MarkUs Blog

MarkUs Developers Blog About Their Project

Archive for March, 2013

Punchlines – UCOSP Winter 2013 – Week 10

without comments

Oussama

Status:
– Related to issue 1035:
– added internatiolization support for label of totals column
– tested changes in both English and French evironments successfully
– within he students view, changed the code that displays the list of of grade entry forms so that the total column is only displayed if the current user is either an admin or a TA

Roadblocks:
– Waiting for changes to be reviewed

Next:
– Not sure exactly as the my approach to the initial feature I planned to work on (allowing TAs to view only student they are assigned to) turned out to be incorrect.

Alysha

Status:
– Submitted pull request-1058 for fix to issue 1027 (the update of the code base to integrate possible existence of a remark result)
– Updated blog post to reflect changes

Roadblocks:

None

Next Week:
– Work on issue 1059  : carrying over bonus/penalty marks on to a remark result (may also be working on issues 1020 and 1013 which are slightly related – have to do with bonus/penalty mark UI section not updating/displaying accordingly)

Mike Wu

Status:
– Completed issue #1028 (highlighting old and new marks in the rubric and populating new remark result with old marks) and submitted a pull request (#1060).

Roadblocks:
– It took me several tries to find a way of properly duplicating marks.

Next week:
Improve the patch #1060 if necessary. (merged)
– Review pull #1058.
– Start working on #1029.

Daniel

Status:
– Had to scrap my attempt from last week of modelling the assignments/id/submissions route. It made searching by group-name and users far too complex because of the resulting XML and JSON (lots of includes, too many nested layers). Instead, I’ve been working on using groupings for that sub collection, and calling it the route assignments/id/groups, so that users don’t need to understand the different class relations (groups/groupings, etc) I could have used groups, however, it would have resulted in an equally complex response as seen with assignments.

Roadblocks:
– I exchanged a few emails with Hanson Wu on the dev list. As of right now, I’m including a new method definition in grouping.rb that allows me to directly access group_name. However, because it’s not an attribute, the get_collection method I wrote in MainApiController doesn’t work out of the box for these nested routes with lots of includes. I’m trying to extend it to be able to achieve something like “collection.find_all { |resource| resource.send(key) == value }”, which compares a method named key to a specific value for each item in a collection. collection, in this class, is an instance of ActiveRecord::Relation, so I’ll have to see what’s available to achieve this.

Next week:
– Solve this problem, and continue work on the assignments routes to finish it up.

Daryn

Status:
– Completed updated_collected_submissions call in the assignments dashboard
– Updated all changes to be compatible with prototype
– Working on dropdown menu in assignments section

Roadblocks:
None

Next Week:
– Continue working on dropdown in assignments section.

Marc

Status:

Reintegrated Prototype back into the current changes involving jQuery so that they work side by side. Did some work on adding additional features to table views throughout the application.

Roadblocks:

Some of the features, such as searching, I am attempting to implement in the Data Table require some changes to the corresponding rails controllers. After looking at these changes, it is unclear on

Next Week:

Wrap up adding extra features to table views and attempt to merge in some changes from MarkUs master into the prototype to jquery feature branch.

Ian

Status:

Allowed students to submit automated tests. Created a simple UI that shows the marks they received on the automated tests.

Roadblocks:

None.

Next Week:

Expand the UI so that students can see all of the information about the test results.

Mike Stewart

Status:

Have the grade entry grader tab UI modified to assign graders to users.

Roadblocks:

Still trying to figure out the correct way to set up the models.

Next Week:

Complete the grader assignment.

Nick

Status:

Moved perform and the functions it depends on to a new job class.
Added configuration settings to let admin choose Redis server and queue name.
Worked on choose_test_server function.

Roadblocks:

None.

Next Week:
Keep working on displaying test run progress.

Written by Marc Bodmer

March 27th, 2013 at 11:01 am

How to Succeed in UCOSP Projects

without comments

Success in a UCOSP project, or any open source project can be achieved using some of the following methods I have used myself, and methods I have observed other people using.

– Time Tracking
It is crucial to keep a log of whatever it is that you have been working on and in which direction you want to head in next.

– Constant Communication
Keep up to date with meetings and mailing list emails so you can understand what the rest of your team is doing. This helps in many ways. Also, ask your mentors or any other projects members involved if you ever have any questions regarding your project. They have most likely been involved in the project for quite a while, and can provide you with valuable advice.

– Don’t give up easily
There are times when you may run into problems such as not being able to understand what is going on in the application or not knowing how to write a specific code in a new language. Don’t give up! Ask others in your communication channels for help, and try to understand what is going on.

Life and Career Lessons Learned from UCOSP

I have learned a couple career lessons that I will keep in mind in my future jobs.

– Time Estimates
This was definitely the biggest lesson. Before I started on my term goal, I had done a lot of research into my topic. I had originally estimated that I could complete the goal before the end of the semester, but as I approached the last few weeks I knew that this could not be done. Essentially, make sure the goal you are attempting to achieve is realistic given your time frame, and in case you cannot achieve your goal have some sort of way to integrate the work you have done into the main project so that others in the future can continue on your work.

– Communication is Key
In the case where you are working with a partner on any sort of project, it is very important to have a central place where you can keep track of changes and look at the work that the other person has done. You can use a project management tool, version control, or even just a simple spreadsheet for this.

I also have a bit more written on my personal blog as well.

Written by Marc Bodmer

March 26th, 2013 at 12:55 pm

Punchlines – ECN PAPPL 2013 OptimMarkus – 2013-03-20

without comments

Claire:

Gaëtan:

  • submitted pull request to add unit tests for issue #559
  • worked on issue #230: suggested UI design, dove into code

 

Written by G_Girin

March 20th, 2013 at 8:51 am

Posted in Developer Essentials

Tagged with , ,

Punchlines – UCOSP Winter 2013 – Week 9

without comments

Mike Wu

Status:

  • Working on issue #1028:
  • Implemented colour-coding for the rubrics: green for selected marks and yellow for old marks.
  • Set unedited criteria to old marks upon changing marking_state to complete.

Roadblocks:

  • Need to decide on some UI details:
  • For example, when starting on a remark request, does the total shown on the page start with the original total mark and them adjust to remarks or start with 0 and reflect the actual remark result in the database (which is not populated with the original result at the beginning)?
  • Populate extra marks (i.e. bonus/deduction)? could and probably should be done in a separated task I think.

Next week:

  • Sort out all the UI details and submit a pull request for #1028

Marc

Status:

  • Reached a point where my level of progress can now be integrated into the master MarkUs branch. I have begun to work on making jQuery compatible with Prototype by using jQuery in a noConflict mode. I have also been making sure none of my changes have broken anything throughout the MarkUs application.

Roadblocks:

  • Have a problem with some of the features of the new data tables for displaying students, but should be resolved once Prototype is renabled.

Next week:

  • Keep working on getting jQuery to work with Prototype with jQuery being in a noConflict mode.

Alysha

Status:

  • Worked on fix for issue 1027 (going over code base and updated it to use appropriate result related methods)

Roadblocks:

  •  takes some time to understand what certain areas of code are doing to know which is the appropriate fix for it

Next week:

  • finish up fix for issue 1027
  • work on some remark related bugs

Ian

Status:

  • Added code to allow for unlimited tokens for test scripts. Tokens are currently not used anywhere so it’s just a boolean somewhere waiting to be used when we implement student run tests. Added code to choose between using the latest or best mark for all test scripts.

Roadblocks:

  •  None.

Next week:

  • Possibly make a full test script or work on getting students to be able to run test scripts.

Mike Stewart

Status:

  • Initial UI for the grader tab for grade entry is complete, with all of the kinks worked out.

Roadblocks:

  • None.

Next week:

  • Modify the functionality to assign graders on a many-to-one basis to users, instead of to groups.

Oussama

Status:

  • Submitted pull request for issue 1035

Roadblocks:

  • Waiting for pull request to be merged

Next week:

  • Keep working on grades sheets’ viewing functionality for TAs

Nick

Status:

  • Worked on displaying test run position with Redis and Resque

Roadblocks:

  • Redis-status requires the perform method to be a class instance method, but right now it’s just a module method. It might take a lot of code refactoring to change it.

Next week:

  • Keep working on displaying test run position and test run progress

Daryn

Status:

  • Began making changes to make jQuery code noConflict() to be compatible with Prototype
  • Working on update_collected_submissions function used in the Assignments dashboard.

Roadblocks:

  • None

Next week:

  • Meeting with Marc this Friday to regroup on progress and goals for the end of the semester

Daniel

Status:

  • Worked on submissions, and integrating the existing API. Also started documentation.

Roadblocks:

  • Thinking of including small examples for working with the API. What language would be preferred out of Java, Python and Ruby?

Next week:

  • Finish up the assignments routes.

Written by Mike Stewart

March 20th, 2013 at 12:55 am

Punchlines – UCOSP Winter 2013 – Week 8

without comments

Daniel

Status:

Roadblocks:

  • Hoping to get the above pull request merged. 🙂

Next week:

  • Work on a submissions resource and refactoring the existing submission downloads controller. Start thinking about what I’ll do for the API Documentation.

Oussama

Status:

  • Resolved issue 1035: changed grades spreadsheet by adding a hide/show hyperlink under “Total”, clicking on this link invokes a JavaScript function that toggles the visibility of all totals.
  • Need to confirm the above functionality with Karen before submitting a pull request.
  • Working on going graders more options in terms of which students records they’re allowed to view in grades sheet.

Roadblocks:

  • None.

Next week:

  • Continue working on grades sheet.

Alysha

Status:

Roadblocks:

  • Would be nice to have pull request-1052 reviewed and merged so i can use the helper methods I implemented to tackle an overall update of the code base with these methods (ie: issue 1027)

Next week:

  • Review pull request as needed
  • Work on issue 1027 once pull request is reviewed and merged
  • Work on other issues logged that aren’t dependent on pull request

Nick

Status:

  • Wrote a short blog about how the test framework can be used in several different real-world situations.
  • Played around with resque and resque-status. Resque-status is this new gem I found! It is a plugin for Resque. It lets you view the progress of one, single test run. So it will tell you things like “This test run is 40% complete” but it will not tell you that “This test run is queued and in position #46”.

Roadblocks:

  • I sent an email around asking about how the test_results table should be re-designed. I haven’t heard back yet, because I sent the email pretty late.
  • Although Resque-status lets us view the progress of one test run, I’m not sure if I am allowed to use it. It’s a new gem that we’ll have to add. Also I’m not sure how to install the resque-status files into MarkUs’s lib/ folder.
  • I still need to find a way to get the position of a test run in the resque queue.

Next week:

  • If I get feedback on the test_results table, then I can work on that.
  • Alternatively, I can keep working on Resque queues. I still need to find a way to get the position of a test run in the resque queue.

Marc

Status:

  • Working on advanced features of table views such as batch operations and pop up boxes for actions within tables (Specifically, working on the students table).

Roadblocks:

  • None so far.

Next week:

  • Going to meet up with Daryn for another code sprint working out any problems encountered thus far. We will also attempt to combine some of the seperate elements we have been working on with eachother. Overall, I will keep working on refactoring.

Daryn

Status:

  • Completed the admin function to “view as” a student. (icon on the top right, left of logout)
  • That sums up the Main views
  • Moved into the Assignments views

Roadblocks:

  • Figuring out the purpose of the update_collected_submissions function as it is used by a periodical prototype function

Next week:

  • Meeting Marc again next week for another day of hacking and consolidate our work.
  • Continue migrations.

Mike Wu

Status:

  • Fixed the bug where the UI states stored in the cookie were not properly restored (#1036).
  • Proposed a design for pre-populating the rubric with original result for remarking (#1028).
  • Started looking into the annotation component for #1029.

Roadblocks:

  • None.

Next week:

Ian

Status:

  • Fixed issues 102210231024, and 1025 (which is the same and issue 1015). All fixes have been merged.

Roadblocks:

  • Not sure what to do next.

Next week:

  • Possibly make a full test script. Work with resque to determine where tests are in the queue (if Nick hasn’t already started this).

Mike

Status:

  • Completing the grader tab for grade entry.

Roadblocks:

  • Working to figure out what database schema changes will be necessary.

Next week:

  • Have a working UI of the grader tab, and hopefully much of the back-end functionality in place.

Written by Daryn Lam

March 13th, 2013 at 1:30 am

Posted in Status Reports

Using the Test Framework Infrastructure in Different Scenarios

without comments

This post describes several scenarios that instructors may run into while using MarkUs for automated testing. These scenarios have all happened before at my school. Below each scenario is a description of how to accomplish it, or why it cannot be done using the current test framework design. In the scenarios, “I” refers to an instructor or TA.

1. I want to let my students run as many tests as they want.

  • Currently there is no way to do this. When students submit, they will always be deducted a token. If they have no tokens left, the test won’t run. Currently, there is no way to disable this token system. A solution that will probably work is to just set the number of tokens to 999. It’s very unlikely that a student will submit more than 999 times in a single day…

2. I have an assignment with some practise questions that are not for marks. Test scripts for these questions should always run, but they should not affect the student’s marks in any way.

  • You can set the maximum mark to 0 and set up the test script so that the marks_earned field is always 0. The students can still see their pass/fail status to see if they got the question correct or not.

3. I want to let the test runs that my students make count for marks. Before the assignment is due, students should be able to run tests only if they have tokens available, and their mark for the assignment is the best mark they’ve earned from any test run they’ve made.

  • There is a token system in place, but the current test framework design does not allow you to use the best mark earned as the mark for the assignment.

4. A student (John) submitted his assignment late and so he is getting 0 on the assignment. But, I want to run the test scripts on his assignment anyways to see what mark he would have gotten. I also want John to be able to see which unit tests he passed and failed, for feedback only (the test run should not change his mark, which is 0).

  • Right now, when you run the test scripts, MarkUs will automatically update the mark on the assignment. You cannot run the test scripts for feedback only. However, you can use Deductions to manually remove any marks earned. You can also manually run the test scripts yourself, and commit the results to John’s repository so that he can see them.

5. I need to access the student’s MarkUs username in my test scripts. How can I do this?

  • Right now, you can’t. One way that this feature can be implemented in MarkUs is to pass the student’s username to testrunner.rb as an environment variable. This should be easy to do; just modify the ssh call in automated_tests_helper.rb.

6. I have some test scripts which students should be able to run without needing any tokens, and some test scripts that do require tokens. How should I set things up in order to do this?

  • You cannot do this in the current test framework design.

7. Many of my courses use “public tests” and “release tests”. Before the assignment is due, students can run the public tests, which check the students’ code for trivial things (like syntax errors). After the assignment deadline, the TA will run the release tests, which do a very rigorous and thorough testing of the students’ code. Students should not have access to the release tests.

  • For the release tests, you can set the “Display Settings” to “Do not display” to hide test results from the students. You should also uncheck “Run On Request” and “Run On Submission” so that students cannot run the test scripts.

8. I have an algorithms assignment and I want to set different timeouts for each unit test. I also want to set some memory usage limits. How can I do this?

  • You will probably want to define the timeouts and memory limits in your test scripts. You can also try editing testrunner.rb. You will have to edit testrunner.rb if you want a timeout of more than 10 minutes, because the 10 minute timeout is hard-coded in testrunner.rb.

9. I want to prevent students from using certain libraries and/or function calls. How can I do this?

  • This will have to be done in the test scripts that you upload. For example you can try stripping out any import or require directives in the student’s code (depending on which programming language the student is using). Or, you can define your own versions of the prohibited functions so that they throw an error when called; if students use these prohibited functions, your error will be thrown.

10. I have an assignment X which is a prerequisite for another assignment Y. Specifically, if students do not pass assignment X, then they should automatically get 0 on the testing portion of assignment Y, regardless of what they submitted for Assignment Y.

  • In the test script that you upload, you will have to somehow check that the student has passed assignment X. I don’t think you can do this right now.

11. It is after the assignment deadline, and I ran my test scripts on all the students’ submissions. However, I later discovered that there was a mistake in my test scripts (this is the instructor’s fault, not the students’). What is the proper way to fix things?

  • First, fix the bugs in your test scripts. Then upload the new test scripts from the Test Framework Settings / Config UI page. Then re-run the tests on all the students. During this process, MarkUs needs to somehow replace the mark earned from the old, broken test script with the mark earned from the new test script.

12. I want to display a long, custom message when students request a test run. Since it’s a graphics assignment, I also want to display a picture in the message. How can I do this?

  • The Description field is the same for every student, so you can’t use that. You can try using the “Expected output” field to display the message. You cannot include an image in the output (even if you include an HTML img tag, Rails will escape the tag).

13. I want to see how many test runs a student has requested and what the results were.

  • You can switch roles and log in as that student. Then you should be able to see the desired information.

14. For an algorithms assignment, students have to sort numbers using heapsort. If they use a different sorting algorithm, they will get 0 marks. The TAs will manually check that the students have used the correct algorithm, and deduct marks accordingly if they don’t. The test scripts cannot check that students have used heapsort. How can the TAs override the test results?

  • Currently, the test results cannot be manually changed. However you can simulate the effect by using Deductions.

Written by Nick Lee

March 12th, 2013 at 6:30 pm

Posted in Automated Testing

Punchlines – ECN PAPPL 2013 OptimMarkus – 2013-03-11

without comments

Claire

Gaëtan

Written by G_Girin

March 11th, 2013 at 4:18 pm

Posted in Developer Essentials

Tagged with , ,

Punchlines – UCOSP Winter 2013 – Week 7

without comments

Status reports for Week 7 (February 27 to March 6, 2013).

Daniel

Status:

  • Mostly completed the assignments part of the API. Expect a pull request in the next couple days once I have time to clean up my commits.

Roadblocks:

  • Waiting for a pull request to go through. It’s already been tested and reviewed by a few others. The modifications to main_api_controller.rb are necessary for future changes to the API.

Next week:

  •  Work on completing another set of routes and api controllers.

Daryn

Status:

  • Worked with Marc on Friday to tackle .rjs file problems.
  • Resolved the issue by replacing files with .js.erb files instead.

Roadblocks:

  • Still looking for some prototype equivalent methods in jQuery.

Next week:

  •  Working on admin functionality to act as different types of users (the button on the top right).

Marc

Status:

Roadblocks:

  • Spent a lot of time figuring out what the best plugin was to replace LivePipe tables was.
  • Also, had to figure out how MarkUs was getting data with Prototype Ajax calls and apply it properly in jQuery.

Next week:

  • Integrate Daryn’s changes regarding modals and make sure modals work correctly and submit the proper form data where applicable.

Alysha

Status:

Roadblocks:

  • Waiting for my pull request (1026) to work on issue 1027 and related issues (updating code to use remark result where applicable).

Next week:

  • Work on bugs I’ve logged either not related to remark or not blocked by merge of pull request. Revise pull request 1026 as needed.

Mike Wu

Status:

  • Planned some UI features for remark with Karen (#1028 and #1029)
  • Reviewed pull request #1026 by Alysha

Roadblocks:

  • All remark features/bugs depend on pull #1026, so it would be nice to get it in soon.

Next week:

  • Start working on #1029 (distinguishing remark annotations from original ones)
  • Fix some bugs once #1026 is merged.

Oussama

Status:

  • Working on implementing the following two changes in terms of how the grades entry sheet is viewed by TAs
  1. displaying the relevant TA name in an added column
  2. adding a menu whereby TAs can choose to view all students or only students that belong to one of their groups
  • Started making changes in UI

Roadblocks:

  • None

Next week:

  •  Work on functionality of above feature as well as any possible DB changes

Ian

Status:

  • Made it so test script max marks will be added to the total marks for an assignment, and the marks earned will be added to the marks earned for an assignment.
  • Added row to summary table for the assignment results view so that user can see what they got total on the test scripts.
  • Changed CSV output to added new column for test script marks.

Roadblocks:

  • None

Next week:

  • Finish up and make a pull request for the stuff I’ve been doing. Work on fixing Issues 1022-1025.

Nick

Status:

  •  Fixed a few issues with my Test Framework’s Config/Settings page and asked for feedback on it.
  • Found some bugs/issues and posted them on Github.
  • Started working on the Test Results tab, which shows the test results to the graders when they are marking.

Roadblocks:

  • I want to display the test results by test run, but the tables test_results and test_script_results do not provide me with the necessary information. Given a row from one of these tables, I cannot tell which test run it came from.

Next week:

  • Continue working on Test Results tab.
  • I want to start looking at how to display the progress of a test run to the admin/instructor.

Written by Nick Lee

March 6th, 2013 at 7:07 am

Posted in Status Reports

Database Schema Change: Submissions and Results

without comments

For the past while, I’ve been working on updating the database schema for the relationship between Submissions and Results. Originally, Submissions was created so it had a one to one relationship with Results. However, a feature was later added so that students could submit remark requests, requiring more than one result to be related to a single submission. The schema was then changed so that submissions had the following relation:

has_one    :result, :dependent => :destroy
belongs_to :remark_result, :class_name => “Result”

Where ideally, we would want a has_many relationship. With the previous implementation, an error appeared about old results sometimes not showing up (documented in this blog post). A quick fix was implemented but it was dependent on database ordering and time stamps. I took the following steps to improve this implementation and schema:

1. The schema
– I updated removed the belongs_to remark_result relation, and changed the has_one result to has_many
– This change means that submission.results will return an array of results and that we can no longer call submission.result and submission.remark_result to get the results

2. Helper methods
– I implemented two helper methods: get_original_result and get_remark_result that would return the appropriate result for the submission
– These helper methods utilized the already existing field: remark_result_id in the Submissions table in order to differentiate between the two results (this removed our dependency on ordering/timestamps)

3. Updating the code base
– I updated the way results were created so that they would add to the array of results, and I had to manually fill in the remark_result_id column of the Submissions table when a remark result is created (since the belongs_to dependency no longer exists)
– All areas of the code that were using either the result or remark_result had to be updated with the helper methods and then tested to ensure functionality did not change

Next Steps
Originally, the suggestion on issue-941 (the schema update issue) was for the two methods to be get_result and get_old_result. The idea behind the methods was that get_result would return the remark result, if it exists, otherwise return the original result. Then, get_old_result, if a remark result exists, would return the original. When I implemented this approach, I realized several errors and differing functionality. Comparing it to the original code, I realized that the remark result was actually not being used in many places in the code base and that the original was result was generally used most of the time (without checking if a remark exists). Another problem is that in some cases you only want the latest completed result to be displayed rather than just the latest existing result.

So, the approach we took was to update the schema without changing any functionality (by using helper methods that would return the same type of result – original or remark – each time), and then follow up by going over sections of the code and updating them so that they will be using the appropriate result in that case. This issue has been opened and can be found here.

My suggestion for implementation is to create another helper method, get_latest_result, that would utilize the current helper methods get_original_result, get_remark_result, has_remark?, and remark_submitted? to return the latest result (note: only use remark result if remark request has been submitted). We would have to determine areas where this method would be appropriate to use instead of always using the original result.

Update
As per the “Next Steps” section, I have added in helper methods and then gone over the code base that uses results, updating method calls where appropriate. I implemented two helper methods: get_latest_result and get_latest_completed_result for the update. The former will return a remark result only if the remark request has been submitted, and the latter will only return a result whose marking state is “completed”. These changes can be found in pull-request 1058.

Written by Alysha Kwok

March 5th, 2013 at 1:07 pm