MarkUs Blog

MarkUs Developers Blog About Their Project

Archive for the ‘test framework’ tag

Test Framewok – State of Art

with one comment

Here is the State of Art of the Markus Test Framework. We will describe what is working for each view (Administrator, Grader and Student).

Administrator view

Create a new assignment with test

When we create a new assignment, we can enable the test framwork. If we do so, we can upload a build.xml and, and add some test files (public or private).

This part is now working well, since the routing bug are fixed.

Due date passed

Buttons “Release Marks” and “Unrelease Marks” lead to a 404 error.

“Collect All Submissions” links to RecordNotFound in SubmissionsController#collect_all_submissions.

For each submission, the “Collect Submission” lead to a 404 error (issue

We can’t test if the Admin can run the test because of the errors posted in our last post on this blog.

Grader’s view

We suppose this configuration being applied by the administrator :

  • Tests have been uploaded
  • Tests are public
  • Tokens are available
  • The return date is passed

First, the student has to create a working group or click he wants to work alone.

The Administrator assign a Grader to the Student.

Then, the Grader is supposed to be able to use the testing framework :

  • Click on one of the current assignment.
  • “Can begin grading” is True
  • On clicking on a Group name : the Marking State changes from « Not Collected » (Blue square) to “In progress” (a yellow pencil) and a confirmation message “The submission has been given collection priority. It will be ready soon.” appears on screen.

Then there are problems with collecting all assignments, applying tests or getting results :

  • On clicking on « Collect all your assigned submissions » an error page appears ” ActiveRecord::RecordNotFound in submissionsController#collect_ta_submissions
  • No test can be run and no results loaded.
  • The button “see and comment the result details” is still unavailable.

Student’s view

We suppose :

  • Tests have been uploaded by the administrator on the administrator view
  • Tests are public
  • Tokens are availables
  • The return date is not passed

Steps to use the testing framework as a student:

First, the student has to create a working group or click he wants to work alone.

Then, he can post his work (1. Click on submissions, 2. Click on “add new” and find the file to upload, 3. Click on “submit”).

Then, he can use the testing framework:

  • Click on “assignments”
  • For the first time of use: the button “collect and prepare tests” is available so he can click on.
  • Click on the button “run the test”

What happens ?

At this point, we can launch the tests, they are running but no result is collected and so there is no result available.

Previously, there were some issues of displaying but now all is well displayed (11/30/2011 patch by Guillaume).

If we change some configurations:

  • We suppose that there is no token available or tests are private.

We have the same situation as previously. (Shouldn’t we have no button available to run the tests in those cases?)

  • If the due date is passed: No problem appears. Indeed, the student cannot launch the tests.

Written by gbugnet

December 1st, 2011 at 12:46 pm

Posted in Uncategorized

Test Framework – Bugs reports

without comments

After many hours spent exploring the different possibilities of the MarkUs tool, the team has found, by trials and errors, some bugs where some are still unresolved at that day. We are still performing some tests about the framework in order to assure the stability of the version against any action (here implied more specifically the misused of the tool itself by a grader or a student). It results from these many tests the (incomplete) list, here below. Here follows some of these errors :


  • Situation : The student tries to run his code before the expiration of the date

ActionController::RoutingError in Assignments#student_interface
No route matches {:action=>”render_test_result”, :controller=>”results”, :id=>1

Bug solved (view the related commit)

  • Situation : The student tries to run his code before the expiration of the date (behind the first error)

ActionController::RoutingError in Assignments#student_interface
No route matches {:result=>#<Result id: 5, submission_id: 5, marking_state: “unmarked”, overall_comment: nil, created_at: “2011-11-17 21:42:50”, updated_at: “2011-11-17 21:42:50″, released_to_students: false, total_mark: 0.0>, :controller=>”automated_tests”}

Bug solved (view the related commit)

  • Situation : The admin or the grader tries to see the students submissions

ActionController::RoutingError in Submissions#browse
No route matches {:action=>”edit”, :controller=>”results”, :id=>#<Result id: 4, submission_id: 4, marking_state: “unmarked”, overall_comment: nil, created_at: “2011-11-17 13:31:52”, updated_at: “2011-11-17 13:31:52”, released_to_students: false, total_mark: 0.0>}

Bug solved (view the related commit)

Record Not Found

  • Situation : The graders tries to collect all the submissions after the deadline is passed

ActiveRecord::RecordNotFound in SubmissionsController#collect_ta_submissions
Couldn’t find Assignment without an ID


  • Situation : Let’s assume that a group of students or maybe only one student (according to the condition of the practical work) has sent his work.
    Then the teacher wants to collect the work in order to go on the grading action. Then he presses the “collect and grade” button.

Let’s solve it !

Written by gbugnet

November 20th, 2011 at 4:14 pm

Posted in Uncategorized

Automated Testing: What still needs to be done?

with one comment

Victor and I met on Saturday to work on the testing framework.  Here are some of the things that we came up with that we want to work on for the rest of the semester.  Victor will be posting a separate article on the security aspects of the testing framework.


Benjamin and Diane did a great job during the summer of developing the test framework.  Most of the functionality is there, but there are still a few features that should be added and some bug fixes.


  • Implement other options for running tests.  Currently we allow a certain number of tokens per day.  We should have the option for allowing certain number of tokens in total and possibly to run tests at scheduled intervals (as is currently used a lot at U of T).
  • Allow students to see the test results that were run by their marker when they were graded.
  • Add a checkbox for “students can run tests” and, if unchecked, the “test” area of the student view will not show up.  (Right now the best we can do is set number of tokens to 0, which does prevent students from running tests, but gives the students the impression that they will be able to run tests until they try and it does not work).
  • Allow an option for tests to be run on all students as part of the collection process before grading.
  • Give instructors access to both raw output and parsed output.
  • Try to give better error feedback if ant fails for some reason (for example, if something has not been correctly installed).  See if we can differentiate when a build fails between an ant/framework problem and a problem due to the student’s code failing to build.  This might be tricky.


  • Fix an issue where students are not able to see their own test results after running a test.
  • Changing the number of tokens allowed for an assignment should update the number of tokens available for each group.
  • Former test results go away for a student if the number of tokens available reaches 0.  They should be still viewable.
  • The test framework should not fail to run if an instructor has not uploaded anything into one of the “test”, “lib” or “parse” folders.
  • Give a better name to the “collect and prepare tests” button for students, or eliminate the need for this button.


We already have some sample testing setups using C, Java and Python that were created by Benjamin.  What still has to be done:

  • Develop a sample testing setup for Scheme.
  • Write documentation for all the examples, so that people can understand what is going on and make modifications.
  • Add parsing to the examples, as none of them include parsing at the moment.

It might be too optimistic to say that we will do everything on this list, as some things could take quite a bit of time.  We’d love to hear your thoughts on what things are important, anything that we missed that needs to be addressed, and any other ideas!

Written by Evan

October 24th, 2010 at 10:36 pm

Posted in Uncategorized