MarkUs Blog

MarkUs Developers Blog About Their Project

Automated Testing: What still needs to be done?

with one comment

Victor and I met on Saturday to work on the testing framework.  Here are some of the things that we came up with that we want to work on for the rest of the semester.  Victor will be posting a separate article on the security aspects of the testing framework.


Benjamin and Diane did a great job during the summer of developing the test framework.  Most of the functionality is there, but there are still a few features that should be added and some bug fixes.


  • Implement other options for running tests.  Currently we allow a certain number of tokens per day.  We should have the option for allowing certain number of tokens in total and possibly to run tests at scheduled intervals (as is currently used a lot at U of T).
  • Allow students to see the test results that were run by their marker when they were graded.
  • Add a checkbox for “students can run tests” and, if unchecked, the “test” area of the student view will not show up.  (Right now the best we can do is set number of tokens to 0, which does prevent students from running tests, but gives the students the impression that they will be able to run tests until they try and it does not work).
  • Allow an option for tests to be run on all students as part of the collection process before grading.
  • Give instructors access to both raw output and parsed output.
  • Try to give better error feedback if ant fails for some reason (for example, if something has not been correctly installed).  See if we can differentiate when a build fails between an ant/framework problem and a problem due to the student’s code failing to build.  This might be tricky.


  • Fix an issue where students are not able to see their own test results after running a test.
  • Changing the number of tokens allowed for an assignment should update the number of tokens available for each group.
  • Former test results go away for a student if the number of tokens available reaches 0.  They should be still viewable.
  • The test framework should not fail to run if an instructor has not uploaded anything into one of the “test”, “lib” or “parse” folders.
  • Give a better name to the “collect and prepare tests” button for students, or eliminate the need for this button.


We already have some sample testing setups using C, Java and Python that were created by Benjamin.  What still has to be done:

  • Develop a sample testing setup for Scheme.
  • Write documentation for all the examples, so that people can understand what is going on and make modifications.
  • Add parsing to the examples, as none of them include parsing at the moment.

It might be too optimistic to say that we will do everything on this list, as some things could take quite a bit of time.  We’d love to hear your thoughts on what things are important, anything that we missed that needs to be addressed, and any other ideas!

Written by Evan

October 24th, 2010 at 10:36 pm

Posted in Uncategorized

One Response to 'Automated Testing: What still needs to be done?'

Subscribe to comments with RSS or TrackBack to 'Automated Testing: What still needs to be done?'.

  1. Evan, Victor, this looks like a really good assessment. Great work! In my mind, making sure to properly report errors if something is wrong with the basic setup is a very important point. If users don’t get it working and/or get proper error messages, this causes a lot of frustration (as you might have experienced yourself). As for setup, I also think that documenting the required software stack for running basic Java/C/Python/Scheme tests is important.

    Having solid examples illustrating all available features (this includes output parsing/filtering) is also important, I think. We should probably ask Waterloo for Scheme examples. I’m pretty sure they’ll have plenty.

    Try to give better error feedback if ant fails for some reason (for example, if something has not been correctly installed). See if we can differentiate when a build fails between an ant/framework problem and a problem due to the student’s code failing to build. This might be tricky.

    If everything is properly setup and required libraries installed should be testable. I’m thinking of shipping build.xml with an example task which always succeeds if everything is properly installed. We could add some file in a “examples” folder which could be used to run basic test-framework installation tests. Hello world should do for them. If those fail, maybe report something like “Could not run installation tests successfully, please see bla.log and make sure you have x, y, z installed.” x, y, z, may be things like ant-contrib not available, and other common problems. Maybe there’s something built into ant which allows one to query available tasks – if some crucial tasks aren’t there – report an error. Not sure if that would work or if something like that exists, though.

    What do you think?


    24 Oct 10 at 11:28 pm

Leave a Reply