MarkUs Blog

MarkUs Developers Blog About Their Project

Automated Testing

with 2 comments

The idea is to integrate some sort of testing infrastructure into MarkUs, so that students get more feedback about their code submitted for assignments. A possible scenario would be the following:

  1. A new assignment is up in MarkUs and students start to work on it
  2. An instructor has written tests for this assignment, which will be run on the code Students submit
  3. Students will get feedback about their code submitted according to the results of those test runs

The question is how to achieve this. I’ll refer to the server or infrastructure which will be executing tests only as test server. Here are some questions we are trying to find the answer for:

  1. Where and how will the tests on students code be run?
  2. Should they be run on a dedicated server? On several servers?
  3. How does MarkUs communicate with the test server?
  4. How to decouple MarkUs from the test server?
  5. Is it a concern if a student compromises the test server either maliciously or accidentally?
  6. What are the required security measures for the test server?
  7. How to satisfy as many programming languages and course models as possible?
  8. What will the test server or test infrastructure look like? Should tests for each MarkUs instance run in separated virtual machines?
  9. Should we use virtual machines? If yes, why? Why not?
  10. How can we secure the test server well enough so that no secret information will be revealed to students? Should the mail system be disabled, network/socket connections be monitored or disallowed? How to secure the test server for courses in which students learn process forking and socket programming?
  11. Will there be performance problems? How to deal with increased load close to assignment deadlines?
  12. Should the amount of test runs be limited for students? If so, how? e.g. 5 test runs per day or max number of test runs?
  13. How much information of tests results should be shown to the student? Number of successes/failures? Detailed test failure messages? Should descriptions of tests be displayed?

These are related projects to get some ideas from. If you know some more, please let me know!

I am looking for ideas and hints regarding automated testing systems, how to integrate them into MarkUs and why to use one approach rather then the other. Please feel free to chime in and comment to this post. Greatly appreciated!

Written by Severin

October 16th, 2009 at 11:00 am

2 Responses to 'Automated Testing'

Subscribe to comments with RSS or TrackBack to 'Automated Testing'.

  1. Hi,

    You have http://www.prologin.org, which is a french website.
    It selects students for a national competition of programming.
    Students have to post their source code on the site and a test will compile & validate their code (C, C++, C#, Python, OCaml, Java et Pascal.)

    Benjamin

    17 Oct 09 at 3:26 am

  2. I would highly recommend the use of sandbox or virtual machines to tackle the problem of students compromising the test server itself.

    Performance will always be a problem with big courses, graph algorithms are a good example and now imagine 400 algorithms running at the same time is going to make the server go crazy. From the top of my mind i would suggest having a queue and only allowing several number of test being ran in the server at a specific time. To deal with deadlines i would consider the easy solution, a warning “As deadline approach the server load might be unresponsive given the amount of users” or something like that.

    I think the question of limiting tests for students is a question for the teachers since they might have different opinions on what to do. Some of my profs didn’t gave us test cases whatsoever because they wanted us to have a better idea on how test ourselves. In some other courses they gave us pretty good ideas on how to test it, so it pretty much depends on each prof.
    I do believe that tests shouldn’t give too much information apart from success/failure and the title of test, the reasons is because any other kind of error will give away good hints of the assignment, and i presume that it is not what we want.

    I have no ideas right now of how to decouple MarkUs and the test server, if i think of something i’ll tell you

    fgarces

    19 Oct 09 at 3:14 pm

Leave a Reply