This post describes several scenarios that instructors may run into while using MarkUs for automated testing. These scenarios have all happened before at my school. Below each scenario is a description of how to accomplish it, or why it cannot be done using the current test framework design. In the scenarios, “I” refers to an instructor or TA.
1. I want to let my students run as many tests as they want.
- Currently there is no way to do this. When students submit, they will always be deducted a token. If they have no tokens left, the test won’t run. Currently, there is no way to disable this token system. A solution that will probably work is to just set the number of tokens to 999. It’s very unlikely that a student will submit more than 999 times in a single day…
2. I have an assignment with some practise questions that are not for marks. Test scripts for these questions should always run, but they should not affect the student’s marks in any way.
- You can set the maximum mark to 0 and set up the test script so that the
marks_earnedfield is always 0. The students can still see their pass/fail status to see if they got the question correct or not.
3. I want to let the test runs that my students make count for marks. Before the assignment is due, students should be able to run tests only if they have tokens available, and their mark for the assignment is the best mark they’ve earned from any test run they’ve made.
- There is a token system in place, but the current test framework design does not allow you to use the best mark earned as the mark for the assignment.
4. A student (John) submitted his assignment late and so he is getting 0 on the assignment. But, I want to run the test scripts on his assignment anyways to see what mark he would have gotten. I also want John to be able to see which unit tests he passed and failed, for feedback only (the test run should not change his mark, which is 0).
- Right now, when you run the test scripts, MarkUs will automatically update the mark on the assignment. You cannot run the test scripts for feedback only. However, you can use Deductions to manually remove any marks earned. You can also manually run the test scripts yourself, and commit the results to John’s repository so that he can see them.
5. I need to access the student’s MarkUs username in my test scripts. How can I do this?
- Right now, you can’t. One way that this feature can be implemented in MarkUs is to pass the student’s username to
testrunner.rbas an environment variable. This should be easy to do; just modify the ssh call in
6. I have some test scripts which students should be able to run without needing any tokens, and some test scripts that do require tokens. How should I set things up in order to do this?
- You cannot do this in the current test framework design.
7. Many of my courses use “public tests” and “release tests”. Before the assignment is due, students can run the public tests, which check the students’ code for trivial things (like syntax errors). After the assignment deadline, the TA will run the release tests, which do a very rigorous and thorough testing of the students’ code. Students should not have access to the release tests.
- For the release tests, you can set the “Display Settings” to “Do not display” to hide test results from the students. You should also uncheck “Run On Request” and “Run On Submission” so that students cannot run the test scripts.
8. I have an algorithms assignment and I want to set different timeouts for each unit test. I also want to set some memory usage limits. How can I do this?
- You will probably want to define the timeouts and memory limits in your test scripts. You can also try editing
testrunner.rb. You will have to edit
testrunner.rbif you want a timeout of more than 10 minutes, because the 10 minute timeout is hard-coded in
9. I want to prevent students from using certain libraries and/or function calls. How can I do this?
- This will have to be done in the test scripts that you upload. For example you can try stripping out any
requiredirectives in the student’s code (depending on which programming language the student is using). Or, you can define your own versions of the prohibited functions so that they throw an error when called; if students use these prohibited functions, your error will be thrown.
10. I have an assignment X which is a prerequisite for another assignment Y. Specifically, if students do not pass assignment X, then they should automatically get 0 on the testing portion of assignment Y, regardless of what they submitted for Assignment Y.
- In the test script that you upload, you will have to somehow check that the student has passed assignment X. I don’t think you can do this right now.
11. It is after the assignment deadline, and I ran my test scripts on all the students’ submissions. However, I later discovered that there was a mistake in my test scripts (this is the instructor’s fault, not the students’). What is the proper way to fix things?
- First, fix the bugs in your test scripts. Then upload the new test scripts from the Test Framework Settings / Config UI page. Then re-run the tests on all the students. During this process, MarkUs needs to somehow replace the mark earned from the old, broken test script with the mark earned from the new test script.
12. I want to display a long, custom message when students request a test run. Since it’s a graphics assignment, I also want to display a picture in the message. How can I do this?
- The Description field is the same for every student, so you can’t use that. You can try using the “Expected output” field to display the message. You cannot include an image in the output (even if you include an HTML
imgtag, Rails will escape the tag).
13. I want to see how many test runs a student has requested and what the results were.
- You can switch roles and log in as that student. Then you should be able to see the desired information.
14. For an algorithms assignment, students have to sort numbers using heapsort. If they use a different sorting algorithm, they will get 0 marks. The TAs will manually check that the students have used the correct algorithm, and deduct marks accordingly if they don’t. The test scripts cannot check that students have used heapsort. How can the TAs override the test results?
- Currently, the test results cannot be manually changed. However you can simulate the effect by using Deductions.