At the beginning of the summer I wrote up a list of goals and list of features that would be nice to have. We didn’t get to any of the new features, but we did a pretty good job on the primary goals. One of the first priorities for the fall is to identify appropriately sized projects for each of the groups to work on.
Before we get fully into the projects, I’d like everyone to have a look at the current tickets and choose a few of the smaller ones to solve as a warm-up exercise and as a way to become more familiar with the code base.
Here is the list of features that I’d really like to see in MarkUs. It is highly likely that we will not complete all of them. It is also likely that some of them will be relatively straightforward. What am I missing? I’m sure a few more feature requests will come in as instructors, TAs and students start to use MarkUs.
I’d like to be able find out how students and TAs are using MarkUs. The purpose of collecting data is to
- Improve MarkUs: I hope that by looking at how all users actually use MarkUs, we may be able to identify workflow issues, UI issues, and training issues.
- Help students: We may be able to see student behaviours that we’d like to reinforce or correct.
- Monitor TAs: It would be nice to know whether I should be asking TAs to provide more feedback, or prod them to get started, or congratulate them for work well done.
- Gather evidence that MarkUs is useful. It would be nice to have some evidence that MarkUs actually helps instructors, TAs, and students get their work done.
It is probably not difficult to add log messages to MarkUs. The interesting part of this task is figuring out which events to log, and how to visualize the data.
Add a different marking scheme
The rubric marking scheme works reasonably well, but it would be nice to have at least one other type of marking scheme. I have one in mind, in which the instructor provides a description for a category or criterion and a total number of marks for that criterion. The TA would be asked to enter a number into a box for each criterion.
Simple grade entry
If I’m already using MarkUs to keep track of assignment marks, it might be nice to have a simple interface to enter grades from labs, midterms, or other activities. I imagine this will be another type of “assignment” that the instructor could create.
Messages visible only to instructors and/or TAS
I’d like to be able to add notes to a students’ results in the grader view that are not visible to the student. I imagine these notes to be automatically time-stamped and labeled with the author’s name or id. A TA might use this mechanism to alert the instructor to a problem in a particular assignment. An instructor might use this to make notes of changes made due to remarks. I envision this mechanism as an additional tab in the grader view.
Automated checking of students’ code
This is the big one. A long term goal is to build the infrastructure so that students could submit code and receive immediate feedback. This infrastructure could be used in a few different ways. For example, students might be able to submit their code before the due date and find out how many reference tests pass on their code, or they might get a coverage analysis of their own tests on their code. Another use of this infrastructure would be to give students small exercises that could be automatically checked.
The biggest problem with this feature is that we have to run student code automatically and securely. Having looked into this problem last summer, the best solution seems to be to run the students’ code in a virtual machine. The issues include
- Security: How do we lock down a VM so that we can be sure that student code will not bring down the server, or could access data it should not be able to access (like solution code for example).
- Performance: Is it really feasible to run one or more VMs on our server and still give tolerable response times.
- UI for the instructor: How does the instructor actually set up the tests? How does the instructor specify what type of feedback the students will receive.
- UI for the student: Students need to get useful feedback when they submit their code for testing.
- Backend: We need to keep track of the results every time students submit their code for testing.