Archive for August, 2010
After a summer of hard work, we have finally released MarkUs 0.9.0!
Changes for MarkUs 0.9.0:
* Multiple bug fixes
* REMOTE_USER authentication support
* Redesigned manage groups and graders pages
* Added in-browser pdf display and annotation
* New batch submission collection
* Improved loading speed of submissions table
* Added ability to assign graders to individual criteria
Unfortunately, we did not have a full-time French speaking person to help update the translations. Thus 0.9.0 was released without French translations. Our friends in Nantes are currently working on getting translations up to speed, and 0.9.1 is expected shortly with full French support restored.
The simulator creates random data to fill in the database.
The simulator creates by default 2 assignments with random due date and total mark. For each assignment, it creates some Tas, students and groupings and then assign the groupings to the Tas. Each grouping may have up to four submissions and it might have been partially or completely marked.
If there is no admin account in the data base, the simulator creates two: “a” and “reid”.
There are four environment variables that can be provided to the simulator:
– NUM_OF_ASSIGNMENTS: it indicates how many assignments the simulator should create. The default value is 2.
– PASSED_DUE_DATE: if it is true, then all the created assignments by the simulator will have their due date passed; if it is false, then the created assignments have due dates two months ahead from the date the assignments were created. If this environment variable was no provided, then the assignments due date randomly can be passed or not.
– NUM_OF_TAS: it is the number of tas each assignment can have. If it was not provided, the number of tas is at least 1 and at most 3.
– NUM_OF_STUDENTS: it is the number of students each ta can have. If it was not provided, the number of students is at least 10 and at most 15.
MarkUs has a new elegant dashboard. The dash board displays
– A description of the assignment.
– Average and median (both in percentage and out of the total mark)
– Number of groupings that submitted the assignment and how many had been marked.
There are three graphs:
1. TA progress: each TA which is assigned to work in this assignment has stacked bars to show how many groupings have he/she completely marked and how many left.
2. Assignment marks: there are ten bars. The bars are labeled form 0-10% … 90-10% and each bar presents the number of groupings that have marks fall in that category.
3. Number of submission: each bar presents number of submissions done by each groupings.
If a graph does not have any data to display (for example if no marking has been done or no submission done by any grouping) then the relative graph is not going to appear.
The assignments are ordered by the last time they were updated.
This summer, one of my more interesting tasks was to speed up MarkUs, namely to hasten the loading of the dreaded submissions view. Going into this, I had a semi-decent understanding of rails, and very little knowledge of databases. I had started by looking at ticket #174 about indexing foreign keys. Although most of the indexing has already been done, I learned about a certain ‘N+1 query’ problem that is common in many applications using databases.
A short description of the problem:
Picture a ‘student’ object having N ‘lecture’ objects that they attend daily. Our application keeps track of all the students, and stores them in a database. Say we want to change the time of each ‘lecture’ for a given ‘student’. Our application logic would loop over student.lectures and perform an action on each ‘lecture’. The app performs 1 database query to select the ‘student’, and N more database queries, one to select each ‘lecture’. This is the essence of the problem, as database queries are expensive.
In the case of MarkUs, the submissions table had hundreds of rows. Each row represented a grouping, and in order to populate the table (each column – group name, final mark, grace credits, etc… – required more information about the grouping), each row needed about 7 extra queries. With database queries in the thousands, it could take over a minute for the page to load.
The solution for this is Rails’ eager loading. When you know that with each grouping you will need info about their submissions, results, members, etc… you simply tell Rails to include all that information in a single large query. Eager loading in rails is done by adding the :include option option to any ‘find’ method calls. Simply by preloading all the required information for the submissions view, I was able to speed up the loading of the page by several times. Of course redundant eager loading can actually slow down the application, but a discussion about eager loading is not the point of this post.
Finding N+1 Queries in Code:
In ticket #174 Mike Gunderloy suggests using a gem called bullet to find the queries. It is very simple to install, and gives feedback right in the console running the server. The gem notices when you have N+1 queries, gives you a specific :include statement and tells you where to place it in the code. I began using this gem and it did work a few times, however in many cases it simply led me in circles, first saying to add an :include somewhere to get rid of N+1 queries, and then saying I am using redundant eager loading in the same place.
Bullet was something I used at first to get me started, however I discontinued using it because the console already displayed to me each SQL query that MarkUs was doing, and I found this to be helpful enough. N+1 queries usually happen on pages which are iterative in nature (large tables for example), but are not limited to them. A good way to detect them is to look at the console, and see if there are a lot of very similar database queries that differ only by the id. For example, 200
Grouping Load (1.5ms) SELECT * FROM “groupings” WHERE (“groupings”.”id” = some number)
would suggest that it is possible to load all the groupings with one query (usually by adding an :include somewhere).
Apart from N+1 queries, there were some other issues which slowed down the application, but most of them had to do with excessive database queries. For example, in the submissions view, each grouping’s total mark would be re-calculated every time the page was loaded (the calculation involved several queries per grouping). This was a case where it was much better to simply add a ‘total_mark’ column in the groupings database table, which would be updated every time a mark is changed (thanks to Rails’ wonderful before_save callback).
I have discussed some of the more significant issues I have encountered, however these are not the only ones that will happen. When tweaking our code to optimize it, I was always aiming for as little database queries as possible. I found that in many cases, it is more desirable to have a bit of extra processing done if it involved reducing the number of database queries.
Testing Framework Overview:
Upon grading a student’s assignment, an instructor may want to execute a set of tests against the student’s code to validate the correctness of the code. The instructor may also have various ways of handling pass/fail cases. The testing framework has been implemented to allow instructors to perform such operations. Using Ant, a Java-based build tool, as the core behind this testing feature, instructors will be allowed to upload their test files, any associated library files necessary for the tests to run, as well as any parsers they may wish to execute against the output, and control the compilation, testing and output parsing through a simple build file.
Part 1 of the testing framework consists of the Test Framework form from the Assignment view which allows a user to upload the Ant, test, library and parser files required to execute tests against students’ code. Part 2 of the testing framework consists of the actual execution of the tests from the Submission view. Part 3 is the student’s Assignment view where the student will be allowed (assuming they have tokens available to them) to execute the tests against their code before submitting their assignments.
The following documents how to create new test files associated to an assignment, as well as how to update and delete the files and the associated expected errors messages that will appear in response to invalid input.
Creating new test files:
1) Create a new assignment and click ‘Submit’
2) Click on ‘Testing Framework’ from the sub menu and you will be redirected to the Test Framework view (by default, the test form is disabled)
3) Check ‘Enable tests for this assignment’ and the test form will be enabled
4) Fill in the ‘Tokens available per group’ as needed. This value represents the number of times a student may be allowed to execute the tests against their code before submitting their assignment.
5) Add the required Ant files, build.xml and build.properties
6) Click on ‘Add Test File’, ‘Add Library File’, ‘Add Parser File’ to add the appropriate test, library and parser files needed to run the tests respectively. Test files represent the test cases that will be executed against the code. Library files are any necessary files required by the tests to execute appropriately. Parser files (if specified) will be executed against the output from the tests to extract and manipulate the output as needed.
7) Click ‘Submit’ to save the changes
*Optional* You can check the test file ‘private’ box to indicate a particular test cannot be executed by the student.
1) Verify table ‘test_files’ to ensure the files were saved correctly
2) Verify table ‘assignments’ to ensure ‘enable_test’ and ‘tokens_per_day’ were saved correctly
1) Verify the following folder structure in the TEST_FRAMEWORK_REPOSITORY:
– test – TestMath.java
– lib – junit-4.8.2.jar
– parse – Parser.java
Updating test files:
1) Update any test, library or parser file already uploaded and click ‘Submit’ (able to replace with the same filename or overwrite with a different filename)
1) Verify updated file has been saved successfully
1) Verify updated file has been overwritten successfully
(If the replaced file has a different filename, it will be deleted from the directory. If it has the same filename, it will simply be overwritten.)
Deleting test files:
1) Delete any test or library file by checking the ‘Remove’ box and click ‘Submit’
1) Verify file has been removed from the database successfully
1) Verify file has been removed from the filesystem successfully
Negative Test Cases:
– Filename Validation –
- cannot be named anything but ‘build.properties’
- cannot be named anything but ‘build.xml’
test, library and parser files:
- cannot be named ‘build.xml’ or ‘build.properties’
- cannot have the same filename as any other existing test file belonging to that assignment
- if you attempt to upload two files simultaneously with the same filename, an error will be displayed
- cannot be blank (File entry will simply be ignored)
– Assignment Validation –
- cannot be negative
- if you update the test form in any way but then disable tests for the assignment and click ‘Submit’, those new changes will not be saved