Here is a shell script, which does the heavy lifting of a proxied setup including basic authentication. This has been tested on Debian and Ubuntu. It might also work on some other distributions with some modifications. The necessary steps are as follows:

1. cd to a clean checkout of MarkUs
2. edit config/environment.rb such that it includes
   config.action_controller.relative_url_root = “/markus”
3. edit setup_apache_auth_proxy.sh and update MARKUS_RAILS_ROOT so
   that it points to your MarkUs checkout.
4. go to your MarkUs checkout and do ./script/server (the usual
   thing)

Become root (e.g. $ sudo -s)
# chmod +x setup_apache_auth_proxy.sh
# ./setup_apache_auth_proxy.sh

Finally go to http://testhost.com/markus (It should ask for authentication; use username=markus_test password=test). You should see a standard MarkUs login screen now. Happy proxy testing/developing!

After doing some research, i found a blog post at http://www.salas.com/2009/06/19/geeky-rmagick-and-memory-leaks/. This talks about the issues of using RMagick, namely the memory leaks it creates as ruby’s garbage collector doesn’t play nice with it. Upon further investigation, I found that this leak made MarkUs use 3 times as much memory as was needed for conversion. Furthermore, the post described a function designed to free memory. However due to the nature of RMagick, this meant that we would be using at least twice necessary memory used to convert the files by ImageMagick, as RMagick would save an object corresponding to an image, and create a new one for every manipulation performed on it. Thus there was always an unavoidable window after a modification and before the original image object can be destroyed to free its memory.

As a result, I have opted for issuing a direct call to ImageMagick’s convert function via ruby.  This is a much better alternative, as it performs all the modifications in one line. The memory use issue still wasn’t solved here though. After more digging I found this formula on ImageMagick’s site http://www.imagemagick.org/script/advanced-unix-installation.php

Memory use = (5 * Quantum Depth * Rows * Columns) / 8

Where Rows and Columns were the size of the pdf file, and quantum depth was a setting configured at the pre-compilation phase of ImageMagick. It was impractical to reduce the quantum depth for several reasons. Firstly, having to recompile ImageMagick would not be very fun, especially since this would have to be done for every server MarkUs is hosted on. Secondly, picture quality would be severely compromised, as quantum depth is responsible for the number of colours in the image (setting it to 8 would impose a 256 colour restriction and we live in 2010 now…). Lastly, the quantum depth was currently set at 16, and its minimal value was 8. All that extra headache would only cut memory use in half, and still not prevent malicious users from crashing server simply by using a pdf file twice as big.

After more headaches, (I blame myself for them, as the solution should have been so obvious) I took a close look at all the optional arguments the convert command takes. The -limit option was exactly what I was looking for – being responsible developers, the ImageMagick team knew of its heavy memory requirements and created a solution for cases just like ours. The documentation can be found here http://www.imagemagick.org/script/command-line-options.php#limit. Essentially, this option allowed me to limit the amount of memory that the convert command was allowed to use. Now, instead of eating up all the server’s RAM and swap, convert gets all the memory it needs from the hard disk, where space is not an issue anymore.

The downside of course, is significantly slower conversion times when the hard disk is used. To counteract this, I have added an option in the markus configuration that will allow sysadmins to set the amount of RAM they want to be accessible by ImageMagick. We found that 100 megabytes will take care of most conversion needs, and should be enough to convert a 10 page US letter sized pdf, and the hard disk only needs to kick in for those extra big files.

This post is meant to be a guide to anyone who decides to tinker with the pdf conversion system in the future, as well as aiding admins in the (highly unlikely…. hopefully) case something goes wrong.

We have added a lot of great features :

Changelog

• We are now using Rails 2.3.8, the latest version of the 2.3.x tree
• Some bug fixes on submission dates and grace period credits
• Python and Ruby Scripts in order to help users to interact with MarkUs through the API
• Displaying and annotating images
• A lot of accessibility features have been implemented :
  • Missing labels & Better focus on forms
  • Adding annotations in downloaded code from students repository
  • Re-arrange criteria using keyboard
• MarkUs is now completely internationalized and a French translation is available

So what are you waiting for? Get MarkUs 0.8.0 right now!

Contributors:
Diane Tam
Dina Sabied
Evan Browning
Anton Braverman
Nelle Varoquaux
Bryan Shen
Severin Gehwolf
Mike Conley
Benjamin Vialle

I hope I didn’t forget someone.

To make things faster, go to about:config and set network.dns.ipv4OnlyDomains to the value “localhost”.  Now Firefox only tries to find an IPV4 address

My approach to this task is to convert the PDF files to JPG format via the ImageMagick software and the RMagick gem designed for ruby and then display it in the browser. This seemed like a reasonable approach, as we already have the ability to annotate JPG’s. Currently, I successfully managed to integrate the PDFs and I am at a point where they are converted and displayed in the browser and the user is able to annotate them.

Great! So why not commit and ship these features right away you say? Right now, the main issue is the fact that PDF-JPG conversion is a very expensive operation on the order of ~2-3 seconds per page and we obviously need to accommodate for multiple-page submissions. For example, it takes a good 25 seconds to convert a 900kb 11-paged PDF file. Class sizes can easily range in the hundreds, so a 300 person class would require 7500 seconds – just over 2 hours to convert all the PDFs.

The length of the conversion is due to the need of preserving the file-content quality. On its default settings, an ImageMagick conversion gives a poor image quality, rendering standard-sized text unreadable and thus the submission is effectively useless. To combat this, I use supersampling – increasing the image resolution while decreasing the image size. This significantly lengthens the conversion process.

In light of this problem, we clearly want as little conversions happening as possible – 1 conversion per file is the best we can do. Thus I have decided to store the binary data of the images (post-conversion) in the database, for quick access. The alternative would be to store the converted file in the student repositories, but this is very undesirable for multiple reasons.

Thus we are faced with several new questions. When do we convert and how many files at a time do we convert? After speaking with Karen, we decided to kick-off a process to collect all submissions and convert them as soon as the first user logs in after an assignments submission date and grace periods have passed. I will create a queue-like object to handle this process. It will be responsible for accommodating any graders that try to mark a submission with a PDF in it before it has been converted, by bumping their job ahead of schedule. Thus waits for conversion of individual submissions may still occur, but they will happen less often. Colour coding the list of submissions may also prevent waits on the conversion process.

Here are the most important issues that he would like dealt with:

• Currently, in order to read annotations, you have to mouse over the corresponding line of code.  It would be great to have an alternate format, perhaps available for download by instructors, that inserts the annotations as comments into the code files themselves.  This would make it easy for blind users to review annotations in context.
• There is a problem with the groups and graders tab that prevents him from adding, removing or moving students between groups.  Since we are planning to redo this section of MarkUs anyway, that is a perfect opportunity to make sure the new groups page is accessible!  It would also be good to be able to download and upload group lists in a more palatable format, something that begins by listing the students in the group for example.  The format (this may not be the same as the new csv format) that Dan was using had the “group_####” name followed by a large list of TAs, before finally eventually getting to the students that are actually in the group, so Dan found it difficult to navigate.
• It would be great to give instructors a link to the SVN repositories of groups, and maybe a way to download a file containing all the repositories of every student.  Dan would be much more able to commit a file with comments it to a student’s repository than he would be able to use our visual annotation tools.

It might not be feasible to do everything that would help with accessibility, but if we can do some of these things then maybe we can stop Dan from editing databases!  He is very interested in helping us out and will set up a development environment on his machine so that he can keep track of our progress and make suggestions.

It was good to hear that some issues have already been fixed.  Dan wanted a way to re-upload rubrics after he had downloaded them, and the latest MarkUs supports that.  He also very much wanted to be able to reorder criteria without using a mouse, and that fix is currently on ReviewBoard.  I’m really happy that someone will be able to immediately benefit from my code!

We defined some specifications for the Automated Testing Framework we wanted to implement. First, the framework must add as few dependencies as possible, and not disturb the core of MarkUs (MarkUs must not be overcrowded by other programs for testing student’s code). The Automated Testing Framework must also support as many languages as possible, allowing users to test any language regardless of the languages MarkUs has been implemented in.

Technical side

Today, C, Java, Python and Scheme are used in the Universities where MarkUs is deployed (Department of Computer Science, University of Toronto.School of Computer Science, University of Waterloo.Ecole Centrale de Nantes).
C++ should be easy to implement according to the work done with C.

Diane and I are going to build our framework by using examples from these languages. When we speak of an Automated Testing Framework, the idea is not to build a new Unit Testing Framework. We aim to build an “abstraction layer” between MarkUs and most used Unit Testing Frameworks for most used languages in University.

A picture is far better than words :

Automated Testing Framework - version 1

MarkUs will call an Ant script written by the TA. This script, and all necessary files for the test environment (student code, data and tests) will be sent to a “working environment”. Later, the working environment could be moved to a dedicated server.

MarkUs will ask the Ant script to do many tasks (see Ant tasks), in an asynchronous way. Next, the Ant script, customizable by each instructor, will compile, test and run student’s code. The Ant script will produce an xml or text output for tests and build. Ant’s output will be saved in MarkUs and will be filtered for instructors and students.

The student will be able to see the result of the compilation and some test results will be made available by the instructor.

To begin with, the screen reader is really fun, but after navigating for sometime you start to get frustrated with the length of time that it takes to do some things, and the voice gets annoying too.  I can see why people who use screen readers all the time make the voice go so fast, you want to be able to listen to everything as fast as you could read it visually.

After working with MarkUs for a few days in this format, here are some key features for accessibility that I need to work on, and which other developers should keep in mind when working on user interfaces:

• Everything has to be accessible using the keyboard only!  This is the only input method that we can assume everyone will be able to use.  The biggest problem with this in MarkUs is the accordion menus, they are only accessible through the mouse.  We will have to either get rid of them or make it so that they can be navigated to using the keyboard.  Another problem is dragging and dropping for reordering things, this is really convenient with the mouse and should not be removed, but we should add alternative buttons for keyboard access.

• Focus is a problem, especially with javascript making new boxes and windows appear.  When these things appear the focus remains where it was before, and when you are using only a screen reader it seems like nothing happened.  For things like the upload file window, the box that seems to be in front of everything else on the screen is actually at the end of the document, so you have to tab through a lot of things until you see that something new has actually appeared.  If we could automatically change the focus when something like this appears, that would be a lot of help, and it would probably help everyone else too.

• Every form input element should have a label!  We are pretty good at doing this so far, but there are places here and there where there is text that describes the form element, but is not formatted as a label.  It’s pretty easy to change this text into a label and it really helps to identify what each element does, otherwise you just hear something like “checkbox unchecked” and have to go hunt for the text that labels it.  There are some places where having a label does not make sense for sighted users (like in a table of students, we don’t want to have a label beside each checkbox) but we can make hidden labels in this case for the benefit of people who don’t see the visual association.

Overall we have already done a good job of making things accessible just by following web conventions well, and if we deal with the points I mentioned above then I think MarkUs will be doing well  accessibility-wise.  The next step is to work with someone who actually requires the use of a screen reader, and I’m sure I will learn a lot more about what we could do to make things better!

I am using the wiki page as a way to keep track of progress for myself, look there for more specific details.

1. MarkUs allows instructors to reuse groups from previous assignments. Motivation for this feature was as follows. Students work on assignments alone throughout the term. In this case there is no need for students to form groups for each assignment. Students work alone anyway and once they’ve logged in a repository will be created for them (alternatively, an instructor has groups already set up so that students have repositories right away). The idea is to have students commit their code into these repositories created by MarkUs. It would be overkill, though, to create a brand new repository (and groups) for each assignment. So there is the possibility for instructors to set things up in a way that repositories are created once. But since the repositories already exist for later assignments we need a way to link those existing repositories with new groups for a new assignment. So one way is to use the “Use another assignment’s groups” feature. So far so good. But what if an instructor wants to have single student groups for A1 (allowing Subversion command line commits only) but wants to allow students to submit via MarkUs web interface for a small lab exercise? Assume for the lab exercise they would be allowed to form groups of two. Problem: If we would allow cloning groups forward for assignments for which web submissions are allowed, MarkUs could potentially link to the repository of A1. So things could end up in a repository called “c5anthei” and student c5bloche and c5anthei would be able to submit to it via the web interface. This seems a bit awkward, since the idea is to have repositories named after the student’s username only when it’s a single student assignment and it’s a command line commit only assignment. Solution: Cloning groupings forward for assignments won’t be allowed for assignments which enable MarkUs’ web submission interface.
2. Using student’s usernames as the repository name opens up the possibility of repository collisions. I.e. MarkUs could end up trying to create a repository twice with the very same name (just imagine an instructor uploading the very same CSV groups file twice). Solution: This will happen for single student and non-web-submit assignments only. If a collision occurs the repository already exists, but that’s everything the CSV upload tries to achieve (apart from creating the groupings in MarkUs). But since MarkUs won’t create a repository if it already exists, this is a minor concern. This cannot happen for web submit assignments, because in this case a unique name for the repository will be generated. If a repository collision happens during a CSV upload, MarkUs will notify the instructor about it.
3. MarkUs “recycles” groups. MarkUs internally knows about groupings and groups. A grouping is basically a snapshot of students and their memberships in groups for a particular assignment. In fact, when you create a “Group” (in the usual sense) using MarkUs’ web interface a grouping will be created. If a group object does not exist yet, it will be created too. This behaviour might be different for the next grouping, because a group object with the required name might already exist in the database. I.e. if a group with a particular name already exists, MarkUs will try to reuse it. Thus, it’s theoretically possible to create a group “Beta 10″ for assignment 1 (say for A1 students work alone and have to commit via command line client). Then for A2, say, the instructor wants to create group “Beta 10″ again, now allowing web interface submits and groups > 1 members. In that case MarkUs will find group “Beta 10″ in the database and will try to use this group instead of creating a brand new object. This is a problem if an instructor would pick a group name which is equal to a student’s username. We’d end up in a similar situation as described in (1). Solution: I think, the most clean solution is to require instructors to use the user’s username as the group name (and those groups can have one group member at most). Otherwise, an autogenerated repository name will be used.

Those three corner cases are the ones I could think of so far. I’m not sure if we are still missing some. Let me know if there is a problem with my reasoning or if there are other cases I haven’t thought of, yet. Thanks! I’ll post an updated review shortly.

Edit: Considering that the repository name will be set in this special way for one member only groups, some of those cases might occur very rarely (if at all). Solution to (3) also means that it should be impossible to produce repository collisions. What do you think?

• A more stable, polished, and usable notes system.  Notes can now be applied to students, groups, and assignment submissions
• An alternative, “flexible” marking scheme type that allows graders to input arbitrary numerical grades, as opposed to the more strict rubric marking scheme
• A new marks spreadsheet feature that allows instructors and graders to input grades from assignments, quizzes, or tests, in an Excel-like web-based spreadsheet.  Grades can also be uploaded and downloaded via CSV file
• The table of student submissions can now be bookmarked, and the back-button works correctly
• Lots of bugfixes and usability fixes.

So what are you waiting for?  Get MarkUs 0.7.0 right now!

Contributors:

Benjamin Vialle
Christian Jacques
Nelle Varoquaux
Victoria Mui
Joseph Maté
Robert Burke
Bryan Shen
Brian Xu
Fernando Garces
Farah Juma
Severin Gehwolf
Mike Conley