MarkUs Blog

MarkUs Developers Blog About Their Project

Archive for the ‘Automated Testing’ Category

Using the Test Framework Infrastructure in Different Scenarios

without comments

This post describes several scenarios that instructors may run into while using MarkUs for automated testing. These scenarios have all happened before at my school. Below each scenario is a description of how to accomplish it, or why it cannot be done using the current test framework design. In the scenarios, “I” refers to an instructor or TA.

1. I want to let my students run as many tests as they want.

  • Currently there is no way to do this. When students submit, they will always be deducted a token. If they have no tokens left, the test won’t run. Currently, there is no way to disable this token system. A solution that will probably work is to just set the number of tokens to 999. It’s very unlikely that a student will submit more than 999 times in a single day…

2. I have an assignment with some practise questions that are not for marks. Test scripts for these questions should always run, but they should not affect the student’s marks in any way.

  • You can set the maximum mark to 0 and set up the test script so that the marks_earned field is always 0. The students can still see their pass/fail status to see if they got the question correct or not.

3. I want to let the test runs that my students make count for marks. Before the assignment is due, students should be able to run tests only if they have tokens available, and their mark for the assignment is the best mark they’ve earned from any test run they’ve made.

  • There is a token system in place, but the current test framework design does not allow you to use the best mark earned as the mark for the assignment.

4. A student (John) submitted his assignment late and so he is getting 0 on the assignment. But, I want to run the test scripts on his assignment anyways to see what mark he would have gotten. I also want John to be able to see which unit tests he passed and failed, for feedback only (the test run should not change his mark, which is 0).

  • Right now, when you run the test scripts, MarkUs will automatically update the mark on the assignment. You cannot run the test scripts for feedback only. However, you can use Deductions to manually remove any marks earned. You can also manually run the test scripts yourself, and commit the results to John’s repository so that he can see them.

5. I need to access the student’s MarkUs username in my test scripts. How can I do this?

  • Right now, you can’t. One way that this feature can be implemented in MarkUs is to pass the student’s username to testrunner.rb as an environment variable. This should be easy to do; just modify the ssh call in automated_tests_helper.rb.

6. I have some test scripts which students should be able to run without needing any tokens, and some test scripts that do require tokens. How should I set things up in order to do this?

  • You cannot do this in the current test framework design.

7. Many of my courses use “public tests” and “release tests”. Before the assignment is due, students can run the public tests, which check the students’ code for trivial things (like syntax errors). After the assignment deadline, the TA will run the release tests, which do a very rigorous and thorough testing of the students’ code. Students should not have access to the release tests.

  • For the release tests, you can set the “Display Settings” to “Do not display” to hide test results from the students. You should also uncheck “Run On Request” and “Run On Submission” so that students cannot run the test scripts.

8. I have an algorithms assignment and I want to set different timeouts for each unit test. I also want to set some memory usage limits. How can I do this?

  • You will probably want to define the timeouts and memory limits in your test scripts. You can also try editing testrunner.rb. You will have to edit testrunner.rb if you want a timeout of more than 10 minutes, because the 10 minute timeout is hard-coded in testrunner.rb.

9. I want to prevent students from using certain libraries and/or function calls. How can I do this?

  • This will have to be done in the test scripts that you upload. For example you can try stripping out any import or require directives in the student’s code (depending on which programming language the student is using). Or, you can define your own versions of the prohibited functions so that they throw an error when called; if students use these prohibited functions, your error will be thrown.

10. I have an assignment X which is a prerequisite for another assignment Y. Specifically, if students do not pass assignment X, then they should automatically get 0 on the testing portion of assignment Y, regardless of what they submitted for Assignment Y.

  • In the test script that you upload, you will have to somehow check that the student has passed assignment X. I don’t think you can do this right now.

11. It is after the assignment deadline, and I ran my test scripts on all the students’ submissions. However, I later discovered that there was a mistake in my test scripts (this is the instructor’s fault, not the students’). What is the proper way to fix things?

  • First, fix the bugs in your test scripts. Then upload the new test scripts from the Test Framework Settings / Config UI page. Then re-run the tests on all the students. During this process, MarkUs needs to somehow replace the mark earned from the old, broken test script with the mark earned from the new test script.

12. I want to display a long, custom message when students request a test run. Since it’s a graphics assignment, I also want to display a picture in the message. How can I do this?

  • The Description field is the same for every student, so you can’t use that. You can try using the “Expected output” field to display the message. You cannot include an image in the output (even if you include an HTML img tag, Rails will escape the tag).

13. I want to see how many test runs a student has requested and what the results were.

  • You can switch roles and log in as that student. Then you should be able to see the desired information.

14. For an algorithms assignment, students have to sort numbers using heapsort. If they use a different sorting algorithm, they will get 0 marks. The TAs will manually check that the students have used the correct algorithm, and deduct marks accordingly if they don’t. The test scripts cannot check that students have used heapsort. How can the TAs override the test results?

  • Currently, the test results cannot be manually changed. However you can simulate the effect by using Deductions.

Written by Nick Lee

March 12th, 2013 at 6:30 pm

Posted in Uncategorized

Ideas for passing data to the Test Runner

with 10 comments

The test runner is the program that would be called to run the instructor-provided test scripts, and would return the data to MarkUs, which would then parse the results.

The test runner is supposed to return the following data as XML:

–          Test id – The name of the test

–          Input – The input used for the test

–          Expected Output – The correct output

–          Actual Output – The actual output (from the submitted code)

–          Result – Pass or Fail, depending on the output, or Error if there was an error during testing

–          Marks – The marks earned by the student

The input for the test runner is currently a file (or input on stdin) that contains the name of each test followed by a flag (to determine whether the program should halt if that test fails), followed by another test, etc.

The issue that arises from this is that there is more data returned than is sent to the test runner in the input file, and the test runner needs some way of receiving that data.

As a result, we must find another way to send that data to the test runner.


There are 3 solutions that I can see, and I’d like to ask for feedback.

1.       Return less data

This is the simplest solution, although arguably the worst (I’m just including it here in the off chance that it’s actually a viable solution).

Instead of returning all of the above information, the test runner could return a subset of it; specifically, it could return the test id, the actual output, and the exit status of the test script.

The advantage of this approach is that the input for the test runner would remain the same.

The disadvantage is that MarkUs would need to compare the results and determine the marks received. This means that the instructor would still need to submit the input and correct answer to MarkUs, which raises the question of why that information wouldn’t be sent to the test runner.


2.       Include more data in the input file

Instead of simply passing a file where each line is “test_name, halt_on_fail”, more data could be included. For instance, “test_name, halt_on_fail, input, expected_output, marks” could be passed instead.

By passing this data, the test runner could compare the actual and expected output, and return the appropriate status, and determine the number of marks to be awarded. The “input” field could likely be omitted since the test runner doesn’t use it, so the only real changes would be the inclusion of the “expected output” and “marks” fields.

The main advantage to this approach is that the input would remain almost the same.

The disadvantage though is that the testing interface would need to be updated to allow the instructor to specify the input, target output, and number of marks, which can be quite tedious to enter if there are a large number of tests.


3.       Have the test script output the data

Rather than change the data that is passed to the test runner, or change the testing interface, we can simply require that the first 3 lines output by each testing script are the input, target output, and number of marks that test is worth.

This seems to be the simplest of the 3 approaches.

Without this approach, the test script would already need to define the input for the student code, as well as the target output. It would be trivial to include a print statement to print that to stdout. The only real addition to the script would be to add a line printing the number of marks that test is worth.

The main advantage of this approach is that the interface does not need to be changed, and the instructor will not be required to submit additional information; they can simply put the information in the test script. As well, no extra input is needed for the test runner.

The main disadvantage though, is that the instructor must output the data in every script, and in the correct order. If the print statements are ordered incorrectly, then the results for that test will be incorrect.

I believe that this may be the best solution to this problem.


Feedback would be appreciated.


Written by mmargel

October 20th, 2012 at 4:18 am

Posted in Uncategorized

Test Framewok – State of Art

with one comment

Here is the State of Art of the Markus Test Framework. We will describe what is working for each view (Administrator, Grader and Student).

Administrator view

Create a new assignment with test

When we create a new assignment, we can enable the test framwork. If we do so, we can upload a build.xml and, and add some test files (public or private).

This part is now working well, since the routing bug are fixed.

Due date passed

Buttons “Release Marks” and “Unrelease Marks” lead to a 404 error.

“Collect All Submissions” links to RecordNotFound in SubmissionsController#collect_all_submissions.

For each submission, the “Collect Submission” lead to a 404 error (issue

We can’t test if the Admin can run the test because of the errors posted in our last post on this blog.

Grader’s view

We suppose this configuration being applied by the administrator :

  • Tests have been uploaded
  • Tests are public
  • Tokens are available
  • The return date is passed

First, the student has to create a working group or click he wants to work alone.

The Administrator assign a Grader to the Student.

Then, the Grader is supposed to be able to use the testing framework :

  • Click on one of the current assignment.
  • “Can begin grading” is True
  • On clicking on a Group name : the Marking State changes from « Not Collected » (Blue square) to “In progress” (a yellow pencil) and a confirmation message “The submission has been given collection priority. It will be ready soon.” appears on screen.

Then there are problems with collecting all assignments, applying tests or getting results :

  • On clicking on « Collect all your assigned submissions » an error page appears ” ActiveRecord::RecordNotFound in submissionsController#collect_ta_submissions
  • No test can be run and no results loaded.
  • The button “see and comment the result details” is still unavailable.

Student’s view

We suppose :

  • Tests have been uploaded by the administrator on the administrator view
  • Tests are public
  • Tokens are availables
  • The return date is not passed

Steps to use the testing framework as a student:

First, the student has to create a working group or click he wants to work alone.

Then, he can post his work (1. Click on submissions, 2. Click on “add new” and find the file to upload, 3. Click on “submit”).

Then, he can use the testing framework:

  • Click on “assignments”
  • For the first time of use: the button “collect and prepare tests” is available so he can click on.
  • Click on the button “run the test”

What happens ?

At this point, we can launch the tests, they are running but no result is collected and so there is no result available.

Previously, there were some issues of displaying but now all is well displayed (11/30/2011 patch by Guillaume).

If we change some configurations:

  • We suppose that there is no token available or tests are private.

We have the same situation as previously. (Shouldn’t we have no button available to run the tests in those cases?)

  • If the due date is passed: No problem appears. Indeed, the student cannot launch the tests.

Written by gbugnet

December 1st, 2011 at 12:46 pm

Posted in Uncategorized

Automated Testing: Options to implement security

with 2 comments

Please comment and add to this post!

There are two ‘general’ ways to implement security in auto-testing: To use VMs or not to use VMs. This is the question. At first, the consensus was in favor of using a VM, while I wanted to prove that it will be possible to achieve the same level of security via existing tools available in Linux (and SE Linux), because it seemed like a more elegant solution at the time, plus it would avoid the runtime overhead of VMs.

What I’ve realized is that there are two problems to that approach: First, this would require some custom tailoring of security features to different/older distributions. A greater problem would be allowing maximum freedom for assignments such as operating-systems, networking, security – while retaining security. In the end, it seems, running a VM seems like the only feasible solution.

Here is the rough workflow of doing automated testing with a VM:

1) Setup (by hand):

  • Install the VM.
  • Set up a virtual hard drive (preferably on a separate hard disk, for speed) and networking.
  • Install the necessary programs on it (minimum: Ant, SSH server)
  • Set up SSH keys to allow automatic SSH from local machine to VM.
  • Save the image.

2) Automated testing (shell script):

  • Start a headless VM from the initial image.
  • FTP test scripts/files
  • SSH-execute the script, pipe output to a file.
  • FTP output file to local machine.
  • Lather-rinse-repeat for every test request. That is, each test will start from new image to avoid possibility of corruption.

Possible improvements on the basic process:

  • Parallelizing test runs. That is, not being limited to serially running VMs. Useful in batch-testing, and for more real-time-like results (imagine 100 students testing their assignments 10 minutes before deadline!)
  • Throttling resource use – not waiting indefinitely on busy-wait, and timely termination of fork-bombs and infinite recursion.

Possible VM products:

  • VMWare Server
  • Oracle VirtualBox
  • Parallels
  • Qemu

I have heard very good things about Oracle’s product, and of course VMWare. Further research required to test for speed (up-time from saved image, system resources, restart) and general compatibility (I think they should be equivalent in this regard.)

That’s it for now. Please leave comments and suggestions!

Written by victor

October 27th, 2010 at 2:53 pm

Posted in Uncategorized

Automated Test Framework – Part 1: Assignment View

without comments

Testing Framework Overview:

Upon grading a student’s assignment, an instructor may want to execute a set of tests against the student’s code to validate the correctness of the code.  The instructor may also have various ways of handling pass/fail cases.  The testing framework has been implemented to allow instructors to perform such operations.  Using Ant, a Java-based build tool, as the core behind this testing feature, instructors will be allowed to upload their test files, any associated library files necessary for the tests to run, as well as any parsers they may wish to execute against the output, and control the compilation, testing and output parsing through a simple build file.

Part 1 of the testing framework consists of the Test Framework form from the Assignment view which allows a user to upload the Ant, test, library and parser files required to execute tests against students’ code.  Part 2 of the testing framework consists of the actual execution of the tests from the Submission view.  Part 3 is the student’s Assignment view where the student will be allowed (assuming they have tokens available to them) to execute the tests against their code before submitting their assignments.

The following documents how to create new test files associated to an assignment, as well as how to update and delete the files and the associated expected errors messages that will appear in response to invalid input.

Creating new test files:
1) Create a new assignment and click ‘Submit’
2) Click on ‘Testing Framework’ from the sub menu and you will be redirected to the Test Framework view (by default, the test form is disabled)
3) Check ‘Enable tests for this assignment’ and the test form will be enabled
4) Fill in the ‘Tokens available per group’ as needed.  This value represents the number of times a student may be allowed to execute the tests against their code before submitting their assignment.
5) Add the required Ant files, build.xml and
6) Click on ‘Add Test File’, ‘Add Library File’, ‘Add Parser File’ to add the appropriate test, library and parser files needed to run the tests respectively.  Test files represent the test cases that will be executed against the code.  Library files are any necessary files required by the tests to execute appropriately.  Parser files (if specified) will be executed against the output from the tests to extract and manipulate the output as needed.
7) Click ‘Submit’ to save the changes

*Optional* You can check the test file ‘private’ box to indicate a particular test cannot be executed by the student.

Testing Framework

Testing Framework

1) Verify table ‘test_files’ to ensure the files were saved correctly
2) Verify table ‘assignments’ to ensure ‘enable_test’ and ‘tokens_per_day’ were saved correctly

1) Verify the following folder structure in the TEST_FRAMEWORK_REPOSITORY:

A1 (folder)


– build.xml

– test –

– lib – junit-4.8.2.jar

– parse –

Updating test files:
1) Update any test, library or parser file already uploaded and click ‘Submit’ (able to replace with the same filename or overwrite with a different filename)

1) Verify updated file has been saved successfully

1) Verify updated file has been overwritten successfully

(If the replaced file has a different filename, it will be deleted from the directory.  If it has the same filename, it will simply be overwritten.)

Deleting test files:
1) Delete any test or library file by checking the ‘Remove’ box and click ‘Submit’

1) Verify file has been removed from the database successfully

1) Verify file has been removed from the filesystem successfully

Negative Test Cases:

Filename Validation –

  • cannot be named anything but ‘’
Invalid Filename

Invalid Filename


  • cannot be named anything but ‘build.xml’
Invalid Filename

Invalid Filename

test, library and parser files:

  • cannot be named ‘build.xml’ or ‘’
Invalid Filename

Invalid Filename

any file:

  • cannot have the same filename as any other existing test file belonging to that assignment
Duplicate Filename

Duplicate Filename

  • if you attempt to upload two files simultaneously with the same filename, an error will be displayed
Duplicate File Entry

Duplicate File Entry

  • cannot be blank (File entry will simply be ignored)

Assignment Validation –


  • cannot be negative
Invalid Token Value

Invalid Token Value

Other Scenarios:

  • if you update the test form in any way but then disable tests for the assignment and click ‘Submit’, those new changes will not be saved

Related Posts:

Automated Test Framework Overview
Automated Test Framework – Part 2: Submission View
Automated Test Framework – Part 3: Student’s View

Written by diane

August 16th, 2010 at 2:34 am

Posted in Uncategorized

Automated Testing Framework

with 3 comments

Talks concerning MarkUs Automated Testing Framework

We defined some specifications for the Automated Testing Framework we wanted to implement. First, the framework must add as few dependencies as possible, and not disturb the core of MarkUs (MarkUs must not be overcrowded by other programs for testing student’s code). The Automated Testing Framework must also support as many languages as possible, allowing users to test any language regardless of the languages MarkUs has been implemented in.

Technical side

Today, C, Java, Python and Scheme are used in the Universities where MarkUs is deployed (Department of Computer Science, University of Toronto.School of Computer Science, University of Waterloo.Ecole Centrale de Nantes).
C++ should be easy to implement according to the work done with C.

Diane and I are going to build our framework by using examples from these languages. When we speak of an Automated Testing Framework, the idea is not to build a new Unit Testing Framework. We aim to build an “abstraction layer” between MarkUs and most used Unit Testing Frameworks for most used languages in University.

A picture is far better than words :

Automated Testing

Automated Testing Framework - version 1

MarkUs will call an Ant script written by the TA. This script, and all necessary files for the test environment (student code, data and tests) will be sent to a “working environment”. Later, the working environment could be moved to a dedicated server.

MarkUs will ask the Ant script to do many tasks (see Ant tasks), in an asynchronous way. Next, the Ant script, customizable by each instructor, will compile, test and run student’s code. The Ant script will produce an xml or text output for tests and build. Ant’s output will be saved in MarkUs and will be filtered for instructors and students.

The student will be able to see the result of the compilation and some test results will be made available by the instructor.

Written by Benjamin Vialle

June 9th, 2010 at 10:29 am

Posted in Uncategorized

Getting Started with Machinist in Markus

with 3 comments

When running, we almost always have to set up some objects to test. We used to set up these objects by loading them from fixtures. There is a problem with fixtures. It’s hard to know the attributes of the objects or the relationships between different objects unless we delve into fixture files. This is not good, because when doing tests and making assertions, we want to know well about the objects we tested against, but looking into fixtures is a time-consuming job. Now don’t you want a mechanism that allows you to explicitly create objects and build up the relationships among them, but without having to specify their attributes that you don’t care about? This is what Machinist does! With Machinist, you can create and save an assignment object by writing one short line in your code: “assignment = Assignment.make”. Isn’t that great?

In his post, Fixtures are not as clear as Machinist, Robert has a deep insight on how Machinist is better than fixtures.

Yes, Machinist is great, and I promise you’ll find it even greater if you spend a little time reading through Machinist’s documentation. Make sure you understand what a blueprint is and how we can pass attributes to “make” method and override its default behaviour.

The documentation pretty much tells all you need to know about Machinist. I’m not going through every details here, but there are some stuffs you may want to know about how we can use Machinist with Markus.

1. Setting up the playground for Machinist

Before using Machinist, please make sure you’ve installed two gems: Machinist and Faker. If they are not on your machine, you can run the following command in a terminal:

 gem install faker machinist --source 

Now that the gems are installed, you want to build some confidence in Machinist by making some easy objects. The easiest way to play around with Machinist is to open a script/console with test environment. You can do this by opening a terminal, change the directory to the root directory of your Markus’ project, and then run the following command:

 script/console test 

Make sure you append the “test” option, which loads test environment for you. Otherwise you’ll be at the risk of screwing up data you use for dev.

If you see something like

Loading test environment (Rails 2.3.5)

you’re already in the script console. In script console, you can execute any Ruby code as you would normally write in .rb source files. If you’ve not used script console before, try to “new” an empty group object to get yourself familiar with it.

>> group =
=> #<Group id: nil, group_name: nil, repo_name: nil>

“group =” creates a new empty group, whose information is displayed in the console.

Now you probably cannot wait to use Machinist. OK, let’s start by making a group which is the easiest model I can find in Markus. Try to execute the following line in the script console.

>> group = Group.make

But you’re likely to get this:

NoMethodError: undefined method `make' for Group(id: integer, group_name: string, repo_name: string):Class
 from /var/lib/gems/1.8/gems/activerecord-2.3.5/lib/active_record/base.rb:1959:in `method_missing_without_paginate'
 from /var/lib/gems/1.8/gems/will_paginate-2.3.12/lib/will_paginate/finder.rb:170:in `method_missing'
 from (irb):5

Why? If you’ve read the Machinist’s documentation, you should’ve already known the reason. You cannot use the “make” method of a model unless you’ve defined the its blueprint. OK, then let’s write the blueprint. Wait, in fact, you don’t need to, because blueprints for most models in Markus are already defined in /test/blueprints/blueprints.rb, which is “required” by /test/blueprints/helper.rb. To make a reference of these blueprints, you only need to run the following line in script console:

>> require "test/blueprints/helper.rb" 

Now try to “make” your group object again, and see what happens. If you still get an error, please don’t panic, and skip ahead to read the next section, where you’ll find the answer. But most likely you will be lucky to see a sign of success in your script console like the one I get here:

=> #<Group id: 930503258, group_name: "group1", repo_name: "group_930503258"> 

See, with only one simple call of “make”, you have already created a group object with all the attributes set properly. That’s why we all like Machinist!

Congratulations. You’ve already set up the playground for Machinist. At this point, you are good to go. Just try out any of your ideas with Machinist.

2. Common Issues

Sometimes, when you make a make an object, you get an error message like this: “ActiveRecord::RecordInvalid: Validation failed: Group name has already been taken”. This message means in the database there is already a group object with the same group name as the one you are just making. The test database can persists objects as a result of your last test, so you should clear the database before playing around with Machinist. To delete all the objects of a model in database, you can call “ModelName.delete_all”, where ModelName is the name of the Model whose objects you want to delete. But it’s is more desirable to clear all the table at the same time. Here is the code to do that:

(ActiveRecord::Base.send :subclasses).each {|c| c.delete_all}

I recommended you to use script console. But it has an issue. Everytime you modify and save blueprints.rb, you need to restart the script console for it to detect the change. This makes it inconvenient for you to do test your blueprints. As an alternative to script console, you can create a .rb file at your project’s root directory, and put the following lines at the beginning:

ENV["RAILS_ENV"] = "test"
require File.expand_path(File.dirname(__FILE__) + "/config/environment")
require File.join(File.dirname(__FILE__), "test/blueprints/helper")

This should also set you up a test environment where you can write code to play around with Machinist. The benefit of this approach is that it makes it easy to test blueprints.

3. Making Objects with Associations

There are three common types of associations: “belongs_to”, “has_many” and “has_one”.

belongs_to“: if A “belongs_to” B, a.b should be not nil and unique. It’s identical to a foreign key relationship from A to B. B should be made prior A. A’s blueprint should take care of making B. For example in Markus, Student belongs Section. The blueprints for Section and Student are:

Section.blueprint do

name {Sham.section_name}

Student.blueprint do
type {'Student'}
user_name {Sham.student_user_name}

The blueprint of Student implicitly calls Section.make. Therefore, when you call Student.make, a section will be made before student. If you already have a section object my_section and want to make a student that belongs to that section, you can call Student.make({:section => my_section}), which overrides the default behaviour specified by Student blueprint and doesn’t make a new section. When you make an object, you may end up making lots of  other objects because the “belongs_to” association “recursively” makes the dependencies.

has_many“:  if A “has_many” B, then B “belongs_to” A. Machinist encourages you to first create the object on the right hand side of “belongs_to” association (seeking for some better wording here). Using the example of Student and Section. Section belongs to Section, and thus Section has many Students. When you want to make a section with many students, you need to make a section first. We are still using the blueprints from the previous example:

section = Section.make


When section.students.make is called, Machinist is smart enough to call  Student.make({:section => section}). So you don’t need to worry about accidentally creating a new section. After the two lines, you have section.students.first.section == section.

has_one“: “has_one” is a special case of “has_many”. if A “has_one” B, then B “belongs_to” A. Same as “has_many”, A should be made prior to B. Suppose we have two models: Coach and Team, where Team “has_one” Coach and Coach “belongs_to” Team. You may attempt to make an object of Team in a similar way to “has_many” association:

team = Team.make

coach =

But it doesn’t work. In the “has_many” example, section.students is a collection which is an object not nil, but here is nil. There is no way for Machinist to know which blueprint to use for making a nil object. So what you have to do is:

team = Team.make

coach = Coach.make({:team => team})

Then you can use the team object.

Submission.blueprint do
submission_version {1}
submission_version_used {true}
revision_number {1}
revision_timestamp {1.days.ago}

Written by bryanshen

March 10th, 2010 at 3:51 am

Posted in Uncategorized

Re-factoring contexts with Shoulda

with one comment

I’ve been playing around with Ruby on Rails for a personal project that I started in December and as we starting doing with MarkUs, I’ve been using Shoulda for my tests. One of the basic tenets of Ruby on Rails is DRY – Don’t Repeat Yourself. And I’ve often found that with Shoulda tests, I was writing the same code over and over again. I’ve had some difficulties scoping the internet for ways to re-factor Shoulda tests, but I’ve come up with two useful ways so far that I thought I would share with you. Perhaps this is just my still naivety with Ruby on Rails, but since most of us are in a similar boat, I thought I’d share my findings.

1. Class methods

This works great if your setup varies significantly between tests of a method, but you still want almost the same thing (or the exact same thing) to happen in each of the cases. You can create class methods in test class to re-factor tests like this. The key part is that you define the method prior to the highest-nested level context of any test that will use the method. I like naming them starting with should_ to clearly indicate what they’re for 🙂

def self.should_flip_active
  # Put your should statements here, including should "..." do ... end and should_*

2. Iterating over a set of test data

With unit tests, we shouldn’t really be using fixtures and loading everything from the database. Instead, you might just set up the few attributes that you need on a model instance to test a method on it. If you want to do this for slightly different data, then you can use either a hash or an array and make a context for each of them!

Structure with an array:

['c6conley', 'g9merika'].each do |user|
  context "with user "+user do
    # Put your setup and should statements in here - you could even make it nested further!

Structure with a hash:

{ 'c6conley' => 5, 'g9merika' => 8 }.each do |user, section|
  context "with user "+user do
    # Do some tests on user with section

One last comment on Shoulda – I’ve found that you can only access assigns, e.g. assigns(:student).some_student_attribute from within should blocks. I’ve run into some problems when running my tests if I try to use assigns or a class variable like @student within an open area in a should statement. Here’s some examples:

should "work" do
  assert_equal 'hello', assigns(:student).user_name

# should not work
should_set_the_flash_to assigns(:student).user_name

should "work" do
  assert_contains flash.values, assigns(:student).user_name

Happy Shoulda testing!

Written by Tara Clark

January 26th, 2010 at 6:10 am

Posted in Uncategorized

Automated Testing: Mockups and some more ideas

with 5 comments

The more that I read up on this and think about it, the more I think this will be a huge project. There is really a lot to concern about. In an ideal world, this is what automated testing would do once implemented.

  1. Students have a system at hand, which they can use to learn writing good tests. For instance they would check if their tests are correct, by running them against a reference implementation an instructor would put up at some point before the assignment due date.
  2. Students would learn which parts of the code they are testing (coverage). This would – again, we are in an ideal world – be reported visually in an Grader View like environment. With little boxes, as to where is unchecked code. See the mockup below.
  3. Students would also learn which areas of their code is particularly fragile. This could be achieved by running a static analysis tool on the code
  4. Students would have the opportunity to see public test results of tests the instructor has created and are marked public. The instructor uploads a public test file and could additionally specify a filter file, which is run on the test output. I can imagine these kinds of things would be shell scripts or Python scripts.
  5. Also, students have the possibility to run some more tests on their code, the restricted test suite as specified by the instructor. Say the instructor specifies that a student or group can run these tests at most 1 or 2 times a day. The idea is to encourage students to start their assignment early and to have lots of opportunity to run those restricted tests. For each restricted test, students would have to log into MarkUs and trigger their execution. We store in the database, how many restricted test runs are left for each grouping. Restricted tests can be filtered similarly as public test. By using some sort of script run on the test output.
  6. An instructor could specify a file, which has to successfully run in order to have the submission graded. This would be useful, if an instructor does not allow submissions which do not compile. Such failing “acceptance tests” would be displayed visually in the submissions view and the grader view.
  7. Graders have a visual report ready, when they start grading. They would see the percentage of failing and passing tests, respectively. See again the mockups for more information.
  8. Tests created by students (for instance to run them on a reference implementation), would be submitted via the web interface in a separate section, “Tests” say. The reason for this is to keep tests and code separate. I can imagine that these tests would go into a different folder in the groupings repository (e.g. /tests/a1). Alternatively, students can submit these tests via SVN command line client into the appropriate folder. Tests would be triggered via MarkUs’ interface in any case. This makes tracking of students testing activity and restricted test runs easier.
  9. Students control their test runs via MarkUs. They trigger runs, and get immediate feedback if they are not allowed to run the desired test or get feedback as to where in the testing queue their run is schedulred. Students can wait a while and check back later if there are test results available for them. Maybe we could plan for the option of email notifications for students once the test has been run and results are available.
  10. Automated testing would support graders in many aspects. All tests will be run which the instructor provided for the assignment. This includes public, restricted, private and basic acceptance tests. If students are required to write tests for their code, too, MarkUs could be configured to also run tests in a specific folder in the grouping’s repository (e.g. /tests/a1). All these tests results would be available to the grader in the grader view. Ideally, failures are already pre-annotated in the Grader View. If basic tests are failing (e.g. program does not compile), automatic deductions could be triggered.

I’ll stop for now. Here are some mocks:


assignment_creation_detailgrader_view_integrationIf you have thoughts, please feel free to chime in. As I mentioned previously, I think this whole topic would be enough material for a master thesis. The big question remains: How would you make sure that you can integrate test runs into the Grader View and other points? It would be nice if test tools would have a standard output format. I can imagine some built in integration points in MarkUs, though. That’s it for now, stay tuned 🙂

Written by Severin

November 22nd, 2009 at 2:13 am

Posted in Uncategorized

Automated Testing

with 2 comments

The idea is to integrate some sort of testing infrastructure into MarkUs, so that students get more feedback about their code submitted for assignments. A possible scenario would be the following:

  1. A new assignment is up in MarkUs and students start to work on it
  2. An instructor has written tests for this assignment, which will be run on the code Students submit
  3. Students will get feedback about their code submitted according to the results of those test runs

The question is how to achieve this. I’ll refer to the server or infrastructure which will be executing tests only as test server. Here are some questions we are trying to find the answer for:

  1. Where and how will the tests on students code be run?
  2. Should they be run on a dedicated server? On several servers?
  3. How does MarkUs communicate with the test server?
  4. How to decouple MarkUs from the test server?
  5. Is it a concern if a student compromises the test server either maliciously or accidentally?
  6. What are the required security measures for the test server?
  7. How to satisfy as many programming languages and course models as possible?
  8. What will the test server or test infrastructure look like? Should tests for each MarkUs instance run in separated virtual machines?
  9. Should we use virtual machines? If yes, why? Why not?
  10. How can we secure the test server well enough so that no secret information will be revealed to students? Should the mail system be disabled, network/socket connections be monitored or disallowed? How to secure the test server for courses in which students learn process forking and socket programming?
  11. Will there be performance problems? How to deal with increased load close to assignment deadlines?
  12. Should the amount of test runs be limited for students? If so, how? e.g. 5 test runs per day or max number of test runs?
  13. How much information of tests results should be shown to the student? Number of successes/failures? Detailed test failure messages? Should descriptions of tests be displayed?

These are related projects to get some ideas from. If you know some more, please let me know!

I am looking for ideas and hints regarding automated testing systems, how to integrate them into MarkUs and why to use one approach rather then the other. Please feel free to chime in and comment to this post. Greatly appreciated!

Written by Severin

October 16th, 2009 at 11:00 am

Posted in Uncategorized