MarkUs Blog

MarkUs Developers Blog About Their Project

Archive for the ‘tests’ Category

Looking Behind the Scenes of MarkUs Development

with one comment

MarkUs is entirely developed by undergraduate students. Since MarkUs is a web-application developed for and by undergraduate students, there are some challenges to overcome. Most notably there is high developer turn-around and development happens in a fairly distributed fashion.  We try to overcome those challenges by instrumenting established means of quality assurance (QA) in the open source world. In order to keep code in shape and well tested some quite heavy tooling is involved at

MarkUs website

What started as a small project at University of Toronto is becoming more and more what one might call a classical open source organization. As FOSS (Free and Open Source Software) projects don’t have a lot of money, they use various other open source software for their development. And so is MarkUs.

MarkUs developers use a mix of method and tools for their development. Some of those tools are:

  • Git (version control)
  • GitHub (wiki, issues, pull requests)
  • ReviewBoard (code review tool)
  • Tests (functional, unit)
  • Test Coverage Analysis
  • Sandbox (MarkUs demo application)
  • Blog (
  • #markus IRC channel on freenode

A subset of those tools fall into the realm of QA.

Git as Source Control Management (SCM)

Git is a decentralized SCM. Every developer has its own local repository. We have chosen a decentralized workflow. Nobody pushes directly to the central repository. Instead, people push to their own public repository hosted on GitHub (fork) and notify core developers, by instrumenting GitHub’s pull requests, to get changes in to the main MarkUs repository. This ensures that only well vetted code goes into the main development repository. What’s more, every patch has to be reviewed before being applied to the main repository (see section ReviewBoard of this blog post).

Git Workflow

GitHub as a Project Management Tool

In addition to providing public repositories, GitHub allows the team to manage issues and documentation (wiki). People can interact with the wiki using the Web interface, but also using Git. Several markup languages are supported. We chose to use the ReStructuredText syntax. The issue tracker is quite simple, but has labels rather than categories, which can come in handy in several cases. On a related note, GitHub allows for issues to be closed automatically.

GitHub also displays a graph of everyone’s forks and branches.


Before code goes into our main repository, it has to be reviewed by the team. This is an excellent way to improve code quality and to quickly spot common mistakes, especially in cases of high developer turn-around, which is the case for MarkUs. Another benefit is that it helps the whole team to keep an eye on every area of the project, even the ones they don’t code on. In other words it is an excellent tool to speed up getting familiar with the code base.

Once the code has been reviewed and/or revised, developers will check the “ship it” flag for a review request indicating that they are OK with submitting this code to the main development repository. Once the person who posted the review request received several “ship it”, the original poster can open a pull request on GitHub. After another once-over of a core developer the code in question gets merged into the main repository.

Dashboard of ReviewBoard

Unit Tests and Functional Tests

A great way to increase confidence if the code is doing what it is supposed to be doing is to write unit and/or functional tests for each piece of contributed code. We think we have been doing a quite good job at this. The number of tests increase almost on a daily bases. Functional and unit tests run every few hours and results are shown on on

Code Coverage: How Well is MarkUs’ Code Exercised When Tests Run?

Trustworthy tests are essential. We think it’s important to check how well our code is tested. Tests coverage analysis provides us with good hints as to where our code lacks testing. Ruby Coverage a.k.a. RCov is an excellent tool to provide visual feedback as to what is tested and what is not.

Test Coverage

Test Coverage

A MarkUs Demo: Sandbox

A current version of MarkUs in action is available to anybody on to play with. We try to keep the running version as up-to-date as possible. Instructions as to how to use it are described in this blog post. Note that the underlying database of this instance is flushed on a daily basis. So please don’t expect your data to persist any longer than a day or two 😉

MarkUs Blog

On our blog we describe progress, propose new features, announce new releases, post individual experiences related to MarkUs development (mostly including solutions too) and various other random things. Following our blog allows you to get a feel of the current state of the project, planned features and many more. For developers it is recommended to keep an eye on the blog every once in a while. You might find an elegant solution a problem you might have encountered. Keep reading it 🙂


#markus IRC channel

We have our IRC channel, #markus, on That’s the place where you can meet developers and can get almost instant help for a MarkUs related problem you might have. You can also just drop by and say hello. It’s a friendly small community hanging out there. In short, it is a good place to chat with current and former developers or other lurkers, who tend to hang out on IRC. See you there! 🙂

Written by Benjamin Vialle

October 16th, 2010 at 1:11 pm

In-memory database speeds up tests, but not in a such great scale.

without comments

It has been observed that test suites with Machinist are slower than those with transactional fixtures . This is because the bottleneck of test performance lies in database operations. Machinist normally requires much more database queries than transactional fixtures do. To solve the performance issue related to Machinist, we should either speed up database operations or reduce the number of database queries.

There is a type of database known as in-memory database. As its name suggests, it stores data in memory instead of file systems. Memory access is a lot faster than file access, and therefore, an in-memory database should have performance advantage over an normal file-based database. By using in-memory database for testing, database operations can be faster and thus test speed will go up.

We tried in-memory SQLite3 following the instructions on I expected in-memory SQLite3 would have an enormous speed boost, but  an experiment comparing PostgreSQL v.s. in-memory SQLite3 shows that it is no more than 20% faster.

The experiment runs several ruby files with test database set to PostgreSQL and in-memory SQLite respectively. Here is the code in config/database.yml that sets the test database to in-memory SQLite3 and PostgreSQL:

# setup test db to in-memory SQLite3

adapter: sqlite3
database: ":memory:"
verbosity: quiet

#setup test db to PostgreSQL

adapter: postgresql
encoding: unicode
database: markus_test
username: markus
password: markus

I pick test/assignment_test.rb as the first benchmark, because this test suite uses Machinist and has heavy database operations.  On my laptop, it takes around 15.6s to run test/assignment_test.rb when in-memory SQLite3 is used, and it takes around 18.5 s when PostgreSQL is used. In-memory SQLite3 is only 18% faster for a typical Machinist test.

The other benchmark I use is a temporary source file, performance_test.rb in the root directory of the project:

# markus_root/performance_test.rb

ENV["RAILS_ENV"] = "test"
require File.expand_path(File.dirname(__FILE__) + "/config/environment")
require File.join(File.dirname(__FILE__), "test/blueprints/helper")

(ActiveRecord::Base.send :subclasses).each {|c| c.delete_all}

t1 =

5000.times { Section.make }

t2 =

puts "time span: #{t2-t1}s"

It gives 53.6s for PostgrSQL and 45.2 for in-memory SQLite3. In this case, in-memory SQLite3 is 19% faster.

19%, not as good as expected, is it? Yet, this is understandable. Using in-memory database is not equal to keeping objects within the same memory space used by the ruby application. The former requires interprocess (or sometimes internet) communications between a ruby application and a database, which itself is slow. And I suspect a big fraction of db operation is the communication between an application and a db, and thus we do not see a significant speed-up when using in-memory SQLite3. For this reason, I think that reducing the number of db queries is a better way to improve test speed. This can be testified if we observe the running time of an earlier version of assignment_test where transactional fixtures were used. This test suite runs in 11.6s even if PostgreSQL is the db, compared to 15.6s using in-memory SQLite3 for Machinist tests. So my conclusion is that in-memory SQLite is faster than file-based databases, but this should not be the reason to abandon transactional tests, because transactional tests are much faster than non-transactional ones.

Written by bryanshen

April 19th, 2010 at 2:32 am

Posted in tests

Tagged with ,

Getting Started with Machinist in Markus

with 3 comments

When running, we almost always have to set up some objects to test. We used to set up these objects by loading them from fixtures. There is a problem with fixtures. It’s hard to know the attributes of the objects or the relationships between different objects unless we delve into fixture files. This is not good, because when doing tests and making assertions, we want to know well about the objects we tested against, but looking into fixtures is a time-consuming job. Now don’t you want a mechanism that allows you to explicitly create objects and build up the relationships among them, but without having to specify their attributes that you don’t care about? This is what Machinist does! With Machinist, you can create and save an assignment object by writing one short line in your code: “assignment = Assignment.make”. Isn’t that great?

In his post, Fixtures are not as clear as Machinist, Robert has a deep insight on how Machinist is better than fixtures.

Yes, Machinist is great, and I promise you’ll find it even greater if you spend a little time reading through Machinist’s documentation. Make sure you understand what a blueprint is and how we can pass attributes to “make” method and override its default behaviour.

The documentation pretty much tells all you need to know about Machinist. I’m not going through every details here, but there are some stuffs you may want to know about how we can use Machinist with Markus.

1. Setting up the playground for Machinist

Before using Machinist, please make sure you’ve installed two gems: Machinist and Faker. If they are not on your machine, you can run the following command in a terminal:

 gem install faker machinist --source 

Now that the gems are installed, you want to build some confidence in Machinist by making some easy objects. The easiest way to play around with Machinist is to open a script/console with test environment. You can do this by opening a terminal, change the directory to the root directory of your Markus’ project, and then run the following command:

 script/console test 

Make sure you append the “test” option, which loads test environment for you. Otherwise you’ll be at the risk of screwing up data you use for dev.

If you see something like

Loading test environment (Rails 2.3.5)

you’re already in the script console. In script console, you can execute any Ruby code as you would normally write in .rb source files. If you’ve not used script console before, try to “new” an empty group object to get yourself familiar with it.

>> group =
=> #<Group id: nil, group_name: nil, repo_name: nil>

“group =” creates a new empty group, whose information is displayed in the console.

Now you probably cannot wait to use Machinist. OK, let’s start by making a group which is the easiest model I can find in Markus. Try to execute the following line in the script console.

>> group = Group.make

But you’re likely to get this:

NoMethodError: undefined method `make' for Group(id: integer, group_name: string, repo_name: string):Class
 from /var/lib/gems/1.8/gems/activerecord-2.3.5/lib/active_record/base.rb:1959:in `method_missing_without_paginate'
 from /var/lib/gems/1.8/gems/will_paginate-2.3.12/lib/will_paginate/finder.rb:170:in `method_missing'
 from (irb):5

Why? If you’ve read the Machinist’s documentation, you should’ve already known the reason. You cannot use the “make” method of a model unless you’ve defined the its blueprint. OK, then let’s write the blueprint. Wait, in fact, you don’t need to, because blueprints for most models in Markus are already defined in /test/blueprints/blueprints.rb, which is “required” by /test/blueprints/helper.rb. To make a reference of these blueprints, you only need to run the following line in script console:

>> require "test/blueprints/helper.rb" 

Now try to “make” your group object again, and see what happens. If you still get an error, please don’t panic, and skip ahead to read the next section, where you’ll find the answer. But most likely you will be lucky to see a sign of success in your script console like the one I get here:

=> #<Group id: 930503258, group_name: "group1", repo_name: "group_930503258"> 

See, with only one simple call of “make”, you have already created a group object with all the attributes set properly. That’s why we all like Machinist!

Congratulations. You’ve already set up the playground for Machinist. At this point, you are good to go. Just try out any of your ideas with Machinist.

2. Common Issues

Sometimes, when you make a make an object, you get an error message like this: “ActiveRecord::RecordInvalid: Validation failed: Group name has already been taken”. This message means in the database there is already a group object with the same group name as the one you are just making. The test database can persists objects as a result of your last test, so you should clear the database before playing around with Machinist. To delete all the objects of a model in database, you can call “ModelName.delete_all”, where ModelName is the name of the Model whose objects you want to delete. But it’s is more desirable to clear all the table at the same time. Here is the code to do that:

(ActiveRecord::Base.send :subclasses).each {|c| c.delete_all}

I recommended you to use script console. But it has an issue. Everytime you modify and save blueprints.rb, you need to restart the script console for it to detect the change. This makes it inconvenient for you to do test your blueprints. As an alternative to script console, you can create a .rb file at your project’s root directory, and put the following lines at the beginning:

ENV["RAILS_ENV"] = "test"
require File.expand_path(File.dirname(__FILE__) + "/config/environment")
require File.join(File.dirname(__FILE__), "test/blueprints/helper")

This should also set you up a test environment where you can write code to play around with Machinist. The benefit of this approach is that it makes it easy to test blueprints.

3. Making Objects with Associations

There are three common types of associations: “belongs_to”, “has_many” and “has_one”.

belongs_to“: if A “belongs_to” B, a.b should be not nil and unique. It’s identical to a foreign key relationship from A to B. B should be made prior A. A’s blueprint should take care of making B. For example in Markus, Student belongs Section. The blueprints for Section and Student are:

Section.blueprint do

name {Sham.section_name}

Student.blueprint do
type {'Student'}
user_name {Sham.student_user_name}

The blueprint of Student implicitly calls Section.make. Therefore, when you call Student.make, a section will be made before student. If you already have a section object my_section and want to make a student that belongs to that section, you can call Student.make({:section => my_section}), which overrides the default behaviour specified by Student blueprint and doesn’t make a new section. When you make an object, you may end up making lots of  other objects because the “belongs_to” association “recursively” makes the dependencies.

has_many“:  if A “has_many” B, then B “belongs_to” A. Machinist encourages you to first create the object on the right hand side of “belongs_to” association (seeking for some better wording here). Using the example of Student and Section. Section belongs to Section, and thus Section has many Students. When you want to make a section with many students, you need to make a section first. We are still using the blueprints from the previous example:

section = Section.make


When section.students.make is called, Machinist is smart enough to call  Student.make({:section => section}). So you don’t need to worry about accidentally creating a new section. After the two lines, you have section.students.first.section == section.

has_one“: “has_one” is a special case of “has_many”. if A “has_one” B, then B “belongs_to” A. Same as “has_many”, A should be made prior to B. Suppose we have two models: Coach and Team, where Team “has_one” Coach and Coach “belongs_to” Team. You may attempt to make an object of Team in a similar way to “has_many” association:

team = Team.make

coach =

But it doesn’t work. In the “has_many” example, section.students is a collection which is an object not nil, but here is nil. There is no way for Machinist to know which blueprint to use for making a nil object. So what you have to do is:

team = Team.make

coach = Coach.make({:team => team})

Then you can use the team object.

Submission.blueprint do
submission_version {1}
submission_version_used {true}
revision_number {1}
revision_timestamp {1.days.ago}

Written by bryanshen

March 10th, 2010 at 3:51 am

Functional Testing

with 2 comments

Functional testing is the process of trying to find errors in a software behaviour, or in a software component behaviour. We expect 2 things from functional tests:
  1. Validating that the software does exactly what we expect;
  2. Validating that the software does not do what we not expect it to do.

In fact, the goal of functional testing is not to demonstrate that the software does not contains any error, but to demonstrate that the software does what the client expect it to do.

The functional test could be as simple as testing every behaviour of a program to validate that the behaviour meets the specifications. Usually, proceeding to an exhaustive testing is just unreasonable. There could be to many cases and would be just impossible to test everything in a reasonable time. So it is important to select within all possible behaviours, the ones that ensure the best program coverage. It means, the behaviours that are the most representatives of the program general behaviour and the particular cases.

To do our functional tests, our suggestion is to use Selenium. Selenium is implemented as a Firefox extension, and allows us to record, edit, and debug tests. With Selenium, we can also play back our tests when required.

Selenium Installation

  1. Download Selenium IDE FireFox plugin(
  2. Make sure java 1.5 or higher is installed on your computer.

How to test with Selenium

1- Open a command window and start the listener

  • Go into the following PATH: selenium-remote-control-1.0.1\selenium-server-1.0.1
  • Type: java -jar selenium-server.jar

2- Open the IDE recorder in FireFox (Tools/Selenium IDE)


3- Be sure that your recorder is on and do every step a user would have to do in the application to get the desired action.  Per example if we want to test that a message of success is displayd when an assignment is created do the following:

  • Create an assignment;
  • Right-click on the text: Successfully created assignment;
  • Choose a verification to do.

Selenium menu

4- In the recorder :

  • Stop recording
  • File/Export Test Suit As…/Ruby-Selenium RC
  • Give a representative file name to your test and save it into the following PATH: test/selenium/MyTest.rb

Selenium Export as Ruby

5- The test is exported as Ruby Code and can be modified.  This is how the test code looks like:


6- Logout from Markus and run your test

  • Open a command window
  •  Type: ruby test/selenium/MyTest.rb


  • Teardown section should be used to delete all the records created by the test
  • We should also create a program for the login that could be used by every test that requires to login the application.
  • A random string library could also be useful.

For more information about Selenium

Written by melagaud

October 6th, 2009 at 6:08 pm

Posted in tests

Unit Testing

with one comment

“The primary goal of unit testing is to take the smallest piece of testable software in the application, isolate it from the remainder of the code, and determine whether it behaves exactly as you expect”1. Unit tests are done before integrating the unit into the system to assure quality and save time due to errors.

While unit testing, other units are considered bug free. Unfortunately, we don’t live in a perfect world and it’s impossible to ensure bug free units. For this reason, other units can affect the behavior of the tested unit. To solve this situation, isolation is required and since all the unit tests have to run under 5 minutes, it also helps to boost the speed.

In this example, while testing the Student class, calls done to the Repository, Grouping and Database have to be stubbed and mocked.

Unit Test Isolation
All the next examples are coded in Ruby and will use the mocha framework.
How to test
Assertions are used to test the behavior of the code.
For example, this is the test of a method changing the “hidden” attribute of the provided students to true.

def test_hide_students
student1 = users(:student1)
student2 = users(:student2)

student_id_list = [,]


students = Student.find(student_id_list)

# The test fails if the assertion is false
assert_equal(true, students[0].hidden)
assert_equal(true, students[1].hidden)

How to achieve isolation

Stub objects are used to bypass a call and return the expected result. For example, to test this method in the Student class, stubs are required.

def self.hide_students(student_id_list)
# A stub will bypass the call to repository_external_commits_only?
#to return false or true depending on the test case.
if group.repository_external_commits_only? &&membership.grouping.is_valid?
group.repo.remove_user(student.user_name) # revoke repo permissions
rescue Repository::UserNotFound
# ignore case when user isn't there any more

In the previous code, to ensure the students are removed from the repository, stubs are used to enter the “if” condition.

Here is the code testing this method :

def test_hide_students_repo_remove_user
student1 = users(:student1)
student2 = users(:student2)

# Stubs to enter into the if
# Calls to the method “repository_external_commits_only?”
for any Group instance will return true

student_id_list = [,]
assert Student.hide_students(student_id_list)

How to guarantee behavior with mock objects
Mock objects are used to assure the unit calls an expected method with expected arguments.

Let’s go back to the previous example, hide_students(student_id_list) in the Student Class. To guarantee the method removes the correct user from the repository, a mock object expecting to be called will be used.

To confirm this method is called :

group.repo.remove_user(student.user_name) # revoke repo permissions

a mock object is used :

# Mock the repository and expect :remove_user with the student's
mock_repo = mock('Repository::AbstractRepository')

#Stub to return true
mock_repo.expects(:remove_user).with(any_of(student1.user_name, student2.user_name)).at_least(2)

During the test, if “remove_user” is not called at least 2 times with either “student1.user_name” or “student2.user_name”, the test fails.

-It helps to think about all the possible cases;
-It is easier to refactor the code when a net (tests) can catch all the possible errors;
-Boost the confidence in your code.

Best practices
-Never commit code that isn’t tested;
-Always run all the tests before committing;
-If you change something and the tests fail, correct the situation;
-Clean tests are the best!

Examples in MarkUs
Look at student_test.rb and annotation_categories_controller_test.rb

Useful links
Unit testing
XUnit Patterns
Mocks aren’t stubs

Frameworks used in MarkUs :

All the next examples are coded in Ruby and will use the mocha framework.
1 Unit Testing –

Written by simonlg

October 3rd, 2009 at 12:51 am

Posted in tests