MarkUs Blog

MarkUs Developers Blog About Their Project

Fixture replacement

with 4 comments

One of the things I’ve done since I joined the project, was to look at how we could make the test suite cleaner and more up-to-date.

That task implied some test debugging, and one annoying thing I bumped into was fixtures. A little internet roaming showed that I’m not alone to be uneasy with them.

Some interesting fixture definition can be found if you follow the first link:

Accessories fixed to structures or land in such a way that they can’t be independently moved without damage to themselves, or the property housing them.

Which applies strangely well to how I fell about fixtures when I use them.

Some Bad Smells

While writing tests, how many times have you asked yourself questions like:

  • Which student was part of group_1?
  • Was assignement_1 submitted?
  • Was student_3 a member of group_3 or group3? (this one is a good example of maintainability issues)
  • Am I going break dozens of tests if modify that fixture field?

To me, fixtures feature the following problems:

  • They make necessary to explore multiple files to understand a particular test;
  • It’s even more complicated to figure out the links between records;
  • They generate more data than needed by any test case;
  • They open the way to tests that pass alone but fail in the suite;
  • Editing a particular fixture may have side-effects in many tests;
  • It is a common statement that all of the above should not be problems if you maintain your fixtures adequately. But I found fixtures are hard to maintain.

The alternatives

Exploring for alternatives I stumbled upon the following projects (reduced the list to those that looked appealing):

I did read about them all, but have not tried them all. A short report on how my experiences went follows.

First attempt – FactoryGirl

My first tries were with FactoryGirl. Influenced by the fact that we were already satisfied with Shoulda and that it got the best rating on the Ruby Toolbox.

I did face some problems though. For instance, “building” an assignment was “building” a SubmissionRule that couldn’t be saved because the assignment_id field was nil. I did not manage to make it build in the correct order.

I am certainly not stating this is impossible. Maybe it’s just that I did not caught the FactoryGirl philosophy quick enough. I probably was missing something.

There were also issues with the way we generate fake memory repositories in the test_helper, but I believe we are going to get those whatever the fixture replacement we opt to use.

Second attempt – Machinist

Then I went with Machinist. First good thing I noticed about it was its syntax. It is much lighter and fun to use that FactoryGirl’s. So I did applied myself in writing blueprints. They are the Machinist artefact that let us tell him how we want our objects to be generated, supplying default values that we can override later according to our particular needs not having to mention the data we do not care about when writing a test.

The whole process is really neat. I created a review request so that you can all take a peek at how I made things work. There still are no blueprints for each of our model object, but I managed to (easily) write working blueprints for many of our classes. I would approximate, twice the number of classes (without hassle) in half the time compared to FactoryGirl.

Once more, I do not want to sound like bashing on FactoryGirl, which definitely look like a great tool and that a lot of people out there are using. I am simply reporting on my experience, and, to me, the bottom line is, Machinist came a lot more naturally.

As for FactoryGirl, Machinist does not happily coexists with fixtures (partly because of fake test repositories), enforcing the idea that if we replace the fixtures, we’ll have to completely migrate our test suite — which makes sense since it all depends on fixtures[1]. This is huge work ahead.

What is to be Gained?

First things that come to mind:

  • Easier to maintain test suite;
  • Reduced side-effects when modifying;
  • More test readability — Tremendous enhancements have been made in that field since the beginning of the semester, but there is always room to do better;

What is to be lost?

We can expect some performance loss (during test execution) if we switch to a data generator (whatever which one). This is bad. But if we want our unit tests to run fast, we should consider letting them be real unit tests and not hit the database at all. Then we could have, say, model tests that verify the relation between our objects using generated data.

Temporary Final Word

An eventual transition from fixtures to, say, Machinist, would take quite some time and would introduce even more inconsistencies in the test suite until completed. That definitely has to be kept in mind. This is no piece of cake. But the final product would most probably be a better/stronger/easier to maintain/easier to read test suite which is not something we should overlook.

My vote? Let’s pack all the courage we got[1] and sail toward a fixture-less world for the benefit of the generations to come!


  1. An Agile development valued practice.

Written by gabrielrl

December 4th, 2009 at 6:51 pm

Posted in Uncategorized

4 Responses to 'Fixture replacement'

Subscribe to comments with RSS or TrackBack to 'Fixture replacement'.

  1. Gabriel:

    Great write up!

    From this, and the review request you posted, I’m pretty sold on Machinist.

    But you’re right – it’s a pretty big job to migrate all of those tests…



    8 Dec 09 at 5:51 pm

  2. “Model” tests are a great idea and should be done. Units tests should never communicate with the database.

    I’m with Gabriel on this one. Fixtures are nice for small applications but really hard to maintain.


    14 Dec 09 at 3:42 am

  3. Thanks for your post, I searched for “Machinist Performance” and finally reached this blog.

    My concern is that, if not implemented carefully, test cases written in machinist would be really slow. I’m trying to find a pattern for writing machinist tests that prevents performance overhead. Do you have any ideas?



    5 Mar 10 at 6:54 pm

  4. Unfortunately I haven’t used Machinist enough to become advanced user. I have no pattern in particular to recommend.

    All I can think of right now is: I would be really careful using shoulda’s nested contexts. Although they’re great for readability, you must not forget that every (nested) setup block (up to the first level) will be executed once per test (should macro). I think I would be careful and make sure that every ”should” in a ”context” makes use of every class attribute defined in the corresponding context.

    That’s nothing new, but the overhead of unused class attributes will be greater when using an object generator like Machinist.

    (sorry for the late response)


    31 Mar 10 at 7:53 pm

Leave a Reply