One of the things I’ve done since I joined the project, was to look at how we could make the test suite cleaner and more up-to-date.
That task implied some test debugging, and one annoying thing I bumped into was fixtures. A little internet roaming showed that I’m not alone to be uneasy with them.
- Are Rails Test Fixtures Good or Evil?
- Rails fixtures – help or hindrance
- Replacing Ruby on Rails fixtures with Factories and Builders
- found on rspec-users mailing list: Rails’ fixtures suck! But what about something like this?
- public survey: Rails fixtures don’t scale well
Some interesting fixture definition can be found if you follow the first link:
Accessories fixed to structures or land in such a way that they can’t be independently moved without damage to themselves, or the property housing them.
Which applies strangely well to how I fell about fixtures when I use them.
Some Bad Smells
While writing tests, how many times have you asked yourself questions like:
- Which student was part of group_1?
- Was assignement_1 submitted?
- Was student_3 a member of group_3 or group3? (this one is a good example of maintainability issues)
- Am I going break dozens of tests if modify that fixture field?
To me, fixtures feature the following problems:
- They make necessary to explore multiple files to understand a particular test;
- It’s even more complicated to figure out the links between records;
- They generate more data than needed by any test case;
- They open the way to tests that pass alone but fail in the suite;
- Editing a particular fixture may have side-effects in many tests;
- It is a common statement that all of the above should not be problems if you maintain your fixtures adequately. But I found fixtures are hard to maintain.
Exploring for alternatives I stumbled upon the following projects (reduced the list to those that looked appealing):
I did read about them all, but have not tried them all. A short report on how my experiences went follows.
First attempt – FactoryGirl
My first tries were with FactoryGirl. Influenced by the fact that we were already satisfied with Shoulda and that it got the best rating on the Ruby Toolbox.
I did face some problems though. For instance, “building” an assignment was “building” a SubmissionRule that couldn’t be saved because the
assignment_id field was
nil. I did not manage to make it build in the correct order.
I am certainly not stating this is impossible. Maybe it’s just that I did not caught the FactoryGirl philosophy quick enough. I probably was missing something.
There were also issues with the way we generate fake memory repositories in the
test_helper, but I believe we are going to get those whatever the fixture replacement we opt to use.
Second attempt – Machinist
Then I went with Machinist. First good thing I noticed about it was its syntax. It is much lighter and fun to use that FactoryGirl’s. So I did applied myself in writing blueprints. They are the Machinist artefact that let us tell him how we want our objects to be generated, supplying default values that we can override later according to our particular needs not having to mention the data we do not care about when writing a test.
The whole process is really neat. I created a review request so that you can all take a peek at how I made things work. There still are no blueprints for each of our model object, but I managed to (easily) write working blueprints for many of our classes. I would approximate, twice the number of classes (without hassle) in half the time compared to FactoryGirl.
Once more, I do not want to sound like bashing on FactoryGirl, which definitely look like a great tool and that a lot of people out there are using. I am simply reporting on my experience, and, to me, the bottom line is, Machinist came a lot more naturally.
As for FactoryGirl, Machinist does not happily coexists with fixtures (partly because of fake test repositories), enforcing the idea that if we replace the fixtures, we’ll have to completely migrate our test suite — which makes sense since it all depends on fixtures. This is huge work ahead.
What is to be Gained?
First things that come to mind:
- Easier to maintain test suite;
- Reduced side-effects when modifying;
- More test readability — Tremendous enhancements have been made in that field since the beginning of the semester, but there is always room to do better;
What is to be lost?
We can expect some performance loss (during test execution) if we switch to a data generator (whatever which one). This is bad. But if we want our unit tests to run fast, we should consider letting them be real unit tests and not hit the database at all. Then we could have, say, model tests that verify the relation between our objects using generated data.
Temporary Final Word
An eventual transition from fixtures to, say, Machinist, would take quite some time and would introduce even more inconsistencies in the test suite until completed. That definitely has to be kept in mind. This is no piece of cake. But the final product would most probably be a better/stronger/easier to maintain/easier to read test suite which is not something we should overlook.
My vote? Let’s pack all the courage we got and sail toward a fixture-less world for the benefit of the generations to come!
- An Agile development valued practice.