MarkUs Blog

MarkUs Developers Blog About Their Project

Fixtures are not as clear as Machinist

with 4 comments

I’m in the process of converting some unit tests (for Students and Assignments) away form fixtures and onto Machinist. I decreed at the weekly meeting that I was going to be finished this by the end of the weekend, and from the time this is posted, I’ve failed. I started later than I should have, and clearly didn’t account for enough debugging time.

“But wait!” I hear you clamour. “You’re just moving tests from one framework to another, right? Surely such things could be done by robots and other automatons?” Indeed! In an ideal world this would be true. But in this world, we will encounter exceptional circumstances and end up using our brain. Here is my tale…

This is a telling observation from the fixture tests:

Half the time, I’m not sure what’s being tested.

Here is a typical example:

<br />
def test_create_group_for_working_alone_student_existing_group<br />
    assignment = assignments(:assignment_1)<br />
    student = users(:student1)<br />
    # The test code...<br />
end<br />

The rote way of converting this to machinist is like so:

<br />
def test_create_group_for_working_alone_student_existing_group<br />
    assignment = Assignment.make<br />
    student = Student.make</p>
<p>    # The test code...<br />
end<br />

Pretty simple right? Awesome!

However, all is not well in testerland. Watch when I run it:

<br />
Testing started at 10:43 PM ...</p>
<p>NoMethodError: undefined method `allow_web_submits' for nil:NilClass<br />
app/models/group.rb:49:in `repository_external_commits_only?'<br />
    app/models/grouping.rb:333:in `update_repository_permissions'<br />
    app/models/student.rb:118:in `create_group_for_working_alone_student'<br />
    test/unit/student_test.rb:142:in `test_create_group_for_working_alone_student_existing_group'<br />
    /Applications/RubyMine 2.0.1.app/rb/testing/patch/testunit/test/unit/ui/testrunnermediator.rb:36:in `run_suite'<br />
    /Applications/RubyMine 2.0.1.app/rb/testing/patch/testunit/test/unit/ui/teamcity/testrunner.rb:215:in `start_mediator'<br />
    /Applications/RubyMine 2.0.1.app/rb/testing/patch/testunit/test/unit/ui/teamcity/testrunner.rb:191:in `start'<br />
1 tests, 0 assertions, 0 failures, 1 errors<br />
Test suite finished: 0.860263 seconds</p>
<p>Process finished with exit code 1<br />

Now there’s a tasty bowl of error soup!
What the heck happened? We were using a student and an assignment, shouldn’t it just work?

I look into the fixtures for both :student1 and :assignment_1 to see what went wrong.

<br />
# in users.yml<br />
student1:<br />
  user_name:      student1<br />
  last_name:      Mason<br />
  first_name:     Hollis<br />
  grace_credits:     5<br />
  type:           Student<br />
  section_id:     <%= Fixtures.identify(:section1) %><br />
  created_at:     <%= Time.now.to_s(:db) %><br />
  updated_at:     <%= Time.now.to_s(:db) %><br />
  hidden: false</p>
<p># in assignments.yml<br />
assignment_1:<br />
   short_identifier: Captain Nemo<br />
   description: "Code!!"<br />
   message:<br />
   due_date: <%= 30.days.from_now.to_s(:db)%><br />
   group_min: 2<br />
   group_max: 4<br />
   student_form_groups: true<br />
   instructor_form_groups: false<br />
   group_name_autogenerated: true<br />
   group_name_displayed: true<br />
   created_at: <%= Time.now.to_s(:db)%><br />
   updated_at: <%= Time.now.to_s(:db)%><br />
   repository_folder: CaptainNemo<br />
   marking_scheme_type: "rubric"<br />
   allow_web_submits: false<br />

Both look pretty routine, so now I’m at a loss of what to do. The only thing I’ve got to go on is the error message. But why am I getting it in the first place? The only thing that looks related to the error is the allow_web_submits property, which is being set to false. I update the make line to the following, to ensure consistent results.

<br />
    assignment = Assignment.make(:allow_web_submits => false)<br />

That’s not the big error though, why in the world is this being checked on a nil object? Where’s that happening? It’s raising in groups.rb, but where the heck were those set up?

Oh. They are… They are just in the yml files for the groups, groupings, and memberships fixtures.

A little tinkering around and my knowledge of how Assignments, Students, Groups, Memberships, and Groupings are organized gives me the final working test, that emulates the relevant conditions in the fixtures.

<br />
  def test_create_group_for_working_alone_student_existing_group<br />
    group = Group.make<br />
    assignment = Assignment.make(:allow_web_submits => false)<br />
    grouping = Grouping.make(:group => group, :assignment => assignment)<br />
    membership = StudentMembership.make(:membership_status => 'inviter', :grouping => grouping)<br />
    student = membership.user<br />
    # Same old Test code....<br />
  end<br />

Not at all the same, is it? The fixtures were hiding all the delicious dependancy information from me. In the end, I had to improvise a bit. From all the information, and time spent in the yml files determining the purpose of this particular test wasn’t easy. You needed to know exactly how the fixtures were set up in relation to each other, which involved look into half a dozen files. This is not ideal.

The purpose of a good test should be clear to anyone with some information about the system tested. In the case above, the test was that a student with a grouping already doesn’t cause an error when put into a separate group by themselves.

Beyond the assertion and the function name, I didn’t know what’s being tested. I didn’t know why those particular fixtures, :student1 and :assignment_1, were picked from the other students and assignment fixtures. The only way to find out is to directly delve into the .yml files that define them. That didn’t even guarantee that I understood the purpose of the test.

Before I submit this, I’ll likely need to check to see if each method is being adequately tested. Even if the tests were comprehensive under fixtures, it’s not clear that the same conditions are being met now. Thats not a vote to stay with fixtures though. Fixtures are bad because they move the preconditions away from the test, making it unclear under what circumstances the desired behaviour is to occur. Such a framework becomes brittle when writing new tests. Would you want add new fixtures, or see if some combination of the previous ones emulate the conditions of your test? With Machinist, you can construct what you need. When some other dev comes along, they might not know exactly why it was set up that way, but they will know what the setup is, and how you want the parts to be tested to interact.

Written by Robert B

February 22nd, 2010 at 1:48 am

4 Responses to 'Fixtures are not as clear as Machinist'

Subscribe to comments with RSS or TrackBack to 'Fixtures are not as clear as Machinist'.

  1. This is really helpful, thanks!

    Tara Clark

    22 Feb 10 at 2:53 pm

  2. […] a time. Occasionally there’s a sticking point that has me butting my head against a wall (converting fixture based tests to machinist for example), but that has taught me to take advantage of the ease of communicating with my fellow devs on the […]

  3. Extremely helpful post. I had the same problem when I need to setup some test data using Machinist. Using Machinist, objects need to be explicitly constructed, whereas Fixtures hides most of the details about how the associated objects are constructed.

    bryanshen

    8 Mar 10 at 2:28 am

  4. […] his post, Fixtures are not as clear as Machinist, Robert has a deep insight on how Machinist is better than […]

Leave a Reply