Archive for the ‘acceptance test’ tag
One thing (among lots of others) I have learned about tests during last semester is that, depending on the environment (or the culture you’re developing in) tests get different names.
Here’s an example where Rails’ naming conventions differ from what I have learned in my QA course:
Rails’ Unit Tests are not Unit Tests
Considering a unit test should test an object in complete isolation, it is a breach to that rule to have a “unit” test access the database. Rails’ unit tests are really functional tests where we make sure that the model behaves accordingly (for one) and that its mapping with the database is correct (for two). Another thing about unit tests is that they should be fast to run. Under 30 seconds is a generally good standard (varying depending on the environment). Let me guess, you need somewhat more than 30 seconds to run
rake test:units on MarkUs. For example, on my machine I get a run time of approximately 31 seconds, which would be good, if it was not for the fact that it does not include the database setup time as well as (I believe) setup and teardown time between each test.
Side-note: I also get 1 error and 1 failure. It makes me want to emphasize that fast-to-run tests get ran more often. It the tests are not used, they loose their ability to help find bugs early, get deprecated and more of a burden that a great tool.
On the other hand. Every model object is so tightly coupled with ActiveRecord that one could argue that we do not really want to test our model classes without ActiveRecord participating.
Anyway, this is an example of what I was stating in the first paragraph. In this case, it is commonly accepted, in Rails’ culture, that, what I could refer to as “open box integration testing of models and ActiveRecord” should be called “a unit test”. The later has some clear advantages over the former and the most important thing is to make sure that everyone is speaking the same language among the team.
Refactoring the test/ folder
As Adam told us during code sprint. It is a common practice, among Rails’ developer teams, to refactor the test folder on the very start of a new project. One way of doing this could give something like:
- test/ remains test/
- unit/ becomes model/
- functional/ becomes controller/
- unit/ gets added (and contains model unit tests that does not hit the database)
- functional/ gets added (and contains selenium tests)
- acceptance/ gets added (and contains cucumber tests)
I have myself been pretty loose all these times I have reffered to Cucumber tests simply as acceptance tests. If you dig a little, you’ll find that I was more specifically talking about “user acceptance” tests.
Convention Over Elite Correctness
It’s written up there, but the key thing is to speak the same language among the team. You’ve experienced it, Rails is all about conventions. Keeping the usual setting may help newcomers to the project, but used Rails developers, to understand exactly what is supposed to go where (and what they can expect to find where) when it comes to test.
On the other hand, from what I can tell, newcomers to the project also are newcomers to Ruby and Rails, which, I believe, leaves the way open for a little structure refactoring.
- Which, in turn, was not proposing an absolute answer to “how a specific test should be named?” but rather focused on providing us with the necessary tools (read: knowledge) to efficiently and intelligently test the software we develop.
- Does anybody know?
- and I am no authority on the question. I also intentionally made it too long for its own good 😉
Cucumber is now integrated in MarkUs’ trunk. Hurray!
I greatly encourage you to try it, it’s pretty simple. First, update to the last revision (just like you do every morning before breakfast, right?) and then visit the wiki page. After reading the requirements and running the tests sections you’re ready for a test run.
I’ll make sure to update the wiki/post/code with any feedback you provide.
About Acceptance Tests
Here are some guidelines to keep in mind when you write acceptance tests.
Speak the Client’s Language
Acceptance test are directed at the client. This is why they should speak your mother tongue and use domain specific language. There should be no reference to the code and it should conceptually stay at the user level.
You Don’t Have to Test Everything
You probably know by now that it’s impossible to test everything. But unlike when you write a unit or a functional test, which you want to be as thorough as you can, an acceptance test aims at asserting that a particular feature is actually there. So, most of the time, one case per features’ functionality is enough. Also, you don’t want to try for error cases. It’s just not the place, unless it is explicitly requested by the client, error case tests belongs to unit/functional tests.
An acceptance test output should not get convoluted with tons of specific border-line test cases. The output should be just what is needed for the client to assert that all of his requested functionalities are implemented and that they do work. That is, give the client some insight on whether or not he should accept the software.