Archive for December, 2009
The last couple of days I’ve been thinking of a way to do user authentication for MarkUs’ Web API. This is what I’ve come up so far. Feedback and suggestions are very welcome.
- Instructors and TAs will be able to get a “key” via MarkUs. When an instructor logs in the API key (a SHA2 512 bits in length) is displayed to him on the dashboard. TAs will see the key below the list of the assignments when they log in.
- The goal is that the private key never shows up in a script.
- In order to use this key for authentication from within a script the following steps are required:
- A MD5 hash of the private key has to be generated.
- This MD5 hash will be then encoded using Base 64 and the result will be used as a token and travels over the wire for each request.
- The token, generated as described above, will be sent to the MarkUs server by facilitating the HTTP header “Authorization”. I.e. each request sent to the MarkUs API has to include this header with the appropriate token.
- Since MarkUs knows about the private key and that the token is a Base 64 encoded MD5 digest of the key a matching user can be easily determined and the private key never has to be included in scripts on semi-private servers, where the scripts are run.
I was thinking that the Authorization HTTP header (which is otherwise used for Basic or Digest schemes) could have the following form:
Authorization: MarkusCustom MDk4ZjZiY2Q0NjIxZDM3M2NhZGU0ZTgzMjYyN2I0ZjYK
So far my thoughts. What do you think? Concerns? Am I on the wrong track? Thanks!
You will recognize this report. It is the complete marking scheme analysis in which I have added one Chapter. Chapter 3 details the development that has been done and the development left to do in order to deliver the complete flexible marking scheme feature.
By the way, eventhough it is published on our DrProject since December first, we forgot to let you know that our screencasts are done. Please follow this links to see theme:
The meeting started at 12:02 and ended at 12:51.
The IRC log can be found here
- Final round table
- Post mortem
- High priority issues
Simple Grade Entry
- Farah said that she ran into some problems with the Table View implementation because she wasn’t able to use the Rails Form Helper
- Farah is going to learn about AJAX over the break and try to make the table cells be saved automatically, instead of requiring the user to hit “Submit”
- We decided that pagination (alphabetically, by user id) should also be implemented
- The Notes tab is done and checked in – for groupings
- Tara is going to create a screencast
- Tara and Fernando are both going to work on documentation
- Fernando is going to work on making sure the entire page doesn’t reload when attaching a note to a group from the Groups manager
Flexible Marking Scheme
- Their presentation went well
- Gabriel would like to add some Selenium tests and acceptance tests
- The application is nicely coded
- Communication was extremely good
- Great team
- Having Mike and Severin with lots of knowledge really jump started everything
- Weekly meeting
- Everyone was more than willing to help each other out
- Meeting everyone at the code sprint
- In-house design of software by students to be used by students
- Everyone jumped in on the reviews, took / gave constructive criticism well
- Karen couldn’t stay on top of everything and not enough careful design review was done
- It was hard to keep up with the reviews and so some stopped reading some of them
- Mike wishes he could have done more development
- Code Reviews were overwhelming
- Not knowing enough about web development was difficult
- Difficult to get design reviews when you’re not in-person
- Didn’t have more face-to-face communication with others
- It was hard to keep track of everything that was going on (eg. DrProject’s EventLog RSS feed)
- Karen said that she couldn’t be prouder of the work that was accomplished, the teamwork, the general atmosphere, the fact that *everyone* pulled their weight
Some high priority issues to think about
- Being able to upload results files without disturbing the repos. (Which also means a mechanism for downloading them.)
- A thorough review of the grace day stuff and making sure students and instructors can see how many grace days have been used.
- csv uploads for rubrics, mappings, comments
- Improving the group stuff so that groups can be formed after the deadline and before the grace period is over and double checking to make sure that groups can be deleted.
- It would be really nice to be able to have a view of the annotations for Dan
A current bug
- Grouping.has_submission? was reporting true (even without caching), and then Grouping.get_submission_used was reporting nil.
One of the things I’ve done since I joined the project, was to look at how we could make the test suite cleaner and more up-to-date.
That task implied some test debugging, and one annoying thing I bumped into was fixtures. A little internet roaming showed that I’m not alone to be uneasy with them.
- Are Rails Test Fixtures Good or Evil?
- Rails fixtures – help or hindrance
- Replacing Ruby on Rails fixtures with Factories and Builders
- found on rspec-users mailing list: Rails’ fixtures suck! But what about something like this?
- public survey: Rails fixtures don’t scale well
Some interesting fixture definition can be found if you follow the first link:
Accessories fixed to structures or land in such a way that they can’t be independently moved without damage to themselves, or the property housing them.
Which applies strangely well to how I fell about fixtures when I use them.
Some Bad Smells
While writing tests, how many times have you asked yourself questions like:
- Which student was part of group_1?
- Was assignement_1 submitted?
- Was student_3 a member of group_3 or group3? (this one is a good example of maintainability issues)
- Am I going break dozens of tests if modify that fixture field?
To me, fixtures feature the following problems:
- They make necessary to explore multiple files to understand a particular test;
- It’s even more complicated to figure out the links between records;
- They generate more data than needed by any test case;
- They open the way to tests that pass alone but fail in the suite;
- Editing a particular fixture may have side-effects in many tests;
- It is a common statement that all of the above should not be problems if you maintain your fixtures adequately. But I found fixtures are hard to maintain.
Exploring for alternatives I stumbled upon the following projects (reduced the list to those that looked appealing):
I did read about them all, but have not tried them all. A short report on how my experiences went follows.
First attempt – FactoryGirl
My first tries were with FactoryGirl. Influenced by the fact that we were already satisfied with Shoulda and that it got the best rating on the Ruby Toolbox.
I did face some problems though. For instance, “building” an assignment was “building” a SubmissionRule that couldn’t be saved because the
assignment_id field was
nil. I did not manage to make it build in the correct order.
I am certainly not stating this is impossible. Maybe it’s just that I did not caught the FactoryGirl philosophy quick enough. I probably was missing something.
There were also issues with the way we generate fake memory repositories in the
test_helper, but I believe we are going to get those whatever the fixture replacement we opt to use.
Second attempt – Machinist
Then I went with Machinist. First good thing I noticed about it was its syntax. It is much lighter and fun to use that FactoryGirl’s. So I did applied myself in writing blueprints. They are the Machinist artefact that let us tell him how we want our objects to be generated, supplying default values that we can override later according to our particular needs not having to mention the data we do not care about when writing a test.
The whole process is really neat. I created a review request so that you can all take a peek at how I made things work. There still are no blueprints for each of our model object, but I managed to (easily) write working blueprints for many of our classes. I would approximate, twice the number of classes (without hassle) in half the time compared to FactoryGirl.
Once more, I do not want to sound like bashing on FactoryGirl, which definitely look like a great tool and that a lot of people out there are using. I am simply reporting on my experience, and, to me, the bottom line is, Machinist came a lot more naturally.
As for FactoryGirl, Machinist does not happily coexists with fixtures (partly because of fake test repositories), enforcing the idea that if we replace the fixtures, we’ll have to completely migrate our test suite — which makes sense since it all depends on fixtures. This is huge work ahead.
What is to be Gained?
First things that come to mind:
- Easier to maintain test suite;
- Reduced side-effects when modifying;
- More test readability — Tremendous enhancements have been made in that field since the beginning of the semester, but there is always room to do better;
What is to be lost?
We can expect some performance loss (during test execution) if we switch to a data generator (whatever which one). This is bad. But if we want our unit tests to run fast, we should consider letting them be real unit tests and not hit the database at all. Then we could have, say, model tests that verify the relation between our objects using generated data.
Temporary Final Word
An eventual transition from fixtures to, say, Machinist, would take quite some time and would introduce even more inconsistencies in the test suite until completed. That definitely has to be kept in mind. This is no piece of cake. But the final product would most probably be a better/stronger/easier to maintain/easier to read test suite which is not something we should overlook.
My vote? Let’s pack all the courage we got and sail toward a fixture-less world for the benefit of the generations to come!
- An Agile development valued practice.
One thing (among lots of others) I have learned about tests during last semester is that, depending on the environment (or the culture you’re developing in) tests get different names.
Here’s an example where Rails’ naming conventions differ from what I have learned in my QA course:
Rails’ Unit Tests are not Unit Tests
Considering a unit test should test an object in complete isolation, it is a breach to that rule to have a “unit” test access the database. Rails’ unit tests are really functional tests where we make sure that the model behaves accordingly (for one) and that its mapping with the database is correct (for two). Another thing about unit tests is that they should be fast to run. Under 30 seconds is a generally good standard (varying depending on the environment). Let me guess, you need somewhat more than 30 seconds to run
rake test:units on MarkUs. For example, on my machine I get a run time of approximately 31 seconds, which would be good, if it was not for the fact that it does not include the database setup time as well as (I believe) setup and teardown time between each test.
Side-note: I also get 1 error and 1 failure. It makes me want to emphasize that fast-to-run tests get ran more often. It the tests are not used, they loose their ability to help find bugs early, get deprecated and more of a burden that a great tool.
On the other hand. Every model object is so tightly coupled with ActiveRecord that one could argue that we do not really want to test our model classes without ActiveRecord participating.
Anyway, this is an example of what I was stating in the first paragraph. In this case, it is commonly accepted, in Rails’ culture, that, what I could refer to as “open box integration testing of models and ActiveRecord” should be called “a unit test”. The later has some clear advantages over the former and the most important thing is to make sure that everyone is speaking the same language among the team.
Refactoring the test/ folder
As Adam told us during code sprint. It is a common practice, among Rails’ developer teams, to refactor the test folder on the very start of a new project. One way of doing this could give something like:
- test/ remains test/
- unit/ becomes model/
- functional/ becomes controller/
- unit/ gets added (and contains model unit tests that does not hit the database)
- functional/ gets added (and contains selenium tests)
- acceptance/ gets added (and contains cucumber tests)
I have myself been pretty loose all these times I have reffered to Cucumber tests simply as acceptance tests. If you dig a little, you’ll find that I was more specifically talking about “user acceptance” tests.
Convention Over Elite Correctness
It’s written up there, but the key thing is to speak the same language among the team. You’ve experienced it, Rails is all about conventions. Keeping the usual setting may help newcomers to the project, but used Rails developers, to understand exactly what is supposed to go where (and what they can expect to find where) when it comes to test.
On the other hand, from what I can tell, newcomers to the project also are newcomers to Ruby and Rails, which, I believe, leaves the way open for a little structure refactoring.
- Which, in turn, was not proposing an absolute answer to “how a specific test should be named?” but rather focused on providing us with the necessary tools (read: knowledge) to efficiently and intelligently test the software we develop.
- Does anybody know?
- and I am no authority on the question. I also intentionally made it too long for its own good 😉
Cucumber is now integrated in MarkUs’ trunk. Hurray!
I greatly encourage you to try it, it’s pretty simple. First, update to the last revision (just like you do every morning before breakfast, right?) and then visit the wiki page. After reading the requirements and running the tests sections you’re ready for a test run.
I’ll make sure to update the wiki/post/code with any feedback you provide.
About Acceptance Tests
Here are some guidelines to keep in mind when you write acceptance tests.
Speak the Client’s Language
Acceptance test are directed at the client. This is why they should speak your mother tongue and use domain specific language. There should be no reference to the code and it should conceptually stay at the user level.
You Don’t Have to Test Everything
You probably know by now that it’s impossible to test everything. But unlike when you write a unit or a functional test, which you want to be as thorough as you can, an acceptance test aims at asserting that a particular feature is actually there. So, most of the time, one case per features’ functionality is enough. Also, you don’t want to try for error cases. It’s just not the place, unless it is explicitly requested by the client, error case tests belongs to unit/functional tests.
An acceptance test output should not get convoluted with tons of specific border-line test cases. The output should be just what is needed for the client to assert that all of his requested functionalities are implemented and that they do work. That is, give the client some insight on whether or not he should accept the software.
- Worked on documentation for simple grade entry (it describes what simple grade entry is, why we need it, implementation details, the current state, and some plans for next semester) (see https://stanley.cdf.toronto.edu/drproject/csc49x/olm_rails/wiki/SimpleGradeEntry)
- Created a screencast which demonstrates how to create and edit grade entry forms (see http://www.youtube.com/watch?v=r7UnaNYe2rw)
- Continued to work on the Table View (ran into some speed bumps which I’ve briefly described in the “Current State” section of my documentation on the wiki)
- Created a blog post related to some performance issues with the Table View (see http://blog.markusproject.org/?p=935)
- Discuss the issues related to the Table View at the meeting tomorrow
- Commit my code since I got my last “Ship It” for my current review requests and then do some ticket clean-up
- None this week
- Added the link so a TA can create or view notes while grading
- Fixed bugs 506 and 505 which are in the branch release
- Did the screen-casts of the notes modal dialog
- As usual time
- Ticket #437 committed
- Created some tickets, feature requests
- Fixed reviews for which I got feedback
- Didn’t get to post my next automated testing blog-post (Hrmph)
- Write a blog post about a feasible scenario which could be implemented next summer regarding automated testing (we need to allow check in of instructor generated (test-) files to student repositories, first)
- Talk with Mike and Karen about must-fix tickets/features for the spring courses round. Think about a schedule until our next deployment round in January.
- Finalized and committed Notes tab code for groupings
- Closed ticket #500 – Grader view: rubrics are nested in the table
- Commit ticket #507 – Groups controller tests missing fixture student7
Wiki documentation on Notes
Make notes on my investigation for ticket #361 (Grader View: shoudn’t unmarked rubric be expanded by default?) since I don’t have time to finish it
- None – just finals left to write!
- Committed Cucumber integration (ticket 460) and created a wiki page about using it.
- Opened a review request to show the tests I made with Machinist and MarkUs.
I’m afraid there are, just give me the week-end.
- Selenium testing added functionalities.
- Add what’s needed to have Selenium used inside Cucumber tests.
- Updated documentation on what’s left to do for the Flexible marking scheme feature.
- Design III, alias: The Robot
- Introduced MarkUs to our colleagues (and shared our experience) during our QA class
- Did nothing else this week except work on a robot for a course but it is now FINISHED !!
- Create tickets for what is left to do on the Flexible criteria feature
- Document the Flexible criteria features state (wiki)
- Look at acceptance tests that could be done.
- Look at the grading view and estimate the time it could take.
- Worked on and finished Ticket #457
- Published 2 screencasts to explain the flexible criterion implemented features.
- Demo for our QA course that presents Cross-Canada, Markus and our implication
- Started the document about the Flexible criteria features state
- Write some Selenium tests
- Publish the document about the Flexible criteria features state
- Study for my final exams
- Reviewed lots of code
- Looked into ReviewBoard large-diff problem: looks like we’ll just have to split into smaller diffs for now
- Now that things have calmed down, have a meeting with Karen to figure out how to prepare MarkUs for next semester
As I’ve been working on the table view for grade entry last week and this week (there are still some kinks to sort out, more on that in my next post), I’ve been noticing some performance issues that I think we might want to deal with before we let instructors use this table. One thing that I needed to decide was whether or not each table cell should be saved right as the instructor enters the mark or if the instructor should enter a bunch of marks and then hit a “Save” button. I decided to go with the “Save” button option. My initial plan was to display the entire table on a single page. However, I’m finding that:
- It’s difficult to navigate
- This page can take a long time to load or save
Severin had suggested earlier that we could implement some sort of pagination. I really like this idea. Here’s what I’m thinking:
- Maybe we could implement pagination alphabetically, by last name (eg. At the bottom of the table, we could have links like “A-D”, “E-J”, etc.). This would help to reduce the number of rows that get displayed on the page at a time. We probably wouldn’t want to implement pagination using numbers because if an instructor wanted to enter grades for a particular student, he/she would have a hard time figuring out which page contains the desired student.
Any thoughts about this?
According to our QA cours requirement, we had to present a demo of the work done during the session. Our demo was split in two parts.
1- PowerPoint presentation. This is the occasion to practice your french 😉
2- Demo online of the application and of some code
The whole presentation went very well, and last around 30 minutes.
The log is right here.
- Meeting starts at 12:01 p.m.
- Everybody is exhausted because of other courses.
- Everybody seems to make good progress.
- The notes system is close to being done.
- Karen wants the notes on grouping to be really solid first, notes on assignments and students are secondary.
- Karen thinks the notes system’s interface is pretty nice.
- Karen wants to be able to create a note from a grader view but not right now.
- Farah is currently working on the table view for the notes system.
- Simon, Mélanie and Gabriel are preparing for a presentation for their QA course.
- Simon had fun adding Selenium tests for the Assignment’s properties’ view.
- Mike added a Selenium test for the login page. For this page CSRF (Cross site request forgery) protection should be turned off.
- Cucumber is waiting for the approval on the review board.
- There is an interesting project for next semester, adapt MarkUs for blind people.
- Html validation is handy to test if the application is usable for blind people.
- Karen wants everyone’s address to send them something.
- Tara got a job for Amazon, congrats !
- Meeting ends at about 12:55 p.m.