Archive for the ‘Developer Essentials’ Category
For this term I have been working on replacing the old image based PDF annotation system with a new one that uses the native file format. The new system allows the students and the markers to view PDF files in a much smoother native fashion. As well as allows markers to annotate the PDF files directly in the viewer. I believe the overall user experience has improved as a result of this change.
This change has been very beneficial for a multitude of reasons:
- It has increased the speed of the assignment collection by completely removing the long process of converting each submitted PDF into a series of images.
- Allows for a smoother workflow when viewing and annotating PDF’s. There is now support for changing the zoom level, and there are page number controls for navigating through the pages.
- It is now resolution independent. It doesn’t matter what resolution the PDF file was saved as, it will always render correctly to fit within the screen nicely. While still allowing the user to zoom in on pages if need be.
- Annotations can be drawn directly in the viewer and thus will change with the zoom level of the document. This is beneficial when small images or specific areas need to be annotated.
The main challenge with working on this component was implementing the annotation system on top of the PDF.js library. The main viewer code was largely undocumented so a fair amount of reverse engineering had to be done in order to understand the rendering process and insert the hooks for annotations. The PDF.js library itself was not modified at all during this process, instead hooks were added into the rendering process so that the annotation system could sit on top of PDF.js without being coupled too heavily to the library. This should make it simpler to upgrade to new PDF.js versions as they are released.
Currently the system does not support downloading PDF files to a user’s computer with the annotations rendered in them. They can only be viewed online. An improvement that could be made in the future would be to render these annotations to the actual PDF file before a user downloads them. [Origami-PDF](https://github.com/mobmewireless/origami-pdf/) is a library that may be able to do this.
Overall I think the system works quite well. The annotation system behaves exactly the same as the image annotations so the user experience will be seamless when transitioning from one system to another. Thus a very small learning curve. And having an actual PDF viewer makes it much simpler to navigate through and markup PDF files.
This term, I have been working on the upgrade from rails 3 to rails 4. The upgrade process consisted of two phases. Firstly, I upgraded MarkUs from rails 3.2 to rails 4.0. Then, I upgraded it from rails 4.0 to rails 4.1. Each phase consisted of some parts that need to be changed by referring to two upgrade guides in Rails 4 Upgrade Notes
Right after each upgrade phase, I ran both unit test and Rspec test. There were many deprecated warnings and errors showing up. In order to make pull requests small, I
ran each test from unit test and Rspec test separately and updated the code.
After all warnings and errors from the tests were fixed, I tested MarkUs manually. There are some issues after the upgrade and need to be fixed.
Also, The following tasks still need to be done.
Since many features that were present in earlier versions of Rails were removed from Rails4 itself and extracted to gems, I have added the following Gems in order to make the upgrade process as smooth as possible.
gem ‘actionpack-action_caching’, ‘~>1.0.0′
gem ‘actionpack-page_caching’, ‘~>1.0.0′
gem ‘actionpack-xml_parser’, ‘~>1.0.0′
gem ‘actionview-encoded_mail_to’, ‘~>1.0.4′
gem ‘activerecord-session_store’, ‘~>0.1.0′
gem ‘rails-observers’, ‘~>0.1.1′
gem ‘rails-perftest’, ‘~>0.0.2’
But not all of them are needed in our application. Those gems that are not used in our application should be removed. For more information about what the gems do, Please refer to upgrading to rails 4 page 71.
Since the final exams are approaching, I didn’t have time to finish the above tasks yet. And I will try to finish them after all finals have completed.
My focus for this term has been testing the Assignment model with RSpec. The major components of this testing included:
- Migrating existing rake tests to RSpec tests,
- Adding additional tests and
- Improving the Assignment model itself (through adding additional validations, removing unused methods and improving existing methods).
I have been creating tests for the model using a unit-method approach. This is what is suggested when testing models. I created a separate describe block to test each Assignment method and only that method. Every method should have an associated describe block upon completion of the Assignment model testing. When I finish testing a method, I delete the associated tests from the rake test suite. Therefore, the old Assignment tests should also all be removed by the time all of the RSpec tests are in place. It is worth noting that there are many methods in the Assignment model without tests for them. One should not rely on simply migrating the tests to ensure good test coverage.
I realized early on that, realistically, there is a good chance no one will inspect the coverage of these tests for a long time after I’ve written them. This is why I was very thorough when testing. I focused on the methods one-at-a-time instead of thinking about the amount of work left to keep myself from just speeding through them. My boiled-down process looks something like:
- Focus on a method that is in the current test suite.
- Inspect the method to see what it does and how it does it.
- Create RSpec tests for that method.
- Ensure all tests in the old test suite have been covered and remove them.
- Commit changes.
I make changes to the method throughout the process if I see anything that could be improved.
I suggest drawing out a diagram of the associations between the models you are working with. Jumping into testing in a new codebase can be a bit daunting and drawing out a rough diagram really helped me understand how the models fit together. In addition, be sure to play around with the application to find where each model matters. It will help create a mental picture of models as you work with them.
I learned RSpec while working on the tests. Here are some useful links I’ve gathered on RSpec testing:
- http://betterspecs.org/ (best practices)
- http://robots.thoughtbot.com/how-we-test-rails-applications (tips and test types)
- https://gist.github.com/kyletcarlson/6234923 (model testing template)
- http://www.methodsandtools.com/tools/rspecbestpractices.php (more tips; realize most tips are for controller testing)
- Bonus: http://www.amazon.ca/Rails-4-Way-3rd/dp/0321944275/ref=sr_1_1?ie=UTF8&qid=1417150279&sr=8-1&keywords=the+rails+4+way (where I started learning about RSpec; a truly phenomenal Rails book)
- Added test cases to Assignments rspec tests
- Updated Main Controller functional test
- Added Marks Spreadsheets quick links to dashboard & colour-coded due dates
- Make some final visual changes in assignment summaries
- looking over React tables to determine which need a custom sort
- custom sort function for grace credits (sort by fractional value remaining/total)
- small refactoring of last week’s sort functions
- addressing comments from @david-yz-liu and @houndci on PR #1906
- making the compare functions more general; moving some of the parsing work back into table.js
- Fixed up some of the changes with the Tags view.
- Moved the Tags view link up to the sub menu. It was previously located in the sub sub menu. Tags are global in scope as opposed to confined to an assignment.
- Got most of the Results view working. Tags can now be assigned and unassigned from particular submissions. This was done using Ajax to prevent page loads every time a tag is assigned.
- Bug fixes?
- Finishing up the Tag feature.
- Refactored Assignment spec:
- grouped certain tests (i.e. association and validation tests)
- replaced `.to be` and `.not_to be` with `.to be true` and `.to be false` when expecting a boolean value
- other small improvements
- RSpec tests for:
- Assignment#section_due_date (and rewrote actual method)
- Assignment#section_past_due_date? (and modified actual method)
- Started putting RSpec blog post together
- Complete blog post and
- You guessed it — more tests!
- Completed drawing/displaying annotations.
- Completed saving annotations.
- Worked on loading existing annotations.
- Complete loading existing annotations.
- Update/Write blog post on the status of annotations and how they work.
- Worked on TODOs in the comment that is related to rails 4
find_all_by_… to where(…),
find_or_create_by_ to find_or_create_by,
find_last_by_ to where(…).last
- Manual tests
- Write a blog post
- I found some errors caused by the new “all()” api today, I will update the code related to it.
- Manual tests and discover more errors
- Fixed Ruby development environment after it was broken from revert merge from master
- Learning how to write tests for class methods, confused about how to create the test data. Worked on rspec tests and updating rake test for new method in Assignments model
- Manual testing on dashboard
- Finish rspec tests and make pull request
- Add spreadsheets summary to dashboard
- Opened PR with changes to associate grouping and tags upon tag creation (in one specific workflow), moved methods to a TagsHelper module to allow access from different controllers, set up a variable in the resultscontroller to display tags not associated to current grouping
- Manual testing to show the association created between tags and groupings work
- reading and research into routes.rb, still very confused!
- continued work on the submissions table and the results view.
- new compare functions for React tables custom sorting
- some time spent doing it the wrong way (passing extra data to the view) and then discussing/clarifying the right way (relevant commit; other work was not pushed to github before I changed it).
- writing up work on sorting for blog
- add custom sorts where needed in other tables (?)
- look into filtering problems
- Added features for the Tag view. These include edit modals, tag delete buttons, create new modal, download yml.
- Worked on the routes file. Created routes to allow for the functions defined above. There are still issues with the routes table and certain things aren’t working properly.
- Added some features to the controller to help assist with CSV upload and download, YML download, tag creation, etc.
- Bug fixes for the tagging features.
- Start on the tagging features on the Submissions view.
- Fixed routes.rb so that routes are defined for the tagging features.
- Continued work on the drawing system for annotations in the PDF view. Originally I was trying to use as much of the existing image annotation system as possible, unfortunately I have discovered that the way they work are too different and I am not creating a specific annotation manager for the PDF’s.
- Finish the drawing, saving, and loading of annotations on the PDF.
- Clean up as much code as possible before the final week.
- Migrated testing of the following methods: Assignment#assigned_groups, Assignment#unassigned_groups and Assignment#add_group
- Updated rspec matchers that checked if two arrays were equal from eq() to match_array() so that order is not considered. This caused intermittent failing when the order of the retrieved records varied with the local array. Order does not matter in the modified cases.
- Improved readability and slightly optimized Assignment#add_group method. Both tests suites still pass with my modifications.
- More RSpec testing/migrations
- Publish blog post: “RSpec tips for Markus”
- Updated and removed some gems
- Upgraded rails from 4.0 to 4.1
- Fix errors caused by the upgrade
- Merge master to rails4_new branch
- Pull Requests:
- Removal of unused Assignment#no_grouping_students_list method.
- Renaming of Assignment#graded_submissions method to ‘graded_submission_results’ (as it returns a set of Results)
- Test migration/improvement of 2 Assignment methods #groups_submitted and #add_csv_group
- Continue migration/improvement of Assignment model tests
- Added strong parameters of assignment model and update helper file
- Updated some test cases according to the new rails API
- Did some manual tests under admin
- Continue working on manual tests
- Merge pull request #1857 (Split view on dashboard)
- Working on the assignment summary view (according to the wireframe)
- Submit pull request for the revamped assignment summary view
- Implement logic to display the most “current” assignment
- Continued work on integrating the annotation system.
- Removed unnecessary files from pdf.js and optimize the viewer loading time
- Tried to use the current annotation system for the pdf’s. I think it is incompatible with the needs for creating annotations on a pdf. I am still trying to use the existing code for selection of areas to draw annotations and then implement a annotation rendering part on top of that.
- Begin working on the server side saving of annotations.
- Worked on the text drawing annotations. Currently I am trying to use interface the image annotation system with the pdf viewer so that the same code can be used in both places.
- Removed unnecessary code in the viewer and added more optimizations.
- Fixed more display and css bugs.
- Continue work on the annotation system.
- Save annotations from the pdf to the database.
test/functional/grade_entry_forms_controller_test.rb (20 errors)
test/functional/groups_controller_test.rb (1 error)
test/functional/marks_graders_controller_test.rb (3 errors)
test/functional/notes_controller_test.rb (18 errors)
test/functional/main_controller_test.rb (12 errors)
test/functional/role_switching_test.rb (2 failures)
test/functional/groups_controller_csv_upload_test.rb (2 failures)
test/functional/submissions_controller_test.rb (1 warning)
test/functional/results_controller_test.rb (4 errors)
test/functional/automated_tests_controller_test.rb (13 tests, 12 assertions, 5 failures, 2 errors, 0 skips)
test/unit/helpers/ensure_config_helper_test.rb (9 tests, 5 assertions, 0 failures, 1 errors, 0 skips)
test/unit/markus_logger_test.rb (20 tests, 20 assertions, 1 failures, 0 errors, 0 skips)
test/unit/submission_collector_test.rb (16 tests, 50 assertions, 1 failures, 0 errors, 0 skips)
spec/models/grade_entry_student_spec.rb (12 examples, 1 failure)
spec/controllers/groups_controller_spec.rb (16 examples, 3 failures)
- Migrated more rspec tests. Assignment rake tests down to 800 lines
- Identified codebase improvements consisting of removing an unused method and renaming a method
- Faced issue creating an rspec test to test Assignment#graded_submissions. I did not realize I needed to set `version_used_submission` for Submission in order for it to be picked up as a valid submission so I kept on receiving zero submissions To solve, I removed FactoryGirl and created all of the necessary records manually, using data from development. When that worked, I started removing attributes and swapping in FactoryGirl for each record to narrow down the source of the issue.
- Create more tests for Assignment methods and remove rake tests that relate the to method
- Remove invalid_override from Markus
- Remove unused Assignment#submissions_by(user)
- Figured out how to add to student repos with svn
- Added subdirectories and previous directory link to repo files table
- Fixed current path/breadcrumbs display (was showing html instead of actual links)
- Removed some unneeded code/files
- Fix sorting in repo files table (probably need a custom sort after all; will most likely refactor table.js to make custom sorting definitions work similar to custom filers, as Lawrence Wu suggests)
- Fix revision selection
- Start on File Manager table conversion
- Some time spent fixing svn ruby bindings on my desktop
- Some time spent looking at how the db is seeded. I don’t really get it.
- Sorting by revised_by and last_modified don’t work. I didn’t notice before modifying the repo since all the files had the same values for those columns.
- Selecting a revision by number or with the date picker don’t appear to update the files table. With some caveman debugging, I can see that @revision_number is changed temporarily, but seems to go back to the most recent revision after the update pdfs script is used. The page says it’s displaying revision N(timestamp of N) but the table data is still still for the most recent revision every time.
- Spent a considerable amount of time figuring out the flow of control in the existing FilterTable; it seemed to bounce around between the view and controller a lot. The new flow models the submissions table more, but I wonder if the old flow had something to do with why the revision selection and sorting aren’t updating appropriately.
- It took some doing to get the file download links to appear, just because I’m not a ruby expert and I wasn’t sure where the links should be formed (it’s in the controller now). Also, I was using the old *_table_row.html.erb for reference and accidentally copy/pasta’d the directory row link info from file manager instead of using the repo browser action.
- Open a pull request with the following changes
- change to tags migration file to capture name and description content
- addition of assignment_tags migration file to capture many to many association
- this change came up looking at Bryan’s UI, may need to sync with him if we actually want this
- when submissions table is loaded, tags for each grouping is being pulled now (not shown in UI yet)
- Add tags functionality is the tag view and in the individual submission view should create a tag now associated with the assignment, not yet associated to the individual submission/grouping
- a few other tag operations in tag_controller I anticipate will support other tagging features.
- created a google doc with Bryan to go sync over what’s needed for each tagging operation
- continue going through the list of features and implement the backend operations needed
- also I want to revisit some of the operations i wrote this week, I think I may need to use params to get the to work the way I’m anticipating they’ll be used. this is a part of ruby im a little hazy about.
- Created a blog post to outline all the new view changes in MarkUs. Looking for feedback from other users.
- Collaborated with Nathan regarding our next steps in connecting the views with the controller.
- Continued to update the views for the Tags. Added in any features that do not require the controller (ie, word counter, moving tags from assigned to not assigned list, etc).
- Finalize the views and start working on linking the models and controller to the views!
- Still trying to implement the functionality for list of assignments to change the assignment details on the right side. Spent 10+ hours on the controller/routing issue but can’t seem to pass the assignment variable. Getting the ‘undefined method `short_identifier’ for nil:NilClass’ error. David is trying to help me but I am still unable to fix the issue at the moment.
- Hopefully move on to implementing the logic to display the ‘current’ assignment.