MarkUs Blog

MarkUs Developers Blog About Their Project

Tagging Backend

without comments

This blog post will discuss the decisions made surrounding the backend of the Tagging Project.

Database Schema

As this is the initial implementation of the Tagging feature, we strove to keep the schema changes as simple as possible.  MarkUs uses ActiveRecord as the M in MVC, and thus we had to create ActiveRecord migration files to implement changes.

We decided to create a Tags Model which contained the following explicitly declared pieces of information:

  1. name (string)
  2. description (string)
  3. user (string)

and an integer id that is implicitly declared.  Furthermore, we created a groupings_tags association to create a many to many association between groupings and tags.

That migration file explicitly associates and a tag id with a grouping id and adds a index of the two ids that must be unique.

An interesting thing with ActiveRecord migration files, and increased its ease of use, is that each migration file must specify the action being done(creating table, adding an attribute to a table etc) but also the action that must be done to rollback these changes.

In this case, the rollback operations for each files solely consisted of dropping each table(and removing the index for the grouping_tags association).  This made it quite easy to create and edit the schema in subsequent iterations.  Editing the schema file would consist of the rollback command, editing schema file, and re-migrating.

Models

In editing the model files, we only had to create a tag.rb and modify the grouping.rb to reflect the new schema.

Controllers

In creating the tag controllers, we had to split up the logic between a tags_controller.rb and a tags_helper.rb module.

The interesting between the tags_helper module and the tags_controller is that other controllers can include the module and directly call its method.  This is not the case for tags_controller; routes.rb needs to consulted to call its methods.

The methods in the tags_helper modules are used by controllers (e.g. tags, results) whereas methods in tags_controller are called from views.

For example, methods in the controller include index, edit, create, destroy.  The module has methods that do work with groupings and tags, which are needed by the results_controller.

Written by Nathan Chow

December 15th, 2014 at 1:31 am

Status Report – December 5

without comments

Chris

This Week:

  • Completed the PDF annotation editor/viewer (i.e. view, load, save, delete)

Next Week

  • Fix an issues from the last pull request.

 

Jakub Subczynski

Final Week

  • renamed methods throughout codebase:
    • Assignment#past_due_date? to #past_all_due_dates?
    • Assignment#what_past_due_date to #section_names_past_due_date
    • Assignment#section_past_due_date?(grouping) to #grouping_past_due_date?(grouping)
  • created Issue #1926 regarding the intended functionality of Assignment#past_all_due_dates? when there the assignment has sections
  • created RSpec tests for:
    • Assignment#grade_distribution_as_percentage
    • Assignment#get_detailed_csv_report

 

Yusi

This week:

  • Updated code about Relation#all
  • Reported some issues

Next time:
Fix

  • https://github.com/MarkUsProject/Markus/issues/1924
  • https://github.com/MarkUsProject/Markus/issues/1923
  • https://github.com/MarkUsProject/Markus/issues/1922
  • https://github.com/MarkUsProject/Markus/issues/1920

Nathan

Last Week:

  • have a code fix updating the submissions table to display real associated tags
  • final exams and end of term projects have kept me pretty busy

Next Week

  • Will be done finals Thursday of next week
  • will work on implementing RSpec tests, any remaining features for tagging and writing a blog post

Written by Chris Kellendonk

December 5th, 2014 at 3:52 pm

PDFJS Annotations II

without comments

For this term I have been working on replacing the old image based PDF annotation system with a new one that uses the native file format. The new system allows the students and the markers to view PDF files in a much smoother native fashion. As well as allows markers to annotate the PDF files directly in the viewer. I believe the overall user experience has improved as a result of this change.

In order to implement this feature the [PDF.js](https://github.com/mozilla/pdf.js/) library by Mozilla was used. This library is designed to render PDF files in the browser using only Javascript and HTML5. But was not designed for adding annotations to a PDF file.

This change has been very beneficial for a multitude of reasons:

  1. It has increased the speed of the assignment collection by completely removing the long process of converting each submitted PDF into a series of images.
  2. Allows for a smoother workflow when viewing and annotating PDF’s. There is now support for changing the zoom level, and there are page number controls for navigating through the pages.
  3. It is now resolution independent. It doesn’t matter what resolution the PDF file was saved as, it will always render correctly to fit within the screen nicely. While still allowing the user to zoom in on pages if need be.
  4. Annotations can be drawn directly in the viewer and thus will change with the zoom level of the document. This is beneficial when small images or specific areas need to be annotated.

The main challenge with working on this component was implementing the annotation system on top of the PDF.js library. The main viewer code was largely undocumented so a fair amount of reverse engineering had to be done in order to understand the rendering process and insert the hooks for annotations. The PDF.js library itself was not modified at all during this process, instead hooks were added into the rendering process so that the annotation system could sit on top of PDF.js without being coupled too heavily to the library. This should make it simpler to upgrade to new PDF.js versions as they are released.

Currently the system does not support downloading PDF files to a user’s computer with the annotations rendered in them. They can only be viewed online. An improvement that could be made in the future would be to render these annotations to the actual PDF file before a user downloads them. [Origami-PDF](https://github.com/mobmewireless/origami-pdf/) is a library that may be able to do this.

Overall I think the system works quite well. The annotation system behaves exactly the same as the image annotations so the user experience will be seamless when transitioning from one system to another. Thus a very small learning curve. And having an actual PDF viewer makes it much simpler to navigate through and markup PDF files.

Written by Chris Kellendonk

December 5th, 2014 at 3:26 am

Rails 4 Upgrade Notes II

without comments

This term, I have been working on the upgrade from rails 3 to rails 4. The upgrade process consisted of two phases. Firstly, I upgraded MarkUs from rails 3.2 to rails 4.0. Then, I upgraded it from rails 4.0 to rails 4.1. Each phase consisted of some parts that need to be changed by referring to two upgrade guides in Rails 4 Upgrade Notes

Right after each upgrade phase, I ran both unit test and Rspec test. There were many deprecated warnings and errors showing up. In order to make pull requests small, I
ran each test from unit test and Rspec test separately and updated the code.

After all warnings and errors from the tests were fixed, I tested MarkUs manually. There are some issues after the upgrade and need to be fixed.

https://github.com/MarkUsProject/Markus/issues/1924
https://github.com/MarkUsProject/Markus/issues/1923
https://github.com/MarkUsProject/Markus/issues/1922
https://github.com/MarkUsProject/Markus/issues/1920

Also, The following tasks still need to be done.

Extracted Gems
Since many features that were present in earlier versions of Rails were removed from Rails4 itself and extracted to gems, I have added the following Gems in order to make the upgrade process as smooth as possible.

gem ‘actionpack-action_caching’, ‘~>1.0.0′
gem ‘actionpack-page_caching’, ‘~>1.0.0′
gem ‘actionpack-xml_parser’, ‘~>1.0.0′
gem ‘actionview-encoded_mail_to’, ‘~>1.0.4′
gem ‘activerecord-session_store’, ‘~>0.1.0′
gem ‘rails-observers’, ‘~>0.1.1′
gem ‘rails-perftest’, ‘~>0.0.2’

But not all of them are needed in our application. Those gems that are not used in our application should be removed. For more information about what the gems do, Please refer to upgrading to rails 4 page 71.

Since the final exams are approaching, I didn’t have time to finish the above tasks yet. And I will try to finish them after all finals have completed.

Written by Yusi Fan

December 4th, 2014 at 5:49 pm

RSpec testing in MarkUs

without comments

My focus for this term has been testing the Assignment model with RSpec. The major components of this testing included:

  1. Migrating existing rake tests to RSpec tests,
  2. Adding additional tests and
  3. Improving the Assignment model itself (through adding additional validations, removing unused methods and improving existing methods).

I have been creating tests for the model using a unit-method approach. This is what is suggested when testing models. I created a separate describe block to test each Assignment method and only that method. Every method should have an associated describe block upon completion of the Assignment model testing. When I finish testing a method, I delete the associated tests from the rake test suite. Therefore, the old Assignment tests should also all be removed by the time all of the RSpec tests are in place. It is worth noting that there are many methods in the Assignment model without tests for them. One should not rely on simply migrating the tests to ensure good test coverage.

I realized early on that, realistically, there is a good chance no one will inspect the coverage of these tests for a long time after I’ve written them. This is why I was very thorough when testing. I focused on the methods one-at-a-time instead of thinking about the amount of work left to keep myself from just speeding through them. My boiled-down process looks something like:

  1. Focus on a method that is in the current test suite.
  2. Inspect the method to see what it does and how it does it.
  3. Create RSpec tests for that method.
  4. Ensure all tests in the old test suite have been covered and remove them.
  5. Commit changes.

I make changes to the method throughout the process if I see anything that could be improved.

I suggest drawing out a diagram of the associations between the models you are working with. Jumping into testing in a new codebase can be a bit daunting and drawing out a rough diagram really helped me understand how the models fit together. In addition, be sure to play around with the application to find where each model matters. It will help create a mental picture of models as you work with them.

I learned RSpec while working on the tests. Here are some useful links I’ve gathered on RSpec testing:

Written by Jakub Subczynski

December 4th, 2014 at 1:05 am

Status Report – December 2

without comments

Irene

Last Week:

  • Added test cases to Assignments rspec tests
  • Updated Main Controller functional test
  • Added Marks Spreadsheets quick links to dashboard & colour-coded due dates

This Week:

  • Make some final visual changes in assignment summaries

Tori

Last Week:

  • looking over React tables to determine which need a custom sort
  • custom sort function for grace credits (sort by fractional value remaining/total)
  • small refactoring of last week’s sort functions
  • addressing comments from @david-yz-liu and @houndci on PR #1906

This Week:

  • making the compare functions more general; moving some of the parsing work back into table.js

Bryan

Last Week:

  • Fixed up some of the changes with the Tags view.
  • Moved the Tags view link up to the sub menu. It was previously located in the sub sub menu. Tags are global in scope as opposed to confined to an assignment.
  • Got most of the Results view working. Tags can now be assigned and unassigned from particular submissions. This was done using Ajax to prevent page loads every time a tag is assigned.

This Week:

  • Bug fixes?
  • Finishing up the Tag feature.

Written by Irene Fung

December 2nd, 2014 at 9:50 pm

Status Report – November 28

without comments

Jakub Subczynski

This Week

  • Refactored Assignment spec:
    • grouped certain tests (i.e. association and validation tests)
    • replaced `.to be` and `.not_to be` with `.to be true` and `.to be false` when expecting a boolean value
    • other small improvements
  • RSpec tests for:
    • Assignment#section_due_date (and rewrote actual method)
    • Assignment#past_due_date?
    • Assignment#latest_due_date
    • Assignment#section_past_due_date? (and modified actual method)
    • Assignment#what_past_due_date
  • Started putting RSpec blog post together

Next Week

  • Complete blog post and
  • You guessed it — more tests!

 

Chris

This Week

  • Completed drawing/displaying annotations.
  • Completed saving annotations.
  • Worked on loading existing annotations.

Next Week

  • Complete loading existing annotations.
  • Update/Write blog post on the status of annotations and how they work.

 

Yusi

This Week

  • Worked on TODOs in the comment that is related to rails 4
  • Change:
    find_all_by_… to where(…),
    find_or_create_by_ to find_or_create_by,
    find_last_by_ to where(…).last
  • Manual tests

Next Week

  • Write a blog post
  • I found some errors caused by the new “all()” api today, I will update the code related to it.
  • Manual tests and discover more errors

Written by Jakub Subczynski

November 28th, 2014 at 2:04 pm

Custom Sort Functions for React Tables

without comments

Recently, we discovered that some columns in the new React.js Tables were not sorting the rows as expected. In particular, the GROUP NAME and REPOSITORY columns in the Submissions and Summaries views exhibited consistent but very strange behaviour, as shown in the figures below.

sort_by_group_name

sort_by_repo_name

(The COMMIT DATE column was also sorting incorrectly, but this issue was less mysterious in that it was obviously performing string comparison on the dates. In order to correctly parse the date strings into a numerically comparable value, a customized sorting method was definitely required.) I was assigned the task of refactoring the React Table class (and related classes) in order to allow customized sorting methods for columns that required special treatment.

As it was, sorting was handled by sort_by_column(). The cell values we converted to lowercase and used the dangerouslySetInnerHTML.__html property for React Components. Values of types other than string were not interpreted.

function sort_by_column(data, column, direction) {
  function makeSortable(a)
  {
    if (typeof a == 'string') {
      return a.toLowerCase().replace(' ', '');
    } else if (a.hasOwnProperty('props')) {
      // Is a react component, get innerHTML
      return a.props.dangerouslySetInnerHTML.__html.toLowerCase();
    } else {
      return a;
    }
  }
  // sorts column id
  var r = data.sort(function(a, b) {
    if (makeSortable(a[column]) > makeSortable(b[column])) {
      return 1;
    } else if (makeSortable(b[column]) > makeSortable(a[column])) {
      return -1;
    }
    return 0;
  });
  // flips order if direction descending.
  if (direction == 'desc') r.reverse();

  return r;
}

 

table.js ]

In some table columns, we use a React <span> components with dangerouslySetInnerHTML.__html set in order to properly render html contents such as links and icons in a cell. (Though not shown above, the cells in the COMMIT DATE column will contain an icon if the submission commit date is past due.)

Of course, when we attempt to sort by dangerouslySetInnerHTML.__html, we are comparing strings of html code rather than the text displayed in the table cells. Naturally, the unexpected behaviours exhibited by the GROUP_NAME and REPOSITORY columns were the result of string comparison between the hrefs for the anchor elements, rather than the hyperlink text. Since both links differed by group number, we were effectively performing an alphanumeric sort on the unformatted group numbers.

For example, group_0007 would appear below group_0011 on an ascending sort because we were comparing “11″ and “7″ rather than the repository names. In the first figure, the group names did not appear to follow any kind of logical order because the display text is completely unrelated to group number.

My solution was to leave the existing function makeComparable() intact, for  primary processing of sortable data, while optionally allowing a custom compare function to be passed in as needed for any particular column.

function sort_by_column(data, column, direction, compare) {
  //...
  // determine sort behaviour
  compare = compare || compare_values;

  // sort row by column id
  var sorted = data.sort(function(a, b) {
    return compare(makeComparable(a[column]), makeComparable(b[column]));
  });

  // flip order if direction descending
  if (direction == 'desc') {
      sorted.reverse();
  }

  return sorted;
}

table.js ]

The compare function to use when the table rows are rendered is stored in the Table state variable sort_compare. When the selected sort column changes sort_compare is updated to the compare member of the appropriate column.

// Header col was clicked. Adjust state accordingly.
synchronizeHeaderColumn: function(sort_column, sort_direction) {
  var compare_func = this.props.columns.filter(function(col) {
      return col.id == sort_column;
  })[0].compare;
  this.setState({
    sort_column: sort_column,
    sort_direction: sort_direction,
    sort_compare: compare_func
  });
}

table.js ]

Since we have the default comparison function compare_values, we are able to leave compare undefined for any given column.

function compare_values(a, b) {
  if (!b || a > b) {
    return 1;
  } else if (!a || a < b) {
    return -1;
  }
  return 0;
}

table_sorts.js ]

Now we define functions such as compare_anchor_text() to handle special cases, and assign the custom sort functions to column objects. In the code below, notice that the SECTION column does not define a compare member since the data is just a string and so the default compare_values will suffice.

function compare_anchor_text(a, b) {
  function parse_anchor(a) {
    var open_tag_end = a.indexOf('>', a.indexOf('<a'));
    var close_tag_start = a.indexOf('</a', open_tag_end + 1);
    if (open_tag_end !== -1 && close_tag_start !== -1) {
      return a.substring(open_tag_end + 1, close_tag_start);
    }
    return a;
  }

  return compare_values(parse_anchor(a), parse_anchor(b));
}

table_sorts.js ]

var SubmissionsTable = React.createClass({
  getDefaultProps: function() {
    // Defines the columns used for the table and whether they
    // are sortable searchable. The default initially sorted
    // column is the first sortable column in the array.
    return {
      columns: [
        {
          id: 'group_name',
          content: '<%= j raw I18n.t(:'browse_submissions.group_name') %>',
          sortable: true,
          compare: compare_anchor_text,
          searchable: true
        },
        {
          id: 'repository',
          content: '<%= j raw I18n.t(:'browse_submissions.repository') %>',
          sortable: true,
          compare: compare_anchor_text,
          searchable: true
        },
        {
          id: 'section',
          content: '<%= j raw I18n.t(:'browse_submissions.section') %>',
          sortable: true,
          searchable: true
        },
//...

_submissions_table.js.jsx.erb ]

Written by Victoria Verlysdonk

November 25th, 2014 at 10:51 pm

Posted in Uncategorized

Status Report – November 25

without comments

Irene

Last Week

  • Fixed Ruby development environment after it was broken from revert merge from master
  • Learning how to write tests for class methods, confused about how to create the test data. Worked on rspec tests and updating rake test for new method in Assignments model
  • Manual testing on dashboard

This Week

  • Finish rspec tests and make pull request
  • Add spreadsheets summary to dashboard

Nathan

Last Week

  • Opened PR with changes to associate grouping and tags upon tag creation (in one specific workflow), moved methods to a TagsHelper module to allow access from different controllers, set up a variable in the resultscontroller to display tags not associated to current grouping
  • Manual testing to show the association created between tags and groupings work
  • reading and research into routes.rb, still very confused!

This Week

  • continued work on the submissions table and the results view.

 

Tori

Last Week

  • new compare functions for React tables custom sorting
    • some time spent doing it the wrong way (passing extra data to the view) and then discussing/clarifying the right way (relevant commit; other work was not pushed to github before I changed it).
  • writing up work on sorting for blog

This Week

  • add custom sorts where needed in other tables (?)
  • look into filtering problems

 

Bryan

Last Week

  • Added features for the Tag view. These include edit modals, tag delete buttons, create new modal, download yml.
  • Worked on the routes file. Created routes to allow for the functions defined above. There are still issues with the routes table and certain things aren’t working properly.
  • Added some features to the controller to help assist with CSV upload and download, YML download, tag creation, etc.

Next Week

  • Bug fixes for the tagging features.
  • Start on the tagging features on the Submissions view.
  • Fixed routes.rb so that routes are defined for the tagging features.

Written by Victoria Verlysdonk

November 25th, 2014 at 1:14 pm

Status Report – November 21

without comments

Chris

This Week:

  • Continued work on the drawing system for annotations in the PDF view. Originally I was trying to use as much of the existing image annotation system as possible, unfortunately I have discovered that the way they work are too different and I am not creating a specific annotation manager for the PDF’s.
  • Read through a ton of Javascript and HTML to decipher how the current annotation drawings are done on images.

Next Week:

  • Finish the drawing, saving, and loading of annotations on the PDF.
  • Clean up as much code as possible before the final week.

 

Jakub Subczynski

This Week

  • Migrated testing of the following methods: Assignment#assigned_groups, Assignment#unassigned_groups and Assignment#add_group
  • Updated rspec matchers that checked if two arrays were equal from eq() to match_array() so that order is not considered. This caused intermittent failing when the order of the retrieved records varied with the local array. Order does not matter in the modified cases.
  • Improved readability and slightly optimized Assignment#add_group method. Both tests suites still pass with my modifications.

Next Week

  • More RSpec testing/migrations
  • Publish blog post: “RSpec tips for Markus”

Yusi

This week:

  • Updated and removed some gems
  • Upgraded rails from 4.0 to 4.1
  • Fix errors caused by the upgrade

Next week:

  • Merge master to rails4_new branch

Written by Chris Kellendonk

November 21st, 2014 at 3:58 pm