MarkUs Blog

MarkUs Developers Blog About Their Project

Archive for the ‘Developer Essentials’ Category

Using Gitolite

with one comment

We are in the process of implementing a git backend for MarkUs, and authentication/authorization on the repos is a big question.

It seems obvious that gitolite is the right choice for authorization, and because I’m planning to use it for courses as well, I thought I would write down what I have learned.  I still have to try it out, so I may have some of the details wrong.

The repos will be managed under a separate userid that I’ll call markus-git.  I suspect it should be a separate userid than the one that the application runs under, for safety. Only the markus-git user has file permissions on the repos, and users do not need any shell access.

I should also point out that if students are only accessing MarkUs through the web interface, then none of the rest of this is relevant to them.

Users will have to register a public key with gitolite, and we’ll want to automate this process so that it doesn’t require human intervention.  Then they can make a request like the following.

git clone markus-git@markus.cdf.toronto.edu:c2reidka.git

How does it work?

Gitolite sets things up so that when the ssh command is run, it first forces an authorization check. It uses the public key to create a message that only the private key on the client can interpret. If the client sends back the correct response, then the user will be granted access to the appropriate repos.

The gitolite-admin repo is cloned by the administrators (and will include the markus process). It contains the public keys as well as a configuration file conf/gitolite.conf that contains all of the access control information for the repos.

The nice thing about using the ssh approach is that there is no additional server to worry about.

One of the headaches we will have to resolve is how to handle students who have problems with their key pairs, and need to update their keys.

Written by Karen Reid

August 1st, 2014 at 12:42 pm

Carpentry in React

without comments

React is a JavaScript library for building user interfaces. It’s used heavily in production in both Facebook and Instagram (Facebook created React), and in my opinion is an awesome tool for building front-end interfaces.

The work I’m currently doing on MarkUs involves re-implementing, using React, the sortable, searchable, and filterable tables used for listing users and managing groups, graders, and submissions. This is a prime target because 1) the previous tables used a library that depended on Prototype, which we are trying to move away from, 2) these tables can be really slow, since the html for each row is rendered in the backend individually, and 3) the previous tables became messy enough (_boot.js.erb files and construct_xxx_table_row methods) to warrant a rewrite. Also because 4) React is awesome. So let’s go into how React works.

React, at its core, is a library for creating components: functional, reusable, and composable web views. It also manages to let you do this in an extremely simple fashion.

Each component has two objects: this.props and this.state. Props are generally for holding immutable data, while state holds mutable data. In a sortable table, for example, the titles of the columns would go into props and the currently sorted column and sort direction would go into the state. A component then defines a render method that is functionally pure; that is, given the same props and state, the output will always be the same. The render method also cannot cause any side effects.

Now, when React detects a state change of any kind, it’ll quickly re-render diffs for the new DOM based on the render method. Managing state suddenly got a lot easier.

Here’s an example from MarkUs: StudentsTable, which is the React component that defines the listing of students in the Users view. StudentsTable has two pieces of state: students, which is the array of student data that will be used to fill the table, and selected_students, which is an array of student ids that represent the selected students (duh) for doing actions on.

The render method then defines some column and filter objects (based on an ad hoc protocol; column objects have boolean attributes such as sortable and searchable while filter objects define a test for each object). Then it returns a div (really a React DOM object: React uses a virtual DOM) containing two other components: an ActionBox and a Table.

Here’s some demo code for StudentsTable:

StudentsTable = React.createClass({
  getInitialState: function() {
  // Sets this.state.students to
  // what the backend returns
  jQuery.ajax({
    method: 'GET',
    url: ...
    dataType: ...
    success: ...
    });
  },
  render: function() {
    // this stuff could and maybe should actually be moved to props
    var columns = [
      { id: "user_name", sortable: true, searchable: true },
      { id: "first_name" , sortable: true, searchable: true }
      ...
    ];
    var filters = [
      { name: 'all', func: function(student) { return student; } },
      { name: 'active', func: function(student) { return student.active; } },
      ...
    ];
    return (
      <div>
        <StudentsActionBox selected_students={this.state.selected_students}
                           onChange={this.refresh} />
        <Table data={this.state.students} columns={columns} filters={filters} />
      </div>
    }
});

Now, StudentsTable will pass down selected_students to ActionBox, which will receive it as props. This may be a little confusing since selected_students was in state originally, but it doesn’t matter. ActionBox doesn’t know, and ActionBox doesn’t care. Its render function should account for both state and props, and in fact it helps to think of selected_students as immutable, since what ActionBox really needs to be worrying about are things like the dropdowns and the grace credits input and managing that stuff.

We can see that the data flow is kept one-way: StudentsTable, as the parent of the ActionBox, doesn’t know about the state of the ActionBox, nor does it care. If you really needed to know something, like when StudentsTable should request new data, you can create a hook by passing down the refresh method as a prop and have ActionBox call it whenever it needs to. But in general, it’s best to keep upward data flow as minimal as possible.

The base Table is its own component with its own subcomponents like TableFilter, TableSearch, and TableHeader which may or may not have their own subcomponents. But these parts all work in the same way; they’re just smaller parts. This way, React makes reasoning about a specific level in the view hierarchy a lot simpler since everything is compartmentalized and data flows only one way. Consider implementing the same thing in jQuery or Prototype: it would probably require a crazy amount of reasoning about state (of which the size is huge — blame combinatorial explosion) and DOM manipulation.

In conclusion: React is dope. Its power comes from its means of combination and abstraction (sup Sussman). Check it out here.

Written by Lawrence Wu

June 16th, 2014 at 10:25 am

Rails 4 Strong Parameters

with one comment

A long time ago the developers of Rails wanted a simple API for creating new instances of model objects. What they came up looks something like this:

User.new({ user_name: ‘markus’, last_name: ‘Us’, first_name: ‘Mark’, grace_credits: 5, section_id: 2, type: ‘Student’ })

 

This form of initialization, with a hash, is referred to as mass assignment. Using a hash as input to the constructor made the initialization process very flexible and it was used for seeding the database and creating mock objects for testing.

The problem with having something as nice as mass assignment is that it ended up being used for everything. It became (too) common for controller code to be written that would initialize an object by taking input straight from web forms:

def create
    # Default attributes: role = TA or role = STUDENT
    # params[:user] is a hash of values passed to the controller
    # by the HTML form with the help of ActiveView::Helper::
    @user = Student.new(params[:user])
    if @user.save
      flash[:success] = I18n.t(‘students.create.success’,
                               :user_name => @user.user_name)
      redirect_to :action => ‘index’ # Redirect
    else
      @sections = Section.all(:order => ‘name’)
      flash[:error] = I18n.t(‘students.create.error’)
      render :new
    end
  end

 

Or, at least you hoped the params were coming from a web form, and that the person filling in the web form didn’t add some extra data you weren’t expecting. In reality, a malicious user could construct their own POST request to the server and create a new user that had administrative powers. This actually happened to Github a couple of years ago.

This kind of exploit is a text book example of incomplete mediation. Checking the validity of user input on the client side does little to make the app secure. An attacker has full control of the client and can manipulate input to the server in any conceivable way. User side checking may be valuable for user experience reasons, but the checks must also be performed again on the server where we have control. More information on the principles of security for protecting computer information can be found here.

Shortly after the Github/Rails issue, the Rails core team wanted safe defaults for new projects. At the time, the existing Rails solution for this issue was to declare model attributes as being mass assignable or not mass assignable. In code, you would declare attributes as attr_accessible (whitelist) or attr_protected (blacklist), and these declarations would be used to maintain a whitelist of attributes which could be set using mass assignment. However, whitelisting was not enabled by default, and so in Rails 3.2.8, all new apps included the following configuration:

config.active_record.whitelist_attributes = true

Which caused all models to start with an empty whitelist. Mass assignable attributes had to be explicitly listed with attr_accessible. Safety by default was implemented, but it came at the cost of being able to use mass assignability for legitimate purposes. The Rails security guide from that era includes more details on the whitelist configuration and how to work with its limitations.

The problem with having all models disallow mass assignability was that programmers would have to go to more trouble to seed databases, generate mock objects, etc. Possibly for this reason, MarkUs never turned on default whitelists.

The logic was that the whitelists would stop incomplete mediation attacks from anywhere that models are created. However, these attacks come from user input, which is first handled (on the server side) by controllers. So, performing sanitization checks in the controller would limit the impact of the whitelists to where it mattered.

During Rails 4 development, the core team removed attr_accessible and the entire mechanism for whitelisting attributes in model classes. The new solution, strong_parameters, enforces attribute whitelists on the input params before passing the data along to the model constructor. Since strong_parameters does not directly affect model code, it is once again easy to use mass assignment for internal machinery and trusted scripts.

MarkUs predates Rails 3.2.8, and the whitelist default was never turned on during the upgrade to Rails 3.2. MarkUs does use attr_accessible in a couple of places, and avoids using mass assignment for input coming from students. However, the example controller code above (that demonstrates the incomplete mediation flaw) comes straight from the MarkUs code base as it exists today. The example is actually fairly safe because it requires an authorized admin to make the POST in the first place, but nonetheless demonstrates unsafe practices.

I think that current strategy for handling student input entirely without mass assignment is the best strategy from a security perspective. This would also make the upgrade to strong_parameters painless.

It is not recommended to switch to strong_parameters and perform the Rails 4 upgrade at the same time. However, strong_parameters is available for Rails 3.2 as a back port and we can migrate to it before the upgrade to Rails 4. I think it would be best to switch to strong_parameters right away and move away from using mass assignment when the input comes from the client side.

A blog post demonstrating how strong_parameters works can can be found here. The API documentation for strong_parameters can be found here.

Written by Mark Rada

June 11th, 2014 at 11:53 am

Status Report for May 1st

without comments

Alex:

This week: Worked on final paper for facebook. Started working on our presentation.
Next week: Last week of our  semester and we will focus on our presentation.

Christopher:

This Week: Fix Bug# 1412, preparing for presentation, write final report for the class and Facebook
Next Week: Our semester ends at Wednesday so we will prepare for the presentation.

Ope:

This week: Course related material
  • Writing Final Reports
  •  Preparing for Presentation
Next week:
  • Update Help System text (if provided by Karen)
  • Push Help System changes to Upstream (Need to determine best approach)
  • Continue with Course related material
Note: Final day of classes is next Wednesday.

Rafael:

This week:  Fixed the compatibility bug between Rugged and Ruby by upgrading Ruby from 1.9.2 to 2.0. Continue to study the svn code in order to implement the git version of it.

Next week: Find out what Tiago and Zach are working on and help them with it.

 

Written by Rafael Soares Padilha

May 6th, 2014 at 9:26 am

Status Report for April 24th

without comments

Cornell Team:

Next week:

Look at other possible bugs we could fix in 1 week. Prepare for our final presentation for the class.

Alex:

Completed the wiki page for the help system, which describes the details about how to add new help boxes and embedded documentation to new pages.

Christopher:

We worked on issue #1412. We still need to figure out whether the link is what we want for every users (admin, TA and students).

Ope:

Worked on issue #1412. Fixed the url generated for links in the assignment drop down menu. The links are now directing the user to the appropriate page. Will create pull request once testing is completed. Worked with Alex on the Wiki page.

Rafael:

Working with Zach and Tiago on the git back-end. I got stuck on a bug with Rugged and ruby… maybe I’ll need to reinstal it and update my ruby to a newer version. Going to work on that during the weekend.

Zach

Upgraded ruby from 1.9 to 2.0. Created commit for time-based functions (which are still under consideration for their implementation).

 

Written by Alexander Kittelberger

April 28th, 2014 at 7:58 am

Status Report for April 17th

without comments

Cornell Team:

Next week:

We’ll start working on the wiki page for Help System and start working on issue # 1412

Alex:

Continued working on the help system with Chris and Ope. Finished up the majority of the pages with placeholder text.

Christopher:

We finish the help system structure and textboxes. We will send the screenshot by the meeting on Friday.

Ope:

Worked on finishing the help system structure. Worked on screenshots to be provided to Karen (Target: Friday).

 

Zach:
This Week: Rafael, Tiago, and myself started an active discussion on the git backend. I implemented some of the time-based functions for review.

Roadblocks: Ruby. I have had to upgrade my ruby version as 1.9.2 was causing error sometimes.

Next Week: Get ready for my UCOSP presentation and resolve the ruby issues

Rafael:

Started working with Zach and Tiago on the gitbackend. Installed everything and now  studying the svn code and the rugged and gitolite libs.

 

Written by Alexander Kittelberger

April 19th, 2014 at 4:51 pm

Status Report for April 10

without comments

Zach Munro-Cape

This Week: Started staging my commit for a clean pull request

Road Blocks: git specific commands to untangle my current repo
Next Week: Meet up with Rafael and Tiago to discuss our approach on the git backend

Jeremy Winter

-Finished issue #1402

-Finished tagging system specs

Cornell Team:

Current Week:

- work on the help system on several pages and style. We decide that we will use 2 style of help text box depending on the pages.  Finished 5 pages.

Next Week:

- continue working on the help system on other pages.

Alex

-continued work on the help system

-implemented the ability to add both tooltips and larger big boxes for embedded documentation

Christopher:

- worked on help-system

- change the style of the help textbox and add 2 styles of boxes (large box on the right and tooltip boxes)

- add another style of help icon for title.

Ope

- Worked with Alex and Chris on new designs for the Help system

- Fixed a minor bug on the help system implementation on the Assignment page that caused the help messages to be rendered incorrectly on other pages using the same form, besides ‘Add Assignment’. Help system implementation now moved to form level for this content.

Tiago 

This Week: Started studying gitbackend code, solved issues 1324 and 1388

Road Blocks: The integration between gitolite gem and rugged gem.
Next Week: I’m still studying gitolite server and rugged and how to change svn code to make it work
Also meet up with Rafael and Zach to discuss our approach on the git backend. I’m wrinting a document to help this implementation, but for the moment is in portuguese and local, as long as I make it work, I’ll update the blog

 

Written by Christopher Jonathan

April 11th, 2014 at 3:04 pm

Help System Update

without comments

Here are some of the latest screen shots from the help system. We have implemented both tool tip boxes that will show up next to specific fields and also larger boxes that can be displayed below a title or in open space on the page. We have also changed the color of the box. We are going through the individual pages and deciding where tool tips and larger boxes are needed. Any feedback would be great, also if you have a suggestion where embedded documentation is needed and whether a large box with a summary of how something works or a short tool tip is better, let us know.

Add Assignment

This page is using the larger box on the right hand side. This box will enlarged based on the amount of text it contains.

Test Framework

This page is using both larger box and tool tip boxes.

Written by Christopher Jonathan

April 9th, 2014 at 2:37 pm

Status Reports for April 3rd

without comments

Rafael S. Padilha:

 This Week:

I started to search and read about the git backend.

Next Weeks:

I’ll continue to study/understand the git backend and I’ll begin to work with Zach and Tiago on it.

Zach Munro-Cape:

 This Week:

Was able to implement a walker such that it can find commits based on time as well as files based on time.

Roadblocks:

Integrating it such that it can be tested from a test file.

Next Week:

Make up for the week I missed due to final assignments.

Shenlong:

This Week:

About finished functional spec for tagging.

Next Week:

Wrap up and finish spec.

Gillian & David:

This Week:

Worked on issue 1404. Looked into how rails includes javascript files. We think the issue may be related to the path of files and the revision number in which they were committed. The program is calling on certain javascript methods but we’re not seeing them actually being run when the bug occurs.

Next Week:

Try to finish figuring out exactly what is going on in issue 1404 and make a fix for it.

Roadblock:

Needing to ramp up on our knowledge of rails server structure and how Assets/Javascript files get included in order to be able to rule out possible reasons for issue 1404.

Jeremy:

This week:

Continued work on #1402. Worked with Andrew and Sheng on finishing the functional specs for the tagging system.

Roadblocks:

None.

Next week:

Put the final touches on for the functional specs with Andrew and Sheng before our semester ends.

Tiago

This week:

   Finish some pull requests unfinisheds

 Roadblocks:

  I updated my svn but shared libraries were not updated. I had to revert the instalation of svn.

Next week:

  Study gitbackend

 

Written by Gillian Chesnais

April 6th, 2014 at 6:50 pm

Status Reports for Mar 27

without comments

Jeremy

This week: Started working on #1379 and found a bug and created issue #1402. Began working on this issue as well. Met with Andrew and Sheng and began writing specs for the tagging system

Roadblocks: Everything was cleared up in the meeting, no current roadblocks

Next week: Meet with Sheng and Andrew again to finish up the specs and post them to the blog. If time permits (exams are close), continue work on the issues above.

Gillian

This week: 

- Fixed issue #1340, escaping characters that get interpreted in HTML when viewing the content of student submissions on the browser.

- Logged issue #1404,

Roadblocks:

- In Master, the Submissions File Manager displays an empty list when there are submitted files (issue #1404), so I spent some time trying to get around that and doublechecking that it was not an error on my part before going to 1.0.x where it did work.

Next week:

- Will work on fixing issue #1404

Shenglong

This week: Discussed implementation for the tagging system further and started writing a functional specification document to use for implementation.

Next week: Continue working on document and work out design issues that arise

Kitiya

This Week: Create mock up ui to display a view of all of an individual student’s work.

Roadblocks: It took quite a lot of time finding files/objects/methods relevant to a page.

Next Week: Crate a stedent’s work view

Zach

This Week: Resolved previous confusion regarding open and close methods for Repositories and the git implementation (no real change in the implementation) as well as figured out the time-based methods that were causing issue for the previous term students

Roadblocks: Last week(s) of school are causing a fair amount of other-course-work that has detracted not only from available time but also quality of time spent.

Rafael

Next week: Catch up on this week’s work before the official “end of term”

This summer: Continue working with Rafael and Tiago on the git backend

David:

This week:

-This report

-Fixed lag and animation bugs (issue #1405)

Roadblocks:

-I have a lot of assignments and tests from other courses, so it is hard to find enough time to work

Cornell Team:

Next week:

Cornell is on Spring break next week, so we will not be working on MarkUs and will continue the week of April 7th.

Current Week:

Ope:

- Worked on the help system: Updated the javascript to recognize help text within the same section. This now allows the help text for fields in an entire section to be toggled when the user clicks on a particular help icon. Completed help functionality using this approach for a couple sections in the Assignment form, displaying placeholder text created in the en.yml file.

Roadblocks:

- Had significant trouble running MarkUs after the latest pull. Had to manually install some gems, modify my gem file and migrate the database changes.

- It may be helpful for people to communicate any major changes and potential impacts to others

Alex:

- Worked on the help system by creating a new JavaScript file which the help system will use. It now has the ability to click on the icon and display many or just one  help text box on the screen.

Roadblocks:

- Had a little trouble with pulling the latest version of Markus, but was able to fix it by re creating the database.

- Trouble understanding prototype and how jquery is integrated into Ruby on Rails but was able to get it working.

Christopher:

- Try to run the newly pulled Markus. (Still failed).

- Reinstall Ruby Gem and still failed.

- Help working on the help system together on Ope’s or Alex’s computer.

- Finally able to run Markus after reinstall everything.

Roadblocks:

- Unable to run MarkUs after pull the latest version from GIT. After pulling, we need to do bundle update / bundle install that always fail with an exception.

 

 

Written by David Das

March 30th, 2014 at 9:36 am