MarkUs Blog

MarkUs Developers Blog About Their Project

Archive for September, 2009

Code Sprinting with MarkUs – Day 3

with 3 comments

The MarkUs team had a meeting yesterday to discuss our tasks for the rest of the semester. As promised, here is a description of the exciting new features that we are planning on adding to MarkUs:

  • Simple Grade Entry
    • Used for entering grades from labs, exams, etc.
    • This will be another type of “assignment” that the instructor can create
    • Considerations and open questions:
      • Permissions (eg. TAs need to be able to modify the table, students are not allowed to see the whole table, etc.)
      • Attaching graders
      • Conditions for releasing marks
      • What’s the workflow for creating the grade entry form? What’s the UI for entering grades? What’s the UI for students? Model?
  • Messages
    • Allow messages about a student’s results (eg. notes about remark requests, concerns, etc.) to be added in the grader view
    • This will be a tab in the grader view
    • Messages are visible to instructors and TAs only
    • Timestamp and mark the user name for annotations
    • Considerations:
      • Ways to aggregate messages for an assignment
      • Ways to download and/or view aggregate
      • Messages per student
      • Viewing messages – notify the instructor / TA about messages since the last time they logged in
  • New Marking Scheme
    • Allow the instructor to specify a description of a criterion and a total number of marks for that criterion
    • Considerations:
      • Ways to create title, description, and weight
      • Ways to upload/download description
      • Format for description
      • Allowing penalties, deductions, bonus marks, etc.
      • UI – the TA will need to enter a number into a box for each criterion
  • Logging
    • Want to find out how students and TAs are using MarkUs
    • Students:
      • Number of submissions and when (can get this information from svn)
      • Who is submitting? (eg. Is one person doing all of the work in a group project?)
      • Who is looking up their grades?
      • Information about groups (eg. accepts, rejects)
    • TAs
      • Number of annotations
      • Tracking setting grades
      • Time spent marking
    • Need to create a log file first and then determine how to visualize the data
    • Considerations and open questions:
      • The size of the log files
      • How should we store this information? DB versus log file?
      • Rotating logs?
      • How many messages might there be?
  • Automated Testing
    • Allow students to run tests against their code
    • Can be used to run style checks, code coverage tests, etc.
    • Considerations:
      • Security
      • UI for the student and instructor
      • Where/how will the tests be run?

After discussing these new features, we split up the tasks based on our interests. Here’s the breakdown:

  • Tara and Fernando will be working together on the messages. Fernando will also do some work on logging first.
  • Melanie, Simon, and Gabriel will work on the new marking scheme. They will be doing test-driven development.
  • Farah will work on the simple grade entry.
  • Severin is going to get started on automated testing.
  • Everyone will be developing tests for their code.

We’re all set for a productive term! Let the fun begin!

Written by Farah Juma

September 28th, 2009 at 10:26 am

Posted in Uncategorized

Codesprinting on MarkUs: Day 2 Blurb

with one comment

Here’s a quick snapshot of what the team did yesterday:

  • Lots of coding!  Tickets were assigned, ReviewBoard was cookin’ with lots of activity
  • A bunch of code from the new team got past review, and went straight into the repo!  Excellent!
  • A unexpectedly long lunch break
  • All of the teams celebrated the end of the day by heading out to a Chinese restaurant.  So much food!

Today is the last day of the sprint.  We’re having a meeting later today to figure out the direction for the rest of term.  We’ll keep you posted!

Written by m_conley

September 27th, 2009 at 9:17 am

Posted in Uncategorized

Codesprinting on MarkUs: Day 1 Blurb

with one comment

I believe the team is planning on doing a large writeup on the Codesprint after the weekend, but I thought I’d wet your whistle with a little preview:

  • All team members arrived in Toronto safely!
  • Everyone was able to get online quickly.  Team began working furiously.
  • Free pizza lunch == amazing
  • Adam Goucher joined us, and gave us some excellent industry-level advice on how to test MarkUs
  • The MarkUs ReviewBoard is getting busy!  All code (even the small snippets) is getting reviewed before being committed to the repo.
  • After a solid day of work, the team eventually headed to a pub for free food and drink

So that’s just a taste.  Stay tuned!

Written by Markus

September 26th, 2009 at 9:32 am

Posted in Uncategorized

IRC Meeting Notes – September 18, 2009

with one comment

Today’s Agenda

Today’s Status Reports

IRC Transcript – starting around 11:54 am and going through to the 3rd page.

Mike Conley (m_conley) ran the meeting today, with Tara (tlclark) taking notes.

  1. Development environments
    • We all have our development environments and MarkUs set up, across all 3 platforms. Yay!
    • Karen and Mike thank us for working hard to update the install documentation.
  2. Code sprint travel arrangements
    • Tara is driving up, all settled.
    • Melanie, Simon, and Gabriel all bought their tickets today.
    • Farah is already in Toronto, as is Severin.
    • Fernando is getting there by bus.
    • Mike Conley will send an e-mail to Greg about how to check-in to the hotel rooms.
  3. Tickets distribution
    • Mike has assigned each of us 1 or 2 tickets on Dr Project since code sprint is next weekend anyways.
      • To view yours: Dr Project > View Tickets, then add a filter on “Owner” and select “is” and your username
      • Or do a group by owner to see who all has what
    • Not exactly critical projects even if they’re priority “high”
    • MarkUs is big, so here are some small things to start with for each of us
    • All code must go through ReviewBoard before being committed to the subversion respoitory.
    • Mike, Severin (jerboaa) and Nelle are around to help with any questions about the code.
    • No questions about tickets (yet).
  4. Code sprint discussion (work part)
    • All of the teams get to mix and mingle together and then we sit down and code afterwards.
    • We’re going to be spread out across several rooms, together in a group though.
    • 8 hour days spent closing tickets, writing tests, and asking questions (and having fun)
    • Big white boards
    • Greg brings donuts
      • If you don’t like (or don’t want them in the morning) donuts, there is a Tim Horton’s near the Bahan Center for bagels, etc.
  5. Code sprint discussion (team hang out)
    • Gabriel suggested doing something on Thursday night
    • Him, Mélanie, and Simon arrive at Pearson at 8:15 pm
    • Tara and Mike don’t drink, but we agree to go for beers at 9 pm.
    • We divulged into a discussion about how drinking on the plane works and whether you can take your own alcohol onto the plane. Simon guessed that beers must cost $18 on the plane.
    • Severin suggested meeting at the Bahan Centre.
    • Tara said that the non-U of T people don’t know where that is and suggested meeting at the hotel, having the Torontonians guide us to the Bahan Center, to a pub, and then back to the hotel.
    • Mike agreed and suggested that we meet at 9 pm at the Bata Shoe Museum by the hotel. You can’t miss it. It’s a shoe museum. Lots of shoes in the windows.
    • Tara suggested that we exchange phone numbers in case we run into any problems trying to meet up. Everyone should e-mail Mike with their phone number (ensure that at least one peson in each traveling group has a cell phone) and he will e-mail us the list of everyone’s phone numbers.
    • Nelle then figured out that Melanie speaks French and we divulged into a short French discussion, with groans from Mike and Severin, as could be expected, but also from Gabriel who wants to practice his English. (what about me who wants to practice my French?!?)

And that was the end of our second meeting! See everyone Thursday evening in Toronto!!!

Written by Tara Clark

September 18th, 2009 at 7:22 pm

Posted in Uncategorized

Meeting Agenda – Sept 18, 2009

with one comment

  1. Great job on getting the development environment installed!  And on all three platforms!  Wow!
  2. Check that all the travel arrangements are set.  (Code sprint is one week away)
  3. Dole out some tickets. Talk about how to approach it.  If there isn’t a test already that shows how it fails, it may be worth writing the test.  (If all that is manageable is to write the test that exposes the bug, that would be a big step.)  Sometimes, it might not be necessary or possible to write the test.
  4. Time management.  It would be amazing if everyone could come to the code sprint having already checked in some code, but we will be spending all of the next weekend working on MarkUs, so everyone should make sure that they spend enough time on their other school work.

Written by m_conley

September 17th, 2009 at 11:09 pm

Posted in Uncategorized

Punchline Status Reports – Sept 18, 2009

with 2 comments

These are the punchline status reports for the Sept 18, 2009 IRC meeting in #markus at 12PM.

Farah

Status

  • Finished setting up my development environment
  • Updated the MarkUs developer installation instructions for Mac OS X with Simon (see Ticket #379)
  • Added instructions for setting up MySQL and updated some information about getting started with MarkUs development and running the rake tasks
  • Continued to read “Agile Web Development With Rails” and continued to go through some Rails tutorials

Next Steps

  • Start to familiarize myself with the MarkUs source code
  • Talk to Mike about tickets that would be good to start with

Roadblocks

  • None this week

Fernando

Status

  • Starting to understand the structure of the code
  • Finished with ticket #376 – got 1000th MarkUs commit!
  • Uploaded a review request
  • Still going through the basics of Rails

Next Steps

  • Get another bug assigned
  • Keep on reading tutorials of Rails

Roadblocks

  • My lack of familiarity with Rails

Gabriel

Status

  • Completed my local MarkUs installation on Linux. Had to try a few variations because the original procedure led me to some errors.
  • Updated the installation doc (along with Severin).
  • Read most of the project developer’s doc referenced on the wiki start page (Guidelines for dev, review board, test guidelines…).
  • Read about Ruby (The pragmatic programmer’s guide).

Next Steps

  • Read about Rails (up to now, I mostly read about plain Ruby).
  • Dive into the code (looking at it, trying it, modifying it and playing with it)

Roadblocks

None at the moment.

Melanie

Status

  • I have listened to the screen casts
  • I have written the MarkUs installation procedure on Windows
  • I have bought a book titled “Ruby on rails” and started to read it

Next Steps

  • I have to buy my tickets to go to Toronto.
  • I will continue reading my book
  • Will go through the documents on the Dr.Project page(I did not have the time to do it yet)

Roadblocks

  • None

Mike

Status

  • Still hanging out in IRC when I can, helping where I can
  • Posted the meeting notes from last week on the blog
  • Collected the punchline status updates for this week
  • Doled out some tickets
  • Reviewed some code from Fernando

Next Steps

  • I’m ready to review code that starts coming in from the tickets I’ve assigned.  Bring it on, people!

Roadblocks

  • Everything is fine.

Severin

Status

Next Steps

Roadblocks:

  • None

Simon

Status

  • I’ve read a book on ruby on rails and my project is fully working.

Next steps

  • I plan on looking at the bugs and go into the code. I also want to read on “shoulda” and look at the tests.

Roadblocks

  • Nothing

Tara

Status

  • I read through the first 7 chapters of “Agile Web Development with Rails” and read over several online Ruby tutorials.
  • InstantRails is up and running on my laptop and MarkUs is fully working.
  • Read through most of the documents on the Dr. Project site.
  • Set up an IDE – currently trying out Komodo Edit.
  • With Mélanie, updated the Windows install wiki page. (see ticket #378 ) *still working on this, will be done before I go to bed tonight

Next Steps

  • I will finish going through the documents on the Dr. Project site.
  • This weekend I would like to set aside some chunks of time to work on a first bug and better step through how some pieces of the code are set up.

Roadblocks

  • none at the moment

Written by m_conley

September 17th, 2009 at 11:07 pm

Posted in Uncategorized

We Are Using ReviewBoard

without comments

We’ve taken some effort to get ReviewBoard up and running. If you are not yet familiar with this tool, now is the time to do so. Please have a look at our “How to use ReviewBoard” document.

Written by Severin

September 17th, 2009 at 11:10 am

Posted in Uncategorized

IRC Meeting Notes – September 11, 2009

with one comment

IRC Meeting Notes

Transcribed by Mike Conley (m_conley)

  • Greeting, General Talk about IRC Meeting Structure
    • Should have agenda.  Karen will post the agenda on the blog next week.
    • Someone should take notes (today it’s Mike – next week it’ll be Tara)
    • Someone should collect status-report “punchlines” each week, and write up a blog post about it, so that people can give them a quick read before the meeting.  Format should be:

      Status: a few lines saying what you did
      Next Steps: What you plan to work on during the next week.  If unsure, say so.
      Roadblocks: What are the things that are getting in the way or making it difficult to complete your current task?

      Mike will collect punchlines next week.  Please send him your punchline on Thursday afternoon. Tara will do it the week after.

      We’ll just give punchlines in IRC today.

  • Status reports
    • Farah
    • At this point, Karen asks for everyone installing the development environment to take notes on the process, and that she would very much like to see everyone updating the how-to pages (or creating new ones) on the wiki for getting started on all these different OS’s.
    • Fernando:
      • has installed MarkUs
      • has been using it for a few days, still learning Rails basics
      • using Gentoo Linux
      • next step is to finish some Rails tutorials, and get familiar with the MarkUs code
      • also noted that uploading TA’s with CVS doesn’t work
      • Using Gentoo Linux
    • Gabriel:
      • has looked at the demo, and some screencasts
      • still needs to set up development environment
      • will start it up next week, get familiar with the code
      • still learning ruby, rails, etc
      • using Ubuntu
    • Severin:
      • ReviewBoard is completely functional (was a sending e-mail problem)
      • Will create a sample review-request for people to play with whilst/after our meeting
      • Development environment is up and running (ready to dive right in)
      • Have had a look at the tickets (added one comment to the “features” blog page)
      • It might be useful if we could annotate pdfs with MarkUs (there are Use Cases for courses such as CSC236)
      • markusproject.org generates test-coverage on a daily basis, now (no need for somebody creating them manually)
      • Using Linux
    • Karen, Mike and Severin direct everybody to https://stanley.cdf.toronto.edu/drproject/csc49x/olm_rails/wiki/DeveloperGuidelines
    • Karen suggests that a sandbox be created in the repo to check in bits of code before it goes into the main trunk – says she’ll talk more about this later
    • Mike:
      • Development environment is ready to go
      • Just been doing support, helping everybody get set up
      • using Ubuntu
    • Melanie:
      • after some difficulty, has installed MarkUs successfully on Windows
      • has played with the MarkUs demo
      • is planning on reading up on Ruby on Rails
      • will go through the documents on the Dr.Project page
      • Karen would like Melanie to write up what actually worked for her installation process, and to add it as a wiki page on Dr. Project.
    • Simon:
      • development environment is up and running
      • is reading a Rails book he got from a friend
      • is unfamiliar with MarkUs, but plans to look at screencasts and play with demo
      • is using Mac OSX
    • Tara:
      • bought a copy of a Rails book
      • read the intro, working on the ruby section to better understand the language
      • goal is to finish the ruby section and the 2nd chapter of the book, and start working on getting development environment set up on Monday
      • using Windows
      • seen the screencasts, and played with demo
      • already has an Apache/MySQL stack set up on her machine
  • Next Steps
    • At this point, Mike Gunderloy suggests bookmarking http://apidock.com/rails for reference.
    • Mike says that he’ll be doing code reviews, and wants everybody to use ReviewBoard
    • Severin will write up ReviewBoard instructions, and put them on the blog
    • Karen wants everyone to have the MarkUs development environment installed by the next meeting
    • Those who already have MarkUs installed should talk to Mike about being assigned small tickets to start getting hands into code
    • Karen wants Simon, Gabriel and Melanie to look at the unit tests, see how they work, and see what we have in place.
    • Karen wants to arrange a time with Adam Goucher where they can all talk a little more about overall term goals.
    • Ainsley Lawson has had to drop out of the project
    • Meetings will now be held on Fridays at 12PM
    • Mike Gunderloy, Blake Winton, Adam Goucher, and Andrew Louis are introduced

Written by m_conley

September 11th, 2009 at 5:56 pm

Posted in Uncategorized

New features and other goals

with one comment

At the beginning of the summer I wrote up a list of goals and list of features that would be nice to have.  We didn’t get to any of the new features, but we did a pretty good job on the primary goals.  One of the first priorities for the fall is to identify appropriately sized projects for each of the groups to work on.

Before we get fully into the projects, I’d like everyone to have a look at the current tickets and choose a few of the smaller ones to solve as a warm-up exercise and as a way to become more familiar with the code base.

Here is the list of features that I’d really like to see in MarkUs.  It is highly likely that we will not complete all of them.  It is also likely that some of them will be relatively straightforward.  What am I missing?  I’m sure a few more feature requests will come in as instructors, TAs and students start to use MarkUs.

Logging

I’d like to be able find out how students and TAs are using MarkUs.  The purpose of collecting data is to

  • Improve MarkUs: I hope that by looking at how all users actually use MarkUs, we may be able to identify workflow issues, UI issues, and training issues.
  • Help students: We may be able to see student behaviours that we’d like to reinforce or correct.
  • Monitor TAs: It would be nice to know whether I should be asking TAs to provide more feedback, or prod them to get started, or congratulate them for work well done.
  • Gather evidence that MarkUs is useful. It would be nice to have some evidence that MarkUs actually helps instructors, TAs, and students get their work done.

It is probably not difficult to add log messages to MarkUs.  The interesting part of this task is figuring out which events to log, and how to visualize the data.

Add a different marking scheme

The rubric marking scheme works reasonably well, but it would be nice to have at least one other type of marking scheme.  I have one in mind, in which the instructor provides a description for a category or criterion and a total number of marks for that criterion.  The TA would be asked to enter a number into a box for each criterion.

Simple grade entry

If I’m already using MarkUs to keep track of assignment marks, it might be nice to have a simple interface to enter grades from labs, midterms, or other activities.  I imagine this will be another type of “assignment” that the instructor could create.

Messages visible only to instructors and/or TAS

I’d like to be able to add notes to a students’ results in the grader view that are not visible to the student.  I imagine these notes to be automatically time-stamped and labeled with the author’s name or id.  A TA might use this mechanism to alert the instructor to a problem in a particular assignment.  An instructor might use this to make notes of changes made due to remarks.  I envision this mechanism as an additional tab in the grader view.

Automated checking of students’ code

This is the big one.  A long term goal is to build the infrastructure so that students could submit code and receive immediate feedback.  This infrastructure could be used in a few different ways.  For example, students might be able to submit their code before the due date and find out how many reference tests pass on their code, or they might get a coverage analysis of their own tests on their code.  Another use of this infrastructure would be to give students small exercises that could be automatically checked.

The biggest problem with this feature is that we have to run student code automatically and securely.  Having looked into this problem last summer, the best solution seems to be to run the students’ code in a virtual machine.  The issues include

  • Security:  How do we lock down a VM so that we can be sure that student code will not bring down the server, or could access data it should not be able to access (like solution code for example).
  • Performance: Is it really feasible to run one or more VMs on our server and still give  tolerable response times.
  • UI for the instructor: How does the instructor actually set up the tests? How does the instructor specify what type of feedback the students will receive.
  • UI for the student: Students need to get useful feedback when they submit their code for testing.
  • Backend: We need to keep track of the results every time students submit their code for testing.

Written by Karen Reid

September 3rd, 2009 at 11:00 pm

Posted in Uncategorized

Seeing MarkUs in Action

without comments

We have done some screencasts over the last couple of months and we would like to share them with you.

Student File Submssions

This screencast demonstrates how students can upload, replace and delete files using MarkUs’ easy to use Web interface. And all these files end up in a Subversion repository. Hence, MarkUs implicitly has a history of submitted files. If need be, revisions other than the latest can be graded provided the instructor wants to allow that.

 

Student Group Formation

MarkUs supports various course models. One possibility of which is that an instructor can allow students to work in teams. If an Assignment is set up this way, students can go and start form groups (mostly) to their liking. For instance, student A can invite student B and C to his group. Students B and C will, then, have the option to join student A’s group, decline student A’s invitation or create a group of their own. Once groups are formed (and are valid), students can begin to submit files for their assignment.

 

The Grader View

This screencast (partially) illustrates the huge progress we’ve made this summer. Although the main grader view functionality was there when we started, the user interface as it is today de facto wasn’t. So what is the grader view? It’s one of the core functionalities of MarkUs, which allows graders (usually TAs) to mark a student’s/group’s work for an assignment. He or she can annotate the submitted source code – which is syntax highlighted – and mark according to rubrics defined by the instructor. As mentioned above, this screencast was done before our general UI overhaul (and name change). Please excuse. The workings of the grader view remained (almost) the same though.

Grader View Screencast

Written by Severin

September 2nd, 2009 at 6:03 pm

Posted in Uncategorized