Archive for September, 2009
The MarkUs team had a meeting yesterday to discuss our tasks for the rest of the semester. As promised, here is a description of the exciting new features that we are planning on adding to MarkUs:
- Simple Grade Entry
- Used for entering grades from labs, exams, etc.
- This will be another type of “assignment” that the instructor can create
- Considerations and open questions:
- Permissions (eg. TAs need to be able to modify the table, students are not allowed to see the whole table, etc.)
- Attaching graders
- Conditions for releasing marks
- What’s the workflow for creating the grade entry form? What’s the UI for entering grades? What’s the UI for students? Model?
- Allow messages about a student’s results (eg. notes about remark requests, concerns, etc.) to be added in the grader view
- This will be a tab in the grader view
- Messages are visible to instructors and TAs only
- Timestamp and mark the user name for annotations
- Ways to aggregate messages for an assignment
- Ways to download and/or view aggregate
- Messages per student
- Viewing messages – notify the instructor / TA about messages since the last time they logged in
- New Marking Scheme
- Allow the instructor to specify a description of a criterion and a total number of marks for that criterion
- Ways to create title, description, and weight
- Ways to upload/download description
- Format for description
- Allowing penalties, deductions, bonus marks, etc.
- UI – the TA will need to enter a number into a box for each criterion
- Want to find out how students and TAs are using MarkUs
- Number of submissions and when (can get this information from svn)
- Who is submitting? (eg. Is one person doing all of the work in a group project?)
- Who is looking up their grades?
- Information about groups (eg. accepts, rejects)
- Number of annotations
- Tracking setting grades
- Time spent marking
- Need to create a log file first and then determine how to visualize the data
- Considerations and open questions:
- The size of the log files
- How should we store this information? DB versus log file?
- Rotating logs?
- How many messages might there be?
- Automated Testing
- Allow students to run tests against their code
- Can be used to run style checks, code coverage tests, etc.
- UI for the student and instructor
- Where/how will the tests be run?
After discussing these new features, we split up the tasks based on our interests. Here’s the breakdown:
- Tara and Fernando will be working together on the messages. Fernando will also do some work on logging first.
- Melanie, Simon, and Gabriel will work on the new marking scheme. They will be doing test-driven development.
- Farah will work on the simple grade entry.
- Severin is going to get started on automated testing.
- Everyone will be developing tests for their code.
We’re all set for a productive term! Let the fun begin!
Here’s a quick snapshot of what the team did yesterday:
- Lots of coding! Tickets were assigned, ReviewBoard was cookin’ with lots of activity
- A bunch of code from the new team got past review, and went straight into the repo! Excellent!
- A unexpectedly long lunch break
- All of the teams celebrated the end of the day by heading out to a Chinese restaurant. So much food!
Today is the last day of the sprint. We’re having a meeting later today to figure out the direction for the rest of term. We’ll keep you posted!
I believe the team is planning on doing a large writeup on the Codesprint after the weekend, but I thought I’d wet your whistle with a little preview:
- All team members arrived in Toronto safely!
- Everyone was able to get online quickly. Team began working furiously.
- Free pizza lunch == amazing
- Adam Goucher joined us, and gave us some excellent industry-level advice on how to test MarkUs
- The MarkUs ReviewBoard is getting busy! All code (even the small snippets) is getting reviewed before being committed to the repo.
- After a solid day of work, the team eventually headed to a pub for free food and drink
So that’s just a taste. Stay tuned!
IRC Transcript – starting around 11:54 am and going through to the 3rd page.
Mike Conley (m_conley) ran the meeting today, with Tara (tlclark) taking notes.
- Development environments
- We all have our development environments and MarkUs set up, across all 3 platforms. Yay!
- Karen and Mike thank us for working hard to update the install documentation.
- Code sprint travel arrangements
- Tara is driving up, all settled.
- Melanie, Simon, and Gabriel all bought their tickets today.
- Farah is already in Toronto, as is Severin.
- Fernando is getting there by bus.
- Mike Conley will send an e-mail to Greg about how to check-in to the hotel rooms.
- Tickets distribution
- Mike has assigned each of us 1 or 2 tickets on Dr Project since code sprint is next weekend anyways.
- To view yours: Dr Project > View Tickets, then add a filter on “Owner” and select “is” and your username
- Or do a group by owner to see who all has what
- Not exactly critical projects even if they’re priority “high”
- MarkUs is big, so here are some small things to start with for each of us
- All code must go through ReviewBoard before being committed to the subversion respoitory.
- Subversion doesn’t require this, so be careful!
- Mike, Severin (jerboaa) and Nelle are around to help with any questions about the code.
- No questions about tickets (yet).
- Mike has assigned each of us 1 or 2 tickets on Dr Project since code sprint is next weekend anyways.
- Code sprint discussion (work part)
- All of the teams get to mix and mingle together and then we sit down and code afterwards.
- We’re going to be spread out across several rooms, together in a group though.
- 8 hour days spent closing tickets, writing tests, and asking questions (and having fun)
- Big white boards
- Greg brings donuts
- If you don’t like (or don’t want them in the morning) donuts, there is a Tim Horton’s near the Bahan Center for bagels, etc.
- Code sprint discussion (team hang out)
- Gabriel suggested doing something on Thursday night
- Him, Mélanie, and Simon arrive at Pearson at 8:15 pm
- Tara and Mike don’t drink, but we agree to go for beers at 9 pm.
- We divulged into a discussion about how drinking on the plane works and whether you can take your own alcohol onto the plane. Simon guessed that beers must cost $18 on the plane.
- Severin suggested meeting at the Bahan Centre.
- Tara said that the non-U of T people don’t know where that is and suggested meeting at the hotel, having the Torontonians guide us to the Bahan Center, to a pub, and then back to the hotel.
- Mike agreed and suggested that we meet at 9 pm at the Bata Shoe Museum by the hotel. You can’t miss it. It’s a shoe museum. Lots of shoes in the windows.
- Tara suggested that we exchange phone numbers in case we run into any problems trying to meet up. Everyone should e-mail Mike with their phone number (ensure that at least one peson in each traveling group has a cell phone) and he will e-mail us the list of everyone’s phone numbers.
- Nelle then figured out that Melanie speaks French and we divulged into a short French discussion, with groans from Mike and Severin, as could be expected, but also from Gabriel who wants to practice his English. (what about me who wants to practice my French?!?)
And that was the end of our second meeting! See everyone Thursday evening in Toronto!!!
- Great job on getting the development environment installed! And on all three platforms! Wow!
- Check that all the travel arrangements are set. (Code sprint is one week away)
- Dole out some tickets. Talk about how to approach it. If there isn’t a test already that shows how it fails, it may be worth writing the test. (If all that is manageable is to write the test that exposes the bug, that would be a big step.) Sometimes, it might not be necessary or possible to write the test.
- Time management. It would be amazing if everyone could come to the code sprint having already checked in some code, but we will be spending all of the next weekend working on MarkUs, so everyone should make sure that they spend enough time on their other school work.
These are the punchline status reports for the Sept 18, 2009 IRC meeting in #markus at 12PM.
- Finished setting up my development environment
- Updated the MarkUs developer installation instructions for Mac OS X with Simon (see Ticket #379)
- Added instructions for setting up MySQL and updated some information about getting started with MarkUs development and running the rake tasks
- Continued to read “Agile Web Development With Rails” and continued to go through some Rails tutorials
- Start to familiarize myself with the MarkUs source code
- Talk to Mike about tickets that would be good to start with
- None this week
- Starting to understand the structure of the code
- Finished with ticket #376 – got 1000th MarkUs commit!
- Uploaded a review request
- Still going through the basics of Rails
- Get another bug assigned
- Keep on reading tutorials of Rails
- My lack of familiarity with Rails
- Completed my local MarkUs installation on Linux. Had to try a few variations because the original procedure led me to some errors.
- Updated the installation doc (along with Severin).
- Read most of the project developer’s doc referenced on the wiki start page (Guidelines for dev, review board, test guidelines…).
- Read about Ruby (The pragmatic programmer’s guide).
- Read about Rails (up to now, I mostly read about plain Ruby).
- Dive into the code (looking at it, trying it, modifying it and playing with it)
None at the moment.
- I have listened to the screen casts
- I have written the MarkUs installation procedure on Windows
- I have bought a book titled “Ruby on rails” and started to read it
- I have to buy my tickets to go to Toronto.
- I will continue reading my book
- Will go through the documents on the Dr.Project page(I did not have the time to do it yet)
- Still hanging out in IRC when I can, helping where I can
- Posted the meeting notes from last week on the blog
- Collected the punchline status updates for this week
- Doled out some tickets
- Reviewed some code from Fernando
- I’m ready to review code that starts coming in from the tickets I’ve assigned. Bring it on, people!
- Everything is fine.
- ReviewBoard HowTo has been written: https://stanley.cdf.toronto.edu/drproject/csc49x/olm_rails/wiki/HowToReviewBoard
- Developer installation instructions for Linux have been improved
- Supported Alan in doing the Stanley install
- Reorganized wiki start page
- Improved InstallProd wiki page: https://stanley.cdf.toronto.edu/drproject/csc49x/olm_rails/wiki/InstallProd
- I’ve read a book on ruby on rails and my project is fully working.
- I plan on looking at the bugs and go into the code. I also want to read on “shoulda” and look at the tests.
- I read through the first 7 chapters of “Agile Web Development with Rails” and read over several online Ruby tutorials.
- InstantRails is up and running on my laptop and MarkUs is fully working.
- Read through most of the documents on the Dr. Project site.
- Set up an IDE – currently trying out Komodo Edit.
- With Mélanie, updated the Windows install wiki page. (see ticket #378 ) *still working on this, will be done before I go to bed tonight
- I will finish going through the documents on the Dr. Project site.
- This weekend I would like to set aside some chunks of time to work on a first bug and better step through how some pieces of the code are set up.
- none at the moment
We’ve taken some effort to get ReviewBoard up and running. If you are not yet familiar with this tool, now is the time to do so. Please have a look at our “How to use ReviewBoard” document.
IRC Meeting Notes
Transcribed by Mike Conley (m_conley)
- Greeting, General Talk about IRC Meeting Structure
- Should have agenda. Karen will post the agenda on the blog next week.
- Someone should take notes (today it’s Mike – next week it’ll be Tara)
- Someone should collect status-report “punchlines” each week, and write up a blog post about it, so that people can give them a quick read before the meeting. Format should be:
Status: a few lines saying what you did
Next Steps: What you plan to work on during the next week. If unsure, say so.
Roadblocks: What are the things that are getting in the way or making it difficult to complete your current task?
Mike will collect punchlines next week. Please send him your punchline on Thursday afternoon. Tara will do it the week after.
We’ll just give punchlines in IRC today.
- Status reports
- has tried out the demo
- seen the screencasts
- still learning Ruby basics
- has gone through some basic Rails tutorials
- starting reading the book that Karen suggested to her
- using Mac OSX
- still needs to get development environment set up
- Mike Gunderloy suggested http://afreshcup.com/2009/09/02/migrating-to-snow-leaopard-for-rails-development-a-definitive-guide/ for Mac OSX users.
- Karen says that next step is to start working on installing the development environment
- At this point, Karen asks for everyone installing the development environment to take notes on the process, and that she would very much like to see everyone updating the how-to pages (or creating new ones) on the wiki for getting started on all these different OS’s.
- has installed MarkUs
- has been using it for a few days, still learning Rails basics
- using Gentoo Linux
- next step is to finish some Rails tutorials, and get familiar with the MarkUs code
- also noted that uploading TA’s with CVS doesn’t work
- Using Gentoo Linux
- has looked at the demo, and some screencasts
- still needs to set up development environment
- will start it up next week, get familiar with the code
- still learning ruby, rails, etc
- using Ubuntu
- ReviewBoard is completely functional (was a sending e-mail problem)
- Will create a sample review-request for people to play with whilst/after our meeting
- Development environment is up and running (ready to dive right in)
- Have had a look at the tickets (added one comment to the “features” blog page)
- It might be useful if we could annotate pdfs with MarkUs (there are Use Cases for courses such as CSC236)
- markusproject.org generates test-coverage on a daily basis, now (no need for somebody creating them manually)
- Using Linux
- Karen, Mike and Severin direct everybody to https://stanley.cdf.toronto.edu/drproject/csc49x/olm_rails/wiki/DeveloperGuidelines
- Karen suggests that a sandbox be created in the repo to check in bits of code before it goes into the main trunk – says she’ll talk more about this later
- Development environment is ready to go
- Just been doing support, helping everybody get set up
- using Ubuntu
- after some difficulty, has installed MarkUs successfully on Windows
- has played with the MarkUs demo
- is planning on reading up on Ruby on Rails
- will go through the documents on the Dr.Project page
- Karen would like Melanie to write up what actually worked for her installation process, and to add it as a wiki page on Dr. Project.
- development environment is up and running
- is reading a Rails book he got from a friend
- is unfamiliar with MarkUs, but plans to look at screencasts and play with demo
- is using Mac OSX
- bought a copy of a Rails book
- read the intro, working on the ruby section to better understand the language
- goal is to finish the ruby section and the 2nd chapter of the book, and start working on getting development environment set up on Monday
- using Windows
- seen the screencasts, and played with demo
- already has an Apache/MySQL stack set up on her machine
- Next Steps
- At this point, Mike Gunderloy suggests bookmarking http://apidock.com/rails for reference.
- Mike says that he’ll be doing code reviews, and wants everybody to use ReviewBoard
- Severin will write up ReviewBoard instructions, and put them on the blog
- Karen wants everyone to have the MarkUs development environment installed by the next meeting
- Those who already have MarkUs installed should talk to Mike about being assigned small tickets to start getting hands into code
- Karen wants Simon, Gabriel and Melanie to look at the unit tests, see how they work, and see what we have in place.
- Karen wants to arrange a time with Adam Goucher where they can all talk a little more about overall term goals.
- Ainsley Lawson has had to drop out of the project
- Meetings will now be held on Fridays at 12PM
- Mike Gunderloy, Blake Winton, Adam Goucher, and Andrew Louis are introduced
At the beginning of the summer I wrote up a list of goals and list of features that would be nice to have. We didn’t get to any of the new features, but we did a pretty good job on the primary goals. One of the first priorities for the fall is to identify appropriately sized projects for each of the groups to work on.
Before we get fully into the projects, I’d like everyone to have a look at the current tickets and choose a few of the smaller ones to solve as a warm-up exercise and as a way to become more familiar with the code base.
Here is the list of features that I’d really like to see in MarkUs. It is highly likely that we will not complete all of them. It is also likely that some of them will be relatively straightforward. What am I missing? I’m sure a few more feature requests will come in as instructors, TAs and students start to use MarkUs.
I’d like to be able find out how students and TAs are using MarkUs. The purpose of collecting data is to
- Improve MarkUs: I hope that by looking at how all users actually use MarkUs, we may be able to identify workflow issues, UI issues, and training issues.
- Help students: We may be able to see student behaviours that we’d like to reinforce or correct.
- Monitor TAs: It would be nice to know whether I should be asking TAs to provide more feedback, or prod them to get started, or congratulate them for work well done.
- Gather evidence that MarkUs is useful. It would be nice to have some evidence that MarkUs actually helps instructors, TAs, and students get their work done.
It is probably not difficult to add log messages to MarkUs. The interesting part of this task is figuring out which events to log, and how to visualize the data.
Add a different marking scheme
The rubric marking scheme works reasonably well, but it would be nice to have at least one other type of marking scheme. I have one in mind, in which the instructor provides a description for a category or criterion and a total number of marks for that criterion. The TA would be asked to enter a number into a box for each criterion.
Simple grade entry
If I’m already using MarkUs to keep track of assignment marks, it might be nice to have a simple interface to enter grades from labs, midterms, or other activities. I imagine this will be another type of “assignment” that the instructor could create.
Messages visible only to instructors and/or TAS
I’d like to be able to add notes to a students’ results in the grader view that are not visible to the student. I imagine these notes to be automatically time-stamped and labeled with the author’s name or id. A TA might use this mechanism to alert the instructor to a problem in a particular assignment. An instructor might use this to make notes of changes made due to remarks. I envision this mechanism as an additional tab in the grader view.
Automated checking of students’ code
This is the big one. A long term goal is to build the infrastructure so that students could submit code and receive immediate feedback. This infrastructure could be used in a few different ways. For example, students might be able to submit their code before the due date and find out how many reference tests pass on their code, or they might get a coverage analysis of their own tests on their code. Another use of this infrastructure would be to give students small exercises that could be automatically checked.
The biggest problem with this feature is that we have to run student code automatically and securely. Having looked into this problem last summer, the best solution seems to be to run the students’ code in a virtual machine. The issues include
- Security: How do we lock down a VM so that we can be sure that student code will not bring down the server, or could access data it should not be able to access (like solution code for example).
- Performance: Is it really feasible to run one or more VMs on our server and still give tolerable response times.
- UI for the instructor: How does the instructor actually set up the tests? How does the instructor specify what type of feedback the students will receive.
- UI for the student: Students need to get useful feedback when they submit their code for testing.
- Backend: We need to keep track of the results every time students submit their code for testing.
We have done some screencasts over the last couple of months and we would like to share them with you.
Student File Submssions
This screencast demonstrates how students can upload, replace and delete files using MarkUs’ easy to use Web interface. And all these files end up in a Subversion repository. Hence, MarkUs implicitly has a history of submitted files. If need be, revisions other than the latest can be graded provided the instructor wants to allow that.
Student Group Formation
MarkUs supports various course models. One possibility of which is that an instructor can allow students to work in teams. If an Assignment is set up this way, students can go and start form groups (mostly) to their liking. For instance, student A can invite student B and C to his group. Students B and C will, then, have the option to join student A’s group, decline student A’s invitation or create a group of their own. Once groups are formed (and are valid), students can begin to submit files for their assignment.
The Grader View
This screencast (partially) illustrates the huge progress we’ve made this summer. Although the main grader view functionality was there when we started, the user interface as it is today de facto wasn’t. So what is the grader view? It’s one of the core functionalities of MarkUs, which allows graders (usually TAs) to mark a student’s/group’s work for an assignment. He or she can annotate the submitted source code – which is syntax highlighted – and mark according to rubrics defined by the instructor. As mentioned above, this screencast was done before our general UI overhaul (and name change). Please excuse. The workings of the grader view remained (almost) the same though.