MarkUs Blog

MarkUs Developers Blog About Their Project

Archive for October, 2009

Notes System: Initial Design

with 4 comments

At the code sprint, we tossed around some ideas for the “Messages between TAs and instructors” feature that Karen asked for. Today, I’ve been brainstorming what I think this feature should look like, functionally.

Feature goal

To allow small pieces of information to be left on various aspects of the course, pertaining to something specific, to alert others to the problems (or good things!), with a timestamp. These could potentially pertain to an assignment, a student, or a grouping, or we could limit these to just one of these types.

Example uses

If students are suspected to be cheating, then the grader could “flag” the assignment for the instructor by leaving a note. Since other graders can also see it, then they can see that this student was previously flagged for cheating. If the student is sick and gets an exemption on part of the assignment, then a note could be placed on the affected assignment for the grader to take that into account during marking.

Can anyone think of other potential uses of this feature?

Where to create these?

Now that we have some idea of how this could be used in a real course – it’s always key to keep the users in mind – we need to think about where in the work flow we would want to create these pieces of information.

While marking an assignment, we should be able to make notes. These would be noted against the grouping, since this encompasses assignments where the students work alone, as well as assignments where the students work together and keeps in mind the idea that one student having a problem can affect all students in the grouping. It also allows us to have notes go against a single object, since a grouping represents an assignment as well as the students working on it.

We should also be able to make a comment about a particular student in general or about a a particular student on an assignment, not from the marking view. Should we also be able to comment about an assignment? This could be incorporated into the viewing of the comments described below.

Replies

What about replies to a comment? Should we incorporate these into our feature? I think that we shouldn’t at this time, since the initial idea is for more one-off ideas, but I think that the model should be designed in a flexible enough way such that we can have multiple notes against the same object, which would allow for multiple notes to be left against an object and thus “replies” of a sort without complicating the model too much.

Where to view these?

I propose adding a new tab in the menu for Admins and Graders, with its text being the name we decide upon for this feature. In here, we can display all of the messages (paged in case there are a lot of them?) and have a link to add new ones as well. Would distinguishing the new comments made since last login from the previous ones be useful? You should be able to sort the view. You should also be able to filter it by assignment or by student ID (which would obviously return all comments left against that student, as well as comments left against any groupings it is a member of).

Model

At the code sprint meeting, someone suggested modifying the annotations model for these notes, but that idea was declined since we don’t want to relate these to the code. The conclusion was that we will need a new model to represent these ideas.

The pieces of information that we need to store are: the user who wrote the note, the timestamp from when it was written, the object ID that it is against, the type of object it is against (grouping, student, assignment, etc.), and the actual text of the note itself.

Naming

We need to consider how we’re going to name this feature. Several initial possibilities come to mind: Messages, Notes, and Comments. (I actually found all three of these used interchangeably in my notes from the Sunday code sprint meeting!)

  • I don’t like the term “Messages” because it implies that they are sent from one person to another, which these won’t be – they’ll be available for viewing by all instructors AND graders. I do, however, like its implication that these are being shared with other people.
  • The term “Comments” is too close to the ideas of commenting code and the overall comments on an assignment that are shared with the students. I think that a new term is in order to ensure that information is not left in this field that the students shouldn’t see.
  • My favourite is definitely the term “Notes” because it implies that these are small pieces of information, pertaining to something specific. It doesn’t carry the connotation that it is just between specific individuals, so it is much more fitting than “Messages”.

I am definitely open to suggestions on other names or further comments on the names that I have suggested.

Anything else?

Are there any other pieces related to this feature that I haven’t outlined? I look forward to hearing your ideas about the upcoming notes feature in MarkUs!

Written by Tara Clark

October 17th, 2009 at 10:47 pm

Posted in Notes System

Tagged with

MarkUs Meeting Minutes Oct. 16

with one comment

IRC logs can be found here.

General

  • Everybody makes steady progress.
  • Proposal of the Laval students looks good.
  • Karen would like to see opinions on the various blog posts.
  • Farah and Karen have met and agreed on the general structure of the grade entry form. Now it’s time to start implementing it.
  • Overall view of grades for MarkUs would be useful (maybe a project for next term).
  • Karen didn’t have a chance to look at the logging work of Fernando or the messaging work of Tara. Karen thinks she can provide more feedback this weekend.
  • Severin met with Karen regarding automated testing. It’s time to write down trade offs which might have to be made (also Pros/Cons) so that they can be evaluated.
  • People should focus on their projects, but it would be helpful if everybody could tackle the occasional ticket.
  • Because Tara is on 0.75 instead of 0.5 for this course Karen proposes to spend some time on those tickets.
  • Tara has tickets #427 and #428 assigned to her.
  • Karen would be happy to arrange Skype or phone meetings, if need be.

MarkUs in Production

  • We are starting to get more feedback from users (new tickets have been filed).
  • Need to go through new tickets and decide if they are solvable. They also need to get prioritized.
  • Mike has been the knight in shining armor by creating urgent fixes for bugs in the 0.5 branch of MarkUs last weekend. Many kudos. Here is more to read on this.
  • Patches for 0.5 have been prepared.

Meeting ends at 12:26 pm

Written by Severin

October 16th, 2009 at 7:15 pm

Posted in Meetings,Minutes

Eclipse/Aptana RadRails/Subclipse pitfall

with 3 comments

UPDATE: Since Severin comment, I doubt that I identified the right cause for my problem. I’ll leave the post public for a couple more days, in case something pops out. If not, I’ll just remove it as I fear it might be more misleading than helpful.

It would be temping to ask you all to try it, but the cost of having you all break your working copies may not be worth it.

—–

In order for my team members to avoid a pitfall
    As a MarkUs developer
    I want to write a blog post.

Scenario: SVN updating in an Aptana driven development environment
    Given I configured an Eclipse IDE with the Aptana RadRails plug-in
    And I checked out MarkUs code with my "Aptana version" of the Subclipse plugin
    When I run an "svn update" from my MarkUs root directory in a shell
    Then I should still be able to use Subclipse.

The above didn’t work for me

It only happened a couple of days ago. I had the habit of calling for updates in the eclipse’s project explorer (pretty convenient) but I always launched my rake tasks from the command line. So one day, as I wanted to update my working copy, I mechanically typed “svn update<return>” in a shell and that was it. Now the Subclipse plug-in complains that he’s too old to use my working copy.

Unfortunately a “Search for software updates” didn’t do the trick.

Please note here that I’m using the “Aptana proposed” Subclipse plug-in, but I prefer not to mingle with that since the whole development environment is Aptana provided. I’m not sure if what I’m talking about here even applies to those who use  the latest Subversive plug-in (I hope not) or even the latest Subclipse. By the way, Subversive is usually my weapon of choice.

So, a word to those of you that uses Aptana RadRails’ provided Subclipse plug-in: Keep yourself from doing command line updates (or commit, I guess) or you’ll suffer from it. Specially if you have some modified files in your working copy, like I had. Not exactly a nightmare, but you might want to keep yourself away from that pit.

Written by gabrielrl

October 16th, 2009 at 2:08 pm

Automated Testing

with 2 comments

The idea is to integrate some sort of testing infrastructure into MarkUs, so that students get more feedback about their code submitted for assignments. A possible scenario would be the following:

  1. A new assignment is up in MarkUs and students start to work on it
  2. An instructor has written tests for this assignment, which will be run on the code Students submit
  3. Students will get feedback about their code submitted according to the results of those test runs

The question is how to achieve this. I’ll refer to the server or infrastructure which will be executing tests only as test server. Here are some questions we are trying to find the answer for:

  1. Where and how will the tests on students code be run?
  2. Should they be run on a dedicated server? On several servers?
  3. How does MarkUs communicate with the test server?
  4. How to decouple MarkUs from the test server?
  5. Is it a concern if a student compromises the test server either maliciously or accidentally?
  6. What are the required security measures for the test server?
  7. How to satisfy as many programming languages and course models as possible?
  8. What will the test server or test infrastructure look like? Should tests for each MarkUs instance run in separated virtual machines?
  9. Should we use virtual machines? If yes, why? Why not?
  10. How can we secure the test server well enough so that no secret information will be revealed to students? Should the mail system be disabled, network/socket connections be monitored or disallowed? How to secure the test server for courses in which students learn process forking and socket programming?
  11. Will there be performance problems? How to deal with increased load close to assignment deadlines?
  12. Should the amount of test runs be limited for students? If so, how? e.g. 5 test runs per day or max number of test runs?
  13. How much information of tests results should be shown to the student? Number of successes/failures? Detailed test failure messages? Should descriptions of tests be displayed?

These are related projects to get some ideas from. If you know some more, please let me know!

I am looking for ideas and hints regarding automated testing systems, how to integrate them into MarkUs and why to use one approach rather then the other. Please feel free to chime in and comment to this post. Greatly appreciated!

Written by Severin

October 16th, 2009 at 11:00 am

New marking scheme proposal

with 11 comments

A PDF file is included in this blog post about the new marking scheme. It contains User Stories, functional and implementation details.

– Three implementations are discussed and we want to know what you think about them.
– We also need approval from Karen about the User Stories.

New marking scheme proposal (v1.1)

In answer to Karen’s comment, this is 2 more database scenarios  Scenario4And5.

We hope to discuss about the new marking scheme on the IRC meeting tomorrow.

– Ulaval Trio

Written by simonlg

October 16th, 2009 at 12:46 am

Posted in Missing Features

Simple Grade Entry: Design Update

with 2 comments

Karen and I met yesterday to discuss the new simple grade entry feature. We went over the ideas and questions I mentioned in my initial design proposal. Here are the changes/implementation details that we discussed:

Grade Entry Form Creation:

A grade entry form is going to be treated as an assignment. This new feature will be found under the “Assignments” tab. We are going to add a new button that says “Create New Grade Entry Form” (this means that the old creation button for assignments will need to be renamed to “Create New Assignment” and “Choose Assignment” will be renamed to simply “Choose…” for now.) The creation form will contain most of the properties I described in my last post. However, in order to specify the columns for the grade entry form, the instructor will click on something like “Add Column” and enter a column name and the total number of marks for that question. The instructor can then continue to add as many columns as desired.

Views for Instructors, TAs, and Students:

1) Instructors

  • For instructors, a grade entry form will have a properties view, a graders view, and a table view.
  • From the graders view, instructors will be able to assign TAs to specific grade entry forms. TAs will be given permission to edit the entire form. We’re also going to allow multiple TAs to be assigned to a single grade entry form. This means that we might need to watch out for race conditions. However, technically, two TAs should never find themselves in a situation where they are both modifying the same table cell at the same time. But, if this does happen, then a reasonable solution is to let the last writer win.
  • From the table view, the instructor should also be able to download or upload the grade entry form in CSV format.

2) TAs

  • Grade entry forms will appear on a TA’s main page along with the assignments he/she needs to mark. The TA will be able to click on a grade entry form and then the table view will get displayed.

3) Students

  • Grade entry forms will appear on a student’s main page along with his/her assignments. The student’s total mark will be displayed on this page and the student will be able to click on a grade entry form to see the breakdown of the marks.

I think we now have the design details sorted out. Now, it’s time to start on the implementation!

Written by Farah Juma

October 15th, 2009 at 5:50 pm

Punchline Status Reports – Oct 16, 2009

without comments

Mike Conley

Status

  • Trying to close some svn_authz related performance issues in 0.5 release

Next Steps

  • Study how svn_authz and groups work – perhaps we can find an optimization

Roadblocks

  • I’ve got two major projects due next week.  Gripe gripe gripe.

Mélanie Gaudet

Status

  • Contributed to write a report about the new marking scheme

Next Steps

  • Ticket #309(Write test for annotationControler) is still on my list
  • Work on new marking scheme dev.

Roadblocks

  • None

Fernando Garces

Status

  • Posted on the blog about logging implementations
  • Requested a code review for v1 of my logging implementation

Next Steps

  • Starting doing modifications based on the feedback

Roadblocks

  • None

Severin Gehwolf

Status

  • Closed ticket #420 (Students can delete groups in which only they are members)
  • Commented on blog
  • Still missing a blog post for automated testing
  • Tested code in branches/release_0.5

Next Steps

  • Move on regarding automated testing
  • Work on some other tickets
  • Merge code from release_0.5 branch into trunk again

Roadblocks

  • Not sure what’s the best approach to get some early results for automated testing. Meeting with Karen might help.

Tara Clark

Status

  • Fixed Ticket #403 (assignments stuff in layouts)
  • Fixed Ticket #408 (bad form for renaming a group as an instructor)
  • Fixed Ticket #357 (bad form for adding members to a group as an instructor)

Next Steps

  • Analysis phase for the Messaging system

Roadblocks

  • Apparently it is midterms season?

Simon Lavigne-Giroux

Status

  • Nearly finished ticket #401
  • Worked on a report about the new marking scheme

Next Steps

  • Create tickets for each task in the new marking scheme
  • Work on the new feature

Roadblocks

  • I can’t work on the new feature before a final decision is made about the database model.

Farah Juma

Status

  • Met with Karen on Wednesday to discuss simple grade entry
  • Posted an update on the blog about the changes/implementation details we discussed
  • Split up the grade entry feature into smaller tasks and created these tickets for myself:
    Ticket #429: Create grade entry creation form
    Ticket #430: Create instructor views
    Ticket #431: Create TA view
    Ticket #432: Create student view

Next Steps

  • Start working on the implementation for simple grade entry.

Roadblocks

  • None this week

Gabriel Roy-Lortie

Status

  • Worked on the Flexible marking scheme design proposal
  • Did some research on more testing tools (cucumber, webrat) and replacement to fixtures (factory_girl looks promising — the same team that brought us shoulda)
  • Started to mingle with cucumber on my local version

Next steps

  • A post about “why fixtures aren’t our friends” and what I’ve learned on testing tools in general

Roadblocks

  • Next week is exam week

Written by melagaud

October 15th, 2009 at 4:42 pm

Posted in Status Reports

Real Users = Real Problems

with 2 comments

We’ve been using MarkUs for a few weeks now.  I’ve used it in my course (CSC209: Software Tools and Systems Programming), and everything was working remarkably well.  I was just starting to relax, when Diane Horton started setting up to mark her first assignment in our first year course, CSC108: Intro to Computer Programming.  Diane was initially very happy because the students had successfully formed groups, and there were no problems with their assignment submissions.  I was thrilled; 450 students had been able to submit assignments without any serious performance problems.

And then we started encountering bugs…

Infinite loops

The primary problem was that operations were running for a long time: 5 minutes or more.   We got Mike involved, and he found a bug that suggested an infinite loop.  Although Diane had set up her assignment to allow for 1 grace period, she did not assign students any grace credits.  The code that checks to see whether students were using or were allowed to use a grace credit was possibly hitting an infinite loop.  This could explain why we were seeing such a slowdown.  If the mongrels were kept busy with infinite loops, we might have a hard time getting other requests through.

Adding the grace credits to all the users and restarting the server didn’t seem to help anything.  We did finally decide on a workaround: change the assignment properties to “accept no late submissions”. That means Diane won’t be able to tell if anyone used a grace period for this assignment.

Timeouts when creating graders

We started looking further when the first potential solution turned out not to make much difference. Diane was in the process of adding graders to MarkUs, and was still finding that it took approximately 5 minutes for the page to timeout and another 5 minutes before she could do anything. (Not surprisingly, she was not a happy user.)

Mike could see the timeouts when we added a grrader in the log file. I think this timeout could be because when a grader or instructor is added, they get added to the authorization file for all the svn repos in the course. Rebuilding those authorization files has been a big performance problem on drproject, and I’m guessing that is what is causing the problem here.  We haven’t completely verified this at this point but Mike worked on a fix.  I’ll let him write up the fix.

At some point a TA was added to some, but not all, of the authz files, but the operation died before the TA was added to the database, so we couldn’t see the TA on any lists, but we also couldn’t re-add the TA.

Getting test results into MarkUs

Then we ran into a problem that was caused by poor design decisions.  We use the commit times to determine whether a student is using a grace or is being assessed a penalty for late submissions.  In CSC108, Diane wanted to check out all the repositories, run tests on them and check the results back in so the students can see the test results and the TAs can see the test results and use them to give the students a grade.

You’ve probably figured out the problem by now.  When head TA committed all the test results to the students’ repos, we wanted to include those files in the official “submission”.  So, we had to tell MarkUs to collect the submissions after the due date which triggered all of the grace day calculations, and all the students “used” their grace day.  Not only that, but we needed an extra step to “collect” the submission that contained the new results files.

The end of the story for CSC108

Unfortunately, despite Mike’s efforts (on a long weekend), we were still unable to guarantee to Diane that things would work well enough, so her TAs are marking the first assignment on paper.  The good news is that we think we have fixes for all but the last problem and will be working with our Alan to get the patches installed.

The two issues we have to take a serious look at are:

  1. Have we got the “right” solution for giving TAs authorization on the repos?
  2. What are we going to do about adding result files? (next blog post)

Written by Karen Reid

October 14th, 2009 at 11:07 am

Posted in Uncategorized

Logging

with 4 comments

Logging is not as nice as I expected it would be, there are different ways of doing it but i believe the only two relevant for us are these:

Option 1:  Add a new class and initiliaze the different log files at the beginning

Advantages:

  • We can separate the log levels as we please
  • Have different rotations for each log file

Disadvantages

  • Debugging messages must be logged manually, we must redirect the stdout to a file. An example of debuggin message is this one: User Load (0.2ms)   SELECT * FROM `users` WHERE (`users`.`id` = 222) LIMIT 1
  • To change the rotation or filename we would need to add methods and views to do it

Option 2: Configure the logger in the environment.rb file

Advantages:

  • We can divide the controller messages from the records, which might come in handy sometime.
  • It is configurable by the administrator without having to add new methods or views to MarkUs
  • It is very simple to do =)

Disadvantages

  • We can only separate the record messages from the controller ones.There would be no special file for errors or warnings

I would go for option 2 given that we can always grep the log file for what we want given that the formats used by the logger are quite clean:

Here is an example of how the log for the controller looks like: http://pastie.org/650165

And here is an example of how the log for the record looks like: http://pastie.org/650166

Written by fgarces

October 11th, 2009 at 2:44 am

Posted in Uncategorized

IRC Meeting Notes – October 9, 2009

without comments

Meeting has started at 12h00 and ended at 12h48

Markus issues in production

  • Reid told us: Sadly, MarkUs for our big 108 course is currently not well.
  • Mike has diagnosed the problem, and think there is a workaround, but the server is still really slow.
  • Whatever Alan did for the first restart didn’t seem to solve anything, so he has been asked to apply more force in killing the servers.

New marking Scheme name:

  • m_conley, Tara and Fernando does not like Ordinal
  • simonlg suggestions: Standard, Open marking scheme(referring to open concept, the less structured nature of the scheme), Free marking scheme
  • gabrl suggestions:
  • jerboaa suggestions: Point-scale, Flexible marking scheme
  • gabrl: Likes Point-scale, likes Flexible better than Open
  • reid: Likes Open, likes Flexible better than Open
  • simonlg: Likes Flexible, Not rubric
  • tclark: Likes Flexible better than Open
  • farahj: Likes Flexible

reid will take a poll amongst her colleagues, and get back to us.

We will go ahead and use Flexible marking scheme in our analysis, but when reid comes back with the poll results, we’ll “refectory” if need be.

Blog views

  • Tara asked if is there a way to see a listing of all the post titles on the blog instead of having to go on wp-admin.
  • fgarces answered: Yes, use the link Site feed
  • jerboaa suggested rss, but rss ony goes back until may 22nd

Iteration 1 today
3 tickets not completed yet

  • Ticket #416(Owner gabrl): Will try to do it this afternoon, If he can not, will adjust is to the next iteration
  • tlclark has adjusted her ticket to the next iteration.
  • simonlg has adjusted his ticket to the next iteration.

New Ticket
Ticket #423(Owner fgarces): Pretend that user.authenticate returns 0 if user/pass is incorrect
The method authenticate returns an error code when the user/password is wrong, thought that the server returned -1 when the user was wrong and self.authenticate was expecting for 1 or 2 if the user/password was wrong, but mike said that it is supposed to return 1 or 2, which is what self.authenticate is expecting, so at the end there was no real problem. user.rb has the constants defined – ranging from 0 to 6

Conclusion
Ticket #423: Rejected with a comment telling that something was wrongly understood.

Posts

  • We have been congratulated by G. Wilson about our blog activity!
  • Farah’s post posed some questions about the organization at the top level.
  • Farah’s post needs some Instructors to look at it, to make sure she has the mental model right.
  • Fara and reid will meet in person to talk about it.
  • Severin had some good comments, If anyone else has some thoughts or suggestions, please chime in on the blog.

Comment System

  • Tara hasn’t had much time to work on the comment system yet. She will work on that this weekend.

Automated Testing

  • Severin has been doing research, and will write up a post soon.
  • Severin and reid should probably meet sometime next week to talk about it.

Written by melagaud

October 9th, 2009 at 1:42 pm

Posted in Meetings,Minutes