MarkUs Blog

MarkUs Developers Blog About Their Project

Archive for the ‘Missing Features’ Category

Notes System: Initial Design

with 4 comments

At the code sprint, we tossed around some ideas for the “Messages between TAs and instructors” feature that Karen asked for. Today, I’ve been brainstorming what I think this feature should look like, functionally.

Feature goal

To allow small pieces of information to be left on various aspects of the course, pertaining to something specific, to alert others to the problems (or good things!), with a timestamp. These could potentially pertain to an assignment, a student, or a grouping, or we could limit these to just one of these types.

Example uses

If students are suspected to be cheating, then the grader could “flag” the assignment for the instructor by leaving a note. Since other graders can also see it, then they can see that this student was previously flagged for cheating. If the student is sick and gets an exemption on part of the assignment, then a note could be placed on the affected assignment for the grader to take that into account during marking.

Can anyone think of other potential uses of this feature?

Where to create these?

Now that we have some idea of how this could be used in a real course – it’s always key to keep the users in mind – we need to think about where in the work flow we would want to create these pieces of information.

While marking an assignment, we should be able to make notes. These would be noted against the grouping, since this encompasses assignments where the students work alone, as well as assignments where the students work together and keeps in mind the idea that one student having a problem can affect all students in the grouping. It also allows us to have notes go against a single object, since a grouping represents an assignment as well as the students working on it.

We should also be able to make a comment about a particular student in general or about a a particular student on an assignment, not from the marking view. Should we also be able to comment about an assignment? This could be incorporated into the viewing of the comments described below.

Replies

What about replies to a comment? Should we incorporate these into our feature? I think that we shouldn’t at this time, since the initial idea is for more one-off ideas, but I think that the model should be designed in a flexible enough way such that we can have multiple notes against the same object, which would allow for multiple notes to be left against an object and thus “replies” of a sort without complicating the model too much.

Where to view these?

I propose adding a new tab in the menu for Admins and Graders, with its text being the name we decide upon for this feature. In here, we can display all of the messages (paged in case there are a lot of them?) and have a link to add new ones as well. Would distinguishing the new comments made since last login from the previous ones be useful? You should be able to sort the view. You should also be able to filter it by assignment or by student ID (which would obviously return all comments left against that student, as well as comments left against any groupings it is a member of).

Model

At the code sprint meeting, someone suggested modifying the annotations model for these notes, but that idea was declined since we don’t want to relate these to the code. The conclusion was that we will need a new model to represent these ideas.

The pieces of information that we need to store are: the user who wrote the note, the timestamp from when it was written, the object ID that it is against, the type of object it is against (grouping, student, assignment, etc.), and the actual text of the note itself.

Naming

We need to consider how we’re going to name this feature. Several initial possibilities come to mind: Messages, Notes, and Comments. (I actually found all three of these used interchangeably in my notes from the Sunday code sprint meeting!)

  • I don’t like the term “Messages” because it implies that they are sent from one person to another, which these won’t be – they’ll be available for viewing by all instructors AND graders. I do, however, like its implication that these are being shared with other people.
  • The term “Comments” is too close to the ideas of commenting code and the overall comments on an assignment that are shared with the students. I think that a new term is in order to ensure that information is not left in this field that the students shouldn’t see.
  • My favourite is definitely the term “Notes” because it implies that these are small pieces of information, pertaining to something specific. It doesn’t carry the connotation that it is just between specific individuals, so it is much more fitting than “Messages”.

I am definitely open to suggestions on other names or further comments on the names that I have suggested.

Anything else?

Are there any other pieces related to this feature that I haven’t outlined? I look forward to hearing your ideas about the upcoming notes feature in MarkUs!

Written by Tara Clark

October 17th, 2009 at 10:47 pm

Posted in Notes System

Tagged with

Automated Testing

with 2 comments

The idea is to integrate some sort of testing infrastructure into MarkUs, so that students get more feedback about their code submitted for assignments. A possible scenario would be the following:

  1. A new assignment is up in MarkUs and students start to work on it
  2. An instructor has written tests for this assignment, which will be run on the code Students submit
  3. Students will get feedback about their code submitted according to the results of those test runs

The question is how to achieve this. I’ll refer to the server or infrastructure which will be executing tests only as test server. Here are some questions we are trying to find the answer for:

  1. Where and how will the tests on students code be run?
  2. Should they be run on a dedicated server? On several servers?
  3. How does MarkUs communicate with the test server?
  4. How to decouple MarkUs from the test server?
  5. Is it a concern if a student compromises the test server either maliciously or accidentally?
  6. What are the required security measures for the test server?
  7. How to satisfy as many programming languages and course models as possible?
  8. What will the test server or test infrastructure look like? Should tests for each MarkUs instance run in separated virtual machines?
  9. Should we use virtual machines? If yes, why? Why not?
  10. How can we secure the test server well enough so that no secret information will be revealed to students? Should the mail system be disabled, network/socket connections be monitored or disallowed? How to secure the test server for courses in which students learn process forking and socket programming?
  11. Will there be performance problems? How to deal with increased load close to assignment deadlines?
  12. Should the amount of test runs be limited for students? If so, how? e.g. 5 test runs per day or max number of test runs?
  13. How much information of tests results should be shown to the student? Number of successes/failures? Detailed test failure messages? Should descriptions of tests be displayed?

These are related projects to get some ideas from. If you know some more, please let me know!

I am looking for ideas and hints regarding automated testing systems, how to integrate them into MarkUs and why to use one approach rather then the other. Please feel free to chime in and comment to this post. Greatly appreciated!

Written by Severin

October 16th, 2009 at 11:00 am

New marking scheme proposal

with 11 comments

A PDF file is included in this blog post about the new marking scheme. It contains User Stories, functional and implementation details.

– Three implementations are discussed and we want to know what you think about them.
– We also need approval from Karen about the User Stories.

New marking scheme proposal (v1.1)

In answer to Karen’s comment, this is 2 more database scenarios  Scenario4And5.

We hope to discuss about the new marking scheme on the IRC meeting tomorrow.

– Ulaval Trio

Written by simonlg

October 16th, 2009 at 12:46 am

Posted in Missing Features

It has waited long enough

with 2 comments

With a catchy title like that, I was sure to get your attention 😉

Something is buggin’ me and fortunately for you (or not) I feel like sharing. We (the Laval’s trio) are working on the New marking scheme (Did you see that? Not yet… it’ll come). So we had a meeting last Monday night and another one today afternoon, always talking about the New marking scheme and the impacts it is going to have on any thinkable part of the application. Then, tonight I started typing my part of the report we’re preparing for the rest of the team. A nice little report from which everyone will have to the chance to get a good picture of what we intend to do, as well as giving everyone a chance to share ideas about our approach. My still-in-a-draft-state report counted 35 times the word new out of 899 words…

Bottom line is : The New marking scheme needs a name!

Since it is going to imply multiple criterion rated from zero to x, I was thinking about names like:

  • Ordinal marking scheme (personal favourite)
  • Numbered marking scheme
  • Scaled marking scheme
  • Integer marking scheme (ew!)
  • Max value marking scheme (even more ew!)
  • (or any combination of the above)

We kept saying, “when it’ll have a name…”, but I can’t stand writing tickets, user stories and acceptance tests that feature the label New marking scheme in them anymore. It has waited long enough!

Propositions are welcome. There’s probably an already defined way of calling such a marking scheme. I’m sure people with more pedagogical knowledge than me (Karen, I’m thinking about you here) can put us in the right direction.

Thanks for you attention folks.

Written by gabrielrl

October 7th, 2009 at 11:06 pm

Posted in Missing Features

FileRepository: An Implementation of AbstractRepository

without comments

When designing the repository library used by MarkUs (it is actually a small, separate library, which could be used by other Ruby software as well) we were planning to implement a file based pendant to SubversionRepository and MemoryRepository. The idea was to remove the Subversion dependency of MarkUs, but still have versioned submissions (by submissions I mean files students upload or submit for an assignment). Basically, FileRepository would be a simple version control system, with the absence of a client to this VCS. In our case, this client would be MarkUs.

One idea to implement FileRepository was to have directories inside each repository folder named by the string representing the date on which files have been checked in. I think we didn’t go any farther than that 🙂

The last couple of days I was looking into the Moin Moin. It’s a wiki implemented in Python which uses flat files to store wiki pages. It turns out that Moin uses directories named by the wiki page to put the actual wiki page into it as a flat file. So, inside the wiki page directory are kept several versions of the wiki page. I.e. that might be something similar we would have to implement for FileRepository.

Conclusion: If I were to implement FileRepository, I would have a look at Moin, Subversion and Git as well as the other obvious candidates. But remember, the implementation of FileRepository would most probably be something simpler than Subversion and co. So what Moin does could be closer to what we want for FileRepository. Maybe this post will help some future developer to get started on this.

Written by Severin

October 7th, 2009 at 2:28 pm