Archive for the ‘Uncategorized’ Category
For our project at Ecole Centrale de Nantes, Christian, Benjamin and I had to meet several professors from the different CS courses that are taught our school during our two first years in order to understand how would Markus be used at Ecole Centrale de Nantes, and list the desired new functionality.
We met Guillaume Moreau, in charge of supervising Markus’s deployment at Centrale Nantes, and head professor of the Object course of third year (during which he would like to test Markus next year), Didier Lime, current head Professor of the first year CS course on algorithm and C programming, and last but not least, Vincent Tourre, head prof of this same course next year.
A few concerns and ideas where raised. Here are the three main ideas.
- Metrics: Guillaume Moreau likes to run metrics on students code for project: number of classes, name of this classes, number of methods, number of variables etc. These metrics are very useful to detect cheating: indeed, in more than 80% of the case, two groups who have exactly the same number of classes and methods just copied/pasted their code. Another type of metrics would be interesting, for third year students at Ecole Centrale de Nantes: metrics on the code itself, that analyses the code, and the use of some key word, and give automatic feedbacks on the code quality (http://pmd.sourceforge.net/).
- PDF export of the results: Didier Lime would like the students to be able to retrieve the results and comment on their code. A simple way to do this is to implement a pdf export of the results page, with the annotation in a PDF. Students could therefore keep all traces of their assignments on their computer.
- Possibility to upload reports & screenshots: Vincent Tourre was concerned about reports. Our first year course is about algorithm, and not code. Students have to write algorithm before starting to code anything, and write reports, with their algorithm. In order to use Markus to grade, we could ask students to only write plain text report. But, in second year, we also have to make the object schema. If students could upload reports and images, it would solve both of this problem (a bit like in review board).
We are working on this feature, so here is my first attempt at describing the requirements.
In some of the large classes, we might have quite a few TAs grading an assignment. One way to divide them up is to assign several TAs to each part of the assignment. For example if we had 12 TAs grading an assignment we might put 4 of them on part 1, 4 on part 2, and 4 on part 3. The only mechanism we have in MarkUs to map graders to parts of an assignment is through rubric elements. So the idea is to assign TAs specific rubric elements and then use some UI features to highlight the rubric elements when a TA opens a student submission.
Assigning rubric elements to a grader is separate from assigning groups to a grader. A grader might mark one rubric element for all submissions, or might mark all rubric elements for a subset of the submissions, or might mark a subset of the rubric elements for a subset of the submissions.
I imagine a UI similar to the mapping of graders to groups or students. In this case we would see a list of the rubric elements with checkboxes (?) and would use much the same mechanism to map the graders to the rubric elements.
If the instructor does not specify rubric elements for a grader, then the default should be that the grader will mark all rubric elements.
After rubric elements have been assigned to a grader, when a grader view a submission, it should be clear which rubric elements they are supposed to mark. I am imagining that these rubric elements would be expanded by default, and would also be coloured differently. The graders should still be able to assign marks for other rubric elements, we just want the ones that they have been assigned to work on to stand out more.
As always, feedback and comments are welcome. What am I missing?
The IRC log can be found at: http://www.cs.toronto.edu/~mconley/irc/markus/markus20100205_pg1.html
Brian Xu fixed three bugs:
512 – UI: Show the graders name in the graders drop down —- http://review.markusproject.org/r/370/
531 – Graders need a way of “un-setting” Criteria —- http://review.markusproject.org/r/369/
571 – Edit Student: “Cancel” button links to something wrong —- http://review.markusproject.org/r/371/
Bryan Shen closed: #393. multi-level folder not allowed. Bryan also plans to blog about some things he has learned from working on Markus.
Brian Xu and Bryan Shen plan to get together and discuss how they are going to complete the flexible grading scheme.
Farah plans to return to working on the grade entry. She will update the review board so we can make comments on the code. The functional tests are still in the works.
Joseph has completed 564 – Validate files with a path with a space and check configuration on startup —- http://review.markusproject.org/r/358/. He hopes to get some time next week to look at will_paginate.
jerboaa figure out that urgent issue that arrived on February 4th.
Mike, feel free to assign this bug to someone else. Everyone else, feel free to jump on it.
The Download Detailed Criteria produces a stack trace when grace days are allowed. It works when there are no grace days.
undefined method `find_by_rubric_criterion_id’ for #<Class:0xb6e475f8>
Application Trace | Framework Trace | Full Trace
Robert finished off the tests for the random Grader assignment. He plans to work on issue 569 – Investigate possible Grader Assignment problem by manually hammering on the grader assignments to see if they always update correctly onscreen.
He also plans to work on 559 – Allow Rubric Weightings to be 0 or negative. However, Karen would like to discuss how it should be done with Byron. Mike believes that Byron will be able to use the Flexible Criteria if we push hard enough.
Mike will be running next week’s meeting. Additionally, Karen would like to have a meeting on the reading week’s Friday Feb 19th.
- Fixed the functional tests! Yeah!
- Put the functional tests on markusproject.org homepage – though they act funny in an “off and on” sort of way…
- Reviewed some code
- Closed #530 (next/prev submission links in grader view no longer skip completed assignments in 0.6 – needs to be pushed to trunk)
- Wrote a blog post about closing #531: http://blog.markusproject.org/?p=1103
- Review more code as it comes in
- Turn the rest of Byron’s email into tickets (can’t believe I haven’t finished that yet)
- Take a look at some other high-priority fixes maybe to patch 0.6
- Brace myself for possible MarkUs support…CSC148 and CSC209 assignments due tomorrow.
- This is a heavy week and a half for me – that’s why I did most of my work over the weekend. Might not respond to email this weekend, and will be pretty bogged down until next Wednesday sometime.
- 1 post on Markus blog
- Finished #570 units test cases, change committed
- Fix some bugs on weekends .
- Job hunting, 1 graduate course that requires intensive reading of papers, 1 research project
- Completed first set of wireframes for the Assign Graders View.
- Got useful feedback from Karen and Farah during last night’s UI meeting.
- Start prototyping for the Group tab. That is, the view users use to create new groups and modify existing groups.
- Summarize new usability issues and ideas from last night’s UI meeting and write a blog post about it.
- Learning how to use a new application for prototyping. I’m slowly getting the hang of it!
- handling validate files with a space and checking configuration on startup is on reviewboard
- ticketed the paging bug where the page number and page size are not saved in a bookmark or when hitting back/forwards
- did rudimentary research on will_paginate – have not found anything so far
- finish up the codereview
- complete research on will_paginate
- conclude which method to approach the pagination problem after research is complete
- see reviewboard for the discussion that just arose about the function in PathHelper: http://review.markusproject.org/r/358/#comment923
- Working on #393, Reviewing code, learning Ruby and Shoulda.
- Solve more tickets.
- Not enough knowledge of Ruby, testing, and Markus’ structure.
- Work on implementing creating section and to adding students to them
- Started to look on how to migrate smoothly to Machinist
- Implementing the group members from same section limitation
- Improving the UI on the different parts implemented this week
- Working on tests for my fix for 510, otherwise Halted.
- Reviewed changes from team members as they come in.
- Find time to work on MarkUs.
- Many consecutive interviews [Google, Microsoft, Amazon, Facebook], midterms, and assignments all at once.
- Fixed a bug related to alphabetical pagination which was causing the alphabetical categories to get re-calculated each time the user switched between pages instead of only when the number of items per page is changed (and learned about session variables)
- Re-factored code to make the GradeEntryFormsController lighter by moving some things to the model instead
- Finished the unit tests for the grades table
- Dropped in on the UI meeting on Wednesday
- Finish functional tests for the grades table
- Student interface for grade entry forms
- None this week
Right now, MarkUs lets graders choose a mark from 0 – 4 to assign to a particular criterion on the rubric. However, MarkUs does not currently allow graders to remove those marks once they’ve been made. They can change the level, but they can’t flat out remove it. This problem is filed as ticket #531.
My approach is to simply add a link that allows graders to remove the mark. Simple as that. I’ve thrown together a quick and dirty UI mockup – see below:
It really might be that quick and easy. In fact, this problem might be ripe for one of my new teammates to sharpen their teeth on. It’ll be a good experience working with RJS, the Rails AJAX helpers, and the Grader View (which is one of my favourite parts of the entire app). So I’m going to assign this ticket to “nobody”, and see who bites (and who’s reading the blog!).
One of the functionality we would like to implement this semester is section management.
Here is the content of this post
- Group management : the groups should be formed of two or three students of the same section ;
- instructors need to be able to add the students section through the web interface or via the CSV file upload.
- the student list should have filter to sort students per section.
- instructors should be able to switch from two modes, in the assignment property tab:
- groups have to be formed with students from the same section.
- groups DO NOT have to be formed with students from the same section.
- when switching to this mode, the application should display a warning message if some of the students don’t have a section associated to them.
- students should see whether the groups have to be formed of students in the same section or not.
- when the mode is activated, inviters should not be able to invited students in another section than theirs.
- if some students don’t have a section, inviters with no section will be able to invited only students without any sections.
- Deadlines : Each section needs to be associated with one specific deadline ;
- this functionality can be enabled only if groups are limited to same-section students.
- when enable, instructors can specify one deadline per section in the assignment property view.
If deadlines are specified per sections, it is logical that submissions rules could be too. For the first implementation, we will disable submission rules.
This table is used to save all the sections. In the case of UofT, it should be from 2 to 3 sections. For Centrale Nantes (a French engineer school), 12 sections (groups).
We added a foreign key to sections. This is mostly used for students, but can also be used for Graders (we could then map graders to groups per sections).
There are a lot of options that can be added with the new section scheme.
This links assignments and section, to add a specific due date to each section for an assignment.
If you have any thought, please chime in !
The last couple of days I’ve been thinking of a way to do user authentication for MarkUs’ Web API. This is what I’ve come up so far. Feedback and suggestions are very welcome.
- Instructors and TAs will be able to get a “key” via MarkUs. When an instructor logs in the API key (a SHA2 512 bits in length) is displayed to him on the dashboard. TAs will see the key below the list of the assignments when they log in.
- The goal is that the private key never shows up in a script.
- In order to use this key for authentication from within a script the following steps are required:
- A MD5 hash of the private key has to be generated.
- This MD5 hash will be then encoded using Base 64 and the result will be used as a token and travels over the wire for each request.
- The token, generated as described above, will be sent to the MarkUs server by facilitating the HTTP header “Authorization”. I.e. each request sent to the MarkUs API has to include this header with the appropriate token.
- Since MarkUs knows about the private key and that the token is a Base 64 encoded MD5 digest of the key a matching user can be easily determined and the private key never has to be included in scripts on semi-private servers, where the scripts are run.
I was thinking that the Authorization HTTP header (which is otherwise used for Basic or Digest schemes) could have the following form:
Authorization: MarkusCustom MDk4ZjZiY2Q0NjIxZDM3M2NhZGU0ZTgzMjYyN2I0ZjYK
So far my thoughts. What do you think? Concerns? Am I on the wrong track? Thanks!
You will recognize this report. It is the complete marking scheme analysis in which I have added one Chapter. Chapter 3 details the development that has been done and the development left to do in order to deliver the complete flexible marking scheme feature.
By the way, eventhough it is published on our DrProject since December first, we forgot to let you know that our screencasts are done. Please follow this links to see theme:
As I’ve been working on the table view for grade entry last week and this week (there are still some kinks to sort out, more on that in my next post), I’ve been noticing some performance issues that I think we might want to deal with before we let instructors use this table. One thing that I needed to decide was whether or not each table cell should be saved right as the instructor enters the mark or if the instructor should enter a bunch of marks and then hit a “Save” button. I decided to go with the “Save” button option. My initial plan was to display the entire table on a single page. However, I’m finding that:
- It’s difficult to navigate
- This page can take a long time to load or save
Severin had suggested earlier that we could implement some sort of pagination. I really like this idea. Here’s what I’m thinking:
- Maybe we could implement pagination alphabetically, by last name (eg. At the bottom of the table, we could have links like “A-D”, “E-J”, etc.). This would help to reduce the number of rows that get displayed on the page at a time. We probably wouldn’t want to implement pagination using numbers because if an instructor wanted to enter grades for a particular student, he/she would have a hard time figuring out which page contains the desired student.
Any thoughts about this?
According to our QA cours requirement, we had to present a demo of the work done during the session. Our demo was split in two parts.
1- PowerPoint presentation. This is the occasion to practice your french
2- Demo online of the application and of some code
The whole presentation went very well, and last around 30 minutes.