MarkUs Blog

MarkUs Developers Blog About Their Project

Archive for October, 2010

Meeting Minutes: October 29th, 2010

without comments

Log of the IRC meeting.

Kurtis and Hora

  • Hora’s review for Issue 117 has stalled, Mike will take a look at it later to get that going
  • Hora has been working on preparing the data for the grade distribution graph – Hora had a question about what kind of graph he should build first (to show distribution by grades or percentage), percentage was agreed upon
  • Hora should have a review in tomorrow
  • Kurtis added the graphing framework into git
  • Kurtis will use Hora’s code to work on TA grade distribution, and will also be working on TA progress reports (including both groupings and criteria)
  • Karen hopes for the graphing/reporting framework to be something that can be used and added to in January

Evan, Jiahui and Victor

  • The blog posts are very useful and informative, Karen will look over and comment on what is necessary
  • Jiahui is identifying bugs while developing testing setups
  • Evan and Jiahui will work on the framework for the rest of the term
  • Evan and Victor will work on the security/VM side for the rest of the term
  • Victor will create increasingly complex prototypes for easier reviewing/integration
  • MarkUs should trust the testing server whether it is virtual or a physical server

Misa and Vivien

  • Great GUI mock-ups by Vivien
  • Misa started implementing the marking state changes, may add another field (ex. remark_exists) in order to display both old and new results after a re-mark
  • Only one re-mark per assignment, student can continue editing the request until they submit it to the professor
  • Vivien started coding the GUI for re-mark settings, should be up on review board soon
  • Assignment will be un-released if the professor starts re-marking, to avoid conflicts
  • Need to make the distinction between saving and submiting a marking request clear
  • Need to be cautious when developing, will need some revised db schema

General Notes

  • More users are now using all corners of MarkUs, more issues are being discovered
  • Git’s issue tracker is lacking, no ‘proiority’ feature – Karen will look into a new tag to prioritize issues we need fixed by Christmas
  • Keep reviewing code! And don’t be afraid to bug Mike and Severin (and others) to review your code to keep everything rolling

Written by horatiu

October 29th, 2010 at 9:47 pm

Posted in Meetings,Minutes

Punchlines for Oct 28

without comments

Kurtis Schmidt

Status

  • Chose Bluff graphing library for MarkUs
  • Integrated Bluff graphing library into MarkUs
  • Created working branch with Hora
  • Began designing graphing framework

Next Steps

  • Generate the graphs

Roadblocks

  • Exams

Hora Hamalghi

Status:

  • still working through my review of issue 117
  • working on preparing grade distribution data for graphing – you should see a review of this soon!

Next steps:

  • finish up preparing grade distribution data
  • figure out how and where to display the grade distribution graphs

Roadblocks:

  • my review of issue 117 has stalled, waiting on some more comments with regard to testing

Evan Browning

Status:

  • Met with Victor, discussed testing framework, security, worked with benjamin’s examples
  • Wrote a blog post giving details on everything that still has to be done on the testing framwork
Next Steps:
  • Add parsing to the examples and write documentation for the examples
  • Fix critical bugs in the testing framework

Roadblocks:

  • Was low on time this week due to other classes

Misa Sakamoto

Status:

  • met up with Karen to discuss paper prototypes and feedback from Diane
  • talked with Paul Gries to hear his feedback on prototype
  • sharing Vivien’s branch and started implementing

Next Steps:

  • continue with implementing

Roadblocks:

  • was sick this week

Vivien Suen

Status

  • Met with Karen to show her our ideas and prototypes
  • Photoshopped together some screenshots of our ideas and posted to blog for feedback
  • Figured out how to share a branch between me and Misa and started implementing

Next Steps

  • Continue implementing and get something up on review board

Roadblocks

  • Midterms this week

Jiahui Xu

Status:

  • Working on issue 98
  • Setting up the work environment for Test Framework

Next Steps:

  • Pick a feature for the test framework from Victor’s blog post and start working on it

Roadblocks:

  • None

Victor Ivri

Status:

  • Nearing the end of the investigation stage, ready to implement prototypes.

Next Steps:

  • Implement prototypes.
  • Further research approach.

Roadblocks:

  • None.

Written by vivien

October 29th, 2010 at 12:40 am

Punchlines for Nov 2 (Centrale Nantes)

with one comment

We will code each Thursday afternoon in an open-space at Centrale Nantes. We will try to have as many students as possible available on IRC. Stay tuned !

Benjamin Vialle

Status:

  • Write up requirement specifications for the up-coming touch-screen module for MarkUs.
  • Draw some mock-ups for the new interface. (using Pencil)
  • Meet with an UI expert at Centrale Nantes
  • Pushed translated end-user documentation to the MarkUs wiki
  • Plan all punchlines on a graph
  • Work on InkSurvey’s source code and on MarkUs’ image-annotations written this summer by Anton
  • This blog post ūüôā

Next Steps:

  • Get feedback on requirement specifications of the new touch-screen feature from the Canadian team.
  • Start to code ‚Ķ

Roadblocks

  • Review Board alters patches produced by Git. This prevents us from downloading and testing diffs. It’s not fixed yet.

Clément Delafargue

Status:

  • Help other students with Prototype
  • Write the blog article on Javascript Frameworks
  • Got familiar with Anton’s code (learned mostly from comments in the code)

Next Steps:

  • Start to code

Roadblocks

  • None

Valentin Roger

Status:

  • Work with No√© and Benjamin on the requirement specifications
  • Draw some mock-ups of the new interface
  • Have a meeting with Vincent Tourre, UI expert at Centrale Nantes

Next Steps:

  • Start to code

Roadblocks

  • Still has some errors from his unfortunate chmod ūüėČ

J√©r√īme Gazel

Status:

  • Helped Cl√©ment Schiano on his work

Next Steps:

  • None for the moment

Roadblocks

  • None

Clément Schiano

Status:

  • Work with J√©r√īme on interpolating lines
  • Start to work with Cl√©ment D. and J√©r√īme on database tables (a way to record lines from the Tablet PC)
  • Write a blog post on findings

Next Steps:

  • ūüôā

Roadblocks

  • None

Noé Bedetti

Status:

  • Drew some mock-ups of the new interface

Next Steps:

  • Start to code

Roadblocks

  • None

Written by Benjamin Vialle

October 28th, 2010 at 7:11 pm

Automated Testing: Options to implement security

with 2 comments

Please comment and add to this post!

There are two ‘general’ ways to implement security in auto-testing: To use VMs or not to use VMs. This is the question. At first, the consensus was in favor of using a VM, while I wanted to prove that it will be possible to achieve the same level of security via existing tools available in Linux (and SE Linux), because it seemed like a more elegant solution at the time, plus it would avoid the runtime overhead of VMs.

What I’ve realized is that there are two problems to that approach: First, this would require some custom tailoring of security features to different/older distributions. A greater problem would be allowing maximum freedom for assignments such as operating-systems, networking, security – while retaining security. In the end, it seems, running a VM seems like the only feasible solution.

Here is the rough workflow of doing automated testing with a VM:

1) Setup (by hand):

  • Install the VM.
  • Set up a virtual hard drive (preferably on a separate hard disk, for speed) and networking.
  • Install the necessary programs on it (minimum: Ant, SSH server)
  • Set up SSH keys to allow automatic SSH from local machine to VM.
  • Save the image.

2) Automated testing (shell script):

  • Start a headless VM from the initial image.
  • FTP test scripts/files
  • SSH-execute the script, pipe output to a file.
  • FTP output file to local machine.
  • Lather-rinse-repeat for every test request. That is, each test will start from new image to avoid possibility of corruption.

Possible improvements on the basic process:

  • Parallelizing test runs. That is, not being limited to serially running VMs. Useful in batch-testing, and for more real-time-like results (imagine 100 students testing their assignments 10 minutes before deadline!)
  • Throttling resource use – not waiting indefinitely on busy-wait, and timely termination of fork-bombs and infinite recursion.

Possible VM products:

  • VMWare Server
  • Oracle VirtualBox
  • Parallels
  • Qemu

I have heard very good things about Oracle’s product, and of course VMWare. Further research required to test for speed (up-time from saved image, system resources, restart) and general compatibility (I think they should be equivalent in this regard.)

That’s it for now. Please leave comments and suggestions!

Written by victor

October 27th, 2010 at 2:53 pm

Automated Testing: What still needs to be done?

with one comment

Victor and I met on Saturday to work on the testing framework.  Here are some of the things that we came up with that we want to work on for the rest of the semester.  Victor will be posting a separate article on the security aspects of the testing framework.

Framework:

Benjamin and Diane did a great job during the summer of developing the test framework.  Most of the functionality is there, but there are still a few features that should be added and some bug fixes.

Features:

  • Implement other options for running tests.¬† Currently we allow a certain number of tokens per day.¬† We should have the option for allowing certain number of tokens in total and possibly to run tests at scheduled intervals (as is currently used a lot at U of T).
  • Allow students to see the test results that were run by their marker when they were graded.
  • Add a checkbox for “students can run tests” and, if unchecked, the “test” area of the student view will not show up.¬† (Right now the best we can do is set number of tokens to 0, which does prevent students from running tests, but gives the students the impression that they will be able to run tests until they try and it does not work).
  • Allow an option for tests to be run on all students as part of the collection process before grading.
  • Give instructors access to both raw output and parsed output.
  • Try to give better error feedback if ant fails for some reason (for example, if something has not been correctly installed).¬† See if we can differentiate when a build fails between an ant/framework problem and a problem due to the student’s code failing to build.¬† This might be tricky.

Bugs:

  • Fix an issue where students are not able to see their own test results after running a test.
  • Changing the number of tokens allowed for an assignment should update the number of tokens available for each group.
  • Former test results go away for a student if the number of tokens available reaches 0.¬† They should be still viewable.
  • The test framework should not fail to run if an instructor has not uploaded anything into one of the “test”, “lib” or “parse” folders.
  • Give a better name to the “collect and prepare tests” button for students, or eliminate the need for this button.

Examples:

We already have some sample testing setups using C, Java and Python that were created by Benjamin.  What still has to be done:

  • Develop a sample testing setup for Scheme.
  • Write documentation for all the examples, so that people can understand what is going on and make modifications.
  • Add parsing to the examples, as none of them include parsing at the moment.

It might be too optimistic to say that we will do everything on this list, as some things could take quite a bit of time.¬† We’d love to hear your thoughts on what things are important, anything that we missed that needs to be addressed, and any other ideas!

Written by Evan

October 24th, 2010 at 10:36 pm

Posted in Developer Essentials

Tagged with

Remark Request GUI Mock-up

with 2 comments

Misa and I have been thinking about ways to implement the remark request feature. These are a few screenshots of our initial design.

When a student wants to request a remark, he/she will first click on Request a Remark at the bottom of the source code. This will create a new tab ‘Remark’ on the right side where the student can make their case about a remark. Students can save, submit, or delete the request. Once a request has been submitted, it cannot be modified unless a student deletes and makes a new request. Instructor instructions (specified when they create an assignment) will also appear in this tab.

When a student submits a request, their assignment gets a new marking state ‘remark’ with a new icon in the instructor/TA view. Then the instructor/TA can start modifying the students grade.

This view was trickier to design because we want graders to be able to see the student’s justifications, their old marks, as well as new marks. For now, we have decided to keep the two tabbed sections and have old marks are highlighted in yellow while allowing the grader to click and assign new marks. (We still have to work out a way for instructors that use manual rubrics). The grader also has space to make his/her comments.

After the instructor completes and releases the student’s mark, the mark breakdown and summary for both the instructor and the student will show the changes that have been made.

Well, this is basically how the remark interface will work. I think there can be improvements, especially on the grader view, but we haven’t been able to think of an elegant way to show all the information at once. If anyone has any ideas or suggestions they’d like to add, please comment!

Written by vivien

October 24th, 2010 at 12:54 am

Meeting Minutes: October 22, 2010

without comments

Log of the IRC meeting can be found here

Round table

Kurtis and Hora

  • Worked through frameworks for graphing, presented top 2 (Highcharts and Bluff) for feedback
  • Bluff seems to be the best option since its open source and no jQuery dependencies
  • Look at Dina’s work for possible chunks of value
  • Karen wants for next week: get¬†a basic framework up for either a basic dashboard page with just some data

Evan, Jiahui, and Victor

  • Evan has more or less got the test framework up and running. There were some issues but they have been fixed
  • ant-contrib is required to run ant properly (see ReviewBoard)
  • Want to work on security, develop good examples for instructors to use, and the auto-generation of .build and .properties files
  • VMs and their limitations?
  • Write blog post soon to map out rest of term

Misa and Vivien

  • Drafted prototypes for possible UI configurations
  • Met with Diane last week for feedback
  • Diane has some issue with Markus in general. Her issues to be posted on github soon
  • Going to meet with more instructors for more feedback today
  • Write blog post soon with mock-ups for feedback from everyone

Written by vivien

October 22nd, 2010 at 2:34 pm

Posted in Meetings,Minutes

Punchlines – Oct’ 22, 2010

without comments

Here are the punchlines from the past week:

Horatiu:

Status:
– experimented with different graphing libraries
– working through my review for issue 117
Next steps:
Рchoose a graphing library (Bluff seems to be the favourite http://blog.markusproject.org/?p=2061#comments) and start working on the dashboard
– finish up issue 117
Roadblocks:
None!

Kurtis:

Status
– Experimented with graphing frameworks
– Write a blog post outlining best solutions
– Received feedback from community on framework choices
Next Steps
– Set up branch for Hora and Me to work with
– Integrate chosen framework into Markus
– Start generating graphs
Roadblocks
– None

Vivien:
Status:
– Thought about potential remark request UI/features and discussed these ideas with Misa
– Drafted some paper prototypes
– Met with Diane for some initial feedback
Next Steps:
– Meet with Karen to show her our current ideas
– Start coding/implementing!
Roadblocks
– None

Misa:
Status:
– Study the current interface to see what types of situations there are/could be
– Play around with randomly created data for scenarios
– Sketched up 4 paper prototypes prior to meeting with Vivien
– Met with Vivien to discuss paper prototype
– Got feedback from Diane Horton regarding the prototypes and functionality
Plan:
– Meet up with Karen possibly Friday afternoon to discuss findings
– Get at least one more professor’s feedback
– Start coding!
Roadblocks:
– none

Evan:
Status:
– Worked on image annotation re-use
– Got the test framework running on my computer
– Workend on fixing some issues with the test framework (and discovered more issues)

Next Steps:
– Keep polishing the testing framework
– Hopefully meet up with Victor to work sometime this week

Roadblocks:
– I made some changes to the Test Framework review but I cannot seem to be able to update the review request for it

Victor:
Status:
– Almost same as last week. Had many deadlines this week, so I could only read about RoR and git during my free time.
Next Steps:
– Investigate existing test framework structure.
– Meet with Evan to discuss project and split responsibilities.
– Start coding!!
Roadblocks:
– Didn’t have time to work on the project this week due to deadlines in other courses.

Jiahui:
Status:
– Still experiencing with Test Framework
– Contacted Evan and he said there might be an error on the server side of the Test Framework, and he would check
Next Steps:
РStart reading the code for the test framework, and found out where the error is

Written by victor

October 21st, 2010 at 10:28 pm

Posted in Status Reports

(Belated) Meeting Minutes Oct 15, 2010

without comments

Log of the IRC meeting can be found here.

Agenda:

– Status report.

– Remind everyone to prototype rather than plan too much.

– Remind everyone to review items on Review Board.

Status report:

– Current usage of MarkUs in UofT: 12 TAs marking 310 assignments. New grader interface seems to be working well. (Karen Reid)

– Hora & Kurtis: Testing different graphs. Will post test results on blog.

– Need to create separate view to see progress of grading.

– Testing group: Getting stuck, will post issue to email board.

– Assignment collection not working properly. Agreement that it needs to be simplified when no pdfs need to be converted.

– Karen’s note about workflow: Better prototype and discuss, than spend long time planning. Need to see results at this point.

Written by victor

October 20th, 2010 at 11:54 pm

Data Display and Graphing Poll

with 8 comments

Hora and I have been experimenting with a number of possible options for graphing data in MarkUs. We have narrowed down our options to two javascript frameworks and are going to use this blog post to solicit input from the Markus community.

Highcharts

Highcharts is a commercial javascript graphing library.  However, they have released their software under a creative commons license for use in non-commercial projects.  Additionally, lazy_high_charts is a rails plugin which allows for simple integration of highcharts into rails projects.

Samples

Pros

  • Thanks to lazy high charts, most of the programming can be done in the controller. ¬†This leads to cleaner code.
  • Graphs are very attractive, animations are professional.
  • Dynamically hide single items in a chart. Example, don’t show one of the TAs.
  • Lots of options for creating charts
  • Great documentation

Cons

  • Built on JQuery. ¬†Markus uses Prototype which can have conflicts with JQuery. ¬†In the few tests we’ve done several javascript errors were reported.
  • As a commercial product, we are unsure about the future possibilities for use. ¬†Licenses can change.

Bluff

Bluff is a JavaScript port of the Gruff graphing library for Ruby.  It is pure javascript and has no conflicting dependencies.

Samples

Pros

  • Small javascript files, with no dependence on JQuery.
  • Easy to generate graphs quickly.
  • MIT License, so no conflicts with Markus.
  • Simple, attractive graphs.

Cons

  • Code must be written in Javascript. ¬†This means that most of the work will be done in the view, which creates ugly code. ¬†If we choose this path, a set of ruby helper methods (or a rubygem) could be created to clean up the code.
  • More simplistic than Highcharts.
  • API exists, but doesn’t provide a lot of examples.

Conclusions

Both graphing methods work well.  Highcharts produces a cleaner code base but has some implementation issues and is (sort of) proprietary.  Bluff is completely open source but will involve embedding javascript into views (or partials).  If anyone has opinions either way please sound off in the comments.

Written by kurtis

October 17th, 2010 at 10:37 pm