Archive for October, 2010
Log of the IRC meeting.
Kurtis and Hora
- Hora’s review for Issue 117 has stalled, Mike will take a look at it later to get that going
- Hora has been working on preparing the data for the grade distribution graph – Hora had a question about what kind of graph he should build first (to show distribution by grades or percentage), percentage was agreed upon
- Hora should have a review in tomorrow
- Kurtis added the graphing framework into git
- Kurtis will use Hora’s code to work on TA grade distribution, and will also be working on TA progress reports (including both groupings and criteria)
- Karen hopes for the graphing/reporting framework to be something that can be used and added to in January
Evan, Jiahui and Victor
- The blog posts are very useful and informative, Karen will look over and comment on what is necessary
- Jiahui is identifying bugs while developing testing setups
- Evan and Jiahui will work on the framework for the rest of the term
- Evan and Victor will work on the security/VM side for the rest of the term
- Victor will create increasingly complex prototypes for easier reviewing/integration
- MarkUs should trust the testing server whether it is virtual or a physical server
Misa and Vivien
- Great GUI mock-ups by Vivien
- Misa started implementing the marking state changes, may add another field (ex. remark_exists) in order to display both old and new results after a re-mark
- Only one re-mark per assignment, student can continue editing the request until they submit it to the professor
- Vivien started coding the GUI for re-mark settings, should be up on review board soon
- Assignment will be un-released if the professor starts re-marking, to avoid conflicts
- Need to make the distinction between saving and submiting a marking request clear
- Need to be cautious when developing, will need some revised db schema
- More users are now using all corners of MarkUs, more issues are being discovered
- Git’s issue tracker is lacking, no ‘proiority’ feature – Karen will look into a new tag to prioritize issues we need fixed by Christmas
- Keep reviewing code! And don’t be afraid to bug Mike and Severin (and others) to review your code to keep everything rolling
- Chose Bluff graphing library for MarkUs
- Integrated Bluff graphing library into MarkUs
- Created working branch with Hora
- Began designing graphing framework
- Generate the graphs
- still working through my review of issue 117
- working on preparing grade distribution data for graphing – you should see a review of this soon!
- finish up preparing grade distribution data
- figure out how and where to display the grade distribution graphs
- my review of issue 117 has stalled, waiting on some more comments with regard to testing
- Met with Victor, discussed testing framework, security, worked with benjamin’s examples
- Wrote a blog post giving details on everything that still has to be done on the testing framwork
- Add parsing to the examples and write documentation for the examples
- Fix critical bugs in the testing framework
- Was low on time this week due to other classes
- met up with Karen to discuss paper prototypes and feedback from Diane
- talked with Paul Gries to hear his feedback on prototype
- sharing Vivien’s branch and started implementing
- continue with implementing
- was sick this week
- Met with Karen to show her our ideas and prototypes
- Photoshopped together some screenshots of our ideas and posted to blog for feedback
- Figured out how to share a branch between me and Misa and started implementing
- Continue implementing and get something up on review board
- Midterms this week
- Working on issue 98
- Setting up the work environment for Test Framework
- Pick a feature for the test framework from Victor’s blog post and start working on it
- Nearing the end of the investigation stage, ready to implement prototypes.
- Implement prototypes.
- Further research approach.
We will code each Thursday afternoon in an open-space at Centrale Nantes. We will try to have as many students as possible available on IRC. Stay tuned !
- Write up requirement specifications for the up-coming touch-screen module for MarkUs.
- Draw some mock-ups for the new interface. (using Pencil)
- Meet with an UI expert at Centrale Nantes
- Pushed translated end-user documentation to the MarkUs wiki
- Plan all punchlines on a graph
- Work on InkSurvey’s source code and on MarkUs’ image-annotations written this summer by Anton
- This blog post 🙂
- Get feedback on requirement specifications of the new touch-screen feature from the Canadian team.
- Start to code …
- Review Board alters patches produced by Git. This prevents us from downloading and testing diffs. It’s not fixed yet.
- Help other students with Prototype
- Got familiar with Anton’s code (learned mostly from comments in the code)
- Start to code
- Work with Noé and Benjamin on the requirement specifications
- Draw some mock-ups of the new interface
- Have a meeting with Vincent Tourre, UI expert at Centrale Nantes
- Start to code
- Still has some errors from his unfortunate chmod 😉
- Helped Clément Schiano on his work
- None for the moment
- Work with Jérôme on interpolating lines
- Start to work with Clément D. and Jérôme on database tables (a way to record lines from the Tablet PC)
- Write a blog post on findings
- Drew some mock-ups of the new interface
- Start to code
Please comment and add to this post!
There are two ‘general’ ways to implement security in auto-testing: To use VMs or not to use VMs. This is the question. At first, the consensus was in favor of using a VM, while I wanted to prove that it will be possible to achieve the same level of security via existing tools available in Linux (and SE Linux), because it seemed like a more elegant solution at the time, plus it would avoid the runtime overhead of VMs.
What I’ve realized is that there are two problems to that approach: First, this would require some custom tailoring of security features to different/older distributions. A greater problem would be allowing maximum freedom for assignments such as operating-systems, networking, security – while retaining security. In the end, it seems, running a VM seems like the only feasible solution.
Here is the rough workflow of doing automated testing with a VM:
1) Setup (by hand):
- Install the VM.
- Set up a virtual hard drive (preferably on a separate hard disk, for speed) and networking.
- Install the necessary programs on it (minimum: Ant, SSH server)
- Set up SSH keys to allow automatic SSH from local machine to VM.
- Save the image.
2) Automated testing (shell script):
- Start a headless VM from the initial image.
- FTP test scripts/files
- SSH-execute the script, pipe output to a file.
- FTP output file to local machine.
- Lather-rinse-repeat for every test request. That is, each test will start from new image to avoid possibility of corruption.
Possible improvements on the basic process:
- Parallelizing test runs. That is, not being limited to serially running VMs. Useful in batch-testing, and for more real-time-like results (imagine 100 students testing their assignments 10 minutes before deadline!)
- Throttling resource use – not waiting indefinitely on busy-wait, and timely termination of fork-bombs and infinite recursion.
Possible VM products:
- VMWare Server
- Oracle VirtualBox
I have heard very good things about Oracle’s product, and of course VMWare. Further research required to test for speed (up-time from saved image, system resources, restart) and general compatibility (I think they should be equivalent in this regard.)
That’s it for now. Please leave comments and suggestions!
Victor and I met on Saturday to work on the testing framework. Here are some of the things that we came up with that we want to work on for the rest of the semester. Victor will be posting a separate article on the security aspects of the testing framework.
Benjamin and Diane did a great job during the summer of developing the test framework. Most of the functionality is there, but there are still a few features that should be added and some bug fixes.
- Implement other options for running tests. Currently we allow a certain number of tokens per day. We should have the option for allowing certain number of tokens in total and possibly to run tests at scheduled intervals (as is currently used a lot at U of T).
- Allow students to see the test results that were run by their marker when they were graded.
- Add a checkbox for “students can run tests” and, if unchecked, the “test” area of the student view will not show up. (Right now the best we can do is set number of tokens to 0, which does prevent students from running tests, but gives the students the impression that they will be able to run tests until they try and it does not work).
- Allow an option for tests to be run on all students as part of the collection process before grading.
- Give instructors access to both raw output and parsed output.
- Try to give better error feedback if ant fails for some reason (for example, if something has not been correctly installed). See if we can differentiate when a build fails between an ant/framework problem and a problem due to the student’s code failing to build. This might be tricky.
- Fix an issue where students are not able to see their own test results after running a test.
- Changing the number of tokens allowed for an assignment should update the number of tokens available for each group.
- Former test results go away for a student if the number of tokens available reaches 0. They should be still viewable.
- The test framework should not fail to run if an instructor has not uploaded anything into one of the “test”, “lib” or “parse” folders.
- Give a better name to the “collect and prepare tests” button for students, or eliminate the need for this button.
We already have some sample testing setups using C, Java and Python that were created by Benjamin. What still has to be done:
- Develop a sample testing setup for Scheme.
- Write documentation for all the examples, so that people can understand what is going on and make modifications.
- Add parsing to the examples, as none of them include parsing at the moment.
It might be too optimistic to say that we will do everything on this list, as some things could take quite a bit of time. We’d love to hear your thoughts on what things are important, anything that we missed that needs to be addressed, and any other ideas!
Misa and I have been thinking about ways to implement the remark request feature. These are a few screenshots of our initial design.
When a student wants to request a remark, he/she will first click on Request a Remark at the bottom of the source code. This will create a new tab ‘Remark’ on the right side where the student can make their case about a remark. Students can save, submit, or delete the request. Once a request has been submitted, it cannot be modified unless a student deletes and makes a new request. Instructor instructions (specified when they create an assignment) will also appear in this tab.
When a student submits a request, their assignment gets a new marking state ‘remark’ with a new icon in the instructor/TA view. Then the instructor/TA can start modifying the students grade.
This view was trickier to design because we want graders to be able to see the student’s justifications, their old marks, as well as new marks. For now, we have decided to keep the two tabbed sections and have old marks are highlighted in yellow while allowing the grader to click and assign new marks. (We still have to work out a way for instructors that use manual rubrics). The grader also has space to make his/her comments.
After the instructor completes and releases the student’s mark, the mark breakdown and summary for both the instructor and the student will show the changes that have been made.
Well, this is basically how the remark interface will work. I think there can be improvements, especially on the grader view, but we haven’t been able to think of an elegant way to show all the information at once. If anyone has any ideas or suggestions they’d like to add, please comment!
Log of the IRC meeting can be found here
Kurtis and Hora
- Worked through frameworks for graphing, presented top 2 (Highcharts and Bluff) for feedback
- Bluff seems to be the best option since its open source and no jQuery dependencies
- Look at Dina’s work for possible chunks of value
- Karen wants for next week: get a basic framework up for either a basic dashboard page with just some data
Evan, Jiahui, and Victor
- Evan has more or less got the test framework up and running. There were some issues but they have been fixed
- ant-contrib is required to run ant properly (see ReviewBoard)
- Want to work on security, develop good examples for instructors to use, and the auto-generation of .build and .properties files
- VMs and their limitations?
- Write blog post soon to map out rest of term
Misa and Vivien
- Drafted prototypes for possible UI configurations
- Met with Diane last week for feedback
- Diane has some issue with Markus in general. Her issues to be posted on github soon
- Going to meet with more instructors for more feedback today
- Write blog post soon with mock-ups for feedback from everyone
Here are the punchlines from the past week:
– Worked on image annotation re-use
– Got the test framework running on my computer
– Workend on fixing some issues with the test framework (and discovered more issues)
– Keep polishing the testing framework
– Hopefully meet up with Victor to work sometime this week
– I made some changes to the Test Framework review but I cannot seem to be able to update the review request for it
Log of the IRC meeting can be found here.
– Status report.
– Remind everyone to prototype rather than plan too much.
– Remind everyone to review items on Review Board.
– Current usage of MarkUs in UofT: 12 TAs marking 310 assignments. New grader interface seems to be working well. (Karen Reid)
– Hora & Kurtis: Testing different graphs. Will post test results on blog.
– Need to create separate view to see progress of grading.
– Testing group: Getting stuck, will post issue to email board.
– Assignment collection not working properly. Agreement that it needs to be simplified when no pdfs need to be converted.
– Karen’s note about workflow: Better prototype and discuss, than spend long time planning. Need to see results at this point.
- Thanks to lazy high charts, most of the programming can be done in the controller. This leads to cleaner code.
- Graphs are very attractive, animations are professional.
- Dynamically hide single items in a chart. Example, don’t show one of the TAs.
- Lots of options for creating charts
- Great documentation
- As a commercial product, we are unsure about the future possibilities for use. Licenses can change.
- Easy to generate graphs quickly.
- MIT License, so no conflicts with Markus.
- Simple, attractive graphs.
- More simplistic than Highcharts.
- API exists, but doesn’t provide a lot of examples.