Archive for October, 2011
The team is pleased to announce MarkUs 0.11.0!
MarkUs 0.11.0 contains fixes made in 0.10.1 and allows instructors to hide assignments from students. This will be our last Rails 2 based release. We are working hard to make MarkUs Rails 3 ready 😉 Keep your eyes peeled for a MarkUs 1.0.0 release.
If you’d like to upgrade from 0.10.1 using a patch, make sure you run
$ bundle exec rake db:migrate RAILS_ENV=production .
Thanks for all participants making this release possible. Keep up the good work!
Here are the punchlines of our group (students from Centrale Nantes), working on the test framework of Markus.
Add corrections in the Specifications according to M.Magnin and B.Vialle comments
Add use cases in the Specifications (schema and details of the scopes) with Claire
Next Step :
Finalize the specifications
Sum up and complete the framework state of the art (in order to post in on the blog and do a report of what was done)
first version of the specifications completed
work on use cases (diagrams and textual descriptions) for the second version of the specifications finished.
Next Step :
Complete the second version of the specifications
Finalize and summarize the state of art (what works and what has to be implemeted), writing off a report and publishing it on the blog
After the establishment of the state of art, the aim is that each of us will focus on the code of a particular directory or file.
Try to dig and explore inside the existing code in Ruby.
I’ve followed the tutorial on Ruby indicated on the the wiki.
Next Step :
Understanding better what the concrete objectives in the Markus Project.
The current functionalities in the Markus Project are still vague in my mind.
Looking for what is working in the existing test framework.
Next Step :
My Markus installation is not working properly, cause I’ve some errors that don’t exist in the demo version.
Find what is wrong, repair it and look at the existing test framework, to have a precise state of what is already working and what is missing.
Working on the test’s state of the art with Guillaume.
Trying to figure out why Markus stopped working on my computer.
Next Step :
Understand how the test are working in Markus.
Try some tests and see where are the limits of Markus.
It seems Svn is not working anymore on my computer. : “no such file to load — svn/repos”
Trying to solve Database-system installation problems
Working on defining the specifications
Next Step :
Work and try Markus to understand the state of arts.
Define roles and specify files to work on.
Even the re-installation of Ubuntu didn’t solve my “E: Unable to locate package mysql” error.
The code sprint was a great success. Everyone made some good progress and it was fun to meet all the new students.
One of the plans for the fall is to fix as many of the issues left because of the switch to Rails 3 as possible, but we would also like to make some progress on some additional features.
Here are a couple of feature requests that have arisen recently:
Assigning Graders to Marks Spreadsheets
For assignments, we can map graders to groups and to rubric criteria, but we do not currently have a similar feature for the Marks Spreadsheets.
One use case is that we would like to use the spreadsheets to record lab marks, and each grader is responsible for a particular subset of the students. They may occasionally see students from other labs in their group and will want to be able to give them a grade, but for the most part will focus on their own students.
We could use the same interface for assigning graders to groups for assignments. There would be some UI work involved to figure out how to display the spreadsheet to the TAs.
Uploading Grades to a Rubric or Flexible Marking Scheme
We currently have a REST API that allows instructors to upload test results or to add new user by writing scripts that send HTTP requests to carry out these operations. The ability to perform these operations depends on knowing the MarkUs instance API key.
There have been several cases recently where instructors wanted the ability to upload at least partial marks for an assignment — often when some aspect of the work can be automatically graded. A REST API would make this possible.
One question is whether the current authentication mechanism is sufficiently secure when we are talking about modifying grades.
Another question is exactly what the interface will look like since each rubric has a different number of categories, different names, and different valid values.
- Meeting started at 3:15 p.m. (EDT) and ended at 3:50 p.m.
- Everyone seems to be making good progress. Luke is working on using render instead of redirect where it makes sense, Alex fixed false passing functional tests, Razvan continues to get a better understanding of Rails and will continue to work on his bug fix. Erik has also been working on bug fixes and is learning functional tests and how code reviews fit in the picture. Severin continues with his performance analysis of MarkUs.
- Good discussions on Review Board have been happening over the last week. Kudos to everyone participating.
- Developers should try to fix some more Rails 3 related MarkUs bug until next week
- Term projects will be up by next week. Everyone is free to get in touch with Karen in order to discuss details.
- Everyone did a great job on the thoroughness of status updates. Luke’s status was a bit thin on his progress this week. He should get back to the team in order to elaborate a bit more what he has been working on this week.
- Shipped hidden assignments. Work in on 0.11.x branch.
- Wrote first blog post about performance analysis
- Created a clustered setup of MarkUs on the server
machine (Apache reverse proxy & mongrel clusters)
- Code review and pushed review for resetting API key
which should make remote forms work.
- Run benchmark scripts and see if I can recreate the problem
we were seeing when 500+ student-repos were created and
- Blog about the above as well as try to look under the covers
what’s going on.
- Fixed bug #504. The add/edit student and the add/edit admin page links were broken. Also made test cases for it.
- Learned about how to add functional tests, as well as the review board process
- Worked on bug #395. The cancel buttons on most forms are different sizes than the submit buttons. I found a solution for this, but after discussion with Severin, I need to try finding an alternate solution with CSS.
- Other bugs: Find some new problems to work on
- Choose project: Pick the big project for the semester
- Working on form submission errors and best way to display full error messages.
- Discuss on mailing list best way to do this and submit a review.
- Still working through my first real bug/issue in the Markus Project
- Thinking about which term project I would like to be involved in
- Still working on learning rails when time permits
- Actually finish the issue I am currently looking at
- Take first steps towards starting on project for the term
- Reformatted all the functional tests to use proper assert statements
- Modified the grade entry forms controller to show proper error messages in order to pass required functional tests
- Added an error message in both languages to provide appropriate response in the grade entry forms controller
- Modified the new and edit pages of grade entry form to show error flash messages
- Continue working on issue #434 – “creating a new criteria fails”
I will be working on analyzing MarkUs’ performance this term. Here is what I have so far:
In order to look at MarkUs’ performance under various conditions I have the following lab setup comprising of a server machine and eight clients. MarkUs will be hosted on the server machine and its performance will be evaluated there. The eight clients will be used to “simulate” load on the server. Here is a picture illustrating the setup:
As illustrated, clients are connected via a network switch to the server. All machines are physical machines (i.e. not virtual machines). The server machine has 2 Gigabytes of RAM and is a Intel Core 2 machine. I.e. it has 2 cores at 1.86GHz clock frequency. It runs on Ubuntu LTS 10.04 and the PostgreSQL database runs on the same machine as well. All filesystems are local and are ext4 formatted. The specs of the client machines should be irrelevant for this analysis.
In order to get some data out of this setup I intend to proceed this analysis incrementally. One focus of this study is to look at MarkUs’ performance when there are 500+ students in a course. In fact, I hope this analysis will help us to come up with some recommendations in terms of the maximum number of students which should not be exceeded per MarkUs instance.
As a first step I will be trying to reproduce a potential performance problem of MarkUs when there are 500+ students in a course and some subset of them is submitting code simultaneously. In order to do so I will use a mongrel cluster on the server behind an Apache httpd reverse proxy (which is used at the University of Toronto and University of Waterloo). In order to simulate 500+ students submissions, I will be using benchmarking scripts in lib/tools of the MarkUs code base. Since MarkUs on Rails 3 is not quite ready yet, I will be using MarkUs 0.10.1 on Rails 2.
Once I was able to reproduce this performance problem, I will start to take a closer look at what might cause this. Questions I’d like to answer are:
- Is there a reproducible performance problem?
- When does performance start to degrade? At 500 students? 5000 students? Something else?
- How does a clustered mongrel setup compare to a setup using Phusion Passenger with respect to performance? Can we get some hard numbers on this?
- What is the cause of the performance breakdown (if any)? Is it one cause? A combination of many things?
- Is there a difference between students working in groups vs. students working alone setups?
- How can we alleviate potential performance problems?
Later throughout my analysis I’d also like to see if there is any difference in results when a DB snapshot of a former production system is being used.
Any other suggestions as to how to go about this? Am I missing something? What would you like to get out of this?
In order to figure out what the cause of potential performance problems of MarkUs is, I plan to use the following tools:
- Analyze response time in production logs with say Rails Analyzer in order to get some concrete numbers as to how response time behaves as the number of students and repositories increases.
- Use a Ruby profiler, for example Ruby Prof, in order to profile problematic code.
- Use a system profiler such as Oprofile in order to get a bigger picture as to where time is mostly spent system wide. In the database? File IO? MarkUs code (i.e. Ruby code)? Something else?
What else should I be using? Thoughts? I anticipate that it’s going to be challenging to go from the big picture – there is a performance problem – down to – what is causing it – since there are a lot of components involved (Subversion, PostgreSQL, Ruby, Rails, MarkUs).
What do you think? Let’s hear it 🙂