MarkUs Blog

MarkUs Developers Blog About Their Project

Archive for November, 2011

Markus UCOSP Fall 2011 – Status Week 9

without comments

Severin

Status:

  •  I think a certain slowdown of MarkUs can be confirmed at this point. When many students log on to MarkUs the first time and try to submit files for assignments simultaneously a slowdown can be reliably reproduced.
  • Wrote analysis script for client logs in order to get concrete numbers on request times (these are still ballpark numbers, though are averaged over ~700 samples each).
  • Ran many different experiments with regards to student submission simulation and analyzed results.
  • Blogged about first preliminary results[1]
  • Updated review with new scripts[2]
  • Created Git repo on Github with results I’ve collected so far[3]

Next Steps:

  •   Run more experiments and continue to blog about it

Roadblocks:

  •   None

[1] http://blog.markusproject.org/?p=3280
[2] http://review.markusproject.org/r/1128/
[3] https://github.com/jerboaa/markus-benchmarking-results

Erik

Status:
  • A bit busy with assignments and marking this week
  • Finally resolved the UJS issue from last week, and got most of the groups page working again. Waiting for a review (id 1131).
Next Steps:
  • Finish fixing the rest of the groups page (“Add new” button and individual table entry buttons are still broken)
Roadblocks:
  • None

Alex

Status:

  • Fixed #426. Routing fix for all file uploads, including regression tests.
  • Created a better fix for #284. Uploading Admin files is possible in multiple different encoding formats which are all converted to Unicode before being added to the database.

Next Steps:

  • Create regression tests for the fix created to #426.
  • Work on new found issues #583 and #584. Missing view code in graders and groups for downloading/uploading files.

Roadblocks:

  • None.

Luke

Status:

  • Just did a quick review of Razvan’s small change
  • Banged head against wall regarding submission rules
  • Wrote blog on render vs. redirect_to

Next Steps:

Roadblocks:

  • The way submission rules are created is quite complicated and I don’t fully understand it yet. This is causing me issues because I’m getting errors and I don’t know what caused them.

Razvan

Status:

  • Submitted a review request for issue 580
  • Got issue 560 into a working state, where the only issue is a deprecated remote_function call
  • Starting looking into issue 527/528

Next Steps:

  • Continue working on issues 527/528
  • Submit changes for issue 580

Roadblocks:

  • Still stuck with how to replace call to remote_function in rails 3
  • The issue given before, 567, seems to have already been fixed so wanted to clear up if this needs to be addressed

Written by akrassikov

November 16th, 2011 at 2:09 pm

Posted in Uncategorized

MarkUs Performance Analysis (3): First Results

with 2 comments

This is a follow-up post of my previous two, MarkUs Performance Analysis (1) and Markus Performance Analysis (2). Please have a look at them as they detail the set-up of the lab machines I’ve been using as well as which scripts have been used and how to use them. Also note that I’ve added a couple of additional scripts in order to make it easier to kick off load test runs. More on the load testing scripts are in this review request. At this point I am able to report first results. If you are interested in the gory details, please have a look at this Git repo which contains all raw logs in addition to the spreadsheets and other things I’ve produced while conducting experiments.

Results

Here is a table with first results of various experiments I’ve conducted so far. A “Runner” is one call to ./post_submissions.sh. Each call to ./post_submissions.sh in turn generates 7 requests to the MarkUs server. Unless otherwise noted, results are for a clean MarkUs installation (no Subversion repositories exist). Note that the timing info is the elapsed time as reported by /usr/bin/time for each curl call. Results listed are averages over number of students (one student is one sample).

Observations

Here are a few observations I’ve made during my experiments.

  1. Both, mongrel based and Passenger based MarkUs set-ups seem to be IO bound under load on a clean MarkUs instance (load averages > 2; up to 17’ish on a dual core server machine). Clean MarkUs instance means no Subversion repositories exist prior to each experiment.
  2. Memory does not seem to be an issue. 2 GB of RAM was no problem for 24 mongrels.
  3. Top reports around 40% IO waiting when experiments are run. The question is where are these IO requests coming from?
  4. When an experiment runs, Ruby processes show up at the top of the list of “top” (for both, mongrel and Passenger setups). If I add the WCHAN field in top, the sleep function of these processes seem to be filesytem related or scheduler related. Note the filesystem of the server is ext4. In particular, the most prominent “sleep functions” are jbd2_log, synch_page, get_write and blk_dev_is.
  5. I’ve used oprofile in order to profile the entire system when an experiment is run. Here is an example of opreport outputof one run. Not surprisingly about 60% of the time the CPU was not halted a Ruby process was running (libruby.so.1.8.7). Note that Subversion and PostgreSQL related percentages are negligible. I’m not sure why. Does anybody have thoughts on this?
  6. A second run of ./run-load-tests.sh with Subversion repositories already existing and submissions resulting in conflicts (i.e. no SVN IO) are significantly faster (~3 times faster). See last experiment in table.
  7. Passenger based setups seem to use 6 ruby processes. Seems to be similar to what could be achieved via an Apache reverse proxy with a cluster of 6 mongrels. Perhaps this is different for non-IO heavy workloads.

Conclusions?

So what does all of this mean? Good question. Here are my thoughs:

  1. I think the main causes of heavy IO and, hence, sources of slowdown are request number 5 (where Subversion repositories get created) and request number 7 (where file submissions are happening and files are stored in Subversion repositories).
  2. There does not seem to be a significant difference between Mongrel and Passenger based set-ups (at least for the IO heavy experiment).
  3. Something else?

Now I’d be interested in your input. How would you interpret the data? What additional experiments should I run? Did I make a mistake somewhere? Please leave your feedback in the comments. Thanks!

Written by Severin

November 12th, 2011 at 1:59 pm

Posted in Uncategorized

Punchlines – 11 november 2011

without comments

Here are the punchlines of our group (students from Centrale Nantes), working on the test framework of Markus.

Charline

Status :

I have applied Benjamin’s patch Solving some routing problems to have a functional version of the framework

Next Step :

Focusing on routing issues

Writing a detail state of the art

Claire

Status :

Keeping on working on the state of art of the student view with Charline Trying to solve Markus bugs on the test framework in order to describe what is working (applying Bertrand & Guillaume patch)

Next Step :

complete user documentation for the test framework code to fix bugs linked to the test framework

Roadblocks :

difficulties to test what is working on my computer due to rails update

Benjamin

Status :

I have applied unsucceesfully the patch

think-tank with the group about the framework

Next Step :

Writing the state of the current project

Roadblocks :

I’ve got to have a look why the patch is not running

Guillaume

Status :

Listing other routing problems

Have a look at ant files (build.properties and build.xml)

Next Step :

Solve routing problems

Create ant files for an ALGPR assignment

Bertrand

Status :

Patched my files with Benjamin_V’s patch

testing my Markus branch with the new test

Working on routing issues

Next Step :

Carry on trying to solve routing problems

Testing Markus when we can use it without routing bugs

Roadblocks :

Ruby on Rails ^^

Karim

Status :

Working on the state of art of the test framework from Grader view.

Putting on the latest modifications from github and installing the patch.

Next Step :

Continue to work on the state of art of the test framework.

Written by gbugnet

November 11th, 2011 at 1:32 pm

Posted in Uncategorized

Markus UCOSP Fall 2011 – Meeting Minutes, Nov 9

without comments

Meeting started at 3:15 p.m. and ended at 3:55 p.m.

  • Luke got a job offer from EA. Due to interviews and travelling he wasn’t able to work on MarkUs this week. He will double up for the next.
  • Erik and Razvan are having trouble with the way Rails 3 does JavaScript. It was suggested to look for help on Ruby on Rails mailing lists and IRC channels.
  • Razvan suffered a hard drive failure and had to reinstall MarkUs.
  • Alex worked on broken routes of CSV uploads due to the Rails 3 migration. He managed to get a nice regression test in for this. He also discovered that some parts of MarkUs controller code are not exposed in views. He’ll have a more closer look and will file issues.
  • Severin didn’t get too much work done on the performance analysis front. He was able to run some variations of the load tests, but didn’t finish analysis. He should have more to report next week.
  • Jordan tried to get the test framework working, but feels he is going in circles. It was suggested to short-circuit with the French students which are also working on the test framework.

Written by Severin

November 9th, 2011 at 11:54 pm

Posted in Uncategorized

Markus UCOSP Fall 2011 – Status Week 8

without comments

Luke

Status:

  • Interview and vacation in Montreal this week so I didn’t get anything done

Next Steps:

  • Finish Submission Rules

Roadblocks:

  • None

Erik

Status:

  • Fixed some raw html output on the student group creation page
  • Continuing work on the admin group creation page. Got the page working using form_remote_tag, (which no longer exists in rails 3), so tried to rewrite it correctly
  • Got hung up with trying to get unobtrusive javascript (UJS) working correctly.

Next Steps:

Roadblocks:

  • Very much stuck with the UJS issue. I have found a large number of resources on google explaining how it works, but I am still having no success with it.

 

Razvan

Status:

  • Submitted a blog post about “Link vs Button” use
  • Fixed part of the JS problem with issue 560
  • Reinstalled Markus from scratch after hard drive failure

Next Steps:

  • Finally submit all the changes for issue 560

Roadblocks:

  • Still stuck with how to replace call to remote_function in rails 3

 

Alex

Status:

  • Created a separate fix for all CSV upload routing errors, not just students and TAs.

Next Steps:

  • Finish up some missing regression tests for the CSV upload routing fix.
  • Create support for multiple encoding formats for uploading files.

Roadblocks:

  • Figuring out how to make regression tests for the CSV routing error took much longer than expected.

Severin

Status:

  • Added timing related info to post_submissions.sh script in order to have something to work with in the analysis.
  • Ran the load simulation on 400 students, 800 students (2 and 4 runners per client each; i.e. 16 and 32 concurrent requests).
  • Result logs have been kept.
  • Posted an update to the review with the new script.
  • Didn’t get to analyzing numbers yet.
  • A bit of code review.

Next Steps:

  • Run more simulations (vary the amount of mongrels, set up passenger, etc.)
  • Analyze the numbers.

Roadblocks:

  •  None

Written by Erik

November 9th, 2011 at 2:17 pm

Posted in Uncategorized

Button VS Link

without comments

During the transition to rails 3 much of the Markus application has be looked over in fine detail in order to ensure continued functionality and adhere to the correct standards and restful design architecture that came with this upgrade. One very common decision that is bound to arise during this process is the ever present design choice of using links or buttons for navigation and input from users.

Whats the difference?
In the overall scheme of things, given enough tweaking, buttons and links can be used interchangeably, but by no means does this mean they should be used as such. Links are by design lightweight and should be used for general navigation in between pages. Buttons on the other hand are designed to represent actions that will cause some change of information (e.g. submitting a form) and thus are generally not as lightweight as their link counterpart.

Why does it really matter?
Making everything a button has the immediate negative effect that pages become more complex very quickly due to the button’s greater overhead. Making everything a link has the immediate effect of creating a very large security issue in terms of privacy of information and integrity of a server’s data for a plethora of reasons. First of all, giving a link any request method other than ‘get’ can break when a user tries to open the link in a new tab/window (because browsers only copy the url, not the request method that goes with it). Also many applications such as web spiders and applications designed to pre-fetch data access all the links on a given page, and if those links had a dynamic effect (such as deleting something) then this can quickly break the integrity of the data that the web server is storing.

Conclusion
When the question comes up of whether to use a link or button on any given page there is almost guaranteed to be a “correct” answer. If only navigation is involved, and the user is being moved throughout the information space of the server without actually modifying it in any way, then a link with the get method should suffice. If the user is in any way changing information by either modifying what already exists, creating something new, or destroying something that already exists, then a button should be used. To quote Mike Dierken, “keep links safe – just say no to ‘links via post'”.

Written by Razvan

November 8th, 2011 at 1:10 pm

Posted in Uncategorized

Punchlines – 4 november 2011

without comments

Here are the punchlines of our group (students from Centrale Nantes), working on the test framework of Markus.

Charline

Status :

Carrying on working on the state of art in the student view with Claire Trying to solve Markus routing bugs in order to test what is working (applying Bertrand & Guillaume patch…)

Next Step :

Complete user documentation for the testing framework Start coding to make the framework working

Roadblock :

Difficulties to test what is working on my personal machine due to Rails update

Claire

Status :

Keeping on working on the state of art of the student view with Charline Trying to solve Markus bugs on the test framework in order to describe what is working (applying Bertrand & Guillaume patch)

Next Step :

complete user documentation for the test framework code to fix bugs linked to the test framework

Roadblocks :

difficulties to test what is working on my computer due to rails update

Benjamin

Status :

Progressing on ruby Reading some documents on github/markus/wiki

Next Step :

focusing on rails

Roadblocks :

none (waiting for our next meeting within our group for sharing our progress in the project)

Guillaume

Status :

Finalizing the integration of the patch (working with Github…) Have a clear look at the admin view of the test framework

Next Step :

Looking deeper in the admin view of the test framework Learn how to write build.xml and build.properties files

Bertrand

Status :

Working on testing the patch we released 2 weeks ago. Trying to find the error due to our change.

Next Step :

Work on Admin’s user documentation

Roadblocks :

None, just looking for the reason of the test fail

Karim

Status :

Working on the test framework for the grader view. Upgrading my own version of Markus to manage and work on the test framework.

Next Step :

keep working on the documentation for the test framework

Written by gbugnet

November 5th, 2011 at 5:34 am

Posted in Uncategorized

Character encoding problems

without comments

Markus currently aims to support both French and English users. A common character encoding in Western Europe is called ISO-8859-1, which allows the use of characters with accents for example. This is different from ASCII or Unicode (UTF-8) encoding which is commonly used by English users. Many other encoding formats are used by users that speak different languages. Currently, support for ISO-8859-1 and other encoding formats is limited.

Can’t we just convert everything to Unicode?

The biggest problem with character encoding is that there is no method to derive the encoding from the file or string itself. The best method we have at our disposal is to use algorithms that try to guess the most appropriate encoding or narrow down the number of possibilities. This still leaves the user to make the final judgement call.

Importing CSV files

CSV importing is currently suffering from this issue. Users could potentially provide a number of different encoding formats for their CSV files. If we simply assume the input format is always Unicode, we could end up with incomprehensible data since the characters are not always represented by the same internal numbers in different encoding formats.

A possible solution is to provide a drop-down menu filled with common encoding formats to allow for an accurate conversion to Unicode. However it is not guaranteed that the user will know which encoding they are using, or even know how to find out. Using an encoding guesser is the other solution, but as mentioned above, it runs into accuracy issues.

As a method of improving the accuracy of the encoding guesser to 100%, the user could be asked to double check the input after it has been converted. Presenting the option to select a different origin encoding and showing the output as it looks in Unicode. Changes between input encoding formats would need to be highlighted in order to reduce the chances of human error.

Another possible fix would be to try to see if entering the data assuming a Unicode encoding results in any changes in the string lengths, and reporting that as an error with the request to convert to UTF-8. This might be hard to comprehend for some users and makes using Markus a bit more tedious.

Student file submissions

This is another area where file encoding will need to be considered. Modifying the files in this case is not possible, as it would alter the student’s work. However viewing them from within Markus would require multiple encoding support. A simple drop-down menu with popular encoding formats should suffice in this case as modification of the file should not be needed (my impression).

Site text inputs

I am not sure this would be an issue as I have not yet tested it. I do not know if text boxes on the site are sent encoded in standard operating system encoding format, or if web browsers output everything in Unicode automatically. I will investigate how this works soon.

Plans for the future

Given the size of some of the potential fixes as well as their potential for creating even more problems, further discussion will be required before action is taken to fix this issue.

Written by akrassikov

November 2nd, 2011 at 5:26 am

Posted in Uncategorized

MarkUs UCOSP Fall 2011 – Status Week 7

without comments

Luke

Status

  • Still working on submission rule types

Next Steps

  • Finish submission rule types and move on

Road Blocks

  • Having some issues with submission rule objects and the way they are referenced. Reproduced some bugs on the sandbox that I am also encountering where objects can’t be found after they are created.

Erik

Status:

  • Working on bug 438: The groups page is broken
  • Fixed the following functions: Adding and removing students from groups, and the global validate, invalidate and delete buttons (nearly, 1 small issue before shipping)

Next Steps:

  • Still need to fix the create groups button as well as the validate, invalidate, notes and delete buttons for each table entry

Roadblocks:

  •   None

Razvan

Status:

  • Worked on and finished issue #560 minus one single syntax issue.
  • Looking for new issues to address.

Next Steps:

  • Find new issues to work on.
  • Write a blog post pertaining to button vs link functionality issue that was brought up a few weeks earlier.

Roadblocks:

  • Can’t really submit fixes for #560 until I re-add some code that I removed that does inline code generation in ruby.

Alex

Status:

  • Worked on issue #284, importing CSV files in ISO-8859-1 encoding resulting in incorrect data being entered.
  • Posted a blog post Character encoding problems.

Next Steps:

  • Create a fix for all the CSV importing locations rather than just Student and TA lists

Roadblocks:

  • Complexity of the file encoding problem is making me unsure of what to do
  • Conversion to Ruby 1.9 will allow for new methods of conversion between encoding formats, but having trouble getting Ruby 1.9 installed for testing

Severin

Status:

  • Continued work on load testing using the curl based script. It looks like I was able to reproduce a performance problem. It seems likely that slowness is related to mongrels getting saturated.
  • Created another script in order to be able to trigger submission posting on many clients.
  • Blogged about how to use the curl based scrip[1]
  • Submitted mid-term evaluation
  • Code review

Next Steps:

  • Clean up new script and add it to the relevant review[2]
  • Dig deeper. Make sure the performance problem is reliably reproducible and get some data to proof this (do some logfile analysis).
  • Talk to Karen about how to proceed from here.

Roadblocks:

  • None
[1] http://blog.markusproject.org/?p=3211
[2] http://review.markusproject.org/r/1128/

Written by lkysow

November 2nd, 2011 at 1:52 am

Posted in Uncategorized