MarkUs Blog

MarkUs Developers Blog About Their Project

Archive for October, 2011

MarkUs Performance Analysis (2)

without comments

As mentioned earlier, I am currently working on analyzing MarkUs’ performance. After a few initial bumps I have a basic curl based script ready which can be used to simulate students submitting code through MarkUs. Here is the review where I introduce this new script. I should mention that this script is not very resilient to errors and is fairly sensitive with respect to the URLs you configure in that script. Have a look at this post as to why that’s the case. So here are the requirements for this to work:

  • Use MarkUs 0.10.x (MarkUs 0.11.x should work as well, but a Rails 3 based MarkUs will likely no quite work; at the very least it has been untested)
  • Prepare the server as described below.
  • Make sure that you have curl installed on the client machine.

The curl based script (lib/benchmarking/post_submissions.sh) comes with a buddy rake task called markus:benchmark:students_create. This rake task can be used to create many student users. It’ll destroy any existing student users and will hand out user names of the form student_1, student_2, student_3 and so on. It is important that you use this rake task on the server in order to create many student users, since post_submissions.sh attempts to log in as student_<#> (<#> will depend on the range you passed as a parameter to this script when you ran it). Ok, without further ado, here is the cookie-cutter-recipe:

Server Setup

First make sure that CSRF is turned on for your MarkUs instance. Make sure you have this line in your config/environments/production.rb:

config.action_controller.allow_forgery_protection    = false

 
Then to do the following:

$ bundle exec rake repos:drop RAILS_ENV=production
$ bundle exec rake db:reset RAILS_ENV=production
$ bundle exec rake db:populate RAILS_ENV=production
$ bundle exec rake markus:benchmark:students_create num=1000 \
  RAILS_ENV=production

 

The last command will create 1000 student users, student_1 to student_1000.

Client Script

After this it’s good to do some sanity checking. Make sure that logging in as student_1 and student_1000 works. Now you are ready to run post_submissions.sh. Here is one way to do it. It will use the MarkUs instance as specified by MARKUS_BASE_URL and submits all files in directory lib/benchmarking/submission_files for students student_1 to student_10.

$ cd path/to/copy/of/lib/benchmarking
$ ./post_submissions.sh 1 10

 
Note that the user who runs post_submissions.sh needs to have write permissions in the current working directory (the script needs to store/update the MarkUs cookie). After running the script you should see a directory created which holds one log file per student submission. At the moment this is the only way to figure out what happened when the script attempted to log in, visit the assignments page and submitted the files for assignment 1. Keep in mind that if you get any conflict while submitting, nothing will be submitted. This is how MarkUs works. Thus, make sure you are not submitting a file (same file name) twice.

If all went well, you should see the submitted files in a browser session as well.

So that’s the status for the time being. I’ll be likely writing another script which allows one to distribute and execute post_submissions.sh on many clients. Keep your eyes peeled for a follow up post 🙂 Questions, concerns? Let me know in the comments. Thanks!

Written by Severin

October 31st, 2011 at 12:05 pm

Posted in Uncategorized

MarkUs UCOSP Fall 2011 – Meeting minutes, Oct 26

without comments

  • Meeting started at 3:20 p.m. (EDT) and ended at 4:00 p.m.
  • Each member is to write at least one blog post or make a significant contribution to the user documentation
  • Midterm review comments are due
  • Luke fixed as section due date bug, as well as found a possible bug in the use of a transaction in the “create section” part of the assignment controller
  • Erik worked on the admin group creation process. The current version requires a major reworking of the code, but a partial fix should be up for review later this week
  • Razvan was busy with midterms, and thus was not able to do much
  • Alex fixed the notes bugs and a CSV import routing bug, plans to continue working on more CSV import bugs
  • Severin got a script simulating students activity. Plans to run it on the server to see how it behaves, afterward turning on profiling if things are successful
  • Danesh and Tobi have joined the core Markus team

Written by akrassikov

October 27th, 2011 at 12:25 pm

Posted in Uncategorized

MarkUs UCOSP Fall 2011 – Status Week 6

without comments

Luke

Status

  • Got first step in assignment creation bugfixing into master with per sections submission date fix

Next Steps

  • Continue with assignment creation by fixing submission rule creation

Road Blocks

  • None

Erik

Status:

  • Working on issue 438: Admins cannot create groups or add students to them
  •   ade changes so that admins can create groups (with errors though), no success with adding students yet

Next Steps:

  • Finish up the issue. Still aways from finishing

Roadblocks:

  •   None

Razvan

Status:

  • Did not get a chance to work on Markus this past week due to midterms and a couple of very big assignments.

Next Steps:

  • Finish up the bugs I started working on a week ago.
  • Find new issues to try and tackle.

Roadblocks:

  • None

Alex

Status:

  • Fixed bug #430. Selector did not appear for groups when creating a new note.
  • Fixed bug #431. Editing a note with invalid values had unescaped HTML preventing editing.
  • Fixed bug #426. Importing a CSV file to add students and TAs works again.

Next Steps:

  • Working on fixing bug #284. Adding support for ISO-8859-1 encoded CSV files to support french characters.

Roadblocks:

  • None

Severin

Status:

Next Steps:

  • Use the prepared script on all 8 client machines to simulate load on the server machine.
  • Blog about how to use the new script for load simulation (together with the rake task for creating many students on the server).

Roadblocks:

  • None

Written by Severin

October 25th, 2011 at 11:33 pm

Posted in Uncategorized

Why script based load simulation for MarkUs is difficult

with one comment

I am trying to get some data about how MarkUs behaves under load. One aspect would also be to get a better feel for how scalable MarkUs is. I’m aiming to simulate a production environment which is something of the order of a couple of hundred students submitting code (and perhaps implicitly creating a Subversion repositories) at the same time. So far I’ve been trying to create a script which logs on to MarkUs, visits relevant pages and submits code. However, I’ve been hitting a bit of a snag. And here is why.

Problem 1: Cookies

Since MarkUs is a web application and the Web is stateless per se, something needs to be done on top of plain HTTP in order to maintain sessions. The most commonly used session tracking mechanism is to use cookies (an alternative would be URL rewriting). In the MarkUs case it’s one cookie which stores the session id of a particular user. If a browser requests a MarkUs page it usually gets a header Set-Cookie with the cookie info back. If not disabled by the user, the browser sends all cookies for this particular server (or path for that matter) which were previously set with the Set-Cookie header back to the server. This happens under the covers of a Web browser and a user usually does not realize that cookies are sent back and forth along with requests and responses. However, when using a script one has to be careful to store the cookie and send it back on subsequent requests. As it turns out this was a problem for my Ruby based script. I didn’t quite figure out why :(. In any case, curl seems to be doing a better job, so this is what I’m going to try next.

Problem 2: HTTP Redirects

Another thing which makes script based load testing painful is that HTTP redirects can happen. Redirects are nothing other than mini responses sent back to the client telling it the actual location of the source. When using a Web browser, this is almost not noticable, since the browser usually follows redirects (i.e. automatically requests the new location). Again, this is different in a script based walk through a web application. One might get a redirect response when you are not expecting one and if you’re not careful, you’ve hammered a certain page many times only to realize that you have been getting redirect responses. What’s more, MarkUs generates redirects if it thinks that the user is not logged in. Forget to send the cookie you’ve gotten earlier and MarkUs will redirect you to the log-in page instead of giving you the response for a particular page or servicing a HTTP POST request.

Why not turn off authentication and sessions entirely?

Some of you might think that I could just turn off sessions and authentication as a whole, but this won’t work well either. First note that I’d have to change a fair bit of MarkUs code to get it working without any authentication. Second, with authentication turned off, there is no notion of different users anymore. All of a sudden all requests belong to one single user. Hence, we don’t want this for script based load testing of MarkUs either. We want to simulate hundreds of users logging in simultaneously and not one single user over and over again.

Conclusion

For script based load testing I’ll have to make cookies work and deal with redirects. For the time being a curl based solution seems most promising. This won’t yield any request timing data on the client side eventually, though. I could manually time requests in order to get a feel of how long MarkUs takes to serve them under load, but not sure if that’s a high priority. Once I can simulate some load, I’ll use request times in the logs to approximate request/response times. Wouldn’t I have to deal with cookies and redirects, Apache Bench could be viable, but it seems it doesn’t handle cookies and redirects very well. I’ll keep you posted as to how it goes.

As usual, please leave a comment if you have thoughts about this or I’m missing something very obvious and am just not doing it right 😉

Written by Severin

October 23rd, 2011 at 5:47 pm

Posted in Uncategorized

Punchlines – 23 october 2011

without comments

Here are the punchlines of our group (students from Centrale Nantes), working on the test framework of Markus.

Charline

Status :

Tests on Markus demo version what is working. Learning of ruby language. Understanding ant

Next Step :

writing users documentation concerning test framework
Carrying on testing what is working.

Claire

Status :

tutorials concerning Ruby and Rails
browsing almost all possibilities of the demo markus version
ant clearings up

Next Step :

writing user guide concerning test framework
applying Ruby and Rails knowledge for reading code fragments concerning test framework

Benjamin

Status :

Improving my knowledge on ruby

Next Step :

Keep going on ruby
Have a better look on the use cases

Roadblocks :

I ought to have a better look on the current work on the use cases in order to increase my point of view on the requirements of the project

Guillaume

Status :

Fix a bug (routing). Nelle assigned me my first ticket !
Learn a bit about Github

Next Step :

Learn how to use Review board and Github, to submit our modifications

Bertrand

Status :

Fixed the routing bug due to the rails 3 migration by adding a function “Update” AutomatedTestControler
Learnt some more about Git and GitHub

Next Step :

Understand what the Update function does exactly.
Carry on understanding the MVC model

Karim

Status :

Understanding the test framework.
Reading documentation about Markus and specifying test needs.

Next Step :

Getting some interest on ant files.
Try to configure new tests.

Written by gbugnet

October 23rd, 2011 at 8:17 am

Posted in Uncategorized

Markus UCOSP 2011 – Meeting minutes, Oct 19

without comments

  • Meeting started at 3:15 p.m. (EDT) and ended at 4:00 p.m.
  • Luke has just finished resolving issues related to creating assignments. Continuing on to work on general bugfixing.
  • Erik has finished solving some grader routing issues, and is now working on group routing issues
  • Razvan is knee deep in anotation text issues
  • Alex is working on some notes bugfixes
  • Severin is having cookie issues, and is moving towards using curl as a workaround. Blog post forthcoming.

 

 

Written by Erik

October 19th, 2011 at 4:03 pm

Posted in Uncategorized

MarkUs UCOSP Fall 2011 – Status Week 5

without comments

Luke

Status

  • Got error display pulled into main
  • Tied up some loose ends with bugs that had been closed

Next Steps

  • Investigate ‘type’ column error

Road Blocks

  • None

Erik

Status:

  • Completed issue 521: Broken link to the add/edit graders page. It was a strange routing issue dealing with the pluralisation of “TA”.
  • Finished functional test cases for TAs
Next steps:
  • Working on issue 438: students cannot be added to groups
Road blocks:
  • None.

Razvan

Status:

  • Fixed delete and create for annotation texts (issue 560)
  • Started working on creation of annotation texts and have it in a half complete state

Next steps:

  • Finish up issue 560
  • Look into cleaning up how buttons/links look in cases where we use both next to eachother

Road blocks:

  • Assignments and midterms just around the corner
  • Trying to figure out the cause of the errors in rendering related to add new annotation text that was introduced now that the controller method is properly being called

Alex

Status:
  • No work done this week
Next steps:
  • Work on issues #430 and #431, both notes related
  • Put in this week’s missed work time into next week
Road blocks:
  • Assignments and midterm prep took longer than expected

Severin Gehwolf

Status:

  • Tested script based load testing using REMOTE_USER + basic auth. There was no gain, since the current user is inside MarkUs is still tracked in a session cookie. Back to square one.
  • Wrote an unofficial MarkUs 0.11.1 patch for U of T (tests results workaround).
  • A bit of code review.

Next Steps:

  • Write blog post about why maintaining sessions in a script bases solution is difficult and why there’s no way without it.
  • Move forward with a curl based scripted solution which seems to work (using its cookie store support).

Roadblocks:

  • Other things are keeping me busy.

Written by Razvan

October 19th, 2011 at 1:05 pm

Posted in Uncategorized

Punchlines – 14 october 2011

with one comment

Here are the punchlines of our group (students from Centrale Nantes), working on the test framework of Markus.

Charline

Status :

Specifications and use cases finished
Test what is working on Markus demo version. And write the user documentation concerning the testing framework.

Next Step :

Continuing the tests on the testing framework

Claire

Status :

writing user cases and the blog article with Charline
begin the test on the Markus demo version

Next Step :

Test the test module on the demo version
Write the user documentation for the test framework

Benjamin

Status :

Improving my knowledge on ruby

Next Step :

Keep going on ruby
Have a better look on the use cases

Roadblocks :

I ought to have a better look on the current work on the use cases in order to increase my point of view on the requirements of the project

Guillaume

Status :

Trying to debug routing issues in the test framework. I looked at commits about routing issues.

Next Step :

Understand how the Reviewboard work, find more info about routing and debug !

Bertrand

Status :

Testing the Markus version running on my server.
Trying to figure out how to fix some routing bugs.
Reading some commits.

Next Step :

I still need to understand how the test are working in Markus.
Make my Markus version work as the demo version

Roadblocks :

Having trouble understanding Ruby

Karim

Status :

Reading some commits in ruby.
Exploring Markus demo and working on documentation.

Next Step :

Continue to learn about the application, state of art, bugs and work to do.
Work on documentation of existant functions.

Written by gbugnet

October 14th, 2011 at 11:58 am

Posted in Uncategorized

MarkUs UCOSP Fall 2011 – Meeting minutes, Oct 12

without comments

  • Meeting started at 3:15 p.m. (EDT) and ended at 4:00 p.m.
  • Erik is going to do some refactoring to fix some issues with TA routes
  • Luke has been and will continue to work on errors and how they’re displayed
  • Alex had some issues with buttons looking bad next to links but he fixed that and now he’s going to work on notes
  • Razvan is working on the annotation text pages
  • Severin released 0.11.0 and is starting work on load testing

Written by lkysow

October 12th, 2011 at 5:08 pm

Posted in Uncategorized

MarkUs UCOSP Fall 2011 – Status Week 4

without comments

Erik

Status:
  • Completed issue 552: Cancel buttons on various forms were appearing smaller than the submit buttons
  • Started working on issue 521: Cannot reach the create/edit graders page. Seems to be some sort of routing issue, but I haven’t really gotten anywhere yet.
  • Implemented functional tests for the TA (graders) page. It previously had no tests, but now has a few simple ones, including ones that fail now, but should succeed once issue 521 is resolved
Next steps:
  • Complete this issue and move onto the next one
Road blocks:
  • None.

Luke

Status

Next Steps

  • Get error fixes reviewed.

Road Blocks

  • None

Severin

Status:

  • Fixed about dialog on master[1].
  • Shipped MarkUs 0.11.0[2] including the possibility of hiding assignments.
  • Experimented with existing benchmarking scripts in order to produce load on the server. Didn’t get very far due to a cookie problem. Will try to go down the HTTP basic authentication route.
  • Code review.

Next Steps:

  •   Setup server with Apache’s basic authentication and MarkUs REMOTE_USER.
  •   Give the load test script another shot with basic auth.

Roadblocks:

  • Sessions in HTTP are difficult to maintain in scripts, which I need for load testing. I hope basic authentication will do the trick.

[1] http://review.markusproject.org/r/1117/
[2] http://blog.markusproject.org/?p=3139

Alex

Status:

  • Modified the refactored functional tests after some changes were pushed to master
  • Fixed the deleting a note bug (#429)
  • Had my previous bug fixed by Nelle before I really got into it (#434)

Next Steps:

  • Work on issues #430 and #431, both notes related

Roadblocks:

  • None

Razvan

Status:
  • Completed issue 520: Delete annotations button not working
  • Started working on issues regarding links not doing anything for annotation texts, still need to create a github issue for this
Next steps:
  • Complete current issue and continue on fixing rails 3 bugs
  • Start planning on term project for Markus
Road blocks:
  • None

Written by akrassikov

October 12th, 2011 at 1:11 pm

Posted in Uncategorized