MarkUs Blog

MarkUs Developers Blog About Their Project

Archive for the ‘testing’ tag

MarkUs 0.9.3 is out!

without comments

The team is pleased to announce MarkUs 0.9.3!

MarkUs 0.9.3 fixes a nasty little bug that prevents students from viewing their test results.  If your students have been complaining that they can’t see their test output, well, this is the release for you.

Or, if you’re running 0.9.2 and want to patch up – download the patch here (you’ll want to apply sequentially).  It’s short, sweet, and shouldn’t require any special configuration.  All wins.

Great job team – keep up the good work!

Written by m_conley

December 2nd, 2010 at 5:09 pm

Posted in Uncategorized

Automated Test Framework – Part 1: Assignment View

without comments

Testing Framework Overview:

Upon grading a student’s assignment, an instructor may want to execute a set of tests against the student’s code to validate the correctness of the code.  The instructor may also have various ways of handling pass/fail cases.  The testing framework has been implemented to allow instructors to perform such operations.  Using Ant, a Java-based build tool, as the core behind this testing feature, instructors will be allowed to upload their test files, any associated library files necessary for the tests to run, as well as any parsers they may wish to execute against the output, and control the compilation, testing and output parsing through a simple build file.

Part 1 of the testing framework consists of the Test Framework form from the Assignment view which allows a user to upload the Ant, test, library and parser files required to execute tests against students’ code.  Part 2 of the testing framework consists of the actual execution of the tests from the Submission view.  Part 3 is the student’s Assignment view where the student will be allowed (assuming they have tokens available to them) to execute the tests against their code before submitting their assignments.

The following documents how to create new test files associated to an assignment, as well as how to update and delete the files and the associated expected errors messages that will appear in response to invalid input.

Creating new test files:
[UI]
1) Create a new assignment and click ‘Submit’
2) Click on ‘Testing Framework’ from the sub menu and you will be redirected to the Test Framework view (by default, the test form is disabled)
3) Check ‘Enable tests for this assignment’ and the test form will be enabled
4) Fill in the ‘Tokens available per group’ as needed.  This value represents the number of times a student may be allowed to execute the tests against their code before submitting their assignment.
5) Add the required Ant files, build.xml and build.properties
6) Click on ‘Add Test File’, ‘Add Library File’, ‘Add Parser File’ to add the appropriate test, library and parser files needed to run the tests respectively.  Test files represent the test cases that will be executed against the code.  Library files are any necessary files required by the tests to execute appropriately.  Parser files (if specified) will be executed against the output from the tests to extract and manipulate the output as needed.
7) Click ‘Submit’ to save the changes

*Optional* You can check the test file ‘private’ box to indicate a particular test cannot be executed by the student.

Testing Framework

Testing Framework

[DATABASE]
1) Verify table ‘test_files’ to ensure the files were saved correctly
2) Verify table ‘assignments’ to ensure ‘enable_test’ and ‘tokens_per_day’ were saved correctly

[FILESYSTEM]
1) Verify the following folder structure in the TEST_FRAMEWORK_REPOSITORY:

A1 (folder)

– build.properties

– build.xml

– test – TestMath.java

– lib – junit-4.8.2.jar

– parse – Parser.java

Updating test files:
[UI]
1) Update any test, library or parser file already uploaded and click ‘Submit’ (able to replace with the same filename or overwrite with a different filename)

[DATABASE]
1) Verify updated file has been saved successfully

[FILESYSTEM]
1) Verify updated file has been overwritten successfully

(If the replaced file has a different filename, it will be deleted from the directory.  If it has the same filename, it will simply be overwritten.)

Deleting test files:
[UI]
1) Delete any test or library file by checking the ‘Remove’ box and click ‘Submit’

[DATABASE]
1) Verify file has been removed from the database successfully

[FILESYSTEM]
1) Verify file has been removed from the filesystem successfully

Negative Test Cases:

Filename Validation –

build.properties:

  • cannot be named anything but ‘build.properties’
Invalid Filename

Invalid Filename

build.xml:

  • cannot be named anything but ‘build.xml’
Invalid Filename

Invalid Filename

test, library and parser files:

  • cannot be named ‘build.xml’ or ‘build.properties’
Invalid Filename

Invalid Filename

any file:

  • cannot have the same filename as any other existing test file belonging to that assignment
Duplicate Filename

Duplicate Filename

  • if you attempt to upload two files simultaneously with the same filename, an error will be displayed
Duplicate File Entry

Duplicate File Entry

  • cannot be blank (File entry will simply be ignored)

Assignment Validation –

tokens:

  • cannot be negative
Invalid Token Value

Invalid Token Value

Other Scenarios:

  • if you update the test form in any way but then disable tests for the assignment and click ‘Submit’, those new changes will not be saved

Related Posts:

Automated Test Framework Overview
Automated Test Framework – Part 2: Submission View
Automated Test Framework – Part 3: Student’s View

Written by diane

August 16th, 2010 at 2:34 am

Posted in Uncategorized

Automated Testing Framework

with 3 comments

Talks concerning MarkUs Automated Testing Framework

We defined some specifications for the Automated Testing Framework we wanted to implement. First, the framework must add as few dependencies as possible, and not disturb the core of MarkUs (MarkUs must not be overcrowded by other programs for testing student’s code). The Automated Testing Framework must also support as many languages as possible, allowing users to test any language regardless of the languages MarkUs has been implemented in.

Technical side

Today, C, Java, Python and Scheme are used in the Universities where MarkUs is deployed (Department of Computer Science, University of Toronto.School of Computer Science, University of Waterloo.Ecole Centrale de Nantes).
C++ should be easy to implement according to the work done with C.

Diane and I are going to build our framework by using examples from these languages. When we speak of an Automated Testing Framework, the idea is not to build a new Unit Testing Framework. We aim to build an “abstraction layer” between MarkUs and most used Unit Testing Frameworks for most used languages in University.

A picture is far better than words :

Automated Testing

Automated Testing Framework - version 1

MarkUs will call an Ant script written by the TA. This script, and all necessary files for the test environment (student code, data and tests) will be sent to a “working environment”. Later, the working environment could be moved to a dedicated server.

MarkUs will ask the Ant script to do many tasks (see Ant tasks), in an asynchronous way. Next, the Ant script, customizable by each instructor, will compile, test and run student’s code. The Ant script will produce an xml or text output for tests and build. Ant’s output will be saved in MarkUs and will be filtered for instructors and students.

The student will be able to see the result of the compilation and some test results will be made available by the instructor.

Written by Benjamin Vialle

June 9th, 2010 at 10:29 am

Posted in Uncategorized

A Sneaky Change in Rails 2.3.2 Temporarily Broke Our Tests

without comments

Today, I’ve decided to revamp the test suites for OLM. I won’t lie – we’ve been a little lazy in terms of testing. We have lots of meetings where we say “our testing is going to be solid”, but then nothing really happens.

I’m sure we’re not alone in this regard.

Anyhow, I decided to dust off our tests and give them a run. Here’s what I got:


./test/unit/../test_helper.rb:22: undefined method `use_transactional_fixtures=' for Test::Unit::TestCase:Class (NoMethodError)

Hrmph. What’s going on? After a little Googling, and it turns out that in Rails 2.3.2, “Test::Unit::TestCase changed to ActiveSupport::TestCase”.

So, I had to go into ./test/test_helper.rb, and change the class to ActiveSupport::TestCase. No big deal really, but it was confusing at first.

It’s a real blow to your self-esteem when your testing framework doesn’t even run.

Anyhow, it’s fixed. Now to write some good tests…

FOLLOW UP: You also need to change Test::Unit::TestCase to ActiveSupport::TestCase in your tests.

Written by m_conley

May 26th, 2009 at 11:02 am

Posted in Uncategorized

Testing Rails with Authentication

without comments

I’ve been putting off doing functional testing with rails for a while now because of the snag that we couldn’t figure out how to test with authentication.  Finally I’ve decided to solve this once and for all and get to the root of the problem. First approach we tried was to explicitly log in using a post request to the login page in the test setup. Of course, it didn’t work as each request I believe is independent of the subsequent requests, at least during functional testing.

The next approach we tried was to go down one level and expose the method that sets the current user for the session, and then call that as a controller function.  This made me a bit uneasy because we were making the current user setter public, which could probably mean that it would be exposed as an action that can be exploited.  Either way, this approach still didn’t work and was still giving us a 302 redirected status.

We decided to go one more level down and tried then to explicitly set the session hash, which is how we keep track of the current user.  One way that I came across was to set the session variable in a request object, that is

def setup
@controller = AssignmentsController.new
@request = ActionController::TestRequest.new
@response = ActionController::TestResponse.new
@admin = users(:admin)
@request.session['uid'] = @admin.id # login before testing
end

or some reason, this didn’t work out either.  Finally, it turns out that session variables can be set on every (get/post/put/delete) request made as an optional parameter. Now, each of the request type methods are wrapped with an extra user parameter to make that request on behalf of that user. For example, we now have a get_as method defined as:

# Performs GET request as the supplied user for authentication
def get_as(user, action, params=nil, flash=nil)
session_vars = { 'uid' => user.id, 'timeout' => 3.days.from_now }
get(action, params, session_vars, flash)
end

Written by Geofrey

November 7th, 2008 at 2:38 pm

Posted in Uncategorized

Pay Attention to Little Things

with one comment

On testing:

If you ever tried running your tests and find yourself getting the error:

your_controller_test.rb:3:in `require': no such file to load -- test_helper (LoadError)
from your_controller_test.rb:3

it’s probably because the generated require call, require test_helper is not being properly recognized.  Instead of the default require 'test_helper', replace it with:

require File.dirname(__FILE__) + '/../test_helper'

If you’re doing functional testing, make sure that the controller being tested is also required (e.g. add the line require 'your_controller' on top).  Be sure that you also have installed the package libtest-unit-ruby (if using Linux), before testing.

Also, thanks to Jeff Balogh, make sure that you append test_ on all your test cases to make sure that they actually run (I know it’s obvious, but it’s nice to be reminded once in a while).

On database constraints:

If you want to enforce foreign key constraints, remember to create the required indexes first.  ActiveRecord Migration supports this using add_index.  This only applies if the column being referred to is not a primary key, if it is then there’s no need to create the index. Once this is set, you can now enforce foreign keys using ActiveRecord Migrations by implementing this.

On implementing sessions:

By default, the rails session handler stores session information on a file in the server.  This can cause scalability issues in the long run if file space runs out or when using multiple servers.  Thus, it’s recommended to use :active_record_store instead. To do this make sure that you do the following:

  • execute rake db:sessions:create to add a create table migration for sessions
  • execute rake db:migrate to apply the new schema
  • change session store to active session store by adding this line to config/environment.rb:

config.action_controller.session_store = :active_record_store

  • Make sure that you comment out or add this line to your ApplicationController:

protect_from_forgery :secret => <32_hex_digit_secret_key>

Here are also some helpful links regarding sessions and implementing it:

Written by Geofrey

July 29th, 2008 at 4:01 pm

Posted in Uncategorized