MarkUs Blog

MarkUs Developers Blog About Their Project

Automated Test Framework – Part 1: Assignment View

without comments

Testing Framework Overview:

Upon grading a student’s assignment, an instructor may want to execute a set of tests against the student’s code to validate the correctness of the code.  The instructor may also have various ways of handling pass/fail cases.  The testing framework has been implemented to allow instructors to perform such operations.  Using Ant, a Java-based build tool, as the core behind this testing feature, instructors will be allowed to upload their test files, any associated library files necessary for the tests to run, as well as any parsers they may wish to execute against the output, and control the compilation, testing and output parsing through a simple build file.

Part 1 of the testing framework consists of the Test Framework form from the Assignment view which allows a user to upload the Ant, test, library and parser files required to execute tests against students’ code.  Part 2 of the testing framework consists of the actual execution of the tests from the Submission view.  Part 3 is the student’s Assignment view where the student will be allowed (assuming they have tokens available to them) to execute the tests against their code before submitting their assignments.

The following documents how to create new test files associated to an assignment, as well as how to update and delete the files and the associated expected errors messages that will appear in response to invalid input.

Creating new test files:
1) Create a new assignment and click ‘Submit’
2) Click on ‘Testing Framework’ from the sub menu and you will be redirected to the Test Framework view (by default, the test form is disabled)
3) Check ‘Enable tests for this assignment’ and the test form will be enabled
4) Fill in the ‘Tokens available per group’ as needed.  This value represents the number of times a student may be allowed to execute the tests against their code before submitting their assignment.
5) Add the required Ant files, build.xml and
6) Click on ‘Add Test File’, ‘Add Library File’, ‘Add Parser File’ to add the appropriate test, library and parser files needed to run the tests respectively.  Test files represent the test cases that will be executed against the code.  Library files are any necessary files required by the tests to execute appropriately.  Parser files (if specified) will be executed against the output from the tests to extract and manipulate the output as needed.
7) Click ‘Submit’ to save the changes

*Optional* You can check the test file ‘private’ box to indicate a particular test cannot be executed by the student.

Testing Framework

Testing Framework

1) Verify table ‘test_files’ to ensure the files were saved correctly
2) Verify table ‘assignments’ to ensure ‘enable_test’ and ‘tokens_per_day’ were saved correctly

1) Verify the following folder structure in the TEST_FRAMEWORK_REPOSITORY:

A1 (folder)


– build.xml

– test –

– lib – junit-4.8.2.jar

– parse –

Updating test files:
1) Update any test, library or parser file already uploaded and click ‘Submit’ (able to replace with the same filename or overwrite with a different filename)

1) Verify updated file has been saved successfully

1) Verify updated file has been overwritten successfully

(If the replaced file has a different filename, it will be deleted from the directory.  If it has the same filename, it will simply be overwritten.)

Deleting test files:
1) Delete any test or library file by checking the ‘Remove’ box and click ‘Submit’

1) Verify file has been removed from the database successfully

1) Verify file has been removed from the filesystem successfully

Negative Test Cases:

Filename Validation –

  • cannot be named anything but ‘’
Invalid Filename

Invalid Filename


  • cannot be named anything but ‘build.xml’
Invalid Filename

Invalid Filename

test, library and parser files:

  • cannot be named ‘build.xml’ or ‘’
Invalid Filename

Invalid Filename

any file:

  • cannot have the same filename as any other existing test file belonging to that assignment
Duplicate Filename

Duplicate Filename

  • if you attempt to upload two files simultaneously with the same filename, an error will be displayed
Duplicate File Entry

Duplicate File Entry

  • cannot be blank (File entry will simply be ignored)

Assignment Validation –


  • cannot be negative
Invalid Token Value

Invalid Token Value

Other Scenarios:

  • if you update the test form in any way but then disable tests for the assignment and click ‘Submit’, those new changes will not be saved

Related Posts:

Automated Test Framework Overview
Automated Test Framework – Part 2: Submission View
Automated Test Framework – Part 3: Student’s View

Written by diane

August 16th, 2010 at 2:34 am

Posted in Automated Testing

Tagged with , ,

Leave a Reply