MarkUs Blog

MarkUs Developers Blog About Their Project

Archive for the ‘ant’ tag

Automated Test Framework – Part 1: Assignment View

without comments

Testing Framework Overview:

Upon grading a student’s assignment, an instructor may want to execute a set of tests against the student’s code to validate the correctness of the code.  The instructor may also have various ways of handling pass/fail cases.  The testing framework has been implemented to allow instructors to perform such operations.  Using Ant, a Java-based build tool, as the core behind this testing feature, instructors will be allowed to upload their test files, any associated library files necessary for the tests to run, as well as any parsers they may wish to execute against the output, and control the compilation, testing and output parsing through a simple build file.

Part 1 of the testing framework consists of the Test Framework form from the Assignment view which allows a user to upload the Ant, test, library and parser files required to execute tests against students’ code.  Part 2 of the testing framework consists of the actual execution of the tests from the Submission view.  Part 3 is the student’s Assignment view where the student will be allowed (assuming they have tokens available to them) to execute the tests against their code before submitting their assignments.

The following documents how to create new test files associated to an assignment, as well as how to update and delete the files and the associated expected errors messages that will appear in response to invalid input.

Creating new test files:
[UI]
1) Create a new assignment and click ‘Submit’
2) Click on ‘Testing Framework’ from the sub menu and you will be redirected to the Test Framework view (by default, the test form is disabled)
3) Check ‘Enable tests for this assignment’ and the test form will be enabled
4) Fill in the ‘Tokens available per group’ as needed.  This value represents the number of times a student may be allowed to execute the tests against their code before submitting their assignment.
5) Add the required Ant files, build.xml and build.properties
6) Click on ‘Add Test File’, ‘Add Library File’, ‘Add Parser File’ to add the appropriate test, library and parser files needed to run the tests respectively.  Test files represent the test cases that will be executed against the code.  Library files are any necessary files required by the tests to execute appropriately.  Parser files (if specified) will be executed against the output from the tests to extract and manipulate the output as needed.
7) Click ‘Submit’ to save the changes

*Optional* You can check the test file ‘private’ box to indicate a particular test cannot be executed by the student.

Testing Framework

Testing Framework

[DATABASE]
1) Verify table ‘test_files’ to ensure the files were saved correctly
2) Verify table ‘assignments’ to ensure ‘enable_test’ and ‘tokens_per_day’ were saved correctly

[FILESYSTEM]
1) Verify the following folder structure in the TEST_FRAMEWORK_REPOSITORY:

A1 (folder)

– build.properties

– build.xml

– test – TestMath.java

– lib – junit-4.8.2.jar

– parse – Parser.java

Updating test files:
[UI]
1) Update any test, library or parser file already uploaded and click ‘Submit’ (able to replace with the same filename or overwrite with a different filename)

[DATABASE]
1) Verify updated file has been saved successfully

[FILESYSTEM]
1) Verify updated file has been overwritten successfully

(If the replaced file has a different filename, it will be deleted from the directory.  If it has the same filename, it will simply be overwritten.)

Deleting test files:
[UI]
1) Delete any test or library file by checking the ‘Remove’ box and click ‘Submit’

[DATABASE]
1) Verify file has been removed from the database successfully

[FILESYSTEM]
1) Verify file has been removed from the filesystem successfully

Negative Test Cases:

Filename Validation –

build.properties:

  • cannot be named anything but ‘build.properties’
Invalid Filename

Invalid Filename

build.xml:

  • cannot be named anything but ‘build.xml’
Invalid Filename

Invalid Filename

test, library and parser files:

  • cannot be named ‘build.xml’ or ‘build.properties’
Invalid Filename

Invalid Filename

any file:

  • cannot have the same filename as any other existing test file belonging to that assignment
Duplicate Filename

Duplicate Filename

  • if you attempt to upload two files simultaneously with the same filename, an error will be displayed
Duplicate File Entry

Duplicate File Entry

  • cannot be blank (File entry will simply be ignored)

Assignment Validation –

tokens:

  • cannot be negative
Invalid Token Value

Invalid Token Value

Other Scenarios:

  • if you update the test form in any way but then disable tests for the assignment and click ‘Submit’, those new changes will not be saved

Related Posts:

Automated Test Framework Overview
Automated Test Framework – Part 2: Submission View
Automated Test Framework – Part 3: Student’s View

Written by diane

August 16th, 2010 at 2:34 am

Posted in Uncategorized

Automated Testing Framework

with 3 comments

Talks concerning MarkUs Automated Testing Framework

We defined some specifications for the Automated Testing Framework we wanted to implement. First, the framework must add as few dependencies as possible, and not disturb the core of MarkUs (MarkUs must not be overcrowded by other programs for testing student’s code). The Automated Testing Framework must also support as many languages as possible, allowing users to test any language regardless of the languages MarkUs has been implemented in.

Technical side

Today, C, Java, Python and Scheme are used in the Universities where MarkUs is deployed (Department of Computer Science, University of Toronto.School of Computer Science, University of Waterloo.Ecole Centrale de Nantes).
C++ should be easy to implement according to the work done with C.

Diane and I are going to build our framework by using examples from these languages. When we speak of an Automated Testing Framework, the idea is not to build a new Unit Testing Framework. We aim to build an “abstraction layer” between MarkUs and most used Unit Testing Frameworks for most used languages in University.

A picture is far better than words :

Automated Testing

Automated Testing Framework - version 1

MarkUs will call an Ant script written by the TA. This script, and all necessary files for the test environment (student code, data and tests) will be sent to a “working environment”. Later, the working environment could be moved to a dedicated server.

MarkUs will ask the Ant script to do many tasks (see Ant tasks), in an asynchronous way. Next, the Ant script, customizable by each instructor, will compile, test and run student’s code. The Ant script will produce an xml or text output for tests and build. Ant’s output will be saved in MarkUs and will be filtered for instructors and students.

The student will be able to see the result of the compilation and some test results will be made available by the instructor.

Written by Benjamin Vialle

June 9th, 2010 at 10:29 am

Posted in Uncategorized