MarkUs Blog

MarkUs Developers Blog About Their Project

Automated Testing: Options to implement security

with 2 comments

Please comment and add to this post!

There are two ‘general’ ways to implement security in auto-testing: To use VMs or not to use VMs. This is the question. At first, the consensus was in favor of using a VM, while I wanted to prove that it will be possible to achieve the same level of security via existing tools available in Linux (and SE Linux), because it seemed like a more elegant solution at the time, plus it would avoid the runtime overhead of VMs.

What I’ve realized is that there are two problems to that approach: First, this would require some custom tailoring of security features to different/older distributions. A greater problem would be allowing maximum freedom for assignments such as operating-systems, networking, security – while retaining security. In the end, it seems, running a VM seems like the only feasible solution.

Here is the rough workflow of doing automated testing with a VM:

1) Setup (by hand):

  • Install the VM.
  • Set up a virtual hard drive (preferably on a separate hard disk, for speed) and networking.
  • Install the necessary programs on it (minimum: Ant, SSH server)
  • Set up SSH keys to allow automatic SSH from local machine to VM.
  • Save the image.

2) Automated testing (shell script):

  • Start a headless VM from the initial image.
  • FTP test scripts/files
  • SSH-execute the script, pipe output to a file.
  • FTP output file to local machine.
  • Lather-rinse-repeat for every test request. That is, each test will start from new image to avoid possibility of corruption.

Possible improvements on the basic process:

  • Parallelizing test runs. That is, not being limited to serially running VMs. Useful in batch-testing, and for more real-time-like results (imagine 100 students testing their assignments 10 minutes before deadline!)
  • Throttling resource use – not waiting indefinitely on busy-wait, and timely termination of fork-bombs and infinite recursion.

Possible VM products:

  • VMWare Server
  • Oracle VirtualBox
  • Parallels
  • Qemu

I have heard very good things about Oracle’s product, and of course VMWare. Further research required to test for speed (up-time from saved image, system resources, restart) and general compatibility (I think they should be equivalent in this regard.)

That’s it for now. Please leave comments and suggestions!

Written by victor

October 27th, 2010 at 2:53 pm

Posted in Uncategorized

2 Responses to 'Automated Testing: Options to implement security'

Subscribe to comments with RSS or TrackBack to 'Automated Testing: Options to implement security'.

  1. Nice summary. Here are a few comments:

    1) Setup: The VM will need to be carefully configured to prevent data from leaking. We want to control only input and output from the VM.

    2) I think it is an open question whether we need to start up a clean VM for every test. I suspect that the likelihood of corruption is small enough that we can avoid the overhead of VM startup.

    We could consider running multiple tests in parallel on the same virtual machine. They would most likely need to be run as different users.

    On the other hand it may be possible to leverage research from our department into fast VM cloning. (

    We may also want to consider maintaining a pool of VMs to run tests on.

    Karen Reid

    27 Oct 10 at 4:58 pm

  2. It would be nice if VMs are as much abstracted as possible. I.e. we shouldn’t rely on any particular virtualization product. I am envisioning something like a pool for Python, Java, C, Scheme. Members of those pools are identified by a SSH host key, IP address and maybe SUDOable remote user.

    The easiest way to control what’s going in and out of VMs would be to put them into a DMZ. Then on the firewall, only allow SSH traffic from the MarkUs host machine to the VM-IP-pool. Similarly in the opposite direction, only allow SSH traffic from the VM-IP-pool to the MarkUs host.

    The MarkUs host could have a second NIC, with a private IP and is, hence, unreachable from the outside (via SSH).

    For a C socket assignment, admins could put a target host machine inside the DMZ and things can go crazy in there (not disrupting other traffic outside the DMZ).

    Otherwise, VMs configured for Python/C/Java and so on would be configured to disallow opening sockets and whatever else is required…

    As for communication sys-admins could set up a pool of users for connecting to and from VMs (using SSH public key without passphrase disallowing password authentication). Then, this randomly generated pool of users may be regenerated every day/week/whatever appropriate interval. For each test run MarkUs picks one of those users to connect to one of the VMs inside the DMZ.

    I’m pretty sure Alan and CSLab folks would have some great input in terms of system and network security, too 🙂


    28 Oct 10 at 7:34 pm

Leave a Reply