Adding a new test to Crucible

This tutorial is a step by step example of how easy it was for me to add a new test to Crucible and subsequently how easy it is to run the new test.

Contents:

Creating the runtest Script
Defining a Test Plan
Automatically Pulling Packages to Test
Running a Test

1. Creating the runtest script

The first step is to create a runtest_[testname] script. Examples of these scripts can be seen in /testing/plans/commands/. Lets take runtest_dbench for example:

	#!/bin/bash

	. run_profile.txt
	. $BASE_DIR/plans/commands/common_with_loop.fns

	SUT=`hostname -s`

	run_dir="$RUNPATH"
	out_dir="$run_dir/test_output"
	mkdir -p $out_dir

	if [ ! -d $run_dir ]; then
	    echo "Error:  Run directory '$run_dir' does not exist"
	    exit -10
	fi

	export $out_dir

	### dbench ###
	echo 1 > "$out_dir/client_count.txt"
	run_test "dbench" "/testing/usr/bin/dbench_runner $out_dir/client_count.txt" "$out_dir" "8"
	rm "$out_dir/client_count.txt"
	exit 0

This is a fairly simple script. The first part of the script handles any necessary setup that the test requires. The second half of the script following "### debench ###" basically says how dbench should be run. As it is, the initial # of simulated clients is stored in client_count.txt. dbench is then executed (via the dbench_runner script) in a loop, 8 times, and the output is captured to $out_dir

2. Defining a Test Plan

We now need to define how the above runtest_[testname] script is going to be executed. It dictates things like which packages we want, do we want to patch the kernel, build the kernel, boot the kernel, etc etc. The test plan decides all of this. Let's take /testing/plans/os-drivers as an example:

	wanted=X-os-drivers
	commands=nfs10:create_patched_kernel,build_kernel,boot_kernel,verify_kernel,install_packages,runtest_dbench,finish

	export nfs10_packages="dbench"

"wanted" means we want packages found in /testing/packages/X-os-drivers (the X stands for eXperimental).

"commands" tells what this test run will do: on machine nfs10, create a patched kernel, build the kernel, boot the kernel, verify we booted to the correct kernel, install packages, run debench, then finish.

"export" means add 'nfs10_packages="dbench"' to the run_profile.txt file and also it directs Crucible to install the latest available version of dbench

3. Automatically Pulling Packages to Test

	mkdir /testing/packages/dbench

To have Crucible automatically detect a new package and download it to the directory created above, you need to set up a section in the /testing/etc/pkgfind.cnf file. For example:

	PACKAGE:     dbench
	URL:         http://samba.org/ftp/tridge/dbench/
	DEPTH:       0
	WANTED:      ^dbench-.*\.tar\.gz$

PACKAGE - is the name of the package to download
URL - is the location to grab the package
DEPTH - is how far down the URL hierarchy to search for the package
WANTED - a regular expression for finding the package

A cron job is set up to detect new packages once per hour.

Now create a LATEST file whose contents contains the latest version of your package. For example, say the latest version of dbench is debench-3.04. The LATEST file would contain the following:

	3.04

4. Running a Test

The crucible/docs/tutorial_sut_operations.txt might be useful to read first. But to run a test, you can do something like:

	queue_package X-os-drivers/linux-2.6.17.tar.bz2

or if you know the test run ID:

  	testrun requeue [ID]

You can then watch the status of your testrun via:

	watch testrun status

Which generates something like:

RUN PLAN               STATE        SUTS         TIME(m)  PACKAGE
1582 netem_test        finished     nfs05,nfs04  38       linux-2.6.17-rc1-CITI_NFS4_ALL-1
1583 netem_test        finished     nfs05,nfs04  46       linux-2.6.17-rc2-CITI_NFS4_ALL-1
. . .
1599 netem_test        running      nfs05,nfs04  27       linux-2.6.17-rc2-CITI_NFS4_ALL-1
1600 nfsv4             running      nfs03,nfs02  200      linux-2.6.18-rc4-g73f0283-nfs-client-stable
1601 os-drivers        running      nfs10        2        linux-2.6.17.tar

You can then do something like:

	testrun status 1601

	testrun info 1601

You can also immediately view the progress of your tests via a web browser. For example:

	http://crucible.osdl.org/runs/1601