Skip to content

Adding a new test to Crucible

This tutorial is a step by step example of how easy it was for me to add a new test to Crucible and subsequently how easy it is to run the new test.


  1. Creating the runtest Script
  2. Defining a Test Plan
  3. Automatically Pulling Packages to Test
  4. Running a Test
  5. 1. Creating the runtest script

    The first step is to create a runtest_[testname] script. Examples of these scripts can be seen in /testing/plans/commands/. Lets take runtest_dbench for example:

    	. run_profile.txt
    	. $BASE_DIR/plans/commands/common_with_loop.fns
    	SUT=`hostname -s`
    	mkdir -p $out_dir
    	if [ ! -d $run_dir ]; then
    	    echo "Error:  Run directory '$run_dir' does not exist"
    	    exit -10
    	export $out_dir
    	### dbench ###
    	echo 1 > "$out_dir/client_count.txt"
    	run_test "dbench" "/testing/usr/bin/dbench_runner $out_dir/client_count.txt" "$out_dir" "8"
    	rm "$out_dir/client_count.txt"
    	exit 0

    This is a fairly simple script. The first part of the script handles any necessary setup that the test requires. The second half of the script following "### debench ###" basically says how dbench should be run. As it is, the initial # of simulated clients is stored in client_count.txt. dbench is then executed (via the dbench_runner script) in a loop, 8 times, and the output is captured to $out_dir

    2. Defining a Test Plan

    We now need to define how the above runtest_[testname] script is going to be executed. It dictates things like which packages we want, do we want to patch the kernel, build the kernel, boot the kernel, etc etc. The test plan decides all of this. Let's take /testing/plans/os-drivers as an example:

    	export nfs10_packages="dbench"

    "wanted" means we want packages found in /testing/packages/X-os-drivers (the X stands for eXperimental).

    "commands" tells what this test run will do: on machine nfs10, create a patched kernel, build the kernel, boot the kernel, verify we booted to the correct kernel, install packages, run debench, then finish.

    "export" means add 'nfs10_packages="dbench"' to the run_profile.txt file and also it directs Crucible to install the latest available version of dbench

    3. Automatically Pulling Packages to Test

    >[? First you need to create a download location for your package:

    	mkdir /testing/packages/dbench

    To have Crucible automatically detect a new package and download it to the directory created above, you need to set up a section in the /testing/etc/pkgfind.cnf file. For example:

    	PACKAGE:     dbench
    	DEPTH:       0
    	WANTED:      ^dbench-.*\.tar\.gz$

    PACKAGE - is the name of the package to download
    URL - is the location to grab the package
    DEPTH - is how far down the URL hierarchy to search for the package
    WANTED - a regular expression for finding the package

    A cron job is set up to detect new packages once per hour.

    Now create a LATEST file whose contents contains the latest version of your package. For example, say the latest version of dbench is debench-3.04. The LATEST file would contain the following:


    4. Running a Test

    The crucible/docs/tutorial_sut_operations.txt might be useful to read first. But to run a test, you can do something like:

    	queue_package X-os-drivers/linux-2.6.17.tar.bz2

    or if you know the test run ID:

      	testrun requeue [ID]

    You can then watch the status of your testrun via:

    	watch testrun status

    Which generates something like:

    RUN PLAN               STATE        SUTS         TIME(m)  PACKAGE
    1582 netem_test        finished     nfs05,nfs04  38       linux-2.6.17-rc1-CITI_NFS4_ALL-1
    1583 netem_test        finished     nfs05,nfs04  46       linux-2.6.17-rc2-CITI_NFS4_ALL-1
    . . .
    1599 netem_test        running      nfs05,nfs04  27       linux-2.6.17-rc2-CITI_NFS4_ALL-1
    1600 nfsv4             running      nfs03,nfs02  200      linux-2.6.18-rc4-g73f0283-nfs-client-stable
    1601 os-drivers        running      nfs10        2        linux-2.6.17.tar

    You can then do something like:

    	testrun status 1601
    	testrun info 1601

    You can also immediately view the progress of your tests via a web browser. For example: