From 6f44fffcea15f7484fe7b2037fa4df4a76347e5c Mon Sep 17 00:00:00 2001 From: Robin Sommer Date: Tue, 7 Jun 2011 22:05:14 -0700 Subject: [PATCH] Starting a small framework for doing regression testing with larger traces and more complex Bro configurations. That's "Type 2" in the terminology of http://www-new.bro-ids.org/devel/projects/testing.html This leverages btest as well and will replace the set of scripts that the old private test-suite is using. It can accomodate tests with both public and private traces. This is a checkpoint commmit, not much there yet in terms of actual testing. Also still need to figure out a mechanmism to allow for different baselines based on which branch we're testing. --- testing/README | 19 +++--- testing/external/.gitignore | 1 + testing/external/Baseline/.gitignore | 0 testing/external/Makefile | 11 ++++ testing/external/README | 84 ++++++++++++++++++++++++ testing/external/btest.cfg | 18 +++++ testing/external/random.seed | 17 +++++ testing/external/scripts/create-new-repo | 42 ++++++++++++ testing/external/scripts/diff-canonifier | 5 ++ testing/external/scripts/test.skeleton | 2 + testing/external/tests/.gitignore | 0 testing/external/tests/test.skeleton | 2 + testing/external/traces/.gitignore | 0 13 files changed, 191 insertions(+), 10 deletions(-) create mode 100644 testing/external/.gitignore create mode 100644 testing/external/Baseline/.gitignore create mode 100644 testing/external/Makefile create mode 100644 testing/external/README create mode 100644 testing/external/btest.cfg create mode 100644 testing/external/random.seed create mode 100755 testing/external/scripts/create-new-repo create mode 100755 testing/external/scripts/diff-canonifier create mode 100644 testing/external/scripts/test.skeleton create mode 100644 testing/external/tests/.gitignore create mode 100644 testing/external/tests/test.skeleton create mode 100644 testing/external/traces/.gitignore diff --git a/testing/README b/testing/README index 1ea54c4431..d5c78c7e03 100644 --- a/testing/README +++ b/testing/README @@ -1,13 +1,12 @@ -This directory contains some of the suites for testing for Bro's -correct operation: +This directory contains suites for testing for Bro's correct +operation: -btest/ - Tests Bro's logging & bundled scripts. See btest/README for more. + btest/ + An ever-growing set of small unit tests testing Bro's + functionality. -istate/ - Tests Bro's independent state facilities. These include persistent - values and inter-process event/value communication. + external/ A framework for downloading additional test sets that + run more complex Bro configuration on large traces files. Due + to the size, these are not included directly. See the README + for more information. -(Note that the Bro developers maintain a separate test suite for - Bro's trace analysis capabilities. This is kept private as it uses - sensitive raw traces for input.) diff --git a/testing/external/.gitignore b/testing/external/.gitignore new file mode 100644 index 0000000000..444ff42e80 --- /dev/null +++ b/testing/external/.gitignore @@ -0,0 +1 @@ +*.git diff --git a/testing/external/Baseline/.gitignore b/testing/external/Baseline/.gitignore new file mode 100644 index 0000000000..e69de29bb2 diff --git a/testing/external/Makefile b/testing/external/Makefile new file mode 100644 index 0000000000..7aa9257333 --- /dev/null +++ b/testing/external/Makefile @@ -0,0 +1,11 @@ + +PUBLIC_REPO=/home/robin/tmp/bro-tests + +all: + @for repo in *.git; do ( cd $$repo && btest ); done + +init: + git clone $(PUBLIC_REPO) public.git + +pull: + @for repo in *.git; do ( cd $$repo && git pull ); done diff --git a/testing/external/README b/testing/external/README new file mode 100644 index 0000000000..88ba634d8d --- /dev/null +++ b/testing/external/README @@ -0,0 +1,84 @@ + +Test Suite for Large Trace Files +================================ + +This test-suite runs more complex Bro configurations on larger trace +files, and compares the results to a preestablished baseline. Due to +their size, both traces and baseline are not part of the main Bro +repository but kept externally. In addition to the publically provided +files, one can also add a local set to the test-suite for running on +private traces. + +Initialization +-------------- + +Before the test-suite can be run, one needs to download the traces and +the corresponding test and baseline files. They are kept in a separate +git repository and a ``Makefile`` is provided to clone a copy: + +.. console: + + > make init + +To later update to upstream changes: + +.. console: + + > make pull + +Running Tests +------------- + +The easiest way to run all tests is simply typing ``make``. Doing so +will iterate through all ``*.git`` repositories and run ``btest`` in +each of them. + +Alternatively, one can also manually run all tests inside a single +test repository: + +.. console: + + > cd public.git + > btest + +All the standard ``btest`` options can be used to run individual +tests, get diagnostic output, etc. + +Updating Baseline +----------------- + +To update a test's baseline, first run ``btest`` in update mode: + +.. console: + + > cd public.git + > btest -U tests/test-you-want-to-update + +Then use ``git`` to commit the changes and push the changes upstream +as usual. + +Adding a Local Repository +------------------------- + +One can add local non-public set of tests (potentially using private +traces) by creating a git repository of a similar structure as the +public one. A helper script is provided to set up such a repository: + +.. console: + + > ./scripts/create-new-repo private-repo /home/git/bro-private-repo + +The first argument is the local name of the repository (it will be +cloned into ``.git``); and then second is the path to the (bare) +git repository. The repository will be initialized with a few standard +directories as well as a skeleton test in ``.git/tests``. You +can then edit files as needed (and in particular add traces to +``.git/traces``). + +If you want the repository to be located somewhere else than the local +machine, just push the created local repository to the right place, +delete ``.git``, and clone from the remote location. Likewise, +if you already have such a private test repository that you want to +include into the test suite, clone it directly into +``.git/tests``. + diff --git a/testing/external/btest.cfg b/testing/external/btest.cfg new file mode 100644 index 0000000000..270bfea93c --- /dev/null +++ b/testing/external/btest.cfg @@ -0,0 +1,18 @@ +[btest] +TestDirs = tests +TmpDir = %(testbase)s/.tmp +BaselineDir = %(testbase)s/Baseline +IgnoreDirs = .svn CVS .tmp +IgnoreFiles = *.tmp *.swp #* *.trace .gitignore + +[environment] +BROPATH=`bash -c %(testbase)s/../../../build/bro-path-dev` +BRO_SEED_FILE=%(testbase)s/../random.seed +TZ=UTC +LC_ALL=C +PATH=%(testbase)s/../../../build/src:%(testbase)s/../../../aux/btest:%(default_path)s +TEST_DIFF_CANONIFIER=%(testbase)s/../scripts/diff-canonifier +TRACES=%(testbase)s/traces +SCRIPTS=%(testbase)s/../scripts +DIST=%(testbase)s/../../.. +BUILD=%(testbase)s/../../../build diff --git a/testing/external/random.seed b/testing/external/random.seed new file mode 100644 index 0000000000..e70c8f85ef --- /dev/null +++ b/testing/external/random.seed @@ -0,0 +1,17 @@ +2983378351 +1299727368 +0 +310447 +0 +1409073626 +3975311262 +34130240 +1450515018 +1466150520 +1342286698 +1193956778 +2188527278 +3361989254 +3912865238 +3596260151 +517973768 diff --git a/testing/external/scripts/create-new-repo b/testing/external/scripts/create-new-repo new file mode 100755 index 0000000000..111dbbeb37 --- /dev/null +++ b/testing/external/scripts/create-new-repo @@ -0,0 +1,42 @@ +#! /usr/bin/env bash + +cwd=`pwd` + +if [ $# != 2 ]; then + echo "usage: $0 " + exit 1 +fi + +name=`pwd`/$1.git +repo=$2 + +if [ -e $repo ]; then + echo "$repo already exists, aborting." + exit 1 +fi + +if [ -e $name ]; then + echo "$name already exists, aborting." + exit 1 +fi + +mkdir $repo +( cd $repo && git init --bare ) + +git clone $repo $name + +cd $name + +for dir in traces tests Baseline; do + mkdir $dir + touch $dir/.gitignore +done + +ln -s ../btest.cfg . + +cp $cwd/`dirname $0`/test.skeleton tests + +git add * + +git commit -m "Repository initialized." +git push origin master diff --git a/testing/external/scripts/diff-canonifier b/testing/external/scripts/diff-canonifier new file mode 100755 index 0000000000..063f1e4900 --- /dev/null +++ b/testing/external/scripts/diff-canonifier @@ -0,0 +1,5 @@ +#! /usr/bin/env bash +# +# Replace anything which looks like timestamps with XXXs. + +sed 's/[0-9]\{10\}\.[0-9]\{2,8\}/XXXXXXXXXX.XXXXXX/g' diff --git a/testing/external/scripts/test.skeleton b/testing/external/scripts/test.skeleton new file mode 100644 index 0000000000..7f79ab0d3d --- /dev/null +++ b/testing/external/scripts/test.skeleton @@ -0,0 +1,2 @@ +@TEST-EXEC: bro -r $TRACES/trace.pcap mt +@TEST-EXEC: for i in *.log; do btest-diff $i; done diff --git a/testing/external/tests/.gitignore b/testing/external/tests/.gitignore new file mode 100644 index 0000000000..e69de29bb2 diff --git a/testing/external/tests/test.skeleton b/testing/external/tests/test.skeleton new file mode 100644 index 0000000000..7f79ab0d3d --- /dev/null +++ b/testing/external/tests/test.skeleton @@ -0,0 +1,2 @@ +@TEST-EXEC: bro -r $TRACES/trace.pcap mt +@TEST-EXEC: for i in *.log; do btest-diff $i; done diff --git a/testing/external/traces/.gitignore b/testing/external/traces/.gitignore new file mode 100644 index 0000000000..e69de29bb2