Starting a small framework for doing regression testing with larger

traces and more complex Bro configurations.

That's "Type 2" in the terminology of
http://www-new.bro-ids.org/devel/projects/testing.html

This leverages btest as well and will replace the set of scripts that
the old private test-suite is using. It can accomodate tests with both
public and private traces.

This is a checkpoint commmit, not much there yet in terms of actual
testing. Also still need to figure out a mechanmism to allow for
different baselines based on which branch we're testing.

testing/README

@@ -1,13 +1,12 @@
-This directory contains some of the suites for testing for Bro's
-correct operation:
+This directory contains suites for testing for Bro's correct
+operation:
 
-btest/
-	Tests Bro's logging & bundled scripts. See btest/README for more.
+    btest/
+        An ever-growing set of small unit tests testing Bro's
+        functionality.
 
-istate/
-	Tests Bro's independent state facilities.  These include persistent
-	values and inter-process event/value communication.
+    external/ A framework for downloading additional test sets that
+        run more complex Bro configuration on large traces files. Due
+        to the size, these are not included directly. See the README
+        for more information. 
 
-(Note that the Bro developers maintain a separate test suite for
- Bro's trace analysis capabilities.  This is kept private as it uses
- sensitive raw traces for input.)

testing/external/.gitignore

@@ -0,0 +1 @@
+*.git

testing/external/Makefile

@@ -0,0 +1,11 @@
+
+PUBLIC_REPO=/home/robin/tmp/bro-tests
+
+all:
+	@for repo in *.git; do ( cd $$repo && btest ); done
+
+init:
+	git clone $(PUBLIC_REPO) public.git
+
+pull:
+	@for repo in *.git; do ( cd $$repo && git pull ); done

testing/external/README

@@ -0,0 +1,84 @@
+
+Test Suite for Large Trace Files
+================================
+
+This test-suite runs more complex Bro configurations on larger trace
+files, and compares the results to a preestablished baseline. Due to
+their size, both traces and baseline are not part of the main Bro
+repository but kept externally. In addition to the publically provided
+files, one can also add a local set to the test-suite for running on
+private traces.
+
+Initialization
+--------------
+
+Before the test-suite can be run, one needs to download the traces and
+the corresponding test and baseline files. They are kept in a separate
+git repository and a ``Makefile`` is provided to clone a copy:
+
+.. console:
+
+    > make init
+
+To later update to upstream changes:
+
+.. console:
+
+    > make pull
+
+Running Tests
+-------------
+
+The easiest way to run all tests is simply typing ``make``. Doing so
+will iterate through all ``*.git`` repositories and run ``btest`` in
+each of them.
+
+Alternatively, one can also manually run all tests inside a single
+test repository:
+
+.. console:
+
+    > cd public.git
+    > btest
+
+All the standard ``btest`` options can be used to run individual
+tests, get diagnostic output, etc.
+
+Updating Baseline
+-----------------
+
+To update a test's baseline, first run ``btest`` in update mode:
+
+.. console:
+
+    > cd public.git
+    > btest -U tests/test-you-want-to-update
+
+Then use ``git`` to commit the changes and push the changes upstream
+as usual.
+
+Adding a Local Repository
+-------------------------
+
+One can add local non-public set of tests (potentially using private
+traces) by creating a git repository of a similar structure as the
+public one. A helper script is provided to set up such a repository:
+
+.. console:
+
+    > ./scripts/create-new-repo private-repo /home/git/bro-private-repo
+
+The first argument is the local name of the repository (it will be
+cloned into ``<name>.git``); and then second is the path to the (bare)
+git repository. The repository will be initialized with a few standard
+directories as well as a skeleton test in ``<name>.git/tests``. You
+can then edit files as needed (and in particular add traces to
+``<name>.git/traces``).
+
+If you want the repository to be located somewhere else than the local
+machine, just push the created local repository to the right place,
+delete ``<name>.git``, and clone from the remote location. Likewise,
+if you already have such a private test repository that you want to
+include into the test suite, clone it directly into
+``<name>.git/tests``.
+

testing/external/btest.cfg

@@ -0,0 +1,18 @@
+[btest]
+TestDirs    = tests
+TmpDir      = %(testbase)s/.tmp
+BaselineDir = %(testbase)s/Baseline
+IgnoreDirs  = .svn CVS .tmp
+IgnoreFiles = *.tmp *.swp #* *.trace .gitignore
+
+[environment]
+BROPATH=`bash -c %(testbase)s/../../../build/bro-path-dev`
+BRO_SEED_FILE=%(testbase)s/../random.seed
+TZ=UTC
+LC_ALL=C
+PATH=%(testbase)s/../../../build/src:%(testbase)s/../../../aux/btest:%(default_path)s
+TEST_DIFF_CANONIFIER=%(testbase)s/../scripts/diff-canonifier
+TRACES=%(testbase)s/traces
+SCRIPTS=%(testbase)s/../scripts
+DIST=%(testbase)s/../../..
+BUILD=%(testbase)s/../../../build

testing/external/random.seed

@@ -0,0 +1,17 @@
+2983378351
+1299727368
+0
+310447
+0
+1409073626
+3975311262
+34130240
+1450515018
+1466150520
+1342286698
+1193956778
+2188527278
+3361989254
+3912865238
+3596260151
+517973768

testing/external/scripts/create-new-repo

@@ -0,0 +1,42 @@
+#! /usr/bin/env bash
+
+cwd=`pwd`
+
+if [ $# != 2 ]; then
+    echo "usage: $0 <name> <dst-repo-dir>"
+    exit 1
+fi
+
+name=`pwd`/$1.git
+repo=$2
+
+if [ -e $repo ]; then
+    echo "$repo already exists, aborting."
+    exit 1
+fi
+
+if [ -e $name ]; then
+    echo "$name already exists, aborting."
+    exit 1
+fi
+
+mkdir $repo
+( cd $repo && git init --bare )
+
+git clone $repo $name
+
+cd $name
+
+for dir in traces tests Baseline; do
+    mkdir $dir
+    touch $dir/.gitignore
+done
+
+ln -s ../btest.cfg .
+
+cp $cwd/`dirname $0`/test.skeleton tests
+
+git add *
+
+git commit -m "Repository initialized."
+git push origin master

testing/external/scripts/diff-canonifier

@@ -0,0 +1,5 @@
+#! /usr/bin/env bash
+#
+# Replace anything which looks like timestamps with XXXs.
+
+sed 's/[0-9]\{10\}\.[0-9]\{2,8\}/XXXXXXXXXX.XXXXXX/g'

testing/external/scripts/test.skeleton

@@ -0,0 +1,2 @@
+@TEST-EXEC: bro -r $TRACES/trace.pcap mt
+@TEST-EXEC: for i in *.log; do btest-diff $i; done

testing/external/tests/test.skeleton

@@ -0,0 +1,2 @@
+@TEST-EXEC: bro -r $TRACES/trace.pcap mt
+@TEST-EXEC: for i in *.log; do btest-diff $i; done