Changes to the testing/external infrastructure.

The traces for external test-suites are no longer kept inside the
repositories themselves but downloaded separately via curl. This is
because git is pretty bad at dealing with large files.

See the README for more information.

testing/external/.gitignore

@@ -1,3 +1,4 @@
 *.git
 diag.log
 bro-testing
+.proxy

testing/external/README

@@ -12,20 +12,28 @@ private traces.
 Initialization
 --------------
 
-Before the test-suite can be run, one needs to download the traces and
+Before the test-suite can be run, one needs to download the necessary
-the corresponding test and baseline files. They are kept in a separate
+files. Test and baselines are kept in git repositories, while any
-git repository and a ``Makefile`` is provided to clone a copy:
+traces are download directly. A ``Makefile`` is provided to get
+everything that's needed initially:
 
 .. console:
 
     > make init
 
+If you need a proxy to download the traces, enter it into a file
+``.proxy`` either in the top-level directory or inside one of the
+repositories.
+
 To later update to upstream changes:
 
 .. console:
 
     > make pull
 
+This updates the tests and the traces as necessary.
+
+
 Running Tests
 -------------
 
@@ -53,7 +61,7 @@ To update a test's baseline, first run ``btest`` in update mode:
 .. console:
 
     > cd bro-testing
-    > btest -U tests/test-you-want-to-update
+    > btest -u tests/test-you-want-to-update
 
 Then use ``git`` to commit the changes and push the changes upstream
 as usual.
@@ -63,20 +71,29 @@ Adding a Local Repository
 
 One can add local non-public set of tests (potentially using private
 traces) by creating a git repository of a similar structure as the
-public one. A helper script is provided to set up such a repository:
+public one.
+
+If you already have such a private test repository that you want to
+include into the test suite, clone it directly into ``<repo-name>``.
+
+If you want to create a new private repository, there's a helper
+script to set that up:
 
 .. console:
 
     > ./scripts/create-new-repo <repo-name> <repo-url>
 
 The first argument is the local name of the repository (it will be
-cloned into ``<repo-name>``); and then second is the path to the git
+cloned into ``<repo-name>``); and the second is the URL of the git
 repository. The repository will be initialized with a few standard
-directories as well as a skeleton test in ``<name>/tests``. You
+directories as well as a skeleton test in ``<name>/tests``. You can
-can then edit files as needed (and in particular add traces to
+then edit files as needed. You add trace files by editing
-``<name>/traces``).
+``Traces/traces.cfg``; see the comments in there. For each trace, you
+also need to calculate a checksum with ``md5sum`` and put it into
+``<url>.md5sum``. The scripts use this to decide if they need to
+redownload the trace. Accordingly, if you update a trace, make sure to
+also recalculate its checksum. Note that the traces will be downloaded
+to ``Traces/`` but must not be added to the git repostiory; there's a
+``.gitignore`` installed to prevent that.
 
-If you already have such a private test repository that you want to
-include into the test suite, clone it directly into
-``<repo-name>``.
 

testing/external/scripts/create-new-repo

@@ -35,6 +35,7 @@ done
 ln -s ../subdir-btest.cfg ./btest.cfg
 
 cp $cwd/`dirname $0`/skel/test.skeleton tests
+cp $cwd/`dirname $0`/skel/traces.cfg .
 cp $cwd/`dirname $0`/skel/Makefile .
 cp $cwd/`dirname $0`/skel/.gitignore .
 

testing/external/scripts/skel/.gitignore

@@ -1 +1,3 @@
+Traces
 diag.log
+.proxy

testing/external/scripts/skel/Makefile

@@ -2,10 +2,13 @@
 DIAG=diag.log
 BTEST=../../../aux/btest/btest
 
-all:
+all: update-traces
 	@rm -f $(DIAG)
 	@$(BTEST) -f $(DIAG)
 
-brief:
+brief: update-traces
 	@rm -f $(DIAG)
 	@$(BTEST) -b -f $(DIAG)
+
+update-traces:
+	../scripts/update-traces Traces

testing/external/scripts/skel/traces.cfg

@@ -0,0 +1,7 @@
+#
+# Format:
+#
+# <url> <md5sum>
+#
+# Use scripts/make-md5sum to calculate the md5sum.
+#

testing/external/scripts/update-traces

@@ -0,0 +1,76 @@
+#! /usr/bin/env bash
+#
+# Downloads all traces as specified in <cwd>/traces.cfg to directory $1.
+#
+# traces.cfg must consist of lines of the form "<url> <md5sum>"
+
+if [ "$1" == "" ]; then
+    echo "usage: `basename $0` <traces-directory>"
+    exit 1
+fi
+
+if [ ! -e $cfg ]; then
+    echo "No $cfg found."
+    exit 1
+fi
+
+cfg=traces.cfg
+
+for proxy in .proxy ../.proxy; do
+    if [ -e $proxy ]; then
+        proxy=`cat $proxy | head -1 | awk '{print $1}'`
+        echo Using proxy $proxy ...
+        proxy="ALL_PROXY=$proxy"
+        break
+    fi
+done
+
+cat $cfg | while read line; do
+
+    if echo $line | grep -q '^[ \t]*$'; then
+        continue
+    fi
+
+    if echo $line | grep -q '^[ \t]*#'; then
+        continue
+    fi
+
+    url=`echo $line | awk '{print $1}'`
+    auth=`echo $line | awk '{print $3}'`
+
+    file=$1/`echo $url | sed 's#^.*/##g'`
+    fp=$file.md5sum
+
+    if [ "$auth" != "" ]; then
+        auth="-u $auth"
+    fi
+
+    # Get the fingerprint file.
+    if ! eval "$proxy curl $auth -fsS --anyauth $url.md5sum -o $fp.tmp"; then
+        echo "Error: Could not get $url.fingerprint, skipping download."
+        continue
+    fi
+
+    download=0
+
+    if [ -e $fp ]; then
+        if ! cmp -s $fp $fp.tmp; then
+            download=1
+        fi
+    else
+       download=1
+    fi
+
+    if [ "$download" = "1" ]; then
+        echo Getting $url ...
+        echo
+        eval "$proxy curl $auth -f --anyauth $url -o $file"
+        echo
+        mv $fp.tmp $fp
+    else
+        echo "`basename $file` already available."
+    fi
+
+done
+
+