Cleaning up some testing stuff.

- The Makefiles now run btest with "-f diag.log" so that diangnostics
  output will always be written into that file.

- Makefiles now hardcode path to btest to avoid picking up the wrong version
  if in PATH.

- The canonifier scripts now live in testing/scripts, and they are
  used from both btest/ and external/.

- There's a new diff-remove-uids scripts that removed connection UIDs
  for diffing. The external/* tests now use that by default.

- Timestamp removal now has its own script: diff-remove-timestamps.
  diff-canonifier calls that.

- All Makefile have a "brief" target that runs btest with -b.

- The higher-level directories have Makefile to call the subdirs.

testing/scripts/diff-canonifier

@@ -0,0 +1,5 @@
+#! /usr/bin/env bash
+#
+# Default canonifier used with the tests in testing/btest/*.
+
+`dirname $0`/diff-remove-timestamps

testing/scripts/diff-canonifier-external

@@ -0,0 +1,5 @@
+#! /usr/bin/env bash
+#
+# Default canonifier used with the trace-based tests in testing/external/*.
+
+`dirname $0`/diff-remove-timestamps | `dirname $0`/diff-remove-uids

testing/scripts/diff-remove-abspath

@@ -0,0 +1,5 @@
+#! /usr/bin/env bash
+#
+# Replace absolute paths with the basename.
+
+sed 's#/\([^/]\{1,\}/\)\{1,\}\([^/]\{1,\}\)#<...>/\2#g'

testing/scripts/diff-remove-timestamps

@@ -0,0 +1,5 @@
+#! /usr/bin/env bash
+#
+# Replace anything which looks like timestamps with XXXs.
+
+sed 's/[0-9]\{10\}\.[0-9]\{2,8\}/XXXXXXXXXX.XXXXXX/g'

testing/scripts/diff-remove-uids

@@ -0,0 +1,21 @@
+#! /usr/bin/awk -f
+#
+# A diff canonifier that removes all connection UIDs.
+
+BEGIN { IFS="\t"; OFS="\t"; }
+
+column > 0 {
+    $column = "XXXXXXXXXXX";
+    }
+
+/^#/ {
+    for ( i = 0; i < NF; ++i ) {
+        if ( $i == "uid" )
+            column = i - 1;
+        }
+    }
+
+{ print }
+
+
+

testing/scripts/doc/example-diff-canonifier.py

@@ -0,0 +1,15 @@
+#!/usr/bin/python
+
+import sys
+import re
+
+# MutableVal derivatives (e.g. sets/tables) don't always generate the same
+# ordering in the reST documentation, so just don't bother diffing
+# the places where example.bro uses them.
+
+RE1 = "\d*/tcp"
+RE2 = "tcp port \d*"
+
+for line in sys.stdin.readlines():
+    if re.search(RE1, line) is None and re.search(RE2, line) is None:
+        print line