Various unit test cleanup.

Updated README and collected coverage-related tests in a common dir.

There are still coverage failures resulting from either the following
scripts not being @load'd in the default bro mode:

base/frameworks/time-machine/notice.bro
base/protocols/http/partial-content.bro
base/protocols/rpc/main.bro

Or the following result in errors when @load'd:

policy/protocols/conn/scan.bro
policy/hot.conn.bro

If these are all scripts-in-progress, can we move them all to live
outside the main scripts/ directory until they're ready?

testing/btest/coverage/bare-load-baseline.test

@@ -0,0 +1,14 @@
+# This test is meant to cover whether the set of scripts that get loaded by
+# default in bare mode matches a baseline of known defaults.  The baseline
+# should only need updating if something new is @load'd from init-bare.bro
+# (or from an @load'd descendent of it).
+#
+# As the output has absolute paths in it, we need to remove the common
+# prefix to make the test work everywhere. That's what the sed magic
+# below does. Don't ask. :-)
+
+# @TEST-EXEC: bro -b misc/loaded-scripts
+# @TEST-EXEC: test -e loaded_scripts.log
+# @TEST-EXEC: cat loaded_scripts.log | egrep -v '#' | awk 'NR>0{print $2}' | sed -e ':a' -e '$!N' -e 's/^\(.*\).*\n\1.*/\1/' -e 'ta' >prefix
+# @TEST-EXEC: cat loaded_scripts.log | sed "s#`cat prefix`##g" >canonified_loaded_scripts.log
+# @TEST-EXEC: btest-diff canonified_loaded_scripts.log

testing/btest/coverage/bare-mode-errors.test

@@ -0,0 +1,11 @@
+# Makes sure any given bro script in the scripts/ tree can be loaded in
+# bare mode without error.  btest-bg-run/btest-bg-wait are used to kill off
+# scripts that block after loading, e.g. start listening on a socket.
+#
+# Commonly, this test may fail if one forgets to @load some base/ scripts
+# when writing a new bro scripts.
+#
+# @TEST-EXEC: test -d $DIST/scripts
+# @TEST-EXEC: for script in `find $DIST/scripts -name \*\.bro`; do echo $script;if [[ "$script" =~ listen-clear|listen-ssl|controllee ]]; then rm -rf load_attempt .bgprocs; btest-bg-run load_attempt bro -b $script; btest-bg-wait -k 2; cat load_attempt/.stderr >>allerrors; else bro -b $script 2>>allerrors; fi done || exit 0
+# @TEST-EXEC: cat allerrors | grep -v "received termination signal" | sort | uniq > unique_errors
+# @TEST-EXEC: btest-diff unique_errors

testing/btest/coverage/default-load-baseline.test

@@ -0,0 +1,14 @@
+# This test is meant to cover whether the set of scripts that get loaded by
+# default matches a baseline of known defaults.  When new scripts are
+# added to the scripts/base/ directory, the baseline will usually just need
+# to be updated.
+#
+# As the output has absolute paths in it, we need to remove the common
+# prefix to make the test work everywhere. That's what the sed magic
+# below does. Don't ask. :-)
+
+# @TEST-EXEC: bro misc/loaded-scripts
+# @TEST-EXEC: test -e loaded_scripts.log
+# @TEST-EXEC: cat loaded_scripts.log | egrep -v '#' | awk 'NR>0{print $2}' | sed -e ':a' -e '$!N' -e 's/^\(.*\).*\n\1.*/\1/' -e 'ta' >prefix
+# @TEST-EXEC: cat loaded_scripts.log | sed "s#`cat prefix`##g" >canonified_loaded_scripts.log
+# @TEST-EXEC: btest-diff canonified_loaded_scripts.log

testing/btest/coverage/doc.test

@@ -0,0 +1,7 @@
+# This tests that we're generating bro script documentation for all the
+# available bro scripts.  If this fails, then the genDocSources.sh needs
+# to be run to produce a new DocSourcesList.cmake or genDocSources.sh needs
+# to be updated to blacklist undesired scripts.
+#
+# @TEST-EXEC: $DIST/doc/scripts/genDocSourcesList.sh
+# @TEST-EXEC: cmp $DIST/doc/scripts/DocSourcesList.cmake ./DocSourcesList.cmake

testing/btest/coverage/init-default.test

@@ -0,0 +1,18 @@
+# Makes sure that all base/* scripts are loaded by default via init-default.bro;
+# and that all scripts loaded there in there actually exist.
+#
+# This test will fail if a new bro script is added under the scripts/base/
+# directory and it is not also added as an @load in base/init-default.bro.
+# In some cases, a script in base is loaded based on the bro configuration
+# (e.g. cluster operation), and in such cases, the missing_loads baseline
+# can be adjusted to tolerate that.
+
+#@TEST-EXEC: test -d $DIST/scripts/base
+#@TEST-EXEC: test -e $DIST/scripts/base/init-default.bro
+#@TEST-EXEC: ( cd $DIST/scripts/base && find . -name '*.bro' ) | sort >"all scripts found"
+#@TEST-EXEC: bro misc/loaded-scripts
+#@TEST-EXEC: cat loaded_scripts.log | egrep -v '/build/|/loaded-scripts.bro|#' | awk 'NR>0{print $2}' | sed 's#/./#/#g'  >loaded_scripts.log.tmp
+#@TEST-EXEC: cat loaded_scripts.log.tmp | sed -e ':a' -e '$!N' -e 's/^\(.*\).*\n\1.*/\1/' -e 'ta' >prefix
+#@TEST-EXEC: cat loaded_scripts.log.tmp | sed "s#`cat prefix`#./#g" | sort >init-default.bro
+#@TEST-EXEC: diff -u "all scripts found" init-default.bro | egrep "^-[^-]" > missing_loads
+#@TEST-EXEC: btest-diff missing_loads

testing/btest/coverage/test-all-policy.test

@@ -0,0 +1,12 @@
+# Makes sure that all policy/* scripts are loaded in
+# scripts/test-all-policy.bro and that all scripts loaded there actually exist.
+#
+# This test will fail if new bro scripts are added to the scripts/policy/
+# directory.  Correcting that just involves updating scripts/test-all-policy.bro
+# to @load the new bro scripts.
+
+@TEST-EXEC: test -e $DIST/scripts/test-all-policy.bro
+@TEST-EXEC: test -d $DIST/scripts
+@TEST-EXEC: ( cd $DIST/scripts/policy && find . -name '*.bro' ) | sort >"all scripts found"
+@TEST-EXEC: cat $DIST/scripts/test-all-policy.bro | grep '@load' | sed 'sm^\( *# *\)\{0,\}@load *m./mg' | sort >test-all-policy.bro
+@TEST-EXEC: diff -u "all scripts found" test-all-policy.bro 1>&2