Fixed some test canonifier scripts to read from stdin instead of
from a filename specified as a cmd-line argument. This is needed
in order to be able to reliably use them in a pipeline with other
test canonifiers.
Also removed some unused test canonifier scripts.
- It's derived from the magic database of libmagic 5.14, but with most
everything not related to mime types removed.
- The custom database is always used by default for mime detection, but
the more verbose file type detection will fall back on the default
libmagic installation's database. The result is: mime type strings
are now guaranteed to be consistent across platforms, but the verbose
file type descriptions are not.
- The custom database gets installed in $prefix/share/bro/magic, and
should even be extensible if files with new patterns are added inside
the directory.
- The search path for the mime magic database can be controlled via
BROMAGIC environment variable.
- Remove mime_desc field from ftp.log.
- Stop using the mime/file type canonifier with unit tests.
- libmagic >= 5.04 is now a requirement.
- Skip diffing of debug.log always.
- Skip diffing of reporter.log if it only contains an error about
missing GeoIP support.
- Canonicalize X.509 Distinguished Name subjects since that can vary
depending on installed OpenSSL version.
- The absolute path canonifier was overzealously canonifying relevant
log fields, so it's no longer generally applied to diffing all
baselines. I don't think there's any logs that require local
filesystem path names that aren't already tested by a unit test,
but if any show up in the future, they can be canonified on a
case-by-case basis.
- Removed some logs from being diff'd in the diff-all script
because they're either already covered by a unit test
(load_scripts.log) or because of difficulty/maintainenance
tradeoff (prof.log).
Baselines for the external bro-testing repo still need updating.
This is because libmagic can return different results depending on
installation.
Note, this still doesn't fix all problems: smtp-entities sets md5
depending on the mime-type returned by libmagic ...
- The Makefiles now run btest with "-f diag.log" so that diangnostics
output will always be written into that file.
- Makefiles now hardcode path to btest to avoid picking up the wrong version
if in PATH.
- The canonifier scripts now live in testing/scripts, and they are
used from both btest/ and external/.
- There's a new diff-remove-uids scripts that removed connection UIDs
for diffing. The external/* tests now use that by default.
- Timestamp removal now has its own script: diff-remove-timestamps.
diff-canonifier calls that.
- All Makefile have a "brief" target that runs btest with -b.
- The higher-level directories have Makefile to call the subdirs.
