- Introducing analyzer::<protocol> namespaces.
- Moving protocol-specific events out of events.bif into analyzer/protocol/<protocol>/events.bif
- Moving ARP over (even though it's not an actual analyzer).
- Moving NetFlow over (even though it's not an actual analyzer).
- Moving MIME over (even though it's not an actual analyzer).
Some of the unit tests revealed different versions of libmagic could
give different mime types for the same input file and magic database.
One way that could happen is because of the use of hardcoded/builtin
token (word) comparisons for ascii files -- MAGIC_NO_CHECK_TOKENS flag
will prevent that from being used (and it's obsoleted in newer
libmagics).
The other problem looked like a bug fixed as of 5.05 where
a match in the magic database that doesn't have a verbose description
but does have a mime type won't actually return that mime type due to
the the missing description. The one case where that kept popping up
was in 5.04 not beign able to identify application/x-dosexec, so I added
a description to the top-level match for that to workaround the issue.
and fix up the hll scripts for it.
Conflicts:
scripts/base/frameworks/sumstats/plugins/__load__.bro
testing/btest/scripts/base/frameworks/measurement/basic.bro
- It's derived from the magic database of libmagic 5.14, but with most
everything not related to mime types removed.
- The custom database is always used by default for mime detection, but
the more verbose file type detection will fall back on the default
libmagic installation's database. The result is: mime type strings
are now guaranteed to be consistent across platforms, but the verbose
file type descriptions are not.
- The custom database gets installed in $prefix/share/bro/magic, and
should even be extensible if files with new patterns are added inside
the directory.
- The search path for the mime magic database can be controlled via
BROMAGIC environment variable.
- Remove mime_desc field from ftp.log.
- Stop using the mime/file type canonifier with unit tests.
- libmagic >= 5.04 is now a requirement.
And added an event called "event_queue_flush_point" to mark where that
occured in the event stream. The FAF now uses an explicit event queue
flush instead of buffering input in order to wait for a file handle to
be returned from script-layer.
hyperloglog.h was missing guards and randomly deleting memory at
addresses equal to variable contents.
I am not entirely sure why that did not crash before...
- FileAnalysis::Info is now just a record used for logging, the fa_file
record type is defined in init-bare.bro as the analogue to a
connection record.
- Starting to transfer policy hook triggers and analyzer results to
events.
