* origin/topic/vern/content-gap-history:
Refined state machine update placement to (1) properly deal with gaps capped by clean FIN handshakes, and (1) fix failure to detect split routing.
added 'g' $history character for content gaps
For trace files that require authentication to download, hide
part of the URL in output messages. This avoids leaking potentially
sensitive info when running tests using a continuous integration
service.
Addig a new random seed for external tests.
I added a wrapper around the siphash() function to make calling it a
little bit safer at least.
BIT-1612 #merged
* origin/topic/johanna/bit-1612:
HLL: Fix missing typecast in test case.
Remove the -K/-J options for setting keys.
Add test checking the quality of HLL by adding a lot of elements.
Fix serializing probabilistic hashers.
Baseline updates after hash function change.
Also switch BloomFilters from H3 to siphash.
Change Hashing from H3 to Siphash.
HLL: Remove unnecessary comparison.
Hyperloglog: change calculation of Rho
This also patches a few tests to contain certificates that were removed.
Furthermore, we include the old CA file with the external tests and load
it automatically. Those traces are kind of old now, more and more of the
CAs in them are no longer valid and it does not really make sense to
update them on each change...
* origin/topic/jsiwek/faf-perf:
Adapt HTTP partial content to cache file analysis IDs.
Adapt SSL analyzer to generate file analysis handles itself.
Adapt more of HTTP analyzer to use cached file analysis IDs.
Adapt IRC/FTP analyzers to cache file analysis IDs.
Refactor regex/signature AcceptingSet data structure and usages.
Enforce data size limit when checking files for MIME matches.
Refactor file analysis file ID lookup.
* origin/topic/seth/json-formatter:
Updating a couple of tests.
Expanded support for modifying the timestamp format in the JSON formatter.
Ascii input reader now supports all config options per-input stream.
Added an option to the JSON formatter to use ISO 8601 for timestamps.
Refactored formatters and updated the the writers a bit.
Includes some minor bugfixes and cleanup at various places, including
in old code.
- Several places were just using old variable names or not loading
scripts correctly after they'd been renamed/moved.
- Revert/adjust a change in how HTTP file handles are generated that
broke partial content responses.
- Turn some libmagic builtin checks back on; seems some are actually
useful (e.g. text detection seems to be a builtin). The rule going
forward probably will be only to turn off a builtin if we confirm it
causes issues.
- Removed some tests that are redundant or not necessary anymore because
the generic file analysis tests cover them.
- A couple FTP tests still fail that I think need an actual solution via
script changes.
- It's derived from the magic database of libmagic 5.14, but with most
everything not related to mime types removed.
- The custom database is always used by default for mime detection, but
the more verbose file type detection will fall back on the default
libmagic installation's database. The result is: mime type strings
are now guaranteed to be consistent across platforms, but the verbose
file type descriptions are not.
- The custom database gets installed in $prefix/share/bro/magic, and
should even be extensible if files with new patterns are added inside
the directory.
- The search path for the mime magic database can be controlled via
BROMAGIC environment variable.
- Remove mime_desc field from ftp.log.
- Stop using the mime/file type canonifier with unit tests.
- libmagic >= 5.04 is now a requirement.
- FileAnalysis::Info is now just a record used for logging, the fa_file
record type is defined in init-bare.bro as the analogue to a
connection record.
- Starting to transfer policy hook triggers and analyzer results to
events.
- BRO_PROFILER_FILE now passes .X* templated filenames to mkstemp
for generating unique coverage state files. All test suites
now use this so each Bro instance writes to a unique coverage file.
- Rearranging Makefile targets. The general rule is that if the
all/brief target fails out due to a test failure, then the dependent
coverage target won't run, but can still be invoked directly later.
(e.g. make brief || make coverage)
* origin/topic/script-reference: (50 commits)
A few updates for the FAQ.
Fixing some doc warnings.
Forgot to add protocol identifier support for TLS 1.2
Finished SSL & syslog autodocs.
Adding the draft SSL extension type next_protocol_negotiation.
Fix some documentation errors.
Tweaks.
A set of script-reference polishing.
fixed a couple typos in comments
Add summary documentation to bif files.
Add ssl and syslog script documentation
Add Conn and DNS protocol script documentation. (fixes#731)
Small updates to the default local.bro.
Documentation updates for HTTP & IRC scripts.
SSH&FTP Documentation updates.
Fixing a warning from the documentation generation.
This completes framework documentation package 4.
Minor notice documentation tweaks.
Fix some malformed Broxygen xref roles.
Minor doc tweaks to init-bare.bro.
...
Conflicts:
aux/broccoli
aux/broctl
src/bro.bif
src/strings.bif
Includes:
- Updated baselines for autodoc tests.
- Now excluding stats.bro from external texts, it's not stable.
- Skip diffing of debug.log always.
- Skip diffing of reporter.log if it only contains an error about
missing GeoIP support.
- Canonicalize X.509 Distinguished Name subjects since that can vary
depending on installed OpenSSL version.
- The absolute path canonifier was overzealously canonifying relevant
log fields, so it's no longer generally applied to diffing all
baselines. I don't think there's any logs that require local
filesystem path names that aren't already tested by a unit test,
but if any show up in the future, they can be canonified on a
case-by-case basis.
- Removed some logs from being diff'd in the diff-all script
because they're either already covered by a unit test
(load_scripts.log) or because of difficulty/maintainenance
tradeoff (prof.log).
Baselines for the external bro-testing repo still need updating.
entities.
Before, whether they did depended on libmagic. To do that,
smpt/entities.bro gets a new option `never_calc_md5`.
Also restructuring the tests a bit so that load a common
testing-setup.bro scripts that can set a global configuration.
