* origin/topic/bernhard/metrics-bug:
add comment for seth to make us not forget about the copy statements
fix the fix (thanks seth)
duct-tape fix of values not propagating after intermediate check in cluster environments.
Fixing coverage.bare-mode-errors test.
observed elements.
Add methods to merge with and without pruning (before only merge
method was with pruning, which invalidates the number of total
observed elements)
* origin/topic/seth/metrics-merge: (70 commits)
Added protocol to the traceroute detection script.
Added an automatic state limiter for threshold based SumStats.
Removed some dead code in scan.bro
Renamed a plugin hook in sumstats framework.
Move loading variance back to where it should be alphabetically.
Fix a bug with path building in FTP. Came up when changing the path utils.
Fix a few tests.
SumStats test checkpoint.
SumStats tests pass.
Checkpoint for SumStats rename.
Fix another occasional reporter error.
Small updates to hopefully correct reporter errors leading to lost memory.
Trying to fix a state maintenance issue.
Updating DocSourcesList
Updated FTP bruteforce detection and a few other small changes.
Test updates and cleanup.
Fixed the measurement "sample" plugin.
Fix path compression to include removing "/./".
Removed the example metrics scripts. Better real world examples exist now.
Measurement framework is ready for testing.
...
I am not (entirely) sure that this is mathematically correct, but
I am (more and more) getting the feeling that it... might be.
In any case - this was the last step and now it should work
in cluster settings.
Note: merging top-k data structures is not yet possible (and is
actually quite awkward/expensive). I will have to think about
how to do that for a bit...
- It's derived from the magic database of libmagic 5.14, but with most
everything not related to mime types removed.
- The custom database is always used by default for mime detection, but
the more verbose file type detection will fall back on the default
libmagic installation's database. The result is: mime type strings
are now guaranteed to be consistent across platforms, but the verbose
file type descriptions are not.
- The custom database gets installed in $prefix/share/bro/magic, and
should even be extensible if files with new patterns are added inside
the directory.
- The search path for the mime magic database can be controlled via
BROMAGIC environment variable.
- Remove mime_desc field from ftp.log.
- Stop using the mime/file type canonifier with unit tests.
- libmagic >= 5.04 is now a requirement.
- FileAnalysis::Info is now just a record used for logging, the fa_file
record type is defined in init-bare.bro as the analogue to a
connection record.
- Starting to transfer policy hook triggers and analyzer results to
events.
I've used the opportunity to also cleanup DPD's expect_connection()
infrastructure, and renamed that bif to schedule_analyzer(), which
seems more appropiate. One can now also schedule more than one
analyzer per connection.
TODOs:
- "make install" is probably broken.
- Broxygen is probably broken for plugin-defined events.
- event groups are broken (do we want to keep them?)
- parallel btest is broken, but I'm not sure why ...
(tests all pass individually, but lots of error when running
in parallel; must be related to *.bif restructuring).
- Document API for src/plugin/*
- Document API for src/analyzer/Analyzer.h
- Document API for scripts/base/frameworks/analyzer
