- New, expanded API.
- Calculations moved into plugins.
- Scripts using measurement framework ported.
- Updated the script-land queue implementation to make it more generic.
-
This is a larger internal change that moves the analyzer
infrastructure to a more flexible model where the available analyzers
don't need to be hardcoded at compile time anymore. While currently
they actually still are, this will in the future enable external
analyzer plugins. For now, it does already add the capability to
dynamically enable/disable analyzers from script-land, replacing the
old Analyzer::Available() methods.
There are three major parts going into this:
- A new plugin infrastructure in src/plugin. This is independent
of analyzers and will eventually support plugins for other parts
of Bro as well (think: readers and writers). The goal is that
plugins can be alternatively compiled in statically or loadead
dynamically at runtime from a shared library. While the latter
isn't there yet, there'll be almost no code change for a plugin
to make it dynamic later (hopefully :)
- New analyzer infrastructure in src/analyzer. I've moved a number
of analyzer-related classes here, including Analyzer and DPM;
the latter now renamed to Analyzer::Manager. More will move here
later. Currently, there's only one plugin here, which provides
*all* existing analyzers. We can modularize this further in the
future (or not).
- A new script interface in base/framework/analyzer. I think that
this will eventually replace the dpm framework, but for now
that's still there as well, though some parts have moved over.
I've also remove the dpd_config table; ports are now configured via
the analyzer framework. For exmaple, for SSH:
const ports = { 22/tcp } &redef;
event bro_init() &priority=5
{
...
Analyzer::register_for_ports(Analyzer::ANALYZER_SSH, ports);
}
As you can see, the old ANALYZER_SSH constants have more into an enum
in the Analyzer namespace.
This is all hardly tested right now, and not everything works yet.
There's also a lot more cleanup to do (moving more classes around;
removing no longer used functionality; documenting script and C++
interfaces; regression tests). But it seems to generally work with a
small trace at least.
The debug stream "dpm" shows more about the loaded/enabled analyzers.
A new option -N lists loaded plugins and what they provide (including
those compiled in statically; i.e., right now it outputs all the
analyzers).
This is all not cast-in-stone yet, for some things we need to see if
they make sense this way. Feedback welcome.
* origin/topic/bernhard/base64:
and re-enable caching of extracted certs
and add bae64 bif tests.
re-unify classes
and modernize script.
add base64-encode functionality and bif.
Closes#965.
This allows replacing an ugly openssl-call from one of
the policy scripts. The openssl call is now replaced with
a still-but-less-ugly call to base64_encode.
I do not know if I split the Base64 classes in a "smart" way... :)
* origin/topic/matthias/notary:
Small cosmetic changes.
Give log buffer the correct name.
Simplify delayed logging of SSL records.
Implement delay-token style SSL logging.
More style tweaks: replace spaces with tabs.
Factor notary code into separte file.
Adhere to Bro coding style guidelines.
Enhance ssl.log with information from notary.
Closes#928
These cases should be avoidable by fixing scripts where they occur and
they can also help catch typos that would lead to unintentional runtime
behavior.
Adding this already revealed several scripts where a field in an inlined
record was never removed after a code refactor.
- The feature was primarily added to allow the value to be
modified for cluster based intermediate threshold checks
without requiring the user to write the metrics filter
differently for cluster consideration. It's also a nice
way to calculate some related information to the metric
without accidently applying thresholds to that value.
- Fixed a few small bugs in ftp detect-bruteforcing script
and adapted it to the new threshold value selection feature.
This commit moves the notary script into the policy directory, along with some
architectural changes: the main SSL script now has functionality to add and
remove tokens for a given record. When adding a token, the script delays the
logging until the token has been removed or until the record exceeds a maximum
delay time.
As before, the base SSL script stores all records sequentially and buffers even
non-delayed records for the sake of having an ordered log file. If this turns
out to be not so important, we can easily revert to a simpler logic.
(This is still WiP, some debuggin statements still linger.)
- Removed default logging. Now a function is available for the new
$period_finished filter field to get the same behavior for logging
named Metrics::write_log.
- Added index rollups for getting multiple metrics result values
as the same time.
* topic/robin/intel-framework-merge: (22 commits)
Fixing tests after intel-framework merge.
Extracting URLs from message bodies over SMTP and sending them to Intel framework.
Small comment updates in the Intel framework CIF support.
Intelligence framework documentation first draft.
Only the manager tries to read files with the input framework now.
Initial support for Bro's Intel framework with the Collective Intelligence Framework.
Initial API for Intel framework is complete.
Fixed an issue with cluster data distribution.
Updating some intel framework test baselines.
Reworked cluster intelligence data distribution mechanism and fixed tests.
Lots more intelligence checking in SMTP traffic.
Added intelligence check for "Received" path checking and a bit of reshuffling.
Added sources to the intel log.
Fixing a problem with intel distribution on clusters.
Updated intel framework test to include matching.
Restructuring the scripts that feed data into the intel framework slightly.
One test for cluster transparency of the intel framework.
Fixed a cluster support bug.
Intelligence framework checkpoint
Major updates to fix the Intel framework API.
...
Closes#914.
* remotes/origin/topic/seth/modbus-merge:
Small modbus documentation update and tiny refactoring.
Final touches to modbus analyzer for now.
Major revisions to Modbus analyzer support (not quite done yet).
put some make-up on Modbus analyser
Modbus analyser, added support: FC=20,21
Modbus analyzer,added support: FC=1,2,15,24
Modbus analyzer, current support: FC=3,4,5,6,7,16,22,23
Closes#915.
* vlad/topic/vladg/dns_txt_queries:
Add detection rate threshold for MHR.
Make sure lookup_hostname_txt isn't checking lookup_hostname's cache for answers.
A couple of lookup_hostname_txt fixes.
Reverting the earlier function-join during merging.
- There are still some broken events in the modbus analyzer because
I don't have traffic to test with (coil and record related events primarily).
- There are a few example scripts in policy/protocols/modbus
