BIT-1054 #merged
* origin/topic/seth/unified2-analyzer:
Fixes in case a packet isn't seen that matches an event.
Finished work on unified2 analyzer.
Fixed some tests.
Working unified2 analyzer.
Unified2 file analyzer updated to new plugin style.
Adding the unified2 analyzer.
Conflicts:
testing/btest/Baseline/coverage.bare-load-baseline/canonified_loaded_scripts.log
testing/btest/Baseline/coverage.default-load-baseline/canonified_loaded_scripts.log
Nice solution with the ComponentManager/TaggedComponent!
BIT-1049 #Merged Merged into master.
* origin/topic/jsiwek/faf-updates:
Fix some build errors.
Minor fix to file/protocol analyzer plugin reference doc.
Internal refactoring of how plugin components are tagged/managed.
Factor out the need for a tag field in Files::AnalyzerArgs record.
Add a distinct tag class for file analyzers.
Fix various documentation, mostly related to file analysis.
CompositeHash.
We do this by hashing values added to a BloomFilter another time more
with a stable hash seeded only by either the filter's name or the
global_hash_seed (or Bro's random() seed if neither is defined).
I'm also adding a new bif bloomfilter_internal_state() that returns a
string representation of a Bloom filter's current internal state. This
is solely for writing tests that check that the filters end up
consistent when seeded with the same value.
* origin/topic/seth/faf-updates: (27 commits)
Undoing the FTP tests I updated earlier.
Update the last two btest FAF tests.
File analysis fixes and test updates.
Fix a bug with getting analyzer tags.
A few test updates.
Some tests work now (at least they all don't fail anymore!)
Forgot a file.
Added protocol description functions that provide a super compressed log representation.
Fix a bug where orig file information in http wasn't working right.
Added mime types to http.log
Clean up queued but unused file_over_new_connections event args.
Add jar files to the default MHR lookups.
Adding CAB files for MHR checking.
Improve malware hash registry script.
Fix a small issue with finding smtp entities.
Added support for files to the notice framework.
Make the custom libmagic database a git submodule.
Add an is_orig parameter to file_over_new_connection event.
Make magic for emitting application/msword mime type less strict.
Disable more libmagic builtin checks that override the magic database.
...
Conflicts:
doc/scripts/DocSourcesList.cmake
scripts/base/init-bare.bro
scripts/test-all-policy.bro
testing/btest/Baseline/coverage.bare-load-baseline/canonified_loaded_scripts.log
testing/btest/Baseline/coverage.default-load-baseline/canonified_loaded_scripts.log
- Fix examples/references in the file analysis how-to/usage doc.
- Add Broxygen-generated docs for file analyzer plugins.
- Break FTP::Info type declaration out in to its own file to get
rid of some circular dependencies (between s/b/p/ftp/main and
s/b/p/ftp/utils).
in.
No more manual includes to pull them in.
(It doesn't quite work fully automatically yet for some bifs that need
script-level types defined, like the input and logging frameworks.
They still do a manual "@load foo.bif" in their main.bro to get the
order right. It's a bit tricky to fix that and would probably need
splitting main.bro into two parts; not sure that's worth it.)
I'm moving the new files into a subdirectory probabilistic, and into a
corresponding namespace. We can later put code for the other
probabilistic data structures there as well.
* origin/topic/matthias/bloom-filter: (45 commits)
Implement and test Bloom filter merging.
Make hash functions equality comparable.
Make counter vectors mergeable.
Use half adder for bitwise addition and subtraction.
Fix and test counting Bloom filter.
Implement missing CounterVector functions.
Tweak hasher interface.
Add missing include for GCC.
Fixing for unserializion error.
Small fixes and style tweaks.
Only serialize Bloom filter type if available.
Create hash policies through factory.
Remove lingering debug code.
Factor implementation and change interface.
Expose Bro's linear congruence PRNG as utility function.
H3 does not check for zero length input.
Support seeding for hashers.
Add utility function to access first random seed.
Update H3 documentation (and minor style nits.)
Make H3 seed configurable.
...
Thanks to git this merge was less troublesome that I was afraid it
would be. Not all tests pass yet though (and file hashes have changed
unfortunately).
Conflicts:
cmake
doc/scripts/DocSourcesList.cmake
scripts/base/init-bare.bro
scripts/base/protocols/ftp/main.bro
scripts/base/protocols/irc/dcc-send.bro
scripts/test-all-policy.bro
src/AnalyzerTags.h
src/CMakeLists.txt
src/analyzer/Analyzer.cc
src/analyzer/protocol/file/File.cc
src/analyzer/protocol/file/File.h
src/analyzer/protocol/http/HTTP.cc
src/analyzer/protocol/http/HTTP.h
src/analyzer/protocol/mime/MIME.cc
src/event.bif
src/main.cc
src/util-config.h.in
testing/btest/Baseline/coverage.bare-load-baseline/canonified_loaded_scripts.log
testing/btest/Baseline/coverage.default-load-baseline/canonified_loaded_scripts.log
testing/btest/Baseline/istate.events-ssl/receiver.http.log
testing/btest/Baseline/istate.events-ssl/sender.http.log
testing/btest/Baseline/istate.events/receiver.http.log
testing/btest/Baseline/istate.events/sender.http.log
- FileAnalysis::Info is now just a record used for logging, the fa_file
record type is defined in init-bare.bro as the analogue to a
connection record.
- Starting to transfer policy hook triggers and analyzer results to
events.
There's now a new directory "src/protocols/", and the plan is for each
protocol analyzer to eventually have its own subdirectory in there
that contains everything it defines (C++/pac/bif). The infrastructure
to make that happen is in place, and two analyzers have been
converted to the new model, HTTP and SSL; there's no further
HTTP/SSL-specific code anywhere else in the core anymore (I believe :-)
Further changes:
- -N lists available plugins, -NN lists more details on what these
plugins provide (analyzers, bif elements). (The latter does not
work for analyzers that haven't been converted yet).
- *.bif.bro files now go into scripts/base/bif/; and
scripts/base/bif/plugins/ for bif files provided by plugins.
- I've factored out the bifcl/binpac CMake magic from
src/CMakeLists.txt to cmake/{BifCl,Binpac}
- There's a new cmake/BroPlugin that contains magic to allow
plugins to have a simple CMakeLists.txt. The hope is that
eventually the same CMakeLists.txt can be used for compiling a
plugin either statically or dynamically.
- bifcl has a new option -c that changes the code it generates so
that it can be used with a plugin.
TODOs:
- "make install" is probably broken.
- Broxygen is probably broken for plugin-defined events.
- event groups are broken (do we want to keep them?)
This is a larger internal change that moves the analyzer
infrastructure to a more flexible model where the available analyzers
don't need to be hardcoded at compile time anymore. While currently
they actually still are, this will in the future enable external
analyzer plugins. For now, it does already add the capability to
dynamically enable/disable analyzers from script-land, replacing the
old Analyzer::Available() methods.
There are three major parts going into this:
- A new plugin infrastructure in src/plugin. This is independent
of analyzers and will eventually support plugins for other parts
of Bro as well (think: readers and writers). The goal is that
plugins can be alternatively compiled in statically or loadead
dynamically at runtime from a shared library. While the latter
isn't there yet, there'll be almost no code change for a plugin
to make it dynamic later (hopefully :)
- New analyzer infrastructure in src/analyzer. I've moved a number
of analyzer-related classes here, including Analyzer and DPM;
the latter now renamed to Analyzer::Manager. More will move here
later. Currently, there's only one plugin here, which provides
*all* existing analyzers. We can modularize this further in the
future (or not).
- A new script interface in base/framework/analyzer. I think that
this will eventually replace the dpm framework, but for now
that's still there as well, though some parts have moved over.
I've also remove the dpd_config table; ports are now configured via
the analyzer framework. For exmaple, for SSH:
const ports = { 22/tcp } &redef;
event bro_init() &priority=5
{
...
Analyzer::register_for_ports(Analyzer::ANALYZER_SSH, ports);
}
As you can see, the old ANALYZER_SSH constants have more into an enum
in the Analyzer namespace.
This is all hardly tested right now, and not everything works yet.
There's also a lot more cleanup to do (moving more classes around;
removing no longer used functionality; documenting script and C++
interfaces; regression tests). But it seems to generally work with a
small trace at least.
The debug stream "dpm" shows more about the loaded/enabled analyzers.
A new option -N lists loaded plugins and what they provide (including
those compiled in statically; i.e., right now it outputs all the
analyzers).
This is all not cast-in-stone yet, for some things we need to see if
they make sense this way. Feedback welcome.
Versus from synchronous function calls, which doesn't work well because
the function call can see a script-layer state that doesn't reflect
the state as it will be in terms of the event/network stream.
A retry happens on every new input and also periodically based on a
timer. If a file handle is returned at those times, the input is
forwarded for analysis, else it keeps retrying until a timeout
threshold.
Added a generic gtpv1_message event generated for any GTP message type.
Added specific events for the create/update/delete PDP context
request/response messages.
Addresses #934.
* topic/robin/exit-after-terminate:
Updating submodule(s).
Fixing exit-after-terminate when used with bare mode.
New option exit_only_after_terminate to prevent Bro from exiting.
Moved this functionality to be internal instead of in the script-layer
event handlers. The issue with the later is that bad things can happen
between the time a reporter event handler is dispatched and the time it
is executed, and if bro crashes in that time, the message may never be
seen/logged.
Addressed #930 (and revisits #836).
* remotes/origin/topic/seth/modbus-merge:
Small modbus documentation update and tiny refactoring.
Final touches to modbus analyzer for now.
Major revisions to Modbus analyzer support (not quite done yet).
put some make-up on Modbus analyser
Modbus analyser, added support: FC=20,21
Modbus analyzer,added support: FC=1,2,15,24
Modbus analyzer, current support: FC=3,4,5,6,7,16,22,23
Closes#915.
This currently supports automatic decapsulation of GTP-U packets on
UDP port 2152.
The GTPv1 headers for such tunnels can be inspected by handling the
"gtpv1_g_pdu_packet" event, which has a parameter of type "gtpv1_hdr".
Analyzer and test cases are derived from submissions by Carsten Langer.
Addresses #690.
This option indicates that the Teredo analyzer should wait until
it sees both sides of a connection using a valid Teredo encapsulation
before issuing a protocol_confirmation. Previous behavior confirmed
on the first instance of a valid encapsulation, which could result
in more false positives (and e.g. bogus entries in known-services.log).
Addresses #890.
Authentication Headers with a Payload Len field set to zero would cause
a crash due to invalid memory allocation because the previous code
assumed Payload Len would always be great enough to contain all
mandatory fields of the header. This changes it so the length of
the header is explicitly checked before attempting to extract fields
located past the minimum length (8 bytes) of an Authentication Header.
Crashes due to this are only possible when handling script-layer events
ipv6_ext_headers, new_packet, esp_packet, or teredo_*. Or also when
implementing one of the discarder_check_* family of functions.
Otherwise, Bro correctly parses past such a header.
- Renamed many data structures to align with most recent standard.
- Reworked modbus events to make them more canonically "Bro".
- Converted the Modbus analyzer to a simpler style for easier maintenance.
- Modbus coil related events still don't work (I haven't finished the
function for converting the data structures).
- Modbus file record events remain incomplete.
* origin/topic/dina/modbus:
put some make-up on Modbus analyser
Modbus analyser, added support: FC=20,21
Modbus analyzer,added support: FC=1,2,15,24
Modbus analyzer, current support: FC=3,4,5,6,7,16,22,23
I cleaned up the code a bit, mainly layout style.
I did not include the *.bro scripts for now, but a test script
../testing/btest/scripts/base/protocols/modbus/events.bro that prints
out the value for each event.
Merged the Modbus traces from the ics repository into a single trace
as input for the test. They currently trigger 20 of the 34 events.
Addresses #870.
