Thanks to git this merge was less troublesome that I was afraid it
would be. Not all tests pass yet though (and file hashes have changed
unfortunately).
Conflicts:
cmake
doc/scripts/DocSourcesList.cmake
scripts/base/init-bare.bro
scripts/base/protocols/ftp/main.bro
scripts/base/protocols/irc/dcc-send.bro
scripts/test-all-policy.bro
src/AnalyzerTags.h
src/CMakeLists.txt
src/analyzer/Analyzer.cc
src/analyzer/protocol/file/File.cc
src/analyzer/protocol/file/File.h
src/analyzer/protocol/http/HTTP.cc
src/analyzer/protocol/http/HTTP.h
src/analyzer/protocol/mime/MIME.cc
src/event.bif
src/main.cc
src/util-config.h.in
testing/btest/Baseline/coverage.bare-load-baseline/canonified_loaded_scripts.log
testing/btest/Baseline/coverage.default-load-baseline/canonified_loaded_scripts.log
testing/btest/Baseline/istate.events-ssl/receiver.http.log
testing/btest/Baseline/istate.events-ssl/sender.http.log
testing/btest/Baseline/istate.events/receiver.http.log
testing/btest/Baseline/istate.events/sender.http.log
- FileAnalysis::Info is now just a record used for logging, the fa_file
record type is defined in init-bare.bro as the analogue to a
connection record.
- Starting to transfer policy hook triggers and analyzer results to
events.
There's now a new directory "src/protocols/", and the plan is for each
protocol analyzer to eventually have its own subdirectory in there
that contains everything it defines (C++/pac/bif). The infrastructure
to make that happen is in place, and two analyzers have been
converted to the new model, HTTP and SSL; there's no further
HTTP/SSL-specific code anywhere else in the core anymore (I believe :-)
Further changes:
- -N lists available plugins, -NN lists more details on what these
plugins provide (analyzers, bif elements). (The latter does not
work for analyzers that haven't been converted yet).
- *.bif.bro files now go into scripts/base/bif/; and
scripts/base/bif/plugins/ for bif files provided by plugins.
- I've factored out the bifcl/binpac CMake magic from
src/CMakeLists.txt to cmake/{BifCl,Binpac}
- There's a new cmake/BroPlugin that contains magic to allow
plugins to have a simple CMakeLists.txt. The hope is that
eventually the same CMakeLists.txt can be used for compiling a
plugin either statically or dynamically.
- bifcl has a new option -c that changes the code it generates so
that it can be used with a plugin.
TODOs:
- "make install" is probably broken.
- Broxygen is probably broken for plugin-defined events.
- event groups are broken (do we want to keep them?)
This is a larger internal change that moves the analyzer
infrastructure to a more flexible model where the available analyzers
don't need to be hardcoded at compile time anymore. While currently
they actually still are, this will in the future enable external
analyzer plugins. For now, it does already add the capability to
dynamically enable/disable analyzers from script-land, replacing the
old Analyzer::Available() methods.
There are three major parts going into this:
- A new plugin infrastructure in src/plugin. This is independent
of analyzers and will eventually support plugins for other parts
of Bro as well (think: readers and writers). The goal is that
plugins can be alternatively compiled in statically or loadead
dynamically at runtime from a shared library. While the latter
isn't there yet, there'll be almost no code change for a plugin
to make it dynamic later (hopefully :)
- New analyzer infrastructure in src/analyzer. I've moved a number
of analyzer-related classes here, including Analyzer and DPM;
the latter now renamed to Analyzer::Manager. More will move here
later. Currently, there's only one plugin here, which provides
*all* existing analyzers. We can modularize this further in the
future (or not).
- A new script interface in base/framework/analyzer. I think that
this will eventually replace the dpm framework, but for now
that's still there as well, though some parts have moved over.
I've also remove the dpd_config table; ports are now configured via
the analyzer framework. For exmaple, for SSH:
const ports = { 22/tcp } &redef;
event bro_init() &priority=5
{
...
Analyzer::register_for_ports(Analyzer::ANALYZER_SSH, ports);
}
As you can see, the old ANALYZER_SSH constants have more into an enum
in the Analyzer namespace.
This is all hardly tested right now, and not everything works yet.
There's also a lot more cleanup to do (moving more classes around;
removing no longer used functionality; documenting script and C++
interfaces; regression tests). But it seems to generally work with a
small trace at least.
The debug stream "dpm" shows more about the loaded/enabled analyzers.
A new option -N lists loaded plugins and what they provide (including
those compiled in statically; i.e., right now it outputs all the
analyzers).
This is all not cast-in-stone yet, for some things we need to see if
they make sense this way. Feedback welcome.
Versus from synchronous function calls, which doesn't work well because
the function call can see a script-layer state that doesn't reflect
the state as it will be in terms of the event/network stream.
A retry happens on every new input and also periodically based on a
timer. If a file handle is returned at those times, the input is
forwarded for analysis, else it keeps retrying until a timeout
threshold.
Added a generic gtpv1_message event generated for any GTP message type.
Added specific events for the create/update/delete PDP context
request/response messages.
Addresses #934.
* topic/robin/exit-after-terminate:
Updating submodule(s).
Fixing exit-after-terminate when used with bare mode.
New option exit_only_after_terminate to prevent Bro from exiting.
Moved this functionality to be internal instead of in the script-layer
event handlers. The issue with the later is that bad things can happen
between the time a reporter event handler is dispatched and the time it
is executed, and if bro crashes in that time, the message may never be
seen/logged.
Addressed #930 (and revisits #836).
* remotes/origin/topic/seth/modbus-merge:
Small modbus documentation update and tiny refactoring.
Final touches to modbus analyzer for now.
Major revisions to Modbus analyzer support (not quite done yet).
put some make-up on Modbus analyser
Modbus analyser, added support: FC=20,21
Modbus analyzer,added support: FC=1,2,15,24
Modbus analyzer, current support: FC=3,4,5,6,7,16,22,23
Closes#915.
This currently supports automatic decapsulation of GTP-U packets on
UDP port 2152.
The GTPv1 headers for such tunnels can be inspected by handling the
"gtpv1_g_pdu_packet" event, which has a parameter of type "gtpv1_hdr".
Analyzer and test cases are derived from submissions by Carsten Langer.
Addresses #690.
This option indicates that the Teredo analyzer should wait until
it sees both sides of a connection using a valid Teredo encapsulation
before issuing a protocol_confirmation. Previous behavior confirmed
on the first instance of a valid encapsulation, which could result
in more false positives (and e.g. bogus entries in known-services.log).
Addresses #890.
Authentication Headers with a Payload Len field set to zero would cause
a crash due to invalid memory allocation because the previous code
assumed Payload Len would always be great enough to contain all
mandatory fields of the header. This changes it so the length of
the header is explicitly checked before attempting to extract fields
located past the minimum length (8 bytes) of an Authentication Header.
Crashes due to this are only possible when handling script-layer events
ipv6_ext_headers, new_packet, esp_packet, or teredo_*. Or also when
implementing one of the discarder_check_* family of functions.
Otherwise, Bro correctly parses past such a header.
- Renamed many data structures to align with most recent standard.
- Reworked modbus events to make them more canonically "Bro".
- Converted the Modbus analyzer to a simpler style for easier maintenance.
- Modbus coil related events still don't work (I haven't finished the
function for converting the data structures).
- Modbus file record events remain incomplete.
* origin/topic/dina/modbus:
put some make-up on Modbus analyser
Modbus analyser, added support: FC=20,21
Modbus analyzer,added support: FC=1,2,15,24
Modbus analyzer, current support: FC=3,4,5,6,7,16,22,23
I cleaned up the code a bit, mainly layout style.
I did not include the *.bro scripts for now, but a test script
../testing/btest/scripts/base/protocols/modbus/events.bro that prints
out the value for each event.
Merged the Modbus traces from the ics repository into a single trace
as input for the test. They currently trigger 20 of the 34 events.
Addresses #870.
This adds a new parameter of type "icmp6_nd_options" to the ICMPv6
neighbor discovery events (icmp_redirect, icmp_router_solicitation,
icmp_router_advertisement, icmp_neighbor_solicitation,
icmp_neighbor_advertisement) which includes data extracted from
all neighbor discovery options (RFC 4861) that are present in the
ICMPv6 message.
- Make the uid field optional since it's conceptually incorrect
for proxies being treated as tunnels to have it.
- Reordered two fields in the log.
- Reduced the default tunnel expiration interface to something more
reasonable (1 hour).
The "tunnel_port" and "parse_udp_tunnels" options are still gone
as those did not work entirely (e.g. IPv6 support and misnaming
of tunnel_port/udp_tunnel_port).
The Tunnel::yielding_teredo_decapsulation (on by default) makes it so
the Teredo analyzer doesn't attempt to decapsulate payloads when
there's already a sibling analyzer that thinks it's parsing the right
protocol. Sometimes, UDP payloads just happen to look like they are
validly Teredo-encapsulated and doing further analysis on the
decapsulated packet can quickly turn into a weird; this change helps
reduce such weirds.
These are called "teredo_packet", "teredo_authentication",
"teredo_origin_indication", and "teredo_bubble" and may be raised
on a per-packet basis depending on Teredo encapsulation method.
* topic/robin/input-threads-merge: (130 commits)
And now it even compiles after my earlier changes.
A set of input framework refactoring, cleanup, and polishing.
another small memory leak in ascii reader:
and another small memory leak when using streaming reads.
fix another memory lead (when updating tables).
Input framework merge in progress.
filters have been called streams for eternity. And I always was too lazy to change it everywhere...
reactivate network_time check in threading manager. previously this line made all input framework tests fail - it works now. Some of the other recent changes of the threading manager must have fixed that problem.
fix up the executeraw test - now it works for the first time and does not always fail
baselines for the autostart removal.
remove last remnants of autostart, which has been removed for quite a while.
make input framework source (hopefully) adhere to the usual indentation style. No functional changes.
fix two memory leaks which occured when one used filters.
update description to current interface.
rename a couple of structures and make the names in manager fit the api more.
fix memory leak in tables and vectors that are read into tables
fix missing get call for heart beat in benchmark reader.
fix heart_beat_interval -- initialization in constructor does not work anymore (probably due to change in init ordering?)
fix memory leak for tables... nearly completely.
fix a couple more leaks. But - still leaking quite a lot with tables.
...
