I've used the opportunity to also cleanup DPD's expect_connection()
infrastructure, and renamed that bif to schedule_analyzer(), which
seems more appropiate. One can now also schedule more than one
analyzer per connection.
TODOs:
- "make install" is probably broken.
- Broxygen is probably broken for plugin-defined events.
- event groups are broken (do we want to keep them?)
- parallel btest is broken, but I'm not sure why ...
(tests all pass individually, but lots of error when running
in parallel; must be related to *.bif restructuring).
- Document API for src/plugin/*
- Document API for src/analyzer/Analyzer.h
- Document API for scripts/base/frameworks/analyzer
All tests pass with one exception: some Broxygen tests are broken
because dpd_config doesn't exist anymore. Need to update the mechanism
for auto-documenting well-known ports.
This is a larger internal change that moves the analyzer
infrastructure to a more flexible model where the available analyzers
don't need to be hardcoded at compile time anymore. While currently
they actually still are, this will in the future enable external
analyzer plugins. For now, it does already add the capability to
dynamically enable/disable analyzers from script-land, replacing the
old Analyzer::Available() methods.
There are three major parts going into this:
- A new plugin infrastructure in src/plugin. This is independent
of analyzers and will eventually support plugins for other parts
of Bro as well (think: readers and writers). The goal is that
plugins can be alternatively compiled in statically or loadead
dynamically at runtime from a shared library. While the latter
isn't there yet, there'll be almost no code change for a plugin
to make it dynamic later (hopefully :)
- New analyzer infrastructure in src/analyzer. I've moved a number
of analyzer-related classes here, including Analyzer and DPM;
the latter now renamed to Analyzer::Manager. More will move here
later. Currently, there's only one plugin here, which provides
*all* existing analyzers. We can modularize this further in the
future (or not).
- A new script interface in base/framework/analyzer. I think that
this will eventually replace the dpm framework, but for now
that's still there as well, though some parts have moved over.
I've also remove the dpd_config table; ports are now configured via
the analyzer framework. For exmaple, for SSH:
const ports = { 22/tcp } &redef;
event bro_init() &priority=5
{
...
Analyzer::register_for_ports(Analyzer::ANALYZER_SSH, ports);
}
As you can see, the old ANALYZER_SSH constants have more into an enum
in the Analyzer namespace.
This is all hardly tested right now, and not everything works yet.
There's also a lot more cleanup to do (moving more classes around;
removing no longer used functionality; documenting script and C++
interfaces; regression tests). But it seems to generally work with a
small trace at least.
The debug stream "dpm" shows more about the loaded/enabled analyzers.
A new option -N lists loaded plugins and what they provide (including
those compiled in statically; i.e., right now it outputs all the
analyzers).
This is all not cast-in-stone yet, for some things we need to see if
they make sense this way. Feedback welcome.
This allows replacing an ugly openssl-call from one of
the policy scripts. The openssl call is now replaced with
a still-but-less-ugly call to base64_encode.
I do not know if I split the Base64 classes in a "smart" way... :)
* origin/topic/matthias/notary:
Small cosmetic changes.
Give log buffer the correct name.
Simplify delayed logging of SSL records.
Implement delay-token style SSL logging.
More style tweaks: replace spaces with tabs.
Factor notary code into separte file.
Adhere to Bro coding style guidelines.
Enhance ssl.log with information from notary.
Closes#928
These cases should be avoidable by fixing scripts where they occur and
they can also help catch typos that would lead to unintentional runtime
behavior.
Adding this already revealed several scripts where a field in an inlined
record was never removed after a code refactor.
This commit moves the notary script into the policy directory, along with some
architectural changes: the main SSL script now has functionality to add and
remove tokens for a given record. When adding a token, the script delays the
logging until the token has been removed or until the record exceeds a maximum
delay time.
As before, the base SSL script stores all records sequentially and buffers even
non-delayed records for the sake of having an ordered log file. If this turns
out to be not so important, we can easily revert to a simpler logic.
(This is still WiP, some debuggin statements still linger.)
There exists one complication: the new file notary.bro requires the definition
of the SSL::Info record, but as does main.bro. Because I did not really know
where to put the common code (it's not a constant, so ssl/const.bro does not
really fit), I put it into __load.bro__ so that it sticks out for now. If
anybody has an idea how to solve this elegantly, please let me know.
* origin/topic/matthias/opaque:
Add new unit test for opaque serialization.
Migrate entropy testing to opaque.
C++ify RandTest.*
Fix a hard-to-spot bug.
Use more descriptive error message.
Fix the fix :-/.
Fix initialization of hash values.
Be clearer about delegation.
Implement serialization of opaque types.
Update hash BiF documentation.
Migrate free SHA* functions to SHA*Val::digest().
Add missing type name that caused failing tests.
Update base scripts and unit tests.
Simplify hash function BiFs.
Add support for opaque hash values.
Adapt BiF & Bro parser to handle opaque types.
More lexer/parser work.
Implement equivalence relation for opaque types.
Support basic serialization of opaque.
Add opaque type to lexer, parser, and BroType.
Closes#925
Conflicts:
aux/broccoli
consisting of letters [A-Za-z].
I had some bogus HTTP sessions now with the test-suite that reported
data as HTTP because it started with "<!... ". Requiring letters seems
a reasonable constraint.
* vlad/topic/vladg/http-verbs:
A test for HTTP methods, including some horribly illegal requests.
Remove hardcoded HTTP verbs from the analyzer (#741)
I added a "bad_HTTP_request" weird for HTTP request lines that don't
have more than a single word.
Closes#741.
* remotes/origin/topic/seth/modbus-merge:
Small modbus documentation update and tiny refactoring.
Final touches to modbus analyzer for now.
Major revisions to Modbus analyzer support (not quite done yet).
put some make-up on Modbus analyser
Modbus analyser, added support: FC=20,21
Modbus analyzer,added support: FC=1,2,15,24
Modbus analyzer, current support: FC=3,4,5,6,7,16,22,23
Closes#915.
- There are still some broken events in the modbus analyzer because
I don't have traffic to test with (coil and record related events primarily).
- There are a few example scripts in policy/protocols/modbus
In addition to checking for a finished SSL handshake over an FTP
connection, it now also requires that the SSL handshake occurs after
the FTP client requested AUTH GSSAPI, more specifically identifying the
characteristics of GridFTP control channels.
Addresses #891.
* origin/topic/jsiwek/gridftp:
Add memory leak unit test for GridFTP.
Enable GridFTP detection by default. Track/log SSL client certs.
Add analyzer for GSI mechanism of GSSAPI FTP AUTH method.
Add an example of a GridFTP data channel detection script.
In the *service* field of connection records, GridFTP control channels
are labeled as "gridftp" and data channels as "gridftp-data".
Added *client_subject* and *client_issuer_subject* as &log'd fields to
SSL::Info record. Also added *client_cert* and *client_cert_chain*
fields to track client cert chain.
GSI authentication involves an encoded TLS/SSL handshake over the FTP
control session. Decoding the exchanged tokens and passing them to an
SSL analyzer instance allows use of all the familiar script-layer events
in inspecting the handshake (e.g. client/server certificats are
available). For FTP sessions that attempt GSI authentication, the
service field of the connection record will have both "ftp" and "ssl".
One additional change is an FTP server's acceptance of an AUTH request
no longer causes analysis of the connection to cease (because further
analysis likely wasn't possible). This decision can be made more
dynamically at the script-layer (plus there's now the fact that further
analysis can be done at least on the GSSAPI AUTH method).
It relies on the heuristics of GridFTP data channels commonly default to
SSL mutual authentication with a NULL bulk cipher and that they usually
transfer large datasets (default threshold of script is 1 GB). The
script also defaults to skip_further_processing() after detection to try
to save cycles analyzing the large, benign connection.
Also added a script in base/protocols/conn/polling that generalizes the
process of polling a connection for interesting features. The GridFTP
data channel detection script depends on it to monitor bytes
transferred.
- Renamed many data structures to align with most recent standard.
- Reworked modbus events to make them more canonically "Bro".
- Converted the Modbus analyzer to a simpler style for easier maintenance.
- Modbus coil related events still don't work (I haven't finished the
function for converting the data structures).
- Modbus file record events remain incomplete.
* origin/topic/dina/modbus:
put some make-up on Modbus analyser
Modbus analyser, added support: FC=20,21
Modbus analyzer,added support: FC=1,2,15,24
Modbus analyzer, current support: FC=3,4,5,6,7,16,22,23
I cleaned up the code a bit, mainly layout style.
I did not include the *.bro scripts for now, but a test script
../testing/btest/scripts/base/protocols/modbus/events.bro that prints
out the value for each event.
Merged the Modbus traces from the ics repository into a single trace
as input for the test. They currently trigger 20 of the 34 events.
Addresses #870.
no reply package was seen.
In the fixed version rcode is only set when a reply packet was seen.
Updates for the baseline have been commited separately in the
topic/bernhard/dns-fix branch.
* origin/topic/seth/tunnels-merge:
Fixed some problems with the SOCKS analyzer and tests.
Conflicts:
scripts/base/protocols/socks/main.bro
src/socks-analyzer.pac
* origin/topic/seth/tunnels-merge:
SOCKS DPD fixes.
Fix a bug in the SOCKS analyzer.
SOCKS and tunnel test updates.
Updates for the SOCKS analyzer.
Very small updates to the tunnels framework.
