* origin/topic/bernhard/file-analysis-x509:
Forgot the preamble for the new leak test
(hopefully) last change -> return real opaque vec instead of any_vec
Fix dump-events - it cannot be used with ssl anymore, because openssl does not give the same string results in all versions.
Finishing touches of the x509 file analyzer.
Revert change to only log certificates once per hour.
Change x509 log - now certificates are only logged once per hour.
Fix circular reference problem and a few other small things.
X509 file analyzer nearly done. Verification and most other policy scripts work fine now.
Add verify functionality, including the ability to get the validated chain. This means that it is now possible to get information about the root-certificates that were used to secure a connection.
Second try on the event interface.
Backport crash fix that made it into master with the x509_extension backport from here.
Make x509 certificates an opaque type
rip out x509 code from ssl analyzer. Note that since at the moment the file analyzer does not yet re-populate the info record that means quite a lot of information is simply not available.
parse out extension. One event for general extensions (just returns the openssl-parsed string-value), one event for basicconstraints (is a certificate a CA or not) and one event for subject-alternative-names (only DNS parts).
Very basic file-analyzer for x509 certificates. Mostly ripped from the ssl-analyzer and the topic/bernhard/x509 branch.
- Since it's just the handshake packets out of order, they're no
longer treated as partial connections, which some protocol analyzers
immediately refuse to look at.
- The TCP_Reassembler "is_orig" state failed to change, which led to
protocol analyzers sometimes using the wrong value for that.
- Add a unit test which exercises the Connection::FlipRoles() code
path (i.e. the SYN/SYN-ACK reversal situation).
Addresses BIT-1148.
CONNECT code.
Removal of support analyzers was broken. The code now actually doesn't
delete them immediately anymore but instead just flags them as
disabled. They'll be destroyed with the parent analyzer later.
Also includes a new leak tests exercising the CONNECT code.
Lines starting # with '#' will be ignored, and an empty message aborts
the commit. # On branch topic/robin/http-connect # Changes to be
committed: # modified: scripts/base/protocols/http/main.bro #
modified: scripts/base/protocols/ssl/consts.bro # modified:
src/analyzer/Analyzer.cc # modified: src/analyzer/Analyzer.h #
modified: src/analyzer/protocol/http/HTTP.cc # new file:
testing/btest/core/leaks/http-connect.bro # modified:
testing/btest/scripts/base/protocols/http/http-connect.bro # #
Untracked files: # .tags # changes.txt # conn.log # debug.log # diff #
mpls-in-vlan.patch # newfile.pcap # packet_filter.log # reporter.log #
src/PktSrc.cc.orig # weird.log #
The previous behavior was to accomodate SYN/FIN/RST-filtered traces by
not reporting missing data (via the content_gap event) for such
connections. The new behavior always reports gaps for connections that
are established and terminate normally, but sequence numbers indicate
that all data packets of the connection were missed. The behavior can
be reverted by redef'ing "detect_filtered_trace".
In the case multiple FIN packets are seen from a TCP endpoint (e.g.
when one is retransmitted), only the first counted towards a byte in the
sequence space. This could cause a subsequent FIN packet to induce an
incorrect wrap around in the sequence numbers (e.g. the retransmitted
FIN packet now is one sequence number behind the the first) and
misleadingly large connection sizes. The change is to always treat a
FIN packet as counting one byte in to the sequence space.
The Plugin.cc file is now just a standard class, with the interface
changed a bit to make it more easy to write. However, there're still
some conventions that one must follow to make everything work (like
using the right namespace structure).
This commit also includes the option to compile built-in plugins
dynamically instead of statically by adding
SET(BRO_PLUGIN_BUILD_DYNAMIC TRUE) to their CMake config. This hasn't
been tested much yet, and I'm still undecided if it's somethign we
would want to do by default--but we could now if wanted. :)
Also some minor other cleanup of plugin APIs and built infrastructure.
All tested on MacOS only currently.
This includes enhanced GRE headers. GRE tunnels are treated just like
IP-in-IP tunnels by parsing past the GRE header in between the delivery
and payload IP packets.
Includes:
- Cleanup of the plugin API, in particular generally changing
const char* to std::string
- Renaming environment variable BRO_PLUGINS to BRO_PLUGIN_PATH,
defaulting to <prefix>/lib/bro/plugins
- Reworking how dynamic plugins are searched and activated. See
doc/devel/plugins.rst for details.
- New @load-plugin directive to explicitly activate a plugin
- Support for Darwin. (Linux untested right now)
- The init-plugin updates come with support for "make test", "make
sdist", and "make bdist" (see how-to).
- Test updates.
Notes: The new hook mechanism, which allows plugins to hook into Bro's
core a well-defined points, is still essentially untested.
lookup_hostname("localhost") occassionally timed out (after allowed 10
secs) when running test suite on some systems. Not sure why, but
changed to use the Exec module for when block conditions instead as the
scope of the test doesn't depend on a particular type of condition, it
just needs something that will work reliably/quickly.
If the input framework was used to read event streams and
those streams contained records with more than one field, not all
elements of the threading Values were cleaned up.
The reason for this is, that the SendEventStreamEvent function
returned the number of record elements in the outmost record
instead of the number of unrolled elements in the whole vector.
This number is later used to determine how many objects to delete.
Also - add a whole bunch of leak checks for the input framework
that would have caught that and should cover quite a number of
use-cases.
May fix a sporadic failure, in which case it was just not getting enough
time to run or lookup_hostname() taking longer than ~3 seconds. Else,
the new output should give more hints on what's going wrong. In any
case, termination conditions for the test are now explicit.
This is essentially the code from the dynamic-plugin branch except for
some pieces that I have split out into separate, earlier commits.
I'm going to updatre things in this branch going forward.
Integral/floating-point division/modulo by zero in C++ is undefined
behavior, so to prevent such cases in a script from crashing Bro,
they're now reported as an error (with script location information) and
the event handler in which it occurred returns immediately.
* origin/topic/bernhard/hyperloglog: (32 commits)
add clustered leak test for hll. No issues.
make gcc happy
(hopefully) fix refcounting problem in hll/bloom-filter opaque vals. Thanks Robin.
re-use same hash class for all add operations
get hll ready for merging
and forgot a file...
adapt to new structure
fix opaqueval-related memleak.
make it compile on case-sensitive file systems and fix warnings
make error rate configureable
add persistence test not using predetermined random seeds.
update cluster test to also use hll
persistence really works.
well, with this commit synchronizing the data structure should work.. ...if we had consistent hashing.
and also serialize the other things we need
ok, this bug was hard to find.
serialization compiles.
change plugin after feedback of seth
Forgot a file. Again. Like always. Basically.
do away with old file.
...
Thanks to git this merge was less troublesome that I was afraid it
would be. Not all tests pass yet though (and file hashes have changed
unfortunately).
Conflicts:
cmake
doc/scripts/DocSourcesList.cmake
scripts/base/init-bare.bro
scripts/base/protocols/ftp/main.bro
scripts/base/protocols/irc/dcc-send.bro
scripts/test-all-policy.bro
src/AnalyzerTags.h
src/CMakeLists.txt
src/analyzer/Analyzer.cc
src/analyzer/protocol/file/File.cc
src/analyzer/protocol/file/File.h
src/analyzer/protocol/http/HTTP.cc
src/analyzer/protocol/http/HTTP.h
src/analyzer/protocol/mime/MIME.cc
src/event.bif
src/main.cc
src/util-config.h.in
testing/btest/Baseline/coverage.bare-load-baseline/canonified_loaded_scripts.log
testing/btest/Baseline/coverage.default-load-baseline/canonified_loaded_scripts.log
testing/btest/Baseline/istate.events-ssl/receiver.http.log
testing/btest/Baseline/istate.events-ssl/sender.http.log
testing/btest/Baseline/istate.events/receiver.http.log
testing/btest/Baseline/istate.events/sender.http.log
