* topic/robin/intel-framework-merge: (22 commits)
Fixing tests after intel-framework merge.
Extracting URLs from message bodies over SMTP and sending them to Intel framework.
Small comment updates in the Intel framework CIF support.
Intelligence framework documentation first draft.
Only the manager tries to read files with the input framework now.
Initial support for Bro's Intel framework with the Collective Intelligence Framework.
Initial API for Intel framework is complete.
Fixed an issue with cluster data distribution.
Updating some intel framework test baselines.
Reworked cluster intelligence data distribution mechanism and fixed tests.
Lots more intelligence checking in SMTP traffic.
Added intelligence check for "Received" path checking and a bit of reshuffling.
Added sources to the intel log.
Fixing a problem with intel distribution on clusters.
Updated intel framework test to include matching.
Restructuring the scripts that feed data into the intel framework slightly.
One test for cluster transparency of the intel framework.
Fixed a cluster support bug.
Intelligence framework checkpoint
Major updates to fix the Intel framework API.
...
Closes#914.
* origin/topic/seth/intel-framework: (21 commits)
Extracting URLs from message bodies over SMTP and sending them to Intel framework.
Small comment updates in the Intel framework CIF support.
Intelligence framework documentation first draft.
Only the manager tries to read files with the input framework now.
Initial support for Bro's Intel framework with the Collective Intelligence Framework.
Initial API for Intel framework is complete.
Fixed an issue with cluster data distribution.
Updating some intel framework test baselines.
Reworked cluster intelligence data distribution mechanism and fixed tests.
Lots more intelligence checking in SMTP traffic.
Added intelligence check for "Received" path checking and a bit of reshuffling.
Added sources to the intel log.
Fixing a problem with intel distribution on clusters.
Updated intel framework test to include matching.
Restructuring the scripts that feed data into the intel framework slightly.
One test for cluster transparency of the intel framework.
Fixed a cluster support bug.
Intelligence framework checkpoint
Major updates to fix the Intel framework API.
Checkpoint commit. This is all a huge mess right now. :)
...
Closes#914.
To do this, the document root and autogenerated bro script docs
are rsync'd in to Sphinx's working directory instead of a plain
recursive copy. There are still some index files that are auto
generated on each build and will trigger Sphinx to re-build them
since it thinks they changed, but overall the build is much faster
this way.
One limitation of this is that old files in the Sphinx work dir
don't automatically get cleaned up (e.g. if you remove some static
documentation from the /doc it will still be in
build/doc/sphinx-sources). So a `make docclean` or at least
`make broxygenclean` is needed at least in that case. (For now,
rsync --delete isn't the right answer since the destination of
the autogenerated stuff overlaps with the document root, the separate
rsyncs end up clobbering each other.)
- "src-ip" and "dst-ip" conditions can now use IPv6 addresses/subnets.
They must be written in colon-hexadecimal representation and enclosed
in square brackets (e.g. [fe80::1]). Addresses #774.
- "icmp6" is now a valid protocol for use with "ip-proto" and "header"
conditions. This allows signatures to be written that can match
against ICMPv6 payloads. Addresses #880.
- "ip6" is now a valid protocol for use with the "header" condition.
(also the "ip-proto" condition, but it results in a no-op in that
case since signatures apply only to the inner-most IP packet when
packets are tunneled). This allows signatures to match specifically
against IPv6 packets (whereas "ip" only matches against IPv4 packets).
- "ip-proto" conditions can now match against IPv6 packets. Before,
IPv6 packets were just silently ignored which meant DPD based on
signatures did not function for IPv6 -- protocol analyzers would only
get attached to a connection over IPv6 based on the well-known ports
set in the "dpd_config" table.
* origin/topic/jsiwek/gridftp:
Add memory leak unit test for GridFTP.
Enable GridFTP detection by default. Track/log SSL client certs.
Add analyzer for GSI mechanism of GSSAPI FTP AUTH method.
Add an example of a GridFTP data channel detection script.
more cases.
It will now not only fire after table-reads have been completed,
but also after the last event of a whole-file-read (or whole-db-read, etc.).
The interface also has been extended a bit to allow readers to
directly fire the event should they so choose. This allows the
event to be fired in direct table-setting/event-sending modes,
which was previously not possible.
In the *service* field of connection records, GridFTP control channels
are labeled as "gridftp" and data channels as "gridftp-data".
Added *client_subject* and *client_issuer_subject* as &log'd fields to
SSL::Info record. Also added *client_cert* and *client_cert_chain*
fields to track client cert chain.
GSI authentication involves an encoded TLS/SSL handshake over the FTP
control session. Decoding the exchanged tokens and passing them to an
SSL analyzer instance allows use of all the familiar script-layer events
in inspecting the handshake (e.g. client/server certificats are
available). For FTP sessions that attempt GSI authentication, the
service field of the connection record will have both "ftp" and "ssl".
One additional change is an FTP server's acceptance of an AUTH request
no longer causes analysis of the connection to cease (because further
analysis likely wasn't possible). This decision can be made more
dynamically at the script-layer (plus there's now the fact that further
analysis can be done at least on the GSSAPI AUTH method).
* origin/fastpath:
Fix construction of ip6_ah (Authentication Header) record values.
Update compile/dependency docs for OS X.
Adjusting Mac binary packaging script.
Unit test reliability adjustment.
Adjusting some unit tests that do cluster communication.
Small change to non-blocking DNS initialization.
reorder a few statements in scan.l to make 1.5msecs etc work.
Add missing description of interval "msec" unit.
Improved description of pattern by clarifying the issue of operand
order and difference between exact and embedded matching.
I've only tested that it compiles, not whether it still works. The
fact that we don't have any tests for this makes me uneasy ...
* remotes/origin/topic/seth/elasticsearch: (35 commits)
Some documentation updates for elasticsearch plugin.
Temporarily removing the ES timeout because it works with signals and is incompatible with Bro threads.
Changed ES index names to localtime and added a meta index.
New script for easily duplicating logs to ElasticSearch.
Some better elasticsearch reliability.
Fixed small elasticsearch problem in configure output.
Re-adding the needed call to FinishedRotation in the ES writer plugin.
Tiny updates.
Bringing elasticsearch branch up to date with master.
Adding a define to make the stdint C macros available.
Adding an extra header.
Fixed a bug with messed up time value passing to elasticsearch.
Small updates and a little standardization for config.h.in naming.
Bug fixes.
Bug fix and feature.
Forgot to call the parent method for DoHeartBeat.
Changed the escaping method.
Flush logs to ES daemon as Bro is shutting down.
Reduce the batch size to 1000 and add a maximum time interval for batches.
Reworked bulk operation string construction to use ODesc and added json escaping.
...
* topic/robin/tunnels-merge: (51 commits)
Updating baselines and NEWS.
Remove &synchronized from Tunnel::active table.
Refactor of interal tunnel analysis code.
Add state management of NetSessions's IP tunnel map.
Add "encap_hdr_size" option back in.
Script-layer tunnel interface cleanup.
Fix performance problem checking packet encapsulation. (addresses #830)
Adding a SOCKS test case.
Updating DataSeries baselines.
Moving my todos over to the tracker ticket.
Extend weird names that occur in core packet processing during decapsulation.
Add Teredo analysis option to reduce false positive decapsulation.
Just some cleanup/documentation of new tunnel-handling code.
Memory leak fixes
Add a config.h definition for IPPROTO_IPV4.
Add AYIYA tunnel decapsulation unit test.
Add Teredo-specific events.
Refactor some of the NetSessions routines that recurse on IP packets.
Add independent options to toggle the different decapsulation methods
Add more sanity checks before recursing on encapsulated IP packets.
...
Conflicts:
src/event.bif
