* origin/topic/seth/intel-framework: (21 commits)
Extracting URLs from message bodies over SMTP and sending them to Intel framework.
Small comment updates in the Intel framework CIF support.
Intelligence framework documentation first draft.
Only the manager tries to read files with the input framework now.
Initial support for Bro's Intel framework with the Collective Intelligence Framework.
Initial API for Intel framework is complete.
Fixed an issue with cluster data distribution.
Updating some intel framework test baselines.
Reworked cluster intelligence data distribution mechanism and fixed tests.
Lots more intelligence checking in SMTP traffic.
Added intelligence check for "Received" path checking and a bit of reshuffling.
Added sources to the intel log.
Fixing a problem with intel distribution on clusters.
Updated intel framework test to include matching.
Restructuring the scripts that feed data into the intel framework slightly.
One test for cluster transparency of the intel framework.
Fixed a cluster support bug.
Intelligence framework checkpoint
Major updates to fix the Intel framework API.
Checkpoint commit. This is all a huge mess right now. :)
...
Closes#914.
more cases.
It will now not only fire after table-reads have been completed,
but also after the last event of a whole-file-read (or whole-db-read, etc.).
The interface also has been extended a bit to allow readers to
directly fire the event should they so choose. This allows the
event to be fired in direct table-setting/event-sending modes,
which was previously not possible.
This could, for example, result in duplicate emails being sent (one from
manager and one from worker) if Notice::emailed_types is redef'd in
local.bro (or any script that gets loaded on all cluster nodes).
The problem was that Notice::policy is used to populate the internal
Notice::ordered_policy vector in a priority 10 bro_init handler (in
scripts/base/frameworks/notice/main.bro) and then that is what is used
when applying policy to notices. In order for
scripts/base/frameworks/notice/cluster.bro to prevent Notice::policy
from being used on non-manager nodes, it needs to clear it in a
bro_init hander of higher priority than 10.
- Intel data distribution on clusters is now pushed in whole
by the manager when a worker connects. Additions after that point
are managed by the normal single-item distribution mechanism already
built into the intelligence framework.
- The manager maintains the complete "minimal" data store that the
workers use to do their matching so that full "minimal" data
distribution is very easy.
- Tests are cleaned up and work.
- Basic API seems to works, but tests aren't updated yet.
- Several scripts are available in policy/frameworks/intel that
call the "seen" function to provide data into the intel
framework to be tested.
- Intel::policy is not done yet and needs to be discussed to
figure out what it needs to have.
- Running the intel framework and having it do something finally
is really cool!
Since the millisecond resolution cannot be harnessed universally and is not
supported by older version of libcurl, we will allow only specifications at the
granularity of seconds.
This commit also fixes a typing issue that causes that prevented the
ElasticSearch timeout to work in the first place: curl_easy_setopt requires a
long but was given a uint64_t.
* origin/topic/bernhard/input-allow_invalid_types:
to be sure - add a small assertion
add an option to the input framework that allows the user to chose to not die upon encountering files/functions.
That's the last feature for 2.1!
* origin/topic/seth/reporter-to-stderr:
A couple of tests for printing reporter messages to STDERR.
Small improvements for printing reporter messages to STDERR.
Reporter warnings and error now print to stderr by default.
Closes#836.
- All 5 intelligence tests pass.
- Some initial memory optimizations done.
- More work needs done to reduce duplicate data in memory.
- Input framework integration.
- Define files to read in the "Bro intelligence format" in Intel::read_files.
- Cluster transparency.
- DNS Zones are a fully supported data type.
- Queries for Intel::DOMAIN values will automatically check in DNS_ZONE intelligence.
Also reenabling the logs-to-elasticsearch.bro script in
test-all-policy.bro, that seems to work now.
* origin/fastpath:
Reworked how the logs-to-elasticsearch scripts works to stop abusing the logging framework.
This change makes it so when differing logging filters on the same
stream attempt to write to the same writer/path combination, the path
of the filter doing the later write will be automatically adjusted so
that it does not conflict with the other. The path is adjusted by
appending "-N", where N is the smallest integer greater or equal to 2
required to resolve the path name conflict.
Addresses #842.
I've only tested that it compiles, not whether it still works. The
fact that we don't have any tests for this makes me uneasy ...
* remotes/origin/topic/seth/elasticsearch: (35 commits)
Some documentation updates for elasticsearch plugin.
Temporarily removing the ES timeout because it works with signals and is incompatible with Bro threads.
Changed ES index names to localtime and added a meta index.
New script for easily duplicating logs to ElasticSearch.
Some better elasticsearch reliability.
Fixed small elasticsearch problem in configure output.
Re-adding the needed call to FinishedRotation in the ES writer plugin.
Tiny updates.
Bringing elasticsearch branch up to date with master.
Adding a define to make the stdint C macros available.
Adding an extra header.
Fixed a bug with messed up time value passing to elasticsearch.
Small updates and a little standardization for config.h.in naming.
Bug fixes.
Bug fix and feature.
Forgot to call the parent method for DoHeartBeat.
Changed the escaping method.
Flush logs to ES daemon as Bro is shutting down.
Reduce the batch size to 1000 and add a maximum time interval for batches.
Reworked bulk operation string construction to use ODesc and added json escaping.
...
Turns out the finish methods weren't called correctly, caused by a
mess up with method names which all sounded too similar and the wrong
one ended up being called. I've reworked this by changing the
thread/writer/reader interfaces, which actually also simplifies them
by getting rid of the requirement for writer backends to call their
parent methods (i.e., less opportunity for errors).
This commit also includes the following (because I noticed the problem
above when working on some of these):
- The ASCII log writer now includes "#start <timestamp>" and
"#end <timestamp> lines in the each file. The latter supersedes
Bernhard's "EOF" patch.
This required a number of tests updates. The standard canonifier
removes the timestamps, but some tests compare files directly,
which doesn't work if they aren't printing out the same
timestamps (like the comm tests).
- The above required yet another change to the writer API to
network_time to methods.
- Renamed ASCII logger "header" options to "meta".
- Fixes#763 "Escape # when first character in log file line".
All btests pass for me on Linux FC15. Will try MacOS next.
