Closes#1021.
* origin/topic/bernhard/input-update:
this event handler fails the unused-event-handlers test because it is a bit of a special case.
...and fix the event ordering issue. Dispatch != QueueEvent
add Terminate to input framework to prevent potential shutdown race-conditions.
fix warning.
fix stderr test. ls behaves differently on errors on linux...
small fixes.
linux does not have strnstr
and close only fds that are currently open (the logging framework really did not like that :) )
A bunch of more changes for the raw reader
make reading from stdout and stderr simultaneously work.
allow sending data to stdin of child process
Streaming reads from external commands work without blocking anything.
replace popen with fork and exec.
change raw reader to use basic c io instead of fdstream encapsulation class.
- Thanks for help from Rafal Lesniak in nailing down the location
of the bug and supplying test traffic.
- Test traffic with a TLS 1.2 connection.
- Addresses ticket #1020
When constructing a Bloom filter, one now has to pass a HashPolicy instance to
it. This separates more clearly the concerns of hashing and Bloom filter
management.
This commit also changes the interface to initialize Bloom filters: there exist
now two initialization functions, one for each type:
(1) bloomfilter_basic_init(fp: double,
capacity: count,
name: string &default=""): opaque of bloomfilter
(2) bloomfilter_counting_init(k: count,
cells: count,
max: count,
name: string &default=""): opaque of bloomfilter
The BiFs for adding elements and performing lookups remain the same. This
essentially gives us "BiF polymorphism" at script land, where the
initialization BiF constructs the most derived type while subsequent BiFs
adhere to the same interface.
The reason why we split up the constructor in this case is that we have not yet
derived the math that computes the optimal number of hash functions for
counting Bloom filters---users have to explicitly parameterize them for now.
it is a bit of a special case.
It is only called via the SendEvent function from a reader. The reader
does (at least with the current interface) however not provide
the function pointer, but looks up the name of the event dynamically.
Hence, internal_handler is never called for the event.
Even if resolving the event in the reader, e.g. in an initialization
function, this would not solve the issue - the initialization function
is only called when the first Raw reader is initialized - and in the
base configuration the raw reader will never be used (hence, internal_handler
also won't be called).
Calling it once in the manager seems like a really dirty hack. So - now
it is the second exception in the testcase, unless anyone has a better
idea :)
Update baseline of stderr test to what it should be. There still is
a message ordering issue there (which is the last issue in the new
Raw reader I know of).
One message that sidesteps a bit of the usual processing does
not always arrive at the correct time (meaning it pops up from the
event queue too early). Even though it sidesteps a bit of the usual
processing that should not happen in my opinion (which clearly
does not matter). And I have not yet fully grasped how this can happen.
http.log now has files taken from request and response bodies in
different fields for each, and can now track multiple files per body.
That is, the "extraction_file" field is now "extracted_request_files"
and "extracted_response_files".
- On-demand access to sumstats results through "return from"
functions named SumStats::request and Sumstats::request_key.
Both functions are tested in standalone and clustered modes.
- $name field has returned to SumStats which simplifies cluster
code and makes the on-demand access stuff possible.
- Clustered results can only be collected for 1 minute from their
time of creation now instead of time of last read.
- Thresholds use doubles instead of counts everywhere now.
- Calculation dependency resolution occurs at start up time now
instead of doing it at observation time which provide a minor
cpu performance improvement. A new plugin registration mechanism
was created to support this change.
- AppStats now has a minimal doc string and is broken into hook-based
plugins.
- AppStats and traceroute detection added to local.bro
It was mostly redundant when logged, but still can be useful to
inspect at runtime. In the future, a better field for logging
will be available which will be similar to the "service" field
for connection records (there's not any file-format-specific
analyzers that would currently make use of such a thing).
