This one would fail intermittently in the cases where log files were
opened or closed on a different second of the time of day from each
other since the "out" baseline contains only a single "#open" and
"#close" tag (indicating all logs opened/closed on same second of time
of day). Piping aggregated log output through the timestamp canonifier
before `uniq` makes it so "#open" and "#close" tags for different
seconds of the time of day are reduced to a single one.
Not sure if more can be done to work around it, but reported to
dataseries devs here: https://github.com/dataseries/DataSeries/issues/1
The core/leaks/dataseries-rotate.bro unit test fails without this.
* origin/fastpath:
Add the Stream record to Log:active_streams to make more dynamic logging possible.
Fix portability of printing to files returned by open("/dev/stderr").
Fix mime type diff canonifier to also skip mime_desc columns
Unit test tweaks/fixes.
Fix memory leak of serialized IDs when compiled with --enable-debug.
One tweak to the open() change: make sure we don't try to rotate the
special files.
The BroFile ctor now wraps /dev/std{in,out,err} string arguments
into the actual FILE* provided by stdio.h because use of the former
directly isn't POSIX compliant and led to subtle differences that
broke unit tests on certain platforms (e.g. OS X redirection of stderr
behavior started differing from Linux). The BroFile (un)serialization
methods already did this kind of logic, so adding it in the ctor also
should make things more consistent.
Some of the reporter-related unit tests looked like they were missing
output because of this, and the coverage test for bare-mode errors
needed tweaking to branch on whether or not libcurl was available
(since the error output differs when elasticsearch isn't there).
- Some baselines for tests in "leaks" group were outdated.
- Changed a few of the cluster/communication tests to terminate
more explicitly instead of relying on btest-bg-wait to kill
processes. This makes the tests finish faster in the success case
and makes the reason for failing clearer in the that case.
When using --enable-debug, values keep track of the last identifier
to which they were bound by storing a ref'd ID pointer. This could
lead to some circular dependencies in which an ID is never reclaimed
because the Val is bound to the ID and the ID is bound to the Val, with
both holding references to each other.
There might be more cases where this feature of --enable-debug caused
a leak, but it showed up in particular when running the
core.leaks.remote unit test due to the internal
SendID("peer_description") call during the handshake between remote
processes. Other tests showed the send_id() BIF leaked more generally.
Tracking the ID last bound to a Val through just the identifier string
instead of a ref'd ID pointer fixes the leak.
* origin/topic/seth/reporter-to-stderr:
A couple of tests for printing reporter messages to STDERR.
Small improvements for printing reporter messages to STDERR.
Reporter warnings and error now print to stderr by default.
Closes#836.
* origin/fastpath:
fix little sneaky bug in input framework with an edge case.
small bug in test script. Still worked, because the internal type checking let this through...
An assertion would trigger in the case when a predicate refuses
a new entry and another entry with the same index elements was
already in the table. (I thought that code block was unreachable
... did not think of this case).
If a log filter attempts to write to a path for which a writer is
already instantiated due to remote logging, it will re-use the writer
as long as the fields of the filter and writer are compatible, else
the filter path will be auto-adjusted to not conflict with existing
writer's. Conflicts between two local filters are still always
auto-adjusted even if field types agree (since they could still
be semantically different).
Addresses #842.
There are now two FinishedRotation() methods, one that triggers
post-processing and one that doesn't. There's also insurance built in
against a writer not calling either (or both), in which case we abort
with an internal error.
This changes writer implementations to always respond to rotation
messages in their DoRotate() method, even for failure/no-op cases
with a new RotationFailedMessage. This informs the manager to
decrement its count of pending rotations.
Addresses #860.
Also reenabling the logs-to-elasticsearch.bro script in
test-all-policy.bro, that seems to work now.
* origin/fastpath:
Reworked how the logs-to-elasticsearch scripts works to stop abusing the logging framework.
* origin/fastpath:
Small (potential performance) improvement for logging framework.
Script-level rotation postprocessor fix.
update input framework documentation to reflect want_record change.
Fix crash when encountering an InterpreterException in a predicate in logging or input Framework.
make want_record=T the default for events
