- Some baselines for tests in "leaks" group were outdated.
- Changed a few of the cluster/communication tests to terminate
more explicitly instead of relying on btest-bg-wait to kill
processes. This makes the tests finish faster in the success case
and makes the reason for failing clearer in the that case.
* origin/topic/seth/reporter-to-stderr:
A couple of tests for printing reporter messages to STDERR.
Small improvements for printing reporter messages to STDERR.
Reporter warnings and error now print to stderr by default.
Closes#836.
An assertion would trigger in the case when a predicate refuses
a new entry and another entry with the same index elements was
already in the table. (I thought that code block was unreachable
... did not think of this case).
* origin/fastpath:
Small (potential performance) improvement for logging framework.
Script-level rotation postprocessor fix.
update input framework documentation to reflect want_record change.
Fix crash when encountering an InterpreterException in a predicate in logging or input Framework.
make want_record=T the default for events
Small tweak: I added the "same writer" constraint to the loop
condition as well. Makes sense?
* origin/fastpath:
Change path conflicts between log filters to be auto-corrected.
There seems to be a race condition in capturing the external shell's
stdout output reliably. As far as I can tell, Bro's doing everything
correctly though, the log postprocessors gets executed as expected. So
I rewrote the test to capture the output in a separate file first, and
that seems to solve the test failures.
This change makes it so when differing logging filters on the same
stream attempt to write to the same writer/path combination, the path
of the filter doing the later write will be automatically adjusted so
that it does not conflict with the other. The path is adjusted by
appending "-N", where N is the smallest integer greater or equal to 2
required to resolve the path name conflict.
Addresses #842.
add missing binary testcase (Baseline is in master, testcase is missing for some reason)
make error output for nonmatching event types much more verbose
Since WriterFrontend objects are looked up internally by writer type and
path, and they also expect to write consistent field arguments, it could
be the case that more than one filter of a given stream attempts to
write to the same path (derived either from $path or $path_func fields
of the filter) with the same writer type. This won't work, so now
WriterFrontend objects are bound to the filter that instantiated them so
that we can warn about other filters attempting to write to the
conflicting writer/path and the write can be skipped. Remote logs don't
appear to suffer the same issue due to pre-filtering.
Addresses #842.
* origin/topic/bernhard/input-fixes:
fix problem with possible access to unititialized memory (thanks robin :) )
and just to be a little bit careful - add check if the field description is long enough. Otherwise there might possibly be an access of uninitialized memory, when someone reads a file that contains just #fields without any following field descriptions.
and like nearly always - forgot the baseline.
Input framework now accepts escaped ascii values as input.
make reading ascii logfiles work when the input separator is different from \t.
Threads will now reliably get a call to DoFinish() no matter how the
thread terminates. This will always be called from within the thread,
whereas the destructor is called from the main thread after the child
thread has already terminated.
Also removing debugging code.
However, two problems remain with the ASCII writer (seeing them only
on MacOS):
- the #start/#end timestamps contain only dummy values right now.
The odd thing is that once I enable strftime() to print actual
timestamps, I get crashes (even though strftime() is supposed to
be thread-safe).
- occassionally, there's still output missing in tests. In those
cases, the file descriptor apparently goes bad: a write() will
suddently return EBADF for reasons I don't understand yet.
I've only tested that it compiles, not whether it still works. The
fact that we don't have any tests for this makes me uneasy ...
* remotes/origin/topic/seth/elasticsearch: (35 commits)
Some documentation updates for elasticsearch plugin.
Temporarily removing the ES timeout because it works with signals and is incompatible with Bro threads.
Changed ES index names to localtime and added a meta index.
New script for easily duplicating logs to ElasticSearch.
Some better elasticsearch reliability.
Fixed small elasticsearch problem in configure output.
Re-adding the needed call to FinishedRotation in the ES writer plugin.
Tiny updates.
Bringing elasticsearch branch up to date with master.
Adding a define to make the stdint C macros available.
Adding an extra header.
Fixed a bug with messed up time value passing to elasticsearch.
Small updates and a little standardization for config.h.in naming.
Bug fixes.
Bug fix and feature.
Forgot to call the parent method for DoHeartBeat.
Changed the escaping method.
Flush logs to ES daemon as Bro is shutting down.
Reduce the batch size to 1000 and add a maximum time interval for batches.
Reworked bulk operation string construction to use ODesc and added json escaping.
...
frameworks.
There were a number of cases that weren't thread-safe. In particular,
we don't use std::string anymore for anything that's passed between
threads (but instead plain old const char*, with manual memmory
managmenet).
This is still a check-point commit, I'll do more testing.
Turns out the finish methods weren't called correctly, caused by a
mess up with method names which all sounded too similar and the wrong
one ended up being called. I've reworked this by changing the
thread/writer/reader interfaces, which actually also simplifies them
by getting rid of the requirement for writer backends to call their
parent methods (i.e., less opportunity for errors).
This commit also includes the following (because I noticed the problem
above when working on some of these):
- The ASCII log writer now includes "#start <timestamp>" and
"#end <timestamp> lines in the each file. The latter supersedes
Bernhard's "EOF" patch.
This required a number of tests updates. The standard canonifier
removes the timestamps, but some tests compare files directly,
which doesn't work if they aren't printing out the same
timestamps (like the comm tests).
- The above required yet another change to the writer API to
network_time to methods.
- Renamed ASCII logger "header" options to "meta".
- Fixes#763 "Escape # when first character in log file line".
All btests pass for me on Linux FC15. Will try MacOS next.
Also removed RefExpr::Eval(Val*) method since it was never called
(Clang emitted warning about this hiding overloaded virtual function
UnaryExpr::Eval(Frame*)) and doesn't appear to be necessary even if it
was called to avoid the default vector handling of UnaryExpr::Eval
(as the comment suggests as the intention).
The output on stderr for this test is the results of many backgrounded
"echo" commands, one for each rotation, so the order in which they
occur may be subject to OS process scheduling and can't be relied upon
* origin/topic/bernhard/reader-info:
fix small bug - now configuration actually is passed.
add mode to readerinfo - no need to have it separately everywhere anymore.
introduce reader-info struct analogous to writer-info.
Introduce support for a table of key/value pairs with further configuration options, with the same userinterface as in the logging interface.
make writer-info work when debugging is enabled
Conflicts:
testing/btest/Baseline/scripts.base.frameworks.input.event/out
testing/btest/Baseline/scripts.base.frameworks.input.executeraw/out
testing/btest/Baseline/scripts.base.frameworks.input.raw/out
testing/btest/Baseline/scripts.base.frameworks.input.rereadraw/out
testing/btest/Baseline/scripts.base.frameworks.input.tableevent/out
Closes#841.
* robin/topic/writer-info:
Extending the log writer DoInit() API.
Reworking log writer API to make it easier to pass additional information to a writer's initialization method.
Conflicts:
src/logging/WriterBackend.cc
src/logging/WriterBackend.h
src/logging/WriterFrontend.cc
