Turns out that in error situations, the final finish message might not
reach the writer anymore, as communication between the threads will be
shut down. Instead of aborting, we now just clean up in that case and
proceed. This isn't changing any other behaviour. The original error
check was in place mostly for helping debug the data flow between the
threads anyways.
Addresses BIT-1331.
is present. Testcase crashes on unpatched versions of Bro.
Found by Aaron Eppert <aeppert@gmail.com>.
This (probably) fixes the crash issue with sqlite a few people have
reported on the mailing list in the past.
Also changed asynchronous data store query code a bit; trying to make
memory management and handling of corner cases a bit clearer (former
maybe could still be better, but I need to lookup queries by memory
address to associate response cookies to them, and so wrapping pointers
kind of just gets in the way).
It now works a bit differently than before: whether to send a remote log
write is now a property of the logging stream, not the logging filter
and it's now up the the receiver side filters to instantiate the desired
writer. i.e. the sender now has no say in what the receiver should use
as the log writer backend.
Under the new style of remote logging, the "Log::enable_remote_logging"
option is repurposed to set the default behavior for new logging
streams. There's also "Comm::{enable,disable}_remote_logging()" to
explicitly set the desired behavior for a given logging stream. To
receive remote logs, one calls "Comm::subscribe_to_logs(<topic>)", where
senders implicitly use topics of the form "bro/log/<stream id>".
Setting the thread name on every heartbeat uses a mild amount of
cycles and there's not much benefit to doing it there to get the
additional info regarding the number of processed messages since thread
names usually get truncated to 16 characters and omit that part anyway.
An unnecessary null pointer check and uninitialized scalar fields.
Don't expect these to be actual problems, but easy enough to fix in
order to silence Coverity.
* origin/topic/seth/json-formatter:
Updating a couple of tests.
Expanded support for modifying the timestamp format in the JSON formatter.
Ascii input reader now supports all config options per-input stream.
Added an option to the JSON formatter to use ISO 8601 for timestamps.
Refactored formatters and updated the the writers a bit.
Includes some minor bugfixes and cleanup at various places, including
in old code.
- It's not *exactly* ISO 8601 which doesn't seem to support
subseconds, but subseconds are very important to us and
most things that support ISO8601 seem to also support subseconds
in the way I'm implemented it.
- Formatters have been abstracted similarly to readers and writers now.
- The Ascii writer has a new option for writing out logs as JSON.
- The Ascii writer now has all options availble as per-filter
options as well as global.
Segmentation fault caused by accessing fields with pos which is
one-based for setting SQLite field values. Fix is to simply subtract one
from pos. Discovered when trying to store HTTP traffic to a SQLite
database with the following Bro script:
event bro_init() {
local filter: Log::Filter = [
$name = "sqlite",
$path = "http",
$config = table(["tablename"] = "http_logs"),
$writer = Log::WRITER_SQLITE
];
Log::add_filter(HTTP::LOG, filter);
}
Sorry for this - I noticed that I named this option quite unfortunately
while writing the documentation.
The patch also removes the dbname configuration option from the sqlite
input reader - it was not used there at all anymore (and I did not notice
that).
statistics, finalize prepared statement before exitting logger.
This might fix the deadlock issue, at least it did not happen for
me on my tried on the test system where it happened quite regularly
before.
It doesn't do anything else than simply forwarding to FlushBuffers().
This is just for consistency in terminate_bro() where components get
their Terminate() called so that the main code doesn't need to know
anything more specific about what particular action to take at
shutdown.
* origin/topic/bernhard/thread-cleanup:
and just to be really sure - always make threads go through OnWaitForStop
hopefully finally fix last interesting race-condition
it is apparently getting a bit late for changes at important code...
spoke to soon (forgot to comment in line again).
Change thread shutdown again to also work with input framework.
Changing semantics of thread stop methods.
Support for cleaning up threads that have terminated.
Seems to work, tests pass, but not really verified.
Major change 1:
finished flag in MsgThread was replaced by 2 flags:
child_finished and main_finished.
child_finished is set by child_thread and means that the processing
loop is stopped immediately (no longer needed, no new input messages
will be processed, if loop continues running there is an ugly delay
on shutdown). (This took me a while to realize...)
main_finished is set by a message that is sent back by the child
to the main thread when Finished() is called (and child_finished
is set). when main_finished is set, processing of output messages
stops. But all messages that the child thread pushed in the queue
before calling Finish() are still processed.
Change 2:
Logging terminate call was replaced by a smaller call that just
flushes out the cache held by the main thread. This call
has to be done before thread shutdown is called - otherwhise
the threads will be shut down before all messages are pushed
on them. (This also took me a while to realize...).
Change 3:
Input framework actually calls it stop methods correctly (everything
was prepared, function call was missing)
* origin/topic/bernhard/sqlite:
fix a few small rough edges (mostly comments that do no longer apply)
fix bug in input-manager regarding enums that a writer reads without 0-terminating the string
actually make sqlite work again (tests passed because the writer was not actually defined because of the define.)
add sqlite distribution.
fix warnings, update baselines, handle rotation
add sqlite tests and fix small vector/set escaping bugs
fix small bug with vectors and sets.
make work with newer AsciiFormatter.
start adding a different text for empty records for the sqlite writer.
no, you will never guess from where I copied this file...
make sqlite support more or less work for logging and input
make sqlite-writer more stable.
make it compile with new version of AsciiInputOutput
and adapt to AsciiInputOutput - seems to work...
make it compile
add SQLite reader.
...adapt to new api...
now the writer supports tables and vectors.
basic sqlite writer seems to work.
