statistics, finalize prepared statement before exitting logger.
This might fix the deadlock issue, at least it did not happen for
me on my tried on the test system where it happened quite regularly
before.
* origin/topic/bernhard/thread-cleanup:
and just to be really sure - always make threads go through OnWaitForStop
hopefully finally fix last interesting race-condition
it is apparently getting a bit late for changes at important code...
spoke to soon (forgot to comment in line again).
Change thread shutdown again to also work with input framework.
Changing semantics of thread stop methods.
Support for cleaning up threads that have terminated.
Now it should work. However - this commit changes a basic assumption
of the threading queue. This basic assumption is, that nothing can
be read out of the out-queue of a dead thread. I think that reading
out of the queue of a dead thread makes perfect sense (when the thread
shuts down, pushes the rest of its work on the queue and says bye,
and wants the main thread to pick it up afterwards) - however, I
guess one can be of a differing opinion here.
In any case, it makes stuff a bit easier to understand - in my opinion.
It took me a while to find out why the messages disappear in thin
air and never arrive in the main thread ;)
Seems to work, tests pass, but not really verified.
Major change 1:
finished flag in MsgThread was replaced by 2 flags:
child_finished and main_finished.
child_finished is set by child_thread and means that the processing
loop is stopped immediately (no longer needed, no new input messages
will be processed, if loop continues running there is an ugly delay
on shutdown). (This took me a while to realize...)
main_finished is set by a message that is sent back by the child
to the main thread when Finished() is called (and child_finished
is set). when main_finished is set, processing of output messages
stops. But all messages that the child thread pushed in the queue
before calling Finish() are still processed.
Change 2:
Logging terminate call was replaced by a smaller call that just
flushes out the cache held by the main thread. This call
has to be done before thread shutdown is called - otherwhise
the threads will be shut down before all messages are pushed
on them. (This also took me a while to realize...).
Change 3:
Input framework actually calls it stop methods correctly (everything
was prepared, function call was missing)
PrepareStop() is now SignalStop() and just signals a thread that it
should terminate. After that's called, WaitForStop() (formerly Stop())
wait for it to actually finish processing.
When stopping writers during operation, we now no longer wait for them
to finish.
Once a BasicThread leaves its run() method, a thread is now marked for
cleaning up, and the ThreadMgr will soon join it to release the OS
resources.
Also, adding a function Log::remove_stream() that remove a logging
stream, stopping all writer threads that are associated with it.
Note, however, that removing a *filter* from a stream still doesn't
clean up any threads. The problem is that because of the output paths
potentially being created dynamically it's unclear if the writer
thread will still be needed in the future. We could add clean writers
up with timeouts, but that doesn't sound great either. So for now, the
only way to sure clean up logging threads is to remove the entire
stream.
Also note that cleanup doesn't work with input threads yet, which
don't seem to terminate (at least in the case I tried).
* origin/topic/bernhard/input-logging-commmon-functions:
add the last of Robins suggestions (separate info-struct for constructors).
port memory leak fix from master
harmonize function naming
move AsciiInputOutput over to threading
and thinking about it, ascii-io doesn't need the separator
change constructors
and factor stuff out the input framework too.
factor out ascii input/output.
std::string accessors to escape_sequence functionality
intermediate commit - it has been over a month since I touched this...
I cleaned up the AsciiInputOutput class somewhat, including renaming
it to AsciiFormatter, renaming some of its methods, and turning the
static methods into members for consistency.
Closes#929.
First step - factored out everything the logging classes
use ( so only output ).
Moved the script-level configuration to logging/main,
and made the individual writers just refer to it -
no idea if this is good design. It works. But I am happy
about opinions :)
Next step - add support for input...
failure.
Once a writer/reader Do* method has returned false, no further ones
will be executed anymore. This is primarily a safety mechanism to make
it easier for writer/reader authors as otherwise they would often need
to track the failure state themselves (because with the now delayed
termination from the earlier commit, furhter messages can now still
arrive for a little bit).
If a thread command fails (like the input framework not finding a
file), that now (1) no longer hangs Bro, and (2) even allows for
propagating error messages back before the thread is stops.
(Actually, the thread doesn't really "stop"; the thread manager keeps
threads around independent of their success; but it no longer polls
them for input.)
Closes#858.
* topic/robin/master-test: (60 commits)
Script fix for Linux.
Updating test base line.
Another small change to MsgThread API.
Bug fix for BasicThread.
make version_ok return true for TLSv12
Sed usage in canonifier script didn't work on non-Linux systems.
Changing HTTP DPD port 3138 to 3128.
Temporarily removing tuning/logs-to-elasticsearch.bro from the test-all-policy.
More documentation updates.
Revert "Fixing calc_next_rotate to use UTC based time functions."
Some documentation updates for elasticsearch plugin.
Give configure a --disable-perftools option.
Updating tests for the #start/#end change.
Further threading and API restructuring for logging and input frameworks.
Reworking forceful thread termination.
Moving the ASCII writer over to use UNIX I/O rather than stdio.
Further reworking the thread API.
Reworking thread termination logic.
If a thread doesn't terminate, we log that but not longer proceed (because it could hang later still).
Removing the thread kill functionality.
...
Threads will now reliably get a call to DoFinish() no matter how the
thread terminates. This will always be called from within the thread,
whereas the destructor is called from the main thread after the child
thread has already terminated.
Also removing debugging code.
However, two problems remain with the ASCII writer (seeing them only
on MacOS):
- the #start/#end timestamps contain only dummy values right now.
The odd thing is that once I enable strftime() to print actual
timestamps, I get crashes (even though strftime() is supposed to
be thread-safe).
- occassionally, there's still output missing in tests. In those
cases, the file descriptor apparently goes bad: a write() will
suddently return EBADF for reasons I don't understand yet.
frameworks.
There were a number of cases that weren't thread-safe. In particular,
we don't use std::string anymore for anything that's passed between
threads (but instead plain old const char*, with manual memmory
managmenet).
This is still a check-point commit, I'll do more testing.
Turns out the finish methods weren't called correctly, caused by a
mess up with method names which all sounded too similar and the wrong
one ended up being called. I've reworked this by changing the
thread/writer/reader interfaces, which actually also simplifies them
by getting rid of the requirement for writer backends to call their
parent methods (i.e., less opportunity for errors).
This commit also includes the following (because I noticed the problem
above when working on some of these):
- The ASCII log writer now includes "#start <timestamp>" and
"#end <timestamp> lines in the each file. The latter supersedes
Bernhard's "EOF" patch.
This required a number of tests updates. The standard canonifier
removes the timestamps, but some tests compare files directly,
which doesn't work if they aren't printing out the same
timestamps (like the comm tests).
- The above required yet another change to the writer API to
network_time to methods.
- Renamed ASCII logger "header" options to "meta".
- Fixes#763 "Escape # when first character in log file line".
All btests pass for me on Linux FC15. Will try MacOS next.
(because it could hang later still).
Also logging to stderr as well to make sure one sees it.
Also adding code to the ASCII writer to catch termination
inconsistencies.
Turns out the finish methods weren't called correctly, caused by a
mess up with method names which all sounded too similar and the wrong
one ended up being called. I've reworked this by changing the
thread/writer/reader interfaces, which actually also simplifies them
by getting rid of the requirement for writer backends to call their
parent methods (i.e., less opportunity for errors).
This commit also includes the following (because I noticed the problem
above when working on some of these):
- The ASCII log writer now includes "#start <timestamp>" and
"#end <timestamp> lines in the each file. The latter supersedes
Bernhard's "EOF" patch.
This required a number of tests updates. The standard canonifier
removes the timestamps, but some tests compare files directly,
which doesn't work if they aren't printing out the same
timestamps (like the comm tests).
- The above required yet another change to the writer API to
network_time to methods.
- Renamed ASCII logger "header" options to "meta".
- Fixes#763 "Escape # when first character in log file line".
All btests pass for me on Linux FC15. Will try MacOS next.
* origin/fastpath:
Fix overrides of TCP_ApplicationAnalyzer::EndpointEOF.
Fix segfault when incrementing whole vector values.
Remove baselines for some leak-detecting unit tests.
Unblock SIGFPE, SIGILL, SIGSEGV and SIGBUS for threads.
According to POSIX, behavior is unspecified if a specific thread receives one of those signals (because of e.g. executing an invalid instruction) if the signal is blocked.
This resulted in segfaults in threads not propagating to the main thread.
Adresses #848
set frontend type before starting the thread. This means that the thread type will be output correctly in the error message.
return errno string of pthread functions called in thread start
* topic/robin/input-threads-merge: (130 commits)
And now it even compiles after my earlier changes.
A set of input framework refactoring, cleanup, and polishing.
another small memory leak in ascii reader:
and another small memory leak when using streaming reads.
fix another memory lead (when updating tables).
Input framework merge in progress.
filters have been called streams for eternity. And I always was too lazy to change it everywhere...
reactivate network_time check in threading manager. previously this line made all input framework tests fail - it works now. Some of the other recent changes of the threading manager must have fixed that problem.
fix up the executeraw test - now it works for the first time and does not always fail
baselines for the autostart removal.
remove last remnants of autostart, which has been removed for quite a while.
make input framework source (hopefully) adhere to the usual indentation style. No functional changes.
fix two memory leaks which occured when one used filters.
update description to current interface.
rename a couple of structures and make the names in manager fit the api more.
fix memory leak in tables and vectors that are read into tables
fix missing get call for heart beat in benchmark reader.
fix heart_beat_interval -- initialization in constructor does not work anymore (probably due to change in init ordering?)
fix memory leak for tables... nearly completely.
fix a couple more leaks. But - still leaking quite a lot with tables.
...
line made all input framework tests fail - it works now. Some of the
other recent changes of the threading manager must have fixed that
problem.
This was easy :)
* origin/topic/robin/dataseries:
Moving trace for rotation test into traces directory.
Fixing a rotation race condition at termination.
Portability fixes.
Extending DS docs with some examples.
Updating doc.
Fixing pack_scale and time-as-int.
Adding format specifier to DS spec to print out double as %.6f.
DataSeries updates and fixes.
DataSeries tuning.
Tweaking DataSeries support.
Extending log post-processor call to include the name of the writer.
Removing an unnecessary const cast.
DataSeries TODO list with open issues/questions.
Starting DataSeries HowTo.
Additional test output canonification for ds2txt's timestamps.
In threads, an internal error now immediately aborts.
DataSeries cleanup.
Working on DataSeries support.
Merging in DataSeries support from topic/gilbert/logging.
Fixing threads' DoFinish() method.
