Turns out the finish methods weren't called correctly, caused by a
mess up with method names which all sounded too similar and the wrong
one ended up being called. I've reworked this by changing the
thread/writer/reader interfaces, which actually also simplifies them
by getting rid of the requirement for writer backends to call their
parent methods (i.e., less opportunity for errors).
This commit also includes the following (because I noticed the problem
above when working on some of these):
- The ASCII log writer now includes "#start <timestamp>" and
"#end <timestamp> lines in the each file. The latter supersedes
Bernhard's "EOF" patch.
This required a number of tests updates. The standard canonifier
removes the timestamps, but some tests compare files directly,
which doesn't work if they aren't printing out the same
timestamps (like the comm tests).
- The above required yet another change to the writer API to
network_time to methods.
- Renamed ASCII logger "header" options to "meta".
- Fixes#763 "Escape # when first character in log file line".
All btests pass for me on Linux FC15. Will try MacOS next.
The output on stderr for this test is the results of many backgrounded
"echo" commands, one for each rotation, so the order in which they
occur may be subject to OS process scheduling and can't be relied upon
* robin/topic/writer-info:
Extending the log writer DoInit() API.
Reworking log writer API to make it easier to pass additional information to a writer's initialization method.
Conflicts:
src/logging/WriterBackend.cc
src/logging/WriterBackend.h
src/logging/WriterFrontend.cc
Generally tried to make them more reliable and execute quicker.
They all now load the listen script as a trick to make sure input
sources are fully read, but also terminate() at appropriate times
so that they don't take more time than needed. They're also all
serialized with the 'comm' group so listening on a port doesn't
interfere with the communication tests.
Using the default scripts, the events from RemoteSerializer::LogStats()
were attempting to use the logging framework after logging/threading
had been terminated which never worked right and sometimes caused
crashes with "fatal error: cannot lock mutex".
Also made communication log baseline test pass more reliably.
We now pass in a Info struct that contains:
- the path name (as before)
- the rotation interval
- the log_rotate_base_time in seconds
- a table of key/value pairs with further configuration options.
To fill the table, log filters have a new field "config: table[string]
of strings". This gives a way to pass arbitrary values from
script-land to writers. Interpretation is left up to the writer.
Also splits calc_next_rotate() into two functions, one of which is
thread-safe and can be used with the log_rotate_base_time value from
DoInit().
Includes also updates to the None writer:
- It gets its own script writers/none.bro.
- New bool option LogNone::debug to enable debug output. It then
prints out all the values passed to DoInit(). That's used by a
btest test to ensure the new DoInit() values are right.
- Fixed a bug that prevented Bro from terminating..
(scripts.base.frameworks.logging.rotate-custom currently fails.
Haven't yet investigated why.)
This lead to hanging bro's because pclose apparently can wait for eternity
if things go wrong. And there probably are a couple of other problems with this approach.
Adjust twotables testcase - now it is faster. Shorten the output
-- because of threading, the results did not always come out in
the same order (it depends on which thread manages to sneak in
the results into the queue earlier).
* origin/topic/robin/dataseries:
Moving trace for rotation test into traces directory.
Fixing a rotation race condition at termination.
Portability fixes.
Extending DS docs with some examples.
Updating doc.
Fixing pack_scale and time-as-int.
Adding format specifier to DS spec to print out double as %.6f.
DataSeries updates and fixes.
DataSeries tuning.
Tweaking DataSeries support.
Extending log post-processor call to include the name of the writer.
Removing an unnecessary const cast.
DataSeries TODO list with open issues/questions.
Starting DataSeries HowTo.
Additional test output canonification for ds2txt's timestamps.
In threads, an internal error now immediately aborts.
DataSeries cleanup.
Working on DataSeries support.
Merging in DataSeries support from topic/gilbert/logging.
Fixing threads' DoFinish() method.
- The option to use integers insteads of double was ignored.
- Renaming script-level options to remove the ds_ prefix.
- Log rotation didn't work.
- A set of simple unit tests.
This do not have to be present in the input file and are marked as &optional in the record description.
Those can e.g. be used to create field values on the file in a predicate while reading a file - example:
Input::add_table([$source="input.log", $name="input", $idx=Idx, $val=Val, $destination=servers,
$pred(typ: Input::Event, left: Idx, right: Val) = { right$notb = !right$b; return T; }
support reading from commands by adppending | to the filename.
support streaming reads from command.
Fix something to make rearead work better. (magically happened)
Note that fdstream.h is from boost and has a separate license:
* (C) Copyright Nicolai M. Josuttis 2001.
* Permission to copy, use, modify, sell and distribute this software
* is granted provided this copyright notice appears in all copies.
* This software is provided "as is" without express or implied
* warranty, and with no claim as to its suitability for any purpose.
compiles, not really tested.
basic test works 70% of the time, coredumps in the other 30 - but was not easy to debug on a first glance (most interestingly the crash happens in the logging framework - I wonder how that works).
Other tests are not adjusted to the new interface yet.
* origin/topic/robin/parallel-btest:
Updating submodule(s).
Adding group "comm" to communication tests for parallelizing execution with new btest version.
This is in preparation for the merge with the parallel btest branch.
Conflicts:
cmake
testing/btest/istate/pybroccoli.py
testing/btest/scripts/base/frameworks/control/configuration_update.bro
testing/btest/scripts/base/frameworks/control/id_value.bro
testing/btest/scripts/base/frameworks/control/shutdown.bro
testing/btest/scripts/base/frameworks/metrics/basic-cluster.bro
* simple testcase for file refresh (check for changes) and streaming reads
* add events for simple put and delete operations
* fix bugs in table filter events (type for first element was wrong)
* and I think a couple of other small bugs
* change internal reader interface again
* remove some quite embarassing bugs that must have been in the interface for rather long
* add different read methods to script & internal interface (like normal, streaming, etc). Not implemented in ascii reader yet.
But: there are still a few places where I am sure that there are race conditions & memory leaks & I do not really like the current interface & I have to add a few more messages between the front and backend.
But - it works :)
* origin/topic/bernhard/software:
change software framework interface again. At the moment everything should worl.
start reworking interface of software framework. working apart from detect-webapps.bro, which direcly manipulates a no longer available interface...
after talking to seth - change host_a field in record back to host.
forgotten policy files.
Software framework stores ports for server software.
