But: there are still a few places where I am sure that there are race conditions & memory leaks & I do not really like the current interface & I have to add a few more messages between the front and backend.
But - it works :)
- Data queued at termination wasn't written out completely.
- Fixed some race conditions.
- Fixing IOSource integration.
- Fixing setting thread names on Linux.
- Fixing minor leaks.
All tests now pass for me on Linux in debug and non-debug compiles.
Remaining TODOs:
- Needs leak check.
- Test on MacOS and FreeBSD.
- More testing:
- High volume traffic.
- Different platforms.
most stuff is inplace, logging framework needs a few changes merged before continuing here...
Conflicts:
src/CMakeLists.txt
src/LogMgr.h
src/logging/Manager.cc
src/main.cc
Sending SIGTERM triggers a normal shutdown of all threads that waits
until they have processed their remaining data. However, sending a 2nd
SIGTERM while waiting for them to finish will immediately kill them
all.
This is based on Gilbert's code but I ended up refactoring it quite a
bit. That's why I didn't do a direct merge but started with a new
branch and copied things over to adapt. It looks quite a bit different
now as I tried to generalize things a bit more to also support the
Input Framework.
The larger changes code are:
- Moved all logging code into subdirectory src/logging/. Code
here is in namespace "logging".
- Moved all threading code into subdirectory src/threading/. Code
here is in namespace "threading".
- Introduced a central thread manager that tracks threads and is
in charge of termination and (eventually) statistics.
- Refactored logging independent threading code into base classes
BasicThread and MsgThread. The former encapsulates all the
pthread code with simple start/stop methods and provides a
single Run() method to override.
The latter is derived from BasicThread and adds bi-directional
message passing between main and child threads. The hope is that
the Input Framework can reuse this part quite directly.
- A log writer is now split into a general WriterFrontend
(LogEmissary in Gilbert's code) and a type-specific
WriterBackend. Specific writers are implemented by deriving from
the latter. (The plugin interface is almost unchanged compared
to the 2.0 version.).
Frontend and backend communicate via MsgThread's message
passing.
- MsgThread (and thus WriterBackend) has a Heartbeat() method that
a thread can override to execute code on a regular basis. It's
triggered roughly once a second by the main thread.
- Integration into "the rest of Bro". Threads can send messages to
the reporter and do debugging output; they are hooked into the
I/O loop for sending messages back; and there's a new debugging
stream "threading" that logs, well, threading activity.
This all seems to work for the most part, but it's not done yet.
TODO list:
- Not all tests pass yet. In particular, diffs for the external
tests seem to indicate some memory problem (no crashes, just an
occasional weird character).
- Only tested in --enable-debug mode.
- Only tested on Linux.
- Needs leak check.
- Each log write is currently a single inter-thread message. Bring
Gilbert's bulk writes back.
- Code needs further cleanup.
- Document the class API.
- Document the internal structure of the logging framework.
- Check for robustness: live traffic, aborting, signals, etc.
- Add thread statistics to profile.log (most of the code is there).
- Customize the OS-visible thread names on platforms that support it.
* origin/topic/jsiwek/brofiler:
Fix superfluous/duplicate data getting in to testing coverage log.
Add "# @no-test" tag to blacklist statements from test coverage analysis.
Test coverage integration for external tests and complete suite.
Integrate Bro script coverage profiling with the btest suite.
Add simple profiling class to accumulate Stmt usage stats across runs.
Renaming environment variable BROFILER_FILE to BRO_PROFILER_FILE for
consistency. Yeah, I know, such a nice name! :)
Also replaced the --snaplen/-l command line option with a
scripting-layer option called "snaplen" (which can also be
redefined on the command line, e.g. `bro -i eth0 snaplen=65535`).
Use the BROFILER_FILE environment variable to point to a file in
which Stmt usage statistics from Bro script-layer can be output.
This should be able to be used to check Bro script coverage that
that e.g. the entire test suite covers.
Some of the changes only clean up at termination to make perftools
happt, but there were some "real" leaks as well.
This fixes all DNS leaks I could reproducem, including most likely
what's reported in #534. Closing #534.
I'm also adding a new btest subdir core/leaks with tests requiring
perftools support. These don't compare against base lines but abort
whenever perftools reports a leak (with stack information to track it
down). Right now, these are passing.
Reading from an interface like `bro -i en0` no longer expects to
start reading stdin for a script to load. Explicitly passing in
'-' as an additional command line argument still allows reading a
script from stdin.
Closes#561
The format string given to the reporter warning call wasn't printing
the handler names. Also changed it so that each warning message has
the full context of the warning.
The communication subsystem is now disabled until a new BiF,
enable_communication(), is called. The base scripts do this
automatically when either a Communication::Node is defined, or Bro is
asked to listen for incoming connections.
- bro.init was renamed to base/init-bare.bro and base/all.bro
was renamed to init-default.bro.
- To run in "bare mode" with only the init-bare.bro and no other
scripts from base/, use either -b or --bare-mode.
- The environment variable to run in "bare mode" has been removed.
- policy/ renamed to scripts/
- By default BROPATH now contains:
- scripts/
- scripts/policy
- scripts/site
- *Nearly* all tests pass.
- All of scripts/base/ is loaded by main.cc
- Can be disabled by setting $BRO_NO_BASE_SCRIPTS
- Scripts in scripts/base/ don't use relative path loading to ease use of BRO_NO_BASE_SCRIPTS (to copy and paste that script).
- The scripts in scripts/base/protocols/ only (or soon will only) do logging and state building.
- The scripts in scripts/base/frameworks/ add functionality without causing any additional overhead.
- All "detection" activity happens through scripts in scripts/policy/.
- Communications framework modified temporarily to need an environment variable to actually enable (ENABLE_COMMUNICATION=1)
- This is so the communications framework can be loaded as part
of the base without causing trouble when it's not needed.
- This will be removed once a resolution to ticket #540 is reached.
Also removing the -l command-line option as that can now be done at
the script-level.
A couple tests fail now that use -l. Leaving that until we have
script-level replacement.
* origin/topic/robin/reporting:
Syslog BiF now goes through the reporter as well.
Avoiding infinite loops when an error message handlers triggers errors itself.
Renaming the Logger to Reporter.
Overhauling the internal reporting of messages to the user.
Updating a bunch of tests/baselines as well.
Conflicts:
aux/broccoli
policy.old/alarm.bro
policy/all.bro
policy/bro.init
policy/frameworks/notice/weird.bro
policy/notice.bro
src/SSL-binpac.cc
src/bro.bif
src/main.cc
* origin/fastpath:
Raise internal error when failing to read contents of state file with -x option
I've changed this to use error() instead of internal_error(). The
latter should only be used for logic errors that indicate a bug in
Bro. In this case, the message flags a problem that's more likely to
be external.
The Logger class is now in charge of reporting all errors, warnings,
informational messages, weirds, and syslogs. All other components
route their messages through the global bro_logger singleton.
The Logger class comes with these reporting methods:
void Message(const char* fmt, ...);
void Warning(const char* fmt, ...);
void Error(const char* fmt, ...);
void FatalError(const char* fmt, ...); // Terminate Bro.
void Weird(const char* name);
[ .. some more Weird() variants ... ]
void Syslog(const char* fmt, ...);
void InternalWarning(const char* fmt, ...);
void InternalError(const char* fmt, ...); // Terminates Bro.
See Logger.h for more information on these.
Generally, the reporting now works as follows:
- All non-fatal message are reported in one of two ways:
(1) At startup (i.e., before we start processing packets),
they are logged to stderr.
(2) During processing, they turn into events:
event log_message%(msg: string, location: string%);
event log_warning%(msg: string, location: string%);
event log_error%(msg: string, location: string%);
The script level can then handle them as desired.
If we don't have an event handler, we fall back to
reporting on stderr.
- All fatal errors are logged to stderr and Bro terminates
immediately.
- Syslog(msg) directly syslogs, but doesn't do anything else.
The three main types of messages can also be generated on the
scripting layer via new Log::* bifs:
Log::error(msg: string);
Log::warning(msg: string);
Log::message(msg: string);
These pass through the bro_logger as well and thus are handled in the
same way. Their output includes location information.
More changes:
- Removed the alarm statement and the alarm_hook event.
- Adapted lots of locations to use the bro_logger, including some
of the messages that were previously either just written to
stdout, or even funneled through the alarm mechanism.
- No distinction anymore between Error() and RunTime(). There's
now only one class of errors; the line was quite blurred already
anyway.
- util.h: all the error()/warn()/message()/run_time()/pinpoint()
functions are gone. Use the bro_logger instead now.
- Script errors are formatted a bit differently due to the
changes. What I've seen so far looks ok to me, but let me know
if there's something odd.
Notes:
- The default handlers for the new log_* events are just dummy
implementations for now since we need to integrate all this into
the new scripts anyway.
- I'm not too happy with the names of the Logger class and its
instance bro_logger. We now have a LogMgr as well, which makes
this all a bit confusing. But I didn't have a good idea for
better names so I stuck with them for now.
Perhaps we should merge Logger and LogMgr?
Originally docs were written right after parsing, but it changed to after
the bro_init event happens when I was experimenting with auto-documenting
logging streams by querying the LogMgr after bro_init. That experiment
dead-ended, and that location is bad for other reasons: the doc framework
may try to access BroObj's that have already been freed.
* remotes/origin/topic/jsiwek/doc-framework:
Adding example documentation for a script's use of logging features.
Adding &log attribute to static attr_names array.
Small typo fix.
Bro doc mode now tracks record redefs that extend its field list.
BroBifDoc was unneeded; now dead code, so removed.
Bro doc mode now only does a "shallow" copy of declared record types
Bro's doc mode now terminates after processing bro_init but before net_run
Fixes related to `make doc` handling of script summary text (##! comments)
Overhaul of "doc" build target for generating policy script documentation.
Add parser error hint when in doc mode about checking ## comment syntax.
Move stuff related to policy script documentation from doc/ to doc/scripts/
Fixing example.bro's auto-reST generation baseline test.
Generated script reST documentation is also written out at this time
instead of at the end of lexical scanning.
The persistence serializer will no longer write out Bro's state to the
.state directory when in doc mode.
filter. (Seth Hall and Robin Sommer)
- Merging in the patch from #264, which provides support for mixed
VLAN and MPLS traffic.
- Changing Bro's default filter from being built dynamically to being
a static "ip or not ip". To get the old behaviour back (i.e., the
dynamically built filter), redef "all_packets" to false.
- print-filter.bro now always prints the filter that Bro is actually
using, even if overriden from the command line.
