Also removing the -l command-line option as that can now be done at
the script-level.
A couple tests fail now that use -l. Leaving that until we have
script-level replacement.
- The Makefiles now run btest with "-f diag.log" so that diangnostics
output will always be written into that file.
- Makefiles now hardcode path to btest to avoid picking up the wrong version
if in PATH.
- The canonifier scripts now live in testing/scripts, and they are
used from both btest/ and external/.
- There's a new diff-remove-uids scripts that removed connection UIDs
for diffing. The external/* tests now use that by default.
- Timestamp removal now has its own script: diff-remove-timestamps.
diff-canonifier calls that.
- All Makefile have a "brief" target that runs btest with -b.
- The higher-level directories have Makefile to call the subdirs.
* origin/topic/robin/reporting:
Syslog BiF now goes through the reporter as well.
Avoiding infinite loops when an error message handlers triggers errors itself.
Renaming the Logger to Reporter.
Overhauling the internal reporting of messages to the user.
Updating a bunch of tests/baselines as well.
Conflicts:
aux/broccoli
policy.old/alarm.bro
policy/all.bro
policy/bro.init
policy/frameworks/notice/weird.bro
policy/notice.bro
src/SSL-binpac.cc
src/bro.bif
src/main.cc
This is for testing only. There are also two test: one that checks
that test-all.bro loads correctly, and one that ensures that test-all
is actually loading all scripts found in policy/*.
traces and more complex Bro configurations.
That's "Type 2" in the terminology of
http://www-new.bro-ids.org/devel/projects/testing.html
This leverages btest as well and will replace the set of scripts that
the old private test-suite is using. It can accomodate tests with both
public and private traces.
This is a checkpoint commmit, not much there yet in terms of actual
testing. Also still need to figure out a mechanmism to allow for
different baselines based on which branch we're testing.
Added an additional master TOC index for Bro script packages that
automatically gets filled in at ``make doc`` time. The master TOC
links to per-package indexes which contains links to all the scripts
contained within it along with their summary. The per-package index
is also now automatically generated/derived from the path component
of the script passed into the rest_target() macro.
Added an arg to the search_for_files() util function that can return
the subpath of BROPATH's policy/ dir in which the loaded file is found.
This subpath is then used in both the the reST file's document title
(so that script's named e.g. "base.bro" actually have some context) and
in figuring out how to interlink with other generated docs of other
scripts that are found in @load directives.
I still need to overhaul things so the loading of "packages" is
documented in a meaningful way and that the CMake targets are able
to generate indexes for packages.
- core.load-unload: scripts that get loaded by default changed, so to
make the test insensitive to that in the future, I changed the test
to just check that the stdout is empty (the @unload'd script would have
had output there)
- policy.frameworks.logging.rotate-custom: I saw that the ordering of
the log postprocessor output caused a failure for me even though the
overall content was the same, so it now sorts that part before diff'ing
- core.print-bpf-filters-ipv[4|6]: packet-filter log file name changed
- policy.protocols.conn.known-services: logging file names changes and
local_nets is now in the Site module
The Logger class is now in charge of reporting all errors, warnings,
informational messages, weirds, and syslogs. All other components
route their messages through the global bro_logger singleton.
The Logger class comes with these reporting methods:
void Message(const char* fmt, ...);
void Warning(const char* fmt, ...);
void Error(const char* fmt, ...);
void FatalError(const char* fmt, ...); // Terminate Bro.
void Weird(const char* name);
[ .. some more Weird() variants ... ]
void Syslog(const char* fmt, ...);
void InternalWarning(const char* fmt, ...);
void InternalError(const char* fmt, ...); // Terminates Bro.
See Logger.h for more information on these.
Generally, the reporting now works as follows:
- All non-fatal message are reported in one of two ways:
(1) At startup (i.e., before we start processing packets),
they are logged to stderr.
(2) During processing, they turn into events:
event log_message%(msg: string, location: string%);
event log_warning%(msg: string, location: string%);
event log_error%(msg: string, location: string%);
The script level can then handle them as desired.
If we don't have an event handler, we fall back to
reporting on stderr.
- All fatal errors are logged to stderr and Bro terminates
immediately.
- Syslog(msg) directly syslogs, but doesn't do anything else.
The three main types of messages can also be generated on the
scripting layer via new Log::* bifs:
Log::error(msg: string);
Log::warning(msg: string);
Log::message(msg: string);
These pass through the bro_logger as well and thus are handled in the
same way. Their output includes location information.
More changes:
- Removed the alarm statement and the alarm_hook event.
- Adapted lots of locations to use the bro_logger, including some
of the messages that were previously either just written to
stdout, or even funneled through the alarm mechanism.
- No distinction anymore between Error() and RunTime(). There's
now only one class of errors; the line was quite blurred already
anyway.
- util.h: all the error()/warn()/message()/run_time()/pinpoint()
functions are gone. Use the bro_logger instead now.
- Script errors are formatted a bit differently due to the
changes. What I've seen so far looks ok to me, but let me know
if there's something odd.
Notes:
- The default handlers for the new log_* events are just dummy
implementations for now since we need to integrate all this into
the new scripts anyway.
- I'm not too happy with the names of the Logger class and its
instance bro_logger. We now have a LogMgr as well, which makes
this all a bit confusing. But I didn't have a good idea for
better names so I stuck with them for now.
Perhaps we should merge Logger and LogMgr?
The http.log that both sides produce is the same, but the
http-related events that each serializes into events.bst
don't look equivalent when read back and cause the test to fail.
I removed the diff'ing against which scripts Bro outputs as loaded
because that's going to be sensitive to changes in which scripts
get loaded by default. What really matters is the output that shows
that Bro was able to load the script via a directory's __load__.bro
This change primarily improves the way Bro detects and prevents
the same script from being loaded twice. It now compares inode
numbers instead of path names.
"conn-id.bro" was the name of a script in the policy/ directory that conn.bro
expected to @load, but when the unit test was also named "conn-id.bro",
it would be loaded twice during the test (once from conn.bro and once
as a command line argument to bro). This means two event handlers were
registered in error and the baseline output contained duplicate lines
that can be removed.
