* topic/robin/input-threads-merge: (130 commits)
And now it even compiles after my earlier changes.
A set of input framework refactoring, cleanup, and polishing.
another small memory leak in ascii reader:
and another small memory leak when using streaming reads.
fix another memory lead (when updating tables).
Input framework merge in progress.
filters have been called streams for eternity. And I always was too lazy to change it everywhere...
reactivate network_time check in threading manager. previously this line made all input framework tests fail - it works now. Some of the other recent changes of the threading manager must have fixed that problem.
fix up the executeraw test - now it works for the first time and does not always fail
baselines for the autostart removal.
remove last remnants of autostart, which has been removed for quite a while.
make input framework source (hopefully) adhere to the usual indentation style. No functional changes.
fix two memory leaks which occured when one used filters.
update description to current interface.
rename a couple of structures and make the names in manager fit the api more.
fix memory leak in tables and vectors that are read into tables
fix missing get call for heart beat in benchmark reader.
fix heart_beat_interval -- initialization in constructor does not work anymore (probably due to change in init ordering?)
fix memory leak for tables... nearly completely.
fix a couple more leaks. But - still leaking quite a lot with tables.
...
Stack trace context descriptions are no longer limited to 1024 chars
and better error messages are relayed when the arguments to print
commands fail to parse (e.g. an "unknown identifier" was given).
most stuff is inplace, logging framework needs a few changes merged before continuing here...
Conflicts:
src/CMakeLists.txt
src/LogMgr.h
src/logging/Manager.cc
src/main.cc
* origin/topic/jsiwek/brofiler:
Fix superfluous/duplicate data getting in to testing coverage log.
Add "# @no-test" tag to blacklist statements from test coverage analysis.
Test coverage integration for external tests and complete suite.
Integrate Bro script coverage profiling with the btest suite.
Add simple profiling class to accumulate Stmt usage stats across runs.
Renaming environment variable BROFILER_FILE to BRO_PROFILER_FILE for
consistency. Yeah, I know, such a nice name! :)
Use the BROFILER_FILE environment variable to point to a file in
which Stmt usage statistics from Bro script-layer can be output.
This should be able to be used to check Bro script coverage that
that e.g. the entire test suite covers.
- The "private interface" section is still tracked, but no longer rendered.
We can judge the usefulness of it later and add back if there's demand.
- Documentation of event handlers was being treated the same as event
declarations. Now handlers are tracked separately, but not currently
rendered in the generated doc output since usefulness is questionable.
Attributes have state to track whether they're in a record and should
apply to a record field, but this state wasn't being set for TypeDecls
that are part of a redef'd record.
Closes#460
* origin/topic/robin/reporting:
Syslog BiF now goes through the reporter as well.
Avoiding infinite loops when an error message handlers triggers errors itself.
Renaming the Logger to Reporter.
Overhauling the internal reporting of messages to the user.
Updating a bunch of tests/baselines as well.
Conflicts:
aux/broccoli
policy.old/alarm.bro
policy/all.bro
policy/bro.init
policy/frameworks/notice/weird.bro
policy/notice.bro
src/SSL-binpac.cc
src/bro.bif
src/main.cc
The Logger class is now in charge of reporting all errors, warnings,
informational messages, weirds, and syslogs. All other components
route their messages through the global bro_logger singleton.
The Logger class comes with these reporting methods:
void Message(const char* fmt, ...);
void Warning(const char* fmt, ...);
void Error(const char* fmt, ...);
void FatalError(const char* fmt, ...); // Terminate Bro.
void Weird(const char* name);
[ .. some more Weird() variants ... ]
void Syslog(const char* fmt, ...);
void InternalWarning(const char* fmt, ...);
void InternalError(const char* fmt, ...); // Terminates Bro.
See Logger.h for more information on these.
Generally, the reporting now works as follows:
- All non-fatal message are reported in one of two ways:
(1) At startup (i.e., before we start processing packets),
they are logged to stderr.
(2) During processing, they turn into events:
event log_message%(msg: string, location: string%);
event log_warning%(msg: string, location: string%);
event log_error%(msg: string, location: string%);
The script level can then handle them as desired.
If we don't have an event handler, we fall back to
reporting on stderr.
- All fatal errors are logged to stderr and Bro terminates
immediately.
- Syslog(msg) directly syslogs, but doesn't do anything else.
The three main types of messages can also be generated on the
scripting layer via new Log::* bifs:
Log::error(msg: string);
Log::warning(msg: string);
Log::message(msg: string);
These pass through the bro_logger as well and thus are handled in the
same way. Their output includes location information.
More changes:
- Removed the alarm statement and the alarm_hook event.
- Adapted lots of locations to use the bro_logger, including some
of the messages that were previously either just written to
stdout, or even funneled through the alarm mechanism.
- No distinction anymore between Error() and RunTime(). There's
now only one class of errors; the line was quite blurred already
anyway.
- util.h: all the error()/warn()/message()/run_time()/pinpoint()
functions are gone. Use the bro_logger instead now.
- Script errors are formatted a bit differently due to the
changes. What I've seen so far looks ok to me, but let me know
if there's something odd.
Notes:
- The default handlers for the new log_* events are just dummy
implementations for now since we need to integrate all this into
the new scripts anyway.
- I'm not too happy with the names of the Logger class and its
instance bro_logger. We now have a LogMgr as well, which makes
this all a bit confusing. But I didn't have a good idea for
better names so I stuck with them for now.
Perhaps we should merge Logger and LogMgr?
* remotes/origin/topic/jsiwek/doc-framework:
Adding example documentation for a script's use of logging features.
Adding &log attribute to static attr_names array.
Small typo fix.
Bro doc mode now tracks record redefs that extend its field list.
BroBifDoc was unneeded; now dead code, so removed.
Bro doc mode now only does a "shallow" copy of declared record types
Bro's doc mode now terminates after processing bro_init but before net_run
Fixes related to `make doc` handling of script summary text (##! comments)
Overhaul of "doc" build target for generating policy script documentation.
Add parser error hint when in doc mode about checking ## comment syntax.
Move stuff related to policy script documentation from doc/ to doc/scripts/
Fixing example.bro's auto-reST generation baseline test.
with the field.
This works now:
type X: record {
a: table[string] of bool &default=table( ["foo"] = T );
b: table[string] of bool &default=table();
c: set[string] &default=set("A", "B", "C");
d: set[string] &default=set();
};
I think previously the intend was to associate &default with the
table/set (i.e., define the default value for non-existing indices).
However, that was already not working: the error checking was
reporting type mismatches. So, this shouldn't break anything and make
things more consistent.
The &log keyword now operates as discussed:
- When associated with individual record fields, it defines them
as being logged.
- When associated with a complete record type, it defines all fields
to be logged.
- When associated with a record extension, it defines all added
fields to be logged.
Note that for nested record types, the inner fields must likewise
be declared with &log. Consequently, conn_id is now declared with
&log in bro.init.
Vectors are now allowed to be logged and will be recorded as an
ordered set of items.
A bro script's public interface is taken to mean any identifier declared
in the global scope that optionally is exported from some namespace/module.
Or more simply: ID::IsGlobal()
The documentation framework now sees "##Text" and "## Text" as
equivalent documentation comments. This prevents unintentional
indentation in the generated reST as a result of the later style, but
still allows embedded reST markup that relies on indentation of more
than two spaces to work as expected.
Comments associated with record fields and enums values are able
to span multiple "##"-stylized comments, allowing for more robust
reST markup to be embedded.
The documentation framework now tracks record fields through
a new CommentedTypeDecl subclass of TypeDecl that the parser constructs
in parallel with the real TypeDecl.
This adds a new subclass of EnumType, CommentedEnumType, and removes
any previous changes to EnumType that were done to support the
autodoc framework.
Dummy CommentedEnumType and ID's are constructed in parallel with the
real EnumType ID's during parsing and passed on to the autodoc framework.
This allows the generated documentation to track enum redefs, with
a special case being the "Notice" enum type.
"##" style comments before identifiers and "##<" style after identifiers
in the body of an enum type declaration will now show up in the
auto-generated reST documentation.
The scanner can now be told to start/stop producing new token types that
assist in documenting record field types (and eventually enums also).
TOK_DOC:
Produced on "##" style comments; documents the field that follows.
TOK_POST_DOC:
Produced on "##<" style comments; documents the previous field.
This follows rather closely how rotation currently works in
rotate-logs.bro. logging.bro now defines:
# Default rotation interval; zero disables rotation.
const default_rotation_interval = 0secs &redef;
# Default naming suffix format.
const default_rotation_date_format = "%y-%m-%d_%H.%M.%S" &redef;
# Default postprocessor for writers outputting into files.
const default_rotation_postprocessor = "" &redef;
# Default function to construct the name of the rotated file.
# The default implementation includes
# default_rotation_date_format into the file name.
global default_rotation_path_func: function(info: RotationInfo) : string &redef;
Writer support for rotation is optional, usually it will only make
sense for file-based writers.
TODO: Currently, there's no way to customize rotation on a per file
basis, there are only the global defaults as described above.
Individual customization is coming next.
