* origin/topic/seth/log-framework-ext:
Log extensions: series of small fixes and new tests.
Change the function for log extension to take a path only and update tests.
Final changes to log framework ext code.
Add logging framework metadata mechanism.
Add unrolling separator & field name map to logging framework.
The extensions now work with optional types, as well with complex types
(like subrecords). Not returning a record in the ext_func no longer
crashes bro.
The default_ext_func was switched to return void in
cases where no extension revord is defined (was bool).
I also got rid of the offsets in the indices - with the rest of the
implementation, that was not really necessary and made the code more
complex.
The "metadata" functionality has been renamed to "ext" to
represent that the logs are being extended. The function that
returns the record which is used to extend the log now receives
a log filter as it's single argument.
The field name "unrolling" is now renamed to "scope" so the variables
names now look like this: "Log::default_scope_sep"
If the analyzer is not found directly attached to the connection,
useless error messages are being output. There are now several
cases where analyzers are attached within other analyzers so the
connection itself doesn't know about the analyzer. This hides
these useless messages.
This adds the capability for the user to attach a reason when removing
or destroying a rule. The message will both be logged in netcontrol.log
and forwarded to the responsible plugins.
Addresses BIT-1655
* origin/topic/dnthayer/ticket1627:
Add a test for starting a cluster with a logger node
Update broctl submodule
Update broctl submodule to branch topic/dnthayer/ticket1627
Change how logger node is detected in cluster framework
Update test baselines for the new logger node type
Update docs for the new logger node type
Add a new node type for logging
This adds an event that is raised once Catch & Release ceases the
block management for an IP address because the IP has not been seen in
traffic during the watch interval.
This allows users who use their own logic on the top of catch and
release know when they will have to start re-blocking the IP if it
occurs in traffic again.
This change introduces error events for Table and Event readers. Users
can now specify an event that is called when an info, warning, or error
is emitted by their input reader. This can, e.g., be used to raise
notices in case errors occur when reading an important input stream.
Example:
event error_event(desc: Input::TableDescription, msg: string, level: Reporter::Level)
{
...
}
event bro_init()
{
Input::add_table([$source="a", $error_ev=error_event, ...]);
}
For the moment, this converts all errors in the Asciiformatter into
warnings (to show that they are non-fatal) - the Reader itself also has
to throw an Error to show that a fatal error occurred and processing
will be abort.
It might be nicer to change this and require readers to mark fatal
errors as such when throwing them.
Addresses BIT-1181
Instead of assuming the logger node always has the name "logger", now
broctl will set a boolean which the cluster framework scripts can use
to determine if there is a logger node or not.
Also removed one line from the manager node script, because it has to do
with logging, which the logger.bro script handles.
Great work, and great documentation!
I'm getting one test failure with
scripts.base.frameworks.netcontrol.catch-and-release-cluster Going
ahead and commiting, Jenkins will show the details I assume.
BIT-1584 #merged
* origin/topic/johanna/netcontrol-improvements:
SMTP does not need to pull in the notice framework.
Write NetControl framework documentation.
Use NetControl for ACTION_DROP of notice framework.
NetControl: slightly update catch and release logging
NetControl: fix several small logging issues
NetControl: more catch and release logging and cluster fix
NetControl: rewrite catch and release and small fixes.
NetControl: find_rules_subnet works in cluster mode
NetControl: fix acld whitelist command
NetControl: add rule exists as state besides added and failure.
NetControl: Suppress duplicate "plugin activated" messages.
NetControl: make new broker plugin options accessible
NetControl: add predicates to broker plugin
Changed the cluster framework scripts by adding a new Bro node type
for doing logging (this is intended to reduce the load on the manager).
If a user chooses not to specify a logger node in the cluster
configuration, then the manager will write logs locally as usual.
This caused test baseline changes in one of the test: notice now ties in
netcontrol due to ACTION_DROP. Catch and release uses the new_connection
event, which was not before triggered and can cause uids to be generated
for connections that are not usually assigned uids in bare mode.
