The extensions now work with optional types, as well with complex types
(like subrecords). Not returning a record in the ext_func no longer
crashes bro.
The default_ext_func was switched to return void in
cases where no extension revord is defined (was bool).
I also got rid of the offsets in the indices - with the rest of the
implementation, that was not really necessary and made the code more
complex.
The "metadata" functionality has been renamed to "ext" to
represent that the logs are being extended. The function that
returns the record which is used to extend the log now receives
a log filter as it's single argument.
The field name "unrolling" is now renamed to "scope" so the variables
names now look like this: "Log::default_scope_sep"
If the analyzer is not found directly attached to the connection,
useless error messages are being output. There are now several
cases where analyzers are attached within other analyzers so the
connection itself doesn't know about the analyzer. This hides
these useless messages.
This adds the capability for the user to attach a reason when removing
or destroying a rule. The message will both be logged in netcontrol.log
and forwarded to the responsible plugins.
Addresses BIT-1655
* origin/topic/dnthayer/ticket1627:
Add a test for starting a cluster with a logger node
Update broctl submodule
Update broctl submodule to branch topic/dnthayer/ticket1627
Change how logger node is detected in cluster framework
Update test baselines for the new logger node type
Update docs for the new logger node type
Add a new node type for logging
This adds an event that is raised once Catch & Release ceases the
block management for an IP address because the IP has not been seen in
traffic during the watch interval.
This allows users who use their own logic on the top of catch and
release know when they will have to start re-blocking the IP if it
occurs in traffic again.
This change introduces error events for Table and Event readers. Users
can now specify an event that is called when an info, warning, or error
is emitted by their input reader. This can, e.g., be used to raise
notices in case errors occur when reading an important input stream.
Example:
event error_event(desc: Input::TableDescription, msg: string, level: Reporter::Level)
{
...
}
event bro_init()
{
Input::add_table([$source="a", $error_ev=error_event, ...]);
}
For the moment, this converts all errors in the Asciiformatter into
warnings (to show that they are non-fatal) - the Reader itself also has
to throw an Error to show that a fatal error occurred and processing
will be abort.
It might be nicer to change this and require readers to mark fatal
errors as such when throwing them.
Addresses BIT-1181
Instead of assuming the logger node always has the name "logger", now
broctl will set a boolean which the cluster framework scripts can use
to determine if there is a logger node or not.
Also removed one line from the manager node script, because it has to do
with logging, which the logger.bro script handles.
Great work, and great documentation!
I'm getting one test failure with
scripts.base.frameworks.netcontrol.catch-and-release-cluster Going
ahead and commiting, Jenkins will show the details I assume.
BIT-1584 #merged
* origin/topic/johanna/netcontrol-improvements:
SMTP does not need to pull in the notice framework.
Write NetControl framework documentation.
Use NetControl for ACTION_DROP of notice framework.
NetControl: slightly update catch and release logging
NetControl: fix several small logging issues
NetControl: more catch and release logging and cluster fix
NetControl: rewrite catch and release and small fixes.
NetControl: find_rules_subnet works in cluster mode
NetControl: fix acld whitelist command
NetControl: add rule exists as state besides added and failure.
NetControl: Suppress duplicate "plugin activated" messages.
NetControl: make new broker plugin options accessible
NetControl: add predicates to broker plugin
Changed the cluster framework scripts by adding a new Bro node type
for doing logging (this is intended to reduce the load on the manager).
If a user chooses not to specify a logger node in the cluster
configuration, then the manager will write logs locally as usual.
When inserting, existance of the given subnet is checked using exact
matching instead of longest prefix matching. Before, inserting a subnet
would have updated the subnet item, which is the longest prefix of the
inserted subnet, if present.
In all versions so far, the identifier string that was used for
comparisons might have been different from the identifier string that
was added (when certain notices are used).
This commit rewrites the way that weirds are logged and fixes a number
of issues on the way. Most prominently, flow weirds now actually log
information about the flow that they occur in (before this change, they
only logged the name of the weird, which is only marginally helpful).
Besides restructuring how weird logging works internally, weirds can now
also be generated by calling Weird::weird with the info record directly,
allowing more fine-granular passing of information. This is e.g. used
for DNS weirds, which do not have the connection record available any
more when they are generated (before data like the connection ID was
just not logged in these instances).
Addresses BIT-1578
File Analysis Framework related code has been moved into a separate
script. Using redefinitions of the corresponding records causes the
file-related columns to appear last.
forgotten messages are only logged on the manager (or standalone host)
now. Logs are not written by default anymore when Bro encounters traffic
that should have been blocked.
This commit rewrites catch and release, fixing issues with it and making
it fully cluster capable. A dedicated netcontrol_catch_release.log is
also added.
This is not quite done yet; a few more log messages are missing. There
should hopefully not be many big issues left.
This introduces two new events, NetControl::rule_new and
NetControl::rule_destroyed, which are raised when rules are first added
and then deleted from the internal state tracking.
- When a log record is being "unrolled" (sub-records flattened
out into a single record), it's now possible to choose the
character/string to separate the outer name from the inner
name. This can be used to work around the problems
with ElasticSearch 2.0 not supporting dots "." in field names.
This value can be provided per-filter as well as a global
default value.
- Log fields can be renamed by providing a table per-filter
(or a global default) to rename fields for any log writer.
The name translation is performed after unrolling so the
value in the field name table must match whatever is being
used to separate field names.
For example if the unrolling separator was set to "*":
redef Log::default_unrolling_sep = "*";
The field name map would need to reflect it:
redef Log::default_field_name_map = {
["id*orig_h"] = "src",
["id*orig_p"] = "src_port",
["id*resp_h"] = "dst",
["id*resp_p"] = "dst_port",
};
The extension mechanism is basically the one that Seth introduced with
his intel extensions. The main difference lies in using a hook instead
of an event. An example policy implements whitelisting.
