* origin/topic/johanna/config-framework-fixes:
Fix test that fails now that options are automatically redefable.
Make options redef-able by default.
Ascii formatter: do not complain about port text.
Make parsing of booleans a little bit more lenient.
The ascii formatter already was happy to read ports in the form
"42/tcp"; however it emitted a warning message for each line.
This patch fixes this and adds a bit more testing for the existing
behavior.
BIT-1958 #merged
* origin/topic/johanna/config-cluster:
Add vector to read_config_cluster test.
Fix special-case-bug for vectors in UnaryExpr.
Config: another cluster test-case, this time reading in a file.
Add sending of values to nodes that dropped out.
Continue work on config framework clusterization.
Teach Option::set to unwrap Broker::Data values
Start clusterizing configuration framework.
Mostly trying to standardize the way tests sleep for arbitrary amounts
of time to make it easier to tell at which particular point the
unit test actually may need the timeout interval increased (or else
debugged further).
This test-case has actually revealed an interesting issue - it works as
is, but as soon as one adds a vector, one gets the fun error-message
fatal error in any: BroType::AsVectorType (any/vector) (any)
This will require a bit more digging :).
* origin/topic/dnthayer/test-improvements:
Fix the ip-broken-header.bro test on macOS
Improve reliability of the logging rotate.bro test
Improve reliability of missing-file-initially.bro test
BIT-1929 #merged
Occasionally a few lines in the first part of the output file were
not in the expected order (this seems to be caused by each line in the
output being created by a process that is run in the background but
bro doesn't wait for it to finish). Fixed by sorting the output.
This test has failed numerous times on Travis CI. Fixes to make this
test more reliable: create the does-not-exist.dat file atomically, and
increase wait time after starting bro in order to give all input
streams a chance to try to read the input file.
Also added the input stream name to the test output, in order to make
output easier to understand if the test fails again.
This small change allows the empty field separator to be empty. This
means that we can represent an empty list by a empty input string, which
was not possible before.
Before, an empty empty field separator meant that there is no empty
field - to get back to this behavior one now has to set the empty field
separator to a string that is guaranteed to not be part of the input
data. Note that we did not use "empty" empty field separators anywhere
and I am not aware of this being used by anyone - the new behavior seems
like it is much more useful in practice.
This also changes the config framework to interpret empty lists as...
empty, instead of interpreting them as lists that have one zero-length
element; this seems like the saner default.
The configure reader had a small bug that caused the tracking of changed
variables to be incorrect after the second update. This resulted in
change-events for unchanged variables.
Currently the destructor would try to free unallocated memory. This
could e.g. be triggered by the input framework reading a set with an
invalid element.
get_filter_names(id: ID) : set[string] returns the names of the current
list of filters for a specified log stream.
Furthermore this commit makes a number of logging functions more robust
by checking existence of values before trying to modify them. This
commit also really implements (and tests) the enable_stream function.
There are two new script level functions to query and lookup files
from the core by their IDs. These are adding feature parity for
similarly named functions for files. The function prototypes are
as follows:
Files::file_exists(fuid: string): bool
Files::lookup_File(fuid: string): fa_file
The configuration framework consists of three mostly distinct parts:
* option variables
* the config reader
* the script level framework
I will describe the three elements in the following.
Internally, this commit also performs a range of changes to the Input
manager; it marks a lot of functions as const and introduces a new
ValueToVal method (which could in theory replace the already existing
one - it is a bit more powerful).
This also changes SerialTypes to have a subtype for Values, just as
Fields already have it; I think it was mostly an oversight that this was
not introduced from the beginning. This should not necessitate any code
changes for people already using SerialTypes.
option variable
===============
The option keyword allows variables to be specified as run-tine options.
Such variables cannot be changed using normal assignments. Instead, they
can be changed using Option::set. It is possible to "subscribe" to
options and be notified when an option value changes.
Change handlers can also change values before they are applied; this
gives them the opportunity to reject changes. Priorities can be
specified if there are several handlers for one option.
Example script:
option testbool: bool = T;
function option_changed(ID: string, new_value: bool): bool
{
print fmt("Value of %s changed from %s to %s", ID, testbool, new_value);
return new_value;
}
event bro_init()
{
print "Old value", testbool;
Option::set_change_handler("testbool", option_changed);
Option::set("testbool", F);
print "New value", testbool;
}
config reader
=============
The config reader provides a way to read configuration files back into
Bro. Most importantly it automatically converts values to the correct
types. This is important because it is at least inconvenient (and
sometimes near impossible) to perform the necessary type conversions in
Bro scripts themselves. This is especially true for sets/vectors.
Configuration generally look like this:
[option name][tab/spaces][new variable value]
so, for example:
testaddr 2607:f8b0:4005:801::200e
testinterval 60
testtime 1507321987
test_set a b c d erdbeerschnitzel
The reader uses the option name to look up the type that variable has in
the Bro core and automatically converts the value to the correct type.
Example script use:
type Idx: record {
option_name: string;
};
type Val: record {
option_val: string;
};
global currconfig: table[string] of string = table();
event InputConfig::new_value(name: string, source: string, id: string, value: any)
{
print id, value;
}
event bro_init()
{
Input::add_table([$reader=Input::READER_CONFIG, $source="../configfile", $name="configuration", $idx=Idx, $val=Val, $destination=currconfig, $want_record=F]);
}
Script-level config framework
=============================
The script-level framework ties these two features together and makes
them a bit more convenient to use. Configuration files can simply be
specified by placing them into Config::config_files. The framework also
creates a config.log that shows all value changes that took place.
Usage example:
redef Config::config_files += {configfile};
export {
option testbool : bool = F;
}
The file is now monitored for changes; when a change occurs the
respective option values are automatically updated and the value change
is written to config.log.
The catch-and-release.bro test was failing whenever three conditions
were all true: sorting the netcontrol.log before comparing to
the baseline, the presence of LC_ALL=C in btest.cfg changes the sort
order, and sometimes the timestamp increases slightly beginning
with one of the rule_id == 5 lines.
As a result of these three conditions, the sorted order of the lines
with rule_id of 5 were different than the baseline.
Fixed by not sorting netcontrol.log, as this doesn't seem necessary.
- Addresses Philip Romero's question from the Bro mailing list.
- Adds Microsoft Edge as a detected browser.
- We are now unescaping encoded characters in software names.
This feature can be enabled globally for all logs by setting
LogAscii::gzip_level to a value greater than 0.
This feature can be enabled on a per-log basis by setting gzip-level in
$confic to a value greater than 0.
The changes are now a bit more succinct with less code changes required.
Behavior is tested a little bit more thoroughly and a memory problem
when reading incomplete lines was fixed. ReadHeader also always directly
returns if header reading failed.
Error messages now are back to what they were before the change, if the
new behavior is not used.
I also tweaked the documentation text a bit.
By default, the ASCII reader does not fail on errors anymore.
If there is a problem parsing a line, a reporter warning is
written and parsing continues. If the file is missing or can't
be read, the input thread just tries again on the next heartbeat.
Options have been added to recreate the previous behavior...
const InputAscii::fail_on_invalid_lines: bool;
and
const InputAscii::fail_on_file_problem: bool;
They are both set to `F` by default which makes the input readers
resilient to failure.
