* origin/topic/jsiwek/gh-211:
GH-208: change invalid subnet expressions to a runtime error
GH-211: improve consistency of how scripting errors are handled
Removed the 'allow_init_errors' option.
Scripting errors/mistakes now consistently generate a runtime error
which have the behavior of unwinding the call stack all the way out of
the current event handler.
Before, such errors were not treated consistently and either aborted
the process entirely or emitted a message while continuing to execute
subsequent statements without well-defined behavior (possibly causing
a cascade of errors).
The previous behavior also would only unwind out of the current
function (if within a function body), not out the current event
handler, which is especially problematic for functions that return
a value: the caller is essentially left a mess with no way to deal
with it.
This also changes the behavior of the startup/initialization process
to abort if there's errors during bro_init() rather than continue one
to the main run loop. The `allow_init_errors` option may change this
new, default behavior.
Added check for the optional sub-header in ERSPAN Type III as well
as additional truncation checks to the GRE parsing logic in general.
Also added a unit test for ERSPAN Type II.
This fixes some use-after-free issues in the shutdown order of
various systems: I/O, plugin, logging, and reporter systems may
interact during shutdown if there's errors emitted (or maybe just
still pending) during the shutdown.
Fixes GH-188
* 'fatemabw/bro' of https://github.com/fatemabw/bro:
DNSSEC support in Bro
I made several changes:
- renamed event/record types
- reformatted the info added to dns.log
- removed the "addl" scripts that added extended dnssec info to dns.log
- simplifications/improvements to the internal parsing logic
This change allows a weird sampling rate of 0, which completely suppresses
all notifications (previously this crashed Bro). If also fixes the sampling
threshold to work with sampling rates of 0.
* origin/topic/jsiwek/empty-lines:
Add 'smtp_excessive_pending_cmds' weird
Fix SMTP command string comparisons
Improve handling of empty lines in several text protocol analyzers
Add rate-limiting sampling mechanism for weird events
Teach timestamp canonifier about timestamps before ~2001
* origin/topic/vern/vec-append:
d'oh, still have a (deprecated) string_array rather than string_vector
forgot to update test suite results for v += e
reap the fruits of v += e
test case for v += e
documentation of v += e
v += e implemented
Fixed a mistake in find_ip_addresses()
* origin/topic/vern/set-ops2:
documentation, test suite update
implemented set relationals
bug fix for set intersection
set intersection implemented
mirroring previous topic/vern/set-ops to get branch up to date, since I'm a n00b
Fixed a couple memory leaks and added a leak test
The generation of weird events, by default, are now rate-limited
according to these tunable options:
- Weird::sampling_whitelist
- Weird::sampling_threshold
- Weird::sampling_rate
- Weird::sampling_duration
The new get_reporter_stats() BIF also allows one to query the
total number of weirds generated (pre-sampling) which the new
policy/misc/weird-stats.bro script uses periodically to populate
a weird_stats.log.
There's also new reporter BIFs to allow generating weirds from the
script-layer such that they go through the same, internal
rate-limiting/sampling mechanisms:
- Reporter::conn_weird
- Reporter::flow_weird
- Reporter::net_weird
Some of the code was adapted from previous work by Johanna Amann.
In some cases one can get the Type() of unaryexpr to be ANY. Vectors so
far did not deal gracefully with this and crashed because trying to
convert any to a vectortype.
This patch fixes this by just using the original vector-type in this
case.
* origin/topic/vern/case-insensitive-patterns:
use PCRE syntax instead of the beautiful new (?i ...) syntax
nitlet in NEWS entry
test suite update for case-insensitive patterns
document use of double quotes to escape case-insensitivity
bug fix for recent memory leak patch
documentation updates for case-insensitive patterns
d'oh there's isalpha. I looked earlier for isletter :-P
fix for handling [:(lower|upper):] in case-insensitive patterns
implemented /re/i for case-insensitive patterns
Mostly trying to standardize the way tests sleep for arbitrary amounts
of time to make it easier to tell at which particular point the
unit test actually may need the timeout interval increased (or else
debugged further).
* 'Reporter/MessageFix' of https://github.com/catenacyber/bro:
Better reporter for Brostring with embedded NUL
I slightly changed the code for beautification purposes and added a
testcase. No functional changes.
The configuration framework consists of three mostly distinct parts:
* option variables
* the config reader
* the script level framework
I will describe the three elements in the following.
Internally, this commit also performs a range of changes to the Input
manager; it marks a lot of functions as const and introduces a new
ValueToVal method (which could in theory replace the already existing
one - it is a bit more powerful).
This also changes SerialTypes to have a subtype for Values, just as
Fields already have it; I think it was mostly an oversight that this was
not introduced from the beginning. This should not necessitate any code
changes for people already using SerialTypes.
option variable
===============
The option keyword allows variables to be specified as run-tine options.
Such variables cannot be changed using normal assignments. Instead, they
can be changed using Option::set. It is possible to "subscribe" to
options and be notified when an option value changes.
Change handlers can also change values before they are applied; this
gives them the opportunity to reject changes. Priorities can be
specified if there are several handlers for one option.
Example script:
option testbool: bool = T;
function option_changed(ID: string, new_value: bool): bool
{
print fmt("Value of %s changed from %s to %s", ID, testbool, new_value);
return new_value;
}
event bro_init()
{
print "Old value", testbool;
Option::set_change_handler("testbool", option_changed);
Option::set("testbool", F);
print "New value", testbool;
}
config reader
=============
The config reader provides a way to read configuration files back into
Bro. Most importantly it automatically converts values to the correct
types. This is important because it is at least inconvenient (and
sometimes near impossible) to perform the necessary type conversions in
Bro scripts themselves. This is especially true for sets/vectors.
Configuration generally look like this:
[option name][tab/spaces][new variable value]
so, for example:
testaddr 2607:f8b0:4005:801::200e
testinterval 60
testtime 1507321987
test_set a b c d erdbeerschnitzel
The reader uses the option name to look up the type that variable has in
the Bro core and automatically converts the value to the correct type.
Example script use:
type Idx: record {
option_name: string;
};
type Val: record {
option_val: string;
};
global currconfig: table[string] of string = table();
event InputConfig::new_value(name: string, source: string, id: string, value: any)
{
print id, value;
}
event bro_init()
{
Input::add_table([$reader=Input::READER_CONFIG, $source="../configfile", $name="configuration", $idx=Idx, $val=Val, $destination=currconfig, $want_record=F]);
}
Script-level config framework
=============================
The script-level framework ties these two features together and makes
them a bit more convenient to use. Configuration files can simply be
specified by placing them into Config::config_files. The framework also
creates a config.log that shows all value changes that took place.
Usage example:
redef Config::config_files += {configfile};
export {
option testbool : bool = F;
}
The file is now monitored for changes; when a change occurs the
respective option values are automatically updated and the value change
is written to config.log.
handlers.
It's well known that changes to mutable event arguments, like tables,
become visible to all places where those values are used, including
subsequent handlers of the same event. However, there's a related case
that's more suprising: simply assigning *a new value* to an event
argument passes through, too. This commit fixes that behaviour. (We
even had a btest with a baseline reflecting the problen).
IP packets that have a header length that is greater than the total
length of the packet cause a integer overflow, which cause range-checks
to fail, which causes OOB reads.
Furthermore Bro does not currently check the version field of IP packets
that are read from tunnels. I added this check - otherwhise Bro reports
bogus IP information in its error messages, just converting the data
from the place where the IP information is supposed to be to IPs.
This behavior brings us closer to what other software (e.g. Wireshark)
displays in these cases.
This is a small caveat to this implementation. The ethernet
header that is carried over the tunnel is ignored. If a user
tries to do MAC address logging, it will only show the MAC
addresses for the outer tunnel and the inner MAC addresses
will be stripped and not available anywhere.
If connection flipping occured in Sessions.cc code (invoked e.g. when
the original SYN is missing), layer 2 flipping was not performed. This
change switches to always use the connection flipping code in Conn.cc
which performs the switch correctly.
non-partial connections.
Before, if we saw a responder-side SYN/ACK, but had not seen the
initial orginator-side SYN, Bro would treat the connection as partial,
meaning that most application-layer analyzers would refuse to inspect
the payload. That was unfortunate because all payload data was
actually there (and even passed to the analyzers). This change make
Bro consider these connections as complete, so that analyzers will
just normally process them.
The leads to couple more connections in the test-suite to now being
analyzed.
Addresses #1492. (I used an HTTP trace for debugging instead of the
HTTPS trace from the ticket, as the clear-text makes it easier to
track the data flow).
