* origin/topic/christian/fix-config-segfault:
Btest tweak for improved type rendering in config framework errors and set types
More precise type information in a config framework error message
Explicitly don't support sets with multiple index types in input/config frameworks
The input framework's Manager::IsCompatibleType() already rejected
sets with multiple index types that aren't all the same (i.e. that are
not pure). Pure ones (e.g. "set[addr,addr]") slipped through and could
cause Zeek to segfault elsewhere in the config framework due to type
comparison subtleties. Note that the ASCII reader can't read such sets
anyway, so this method now rejects sets with any kind of index-type
tuple.
In the config framework, the script-level change handler has a risky
conversion from any to set[bool], which can trigger segfaults when the
underlying set's index is a type tuple. We now prevent this code path
by ensuring it only applies to sets with a single index type.
This change allows users to specify an epoch length of 0, which means
that the user manually has to finish the epochs. A new next_epoch
function is introduced to allow users to manually end epochs.
Addresses GH-348
In non-clustered mode, epoch_finished was not called when there was no
data during the epoch.
This behavior does not fit the documentation, and also is different in
cluster-mode, where epoch_finished is, indeed, called after every epoch.
This small change fixes this behavior.
(Adding a NEWS entry.)
* origin/topic/christian/364-logfilter-hooks:
Update testing/btest/scripts/base/frameworks/logging/hooks.zeek
Btests for log filter policy hooks
Btest baseline updates to reflect new logging policy hooks
Migrate existing use of filter predicates to policy hooks
Support for log filter policy hooks
- Improved documentation/comment for the new option
* 'logging-ascii-enable-shadow-logs' of https://github.com/awelzel/zeek:
logging/ascii: Support leftover log rotation in non-supervisor setups
We have a use case to rotate leftover log files in a non-supervisor
setup. There doesn't seem to be a strict requirement on supervisor
functionality. Allow enabling leftover log rotation through
LogAscii::enable_leftover_log_rotation and redef this for the
logger node in a supervisor setup individually.
This adds a "policy" hook into the logging framework's streams and
filters to replace the existing log filter predicates. The hook
signature is as follows:
hook(rec: any, id: Log::ID, filter: Log::Filter);
The logging manager invokes hooks on each log record. Hooks can veto
log records via a break, and modify them if necessary. Log filters
inherit the stream-level hook, but can override or remove the hook as
needed.
The distribution's existing log streams now come with pre-defined
hooks that users can add handlers to. Their name is standardized as
"log_policy" by convention, with additional suffixes when a module
provides multiple streams. The following adds a handler to the Conn
module's default log policy hook:
hook Conn::log_policy(rec: Conn::Info, id: Log::ID, filter: Log::Filter)
{
if ( some_veto_reason(rec) )
break;
}
By default, this handler will get invoked for any log filter
associated with the Conn::LOG stream.
The existing predicates are deprecated for removal in 4.1 but continue
to work.
This adds two new functions: `Conn::register_removal_hook()` and
`Conn::unregister_removal_hook()` for registering a hook function to be
called back during `connection_state_remove`. The benefit of using hook
callback approach is better scalability: the overhead of unrelated
protocols having to dispatch no-op `connection_state_remove` handlers is
avoided.
* 'action-drop' of https://github.com/LBL-gov/zeek:
Moved verb ACTION_DROP from policy/frameworks/netcontrol/catch-and-release.zeek to base/frameworks/notice/main.zeek.
ACTION_DROP is not only part of catch-n-release subsystem.
Also, historically ACTION_DROP has been bundled with ACTION_LOG, ACTION_ALARM, ACTION_EMAIL... and its helpful that this verb remains in base/frameworks/notice/main.zeek
These are to support actions taken on corsa and keep them distinct from ACLD drops/restore or BGP nullzero/nonullzero.
This does not negatively impacts anything - only adds new verbs for handling zeek-netcontrol actions.
``NetControl::DROP`` had 3 conflicting definitions that could potentially
be used incorrectly without any warnings or type-checking errors.
Such enum redefinition conflicts are now caught and treated as errors,
so the ``NetControl::DROP`` enums had to be renamed:
* The use as enum of type ``Log::ID`` is renamed to ``NetControl::DROP_LOG``
* The use as enum of type ``NetControl::CatchReleaseInfo`` is renamed to
``NetControl::DROP_REQUESTED``
* The use as enum of type ``NetControl::RuleType`` is unchanged and still
named ``NetControl::DROP``
* origin/topic/johanna/table-changes: (26 commits)
TableSync: try to make test more robust & add debug output
Increase timeouts to see if FreeBSD will be happy with this.
Try to make FreeBSD test happy with larger timeout.
TableSync: refactor common functionality into function
TableSync: don't raise &on_change, smaller fixes
TableSync: rename auto_store -> table_store
SyncTables: address feedback part 1 - naming (broker and zeek)
BrokerStore <-> Zeek Tables: cleanup and bug workaround
Zeek Table<->Brokerstore: cleanup, documentation, small fixes
BrokerStore<->Zeek table: adopt to recent Zeek API changes
BrokerStore<->Zeek Tables Fix a few small test failures.
BrokerStore<->Zeek tables: allow setting storage location & tests
BrokerStore<->Zeek tables: &backend works for in-memory stores.
BrokerStore<->Zeek table - introdude &backend attribute
BrokerStore<->Zeek tables: test for clones synchronizing to a master
BrokerStore<->Zeek tables: load persistent tables on startup.
Brokerstore<->Tables: attribute conflicts
Zeek/Brokerstore updates: expiration
Zeek/Brokerstore updates: add test that includes updates from clones
Zeek/Brokerstore updates: first working end-to-end test
...
This commit adds script/c++ documentation and fixes a few loose ends.
It also adds tests for corner cases and massively improves error
messages.
This also actually introduces type-compatibility checking and introduces
a new attribute that lets a user override this if they really know what
they are doing. I am not quite sure if we should really let that stay in
- but it can be very convenient to have this functionality.
One test is continuing to fail - the expiry test is very flaky. This is,
I think, caused by delays of the broker store forwarding. I am unsure if
we can actually do anything about that.
* origin/master: (47 commits)
scan.l: Remove "constant" did_module_restore logic
Fix FreeBSD CI script to install right SWIG package
Update submodule(s)
GH-928: use realpath() instead of inode to de-duplicate scripts
Update submodule(s)
GH-1040: Add zero-indexed version of str_split
Fix WhileStmt to call Stmt(Tag) ctor
GH-1041: Move compress_path to a bif that uses normalize_path
Update submodule(s)
Update submodule(s)
Update submodule(s)
Fix --enable-mobile-ipv6 build
Fix namespace of GetCurrentLocation() to zeek::detail
Add backtrace() and print_backtrace()
Rename BroString files to ZeekString
Update NEWS entry with note about class renames
Rename BroObj to Obj
Rename BroString to zeek::String
Move Func up to zeek namespace, rename BroFunc to ScriptFunc
Mark global val_mgr as deprecated and fix uses of it to use namespaced version
...
These may be redefined to customize log rotation path prefixes,
including use of a directory. File extensions are still up to
individual log writers to add themselves during the actual rotation.
These new also allow for some simplication to the default
ASCII postprocessor function: it eliminates the need for it doing an
extra/awkward rename() operation that only changes the timestamp format.
This also teaches the supervisor framework to use these new options
to rotate ascii logs into a log-queue/ directory with a specific
file name format (intended for an external archiver process to
monitor separately).
Currently this requires using this with a normal cluster - or sending
messages by yourself.
It, in principle, should also work with SQLITE - but that is a bit
nonsensical without being able to change the storage location.
The &backend attribute allows for a much more convenient way of
interacting with brokerstores. One does not need to create a broker
store anymore - instead all of this is done internally.
The current state of this partially works. This should work fine for
persistence - but clones are currently not yet correctly attached.
This introduces a new sampling state-map for expired connections to fix
segfaults that previously occured when passing in a `connection` record
to `Reporter::conn_weird()` for which the internal `Connection` object
had already been expired and deleted. This also introduces a new event
called `expired_conn_weird`, which is similar to `conn_weird`, except
the full `connection` record is no longer available, just the `conn_id`
and UID string.
This commit switches UID hashing from md5 to a highway hash. It also
moves the salt value out of the file plugin - and makes it
installation-specific instead - it is moved to the global namespace.
There now are digest hash functions to make "static"
installation-specific hashes that are stable over workers available to
everyone; hashes can be 64, 128 or 256 bits in size.
Due to the fact that we switch the file hashing algorithm, all file
hashes change.
The underlyigng algorithm that is used for hashing is highwayhash-128,
which is significantly faster than md5.
