This adds two new functions: `Conn::register_removal_hook()` and
`Conn::unregister_removal_hook()` for registering a hook function to be
called back during `connection_state_remove`. The benefit of using hook
callback approach is better scalability: the overhead of unrelated
protocols having to dispatch no-op `connection_state_remove` handlers is
avoided.
* 'action-drop' of https://github.com/LBL-gov/zeek:
Moved verb ACTION_DROP from policy/frameworks/netcontrol/catch-and-release.zeek to base/frameworks/notice/main.zeek.
ACTION_DROP is not only part of catch-n-release subsystem.
Also, historically ACTION_DROP has been bundled with ACTION_LOG, ACTION_ALARM, ACTION_EMAIL... and its helpful that this verb remains in base/frameworks/notice/main.zeek
These are to support actions taken on corsa and keep them distinct from ACLD drops/restore or BGP nullzero/nonullzero.
This does not negatively impacts anything - only adds new verbs for handling zeek-netcontrol actions.
``NetControl::DROP`` had 3 conflicting definitions that could potentially
be used incorrectly without any warnings or type-checking errors.
Such enum redefinition conflicts are now caught and treated as errors,
so the ``NetControl::DROP`` enums had to be renamed:
* The use as enum of type ``Log::ID`` is renamed to ``NetControl::DROP_LOG``
* The use as enum of type ``NetControl::CatchReleaseInfo`` is renamed to
``NetControl::DROP_REQUESTED``
* The use as enum of type ``NetControl::RuleType`` is unchanged and still
named ``NetControl::DROP``
* origin/topic/johanna/table-changes: (26 commits)
TableSync: try to make test more robust & add debug output
Increase timeouts to see if FreeBSD will be happy with this.
Try to make FreeBSD test happy with larger timeout.
TableSync: refactor common functionality into function
TableSync: don't raise &on_change, smaller fixes
TableSync: rename auto_store -> table_store
SyncTables: address feedback part 1 - naming (broker and zeek)
BrokerStore <-> Zeek Tables: cleanup and bug workaround
Zeek Table<->Brokerstore: cleanup, documentation, small fixes
BrokerStore<->Zeek table: adopt to recent Zeek API changes
BrokerStore<->Zeek Tables Fix a few small test failures.
BrokerStore<->Zeek tables: allow setting storage location & tests
BrokerStore<->Zeek tables: &backend works for in-memory stores.
BrokerStore<->Zeek table - introdude &backend attribute
BrokerStore<->Zeek tables: test for clones synchronizing to a master
BrokerStore<->Zeek tables: load persistent tables on startup.
Brokerstore<->Tables: attribute conflicts
Zeek/Brokerstore updates: expiration
Zeek/Brokerstore updates: add test that includes updates from clones
Zeek/Brokerstore updates: first working end-to-end test
...
This commit adds script/c++ documentation and fixes a few loose ends.
It also adds tests for corner cases and massively improves error
messages.
This also actually introduces type-compatibility checking and introduces
a new attribute that lets a user override this if they really know what
they are doing. I am not quite sure if we should really let that stay in
- but it can be very convenient to have this functionality.
One test is continuing to fail - the expiry test is very flaky. This is,
I think, caused by delays of the broker store forwarding. I am unsure if
we can actually do anything about that.
* origin/master: (47 commits)
scan.l: Remove "constant" did_module_restore logic
Fix FreeBSD CI script to install right SWIG package
Update submodule(s)
GH-928: use realpath() instead of inode to de-duplicate scripts
Update submodule(s)
GH-1040: Add zero-indexed version of str_split
Fix WhileStmt to call Stmt(Tag) ctor
GH-1041: Move compress_path to a bif that uses normalize_path
Update submodule(s)
Update submodule(s)
Update submodule(s)
Fix --enable-mobile-ipv6 build
Fix namespace of GetCurrentLocation() to zeek::detail
Add backtrace() and print_backtrace()
Rename BroString files to ZeekString
Update NEWS entry with note about class renames
Rename BroObj to Obj
Rename BroString to zeek::String
Move Func up to zeek namespace, rename BroFunc to ScriptFunc
Mark global val_mgr as deprecated and fix uses of it to use namespaced version
...
These may be redefined to customize log rotation path prefixes,
including use of a directory. File extensions are still up to
individual log writers to add themselves during the actual rotation.
These new also allow for some simplication to the default
ASCII postprocessor function: it eliminates the need for it doing an
extra/awkward rename() operation that only changes the timestamp format.
This also teaches the supervisor framework to use these new options
to rotate ascii logs into a log-queue/ directory with a specific
file name format (intended for an external archiver process to
monitor separately).
Currently this requires using this with a normal cluster - or sending
messages by yourself.
It, in principle, should also work with SQLITE - but that is a bit
nonsensical without being able to change the storage location.
The &backend attribute allows for a much more convenient way of
interacting with brokerstores. One does not need to create a broker
store anymore - instead all of this is done internally.
The current state of this partially works. This should work fine for
persistence - but clones are currently not yet correctly attached.
This introduces a new sampling state-map for expired connections to fix
segfaults that previously occured when passing in a `connection` record
to `Reporter::conn_weird()` for which the internal `Connection` object
had already been expired and deleted. This also introduces a new event
called `expired_conn_weird`, which is similar to `conn_weird`, except
the full `connection` record is no longer available, just the `conn_id`
and UID string.
This commit switches UID hashing from md5 to a highway hash. It also
moves the salt value out of the file plugin - and makes it
installation-specific instead - it is moved to the global namespace.
There now are digest hash functions to make "static"
installation-specific hashes that are stable over workers available to
everyone; hashes can be 64, 128 or 256 bits in size.
Due to the fact that we switch the file hashing algorithm, all file
hashes change.
The underlyigng algorithm that is used for hashing is highwayhash-128,
which is significantly faster than md5.
Node-specific topic prefix subscriptions/publications now add a trailing
slash like "zeek/cluster/node/<name>/". Without the trailing slash,
messages attempting to target "proxy-10" may also be sent to "proxy-1"
since subscription matching is prefix-based.
For event/hook handlers that had a previous declaration, any &default
arguments are ineffective. Only &default uses in the initial
prototype's arguments have an effect (that includes if the handler
is actually the site at which the declaration occurs).
Logs that got sent sparsely or burstily would get buffered for long
periods of time since the logic to flush them only does so on the next
log write. In the worst case, a subsequent log write could never happen
and cause a log entry to be indefinitely buffered.
This fix introduces a recurring event/timer to simply flush all pending
logs at frequency of Broker::log_batch_interval.
During merge I split the test for bro_init/bro_done/bro_script_loaded
event errors into individual tests since the other testing of the zeek
versions of those events seemed fine to otherwise keep.
* origin/topic/robin/631-deprecation-v2:
Update NEWS for naming changes.
Small cleanup and updating submodules.
Remove test for legacy plugin.
Remove legancy symlinks in aux/.
Add warnings when loading scripts ending in ".bro", or using legacy environment variables.
Fix missing rename.
No longer symlink local.zeek to local.bro.
Update notice user agent.
Remove old_comm_usage_is_ok.
Remove bro-config.h.in and bro-path-dev.in.
Change Bro wrapper script to now abort when old executable names are still used.
Remove APIs that were explicitly deprecated to be removed in 3.1.
* origin/topic/jsiwek/supervisor: (44 commits)
Add note that Supervisor script APIs are unstable until 4.0
Move command-line arg parsing functions to Options.{h,cc}
Add btests for supervisor stem/leaf process revival
Move supervisor control events into SupervisorControl namespace
Fix supervisor "destroy" call on nodes not currently alive
Move supervisor source files into supervisor/
Address supervisor code re-factoring feedback from Robin
Convert supervisor internals to rapidjson
Add Supervisor documentation
Add supervisor btests
Improve logging of supervised node errors
Fix supervised node inheritence of command-line script paths
Improve normalize_path() util function
Use a timer to check for death of supervised node's parent
Improve supervisor checks for parent process termination
Improve handling of premature supervisor stem exit
Improve supervisor signal handler safety
Remove unused supervisor config options
Cleanup minor Supervisor TODOs
Improve supervisor debug logging
...
Typically in base scripts, Log::create_stream() is called in zeek_init()
handler with &priority=5 such that it will have already been created
in the default zeek_init() &priority=0.
Also have stem process execv() with original command-line arguments so
that they're re-parsed and inherited correctly by supervised-nodes in
the event the stem process needs to be re-created.
