* topic/christian/disconnect-slow-peers:
Bump cluster testsuite to pull in Broker backpressure tests
Expand documentation of Broker events.
Add sleep() BiF.
Add backpressure disconnect notification to cluster.log and via telemetry
Remove unneeded @loads from base/misc/version.zeek
Add Cluster::nodeid_to_node() helper function
Support re-peering with Broker peers that fall behind
Add Zeek-level configurability of Broker slow-peer disconnects
Bump Broker to pull in disconnect feature and infinite-loop fix
No need to namespace Cluster:: functions in their own namespace
* origin/topic/vern/CPP-streamlining:
BTest baseline updates for compile-to-C++
mark ZAM regression BTests as not suitable for compile-to-C++
fix for -O gen-C++ maintenance helper to skip BTest intermediary files
introduced simplified initialization for non-standalone -O gen-C++ code tied -O gen-standalone-C++ to use of --optimize-files
streamline generated -O C++ code by relying on per-function profiles rather than aggregate profile
when reporting available/unavailble C++ script bodies, flag those that are skipped
modified AST profiling to mark (and fully skip) non-optimizable functions
modified merge_types() to skip work if given identical types, which also preserves type names (useful for -O gen-C++)
Yes, really. :-) We've hit the need for this on occasion in very specific
settings and always worked around it via ugly nested loops or similars.
This has ample warning that folks normally won't want to use this.
Not sure that ZAM btest should baseline the number of BiFs.
This adds a Broker-specific script to the cluster framework, loaded only when
Zeek is running in cluster mode. It adds logging in cluster.log as well as
telemetry via a metrics counter for Broker-observed backpressure disconnects.
The new zeek_broker_backpressure_disconnects counter, labeled by the neighboring
peer that the reporting node has determined to be unresponsive, counts the
number of unpeerings for this reason.
Here the node "worker" has observed node "proxy" falling behind once:
# HELP zeek_broker_backpressure_disconnects_total Number of Broker peering drops due to a neighbor falling too far behind in message I/O
# TYPE zeek_broker_backpressure_disconnects_total counter
zeek_broker_backpressure_disconnects_total{endpoint="worker",peer="proxy"} 1
Includes small btest baseline update to reflect @load of a new script.
This module is loaded by the telemetry framework, which we're now loading via
the cluster framework, i.e. also in bare mode. The resulting additional
thread (for creating reporter.log) trips up a number of btest baselines.
version.zeek doesn't use any of the string helper functions.
This adds re-peering at the Broker level for peers that Broker decided to
unpeer. We keep this at the Broker level since this behavior is specific to
it (as opposed to other cluster backends).
Includes baseline updates for btests that pick up on the new script's @load.
This analyzer can be used to transport raw stream data for a given
connection to the script layer. For example, adding this analyzer into
the HTTP::upgrade_analyzer or using it to configure a child WebSocket
analyzer allows to get access to the raw stream data in script land
when no more appropriate protocol analyzer is available.
* origin/topic/vern/script-opt-keep-asserts:
ZAM documentation updates for asserts and event handler run-time errors
BTest updates for ZAM support of (optionally) keeping "assert" statements
command-line options for controlling script optimization: keeping asserts, avoiding event handler coalescence
ZAM support for option to not coalesce event handlers
ZAM support for keeping "assert" statements
internal support for script optimization options for keeping asserts, not consolidating event handlers
ZAM operations to support asserts
simplified "assert" by not trying to catch messages that themselves have errors
Fixed some TEST-REQUIRES "${ZEEK_ZAM}" == "1" to use "=" instead to
be /bin/sh compatible.
* origin/topic/vern/zam-pattern-comparison:
update of BTest that tracks number of (and validates) ZAM operations
ZAM support for pattern equality/inequality operations
expanded ZAM operations for bit-shifting to allow for int/count shift values
added type coercion for bit-shifting expressions
The pcap comes from the following dataset [1]:
CTU-SME-11: a labeled dataset with real benign and malicious network
traffic mimicking a small medium-size enterprise environment
[1] https://zenodo.org/records/7958259
This is a serializer for log records that is using SerialTypes
for serializing and un-serializing rather. Essentially, this is
similar to what broker does except for the envelope.
* origin/topic/johanna/sqlite-pragmas:
Options for SQLite log writer, eliminate duplicate definitions
Test synchronous/journal mode options for SQLite log writer
Added default options for synchronous and journal mode
Support for synchronous and journal_mode
* origin/topic/awelzel/pluggable-cluster-backends-part1:
btest: Test Broker::make_event() together with Cluster::publish_hrw()
btest: Add cluster dir, minimal test for enum value
broker: Add shim plugin adding a backend component
zeek-setup: Instantiate backend::manager
cluster: Add to src/CMakeLists.txt
cluster: Add Components and ComponentManager for new components
cluster/Backend: Interface for cluster backends
cluster/Serializer: Interface for event and log serializers
logging: Introduce logging/Types.h
SerialTypes/Field: Allow default construction and add move constructor
DebugLogger: Add cluster debugging stream
plugin: Add component enums for pluggable cluster backends
broker: Pass frame to MakeEvent()
This ordering fixes a test failure we're seeing on Alpine for the
signatures/tcp-end-of-match btest, since discrepancies in rule match traversal
could lead to discrepancies in corresponding event ordering.
It looks safe to rely on across platforms since the index is driven by signature
load order, which shouldn't deviate. If this somehow doesn't hold in the future,
we'll only wind up with a test failure, not incorrect match behavior.
(Correction to 2e03fbb8b0, which I pushed
accidentally.)
