Yes, really. :-) We've hit the need for this on occasion in very specific
settings and always worked around it via ugly nested loops or similars.
This has ample warning that folks normally won't want to use this.
Not sure that ZAM btest should baseline the number of BiFs.
This adds a Broker-specific script to the cluster framework, loaded only when
Zeek is running in cluster mode. It adds logging in cluster.log as well as
telemetry via a metrics counter for Broker-observed backpressure disconnects.
The new zeek_broker_backpressure_disconnects counter, labeled by the neighboring
peer that the reporting node has determined to be unresponsive, counts the
number of unpeerings for this reason.
Here the node "worker" has observed node "proxy" falling behind once:
# HELP zeek_broker_backpressure_disconnects_total Number of Broker peering drops due to a neighbor falling too far behind in message I/O
# TYPE zeek_broker_backpressure_disconnects_total counter
zeek_broker_backpressure_disconnects_total{endpoint="worker",peer="proxy"} 1
Includes small btest baseline update to reflect @load of a new script.
This module is loaded by the telemetry framework, which we're now loading via
the cluster framework, i.e. also in bare mode. The resulting additional
thread (for creating reporter.log) trips up a number of btest baselines.
version.zeek doesn't use any of the string helper functions.
This adds re-peering at the Broker level for peers that Broker decided to
unpeer. We keep this at the Broker level since this behavior is specific to
it (as opposed to other cluster backends).
Includes baseline updates for btests that pick up on the new script's @load.
This analyzer can be used to transport raw stream data for a given
connection to the script layer. For example, adding this analyzer into
the HTTP::upgrade_analyzer or using it to configure a child WebSocket
analyzer allows to get access to the raw stream data in script land
when no more appropriate protocol analyzer is available.
* origin/topic/vern/script-opt-keep-asserts:
ZAM documentation updates for asserts and event handler run-time errors
BTest updates for ZAM support of (optionally) keeping "assert" statements
command-line options for controlling script optimization: keeping asserts, avoiding event handler coalescence
ZAM support for option to not coalesce event handlers
ZAM support for keeping "assert" statements
internal support for script optimization options for keeping asserts, not consolidating event handlers
ZAM operations to support asserts
simplified "assert" by not trying to catch messages that themselves have errors
Fixed some TEST-REQUIRES "${ZEEK_ZAM}" == "1" to use "=" instead to
be /bin/sh compatible.
* origin/topic/vern/zam-pattern-comparison:
update of BTest that tracks number of (and validates) ZAM operations
ZAM support for pattern equality/inequality operations
expanded ZAM operations for bit-shifting to allow for int/count shift values
added type coercion for bit-shifting expressions
The pcap comes from the following dataset [1]:
CTU-SME-11: a labeled dataset with real benign and malicious network
traffic mimicking a small medium-size enterprise environment
[1] https://zenodo.org/records/7958259
* origin/topic/awelzel/pluggable-cluster-backends-part3:
init-bare/zeek-setup: Groundwork for instantiating cluster backends
cluster/serializer: Add binary-serialization-format
logging/WriterFrontend: Add logic for non-broker cluster backends
logging/WriterBackend: Include logging/Types.h
logging/Manager: Implement new WriteBatchFromRemote()
logging/WriterFrontend: Add LogWriteHeader as member
logging: Add filter_name to WriterInfo
This is a serializer for log records that is using SerialTypes
for serializing and un-serializing rather. Essentially, this is
similar to what broker does except for the envelope.
If cluster::backend isn't broker_mgr, use the WriterFrontend's buffering
logic and send a whole batch of log writes during FlushWriteBuffer().
This is a different path than broker's own logging logic.
Preferably we adapt broker to a model where it isn't
buffering either.
...with this change, it'll be possible to identify WriterFrontend's
based on (stream, filter_name, path) pairs in addition to (stream,
writer, path) pairs.
