Merge branch 'topic/christian/disconnect-slow-peers'

* topic/christian/disconnect-slow-peers:
  Bump cluster testsuite to pull in Broker backpressure tests
  Expand documentation of Broker events.
  Add sleep() BiF.
  Add backpressure disconnect notification to cluster.log and via telemetry
  Remove unneeded @loads from base/misc/version.zeek
  Add Cluster::nodeid_to_node() helper function
  Support re-peering with Broker peers that fall behind
  Add Zeek-level configurability of Broker slow-peer disconnects
  Bump Broker to pull in disconnect feature and infinite-loop fix
  No need to namespace Cluster:: functions in their own namespace

CHANGES

@@ -1,3 +1,18 @@
+7.1.0-dev.732 | 2024-12-09 23:28:30 -0800
+
+  * Support for Broker I/O backpressure overflow policies (Christian Kreibich, Corelight, and Dominik Charousset)
+
+    - Add sleep() BiF
+    - Add backpressure disconnect notification to cluster.log and via telemetry
+    - Remove unneeded @loads from base/misc/version.zeek
+    - Add Cluster::nodeid_to_node() helper function
+    - Support re-peering with Broker peers that fall behind
+    - Add Zeek-level configurability of Broker slow-peer disconnects
+    - Bump Broker to pull in disconnect feature and infinite-loop fix
+    - No need to namespace Cluster:: functions in their own namespace
+
+  * Update doc submodule [nomail] [skip ci] (zeek-bot)
+
 7.1.0-dev.720 | 2024-12-09 12:22:44 -0700
 
   * Add missing copyright line to headers and cc files (Arne Welzel, Corelight)

NEWS

@@ -55,6 +55,54 @@ New Functionality
   If you observe PostgreSQL traffic in your environment, please provide feedback
   about the analyzer and structure of the new log.
 
+- Broker's message I/O buffering now operates on per-peering granularity at the
+  sender (it was previously global) and provides configurable overflow handling
+  when a fast sender overwhelms a slow receiver, via the following new tunables
+  in the ``Broker`` module:
+
+	const peer_buffer_size = 2048 &redef;
+	const peer_overflow_policy = "disconnect" &redef;
+	const web_socket_buffer_size = 512 &redef;
+	const web_socket_overflow_policy = "disconnect" &redef;
+
+  When a send buffer overflows (i.e., it is full when a node tries to transmit
+  another message), the sender may drop the message and unpeer the slow receiver
+  (policy ``disconnect``, the default), drop the newest message in the buffer
+  (``drop_newest``), or drop the oldest (``drop_oldest``). Buffer sizes are
+  measured in number of messages, not bytes. Note that "sender" and "receiver"
+  are independent of the direction in which Zeek established the peering. After
+  disconnects Zeek automatically tries to re-establish peering with the slow
+  node, in case it recovers.
+
+  Zeek notifies you in two ways of such disconnects:
+
+    * A cluster.log entry for the sending node indicates that a slow peered node
+      has been removed. Here node ``worker01`` has removed a peered ``proxy01`:
+
+	1733468802.626622  worker01  removed due to backpressure overflow: 127.0.0.1:42204/tcp (proxy01)
+
+    * The labeled counter metric ``zeek_broker_backpressure_disconnects_total``
+      in the telemetry framework tracks the number of times such disconnects
+      happen between respective nodes. The following scraped telemetry indicates
+      the same disconnect as above:
+
+	zeek_broker_backpressure_disconnects_total{endpoint="worker01",peer="proxy01"} 1
+
+  To implement custom handling of a backpressure-induced disconnect, add a
+  ``Broker::peer_removed`` event handler, as follows:
+
+	event Broker::peer_removed(endpoint: Broker::EndpointInfo, msg: string)
+		{
+		if ( "caf::sec::backpressure_overflow" !in msg )
+			return;
+
+		# The local node has disconnected the given endpoint,
+		# add your logic here.
+                }
+
+  These new policies fix a problem in which misbehaving nodes could trigger
+  cascading "lockups" of nodes, each ceasing to transmit any messages.
+
 * The LDAP analyzer now supports handling of non-sealed GSS-API WRAP tokens.
 
 * StartTLS support was added to the LDAP analyzer. The SSL analyzer is enabled

VERSION

@@ -1 +1 @@
-7.1.0-dev.720
+7.1.0-dev.732

auxil/broker

@@ -1 +1 @@
-Subproject commit 2a6e6201f7b43e213f2bac3863ca571b659e8a16
+Subproject commit 28cdb7524f73ffa37315f4058f4f48948fe1683a

scripts/base/frameworks/broker/__load__.zeek

@@ -1,3 +1,4 @@
 @load ./main
 @load ./store
 @load ./log
+@load ./backpressure

scripts/base/frameworks/broker/backpressure.zeek

@@ -0,0 +1,35 @@
+##! This handles Broker peers that fall so far behind in handling messages that
+##! this node sends it that the local Broker endpoint decides to unpeer them.
+##! Zeek captures this as follows:
+##!
+##! - In broker.log, with a regular "peer-removed" entry indicating CAF's reason.
+##! - Via eventing through :zeek:see:`Broker::peer_removed` as done in this script.
+##!
+##! The cluster framework additionally captures the unpeering as follows:
+##!
+##! - In cluster.log, with a higher-level message indicating the node names involved.
+##! - Via telemetry, using a labeled counter.
+
+event Broker::peer_removed(endpoint: Broker::EndpointInfo, msg: string)
+	{
+	if ( "caf::sec::backpressure_overflow" !in msg ) {
+		return;
+	}
+
+	if ( ! endpoint?$network ) {
+		Reporter::error(fmt("Missing network info to re-peer with %s", endpoint$id));
+		return;
+	}
+
+	# Re-establish the peering so Broker's reconnect behavior kicks in once
+	# the other endpoint catches up. Broker will periodically re-try
+	# connecting as necessary. If the other endpoint originally connected to
+	# us, our attempt will fail (since we attempt to connect to the peer's
+	# ephemeral port), but in that case the peer will reconnect with us once
+	# it recovers.
+	#
+	# We could do this more cleanly by leveraging information from the
+	# cluster framework (since it knows who connects to whom), but that
+	# would further entangle Broker into it.
+	Broker::peer(endpoint$network$address, endpoint$network$bound_port);
+}

scripts/base/frameworks/broker/main.zeek

@@ -86,6 +86,24 @@ export {
 	## ZEEK_BROKER_MAX_THREADS environment variable overrides this setting.
 	const max_threads = 1 &redef;
 
+	## Max number of items we buffer at most per peer. What action to take when
+	## the buffer reaches its maximum size is determined by
+	## `peer_overflow_policy`.
+	const peer_buffer_size = 2048 &redef;
+
+	## Configures how Broker responds to peers that cannot keep up with the
+	## incoming message rate. Available strategies:
+	## - disconnect: drop the connection to the unresponsive peer
+	## - drop_newest: replace the newest message in the buffer
+	## - drop_oldest: removed the olsted message from the buffer, then append
+	const peer_overflow_policy = "disconnect" &redef;
+
+	## Same as `peer_buffer_size` but for WebSocket clients.
+	const web_socket_buffer_size = 512 &redef;
+
+	## Same as `peer_overflow_policy` but for WebSocket clients.
+	const web_socket_overflow_policy = "disconnect" &redef;
+
 	## The CAF scheduling policy to use.  Available options are "sharing" and
 	## "stealing".  The "sharing" policy uses a single, global work queue along
 	## with mutex and condition variable used for accessing it, which may be

scripts/base/frameworks/cluster/__load__.zeek

@@ -14,6 +14,9 @@ redef Broker::log_topic = Cluster::rr_log_topic;
 # Add a cluster prefix.
 @prefixes += cluster
 
+# This should soon condition on loading only when Broker is in use.
+@load ./broker-backpressure
+
 @if ( Supervisor::is_supervised() )
 # When running a supervised cluster, populate Cluster::nodes from the node table
 # the Supervisor provides to new Zeek nodes. The management framework configures

scripts/base/frameworks/cluster/broker-backpressure.zeek

@@ -0,0 +1,29 @@
+# Notifications for Broker-reported backpressure overflow.
+# See base/frameworks/broker/backpressure.zeek for context.
+
+@load base/frameworks/telemetry
+
+module Cluster;
+
+global broker_backpressure_disconnects_cf = Telemetry::register_counter_family([
+    $prefix="zeek",
+    $name="broker-backpressure-disconnects",
+    $unit="",
+    $label_names=vector("peer"),
+    $help_text="Number of Broker peerings dropped due to a neighbor falling behind in message I/O",
+]);
+
+event Broker::peer_removed(endpoint: Broker::EndpointInfo, msg: string)
+	{
+	if ( ! endpoint?$network || "caf::sec::backpressure_overflow" !in msg )
+		return;
+
+	local nn = nodeid_to_node(endpoint$id);
+
+	Cluster::log(fmt("removed due to backpressure overflow: %s%s:%s (%s)",
+	                 nn$name != "" ? "" : "non-cluster peer ",
+	                 endpoint$network$address, endpoint$network$bound_port,
+	                 nn$name != "" ? nn$name : endpoint$id));
+	Telemetry::counter_family_inc(broker_backpressure_disconnects_cf,
+	                              vector(nn$name != "" ? nn$name : "unknown"));
+	}

scripts/base/frameworks/cluster/main.zeek

@@ -281,6 +281,15 @@ export {
 	##          a given cluster node.
 	global nodeid_topic: function(id: string): string;
 
+	## Retrieve the cluster-level naming of a node based on its node ID,
+	## a backend-specific identifier.
+	##
+	## id: the node ID of a peer.
+	##
+	## Returns: the :zeek:see:`Cluster::NamedNode` for the requested node, if
+	##          known, otherwise a "null" instance with an empty name field.
+	global nodeid_to_node: function(id: string): NamedNode;
+
 	## Initialize the cluster backend.
 	##
 	## Cluster backends usually invoke this from a :zeek:see:`zeek_init` handler.
@@ -336,7 +345,7 @@ function nodes_with_type(node_type: NodeType): vector of NamedNode
 		{ return strcmp(n1$name, n2$name); });
 	}
 
-function Cluster::get_node_count(node_type: NodeType): count
+function get_node_count(node_type: NodeType): count
 	{
 	local cnt = 0;
 
@@ -349,7 +358,7 @@ function Cluster::get_node_count(node_type: NodeType): count
 	return cnt;
 	}
 
-function Cluster::get_active_node_count(node_type: NodeType): count
+function get_active_node_count(node_type: NodeType): count
 	{
 	return node_type in active_node_ids ? |active_node_ids[node_type]| : 0;
 	}
@@ -394,6 +403,17 @@ function nodeid_topic(id: string): string
 	return nodeid_topic_prefix + id + "/";
 	}
 
+function nodeid_to_node(id: string): NamedNode
+	{
+	for ( name, n in nodes )
+		{
+		if ( n?$id && n$id == id )
+			return NamedNode($name=name, $node=n);
+		}
+
+	return NamedNode($name="", $node=[$node_type=NONE, $ip=0.0.0.0]);
+	}
+
 event Cluster::hello(name: string, id: string) &priority=10
 	{
 	if ( name !in nodes )

scripts/base/misc/version.zeek

@@ -2,9 +2,6 @@
 ##! The most convenient way to access this are the Version::number
 ##! and Version::info constants.
 
-@load base/frameworks/reporter
-@load base/utils/strings
-
 module Version;
 
 export {

src/broker/Manager.cc

@@ -257,6 +257,36 @@ void Manager::DoInitPostScript() {
     options.disable_forwarding = ! get_option("Broker::forward_messages")->AsBool();
     options.use_real_time = use_real_time;
 
+    options.peer_buffer_size = get_option("Broker::peer_buffer_size")->AsCount();
+    auto peer_overflow_policy = get_option("Broker::peer_overflow_policy")->AsString()->CheckString();
+    if ( util::streq(peer_overflow_policy, "disconnect") ) {
+        options.peer_overflow_policy = broker::overflow_policy::disconnect;
+    }
+    else if ( util::streq(peer_overflow_policy, "drop_oldest") ) {
+        options.peer_overflow_policy = broker::overflow_policy::drop_oldest;
+    }
+    else if ( util::streq(peer_overflow_policy, "drop_newest") ) {
+        options.peer_overflow_policy = broker::overflow_policy::drop_newest;
+    }
+    else {
+        reporter->FatalError("Invalid Broker::peer_overflow_policy: %s", peer_overflow_policy);
+    }
+
+    options.web_socket_buffer_size = get_option("Broker::web_socket_buffer_size")->AsCount();
+    auto web_socket_overflow_policy = get_option("Broker::web_socket_overflow_policy")->AsString()->CheckString();
+    if ( util::streq(web_socket_overflow_policy, "disconnect") ) {
+        options.web_socket_overflow_policy = broker::overflow_policy::disconnect;
+    }
+    else if ( util::streq(web_socket_overflow_policy, "drop_oldest") ) {
+        options.web_socket_overflow_policy = broker::overflow_policy::drop_oldest;
+    }
+    else if ( util::streq(web_socket_overflow_policy, "drop_newest") ) {
+        options.web_socket_overflow_policy = broker::overflow_policy::drop_newest;
+    }
+    else {
+        reporter->FatalError("Invalid Broker::web_socket_overflow_policy: %s", web_socket_overflow_policy);
+    }
+
     broker::configuration config{std::move(options)};
 
     config.openssl_cafile(get_option("Broker::ssl_cafile")->AsString()->CheckString());

src/broker/comm.bif

@@ -7,16 +7,56 @@
 
 module Broker;
 
-## Generated when something changes in the Broker sub-system.
-event Broker::status%(endpoint: EndpointInfo, msg: string%);
-
-## Generated when a new peering has been established.
+## Generated when a new peering has been established. Both sides of the peering
+## receive this event, created independently in each endpoint. For the endpoint
+## establishing the peering, the added endpoint's network information will match
+## the address and port provided to :zeek:see:`Broker::peer`; for the listening
+## endpoint it's the peer's TCP client's address and (likely ephemeral) TCP
+## port.
+##
+## endpoint: the added endpoint's Broker ID and connection information.
+##
+## msg: a message providing additional context.
+##
+## .. zeek:see:: Broker::peer_removed Broker::peer_lost
+##    Broker::endpoint_discovered Broker::endpoint_unreachable
+##    Broker::status Broker::error
 event Broker::peer_added%(endpoint: EndpointInfo, msg: string%);
 
-## Generated when an existing peer has been removed.
+## Generated when the local endpoint has removed its peering with another
+## endpoint. This event can fire for multiple reasons, such as a local call to
+## :zeek:see:`Broker::unpeer`, or because Broker autonomously decides to
+## unpeer. One reason it might do this is message I/O backpressure overflow,
+## meaning that the remote peer cannot keep up with the stream of messages the
+## local endpoint sends it. Regardless of the cause, the remote endpoint will
+## locally trigger a corresponding :zeek:see:`Broker::peer_lost` event once the
+## peering ends. These events are independent of the original directionality of
+## TCP connection establishment and only reflect which endpoint terminates the
+## peering.
+##
+## endpoint: the removed endpoint's Broker ID and connection information.
+##
+## msg: a message providing additional context. If backpressure overflow
+##      caused this unpeering, the message contains the string
+##      *caf::sec::backpressure_overflow*.
+##
+## .. zeek:see:: Broker::peer_added Broker::peer_lost
+##    Broker::endpoint_discovered Broker::endpoint_unreachable
+##    Broker::status Broker::error
 event Broker::peer_removed%(endpoint: EndpointInfo, msg: string%);
 
-## Generated when an existing peering has been lost.
+## Generated when the local endpoint has lost its peering with another
+## endpoint. This event fires when the other endpoint stops or removes the
+## peering for some other reason. This event is independent of the original
+## directionality of connection establishment.
+##
+## endpoint: the lost endpoint's Broker ID and connection information.
+##
+## msg: a message providing additional context.
+##
+## .. zeek:see:: Broker::peer_added Broker::peer_removed
+##    Broker::endpoint_discovered Broker::endpoint_unreachable
+##    Broker::status Broker::error
 event Broker::peer_lost%(endpoint: EndpointInfo, msg: string%);
 
 ## Generated when a new Broker endpoint appeared.
@@ -25,7 +65,29 @@ event Broker::endpoint_discovered%(endpoint: EndpointInfo, msg: string%);
 ## Generated when the last path to a Broker endpoint has been lost.
 event Broker::endpoint_unreachable%(endpoint: EndpointInfo, msg: string%);
 
-## Generated when an error occurs in the Broker sub-system.
+## Generated when an unspecified change occurs in Broker. This event only fires
+## when the status change isn't covered by more specific Broker events. The
+## provided message string may be empty.
+##
+## endpoint: the Broker ID and connection information, if available,
+##           of the endpoint the update relates to.
+##
+## msg: a message providing additional context.
+##
+## .. zeek:see:: Broker::peer_added Broker::peer_removed Broker::peer_lost
+##    Broker::endpoint_discovered Broker::endpoint_unreachable Broker::error
+event Broker::status%(endpoint: EndpointInfo, msg: string%);
+
+## Generated when an error occurs in the Broker sub-system. This event
+## reports local errors in Broker, as indicated by the provided
+## :zeek:type:`Broker::ErrorCode`.
+##
+## code: the type of error that triggered this event.
+##
+## msg: a message providing additional context.
+##
+## .. zeek:see:: Broker::peer_added Broker::peer_removed Broker::peer_lost
+##    Broker::endpoint_discovered Broker::endpoint_unreachable Broker::status
 event Broker::error%(code: ErrorCode, msg: string%);
 
 ## Enumerates the possible error types.

src/script_opt/FuncInfo.cc

@@ -431,6 +431,7 @@ static std::unordered_map<std::string, unsigned int> func_attrs = {
     {"skip_further_processing", ATTR_NO_SCRIPT_SIDE_EFFECTS},
     {"skip_http_entity_data", ATTR_NO_SCRIPT_SIDE_EFFECTS},
     {"skip_smtp_data", ATTR_NO_SCRIPT_SIDE_EFFECTS},
+    {"sleep", ATTR_NO_SCRIPT_SIDE_EFFECTS},
     {"split_string", ATTR_FOLDABLE},
     {"split_string1", ATTR_FOLDABLE},
     {"split_string_all", ATTR_FOLDABLE},

src/zeek.bif

@@ -600,6 +600,27 @@ function piped_exec%(program: string, to_write: string%): bool
 	return zeek::val_mgr->True();
 	%}
 
+## Sleeps for the given amount of time.
+##
+## i: The time interval to sleep for.
+##
+## Returns: The :zeek:type:`interval` Zeek actually slept for.
+##
+## .. note::
+##
+##      This is a blocking sleep! Zeek will not run most of its processing
+##      during that time. You almost certainly DO NOT WANT THIS outside
+##      of specific testing/troubleshooting scenarios. To sleep asynchronously,
+##      :zeek:see:`schedule` an event, or consider :zeek:id:`Exec::run`.
+function sleep%(i: interval%): interval
+	%{
+	const auto start = std::chrono::high_resolution_clock::now();
+	std::this_thread::sleep_for(std::chrono::duration<double>(i));
+	const auto end = std::chrono::high_resolution_clock::now();
+	const auto slept = std::chrono::duration<double>(end - start).count();
+	return zeek::make_intrusive<zeek::IntervalVal>(slept);
+        %}
+
 %%{
 #include "zeek/OpaqueVal.h"
 %%}

testing/btest/Baseline.zam/opt.ZAM-bif-tracking/output

@@ -1,2 +1,2 @@
 ### BTest baseline data generated by btest-diff. Do not edit. Use "btest -U/-u" to update. Requires BTest >= 0.63.
-539 seen BiFs, 0 unseen BiFs (), 0 new BiFs ()
+540 seen BiFs, 0 unseen BiFs (), 0 new BiFs ()

testing/btest/Baseline/bifs.sleep/out

@@ -0,0 +1 @@
+### BTest baseline data generated by btest-diff. Do not edit. Use "btest -U/-u" to update. Requires BTest >= 0.63.

testing/btest/Baseline/coverage.bare-load-baseline/canonified_loaded_scripts.log

@@ -119,6 +119,7 @@ scripts/base/init-frameworks-and-bifs.zeek
       build/scripts/base/bif/data.bif.zeek
       build/scripts/base/bif/store.bif.zeek
     scripts/base/frameworks/broker/log.zeek
+    scripts/base/frameworks/broker/backpressure.zeek
   scripts/base/frameworks/supervisor/__load__.zeek
     scripts/base/frameworks/supervisor/control.zeek
     scripts/base/frameworks/supervisor/main.zeek

testing/btest/Baseline/coverage.default-load-baseline/canonified_loaded_scripts.log

@@ -119,6 +119,7 @@ scripts/base/init-frameworks-and-bifs.zeek
       build/scripts/base/bif/data.bif.zeek
       build/scripts/base/bif/store.bif.zeek
     scripts/base/frameworks/broker/log.zeek
+    scripts/base/frameworks/broker/backpressure.zeek
   scripts/base/frameworks/supervisor/__load__.zeek
     scripts/base/frameworks/supervisor/control.zeek
     scripts/base/frameworks/supervisor/main.zeek

testing/btest/Baseline/coverage.init-default/missing_loads

@@ -1,4 +1,5 @@
 ### BTest baseline data generated by btest-diff. Do not edit. Use "btest -U/-u" to update. Requires BTest >= 0.63.
+-./frameworks/cluster/broker-backpressure.zeek
 -./frameworks/cluster/broker-stores.zeek
 -./frameworks/cluster/nodes/logger.zeek
 -./frameworks/cluster/nodes/manager.zeek

testing/btest/Baseline/plugins.hooks/output

@@ -461,6 +461,7 @@
 0.000000   MetaHookPost  LoadFile(0, ./addrs, <...>/addrs.zeek) -> -1
 0.000000   MetaHookPost  LoadFile(0, ./analyzer.bif.zeek, <...>/analyzer.bif.zeek) -> -1
 0.000000   MetaHookPost  LoadFile(0, ./api, <...>/api.zeek) -> -1
+0.000000   MetaHookPost  LoadFile(0, ./backpressure, <...>/backpressure.zeek) -> -1
 0.000000   MetaHookPost  LoadFile(0, ./bloom-filter.bif.zeek, <...>/bloom-filter.bif.zeek) -> -1
 0.000000   MetaHookPost  LoadFile(0, ./cardinality-counter.bif.zeek, <...>/cardinality-counter.bif.zeek) -> -1
 0.000000   MetaHookPost  LoadFile(0, ./cluster.bif.zeek, <...>/cluster.bif.zeek) -> -1
@@ -766,6 +767,7 @@
 0.000000   MetaHookPost  LoadFileExtended(0, ./addrs, <...>/addrs.zeek) -> (-1, <no content>)
 0.000000   MetaHookPost  LoadFileExtended(0, ./analyzer.bif.zeek, <...>/analyzer.bif.zeek) -> (-1, <no content>)
 0.000000   MetaHookPost  LoadFileExtended(0, ./api, <...>/api.zeek) -> (-1, <no content>)
+0.000000   MetaHookPost  LoadFileExtended(0, ./backpressure, <...>/backpressure.zeek) -> (-1, <no content>)
 0.000000   MetaHookPost  LoadFileExtended(0, ./bloom-filter.bif.zeek, <...>/bloom-filter.bif.zeek) -> (-1, <no content>)
 0.000000   MetaHookPost  LoadFileExtended(0, ./cardinality-counter.bif.zeek, <...>/cardinality-counter.bif.zeek) -> (-1, <no content>)
 0.000000   MetaHookPost  LoadFileExtended(0, ./cluster.bif.zeek, <...>/cluster.bif.zeek) -> (-1, <no content>)
@@ -1403,6 +1405,7 @@
 0.000000   MetaHookPre   LoadFile(0, ./addrs, <...>/addrs.zeek)
 0.000000   MetaHookPre   LoadFile(0, ./analyzer.bif.zeek, <...>/analyzer.bif.zeek)
 0.000000   MetaHookPre   LoadFile(0, ./api, <...>/api.zeek)
+0.000000   MetaHookPre   LoadFile(0, ./backpressure, <...>/backpressure.zeek)
 0.000000   MetaHookPre   LoadFile(0, ./bloom-filter.bif.zeek, <...>/bloom-filter.bif.zeek)
 0.000000   MetaHookPre   LoadFile(0, ./cardinality-counter.bif.zeek, <...>/cardinality-counter.bif.zeek)
 0.000000   MetaHookPre   LoadFile(0, ./cluster.bif.zeek, <...>/cluster.bif.zeek)
@@ -1708,6 +1711,7 @@
 0.000000   MetaHookPre   LoadFileExtended(0, ./addrs, <...>/addrs.zeek)
 0.000000   MetaHookPre   LoadFileExtended(0, ./analyzer.bif.zeek, <...>/analyzer.bif.zeek)
 0.000000   MetaHookPre   LoadFileExtended(0, ./api, <...>/api.zeek)
+0.000000   MetaHookPre   LoadFileExtended(0, ./backpressure, <...>/backpressure.zeek)
 0.000000   MetaHookPre   LoadFileExtended(0, ./bloom-filter.bif.zeek, <...>/bloom-filter.bif.zeek)
 0.000000   MetaHookPre   LoadFileExtended(0, ./cardinality-counter.bif.zeek, <...>/cardinality-counter.bif.zeek)
 0.000000   MetaHookPre   LoadFileExtended(0, ./cluster.bif.zeek, <...>/cluster.bif.zeek)
@@ -2346,6 +2350,7 @@
 0.000000 | HookLoadFile  ./api <...>/api.zeek
 0.000000 | HookLoadFile  ./archive <...>/archive.sig
 0.000000 | HookLoadFile  ./audio <...>/audio.sig
+0.000000 | HookLoadFile  ./backpressure <...>/backpressure.zeek
 0.000000 | HookLoadFile  ./bloom-filter.bif.zeek <...>/bloom-filter.bif.zeek
 0.000000 | HookLoadFile  ./cardinality-counter.bif.zeek <...>/cardinality-counter.bif.zeek
 0.000000 | HookLoadFile  ./cluster.bif.zeek <...>/cluster.bif.zeek
@@ -2651,6 +2656,7 @@
 0.000000 | HookLoadFileExtended ./api <...>/api.zeek
 0.000000 | HookLoadFileExtended ./archive <...>/archive.sig
 0.000000 | HookLoadFileExtended ./audio <...>/audio.sig
+0.000000 | HookLoadFileExtended ./backpressure <...>/backpressure.zeek
 0.000000 | HookLoadFileExtended ./bloom-filter.bif.zeek <...>/bloom-filter.bif.zeek
 0.000000 | HookLoadFileExtended ./cardinality-counter.bif.zeek <...>/cardinality-counter.bif.zeek
 0.000000 | HookLoadFileExtended ./cluster.bif.zeek <...>/cluster.bif.zeek

testing/btest/Baseline/scripts.base.misc.version/.stderr

@@ -1,4 +1,4 @@
 ### BTest baseline data generated by btest-diff. Do not edit. Use "btest -U/-u" to update. Requires BTest >= 0.63.
-error in <...>/version.zeek, line 63: Version string 1 cannot be parsed
-error in <...>/version.zeek, line 63: Version string 1.12-beta-drunk-too-much cannot be parsed
-error in <...>/version.zeek, line 63: Version string JustARandomString cannot be parsed
+error in <...>/version.zeek, line 60: Version string 1 cannot be parsed
+error in <...>/version.zeek, line 60: Version string 1.12-beta-drunk-too-much cannot be parsed
+error in <...>/version.zeek, line 60: Version string JustARandomString cannot be parsed

testing/btest/bifs/sleep.zeek

@@ -0,0 +1,21 @@
+# Verifies sleep()'s reported latencies.
+#
+# @TEST-EXEC: zeek -b %INPUT 2>out
+# @TEST-EXEC: btest-diff out
+
+function test_sleep(i: interval)
+	{
+	local start = current_time();
+	local sleep_delay = sleep(i);
+	local script_delay = current_time() - start;
+
+	assert script_delay >= i, fmt("sleep() took %s, less than %s", script_delay, i);
+	assert sleep_delay >= i, fmt("slept for %s, less than %s", script_delay, i);
+	assert sleep_delay <= script_delay, fmt("sleep() claims %s, longer than %s", sleep_delay, script_delay);
+	}
+
+event zeek_init()
+	{
+	test_sleep(100msec);
+	test_sleep(1sec);
+	}

testing/btest/opt/ZAM-bif-tracking.zeek

@@ -464,6 +464,7 @@ global known_BiFs = set(
 	"skip_further_processing",
 	"skip_http_entity_data",
 	"skip_smtp_data",
+	"sleep",
 	"sort",
 	"split_string",
 	"split_string1",

testing/external/commit-hash.zeek-testing-cluster

@@ -1 +1 @@
-d2987b0bc07cb70bd2f8f707b372fb852147b71f
+aa361fc9f5fba202a9df68717a1d403be5f1e6b9