* topic/christian/disconnect-slow-peers:
Bump cluster testsuite to pull in Broker backpressure tests
Expand documentation of Broker events.
Add sleep() BiF.
Add backpressure disconnect notification to cluster.log and via telemetry
Remove unneeded @loads from base/misc/version.zeek
Add Cluster::nodeid_to_node() helper function
Support re-peering with Broker peers that fall behind
Add Zeek-level configurability of Broker slow-peer disconnects
Bump Broker to pull in disconnect feature and infinite-loop fix
No need to namespace Cluster:: functions in their own namespace
The old `c_str_safe` utility function allowed Zeek to operator on
`broker::data` and `broker::variant`. The former grants access to actual
`std::string` objects while the latter only provides access to fields
via `std::string_view`. Since the Zeek formatting functions need null
terminated strings, we need to copy the characters into a
null-terminated container first.
After removing support for `broker::data` and `broker::variant` from the
same code paths, we can drop `c_str_safe` and always do the copying
(since we are always dealing with `broker::variant` now).
Since the transition to broker::variant has been long finalized, there
is no more need to be able to go back to a pre-variant version of
Broker. Hence, we can drop various utilities that allow Zeek to run with
older Broker releases.
This was added previously in the 7.1 cycle. Now that MakeEvent() was
removed from cluster::Backend, there's no need for Broker to provide
this version.
Discussed with @J-Gras, calling Broker::publish() within a scheduled
should use the "intended timestamp" implicitly.
This is subtle, but supposedly more expected when running
a pcap replay cluster.
Now that we run callbacks on the main loop, we can move callback support
for Counter and Gauge instances directly into Zeek and don't need to patch
prometheus-cpp anymore.
This allows to leverage automatic memory management, less allocations
and using move semantics for expressing ownership.
This breaks the existing logging and broker API, but keeps the plugin
DoWrite() and HookLogWrite() methods functioning.
It further changes ValToLogVal to return a threading::Value rather than
a threading::Value*. The vector_val and set_val fields unfortunately
use the same pointer-to-array-of-pointers approach. this can'tbe changed
as it'd break backwards compatibility for plugin provided input readers
and log writers.
The variadic broker messaging BIFs currently convert @ARGS@ into a
ValPList before passing it on to MakeEvent(). This appears historic
plumbing. Implement the same functionality using Span<const ValPtr>
and do the extra copying in the now deprecated MakeEvent().
Further, make passing a frame optional as not all callers may
have one available.
- With `broker::data`, we always have actual `std::string` objects that
we can pass to C functions expecting a null-terminated string.
However, `broker::variant` will return a `std::string_view` where we
have previously received a `std::string`. Hence, we add an extra level
of indirection that ensures that views are converted to
null-terminated strings and also use `c_str()` where we have
previously used `data()`. The former is not present on a
`std::string_view`. Using this member function instead acts as an
extra level of insurance that we do not accidentally pass the bytes
from a view to a C function.
- Switch from error and status views to actual error and status objects.
The view types from Broker only work with `broker::data` and thus
won't be available with `broker::variant`.
By avoiding to use `broker::data` directly, we gain a degree of freedom
that allows us to swap out `broker::data` for something else (e.g.,
`broker::variant`) in the future. Furthermore, it also helps us to keep
Broker types "local" to the Broker manager and gives us a nicer
interface.
Also replaces uses of `broker::expected` with `std::optional`. While an
`expected `can carry additional information as to why a value is not
present, nothing in Zeek ever cared about that. Hence, using
`std::optional` removes an unnecessary dependency on a Broker detail
while also being more efficient (no extra heap allocation when no value
is present).
This largely copies over Spicy's `.clang-format` configuration file. The
one place where we deviate is header include order since Zeek depends on
headers being included in a certain order.
Remove the special case of initializing network time if it hasn't
happened yet. The argument about broker.log containing 0.0 timestamps
is more a problem of the log, not something that would justify modifying
network time globally. For broker.log and possibly cluster.log, it might
be more reasonable to use current time, anyway.
I was a bit wary about tables backed by broker stores being populated
with network_time set to 0.0, but there seems to exist logic and assumptions
that this is okay: It should be the same as if one populates a table with
expirations set within zeek_init().
In fact, staring a bit more, *not setting* network time might be more correct
as workers that don't see packets would never set zeek_start_network_time
which is used within the expiration computation.
Run the broker in non-realtime mode when allow_network_time_forward=F.
This may need an extra option for really advanced use-cases, but for
now this seems reasonable.
network_time forwarding will happen in the main-loop before draining the
EventMgr so timers/events scheduled based on broker messages should
behave similarly. This also keeps network_time unaffected during
non pseudo-realtime trace processing.
This exposes Broker's new WebSocket support in Zeek. To enable it,
call `Broker::listen_websocket()`. Zeek will then start listening on
port 9997 for incoming WebSocket connections.
See the Broker documentation for a description of the message format
expected over these WebSocket connections.
Broker sends record values as a vector of the corresponding fields. If
the received values doesn't match the actual record fields, Zeek has
so far been reporting this not-so-helpful error message:
warning: failed to convert remote event 'got_result' arg #1, got vector, expected record
This change turns the latter part into a list of the field types we
actually received, which makes debugging much easier.
(No test, seems overkill to add a spawn a Broker client to exercise
this.)
1469562/1469558: Uninitialized fields in Func constructor
1469571/1469566: Null pointer dereference in Trigger::Init()
1469568: Uninitialized fields in CounterVector constructor
1469570: Uncaught exception in plugin manager
1469569: Resource leak in script_opt::Stmt
1469561/1469561: Uninitialized fields in ZBody constructor
1469559: Uninitialized fields in logging::Manager
1469563: Resource leak in ZAMCompiler::CompileDel
1469549/1469553/1469556: Context not fully initialized in HashVals
1469548: Remove dead code from IPAddr
1469551/1469554: Handle iosource_mgr registration failure in broker::Manager
1469552/1469572: Resource leaks in input::Manager
