Commit graph

10 commits

Author SHA1 Message Date
Tim Wojtulewicz
1169fcf2a2 Move byte_buffer types from cluster and storage into util 2025-04-14 10:11:13 -07:00
Arne Welzel
bfffc8dac8 cluster/zeromq: Improve XPUB stall behavior, add a metric
Instead of fprintf, track the number of occurrences via a metric and
change the sleep loop to a blocking send instead.
2025-03-26 14:23:09 +01:00
Arne Welzel
aad512c616 cluster/zeromq: Support configuring IO threads for proxy thread 2025-03-10 17:07:30 +01:00
Arne Welzel
ba7b605a97 cluster/zeromq: Move variable lookups from DoInit() to DoInitPostScript() 2025-03-10 17:07:30 +01:00
Arne Welzel
827eccb732 cluster/zeromq: Adapt for OnLoopProcess changes 2025-03-10 17:07:30 +01:00
Arne Welzel
6008e67008 cluster/zeromq: Call DoTerminate() in destructor
Normal life-cycle is that Terminate() / DoTerminate() is called
by zeek-setup code. If that doesn't happen, shutdown and join
threads during destructor.

try { } catch (...) suggested by Benjamin.
2025-02-05 16:39:44 +01:00
Arne Welzel
e8f87019c6 cluster: Add SubscribeCallback support
This allows callers of Subscribe() to pass in a callback that will be invoked
once the subscription is established or failed to establish. It is the
backend's responsibility to execute the callback on the main thread either
synchronously, or preferably asynchronously at a later point, by
scheduling a task on the IO main loop.

This turns on ZMQ_XPUB_VERBOSE for ZeroMQ so that notifications about
subscriptions are raised even if the subscriptions has previously been
observed.
2025-02-05 10:39:56 +01:00
Arne Welzel
df78a94c76 cluster/zeromq: Use NodeId(), drop my_node_id 2025-02-05 10:39:56 +01:00
Arne Welzel
0b7a660a34 cluster/Backend: Make backend event processing customizable
This allows configurability at the code level to decide what to do with
a received remote events and events produced by a backend. For now, only
enqueue events into the process's script layer, but for the WebSocket
interface, the action would be to send out the event on a WebSocket
connection instead.
2025-02-05 10:39:56 +01:00
Arne Welzel
35c79ab2e3 cluster/backend/zeromq: Add ZeroMQ based cluster backend
This is a cluster backend implementation using a central XPUB/XSUB proxy
that by default runs on the manager node. Logging is implemented leveraging
PUSH/PULL sockets between logger and other nodes, rather than going
through XPUB/XSUB.

The test-all-policy-cluster baseline changed: Previously, Broker::peer()
would be called from setup-connections.zeek, causing the IO loop to be
alive. With the ZeroMQ backend, the IO loop is only alive when
Cluster::init() is called, but that doesn't happen anymore.
2024-12-10 20:33:02 +01:00