Same as what we do in Broker. Use the expected type if publishing
a table() or set() parameter.
This fixes issues when switching sumstats to Cluster::publish()
The broker serializer leverages the existing data_to_val() function.
During unserialization, if the destination type is any, the logic
simply wraps the broker::data value into a Broker::Data record.
Therefore, events with any parameters are currently exposed to
the Broker::Data type.
There is a bigger issue in that re-publishing such Broker::Data
instances would encode them as a normal record. Explicitly prevent
this by serializing the contained data value directly instead, similar
to what Broker already did when publishing a record.
Mostly due to spending too much time wondering why nodes didn't connect
when there was a mismatch between "manager" and "manager-1" in the
cluster layout. Remove manager from test-all-policy-cluster test to
avoid connection attempts in this test.
* origin/topic/timw/python-upgrade: (65 commits)
Swap pre-commit yapf for ruff/ruff-format, fix findings
Upgrade package-manager submodule to get python upgrade
Upgrade broker submodule to get python upgrade
Upgrade zeekctl submodule to get python upgrade
Upgrade zeek-client submodule to get python upgrade
Upgrade btest submodule to get python upgrade
Require Python 3.9 in CMakeLists.txt
CI: Install python 3.9 on ubuntu 20
btest: Avoid loading ZeroMQ if not compiled in
btest/coverage: Avoid warnings in test-all-policy-cluster
Broker::publish: Warn on using Broker::publish() when inactive
Update doc submodule [nomail] [skip ci]
Update zeek-testing and zeek-testing-cluster commit hashes
Update ZAM BiF-tracking
configure: Add --disable-cluster-backend-zeromq
CMakeLists: Cluster backends output
zeromq: Conditionally enable by default
btest/generic: Add publish_hrw(), publish_rr() and logging tests
generate-docs: Run on Ubuntu 24.04, add cppzmq
docker: Add cppzmq/libzmq dependencies
...
* origin/topic/awelzel/broker-publish-warn:
btest: Avoid loading ZeroMQ if not compiled in
btest/coverage: Avoid warnings in test-all-policy-cluster
Broker::publish: Warn on using Broker::publish() when inactive
...at the same time, add some `TEST-REQUIRES: have-zeromq` which
unfortunately means that developers will usually want libzmq
installed on their system.
This is a cluster backend implementation using a central XPUB/XSUB proxy
that by default runs on the manager node. Logging is implemented leveraging
PUSH/PULL sockets between logger and other nodes, rather than going
through XPUB/XSUB.
The test-all-policy-cluster baseline changed: Previously, Broker::peer()
would be called from setup-connections.zeek, causing the IO loop to be
alive. With the ZeroMQ backend, the IO loop is only alive when
Cluster::init() is called, but that doesn't happen anymore.
Not all supported platforms provide a recent enough cppzmq version,
add a fallback as submodule. cppzmq is a header-only library, so there's
no build step involved.
Also, run the ZAM-bif-tracking test in non-ZAM environments so
failures are caught immediately. There's nothing overly ZAM specific about
running this test.
I'm not sure I like the fact that any new contributor adding a BiF
will need to dig into this... it might be a bit intimidating.
