Commit graph

17462 commits

Author SHA1 Message Date
Tim Wojtulewicz
9e19b51f41 Merge remote-tracking branch 'origin/topic/johanna/spicysslnews'
* origin/topic/johanna/spicysslnews:
  Add blurb about Spicy SSL to NEWS
2024-12-11 12:59:28 -07:00
Johanna Amann
3711af53d5 Add blurb about Spicy SSL to NEWS 2024-12-11 12:58:35 -07:00
Tim Wojtulewicz
9f1a30d3fe Merge remote-tracking branch 'origin/topic/timw/add-spicy-to-coverity'
* origin/topic/timw/add-spicy-to-coverity:
  Add Spicy analyzers to Coverity builds
2024-12-11 12:38:59 -07:00
Tim Wojtulewicz
b1d077cab9 Add Spicy analyzers to Coverity builds 2024-12-11 12:38:34 -07:00
Tim Wojtulewicz
649cedb693 Merge remote-tracking branch 'origin/topic/timw/python-upgrade'
* origin/topic/timw/python-upgrade: (65 commits)
  Swap pre-commit yapf for ruff/ruff-format, fix findings
  Upgrade package-manager submodule to get python upgrade
  Upgrade broker submodule to get python upgrade
  Upgrade zeekctl submodule to get python upgrade
  Upgrade zeek-client submodule to get python upgrade
  Upgrade btest submodule to get python upgrade
  Require Python 3.9 in CMakeLists.txt
  CI: Install python 3.9 on ubuntu 20
  btest: Avoid loading ZeroMQ if not compiled in
  btest/coverage: Avoid warnings in test-all-policy-cluster
  Broker::publish: Warn on using Broker::publish() when inactive
  Update doc submodule [nomail] [skip ci]
  Update zeek-testing and zeek-testing-cluster commit hashes
  Update ZAM BiF-tracking
  configure: Add --disable-cluster-backend-zeromq
  CMakeLists: Cluster backends output
  zeromq: Conditionally enable by default
  btest/generic: Add publish_hrw(), publish_rr() and logging tests
  generate-docs: Run on Ubuntu 24.04, add cppzmq
  docker: Add cppzmq/libzmq dependencies
  ...
2024-12-11 11:46:02 -07:00
Tim Wojtulewicz
49f82b325b Swap pre-commit yapf for ruff/ruff-format, fix findings 2024-12-11 11:08:37 -07:00
Tim Wojtulewicz
b02f812e26 Upgrade package-manager submodule to get python upgrade 2024-12-11 11:08:37 -07:00
Tim Wojtulewicz
e65e92edcf Upgrade broker submodule to get python upgrade 2024-12-11 11:08:37 -07:00
Tim Wojtulewicz
c98ed621d6 Upgrade zeekctl submodule to get python upgrade 2024-12-11 11:08:37 -07:00
Tim Wojtulewicz
38fb76d52d Upgrade zeek-client submodule to get python upgrade 2024-12-11 11:08:37 -07:00
Tim Wojtulewicz
641306b912 Upgrade btest submodule to get python upgrade 2024-12-11 11:08:37 -07:00
Tim Wojtulewicz
566fa7c6d0 Require Python 3.9 in CMakeLists.txt 2024-12-11 11:08:37 -07:00
Tim Wojtulewicz
beb7b5bbbf CI: Install python 3.9 on ubuntu 20 2024-12-11 11:08:37 -07:00
Tim Wojtulewicz
7f3bea7ce1 Merge remote-tracking branch 'origin/topic/awelzel/broker-publish-warn'
* origin/topic/awelzel/broker-publish-warn:
  btest: Avoid loading ZeroMQ if not compiled in
  btest/coverage: Avoid warnings in test-all-policy-cluster
  Broker::publish: Warn on using Broker::publish() when inactive
2024-12-11 11:08:11 -07:00
Arne Welzel
85189ca918 btest: Avoid loading ZeroMQ if not compiled in
...at the same time, add some `TEST-REQUIRES: have-zeromq` which
unfortunately means that developers will usually want libzmq
installed on their system.
2024-12-11 18:08:43 +01:00
Arne Welzel
c6c6d88b4c btest/coverage: Avoid warnings in test-all-policy-cluster 2024-12-11 17:20:45 +01:00
Arne Welzel
0ad3210177 Broker::publish: Warn on using Broker::publish() when inactive
This is mostly for transitioning base scripts to Cluster::publish() and
avoid silent surprises why certain things don't work when using ZeroMQ.
2024-12-11 17:20:42 +01:00
Arne Welzel
f674fc2634 Merge remote-tracking branch 'origin/topic/timw/update-zeek-testing'
* origin/topic/timw/update-zeek-testing:
  Update zeek-testing and zeek-testing-cluster commit hashes
2024-12-11 13:19:33 +01:00
Arne Welzel
4ee2f9256b Merge remote-tracking branch 'origin/topic/awelzel/pluggable-cluster-backends-part4'
* origin/topic/awelzel/pluggable-cluster-backends-part4:
  Update ZAM BiF-tracking
  configure: Add --disable-cluster-backend-zeromq
  CMakeLists: Cluster backends output
  zeromq: Conditionally enable by default
  btest/generic: Add publish_hrw(), publish_rr() and logging tests
  generate-docs: Run on Ubuntu 24.04, add cppzmq
  docker: Add cppzmq/libzmq dependencies
  tsan_suppressions: Add called_from_lib: libzmq
  ci: Add cppzmq and libzmq to most platforms
  cluster/backend/zeromq: Add ZeroMQ based cluster backend
  cluster/backend/zeromq: Add cppzmq submodule
  cluster: Add Cluster::node_id(), allow redef of node_topic(), nodeid_topic()
  cluster: Move publish_hrw() and publish_rr() to cluster.bif
2024-12-11 09:52:41 +01:00
zeek-bot
567d2f356b Update doc submodule [nomail] [skip ci] 2024-12-11 00:19:44 +00:00
Tim Wojtulewicz
0701895132 Merge remote-tracking branch 'J-Gras/topic/jgras/add-unknown-proto-count'
* J-Gras/topic/jgras/add-unknown-proto-count:
  Add btest for unknown_protocols.log
  Add protocol_id count to unknown protocol record
2024-12-10 16:16:12 -07:00
Tim Wojtulewicz
f61ba4df29 Update zeek-testing and zeek-testing-cluster commit hashes 2024-12-10 16:13:00 -07:00
Tim Wojtulewicz
666d319cfa Merge remote-tracking branch 'origin/topic/christian/news-tweaks'
* origin/topic/christian/news-tweaks:
  NEWS tweaks [skip ci]
2024-12-10 14:41:28 -07:00
Arne Welzel
759281dabd Update ZAM BiF-tracking 2024-12-10 20:35:41 +01:00
Arne Welzel
b93a4f5d2f configure: Add --disable-cluster-backend-zeromq 2024-12-10 20:33:02 +01:00
Arne Welzel
9e9d0ba7aa CMakeLists: Cluster backends output 2024-12-10 20:33:02 +01:00
Arne Welzel
07e23fb95e zeromq: Conditionally enable by default
Instead of having ZeroMQ as a new dependency, enable the ZeroMQ backend only
if ZeroMQ is available on the system as suggested by Tim.
2024-12-10 20:33:02 +01:00
Arne Welzel
d816bfb249 btest/generic: Add publish_hrw(), publish_rr() and logging tests
They currently use zeromq, but technically they should be valid for
any other backend, too, even broker.
2024-12-10 20:33:02 +01:00
Arne Welzel
df69ec1279 generate-docs: Run on Ubuntu 24.04, add cppzmq 2024-12-10 20:33:02 +01:00
Arne Welzel
d946be878e docker: Add cppzmq/libzmq dependencies 2024-12-10 20:33:02 +01:00
Arne Welzel
2f37dcf505 tsan_suppressions: Add called_from_lib: libzmq 2024-12-10 20:33:02 +01:00
Arne Welzel
34275afc1f ci: Add cppzmq and libzmq to most platforms 2024-12-10 20:33:02 +01:00
Arne Welzel
35c79ab2e3 cluster/backend/zeromq: Add ZeroMQ based cluster backend
This is a cluster backend implementation using a central XPUB/XSUB proxy
that by default runs on the manager node. Logging is implemented leveraging
PUSH/PULL sockets between logger and other nodes, rather than going
through XPUB/XSUB.

The test-all-policy-cluster baseline changed: Previously, Broker::peer()
would be called from setup-connections.zeek, causing the IO loop to be
alive. With the ZeroMQ backend, the IO loop is only alive when
Cluster::init() is called, but that doesn't happen anymore.
2024-12-10 20:33:02 +01:00
Arne Welzel
889c7d888a cluster/backend/zeromq: Add cppzmq submodule
Not all supported platforms provide a recent enough cppzmq version,
add a fallback as submodule. cppzmq is a header-only library, so there's
no build step involved.
2024-12-10 20:33:02 +01:00
Arne Welzel
a2249f7ecb cluster: Add Cluster::node_id(), allow redef of node_topic(), nodeid_topic()
This provides a way for non-broker cluster backends to override a
node's identifier and its own topics that it listens on by default.
2024-12-10 20:33:02 +01:00
Arne Welzel
210b54799e cluster: Move publish_hrw() and publish_rr() to cluster.bif
From this point on, Cluster::publish_hrw() and Cluster::publish_rr()
go through cluster/Backend.cc code.
2024-12-10 20:33:02 +01:00
Arne Welzel
79a71357c7 Merge remote-tracking branch 'origin/topic/awelzel/bif-tracking-fix'
* origin/topic/awelzel/bif-tracking-fix:
  Update BiF-tracking, add get_current_packet_ts()
2024-12-10 20:31:47 +01:00
Arne Welzel
7449b050b3 Update BiF-tracking, add get_current_packet_ts()
Also, run the ZAM-bif-tracking test in non-ZAM environments so
failures are caught immediately. There's nothing overly ZAM specific about
running this test.

I'm not sure I like the fact that any new contributor adding a BiF
will need to dig into this... it might be a bit intimidating.
2024-12-10 20:07:56 +01:00
Arne Welzel
77465a90b5 Merge remote-tracking branch 'jgras/topic/jgras/fix-packet-lag'
* jgras/topic/jgras/fix-packet-lag:
  Introduce get_packet_lag()
  Add btest for get_current_packet_ts()
  Introduce get_current_packet_ts to fix packet lag
2024-12-10 18:26:54 +01:00
Jan Grashoefer
1805afe5d9 Add btest for unknown_protocols.log 2024-12-10 14:54:19 +01:00
Jan Grashoefer
9228f0ff9f Add protocol_id count to unknown protocol record
The count representation is not logged and added for access to the value
in log policy hooks without converting the logged hex representation.
2024-12-10 14:49:02 +01:00
Arne Welzel
9034e135d1 Merge remote-tracking branch 'origin/topic/awelzel/fix-dns-binds-complte-typo'
* origin/topic/awelzel/fix-dns-binds-complte-typo:
  DNS/dns_binds_rr: Fix complte to complete typo, switch to count
2024-12-10 09:57:00 +01:00
Christian Kreibich
a32ea43664 NEWS tweaks [skip ci]
- Switch list items back to "-" from "*" -- we hadn't used "*" since
  2.5.4 but started in 7.1, probably by accident? :-)

- Fix a typo.
2024-12-09 23:59:58 -08:00
Christian Kreibich
1c42bfc715 Merge branch 'topic/christian/disconnect-slow-peers'
* topic/christian/disconnect-slow-peers:
  Bump cluster testsuite to pull in Broker backpressure tests
  Expand documentation of Broker events.
  Add sleep() BiF.
  Add backpressure disconnect notification to cluster.log and via telemetry
  Remove unneeded @loads from base/misc/version.zeek
  Add Cluster::nodeid_to_node() helper function
  Support re-peering with Broker peers that fall behind
  Add Zeek-level configurability of Broker slow-peer disconnects
  Bump Broker to pull in disconnect feature and infinite-loop fix
  No need to namespace Cluster:: functions in their own namespace
2024-12-09 23:33:35 -08:00
zeek-bot
09541d5ef2 Update doc submodule [nomail] [skip ci] 2024-12-10 00:24:53 +00:00
Tim Wojtulewicz
cb7cda9808 Merge remote-tracking branch 'origin/topic/awelzel/license-header-cleanup'
* origin/topic/awelzel/license-header-cleanup:
  Add missing copyright line to headers and cc files
  pre-commit: Add license-header check inspired by Spicy
  Add missing "COPYING" in file comments
2024-12-09 12:23:00 -07:00
Tim Wojtulewicz
6e75417032 Merge remote-tracking branch 'origin/topic/vern/CPP-streamlining'
* origin/topic/vern/CPP-streamlining:
  BTest baseline updates for compile-to-C++
  mark ZAM regression BTests as not suitable for compile-to-C++
  fix for -O gen-C++ maintenance helper to skip BTest intermediary files
  introduced simplified initialization for non-standalone -O gen-C++ code tied -O gen-standalone-C++ to use of --optimize-files
  streamline generated -O C++ code by relying on per-function profiles rather than aggregate profile
  when reporting available/unavailble C++ script bodies, flag those that are skipped
  modified AST profiling to mark (and fully skip) non-optimizable functions
  modified merge_types() to skip work if given identical types, which also preserves type names (useful for -O gen-C++)
2024-12-09 12:16:00 -07:00
Tim Wojtulewicz
832b9b6cf0 Merge remote-tracking branch 'origin/topic/timw/python-magic-numbers'
* origin/topic/timw/python-magic-numbers:
  Move python signatures to a separate file
  Add signatures for Python bytecode for 3.8-3.14
2024-12-09 12:09:20 -07:00
Jan Grashoefer
c2b17f9d30 Introduce get_packet_lag() 2024-12-09 19:19:29 +01:00
Tim Wojtulewicz
ccefd66d37 Move python signatures to a separate file 2024-12-09 11:08:30 -07:00