Commit graph

17644 commits

Author SHA1 Message Date
Arne Welzel
4ee2f9256b Merge remote-tracking branch 'origin/topic/awelzel/pluggable-cluster-backends-part4'
* origin/topic/awelzel/pluggable-cluster-backends-part4:
  Update ZAM BiF-tracking
  configure: Add --disable-cluster-backend-zeromq
  CMakeLists: Cluster backends output
  zeromq: Conditionally enable by default
  btest/generic: Add publish_hrw(), publish_rr() and logging tests
  generate-docs: Run on Ubuntu 24.04, add cppzmq
  docker: Add cppzmq/libzmq dependencies
  tsan_suppressions: Add called_from_lib: libzmq
  ci: Add cppzmq and libzmq to most platforms
  cluster/backend/zeromq: Add ZeroMQ based cluster backend
  cluster/backend/zeromq: Add cppzmq submodule
  cluster: Add Cluster::node_id(), allow redef of node_topic(), nodeid_topic()
  cluster: Move publish_hrw() and publish_rr() to cluster.bif
2024-12-11 09:52:41 +01:00
zeek-bot
567d2f356b Update doc submodule [nomail] [skip ci] 2024-12-11 00:19:44 +00:00
Tim Wojtulewicz
0701895132 Merge remote-tracking branch 'J-Gras/topic/jgras/add-unknown-proto-count'
* J-Gras/topic/jgras/add-unknown-proto-count:
  Add btest for unknown_protocols.log
  Add protocol_id count to unknown protocol record
2024-12-10 16:16:12 -07:00
Tim Wojtulewicz
f61ba4df29 Update zeek-testing and zeek-testing-cluster commit hashes 2024-12-10 16:13:00 -07:00
Tim Wojtulewicz
666d319cfa Merge remote-tracking branch 'origin/topic/christian/news-tweaks'
* origin/topic/christian/news-tweaks:
  NEWS tweaks [skip ci]
2024-12-10 14:41:28 -07:00
Arne Welzel
759281dabd Update ZAM BiF-tracking 2024-12-10 20:35:41 +01:00
Arne Welzel
b93a4f5d2f configure: Add --disable-cluster-backend-zeromq 2024-12-10 20:33:02 +01:00
Arne Welzel
9e9d0ba7aa CMakeLists: Cluster backends output 2024-12-10 20:33:02 +01:00
Arne Welzel
07e23fb95e zeromq: Conditionally enable by default
Instead of having ZeroMQ as a new dependency, enable the ZeroMQ backend only
if ZeroMQ is available on the system as suggested by Tim.
2024-12-10 20:33:02 +01:00
Arne Welzel
d816bfb249 btest/generic: Add publish_hrw(), publish_rr() and logging tests
They currently use zeromq, but technically they should be valid for
any other backend, too, even broker.
2024-12-10 20:33:02 +01:00
Arne Welzel
df69ec1279 generate-docs: Run on Ubuntu 24.04, add cppzmq 2024-12-10 20:33:02 +01:00
Arne Welzel
d946be878e docker: Add cppzmq/libzmq dependencies 2024-12-10 20:33:02 +01:00
Arne Welzel
2f37dcf505 tsan_suppressions: Add called_from_lib: libzmq 2024-12-10 20:33:02 +01:00
Arne Welzel
34275afc1f ci: Add cppzmq and libzmq to most platforms 2024-12-10 20:33:02 +01:00
Arne Welzel
35c79ab2e3 cluster/backend/zeromq: Add ZeroMQ based cluster backend
This is a cluster backend implementation using a central XPUB/XSUB proxy
that by default runs on the manager node. Logging is implemented leveraging
PUSH/PULL sockets between logger and other nodes, rather than going
through XPUB/XSUB.

The test-all-policy-cluster baseline changed: Previously, Broker::peer()
would be called from setup-connections.zeek, causing the IO loop to be
alive. With the ZeroMQ backend, the IO loop is only alive when
Cluster::init() is called, but that doesn't happen anymore.
2024-12-10 20:33:02 +01:00
Arne Welzel
889c7d888a cluster/backend/zeromq: Add cppzmq submodule
Not all supported platforms provide a recent enough cppzmq version,
add a fallback as submodule. cppzmq is a header-only library, so there's
no build step involved.
2024-12-10 20:33:02 +01:00
Arne Welzel
a2249f7ecb cluster: Add Cluster::node_id(), allow redef of node_topic(), nodeid_topic()
This provides a way for non-broker cluster backends to override a
node's identifier and its own topics that it listens on by default.
2024-12-10 20:33:02 +01:00
Arne Welzel
210b54799e cluster: Move publish_hrw() and publish_rr() to cluster.bif
From this point on, Cluster::publish_hrw() and Cluster::publish_rr()
go through cluster/Backend.cc code.
2024-12-10 20:33:02 +01:00
Arne Welzel
79a71357c7 Merge remote-tracking branch 'origin/topic/awelzel/bif-tracking-fix'
* origin/topic/awelzel/bif-tracking-fix:
  Update BiF-tracking, add get_current_packet_ts()
2024-12-10 20:31:47 +01:00
Arne Welzel
7449b050b3 Update BiF-tracking, add get_current_packet_ts()
Also, run the ZAM-bif-tracking test in non-ZAM environments so
failures are caught immediately. There's nothing overly ZAM specific about
running this test.

I'm not sure I like the fact that any new contributor adding a BiF
will need to dig into this... it might be a bit intimidating.
2024-12-10 20:07:56 +01:00
Arne Welzel
77465a90b5 Merge remote-tracking branch 'jgras/topic/jgras/fix-packet-lag'
* jgras/topic/jgras/fix-packet-lag:
  Introduce get_packet_lag()
  Add btest for get_current_packet_ts()
  Introduce get_current_packet_ts to fix packet lag
2024-12-10 18:26:54 +01:00
Jan Grashoefer
1805afe5d9 Add btest for unknown_protocols.log 2024-12-10 14:54:19 +01:00
Jan Grashoefer
9228f0ff9f Add protocol_id count to unknown protocol record
The count representation is not logged and added for access to the value
in log policy hooks without converting the logged hex representation.
2024-12-10 14:49:02 +01:00
Arne Welzel
9034e135d1 Merge remote-tracking branch 'origin/topic/awelzel/fix-dns-binds-complte-typo'
* origin/topic/awelzel/fix-dns-binds-complte-typo:
  DNS/dns_binds_rr: Fix complte to complete typo, switch to count
2024-12-10 09:57:00 +01:00
Christian Kreibich
a32ea43664 NEWS tweaks [skip ci]
- Switch list items back to "-" from "*" -- we hadn't used "*" since
  2.5.4 but started in 7.1, probably by accident? :-)

- Fix a typo.
2024-12-09 23:59:58 -08:00
Christian Kreibich
1c42bfc715 Merge branch 'topic/christian/disconnect-slow-peers'
* topic/christian/disconnect-slow-peers:
  Bump cluster testsuite to pull in Broker backpressure tests
  Expand documentation of Broker events.
  Add sleep() BiF.
  Add backpressure disconnect notification to cluster.log and via telemetry
  Remove unneeded @loads from base/misc/version.zeek
  Add Cluster::nodeid_to_node() helper function
  Support re-peering with Broker peers that fall behind
  Add Zeek-level configurability of Broker slow-peer disconnects
  Bump Broker to pull in disconnect feature and infinite-loop fix
  No need to namespace Cluster:: functions in their own namespace
2024-12-09 23:33:35 -08:00
zeek-bot
09541d5ef2 Update doc submodule [nomail] [skip ci] 2024-12-10 00:24:53 +00:00
Tim Wojtulewicz
cb7cda9808 Merge remote-tracking branch 'origin/topic/awelzel/license-header-cleanup'
* origin/topic/awelzel/license-header-cleanup:
  Add missing copyright line to headers and cc files
  pre-commit: Add license-header check inspired by Spicy
  Add missing "COPYING" in file comments
2024-12-09 12:23:00 -07:00
Tim Wojtulewicz
6e75417032 Merge remote-tracking branch 'origin/topic/vern/CPP-streamlining'
* origin/topic/vern/CPP-streamlining:
  BTest baseline updates for compile-to-C++
  mark ZAM regression BTests as not suitable for compile-to-C++
  fix for -O gen-C++ maintenance helper to skip BTest intermediary files
  introduced simplified initialization for non-standalone -O gen-C++ code tied -O gen-standalone-C++ to use of --optimize-files
  streamline generated -O C++ code by relying on per-function profiles rather than aggregate profile
  when reporting available/unavailble C++ script bodies, flag those that are skipped
  modified AST profiling to mark (and fully skip) non-optimizable functions
  modified merge_types() to skip work if given identical types, which also preserves type names (useful for -O gen-C++)
2024-12-09 12:16:00 -07:00
Tim Wojtulewicz
832b9b6cf0 Merge remote-tracking branch 'origin/topic/timw/python-magic-numbers'
* origin/topic/timw/python-magic-numbers:
  Move python signatures to a separate file
  Add signatures for Python bytecode for 3.8-3.14
2024-12-09 12:09:20 -07:00
Jan Grashoefer
c2b17f9d30 Introduce get_packet_lag() 2024-12-09 19:19:29 +01:00
Tim Wojtulewicz
ccefd66d37 Move python signatures to a separate file 2024-12-09 11:08:30 -07:00
Arne Welzel
f6c44e3f7a DNS/dns_binds_rr: Fix complte to complete typo, switch to count
From my reading in the docs the complete_flag should only ever be a
single byte, so add a weird for when it is longer, but use count
as the new type.
2024-12-09 19:02:28 +01:00
Benjamin Bannier
e94fe45865 Merge remote-tracking branch 'origin/topic/bbannier/bump-pre-commit-hook' 2024-12-09 17:42:39 +01:00
Jan Grashoefer
6977c07a25 Add btest for get_current_packet_ts() 2024-12-09 16:39:38 +01:00
Jan Grashoefer
08c5a9c66d Introduce get_current_packet_ts to fix packet lag
Using network_time to calculate packet lag will produce wrong results
when there is no packet available but network time does not (yet) fall
back to wall clock.
2024-12-09 16:39:38 +01:00
Benjamin Bannier
29a49a59bd Bump pre-commit hooks 2024-12-09 16:14:25 +01:00
Benjamin Bannier
24894febb6 Bump typos pre-commit hooks
This now picks up additional typical misspellings, but also triggers on
more identifiers we use. I opted for fixing the obvious misspellings and
updated the allowlist for anything else.
2024-12-09 16:13:28 +01:00
Benjamin Bannier
97c0df29d4 Bump clang-format pre-commit hooks
This brings in slightly better formatting around uniform initialization,
and comments after blocks not surrounded by `{ .. }`.
2024-12-09 16:12:36 +01:00
Arne Welzel
0ebcd9608f Merge remote-tracking branch 'origin/topic/vern/script-opt-uncompilable-AST'
* origin/topic/vern/script-opt-uncompilable-AST:
  ScriptOpt: Fail compilation if known exprs/stmts is outdated
  skip optimization of functions with AST nodes unknown to script optimization
2024-12-09 14:04:01 +01:00
Arne Welzel
28f76d5da9 ScriptOpt: Fail compilation if known exprs/stmts is outdated 2024-12-09 11:03:20 +01:00
Arne Welzel
99989b8055 Disable CTU-SME test under TSAN 2024-12-09 09:26:42 +01:00
Christian Kreibich
83d16f9ef4 Bump cluster testsuite to pull in Broker backpressure tests 2024-12-07 00:36:45 -08:00
Christian Kreibich
feb2aa890d Expand documentation of Broker events. 2024-12-07 00:36:45 -08:00
Christian Kreibich
e6d0c8aa04 Add sleep() BiF.
Yes, really. :-) We've hit the need for this on occasion in very specific
settings and always worked around it via ugly nested loops or similars.
This has ample warning that folks normally won't want to use this.

Not sure that ZAM btest should baseline the number of BiFs.
2024-12-07 00:36:45 -08:00
Vern Paxson
96305aa4aa BTest baseline updates for compile-to-C++ 2024-12-06 16:30:44 -08:00
Vern Paxson
2047ae980a mark ZAM regression BTests as not suitable for compile-to-C++ 2024-12-06 16:28:06 -08:00
Vern Paxson
de98f1c36c fix for -O gen-C++ maintenance helper to skip BTest intermediary files 2024-12-06 16:26:48 -08:00
Vern Paxson
2e69a8870a introduced simplified initialization for non-standalone -O gen-C++ code
tied -O gen-standalone-C++ to use of --optimize-files
2024-12-06 16:25:22 -08:00
Vern Paxson
612d99e751 streamline generated -O C++ code by relying on per-function profiles rather than aggregate profile 2024-12-06 16:23:32 -08:00