Commit graph

17426 commits

Author SHA1 Message Date
Arne Welzel
889c7d888a cluster/backend/zeromq: Add cppzmq submodule
Not all supported platforms provide a recent enough cppzmq version,
add a fallback as submodule. cppzmq is a header-only library, so there's
no build step involved.
2024-12-10 20:33:02 +01:00
Arne Welzel
a2249f7ecb cluster: Add Cluster::node_id(), allow redef of node_topic(), nodeid_topic()
This provides a way for non-broker cluster backends to override a
node's identifier and its own topics that it listens on by default.
2024-12-10 20:33:02 +01:00
Arne Welzel
210b54799e cluster: Move publish_hrw() and publish_rr() to cluster.bif
From this point on, Cluster::publish_hrw() and Cluster::publish_rr()
go through cluster/Backend.cc code.
2024-12-10 20:33:02 +01:00
Arne Welzel
79a71357c7 Merge remote-tracking branch 'origin/topic/awelzel/bif-tracking-fix'
* origin/topic/awelzel/bif-tracking-fix:
  Update BiF-tracking, add get_current_packet_ts()
2024-12-10 20:31:47 +01:00
Arne Welzel
7449b050b3 Update BiF-tracking, add get_current_packet_ts()
Also, run the ZAM-bif-tracking test in non-ZAM environments so
failures are caught immediately. There's nothing overly ZAM specific about
running this test.

I'm not sure I like the fact that any new contributor adding a BiF
will need to dig into this... it might be a bit intimidating.
2024-12-10 20:07:56 +01:00
Arne Welzel
77465a90b5 Merge remote-tracking branch 'jgras/topic/jgras/fix-packet-lag'
* jgras/topic/jgras/fix-packet-lag:
  Introduce get_packet_lag()
  Add btest for get_current_packet_ts()
  Introduce get_current_packet_ts to fix packet lag
2024-12-10 18:26:54 +01:00
Arne Welzel
9034e135d1 Merge remote-tracking branch 'origin/topic/awelzel/fix-dns-binds-complte-typo'
* origin/topic/awelzel/fix-dns-binds-complte-typo:
  DNS/dns_binds_rr: Fix complte to complete typo, switch to count
2024-12-10 09:57:00 +01:00
Christian Kreibich
1c42bfc715 Merge branch 'topic/christian/disconnect-slow-peers'
* topic/christian/disconnect-slow-peers:
  Bump cluster testsuite to pull in Broker backpressure tests
  Expand documentation of Broker events.
  Add sleep() BiF.
  Add backpressure disconnect notification to cluster.log and via telemetry
  Remove unneeded @loads from base/misc/version.zeek
  Add Cluster::nodeid_to_node() helper function
  Support re-peering with Broker peers that fall behind
  Add Zeek-level configurability of Broker slow-peer disconnects
  Bump Broker to pull in disconnect feature and infinite-loop fix
  No need to namespace Cluster:: functions in their own namespace
2024-12-09 23:33:35 -08:00
zeek-bot
09541d5ef2 Update doc submodule [nomail] [skip ci] 2024-12-10 00:24:53 +00:00
Tim Wojtulewicz
cb7cda9808 Merge remote-tracking branch 'origin/topic/awelzel/license-header-cleanup'
* origin/topic/awelzel/license-header-cleanup:
  Add missing copyright line to headers and cc files
  pre-commit: Add license-header check inspired by Spicy
  Add missing "COPYING" in file comments
2024-12-09 12:23:00 -07:00
Tim Wojtulewicz
6e75417032 Merge remote-tracking branch 'origin/topic/vern/CPP-streamlining'
* origin/topic/vern/CPP-streamlining:
  BTest baseline updates for compile-to-C++
  mark ZAM regression BTests as not suitable for compile-to-C++
  fix for -O gen-C++ maintenance helper to skip BTest intermediary files
  introduced simplified initialization for non-standalone -O gen-C++ code tied -O gen-standalone-C++ to use of --optimize-files
  streamline generated -O C++ code by relying on per-function profiles rather than aggregate profile
  when reporting available/unavailble C++ script bodies, flag those that are skipped
  modified AST profiling to mark (and fully skip) non-optimizable functions
  modified merge_types() to skip work if given identical types, which also preserves type names (useful for -O gen-C++)
2024-12-09 12:16:00 -07:00
Tim Wojtulewicz
832b9b6cf0 Merge remote-tracking branch 'origin/topic/timw/python-magic-numbers'
* origin/topic/timw/python-magic-numbers:
  Move python signatures to a separate file
  Add signatures for Python bytecode for 3.8-3.14
2024-12-09 12:09:20 -07:00
Jan Grashoefer
c2b17f9d30 Introduce get_packet_lag() 2024-12-09 19:19:29 +01:00
Tim Wojtulewicz
ccefd66d37 Move python signatures to a separate file 2024-12-09 11:08:30 -07:00
Arne Welzel
f6c44e3f7a DNS/dns_binds_rr: Fix complte to complete typo, switch to count
From my reading in the docs the complete_flag should only ever be a
single byte, so add a weird for when it is longer, but use count
as the new type.
2024-12-09 19:02:28 +01:00
Benjamin Bannier
e94fe45865 Merge remote-tracking branch 'origin/topic/bbannier/bump-pre-commit-hook' 2024-12-09 17:42:39 +01:00
Jan Grashoefer
6977c07a25 Add btest for get_current_packet_ts() 2024-12-09 16:39:38 +01:00
Jan Grashoefer
08c5a9c66d Introduce get_current_packet_ts to fix packet lag
Using network_time to calculate packet lag will produce wrong results
when there is no packet available but network time does not (yet) fall
back to wall clock.
2024-12-09 16:39:38 +01:00
Benjamin Bannier
29a49a59bd Bump pre-commit hooks 2024-12-09 16:14:25 +01:00
Benjamin Bannier
24894febb6 Bump typos pre-commit hooks
This now picks up additional typical misspellings, but also triggers on
more identifiers we use. I opted for fixing the obvious misspellings and
updated the allowlist for anything else.
2024-12-09 16:13:28 +01:00
Benjamin Bannier
97c0df29d4 Bump clang-format pre-commit hooks
This brings in slightly better formatting around uniform initialization,
and comments after blocks not surrounded by `{ .. }`.
2024-12-09 16:12:36 +01:00
Arne Welzel
0ebcd9608f Merge remote-tracking branch 'origin/topic/vern/script-opt-uncompilable-AST'
* origin/topic/vern/script-opt-uncompilable-AST:
  ScriptOpt: Fail compilation if known exprs/stmts is outdated
  skip optimization of functions with AST nodes unknown to script optimization
2024-12-09 14:04:01 +01:00
Arne Welzel
28f76d5da9 ScriptOpt: Fail compilation if known exprs/stmts is outdated 2024-12-09 11:03:20 +01:00
Arne Welzel
99989b8055 Disable CTU-SME test under TSAN 2024-12-09 09:26:42 +01:00
Christian Kreibich
83d16f9ef4 Bump cluster testsuite to pull in Broker backpressure tests 2024-12-07 00:36:45 -08:00
Christian Kreibich
feb2aa890d Expand documentation of Broker events. 2024-12-07 00:36:45 -08:00
Christian Kreibich
e6d0c8aa04 Add sleep() BiF.
Yes, really. :-) We've hit the need for this on occasion in very specific
settings and always worked around it via ugly nested loops or similars.
This has ample warning that folks normally won't want to use this.

Not sure that ZAM btest should baseline the number of BiFs.
2024-12-07 00:36:45 -08:00
Vern Paxson
96305aa4aa BTest baseline updates for compile-to-C++ 2024-12-06 16:30:44 -08:00
Vern Paxson
2047ae980a mark ZAM regression BTests as not suitable for compile-to-C++ 2024-12-06 16:28:06 -08:00
Vern Paxson
de98f1c36c fix for -O gen-C++ maintenance helper to skip BTest intermediary files 2024-12-06 16:26:48 -08:00
Vern Paxson
2e69a8870a introduced simplified initialization for non-standalone -O gen-C++ code
tied -O gen-standalone-C++ to use of --optimize-files
2024-12-06 16:25:22 -08:00
Vern Paxson
612d99e751 streamline generated -O C++ code by relying on per-function profiles rather than aggregate profile 2024-12-06 16:23:32 -08:00
Vern Paxson
79c5790bbf when reporting available/unavailble C++ script bodies, flag those that are skipped 2024-12-06 16:18:43 -08:00
Vern Paxson
62f891dcba modified AST profiling to mark (and fully skip) non-optimizable functions 2024-12-06 16:17:44 -08:00
Vern Paxson
7ed3f79c87 modified merge_types() to skip work if given identical types, which
also preserves type names (useful for -O gen-C++)
2024-12-06 16:16:35 -08:00
zeek-bot
db79c88fb8 Update doc submodule [nomail] [skip ci] 2024-12-07 00:12:05 +00:00
Christian Kreibich
ead6134501 Add backpressure disconnect notification to cluster.log and via telemetry
This adds a Broker-specific script to the cluster framework, loaded only when
Zeek is running in cluster mode. It adds logging in cluster.log as well as
telemetry via a metrics counter for Broker-observed backpressure disconnects.

The new zeek_broker_backpressure_disconnects counter, labeled by the neighboring
peer that the reporting node has determined to be unresponsive, counts the
number of unpeerings for this reason.

Here the node "worker" has observed node "proxy" falling behind once:

# HELP zeek_broker_backpressure_disconnects_total Number of Broker peering drops due to a neighbor falling too far behind in message I/O
# TYPE zeek_broker_backpressure_disconnects_total counter
zeek_broker_backpressure_disconnects_total{endpoint="worker",peer="proxy"} 1

Includes small btest baseline update to reflect @load of a new script.
2024-12-06 15:18:05 -08:00
Christian Kreibich
d260a5b7a9 Remove unneeded @loads from base/misc/version.zeek
This module is loaded by the telemetry framework, which we're now loading via
the cluster framework, i.e. also in bare mode. The resulting additional
thread (for creating reporter.log) trips up a number of btest baselines.

version.zeek doesn't use any of the string helper functions.
2024-12-06 15:18:05 -08:00
Christian Kreibich
46a11ec37d Add Cluster::nodeid_to_node() helper function
This translates backend-specific node identifiers (like Broker IDs) to
cluster nodes and their names, if available.
2024-12-06 15:18:05 -08:00
Christian Kreibich
0010e65f6d Support re-peering with Broker peers that fall behind
This adds re-peering at the Broker level for peers that Broker decided to
unpeer. We keep this at the Broker level since this behavior is specific to
it (as opposed to other cluster backends).

Includes baseline updates for btests that pick up on the new script's @load.
2024-12-06 15:18:05 -08:00
Dominik Charousset
4c4eb4b8e2 Add Zeek-level configurability of Broker slow-peer disconnects 2024-12-06 15:18:05 -08:00
Christian Kreibich
b9df1674b7 Bump Broker to pull in disconnect feature and infinite-loop fix 2024-12-06 15:18:05 -08:00
Christian Kreibich
e81856a4af No need to namespace Cluster:: functions in their own namespace 2024-12-06 15:18:05 -08:00
Tim Wojtulewicz
bbd7f56dcc Add signatures for Python bytecode for 3.8-3.14 2024-12-06 13:45:46 -07:00
Arne Welzel
27517ee4cb Merge remote-tracking branch 'origin/topic/awelzel/4068-http-upgrade-content-type'
* origin/topic/awelzel/4068-http-upgrade-content-type:
  btest/http: Demo StreamEvent analyzer with HTTP::upgrade_analyzers
  protocol: Add StreamEvent analyzer
2024-12-06 17:33:52 +01:00
Arne Welzel
079ae460a7 btest/http: Demo StreamEvent analyzer with HTTP::upgrade_analyzers
Relates to #4068
2024-12-06 16:12:40 +01:00
Arne Welzel
51836d08ae protocol: Add StreamEvent analyzer
This analyzer can be used to transport raw stream data for a given
connection to the script layer. For example, adding this analyzer into
the HTTP::upgrade_analyzer or using it to configure a child WebSocket
analyzer allows to get access to the raw stream data in script land
when no more appropriate protocol analyzer is available.
2024-12-06 16:12:40 +01:00
Arne Welzel
9619cd0f17 Add missing copyright line to headers and cc files 2024-12-06 12:50:58 +01:00
Arne Welzel
d93249eeab pre-commit: Add license-header check inspired by Spicy 2024-12-06 12:50:58 +01:00
Arne Welzel
4958c56c84 Add missing "COPYING" in file comments
This was just done via sed. There's a number of files that don't
have a license entry at all.
2024-12-06 11:55:46 +01:00