Commit graph

17615 commits

Author SHA1 Message Date
Tim Wojtulewicz
832b9b6cf0 Merge remote-tracking branch 'origin/topic/timw/python-magic-numbers'
* origin/topic/timw/python-magic-numbers:
  Move python signatures to a separate file
  Add signatures for Python bytecode for 3.8-3.14
2024-12-09 12:09:20 -07:00
Jan Grashoefer
c2b17f9d30 Introduce get_packet_lag() 2024-12-09 19:19:29 +01:00
Tim Wojtulewicz
ccefd66d37 Move python signatures to a separate file 2024-12-09 11:08:30 -07:00
Arne Welzel
f6c44e3f7a DNS/dns_binds_rr: Fix complte to complete typo, switch to count
From my reading in the docs the complete_flag should only ever be a
single byte, so add a weird for when it is longer, but use count
as the new type.
2024-12-09 19:02:28 +01:00
Benjamin Bannier
e94fe45865 Merge remote-tracking branch 'origin/topic/bbannier/bump-pre-commit-hook' 2024-12-09 17:42:39 +01:00
Jan Grashoefer
6977c07a25 Add btest for get_current_packet_ts() 2024-12-09 16:39:38 +01:00
Jan Grashoefer
08c5a9c66d Introduce get_current_packet_ts to fix packet lag
Using network_time to calculate packet lag will produce wrong results
when there is no packet available but network time does not (yet) fall
back to wall clock.
2024-12-09 16:39:38 +01:00
Benjamin Bannier
29a49a59bd Bump pre-commit hooks 2024-12-09 16:14:25 +01:00
Benjamin Bannier
24894febb6 Bump typos pre-commit hooks
This now picks up additional typical misspellings, but also triggers on
more identifiers we use. I opted for fixing the obvious misspellings and
updated the allowlist for anything else.
2024-12-09 16:13:28 +01:00
Benjamin Bannier
97c0df29d4 Bump clang-format pre-commit hooks
This brings in slightly better formatting around uniform initialization,
and comments after blocks not surrounded by `{ .. }`.
2024-12-09 16:12:36 +01:00
Arne Welzel
0ebcd9608f Merge remote-tracking branch 'origin/topic/vern/script-opt-uncompilable-AST'
* origin/topic/vern/script-opt-uncompilable-AST:
  ScriptOpt: Fail compilation if known exprs/stmts is outdated
  skip optimization of functions with AST nodes unknown to script optimization
2024-12-09 14:04:01 +01:00
Arne Welzel
28f76d5da9 ScriptOpt: Fail compilation if known exprs/stmts is outdated 2024-12-09 11:03:20 +01:00
Arne Welzel
99989b8055 Disable CTU-SME test under TSAN 2024-12-09 09:26:42 +01:00
Christian Kreibich
83d16f9ef4 Bump cluster testsuite to pull in Broker backpressure tests 2024-12-07 00:36:45 -08:00
Christian Kreibich
feb2aa890d Expand documentation of Broker events. 2024-12-07 00:36:45 -08:00
Christian Kreibich
e6d0c8aa04 Add sleep() BiF.
Yes, really. :-) We've hit the need for this on occasion in very specific
settings and always worked around it via ugly nested loops or similars.
This has ample warning that folks normally won't want to use this.

Not sure that ZAM btest should baseline the number of BiFs.
2024-12-07 00:36:45 -08:00
Vern Paxson
96305aa4aa BTest baseline updates for compile-to-C++ 2024-12-06 16:30:44 -08:00
Vern Paxson
2047ae980a mark ZAM regression BTests as not suitable for compile-to-C++ 2024-12-06 16:28:06 -08:00
Vern Paxson
de98f1c36c fix for -O gen-C++ maintenance helper to skip BTest intermediary files 2024-12-06 16:26:48 -08:00
Vern Paxson
2e69a8870a introduced simplified initialization for non-standalone -O gen-C++ code
tied -O gen-standalone-C++ to use of --optimize-files
2024-12-06 16:25:22 -08:00
Vern Paxson
612d99e751 streamline generated -O C++ code by relying on per-function profiles rather than aggregate profile 2024-12-06 16:23:32 -08:00
Vern Paxson
79c5790bbf when reporting available/unavailble C++ script bodies, flag those that are skipped 2024-12-06 16:18:43 -08:00
Vern Paxson
62f891dcba modified AST profiling to mark (and fully skip) non-optimizable functions 2024-12-06 16:17:44 -08:00
Vern Paxson
7ed3f79c87 modified merge_types() to skip work if given identical types, which
also preserves type names (useful for -O gen-C++)
2024-12-06 16:16:35 -08:00
zeek-bot
db79c88fb8 Update doc submodule [nomail] [skip ci] 2024-12-07 00:12:05 +00:00
Christian Kreibich
ead6134501 Add backpressure disconnect notification to cluster.log and via telemetry
This adds a Broker-specific script to the cluster framework, loaded only when
Zeek is running in cluster mode. It adds logging in cluster.log as well as
telemetry via a metrics counter for Broker-observed backpressure disconnects.

The new zeek_broker_backpressure_disconnects counter, labeled by the neighboring
peer that the reporting node has determined to be unresponsive, counts the
number of unpeerings for this reason.

Here the node "worker" has observed node "proxy" falling behind once:

# HELP zeek_broker_backpressure_disconnects_total Number of Broker peering drops due to a neighbor falling too far behind in message I/O
# TYPE zeek_broker_backpressure_disconnects_total counter
zeek_broker_backpressure_disconnects_total{endpoint="worker",peer="proxy"} 1

Includes small btest baseline update to reflect @load of a new script.
2024-12-06 15:18:05 -08:00
Christian Kreibich
d260a5b7a9 Remove unneeded @loads from base/misc/version.zeek
This module is loaded by the telemetry framework, which we're now loading via
the cluster framework, i.e. also in bare mode. The resulting additional
thread (for creating reporter.log) trips up a number of btest baselines.

version.zeek doesn't use any of the string helper functions.
2024-12-06 15:18:05 -08:00
Christian Kreibich
46a11ec37d Add Cluster::nodeid_to_node() helper function
This translates backend-specific node identifiers (like Broker IDs) to
cluster nodes and their names, if available.
2024-12-06 15:18:05 -08:00
Christian Kreibich
0010e65f6d Support re-peering with Broker peers that fall behind
This adds re-peering at the Broker level for peers that Broker decided to
unpeer. We keep this at the Broker level since this behavior is specific to
it (as opposed to other cluster backends).

Includes baseline updates for btests that pick up on the new script's @load.
2024-12-06 15:18:05 -08:00
Dominik Charousset
4c4eb4b8e2 Add Zeek-level configurability of Broker slow-peer disconnects 2024-12-06 15:18:05 -08:00
Christian Kreibich
b9df1674b7 Bump Broker to pull in disconnect feature and infinite-loop fix 2024-12-06 15:18:05 -08:00
Christian Kreibich
e81856a4af No need to namespace Cluster:: functions in their own namespace 2024-12-06 15:18:05 -08:00
Tim Wojtulewicz
bbd7f56dcc Add signatures for Python bytecode for 3.8-3.14 2024-12-06 13:45:46 -07:00
Arne Welzel
27517ee4cb Merge remote-tracking branch 'origin/topic/awelzel/4068-http-upgrade-content-type'
* origin/topic/awelzel/4068-http-upgrade-content-type:
  btest/http: Demo StreamEvent analyzer with HTTP::upgrade_analyzers
  protocol: Add StreamEvent analyzer
2024-12-06 17:33:52 +01:00
Arne Welzel
079ae460a7 btest/http: Demo StreamEvent analyzer with HTTP::upgrade_analyzers
Relates to #4068
2024-12-06 16:12:40 +01:00
Arne Welzel
51836d08ae protocol: Add StreamEvent analyzer
This analyzer can be used to transport raw stream data for a given
connection to the script layer. For example, adding this analyzer into
the HTTP::upgrade_analyzer or using it to configure a child WebSocket
analyzer allows to get access to the raw stream data in script land
when no more appropriate protocol analyzer is available.
2024-12-06 16:12:40 +01:00
Arne Welzel
9619cd0f17 Add missing copyright line to headers and cc files 2024-12-06 12:50:58 +01:00
Arne Welzel
d93249eeab pre-commit: Add license-header check inspired by Spicy 2024-12-06 12:50:58 +01:00
Arne Welzel
4958c56c84 Add missing "COPYING" in file comments
This was just done via sed. There's a number of files that don't
have a license entry at all.
2024-12-06 11:55:46 +01:00
Arne Welzel
2f27db6542 Merge remote-tracking branch 'origin/topic/vern/zam-assert-fix'
* origin/topic/vern/zam-assert-fix:
  fix for ZAM "assert" statements potentially evaluating invalid expressions
2024-12-06 10:22:24 +01:00
zeek-bot
a1776540f4 Update doc submodule [nomail] [skip ci] 2024-12-06 00:22:21 +00:00
Vern Paxson
e6e31bc25a fix for ZAM "assert" statements potentially evaluating invalid expressions 2024-12-05 14:29:06 -08:00
Tim Wojtulewicz
682789a8e9 Merge remote-tracking branch 'origin/topic/awelzel/deprecate-tcp-reassembler-ports'
* origin/topic/awelzel/deprecate-tcp-reassembler-ports:
  init-bare: Deprecate tcp_reassembler_ports
2024-12-05 13:48:57 -07:00
Arne Welzel
93a3a11d36 Merge remote-tracking branch 'origin/topic/vern/script-opt-keep-asserts'
* origin/topic/vern/script-opt-keep-asserts:
  ZAM documentation updates for asserts and event handler run-time errors
  BTest updates for ZAM support of (optionally) keeping "assert" statements
  command-line options for controlling script optimization: keeping asserts, avoiding event handler coalescence
  ZAM support for option to not coalesce event handlers
  ZAM support for keeping "assert" statements
  internal support for script optimization options for keeping asserts, not consolidating event handlers
  ZAM operations to support asserts
  simplified "assert" by not trying to catch messages that themselves have errors

Fixed some TEST-REQUIRES "${ZEEK_ZAM}" == "1" to use "=" instead to
be /bin/sh compatible.
2024-12-05 21:45:07 +01:00
Arne Welzel
9efa12b055 Merge remote-tracking branch 'origin/topic/vern/ZAM-empty-hook-opt'
* origin/topic/vern/ZAM-empty-hook-opt:
  ZAM optimization now removes hook calls to hooks without any bodies
2024-12-05 19:12:59 +01:00
Tim Wojtulewicz
488ac7ddf0 Merge remote-tracking branch 'origin/topic/vern/zam-pattern-comparison'
* origin/topic/vern/zam-pattern-comparison:
  update of BTest that tracks number of (and validates) ZAM operations
  ZAM support for pattern equality/inequality operations
  expanded ZAM operations for bit-shifting to allow for int/count shift values
  added type coercion for bit-shifting expressions
2024-12-05 11:09:15 -07:00
Arne Welzel
e773fe0506 init-bare: Deprecate tcp_reassembler_ports
...and remove from NetVar, seems left-over.
2024-12-05 16:18:11 +01:00
Arne Welzel
6482a9a8c2 Merge remote-tracking branch 'origin/topic/awelzel/ctu-sme-11-vm-win7ad'
* origin/topic/awelzel/ctu-sme-11-vm-win7ad:
  zeek-testing: Add Experiment-VM-Microsoft-Windows7AD-1 PCAP and baselines
2024-12-05 14:52:14 +01:00
zeek-bot
55929e5acd Update doc submodule [nomail] [skip ci] 2024-12-05 00:19:43 +00:00
Vern Paxson
678db505fb update of BTest that tracks number of (and validates) ZAM operations 2024-12-04 15:26:13 -07:00