Commit graph

3576 commits

Author SHA1 Message Date
Tim Wojtulewicz
98ec627404 Merge branch 'topic/timw/7.1-news-updates'
* topic/timw/7.1-news-updates:
  Add note about various dependency updates [nomail] [skip ci]
  NEWS additions for v7.1 [nomail] [skip ci]
2024-12-12 11:33:09 -07:00
Arne Welzel
953fe8e029 Merge remote-tracking branch 'origin/topic/justin/pools-cache-node-topic'
* origin/topic/justin/pools-cache-node-topic:
  Pre-compute the node topics for all pool entries.
2024-12-12 14:45:50 +01:00
Tim Wojtulewicz
b547c7287d Merge remote-tracking branch 'origin/topic/timw/remove-brocontrol-dir'
* origin/topic/timw/remove-brocontrol-dir:
  Update zeekctl submodule to remove BroControl dir
2024-12-11 15:33:17 -07:00
Tim Wojtulewicz
9e19b51f41 Merge remote-tracking branch 'origin/topic/johanna/spicysslnews'
* origin/topic/johanna/spicysslnews:
  Add blurb about Spicy SSL to NEWS
2024-12-11 12:59:28 -07:00
Tim Wojtulewicz
9f1a30d3fe Merge remote-tracking branch 'origin/topic/timw/add-spicy-to-coverity'
* origin/topic/timw/add-spicy-to-coverity:
  Add Spicy analyzers to Coverity builds
2024-12-11 12:38:59 -07:00
Tim Wojtulewicz
649cedb693 Merge remote-tracking branch 'origin/topic/timw/python-upgrade'
* origin/topic/timw/python-upgrade: (65 commits)
  Swap pre-commit yapf for ruff/ruff-format, fix findings
  Upgrade package-manager submodule to get python upgrade
  Upgrade broker submodule to get python upgrade
  Upgrade zeekctl submodule to get python upgrade
  Upgrade zeek-client submodule to get python upgrade
  Upgrade btest submodule to get python upgrade
  Require Python 3.9 in CMakeLists.txt
  CI: Install python 3.9 on ubuntu 20
  btest: Avoid loading ZeroMQ if not compiled in
  btest/coverage: Avoid warnings in test-all-policy-cluster
  Broker::publish: Warn on using Broker::publish() when inactive
  Update doc submodule [nomail] [skip ci]
  Update zeek-testing and zeek-testing-cluster commit hashes
  Update ZAM BiF-tracking
  configure: Add --disable-cluster-backend-zeromq
  CMakeLists: Cluster backends output
  zeromq: Conditionally enable by default
  btest/generic: Add publish_hrw(), publish_rr() and logging tests
  generate-docs: Run on Ubuntu 24.04, add cppzmq
  docker: Add cppzmq/libzmq dependencies
  ...
2024-12-11 11:46:02 -07:00
Tim Wojtulewicz
7f3bea7ce1 Merge remote-tracking branch 'origin/topic/awelzel/broker-publish-warn'
* origin/topic/awelzel/broker-publish-warn:
  btest: Avoid loading ZeroMQ if not compiled in
  btest/coverage: Avoid warnings in test-all-policy-cluster
  Broker::publish: Warn on using Broker::publish() when inactive
2024-12-11 11:08:11 -07:00
Arne Welzel
4ee2f9256b Merge remote-tracking branch 'origin/topic/awelzel/pluggable-cluster-backends-part4'
* origin/topic/awelzel/pluggable-cluster-backends-part4:
  Update ZAM BiF-tracking
  configure: Add --disable-cluster-backend-zeromq
  CMakeLists: Cluster backends output
  zeromq: Conditionally enable by default
  btest/generic: Add publish_hrw(), publish_rr() and logging tests
  generate-docs: Run on Ubuntu 24.04, add cppzmq
  docker: Add cppzmq/libzmq dependencies
  tsan_suppressions: Add called_from_lib: libzmq
  ci: Add cppzmq and libzmq to most platforms
  cluster/backend/zeromq: Add ZeroMQ based cluster backend
  cluster/backend/zeromq: Add cppzmq submodule
  cluster: Add Cluster::node_id(), allow redef of node_topic(), nodeid_topic()
  cluster: Move publish_hrw() and publish_rr() to cluster.bif
2024-12-11 09:52:41 +01:00
Tim Wojtulewicz
0701895132 Merge remote-tracking branch 'J-Gras/topic/jgras/add-unknown-proto-count'
* J-Gras/topic/jgras/add-unknown-proto-count:
  Add btest for unknown_protocols.log
  Add protocol_id count to unknown protocol record
2024-12-10 16:16:12 -07:00
Tim Wojtulewicz
666d319cfa Merge remote-tracking branch 'origin/topic/christian/news-tweaks'
* origin/topic/christian/news-tweaks:
  NEWS tweaks [skip ci]
2024-12-10 14:41:28 -07:00
Arne Welzel
79a71357c7 Merge remote-tracking branch 'origin/topic/awelzel/bif-tracking-fix'
* origin/topic/awelzel/bif-tracking-fix:
  Update BiF-tracking, add get_current_packet_ts()
2024-12-10 20:31:47 +01:00
Arne Welzel
77465a90b5 Merge remote-tracking branch 'jgras/topic/jgras/fix-packet-lag'
* jgras/topic/jgras/fix-packet-lag:
  Introduce get_packet_lag()
  Add btest for get_current_packet_ts()
  Introduce get_current_packet_ts to fix packet lag
2024-12-10 18:26:54 +01:00
Arne Welzel
9034e135d1 Merge remote-tracking branch 'origin/topic/awelzel/fix-dns-binds-complte-typo'
* origin/topic/awelzel/fix-dns-binds-complte-typo:
  DNS/dns_binds_rr: Fix complte to complete typo, switch to count
2024-12-10 09:57:00 +01:00
Christian Kreibich
1c42bfc715 Merge branch 'topic/christian/disconnect-slow-peers'
* topic/christian/disconnect-slow-peers:
  Bump cluster testsuite to pull in Broker backpressure tests
  Expand documentation of Broker events.
  Add sleep() BiF.
  Add backpressure disconnect notification to cluster.log and via telemetry
  Remove unneeded @loads from base/misc/version.zeek
  Add Cluster::nodeid_to_node() helper function
  Support re-peering with Broker peers that fall behind
  Add Zeek-level configurability of Broker slow-peer disconnects
  Bump Broker to pull in disconnect feature and infinite-loop fix
  No need to namespace Cluster:: functions in their own namespace
2024-12-09 23:33:35 -08:00
Tim Wojtulewicz
cb7cda9808 Merge remote-tracking branch 'origin/topic/awelzel/license-header-cleanup'
* origin/topic/awelzel/license-header-cleanup:
  Add missing copyright line to headers and cc files
  pre-commit: Add license-header check inspired by Spicy
  Add missing "COPYING" in file comments
2024-12-09 12:23:00 -07:00
Tim Wojtulewicz
6e75417032 Merge remote-tracking branch 'origin/topic/vern/CPP-streamlining'
* origin/topic/vern/CPP-streamlining:
  BTest baseline updates for compile-to-C++
  mark ZAM regression BTests as not suitable for compile-to-C++
  fix for -O gen-C++ maintenance helper to skip BTest intermediary files
  introduced simplified initialization for non-standalone -O gen-C++ code tied -O gen-standalone-C++ to use of --optimize-files
  streamline generated -O C++ code by relying on per-function profiles rather than aggregate profile
  when reporting available/unavailble C++ script bodies, flag those that are skipped
  modified AST profiling to mark (and fully skip) non-optimizable functions
  modified merge_types() to skip work if given identical types, which also preserves type names (useful for -O gen-C++)
2024-12-09 12:16:00 -07:00
Tim Wojtulewicz
832b9b6cf0 Merge remote-tracking branch 'origin/topic/timw/python-magic-numbers'
* origin/topic/timw/python-magic-numbers:
  Move python signatures to a separate file
  Add signatures for Python bytecode for 3.8-3.14
2024-12-09 12:09:20 -07:00
Arne Welzel
0ebcd9608f Merge remote-tracking branch 'origin/topic/vern/script-opt-uncompilable-AST'
* origin/topic/vern/script-opt-uncompilable-AST:
  ScriptOpt: Fail compilation if known exprs/stmts is outdated
  skip optimization of functions with AST nodes unknown to script optimization
2024-12-09 14:04:01 +01:00
Arne Welzel
27517ee4cb Merge remote-tracking branch 'origin/topic/awelzel/4068-http-upgrade-content-type'
* origin/topic/awelzel/4068-http-upgrade-content-type:
  btest/http: Demo StreamEvent analyzer with HTTP::upgrade_analyzers
  protocol: Add StreamEvent analyzer
2024-12-06 17:33:52 +01:00
Arne Welzel
2f27db6542 Merge remote-tracking branch 'origin/topic/vern/zam-assert-fix'
* origin/topic/vern/zam-assert-fix:
  fix for ZAM "assert" statements potentially evaluating invalid expressions
2024-12-06 10:22:24 +01:00
Tim Wojtulewicz
682789a8e9 Merge remote-tracking branch 'origin/topic/awelzel/deprecate-tcp-reassembler-ports'
* origin/topic/awelzel/deprecate-tcp-reassembler-ports:
  init-bare: Deprecate tcp_reassembler_ports
2024-12-05 13:48:57 -07:00
Arne Welzel
93a3a11d36 Merge remote-tracking branch 'origin/topic/vern/script-opt-keep-asserts'
* origin/topic/vern/script-opt-keep-asserts:
  ZAM documentation updates for asserts and event handler run-time errors
  BTest updates for ZAM support of (optionally) keeping "assert" statements
  command-line options for controlling script optimization: keeping asserts, avoiding event handler coalescence
  ZAM support for option to not coalesce event handlers
  ZAM support for keeping "assert" statements
  internal support for script optimization options for keeping asserts, not consolidating event handlers
  ZAM operations to support asserts
  simplified "assert" by not trying to catch messages that themselves have errors

Fixed some TEST-REQUIRES "${ZEEK_ZAM}" == "1" to use "=" instead to
be /bin/sh compatible.
2024-12-05 21:45:07 +01:00
Tim Wojtulewicz
488ac7ddf0 Merge remote-tracking branch 'origin/topic/vern/zam-pattern-comparison'
* origin/topic/vern/zam-pattern-comparison:
  update of BTest that tracks number of (and validates) ZAM operations
  ZAM support for pattern equality/inequality operations
  expanded ZAM operations for bit-shifting to allow for int/count shift values
  added type coercion for bit-shifting expressions
2024-12-05 11:09:15 -07:00
Arne Welzel
6482a9a8c2 Merge remote-tracking branch 'origin/topic/awelzel/ctu-sme-11-vm-win7ad'
* origin/topic/awelzel/ctu-sme-11-vm-win7ad:
  zeek-testing: Add Experiment-VM-Microsoft-Windows7AD-1 PCAP and baselines
2024-12-05 14:52:14 +01:00
Arne Welzel
a44cafafe7 Merge remote-tracking branch 'origin/topic/awelzel/pluggable-cluster-backends-part3'
* origin/topic/awelzel/pluggable-cluster-backends-part3:
  init-bare/zeek-setup: Groundwork for instantiating cluster backends
  cluster/serializer: Add binary-serialization-format
  logging/WriterFrontend: Add logic for non-broker cluster backends
  logging/WriterBackend: Include logging/Types.h
  logging/Manager: Implement new WriteBatchFromRemote()
  logging/WriterFrontend: Add LogWriteHeader as member
  logging: Add filter_name to WriterInfo
2024-12-04 13:47:49 +01:00
Arne Welzel
2c52e533b2 Merge remote-tracking branch 'origin/topic/awelzel/spicy-type-convert-asan-options'
* origin/topic/awelzel/spicy-type-convert-asan-options:
  btest/spicy.type-converter: Drop ASAN_OPTIONS from test
2024-12-04 11:14:10 +01:00
Arne Welzel
067c40a545 Merge remote-tracking branch 'origin/topic/neverlord/broker-cleanup'
* origin/topic/neverlord/broker-cleanup:
  Remove obsolete c_str_safe utility
  Remove obsolete Broker compatibility layer
2024-12-04 11:08:08 +01:00
Tim Wojtulewicz
deb2acaeda Merge remote-tracking branch 'origin/topic/timw/json-interval-conversion'
* origin/topic/timw/json-interval-conversion:
  Add interval_as_double argument to control how intervals are converted to JSON
  Add btest for round-trip JSON conversion
  Allow comparing two PatternVals
  Handle conversion between data from Val::ToJSON and ValFromJSON better
2024-12-03 10:10:37 -07:00
Arne Welzel
d2ecdc2a11 Merge remote-tracking branch 'origin/topic/vern/ZAM-cluster-bifs'
* origin/topic/vern/ZAM-cluster-bifs:
  added new Cluster:: BiFs to script optimization tracking
2024-12-02 13:43:36 +01:00
Arne Welzel
423c2de4c8 Merge remote-tracking branch 'origin/topic/vern/dump-no-opt-ZAM'
* origin/topic/vern/dump-no-opt-ZAM:
  fixed -O dump-final-ZAM to work in conjunction with -O no-ZAM-opt
2024-12-02 10:59:16 +01:00
Benjamin Bannier
f1bb009e44 Merge branch 'topic/bbannier/bump-spicy' 2024-12-02 10:35:34 +01:00
Tim Wojtulewicz
8d625715ca Merge remote-tracking branch 'origin/topic/johanna/jq-guard'
* origin/topic/johanna/jq-guard:
  Add TEST-REQUIRES: which jq to two new tests that are using jq
2024-11-27 12:31:59 -07:00
Johanna Amann
7b582bc345 Merge remote-tracking branch 'origin/topic/johanna/sqlite-pragmas'
* origin/topic/johanna/sqlite-pragmas:
  Options for SQLite log writer, eliminate duplicate definitions
  Test synchronous/journal mode options for SQLite log writer
  Added default options for synchronous and journal mode
  Support for synchronous and journal_mode
2024-11-27 08:32:08 +00:00
Arne Welzel
3ce41f1eff Merge remote-tracking branch 'origin/topic/awelzel/pluggable-cluster-backends-part2'
* origin/topic/awelzel/pluggable-cluster-backends-part2:
  ci/test.sh: Run doctest with TZ=UTC
  cluster/setup-connections: Switch to Cluster::subscribe(), short-circuit broker
  cluster/serializer: Add Broker based event serializers
  cluster: Add Cluster scoped bifs
  Reporter: Add ScriptLocationScope helper
  init-bare/zeek-setup: Add Cluster::backend const &redef
  broker: Implement cluster::Backend interface
  Broker: Fix some error messages
  broker: Remove MakeEvent(ArgsSpan)
2024-11-26 18:01:28 +01:00
Arne Welzel
c3b30b187e Merge remote-tracking branch 'origin/topic/vern/zam-exception-leaks'
* origin/topic/vern/zam-exception-leaks:
  More robust memory management for ZAM execution - fixes #4052
2024-11-26 17:17:10 +01:00
Arne Welzel
f65c4f0b7e Merge remote-tracking branch 'origin/topic/awelzel/deprecate-val-list-to-args'
* origin/topic/awelzel/deprecate-val-list-to-args:
  ZeekArgs: Deprecate val_list_to_args()
2024-11-26 15:25:25 +01:00
Arne Welzel
40a780bb24 Merge remote-tracking branch 'origin/topic/awelzel/test-init-hooks-plugin'
* origin/topic/awelzel/test-init-hooks-plugin:
  btest/plugins: Add a plugin testing Init and Done hooks
2024-11-22 18:38:06 +01:00
Arne Welzel
542640db00 Merge remote-tracking branch 'origin/topic/awelzel/skip-core-expr-error'
* origin/topic/awelzel/skip-core-expr-error:
  Disable core.expr-execption btest under ZAM to fix CI builds
2024-11-22 15:29:48 +01:00
Arne Welzel
97f05b2f8c Merge remote-tracking branch 'origin/topic/awelzel/pluggable-cluster-backends-part1'
* origin/topic/awelzel/pluggable-cluster-backends-part1:
  btest: Test Broker::make_event() together with Cluster::publish_hrw()
  btest: Add cluster dir, minimal test for enum value
  broker: Add shim plugin adding a backend component
  zeek-setup: Instantiate backend::manager
  cluster: Add to src/CMakeLists.txt
  cluster: Add Components and ComponentManager for new components
  cluster/Backend: Interface for cluster backends
  cluster/Serializer: Interface for event and log serializers
  logging: Introduce logging/Types.h
  SerialTypes/Field: Allow default construction and add move constructor
  DebugLogger: Add cluster debugging stream
  plugin: Add component enums for pluggable cluster backends
  broker: Pass frame to MakeEvent()
2024-11-22 12:53:23 +01:00
Arne Welzel
9883b3a917 Merge remote-tracking branch 'origin/topic/awelzel/sumstats-non-cluster-final-epoch'
* origin/topic/awelzel/sumstats-non-cluster-final-epoch:
  sumstat/non-cluster: Move last epoch processing to zeek_done()
2024-11-21 16:39:18 +01:00
Arne Welzel
94ffcd6535 Merge remote-tracking branch 'origin/topic/awelzel/plugin-init-pre-execution'
* origin/topic/awelzel/plugin-init-pre-execution:
  plugin: Add InitPreExecution()
2024-11-21 14:41:14 +01:00
Arne Welzel
aac640ebff Merge remote-tracking branch 'origin/topic/awelzel/bump-zeekjs-0-14-0'
* origin/topic/awelzel/bump-zeekjs-0-14-0:
  Bump zeekjs to v0.14.0
2024-11-21 10:22:44 +01:00
Arne Welzel
6e916efe8d Merge remote-tracking branch 'origin/topic/awelzel/bump-zeekjs-0-13-2'
* origin/topic/awelzel/bump-zeekjs-0-13-2:
  Bump zeekjs to 0.13.2
2024-11-19 19:17:48 +01:00
Tim Wojtulewicz
d9a74680e0 Merge remote-tracking branch 'origin/topic/timw/speed-up-zam-ci-testing'
* origin/topic/timw/speed-up-zam-ci-testing:
  CI: Use test.sh script for running ZAM tests, but disable parts of it
2024-11-19 10:53:21 -07:00
Christian Kreibich
96273b4550 Merge branch 'topic/christian/fix-rulematcher-iteration'
* topic/christian/fix-rulematcher-iteration:
  Order rule traversal in RuleMatcher::Match() operations by Rule index
2024-11-18 10:13:18 -08:00
Christian Kreibich
5e0e2a8bd8 Merge branch 'topic/christian/fix-rulematcher-iteration'
* topic/christian/fix-rulematcher-iteration:
  Define ordering on Rule instances and use on sets in RuleMatcher
2024-11-18 09:39:22 -08:00
Arne Welzel
7ae52ea4d2 Merge remote-tracking branch 'origin/topic/awelzel/zam-baseline-updates'
* origin/topic/awelzel/zam-baseline-updates:
  communityid: Use GetField() over GetFieldAs()
  btest/Baseline.zam: Updates for proto addition
2024-11-18 17:28:37 +01:00
Tim Wojtulewicz
62bc324fe1 Merge remote-tracking branch 'origin/topic/christian/debug-stream-tweaks'
* origin/topic/christian/debug-stream-tweaks:
  Make debug stream names use "-", and handle "_" transparently
  Make "-B all" apply to plugin streams as well.
  Sort streams in "-B help" output, and match case-insensitively throughout
2024-11-16 09:34:42 -07:00
Christian Kreibich
26cc378471 Merge branch 'topic/timw/avoid-unknown-ip-transport-weirds'
* topic/timw/avoid-unknown-ip-transport-weirds:
  Don't forward out of the unknown IP transport analyzer
2024-11-15 19:08:33 -08:00
Arne Welzel
2125a1f558 Merge remote-tracking branch 'origin/topic/awelzel/log-broker-io-telemetry-rules-includes'
* origin/topic/awelzel/log-broker-io-telemetry-rules-includes:
  zeek-setup: Remove some unused headers
  clang-format: Sort doctest header at the bottom
  RuleMatcher: Move plugin/Manager.h include from .h to .cc
  iosource/Manager: Remove superflous includes
  telemetry/Manager: Remove broker header include, add fnmatch.h
  logging/Manager: Fix using filename from input.h in debug log
2024-11-15 17:40:40 +01:00