Commit graph

5062 commits

Author SHA1 Message Date
Arne Welzel
e5c0a597d0 HTTP: Support HTTP::upgrade_content_type_analyzers
This allows to select the Upgrade analyzer based on the tuple
of [Upgrade, Content-Type] as motivated by the Docker API trace
that sets a very specific Content-Type in the HTTP response.

Closes #4068
2024-12-06 18:49:51 +01:00
Arne Welzel
079ae460a7 btest/http: Demo StreamEvent analyzer with HTTP::upgrade_analyzers
Relates to #4068
2024-12-06 16:12:40 +01:00
Arne Welzel
51836d08ae protocol: Add StreamEvent analyzer
This analyzer can be used to transport raw stream data for a given
connection to the script layer. For example, adding this analyzer into
the HTTP::upgrade_analyzer or using it to configure a child WebSocket
analyzer allows to get access to the raw stream data in script land
when no more appropriate protocol analyzer is available.
2024-12-06 16:12:40 +01:00
Vern Paxson
e6e31bc25a fix for ZAM "assert" statements potentially evaluating invalid expressions 2024-12-05 14:29:06 -08:00
Arne Welzel
93a3a11d36 Merge remote-tracking branch 'origin/topic/vern/script-opt-keep-asserts'
* origin/topic/vern/script-opt-keep-asserts:
  ZAM documentation updates for asserts and event handler run-time errors
  BTest updates for ZAM support of (optionally) keeping "assert" statements
  command-line options for controlling script optimization: keeping asserts, avoiding event handler coalescence
  ZAM support for option to not coalesce event handlers
  ZAM support for keeping "assert" statements
  internal support for script optimization options for keeping asserts, not consolidating event handlers
  ZAM operations to support asserts
  simplified "assert" by not trying to catch messages that themselves have errors

Fixed some TEST-REQUIRES "${ZEEK_ZAM}" == "1" to use "=" instead to
be /bin/sh compatible.
2024-12-05 21:45:07 +01:00
Tim Wojtulewicz
488ac7ddf0 Merge remote-tracking branch 'origin/topic/vern/zam-pattern-comparison'
* origin/topic/vern/zam-pattern-comparison:
  update of BTest that tracks number of (and validates) ZAM operations
  ZAM support for pattern equality/inequality operations
  expanded ZAM operations for bit-shifting to allow for int/count shift values
  added type coercion for bit-shifting expressions
2024-12-05 11:09:15 -07:00
Arne Welzel
6482a9a8c2 Merge remote-tracking branch 'origin/topic/awelzel/ctu-sme-11-vm-win7ad'
* origin/topic/awelzel/ctu-sme-11-vm-win7ad:
  zeek-testing: Add Experiment-VM-Microsoft-Windows7AD-1 PCAP and baselines
2024-12-05 14:52:14 +01:00
Vern Paxson
678db505fb update of BTest that tracks number of (and validates) ZAM operations 2024-12-04 15:26:13 -07:00
Arne Welzel
df5a6a1ea0 zeek-testing: Add Experiment-VM-Microsoft-Windows7AD-1 PCAP and baselines
The pcap comes from the following dataset [1]:

    CTU-SME-11: a labeled dataset with real benign and malicious network
    traffic mimicking a small medium-size enterprise environment

[1] https://zenodo.org/records/7958259
2024-12-04 20:12:30 +01:00
Arne Welzel
fdde1e9841 cluster/serializer: Add binary-serialization-format
This is a serializer for log records that is using SerialTypes
for serializing and un-serializing rather. Essentially, this is
similar to what broker does except for the envelope.
2024-12-04 12:40:35 +01:00
Arne Welzel
2c52e533b2 Merge remote-tracking branch 'origin/topic/awelzel/spicy-type-convert-asan-options'
* origin/topic/awelzel/spicy-type-convert-asan-options:
  btest/spicy.type-converter: Drop ASAN_OPTIONS from test
2024-12-04 11:14:10 +01:00
Arne Welzel
a2cf0d874a btest/spicy.type-converter: Drop ASAN_OPTIONS from test
...doesn't seem it's needed anymore.

Fixes #4077
2024-12-04 11:11:47 +01:00
Vern Paxson
47a555da87 BTest updates for ZAM support of (optionally) keeping "assert" statements 2024-12-03 10:40:41 -07:00
Tim Wojtulewicz
43e3de5c79 Add interval_as_double argument to control how intervals are converted to JSON 2024-12-03 09:26:08 -07:00
Tim Wojtulewicz
8ff10e4d21 Add btest for round-trip JSON conversion 2024-12-03 09:26:08 -07:00
Vern Paxson
05e913db1b simplified "assert" by not trying to catch messages that themselves have errors 2024-12-02 10:37:10 -08:00
Vern Paxson
2a9c1d900f added new Cluster:: BiFs to script optimization tracking 2024-11-30 08:36:39 -08:00
Johanna Amann
079581aa6b Add TEST-REQUIRES: which jq to two new tests that are using jq 2024-11-27 16:52:15 +00:00
Johanna Amann
3523d393bf Explain why modbus test does not work with the Spicy SSL analyzer
This took a _long_ time to figure out.
2024-11-27 14:51:26 +00:00
Johanna Amann
7b582bc345 Merge remote-tracking branch 'origin/topic/johanna/sqlite-pragmas'
* origin/topic/johanna/sqlite-pragmas:
  Options for SQLite log writer, eliminate duplicate definitions
  Test synchronous/journal mode options for SQLite log writer
  Added default options for synchronous and journal mode
  Support for synchronous and journal_mode
2024-11-27 08:32:08 +00:00
Arne Welzel
3ce41f1eff Merge remote-tracking branch 'origin/topic/awelzel/pluggable-cluster-backends-part2'
* origin/topic/awelzel/pluggable-cluster-backends-part2:
  ci/test.sh: Run doctest with TZ=UTC
  cluster/setup-connections: Switch to Cluster::subscribe(), short-circuit broker
  cluster/serializer: Add Broker based event serializers
  cluster: Add Cluster scoped bifs
  Reporter: Add ScriptLocationScope helper
  init-bare/zeek-setup: Add Cluster::backend const &redef
  broker: Implement cluster::Backend interface
  Broker: Fix some error messages
  broker: Remove MakeEvent(ArgsSpan)
2024-11-26 18:01:28 +01:00
Arne Welzel
c3b30b187e Merge remote-tracking branch 'origin/topic/vern/zam-exception-leaks'
* origin/topic/vern/zam-exception-leaks:
  More robust memory management for ZAM execution - fixes #4052
2024-11-26 17:17:10 +01:00
Vern Paxson
847fcc66d6 More robust memory management for ZAM execution - fixes #4052 2024-11-26 17:05:38 +01:00
Johanna Amann
d592942ccb Test synchronous/journal mode options for SQLite log writer
Also adds some small tweaks and adds the new feature to NEWS.
2024-11-26 12:26:38 +00:00
Arne Welzel
9ec872d161 cluster/serializer: Add Broker based event serializers
This adds the first event serializers that use
broker functionality. Binary and JSON formats.
2024-11-26 12:58:23 +01:00
Arne Welzel
ef04a199c8 cluster: Add Cluster scoped bifs
... and a broker based test using Cluster::publish() and
Cluster::subscribe().
2024-11-26 12:58:23 +01:00
Arne Welzel
40a780bb24 Merge remote-tracking branch 'origin/topic/awelzel/test-init-hooks-plugin'
* origin/topic/awelzel/test-init-hooks-plugin:
  btest/plugins: Add a plugin testing Init and Done hooks
2024-11-22 18:38:06 +01:00
Tim Wojtulewicz
24d913c0a9 Disable core.expr-execption btest under ZAM to fix CI builds
Relates to #4052.
2024-11-22 15:28:14 +01:00
Arne Welzel
39b43e8dd6 btest/plugins: Add a plugin testing Init and Done hooks
Follow-up for #4047.
2024-11-22 13:37:03 +01:00
Arne Welzel
97f05b2f8c Merge remote-tracking branch 'origin/topic/awelzel/pluggable-cluster-backends-part1'
* origin/topic/awelzel/pluggable-cluster-backends-part1:
  btest: Test Broker::make_event() together with Cluster::publish_hrw()
  btest: Add cluster dir, minimal test for enum value
  broker: Add shim plugin adding a backend component
  zeek-setup: Instantiate backend::manager
  cluster: Add to src/CMakeLists.txt
  cluster: Add Components and ComponentManager for new components
  cluster/Backend: Interface for cluster backends
  cluster/Serializer: Interface for event and log serializers
  logging: Introduce logging/Types.h
  SerialTypes/Field: Allow default construction and add move constructor
  DebugLogger: Add cluster debugging stream
  plugin: Add component enums for pluggable cluster backends
  broker: Pass frame to MakeEvent()
2024-11-22 12:53:23 +01:00
Arne Welzel
baca6ba769 btest: Test Broker::make_event() together with Cluster::publish_hrw() 2024-11-22 10:43:55 +01:00
Arne Welzel
de9d39cd01 btest: Add cluster dir, minimal test for enum value 2024-11-22 10:43:55 +01:00
Arne Welzel
6fb73aa9da broker: Add shim plugin adding a backend component
For broker, this isn't really functioning, but at least makes the
CLUSTER_BACKEND_BROKER enum available.
2024-11-22 10:43:55 +01:00
Arne Welzel
9883b3a917 Merge remote-tracking branch 'origin/topic/awelzel/sumstats-non-cluster-final-epoch'
* origin/topic/awelzel/sumstats-non-cluster-final-epoch:
  sumstat/non-cluster: Move last epoch processing to zeek_done()
2024-11-21 16:39:18 +01:00
Christian Kreibich
b24c5c0e46 Order rule traversal in RuleMatcher::Match() operations by Rule index
This ordering fixes a test failure we're seeing on Alpine for the
signatures/tcp-end-of-match btest, since discrepancies in rule match traversal
could lead to discrepancies in corresponding event ordering.

It looks safe to rely on across platforms since the index is driven by signature
load order, which shouldn't deviate. If this somehow doesn't hold in the future,
we'll only wind up with a test failure, not incorrect match behavior.

(Correction to 2e03fbb8b0, which I pushed
accidentally.)
2024-11-18 09:45:09 -08:00
Christian Kreibich
5e0e2a8bd8 Merge branch 'topic/christian/fix-rulematcher-iteration'
* topic/christian/fix-rulematcher-iteration:
  Define ordering on Rule instances and use on sets in RuleMatcher
2024-11-18 09:39:22 -08:00
Arne Welzel
7ae52ea4d2 Merge remote-tracking branch 'origin/topic/awelzel/zam-baseline-updates'
* origin/topic/awelzel/zam-baseline-updates:
  communityid: Use GetField() over GetFieldAs()
  btest/Baseline.zam: Updates for proto addition
2024-11-18 17:28:37 +01:00
Arne Welzel
91f5945f92 sumstat/non-cluster: Move last epoch processing to zeek_done()
@Sheco reported that standalone epoch processing may exclude scheduled
events when the final sumstat epoch runs before. For example, this easily
happens when attempting to do sumstat observations within connection_state_remove().

Delay final epoch processing to zeek_done() instead.

This doesn't deal with the clustered version - this would need something
more elaborate and potentially a mechanism to delay the shutdown of
other cluster nodes until/after sumstat processing completed.
2024-11-18 15:58:01 +01:00
Tim Wojtulewicz
62bc324fe1 Merge remote-tracking branch 'origin/topic/christian/debug-stream-tweaks'
* origin/topic/christian/debug-stream-tweaks:
  Make debug stream names use "-", and handle "_" transparently
  Make "-B all" apply to plugin streams as well.
  Sort streams in "-B help" output, and match case-insensitively throughout
2024-11-16 09:34:42 -07:00
Christian Kreibich
b1e0e26484 Make debug stream names use "-", and handle "_" transparently
This looks consistent, and doesn't break any existing uses with underscores.
2024-11-15 17:53:29 -08:00
Tim Wojtulewicz
d30f480ca9 The cisco-fabric-path pcap used in tests doesn't need the -C argument 2024-11-15 16:50:10 -07:00
Tim Wojtulewicz
fbaf438a1d Add btests for ip_proto policy files 2024-11-15 16:49:10 -07:00
Christian Kreibich
7563af4906 Make "-B all" apply to plugin streams as well. 2024-11-15 13:53:06 -08:00
Christian Kreibich
5e4942bcc0 Sort streams in "-B help" output, and match case-insensitively throughout
Matching of plugins' debug streams was still case-sensitive. Also contains some
minor output tweaks.

It'd be nice to only list plugin debug streams actually _used_ by plugins. I
didn't see a quick way to do that so that's for another time.
2024-11-15 12:50:13 -08:00
Arne Welzel
173e6683f5 btest/Baseline.zam: Updates for proto addition 2024-11-15 17:36:49 +01:00
Christian Kreibich
2e03fbb8b0 Define ordering on Rule instances and use on sets in RuleMatcher
Establishing reliable ordering fixes a test failure we're seeing on Alpine for
the signatures/tcp-end-of-match btest, since discrepancies in rule match
traversal could lead to discrepancies in corresponding event ordering.
2024-11-14 20:03:07 -08:00
Arne Welzel
42cf86b503 Update external commit hashes 2024-11-14 14:37:05 +01:00
Arne Welzel
8ff49f9910 Merge remote-tracking branch 'origin/topic/awelzel/communityid-non-tcp-udp-icmp'
* origin/topic/awelzel/communityid-non-tcp-udp-icmp:
  communityid: Do not include ports for non TCP, UDP, ICMP
2024-11-14 14:21:36 +01:00
Arne Welzel
18bfdb8a2b Merge remote-tracking branch 'origin/topic/awelzel/deprecate-broker-auto-publish'
* origin/topic/awelzel/deprecate-broker-auto-publish:
  sumstats: Remove copy() for Broker::publish() calls
  broker/Publish: Use event time instead of network time
  broker/Eventhandler: Deprecate Broker::auto_publish() for v8.1
  btest: Remove Broker::auto_publish() usages
  frameworks/control: Remove Broker::auto_publish()
  catch-and-release: Remove Broker::auto_publish()
  ssl/validate-certs: Remove Broker::auto_publish()
  sumstats: Remove Broker::auto_publish()
  cluster_started: No Broker::auto_publish() use
  openflow: Remove Broker::auto_publish()
  dhcp: Remove Broker::auto_publish()
  frameworks/notice: Remove Broker::auto_publish()
  netcontrol: Replace Broker::auto_publish()
  intel: Switch to Cluster::publish()
  broker: Support publish() of unspecified set() / table()
  types: Fix table() resulting in table_type->IsSet() == true
2024-11-14 14:17:13 +01:00
Arne Welzel
831614f907 broker/Publish: Use event time instead of network time
Discussed with @J-Gras, calling Broker::publish() within a scheduled
should use the "intended timestamp" implicitly.

This is subtle, but supposedly more expected when running
a pcap replay cluster.
2024-11-14 12:59:22 +01:00