Commit graph

5050 commits

Author SHA1 Message Date
Arne Welzel
2c52e533b2 Merge remote-tracking branch 'origin/topic/awelzel/spicy-type-convert-asan-options'
* origin/topic/awelzel/spicy-type-convert-asan-options:
  btest/spicy.type-converter: Drop ASAN_OPTIONS from test
2024-12-04 11:14:10 +01:00
Arne Welzel
a2cf0d874a btest/spicy.type-converter: Drop ASAN_OPTIONS from test
...doesn't seem it's needed anymore.

Fixes #4077
2024-12-04 11:11:47 +01:00
Tim Wojtulewicz
43e3de5c79 Add interval_as_double argument to control how intervals are converted to JSON 2024-12-03 09:26:08 -07:00
Tim Wojtulewicz
8ff10e4d21 Add btest for round-trip JSON conversion 2024-12-03 09:26:08 -07:00
Vern Paxson
2a9c1d900f added new Cluster:: BiFs to script optimization tracking 2024-11-30 08:36:39 -08:00
Johanna Amann
079581aa6b Add TEST-REQUIRES: which jq to two new tests that are using jq 2024-11-27 16:52:15 +00:00
Johanna Amann
3523d393bf Explain why modbus test does not work with the Spicy SSL analyzer
This took a _long_ time to figure out.
2024-11-27 14:51:26 +00:00
Johanna Amann
7b582bc345 Merge remote-tracking branch 'origin/topic/johanna/sqlite-pragmas'
* origin/topic/johanna/sqlite-pragmas:
  Options for SQLite log writer, eliminate duplicate definitions
  Test synchronous/journal mode options for SQLite log writer
  Added default options for synchronous and journal mode
  Support for synchronous and journal_mode
2024-11-27 08:32:08 +00:00
Arne Welzel
3ce41f1eff Merge remote-tracking branch 'origin/topic/awelzel/pluggable-cluster-backends-part2'
* origin/topic/awelzel/pluggable-cluster-backends-part2:
  ci/test.sh: Run doctest with TZ=UTC
  cluster/setup-connections: Switch to Cluster::subscribe(), short-circuit broker
  cluster/serializer: Add Broker based event serializers
  cluster: Add Cluster scoped bifs
  Reporter: Add ScriptLocationScope helper
  init-bare/zeek-setup: Add Cluster::backend const &redef
  broker: Implement cluster::Backend interface
  Broker: Fix some error messages
  broker: Remove MakeEvent(ArgsSpan)
2024-11-26 18:01:28 +01:00
Arne Welzel
c3b30b187e Merge remote-tracking branch 'origin/topic/vern/zam-exception-leaks'
* origin/topic/vern/zam-exception-leaks:
  More robust memory management for ZAM execution - fixes #4052
2024-11-26 17:17:10 +01:00
Vern Paxson
847fcc66d6 More robust memory management for ZAM execution - fixes #4052 2024-11-26 17:05:38 +01:00
Johanna Amann
d592942ccb Test synchronous/journal mode options for SQLite log writer
Also adds some small tweaks and adds the new feature to NEWS.
2024-11-26 12:26:38 +00:00
Arne Welzel
9ec872d161 cluster/serializer: Add Broker based event serializers
This adds the first event serializers that use
broker functionality. Binary and JSON formats.
2024-11-26 12:58:23 +01:00
Arne Welzel
ef04a199c8 cluster: Add Cluster scoped bifs
... and a broker based test using Cluster::publish() and
Cluster::subscribe().
2024-11-26 12:58:23 +01:00
Arne Welzel
40a780bb24 Merge remote-tracking branch 'origin/topic/awelzel/test-init-hooks-plugin'
* origin/topic/awelzel/test-init-hooks-plugin:
  btest/plugins: Add a plugin testing Init and Done hooks
2024-11-22 18:38:06 +01:00
Tim Wojtulewicz
24d913c0a9 Disable core.expr-execption btest under ZAM to fix CI builds
Relates to #4052.
2024-11-22 15:28:14 +01:00
Arne Welzel
39b43e8dd6 btest/plugins: Add a plugin testing Init and Done hooks
Follow-up for #4047.
2024-11-22 13:37:03 +01:00
Arne Welzel
97f05b2f8c Merge remote-tracking branch 'origin/topic/awelzel/pluggable-cluster-backends-part1'
* origin/topic/awelzel/pluggable-cluster-backends-part1:
  btest: Test Broker::make_event() together with Cluster::publish_hrw()
  btest: Add cluster dir, minimal test for enum value
  broker: Add shim plugin adding a backend component
  zeek-setup: Instantiate backend::manager
  cluster: Add to src/CMakeLists.txt
  cluster: Add Components and ComponentManager for new components
  cluster/Backend: Interface for cluster backends
  cluster/Serializer: Interface for event and log serializers
  logging: Introduce logging/Types.h
  SerialTypes/Field: Allow default construction and add move constructor
  DebugLogger: Add cluster debugging stream
  plugin: Add component enums for pluggable cluster backends
  broker: Pass frame to MakeEvent()
2024-11-22 12:53:23 +01:00
Arne Welzel
baca6ba769 btest: Test Broker::make_event() together with Cluster::publish_hrw() 2024-11-22 10:43:55 +01:00
Arne Welzel
de9d39cd01 btest: Add cluster dir, minimal test for enum value 2024-11-22 10:43:55 +01:00
Arne Welzel
6fb73aa9da broker: Add shim plugin adding a backend component
For broker, this isn't really functioning, but at least makes the
CLUSTER_BACKEND_BROKER enum available.
2024-11-22 10:43:55 +01:00
Arne Welzel
9883b3a917 Merge remote-tracking branch 'origin/topic/awelzel/sumstats-non-cluster-final-epoch'
* origin/topic/awelzel/sumstats-non-cluster-final-epoch:
  sumstat/non-cluster: Move last epoch processing to zeek_done()
2024-11-21 16:39:18 +01:00
Christian Kreibich
b24c5c0e46 Order rule traversal in RuleMatcher::Match() operations by Rule index
This ordering fixes a test failure we're seeing on Alpine for the
signatures/tcp-end-of-match btest, since discrepancies in rule match traversal
could lead to discrepancies in corresponding event ordering.

It looks safe to rely on across platforms since the index is driven by signature
load order, which shouldn't deviate. If this somehow doesn't hold in the future,
we'll only wind up with a test failure, not incorrect match behavior.

(Correction to 2e03fbb8b0, which I pushed
accidentally.)
2024-11-18 09:45:09 -08:00
Christian Kreibich
5e0e2a8bd8 Merge branch 'topic/christian/fix-rulematcher-iteration'
* topic/christian/fix-rulematcher-iteration:
  Define ordering on Rule instances and use on sets in RuleMatcher
2024-11-18 09:39:22 -08:00
Arne Welzel
7ae52ea4d2 Merge remote-tracking branch 'origin/topic/awelzel/zam-baseline-updates'
* origin/topic/awelzel/zam-baseline-updates:
  communityid: Use GetField() over GetFieldAs()
  btest/Baseline.zam: Updates for proto addition
2024-11-18 17:28:37 +01:00
Arne Welzel
91f5945f92 sumstat/non-cluster: Move last epoch processing to zeek_done()
@Sheco reported that standalone epoch processing may exclude scheduled
events when the final sumstat epoch runs before. For example, this easily
happens when attempting to do sumstat observations within connection_state_remove().

Delay final epoch processing to zeek_done() instead.

This doesn't deal with the clustered version - this would need something
more elaborate and potentially a mechanism to delay the shutdown of
other cluster nodes until/after sumstat processing completed.
2024-11-18 15:58:01 +01:00
Tim Wojtulewicz
62bc324fe1 Merge remote-tracking branch 'origin/topic/christian/debug-stream-tweaks'
* origin/topic/christian/debug-stream-tweaks:
  Make debug stream names use "-", and handle "_" transparently
  Make "-B all" apply to plugin streams as well.
  Sort streams in "-B help" output, and match case-insensitively throughout
2024-11-16 09:34:42 -07:00
Christian Kreibich
b1e0e26484 Make debug stream names use "-", and handle "_" transparently
This looks consistent, and doesn't break any existing uses with underscores.
2024-11-15 17:53:29 -08:00
Tim Wojtulewicz
d30f480ca9 The cisco-fabric-path pcap used in tests doesn't need the -C argument 2024-11-15 16:50:10 -07:00
Tim Wojtulewicz
fbaf438a1d Add btests for ip_proto policy files 2024-11-15 16:49:10 -07:00
Christian Kreibich
7563af4906 Make "-B all" apply to plugin streams as well. 2024-11-15 13:53:06 -08:00
Christian Kreibich
5e4942bcc0 Sort streams in "-B help" output, and match case-insensitively throughout
Matching of plugins' debug streams was still case-sensitive. Also contains some
minor output tweaks.

It'd be nice to only list plugin debug streams actually _used_ by plugins. I
didn't see a quick way to do that so that's for another time.
2024-11-15 12:50:13 -08:00
Arne Welzel
173e6683f5 btest/Baseline.zam: Updates for proto addition 2024-11-15 17:36:49 +01:00
Christian Kreibich
2e03fbb8b0 Define ordering on Rule instances and use on sets in RuleMatcher
Establishing reliable ordering fixes a test failure we're seeing on Alpine for
the signatures/tcp-end-of-match btest, since discrepancies in rule match
traversal could lead to discrepancies in corresponding event ordering.
2024-11-14 20:03:07 -08:00
Arne Welzel
42cf86b503 Update external commit hashes 2024-11-14 14:37:05 +01:00
Arne Welzel
8ff49f9910 Merge remote-tracking branch 'origin/topic/awelzel/communityid-non-tcp-udp-icmp'
* origin/topic/awelzel/communityid-non-tcp-udp-icmp:
  communityid: Do not include ports for non TCP, UDP, ICMP
2024-11-14 14:21:36 +01:00
Arne Welzel
18bfdb8a2b Merge remote-tracking branch 'origin/topic/awelzel/deprecate-broker-auto-publish'
* origin/topic/awelzel/deprecate-broker-auto-publish:
  sumstats: Remove copy() for Broker::publish() calls
  broker/Publish: Use event time instead of network time
  broker/Eventhandler: Deprecate Broker::auto_publish() for v8.1
  btest: Remove Broker::auto_publish() usages
  frameworks/control: Remove Broker::auto_publish()
  catch-and-release: Remove Broker::auto_publish()
  ssl/validate-certs: Remove Broker::auto_publish()
  sumstats: Remove Broker::auto_publish()
  cluster_started: No Broker::auto_publish() use
  openflow: Remove Broker::auto_publish()
  dhcp: Remove Broker::auto_publish()
  frameworks/notice: Remove Broker::auto_publish()
  netcontrol: Replace Broker::auto_publish()
  intel: Switch to Cluster::publish()
  broker: Support publish() of unspecified set() / table()
  types: Fix table() resulting in table_type->IsSet() == true
2024-11-14 14:17:13 +01:00
Arne Welzel
831614f907 broker/Publish: Use event time instead of network time
Discussed with @J-Gras, calling Broker::publish() within a scheduled
should use the "intended timestamp" implicitly.

This is subtle, but supposedly more expected when running
a pcap replay cluster.
2024-11-14 12:59:22 +01:00
Arne Welzel
455e05bc2e btest: Remove Broker::auto_publish() usages
The ones that seemed to test Broker::auto_publish() were annotated
for removal.
2024-11-14 12:59:22 +01:00
Arne Welzel
cb10852f99 dhcp: Remove Broker::auto_publish()
This isn't prettier, but neither worse IMO. A test would be good.
2024-11-14 12:59:22 +01:00
Arne Welzel
b05f7a4d0e communityid: Do not include ports for non TCP, UDP, ICMP
Checked against the result of pycommunityid. The SCTP case
isn't quite right, because Zeek's core will not have extracted
any ports for SCTP.
2024-11-14 11:05:43 +01:00
Tim Wojtulewicz
43e77a3338 Fixes for community ID hashing with new proto values 2024-11-13 14:08:30 -07:00
Tim Wojtulewicz
623fea9014 Add policy script to remove ip_proto field, rename protocol naming script 2024-11-13 14:08:04 -07:00
Tim Wojtulewicz
5e5aceb6f7 Rename protocol_id field to ip_proto and similar renaming for name field 2024-11-13 12:02:00 -07:00
Tim Wojtulewicz
d0896e81d6 Increase size of proto fields to uint16_t, add common default value 2024-11-13 11:25:46 -07:00
Tim Wojtulewicz
f762a45e83 Disable part of core/dict-iteration-expire5 btest to avoid iteration bug
The second set of seeds in this test trip the bug reported in #3538
2024-11-13 11:25:46 -07:00
Tim Wojtulewicz
35ec9733c0 Add conn.log entries for connections with unhandled IP protocols 2024-11-13 11:25:40 -07:00
Tim Wojtulewicz
0217208c49 Merge remote-tracking branch 'origin/topic/timw/remove-abspath-cleanup'
* origin/topic/timw/remove-abspath-cleanup:
  diff-remove-abspath: Add separate handling of Windows paths
  diff-remove-abspath: Remove capture of windows drive letters from POSIX regex
2024-11-12 12:26:56 -07:00
Arne Welzel
9e27334596 Merge remote-tracking branch 'origin/topic/vern/zam-asan-fixes'
* origin/topic/vern/zam-asan-fixes:
  ZAM fixes for assignments involving "any" record fields
  fixes for (mostly ZAM) vector operation issues found by ASAN

Including a fix for mmdb/explicit-open.zeek to avoid using assert.
2024-11-12 10:29:56 +01:00
Robin Sommer
0285196626
Merge remote-tracking branch 'origin/topic/robin/gh-3988-evt-assert'
* origin/topic/robin/gh-3988-evt-assert:
  Spicy: Improve error messages reporting malformed unit names in EVT files.
  Spicy:: Remove unhelpful assertion.
2024-11-11 14:02:15 +01:00