Commit graph

17261 commits

Author SHA1 Message Date
Christian Kreibich
b24c5c0e46 Order rule traversal in RuleMatcher::Match() operations by Rule index
This ordering fixes a test failure we're seeing on Alpine for the
signatures/tcp-end-of-match btest, since discrepancies in rule match traversal
could lead to discrepancies in corresponding event ordering.

It looks safe to rely on across platforms since the index is driven by signature
load order, which shouldn't deviate. If this somehow doesn't hold in the future,
we'll only wind up with a test failure, not incorrect match behavior.

(Correction to 2e03fbb8b0, which I pushed
accidentally.)
2024-11-18 09:45:09 -08:00
Christian Kreibich
5e0e2a8bd8 Merge branch 'topic/christian/fix-rulematcher-iteration'
* topic/christian/fix-rulematcher-iteration:
  Define ordering on Rule instances and use on sets in RuleMatcher
2024-11-18 09:39:22 -08:00
Arne Welzel
7ae52ea4d2 Merge remote-tracking branch 'origin/topic/awelzel/zam-baseline-updates'
* origin/topic/awelzel/zam-baseline-updates:
  communityid: Use GetField() over GetFieldAs()
  btest/Baseline.zam: Updates for proto addition
2024-11-18 17:28:37 +01:00
Tim Wojtulewicz
62bc324fe1 Merge remote-tracking branch 'origin/topic/christian/debug-stream-tweaks'
* origin/topic/christian/debug-stream-tweaks:
  Make debug stream names use "-", and handle "_" transparently
  Make "-B all" apply to plugin streams as well.
  Sort streams in "-B help" output, and match case-insensitively throughout
2024-11-16 09:34:42 -07:00
Christian Kreibich
26cc378471 Merge branch 'topic/timw/avoid-unknown-ip-transport-weirds'
* topic/timw/avoid-unknown-ip-transport-weirds:
  Don't forward out of the unknown IP transport analyzer
2024-11-15 19:08:33 -08:00
Christian Kreibich
b1e0e26484 Make debug stream names use "-", and handle "_" transparently
This looks consistent, and doesn't break any existing uses with underscores.
2024-11-15 17:53:29 -08:00
Tim Wojtulewicz
5c1c592554 Don't forward out of the unknown IP transport analyzer
We already accept that the packets coming into this analyzer won't
be processed, so forwarding out of it will just result in a failure.
Forwarding will also report a weird for every packet which just
results in extra noise.
2024-11-15 16:54:53 -07:00
Tim Wojtulewicz
bb2456129a Merge remote-tracking branch 'origin/topic/timw/ip-proto-btests'
* origin/topic/timw/ip-proto-btests:
  The cisco-fabric-path pcap used in tests doesn't need the -C argument
  Add btests for ip_proto policy files
2024-11-15 16:51:15 -07:00
Tim Wojtulewicz
d30f480ca9 The cisco-fabric-path pcap used in tests doesn't need the -C argument 2024-11-15 16:50:10 -07:00
Tim Wojtulewicz
fbaf438a1d Add btests for ip_proto policy files 2024-11-15 16:49:10 -07:00
Christian Kreibich
7563af4906 Make "-B all" apply to plugin streams as well. 2024-11-15 13:53:06 -08:00
Christian Kreibich
5e4942bcc0 Sort streams in "-B help" output, and match case-insensitively throughout
Matching of plugins' debug streams was still case-sensitive. Also contains some
minor output tweaks.

It'd be nice to only list plugin debug streams actually _used_ by plugins. I
didn't see a quick way to do that so that's for another time.
2024-11-15 12:50:13 -08:00
Arne Welzel
2125a1f558 Merge remote-tracking branch 'origin/topic/awelzel/log-broker-io-telemetry-rules-includes'
* origin/topic/awelzel/log-broker-io-telemetry-rules-includes:
  zeek-setup: Remove some unused headers
  clang-format: Sort doctest header at the bottom
  RuleMatcher: Move plugin/Manager.h include from .h to .cc
  iosource/Manager: Remove superflous includes
  telemetry/Manager: Remove broker header include, add fnmatch.h
  logging/Manager: Fix using filename from input.h in debug log
2024-11-15 17:40:40 +01:00
Arne Welzel
a75ade222b communityid: Use GetField() over GetFieldAs()
GetFieldAs() does not initialize an optional or default field. Apparently,
for ZAM that makes a difference, possibly the [] record construction is
optimized, not initializing proto until actually accessed. Not quite
sure why that's not happening for classic script interpretation
though.
2024-11-15 17:38:53 +01:00
Arne Welzel
173e6683f5 btest/Baseline.zam: Updates for proto addition 2024-11-15 17:36:49 +01:00
Arne Welzel
a46387075c zeek-setup: Remove some unused headers
The zeek_binpac.h one isn't used directly, but keeping it over
<binpac.h> include. Also do some std prefixing. binpac.h has
'using namespace std', so these slip through :-(
2024-11-15 17:00:02 +01:00
Arne Welzel
0f1c1cb754 clang-format: Sort doctest header at the bottom 2024-11-15 17:00:00 +01:00
Arne Welzel
a02ae82778 RuleMatcher: Move plugin/Manager.h include from .h to .cc 2024-11-15 16:00:23 +01:00
Arne Welzel
c380ee68ca iosource/Manager: Remove superflous includes 2024-11-15 15:55:46 +01:00
Arne Welzel
da291272f9 telemetry/Manager: Remove broker header include, add fnmatch.h 2024-11-15 15:55:42 +01:00
Arne Welzel
65037fa822 logging/Manager: Fix using filename from input.h in debug log
...and remove network_time, it's always included.
2024-11-15 15:46:24 +01:00
Christian Kreibich
2e03fbb8b0 Define ordering on Rule instances and use on sets in RuleMatcher
Establishing reliable ordering fixes a test failure we're seeing on Alpine for
the signatures/tcp-end-of-match btest, since discrepancies in rule match
traversal could lead to discrepancies in corresponding event ordering.
2024-11-14 20:03:07 -08:00
zeek-bot
b4ddf73e22 Update doc submodule [nomail] [skip ci] 2024-11-15 00:22:21 +00:00
Arne Welzel
42cf86b503 Update external commit hashes 2024-11-14 14:37:05 +01:00
Arne Welzel
8ff49f9910 Merge remote-tracking branch 'origin/topic/awelzel/communityid-non-tcp-udp-icmp'
* origin/topic/awelzel/communityid-non-tcp-udp-icmp:
  communityid: Do not include ports for non TCP, UDP, ICMP
2024-11-14 14:21:36 +01:00
Arne Welzel
18bfdb8a2b Merge remote-tracking branch 'origin/topic/awelzel/deprecate-broker-auto-publish'
* origin/topic/awelzel/deprecate-broker-auto-publish:
  sumstats: Remove copy() for Broker::publish() calls
  broker/Publish: Use event time instead of network time
  broker/Eventhandler: Deprecate Broker::auto_publish() for v8.1
  btest: Remove Broker::auto_publish() usages
  frameworks/control: Remove Broker::auto_publish()
  catch-and-release: Remove Broker::auto_publish()
  ssl/validate-certs: Remove Broker::auto_publish()
  sumstats: Remove Broker::auto_publish()
  cluster_started: No Broker::auto_publish() use
  openflow: Remove Broker::auto_publish()
  dhcp: Remove Broker::auto_publish()
  frameworks/notice: Remove Broker::auto_publish()
  netcontrol: Replace Broker::auto_publish()
  intel: Switch to Cluster::publish()
  broker: Support publish() of unspecified set() / table()
  types: Fix table() resulting in table_type->IsSet() == true
2024-11-14 14:17:13 +01:00
Arne Welzel
aabc4a4114 sumstats: Remove copy() for Broker::publish() calls
Serialization happens immediately at Broker::publish() time, there
should be no caching issues.
2024-11-14 12:59:22 +01:00
Arne Welzel
831614f907 broker/Publish: Use event time instead of network time
Discussed with @J-Gras, calling Broker::publish() within a scheduled
should use the "intended timestamp" implicitly.

This is subtle, but supposedly more expected when running
a pcap replay cluster.
2024-11-14 12:59:22 +01:00
Arne Welzel
6abb9d7eda broker/Eventhandler: Deprecate Broker::auto_publish() for v8.1
Relates to #3637
2024-11-14 12:59:22 +01:00
Arne Welzel
455e05bc2e btest: Remove Broker::auto_publish() usages
The ones that seemed to test Broker::auto_publish() were annotated
for removal.
2024-11-14 12:59:22 +01:00
Arne Welzel
927e936653 frameworks/control: Remove Broker::auto_publish() 2024-11-14 12:59:22 +01:00
Arne Welzel
6aca4d1dc7 catch-and-release: Remove Broker::auto_publish() 2024-11-14 12:59:22 +01:00
Arne Welzel
44c4a91cc8 ssl/validate-certs: Remove Broker::auto_publish() 2024-11-14 12:59:22 +01:00
Arne Welzel
883ae3694c sumstats: Remove Broker::auto_publish() 2024-11-14 12:59:22 +01:00
Arne Welzel
416887157c cluster_started: No Broker::auto_publish() use 2024-11-14 12:59:22 +01:00
Arne Welzel
b32153037a openflow: Remove Broker::auto_publish() 2024-11-14 12:59:22 +01:00
Arne Welzel
cb10852f99 dhcp: Remove Broker::auto_publish()
This isn't prettier, but neither worse IMO. A test would be good.
2024-11-14 12:59:22 +01:00
Arne Welzel
08f2198d3e frameworks/notice: Remove Broker::auto_publish() 2024-11-14 12:59:22 +01:00
Arne Welzel
b05f7a4d0e communityid: Do not include ports for non TCP, UDP, ICMP
Checked against the result of pycommunityid. The SCTP case
isn't quite right, because Zeek's core will not have extracted
any ports for SCTP.
2024-11-14 11:05:43 +01:00
Christian Kreibich
af4c21763f Merge branch 'topic/christian/ci-updates'
* topic/christian/ci-updates:
  CI: Use FEDORA40 crypto policy in Fedora 41
  Bump zeekjs to 0.13.0
  CI: bump FreeBSD 13 to 13.4, released in September
  CI: drop Fedora 39, add 41
2024-11-13 17:29:23 -08:00
Johanna Amann
09d6be7f68 CI: Use FEDORA40 crypto policy in Fedora 41
Fedora 41 distrusts SHA-1 signatures by default. Switching to this policy is
Fedora's recommended way of re-enabling support for at least the next several
releases.

A few references:

https://fedoraproject.org/wiki/Changes/OpenSSLDistrustSHA1SigVer
https://fedoraproject.org/wiki/SHA1SignaturesGuidance
https://www.redhat.com/en/blog/rhel-security-sha-1-package-signatures-distrusted-rhel-9
2024-11-13 17:05:08 -08:00
zeek-bot
ba0e8290ed Update doc submodule [nomail] [skip ci] 2024-11-14 00:24:48 +00:00
Tim Wojtulewicz
3c08c57be3 Merge remote-tracking branch 'origin/topic/timw/3915-unknown-ip-protocol'
* origin/topic/timw/3915-unknown-ip-protocol:
  Add NEWS entry for ip_proto feature
  Move IP protocol names table out of policy script to init-bare
  Minor review nits
  Fixes for community ID hashing with new proto values
  Use new_connection instead of connection_state_remove
  Add policy script to remove ip_proto field, rename protocol naming script
  Rename protocol_id field to ip_proto and similar renaming for name field
  Increase size of proto fields to uint16_t, add common default value
  Disable part of core/dict-iteration-expire5 btest to avoid iteration bug
  Add conn.log entries for connections with unhandled IP protocols
2024-11-13 14:36:22 -07:00
Tim Wojtulewicz
ec3794b43e Add NEWS entry for ip_proto feature 2024-11-13 14:15:57 -07:00
Tim Wojtulewicz
e33aee8ca2 Move IP protocol names table out of policy script to init-bare 2024-11-13 14:08:30 -07:00
Tim Wojtulewicz
fd67206865 Minor review nits 2024-11-13 14:08:30 -07:00
Tim Wojtulewicz
43e77a3338 Fixes for community ID hashing with new proto values 2024-11-13 14:08:30 -07:00
Tim Wojtulewicz
5a3d16e16f Use new_connection instead of connection_state_remove 2024-11-13 14:08:30 -07:00
Tim Wojtulewicz
623fea9014 Add policy script to remove ip_proto field, rename protocol naming script 2024-11-13 14:08:04 -07:00
Tim Wojtulewicz
5e5aceb6f7 Rename protocol_id field to ip_proto and similar renaming for name field 2024-11-13 12:02:00 -07:00