Commit graph

5556 commits

Author SHA1 Message Date
Tim Wojtulewicz
339d46ae26 Add a weird that gets emitted when strings/containers are over the limits 2025-08-12 17:31:28 -07:00
Tim Wojtulewicz
837fde1a08 Add metrics to track string and container fields limited by length 2025-08-12 17:31:28 -07:00
Tim Wojtulewicz
e2e7ab28da Implement string- and container-length filtering at the log record level 2025-08-12 17:31:28 -07:00
Tim Wojtulewicz
d95affde4d Remove deprecations tagged for v8.1 2025-08-12 10:19:03 -07:00
Arne Welzel
b2a2ad7e10 smb2/read: Parse only 1 byte for data_offset, ignore reserved1
A user provided a SMB2 pcap with the reserved1 field of a ReadResponse
set to 1 instead of 0. This confused the padding computation due to
including this byte into the offset. Properly split data_offset and
reserved1 into individual byte fields.

Closes #4730
2025-08-08 16:12:20 +02:00
Arne Welzel
13f613eb1d Merge remote-tracking branch 'origin/topic/awelzel/4176-cluster-on-sub-unsub-hooks'
* origin/topic/awelzel/4176-cluster-on-sub-unsub-hooks:
  cluster: Add on_subscribe() and on_unsubscribe() hooks
2025-08-08 14:24:18 +02:00
Tim Wojtulewicz
54d67c3322 Merge remote-tracking branch 'origin/topic/timw/cleanup-warnings-from-plugin-btest-builds'
* origin/topic/timw/cleanup-warnings-from-plugin-btest-builds:
  Update zeek-aux to remove BRO_DIST from plugin skeleton
  cmake_minimum_required() should come before project()
2025-08-07 08:39:40 -07:00
Tim Wojtulewicz
162ecc022e Update zeek-aux to remove BRO_DIST from plugin skeleton 2025-08-07 08:39:08 -07:00
Tim Wojtulewicz
3c535ec215 cmake_minimum_required() should come before project() 2025-08-06 12:10:41 -07:00
Arne Welzel
f98508bbb0 btest/tap-analyzer: Update existing test and add new one for UpdateConnVal()
This also changes the output of connection UIDs from the tap analyzer to be
prefixed with C for easier correlation with other logs.

Relates to #4337 #4725 #4734 #4737
2025-08-06 17:22:59 +02:00
Johanna Amann
2f2f328a72 Merge remote-tracking branch 'origin/topic/johanna/analyzer-log-proto'
* origin/topic/johanna/analyzer-log-proto:
  Add proto to analyzer.log
2025-08-06 14:38:47 +01:00
Evan Typanski
22f77248f5 Merge remote-tracking branch 'origin/topic/etyp/fix-record-vec-type-conflict'
* origin/topic/etyp/fix-record-vec-type-conflict:
  Fix record coercion with compatible types
2025-08-06 09:10:19 -04:00
Arne Welzel
ac776b0aad btest/plugins/tap-analyzer: Update baseline
Relates to #4337 #4725 #4734
2025-08-06 14:17:42 +02:00
Johanna Amann
82266b1e78 Add proto to analyzer.log
The analyzer.log file was missing the protocol field to distinguish
tcp/udp connections.
2025-08-06 11:34:57 +01:00
Arne Welzel
b4925fbd16 TapAnalyzer: More verdict to action rename
Relates to #4725 #4337
2025-08-05 19:59:06 +02:00
Arne Welzel
4bc7f9532c IPBasedAnalyzer: Call TapPacket() when skipping
When skip_further_processing() is called, a TapAnalyzer should still see
the packets as skipped with SkipReason "skipping".
2025-08-05 19:47:04 +02:00
Arne Welzel
dc904b2216 SessionAdapter: Introduce TapAnalyzer for session adapter
This commit introduces a mechanism to attach light weight analyzers to
the root analyzer of sessions in order to tap into the packets delivered
to child analyzer.
2025-08-05 19:47:02 +02:00
Evan Typanski
006bef71b5 Fix record coercion with compatible types
Fixes #4722
2025-08-04 17:09:26 -04:00
Tim Wojtulewicz
6afeeca090 Start of 8.1.0 development 2025-08-04 08:26:29 -07:00
Arne Welzel
1a87ebab72 cluster: Add on_subscribe() and on_unsubscribe() hooks
Closes #4176
2025-08-01 14:06:19 +02:00
Tim Wojtulewicz
f2e155d7fa Merge remote-tracking branch 'origin/topic/timw/update-ct-ca-lists'
* origin/topic/timw/update-ct-ca-lists:
  Update CT/CA lists to versions from NSS 3.114
2025-07-31 14:32:21 -07:00
Tim Wojtulewicz
1daead9edd Update CT/CA lists to versions from NSS 3.114 2025-07-31 11:34:23 -07:00
Johanna Amann
136bdb43fd Merge remote-tracking branch 'origin/topic/johanna/gh-4694'
* origin/topic/johanna/gh-4694:
  Add tests for the deprecated-dpd-log.zeek policy script
  Move c$service_violation to deprecated-dpd-log.zeek
2025-07-31 16:11:00 +01:00
Arne Welzel
10e7f14f78 Merge remote-tracking branch 'origin/topic/awelzel/defer-more-stuff'
* origin/topic/awelzel/defer-more-stuff:
  RecordType: Ensure &default fields are always re-initialized
  Attr: Deprecate using &default and &optional together on record fields
  RecordType: Allow deferring &default=vector(), set(), table() fields
2025-07-30 10:35:56 +02:00
Arne Welzel
9eb94ee151 RecordType: Ensure &default fields are always re-initialized
This started working partly after the deferral logic introduced with
Zeek 6.0 so this finishes it :-)
2025-07-30 10:26:06 +02:00
Arne Welzel
473723cc47 Attr: Deprecate using &default and &optional together on record fields
If &default implies re-initialization of the field, using them together
doesn't make much sense.
2025-07-30 10:26:06 +02:00
Arne Welzel
23181e4811 RecordType: Allow deferring &default=vector(), set(), table() fields 2025-07-30 10:26:06 +02:00
Arne Welzel
d7fbd49d9e Merge remote-tracking branch 'origin/topic/vern/zam-record-fields-fixes'
* origin/topic/vern/zam-record-fields-fixes:
  fixes for specialized ZAM operations needing to check whether record fields exist
2025-07-30 10:08:21 +02:00
Johanna Amann
a90969800c Add tests for the deprecated-dpd-log.zeek policy script
This re-adds baselines for the old dpd.log to check functionality until
its removal in 8.1
2025-07-30 07:58:36 +01:00
Johanna Amann
8de178d923 Move c$service_violation to deprecated-dpd-log.zeek
This moves c$service_violation to the deprecated-dpd-log policy script.

This is the only script in the distribution that uses the field, and it
is unlikely to be used externally. It is also responsible for a
significant amount of memory use by itself.

This also restores the field being populated, which was broken in
GH-4362
2025-07-30 07:58:36 +01:00
Vern Paxson
47bf6af6a5 fixes for specialized ZAM operations needing to check whether record fields exist 2025-07-30 08:36:04 +02:00
Johanna Amann
a22b45c69e Merge remote-tracking branch 'origin/topic/johanna/gh-4202'
* origin/topic/johanna/gh-4202:
  Update NEWS for Conn::set_conn changes
  DNS-fuzzer: raise new_connection event
  Optimize Conn::set_conn to minimize operations
  Move Conn::set_conn() from connection_state_remove to new_connection
2025-07-29 21:01:51 +01:00
Arne Welzel
bda70067ec cluster: Move cluster::detail::Event to cluster::Event
This class is a parameter of virtual methods of the Backend API for users
to implement and also a parameter to the HookPublishEvent() API. Seems it
shouldn't be in detail and instead we should own it.

Alternatively, could mark the cluster APIs as not-stable-yet, but I
think we can move forward and make it non-detail for 8.0.
2025-07-29 18:13:59 +02:00
Arne Welzel
cd7836dda2 Merge remote-tracking branch 'origin/topic/awelzel/4431-zeromq-drop-policy-v2'
* origin/topic/awelzel/4431-zeromq-drop-policy-v2:
  cluster.bif: Improve Cluster::publish() docstring
  btest/cluster/zeromq: Add tests for overload behavior
  cluster/zeromq: Metric for msg errors
  cluster/zeromq: Drop events when overloaded
  cluster/zeromq: Comments and move lookups to InitPostScript()
  cluster/zeromq: Rework lambdas to member functions
  cluster/zeromq: Support local XPUB/XSUB hwm and buf configurability
  cluster/OnLoop: Support DontBlock and Force flags for queueing
  cluster/ThreadedBackend: Injectable OnLoopProcess instance
2025-07-29 11:38:49 +02:00
Arne Welzel
c8307487d1 btest/cluster/zeromq: Add tests for overload behavior
The overload-drop.zeek and overload-no-drop.zeek tests have proxy,
worker-1 and worker-2 publish to the manager topic. For the drop
case, we verify that both, the senders, but also the manager drops
events. For the no-drop test, the HWMs are set such that all events
are buffered.

The overload-worker-proxy-topic*.zeek tests are similar, but instead
of publishing to the manager topic, proxy, worker-1 and worker-2 publish
to the proxy and worker topics to overload each other. This had
previously resulted in lockups and these tests verify that this doesn't
happen anymore.
2025-07-29 11:23:53 +02:00
Arne Welzel
12518e8256 Merge remote-tracking branch 'origin/topic/awelzel/expose-num-packets-unprocessed'
* origin/topic/awelzel/expose-num-packets-unprocessed:
  ConnStats: Expose num_packets_unprocessed
  packet_analysis/Manager: Rename GetUnprocessedCount() to PacketsUnprocessed()
2025-07-29 10:12:46 +02:00
Johanna Amann
83d5243cf6 Optimize Conn::set_conn to minimize operations
Now that Conn::set_conn is guaranteed to be called at the beginning and
at the end of the connection, we can skip re-setting the elements that
we know will not have changed. This prevents repeated lookups, e.g. to
check that addresses are in the local networks.

During `connection_state_remove`, only the duration, number of
packets, service, and history fields are updated.

local_orig and local_resp are updated when the connection is flipped. A
test was added for that purpose. It uses the already existing
http.zeek-image-post-1080-8000-x.pcap, which was slightly rewritten for
this, so that one side of the connection has IP addresses different from
127.0.0.1.

The existing history-flip test also was updated to have one side being
in a local-net, to check that the flipping of local_orig and local_resp
works correctly at the beginning of a connection.
2025-07-29 09:11:57 +01:00
Johanna Amann
3e4f67e67c Move Conn::set_conn() from connection_state_remove to new_connection
c$conn is often needed for connection events, but it being established
in connection_state_removed can be a problem because event handlers have
to remember to call Con::set_conn().

This commit moves to call Conn::set_conn() in new_connection.

Addresses GH-4202

update logs
2025-07-29 09:11:57 +01:00
Tim Wojtulewicz
22fc57a90a Merge remote-tracking branch 'origin/topic/johanna/gh-4656'
* origin/topic/johanna/gh-4656:
  Fix parsing of EDNS rcode
2025-07-28 14:16:24 -07:00
Tim Wojtulewicz
e2a702f1ab Expand coverage of IRC analyzer with more commands 2025-07-28 14:13:55 -07:00
Tim Wojtulewicz
a33f100880 Add support for ircv3 tags
This also replaces the irc-dcc-send.trace file in our pcap library with
a cleaner one that includes ircv3 tags.
2025-07-28 14:06:33 -07:00
Arne Welzel
120f061bcd ConnStats: Expose num_packets_unprocessed
Not sure it's the best place to put, but we don't have packet analysis stats
bif and also num_packets is already there, so seems reasonable to put the
num_packets_unprocessed into
2025-07-28 14:10:53 +02:00
Tim Wojtulewicz
d5db359772 Add process_start_time_seconds and process_pid metrics 2025-07-24 10:39:14 -07:00
Johanna Amann
383dce9343 Fix parsing of EDNS rcode
The EDNS rcode was incorrectly calculated. The extended rcode is formed
by taking the upper 8 bits of the extended rcode field, plus the lower 4
bits of the existing rcode.

This also adds a new trace with an extended rcode, and a testcase
parsing it.

Reported by dwhitemv25.

Fixes GH-4656
2025-07-24 14:57:00 +01:00
Arne Welzel
785bb2ee13 conn_key/fivetuple: Handle one-way ICMP conns in DoConnKeyFromVal()
When a conn_id represents a ICMP "connection", we need to determine the
is_one_way flag for InitTuple() in order to skip any flipping of address
and ports for one-way ICMP connections.

Fixes #4645
2025-07-24 09:48:33 +02:00
Johanna Amann
4399f171ae Merge remote-tracking branch 'origin/topic/johanna/pppoe-session-id-logging'
* origin/topic/johanna/pppoe-session-id-logging:
  Update external tests for pppoe-session-id conn.log changes
  PPPoE: add session id logging
2025-07-24 07:57:18 +01:00
Tim Wojtulewicz
146837fe73 Fix swapped storage metrics names 2025-07-23 13:14:46 -07:00
Tim Wojtulewicz
7e3ed2010d Add flag to force synchronous mode when calling storage script-land functions 2025-07-23 13:14:34 -07:00
Johanna Amann
9ab7b768c6 Update external tests for pppoe-session-id conn.log changes 2025-07-23 14:09:17 +01:00
Johanna Amann
e5a434c392 PPPoE: add session id logging
This adds a new PacketAnalyzer::PPPoE::session_id bif, which extracts
the PPPoE session ID from the current packet.

Furthermore, a new policy script is added which adds the pppoe session
id to the connection log.

Related to GH-4602
2025-07-23 13:43:45 +01:00