Commit graph

5230 commits

Author SHA1 Message Date
Arne Welzel
a318463c1c cluster/zeromq: Improve EINTR handling
When using ZeroMQ also within the Supervisor process, zmq::poll() and
recv() were observed to return EINTR, handle these.
2025-09-25 13:52:12 +02:00
Vern Paxson
dc7f4dca1c BTests & baselines for testing selective skipping of script optimization 2025-09-23 09:05:40 -07:00
Arne Welzel
b3b0c3db7d utils/decompose_uri: Support URIs containing IPv6 addresses
An URI containing a bracketed or non-bracketed IPv6 address of the form
http://[::1]:42 was previously split on the first colon for port extraction,
causing a subsequent to_count() call to fail. Harden this to check for a
digits in the last :[0-9]+ component.

Fixes #4842
2025-09-23 13:04:29 +02:00
Tim Wojtulewicz
50581bfd3c Avoid starting up storage expiration thread with zero backends 2025-09-15 14:18:03 -07:00
Arne Welzel
9e7537044f btest/core/suspend_processing: Add WebSocket example
Add a test/example forwarding all new_connection() events produced during
`zeek -r wikipedia.trace` as my_new_connection() to a WebSocket client.

This is mostly to demonstrate and verify usage of suspend_processing(),
websocket_client_added(), resume_processing(), Pcap::file_done(),
websocket_client_lost() and terminate() together.
2025-09-08 11:11:13 -07:00
Tim Wojtulewicz
7142b9efde Merge remote-tracking branch 'klemensya/svcparams'
* klemensya/svcparams:
  Parse SVCB/HTTPS SvcParams list
2025-09-03 15:36:03 -07:00
Tim Wojtulewicz
e3b22cd21f Merge remote-tracking branch 'origin/topic/vern/emphasize-IDPtrs'
* origin/topic/vern/emphasize-IDPtrs:
  fixup! fixup! shift much of the internal use of ID* identifier pointers over to IDPtr objects
  fixup! shift much of the internal use of ID* identifier pointers over to IDPtr objects
  fixup! shift much of the internal use of ID* identifier pointers over to IDPtr objects
  fixup! shift much of the internal use of ID* identifier pointers over to IDPtr objects
  annotate a number of BTests as unsuited for -O gen-C++ testing due to multiple Zeek runs
  BTest baseline updates for -O gen-C++ - all minor tweaks
  BTest updates for script optimization tracking of BiFs
  regression test for former ASAN issue with script optimization of lambdas
  shift much of the internal use of ID* identifier pointers over to IDPtr objects
  maintenance update for script optimization's knowledge of BiFs
  logger fix for interoperability with -O gen-C++ code
2025-09-03 15:33:43 -07:00
Evan Typanski
42f7a2d88c Merge remote-tracking branch 'origin/topic/vern/field-assign-parsing'
* origin/topic/vern/field-assign-parsing:
  tightened up parsing of $field=X record constructor expressions
2025-09-03 16:21:24 -04:00
Vern Paxson
b6b8484819 annotate a number of BTests as unsuited for -O gen-C++ testing due to multiple Zeek runs 2025-09-03 11:19:38 -07:00
Vern Paxson
77bb32ff78 BTest baseline updates for -O gen-C++ - all minor tweaks 2025-09-03 11:19:38 -07:00
Vern Paxson
1b4bc72802 BTest updates for script optimization tracking of BiFs 2025-09-03 11:19:38 -07:00
Vern Paxson
e9971e31d5 regression test for former ASAN issue with script optimization of lambdas 2025-09-03 11:19:38 -07:00
Benjamin Bannier
07dd2a1c6f Merge branch 'topic/bbannier/test-bittorrent-unnessary-sigs' 2025-09-03 14:51:16 +02:00
Klemens Nanni
397f7e5c0e Parse SVCB/HTTPS SvcParams list
Add full support for RFC 9460's SvcParams list.

Amend the existing `dns_svcb_rr` record by a vector of new
`dns_svcb_param` records containing aptly typed SvcParamKey and
SvcParamValue pairs.  Example output:

```
@load base/protocols/dns
event dns_HTTPS( c: connection , msg: dns_msg , ans: dns_answer , https: dns_svcb_rr ) {
	for (_, param in https$svc_params)
		print to_json(param);  # filter uninitialised values
}
```

```
$ dig https cloudflare-ech.com +short | tr [:space:] \\n
1
.
alpn="h3,h2"
ipv4hint=104.18.10.118,104.18.11.118
ech=AEX+DQBBHgAgACBGL2e9TiFwjK/w1Zg9AmRm7mgXHz3PjffP0mTFNMxmDQAEAAEAAQASY2xvdWRmbGFyZS1lY2guY29tAAA=
ipv6hint=2606:4700::6812:a76,2606:4700::6812:b76
```

```
{"key":1,"alpn":["h3","h2"]}
{"key":4,"hint":["104.18.10.118","104.18.11.118"]}
{"key":5,"ech":"AEX+DQBBHgAgACBGL2e9TiFwjK/w1Zg9AmRm7mgXHz3PjffP0mTFNMxmDQAEAAEAAQASY2xvdWRmbGFyZS1lY2guY29tAAA="}
{"key":6,"hint":["2606:4700::6812:a76","2606:4700::6812:b76"]}
```

Values with malformed data or belonging to invalid/reserved keys
are passed raw bytes in network order for script-level inspection.

Follow up to "Initial Support to DNS SVCB/HTTPS RR"
https://github.com/zeek/zeek/pull/1808
2025-09-03 09:53:55 +03:00
Tim Wojtulewicz
aa3940f16d Use std::numbers::pi instead of hard-coded value 2025-09-02 16:51:23 +00:00
Benjamin Bannier
4c9cbeb43f Remove unnecessary peer signature from test scripts.base.protocols.bittorrent.tracker 2025-09-02 10:53:38 +02:00
Vern Paxson
36862a18d6 tightened up parsing of $field=X record constructor expressions 2025-08-31 15:14:26 -07:00
Arne Welzel
9a7678f15a cluster/Backend: Fallback to current network time when current event has not timestamp
When a WebSocket client sends an event to Zeek without explicit network
timestamp metadata, Zeek would use -1.0 as a timestamp for any events
published while handling this event. Instead, it seems far more sensible
to use the current network time in that scenario.
2025-08-20 19:56:50 +02:00
Arne Welzel
01fe022e07 cluster/serializer/broker: Do not send empty metadata vectors around
Event when there's no metadata attached to an event, we'd still use the
constructor passing an empty metadata vector, resulting in an on-the-wire
representation with an empty trailing vector.

Particularly visible when just snooping events via websocat. There also
seems to be some bug with the timestamp -1 handling.
2025-08-20 19:55:38 +02:00
Evan Typanski
59e84e06f6 Merge remote-tracking branch 'origin/topic/etyp/fix-optional-attr-errors'
* origin/topic/etyp/fix-optional-attr-errors:
  Fix parameter attributes pretending to be records
  Only allow `&optional` in records
2025-08-18 14:49:56 -04:00
Arne Welzel
2c9015d247 Merge branch 'master' of https://github.com/blightzero/zeek
* 'master' of https://github.com/blightzero/zeek:
  Changed behavior of var-extraction-uri.zeek from policy/protocol/http to extract only the URI parameter names. Do not include the path in the first parameter name. Only extract uri vars if parameters actually exist.
2025-08-18 13:14:34 +02:00
Arne Welzel
e04f725523 Merge remote-tracking branch 'amazing-pp/t/psql-login-no-role'
* amazing-pp/t/psql-login-no-role:
  Report PostgreSQL login success only after ReadyForQuery
2025-08-18 09:40:22 +02:00
Fupeng Zhao
e4e56789db
Report PostgreSQL login success only after ReadyForQuery
Previously, Zeek treated the receipt of `AuthenticationOk` as a
successful login. However, according to the PostgreSQL
Frontend/Backend Protocol, the startup phase is not complete until
the server sends `ReadyForQuery`. It is still possible for the server
to emit an `ErrorResponse` (e.g. ERRCODE_INVALID_AUTHORIZATION_SPECIFICATION)
after `AuthenticationOk` but before `ReadyForQuery`.

This change updates the PostgreSQL analyzer to defer reporting login
success until `ReadyForQuery` is observed. This prevents false
positives in cases where authentication succeeds but session startup
fails.
2025-08-18 10:59:44 +08:00
Arne Welzel
c0a863cba0 Merge remote-tracking branch 'origin/topic/vern/stmt-line-numbers'
* origin/topic/vern/stmt-line-numbers:
  maintenance updates for ZAM BiF-tracking
  fix line numbers associated with "if" and initialization statements
2025-08-17 17:29:09 +02:00
Arne Welzel
f57a1263d4 cluster/serializer/broker: Do not special case Broker::Data anymore
The previous approach ignored the fact that nested / inner values might
also be Broker::Data values. I'm not super sure about the validity of
the test, because it's essentially demonstrating any-nesting, but
it's not leading to extra Broker::Data encoding.
2025-08-17 16:56:20 +02:00
Vern Paxson
5b74b3d0ac maintenance updates for ZAM BiF-tracking 2025-08-16 14:10:17 -07:00
Tim Wojtulewicz
e64ec54172
Revert "Move BinPAC, bifcl, af_packet, and gen_zam submodules into main zeek repo" 2025-08-15 15:11:22 -07:00
Tim Wojtulewicz
a1f5f0a40e Fix some clang-tidy findings in generated BIF code 2025-08-15 13:57:27 -07:00
Evan Typanski
4445bc1daf Fix parameter attributes pretending to be records
Parameters relied on is_record for a couple of validations, but they are
not records and should not be treated as such. This way we can validate
&optional better.
2025-08-14 12:00:15 -04:00
Evan Typanski
4e5a56c5e0 Only allow &optional in records
There was some confusing behavior with &optional and locals, so this
should get rid of that by making it an error. However, there is a case
where function parameters are still allowed to have &optional - this is
because there are checks for &default in parameters as well.
2025-08-14 11:38:19 -04:00
Benjamin Grap
fafc0212a5 Changed behavior of var-extraction-uri.zeek from policy/protocol/http to extract only the URI parameter names. Do not include the path in the first parameter name. Only extract uri vars if parameters actually exist. 2025-08-13 22:45:20 +02:00
Tim Wojtulewicz
c8818d76bd Remove length limiting on string fields for HTTP 2025-08-12 17:31:29 -07:00
Tim Wojtulewicz
0ec2161b04 Add options to filter at the stream level as well as globally 2025-08-12 17:31:28 -07:00
Tim Wojtulewicz
339d46ae26 Add a weird that gets emitted when strings/containers are over the limits 2025-08-12 17:31:28 -07:00
Tim Wojtulewicz
837fde1a08 Add metrics to track string and container fields limited by length 2025-08-12 17:31:28 -07:00
Tim Wojtulewicz
e2e7ab28da Implement string- and container-length filtering at the log record level 2025-08-12 17:31:28 -07:00
Tim Wojtulewicz
d95affde4d Remove deprecations tagged for v8.1 2025-08-12 10:19:03 -07:00
Arne Welzel
b2a2ad7e10 smb2/read: Parse only 1 byte for data_offset, ignore reserved1
A user provided a SMB2 pcap with the reserved1 field of a ReadResponse
set to 1 instead of 0. This confused the padding computation due to
including this byte into the offset. Properly split data_offset and
reserved1 into individual byte fields.

Closes #4730
2025-08-08 16:12:20 +02:00
Arne Welzel
13f613eb1d Merge remote-tracking branch 'origin/topic/awelzel/4176-cluster-on-sub-unsub-hooks'
* origin/topic/awelzel/4176-cluster-on-sub-unsub-hooks:
  cluster: Add on_subscribe() and on_unsubscribe() hooks
2025-08-08 14:24:18 +02:00
Tim Wojtulewicz
54d67c3322 Merge remote-tracking branch 'origin/topic/timw/cleanup-warnings-from-plugin-btest-builds'
* origin/topic/timw/cleanup-warnings-from-plugin-btest-builds:
  Update zeek-aux to remove BRO_DIST from plugin skeleton
  cmake_minimum_required() should come before project()
2025-08-07 08:39:40 -07:00
Tim Wojtulewicz
162ecc022e Update zeek-aux to remove BRO_DIST from plugin skeleton 2025-08-07 08:39:08 -07:00
Tim Wojtulewicz
3c535ec215 cmake_minimum_required() should come before project() 2025-08-06 12:10:41 -07:00
Arne Welzel
f98508bbb0 btest/tap-analyzer: Update existing test and add new one for UpdateConnVal()
This also changes the output of connection UIDs from the tap analyzer to be
prefixed with C for easier correlation with other logs.

Relates to #4337 #4725 #4734 #4737
2025-08-06 17:22:59 +02:00
Johanna Amann
2f2f328a72 Merge remote-tracking branch 'origin/topic/johanna/analyzer-log-proto'
* origin/topic/johanna/analyzer-log-proto:
  Add proto to analyzer.log
2025-08-06 14:38:47 +01:00
Evan Typanski
22f77248f5 Merge remote-tracking branch 'origin/topic/etyp/fix-record-vec-type-conflict'
* origin/topic/etyp/fix-record-vec-type-conflict:
  Fix record coercion with compatible types
2025-08-06 09:10:19 -04:00
Arne Welzel
ac776b0aad btest/plugins/tap-analyzer: Update baseline
Relates to #4337 #4725 #4734
2025-08-06 14:17:42 +02:00
Johanna Amann
82266b1e78 Add proto to analyzer.log
The analyzer.log file was missing the protocol field to distinguish
tcp/udp connections.
2025-08-06 11:34:57 +01:00
Arne Welzel
b4925fbd16 TapAnalyzer: More verdict to action rename
Relates to #4725 #4337
2025-08-05 19:59:06 +02:00
Arne Welzel
4bc7f9532c IPBasedAnalyzer: Call TapPacket() when skipping
When skip_further_processing() is called, a TapAnalyzer should still see
the packets as skipped with SkipReason "skipping".
2025-08-05 19:47:04 +02:00
Arne Welzel
dc904b2216 SessionAdapter: Introduce TapAnalyzer for session adapter
This commit introduces a mechanism to attach light weight analyzers to
the root analyzer of sessions in order to tap into the packets delivered
to child analyzer.
2025-08-05 19:47:02 +02:00