Commit graph

4584 commits

Author SHA1 Message Date
Vern Paxson
4b39e1f5f5 shift ZAM maintenance monitoring of new BiFs to a BTest so it automatically runs 2024-04-25 09:15:12 -07:00
Vern Paxson
c11c2830b1 performance speed-up for SMB base scripts 2024-04-25 09:15:12 -07:00
Benjamin Bannier
3c89a5492c Fix typo in hook name
While in Spicy code a hook priority is spelled `priority=4711` the
attribute is still called `&priority` (like in HILTI) and we rely on
exactly that name when e.g., extracting hook priorities for scheduling.

This change was introduced as part of
db98dc4193 and caused the default hook
priority for hooks defined in EVT files (intended to be -1000 to likely
schedule after e.g., hooks in the Spicy grammars) to be ignored. This
could then e.g., introduce issue when a `%done` hook would mutate state
exposed in an EVT hook (which now might not have seen the updated state
due to different scheduling).
2024-04-25 13:43:09 +02:00
Tim Wojtulewicz
bac6ad62b5 Allow SMB_TCP record to contain multiple protocol identifiers/headers 2024-04-22 15:55:25 -07:00
Tim Wojtulewicz
5aa18287f1 Merge remote-tracking branch 'origin/topic/timw/3456-when-unknown-capture'
* origin/topic/timw/3456-when-unknown-capture:
  Check for valid identifier in WhenStmt::Describe
2024-04-22 15:16:16 -07:00
Tim Wojtulewicz
d566961da5 Merge remote-tracking branch 'pbcullen/topic/pbcullen/remove_exclude'
* pbcullen/topic/pbcullen/remove_exclude:
  Update core.pcap.filter-warning baseline
  Add PacketFilter::remove_exclude function
2024-04-18 09:03:57 -07:00
Peter Cullen
f1941186eb Update core.pcap.filter-warning baseline
Only the line numbers changed.
2024-04-18 12:27:46 +00:00
Peter Cullen
187f93cb96 Parse and store localversion string
Add localversion to the VersionDescription record and populate it
during version string parsing.
This change also modifies the version string syntax, removing the
deprecated dash (-) between beta|dev|rc and the commmit count; those
must now be separated by a period.
The test version strings were updated accordingly along with the
baseline.
2024-04-17 14:17:22 -07:00
Peter Cullen
32289521f5 Update version string btests for localversion
Added some more version strings to be tested that include
a localversion component. Baseline updates, as well.
2024-04-17 14:17:10 -07:00
Tim Wojtulewicz
9abf8ea8b7 Check for valid identifier in WhenStmt::Describe 2024-04-10 16:20:36 -07:00
Vern Paxson
0538c68221 ZAM-specific BTest baseline changes for tweak to how ZAM bodies print 2024-03-19 10:59:49 -07:00
Vern Paxson
378627448d BTest baseline changes for tweak to how ZAM bodies print 2024-03-19 10:59:49 -07:00
Tim Wojtulewicz
9de1dd16d5 Remove variant from StdFunctionStmt
The variant ended up conflicting with std::bind, which resulted in failures
on the btest invoking it. Change back to a single function that takes a
flow, and default it to a value in Exec.
2024-03-19 09:07:45 -07:00
Tim Wojtulewicz
39890fda5e Add a method to register an event handler to a std::function via C++ 2024-03-18 16:36:25 -07:00
Arne Welzel
62928ddb61 btest/spicy: Remove most port usages 2024-03-15 17:36:11 +01:00
Arne Welzel
54f9b76f8e spicy: Deprecate port/ports in .evt files 2024-03-15 17:36:11 +01:00
Arne Welzel
2d1588277e Merge remote-tracking branch 'origin/topic/awelzel/3640-void-size-expr-assign-stmt'
* origin/topic/awelzel/3640-void-size-expr-assign-stmt:
  Expr: Handle TYPE_VOID in SizeExpr and AssignExpr::Typecheck()
2024-03-12 10:31:46 +01:00
Vern Paxson
99f43d3c8b AST location fixes for -O gen-C++ 2024-03-08 19:09:46 +01:00
Vern Paxson
54877e9dec fixed bug where appending an empty vector to a vector-of-vector's had no effect 2024-03-08 19:09:43 +01:00
Arne Welzel
8cb1a1518f Expr: Handle TYPE_VOID in SizeExpr and AssignExpr::Typecheck()
@vpax reported surprising behavior when working with "void values".
While these are not exposed to script land, plumb the places he
pointed out are causing confusing behavior.

Closes #3640.
2024-03-07 11:24:28 +01:00
Robin Sommer
732a3e1d9a
Bump Spicy. 2024-03-07 09:08:44 +01:00
Jan Grashoefer
08daef8d42 Make worker restart in cluster btest race-proof 2024-03-05 23:10:18 +01:00
Arne Welzel
6867eda621 Merge remote-tracking branch 'origin/topic/awelzel/fix-assert-cond-twice'
* origin/topic/awelzel/fix-assert-cond-twice:
  Stmt: Fix assert evaluating cond twice
2024-03-04 18:22:43 +01:00
Robin Sommer
db98dc4193
Port Spicy integration to new AST API. 2024-03-04 15:48:39 +01:00
Arne Welzel
6572324b8c Stmt: Fix assert evaluating cond twice
Since 81a9745fb3, the assert condition is
evaluated twice. This leads to unexpected behavior when cond has a side
effect like publishing a message or creating a log stream or filter.

Found while using the following in ad-hoc testing code and wondering
why two messages were published.

    assert publish(Cluster::worker_topic, hello, "abc")
2024-03-04 15:30:09 +01:00
Arne Welzel
d728981b77 Merge remote-tracking branch 'origin/topic/awelzel/move-iso-9660-sig-to-policy'
* origin/topic/awelzel/move-iso-9660-sig-to-policy:
  signatures/iso-9660: Add \x01 suffix to CD001
  test-all-policy: Do not load iso-9660.zeek
  signatures: Move ISO 9660 signature to policy
2024-02-26 21:16:48 +01:00
Arne Welzel
e87272f5a7 Merge remote-tracking branch 'origin/topic/awelzel/3490-global-ids-type-name-change'
* origin/topic/awelzel/3490-global-ids-type-name-change:
  global_ids: Align script_id$type_name field with type_name()
2024-02-26 21:15:25 +01:00
Arne Welzel
77d57fccfd Merge remote-tracking branch 'origin/topic/awelzel/find-in-zeekpath'
* origin/topic/awelzel/find-in-zeekpath:
  NEWS: Mention can_load() and find_in_zeekpath()
  utils: Introduce packages.zeek with can_load() helper
  zeek.bif: Add find_in_zeekpath() helper
2024-02-26 21:14:59 +01:00
Arne Welzel
e11c20e1eb test-all-policy: Do not load iso-9660.zeek
Changing the default_file_bof_buffer_size has subtle impact on
MIME type detection and changed the zeek-testing baseline. Do
not load this new script via test-all-policy to avoid this.

The new test was mainly an aid to understand what is actually going on.
In short, if default_file_bof_buffer_size is larger than the file MIME
detection only runs when the buffer is full, or when the file is removed.
When a file transfer happens over multiple HTTP connections, only
some or one of the http.log entries will have a proper response MIME type.

PCAP extracted from 2009-M57-day11-18.trace.gz.
2024-02-26 17:58:26 +01:00
Arne Welzel
1a5ce65e3d signatures: Move ISO 9660 signature to policy
The previous "fix" caused significant performance degradation without
the signature ever having a chance to trigger. Moving it to policy
seems the best compromise, the alternative being outright removing it.
2024-02-26 13:35:23 +01:00
Arne Welzel
1c2d1b3717 global_ids: Align script_id$type_name field with type_name()
Populate script_id$type_name with what the type_name() bif
would produce for the same identifier.

Closes #3490
2024-02-23 17:26:56 +01:00
Johanna Amann
c81274b549 Update Mozilla CA and Google CT list and related tests.
Test updates are necessary due to removed CT logs.

Fixes GH-3620
2024-02-23 11:50:31 +00:00
Arne Welzel
d2409dd432 signatures: Fix ISO 9960 signature
This signature only really works when default_file_bof_buffer_size is bumped
to a sufficient value (40k).
2024-02-22 12:37:40 +01:00
Arne Welzel
036bcfe919 Merge remote-tracking branch 'origin/topic/awelzel/ftp-clear-fuid-after-logging'
* origin/topic/awelzel/ftp-clear-fuid-after-logging:
  testing: Update zeek-testing-private commit hash
  ftp: Reset fuid after logging
2024-02-22 12:23:21 +01:00
Arne Welzel
947294efab utils: Introduce packages.zeek with can_load() helper 2024-02-22 11:41:50 +01:00
Arne Welzel
704f75a214 zeek.bif: Add find_in_zeekpath() helper
Relates to #3594. This helper can be used to determine the path that
will be used by @load, if at all.
2024-02-22 11:33:03 +01:00
Arne Welzel
2f1893bc58 Allow delete statement for tables, sets and vectors
Relates to #3472. This allow "delete tbl" as an alternative for
clear_table(tbl). Also works for vectors.
2024-02-21 18:00:00 +01:00
Arne Welzel
31b548babc ftp: Reset fuid after logging
A user reported being confused about the fuid association of subsequent
FTP commands when a data transfer has completed. It seems reasonable to
unset fuid upon logging a FTP command which had a fuid.

The current behavior results in the PORT or PASV commands after a RETR or STOR
to have the fuid of the prior file transfer. Similarly, any CWD or DEL commands
following a file transfer will unnecessarily be logged with the fuid of the
prior file transfer.

This tickles the baselines for the private testing PCAP a lot, primarily
because there data connections in that pcap are never established properly.
E.g, the fuids FzDzid1Dxm9srVKHXf and FEfYX73q5C6GEQZXX9 have been re-used
for multiple commands.

This may look like we're losing information, but the fuids vanishing
in the normal btests belong to a LIST command that isn't logged by
default into ftp.log. If it was, the fuid would be attached to it.
2024-02-21 12:41:32 +01:00
Tim Wojtulewicz
6d86a48a6a Start of 7.0.0 development 2024-02-20 11:51:06 -07:00
Dominik Charousset
0a84aad417 Pull in variant-enabled Broker
Switch the Broker submodule to a version with `broker::variant` enabled
and provide new overloads for `broker::variant` in the Broker manager.
2024-02-20 09:47:32 -07:00
Arne Welzel
c1a685a05d websocket: Add Spicy parser version, too.
The Spicy analyzer is added as a child analyzer when enabled and the
WebSocket.cc logic dispatches between the BinPac and Spicy version.

It substantially slower when tested against a somewhat artificial
2.4GB PCAP. The first flamegraph indicates that the unmask() function
stands out with 35% of all samples, and above it shared_ptr samples.
2024-02-06 17:29:55 +01:00
Johanna Amann
160ccda45f Merge remote-tracking branch 'origin/topic/johanna/netcontrol-updates'
* origin/topic/johanna/netcontrol-updates:
  Netcontrol: add rule_added_policy
  Netcontrol: more logging in catch-and-release
  Netcontrol: allow supplying explicit name to Debug plugin
2024-02-06 11:54:26 +00:00
Johanna Amann
737f828bcb Netcontrol: add rule_added_policy
rule_added_policy allows the modification of rules just after they have
been added. This allows the implementation of some more complex features
- like changing rule states depending on insertion in other plugins.
2024-02-05 18:52:27 +00:00
Johanna Amann
979d43eac0 Netcontrol: more logging in catch-and-release
Catch-and-release logs now include the plugin that is responsible for an
action. Furthermore, the catch-and-release log also includes instances
where a rule already existed, and where an error occurred during an
operation.
2024-02-05 14:38:21 +00:00
Vern Paxson
9e32443504 minor updates and fixes for script optimization maintenance 2024-02-04 07:57:45 -08:00
Arne Welzel
1c516f738c quic: Log client's source connection id, too.
Seem reasonable give we log the server SCID. Interestingly, the Chromium
examples actually have zero length (empty) source connection IDs. I wonder
if that's part of their "protocol ossification avoidance" effort.
2024-01-30 21:46:38 +01:00
Arne Welzel
33acfdc630 quic: Continue decryption attempts for later INITIAL packets
The original logic stopped decrypting any INITIAL packets after the
first. The Firefox/cloudflare pcaps actually show that the server
replies with a QUIC INITAL packet containing just ACK frames and no
CRYPTO frames. Only the second QUIC INITIAL packet from the server
then contains the CRYPTO frames.

There's no good reason to stop decryption attempts, either we succeed
down the road and then stop, or we fail and raise analyzer violations.
2024-01-30 21:46:38 +01:00
Robin Sommer
1837e0289a
Bump Spicy to current main.
This requires a few baseline updates, and a tweak to one test
canonifier.
2024-01-29 09:59:46 +01:00
Christian Kreibich
173f5e7137 Move GeoIP availability test in btests to zeek-config --have-geoip 2024-01-26 14:27:19 -08:00
Christian Kreibich
1044f4db0d Add btest for succeeding/failing IPv4/IPv6 lookups
The existing tests happen to test only succeeding IP addresses.
2024-01-25 23:54:20 -08:00