Commit graph

4894 commits

Author SHA1 Message Date
Arne Welzel
e85a12f286 btest/files/x509/files: Sort analyzers in baseline
This test is very sensitive to new enum values that perturb the final
order of the output. Listing the commits that modify that file in the
past clearly shows changes to the baseline happen when a new component
with an enum value is introduced.

    40f04d4ccf broker: Add WebSocketShim backend
    51836d08ae protocol: Add StreamEvent analyzer
    fdde1e9841 cluster/serializer: Add binary-serialization-format
    6fb73aa9da broker: Add shim plugin adding a backend component
    85ca59484b postgresql: Initial parser implementation
    efc2681152 WebSocket: Introduce new analyzer and log
    ...

That's just distraction. Point fix the test to sort the analyzers before
writing the final log line.

I've previously attempted this in a more complete fashion in PR #3365
(issue #3364), but seems this direct fix might just make more sense.

(cherry picked from commit 2f6c31df89)
2025-07-02 18:45:47 +02:00
Arne Welzel
b76a75d86e Merge remote-tracking branch 'origin/topic/awelzel/4035-btest-openssl-sha1-certs'
* origin/topic/awelzel/4035-btest-openssl-sha1-certs:
  external/subdir-btest.cfg: Set OPENSSL_ENABLE_SHA1_SIGNATURES=1
  btest/x509_verify: Drop OpenSSL 1.0 hack
  testing/btest: Use OPENSSL_ENABLE_SHA1_SIGNATURES

(cherry picked from commit 280e7acc6e)
2025-05-19 11:18:20 -07:00
Arne Welzel
94700130ed Merge remote-tracking branch 'origin/topic/vern/zam-aggr-change-in-loop'
* origin/topic/vern/zam-aggr-change-in-loop:
  fix for ZAM optimization when an aggregate is modified inside of a loop

(cherry picked from commit 2255fa23b8)
2025-05-19 09:16:10 -07:00
Tim Wojtulewicz
37e7b57664 Update quic baselines due to service ordering 2025-05-06 10:09:16 -07:00
Arne Welzel
ceb798b42a Merge remote-tracking branch 'origin/topic/awelzel/4275-ldap-gss-spnego-auth-miss'
* origin/topic/awelzel/4275-ldap-gss-spnego-auth-miss:
  ldap: Clean up from code review
  ldap: Add Sicily Authentication constants
  ldap: Only switch into MS_KRB5 mode if responseToken exists

(cherry picked from commit a2a535d0c9)
2025-05-06 09:46:49 +02:00
Arne Welzel
ec18da8baa Merge remote-tracking branch 'origin/topic/awelzel/4405-quic-fragmented-crypto'
* origin/topic/awelzel/4405-quic-fragmented-crypto:
  Bump external/zeek-testing
  QUIC: Extract reset_crypto() function
  QUIC: Rename ConnectionIDInfo to Context
  QUIC: Switch initial_destination_conn_id to optional
  QUIC: Use initial destination conn_id for decryption
  QUIC: Handle CRYPTO frames across multiple INITIAL packets
  QUIC: Do not consume EncryptedLongPacketPayload
  QUIC: Fix ACK frame parsing

(cherry picked from commit 50ac8d1468)
2025-05-05 12:56:53 -07:00
Tim Wojtulewicz
a041080e3f Update core/vntag-in-vlan baseline to remove ip_proto field for 7.0 2025-03-18 17:03:05 -07:00
Tim Wojtulewicz
ed081212ae Merge remote-tracking branch 'origin/topic/timw/vntag-in-vlan'
* origin/topic/timw/vntag-in-vlan:
  Add analyzer registration from VLAN to VNTAG

(cherry picked from commit cb5e3d0054)
2025-03-18 16:18:13 -07:00
Arne Welzel
ec04c925a0 Merge remote-tracking branch 'origin/topic/awelzel/2311-load-plugin-bare-mode'
* origin/topic/awelzel/2311-load-plugin-bare-mode:
  scan.l: Fix @load-plugin scripts loading
  scan.l: Extract switch_to() from load_files()
  ScannedFile: Allow skipping canonicalization

(cherry picked from commit a3a08fa0f3)
2025-03-18 16:16:39 -07:00
Arne Welzel
de8127f3cd Merge remote-tracking branch 'origin/topic/awelzel/4198-4201-quic-maintenance'
* origin/topic/awelzel/4198-4201-quic-maintenance:
  QUIC/decrypt_crypto: Rename all_data to data
  QUIC: Confirm before forwarding data to SSL
  QUIC: Parse all QUIC packets in a UDP datagram
  QUIC: Only slurp till packet end, not till &eod

(cherry picked from commit 44304973fb)
2025-03-18 16:15:34 -07:00
Arne Welzel
b5774f2de9 Merge remote-tracking branch 'origin/topic/vern/ZAM-field-assign-in-op'
* origin/topic/vern/ZAM-field-assign-in-op:
  pre-commit: Bump spicy-format to 0.23
  fix for ZAM optimization of assigning a record field to result of "in" operation

(cherry picked from commit 991bc9644d)
2025-03-18 16:13:01 -07:00
Arne Welzel
c2f2388f18 QUIC/decrypt_crypto: Actually check if decryption was successful
...and bail if it wasn't.

PCAP was produced using OSS-Fuzz input from issue 383379789.
2024-12-13 13:10:45 -07:00
Arne Welzel
f54416eae4 Merge remote-tracking branch 'origin/topic/christian/fix-zam-analyzer-name'
* origin/topic/christian/fix-zam-analyzer-name:
  Fix ZAM's implementation of Analyzer::name() BiF

(cherry picked from commit e100a8e698)
2024-12-12 13:14:10 -07:00
Arne Welzel
68bfe8d1c0 Merge remote-tracking branch 'origin/topic/vern/zam-exception-leaks'
* origin/topic/vern/zam-exception-leaks:
  More robust memory management for ZAM execution - fixes #4052

(cherry picked from commit c3b30b187e)
2024-12-12 13:05:13 -07:00
Arne Welzel
43ab74b70f Merge branch 'sqli-spaces-encode-to-plus' of https://github.com/cooper-grill/zeek
* 'sqli-spaces-encode-to-plus' of https://github.com/cooper-grill/zeek:
  account for spaces encoding to plus signs in sqli regex detection

(cherry picked from commit 5200b84fb3)
2024-11-19 09:33:22 -07:00
Arne Welzel
887d92e26c Merge remote-tracking branch 'upstream/topic/awelzel/3774-skip-script-args-test-under-tsan'
* upstream/topic/awelzel/3774-skip-script-args-test-under-tsan:
  btest: Skip core.script-args under TSAN

(cherry picked from commit 159f40a4bf)
2024-11-14 19:07:51 -07:00
Tim Wojtulewicz
b1fec3284e Disable core.expr-execption btest under ZAM to fix CI builds 2024-11-14 16:04:41 -07:00
Tim Wojtulewicz
5ce0f2edb6 Fix ubsan warning with ZAM and mmdb btest 2024-11-14 13:14:58 -07:00
Arne Welzel
056b70bd2d Merge remote-tracking branch 'origin/topic/awelzel/community-id-new-connection'
* origin/topic/awelzel/community-id-new-connection:
  policy/community-id: Populate conn$community_id in new_connection()

(cherry picked from commit d3579c1f34)
2024-11-14 12:15:27 -07:00
Arne Welzel
3ebe867193 Merge branch 'modbus-fixes' of https://github.com/zambo99/zeek
* 'modbus-fixes' of https://github.com/zambo99/zeek:
  Prevent non-Modbus on port 502 to be reported as Modbus

(cherry picked from commit 4763282f36)
2024-11-14 11:32:17 -07:00
Christian Kreibich
300b7a11ac Merge branch 'topic/awelzel/3957-raw-reader-spinning'
* topic/awelzel/3957-raw-reader-spinning:
  input/Raw: Rework GetLine()

(cherry picked from commit 2a23e9fc19)
2024-11-14 11:30:55 -07:00
Vern Paxson
3281aa6284 import of GH-4022 BTest additions
ZAM baseline update
2024-11-14 10:19:07 -07:00
Christian Kreibich
ea44c30272 Merge remote-tracking branch 'security/topic/awelzel/215-pop3-mail-null-deref'
* security/topic/awelzel/215-pop3-mail-null-deref:
  POP3: Rework unbounded pending command fix

(cherry picked from commit 7fea32c6edc5d4d14646366f87c9208c8c9cf555)
2024-10-04 10:46:40 -07:00
Tim Wojtulewicz
88c37d0be8 Merge remote-tracking branch 'origin/topic/awelzel/3936-pop3-and-redis'
* origin/topic/awelzel/3936-pop3-and-redis:
  pop3: Remove unused headers
  pop3: Prevent unbounded state growth
  btest/pop3: Add somewhat more elaborate testing

(cherry picked from commit 702fb031a4)
2024-09-23 11:12:54 -07:00
Johanna Amann
40db8463df Merge remote-tracking branch 'origin/topic/timw/remove-negative-timestamp-test'
* origin/topic/timw/remove-negative-timestamp-test:
  Remove core.negative-time btest

(cherry picked from commit 899f7297d7)
2024-09-23 10:27:19 -07:00
Arne Welzel
5a0e2bf771 Merge remote-tracking branch 'origin/topic/awelzel/3919-ldap-logs-missing'
* origin/topic/awelzel/3919-ldap-logs-missing:
  btest/ldap: Add regression test for #3919

(cherry picked from commit a339cfa4c0)
2024-09-23 09:24:52 -07:00
Arne Welzel
95e7c5a63e Merge remote-tracking branch 'origin/topic/awelzel/3853-ldap-spnego-ntlmssp'
* origin/topic/awelzel/3853-ldap-spnego-ntlmssp:
  ldap: Recognize SASL+SPNEGO+NTLMSSP

(cherry picked from commit 152bbbd680)
2024-09-23 09:23:19 -07:00
Robin Sommer
15be682f63 Merge remote-tracking branch 'origin/topic/robin/gh-3881-spicy-ports'
* origin/topic/robin/gh-3881-spicy-ports:
  Spicy: Register well-known ports through an event handler.
  Revert "Remove deprecated port/ports fields for spicy analyzers"

(cherry picked from commit a2079bcda6)
2024-08-30 13:26:16 -07:00
Arne Welzel
6f65b88f1b Merge remote-tracking branch 'origin/topic/awelzel/ldap-extended-request-response-starttls'
* origin/topic/awelzel/ldap-extended-request-response-starttls:
  ldap: Add heuristic for wrap tokens
  ldap: Ignore ec/rrc for sealed wrap tokens
  ldap: Add LDAP sample with SASL-SRP mechanism
  ldap: Reintroduce encryption after SASL heuristic
  ldap: Fix assuming GSS-SPNEGO for all bindResponses
  ldap: Implement extended request/response and StartTLS support

(cherry picked from commit 6a6a5c3d0d)
2024-08-30 11:47:08 -07:00
Arne Welzel
cfe47f40a4 Merge remote-tracking branch 'origin/topic/awelzel/spicy-ldap-krb-wrap-tokens'
* origin/topic/awelzel/spicy-ldap-krb-wrap-tokens:
  ldap: Remove MessageWrapper with magic 0x30 searching
  ldap: Harden parsing a bit
  ldap: Handle integrity-only KRB wrap tokens

(cherry picked from commit 2ea3a651bd)
2024-08-30 11:46:47 -07:00
Arne Welzel
0fd6672dde Merge branch 'fix-http-password-capture' of https://github.com/p-l-/zeek
* 'fix-http-password-capture' of https://github.com/p-l-/zeek:
  http: fix password capture when enabled

(cherry picked from commit c27e18631c)
2024-08-30 11:34:24 -07:00
Tim Wojtulewicz
dd4597865a Merge remote-tracking branch 'origin/topic/timw/telemetry-threading'
* origin/topic/timw/telemetry-threading:
  Process metric callbacks from the main-loop thread

(cherry picked from commit 3c3853dc7d)
2024-08-30 11:29:17 -07:00
Christian Kreibich
3a44bda957 Bump zeek-testing-cluster to reflect deprecation of prometheus.zeek
(cherry picked from commit 146cf99ff6)
2024-07-24 17:07:14 -07:00
Tim Wojtulewicz
a4b746e5e8 Merge remote-tracking branch 'origin/topic/timw/smb2-ioctl-errors'
* origin/topic/timw/smb2-ioctl-errors:
  Update 7.0 NEWS with blurb about multi-PDU parsing causing increased load [nomail] [skip ci]
  Fix handling of zero-length SMB2 error responses

(cherry picked from commit bd208f4c54)
2024-07-24 13:29:09 -07:00
Arne Welzel
8014c4b8c3 telemetry: Deprecate prometheus.zeek policy script
With Cluster::Node$metrics_port being optional, there's not really
a need for the extra script. New rule, if a metrics_port is set, the
node will attempt to listen on it.

Users can still redef Telemetry::metrics_port *after*
base/frameworks/telemetry was loaded to change the port defined
in cluster-layout.zeek.

(cherry picked from commit bf9704f339)
2024-07-23 10:05:46 +02:00
Christian Kreibich
d17a1f9822 Bump zeek-testing-cluster to pull in tee SIGPIPE fix
(cherry picked from commit b51a46f94d)
2024-07-17 15:39:45 -07:00
Christian Kreibich
c4bc9078ef Merge branch 'topic/christian/broker-prometheus-cpp'
* topic/christian/broker-prometheus-cpp:
  Update the scripts.base.frameworks.telemetry.internal-metrics test
  Revert "Temporarily disable the scripts/base/frameworks/telemetry/internal-metrics btest"
  Bump Broker to pull in new Prometheus support and pass in Zeek's registry
2024-07-11 12:15:16 -07:00
Christian Kreibich
77816f9a6b Update the scripts.base.frameworks.telemetry.internal-metrics test
This now uses different record fields, and for now we no longer have CAF
telemetry. We indicate we're running under test to get reliable ordering in the
baselined output.
2024-07-11 12:13:05 -07:00
Christian Kreibich
b387da3489 Revert "Temporarily disable the scripts/base/frameworks/telemetry/internal-metrics btest"
This reverts commit d6e97ab306.
Broker's telemetry is now available again.
2024-07-11 12:13:05 -07:00
Arne Welzel
a9c4daaa22 Merge remote-tracking branch 'jgras/topic/jgras/log-disable-analyzer'
* jgras/topic/jgras/log-disable-analyzer:
  Extend btest for logging of disabled analyzers
  Add logging of disabled analyzers to analyzer.log
2024-07-10 18:51:26 +02:00
Arne Welzel
7b99fc01a9 testing/btest: Default to HILTI_JIT_PARALLELISM=1
This is a rework of b59bed9d06 moving
HILTI_JIT_PARALLELISM=1 into btest.cfg to make it default applicable to
btest -j users (and CI).

The background for this change is that spicyz may spawn up to nproc compiler
instances by default. Combined with btest -j, this may be nproc x nproc
instances worst case. Particularly with gcc, this easily overloads CI or
local systems, putting them into hard-to-recover-from thrashing/OOM states.

Exporting HILTI_JIT_PARALLELISM in the shell allows overriding.
2024-07-10 11:04:47 +02:00
Jan Grashoefer
c6c8d078c0 Extend btest for logging of disabled analyzers 2024-07-09 20:15:46 +02:00
Tim Wojtulewicz
c557b2156a Merge remote-tracking branch 'origin/topic/vern/script-opt-maint.Jul24'
* origin/topic/vern/script-opt-maint.Jul24:
  minor script optimization updates to reflect recent changes, Coverity findings
2024-07-09 10:15:26 -07:00
Christian Kreibich
cdd5062f45 Management framework: bump cluster testsuite to pull in telemetry tests 2024-07-08 23:05:24 -07:00
Vern Paxson
a53e28be87 minor script optimization updates to reflect recent changes, Coverity findings 2024-07-06 16:44:27 -07:00
Arne Welzel
377fd711bd HTTP: Implement FlipRoles()
When Zeek flips roles of a HTTP connection subsequent to the HTTP analyzer
being attached, that analyzer would not update its own ContentLine analyzer
state, resulting in the wrong ContentLine analyzer being switched into
plain delivery mode.

In debug builds, this would result in assertion failures, in production
builds, the HTTP analyzer would receive HTTP bodies as individual header
lines, or conversely, individual header lines would be delivered as a
large chunk from the ContentLine analyzer.

PCAPs were generated locally using tcprewrite to select well-known-http ports
for both endpoints, then editcap to drop the first SYN packet.

Kudos to @JordanBarnartt for keeping at it.

Closes #3789
2024-07-04 11:38:33 +02:00
Christian Kreibich
737b1a2013 Remove the Supervisor's internal ClusterEndpoint struct.
This eliminates one place in which we currently need to mirror changes to the
script-land Cluster::Node record. Instead of keeping an exact in-core equivalent, the
Supervisor now treats the data structure as opaque, and stores the whole cluster
table as a JSON string.

We may replace the script-layer Supervisor::ClusterEndpoint in the future, using
Cluster::Node directly. But that's a more invasive change that will affect how
people invoke Supervisor::create() and similars.

Relying on JSON for serialization has the side-effect of removing the
Supervisor's earlier quirk of using 0/tcp, not 0/unknown, to indicate unused
ports in the Supervisor::ClusterEndpoint record.
2024-07-02 14:52:17 -07:00
Christian Kreibich
0179a5e75c Support JSON roundtripping via to_json()/from_json() for patterns
This needed a small tweak in the deserialization, since each roundtrip
would otherwise pad the prior pattern with an extra /^?(...)$?/.

This expands the language.set test to also verify serializing/unserializing for
sets, similarly to tables in the previous commit.
2024-07-02 14:46:16 -07:00
Christian Kreibich
92c1098e97 Support table deserialization in from_json()
This allows additional data roundtripping through JSON since to_json() already
supports tables. There are some subtleties around the formatting of strings in
JSON object keys, for which this adds a bit of helper infrastructure.

This also expands the language.table test to verify the roundtrips, and adapts
bif.from_json to include a table in the test record.
2024-07-02 14:46:16 -07:00
Christian Kreibich
df645e9bb2 Support map-based definition of ports in from_json()
The from_json() BiF and its underlying code in Val.cc currently expect ports
expressed as a string ('80/tcp' etc). Zeek's own serialization via ToJSON()
renders them as an object ('{"port":80, "proto":"tcp"}'). This adds support
for the latter format to from_json(), so serialized values can be read back.
2024-07-02 14:46:16 -07:00