Commit graph

18601 commits

Author SHA1 Message Date
Tim Wojtulewicz
da7c3d9138 Merge remote-tracking branch 'origin/topic/bbannier/bump-spicy'
* origin/topic/bbannier/bump-spicy:
  Bump auxil/spicy to latest development snapshot
2024-07-23 08:51:55 -07:00
Johanna Amann
da131fae60 Update Mozilla CA list and CT list 2024-07-23 16:05:30 +01:00
Johanna Amann
7c0c48b290 Spicy TLS - refactoring and partial connection fix
The analyzer now detects partial connections at the beginning of a
connection - and will skip them. This makes behavior more similar to the
binpac analyzer.

The decryption test is skipped.

And some minor refacoring.
2024-07-23 15:34:10 +01:00
Arne Welzel
6a6a5c3d0d Merge remote-tracking branch 'origin/topic/awelzel/ldap-extended-request-response-starttls'
* origin/topic/awelzel/ldap-extended-request-response-starttls:
  ldap: Add heuristic for wrap tokens
  ldap: Ignore ec/rrc for sealed wrap tokens
  ldap: Add LDAP sample with SASL-SRP mechanism
  ldap: Reintroduce encryption after SASL heuristic
  ldap: Fix assuming GSS-SPNEGO for all bindResponses
  ldap: Implement extended request/response and StartTLS support
2024-07-23 12:39:14 +02:00
Benjamin Bannier
dc61cd7a54 Bump auxil/spicy to latest development snapshot 2024-07-23 11:43:01 +02:00
Arne Welzel
d4778f451c ldap: Add heuristic for wrap tokens
Instead of dissecting the GSSAPI handshake, add another heuristic
into MaybeEncrypted to check for the WRAP token identifier.

After this change, the pcap on the following ticket is processed
nicely: https://gitlab.com/wireshark/migration-test/-/issues/9398
2024-07-23 11:29:00 +02:00
Arne Welzel
ca25516e03 ldap: Ignore ec/rrc for sealed wrap tokens
It shouldn't matter for the encrypted payload that we'll
just consume and ignore.
2024-07-23 11:29:00 +02:00
Arne Welzel
a70ccc51ab ldap: Add LDAP sample with SASL-SRP mechanism
This is what @dopheide-esnet actually saw. Produced with a custom
cyrus-sasl and openldap build :-(
2024-07-23 11:29:00 +02:00
Arne Welzel
3846db6ccf ldap: Reintroduce encryption after SASL heuristic
@dopheide-esnet provided sample captures where SASL SRP is used as
a SASL mechanism and the follow-up LDAP messages are encrypted. It's
not clear how to determine whether encryption will or will not happen,
so re-add a heuristic to determine this based on the first byte of
the first message *after* the successful bindResponse handshake. If
that byte is 0x30, assume cleartext.

I haven't been able to produce such pcaps, unfortunately, but the
cleartext path is tested via the existing sasl-ntlm.pcap.
2024-07-23 11:29:00 +02:00
Arne Welzel
724c08f286 ldap: Fix assuming GSS-SPNEGO for all bindResponses
In retrospect that's an obvious bug.
2024-07-23 11:29:00 +02:00
Arne Welzel
09a48c7028 ldap: Implement extended request/response and StartTLS support
PCAP was produced with a local OpenLDAP server configured to support StartTLS.

This puts the Zeek calls into a separate ldap_zeek.spicy file/module
to separate it from LDAP.
2024-07-23 11:29:00 +02:00
Johanna Amann
f95f5d2adb Merge remote-tracking branch 'origin/master' into topic/johanna/spicy-tls
* origin/master:
  Update broker submodule [nomail]
  telemetry: Deprecate prometheus.zeek policy script
  input/Manager: Improve type checks of record fields with type any
  Bump zeek-testing-cluster to pull in tee SIGPIPE fix
  ldap: Remove MessageWrapper with magic 0x30 searching
  ldap: Harden parsing a bit
  ldap: Handle integrity-only KRB wrap tokens
  Bump auxil/spicy to latest development snapshot
  CI: Set FETCH_CONTENT_FULLY_DISCONNECTED flag for configure
  Update broker and cmake submodules [nomail]
  Fix a broken merge
  Do not emit hook files for builtin modules
  Fix warning about grealpath when running 'make dist' on Linux
  Start of 7.1.0 development
  Updating submodule(s) [nomail]
  Update the scripts.base.frameworks.telemetry.internal-metrics test
  Revert "Temporarily disable the scripts/base/frameworks/telemetry/internal-metrics btest"
  Bump Broker to pull in new Prometheus support and pass in Zeek's registry
  Do not emit hook files for builtin modules
2024-07-23 10:21:49 +01:00
Arne Welzel
f4a79fa703 Merge remote-tracking branch 'origin/topic/awelzel/deprecate-prometheus-script'
* origin/topic/awelzel/deprecate-prometheus-script:
  telemetry: Deprecate prometheus.zeek policy script
2024-07-23 10:03:13 +02:00
Tim Wojtulewicz
f500c424fe Update broker submodule [nomail] 2024-07-22 14:59:17 -07:00
Arne Welzel
bf9704f339 telemetry: Deprecate prometheus.zeek policy script
With Cluster::Node$metrics_port being optional, there's not really
a need for the extra script. New rule, if a metrics_port is set, the
node will attempt to listen on it.

Users can still redef Telemetry::metrics_port *after*
base/frameworks/telemetry was loaded to change the port defined
in cluster-layout.zeek.
2024-07-21 17:49:21 +02:00
Arne Welzel
e982a18792 Merge remote-tracking branch 'origin/topic/awelzel/3836-input-create-any-abort-crash'
* origin/topic/awelzel/3836-input-create-any-abort-crash:
  input/Manager: Improve type checks of record fields with type any
2024-07-19 19:51:52 +02:00
Arne Welzel
3797622152 input/Manager: Improve type checks of record fields with type any
Calling AsRecordType() or AsFunc() on a Val of type any isn't safe.

Closes #3836
2024-07-19 11:35:54 +02:00
Christian Kreibich
0ba80d13b4 Merge branch 'topic/christian/bump-cluster-testsuite'
* topic/christian/bump-cluster-testsuite:
  Bump zeek-testing-cluster to pull in tee SIGPIPE fix
2024-07-17 15:37:29 -07:00
Christian Kreibich
b51a46f94d Bump zeek-testing-cluster to pull in tee SIGPIPE fix 2024-07-17 15:34:24 -07:00
Arne Welzel
2ea3a651bd Merge remote-tracking branch 'origin/topic/awelzel/spicy-ldap-krb-wrap-tokens'
* origin/topic/awelzel/spicy-ldap-krb-wrap-tokens:
  ldap: Remove MessageWrapper with magic 0x30 searching
  ldap: Harden parsing a bit
  ldap: Handle integrity-only KRB wrap tokens
2024-07-17 16:48:58 +02:00
Arne Welzel
e7aca5b388 ldap: Remove MessageWrapper with magic 0x30 searching
This unit implements a heuristic to search for the 0x30 sequence
byte if Message couldn't readily be parsed. Remove it with the
idea of explicit and predictable support for SASL mechanisms.
2024-07-17 15:28:27 +02:00
Arne Welzel
0cab87c185 ldap: Harden parsing a bit
ASN1Message(True) may go off parsing arbitrary input data as
"something ASN.1" This could be GBs of octet strings or just very
long sequences. Avoid this by open-coding some top-level types expected.

This also tries to avoid some of the &parse-from usages that result
in unnecessary copies of data.

Adds a locally generated PCAP with addRequest/addResponse that we
don't currently handle.
2024-07-17 15:27:14 +02:00
Arne Welzel
31122f335f ldap: Handle integrity-only KRB wrap tokens
Mostly staring at the PCAPs and opened a few RFCs. For now, only if the
MS_KRB5 OID is used and accepted in a bind response, start stripping
KRB5 wrap tokens for both, client and server traffic.

Would probably be nice to forward the GSS-API data to the analyzer...

Closes zeek/spicy-ldap#29.
2024-07-17 15:26:30 +02:00
Johanna Amann
990c97b5ba Spicy SSL - run spicy-format 2024-07-17 12:51:13 +01:00
Johanna Amann
f36be3dafc Spicy-TLS: address review feedback re convert-functions
This switches convert-functions from being called in the evt file to
being called directly in the spicy file.

See GH-3765 for details.
2024-07-17 12:44:14 +01:00
Tim Wojtulewicz
9ba7c2ddaf Merge remote-tracking branch 'origin/topic/bbannier/bump-spicy'
* origin/topic/bbannier/bump-spicy:
  Bump auxil/spicy to latest development snapshot
2024-07-16 10:16:15 -07:00
Theo Buehler
3ea67a3109 Use accessor to reach into X509_ALGOR
Despite already having an accessor, X509_ALGOR wasn't made opaque
during OpenSSL 1.1.0 development. It would be nice if this could be
fixed at some point, so avoid reaching into that struct by using the
accessor
2024-07-16 18:06:10 +02:00
Benjamin Bannier
41db05238b Bump auxil/spicy to latest development snapshot
This patch bump Spicy to the latest development snapshot. This
introduces a backwards-incompatible change in that it removes support
for a never officially supported syntax to specify unit fields (so I
would argue: not strictly a breaking change).
2024-07-16 08:17:31 +02:00
Tim Wojtulewicz
cb56837f07 Merge remote-tracking branch 'origin/topic/timw/fetchcontent_mkdir'
* origin/topic/timw/fetchcontent_mkdir:
  CI: Set FETCH_CONTENT_FULLY_DISCONNECTED flag for configure
  Update broker and cmake submodules [nomail]
2024-07-12 16:02:37 -07:00
Tim Wojtulewicz
4757536d7d CI: Set FETCH_CONTENT_FULLY_DISCONNECTED flag for configure 2024-07-12 15:14:31 -07:00
Tim Wojtulewicz
a46be1ea29 Update broker and cmake submodules [nomail] 2024-07-12 15:14:31 -07:00
Tim Wojtulewicz
f3bcf1a55d Fix a broken merge
I merged an old version of the branch on accident and then merged the right
one over top of it, but git ended up including both versions. This fixes
that mistake.
2024-07-12 10:00:13 -07:00
Tim Wojtulewicz
b935d2f59a Merge remote-tracking branch 'origin/topic/bbannier/lib-spicy-hooks'
* origin/topic/bbannier/lib-spicy-hooks:
  Do not emit hook files for builtin modules
2024-07-12 09:51:55 -07:00
Tim Wojtulewicz
7a38cee81f Merge remote-tracking branch 'origin/topic/bbannier/lib-spicy-hooks'
* origin/topic/bbannier/lib-spicy-hooks:
  Do not emit hook files for builtin modules
2024-07-12 09:48:12 -07:00
Tim Wojtulewicz
e4716b6c91 Merge remote-tracking branch 'origin/topic/timw/grealpath-make-dist-warning'
* origin/topic/timw/grealpath-make-dist-warning:
  Fix warning about grealpath when running 'make dist' on Linux
2024-07-12 09:46:51 -07:00
Benjamin Bannier
24d3454d61 Do not emit hook files for builtin modules
We would previously emit a C++ file with hooks for at least the builtin
`spicy` module even though that module like any other builtin module
never contains implementations of hooks for types in user code.

This patch prevents modules with skipped implementations (such as our
builtin modules) from being added to the compilation which prevents
generating their hook files.
2024-07-12 09:06:12 +02:00
Tim Wojtulewicz
5fd563da79 Fix warning about grealpath when running 'make dist' on Linux 2024-07-11 13:45:14 -07:00
Tim Wojtulewicz
822102382c Start of 7.1.0 development 2024-07-11 13:19:15 -07:00
Tim Wojtulewicz
f124b31904 Updating submodule(s) [nomail] 2024-07-11 12:21:02 -07:00
Christian Kreibich
c4bc9078ef Merge branch 'topic/christian/broker-prometheus-cpp'
* topic/christian/broker-prometheus-cpp:
  Update the scripts.base.frameworks.telemetry.internal-metrics test
  Revert "Temporarily disable the scripts/base/frameworks/telemetry/internal-metrics btest"
  Bump Broker to pull in new Prometheus support and pass in Zeek's registry
2024-07-11 12:15:16 -07:00
Christian Kreibich
77816f9a6b Update the scripts.base.frameworks.telemetry.internal-metrics test
This now uses different record fields, and for now we no longer have CAF
telemetry. We indicate we're running under test to get reliable ordering in the
baselined output.
2024-07-11 12:13:05 -07:00
Christian Kreibich
b387da3489 Revert "Temporarily disable the scripts/base/frameworks/telemetry/internal-metrics btest"
This reverts commit d6e97ab306.
Broker's telemetry is now available again.
2024-07-11 12:13:05 -07:00
Dominik Charousset
40e1e43c25 Bump Broker to pull in new Prometheus support and pass in Zeek's registry
Co-authored-by: Christian Kreibich <christian@corelight.com>
2024-07-11 12:13:05 -07:00
Johanna Amann
4acd0297b0 Merge remote-tracking branch 'origin/master' into topic/johanna/spicy-tls
* origin/master: (89 commits)
  Update doc submodule [nomail] [skip ci]
  Bump cmake submodule [nomail]
  testing/btest: Default to HILTI_JIT_PARALLELISM=1
  Revert "CI: Use ccache and a single CPU when building spicy analyzers for btests"
  Update doc submodule [nomail] [skip ci]
  CI: Use ccache and a single CPU when building spicy analyzers for btests
  Extend btest for logging of disabled analyzers
  Update zeek-aux submodule [nomail]
  Add logging of disabled analyzers to analyzer.log
  Bump auxil/spicy to latest development snapshot
  Management framework: bump cluster testsuite to pull in telemetry tests
  Management framework: bump zeek-client
  Management framework: augment deployed configs with instance IP addresses
  Management framework: add auto-enumeration of metrics ports
  Management framework: propagate metrics port from agent
  Management framework: add metrics port in management & Supervisor node records
  Harden the telemetry manager against unset Telemetry::metrics_address
  Comment-only tweaks for telemetry-related settings.
  Fix for --display-cmake in configure Moved build directory creation further down in the script so that --display-cmake has a chance to happen before build tree setup.
  Update submodules [nomail]
  ...
2024-07-11 14:02:28 +01:00
Benjamin Bannier
596f4114fc Do not emit hook files for builtin modules
We would previously emit a C++ file with hooks for at least the builtin
`spicy` module even though that module like any other builtin module
never contains implementations of hooks for types in user code.

This patch adds a blocklist of builtin modules which are skipped for
generating hook files.
2024-07-11 14:15:21 +02:00
zeek-bot
e99b94c18f Update doc submodule [nomail] [skip ci] 2024-07-11 00:10:52 +00:00
Tim Wojtulewicz
3f475ebf7d Bump cmake submodule [nomail] 2024-07-10 12:37:21 -07:00
Arne Welzel
a9c4daaa22 Merge remote-tracking branch 'jgras/topic/jgras/log-disable-analyzer'
* jgras/topic/jgras/log-disable-analyzer:
  Extend btest for logging of disabled analyzers
  Add logging of disabled analyzers to analyzer.log
2024-07-10 18:51:26 +02:00
Arne Welzel
1c8fedaf09 Merge remote-tracking branch 'origin/topic/awelzel/move-hilti-jit-parallelism-to-btest-cfg'
* origin/topic/awelzel/move-hilti-jit-parallelism-to-btest-cfg:
  testing/btest: Default to HILTI_JIT_PARALLELISM=1
  Revert "CI: Use ccache and a single CPU when building spicy analyzers for btests"
2024-07-10 11:06:46 +02:00
Arne Welzel
7b99fc01a9 testing/btest: Default to HILTI_JIT_PARALLELISM=1
This is a rework of b59bed9d06 moving
HILTI_JIT_PARALLELISM=1 into btest.cfg to make it default applicable to
btest -j users (and CI).

The background for this change is that spicyz may spawn up to nproc compiler
instances by default. Combined with btest -j, this may be nproc x nproc
instances worst case. Particularly with gcc, this easily overloads CI or
local systems, putting them into hard-to-recover-from thrashing/OOM states.

Exporting HILTI_JIT_PARALLELISM in the shell allows overriding.
2024-07-10 11:04:47 +02:00