The analyzer now detects partial connections at the beginning of a
connection - and will skip them. This makes behavior more similar to the
binpac analyzer.
The decryption test is skipped.
And some minor refacoring.
Instead of dissecting the GSSAPI handshake, add another heuristic
into MaybeEncrypted to check for the WRAP token identifier.
After this change, the pcap on the following ticket is processed
nicely: https://gitlab.com/wireshark/migration-test/-/issues/9398
@dopheide-esnet provided sample captures where SASL SRP is used as
a SASL mechanism and the follow-up LDAP messages are encrypted. It's
not clear how to determine whether encryption will or will not happen,
so re-add a heuristic to determine this based on the first byte of
the first message *after* the successful bindResponse handshake. If
that byte is 0x30, assume cleartext.
I haven't been able to produce such pcaps, unfortunately, but the
cleartext path is tested via the existing sasl-ntlm.pcap.
PCAP was produced with a local OpenLDAP server configured to support StartTLS.
This puts the Zeek calls into a separate ldap_zeek.spicy file/module
to separate it from LDAP.
* origin/master:
Update broker submodule [nomail]
telemetry: Deprecate prometheus.zeek policy script
input/Manager: Improve type checks of record fields with type any
Bump zeek-testing-cluster to pull in tee SIGPIPE fix
ldap: Remove MessageWrapper with magic 0x30 searching
ldap: Harden parsing a bit
ldap: Handle integrity-only KRB wrap tokens
Bump auxil/spicy to latest development snapshot
CI: Set FETCH_CONTENT_FULLY_DISCONNECTED flag for configure
Update broker and cmake submodules [nomail]
Fix a broken merge
Do not emit hook files for builtin modules
Fix warning about grealpath when running 'make dist' on Linux
Start of 7.1.0 development
Updating submodule(s) [nomail]
Update the scripts.base.frameworks.telemetry.internal-metrics test
Revert "Temporarily disable the scripts/base/frameworks/telemetry/internal-metrics btest"
Bump Broker to pull in new Prometheus support and pass in Zeek's registry
Do not emit hook files for builtin modules
This unit implements a heuristic to search for the 0x30 sequence
byte if Message couldn't readily be parsed. Remove it with the
idea of explicit and predictable support for SASL mechanisms.
ASN1Message(True) may go off parsing arbitrary input data as
"something ASN.1" This could be GBs of octet strings or just very
long sequences. Avoid this by open-coding some top-level types expected.
This also tries to avoid some of the &parse-from usages that result
in unnecessary copies of data.
Adds a locally generated PCAP with addRequest/addResponse that we
don't currently handle.
Mostly staring at the PCAPs and opened a few RFCs. For now, only if the
MS_KRB5 OID is used and accepted in a bind response, start stripping
KRB5 wrap tokens for both, client and server traffic.
Would probably be nice to forward the GSS-API data to the analyzer...
Closeszeek/spicy-ldap#29.
This patch bump Spicy to the latest development snapshot. This
introduces a backwards-incompatible change in that it removes support
for a never officially supported syntax to specify unit fields (so I
would argue: not strictly a breaking change).
* origin/master: (89 commits)
Update doc submodule [nomail] [skip ci]
Bump cmake submodule [nomail]
testing/btest: Default to HILTI_JIT_PARALLELISM=1
Revert "CI: Use ccache and a single CPU when building spicy analyzers for btests"
Update doc submodule [nomail] [skip ci]
CI: Use ccache and a single CPU when building spicy analyzers for btests
Extend btest for logging of disabled analyzers
Update zeek-aux submodule [nomail]
Add logging of disabled analyzers to analyzer.log
Bump auxil/spicy to latest development snapshot
Management framework: bump cluster testsuite to pull in telemetry tests
Management framework: bump zeek-client
Management framework: augment deployed configs with instance IP addresses
Management framework: add auto-enumeration of metrics ports
Management framework: propagate metrics port from agent
Management framework: add metrics port in management & Supervisor node records
Harden the telemetry manager against unset Telemetry::metrics_address
Comment-only tweaks for telemetry-related settings.
Fix for --display-cmake in configure Moved build directory creation further down in the script so that --display-cmake has a chance to happen before build tree setup.
Update submodules [nomail]
...
When Zeek flips roles of a HTTP connection subsequent to the HTTP analyzer
being attached, that analyzer would not update its own ContentLine analyzer
state, resulting in the wrong ContentLine analyzer being switched into
plain delivery mode.
In debug builds, this would result in assertion failures, in production
builds, the HTTP analyzer would receive HTTP bodies as individual header
lines, or conversely, individual header lines would be delivered as a
large chunk from the ContentLine analyzer.
PCAPs were generated locally using tcprewrite to select well-known-http ports
for both endpoints, then editcap to drop the first SYN packet.
Kudos to @JordanBarnartt for keeping at it.
Closes#3789
This reverts part of commit a0888b7e36 due
to inhibiting analyzer violations when parsing non SSH traffic when
the &restofdata path is entered.
@J-Gras reported the analyzer not being disabled when sending HTTP
traffic on port 22.
This adds the verbose analyzer.log baselines such that future improvements
of these scenarios become visible.
Spicy SSL is now only enabled when specifying the --enable-spicy-ssl
configure-time option.
This should allow merging this into Zeek on an experimental basis.
* origin/master: (173 commits)
Bump Spicy to latest dev snapshot
Update docs submodule [nomail] [skip ci]
Add type aliases for instrument and family shared_ptrs
Update NEWS for double and is_sum changes
Remove is_sum arguments from counters and gauges
Change all instruments to only handle doubles
Add comment to telemetry::Manager::InitPostScript
Remove all of the ZEEK_METRICS_ environment variables
Fix header comments in scripts/policy/frameworks/telemetry/prometheus.zeek
Change all prometheus #includes to use angle brackets
Update zeekctl submodule for metrics_port node.cfg option
Regenerate docs [nomail]
Remove the is_sum argument from BIF histogram creation methods
Update NEWS for Telemetry rework
Remove Telemetry::metrics_export_prefixes option
Validate that label names are constant in non-zeek metrics
Avoid calling Collect() in counter/gauge Value() methods if not needed
Fix some determinism issues with btests
Temporarily disable the scripts/base/frameworks/telemetry/internal-metrics btest
Fix the scripts.policy.frameworks.telemetry.prometheus btest to use the service discovery endpoint
...
* origin/topic/robin/gh-3561-forward-to-udp:
Update docs.
Add explicit children life-cycle management method to analyzers.
Spicy: Support UDP in Spicy's `protocol_*` runtime functions.
Add method to analyzer to retrieve direct child by name.
Extend PIA's `FirstPacket` API.
Spicy: Prepare for supporting forwarding to protocols other than TCP.
`FirstPacket()` so far supported only TCP. To extend this to UDP, we
move the method into the PIA base class; give it a protocol parameter
for the case that there's no actual packet is available; and add the
ability to create fake UDP packets as well, not just TCP.
This whole thing is pretty ugly to begin with, and this doesn't make
it nicer, but we need this extension that so we can feed UDP data into
the signature engine that's tunneled over other protocols. Without the
fake packets, DPD signatures in particular wouldn't have anything to
match on.
When a specific component is requested through its tag or name, one
can now have the component manager transparently return a different
one that has been registered to replace the original one. We limit
this to disabled components to avoid unnecessary confusion. That also
means that remappings are currently only supported for analyzers
(because other types of components cannot be disabled for now, per the
previous change).
The different analyzers types all had their own methods for
enabling/disabling their availability. This change abstracts that into
a new API inside their base class (`plugin::Component`) so that they
can be toggled in a unified way.
In principle, other types of components could/should use this as well
now, so that, e.g., an input reader's availability could be toggled at
runtime. The code doesn't make that broader change for now because it
would requires a series of changes wherever these other component
types are being used. However, that means that one now could try
toggling some other component through the new API without that having
any effect. To catch that, there's a runtime check in place that turns
any such attempt into an internal error.
* origin/master: (352 commits)
Bump Spicy.
Remove support for old Spicy versions from QUIC analyzer.
Make sure that vcpkg isn't preferred if pcap_root_dir is passed in
Remove some unused Spicy state.
Bump Spicy.
ZAM fix for concretizing vectors in record constructors
improve ZAM's estimation of profiling overheads
CI: Remove commented openssl 1.1 workaround from macOS preparation script
CI: Fix installation of python package on macOS
Address review feedback for configure error change
Raise configure error message for unsupported archives
fix ZAM "cat" of doubles/times to include trailing ".0" per normal BiF behavior
CI: Specify the xcode version of the macOS Sonoma instance
Remove vestigial Conan bit in CMakeLists.txt
When configuring Spicy, be prepated for zeek_lib or zeek_exe targets.
Fix a typo in CMakeLists.txt when building Zeek as a library
Bump Spicy to current `main`.
tie into updates to gen-zam
ZAM documentation updated to reflect finer-grained profiling
ZAM-specific BTest baseline changes for tweak to how ZAM bodies print
...
The Spicy analyzer is added as a child analyzer when enabled and the
WebSocket.cc logic dispatches between the BinPac and Spicy version.
It substantially slower when tested against a somewhat artificial
2.4GB PCAP. The first flamegraph indicates that the unmask() function
stands out with 35% of all samples, and above it shared_ptr samples.
The original logic stopped decrypting any INITIAL packets after the
first. The Firefox/cloudflare pcaps actually show that the server
replies with a QUIC INITAL packet containing just ACK frames and no
CRYPTO frames. Only the second QUIC INITIAL packet from the server
then contains the CRYPTO frames.
There's no good reason to stop decryption attempts, either we succeed
down the road and then stop, or we fail and raise analyzer violations.
A continuation frame has the same type as the first frame, but that
information wasn't used nor kept, resulting payload of continuation
frames not being forwarded. The pcap was created with a fake Python
server and a bit of message crafting.
