This adds a new lookup_connection_analyzer_id() BiF to find a given connection's
numeric identifier for a given protocol analyzer (as defined by the underlying
Analyzer::id_counter).
This enables users to call disable_analyzer(), which requires a numeric analyzer
ID, outside of analyzer_confirmation_info and analyzer_violation_info events
handlers.
Like traditional file analyzers, we now query Zeek's
`get_file_handle()` event for handles when a connection begins
analyzing an embedded file. That means that Spicy-side protocol
analyzers that are forwarding data into file analysis now need to call
Zeek's `Files::register_protocol()` and provide a callback for
computing file handles. If that's missing, Zeek will now issue a
warning. This aligns with the requirements Zeek's traditional protocol
analyzers. (If the EVT file defines a protocol analyzer to `replace`
an existing one, that one's `register_protocol()` will be consulted.)
Because Zeek's `get_file_handle()` event requires a current
connection, if a Spicy file analyzer isn't directly part of a
connection context (e.g., with nested files), we continue to use
hardcoded, built-in file handle. Scriptland won't be consulted in
that case, just like before.
Closes#3440.
* topic/christian/localversion:
Parse and store localversion string
Remove commented-out code
Check ZEEK_VERSION_LOCAL for dashes
Update version string btests for localversion
Modify version parsing for localversion
Update version used by spicyz
Update build script
Support for configurable localversion
* origin/topic/awelzel/move-iso-9660-sig-to-policy:
signatures/iso-9660: Add \x01 suffix to CD001
test-all-policy: Do not load iso-9660.zeek
signatures: Move ISO 9660 signature to policy
The previous "fix" caused significant performance degradation without
the signature ever having a chance to trigger. Moving it to policy
seems the best compromise, the alternative being outright removing it.
* origin/topic/johanna/netcontrol-updates:
Netcontrol: add rule_added_policy
Netcontrol: more logging in catch-and-release
Netcontrol: allow supplying explicit name to Debug plugin
This introduces a new hook into the Intel::seen() function that allows
users to directly interact with the result of a find() call via external
scripts.
This should solve the use-case brought up by @chrisanag1985 in
discussion #3256: Recording and acting on "no intel match found".
@Canon88 was recently asking on Slack about enabling HTTP logging for a
given connection only when an Intel match occurred and found that the
Intel::match() event would only occur on the manager. The
Intel::match_remote() event might be a workaround, but possibly running a
bit too late and also it's just an internal "detail" event that might not
be stable.
Another internal use case revolved around enabling packet recording
based on Intel matches which necessarily needs to happen on the worker
where the match happened. The proposed workaround is similar to the above
using Intel::match_remote().
This hook also provides an opportunity to rate-limit heavy hitter intel
items locally on the worker nodes, or even replacing the event approach
currently used with a customized approach.
* origin/topic/awelzel/3424-http-upgrade-websocket-v1:
websocket: Handle breaking from WebSocket::configure_analyzer()
websocket: Address review feedback for BinPac code
fuzzers: Add WebSocket fuzzer
websocket: Fix crash for fragmented messages
websocket: Verify Sec-WebSocket-Key/Accept headers and review feedback
btest/websocket: Test for coalesced reply-ping
HTTP/CONNECT: Also weird on extra data in reply
HTTP/Upgrade: Weird when more data is available
ContentLine: Add GetDeliverStreamRemainingLength() accessor
HTTP: Drain event queue after instantiating upgrade analyzer
btest/http: Explain switching-protocols test change as comment
WebSocket: Introduce new analyzer and log
HTTP: Add mechanism to instantiate Upgrade analyzer
OSS-Fuzz managed to produce a MIME multipart message construction with
thousands of nested entities (or that's what Zeek makes out of it anyhow).
Prevent such deep analysis by capping at a nesting depth of 100,
preventing unnecessary resource usage. A new weird named exceeded_mime_max_depth
is reported when this limit is reached.
This change reduces the runtime of the OSS-Fuzz reproducer from ~45 seconds
to ~2.5 seconds.
The test PCAP was produced from a Python script using the email package
and sending the rendered version via POST to a HTTP server.
Closes#208
* origin/topic/awelzel/3504-ldap-logs-scalars:
Update external baselines
ldap: Use scalar values in logs where appropriate
ldap: Rename LDAP::search_result to LDAP::search_result_entry
Skimming through the RFC, the previous approach of having containers for most
fields seems unfounded for normal protocol operation. The new weirds could just
as well be considered protocol violations. Outside of duplicated or missed data
they just shouldn't happen for well-behaved client/server behavior.
Additionally, with non-conformant traffic it would be trivial to cause
unbounded state growth and immense log record sizes.
Unfortunately, things have become a bit clunky now.
Closes#3504
While it seems interesting functionality, this hasn't been documented,
maintained or knowingly leveraged for many years.
There are various other approaches today, too:
* We track the number of event handler invocations regardless of
profiling. It's possible to approximate a load_sample event by
comparing the result of two get_event_stats() calls. Or, visualize
the corresponding counters in a Prometheus setup to get an idea of
event/s broken down by event names.
* HookCallFunction() allows to intercept script execution, including
measuring the time execution takes.
* The global call_stack and g_frame_stack can be used from plugins
(and even external processes) to walk the Zeek script stack at certain
points to implement a sampling profiler.
* USDT probes or more plugin hooks will likely be preferred over Zeek
builtin functionality in the future.
Relates to #3458
This change allows to specify a per signature specific event, overriding
the default signature_match event. It further removes the message
parameter from such events if not provided in the signature.
This also tracks the message as StringValPtr directly to avoid
allocating the same StringVal for every DoAction() call.
Closes#3403
* origin/topic/neverlord/avoid-openssl-header-deps:
Integrate review feedback
Avoid OpenSSL header dependencies
Added NEWS during merge and removed never set OPENSSL_INCLUDE_DIR reference
in main CMakeLists.txt.
* origin/topic/awelzel/pattern-tables-reuse-compile-set:
TableVal: Replace raw subnets/pattern_matcher with unique_ptr
TablePatternMatcher: Drop Insert()/Remove(), use Clear()
TableType: Simplify and inline Is...Index tests
NEWS: Add small table[pattern] section
Expr/Val: Add support for in set[pattern]
zeek.bif: Implement table_pattern_matcher_stats() bif for introspection
DFA: Extract inner Stats struct from DFA_State_Cache
Expr: Implement string in table[pattern] of X
RE: Remove RE_DisjunctiveMatcher and re-use MatchAll()
TablePatternMatcher: Use const StringValPtr& instead of const StringVal*
Val: Move TablePatternMatcher into detail namespace
TablePatternMatcher: Use unique_ptr
IndexType: Add IsPatternIndex(), like IsSubNetIndex()
btest: Add test for pattern tables and when
Reuse CompileSet() instead of || string formatting
script optimization support for using strings to index table[pattern] values
BTests for indexing "table[pattern] of T" with strings
support for indexing "table[pattern] of T" with strings to get multi-matches
Anchors within pattern passed to sub() or gsub() were previously ignored,
replacing any occurrence of '<text>' even when '^<text>' was used as a
pattern.
This is a pretty user-visible change (and we even have anchored patterns
within the base scripts), but seems "the right thing to do".
Relates to #3455
* origin/topic/awelzel/3455-do-split-string-2:
strings.bif/do_split_string: Pass bol and eol to MatchPrefix()
RE_Matcher: Add MatchPrefix with bol/eol control
For deeply encapsulated connections (think AWS traffic mirroring format
like IP,UDP,GENEVE,IP,UDP,VXLAN,ETH,IP,TCP), the Dispatcher::Lookup()
method is fairly visible in profiles when running in bare mode.
This changes the Analyzer::Lookup() and Dispatcher::Lookup() return value
breaking the API in favor of the performance improvement.
Relates to zeek/zeek#3379.
The ssl_history field may grow unbounded (e.g., ssl_alert event). Prevent this
by capping using a configurable limit (default 100) and raise a weird once reached.
Limit the number of events raised from an SSL record with content_type
alert (21) to a configurable maximum number (default 10). For TLS 1.3,
the limit is set to 1 as specified in the RFC. Add a new weird cases
where the limit is exceeded.
OSS-Fuzz managed to generate a reproducer that raised ~660k ssl_plaintext
and ssl_alert events given ~810kb of input data. This change prevents this
with hopefully no negative side-effect in the real-world.
