* origin/topic/awelzel/3504-ldap-logs-scalars:
Update external baselines
ldap: Use scalar values in logs where appropriate
ldap: Rename LDAP::search_result to LDAP::search_result_entry
Skimming through the RFC, the previous approach of having containers for most
fields seems unfounded for normal protocol operation. The new weirds could just
as well be considered protocol violations. Outside of duplicated or missed data
they just shouldn't happen for well-behaved client/server behavior.
Additionally, with non-conformant traffic it would be trivial to cause
unbounded state growth and immense log record sizes.
Unfortunately, things have become a bit clunky now.
Closes#3504
While it seems interesting functionality, this hasn't been documented,
maintained or knowingly leveraged for many years.
There are various other approaches today, too:
* We track the number of event handler invocations regardless of
profiling. It's possible to approximate a load_sample event by
comparing the result of two get_event_stats() calls. Or, visualize
the corresponding counters in a Prometheus setup to get an idea of
event/s broken down by event names.
* HookCallFunction() allows to intercept script execution, including
measuring the time execution takes.
* The global call_stack and g_frame_stack can be used from plugins
(and even external processes) to walk the Zeek script stack at certain
points to implement a sampling profiler.
* USDT probes or more plugin hooks will likely be preferred over Zeek
builtin functionality in the future.
Relates to #3458
This change allows to specify a per signature specific event, overriding
the default signature_match event. It further removes the message
parameter from such events if not provided in the signature.
This also tracks the message as StringValPtr directly to avoid
allocating the same StringVal for every DoAction() call.
Closes#3403
* origin/topic/neverlord/avoid-openssl-header-deps:
Integrate review feedback
Avoid OpenSSL header dependencies
Added NEWS during merge and removed never set OPENSSL_INCLUDE_DIR reference
in main CMakeLists.txt.
* origin/topic/awelzel/pattern-tables-reuse-compile-set:
TableVal: Replace raw subnets/pattern_matcher with unique_ptr
TablePatternMatcher: Drop Insert()/Remove(), use Clear()
TableType: Simplify and inline Is...Index tests
NEWS: Add small table[pattern] section
Expr/Val: Add support for in set[pattern]
zeek.bif: Implement table_pattern_matcher_stats() bif for introspection
DFA: Extract inner Stats struct from DFA_State_Cache
Expr: Implement string in table[pattern] of X
RE: Remove RE_DisjunctiveMatcher and re-use MatchAll()
TablePatternMatcher: Use const StringValPtr& instead of const StringVal*
Val: Move TablePatternMatcher into detail namespace
TablePatternMatcher: Use unique_ptr
IndexType: Add IsPatternIndex(), like IsSubNetIndex()
btest: Add test for pattern tables and when
Reuse CompileSet() instead of || string formatting
script optimization support for using strings to index table[pattern] values
BTests for indexing "table[pattern] of T" with strings
support for indexing "table[pattern] of T" with strings to get multi-matches
Anchors within pattern passed to sub() or gsub() were previously ignored,
replacing any occurrence of '<text>' even when '^<text>' was used as a
pattern.
This is a pretty user-visible change (and we even have anchored patterns
within the base scripts), but seems "the right thing to do".
Relates to #3455
* origin/topic/awelzel/3455-do-split-string-2:
strings.bif/do_split_string: Pass bol and eol to MatchPrefix()
RE_Matcher: Add MatchPrefix with bol/eol control
For deeply encapsulated connections (think AWS traffic mirroring format
like IP,UDP,GENEVE,IP,UDP,VXLAN,ETH,IP,TCP), the Dispatcher::Lookup()
method is fairly visible in profiles when running in bare mode.
This changes the Analyzer::Lookup() and Dispatcher::Lookup() return value
breaking the API in favor of the performance improvement.
Relates to zeek/zeek#3379.
The ssl_history field may grow unbounded (e.g., ssl_alert event). Prevent this
by capping using a configurable limit (default 100) and raise a weird once reached.
Limit the number of events raised from an SSL record with content_type
alert (21) to a configurable maximum number (default 10). For TLS 1.3,
the limit is set to 1 as specified in the RFC. Add a new weird cases
where the limit is exceeded.
OSS-Fuzz managed to generate a reproducer that raised ~660k ssl_plaintext
and ssl_alert events given ~810kb of input data. This change prevents this
with hopefully no negative side-effect in the real-world.
* origin/topic/awelzel/2326-import-quic:
ci/btest: Remove spicy-quic helper, disable Spicy on CentOS 7
btest/core/ppp: Run test in bare mode
btest/quic: Update other tests
testing/quic: Fixups and simplification after Zeek integration
quic: Integrate as default analyzer
quic: Include Copyright lines to the analyzer's source code contributed by Fox-IT
quic: Squashed follow-ups: quic.log, tests, various fixes, performance
quic: Initial implementation
* origin/topic/bbannier/issue-3234:
Introduce dedicated `LDAP::Info`
Remove redundant storing of protocol in LDAP logs
Use LDAP `RemovalHook` instead of implementing `connection_state_remove`
Tidy up LDAP code by using local references
Pluralize container names in LDAP types
Move LDAP script constants to their own file
Name `LDAP::Message` and `LDAP::Search` `*Info`
Make ports for LDAP analyzers fully configurable
Require have-spicy for tests which log spicy-ldap information
Fix LDAP analyzer setup for when Spicy analyzers are disabled
Bump zeek-testing-private
Integrate spicy-ldap test suite
Move spicy-ldap into Zeek protocol analyzer tree
Explicitly use all of spicy-ldap's modules
Explicitly list `asn1.spicy` as spicy-ldap source
Remove uses of `zeek` module in spicy-ldap
Fix typos in spicy-ldap
Remove project configuration files in spicy-ldap
Integrate spicy-ldap into build
Import zeek/spicy-ldap@57b5eff988
On Linux with a default ext4 or tmpfs filesystem, the default buffer size for
reading a pcap is chosen as 4k (strace/gdb validated). When reading large pcaps
containing raw data transfers, the syscall overhead for read becomes visible
in profiles. Support configurability of the buffer size and default to 128kb.
When processing a ~830M PCAP (16 UDP connections, each transferring ~50MB) in
bare mode, this change improves runtime from 1.39 sec to 1.29 sec. Increasing
the buffer further didn't provide a noticeable boost.
* origin/topic/awelzel/3314-lambda-redefinition-segfault:
Var/Func: Render function parameters using comma, not semicolon
Var: Fix null-pointer deref on redefinition of lambdas
* origin/topic/awelzel/deferred-default-non-const-v4:
CreationInitsOptimizer: Use PreTypedef() instead of PreType()
Fix deferred record initialization
testing/btest: Un-deferred record initalization tests
* origin/topic/timw/3059-set-vector-conversion:
Fix conversion with record types
Add conversion between set and vector using 'as' keyword
Add std::move for a couple of variables passed by value
This is based on the discussion in zeek/zeek#2668. Using &default with tables
can be confusing as the default value is not inserted. The following example
prints an empty table at the end even new Service records was instantiated.
type Service: record {
occurrences: count &default=0;
last_seen: time &default=network_time();
};
global services: table[string] of Service &default=Service();
event zeek_init()
{
services["http"]$occurrences += 1;
services["http"]$last_seen = network_time();
print services;
}
Changing above &default to &default_insert will insert the newly created
default value upon a missed lookup and act less surprising.
Other examples that caused confusion previously revolved around table of sets
or table of vectors and `add` or `+=` not working as expected.
tbl_of_vector["http"] += 1
add tbl_of_set["http"][1];
Ad-hoc include module names in the global_ids() table. Table values will
have the type_name field set to "module" and their key in the table is
prefixed with "module " to avoid clashes with existing global identifiers
shadowing module names (Management::Node being an existing example).
Closes#3136
Avoids loosing state on a connection value when a connection is flipped.
Fixes up the NTP baseline as well where this was visible: analyzer_confirmation_info()
was raised for a connection value which was immediately forgotten due to
the subsequent connection flipping.
Closed#3028
