- Recomputes checksums for pcaps to keep clean
- Removes some tests that had big pcaps or weren't necessary
- Cleans up scripting names and minor points
- Comments out Spicy code that causes a build failure now with a TODO to
uncomment it
* origin/topic/johanna/ssh-server-banners-can-be-wild:
SSH analyzer - tiny aesthetic fixes
SSH: make banner parsing more robust
SSH: split banner into client/server parts
This change revamps SSH banner parsing. The previous behavior was both
a bit too strict in some regards, and too permissive in other.
Specifically, clients are now required to send a line starting with
"SSH-" as the first line. This is in line with the RFC, as well with
observed behavior. This also prevents the creation of `ssh.log` for
non-SSH traffic on port 22.
For the server side, we now accept text before the SSH banner. This
previously led to a protocol violation but is allowed by the spec.
New tests are added to cover these cases.
* origin/topic/johanna/dpd-changes:
DPD: failed services logging alignment
DPD: update test baselines; change options for external tests.
DPD: change policy script for service violation logging; add NEWS
DPD changes - small script fixes and renames.
Update public and private test suite for DPD changes.
Allow to track service violations in conn.log.
Make conn.log service field ordered
DPD: change handling of pre-confirmation violations, remove max_violations
DPD: log analyzers that have confirmed
IRC analyzer - make protocol confirmation more robust.
As services are sorted by default now, this disables the canonifier that
sorts the service field for the external baseline.
This also adds the tracking of disabled services in the service field
via DPD::track_removed_services_in_connection - to make such changes
more visible, and check that the feature works as desired.
The changes are mostly quite minor. The main change reasons are:
* analyzers that were confirmed, and later removed now show up in the
conn.log.
* a couple of removed lines in analyzer.log, because non-confirmed
analyzers get removed more quickly.
* in some cases there are additional lines in analyzer.log. These are
cases in which an analyzer gets removed due to a violation and then
re-attached because of a later signature match, which replays the
violating content. In all examples that I have so far, this is caused
by both sides of a connection speaking a differing protocol. There
probably should be a better way to handle this - but it works.
* new column for failed analyzers in conn.log
* origin/master: (27 commits)
Update doc submodule [nomail] [skip ci]
btest/ldap: Add regression test for #3919
postgresql: Simplify SSL buffering and forwarding
postgresql: Initial parser implementation
testing/external: Update private baselines
analyzer/syslog: Reformat with spicy-format
analyzer/finger: Reformat with spicy-format
scripts/spicy: Reformat with spicy-format
pre-commit: Add spicy-format
Check for netbios to avoid reporting extra bad DNS opcodes
Add weird for unhandled opcodes in DNS analyzer
Bump zeek-aux for zeek/zeek-aux#57
Remove pre-commit exclusions for clang-format
Bump clang-format
Bump auxil/spicy to latest development snapshot
RunState: Drop broker_mgr->Active() usage
script_opt/ZAM/IterInfo.h: Add missing Dict.h dependency
script_opt/ZAM: ZBody.h / Support.h: Cleanup includes, use forward declarations
script_opt/ZAM/Profile: Remove Zeek header includes
script_opt: Extend Support.h to break include dependencies
...
The reason that this is necessary is the end-of-connection-handling of
spicy. If spicy is in the middle of parsing some bytes while the
connection ends, an error is raised. This behavior cannot be changed,
and means that there will be a DPD-log entry, etc. for connections that
are completely valid TLS connections - that just happen to be truncated
and end in the middle.
It turns out that, for probably a long time, we have reported an
incorrect version when parsing an SSLv2 client hello. We always reported
this as SSLv2, no matter which version the client hello actually
contained.
This bug probably went unnoticed for a long time, as SSLv2 is
essentially unused nowadays, and as this field does not show up in the
default logs.
This was found due to a baseline difference when writing the Spicy SSL
analyzer.
* origin/topic/johanna/ssl-history-also-for-sslv2-not-only-for-things-that-use-the-more-modern-handshake:
Make ssl_history work for SSLv2 handshakes/connections
It turns out that the ssl_history field never was populated with C/S for
SSLv2 connections, or connections using the SSLv2 handshake. In our
testcases, the latter is especially common - with connections up to TLS1
using the old SSLv2 client hello for backwards compatibility.
This change resolves this issue. As the history is not by default
enabled in a lot of locations, baseline impact is minor.
* origin/topic/awelzel/3504-ldap-logs-scalars:
Update external baselines
ldap: Use scalar values in logs where appropriate
ldap: Rename LDAP::search_result to LDAP::search_result_entry
* origin/topic/bbannier/issue-3234:
Introduce dedicated `LDAP::Info`
Remove redundant storing of protocol in LDAP logs
Use LDAP `RemovalHook` instead of implementing `connection_state_remove`
Tidy up LDAP code by using local references
Pluralize container names in LDAP types
Move LDAP script constants to their own file
Name `LDAP::Message` and `LDAP::Search` `*Info`
Make ports for LDAP analyzers fully configurable
Require have-spicy for tests which log spicy-ldap information
Fix LDAP analyzer setup for when Spicy analyzers are disabled
Bump zeek-testing-private
Integrate spicy-ldap test suite
Move spicy-ldap into Zeek protocol analyzer tree
Explicitly use all of spicy-ldap's modules
Explicitly list `asn1.spicy` as spicy-ldap source
Remove uses of `zeek` module in spicy-ldap
Fix typos in spicy-ldap
Remove project configuration files in spicy-ldap
Integrate spicy-ldap into build
Import zeek/spicy-ldap@57b5eff988
This moves the ports the LDAP analyzers should be triggered on from the
EVT file to the Zeek module. This gives users full control over which
ports the analyzers are registered for while previously they could only
register them for additional ports (there is no Zeek script equivalent
of `Manager::UnregisterAnalyzerForPort`).
The analyzers could still be triggered via DPD, but this is intentional.
To fully disable analyzers users can use e.g.,
```zeek
event zeek_init()
{
Analyzer::disable_analyzer(Analyzer::ANALYZER_LDAP_TCP);
}
```
The diffs produced by telemetry.log when introducing a weird or
removing/adding protocol specific logs is overwhelming and distracting
without providing value. Exclude telemetry.log similar to how we already
exclude stats.log.
Some more targeted telemetry.log tests exists in the normal testing/btest
suite and that appears more sensible.
This change makes the community-id script that adds the community id to
notice.log automatically load the main script if this was not already
loaded.
In the past, the script just did not perform any action if the main
script was not loaded.
This change also makes the notice script respect the seed/base64
settings that were set in the main script.
Fixes GH-3242
* origin/topic/vern/zam-feature-complete: (23 commits)
documentation updates
updates to ZAM-specific BTest baseline
Address PR review feedback on zam-feature-complete
Updates to BTest baselines due to previous BTest tweaks
tweaks to BTests for ZAM feature completeness; new ZAM-only btest
removal of unused functionality and some follow-on simplifications
feature completeness for ZAM
-O gen-C++ tweaks to be compatible with ZAM changes
ZAM support for "when" statements
ZAM changes intermixed with lambda and "when" support
WhenStmt/WhenInfo restructuring in support of ZAM "when" statements
ZAM support for lambdas
ZAM internals have a notion of "captures" as global-like variables
AST profiling enhnacements in support of script optimization for lambdas/whens
refinements to LambdaExpr's to provide flexibility, support for ZVal captures
support in ScriptFunc class for ZVal-oriented vector of captures
simplifications to the Frame class now that it no longer has to support old-style captures
use Ingredients directly for constructing functions
the "Capture" struct is now a class
more debugging information when dumping script optimization data structures
...
* origin/topic/timw/2914-truncated-tunnel-plus-things:
Address wire/capture length feedback
packet_analysis/TCP: Do not use untrusted len for DeliverPacket()
Add regression test using pcap from GH-2683
Add btest to test Geneve->VXLAN->Truncated inner packet
IP: Update packet->len with accumulated fragment size
UDP: Forward any remaining data (also empty) to session-analysis
IPTunnel: Compute inner wire length based on cap_len differences.
IP: fix weird name to not be ipv6 specific
UDP: don't validate checksum if caplen < len
PIA: Modernize how struct initialization is done
