* origin/topic/vern/zam-asan-fixes:
ZAM fixes for assignments involving "any" record fields
fixes for (mostly ZAM) vector operation issues found by ASAN
Including a fix for mmdb/explicit-open.zeek to avoid using assert.
* origin/master: (60 commits)
Update gen-zam submodule [nomail] [skip ci]
Update doc submodule [nomail] [skip ci]
Remove unused wrapper packet analyzer
Add DNS TKEY event
ScriptOpt: Ensure global statements have non-null scope
simpler and more robust identification of function parameters for AST profiling
fixes to limit AST traversal in the face of recursive types
address some script optimization compiler warnings under Linux
fix for -O C++ construction of variable names that use multiple module namespaces
fix for script optimization of "opaque" values that are run-time constants
fix for script optimization of nested switch statements
script optimization fix for complex "in" expressions in conditionals
updates to typos allow-list reflecting ZAM regularization changes
BTest updates for ZAM regularization changes
convert new ZAM operations to use typed operands
complete migration of ZAM to use only public ZVal methods
"-O validate-ZAM" option to validate generated ZAM instructions
internal option to suppress control-flow optimization
exposing some functionality for greater flexibility in structuring run-time execution
rework ZAM compilation of type switches to leverage value switches
...
With this commit, the entire Zeek test suite passes using spicy TLS.
Tests that either use a SSLv2 handshake, or DTLS are skipped, as the
parser currently does not support either.
Similarly, tests that rely on behavior we cannot replicate (baseline,
hooks, exact error messages) are passed. Other than that, all the
TLS-based tests pass with 100% the exact same baseline results.
This necessitated a couple of small tweaks to the spicy file - the
testcases uncovered several small problems.
This commit also enables cirrus tests for Spicy SSL/TLS.
The core.file-analyzer-violation test showed that it's possible to
create new threads (log writers) when Zeek is in the process of
terminating. This can result in the IO manager's deconstructor
deleting IO sources for threads that are still running.
This is sort of a scripting issue, so for now log a reporter warning
when it happens to have a bit of a bread-crumb what might be
going on. In the future it might make sense to plug APIs with
zeek_is_terminating().
* origin/topic/christian/mmdb-configurability:
Modernize various C++/Zeek-isms in the MMDB code.
Fix MMDB code to re-open explicitly opened DBs correctly
Add btest to verify behavior of re-opened MMDBs opened directly via BIFs
Simplify MMDB code by moving more lookup functionality into MMDB class
Move MMDB logic out of mmdb.bif and into MMDB.cc/h.
Fix mmdb.temporary-error testcase when MMDBs are installed on system
Adapt MMDB BiF code to new script-layer variables
Update btest baselines to reflect introduction of mmdb.bif
Move MaxMind/GeoIP BiF functionality into separate file
Provide script-level configurability of MaxMind DB placement on disk
Sort toplevel .bif list in CMakeLists
In AWS GLB environments, the max_depth of 2 is easily reached due to packets
being encapsulated with GENEVE and VXLAN [1]. Any additional encapsulation
layer causes Zeek raise a weird and ignore the inner traffic. Bump the default
maximum depth to 4, while not common it's not unusual either to observe
this in the wild.
[1] https://docs.aws.amazon.com/vpc/latest/mirroring/traffic-mirroring-packet-formats.htmlCloses#3439
The mmdb_open_location_db() and mmdb_open_asn_db() BiFs were untested, and Zeek
has a bug that makes any DBs opened that way fall back to looking up DBs via the
existing script-level config mechanism (via mmdb_dir), which is at least
unexpected and might well be unconfigured if somebody uses the direct BiFs.
The test would previously fail in settings where the user has Maxmind DBs
installed in the hardwired system locations, because the fallback logic still
picked those up.
* origin/topic/awelzel/2326-import-quic:
ci/btest: Remove spicy-quic helper, disable Spicy on CentOS 7
btest/core/ppp: Run test in bare mode
btest/quic: Update other tests
testing/quic: Fixups and simplification after Zeek integration
quic: Integrate as default analyzer
quic: Include Copyright lines to the analyzer's source code contributed by Fox-IT
quic: Squashed follow-ups: quic.log, tests, various fixes, performance
quic: Initial implementation
This moves the ports the LDAP analyzers should be triggered on from the
EVT file to the Zeek module. This gives users full control over which
ports the analyzers are registered for while previously they could only
register them for additional ports (there is no Zeek script equivalent
of `Manager::UnregisterAnalyzerForPort`).
The analyzers could still be triggered via DPD, but this is intentional.
To fully disable analyzers users can use e.g.,
```zeek
event zeek_init()
{
Analyzer::disable_analyzer(Analyzer::ANALYZER_LDAP_TCP);
}
```
On Linux with a default ext4 or tmpfs filesystem, the default buffer size for
reading a pcap is chosen as 4k (strace/gdb validated). When reading large pcaps
containing raw data transfers, the syscall overhead for read becomes visible
in profiles. Support configurability of the buffer size and default to 128kb.
When processing a ~830M PCAP (16 UDP connections, each transferring ~50MB) in
bare mode, this change improves runtime from 1.39 sec to 1.29 sec. Increasing
the buffer further didn't provide a noticeable boost.
Using pcaps from https://interop.seemann.io/ as samples for QUIC protocol
data didn't produce a conn.log for the contained data. `tcpdump -r`
and Wireshark do show the contained IP/UDP packets. Teach Zeek how
to handle link type DLT_PPP 0x09 using a new PPP analyzer based on the
PPPSerial analyzer code.
Usual update to files/x509 baseline after adding new analyzer due
to enum values changing.
This is similar to GH-3206. There do not seem to be practical
consequences - but we should still fix it.
This also includes the udp-testcase that was forgotten in GH-3206.
Avoids loosing state on a connection value when a connection is flipped.
Fixes up the NTP baseline as well where this was visible: analyzer_confirmation_info()
was raised for a connection value which was immediately forgotten due to
the subsequent connection flipping.
Closed#3028
For "individually different but deterministic" runs specifying Zeek's
seed as an environment variable eases setups as one can avoid creating
extra seed files for each of the individual processes.
It is an error to specify the new ZEEK_SEED_VALUES variable together
with the existing ZEEK_SEED_FILE and -G. ZEEK_SEED takes precedence over
deterministic mode (-D) like ZEEK_SEED_FILE does today already.
