* origin/topic/timw/2914-truncated-tunnel-plus-things:
Address wire/capture length feedback
packet_analysis/TCP: Do not use untrusted len for DeliverPacket()
Add regression test using pcap from GH-2683
Add btest to test Geneve->VXLAN->Truncated inner packet
IP: Update packet->len with accumulated fragment size
UDP: Forward any remaining data (also empty) to session-analysis
IPTunnel: Compute inner wire length based on cap_len differences.
IP: fix weird name to not be ipv6 specific
UDP: don't validate checksum if caplen < len
PIA: Modernize how struct initialization is done
This patch changes invocations of `spicyz` and similar Spicy tools in
tests which perform compilation to use debug mode via passing `-d`. This
in turn leads to Spicy compiling generated C++ code in debug as opposed
to release mode which typically seems to require less CPU time and RAM.
For a local test running with `btest -j 16` and no caching via
`HILTI_CXX_COMPILER_LAUNCER` this sped up running of BTests under
`spicy/` by about 40s on my machine (120s vs 160s).
If CMakeCache.txt indicates the Zeek build was done with ccache,
take the CCACHE_PROGRAM:FILEPATH= line and populate the
HILTI_CXX_COMPILER_LAUNCHER environment.
For local development with ccache, this speeds up the spicy tests
significantly after the initial run without knowing or needing know
that setting HILTI_CXX_COMPILER_LAUNCHER would've helped.
We should not be passing the untrusted TCP header length into
DeliverPacket(). Also, DeliverPacket() cap len parameter should
be the capture length of the packet, not remaining data.
With packet->len representing the wire length and other places
relying on it, ensure it's updated for fragments as well. This
assumes non-truncated fragments right now. Otherwise we'd need
to teach the FragmentReassembler to somehow track this independently
but it would be a mess.
The protocol analyzers are prepared to receive truncated data and
this way we give analyzers a chance to look at data. We previously
allowed empty data being passed: When len ended up 0 and remaining
was 0 too.
This may happen with truncated packets and will cause asan builds to bail out
before the packet can be forwarded along. The TCP analyzer already has this
check, but it's missing for UDP.
While writing documentation about troubleshooting and looking a bit
at the older stats.log, realized we don't have the packet lag metric
exposed as metric/telemetry. Add it.
This is a Zeek instance lagging behind in network time ~6second because
it's very overloaded:
zeek_net_packet_lag_seconds{endpoint=""} 6.169406 1684848998092
This seems to have relied on the reading file twice behavior simply
testing that 16 lines are observed. Switch to using two separate
files and doing a system("mv ...") to trigger the REREAD logic, there's
not force_update() needed and it wouldn't do anything if the file
hadn't changed anyway.
Found while writing documentation and being confused why
all lines and end_of_data() arrive twice during startup.
The test is a bit fuzzy, but does fail reliably without
the changes to Raw.cc
Also fix not checking dev in the MODE_REREAD path.
Closes#3053
It needs to be defined by the time we create zeek-config, which happens before
its current definition. To avoid a redundant TOLOWER when we check for presence
of --enable-debug at the beginning, this also switches this to a case-unadjusted
comparison to "Debug", which we use elsewhere in the file too.
Repeating the message for every new call to get_file_handle() is not
very useful. It's pretty much an analyzer configuration issue so logging
it once should be enough.
