This addresses review feedback of GH-1831 and additionally fixes one
case in which PayloadLen was used in a way that would have given
problematic results when TSO is enabled.
When checksum offloading is enabled, we now forward packets that
have 0 header lengths set - and assume that they have TSO enabled.
If checksum offloading is not enabled, we drop the packets.
Addresses GH-1829
Merge includes small changes, e.g. fixing the comsumption of remaining
raw data.
* 'master' of https://github.com/FlyingWithJerome/zeek:
remove excussive fields in dns_svcb_rr
address code reviews (formatting and type and intrusiveptr)
newlines at the end of test outputs
lazy commit
use tabs in init-bare.zeek
add svcb test case
add a dns https test case
remove test logs
fix a few syntax errors
initial commit for SVCB/HTTPS records
We're using shadow files for log rotation on systems with ext4 running
Linux 4.19. We've observed zero-length shadow files in the logger's working
directory after a power-outage. This leads to a broken/stuck logger
process due to empty shadow files being considered invalid and the
process exiting:
error: failed to process leftover log 'conn.log.gz': Found leftover log, 'conn.log.gz', but the associated shadow file, '.shadow.conn.log.gz', required to process it is invalid
PR #1137 introduced atomic renaming of shadow files and was supposed to
handle this. However, after more investigation, the rename() has to be
preceded by an fsync() in order to avoid zero-length files in the presence
of hard-crashes or power-failures. This is generally operating system
and filesystem dependent, but should not hurt to add. The performance impact
can likely be neglected due to the low frequency and limited number of
log streams.
This has happened to others, too. Some references around this issue:
* https://stackoverflow.com/questions/7433057/is-rename-without-fsync-safe
* https://unix.stackexchange.com/questions/464382/which-filesystems-require-fsync-for-crash-safety-when-replacing-an-existing-fi
* https://bugzilla.kernel.org/show_bug.cgi?id=15910
Reproducer
This issue was reproduced artificially on Linux using the sysrq-trigger
functionality to hard-reset the system shortly after a .shadow file was
renamed to it's final destination with the following script watching for
.shadow.conn.log.gz:
#!/bin/bash
set -eu
dir=/data/logger-01/
# Allow everything via /proc/sysrq-trigger
echo "1" > /proc/sys/kernel/sysrq
inotifywait -m -e MOVED_TO --format '%e %w%f' "${dir}" | while read -r line; do
if echo "${line}" | grep -q '^MOVED_TO .*/.shadow.conn.log.gz$'; then
echo "RESET: $line"
sleep 4
# Trigger a hard-reset without sync/unmount
echo "b" > /proc/sysrq-trigger
fi
done
This quite reliably (4 out of 4 times) yielded a system with zero-length
shadow files and a broken logger after it came back online:
$ ls -lha /data/logger-01/.shadow.*
-rw-r--r-- 1 bro bro 0 Oct 14 02:26 .shadow.conn.log.gz
-rw-r--r-- 1 bro bro 0 Oct 14 02:26 .shadow.dns.log.gz
-rw-r--r-- 1 bro bro 0 Oct 14 02:26 .shadow.files.log.gz
After this change while running the reproducer, the shadow files always
contained content after a hard-reset.
Rework with util::safe_fsync helper
* topic/christian/ci-tweaks-freebsd-testsuite:
Don't initialize private testsuite in CI when not running out of Zeek project
Drop FreeBSD 11.4 (now EOL), add FreeBSD 14 as informational build
The initialization of the private testsuite currently fails builds run in
forks. Cirrus's encrypted variables won't work outsize of the Zeek project, so
skip that setting altogether.
* topic/christian/complex-indices:
Expand table/set tests to cover new index types
Expression tweaks to get complex types to work as container indices
Whitespace tweak in a btest, no other change
* topic/christian/docker-tweaks:
Enable customization of the Docker image's configure invocation
Add Python Broker module load test to Docker image btests
Set PYTHONPATH in Docker to automatically find Zeek's modules
Docker image fix: libpython needs to be 3.9 on Debian Bullseye
* topic/christian/bump-highwayhash:
CI tweak: upgrade curl on FreeBSD to avoid cert expiration
CI tweak: add a datestamp to Debian 9 Dockerfiles to invalidate Cirrus build cache
Bump highwayhash submodule to pull in another FreeBSD header fix
We missed the fact that the broker module didn't load due to a wrong libpython
version because we didn't test that load.
Also remove an unused environment variable from btest.cfg.
* origin/topic/timw/clang-format-fixes:
clang-format: Set penalty for breaking after assignment operator
clang-format: Set IndentCaseBlocks to false
clang-format: Other minor formatting changes
clang-format: Other include ordering changes
clang-format: Enforce ordering of includes in ZBody
clang-format: A few minor comment-spacing fixes
clang-format: Force zeek-config.h to be earlier in the config ordering
