* topic/AbdelSaTd/case-insensitive-find:
testing-do-find-str_case-insensitive
case-insensitive-search-features-for-do_find_str
I did a few small style-fixes while merging this and expanded the test
a bit. I also removed unecessary commits.
Closes GH-1828
* origin/topic/robin/gh-1757-loadfile2:
Tweaking a couple of debug message.
Add new hook `HookLoadFileExtended` that allows plugins to supply Zeek script code to parse.
Move logic to execute `HookLoadFile` for signatures into rule matcher code.
This is a fixup commit for 72cbc7cd13
where we move some header files from `src/` to `src/3rdparty/` but
missed adding install rules for these header. Since some of these
headers are exposed in installed headers they need to be installed as
well.
* origin/topic/bbannier/issue-1780:
Add Github action exercising pre-commit
Add pre-commit config.
Format code with `clang-format`
Remove stale files `src/DebugCmdInfoConstants.*`
Disable formatting for files in `testing/btest/plugins`
Move 3rdparty source files to `3rdparty/`
This addresses review feedback of GH-1831 and additionally fixes one
case in which PayloadLen was used in a way that would have given
problematic results when TSO is enabled.
The files generated from `src/DebugCmdInfoConstants.in` are placed in
`build/src/` by the build setup, and generated file in `src/` removed
here were unused and possibly out-of-date.
The new hooks works similar to the existing `HookLoadFile` but,
additionally, allows the plugin to return a string that contains the
code to be used for the file being loaded. If the plugin does so, the
content of any actual file on disk will be ignored (in fact, there
doesn't even need to be a file on disk in that case). This works for
both Zeek scripts and signatures.
There's a new test that covers the new functionality, testing loading
both scripts and signatures from memory. I also manually tested that the
debugger integration works, but I don't see much of a way to add a
regression test for that part.
We keep the existing hook as well for backwards compatibility. We could
decide to deprecate it, but not sure that buys us much, so left that
out.
Closes#1757.
This (1) fixes an issue where signature files supplied on the command
line wouldn't pass through the hooks, and (2) prepares for allowing
hooks to supply the content of a signature file directly.
This is apparently a problem with the Myricom version of libpcap, where
instead of returning a null or a zero if no packets are available, it
returns the previous packet. This causes Zeek to improperly parse the
packet and crash. We thought we had fixed this previously with a check
for a null packet but that fix was not enough.
When checksum offloading is enabled, we now forward packets that
have 0 header lengths set - and assume that they have TSO enabled.
If checksum offloading is not enabled, we drop the packets.
Addresses GH-1829
Merge includes small changes, e.g. fixing the comsumption of remaining
raw data.
* 'master' of https://github.com/FlyingWithJerome/zeek:
remove excussive fields in dns_svcb_rr
address code reviews (formatting and type and intrusiveptr)
newlines at the end of test outputs
lazy commit
use tabs in init-bare.zeek
add svcb test case
add a dns https test case
remove test logs
fix a few syntax errors
initial commit for SVCB/HTTPS records
We're using shadow files for log rotation on systems with ext4 running
Linux 4.19. We've observed zero-length shadow files in the logger's working
directory after a power-outage. This leads to a broken/stuck logger
process due to empty shadow files being considered invalid and the
process exiting:
error: failed to process leftover log 'conn.log.gz': Found leftover log, 'conn.log.gz', but the associated shadow file, '.shadow.conn.log.gz', required to process it is invalid
PR #1137 introduced atomic renaming of shadow files and was supposed to
handle this. However, after more investigation, the rename() has to be
preceded by an fsync() in order to avoid zero-length files in the presence
of hard-crashes or power-failures. This is generally operating system
and filesystem dependent, but should not hurt to add. The performance impact
can likely be neglected due to the low frequency and limited number of
log streams.
This has happened to others, too. Some references around this issue:
* https://stackoverflow.com/questions/7433057/is-rename-without-fsync-safe
* https://unix.stackexchange.com/questions/464382/which-filesystems-require-fsync-for-crash-safety-when-replacing-an-existing-fi
* https://bugzilla.kernel.org/show_bug.cgi?id=15910
Reproducer
This issue was reproduced artificially on Linux using the sysrq-trigger
functionality to hard-reset the system shortly after a .shadow file was
renamed to it's final destination with the following script watching for
.shadow.conn.log.gz:
#!/bin/bash
set -eu
dir=/data/logger-01/
# Allow everything via /proc/sysrq-trigger
echo "1" > /proc/sys/kernel/sysrq
inotifywait -m -e MOVED_TO --format '%e %w%f' "${dir}" | while read -r line; do
if echo "${line}" | grep -q '^MOVED_TO .*/.shadow.conn.log.gz$'; then
echo "RESET: $line"
sleep 4
# Trigger a hard-reset without sync/unmount
echo "b" > /proc/sysrq-trigger
fi
done
This quite reliably (4 out of 4 times) yielded a system with zero-length
shadow files and a broken logger after it came back online:
$ ls -lha /data/logger-01/.shadow.*
-rw-r--r-- 1 bro bro 0 Oct 14 02:26 .shadow.conn.log.gz
-rw-r--r-- 1 bro bro 0 Oct 14 02:26 .shadow.dns.log.gz
-rw-r--r-- 1 bro bro 0 Oct 14 02:26 .shadow.files.log.gz
After this change while running the reproducer, the shadow files always
contained content after a hard-reset.
Rework with util::safe_fsync helper
