* origin/topic/awelzel/cluster-event-metadata-fixes-for-8.0:
cluster/Backend: Fallback to current network time when current event has not timestamp
cluster/serializer/broker: Do not send empty metadata vectors around
When a WebSocket client sends an event to Zeek without explicit network
timestamp metadata, Zeek would use -1.0 as a timestamp for any events
published while handling this event. Instead, it seems far more sensible
to use the current network time in that scenario.
Event when there's no metadata attached to an event, we'd still use the
constructor passing an empty metadata vector, resulting in an on-the-wire
representation with an empty trailing vector.
Particularly visible when just snooping events via websocat. There also
seems to be some bug with the timestamp -1 handling.
* origin/topic/timw/move-submodules-to-main-repo-take-2: (343 commits)
Add NEWS entries for submodule moves
Remove configure --with-gen-zam argument and the CMake summaries
af_packet: Remove submodule, adapt CMake/code for Zeek build
af_packet: pre-commit fixes
af_packet: Fix initialization
af_packet: Ensure all of the member fields get initialized
af_packet: Require CMake 3.15 to match Zeek's requirement
af_packet: Note that Zeek ships with a built-in version.
af_packet: Use cstdint instead of stdint.h
af_packet: Use override for overriding parent methods
af_packet: Use 'pragma once' instead of include guards
af_packet: AF_Packet: Fix wrong vlan when PCP or DEI bits are set in tp_vlan_tci
af_packet: AF_Packet: Check interface for upness
af_packet: AF_Packet: Use negative socket_fd for error indication
af_packet: AF_Packet: Remove usages of inline
af_packet: Add guarded zeek/zeek-version.h include.
af_packet: RX_Ring: Add include for string
af_packet: Increase version number.
af_packet: Add info if TP_STATUS_CSUM_VALID is not defined.
af_packet: Define TP_STATUS_CSUM_VALID when not defined
...
A user reported vlan ids > 4095 being logged by Zeek [1]. For populating
packet->vlan, mask away Priority Code Point (PCP) and Drop Eligible
Indicator (DEI) bits from the tp_vlan_tci field, else we're not setting
the correct value on the packet.
Fixes#60
[1] https://community.zeek.org/t/zeek-reporting-vlan-ids-above-4095-bug-found/7000
When using af_packet with an interface that was not up, the following
non-informative error was reported:
$ /opt/zeek-5.2/bin/zeek -i af_packet::replay
fatal error: problem with interface af_packet::replay (Invalid argument)
With this change, the error now includes information about the
interface being down:
$ ZEEK_PLUGIN_PATH=$(pwd)/build zeek -Ci af_packet::replay
fatal error: problem with interface af_packet::replay (interface is down)
Fixes#51
Technically, socket() can return 0, so shouldn't use it as an
indication of a non existent / closed socket.
I'm not 100% sure about the Close() contract here: If something
goes haywire with a packet source Zeek calls FatalError without
calling Close() nor properly destructing the PktSrc. Oh yikes.
With zeek/zeek#2802, zeek-config.h will not provide ZEEK_VERSION_NUMBER
when a plugin is compiled as a builtin/static plugin into Zeek. This is
done to avoid tree-wide ccache busting when just the version changes.
When compiling with clang and libc++ like done when using Google's
hongfuzz in the oss-fuzz setup, the following errors are produced:
Step #3 - "compile-honggfuzz-address-x86_64": /src/zeek/auxil/zeek-af_packet-plugin/src/RX_Ring.cc:19:9: error: no matching conversion for functional-style cast from 'const char[15]' to 'RX_RingException'
Step #3 - "compile-honggfuzz-address-x86_64": throw RX_RingException("invalid socket");
Step #3 - "compile-honggfuzz-address-x86_64": ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Step #3 - "compile-honggfuzz-address-x86_64": /src/zeek/auxil/zeek-af_packet-plugin/src/RX_Ring.h:14:7: note: candidate constructor (the implicit copy constructor) not viable: no known conversion from 'const char[15]' to 'const RX_RingException' for 1st argument
Step #3 - "compile-honggfuzz-address-x86_64": class RX_RingException : public std::runtime_error {
Step #3 - "compile-honggfuzz-address-x86_64": ^
Step #3 - "compile-honggfuzz-address-x86_64": /src/zeek/auxil/zeek-af_packet-plugin/src/RX_Ring.h:14:7: note: candidate constructor (the implicit move constructor) not viable: no known conversion from 'const char[15]' to 'RX_RingException' for 1st argument
Step #3 - "compile-honggfuzz-address-x86_64": /src/zeek/auxil/zeek-af_packet-plugin/src/RX_Ring.h:16:2: note: candidate constructor not viable: no known conversion from 'const char[15]' to 'const std::string' (aka 'const basic_string<char, char_traits<char>, allocator<char>>') for 1st argument
Step #3 - "compile-honggfuzz-address-x86_64": RX_RingException(const std::string& what_arg) : std::runtime_error(what_arg) {}
Step #3 - "compile-honggfuzz-address-x86_64":
The problem can be reproduced outside of hongfuzz by compiling this
plugin with with clang/libc++:
export CXX=clang++-14
export CXXFLAGS=-stdlib=libc++
./configure
make
Include <string> in RX_Ring.h to ensure the required const char * to
std::string conversion are available to any users of RX_RingException.
On some older Linux distributions (CentOS 7), the if_packet.h header does
not yet include TP_STATUS_CSUM_VALID (introduced in March 2015). Simply
define it if it's not there.
This lowers the default timeout from 100msec to 10msec and increases the default
block_size from 16KB to 32KB. Both are aligned with what Suricata uses as defaults.
The block_size is likely too conservative for high-performance, tuning
guides recommend starting with 1MB block size.
Fixes#37.
There's been some wondering why kernel headers are required to compile
this plugin as it's not providing a kernel module or otherwise provides
functionality related to kernel APIs. AF_PACKET sockets are provided
through user-space APIs.
There may have been historical reasons, but let's move forward and
remove the dependency.
Fixes#29#24
The plugin mechanism automatically picks up the scripts folder for
plugin-related Zeek scripts. When a package's zkg.meta also declares the scripts
folder the location for package-level scripts, the scripts get installed
redundantly in two places. Zeek cannot recognize them as duplicates, triggering
various errors relating to double-defined symbols.
This moves the package-level scripts to a subdirectory with a placeholder. The
placeholder avoids the zkg install-time warning, whose original fix in e3aea1
introduced this problem.
