A std::unordered_map takes 56 bytes on my system. Switch to a unique_ptr
an initialize weird_state lazily. That saves ~48 bytes per connection.
Particularly for scan or non-weird traffic, this should allow some
memory savings.
When the stem process terminates and SIGCHLD is sent to the supervisor,
the signal might be handled by the main thread or any other threads that
aren't blocking SIGCHLD explicitly. Convert last_signal to a std::atomic<int>
such that non-main threads can safely set last_signal without triggering
data race as reported by TSAN. This doesn't make it less racy to work
last_signal, but it appears we only use it for debug printing anyhow and
another option might have been to just remove last_signal altogether.
Follow-up for #4849
* origin/topic/timw/merge-3rdparty-repo-into-main: (30 commits)
Ignore src/3rdparty for pre-commit
src/3rdparty: Port doctest fix for including <ciso646> from upstream
src/3rdparty: Update doctest to v2.4.12
src/3rdparty: Move jthread/stop_token out of std namespace to prevent collisions
src/3rdparty: Fix clang-tidy bugprone-casting-through-void warning
src/3rdparty: Add jthread and stop_token headers
src/3rdparty: Update SQLite to 3.47.1
src/3rdparty: Upgrade sqlite3 to 3.45.0
src/3rdparty: modp: Disable deprecation warning for sprintf
src/3rdparty: Update SQLite to 3.41.2
src/3rdparty: Handle zeek-inet-ntop snprintf() return value correctly
src/3rdparty: Rework inet-ntop change to build on Windows
src/3rdparty: Use snprintf in zeek_inet_ntop.c to silence macOS compiler warnings
src/3rdparty: Use snprintf in patricia.cc to silence macOS compiler warnings
src/3rdparty: Update ConvertUTF Unicode license as per request from LLVM upstream
src/3rdparty: Update doctest to 2.4.9 and sqlite to 3.39.4
src/3rdparty: Fix rampant off-by-one error in last change
src/3rdparty: numeric conversion functions now return the number of characters added
src/3rdparty: Fix GCC 12.2 warning in access to patricia_t members
src/3rdparty: Rebase patricia.{h,c} on upstream version
...
This has been failing on our weekly builds for a bit. It's fixed upstream
but there's no idea when they'll release a new version that includes it.
This cherry-picks 5eb20178ba
Access to members of the larger prefix_t, while safe, throws warnings
when the compiler knows it previously allocated the smaller prefix4_t.
Adapted from 11ae12079ca716f8cfac42cccfd905dad45da18f in the upstream repo.
If Cluster::init() hasn't been invoked yet, Cluster::subscribe() with the
ZeroMQ backend would block because the main_inproc socket didn't
yet have a connection from the child thread. Prevent this by connecting
the main and child socket pair at construction time.
This will queue the subscriptions and start processing them once the
child thread has started.
An URI containing a bracketed or non-bracketed IPv6 address of the form
http://[::1]:42 was previously split on the first colon for port extraction,
causing a subsequent to_count() call to fail. Harden this to check for a
digits in the last :[0-9]+ component.
Fixes#4842
build
Whatever is used with recent OpenSSL is also available with latest LibreSSL
on OpenBSD 7.8-beta as of today.
Some of these hunks have been in the net/bro port for years, others I
recently added whilst gradually updating from 6.0.5 to 8.0.1.
* origin/topic/timw/4742-skip-storage-thread:
Avoid starting up storage expiration thread with zero backends
Remove a couple of obsolete TODOs in storage manager
* origin/topic/timw/windows-zeromq-support:
Reword comment about when ZeroMQ is required for the build
CI: Add label for enabling the Windows build in PRs
Enable building ZeroMQ support on Windows
Add a test/example forwarding all new_connection() events produced during
`zeek -r wikipedia.trace` as my_new_connection() to a WebSocket client.
This is mostly to demonstrate and verify usage of suspend_processing(),
websocket_client_added(), resume_processing(), Pcap::file_done(),
websocket_client_lost() and terminate() together.
The upstream recently merge Christian's fixes for the fd_map allocations
so that patch doesn't need to be in our fork anymore. I also tested removing
the patch about timers from our fork, and it seems to only matter for Linux
kernels earlier than 5.3. All of our supported platforms are newer than
that.
* origin/topic/vern/emphasize-IDPtrs:
fixup! fixup! shift much of the internal use of ID* identifier pointers over to IDPtr objects
fixup! shift much of the internal use of ID* identifier pointers over to IDPtr objects
fixup! shift much of the internal use of ID* identifier pointers over to IDPtr objects
fixup! shift much of the internal use of ID* identifier pointers over to IDPtr objects
annotate a number of BTests as unsuited for -O gen-C++ testing due to multiple Zeek runs
BTest baseline updates for -O gen-C++ - all minor tweaks
BTest updates for script optimization tracking of BiFs
regression test for former ASAN issue with script optimization of lambdas
shift much of the internal use of ID* identifier pointers over to IDPtr objects
maintenance update for script optimization's knowledge of BiFs
logger fix for interoperability with -O gen-C++ code
Add full support for RFC 9460's SvcParams list.
Amend the existing `dns_svcb_rr` record by a vector of new
`dns_svcb_param` records containing aptly typed SvcParamKey and
SvcParamValue pairs. Example output:
```
@load base/protocols/dns
event dns_HTTPS( c: connection , msg: dns_msg , ans: dns_answer , https: dns_svcb_rr ) {
for (_, param in https$svc_params)
print to_json(param); # filter uninitialised values
}
```
```
$ dig https cloudflare-ech.com +short | tr [:space:] \\n
1
.
alpn="h3,h2"
ipv4hint=104.18.10.118,104.18.11.118
ech=AEX+DQBBHgAgACBGL2e9TiFwjK/w1Zg9AmRm7mgXHz3PjffP0mTFNMxmDQAEAAEAAQASY2xvdWRmbGFyZS1lY2guY29tAAA=
ipv6hint=2606:4700::6812:a76,2606:4700::6812:b76
```
```
{"key":1,"alpn":["h3","h2"]}
{"key":4,"hint":["104.18.10.118","104.18.11.118"]}
{"key":5,"ech":"AEX+DQBBHgAgACBGL2e9TiFwjK/w1Zg9AmRm7mgXHz3PjffP0mTFNMxmDQAEAAEAAQASY2xvdWRmbGFyZS1lY2guY29tAAA="}
{"key":6,"hint":["2606:4700::6812:a76","2606:4700::6812:b76"]}
```
Values with malformed data or belonging to invalid/reserved keys
are passed raw bytes in network order for script-level inspection.
Follow up to "Initial Support to DNS SVCB/HTTPS RR"
https://github.com/zeek/zeek/pull/1808
* origin/topic/timw/readability-container-contains:
Fix a few more random clang-tidy findings
Use std::numbers::pi instead of hard-coded value
Use std::scoped_lock instead of std::lock_guard
Use .contains() instead of .find() or .count()
* topic/christian/management-framework-tls:
Bump zeek-testing-cluster to pull in WebSocket TLS updates
Bump zeek-client to pull in TLS config updates
Management framework: add TLS options for controller's websocket server
* origin/topic/timw/switch-spicy-zeek-builds-to-ubuntu-24:
Move benchmarking to Ubnutu 24 task, add to normal PR builds
Reorder ubuntu builds so 22.04 comes last
Rename Ubuntu-based tasks to have the full version name
Switch Zeek Spicy builds to Ubuntu 24
* origin/topic/timw/tools-clang-tidy:
gen-zam: Move source files up a directory
bifcl: Fix clang-tidy warnings
bifcl: Run clang-format on c++ code embedded in builtin-func.{l,y}
binpac: Fix a large number of clang-tidy warnings
binpac: Replace delete_list macro uses
binpac: Run clang-format on c++ code embedded in pac_parse.{ll,yy}
* origin/topic/awelzel/cluster-event-metadata-fixes-for-8.0:
cluster/Backend: Fallback to current network time when current event has not timestamp
cluster/serializer/broker: Do not send empty metadata vectors around
We install test data which we also make available under an alternative
path for backwards compatibility. The installation of this symlink did
not take `DESTDIR` installs like used by Zeek's packaging into account
which caused installations from packages to behave different from
installs from source.
This patch fixes the symlink to respect a possible `DESTDIR`.
Closes#3266.
When a WebSocket client sends an event to Zeek without explicit network
timestamp metadata, Zeek would use -1.0 as a timestamp for any events
published while handling this event. Instead, it seems far more sensible
to use the current network time in that scenario.
Event when there's no metadata attached to an event, we'd still use the
constructor passing an empty metadata vector, resulting in an on-the-wire
representation with an empty trailing vector.
Particularly visible when just snooping events via websocat. There also
seems to be some bug with the timestamp -1 handling.
* origin/topic/timw/move-submodules-to-main-repo-take-2: (343 commits)
Add NEWS entries for submodule moves
Remove configure --with-gen-zam argument and the CMake summaries
af_packet: Remove submodule, adapt CMake/code for Zeek build
af_packet: pre-commit fixes
af_packet: Fix initialization
af_packet: Ensure all of the member fields get initialized
af_packet: Require CMake 3.15 to match Zeek's requirement
af_packet: Note that Zeek ships with a built-in version.
af_packet: Use cstdint instead of stdint.h
af_packet: Use override for overriding parent methods
af_packet: Use 'pragma once' instead of include guards
af_packet: AF_Packet: Fix wrong vlan when PCP or DEI bits are set in tp_vlan_tci
af_packet: AF_Packet: Check interface for upness
af_packet: AF_Packet: Use negative socket_fd for error indication
af_packet: AF_Packet: Remove usages of inline
af_packet: Add guarded zeek/zeek-version.h include.
af_packet: RX_Ring: Add include for string
af_packet: Increase version number.
af_packet: Add info if TP_STATUS_CSUM_VALID is not defined.
af_packet: Define TP_STATUS_CSUM_VALID when not defined
...
A user reported vlan ids > 4095 being logged by Zeek [1]. For populating
packet->vlan, mask away Priority Code Point (PCP) and Drop Eligible
Indicator (DEI) bits from the tp_vlan_tci field, else we're not setting
the correct value on the packet.
Fixes#60
[1] https://community.zeek.org/t/zeek-reporting-vlan-ids-above-4095-bug-found/7000
When using af_packet with an interface that was not up, the following
non-informative error was reported:
$ /opt/zeek-5.2/bin/zeek -i af_packet::replay
fatal error: problem with interface af_packet::replay (Invalid argument)
With this change, the error now includes information about the
interface being down:
$ ZEEK_PLUGIN_PATH=$(pwd)/build zeek -Ci af_packet::replay
fatal error: problem with interface af_packet::replay (interface is down)
Fixes#51
Technically, socket() can return 0, so shouldn't use it as an
indication of a non existent / closed socket.
I'm not 100% sure about the Close() contract here: If something
goes haywire with a packet source Zeek calls FatalError without
calling Close() nor properly destructing the PktSrc. Oh yikes.
With zeek/zeek#2802, zeek-config.h will not provide ZEEK_VERSION_NUMBER
when a plugin is compiled as a builtin/static plugin into Zeek. This is
done to avoid tree-wide ccache busting when just the version changes.
When compiling with clang and libc++ like done when using Google's
hongfuzz in the oss-fuzz setup, the following errors are produced:
Step #3 - "compile-honggfuzz-address-x86_64": /src/zeek/auxil/zeek-af_packet-plugin/src/RX_Ring.cc:19:9: error: no matching conversion for functional-style cast from 'const char[15]' to 'RX_RingException'
Step #3 - "compile-honggfuzz-address-x86_64": throw RX_RingException("invalid socket");
Step #3 - "compile-honggfuzz-address-x86_64": ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Step #3 - "compile-honggfuzz-address-x86_64": /src/zeek/auxil/zeek-af_packet-plugin/src/RX_Ring.h:14:7: note: candidate constructor (the implicit copy constructor) not viable: no known conversion from 'const char[15]' to 'const RX_RingException' for 1st argument
Step #3 - "compile-honggfuzz-address-x86_64": class RX_RingException : public std::runtime_error {
Step #3 - "compile-honggfuzz-address-x86_64": ^
Step #3 - "compile-honggfuzz-address-x86_64": /src/zeek/auxil/zeek-af_packet-plugin/src/RX_Ring.h:14:7: note: candidate constructor (the implicit move constructor) not viable: no known conversion from 'const char[15]' to 'RX_RingException' for 1st argument
Step #3 - "compile-honggfuzz-address-x86_64": /src/zeek/auxil/zeek-af_packet-plugin/src/RX_Ring.h:16:2: note: candidate constructor not viable: no known conversion from 'const char[15]' to 'const std::string' (aka 'const basic_string<char, char_traits<char>, allocator<char>>') for 1st argument
Step #3 - "compile-honggfuzz-address-x86_64": RX_RingException(const std::string& what_arg) : std::runtime_error(what_arg) {}
Step #3 - "compile-honggfuzz-address-x86_64":
The problem can be reproduced outside of hongfuzz by compiling this
plugin with with clang/libc++:
export CXX=clang++-14
export CXXFLAGS=-stdlib=libc++
./configure
make
Include <string> in RX_Ring.h to ensure the required const char * to
std::string conversion are available to any users of RX_RingException.
On some older Linux distributions (CentOS 7), the if_packet.h header does
not yet include TP_STATUS_CSUM_VALID (introduced in March 2015). Simply
define it if it's not there.
This lowers the default timeout from 100msec to 10msec and increases the default
block_size from 16KB to 32KB. Both are aligned with what Suricata uses as defaults.
The block_size is likely too conservative for high-performance, tuning
guides recommend starting with 1MB block size.
Fixes#37.
There's been some wondering why kernel headers are required to compile
this plugin as it's not providing a kernel module or otherwise provides
functionality related to kernel APIs. AF_PACKET sockets are provided
through user-space APIs.
There may have been historical reasons, but let's move forward and
remove the dependency.
Fixes#29#24
The plugin mechanism automatically picks up the scripts folder for
plugin-related Zeek scripts. When a package's zkg.meta also declares the scripts
folder the location for package-level scripts, the scripts get installed
redundantly in two places. Zeek cannot recognize them as duplicates, triggering
various errors relating to double-defined symbols.
This moves the package-level scripts to a subdirectory with a placeholder. The
placeholder avoids the zkg install-time warning, whose original fix in e3aea1
introduced this problem.
Under MSVC regex objects are multiline by default and there is no apparant way of changing this behaviour. Under clang/gcc regex is singleline unless std::regeX_constants::multiline is specificied.
This behaviour is assumed in Gen-ZAM.cc when auto-generating header files.
Example: https://godbolt.org/z/aP59x3EhT
1491335: AUTO_CAUSES_COPY due to a for loop using auto& instead of const auto&
1491338: AUTO_CAUSES_COPY due to a for loop using auto& instead of const auto&
Currently, Zeek disables any static type checking for var_arg bifs.
However, the generated preamble for var_args bifs assume that
typed positional arguments are correctly typed and blindly calls
the type converters on them. This easily triggers abort()s at runtime
currently when a script mistakenly uses the wrong types for var_arg
bifs. For example, calling publish_rr() with a port instead of a string
causes a hard-abort with Zeek 5.0.8.
$ zeek -e 'Cluster::publish_rr(Cluster::Pool(), 80/tcp)'
fatal error in <no location>: Val::CONVERTER (port/string) (80/tcp)
Aborted (core dumped)
Extend bifcl so that for var_arg functions and the types that bifcl understands,
we render a runtime type check and explicit early return to avoid the abort().
For any/other types, the implementer of the bif continuous to be responsible
for type checking.
This isn't solving the var_args situation generally, but avoids some
ad-hoc fixes trickling in current bif implementations.
Some references:
https://github.com/zeek/zeek/issues/1523https://github.com/zeek/zeek/issues/2425https://github.com/zeek/zeek/issues/2935https://github.com/zeek/zeek/pull/2950
I'm not sure why this was restricted to only two components,
the following appears functional in Zeek scripts.
module A::B::C;
export {
type MyRecord: record {
a: string;
};
}
Closes#25.
This fixes the DWARF information gets output by the compiler, and allows debuggers
to use the full path name to display contextual information when a session stops
inside of BIF code.
The Yacc grammar treats comments like other whitespace and when seeing
the first definition in a file would previously emit all whitespace
before emitting the `export` section containing the definition. This
lead to the first definition being separated from their zeekygen
documention (separated by `export {`).
With this patch we start the export section before emitting whitespace.
While this might now pull more "whitespace" into the exported part, it
avoids breaking the association between zeekygen comments and
definitions.
Closes#15.
This avoids potential problems with libc++ 8+ on case-insensitive file
systems due to inclusion of a new header called <version> which will end
up conflicting with the VERSION file if the search path includes the
project root.
Binpac generates a lot of switch statements with repeated blocks in
them (typically empty blocks). Running clang-tidy on the generated code
with bugprone-branch-clone generates a lot of warnings. Instead of
doing a ton of analysis in binpac to avoid generating the duplicates,
just mark any switch generated with an annotation to avoid reporting
them.
Elsewhere (zeek/zeek#2482), it was observed that when binpac encounters
include failures, it still exits with 0 indicating success. Subsequent
compilation of the produced .h and .cc files likely fails.
Exit with 1 on include errors to make pin pointing issues easier by
having make/ninja stop earlier.
CMake documentation says that CMAKE_CFG_INTDIR is no longer necessary to
find the right binary for the configuration and is in fact deprecated in
recent versions of CMake.
For example:
inum: uint32 = case (ed & 0x0f) of {
0x00 -> n_8; # n_8 is a uint8
0x01 -> n_16; # n_16 is a uint16
0x02 -> n_32; # n_32 is a uint32
default -> 0;
};
Previously, the temporary storage used for evaluating the
case-expression was based on whatever type the first case yields, which
is a uint8 in the above example. That behavior can lead to a narrowing
conversion whenever the 0x01 or 0x02 cases occur.
The new behavior is to base the temporary storage's type on the largest
numeric type that the case-expression can yield, which is uint32 in the
above example.
Incremental flowbuffer parsing sought to first parse the "minimum header
length" required to get the full frame length, possibly from a record
field, but generating the logic to parse that field could greedily
bundle in additional boundary-checks for all subsequent fields of
known-size.
E.g. for flowunit parsing of this:
type HDR = record {
version: uint8;
reserved: uint8;
len: uint16;
} &byteorder=bigendian;
type FOO_PDU(is_orig: bool) = record {
hdr: HDR;
plen: uint8;
ptype: uint8;
something: bytestring &restofdata;
} &byteorder=bigendian, &length=hdr.len;
The flowbuffer was correctly seeking to buffer 4 bytes and parse the
"hdr.len" field, but the generated parsing logic for "hdr.len" included
a boundary check all the way up to include "plen" and "ptype".
This causes out-of-bounds exceptions to be thrown for inputs that should
actually be possible to incrementally parse via flowbuffer.
OpenBSD shared library names are like "libfoo.so.major.minor" and
binpac was previously letting the post-release number into the name
like "libbinpac.so.0.54-7", which isn't compatible with that scheme.
Related to https://github.com/zeek/zeek/issues/649
Array lengths use signed integer storage, so multiplication of that
by the element size for purpose of bounds checking against available
data may produce a signed integer overlow, which is undefined behavior.
The type used to store the index for a case-type now tracks the
type of the index expression rather than always using an "int".
The case fields also now have some checking done at code-gen-time to
ensure the constants used for cases does not exceed the numeric limit
of the type used in the case's index expression. Then, assuming, it
looks safe, the C++ case labels are generated with casts to the type
of the Binpac case's index expression to ensure compilers accept it
(since all Binpac numbers use "int" for storage/printing internally).
This allows for tunability of the following behaviors:
* Minimum flowbuffer capacity to use when parsing a new unit
* Threshold at which flowbuffer capacity is contracted back to the
minimum after parsing a complete unit and before parsing the next
* Maximum flowbuffer capacity to allow when parsing a given unit
Failed flowbuffer allocations due to reaching maximum capacity or any
other reason now throw ExceptionFlowBufferAlloc.
For arrays that are fields within a record, the bounds check was based
on a pointer to the start of the record rather than the start of the
array field.
For arrays with unknown element size, it used to cap the array length to
be the maximum it could be for the given input buffer, assuming 1-byte
elements. An exception is instead now raised for cases where the
maximum array size (number of elements) exceeds what's possibly in the
buffer. Using an exception versus capping the length may help prevent
protocol analyzers from unintentionally accessing array indices that
were not actually parsed even if the evauluated-length-expression for
that given array implies it may have been.
The former is easy to misuse by accidentally storing the contents of
the temporary string return value and accessing it later. There's also
potential pitfalls in changing it to return a pointer into a static
buffer, so instead start using strfmt() uniformly across the codebase
and change some methods to use strings instead of char*.
In this case, the bounds checking for individual elements can be
optimized out of the parsing-loop in favor of a single, array-wide
bounds check beforehand.
For arrays with a length expression (e.g. uint16[size] instead of
uint16[]), the parsing loop would consider reaching the end of the
data buffer as a successful loop termination condition even if it's
not yet parsed the required number of elements.
Now, for such arrays, the loop will only terminate based on the loop
counter (derived from the length expression) or else it will throw an
OOB exception when trying to parse an element and finding not enough
data in the buffer.
Credit to Tomas Bortoli for reporting the problem and proposing
patches.
It should only suppress the parsing-loop boundary check in the case
where array elaments are a single byte in length and thus covered by
the boundary check (generated as a result of &length) that is placed
before the parsing-loop.
The issue is that t_begin_of_data + %s can sometimes overflow.
Bug reported and patch proposed by
Philippe Antoine <p.antoine@catenacyber.fr> from Catena cyber.
How to reproduce:
>>>code
int32 n = 0xF71B0000;
int32 n1 = pac_swap(n);
code<<<
n1 becomes 0xFFFFFFF7 instead of 0x00001BF7
Reason: Undefined behaviour after bit shift operation because of
negative value of the argument. See C++ standard (2011) 5.8.2 (Shift
operators).
A common BinPAC construct for parsing records is a switch statement,
with no breaks between the cases, as control is expected to fall
through.
Coverity raises an error about this; this commit should fix that.
Specifying &length on a record no longer skips generating boundary
checks for individual fields. E.g. a record field that specifies a
&length that extends beyond the &length of the record containing it
should throw binpac::ExceptionOutOfBound, the usual way of handling
out-of-bounds conditions.
Field lengths derived from other data in the input could potentially
lead to reading from outside the bounds of the input buffer.
Reported by John Villamil and Chris Rohlf - Yahoo Paranoids
before anything else.
Internally, this function compiles all regular expressions, avoiding
to that inside the regexp constructore. The code is a bit hackish due
to the way the regexp code depends on the Bro header.
This helps ensure the availability of PRI* macros from .pac files,
which cannot create this definition themselves since the inclusion
of binpac.h is hardcoded to be placed very early in the generated
code and already includes inttypes.h itself.
Strings with a constant &length expression can be checked for negative
length values while generating the parser instead of in the parser
itself (which likely just ends up being dead code).
The code generated for types w/ &refcount will subclass RefCount and
Unref definitely deletes via a pointer to that base class so it needs a
virtual dtor.
parsers.
This consists of two parts:
1. The generated Flow classes expose their flow buffers via a new
method flow_buffer().
2. Flow buffers get two new methods:
// Interface for delayed parsing. Sometimes BinPAC doesn't get the
// buffering right and then one can use these to feed parts
// individually and assemble them internally. After calling
// FinishBuffer(), one can send the uppper-layer flow an FlowEOF()
// to trigger parsing.
void BufferData(const_byteptr data, const_byteptr end);
void FinishBuffer();
Switch to using a no-argument throw to preserve the dynamic type of
the binpac exception. Otherwise, the exception is "sliced" and can only
be subsequently handled as binpac::Exception and not a derived type.
This allows analyzers to define their own types of the same name
without mistakingly overshadowing the usages of binpac::Exception
and its derived types in the generated parser code.
If set, parsed elements won't actually be added to the array, and read
access to the array aren't permitted. This is helpful to save memory
in the case of large arrays for which elements don't need (or can't)
be buffered.
Also set some convenience variables that mimic what a FindBinPAC
module would set. These could be used by a superproject that
wants to decide whether to use an existing BinPAC installation
or (re)build from source.
- Moved find_package() calls up to top directory so failure can occur
right away if any dependencies are missing
- Got rid of a bunch of header checks and preprocessor definitions that
aren't ever used; not sure why I had them there in the first place
- Renamed VERSION preprocessor definition to not conflict with Bro's.
- Removed check for HAVE_CONFIG_H; config.h will always be generated in the
binpac build directory, but if a parent directory has generated a config.h
and added that to compiler include search paths via include_directories(),
then that one should be used -- include_directories() appends paths by
default and the parent CMakeLists.txt will have called it first.
* origin/topic/awelzel/no-null-strings-in-threading-vals:
logging/Manager: Also pass non-null vector and set
logging/Manager: Non-null strings for empty strings
After #4724, empty strings would result in nullptrs being stored in the
threading::Value's string_val.data field instead of a valid pointer to
an empty strings. This upsets UBSAN's nonnull check for memcpy()
[01:29:45.807] ../../src/SerializationFormat.cc:80:33: runtime error: null pointer passed as argument 2, which is declared to never be null
[01:29:45.807] /usr/include/string.h:44:28: note: nonnull attribute specified here
[01:29:45.807] #0 0x5b2e9c933a3f in zeek::detail::SerializationFormat::WriteData(void const*, unsigned long) /zeek/build/src/../../src/SerializationFormat.cc:80:5
[01:29:45.807] #1 0x5b2e9c935184 in zeek::detail::BinarySerializationFormat::Write(char const*, int, char const*) /zeek/build/src/../../src/SerializationFormat.cc:371:40
Continue to allocate the empty string for now as a fix.
* 'master' of https://github.com/blightzero/zeek:
Changed behavior of var-extraction-uri.zeek from policy/protocol/http to extract only the URI parameter names. Do not include the path in the first parameter name. Only extract uri vars if parameters actually exist.
Previously, Zeek treated the receipt of `AuthenticationOk` as a
successful login. However, according to the PostgreSQL
Frontend/Backend Protocol, the startup phase is not complete until
the server sends `ReadyForQuery`. It is still possible for the server
to emit an `ErrorResponse` (e.g. ERRCODE_INVALID_AUTHORIZATION_SPECIFICATION)
after `AuthenticationOk` but before `ReadyForQuery`.
This change updates the PostgreSQL analyzer to defer reporting login
success until `ReadyForQuery` is observed. This prevents false
positives in cases where authentication succeeds but session startup
fails.
* origin/topic/awelzel/4754-double-wrapped-broker-data-records:
cluster/serializer/broker: Do not special case Broker::Data anymore
broker/Data: Support unwrapping Broker::Data records
The previous approach ignored the fact that nested / inner values might
also be Broker::Data values. I'm not super sure about the validity of
the test, because it's essentially demonstrating any-nesting, but
it's not leading to extra Broker::Data encoding.
Calling val_to_data() on a Broker::Data ends up wrapping the
Broker::Data record instead of using the contained broker::value
directly.
Seems this should be the default behavior and wonder if the flag
even makes sense, but for a 8.0 backport that seems more reasonable.
* origin/topic/timw/move-submodules-to-main-repo:
Fix some clang-tidy findings in generated BIF code
Fix clang-tidy and pre-commit warnings for gen-zam code files
Move gen-zam code into the main Zeek repository
Move zeek-af_packet-plugin code into the main Zeek repository
Move the bifcl code into the main Zeek repository
Fix clang-tidy findings in the binpac lib code
Add copyright headers to all of the binpac source files
Move binpac code into the main Zeek repository
Parameters relied on is_record for a couple of validations, but they are
not records and should not be treated as such. This way we can validate
&optional better.
There was some confusing behavior with &optional and locals, so this
should get rid of that by making it an error. However, there is a case
where function parameters are still allowed to have &optional - this is
because there are checks for &default in parameters as well.
* origin/topic/awelzel/docker-trixie:
ci: Run zeekctl and builtin tasks with Debian 13, too
ci: Prepend timestamps to output
ci: Enable Spicy for arm_debian13
ci: Add Debian 13.0 (trixie)
docker: Bump to debian:trixie-slim
* origin/topic/timw/limit-string-and-container-lengths-in-logs:
Add NEWS entry for field length limiting
Tag truncated values with a flag, plus pack threading::Value better
Remove length limiting on string fields for HTTP
Make total_size counter a member in logging::Manager
Remove using numeric_limits and just check for zero instead
Expand the size of the log-size filters for x509
Add options to filter at the stream level as well as globally
Add a weird that gets emitted when strings/containers are over the limits
Add metrics to track string and container fields limited by length
Replace unused stream argument from RecordToLogRecord with WriterInfo
Implement string- and container-length filtering at the log record level
* origin/topic/timw/remove-8.1-deprecations:
Pass DNS complete_flag along as a uint8_t instead of a String
Update docs submodule with 8.1 deprecation removals
Update zeekjs submodule with 8.1 deprecation fixes
Remove deprecations tagged for v8.1
A user provided a SMB2 pcap with the reserved1 field of a ReadResponse
set to 1 instead of 0. This confused the padding computation due to
including this byte into the offset. Properly split data_offset and
reserved1 into individual byte fields.
Closes#4730
* origin/topic/timw/cleanup-warnings-from-plugin-btest-builds:
Update zeek-aux to remove BRO_DIST from plugin skeleton
cmake_minimum_required() should come before project()
* origin/topic/awelzel/tap-analyzer-take-four-thanks-clang-tidy:
btest/tap-analyzer: Update existing test and add new one for UpdateConnVal()
SessionAdapter: Keep tap_analyzers until destruction
tcp,udp,icmp adapters: Move TapPacket() to earlier
tcp,udp,icmp adapters: Fix UpdateConnVal() superclass call
This also changes the output of connection UIDs from the tap analyzer to be
prefixed with C for easier correlation with other logs.
Relates to #4337#4725#4734#4737
connection_state_remove() is invoked after Done(), so it's not a good
idea to remove the tap analyzers before in case they have up-to-date
information for the connection val.
Relates to #4337#4725#4734#4737
Writing a test, the packet was tapped after protocol analysis at least
for TCP. Ensure tapping happens before. The adapter->Process() moving
after pkt->session made me a bit wondering if things are underspecified
here, but seems reasonable to set the session on pkt before adapter->Process().
Relates to #4337#4725#4734#4737
Now that SessionAdapter implements UpdateConnVal(), the individual
adapters need to call that instead of Analyzer::UpdateConnVal()
Thanks clang-tidy.
Relates to #4337#4725#4734#4737
This commit introduces a mechanism to attach light weight analyzers to
the root analyzer of sessions in order to tap into the packets delivered
to child analyzer.
* origin/topic/awelzel/defer-more-stuff:
RecordType: Ensure &default fields are always re-initialized
Attr: Deprecate using &default and &optional together on record fields
RecordType: Allow deferring &default=vector(), set(), table() fields
This moves c$service_violation to the deprecated-dpd-log policy script.
This is the only script in the distribution that uses the field, and it
is unlikely to be used externally. It is also responsible for a
significant amount of memory use by itself.
This also restores the field being populated, which was broken in
GH-4362
The conn protocol scripts now assume that new_connection is run before
connection_state_remove. Update the DNS analyzer to raise the
new_connection event.
This class is a parameter of virtual methods of the Backend API for users
to implement and also a parameter to the HookPublishEvent() API. Seems it
shouldn't be in detail and instead we should own it.
Alternatively, could mark the cluster APIs as not-stable-yet, but I
think we can move forward and make it non-detail for 8.0.
* origin/topic/awelzel/4431-zeromq-drop-policy-v2:
cluster.bif: Improve Cluster::publish() docstring
btest/cluster/zeromq: Add tests for overload behavior
cluster/zeromq: Metric for msg errors
cluster/zeromq: Drop events when overloaded
cluster/zeromq: Comments and move lookups to InitPostScript()
cluster/zeromq: Rework lambdas to member functions
cluster/zeromq: Support local XPUB/XSUB hwm and buf configurability
cluster/OnLoop: Support DontBlock and Force flags for queueing
cluster/ThreadedBackend: Injectable OnLoopProcess instance
The overload-drop.zeek and overload-no-drop.zeek tests have proxy,
worker-1 and worker-2 publish to the manager topic. For the drop
case, we verify that both, the senders, but also the manager drops
events. For the no-drop test, the HWMs are set such that all events
are buffered.
The overload-worker-proxy-topic*.zeek tests are similar, but instead
of publishing to the manager topic, proxy, worker-1 and worker-2 publish
to the proxy and worker topics to overload each other. This had
previously resulted in lockups and these tests verify that this doesn't
happen anymore.
When either the XPUB socket's hwm is reached, or the onloop queue is
full, drop the events. Users can set ths xpub_sndhwm and
onloop_queue_hwm to 0 to avoid these drops at the risk of unbounded
memory growth.
Now that Conn::set_conn is guaranteed to be called at the beginning and
at the end of the connection, we can skip re-setting the elements that
we know will not have changed. This prevents repeated lookups, e.g. to
check that addresses are in the local networks.
During `connection_state_remove`, only the duration, number of
packets, service, and history fields are updated.
local_orig and local_resp are updated when the connection is flipped. A
test was added for that purpose. It uses the already existing
http.zeek-image-post-1080-8000-x.pcap, which was slightly rewritten for
this, so that one side of the connection has IP addresses different from
127.0.0.1.
The existing history-flip test also was updated to have one side being
in a local-net, to check that the flipping of local_orig and local_resp
works correctly at the beginning of a connection.
c$conn is often needed for connection events, but it being established
in connection_state_removed can be a problem because event handlers have
to remember to call Con::set_conn().
This commit moves to call Conn::set_conn() in new_connection.
Addresses GH-4202
update logs
* origin/topic/timw/lazy-allocate-id-option-handlers:
Move ID::type in structure to fill memory padding
Use sorted forward_list instead of multimap for ID option change handlers
* origin/topic/timw/cpp20-modernization:
Remove intermediate cipher vectors in ssl-analyzer.pac
Reduce the size of Func::Body quite a bit with some extra packing
Switch to using std::ranges algorithms
Enable modernize-std-numbers clang-tidy checker, fix findings
Switch to using c++20 constraints instead of std::enable_if
Not sure it's the best place to put, but we don't have packet analysis stats
bif and also num_packets is already there, so seems reasonable to put the
num_packets_unprocessed into
* origin/topic/awelzel/conn-id-ctx-singleton:
RecordType: Allow field init deferral of deferrable record constructors
Conn: Use conn_id_ctx singleton
Conn: Add InitPostScript() and conn_id_ctx singleton
ID: Add conn_id_ctx
The ctx: conn_id_ctx &default=conn_id_ctx() field was not optimized
as deferrable even though by default its an empty record and so deferring
initialization seems safe. Open-code the special record constructor
expression case in ExprFieldInit so that the ctx field is not default
initialized at record construction anymore. I am wondering a bit if the
same applies to &default=set() and &default=table().
The EDNS rcode was incorrectly calculated. The extended rcode is formed
by taking the upper 8 bits of the extended rcode field, plus the lower 4
bits of the existing rcode.
This also adds a new trace with an extended rcode, and a testcase
parsing it.
Reported by dwhitemv25.
Fixes GH-4656
When a conn_id represents a ICMP "connection", we need to determine the
is_one_way flag for InitTuple() in order to skip any flipping of address
and ports for one-way ICMP connections.
Fixes#4645
* origin/topic/timw/coverity-fixes:
Fix a few other minor issues reported by Coverity
Add a few extra null checks, plus a missing initialization that led to a bad null check
Fix some integer overflow issues reported by Coverity
Ignore a couple of known-unused results reported by Coverity
Fix some bit-shifting overflow/UB issues reported by Coverity
Reset the value of a status variable in SQLite backend before using it in a loop
Fix a potential memory leak reported by Coverity
Avoid some string copies in IRC analyzer
Add some additional std::moves reported by Coverity
Fix an unsigned integer comparison reported by Coverity
Fix uninitialized class member Coverity findings
Handle uncaught exception during setup
Update gen-zam submodule for Coverity findings
* origin/topic/timw/storage-forced-sync:
Fix swapped storage metrics names
Add flag to force synchronous mode when calling storage script-land functions
* origin/topic/awelzel/control-switch-to-cluster:
NEWS: ZeekControl, ZeroMQ and WebSocket
Update zeekctl module for ClusterBackend and UseWebSocket
control: Use Cluster::publish() for replying
This adds a new PacketAnalyzer::PPPoE::session_id bif, which extracts
the PPPoE session ID from the current packet.
Furthermore, a new policy script is added which adds the pppoe session
id to the connection log.
Related to GH-4602
Switching to ZeroMQ as cluster backend and dabbling with zeekctl
and WebSocket, replies didn't arrive due to the usage of
Broker::publish() rather than Cluster::publish(). Additionally,
add the node name to the topic on which we reply so that the
receiver can figure out which node sent the reply. It could've
been a separate event parameter, but the topic appears just fine.
This allows injecting a custom onloop process to configure the
max_queue_size at instantiation time. Also allow access to the
instance directly and deprecate the QueueForProcessing() helper
* origin/topic/timw/storage-metrics:
Add SQLite page_count and file_size metrics
Add btests to cover storage metrics
Add storage metrics for operations, expirations, data transferred
Fix ordering of telemtry metrics when running under test
Make RunPragma take an optional value parser to return data
Make SQLite::Step take a callback function for parsing result data
The util:: versions of these methods remain as a thin wrapper around them so
they can be used with const char* arguments. Otherwise callers have to manually
make string_view objects from the input.
s Please enter the commit message for your changes. Lines starting
* origin/topic/robin/gh-4481-test-analyzer:
Spicy: Fix missing include.
Bump Spicy.
Spicy: Add functions to check if Zeek provides an analyzer of a given name.
Specifically, set a MIME part's parent_id to the rfc822_msg_fuid if it
is set and take into account the current rfc822_msg_fuid for describe_file()
to avoid fuid collisions of the top-level RFC822 message and the first
MIME part.
* origin/topic/bbannier/sqlite-no-stale-entries:
Make PUT on SQLite backend implicitly overwrite expired entries
Prevent SQLite storage backend from serving expired entries
The backend does not serve expired but still present entries so to a
user they do not exist. When they put new data over such an entry their
expecation is that the value is overwritten, even if not explicitly
requested.
The SQLite storage backend implements expiration by hand and garbage
collection is done in `DoExpire`. This previously relied exclusively on
gets not running within `Storage::expire_interval` of the put, otherwise
we would potentially serve expired entries.
With this patch we explictly check that entries are not expired before
serving them so that the SQLite backend should never serve expired
entries.
```
## Checks if there is a Zeek analyzer of a given name.
##
## analyzer: the Zeek-side name of the analyzer to check for
## if_enabled: if true, only checks for analyzers that are enabled
##
## Returns the type of the analyzer if it exists, or ``Undef`` if it does not.
public function has_analyzer(analyzer: string, if_enabled: bool = True): bool &cxxname="zeek::spicy::rt::has_analyzer";
## Differentiates between the types of analyzers Zeek provides.
public type AnalyzerType = enum { Protocol, File, Packet, };
## Returns the type of a Zeek analyzer of a given name.
##
## analyzer: the Zeek-side name of the analyzer to check
## if_enabled: if true, only checks for analyzers that are enabled
##
## Returns the type of the analyzer if it exists, or ``Undef`` if it does not.
public function analyzer_type(analyzer: string, if_enabled: bool = True): AnalyzerType &cxxname="zeek::spicy::rt::analyzer_type";
```
Closes#4481.
* origin/topic/timw/std-filesystem:
Fix some missing #includes resulting from removal of ghc::filesystem
Remove ghc::filesystem submodule, switch to std::filesystem
* origin/topic/timw/cpp20-clang-tidy-updates:
Use ranges::reverse_view to fix a few reverse ranged-for loops
Disable a few new modernize clang-tidy checkers, enabled by C++20
While we support initializing records via coercion from an expression
list, e.g.,
local x: X = [$x1=1, $x2=2];
this can sometimes obscure the code to readers, e.g., when assigning to
value declared and typed elsewhere. The language runtime has a similar
overhead since instead of just constructing a known type it needs to
check at runtime that the coercion from the expression list is valid;
this can be slower than just writing the readible code in the first
place, see #4559.
With this patch we use explicit construction, e.g.,
local x = X($x1=1, $x2=2);
* topic/timw/c++20:
Reduce some false-positive warnings from GCC to warnings with -Werror
Add zeek-namespaced alias for jthread
Include RequireCXXStd.cmake from main CMakeLists.txt
Update submodules to C++20-enabled versions
Apple Clang (as of version 17.0.0) does not support jthread unless
you enable experimental libcpp features by defining
_LIBCPP_ENABLE_EXPERIMENTAL or passing -fexperimental_library to
the compiler. Even if you don't do those, our 3rdparty jthread
library fails to build because of a bunch of namespace/naming
collisions. I moved our 3rdparty one to the `nonstd` namespace, and
am using an alias under __APPLE__ to avoid the collisions.
Zeek's analyzer API makes it hard to determine during analyzer
shutdown whether a regular end-of-data has been reached, or if we're
aborting in the middle of a session (e.g., because Zeek missed the
remaining packets): the corresponding analyzer method, `EndOfData()`
gets called in both cases.
In an earlier change, we had stopped signaling Spicy analyzers a
regular finish when that `EndOfData()` method executes, because doing
so could trigger a parse error if it wasn't a regular shutdown—-which
isn't desired, a user request was to just silently stop processing in
this case.
However, that behavior now seems unfortunate in the case that one
deliberately calls `zeek::protocol_handle_close()` to terminate an
analyzer: this feels like a regular shutdown that should just
immediately happen. We achieve this now in this function by
additionally signaling the shutdown at the TCP layer as an "end of
file", which, for Spicy analyzers, happens to run the final, orderly
tear-down.
Not exactly great, but ti seems to thread the needle to achieve the
desired semantics in both cases.
pre-commit ignores Cargo.lock files for Rust projects, so any movement
in a Rust project's dependencies can break a hook, even if no code in
the hook changed. I have tried to work with upstream on a fix, but they
basically told me they weren't interested and to get lost.
This bumps the `spicy-format` pre-commit hook to a new version which
explicitly deals with bumps of its dependencies. Having to do this
semi-regularly is not fun, and ideally somebody interested in using this
hook would help set up infrastructure in the hook so it just pulls
pre-built binaries. This is not directly supported by pre-commit, but
many projects work around this by declaring a Python module which then
pulls pre-build binaries which already exist for spicy-format.
This changes the PPPoE parser so that it doesn't forward extra bytes
that might be appended after the payload. Instead, it raises a weird if
the payload size doesn't match the size indicated by the header.
This is in line with what other protocol parsers (like UDP) are doing.
Two tests needed to be updated - with this change, the traffic in
pppoe-over-qinq.pcap is now valid TLS. A new trace was introduced for
the confirmation-violation-info test.
Addresses GH-4602
* origin/topic/vern/line-number-ordering:
Bump ZeekJS to work with new Location constructor
remove non-functional column information from Location objects
isolate Location specifics to private class variables to enforce correct line number ordering
The .rst generation doesn't escape the trailing `_` and the docs build
gets upset due to using `type` as a reference target then.
For the better or worse, revert to using tpe. Though I acknowledge this
means we need to be careful with trailing underscores because our docs
build is so fragile.
Partly reverts b9eabbabba.
* origin/topic/awelzel/4605-conn-id-context:
NEWS: Adapt for conn_id$ctx introduction
conn_key/fivetuple: Drop support for non conn_id records
Conn: Move conn_id init and flip to IPBasedConnKey
IPBasedConnKey: Add GetTransportProto() helper
input/Manager: Ignore empty record types
external: Bump commit hashes for external suites
ip/vlan_fivetuple: Populate nested conn_id_context, not conn_id
ConnKey: Extend DoPopulateConnIdVal() with ctx
btest: Update tests and baselines after adding ctx to conn_id
init-bare: Add conn_id_ctx to conn_id
Previously, we supported any records that happened to have orig_h,
resp_h, etc. fields, but it's not exactly clear why we ever did. Users
that relied on this can instantiate an explicit conn_id instance, too.
This loosens the coupling of the script-layer conn_id record and
the code in Conn a bit, moving more into the IPBasedConnKey class.
I'm not quite sure whether moving the flipping logic is worth it,
but assuming Conn could become non-IP in the future, it might.
Somewhere record types with zero fields get the optional attribute
apparently. The input/sqlite/basic test failed due to complaining
that ctx is optional. It isn't optional and when it has zero fields
we can just ignore it, too.
Also adds a input framework test with an explicit empty record type
get_file_handle() may include c$id and perturbs their values when adding new
fields. I think that's reasonable, as files transferred in one VLAN should
be treated separate from files transferred in a different VLAN.
This prepares the move where ConnKey implementations should fill out
ctx rather than filling conn_id directly. The API continues to receive
both, conn_id and ctx, as adding fields to `conn_id` is reasonable
use-case even if it's just for logging purposes.
This nested record can be used to discriminate orig_h or resp_h being
observed in different "contexts". A context can be based on VLAN tags,
but any custom ConnKey implementation should populate the ctx field,
allowing to write context-aware Zeek scripts without needing to know
what the context really is.
Changes in endpoint stats are a side-effect caused by the ConnSize
analyzer updating the conn record triggering the threshold event. The
phenomenon is described in https://github.com/zeek/zeek/issues/4214.
* origin/topic/johanna/ssl-small-tweaks:
Spicy SSL analyzer: move unknown version out of possible range
Spicy SSL analyzer: move exceptions to &requires & throw
* origin/topic/etyp/redis-resp3:
Touchup TODOs in the Redis analyzer
Handle more Redis RESP3 protocol pieces
Stringify all Redis-RESP serialized data
Handle Redis protocol `message` separately
Add Redis analyzer array stringification
Closes#4504
Messages are not typical responses, so they need special handling. This
is different between RESP2 and 3, so this is the first instance where
the script layer needs to tell the difference.
* origin/topic/bbannier/readability-isolate-declaration:
Make clang-tidy warnings report as errors to cause CI build to fail
Fix a clang-tidy finding in cluster telemetry code
Fix clang-tidy readability-isolate-declaration warnings
I missed one of these in review so a machine is probably better at
catching them.
I fixed the existing instances which where largely in code which look
dated. Where possible I slightly reorganized the code so we do not have
to leave values uninitialized, but did not touch up anything else.
A bit ad-hoc formatting for the log, but that's mostly because cluster.log
only has message field and I don't think having a dedicated application_name
column is worth it. That could also be added by custom scripts if it's really
wanted for a given deployment.
* origin/topic/awelzel/cluster-telemetry-follow-up:
Bump cluster test suite
cluster/Telemetry: Cache CallExpr locations
cluster/Telemetry: Avoid unneeded StringVal() construction
Val: Switch TablePatternMatcher to std::string_view
RE: Add MatchAll() and MatchSet() for std::string_view
cluster/websocket: Fix and test for invalid X-Application-Name
cluster/telemetry: Move topic_normalization redef to zeromq
This moves the negotiated version outside of the possible range of
protocol values. I don't think it was possible to cause a problem with
this in the past - but this approach seems safer.
The Spicy based SSL analyzer was, so far, more permissive with the
record layer versions that it would accept.
This change brings the parsing of record layer versions in line with the
binpac based analyzer. This behavioral difference was discovered due to
a test that changed with the recent dpd log changes.
* origin/topic/awelzel/1474-cluster-telemetry:
btest/cluster/telemetry: Add smoke testing for telemetry
cluster/WebSocket: Fetch X-Application-Name header as app label
cluster/WebSocket: Pass X-Application-Name to dispatcher
broker/WebSocketShim: Add calls to Telemetry hooks
cluster/WebSocket: Configure telemetry for WebSocket backends
broker: Hook up generic cluster telemetry
cluster: Introduce telemetry component
One bug fix removing static from a variable that shouldn't be static.
* origin/topic/christian/extensible-conntuples:
btest/plugins: Add test for custom ConnKey factory
NEWS updates for pluggable connection tuples.
Add a VLAN-aware flow tuple implementation.
Deprecate ConnTuple and related APIs.
Deprecate the old Connection constructor and detail::ConnKey class.
Switch to virtualized use of new zeek::ConnKey class tree
Provide a connkey factory for Zeek's default five-tuples.
Add IP-specific ConnKey implementation.
Establish plugin infrastructure for ConnKey factories.
Add new ConnKey abstraction.
Given IP-aware ConnKeys, ConnTuples aren't really required any more. ConnTuple
had two benefits:
- It preserved the original src/dst orientation from the packet headers it was
based on, which IPBasedConnKey now tracks and provides accessor methods for.
- In IPBasedAnalyzer::AnalyzePacket() its instance survived past the std:move()
of the key into NewConn(), which we sidestep by keeping the original src address
and port around until we need after the connection is obtained.
Since the base factory is pure virtual this is now the first full
implementation, but still a bit of a special case because it implements Zeek's
default behavior and doesn't add "custom" content to the tuple.
Using a label named "endpoint" is not intuitive and requires explaining to
users that it's really just the Cluster::node value. Change the label to
"node", so that we don't need to do the explaining.
This probably breaks some existing users of the Prometheus metrics, but after
looking more at metrics recently, "endpoint" really is a thorn in my eye.
This commit fixes the parsing of the data field in the SSL analyzer. So
far, this field contained two extra bytes at the beginning, which
contain the length of the following data.
Now, the data passed to the event only contains the actual value of the
session ticket.
The Spicy analyzer already contains the correct handling of this field,
and does not need to be updated. A test that uses the event and
exhibited the bug was added.
* origin/topic/awelzel/4586-zeromq-ipv6:
cluster/zeromq: Short-circuit DoPublishLogWrite() when not initialized
cluster/zeromq: Hook up and enable IPV6 by default
cluster/zeromq/connect: Make failures fatal
cluster/zeromq: Move log_push creation to DoInit()
After moving the log_push initialization from the constructor to the
DoInit() method, it's now possible that DoPublishLogWrites() is invoked
even if DoInit() was never called. Handle this by short-circuiting. This
is sort of an error, but can happen during tests if scripts are loaded
somewhat arbitrarily.
ZeroMQ's IPv6 support isn't enabled by default, resulting in
"No such device" errors when attempting to listen on an IPv6
address. This change adds a ipv6 option to the ZeroMQ module
and enables it by default. Further, adds a test configuring
everything to listen on IPv6 ::1 as well, and one test to provoke
the original error. This also regularizes some error messages.
The addr_to_uri() calls weren't actually needed, but they apparently do
not hurt and the result is easier on the eyes, so use them :-)
* origin/topic/johanna/default-canonifier-only-first-timestamp:
Default canonifier change to only remove first timestamp in line
Align SMB timestamp calculation between operating systems
This one is weird. This checker complains because we tend to use PList,
which sets the type of T to a pointer. The checker is making note that
we're doing sizeof() on a pointer, which always returns the same value
every time. It's asking whether we meant to do that, or if we meant to
pass the actual type to sizeof.
The schema of cluster WebSocket error messages deviated from the
existing one in Broker which breaks seamless migration from the Broker
WebSocket bindings.
This patch aligns the serialization in cluster with the one in Broker.
This is technically a breaking change of the cluster schema, but since
it never worked like documented and is still experimental this is
probably fine.
Closes#4594.
The cluster is borked if the initialization fails, so may as well just
completely abort Zeek at that point with a fatal error. There's no real
point in continuing to run.
In the past, we used a default canonifier, which removes everything that
looks like a timestamp from log files. The goal of this is to prevent
logs from changing, e.g., due to local system times ending up in log
files.
This, however, also has the side-effect of removing information that is
parsed from protocols which probably should be part of our tests.
There is at least one test (1999 certificates) where the entire test
output was essentially removed by the canonifier.
GH-4521 was similarly masked by this.
This commit changes the default canonifier, so that only the first
timestamp in a line is removed. This should skip timestamps that are
likely to change while keeping timestamps that are parsed
from protocol information.
A pass has been made over the tests, with some additional adjustments
for cases which require the old canonifier.
There are some cases in which we probably could go further and not
remove timestamps at all - that, however, seems like a follow-up
project.
The way that SMB timestamps were calculated used a "long double" for
the conversion calculation.
OS-X and Linux seem to have different sizes for long doubles. On the
Linux distributions that I have available, a long double is 128bits, vs
64 bits in OS-X.
This leads to slight discrepancies in the timestamps. This commit
changes this calculation to use a double on both systems, which is 64
bits and eliminates this difference.
Not the not_before/not_after fields output GMT based times.
Also adds a new btest diff canonifier which only removes the first
timestamp in a line.
Fixes GH-4521
The Spicy SSL analyzer was using references pretty heavily, probably to
work around now fixed issues with `inout` parameters in older Spicy
versions. At least for units this seems not needed anymore, and was also
partially incorrect, e.g., the Spicy docs call out that when using
`inout` parameters, passed and expected types should match exactly so
passing a reference as an `inout` value seems incorrect. Additionally,
one use case for references in Spicy is to use their interior
mutability, i.e., a reference never needs to be passed `inout` since
their can always be mutated.
Internally units are stored as reference-counted values, and references
to units are not much cheaper since they also need to be refcounted.
With that there seems litle reason to use references in this analyzer at
all, and this patch drops their use completely; instead we either pass
values, or values declared `inout`.
We leave the use of references for sharing sinks in place.
When looking up the postprocessor function from shadow files, id::find_func()
would abort() if the function wasn't available instead of falling back
to the default postprocessor.
Fix by using id::find() and checking the type explicitly and also adding a
strict type check while at it.
This issue was tickled by loading the json-streaming-logs package,
Zeek creating shadow files containing its custom postprocessor function,
then restarting Zeek without the package loaded.
Closes#4562
* origin/topic/timw/util-types:
Remove some unused #includes from spicy code
Remove using util.h in various headers in favor of util-types.h
Move type definitions/aliases from util.h to a separate file
After a public discussion and also chatting with Vern directly, disable the
ZAM bif tracking test to avoid an update every time new functions are
added. Usually these aren't performance critical and the defaults
characterization is fine. If they are performance critical, then Vern
is currently best positioned to properly integrate an optimized version.
* topic/christian/btest-trace-cleanup:
Btests: don't use -C in Zeek invocations that don't actually need it
Remove executable file permission bits from a bunch of our pcaps
The response to BDAT LAST was never recognized, resulting in the
BDAT LAST commands not being logged in a timely fashion and receiving
the wrong status.
This likely doesn't handle complex pipeline scenarios, but it fixes
the wrong behavior for smtp_reply() not handling simple BDAT commands
responses.
Thanks @cccs-jsjm for the report!
Closes#4522
Add `net-tools` and `procps` to the final.Dockerfile, to avoid warning in
zeekctl commands like: "failed to find local IP addresses [...]" and the
error in `zeekctl top` command caused respectively by missing `ifconfig`
and `top` commands.
Nobody generates docs at startup while then moving on to regular Zeek operation,
and the generated runtime output when loading scripts/zeekygen is both noisy and
confusing.
* origin/topic/timw/storage-expire-contention:
Add busy_timeout script-level option, override any busy_timeout pragma
Handle potential contention when running sqlite expiration
Add expiration to sqlite-cluster.btest
Use unique_ptr to avoid needing to call sqlite3_reset manually
Move Deferred class from ZeroMQ to util
* origin/topic/johanna/new-style-analyzer-log:
NEWS entries for analyzer log changes
Move detect-protocol from frameworks/dpd to frameworks/analyzer
Introduce new c$failed_analyzers field
Settle on analyzer.log for the dpd.log replacement
dpd->analyzer.log change - rename files
Analyzer failure logging: tweaks and test fixes
Introduce analyzer-failed.log, as a replacement for dpd.log
Rename analyzer.log to analyzer.debug log; move to policy
Move dpd.log to policy script
Currently, coverage/bare-mode-errors is one of the slowest tests in the
entire test suite. This is caused by the fact that it has to repeatedly
launch Zeek for every script that we ship. This is done sequentially.
This commit changes this test to use xargs to spawn 20 parallell
processes.
detect-protocol.zeek was the last non-deprecated script left in
policy/frameworks/dpd. It was moved to policy/frameworks/analyzer. A
script that loads the script from the new location with a deprecation
warning was added.
This field is used internally to trace which analyzers already had a
violation. This is mostly used to prevent duplicate logging.
In the past, c$service_violation was used for a similar purpose -
however it has slightly different semantics. Where c$failed_analyzers
tracks analyzers that were removed due to a violation,
c$service_violation tracks violations - and doesn't care if an analyzer
was actually removed due to it.
* origin/topic/timw/4350-redis-passwords:
Redis: Add support for sending AUTH commands during connection
Redis: disconnect cleanly if INFO request fails
Fix segfault if storage sync open_backend returns bad code
Add ToStdString and ToStdStringView to ZeekString
To address review feedback in GH-4362: rename analyzer-failed-log.zeek
to loggig.zeek, analyzer-debug-log.zeek to debug-logging.zeek and
dpd-log.zeek to deprecated-dpd-log.zeek.
Includes respective test, NEWS, etc updates.
The main part of this commit are changes in tests. A lot of the tests
that previously relied on analyzer.log or dpd.log now use the new
analyzer-failed.log.
I verified all the changes and, as far as I can tell, everything
behaves as it should. This includes the external test baselines.
This change also enables logging of file and packet analyzer to
analyzer_failed.log and fixes some small behavior issues.
The analyzer_failed event is no longer raised when the removal of an
analyzer is vetoed.
If an analyzer is no longer active when an analyzer violation is raised,
currently the analyzer_failed event is raised. This can, e.g., happen
when an analyzer error happens at the very end of the connection. This
makes the behavior more similar to what happened in the past, and also
intuitively seems to make sense.
A bug introduced in the failed service logging was fixed.
Analyzer-failed.log is, essentially, the replacement for dpd.log. The
name should make more sense, as it does now log analyzer failures. For
protocol analyzers specifically, these are failures that lead to the
analyzer being disabled.
The current analyzer.log is more useful for debugging than for
operational purposes. Hence this is disabled by default, moved to a
policy script, and the log is renamed to analyzer-debug.log.
Furthermore, logging of analyzer confirmations and disabling analyzers
are now enabled by default.
This is the first phase of moving from the current dpd log to a more
modern logfile, without some of the weirdnesses that the current dpd log
contains.
Tests will not pass in the current state; this is just splitting out
functionality.
Something is off with the libhiredis package that's installed
as part of the current image, and 'zypper patch' is failing to
update it during the prepare step. Force a rebuild so the
updated package is part of the image to start with.
* origin/topic/awelzel/4177-4178-custom-event-metadata-part-2:
Event: Bail on add_missing_remote_network_timestamp without add_network_timestamp
btest/plugin: Test custom metadata publish
NEWS: Add note about generic event metadata
cluster: Remove deprecated Event constructor
cluster: Remove some explicit timestamp handling
broker/Manager: Fetch and forward all metadata from events
Event/init-bare: Add add_missing_remote_network_timestamp logic
cluster/Backend/DoProcessEvent: Use generic metadata, not just timestamps
cluster/Event: Support moving args and metadata from event
cluster/serializer/broker: Support generic metadata
cluster/Event: Generic metadata support
Event: Use -1.0 for undefined/unset timestamps
cluster: Use shorter obj_desc versions
Desc: Add obj_desc() / obj_desc_short() overloads for IntrusivePtr
Also use the generic metadata version for publishing, keep the
ts-based API for now, but only add timestamps when
EventMetadata::add_network_timestamp is T. I'm not sure what the
right way forward here is, maybe deprecating Broker's publish event
variations and funneling through cluster.
Instead of a timestamp attribute, switch to holding a EventMetadataVectorPtr
like zeek::Event instances do. Keep the old constructor until the end of
the patch series.
This can happen if either there's no network timestamp associated with
an event, or there's currently no event being dispatched. Using 0.0
isn't great as it's the normal start timestamp before reading a network
packet. Using -1.0 gives the caller a chance to check and realize what's
going on.
* origin/topic/vern/CPP-maint.May25:
minor BTest maintenance updates for -O gen-C++
fix for more robustly finding BTests to assess for -O gen-C++
fix for -O gen-C++ dealing with type constants of unnamed compound types
This change adds two new hooks to the Intel framework that can be used
to intercept added and removed indicators and their type.
These hooks are fairly low-level. One immediate use-case is to count the
number of indicators loaded per Intel::Type and enable and disable the
corresponding event groups of the intel/seen scripts.
I attempted to gauge the overhead and while it's definitely there, loading
a file with ~500k DOMAIN entries takes somewhere around ~0.5 seconds hooks
when populated via the min_data_store store mechanism. While that
doesn't sound great, it actually takes the manager on my system 2.5
seconds to serialize and Cluster::publish() the min_data_store alone
and its doing that serially for every active worker. Mostly to say that
the bigger overhead in that area on the manager doing redundant work
per worker.
Co-authored-by: Mohan Dhawan <mohan@corelight.com>
* origin/topic/vern/ZAM-maint.May25:
fix for crash when interpreting transformed ASTs that include multi-field record assignments/additions
Remove unused ZAM compiler method
* origin/topic/timw/ci-clang-tidy:
CI: Add new task to run clang-tidy as part of nightly builds
CI: Update to clang 19 on ubuntu 24.04, add clang-tidy package
This only changes the script-layer API, but keeps the std::string host
in the C++ layer's ServerOptions. Mostly because the ixwebsocket library
takes host as std::string. Also, maybe at some point we'd want to
support something scheme-based like unix:///var/run/zeek.sock and placing
that in a string could not be totally wrong.
Add tests for IPV6, too.
* origin/topic/timw/coverity:
Add a few std::moves to fix Coverity warnings
Fix possible memory leak in broker::Manager
fixup! Use SCN* constants when reading into sized integer types
from_json: Statically lookup err index in result type
Use SCN* constants when reading into sized integer types
Remove setting unused variable value in FTP::parse_eftp
This one renames the Modbus Exception binpac type to ExcResponse. clang-tidy
insisted that anything named started with Exception was actually an exception
and needed to be thrown.
One instance of this method is noexcept and one isn't. The version
that is noexcept uses std::vector::emplace_back, which may throw
exceptions. Instead of adding a try/catch block, opt for just making
the two functions able to throw exceptions.
This fixes a clang-tidy bugprone-exception-escape warning.
* origin/topic/etyp/redis-analyzer:
spicy-redis: Add NEWS entry
spicy-redis: Separate error replies from success
spicy-redis: Cleanup scripts and tests
spciy-redis: Bring Redis analyzer into Zeek proper
spicy-redis: Abort parsing if server data comes first
spicy-redis: Add recursion depth to server data
spicy-redis: Make client data only accept bulk strings
spicy-redis: Add dpd signature and clean pcaps
spicy-redis: Add some commands and touch up parsing
spicy-redis: Add some script logic for logging
spicy-redis: Separate client/server
spicy-redis: Touchup logging and Spicy issues
spicy-redis: Add synchronization and pipeline support
spicy-redis: Begin Spicy Redis analyzer
- Recomputes checksums for pcaps to keep clean
- Removes some tests that had big pcaps or weren't necessary
- Cleans up scripting names and minor points
- Comments out Spicy code that causes a build failure now with a TODO to
uncomment it
Redis seems to only want client data first to request server data. The
DPD signature seems to pick up on some cases where server data comes
first, but is otherwise "valid" RESP. See if this helps lower FP rates.
* origin/topic/awelzel/fix-no-zero-timestamp-metadata:
btest: Add test for Cluster::hello zero-timestamp
EventMgr/Enqueue: Add automatic timestamp metadata to local events, only
cluster and broker: Propagate zero-timestamp as metadata, too.
It seems less surprising if only local events receive automatic network
timestamp metadata. For remote events the automatic value will most
likely be misleading.
This will be cleaned up later to just pass all contained metadata from
a cluster event to the queued event, but for now do this here, otherwise
we break some internal tests.
* origin/topic/awelzel/4177-4178-custom-event-metadata-part-1:
Event: Move meta after args
Event: Use IntrusivePtr to manage obj refcount
btest/zam: Update for new EventMetadata bifs
broker and cluster: Switch to new Enqueue() API
Event/zeek.bif: Add EventMetadata current() and current_values() accessors
Event: Deprecate default network timestamp metadata
Event: Store timestamp in metadata vector
EventRegistry/zeek.bif/init-bare: Add event metadata infrastructure
EventMgr: Add CurrentEvent() accessor
This deprecates the Event constructor and the ``ts`` parameter of Enqueue()
Instead, versions are introduced that take a detail::MetadataVectorPtr which
can hold the network timestamp metadata and is meant to be allocated by the
caller instead of automatically during Enqueue() or within the Event
constructor.
This also introduces a BifConst ``EventMetadata::add_network_timestamp`` to
opt-in adding network timestamps to events globally. It's disabled by
default as there are not a lot of known use cases that need this.
This removes the ts attribute from Event and instead allocates a vector for
storing metadata. By default, adds the network time as a TimeVal. Later
patches will make the allocation of the vector optional by introducing a
different constructor so that users that are not interested in network
timestamp metadata do not take the allocation hit.
Moving the explicit ``ts`` out of the event is done in order to treat it
just as generic metadata, too. However, the Time() accessor is adapted to
lookup the value from the metadata vector instead.
* origin/topic/timw/sqlite-cluster-test-followup:
SQLite: Add TODO note about possibly using sqlite3_busy_timeout
SQLite: Fix typo in variable name causing pragmas not to retry on busy
SQLite: Use tableval iteration instead of ToMap for pragmas
SQLite: Fix logging/error messages around executing pragmas
* origin/topic/timw/sqlite-cluster-test:
SQLite: Move integrity_check to pragma table
SQLite: Add backend option for pragma timeout
SQLite: Rename tuning_params to pragma_commands, move running pragmas to utility method
SQLite: Retry pragma statements at startup to avoid contention
SQLite: Check for locked database as well as busy databases
SQLite: Fix some string-sizing issues
SQLite: Run pragmas on connection before creating table
SQLite: Add busy_timeout pragma to default options
Prefix sqlite-based btests with sqlite- to match redis tests
Add sqlite cluster storage btest
* origin/topic/timw/3913-parse-port-invalid-read:
Add extra input files to ftp fuzzer corpus
Use bool instead of int flag in FTP analyzer's parse_eftp method
Fix undefined behavior in FTP analyzer's parse_port method
Fix invalid-read in FTP analyzer's parse_port method
* origin/topic/awelzel/btest-cluster-layout-replacement:
btest remaining: Use generic cluster-layout.zeek
btest/frameworks/logging: Use generic cluster-layout.zeek
btest/files/x509: Use generic cluster-layout.zeek
btest/frameworks/notice: Use generic cluster-layout.zeek
btest/policy: Use generic cluster-layout.zeek
btest/frameworks/intel: Use generic cluster-layout.zeek
btest/frameworks/sumstats: Use generic cluster-layout.zeek
btest/frameworks/cluster: Use generic cluster-layout.zeek
btest/frameworks/config: Use generic cluster-layout.zeek
btest/broker: Use generic cluster-layout.zeek
btest/cluster: Use generic cluster-layout.zeek
testing/btest/Files: Add generic broker/cluster-layout.zeek
This is a cluster-layout.zeek template that can be copied into a testing
directory if needed. The idea is that a developer sets environment
variables within their btest and the Cluster::nodes variable is
implicitly extended by appropriate nodes.
For example, using @TEST-PORT BROKER_LOGGER1_PORT will add an appropriate
logger-1 node to Cluster::nodes, based on the existence of the
BROKER_LOGGER1_PORT environment variable.
* origin/topic/vern/http-sqli-replacement:
site/local: Switch to detect-sql-injection
Add a revised script for detecting HTTP SQL injection, deprecate original
New test triggered the following error:
runtime error: downcast of address 0x57021a323ea0 which does not point to an object of type 'const FileVal' 0x57021a323ea0: note: object is of type 'zeek::FuncVal'
* 'smoot-improve-from_json' of github.com:/stevesmoot/zeek:
update baseline for zam
Update src/zeek.bif
Change from_json to return an error rather than print it.
* origin/topic/timw/include-cleanup-part-2:
Remove unnecessary #includes in script_opt
Remove unnecessary #includes in telemetry and supervisor
Remove unnecessary #includes in cluster/broker/iosource/probabilistic/session
Remove unnecessary #includes in zeekygen and base plugin files
Remove unnecessary #includes in input/logging/threading
Remove unnecessary #includes in analyzer/packet analyzer/file analyzer source files
Remove unnecessary #includes in analyzer/packet analyzer/file analyzer headers
Remove unnecessary #includes in base files in repo
Coverity complains about a missing fclose() in a non-existing
destructor. Also sprinkle in a strerror() call for fopen() to
provide a bit of a hint what might have gone wrong.
* origin/topic/timw/include-cleanup-part-1:
Remove including <cinttypes> from util.h
Remove telemetry #includes from OpaqueVal.h
Reduce includes in plugin/Component.h
Remove zeek/Stats.h include from NetVar.h
Include StmtBase/StmtEnums in Func.h instead of Stmt.h
Use modern names for standard headers
Remove fix for CentOS 7 from TCP_Flags.h
Fix usage of std::string in http analyzer
Reorder top section of net_util.h to batch includes together
Use quotes instead of <> for zeek includes
Fix Obj.h include in IntrusivePtr.h to have full path
* origin/topic/awelzel/4440-websocket-pong-timeout:
btest/cluster/websocket: Update tests for new event signature
cluster/websocket: Propagate code and reason to websocket_client_lost()
cluster/websocket: Support configurable ping interval
IXWebSocket: Bump to improve O^2 rxbuf erase()
* origin/topic/bbannier/fix-spicy-main:
Fix clang-tidy `bugprone-inc-dec-in-conditions` report in Spicy plugins glue compiler
Fix clang-tidy `performance-enum-size` reports in Spicy plugin's glue compiler
Remove unneeded copies in Spicy plugin glue compiler
Adjust for renamed function flavor in Spicy development version
With zeek/spicy#2048 the flavor for functions was renamed from
`Standard` to `Function`. This patch adapts the code for that while
still allowing using earlier Spicy versions.
* origin/topic/awelzel/ci-fedora-42:
probabilistic/BitVector: Add include <cstdint>
Bump spicy to fix build with GCC 15.1
CI: Drop fedora-40
CI: Add fedora-42
When a node restarts or a peering between two nodes starts over for other
reasons, the internal tracking in the Broker manager resets its state (since
it's per-peering), and thus the message overflow counter. The script layer was
unaware of this, and threw errors when trying to reset the corresponding counter
metric down to zero at sync time.
We now track past buffer overflows via a separate epoch table, using Broker peer
ID comparisons to identify new peerings, and set the counter to the sum of past
and current overflows.
I considered just making this a gauge, but it seems more helpful to be able to
look at a counter to see whether any messages have ever been dropped over the
lifetime of the node process.
As an aside, this now also avoids repeatedly creating the labels vector,
re-using the same one for each metric.
Thanks to @pbcullen for identifying this one!
I believe there's a bug/usage issue in the websockets library
where during send(), EOF is detected and stored, but the receiving
thread is then discarding the last received frame. Avoid the bug
by replacing the close_socket() implementation of the websockets
library just for that test and leave detecting the EOF condition
to the receiving thread.
The terminate-while-queueing test added for #4428 failed spuriously
indicating that sometimes WebSocket clients receive code 1000 instead of 1001.
This happens if the ixwebsocket server is shutdown before the reply thread had a
chance to process queued close messages.
Fix by signaling and waiting for the dispatcher's reply thread to terminate
before returning from Terminate().
In GH-4422 it was pointed out that the protocols/conn/failed-service-logging.zeek
policy script only works when
`DPD::track_removed_services_in_connection=T` is set.
This was caused by a logic error in the script. This commit fixes this
logic error and introduces an additional test that checks that
failed-service-logging works even when the option is not set to true.
Terminate() is called when Zeek shuts down. If WebSocket client threads
were blocked in QueueForProcessing() due to reaching queue limits, these
previously would not exit QueueForProcessing() and instead block
indefinitely, resulting in the ixwebsocket library blocking and its
garbage collection thread running at 100%. Not great.
Closing the onloop instance will unblock the WebSocket client threads
for a timely shutdown.
Closes#4420
* origin/topic/timw/storage-check-redis-server-version:
Add commands to the static methods for the Redis implementation
Redis: Check server version when connecting
Ensure the client side also uses the initial destination connection ID
for decryption purposes instead of the one from the current long header
packet. PCAP from local WiFi hotspot.
Instead of sending the accumulated CRYPTO frames after processing an
INITIAL packet, add logic to determine the total length of the TLS
Client or Server Hello (by peeking into the first 4 byte). Once all
CRYPTO frames have arrived, flush the reassembled data to the TLS
analyzer at once.
The payload is already consumed within the InitialPacket unit. Consuming
it again resulted in UDP datagrams with multiple packets to ignore
the remaining packets in the same UDP datagram. The baseline changes
showing I being followed by a new H indicates that the INITIAL packet
was followed by a HANDSHAKE packet, but previously Zeek discarded
these.
* origin/topic/awelzel/event-publish-hook:
NEWS: Add HookPublishEvent() note
btest/plugin: Test for PublishEventHook()
broker and cluster: Wire up HookPublishEvent
plugin: Add HookPublishEvent hook
Vern didn't have ZeroMQ installed and the test was skipped for him.
Generally would recommend anyone working on core Zeek to install
libzmq-dev or the equivalent for their environment, but until it is a
real required dependency, loosen the requirements on the test.
* origin/topic/timw/update-ct-ca-lists:
External tests: add removed logs to CT list to prevent baseline changes
Update Mozilla CA list and CT list to NSS 3.110
Instead of simply removing holes from vectors or lists when converting
from Val to Broker format, error out as the receiver has no chance to
reconstruct where the hole might have been.
We could encode holes with broker::none, but this will put unnecessary
burden on language bindings and users due to the potential optionality.
Think a std::vector<uint64_t> that technically needs to be a
std::vector<std::optional<uint64_t>> to represent optional elements
properly.
Closes#3045
Coverity complains proc is set under a lock, but accessed in Process()
without a lock. Fix this by setting it in Close() also without locking.
The proc member should only ever be accessed my the main thread.
--with-bifcl=PATH path to Zeek BIF compiler executable
(useful for cross-compiling)
--with-bind=PATH path to BIND install root
--with-binpac=PATH path to BinPAC executable
(useful for cross-compiling)
--with-bison=PATH path to bison executable
--with-bison=PATH path to bison executable
--with-broker=PATH path to Broker install root
--with-broker=PATH path to Broker install root
(Zeek uses an embedded version by default)
(Zeek uses an embedded version by default)
--with-gen-zam=PATH path to Gen-ZAM code generator
(Zeek uses an embedded version by default)
--with-flex=PATH path to flex executable
--with-flex=PATH path to flex executable
--with-libkqueue=PATH path to libkqueue install root
--with-libkqueue=PATH path to libkqueue install root
(Zeek uses an embedded version by default)
(Zeek uses an embedded version by default)
@ -328,9 +321,6 @@ while [ $# -ne 0 ]; do
--disable-af-packet)
--disable-af-packet)
append_cache_entry DISABLE_AF_PACKET BOOL true
append_cache_entry DISABLE_AF_PACKET BOOL true
;;
;;
--disable-archiver)
has_disable_archiver=1
;;
--disable-auxtools)
--disable-auxtools)
append_cache_entry INSTALL_AUX_TOOLS BOOL false
append_cache_entry INSTALL_AUX_TOOLS BOOL false
;;
;;
@ -371,15 +361,9 @@ while [ $# -ne 0 ]; do
--disable-zkg)
--disable-zkg)
append_cache_entry INSTALL_ZKG BOOL false
append_cache_entry INSTALL_ZKG BOOL false
;;
;;
--with-bifcl=*)
append_cache_entry BIFCL_EXE_PATH PATH $optarg
;;
--with-bind=*)
--with-bind=*)
append_cache_entry BIND_ROOT_DIR PATH $optarg
append_cache_entry BIND_ROOT_DIR PATH $optarg
;;
;;
--with-binpac=*)
append_cache_entry BINPAC_EXE_PATH PATH $optarg
;;
--with-bison=*)
--with-bison=*)
append_cache_entry BISON_EXECUTABLE PATH $optarg
append_cache_entry BISON_EXECUTABLE PATH $optarg
;;
;;
@ -392,9 +376,6 @@ while [ $# -ne 0 ]; do
--with-flex=*)
--with-flex=*)
append_cache_entry FLEX_EXECUTABLE PATH $optarg
append_cache_entry FLEX_EXECUTABLE PATH $optarg
;;
;;
--with-gen-zam=*)
append_cache_entry GEN_ZAM_EXE_PATH PATH $optarg
;;
--with-geoip=*)
--with-geoip=*)
append_cache_entry LibMMDB_ROOT_DIR PATH $optarg
append_cache_entry LibMMDB_ROOT_DIR PATH $optarg
;;
;;
@ -510,8 +491,3 @@ eval ${cmake} 2>&1
echo "# This is the command used to configure this build" >config.status
echo "# This is the command used to configure this build" >config.status
echo $command >>config.status
echo $command >>config.status
chmod u+x config.status
chmod u+x config.status
if [ $has_disable_archiver -eq 1 ]; then
echo
echo "NOTE: The --disable-archiver argument no longer has any effect and will be removed in v7.1. zeek-archiver is now part of zeek-aux, so consider --disable-auxtools instead."
##! Activates port-independent protocol detection and selectively disables
##! Disables analyzers if protocol violations occur, and adds service information
##! analyzers if protocol violations occur.
##! to connection log.
@load ./main
module DPD;
module DPD;
export {
export {
## Add the DPD logging stream identifier.
## Analyzers which you don't want to remove on violations.
redef enum Log::ID += { LOG };
## A default logging policy hook for the stream.
global log_policy: Log::PolicyHook;
## The record type defining the columns to log in the DPD logging stream.
type Info: record {
## Timestamp for when protocol analysis failed.
ts: time &log;
## Connection unique ID.
uid: string &log;
## Connection ID containing the 4-tuple which identifies endpoints.
id: conn_id &log;
## Transport protocol for the violation.
proto: transport_proto &log;
## The analyzer that generated the violation.
analyzer: string &log;
## The textual reason for the analysis failure.
failure_reason: string &log;
};
## Deprecated, please see https://github.com/zeek/zeek/pull/4200 for details
option max_violations: table[Analyzer::Tag] of count = table() &deprecated="Remove in v8.1: This has become non-functional in Zeek 7.2, see PR #4200" &default = 5;
## a: an address string on which to accept connections, e.g.
## "127.0.0.1". An empty string refers to INADDR_ANY.
##
## p: the TCP port to listen on. The value 0 means that the OS should choose
## the next available free port.
##
## retry: If non-zero, retries listening in regular intervals if the port cannot be
## acquired immediately. 0 disables retries. If the
## ZEEK_DEFAULT_LISTEN_RETRY environment variable is set (as number
## of seconds), it overrides any value given here.
##
## Returns: the bound port or 0/? on failure.
##
## .. zeek:see:: Broker::status
global listen_websocket: function(a: string &default = default_listen_address_websocket,
p: port &default = default_port_websocket,
retry: interval &default = default_listen_retry): port
&deprecated="Remove in v8.1. Switch to Cluster::listen_websocket() instead.";
## Initiate a remote connection.
## Initiate a remote connection.
##
##
## a: an address to connect to, e.g. "localhost" or "127.0.0.1".
## a: an address to connect to, e.g. "localhost" or "127.0.0.1".
@ -450,29 +424,6 @@ export {
##
##
## Returns: true if a new event forwarding/subscription is now registered.
## Returns: true if a new event forwarding/subscription is now registered.
global forward: function(topic_prefix: string): bool;
global forward: function(topic_prefix: string): bool;
## Automatically send an event to any interested peers whenever it is
## locally dispatched. (For example, using "event my_event(...);" in a
## script.)
##
## topic: a topic string associated with the event message.
## Peers advertise interest by registering a subscription to some
## prefix of this topic name.
##
## ev: a Zeek event value.
##
## Returns: true if automatic event sending is now enabled.
global auto_publish: function(topic: string, ev: any): bool &deprecated="Remove in v8.1. Switch to explicit Cluster::publish() calls. Auto-publish won't work with all cluster backends.";
## Stop automatically sending an event to peers upon local dispatch.
##
## topic: a topic originally given to :zeek:see:`Broker::auto_publish`.
##
## ev: an event originally given to :zeek:see:`Broker::auto_publish`.
##
## Returns: true if automatic events will not occur for the topic/event
## pair.
global auto_unpublish: function(topic: string, ev: any): bool &deprecated="Remove in v8.1. See Broker::auto_publish()";
}
}
@load base/bif/comm.bif
@load base/bif/comm.bif
@ -514,31 +465,6 @@ function listen(a: string, p: port, retry: interval): port
