* topic/awelzel/3112-log-suffix-left-over-log-rotation:
cluster/logger: Fix leftover-log-rotation in multi-logger setups
cluster/logger: Fix global var reference
Populating log_metadata during zeek_init() is too late for the
leftover-log-rotation functionality, so do it at script parse time.
Also, prepend archiver_ to the log_metadata table and encoding function
due to being in the global namespace and to align with the
archiver_rotation_format_func. This hasn't been in a released
version yet, so fine to rename still.
Closes#3112
* origin/topic/awelzel/cluster-at-if-removal:
test-all-policy: Do not load nodes-experimental/manager.zeek
cluster/main: Remove extra @if ( Cluster::is_enabled() )
These have been discussed in the context of "@if &analyze" [1] and
am much in favor for not disabling/removing ~100 lines (more than
fits on a single terminal) out from the middle of a file. There's no
performance impact for having these handlers enabled unconditionally.
Also, any future work on "@if &analyze" will look at them again which
we could also skip.
This also reverts back to the behavior where the Cluster::LOG stream
is created even in non cluster setups like in previous Zeek versions.
As long as no one writes to it there's essentially no difference. If
someone does write to Cluster::LOG, I'd argue not black holing these
messages is better. Schema generators using Log::active_streams will
continue to discover Cluster::LOG even if they run in non-cluster
mode.
https://github.com/zeek/zeek/pull/3062#discussion_r1200498905
After the introduction of @if ... analyze, a lot of warnings were
triggered due to nested @if and @if .. analyze usage.
Add a test for coverage of all policy scripts in cluster mode
for the usual node types so this does not happen again.
* origin/topic/vern/at-if-analyze:
updates reflecting review comments
change base scripts to use run-time if's or @if ... &analyze
a number of BTests updated with @if ... &analyze
update for scripting coverage BTest demonstrating utility of @if ... &analyze
BTests for new @if ... &analyze functionality
"if ( ... ) &analyze" language feature
classes for tracking "@if (...) &analyze" notion of code being/not being "activated"
RemoveGlobal() method for Scope class + simplifying interfaces
When a Spicy unit type was used in an EVT event definition, it could
end up being cleared out at the time the logic for `export` got to see it.
Closes#3083.
Note: This needs backporting to the legacy plugin.
* origin/topic/timw/2914-truncated-tunnel-plus-things:
Address wire/capture length feedback
packet_analysis/TCP: Do not use untrusted len for DeliverPacket()
Add regression test using pcap from GH-2683
Add btest to test Geneve->VXLAN->Truncated inner packet
IP: Update packet->len with accumulated fragment size
UDP: Forward any remaining data (also empty) to session-analysis
IPTunnel: Compute inner wire length based on cap_len differences.
IP: fix weird name to not be ipv6 specific
UDP: don't validate checksum if caplen < len
PIA: Modernize how struct initialization is done
This patch changes invocations of `spicyz` and similar Spicy tools in
tests which perform compilation to use debug mode via passing `-d`. This
in turn leads to Spicy compiling generated C++ code in debug as opposed
to release mode which typically seems to require less CPU time and RAM.
For a local test running with `btest -j 16` and no caching via
`HILTI_CXX_COMPILER_LAUNCER` this sped up running of BTests under
`spicy/` by about 40s on my machine (120s vs 160s).
If CMakeCache.txt indicates the Zeek build was done with ccache,
take the CCACHE_PROGRAM:FILEPATH= line and populate the
HILTI_CXX_COMPILER_LAUNCHER environment.
For local development with ccache, this speeds up the spicy tests
significantly after the initial run without knowing or needing know
that setting HILTI_CXX_COMPILER_LAUNCHER would've helped.
This seems to have relied on the reading file twice behavior simply
testing that 16 lines are observed. Switch to using two separate
files and doing a system("mv ...") to trigger the REREAD logic, there's
not force_update() needed and it wouldn't do anything if the file
hadn't changed anyway.
Found while writing documentation and being confused why
all lines and end_of_data() arrive twice during startup.
The test is a bit fuzzy, but does fail reliably without
the changes to Raw.cc
Also fix not checking dev in the MODE_REREAD path.
Closes#3053
If DataIn() was called and a cur_entity_id (file_id) has been produced
previously, re-use it for calls to EndOfFile(). This avoids a costly
event_mgr.Drain() when we already have that information. It should be safer,
too, as `get_file_handle()` in script may generate a different ID and
thereby de-synchronizing.
We're currently rendering parameter descriptions from .bif file into
the .rst as follows:
:cid: The connection identifier.
:aid: The analyzer ID.
Switch this to :param cid: instead so that we can have Sphinx deal with
this as param docfield and group all parameters into a single section.
Currently, having the bare :cid: style causes sphinx to treat it as an
unknown field type, capitalize it and render it.
This moves `auxil/spicy/spicy` to `auxil/spicy`. It relocates the
pieces previously inside that intermediary directory. The main change
is that now tweak the compilation flags for Spicy through target
options.
This is on top of `topic/robin/spicy-plugin`.
This reflects the `spicy-plugin` code as of `d8c296b81cc2a11`.
In addition to moving the code into Zeek's source tree, this comes
with a couple small functional changes:
- `spicyz` no longer tries to infer if it's running from the build
directory. Instead `ZEEK_SPICY_LIBRARY` can be set to a custom
location. `zeek-set-path.sh` does that now.
- ZEEK_CONFIG can be set to change what `spicyz -z` print out. This is
primarily for backwards compatibility.
Some further notes on specifics:
- We raise the minimum Spicy version to 1.8 (i.e., current `main`
branch).
- Renamed the `compiler/` subdirectory to `spicyz` to avoid
include-path conflicts with the Spicy headers.
- In `cmake/`, the corresponding PR brings a new/extended version of
`FindZeek`, which Spicy analyzer packages need. We also now install
some of the files that the Spicy plugin used to bring for testing,
so that existing packages keep working.
- For now, this all remains backwards compatible with the current
`zkg` analyzer templates so that they work with both external and
integrated Spicy support. Later, once we don't need to support any
external Spicy plugin versions anymore, we can clean up the
templates as well.
- All the plugin's tests have moved into the standard test suite. They
are skipped if configure with `--disable-spicy`.
This holds off on adapting the new code further to Zeek's coding
conventions, so that it remains easier to maintain it in parallel to
the (now legacy) external plugin. We'll make a pass over the
formatting for (presumable) Zeek 6.1.
For "individually different but deterministic" runs specifying Zeek's
seed as an environment variable eases setups as one can avoid creating
extra seed files for each of the individual processes.
It is an error to specify the new ZEEK_SEED_VALUES variable together
with the existing ZEEK_SEED_FILE and -G. ZEEK_SEED takes precedence over
deterministic mode (-D) like ZEEK_SEED_FILE does today already.
This adds compatibility tests for receiving non-timestamped events as
well as providing timestamps via broker websockets.
Co-authored-by: Arne Welzel <arne.welzel@corelight.com>
* origin/topic/neverlord/version-header:
testing/zeek-version-link: Assume nm is there
Drop dependency for zeek_dynamic_plugin_base
Fixup ifdef check in Plugin.h
Update cmake_minimum_required() in test plugins
testing: Add zeek-version-link tests
Clean up ZEEK_CONFIG_SKIP_VERSION_H issues
cmake-format during merge
