When a JSON document contains key names containing colons or other
special characters that are not valid in Zeek identifiers, from_json()
cannot be used to parse such input.
This change allows a customizable normalization function.
Closes#3142.
* origin/topic/vern/ZAM-Jun23-maint:
ZAM documentation updates
ZAM bug fix for incorrect management of global variables
ZAM bug fix (simplification) for nested inline functions that don't do anything
ZAM support for negative vector indices
ZAM ignores new "assert" statement
fixes for doing "script validation" to check for ZAM compile-ability
tweak for Broker BTest that needs a delay for ZAM
BTest baseline updates for minor Zeek differences (in one case, no difference)
BTest to check for invalid negative vector indices
This was brought up on Slack as a potential source of confusion during
development as it's not visible when plugin directories are ignored outside
of looking into the plugin debug stream. I'd actually prefer to just
FatalError() this, but a warning seems reasonably visible for interactive
usage.
This test triggered ubsan by putting a function with the wrong type
as a post-processor into the .shadow file. Don't do that.
Likely Zeek should provide a better error message, but hand-crafting
.shadow files isn't what is normally done and this is to fix the
master build for now.
* topic/awelzel/3112-log-suffix-left-over-log-rotation:
cluster/logger: Fix leftover-log-rotation in multi-logger setups
cluster/logger: Fix global var reference
Using break in either of the hooks allows to suppress the default reporter
error message rather than suppressing solely based on the existence of an
assertion_failure() handler.
Populating log_metadata during zeek_init() is too late for the
leftover-log-rotation functionality, so do it at script parse time.
Also, prepend archiver_ to the log_metadata table and encoding function
due to being in the global namespace and to align with the
archiver_rotation_format_func. This hasn't been in a released
version yet, so fine to rename still.
Closes#3112
* origin/topic/awelzel/cluster-at-if-removal:
test-all-policy: Do not load nodes-experimental/manager.zeek
cluster/main: Remove extra @if ( Cluster::is_enabled() )
These have been discussed in the context of "@if &analyze" [1] and
am much in favor for not disabling/removing ~100 lines (more than
fits on a single terminal) out from the middle of a file. There's no
performance impact for having these handlers enabled unconditionally.
Also, any future work on "@if &analyze" will look at them again which
we could also skip.
This also reverts back to the behavior where the Cluster::LOG stream
is created even in non cluster setups like in previous Zeek versions.
As long as no one writes to it there's essentially no difference. If
someone does write to Cluster::LOG, I'd argue not black holing these
messages is better. Schema generators using Log::active_streams will
continue to discover Cluster::LOG even if they run in non-cluster
mode.
https://github.com/zeek/zeek/pull/3062#discussion_r1200498905
After the introduction of @if ... analyze, a lot of warnings were
triggered due to nested @if and @if .. analyze usage.
Add a test for coverage of all policy scripts in cluster mode
for the usual node types so this does not happen again.
* origin/topic/vern/at-if-analyze:
updates reflecting review comments
change base scripts to use run-time if's or @if ... &analyze
a number of BTests updated with @if ... &analyze
update for scripting coverage BTest demonstrating utility of @if ... &analyze
BTests for new @if ... &analyze functionality
"if ( ... ) &analyze" language feature
classes for tracking "@if (...) &analyze" notion of code being/not being "activated"
RemoveGlobal() method for Scope class + simplifying interfaces
When a Spicy unit type was used in an EVT event definition, it could
end up being cleared out at the time the logic for `export` got to see it.
Closes#3083.
Note: This needs backporting to the legacy plugin.
* origin/topic/timw/2914-truncated-tunnel-plus-things:
Address wire/capture length feedback
packet_analysis/TCP: Do not use untrusted len for DeliverPacket()
Add regression test using pcap from GH-2683
Add btest to test Geneve->VXLAN->Truncated inner packet
IP: Update packet->len with accumulated fragment size
UDP: Forward any remaining data (also empty) to session-analysis
IPTunnel: Compute inner wire length based on cap_len differences.
IP: fix weird name to not be ipv6 specific
UDP: don't validate checksum if caplen < len
PIA: Modernize how struct initialization is done
This patch changes invocations of `spicyz` and similar Spicy tools in
tests which perform compilation to use debug mode via passing `-d`. This
in turn leads to Spicy compiling generated C++ code in debug as opposed
to release mode which typically seems to require less CPU time and RAM.
For a local test running with `btest -j 16` and no caching via
`HILTI_CXX_COMPILER_LAUNCER` this sped up running of BTests under
`spicy/` by about 40s on my machine (120s vs 160s).
If CMakeCache.txt indicates the Zeek build was done with ccache,
take the CCACHE_PROGRAM:FILEPATH= line and populate the
HILTI_CXX_COMPILER_LAUNCHER environment.
For local development with ccache, this speeds up the spicy tests
significantly after the initial run without knowing or needing know
that setting HILTI_CXX_COMPILER_LAUNCHER would've helped.
