This moves the Telemetry framework's BIF-defined functionalit from the
secondary-BIFs stage to the primary one. That is, this functionality is now
available from the end of init-bare.zeek, not only after the end of
init-frameworks-and-bifs.zeek.
This allows us to use script-layer telemetry in our Zeek's own code that get
pulled in during init-frameworks-and-bifs.
This change splits up the BIF features into functions, constants, and types,
because that's the granularity most workable in Func.cc and NetVar. It also now
defines the Telemetry::MetricsType enum once, not redundantly in BIFs and script
layer.
Due to subtle load ordering issues between the telemetry and cluster frameworks
this pushes the redef stage of Telemetry::metrics_port and address into
base/frameworks/telemetry/options.zeek, which is loaded sufficiently late in
init-frameworks-and-bifs.zeek to sidestep those issues. (When not doing this,
the effect is that the redef in telemetry/main.zeek doesn't yet find the
cluster-provided values, and Zeek does not end up listening on these ports.)
The need to add basic Zeek headers in script_opt/ZAM/ZBody.cc as a side-effect
of this is curious, but looks harmless.
Also includes baseline updates for the usual btests and adds a few doc strings.
* origin/topic/vern/zam-regularization: (33 commits)
simpler and more robust identification of function parameters for AST profiling
fixes to limit AST traversal in the face of recursive types
address some script optimization compiler warnings under Linux
fix for -O C++ construction of variable names that use multiple module namespaces
fix for script optimization of "opaque" values that are run-time constants
fix for script optimization of nested switch statements
script optimization fix for complex "in" expressions in conditionals
updates to typos allow-list reflecting ZAM regularization changes
BTest updates for ZAM regularization changes
convert new ZAM operations to use typed operands
complete migration of ZAM to use only public ZVal methods
"-O validate-ZAM" option to validate generated ZAM instructions
internal option to suppress control-flow optimization
exposing some functionality for greater flexibility in structuring run-time execution
rework ZAM compilation of type switches to leverage value switches
add tracking of control flow information
factoring of ZAM operation specifications into separate files
updates to ZAM operations / gen-zam regularization, other than the operations themselves
type-checking fix for vector-of-string operations
ZVal constructor for booleans
...
We package vanilla sqlite from upstream and on Fedora 40 with sqlite 3.46
there's the following compiler warning:
In function 'sqlite3Strlen30',
inlined from 'sqlite3ColumnSetColl' at
../../src/3rdparty/sqlite3.c:122105:10:
../../src/3rdparty/sqlite3.c:35003:28: error: 'strlen' reading 1 or more bytes from a region of size 0 [-Werror=stringop-overread]
35003 | return 0x3fffffff & (int)strlen(z);
| ^~~~~~~~~
In function 'sqlite3ColumnSetColl':
Disabling -Werror on sqlite3.c seems sensible given we have little
control over that code.
threading/formatters/JSON.h currently includes rapidjson headers for declaring
the NullDoubleWriter. This appears mostly an internal detail, but
results in the situation that 1) we need to ship rapidjson headers with
the Zeek install tree and 2) taking care that external plugins are able
to find these headers should they include formatters/JSON.h.
There are currently no other Zeek headers that include rapidjson, so this
seems very unfortunate and self-inflicted given it's not actually required.
Attempt to hide this implementation detail with the goal to remove the
rapidjson includes with v7.1 and then also stop bundling and exposing
the include path to external plugins.
The NullDoubleWriter implementation moves into a new formatters/detail/json.h
header which is not installed.
Closes#3128
* origin/topic/vern/at-if-analyze:
updates reflecting review comments
change base scripts to use run-time if's or @if ... &analyze
a number of BTests updated with @if ... &analyze
update for scripting coverage BTest demonstrating utility of @if ... &analyze
BTests for new @if ... &analyze functionality
"if ( ... ) &analyze" language feature
classes for tracking "@if (...) &analyze" notion of code being/not being "activated"
RemoveGlobal() method for Scope class + simplifying interfaces
I've continued to see somewhat slower builds after Zeek version bumps. It
appears files covered by zeek_objs didn't have -DZEEK_CONFIG_SKIP_VERSION_H
set causing ccache invalidation after a version bump.
The util-config.h has never been installed previously. Skimming the history,
it was only meant for inclusion from util.cc, hence the name. Now that it's
included from some other headers, rename it to align with what it
contains and install it, too.
This reflects the `spicy-plugin` code as of `d8c296b81cc2a11`.
In addition to moving the code into Zeek's source tree, this comes
with a couple small functional changes:
- `spicyz` no longer tries to infer if it's running from the build
directory. Instead `ZEEK_SPICY_LIBRARY` can be set to a custom
location. `zeek-set-path.sh` does that now.
- ZEEK_CONFIG can be set to change what `spicyz -z` print out. This is
primarily for backwards compatibility.
Some further notes on specifics:
- We raise the minimum Spicy version to 1.8 (i.e., current `main`
branch).
- Renamed the `compiler/` subdirectory to `spicyz` to avoid
include-path conflicts with the Spicy headers.
- In `cmake/`, the corresponding PR brings a new/extended version of
`FindZeek`, which Spicy analyzer packages need. We also now install
some of the files that the Spicy plugin used to bring for testing,
so that existing packages keep working.
- For now, this all remains backwards compatible with the current
`zkg` analyzer templates so that they work with both external and
integrated Spicy support. Later, once we don't need to support any
external Spicy plugin versions anymore, we can clean up the
templates as well.
- All the plugin's tests have moved into the standard test suite. They
are skipped if configure with `--disable-spicy`.
This holds off on adapting the new code further to Zeek's coding
conventions, so that it remains easier to maintain it in parallel to
the (now legacy) external plugin. We'll make a pass over the
formatting for (presumable) Zeek 6.1.
* origin/topic/neverlord/gh-2806:
test/builtin-plugins: Add plugin using ZEEK_VERSION_NUMBER
Revert putting plugins into a fresh scope for now
Propagate zeek-version.h skip via CMake properties
Add a define for `ZEEK_CONFIG_SKIP_VERSION_H` to static plugins as well
as to dynamic plugins that we build alongside Zeek. When including
`zeek-config.h` with this macro defined, the header skips including
`zeek-version.h`.
"Community ID" has become an established flow hash for connection correlation
across different monitoring and storage systems. Other NSMs have had native
and built-in support for Community ID since late 2018. And even though the
roots of "Community ID" are very close to Zeek, Zeek itself has never provided
out-of-the-box support and instead required users to install an external plugin.
While we try to make that installation as easy as possible, an external plugin
always sets the bar higher for an initial setup and can be intimidating.
It also requires a rebuild operation of the plugin during upgrades. Nothing
overly complicated, but somewhat unnecessary for such popular functionality.
This isn't a 1:1 import. The options are parameters and the "verbose"
functionality has been removed. Further, instead of a `connection`
record, the new bif works with `conn_id`, allowing computation of the
hash with little effort on the command line:
$ zeek -e 'print community_id_v1([$orig_h=1.2.3.4, $orig_p=1024/tcp, $resp_h=5.6.7.8, $resp_p=80/tcp])'
1:RcCrCS5fwYUeIzgDDx64EN3+okU
Reference: https://github.com/corelight/zeek-community-id/
This adds a new utility called ci/collect-repo-info.py to produce a JSON
document that is then baked into the Zeek executable file. Further, when
creating a tarball via `make dist`, put a top-level repo-info.json file
in place that is picked when no .git directory exists.
Closes#1405
- Ignore conan libpcap if PCAP_ROOT_DIR is passed
- Update the cmake submodule to pick up changes for finding the right
paths to npcap
- Add lazy-loading of npcap so the library path gets set correctly
at startup
