This reflects the `spicy-plugin` code as of `d8c296b81cc2a11`.
In addition to moving the code into Zeek's source tree, this comes
with a couple small functional changes:
- `spicyz` no longer tries to infer if it's running from the build
directory. Instead `ZEEK_SPICY_LIBRARY` can be set to a custom
location. `zeek-set-path.sh` does that now.
- ZEEK_CONFIG can be set to change what `spicyz -z` print out. This is
primarily for backwards compatibility.
Some further notes on specifics:
- We raise the minimum Spicy version to 1.8 (i.e., current `main`
branch).
- Renamed the `compiler/` subdirectory to `spicyz` to avoid
include-path conflicts with the Spicy headers.
- In `cmake/`, the corresponding PR brings a new/extended version of
`FindZeek`, which Spicy analyzer packages need. We also now install
some of the files that the Spicy plugin used to bring for testing,
so that existing packages keep working.
- For now, this all remains backwards compatible with the current
`zkg` analyzer templates so that they work with both external and
integrated Spicy support. Later, once we don't need to support any
external Spicy plugin versions anymore, we can clean up the
templates as well.
- All the plugin's tests have moved into the standard test suite. They
are skipped if configure with `--disable-spicy`.
This holds off on adapting the new code further to Zeek's coding
conventions, so that it remains easier to maintain it in parallel to
the (now legacy) external plugin. We'll make a pass over the
formatting for (presumable) Zeek 6.1.
* origin/topic/neverlord/gh-2711:
Integrate review feedback
Include compiler in --show-config output
Fix CMake option defaults on Windows
Move build defaults from configure to CMake
Moving the defaults for build variables from the `configure` script to
`CMakeLists.txt` gives the same default behavior on platforms where the
`configure` script is not available (Windows) and also allows a pure
CMake-based work flow (e.g., the standard `cmake -S . -B build`) without
having to manually adjust the defaults.
The `configure` script also becomes much simpler as a result.
zeek.on('zeek_init', () => {
console.log('Hello, Zeek!');
});
For interaction with external systems and HTTP APIs, JavaScript and the
Node.js ecosystem beat Zeek script. Make it more easily accessible by
including ZeekJS with Zeek directly.
When a recent enough libnode version is found on the build system, ZeekJS is
added as a builtin plugin. This behavior can be disabled via
``--disable-javascript``. Linux distributions providing such a package are
Ubuntu (22.10) and Debian (testing/bookworm) as libnode-dev.
Fedora provides it as nodejs-devel.
This plugin takes over loading of .js or .cjs files. When no such files
are provided to Zeek, Node and the V8 engine are not initialized and
should not get into the way.
This should be considered experimental.
This has come up a few times and the motivation is mainly better "first timer"
experience with Zeek. Concretely, if one wants to run a Zeek cluster with
multiple workers and reasonable load balancing on Linux, AF_PACKET is a decent
start. Without AF_PACKET support being built into Zeek, however, a new user's
next experience is that of setting up a development environment in order
to compile an external plugin (think compiler, kernel headers, zkg, ...).
Only to get what could be termed basic functionality.
This is using the ZEEK_INCLUDE_PLUGINS infrastructure. I've used the all
upper case spelling of AF_PACKET in the help output because it seems everyone
else references/writes it like that. I think we should also write it
like that in the docs.
spicy-plugin can find Spicy in paths given by `SPICY_ROOT_DIR` while
`./configure` instead set `SPICY_ROOT`. With this patch we now set the
correct variable.
We also adjust variations of the previous variable name with different
capitalization which caused us to not properly configure spicy-plugin
(which triggers finding Spicy in its given prefix).
Closes#2363.
The user so far had to configure with --enable-zeek-client to trigger
installation of the client (from auxil/zeek-client). This flips it around to
allow disabling the installation, and removes --enable-zeek-client from the
Docker build in CI, where we've already been using it to allow the cluster
testsuite to run tests with that image.
This establishes several of the Zeek installation's directories in
CMakeLists.txt, and adds a new ZEEK_STATE_DIR (via `configure --state-dir`) for
arbitrary Zeek-related variable state. It defaults to ZEEK_ROOT/var/lib.
This leaves the existing, zeekctl-related ZEEK_LOCAL_STATE_DIR unchanged and
does not use it. We'll likely deprecate it when removal of zeekctl approaches.
This separates Gen-ZAM from the main Zeek distribution and places it in a
submodule at auxil/gen-zam, reflecting binpac and bifcl. Configuring with
--with-gen-zam=... allws reuse of an existing executable.
All changes in this patch were performed automatically with `shfmt` with
configuration flags specified in `.pre-commit-config.yaml`.
In addition to fixing whitespace the roundtrip through shfmt's AST also
transforms command substitutions
`cmd`
# becomes
$(cmd)
and some redirects
>&2 echo "msg"
# becomes
echo >&2 "msg"
* origin/topic/christian/install-btest-tooling:
Explain zeek-config options in help output
Sort variables at top of zeek-config alphabetically
Install Zeek's btest tooling with the distribution
This creates $PREFIX/share/btest in the install tree, with the
following folders:
- scripts/ for the canonifiers
- data/ for random.seed
- data/pcaps for the test pcaps
The pcaps can be skipped by configuring with --disable-btest-pcaps.
Configuring with --disable-btest skips the installation. The main
reason for doing this is that many Zeek packages come with testsuites
that nearly always require btest, so providing btest out of the box
makes sense. Note that zkg's recent additions to its PATH environment
variable management mean that this btest instance is automatically
found also when the Zeek installation's binary folder isn't in the
path.
- Add auxil/package-manager submodule as an optional build
source. When the submodule is present, zkg gets installed into the
Zeek installation's bin directory, its config file into etc/zkg, and
its state into var/lib/zkg. Like zeekctl, zkg finds its own module
independently of any PYTHONPATH. Installation via pip remains
supported. You can skip zkg explicitly via --disable-zkg. See the
NEWS update for details.
- Establish a "zeek/python" subdirectory under libdir as the common place
for Python modules in the Zeek distribution. This now separates out
the Broker Python bindings, ZeekControl, and zkg's Python module.
- Add configure flags to allow customizing this Python folder, in
three ways: --python-dir, --python-prefix, and --python-home. These
differ in the logic they automatically add to the path, and build on
the logic already used in Broker.
- Include a (comented-out) @load for zkg's packages folder in
local.zeek.
- Bump zeekctl to move to this new location.
- Bump doc to include installation instructions
- Update NEWS accordingly.
- Remove hardwiring of $ZEEK_ROOT/lib throughout the three and
defaults the name of Zeek's library directory to the default on the
given platform (e.g. lib64), via GNUInstallDirs.
- Consistently use that lib directory, instead of two lib folders
resulting when using a custom libdir.
- Remove the old lib directory in the installation prefix, if one exists
- Add --lib_dir to zeek-config (and sort its options a bit).
- Bump submodules for corresponding changes
This is added to the default Zeek build as a convenience since it's the
new suggested method for archiving the logs produced by a Supervised
Zeek Cluster.
General changes:
* Add -D/--deterministic command line option as
convenience/alternative to -G/--load-seeds (i.e. no file needed, it just
uses zero-initialized random seeds). It also changes Broker data
stores over to using deterministic timing rather than real time.
* Add option to make Reporter abort on runtime scripting errors
- Also removed the setting of BinPAC_ROOT_DIR in the configure
script's --with-binpac= option as that breaks the cross-compilation
use-case
* 'master' of https://github.com/ffontaine/zeek:
CMakeLists.txt: fix cross-compilation with binpac
- Removes entire FindSoonest method that includes all of the calls to select() for checking for ready sources
- Removes FD_Set checking against IOSources
- Adds system for registering and unregistering file descriptors from IOSources. This allows individual sources to mark themselves as ready to be checked by the loop as they become available.
- Adds entirely new loop architecture based on checking the IOSources for when their next timeout is, and then waiting for either that timeout or when the next source is ready. This also implements the polling based on what the OS supports, instead of just calling select() on all platforms. Currently it supports kqueue, epoll, and plain poll.
- Adds system for pinging the loop to force it to wake up
* Rename SANITIZERS CMake variable to ZEEK_SANITIZERS for clarity
* Use -O1 by default to improve speed (set NO_OPTIMIZATIONS env. var.
to override and use -O0). Uses -fno-optimize-sibling-calls with -O1
to still get "perfect stack traces".
* Updates various sub-projects with sanitizer improvements:
binpac and bifcl, by default, now ignore leaks reported by LeakSanitizer
so that it doesn't interfere with the Zeek build
- Minor whitespace tweaks
- Add line to build summary output for whether unit tests are enabled
* origin/topic/neverlord/doctest:
Add doctest license and copyright
Integrate review feedback
Fix submodule reference for doctest
Add initial scaffold for unit testing via doctest
- Adjusted the hint text a bit during merge
* 'topic/simon/configure-fail-hint' of https://github.com/simonhf/zeek:
Add hint to run make distclean if configure fails
This alias makes it easier for tooling that deals with both Zeek and
Broker (which uses `--build-dir`). Also, it's one less quirk to remember
when working with both repositories.
