This is mostly involved with moving CMAKE_SOURCE_DIR and
CMAKE_BINARY_DIR to PROJECT_SOURCE_DIR and PROJECT_BINARY_DIR because it
enables zeek to be built as a subproject of another.
There is another branch in the cmake repository with the same name that
is also required for this to fully work.
* origin/topic/christian/install-btest-tooling:
Explain zeek-config options in help output
Sort variables at top of zeek-config alphabetically
Install Zeek's btest tooling with the distribution
Previously, a system-wide installation of any bundled auxil/ software
(like CAF) may get found/included rather than the bundled version and
possibly break the build.
This creates $PREFIX/share/btest in the install tree, with the
following folders:
- scripts/ for the canonifiers
- data/ for random.seed
- data/pcaps for the test pcaps
The pcaps can be skipped by configuring with --disable-btest-pcaps.
Configuring with --disable-btest skips the installation. The main
reason for doing this is that many Zeek packages come with testsuites
that nearly always require btest, so providing btest out of the box
makes sense. Note that zkg's recent additions to its PATH environment
variable management mean that this btest instance is automatically
found also when the Zeek installation's binary folder isn't in the
path.
- Add auxil/package-manager submodule as an optional build
source. When the submodule is present, zkg gets installed into the
Zeek installation's bin directory, its config file into etc/zkg, and
its state into var/lib/zkg. Like zeekctl, zkg finds its own module
independently of any PYTHONPATH. Installation via pip remains
supported. You can skip zkg explicitly via --disable-zkg. See the
NEWS update for details.
- Establish a "zeek/python" subdirectory under libdir as the common place
for Python modules in the Zeek distribution. This now separates out
the Broker Python bindings, ZeekControl, and zkg's Python module.
- Add configure flags to allow customizing this Python folder, in
three ways: --python-dir, --python-prefix, and --python-home. These
differ in the logic they automatically add to the path, and build on
the logic already used in Broker.
- Include a (comented-out) @load for zkg's packages folder in
local.zeek.
- Bump zeekctl to move to this new location.
- Bump doc to include installation instructions
- Update NEWS accordingly.
Merge adjustments:
- Fix the `--with-caf=` option from emitting messages about a variable
having an unset value.
- Fix the `zeek-config --caf_root` option
* origin/topic/neverlord/caf-migration:
Update submodule(s)
Fix message ordering of Broker messages
Fix UB in shutdown of Broker manager
Preserve string output of Broker errors
Migrate to CAF 0.18
Also now uses CMake's ENABLE_EXPORTS target property for the zeek
executable to ensure symbols are visible to plugins. Prior to CMake
3.4, the policy was to export symbols by default for certain platforms,
but later versions need either the explicit target property or policy.
- Remove hardwiring of $ZEEK_ROOT/lib throughout the three and
defaults the name of Zeek's library directory to the default on the
given platform (e.g. lib64), via GNUInstallDirs.
- Consistently use that lib directory, instead of two lib folders
resulting when using a custom libdir.
- Remove the old lib directory in the installation prefix, if one exists
- Add --lib_dir to zeek-config (and sort its options a bit).
- Bump submodules for corresponding changes
Addresses https://github.com/zeek/zeek/issues/1180
This helps ensure successful compilation of a plugin that includes a
Zeek header which includes an external dependency header even if the
plugin itself doesn't necessarily depend on that external library.
This is added to the default Zeek build as a convenience since it's the
new suggested method for archiving the logs produced by a Supervised
Zeek Cluster.
* origin/master:
Use zeek::detail namespace for fuzzer utils
Set terminating flag during fuzzer cleanup
Add missing include to standalone fuzzer driver
Improve standalone fuzzer driver error messages
Test fuzzers against seed corpus under CI ASan build
Update fuzzing README with OSS-Fuzz integration notes
Link fuzzers against shared library to reduce executable sizes
Improve FuzzBuffer chunking
Fix compiler warning in standalone fuzzer driver
Adjust minor fuzzing documentation
Exit immediately after running unit tests
Add OSS-Fuzz Zeek script search path to fuzzers
Assume libFuzzer when LIB_FUZZING_ENGINE file doesn't exist
Change handling of LIB_FUZZING_ENGINE
Change --enable-fuzzing to --enable-fuzzers
Add standalone driver for fuzz targets
Add basic structure for fuzzing targets
Now we use cmake to determine which compiler flags are set to
determine which implementation of highwayhash to compile. This should
be much more robust.
Also - fix missing include on old ubuntus.
Useful for cases that don't need to use a fuzzing engine, but just run
the fuzz targets over some set of inputs, like for regression/CI tests.
Also added a POP3 fuzzer dictionary, seed corpus, and README with
examples.
This also now installs the full rapidjson include/ tree in order to
allow external plugins to properly build if they include
threading/formatters/JSON.h
This is the result of (major * 10000 + minor * 100 + patch), for example
3.1.2 becomes 30102. This definition may be helpful for external code
that requires conditional compilation to support multiple Zeek
versions with differing APIs.
When cross-compiling, BINPAC_EXE_PATH will be set by the user to the
host binpac binary which is fine however aux/binpac won't be built which
will raise a build failure as target binpac (headers, library) won't be
installed or built
Signed-off-by: Fabrice Fontaine <fontaine.fabrice@gmail.com>
- Removes entire FindSoonest method that includes all of the calls to select() for checking for ready sources
- Removes FD_Set checking against IOSources
- Adds system for registering and unregistering file descriptors from IOSources. This allows individual sources to mark themselves as ready to be checked by the loop as they become available.
- Adds entirely new loop architecture based on checking the IOSources for when their next timeout is, and then waiting for either that timeout or when the next source is ready. This also implements the polling based on what the OS supports, instead of just calling select() on all platforms. Currently it supports kqueue, epoll, and plain poll.
- Adds system for pinging the loop to force it to wake up
* Rename SANITIZERS CMake variable to ZEEK_SANITIZERS for clarity
* Use -O1 by default to improve speed (set NO_OPTIMIZATIONS env. var.
to override and use -O0). Uses -fno-optimize-sibling-calls with -O1
to still get "perfect stack traces".
* Updates various sub-projects with sanitizer improvements:
binpac and bifcl, by default, now ignore leaks reported by LeakSanitizer
so that it doesn't interfere with the Zeek build
* origin/topic/jsiwek/openbsd:
Adjust btests for OpenBSD portability
Convert pcapng test suite files to pcap format
Fix undefined symbols loading libbroker on OpenBSD
Fix compile warnings on OpenBSD
- Minor whitespace tweaks
- Add line to build summary output for whether unit tests are enabled
* origin/topic/neverlord/doctest:
Add doctest license and copyright
Integrate review feedback
Fix submodule reference for doctest
Add initial scaffold for unit testing via doctest
python3.7:/usr/local/lib/libbroker.so.2: undefined symbol '__inet_ntop'
python3.7:/usr/local/lib/libbroker.so.2: undefined symbol '__inet_pton'
An independent Broker build doesn't have that problem because it
usually picks up those functions from libc. But when building Broker
as part of Zeek, include search paths were already modified to reflect
Zeek's libbind dependency and Broker ends up using libbind headers
which defines its own version of these, but then does not link to
libbind (since it's a dependency of Zeek, not Broker) to actually get
those symbols (e.g. when dlopen'ing independently from Zeek, like for
the Python bindings). Solution is to re-order include_directories()
for Zeek dependencies such that they won't effect sub-projects.
Calling `find_package(CAF)` puts all include directory paths into a
single variable. Picking the paths individually is not only error prone
and cumbersome, but can also lead to bulid errors. For example, when
using a build directory for CAF_ROOT then CAF_INCLUDE_DIRS will have one
extra path to find CAF's `config.hpp` (which is part of the build
directory).
