Using dnsmasq --address covers an entire domain, so dns.example.com
AAAA queries would also return IPv6 addresses for example.com
Instead, this was always meant to use host entries.
As services are sorted by default now, this disables the canonifier that
sorts the service field for the external baseline.
This also adds the tracking of disabled services in the service field
via DPD::track_removed_services_in_connection - to make such changes
more visible, and check that the feature works as desired.
This is a cluster backend implementation using a central XPUB/XSUB proxy
that by default runs on the manager node. Logging is implemented leveraging
PUSH/PULL sockets between logger and other nodes, rather than going
through XPUB/XSUB.
The test-all-policy-cluster baseline changed: Previously, Broker::peer()
would be called from setup-connections.zeek, causing the IO loop to be
alive. With the ZeroMQ backend, the IO loop is only alive when
Cluster::init() is called, but that doesn't happen anymore.
* origin/topic/timw/remove-abspath-cleanup:
diff-remove-abspath: Add separate handling of Windows paths
diff-remove-abspath: Remove capture of windows drive letters from POSIX regex
That test got flaky probably from #3949 on centosstream9 CI. You can
replicate that behavior by increasing the sleep time when waiting for
the file such that the test will attempt to read the missing file again.
Since the one second wait for file is glacially slow for this, speeding
it up should mean that the file gets created sooner and so the test
won't try to open the file again. But, it's always still technically
possible, since the test will wait for 10 seconds and the heartbeat
seems to be 1 second. At least if that happens, it's probably a bug or
massive slowdown of some kind.
I'm always a bit worried to use sed -E anywhere, because the canonifiers
give the impression it won't work everywhere consistently. My manpage says
sed -E should be preferred for portability, so lets remove the
sed -r / sed -E differentiation assuming it's just a thing from the past.
...except for clang-format, because versions after v13.0.0 have
borked the Whitesmith formatting. Also moves yapf from
pre-commit/mirrors-yapf to google/yapf.
This reflects the `spicy-plugin` code as of `d8c296b81cc2a11`.
In addition to moving the code into Zeek's source tree, this comes
with a couple small functional changes:
- `spicyz` no longer tries to infer if it's running from the build
directory. Instead `ZEEK_SPICY_LIBRARY` can be set to a custom
location. `zeek-set-path.sh` does that now.
- ZEEK_CONFIG can be set to change what `spicyz -z` print out. This is
primarily for backwards compatibility.
Some further notes on specifics:
- We raise the minimum Spicy version to 1.8 (i.e., current `main`
branch).
- Renamed the `compiler/` subdirectory to `spicyz` to avoid
include-path conflicts with the Spicy headers.
- In `cmake/`, the corresponding PR brings a new/extended version of
`FindZeek`, which Spicy analyzer packages need. We also now install
some of the files that the Spicy plugin used to bring for testing,
so that existing packages keep working.
- For now, this all remains backwards compatible with the current
`zkg` analyzer templates so that they work with both external and
integrated Spicy support. Later, once we don't need to support any
external Spicy plugin versions anymore, we can clean up the
templates as well.
- All the plugin's tests have moved into the standard test suite. They
are skipped if configure with `--disable-spicy`.
This holds off on adapting the new code further to Zeek's coding
conventions, so that it remains easier to maintain it in parallel to
the (now legacy) external plugin. We'll make a pass over the
formatting for (presumable) Zeek 6.1.
zeek.on('zeek_init', () => {
console.log('Hello, Zeek!');
});
For interaction with external systems and HTTP APIs, JavaScript and the
Node.js ecosystem beat Zeek script. Make it more easily accessible by
including ZeekJS with Zeek directly.
When a recent enough libnode version is found on the build system, ZeekJS is
added as a builtin plugin. This behavior can be disabled via
``--disable-javascript``. Linux distributions providing such a package are
Ubuntu (22.10) and Debian (testing/bookworm) as libnode-dev.
Fedora provides it as nodejs-devel.
This plugin takes over loading of .js or .cjs files. When no such files
are provided to Zeek, Node and the V8 engine are not initialized and
should not get into the way.
This should be considered experimental.
As initial examples, this branch ports the Syslog and Finger analyzers
over. We leave the old analyzers in place for now and activate them
iff we compile without any Spicy.
Needs `zeek-spicy-infra` branches in `spicy/`, `spicy-plugin/`,
`CMake/`, and `zeek/zeek-testing-private`.
Note that the analyzer events remain associated with the Spicy plugin
for now: that's where they will show up with `-NN`, and also inside
the Zeekygen documentation.
We switch CMake over to linking the runtime library into the plugin,
vs. at the top-level through object libraries.
This has come up a few times and the motivation is mainly better "first timer"
experience with Zeek. Concretely, if one wants to run a Zeek cluster with
multiple workers and reasonable load balancing on Linux, AF_PACKET is a decent
start. Without AF_PACKET support being built into Zeek, however, a new user's
next experience is that of setting up a development environment in order
to compile an external plugin (think compiler, kernel headers, zkg, ...).
Only to get what could be termed basic functionality.
This is using the ZEEK_INCLUDE_PLUGINS infrastructure. I've used the all
upper case spelling of AF_PACKET in the help output because it seems everyone
else references/writes it like that. I think we should also write it
like that in the docs.
We also add a very basic yapf configuration file. Most of the changes in
this patch were performed automatically, but we broke one overly long
string into multiple components on `src/make_dbg_constants.py`.
All changes in this patch were performed automatically with `shfmt` with
configuration flags specified in `.pre-commit-config.yaml`.
In addition to fixing whitespace the roundtrip through shfmt's AST also
transforms command substitutions
`cmd`
# becomes
$(cmd)
and some redirects
>&2 echo "msg"
# becomes
echo >&2 "msg"
On Alpine (BusyBox `sed`), the previous `sed -e '$a\'` invocation always
added a newline, breaking most every diff. There doesn't seem to be a
need to attempt normalizing EOF newlines at the moment and doing it that
way doesn't seem to be portable anyway. If canonifiers need to be
portable, `sed` should be treated as a text-processing tool and POSIX
definition of text-file is zero or more newline-terminated
character-sequences, so if canonification of Baselines via `sed` is
required, those Baselines should always end with a newline to be
considered text files. I.e. that's not the job of this canonifier, and
changing it also doesn't necessarily generalize since it could be
considered coincidental that diff-remove-timestamps in particular is the
default canonifier that's commonly used while there's still others that
also make use of `sed`.
By default all baslines are run through diff-remove-timestamp. On a BSD
sed implementation, this means that a newline is added to the end of the
file, if no newline was there originally. This behavior differs from GNU
sed, which does not add a newline.
In this commit we unify this behavior by always adding a newline, even
when using GNU sed. This commit also disables the canonifier for a bunch
of binary baselines, so we do not have to change them.
This avoids swallowing multiple separate paths separated by unrelated
content into one substitution, like here:
orig_p=59856<...>/tcp] -> orig_p=59856/tcp, resp_h=192.150.187.43, resp_p=80/tcp]
This pattern got mislead by matching suffixes of other numbers, and
noramlizing exact 0-timestamps isn't really required.
- Remove eplicit "0.000000" number pattern from timestamp normalization
- Require beginning of line or non-numeric character before the
beginning of the number replacement
- Use `-b` most everywhere, it will save time.
- Start some intel tests upon the input file being fully read instead of
at an arbitrary time.
- Improve termination condition for some sumstats/cluster tests.
- Filter uninteresting output from some supervisor tests.
- Test for `notice_policy.log` is no longer needed.
