detect-protocol.zeek was the last non-deprecated script left in
policy/frameworks/dpd. It was moved to policy/frameworks/analyzer. A
script that loads the script from the new location with a deprecation
warning was added.
To address review feedback in GH-4362: rename analyzer-failed-log.zeek
to loggig.zeek, analyzer-debug-log.zeek to debug-logging.zeek and
dpd-log.zeek to deprecated-dpd-log.zeek.
Includes respective test, NEWS, etc updates.
The main part of this commit are changes in tests. A lot of the tests
that previously relied on analyzer.log or dpd.log now use the new
analyzer-failed.log.
I verified all the changes and, as far as I can tell, everything
behaves as it should. This includes the external test baselines.
This change also enables logging of file and packet analyzer to
analyzer_failed.log and fixes some small behavior issues.
The analyzer_failed event is no longer raised when the removal of an
analyzer is vetoed.
If an analyzer is no longer active when an analyzer violation is raised,
currently the analyzer_failed event is raised. This can, e.g., happen
when an analyzer error happens at the very end of the connection. This
makes the behavior more similar to what happened in the past, and also
intuitively seems to make sense.
A bug introduced in the failed service logging was fixed.
The nodes-experimental/manager.zeek file ends up calling Broker::publish()
unconditionally, resulting in a warning. Skip running that code when
generating documentation.
...at the same time, add some `TEST-REQUIRES: have-zeromq` which
unfortunately means that developers will usually want libzmq
installed on their system.
This is a cluster backend implementation using a central XPUB/XSUB proxy
that by default runs on the manager node. Logging is implemented leveraging
PUSH/PULL sockets between logger and other nodes, rather than going
through XPUB/XSUB.
The test-all-policy-cluster baseline changed: Previously, Broker::peer()
would be called from setup-connections.zeek, causing the IO loop to be
alive. With the ZeroMQ backend, the IO loop is only alive when
Cluster::init() is called, but that doesn't happen anymore.
Changing the default_file_bof_buffer_size has subtle impact on
MIME type detection and changed the zeek-testing baseline. Do
not load this new script via test-all-policy to avoid this.
The new test was mainly an aid to understand what is actually going on.
In short, if default_file_bof_buffer_size is larger than the file MIME
detection only runs when the buffer is full, or when the file is removed.
When a file transfer happens over multiple HTTP connections, only
some or one of the http.log entries will have a proper response MIME type.
PCAP extracted from 2009-M57-day11-18.trace.gz.
Turns out loading this script in non-cluster mode uses Cluster::log()
and creates cluster.log in the external baselines saying "cluster
started". Do not load it into test-all-policy.zeek and instead rely
on the new test-all-policy-cluster.test to load it transitively
when running as manager for basic checking.
This provides Broker-level plumbing that allows agents to reach out to their
managed Zeek nodes and collect responses.
As a first event, it establishes Management::Node::API::notify_agent_hello,
to notify the agent when the cluster node is ready to communicate.
Also a bit of comment rewording to replace use of "data cluster" with simply
"cluster", to avoid ambiguity with data nodes in SumStats, and expansion of
test-all-policy.zeek and related/dependent tests, since we're introducing new
scripts.
Documentation is missing and will be added in the next couple of hours.
* origin/topic/johanna/tls12-decryption: (24 commits)
TLS decryption: add test, fix small issues
Address PR feedback
TLS decryption: refactoring, more comments, less bare pointers
Small code fix and test baseline update.
SSL decryption: refactor TLS12_PRF
SSL decryption: small style changes, a bit of documentation
Deprecation and warning fixes
Clang-format updates
add missing call to EVP_KDF_CTX_set_params
TLS decryption: remove payload from ssl_encrypted_data again.
TLS 1.2 decryption: adapt OpenSSL 3.0 changes for 1.1
ssl: adapt TLS-PRF to openSSL 3.0
ssl/analyzer: potentially fix memory leaks caused by bytestrings
analyzer/ssl: several improvements
analyzer/ssl: defensive key length check + more debug logging
testing: feature gate ssl/decryption test
testing: add ssl/decryption test
analyzer/ssl: handle missing <openssl/kdf.h>
analyzer/ssl: silence warning in DTLS analyzer
analyzer/ssl: move proc-{client,server}-hello into the respective analyzers
...
- This gives the cluster controller and agent the common name "Management
framework" and changes the start directory of the sources from
"policy/frameworks/cluster" to "policy/frameworks/management". This avoids
ambiguity with the existing cluster framework.
- It renames the "ClusterController" and "ClusterAgent" script modules to
"Management::Controller" and "Management::Agent", respectively. This allows us
to anchor tooling common to both controller and agent at the "Management"
module.
- It moves common configuration settings, logging, requests, types, and
utilities to the common "Management" module.
- It removes the explicit "::Types" submodule (so a request/response result is
now a Management::Result, not a Management::Types::Result), which makes
typenames more readable.
- It updates tests that depend on module naming and full set of scripts.
Extract-certs-pem writes pem files to a dedicated file; since it does
not really work in cluster-environments it was never super helpful.
This commit deprecates this file and, instead, adds
log-certs-base64.zeek, which adds the base64-encoded certificate (which
is basically equivalent with a PEM) to the log-file. Since, nowadays,
the log-files are deduplicates this should not add a huge overhead.
Several limitations still apply:
- TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 is the only supported cipher suite
- Some tests are broken due to a failing assertion regarding bytestring
- No newly written tests for decryption (the patch was tested extensively for our paper)
- Several small open technical questions marked with FIXME
- Architecture in the ssl module might not be optimal
