Prior to this, static configuration needed to be in place to configure the
controller/agent layout. The configuration update can now include new instances
that the controller will connect to, assuming they're instances with a listening
agent.
Request records for configuration updates now store the full configuration. The
ClusterController::Request module now provies a to_string() function for
rendering requests to a string.
* topic/christian/cluster-controller:
Add a cluster controller testcase for agent-controller checkin
Add zeek-client via new submodule
Update baselines affected by cluster controller changes
Introduce cluster controller and cluster agent scripting
Establish a separate init script when using the supervisor
Add optional bare-mode boolean flag to Supervisor's node configuration
Add support for making the supervisor listen for requests
Add support for setting environment variables via supervisor
This is a preliminary implementation of a subset of the functionality set out in
our cluster controller architecture. The controller is the central management
node, existing once in any Zeek cluster. The agent is a node that runs once per
instance, where an instance will commonly be a physical machine. The agent in
turn manages the "data cluster", i.e. the traditional notion of a Zeek cluster
with manager, worker nodes, etc.
Agent and controller live in the policy folder, and are activated when loading
policy/frameworks/cluster/agent and policy/frameworks/cluster/controller,
respectively. Both run in nodes forked by the supervisor. When Zeek doesn't use
the supervisor, they do nothing. Otherwise, boot.zeek instructs the supervisor
to create the respective node, running main.zeek.
Both controller and agent have their own config.zeek with relevant knobs. For
both, controller/types.zeek provides common data types, and controller/log.zeek
provides basic logging (without logger communication -- no such node might
exist).
A primitive request-tracking abstraction can be found in controller/request.zeek
to track outstanding request events and their subsequent responses.
The added disable-certificate-events-known-certs.zeek disables repeated
X509 events in SSL connections, given that the connection terminates at
the same server and used the samt SNI as a previously seen connection
with the same certificate.
For people that see significant amounts of TLS 1.2 traffic, this could
reduce the amount of raised events significantly - especially when a
lot of connections are repeat connections to the same servers.
The practical impact of not raising these events is actually very little
- unless a script directly interacts with the x509 events, everything
works as before - the x509 variables in the connection records are still
being set (from the cache).
This policy script significantly extends the details that are logged
about SSL/TLS handshakes.
I am a bit tempted to just make this part of the default log - but it
does add a bunch logging overhead for each connection.
Extract-certs-pem writes pem files to a dedicated file; since it does
not really work in cluster-environments it was never super helpful.
This commit deprecates this file and, instead, adds
log-certs-base64.zeek, which adds the base64-encoded certificate (which
is basically equivalent with a PEM) to the log-file. Since, nowadays,
the log-files are deduplicates this should not add a huge overhead.
The ICSI notary is pretty much inactive. Furthermore - this approach
does no longer make much sense at this point of time - performing, e.g.,
signed certificate timestamp validation is much more worthwhile.
This commit changes the SSL and X.509 logging formats to something that,
hopefully, slowly approaches what they will look like in the future.
X.509 log is not yet deduplicated; this will come in the future.
This commit introduces two new options, which determine if certificate
issuers and subjects are still logged in ssl.log. The default is to have
the host subject/issuer logged, but to remove client-certificate
information. Client-certificates are not a typically used feature
nowadays.
In the past I thought that this is not super interesting. However, it
turns out that this can actually contain a slew of interresting
information - like operating systems querying for the revocation of
software signing certificates, e.g.
So - let's just enable this as a default log for the future.
- make TLS12 PRF a member function of the analyzer
- use std::string in PRF
- use StringValPtr instead of zeek::StringVal
- replace malloc/free with C++ style allocations
Several limitations still apply:
- TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 is the only supported cipher suite
- Some tests are broken due to a failing assertion regarding bytestring
- No newly written tests for decryption (the patch was tested extensively for our paper)
- Several small open technical questions marked with FIXME
- Architecture in the ssl module might not be optimal
The larger number was substracted from the smaller one leading to an
integer overflow. However, no information was lost due to everything
also being present in the notice message.
Fixes GH-1454
Changes \x00-\x37 ranges to \x00-\x1f with assumption that the former
was attempting to match ASCII control characters, but mistook an octal
range for hex. This change reduces some false positives.
- Tweaked the Too_Little_Traffic notice message to avoid
cluster-specific terminology.
* origin/topic/vlad/caploss_no_traffic:
Fix scheduling due to network_time being 0 in zeek_init
Add test for CaptureLoss::Too_Little_Traffic
Add CaptureLoss::Too_Little_Traffic
Add CaptureLoss::initial_watch_interval for a quick read on cluster health after startup.
Documentation update, reference the threshold variable. [nomail] [skip ci]
Whitespace fixes only [nomail] [skip ci]
generate_all_events causes all events to be raised internally; this
makes it possible for dump_events to really capture all events (and not
just those that were handled).
Addresses GH-169
This adds a "policy" hook into the logging framework's streams and
filters to replace the existing log filter predicates. The hook
signature is as follows:
hook(rec: any, id: Log::ID, filter: Log::Filter);
The logging manager invokes hooks on each log record. Hooks can veto
log records via a break, and modify them if necessary. Log filters
inherit the stream-level hook, but can override or remove the hook as
needed.
The distribution's existing log streams now come with pre-defined
hooks that users can add handlers to. Their name is standardized as
"log_policy" by convention, with additional suffixes when a module
provides multiple streams. The following adds a handler to the Conn
module's default log policy hook:
hook Conn::log_policy(rec: Conn::Info, id: Log::ID, filter: Log::Filter)
{
if ( some_veto_reason(rec) )
break;
}
By default, this handler will get invoked for any log filter
associated with the Conn::LOG stream.
The existing predicates are deprecated for removal in 4.1 but continue
to work.
This adds two new functions: `Conn::register_removal_hook()` and
`Conn::unregister_removal_hook()` for registering a hook function to be
called back during `connection_state_remove`. The benefit of using hook
callback approach is better scalability: the overhead of unrelated
protocols having to dispatch no-op `connection_state_remove` handlers is
avoided.
