This adds a new function for validating UTF-8 sequences by converting to
UTF-32. This allows us to also check for various blocks of codepointsi
that we consider invalid while checking for valid sequences in general.
We previously computed the length of the Geneve options field
incorrectly which lead to us passing data at an incorrect offset to
inner analyzers. With this patch we now interpret the length field
correctly, according the the spec
https://datatracker.ietf.org/doc/html/rfc8926#section-3.4.
Closes#1726.
This could lead to a noticeable (single-percent) performance
improvement.
Most of the functionality for this is in the packet analyzers that now
cache ignore_chesksums_nets.
Based on a patch by Arne Welzel (Corelight).
When cloning TableVals, a new timer was created for the wrong object
(the existing TableVal, not the clone). This lead to the already
existing timer being no longer accessible. Which, in turn, leads to an
abandoned timer reading into no longer allocated data when the original
TableVal is deleted.
Fixes GH-1687
With the recent packet manager work, it broke to register a protocol
analyzer for a specific port from inside a plugin's initialization code.
That's because that registration now depends on the packet manager being
set up, which isn't case at that time a plugin's `InitPostInit()` runs.
This fix contains two parts:
- Initialize the packet manager before the analyzer manager, so that
the latter's `InitPostScript()` can rely on the former being
ready.
- Change the analyzer manager to (only) record port registrations
happening before it's fully initialized. Its `InitPostScript()`
then performs the actual registrations, knowing it can use the
packet manager now.
This comes with a `cmake/` to add a missing include directory.
* origin/topic/vlad/dns_rr_fixes:
Add a TODO to return a correct status for ParseRR_WKS
Remove unnecessary if
Generate dns_WKS_reply event
dns_HINFO_reply event was never being generated.
On FreeBSD, this test showed two problems: (1) reordering problems
based on writing the predicate, event, and end-of-data updates into a
single file, (2) a race condition based on printing the entirety of
the table description argument in update events. The description
contains the destination table, and its content at the time an update
event gets processed isn't deterministic: depending on the number
of updates the reader thread has sent, the table will contain a
varying number of entries.
The invalidset and invalidtext tests loaded an input file via table
and event reads, in parallel. On FreeBSD this triggers an occasional
reordering of messages coming out of the reader thread vs the input
managers. This commit makes the table and event reads sequential,
avoiding the race.
When omitted, the node inherits the Supervisor's bare-mode
status. When true/false, the new Zeek node will enable/disable bare
mode, respectively. It continues to load any scripts passed at the
command line and in the additional scripts list already provided in
the node configuration.
Includes testcase.
The NodeConfig record now has a table for specifying environment variable names
and values, which the supervisor sets in the created node.
This also repositions the cpu_affinity member to keep the order the same in
the corresponding script-layer and in-core types.
Includes testcase.
* origin/topic/johanna/gh-859:
Add X509/SSL changes to NEWS
X509: add check if function succeeds
GH-1634: Address feedback
Small indentation fixes in ssl-log-ext.zeek
Fix memory leak in x509_check_cert_hostname bif
Small bugfix and updates for external test hashes (SSL/X509)
Baseline updates for recent SSL changes.
Add ability to check if hostname is valid for a specific cert
Add ssl_history field to ssl.log
Add policy script suppressing certificate events
Add new ssl-log-ext policy script
Deprecate extract-certs-pem.zeek and add log-certs-base64.zeek
Implement X509 certificate log caching
Deprecate ICSI SSL notary script.
Change SSL and X.509 logging format
Enable OCSP logging by default.
Split the code that handles X509 event hashing into its own file
Closes GH-859
This addresses the need for a central hook on any log write, which
wasn't previously doable without a lot of effort. The log manager
invokes the new Log::log_stream_policy hook prior to any filter-specific
hooks. Like filter-level hooks, it may veto a log write. Even when
it does, filter-level hooks still get invoked, but cannot "un-veto".
Includes test cases.
The framework so far populated data structures with missing fields
even when those fields are defined without the &optional
attribute. When using the attribute, such entries continue to get
populated.
Update tests to reflect focus on unset fields.
* origin/topic/seth/small-builtin-plugin-fixes:
Fix tests
Removed a non-functional builtin plugin preload loading mechanism
Load the builtin-plugin preload files and fix the dev path.
This commit switches to only allowing the CT logs that are currently
accepted by Google Chrome - which makes much more sense for us since
this is (potentially) used for validation.
Additional CT logs can be added in user-scripts.
Due to different double precision on M1, file IDs for SMB could end up
changing on M1 because the access time of a file goes into their
computation. The real solution for this would be changing Zeek's
internal "time" representation to uint64; that's planned, but requires
major surgery. For now, this PR changes the SMB code to also pass SMB's
original time representation (which is a uint64) into script-land, and
then use that for computing the file ID.
Closes#1406
This commit adds two new bifs, x509_check_hostname and
x509_check_cert_hostname. These bifs can be used to check if a given
hostname which can, e.g., be sent in a SNI is valid for a specific
certificate.
This PR furthermore modifies the ssl logs again, and adds information
about this to the log-file. Furthermore we now by default remove the
server certificate information from ssl.log - I doubt that this is often
looked at, it is not present in TLS 1.3, we do still have the SNI, and
if you need it you have the information in x509.log.
This also fixes a small potential problem in X509.cc assuming there
might be SAN-entries that contain null-bytes.
Baseline update will follow in another commit.
This is the equivalent to a connection history for SSL - and contains
information about which protocol messages were exchanged in which order.
Tests currently don't pass - I will update the ssl.log baselines after
doing another a bit invasive change that will change all the logs.
The added disable-certificate-events-known-certs.zeek disables repeated
X509 events in SSL connections, given that the connection terminates at
the same server and used the samt SNI as a previously seen connection
with the same certificate.
For people that see significant amounts of TLS 1.2 traffic, this could
reduce the amount of raised events significantly - especially when a
lot of connections are repeat connections to the same servers.
The practical impact of not raising these events is actually very little
- unless a script directly interacts with the x509 events, everything
works as before - the x509 variables in the connection records are still
being set (from the cache).
This policy script significantly extends the details that are logged
about SSL/TLS handshakes.
I am a bit tempted to just make this part of the default log - but it
does add a bunch logging overhead for each connection.
Extract-certs-pem writes pem files to a dedicated file; since it does
not really work in cluster-environments it was never super helpful.
This commit deprecates this file and, instead, adds
log-certs-base64.zeek, which adds the base64-encoded certificate (which
is basically equivalent with a PEM) to the log-file. Since, nowadays,
the log-files are deduplicates this should not add a huge overhead.
By default, each certificate is now output only once per hour. This also
should work in cluster mode, where we use the net broker-table-syncing
feature to distribute the information about already seen certificates
across the entire cluster.
Log caching is also pretty configureable and can be changed using a
range of confiuration options and hooks.
Note that this is currently completely separate from X509 events
caching, which prevents duplicate parsing of X509 certificates.
The ICSI notary is pretty much inactive. Furthermore - this approach
does no longer make much sense at this point of time - performing, e.g.,
signed certificate timestamp validation is much more worthwhile.
This commit changes the SSL and X.509 logging formats to something that,
hopefully, slowly approaches what they will look like in the future.
X.509 log is not yet deduplicated; this will come in the future.
This commit introduces two new options, which determine if certificate
issuers and subjects are still logged in ssl.log. The default is to have
the host subject/issuer logged, but to remove client-certificate
information. Client-certificates are not a typically used feature
nowadays.
In the past I thought that this is not super interesting. However, it
turns out that this can actually contain a slew of interresting
information - like operating systems querying for the revocation of
software signing certificates, e.g.
So - let's just enable this as a default log for the future.
* origin/topic/vern/ZAM-prep: (45 commits)
whoops overlooked the need to canonicalize filenames
another set of tweaks per review comments
addressed a number of code review comments
baseline updates for merge
support "any" coercions for "-O gen-C++"
better descriptions for named record constructors
test suite baseline updates for "-a opt" optimize-AST alternative
test suite baseline updates for "-a xform" alternative / AST transformation
error propagation fix for AST reduction
updates to "-a inline" test suite alternative baseline
updates for the main test suite baseline
updates to test suite tests for compatibility with upcoming ZAM functionality
"-O compile-all" option to specify compilation of inlined functions
compile inlined functions if they're also used indirectly
provide ZAM-generated code with low-level access to record fields
fix for cloning records with fields of type "any"
direct access for ZAM to VectorVal internal vector
ZVal constructors, accessors & methods in support of ZAM
switch ZVal representation of types from Type objects to TypeVal's
revised error-reporting interface for ZVal's, to accommodate ZAM inner loop
...
* origin/topic/christian/gh-1600:
Remove unnecessary -B arguments from Zeek invocations in testsuite
Fix perftools-enabled build
Minor tweaks to single-character command line option handling
* origin/topic/vlad/page_hostnames:
Expand the tests, and have email_admin email src contacts in one e-mail, and dst contacts in another.
Simplify the logic a big by making Notice$email_dest have a default rather than be optional
Missed some baselines
Update hashes in external testing repos
Update notice.log baselines
Remove unneccessary @load
Add tests for e-mail actions, and cleanup the new logic a bit.
Fixup priorities
Add a new field to the notice, which defines where to send the e-mail
Add page and email_admin to hostnames extension
