The invalidset and invalidtext tests loaded an input file via table
and event reads, in parallel. On FreeBSD this triggers an occasional
reordering of messages coming out of the reader thread vs the input
managers. This commit makes the table and event reads sequential,
avoiding the race.
* origin/topic/johanna/gh-859:
Add X509/SSL changes to NEWS
X509: add check if function succeeds
GH-1634: Address feedback
Small indentation fixes in ssl-log-ext.zeek
Fix memory leak in x509_check_cert_hostname bif
Small bugfix and updates for external test hashes (SSL/X509)
Baseline updates for recent SSL changes.
Add ability to check if hostname is valid for a specific cert
Add ssl_history field to ssl.log
Add policy script suppressing certificate events
Add new ssl-log-ext policy script
Deprecate extract-certs-pem.zeek and add log-certs-base64.zeek
Implement X509 certificate log caching
Deprecate ICSI SSL notary script.
Change SSL and X.509 logging format
Enable OCSP logging by default.
Split the code that handles X509 event hashing into its own file
Closes GH-859
This addresses the need for a central hook on any log write, which
wasn't previously doable without a lot of effort. The log manager
invokes the new Log::log_stream_policy hook prior to any filter-specific
hooks. Like filter-level hooks, it may veto a log write. Even when
it does, filter-level hooks still get invoked, but cannot "un-veto".
Includes test cases.
The framework so far populated data structures with missing fields
even when those fields are defined without the &optional
attribute. When using the attribute, such entries continue to get
populated.
Update tests to reflect focus on unset fields.
This commit adds two new bifs, x509_check_hostname and
x509_check_cert_hostname. These bifs can be used to check if a given
hostname which can, e.g., be sent in a SNI is valid for a specific
certificate.
This PR furthermore modifies the ssl logs again, and adds information
about this to the log-file. Furthermore we now by default remove the
server certificate information from ssl.log - I doubt that this is often
looked at, it is not present in TLS 1.3, we do still have the SNI, and
if you need it you have the information in x509.log.
This also fixes a small potential problem in X509.cc assuming there
might be SAN-entries that contain null-bytes.
Baseline update will follow in another commit.
The added disable-certificate-events-known-certs.zeek disables repeated
X509 events in SSL connections, given that the connection terminates at
the same server and used the samt SNI as a previously seen connection
with the same certificate.
For people that see significant amounts of TLS 1.2 traffic, this could
reduce the amount of raised events significantly - especially when a
lot of connections are repeat connections to the same servers.
The practical impact of not raising these events is actually very little
- unless a script directly interacts with the x509 events, everything
works as before - the x509 variables in the connection records are still
being set (from the cache).
By default, each certificate is now output only once per hour. This also
should work in cluster mode, where we use the net broker-table-syncing
feature to distribute the information about already seen certificates
across the entire cluster.
Log caching is also pretty configureable and can be changed using a
range of confiuration options and hooks.
Note that this is currently completely separate from X509 events
caching, which prevents duplicate parsing of X509 certificates.
This commit changes the SSL and X.509 logging formats to something that,
hopefully, slowly approaches what they will look like in the future.
X.509 log is not yet deduplicated; this will come in the future.
This commit introduces two new options, which determine if certificate
issuers and subjects are still logged in ssl.log. The default is to have
the host subject/issuer logged, but to remove client-certificate
information. Client-certificates are not a typically used feature
nowadays.
In the past I thought that this is not super interesting. However, it
turns out that this can actually contain a slew of interresting
information - like operating systems querying for the revocation of
software signing certificates, e.g.
So - let's just enable this as a default log for the future.
* origin/topic/vern/ZAM-prep: (45 commits)
whoops overlooked the need to canonicalize filenames
another set of tweaks per review comments
addressed a number of code review comments
baseline updates for merge
support "any" coercions for "-O gen-C++"
better descriptions for named record constructors
test suite baseline updates for "-a opt" optimize-AST alternative
test suite baseline updates for "-a xform" alternative / AST transformation
error propagation fix for AST reduction
updates to "-a inline" test suite alternative baseline
updates for the main test suite baseline
updates to test suite tests for compatibility with upcoming ZAM functionality
"-O compile-all" option to specify compilation of inlined functions
compile inlined functions if they're also used indirectly
provide ZAM-generated code with low-level access to record fields
fix for cloning records with fields of type "any"
direct access for ZAM to VectorVal internal vector
ZVal constructors, accessors & methods in support of ZAM
switch ZVal representation of types from Type objects to TypeVal's
revised error-reporting interface for ZVal's, to accommodate ZAM inner loop
...
* origin/topic/vlad/page_hostnames:
Expand the tests, and have email_admin email src contacts in one e-mail, and dst contacts in another.
Simplify the logic a big by making Notice$email_dest have a default rather than be optional
Missed some baselines
Update hashes in external testing repos
Update notice.log baselines
Remove unneccessary @load
Add tests for e-mail actions, and cleanup the new logic a bit.
Fixup priorities
Add a new field to the notice, which defines where to send the e-mail
Add page and email_admin to hostnames extension
In 3769ed6c66 we added handling for SSH version 1.99 which unsed a SSH
version of 0 to indicate weird cases where no version could be
determined.
This patch is a fixup for that patch. Instead of using a magic version
of 0 we now use an `&optional` version value. If no SSH version can be
extracted the version will be unset; additionally a `conn_weird` event
will be raised.
Closes#1590.
* 'logging/script-logdir' of https://github.com/kramse/zeek:
Copy of ascii-empty test, just changed path in the beginning
Logdir: Change requested by 0xxon, no problem
Introduce script-land variable that can be used to set logdir.
Closes GH-772
* origin/topic/timw/1487-not-valid-enum:
Move an assert() in input/Manager.cc to account for ValueToVal errors
Add test for config framework
Fix similar issues with ValueTo* methods in the input framework
GH-1487: Handle error from ValueToVal instead of ignoring it
This allows for data that won't match a SIP request method to precede an
actual request and generates a new 'sip_junk_before_request' weird when
encountering such a situation.
The modp_dtoa/modp_dtoa2 functions aren't capable of handling double
values larger than INT_MAX and fallback on using sprintf() in that
situation. Previously, the format string to that sprintf() was "%e",
defaulting to a precision of 6, which is already too few digits to
represent a number known to be larger than INT_MAX. Now, an sprintf()
is still performed for values larger than INT_MAX and still uses a
scientific notation format, but in a way that uses as many decimal
digits as needed to preserve information.
When a text with an (escaped) zero byte was passed to ParseValue, only
the part of the string up to the zero byte was copied, but the length of
the full string was passed to the input framework.
This leads to the input manager reading over the end of the buffer.
Fixeszeek/zeek#1398
Merge adjustments:
- Removed some stale str_split() references from docs
- Renumbered TypeTag enum comments
- Simplified test-case for @unload (don't need .bro files anymore)
* origin/topic/timw/deprecation-cleanup:
Doc updates
Fix language.init-in-anon-function btest due to changes to log filter predicates
Remove deprecated log filter predicates for 4.1
Remove Plugin::HookCallFunction and fix tests related to it
Remove support for .bro script extension and BRO_ environment variables
Remove deprecated ICMP events
Remove some deprected methods/events from bif files
Remove TYPE_COUNTER
Remove all of the random single-file deprecations
Remove all fully-deprecated files
Update bifcl submodule to remove deprecations from generated code
Script-layer counts, when provided as negative integers in an input
file, got cast to unsigned values because strtoull() does not complain
about negative values. For example, input string "-1" would lead to
value 18446744073709551615 (an all-ones 64-bit int) on x86_64. This is
more likely to be an error than an intent to get very large,
platform-dependent values, so these input lines are now skipped with
according messaging in the reporter.log/stderr.
This also affected ports: -1/tcp got cast to unsigned and only thrown
out because PortVal rejects values > 65535, mapping them to 0. We now
skip such inputs as well.
Updates existing input framework tests to capture the new behavior.
Update the logging framework tests: since hooks operate
by name, they cannot be anonymous. I'm also dropping the &optional
attribute from the status field, since here know that the values are
actually defined, and access to an optional status field should
normally be guarded by the existence test operator.
Also includes baseline update for plugins.hooks, which picks up the
fact that the pred record field is now gone.
