We used to attempt to remove any port specification before recording
HTTP host headers in logs. Doing so would (1) remove potentially useful
information, (2) not match what the documentation seemed to suggest, and
(3) fail for IP6 addresses containing colons.
We now record the original HOST header as is.
Addresses #1844.
Merge includes small changes, e.g. fixing the comsumption of remaining
raw data.
* 'master' of https://github.com/FlyingWithJerome/zeek:
remove excussive fields in dns_svcb_rr
address code reviews (formatting and type and intrusiveptr)
newlines at the end of test outputs
lazy commit
use tabs in init-bare.zeek
add svcb test case
add a dns https test case
remove test logs
fix a few syntax errors
initial commit for SVCB/HTTPS records
* origin/topic/johanna/gh-859:
Add X509/SSL changes to NEWS
X509: add check if function succeeds
GH-1634: Address feedback
Small indentation fixes in ssl-log-ext.zeek
Fix memory leak in x509_check_cert_hostname bif
Small bugfix and updates for external test hashes (SSL/X509)
Baseline updates for recent SSL changes.
Add ability to check if hostname is valid for a specific cert
Add ssl_history field to ssl.log
Add policy script suppressing certificate events
Add new ssl-log-ext policy script
Deprecate extract-certs-pem.zeek and add log-certs-base64.zeek
Implement X509 certificate log caching
Deprecate ICSI SSL notary script.
Change SSL and X.509 logging format
Enable OCSP logging by default.
Split the code that handles X509 event hashing into its own file
Closes GH-859
This commit switches to only allowing the CT logs that are currently
accepted by Google Chrome - which makes much more sense for us since
this is (potentially) used for validation.
Additional CT logs can be added in user-scripts.
Due to different double precision on M1, file IDs for SMB could end up
changing on M1 because the access time of a file goes into their
computation. The real solution for this would be changing Zeek's
internal "time" representation to uint64; that's planned, but requires
major surgery. For now, this PR changes the SMB code to also pass SMB's
original time representation (which is a uint64) into script-land, and
then use that for computing the file ID.
Closes#1406
This commit adds two new bifs, x509_check_hostname and
x509_check_cert_hostname. These bifs can be used to check if a given
hostname which can, e.g., be sent in a SNI is valid for a specific
certificate.
This PR furthermore modifies the ssl logs again, and adds information
about this to the log-file. Furthermore we now by default remove the
server certificate information from ssl.log - I doubt that this is often
looked at, it is not present in TLS 1.3, we do still have the SNI, and
if you need it you have the information in x509.log.
This also fixes a small potential problem in X509.cc assuming there
might be SAN-entries that contain null-bytes.
Baseline update will follow in another commit.
This is the equivalent to a connection history for SSL - and contains
information about which protocol messages were exchanged in which order.
Tests currently don't pass - I will update the ssl.log baselines after
doing another a bit invasive change that will change all the logs.
This commit changes the SSL and X.509 logging formats to something that,
hopefully, slowly approaches what they will look like in the future.
X.509 log is not yet deduplicated; this will come in the future.
This commit introduces two new options, which determine if certificate
issuers and subjects are still logged in ssl.log. The default is to have
the host subject/issuer logged, but to remove client-certificate
information. Client-certificates are not a typically used feature
nowadays.
* origin/topic/bbannier/issue-1590:
Change SSH version field to be `&optional`.
Reformat function in SSH base script.
Changes during merge:
* rename weirds to fit into our naming scheme
* add NEWS entry
Closes GH-1590
In 3769ed6c66 we added handling for SSH version 1.99 which unsed a SSH
version of 0 to indicate weird cases where no version could be
determined.
This patch is a fixup for that patch. Instead of using a magic version
of 0 we now use an `&optional` version value. If no SSH version can be
extracted the version will be unset; additionally a `conn_weird` event
will be raised.
Closes#1590.
Fixes to `decode_netbios_name`:
* Improve validation that input string is a NetBIOS encoding
(32 bytes, with characters ranging from 'A' to 'P'). This helps
prevent Undefined Behavior of left-shifting negative values.
Invalid encodings now cause a return-value of an empty string.
* More liberal in what decoded characters are allowed. Namely,
spaces are now allowed (but any trailing null-bytes and spaces
are trimmed, similar to before).
Fixes to `decode_netbios_name_type`:
* Improve validation that input string is a NetBIOS encoding
(32 bytes, with characters ranging from 'A' to 'P'). This helps
prevent Undefined Behavior of left-shifting negative values and
a heap-buffer-overread when the input string is too small.
Invalid encodings now cause a return-value of 256.
This avoids a problem identified by amanbansal2709 in pull
request #1288. I fixed it in a different way than that pull request
by making sure the ts field is always set so that this isssue doesn't
return in the future.
This commit changes the logic that is used to tracks connection
establishment - and moves it from scriptland into the core.
TLS 1.3 connection establishment is much more finnicky for us than the
establishment of earlier versions - since we cannot rely on the CCS
message anymore (which is meaningless and not sent in a lot of cases).
With this commit, the ssl_encrypted_data message gets raised for
encrypted TLS 1.3 handshake messages - which is much more correct than
the behavior before that just interpreted them as plaintext messages.
I will refine this a bit more - at the moment the connection established
event happens a bit too early - earlier than TLS 1.3 connections
actually can be estasblished.
Part of GH-1323
* origin/topic/jsiwek/gh-1264-ssh-host-key-fingerprints:
Simply ssh/main.zeek by using "ssh_server_host_key" for fingerprinting
Deprecate "ssh1_server_host_key" parameters *e* and *p*
GH-1264: Implement "ssh_server_host_key" event
SSH can set in its identification a version 1.99 (SSH-1.99-xxx).
That means the client/server is compatible with SSHv1 and SSHv2.
So the version choice depends of the both side.
1.99 : 1.99 => 2.0
1.99 : 1.x => 1.x
1.99 : 2.0 => 2.O
(see "Compatibility With Old SSH Versions" in RFC 4253)
This adds a "policy" hook into the logging framework's streams and
filters to replace the existing log filter predicates. The hook
signature is as follows:
hook(rec: any, id: Log::ID, filter: Log::Filter);
The logging manager invokes hooks on each log record. Hooks can veto
log records via a break, and modify them if necessary. Log filters
inherit the stream-level hook, but can override or remove the hook as
needed.
The distribution's existing log streams now come with pre-defined
hooks that users can add handlers to. Their name is standardized as
"log_policy" by convention, with additional suffixes when a module
provides multiple streams. The following adds a handler to the Conn
module's default log policy hook:
hook Conn::log_policy(rec: Conn::Info, id: Log::ID, filter: Log::Filter)
{
if ( some_veto_reason(rec) )
break;
}
By default, this handler will get invoked for any log filter
associated with the Conn::LOG stream.
The existing predicates are deprecated for removal in 4.1 but continue
to work.
The RSTOS0 `conn_state` label is documented as "Originator sent a SYN
followed by a RST, never saw SYN-ACK from responder", but was previously
applied to cases where no originator SYN exists, like a single RST-only
packet.
