Commit graph

860 commits

Author SHA1 Message Date
FlyingWithJerome
c957e3e91e address code reviews (formatting and type and intrusiveptr) 2021-10-12 20:36:35 -04:00
FlyingWithJerome
8fce51bf83 initial commit for SVCB/HTTPS records 2021-10-12 17:43:32 -04:00
Vern Paxson
385e49491b script simplification that removes an unnecessary &is_assigned 2021-09-22 11:18:52 -07:00
FOX-DS
4a19acbef2 Add IType opnum mapping 2021-09-20 03:56:20 -04:00
Robin Sommer
6c128a21e1 Merge branch 'topic/foxds/dcerpc_ms-oaut' of ssh://github.com/fox-ds/zeek
* 'topic/foxds/dcerpc_ms-oaut' of ssh://github.com/fox-ds/zeek:
  Added four new opnum mappings for MS-OAUT IDispatch methods
2021-09-16 11:15:36 +02:00
FOX-DS
d3ca226e96 Added four new opnum mappings for MS-OAUT IDispatch methods 2021-09-14 15:31:08 +02:00
Tim Wojtulewicz
569552b320 GH-1589: Avoid extracting IP-like strings from SMTP headers 2021-09-03 17:35:10 +00:00
Johanna Amann
132b9ef2cc Add a few DNS query types. 2021-07-06 10:40:22 +01:00
Johanna Amann
7ec50bf434 Merge remote-tracking branch 'origin/topic/johanna/gh-859'
* origin/topic/johanna/gh-859:
  Add X509/SSL changes to NEWS
  X509: add check if function succeeds
  GH-1634: Address feedback
  Small indentation fixes in ssl-log-ext.zeek
  Fix memory leak in x509_check_cert_hostname bif
  Small bugfix and updates for external test hashes (SSL/X509)
  Baseline updates for recent SSL changes.
  Add ability to check if hostname is valid for a specific cert
  Add ssl_history field to ssl.log
  Add policy script suppressing certificate events
  Add new ssl-log-ext policy script
  Deprecate extract-certs-pem.zeek and add log-certs-base64.zeek
  Implement X509 certificate log caching
  Deprecate ICSI SSL notary script.
  Change SSL and X.509 logging format
  Enable OCSP logging by default.
  Split the code that handles X509 event hashing into its own file

Closes GH-859
2021-07-05 10:12:46 +01:00
Johanna Amann
8972676e92 GH-1634: Address feedback
This commit addreses feedback for GH-1643, changing typos and renaming
one of the fields in x509.log.
2021-07-02 15:12:58 +01:00
Johanna Amann
a193b08ecf Update Mozilla CA list and CT anchors.
This commit switches to only allowing the CT logs that are currently
accepted by Google Chrome - which makes much more sense for us since
this is (potentially) used for validation.

Additional CT logs can be added in user-scripts.
2021-06-30 15:14:22 +01:00
Johanna Amann
509b501e1b Merge remote-tracking branch 'origin/master' into topic/johanna/gh-859 2021-06-30 10:36:54 +01:00
Robin Sommer
369e42a6e4 Fix SMB tests on Apple M1.
Due to different double precision on M1, file IDs for SMB could end up
changing on M1 because the access time of a file goes into their
computation. The real solution for this would be changing Zeek's
internal "time" representation to uint64; that's planned, but requires
major surgery. For now, this PR changes the SMB code to also pass SMB's
original time representation (which is a uint64) into script-land, and
then use that for computing the file ID.

Closes #1406
2021-06-29 20:17:02 +02:00
Johanna Amann
93d7778f97 Small bugfix and updates for external test hashes (SSL/X509) 2021-06-29 15:25:08 +01:00
Johanna Amann
e4b2fa50a9 Merge remote-tracking branch 'origin/master' into topic/johanna/gh-859 2021-06-29 15:09:56 +01:00
Johanna Amann
833168090a Add ability to check if hostname is valid for a specific cert
This commit adds two new bifs, x509_check_hostname and
x509_check_cert_hostname. These bifs can be used to check if a given
hostname which can, e.g., be sent in a SNI is valid for a specific
certificate.

This PR furthermore modifies the ssl logs again, and adds information
about this to the log-file. Furthermore we now by default remove the
server certificate information from ssl.log - I doubt that this is often
looked at, it is not present in TLS 1.3, we do still have the SNI, and
if you need it you have the information in x509.log.

This also fixes a small potential problem in X509.cc assuming there
might be SAN-entries that contain null-bytes.

Baseline update will follow in another commit.
2021-06-29 15:00:48 +01:00
Johanna Amann
5479ce607a Add ssl_history field to ssl.log
This is the equivalent to a connection history for SSL - and contains
information about which protocol messages were exchanged in which order.

Tests currently don't pass - I will update the ssl.log baselines after
doing another a bit invasive change that will change all the logs.
2021-06-29 14:10:02 +01:00
Johanna Amann
b02f22a667 Change SSL and X.509 logging format
This commit changes the SSL and X.509 logging formats to something that,
hopefully, slowly approaches what they will look like in the future.

X.509 log is not yet deduplicated; this will come in the future.

This commit introduces two new options, which determine if certificate
issuers and subjects are still logged in ssl.log. The default is to have
the host subject/issuer logged, but to remove client-certificate
information. Client-certificates are not a typically used feature
nowadays.
2021-06-29 09:26:43 +01:00
Johanna Amann
a995d73fdf Merge remote-tracking branch 'origin/topic/bbannier/issue-1590'
* origin/topic/bbannier/issue-1590:
  Change SSH version field to be `&optional`.
  Reformat function in SSH base script.

Changes during merge:
  * rename weirds to fit into our naming scheme
  * add NEWS entry

Closes GH-1590
2021-06-17 10:52:49 +01:00
Benjamin Bannier
daa9537f92 Change SSH version field to be &optional.
In 3769ed6c66 we added handling for SSH version 1.99 which unsed a SSH
version of 0 to indicate weird cases where no version could be
determined.

This patch is a fixup for that patch. Instead of using a magic version
of 0 we now use an `&optional` version value. If no SSH version can be
extracted the version will be unset; additionally a `conn_weird` event
will be raised.

Closes #1590.
2021-06-17 09:24:46 +02:00
Benjamin Bannier
e6e30240ec Reformat function in SSH base script. 2021-06-16 15:03:43 +02:00
Vern Paxson
66fb3e0f56 low-level tidying / documentation tweaks 2021-05-05 16:50:29 -07:00
Jon Siwek
76fb1e7fd0 Fixes to decode_netbios_name and decode_netbios_name_type BIFs
Fixes to `decode_netbios_name`:

* Improve validation that input string is a NetBIOS encoding
  (32 bytes, with characters ranging from 'A' to 'P').  This helps
  prevent Undefined Behavior of left-shifting negative values.
  Invalid encodings now cause a return-value of an empty string.

* More liberal in what decoded characters are allowed.  Namely,
  spaces are now allowed (but any trailing null-bytes and spaces
  are trimmed, similar to before).

Fixes to `decode_netbios_name_type`:

* Improve validation that input string is a NetBIOS encoding
  (32 bytes, with characters ranging from 'A' to 'P').  This helps
  prevent Undefined Behavior of left-shifting negative values and
  a heap-buffer-overread when the input string is too small.
  Invalid encodings now cause a return-value of 256.
2021-04-27 15:27:04 -07:00
Vern Paxson
5a8ba8c3e5 lint fixes: ensuring functions return values, robustness to nil Val's 2021-03-18 08:21:19 -07:00
Seth Hall
db7aba0835 ts fields in SMB logs now default to network_time()
This avoids a problem identified by amanbansal2709 in pull
request #1288. I fixed it in a different way than that pull request
by making sure the ts field is always set so that this isssue doesn't
return in the future.
2021-03-10 13:52:44 +00:00
Vern Paxson
4f2b138e72 removing unused assignments from base scripts 2021-02-06 10:59:37 -08:00
Vern Paxson
c991c54690 &is_set => &is_assigned 2021-02-04 12:18:46 -08:00
Vern Paxson
0d77b474e6 adding &is_set attributes to base scripts so -u output isn't cluttered 2021-01-23 10:55:27 -08:00
Jon Siwek
3449bc81af Merge remote-tracking branch 'origin/topic/johanna/update-tls-constants-which-changed-a-lot-since-I-last-looked-which-just-shows-i-should-really-look-at-them-more-regularly-but-luckily-they-are-not-really-used-for-anything-so-it-does-not-really-matter-that-much'
* origin/topic/johanna/update-tls-constants-which-changed-a-lot-since-I-last-looked-which-just-shows-i-should-really-look-at-them-more-regularly-but-luckily-they-are-not-really-used-for-anything-so-it-does-not-really-matter-that-much:
  Update SSL consts
2021-01-21 17:33:36 -08:00
Jon Oakley
25de6f28e5 GH-1352: Added flag to stop processing SMTP headers in attached
messages
2021-01-21 14:55:10 -05:00
Johanna Amann
8ff05bae1b Update SSL consts
There have been a bunch of mostly TLS 1.3 related changes.
2021-01-20 14:14:47 +00:00
Johanna Amann
886d7178ef TLS 1.3 changes: Address review feedback
Only minor changes, new consts, and documentation updates.

Part of GH-1335. Addresses GH-1323.
2020-12-18 10:51:36 +00:00
Johanna Amann
3c95c9a956 Fix TLS 1.3 session resumption detection.
Now we detect TLS 1.3 session resumption by looking if both sides have
the PSK extension set, which is much more exact than the previous
approach.
2020-12-15 16:34:47 +00:00
Johanna Amann
bea3075c1f TLS analyzer: change logic to track TLS 1.3 connection establishment
This commit changes the logic that is used to tracks connection
establishment - and moves it from scriptland into the core.

TLS 1.3 connection establishment is much more finnicky for us than the
establishment of earlier versions - since we cannot rely on the CCS
message anymore (which is meaningless and not sent in a lot of cases).

With this commit, the ssl_encrypted_data message gets raised for
encrypted TLS 1.3 handshake messages - which is much more correct than
the behavior before that just interpreted them as plaintext messages.

I will refine this a bit more - at the moment the connection established
event happens a bit too early - earlier than TLS 1.3 connections
actually can be estasblished.

Part of GH-1323
2020-12-14 19:51:05 +00:00
Johanna Amann
14a448cddc Update Mozilla Root Store 2020-12-09 21:46:32 +00:00
Jon Siwek
02c0b33b54 Merge remote-tracking branch 'origin/topic/jsiwek/gh-1264-ssh-host-key-fingerprints' into master
* origin/topic/jsiwek/gh-1264-ssh-host-key-fingerprints:
  Simply ssh/main.zeek by using "ssh_server_host_key" for fingerprinting
  Deprecate "ssh1_server_host_key" parameters *e* and *p*
  GH-1264: Implement "ssh_server_host_key" event
2020-11-16 11:22:37 -08:00
Jon Siwek
331b94db39 Simply ssh/main.zeek by using "ssh_server_host_key" for fingerprinting 2020-11-14 08:40:27 -08:00
Brevet Vivien
3769ed6c66 [SSH] Handle SSH version 1.99
SSH can set in its identification a version 1.99 (SSH-1.99-xxx).
That means the client/server is compatible with SSHv1 and SSHv2.
So the version choice depends of the both side.

1.99 : 1.99 => 2.0
1.99 : 1.x  => 1.x
1.99 : 2.0  => 2.O

(see "Compatibility With Old SSH Versions" in RFC 4253)
2020-11-14 15:33:34 +01:00
Jon Siwek
45449dad72 Deprecate "ssh1_server_host_key" parameters *e* and *p*
They are named such that *e* is actually the modulus, not the exponent.
The replacement parameters are named *exponent* and *modulus* for
clarity.
2020-11-13 22:58:56 -08:00
Jon Siwek
bd40a97a78 GH-1264: Implement "ssh_server_host_key" event
This event provides host key fingerprints for both SSH1 and SSH2.
2020-11-13 22:58:56 -08:00
Tim Wojtulewicz
a6a4b976ec Support for additional DNS RR Type: LOC[29], SSHFP[44], NSEC3PARAM[51], custom BIND9 signaling[65534] 2020-11-11 13:35:51 -07:00
Christian Kreibich
1bd658da8f Support for log filter policy hooks
This adds a "policy" hook into the logging framework's streams and
filters to replace the existing log filter predicates. The hook
signature is as follows:

    hook(rec: any, id: Log::ID, filter: Log::Filter);

The logging manager invokes hooks on each log record. Hooks can veto
log records via a break, and modify them if necessary. Log filters
inherit the stream-level hook, but can override or remove the hook as
needed.

The distribution's existing log streams now come with pre-defined
hooks that users can add handlers to. Their name is standardized as
"log_policy" by convention, with additional suffixes when a module
provides multiple streams. The following adds a handler to the Conn
module's default log policy hook:

    hook Conn::log_policy(rec: Conn::Info, id: Log::ID, filter: Log::Filter)
            {
            if ( some_veto_reason(rec) )
                break;
            }

By default, this handler will get invoked for any log filter
associated with the Conn::LOG stream.

The existing predicates are deprecated for removal in 4.1 but continue
to work.
2020-09-30 12:32:45 -07:00
Tim Wojtulewicz
6b93020c6f Merge remote-tracking branch 'origin/topic/jsiwek/gh-1164-fix-rst-conn-state'
* origin/topic/jsiwek/gh-1164-fix-rst-conn-state:
  GH-1164: Fix incorrect RSTOS0 conn_state determinations
2020-09-17 13:40:06 -07:00
Jon Siwek
a16bd47bf7 GH-1164: Fix incorrect RSTOS0 conn_state determinations
The RSTOS0 `conn_state` label is documented as "Originator sent a SYN
followed by a RST, never saw SYN-ACK from responder", but was previously
applied to cases where no originator SYN exists, like a single RST-only
packet.
2020-09-11 16:14:41 -07:00
Jon Siwek
05cf511f18 GH-1119: add base/protcols/conn/removal-hooks.zeek
This adds two new functions: `Conn::register_removal_hook()` and
`Conn::unregister_removal_hook()` for registering a hook function to be
called back during `connection_state_remove`.  The benefit of using hook
callback approach is better scalability: the overhead of unrelated
protocols having to dispatch no-op `connection_state_remove` handlers is
avoided.
2020-09-11 12:12:10 -07:00
Jon Siwek
5f435c2644 Remove connection_successful and successful_connection_remove events
Related to https://github.com/zeek/zeek/issues/1119
2020-09-10 12:06:50 -07:00
Robin Sommer
0af57d12b2 Change HTTP's DPD signatures so that each side can trigger the analyzer on its own.
This is to avoid missing large sessions where a single side exceeds
the DPD buffer size. It comes with the trade-off that now the analyzer
can be triggered by anybody controlling one of the endpoints (instead
of both).

Test suite changes are minor, and nothing in "external".

Closes #343.
2020-09-08 07:33:36 +00:00
FlyingWithJerome
74efbd0abc add EDNS cookie parsing 2020-08-20 09:04:56 -04:00
Johanna Amann
b948180247 Fix minimize_info in ftp/main not returning a value.
Fixes GH-1120
2020-08-12 19:53:53 +00:00
Jon Siwek
fef4fda2f7 Fix DNS script deleting a table element while iterating
Doesn't typically cause any problems since the loop breaks after
deleting, except there's now an assert in debug builds catching
potential problems like this.
2020-08-11 11:26:21 -07:00