Commit graph

185 commits

Author SHA1 Message Date
Johanna Amann
273731e1ce Merge branch 'topic/johanna/fix-logging-of-ssl-log-ext-in-some-cases'
* topic/johanna/fix-logging-of-ssl-log-ext-in-some-cases:
  Fix ssl-log-ext omitting data in some cases
2024-01-16 13:10:35 +00:00
Johanna Amann
46e54bab76 Fix ssl-log-ext omitting data in some cases
ssl-log-ext had a bug that caused data present in the SSL connection to
not be logged in some cases. Specifically, the script relied on the base
ssl script to initialize some data structures; however, this means that
protocol messages that arrive before a message is handled by the base
ssl script are not logged.

This commit changes the ssl-log-ext script to also initialize the data
structures; now messages are correctly included in the log in all cases.
2024-01-11 17:12:33 +00:00
Arne Welzel
60d8872969 Known: Keep &create_expire on local tables/sets valid
After switching the known scripts away from broker stores, the
&create_expire value of the local tables/sets of the known scripts
wasn't in effect due to Cluster::node_up() and Cluster::node_down()
re-assigning these without keeping the &create_expire attribute
intact. This broke the "log hosts every 24h" behavior.

Closes #3540
2024-01-10 09:07:43 +01:00
Vern Paxson
a927d61e59 minor fixes for script optimization, exporting of attr_name, script layout tweak 2023-12-04 16:48:19 -08:00
Tim Wojtulewicz
5a3abbe364 Revert "Merge remote-tracking branch 'origin/topic/vern/at-if-analyze'"
This reverts commit 4e797ddbbc, reversing
changes made to 3ac28ba5a2.
2023-05-31 09:20:33 +02:00
Vern Paxson
e441ba394a updates reflecting review comments 2023-05-25 18:00:13 -07:00
Vern Paxson
890010915a change base scripts to use run-time if's or @if ... &analyze 2023-05-19 13:26:27 -07:00
Vern Paxson
9560e38784 fixed type mismatch for ssl_certificate_request event 2023-04-01 12:56:54 -07:00
Johanna Amann
b8d658ac77 SSL/TLS CertificateRequest message: Address review feedback
Minor stylistic changes; see https://github.com/zeek/zeek/pull/2855 for
details
2023-03-15 09:19:26 +01:00
Johanna Amann
b56b856da9 SSL/TLS: Parse CertificateRequest message
This commit introduces parsing of the CertificateRequest message in the
TLS handshake. It introduces a new event ssl_certificate_request, as
well as a new function parse_distinguished_name, which can be used to
parse part of the ssl_certificate_request event parameters.

This commit also introduces a new policy script, which appends
information about the CAs a TLS server requests in the
CertificateRequest message, if it sends it.
2023-03-09 09:12:29 +01:00
Tim Wojtulewicz
f6d3ed2395 Default known stores to false to avoid Broker stores 2023-03-02 12:48:53 -07:00
Josh Soref
21e0d777b3 Spelling fixes: scripts
* accessing
* across
* adding
* additional
* addresses
* afterwards
* analyzer
* ancillary
* answer
* associated
* attempts
* because
* belonging
* buffer
* cleanup
* committed
* connects
* database
* destination
* destroy
* distinguished
* encoded
* entries
* entry
* hopefully
* image
* include
* incorrect
* information
* initial
* initiate
* interval
* into
* java
* negotiation
* nodes
* nonexistent
* ntlm
* occasional
* omitted
* otherwise
* ourselves
* paragraphs
* particular
* perform
* received
* receiver
* referring
* release
* repetitions
* request
* responded
* retrieval
* running
* search
* separate
* separator
* should
* synchronization
* target
* that
* the
* threshold
* timeout
* transaction
* transferred
* transmission
* triggered
* vetoes
* virtual

Signed-off-by: Josh Soref <2119212+jsoref@users.noreply.github.com>
2022-11-02 17:36:39 -04:00
Arne Welzel
8c5896a74d scripts: Migrate table iteration to blank identifiers
No obvious hot-cases. Maybe the describe_file() ones or the intel ones
if/when there are hot intel hits.
2022-10-24 10:36:09 +02:00
Johanna Amann
6e1e6fefe5 Merge remote-tracking branch 'origin/topic/johanna/2198'
* origin/topic/johanna/2198:
  SSL/GH-2211: Address review feedback, remove USE_FLIPPED
  SSL Analyzer: track connection direction by messages
2022-07-05 15:09:34 +01:00
Tim Wojtulewicz
509718b51c GH-2229: Fix some typos in weak-keys.zeek 2022-06-30 15:12:10 -07:00
Tim Wojtulewicz
6130d32440 Remove some deprecated ocsp/ssl base scripts 2022-06-30 19:17:08 +00:00
Johanna Amann
e14eddeb97 SSL Analyzer: track connection direction by messages
This PR changes the way in which the SSL analyzer tracks the direction
of connections. So far, the SSL analyzer assumed that the originator of
a connection would send the client hello (and other associated
client-side events), and that the responder would be the SSL servers.

In some circumstances this is not true, and the initiator of a
connection is the server, with the responder being the client. So far
this confused some of the internal statekeeping logic and could lead to
mis-parsing of extensions.

This reversal of roles can happen in DTLS, if a connection uses STUN -
and potentially in some StartTLS protocols.

This PR tracks the direction of a TLS connection using the hello
request, client hello and server hello handshake messages. Furthermore,
it changes the SSL events from providing is_orig to providing is_client,
where is_client is true for the client_side of a connection. Since the
argument positioning in the event has not changed, old scripts will
continue to work seamlessly - the new semantics are what everyone
writing SSL scripts will have expected in any case.

There is a new event that is raised when a connection is flipped. A
weird is raised if a flip happens repeatedly.

Addresses GH-2198.
2022-06-24 18:35:44 +01:00
Johanna Amann
5f04f216bc Include certificate information in SSL::Weak_Key notice 2022-05-11 18:56:04 +01:00
Johanna Amann
d38923cfcf Merge remote-tracking branch 'origin/topic/johanna/tls12-decryption'
Documentation is missing and will be added in the next couple of hours.

* origin/topic/johanna/tls12-decryption: (24 commits)
  TLS decryption: add test, fix small issues
  Address PR feedback
  TLS decryption: refactoring, more comments, less bare pointers
  Small code fix and test baseline update.
  SSL decryption: refactor TLS12_PRF
  SSL decryption: small style changes, a bit of documentation
  Deprecation and warning fixes
  Clang-format updates
  add missing call to EVP_KDF_CTX_set_params
  TLS decryption: remove payload from ssl_encrypted_data again.
  TLS 1.2 decryption: adapt OpenSSL 3.0 changes for 1.1
  ssl: adapt TLS-PRF to openSSL 3.0
  ssl/analyzer: potentially fix memory leaks caused by bytestrings
  analyzer/ssl: several improvements
  analyzer/ssl: defensive key length check + more debug logging
  testing: feature gate ssl/decryption test
  testing: add ssl/decryption test
  analyzer/ssl: handle missing <openssl/kdf.h>
  analyzer/ssl: silence warning in DTLS analyzer
  analyzer/ssl: move proc-{client,server}-hello into the respective analyzers
  ...
2022-03-02 08:20:39 +00:00
Johanna Amann
590d4aa13e TLS decryption: add test, fix small issues
Add a test loading keys from an external file. Make some debug messages
slightly better and remove unnecessary debug output.
2022-03-01 17:45:11 +00:00
Johanna Amann
1c9ea09d9f Address PR feedback
This addresses feedback to GH-1814. The most significant change is the
fact that the ChipertextRecord now can remain &transient - which might
lead to improved speed.
2022-02-23 11:31:21 +00:00
Johanna Amann
95f1565498 Match DPD TLS signature on one-sided connections.
This commit changes DPD matching for TLS connections. A one-sided match
is enough to enable DPD now.

This commit also removes DPD for SSLv2 connections. SSLv2 connections do
basically no longer happen in the wild. SSLv2 is also really finnicky to
identify correctly - there is very little data required to match it, and
basically all matches today will be false positives. If DPD for SSLv2 is
still desired, the optional signature in policy/protocols/ssl/dpd-v2.sig
can be loaded.

Fixes GH-1952
2022-02-01 16:51:21 +00:00
Johanna Amann
b78f30339f TLS decryption: refactoring, more comments, less bare pointers
This commit refactors TLS decryption, adds more comments in scripts and
in C++ source-code, and removes use of bare pointers, instead relying
more on stl data types.
2022-01-17 15:04:44 +00:00
Johanna Amann
689b06d9bd Merge remote-tracking branch 'origin/master' into topic/johanna/tls12-decryption 2022-01-17 10:56:06 +00:00
Vern Paxson
98cd3f2213 update uses of "when" in base scripts to include captures 2022-01-07 14:53:33 -08:00
Johanna Amann
4204615997 SSL decryption: small style changes, a bit of documentation 2022-01-05 15:44:36 +00:00
Johanna Amann
d1e7134156 Merge remote-tracking branch 'origin/master' into topic/johanna/tls12-decryption 2022-01-05 10:27:55 +00:00
Tim Wojtulewicz
a6378531db Remove trailing whitespace from script files 2021-10-20 09:57:09 -07:00
Johanna Amann
fe4e06e8ca TLS decryption: remove payload from ssl_encrypted_data again.
There is no reason to make the payload available in the event - it is
still encrypted.
2021-10-19 17:36:48 +02:00
Johanna Amann
b8b6ac744e Merge remote-tracking branch 'origin/master' into topic/johanna/tls12-decryption 2021-10-13 10:49:29 +01:00
Johanna Amann
6a6961b474 Small indentation fixes in ssl-log-ext.zeek 2021-06-30 15:16:49 +01:00
Johanna Amann
e310734d7b Add new ssl-log-ext policy script
This policy script significantly extends the details that are logged
about SSL/TLS handshakes.

I am a bit tempted to just make this part of the default log - but it
does add a bunch logging overhead for each connection.
2021-06-29 09:45:25 +01:00
Johanna Amann
279a060fae Deprecate extract-certs-pem.zeek and add log-certs-base64.zeek
Extract-certs-pem writes pem files to a dedicated file; since it does
not really work in cluster-environments it was never super helpful.

This commit deprecates this file and, instead, adds
log-certs-base64.zeek, which adds the base64-encoded certificate (which
is basically equivalent with a PEM) to the log-file. Since, nowadays,
the log-files are deduplicates this should not add a huge overhead.
2021-06-29 09:45:18 +01:00
Johanna Amann
311e113ff6 Deprecate ICSI SSL notary script.
The ICSI notary is pretty much inactive. Furthermore - this approach
does no longer make much sense at this point of time - performing, e.g.,
signed certificate timestamp validation is much more worthwhile.
2021-06-29 09:34:29 +01:00
Johanna Amann
b02f22a667 Change SSL and X.509 logging format
This commit changes the SSL and X.509 logging formats to something that,
hopefully, slowly approaches what they will look like in the future.

X.509 log is not yet deduplicated; this will come in the future.

This commit introduces two new options, which determine if certificate
issuers and subjects are still logged in ssl.log. The default is to have
the host subject/issuer logged, but to remove client-certificate
information. Client-certificates are not a typically used feature
nowadays.
2021-06-29 09:26:43 +01:00
Florian Wilkens
febc69d320 analyzer/ssl: cleanup SSL_Analyzer
- make TLS12 PRF a member function of the analyzer
- use std::string in PRF
- use StringValPtr instead of zeek::StringVal
- replace malloc/free with C++ style allocations
2021-05-11 19:12:16 +02:00
Florian Wilkens
c1c0cb6f3c analyzer/ssl: Formatting, printf -> DBG_LOG, namespacing 2021-05-05 15:58:15 +02:00
Florian Wilkens
f73935aa45 ssl/decryption.zeek: cleanup 2021-05-05 15:41:36 +02:00
Florian Wilkens
2d950ffde9 ssl: rudimentary decryption for TLS 1.2
Several limitations still apply:
- TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 is the only supported cipher suite
- Some tests are broken due to a failing assertion regarding bytestring
- No newly written tests for decryption (the patch was tested extensively for our paper)
- Several small open technical questions marked with FIXME
- Architecture in the ssl module might not be optimal
2021-04-19 17:38:02 +02:00
Johanna Amann
c23e3ca105 Heartbleed: fix substraction order.
The larger number was substracted from the smaller one leading to an
integer overflow. However, no information was lost due to everything
also being present in the notice message.

Fixes GH-1454
2021-03-25 13:41:12 +00:00
Vern Paxson
c991c54690 &is_set => &is_assigned 2021-02-04 12:18:46 -08:00
Vern Paxson
0d77b474e6 adding &is_set attributes to base scripts so -u output isn't cluttered 2021-01-23 10:55:27 -08:00
Christian Kreibich
7bd8351719 Migrate existing use of filter predicates to policy hooks 2020-09-30 12:45:20 -07:00
Christian Kreibich
1bd658da8f Support for log filter policy hooks
This adds a "policy" hook into the logging framework's streams and
filters to replace the existing log filter predicates. The hook
signature is as follows:

    hook(rec: any, id: Log::ID, filter: Log::Filter);

The logging manager invokes hooks on each log record. Hooks can veto
log records via a break, and modify them if necessary. Log filters
inherit the stream-level hook, but can override or remove the hook as
needed.

The distribution's existing log streams now come with pre-defined
hooks that users can add handlers to. Their name is standardized as
"log_policy" by convention, with additional suffixes when a module
provides multiple streams. The following adds a handler to the Conn
module's default log policy hook:

    hook Conn::log_policy(rec: Conn::Info, id: Log::ID, filter: Log::Filter)
            {
            if ( some_veto_reason(rec) )
                break;
            }

By default, this handler will get invoked for any log filter
associated with the Conn::LOG stream.

The existing predicates are deprecated for removal in 4.1 but continue
to work.
2020-09-30 12:32:45 -07:00
Johanna Amann
7c012f9b91 Merge branch 'master' of https://github.com/mmguero-dev/zeek
* 'master' of https://github.com/mmguero-dev/zeek:
  check for the existance of f?$conns in file_sniff event in policy/protocols/ssl/log-hostcerts-only.zeek
2020-04-16 12:59:12 -07:00
SG
42bf41aca1 check for the existance of f?$conns in file_sniff event in policy/protocols/ssl/log-hostcerts-only.zeek
In using the corelight/bro-xor-exe-plugin (https://github.com/corelight/bro-xor-exe-plugin) I noticed this error when running the PCAP trace file in its tests directory:

1428602842.525435 expression error in /opt/zeek/share/zeek/policy/protocols/ssl/log-hostcerts-only.zeek, line 44: field value missing (X509::f$conns)

Examining log-hostcerts-only.zeek, I saw that although f$conns is being checked for length, it's not being checked to see if it exists first.

This commit changes "if ( |f$conns| != 1 )" to "if (( ! f?$conns ) || ( |f$conns| != 1 ))" so that the script returns if there is no f$conns field.

In my local testing, this seems to fix the error. My testing was being done with v3.0.5, but I think this patch can be applied to both the 3.0.x and 3.1.x branches.
2020-04-16 10:19:59 -06:00
Jon Siwek
1ce0fcce49 GH-387: update Broker topic names to use "zeek/" prefix 2019-05-29 15:56:37 -07:00
Daniel Thayer
be182aac83 More bro-to-zeek renaming in scripts and other files 2019-05-16 02:36:41 -05:00
Jon Siwek
aebcb1415d GH-234: rename Broxygen to Zeexygen along with roles/directives
* All "Broxygen" usages have been replaced in
  code, documentation, filenames, etc.

* Sphinx roles/directives like ":bro:see" are now ":zeek:see"

* The "--broxygen" command-line option is now "--zeexygen"
2019-04-22 19:45:50 -07:00
Jon Siwek
a994be9eeb Merge remote-tracking branch 'origin/topic/seth/zeek_init'
* origin/topic/seth/zeek_init:
  Some more testing fixes.
  Update docs and tests for bro_(init|done) -> zeek_(init|done)
  Implement the zeek_init handler.
2019-04-19 11:24:29 -07:00