zeek/doc/frameworks/tls-decryption.rst

.. _framework-tls-decryption:

==============
TLS Decryption
==============

.. rst-class:: opening


   Zeek has limited support for decrypting TLS connections, if the necessary key material is
   available. If decryption is possible, Zeek can forward the decrypted data to other analyzers - like
   the HTTP analyzer.

   Note that this support is currently limited to a single version of TLS and a single cipher suite.
   Zeek can currently only decrypt TLS 1.2 connections that use the TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
   cipher. Any other TLS version or cipher will not be decrypted. We do not currently plan to extend
   this support to other versions of TLS, or to other ciphersuites.

Capturing and decrypting a trace file
=====================================

The most common use-case for TLS decryption is to capture a trace file together with the necessary key material
to allow Zeek to decrypt it. In principle, it is possible to allow Zeek to decrypt TLS connections in live traffic.
However, this kind of setup is much more complex and will require the user to set up a way to transport the
key material to Zeek in real-time. We will talk a bit about the possibility of this below.

Capturing a trace file with keys
--------------------------------

To be able to decrypt the TLS connections contained in a trace file, we need access to the symmetric keys that
were used to encrypt the connection. Specifically, for our supported version of TLS, we need the pre-master secret
for the TLS connection.

Firefox and Chrome allow users to capture the key material of all TLS connections that they perform by setting
the ``SSLKEYLOGFILE`` environment variable. For example, on Mac OS, you can instruct Firefox to record the TLS
key material by starting it in the following way:

.. code-block:: console

   export SSLKEYLOGFILE=$HOME/keylogfile.txt
   open -a firefox

After running Firefox like this, and accessing some web pages, you should end up with the file ``keylogfile.txt``
in your home directory that contains lines like this::

   # SSL/TLS secrets log file, generated by NSS
   CLIENT_RANDOM 47d1becb619e0851ee363c2cf37187228227ca4e680f9a7c0bd15069aa7a5970 ad03ceda4890fa581e989f5e3862023e2a4e3e8ad81325238d908066e1d35cc875979e34c08e6fdfd9d8c6f356e385c1
   CLIENT_RANDOM 2095006fcb3f93d255cbb6562587f0dd010212fdee9d233aff64e6ed36cd5c45 0d36faaa2eadbda2a8095f951de1cbac46b81b008fbf391d91951b3485476bab73288a1e17cd0ce80e0fc0401dbe9e3f
   CLIENT_RANDOM 8f58b32bf97e7d3856e2fccbbe80798ec2e3f515251082ad63bbc7c231d8bee0 9a7cf946a04718a19f4d20c3f80c1cf8c823c3e2b1c337ef64322d751b410543315f6ecf7dbf45ec9be194a3cc7c1a0f

These log lines contain the pre-master secrets for the connections that your browser established. The secrets
are indexed with the client random of the connections. This allows applications (like Zeek) to identify which
secret to use to decrypt a connection.

If you capture this key log file together with a trace-file, you will be able to decrypt the sessions using Zeek
(assuming they use a supported TLS version and ciphersuite).

Decrypting a trace file
-----------------------

The next step is to convert the keylogfile into a format that can be ingested by the Zeek. This bash-script
will perform the conversion:

.. code-block:: bash

   #!/usr/bin/env bash

   if [ $# -ne 1 ]; then
   	echo "Script expects one argument (key log filename)" >/dev/stderr
   	exit -1
   fi

   FILE=$1

   if [ ! -f ${FILE} ]; then
   	echo "${FILE} does not exist or is not readable" >/dev/stderr
   	exit -1
   fi

   echo "#fields	client_random	secret"
   grep CLIENT_RANDOM ${FILE} | sed 's/^CLIENT_RANDOM ........\(.*\) \(.*\)$/\1	\2/' | sed 's/[A-Za-z0-9][A-Za-z0-9]/\\x&/g'

Note that the script just converts the keylog file in a standard Zeek tsv-file. Furthermore, it removes
the first 16 characters of the CLIENT_RANDOM; this is needed due to a design-choice of Zeek that makes accessing
the first 8 bytes (equivalent to 16 hex-characters) of the client random inconvenient - thus these bytes are not
used for matching.

If you run the bash script on the ``keylogfile.txt`` you created earlier, you will get a Zeek tsv-file.

.. code-block:: console

   ./convert-keylog.sh ~/keylogfile.txt > ~/keylogfile.log

   cat ~/keylogfile.log
   #fields	client_random	secret
   \x0e\x78\x2d\x35\x63\x95\x5d\x8a\x30\xa9\xcf\xb6\x4f\x47\xf3\x96\x34\x8a\x1e\x79\x1a\xa2\x32\x55\xe2\x2f\xc5\x7a	\x34\x4f\x12\x65\xbf\x43\x40\xb3\x61\x6b\xa0\x16\x5d\x2b\x4d\xb9\xb1\xe8\x4a\x3d\xa2\x42\x0e\x38\xab\x01\x50\x62\x84\xcc\x34\xcd\xe0\x34\x10\xfe\x1a\x02\x30\x49\x74\x6c\x46\x43\xa7\x0c\x67\x0d
   \x24\x8c\x7e\x24\xee\xfb\x13\xcd\xee\xde\xb1\xf4\xb6\xd6\xd5\xee\x67\x8d\xd3\xff\xc7\xe7\x39\x23\x18\x3f\x99\xb4	\xe7\xed\x24\x26\x0d\x25\xd9\xfd\xf5\x0f\xc0\xf4\x56\x51\x0e\x4e\xec\x7f\x58\x9c\xaf\x39\x25\x14\x16\xa6\x71\xdd\xea\xfe\xe9\xc0\x93\xbe\x89\x4c\xab\xcc\xff\xb2\xf0\x9a\xea\x98\xf5\xb2\x53\x1e
   \x57\xd7\xc7\x7a\x2d\x5e\x35\x29\x2c\xd7\xe7\x94\xee\xf8\x6f\x31\x45\xf6\xbe\x25\x08\xed\x1d\x92\xd2\x0b\x9b\x04	\xc1\x93\x17\x93\xd9\x7d\xd2\x98\xb3\xe0\xdb\x2c\x5d\xbe\x71\x31\xa7\x9a\xf5\x91\xf9\x87\x90\xee\xb7\x79\x9f\x6b\xb4\x1f\x47\xa7\x69\x62\x4b\xa3\x99\x0c\xa9\x43\xf9\xea\x3b\x4d\x5f\x2f\xfe\xfb

Now we can run Zeek on the trace-file that we recorded. We need a small additional script for this, which
stops processing while the TLS keylog file is loaded. It also loads the required policy script.

.. literalinclude:: tls_decryption-1-suspend-processing.zeek
   :caption:
   :language: zeek
   :linenos:
   :tab-width: 4

.. code-block:: console

   $ export ZEEK_TLS_KEYLOG_FILE=~/keylogfile.log
   $ zeek -C -r tls/tls-1.2-stream-keylog.pcap tls_decryption-1-suspend-processing.zeek

   $ cat conn.log
   #separator \x09
   #set_separator	,
   #empty_field	(empty)
   #unset_field	-
   #path	conn
   #open	2022-03-01-16-57-26
   #fields	ts	uid	id.orig_h	id.orig_p	id.resp_h	id.resp_p	proto	service	duration	orig_bytes	resp_bytes	conn_state	local_orig	local_resp	missed_bytes	history	orig_pkts	orig_ip_bytes	resp_pkts	resp_ip_bytes	tunnel_parents
   #types	time	string	addr	port	addr	port	enum	string	interval	count	count	string	bool	bool	count	string	count	count	count	count	set[string]
   1646150638.631834	CTy5Us4OUaTOcyrPvc	192.168.20.12	60679	193.99.144.85	443	tcp	http,ssl	7.246461	10853	151695	SF	-	-	0	ShADadFf	98	15961	139	158931	-
   #close	2022-03-01-16-57-26

   $ cat http.log
   #separator \x09
   #set_separator	,
   #empty_field	(empty)
   #unset_field	-
   #path	http
   #open	2022-03-01-16-57-25
   #fields	ts	uid	id.orig_h	id.orig_p	id.resp_h	id.resp_p	trans_depth	method	host	uri	referrer	version	user_agent	origin	request_body_len	response_body_len	status_code	status_msg	info_code	info_msg	tags	username	password	proxied	orig_fuids	orig_filenames	orig_mime_types	resp_fuids	resp_filenames	resp_mime_types
   #types	time	string	addr	port	addr	port	count	string	string	string	string	string	string	string	count	count	count	string	count	string	set[enum]	string	string	set[string]	vector[string]	vector[string]	vector[string]	vector[string]	vector[string]	vector[string]
   1646150638.735969	CTy5Us4OUaTOcyrPvc	192.168.20.12	60679	193.99.144.85	443	1	GET	www.heise.de	/assets/akwa/v24/js/akwa.js?.ltc.c61e84978682308f631c	https://www.heise.de/	1.1	Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:97.0) Gecko/20100101 Firefox/97.0	-	0	375340	200	OK	-	-	(empty)	-	-	-	-	-	-	FSJiWr34wfIujxxtm3	-	text/plain
   1646150638.944774	CTy5Us4OUaTOcyrPvc	192.168.20.12	60679	193.99.144.85	443	2	GET	www.heise.de	/assets/heise/images/mit_technology_review_singleline.b768.ltc.svg	https://www.heise.de/	1.1	Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:97.0) Gecko/20100101 Firefox/97.0	-	0	3430	200	OK	-	-	(empty)	-	-	-	-	-	-	FgivhC1pvnYeQS4u18	-	text/plain
   1646150638.976118	CTy5Us4OUaTOcyrPvc	192.168.20.12	60679	193.99.144.85	443	3	GET	www.heise.de	/assets/heise/hobell/css/hobell.css?.ltc.3746e7e49abafa23b5fb	https://www.heise.de/	1.1	Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:97.0) Gecko/20100101 Firefox/97.0	-	0	85280	200	OK	-	-	(empty)	-	-	-	-	-	-	FyvBkl2nwRXf0hkDO1	-	text/plain
   ...

Now :file:`conn.log` shows that the HTTP as well as the SSL analyzers were attached. :file:`http.log` shows the
information from the decrypted HTTP session.

If you try this yourself note that today a lot of encrypted Internet traffic uses HTTP/2. Zeek currently does not
ship with an HTTP/2 parser by default. If you capture your own traffic make sure that your browser uses
HTTP/1. Alternatively, you can add an HTTP/2 analyzer to Zeek, e.g. using a package.

Decrypting live traffic
=======================

In principle, it is possible to decrypt live traffic using this approach. When you want to do this, you have to supply the secrets
to Zeek as the connections are happening. Note that there are timing constraints here - the secrets should arrive at the Zeek instance
that will decrypt the traffic before encrypted application data is exchanged.

The :doc:`/scripts/policy/protocols/ssl/decryption.zeek` policy script sets up a two events for this purpose. You can send key material
to the Zeek worker in question via Broker, using the ``/zeek/tls/decryption`` topic. The two events used for this are
:zeek:see:`SSL::add_keys` and :zeek:see:`SSL::add_secret`.

TLS Decryption API
==================

If the policy script does not suit your use-case, you can use the TLS decryption API directly to decrypt a connection. You can use either the
:zeek:see:`set_secret` or the :zeek:see:`set_keys` functions to provide the decryption keys for an ongoing SSL connection.

Note that you will have to make sure to set :zeek:see:`SSL::disable_analyzer_after_detection` to false if you use this functionality directly.