:tocdepth: 3

base/protocols/http/main.zeek
=============================
.. zeek:namespace:: HTTP

Implements base functionality for HTTP analysis.  The logging model is
to log request/response pairs and all relevant metadata together in
a single record.

:Namespace: HTTP
:Imports: :doc:`base/frameworks/tunnels </scripts/base/frameworks/tunnels/index>`, :doc:`base/protocols/conn/removal-hooks.zeek </scripts/base/protocols/conn/removal-hooks.zeek>`, :doc:`base/utils/files.zeek </scripts/base/utils/files.zeek>`, :doc:`base/utils/numbers.zeek </scripts/base/utils/numbers.zeek>`

Summary
~~~~~~~
Runtime Options
###############
================================================================================ ====================================================================
:zeek:id:`HTTP::default_capture_password`: :zeek:type:`bool` :zeek:attr:`&redef` This setting changes if passwords used in Basic-Auth are captured or
                                                                                 not.
:zeek:id:`HTTP::http_methods`: :zeek:type:`set` :zeek:attr:`&redef`              A list of HTTP methods.
:zeek:id:`HTTP::max_pending_requests`: :zeek:type:`count` :zeek:attr:`&redef`    Only allow that many pending requests on a single connection.
:zeek:id:`HTTP::proxy_headers`: :zeek:type:`set` :zeek:attr:`&redef`             A list of HTTP headers typically used to indicate proxied requests.
================================================================================ ====================================================================

Redefinable Options
###################
======================================================================================= =======================================================================
:zeek:id:`HTTP::default_max_field_string_bytes`: :zeek:type:`count` :zeek:attr:`&redef` The maximum number of bytes that a single string field can contain when
                                                                                        logging.
======================================================================================= =======================================================================

Types
#####
============================================= ===================================================================
:zeek:type:`HTTP::Info`: :zeek:type:`record`  The record type which contains the fields of the HTTP log.
:zeek:type:`HTTP::State`: :zeek:type:`record` Structure to maintain state for an HTTP connection with multiple
                                              requests and responses.
:zeek:type:`HTTP::Tags`: :zeek:type:`enum`    Indicate a type of attack or compromise in the record to be logged.
============================================= ===================================================================

Redefinitions
#############
==================================================================== =============================================================
:zeek:type:`Log::ID`: :zeek:type:`enum`                              
                                                                     
                                                                     * :zeek:enum:`HTTP::LOG`
:zeek:type:`connection`: :zeek:type:`record`                         
                                                                     
                                                                     :New Fields: :zeek:type:`connection`
                                                                     
                                                                       http: :zeek:type:`HTTP::Info` :zeek:attr:`&optional`
                                                                     
                                                                       http_state: :zeek:type:`HTTP::State` :zeek:attr:`&optional`
:zeek:id:`likely_server_ports`: :zeek:type:`set` :zeek:attr:`&redef` 
==================================================================== =============================================================

Events
######
============================================= ====================================================================
:zeek:id:`HTTP::log_http`: :zeek:type:`event` Event that can be handled to access the HTTP record as it is sent on
                                              to the logging framework.
============================================= ====================================================================

Hooks
#####
============================================================== =======================
:zeek:id:`HTTP::finalize_http`: :zeek:type:`Conn::RemovalHook` HTTP finalization hook.
:zeek:id:`HTTP::log_policy`: :zeek:type:`Log::PolicyHook`      
============================================================== =======================


Detailed Interface
~~~~~~~~~~~~~~~~~~
Runtime Options
###############
.. zeek:id:: HTTP::default_capture_password
   :source-code: base/protocols/http/main.zeek 25 25

   :Type: :zeek:type:`bool`
   :Attributes: :zeek:attr:`&redef`
   :Default: ``F``

   This setting changes if passwords used in Basic-Auth are captured or
   not.

.. zeek:id:: HTTP::http_methods
   :source-code: base/protocols/http/main.zeek 120 120

   :Type: :zeek:type:`set` [:zeek:type:`string`]
   :Attributes: :zeek:attr:`&redef`
   :Default:

      ::

         {
            "POST",
            "PUT",
            "CONNECT",
            "BMOVE",
            "SEARCH",
            "TRACE",
            "LOCK",
            "PROPPATCH",
            "HEAD",
            "OPTIONS",
            "POLL",
            "REPORT",
            "SUBSCRIBE",
            "MOVE",
            "GET",
            "UNLOCK",
            "DELETE",
            "COPY",
            "MKCOL",
            "PROPFIND"
         }


   A list of HTTP methods. Other methods will generate a weird. Note
   that the HTTP analyzer will only accept methods consisting solely
   of letters ``[A-Za-z]``.

.. zeek:id:: HTTP::max_pending_requests
   :source-code: base/protocols/http/main.zeek 141 141

   :Type: :zeek:type:`count`
   :Attributes: :zeek:attr:`&redef`
   :Default: ``100``

   Only allow that many pending requests on a single connection.
   If this number is exceeded, all pending requests are flushed
   out and request/response tracking reset to prevent unbounded
   state growth.

.. zeek:id:: HTTP::proxy_headers
   :source-code: base/protocols/http/main.zeek 107 107

   :Type: :zeek:type:`set` [:zeek:type:`string`]
   :Attributes: :zeek:attr:`&redef`
   :Default:

      ::

         {
            "CLIENT-IP",
            "X-FORWARDED-FROM",
            "VIA",
            "XROXY-CONNECTION",
            "PROXY-CONNECTION",
            "X-FORWARDED-FOR",
            "FORWARDED"
         }


   A list of HTTP headers typically used to indicate proxied requests.

Redefinable Options
###################
.. zeek:id:: HTTP::default_max_field_string_bytes
   :source-code: base/protocols/http/main.zeek 149 149

   :Type: :zeek:type:`count`
   :Attributes: :zeek:attr:`&redef`
   :Default: ``0``

   The maximum number of bytes that a single string field can contain when
   logging. If a string reaches this limit, the log output for the field will be
   truncated. Setting this to zero disables the limiting. HTTP has no maximum
   length for various fields such as the URI, so this is set to zero by default.
   
   .. zeek:see:: Log::default_max_field_string_bytes

Types
#####
.. zeek:type:: HTTP::Info
   :source-code: base/protocols/http/main.zeek 28 89

   :Type: :zeek:type:`record`


   .. zeek:field:: ts :zeek:type:`time` :zeek:attr:`&log`

      Timestamp for when the request happened.


   .. zeek:field:: uid :zeek:type:`string` :zeek:attr:`&log`

      Unique ID for the connection.


   .. zeek:field:: id :zeek:type:`conn_id` :zeek:attr:`&log`

      The connection's 4-tuple of endpoint addresses/ports.


   .. zeek:field:: trans_depth :zeek:type:`count` :zeek:attr:`&log`

      Represents the pipelined depth into the connection of this
      request/response transaction.


   .. zeek:field:: method :zeek:type:`string` :zeek:attr:`&log` :zeek:attr:`&optional`

      Verb used in the HTTP request (GET, POST, HEAD, etc.).


   .. zeek:field:: host :zeek:type:`string` :zeek:attr:`&log` :zeek:attr:`&optional`

      Value of the HOST header.


   .. zeek:field:: uri :zeek:type:`string` :zeek:attr:`&log` :zeek:attr:`&optional`

      URI used in the request.


   .. zeek:field:: referrer :zeek:type:`string` :zeek:attr:`&log` :zeek:attr:`&optional`

      Value of the "referer" header.  The comment is deliberately
      misspelled like the standard declares, but the name used here
      is "referrer", spelled correctly.


   .. zeek:field:: version :zeek:type:`string` :zeek:attr:`&log` :zeek:attr:`&optional`

      Value of the version portion of the reply. If you require
      message-level detail, consider the :zeek:see:`http_request` and
      :zeek:see:`http_reply` events, which report each message's
      version string.


   .. zeek:field:: user_agent :zeek:type:`string` :zeek:attr:`&log` :zeek:attr:`&optional`

      Value of the User-Agent header from the client.


   .. zeek:field:: origin :zeek:type:`string` :zeek:attr:`&log` :zeek:attr:`&optional`

      Value of the Origin header from the client.


   .. zeek:field:: request_body_len :zeek:type:`count` :zeek:attr:`&log` :zeek:attr:`&default` = ``0`` :zeek:attr:`&optional`

      Actual uncompressed content size of the data transferred from
      the client.


   .. zeek:field:: response_body_len :zeek:type:`count` :zeek:attr:`&log` :zeek:attr:`&default` = ``0`` :zeek:attr:`&optional`

      Actual uncompressed content size of the data transferred from
      the server.


   .. zeek:field:: status_code :zeek:type:`count` :zeek:attr:`&log` :zeek:attr:`&optional`

      Status code returned by the server.


   .. zeek:field:: status_msg :zeek:type:`string` :zeek:attr:`&log` :zeek:attr:`&optional`

      Status message returned by the server.


   .. zeek:field:: info_code :zeek:type:`count` :zeek:attr:`&log` :zeek:attr:`&optional`

      Last seen 1xx informational reply code returned by the server.


   .. zeek:field:: info_msg :zeek:type:`string` :zeek:attr:`&log` :zeek:attr:`&optional`

      Last seen 1xx informational reply message returned by the server.


   .. zeek:field:: tags :zeek:type:`set` [:zeek:type:`HTTP::Tags`] :zeek:attr:`&log`

      A set of indicators of various attributes discovered and
      related to a particular request/response pair.


   .. zeek:field:: username :zeek:type:`string` :zeek:attr:`&log` :zeek:attr:`&optional`

      Username if basic-auth is performed for the request.


   .. zeek:field:: password :zeek:type:`string` :zeek:attr:`&log` :zeek:attr:`&optional`

      Password if basic-auth is performed for the request.


   .. zeek:field:: capture_password :zeek:type:`bool` :zeek:attr:`&default` = :zeek:see:`HTTP::default_capture_password` :zeek:attr:`&optional`

      Determines if the password will be captured for this request.


   .. zeek:field:: proxied :zeek:type:`set` [:zeek:type:`string`] :zeek:attr:`&log` :zeek:attr:`&optional`

      All of the headers that may indicate if the request was proxied.


   .. zeek:field:: range_request :zeek:type:`bool` :zeek:attr:`&default` = ``F`` :zeek:attr:`&optional`

      Indicates if this request can assume 206 partial content in
      response.


   .. zeek:field:: orig_fuids :zeek:type:`vector` of :zeek:type:`string` :zeek:attr:`&log` :zeek:attr:`&optional`

      (present if :doc:`/scripts/base/protocols/http/entities.zeek` is loaded)

      An ordered vector of file unique IDs.
      Limited to :zeek:see:`HTTP::max_files_orig` entries.


   .. zeek:field:: orig_filenames :zeek:type:`vector` of :zeek:type:`string` :zeek:attr:`&log` :zeek:attr:`&optional`

      (present if :doc:`/scripts/base/protocols/http/entities.zeek` is loaded)

      An ordered vector of filenames from the client.
      Limited to :zeek:see:`HTTP::max_files_orig` entries.


   .. zeek:field:: orig_mime_types :zeek:type:`vector` of :zeek:type:`string` :zeek:attr:`&log` :zeek:attr:`&optional`

      (present if :doc:`/scripts/base/protocols/http/entities.zeek` is loaded)

      An ordered vector of mime types.
      Limited to :zeek:see:`HTTP::max_files_orig` entries.


   .. zeek:field:: resp_fuids :zeek:type:`vector` of :zeek:type:`string` :zeek:attr:`&log` :zeek:attr:`&optional`

      (present if :doc:`/scripts/base/protocols/http/entities.zeek` is loaded)

      An ordered vector of file unique IDs.
      Limited to :zeek:see:`HTTP::max_files_resp` entries.


   .. zeek:field:: resp_filenames :zeek:type:`vector` of :zeek:type:`string` :zeek:attr:`&log` :zeek:attr:`&optional`

      (present if :doc:`/scripts/base/protocols/http/entities.zeek` is loaded)

      An ordered vector of filenames from the server.
      Limited to :zeek:see:`HTTP::max_files_resp` entries.


   .. zeek:field:: resp_mime_types :zeek:type:`vector` of :zeek:type:`string` :zeek:attr:`&log` :zeek:attr:`&optional`

      (present if :doc:`/scripts/base/protocols/http/entities.zeek` is loaded)

      An ordered vector of mime types.
      Limited to :zeek:see:`HTTP::max_files_resp` entries.


   .. zeek:field:: current_entity :zeek:type:`HTTP::Entity` :zeek:attr:`&optional`

      (present if :doc:`/scripts/base/protocols/http/entities.zeek` is loaded)

      The current entity.


   .. zeek:field:: orig_mime_depth :zeek:type:`count` :zeek:attr:`&default` = ``0`` :zeek:attr:`&optional`

      (present if :doc:`/scripts/base/protocols/http/entities.zeek` is loaded)

      Current number of MIME entities in the HTTP request message
      body.


   .. zeek:field:: resp_mime_depth :zeek:type:`count` :zeek:attr:`&default` = ``0`` :zeek:attr:`&optional`

      (present if :doc:`/scripts/base/protocols/http/entities.zeek` is loaded)

      Current number of MIME entities in the HTTP response message
      body.


   .. zeek:field:: client_header_names :zeek:type:`vector` of :zeek:type:`string` :zeek:attr:`&log` :zeek:attr:`&optional`

      (present if :doc:`/scripts/policy/protocols/http/header-names.zeek` is loaded)

      The vector of HTTP header names sent by the client.  No
      header values are included here, just the header names.


   .. zeek:field:: server_header_names :zeek:type:`vector` of :zeek:type:`string` :zeek:attr:`&log` :zeek:attr:`&optional`

      (present if :doc:`/scripts/policy/protocols/http/header-names.zeek` is loaded)

      The vector of HTTP header names sent by the server.  No
      header values are included here, just the header names.


   .. zeek:field:: omniture :zeek:type:`bool` :zeek:attr:`&default` = ``F`` :zeek:attr:`&optional`

      (present if :doc:`/scripts/policy/protocols/http/software-browser-plugins.zeek` is loaded)

      Indicates if the server is an omniture advertising server.


   .. zeek:field:: flash_version :zeek:type:`string` :zeek:attr:`&optional`

      (present if :doc:`/scripts/policy/protocols/http/software-browser-plugins.zeek` is loaded)

      The unparsed Flash version, if detected.


   .. zeek:field:: cookie_vars :zeek:type:`vector` of :zeek:type:`string` :zeek:attr:`&optional` :zeek:attr:`&log`

      (present if :doc:`/scripts/policy/protocols/http/var-extraction-cookies.zeek` is loaded)

      Variable names extracted from all cookies.


   .. zeek:field:: uri_vars :zeek:type:`vector` of :zeek:type:`string` :zeek:attr:`&optional` :zeek:attr:`&log`

      (present if :doc:`/scripts/policy/protocols/http/var-extraction-uri.zeek` is loaded)

      Variable names from the URI.


   The record type which contains the fields of the HTTP log.

.. zeek:type:: HTTP::State
   :source-code: base/protocols/http/main.zeek 93 104

   :Type: :zeek:type:`record`


   .. zeek:field:: pending :zeek:type:`table` [:zeek:type:`count`] of :zeek:type:`HTTP::Info`

      Pending requests.


   .. zeek:field:: current_request :zeek:type:`count` :zeek:attr:`&default` = ``0`` :zeek:attr:`&optional`

      Current request in the pending queue.


   .. zeek:field:: current_response :zeek:type:`count` :zeek:attr:`&default` = ``0`` :zeek:attr:`&optional`

      Current response in the pending queue.


   .. zeek:field:: trans_depth :zeek:type:`count` :zeek:attr:`&default` = ``0`` :zeek:attr:`&optional`

      Track the current deepest transaction.
      This is meant to cope with missing requests
      and responses.


   Structure to maintain state for an HTTP connection with multiple
   requests and responses.

.. zeek:type:: HTTP::Tags
   :source-code: base/protocols/http/main.zeek 18 22

   :Type: :zeek:type:`enum`

      .. zeek:enum:: HTTP::EMPTY HTTP::Tags

         Placeholder.

      .. zeek:enum:: HTTP::URI_SQLI HTTP::Tags

         (present if :doc:`/scripts/policy/protocols/http/detect-sql-injection.zeek` is loaded)


         Indicator of a URI based SQL injection attack.

   Indicate a type of attack or compromise in the record to be logged.

Events
######
.. zeek:id:: HTTP::log_http
   :source-code: base/protocols/http/main.zeek 132 132

   :Type: :zeek:type:`event` (rec: :zeek:type:`HTTP::Info`)

   Event that can be handled to access the HTTP record as it is sent on
   to the logging framework.

Hooks
#####
.. zeek:id:: HTTP::finalize_http
   :source-code: base/protocols/http/main.zeek 393 405

   :Type: :zeek:type:`Conn::RemovalHook`

   HTTP finalization hook.  Remaining HTTP info may get logged when it's called.

.. zeek:id:: HTTP::log_policy
   :source-code: base/protocols/http/main.zeek 15 15

   :Type: :zeek:type:`Log::PolicyHook`



