zeek/doc/user-manual/scripting.rst

=========
Scripting
=========

.. toctree::
    :maxdepth: 2
    :numbered:

    Understanding Bro Scripts
    Event Handlers
    Data types
    A complete script
    Guidelines for writing scripts

Understanding Bro Scripts
=========================

Bro includes an event queue driven scripting language that provides the primary means for an organization to extend and customize Bro's functionality.  An overwhelming amount of the output generated by Bro is, in fact, generated by bro scripts.  It's almost easier to consider Bro to be an entity behind-the-scenes processing connections and generating events while Bro's scripting language is the medium through which we mere mortals can achieve communication.  Bro scripts effectively notify Bro that should there be an event of a type we define, then let us have the information about the connection so we can perform some function on it.  For example, the ssl.log file is generated by a bro script that walks the entire certificate chain and issues notifications if any of the steps along the cert chain are invalid.  This entire process is setup by telling Bro that should it see a server or client issue an SSL HELLO message, we want to know about the information about that connection.

It's often easier to understand Bro's scripting language by looking at a complete script and breaking it down into its identifiable components.  In this example, we'll take a look at how Bro queries the Team Cymru malware hash registry for detected downloads via HTTP. Part of the Team Cymru Malware Hash registry includes the ability to do a host lookup on a domain with the format MALWARE_HASH.malware.hash.cymru.com where MALWARE_HASH is the md5 or sha1 hash of a file.  The important aspect to understand is Bro already generates hashes for files it can parse from HTTP streams, but the script detect-MHR.bro is responsible for generating the appropriate DNS lookup and parsing the response.


.. literalinclude:: ../../scripts/policy/protocols/http/detect-MHR.bro
   :language: bro
   :linenos:


Visually, there are three distinct sections of the script.  A base level with no identation followed by an indented and formatted section explaining custom the variables being set (export) and another indented and formatted section describing the instructions for a specific event (event log_http).  Don't get discouraged if you don't understand every section of the script; we'll cover the basics of the script and much more in following sections.

.. literalinclude:: ../../scripts/policy/protocols/http/detect-MHR.bro
   :language: bro
   :linenos:
   :lines: 7-11

Lines 7 and 8 of the script process the __load__.bro script in the respective directories being loaded.  In a full production deployment of Bro it's likely that these files would already be loaded and their contents made available to the script, but including them explicitly ensures that Bro can be run in modes such as "bare" and still have scripts that reliably work.

.. literalinclude:: ../../scripts/policy/protocols/http/detect-MHR.bro
   :language: bro
   :linenos:
   :lines: 12-18

The export section redefines an enumerable constant that describes the type of notice we will generate with the logging framework.  The notice type listed allows for the use of the NOTICE() function to generate notices of type Malware_Hash_Registry_Match as done in the next section.

.. literalinclude:: ../../scripts/policy/protocols/http/detect-MHR.bro
   :language: bro
   :linenos:
   :lines: 20-37

The workhorse of the script is contained in the event handler for log_http.  The log_http event is defined as an event-hook in the base/protocols/http/main.bro script and allows scripts to handle connection as it is being passed to the logging framework. The event handler is passed an HTTP::Info data structure which will be referred to as "rec" in body of the event handler.

An if statement is used to check for the existence of a datastructure named "md5" nested within the rec data structure.  Bro uses the "$" as a deference operator and as such, and it is employed in this script to check if rec$md5 is present by including the "?" operator within the path.  If the rec data structure includes a nested datastructure named "md5", the statement is processed as true and a local variable named "hash_domain" is provisioned and given a format string based on the contents of rec$md5.  The script then provisions another local variable named "addrs" and fills it with the output of the lookup_hostname() function which will perform a DNS lookup on the arguments passed to it.

DNS lookups against the Team Cymru Malware Hash Registry will return a '127.0.0.2' if the hash being queried is found in the hash registry.  If the response includes 127.0.0.2, the script proceeds to build two new local variables to be used in the notice and then issues the NOTICE().

In approximately 15 lines of actual code, Bro provides an amazing utility that would be incredibly difficult to implement and deploy with other products.  In truth, claiming that Bro does this in 15 lines is a misdirection; There is a truly massive number of things going on behind-the-scenes in Bro, but it is the inclusion of the scripting language that gives analysts access to those underlying layers in a succinct and well defined manner.  

Event Handlers
==============

Bro's scripting language is event driven which is a gear change from the majority of scripting languages with which most users will have previous experience.  Scripting in Bro depends on handling the events generated by Bro as it processes network traffic, altering the state of data structures through those events, and making decisions on the information provided.  This approach to scripting can often cause confusion to users who come to Bro from a procedural or functional language, but once the initial shock wears off it becomes more clear with each exposure.  Gaining familiarity with the specific events generated by Bro is a big step towards building a mindset for working with Bro scripts.  

Most, if not all, of Bro's events are defined in the built-in-function files or .bif files.  Whether starting a script from scratch or reading and maintaining someone else's script, having the built-in-function files available is an excellent resource to have on hand.  Before release version 2.0 the Bro developers put significant effort into organization and documentation of every event.  This effort resulted in built-in-function files organized such that each entry contains a descriptive event name, the arguments passed to the event, and a concise explanation of the functions use.

.. literalinclude:: ../../build/src/event.bif.bro
   :language: bro
   :linenos:
   :lines: 4124-4149

Above is a segment of event.bif.bro showing the documentation for the event dns_request().  It's organized such that the documentation, commentary, and list of arguments preceed the actual event definition used by Bro.  As Bro detects DNS requests being issued by an originator, it issues this event and any number of scripts then have access to the data Bro passes along with the event.  In this example, Bro passes not only the message, the query, query type and query class for the DNS request, but also a then record used for the connection itself.

The Connection Record Data Type
===============================

A majority of the events in Bro's event.bif.bro file are passed the connection record data type, in effect, making backbone of many scripting solutions.  The connection record itself, as we will see in a moment, is a mass of nested data types used to track state on a connection through its lifetime.  

   


Not only are events defined in the built-in-functions in Bro, but as seen in the last section, users can define events as well.

As events are handled, the script is given a chance to save or alter state.