diff --git a/doc/user-manual/scripting.rst b/doc/user-manual/scripting.rst index a8ec45e53d..bd57d2f06f 100644 --- a/doc/user-manual/scripting.rst +++ b/doc/user-manual/scripting.rst @@ -8,10 +8,11 @@ Scripting :numbered: Understanding Bro Scripts - Event Handlers - Data types - A complete script - Guidelines for writing scripts + The Event Queue and Event Handlers + The Connection Record Datatype + Data Types + Variables and Scope + Branching and Iteration Understanding Bro Scripts ========================= @@ -21,27 +22,27 @@ Bro includes an event queue driven scripting language that provides the primary It's often easier to understand Bro's scripting language by looking at a complete script and breaking it down into its identifiable components. In this example, we'll take a look at how Bro queries the Team Cymru Malware hash registry for detected downloads via HTTP. Part of the Team Cymru Malware Hash registry includes the ability to do a host lookup on a domain with the format MALWARE_HASH.malware.hash.cymru.com where MALWARE_HASH is the md5 or sha1 hash of a file. Team Cymru also populates the TXT record of their DNS responses with both a "last seen" timestamp and a numerical "detection rate". The important aspect to understand is Bro already generates hashes for files it can parse from HTTP streams, but the script detect-MHR.bro is responsible for generating the appropriate DNS lookup and parsing the response. -.. literalinclude:: ../../scripts/policy/protocols/http/detect-MHR.bro +.. literalinclude:: ../scripts/policy/protocols/http/detect-MHR.bro :language: bro :linenos: Visually, there are three distinct sections of the script. A base level with no indentation followed by an indented and formatted section explaining the custom variables being set (export) and another indented and formatted section describing the instructions for a specific event (event log_http). Don't get discouraged if you don't understand every section of the script; we'll cover the basics of the script and much more in following sections. -.. literalinclude:: ../../scripts/policy/protocols/http/detect-MHR.bro +.. literalinclude:: ../scripts/policy/protocols/http/detect-MHR.bro :language: bro :linenos: :lines: 7-11 Lines 7 and 8 of the script process the __load__.bro script in the respective directories being loaded. In a full production deployment of Bro it's likely that these files would already be loaded and their contents made available to the script, but including them explicitly ensures that Bro can be run in modes such as "bare" and still have scripts that reliably work. Consider it a "best practice" to include appropriate @load statments when possible. -.. literalinclude:: ../../scripts/policy/protocols/http/detect-MHR.bro +.. literalinclude:: ../scripts/policy/protocols/http/detect-MHR.bro :language: bro :linenos: :lines: 12-24 The export section redefines an enumerable constant that describes the type of notice we will generate with the logging framework. The notice type listed allows for the use of the NOTICE() function to generate notices of type Malware_Hash_Registry_Match as done in the next section. -.. literalinclude:: ../../scripts/policy/protocols/http/detect-MHR.bro +.. literalinclude:: ../scripts/policy/protocols/http/detect-MHR.bro :language: bro :linenos: :lines: 26-44 @@ -63,7 +64,7 @@ Bro's core acts to place events into an ordered "Event Queue", allowing event ha Gaining familiarity with the specific events generated by Bro is a big step towards building a mind set for working with Bro scripts. The majority of events generated by Bro are defined in the built-in-function files or .bif files which also act as the basis for online event documentation. Whether starting a script from scratch or reading and maintaining someone else's script, having the built-in event definitions available is an excellent resource to have on hand. Before release version 2.0 the Bro developers put significant effort into organization and documentation of every event. This effort resulted in built-in-function files organized such that each entry contains a descriptive event name, the arguments passed to the event, and a concise explanation of the functions use. -.. literalinclude:: ../../build/src/base/event.bif.bro +.. literalinclude:: ../../../../build/src/base/event.bif.bro :language: bro :linenos: :lines: 4124-4149 @@ -77,12 +78,81 @@ Of all the events defined in Bro's event.bif.bro file, an overwhelmingly large n While Bro is capable of packet level processing, its strengths lay in the context of a connection between an originator and a responder. As such, there are events defined for the primary parts of the connection life-cycle as you'll see from the small selection of connection-related events below. -.. literalinclude:: ../../build/src/base/event.bif.bro +.. literalinclude:: ../../../../build/src/base/event.bif.bro :language: bro :linenos: :lines: 135-138,154,204-208,218,255-256,266,335-340,351 -Of the events listed, the event that will give us the best insight into the connection record data type will be connection_state_remove(). As detailed in the in-line documentation, Bro generates this event just before it decides to remove this event from memory, effectively forgetting about it. +Of the events listed, the event that will give us the best insight into the connection record data type will be connection_state_remove(). As detailed in the in-line documentation, Bro generates this event just before it decides to remove this event from memory, effectively forgetting about it. Let's take a look at a simple script that will output the connection record for a single connection. +.. literalinclude:: ../../../../testing/btest/doc/manual/connection_record_01.bro + :language: bro + :linenos: + :lines: 4-9 +Again, we start with @load, this time importing the base/protocols/conn scripts which supply the tracking and logging of general information and state of connections. We handle the connection_state_remove() event and simply print the contents of the argument passed to it. For this example we're going to run Bro in "bare mode" which loads only the minimum number of scripts to retain operability and leaves the burden of loading required scripts to the script being run. This will give us a chance to see the contents of the connection record without it being overly populated. + +.. btest:: connection-record-01 + + @TEST-EXEC: btest-rst-cmd bro -b -r ${TRACES}/dns-session.trace ${TESTBASE}/doc/manual/connection_record_01.bro + +As you can see from the output, the connection record is something of a jumble when printed on its own. Regularly taking a peek at a populated connection record helps to understand the relationship between its fields as well as allowing an opportunity to build a frame of reference for accessing data in a script. + +Bro uses the dollar sign as it's field delimiter and a direct correlation exists between the output of the connection record and the proper format of a dereferenced variable in scripts. In the output of the script above, groups of information are collected between brackets, which would correspond to the $-delimiter in a Bro script. For example, the originating host is referenced by c$id$orig_h which breaks down to "orig_h which is a member of id which is a member of the data structure referred to as c that was passed into the event handler." Given that the responder port (c$id$resp_p) is 53/tcp, it's likely that Bro's base DNS scripts can further populate the connection record. Let's load the base/protocols/dns scripts and check the output of our script. + +.. literalinclude:: ../../../../testing/btest/doc/manual/connection_record_02.bro + :language: bro + :linenos: + :lines: 4-10 + +.. btest:: connection-record-02 + + @TEST-EXEC: btest-rst-cmd bro -b -r ${TRACES}/dns-session.trace ${TESTBASE}/doc/manual/connection_record_02.bro + +The addition of the base/protocols/dns scripts populates the dns=[] member of the connection record. While Bro is doing a massive amount of work in the background, it is in what is commonly called "script land" that details are being refined and decisions being made. Were we to continue running in "bare mode" we could slowly keep adding infrastructure through @load statements. For example, were we to load base/frameworks/logging, Bro would generate a conn.log and dns.log for us in the current working directory. Not only is it good practice to include appropriate load statements in your scripts, but it also helps to illuminate the various functionalities within Bro's scripting language. + +Data Types +========== + +The table below shows the common data types used in Bro, of which, the first four should seem familiar while the remaining six are less common in other languages. A basic understanding of the extra data types in Bro might save you a light night of reinventing the wheel. It should come as no surprise that a scripting language for a Network Security Monitoring platform has a fairly robust set of network-centric data types. While each of the network centric data types should be be familiar in the type of data they represent, seeing the data types in action can be a better starting point. + ++-----------+-------------------------------------+ +| Data Type | Description | ++===========+=====================================+ +| int | 64 bit signed integer | ++-----------+-------------------------------------+ +| count | 64 bit unsigned integer | ++-----------+-------------------------------------+ +| double | double precision floating precision | ++-----------+-------------------------------------+ +| bool | boolean (T/F) | ++-----------+-------------------------------------+ +| addr | ip address, ipv4 and ipv6 | ++-----------+-------------------------------------+ +| port | transport layer port | ++-----------+-------------------------------------+ +| subnet | CIDR subnet mask | ++-----------+-------------------------------------+ +| time | absolute epoch time | ++-----------+-------------------------------------+ +| interval | a time interval | ++-----------+-------------------------------------+ +| pattern | regular expression | ++-----------+-------------------------------------+ + +subnet +------ + +Bro has full support for CIDR notation subnets as a base data type. There is no need to manage the IP and the subnet mask as two seperate entities when you can provide the same information in CIDR notation in your scripts. The following example below uses a Bro script to determine if a series of IP addresses are within a set of subnets using a 20 bit subnet mask. We'll be using some structures we have not yet introduced, as well as some operationts, but we'll touch on them here before going into more detail with them later on. + +.. literalinclude:: ${TESTBASE}/doc/manual/data_type_subnets.bro + :language: bro + :linenos: + :lines: 4-19 + +Because this is a script that doesn't use any kind of network analysis, we can handle the event bro_init() which is always generated by Bro's core upon startup. On lines six and seven, two locally scoped vectors (vectors are like lists in Python or arrays in C) are created to hold our lists of subnets and IP addresses respectively. Then, using a set of nested for loops, we iterate over every subnet and every IP address and use an if statement to compare an IP address against a subnet using the in operator. The in operator returns true if the IP address falls within a given subnet. For example, 10.0.0.1 in 10.0.0.0/8 would return true while 192.168.2.1 in 192.168.1.0/24 would return false. When we run the script, we get the output listing the IP address and the subnet in which it belongs. + +.. btest:: data_type_subnets + + @TEST-EXEC: btest-rst-cmd bro ${TESTBASE}/doc/manual/data_type_subnets.bro diff --git a/testing/btest/Baseline/doc.manual.data_type_subnets/.stdout b/testing/btest/Baseline/doc.manual.data_type_subnets/.stdout new file mode 100644 index 0000000000..facaaabe64 --- /dev/null +++ b/testing/btest/Baseline/doc.manual.data_type_subnets/.stdout @@ -0,0 +1,4 @@ +172.16.4.56 belongs to subnet 172.16.0.0/20 +172.16.47.254 belongs to subnet 172.16.32.0/20 +172.16.22.45 belongs to subnet 172.16.16.0/20 +172.16.1.1 belongs to subnet 172.16.0.0/20 diff --git a/testing/btest/doc/manual/data_type_subnets.bro b/testing/btest/doc/manual/data_type_subnets.bro new file mode 100644 index 0000000000..b0d174069a --- /dev/null +++ b/testing/btest/doc/manual/data_type_subnets.bro @@ -0,0 +1,22 @@ +# @TEST-EXEC: bro -b %INPUT +# @TEST-EXEC: btest-diff .stdout + +event bro_init() + { + local subnets = vector(172.16.0.0/20, 172.16.16.0/20, 172.16.32.0/20, 172.16.48.0/20); + local addresses = vector(172.16.4.56, 172.16.47.254, 172.16.22.45, 172.16.1.1); + for (a in addresses) + { + for (s in subnets) + { + if (addresses[a] in subnets[s]) + { + print fmt("%s belongs to subnet %s", addresses[a], subnets[s]); + } + } + } + + } + + +