Merge remote-tracking branch 'origin/master' into topic/documentation

Conflicts:
	doc/index.rst

CHANGES

@@ -1,4 +1,59 @@
 
+2.1-152 | 2012-11-05 16:52:34 -0800
+
+  * Initial version of a completely reworked intelligence framework.
+    See doc/intel.rst for more information. (Seth Hall)
+
+  * Experimental Modbus analyzer. See policy/protocols/modbus/* for
+    example policies. (Dina Hadziosmanovic, Seth Hall)
+
+2.1-112 | 2012-11-05 13:58:20 -0800
+
+  * New base script for detecting cases of checksum offloading. 
+    Reporter messages will now tell if one has bad checksums. (Seth
+    Hall)
+
+  * Clarifying ownership rules for BroString constructors. (Robin
+    Sommer)
+
+2.1-109 | 2012-11-05 13:39:34 -0800
+
+  * Add detection rate threshold for MHR. (Vlad Grigorescu)
+    
+  * lookup_hostname_txt fixes. (Vlad Grigorescu)
+    
+2.1-104 | 2012-11-01 10:37:50 -0700
+
+  * A new built-in function lookup_hostname_txt() provides support for
+    DNS TXT queries. (Vlad Grigorescu)
+
+2.1-101 | 2012-10-31 14:30:26 -0700
+
+  * Documentation reorg: The install info has been consolidated into a
+    single document (INSTALL), the upgrade info has been moved from
+    the FAQ to a section in the install doc, and the "upgrading from
+    1.5 to 2.0" document has been updated (and renamed) to also
+    include 2.0 to 2.1 upgrade info. (Daniel Thayer)
+
+2.1-96 | 2012-10-31 14:23:50 -0700
+
+  * Renaming option defining the frequency of alarm summary mails to
+    'Logging::default_alarm_mail_interval'. (Daniel Thayer)
+
+2.1-91 | 2012-10-24 16:04:47 -0700
+
+  * Adding PPPoE support to Bro. (Seth Hall)
+
+2.1-87 | 2012-10-24 15:40:06 -0700
+
+  * Adding missing &redef for some TCP options. Addresses #905, #906,
+    #907. (Carsten Langer)
+
+2.1-86 | 2012-10-24 15:37:11 -0700
+
+  * Add parsing rules for IPv4/IPv6 subnet literal constants.
+    Addresses #888. (Jon Siwek)
+
 2.1-84 | 2012-10-19 15:12:56 -0700
 
   * Added a BiF strptime() to wrap the corresponding C function. (Seth

INSTALL

@@ -1,35 +1,55 @@
+.. _CMake: http://www.cmake.org
+.. _SWIG: http://www.swig.org
+.. _Xcode: https://developer.apple.com/xcode/
+.. _MacPorts: http://www.macports.org
+.. _Fink: http://www.finkproject.org
+.. _Homebrew: http://mxcl.github.com/homebrew
+.. _bro downloads page: http://bro-ids.org/download/index.html
+
 ==============
 Installing Bro
 ==============
 
+Bro can be downloaded in either pre-built binary package or
+source code forms.
+
 Prerequisites
 =============
 
 Bro requires the following libraries and tools to be installed
 before you begin:
 
-    * CMake 2.6.3 or greater            http://www.cmake.org
+    * Libpcap                           http://www.tcpdump.org
 
-    * Perl  (used only during the Bro build process)
+    * OpenSSL libraries                 http://www.openssl.org
 
-    * Libpcap headers and libraries     http://www.tcpdump.org
+    * BIND8 library
-
-    * OpenSSL headers and libraries     http://www.openssl.org
-
-    * BIND8 headers and libraries
 
     * Libmagic
 
     * Libz
 
+    * Bash  (for BroControl)
+
+To build Bro from source, the following additional dependencies are required:
+
+    * CMake 2.6.3 or greater            http://www.cmake.org
+
     * SWIG                              http://www.swig.org
 
     * Bison (GNU Parser Generator)
 
     * Flex  (Fast Lexical Analyzer)
 
-    * Bash  (for BroControl)
+    * Libpcap headers                   http://www.tcpdump.org
 
+    * OpenSSL headers                   http://www.openssl.org
+
+    * libmagic headers
+
+    * zlib headers
+
+    * Perl
 
 Bro can make use of some optional libraries and tools if they are found at
 build time:
@@ -45,29 +65,161 @@ build time:
     * Ruby executable, library, and headers (for Broccoli Ruby bindings)
 
 
-Installation
+Installing From Pre-Built Binary Release Packages
-============
+=================================================
 
-To build and install into ``/usr/local/bro``::
+See the `bro downloads page`_ for currently supported/targeted platforms.
+
+* RPM
+
+  .. console::
+
+      sudo yum localinstall Bro-*.rpm
+
+* DEB
+
+  .. console::
+
+      sudo gdebi Bro-*.deb
+
+* MacOS Disk Image with Installer
+
+  Just open the ``Bro-*.dmg`` and then run the ``.pkg`` installer.
+  Everything installed by the package will go into ``/opt/bro``.
+
+The primary install prefix for binary packages is ``/opt/bro``.
+Non-MacOS packages that include BroControl also put variable/runtime
+data (e.g. Bro logs) in ``/var/opt/bro``.
+
+
+Installing From Source
+======================
+
+Required Dependencies
+~~~~~~~~~~~~~~~~~~~~~
+
+The following dependencies are required to build Bro:
+
+* RPM/RedHat-based Linux:
+
+  .. console::
+
+     sudo yum install cmake make gcc gcc-c++ flex bison libpcap-devel openssl-devel python-devel swig zlib-devel file-devel
+
+* DEB/Debian-based Linux:
+
+  .. console::
+
+     sudo apt-get install cmake make gcc g++ flex bison libpcap-dev libssl-dev python-dev swig zlib1g-dev libmagic-dev
+
+* FreeBSD
+
+  Most required dependencies should come with a minimal FreeBSD install
+  except for the following.
+
+  .. console::
+
+      sudo pkg_add -r bash cmake swig bison python
+
+  Note that ``bash`` needs to be in ``PATH``, which by default it is
+  not. The FreeBSD package installs the binary into
+  ``/usr/local/bin``.
+
+* Mac OS X
+
+  Compiling source code on Macs requires first downloading Xcode_,
+  then going through its "Preferences..." -> "Downloads" menus to
+  install the "Command Line Tools" component.
+
+  Lion (10.7) and Mountain Lion (10.8) come with all required
+  dependencies except for CMake_, SWIG_, and ``libmagic``.
+
+  Distributions of these dependencies can likely be obtained from your
+  preferred Mac OS X package management system (e.g. MacPorts_, Fink_,
+  or Homebrew_).
+
+  Specifically for MacPorts, the ``swig``, ``swig-ruby``, ``swig-python``
+  and ``file`` packages provide the required dependencies.
+
+Optional Dependencies
+~~~~~~~~~~~~~~~~~~~~~
+
+Bro can use libGeoIP for geo-locating IP addresses, and sendmail for
+sending emails.
+
+* RedHat Enterprise Linux:
+
+  .. console::
+
+      sudo yum install geoip-devel sendmail
+
+* CentOS Linux:
+
+  .. console::
+  
+      sudo yum install GeoIP-devel sendmail
+
+* DEB/Debian-based Linux:
+
+  .. console::
+
+      sudo apt-get install libgeoip-dev sendmail
+
+* Ports-based FreeBSD
+
+  .. console::
+
+      sudo pkg_add -r GeoIP
+
+  sendmail is typically already available.
+
+* Mac OS X
+
+  Vanilla OS X installations don't ship with libGeoIP, but
+  if installed from your preferred package management system (e.g. MacPorts,
+  Fink, or Homebrew), they should be automatically detected and Bro will
+  compile against them.
+
+Additional steps may be needed to :doc:`get the right GeoIP database <geoip>`.
+
+Compiling Bro Source Code
+~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Bro releases are bundled into source packages for convenience and
+available from the `bro downloads page`_.
+
+Alternatively, the latest Bro development version can be obtained through git
+repositories hosted at `git.bro-ids.org <http://git.bro-ids.org>`_.  See
+our `git development documentation
+<http://bro-ids.org/development/process.html>`_ for comprehensive
+information on Bro's use of git revision control, but the short story
+for downloading the full source code experience for Bro via git is:
+
+.. console::
+
+    git clone --recursive git://git.bro-ids.org/bro
+
+.. note:: If you choose to clone the ``bro`` repository non-recursively for
+   a "minimal Bro experience", be aware that compiling it depends on
+   BinPAC, which has its own ``binpac`` repository.  Either install it
+   first or initialize/update the cloned ``bro`` repository's
+   ``aux/binpac`` submodule.
+
+The typical way to build and install from source is (for more options,
+run ``./configure --help``):
+
+.. console::
 
     ./configure
     make
     make install
 
-This will first build Bro in a directory inside the distribution
+The default installation path is ``/usr/local/bro``, which would typically
-called ``build/``, using default build options. It then installs all
+require root privileges when doing the ``make install``.  A different 
-required files into ``/usr/local/bro``, including the Bro binary in
+installation path can be chosen by specifying the ``--prefix`` option.
-``/usr/local/bro/bin/bro``.
+Note that ``/usr`` and ``/opt/bro`` are the
-
+standard prefixes for binary Bro packages to be installed, so those are
-You can specify a different installation directory with::
+typically not good choices unless you are creating such a package.
-
-    ./configure --prefix=<dir>
-
-Note that ``/usr`` and ``/opt/bro`` are the standard prefixes for
-binary Bro packages to be installed, so those are typically not good
-choices unless you are creating such a package.
-
-Run ``./configure --help`` for more options.
 
 Depending on the Bro package you downloaded, there may be auxiliary
 tools and libraries available in the ``aux/`` directory. Some of them
@@ -81,6 +233,66 @@ OpenBSD users, please see our FAQ at
 http://www.bro-ids.org/documentation/faq.html if you are having
 problems installing Bro.
 
+
+Upgrading From a Previous Version of Bro
+========================================
+
+If you're doing an upgrade install (rather than a fresh install),
+there's two suggested approaches: either install Bro using the same
+installation prefix directory as before, or pick a new prefix and copy
+local customizations over.
+
+Re-Use Previous Install Prefix
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+If you choose to configure and install Bro with the same prefix
+directory as before, local customization and configuration to files in
+``$prefix/share/bro/site`` and ``$prefix/etc`` won't be overwritten
+(``$prefix`` indicating the root of where Bro was installed). Also, logs
+generated at run-time won't be touched by the upgrade. (But making
+a backup of local changes before upgrading is still recommended.)
+
+After upgrading, remember to check ``$prefix/share/bro/site`` and
+``$prefix/etc`` for ``.example`` files, which indicate the
+distribution's version of the file differs from the local one, which may
+include local changes.  Review the differences, and make adjustments
+as necessary (for differences that aren't the result of a local change,
+use the new version's).
+
+Pick a New Install prefix
+~~~~~~~~~~~~~~~~~~~~~~~~~
+
+If you want to install the newer version in a different prefix
+directory than before, you can just copy local customization and
+configuration files from ``$prefix/share/bro/site`` and ``$prefix/etc``
+to the new location (``$prefix`` indicating the root of where Bro was
+originally installed).  Make sure to review the files for difference
+before copying and make adjustments as necessary (for differences that
+aren't the result of a local change, use the new version's).  Of
+particular note, the copied version of ``$prefix/etc/broctl.cfg`` is
+likely to need changes to the ``SpoolDir`` and ``LogDir`` settings.
+
+
+Configure the Run-Time Environment
+==================================
+
+Just remember that you may need to adjust your ``PATH`` environment variable
+according to the platform/shell/package you're using.  For example:
+
+Bourne-Shell Syntax:
+
+.. console::
+
+   export PATH=/usr/local/bro/bin:$PATH
+
+C-Shell Syntax:
+
+.. console::
+
+   setenv PATH /usr/local/bro/bin:$PATH
+
+Or substitute ``/opt/bro/bin`` instead if you installed from a binary package.
+
 Running Bro
 ===========
 

NEWS

@@ -39,6 +39,11 @@ Changed Functionality
   completed, but also after the last event of a whole-file-read (or
   whole-db-read, etc.).
 
+- Renamed the option defining the frequency of alarm summary mails to
+  'Logging::default_alarm_mail_interval'. When using BroControl, the
+  value can now be set with the new broctl.cfg option
+  "MailAlarmsInterval".
+
 Bro 2.1
 -------
 

VERSION

@@ -1 +1 @@
-2.1-84
+2.1-152

aux/bro-aux

@@ -1 +1 @@
-Subproject commit 01bb93cb23f31a98fb400584e8d2f2fbe8a589ef
+Subproject commit bea556198b69d30d64c0cf1b594e6de71176df6f

aux/broctl

@@ -1 +1 @@
-Subproject commit fd0e7e0b0cf50131efaf536a5683266cfe169455
+Subproject commit 834131cd0ec0f63cce9de818726fe6167dedbf34

doc.old/faq.rst

@@ -12,43 +12,6 @@ Frequently Asked Questions
 Installation and Configuration
 ==============================
 
-How do I upgrade to a new version of Bro?
------------------------------------------
-
-There's two suggested approaches, either install Bro using the same
-installation prefix directory as before, or pick a new prefix and copy
-local customizations over.
-
-Re-Use Previous Install Prefix
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-
-If you choose to configure and install Bro with the same prefix
-directory as before, local customization and configuration to files in
-``$prefix/share/bro/site`` and ``$prefix/etc`` won't be overwritten
-(``$prefix`` indicating the root of where Bro was installed). Also, logs
-generated at run-time won't be touched by the upgrade. (But making
-a backup of local changes before proceeding is still recommended.)
-
-After upgrading, remember to check ``$prefix/share/bro/site`` and
-``$prefix/etc`` for ``.example`` files, which indicate the
-distribution's version of the file differs from the local one, which may
-include local changes.  Review the differences, and make adjustments
-as necessary (for differences that aren't the result of a local change,
-use the new version's).
-
-Pick a New Install prefix
-^^^^^^^^^^^^^^^^^^^^^^^^^
-
-If you want to install the newer version in a different prefix
-directory than before, you can just copy local customization and
-configuration files from ``$prefix/share/bro/site`` and ``$prefix/etc``
-to the new location (``$prefix`` indicating the root of where Bro was
-originally installed).  Make sure to review the files for difference
-before copying and make adjustments as necessary (for differences that
-aren't the result of a local change, use the new version's).  Of
-particular note, the copied version of ``$prefix/etc/broctl.cfg`` is
-likely to need changes to the ``SpoolDir`` and ``LogDir`` settings.
-
 How can I tune my operating system for best capture performance?
 ----------------------------------------------------------------
 

doc.old/quickstart.rst

@@ -1,10 +1,3 @@
-.. _CMake: http://www.cmake.org
-.. _SWIG: http://www.swig.org
-.. _Xcode: https://developer.apple.com/xcode/
-.. _MacPorts: http://www.macports.org
-.. _Fink: http://www.finkproject.org
-.. _Homebrew: http://mxcl.github.com/homebrew
-.. _bro downloads page: http://bro-ids.org/download/index.html
 
 =================
 Quick Start Guide
@@ -23,181 +16,11 @@ Installation
 
 Bro works on most modern, Unix-based systems and requires no custom
 hardware.  It can be downloaded in either pre-built binary package or
-source code forms.
+source code forms.  See :doc:`Installing Bro <INSTALL>` for instructions
+on how to install Bro.
 
-Pre-Built Binary Release Packages
+.. note:: Below, ``$PREFIX`` is used to reference the Bro installation
----------------------------------
+   root directory.
-
-See the `bro downloads page`_ for currently supported/targeted platforms.
-
-* RPM
-
-  .. console::
-
-      sudo yum localinstall Bro-*.rpm
-
-* DEB
-
-  .. console::
-
-      sudo gdebi Bro-*.deb
-
-* MacOS Disk Image with Installer
-
-  Just open the ``Bro-*.dmg`` and then run the ``.pkg`` installer.
-  Everything installed by the package will go into ``/opt/bro``.
-
-The primary install prefix for binary packages is ``/opt/bro``.
-Non-MacOS packages that include BroControl also put variable/runtime
-data (e.g. Bro logs) in ``/var/opt/bro``.
-
-Building From Source
---------------------
-
-Required Dependencies
-~~~~~~~~~~~~~~~~~~~~~
-
-The following dependencies are required to build Bro:
-
-* RPM/RedHat-based Linux:
-
-  .. console::
-
-     sudo yum install cmake make gcc gcc-c++ flex bison libpcap-devel openssl-devel python-devel swig zlib-devel file-devel
-
-* DEB/Debian-based Linux:
-
-  .. console::
-
-     sudo apt-get install cmake make gcc g++ flex bison libpcap-dev libssl-dev python-dev swig zlib1g-dev libmagic-dev
-
-* FreeBSD
-
-  Most required dependencies should come with a minimal FreeBSD install
-  except for the following.
-
-  .. console::
-
-      sudo pkg_add -r bash cmake swig bison python
-
-  Note that ``bash`` needs to be in ``PATH``, which by default it is
-  not. The FreeBSD package installs the binary into
-  ``/usr/local/bin``.
-
-* Mac OS X
-
-  Compiling source code on Macs requires first downloading Xcode_,
-  then going through its "Preferences..." -> "Downloads" menus to
-  install the "Command Line Tools" component.
-
-  Lion (10.7) and Mountain Lion (10.8) come with all required
-  dependencies except for CMake_, SWIG_, and ``libmagic``.
-
-  Distributions of these dependencies can be obtained from the project
-  websites linked above, but they're also likely available from your
-  preferred Mac OS X package management system (e.g. MacPorts_, Fink_,
-  or Homebrew_).
-
-  Specifically for MacPorts, the ``swig``, ``swig-ruby``, ``swig-python``
-  and ``file`` packages provide the required dependencies.
-
-Optional Dependencies
-~~~~~~~~~~~~~~~~~~~~~
-
-Bro can use libGeoIP for geo-locating IP addresses, and sendmail for
-sending emails.
-
-* RedHat Enterprise Linux:
-
-  .. console::
-
-      sudo yum install geoip-devel sendmail
-
-* CentOS Linux:
-
-  .. console::
-  
-      sudo yum install GeoIP-devel sendmail
-
-* DEB/Debian-based Linux:
-
-  .. console::
-
-      sudo apt-get install libgeoip-dev sendmail
-
-* Ports-based FreeBSD
-
-  .. console::
-
-      sudo pkg_add -r GeoIP
-
-  sendmail is typically already available.
-
-* Mac OS X
-
-  Vanilla OS X installations don't ship with libmagic or libGeoIP, but
-  if installed from your preferred package management system (e.g. MacPorts,
-  Fink, or Homebrew), they should be automatically detected and Bro will compile
-  against them.
-
-Additional steps may be needed to :doc:`get the right GeoIP database <geoip>`
-
-Compiling Bro Source Code
-~~~~~~~~~~~~~~~~~~~~~~~~~
-
-Bro releases are bundled into source packages for convenience and
-available from the `bro downloads page`_.
-
-The latest Bro development versions are obtainable through git
-repositories hosted at `git.bro-ids.org <http://git.bro-ids.org>`_.  See
-our `git development documentation
-<http://bro-ids.org/development/process.html>`_ for comprehensive
-information on Bro's use of git revision control, but the short story
-for downloading the full source code experience for Bro via git is:
-
-.. console::
-
-    git clone --recursive git://git.bro-ids.org/bro
-
-.. note:: If you choose to clone the ``bro`` repository non-recursively for
-   a "minimal Bro experience", be aware that compiling it depends on
-   BinPAC, which has its own ``binpac`` repository.  Either install it
-   first or initialize/update the cloned ``bro`` repository's
-   ``aux/binpac`` submodule.
-
-See the ``INSTALL`` file included with the source code for more information
-on compiling, but this is the typical way to build and install from source
-(of course, changing the value of the ``--prefix`` option to point to the
-desired root install path):
-
-.. console::
-
-    ./configure --prefix=/desired/install/path
-    make
-    make install
-
-The default installation prefix is ``/usr/local/bro``, which would typically
-require root privileges when doing the ``make install``.
-
-Configure the Run-Time Environment
-----------------------------------
-
-Just remember that you may need to adjust your ``PATH`` environment variable
-according to the platform/shell/package you're using.  For example:
-
-Bourne-Shell Syntax:
-
-.. console::
-
-   export PATH=/usr/local/bro/bin:$PATH
-
-C-Shell Syntax:
-
-.. console::
-
-   setenv PATH /usr/local/bro/bin:$PATH
-
-Or substitute ``/opt/bro/bin`` instead if you installed from a binary package.
 
 Using BroControl
 ================
@@ -206,9 +29,6 @@ BroControl is an interactive shell for easily operating/managing Bro
 installations on a single system or even across multiple systems in a
 traffic-monitoring cluster.
 
-.. note:: Below, ``$PREFIX`` is used to reference the Bro installation
-   root directory.
-
 A Minimal Starting Configuration
 --------------------------------
 

doc.old/upgrade.rst

@@ -1,19 +1,80 @@
 
-=============================
+==========================================
-Upgrading From Bro 1.5 to 2.0
+Upgrading From the Previous Version of Bro
-=============================
+==========================================
 
 .. rst-class:: opening
 
-   This guide details differences between Bro versions 1.5 and 2.0
+   This guide details specific differences between Bro versions
    that may be important for users to know as they work on updating
    their Bro deployment/configuration to the later version.
 
 .. contents::
 
 
-Introduction
+Upgrading From Bro 2.0 to 2.1
-============
+=============================
+
+In Bro 2.1, IPv6 is enabled by default.  Therefore, when building Bro from
+source, the "--enable-brov6" configure option has been removed because it
+is no longer relevant.  
+
+Other configure changes include renaming the "--enable-perftools" option
+to "--enable-perftools-debug" to indicate that the option is only relevant
+for debugging the heap.  One other change involves what happens when
+tcmalloc (part of Google perftools) is found at configure time.  On Linux,
+it will automatically be linked with Bro, but on other platforms you
+need to use the "--enable-perftools" option to enable linking to tcmalloc.
+
+There are a couple of changes to the Bro scripting language to better
+support IPv6. First, IPv6 literals appearing in a Bro script must now be
+enclosed in square brackets (for example, ``[fe80::db15]``). For subnet
+literals, the slash "/" appears after the closing square bracket (for
+example, ``[fe80:1234::]/32``). Second, when an IP address variable or IP
+address literal is enclosed in pipes (for example, ``|[fe80::db15]|``) the
+result is now the size of the address in bits (32 for IPv4 and 128 for IPv6).
+
+In the Bro scripting language, "match" and "using" are no longer reserved
+keywords.
+
+Some built-in functions have been removed: "addr_to_count" (use
+"addr_to_counts" instead), "bro_has_ipv6" (this is no longer relevant
+because Bro now always supports IPv6), "active_connection" (use
+"connection_exists" instead), and "connection_record" (use "lookup_connection"
+instead).
+
+The "NFS3::mode2string" built-in function has been renamed to "file_mode".
+
+Some built-in functions have been changed:  "exit" (now takes the exit code
+as a parameter), "to_port" (now takes a string as parameter instead
+of a count and transport protocol, but "count_to_port" is still available),
+"connect" (now takes an additional string parameter specifying the zone of
+a non-global IPv6 address), and "listen" (now takes three additional
+parameters to enable listening on IPv6 addresses).
+
+Some Bro script variables have been renamed:  "LogAscii::header_prefix"
+has been renamed to "LogAscii::meta_prefix", "LogAscii::include_header"
+has been renamed to "LogAscii::include_meta".
+
+Some Bro script variables have been removed: "tunnel_port",
+"parse_udp_tunnels", "use_connection_compressor", "cc_handle_resets",
+"cc_handle_only_syns", and "cc_instantiate_on_data".
+
+A couple events have changed:  the "icmp_redirect" event now includes
+the target and destination addresses and any Neighbor Discovery options
+in the message, and the last parameter of the "dns_AAAA_reply" event has
+been removed because it was unused.
+
+The format of the ASCII log files has changed very slightly.  Two new lines
+are automatically added, one to record the time when the log was opened,
+and the other to record the time when the log was closed.
+
+In BroControl, the option (in broctl.cfg) "CFlowAddr" was renamed
+to "CFlowAddress".
+
+
+Upgrading From Bro 1.5 to 2.0
+=============================
 
 As the version number jump suggests, Bro 2.0 is a major upgrade and
 lots of things have changed. Most importantly, we have rewritten

doc/index.rst

@@ -5,6 +5,33 @@
 Bro Documentation
 =================
 
+Guides
+------
+
+.. toctree::
+   :maxdepth: 1
+
+   INSTALL
+   upgrade
+   quickstart
+   faq
+   reporting-problems
+
+xFrameworks
+----------
+
+.. toctree::
+   :maxdepth: 1
+
+   notice
+   logging
+   input
+   cluster
+   signatures
+
+How-Tos
+-------
+
 .. toctree::
     :maxdepth: 2
     :numbered:

doc/intel.rst

@@ -0,0 +1,125 @@
+Intel Framework
+===============
+
+Intro
+-----
+
+Intelligence data is critical to the process of monitoring for
+security purposes.  There is always data which will be discovered
+through the incident response process and data which is shared through
+private communities.  The goals of Bro's Intelligence Framework are to
+consume that data, make it available for matching, and provide
+infrastructure around improving performance, memory utilization, and
+generally making all of this easier.
+
+Data in the Intelligence Framework is the atomic piece of intelligence
+such as an IP address or an e-mail address along with a suite of
+metadata about it such as a freeform source field, a freeform
+descriptive field and a URL which might lead to more information about
+the specific item.  The metadata in the default scripts has been
+deliberately kept minimal so that the community can find the
+appropriate fields that need added by writing scripts which extend the
+base record using the normal record extension mechanism.
+
+Quick Start
+-----------
+
+Load the package of scripts that sends data into the Intelligence
+Framework to be checked by loading this script in local.bro::
+
+	@load policy/frameworks/intel
+
+(TODO: find some good mechanism for getting setup with good data
+quickly)
+
+Refer to the "Loading Intelligence" section below to see the format
+for Intelligence Framework text files, then load those text files with
+this line in local.bro::
+
+	redef Intel::read_files += { "/somewhere/yourdata.txt" };
+
+The data itself only needs to reside on the manager if running in a
+cluster.
+
+Architecture
+------------
+
+The Intelligence Framework can be thought of as containing three
+separate portions.  The first part is how intelligence is loaded,
+followed by the mechanism for indicating to the intelligence framework
+that a piece of data which needs to be checked has been seen, and
+thirdly the part where a positive match has been discovered.
+
+Loading Intelligence
+********************
+
+Intelligence data can only be loaded through plain text files using
+the Input Framework conventions.  Additionally, on clusters the
+manager is the only node that needs the intelligence data.  The
+intelligence framework has distribution mechanisms which will push
+data out to all of the nodes that need it.
+
+Here is an example of the intelligence data format.  Note that all
+whitespace separators are literal tabs and fields containing only a
+hyphen a considered to be null values.::
+
+	#fields	host	net	str	str_type	meta.source	meta.desc	meta.url
+	1.2.3.4	-	-	-	source1	Sending phishing email	http://source1.com/badhosts/1.2.3.4
+	-	31.131.248.0/21	-	-	spamhaus-drop	SBL154982	-	-
+	-	-	a.b.com	Intel::DOMAIN	source2	Name used for data exfiltration	-
+
+For more examples of built in `str_type` values, please refer to the
+autogenerated documentation for the intelligence framework (TODO:
+figure out how to do this link).
+
+To load the data once files are created, use the following example
+code to define files to load with your own file names of course::
+
+	redef Intel::read_files += {
+		"/somewhere/feed1.txt",
+		"/somewhere/feed2.txt",
+	};
+
+Remember, the files only need to be present on the file system of the
+manager node on cluster deployments.
+
+Seen Data
+*********
+
+When some bit of data is extracted (such as an email address in the
+"From" header in a message over SMTP), the Intelligence Framework
+needs to be informed that this data was discovered and it's presence
+should be checked within the intelligence data set.  This is
+accomplished through the Intel::seen (TODO: do a reference link)
+function.
+
+Typically users won't need to work with this function due to built in
+hook scripts that Bro ships with that will "see" data and send it into
+the intelligence framework.  A user may only need to load the entire
+package of hook scripts as a module or pick and choose specific
+scripts to load.  Keep in mind that as more data is sent into the
+intelligence framework, the CPU load consumed by Bro will increase
+depending on how many times the Intel::seen function is being called
+which is heavily traffic dependent.
+
+The full package of hook scripts that Bro ships with for sending this
+"seen" data into the intelligence framework can be loading by adding
+this line to local.bro::
+
+	@load policy/frameworks/intel
+
+Intelligence Matches
+********************
+
+Against all hopes, most networks will eventually have a hit on
+intelligence data which could indicate a possible compromise or other
+unwanted activity.  The Intelligence Framework provides an event that
+is generated whenever a match is discovered named Intel::match (TODO:
+make a link to inline docs).  Due to design restrictions placed upon
+the intelligence framework, there is no assurance as to where this
+event will be generated.  It could be generated on the worker where
+the data was seen or on the manager.  When the Intel::match event is
+handled, only the data given as event arguments to the event can be
+assured since the host where the data was seen may not be where
+Intel::match is handled.
+

doc/scripts/DocSourcesList.cmake

@@ -36,6 +36,8 @@ rest_target(${psd} base/frameworks/input/main.bro)
 rest_target(${psd} base/frameworks/input/readers/ascii.bro)
 rest_target(${psd} base/frameworks/input/readers/benchmark.bro)
 rest_target(${psd} base/frameworks/input/readers/raw.bro)
+rest_target(${psd} base/frameworks/intel/cluster.bro)
+rest_target(${psd} base/frameworks/intel/input.bro)
 rest_target(${psd} base/frameworks/intel/main.bro)
 rest_target(${psd} base/frameworks/logging/main.bro)
 rest_target(${psd} base/frameworks/logging/postprocessors/scp.bro)
@@ -62,6 +64,7 @@ rest_target(${psd} base/frameworks/reporter/main.bro)
 rest_target(${psd} base/frameworks/signatures/main.bro)
 rest_target(${psd} base/frameworks/software/main.bro)
 rest_target(${psd} base/frameworks/tunnels/main.bro)
+rest_target(${psd} base/misc/find-checksum-offloading.bro)
 rest_target(${psd} base/protocols/conn/contents.bro)
 rest_target(${psd} base/protocols/conn/inactivity.bro)
 rest_target(${psd} base/protocols/conn/main.bro)
@@ -79,6 +82,8 @@ rest_target(${psd} base/protocols/http/main.bro)
 rest_target(${psd} base/protocols/http/utils.bro)
 rest_target(${psd} base/protocols/irc/dcc-send.bro)
 rest_target(${psd} base/protocols/irc/main.bro)
+rest_target(${psd} base/protocols/modbus/consts.bro)
+rest_target(${psd} base/protocols/modbus/main.bro)
 rest_target(${psd} base/protocols/smtp/entities-excerpt.bro)
 rest_target(${psd} base/protocols/smtp/entities.bro)
 rest_target(${psd} base/protocols/smtp/main.bro)
@@ -100,11 +105,21 @@ rest_target(${psd} base/utils/patterns.bro)
 rest_target(${psd} base/utils/site.bro)
 rest_target(${psd} base/utils/strings.bro)
 rest_target(${psd} base/utils/thresholds.bro)
+rest_target(${psd} base/utils/urls.bro)
 rest_target(${psd} policy/frameworks/communication/listen.bro)
 rest_target(${psd} policy/frameworks/control/controllee.bro)
 rest_target(${psd} policy/frameworks/control/controller.bro)
 rest_target(${psd} policy/frameworks/dpd/detect-protocols.bro)
 rest_target(${psd} policy/frameworks/dpd/packet-segment-logging.bro)
+rest_target(${psd} policy/frameworks/intel/conn-established.bro)
+rest_target(${psd} policy/frameworks/intel/dns.bro)
+rest_target(${psd} policy/frameworks/intel/http-host-header.bro)
+rest_target(${psd} policy/frameworks/intel/http-url.bro)
+rest_target(${psd} policy/frameworks/intel/http-user-agents.bro)
+rest_target(${psd} policy/frameworks/intel/smtp-url-extraction.bro)
+rest_target(${psd} policy/frameworks/intel/smtp.bro)
+rest_target(${psd} policy/frameworks/intel/ssl.bro)
+rest_target(${psd} policy/frameworks/intel/where-locations.bro)
 rest_target(${psd} policy/frameworks/metrics/conn-example.bro)
 rest_target(${psd} policy/frameworks/metrics/http-example.bro)
 rest_target(${psd} policy/frameworks/metrics/ssl-example.bro)
@@ -112,6 +127,7 @@ rest_target(${psd} policy/frameworks/software/version-changes.bro)
 rest_target(${psd} policy/frameworks/software/vulnerable.bro)
 rest_target(${psd} policy/integration/barnyard2/main.bro)
 rest_target(${psd} policy/integration/barnyard2/types.bro)
+rest_target(${psd} policy/integration/collective-intel/main.bro)
 rest_target(${psd} policy/misc/analysis-groups.bro)
 rest_target(${psd} policy/misc/capture-loss.bro)
 rest_target(${psd} policy/misc/loaded-scripts.bro)
@@ -126,7 +142,6 @@ rest_target(${psd} policy/protocols/dns/detect-external-names.bro)
 rest_target(${psd} policy/protocols/ftp/detect.bro)
 rest_target(${psd} policy/protocols/ftp/software.bro)
 rest_target(${psd} policy/protocols/http/detect-MHR.bro)
-rest_target(${psd} policy/protocols/http/detect-intel.bro)
 rest_target(${psd} policy/protocols/http/detect-sqli.bro)
 rest_target(${psd} policy/protocols/http/detect-webapps.bro)
 rest_target(${psd} policy/protocols/http/header-names.bro)
@@ -134,6 +149,8 @@ rest_target(${psd} policy/protocols/http/software-browser-plugins.bro)
 rest_target(${psd} policy/protocols/http/software.bro)
 rest_target(${psd} policy/protocols/http/var-extraction-cookies.bro)
 rest_target(${psd} policy/protocols/http/var-extraction-uri.bro)
+rest_target(${psd} policy/protocols/modbus/known-masters-slaves.bro)
+rest_target(${psd} policy/protocols/modbus/track-memmap.bro)
 rest_target(${psd} policy/protocols/smtp/blocklists.bro)
 rest_target(${psd} policy/protocols/smtp/detect-suspicious-orig.bro)
 rest_target(${psd} policy/protocols/smtp/software.bro)

scripts/base/frameworks/cluster/nodes/manager.bro

@@ -13,8 +13,12 @@
 ## Turn off remote logging since this is the manager and should only log here.
 redef Log::enable_remote_logging = F;
 
+## Log rotation interval.
 redef Log::default_rotation_interval = 1 hrs;
 
+## Alarm summary mail interval.
+redef Log::default_mail_alarms_interval = 24 hrs;
+
 ## Use the cluster's archive logging script.
 redef Log::default_rotation_postprocessor_cmd = "archive-log";
 

scripts/base/frameworks/intel/__load__.bro

@@ -1 +1,11 @@
-@load ./main
+@load ./main
+
+# The cluster framework must be loaded first.
+@load base/frameworks/cluster
+
+@if ( Cluster::is_enabled() )
+@load ./cluster
+@endif
+
+# This needs cluster support to only read on the manager.
+@load ./input

scripts/base/frameworks/intel/cluster.bro

@@ -0,0 +1,61 @@
+##! Cluster transparency support for the intelligence framework.  This is mostly oriented
+##! toward distributing intelligence information across clusters.
+
+@load base/frameworks/cluster
+@load ./input
+
+module Intel;
+
+redef record Item += {
+	## This field is used internally for cluster transparency to avoid 
+	## re-dispatching intelligence items over and over from workers.
+	first_dispatch: bool &default=T;
+};
+
+# If this process is not a manager process, we don't want the full metadata
+@if ( Cluster::local_node_type() != Cluster::MANAGER )
+redef have_full_data = F;
+@endif
+
+global cluster_new_item: event(item: Item);
+
+# Primary intelligence distribution comes from manager.
+redef Cluster::manager2worker_events += /^Intel::(cluster_new_item)$/;
+# If a worker finds intelligence and adds it, it should share it back to the manager.
+redef Cluster::worker2manager_events += /^Intel::(cluster_new_item|match_no_items)$/;
+
+@if ( Cluster::local_node_type() == Cluster::MANAGER )
+event Intel::match_no_items(s: Seen) &priority=5
+	{
+	event Intel::match(s, Intel::get_items(s));
+	}
+
+event remote_connection_handshake_done(p: event_peer)
+	{
+	# When a worker connects, send it the complete minimal data store.
+	# It will be kept up to date after this by the cluster_new_item event.
+	if ( Cluster::nodes[p$descr]$node_type == Cluster::WORKER )
+		{
+		send_id(p, "Intel::min_data_store");
+		}
+	}
+@endif
+
+event Intel::cluster_new_item(item: Intel::Item) &priority=5
+	{
+	# Ignore locally generated events to avoid event storms.
+	if ( is_remote_event() )
+		Intel::insert(item);
+	}
+
+event Intel::new_item(item: Intel::Item) &priority=5
+	{
+	# The cluster manager always rebroadcasts intelligence.
+	# Workers redistribute it if it was locally generated.
+	if ( Cluster::local_node_type() == Cluster::MANAGER ||
+	     item$first_dispatch )
+		{
+		item$first_dispatch=F;
+		event Intel::cluster_new_item(item);
+		}
+	}

scripts/base/frameworks/intel/input.bro

@@ -0,0 +1,33 @@
+@load ./main
+
+module Intel;
+
+export {
+	## Intelligence files that will be read off disk.  The files are
+	## reread everytime they are updated so updates much be atomic with
+	## "mv" instead of writing the file in place.
+	const read_files: set[string] = {} &redef;
+}
+
+event Intel::read_entry(desc: Input::EventDescription, tpe: Input::Event, item: Intel::Item)
+	{
+	Intel::insert(item);
+	}
+
+event bro_init() &priority=5
+	{
+	if ( ! Cluster::is_enabled() ||
+	     Cluster::local_node_type() == Cluster::MANAGER )
+		{
+		for ( a_file in read_files )
+			{
+			Input::add_event([$source=a_file,
+			                  $reader=Input::READER_ASCII,
+			                  $mode=Input::REREAD,
+			                  $name=cat("intel-", a_file),
+			                  $fields=Intel::Item,
+			                  $ev=Intel::read_entry]);
+			}
+		}
+	}
+

scripts/base/frameworks/intel/main.bro

@@ -1,323 +1,345 @@
 ##! The intelligence framework provides a way to store and query IP addresses,
-##! strings (with a subtype), and numeric (with a subtype) data.  Metadata 
+##! and strings (with a str_type).  Metadata can
-##! also be associated with the intelligence like tags which are arbitrary
+##! also be associated with the intelligence like for making more informed
-##! strings, time values, and longer descriptive strings.
+##! decisions about matching and handling of intelligence.
-
-# Example string subtypes:
-#   url
-#   email
-#   domain
-#   software
-#   user_name
-#   file_name
-#   file_md5
-#   x509_md5
-
-# Example tags: 
-#   infrastructure
-#   malicious
-#   sensitive
-#   canary
-#   friend
 
 @load base/frameworks/notice
 
 module Intel;
 
 export {
-	## The intel logging stream identifier.
 	redef enum Log::ID += { LOG };
 	
-	redef enum Notice::Type += {
+	## String data needs to be further categoried since it could represent
-		## This notice should be used in all detector scripts to indicate 
+	## and number of types of data.
-		## an intelligence based detection.
+	type StrType: enum {
-		Detection,
+		## A complete URL without the prefix "http://".
+		URL,
+		## User-Agent string, typically HTTP or mail message body.
+		USER_AGENT,
+		## Email address.
+		EMAIL,
+		## DNS domain name.
+		DOMAIN,
+		## A user name.
+		USER_NAME,
+		## File hash which is non-hash type specific.  It's up to the user to query
+		## for any relevant hash types.
+		FILE_HASH,
+		## Certificate SHA-1 hash.
+		CERT_HASH,
 	};
 	
-	## Record type used for logging information from the intelligence framework.
+	## Data about an :bro:type:`Intel::Item`
-	## Primarily for problems or oddities with inserting and querying data.  
-	## This is important since the content of the intelligence framework can 
-	## change quite dramatically during runtime and problems may be introduced 
-	## into the data.
-	type Info: record {
-		## The current network time.
-		ts:      time   &log;
-		## Represents the severity of the message. 
-		## This value should be one of: "info", "warn", "error"
-		level:   string &log;
-		## The message.
-		message: string &log;
-	};
-	
-	## Record to represent metadata associated with a single piece of
-	## intelligence.
 	type MetaData: record {
-		## A description for the data.
+		## An arbitrary string value representing the data source.  Typically,
+		## the convention for this field will be the source name and feed name
+		## separated by a hyphen.  For example: "source1-c&c".
+		source:      string;
+		## A freeform description for the data.
 		desc:        string      &optional;
-		## A URL where more information may be found about the intelligence.
+		## A URL for more information about the data.
 		url:         string      &optional;
-		## The time at which the data was first declared to be intelligence.
-		first_seen:  time        &optional;
-		## When this data was most recent inserted into the framework.
-		latest_seen: time        &optional;
-		## Arbitrary text tags for the data.
-		tags:        set[string];
 	};
 	
-	## Record to represent a singular piece of intelligence.
+	## Represents a piece of intelligence.
 	type Item: record {
-		## If the data is an IP address, this hold the address.
+		## The IP address if the intelligence is about an IP address.
-		ip:          addr        &optional;
+		host:        addr           &optional;
-		## If the data is textual, this holds the text.
+		## The network if the intelligence is about a CIDR block.
-		str:         string      &optional;
+		net:         subnet         &optional;
-		## If the data is numeric, this holds the number.
+		## The string if the intelligence is about a string.
-		num:         int         &optional;
+		str:         string         &optional;
-		## The subtype of the data for when either the $str or $num fields are
+		## The type of data that is in the string if the $str field is set.
-		## given.  If one of those fields are given, this field must be present.
+		str_type:    StrType        &optional;
-		subtype:     string      &optional;
 		
-		## The next five fields are temporary until a better model for 
+		## Metadata for the item.  Typically represents more deeply \
-		## attaching metadata to an intelligence item is created.
+		## descriptive data for a piece of intelligence.
-		desc:        string      &optional;
+		meta:        MetaData;
-		url:         string      &optional;
-		first_seen:  time        &optional;
-		latest_seen: time        &optional;
-		tags:        set[string];
-		
-		## These single string tags are throw away until pybroccoli supports sets.
-		tag1: string &optional;
-		tag2: string &optional;
-		tag3: string &optional;
 	};
 	
-	## Record model used for constructing queries against the intelligence 
+	## Enum to represent where data came from when it was discovered.
-	## framework.
+	## The convenction is to prefix the name with "IN_".
-	type QueryItem: record {
+	type Where: enum {
-		## If an IP address is being queried for, this field should be given.
+		## A catchall value to represent data of unknown provenance.
-		ip:        addr        &optional;
+		IN_ANYWHERE,
-		## If a string is being queried for, this field should be given.
-		str:       string      &optional;
-		## If numeric data is being queried for, this field should be given.
-		num:       int         &optional;
-		## If either a string or number is being queried for, this field should
-		## indicate the subtype of the data.
-		subtype:   string      &optional;
-		
-		## A set of tags where if a single metadata record attached to an item
-		## has any one of the tags defined in this field, it will match.
-		or_tags:   set[string] &optional;
-		## A set of tags where a single metadata record attached to an item 
-		## must have all of the tags defined in this field.
-		and_tags:  set[string] &optional; 
-		
-		## The predicate can be given when searching for a match.  It will
-		## be tested against every :bro:type:`Intel::MetaData` item associated
-		## with the data being matched on.  If it returns T a single time, the 
-		## matcher will consider that the item has matched.  This field can
-		## be used for constructing arbitrarily complex queries that may not
-		## be possible with the $or_tags or $and_tags fields.
-		pred:      function(meta: Intel::MetaData): bool &optional;
 	};
-	
+
-	## Function to insert data into the intelligence framework.
+	## The $host field and combination of $str and $str_type fields are mutually 
-	## 
+	## exclusive.  These records *must* represent either an IP address being
-	## item: The data item.
+	## seen or a string being seen.
+	type Seen: record {
+		## The IP address if the data seen is an IP address.
+		host:      addr          &log &optional;
+		## The string if the data is about a string.
+		str:       string        &log &optional;
+		## The type of data that is in the string if the $str field is set.
+		str_type:  StrType       &log &optional;
+
+		## Where the data was discovered.
+		where:     Where         &log;
+		
+		## If the data was discovered within a connection, the 
+		## connection record should go into get to give context to the data.
+		conn:      connection    &optional;
+	};
+
+	## Record used for the logging framework representing a positive
+	## hit within the intelligence framework.
+	type Info: record {
+		## Timestamp when the data was discovered.
+		ts:       time           &log;
+
+		## If a connection was associated with this intelligence hit,
+		## this is the uid for the connection
+		uid:      string         &log &optional;
+		## If a connection was associated with this intelligence hit,
+		## this is the conn_id for the connection.
+		id:       conn_id        &log &optional;
+
+		## Where the data was seen.
+		seen:     Seen           &log;
+		## Sources which supplied data that resulted in this match.
+		sources:  set[string]    &log;
+	};
+
+	## Intelligence data manipulation functions.
+	global insert: function(item: Item);
+
+	## Function to declare discovery of a piece of data in order to check
+	## it against known intelligence for matches.
+	global seen: function(s: Seen);
+
+	## Event to represent a match in the intelligence data from data that was seen.  
+	## On clusters there is no assurance as to where this event will be generated 
+	## so do not assume that arbitrary global state beyond the given data
+	## will be available.
 	##
-	## Returns: T if the data was successfully inserted into the framework,
+	## This is the primary mechanism where a user will take actions based on data
-	##          otherwise it returns F.
+	## within the intelligence framework.
-	global insert: function(item: Item): bool;
+	global match: event(s: Seen, items: set[Item]);
-	
+
-	## A wrapper for the :bro:id:`Intel::insert` function.  This is primarily
+	global log_intel: event(rec: Info);
-	## used as the external API for inserting data into the intelligence 
-	## using Broccoli.
-	global insert_event: event(item: Item);
-	
-	## Function for matching data within the intelligence framework.
-	global matcher: function(item: QueryItem): bool;
 }
 
-type MetaDataStore: table[count] of MetaData;
+# Internal handler for matches with no metadata available.
+global match_no_items: event(s: Seen);
+
+# Internal events for cluster data distribution
+global new_item: event(item: Item);
+global updated_item: event(item: Item);
+
+# Optionally store metadata.  This is used internally depending on
+# if this is a cluster deployment or not.
+const have_full_data = T &redef;
+
+# The in memory data structure for holding intelligence.
 type DataStore: record {
-	ip_data:     table[addr] of MetaDataStore;
+	net_data:    table[subnet] of set[MetaData];
-	# The first string is the actual value and the second string is the subtype.
+	string_data: table[string, StrType] of set[MetaData];
-	string_data: table[string, string] of MetaDataStore;
-	int_data:    table[int, string] of MetaDataStore;
 };
-global data_store: DataStore;
+global data_store: DataStore &redef;
 
-event bro_init()
+# The in memory data structure for holding the barest matchable intelligence.
+# This is primarily for workers to do the initial quick matches and store
+# a minimal amount of data for the full match to happen on the manager.
+type MinDataStore: record {
+	net_data:    set[subnet];
+	string_data: set[string, StrType];
+};
+global min_data_store: MinDataStore &redef;
+
+
+event bro_init() &priority=5
 	{
-	Log::create_stream(Intel::LOG, [$columns=Info]);
+	Log::create_stream(LOG, [$columns=Info, $ev=log_intel]);
 	}
 
-
+function find(s: Seen): bool
-function insert(item: Item): bool
 	{
-	local err_msg = "";
+	if ( s?$host && 
-	if ( (item?$str || item?$num) && ! item?$subtype )
+	     ((have_full_data && s$host in data_store$net_data) || 
-		err_msg = "You must provide a subtype to insert_sync or this item doesn't make sense.";
+	      (s$host in min_data_store$net_data)))
-	
-	if ( err_msg == "" )
 		{
-		# Create and fill out the meta data item.
+		return T;
-		local meta: MetaData;
+		}
-		if ( item?$first_seen )
+	else if ( s?$str && s?$str_type &&
-			meta$first_seen = item$first_seen;
+	          ((have_full_data && [s$str, s$str_type] in data_store$string_data) ||
-		if ( item?$latest_seen )
+	           ([s$str, s$str_type] in min_data_store$string_data)))
-			meta$latest_seen = item$latest_seen;
+		{
-		if ( item?$tags )
+		return T;
-			meta$tags = item$tags;
+		}
-		if ( item?$desc )
+	else
-			meta$desc = item$desc;
+		{
-		if ( item?$url )
+		return F;
-			meta$url = item$url;
+		}
-		
+	}
-		
-		# This is hopefully only temporary until pybroccoli supports sets.
-		if ( item?$tag1 )
-			add item$tags[item$tag1];
-		if ( item?$tag2 )
-			add item$tags[item$tag2];
-		if ( item?$tag3 )
-			add item$tags[item$tag3];
-		
-		if ( item?$ip )
-			{
-			if ( item$ip !in data_store$ip_data )
-				data_store$ip_data[item$ip] = table();
-			data_store$ip_data[item$ip][|data_store$ip_data[item$ip]|] = meta;
-			return T;
-			}
-		else if ( item?$str )
-			{
-			if ( [item$str, item$subtype] !in data_store$string_data )
-				data_store$string_data[item$str, item$subtype] = table();
-			
-			data_store$string_data[item$str, item$subtype][|data_store$string_data[item$str, item$subtype]|] = meta;
-			return T;
-			}
-		else if ( item?$num )
-			{
-			if ( [item$num, item$subtype] !in data_store$int_data )
-				data_store$int_data[item$num, item$subtype] = table();
 
-			data_store$int_data[item$num, item$subtype][|data_store$int_data[item$num, item$subtype]|] = meta;
+function get_items(s: Seen): set[Item]
-			return T;
+	{
+	local item: Item;
+	local return_data: set[Item] = set();
+
+	if ( ! have_full_data )
+		{
+		# A reporter warning should be generated here because this function
+		# should never be called from a host that doesn't have the full data.
+		# TODO: do a reporter warning.
+		return return_data;
+		}
+
+	if ( s?$host )
+		{
+		# See if the host is known about and it has meta values
+		if ( s$host in data_store$net_data )
+			{
+			for ( m in data_store$net_data[s$host] )
+				{
+				# TODO: the lookup should be finding all and not just most specific
+				#       and $host/$net should have the correct value.
+				item = [$host=s$host, $meta=m];
+				add return_data[item];
+				}
+			}
+		}
+	else if ( s?$str && s?$str_type )
+		{
+		# See if the string is known about and it has meta values
+		if ( [s$str, s$str_type] in data_store$string_data )
+			{
+			for ( m in data_store$string_data[s$str, s$str_type] )
+				{
+				item = [$str=s$str, $str_type=s$str_type, $meta=m];
+				add return_data[item];
+				}
+			}
+		}
+
+	return return_data;
+	}
+
+function Intel::seen(s: Seen)
+	{
+	if ( find(s) )
+		{
+		if ( have_full_data )
+			{
+			local items = get_items(s);
+			event Intel::match(s, items);
 			}
 		else
-			err_msg = "Failed to insert intelligence item for some unknown reason.";
-		}
-	
-	if ( err_msg != "" )
-		Log::write(Intel::LOG, [$ts=network_time(), $level="warn", $message=fmt(err_msg)]);
-	return F;
-	}
-	
-event insert_event(item: Item)
-	{
-	insert(item);
-	}
-	
-function match_item_with_metadata(item: QueryItem, meta: MetaData): bool
-	{
-	if ( item?$and_tags )
-		{
-		local matched = T;
-		# Every tag given has to match in a single MetaData entry.
-		for ( tag in item$and_tags )
 			{
-			if ( tag !in meta$tags )
+			event Intel::match_no_items(s);
-				matched = F;
 			}
-		if ( matched )
+		}
+	}
+
+
+function has_meta(check: MetaData, metas: set[MetaData]): bool
+	{
+	local check_hash = md5_hash(check);
+	for ( m in metas )
+		{
+		if ( check_hash == md5_hash(m) )
 			return T;
 		}
-	else if ( item?$or_tags )
-		{
-		# For OR tags, only a single tag has to match.
-		for ( tag in item$or_tags )
-			{
-			if ( tag in meta$tags )
-				return T;
-			}
-		}
-	else if ( item?$pred )
-		return item$pred(meta);
 
-	# This indicates some sort of failure in the query
+	# The records must not be equivalent if we made it this far.
 	return F;
 	}
-	
+
-function matcher(item: QueryItem): bool
+event Intel::match(s: Seen, items: set[Item]) &priority=5
 	{
-	local err_msg = "";
+	local empty_set: set[string] = set();
-	if ( ! (item?$ip || item?$str || item?$num) )
+	local info: Info = [$ts=network_time(), $seen=s, $sources=empty_set];
-		err_msg = "You must supply one of the $ip, $str, or $num fields to search on";
-	else if ( (item?$or_tags || item?$and_tags) && item?$pred )
-		err_msg = "You can't match with both tags and a predicate.";
-	else if ( item?$or_tags && item?$and_tags )
-		err_msg = "You can't match with both OR'd together tags and AND'd together tags";
-	else if ( (item?$str || item?$num) && ! item?$subtype )
-		err_msg = "You must provide a subtype to matcher or this item doesn't make sense.";
-	else if ( item?$str && item?$num )
-		err_msg = "You must only provide $str or $num, not both.";
-		
-	local meta: MetaData;
 
-	if ( err_msg == "" )
+	if ( s?$conn )
 		{
-		if ( item?$ip )
+		info$uid = s$conn$uid;
-			{
+		info$id  = s$conn$id;
-			if ( item$ip in data_store$ip_data )
-				{
-				if ( ! item?$and_tags && ! item?$or_tags && ! item?$pred )
-					return T;
-			
-				for ( i in data_store$ip_data[item$ip] )
-					{
-					meta = data_store$ip_data[item$ip][i];
-					if ( match_item_with_metadata(item, meta) )
-						return T;
-					}
-				}
-			}
-		
-		else if ( item?$str )
-			{
-			if ( [item$str, item$subtype] in data_store$string_data )
-				{
-				if ( ! item?$and_tags && ! item?$or_tags && ! item?$pred )
-					return T;
-
-				for ( i in data_store$string_data[item$str, item$subtype] )
-					{
-					meta = data_store$string_data[item$str, item$subtype][i];
-					if ( match_item_with_metadata(item, meta) )
-						return T;
-					}
-				}
-			}
-		
-		else if ( item?$num )
-			{
-			if ( [item$num, item$subtype] in data_store$int_data )
-				{
-				if ( ! item?$and_tags && ! item?$or_tags && ! item?$pred )
-					return T;
-
-				for ( i in data_store$int_data[item$num, item$subtype] )
-					{
-					meta = data_store$int_data[item$num, item$subtype][i];
-					if ( match_item_with_metadata(item, meta) )
-						return T;
-					}
-				}
-			}
-		else
-			err_msg = "Failed to query intelligence data for some unknown reason.";
 		}
-		
+
-	if ( err_msg != "" )
+	for ( item in items )
-		Log::write(Intel::LOG, [$ts=network_time(), $level="error", $message=fmt(err_msg)]);
+		add info$sources[item$meta$source];
-	return F;
+
+	Log::write(Intel::LOG, info);
 	}
+
+function insert(item: Item)
+	{
+	if ( item?$str && !item?$str_type )
+		{
+		event reporter_warning(network_time(), fmt("You must provide a str_type for strings or this item doesn't make sense.  Item: %s", item), "");
+		return;
+		}
+
+	# Create and fill out the meta data item.
+	local meta = item$meta;
+	local metas: set[MetaData];
+
+	if ( item?$host )
+		{
+		local host = mask_addr(item$host, is_v4_addr(item$host) ? 32 : 128);
+		if ( have_full_data )
+			{
+			if ( host !in data_store$net_data )
+				data_store$net_data[host] = set();
+
+			metas = data_store$net_data[host];
+			}
+
+		add min_data_store$net_data[host];
+		}
+	else if ( item?$net )
+		{
+		if ( have_full_data )
+			{
+			if ( item$net !in data_store$net_data )
+				data_store$net_data[item$net] = set();
+
+			metas = data_store$net_data[item$net];
+			}
+
+		add min_data_store$net_data[item$net];
+		}
+	else if ( item?$str )
+		{
+		if ( have_full_data )
+			{
+			if ( [item$str, item$str_type] !in data_store$string_data )
+				data_store$string_data[item$str, item$str_type] = set();
+
+			metas = data_store$string_data[item$str, item$str_type];
+			}
+
+		add min_data_store$string_data[item$str, item$str_type];
+		}
+
+	local updated = F;
+	if ( have_full_data )
+		{
+		for ( m in metas )
+			{
+			if ( meta$source == m$source )
+				{
+				if ( has_meta(meta, metas) )
+					{
+					# It's the same item being inserted again.
+					return;
+					}
+				else
+					{
+					# Same source, different metadata means updated item.
+					updated = T;
+					}
+				}
+			}
+		add metas[item$meta];
+		}
+	
+	if ( updated )
+		event Intel::updated_item(item);
+	else
+		event Intel::new_item(item);
+	}
+	

scripts/base/frameworks/logging/main.bro

@@ -60,6 +60,9 @@ export {
 	## Default rotation interval. Zero disables rotation.
 	const default_rotation_interval = 0secs &redef;
 
+	## Default alarm summary mail interval. Zero disables alarm summary mails.
+	const default_mail_alarms_interval = 0secs &redef;
+
 	## Default naming format for timestamps embedded into filenames.
 	## Uses a ``strftime()`` style.
 	const default_rotation_date_format = "%Y-%m-%d-%H-%M-%S" &redef;

scripts/base/frameworks/notice/actions/pp-alarms.bro

@@ -101,7 +101,7 @@ event bro_init()
 	# This replaces the standard non-pretty-printing filter.
 	Log::add_filter(Notice::ALARM_LOG,
 			[$name="alarm-mail", $writer=Log::WRITER_NONE,
-			 $interv=Log::default_rotation_interval,
+			 $interv=Log::default_mail_alarms_interval,
 			 $postprocessor=pp_postprocessor]);
 	}
 

scripts/base/init-bare.bro

@@ -826,7 +826,7 @@ const tcp_storm_interarrival_thresh = 1 sec &redef;
 ## peer's ACKs.  Set to zero to turn off this determination.
 ##
 ## .. bro:see:: tcp_max_above_hole_without_any_acks tcp_excessive_data_without_further_acks
-const tcp_max_initial_window = 4096;
+const tcp_max_initial_window = 4096 &redef;
 
 ## If we're not seeing our peer's ACKs, the maximum volume of data above a sequence
 ## hole that we'll tolerate before assuming that there's been a packet drop and we
@@ -834,7 +834,7 @@ const tcp_max_initial_window = 4096;
 ## up.
 ##
 ## .. bro:see:: tcp_max_initial_window tcp_excessive_data_without_further_acks
-const tcp_max_above_hole_without_any_acks = 4096;
+const tcp_max_above_hole_without_any_acks = 4096 &redef;
 
 ## If we've seen this much data without any of it being acked, we give up
 ## on that connection to avoid memory exhaustion due to buffering all that
@@ -843,7 +843,7 @@ const tcp_max_above_hole_without_any_acks = 4096;
 ## has in fact gone too far, but for now we just make this quite beefy.
 ##
 ## .. bro:see:: tcp_max_initial_window tcp_max_above_hole_without_any_acks
-const tcp_excessive_data_without_further_acks = 10 * 1024 * 1024;
+const tcp_excessive_data_without_further_acks = 10 * 1024 * 1024 &redef;
 
 ## For services without an a handler, these sets define originator-side ports that
 ## still trigger reassembly.
@@ -2457,6 +2457,16 @@ type bittorrent_benc_dir: table[string] of bittorrent_benc_value;
 ##    bt_tracker_response_not_ok
 type bt_tracker_headers: table[string] of string;
 
+type ModbusCoils: vector of bool;
+type ModbusRegisters: vector of count;
+
+type ModbusHeaders: record {
+	tid:           count;
+	pid:           count;
+	uid:           count;
+	function_code: count;
+};
+
 module SOCKS;
 export {
 	## This record is for a SOCKS client or server to provide either a 

scripts/base/init-default.bro

@@ -14,6 +14,7 @@
 @load base/utils/patterns
 @load base/utils/strings
 @load base/utils/thresholds
+@load base/utils/urls
 
 # This has some deep interplay between types and BiFs so it's 
 # loaded in base/init-bare.bro
@@ -36,8 +37,11 @@
 @load base/protocols/ftp
 @load base/protocols/http
 @load base/protocols/irc
+@load base/protocols/modbus
 @load base/protocols/smtp
 @load base/protocols/socks
 @load base/protocols/ssh
 @load base/protocols/ssl
 @load base/protocols/syslog
+
+@load base/misc/find-checksum-offloading

scripts/base/misc/find-checksum-offloading.bro

@@ -0,0 +1,57 @@
+##! Discover cases where the local interface is sniffed and outbound packets
+##! have checksum offloading.  Load this script to receive a notice if it's
+##! likely that checksum offload effects are being seen on a live interface or
+##! in a packet trace file.
+
+@load base/frameworks/notice
+
+module ChecksumOffloading;
+
+export {
+	## The interval which is used for checking packet statistics
+	## to see if checksum offloading is affecting analysis.
+	const check_interval = 10secs &redef;
+}
+
+# Keep track of how many bad checksums have been seen.
+global bad_checksums = 0;
+
+# Track to see if this script is done so that messages aren't created multiple times.
+global done = F;
+
+event ChecksumOffloading::check()
+	{
+	if ( done )
+		return;
+
+	local pkts_recvd = net_stats()$pkts_recvd;
+	if ( (bad_checksums*1.0 / net_stats()$pkts_recvd*1.0) > 0.05 )
+		{
+		local packet_src = reading_traces() ? "trace file likely has" : "interface is likely receiving";
+		local message = fmt("Your %s invalid IP checksums, most likely from NIC checksum offloading.", packet_src);
+		Reporter::warning(message);
+		done = T;
+		}
+	else if ( pkts_recvd < 20 )
+		{
+		# Keep scheduling this event until we've seen some lower threshold of
+		# total packets.
+		schedule check_interval { ChecksumOffloading::check() };
+		}
+	}
+
+event bro_init()
+	{
+	schedule check_interval { ChecksumOffloading::check() };
+	}
+
+event net_weird(name: string)
+	{
+	if ( name == "bad_IP_checksum" )
+		++bad_checksums;
+	}
+
+event bro_done()
+	{
+	event ChecksumOffloading::check();
+	}

scripts/base/protocols/modbus/__load__.bro

@@ -0,0 +1,2 @@
+@load ./consts
+@load ./main

scripts/base/protocols/modbus/consts.bro

@@ -0,0 +1,67 @@
+
+module Modbus;
+
+export {
+	## Standard defined Modbus function codes.
+	const function_codes = {
+		[0x01] = "READ_COILS",
+		[0x02] = "READ_DISCRETE_INPUTS",
+		[0x03] = "READ_HOLDING_REGISTERS",
+		[0x04] = "READ_INPUT_REGISTERS",
+		[0x05] = "WRITE_SINGLE_COIL",
+		[0x06] = "WRITE_SINGLE_REGISTER",
+		[0x07] = "READ_EXCEPTION_STATUS",
+		[0x08] = "DIAGNOSTICS",
+		[0x0B] = "GET_COMM_EVENT_COUNTER",
+		[0x0C] = "GET_COMM_EVENT_LOG",
+		[0x0F] = "WRITE_MULTIPLE_COILS",
+		[0x10] = "WRITE_MULTIPLE_REGISTERS",
+		[0x11] = "REPORT_SLAVE_ID",
+		[0x14] = "READ_FILE_RECORD",
+		[0x15] = "WRITE_FILE_RECORD",
+		[0x16] = "MASK_WRITE_REGISTER",
+		[0x17] = "READ_WRITE_MULTIPLE_REGISTERS",
+		[0x18] = "READ_FIFO_QUEUE",
+		[0x2B] = "ENCAP_INTERFACE_TRANSPORT",
+
+		# Machine/vendor/network specific functions
+		[0x09] = "PROGRAM_484",
+		[0x0A] = "POLL_484",
+		[0x0D] = "PROGRAM_584_984",
+		[0x0E] = "POLL_584_984",
+		[0x12] = "PROGRAM_884_U84",
+		[0x13] = "RESET_COMM_LINK_884_U84",
+		[0x28] = "PROGRAM_CONCEPT",
+		[0x7D] = "FIRMWARE_REPLACEMENT",
+		[0x7E] = "PROGRAM_584_984_2",
+		[0x7F] = "REPORT_LOCAL_ADDRESS",
+
+		# Exceptions
+		[0x81] = "READ_COILS_EXCEPTION",
+		[0x82] = "READ_DISCRETE_INPUTS_EXCEPTION",
+		[0x83] = "READ_HOLDING_REGISTERS_EXCEPTION",
+		[0x84] = "READ_INPUT_REGISTERS_EXCEPTION",
+		[0x85] = "WRITE_SINGLE_COIL_EXCEPTION",
+		[0x86] = "WRITE_SINGLE_REGISTER_EXCEPTION",
+		[0x87] = "READ_EXCEPTION_STATUS_EXCEPTION",
+		[0x8F] = "WRITE_MULTIPLE_COILS_EXCEPTION",
+		[0x90] = "WRITE_MULTIPLE_REGISTERS_EXCEPTION",
+		[0x94] = "READ_FILE_RECORD_EXCEPTION",
+		[0x95] = "WRITE_FILE_RECORD_EXCEPTION",
+		[0x96] = "MASK_WRITE_REGISTER_EXCEPTION",
+		[0x97] = "READ_WRITE_MULTIPLE_REGISTERS_EXCEPTION",
+		[0x98] = "READ_FIFO_QUEUE_EXCEPTION",
+	} &default=function(i: count):string { return fmt("unknown-%d", i); } &redef;
+
+	const exception_codes = {
+		[0x01] = "ILLEGAL_FUNCTION",
+		[0x02] = "ILLEGAL_DATA_ADDRESS",
+		[0x03] = "ILLEGAL_DATA_VALUE",
+		[0x04] = "SLAVE_DEVICE_FAILURE",
+		[0x05] = "ACKNOWLEDGE",
+		[0x06] = "SLAVE_DEVICE_BUSY",
+		[0x08] = "MEMORY_PARITY_ERROR",
+		[0x0A] = "GATEWAY_PATH_UNAVAILABLE",
+		[0x0B] = "GATEWAY_TARGET_DEVICE_FAILED_TO_RESPOND",
+	} &default=function(i: count):string { return fmt("unknown-%d", i); } &redef;
+}

scripts/base/protocols/modbus/main.bro

@@ -0,0 +1,71 @@
+##! Base Modbus analysis script.
+
+module Modbus;
+
+@load ./consts
+
+export {
+	redef enum Log::ID += { LOG };
+
+	type Info: record {
+		## Time of the request.
+		ts:        time           &log;
+		## Unique identifier for the connnection.
+		uid:       string         &log;
+		## Identifier for the connection.
+		id:        conn_id        &log;
+		## The name of the function message that was sent.
+		func:      string         &log &optional;
+		## The exception if the response was a failure.
+		exception: string         &log &optional;
+	};
+
+	## Event that can be handled to access the Modbus record as it is sent on
+	## to the logging framework.
+	global log_modbus: event(rec: Info);
+}
+
+redef record connection += {
+	modbus: Info &optional;
+};
+
+# Configure DPD and the packet filter.
+redef capture_filters += { ["modbus"] = "tcp port 502" };
+redef dpd_config += { [ANALYZER_MODBUS] = [$ports = set(502/tcp)] };
+redef likely_server_ports += { 502/tcp };
+
+event bro_init() &priority=5
+	{
+	Log::create_stream(Modbus::LOG, [$columns=Info, $ev=log_modbus]);
+	}
+
+event modbus_message(c: connection, headers: ModbusHeaders, is_orig: bool) &priority=5
+	{
+	if ( ! c?$modbus )
+		{
+		c$modbus = [$ts=network_time(), $uid=c$uid, $id=c$id];
+		}
+
+	c$modbus$ts   = network_time();
+	c$modbus$func = function_codes[headers$function_code];
+	}
+
+event modbus_message(c: connection, headers: ModbusHeaders, is_orig: bool) &priority=-5
+	{
+	# Only log upon replies.
+	# Also, don't log now if this is an exception (log in the exception event handler)
+	if ( ! is_orig && ( headers$function_code <= 0x81 || headers$function_code >= 0x98 ) )
+		Log::write(LOG, c$modbus);
+	}
+
+event modbus_exception(c: connection, headers: ModbusHeaders, code: count) &priority=5
+	{
+	c$modbus$exception = exception_codes[code];
+	}
+
+event modbus_exception(c: connection, headers: ModbusHeaders, code: count) &priority=-5
+	{
+	Log::write(LOG, c$modbus);
+	delete c$modbus$exception;
+	}
+

scripts/base/utils/patterns.bro

@@ -1,5 +1,7 @@
 ##! Functions for creating and working with patterns.
 
+module GLOBAL;
+
 ## Given a pattern as a string with two tildes (~~) contained in it, it will
 ## return a pattern with string set's elements OR'd together where the
 ## double-tilde was given (this function only works at or before init time).

scripts/base/utils/urls.bro

@@ -0,0 +1,25 @@
+## Functions for URL handling.
+
+## A regular expression for matching and extracting URLs.
+const url_regex = /^([a-zA-Z\-]{3,5})(:\/\/[^\/?#"'\r\n><]*)([^?#"'\r\n><]*)([^[:blank:]\r\n"'><]*|\??[^"'\r\n><]*)/ &redef;
+
+## Extracts URLs discovered in arbitrary text.
+function find_all_urls(s: string): string_set
+	{
+	return find_all(s, url_regex);
+	}
+
+## Extracts URLs discovered in arbitrary text without
+## the URL scheme included.
+function find_all_urls_without_scheme(s: string): string_set
+	{
+	local urls = find_all_urls(s);
+	local return_urls: set[string] = set();
+	for ( url in urls )
+		{
+		local no_scheme = sub(url, /^([a-zA-Z\-]{3,5})(:\/\/)/, "");
+		add return_urls[no_scheme];
+		}
+
+	return return_urls;
+	}

scripts/policy/frameworks/intel/__load__.bro

@@ -0,0 +1,8 @@
+@load ./conn-established
+@load ./dns
+@load ./http-host-header
+@load ./http-url
+@load ./http-user-agents
+@load ./ssl
+@load ./smtp
+@load ./smtp-url-extraction

scripts/policy/frameworks/intel/conn-established.bro

@@ -0,0 +1,8 @@
+@load base/frameworks/intel
+@load ./where-locations
+
+event connection_established(c: connection)
+	{
+	Intel::seen([$host=c$id$orig_h, $conn=c, $where=Conn::IN_ORIG]);
+	Intel::seen([$host=c$id$resp_h, $conn=c, $where=Conn::IN_RESP]);
+	}

scripts/policy/frameworks/intel/dns.bro

@@ -0,0 +1,10 @@
+@load base/frameworks/intel
+@load ./where-locations
+
+event dns_request(c: connection, msg: dns_msg, query: string, qtype: count, qclass: count)
+	{
+	Intel::seen([$str=query,
+	             $str_type=Intel::DOMAIN,
+	             $conn=c,
+	             $where=DNS::IN_REQUEST]);
+	}

scripts/policy/frameworks/intel/http-host-header.bro

@@ -0,0 +1,11 @@
+@load base/frameworks/intel
+@load ./where-locations
+
+event http_header(c: connection, is_orig: bool, name: string, value: string)
+	{
+	if ( is_orig && name == "HOST" )
+		Intel::seen([$str=value,
+		             $str_type=Intel::DOMAIN,
+		             $conn=c,
+		             $where=HTTP::IN_HOST_HEADER]);
+	}

scripts/policy/frameworks/intel/http-url.bro

@@ -0,0 +1,12 @@
+@load base/frameworks/intel
+@load base/protocols/http/utils
+@load ./where-locations
+
+event http_message_done(c: connection, is_orig: bool, stat: http_message_stat)
+	{
+	if ( is_orig && c?$http )
+		Intel::seen([$str=HTTP::build_url(c$http),
+		             $str_type=Intel::URL,
+		             $conn=c,
+		             $where=HTTP::IN_URL]);
+	}

scripts/policy/frameworks/intel/http-user-agents.bro

@@ -0,0 +1,12 @@
+@load base/frameworks/intel
+@load ./where-locations
+
+event http_header(c: connection, is_orig: bool, name: string, value: string)
+	{
+	if ( is_orig && name == "USER-AGENT" )
+		Intel::seen([$str=value,
+		             $str_type=Intel::USER_AGENT,
+		             $conn=c,
+		             $where=HTTP::IN_USER_AGENT_HEADER]);
+	}
+

scripts/policy/frameworks/intel/smtp-url-extraction.bro

@@ -0,0 +1,15 @@
+@load base/frameworks/intel
+@load base/utils/urls
+@load ./where-locations
+
+event mime_segment_data(c: connection, length: count, data: string) &priority=3
+	{
+	local urls = find_all_urls_without_scheme(data);
+	for ( url in urls )
+		{
+		Intel::seen([$str=url,
+		             $str_type=Intel::URL,
+		             $conn=c,
+		             $where=SMTP::IN_MESSAGE]);
+		}
+	}

scripts/policy/frameworks/intel/smtp.bro

@@ -0,0 +1,71 @@
+@load base/frameworks/intel
+@load base/protocols/smtp
+@load ./where-locations
+
+event mime_end_entity(c: connection)
+	{
+	if ( c?$smtp )
+		{
+		if ( c$smtp?$path )
+			{
+			local path = c$smtp$path;
+			for ( i in path )
+				{
+				Intel::seen([$host=path[i],
+				             $conn=c,
+				             $where=SMTP::IN_RECEIVED_HEADER]);
+				}
+			}
+
+		if ( c$smtp?$user_agent )
+			Intel::seen([$str=c$smtp$user_agent,
+			             $str_type=Intel::USER_AGENT,
+			             $conn=c,
+			             $where=SMTP::IN_HEADER]);
+
+		if ( c$smtp?$x_originating_ip )
+			Intel::seen([$host=c$smtp$x_originating_ip,
+			             $conn=c,
+			             $where=SMTP::IN_X_ORIGINATING_IP_HEADER]);
+
+		if ( c$smtp?$mailfrom )
+			Intel::seen([$str=c$smtp$mailfrom,
+			             $str_type=Intel::EMAIL,
+			             $conn=c,
+			             $where=SMTP::IN_MAIL_FROM]);
+
+		if ( c$smtp?$rcptto )
+			{
+			for ( rcptto in c$smtp$rcptto )
+				{
+				Intel::seen([$str=rcptto,
+				             $str_type=Intel::EMAIL,
+				             $conn=c,
+				             $where=SMTP::IN_RCPT_TO]);
+				}
+			}
+
+		if ( c$smtp?$from )
+			Intel::seen([$str=c$smtp$from,
+			             $str_type=Intel::EMAIL,
+			             $conn=c,
+			             $where=SMTP::IN_FROM]);
+
+		if ( c$smtp?$to )
+			{
+			for ( email_to in c$smtp$to )
+				{
+				Intel::seen([$str=email_to,
+				             $str_type=Intel::EMAIL,
+				             $conn=c,
+				             $where=SMTP::IN_TO]);
+				}
+			}
+
+		if ( c$smtp?$reply_to )
+			Intel::seen([$str=c$smtp$reply_to,
+			             $str_type=Intel::EMAIL,
+			             $conn=c,
+			             $where=SMTP::IN_REPLY_TO]);
+		}
+	}

scripts/policy/frameworks/intel/ssl.bro

@@ -0,0 +1,34 @@
+@load base/frameworks/intel
+@load base/protocols/ssl
+@load ./where-locations
+
+event x509_certificate(c: connection, is_orig: bool, cert: X509, chain_idx: count, chain_len: count, der_cert: string)
+	{
+	if ( chain_idx == 0 )
+		{
+		if ( /emailAddress=/ in cert$subject )
+			{
+			local email = sub(cert$subject, /^.*emailAddress=/, "");
+			email = sub(email, /,.*$/, "");
+			Intel::seen([$str=email,
+			             $str_type=Intel::EMAIL,
+			             $conn=c,
+			             $where=(is_orig ? SSL::IN_CLIENT_CERT : SSL::IN_SERVER_CERT)]);
+			}
+
+		Intel::seen([$str=sha1_hash(der_cert),
+		             $str_type=Intel::CERT_HASH,
+		             $conn=c,
+		             $where=(is_orig ? SSL::IN_CLIENT_CERT : SSL::IN_SERVER_CERT)]);
+		}
+	}
+
+event ssl_extension(c: connection, is_orig: bool, code: count, val: string)
+	{
+	if ( is_orig && SSL::extensions[code] == "server_name" && 
+	     c?$ssl && c$ssl?$server_name )
+		Intel::seen([$str=c$ssl$server_name,
+		             $str_type=Intel::DOMAIN,
+		             $conn=c,
+		             $where=SSL::IN_SERVER_NAME]);
+	}

scripts/policy/frameworks/intel/where-locations.bro

@@ -0,0 +1,25 @@
+@load base/frameworks/intel
+
+export {
+	redef enum Intel::Where += {
+		Conn::IN_ORIG,
+		Conn::IN_RESP,
+		DNS::IN_REQUEST,
+		DNS::IN_RESPONSE,
+		HTTP::IN_HOST_HEADER,
+		HTTP::IN_USER_AGENT_HEADER,
+		HTTP::IN_URL,
+		SMTP::IN_MAIL_FROM,
+		SMTP::IN_RCPT_TO,
+		SMTP::IN_FROM,
+		SMTP::IN_TO,
+		SMTP::IN_RECEIVED_HEADER,
+		SMTP::IN_REPLY_TO,
+		SMTP::IN_X_ORIGINATING_IP_HEADER,
+		SMTP::IN_MESSAGE,
+		SSL::IN_SERVER_CERT,
+		SSL::IN_CLIENT_CERT,
+		SSL::IN_SERVER_NAME,
+		SMTP::IN_HEADER,
+	};
+}

scripts/policy/integration/collective-intel/README

@@ -0,0 +1,6 @@
+Collective Intelligence Framework Integration
+=============================================
+
+The scripts in this module are for deeper integration with the Collective Intelligence
+Framework (CIF) since Bro's Intel framework doesn't natively behave the same as CIF nor
+does it store and maintain the same data in all cases.

scripts/policy/integration/collective-intel/__load__.bro

@@ -0,0 +1 @@
+@load ./main

scripts/policy/integration/collective-intel/main.bro

@@ -0,0 +1,15 @@
+
+@load base/frameworks/intel
+
+module Intel;
+
+## These are some fields to add extended compatibility between Bro and the Collective
+## Intelligence Framework
+redef record Intel::MetaData += {
+	## Maps to the Impact field in the Collective Intelligence Framework.
+	cif_impact:     string &optional;
+	## Maps to the Severity field in the Collective Intelligence Framework.
+	cif_severity:   string &optional;
+	## Maps to the Confidence field in the Collective Intelligence Framework.
+	cif_confidence: double &optional;
+};

scripts/policy/protocols/http/detect-MHR.bro

@@ -1,4 +1,4 @@
-##! Detect file downloads over HTTP that have MD5 sums matching files in Team 
+##! Detect file downloads over HTTP that have MD5 sums matching files in Team
 ##! Cymru's Malware Hash Registry (http://www.team-cymru.org/Services/MHR/).
 ##! By default, not all file transfers will have MD5 sums calculated.  Read the
 ##! documentation for the :doc:base/protocols/http/file-hash.bro script to see
@@ -10,11 +10,17 @@
 module HTTP;
 
 export {
-	redef enum Notice::Type += { 
+	redef enum Notice::Type += {
 		## The MD5 sum of a file transferred over HTTP matched in the
 		## malware hash registry.
 		Malware_Hash_Registry_Match
 	};
+
+	## The malware hash registry runs each malware sample through several A/V engines.
+	## Team Cymru returns a percentage to indicate how many A/V engines flagged the
+	## sample as malicious. This threshold allows you to require a minimum detection
+	## rate (default: 50%).
+	const MHR_threshold = 50 &redef;
 }
 
 event log_http(rec: HTTP::Info)
@@ -22,14 +28,15 @@ event log_http(rec: HTTP::Info)
 	if ( rec?$md5 )
 		{
 		local hash_domain = fmt("%s.malware.hash.cymru.com", rec$md5);
-		when ( local addrs = lookup_hostname(hash_domain) )
+		when ( local MHR_result = lookup_hostname_txt(hash_domain) )
 			{
-			# 127.0.0.2 indicates that the md5 sum was found in the MHR.
+			# Data is returned as "<dateFirstDetected> <detectionRate>"
-			if ( 127.0.0.2 in addrs )
+			local MHR_answer = split1(MHR_result, / /);
+			if ( length(MHR_answer) == 2 && to_count(MHR_answer[2]) >= MHR_threshold )
 				{
 				local url = HTTP::build_url_http(rec);
 				local message = fmt("%s %s %s", rec$id$orig_h, rec$md5, url);
-				NOTICE([$note=Malware_Hash_Registry_Match, 
+				NOTICE([$note=Malware_Hash_Registry_Match,
 				        $msg=message, $id=rec$id, $URL=url]);
 				}
 			}

scripts/policy/protocols/http/detect-intel.bro

@@ -1,21 +0,0 @@
-##! Intelligence based HTTP detections.  Not yet working!
-
-@load base/protocols/http/main
-@load base/protocols/http/utils
-@load base/frameworks/intel/main
-
-module HTTP;
-
-event log_http(rec: Info)
-	{
-	local url = HTTP::build_url(rec);
-	local query = [$str=url, $subtype="url", $or_tags=set("malicious", "malware")];
-	if ( Intel::matcher(query) )
-		{
-		local msg = fmt("%s accessed a malicious URL from the intelligence framework", rec$id$orig_h);
-		NOTICE([$note=Intel::Detection, 
-		        $msg=msg, 
-		        $sub=HTTP::build_url_http(rec), 
-		        $id=rec$id]);
-		}
-	}

scripts/policy/protocols/modbus/known-masters-slaves.bro

@@ -0,0 +1,58 @@
+##! Script for tracking known Modbus masters and slaves.
+##!
+##! .. todo: This script needs a lot of work.  What might be more interesting is to track
+##!          master/slave relationships based on commands sent and successful (non-exception)
+##!          responses.
+
+@load base/protocols/modbus
+
+module Known;
+
+export {
+	redef enum Log::ID += { MODBUS_LOG };
+
+	type ModbusDeviceType: enum {
+		MODBUS_MASTER,
+		MODBUS_SLAVE,
+	};
+
+	type ModbusInfo: record {
+		## The time the device was discovered.
+		ts:          time             &log;
+		## The IP address of the host.
+		host:        addr             &log;
+		## The type of device being tracked.
+		device_type: ModbusDeviceType &log;
+	};
+
+	## The Modbus nodes being tracked.
+	global modbus_nodes: set[addr, ModbusDeviceType] &create_expire=1day &redef;
+
+	## Event that can be handled to access the loggable record as it is sent
+	## on to the logging framework.
+	global log_known_modbus: event(rec: ModbusInfo);
+}
+
+event bro_init() &priority=5
+	{
+	Log::create_stream(Known::MODBUS_LOG, [$columns=ModbusInfo, $ev=log_known_modbus]);
+	}
+
+event modbus_message(c: connection, headers: ModbusHeaders, is_orig: bool)
+	{
+	local master = c$id$orig_h;
+	local slave  = c$id$resp_h;
+
+	if ( [master, MODBUS_MASTER] !in modbus_nodes )
+		{
+		add modbus_nodes[master, MODBUS_MASTER];
+		Log::write(MODBUS_LOG, [$ts=network_time(), $host=master, $device_type=MODBUS_MASTER]);
+		}
+
+	if ( [slave, MODBUS_SLAVE] !in modbus_nodes )
+		{
+		add modbus_nodes[slave, MODBUS_SLAVE];
+		Log::write(MODBUS_LOG, [$ts=network_time(), $host=slave, $device_type=MODBUS_SLAVE]);
+		}
+
+	}

scripts/policy/protocols/modbus/track-memmap.bro

@@ -0,0 +1,104 @@
+##! This script tracks the memory map of holding (read/write) registers and logs
+##! changes as they are discovered.
+##!
+##! .. todo: Not all register reads and write functions are being supported yet.
+module Modbus;
+
+@load base/protocols/modbus
+@load base/utils/directions-and-hosts
+
+export {
+	redef enum Log::ID += { Modbus::REGISTER_CHANGE_LOG };
+
+	## The hosts that should have memory mapping enabled.
+	const track_memmap: Host = ALL_HOSTS &redef;
+
+	type MemmapInfo: record {
+		## Timestamp for the detected register change
+		ts:        time     &log;
+		## Unique ID for the connection
+		uid:       string   &log;
+		## Connection ID.
+		id:        conn_id  &log;
+		## The device memory offset.
+		register:  count    &log;
+		## The old value stored in the register.
+		old_val:   count    &log;
+		## The new value stored in the register.
+		new_val:   count    &log;
+		## The time delta between when the 'old_val' and 'new_val' were seen.
+		delta:     interval &log;
+	};
+
+	type RegisterValue: record {
+		last_set: time;
+		value:    count;
+	};
+
+	## Indexed on the device register value and yielding the register value.
+	type Registers: table[count] of RegisterValue;
+
+	## The memory map of slaves is tracked with this variable.
+	global device_registers: table[addr] of Registers;
+
+	## This event is generated every time a register is seen to be different than
+	## it was previously seen to be.
+	global changed_register: event(c: connection, register: count, old_val: count, new_val: count, delta: interval);
+}
+
+redef record Modbus::Info += {
+	track_address: count &default=0;
+};
+
+event bro_init() &priority=5
+	{
+	Log::create_stream(Modbus::REGISTER_CHANGE_LOG, [$columns=MemmapInfo]);
+	}
+
+event modbus_read_holding_registers_request(c: connection, headers: ModbusHeaders, start_address: count, quantity: count)
+	{
+	c$modbus$track_address = start_address+1;
+	}
+
+event modbus_read_holding_registers_response(c: connection, headers: ModbusHeaders, registers: ModbusRegisters)
+	{
+	local slave = c$id$resp_h;
+
+	if ( ! addr_matches_host(slave, track_memmap ) )
+		return;
+
+	if ( slave !in device_registers )
+		device_registers[slave] = table();
+
+	local slave_regs = device_registers[slave];
+	for ( i in registers )
+		{
+		if ( c$modbus$track_address in slave_regs )
+			{
+			if ( slave_regs[c$modbus$track_address]$value != registers[i] )
+				{
+				local delta = network_time() - slave_regs[c$modbus$track_address]$last_set;
+				event Modbus::changed_register(c, c$modbus$track_address, 
+				                               slave_regs[c$modbus$track_address]$value, registers[i],
+				                               delta);
+				
+				slave_regs[c$modbus$track_address]$last_set = network_time();
+				slave_regs[c$modbus$track_address]$value = registers[i];
+				}
+			}
+		else
+			{
+			local tmp_reg: RegisterValue = [$last_set=network_time(), $value=registers[i]];
+			slave_regs[c$modbus$track_address] = tmp_reg;
+			}
+
+		++c$modbus$track_address;
+		}
+	}
+
+event Modbus::changed_register(c: connection, register: count, old_val: count, new_val: count, delta: interval)
+	{
+	local rec: MemmapInfo = [$ts=network_time(), $uid=c$uid, $id=c$id, 
+	                         $register=register, $old_val=old_val, $new_val=new_val, $delta=delta];
+	Log::write(REGISTER_CHANGE_LOG, rec);
+	}

scripts/test-all-policy.bro

@@ -14,6 +14,16 @@
 # @load frameworks/control/controller.bro
 @load frameworks/dpd/detect-protocols.bro
 @load frameworks/dpd/packet-segment-logging.bro
+@load frameworks/intel/__load__.bro
+@load frameworks/intel/conn-established.bro
+@load frameworks/intel/dns.bro
+@load frameworks/intel/http-host-header.bro
+@load frameworks/intel/http-url.bro
+@load frameworks/intel/http-user-agents.bro
+@load frameworks/intel/smtp-url-extraction.bro
+@load frameworks/intel/smtp.bro
+@load frameworks/intel/ssl.bro
+@load frameworks/intel/where-locations.bro
 @load frameworks/metrics/conn-example.bro
 @load frameworks/metrics/http-example.bro
 @load frameworks/metrics/ssl-example.bro
@@ -22,6 +32,8 @@
 @load integration/barnyard2/__load__.bro
 @load integration/barnyard2/main.bro
 @load integration/barnyard2/types.bro
+@load integration/collective-intel/__load__.bro
+@load integration/collective-intel/main.bro
 @load misc/analysis-groups.bro
 @load misc/capture-loss.bro
 @load misc/loaded-scripts.bro
@@ -35,7 +47,6 @@
 @load protocols/dns/detect-external-names.bro
 @load protocols/ftp/detect.bro
 @load protocols/ftp/software.bro
-@load protocols/http/detect-intel.bro
 @load protocols/http/detect-MHR.bro
 @load protocols/http/detect-sqli.bro
 @load protocols/http/detect-webapps.bro
@@ -44,6 +55,8 @@
 @load protocols/http/software.bro
 @load protocols/http/var-extraction-cookies.bro
 @load protocols/http/var-extraction-uri.bro
+@load protocols/modbus/known-masters-slaves.bro
+@load protocols/modbus/track-memmap.bro
 @load protocols/smtp/blocklists.bro
 @load protocols/smtp/detect-suspicious-orig.bro
 @load protocols/smtp/software.bro

src/Analyzer.cc

@@ -28,6 +28,7 @@
 #include "DCE_RPC.h"
 #include "Gnutella.h"
 #include "Ident.h"
+#include "Modbus.h"
 #include "NCP.h"
 #include "NetbiosSSN.h"
 #include "SMB.h"
@@ -129,6 +130,9 @@ const Analyzer::Config Analyzer::analyzer_configs[] = {
 	{ AnalyzerTag::SYSLOG_BINPAC, "SYSLOG_BINPAC",
 		Syslog_Analyzer_binpac::InstantiateAnalyzer,
 		Syslog_Analyzer_binpac::Available, 0, false },
+	{ AnalyzerTag::Modbus, "MODBUS",
+		ModbusTCP_Analyzer::InstantiateAnalyzer,
+		ModbusTCP_Analyzer::Available, 0, false },
 
 	{ AnalyzerTag::AYIYA, "AYIYA",
 		AYIYA_Analyzer::InstantiateAnalyzer,

src/AnalyzerTags.h

@@ -32,6 +32,7 @@ namespace AnalyzerTag {
 		// Application-layer analyzers, binpac-generated.
 		DHCP_BINPAC, DNS_TCP_BINPAC, DNS_UDP_BINPAC,
 		HTTP_BINPAC, SSL, SYSLOG_BINPAC,
+		Modbus,
 
 		// Decapsulation analyzers.
 		AYIYA,

src/BroString.h

@@ -34,11 +34,15 @@ public:
 	typedef IdxVec::iterator IdxVecIt;
 	typedef IdxVec::const_iterator IdxVecCIt;
 
-	BroString(int arg_final_NUL, byte_vec str, int arg_n);
+	// Constructors creating internal copies of the data passed in.
 	BroString(const u_char* str, int arg_n, int add_NUL);
 	BroString(const char* str);
 	BroString(const string& str);
 	BroString(const BroString& bs);
+
+	// Constructor that takes owernship of the vector passed in.
+	BroString(int arg_final_NUL, byte_vec str, int arg_n);
+
 	BroString();
 	~BroString()	{ Reset(); }
 

src/CMakeLists.txt

@@ -216,6 +216,8 @@ binpac_target(ssl.pac
     ssl-defs.pac ssl-protocol.pac ssl-analyzer.pac)
 binpac_target(syslog.pac
     syslog-protocol.pac syslog-analyzer.pac)
+binpac_target(modbus.pac
+    modbus-protocol.pac modbus-analyzer.pac)
 
 ########################################################################
 ## bro target
@@ -346,6 +348,7 @@ set(bro_SRCS
     Reporter.cc
     Login.cc
     MIME.cc
+    Modbus.cc
     NCP.cc
     NFA.cc
     NFS.cc

src/DNS_Mgr.cc

@@ -46,13 +46,15 @@ extern int select(int, fd_set *, fd_set *, fd_set *, struct timeval *);
 
 class DNS_Mgr_Request {
 public:
-	DNS_Mgr_Request(const char* h, int af)	{ host = copy_string(h); fam = af; }
+	DNS_Mgr_Request(const char* h, int af, bool is_txt)
+		{ host = copy_string(h); fam = af; qtype = is_txt ? 16 : 0; }
 	DNS_Mgr_Request(const IPAddr& a)		{ addr = a; host = 0; fam = 0; }
 	~DNS_Mgr_Request()			{ delete [] host; }
 
 	// Returns nil if this was an address request.
 	const char* ReqHost() const	{ return host; }
 	const IPAddr& ReqAddr() const		{ return addr; }
+	const bool ReqIsTxt() const	{ return qtype == 16; }
 
 	int MakeRequest(nb_dns_info* nb_dns);
 	int RequestPending() const	{ return request_pending; }
@@ -61,7 +63,8 @@ public:
 
 protected:
 	char* host;	// if non-nil, this is a host request
-	int fam; // address family query type for host requests
+	int fam;	// address family query type for host requests
+	int qtype;	// Query type
 	IPAddr addr;
 	int request_pending;
 };
@@ -75,7 +78,7 @@ int DNS_Mgr_Request::MakeRequest(nb_dns_info* nb_dns)
 
 	char err[NB_DNS_ERRSIZE];
 	if ( host )
-		return nb_dns_host_request2(nb_dns, host, fam, (void*) this, err) >= 0;
+		return nb_dns_host_request2(nb_dns, host, fam, qtype, (void*) this, err) >= 0;
 	else
 		{
 		const uint32* bytes;
@@ -475,8 +478,8 @@ TableVal* DNS_Mgr::LookupHost(const char* name)
 	// Not found, or priming.
 	switch ( mode ) {
 	case DNS_PRIME:
-		requests.append(new DNS_Mgr_Request(name, AF_INET));
+		requests.append(new DNS_Mgr_Request(name, AF_INET, false));
-		requests.append(new DNS_Mgr_Request(name, AF_INET6));
+		requests.append(new DNS_Mgr_Request(name, AF_INET6, false));
 		return empty_addr_set();
 
 	case DNS_FORCE:
@@ -484,8 +487,8 @@ TableVal* DNS_Mgr::LookupHost(const char* name)
 		return 0;
 
 	case DNS_DEFAULT:
-		requests.append(new DNS_Mgr_Request(name, AF_INET));
+		requests.append(new DNS_Mgr_Request(name, AF_INET, false));
-		requests.append(new DNS_Mgr_Request(name, AF_INET6));
+		requests.append(new DNS_Mgr_Request(name, AF_INET6, false));
 		Resolve();
 		return LookupHost(name);
 
@@ -636,6 +639,7 @@ int DNS_Mgr::Save()
 
 	Save(f, host_mappings);
 	Save(f, addr_mappings);
+	// Save(f, text_mappings); // We don't save the TXT mappings (yet?).
 
 	fclose(f);
 
@@ -704,39 +708,48 @@ void DNS_Mgr::AddResult(DNS_Mgr_Request* dr, struct nb_dns_result* r)
 		new_dm = new DNS_Mapping(dr->ReqHost(), h, ttl);
 		prev_dm = 0;
 
-		HostMap::iterator it = host_mappings.find(dr->ReqHost());
+		if ( dr->ReqIsTxt() )
-		if ( it == host_mappings.end() )
 			{
-			host_mappings[dr->ReqHost()].first =
+			TextMap::iterator it = text_mappings.find(dr->ReqHost());
-					new_dm->Type() == AF_INET ? new_dm : 0;
+			if ( it == text_mappings.end() )
-			host_mappings[dr->ReqHost()].second =
+				text_mappings[dr->ReqHost()] = new_dm;
-					new_dm->Type() == AF_INET ? 0 : new_dm;
 			}
-
 		else
 			{
-			if ( new_dm->Type() == AF_INET )
+			HostMap::iterator it = host_mappings.find(dr->ReqHost());
+			if ( it == host_mappings.end() )
 				{
-				prev_dm = it->second.first;
+				host_mappings[dr->ReqHost()].first =
-				it->second.first = new_dm;
+					new_dm->Type() == AF_INET ? new_dm : 0;
+
+				host_mappings[dr->ReqHost()].second =
+					new_dm->Type() == AF_INET ? 0 : new_dm;
 				}
 			else
 				{
-				prev_dm = it->second.second;
+				if ( new_dm->Type() == AF_INET )
-				it->second.second = new_dm;
+					{
+					prev_dm = it->second.first;
+					it->second.first = new_dm;
+					}
+				else
+					{
+					prev_dm = it->second.second;
+					it->second.second = new_dm;
+					}
 				}
-			}
 
-		if ( new_dm->Failed() && prev_dm && prev_dm->Valid() )
+			if ( new_dm->Failed() && prev_dm && prev_dm->Valid() )
-			{
+				{
-			// Put previous, valid entry back - CompareMappings
+				// Put previous, valid entry back - CompareMappings
-			// will generate a corresponding warning.
+				// will generate a corresponding warning.
-			if ( prev_dm->Type() == AF_INET )
+				if ( prev_dm->Type() == AF_INET )
-				host_mappings[dr->ReqHost()].first = prev_dm;
+					host_mappings[dr->ReqHost()].first = prev_dm;
-			else
+				else
-				host_mappings[dr->ReqHost()].second = prev_dm;
+					host_mappings[dr->ReqHost()].second = prev_dm;
 
-			++keep_prev;
+				++keep_prev;
+				}
 			}
 		}
 	else
@@ -928,7 +941,10 @@ TableVal* DNS_Mgr::LookupNameInCache(string name)
 	{
 	HostMap::iterator it = dns_mgr->host_mappings.find(name);
 	if ( it == dns_mgr->host_mappings.end() )
+		{
+		it = dns_mgr->host_mappings.begin();
 		return 0;
+		}
 
 	DNS_Mapping* d4 = it->second.first;
 	DNS_Mapping* d6 = it->second.second;
@@ -951,6 +967,26 @@ TableVal* DNS_Mgr::LookupNameInCache(string name)
 	return tv6;
 	}
 
+const char* DNS_Mgr::LookupTextInCache(string name)
+	{
+	TextMap::iterator it = dns_mgr->text_mappings.find(name);
+	if ( it == dns_mgr->text_mappings.end() )
+		return 0;
+
+	DNS_Mapping* d = it->second;
+
+	if ( d->Expired() )
+		{
+		dns_mgr->text_mappings.erase(it);
+		delete d;
+		return 0;
+		}
+
+	// The escapes in the following strings are to avoid having it
+	// interpreted as a trigraph sequence.
+	return d->names ? d->names[0] : "<\?\?\?>";
+	}
+
 void DNS_Mgr::AsyncLookupAddr(const IPAddr& host, LookupCallback* callback)
 	{
 	if ( ! did_init )
@@ -976,6 +1012,7 @@ void DNS_Mgr::AsyncLookupAddr(const IPAddr& host, LookupCallback* callback)
 		// A new one.
 		req = new AsyncRequest;
 		req->host = host;
+		req->is_txt = false;
 		asyncs_queued.push_back(req);
 		asyncs_addrs.insert(AsyncRequestAddrMap::value_type(host, req));
 		}
@@ -1020,6 +1057,35 @@ void DNS_Mgr::AsyncLookupName(string name, LookupCallback* callback)
 	IssueAsyncRequests();
 	}
 
+void DNS_Mgr::AsyncLookupNameText(string name, LookupCallback* callback)
+	{
+	if ( ! did_init )
+		Init();
+
+	// Do we already know the answer?
+	TableVal* addrs;
+
+	AsyncRequest* req = 0;
+
+	// Have we already a request waiting for this host?
+	AsyncRequestTextMap::iterator i = asyncs_texts.find(name);
+	if ( i != asyncs_texts.end() )
+		req = i->second;
+	else
+		{
+		// A new one.
+		req = new AsyncRequest;
+		req->name = name;
+		req->is_txt = true;
+		asyncs_queued.push_back(req);
+		asyncs_texts.insert(AsyncRequestTextMap::value_type(name, req));
+		}
+
+	req->callbacks.push_back(callback);
+
+	IssueAsyncRequests();
+	}
+
 void DNS_Mgr::IssueAsyncRequests()
 	{
 	while ( asyncs_queued.size() && asyncs_pending < MAX_PENDING_REQUESTS )
@@ -1036,8 +1102,9 @@ void DNS_Mgr::IssueAsyncRequests()
 			dr = new DNS_Mgr_Request(req->host);
 		else
 			{
-			dr = new DNS_Mgr_Request(req->name.c_str(), AF_INET);
+			dr = new DNS_Mgr_Request(req->name.c_str(), AF_INET, req->is_txt);
-			dr6 = new DNS_Mgr_Request(req->name.c_str(), AF_INET6);
+			if ( ! req->is_txt )
+				dr6 = new DNS_Mgr_Request(req->name.c_str(), AF_INET6, req->is_txt);
 			}
 
 		if ( ! dr->MakeRequest(nb_dns) )
@@ -1109,6 +1176,38 @@ void DNS_Mgr::CheckAsyncAddrRequest(const IPAddr& addr, bool timeout)
 
 	}
 
+void DNS_Mgr::CheckAsyncTextRequest(const char* host, bool timeout)
+	{
+	// Note that this code is a mirror of that for CheckAsyncAddrRequest.
+
+	AsyncRequestTextMap::iterator i = asyncs_texts.find(host);
+	if ( i != asyncs_texts.end() )
+		{
+		const char* name = LookupTextInCache(host);
+		if ( name )
+			{
+			++successful;
+			i->second->Resolved(name);
+			}
+
+		else if ( timeout )
+			{
+			AsyncRequestTextMap::iterator it = asyncs_texts.begin();
+			++failed;
+			i->second->Timeout();
+			}
+
+		else
+			return;
+
+		asyncs_texts.erase(i);
+		--asyncs_pending;
+
+		// Don't delete the request.  That will be done once it
+		// eventually times out.
+		}
+	}
+
 void DNS_Mgr::CheckAsyncHostRequest(const char* host, bool timeout)
 	{
 	// Note that this code is a mirror of that for CheckAsyncAddrRequest.
@@ -1157,8 +1256,12 @@ void DNS_Mgr::Flush()
 	for ( AddrMap::iterator it2 = addr_mappings.begin(); it2 != addr_mappings.end(); ++it2 )
 		delete it2->second;
 
+	for ( TextMap::iterator it3 = text_mappings.begin(); it3 != text_mappings.end(); ++it3 )
+		delete it3->second;
+
 	host_mappings.clear();
 	addr_mappings.clear();
+	text_mappings.clear();
 	}
 
 void DNS_Mgr::Process()
@@ -1177,6 +1280,8 @@ void DNS_Mgr::DoProcess(bool flush)
 
 		if ( req->IsAddrReq() )
 			CheckAsyncAddrRequest(req->host, true);
+		else if ( req->is_txt )
+			CheckAsyncTextRequest(req->name.c_str(), true);
 		else
 			CheckAsyncHostRequest(req->name.c_str(), true);
 
@@ -1184,7 +1289,7 @@ void DNS_Mgr::DoProcess(bool flush)
 		delete req;
 		}
 
-	if ( asyncs_addrs.size() == 0 && asyncs_names.size() == 0 )
+	if ( asyncs_addrs.size() == 0 && asyncs_names.size() == 0 && asyncs_texts.size() == 0 )
 		return;
 
 	if ( AnswerAvailable(0) <= 0 )
@@ -1217,6 +1322,8 @@ void DNS_Mgr::DoProcess(bool flush)
 
 		if ( ! dr->ReqHost() )
 			CheckAsyncAddrRequest(dr->ReqAddr(), true);
+		else if ( dr->ReqIsTxt() )
+			CheckAsyncTextRequest(dr->ReqHost(), do_host_timeout);
 		else
 			CheckAsyncHostRequest(dr->ReqHost(), do_host_timeout);
 
@@ -1271,5 +1378,6 @@ void DNS_Mgr::GetStats(Stats* stats)
 	stats->pending = asyncs_pending;
 	stats->cached_hosts = host_mappings.size();
 	stats->cached_addresses = addr_mappings.size();
+	stats->cached_texts = text_mappings.size();
 	}
 

src/DNS_Mgr.h

@@ -63,6 +63,7 @@ public:
 
 	const char* LookupAddrInCache(const IPAddr& addr);
 	TableVal* LookupNameInCache(string name);
+	const char* LookupTextInCache(string name);
 
 	// Support for async lookups.
 	class LookupCallback {
@@ -77,6 +78,7 @@ public:
 
 	void AsyncLookupAddr(const IPAddr& host, LookupCallback* callback);
 	void AsyncLookupName(string name, LookupCallback* callback);
+	void AsyncLookupNameText(string name, LookupCallback* callback);
 
 	struct Stats {
 		unsigned long requests;	// These count only async requests.
@@ -85,6 +87,7 @@ public:
 		unsigned long pending;
 		unsigned long cached_hosts;
 		unsigned long cached_addresses;
+		unsigned long cached_texts;
 	};
 
 	void GetStats(Stats* stats);
@@ -106,6 +109,7 @@ protected:
 
 	typedef map<string, pair<DNS_Mapping*, DNS_Mapping*> > HostMap;
 	typedef map<IPAddr, DNS_Mapping*> AddrMap;
+	typedef map<string, DNS_Mapping*> TextMap;
 	void LoadCache(FILE* f);
 	void Save(FILE* f, const AddrMap& m);
 	void Save(FILE* f, const HostMap& m);
@@ -122,6 +126,7 @@ protected:
 	// requested.
 	void CheckAsyncAddrRequest(const IPAddr& addr, bool timeout);
 	void CheckAsyncHostRequest(const char* host, bool timeout);
+	void CheckAsyncTextRequest(const char* host, bool timeout);
 
 	// Process outstanding requests.
 	void DoProcess(bool flush);
@@ -138,6 +143,7 @@ protected:
 
 	HostMap host_mappings;
 	AddrMap addr_mappings;
+	TextMap text_mappings;
 
 	DNS_mgr_request_list requests;
 
@@ -165,6 +171,7 @@ protected:
 		double time;
 		IPAddr host;
 		string name;
+		bool is_txt;
 		CallbackList callbacks;
 
 		bool IsAddrReq() const	{ return name.length() == 0; }
@@ -210,6 +217,9 @@ protected:
 	typedef map<string, AsyncRequest*> AsyncRequestNameMap;
 	AsyncRequestNameMap asyncs_names;
 
+	typedef map<string, AsyncRequest*> AsyncRequestTextMap;
+	AsyncRequestTextMap asyncs_texts;
+
 	typedef list<AsyncRequest*> QueuedList;
 	QueuedList asyncs_queued;
 

src/IPAddr.cc

@@ -248,10 +248,10 @@ IPPrefix::IPPrefix(const in6_addr& in6, uint8_t length)
 	prefix.Mask(this->length);
 	}
 
-IPPrefix::IPPrefix(const IPAddr& addr, uint8_t length)
+IPPrefix::IPPrefix(const IPAddr& addr, uint8_t length, bool len_is_v6_relative)
 	: prefix(addr)
 	{
-	if ( prefix.GetFamily() == IPv4 )
+	if ( prefix.GetFamily() == IPv4 && ! len_is_v6_relative )
 		{
 		if ( length > 32 )
 			reporter->InternalError("Bad IPAddr(v4) IPPrefix length : %d",

src/IPAddr.h

@@ -496,8 +496,15 @@ public:
 	 * @param addr The IP address.
 	 *
 	 * @param length The prefix length in the range from 0 to 128
+	 *
+	 * @param len_is_v6_relative Whether \a length is relative to the full
+	 * 128 bits of an IPv6 address.  If false and \a addr is an IPv4
+	 * address, then \a length is expected to range from 0 to 32.  If true
+	 * \a length is expected to range from 0 to 128 even if \a addr is IPv4,
+	 * meaning that the mask is to apply to the IPv4-mapped-IPv6 representation.
 	 */
-	IPPrefix(const IPAddr& addr, uint8_t length);
+	IPPrefix(const IPAddr& addr, uint8_t length,
+	         bool len_is_v6_relative = false);
 
 	/**
 	 * Copy constructor.

src/Modbus.cc

@@ -0,0 +1,41 @@
+
+#include "Modbus.h"
+#include "TCP_Reassembler.h"
+
+ModbusTCP_Analyzer::ModbusTCP_Analyzer(Connection* c)
+	: TCP_ApplicationAnalyzer(AnalyzerTag::Modbus, c)
+	{
+	interp = new binpac::ModbusTCP::ModbusTCP_Conn(this);
+	}
+
+ModbusTCP_Analyzer::~ModbusTCP_Analyzer()
+	{
+	delete interp;
+	}
+
+void ModbusTCP_Analyzer::Done()
+	{
+	TCP_ApplicationAnalyzer::Done();
+
+	interp->FlowEOF(true);
+	interp->FlowEOF(false);
+	}
+
+void ModbusTCP_Analyzer::DeliverStream(int len, const u_char* data, bool orig)
+	{
+	TCP_ApplicationAnalyzer::DeliverStream(len, data, orig);
+	interp->NewData(orig, data, data + len);
+	}
+
+void ModbusTCP_Analyzer::Undelivered(int seq, int len, bool orig)
+	{
+	TCP_ApplicationAnalyzer::Undelivered(seq, len, orig);
+	interp->NewGap(orig, len);
+	}
+
+void ModbusTCP_Analyzer::EndpointEOF(bool is_orig)
+	{
+	TCP_ApplicationAnalyzer::EndpointEOF(is_orig);
+	interp->FlowEOF(is_orig);
+	}
+

src/Modbus.h

@@ -0,0 +1,58 @@
+#ifndef MODBUS_H
+#define MODBUS_H
+
+#include "TCP.h"
+#include "modbus_pac.h"
+
+class ModbusTCP_Analyzer : public TCP_ApplicationAnalyzer {
+public:
+	ModbusTCP_Analyzer(Connection* conn);
+	virtual ~ModbusTCP_Analyzer();
+
+	virtual void Done();
+	virtual void DeliverStream(int len, const u_char* data, bool orig);
+
+	virtual void Undelivered(int seq, int len, bool orig);
+	virtual void EndpointEOF(bool is_orig);
+
+	static Analyzer* InstantiateAnalyzer(Connection* conn)
+		{ return new ModbusTCP_Analyzer(conn); }
+
+	// Put event names in this function
+	static bool Available()
+		{
+		return modbus_message
+		     | modbus_exception
+		     | modbus_read_coils_request
+		     | modbus_read_coils_response
+		     | modbus_read_discrete_inputs_request
+		     | modbus_read_discrete_inputs_response
+		     | modbus_read_holding_registers_request
+		     | modbus_read_holding_registers_response
+		     | modbus_read_input_registers_request
+		     | modbus_read_input_registers_response
+		     | modbus_write_single_coil_request
+		     | modbus_write_single_coil_response
+		     | modbus_write_single_register_request
+		     | modbus_write_single_register_response
+		     | modbus_write_multiple_coils_request
+		     | modbus_write_multiple_coils_response
+		     | modbus_write_multiple_registers_request
+		     | modbus_write_multiple_registers_response
+		     | modbus_read_file_record_request
+		     | modbus_read_file_record_response
+		     | modbus_write_file_record_request
+		     | modbus_write_file_record_response
+		     | modbus_mask_write_register_request
+		     | modbus_mask_write_register_response
+		     | modbus_read_write_multiple_registers_request
+		     | modbus_read_write_multiple_registers_response
+		     | modbus_read_fifo_queue_request
+		     | modbus_read_fifo_queue_response;
+		}
+
+protected:
+	binpac::ModbusTCP::ModbusTCP_Conn* interp;
+};
+
+#endif

src/PktSrc.cc

@@ -219,16 +219,35 @@ void PktSrc::Process()
 		// Get protocol being carried from the ethernet frame.
 		protocol = (data[12] << 8) + data[13];
 
-		// MPLS carried over the ethernet frame.
+		switch ( protocol )
-		if ( protocol == 0x8847 )
-			have_mpls = true;
-
-		// VLAN carried over ethernet frame.
-		else if ( protocol == 0x8100 )
 			{
-			data += get_link_header_size(datalink);
+			// MPLS carried over the ethernet frame.
-			data += 4; // Skip the vlan header
+			case 0x8847:
-			pkt_hdr_size = 0;
+				have_mpls = true;
+				break;
+
+			// VLAN carried over the ethernet frame.
+			case 0x8100:
+				data += get_link_header_size(datalink);
+				data += 4; // Skip the vlan header
+				pkt_hdr_size = 0;
+				break;
+
+			// PPPoE carried over the ethernet frame.
+			case 0x8864:
+				data += get_link_header_size(datalink);
+				protocol = (data[6] << 8) + data[7];
+				data += 8; // Skip the PPPoE session and PPP header
+				pkt_hdr_size = 0;
+
+				if ( protocol != 0x0021 && protocol != 0x0057 )
+					{
+					// Neither IPv4 nor IPv6.
+					sessions->Weird("non_ip_packet_in_pppoe_encapsulation", &hdr, data);
+					data = 0;
+					return;
+					}
+				break;
 			}
 
 		break;

src/bro.bif

@@ -3770,6 +3770,35 @@ function lookup_addr%(host: addr%) : string
 	return 0;
 	%}
 
+## Issues an asynchronous TEXT DNS lookup and delays the function result.
+## This function can therefore only be called inside a ``when`` condition,
+## e.g., ``when ( local h = lookup_hostname_txt("www.bro-ids.org") ) { f(h); }``.
+##
+## host: The hostname to lookup.
+##
+## Returns: The DNS TXT record associated with *host*.
+##
+## .. bro:see:: lookup_hostname
+function lookup_hostname_txt%(host: string%) : string
+	%{
+	// FIXME: Is should be easy to adapt the function to synchronous
+	// lookups if we're reading a trace.
+	Trigger* trigger = frame->GetTrigger();
+
+	if ( ! trigger)
+		{
+		builtin_error("lookup_hostname_txt() can only be called inside a when-condition");
+		return new StringVal("<error>");
+		}
+
+	frame->SetDelayed();
+	trigger->Hold();
+
+	dns_mgr->AsyncLookupNameText(host->CheckString(),
+				 new LookupHostCallback(trigger, frame->GetCall(), true));
+	return 0;
+	%}
+
 ## Issues an asynchronous DNS lookup and delays the function result.
 ## This function can therefore only be called inside a ``when`` condition,
 ## e.g., ``when ( local h = lookup_hostname("www.bro-ids.org") ) { f(h); }``.

src/event.bif

@@ -150,7 +150,7 @@ event new_connection%(c: connection%);
 ## encapsulation value of *e* after this event is raised.  If the desired
 ## behavior is to track the latest tunnel encapsulation per-connection,
 ## then a handler of this event should assign *e* to ``c$tunnel`` (which Bro's
-## default scripts are doing). 
+## default scripts are doing).
 ##
 ## c: The connection whose tunnel/encapsulation changed.
 ##
@@ -6547,6 +6547,301 @@ event netflow_v5_header%(h: nf_v5_header%);
 ## .. bro:see:: netflow_v5_record
 event netflow_v5_record%(r: nf_v5_record%);
 
+## Generated for any modbus message regardless if the particular function
+## is further supported or not.
+##
+## c: The connection.
+##
+## headers: The headers for the modbus function.
+##
+## is_orig: True if the event is raised for the originator side.
+event modbus_message%(c: connection, headers: ModbusHeaders, is_orig: bool%);
+
+## Generated for any modbus exception message.
+##
+## c: The connection.
+##
+## headers: The headers for the modbus function.
+##
+## code: The exception code.
+event modbus_exception%(c: connection, headers: ModbusHeaders, code: count%);
+
+## Generated for a Modbus read coils request.
+##
+## c: The connection.
+##
+## headers: The headers for the modbus function.
+##
+## start_address: The memory address where of the first coil to be read.
+##
+## quantity: The number of coils to be read.
+event modbus_read_coils_request%(c: connection, headers: ModbusHeaders, start_address: count, quantity: count%);
+
+## Generated for a Modbus read coils response.
+##
+## c: The connection.
+##
+## headers: The headers for the modbus function.
+##
+## coils: The coil values returned from the device.
+event modbus_read_coils_response%(c: connection, headers: ModbusHeaders, coils: ModbusCoils%);
+
+## Generated for a Modbus read discrete inputs request.
+##
+## c: The connection.
+##
+## headers: The headers for the modbus function.
+##
+## start_address: The memory address of the first coil to be read.
+##
+## quantity: The number of coils to be read.
+event modbus_read_discrete_inputs_request%(c: connection, headers: ModbusHeaders, start_address: count, quantity: count%);
+
+## Generated for a Modbus read discrete inputs response.
+##
+## c: The connection.
+##
+## headers: The headers for the modbus function.
+##
+## coils: The coil values returned from the device.
+event modbus_read_discrete_inputs_response%(c: connection, headers: ModbusHeaders, coils: ModbusCoils%);
+
+## Generated for a Modbus read holding registers request.
+##
+## c: The connection.
+##
+## headers: The headers for the modbus function.
+##
+## start_address: The memory address of the first register to be read.
+##
+## quantity: The number of registers to be read.
+event modbus_read_holding_registers_request%(c: connection, headers: ModbusHeaders, start_address: count, quantity: count%);
+
+## Generated for a Modbus read holding registers response.
+##
+## c: The connection.
+##
+## headers: The headers for the modbus function.
+##
+## registers: The register values returned from the device.
+event modbus_read_holding_registers_response%(c: connection, headers: ModbusHeaders, registers: ModbusRegisters%);
+
+## Generated for a Modbus read input registers request.
+##
+## c: The connection.
+##
+## headers: The headers for the modbus function.
+##
+## start_address: The memory address of the first register to be read.
+##
+## quantity: The number of registers to be read.
+event modbus_read_input_registers_request%(c: connection, headers: ModbusHeaders, start_address: count, quantity: count%);
+
+## Generated for a Modbus read input registers response.
+##
+## c: The connection.
+##
+## headers: The headers for the modbus function.
+##
+## registers: The register values returned from the device.
+event modbus_read_input_registers_response%(c: connection, headers: ModbusHeaders, registers: ModbusRegisters%);
+
+## Generated for a Modbus write single coil request.
+##
+## c: The connection.
+##
+## headers: The headers for the modbus function.
+##
+## address: The memory address of the coil to be written.
+##
+## value: The value to be written to the coil.
+event modbus_write_single_coil_request%(c: connection, headers: ModbusHeaders, address: count, value: bool%);
+
+## Generated for a Modbus write single coil response.
+##
+## c: The connection.
+##
+## headers: The headers for the modbus function.
+##
+## address: The memory address of the coil that was written.
+##
+## value: The value that was written to the coil.
+event modbus_write_single_coil_response%(c: connection, headers: ModbusHeaders, address: count, value: bool%);
+
+## Generated for a Modbus write single register request.
+##
+## c: The connection.
+##
+## headers: The headers for the modbus function.
+##
+## address: The memory address of the register to be written.
+##
+## value: The value to be written to the register.
+event modbus_write_single_register_request%(c: connection, headers: ModbusHeaders, address: count, value: count%);
+
+## Generated for a Modbus write single register response.
+##
+## c: The connection.
+##
+## headers: The headers for the modbus function.
+##
+## address: The memory address of the register that was written.
+##
+## value: The value that was written to the register.
+event modbus_write_single_register_response%(c: connection, headers: ModbusHeaders, address: count, value: count%);
+
+## Generated for a Modbus write multiple coils request.
+##
+## c: The connection.
+##
+## headers: The headers for the modbus function.
+##
+## start_address: The memory address of the first coil to be written.
+##
+## value: The values to be written to the coils.
+event modbus_write_multiple_coils_request%(c: connection, headers: ModbusHeaders, start_address: count, coils: ModbusCoils%);
+
+## Generated for a Modbus write multiple coils response.
+##
+## c: The connection.
+##
+## headers: The headers for the modbus function.
+##
+## start_address: The memory address of the first coil that was written.
+##
+## quantity: The quantity of coils that were written.
+event modbus_write_multiple_coils_response%(c: connection, headers: ModbusHeaders, start_address: count, quantity: count%);
+
+## Generated for a Modbus write multiple registers request.
+##
+## c: The connection.
+##
+## headers: The headers for the modbus function.
+##
+## start_address: The memory address of the first register to be written.
+##
+## registers: The values to be written to the registers.
+event modbus_write_multiple_registers_request%(c: connection, headers: ModbusHeaders, start_address: count, registers: ModbusRegisters%);
+
+## Generated for a Modbus write multiple registers response.
+##
+## c: The connection.
+##
+## headers: The headers for the modbus function.
+##
+## start_address: The memory address of the first register that was written.
+##
+## quantity: The quantity of registers that were written.
+event modbus_write_multiple_registers_response%(c: connection, headers: ModbusHeaders, start_address: count, quantity: count%);
+
+## Generated for a Modbus read file record request.
+##
+## c: The connection.
+##
+## headers: The headers for the modbus function.
+##
+## .. note: This event is incomplete.  The information from the data structure is not
+##          yet passed through to the event.
+event modbus_read_file_record_request%(c: connection, headers: ModbusHeaders%);
+
+## Generated for a Modbus read file record response.
+##
+## c: The connection.
+##
+## headers: The headers for the modbus function.
+##
+## .. note: This event is incomplete.  The information from the data structure is not
+##          yet passed through to the event.
+event modbus_read_file_record_response%(c: connection, headers: ModbusHeaders%);
+
+## Generated for a Modbus write file record request.
+##
+## c: The connection.
+##
+## headers: The headers for the modbus function.
+##
+## .. note: This event is incomplete.  The information from the data structure is not
+##          yet passed through to the event.
+event modbus_write_file_record_request%(c: connection, headers: ModbusHeaders%);
+
+## Generated for a Modbus write file record response.
+##
+## c: The connection.
+##
+## headers: The headers for the modbus function.
+##
+## .. note: This event is incomplete.  The information from the data structure is not
+##          yet passed through to the event.
+event modbus_write_file_record_response%(c: connection, headers: ModbusHeaders%);
+
+## Generated for a Modbus mask write register request.
+##
+## c: The connection.
+##
+## headers: The headers for the modbus function.
+##
+## address: The memory address of the register where the masks should be applied.
+##
+## and_mask: The value of the logical AND mask to apply to the register.
+##
+## or_mask: The value of the logical OR mask to apply to the register.
+event modbus_mask_write_register_request%(c: connection, headers: ModbusHeaders, address: count, and_mask: count, or_mask: count%);
+
+## Generated for a Modbus mask write register request.
+##
+## c: The connection.
+##
+## headers: The headers for the modbus function.
+##
+## address: The memory address of the register where the masks were applied.
+##
+## and_mask: The value of the logical AND mask applied register.
+##
+## or_mask: The value of the logical OR mask applied to the register.
+event modbus_mask_write_register_response%(c: connection, headers: ModbusHeaders, address: count, and_mask: count, or_mask: count%);
+
+## Generated for a Modbus read/write multiple registers request.
+##
+## c: The connection.
+##
+## headers: The headers for the modbus function.
+##
+## read_start_address: The memory address of the first register to be read.
+##
+## read_quantity: The number of registers to read.
+##
+## write_start_address: The memory address of the first register to be written.
+##
+## write_registers: The values to be written to the registers.
+event modbus_read_write_multiple_registers_request%(c: connection, headers: ModbusHeaders, read_start_address: count, read_quantity: count, write_start_address: count, write_registers: ModbusRegisters%);
+
+## Generated for a Modbus read/write multiple registers response.
+##
+## c: The connection.
+##
+## headers: The headers for the modbus function.
+##
+## written_registers: The register values read from the registers specified in the request.
+event modbus_read_write_multiple_registers_response%(c: connection, headers: ModbusHeaders, written_registers: ModbusRegisters%);
+
+## Generated for a Modbus read FIFO queue request.
+##
+## c: The connection.
+##
+## headers: The headers for the modbus function.
+##
+## start_address: The address of the FIFO queue to read.
+event modbus_read_fifo_queue_request%(c: connection, headers: ModbusHeaders, start_address: count%);
+
+## Generated for a Modbus read FIFO queue response.
+##
+## c: The connection.
+##
+## headers: The headers for the modbus function.
+##
+## fifos: The register values read from the FIFO queue on the device.
+event modbus_read_fifo_queue_response%(c: connection, headers: ModbusHeaders, fifos: ModbusRegisters%);
+
 ## Raised for informational messages reported via Bro's reporter framework. Such
 ## messages may be generated internally by the event engine and also by other
 ## scripts calling :bro:id:`Reporter::info`.

src/modbus-analyzer.pac

@@ -0,0 +1,564 @@
+#
+# The development of Bro's Modbus analyzer has been made possible thanks to
+# the support of the Ministry of Security and Justice of the Kingdom of the
+# Netherlands within the projects of Hermes, Castor and Midas.
+#
+# Useful references: http://www.modbus.org/docs/Modbus_Application_Protocol_V1_1b.pdf
+#                    http://www.simplymodbus.ca/faq.htm
+#
+
+%header{
+	VectorVal* bytestring_to_coils(bytestring coils, uint quantity);
+	RecordVal* HeaderToBro(ModbusTCP_TransportHeader *header);
+	%}
+
+%code{
+	VectorVal* bytestring_to_coils(bytestring coils, uint quantity)
+		{
+		VectorVal* modbus_coils = new VectorVal(BifType::Vector::ModbusCoils);
+
+		return modbus_coils;
+		}
+
+	RecordVal* HeaderToBro(ModbusTCP_TransportHeader *header)
+		{
+		RecordVal* modbus_header = new RecordVal(BifType::Record::ModbusHeaders);
+		modbus_header->Assign(0, new Val(header->tid(), TYPE_COUNT));
+		modbus_header->Assign(1, new Val(header->pid(), TYPE_COUNT));
+		modbus_header->Assign(2, new Val(header->uid(), TYPE_COUNT));
+		modbus_header->Assign(3, new Val(header->fc(), TYPE_COUNT));
+		return modbus_header;
+		}
+
+	%}
+
+refine flow ModbusTCP_Flow += {
+
+	function deliver_message(header: ModbusTCP_TransportHeader): bool
+		%{
+		if ( ::modbus_message )
+			{
+			BifEvent::generate_modbus_message(connection()->bro_analyzer(),
+			                                  connection()->bro_analyzer()->Conn(),
+			                                  HeaderToBro(header),
+			                                  is_orig());
+			}
+
+		return true;
+		%}
+
+	# EXCEPTION
+	function deliver_Exception(header: ModbusTCP_TransportHeader, message: Exception): bool
+		%{
+		if ( ::modbus_exception )
+			{
+			BifEvent::generate_modbus_exception(connection()->bro_analyzer(),
+			                                    connection()->bro_analyzer()->Conn(),
+			                                    HeaderToBro(header),
+			                                    ${message.code});
+			}
+
+		return true;
+		%}
+
+	# REQUEST FC=1
+	function deliver_ReadCoilsRequest(header: ModbusTCP_TransportHeader, message: ReadCoilsRequest): bool
+		%{
+		if ( ::modbus_read_coils_request )
+			{
+			BifEvent::generate_modbus_read_coils_request(connection()->bro_analyzer(),
+			                                             connection()->bro_analyzer()->Conn(),
+			                                             HeaderToBro(header),
+			                                             ${message.start_address},
+			                                             ${message.quantity});
+			}
+
+		return true;
+		%}
+
+	# RESPONSE FC=1
+	function deliver_ReadCoilsResponse(header: ModbusTCP_TransportHeader, message: ReadCoilsResponse): bool
+		%{
+		if ( ::modbus_read_coils_response )
+			{
+			BifEvent::generate_modbus_read_coils_response(connection()->bro_analyzer(),
+			                                              connection()->bro_analyzer()->Conn(),
+			                                              HeaderToBro(header),
+			                                              bytestring_to_coils(${message.bits}, ${message.bits}.length()*8));
+			}
+		return true;
+		%}
+
+	# REQUEST FC=2
+	function deliver_ReadDiscreteInputsRequest(header: ModbusTCP_TransportHeader, message: ReadDiscreteInputsRequest): bool
+		%{
+		if ( ::modbus_read_discrete_inputs_request )
+			{
+			BifEvent::generate_modbus_read_discrete_inputs_request(connection()->bro_analyzer(),
+			                                                       connection()->bro_analyzer()->Conn(),
+			                                                       HeaderToBro(header),
+			                                                       ${message.start_address}, ${message.quantity});
+			}
+
+		return true;
+		%}
+
+	# RESPONSE FC=2
+	function deliver_ReadDiscreteInputsResponse(header: ModbusTCP_TransportHeader, message: ReadDiscreteInputsResponse): bool
+		%{
+		if ( ::modbus_read_discrete_inputs_response )
+			{
+			BifEvent::generate_modbus_read_discrete_inputs_response(connection()->bro_analyzer(),
+			                                                        connection()->bro_analyzer()->Conn(),
+			                                                        HeaderToBro(header),
+			                                                        bytestring_to_coils(${message.bits}, ${message.bits}.length()*8));
+			}
+
+		return true;
+		%}
+
+
+	# REQUEST FC=3
+	function deliver_ReadHoldingRegistersRequest(header: ModbusTCP_TransportHeader, message: ReadHoldingRegistersRequest): bool
+		%{
+		if ( ::modbus_read_holding_registers_request )
+			{
+			BifEvent::generate_modbus_read_holding_registers_request(connection()->bro_analyzer(),
+			                                                         connection()->bro_analyzer()->Conn(),
+			                                                         HeaderToBro(header),
+			                                                         ${message.start_address}, ${message.quantity});
+			}
+
+		return true;
+		%}
+
+	# RESPONSE FC=3
+	function deliver_ReadHoldingRegistersResponse(header: ModbusTCP_TransportHeader, message: ReadHoldingRegistersResponse): bool
+		%{
+		if ( ::modbus_read_holding_registers_response )
+			{
+			VectorVal* t = new VectorVal(BifType::Vector::ModbusRegisters);
+			for ( unsigned int i=0; i < ${message.registers}->size(); ++i )
+				{
+				Val* r = new Val(${message.registers[i]}, TYPE_COUNT);
+				t->Assign(i, r, 0, OP_ASSIGN);
+				}
+
+			BifEvent::generate_modbus_read_holding_registers_response(connection()->bro_analyzer(),
+			                                                          connection()->bro_analyzer()->Conn(),
+			                                                          HeaderToBro(header),
+			                                                          t);
+			}
+
+		return true;
+		%}
+
+
+	# REQUEST FC=4
+	function deliver_ReadInputRegistersRequest(header: ModbusTCP_TransportHeader, message: ReadInputRegistersRequest): bool
+		%{
+		if ( ::modbus_read_input_registers_request )
+			{
+			BifEvent::generate_modbus_read_input_registers_request(connection()->bro_analyzer(),
+			                                                       connection()->bro_analyzer()->Conn(),
+			                                                       HeaderToBro(header),
+			                                                       ${message.start_address}, ${message.quantity});
+			}
+
+		return true;
+		%}
+
+	# RESPONSE FC=4
+	function deliver_ReadInputRegistersResponse(header: ModbusTCP_TransportHeader, message: ReadInputRegistersResponse): bool
+		%{
+		if ( ::modbus_read_input_registers_response )
+			{
+			VectorVal* t = new VectorVal(BifType::Vector::ModbusRegisters);
+			for ( unsigned int i=0; i < (${message.registers})->size(); ++i )
+				{
+				Val* r = new Val(${message.registers[i]}, TYPE_COUNT);
+				t->Assign(i, r, 0, OP_ASSIGN);
+				}
+
+			BifEvent::generate_modbus_read_input_registers_response(connection()->bro_analyzer(),
+			                                                        connection()->bro_analyzer()->Conn(),
+			                                                        HeaderToBro(header), t);
+			}
+
+		return true;
+		%}
+
+
+	# REQUEST FC=5
+	function deliver_WriteSingleCoilRequest(header: ModbusTCP_TransportHeader, message: WriteSingleCoilRequest): bool
+		%{
+		if ( ::modbus_write_single_coil_request )
+			{
+			int val;
+			if ( ${message.value} == 0x0000 )
+				val = 0;
+			else if ( ${message.value} == 0xFF00 )
+				val = 1;
+			else
+				{
+				connection()->bro_analyzer()->ProtocolViolation(fmt("invalid value for modbus write single coil request %d",
+				                                                    ${message.value}));
+				return false;
+				}
+
+			BifEvent::generate_modbus_write_single_coil_request(connection()->bro_analyzer(),
+			                                                    connection()->bro_analyzer()->Conn(),
+			                                                    HeaderToBro(header),
+			                                                    ${message.address},
+			                                                    val);
+			}
+
+		return true;
+		%}
+
+	# RESPONSE FC=5
+	function deliver_WriteSingleCoilResponse(header: ModbusTCP_TransportHeader, message: WriteSingleCoilResponse): bool
+		%{
+		if ( ::modbus_write_single_coil_response )
+			{
+			int val;
+			if ( ${message.value} == 0x0000 )
+				val = 0;
+			else if ( ${message.value} == 0xFF00 )
+				val = 1;
+			else
+				{
+				connection()->bro_analyzer()->ProtocolViolation(fmt("invalid value for modbus write single coil response %d",
+				                                                    ${message.value}));
+				return false;
+				}
+
+			BifEvent::generate_modbus_write_single_coil_response(connection()->bro_analyzer(),
+			                                                     connection()->bro_analyzer()->Conn(),
+			                                                     HeaderToBro(header),
+			                                                     ${message.address},
+			                                                     val);
+			}
+
+		return true;
+		%}
+
+
+	# REQUEST FC=6
+	function deliver_WriteSingleRegisterRequest(header: ModbusTCP_TransportHeader, message: WriteSingleRegisterRequest): bool
+		%{
+		if ( ::modbus_write_single_register_request )
+			{
+			BifEvent::generate_modbus_write_single_register_request(connection()->bro_analyzer(),
+			                                                        connection()->bro_analyzer()->Conn(),
+			                                                        HeaderToBro(header),
+			                                                        ${message.address}, ${message.value});
+			}
+
+		return true;
+		%}
+
+	# RESPONSE FC=6
+	function deliver_WriteSingleRegisterResponse(header: ModbusTCP_TransportHeader, message: WriteSingleRegisterResponse): bool
+		%{
+		if ( ::modbus_write_single_register_response )
+			{
+			BifEvent::generate_modbus_write_single_register_response(connection()->bro_analyzer(),
+			                                                         connection()->bro_analyzer()->Conn(),
+			                                                         HeaderToBro(header),
+			                                                         ${message.address}, ${message.value});
+			}
+
+		return true;
+		%}
+
+
+	# REQUEST FC=15
+	function deliver_WriteMultipleCoilsRequest(header: ModbusTCP_TransportHeader, message: WriteMultipleCoilsRequest): bool
+		%{
+		if ( ::modbus_write_multiple_coils_request )
+			{
+			BifEvent::generate_modbus_write_multiple_coils_request(connection()->bro_analyzer(),
+			                                                       connection()->bro_analyzer()->Conn(),
+			                                                       HeaderToBro(header),
+			                                                       ${message.start_address},
+			                                                       bytestring_to_coils(${message.coils}, ${message.quantity}));
+			}
+
+		return true;
+		%}
+
+	# RESPONSE FC=15
+	function deliver_WriteMultipleCoilsResponse(header: ModbusTCP_TransportHeader, message: WriteMultipleCoilsResponse): bool
+		%{
+		if ( ::modbus_write_multiple_coils_response )
+			{
+			BifEvent::generate_modbus_write_multiple_coils_response(connection()->bro_analyzer(),
+			                                                        connection()->bro_analyzer()->Conn(),
+			                                                        HeaderToBro(header),
+			                                                        ${message.start_address}, ${message.quantity});
+			}
+
+		return true;
+		%}
+
+
+	# REQUEST FC=16
+	function deliver_WriteMultipleRegistersRequest(header: ModbusTCP_TransportHeader, message: WriteMultipleRegistersRequest): bool
+		%{
+		if ( ::modbus_write_multiple_registers_request )
+			{
+			VectorVal * t = new VectorVal(BifType::Vector::ModbusRegisters);
+			for ( unsigned int i = 0; i < (${message.registers}->size()); ++i )
+				{
+				Val* r = new Val(${message.registers[i]}, TYPE_COUNT);
+				t->Assign(i, r, 0, OP_ASSIGN);
+				}
+
+				BifEvent::generate_modbus_write_multiple_registers_request(connection()->bro_analyzer(),
+				                                                           connection()->bro_analyzer()->Conn(),
+				                                                           HeaderToBro(header),
+				                                                           ${message.start_address}, t);
+			}
+
+		return true;
+		%}
+
+	# RESPONSE FC=16
+	function deliver_WriteMultipleRegistersResponse(header: ModbusTCP_TransportHeader, message: WriteMultipleRegistersResponse): bool
+		%{
+		if ( ::modbus_write_multiple_registers_response )
+			{
+			BifEvent::generate_modbus_write_multiple_registers_response(connection()->bro_analyzer(),
+			                                                            connection()->bro_analyzer()->Conn(),
+			                                                            HeaderToBro(header),
+			                                                            ${message.start_address}, ${message.quantity});
+			}
+
+		return true;
+		%}
+
+	# REQUEST FC=20
+	function deliver_ReadFileRecordRequest(header: ModbusTCP_TransportHeader, message: ReadFileRecordRequest): bool
+		%{
+		if ( ::modbus_read_file_record_request )
+			{
+			//TODO: this need to be a vector of some Reference Request record type
+			//VectorVal *t = new VectorVal(new VectorType(base_type(TYPE_COUNT)));
+			//for ( unsigned int i = 0; i < (${message.references}->size()); ++i )
+			//	{
+			//	Val* r = new Val((${message.references[i].ref_type}), TYPE_COUNT);
+			//	t->Assign(i, r, 0, OP_ASSIGN);
+			//
+			//	Val* k = new Val((${message.references[i].file_num}), TYPE_COUNT);
+			//	t->Assign(i, k, 0, OP_ASSIGN);
+			//
+			//	Val* l = new Val((${message.references[i].record_num}), TYPE_COUNT);
+			//	t->Assign(i, l, 0, OP_ASSIGN);
+			//	}
+
+			BifEvent::generate_modbus_read_file_record_request(connection()->bro_analyzer(),
+			                                                   connection()->bro_analyzer()->Conn(),
+			                                                   HeaderToBro(header));
+			}
+
+		return true;
+		%}
+
+	# RESPONSE FC=20
+	function deliver_ReadFileRecordResponse(header: ModbusTCP_TransportHeader, message: ReadFileRecordResponse): bool
+		%{
+		if ( ::modbus_read_file_record_response )
+			{
+			//VectorVal *t = new VectorVal(new VectorType(base_type(TYPE_COUNT)));
+			//for ( unsigned int i = 0; i < ${message.references}->size(); ++i )
+			//	{
+			//	//TODO: work the reference type in here somewhere
+			//	Val* r = new Val(${message.references[i].record_data}), TYPE_COUNT);
+			//	t->Assign(i, r, 0, OP_ASSIGN);
+			//	}
+
+			BifEvent::generate_modbus_read_file_record_response(connection()->bro_analyzer(),
+			                                                    connection()->bro_analyzer()->Conn(),
+			                                                    HeaderToBro(header));
+			}
+
+		return true;
+		%}
+
+	# REQUEST FC=21
+	function deliver_WriteFileRecordRequest(header: ModbusTCP_TransportHeader, message: WriteFileRecordRequest): bool
+		%{
+		if ( ::modbus_write_file_record_request )
+			{
+			//VectorVal* t = new VectorVal(new VectorType(base_type(TYPE_COUNT)));
+			//for ( unsigned int i = 0; i < (${message.references}->size()); ++i )
+			//	{
+			//	Val* r = new Val((${message.references[i].ref_type}), TYPE_COUNT);
+			//	t->Assign(i, r, 0, OP_ASSIGN);
+			//
+			//	Val* k = new Val((${message.references[i].file_num}), TYPE_COUNT);
+			//	t->Assign(i, k, 0, OP_ASSIGN);
+			//
+			//	Val* n = new Val((${message.references[i].record_num}), TYPE_COUNT);
+			//	t->Assign(i, n, 0, OP_ASSIGN);
+			//
+			//	for ( unsigned int j = 0; j < (${message.references[i].register_value}->size()); ++j )
+			//		{
+			//		k = new Val((${message.references[i].register_value[j]}), TYPE_COUNT);
+			//		t->Assign(i, k, 0, OP_ASSIGN);
+			//		}
+			//	}
+
+			BifEvent::generate_modbus_write_file_record_request(connection()->bro_analyzer(),
+			                                                    connection()->bro_analyzer()->Conn(),
+			                                                    HeaderToBro(header));
+			}
+
+		return true;
+		%}
+
+
+	# RESPONSE FC=21
+	function deliver_WriteFileRecordResponse(header: ModbusTCP_TransportHeader, message: WriteFileRecordResponse): bool
+		%{
+		if ( ::modbus_write_file_record_response )
+			{
+			//VectorVal* t = new VectorVal(new VectorType(base_type(TYPE_COUNT)));
+			//for ( unsigned int i = 0; i < (${messages.references}->size()); ++i )
+			//	{
+			//	Val* r = new Val((${message.references[i].ref_type}), TYPE_COUNT);
+			//	t->Assign(i, r, 0, OP_ASSIGN);
+			//
+			//	Val* f = new Val((${message.references[i].file_num}), TYPE_COUNT);
+			//	t->Assign(i, f, 0, OP_ASSIGN);
+			//
+			//	Val* rn = new Val((${message.references[i].record_num}), TYPE_COUNT);
+			//	t->Assign(i, rn, 0, OP_ASSIGN);
+			//
+			//	for ( unsigned int j = 0; j<(${message.references[i].register_value}->size()); ++j )
+			//		{
+			//		Val* k = new Val((${message.references[i].register_value[j]}), TYPE_COUNT);
+			//		t->Assign(i, k, 0, OP_ASSIGN);
+			//		}
+
+			BifEvent::generate_modbus_write_file_record_response(connection()->bro_analyzer(),
+			                                                     connection()->bro_analyzer()->Conn(),
+			                                                     HeaderToBro(header));
+			}
+
+		return true;
+		%}
+
+	# REQUEST FC=22
+	function deliver_MaskWriteRegisterRequest(header: ModbusTCP_TransportHeader, message: MaskWriteRegisterRequest): bool
+		%{
+		if ( ::modbus_mask_write_register_request )
+			{
+			BifEvent::generate_modbus_mask_write_register_request(connection()->bro_analyzer(),
+			                                                      connection()->bro_analyzer()->Conn(),
+			                                                      HeaderToBro(header),
+			                                                      ${message.address},
+			                                                      ${message.and_mask}, ${message.or_mask});
+			}
+
+		return true;
+		%}
+
+	# RESPONSE FC=22
+	function deliver_MaskWriteRegisterResponse(header: ModbusTCP_TransportHeader, message: MaskWriteRegisterResponse): bool
+		%{
+		if ( ::modbus_mask_write_register_response )
+			{
+			BifEvent::generate_modbus_mask_write_register_response(connection()->bro_analyzer(),
+			                                                       connection()->bro_analyzer()->Conn(),
+			                                                       HeaderToBro(header),
+			                                                       ${message.address},
+			                                                       ${message.and_mask}, ${message.or_mask});
+			}
+
+		return true;
+		%}
+
+	# REQUEST FC=23
+	function deliver_ReadWriteMultipleRegistersRequest(header: ModbusTCP_TransportHeader, message: ReadWriteMultipleRegistersRequest): bool
+		%{
+		if ( ::modbus_read_write_multiple_registers_request )
+			{
+			VectorVal* t = new VectorVal(BifType::Vector::ModbusRegisters);
+			for ( unsigned int i = 0; i < ${message.write_register_values}->size(); ++i )
+				{
+				Val* r = new Val(${message.write_register_values[i]}, TYPE_COUNT);
+				t->Assign(i, r, 0, OP_ASSIGN);
+				}
+
+			BifEvent::generate_modbus_read_write_multiple_registers_request(connection()->bro_analyzer(),
+			                                                                connection()->bro_analyzer()->Conn(),
+			                                                                HeaderToBro(header),
+			                                                                ${message.read_start_address},
+			                                                                ${message.read_quantity},
+			                                                                ${message.write_start_address},
+			                                                                t);
+			}
+
+		return true;
+		%}
+
+	# RESPONSE FC=23
+	function deliver_ReadWriteMultipleRegistersResponse(header: ModbusTCP_TransportHeader, message: ReadWriteMultipleRegistersResponse): bool
+		%{
+		if ( ::modbus_read_write_multiple_registers_response )
+			{
+			VectorVal* t = new VectorVal(BifType::Vector::ModbusRegisters);
+			for ( unsigned int i = 0; i < ${message.registers}->size(); ++i )
+				{
+				Val* r = new Val(${message.registers[i]}, TYPE_COUNT);
+				t->Assign(i, r, 0, OP_ASSIGN);
+				}
+
+			BifEvent::generate_modbus_read_write_multiple_registers_response(connection()->bro_analyzer(),
+			                                                                 connection()->bro_analyzer()->Conn(),
+			                                                                 HeaderToBro(header), t);
+			}
+
+		return true;
+		%}
+
+	# REQUEST FC=24
+	function deliver_ReadFIFOQueueRequest(header: ModbusTCP_TransportHeader, message: ReadFIFOQueueRequest): bool
+		%{
+		if ( ::modbus_read_fifo_queue_request )
+			{
+			BifEvent::generate_modbus_read_fifo_queue_request(connection()->bro_analyzer(),
+			                                                  connection()->bro_analyzer()->Conn(),
+			                                                  HeaderToBro(header),
+			                                                  ${message.start_address});
+			}
+
+		return true;
+		%}
+
+
+	# RESPONSE FC=24
+	function deliver_ReadFIFOQueueResponse(header: ModbusTCP_TransportHeader, message: ReadFIFOQueueResponse): bool
+		%{
+		if ( ::modbus_read_fifo_queue_response )
+			{
+			VectorVal* t = new VectorVal(new VectorType(base_type(TYPE_COUNT)));
+			for ( unsigned int i = 0; i < (${message.register_data})->size(); ++i )
+				{
+				Val* r = new Val(${message.register_data[i]}, TYPE_COUNT);
+				t->Assign(i, r, 0, OP_ASSIGN);
+				}
+
+			BifEvent::generate_modbus_read_fifo_queue_response(connection()->bro_analyzer(),
+			                                                   connection()->bro_analyzer()->Conn(),
+			                                                   HeaderToBro(header), t);
+			}
+
+		return true;
+		%}
+};
+
+
+

src/modbus-protocol.pac

@@ -0,0 +1,394 @@
+#
+# The development of Bro's Modbus analyzer has been made possible thanks to
+# the support of the Ministry of Security and Justice of the Kingdom of the
+# Netherlands within the projects of Hermes, Castor and Midas.
+#
+# Useful references: http://www.modbus.org/docs/Modbus_Application_Protocol_V1_1b.pdf
+#                    http://www.simplymodbus.ca/faq.htm
+
+enum function_codes {
+	# Standard functions
+	READ_COILS                    = 0x01,
+	READ_DISCRETE_INPUTS          = 0x02,
+	READ_HOLDING_REGISTERS        = 0x03,
+	READ_INPUT_REGISTERS          = 0x04,
+	WRITE_SINGLE_COIL             = 0x05,
+	WRITE_SINGLE_REGISTER         = 0x06,
+	# READ_EXCEPTION_STATUS         = 0x07,
+	# DIAGNOSTICS                   = 0x08,
+	# GET_COMM_EVENT_COUNTER        = 0x0B,
+	# GET_COMM_EVENT_LOG            = 0x0C,
+	WRITE_MULTIPLE_COILS          = 0x0F,
+	WRITE_MULTIPLE_REGISTERS      = 0x10,
+	# REPORT_SLAVE_ID               = 0x11,
+	READ_FILE_RECORD              = 0x14,
+	WRITE_FILE_RECORD             = 0x15,
+	MASK_WRITE_REGISTER           = 0x16,
+	READ_WRITE_MULTIPLE_REGISTERS = 0x17,
+	READ_FIFO_QUEUE               = 0x18,
+	ENCAP_INTERFACE_TRANSPORT     = 0x2B,
+
+	# Machine/vendor/network specific functions
+	PROGRAM_484                   = 0x09,
+	POLL_484                      = 0x0A,
+	PROGRAM_584_984               = 0x0D,
+	POLL_584_984                  = 0x0E,
+	PROGRAM_884_U84               = 0x12,
+	RESET_COMM_LINK_884_U84       = 0x13,
+	PROGRAM_CONCEPT               = 0x28,
+	FIRMWARE_REPLACEMENT          = 0x7D,
+	PROGRAM_584_984_2             = 0x7E,
+	REPORT_LOCAL_ADDRESS          = 0x7F,
+
+	# Exceptions (not really function codes but they are used similarly)
+	READ_COILS_EXCEPTION                    = 0x81,
+	READ_DISCRETE_INPUTS_EXCEPTION          = 0x82,
+	READ_HOLDING_REGISTERS_EXCEPTION        = 0x83,
+	READ_INPUT_REGISTERS_EXCEPTION          = 0x84,
+	WRITE_SINGLE_COIL_EXCEPTION             = 0x85,
+	WRITE_SINGLE_REGISTER_EXCEPTION         = 0x86,
+	READ_EXCEPTION_STATUS_EXCEPTION         = 0x87,
+	WRITE_MULTIPLE_COILS_EXCEPTION          = 0x8F,
+	WRITE_MULTIPLE_REGISTERS_EXCEPTION      = 0x90,
+	READ_FILE_RECORD_EXCEPTION              = 0x94,
+	WRITE_FILE_RECORD_EXCEPTION             = 0x95,
+	MASK_WRITE_REGISTER_EXCEPTION           = 0x96,
+	READ_WRITE_MULTIPLE_REGISTERS_EXCEPTION = 0x97,
+	READ_FIFO_QUEUE_EXCEPTION               = 0x98,
+};
+
+# Main Modbus/TCP PDU
+type ModbusTCP_PDU(is_orig: bool) = record {
+	header: ModbusTCP_TransportHeader;
+	body: case is_orig of {
+		true  -> request:  ModbusTCP_Request(header);
+		false -> response: ModbusTCP_Response(header);
+	};
+} &length=header.len+6, &byteorder=bigendian;
+
+type ModbusTCP_TransportHeader = record {
+	tid: uint16; # Transaction identifier
+	pid: uint16; # Protocol identifier
+	len: uint16; # Length of everyting after this field
+	uid: uint8;  # Unit identifier (previously 'slave address')
+	fc:  uint8;  # MODBUS function code (see function_codes enum)
+} &byteorder=bigendian, &let {
+	deliver: bool = $context.flow.deliver_message(this);
+};
+
+type ModbusTCP_Request(header: ModbusTCP_TransportHeader) = case header.fc of {
+	READ_COILS                    -> readCoils:                  ReadCoilsRequest(header);
+	READ_DISCRETE_INPUTS          -> readDiscreteInputs:         ReadDiscreteInputsRequest(header);
+	READ_HOLDING_REGISTERS        -> readHoldingRegisters:       ReadHoldingRegistersRequest(header);
+	READ_INPUT_REGISTERS          -> readInputRegisters:         ReadInputRegistersRequest(header);
+	WRITE_SINGLE_COIL             -> writeSingleCoil:            WriteSingleCoilRequest(header);
+	WRITE_SINGLE_REGISTER         -> writeSingleRegister:        WriteSingleRegisterRequest(header);
+	#READ_EXCEPTION_STATUS         -> readExceptionStatus:        ReadExceptionStatusRequest(header);
+	#DIAGNOSTICS                   -> diagnostics:                DiagnosticsRequest(header);
+	#GET_COMM_EVENT_COUNTER        -> getCommEventCounter:        GetCommEventCounterRequest(header);
+	#GET_COMM_EVENT_LOG            -> getCommEventLog:            GetCommEventLogRequest(header);
+	WRITE_MULTIPLE_COILS          -> writeMultipleCoils:         WriteMultipleCoilsRequest(header);
+	WRITE_MULTIPLE_REGISTERS      -> writeMultRegisters:         WriteMultipleRegistersRequest(header);
+	#REPORT_SLAVE_ID
+	READ_FILE_RECORD              -> readFileRecord:             ReadFileRecordRequest(header);
+	WRITE_FILE_RECORD             -> writeFileRecord:            WriteFileRecordRequest(header);
+	MASK_WRITE_REGISTER           -> maskWriteRegister:          MaskWriteRegisterRequest(header);
+	READ_WRITE_MULTIPLE_REGISTERS -> readWriteMultipleRegisters: ReadWriteMultipleRegistersRequest(header);
+	READ_FIFO_QUEUE               -> readFIFOQueue:              ReadFIFOQueueRequest(header);
+	#ENCAP_INTERFACE_TRANSPORT
+
+	# All the rest
+	default                       -> unknown:                    bytestring &restofdata;
+};
+
+# Responses
+#
+type ModbusTCP_Response(header: ModbusTCP_TransportHeader) = case header.fc of {
+	READ_COILS                          -> readCoils:                       ReadCoilsResponse(header);
+	READ_DISCRETE_INPUTS                -> readDiscreteInputs:              ReadDiscreteInputsResponse(header);
+	READ_HOLDING_REGISTERS              -> readHoldingRegisters:            ReadHoldingRegistersResponse(header);
+	READ_INPUT_REGISTERS                -> readInputRegisters:              ReadInputRegistersResponse(header);
+	WRITE_SINGLE_COIL                   -> writeSingleCoil:                 WriteSingleCoilResponse(header);
+	WRITE_SINGLE_REGISTER               -> writeSingleRegister:             WriteSingleRegisterResponse(header);
+	#READ_EXCEPTION_STATUS               -> readExceptionStatus:             ReadExceptionStatusResponse(header);
+	#DIAGNOSTICS                         -> diagnostics:                     DiagnosticsResponse(header);
+	#GET_COMM_EVENT_COUNTER              -> getCommEventCounter:             GetCommEventCounterResponse(header);
+	#GET_COMM_EVENT_LOG                  -> getCommEventLog:                 GetCommEventLogResponse(header);
+	WRITE_MULTIPLE_COILS                -> writeMultipleCoils:              WriteMultipleCoilsResponse(header);
+	WRITE_MULTIPLE_REGISTERS            -> writeMultRegisters:              WriteMultipleRegistersResponse(header);
+	#REPORT_SLAVE_ID
+	READ_FILE_RECORD                    -> readFileRecord:                  ReadFileRecordResponse(header);
+	WRITE_FILE_RECORD                   -> writeFileRecord:                 WriteFileRecordResponse(header);
+	MASK_WRITE_REGISTER                 -> maskWriteRegister:               MaskWriteRegisterResponse(header);
+	READ_WRITE_MULTIPLE_REGISTERS       -> readWriteMultipleRegisters:      ReadWriteMultipleRegistersResponse(header);
+	READ_FIFO_QUEUE                     -> readFIFOQueue:                   ReadFIFOQueueResponse(header);
+
+	# Exceptions
+	READ_HOLDING_REGISTERS_EXCEPTION        -> readHoldingRegistersException:   Exception(header);
+	WRITE_MULTIPLE_REGISTERS_EXCEPTION      -> writeMultRegistersException:     Exception(header);
+	READ_COILS_EXCEPTION                    -> readCoilsException:              Exception(header);
+	READ_DISCRETE_INPUTS_EXCEPTION          -> readDiscreteInputsException:     Exception(header);
+	READ_INPUT_REGISTERS_EXCEPTION          -> readInputRegistersException:     Exception(header);
+	WRITE_SINGLE_COIL_EXCEPTION             -> writeCoilException:              Exception(header);
+	WRITE_SINGLE_REGISTER_EXCEPTION         -> writeSingleRegisterException:    Exception(header);
+	READ_EXCEPTION_STATUS_EXCEPTION         -> readExceptionStatusException:    Exception(header);
+	WRITE_MULTIPLE_COILS_EXCEPTION          -> forceMultipleCoilsException:     Exception(header);
+	READ_FILE_RECORD_EXCEPTION              -> readGeneralReferenceException:   Exception(header);
+	WRITE_FILE_RECORD_EXCEPTION             -> writeGeneralReferenceException:  Exception(header);
+	MASK_WRITE_REGISTER_EXCEPTION           -> maskWriteRegisterException:      Exception(header);
+	READ_WRITE_MULTIPLE_REGISTERS_EXCEPTION -> readWriteRegistersException:     Exception(header);
+	READ_FIFO_QUEUE_EXCEPTION               -> readFIFOQueueException:          Exception(header);
+
+	# All the rest
+	default                                 -> unknown:                         bytestring &restofdata;
+};
+
+type Exception(header: ModbusTCP_TransportHeader) = record {
+	code: uint8;
+} &let {
+	deliver: bool = $context.flow.deliver_Exception(header, this);
+};
+
+# REQUEST FC=1
+type ReadCoilsRequest(header: ModbusTCP_TransportHeader) = record {
+	start_address:  uint16;
+	quantity:       uint16 &check(quantity <= 2000);
+} &let {
+	deliver: bool = $context.flow.deliver_ReadCoilsRequest(header, this);
+} &byteorder=bigendian;
+
+# RESPONSE FC=1
+type ReadCoilsResponse(header: ModbusTCP_TransportHeader) = record {
+	byte_count: uint8;
+	bits:       bytestring &length=byte_count;
+} &let {
+	deliver: bool = $context.flow.deliver_ReadCoilsResponse(header, this);
+} &byteorder=bigendian;
+
+# REQUEST FC=2
+type ReadDiscreteInputsRequest(header: ModbusTCP_TransportHeader) = record {
+	start_address: uint16;
+	quantity:      uint16 &check(quantity <= 2000);
+} &let {
+	deliver: bool = $context.flow.deliver_ReadDiscreteInputsRequest(header, this);
+} &byteorder=bigendian;
+
+# RESPONSE FC=2
+type ReadDiscreteInputsResponse(header: ModbusTCP_TransportHeader) = record {
+	byte_count: uint8;
+	bits:       bytestring &length=byte_count;
+} &let {
+	deliver: bool = $context.flow.deliver_ReadDiscreteInputsResponse(header, this);
+} &byteorder=bigendian;
+
+# REQUEST FC=3
+type ReadHoldingRegistersRequest(header: ModbusTCP_TransportHeader) = record {
+	start_address: uint16;
+	quantity:      uint16 &check(1 <= quantity && quantity <= 125);
+} &let {
+	deliver: bool = $context.flow.deliver_ReadHoldingRegistersRequest(header, this);
+} &byteorder=bigendian;
+
+# RESPONSE FC=3
+type ReadHoldingRegistersResponse(header: ModbusTCP_TransportHeader) = record {
+	byte_count: uint8;
+	registers:  uint16[] &length=byte_count;
+} &let {
+	deliver: bool = $context.flow.deliver_ReadHoldingRegistersResponse(header, this);
+} &byteorder=bigendian;
+
+# REQUEST FC=4
+type ReadInputRegistersRequest(header: ModbusTCP_TransportHeader) = record {
+	start_address: uint16;
+	quantity:      uint16 &check(1 <= quantity && quantity <= 125);
+} &let {
+	deliver: bool = $context.flow.deliver_ReadInputRegistersRequest(header, this);
+} &byteorder=bigendian;
+
+# RESPONSE FC=4
+type ReadInputRegistersResponse(header: ModbusTCP_TransportHeader) = record {
+	byte_count: uint8;
+	registers:  uint16[] &length=byte_count;
+} &let {
+	deliver: bool = $context.flow.deliver_ReadInputRegistersResponse(header, this);
+} &byteorder=bigendian;
+
+# REQUEST FC=5
+type WriteSingleCoilRequest(header: ModbusTCP_TransportHeader) = record {
+	address: uint16;
+	value:   uint16 &check(value == 0x0000 || value == 0xFF00);
+} &let {
+	deliver: bool = $context.flow.deliver_WriteSingleCoilRequest(header, this);
+} &byteorder=bigendian;
+
+# RESPONSE FC=5
+type WriteSingleCoilResponse(header: ModbusTCP_TransportHeader) = record {
+	address: uint16;
+	value:   uint16 &check(value == 0x0000 || value == 0xFF00);
+} &let {
+	deliver: bool = $context.flow.deliver_WriteSingleCoilResponse(header, this);
+} &byteorder=bigendian;
+
+# REQUEST FC=6
+type WriteSingleRegisterRequest(header: ModbusTCP_TransportHeader) = record {
+	address:  uint16;
+	value:    uint16;
+} &let {
+	deliver: bool = $context.flow.deliver_WriteSingleRegisterRequest(header, this);
+} &byteorder=bigendian;
+
+# RESPONSE FC=6
+type WriteSingleRegisterResponse(header: ModbusTCP_TransportHeader) = record {
+	address:   uint16;
+	value:     uint16;
+} &let {
+	deliver: bool = $context.flow.deliver_WriteSingleRegisterResponse(header, this);
+} &byteorder=bigendian;
+
+# REQUEST FC=15
+type WriteMultipleCoilsRequest(header: ModbusTCP_TransportHeader) = record {
+	start_address:    uint16;
+	quantity:         uint16     &check(quantity <= 0x07B0);
+	byte_count:       uint8      &check(byte_count == (quantity + 7)/8);
+	coils:            bytestring &length=byte_count;
+} &let {
+	deliver: bool = $context.flow.deliver_WriteMultipleCoilsRequest(header, this);
+} &byteorder=bigendian;
+
+# RESPONSE FC=15
+type WriteMultipleCoilsResponse(header: ModbusTCP_TransportHeader) = record {
+	start_address:   uint16;
+	quantity:        uint16 &check(quantity <= 0x07B0);
+} &let {
+	deliver: bool = $context.flow.deliver_WriteMultipleCoilsResponse(header, this);
+} &byteorder=bigendian;
+
+# REQUEST FC=16
+type WriteMultipleRegistersRequest(header: ModbusTCP_TransportHeader) = record {
+	start_address: uint16;
+	quantity:      uint16;
+	byte_count:    uint8;
+	# We specify registers buffer with quantity and byte_count so that the analyzer 
+	# will choke if something doesn't match right (correct devices should make it right).
+	registers:     uint16[quantity] &length=byte_count;
+} &let {
+	deliver: bool = $context.flow.deliver_WriteMultipleRegistersRequest(header, this);
+} &byteorder=bigendian;
+
+# RESPONSE FC=16
+type WriteMultipleRegistersResponse(header: ModbusTCP_TransportHeader) = record {
+	start_address: uint16;
+	quantity:      uint16;
+} &let {
+	deliver: bool = $context.flow.deliver_WriteMultipleRegistersResponse(header, this);
+} &byteorder=bigendian;
+
+# Support data structure for following message type.
+type FileRecordRequest = record {
+	ref_type:   uint8  &check(ref_type == 6);
+	file_num:   uint16 &check(file_num > 0);
+	record_num: uint16 &check(record_num <= 0x270F);
+	record_len: uint16;
+} &byteorder=bigendian;
+
+# REQUEST FC=20
+type ReadFileRecordRequest(header: ModbusTCP_TransportHeader) = record {
+	byte_count: uint8               &check(byte_count >= 0x07 && byte_count <= 0xF5);
+	references: FileRecordRequest[] &length=byte_count;
+} &let {
+	deliver: bool = $context.flow.deliver_ReadFileRecordRequest(header, this);
+} &byteorder=bigendian;
+
+# Support data structure for the following message type.
+type FileRecordResponse = record {
+	file_len:    uint8    &check(file_len >= 0x07 && file_len <= 0xF5);
+	ref_type:    uint8    &check(ref_type == 6);
+	record_data: uint16[] &length=file_len;
+} &byteorder=bigendian;
+
+# RESPONSE FC=20
+type ReadFileRecordResponse(header: ModbusTCP_TransportHeader) = record {
+	byte_count: uint8 &check(byte_count >= 0x07 && byte_count <= 0xF5);
+	references: FileRecordResponse[] &length=byte_count;
+} &let {
+	deliver: bool = $context.flow.deliver_ReadFileRecordResponse(header, this);
+} &byteorder=bigendian;
+
+# Support data structure for the two following message types.
+type ReferenceWithData = record {
+	ref_type:       uint8;
+	file_num:       uint16;
+	record_num:     uint16;
+	word_count:     uint16;
+	register_value: uint16[word_count];
+} &byteorder=bigendian;
+
+# REQUEST FC=21
+type WriteFileRecordRequest(header: ModbusTCP_TransportHeader) = record {
+	byte_count: uint8;
+	references: ReferenceWithData[] &length=byte_count;
+} &let {
+	deliver: bool = $context.flow.deliver_WriteFileRecordRequest(header, this);
+} &byteorder=bigendian;
+
+# RESPONSE FC=21
+type WriteFileRecordResponse(header: ModbusTCP_TransportHeader) = record {
+	byte_count: uint8;
+	references: ReferenceWithData[] &length=byte_count;
+} &let {
+	deliver: bool = $context.flow.deliver_WriteFileRecordResponse(header, this);
+} &byteorder=bigendian;
+
+# REQUEST FC=22
+type MaskWriteRegisterRequest(header: ModbusTCP_TransportHeader) = record {
+	address:    uint16;
+	and_mask:   uint16;
+	or_mask:    uint16;
+} &let {
+	deliver: bool = $context.flow.deliver_MaskWriteRegisterRequest(header, this);
+} &byteorder=bigendian;
+
+# RESPONSE FC=22
+type MaskWriteRegisterResponse(header: ModbusTCP_TransportHeader) = record {
+	address:  uint16;
+	and_mask: uint16;
+	or_mask:  uint16;
+} &let {
+	deliver: bool = $context.flow.deliver_MaskWriteRegisterResponse(header, this);
+} &byteorder=bigendian;
+
+# REQUEST FC=23
+type ReadWriteMultipleRegistersRequest(header: ModbusTCP_TransportHeader) = record {
+	read_start_address:    uint16;
+	read_quantity:         uint16                 &check(read_quantity <= 125);
+	write_start_address:   uint16;
+	write_quantity:        uint16                 &check(write_quantity <= 100);
+	write_byte_count:      uint8;
+	write_register_values: uint16[write_quantity] &length=write_byte_count;
+} &let {
+	 deliver: bool = $context.flow.deliver_ReadWriteMultipleRegistersRequest(header, this);
+} &byteorder=bigendian;
+
+# RESPONSE FC=23
+type ReadWriteMultipleRegistersResponse(header: ModbusTCP_TransportHeader) = record {
+	byte_count:  uint8;
+	registers:   uint16[] &length=byte_count;
+} &let {
+	deliver: bool = $context.flow.deliver_ReadWriteMultipleRegistersResponse(header, this);
+} &byteorder=bigendian;
+
+# REQUEST FC=24
+type ReadFIFOQueueRequest(header: ModbusTCP_TransportHeader) = record {
+	start_address: uint16;
+} &let{
+	deliver: bool = $context.flow.deliver_ReadFIFOQueueRequest(header, this);
+} &byteorder=bigendian;
+
+# RESPONSE FC=24
+type ReadFIFOQueueResponse(header: ModbusTCP_TransportHeader) = record {
+	byte_count:    uint16             &check(byte_count <= 62);
+	fifo_count:    uint16             &check(fifo_count <= 31);
+	register_data: uint16[fifo_count] &length=byte_count/2;
+} &let {
+	deliver: bool = $context.flow.deliver_ReadFIFOQueueResponse(header, this);
+} &byteorder=bigendian;

src/modbus.pac

@@ -0,0 +1,28 @@
+#
+# The development of Bro's Modbus analyzer has been made possible thanks to
+# the support of the Ministry of Security and Justice of the Kingdom of the
+# Netherlands within the projects of Hermes, Castor and Midas.
+#
+# Useful references: http://www.modbus.org/docs/Modbus_Application_Protocol_V1_1b.pdf
+#                    http://www.simplymodbus.ca/faq.htm
+
+%include binpac.pac
+%include bro.pac
+
+analyzer ModbusTCP withcontext {
+	connection: ModbusTCP_Conn;
+	flow:       ModbusTCP_Flow;
+};
+
+connection ModbusTCP_Conn(bro_analyzer: BroAnalyzer) {
+	upflow = ModbusTCP_Flow(true);
+	downflow = ModbusTCP_Flow(false);
+};
+
+%include modbus-protocol.pac
+
+flow ModbusTCP_Flow(is_orig: bool) {
+	flowunit = ModbusTCP_PDU(is_orig) withcontext (connection, this);
+}
+
+%include modbus-analyzer.pac

src/nb_dns.c

@@ -307,31 +307,32 @@ nb_dns_host_request(register struct nb_dns_info *nd, register const char *name,
     register void *cookie, register char *errstr)
 {
 
-	return (nb_dns_host_request2(nd, name, AF_INET, cookie, errstr));
+	return (nb_dns_host_request2(nd, name, AF_INET, 0, cookie, errstr));
 }
 
 int
 nb_dns_host_request2(register struct nb_dns_info *nd, register const char *name,
-    register int af, register void *cookie, register char *errstr)
+    register int af, register int qtype, register void *cookie, register char *errstr)
 {
-	register int qtype;
+	if (qtype != 16) {
 
-	switch (af) {
+		switch (af) {
 
-	case AF_INET:
+		case AF_INET:
-		qtype = T_A;
+			qtype = T_A;
-		break;
+			break;
 
 #ifdef AF_INET6
-	case AF_INET6:
+		case AF_INET6:
-		qtype = T_AAAA;
+			qtype = T_AAAA;
-		break;
+			break;
 #endif
 
-	default:
+		default:
-		snprintf(errstr, NB_DNS_ERRSIZE,
+			snprintf(errstr, NB_DNS_ERRSIZE,
-		    "nb_dns_host_request2(): uknown address family %d", af);
+			    "nb_dns_host_request2(): unknown address family %d", af);
-		return (-1);
+			return (-1);
+		}
 	}
 	return (_nb_dns_mkquery(nd, name, af, qtype, cookie, errstr));
 }
@@ -579,6 +580,7 @@ nb_dns_activity(struct nb_dns_info *nd, struct nb_dns_result *nr, char *errstr)
 				nr->host_errno = NO_RECOVERY;
 				return (-1);
 			}
+
 			memcpy(bp, rdata, rdlen);
 			*hap++ = bp;
 			bp += rdlen;
@@ -587,6 +589,20 @@ nb_dns_activity(struct nb_dns_info *nd, struct nb_dns_result *nr, char *errstr)
 			/* Keep looking for more A records */
 			break;
 
+		case T_TXT:
+			if (bp + rdlen >= ep) {
+				snprintf(errstr, NB_DNS_ERRSIZE,
+				    "nb_dns_activity(): overflow 1 for txt");
+				nr->host_errno = NO_RECOVERY;
+				return (-1);
+			}
+
+			memcpy(bp, rdata, rdlen);
+			he->h_name = bp+1; /* First char is a control character. */
+			nr->hostent = he;
+			nr->ttl = rttl;
+			return (1);
+
 		case T_PTR:
 			n = dn_expand((const u_char *)msg,
 			    (const u_char *)msg + msglen, rdata, bp, ep - bp);

src/nb_dns.h

@@ -22,7 +22,7 @@ void nb_dns_finish(struct nb_dns_info *);
 int nb_dns_fd(struct nb_dns_info *);
 
 int nb_dns_host_request(struct nb_dns_info *, const char *, void *, char *);
-int nb_dns_host_request2(struct nb_dns_info *, const char *, int,
+int nb_dns_host_request2(struct nb_dns_info *, const char *, int, int,
 				void *, char *);
 
 int nb_dns_addr_request(struct nb_dns_info *, nb_uint32_t, void *, char *);

src/rule-scan.l

@@ -11,16 +11,6 @@
 #include "rule-parse.h"
 
 int rules_line_number = 0;
-
-static string extract_ipv6(string s)
-	{
-	if ( s.substr(0, 3) == "[0x" )
-		s = s.substr(3, s.find("]") - 3);
-	else
-		s = s.substr(1, s.find("]") - 1);
-
-	return s;
-	}
 %}
 
 %x PS
@@ -49,15 +39,14 @@ PID	([0-9a-zA-Z_-]|"::")+
 }
 
 {IP6}	{
-	rules_lval.prefixval = new IPPrefix(IPAddr(extract_ipv6(yytext)), 128);
+	rules_lval.prefixval = new IPPrefix(IPAddr(extract_ip(yytext)), 128, true);
 	return TOK_IP6;
 	}
 
 {IP6}{OWS}"/"{OWS}{D}	{
-	char* l = strchr(yytext, '/');
+	int len = 0;
-	*l++ = '\0';
+	string ip = extract_ip_and_len(yytext, &len);
-	int len = atoi(l);
+	rules_lval.prefixval = new IPPrefix(IPAddr(ip), len, true);
-	rules_lval.prefixval = new IPPrefix(IPAddr(extract_ipv6(yytext)), len);
 	return TOK_IP6;
 	}
 

src/scan.l

@@ -148,6 +148,7 @@ D	[0-9]+
 HEX	[0-9a-fA-F]+
 IDCOMPONENT [A-Za-z_][A-Za-z_0-9]*
 ID	{IDCOMPONENT}(::{IDCOMPONENT})*
+IP6 ("["({HEX}:){7}{HEX}"]")|("["0x{HEX}({HEX}|:)*"::"({HEX}|:)*"]")|("["({HEX}|:)*"::"({HEX}|:)*"]")|("["({HEX}|:)*"::"({HEX}|:)*({D}"."){3}{D}"]")
 FILE	[^ \t\n]+
 PREFIX	[^ \t\n]+
 FLOAT	(({D}*"."?{D})|({D}"."?{D}*))([eE][-+]?{D})?
@@ -229,21 +230,23 @@ ESCSEQ	(\\([^\n]|[0-7]+|x[[:xdigit:]]+))
 			}
 
 	/* IPv6 literal constant patterns */
-"["({HEX}:){7}{HEX}"]" {
+{IP6}	{
-	string s(yytext+1);
+	RET_CONST(new AddrVal(extract_ip(yytext)))
-	RET_CONST(new AddrVal(s.erase(s.size()-1)))
 }
-"["0x{HEX}({HEX}|:)*"::"({HEX}|:)*"]" {
+
-	string s(yytext+3);
+{IP6}{OWS}"/"{OWS}{D}	{
-	RET_CONST(new AddrVal(s.erase(s.size()-1)))
+	int len = 0;
+	string ip = extract_ip_and_len(yytext, &len);
+	RET_CONST(new SubNetVal(IPPrefix(IPAddr(ip), len, true)))
 }
-"["({HEX}|:)*"::"({HEX}|:)*"]" {
+
-	string s(yytext+1);
+	/* IPv4 literal constant patterns */
-	RET_CONST(new AddrVal(s.erase(s.size()-1)))
+({D}"."){3}{D}		RET_CONST(new AddrVal(yytext))
-}
+
-"["({HEX}|:)*"::"({HEX}|:)*({D}"."){3}{D}"]" {
+({D}"."){3}{D}{OWS}"/"{OWS}{D}	{
-	string s(yytext+1);
+	int len = 0;
-	RET_CONST(new AddrVal(s.erase(s.size()-1)))
+	string ip = extract_ip_and_len(yytext, &len);
+	RET_CONST(new SubNetVal(IPPrefix(IPAddr(ip), len)))
 }
 
 [!%*/+\-,:;<=>?()\[\]{}~$|]	return yytext[0];
@@ -484,8 +487,6 @@ F	RET_CONST(new Val(false, TYPE_BOOL))
 {FLOAT}{OWS}msec(s?)	RET_CONST(new IntervalVal(atof(yytext),Milliseconds))
 {FLOAT}{OWS}usec(s?)	RET_CONST(new IntervalVal(atof(yytext),Microseconds))
 
-({D}"."){3}{D}		RET_CONST(new AddrVal(yytext))
-
 "0x"{HEX}+	RET_CONST(new Val(static_cast<bro_uint_t>(strtoull(yytext, 0, 16)), TYPE_COUNT))
 
 {H}("."{H})+		RET_CONST(dns_mgr->LookupHost(yytext))

src/types.bif

@@ -156,6 +156,13 @@ type readdir_reply_t: record;
 
 type fsstat_t: record;
 
+
+module GLOBAL;
+
+type ModbusHeaders: record;
+type ModbusCoils: vector;
+type ModbusRegisters: vector;
+
 module Log;
 
 enum Writer %{

src/util.cc

@@ -43,6 +43,40 @@
 #include "Net.h"
 #include "Reporter.h"
 
+/**
+ * Return IP address without enclosing brackets and any leading 0x.
+ */
+std::string extract_ip(const std::string& i)
+	{
+	std::string s(skip_whitespace(i.c_str()));
+	if ( s.size() > 0 && s[0] == '[' )
+		s.erase(0, 1);
+
+	if ( s.size() > 1 && s.substr(0, 2) == "0x" )
+		s.erase(0, 2);
+
+	size_t pos = 0;
+	if ( (pos = s.find(']')) != std::string::npos )
+		s = s.substr(0, pos);
+
+	return s;
+	}
+
+/**
+ * Given a subnet string, return IP address and subnet length separately.
+ */
+std::string extract_ip_and_len(const std::string& i, int* len)
+	{
+	size_t pos = i.find('/');
+	if ( pos == std::string::npos )
+		return i;
+
+	if ( len )
+		*len = atoi(i.substr(pos + 1).c_str());
+
+	return extract_ip(i.substr(0, pos));
+	}
+
 /**
  * Takes a string, unescapes all characters that are escaped as hex codes
  * (\x##) and turns them into the equivalent ascii-codes. Returns a string

src/util.h

@@ -91,6 +91,9 @@ void delete_each(T* t)
 		delete *it;
 	}
 
+std::string extract_ip(const std::string& i);
+std::string extract_ip_and_len(const std::string& i, int* len);
+
 std::string get_unescaped_string(const std::string& str);
 std::string get_escaped_string(const std::string& str, bool escape_all);
 

testing/btest/Baseline/core.pppoe/conn.log

@@ -0,0 +1,16 @@
+#separator \x09
+#set_separator	,
+#empty_field	(empty)
+#unset_field	-
+#path	conn
+#open	2012-10-24-05-04-16
+#fields	ts	uid	id.orig_h	id.orig_p	id.resp_h	id.resp_p	proto	service	duration	orig_bytes	resp_bytes	conn_state	local_orig	missed_bytes	history	orig_pkts	orig_ip_bytes	resp_pkts	resp_ip_bytes	tunnel_parents
+#types	time	string	addr	port	addr	port	enum	string	interval	count	count	string	bool	count	string	count	count	count	count	table[string]
+1284385418.014560	TEfuqmmG4bh	fe80::c801:eff:fe88:8	547	fe80::ce05:eff:fe88:0	546	udp	-	0.096000	192	0	S0	-	0	D	2	288	0	0	(empty)
+1284385417.962560	j4u32Pc5bif	fe80::ce05:eff:fe88:0	546	ff02::1:2	547	udp	-	0.078000	114	0	S0	-	0	D	2	210	0	0	(empty)
+1284385411.091560	arKYeMETxOg	fe80::c801:eff:fe88:8	136	ff02::1	135	icmp	-	-	-	-	OTH	-	0	-	1	64	0	0	(empty)
+1284385411.035560	UWkUyAuUGXf	fe80::c801:eff:fe88:8	143	ff02::16	0	icmp	-	0.835000	160	0	OTH	-	0	-	8	608	0	0	(empty)
+1284385451.658560	FrJExwHcSal	fc00:0:2:100::1:1	128	fc00::1	129	icmp	-	0.156000	260	260	OTH	-	0	-	5	500	5	500	(empty)
+1284385413.027560	nQcgTWjvg4c	fe80::c801:eff:fe88:8	134	fe80::ce05:eff:fe88:0	133	icmp	-	-	-	-	OTH	-	0	-	1	64	0	0	(empty)
+1284385412.963560	k6kgXLOoSKl	fe80::ce05:eff:fe88:0	133	ff02::2	134	icmp	-	-	-	-	OTH	-	0	-	1	48	0	0	(empty)
+#close	2012-10-24-05-04-16

testing/btest/Baseline/core.print-bpf-filters/output

@@ -3,38 +3,38 @@
 #empty_field	(empty)
 #unset_field	-
 #path	packet_filter
-#open	2012-10-08-16-16-08
+#open	2012-11-06-00-53-09
 #fields	ts	node	filter	init	success
 #types	time	string	string	bool	bool
-1349712968.812610	-	ip or not ip	T	T
+1352163189.729807	-	ip or not ip	T	T
-#close	2012-10-08-16-16-08
+#close	2012-11-06-00-53-09
 #separator \x09
 #set_separator	,
 #empty_field	(empty)
 #unset_field	-
 #path	packet_filter
-#open	2012-10-08-16-16-09
+#open	2012-11-06-00-53-10
 #fields	ts	node	filter	init	success
 #types	time	string	string	bool	bool
-1349712969.042094	-	(((((((((((((((((((((((((port 53) or (tcp port 989)) or (tcp port 443)) or (port 6669)) or (udp and port 5353)) or (port 6668)) or (tcp port 1080)) or (udp and port 5355)) or (tcp port 995)) or (tcp port 22)) or (port 21 and port 2811)) or (tcp port 25 or tcp port 587)) or (tcp port 614)) or (tcp port 990)) or (port 6667)) or (udp port 137)) or (tcp port 993)) or (tcp port 5223)) or (port 514)) or (tcp port 585)) or (tcp port 992)) or (tcp port 563)) or (tcp port 994)) or (tcp port 636)) or (tcp and port (80 or 81 or 631 or 1080 or 3138 or 8000 or 8080 or 8888))) or (port 6666)	T	T
+1352163190.114261	-	((((((((((((((((((((((((((port 53) or (tcp port 989)) or (tcp port 443)) or (port 6669)) or (udp and port 5353)) or (port 6668)) or (tcp port 1080)) or (udp and port 5355)) or (tcp port 502)) or (tcp port 995)) or (tcp port 22)) or (port 21 and port 2811)) or (tcp port 25 or tcp port 587)) or (tcp port 614)) or (tcp port 990)) or (port 6667)) or (udp port 137)) or (tcp port 993)) or (tcp port 5223)) or (port 514)) or (tcp port 585)) or (tcp port 992)) or (tcp port 563)) or (tcp port 994)) or (tcp port 636)) or (tcp and port (80 or 81 or 631 or 1080 or 3138 or 8000 or 8080 or 8888))) or (port 6666)	T	T
-#close	2012-10-08-16-16-09
+#close	2012-11-06-00-53-10
 #separator \x09
 #set_separator	,
 #empty_field	(empty)
 #unset_field	-
 #path	packet_filter
-#open	2012-10-08-16-16-09
+#open	2012-11-06-00-53-10
 #fields	ts	node	filter	init	success
 #types	time	string	string	bool	bool
-1349712969.270826	-	port 42	T	T
+1352163190.484506	-	port 42	T	T
-#close	2012-10-08-16-16-09
+#close	2012-11-06-00-53-10
 #separator \x09
 #set_separator	,
 #empty_field	(empty)
 #unset_field	-
 #path	packet_filter
-#open	2012-10-08-16-16-09
+#open	2012-11-06-00-53-10
 #fields	ts	node	filter	init	success
 #types	time	string	string	bool	bool
-1349712969.499878	-	port 56730	T	T
+1352163190.855090	-	port 56730	T	T
-#close	2012-10-08-16-16-09
+#close	2012-11-06-00-53-10

testing/btest/Baseline/coverage.default-load-baseline/canonified_loaded_scripts.log

@@ -3,7 +3,7 @@
 #empty_field	(empty)
 #unset_field	-
 #path	loaded_scripts
-#open	2012-07-20-14-34-40
+#open	2012-11-05-23-29-45
 #fields	name
 #types	string
 scripts/base/init-bare.bro
@@ -40,6 +40,7 @@ scripts/base/init-default.bro
   scripts/base/utils/paths.bro
   scripts/base/utils/strings.bro
   scripts/base/utils/thresholds.bro
+  scripts/base/utils/urls.bro
   scripts/base/frameworks/notice/__load__.bro
     scripts/base/frameworks/notice/./main.bro
     scripts/base/frameworks/notice/./weird.bro
@@ -69,6 +70,7 @@ scripts/base/init-default.bro
     scripts/base/frameworks/metrics/./non-cluster.bro
   scripts/base/frameworks/intel/__load__.bro
     scripts/base/frameworks/intel/./main.bro
+    scripts/base/frameworks/intel/./input.bro
   scripts/base/frameworks/reporter/__load__.bro
     scripts/base/frameworks/reporter/./main.bro
   scripts/base/frameworks/tunnels/__load__.bro
@@ -99,6 +101,9 @@ scripts/base/init-default.bro
   scripts/base/protocols/irc/__load__.bro
     scripts/base/protocols/irc/./main.bro
     scripts/base/protocols/irc/./dcc-send.bro
+  scripts/base/protocols/modbus/__load__.bro
+    scripts/base/protocols/modbus/./consts.bro
+    scripts/base/protocols/modbus/./main.bro
   scripts/base/protocols/smtp/__load__.bro
     scripts/base/protocols/smtp/./main.bro
     scripts/base/protocols/smtp/./entities.bro
@@ -111,5 +116,6 @@ scripts/base/init-default.bro
   scripts/base/protocols/syslog/__load__.bro
     scripts/base/protocols/syslog/./consts.bro
     scripts/base/protocols/syslog/./main.bro
+  scripts/base/misc/find-checksum-offloading.bro
 scripts/policy/misc/loaded-scripts.bro
-#close	2012-07-20-14-34-40
+#close	2012-11-05-23-29-45

testing/btest/Baseline/coverage.init-default/missing_loads

@@ -2,5 +2,6 @@
 -./frameworks/cluster/nodes/proxy.bro
 -./frameworks/cluster/nodes/worker.bro
 -./frameworks/cluster/setup-connections.bro
+-./frameworks/intel/cluster.bro
 -./frameworks/metrics/cluster.bro
 -./frameworks/notice/cluster.bro

testing/btest/Baseline/language.addr/out

@@ -13,3 +13,5 @@ IPv6 address not case-sensitive (PASS)
 size of IPv6 address (PASS)
 IPv6 address type inference (PASS)
 IPv4 and IPv6 address inequality (PASS)
+IPv4-mapped-IPv6 equality to IPv4 (PASS)
+IPv4-mapped-IPv6 is IPv4 (PASS)

testing/btest/Baseline/language.subnet/out

@@ -10,3 +10,11 @@ IPv6 subnet !in operator (PASS)
 IPv6 subnet type inference (PASS)
 IPv4 and IPv6 subnet inequality (PASS)
 IPv4 address and IPv6 subnet (PASS)
+IPv4 in IPv4-mapped-IPv6 subnet (PASS)
+IPv6 !in IPv4-mapped-IPv6 subnet (PASS)
+IPv4-mapped-IPv6 in IPv4-mapped-IPv6 subnet (PASS)
+IPv4-mapped-IPv6 subnet equality (PASS)
+subnet literal const whitespace (PASS)
+subnet literal const whitespace (PASS)
+subnet literal const whitespace (PASS)
+subnet literal const whitespace (PASS)

testing/btest/Baseline/scripts.base.frameworks.intel.cluster-transparency/manager-1..stdout

@@ -0,0 +1,2 @@
+cluster_new_item: 123.123.123.123 inserted by worker-1 (from peer: worker-1)
+cluster_new_item: 4.3.2.1 inserted by worker-2 (from peer: worker-2)

testing/btest/Baseline/scripts.base.frameworks.intel.cluster-transparency/manager-1.intel.log

@@ -0,0 +1,10 @@
+#separator \x09
+#set_separator	,
+#empty_field	(empty)
+#unset_field	-
+#path	intel
+#open	2012-10-03-20-20-39
+#fields	ts	uid	id.orig_h	id.orig_p	id.resp_h	id.resp_p	seen.host	seen.str	seen.str_type	seen.where	sources
+#types	time	string	addr	port	addr	port	addr	string	enum	enum	table[string]
+1349295639.424940	-	-	-	-	-	123.123.123.123	-	-	Intel::IN_ANYWHERE	worker-1
+#close	2012-10-03-20-20-49

testing/btest/Baseline/scripts.base.frameworks.intel.cluster-transparency/worker-1..stdout

@@ -0,0 +1,3 @@
+cluster_new_item: 1.2.3.4 inserted by manager (from peer: manager-1)
+cluster_new_item: 123.123.123.123 inserted by worker-1 (from peer: manager-1)
+cluster_new_item: 4.3.2.1 inserted by worker-2 (from peer: manager-1)

testing/btest/Baseline/scripts.base.frameworks.intel.cluster-transparency/worker-2..stdout

@@ -0,0 +1,4 @@
+cluster_new_item: 1.2.3.4 inserted by manager (from peer: manager-1)
+cluster_new_item: 123.123.123.123 inserted by worker-1 (from peer: manager-1)
+cluster_new_item: 4.3.2.1 inserted by worker-2 (from peer: manager-1)
+Doing a lookup

testing/btest/Baseline/scripts.base.frameworks.intel.input-and-match/broproc.intel.log

@@ -0,0 +1,11 @@
+#separator \x09
+#set_separator	,
+#empty_field	(empty)
+#unset_field	-
+#path	intel
+#open	2012-10-03-20-18-05
+#fields	ts	uid	id.orig_h	id.orig_p	id.resp_h	id.resp_p	seen.host	seen.str	seen.str_type	seen.where	sources
+#types	time	string	addr	port	addr	port	addr	string	enum	enum	table[string]
+1349295485.114156	-	-	-	-	-	-	e@mail.com	Intel::EMAIL	SOMEWHERE	source1
+1349295485.114156	-	-	-	-	-	1.2.3.4	-	-	SOMEWHERE	source1
+#close	2012-10-03-20-18-05

testing/btest/Baseline/scripts.base.frameworks.intel.insert-and-matcher/out

@@ -1,3 +0,0 @@
-VALID
-VALID
-VALID

testing/btest/Baseline/scripts.base.frameworks.intel.read-file-dist-cluster/manager-1.intel.log

@@ -0,0 +1,13 @@
+#separator \x09
+#set_separator	,
+#empty_field	(empty)
+#unset_field	-
+#path	intel
+#open	2012-10-10-15-05-23
+#fields	ts	uid	id.orig_h	id.orig_p	id.resp_h	id.resp_p	seen.host	seen.str	seen.str_type	seen.where	sources
+#types	time	string	addr	port	addr	port	addr	string	enum	enum	table[string]
+1349881523.548946	-	-	-	-	-	1.2.3.4	-	-	Intel::IN_A_TEST	source1
+1349881523.548946	-	-	-	-	-	-	e@mail.com	Intel::EMAIL	Intel::IN_A_TEST	source1
+1349881524.567896	-	-	-	-	-	1.2.3.4	-	-	Intel::IN_A_TEST	source1
+1349881524.567896	-	-	-	-	-	-	e@mail.com	Intel::EMAIL	Intel::IN_A_TEST	source1
+#close	2012-10-10-15-05-24

testing/btest/Baseline/scripts.base.protocols.modbus.events/coverage

@@ -0,0 +1 @@
+2 of 28 events triggered by trace

testing/btest/Baseline/scripts.base.protocols.modbus.policy/known_modbus.log

@@ -0,0 +1,17 @@
+#separator \x09
+#set_separator	,
+#empty_field	(empty)
+#unset_field	-
+#path	known_modbus
+#open	2012-11-06-00-51-15
+#fields	ts	host	device_type
+#types	time	addr	enum
+1093521694.211940	10.0.0.57	Known::MODBUS_MASTER
+1093521694.211940	10.0.0.3	Known::MODBUS_SLAVE
+1093521958.375300	10.0.0.8	Known::MODBUS_SLAVE
+1093522338.985618	10.0.0.9	Known::MODBUS_MASTER
+1153491892.212845	192.168.66.235	Known::MODBUS_MASTER
+1153491892.212845	166.161.16.230	Known::MODBUS_SLAVE
+1342774499.589057	10.1.1.234	Known::MODBUS_MASTER
+1342774499.589057	10.10.5.85	Known::MODBUS_SLAVE
+#close	2012-11-06-00-51-23

testing/btest/Baseline/scripts.base.protocols.modbus.policy/modbus_register_change.log

@@ -0,0 +1,49 @@
+#separator \x09
+#set_separator	,
+#empty_field	(empty)
+#unset_field	-
+#path	modbus_register_change
+#open	2012-11-06-00-51-15
+#fields	ts	uid	id.orig_h	id.orig_p	id.resp_h	id.resp_p	register	old_val	new_val	delta
+#types	time	string	addr	port	addr	port	count	count	count	interval
+1342774501.024564	3PKsZ2Uye21	10.1.1.234	51411	10.10.5.85	502	101	0	1	1.250066
+1342774540.946501	3PKsZ2Uye21	10.1.1.234	51411	10.10.5.85	502	101	1	0	39.921937
+1342774540.946501	3PKsZ2Uye21	10.1.1.234	51411	10.10.5.85	502	103	0	1	41.172003
+1342774811.727563	3PKsZ2Uye21	10.1.1.234	51411	10.10.5.85	502	102	0	1	311.953065
+1342774811.727563	3PKsZ2Uye21	10.1.1.234	51411	10.10.5.85	502	103	1	0	270.781062
+1342774831.727542	3PKsZ2Uye21	10.1.1.234	51411	10.10.5.85	502	101	0	1	290.781041
+1342774831.727542	3PKsZ2Uye21	10.1.1.234	51411	10.10.5.85	502	102	1	0	19.999979
+1342774872.821282	3PKsZ2Uye21	10.1.1.234	51411	10.10.5.85	502	101	1	0	41.093740
+1342774872.821282	3PKsZ2Uye21	10.1.1.234	51411	10.10.5.85	502	103	0	1	61.093719
+1342775143.602482	3PKsZ2Uye21	10.1.1.234	51411	10.10.5.85	502	102	0	1	311.874940
+1342775143.602482	3PKsZ2Uye21	10.1.1.234	51411	10.10.5.85	502	103	1	0	270.781200
+1342775164.774350	3PKsZ2Uye21	10.1.1.234	51411	10.10.5.85	502	101	0	1	291.953068
+1342775164.774350	3PKsZ2Uye21	10.1.1.234	51411	10.10.5.85	502	102	1	0	21.171868
+1342775204.696194	3PKsZ2Uye21	10.1.1.234	51411	10.10.5.85	502	101	1	0	39.921844
+1342775204.696194	3PKsZ2Uye21	10.1.1.234	51411	10.10.5.85	502	103	0	1	61.093712
+1342775475.477365	3PKsZ2Uye21	10.1.1.234	51411	10.10.5.85	502	102	0	1	310.703015
+1342775475.477365	3PKsZ2Uye21	10.1.1.234	51411	10.10.5.85	502	103	1	0	270.781171
+1342775495.477389	3PKsZ2Uye21	10.1.1.234	51411	10.10.5.85	502	101	0	1	290.781195
+1342775495.477389	3PKsZ2Uye21	10.1.1.234	51411	10.10.5.85	502	102	1	0	20.000024
+1342775535.399236	3PKsZ2Uye21	10.1.1.234	51411	10.10.5.85	502	101	1	0	39.921847
+1342775535.399236	3PKsZ2Uye21	10.1.1.234	51411	10.10.5.85	502	103	0	1	59.921871
+1342775806.180404	3PKsZ2Uye21	10.1.1.234	51411	10.10.5.85	502	102	0	1	310.703015
+1342775806.180404	3PKsZ2Uye21	10.1.1.234	51411	10.10.5.85	502	103	1	0	270.781168
+1342775826.180415	3PKsZ2Uye21	10.1.1.234	51411	10.10.5.85	502	101	0	1	290.781179
+1342775826.180415	3PKsZ2Uye21	10.1.1.234	51411	10.10.5.85	502	102	1	0	20.000011
+1342775848.508596	3PKsZ2Uye21	10.1.1.234	51411	10.10.5.85	502	501	80	90	1348.671590
+1342775871.961652	3PKsZ2Uye21	10.1.1.234	51411	10.10.5.85	502	101	1	0	45.781237
+1342775871.961652	3PKsZ2Uye21	10.1.1.234	51411	10.10.5.85	502	103	0	1	65.781248
+1342776142.758456	3PKsZ2Uye21	10.1.1.234	51411	10.10.5.85	502	102	0	1	316.578041
+1342776142.758456	3PKsZ2Uye21	10.1.1.234	51411	10.10.5.85	502	103	1	0	270.796804
+1342776167.445943	3PKsZ2Uye21	10.1.1.234	51411	10.10.5.85	502	101	0	1	295.484291
+1342776167.445943	3PKsZ2Uye21	10.1.1.234	51411	10.10.5.85	502	102	1	0	24.687487
+1342776213.274085	3PKsZ2Uye21	10.1.1.234	51411	10.10.5.85	502	101	1	0	45.828142
+1342776213.274085	3PKsZ2Uye21	10.1.1.234	51411	10.10.5.85	502	103	0	1	70.515629
+1342776484.055366	3PKsZ2Uye21	10.1.1.234	51411	10.10.5.85	502	102	0	1	316.609423
+1342776484.055366	3PKsZ2Uye21	10.1.1.234	51411	10.10.5.85	502	103	1	0	270.781281
+1342776507.570851	3PKsZ2Uye21	10.1.1.234	51411	10.10.5.85	502	101	0	1	294.296766
+1342776507.570851	3PKsZ2Uye21	10.1.1.234	51411	10.10.5.85	502	102	1	0	23.515485
+1342776553.352098	3PKsZ2Uye21	10.1.1.234	51411	10.10.5.85	502	101	1	0	45.781247
+1342776553.352098	3PKsZ2Uye21	10.1.1.234	51411	10.10.5.85	502	103	0	1	69.296732
+#close	2012-11-06-00-51-23

testing/btest/core/pppoe.test

@@ -0,0 +1,2 @@
+# @TEST-EXEC: bro -r $TRACES/pppoe.trace %INPUT 
+# @TEST-EXEC: btest-diff conn.log

testing/btest/language/addr.bro

@@ -1,4 +1,4 @@
-# @TEST-EXEC: bro %INPUT >out
+# @TEST-EXEC: bro -b %INPUT >out
 # @TEST-EXEC: btest-diff out
 
 function test_case(msg: string, expect: bool)
@@ -43,5 +43,10 @@ event bro_init()
 
 	test_case( "IPv4 and IPv6 address inequality", a1 != b1 );
 
+	# IPv4-mapped-IPv6 (internally treated as IPv4)
+	local c1: addr = [::ffff:1.2.3.4];
+
+	test_case( "IPv4-mapped-IPv6 equality to IPv4", c1 == 1.2.3.4 );
+	test_case( "IPv4-mapped-IPv6 is IPv4", is_v4_addr(c1) == T );
 }
 

testing/btest/language/subnet.bro

@@ -1,4 +1,4 @@
-# @TEST-EXEC: bro %INPUT >out
+# @TEST-EXEC: bro -b %INPUT >out
 # @TEST-EXEC: btest-diff out
 
 function test_case(msg: string, expect: bool)
@@ -43,5 +43,22 @@ event bro_init()
 	test_case( "IPv4 and IPv6 subnet inequality", s1 != t1 );
 	test_case( "IPv4 address and IPv6 subnet", a1 !in t2 );
 
+	# IPv4-mapped-IPv6 subnets
+	local u1: subnet = [::ffff:0:0]/96;
+
+	test_case( "IPv4 in IPv4-mapped-IPv6 subnet", 1.2.3.4 in u1 );
+	test_case( "IPv6 !in IPv4-mapped-IPv6 subnet", [fe80::1] !in u1 );
+	test_case( "IPv4-mapped-IPv6 in IPv4-mapped-IPv6 subnet",
+	           [::ffff:1.2.3.4] in u1 );
+	test_case( "IPv4-mapped-IPv6 subnet equality",
+	           [::ffff:1.2.3.4]/112 == 1.2.0.0/16 );
+	test_case( "subnet literal const whitespace",
+	           [::ffff:1.2.3.4] / 112 == 1.2.0.0 / 16 );
+	test_case( "subnet literal const whitespace",
+	           [::ffff:1.2.3.4]/ 128 == 1.2.3.4/ 32 );
+	test_case( "subnet literal const whitespace",
+	           [::ffff:1.2.3.4] /96 == 1.2.3.4 /0 );
+	test_case( "subnet literal const whitespace",
+	           [::ffff:1.2.3.4]   /    92 == [::fffe:1.2.3.4]    /   92 );
 }
 

testing/btest/scripts/base/frameworks/intel/cluster-transparency.bro

@@ -0,0 +1,80 @@
+# @TEST-SERIALIZE: comm
+#
+# @TEST-EXEC: btest-bg-run manager-1 BROPATH=$BROPATH:.. CLUSTER_NODE=manager-1 bro %INPUT
+# @TEST-EXEC: btest-bg-run worker-1  BROPATH=$BROPATH:.. CLUSTER_NODE=worker-1 bro %INPUT
+# @TEST-EXEC: btest-bg-run worker-2  BROPATH=$BROPATH:.. CLUSTER_NODE=worker-2 bro %INPUT
+# @TEST-EXEC: btest-bg-wait -k 10
+# @TEST-EXEC: btest-diff manager-1/.stdout
+# @TEST-EXEC: btest-diff manager-1/intel.log
+# @TEST-EXEC: btest-diff worker-1/.stdout
+# @TEST-EXEC: btest-diff worker-2/.stdout
+
+@TEST-START-FILE cluster-layout.bro
+redef Cluster::nodes = {
+	["manager-1"] = [$node_type=Cluster::MANAGER, $ip=127.0.0.1, $p=37757/tcp, $workers=set("worker-1", "worker-2")],
+	["worker-1"]  = [$node_type=Cluster::WORKER,  $ip=127.0.0.1, $p=37760/tcp, $manager="manager-1"],
+	["worker-2"]  = [$node_type=Cluster::WORKER,  $ip=127.0.0.1, $p=37761/tcp, $manager="manager-1"],
+};
+@TEST-END-FILE
+
+@load base/frameworks/control
+
+module Intel;
+
+redef Log::default_rotation_interval=0sec;
+
+event remote_connection_handshake_done(p: event_peer)
+	{
+	# Insert the data once both workers are connected.
+	if ( Cluster::local_node_type() == Cluster::MANAGER && Cluster::worker_count == 2 )
+		{
+		Intel::insert([$host=1.2.3.4,$meta=[$source="manager"]]);
+		}
+	}
+
+global worker2_data = 0;
+global sent_data = F;
+event Intel::cluster_new_item(item: Intel::Item)
+	{
+	if ( ! is_remote_event() )
+		return;
+
+	print fmt("cluster_new_item: %s inserted by %s (from peer: %s)", item$host, item$meta$source, get_event_peer()$descr);
+
+	if ( ! sent_data )
+		{
+		# We wait to insert data here because we can now be sure the 
+		# full cluster is constructed.
+		sent_data = T;
+		if ( Cluster::node == "worker-1" )
+			Intel::insert([$host=123.123.123.123,$meta=[$source="worker-1"]]);
+		if ( Cluster::node == "worker-2" )
+			Intel::insert([$host=4.3.2.1,$meta=[$source="worker-2"]]);
+		}
+
+	# We're forcing worker-2 to do a lookup when it has three intelligence items
+	# which were distributed over the cluster (data inserted locally is resent).
+	if ( Cluster::node == "worker-2" )
+		{
+		++worker2_data;
+		if ( worker2_data == 3 )
+			{
+			# Now that everything is inserted, see if we can match on the data inserted
+			# by worker-1.
+			print "Doing a lookup";
+			Intel::seen([$host=123.123.123.123, $where=Intel::IN_ANYWHERE]);
+			}
+		}
+	}
+
+event Intel::log_intel(rec: Intel::Info)
+	{
+	event Control::shutdown_request();
+	}
+
+event remote_connection_closed(p: event_peer)
+	{
+	# Cascading termination
+	#print fmt("disconnected from: %s", p);
+	terminate_communication();
+	}

testing/btest/scripts/base/frameworks/intel/input-and-match.bro

@@ -0,0 +1,40 @@
+# @TEST-SERIALIZE: comm
+
+# @TEST-EXEC: btest-bg-run broproc bro %INPUT
+# @TEST-EXEC: btest-bg-wait -k 5
+# @TEST-EXEC: btest-diff broproc/intel.log
+
+@TEST-START-FILE intel.dat
+#fields	host	net	str	str_type	meta.source	meta.desc	meta.url
+1.2.3.4	-	-	-	source1	this host is just plain baaad	http://some-data-distributor.com/1234
+1.2.3.4	-	-	-	source1	this host is just plain baaad	http://some-data-distributor.com/1234
+-	-	e@mail.com	Intel::EMAIL	source1	Phishing email source	http://some-data-distributor.com/100000
+@TEST-END-FILE
+
+@load frameworks/communication/listen
+
+redef Intel::read_files += { "../intel.dat" };
+redef enum Intel::Where += { SOMEWHERE };
+
+event do_it()
+	{
+	Intel::seen([$str="e@mail.com",
+	             $str_type=Intel::EMAIL,
+	             $where=SOMEWHERE]);
+
+	Intel::seen([$host=1.2.3.4,
+	             $where=SOMEWHERE]);
+	}
+
+global log_lines = 0;
+event Intel::log_intel(rec: Intel::Info)
+	{
+	++log_lines;
+	if ( log_lines == 2 )
+		terminate();
+	}
+
+event bro_init() &priority=-10
+	{
+	schedule 1sec { do_it() };
+	}

testing/btest/scripts/base/frameworks/intel/insert-and-matcher.bro

@@ -1,34 +0,0 @@
-#
-# @TEST-EXEC: bro %INPUT >out
-# @TEST-EXEC: btest-diff out
-
-event bro_init()
-	{
-	Intel::insert([$ip=1.2.3.4, $tags=set("zeustracker.abuse.ch", "malicious")]);
-	Intel::insert([$str="http://www.google.com/", $subtype="url", $tags=set("infrastructure", "google")]);
-	Intel::insert([$str="Ab439G32F...", $subtype="x509_cert", $tags=set("bad")]);
-	Intel::insert([$str="Ab439G32F...", $tags=set("bad")]);
-	}
-
-event bro_done()
-	{
-	local orig_h = 1.2.3.4;
-	
-	if ( Intel::matcher([$ip=orig_h, $and_tags=set("malicious")]) )
-		print "VALID";
-	
-	if ( Intel::matcher([$ip=orig_h, $and_tags=set("don't match")]) )
-		print "INVALID";
-	
-	if ( Intel::matcher([$ip=orig_h, $pred=function(meta: Intel::MetaData): bool { return T; } ]) )
-		print "VALID";
-		
-	if ( Intel::matcher([$ip=orig_h, $pred=function(meta: Intel::MetaData): bool { return F; } ]) )
-		print "INVALID";
-		
-	if ( Intel::matcher([$str="http://www.google.com/", $subtype="url", $tags=set("google")]) )
-		print "VALID";
-		
-	if ( Intel::matcher([$str="http://www.example.com", $subtype="url"]) )
-		print "INVALID";
-	}

testing/btest/scripts/base/frameworks/intel/read-file-dist-cluster.bro

@@ -0,0 +1,66 @@
+# @TEST-SERIALIZE: comm
+#
+# @TEST-EXEC: btest-bg-run manager-1 BROPATH=$BROPATH:.. CLUSTER_NODE=manager-1 bro %INPUT
+# @TEST-EXEC: sleep 2
+# @TEST-EXEC: btest-bg-run worker-1  BROPATH=$BROPATH:.. CLUSTER_NODE=worker-1 bro %INPUT
+# @TEST-EXEC: btest-bg-run worker-2  BROPATH=$BROPATH:.. CLUSTER_NODE=worker-2 bro %INPUT
+# @TEST-EXEC: btest-bg-wait -k 10
+# @TEST-EXEC: btest-diff manager-1/.stdout
+# @TEST-EXEC: btest-diff manager-1/intel.log
+# @TEST-EXEC: btest-diff worker-1/.stdout
+# @TEST-EXEC: btest-diff worker-2/.stdout
+
+@TEST-START-FILE cluster-layout.bro
+redef Cluster::nodes = {
+	["manager-1"] = [$node_type=Cluster::MANAGER, $ip=127.0.0.1, $p=37757/tcp, $workers=set("worker-1", "worker-2")],
+	["worker-1"]  = [$node_type=Cluster::WORKER,  $ip=127.0.0.1, $p=37760/tcp, $manager="manager-1"],
+	["worker-2"]  = [$node_type=Cluster::WORKER,  $ip=127.0.0.1, $p=37761/tcp, $manager="manager-1"],
+};
+@TEST-END-FILE
+
+@TEST-START-FILE intel.dat
+#fields	host	net	str	str_type	meta.source	meta.desc	meta.url
+1.2.3.4	-	-	-	source1	this host is just plain baaad	http://some-data-distributor.com/1234
+1.2.3.4	-	-	-	source1	this host is just plain baaad	http://some-data-distributor.com/1234
+-	-	e@mail.com	Intel::EMAIL	source1	Phishing email source	http://some-data-distributor.com/100000
+@TEST-END-FILE
+
+@load base/frameworks/control
+redef Log::default_rotation_interval=0sec;
+
+module Intel;
+
+@if ( Cluster::local_node_type() == Cluster::MANAGER )
+redef Intel::read_files += { "../intel.dat" };
+@endif
+
+redef enum Intel::Where += {
+	Intel::IN_A_TEST,
+};
+
+event do_it()
+	{
+	Intel::seen([$host=1.2.3.4, $where=Intel::IN_A_TEST]);
+	Intel::seen([$str="e@mail.com", $str_type=Intel::EMAIL, $where=Intel::IN_A_TEST]);
+	}
+
+event bro_init()
+	{
+	# Delay the workers searching for hits briefly to allow for the data distribution
+	# mechanism to distribute the data to the workers.
+	if ( Cluster::local_node_type() == Cluster::WORKER )
+		schedule 2sec { do_it() };
+	}
+
+global intel_hits=0;
+event Intel::log_intel(rec: Intel::Info)
+	{
+	++intel_hits;
+	# There should be 4 hits since each worker is "seeing" 2 things.
+	if ( intel_hits == 4 )
+		{
+		# We're delaying shutdown for a second here to make sure that no other
+		# matches happen (which would be wrong!).
+		schedule 1sec { Control::shutdown_request() };
+		}
+	}

testing/btest/scripts/base/protocols/modbus/events.bro

@@ -0,0 +1,148 @@
+#
+# @TEST-EXEC: bro -r $TRACES/modbus.trace %INPUT | sort | uniq -c | sed 's/^ *//g' >output
+# @TEST-EXEC: btest-diff output
+# @TEST-EXEC: cat output | awk '{print $1}' | sort | uniq | wc -l >covered
+# @TEST-EXEC: cat ${DIST}/src/event.bif  | grep "^event modbus_" | wc -l >total
+# @TEST-EXEC: echo `cat covered` of `cat total` events triggered by trace >coverage
+# @TEST-EXEC: btest-diff coverage
+
+event modbus_message(c: connection, headers: ModbusHeaders, is_orig: bool)
+{
+    print "modbus_message", c, headers, is_orig;
+}
+
+event modbus_exception(c: connection, headers: ModbusHeaders, code: count)
+{
+    print "modbus_exception", c, headers, code;
+}
+
+event modbus_read_coils_request(c: connection, headers: ModbusHeaders, start_address: count, quantity: count)
+{
+    print "modbus_read_coils_request", c, headers, start_address, quantity;
+}
+
+event modbus_read_coils_response(c: connection, headers: ModbusHeaders, coils: ModbusCoils)
+{
+    print "modbus_read_coils_response", c, headers, coils;
+}
+
+event modbus_read_discrete_inputs_request(c: connection, headers: ModbusHeaders, start_address: count, quantity: count)
+{
+    print "modbus_read_discrete_inputs_request", c, headers, start_address, quantity;
+}
+
+event modbus_read_discrete_inputs_response(c: connection, headers: ModbusHeaders, coils: ModbusCoils)
+{
+    print "modbus_read_discrete_inputs_response", c, headers, coils;
+}
+
+event modbus_read_holding_registers_request(c: connection, headers: ModbusHeaders, start_address: count, quantity: count)
+{
+    print "modbus_read_holding_registers_request", c, headers, start_address, quantity;
+}
+
+event modbus_read_holding_registers_response(c: connection, headers: ModbusHeaders, registers: ModbusRegisters)
+{
+    print "modbus_read_holding_registers_response", c, headers, registers;
+}
+
+event modbus_read_input_registers_request(c: connection, headers: ModbusHeaders, start_address: count, quantity: count)
+{
+    print "modbus_read_input_registers_request", c, headers, start_address, quantity;
+}
+
+event modbus_read_input_registers_response(c: connection, headers: ModbusHeaders, registers: ModbusRegisters)
+{
+    print "modbus_read_input_registers_response", c, headers, registers;
+}
+
+event modbus_write_single_coil_request(c: connection, headers: ModbusHeaders, address: count, value: bool)
+{
+    print "modbus_write_single_coil_request", c, headers, address, value;
+}
+
+event modbus_write_single_coil_response(c: connection, headers: ModbusHeaders, address: count, value: bool)
+{
+    print "modbus_write_single_coil_response", c, headers, address, value;
+}
+
+event modbus_write_single_register_request(c: connection, headers: ModbusHeaders, address: count, value: count)
+{
+    print "modbus_write_single_register_request", c, headers, address, value;
+}
+
+event modbus_write_single_register_response(c: connection, headers: ModbusHeaders, address: count, value: count)
+{
+    print "modbus_write_single_register_response", c, headers, address, value;
+}
+
+event modbus_write_multiple_coils_request(c: connection, headers: ModbusHeaders, start_address: count, coils: ModbusCoils)
+{
+    print "modbus_write_multiple_coils_request", c, headers, start_address, coils;
+}
+
+event modbus_write_multiple_coils_response(c: connection, headers: ModbusHeaders, start_address: count, quantity: count)
+{
+    print "modbus_write_multiple_coils_response", c, headers, start_address, quantity;
+}
+
+event modbus_write_multiple_registers_request(c: connection, headers: ModbusHeaders, start_address: count, registers: ModbusRegisters)
+{
+    print "modbus_write_multiple_registers_request", c, headers, start_address, registers;
+}
+
+event modbus_write_multiple_registers_response(c: connection, headers: ModbusHeaders, start_address: count, quantity: count)
+{
+    print "modbus_write_multiple_registers_response", c, headers, start_address, quantity;
+}
+
+event modbus_read_file_record_request(c: connection, headers: ModbusHeaders)
+{
+    print "modbus_read_file_record_request", c, headers;
+}
+
+event modbus_read_file_record_response(c: connection, headers: ModbusHeaders)
+{
+    print "modbus_read_file_record_response", c, headers;
+}
+
+event modbus_write_file_record_request(c: connection, headers: ModbusHeaders)
+{
+    print "modbus_write_file_record_request", c, headers;
+}
+
+event modbus_write_file_record_response(c: connection, headers: ModbusHeaders)
+{
+    print "modbus_write_file_record_response", c, headers;
+}
+
+event modbus_mask_write_register_request(c: connection, headers: ModbusHeaders, address: count, and_mask: count, or_mask: count)
+{
+    print "modbus_mask_write_register_request", c, headers, address, and_mask, or_mask;
+}
+
+event modbus_mask_write_register_response(c: connection, headers: ModbusHeaders, address: count, and_mask: count, or_mask: count)
+{
+    print "modbus_mask_write_register_response", c, headers, address, and_mask, or_mask;
+}
+
+event modbus_read_write_multiple_registers_request(c: connection, headers: ModbusHeaders, read_start_address: count, read_quantity: count, write_start_address: count, write_registers: ModbusRegisters)
+{
+    print "modbus_read_write_multiple_registers_request", c, headers, read_start_address, read_quantity, write_start_address, write_registers;
+}
+
+event modbus_read_write_multiple_registers_response(c: connection, headers: ModbusHeaders, written_registers: ModbusRegisters)
+{
+    print "modbus_read_write_multiple_registers_response", c, headers, written_registers;
+}
+
+event modbus_read_fifo_queue_request(c: connection, headers: ModbusHeaders, start_address: count)
+{
+    print "modbus_read_fifo_queue_request", c, headers, start_address;
+}
+
+event modbus_read_fifo_queue_response(c: connection, headers: ModbusHeaders, fifos: ModbusRegisters)
+{
+    print "modbus_read_fifo_queue_response", c, headers, fifos;
+}
+

testing/btest/scripts/base/protocols/modbus/policy.bro

@@ -0,0 +1,9 @@
+#
+# @TEST-EXEC: bro -r $TRACES/modbus.trace %INPUT 
+# @TEST-EXEC: btest-diff modbus.log
+# @TEST-EXEC: btest-diff modbus_register_change.log
+# @TEST-EXEC: btest-diff known_modbus.log
+#
+
+@load protocols/modbus/known-masters-slaves.bro
+@load protocols/modbus/track-memmap.bro