Merge branch 'master' into topic/jsiwek/gtp

CHANGES

@@ -1,4 +1,146 @@
 
+2.1-178 | 2012-11-23 19:35:32 -0800
+
+  * The ASCII writer now supports a new filter config option
+    "only_single_header_row" that turns the output into CSV format
+    when set to "T". (Carsten Langer)
+
+  * Add new function flavor called a "hook". This new flavor of
+    function behaves like a "synchronous event". See
+    doc/scripts/builtins.rst more details on usage. (Jon Siwek)
+
+  * Improve auto-generated enum documentation. The names of enum types
+    are tracked so that variables holding a value of a given enum type
+    can generate a reference to it instead of just listing the type as
+    a generic "enum". (Jon Siwek)
+
+2.1-171 | 2012-11-23 18:24:15 -0800
+
+  * Fix ambiguity between composite table index and record ctor
+    expressions. If a table type is "global t = table[conn_id, bool]
+    of count", then checking membership like "[c$id, is_orig] in t"
+    now works. Addresses #80. (Jon Siwek)
+
+2.1-169 | 2012-11-23 18:21:32 -0800
+
+  * Fix some warnings from sphinx when building docs. (Jon Siwek)
+
+2.1-167 | 2012-11-14 13:19:17 -0800
+
+  * Add a new BIF "bytestring_to_double" for converting from a binary
+    representation of a double. Addresses #908. (Carsten Langer/Daniel
+    Thayer)
+
+2.1-162 | 2012-11-13 17:29:00 -0800
+
+  * Fix modbus register array parsing. (Jon Siwek)
+
+  * Adjustments to modbus test cases. (Jon Siwek)
+
+2.1-157 | 2012-11-08 16:22:00 -0800
+
+  * Fix for lookup_hostname BIF. (Jon Siwek)
+
+  * Fix for modbus test portability. (Robin Sommer)
+
+2.1-152 | 2012-11-05 16:52:34 -0800
+
+  * Initial version of a completely reworked intelligence framework.
+    See doc/intel.rst for more information. (Seth Hall)
+
+  * Experimental Modbus analyzer. See policy/protocols/modbus/* for
+    example policies. (Dina Hadziosmanovic, Seth Hall)
+
+2.1-112 | 2012-11-05 13:58:20 -0800
+
+  * New base script for detecting cases of checksum offloading. 
+    Reporter messages will now tell if one has bad checksums. (Seth
+    Hall)
+
+  * Clarifying ownership rules for BroString constructors. (Robin
+    Sommer)
+
+2.1-109 | 2012-11-05 13:39:34 -0800
+
+  * Add detection rate threshold for MHR. (Vlad Grigorescu)
+    
+  * lookup_hostname_txt fixes. (Vlad Grigorescu)
+    
+2.1-104 | 2012-11-01 10:37:50 -0700
+
+  * A new built-in function lookup_hostname_txt() provides support for
+    DNS TXT queries. (Vlad Grigorescu)
+
+2.1-101 | 2012-10-31 14:30:26 -0700
+
+  * Documentation reorg: The install info has been consolidated into a
+    single document (INSTALL), the upgrade info has been moved from
+    the FAQ to a section in the install doc, and the "upgrading from
+    1.5 to 2.0" document has been updated (and renamed) to also
+    include 2.0 to 2.1 upgrade info. (Daniel Thayer)
+
+2.1-96 | 2012-10-31 14:23:50 -0700
+
+  * Renaming option defining the frequency of alarm summary mails to
+    'Logging::default_alarm_mail_interval'. (Daniel Thayer)
+
+2.1-91 | 2012-10-24 16:04:47 -0700
+
+  * Adding PPPoE support to Bro. (Seth Hall)
+
+2.1-87 | 2012-10-24 15:40:06 -0700
+
+  * Adding missing &redef for some TCP options. Addresses #905, #906,
+    #907. (Carsten Langer)
+
+2.1-86 | 2012-10-24 15:37:11 -0700
+
+  * Add parsing rules for IPv4/IPv6 subnet literal constants.
+    Addresses #888. (Jon Siwek)
+
+2.1-84 | 2012-10-19 15:12:56 -0700
+
+  * Added a BiF strptime() to wrap the corresponding C function. (Seth
+    Hall)
+
+2.1-82 | 2012-10-19 15:05:40 -0700
+
+  * Add IPv6 support to signature header conditions. (Jon Siwek)
+
+    - "src-ip" and "dst-ip" conditions can now use IPv6 addresses/subnets.
+      They must be written in colon-hexadecimal representation and enclosed
+      in square brackets (e.g. [fe80::1]).  Addresses #774.
+
+    - "icmp6" is now a valid protocol for use with "ip-proto" and "header"
+      conditions.  This allows signatures to be written that can match
+      against ICMPv6 payloads.  Addresses #880.
+
+    - "ip6" is now a valid protocol for use with the "header" condition.
+      (also the "ip-proto" condition, but it results in a no-op in that
+      case since signatures apply only to the inner-most IP packet when
+      packets are tunneled).  This allows signatures to match specifically
+      against IPv6 packets (whereas "ip" only matches against IPv4 packets).
+
+    - "ip-proto" conditions can now match against IPv6 packets.  Before,
+      IPv6 packets were just silently ignored which meant DPD based on
+      signatures did not function for IPv6 -- protocol analyzers would only
+      get attached to a connection over IPv6 based on the well-known ports
+      set in the "dpd_config" table.
+
+2.1-80 | 2012-10-19 14:48:42 -0700
+
+  * Change how "gridftp" gets added to service field of connection
+    records. In addition to checking for a finished SSL handshake over
+    an FTP connection, it now also requires that the SSL handshake
+    occurs after the FTP client requested AUTH GSSAPI, more
+    specifically identifying the characteristics of GridFTP control
+    channels. Addresses #891. (Jon Siwek)
+
+  * Allow faster rebuilds in certain cases. Previously, when
+    rebuilding with a different "--prefix" or "--scriptdir", all Bro
+    source files were recompiled.  With this change, only util.cc is
+    recompiled. (Daniel Thayer)
+
 2.1-76 | 2012-10-12 10:32:39 -0700
 
   * Add support for recognizing GridFTP connections as an extension to

INSTALL

@@ -1,35 +1,55 @@
+.. _CMake: http://www.cmake.org
+.. _SWIG: http://www.swig.org
+.. _Xcode: https://developer.apple.com/xcode/
+.. _MacPorts: http://www.macports.org
+.. _Fink: http://www.finkproject.org
+.. _Homebrew: http://mxcl.github.com/homebrew
+.. _bro downloads page: http://bro-ids.org/download/index.html
+
 ==============
 Installing Bro
 ==============
 
+Bro can be downloaded in either pre-built binary package or
+source code forms.
+
 Prerequisites
 =============
 
 Bro requires the following libraries and tools to be installed
 before you begin:
 
-    * CMake 2.6.3 or greater            http://www.cmake.org
+    * Libpcap                           http://www.tcpdump.org
 
-    * Perl  (used only during the Bro build process)
+    * OpenSSL libraries                 http://www.openssl.org
 
-    * Libpcap headers and libraries     http://www.tcpdump.org
+    * BIND8 library
-
-    * OpenSSL headers and libraries     http://www.openssl.org
-
-    * BIND8 headers and libraries
 
     * Libmagic
 
     * Libz
 
+    * Bash  (for BroControl)
+
+To build Bro from source, the following additional dependencies are required:
+
+    * CMake 2.6.3 or greater            http://www.cmake.org
+
     * SWIG                              http://www.swig.org
 
     * Bison (GNU Parser Generator)
 
     * Flex  (Fast Lexical Analyzer)
 
-    * Bash  (for BroControl)
+    * Libpcap headers                   http://www.tcpdump.org
 
+    * OpenSSL headers                   http://www.openssl.org
+
+    * libmagic headers
+
+    * zlib headers
+
+    * Perl
 
 Bro can make use of some optional libraries and tools if they are found at
 build time:
@@ -45,29 +65,161 @@ build time:
     * Ruby executable, library, and headers (for Broccoli Ruby bindings)
 
 
-Installation
+Installing From Pre-Built Binary Release Packages
-============
+=================================================
 
-To build and install into ``/usr/local/bro``::
+See the `bro downloads page`_ for currently supported/targeted platforms.
+
+* RPM
+
+  .. console::
+
+      sudo yum localinstall Bro-*.rpm
+
+* DEB
+
+  .. console::
+
+      sudo gdebi Bro-*.deb
+
+* MacOS Disk Image with Installer
+
+  Just open the ``Bro-*.dmg`` and then run the ``.pkg`` installer.
+  Everything installed by the package will go into ``/opt/bro``.
+
+The primary install prefix for binary packages is ``/opt/bro``.
+Non-MacOS packages that include BroControl also put variable/runtime
+data (e.g. Bro logs) in ``/var/opt/bro``.
+
+
+Installing From Source
+======================
+
+Required Dependencies
+~~~~~~~~~~~~~~~~~~~~~
+
+The following dependencies are required to build Bro:
+
+* RPM/RedHat-based Linux:
+
+  .. console::
+
+     sudo yum install cmake make gcc gcc-c++ flex bison libpcap-devel openssl-devel python-devel swig zlib-devel file-devel
+
+* DEB/Debian-based Linux:
+
+  .. console::
+
+     sudo apt-get install cmake make gcc g++ flex bison libpcap-dev libssl-dev python-dev swig zlib1g-dev libmagic-dev
+
+* FreeBSD
+
+  Most required dependencies should come with a minimal FreeBSD install
+  except for the following.
+
+  .. console::
+
+      sudo pkg_add -r bash cmake swig bison python
+
+  Note that ``bash`` needs to be in ``PATH``, which by default it is
+  not. The FreeBSD package installs the binary into
+  ``/usr/local/bin``.
+
+* Mac OS X
+
+  Compiling source code on Macs requires first downloading Xcode_,
+  then going through its "Preferences..." -> "Downloads" menus to
+  install the "Command Line Tools" component.
+
+  Lion (10.7) and Mountain Lion (10.8) come with all required
+  dependencies except for CMake_, SWIG_, and ``libmagic``.
+
+  Distributions of these dependencies can likely be obtained from your
+  preferred Mac OS X package management system (e.g. MacPorts_, Fink_,
+  or Homebrew_).
+
+  Specifically for MacPorts, the ``swig``, ``swig-ruby``, ``swig-python``
+  and ``file`` packages provide the required dependencies.
+
+Optional Dependencies
+~~~~~~~~~~~~~~~~~~~~~
+
+Bro can use libGeoIP for geo-locating IP addresses, and sendmail for
+sending emails.
+
+* RedHat Enterprise Linux:
+
+  .. console::
+
+      sudo yum install geoip-devel sendmail
+
+* CentOS Linux:
+
+  .. console::
+  
+      sudo yum install GeoIP-devel sendmail
+
+* DEB/Debian-based Linux:
+
+  .. console::
+
+      sudo apt-get install libgeoip-dev sendmail
+
+* Ports-based FreeBSD
+
+  .. console::
+
+      sudo pkg_add -r GeoIP
+
+  sendmail is typically already available.
+
+* Mac OS X
+
+  Vanilla OS X installations don't ship with libGeoIP, but
+  if installed from your preferred package management system (e.g. MacPorts,
+  Fink, or Homebrew), they should be automatically detected and Bro will
+  compile against them.
+
+Additional steps may be needed to :doc:`get the right GeoIP database <geoip>`.
+
+Compiling Bro Source Code
+~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Bro releases are bundled into source packages for convenience and
+available from the `bro downloads page`_.
+
+Alternatively, the latest Bro development version can be obtained through git
+repositories hosted at `git.bro-ids.org <http://git.bro-ids.org>`_.  See
+our `git development documentation
+<http://bro-ids.org/development/process.html>`_ for comprehensive
+information on Bro's use of git revision control, but the short story
+for downloading the full source code experience for Bro via git is:
+
+.. console::
+
+    git clone --recursive git://git.bro-ids.org/bro
+
+.. note:: If you choose to clone the ``bro`` repository non-recursively for
+   a "minimal Bro experience", be aware that compiling it depends on
+   BinPAC, which has its own ``binpac`` repository.  Either install it
+   first or initialize/update the cloned ``bro`` repository's
+   ``aux/binpac`` submodule.
+
+The typical way to build and install from source is (for more options,
+run ``./configure --help``):
+
+.. console::
 
     ./configure
     make
     make install
 
-This will first build Bro in a directory inside the distribution
+The default installation path is ``/usr/local/bro``, which would typically
-called ``build/``, using default build options. It then installs all
+require root privileges when doing the ``make install``.  A different 
-required files into ``/usr/local/bro``, including the Bro binary in
+installation path can be chosen by specifying the ``--prefix`` option.
-``/usr/local/bro/bin/bro``.
+Note that ``/usr`` and ``/opt/bro`` are the
-
+standard prefixes for binary Bro packages to be installed, so those are
-You can specify a different installation directory with::
+typically not good choices unless you are creating such a package.
-
-    ./configure --prefix=<dir>
-
-Note that ``/usr`` and ``/opt/bro`` are the standard prefixes for
-binary Bro packages to be installed, so those are typically not good
-choices unless you are creating such a package.
-
-Run ``./configure --help`` for more options.
 
 Depending on the Bro package you downloaded, there may be auxiliary
 tools and libraries available in the ``aux/`` directory. Some of them
@@ -81,6 +233,66 @@ OpenBSD users, please see our FAQ at
 http://www.bro-ids.org/documentation/faq.html if you are having
 problems installing Bro.
 
+
+Upgrading From a Previous Version of Bro
+========================================
+
+If you're doing an upgrade install (rather than a fresh install),
+there's two suggested approaches: either install Bro using the same
+installation prefix directory as before, or pick a new prefix and copy
+local customizations over.
+
+Re-Use Previous Install Prefix
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+If you choose to configure and install Bro with the same prefix
+directory as before, local customization and configuration to files in
+``$prefix/share/bro/site`` and ``$prefix/etc`` won't be overwritten
+(``$prefix`` indicating the root of where Bro was installed). Also, logs
+generated at run-time won't be touched by the upgrade. (But making
+a backup of local changes before upgrading is still recommended.)
+
+After upgrading, remember to check ``$prefix/share/bro/site`` and
+``$prefix/etc`` for ``.example`` files, which indicate the
+distribution's version of the file differs from the local one, which may
+include local changes.  Review the differences, and make adjustments
+as necessary (for differences that aren't the result of a local change,
+use the new version's).
+
+Pick a New Install prefix
+~~~~~~~~~~~~~~~~~~~~~~~~~
+
+If you want to install the newer version in a different prefix
+directory than before, you can just copy local customization and
+configuration files from ``$prefix/share/bro/site`` and ``$prefix/etc``
+to the new location (``$prefix`` indicating the root of where Bro was
+originally installed).  Make sure to review the files for difference
+before copying and make adjustments as necessary (for differences that
+aren't the result of a local change, use the new version's).  Of
+particular note, the copied version of ``$prefix/etc/broctl.cfg`` is
+likely to need changes to the ``SpoolDir`` and ``LogDir`` settings.
+
+
+Configure the Run-Time Environment
+==================================
+
+Just remember that you may need to adjust your ``PATH`` environment variable
+according to the platform/shell/package you're using.  For example:
+
+Bourne-Shell Syntax:
+
+.. console::
+
+   export PATH=/usr/local/bro/bin:$PATH
+
+C-Shell Syntax:
+
+.. console::
+
+   setenv PATH /usr/local/bro/bin:$PATH
+
+Or substitute ``/opt/bro/bin`` instead if you installed from a binary package.
+
 Running Bro
 ===========
 

NEWS

@@ -17,6 +17,11 @@ New Functionality
 
 - ssl.log now also records the subject client and issuer certificates.
 
+- Hooks: TODO: Briefly summarize the documention from
+  doc/scripts/builtins.rst here.
+
+- The ASCII writer can now output CSV files on a per filter basis.
+
 Changed Functionality
 ~~~~~~~~~~~~~~~~~~~~~
 
@@ -39,6 +44,11 @@ Changed Functionality
   completed, but also after the last event of a whole-file-read (or
   whole-db-read, etc.).
 
+- Renamed the option defining the frequency of alarm summary mails to
+  'Logging::default_alarm_mail_interval'. When using BroControl, the
+  value can now be set with the new broctl.cfg option
+  "MailAlarmsInterval".
+
 Bro 2.1
 -------
 

VERSION

@@ -1 +1 @@
-2.1-76
+2.1-179

aux/binpac

@@ -1 +1 @@
-Subproject commit 74e6a5401c4228d5293c0e309283f43c389e7c12
+Subproject commit 2fd9086c9dc0e76f6ff1ae04a60cbbce60507aab

aux/bro-aux

@@ -1 +1 @@
-Subproject commit 01bb93cb23f31a98fb400584e8d2f2fbe8a589ef
+Subproject commit bea556198b69d30d64c0cf1b594e6de71176df6f

aux/broccoli

@@ -1 +1 @@
-Subproject commit 907210ce1470724fb386f939cc1b10a4caa2ae39
+Subproject commit a8846fc5b004ffe4e3d00e826d0077ba19518192

aux/broctl

@@ -1 +1 @@
-Subproject commit b8cbd5a46fd275c900b5c67f4c6abd5785b83a8a
+Subproject commit 834131cd0ec0f63cce9de818726fe6167dedbf34

aux/btest

@@ -1 +1 @@
-Subproject commit 44a43e62452302277f88e8fac08d1f979dc53f98
+Subproject commit d83e10c5f76cbfdf81c843575351fbc7b544fc93

doc/faq.rst

@@ -12,43 +12,6 @@ Frequently Asked Questions
 Installation and Configuration
 ==============================
 
-How do I upgrade to a new version of Bro?
------------------------------------------
-
-There's two suggested approaches, either install Bro using the same
-installation prefix directory as before, or pick a new prefix and copy
-local customizations over.
-
-Re-Use Previous Install Prefix
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-
-If you choose to configure and install Bro with the same prefix
-directory as before, local customization and configuration to files in
-``$prefix/share/bro/site`` and ``$prefix/etc`` won't be overwritten
-(``$prefix`` indicating the root of where Bro was installed). Also, logs
-generated at run-time won't be touched by the upgrade. (But making
-a backup of local changes before proceeding is still recommended.)
-
-After upgrading, remember to check ``$prefix/share/bro/site`` and
-``$prefix/etc`` for ``.example`` files, which indicate the
-distribution's version of the file differs from the local one, which may
-include local changes.  Review the differences, and make adjustments
-as necessary (for differences that aren't the result of a local change,
-use the new version's).
-
-Pick a New Install prefix
-^^^^^^^^^^^^^^^^^^^^^^^^^
-
-If you want to install the newer version in a different prefix
-directory than before, you can just copy local customization and
-configuration files from ``$prefix/share/bro/site`` and ``$prefix/etc``
-to the new location (``$prefix`` indicating the root of where Bro was
-originally installed).  Make sure to review the files for difference
-before copying and make adjustments as necessary (for differences that
-aren't the result of a local change, use the new version's).  Of
-particular note, the copied version of ``$prefix/etc/broctl.cfg`` is
-likely to need changes to the ``SpoolDir`` and ``LogDir`` settings.
-
 How can I tune my operating system for best capture performance?
 ----------------------------------------------------------------
 

doc/index.rst

@@ -11,8 +11,8 @@ Guides
    :maxdepth: 1
 
    INSTALL
-   quickstart
    upgrade
+   quickstart
    faq
    reporting-problems
 

doc/intel.rst

@@ -0,0 +1,125 @@
+Intel Framework
+===============
+
+Intro
+-----
+
+Intelligence data is critical to the process of monitoring for
+security purposes.  There is always data which will be discovered
+through the incident response process and data which is shared through
+private communities.  The goals of Bro's Intelligence Framework are to
+consume that data, make it available for matching, and provide
+infrastructure around improving performance, memory utilization, and
+generally making all of this easier.
+
+Data in the Intelligence Framework is the atomic piece of intelligence
+such as an IP address or an e-mail address along with a suite of
+metadata about it such as a freeform source field, a freeform
+descriptive field and a URL which might lead to more information about
+the specific item.  The metadata in the default scripts has been
+deliberately kept minimal so that the community can find the
+appropriate fields that need added by writing scripts which extend the
+base record using the normal record extension mechanism.
+
+Quick Start
+-----------
+
+Load the package of scripts that sends data into the Intelligence
+Framework to be checked by loading this script in local.bro::
+
+	@load policy/frameworks/intel
+
+(TODO: find some good mechanism for getting setup with good data
+quickly)
+
+Refer to the "Loading Intelligence" section below to see the format
+for Intelligence Framework text files, then load those text files with
+this line in local.bro::
+
+	redef Intel::read_files += { "/somewhere/yourdata.txt" };
+
+The data itself only needs to reside on the manager if running in a
+cluster.
+
+Architecture
+------------
+
+The Intelligence Framework can be thought of as containing three
+separate portions.  The first part is how intelligence is loaded,
+followed by the mechanism for indicating to the intelligence framework
+that a piece of data which needs to be checked has been seen, and
+thirdly the part where a positive match has been discovered.
+
+Loading Intelligence
+********************
+
+Intelligence data can only be loaded through plain text files using
+the Input Framework conventions.  Additionally, on clusters the
+manager is the only node that needs the intelligence data.  The
+intelligence framework has distribution mechanisms which will push
+data out to all of the nodes that need it.
+
+Here is an example of the intelligence data format.  Note that all
+whitespace separators are literal tabs and fields containing only a
+hyphen a considered to be null values.::
+
+	#fields	host	net	str	str_type	meta.source	meta.desc	meta.url
+	1.2.3.4	-	-	-	source1	Sending phishing email	http://source1.com/badhosts/1.2.3.4
+	-	31.131.248.0/21	-	-	spamhaus-drop	SBL154982	-	-
+	-	-	a.b.com	Intel::DOMAIN	source2	Name used for data exfiltration	-
+
+For more examples of built in `str_type` values, please refer to the
+autogenerated documentation for the intelligence framework (TODO:
+figure out how to do this link).
+
+To load the data once files are created, use the following example
+code to define files to load with your own file names of course::
+
+	redef Intel::read_files += {
+		"/somewhere/feed1.txt",
+		"/somewhere/feed2.txt",
+	};
+
+Remember, the files only need to be present on the file system of the
+manager node on cluster deployments.
+
+Seen Data
+*********
+
+When some bit of data is extracted (such as an email address in the
+"From" header in a message over SMTP), the Intelligence Framework
+needs to be informed that this data was discovered and it's presence
+should be checked within the intelligence data set.  This is
+accomplished through the Intel::seen (TODO: do a reference link)
+function.
+
+Typically users won't need to work with this function due to built in
+hook scripts that Bro ships with that will "see" data and send it into
+the intelligence framework.  A user may only need to load the entire
+package of hook scripts as a module or pick and choose specific
+scripts to load.  Keep in mind that as more data is sent into the
+intelligence framework, the CPU load consumed by Bro will increase
+depending on how many times the Intel::seen function is being called
+which is heavily traffic dependent.
+
+The full package of hook scripts that Bro ships with for sending this
+"seen" data into the intelligence framework can be loading by adding
+this line to local.bro::
+
+	@load policy/frameworks/intel
+
+Intelligence Matches
+********************
+
+Against all hopes, most networks will eventually have a hit on
+intelligence data which could indicate a possible compromise or other
+unwanted activity.  The Intelligence Framework provides an event that
+is generated whenever a match is discovered named Intel::match (TODO:
+make a link to inline docs).  Due to design restrictions placed upon
+the intelligence framework, there is no assurance as to where this
+event will be generated.  It could be generated on the worker where
+the data was seen or on the manager.  When the Intel::match event is
+handled, only the data given as event arguments to the event can be
+assured since the host where the data was seen may not be where
+Intel::match is handled.
+

doc/quickstart.rst

@@ -1,10 +1,3 @@
-.. _CMake: http://www.cmake.org
-.. _SWIG: http://www.swig.org
-.. _Xcode: https://developer.apple.com/xcode/
-.. _MacPorts: http://www.macports.org
-.. _Fink: http://www.finkproject.org
-.. _Homebrew: http://mxcl.github.com/homebrew
-.. _bro downloads page: http://bro-ids.org/download/index.html
 
 =================
 Quick Start Guide
@@ -23,181 +16,11 @@ Installation
 
 Bro works on most modern, Unix-based systems and requires no custom
 hardware.  It can be downloaded in either pre-built binary package or
-source code forms.
+source code forms.  See :doc:`Installing Bro <INSTALL>` for instructions
+on how to install Bro.
 
-Pre-Built Binary Release Packages
+.. note:: Below, ``$PREFIX`` is used to reference the Bro installation
----------------------------------
+   root directory.
-
-See the `bro downloads page`_ for currently supported/targeted platforms.
-
-* RPM
-
-  .. console::
-
-      sudo yum localinstall Bro-*.rpm
-
-* DEB
-
-  .. console::
-
-      sudo gdebi Bro-*.deb
-
-* MacOS Disk Image with Installer
-
-  Just open the ``Bro-*.dmg`` and then run the ``.pkg`` installer.
-  Everything installed by the package will go into ``/opt/bro``.
-
-The primary install prefix for binary packages is ``/opt/bro``.
-Non-MacOS packages that include BroControl also put variable/runtime
-data (e.g. Bro logs) in ``/var/opt/bro``.
-
-Building From Source
---------------------
-
-Required Dependencies
-~~~~~~~~~~~~~~~~~~~~~
-
-The following dependencies are required to build Bro:
-
-* RPM/RedHat-based Linux:
-
-  .. console::
-
-     sudo yum install cmake make gcc gcc-c++ flex bison libpcap-devel openssl-devel python-devel swig zlib-devel file-devel
-
-* DEB/Debian-based Linux:
-
-  .. console::
-
-     sudo apt-get install cmake make gcc g++ flex bison libpcap-dev libssl-dev python-dev swig zlib1g-dev libmagic-dev
-
-* FreeBSD
-
-  Most required dependencies should come with a minimal FreeBSD install
-  except for the following.
-
-  .. console::
-
-      sudo pkg_add -r bash cmake swig bison python
-
-  Note that ``bash`` needs to be in ``PATH``, which by default it is
-  not. The FreeBSD package installs the binary into
-  ``/usr/local/bin``.
-
-* Mac OS X
-
-  Compiling source code on Macs requires first downloading Xcode_,
-  then going through its "Preferences..." -> "Downloads" menus to
-  install the "Command Line Tools" component.
-
-  Lion (10.7) and Mountain Lion (10.8) come with all required
-  dependencies except for CMake_, SWIG_, and ``libmagic``.
-
-  Distributions of these dependencies can be obtained from the project
-  websites linked above, but they're also likely available from your
-  preferred Mac OS X package management system (e.g. MacPorts_, Fink_,
-  or Homebrew_).
-
-  Specifically for MacPorts, the ``swig``, ``swig-ruby``, ``swig-python``
-  and ``file`` packages provide the required dependencies.
-
-Optional Dependencies
-~~~~~~~~~~~~~~~~~~~~~
-
-Bro can use libGeoIP for geo-locating IP addresses, and sendmail for
-sending emails.
-
-* RedHat Enterprise Linux:
-
-  .. console::
-
-      sudo yum install geoip-devel sendmail
-
-* CentOS Linux:
-
-  .. console::
-  
-      sudo yum install GeoIP-devel sendmail
-
-* DEB/Debian-based Linux:
-
-  .. console::
-
-      sudo apt-get install libgeoip-dev sendmail
-
-* Ports-based FreeBSD
-
-  .. console::
-
-      sudo pkg_add -r GeoIP
-
-  sendmail is typically already available.
-
-* Mac OS X
-
-  Vanilla OS X installations don't ship with libmagic or libGeoIP, but
-  if installed from your preferred package management system (e.g. MacPorts,
-  Fink, or Homebrew), they should be automatically detected and Bro will compile
-  against them.
-
-Additional steps may be needed to :doc:`get the right GeoIP database <geoip>`
-
-Compiling Bro Source Code
-~~~~~~~~~~~~~~~~~~~~~~~~~
-
-Bro releases are bundled into source packages for convenience and
-available from the `bro downloads page`_.
-
-The latest Bro development versions are obtainable through git
-repositories hosted at `git.bro-ids.org <http://git.bro-ids.org>`_.  See
-our `git development documentation
-<http://bro-ids.org/development/process.html>`_ for comprehensive
-information on Bro's use of git revision control, but the short story
-for downloading the full source code experience for Bro via git is:
-
-.. console::
-
-    git clone --recursive git://git.bro-ids.org/bro
-
-.. note:: If you choose to clone the ``bro`` repository non-recursively for
-   a "minimal Bro experience", be aware that compiling it depends on
-   BinPAC, which has its own ``binpac`` repository.  Either install it
-   first or initialize/update the cloned ``bro`` repository's
-   ``aux/binpac`` submodule.
-
-See the ``INSTALL`` file included with the source code for more information
-on compiling, but this is the typical way to build and install from source
-(of course, changing the value of the ``--prefix`` option to point to the
-desired root install path):
-
-.. console::
-
-    ./configure --prefix=/desired/install/path
-    make
-    make install
-
-The default installation prefix is ``/usr/local/bro``, which would typically
-require root privileges when doing the ``make install``.
-
-Configure the Run-Time Environment
-----------------------------------
-
-Just remember that you may need to adjust your ``PATH`` environment variable
-according to the platform/shell/package you're using.  For example:
-
-Bourne-Shell Syntax:
-
-.. console::
-
-   export PATH=/usr/local/bro/bin:$PATH
-
-C-Shell Syntax:
-
-.. console::
-
-   setenv PATH /usr/local/bro/bin:$PATH
-
-Or substitute ``/opt/bro/bin`` instead if you installed from a binary package.
 
 Using BroControl
 ================
@@ -206,9 +29,6 @@ BroControl is an interactive shell for easily operating/managing Bro
 installations on a single system or even across multiple systems in a
 traffic-monitoring cluster.
 
-.. note:: Below, ``$PREFIX`` is used to reference the Bro installation
-   root directory.
-
 A Minimal Starting Configuration
 --------------------------------
 

doc/scripts/DocSourcesList.cmake

@@ -36,6 +36,8 @@ rest_target(${psd} base/frameworks/input/main.bro)
 rest_target(${psd} base/frameworks/input/readers/ascii.bro)
 rest_target(${psd} base/frameworks/input/readers/benchmark.bro)
 rest_target(${psd} base/frameworks/input/readers/raw.bro)
+rest_target(${psd} base/frameworks/intel/cluster.bro)
+rest_target(${psd} base/frameworks/intel/input.bro)
 rest_target(${psd} base/frameworks/intel/main.bro)
 rest_target(${psd} base/frameworks/logging/main.bro)
 rest_target(${psd} base/frameworks/logging/postprocessors/scp.bro)
@@ -62,6 +64,7 @@ rest_target(${psd} base/frameworks/reporter/main.bro)
 rest_target(${psd} base/frameworks/signatures/main.bro)
 rest_target(${psd} base/frameworks/software/main.bro)
 rest_target(${psd} base/frameworks/tunnels/main.bro)
+rest_target(${psd} base/misc/find-checksum-offloading.bro)
 rest_target(${psd} base/protocols/conn/contents.bro)
 rest_target(${psd} base/protocols/conn/inactivity.bro)
 rest_target(${psd} base/protocols/conn/main.bro)
@@ -79,6 +82,8 @@ rest_target(${psd} base/protocols/http/main.bro)
 rest_target(${psd} base/protocols/http/utils.bro)
 rest_target(${psd} base/protocols/irc/dcc-send.bro)
 rest_target(${psd} base/protocols/irc/main.bro)
+rest_target(${psd} base/protocols/modbus/consts.bro)
+rest_target(${psd} base/protocols/modbus/main.bro)
 rest_target(${psd} base/protocols/smtp/entities-excerpt.bro)
 rest_target(${psd} base/protocols/smtp/entities.bro)
 rest_target(${psd} base/protocols/smtp/main.bro)
@@ -100,11 +105,21 @@ rest_target(${psd} base/utils/patterns.bro)
 rest_target(${psd} base/utils/site.bro)
 rest_target(${psd} base/utils/strings.bro)
 rest_target(${psd} base/utils/thresholds.bro)
+rest_target(${psd} base/utils/urls.bro)
 rest_target(${psd} policy/frameworks/communication/listen.bro)
 rest_target(${psd} policy/frameworks/control/controllee.bro)
 rest_target(${psd} policy/frameworks/control/controller.bro)
 rest_target(${psd} policy/frameworks/dpd/detect-protocols.bro)
 rest_target(${psd} policy/frameworks/dpd/packet-segment-logging.bro)
+rest_target(${psd} policy/frameworks/intel/conn-established.bro)
+rest_target(${psd} policy/frameworks/intel/dns.bro)
+rest_target(${psd} policy/frameworks/intel/http-host-header.bro)
+rest_target(${psd} policy/frameworks/intel/http-url.bro)
+rest_target(${psd} policy/frameworks/intel/http-user-agents.bro)
+rest_target(${psd} policy/frameworks/intel/smtp-url-extraction.bro)
+rest_target(${psd} policy/frameworks/intel/smtp.bro)
+rest_target(${psd} policy/frameworks/intel/ssl.bro)
+rest_target(${psd} policy/frameworks/intel/where-locations.bro)
 rest_target(${psd} policy/frameworks/metrics/conn-example.bro)
 rest_target(${psd} policy/frameworks/metrics/http-example.bro)
 rest_target(${psd} policy/frameworks/metrics/ssl-example.bro)
@@ -112,6 +127,7 @@ rest_target(${psd} policy/frameworks/software/version-changes.bro)
 rest_target(${psd} policy/frameworks/software/vulnerable.bro)
 rest_target(${psd} policy/integration/barnyard2/main.bro)
 rest_target(${psd} policy/integration/barnyard2/types.bro)
+rest_target(${psd} policy/integration/collective-intel/main.bro)
 rest_target(${psd} policy/misc/analysis-groups.bro)
 rest_target(${psd} policy/misc/capture-loss.bro)
 rest_target(${psd} policy/misc/loaded-scripts.bro)
@@ -126,7 +142,6 @@ rest_target(${psd} policy/protocols/dns/detect-external-names.bro)
 rest_target(${psd} policy/protocols/ftp/detect.bro)
 rest_target(${psd} policy/protocols/ftp/software.bro)
 rest_target(${psd} policy/protocols/http/detect-MHR.bro)
-rest_target(${psd} policy/protocols/http/detect-intel.bro)
 rest_target(${psd} policy/protocols/http/detect-sqli.bro)
 rest_target(${psd} policy/protocols/http/detect-webapps.bro)
 rest_target(${psd} policy/protocols/http/header-names.bro)
@@ -134,6 +149,8 @@ rest_target(${psd} policy/protocols/http/software-browser-plugins.bro)
 rest_target(${psd} policy/protocols/http/software.bro)
 rest_target(${psd} policy/protocols/http/var-extraction-cookies.bro)
 rest_target(${psd} policy/protocols/http/var-extraction-uri.bro)
+rest_target(${psd} policy/protocols/modbus/known-masters-slaves.bro)
+rest_target(${psd} policy/protocols/modbus/track-memmap.bro)
 rest_target(${psd} policy/protocols/smtp/blocklists.bro)
 rest_target(${psd} policy/protocols/smtp/detect-suspicious-orig.bro)
 rest_target(${psd} policy/protocols/smtp/software.bro)

doc/scripts/builtins.rst

@@ -417,10 +417,6 @@ The Bro scripting language supports the following built-in types.
     Writing to files like this for logging usually isn't recommended, for better
     logging support see :doc:`/logging`.
 
-.. bro:type:: func
-
-    See :bro:type:`function`.
-
 .. bro:type:: function
 
     Function types in Bro are declared using::
@@ -504,6 +500,74 @@ The Bro scripting language supports the following built-in types.
     identifier and the body of each will be executed in turn.  Ordering
     of execution can be influenced with :bro:attr:`&priority`.
 
+.. bro:type:: hook
+
+    A hook is another flavor of function that shares characteristics of
+    both a :bro:type:`function` and a :bro:type:`event`.  They are like
+    events in that many handler bodies can be defined for the same hook
+    identifier, they have no return vale, and the order of execution
+    can be enforced with :bro:attr:`&priority`.  They are more like
+    functions in the way they are invoked/called, because, unlike
+    events, their execution is immediate and they do not get scheduled
+    through an event queue.  Also, a unique feature of a hook is that
+    a given hook handler body can short-circuit the execution of
+    remaining hook handlers simply by exiting from the body as a result
+    of a ``break`` statement (as opposed to a ``return`` or just
+    reaching the end of the body).
+
+    A hook type is declared like::
+
+        hook( argument* )
+
+    where *argument* is a (possibly empty) comma-separated list of
+    arguments.  For example:
+
+    .. code:: bro
+
+        global myhook: hook(s: string)
+
+    Here ``myhook`` is the hook type identifier and no hook handler
+    bodies have been defined for it yet.  To define some hook handler
+    bodies the syntax looks like:
+
+    .. code:: bro
+
+        hook myhook(s: string) &priority=10
+            {
+            print "priority 10 myhook handler", s;
+            s = "bye";
+            }
+
+        hook myhook(s: string)
+            {
+            print "break out of myhook handling", s;
+            break;
+            }
+
+        hook myhook(s: string) &priority=-5
+            {
+            print "not going to happen", s;
+            }
+
+    Note that, although the first (forward) declaration of ``myhook`` as
+    a hook type isn't strictly required, when it is provided, the
+    argument types must match.
+
+    To invoke immediate execution of all hook handler bodies, a ``hook``
+    statement must be used:
+
+    .. code:: bro
+
+        hook myhook("hi");
+
+    And the output would like like::
+
+        priority 10 myhook handler, hi
+        break out of myhook handling, bye
+
+    Note how the modification to arguments can be seen by remaining
+    hook handlers.
+
 Attributes
 ----------
 

doc/signatures.rst

@@ -83,9 +83,8 @@ Header Conditions
 ~~~~~~~~~~~~~~~~~
 
 Header conditions limit the applicability of the signature to a subset
-of traffic that contains matching packet headers. For TCP, this match
+of traffic that contains matching packet headers.  This type of matching
-is performed only for the first packet of a connection. For other
+is performed only for the first packet of a connection.
-protocols, it is done on each individual packet.
 
 There are pre-defined header conditions for some of the most used
 header fields. All of them generally have the format ``<keyword> <cmp>
@@ -95,14 +94,22 @@ one of ``==``, ``!=``, ``<``, ``<=``, ``>``, ``>=``; and
 against. The following keywords are defined:
 
 ``src-ip``/``dst-ip <cmp> <address-list>``
-    Source and destination address, respectively. Addresses can be
+    Source and destination address, respectively. Addresses can be given
-    given as IP addresses or CIDR masks.
+    as IPv4 or IPv6 addresses or CIDR masks.  For IPv6 addresses/masks
+    the colon-hexadecimal representation of the address must be enclosed
+    in square brackets (e.g. ``[fe80::1]`` or ``[fe80::0]/16``).
 
-``src-port``/``dst-port`` ``<int-list>``
+``src-port``/``dst-port <cmp> <int-list>``
     Source and destination port, respectively.
 
-``ip-proto tcp|udp|icmp``
+``ip-proto <cmp> tcp|udp|icmp|icmp6|ip|ip6``
-    IP protocol.
+    IPv4 header's Protocol field or the Next Header field of the final
+    IPv6 header (i.e. either Next Header field in the fixed IPv6 header
+    if no extension headers are present or that field from the last
+    extension header in the chain).  Note that the IP-in-IP forms of
+    tunneling are automatically decapsulated by default and signatures
+    apply to only the inner-most packet, so specifying ``ip`` or ``ip6``
+    is a no-op.
 
 For lists of multiple values, they are sequentially compared against
 the corresponding header field. If at least one of the comparisons
@@ -116,20 +123,22 @@ condition can be defined either as
 
     header <proto>[<offset>:<size>] [& <integer>] <cmp> <value-list>
 
-This compares the value found at the given position of the packet
+This compares the value found at the given position of the packet header
-header with a list of values. ``offset`` defines the position of the
+with a list of values. ``offset`` defines the position of the value
-value within the header of the protocol defined by ``proto`` (which
+within the header of the protocol defined by ``proto`` (which can be
-can be ``ip``, ``tcp``, ``udp`` or ``icmp``). ``size`` is either 1, 2,
+``ip``, ``ip6``, ``tcp``, ``udp``, ``icmp`` or ``icmp6``). ``size`` is
-or 4 and specifies the value to have a size of this many bytes. If the
+either 1, 2, or 4 and specifies the value to have a size of this many
-optional ``& <integer>`` is given, the packet's value is first masked
+bytes. If the optional ``& <integer>`` is given, the packet's value is
-with the integer before it is compared to the value-list. ``cmp`` is
+first masked with the integer before it is compared to the value-list.
-one of ``==``, ``!=``, ``<``, ``<=``, ``>``, ``>=``. ``value-list`` is
+``cmp`` is one of ``==``, ``!=``, ``<``, ``<=``, ``>``, ``>=``.
-a list of comma-separated integers similar to those described above.
+``value-list`` is a list of comma-separated integers similar to those
-The integers within the list may be followed by an additional ``/
+described above.  The integers within the list may be followed by an
-mask`` where ``mask`` is a value from 0 to 32. This corresponds to the
+additional ``/ mask`` where ``mask`` is a value from 0 to 32. This
-CIDR notation for netmasks and is translated into a corresponding
+corresponds to the CIDR notation for netmasks and is translated into a
-bitmask applied to the packet's value prior to the comparison (similar
+corresponding bitmask applied to the packet's value prior to the
-to the optional ``& integer``).
+comparison (similar to the optional ``& integer``).  IPv6 address values
+are not allowed in the value-list, though you can still inspect any 1,
+2, or 4 byte section of an IPv6 header using this keyword.
 
 Putting it all together, this is an example condition that is
 equivalent to ``dst-ip == 1.2.3.4/16, 5.6.7.8/24``:
@@ -138,8 +147,8 @@ equivalent to ``dst-ip == 1.2.3.4/16, 5.6.7.8/24``:
 
     header ip[16:4] == 1.2.3.4/16, 5.6.7.8/24
 
-Internally, the predefined header conditions are in fact just
+Note that the analogous example for IPv6 isn't currently possible since
-short-cuts and mapped into a generic condition.
+4 bytes is the max width of a value that can be compared.
 
 Content Conditions
 ~~~~~~~~~~~~~~~~~~

doc/upgrade.rst

@@ -1,19 +1,80 @@
 
-=============================
+==========================================
-Upgrading From Bro 1.5 to 2.0
+Upgrading From the Previous Version of Bro
-=============================
+==========================================
 
 .. rst-class:: opening
 
-   This guide details differences between Bro versions 1.5 and 2.0
+   This guide details specific differences between Bro versions
    that may be important for users to know as they work on updating
    their Bro deployment/configuration to the later version.
 
 .. contents::
 
 
-Introduction
+Upgrading From Bro 2.0 to 2.1
-============
+=============================
+
+In Bro 2.1, IPv6 is enabled by default.  Therefore, when building Bro from
+source, the "--enable-brov6" configure option has been removed because it
+is no longer relevant.  
+
+Other configure changes include renaming the "--enable-perftools" option
+to "--enable-perftools-debug" to indicate that the option is only relevant
+for debugging the heap.  One other change involves what happens when
+tcmalloc (part of Google perftools) is found at configure time.  On Linux,
+it will automatically be linked with Bro, but on other platforms you
+need to use the "--enable-perftools" option to enable linking to tcmalloc.
+
+There are a couple of changes to the Bro scripting language to better
+support IPv6. First, IPv6 literals appearing in a Bro script must now be
+enclosed in square brackets (for example, ``[fe80::db15]``). For subnet
+literals, the slash "/" appears after the closing square bracket (for
+example, ``[fe80:1234::]/32``). Second, when an IP address variable or IP
+address literal is enclosed in pipes (for example, ``|[fe80::db15]|``) the
+result is now the size of the address in bits (32 for IPv4 and 128 for IPv6).
+
+In the Bro scripting language, "match" and "using" are no longer reserved
+keywords.
+
+Some built-in functions have been removed: "addr_to_count" (use
+"addr_to_counts" instead), "bro_has_ipv6" (this is no longer relevant
+because Bro now always supports IPv6), "active_connection" (use
+"connection_exists" instead), and "connection_record" (use "lookup_connection"
+instead).
+
+The "NFS3::mode2string" built-in function has been renamed to "file_mode".
+
+Some built-in functions have been changed:  "exit" (now takes the exit code
+as a parameter), "to_port" (now takes a string as parameter instead
+of a count and transport protocol, but "count_to_port" is still available),
+"connect" (now takes an additional string parameter specifying the zone of
+a non-global IPv6 address), and "listen" (now takes three additional
+parameters to enable listening on IPv6 addresses).
+
+Some Bro script variables have been renamed:  "LogAscii::header_prefix"
+has been renamed to "LogAscii::meta_prefix", "LogAscii::include_header"
+has been renamed to "LogAscii::include_meta".
+
+Some Bro script variables have been removed: "tunnel_port",
+"parse_udp_tunnels", "use_connection_compressor", "cc_handle_resets",
+"cc_handle_only_syns", and "cc_instantiate_on_data".
+
+A couple events have changed:  the "icmp_redirect" event now includes
+the target and destination addresses and any Neighbor Discovery options
+in the message, and the last parameter of the "dns_AAAA_reply" event has
+been removed because it was unused.
+
+The format of the ASCII log files has changed very slightly.  Two new lines
+are automatically added, one to record the time when the log was opened,
+and the other to record the time when the log was closed.
+
+In BroControl, the option (in broctl.cfg) "CFlowAddr" was renamed
+to "CFlowAddress".
+
+
+Upgrading From Bro 1.5 to 2.0
+=============================
 
 As the version number jump suggests, Bro 2.0 is a major upgrade and
 lots of things have changed. Most importantly, we have rewritten

scripts/base/frameworks/cluster/nodes/manager.bro

@@ -13,8 +13,12 @@
 ## Turn off remote logging since this is the manager and should only log here.
 redef Log::enable_remote_logging = F;
 
+## Log rotation interval.
 redef Log::default_rotation_interval = 1 hrs;
 
+## Alarm summary mail interval.
+redef Log::default_mail_alarms_interval = 24 hrs;
+
 ## Use the cluster's archive logging script.
 redef Log::default_rotation_postprocessor_cmd = "archive-log";
 

scripts/base/frameworks/intel/__load__.bro

@@ -1 +1,11 @@
 @load ./main
+
+# The cluster framework must be loaded first.
+@load base/frameworks/cluster
+
+@if ( Cluster::is_enabled() )
+@load ./cluster
+@endif
+
+# This needs cluster support to only read on the manager.
+@load ./input

scripts/base/frameworks/intel/cluster.bro

@@ -0,0 +1,61 @@
+##! Cluster transparency support for the intelligence framework.  This is mostly oriented
+##! toward distributing intelligence information across clusters.
+
+@load base/frameworks/cluster
+@load ./input
+
+module Intel;
+
+redef record Item += {
+	## This field is used internally for cluster transparency to avoid 
+	## re-dispatching intelligence items over and over from workers.
+	first_dispatch: bool &default=T;
+};
+
+# If this process is not a manager process, we don't want the full metadata
+@if ( Cluster::local_node_type() != Cluster::MANAGER )
+redef have_full_data = F;
+@endif
+
+global cluster_new_item: event(item: Item);
+
+# Primary intelligence distribution comes from manager.
+redef Cluster::manager2worker_events += /^Intel::(cluster_new_item)$/;
+# If a worker finds intelligence and adds it, it should share it back to the manager.
+redef Cluster::worker2manager_events += /^Intel::(cluster_new_item|match_no_items)$/;
+
+@if ( Cluster::local_node_type() == Cluster::MANAGER )
+event Intel::match_no_items(s: Seen) &priority=5
+	{
+	event Intel::match(s, Intel::get_items(s));
+	}
+
+event remote_connection_handshake_done(p: event_peer)
+	{
+	# When a worker connects, send it the complete minimal data store.
+	# It will be kept up to date after this by the cluster_new_item event.
+	if ( Cluster::nodes[p$descr]$node_type == Cluster::WORKER )
+		{
+		send_id(p, "Intel::min_data_store");
+		}
+	}
+@endif
+
+event Intel::cluster_new_item(item: Intel::Item) &priority=5
+	{
+	# Ignore locally generated events to avoid event storms.
+	if ( is_remote_event() )
+		Intel::insert(item);
+	}
+
+event Intel::new_item(item: Intel::Item) &priority=5
+	{
+	# The cluster manager always rebroadcasts intelligence.
+	# Workers redistribute it if it was locally generated.
+	if ( Cluster::local_node_type() == Cluster::MANAGER ||
+	     item$first_dispatch )
+		{
+		item$first_dispatch=F;
+		event Intel::cluster_new_item(item);
+		}
+	}

scripts/base/frameworks/intel/input.bro

@@ -0,0 +1,33 @@
+@load ./main
+
+module Intel;
+
+export {
+	## Intelligence files that will be read off disk.  The files are
+	## reread everytime they are updated so updates much be atomic with
+	## "mv" instead of writing the file in place.
+	const read_files: set[string] = {} &redef;
+}
+
+event Intel::read_entry(desc: Input::EventDescription, tpe: Input::Event, item: Intel::Item)
+	{
+	Intel::insert(item);
+	}
+
+event bro_init() &priority=5
+	{
+	if ( ! Cluster::is_enabled() ||
+	     Cluster::local_node_type() == Cluster::MANAGER )
+		{
+		for ( a_file in read_files )
+			{
+			Input::add_event([$source=a_file,
+			                  $reader=Input::READER_ASCII,
+			                  $mode=Input::REREAD,
+			                  $name=cat("intel-", a_file),
+			                  $fields=Intel::Item,
+			                  $ev=Intel::read_entry]);
+			}
+		}
+	}
+

scripts/base/frameworks/intel/main.bro

@@ -1,323 +1,345 @@
 ##! The intelligence framework provides a way to store and query IP addresses,
-##! strings (with a subtype), and numeric (with a subtype) data.  Metadata 
+##! and strings (with a str_type).  Metadata can
-##! also be associated with the intelligence like tags which are arbitrary
+##! also be associated with the intelligence like for making more informed
-##! strings, time values, and longer descriptive strings.
+##! decisions about matching and handling of intelligence.
-
-# Example string subtypes:
-#   url
-#   email
-#   domain
-#   software
-#   user_name
-#   file_name
-#   file_md5
-#   x509_md5
-
-# Example tags: 
-#   infrastructure
-#   malicious
-#   sensitive
-#   canary
-#   friend
 
 @load base/frameworks/notice
 
 module Intel;
 
 export {
-	## The intel logging stream identifier.
 	redef enum Log::ID += { LOG };
 	
-	redef enum Notice::Type += {
+	## String data needs to be further categoried since it could represent
-		## This notice should be used in all detector scripts to indicate 
+	## and number of types of data.
-		## an intelligence based detection.
+	type StrType: enum {
-		Detection,
+		## A complete URL without the prefix "http://".
+		URL,
+		## User-Agent string, typically HTTP or mail message body.
+		USER_AGENT,
+		## Email address.
+		EMAIL,
+		## DNS domain name.
+		DOMAIN,
+		## A user name.
+		USER_NAME,
+		## File hash which is non-hash type specific.  It's up to the user to query
+		## for any relevant hash types.
+		FILE_HASH,
+		## Certificate SHA-1 hash.
+		CERT_HASH,
 	};
 	
-	## Record type used for logging information from the intelligence framework.
+	## Data about an :bro:type:`Intel::Item`
-	## Primarily for problems or oddities with inserting and querying data.  
-	## This is important since the content of the intelligence framework can 
-	## change quite dramatically during runtime and problems may be introduced 
-	## into the data.
-	type Info: record {
-		## The current network time.
-		ts:      time   &log;
-		## Represents the severity of the message. 
-		## This value should be one of: "info", "warn", "error"
-		level:   string &log;
-		## The message.
-		message: string &log;
-	};
-	
-	## Record to represent metadata associated with a single piece of
-	## intelligence.
 	type MetaData: record {
-		## A description for the data.
+		## An arbitrary string value representing the data source.  Typically,
+		## the convention for this field will be the source name and feed name
+		## separated by a hyphen.  For example: "source1-c&c".
+		source:      string;
+		## A freeform description for the data.
 		desc:        string      &optional;
-		## A URL where more information may be found about the intelligence.
+		## A URL for more information about the data.
 		url:         string      &optional;
-		## The time at which the data was first declared to be intelligence.
-		first_seen:  time        &optional;
-		## When this data was most recent inserted into the framework.
-		latest_seen: time        &optional;
-		## Arbitrary text tags for the data.
-		tags:        set[string];
 	};
 	
-	## Record to represent a singular piece of intelligence.
+	## Represents a piece of intelligence.
 	type Item: record {
-		## If the data is an IP address, this hold the address.
+		## The IP address if the intelligence is about an IP address.
-		ip:          addr        &optional;
+		host:        addr           &optional;
-		## If the data is textual, this holds the text.
+		## The network if the intelligence is about a CIDR block.
-		str:         string      &optional;
+		net:         subnet         &optional;
-		## If the data is numeric, this holds the number.
+		## The string if the intelligence is about a string.
-		num:         int         &optional;
+		str:         string         &optional;
-		## The subtype of the data for when either the $str or $num fields are
+		## The type of data that is in the string if the $str field is set.
-		## given.  If one of those fields are given, this field must be present.
+		str_type:    StrType        &optional;
-		subtype:     string      &optional;
 		
-		## The next five fields are temporary until a better model for 
+		## Metadata for the item.  Typically represents more deeply \
-		## attaching metadata to an intelligence item is created.
+		## descriptive data for a piece of intelligence.
-		desc:        string      &optional;
+		meta:        MetaData;
-		url:         string      &optional;
-		first_seen:  time        &optional;
-		latest_seen: time        &optional;
-		tags:        set[string];
-		
-		## These single string tags are throw away until pybroccoli supports sets.
-		tag1: string &optional;
-		tag2: string &optional;
-		tag3: string &optional;
 	};
 	
-	## Record model used for constructing queries against the intelligence 
+	## Enum to represent where data came from when it was discovered.
-	## framework.
+	## The convention is to prefix the name with ``IN_``.
-	type QueryItem: record {
+	type Where: enum {
-		## If an IP address is being queried for, this field should be given.
+		## A catchall value to represent data of unknown provenance.
-		ip:        addr        &optional;
+		IN_ANYWHERE,
-		## If a string is being queried for, this field should be given.
-		str:       string      &optional;
-		## If numeric data is being queried for, this field should be given.
-		num:       int         &optional;
-		## If either a string or number is being queried for, this field should
-		## indicate the subtype of the data.
-		subtype:   string      &optional;
-		
-		## A set of tags where if a single metadata record attached to an item
-		## has any one of the tags defined in this field, it will match.
-		or_tags:   set[string] &optional;
-		## A set of tags where a single metadata record attached to an item 
-		## must have all of the tags defined in this field.
-		and_tags:  set[string] &optional; 
-		
-		## The predicate can be given when searching for a match.  It will
-		## be tested against every :bro:type:`Intel::MetaData` item associated
-		## with the data being matched on.  If it returns T a single time, the 
-		## matcher will consider that the item has matched.  This field can
-		## be used for constructing arbitrarily complex queries that may not
-		## be possible with the $or_tags or $and_tags fields.
-		pred:      function(meta: Intel::MetaData): bool &optional;
 	};
 
-	## Function to insert data into the intelligence framework.
+	## The $host field and combination of $str and $str_type fields are mutually 
-	## 
+	## exclusive.  These records *must* represent either an IP address being
-	## item: The data item.
+	## seen or a string being seen.
-	##
+	type Seen: record {
-	## Returns: T if the data was successfully inserted into the framework,
+		## The IP address if the data seen is an IP address.
-	##          otherwise it returns F.
+		host:      addr          &log &optional;
-	global insert: function(item: Item): bool;
+		## The string if the data is about a string.
+		str:       string        &log &optional;
+		## The type of data that is in the string if the $str field is set.
+		str_type:  StrType       &log &optional;
 
-	## A wrapper for the :bro:id:`Intel::insert` function.  This is primarily
+		## Where the data was discovered.
-	## used as the external API for inserting data into the intelligence 
+		where:     Where         &log;
-	## using Broccoli.
-	global insert_event: event(item: Item);
 		
-	## Function for matching data within the intelligence framework.
+		## If the data was discovered within a connection, the 
-	global matcher: function(item: QueryItem): bool;
+		## connection record should go into get to give context to the data.
+		conn:      connection    &optional;
+	};
+
+	## Record used for the logging framework representing a positive
+	## hit within the intelligence framework.
+	type Info: record {
+		## Timestamp when the data was discovered.
+		ts:       time           &log;
+
+		## If a connection was associated with this intelligence hit,
+		## this is the uid for the connection
+		uid:      string         &log &optional;
+		## If a connection was associated with this intelligence hit,
+		## this is the conn_id for the connection.
+		id:       conn_id        &log &optional;
+
+		## Where the data was seen.
+		seen:     Seen           &log;
+		## Sources which supplied data that resulted in this match.
+		sources:  set[string]    &log;
+	};
+
+	## Intelligence data manipulation functions.
+	global insert: function(item: Item);
+
+	## Function to declare discovery of a piece of data in order to check
+	## it against known intelligence for matches.
+	global seen: function(s: Seen);
+
+	## Event to represent a match in the intelligence data from data that was seen.  
+	## On clusters there is no assurance as to where this event will be generated 
+	## so do not assume that arbitrary global state beyond the given data
+	## will be available.
+	##
+	## This is the primary mechanism where a user will take actions based on data
+	## within the intelligence framework.
+	global match: event(s: Seen, items: set[Item]);
+
+	global log_intel: event(rec: Info);
 }
 
-type MetaDataStore: table[count] of MetaData;
+# Internal handler for matches with no metadata available.
+global match_no_items: event(s: Seen);
+
+# Internal events for cluster data distribution
+global new_item: event(item: Item);
+global updated_item: event(item: Item);
+
+# Optionally store metadata.  This is used internally depending on
+# if this is a cluster deployment or not.
+const have_full_data = T &redef;
+
+# The in memory data structure for holding intelligence.
 type DataStore: record {
-	ip_data:     table[addr] of MetaDataStore;
+	net_data:    table[subnet] of set[MetaData];
-	# The first string is the actual value and the second string is the subtype.
+	string_data: table[string, StrType] of set[MetaData];
-	string_data: table[string, string] of MetaDataStore;
-	int_data:    table[int, string] of MetaDataStore;
 };
-global data_store: DataStore;
+global data_store: DataStore &redef;
 
-event bro_init()
+# The in memory data structure for holding the barest matchable intelligence.
+# This is primarily for workers to do the initial quick matches and store
+# a minimal amount of data for the full match to happen on the manager.
+type MinDataStore: record {
+	net_data:    set[subnet];
+	string_data: set[string, StrType];
+};
+global min_data_store: MinDataStore &redef;
+
+
+event bro_init() &priority=5
 	{
-	Log::create_stream(Intel::LOG, [$columns=Info]);
+	Log::create_stream(LOG, [$columns=Info, $ev=log_intel]);
 	}
 
-
+function find(s: Seen): bool
-function insert(item: Item): bool
 	{
-	local err_msg = "";
+	if ( s?$host && 
-	if ( (item?$str || item?$num) && ! item?$subtype )
+	     ((have_full_data && s$host in data_store$net_data) || 
-		err_msg = "You must provide a subtype to insert_sync or this item doesn't make sense.";
+	      (s$host in min_data_store$net_data)))
-	
-	if ( err_msg == "" )
 		{
-		# Create and fill out the meta data item.
+		return T;
-		local meta: MetaData;
+		}
-		if ( item?$first_seen )
+	else if ( s?$str && s?$str_type &&
-			meta$first_seen = item$first_seen;
+	          ((have_full_data && [s$str, s$str_type] in data_store$string_data) ||
-		if ( item?$latest_seen )
+	           ([s$str, s$str_type] in min_data_store$string_data)))
-			meta$latest_seen = item$latest_seen;
+		{
-		if ( item?$tags )
+		return T;
-			meta$tags = item$tags;
+		}
-		if ( item?$desc )
+	else
-			meta$desc = item$desc;
+		{
-		if ( item?$url )
+		return F;
-			meta$url = item$url;
+		}
+	}
 
+function get_items(s: Seen): set[Item]
+	{
+	local item: Item;
+	local return_data: set[Item] = set();
 
-		# This is hopefully only temporary until pybroccoli supports sets.
+	if ( ! have_full_data )
-		if ( item?$tag1 )
+		{
-			add item$tags[item$tag1];
+		# A reporter warning should be generated here because this function
-		if ( item?$tag2 )
+		# should never be called from a host that doesn't have the full data.
-			add item$tags[item$tag2];
+		# TODO: do a reporter warning.
-		if ( item?$tag3 )
+		return return_data;
-			add item$tags[item$tag3];
+		}
 
-		if ( item?$ip )
+	if ( s?$host )
+		{
+		# See if the host is known about and it has meta values
+		if ( s$host in data_store$net_data )
 			{
-			if ( item$ip !in data_store$ip_data )
+			for ( m in data_store$net_data[s$host] )
-				data_store$ip_data[item$ip] = table();
+				{
-			data_store$ip_data[item$ip][|data_store$ip_data[item$ip]|] = meta;
+				# TODO: the lookup should be finding all and not just most specific
-			return T;
+				#       and $host/$net should have the correct value.
+				item = [$host=s$host, $meta=m];
+				add return_data[item];
+				}
 			}
-		else if ( item?$str )
+		}
+	else if ( s?$str && s?$str_type )
+		{
+		# See if the string is known about and it has meta values
+		if ( [s$str, s$str_type] in data_store$string_data )
 			{
-			if ( [item$str, item$subtype] !in data_store$string_data )
+			for ( m in data_store$string_data[s$str, s$str_type] )
-				data_store$string_data[item$str, item$subtype] = table();
+				{
-			
+				item = [$str=s$str, $str_type=s$str_type, $meta=m];
-			data_store$string_data[item$str, item$subtype][|data_store$string_data[item$str, item$subtype]|] = meta;
+				add return_data[item];
-			return T;
+				}
 			}
-		else if ( item?$num )
+		}
-			{
-			if ( [item$num, item$subtype] !in data_store$int_data )
-				data_store$int_data[item$num, item$subtype] = table();
 
-			data_store$int_data[item$num, item$subtype][|data_store$int_data[item$num, item$subtype]|] = meta;
+	return return_data;
-			return T;
+	}
+
+function Intel::seen(s: Seen)
+	{
+	if ( find(s) )
+		{
+		if ( have_full_data )
+			{
+			local items = get_items(s);
+			event Intel::match(s, items);
 			}
 		else
-			err_msg = "Failed to insert intelligence item for some unknown reason.";
-		}
-	
-	if ( err_msg != "" )
-		Log::write(Intel::LOG, [$ts=network_time(), $level="warn", $message=fmt(err_msg)]);
-	return F;
-	}
-	
-event insert_event(item: Item)
-	{
-	insert(item);
-	}
-	
-function match_item_with_metadata(item: QueryItem, meta: MetaData): bool
-	{
-	if ( item?$and_tags )
-		{
-		local matched = T;
-		# Every tag given has to match in a single MetaData entry.
-		for ( tag in item$and_tags )
 			{
-			if ( tag !in meta$tags )
+			event Intel::match_no_items(s);
-				matched = F;
 			}
-		if ( matched )
+		}
+	}
+
+
+function has_meta(check: MetaData, metas: set[MetaData]): bool
+	{
+	local check_hash = md5_hash(check);
+	for ( m in metas )
+		{
+		if ( check_hash == md5_hash(m) )
 			return T;
 		}
-	else if ( item?$or_tags )
-		{
-		# For OR tags, only a single tag has to match.
-		for ( tag in item$or_tags )
-			{
-			if ( tag in meta$tags )
-				return T;
-			}
-		}
-	else if ( item?$pred )
-		return item$pred(meta);
 
-	# This indicates some sort of failure in the query
+	# The records must not be equivalent if we made it this far.
 	return F;
 	}
 
-function matcher(item: QueryItem): bool
+event Intel::match(s: Seen, items: set[Item]) &priority=5
 	{
-	local err_msg = "";
+	local empty_set: set[string] = set();
-	if ( ! (item?$ip || item?$str || item?$num) )
+	local info: Info = [$ts=network_time(), $seen=s, $sources=empty_set];
-		err_msg = "You must supply one of the $ip, $str, or $num fields to search on";
-	else if ( (item?$or_tags || item?$and_tags) && item?$pred )
-		err_msg = "You can't match with both tags and a predicate.";
-	else if ( item?$or_tags && item?$and_tags )
-		err_msg = "You can't match with both OR'd together tags and AND'd together tags";
-	else if ( (item?$str || item?$num) && ! item?$subtype )
-		err_msg = "You must provide a subtype to matcher or this item doesn't make sense.";
-	else if ( item?$str && item?$num )
-		err_msg = "You must only provide $str or $num, not both.";
 
-	local meta: MetaData;
+	if ( s?$conn )
-
-	if ( err_msg == "" )
 		{
-		if ( item?$ip )
+		info$uid = s$conn$uid;
-			{
+		info$id  = s$conn$id;
-			if ( item$ip in data_store$ip_data )
-				{
-				if ( ! item?$and_tags && ! item?$or_tags && ! item?$pred )
-					return T;
-			
-				for ( i in data_store$ip_data[item$ip] )
-					{
-					meta = data_store$ip_data[item$ip][i];
-					if ( match_item_with_metadata(item, meta) )
-						return T;
-					}
-				}
-			}
-		
-		else if ( item?$str )
-			{
-			if ( [item$str, item$subtype] in data_store$string_data )
-				{
-				if ( ! item?$and_tags && ! item?$or_tags && ! item?$pred )
-					return T;
-
-				for ( i in data_store$string_data[item$str, item$subtype] )
-					{
-					meta = data_store$string_data[item$str, item$subtype][i];
-					if ( match_item_with_metadata(item, meta) )
-						return T;
-					}
-				}
-			}
-		
-		else if ( item?$num )
-			{
-			if ( [item$num, item$subtype] in data_store$int_data )
-				{
-				if ( ! item?$and_tags && ! item?$or_tags && ! item?$pred )
-					return T;
-
-				for ( i in data_store$int_data[item$num, item$subtype] )
-					{
-					meta = data_store$int_data[item$num, item$subtype][i];
-					if ( match_item_with_metadata(item, meta) )
-						return T;
-					}
-				}
-			}
-		else
-			err_msg = "Failed to query intelligence data for some unknown reason.";
 		}
 
-	if ( err_msg != "" )
+	for ( item in items )
-		Log::write(Intel::LOG, [$ts=network_time(), $level="error", $message=fmt(err_msg)]);
+		add info$sources[item$meta$source];
-	return F;
+
+	Log::write(Intel::LOG, info);
 	}
+
+function insert(item: Item)
+	{
+	if ( item?$str && !item?$str_type )
+		{
+		event reporter_warning(network_time(), fmt("You must provide a str_type for strings or this item doesn't make sense.  Item: %s", item), "");
+		return;
+		}
+
+	# Create and fill out the meta data item.
+	local meta = item$meta;
+	local metas: set[MetaData];
+
+	if ( item?$host )
+		{
+		local host = mask_addr(item$host, is_v4_addr(item$host) ? 32 : 128);
+		if ( have_full_data )
+			{
+			if ( host !in data_store$net_data )
+				data_store$net_data[host] = set();
+
+			metas = data_store$net_data[host];
+			}
+
+		add min_data_store$net_data[host];
+		}
+	else if ( item?$net )
+		{
+		if ( have_full_data )
+			{
+			if ( item$net !in data_store$net_data )
+				data_store$net_data[item$net] = set();
+
+			metas = data_store$net_data[item$net];
+			}
+
+		add min_data_store$net_data[item$net];
+		}
+	else if ( item?$str )
+		{
+		if ( have_full_data )
+			{
+			if ( [item$str, item$str_type] !in data_store$string_data )
+				data_store$string_data[item$str, item$str_type] = set();
+
+			metas = data_store$string_data[item$str, item$str_type];
+			}
+
+		add min_data_store$string_data[item$str, item$str_type];
+		}
+
+	local updated = F;
+	if ( have_full_data )
+		{
+		for ( m in metas )
+			{
+			if ( meta$source == m$source )
+				{
+				if ( has_meta(meta, metas) )
+					{
+					# It's the same item being inserted again.
+					return;
+					}
+				else
+					{
+					# Same source, different metadata means updated item.
+					updated = T;
+					}
+				}
+			}
+		add metas[item$meta];
+		}
+	
+	if ( updated )
+		event Intel::updated_item(item);
+	else
+		event Intel::new_item(item);
+	}
+	

scripts/base/frameworks/logging/main.bro

@@ -60,6 +60,9 @@ export {
 	## Default rotation interval. Zero disables rotation.
 	const default_rotation_interval = 0secs &redef;
 
+	## Default alarm summary mail interval. Zero disables alarm summary mails.
+	const default_mail_alarms_interval = 0secs &redef;
+
 	## Default naming format for timestamps embedded into filenames.
 	## Uses a ``strftime()`` style.
 	const default_rotation_date_format = "%Y-%m-%d-%H-%M-%S" &redef;

scripts/base/frameworks/logging/writers/ascii.bro

@@ -1,5 +1,13 @@
 ##! Interface for the ASCII log writer.  Redefinable options are available
 ##! to tweak the output format of ASCII logs.
+##!
+##! The ASCII writer supports currently one writer-specific filter option via
+##! ``config``: setting ``only_single_header_row`` to ``T`` turns the output into
+##! into CSV mode where only a single header row with the column names is printed
+##! out as meta information. Example filter using this::
+##! 
+##!    local my_filter: Log::Filter = [$name = "my-filter", $writer = Log::WRITER_ASCII, $config = table(["only_single_header_row"] = "T")]; 
+##!
 
 module LogAscii;
 

scripts/base/frameworks/notice/actions/pp-alarms.bro

@@ -101,7 +101,7 @@ event bro_init()
 	# This replaces the standard non-pretty-printing filter.
 	Log::add_filter(Notice::ALARM_LOG,
 			[$name="alarm-mail", $writer=Log::WRITER_NONE,
-			 $interv=Log::default_rotation_interval,
+			 $interv=Log::default_mail_alarms_interval,
 			 $postprocessor=pp_postprocessor]);
 	}
 

scripts/base/init-bare.bro

@@ -826,7 +826,7 @@ const tcp_storm_interarrival_thresh = 1 sec &redef;
 ## peer's ACKs.  Set to zero to turn off this determination.
 ##
 ## .. bro:see:: tcp_max_above_hole_without_any_acks tcp_excessive_data_without_further_acks
-const tcp_max_initial_window = 4096;
+const tcp_max_initial_window = 4096 &redef;
 
 ## If we're not seeing our peer's ACKs, the maximum volume of data above a sequence
 ## hole that we'll tolerate before assuming that there's been a packet drop and we
@@ -834,7 +834,7 @@ const tcp_max_initial_window = 4096;
 ## up.
 ##
 ## .. bro:see:: tcp_max_initial_window tcp_excessive_data_without_further_acks
-const tcp_max_above_hole_without_any_acks = 4096;
+const tcp_max_above_hole_without_any_acks = 4096 &redef;
 
 ## If we've seen this much data without any of it being acked, we give up
 ## on that connection to avoid memory exhaustion due to buffering all that
@@ -843,7 +843,7 @@ const tcp_max_above_hole_without_any_acks = 4096;
 ## has in fact gone too far, but for now we just make this quite beefy.
 ##
 ## .. bro:see:: tcp_max_initial_window tcp_max_above_hole_without_any_acks
-const tcp_excessive_data_without_further_acks = 10 * 1024 * 1024;
+const tcp_excessive_data_without_further_acks = 10 * 1024 * 1024 &redef;
 
 ## For services without an a handler, these sets define originator-side ports that
 ## still trigger reassembly.
@@ -2495,6 +2495,16 @@ type bittorrent_benc_dir: table[string] of bittorrent_benc_value;
 ##    bt_tracker_response_not_ok
 type bt_tracker_headers: table[string] of string;
 
+type ModbusCoils: vector of bool;
+type ModbusRegisters: vector of count;
+
+type ModbusHeaders: record {
+	tid:           count;
+	pid:           count;
+	uid:           count;
+	function_code: count;
+};
+
 module SOCKS;
 export {
 	## This record is for a SOCKS client or server to provide either a 

scripts/base/init-default.bro

@@ -14,6 +14,7 @@
 @load base/utils/patterns
 @load base/utils/strings
 @load base/utils/thresholds
+@load base/utils/urls
 
 # This has some deep interplay between types and BiFs so it's 
 # loaded in base/init-bare.bro
@@ -36,8 +37,11 @@
 @load base/protocols/ftp
 @load base/protocols/http
 @load base/protocols/irc
+@load base/protocols/modbus
 @load base/protocols/smtp
 @load base/protocols/socks
 @load base/protocols/ssh
 @load base/protocols/ssl
 @load base/protocols/syslog
+
+@load base/misc/find-checksum-offloading

scripts/base/misc/find-checksum-offloading.bro

@@ -0,0 +1,57 @@
+##! Discover cases where the local interface is sniffed and outbound packets
+##! have checksum offloading.  Load this script to receive a notice if it's
+##! likely that checksum offload effects are being seen on a live interface or
+##! in a packet trace file.
+
+@load base/frameworks/notice
+
+module ChecksumOffloading;
+
+export {
+	## The interval which is used for checking packet statistics
+	## to see if checksum offloading is affecting analysis.
+	const check_interval = 10secs &redef;
+}
+
+# Keep track of how many bad checksums have been seen.
+global bad_checksums = 0;
+
+# Track to see if this script is done so that messages aren't created multiple times.
+global done = F;
+
+event ChecksumOffloading::check()
+	{
+	if ( done )
+		return;
+
+	local pkts_recvd = net_stats()$pkts_recvd;
+	if ( (bad_checksums*1.0 / net_stats()$pkts_recvd*1.0) > 0.05 )
+		{
+		local packet_src = reading_traces() ? "trace file likely has" : "interface is likely receiving";
+		local message = fmt("Your %s invalid IP checksums, most likely from NIC checksum offloading.", packet_src);
+		Reporter::warning(message);
+		done = T;
+		}
+	else if ( pkts_recvd < 20 )
+		{
+		# Keep scheduling this event until we've seen some lower threshold of
+		# total packets.
+		schedule check_interval { ChecksumOffloading::check() };
+		}
+	}
+
+event bro_init()
+	{
+	schedule check_interval { ChecksumOffloading::check() };
+	}
+
+event net_weird(name: string)
+	{
+	if ( name == "bad_IP_checksum" )
+		++bad_checksums;
+	}
+
+event bro_done()
+	{
+	event ChecksumOffloading::check();
+	}

scripts/base/protocols/ftp/gridftp.bro

@@ -68,6 +68,16 @@ export {
 	const data_channel_initial_criteria: function(c: connection): bool &redef;
 }
 
+redef record FTP::Info += {
+	last_auth_requested: string &optional;
+};
+
+event ftp_request(c: connection, command: string, arg: string) &priority=4
+	{
+	if ( command == "AUTH" && c?$ftp )
+		c$ftp$last_auth_requested = arg;
+	}
+
 function size_callback(c: connection, cnt: count): interval
 	{
 	if ( c$orig$size > size_threshold || c$resp$size > size_threshold )
@@ -89,8 +99,10 @@ function size_callback(c: connection, cnt: count): interval
 
 event ssl_established(c: connection) &priority=5
 	{
-	# Add service label to control channels.
+	# If an FTP client requests AUTH GSSAPI and later an SSL handshake
-	if ( "FTP" in c$service )
+	# finishes, it's likely a GridFTP control channel, so add service label.
+	if ( c?$ftp && c$ftp?$last_auth_requested &&
+	     /GSSAPI/ in c$ftp$last_auth_requested )
 		add c$service["gridftp"];
 	}
 

scripts/base/protocols/modbus/__load__.bro

@@ -0,0 +1,2 @@
+@load ./consts
+@load ./main

scripts/base/protocols/modbus/consts.bro

@@ -0,0 +1,67 @@
+
+module Modbus;
+
+export {
+	## Standard defined Modbus function codes.
+	const function_codes = {
+		[0x01] = "READ_COILS",
+		[0x02] = "READ_DISCRETE_INPUTS",
+		[0x03] = "READ_HOLDING_REGISTERS",
+		[0x04] = "READ_INPUT_REGISTERS",
+		[0x05] = "WRITE_SINGLE_COIL",
+		[0x06] = "WRITE_SINGLE_REGISTER",
+		[0x07] = "READ_EXCEPTION_STATUS",
+		[0x08] = "DIAGNOSTICS",
+		[0x0B] = "GET_COMM_EVENT_COUNTER",
+		[0x0C] = "GET_COMM_EVENT_LOG",
+		[0x0F] = "WRITE_MULTIPLE_COILS",
+		[0x10] = "WRITE_MULTIPLE_REGISTERS",
+		[0x11] = "REPORT_SLAVE_ID",
+		[0x14] = "READ_FILE_RECORD",
+		[0x15] = "WRITE_FILE_RECORD",
+		[0x16] = "MASK_WRITE_REGISTER",
+		[0x17] = "READ_WRITE_MULTIPLE_REGISTERS",
+		[0x18] = "READ_FIFO_QUEUE",
+		[0x2B] = "ENCAP_INTERFACE_TRANSPORT",
+
+		# Machine/vendor/network specific functions
+		[0x09] = "PROGRAM_484",
+		[0x0A] = "POLL_484",
+		[0x0D] = "PROGRAM_584_984",
+		[0x0E] = "POLL_584_984",
+		[0x12] = "PROGRAM_884_U84",
+		[0x13] = "RESET_COMM_LINK_884_U84",
+		[0x28] = "PROGRAM_CONCEPT",
+		[0x7D] = "FIRMWARE_REPLACEMENT",
+		[0x7E] = "PROGRAM_584_984_2",
+		[0x7F] = "REPORT_LOCAL_ADDRESS",
+
+		# Exceptions
+		[0x81] = "READ_COILS_EXCEPTION",
+		[0x82] = "READ_DISCRETE_INPUTS_EXCEPTION",
+		[0x83] = "READ_HOLDING_REGISTERS_EXCEPTION",
+		[0x84] = "READ_INPUT_REGISTERS_EXCEPTION",
+		[0x85] = "WRITE_SINGLE_COIL_EXCEPTION",
+		[0x86] = "WRITE_SINGLE_REGISTER_EXCEPTION",
+		[0x87] = "READ_EXCEPTION_STATUS_EXCEPTION",
+		[0x8F] = "WRITE_MULTIPLE_COILS_EXCEPTION",
+		[0x90] = "WRITE_MULTIPLE_REGISTERS_EXCEPTION",
+		[0x94] = "READ_FILE_RECORD_EXCEPTION",
+		[0x95] = "WRITE_FILE_RECORD_EXCEPTION",
+		[0x96] = "MASK_WRITE_REGISTER_EXCEPTION",
+		[0x97] = "READ_WRITE_MULTIPLE_REGISTERS_EXCEPTION",
+		[0x98] = "READ_FIFO_QUEUE_EXCEPTION",
+	} &default=function(i: count):string { return fmt("unknown-%d", i); } &redef;
+
+	const exception_codes = {
+		[0x01] = "ILLEGAL_FUNCTION",
+		[0x02] = "ILLEGAL_DATA_ADDRESS",
+		[0x03] = "ILLEGAL_DATA_VALUE",
+		[0x04] = "SLAVE_DEVICE_FAILURE",
+		[0x05] = "ACKNOWLEDGE",
+		[0x06] = "SLAVE_DEVICE_BUSY",
+		[0x08] = "MEMORY_PARITY_ERROR",
+		[0x0A] = "GATEWAY_PATH_UNAVAILABLE",
+		[0x0B] = "GATEWAY_TARGET_DEVICE_FAILED_TO_RESPOND",
+	} &default=function(i: count):string { return fmt("unknown-%d", i); } &redef;
+}

scripts/base/protocols/modbus/main.bro

@@ -0,0 +1,71 @@
+##! Base Modbus analysis script.
+
+module Modbus;
+
+@load ./consts
+
+export {
+	redef enum Log::ID += { LOG };
+
+	type Info: record {
+		## Time of the request.
+		ts:        time           &log;
+		## Unique identifier for the connnection.
+		uid:       string         &log;
+		## Identifier for the connection.
+		id:        conn_id        &log;
+		## The name of the function message that was sent.
+		func:      string         &log &optional;
+		## The exception if the response was a failure.
+		exception: string         &log &optional;
+	};
+
+	## Event that can be handled to access the Modbus record as it is sent on
+	## to the logging framework.
+	global log_modbus: event(rec: Info);
+}
+
+redef record connection += {
+	modbus: Info &optional;
+};
+
+# Configure DPD and the packet filter.
+redef capture_filters += { ["modbus"] = "tcp port 502" };
+redef dpd_config += { [ANALYZER_MODBUS] = [$ports = set(502/tcp)] };
+redef likely_server_ports += { 502/tcp };
+
+event bro_init() &priority=5
+	{
+	Log::create_stream(Modbus::LOG, [$columns=Info, $ev=log_modbus]);
+	}
+
+event modbus_message(c: connection, headers: ModbusHeaders, is_orig: bool) &priority=5
+	{
+	if ( ! c?$modbus )
+		{
+		c$modbus = [$ts=network_time(), $uid=c$uid, $id=c$id];
+		}
+
+	c$modbus$ts   = network_time();
+	c$modbus$func = function_codes[headers$function_code];
+	}
+
+event modbus_message(c: connection, headers: ModbusHeaders, is_orig: bool) &priority=-5
+	{
+	# Only log upon replies.
+	# Also, don't log now if this is an exception (log in the exception event handler)
+	if ( ! is_orig && ( headers$function_code <= 0x81 || headers$function_code >= 0x98 ) )
+		Log::write(LOG, c$modbus);
+	}
+
+event modbus_exception(c: connection, headers: ModbusHeaders, code: count) &priority=5
+	{
+	c$modbus$exception = exception_codes[code];
+	}
+
+event modbus_exception(c: connection, headers: ModbusHeaders, code: count) &priority=-5
+	{
+	Log::write(LOG, c$modbus);
+	delete c$modbus$exception;
+	}
+

scripts/base/utils/patterns.bro

@@ -1,5 +1,7 @@
 ##! Functions for creating and working with patterns.
 
+module GLOBAL;
+
 ## Given a pattern as a string with two tildes (~~) contained in it, it will
 ## return a pattern with string set's elements OR'd together where the
 ## double-tilde was given (this function only works at or before init time).

scripts/base/utils/urls.bro

@@ -0,0 +1,25 @@
+## Functions for URL handling.
+
+## A regular expression for matching and extracting URLs.
+const url_regex = /^([a-zA-Z\-]{3,5})(:\/\/[^\/?#"'\r\n><]*)([^?#"'\r\n><]*)([^[:blank:]\r\n"'><]*|\??[^"'\r\n><]*)/ &redef;
+
+## Extracts URLs discovered in arbitrary text.
+function find_all_urls(s: string): string_set
+	{
+	return find_all(s, url_regex);
+	}
+
+## Extracts URLs discovered in arbitrary text without
+## the URL scheme included.
+function find_all_urls_without_scheme(s: string): string_set
+	{
+	local urls = find_all_urls(s);
+	local return_urls: set[string] = set();
+	for ( url in urls )
+		{
+		local no_scheme = sub(url, /^([a-zA-Z\-]{3,5})(:\/\/)/, "");
+		add return_urls[no_scheme];
+		}
+
+	return return_urls;
+	}

scripts/policy/frameworks/intel/__load__.bro

@@ -0,0 +1,8 @@
+@load ./conn-established
+@load ./dns
+@load ./http-host-header
+@load ./http-url
+@load ./http-user-agents
+@load ./ssl
+@load ./smtp
+@load ./smtp-url-extraction

scripts/policy/frameworks/intel/conn-established.bro

@@ -0,0 +1,8 @@
+@load base/frameworks/intel
+@load ./where-locations
+
+event connection_established(c: connection)
+	{
+	Intel::seen([$host=c$id$orig_h, $conn=c, $where=Conn::IN_ORIG]);
+	Intel::seen([$host=c$id$resp_h, $conn=c, $where=Conn::IN_RESP]);
+	}

scripts/policy/frameworks/intel/dns.bro

@@ -0,0 +1,10 @@
+@load base/frameworks/intel
+@load ./where-locations
+
+event dns_request(c: connection, msg: dns_msg, query: string, qtype: count, qclass: count)
+	{
+	Intel::seen([$str=query,
+	             $str_type=Intel::DOMAIN,
+	             $conn=c,
+	             $where=DNS::IN_REQUEST]);
+	}

scripts/policy/frameworks/intel/http-host-header.bro

@@ -0,0 +1,11 @@
+@load base/frameworks/intel
+@load ./where-locations
+
+event http_header(c: connection, is_orig: bool, name: string, value: string)
+	{
+	if ( is_orig && name == "HOST" )
+		Intel::seen([$str=value,
+		             $str_type=Intel::DOMAIN,
+		             $conn=c,
+		             $where=HTTP::IN_HOST_HEADER]);
+	}

scripts/policy/frameworks/intel/http-url.bro

@@ -0,0 +1,12 @@
+@load base/frameworks/intel
+@load base/protocols/http/utils
+@load ./where-locations
+
+event http_message_done(c: connection, is_orig: bool, stat: http_message_stat)
+	{
+	if ( is_orig && c?$http )
+		Intel::seen([$str=HTTP::build_url(c$http),
+		             $str_type=Intel::URL,
+		             $conn=c,
+		             $where=HTTP::IN_URL]);
+	}

scripts/policy/frameworks/intel/http-user-agents.bro

@@ -0,0 +1,12 @@
+@load base/frameworks/intel
+@load ./where-locations
+
+event http_header(c: connection, is_orig: bool, name: string, value: string)
+	{
+	if ( is_orig && name == "USER-AGENT" )
+		Intel::seen([$str=value,
+		             $str_type=Intel::USER_AGENT,
+		             $conn=c,
+		             $where=HTTP::IN_USER_AGENT_HEADER]);
+	}
+

scripts/policy/frameworks/intel/smtp-url-extraction.bro

@@ -0,0 +1,15 @@
+@load base/frameworks/intel
+@load base/utils/urls
+@load ./where-locations
+
+event mime_segment_data(c: connection, length: count, data: string) &priority=3
+	{
+	local urls = find_all_urls_without_scheme(data);
+	for ( url in urls )
+		{
+		Intel::seen([$str=url,
+		             $str_type=Intel::URL,
+		             $conn=c,
+		             $where=SMTP::IN_MESSAGE]);
+		}
+	}

scripts/policy/frameworks/intel/smtp.bro

@@ -0,0 +1,71 @@
+@load base/frameworks/intel
+@load base/protocols/smtp
+@load ./where-locations
+
+event mime_end_entity(c: connection)
+	{
+	if ( c?$smtp )
+		{
+		if ( c$smtp?$path )
+			{
+			local path = c$smtp$path;
+			for ( i in path )
+				{
+				Intel::seen([$host=path[i],
+				             $conn=c,
+				             $where=SMTP::IN_RECEIVED_HEADER]);
+				}
+			}
+
+		if ( c$smtp?$user_agent )
+			Intel::seen([$str=c$smtp$user_agent,
+			             $str_type=Intel::USER_AGENT,
+			             $conn=c,
+			             $where=SMTP::IN_HEADER]);
+
+		if ( c$smtp?$x_originating_ip )
+			Intel::seen([$host=c$smtp$x_originating_ip,
+			             $conn=c,
+			             $where=SMTP::IN_X_ORIGINATING_IP_HEADER]);
+
+		if ( c$smtp?$mailfrom )
+			Intel::seen([$str=c$smtp$mailfrom,
+			             $str_type=Intel::EMAIL,
+			             $conn=c,
+			             $where=SMTP::IN_MAIL_FROM]);
+
+		if ( c$smtp?$rcptto )
+			{
+			for ( rcptto in c$smtp$rcptto )
+				{
+				Intel::seen([$str=rcptto,
+				             $str_type=Intel::EMAIL,
+				             $conn=c,
+				             $where=SMTP::IN_RCPT_TO]);
+				}
+			}
+
+		if ( c$smtp?$from )
+			Intel::seen([$str=c$smtp$from,
+			             $str_type=Intel::EMAIL,
+			             $conn=c,
+			             $where=SMTP::IN_FROM]);
+
+		if ( c$smtp?$to )
+			{
+			for ( email_to in c$smtp$to )
+				{
+				Intel::seen([$str=email_to,
+				             $str_type=Intel::EMAIL,
+				             $conn=c,
+				             $where=SMTP::IN_TO]);
+				}
+			}
+
+		if ( c$smtp?$reply_to )
+			Intel::seen([$str=c$smtp$reply_to,
+			             $str_type=Intel::EMAIL,
+			             $conn=c,
+			             $where=SMTP::IN_REPLY_TO]);
+		}
+	}

scripts/policy/frameworks/intel/ssl.bro

@@ -0,0 +1,34 @@
+@load base/frameworks/intel
+@load base/protocols/ssl
+@load ./where-locations
+
+event x509_certificate(c: connection, is_orig: bool, cert: X509, chain_idx: count, chain_len: count, der_cert: string)
+	{
+	if ( chain_idx == 0 )
+		{
+		if ( /emailAddress=/ in cert$subject )
+			{
+			local email = sub(cert$subject, /^.*emailAddress=/, "");
+			email = sub(email, /,.*$/, "");
+			Intel::seen([$str=email,
+			             $str_type=Intel::EMAIL,
+			             $conn=c,
+			             $where=(is_orig ? SSL::IN_CLIENT_CERT : SSL::IN_SERVER_CERT)]);
+			}
+
+		Intel::seen([$str=sha1_hash(der_cert),
+		             $str_type=Intel::CERT_HASH,
+		             $conn=c,
+		             $where=(is_orig ? SSL::IN_CLIENT_CERT : SSL::IN_SERVER_CERT)]);
+		}
+	}
+
+event ssl_extension(c: connection, is_orig: bool, code: count, val: string)
+	{
+	if ( is_orig && SSL::extensions[code] == "server_name" && 
+	     c?$ssl && c$ssl?$server_name )
+		Intel::seen([$str=c$ssl$server_name,
+		             $str_type=Intel::DOMAIN,
+		             $conn=c,
+		             $where=SSL::IN_SERVER_NAME]);
+	}

scripts/policy/frameworks/intel/where-locations.bro

@@ -0,0 +1,25 @@
+@load base/frameworks/intel
+
+export {
+	redef enum Intel::Where += {
+		Conn::IN_ORIG,
+		Conn::IN_RESP,
+		DNS::IN_REQUEST,
+		DNS::IN_RESPONSE,
+		HTTP::IN_HOST_HEADER,
+		HTTP::IN_USER_AGENT_HEADER,
+		HTTP::IN_URL,
+		SMTP::IN_MAIL_FROM,
+		SMTP::IN_RCPT_TO,
+		SMTP::IN_FROM,
+		SMTP::IN_TO,
+		SMTP::IN_RECEIVED_HEADER,
+		SMTP::IN_REPLY_TO,
+		SMTP::IN_X_ORIGINATING_IP_HEADER,
+		SMTP::IN_MESSAGE,
+		SSL::IN_SERVER_CERT,
+		SSL::IN_CLIENT_CERT,
+		SSL::IN_SERVER_NAME,
+		SMTP::IN_HEADER,
+	};
+}

scripts/policy/integration/collective-intel/README

@@ -0,0 +1,4 @@
+The scripts in this module are for deeper integration with the
+Collective Intelligence Framework (CIF) since Bro's Intel framework
+doesn't natively behave the same as CIF nor does it store and maintain
+the same data in all cases.

scripts/policy/integration/collective-intel/__load__.bro

@@ -0,0 +1 @@
+@load ./main

scripts/policy/integration/collective-intel/main.bro

@@ -0,0 +1,15 @@
+
+@load base/frameworks/intel
+
+module Intel;
+
+## These are some fields to add extended compatibility between Bro and the Collective
+## Intelligence Framework
+redef record Intel::MetaData += {
+	## Maps to the Impact field in the Collective Intelligence Framework.
+	cif_impact:     string &optional;
+	## Maps to the Severity field in the Collective Intelligence Framework.
+	cif_severity:   string &optional;
+	## Maps to the Confidence field in the Collective Intelligence Framework.
+	cif_confidence: double &optional;
+};

scripts/policy/protocols/http/detect-MHR.bro

@@ -15,6 +15,12 @@ export {
 		## malware hash registry.
 		Malware_Hash_Registry_Match
 	};
+
+	## The malware hash registry runs each malware sample through several A/V engines.
+	## Team Cymru returns a percentage to indicate how many A/V engines flagged the
+	## sample as malicious. This threshold allows you to require a minimum detection
+	## rate (default: 50%).
+	const MHR_threshold = 50 &redef;
 }
 
 event log_http(rec: HTTP::Info)
@@ -22,10 +28,11 @@ event log_http(rec: HTTP::Info)
 	if ( rec?$md5 )
 		{
 		local hash_domain = fmt("%s.malware.hash.cymru.com", rec$md5);
-		when ( local addrs = lookup_hostname(hash_domain) )
+		when ( local MHR_result = lookup_hostname_txt(hash_domain) )
 			{
-			# 127.0.0.2 indicates that the md5 sum was found in the MHR.
+			# Data is returned as "<dateFirstDetected> <detectionRate>"
-			if ( 127.0.0.2 in addrs )
+			local MHR_answer = split1(MHR_result, / /);
+			if ( length(MHR_answer) == 2 && to_count(MHR_answer[2]) >= MHR_threshold )
 				{
 				local url = HTTP::build_url_http(rec);
 				local message = fmt("%s %s %s", rec$id$orig_h, rec$md5, url);

scripts/policy/protocols/http/detect-intel.bro

@@ -1,21 +0,0 @@
-##! Intelligence based HTTP detections.  Not yet working!
-
-@load base/protocols/http/main
-@load base/protocols/http/utils
-@load base/frameworks/intel/main
-
-module HTTP;
-
-event log_http(rec: Info)
-	{
-	local url = HTTP::build_url(rec);
-	local query = [$str=url, $subtype="url", $or_tags=set("malicious", "malware")];
-	if ( Intel::matcher(query) )
-		{
-		local msg = fmt("%s accessed a malicious URL from the intelligence framework", rec$id$orig_h);
-		NOTICE([$note=Intel::Detection, 
-		        $msg=msg, 
-		        $sub=HTTP::build_url_http(rec), 
-		        $id=rec$id]);
-		}
-	}

scripts/policy/protocols/modbus/known-masters-slaves.bro

@@ -0,0 +1,58 @@
+##! Script for tracking known Modbus masters and slaves.
+##!
+##! .. todo: This script needs a lot of work.  What might be more interesting is to track
+##!          master/slave relationships based on commands sent and successful (non-exception)
+##!          responses.
+
+@load base/protocols/modbus
+
+module Known;
+
+export {
+	redef enum Log::ID += { MODBUS_LOG };
+
+	type ModbusDeviceType: enum {
+		MODBUS_MASTER,
+		MODBUS_SLAVE,
+	};
+
+	type ModbusInfo: record {
+		## The time the device was discovered.
+		ts:          time             &log;
+		## The IP address of the host.
+		host:        addr             &log;
+		## The type of device being tracked.
+		device_type: ModbusDeviceType &log;
+	};
+
+	## The Modbus nodes being tracked.
+	global modbus_nodes: set[addr, ModbusDeviceType] &create_expire=1day &redef;
+
+	## Event that can be handled to access the loggable record as it is sent
+	## on to the logging framework.
+	global log_known_modbus: event(rec: ModbusInfo);
+}
+
+event bro_init() &priority=5
+	{
+	Log::create_stream(Known::MODBUS_LOG, [$columns=ModbusInfo, $ev=log_known_modbus]);
+	}
+
+event modbus_message(c: connection, headers: ModbusHeaders, is_orig: bool)
+	{
+	local master = c$id$orig_h;
+	local slave  = c$id$resp_h;
+
+	if ( [master, MODBUS_MASTER] !in modbus_nodes )
+		{
+		add modbus_nodes[master, MODBUS_MASTER];
+		Log::write(MODBUS_LOG, [$ts=network_time(), $host=master, $device_type=MODBUS_MASTER]);
+		}
+
+	if ( [slave, MODBUS_SLAVE] !in modbus_nodes )
+		{
+		add modbus_nodes[slave, MODBUS_SLAVE];
+		Log::write(MODBUS_LOG, [$ts=network_time(), $host=slave, $device_type=MODBUS_SLAVE]);
+		}
+
+	}

scripts/policy/protocols/modbus/track-memmap.bro

@@ -0,0 +1,105 @@
+##! This script tracks the memory map of holding (read/write) registers and logs
+##! changes as they are discovered.
+##!
+##! .. todo: Not all register reads and write functions are being supported yet.
+
+@load base/protocols/modbus
+@load base/utils/directions-and-hosts
+
+module Modbus;
+
+export {
+	redef enum Log::ID += { Modbus::REGISTER_CHANGE_LOG };
+
+	## The hosts that should have memory mapping enabled.
+	const track_memmap: Host = ALL_HOSTS &redef;
+
+	type MemmapInfo: record {
+		## Timestamp for the detected register change
+		ts:        time     &log;
+		## Unique ID for the connection
+		uid:       string   &log;
+		## Connection ID.
+		id:        conn_id  &log;
+		## The device memory offset.
+		register:  count    &log;
+		## The old value stored in the register.
+		old_val:   count    &log;
+		## The new value stored in the register.
+		new_val:   count    &log;
+		## The time delta between when the 'old_val' and 'new_val' were seen.
+		delta:     interval &log;
+	};
+
+	type RegisterValue: record {
+		last_set: time;
+		value:    count;
+	};
+
+	## Indexed on the device register value and yielding the register value.
+	type Registers: table[count] of RegisterValue;
+
+	## The memory map of slaves is tracked with this variable.
+	global device_registers: table[addr] of Registers;
+
+	## This event is generated every time a register is seen to be different than
+	## it was previously seen to be.
+	global changed_register: event(c: connection, register: count, old_val: count, new_val: count, delta: interval);
+}
+
+redef record Modbus::Info += {
+	track_address: count &default=0;
+};
+
+event bro_init() &priority=5
+	{
+	Log::create_stream(Modbus::REGISTER_CHANGE_LOG, [$columns=MemmapInfo]);
+	}
+
+event modbus_read_holding_registers_request(c: connection, headers: ModbusHeaders, start_address: count, quantity: count)
+	{
+	c$modbus$track_address = start_address+1;
+	}
+
+event modbus_read_holding_registers_response(c: connection, headers: ModbusHeaders, registers: ModbusRegisters)
+	{
+	local slave = c$id$resp_h;
+
+	if ( ! addr_matches_host(slave, track_memmap ) )
+		return;
+
+	if ( slave !in device_registers )
+		device_registers[slave] = table();
+
+	local slave_regs = device_registers[slave];
+	for ( i in registers )
+		{
+		if ( c$modbus$track_address in slave_regs )
+			{
+			if ( slave_regs[c$modbus$track_address]$value != registers[i] )
+				{
+				local delta = network_time() - slave_regs[c$modbus$track_address]$last_set;
+				event Modbus::changed_register(c, c$modbus$track_address, 
+				                               slave_regs[c$modbus$track_address]$value, registers[i],
+				                               delta);
+				
+				slave_regs[c$modbus$track_address]$last_set = network_time();
+				slave_regs[c$modbus$track_address]$value = registers[i];
+				}
+			}
+		else
+			{
+			local tmp_reg: RegisterValue = [$last_set=network_time(), $value=registers[i]];
+			slave_regs[c$modbus$track_address] = tmp_reg;
+			}
+
+		++c$modbus$track_address;
+		}
+	}
+
+event Modbus::changed_register(c: connection, register: count, old_val: count, new_val: count, delta: interval)
+	{
+	local rec: MemmapInfo = [$ts=network_time(), $uid=c$uid, $id=c$id, 
+	                         $register=register, $old_val=old_val, $new_val=new_val, $delta=delta];
+	Log::write(REGISTER_CHANGE_LOG, rec);
+	}

scripts/test-all-policy.bro

@@ -14,6 +14,16 @@
 # @load frameworks/control/controller.bro
 @load frameworks/dpd/detect-protocols.bro
 @load frameworks/dpd/packet-segment-logging.bro
+@load frameworks/intel/__load__.bro
+@load frameworks/intel/conn-established.bro
+@load frameworks/intel/dns.bro
+@load frameworks/intel/http-host-header.bro
+@load frameworks/intel/http-url.bro
+@load frameworks/intel/http-user-agents.bro
+@load frameworks/intel/smtp-url-extraction.bro
+@load frameworks/intel/smtp.bro
+@load frameworks/intel/ssl.bro
+@load frameworks/intel/where-locations.bro
 @load frameworks/metrics/conn-example.bro
 @load frameworks/metrics/http-example.bro
 @load frameworks/metrics/ssl-example.bro
@@ -22,6 +32,8 @@
 @load integration/barnyard2/__load__.bro
 @load integration/barnyard2/main.bro
 @load integration/barnyard2/types.bro
+@load integration/collective-intel/__load__.bro
+@load integration/collective-intel/main.bro
 @load misc/analysis-groups.bro
 @load misc/capture-loss.bro
 @load misc/loaded-scripts.bro
@@ -35,7 +47,6 @@
 @load protocols/dns/detect-external-names.bro
 @load protocols/ftp/detect.bro
 @load protocols/ftp/software.bro
-@load protocols/http/detect-intel.bro
 @load protocols/http/detect-MHR.bro
 @load protocols/http/detect-sqli.bro
 @load protocols/http/detect-webapps.bro
@@ -44,6 +55,8 @@
 @load protocols/http/software.bro
 @load protocols/http/var-extraction-cookies.bro
 @load protocols/http/var-extraction-uri.bro
+@load protocols/modbus/known-masters-slaves.bro
+@load protocols/modbus/track-memmap.bro
 @load protocols/smtp/blocklists.bro
 @load protocols/smtp/detect-suspicious-orig.bro
 @load protocols/smtp/software.bro

src/Analyzer.cc

@@ -28,6 +28,7 @@
 #include "DCE_RPC.h"
 #include "Gnutella.h"
 #include "Ident.h"
+#include "Modbus.h"
 #include "NCP.h"
 #include "NetbiosSSN.h"
 #include "SMB.h"
@@ -130,6 +131,9 @@ const Analyzer::Config Analyzer::analyzer_configs[] = {
 	{ AnalyzerTag::SYSLOG_BINPAC, "SYSLOG_BINPAC",
 		Syslog_Analyzer_binpac::InstantiateAnalyzer,
 		Syslog_Analyzer_binpac::Available, 0, false },
+	{ AnalyzerTag::Modbus, "MODBUS",
+		ModbusTCP_Analyzer::InstantiateAnalyzer,
+		ModbusTCP_Analyzer::Available, 0, false },
 
 	{ AnalyzerTag::AYIYA, "AYIYA",
 		AYIYA_Analyzer::InstantiateAnalyzer,

src/AnalyzerTags.h

@@ -32,6 +32,7 @@ namespace AnalyzerTag {
 		// Application-layer analyzers, binpac-generated.
 		DHCP_BINPAC, DNS_TCP_BINPAC, DNS_UDP_BINPAC,
 		HTTP_BINPAC, SSL, SYSLOG_BINPAC,
+		Modbus,
 
 		// Decapsulation analyzers.
 		AYIYA,

src/Attr.cc

@@ -71,7 +71,9 @@ void Attr::DescribeReST(ODesc* d) const
 
 		else if ( expr->Type()->Tag() == TYPE_FUNC )
 			{
-			d->Add(":bro:type:`func`");
+			d->Add(":bro:type:`");
+			d->Add(expr->Type()->AsFuncType()->FlavorString());
+			d->Add("`");
 			}
 
 		else
@@ -401,13 +403,13 @@ void Attributes::CheckAttr(Attr* a)
 
 	case ATTR_GROUP:
 		if ( type->Tag() != TYPE_FUNC ||
-		     ! type->AsFuncType()->IsEvent() )
+		     type->AsFuncType()->Flavor() != FUNC_FLAVOR_EVENT )
 			Error("&group only applicable to events");
 		break;
 
 	case ATTR_ERROR_HANDLER:
 		if ( type->Tag() != TYPE_FUNC ||
-		     ! type->AsFuncType()->IsEvent() )
+		     type->AsFuncType()->Flavor() != FUNC_FLAVOR_EVENT )
 			Error("&error_handler only applicable to events");
 		break;
 

src/BroDoc.cc

@@ -271,6 +271,7 @@ void BroDoc::WriteInterface(const char* heading, char underline,
 	WriteBroDocObjList(state_vars, isPublic, "State Variables", sub, isShort);
 	WriteBroDocObjList(types, isPublic, "Types", sub, isShort);
 	WriteBroDocObjList(events, isPublic, "Events", sub, isShort);
+	WriteBroDocObjList(hooks, isPublic, "Hooks", sub, isShort);
 	WriteBroDocObjList(functions, isPublic, "Functions", sub, isShort);
 	WriteBroDocObjList(redefs, isPublic, "Redefinitions", sub, isShort);
 	}

src/BroDoc.h

@@ -179,6 +179,30 @@ public:
 		all.push_back(o);
 		}
 
+	/**
+	 * Schedules documentation of a hook declared by the script.
+	 * @param o A pointer to a BroDocObj which contains the internal
+	 *        Bro language representation of the script hook and
+	 *        also any associated comments about it.
+	 */
+	void AddHook(const BroDocObj* o)
+		{
+		hooks.push_back(o);
+		all.push_back(o);
+		}
+
+	/**
+	 * Schedules documentation of a hook handler declared by the script.
+	 * @param o A pointer to a BroDocObj which contains the internal
+	 *        Bro language representation of the script hook handler and
+	 *        also any associated comments about it.
+	 */
+	void AddHookHandler(const BroDocObj* o)
+		{
+		hook_handlers.push_back(o);
+		all.push_back(o);
+		}
+
 	/**
 	 * Schedules documentation of a function declared by the script.
 	 * @param o A pointer to a BroDocObj which contains the internal
@@ -241,6 +265,8 @@ protected:
 	BroDocObjList notices;
 	BroDocObjList events;
 	BroDocObjList event_handlers;
+	BroDocObjList hooks;
+	BroDocObjList hook_handlers;
 	BroDocObjMap functions;
 	BroDocObjList redefs;
 

src/BroString.h

@@ -34,11 +34,15 @@ public:
 	typedef IdxVec::iterator IdxVecIt;
 	typedef IdxVec::const_iterator IdxVecCIt;
 
-	BroString(int arg_final_NUL, byte_vec str, int arg_n);
+	// Constructors creating internal copies of the data passed in.
 	BroString(const u_char* str, int arg_n, int add_NUL);
 	BroString(const char* str);
 	BroString(const string& str);
 	BroString(const BroString& bs);
+
+	// Constructor that takes owernship of the vector passed in.
+	BroString(int arg_final_NUL, byte_vec str, int arg_n);
+
 	BroString();
 	~BroString()	{ Reset(); }
 

src/CMakeLists.txt

@@ -218,6 +218,8 @@ binpac_target(ssl.pac
     ssl-defs.pac ssl-protocol.pac ssl-analyzer.pac)
 binpac_target(syslog.pac
     syslog-protocol.pac syslog-analyzer.pac)
+binpac_target(modbus.pac
+    modbus-protocol.pac modbus-analyzer.pac)
 
 ########################################################################
 ## bro target
@@ -349,6 +351,7 @@ set(bro_SRCS
     Reporter.cc
     Login.cc
     MIME.cc
+    Modbus.cc
     NCP.cc
     NFA.cc
     NFS.cc

src/DNS_Mgr.cc

@@ -46,13 +46,15 @@ extern int select(int, fd_set *, fd_set *, fd_set *, struct timeval *);
 
 class DNS_Mgr_Request {
 public:
-	DNS_Mgr_Request(const char* h, int af)	{ host = copy_string(h); fam = af; }
+	DNS_Mgr_Request(const char* h, int af, bool is_txt)
+		{ host = copy_string(h); fam = af; qtype = is_txt ? 16 : 0; }
 	DNS_Mgr_Request(const IPAddr& a)		{ addr = a; host = 0; fam = 0; }
 	~DNS_Mgr_Request()			{ delete [] host; }
 
 	// Returns nil if this was an address request.
 	const char* ReqHost() const	{ return host; }
 	const IPAddr& ReqAddr() const		{ return addr; }
+	const bool ReqIsTxt() const	{ return qtype == 16; }
 
 	int MakeRequest(nb_dns_info* nb_dns);
 	int RequestPending() const	{ return request_pending; }
@@ -61,7 +63,8 @@ public:
 
 protected:
 	char* host;	// if non-nil, this is a host request
-	int fam; // address family query type for host requests
+	int fam;	// address family query type for host requests
+	int qtype;	// Query type
 	IPAddr addr;
 	int request_pending;
 };
@@ -75,7 +78,7 @@ int DNS_Mgr_Request::MakeRequest(nb_dns_info* nb_dns)
 
 	char err[NB_DNS_ERRSIZE];
 	if ( host )
-		return nb_dns_host_request2(nb_dns, host, fam, (void*) this, err) >= 0;
+		return nb_dns_host_request2(nb_dns, host, fam, qtype, (void*) this, err) >= 0;
 	else
 		{
 		const uint32* bytes;
@@ -475,8 +478,8 @@ TableVal* DNS_Mgr::LookupHost(const char* name)
 	// Not found, or priming.
 	switch ( mode ) {
 	case DNS_PRIME:
-		requests.append(new DNS_Mgr_Request(name, AF_INET));
+		requests.append(new DNS_Mgr_Request(name, AF_INET, false));
-		requests.append(new DNS_Mgr_Request(name, AF_INET6));
+		requests.append(new DNS_Mgr_Request(name, AF_INET6, false));
 		return empty_addr_set();
 
 	case DNS_FORCE:
@@ -484,8 +487,8 @@ TableVal* DNS_Mgr::LookupHost(const char* name)
 		return 0;
 
 	case DNS_DEFAULT:
-		requests.append(new DNS_Mgr_Request(name, AF_INET));
+		requests.append(new DNS_Mgr_Request(name, AF_INET, false));
-		requests.append(new DNS_Mgr_Request(name, AF_INET6));
+		requests.append(new DNS_Mgr_Request(name, AF_INET6, false));
 		Resolve();
 		return LookupHost(name);
 
@@ -636,6 +639,7 @@ int DNS_Mgr::Save()
 
 	Save(f, host_mappings);
 	Save(f, addr_mappings);
+	// Save(f, text_mappings); // We don't save the TXT mappings (yet?).
 
 	fclose(f);
 
@@ -704,39 +708,48 @@ void DNS_Mgr::AddResult(DNS_Mgr_Request* dr, struct nb_dns_result* r)
 		new_dm = new DNS_Mapping(dr->ReqHost(), h, ttl);
 		prev_dm = 0;
 
-		HostMap::iterator it = host_mappings.find(dr->ReqHost());
+		if ( dr->ReqIsTxt() )
-		if ( it == host_mappings.end() )
 			{
-			host_mappings[dr->ReqHost()].first =
+			TextMap::iterator it = text_mappings.find(dr->ReqHost());
-					new_dm->Type() == AF_INET ? new_dm : 0;
+			if ( it == text_mappings.end() )
-			host_mappings[dr->ReqHost()].second =
+				text_mappings[dr->ReqHost()] = new_dm;
-					new_dm->Type() == AF_INET ? 0 : new_dm;
 			}
-
 		else
 			{
-			if ( new_dm->Type() == AF_INET )
+			HostMap::iterator it = host_mappings.find(dr->ReqHost());
+			if ( it == host_mappings.end() )
 				{
-				prev_dm = it->second.first;
+				host_mappings[dr->ReqHost()].first =
-				it->second.first = new_dm;
+					new_dm->Type() == AF_INET ? new_dm : 0;
+
+				host_mappings[dr->ReqHost()].second =
+					new_dm->Type() == AF_INET ? 0 : new_dm;
 				}
 			else
 				{
-				prev_dm = it->second.second;
+				if ( new_dm->Type() == AF_INET )
-				it->second.second = new_dm;
+					{
+					prev_dm = it->second.first;
+					it->second.first = new_dm;
+					}
+				else
+					{
+					prev_dm = it->second.second;
+					it->second.second = new_dm;
+					}
 				}
-			}
 
-		if ( new_dm->Failed() && prev_dm && prev_dm->Valid() )
+			if ( new_dm->Failed() && prev_dm && prev_dm->Valid() )
-			{
+				{
-			// Put previous, valid entry back - CompareMappings
+				// Put previous, valid entry back - CompareMappings
-			// will generate a corresponding warning.
+				// will generate a corresponding warning.
-			if ( prev_dm->Type() == AF_INET )
+				if ( prev_dm->Type() == AF_INET )
-				host_mappings[dr->ReqHost()].first = prev_dm;
+					host_mappings[dr->ReqHost()].first = prev_dm;
-			else
+				else
-				host_mappings[dr->ReqHost()].second = prev_dm;
+					host_mappings[dr->ReqHost()].second = prev_dm;
 
-			++keep_prev;
+				++keep_prev;
+				}
 			}
 		}
 	else
@@ -928,7 +941,10 @@ TableVal* DNS_Mgr::LookupNameInCache(string name)
 	{
 	HostMap::iterator it = dns_mgr->host_mappings.find(name);
 	if ( it == dns_mgr->host_mappings.end() )
+		{
+		it = dns_mgr->host_mappings.begin();
 		return 0;
+		}
 
 	DNS_Mapping* d4 = it->second.first;
 	DNS_Mapping* d6 = it->second.second;
@@ -951,6 +967,26 @@ TableVal* DNS_Mgr::LookupNameInCache(string name)
 	return tv6;
 	}
 
+const char* DNS_Mgr::LookupTextInCache(string name)
+	{
+	TextMap::iterator it = dns_mgr->text_mappings.find(name);
+	if ( it == dns_mgr->text_mappings.end() )
+		return 0;
+
+	DNS_Mapping* d = it->second;
+
+	if ( d->Expired() )
+		{
+		dns_mgr->text_mappings.erase(it);
+		delete d;
+		return 0;
+		}
+
+	// The escapes in the following strings are to avoid having it
+	// interpreted as a trigraph sequence.
+	return d->names ? d->names[0] : "<\?\?\?>";
+	}
+
 void DNS_Mgr::AsyncLookupAddr(const IPAddr& host, LookupCallback* callback)
 	{
 	if ( ! did_init )
@@ -1020,6 +1056,35 @@ void DNS_Mgr::AsyncLookupName(string name, LookupCallback* callback)
 	IssueAsyncRequests();
 	}
 
+void DNS_Mgr::AsyncLookupNameText(string name, LookupCallback* callback)
+	{
+	if ( ! did_init )
+		Init();
+
+	// Do we already know the answer?
+	TableVal* addrs;
+
+	AsyncRequest* req = 0;
+
+	// Have we already a request waiting for this host?
+	AsyncRequestTextMap::iterator i = asyncs_texts.find(name);
+	if ( i != asyncs_texts.end() )
+		req = i->second;
+	else
+		{
+		// A new one.
+		req = new AsyncRequest;
+		req->name = name;
+		req->is_txt = true;
+		asyncs_queued.push_back(req);
+		asyncs_texts.insert(AsyncRequestTextMap::value_type(name, req));
+		}
+
+	req->callbacks.push_back(callback);
+
+	IssueAsyncRequests();
+	}
+
 void DNS_Mgr::IssueAsyncRequests()
 	{
 	while ( asyncs_queued.size() && asyncs_pending < MAX_PENDING_REQUESTS )
@@ -1036,8 +1101,9 @@ void DNS_Mgr::IssueAsyncRequests()
 			dr = new DNS_Mgr_Request(req->host);
 		else
 			{
-			dr = new DNS_Mgr_Request(req->name.c_str(), AF_INET);
+			dr = new DNS_Mgr_Request(req->name.c_str(), AF_INET, req->is_txt);
-			dr6 = new DNS_Mgr_Request(req->name.c_str(), AF_INET6);
+			if ( ! req->is_txt )
+				dr6 = new DNS_Mgr_Request(req->name.c_str(), AF_INET6, req->is_txt);
 			}
 
 		if ( ! dr->MakeRequest(nb_dns) )
@@ -1109,6 +1175,38 @@ void DNS_Mgr::CheckAsyncAddrRequest(const IPAddr& addr, bool timeout)
 
 	}
 
+void DNS_Mgr::CheckAsyncTextRequest(const char* host, bool timeout)
+	{
+	// Note that this code is a mirror of that for CheckAsyncAddrRequest.
+
+	AsyncRequestTextMap::iterator i = asyncs_texts.find(host);
+	if ( i != asyncs_texts.end() )
+		{
+		const char* name = LookupTextInCache(host);
+		if ( name )
+			{
+			++successful;
+			i->second->Resolved(name);
+			}
+
+		else if ( timeout )
+			{
+			AsyncRequestTextMap::iterator it = asyncs_texts.begin();
+			++failed;
+			i->second->Timeout();
+			}
+
+		else
+			return;
+
+		asyncs_texts.erase(i);
+		--asyncs_pending;
+
+		// Don't delete the request.  That will be done once it
+		// eventually times out.
+		}
+	}
+
 void DNS_Mgr::CheckAsyncHostRequest(const char* host, bool timeout)
 	{
 	// Note that this code is a mirror of that for CheckAsyncAddrRequest.
@@ -1157,8 +1255,12 @@ void DNS_Mgr::Flush()
 	for ( AddrMap::iterator it2 = addr_mappings.begin(); it2 != addr_mappings.end(); ++it2 )
 		delete it2->second;
 
+	for ( TextMap::iterator it3 = text_mappings.begin(); it3 != text_mappings.end(); ++it3 )
+		delete it3->second;
+
 	host_mappings.clear();
 	addr_mappings.clear();
+	text_mappings.clear();
 	}
 
 void DNS_Mgr::Process()
@@ -1177,6 +1279,8 @@ void DNS_Mgr::DoProcess(bool flush)
 
 		if ( req->IsAddrReq() )
 			CheckAsyncAddrRequest(req->host, true);
+		else if ( req->is_txt )
+			CheckAsyncTextRequest(req->name.c_str(), true);
 		else
 			CheckAsyncHostRequest(req->name.c_str(), true);
 
@@ -1184,7 +1288,7 @@ void DNS_Mgr::DoProcess(bool flush)
 		delete req;
 		}
 
-	if ( asyncs_addrs.size() == 0 && asyncs_names.size() == 0 )
+	if ( asyncs_addrs.size() == 0 && asyncs_names.size() == 0 && asyncs_texts.size() == 0 )
 		return;
 
 	if ( AnswerAvailable(0) <= 0 )
@@ -1217,6 +1321,8 @@ void DNS_Mgr::DoProcess(bool flush)
 
 		if ( ! dr->ReqHost() )
 			CheckAsyncAddrRequest(dr->ReqAddr(), true);
+		else if ( dr->ReqIsTxt() )
+			CheckAsyncTextRequest(dr->ReqHost(), do_host_timeout);
 		else
 			CheckAsyncHostRequest(dr->ReqHost(), do_host_timeout);
 
@@ -1271,5 +1377,6 @@ void DNS_Mgr::GetStats(Stats* stats)
 	stats->pending = asyncs_pending;
 	stats->cached_hosts = host_mappings.size();
 	stats->cached_addresses = addr_mappings.size();
+	stats->cached_texts = text_mappings.size();
 	}
 

src/DNS_Mgr.h

@@ -63,6 +63,7 @@ public:
 
 	const char* LookupAddrInCache(const IPAddr& addr);
 	TableVal* LookupNameInCache(string name);
+	const char* LookupTextInCache(string name);
 
 	// Support for async lookups.
 	class LookupCallback {
@@ -77,6 +78,7 @@ public:
 
 	void AsyncLookupAddr(const IPAddr& host, LookupCallback* callback);
 	void AsyncLookupName(string name, LookupCallback* callback);
+	void AsyncLookupNameText(string name, LookupCallback* callback);
 
 	struct Stats {
 		unsigned long requests;	// These count only async requests.
@@ -85,6 +87,7 @@ public:
 		unsigned long pending;
 		unsigned long cached_hosts;
 		unsigned long cached_addresses;
+		unsigned long cached_texts;
 	};
 
 	void GetStats(Stats* stats);
@@ -106,6 +109,7 @@ protected:
 
 	typedef map<string, pair<DNS_Mapping*, DNS_Mapping*> > HostMap;
 	typedef map<IPAddr, DNS_Mapping*> AddrMap;
+	typedef map<string, DNS_Mapping*> TextMap;
 	void LoadCache(FILE* f);
 	void Save(FILE* f, const AddrMap& m);
 	void Save(FILE* f, const HostMap& m);
@@ -122,6 +126,7 @@ protected:
 	// requested.
 	void CheckAsyncAddrRequest(const IPAddr& addr, bool timeout);
 	void CheckAsyncHostRequest(const char* host, bool timeout);
+	void CheckAsyncTextRequest(const char* host, bool timeout);
 
 	// Process outstanding requests.
 	void DoProcess(bool flush);
@@ -138,6 +143,7 @@ protected:
 
 	HostMap host_mappings;
 	AddrMap addr_mappings;
+	TextMap text_mappings;
 
 	DNS_mgr_request_list requests;
 
@@ -165,8 +171,11 @@ protected:
 		double time;
 		IPAddr host;
 		string name;
+		bool is_txt;
 		CallbackList callbacks;
 
+		AsyncRequest() : time(0.0), is_txt(false) { }
+
 		bool IsAddrReq() const	{ return name.length() == 0; }
 
 		void Resolved(const char* name)
@@ -210,6 +219,9 @@ protected:
 	typedef map<string, AsyncRequest*> AsyncRequestNameMap;
 	AsyncRequestNameMap asyncs_names;
 
+	typedef map<string, AsyncRequest*> AsyncRequestTextMap;
+	AsyncRequestTextMap asyncs_texts;
+
 	typedef list<AsyncRequest*> QueuedList;
 	QueuedList asyncs_queued;
 

src/Expr.cc

@@ -4374,7 +4374,7 @@ bool InExpr::DoUnserialize(UnserialInfo* info)
 	return true;
 	}
 
-CallExpr::CallExpr(Expr* arg_func, ListExpr* arg_args)
+CallExpr::CallExpr(Expr* arg_func, ListExpr* arg_args, bool in_hook)
 : Expr(EXPR_CALL)
 	{
 	func = arg_func;
@@ -4402,8 +4402,33 @@ CallExpr::CallExpr(Expr* arg_func, ListExpr* arg_args)
 
 		if ( ! yield )
 			{
-			Error("event called in expression");
+			switch ( func_type->AsFuncType()->Flavor() ) {
-			SetError();
+
+			case FUNC_FLAVOR_FUNCTION:
+				Error("function has no yield type");
+				SetError();
+				break;
+
+			case FUNC_FLAVOR_EVENT:
+				Error("event called in expression, use event statement instead");
+				SetError();
+				break;
+
+			case FUNC_FLAVOR_HOOK:
+				// It's fine to not have a yield if it's known that the call
+				// is being done from a hook statement.
+				if ( ! in_hook )
+					{
+					Error("hook called in expression, use hook statement instead");
+					SetError();
+					}
+				break;
+
+			default:
+				Error("invalid function flavor");
+				SetError();
+				break;
+			}
 			}
 		else
 			SetType(yield->Ref());

src/Expr.h

@@ -959,7 +959,7 @@ protected:
 
 class CallExpr : public Expr {
 public:
-	CallExpr(Expr* func, ListExpr* args);
+	CallExpr(Expr* func, ListExpr* args, bool in_hook = false);
 	~CallExpr();
 
 	Expr* Func() const	{ return func; }

src/Func.cc

@@ -284,8 +284,8 @@ Val* BroFunc::Call(val_list* args, Frame* parent) const
 #endif
 	if ( ! bodies.size() ) 
 		{
-		// Can only happen for events.
+		// Can only happen for events and hooks.
-		assert(IsEvent());
+		assert(Flavor() == FUNC_FLAVOR_EVENT || Flavor() == FUNC_FLAVOR_HOOK);
 		loop_over_list(*args, i)
 			Unref((*args)[i]);
 		return 0 ;
@@ -309,7 +309,7 @@ Val* BroFunc::Call(val_list* args, Frame* parent) const
 		DescribeDebug(&d, args);
 
 		g_trace_state.LogTrace("%s called: %s\n",
-			IsEvent() ? "event" : "function", d.Description());
+			FType()->FlavorString().c_str(), d.Description());
 		}
 
 	loop_over_list(*args, i)
@@ -348,6 +348,12 @@ Val* BroFunc::Call(val_list* args, Frame* parent) const
 			parent->SetDelayed();
 			break;
 			}
+
+		if ( flow == FLOW_BREAK && Flavor() == FUNC_FLAVOR_HOOK )
+			{
+			// short-circuit execution of remaining hook handler bodies
+			break;
+			}
 		}
 
 	// Warn if the function returns something, but we returned from
@@ -380,7 +386,7 @@ void BroFunc::AddBody(Stmt* new_body, id_list* new_inits, int new_frame_size,
 
 	new_body = AddInits(new_body, new_inits);
 
-	if ( ! IsEvent() )
+	if ( Flavor() == FUNC_FLAVOR_FUNCTION )
 		{
 		// For functions, we replace the old body with the new one.
 		assert(bodies.size() <= 1);

src/Func.h

@@ -25,7 +25,7 @@ public:
 	virtual ~Func();
 
 	virtual int IsPure() const = 0;
-	int IsEvent() const	{ return FType()->IsEvent(); }
+	function_flavor Flavor() const	{ return FType()->Flavor(); }
 
 	struct Body {
 		Stmt* stmts;

src/ID.cc

@@ -107,7 +107,8 @@ void ID::SetVal(Val* v, Opcode op, bool arg_weak_ref)
 #endif
 
 	if ( type && val &&
-	     type->Tag() == TYPE_FUNC && type->AsFuncType()->IsEvent() )
+	     type->Tag() == TYPE_FUNC &&
+	     type->AsFuncType()->Flavor() == FUNC_FLAVOR_EVENT )
 		{
 		EventHandler* handler = event_registry->Lookup(name);
 		if ( ! handler )
@@ -657,11 +658,19 @@ void ID::DescribeReSTShort(ODesc* d) const
 				break;
 
 			case TYPE_FUNC:
-				d->Add(type->AsFuncType()->IsEvent() ? "event" : type_name(t));
+				d->Add(type->AsFuncType()->FlavorString());
+				break;
+
+			case TYPE_ENUM:
+				if ( is_type )
+					d->Add(type_name(t));
+				else
+					d->Add(type->AsEnumType()->Name().c_str());
 				break;
 
 			default:
 				d->Add(type_name(t));
+				break;
 			}
 			}
 

src/IPAddr.cc

@@ -248,10 +248,10 @@ IPPrefix::IPPrefix(const in6_addr& in6, uint8_t length)
 	prefix.Mask(this->length);
 	}
 
-IPPrefix::IPPrefix(const IPAddr& addr, uint8_t length)
+IPPrefix::IPPrefix(const IPAddr& addr, uint8_t length, bool len_is_v6_relative)
 	: prefix(addr)
 	{
-	if ( prefix.GetFamily() == IPv4 )
+	if ( prefix.GetFamily() == IPv4 && ! len_is_v6_relative )
 		{
 		if ( length > 32 )
 			reporter->InternalError("Bad IPAddr(v4) IPPrefix length : %d",

src/IPAddr.h

@@ -342,6 +342,21 @@ public:
 		return memcmp(&addr1.in6, &addr2.in6, sizeof(in6_addr)) < 0;
 		}
 
+	friend bool operator<=(const IPAddr& addr1, const IPAddr& addr2)
+		{
+		return addr1 < addr2 || addr1 == addr2;
+		}
+
+	friend bool operator>=(const IPAddr& addr1, const IPAddr& addr2)
+		{
+		return ! ( addr1 < addr2 );
+		}
+
+	friend bool operator>(const IPAddr& addr1, const IPAddr& addr2)
+		{
+		return ! ( addr1 <= addr2 );
+		}
+
 	/** Converts the address into the type used internally by the
 	  * inter-thread communication.
 	  */
@@ -481,8 +496,15 @@ public:
 	 * @param addr The IP address.
 	 *
 	 * @param length The prefix length in the range from 0 to 128
+	 *
+	 * @param len_is_v6_relative Whether \a length is relative to the full
+	 * 128 bits of an IPv6 address.  If false and \a addr is an IPv4
+	 * address, then \a length is expected to range from 0 to 32.  If true
+	 * \a length is expected to range from 0 to 128 even if \a addr is IPv4,
+	 * meaning that the mask is to apply to the IPv4-mapped-IPv6 representation.
 	 */
-	IPPrefix(const IPAddr& addr, uint8_t length);
+	IPPrefix(const IPAddr& addr, uint8_t length,
+	         bool len_is_v6_relative = false);
 
 	/**
 	 * Copy constructor.
@@ -583,6 +605,11 @@ public:
 		return net1.Prefix() == net2.Prefix() && net1.Length() == net2.Length();
 		}
 
+	friend bool operator!=(const IPPrefix& net1, const IPPrefix& net2)
+		{
+		return ! (net1 == net2);
+		}
+
 	/**
 	 * Comparison operator IP prefixes. This defines a well-defined order for
 	 * IP prefix. However, the order does not necessarily corresponding to their
@@ -600,6 +627,21 @@ public:
 			return false;
 		}
 
+	friend bool operator<=(const IPPrefix& net1, const IPPrefix& net2)
+		{
+		return net1 < net2 || net1 == net2;
+		}
+
+	friend bool operator>=(const IPPrefix& net1, const IPPrefix& net2)
+		{
+		return ! (net1 < net2 );
+		}
+
+	friend bool operator>(const IPPrefix& net1, const IPPrefix& net2)
+		{
+		return ! ( net1 <= net2 );
+		}
+
 private:
 	IPAddr prefix;	// We store it as an address with the non-prefix bits masked out via Mask().
 	uint8_t length;	// The bit length of the prefix relative to full IPv6 addr.

src/Modbus.cc

@@ -0,0 +1,41 @@
+
+#include "Modbus.h"
+#include "TCP_Reassembler.h"
+
+ModbusTCP_Analyzer::ModbusTCP_Analyzer(Connection* c)
+	: TCP_ApplicationAnalyzer(AnalyzerTag::Modbus, c)
+	{
+	interp = new binpac::ModbusTCP::ModbusTCP_Conn(this);
+	}
+
+ModbusTCP_Analyzer::~ModbusTCP_Analyzer()
+	{
+	delete interp;
+	}
+
+void ModbusTCP_Analyzer::Done()
+	{
+	TCP_ApplicationAnalyzer::Done();
+
+	interp->FlowEOF(true);
+	interp->FlowEOF(false);
+	}
+
+void ModbusTCP_Analyzer::DeliverStream(int len, const u_char* data, bool orig)
+	{
+	TCP_ApplicationAnalyzer::DeliverStream(len, data, orig);
+	interp->NewData(orig, data, data + len);
+	}
+
+void ModbusTCP_Analyzer::Undelivered(int seq, int len, bool orig)
+	{
+	TCP_ApplicationAnalyzer::Undelivered(seq, len, orig);
+	interp->NewGap(orig, len);
+	}
+
+void ModbusTCP_Analyzer::EndpointEOF(bool is_orig)
+	{
+	TCP_ApplicationAnalyzer::EndpointEOF(is_orig);
+	interp->FlowEOF(is_orig);
+	}
+

src/Modbus.h

@@ -0,0 +1,58 @@
+#ifndef MODBUS_H
+#define MODBUS_H
+
+#include "TCP.h"
+#include "modbus_pac.h"
+
+class ModbusTCP_Analyzer : public TCP_ApplicationAnalyzer {
+public:
+	ModbusTCP_Analyzer(Connection* conn);
+	virtual ~ModbusTCP_Analyzer();
+
+	virtual void Done();
+	virtual void DeliverStream(int len, const u_char* data, bool orig);
+
+	virtual void Undelivered(int seq, int len, bool orig);
+	virtual void EndpointEOF(bool is_orig);
+
+	static Analyzer* InstantiateAnalyzer(Connection* conn)
+		{ return new ModbusTCP_Analyzer(conn); }
+
+	// Put event names in this function
+	static bool Available()
+		{
+		return modbus_message
+		     | modbus_exception
+		     | modbus_read_coils_request
+		     | modbus_read_coils_response
+		     | modbus_read_discrete_inputs_request
+		     | modbus_read_discrete_inputs_response
+		     | modbus_read_holding_registers_request
+		     | modbus_read_holding_registers_response
+		     | modbus_read_input_registers_request
+		     | modbus_read_input_registers_response
+		     | modbus_write_single_coil_request
+		     | modbus_write_single_coil_response
+		     | modbus_write_single_register_request
+		     | modbus_write_single_register_response
+		     | modbus_write_multiple_coils_request
+		     | modbus_write_multiple_coils_response
+		     | modbus_write_multiple_registers_request
+		     | modbus_write_multiple_registers_response
+		     | modbus_read_file_record_request
+		     | modbus_read_file_record_response
+		     | modbus_write_file_record_request
+		     | modbus_write_file_record_response
+		     | modbus_mask_write_register_request
+		     | modbus_mask_write_register_response
+		     | modbus_read_write_multiple_registers_request
+		     | modbus_read_write_multiple_registers_response
+		     | modbus_read_fifo_queue_request
+		     | modbus_read_fifo_queue_response;
+		}
+
+protected:
+	binpac::ModbusTCP::ModbusTCP_Conn* interp;
+};
+
+#endif

src/PktSrc.cc

@@ -219,16 +219,35 @@ void PktSrc::Process()
 		// Get protocol being carried from the ethernet frame.
 		protocol = (data[12] << 8) + data[13];
 
-		// MPLS carried over the ethernet frame.
+		switch ( protocol )
-		if ( protocol == 0x8847 )
-			have_mpls = true;
-
-		// VLAN carried over ethernet frame.
-		else if ( protocol == 0x8100 )
 			{
-			data += get_link_header_size(datalink);
+			// MPLS carried over the ethernet frame.
-			data += 4; // Skip the vlan header
+			case 0x8847:
-			pkt_hdr_size = 0;
+				have_mpls = true;
+				break;
+
+			// VLAN carried over the ethernet frame.
+			case 0x8100:
+				data += get_link_header_size(datalink);
+				data += 4; // Skip the vlan header
+				pkt_hdr_size = 0;
+				break;
+
+			// PPPoE carried over the ethernet frame.
+			case 0x8864:
+				data += get_link_header_size(datalink);
+				protocol = (data[6] << 8) + data[7];
+				data += 8; // Skip the PPPoE session and PPP header
+				pkt_hdr_size = 0;
+
+				if ( protocol != 0x0021 && protocol != 0x0057 )
+					{
+					// Neither IPv4 nor IPv6.
+					sessions->Weird("non_ip_packet_in_pppoe_encapsulation", &hdr, data);
+					data = 0;
+					return;
+					}
+				break;
 			}
 
 		break;

src/RuleMatcher.cc

@@ -1,4 +1,5 @@
 #include <algorithm>
+#include <functional>
 
 #include "config.h"
 
@@ -41,6 +42,23 @@ RuleHdrTest::RuleHdrTest(Prot arg_prot, uint32 arg_offset, uint32 arg_size,
 	level = 0;
 	}
 
+RuleHdrTest::RuleHdrTest(Prot arg_prot, Comp arg_comp, vector<IPPrefix> arg_v)
+	{
+	prot = arg_prot;
+	offset = 0;
+	size = 0;
+	comp = arg_comp;
+	vals = new maskedvalue_list;
+	prefix_vals = arg_v;
+	sibling = 0;
+	child = 0;
+	pattern_rules = 0;
+	pure_rules = 0;
+	ruleset = new IntSet;
+	id = ++idcounter;
+	level = 0;
+	}
+
 Val* RuleMatcher::BuildRuleStateValue(const Rule* rule,
 					const RuleEndpointState* state) const
 	{
@@ -63,6 +81,8 @@ RuleHdrTest::RuleHdrTest(RuleHdrTest& h)
 	loop_over_list(*h.vals, i)
 		vals->append(new MaskedValue(*(*h.vals)[i]));
 
+	prefix_vals = h.prefix_vals;
+
 	for ( int j = 0; j < Rule::TYPES; ++j )
 		{
 		loop_over_list(h.psets[j], k)
@@ -114,6 +134,10 @@ bool RuleHdrTest::operator==(const RuleHdrTest& h)
 		     (*vals)[i]->mask != (*h.vals)[i]->mask )
 			return false;
 
+	for ( size_t i = 0; i < prefix_vals.size(); ++i )
+		if ( ! (prefix_vals[i] == h.prefix_vals[i]) )
+			return false;
+
 	return true;
 	}
 
@@ -129,6 +153,9 @@ void RuleHdrTest::PrintDebug()
 		fprintf(stderr, " 0x%08x/0x%08x",
 				(*vals)[i]->val, (*vals)[i]->mask);
 
+	for ( size_t i = 0; i < prefix_vals.size(); ++i )
+		fprintf(stderr, " %s", prefix_vals[i].AsString().c_str());
+
 	fprintf(stderr, "\n");
 	}
 
@@ -410,29 +437,129 @@ static inline uint32 getval(const u_char* data, int size)
 	}
 
 
-// A line which can be inserted into the macros below for debugging
-// fprintf(stderr, "%.06f %08x & %08x %s %08x\n", network_time, v, (mvals)[i]->mask, #op, (mvals)[i]->val);
-
 // Evaluate a value list (matches if at least one value matches).
-#define DO_MATCH_OR( mvals, v, op )	\
+template <typename FuncT>
-	{	\
+static inline bool match_or(const maskedvalue_list& mvals, uint32 v, FuncT comp)
-	loop_over_list((mvals), i)	\
+	{
-		{	\
+	loop_over_list(mvals, i)
-		if ( ((v) & (mvals)[i]->mask) op (mvals)[i]->val )	\
+		{
-			goto match;	\
+		if ( comp(v & mvals[i]->mask, mvals[i]->val) )
-		}	\
+			return true;
-	goto no_match;	\
+		}
+	return false;
+	}
+
+// Evaluate a prefix list (matches if at least one value matches).
+template <typename FuncT>
+static inline bool match_or(const vector<IPPrefix>& prefixes, const IPAddr& a,
+                            FuncT comp)
+	{
+	for ( size_t i = 0; i < prefixes.size(); ++i )
+		{
+		IPAddr masked(a);
+		masked.Mask(prefixes[i].LengthIPv6());
+		if ( comp(masked, prefixes[i].Prefix()) )
+			return true;
+		}
+	return false;
 	}
 
 // Evaluate a value list (doesn't match if any value matches).
-#define DO_MATCH_NOT_AND( mvals, v, op )	\
+template <typename FuncT>
-	{	\
+static inline bool match_not_and(const maskedvalue_list& mvals, uint32 v,
-	loop_over_list((mvals), i)	\
+                                 FuncT comp)
-		{	\
+	{
-		if ( ((v) & (mvals)[i]->mask) op (mvals)[i]->val )	\
+	loop_over_list(mvals, i)
-			goto no_match;	\
+		{
-		}	\
+		if ( comp(v & mvals[i]->mask, mvals[i]->val) )
-	goto match;	\
+			return false;
+		}
+	return true;
+	}
+
+// Evaluate a prefix list (doesn't match if any value matches).
+template <typename FuncT>
+static inline bool match_not_and(const vector<IPPrefix>& prefixes,
+                                 const IPAddr& a, FuncT comp)
+	{
+	for ( size_t i = 0; i < prefixes.size(); ++i )
+		{
+		IPAddr masked(a);
+		masked.Mask(prefixes[i].LengthIPv6());
+		if ( comp(masked, prefixes[i].Prefix()) )
+			return false;
+		}
+	return true;
+	}
+
+static inline bool compare(const maskedvalue_list& mvals, uint32 v,
+                           RuleHdrTest::Comp comp)
+	{
+	switch ( comp ) {
+		case RuleHdrTest::EQ:
+			return match_or(mvals, v, std::equal_to<uint32>());
+			break;
+
+		case RuleHdrTest::NE:
+			return match_not_and(mvals, v, std::equal_to<uint32>());
+			break;
+
+		case RuleHdrTest::LT:
+			return match_or(mvals, v, std::less<uint32>());
+			break;
+
+		case RuleHdrTest::GT:
+			return match_or(mvals, v, std::greater<uint32>());
+			break;
+
+		case RuleHdrTest::LE:
+			return match_or(mvals, v, std::less_equal<uint32>());
+			break;
+
+		case RuleHdrTest::GE:
+			return match_or(mvals, v, std::greater_equal<uint32>());
+			break;
+
+		default:
+			reporter->InternalError("unknown comparison type");
+			break;
+	}
+	return false;
+	}
+
+static inline bool compare(const vector<IPPrefix>& prefixes, const IPAddr& a,
+                           RuleHdrTest::Comp comp)
+	{
+	switch ( comp ) {
+		case RuleHdrTest::EQ:
+			return match_or(prefixes, a, std::equal_to<IPAddr>());
+			break;
+
+		case RuleHdrTest::NE:
+			return match_not_and(prefixes, a, std::equal_to<IPAddr>());
+			break;
+
+		case RuleHdrTest::LT:
+			return match_or(prefixes, a, std::less<IPAddr>());
+			break;
+
+		case RuleHdrTest::GT:
+			return match_or(prefixes, a, std::greater<IPAddr>());
+			break;
+
+		case RuleHdrTest::LE:
+			return match_or(prefixes, a, std::less_equal<IPAddr>());
+			break;
+
+		case RuleHdrTest::GE:
+			return match_or(prefixes, a, std::greater_equal<IPAddr>());
+			break;
+
+		default:
+			reporter->InternalError("unknown comparison type");
+			break;
+	}
+	return false;
 	}
 
 RuleEndpointState* RuleMatcher::InitEndpoint(Analyzer* analyzer,
@@ -492,66 +619,54 @@ RuleEndpointState* RuleMatcher::InitEndpoint(Analyzer* analyzer,
 
 		if ( ip )
 			{
-			// Get start of transport layer.
-			const u_char* transport = ip->Payload();
-
 			// Descend the RuleHdrTest tree further.
 			for ( RuleHdrTest* h = hdr_test->child; h;
 			      h = h->sibling )
 				{
-				const u_char* data;
+				bool match = false;
 
 				// Evaluate the header test.
 				switch ( h->prot ) {
+				case RuleHdrTest::NEXT:
+					match = compare(*h->vals, ip->NextProto(), h->comp);
+					break;
+
 				case RuleHdrTest::IP:
-					data = (const u_char*) ip->IP4_Hdr();
+					if ( ! ip->IP4_Hdr() )
+						continue;
+
+					match = compare(*h->vals, getval((const u_char*)ip->IP4_Hdr() + h->offset, h->size), h->comp);
+					break;
+
+				case RuleHdrTest::IPv6:
+					if ( ! ip->IP6_Hdr() )
+						continue;
+
+					match = compare(*h->vals, getval((const u_char*)ip->IP6_Hdr() + h->offset, h->size), h->comp);
 					break;
 
 				case RuleHdrTest::ICMP:
+				case RuleHdrTest::ICMPv6:
 				case RuleHdrTest::TCP:
 				case RuleHdrTest::UDP:
-					data = transport;
+					match = compare(*h->vals, getval(ip->Payload() + h->offset, h->size), h->comp);
+					break;
+
+				case RuleHdrTest::IPSrc:
+					match = compare(h->prefix_vals, ip->IPHeaderSrcAddr(), h->comp);
+					break;
+
+				case RuleHdrTest::IPDst:
+					match = compare(h->prefix_vals, ip->IPHeaderDstAddr(), h->comp);
 					break;
 
 				default:
-					data = 0;
 					reporter->InternalError("unknown protocol");
+					break;
 				}
 
-				// ### data can be nil here if it's an
+				if ( match )
-				// IPv6 packet and we're doing an IP test.
+					tests.append(h);
-				if ( ! data )
-					continue;
-
-				// Sorry for the hidden gotos :-)
-				switch ( h->comp ) {
-					case RuleHdrTest::EQ:
-						DO_MATCH_OR(*h->vals, getval(data + h->offset, h->size), ==);
-
-					case RuleHdrTest::NE:
-						DO_MATCH_NOT_AND(*h->vals, getval(data + h->offset, h->size), ==);
-
-					case RuleHdrTest::LT:
-						DO_MATCH_OR(*h->vals, getval(data + h->offset, h->size), <);
-
-					case RuleHdrTest::GT:
-						DO_MATCH_OR(*h->vals, getval(data + h->offset, h->size), >);
-
-					case RuleHdrTest::LE:
-						DO_MATCH_OR(*h->vals, getval(data + h->offset, h->size), <=);
-
-					case RuleHdrTest::GE:
-						DO_MATCH_OR(*h->vals, getval(data + h->offset, h->size), >=);
-
-					default:
-						reporter->InternalError("unknown comparision type");
-				}
-
-no_match:
-				continue;
-
-match:
-				tests.append(h);
 				}
 			}
 		}
@@ -1050,8 +1165,11 @@ static Val* get_bro_val(const char* label)
 	}
 
 
-// Converts an atomic Val and appends it to the list
+// Converts an atomic Val and appends it to the list.  For subnet types,
-static bool val_to_maskedval(Val* v, maskedvalue_list* append_to)
+// if the prefix_vector param isn't null, appending to that is preferred
+// over appending to the masked val list.
+static bool val_to_maskedval(Val* v, maskedvalue_list* append_to,
+                             vector<IPPrefix>* prefix_vector)
 	{
 	MaskedValue* mval = new MaskedValue;
 
@@ -1071,29 +1189,37 @@ static bool val_to_maskedval(Val* v, maskedvalue_list* append_to)
 
 		case TYPE_SUBNET:
 			{
-			const uint32* n;
+			if ( prefix_vector )
-			uint32 m[4];
-			v->AsSubNet().Prefix().GetBytes(&n);
-			v->AsSubNetVal()->Mask().CopyIPv6(m);
-
-			for ( unsigned int i = 0; i < 4; ++i )
-				m[i] = ntohl(m[i]);
-
-			bool is_v4_mask = m[0] == 0xffffffff &&
-						m[1] == m[0] && m[2] == m[0];
-
-			if ( v->AsSubNet().Prefix().GetFamily() == IPv4 &&
-			     is_v4_mask )
 				{
-				mval->val = ntohl(*n);
+				prefix_vector->push_back(v->AsSubNet());
-				mval->mask = m[3];
+				delete mval;
+				return true;
 				}
-
 			else
 				{
-				rules_error("IPv6 subnets not supported");
+				const uint32* n;
-				mval->val = 0;
+				uint32 m[4];
-				mval->mask = 0;
+				v->AsSubNet().Prefix().GetBytes(&n);
+				v->AsSubNetVal()->Mask().CopyIPv6(m);
+
+				for ( unsigned int i = 0; i < 4; ++i )
+					m[i] = ntohl(m[i]);
+
+				bool is_v4_mask = m[0] == 0xffffffff &&
+				                          m[1] == m[0] && m[2] == m[0];
+
+
+				if ( v->AsSubNet().Prefix().GetFamily() == IPv4 && is_v4_mask )
+					{
+					mval->val = ntohl(*n);
+					mval->mask = m[3];
+					}
+				else
+					{
+					rules_error("IPv6 subnets not supported");
+					mval->val = 0;
+					mval->mask = 0;
+					}
 				}
 			}
 			break;
@@ -1108,7 +1234,8 @@ static bool val_to_maskedval(Val* v, maskedvalue_list* append_to)
 	return true;
 	}
 
-void id_to_maskedvallist(const char* id, maskedvalue_list* append_to)
+void id_to_maskedvallist(const char* id, maskedvalue_list* append_to,
+                         vector<IPPrefix>* prefix_vector)
 	{
 	Val* v = get_bro_val(id);
 	if ( ! v )
@@ -1118,7 +1245,7 @@ void id_to_maskedvallist(const char* id, maskedvalue_list* append_to)
 		{
 		val_list* vals = v->AsTableVal()->ConvertToPureList()->Vals();
 		loop_over_list(*vals, i )
-			if ( ! val_to_maskedval((*vals)[i], append_to) )
+			if ( ! val_to_maskedval((*vals)[i], append_to, prefix_vector) )
 			{
 				delete_vals(vals);
 				return;
@@ -1128,7 +1255,7 @@ void id_to_maskedvallist(const char* id, maskedvalue_list* append_to)
 		}
 
 	else
-		val_to_maskedval(v, append_to);
+		val_to_maskedval(v, append_to, prefix_vector);
 	}
 
 char* id_to_str(const char* id)

src/RuleMatcher.h

@@ -2,7 +2,9 @@
 #define sigs_h
 
 #include <limits.h>
+#include <vector>
 
+#include "IPAddr.h"
 #include "BroString.h"
 #include "List.h"
 #include "RE.h"
@@ -59,17 +61,19 @@ declare(PList, BroString);
 typedef PList(BroString) bstr_list;
 
 // Get values from Bro's script-level variables.
-extern void id_to_maskedvallist(const char* id, maskedvalue_list* append_to);
+extern void id_to_maskedvallist(const char* id, maskedvalue_list* append_to,
+                                vector<IPPrefix>* prefix_vector = 0);
 extern char* id_to_str(const char* id);
 extern uint32 id_to_uint(const char* id);
 
 class RuleHdrTest {
 public:
 	enum Comp { LE, GE, LT, GT, EQ, NE };
-	enum Prot { NOPROT, IP, ICMP, TCP, UDP };
+	enum Prot { NOPROT, IP, IPv6, ICMP, ICMPv6, TCP, UDP, NEXT, IPSrc, IPDst };
 
 	RuleHdrTest(Prot arg_prot, uint32 arg_offset, uint32 arg_size,
 			Comp arg_comp, maskedvalue_list* arg_vals);
+	RuleHdrTest(Prot arg_prot, Comp arg_comp, vector<IPPrefix> arg_v);
 	~RuleHdrTest();
 
 	void PrintDebug();
@@ -86,6 +90,7 @@ private:
 	Prot prot;
 	Comp comp;
 	maskedvalue_list* vals;
+	vector<IPPrefix> prefix_vals; // for use with IPSrc/IPDst comparisons
 	uint32 offset;
 	uint32 size;
 

src/SerialTypes.h

@@ -165,6 +165,7 @@ SERIAL_STMT(EVENT_BODY_LIST, 16)
 SERIAL_STMT(INIT_STMT, 17)
 SERIAL_STMT(NULL_STMT, 18)
 SERIAL_STMT(WHEN_STMT, 19)
+SERIAL_STMT(HOOK_STMT, 20)
 
 #define SERIAL_TYPE(name, val) SERIAL_CONST(name, val, BRO_TYPE)
 SERIAL_TYPE(BRO_TYPE, 1)

src/Serializer.cc

@@ -389,23 +389,35 @@ bool Serializer::UnserializeCall(UnserialInfo* info)
 		{
 		if ( info->print )
 			fprintf(info->print, "%s [%.06f] %s(%s)\n",
-				functype->IsEvent() ? "Event" : "Function call",
+				functype->FlavorString().c_str(),
 				time, name, types ? d.Description() : "<ignored>");
 
-		if ( functype->IsEvent() )
+		switch ( functype->Flavor() ) {
+
+		case FUNC_FLAVOR_EVENT:
 			{
 			EventHandler* handler = event_registry->Lookup(name);
 			assert(handler);
+
 			if ( ! info->ignore_callbacks )
 				GotEvent(name, time, handler, args);
+
+			break;
 			}
-		else
+
+		case FUNC_FLAVOR_FUNCTION:
+		case FUNC_FLAVOR_HOOK:
 			if ( ! info->ignore_callbacks )
 				GotFunctionCall(name, time, id->ID_Val()->AsFunc(), args);
+			break;
+
+		default:
+			reporter->InternalError("invalid function flavor");
+			break;
+		}
 
 		if ( info->ignore_callbacks )
 			delete_vals(args);
-
 		}
 	else
 		delete_vals(args);

src/Serializer.h

@@ -125,7 +125,7 @@ protected:
 
 	// This will be increased whenever there is an incompatible change
 	// in the data format.
-	static const uint32 DATA_FORMAT_VERSION = 22;
+	static const uint32 DATA_FORMAT_VERSION = 23;
 
 	ChunkedIO* io;
 

src/Stmt.cc

@@ -23,7 +23,7 @@ const char* stmt_name(BroStmtTag t)
 		"print", "event", "expr", "if", "when", "switch",
 		"for", "next", "break", "return", "add", "delete",
 		"list", "bodylist",
-		"<init>",
+		"<init>", "hook",
 		"null",
 	};
 
@@ -933,6 +933,52 @@ bool EventStmt::DoUnserialize(UnserialInfo* info)
 	return event_expr != 0;
 	}
 
+HookStmt::HookStmt(CallExpr* arg_e) : ExprStmt(STMT_HOOK, arg_e)
+	{
+	call_expr = arg_e;
+	}
+
+Val* HookStmt::Exec(Frame* f, stmt_flow_type& flow) const
+	{
+	RegisterAccess();
+
+	Val* ret = call_expr->Eval(f);
+	Unref(ret);
+
+	flow = FLOW_NEXT;
+
+	return 0;
+	}
+
+TraversalCode HookStmt::Traverse(TraversalCallback* cb) const
+	{
+	TraversalCode tc = cb->PreStmt(this);
+	HANDLE_TC_STMT_PRE(tc);
+
+	// call expr is stored in base class's "e" field.
+	tc = e->Traverse(cb);
+	HANDLE_TC_STMT_PRE(tc);
+
+	tc = cb->PostStmt(this);
+	HANDLE_TC_STMT_POST(tc);
+	}
+
+IMPLEMENT_SERIAL(HookStmt, SER_HOOK_STMT);
+
+bool HookStmt::DoSerialize(SerialInfo* info) const
+	{
+	DO_SERIALIZE(SER_HOOK_STMT, ExprStmt);
+	return call_expr->Serialize(info);
+	}
+
+bool HookStmt::DoUnserialize(UnserialInfo* info)
+	{
+	DO_UNSERIALIZE(ExprStmt);
+
+	call_expr = (CallExpr*) Expr::Unserialize(info, EXPR_CALL);
+	return call_expr != 0;
+	}
+
 ForStmt::ForStmt(id_list* arg_loop_vars, Expr* loop_expr)
 : ExprStmt(STMT_FOR, loop_expr)
 	{
@@ -1944,6 +1990,7 @@ int same_stmt(const Stmt* s1, const Stmt* s2)
 	case STMT_RETURN:
 	case STMT_EXPR:
 	case STMT_EVENT:
+	case STMT_HOOK:
 		{
 		const ExprStmt* e1 = (const ExprStmt*) s1;
 		const ExprStmt* e2 = (const ExprStmt*) s2;

src/Stmt.h

@@ -286,6 +286,24 @@ protected:
 	EventExpr* event_expr;
 };
 
+class HookStmt : public ExprStmt {
+public:
+	HookStmt(CallExpr* e);
+
+	Val* Exec(Frame* f, stmt_flow_type& flow) const;
+
+	TraversalCode Traverse(TraversalCallback* cb) const;
+
+protected:
+	friend class Stmt;
+
+	HookStmt() { call_expr = 0; }
+
+	DECLARE_SERIAL(HookStmt);
+
+	CallExpr* call_expr;
+};
+
 class ForStmt : public ExprStmt {
 public:
 	ForStmt(id_list* loop_vars, Expr* loop_expr);

src/StmtEnums.h

@@ -15,7 +15,7 @@ typedef enum {
 	STMT_RETURN,
 	STMT_ADD, STMT_DELETE,
 	STMT_LIST, STMT_EVENT_BODY_LIST,
-	STMT_INIT,
+	STMT_INIT, STMT_HOOK,
 	STMT_NULL
 #define NUM_STMTS (int(STMT_NULL) + 1)
 } BroStmtTag;

src/Type.cc

@@ -651,12 +651,12 @@ bool SetType::DoUnserialize(UnserialInfo* info)
 	return true;
 	}
 
-FuncType::FuncType(RecordType* arg_args, BroType* arg_yield, int arg_is_event)
+FuncType::FuncType(RecordType* arg_args, BroType* arg_yield, function_flavor arg_flavor)
 : BroType(TYPE_FUNC)
 	{
 	args = arg_args;
 	yield = arg_yield;
-	is_event = arg_is_event;
+	flavor = arg_flavor;
 
 	arg_types = new TypeList();
 
@@ -664,6 +664,25 @@ FuncType::FuncType(RecordType* arg_args, BroType* arg_yield, int arg_is_event)
 		arg_types->Append(args->FieldType(i)->Ref());
 	}
 
+string FuncType::FlavorString() const
+	{
+	switch ( flavor ) {
+
+	case FUNC_FLAVOR_FUNCTION:
+		return "function";
+
+	case FUNC_FLAVOR_EVENT:
+		return "event";
+
+	case FUNC_FLAVOR_HOOK:
+		return "hook";
+
+	default:
+		reporter->InternalError("Invalid function flavor");
+		return "invalid_func_flavor";
+	}
+	}
+
 FuncType::~FuncType()
 	{
 	Unref(arg_types);
@@ -698,7 +717,7 @@ void FuncType::Describe(ODesc* d) const
 	{
 	if ( d->IsReadable() )
 		{
-		d->Add(is_event ? "event" : "function");
+		d->Add(FlavorString());
 		d->Add("(");
 		args->DescribeFields(d);
 		d->Add(")");
@@ -712,7 +731,7 @@ void FuncType::Describe(ODesc* d) const
 	else
 		{
 		d->Add(int(Tag()));
-		d->Add(is_event);
+		d->Add(flavor);
 		d->Add(yield != 0);
 		args->DescribeFields(d);
 		if ( yield )
@@ -723,7 +742,7 @@ void FuncType::Describe(ODesc* d) const
 void FuncType::DescribeReST(ODesc* d) const
 	{
 	d->Add(":bro:type:`");
-	d->Add(is_event ? "event" : "function");
+	d->Add(FlavorString());
 	d->Add("`");
 	d->Add(" (");
 	args->DescribeFieldsReST(d, true);
@@ -755,9 +774,30 @@ bool FuncType::DoSerialize(SerialInfo* info) const
 
 	SERIALIZE_OPTIONAL(yield);
 
+	int ser_flavor = 0;
+
+	switch ( flavor ) {
+
+	case FUNC_FLAVOR_FUNCTION:
+		ser_flavor = 0;
+		break;
+
+	case FUNC_FLAVOR_EVENT:
+		ser_flavor = 1;
+		break;
+
+	case FUNC_FLAVOR_HOOK:
+		ser_flavor = 2;
+		break;
+
+	default:
+		reporter->InternalError("Invalid function flavor serialization");
+		break;
+	}
+
 	return args->Serialize(info) &&
 		arg_types->Serialize(info) &&
-		SERIALIZE(is_event);
+		SERIALIZE(ser_flavor);
 	}
 
 bool FuncType::DoUnserialize(UnserialInfo* info)
@@ -774,7 +814,27 @@ bool FuncType::DoUnserialize(UnserialInfo* info)
 	if ( ! arg_types )
 		return false;
 
-	return UNSERIALIZE(&is_event);
+	int ser_flavor = 0;
+
+	if ( ! UNSERIALIZE(&ser_flavor) )
+		return false;
+
+	switch ( ser_flavor ) {
+	case 0:
+		flavor = FUNC_FLAVOR_FUNCTION;
+		break;
+	case 1:
+		flavor = FUNC_FLAVOR_EVENT;
+		break;
+	case 2:
+		flavor = FUNC_FLAVOR_HOOK;
+		break;
+	default:
+		reporter->InternalError("Invalid function flavor unserialization");
+		break;
+	}
+
+	return true;
 	}
 
 TypeDecl::TypeDecl(BroType* t, const char* i, attr_list* arg_attrs, bool in_record)
@@ -1202,9 +1262,10 @@ bool FileType::DoUnserialize(UnserialInfo* info)
 	return yield != 0;
 	}
 
-EnumType::EnumType()
+EnumType::EnumType(const string& arg_name)
 : BroType(TYPE_ENUM)
 	{
+	name = arg_name;
 	counter = 0;
 	}
 
@@ -1327,6 +1388,13 @@ const char* EnumType::Lookup(bro_int_t value)
 	return 0;
 	}
 
+void EnumType::DescribeReST(ODesc* d) const
+	{
+	d->Add(":bro:type:`");
+	d->Add(name.c_str());
+	d->Add("`");
+	}
+
 void CommentedEnumType::DescribeReST(ODesc* d) const
 	{
 	// create temporary, reverse name map so that enums can be documented
@@ -1595,7 +1663,7 @@ int same_type(const BroType* t1, const BroType* t2, int is_init)
 		const FuncType* ft1 = (const FuncType*) t1;
 		const FuncType* ft2 = (const FuncType*) t2;
 
-		if ( ft1->IsEvent() != ft2->IsEvent() )
+		if ( ft1->Flavor() != ft2->Flavor() )
 			return 0;
 
 		if ( t1->YieldType() || t2->YieldType() )
@@ -1882,7 +1950,7 @@ BroType* merge_types(const BroType* t1, const BroType* t2)
 		BroType* yield = t1->YieldType() ?
 			merge_types(t1->YieldType(), t2->YieldType()) : 0;
 
-		return new FuncType(args->AsRecordType(), yield, ft1->IsEvent());
+		return new FuncType(args->AsRecordType(), yield, ft1->Flavor());
 		}
 
 	case TYPE_RECORD:

src/Type.h

@@ -35,7 +35,11 @@ typedef enum {
 #define NUM_TYPES (int(TYPE_ERROR) + 1)
 } TypeTag;
 
-typedef enum { FUNC_FLAVOR_FUNCTION, FUNC_FLAVOR_EVENT } function_flavor;
+typedef enum {
+	FUNC_FLAVOR_FUNCTION,
+	FUNC_FLAVOR_EVENT,
+	FUNC_FLAVOR_HOOK
+} function_flavor;
 
 typedef enum {
 	TYPE_INTERNAL_VOID,
@@ -350,18 +354,19 @@ protected:
 
 class FuncType : public BroType {
 public:
-	FuncType(RecordType* args, BroType* yield, int is_event);
+	FuncType(RecordType* args, BroType* yield, function_flavor f);
 
 	~FuncType();
 
 	RecordType* Args() const	{ return args; }
 	BroType* YieldType();
 	void SetYieldType(BroType* arg_yield)	{ yield = arg_yield; }
-	int IsEvent() const		{ return is_event; }
+	function_flavor Flavor() const { return flavor; }
+	string FlavorString() const;
 
-	// Used to convert a function type to an event type.
+	// Used to convert a function type to an event or hook type.
-	void ClearYieldType()
+	void ClearYieldType(function_flavor arg_flav)
-		{ Unref(yield); yield = 0; is_event = 1; }
+		{ Unref(yield); yield = 0; flavor = arg_flav; }
 
 	int MatchesIndex(ListExpr*& index) const;
 	int CheckArgs(const type_list* args) const;
@@ -374,14 +379,13 @@ public:
 	void DescribeReST(ODesc* d) const;
 
 protected:
-	FuncType()	{ args = 0; arg_types = 0; yield = 0; return_value = 0; }
+	FuncType()	{ args = 0; arg_types = 0; yield = 0; flavor = FUNC_FLAVOR_FUNCTION; }
 	DECLARE_SERIAL(FuncType)
 
 	RecordType* args;
 	TypeList* arg_types;
 	BroType* yield;
-	int is_event;
+	function_flavor flavor;
-	ID* return_value;
 };
 
 class TypeType : public BroType {
@@ -497,7 +501,7 @@ protected:
 
 class EnumType : public BroType {
 public:
-	EnumType();
+	EnumType(const string& arg_name);
 	~EnumType();
 
 	// The value of this name is next internal counter value, starting
@@ -513,7 +517,12 @@ public:
 	bro_int_t Lookup(const string& module_name, const char* name);
 	const char* Lookup(bro_int_t value); // Returns 0 if not found
 
+	string Name() const { return name; }
+
+	void DescribeReST(ODesc* d) const;
+
 protected:
+	EnumType() { counter = 0; }
 	DECLARE_SERIAL(EnumType)
 
 	virtual void AddNameInternal(const string& module_name,
@@ -529,11 +538,14 @@ protected:
 	// as a flag to prevent mixing of auto-increment and explicit
 	// enumerator specifications.
 	bro_int_t counter;
+
+	// The name of the enum type is stored for documentation purposes.
+	string name;
 };
 
 class CommentedEnumType: public EnumType {
 public:
-	CommentedEnumType() {}
+	CommentedEnumType(const string& arg_name) : EnumType(arg_name) {}
 	~CommentedEnumType();
 
 	void DescribeReST(ODesc* d) const;

src/Var.cc

@@ -171,7 +171,9 @@ static void make_var(ID* id, BroType* t, init_class c, Expr* init,
 
 	id->UpdateValAttrs();
 
-	if ( t && t->Tag() == TYPE_FUNC && t->AsFuncType()->IsEvent() )
+	if ( t && t->Tag() == TYPE_FUNC &&
+	     (t->AsFuncType()->Flavor() == FUNC_FLAVOR_EVENT ||
+	      t->AsFuncType()->Flavor() == FUNC_FLAVOR_HOOK) )
 		{
 		// For events, add a function value (without any body) here so that
 		// we can later access the ID even if no implementations have been
@@ -258,7 +260,7 @@ void add_type(ID* id, BroType* t, attr_list* attr, int /* is_event */)
 		case TYPE_FUNC:
 			tnew = new FuncType(t->AsFuncType()->Args(),
 			                    t->AsFuncType()->YieldType(),
-			                    t->AsFuncType()->IsEvent());
+			                    t->AsFuncType()->Flavor());
 			break;
 		default:
 			SerializationFormat* form = new BinarySerializationFormat();
@@ -292,14 +294,14 @@ void add_type(ID* id, BroType* t, attr_list* attr, int /* is_event */)
 void begin_func(ID* id, const char* module_name, function_flavor flavor,
 		int is_redef, FuncType* t)
 	{
-	if ( flavor == FUNC_FLAVOR_EVENT )
+	if ( flavor == FUNC_FLAVOR_EVENT || flavor == FUNC_FLAVOR_HOOK )
 		{
 		const BroType* yt = t->YieldType();
 
 		if ( yt && yt->Tag() != TYPE_VOID )
-			id->Error("event cannot yield a value", t);
+			id->Error("event/hook cannot yield a value", t);
 
-		t->ClearYieldType();
+		t->ClearYieldType(flavor);
 		}
 
 	if ( id->Type() )
@@ -313,21 +315,29 @@ void begin_func(ID* id, const char* module_name, function_flavor flavor,
 
 	if ( id->HasVal() )
 		{
-		int id_is_event = id->ID_Val()->AsFunc()->IsEvent();
+		function_flavor id_flavor = id->ID_Val()->AsFunc()->Flavor();
 
-		if ( id_is_event != (flavor == FUNC_FLAVOR_EVENT) )
+		if ( id_flavor != flavor )
-			id->Error("inconsistency between event and function", t);
+			id->Error("inconsistent function flavor", t);
-		if ( id_is_event )
+
-			{
+		switch ( id_flavor ) {
+
+		case FUNC_FLAVOR_EVENT:
+		case FUNC_FLAVOR_HOOK:
 			if ( is_redef )
 				// Clear out value so it will be replaced.
 				id->SetVal(0);
-			}
+			break;
-		else
+
-			{
+		case FUNC_FLAVOR_FUNCTION:
 			if ( ! id->IsRedefinable() )
 				id->Error("already defined");
-			}
+			break;
+
+		default:
+			reporter->InternalError("invalid function flavor");
+			break;
+		}
 		}
 	else
 		id->SetType(t);

src/bro.bif

@@ -11,6 +11,7 @@
 #include <cmath>
 #include <sys/stat.h>
 #include <cstdio>
+#include <time.h>
 
 #include "digest.h"
 #include "Reporter.h"
@@ -2700,6 +2701,27 @@ function to_port%(s: string%): port
     return new PortVal(port, TRANSPORT_UNKNOWN);
 	%}
 
+## Converts a string of bytes (in network byte order) to a :bro:type:`double`.
+##
+## s: A string of bytes containing the binary representation of a double value.
+##
+## Returns: The double value contained in *s*, or 0 if the conversion
+##          failed.
+##
+function bytestring_to_double%(s: string%): double
+	%{
+	if ( s->Len() != sizeof(double) )
+		{
+		builtin_error("bad conversion to double");
+		return new Val(0.0, TYPE_DOUBLE);
+		}
+
+	// See #908 for a discussion of portability.
+	double d;
+	memcpy(&d, s->Bytes(), sizeof(double));
+	return new Val(ntohd(d), TYPE_DOUBLE);
+	%}
+
 ## Converts a reverse pointer name to an address. For example,
 ## ``1.0.168.192.in-addr.arpa`` to ``192.168.0.1``.
 ##
@@ -3285,6 +3307,31 @@ function strftime%(fmt: string, d: time%) : string
 	return new StringVal(buffer);
 	%}
 
+
+## Parse a textual representation of a date/time value into a ``time`` type value.
+##
+## fmt: The format string used to parse the following *d* argument. See ``man strftime``
+##      for the syntax.
+##
+## d: The string representing the time.
+##
+## Returns: The time value calculated from parsing *d* with *fmt*.
+function strptime%(fmt: string, d: string%) : time
+	%{
+	const time_t timeval = time_t(NULL);
+	struct tm t = *localtime(&timeval);
+
+	if ( strptime(d->CheckString(), fmt->CheckString(), &t) == NULL )
+		{
+		reporter->Warning("strptime conversion failed: fmt:%s d:%s", fmt->CheckString(), d->CheckString());
+		return new Val(0.0, TYPE_TIME);
+		}
+
+	double ret = mktime(&t);
+	return new Val(ret, TYPE_TIME);
+	%}
+
+
 # ===========================================================================
 #
 #                           Network Type Processing
@@ -3744,6 +3791,35 @@ function lookup_addr%(host: addr%) : string
 	return 0;
 	%}
 
+## Issues an asynchronous TEXT DNS lookup and delays the function result.
+## This function can therefore only be called inside a ``when`` condition,
+## e.g., ``when ( local h = lookup_hostname_txt("www.bro-ids.org") ) { f(h); }``.
+##
+## host: The hostname to lookup.
+##
+## Returns: The DNS TXT record associated with *host*.
+##
+## .. bro:see:: lookup_hostname
+function lookup_hostname_txt%(host: string%) : string
+	%{
+	// FIXME: Is should be easy to adapt the function to synchronous
+	// lookups if we're reading a trace.
+	Trigger* trigger = frame->GetTrigger();
+
+	if ( ! trigger)
+		{
+		builtin_error("lookup_hostname_txt() can only be called inside a when-condition");
+		return new StringVal("<error>");
+		}
+
+	frame->SetDelayed();
+	trigger->Hold();
+
+	dns_mgr->AsyncLookupNameText(host->CheckString(),
+				 new LookupHostCallback(trigger, frame->GetCall(), true));
+	return 0;
+	%}
+
 ## Issues an asynchronous DNS lookup and delays the function result.
 ## This function can therefore only be called inside a ``when`` condition,
 ## e.g., ``when ( local h = lookup_hostname("www.bro-ids.org") ) { f(h); }``.

src/event.bif

@@ -6560,6 +6560,301 @@ event netflow_v5_header%(h: nf_v5_header%);
 ## .. bro:see:: netflow_v5_record
 event netflow_v5_record%(r: nf_v5_record%);
 
+## Generated for any modbus message regardless if the particular function
+## is further supported or not.
+##
+## c: The connection.
+##
+## headers: The headers for the modbus function.
+##
+## is_orig: True if the event is raised for the originator side.
+event modbus_message%(c: connection, headers: ModbusHeaders, is_orig: bool%);
+
+## Generated for any modbus exception message.
+##
+## c: The connection.
+##
+## headers: The headers for the modbus function.
+##
+## code: The exception code.
+event modbus_exception%(c: connection, headers: ModbusHeaders, code: count%);
+
+## Generated for a Modbus read coils request.
+##
+## c: The connection.
+##
+## headers: The headers for the modbus function.
+##
+## start_address: The memory address where of the first coil to be read.
+##
+## quantity: The number of coils to be read.
+event modbus_read_coils_request%(c: connection, headers: ModbusHeaders, start_address: count, quantity: count%);
+
+## Generated for a Modbus read coils response.
+##
+## c: The connection.
+##
+## headers: The headers for the modbus function.
+##
+## coils: The coil values returned from the device.
+event modbus_read_coils_response%(c: connection, headers: ModbusHeaders, coils: ModbusCoils%);
+
+## Generated for a Modbus read discrete inputs request.
+##
+## c: The connection.
+##
+## headers: The headers for the modbus function.
+##
+## start_address: The memory address of the first coil to be read.
+##
+## quantity: The number of coils to be read.
+event modbus_read_discrete_inputs_request%(c: connection, headers: ModbusHeaders, start_address: count, quantity: count%);
+
+## Generated for a Modbus read discrete inputs response.
+##
+## c: The connection.
+##
+## headers: The headers for the modbus function.
+##
+## coils: The coil values returned from the device.
+event modbus_read_discrete_inputs_response%(c: connection, headers: ModbusHeaders, coils: ModbusCoils%);
+
+## Generated for a Modbus read holding registers request.
+##
+## c: The connection.
+##
+## headers: The headers for the modbus function.
+##
+## start_address: The memory address of the first register to be read.
+##
+## quantity: The number of registers to be read.
+event modbus_read_holding_registers_request%(c: connection, headers: ModbusHeaders, start_address: count, quantity: count%);
+
+## Generated for a Modbus read holding registers response.
+##
+## c: The connection.
+##
+## headers: The headers for the modbus function.
+##
+## registers: The register values returned from the device.
+event modbus_read_holding_registers_response%(c: connection, headers: ModbusHeaders, registers: ModbusRegisters%);
+
+## Generated for a Modbus read input registers request.
+##
+## c: The connection.
+##
+## headers: The headers for the modbus function.
+##
+## start_address: The memory address of the first register to be read.
+##
+## quantity: The number of registers to be read.
+event modbus_read_input_registers_request%(c: connection, headers: ModbusHeaders, start_address: count, quantity: count%);
+
+## Generated for a Modbus read input registers response.
+##
+## c: The connection.
+##
+## headers: The headers for the modbus function.
+##
+## registers: The register values returned from the device.
+event modbus_read_input_registers_response%(c: connection, headers: ModbusHeaders, registers: ModbusRegisters%);
+
+## Generated for a Modbus write single coil request.
+##
+## c: The connection.
+##
+## headers: The headers for the modbus function.
+##
+## address: The memory address of the coil to be written.
+##
+## value: The value to be written to the coil.
+event modbus_write_single_coil_request%(c: connection, headers: ModbusHeaders, address: count, value: bool%);
+
+## Generated for a Modbus write single coil response.
+##
+## c: The connection.
+##
+## headers: The headers for the modbus function.
+##
+## address: The memory address of the coil that was written.
+##
+## value: The value that was written to the coil.
+event modbus_write_single_coil_response%(c: connection, headers: ModbusHeaders, address: count, value: bool%);
+
+## Generated for a Modbus write single register request.
+##
+## c: The connection.
+##
+## headers: The headers for the modbus function.
+##
+## address: The memory address of the register to be written.
+##
+## value: The value to be written to the register.
+event modbus_write_single_register_request%(c: connection, headers: ModbusHeaders, address: count, value: count%);
+
+## Generated for a Modbus write single register response.
+##
+## c: The connection.
+##
+## headers: The headers for the modbus function.
+##
+## address: The memory address of the register that was written.
+##
+## value: The value that was written to the register.
+event modbus_write_single_register_response%(c: connection, headers: ModbusHeaders, address: count, value: count%);
+
+## Generated for a Modbus write multiple coils request.
+##
+## c: The connection.
+##
+## headers: The headers for the modbus function.
+##
+## start_address: The memory address of the first coil to be written.
+##
+## value: The values to be written to the coils.
+event modbus_write_multiple_coils_request%(c: connection, headers: ModbusHeaders, start_address: count, coils: ModbusCoils%);
+
+## Generated for a Modbus write multiple coils response.
+##
+## c: The connection.
+##
+## headers: The headers for the modbus function.
+##
+## start_address: The memory address of the first coil that was written.
+##
+## quantity: The quantity of coils that were written.
+event modbus_write_multiple_coils_response%(c: connection, headers: ModbusHeaders, start_address: count, quantity: count%);
+
+## Generated for a Modbus write multiple registers request.
+##
+## c: The connection.
+##
+## headers: The headers for the modbus function.
+##
+## start_address: The memory address of the first register to be written.
+##
+## registers: The values to be written to the registers.
+event modbus_write_multiple_registers_request%(c: connection, headers: ModbusHeaders, start_address: count, registers: ModbusRegisters%);
+
+## Generated for a Modbus write multiple registers response.
+##
+## c: The connection.
+##
+## headers: The headers for the modbus function.
+##
+## start_address: The memory address of the first register that was written.
+##
+## quantity: The quantity of registers that were written.
+event modbus_write_multiple_registers_response%(c: connection, headers: ModbusHeaders, start_address: count, quantity: count%);
+
+## Generated for a Modbus read file record request.
+##
+## c: The connection.
+##
+## headers: The headers for the modbus function.
+##
+## .. note: This event is incomplete.  The information from the data structure is not
+##          yet passed through to the event.
+event modbus_read_file_record_request%(c: connection, headers: ModbusHeaders%);
+
+## Generated for a Modbus read file record response.
+##
+## c: The connection.
+##
+## headers: The headers for the modbus function.
+##
+## .. note: This event is incomplete.  The information from the data structure is not
+##          yet passed through to the event.
+event modbus_read_file_record_response%(c: connection, headers: ModbusHeaders%);
+
+## Generated for a Modbus write file record request.
+##
+## c: The connection.
+##
+## headers: The headers for the modbus function.
+##
+## .. note: This event is incomplete.  The information from the data structure is not
+##          yet passed through to the event.
+event modbus_write_file_record_request%(c: connection, headers: ModbusHeaders%);
+
+## Generated for a Modbus write file record response.
+##
+## c: The connection.
+##
+## headers: The headers for the modbus function.
+##
+## .. note: This event is incomplete.  The information from the data structure is not
+##          yet passed through to the event.
+event modbus_write_file_record_response%(c: connection, headers: ModbusHeaders%);
+
+## Generated for a Modbus mask write register request.
+##
+## c: The connection.
+##
+## headers: The headers for the modbus function.
+##
+## address: The memory address of the register where the masks should be applied.
+##
+## and_mask: The value of the logical AND mask to apply to the register.
+##
+## or_mask: The value of the logical OR mask to apply to the register.
+event modbus_mask_write_register_request%(c: connection, headers: ModbusHeaders, address: count, and_mask: count, or_mask: count%);
+
+## Generated for a Modbus mask write register request.
+##
+## c: The connection.
+##
+## headers: The headers for the modbus function.
+##
+## address: The memory address of the register where the masks were applied.
+##
+## and_mask: The value of the logical AND mask applied register.
+##
+## or_mask: The value of the logical OR mask applied to the register.
+event modbus_mask_write_register_response%(c: connection, headers: ModbusHeaders, address: count, and_mask: count, or_mask: count%);
+
+## Generated for a Modbus read/write multiple registers request.
+##
+## c: The connection.
+##
+## headers: The headers for the modbus function.
+##
+## read_start_address: The memory address of the first register to be read.
+##
+## read_quantity: The number of registers to read.
+##
+## write_start_address: The memory address of the first register to be written.
+##
+## write_registers: The values to be written to the registers.
+event modbus_read_write_multiple_registers_request%(c: connection, headers: ModbusHeaders, read_start_address: count, read_quantity: count, write_start_address: count, write_registers: ModbusRegisters%);
+
+## Generated for a Modbus read/write multiple registers response.
+##
+## c: The connection.
+##
+## headers: The headers for the modbus function.
+##
+## written_registers: The register values read from the registers specified in the request.
+event modbus_read_write_multiple_registers_response%(c: connection, headers: ModbusHeaders, written_registers: ModbusRegisters%);
+
+## Generated for a Modbus read FIFO queue request.
+##
+## c: The connection.
+##
+## headers: The headers for the modbus function.
+##
+## start_address: The address of the FIFO queue to read.
+event modbus_read_fifo_queue_request%(c: connection, headers: ModbusHeaders, start_address: count%);
+
+## Generated for a Modbus read FIFO queue response.
+##
+## c: The connection.
+##
+## headers: The headers for the modbus function.
+##
+## fifos: The register values read from the FIFO queue on the device.
+event modbus_read_fifo_queue_response%(c: connection, headers: ModbusHeaders, fifos: ModbusRegisters%);
+
 ## Raised for informational messages reported via Bro's reporter framework. Such
 ## messages may be generated internally by the event engine and also by other
 ## scripts calling :bro:id:`Reporter::info`.

src/input/Manager.cc

@@ -393,7 +393,7 @@ bool Manager::CreateEventStream(RecordVal* fval)
 
 	bool allow_file_func = false;
 
-	if ( ! etype->IsEvent() )
+	if ( etype->Flavor() != FUNC_FLAVOR_EVENT )
 		{
 		reporter->Error("stream event is a function, not an event");
 		return false;
@@ -566,7 +566,7 @@ bool Manager::CreateTableStream(RecordVal* fval)
 		{
 		FuncType* etype = event->FType()->AsFuncType();
 
-		if ( ! etype->IsEvent() )
+		if ( etype->Flavor() != FUNC_FLAVOR_EVENT )
 			{
 			reporter->Error("stream event is a function, not an event");
 			return false;

src/logging/Manager.cc

@@ -327,7 +327,7 @@ bool Manager::CreateStream(EnumVal* id, RecordVal* sval)
 		// Make sure the event is prototyped as expected.
 		FuncType* etype = event->FType()->AsFuncType();
 
-		if ( ! etype->IsEvent() )
+		if ( etype->Flavor() != FUNC_FLAVOR_EVENT )
 			{
 			reporter->Error("stream event is a function, not an event");
 			return false;

src/logging/writers/Ascii.cc

@@ -19,6 +19,7 @@ Ascii::Ascii(WriterFrontend* frontend) : WriterBackend(frontend)
 	{
 	fd = 0;
 	ascii_done = false;
+	only_single_header_row = false;
 
 	output_to_stdout = BifConst::LogAscii::output_to_stdout;
 	include_meta = BifConst::LogAscii::include_meta;
@@ -80,7 +81,7 @@ void Ascii::CloseFile(double t)
 	if ( ! fd )
 		return;
 
-	if ( include_meta )
+	if ( include_meta && ! only_single_header_row )
 		WriteHeaderField("close", Timestamp(0));
 
 	safe_close(fd);
@@ -108,29 +109,29 @@ bool Ascii::DoInit(const WriterInfo& info, int num_fields, const Field* const *
 		return false;
 		}
 
+	for ( WriterInfo::config_map::const_iterator i = info.config.begin(); i != info.config.end(); i++ )
+		{
+		if ( strcmp(i->first, "only_single_header_row") == 0 )
+			{
+			if ( strcmp(i->second, "T") == 0 )
+				only_single_header_row = true;
+
+			else if ( strcmp(i->second, "F") == 0 )
+				only_single_header_row = false;
+
+			else
+				{
+				Error("invalid value for 'only_single_header_row', must be boolean (T/F)");
+				return false;
+				}
+			}
+		}
+
 	if ( include_meta )
 		{
 		string names;
 		string types;
 
-		string str = string(meta_prefix, meta_prefix_len)
-			+ "separator " // Always use space as separator here.
-			+ get_escaped_string(string(separator, separator_len), false)
-			+ "\n";
-
-		if ( ! safe_write(fd, str.c_str(), str.length()) )
-			goto write_error;
-
-		if ( ! (WriteHeaderField("set_separator", get_escaped_string(
-		            string(set_separator, set_separator_len), false)) &&
-		        WriteHeaderField("empty_field", get_escaped_string(
-		            string(empty_field, empty_field_len), false)) &&
-		        WriteHeaderField("unset_field", get_escaped_string(
-		            string(unset_field, unset_field_len), false)) &&
-		        WriteHeaderField("path", get_escaped_string(path, false)) &&
-		        WriteHeaderField("open", Timestamp(0))) )
-			goto write_error;
-
 		for ( int i = 0; i < num_fields; ++i )
 			{
 			if ( i > 0 )
@@ -143,6 +144,34 @@ bool Ascii::DoInit(const WriterInfo& info, int num_fields, const Field* const *
 			types += fields[i]->TypeName().c_str();
 			}
 
+		if ( only_single_header_row )
+			{
+			// A single CSV-style line is all we need.
+			string str = names + "\n";
+			if ( ! safe_write(fd, str.c_str(), str.length()) )
+				goto write_error;
+
+			return true;
+			}
+
+		string str = string(meta_prefix, meta_prefix_len)
+			+ "separator " // Always use space as separator here.
+			+ get_escaped_string(string(separator, separator_len), false)
+			+ "\n";
+
+		if ( ! safe_write(fd, str.c_str(), str.length()) )
+			goto write_error;
+
+		if ( ! (WriteHeaderField("set_separator", get_escaped_string(
+			    string(set_separator, set_separator_len), false)) &&
+			WriteHeaderField("empty_field", get_escaped_string(
+			    string(empty_field, empty_field_len), false)) &&
+			WriteHeaderField("unset_field", get_escaped_string(
+			    string(unset_field, unset_field_len), false)) &&
+			WriteHeaderField("path", get_escaped_string(path, false)) &&
+			WriteHeaderField("open", Timestamp(0))) )
+			goto write_error;
+
 		if ( ! (WriteHeaderField("fields", names)
 			&& WriteHeaderField("types", types)) )
 			goto write_error;

src/logging/writers/Ascii.h

@@ -45,6 +45,7 @@ private:
 	// Options set from the script-level.
 	bool output_to_stdout;
 	bool include_meta;
+	bool only_single_header_row;
 
 	char* separator;
 	int separator_len;

src/modbus-analyzer.pac

@@ -0,0 +1,610 @@
+#
+# The development of Bro's Modbus analyzer has been made possible thanks to
+# the support of the Ministry of Security and Justice of the Kingdom of the
+# Netherlands within the projects of Hermes, Castor and Midas.
+#
+# Useful references: http://www.modbus.org/docs/Modbus_Application_Protocol_V1_1b.pdf
+#                    http://www.simplymodbus.ca/faq.htm
+#
+
+%header{
+	VectorVal* bytestring_to_coils(bytestring coils, uint quantity);
+	RecordVal* HeaderToBro(ModbusTCP_TransportHeader *header);
+	%}
+
+%code{
+	VectorVal* bytestring_to_coils(bytestring coils, uint quantity)
+		{
+		VectorVal* modbus_coils = new VectorVal(BifType::Vector::ModbusCoils);
+
+		return modbus_coils;
+		}
+
+	RecordVal* HeaderToBro(ModbusTCP_TransportHeader *header)
+		{
+		RecordVal* modbus_header = new RecordVal(BifType::Record::ModbusHeaders);
+		modbus_header->Assign(0, new Val(header->tid(), TYPE_COUNT));
+		modbus_header->Assign(1, new Val(header->pid(), TYPE_COUNT));
+		modbus_header->Assign(2, new Val(header->uid(), TYPE_COUNT));
+		modbus_header->Assign(3, new Val(header->fc(), TYPE_COUNT));
+		return modbus_header;
+		}
+
+	%}
+
+refine flow ModbusTCP_Flow += {
+
+	function deliver_message(header: ModbusTCP_TransportHeader): bool
+		%{
+		if ( ::modbus_message )
+			{
+			BifEvent::generate_modbus_message(connection()->bro_analyzer(),
+			                                  connection()->bro_analyzer()->Conn(),
+			                                  HeaderToBro(header),
+			                                  is_orig());
+			}
+
+		return true;
+		%}
+
+	# EXCEPTION
+	function deliver_Exception(header: ModbusTCP_TransportHeader, message: Exception): bool
+		%{
+		if ( ::modbus_exception )
+			{
+			BifEvent::generate_modbus_exception(connection()->bro_analyzer(),
+			                                    connection()->bro_analyzer()->Conn(),
+			                                    HeaderToBro(header),
+			                                    ${message.code});
+			}
+
+		return true;
+		%}
+
+	# REQUEST FC=1
+	function deliver_ReadCoilsRequest(header: ModbusTCP_TransportHeader, message: ReadCoilsRequest): bool
+		%{
+		if ( ::modbus_read_coils_request )
+			{
+			BifEvent::generate_modbus_read_coils_request(connection()->bro_analyzer(),
+			                                             connection()->bro_analyzer()->Conn(),
+			                                             HeaderToBro(header),
+			                                             ${message.start_address},
+			                                             ${message.quantity});
+			}
+
+		return true;
+		%}
+
+	# RESPONSE FC=1
+	function deliver_ReadCoilsResponse(header: ModbusTCP_TransportHeader, message: ReadCoilsResponse): bool
+		%{
+		if ( ::modbus_read_coils_response )
+			{
+			BifEvent::generate_modbus_read_coils_response(connection()->bro_analyzer(),
+			                                              connection()->bro_analyzer()->Conn(),
+			                                              HeaderToBro(header),
+			                                              bytestring_to_coils(${message.bits}, ${message.bits}.length()*8));
+			}
+		return true;
+		%}
+
+	# REQUEST FC=2
+	function deliver_ReadDiscreteInputsRequest(header: ModbusTCP_TransportHeader, message: ReadDiscreteInputsRequest): bool
+		%{
+		if ( ::modbus_read_discrete_inputs_request )
+			{
+			BifEvent::generate_modbus_read_discrete_inputs_request(connection()->bro_analyzer(),
+			                                                       connection()->bro_analyzer()->Conn(),
+			                                                       HeaderToBro(header),
+			                                                       ${message.start_address}, ${message.quantity});
+			}
+
+		return true;
+		%}
+
+	# RESPONSE FC=2
+	function deliver_ReadDiscreteInputsResponse(header: ModbusTCP_TransportHeader, message: ReadDiscreteInputsResponse): bool
+		%{
+		if ( ::modbus_read_discrete_inputs_response )
+			{
+			BifEvent::generate_modbus_read_discrete_inputs_response(connection()->bro_analyzer(),
+			                                                        connection()->bro_analyzer()->Conn(),
+			                                                        HeaderToBro(header),
+			                                                        bytestring_to_coils(${message.bits}, ${message.bits}.length()*8));
+			}
+
+		return true;
+		%}
+
+
+	# REQUEST FC=3
+	function deliver_ReadHoldingRegistersRequest(header: ModbusTCP_TransportHeader, message: ReadHoldingRegistersRequest): bool
+		%{
+		if ( ::modbus_read_holding_registers_request )
+			{
+			BifEvent::generate_modbus_read_holding_registers_request(connection()->bro_analyzer(),
+			                                                         connection()->bro_analyzer()->Conn(),
+			                                                         HeaderToBro(header),
+			                                                         ${message.start_address}, ${message.quantity});
+			}
+
+		return true;
+		%}
+
+	# RESPONSE FC=3
+	function deliver_ReadHoldingRegistersResponse(header: ModbusTCP_TransportHeader, message: ReadHoldingRegistersResponse): bool
+		%{
+		if ( ${message.byte_count} % 2 != 0 )
+			{
+			connection()->bro_analyzer()->ProtocolViolation(
+			    fmt("invalid value for modbus read holding register response byte count %d", ${message.byte_count}));
+			return false;
+			}
+
+		if ( ::modbus_read_holding_registers_response )
+			{
+
+			VectorVal* t = new VectorVal(BifType::Vector::ModbusRegisters);
+			for ( unsigned int i=0; i < ${message.registers}->size(); ++i )
+				{
+				Val* r = new Val(${message.registers[i]}, TYPE_COUNT);
+				t->Assign(i, r, 0, OP_ASSIGN);
+				}
+
+			BifEvent::generate_modbus_read_holding_registers_response(connection()->bro_analyzer(),
+			                                                          connection()->bro_analyzer()->Conn(),
+			                                                          HeaderToBro(header),
+			                                                          t);
+			}
+
+		return true;
+		%}
+
+
+	# REQUEST FC=4
+	function deliver_ReadInputRegistersRequest(header: ModbusTCP_TransportHeader, message: ReadInputRegistersRequest): bool
+		%{
+		if ( ::modbus_read_input_registers_request )
+			{
+			BifEvent::generate_modbus_read_input_registers_request(connection()->bro_analyzer(),
+			                                                       connection()->bro_analyzer()->Conn(),
+			                                                       HeaderToBro(header),
+			                                                       ${message.start_address}, ${message.quantity});
+			}
+
+		return true;
+		%}
+
+	# RESPONSE FC=4
+	function deliver_ReadInputRegistersResponse(header: ModbusTCP_TransportHeader, message: ReadInputRegistersResponse): bool
+		%{
+		if ( ${message.byte_count} % 2 != 0 )
+			{
+			connection()->bro_analyzer()->ProtocolViolation(
+			    fmt("invalid value for modbus read input register response byte count %d", ${message.byte_count}));
+			return false;
+			}
+
+		if ( ::modbus_read_input_registers_response )
+			{
+			VectorVal* t = new VectorVal(BifType::Vector::ModbusRegisters);
+			for ( unsigned int i=0; i < (${message.registers})->size(); ++i )
+				{
+				Val* r = new Val(${message.registers[i]}, TYPE_COUNT);
+				t->Assign(i, r, 0, OP_ASSIGN);
+				}
+
+			BifEvent::generate_modbus_read_input_registers_response(connection()->bro_analyzer(),
+			                                                        connection()->bro_analyzer()->Conn(),
+			                                                        HeaderToBro(header),
+			                                                        t);
+			}
+
+		return true;
+		%}
+
+
+	# REQUEST FC=5
+	function deliver_WriteSingleCoilRequest(header: ModbusTCP_TransportHeader, message: WriteSingleCoilRequest): bool
+		%{
+		if ( ::modbus_write_single_coil_request )
+			{
+			int val;
+			if ( ${message.value} == 0x0000 )
+				val = 0;
+			else if ( ${message.value} == 0xFF00 )
+				val = 1;
+			else
+				{
+				connection()->bro_analyzer()->ProtocolViolation(fmt("invalid value for modbus write single coil request %d",
+				                                                    ${message.value}));
+				return false;
+				}
+
+			BifEvent::generate_modbus_write_single_coil_request(connection()->bro_analyzer(),
+			                                                    connection()->bro_analyzer()->Conn(),
+			                                                    HeaderToBro(header),
+			                                                    ${message.address},
+			                                                    val);
+			}
+
+		return true;
+		%}
+
+	# RESPONSE FC=5
+	function deliver_WriteSingleCoilResponse(header: ModbusTCP_TransportHeader, message: WriteSingleCoilResponse): bool
+		%{
+		if ( ::modbus_write_single_coil_response )
+			{
+			int val;
+			if ( ${message.value} == 0x0000 )
+				val = 0;
+			else if ( ${message.value} == 0xFF00 )
+				val = 1;
+			else
+				{
+				connection()->bro_analyzer()->ProtocolViolation(fmt("invalid value for modbus write single coil response %d",
+				                                                    ${message.value}));
+				return false;
+				}
+
+			BifEvent::generate_modbus_write_single_coil_response(connection()->bro_analyzer(),
+			                                                     connection()->bro_analyzer()->Conn(),
+			                                                     HeaderToBro(header),
+			                                                     ${message.address},
+			                                                     val);
+			}
+
+		return true;
+		%}
+
+
+	# REQUEST FC=6
+	function deliver_WriteSingleRegisterRequest(header: ModbusTCP_TransportHeader, message: WriteSingleRegisterRequest): bool
+		%{
+		if ( ::modbus_write_single_register_request )
+			{
+			BifEvent::generate_modbus_write_single_register_request(connection()->bro_analyzer(),
+			                                                        connection()->bro_analyzer()->Conn(),
+			                                                        HeaderToBro(header),
+			                                                        ${message.address}, ${message.value});
+			}
+
+		return true;
+		%}
+
+	# RESPONSE FC=6
+	function deliver_WriteSingleRegisterResponse(header: ModbusTCP_TransportHeader, message: WriteSingleRegisterResponse): bool
+		%{
+		if ( ::modbus_write_single_register_response )
+			{
+			BifEvent::generate_modbus_write_single_register_response(connection()->bro_analyzer(),
+			                                                         connection()->bro_analyzer()->Conn(),
+			                                                         HeaderToBro(header),
+			                                                         ${message.address}, ${message.value});
+			}
+
+		return true;
+		%}
+
+
+	# REQUEST FC=15
+	function deliver_WriteMultipleCoilsRequest(header: ModbusTCP_TransportHeader, message: WriteMultipleCoilsRequest): bool
+		%{
+		if ( ::modbus_write_multiple_coils_request )
+			{
+			BifEvent::generate_modbus_write_multiple_coils_request(connection()->bro_analyzer(),
+			                                                       connection()->bro_analyzer()->Conn(),
+			                                                       HeaderToBro(header),
+			                                                       ${message.start_address},
+			                                                       bytestring_to_coils(${message.coils}, ${message.quantity}));
+			}
+
+		return true;
+		%}
+
+	# RESPONSE FC=15
+	function deliver_WriteMultipleCoilsResponse(header: ModbusTCP_TransportHeader, message: WriteMultipleCoilsResponse): bool
+		%{
+		if ( ::modbus_write_multiple_coils_response )
+			{
+			BifEvent::generate_modbus_write_multiple_coils_response(connection()->bro_analyzer(),
+			                                                        connection()->bro_analyzer()->Conn(),
+			                                                        HeaderToBro(header),
+			                                                        ${message.start_address}, ${message.quantity});
+			}
+
+		return true;
+		%}
+
+
+	# REQUEST FC=16
+	function deliver_WriteMultipleRegistersRequest(header: ModbusTCP_TransportHeader, message: WriteMultipleRegistersRequest): bool
+		%{
+		if ( ${message.byte_count} % 2 != 0 )
+			{
+			connection()->bro_analyzer()->ProtocolViolation(
+			    fmt("invalid value for modbus write multiple registers request byte count %d", ${message.byte_count}));
+			return false;
+			}
+
+		if ( ::modbus_write_multiple_registers_request )
+			{
+			VectorVal * t = new VectorVal(BifType::Vector::ModbusRegisters);
+			for ( unsigned int i = 0; i < (${message.registers}->size()); ++i )
+				{
+				Val* r = new Val(${message.registers[i]}, TYPE_COUNT);
+				t->Assign(i, r, 0, OP_ASSIGN);
+				}
+
+				BifEvent::generate_modbus_write_multiple_registers_request(connection()->bro_analyzer(),
+				                                                           connection()->bro_analyzer()->Conn(),
+				                                                           HeaderToBro(header),
+				                                                           ${message.start_address}, t);
+			}
+
+		return true;
+		%}
+
+	# RESPONSE FC=16
+	function deliver_WriteMultipleRegistersResponse(header: ModbusTCP_TransportHeader, message: WriteMultipleRegistersResponse): bool
+		%{
+		if ( ::modbus_write_multiple_registers_response )
+			{
+			BifEvent::generate_modbus_write_multiple_registers_response(connection()->bro_analyzer(),
+			                                                            connection()->bro_analyzer()->Conn(),
+			                                                            HeaderToBro(header),
+			                                                            ${message.start_address}, ${message.quantity});
+			}
+
+		return true;
+		%}
+
+	# REQUEST FC=20
+	function deliver_ReadFileRecordRequest(header: ModbusTCP_TransportHeader, message: ReadFileRecordRequest): bool
+		%{
+		if ( ::modbus_read_file_record_request )
+			{
+			//TODO: this need to be a vector of some Reference Request record type
+			//VectorVal *t = new VectorVal(new VectorType(base_type(TYPE_COUNT)));
+			//for ( unsigned int i = 0; i < (${message.references}->size()); ++i )
+			//	{
+			//	Val* r = new Val((${message.references[i].ref_type}), TYPE_COUNT);
+			//	t->Assign(i, r, 0, OP_ASSIGN);
+			//
+			//	Val* k = new Val((${message.references[i].file_num}), TYPE_COUNT);
+			//	t->Assign(i, k, 0, OP_ASSIGN);
+			//
+			//	Val* l = new Val((${message.references[i].record_num}), TYPE_COUNT);
+			//	t->Assign(i, l, 0, OP_ASSIGN);
+			//	}
+
+			BifEvent::generate_modbus_read_file_record_request(connection()->bro_analyzer(),
+			                                                   connection()->bro_analyzer()->Conn(),
+			                                                   HeaderToBro(header));
+			}
+
+		return true;
+		%}
+
+	# RESPONSE FC=20
+	function deliver_ReadFileRecordResponse(header: ModbusTCP_TransportHeader, message: ReadFileRecordResponse): bool
+		%{
+		if ( ::modbus_read_file_record_response )
+			{
+			//VectorVal *t = new VectorVal(new VectorType(base_type(TYPE_COUNT)));
+			//for ( unsigned int i = 0; i < ${message.references}->size(); ++i )
+			//	{
+			//	//TODO: work the reference type in here somewhere
+			//	Val* r = new Val(${message.references[i].record_data}), TYPE_COUNT);
+			//	t->Assign(i, r, 0, OP_ASSIGN);
+			//	}
+
+			BifEvent::generate_modbus_read_file_record_response(connection()->bro_analyzer(),
+			                                                    connection()->bro_analyzer()->Conn(),
+			                                                    HeaderToBro(header));
+			}
+
+		return true;
+		%}
+
+	# REQUEST FC=21
+	function deliver_WriteFileRecordRequest(header: ModbusTCP_TransportHeader, message: WriteFileRecordRequest): bool
+		%{
+		if ( ::modbus_write_file_record_request )
+			{
+			//VectorVal* t = new VectorVal(new VectorType(base_type(TYPE_COUNT)));
+			//for ( unsigned int i = 0; i < (${message.references}->size()); ++i )
+			//	{
+			//	Val* r = new Val((${message.references[i].ref_type}), TYPE_COUNT);
+			//	t->Assign(i, r, 0, OP_ASSIGN);
+			//
+			//	Val* k = new Val((${message.references[i].file_num}), TYPE_COUNT);
+			//	t->Assign(i, k, 0, OP_ASSIGN);
+			//
+			//	Val* n = new Val((${message.references[i].record_num}), TYPE_COUNT);
+			//	t->Assign(i, n, 0, OP_ASSIGN);
+			//
+			//	for ( unsigned int j = 0; j < (${message.references[i].register_value}->size()); ++j )
+			//		{
+			//		k = new Val((${message.references[i].register_value[j]}), TYPE_COUNT);
+			//		t->Assign(i, k, 0, OP_ASSIGN);
+			//		}
+			//	}
+
+			BifEvent::generate_modbus_write_file_record_request(connection()->bro_analyzer(),
+			                                                    connection()->bro_analyzer()->Conn(),
+			                                                    HeaderToBro(header));
+			}
+
+		return true;
+		%}
+
+
+	# RESPONSE FC=21
+	function deliver_WriteFileRecordResponse(header: ModbusTCP_TransportHeader, message: WriteFileRecordResponse): bool
+		%{
+		if ( ::modbus_write_file_record_response )
+			{
+			//VectorVal* t = new VectorVal(new VectorType(base_type(TYPE_COUNT)));
+			//for ( unsigned int i = 0; i < (${messages.references}->size()); ++i )
+			//	{
+			//	Val* r = new Val((${message.references[i].ref_type}), TYPE_COUNT);
+			//	t->Assign(i, r, 0, OP_ASSIGN);
+			//
+			//	Val* f = new Val((${message.references[i].file_num}), TYPE_COUNT);
+			//	t->Assign(i, f, 0, OP_ASSIGN);
+			//
+			//	Val* rn = new Val((${message.references[i].record_num}), TYPE_COUNT);
+			//	t->Assign(i, rn, 0, OP_ASSIGN);
+			//
+			//	for ( unsigned int j = 0; j<(${message.references[i].register_value}->size()); ++j )
+			//		{
+			//		Val* k = new Val((${message.references[i].register_value[j]}), TYPE_COUNT);
+			//		t->Assign(i, k, 0, OP_ASSIGN);
+			//		}
+
+			BifEvent::generate_modbus_write_file_record_response(connection()->bro_analyzer(),
+			                                                     connection()->bro_analyzer()->Conn(),
+			                                                     HeaderToBro(header));
+			}
+
+		return true;
+		%}
+
+	# REQUEST FC=22
+	function deliver_MaskWriteRegisterRequest(header: ModbusTCP_TransportHeader, message: MaskWriteRegisterRequest): bool
+		%{
+		if ( ::modbus_mask_write_register_request )
+			{
+			BifEvent::generate_modbus_mask_write_register_request(connection()->bro_analyzer(),
+			                                                      connection()->bro_analyzer()->Conn(),
+			                                                      HeaderToBro(header),
+			                                                      ${message.address},
+			                                                      ${message.and_mask}, ${message.or_mask});
+			}
+
+		return true;
+		%}
+
+	# RESPONSE FC=22
+	function deliver_MaskWriteRegisterResponse(header: ModbusTCP_TransportHeader, message: MaskWriteRegisterResponse): bool
+		%{
+		if ( ::modbus_mask_write_register_response )
+			{
+			BifEvent::generate_modbus_mask_write_register_response(connection()->bro_analyzer(),
+			                                                       connection()->bro_analyzer()->Conn(),
+			                                                       HeaderToBro(header),
+			                                                       ${message.address},
+			                                                       ${message.and_mask}, ${message.or_mask});
+			}
+
+		return true;
+		%}
+
+	# REQUEST FC=23
+	function deliver_ReadWriteMultipleRegistersRequest(header: ModbusTCP_TransportHeader, message: ReadWriteMultipleRegistersRequest): bool
+		%{
+		if ( ${message.write_byte_count} % 2 != 0 )
+			{
+			connection()->bro_analyzer()->ProtocolViolation(
+			    fmt("invalid value for modbus read write multiple registers request write byte count %d", ${message.write_byte_count}));
+			return false;
+			}
+
+		if ( ::modbus_read_write_multiple_registers_request )
+			{
+			VectorVal* t = new VectorVal(BifType::Vector::ModbusRegisters);
+			for ( unsigned int i = 0; i < ${message.write_register_values}->size(); ++i )
+				{
+				Val* r = new Val(${message.write_register_values[i]}, TYPE_COUNT);
+				t->Assign(i, r, 0, OP_ASSIGN);
+				}
+
+			BifEvent::generate_modbus_read_write_multiple_registers_request(connection()->bro_analyzer(),
+			                                                                connection()->bro_analyzer()->Conn(),
+			                                                                HeaderToBro(header),
+			                                                                ${message.read_start_address},
+			                                                                ${message.read_quantity},
+			                                                                ${message.write_start_address},
+			                                                                t);
+			}
+
+		return true;
+		%}
+
+	# RESPONSE FC=23
+	function deliver_ReadWriteMultipleRegistersResponse(header: ModbusTCP_TransportHeader, message: ReadWriteMultipleRegistersResponse): bool
+		%{
+		if ( ${message.byte_count} % 2 != 0 )
+			{
+			connection()->bro_analyzer()->ProtocolViolation(
+			    fmt("invalid value for modbus read write multiple registers response byte count %d", ${message.byte_count}));
+			return false;
+			}
+
+		if ( ::modbus_read_write_multiple_registers_response )
+			{
+			VectorVal* t = new VectorVal(BifType::Vector::ModbusRegisters);
+			for ( unsigned int i = 0; i < ${message.registers}->size(); ++i )
+				{
+				Val* r = new Val(${message.registers[i]}, TYPE_COUNT);
+				t->Assign(i, r, 0, OP_ASSIGN);
+				}
+
+			BifEvent::generate_modbus_read_write_multiple_registers_response(connection()->bro_analyzer(),
+			                                                                 connection()->bro_analyzer()->Conn(),
+			                                                                 HeaderToBro(header),
+			                                                                 t);
+			}
+
+		return true;
+		%}
+
+	# REQUEST FC=24
+	function deliver_ReadFIFOQueueRequest(header: ModbusTCP_TransportHeader, message: ReadFIFOQueueRequest): bool
+		%{
+		if ( ::modbus_read_fifo_queue_request )
+			{
+			BifEvent::generate_modbus_read_fifo_queue_request(connection()->bro_analyzer(),
+			                                                  connection()->bro_analyzer()->Conn(),
+			                                                  HeaderToBro(header),
+			                                                  ${message.start_address});
+			}
+
+		return true;
+		%}
+
+
+	# RESPONSE FC=24
+	function deliver_ReadFIFOQueueResponse(header: ModbusTCP_TransportHeader, message: ReadFIFOQueueResponse): bool
+		%{
+		if ( ${message.byte_count} % 2 != 0 )
+			{
+			connection()->bro_analyzer()->ProtocolViolation(
+			    fmt("invalid value for modbus read FIFO queue response byte count %d", ${message.byte_count}));
+			return false;
+			}
+
+		if ( ::modbus_read_fifo_queue_response )
+			{
+			VectorVal* t = new VectorVal(new VectorType(base_type(TYPE_COUNT)));
+			for ( unsigned int i = 0; i < (${message.register_data})->size(); ++i )
+				{
+				Val* r = new Val(${message.register_data[i]}, TYPE_COUNT);
+				t->Assign(i, r, 0, OP_ASSIGN);
+				}
+
+			BifEvent::generate_modbus_read_fifo_queue_response(connection()->bro_analyzer(),
+			                                                   connection()->bro_analyzer()->Conn(),
+			                                                   HeaderToBro(header),
+			                                                   t);
+			}
+
+		return true;
+		%}
+};
+
+
+

src/modbus-protocol.pac

@@ -0,0 +1,394 @@
+#
+# The development of Bro's Modbus analyzer has been made possible thanks to
+# the support of the Ministry of Security and Justice of the Kingdom of the
+# Netherlands within the projects of Hermes, Castor and Midas.
+#
+# Useful references: http://www.modbus.org/docs/Modbus_Application_Protocol_V1_1b.pdf
+#                    http://www.simplymodbus.ca/faq.htm
+
+enum function_codes {
+	# Standard functions
+	READ_COILS                    = 0x01,
+	READ_DISCRETE_INPUTS          = 0x02,
+	READ_HOLDING_REGISTERS        = 0x03,
+	READ_INPUT_REGISTERS          = 0x04,
+	WRITE_SINGLE_COIL             = 0x05,
+	WRITE_SINGLE_REGISTER         = 0x06,
+	# READ_EXCEPTION_STATUS         = 0x07,
+	# DIAGNOSTICS                   = 0x08,
+	# GET_COMM_EVENT_COUNTER        = 0x0B,
+	# GET_COMM_EVENT_LOG            = 0x0C,
+	WRITE_MULTIPLE_COILS          = 0x0F,
+	WRITE_MULTIPLE_REGISTERS      = 0x10,
+	# REPORT_SLAVE_ID               = 0x11,
+	READ_FILE_RECORD              = 0x14,
+	WRITE_FILE_RECORD             = 0x15,
+	MASK_WRITE_REGISTER           = 0x16,
+	READ_WRITE_MULTIPLE_REGISTERS = 0x17,
+	READ_FIFO_QUEUE               = 0x18,
+	ENCAP_INTERFACE_TRANSPORT     = 0x2B,
+
+	# Machine/vendor/network specific functions
+	PROGRAM_484                   = 0x09,
+	POLL_484                      = 0x0A,
+	PROGRAM_584_984               = 0x0D,
+	POLL_584_984                  = 0x0E,
+	PROGRAM_884_U84               = 0x12,
+	RESET_COMM_LINK_884_U84       = 0x13,
+	PROGRAM_CONCEPT               = 0x28,
+	FIRMWARE_REPLACEMENT          = 0x7D,
+	PROGRAM_584_984_2             = 0x7E,
+	REPORT_LOCAL_ADDRESS          = 0x7F,
+
+	# Exceptions (not really function codes but they are used similarly)
+	READ_COILS_EXCEPTION                    = 0x81,
+	READ_DISCRETE_INPUTS_EXCEPTION          = 0x82,
+	READ_HOLDING_REGISTERS_EXCEPTION        = 0x83,
+	READ_INPUT_REGISTERS_EXCEPTION          = 0x84,
+	WRITE_SINGLE_COIL_EXCEPTION             = 0x85,
+	WRITE_SINGLE_REGISTER_EXCEPTION         = 0x86,
+	READ_EXCEPTION_STATUS_EXCEPTION         = 0x87,
+	WRITE_MULTIPLE_COILS_EXCEPTION          = 0x8F,
+	WRITE_MULTIPLE_REGISTERS_EXCEPTION      = 0x90,
+	READ_FILE_RECORD_EXCEPTION              = 0x94,
+	WRITE_FILE_RECORD_EXCEPTION             = 0x95,
+	MASK_WRITE_REGISTER_EXCEPTION           = 0x96,
+	READ_WRITE_MULTIPLE_REGISTERS_EXCEPTION = 0x97,
+	READ_FIFO_QUEUE_EXCEPTION               = 0x98,
+};
+
+# Main Modbus/TCP PDU
+type ModbusTCP_PDU(is_orig: bool) = record {
+	header: ModbusTCP_TransportHeader;
+	body: case is_orig of {
+		true  -> request:  ModbusTCP_Request(header);
+		false -> response: ModbusTCP_Response(header);
+	};
+} &length=header.len+6, &byteorder=bigendian;
+
+type ModbusTCP_TransportHeader = record {
+	tid: uint16; # Transaction identifier
+	pid: uint16; # Protocol identifier
+	len: uint16; # Length of everyting after this field
+	uid: uint8;  # Unit identifier (previously 'slave address')
+	fc:  uint8;  # MODBUS function code (see function_codes enum)
+} &byteorder=bigendian, &let {
+	deliver: bool = $context.flow.deliver_message(this);
+};
+
+type ModbusTCP_Request(header: ModbusTCP_TransportHeader) = case header.fc of {
+	READ_COILS                    -> readCoils:                  ReadCoilsRequest(header);
+	READ_DISCRETE_INPUTS          -> readDiscreteInputs:         ReadDiscreteInputsRequest(header);
+	READ_HOLDING_REGISTERS        -> readHoldingRegisters:       ReadHoldingRegistersRequest(header);
+	READ_INPUT_REGISTERS          -> readInputRegisters:         ReadInputRegistersRequest(header);
+	WRITE_SINGLE_COIL             -> writeSingleCoil:            WriteSingleCoilRequest(header);
+	WRITE_SINGLE_REGISTER         -> writeSingleRegister:        WriteSingleRegisterRequest(header);
+	#READ_EXCEPTION_STATUS         -> readExceptionStatus:        ReadExceptionStatusRequest(header);
+	#DIAGNOSTICS                   -> diagnostics:                DiagnosticsRequest(header);
+	#GET_COMM_EVENT_COUNTER        -> getCommEventCounter:        GetCommEventCounterRequest(header);
+	#GET_COMM_EVENT_LOG            -> getCommEventLog:            GetCommEventLogRequest(header);
+	WRITE_MULTIPLE_COILS          -> writeMultipleCoils:         WriteMultipleCoilsRequest(header);
+	WRITE_MULTIPLE_REGISTERS      -> writeMultRegisters:         WriteMultipleRegistersRequest(header);
+	#REPORT_SLAVE_ID
+	READ_FILE_RECORD              -> readFileRecord:             ReadFileRecordRequest(header);
+	WRITE_FILE_RECORD             -> writeFileRecord:            WriteFileRecordRequest(header);
+	MASK_WRITE_REGISTER           -> maskWriteRegister:          MaskWriteRegisterRequest(header);
+	READ_WRITE_MULTIPLE_REGISTERS -> readWriteMultipleRegisters: ReadWriteMultipleRegistersRequest(header);
+	READ_FIFO_QUEUE               -> readFIFOQueue:              ReadFIFOQueueRequest(header);
+	#ENCAP_INTERFACE_TRANSPORT
+
+	# All the rest
+	default                       -> unknown:                    bytestring &restofdata;
+};
+
+# Responses
+#
+type ModbusTCP_Response(header: ModbusTCP_TransportHeader) = case header.fc of {
+	READ_COILS                          -> readCoils:                       ReadCoilsResponse(header);
+	READ_DISCRETE_INPUTS                -> readDiscreteInputs:              ReadDiscreteInputsResponse(header);
+	READ_HOLDING_REGISTERS              -> readHoldingRegisters:            ReadHoldingRegistersResponse(header);
+	READ_INPUT_REGISTERS                -> readInputRegisters:              ReadInputRegistersResponse(header);
+	WRITE_SINGLE_COIL                   -> writeSingleCoil:                 WriteSingleCoilResponse(header);
+	WRITE_SINGLE_REGISTER               -> writeSingleRegister:             WriteSingleRegisterResponse(header);
+	#READ_EXCEPTION_STATUS               -> readExceptionStatus:             ReadExceptionStatusResponse(header);
+	#DIAGNOSTICS                         -> diagnostics:                     DiagnosticsResponse(header);
+	#GET_COMM_EVENT_COUNTER              -> getCommEventCounter:             GetCommEventCounterResponse(header);
+	#GET_COMM_EVENT_LOG                  -> getCommEventLog:                 GetCommEventLogResponse(header);
+	WRITE_MULTIPLE_COILS                -> writeMultipleCoils:              WriteMultipleCoilsResponse(header);
+	WRITE_MULTIPLE_REGISTERS            -> writeMultRegisters:              WriteMultipleRegistersResponse(header);
+	#REPORT_SLAVE_ID
+	READ_FILE_RECORD                    -> readFileRecord:                  ReadFileRecordResponse(header);
+	WRITE_FILE_RECORD                   -> writeFileRecord:                 WriteFileRecordResponse(header);
+	MASK_WRITE_REGISTER                 -> maskWriteRegister:               MaskWriteRegisterResponse(header);
+	READ_WRITE_MULTIPLE_REGISTERS       -> readWriteMultipleRegisters:      ReadWriteMultipleRegistersResponse(header);
+	READ_FIFO_QUEUE                     -> readFIFOQueue:                   ReadFIFOQueueResponse(header);
+
+	# Exceptions
+	READ_HOLDING_REGISTERS_EXCEPTION        -> readHoldingRegistersException:   Exception(header);
+	WRITE_MULTIPLE_REGISTERS_EXCEPTION      -> writeMultRegistersException:     Exception(header);
+	READ_COILS_EXCEPTION                    -> readCoilsException:              Exception(header);
+	READ_DISCRETE_INPUTS_EXCEPTION          -> readDiscreteInputsException:     Exception(header);
+	READ_INPUT_REGISTERS_EXCEPTION          -> readInputRegistersException:     Exception(header);
+	WRITE_SINGLE_COIL_EXCEPTION             -> writeCoilException:              Exception(header);
+	WRITE_SINGLE_REGISTER_EXCEPTION         -> writeSingleRegisterException:    Exception(header);
+	READ_EXCEPTION_STATUS_EXCEPTION         -> readExceptionStatusException:    Exception(header);
+	WRITE_MULTIPLE_COILS_EXCEPTION          -> forceMultipleCoilsException:     Exception(header);
+	READ_FILE_RECORD_EXCEPTION              -> readGeneralReferenceException:   Exception(header);
+	WRITE_FILE_RECORD_EXCEPTION             -> writeGeneralReferenceException:  Exception(header);
+	MASK_WRITE_REGISTER_EXCEPTION           -> maskWriteRegisterException:      Exception(header);
+	READ_WRITE_MULTIPLE_REGISTERS_EXCEPTION -> readWriteRegistersException:     Exception(header);
+	READ_FIFO_QUEUE_EXCEPTION               -> readFIFOQueueException:          Exception(header);
+
+	# All the rest
+	default                                 -> unknown:                         bytestring &restofdata;
+};
+
+type Exception(header: ModbusTCP_TransportHeader) = record {
+	code: uint8;
+} &let {
+	deliver: bool = $context.flow.deliver_Exception(header, this);
+};
+
+# REQUEST FC=1
+type ReadCoilsRequest(header: ModbusTCP_TransportHeader) = record {
+	start_address:  uint16;
+	quantity:       uint16 &check(quantity <= 2000);
+} &let {
+	deliver: bool = $context.flow.deliver_ReadCoilsRequest(header, this);
+} &byteorder=bigendian;
+
+# RESPONSE FC=1
+type ReadCoilsResponse(header: ModbusTCP_TransportHeader) = record {
+	byte_count: uint8;
+	bits:       bytestring &length=byte_count;
+} &let {
+	deliver: bool = $context.flow.deliver_ReadCoilsResponse(header, this);
+} &byteorder=bigendian;
+
+# REQUEST FC=2
+type ReadDiscreteInputsRequest(header: ModbusTCP_TransportHeader) = record {
+	start_address: uint16;
+	quantity:      uint16 &check(quantity <= 2000);
+} &let {
+	deliver: bool = $context.flow.deliver_ReadDiscreteInputsRequest(header, this);
+} &byteorder=bigendian;
+
+# RESPONSE FC=2
+type ReadDiscreteInputsResponse(header: ModbusTCP_TransportHeader) = record {
+	byte_count: uint8;
+	bits:       bytestring &length=byte_count;
+} &let {
+	deliver: bool = $context.flow.deliver_ReadDiscreteInputsResponse(header, this);
+} &byteorder=bigendian;
+
+# REQUEST FC=3
+type ReadHoldingRegistersRequest(header: ModbusTCP_TransportHeader) = record {
+	start_address: uint16;
+	quantity:      uint16 &check(1 <= quantity && quantity <= 125);
+} &let {
+	deliver: bool = $context.flow.deliver_ReadHoldingRegistersRequest(header, this);
+} &byteorder=bigendian;
+
+# RESPONSE FC=3
+type ReadHoldingRegistersResponse(header: ModbusTCP_TransportHeader) = record {
+	byte_count: uint8;
+	registers:  uint16[byte_count/2] &length=byte_count;
+} &let {
+	deliver: bool = $context.flow.deliver_ReadHoldingRegistersResponse(header, this);
+} &byteorder=bigendian;
+
+# REQUEST FC=4
+type ReadInputRegistersRequest(header: ModbusTCP_TransportHeader) = record {
+	start_address: uint16;
+	quantity:      uint16 &check(1 <= quantity && quantity <= 125);
+} &let {
+	deliver: bool = $context.flow.deliver_ReadInputRegistersRequest(header, this);
+} &byteorder=bigendian;
+
+# RESPONSE FC=4
+type ReadInputRegistersResponse(header: ModbusTCP_TransportHeader) = record {
+	byte_count: uint8;
+	registers:  uint16[byte_count/2] &length=byte_count;
+} &let {
+	deliver: bool = $context.flow.deliver_ReadInputRegistersResponse(header, this);
+} &byteorder=bigendian;
+
+# REQUEST FC=5
+type WriteSingleCoilRequest(header: ModbusTCP_TransportHeader) = record {
+	address: uint16;
+	value:   uint16 &check(value == 0x0000 || value == 0xFF00);
+} &let {
+	deliver: bool = $context.flow.deliver_WriteSingleCoilRequest(header, this);
+} &byteorder=bigendian;
+
+# RESPONSE FC=5
+type WriteSingleCoilResponse(header: ModbusTCP_TransportHeader) = record {
+	address: uint16;
+	value:   uint16 &check(value == 0x0000 || value == 0xFF00);
+} &let {
+	deliver: bool = $context.flow.deliver_WriteSingleCoilResponse(header, this);
+} &byteorder=bigendian;
+
+# REQUEST FC=6
+type WriteSingleRegisterRequest(header: ModbusTCP_TransportHeader) = record {
+	address:  uint16;
+	value:    uint16;
+} &let {
+	deliver: bool = $context.flow.deliver_WriteSingleRegisterRequest(header, this);
+} &byteorder=bigendian;
+
+# RESPONSE FC=6
+type WriteSingleRegisterResponse(header: ModbusTCP_TransportHeader) = record {
+	address:   uint16;
+	value:     uint16;
+} &let {
+	deliver: bool = $context.flow.deliver_WriteSingleRegisterResponse(header, this);
+} &byteorder=bigendian;
+
+# REQUEST FC=15
+type WriteMultipleCoilsRequest(header: ModbusTCP_TransportHeader) = record {
+	start_address:    uint16;
+	quantity:         uint16     &check(quantity <= 0x07B0);
+	byte_count:       uint8      &check(byte_count == (quantity + 7)/8);
+	coils:            bytestring &length=byte_count;
+} &let {
+	deliver: bool = $context.flow.deliver_WriteMultipleCoilsRequest(header, this);
+} &byteorder=bigendian;
+
+# RESPONSE FC=15
+type WriteMultipleCoilsResponse(header: ModbusTCP_TransportHeader) = record {
+	start_address:   uint16;
+	quantity:        uint16 &check(quantity <= 0x07B0);
+} &let {
+	deliver: bool = $context.flow.deliver_WriteMultipleCoilsResponse(header, this);
+} &byteorder=bigendian;
+
+# REQUEST FC=16
+type WriteMultipleRegistersRequest(header: ModbusTCP_TransportHeader) = record {
+	start_address: uint16;
+	quantity:      uint16;
+	byte_count:    uint8;
+	# We specify registers buffer with quantity and byte_count so that the analyzer 
+	# will choke if something doesn't match right (correct devices should make it right).
+	registers:     uint16[quantity] &length=byte_count;
+} &let {
+	deliver: bool = $context.flow.deliver_WriteMultipleRegistersRequest(header, this);
+} &byteorder=bigendian;
+
+# RESPONSE FC=16
+type WriteMultipleRegistersResponse(header: ModbusTCP_TransportHeader) = record {
+	start_address: uint16;
+	quantity:      uint16;
+} &let {
+	deliver: bool = $context.flow.deliver_WriteMultipleRegistersResponse(header, this);
+} &byteorder=bigendian;
+
+# Support data structure for following message type.
+type FileRecordRequest = record {
+	ref_type:   uint8  &check(ref_type == 6);
+	file_num:   uint16 &check(file_num > 0);
+	record_num: uint16 &check(record_num <= 0x270F);
+	record_len: uint16;
+} &byteorder=bigendian;
+
+# REQUEST FC=20
+type ReadFileRecordRequest(header: ModbusTCP_TransportHeader) = record {
+	byte_count: uint8               &check(byte_count >= 0x07 && byte_count <= 0xF5);
+	references: FileRecordRequest[] &length=byte_count;
+} &let {
+	deliver: bool = $context.flow.deliver_ReadFileRecordRequest(header, this);
+} &byteorder=bigendian;
+
+# Support data structure for the following message type.
+type FileRecordResponse = record {
+	file_len:    uint8    &check(file_len >= 0x07 && file_len <= 0xF5);
+	ref_type:    uint8    &check(ref_type == 6);
+	record_data: uint16[file_len/2] &length=file_len;
+} &byteorder=bigendian;
+
+# RESPONSE FC=20
+type ReadFileRecordResponse(header: ModbusTCP_TransportHeader) = record {
+	byte_count: uint8 &check(byte_count >= 0x07 && byte_count <= 0xF5);
+	references: FileRecordResponse[] &length=byte_count;
+} &let {
+	deliver: bool = $context.flow.deliver_ReadFileRecordResponse(header, this);
+} &byteorder=bigendian;
+
+# Support data structure for the two following message types.
+type ReferenceWithData = record {
+	ref_type:       uint8;
+	file_num:       uint16;
+	record_num:     uint16;
+	word_count:     uint16;
+	register_value: uint16[word_count];
+} &byteorder=bigendian;
+
+# REQUEST FC=21
+type WriteFileRecordRequest(header: ModbusTCP_TransportHeader) = record {
+	byte_count: uint8;
+	references: ReferenceWithData[] &length=byte_count;
+} &let {
+	deliver: bool = $context.flow.deliver_WriteFileRecordRequest(header, this);
+} &byteorder=bigendian;
+
+# RESPONSE FC=21
+type WriteFileRecordResponse(header: ModbusTCP_TransportHeader) = record {
+	byte_count: uint8;
+	references: ReferenceWithData[] &length=byte_count;
+} &let {
+	deliver: bool = $context.flow.deliver_WriteFileRecordResponse(header, this);
+} &byteorder=bigendian;
+
+# REQUEST FC=22
+type MaskWriteRegisterRequest(header: ModbusTCP_TransportHeader) = record {
+	address:    uint16;
+	and_mask:   uint16;
+	or_mask:    uint16;
+} &let {
+	deliver: bool = $context.flow.deliver_MaskWriteRegisterRequest(header, this);
+} &byteorder=bigendian;
+
+# RESPONSE FC=22
+type MaskWriteRegisterResponse(header: ModbusTCP_TransportHeader) = record {
+	address:  uint16;
+	and_mask: uint16;
+	or_mask:  uint16;
+} &let {
+	deliver: bool = $context.flow.deliver_MaskWriteRegisterResponse(header, this);
+} &byteorder=bigendian;
+
+# REQUEST FC=23
+type ReadWriteMultipleRegistersRequest(header: ModbusTCP_TransportHeader) = record {
+	read_start_address:    uint16;
+	read_quantity:         uint16                 &check(read_quantity <= 125);
+	write_start_address:   uint16;
+	write_quantity:        uint16                 &check(write_quantity <= 100);
+	write_byte_count:      uint8;
+	write_register_values: uint16[write_quantity] &length=write_byte_count;
+} &let {
+	 deliver: bool = $context.flow.deliver_ReadWriteMultipleRegistersRequest(header, this);
+} &byteorder=bigendian;
+
+# RESPONSE FC=23
+type ReadWriteMultipleRegistersResponse(header: ModbusTCP_TransportHeader) = record {
+	byte_count:  uint8;
+	registers:   uint16[byte_count/2] &length=byte_count;
+} &let {
+	deliver: bool = $context.flow.deliver_ReadWriteMultipleRegistersResponse(header, this);
+} &byteorder=bigendian;
+
+# REQUEST FC=24
+type ReadFIFOQueueRequest(header: ModbusTCP_TransportHeader) = record {
+	start_address: uint16;
+} &let{
+	deliver: bool = $context.flow.deliver_ReadFIFOQueueRequest(header, this);
+} &byteorder=bigendian;
+
+# RESPONSE FC=24
+type ReadFIFOQueueResponse(header: ModbusTCP_TransportHeader) = record {
+	byte_count:    uint16             &check(byte_count <= 62);
+	fifo_count:    uint16             &check(fifo_count <= 31);
+	register_data: uint16[fifo_count] &length=fifo_count*2;
+} &let {
+	deliver: bool = $context.flow.deliver_ReadFIFOQueueResponse(header, this);
+} &byteorder=bigendian;

src/modbus.pac

@@ -0,0 +1,28 @@
+#
+# The development of Bro's Modbus analyzer has been made possible thanks to
+# the support of the Ministry of Security and Justice of the Kingdom of the
+# Netherlands within the projects of Hermes, Castor and Midas.
+#
+# Useful references: http://www.modbus.org/docs/Modbus_Application_Protocol_V1_1b.pdf
+#                    http://www.simplymodbus.ca/faq.htm
+
+%include binpac.pac
+%include bro.pac
+
+analyzer ModbusTCP withcontext {
+	connection: ModbusTCP_Conn;
+	flow:       ModbusTCP_Flow;
+};
+
+connection ModbusTCP_Conn(bro_analyzer: BroAnalyzer) {
+	upflow = ModbusTCP_Flow(true);
+	downflow = ModbusTCP_Flow(false);
+};
+
+%include modbus-protocol.pac
+
+flow ModbusTCP_Flow(is_orig: bool) {
+	flowunit = ModbusTCP_PDU(is_orig) withcontext (connection, this);
+}
+
+%include modbus-analyzer.pac

src/nb_dns.c

@@ -307,31 +307,32 @@ nb_dns_host_request(register struct nb_dns_info *nd, register const char *name,
     register void *cookie, register char *errstr)
 {
 
-	return (nb_dns_host_request2(nd, name, AF_INET, cookie, errstr));
+	return (nb_dns_host_request2(nd, name, AF_INET, 0, cookie, errstr));
 }
 
 int
 nb_dns_host_request2(register struct nb_dns_info *nd, register const char *name,
-    register int af, register void *cookie, register char *errstr)
+    register int af, register int qtype, register void *cookie, register char *errstr)
 {
-	register int qtype;
+	if (qtype != 16) {
 
-	switch (af) {
+		switch (af) {
 
-	case AF_INET:
+		case AF_INET:
-		qtype = T_A;
+			qtype = T_A;
-		break;
+			break;
 
 #ifdef AF_INET6
-	case AF_INET6:
+		case AF_INET6:
-		qtype = T_AAAA;
+			qtype = T_AAAA;
-		break;
+			break;
 #endif
 
-	default:
+		default:
-		snprintf(errstr, NB_DNS_ERRSIZE,
+			snprintf(errstr, NB_DNS_ERRSIZE,
-		    "nb_dns_host_request2(): uknown address family %d", af);
+			    "nb_dns_host_request2(): unknown address family %d", af);
-		return (-1);
+			return (-1);
+		}
 	}
 	return (_nb_dns_mkquery(nd, name, af, qtype, cookie, errstr));
 }
@@ -579,6 +580,7 @@ nb_dns_activity(struct nb_dns_info *nd, struct nb_dns_result *nr, char *errstr)
 				nr->host_errno = NO_RECOVERY;
 				return (-1);
 			}
+
 			memcpy(bp, rdata, rdlen);
 			*hap++ = bp;
 			bp += rdlen;
@@ -587,6 +589,20 @@ nb_dns_activity(struct nb_dns_info *nd, struct nb_dns_result *nr, char *errstr)
 			/* Keep looking for more A records */
 			break;
 
+		case T_TXT:
+			if (bp + rdlen >= ep) {
+				snprintf(errstr, NB_DNS_ERRSIZE,
+				    "nb_dns_activity(): overflow 1 for txt");
+				nr->host_errno = NO_RECOVERY;
+				return (-1);
+			}
+
+			memcpy(bp, rdata, rdlen);
+			he->h_name = bp+1; /* First char is a control character. */
+			nr->hostent = he;
+			nr->ttl = rttl;
+			return (1);
+
 		case T_PTR:
 			n = dn_expand((const u_char *)msg,
 			    (const u_char *)msg + msglen, rdata, bp, ep - bp);

src/nb_dns.h

@@ -22,7 +22,7 @@ void nb_dns_finish(struct nb_dns_info *);
 int nb_dns_fd(struct nb_dns_info *);
 
 int nb_dns_host_request(struct nb_dns_info *, const char *, void *, char *);
-int nb_dns_host_request2(struct nb_dns_info *, const char *, int,
+int nb_dns_host_request2(struct nb_dns_info *, const char *, int, int,
 				void *, char *);
 
 int nb_dns_addr_request(struct nb_dns_info *, nb_uint32_t, void *, char *);

src/parse.y

@@ -2,14 +2,14 @@
 // See the file "COPYING" in the main distribution directory for copyright.
 %}
 
-%expect 87
+%expect 88
 
 %token TOK_ADD TOK_ADD_TO TOK_ADDR TOK_ANY
 %token TOK_ATENDIF TOK_ATELSE TOK_ATIF TOK_ATIFDEF TOK_ATIFNDEF
 %token TOK_BOOL TOK_BREAK TOK_CASE TOK_CONST
 %token TOK_CONSTANT TOK_COPY TOK_COUNT TOK_COUNTER TOK_DEFAULT TOK_DELETE
 %token TOK_DOUBLE TOK_ELSE TOK_ENUM TOK_EVENT TOK_EXPORT TOK_FILE TOK_FOR
-%token TOK_FUNCTION TOK_GLOBAL TOK_ID TOK_IF TOK_INT
+%token TOK_FUNCTION TOK_GLOBAL TOK_HOOK TOK_ID TOK_IF TOK_INT
 %token TOK_INTERVAL TOK_LIST TOK_LOCAL TOK_MODULE
 %token TOK_NEXT TOK_OF TOK_PATTERN TOK_PATTERN_TEXT
 %token TOK_PORT TOK_PRINT TOK_RECORD TOK_REDEF
@@ -56,6 +56,7 @@
 %type <re> pattern
 %type <expr> expr init anonymous_function
 %type <event_expr> event
+%type <call_expr> hook
 %type <stmt> stmt stmt_list func_body for_head
 %type <type> type opt_type enum_body
 %type <func_type> func_hdr func_params
@@ -131,16 +132,18 @@ const char* cur_enum_elem_id = 0;
 type_decl_list* fake_type_decl_list = 0;
 TypeDecl* last_fake_type_decl = 0;
 
+static ID* cur_decl_type_id = 0;
+
 static void parser_new_enum (void)
 	{
 	/* Starting a new enum definition. */
 	assert(cur_enum_type == NULL);
-	cur_enum_type = new EnumType();
+	cur_enum_type = new EnumType(cur_decl_type_id->Name());
 
 	// For documentation purposes, a separate type object is created
 	// in order to avoid overlap that can be caused by redefs.
 	if ( generate_documentation )
-		cur_enum_type_doc = new CommentedEnumType();
+		cur_enum_type_doc = new CommentedEnumType(cur_decl_type_id->Name());
 	}
 
 static void parser_redef_enum (ID *id)
@@ -158,7 +161,7 @@ static void parser_redef_enum (ID *id)
 		}
 
 	if ( generate_documentation )
-		cur_enum_type_doc = new CommentedEnumType();
+		cur_enum_type_doc = new CommentedEnumType(id->Name());
 	}
 
 static void add_enum_comment (std::list<std::string>* comments)
@@ -456,17 +459,24 @@ expr:
 
 	|	'[' expr_list ']'
 			{
-			// A little crufty: we peek at the type of
-			// the first expression in the list.  If it's
-			// a record or a field assignment, then this
-			// is a record constructor.  If not, then this
-			// is a list used for an initializer.
-
 			set_location(@1, @3);
 
-			Expr* e0 = $2->Exprs()[0];
+			bool is_record_ctor = true;
-			if ( e0->Tag() == EXPR_FIELD_ASSIGN ||
+
-			     e0->Type()->Tag() == TYPE_RECORD )
+			// If every expression in the list is a field assignment,
+			// then treat it as a record constructor, else as a list
+			// used for an initializer.
+
+			for ( int i = 0; i < $2->Exprs().length(); ++i )
+				{
+				if ( $2->Exprs()[i]->Tag() != EXPR_FIELD_ASSIGN )
+					{
+					is_record_ctor = false;
+					break;
+					}
+				}
+
+			if ( is_record_ctor )
 				$$ = new RecordConstructorExpr($2);
 			else
 				$$ = $2;
@@ -859,7 +869,13 @@ type:
 	|	TOK_EVENT '(' formal_args ')'
 				{
 				set_location(@1, @3);
-				$$ = new FuncType($3, 0, 1);
+				$$ = new FuncType($3, 0, FUNC_FLAVOR_EVENT);
+				}
+
+	|	TOK_HOOK '(' formal_args ')'
+				{
+				set_location(@1, @3);
+				$$ = new FuncType($3, 0, FUNC_FLAVOR_HOOK);
 				}
 
 	|	TOK_FILE TOK_OF type
@@ -991,12 +1007,27 @@ decl:
 				ID* id = $2;
 				if ( id->Type()->Tag() == TYPE_FUNC )
 					{
-					if ( id->Type()->AsFuncType()->IsEvent() )
+					switch ( id->Type()->AsFuncType()->Flavor() ) {
-						current_reST_doc->AddEvent(
+
-							new BroDocObj(id, reST_doc_comments));
+					case FUNC_FLAVOR_FUNCTION:
-					else
 						current_reST_doc->AddFunction(
 							new BroDocObj(id, reST_doc_comments));
+						break;
+
+					case FUNC_FLAVOR_EVENT:
+						current_reST_doc->AddEvent(
+							new BroDocObj(id, reST_doc_comments));
+						break;
+
+					case FUNC_FLAVOR_HOOK:
+						current_reST_doc->AddHook(
+							new BroDocObj(id, reST_doc_comments));
+						break;
+
+					default:
+						reporter->InternalError("invalid function flavor");
+						break;
+					}
 					}
 
 				else
@@ -1098,9 +1129,10 @@ decl:
 				}
 			}
 
-	|	TOK_TYPE global_id ':' type opt_attr ';'
+	|	TOK_TYPE global_id ':' { cur_decl_type_id = $2; } type opt_attr ';'
 			{
-			add_type($2, $4, $5, 0);
+			cur_decl_type_id = 0;
+			add_type($2, $5, $6, 0);
 
 			if ( generate_documentation )
 				{
@@ -1175,6 +1207,15 @@ func_hdr:
 				current_reST_doc->AddEventHandler(
 					new BroDocObj($2, reST_doc_comments));
 			}
+	|	TOK_HOOK def_global_id func_params
+			{
+			begin_func($2, current_module.c_str(),
+				   FUNC_FLAVOR_HOOK, 0, $3);
+			$$ = $3;
+			if ( generate_documentation )
+				current_reST_doc->AddHookHandler(
+					new BroDocObj($2, reST_doc_comments));
+			}
 	|	TOK_REDEF TOK_EVENT event_id func_params
 			{
 			begin_func($3, current_module.c_str(),
@@ -1209,9 +1250,9 @@ begin_func:
 
 func_params:
 		'(' formal_args ')' ':' type
-			{ $$ = new FuncType($2, $5, 0); }
+			{ $$ = new FuncType($2, $5, FUNC_FLAVOR_FUNCTION); }
 	|	'(' formal_args ')'
-			{ $$ = new FuncType($2, base_type(TYPE_VOID), 0); }
+			{ $$ = new FuncType($2, base_type(TYPE_VOID), FUNC_FLAVOR_FUNCTION); }
 	;
 
 opt_type:
@@ -1331,6 +1372,14 @@ stmt:
 			    brofiler.AddStmt($$);
 			}
 
+	|	TOK_HOOK hook ';' opt_no_test
+			{
+			set_location(@1, @4);
+			$$ = new HookStmt($2);
+			if ( ! $4 )
+			    brofiler.AddStmt($$);
+			}
+
 	|	TOK_IF '(' expr ')' stmt
 			{
 			set_location(@1, @4);
@@ -1484,6 +1533,14 @@ event:
 			}
 	;
 
+hook:
+		expr '(' opt_expr_list ')'
+			{
+			set_location(@1, @4);
+			$$ = new CallExpr($1, $3, true);
+			}
+	;
+
 case_list:
 		case_list case
 			{ $1->append($2); }

src/rule-parse.y

@@ -1,13 +1,30 @@
 %{
 #include <stdio.h>
+#include <netinet/in.h>
+#include <vector>
+#include "config.h"
 #include "RuleMatcher.h"
 #include "Reporter.h"
+#include "IPAddr.h"
+#include "net_util.h"
 
 extern void begin_PS();
 extern void end_PS();
 
 Rule* current_rule = 0;
 const char* current_rule_file = 0;
+
+static uint8_t mask_to_len(uint32_t mask)
+	{
+	if ( mask == 0xffffffff )
+	    return 32;
+
+	uint32_t x = ~mask + 1;
+	uint8_t len;
+	for ( len = 0; len < 32 && (! (x & (1 << len))); ++len );
+
+	return len;
+	}
 %}
 
 %token TOK_COMP
@@ -21,6 +38,7 @@ const char* current_rule_file = 0;
 %token TOK_IDENT
 %token TOK_INT
 %token TOK_IP
+%token TOK_IP6
 %token TOK_IP_OPTIONS
 %token TOK_IP_OPTION_SYM
 %token TOK_IP_PROTO
@@ -49,7 +67,9 @@ const char* current_rule_file = 0;
 %type <hdr_test> hdr_expr
 %type <range> range rangeopt
 %type <vallist> value_list
+%type <prefix_val_list> prefix_value_list
 %type <mval> TOK_IP value
+%type <prefixval> TOK_IP6 prefix_value
 %type <prot> TOK_PROT
 %type <ptype> TOK_PATTERN_TYPE
 
@@ -57,6 +77,8 @@ const char* current_rule_file = 0;
 	Rule* rule;
 	RuleHdrTest* hdr_test;
 	maskedvalue_list* vallist;
+	vector<IPPrefix>* prefix_val_list;
+	IPPrefix* prefixval;
 
 	bool bl;
 	int val;
@@ -91,11 +113,11 @@ rule_attr_list:
 	;
 
 rule_attr:
-		TOK_DST_IP TOK_COMP value_list
+		TOK_DST_IP TOK_COMP prefix_value_list
 			{
 			current_rule->AddHdrTest(new RuleHdrTest(
-				RuleHdrTest::IP, 16, 4,
+				RuleHdrTest::IPDst,
-				(RuleHdrTest::Comp) $2, $3));
+				(RuleHdrTest::Comp) $2, *($3)));
 			}
 
 	|	TOK_DST_PORT TOK_COMP value_list
@@ -123,10 +145,14 @@ rule_attr:
 			{
 			int proto = 0;
 			switch ( $3 ) {
-			case RuleHdrTest::ICMP: proto = 1; break;
+			case RuleHdrTest::ICMP: proto = IPPROTO_ICMP; break;
+			case RuleHdrTest::ICMPv6: proto = IPPROTO_ICMPV6; break;
+			// signature matching against outer packet headers of IP-in-IP
+			// tunneling not supported, so do a no-op there
 			case RuleHdrTest::IP: proto = 0; break;
-			case RuleHdrTest::TCP: proto = 6; break;
+			case RuleHdrTest::IPv6: proto = 0; break;
-			case RuleHdrTest::UDP: proto = 17; break;
+			case RuleHdrTest::TCP: proto = IPPROTO_TCP; break;
+			case RuleHdrTest::UDP: proto = IPPROTO_UDP; break;
 			default:
 				rules_error("internal_error: unknown protocol");
 			}
@@ -140,16 +166,20 @@ rule_attr:
 				val->mask = 0xffffffff;
 				vallist->append(val);
 
+				// offset & size params are dummies, actual next proto value in
+				// header is retrieved dynamically via IP_Hdr::NextProto()
 				current_rule->AddHdrTest(new RuleHdrTest(
-					RuleHdrTest::IP, 9, 1,
+					RuleHdrTest::NEXT, 0, 0,
 					(RuleHdrTest::Comp) $2, vallist));
 				}
 			}
 
 	|	TOK_IP_PROTO TOK_COMP value_list
 			{
+			// offset & size params are dummies, actual next proto value in
+			// header is retrieved dynamically via IP_Hdr::NextProto()
 			current_rule->AddHdrTest(new RuleHdrTest(
-				RuleHdrTest::IP, 9, 1,
+				RuleHdrTest::NEXT, 0, 0,
 				(RuleHdrTest::Comp) $2, $3));
 			}
 
@@ -193,11 +223,11 @@ rule_attr:
 	|	TOK_SAME_IP
 			{ current_rule->AddCondition(new RuleConditionSameIP()); }
 
-	|	TOK_SRC_IP TOK_COMP value_list
+	|	TOK_SRC_IP TOK_COMP prefix_value_list
 			{
 			current_rule->AddHdrTest(new RuleHdrTest(
-				RuleHdrTest::IP, 12, 4,
+				RuleHdrTest::IPSrc,
-				(RuleHdrTest::Comp) $2, $3));
+				(RuleHdrTest::Comp) $2, *($3)));
 			}
 
 	|	TOK_SRC_PORT TOK_COMP value_list
@@ -254,6 +284,38 @@ value_list:
 			}
 	;
 
+prefix_value_list:
+		prefix_value_list ',' prefix_value
+			{
+			$$ = $1;
+			$$->push_back(*($3));
+			}
+	|	prefix_value_list ',' TOK_IDENT
+			{
+			$$ = $1;
+			id_to_maskedvallist($3, 0, $1);
+			}
+	|	prefix_value
+			{
+			$$ = new vector<IPPrefix>();
+			$$->push_back(*($1));
+			}
+	|	TOK_IDENT
+			{
+			$$ = new vector<IPPrefix>();
+			id_to_maskedvallist($1, 0, $$);
+			}
+	;
+
+prefix_value:
+		TOK_IP
+			{
+			$$ = new IPPrefix(IPAddr(IPv4, &($1.val), IPAddr::Host),
+			                  mask_to_len($1.mask));
+			}
+	|	TOK_IP6
+	;
+
 value:
 		TOK_INT
 			{ $$.val = $1; $$.mask = 0xffffffff; }

src/rule-scan.l

@@ -1,12 +1,13 @@
 %{
-typedef unsigned int uint32;
-
 #include <string.h>
+#include <string>
 #include <sys/types.h>
 #include <sys/socket.h>
 #include <netinet/in.h>
 #include <arpa/inet.h>
 #include "RuleMatcher.h"
+#include "IPAddr.h"
+#include "util.h"
 #include "rule-parse.h"
 
 int rules_line_number = 0;
@@ -14,11 +15,14 @@ int rules_line_number = 0;
 
 %x PS
 
+OWS	[ \t]*
 WS	[ \t]+
 D	[0-9]+
 H	[0-9a-fA-F]+
+HEX {H}
 STRING	\"([^\n\"]|\\\")*\"
 ID	([0-9a-zA-Z_-]+::)*[0-9a-zA-Z_-]+
+IP6	("["({HEX}:){7}{HEX}"]")|("["0x{HEX}({HEX}|:)*"::"({HEX}|:)*"]")|("["({HEX}|:)*"::"({HEX}|:)*"]")|("["({HEX}|:)*"::"({HEX}|:)*({D}"."){3}{D}"]")
 RE	\/(\\\/)?([^/]|[^\\]\\\/)*\/
 META	\.[^ \t]+{WS}[^\n]+
 PID	([0-9a-zA-Z_-]|"::")+
@@ -34,6 +38,18 @@ PID	([0-9a-zA-Z_-]|"::")+
 	\n	++rules_line_number;
 }
 
+{IP6}	{
+	rules_lval.prefixval = new IPPrefix(IPAddr(extract_ip(yytext)), 128, true);
+	return TOK_IP6;
+	}
+
+{IP6}{OWS}"/"{OWS}{D}	{
+	int len = 0;
+	string ip = extract_ip_and_len(yytext, &len);
+	rules_lval.prefixval = new IPPrefix(IPAddr(ip), len, true);
+	return TOK_IP6;
+	}
+
 [!\]\[{}&:,]	return rules_text[0];
 
 "<="	{ rules_lval.val = RuleHdrTest::LE; return TOK_COMP; }
@@ -45,7 +61,9 @@ PID	([0-9a-zA-Z_-]|"::")+
 "!="	{ rules_lval.val = RuleHdrTest::NE; return TOK_COMP; }
 
 ip	{ rules_lval.val = RuleHdrTest::IP; return TOK_PROT; }
+ip6	{ rules_lval.val = RuleHdrTest::IPv6; return TOK_PROT; }
 icmp	{ rules_lval.val = RuleHdrTest::ICMP; return TOK_PROT; }
+icmp6	{ rules_lval.val = RuleHdrTest::ICMPv6; return TOK_PROT; }
 tcp	{ rules_lval.val = RuleHdrTest::TCP; return TOK_PROT; }
 udp	{ rules_lval.val = RuleHdrTest::UDP; return TOK_PROT; }
 
@@ -123,7 +141,7 @@ http	{ rules_lval.val = Rule::HTTP_REQUEST; return TOK_PATTERN_TYPE; }
 ftp	{ rules_lval.val = Rule::FTP; return TOK_PATTERN_TYPE; }
 finger	{ rules_lval.val = Rule::FINGER; return TOK_PATTERN_TYPE; }
 
-{D}("."{D}){3}"/"{D}	{
+{D}("."{D}){3}{OWS}"/"{OWS}{D}	{
 	char* s = strchr(yytext, '/');
 	*s++ = '\0';
 

src/scan.l

@@ -148,6 +148,7 @@ D	[0-9]+
 HEX	[0-9a-fA-F]+
 IDCOMPONENT [A-Za-z_][A-Za-z_0-9]*
 ID	{IDCOMPONENT}(::{IDCOMPONENT})*
+IP6 ("["({HEX}:){7}{HEX}"]")|("["0x{HEX}({HEX}|:)*"::"({HEX}|:)*"]")|("["({HEX}|:)*"::"({HEX}|:)*"]")|("["({HEX}|:)*"::"({HEX}|:)*({D}"."){3}{D}"]")
 FILE	[^ \t\n]+
 PREFIX	[^ \t\n]+
 FLOAT	(({D}*"."?{D})|({D}"."?{D}*))([eE][-+]?{D})?
@@ -229,21 +230,23 @@ ESCSEQ	(\\([^\n]|[0-7]+|x[[:xdigit:]]+))
 			}
 
 	/* IPv6 literal constant patterns */
-"["({HEX}:){7}{HEX}"]" {
+{IP6}	{
-	string s(yytext+1);
+	RET_CONST(new AddrVal(extract_ip(yytext)))
-	RET_CONST(new AddrVal(s.erase(s.size()-1)))
 }
-"["0x{HEX}({HEX}|:)*"::"({HEX}|:)*"]" {
+
-	string s(yytext+3);
+{IP6}{OWS}"/"{OWS}{D}	{
-	RET_CONST(new AddrVal(s.erase(s.size()-1)))
+	int len = 0;
+	string ip = extract_ip_and_len(yytext, &len);
+	RET_CONST(new SubNetVal(IPPrefix(IPAddr(ip), len, true)))
 }
-"["({HEX}|:)*"::"({HEX}|:)*"]" {
+
-	string s(yytext+1);
+	/* IPv4 literal constant patterns */
-	RET_CONST(new AddrVal(s.erase(s.size()-1)))
+({D}"."){3}{D}		RET_CONST(new AddrVal(yytext))
-}
+
-"["({HEX}|:)*"::"({HEX}|:)*({D}"."){3}{D}"]" {
+({D}"."){3}{D}{OWS}"/"{OWS}{D}	{
-	string s(yytext+1);
+	int len = 0;
-	RET_CONST(new AddrVal(s.erase(s.size()-1)))
+	string ip = extract_ip_and_len(yytext, &len);
+	RET_CONST(new SubNetVal(IPPrefix(IPAddr(ip), len)))
 }
 
 [!%*/+\-,:;<=>?()\[\]{}~$|]	return yytext[0];
@@ -284,6 +287,7 @@ for	return TOK_FOR;
 function	return TOK_FUNCTION;
 global	return TOK_GLOBAL;
 "?$"	return TOK_HAS_FIELD;
+hook	return TOK_HOOK;
 if	return TOK_IF;
 in	return TOK_IN;
 "!"{OWS}in/[^A-Za-z0-9]	return TOK_NOT_IN;	/* don't confuse w "! infoo"! */
@@ -484,8 +488,6 @@ F	RET_CONST(new Val(false, TYPE_BOOL))
 {FLOAT}{OWS}msec(s?)	RET_CONST(new IntervalVal(atof(yytext),Milliseconds))
 {FLOAT}{OWS}usec(s?)	RET_CONST(new IntervalVal(atof(yytext),Microseconds))
 
-({D}"."){3}{D}		RET_CONST(new AddrVal(yytext))
-
 "0x"{HEX}+	RET_CONST(new Val(static_cast<bro_uint_t>(strtoull(yytext, 0, 16)), TYPE_COUNT))
 
 {H}("."{H})+		RET_CONST(dns_mgr->LookupHost(yytext))

src/types.bif

@@ -156,6 +156,13 @@ type readdir_reply_t: record;
 
 type fsstat_t: record;
 
+
+module GLOBAL;
+
+type ModbusHeaders: record;
+type ModbusCoils: vector;
+type ModbusRegisters: vector;
+
 module Log;
 
 enum Writer %{

src/util.cc

@@ -43,6 +43,40 @@
 #include "Net.h"
 #include "Reporter.h"
 
+/**
+ * Return IP address without enclosing brackets and any leading 0x.
+ */
+std::string extract_ip(const std::string& i)
+	{
+	std::string s(skip_whitespace(i.c_str()));
+	if ( s.size() > 0 && s[0] == '[' )
+		s.erase(0, 1);
+
+	if ( s.size() > 1 && s.substr(0, 2) == "0x" )
+		s.erase(0, 2);
+
+	size_t pos = 0;
+	if ( (pos = s.find(']')) != std::string::npos )
+		s = s.substr(0, pos);
+
+	return s;
+	}
+
+/**
+ * Given a subnet string, return IP address and subnet length separately.
+ */
+std::string extract_ip_and_len(const std::string& i, int* len)
+	{
+	size_t pos = i.find('/');
+	if ( pos == std::string::npos )
+		return i;
+
+	if ( len )
+		*len = atoi(i.substr(pos + 1).c_str());
+
+	return extract_ip(i.substr(0, pos));
+	}
+
 /**
  * Takes a string, unescapes all characters that are escaped as hex codes
  * (\x##) and turns them into the equivalent ascii-codes. Returns a string

src/util.h

@@ -91,6 +91,9 @@ void delete_each(T* t)
 		delete *it;
 	}
 
+std::string extract_ip(const std::string& i);
+std::string extract_ip_and_len(const std::string& i, int* len);
+
 std::string get_unescaped_string(const std::string& str);
 std::string get_escaped_string(const std::string& str, bool escape_all);