Merge branch 'master' into topic/v6-addr

Conflicts:
	src/bro.bif

CHANGES

@@ -1,3 +1,78 @@
+2.0-41 | 2012-02-03 04:10:53 -0500
+
+ * Updates to the Software framework to simplify the API.
+  (Bernhard Amann)
+
+2.0-40 | 2012-02-03 01:55:27 -0800
+
+ * Fix typos in documentation. (Daniel Thayer)
+
+ * Fix sorting of lines in Brofiler coverage.log. (Daniel Thayer)
+   
+2.0-38 | 2012-01-31 11:50:53 -0800
+
+  * Canonify sorting of lines in Brofiler coverage.log. (Daniel
+    Thayer)
+
+2.0-36 | 2012-01-27 10:38:14 -0800
+
+  * New "Brofiler" mode that tracks and records script statements
+    executed during runtime. (Jon Siwek)
+
+    Use the BROFILER_FILE environment variable to point to a file in
+    which statement usage statistics from Bro script-layer can be
+    output.
+
+    Script statements that should be ignored can be marked with a "#
+    @no-test" comment. For example:
+
+        print "don't cover"; # @no-test
+
+        if ( F )
+            { # @no-test
+            ...
+            }
+
+  * Integrated coverage measurement into test-suite. (Jon Siwek)
+
+2.0-20 | 2012-01-25 16:34:51 -0800
+
+  * BiF cleanup (Matthias Vallentin)
+
+      - Rename NFS3::mode2string to a more generic file_mode().
+
+      - Unify do_profiling()/make_connection_persistent()/expect_connection()
+        to return any (i.e., nothing) instead of bools.
+
+      - Perform type checking on count-to-port conversion. Related to #684.
+
+      - Remove redundant connection_record() BiF. The same
+        functionality is provided by lookup_connection().
+
+      - Remove redundant active_connection() BiF. The same
+        functionality is provided by connection_exists().
+
+      - exit() now takes the exit code as argument.
+
+      - to_port() now received a string instead of a count.
+        
+2.0-9 | 2012-01-25 13:47:13 -0800
+
+  * Allow local table variables to be initialized with {} list
+    expressions. (Jon Siwek)
+
+2.0-7 | 2012-01-25 13:38:09 -0800
+
+  * Teach CompHash to allow indexing by records with vector/table/set
+    fields. Addresses #464. (Jon Siwek)
+
+2.0-5 | 2012-01-25 13:25:19 -0800
+
+  * Fixed a bug resulting in over-logging of detected webapps. (Seth Hall)
+
+  * Make communication log baseline test more reliable. (Jon Siwek)
+    
+  * Fixed some broken links in documentation. (Daniel Thayer)
     
 2.0 | 2012-01-11 13:52:22 -0800
 

VERSION

@@ -1 +1 @@
-2.0
+2.0-41

aux/binpac

@@ -1 +1 @@
-Subproject commit aa1aa85ddcf524ffcfcf9efa5277bfac341871f7
+Subproject commit 43308aab47a3357ca1885e1b6954154a2744d821

aux/bro-aux

@@ -1 +1 @@
-Subproject commit 1ac5080de06bb29f2bc9fc73bd7dbbf945c7dc81
+Subproject commit 139cc2e1e049c4e1cc7e95f20866102be1d3d599

aux/broccoli

@@ -1 +1 @@
-Subproject commit 722ca1f266d8cf2f79a89e24b6b0a15174365ccd
+Subproject commit 930e7c78221929849086a578308e2fdc99ac3fb8

aux/broctl

@@ -1 +1 @@
-Subproject commit 84636789bf48376d7278b31e0bd7be19ff72c566
+Subproject commit e908ba686dceb56065bdf569c18dd0f67f662f6b

aux/btest

@@ -1 +1 @@
-Subproject commit 5350e4652b44ce1fbd9fffe1228d097fb04247cd
+Subproject commit ee87db37b520b88a55323a9767234c30b801e439

cmake

@@ -1 +1 @@
-Subproject commit ca4ed1a237215765ce9a7f2bc4b57b56958039ef
+Subproject commit 2cc105577044a2d214124568f3f2496ed2ccbb34

doc/cluster.rst

@@ -45,7 +45,7 @@ This is the Bro process that sniffs network traffic and does protocol analysis o
 
 The rule of thumb we have followed recently is to allocate approximately 1 core for every 80Mbps of traffic that is being analyzed, however this estimate could be extremely traffic mix specific.  It has generally worked for mixed traffic with many users and servers.  For example, if your traffic peaks around 2Gbps (combined) and you want to handle traffic at peak load, you may want to have 26 cores available (2048 / 80 == 25.6).  If the 80Mbps estimate works for your traffic, this could be handled by 3 physical hosts dedicated to being workers with each one containing dual 6-core processors.  
 
-Once a flow based load balancer is put into place this model is extremely easy to scale as well so it’s recommended that you guess at the amount of hardware you will need to fully analyze your traffic.  If it turns out that you need more, it’s relatively easy to easy increase the size of the cluster in most cases.
+Once a flow based load balancer is put into place this model is extremely easy to scale as well so it’s recommended that you guess at the amount of hardware you will need to fully analyze your traffic.  If it turns out that you need more, it’s relatively easy to increase the size of the cluster in most cases.
 
 Frontend Options
 ----------------
@@ -58,7 +58,7 @@ Discrete hardware flow balancers
 cPacket
 ^^^^^^^
 
-If you are monitoring one or more 10G physical interfaces, the recommended solution is to use either a cFlow or cVu device from cPacket because they are currently being used very successfully at a number of sites.  These devices will perform layer-2 load balancing by rewriting the destination ethernet MAC address to cause each packet associated with a particular flow to have the same destination MAC.  The packets can then be passed directly to a monitoring host where each worker has a BPF filter to limit it's visibility to only that stream of flows or onward to a commodity switch to split the traffic out to multiple 1G interfaces for the workers.  This can ultimately greatly reduce costs since workers can use relatively inexpensive 1G interfaces.
+If you are monitoring one or more 10G physical interfaces, the recommended solution is to use either a cFlow or cVu device from cPacket because they are currently being used very successfully at a number of sites.  These devices will perform layer-2 load balancing by rewriting the destination ethernet MAC address to cause each packet associated with a particular flow to have the same destination MAC.  The packets can then be passed directly to a monitoring host where each worker has a BPF filter to limit its visibility to only that stream of flows or onward to a commodity switch to split the traffic out to multiple 1G interfaces for the workers.  This can ultimately greatly reduce costs since workers can use relatively inexpensive 1G interfaces.
 
 OpenFlow Switches
 ^^^^^^^^^^^^^^^^^
@@ -76,7 +76,7 @@ The PF_RING software for Linux has a “clustering” feature which will do flow
 Netmap
 ^^^^^^
 
-FreeBSD has an in-progress project named Netmap which will enabled flow based load balancing as well.  When it becomes viable for real world use, this document will be updated.
+FreeBSD has an in-progress project named Netmap which will enable flow based load balancing as well.  When it becomes viable for real world use, this document will be updated.
 
 Click! Software Router
 ^^^^^^^^^^^^^^^^^^^^^^

doc/faq.rst

@@ -156,8 +156,8 @@ alteration tools.
 Bro has two options to workaround such situations and ignore bad checksums:
 
 1) The ``-C`` command line option to ``bro``.
-2) An option called ``ignore_checksums`` that can be redefined at the policy
+2) An option called ``ignore_checksums`` that can be redefined at the
-   policy script layer (e.g. in your ``$PREFIX/share/bro/site/local/bro``):
+   policy script layer (e.g. in your ``$PREFIX/share/bro/site/local.bro``):
 
     .. code:: bro
 

doc/logging.rst

@@ -63,7 +63,7 @@ to work with.
 Filtering
 ---------
 
-To create new a new output file for an existing stream, you can add a
+To create a new output file for an existing stream, you can add a
 new filter. A filter can, e.g., restrict the set of fields being
 logged:
 
@@ -195,7 +195,7 @@ predicate that will be called for each log record:
         Log::add_filter(Conn::LOG, filter);
         }
 
-This will results in a log file ``conn-http.log`` that contains only
+This will result in a log file ``conn-http.log`` that contains only
 traffic detected and analyzed as HTTP traffic.
 
 Extending
@@ -254,7 +254,7 @@ being logged. For these cases, a stream can specify an event that will
 be generated every time a log record is written to it. All of Bro's
 default log streams define such an event. For example, the connection
 log stream raises the event :bro:id:`Conn::log_conn`. You
-could use that for example for flagging when a connection to
+could use that for example for flagging when a connection to a
 specific destination exceeds a certain duration:
 
 .. code:: bro
@@ -269,7 +269,7 @@ specific destination exceeds a certain duration:
         {
         if ( rec$duration > 5mins )
             NOTICE([$note=Long_Conn_Found, 
-                    $msg=fmt("unsually long conn to %s", rec$id$resp_h), 
+                    $msg=fmt("unusually long conn to %s", rec$id$resp_h), 
                     $id=rec$id]);
         }
 
@@ -319,7 +319,7 @@ example for the ``Foo`` module:
     module Foo;
 
     export {
-        # Create an ID for the our new stream. By convention, this is
+        # Create an ID for our new stream. By convention, this is
         # called "LOG".
         redef enum Log::ID += { LOG };
 

doc/notice.rst

@@ -21,7 +21,7 @@ Let's start with a little bit of background on Bro's philosophy on reporting
 things. Bro ships with a large number of policy scripts which perform a wide
 variety of analyses. Most of these scripts monitor for activity which might be
 of interest for the user. However, none of these scripts determines the
-importance of what it finds itself. Instead, the scripts only flags situations
+importance of what it finds itself. Instead, the scripts only flag situations
 as *potentially* interesting, leaving it to the local configuration to define
 which of them are in fact actionable. This decoupling of detection and
 reporting allows Bro to address the different needs that sites have:
@@ -221,7 +221,7 @@ framework.
 
     * - :bro:see:`Notice::not_suppressed_types`
       - Adding a :bro:see:`Notice::Type` to this set results in that notice
-        no longer undergoes the normal notice suppression that would
+        no longer undergoing the normal notice suppression that would
         take place. Be careful when using this in production it could
         result in a dramatic increase in the number of notices being
         processed.
@@ -270,21 +270,21 @@ fields used when raising notices are described in the following table:
         information about this particular instance of the notice type.
 
     * - ``$sub``
-      - This is a sub-message which meant for human readability but will
+      - This is a sub-message meant for human readability but will
         frequently also be used to contain data meant to be matched with the
         ``Notice::policy``.
 
     * - ``$conn``
       - If a connection record is available when the notice is being raised
-        and the notice represents some attribute of the connection the
+        and the notice represents some attribute of the connection, then the
         connection record can be given here. Other fields such as ``$id`` and
         ``$src`` will automatically be populated from this value.
 
     * - ``$id``
       - If a conn_id record is available when the notice is being raised and
-        the notice represents some attribute of the connection, the connection
+        the notice represents some attribute of the connection, then the
-        be given here. Other fields such as ``$src`` will automatically be
+        connection can be given here. Other fields such as ``$src`` will
-        populated from this value.
+        automatically be populated from this value.
 
     * - ``$src``
       - If the notice represents an attribute of a single host then it's
@@ -313,7 +313,7 @@ of the notice the best information about the notice. If the notice is
 representative of many connections and is an attribute of a host (e.g. a
 scanning host) it probably makes most sense to fill out the ``$src`` field and
 not give a connection or conn_id. If a notice is representative of a
-connection attribute (e.g. an apparent SSH login) the it makes sense to fill
+connection attribute (e.g. an apparent SSH login) then it makes sense to fill
 out either ``$conn`` or ``$id`` based on the data that is available when the
 notice is raised. Using care when inserting data into a notice will make later
 analysis easier when only the data to fully represent the occurrence that

doc/quickstart.rst

@@ -100,7 +100,7 @@ The following dependencies are required to build Bro:
 Optional Dependencies
 ~~~~~~~~~~~~~~~~~~~~~
 
-Bro can use libGeoIP for geo-locating IP addresses and sendmail for
+Bro can use libGeoIP for geo-locating IP addresses, and sendmail for
 sending emails.
 
 * RPM/RedHat-based Linux:
@@ -127,7 +127,7 @@ sending emails.
 
   Vanilla OS X installations don't ship with libmagic or libGeoIP, but
   if installed from your preferred package management system (e.g. MacPorts,
-  Fink Homebrew), they should be automatically detected and Bro will compile
+  Fink, or Homebrew), they should be automatically detected and Bro will compile
   against them.
 
 Additional steps may be needed to :doc:`get the right GeoIP database <geoip>`
@@ -151,8 +151,8 @@ for downloading the full source code experience for Bro via git is:
 
 .. note:: If you choose to clone the ``bro`` repository non-recursively for
    a "minimal Bro experience", be aware that compiling it depends on
-   BinPAC, which has it's own ``binpac`` repository.  Either install it
+   BinPAC, which has its own ``binpac`` repository.  Either install it
-   first or initizalize/update the cloned ``bro`` repository's
+   first or initialize/update the cloned ``bro`` repository's
    ``aux/binpac`` submodule.
 
 See the ``INSTALL`` file included with the source code for more information
@@ -196,7 +196,7 @@ BroControl is an interactive shell for easily operating/managing Bro
 installations on a single system or even across multiple systems in a
 traffic-monitoring cluster.
 
-.. note:: Below, ``$PREFIX``, is used to reference the Bro installation
+.. note:: Below, ``$PREFIX`` is used to reference the Bro installation
    root directory.
 
 A Minimal Starting Configuration
@@ -449,7 +449,7 @@ that only takes the email action for SSH logins to a defined set of servers:
         ]
     };
 
-You'll just have to trust the syntax for now, but what we've done is first
+You'll just have to trust the syntax for now, but what we've done is
 first declare our own variable to hold a set of watched addresses,
 ``watched_servers``; then added a record to the policy that will generate
 an email on the condition that the predicate function evaluates to true, which

doc/reporting-problems.rst

@@ -4,7 +4,7 @@ Reporting Problems
 
 .. rst-class:: opening
 
-    Here we summarizes some steps to follow when you see Bro doing
+    Here we summarize some steps to follow when you see Bro doing
     something it shouldn't. To provide help, it is often crucial for
     us to have a way of reliably reproducing the effect you're seeing.
     Unfortunately, reproducing problems can be rather tricky with Bro
@@ -20,16 +20,16 @@ Reporting Problems
 Generally, when you encounter a problem with Bro, the best thing to do
 is opening a new ticket in `Bro's issue tracker
 <http://tracker.bro-ids.org/>`__ and include information on how to
-reproduce the issue. Ideallt, your ticket should come with the
+reproduce the issue. Ideally, your ticket should come with the
 following:
 
 * The Bro version you're using (if working directly from the git
   repository, the branch and revision number.)
 
-* The output you're seeing along with a description what you'd expect
+* The output you're seeing along with a description of what you'd expect
   Bro to do instead.
 
-* A *small* trace in `libpcap format <http://tcpdump.org>`__
+* A *small* trace in `libpcap format <http://www.tcpdump.org>`__
   demonstrating the effect (assuming the problem doesn't happen right
   at startup already).
 
@@ -39,7 +39,7 @@ following:
 
 * Any non-standard scripts you're using (but please only those really
   necessary; just a small code snippet triggering the problem would
-  perfect).
+  be perfect).
 
 * If you encounter a crash, information from the core dump, such as
   the stack backtrace, can be very helpful. See below for more on
@@ -51,10 +51,10 @@ How Do I Get a Trace File?
 
 As Bro is usually running live, coming up with a small trace file that
 reproduces a problem can turn out to be quite a challenge. Often it
-works to best to start with a large trace that triggers the problem,
+works best to start with a large trace that triggers the problem,
-and then successively thin it out as much a possible.
+and then successively thin it out as much as possible.
 
-To get to the initial large trace, here are few things you can try:
+To get to the initial large trace, here are a few things you can try:
 
 * Capture a trace with `tcpdump <http://www.tcpdump.org/>`__, either
   on the same interface Bro is running on, or on another host where
@@ -66,14 +66,14 @@ To get to the initial large trace, here are few things you can try:
   (e.g., for HTTP only, try ``port 80``). 
 
 * Bro's command-line option ``-w <trace>`` records all packets it
-  processes into the given the file. You can then later run Bro
+  processes into the given file. You can then later run Bro
   offline on this trace and it will process the packets in the same
   way as it did live. This is particularly helpful with problems that
   only occur after Bro has already been running for some time. For
   example, sometimes a crash may be triggered by a particular kind of
   traffic only occurring rarely. Running Bro live with ``-w`` and
   then, after the crash, offline on the recorded trace might, with a
-  little bit of luck, reproduce the the problem reliably. However, be
+  little bit of luck, reproduce the problem reliably. However, be
   careful with ``-w``: it can result in huge trace files, quickly
   filling up your disk. (One way to mitigate the space issues is to
   periodically delete the trace file by configuring
@@ -96,7 +96,7 @@ much as possible. Here are a few things you can try to this end:
 * Very often, a single connection is able to demonstrate the problem.
   If you can identify which one it is (e.g., from one of Bro's
   ``*.log`` files) you can extract the connection's packets from the
-  trace usong tcpdump by filtering for the corresponding 4-tuple of
+  trace using tcpdump by filtering for the corresponding 4-tuple of
   addresses and ports:
 
   .. console::
@@ -131,8 +131,8 @@ First, you should configure Bro with the option ``--enable-debug`` and
 recompile; this will disable all compiler optimizations and thus make
 the core dump more useful (don't expect great performance with this
 version though; compiling Bro without optimization has a noticeable
-impact on its CPU usage.). Then enable core dumps if you don't have
+impact on its CPU usage.). Then enable core dumps if you haven't
-already (e.g., ``ulimit -c unlimited`` if you're using a bash).
+already (e.g., ``ulimit -c unlimited`` if you're using bash).
 
 Once Bro has crashed, start gdb with the Bro binary and the file
 containing the core dump. (Alternatively, you can also run Bro
@@ -188,7 +188,7 @@ belonging to the ``Connection`` class. That class has members
 
 Note that these values are stored in `network byte order
 <http://en.wikipedia.org/wiki/Endianness#Endianness_in_networking>`__
-so you will need flip the bytes around if you are on a low-endian
+so you will need to flip the bytes around if you are on a low-endian
 machine (which is why the above example prints them in hex). For
 example, if an IP address prints as ``0100007f`` , that's 127.0.0.1 .
 

doc/signatures.rst

@@ -57,7 +57,7 @@ contain signatures: By using the ``-s`` flag when you invoke Bro, or
 by extending the Bro variable :bro:id:`signature_files` using the ``+=``
 operator. If a signature file is given without a path, it is searched
 along the normal ``BROPATH``. The default extension of the file name
-is ``.sig``, and Bro appends that automatically when neccesary.
+is ``.sig``, and Bro appends that automatically when necessary.
 
 Signature language
 ==================
@@ -94,7 +94,7 @@ against. The following keywords are defined:
     given as IP addresses or CIDR masks.
 
 ``src-port``/``dst-port`` ``<int-list>``
-    Source and destination port, repectively.
+    Source and destination port, respectively.
 
 ``ip-proto tcp|udp|icmp``
     IP protocol.
@@ -126,8 +126,8 @@ CIDR notation for netmasks and is translated into a corresponding
 bitmask applied to the packet's value prior to the comparison (similar
 to the optional ``& integer``).
 
-Putting all together, this is an example condition that is
+Putting it all together, this is an example condition that is
-equivalent to ``dst- ip == 1.2.3.4/16, 5.6.7.8/24``:
+equivalent to ``dst-ip == 1.2.3.4/16, 5.6.7.8/24``:
 
 .. code:: bro-sig
 
@@ -143,7 +143,7 @@ Content conditions are defined by regular expressions. We
 differentiate two kinds of content conditions: first, the expression
 may be declared with the ``payload`` statement, in which case it is
 matched against the raw payload of a connection (for reassembled TCP
-streams) or of a each packet (for ICMP, UDP, and non-reassembled TCP).
+streams) or of each packet (for ICMP, UDP, and non-reassembled TCP).
 Second, it may be prefixed with an analyzer-specific label, in which
 case the expression is matched against the data as extracted by the
 corresponding analyzer.
@@ -208,7 +208,7 @@ To define dependencies between signatures, there are two conditions:
 
 ``requires-reverse-signature [!] <id>``
     Similar to ``requires-signature``, but ``id`` has to match for the
-    opposite direction of the same connection, compared the current
+    opposite direction of the same connection, compared to the current
     signature. This allows to model the notion of requests and
     replies.
 
@@ -338,7 +338,7 @@ Things to keep in mind when writing signatures
   signature engine and can be matched with ``\r`` and ``\n``,
   respectively. Generally, Bro follows `flex's regular expression
   syntax
-  <http://www.gnu.org/software/flex/manual/html_chapter/flex_7.html>`_.
+  <http://flex.sourceforge.net/manual/Patterns.html>`_.
   See the DPD signatures in ``base/frameworks/dpd/dpd.sig`` for some examples
   of fairly complex payload patterns.
 

doc/upgrade.rst

@@ -55,13 +55,13 @@ renamed to ``scripts/`` and contains major subdirectories ``base/``,
 further.
 
 The contents of the new ``scripts/`` directory, like the old/flat
-``policy/`` still gets installed under under the ``share/bro``
+``policy/`` still gets installed under the ``share/bro``
 subdirectory of the installation prefix path just like previous
 versions.  For example, if Bro was compiled like ``./configure
 --prefix=/usr/local/bro && make && make install``, then the script
 hierarchy can be found in ``/usr/local/bro/share/bro``.
 
-THe main
+The main
 subdirectories of that hierarchy are as follows:
 
 - ``base/`` contains all scripts that are loaded by Bro by default
@@ -132,7 +132,7 @@ Logging Framework
 
 - The new logging framework makes it possible to extend, customize,
   and filter logs very easily. See the :doc:`logging framework <logging>`
-  more information on usage.
+  for more information on usage.
 
 - A common pattern found in the new scripts is to store logging stream
   records for protocols inside the ``connection`` records so that
@@ -209,8 +209,8 @@ live analysis.
 BroControl now has an extensive plugin interface for adding new
 commands and options. Note that this is still considered experimental.
 
-We have remove the ``analysis`` command, and BroControl does currently
+We have removed the ``analysis`` command, and BroControl currently
-not not send daily alarm summaries anymore (this may be restored
+does not send daily alarm summaries anymore (this may be restored
 later).
 
 Removed Functionality
@@ -233,11 +233,11 @@ Development Infrastructure
 ==========================
 
 Bro development has moved from using SVN to Git for revision control.
-Users that like to use the latest Bro developments by checking it out
+Users that want to use the latest Bro development snapshot by checking it out
 from the source repositories should see the `development process
 <http://www.bro-ids.org/development/process.html>`_. Note that all the various
-sub-components now reside on their own repositories. However, the
+sub-components now reside in their own repositories. However, the
-top-level Bro repository includes them as git submodules so it's easu
+top-level Bro repository includes them as git submodules so it's easy
 to check them all out simultaneously.
 
 Bro now uses `CMake <http://www.cmake.org>`_ for its build system so

scripts/base/frameworks/software/main.bro

@@ -36,15 +36,17 @@ export {
 	## The record type that is used for representing and logging software.
 	type Info: record {
 		## The time at which the software was detected.
-		ts:               time &log;
+		ts:               time &log &optional;
 		## The IP address detected running the software.
 		host:             addr &log;
+		## The Port on which the software is running. Only sensible for server software.
+		host_p:           port &log &optional;
 		## The type of software detected (e.g. :bro:enum:`HTTP::SERVER`).
 		software_type:    Type &log &default=UNKNOWN;
 		## Name of the software (e.g. Apache).
-		name:             string &log;
+		name:             string &log &optional;
 		## Version of the software.
-		version:          Version &log;
+		version:          Version &log &optional;
 		## The full unparsed version string found because the version parsing 
 		## doesn't always work reliably in all cases and this acts as a 
 		## fallback in the logs.
@@ -65,31 +67,12 @@ export {
 	const asset_tracking = LOCAL_HOSTS &redef;
 	
 	## Other scripts should call this function when they detect software.
-	## unparsed_version: This is the full string from which the
-	##                   :bro:type:`Software::Info` was extracted.
-	##
 	## id: The connection id where the software was discovered.
 	##
 	## info: A record representing the software discovered.
 	## 
 	## Returns: T if the software was logged, F otherwise.
-	global found: function(id: conn_id, info: Software::Info): bool;
+	global found: function(id: conn_id, info: Info): bool;
-	
-	## Take many common software version strings and parse them 
-	## into a sensible :bro:type:`Software::Version` record.  There are 
-	## still many cases where scripts may have to have their own specific 
-	## version parsing though.
-	##
-	## unparsed_version: The raw version string.
-	##
-	## host: The host where the software was discovered.
-	##
-	## software_type: The type of software.
-	##
-	## Returns: A complete record ready for the :bro:id:`Software::found` function.
-	global parse: function(unparsed_version: string,
-	                       host: addr,
-	                       software_type: Type): Info;
 	
 	## Compare two version records.
 	## 
@@ -117,112 +100,23 @@ export {
 	global log_software: event(rec: Info);
 }
 
-event bro_init()
+event bro_init() &priority=5
 	{
 	Log::create_stream(Software::LOG, [$columns=Info, $ev=log_software]);
 	}
 	
-function parse_mozilla(unparsed_version: string, 
+type Description: record {
-	                   host: addr, 
+	name:             string;
-	                   software_type: Type): Info
+	version:          Version;
-	{
+	unparsed_version: string;
-	local software_name = "<unknown browser>";
+};
-	local v: Version;
-	local parts: table[count] of string;
 
-	if ( /Opera [0-9\.]*$/ in unparsed_version )
+# Defining this here because of a circular dependency between two functions.
-		{
+global parse_mozilla: function(unparsed_version: string): Description;
-		software_name = "Opera";
-		parts = split_all(unparsed_version, /Opera [0-9\.]*$/);
-		if ( 2 in parts )
-			v = parse(parts[2], host, software_type)$version;
-		}
-	else if ( / MSIE / in unparsed_version )
-		{
-		software_name = "MSIE";
-		if ( /Trident\/4\.0/ in unparsed_version )
-			v = [$major=8,$minor=0];
-		else if ( /Trident\/5\.0/ in unparsed_version )
-			v = [$major=9,$minor=0];
-		else if ( /Trident\/6\.0/ in unparsed_version )
-			v = [$major=10,$minor=0];
-		else
-			{
-			parts = split_all(unparsed_version, /MSIE [0-9]{1,2}\.*[0-9]*b?[0-9]*/);
-			if ( 2 in parts )
-				v = parse(parts[2], host, software_type)$version;
-			}
-		}
-	else if ( /Version\/.*Safari\// in unparsed_version )
-		{
-		software_name = "Safari";
-		parts = split_all(unparsed_version, /Version\/[0-9\.]*/);
-		if ( 2 in parts )
-			{
-			v = parse(parts[2], host, software_type)$version;
-			if ( / Mobile\/?.* Safari/ in unparsed_version )
-				v$addl = "Mobile";
-			}
-		}
-	else if ( /(Firefox|Netscape|Thunderbird)\/[0-9\.]*/ in unparsed_version )
-		{
-		parts = split_all(unparsed_version, /(Firefox|Netscape|Thunderbird)\/[0-9\.]*/);
-		if ( 2 in parts )
-			{
-			local tmp_s = parse(parts[2], host, software_type);
-			software_name = tmp_s$name;
-			v = tmp_s$version;
-			}
-		}
-	else if ( /Chrome\/.*Safari\// in unparsed_version )
-		{
-		software_name = "Chrome";
-		parts = split_all(unparsed_version, /Chrome\/[0-9\.]*/);
-		if ( 2 in parts )
-			v = parse(parts[2], host, software_type)$version;
-		}
-	else if ( /^Opera\// in unparsed_version )
-		{
-		if ( /Opera M(ini|obi)\// in unparsed_version )
-			{
-			parts = split_all(unparsed_version, /Opera M(ini|obi)/);
-			if ( 2 in parts )
-				software_name = parts[2];
-			parts = split_all(unparsed_version, /Version\/[0-9\.]*/);
-			if ( 2 in parts )
-				v = parse(parts[2], host, software_type)$version;
-			else
-				{
-				parts = split_all(unparsed_version, /Opera Mini\/[0-9\.]*/);
-				if ( 2 in parts )
-					v = parse(parts[2], host, software_type)$version;
-				}
-			}
-		else
-			{
-			software_name = "Opera";
-			parts = split_all(unparsed_version, /Version\/[0-9\.]*/);
-			if ( 2 in parts )
-				v = parse(parts[2], host, software_type)$version;
-			}
-		}
-	else if ( /AppleWebKit\/[0-9\.]*/ in unparsed_version )
-		{
-		software_name = "Unspecified WebKit";
-		parts = split_all(unparsed_version, /AppleWebKit\/[0-9\.]*/);
-		if ( 2 in parts )
-			v = parse(parts[2], host, software_type)$version;
-		}
-
-	return [$ts=network_time(), $host=host, $name=software_name, $version=v,
-	        $software_type=software_type, $unparsed_version=unparsed_version];
-	}
 
 # Don't even try to understand this now, just make sure the tests are 
 # working.
-function parse(unparsed_version: string,
+function parse(unparsed_version: string): Description
-	           host: addr,
-	           software_type: Type): Info
 	{
 	local software_name = "<parse error>";
 	local v: Version;
@@ -230,7 +124,7 @@ function parse(unparsed_version: string,
 	# Parse browser-alike versions separately
 	if ( /^(Mozilla|Opera)\/[0-9]\./ in unparsed_version )
 		{
-		return parse_mozilla(unparsed_version, host, software_type);
+		return parse_mozilla(unparsed_version);
 		}
 	else
 		{
@@ -292,9 +186,102 @@ function parse(unparsed_version: string,
 				v$major = extract_count(version_numbers[1]);
 			}
 		}
-	return [$ts=network_time(), $host=host, $name=software_name,
+	
-	        $version=v, $unparsed_version=unparsed_version,
+	return [$version=v, $unparsed_version=unparsed_version, $name=software_name];
-	        $software_type=software_type];
+	}
+
+
+function parse_mozilla(unparsed_version: string): Description
+	{
+	local software_name = "<unknown browser>";
+	local v: Version;
+	local parts: table[count] of string;
+	
+	if ( /Opera [0-9\.]*$/ in unparsed_version )
+		{
+		software_name = "Opera";
+		parts = split_all(unparsed_version, /Opera [0-9\.]*$/);
+		if ( 2 in parts )
+			v = parse(parts[2])$version;
+		}
+	else if ( / MSIE / in unparsed_version )
+		{
+		software_name = "MSIE";
+		if ( /Trident\/4\.0/ in unparsed_version )
+			v = [$major=8,$minor=0];
+		else if ( /Trident\/5\.0/ in unparsed_version )
+			v = [$major=9,$minor=0];
+		else if ( /Trident\/6\.0/ in unparsed_version )
+			v = [$major=10,$minor=0];
+		else
+			{
+			parts = split_all(unparsed_version, /MSIE [0-9]{1,2}\.*[0-9]*b?[0-9]*/);
+			if ( 2 in parts )
+				v = parse(parts[2])$version;
+			}
+		}
+	else if ( /Version\/.*Safari\// in unparsed_version )
+		{
+		software_name = "Safari";
+		parts = split_all(unparsed_version, /Version\/[0-9\.]*/);
+		if ( 2 in parts )
+			{
+			v = parse(parts[2])$version;
+			if ( / Mobile\/?.* Safari/ in unparsed_version )
+				v$addl = "Mobile";
+			}
+		}
+	else if ( /(Firefox|Netscape|Thunderbird)\/[0-9\.]*/ in unparsed_version )
+		{
+		parts = split_all(unparsed_version, /(Firefox|Netscape|Thunderbird)\/[0-9\.]*/);
+		if ( 2 in parts )
+			{
+			local tmp_s = parse(parts[2]);
+			software_name = tmp_s$name;
+			v = tmp_s$version;
+			}
+		}
+	else if ( /Chrome\/.*Safari\// in unparsed_version )
+		{
+		software_name = "Chrome";
+		parts = split_all(unparsed_version, /Chrome\/[0-9\.]*/);
+		if ( 2 in parts )
+			v = parse(parts[2])$version;
+		}
+	else if ( /^Opera\// in unparsed_version )
+		{
+		if ( /Opera M(ini|obi)\// in unparsed_version )
+			{
+			parts = split_all(unparsed_version, /Opera M(ini|obi)/);
+			if ( 2 in parts )
+				software_name = parts[2];
+			parts = split_all(unparsed_version, /Version\/[0-9\.]*/);
+			if ( 2 in parts )
+				v = parse(parts[2])$version;
+			else
+				{
+				parts = split_all(unparsed_version, /Opera Mini\/[0-9\.]*/);
+				if ( 2 in parts )
+					v = parse(parts[2])$version;
+				}
+			}
+		else
+			{
+			software_name = "Opera";
+			parts = split_all(unparsed_version, /Version\/[0-9\.]*/);
+			if ( 2 in parts )
+				v = parse(parts[2])$version;
+			}
+		}
+	else if ( /AppleWebKit\/[0-9\.]*/ in unparsed_version )
+		{
+		software_name = "Unspecified WebKit";
+		parts = split_all(unparsed_version, /AppleWebKit\/[0-9\.]*/);
+		if ( 2 in parts )
+			v = parse(parts[2])$version;
+		}
+
+	return [$version=v, $unparsed_version=unparsed_version, $name=software_name];
 	}
 
 
@@ -379,7 +366,7 @@ function software_fmt(i: Info): string
 
 # Insert a mapping into the table
 # Overides old entries for the same software and generates events if needed.
-event software_register(id: conn_id, info: Info)
+event register(id: conn_id, info: Info)
 	{
 	# Host already known?
 	if ( info$host !in tracked )
@@ -407,7 +394,31 @@ function found(id: conn_id, info: Info): bool
 	{
 	if ( info$force_log || addr_matches_host(info$host, asset_tracking) )
 		{
-		event software_register(id, info);
+		if ( !info?$ts ) 
+			info$ts=network_time();
+		
+		if ( info?$version ) # we have a version number and don't have to parse. check if the name is also set...
+			{
+				if ( ! info?$name ) 
+					{
+					Reporter::error("Required field name not present in Software::found");
+					return F;
+					}
+			}
+		else  # no version present, we have to parse...
+			{
+			if ( !info?$unparsed_version ) 
+				{
+				Reporter::error("No unparsed version string present in Info record with version in Software::found");
+				return F;
+				} 
+			local sw = parse(info$unparsed_version);
+			info$unparsed_version = sw$unparsed_version;
+			info$name = sw$name;
+			info$version = sw$version;
+			}
+		
+		event register(id, info);
 		return T;
 		}
 	else

scripts/base/protocols/irc/dcc-send.bro

@@ -103,7 +103,7 @@ event irc_dcc_message(c: connection, is_orig: bool,
             return;
 	c$irc$dcc_file_name = argument;
 	c$irc$dcc_file_size = size;
-	local p = to_port(dest_port, tcp);
+	local p = count_to_port(dest_port, tcp);
 	expect_connection(to_addr("0.0.0.0"), address, p, ANALYZER_FILE, 5 min);
 	dcc_expected_transfers[address, p] = c$irc;
 	}

scripts/policy/protocols/ftp/software.bro

@@ -23,7 +23,6 @@ event ftp_request(c: connection, command: string, arg: string) &priority=4
 	{
 	if ( command == "CLNT" )
 		{
-		local si = Software::parse(arg, c$id$orig_h, CLIENT);
+		Software::found(c$id, [$unparsed_version=arg, $host=c$id$orig_h, $software_type=CLIENT]);
-		Software::found(c$id, si);
 		}
 	}

scripts/policy/protocols/http/detect-webapps.bro

@@ -27,7 +27,8 @@ event signature_match(state: signature_state, msg: string, data: string) &priori
 	if ( /^webapp-/ !in state$sig_id ) return;
 	
 	local c = state$conn;
-	local si = Software::parse(msg, c$id$resp_h, WEB_APPLICATION);
+	local si = Software::Info;
+	si = [$unparsed_version=msg, $host=c$id$resp_h, $host_p=c$id$resp_p, $software_type=WEB_APPLICATION];
 	si$url = build_url_http(c$http);
 	if ( c$id$resp_h in Software::tracked &&
 	     si$name in Software::tracked[c$id$resp_h] )
@@ -36,7 +37,8 @@ event signature_match(state: signature_state, msg: string, data: string) &priori
 		# use that as the new url for the software.
 		# PROBLEM: different version of the same software on the same server with a shared root path
 		local is_substring = 0;
-		if ( Software::tracked[c$id$resp_h][si$name]?$url )
+		if ( Software::tracked[c$id$resp_h][si$name]?$url &&
+		     |si$url| <= |Software::tracked[c$id$resp_h][si$name]$url| )
 			is_substring = strstr(Software::tracked[c$id$resp_h][si$name]$url, si$url);
 		
 		if ( is_substring == 1 )

scripts/policy/protocols/http/software-browser-plugins.bro

@@ -27,8 +27,7 @@ event http_header(c: connection, is_orig: bool, name: string, value: string) &pr
 			# Flash doesn't include it's name so we'll add it here since it 
 			# simplifies the version parsing.
 			value = cat("Flash/", value);
-			local flash_version = Software::parse(value, c$id$orig_h, BROWSER_PLUGIN);
+			Software::found(c$id, [$unparsed_version=value, $host=c$id$orig_h, $software_type=BROWSER_PLUGIN]);
-			Software::found(c$id, flash_version);
 			}
 		}
 	else
@@ -55,7 +54,7 @@ event log_http(rec: Info)
 			local plugins = split(sw, /[[:blank:]]*;[[:blank:]]*/);
 			
 			for ( i in plugins )
-				Software::found(rec$id, Software::parse(plugins[i], rec$id$orig_h, BROWSER_PLUGIN));
+				Software::found(rec$id, [$unparsed_version=plugins[i], $host=rec$id$orig_h, $software_type=BROWSER_PLUGIN]);
 			}
 		}
 	}

scripts/policy/protocols/http/software.bro

@@ -23,18 +23,18 @@ event http_header(c: connection, is_orig: bool, name: string, value: string) &pr
 	if ( is_orig )
 		{
 		if ( name == "USER-AGENT" && ignored_user_agents !in value )
-			Software::found(c$id, Software::parse(value, c$id$orig_h, BROWSER));
+			Software::found(c$id, [$unparsed_version=value, $host=c$id$orig_h, $software_type=BROWSER]);
 		}
 	else
 		{
 		if ( name == "SERVER" )
-			Software::found(c$id, Software::parse(value, c$id$resp_h, SERVER));
+			Software::found(c$id, [$unparsed_version=value, $host=c$id$resp_h, $host_p=c$id$resp_p, $software_type=SERVER]);
 		else if ( name == "X-POWERED-BY" )
-			Software::found(c$id, Software::parse(value, c$id$resp_h, APPSERVER));
+			Software::found(c$id, [$unparsed_version=value, $host=c$id$resp_h, $host_p=c$id$resp_p, $software_type=APPSERVER]);
 		else if ( name == "MICROSOFTSHAREPOINTTEAMSERVICES" )
 			{
 			value = cat("SharePoint/", value);
-			Software::found(c$id, Software::parse(value, c$id$resp_h, APPSERVER));
+			Software::found(c$id, [$unparsed_version=value, $host=c$id$resp_h, $host_p=c$id$resp_p, $software_type=APPSERVER]);
 			}
 		}
 	}

scripts/policy/protocols/smtp/software.bro

@@ -75,8 +75,7 @@ event log_smtp(rec: Info)
 		if ( addr_matches_host(rec$id$orig_h,
 		                       detect_clients_in_messages_from) )
 			{
-			local s = Software::parse(rec$user_agent, client_ip, s_type);
+			Software::found(rec$id, [$unparsed_version=rec$user_agent, $host=client_ip, $software_type=s_type]);
-			Software::found(rec$id, s);
 			}
 		}
 	}

scripts/policy/protocols/ssh/software.bro

@@ -18,14 +18,12 @@ event ssh_client_version(c: connection, version: string) &priority=4
 	{
 	# Get rid of the protocol information when passing to the software framework.
 	local cleaned_version = sub(version, /^SSH[0-9\.\-]+/, "");
-	local si = Software::parse(cleaned_version, c$id$orig_h, CLIENT);
+	Software::found(c$id, [$unparsed_version=cleaned_version, $host=c$id$orig_h, $software_type=CLIENT]);
-	Software::found(c$id, si);
 	}
 
 event ssh_server_version(c: connection, version: string) &priority=4
 	{
 	# Get rid of the protocol information when passing to the software framework.
 	local cleaned_version = sub(version, /SSH[0-9\.\-]{2,}/, "");
-	local si = Software::parse(cleaned_version, c$id$resp_h, SERVER);
+	Software::found(c$id, [$unparsed_version=cleaned_version, $host=c$id$resp_h, $host_p=c$id$resp_p, $software_type=SERVER]);
-	Software::found(c$id, si);
 	}

src/Brofiler.cc

@@ -0,0 +1,84 @@
+#include <cstdio>
+#include <utility>
+#include <algorithm>
+#include "Brofiler.h"
+#include "util.h"
+
+Brofiler::Brofiler()
+	: delim('\t'), ignoring(0)
+	{
+	}
+
+Brofiler::~Brofiler()
+	{
+	}
+
+bool Brofiler::ReadStats()
+	{
+	char* bf = getenv("BRO_PROFILER_FILE");
+	if ( ! bf )
+		return false;
+
+	FILE* f = fopen(bf, "r");
+	if ( ! f )
+		return false;
+
+	char line[16384];
+	string delimiter;
+	delimiter = delim;
+
+	while( fgets(line, sizeof(line), f) )
+		{
+		line[strlen(line) - 1] = 0; //remove newline
+		string cnt(strtok(line, delimiter.c_str()));
+		string location(strtok(0, delimiter.c_str()));
+		string desc(strtok(0, delimiter.c_str()));
+		pair<string, string> location_desc(location, desc);
+		uint64 count;
+		atoi_n(cnt.size(), cnt.c_str(), 0, 10, count);
+		usage_map[location_desc] = count;
+		}
+
+	fclose(f);
+	return true;
+	}
+
+bool Brofiler::WriteStats()
+	{
+	char* bf = getenv("BRO_PROFILER_FILE");
+	if ( ! bf ) return false;
+
+	FILE* f = fopen(bf, "w");
+	if ( ! f )
+		{
+		reporter->Error("Failed to open BRO_PROFILER_FILE destination '%s' for writing\n", bf);
+		return false;
+		}
+
+	for ( list<const Stmt*>::const_iterator it = stmts.begin();
+	      it != stmts.end(); ++it )
+		{
+		ODesc location_info;
+		(*it)->GetLocationInfo()->Describe(&location_info);
+		ODesc desc_info;
+		(*it)->Describe(&desc_info);
+		string desc(desc_info.Description());
+		for_each(desc.begin(), desc.end(), canonicalize_desc());
+		pair<string, string> location_desc(location_info.Description(), desc);
+		if ( usage_map.find(location_desc) != usage_map.end() )
+			usage_map[location_desc] += (*it)->GetAccessCount();
+		else
+			usage_map[location_desc] = (*it)->GetAccessCount();
+		}
+
+	map<pair<string, string>, uint64 >::const_iterator it;
+	for ( it = usage_map.begin(); it != usage_map.end(); ++it )
+		{
+		fprintf(f, "%"PRIu64"%c%s%c%s\n", it->second, delim,
+				it->first.first.c_str(), delim, it->first.second.c_str());
+		}
+
+	fclose(f);
+	return true;
+	}
+

src/Brofiler.h

@@ -0,0 +1,79 @@
+#ifndef BROFILER_H_
+#define BROFILER_H_
+
+#include <map>
+#include <utility>
+#include <list>
+#include <Stmt.h>
+
+
+/**
+ * A simple class for managing stats of Bro script coverage across Bro runs.
+ */
+class Brofiler {
+public:
+	Brofiler();
+	virtual ~Brofiler();
+
+	/**
+	 * Imports Bro script Stmt usage information from file pointed to by
+	 * environment variable BRO_PROFILER_FILE.
+	 *
+	 * @return: true if usage info was read, otherwise false.
+	 */
+	bool ReadStats();
+
+	/**
+	 * Combines usage stats from current run with any read from ReadStats(),
+	 * then writes information to file pointed to by environment variable
+	 * BRO_PROFILER_FILE.
+	 *
+	 * @return: true when usage info is written, otherwise false.
+	 */
+	bool WriteStats();
+
+	void SetDelim(char d) { delim = d; }
+
+	void IncIgnoreDepth() { ignoring++; }
+	void DecIgnoreDepth() { ignoring--; }
+
+	void AddStmt(const Stmt* s) { if ( ignoring == 0 ) stmts.push_back(s); }
+
+private:
+	/**
+	 * The current, global Brofiler instance creates this list at parse-time.
+	 */
+	list<const Stmt*> stmts;
+
+	/**
+	 * Indicates whether new statments will not be considered as part of
+	 * coverage statistics because it was marked with the @no-test tag.
+	 */
+	unsigned int ignoring;
+
+	/**
+	 * This maps Stmt location-desc pairs to the total number of times that
+	 * Stmt has been executed.  The map can be initialized from a file at
+	 * startup time and modified at shutdown time before writing back
+	 * to a file.
+	 */
+	map<pair<string, string>, uint64> usage_map;
+
+	/**
+	 * The character to use to delimit Brofiler output files.  Default is '\t'.
+	 */
+	char delim;
+
+	/**
+	 * A canonicalization routine for Stmt descriptions containing characters
+	 * that don't agree with the output format of Brofiler.
+	 */
+	struct canonicalize_desc {
+		void operator() (char& c)
+			{
+			if ( c == '\n' ) c = ' ';
+			}
+	};
+};
+
+#endif /* BROFILER_H_ */

src/CMakeLists.txt

@@ -282,6 +282,7 @@ set(bro_SRCS
     BPF_Program.cc
     BroDoc.cc
     BroDocObj.cc
+    Brofiler.cc
     BroString.cc
     CCL.cc
     ChunkedIO.cc

src/CompHash.cc

@@ -145,14 +145,16 @@ char* CompositeHash::SingleValHash(int type_check, char* kp0,
 	case TYPE_INTERNAL_VOID:
 	case TYPE_INTERNAL_OTHER:
 		{
-		if ( v->Type()->Tag() == TYPE_FUNC )
+		switch ( v->Type()->Tag() ) {
+		case TYPE_FUNC:
 			{
 			uint32* kp = AlignAndPadType<uint32>(kp0);
 			*kp = v->AsFunc()->GetUniqueFuncID();
 			kp1 = reinterpret_cast<char*>(kp+1);
+			break;
 			}
 
-		else if ( v->Type()->Tag() == TYPE_RECORD )
+		case TYPE_RECORD:
 			{
 			char* kp = kp0;
 			RecordVal* rv = v->AsRecordVal();
@@ -176,14 +178,87 @@ char* CompositeHash::SingleValHash(int type_check, char* kp0,
 				}
 
 			kp1 = kp;
+			break;
 			}
-		else
+
+		case TYPE_TABLE:
+			{
+			int* kp = AlignAndPadType<int>(kp0);
+			TableVal* tv = v->AsTableVal();
+			ListVal* lv = tv->ConvertToList();
+			*kp = tv->Size();
+			kp1 = reinterpret_cast<char*>(kp+1);
+			for ( int i = 0; i < tv->Size(); ++i )
+				{
+				Val* key = lv->Index(i);
+				if ( ! (kp1 = SingleValHash(type_check, kp1, key->Type(), key,
+				                            false)) )
+					return 0;
+
+				if ( ! v->Type()->IsSet() )
+					{
+					Val* val = tv->Lookup(key);
+					if ( ! (kp1 = SingleValHash(type_check, kp1, val->Type(),
+								    val, false)) )
+						return 0;
+					}
+				}
+			}
+			break;
+
+		case TYPE_VECTOR:
+			{
+			unsigned int* kp = AlignAndPadType<unsigned int>(kp0);
+			VectorVal* vv = v->AsVectorVal();
+			VectorType* vt = v->Type()->AsVectorType();
+			vector<Val*>* indices = v->AsVector();
+			*kp = vv->Size();
+			kp1 = reinterpret_cast<char*>(kp+1);
+			for ( unsigned int i = 0; i < vv->Size(); ++i )
+				{
+				Val* val = vv->Lookup(i);
+				unsigned int* kp = AlignAndPadType<unsigned int>(kp1);
+				*kp = i;
+				kp1 = reinterpret_cast<char*>(kp+1);
+				kp = AlignAndPadType<unsigned int>(kp1);
+				*kp = val ? 1 : 0;
+				kp1 = reinterpret_cast<char*>(kp+1);
+
+				if ( val )
+					{
+					if ( ! (kp1 = SingleValHash(type_check, kp1,
+					                            vt->YieldType(), val, false)) )
+						return 0;
+					}
+				}
+			}
+			break;
+
+		case TYPE_LIST:
+			{
+			int* kp = AlignAndPadType<int>(kp0);
+			ListVal* lv = v->AsListVal();
+			*kp = lv->Length();
+			kp1 = reinterpret_cast<char*>(kp+1);
+			for ( int i = 0; i < lv->Length(); ++i )
+				{
+				Val* v = lv->Index(i);
+				if ( ! (kp1 = SingleValHash(type_check, kp1, v->Type(), v,
+				                            false)) )
+					return 0;
+				}
+			}
+			break;
+
+		default:
 			{
 			reporter->InternalError("bad index type in CompositeHash::SingleValHash");
 			return 0;
 			}
 		}
-		break;
+
+		break; // case TYPE_INTERNAL_VOID/OTHER
+		}
 
 	case TYPE_INTERNAL_STRING:
 		{
@@ -355,10 +430,14 @@ int CompositeHash::SingleTypeKeySize(BroType* bt, const Val* v,
 	case TYPE_INTERNAL_VOID:
 	case TYPE_INTERNAL_OTHER:
 		{
-		if ( bt->Tag() == TYPE_FUNC )
+		switch ( bt->Tag() ) {
+		case TYPE_FUNC:
+			{
 			sz = SizeAlign(sz, sizeof(uint32));
+			break;
+			}
 
-		else if ( bt->Tag() == TYPE_RECORD )
+		case TYPE_RECORD:
 			{
 			const RecordVal* rv = v ? v->AsRecordVal() : 0;
 			RecordType* rt = bt->AsRecordType();
@@ -376,14 +455,81 @@ int CompositeHash::SingleTypeKeySize(BroType* bt, const Val* v,
 				if ( ! sz )
 					return 0;
 				}
+
+			break;
 			}
-		else
+
+		case TYPE_TABLE:
+			{
+			if ( ! v )
+				return (optional && ! calc_static_size) ? sz : 0;
+
+			sz = SizeAlign(sz, sizeof(int));
+			TableVal* tv = const_cast<TableVal*>(v->AsTableVal());
+			ListVal* lv = tv->ConvertToList();
+			for ( int i = 0; i < tv->Size(); ++i )
+				{
+				Val* key = lv->Index(i);
+				sz = SingleTypeKeySize(key->Type(), key, type_check, sz, false,
+				                       calc_static_size);
+				if ( ! sz ) return 0;
+				if ( ! bt->IsSet() )
+					{
+					Val* val = tv->Lookup(key);
+					sz = SingleTypeKeySize(val->Type(), val, type_check, sz,
+					                       false, calc_static_size);
+					if ( ! sz ) return 0;
+					}
+				}
+
+			break;
+			}
+
+		case TYPE_VECTOR:
+			{
+			if ( ! v )
+				return (optional && ! calc_static_size) ? sz : 0;
+
+			sz = SizeAlign(sz, sizeof(unsigned int));
+			VectorVal* vv = const_cast<VectorVal*>(v->AsVectorVal());
+			for ( unsigned int i = 0; i < vv->Size(); ++i )
+				{
+				Val* val = vv->Lookup(i);
+				sz = SizeAlign(sz, sizeof(unsigned int));
+				sz = SizeAlign(sz, sizeof(unsigned int));
+				if ( val )
+					sz = SingleTypeKeySize(bt->AsVectorType()->YieldType(),
+					                       val, type_check, sz, false,
+					                       calc_static_size);
+				if ( ! sz ) return 0;
+				}
+
+			break;
+			}
+
+		case TYPE_LIST:
+			{
+			sz = SizeAlign(sz, sizeof(int));
+			ListVal* lv = const_cast<ListVal*>(v->AsListVal());
+			for ( int i = 0; i < lv->Length(); ++i )
+				{
+				sz = SingleTypeKeySize(lv->Index(i)->Type(), lv->Index(i),
+				                       type_check, sz, false, calc_static_size);
+				if ( ! sz) return 0;
+				}
+
+			break;
+			}
+
+		default:
 			{
 			reporter->InternalError("bad index type in CompositeHash::CompositeHash");
 			return 0;
 			}
 		}
-		break;
+
+		break; // case TYPE_INTERNAL_VOID/OTHER
+		}
 
 	case TYPE_INTERNAL_STRING:
 		if ( ! v )
@@ -608,7 +754,8 @@ const char* CompositeHash::RecoverOneVal(const HashKey* k, const char* kp0,
 	case TYPE_INTERNAL_VOID:
 	case TYPE_INTERNAL_OTHER:
 		{
-		if ( t->Tag() == TYPE_FUNC )
+		switch ( t->Tag() ) {
+		case TYPE_FUNC:
 			{
 			const uint32* const kp = AlignType<uint32>(kp0);
 			kp1 = reinterpret_cast<const char*>(kp+1);
@@ -634,8 +781,9 @@ const char* CompositeHash::RecoverOneVal(const HashKey* k, const char* kp0,
 				  pval->Type()->Tag() != TYPE_FUNC )
 				reporter->InternalError("inconsistent aggregate Val in CompositeHash::RecoverOneVal()");
 			}
+			break;
 
-		else if ( t->Tag() == TYPE_RECORD )
+		case TYPE_RECORD:
 			{
 			const char* kp = kp0;
 			RecordType* rt = t->AsRecordType();
@@ -672,11 +820,91 @@ const char* CompositeHash::RecoverOneVal(const HashKey* k, const char* kp0,
 			pval = rv;
 			kp1 = kp;
 			}
-		else
+			break;
+
+		case TYPE_TABLE:
+			{
+			int n;
+			const int* const kp = AlignType<int>(kp0);
+			n = *kp;
+			kp1 = reinterpret_cast<const char*>(kp+1);
+			TableType* tt = t->AsTableType();
+			TableVal* tv = new TableVal(tt);
+			vector<Val*> keys, values;
+			for ( int i = 0; i < n; ++i )
+				{
+				Val* key;
+				kp1 = RecoverOneVal(k, kp1, k_end, tt->Indices(), key, false);
+				keys.push_back(key);
+				if ( ! t->IsSet() )
+					{
+					Val* value;
+					kp1 = RecoverOneVal(k, kp1, k_end, tt->YieldType(), value,
+					                    false);
+					values.push_back(value);
+					}
+				}
+
+			for ( int i = 0; i < n; ++i )
+				tv->Assign(keys[i], t->IsSet() ? 0 : values[i]);
+
+			pval = tv;
+			}
+			break;
+
+		case TYPE_VECTOR:
+			{
+			unsigned int n;
+			const unsigned int* kp = AlignType<unsigned int>(kp0);
+			n = *kp;
+			kp1 = reinterpret_cast<const char*>(kp+1);
+			VectorType* vt = t->AsVectorType();
+			VectorVal* vv = new VectorVal(vt);
+			for ( unsigned int i = 0; i < n; ++i )
+				{
+				kp = AlignType<unsigned int>(kp1);
+				unsigned int index = *kp;
+				kp1 = reinterpret_cast<const char*>(kp+1);
+				kp = AlignType<unsigned int>(kp1);
+				unsigned int have_val = *kp;
+				kp1 = reinterpret_cast<const char*>(kp+1);
+				Val* value = 0;
+				if ( have_val )
+					kp1 = RecoverOneVal(k, kp1, k_end, vt->YieldType(), value,
+					                    false);
+				vv->Assign(index, value, 0);
+				}
+
+			pval = vv;
+			}
+			break;
+
+		case TYPE_LIST:
+			{
+			int n;
+			const int* const kp = AlignType<int>(kp0);
+			n = *kp;
+			kp1 = reinterpret_cast<const char*>(kp+1);
+			TypeList* tl = t->AsTypeList();
+			ListVal* lv = new ListVal(TYPE_ANY);
+			for ( int i = 0; i < n; ++i )
+				{
+				Val* v;
+				BroType* it = (*tl->Types())[i];
+				kp1 = RecoverOneVal(k, kp1, k_end, it, v, false);
+				lv->Append(v);
+				}
+
+			pval = lv;
+			}
+			break;
+
+		default:
 			{
 			reporter->InternalError("bad index type in CompositeHash::DescribeKey");
 			}
 		}
+		}
 		break;
 
 	case TYPE_INTERNAL_STRING:

src/Expr.cc

@@ -2419,7 +2419,7 @@ bool RefExpr::DoUnserialize(UnserialInfo* info)
 	}
 
 AssignExpr::AssignExpr(Expr* arg_op1, Expr* arg_op2, int arg_is_init,
-			Val* arg_val)
+		       Val* arg_val, attr_list* arg_attrs)
 : BinaryExpr(EXPR_ASSIGN,
 		arg_is_init ? arg_op1 : arg_op1->MakeLvalue(), arg_op2)
 	{
@@ -2439,14 +2439,14 @@ AssignExpr::AssignExpr(Expr* arg_op1, Expr* arg_op2, int arg_is_init,
 
 	// We discard the status from TypeCheck since it has already
 	// generated error messages.
-	(void) TypeCheck();
+	(void) TypeCheck(arg_attrs);
 
 	val = arg_val ? arg_val->Ref() : 0;
 
 	SetLocationInfo(arg_op1->GetLocationInfo(), arg_op2->GetLocationInfo());
 	}
 
-bool AssignExpr::TypeCheck()
+bool AssignExpr::TypeCheck(attr_list* attrs)
 	{
 	TypeTag bt1 = op1->Type()->Tag();
 	TypeTag bt2 = op2->Type()->Tag();
@@ -2478,6 +2478,21 @@ bool AssignExpr::TypeCheck()
 		return true;
 		}
 
+	if ( bt1 == TYPE_TABLE && op2->Tag() == EXPR_LIST )
+		{
+		attr_list* attr_copy = 0;
+
+		if ( attrs )
+			{
+			attr_copy = new attr_list;
+			loop_over_list(*attrs, i)
+				attr_copy->append((*attrs)[i]);
+			}
+
+		op2 = new TableConstructorExpr(op2->AsListExpr(), attr_copy);
+		return true;
+		}
+
 	if ( bt1 == TYPE_VECTOR && bt2 == bt1 &&
 	     op2->Type()->AsVectorType()->IsUnspecifiedVector() )
 		{

src/Expr.h

@@ -623,7 +623,7 @@ class AssignExpr : public BinaryExpr {
 public:
 	// If val is given, evaluating this expression will always yield the val
 	// yet still perform the assignment.  Used for triggers.
-	AssignExpr(Expr* op1, Expr* op2, int is_init, Val* val = 0);
+	AssignExpr(Expr* op1, Expr* op2, int is_init, Val* val = 0, attr_list* attrs = 0);
 	virtual ~AssignExpr()	{ Unref(val); }
 
 	Expr* Simplify(SimplifyType simp_type);
@@ -638,7 +638,7 @@ protected:
 	friend class Expr;
 	AssignExpr()	{ }
 
-	bool TypeCheck();
+	bool TypeCheck(attr_list* attrs = 0);
 	bool TypeCheckArithmetics(TypeTag bt1, TypeTag bt2);
 
 	DECLARE_SERIAL(AssignExpr);

src/Stmt.cc

@@ -258,6 +258,8 @@ static BroFile* print_stdout = 0;
 
 Val* PrintStmt::DoExec(val_list* vals, stmt_flow_type& /* flow */) const
 	{
+	RegisterAccess();
+
 	if ( ! print_stdout )
 		print_stdout = new BroFile(stdout);
 

src/Stmt.h

@@ -52,6 +52,7 @@ public:
 
 	void RegisterAccess() const	{ last_access = network_time; access_count++; }
 	void AccessStats(ODesc* d) const;
+	uint32 GetAccessCount() const { return access_count; }
 
 	virtual void Describe(ODesc* d) const;
 

src/Var.cc

@@ -202,7 +202,8 @@ Stmt* add_local(ID* id, BroType* t, init_class c, Expr* init,
 		Ref(id);
 
 		Stmt* stmt =
-			new ExprStmt(new AssignExpr(new NameExpr(id), init, 0));
+		    new ExprStmt(new AssignExpr(new NameExpr(id), init, 0, 0,
+		        id->Attrs() ? id->Attrs()->Attrs() : 0 ));
 		stmt->SetLocationInfo(init->GetLocationInfo());
 
 		return stmt;

src/bro.bif

@@ -395,10 +395,12 @@ function setenv%(var: string, val: string%): bool
 
 ## Shuts down the Bro process immediately.
 ##
-## .. todo: Change function signature to ``exit(code: int): any``.
+## code: The exit code to return with.
-function exit%(%): int
+##
+## .. bro:see:: terminate
+function exit%(code: int%): any
 	%{
-	exit(0);
+	exit(code);
 	return 0;
 	%}
 
@@ -407,7 +409,7 @@ function exit%(%): int
 ## Returns: True after successful termination and false when Bro is still in
 ##          the process of shutting down.
 ##
-## .. bro:see:: bro_is_terminating
+## .. bro:see:: exit bro_is_terminating
 function terminate%(%): bool
 	%{
 	if ( terminating )
@@ -1945,12 +1947,12 @@ function record_fields%(rec: any%): record_field_table
 ##              get_matcher_stats
 ##              dump_rule_stats
 ##              get_gap_summary
-function do_profiling%(%) : bool
+function do_profiling%(%) : any
 	%{
 	if ( profiling_logger )
 		profiling_logger->Log();
 
-	return new Val(1, TYPE_BOOL);
+	return 0;
 	%}
 
 ## Checks whether a given IP address belongs to a local interface.
@@ -2210,14 +2212,16 @@ function port_to_count%(p: port%): count
 
 ## Converts a :bro:type:`count` and ``transport_proto`` to a :bro:type:`port`.
 ##
-## c: The :bro:type:`count` to convert.
+## num: The :bro:type:`port` number.
+##
+## proto: The transport protocol.
 ##
 ## Returns: The :bro:type:`count` *c* as :bro:type:`port`.
 ##
 ## .. bro:see:: port_to_count
-function count_to_port%(c: count, t: transport_proto%): port
+function count_to_port%(num: count, proto: transport_proto%): port
 	%{
-	return new PortVal(c, (TransportProto)(t->InternalInt()));
+	return new PortVal(num, (TransportProto)proto->AsEnum());
 	%}
 
 ## Converts a :bro:type:`string` to an :bro:type:`addr`.
@@ -2278,19 +2282,34 @@ function raw_bytes_to_v4_addr%(b: string%): addr
 	return new AddrVal(htonl(a));
 	%}
 
-## Creates a :bro:type:`port` from a given number and transport protocol.
+## Converts a :bro:type:`string` to an :bro:type:`port`.
 ##
-## num: The port number.
+## s: The :bro:type:`string` to convert.
 ##
-## proto: THe transport protocol of the port.
+## Returns: A :bro:type:`port` converted from *s*.
-##
-## Returns: A :bro:type:`port` with number *num* and transport protocol
-##          *proto*.
 ##
 ## .. bro:see:: to_addr to_count to_int
-function to_port%(num: count, proto: transport_proto%): port
+function to_port%(s: string%): port
     %{
-	return new PortVal(num, (TransportProto)proto->AsEnum());
+    int port = 0;
+    if ( s->Len() < 10 )
+        {
+        char* slash;
+        port = strtol(s->CheckString(), &slash, 10);
+        if ( port )
+            {
+            ++slash;
+            if ( streq(slash, "tcp") )
+                return new PortVal(port, TRANSPORT_TCP);
+            else if ( streq(slash, "udp") )
+                return new PortVal(port, TRANSPORT_UDP);
+            else if ( streq(slash, "icmp") )
+                return new PortVal(port, TRANSPORT_ICMP);
+            }
+        }
+
+    builtin_error("wrong port format, must be /[0-9]{1,5}\\/(tcp|udp|icmp)/");
+    return new PortVal(port, TRANSPORT_UNKNOWN);
 	%}
 
 ## Converts a reverse pointer name to an address. For example,
@@ -3728,7 +3747,7 @@ function x509_err2str%(err_num: count%): string
 ##
 ## Returns: A string representation of *mode* in the format
 ##          ``rw[xsS]rw[xsS]rw[xtT]``.
-function NFS3::mode2string%(mode: count%): string
+function file_mode%(mode: count%): string
 	%{
 	char str[12];
 	char *p = str;
@@ -3844,7 +3863,7 @@ function NFS3::mode2string%(mode: count%): string
 ##
 ## .. todo:: The return value should be changed to any.
 function expect_connection%(orig: addr, resp: addr, resp_p: port,
-				analyzer: count, tout: interval%) : bool
+				analyzer: count, tout: interval%) : any
 	%{
 	dpm->ExpectConnection(*orig->AsAddr(), *resp->AsAddr(), resp_p->Port(),
 				 resp_p->PortType(), (AnalyzerTag::Tag) analyzer, tout, 0);
@@ -5365,28 +5384,6 @@ function anonymize_addr%(a: addr, cl: IPAddrAnonymizationClass%): addr
 		}
 	%}
 
-## Deprecated. Will be removed.
-function active_connection%(id: conn_id%): bool
-	%{
-	Connection* c = sessions->FindConnection(id);
-	return new Val(c ? 1 : 0, TYPE_BOOL);
-	%}
-
-## Deprecated. Will be removed.
-function connection_record%(cid: conn_id%): connection
-	%{
-	Connection* c = sessions->FindConnection(cid);
-	if ( c )
-		return c->BuildConnVal();
-	else
-		{
-		// Hard to recover from this until we have union types ...
-		builtin_error("connection ID not a known connection (fatal)", cid);
-		exit(0);
-		return 0;
-		}
-	%}
-
 ## Deprecated. Will be removed.
 function dump_config%(%) : bool
 	%{
@@ -5394,10 +5391,10 @@ function dump_config%(%) : bool
 	%}
 
 ## Deprecated. Will be removed.
-function make_connection_persistent%(c: connection%) : bool
+function make_connection_persistent%(c: connection%) : any
 	%{
 	c->MakePersistent();
-	return new Val(1, TYPE_BOOL);
+	return 0;
 	%}
 
 %%{

src/event.bif

@@ -2574,7 +2574,7 @@ event nfs_proc_null%(c: connection, info: NFS3::info_t%);
 ## .. bro:see:: nfs_proc_create  nfs_proc_lookup nfs_proc_mkdir
 ##    nfs_proc_not_implemented nfs_proc_null nfs_proc_read nfs_proc_readdir
 ##    nfs_proc_readlink nfs_proc_remove nfs_proc_rmdir nfs_proc_write nfs_reply_status
-##    rpc_call rpc_dialogue rpc_reply NFS3::mode2string
+##    rpc_call rpc_dialogue rpc_reply file_mode
 ##
 ## .. todo:: Bro's current default configuration does not activate the protocol
 ##    analyzer that generates this event; the corresponding script has not yet

src/main.cc

@@ -47,10 +47,13 @@ extern "C" void OPENSSL_add_all_algorithms_conf(void);
 #include "ConnCompressor.h"
 #include "DPM.h"
 #include "BroDoc.h"
+#include "Brofiler.h"
 #include "LogWriterAscii.h"
 
 #include "binpac_bro.h"
 
+Brofiler brofiler;
+
 #ifndef HAVE_STRSEP
 extern "C" {
 char* strsep(char**, const char*);
@@ -195,6 +198,7 @@ void usage()
 	fprintf(stderr, "    $BRO_DNS_FAKE                  | disable DNS lookups (%s)\n", bro_dns_fake());
 	fprintf(stderr, "    $BRO_SEED_FILE                 | file to load seeds from (not set)\n");
 	fprintf(stderr, "    $BRO_LOG_SUFFIX                | ASCII log file extension (.%s)\n", LogWriterAscii::LogExt().c_str());
+	fprintf(stderr, "    $BRO_PROFILER_FILE             | Output file for script execution statistics (not set)\n");
 
 	exit(1);
 	}
@@ -261,6 +265,8 @@ void terminate_bro()
 
 	terminating = true;
 
+	brofiler.WriteStats();
+
 	EventHandlerPtr bro_done = internal_handler("bro_done");
 	if ( bro_done )
 		mgr.QueueEvent(bro_done, new val_list);
@@ -336,6 +342,8 @@ static void bro_new_handler()
 
 int main(int argc, char** argv)
 	{
+	brofiler.ReadStats();
+
 	bro_argc = argc;
 	bro_argv = new char* [argc];
 

src/parse.y

@@ -29,6 +29,8 @@
 
 %token TOK_DOC TOK_POST_DOC
 
+%token TOK_NO_TEST
+
 %left ',' '|'
 %right '=' TOK_ADD_TO TOK_REMOVE_FROM
 %right '?' ':' TOK_USING
@@ -42,6 +44,7 @@
 %right '!'
 %left '$' '[' ']' '(' ')' TOK_HAS_FIELD TOK_HAS_ATTR
 
+%type <b> opt_no_test opt_no_test_block
 %type <str> TOK_ID TOK_PATTERN_TEXT single_pattern TOK_DOC TOK_POST_DOC
 %type <str_l> opt_doc_list opt_post_doc_list
 %type <id> local_id global_id def_global_id event_id global_or_event_id resolve_id begin_func
@@ -80,10 +83,12 @@
 #include "Reporter.h"
 #include "BroDoc.h"
 #include "BroDocObj.h"
+#include "Brofiler.h"
 
 #include <list>
 #include <string>
 
+extern Brofiler brofiler;
 extern BroDoc* current_reST_doc;
 extern int generate_documentation;
 extern std::list<std::string>* reST_doc_comments;
@@ -194,6 +199,7 @@ static std::list<std::string>* concat_opt_docs (std::list<std::string>* pre,
 %}
 
 %union {
+	bool b;
 	char* str;
 	std::list<std::string>* str_l;
 	ID* id;
@@ -1313,22 +1319,28 @@ attr:
 	;
 
 stmt:
-		'{' stmt_list '}'
+		'{' opt_no_test_block stmt_list '}'
 			{
-			set_location(@1, @3);
+			set_location(@1, @4);
-			$$ = $2;
+			$$ = $3;
+			if ( $2 )
+			    brofiler.DecIgnoreDepth();
 			}
 
-	|	TOK_PRINT expr_list ';'
+	|	TOK_PRINT expr_list ';' opt_no_test
 			{
 			set_location(@1, @3);
 			$$ = new PrintStmt($2);
+			if ( ! $4 )
+			    brofiler.AddStmt($$);
 			}
 
-	|	TOK_EVENT event ';'
+	|	TOK_EVENT event ';' opt_no_test
 			{
 			set_location(@1, @3);
 			$$ = new EventStmt($2);
+			if ( ! $4 )
+			    brofiler.AddStmt($$);
 			}
 
 	|	TOK_IF '(' expr ')' stmt
@@ -1350,54 +1362,72 @@ stmt:
 			}
 
 	|	for_head stmt
-			{ $1->AsForStmt()->AddBody($2); }
+			{
+			$1->AsForStmt()->AddBody($2);
+			}
 
-	|	TOK_NEXT ';'
+	|	TOK_NEXT ';' opt_no_test
 			{
 			set_location(@1, @2);
 			$$ = new NextStmt;
+			if ( ! $3 )
+			    brofiler.AddStmt($$);
 			}
 
-	|	TOK_BREAK ';'
+	|	TOK_BREAK ';' opt_no_test
 			{
 			set_location(@1, @2);
 			$$ = new BreakStmt;
+			if ( ! $3 )
+			    brofiler.AddStmt($$);
 			}
 
-	|	TOK_RETURN ';'
+	|	TOK_RETURN ';' opt_no_test
 			{
 			set_location(@1, @2);
 			$$ = new ReturnStmt(0);
+			if ( ! $3 )
+			    brofiler.AddStmt($$);
 			}
 
-	|	TOK_RETURN expr ';'
+	|	TOK_RETURN expr ';' opt_no_test
 			{
 			set_location(@1, @2);
 			$$ = new ReturnStmt($2);
+			if ( ! $4 )
+			    brofiler.AddStmt($$);
 			}
 
-	|	TOK_ADD expr ';'
+	|	TOK_ADD expr ';' opt_no_test
 			{
 			set_location(@1, @3);
 			$$ = new AddStmt($2);
+			if ( ! $4 )
+			    brofiler.AddStmt($$);
 			}
 
-	|	TOK_DELETE expr ';'
+	|	TOK_DELETE expr ';' opt_no_test
 			{
 			set_location(@1, @3);
 			$$ = new DelStmt($2);
+			if ( ! $4 )
+			    brofiler.AddStmt($$);
 			}
 
-	|	TOK_LOCAL local_id opt_type init_class opt_init opt_attr ';'
+	|	TOK_LOCAL local_id opt_type init_class opt_init opt_attr ';' opt_no_test
 			{
 			set_location(@1, @7);
 			$$ = add_local($2, $3, $4, $5, $6, VAR_REGULAR);
+			if ( ! $8 )
+			    brofiler.AddStmt($$);
 			}
 
-	|	TOK_CONST local_id opt_type init_class opt_init opt_attr ';'
+	|	TOK_CONST local_id opt_type init_class opt_init opt_attr ';' opt_no_test
 			{
 			set_location(@1, @6);
 			$$ = add_local($2, $3, $4, $5, $6, VAR_CONST);
+			if ( ! $8 )
+			    brofiler.AddStmt($$);
 			}
 
 	|	TOK_WHEN '(' expr ')' stmt
@@ -1406,10 +1436,12 @@ stmt:
 			$$ = new WhenStmt($3, $5, 0, 0, false);
 			}
 
-	|	TOK_WHEN '(' expr ')' stmt TOK_TIMEOUT expr '{' stmt_list '}'
+	|	TOK_WHEN '(' expr ')' stmt TOK_TIMEOUT expr '{' opt_no_test_block stmt_list '}'
 			{
-			set_location(@3, @8);
+			set_location(@3, @9);
-			$$ = new WhenStmt($3, $5, $9, $7, false);
+			$$ = new WhenStmt($3, $5, $10, $7, false);
+			if ( $9 )
+			    brofiler.DecIgnoreDepth();
 			}
 
 
@@ -1419,16 +1451,20 @@ stmt:
 			$$ = new WhenStmt($4, $6, 0, 0, true);
 			}
 
-	|	TOK_RETURN TOK_WHEN '(' expr ')' stmt TOK_TIMEOUT expr '{' stmt_list '}'
+	|	TOK_RETURN TOK_WHEN '(' expr ')' stmt TOK_TIMEOUT expr '{' opt_no_test_block stmt_list '}'
 			{
-			set_location(@4, @9);
+			set_location(@4, @10);
-			$$ = new WhenStmt($4, $6, $10, $8, true);
+			$$ = new WhenStmt($4, $6, $11, $8, true);
+			if ( $10 )
+			    brofiler.DecIgnoreDepth();
 			}
 
-	|	expr ';'
+	|	expr ';' opt_no_test
 			{
 			set_location(@1, @2);
 			$$ = new ExprStmt($1);
+			if ( ! $3 )
+			    brofiler.AddStmt($$);
 			}
 
 	|	';'
@@ -1626,6 +1662,18 @@ opt_doc_list:
 			{ $$ = 0; }
 	;
 
+opt_no_test:
+		TOK_NO_TEST
+			{ $$ = true; }
+	|
+			{ $$ = false; }
+
+opt_no_test_block:
+		TOK_NO_TEST
+			{ $$ = true; brofiler.IncIgnoreDepth(); }
+	|
+			{ $$ = false; }
+
 %%
 
 int yyerror(const char msg[])

src/scan.l

@@ -216,6 +216,8 @@ ESCSEQ	(\\([^\n]|[0-7]+|x[[:xdigit:]]+))
 		}
 }
 
+#{OWS}@no-test.* return TOK_NO_TEST;
+
 #.*	/* eat comments */
 
 {WS}	/* eat whitespace */

src/util.cc

@@ -376,6 +376,7 @@ template<class T> int atoi_n(int len, const char* s, const char** end, int base,
 // Instantiate the ones we need.
 template int atoi_n<int>(int len, const char* s, const char** end, int base, int& result);
 template int atoi_n<int64_t>(int len, const char* s, const char** end, int base, int64_t& result);
+template int atoi_n<uint64_t>(int len, const char* s, const char** end, int base, uint64_t& result);
 
 char* uitoa_n(uint64 value, char* str, int n, int base, const char* prefix)
 	{

testing/.gitignore

@@ -0,0 +1 @@
+coverage.log

testing/Makefile

@@ -1,8 +1,20 @@
 
 DIRS=btest external
 
-all:
+all: make-verbose coverage
+
+brief: make-brief coverage
+
+make-verbose:
 	@for repo in $(DIRS); do (cd $$repo && make ); done
 
-brief:
+make-brief:
 	@for repo in $(DIRS); do (cd $$repo && make brief ); done
+
+coverage:
+	@test -f btest/coverage.log && cp btest/coverage.log `mktemp brocov.tmp.XXX` || true
+	@for f in external/*/coverage.log; do test -f $$f && cp $$f `mktemp brocov.tmp.XXX` || true; done
+	@echo "Complete test suite code coverage:"
+	@./scripts/coverage-calc "brocov.tmp.*" coverage.log `pwd`/../scripts
+	@rm -f brocov.tmp.*
+

testing/btest/.gitignore

@@ -1,2 +1,3 @@
 .tmp
 diag.log
+coverage.log

testing/btest/Baseline/coverage.coverage-blacklist/output

@@ -0,0 +1,5 @@
+1	/Users/jsiwek/Projects/bro/bro/testing/btest/.tmp/coverage.coverage-blacklist/coverage-blacklist.bro, line 13	print cover me;
+1	/Users/jsiwek/Projects/bro/bro/testing/btest/.tmp/coverage.coverage-blacklist/coverage-blacklist.bro, line 17	print always executed;
+0	/Users/jsiwek/Projects/bro/bro/testing/btest/.tmp/coverage.coverage-blacklist/coverage-blacklist.bro, line 26	print also impossible, but included in code coverage analysis;
+1	/Users/jsiwek/Projects/bro/bro/testing/btest/.tmp/coverage.coverage-blacklist/coverage-blacklist.bro, line 29	print success;
+1	/Users/jsiwek/Projects/bro/bro/testing/btest/.tmp/coverage.coverage-blacklist/coverage-blacklist.bro, line 5	print first;

testing/btest/Baseline/language.record-index-complex-fields/output

@@ -0,0 +1,27 @@
+{
+[1.2.3.4] = {
+[a=4, tags_v=[0, 1], tags_t={
+[two] = 2,
+[one] = 1
+}, tags_s={
+b,
+a
+}]
+}
+}
+{
+[a=13, tags_v=[, , 2, 3], tags_t={
+[four] = 4,
+[five] = 5
+}, tags_s={
+d,
+c
+}],
+[a=4, tags_v=[0, 1], tags_t={
+[two] = 2,
+[one] = 1
+}, tags_s={
+b,
+a
+}]
+}

testing/btest/Baseline/language.table-init/output

@@ -0,0 +1,10 @@
+{
+[2] = two,
+[1] = one
+}
+global table default
+{
+[4] = four,
+[3] = three
+}
+local table default

testing/btest/Baseline/scripts.base.frameworks.communication.communication_log_baseline/send.log

@@ -5,15 +5,17 @@
 #path	communication
 #fields	ts	peer	src_name	connected_peer_desc	connected_peer_addr	connected_peer_port	level	message
 #types	time	string	string	string	addr	port	string	string
-1324314302.411344	bro	parent	-	-	-	info	[#1/127.0.0.1:47757] added peer
+1326492291.485390	bro	parent	-	-	-	info	[#1/127.0.0.1:47757] added peer
-1324314302.414978	bro	child	-	-	-	info	[#1/127.0.0.1:47757] connected
+1326492291.491731	bro	child	-	-	-	info	[#1/127.0.0.1:47757] connected
-1324314302.415099	bro	parent	-	-	-	info	[#1/127.0.0.1:47757] peer connected
+1326492291.492024	bro	parent	-	-	-	info	[#1/127.0.0.1:47757] peer connected
-1324314302.415099	bro	parent	-	-	-	info	[#1/127.0.0.1:47757] phase: version
+1326492291.492024	bro	parent	-	-	-	info	[#1/127.0.0.1:47757] phase: version
-1324314302.417446	bro	script	-	-	-	info	connection established
+1326492291.492740	bro	script	-	-	-	info	connection established
-1324314302.417446	bro	script	-	-	-	info	requesting events matching /^?(NOTHING)$?/
+1326492291.492740	bro	script	-	-	-	info	requesting events matching /^?(NOTHING)$?/
-1324314302.417446	bro	script	-	-	-	info	accepting state
+1326492291.492740	bro	script	-	-	-	info	accepting state
-1324314302.418003	bro	parent	-	-	-	info	[#1/127.0.0.1:47757] phase: handshake
+1326492291.493800	bro	parent	-	-	-	info	[#1/127.0.0.1:47757] phase: handshake
-1324314302.418003	bro	parent	-	-	-	info	warning: no events to request
+1326492291.493800	bro	parent	-	-	-	info	warning: no events to request
-1324314302.418003	bro	parent	-	-	-	info	terminating...
+1326492291.494161	bro	parent	-	-	-	info	[#1/127.0.0.1:47757] peer_description is bro
-1324314302.418003	bro	parent	-	-	-	info	[#1/127.0.0.1:47757] peer_description is bro
+1326492291.494404	bro	parent	-	-	-	info	[#1/127.0.0.1:47757] peer supports keep-in-cache; using that
-1324314302.418003	bro	parent	-	-	-	info	[#1/127.0.0.1:47757] closing connection
+1326492291.494404	bro	parent	-	-	-	info	[#1/127.0.0.1:47757] phase: running
+1326492291.494404	bro	parent	-	-	-	info	terminating...
+1326492291.494404	bro	parent	-	-	-	info	[#1/127.0.0.1:47757] closing connection

testing/btest/Makefile

@@ -2,12 +2,16 @@
 DIAG=diag.log
 BTEST=../../aux/btest/btest
 
-all:
+all: cleanup
     # Showing all tests.
-	@rm -f $(DIAG)
 	@$(BTEST) -f $(DIAG)
+	@../scripts/coverage-calc ".tmp/script-coverage*" coverage.log `pwd`/../../scripts
 
-brief:
+brief: cleanup
     # Brief output showing only failed tests.
-	@rm -f $(DIAG)
 	@$(BTEST) -b -f $(DIAG)
+	@../scripts/coverage-calc ".tmp/script-coverage*" coverage.log `pwd`/../../scripts
+
+cleanup:
+	@rm -f $(DIAG)
+	@rm -f .tmp/script-coverage*

testing/btest/btest.cfg

@@ -10,9 +10,12 @@ BROPATH=`bash -c %(testbase)s/../../build/bro-path-dev`
 BRO_SEED_FILE=%(testbase)s/random.seed
 TZ=UTC
 LC_ALL=C
-PATH=%(testbase)s/../../build/src:%(testbase)s/../../aux/btest:%(default_path)s
+BTEST_PATH=%(testbase)s/../../aux/btest
+PATH=%(testbase)s/../../build/src:%(testbase)s/../scripts:%(testbase)s/../../aux/btest:%(default_path)s
 TRACES=%(testbase)s/Traces
 SCRIPTS=%(testbase)s/../scripts
 DIST=%(testbase)s/../..
 BUILD=%(testbase)s/../../build
 TEST_DIFF_CANONIFIER=$SCRIPTS/diff-canonifier
+TMPDIR=%(testbase)s/.tmp
+BRO_PROFILER_FILE=%(testbase)s/.tmp/script-coverage

testing/btest/coverage/coverage-blacklist.bro

@@ -0,0 +1,29 @@
+# @TEST-EXEC: BRO_PROFILER_FILE=coverage bro -b %INPUT
+# @TEST-EXEC: grep %INPUT coverage | sort -k2 >output
+# @TEST-EXEC: TEST_DIFF_CANONIFIER=$SCRIPTS/diff-remove-abspath btest-diff output
+
+print "first";
+
+if ( F )
+	{ # @no-test
+	print "hello";
+	print "world";
+	}
+
+print "cover me";
+
+if ( T )
+	{
+	print "always executed";
+	}
+
+print "don't cover me"; # @no-test
+
+if ( 0 + 0 == 1 ) print "impossible"; # @no-test
+
+if ( 1 == 0 )
+	{
+	print "also impossible, but included in code coverage analysis";
+	}
+
+print "success";

testing/btest/language/record-index-complex-fields.bro

@@ -0,0 +1,38 @@
+# @TEST-EXEC: bro -b %INPUT >output
+# @TEST-EXEC: btest-diff output
+
+# This test checks whether records with complex fields (tables, sets, vectors)
+# can be used as table/set indices.
+
+type MetaData: record {
+	a: count;
+	tags_v: vector of count;
+	tags_t: table[string] of count;
+	tags_s: set[string];
+};
+
+global ip_data: table[addr] of set[MetaData] = table();
+
+global t1_t: table[string] of count = { ["one"] = 1, ["two"] = 2 };
+global t2_t: table[string] of count = { ["four"] = 4, ["five"] = 5 };
+
+global t1_v: vector of count = vector();
+global t2_v: vector of count = vector();
+t1_v[0] = 0;
+t1_v[1] = 1;
+t2_v[2] = 2;
+t2_v[3] = 3;
+
+local m: MetaData = [$a=4, $tags_v=t1_v, $tags_t=t1_t, $tags_s=set("a", "b")];
+local n: MetaData = [$a=13, $tags_v=t2_v, $tags_t=t2_t, $tags_s=set("c", "d")];
+
+if ( 1.2.3.4 !in ip_data )
+	ip_data[1.2.3.4] = set(m);
+else
+	add ip_data[1.2.3.4][m];
+
+print ip_data;
+
+add ip_data[1.2.3.4][n];
+
+print ip_data[1.2.3.4];

testing/btest/language/table-init.bro

@@ -0,0 +1,20 @@
+# @TEST-EXEC: bro %INPUT >output
+# @TEST-EXEC: btest-diff output
+
+global global_table: table[count] of string = {
+    [1] = "one",
+	[2] = "two"
+} &default = "global table default";
+
+event bro_init()
+    {
+    local local_table: table[count] of string = {
+         [3] = "three",
+		 [4] = "four"
+    } &default = "local table default";
+
+    print global_table;
+    print global_table[0];
+    print local_table;
+    print local_table[0];
+    }

testing/btest/scripts/base/frameworks/communication/communication_log_baseline.bro

@@ -15,7 +15,7 @@ redef Communication::nodes += {
     ["foo"] = [$host = 127.0.0.1, $events = /NOTHING/, $connect=T]
 };
 
-event remote_connection_established(p: event_peer)
+event remote_connection_handshake_done(p: event_peer)
 	{
 	terminate_communication();
 	terminate();
@@ -29,8 +29,9 @@ event remote_connection_established(p: event_peer)
 
 @load frameworks/communication/listen
 
-event remote_connection_closed(p: event_peer)
+event remote_connection_handshake_done(p: event_peer)
 	{
+	terminate_communication();
 	terminate();
 	}
 

testing/btest/scripts/base/frameworks/software/version-parsing.bro

@@ -1,116 +1,116 @@
 # @TEST-EXEC: bro %INPUT > output
 # @TEST-EXEC: btest-diff output
 
-global ts = network_time();
+module Software;
-global host = 0.0.0.0;
 
-global matched_software: table[string] of Software::Info = {
+global matched_software: table[string] of Software::Description = {
 	["OpenSSH_4.4"] = 
-		[$name="OpenSSH", $version=[$major=4,$minor=4], $host=host, $ts=ts],
+		[$name="OpenSSH", $version=[$major=4,$minor=4], $unparsed_version=""],
 	["OpenSSH_5.2"] = 
-		[$name="OpenSSH", $version=[$major=5,$minor=2], $host=host, $ts=ts],
+		[$name="OpenSSH", $version=[$major=5,$minor=2], $unparsed_version=""],
 	["Apache/2.0.63 (Unix) mod_auth_kerb/5.3 mod_ssl/2.0.63 OpenSSL/0.9.7a mod_fastcgi/2.4.2"] =
-		[$name="Apache", $version=[$major=2,$minor=0,$minor2=63,$addl="Unix"], $host=host, $ts=ts],
+		[$name="Apache", $version=[$major=2,$minor=0,$minor2=63,$addl="Unix"], $unparsed_version=""],
 	["Apache/1.3.19 (Unix)"] =
-		[$name="Apache", $version=[$major=1,$minor=3,$minor2=19,$addl="Unix"], $host=host, $ts=ts],
+		[$name="Apache", $version=[$major=1,$minor=3,$minor2=19,$addl="Unix"], $unparsed_version=""],
 	["ProFTPD 1.2.5rc1 Server (Debian)"] =
-		[$name="ProFTPD", $version=[$major=1,$minor=2,$minor2=5,$addl="rc1"], $host=host, $ts=ts],
+		[$name="ProFTPD", $version=[$major=1,$minor=2,$minor2=5,$addl="rc1"], $unparsed_version=""],
 	["wu-2.4.2-academ[BETA-18-VR14](1)"] = 
-		[$name="wu", $version=[$major=2,$minor=4,$minor2=2,$addl="academ"], $host=host, $ts=ts],
+		[$name="wu", $version=[$major=2,$minor=4,$minor2=2,$addl="academ"], $unparsed_version=""],
 	["wu-2.6.2(1)"] =
-		[$name="wu", $version=[$major=2,$minor=6,$minor2=2,$addl="1"], $host=host, $ts=ts],
+		[$name="wu", $version=[$major=2,$minor=6,$minor2=2,$addl="1"], $unparsed_version=""],
 	["Java1.2.2-JDeveloper"] =
-		[$name="Java", $version=[$major=1,$minor=2,$minor2=2,$addl="JDeveloper"], $host=host, $ts=ts],
+		[$name="Java", $version=[$major=1,$minor=2,$minor2=2,$addl="JDeveloper"], $unparsed_version=""],
 	["Java/1.6.0_13"] = 
-		[$name="Java", $version=[$major=1,$minor=6,$minor2=0,$addl="13"], $host=host, $ts=ts],
+		[$name="Java", $version=[$major=1,$minor=6,$minor2=0,$addl="13"], $unparsed_version=""],
 	["Python-urllib/3.1"] = 
-		[$name="Python-urllib", $version=[$major=3,$minor=1], $host=host, $ts=ts],
+		[$name="Python-urllib", $version=[$major=3,$minor=1], $unparsed_version=""],
 	["libwww-perl/5.820"] = 
-		[$name="libwww-perl", $version=[$major=5,$minor=820], $host=host, $ts=ts],
+		[$name="libwww-perl", $version=[$major=5,$minor=820], $unparsed_version=""],
 	["Wget/1.9+cvs-stable (Red Hat modified)"] = 
-		[$name="Wget", $version=[$major=1,$minor=9,$addl="+cvs"], $host=host, $ts=ts],
+		[$name="Wget", $version=[$major=1,$minor=9,$addl="+cvs"], $unparsed_version=""],
 	["Wget/1.11.4 (Red Hat modified)"] = 
-		[$name="Wget", $version=[$major=1,$minor=11,$minor2=4,$addl="Red Hat modified"], $host=host, $ts=ts],
+		[$name="Wget", $version=[$major=1,$minor=11,$minor2=4,$addl="Red Hat modified"], $unparsed_version=""],
 	["curl/7.15.1 (i486-pc-linux-gnu) libcurl/7.15.1 OpenSSL/0.9.8a zlib/1.2.3 libidn/0.5.18"] =
-		[$name="curl", $version=[$major=7,$minor=15,$minor2=1,$addl="i486-pc-linux-gnu"], $host=host, $ts=ts],
+		[$name="curl", $version=[$major=7,$minor=15,$minor2=1,$addl="i486-pc-linux-gnu"], $unparsed_version=""],
 	["Apache"] = 
-		[$name="Apache", $host=host, $ts=ts],
+		[$name="Apache", $unparsed_version=""],
 	["Zope/(Zope 2.7.8-final, python 2.3.5, darwin) ZServer/1.1 Plone/Unknown"] =
-		[$name="Zope/(Zope", $version=[$major=2,$minor=7,$minor2=8,$addl="final"], $host=host, $ts=ts],
+		[$name="Zope/(Zope", $version=[$major=2,$minor=7,$minor2=8,$addl="final"], $unparsed_version=""],
 	["The Bat! (v2.00.9) Personal"] =
-		[$name="The Bat!", $version=[$major=2,$minor=0,$minor2=9,$addl="Personal"], $host=host, $ts=ts],
+		[$name="The Bat!", $version=[$major=2,$minor=0,$minor2=9,$addl="Personal"], $unparsed_version=""],
 	["Flash/10,2,153,1"] =
-		[$name="Flash", $version=[$major=10,$minor=2,$minor2=153,$addl="1"], $host=host, $ts=ts],
+		[$name="Flash", $version=[$major=10,$minor=2,$minor2=153,$addl="1"], $unparsed_version=""],
 	["mt2/1.2.3.967 Oct 13 2010-13:40:24 ord-pixel-x2 pid 0x35a3 13731"] = 
-		[$name="mt2", $version=[$major=1,$minor=2,$minor2=3,$addl="967"], $host=host, $ts=ts],
+		[$name="mt2", $version=[$major=1,$minor=2,$minor2=3,$addl="967"], $unparsed_version=""],
 	["CacheFlyServe v26b"] =
-		[$name="CacheFlyServe", $version=[$major=26,$addl="b"], $host=host, $ts=ts],
+		[$name="CacheFlyServe", $version=[$major=26,$addl="b"], $unparsed_version=""],
 	["Apache/2.0.46 (Win32) mod_ssl/2.0.46 OpenSSL/0.9.7b mod_jk2/2.0.4"] =
-		[$name="Apache", $version=[$major=2,$minor=0,$minor2=46,$addl="Win32"], $host=host, $ts=ts],
+		[$name="Apache", $version=[$major=2,$minor=0,$minor2=46,$addl="Win32"], $unparsed_version=""],
 	# I have no clue how I'd support this without a special case.
 	#["Apache mod_fcgid/2.3.6 mod_auth_passthrough/2.1 mod_bwlimited/1.4 FrontPage/5.0.2.2635"] =
-	#	[$name="Apache", $version=[], $host=host, $ts=ts],
+	#	[$name="Apache", $version=[], $unparsed_version=""],
 	["Apple iPhone v4.3.1 Weather v1.0.0.8G4"] =
-		[$name="Apple iPhone", $version=[$major=4,$minor=3,$minor2=1,$addl="Weather"], $host=host, $ts=ts],
+		[$name="Apple iPhone", $version=[$major=4,$minor=3,$minor2=1,$addl="Weather"], $unparsed_version=""],
 	["Mozilla/5.0 (iPhone; U; CPU iPhone OS 4_3_2 like Mac OS X; en-us) AppleWebKit/533.17.9 (KHTML, like Gecko) Version/5.0.2 Mobile/8H7 Safari/6533.18.5"] =
-		[$name="Safari", $version=[$major=5,$minor=0,$minor2=2,$addl="Mobile"], $host=host, $ts=ts],
+		[$name="Safari", $version=[$major=5,$minor=0,$minor2=2,$addl="Mobile"], $unparsed_version=""],
 	["Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_6_7; en-US) AppleWebKit/534.16 (KHTML, like Gecko) Chrome/10.0.648.205 Safari/534.16"] = 
-		[$name="Chrome", $version=[$major=10,$minor=0,$minor2=648,$addl="205"], $host=host, $ts=ts],
+		[$name="Chrome", $version=[$major=10,$minor=0,$minor2=648,$addl="205"], $unparsed_version=""],
 	["Opera/9.80 (Windows NT 6.1; U; sv) Presto/2.7.62 Version/11.01"] =
-		[$name="Opera", $version=[$major=11,$minor=1], $host=host, $ts=ts],
+		[$name="Opera", $version=[$major=11,$minor=1], $unparsed_version=""],
 	["Mozilla/5.0 (Windows; U; Windows NT 5.1; de; rv:1.9.2.11) Gecko/20101013 Lightning/1.0b2 Thunderbird/3.1.5"] =
-		[$name="Thunderbird", $version=[$major=3,$minor=1,$minor2=5], $host=host, $ts=ts],
+		[$name="Thunderbird", $version=[$major=3,$minor=1,$minor2=5], $unparsed_version=""],
 	["iTunes/9.0 (Macintosh; Intel Mac OS X 10.5.8) AppleWebKit/531.9"] = 
-		[$name="iTunes", $version=[$major=9,$minor=0,$addl="Macintosh"], $host=host, $ts=ts],
+		[$name="iTunes", $version=[$major=9,$minor=0,$addl="Macintosh"], $unparsed_version=""],
 	["Java1.3.1_04"] =
-		[$name="Java", $version=[$major=1,$minor=3,$minor2=1,$addl="04"], $host=host, $ts=ts],
+		[$name="Java", $version=[$major=1,$minor=3,$minor2=1,$addl="04"], $unparsed_version=""],
 	["Mozilla/5.0 (Linux; U; Android 2.3.3; zh-tw; HTC Pyramid Build/GRI40) AppleWebKit/533.1 (KHTML, like Gecko) Version/4.0 Mobile Safari/533.1"] = 
-		[$name="Safari", $version=[$major=4,$minor=0,$addl="Mobile"], $host=host, $ts=ts],
+		[$name="Safari", $version=[$major=4,$minor=0,$addl="Mobile"], $unparsed_version=""],
 	["Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_6_6; en-us) AppleWebKit/533.20.25 (KHTML, like Gecko) Version/5.0.4 Safari/533.20.27"] =
-		[$name="Safari", $version=[$major=5,$minor=0,$minor2=4], $host=host, $ts=ts],
+		[$name="Safari", $version=[$major=5,$minor=0,$minor2=4], $unparsed_version=""],
 	["Mozilla/5.0 (iPod; U; CPU iPhone OS 4_0 like Mac OS X; en-us) AppleWebKit/532.9 (KHTML, like Gecko) Version/4.0.5 Mobile/8A293 Safari/6531.22.7"] = 
-		[$name="Safari", $version=[$major=4,$minor=0,$minor2=5,$addl="Mobile"], $host=host, $ts=ts],
+		[$name="Safari", $version=[$major=4,$minor=0,$minor2=5,$addl="Mobile"], $unparsed_version=""],
 	["Opera/9.80 (J2ME/MIDP; Opera Mini/9.80 (S60; SymbOS; Opera Mobi/23.348; U; en) Presto/2.5.25 Version/10.54"] = 
-		[$name="Opera Mini", $version=[$major=10,$minor=54], $host=host, $ts=ts],
+		[$name="Opera Mini", $version=[$major=10,$minor=54], $unparsed_version=""],
 	["Opera/9.80 (J2ME/MIDP; Opera Mini/5.0.18741/18.794; U; en) Presto/2.4.15"] =
-		[$name="Opera Mini", $version=[$major=5,$minor=0,$minor2=18741], $host=host, $ts=ts],
+		[$name="Opera Mini", $version=[$major=5,$minor=0,$minor2=18741], $unparsed_version=""],
 	["Opera/9.80 (Windows NT 5.1; Opera Mobi/49; U; en) Presto/2.4.18 Version/10.00"] =
-		[$name="Opera Mobi", $version=[$major=10,$minor=0], $host=host, $ts=ts],
+		[$name="Opera Mobi", $version=[$major=10,$minor=0], $unparsed_version=""],
 	["Mozilla/4.0 (compatible; MSIE 8.0; Android 2.2.2; Linux; Opera Mobi/ADR-1103311355; en) Opera 11.00"] =
-		[$name="Opera", $version=[$major=11,$minor=0], $host=host, $ts=ts],
+		[$name="Opera", $version=[$major=11,$minor=0], $unparsed_version=""],
 	["Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.7.2) Gecko/20040804 Netscape/7.2 (ax)"] =
-		[$name="Netscape", $version=[$major=7,$minor=2], $host=host, $ts=ts],
+		[$name="Netscape", $version=[$major=7,$minor=2], $unparsed_version=""],
 	["Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.0; GTB5; SLCC1; .NET CLR 2.0.50727; Media Center PC 5.0; .NET CLR 3.0.04506; InfoPath.2)"] =
-		[$name="MSIE", $version=[$major=7,$minor=0], $host=host, $ts=ts],
+		[$name="MSIE", $version=[$major=7,$minor=0], $unparsed_version=""],
 	["Mozilla/4.0 (compatible; MSIE 7.0b; Windows NT 5.1; Media Center PC 3.0; .NET CLR 1.0.3705; .NET CLR 1.1.4322; .NET CLR 2.0.50727; InfoPath.1)"] =
-		[$name="MSIE", $version=[$major=7,$minor=0,$addl="b"], $host=host, $ts=ts],
+		[$name="MSIE", $version=[$major=7,$minor=0,$addl="b"], $unparsed_version=""],
 	["Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.1; Trident/4.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC 6.0; Tablet PC 2.0; InfoPath.2; InfoPath.3)"] =
-		[$name="MSIE", $version=[$major=8,$minor=0], $host=host, $ts=ts],
+		[$name="MSIE", $version=[$major=8,$minor=0], $unparsed_version=""],
 	["Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; WOW64; Trident/5.0)"] =
-		[$name="MSIE", $version=[$major=9,$minor=0], $host=host, $ts=ts],
+		[$name="MSIE", $version=[$major=9,$minor=0], $unparsed_version=""],
 	["Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.1; WOW64; Trident/5.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC 6.0; .NET4.0C; .NET4.0E; InfoPath.3; Creative AutoUpdate v1.40.02)"] =
-		[$name="MSIE", $version=[$major=9,$minor=0], $host=host, $ts=ts],
+		[$name="MSIE", $version=[$major=9,$minor=0], $unparsed_version=""],
 	["Mozilla/5.0 (compatible; MSIE 10.0; Windows NT 6.1; WOW64; Trident/6.0)"] =
-		[$name="MSIE", $version=[$major=10,$minor=0], $host=host, $ts=ts],
+		[$name="MSIE", $version=[$major=10,$minor=0], $unparsed_version=""],
 	["The Bat! (3.0.1 RC3) Professional"] =
-		[$name="The Bat!", $version=[$major=3,$minor=0,$minor2=1,$addl="RC3"], $host=host, $ts=ts],
+		[$name="The Bat!", $version=[$major=3,$minor=0,$minor2=1,$addl="RC3"], $unparsed_version=""],
 	# This is an FTP client (found with CLNT command)
 	["Total Commander"] =
-		[$name="Total Commander", $version=[], $host=host, $ts=ts],
+		[$name="Total Commander", $version=[], $unparsed_version=""],
 	["(vsFTPd 2.0.5)"] =
-		[$name="vsFTPd", $version=[$major=2,$minor=0,$minor2=5], $host=host, $ts=ts],
+		[$name="vsFTPd", $version=[$major=2,$minor=0,$minor2=5], $unparsed_version=""],
 	["Apple Mail (2.1084)"] = 
-		[$name="Apple Mail", $version=[$major=2,$minor=1084], $host=host, $ts=ts],
+		[$name="Apple Mail", $version=[$major=2,$minor=1084], $unparsed_version=""],
 };
 
 event bro_init()
 	{
 	for ( sw in matched_software )
 		{
-		local output = Software::parse(sw, host, Software::UNKNOWN);
+		local output = Software::parse(sw);
-		local baseline: Software::Info;
+		local baseline = matched_software[sw];
-		baseline = matched_software[sw];
+		
 		if ( baseline$name == output$name &&
+		     sw == output$unparsed_version &&
 		     Software::cmp_versions(baseline$version,output$version) == 0 )
 			print fmt("success on: %s", sw);
 		else

testing/external/subdir-btest.cfg

@@ -17,3 +17,4 @@ TRACES=%(testbase)s/Traces
 SCRIPTS=%(testbase)s/../scripts
 DIST=%(testbase)s/../../..
 BUILD=%(testbase)s/../../../build
+BRO_PROFILER_FILE=%(testbase)s/.tmp/script-coverage

testing/scripts/btest-bg-run

@@ -0,0 +1,7 @@
+#! /usr/bin/env bash
+
+# This is a wrapper script to btest's real btest-bg-run.  It's used
+# when collecting Bro script coverage statistics so that two independent
+# Bro processing don't try to write those usage statistics to the same file.
+
+BRO_PROFILER_FILE=`mktemp $TMPDIR/script-coverage.XXXX` $BTEST_PATH/btest-bg-run $@

testing/scripts/coverage-calc

@@ -0,0 +1,59 @@
+#! /usr/bin/env python
+
+# This script aggregates many files containing Bro script coverage information
+# into a single file and reports the overall coverage information. Usage:
+#
+#   coverage-calc <quoted glob of filenames> <output file> <script dir>
+#
+# The last argument is used to point to a root directory containing all
+# the Bro distribution's scripts.  It's used to cull out test scripts
+# that are not part of the distribution and which should not count towards
+# the coverage calculation.
+
+import os
+import sys
+import glob
+
+stats = {}
+inputglob = sys.argv[1]
+outputfile = sys.argv[2]
+scriptdir = os.path.abspath(sys.argv[3])
+
+for filename in glob.glob(inputglob):
+    with open(filename, 'r') as f:
+        for line in f.read().splitlines():
+            parts = line.split("\t")
+            exec_count = int(parts[0])
+            # grab file path and line numbers separately
+            filepath, srclines = parts[1].rsplit(",", 1)
+            filepath = os.path.normpath(filepath)
+            # ignore scripts that don't appear to be part of Bro distribution
+            if not filepath.startswith(scriptdir):
+                continue
+            # keep only the line number (or line number range)
+            srclines = srclines.split()[1]
+            # For sorting purposes (so that line numbers get sorted correctly),
+            # construct a specially-formatted key string.
+            sortkey = filepath + ", line " + ("%6s" % srclines.split("-")[0])
+            location = filepath + ", line " + srclines
+            desc = parts[2]
+            # Keying by location + desc may result in duplicate data
+            # as some descs change as a result of differing configurations
+            # producing record (re)definitions
+            key = location
+            if key in stats:
+                stats[key][0] += exec_count
+            else:
+                stats[key] = [exec_count, location, desc, sortkey]
+
+with open(outputfile, 'w') as f:
+    for k in sorted(stats, key=lambda i: stats[i][3]):
+        f.write("%s\t%s\t%s\n" % (stats[k][0], stats[k][1], stats[k][2]))
+
+num_covered = 0
+for k in stats:
+    if stats[k][0] > 0:
+        num_covered += 1
+
+if len(stats) > 0:
+    print "%s/%s (%.1f%%) Bro script statements covered." % (num_covered, len(stats), float(num_covered)/len(stats)*100)