Merge branch 'topic/bernhard/input-logging-commmon-functions' into topic/bernhard/sqlite

CHANGES

@@ -1,4 +1,232 @@
 
+2.1-271 | 2013-01-08 10:18:57 -0800
+
+  * Change substring index notation to use a colon.  String slice
+    notation is now written as `s[1:2]`. Addresses #422. (Jon Siwek)
+
+2.1-268 | 2013-01-07 09:43:44 -0800
+
+  * Fix memory leak in OpaqueType::DoUnserialize. (Jon Siwek)
+
+2.1-265 | 2012-12-20 17:38:42 -0800
+
+  * Add array-style index accessor for strings.  Addresses #422. (Jon
+    Siwek)
+
+    The index expression can take up to two indices for the start and
+    end index of the substring to return (e.g. "mystring[1,3]").
+    Negative indices are allowed, with -1 representing the last
+    character in the string.  The indexing is not cyclic -- if the
+    starting index is >= the length of the string an empty string is
+    returned, and if the ending index is >= the length of the string
+    then it's interpreted as the last index of the string.  Assigning
+    to substrings accessed like this isn't allowed.
+
+2.1-263 | 2012-12-20 16:22:09 -0800
+
+  * Bro's language now has a new set of types "opaque of X". (Matthias
+    Vallentin)
+
+    Opaque values can be passed around like other values but they can
+    only be manipulated with BiF functions, not with other operators.
+    Currently, the following opaque types are supported:
+
+        - opaque of md5
+        - opaque of sha1
+        - opaque of sha256
+        - opaquey of entropy.
+
+    They go along with the corrsponding BiF functions md5_*, sha1_*,
+    sha256_*, and entropy_*, respectively. Note that these functions
+    have changed their signatures to work with opaques types rather
+    than global state as it was before.
+    
+2.1-240 | 2012-12-20 15:21:07 -0800
+
+  * Improve error for invalid use of types as values. Addresses #923.
+    (Jon Siwek)
+
+2.1-238 | 2012-12-20 15:11:25 -0800
+
+  * Finish implementation of script-layer switch statement. Addresses
+    #754. (Jon Siwek)
+
+    They behave like C-style switches except case labels can be
+    comprised of multiple literal constants delimited by commas.  Only
+    atomic types are allowed for now.  Case label bodies that don't
+    execute a "return" or "break" statement will fall through to
+    subsequent cases.  A default case label is allowed.
+
+  * Fix a case where c$resp$size is misrepresented.  Addresses #730.
+    (Jon Siwek)
+
+2.1-234 | 2012-12-20 12:12:19 -0800
+
+  * Fix return value of hook calls that have no handlers. For this
+    case, the return value is always true. (Jon Siwek)
+
+  * Fix to_port() BIF for port strings with a port number of zero.
+    (Jon Siwek)
+
+2.1-231 | 2012-12-14 14:51:35 -0800
+
+  * Make const variables actually constant. Both local and global
+    variables declared with "const" could be modified, but now
+    expressions that would modify them generate an error message at
+    parse-time. Addresses #922. (Jon Siwek)
+
+2.1-229 | 2012-12-14 14:46:12 -0800
+
+  * Fix memory leak in ASCII reader when encoutering errors in input.
+    (Bernhard Amann)
+
+  * Improvements for the "bad checksums" detector to make it detect
+    bad TCP checksums. (Seth Hall)
+
+2.1-223 | 2012-12-12 14:25:15 -0800
+
+  * Trick for parallelizing input framework unit tests. Instead of
+    loading listen.bro to block until files are read, just read a pcap
+    file in pseudo-realtime. (Jon Siwek)
+
+  * Fix reliability of a unit test that relies on when statements.
+    (Jon Siwek)
+
+  * Remove unused attributes. (Daniel Thayer)
+        - Removed attributes &postprocessor and &match from documentation and source code.
+        - Removed undocumented attribute &attr from source code.
+        - Removed internal attribute "(&tracked)" from documentation.
+
+2.1-218 | 2012-12-10 14:45:04 -0800
+
+  * Add GPRS Tunnelling Protocol (GTPv1) decapsulation. This currently
+    supports automatic decapsulation of GTP-U packets on UDP port 2152.
+    The GTPv1 headers for such tunnels can be inspected by handling
+    the "gtpv1_g_pdu_packet" event, which has a parameter of type
+    "gtpv1_hdr". Addresses #690. (Jon Siwek; derived from patch by
+    Carsten Langer)
+
+  * Change BinPAC exceptions in AYIYA/GTP analyzers to do
+    "protocol_violation". (Jon Siwek)
+
+2.1-212 | 2012-12-07 19:42:03 -0800
+
+  * Changing the HTTP parser to accept request methods in alignment
+    with the RFC. (Robin Sommer)
+
+2.1-209 | 2012-12-05 16:44:04 -0800
+
+  * Adapting the HTTP request line parsing to only accept methods
+    consisting of letters [A-Za-z]. (Robin Sommer)
+
+2.1-207 | 2012-12-05 15:47:32 -0800
+
+  * Reporting warnings if kill/waitpid fail in communication system.
+    (Bill Parker)
+
+  * Replace() bzero with memset(). (Bill Parker)
+
+  * Merge remote-tracking branch 'vlad/topic/vladg/http-verbs'
+
+    * vlad/topic/vladg/http-verbs:
+      A test for HTTP methods, including some horribly illegal requests.
+      Remove hardcoded HTTP verbs from the analyzer (#741)
+
+    I added a "bad_HTTP_request" weird for HTTP request lines that don't
+    have more than a single word.
+
+    Closes #741. (Robin Sommer)
+
+  * A test for HTTP methods, including some horribly illegal requests. (Vlad Grigorescu)
+
+  * Remove hardcoded HTTP verbs from the analyzer (#741) (Vlad Grigorescu)
+
+
+2.1-203 | 2012-12-05 14:36:56 -0800
+
+  * Fix segfault: Synchronization of state between connecting peers
+    now skips over identifiers that aren't initialized with a value
+    yet. Addresses #66. (Jon Siwek)
+
+  * Fix segfault: Delete correct entry in error case in input
+    framework. (Bernhard Amann)
+
+  * Bad record constructor initializers now give an error. Addresses
+    #34. (Jon Siwek)
+
+  * Invalid vector indices now generate error message. Addresses #24.
+    (Jon Siwek)
+
+  * Bump CPack RPM package requirement to Python >= 2.6.0. (Jon Siwek)
+
+  * Interpreter exceptions occurring in "when" blocks are now handled.
+    Addresses #779 (Jon Siwek)
+
+2.1-195 | 2012-12-03 14:50:33 -0800
+
+  * Catching out-of-memory in patricia tree code. (Bill Parker)
+    
+2.1-194 | 2012-12-03 14:36:26 -0800
+
+  * Renaming ASCII writer filter option 'only_single_header_row' to
+    'tsv'. Also clarifying usage. Closes #912. (Robin Sommer)
+
+2.1-193 | 2012-12-03 14:11:14 -0800
+
+  * Fix a set of bugs with table/set attributes. (Jon Siwek)
+
+    - Identifiers that are initialized with set()/table() constructor
+      expressions now inherit attributes from the expression.  Before,
+      statements like
+
+         const i: set[string] = set() &redef;
+
+      associated the attribute with the set() constructor, but not the
+      "i" identifier, preventing redefinition.  Addresses #866.
+
+    - Allow &default attribute to apply to tables initialized as empty
+      (via either "{ }" or "table()") or if the expression supplied to it
+      can evaluate to a type that's promotable to the same yield type as
+      the table.
+
+2.1-191 | 2012-12-03 14:08:56 -0800
+
+  * Add test of record() constructor to table initializer unit test.
+    (Jon Siwek)
+
+  * Fix table(), set(), vector() constructors in table initializer
+    lists. Also adds type checking of yield values to table()
+    constructor and fixes the type checking of yield values in
+    vector() constructor. Addresses #5. (Jon Siwek)
+
+2.1-188 | 2012-12-03 14:04:29 -0800
+
+  * Hook functions now callable with "hook" expression (i.e., hook is
+    no longer a statement). The return value of the call is an
+    implicit boolean value of T if all hook handlers ran, or F if one
+    hook handler exited as a result of a break statement and
+    potentially prevented other handlers from running.
+
+    Scripts don't need to declare hooks with an explicit return type of bool
+    (internally, that's assumed), and any values given to (optional) return
+    statements in handler definitions are just ignored.
+
+    Addresses #918. (Jon Siwek)
+
+  * Clarification in hook documentation. (Jon Siwek)
+
+2.1-184 | 2012-12-03 13:59:50 -0800
+
+  * Slightly fix up file name extraction from Content-Disposition
+    headers. (Seth Hall)
+
+  * Adding -b flag to bro in unit tests so they run faster.
+
+  * Fixed a DNS attribute issue. Reported by Matt Thompson. (Seth
+    Hall)
+
+  * Adding NEWS placeholder for hooks and CSV mode. (Robin Sommer)
+    
 2.1-178 | 2012-11-23 19:35:32 -0800
 
   * The ASCII writer now supports a new filter config option

CMakeLists.txt

@@ -195,7 +195,7 @@ CheckOptionalBuildSources(aux/broccoli Broccoli INSTALL_BROCCOLI)
 
 if (INSTALL_BROCTL)
     # CPack RPM Generator may not automatically detect this
-    set(CPACK_RPM_PACKAGE_REQUIRES "python >= 2.4.0")
+    set(CPACK_RPM_PACKAGE_REQUIRES "python >= 2.6.0")
 endif ()
 
 # If this CMake project is a sub-project of another, we will not

NEWS

@@ -13,6 +13,8 @@ Bro 2.2
 New Functionality
 ~~~~~~~~~~~~~~~~~
 
+- GPRS Tunnelling Protocol (GTPv1) decapsulation.
+
 - GridFTP support. TODO: Extend.
 
 - ssl.log now also records the subject client and issuer certificates.
@@ -22,6 +24,33 @@ New Functionality
 
 - The ASCII writer can now output CSV files on a per filter basis.
 
+- Bro's language now has a working "switch" statement that generally
+  behaves like C-style switches except case labels can be comprised of
+  multiple literal constants delimited by commas.  Only atomic types
+  are allowed for now.  Case label bodies that don't execute a
+  "return" or "break" statement will fall through to subsequent cases.
+  A default case label is allowed.
+
+- Bro's language now has a new set of types "opaque of X". Opaque
+  values can be passed around like other values but they can only be
+  manipulated with BiF functions, not with other operators. Currently,
+  the following opaque types are supported:
+
+        - opaque of md5
+        - opaque of sha1
+        - opaque of sha256
+        - opaquey of entropy.
+
+  They go along with the corrsponding BiF functions md5_*, sha1_*,
+  sha256_*, and entropy_*, respectively. Note that these functions
+  have changed their signatures to work with opaques types rather
+  than global state as it was before.
+
+- Strings now support the subscript operator to extract individual
+  characters and substrings (e.g., s[4], s[1,5]). The index expression
+  can take up to two indices for the start and end index of the
+  substring to return (e.g. "mystring[1,3]").
+
 Changed Functionality
 ~~~~~~~~~~~~~~~~~~~~~
 
@@ -35,6 +64,9 @@ Changed Functionality
           make_connection_persistent(), generate_idmef(),
           split_complete()
 
+        - md5_*, sha1_*, sha256_*, and entropy_* have all changed
+        their signatures to work with opaque types (see above).
+
 - Removed a now unused argument from "do_split" helper function.
 
 - "this" is no longer a reserved keyword.

VERSION

@@ -1 +1 @@
-2.1-179
+2.1-271

aux/broccoli

@@ -1 +1 @@
-Subproject commit a8846fc5b004ffe4e3d00e826d0077ba19518192
+Subproject commit 073404dd29dc6e90ff0e4eb8bc836f8adbf3931e

aux/broctl

@@ -1 +1 @@
-Subproject commit 834131cd0ec0f63cce9de818726fe6167dedbf34
+Subproject commit 2bf6b37177b895329173acac2bb98f38a8783bc1

aux/btest

@@ -1 +1 @@
-Subproject commit d83e10c5f76cbfdf81c843575351fbc7b544fc93
+Subproject commit 3585dc9a7afe20d70cb77fc2472cc6bce3850b67

doc/scripts/builtins.rst

@@ -505,15 +505,14 @@ The Bro scripting language supports the following built-in types.
     A hook is another flavor of function that shares characteristics of
     both a :bro:type:`function` and a :bro:type:`event`.  They are like
     events in that many handler bodies can be defined for the same hook
-    identifier, they have no return vale, and the order of execution
-    can be enforced with :bro:attr:`&priority`.  They are more like
-    functions in the way they are invoked/called, because, unlike
-    events, their execution is immediate and they do not get scheduled
-    through an event queue.  Also, a unique feature of a hook is that
-    a given hook handler body can short-circuit the execution of
-    remaining hook handlers simply by exiting from the body as a result
-    of a ``break`` statement (as opposed to a ``return`` or just
-    reaching the end of the body).
+    identifier and the order of execution can be enforced with
+    :bro:attr:`&priority`.  They are more like functions in the way they
+    are invoked/called, because, unlike events, their execution is
+    immediate and they do not get scheduled through an event queue.
+    Also, a unique feature of a hook is that a given hook handler body
+    can short-circuit the execution of remaining hook handlers simply by
+    exiting from the body as a result of a ``break`` statement (as
+    opposed to a ``return`` or just reaching the end of the body).
 
     A hook type is declared like::
 
@@ -549,18 +548,26 @@ The Bro scripting language supports the following built-in types.
             print "not going to happen", s;
             }
 
-    Note that, although the first (forward) declaration of ``myhook`` as
-    a hook type isn't strictly required, when it is provided, the
-    argument types must match.
+    Note that the first (forward) declaration of ``myhook`` as a hook
+    type isn't strictly required.  Argument types must match for all
+    hook handlers and any forward declaration of a given hook.
 
-    To invoke immediate execution of all hook handler bodies, a ``hook``
-    statement must be used:
+    To invoke immediate execution of all hook handler bodies, they
+    are called similarly to a function, except preceded by the ``hook``
+    keyword:
 
     .. code:: bro
 
         hook myhook("hi");
 
-    And the output would like like::
+    or
+
+    .. code:: bro
+
+        if ( hook myhook("hi") )
+            print "all handlers ran";
+
+    And the output would look like::
 
         priority 10 myhook handler, hi
         break out of myhook handling, bye
@@ -568,6 +575,12 @@ The Bro scripting language supports the following built-in types.
     Note how the modification to arguments can be seen by remaining
     hook handlers.
 
+    The return value of a hook call is an implicit :bro:type:`bool`
+    value with ``T`` meaning that all handlers for the hook were
+    executed and ``F`` meaning that only some of the handlers may have
+    executed due to one handler body exiting as a result of a ``break``
+    statement.
+
 Attributes
 ----------
 
@@ -650,20 +663,12 @@ scripting language supports the following built-in attributes.
     ``&synchronized`` variable is automatically propagated to all peers
     when it changes.
 
-.. bro:attr:: &postprocessor
-
-.. TODO: needs to be documented.
-
 .. bro:attr:: &encrypt
 
     Encrypts files right before writing them to disk.
 
 .. TODO: needs to be documented in more detail.
 
-.. bro:attr:: &match
-
-.. TODO: needs to be documented.
-
 .. bro:attr:: &raw_output
 
     Opens a file in raw mode, i.e., non-ASCII characters are not
@@ -697,6 +702,3 @@ scripting language supports the following built-in attributes.
 
 .. TODO: needs documented
 
-.. bro:attr:: (&tracked)
-
-.. TODO: needs documented or removed if it's not used anywhere.

scripts/base/frameworks/logging/writers/ascii.bro

@@ -2,11 +2,14 @@
 ##! to tweak the output format of ASCII logs.
 ##!
 ##! The ASCII writer supports currently one writer-specific filter option via
-##! ``config``: setting ``only_single_header_row`` to ``T`` turns the output into
-##! into CSV mode where only a single header row with the column names is printed
-##! out as meta information. Example filter using this::
+##! ``config``: setting ``tsv`` to the string ``T`` turns the output into into
+##! "tab-separated-value" mode where only a single header row with the column names
+##! is printed out as meta information, with no "# fields" prepended; no other meta
+##! data gets included in that mode.   
 ##! 
-##!    local my_filter: Log::Filter = [$name = "my-filter", $writer = Log::WRITER_ASCII, $config = table(["only_single_header_row"] = "T")]; 
+##! Example filter using this::
+##! 
+##!    local my_filter: Log::Filter = [$name = "my-filter", $writer = Log::WRITER_ASCII, $config = table(["tsv"] = "T")]; 
 ##!
 
 module LogAscii;

scripts/base/frameworks/notice/cluster.bro

@@ -21,12 +21,10 @@ redef Cluster::manager2worker_events += /Notice::begin_suppression/;
 redef Cluster::worker2manager_events += /Notice::cluster_notice/;
 
 @if ( Cluster::local_node_type() != Cluster::MANAGER )
+
 # The notice policy is completely handled by the manager and shouldn't be 
 # done by workers or proxies to save time for packet processing.
-event bro_init() &priority=11
-	{
-	Notice::policy = table();
-	}
+redef Notice::policy = table();
 
 event Notice::begin_suppression(n: Notice::Info)
 	{

scripts/base/frameworks/tunnels/main.bro

@@ -88,7 +88,10 @@ redef dpd_config += { [ANALYZER_AYIYA] = [$ports = ayiya_ports] };
 const teredo_ports = { 3544/udp };
 redef dpd_config += { [ANALYZER_TEREDO] = [$ports = teredo_ports] };
 
-redef likely_server_ports += { ayiya_ports, teredo_ports };
+const gtpv1u_ports = { 2152/udp };
+redef dpd_config += { [ANALYZER_GTPV1] = [$ports = gtpv1u_ports] };
+
+redef likely_server_ports += { ayiya_ports, teredo_ports, gtpv1u_ports };
 
 event bro_init() &priority=5
 	{

scripts/base/init-bare.bro

@@ -1450,6 +1450,44 @@ type teredo_hdr: record {
 	hdr:    pkt_hdr;                 ##< IPv6 and transport protocol headers.
 };
 
+## A GTPv1 (GPRS Tunneling Protocol) header.
+type gtpv1_hdr: record {
+	## The 3-bit version field, which for GTPv1 should be 1.
+	version:   count;
+	## Protocol Type value differentiates GTP (value 1) from GTP' (value 0).
+	pt_flag:   bool;
+	## Reserved field, should be 0.
+	rsv:       bool;
+	## Extension Header flag.  When 0, the *next_type* field may or may not
+	## be present, but shouldn't be meaningful.  When 1, *next_type* is
+	## present and meaningful.
+	e_flag:    bool;
+	## Sequence Number flag.  When 0, the *seq* field may or may not
+	## be present, but shouldn't be meaningful.  When 1, *seq* is
+	## present and meaningful.
+	s_flag:    bool;
+	## N-PDU flag.  When 0, the *n_pdu* field may or may not
+	## be present, but shouldn't be meaningful.  When 1, *n_pdu* is
+	## present and meaningful.
+	pn_flag:   bool;
+	## Message Type.  A value of 255 indicates user-plane data is encapsulated.
+	msg_type:  count;
+	## Length of the GTP packet payload (the rest of the packet following the
+	## mandatory 8-byte GTP header).
+	length:    count;
+	## Tunnel Endpoint Identifier.  Unambiguously identifies a tunnel endpoint
+	## in receiving GTP-U or GTP-C protocol entity.
+	teid:      count;
+	## Sequence Number.  Set if any *e_flag*, *s_flag*, or *pn_flag* field is
+	## set.
+	seq:       count &optional;
+	## N-PDU Number.  Set if any *e_flag*, *s_flag*, or *pn_flag* field is set.
+	n_pdu:     count &optional;
+	## Next Extension Header Type.  Set if any *e_flag*, *s_flag*, or *pn_flag*
+	## field is set.
+	next_type: count &optional;
+};
+
 ## Definition of "secondary filters". A secondary filter is a BPF filter given as
 ## index in this table. For each such filter, the corresponding event is raised for
 ## all matching packets.
@@ -2786,6 +2824,9 @@ export {
 	## Toggle whether to do IPv6-in-Teredo decapsulation.
 	const enable_teredo = T &redef;
 
+	## Toggle whether to do GTPv1 decapsulation.
+	const enable_gtpv1 = T &redef;
+
 	## With this option set, the Teredo analysis will first check to see if
 	## other protocol analyzers have confirmed that they think they're
 	## parsing the right protocol and only continue with Teredo tunnel
@@ -2802,6 +2843,15 @@ export {
 	## :bro:see:`Tunnel::yielding_teredo_decapsulation`.
 	const delay_teredo_confirmation = T &redef;
 
+	## With this set, the GTP analyzer waits until the most-recent upflow
+	## and downflow packets are a valid GTPv1 encapsulation before
+	## issuing :bro:see:`protocol_confirmation`.  If it's false, the
+	## first occurence of a packet with valid GTPv1 encapsulation causes
+	## confirmation.  Since the same inner connection can be carried
+	## differing outer upflow/downflow connections, setting to false
+	## may work better.
+	const delay_gtp_confirmation = F &redef;
+
 	## How often to cleanup internal state for inactive IP tunnels.
 	const ip_tunnel_timeout = 24hrs &redef;
 } # end export

scripts/base/misc/find-checksum-offloading.bro

@@ -14,7 +14,8 @@ export {
 }
 
 # Keep track of how many bad checksums have been seen.
-global bad_checksums = 0;
+global bad_ip_checksums  = 0;
+global bad_tcp_checksums = 0;
 
 # Track to see if this script is done so that messages aren't created multiple times.
 global done = F;
@@ -25,10 +26,19 @@ event ChecksumOffloading::check()
 		return;
 
 	local pkts_recvd = net_stats()$pkts_recvd;
-	if ( (bad_checksums*1.0 / net_stats()$pkts_recvd*1.0) > 0.05 )
+	local bad_ip_checksum_pct = (pkts_recvd != 0) ? (bad_ip_checksums*1.0 / pkts_recvd*1.0) : 0;
+	local bad_tcp_checksum_pct = (pkts_recvd != 0) ? (bad_tcp_checksums*1.0 / pkts_recvd*1.0) : 0;
+	if ( bad_ip_checksum_pct > 0.05 || bad_tcp_checksum_pct > 0.05 )
 		{
 		local packet_src = reading_traces() ? "trace file likely has" : "interface is likely receiving";
-		local message = fmt("Your %s invalid IP checksums, most likely from NIC checksum offloading.", packet_src);
+		local bad_checksum_msg = (bad_ip_checksum_pct > 0.0) ? "IP" : "";
+		if ( bad_tcp_checksum_pct > 0.0 )
+			{
+			if ( |bad_checksum_msg| > 0 )
+				bad_checksum_msg += " and ";
+			bad_checksum_msg += "TCP";
+			}
+		local message = fmt("Your %s invalid %s checksums, most likely from NIC checksum offloading.", packet_src, bad_checksum_msg);
 		Reporter::warning(message);
 		done = T;
 		}
@@ -48,7 +58,13 @@ event bro_init()
 event net_weird(name: string)
 	{
 	if ( name == "bad_IP_checksum" )
-		++bad_checksums;
+		++bad_ip_checksums;
+	}
+
+event conn_weird(name: string, c: connection, addl: string)
+	{
+	if ( name == "bad_TCP_checksum" )
+		++bad_tcp_checksums;
 	}
 
 event bro_done()

scripts/base/protocols/dns/main.bro

@@ -67,7 +67,7 @@ export {
 		ready:         bool            &default=F;
 		## The total number of resource records in a reply message's answer
 		## section.
-		total_answers: count           &default=0;
+		total_answers: count           &optional;
 		## The total number of resource records in a reply message's answer,
 		## authority, and additional sections.
 		total_replies: count           &optional;
@@ -231,6 +231,7 @@ event DNS::do_reply(c: connection, msg: dns_msg, ans: dns_answer, reply: string)
 		Log::write(DNS::LOG, c$dns);
 		# This record is logged and no longer pending.
 		delete c$dns_state$pending[c$dns$trans_id];
+		delete c$dns;
 		}
 	}
 

scripts/base/protocols/http/file-hash.bro

@@ -13,16 +13,16 @@ export {
 	redef record Info += {
 		## MD5 sum for a file transferred over HTTP calculated from the 
 		## response body.
-		md5:             string   &log &optional;
+		md5:             string     &log &optional;
 		
 		## This value can be set per-transfer to determine per request
 		## if a file should have an MD5 sum generated.  It must be
 		## set to T at the time of or before the first chunk of body data.
-		calc_md5:        bool     &default=F;
+		calc_md5:        bool       &default=F;
 		
 		## Indicates if an MD5 sum is being calculated for the current 
 		## request/response pair.
-		calculating_md5: bool     &default=F;
+		md5_handle: opaque of md5   &optional;
 	};
 	
 	## Generate MD5 sums for these filetypes.
@@ -41,13 +41,12 @@ event http_entity_data(c: connection, is_orig: bool, length: count, data: string
 		if ( c$http$calc_md5 || 
 		     (c$http?$mime_type && generate_md5 in c$http$mime_type) )
 			{
-			c$http$calculating_md5 = T;
-			md5_hash_init(c$id);
+			c$http$md5_handle = md5_hash_init();
 			}
 		}
 	
-	if ( c$http$calculating_md5 )
-		md5_hash_update(c$id, data);
+	if ( c$http?$md5_handle )
+		md5_hash_update(c$http$md5_handle, data);
 	}
 	
 ## In the event of a content gap during a file transfer, detect the state for
@@ -55,11 +54,11 @@ event http_entity_data(c: connection, is_orig: bool, length: count, data: string
 ## incorrect anyway.
 event content_gap(c: connection, is_orig: bool, seq: count, length: count) &priority=5
 	{
-	if ( is_orig || ! c?$http || ! c$http$calculating_md5 ) return;
+	if ( is_orig || ! c?$http || ! c$http?$md5_handle ) return;
 	
 	set_state(c, F, is_orig);
-	c$http$calculating_md5 = F;
-	md5_hash_finish(c$id);
+	md5_hash_finish(c$http$md5_handle); # Ignore return value.
+	delete c$http$md5_handle;
 	}
 
 ## When the file finishes downloading, finish the hash and generate a notice.
@@ -67,11 +66,11 @@ event http_message_done(c: connection, is_orig: bool, stat: http_message_stat) &
 	{
 	if ( is_orig || ! c?$http ) return;
 	
-	if ( c$http$calculating_md5 )
+	if ( c$http?$md5_handle )
 		{
 		local url = build_url_http(c$http);
-		c$http$calculating_md5 = F;
-		c$http$md5 = md5_hash_finish(c$id);
+		c$http$md5 = md5_hash_finish(c$http$md5_handle);
+		delete c$http$md5_handle;
 		
 		NOTICE([$note=MD5, $msg=fmt("%s %s %s", c$id$orig_h, c$http$md5, url),
 		        $sub=c$http$md5, $conn=c, $URL=url]);
@@ -82,11 +81,12 @@ event connection_state_remove(c: connection) &priority=-5
 	{
 	if ( c?$http_state && 
 	     c$http_state$current_response in c$http_state$pending &&
-	     c$http_state$pending[c$http_state$current_response]$calculating_md5 )
+	     c$http_state$pending[c$http_state$current_response]?$md5_handle )
 		{
 		# The MD5 sum isn't going to be saved anywhere since the entire 
 		# body wouldn't have been seen anyway and we'd just be giving an
 		# incorrect MD5 sum.
-		md5_hash_finish(c$id);
+		md5_hash_finish(c$http$md5_handle);
+		delete c$http$md5_handle;
 		}
 	}

scripts/base/protocols/http/main.bro

@@ -95,6 +95,19 @@ export {
 		"PROXY-CONNECTION",
 	} &redef;
 
+	## A list of HTTP methods. Other methods will generate a weird. Note
+        ## that the HTTP analyzer will only accept methods consisting solely
+        ## of letters ``[A-Za-z]``.
+	const http_methods: set[string] = {
+		"GET", "POST", "HEAD", "OPTIONS",
+		"PUT", "DELETE", "TRACE", "CONNECT",
+		# HTTP methods for distributed authoring:
+		"PROPFIND", "PROPPATCH", "MKCOL",
+		"COPY", "MOVE", "LOCK", "UNLOCK",
+		"POLL", "REPORT", "SUBSCRIBE", "BMOVE",
+		"SEARCH"
+	} &redef;
+	
 	## Event that can be handled to access the HTTP record as it is sent on 
 	## to the logging framework.
 	global log_http: event(rec: Info);
@@ -180,6 +193,9 @@ event http_request(c: connection, method: string, original_URI: string,
 	
 	c$http$method = method;
 	c$http$uri = unescaped_URI;
+
+	if ( method !in http_methods )
+		event conn_weird("unknown_HTTP_method", c, method);
 	}
 	
 event http_reply(c: connection, version: string, code: count, reason: string) &priority=5

scripts/base/protocols/smtp/entities.bro

@@ -16,33 +16,33 @@ export {
 
 	type EntityInfo: record {
 		## This is the timestamp of when the MIME content transfer began.
-		ts:               time    &log;
-		uid:              string  &log;
-		id:               conn_id &log;
+		ts:               time            &log;
+		uid:              string          &log;
+		id:               conn_id         &log;
 		## A count to represent the depth of this message transaction in a 
 		## single connection where multiple messages were transferred.
-		trans_depth:      count  &log;
+		trans_depth:      count           &log;
 		## The filename seen in the Content-Disposition header.
-		filename:         string  &log &optional;
+		filename:         string          &log &optional;
 		## Track how many bytes of the MIME encoded file have been seen.
-		content_len:      count   &log &default=0;
+		content_len:      count           &log &default=0;
 		## The mime type of the entity discovered through magic bytes identification.
-		mime_type:        string  &log &optional;
+		mime_type:        string          &log &optional;
 		
 		## The calculated MD5 sum for the MIME entity.
-		md5:              string  &log &optional;
+		md5:              string          &log &optional;
 		## Optionally calculate the file's MD5 sum.  Must be set prior to the 
 		## first data chunk being see in an event.
-		calc_md5:         bool    &default=F;
+		calc_md5:         bool            &default=F;
 		## This boolean value indicates if an MD5 sum is being calculated 
 		## for the current file transfer.
-		calculating_md5:  bool    &default=F;
+		md5_handle:       opaque of md5   &optional;
 		
 		## Optionally write the file to disk.  Must be set prior to first 
 		## data chunk being seen in an event.
-		extract_file:     bool    &default=F;
+		extract_file:     bool            &default=F;
 		## Store the file handle here for the file currently being extracted.
-		extraction_file:  file    &log &optional;
+		extraction_file:  file            &log &optional;
 	};
 
 	redef record Info += {
@@ -126,18 +126,16 @@ event mime_segment_data(c: connection, length: count, data: string) &priority=-5
 
 	if ( c$smtp$current_entity$content_len == 0 )
 		{
-		if ( generate_md5 in c$smtp$current_entity$mime_type && ! never_calc_md5 )
-			c$smtp$current_entity$calc_md5 = T;
+		local entity = c$smtp$current_entity;
+		if ( generate_md5 in entity$mime_type && ! never_calc_md5 )
+			entity$calc_md5 = T;
 
-		if ( c$smtp$current_entity$calc_md5 )
-			{
-			c$smtp$current_entity$calculating_md5 = T;
-			md5_hash_init(c$id);
-			}
+		if ( entity$calc_md5 )
+			entity$md5_handle = md5_hash_init();
 		}
 
-	if ( c$smtp$current_entity$calculating_md5 )
-		md5_hash_update(c$id, data);
+	if ( c$smtp$current_entity?$md5_handle )
+		md5_hash_update(entity$md5_handle, data);
 }
 
 ## In the event of a content gap during the MIME transfer, detect the state for
@@ -147,10 +145,11 @@ event content_gap(c: connection, is_orig: bool, seq: count, length: count) &prio
 	{
 	if ( is_orig || ! c?$smtp || ! c$smtp?$current_entity ) return;
 
-	if ( c$smtp$current_entity$calculating_md5 )
+	local entity = c$smtp$current_entity;
+	if ( entity?$md5_handle )
 		{
-		c$smtp$current_entity$calculating_md5 = F;
-		md5_hash_finish(c$id);
+		md5_hash_finish(entity$md5_handle);
+		delete entity$md5_handle;
 		}
 	}
 
@@ -161,12 +160,14 @@ event mime_end_entity(c: connection) &priority=-3
 	if ( ! c?$smtp || ! c$smtp?$current_entity )
 		return;
 
-	if ( c$smtp$current_entity$calculating_md5 )
+	local entity = c$smtp$current_entity;
+	if ( entity?$md5_handle )
 		{
-		c$smtp$current_entity$md5 = md5_hash_finish(c$id);
+		entity$md5 = md5_hash_finish(entity$md5_handle);
+		delete entity$md5_handle;
 
 		NOTICE([$note=MD5, $msg=fmt("Calculated a hash for a MIME entity from %s", c$id$orig_h),
-				$sub=c$smtp$current_entity$md5, $conn=c]);
+				$sub=entity$md5, $conn=c]);
 		}
 	}
 
@@ -177,6 +178,10 @@ event mime_one_header(c: connection, h: mime_header_rec)
 	if ( h$name == "CONTENT-DISPOSITION" &&
 	     /[fF][iI][lL][eE][nN][aA][mM][eE]/ in h$value )
 		c$smtp$current_entity$filename = extract_filename_from_content_disposition(h$value);
+
+	if ( h$name == "CONTENT-TYPE" && 
+	     /[nN][aA][mM][eE][:blank:]*=/ in h$value )
+		c$smtp$current_entity$filename = extract_filename_from_content_disposition(h$value);
 	}
 
 event mime_end_entity(c: connection) &priority=-5

scripts/base/utils/files.bro

@@ -19,7 +19,7 @@ function generate_extraction_filename(prefix: string, c: connection, suffix: str
 ## the filename.
 function extract_filename_from_content_disposition(data: string): string
 	{
-	local filename = sub(data, /^.*[fF][iI][lL][eE][nN][aA][mM][eE]=/, "");
+	local filename = sub(data, /^.*[nN][aA][mM][eE][[:blank:]]*=[[:blank:]]*/, "");
 	# Remove quotes around the filename if they are there.
 	if ( /^\"/ in filename )
 		filename =  split_n(filename, /\"/, F, 2)[2];

scripts/policy/protocols/http/detect-webapps.bro

@@ -28,7 +28,7 @@ event signature_match(state: signature_state, msg: string, data: string) &priori
 	if ( /^webapp-/ !in state$sig_id ) return;
 	
 	local c = state$conn;
-	local si = Software::Info;
+	local si: Software::Info;
 	si = [$name=msg, $unparsed_version=msg, $host=c$id$resp_h, $host_p=c$id$resp_p, $software_type=WEB_APPLICATION];
 	si$url = build_url_http(c$http);
 	if ( c$id$resp_h in Software::tracked &&

src/AYIYA.cc

@@ -20,5 +20,13 @@ void AYIYA_Analyzer::Done()
 void AYIYA_Analyzer::DeliverPacket(int len, const u_char* data, bool orig, int seq, const IP_Hdr* ip, int caplen)
 	{
 	Analyzer::DeliverPacket(len, data, orig, seq, ip, caplen);
-	interp->NewData(orig, data, data + len);
+
+	try
+		{
+		interp->NewData(orig, data, data + len);
+		}
+	catch ( const binpac::Exception& e )
+		{
+		ProtocolViolation(fmt("Binpac exception: %s", e.c_msg()));
+		}
 	}

src/Analyzer.cc

@@ -41,6 +41,7 @@
 #include "Syslog-binpac.h"
 #include "Teredo.h"
 #include "ConnSizeAnalyzer.h"
+#include "GTPv1.h"
 
 // Keep same order here as in AnalyzerTag definition!
 const Analyzer::Config Analyzer::analyzer_configs[] = {
@@ -143,6 +144,9 @@ const Analyzer::Config Analyzer::analyzer_configs[] = {
 	{ AnalyzerTag::Teredo, "TEREDO",
 		Teredo_Analyzer::InstantiateAnalyzer,
 		Teredo_Analyzer::Available, 0, false },
+	{ AnalyzerTag::GTPv1, "GTPV1",
+		GTPv1_Analyzer::InstantiateAnalyzer,
+		GTPv1_Analyzer::Available, 0, false },
 
 	{ AnalyzerTag::File, "FILE", File_Analyzer::InstantiateAnalyzer,
 		File_Analyzer::Available, 0, false },

src/AnalyzerTags.h

@@ -38,6 +38,7 @@ namespace AnalyzerTag {
 		AYIYA,
 		SOCKS,
 		Teredo,
+		GTPv1,
 
 		// Other
 		File, Backdoor, InterConn, SteppingStone, TCPStats,

src/Attr.cc

@@ -14,8 +14,8 @@ const char* attr_name(attr_tag t)
 		"&rotate_interval", "&rotate_size",
 		"&add_func", "&delete_func", "&expire_func",
 		"&read_expire", "&write_expire", "&create_expire",
-		"&persistent", "&synchronized", "&postprocessor",
-		"&encrypt", "&match",
+		"&persistent", "&synchronized",
+		"&encrypt",
 		"&raw_output", "&mergeable", "&priority",
 		"&group", "&log", "&error_handler", "&type_column",
 		"(&tracked)",
@@ -260,6 +260,11 @@ void Attributes::CheckAttr(Attr* a)
 				// Ok.
 				break;
 
+			if ( type->Tag() == TYPE_TABLE &&
+			     type->AsTableType()->IsUnspecifiedTable() )
+				// Ok.
+				break;
+
 			a->AttrExpr()->Error("&default value has inconsistent type", type);
 			}
 
@@ -290,6 +295,11 @@ void Attributes::CheckAttr(Attr* a)
 					// Ok.
 					break;
 
+				Expr* e = a->AttrExpr();
+				if ( check_and_promote_expr(e, ytype) )
+					// Ok.
+					break;
+
 				Error("&default value has inconsistent type 2");
 				}
 
@@ -327,11 +337,6 @@ void Attributes::CheckAttr(Attr* a)
 			Error("&rotate_size only applicable to files");
 		break;
 
-	case ATTR_POSTPROCESSOR:
-		if ( type->Tag() != TYPE_FILE )
-			Error("&postprocessor only applicable to files");
-		break;
-
 	case ATTR_ENCRYPT:
 		if ( type->Tag() != TYPE_FILE )
 			Error("&encrypt only applicable to files");

src/Attr.h

@@ -25,9 +25,7 @@ typedef enum {
 	ATTR_EXPIRE_CREATE,
 	ATTR_PERSISTENT,
 	ATTR_SYNCHRONIZED,
-	ATTR_POSTPROCESSOR,
 	ATTR_ENCRYPT,
-	ATTR_MATCH,
 	ATTR_RAW_OUTPUT,
 	ATTR_MERGEABLE,
 	ATTR_PRIORITY,

src/CMakeLists.txt

@@ -203,6 +203,8 @@ binpac_target(dns.pac
     dns-protocol.pac dns-analyzer.pac)
 binpac_target(dns_tcp.pac
     dns.pac)
+binpac_target(gtpv1.pac
+    gtpv1-protocol.pac gtpv1-analyzer.pac)
 binpac_target(http.pac
     http-protocol.pac http-analyzer.pac)
 binpac_target(ncp.pac)
@@ -284,7 +286,6 @@ set(bro_SRCS
     Analyzer.cc
     Anon.cc
     ARP.cc
-    AsciiInputOutput.cc
     Attr.cc
     AYIYA.cc
     BackDoor.cc
@@ -333,6 +334,7 @@ set(bro_SRCS
     Frame.cc
     Func.cc
     Gnutella.cc
+    GTPv1.cc
     HTTP.cc
     HTTP-binpac.cc
     Hash.cc
@@ -359,6 +361,7 @@ set(bro_SRCS
     NetVar.cc
     NetbiosSSN.cc
     Obj.cc
+    OpaqueVal.cc
     OSFinger.cc
     PacketFilter.cc
     PacketSort.cc
@@ -423,6 +426,7 @@ set(bro_SRCS
     strsep.c
     modp_numtoa.c
 
+    threading/AsciiInputOutput.cc
     threading/BasicThread.cc
     threading/Manager.cc
     threading/MsgThread.cc

src/Expr.cc

@@ -229,10 +229,15 @@ bool Expr::DoUnserialize(UnserialInfo* info)
 	}
 
 
-NameExpr::NameExpr(ID* arg_id) : Expr(EXPR_NAME)
+NameExpr::NameExpr(ID* arg_id, bool const_init) : Expr(EXPR_NAME)
 	{
 	id = arg_id;
-	SetType(id->Type()->Ref());
+	in_const_init = const_init;
+
+	if ( id->AsType() )
+		SetType(new TypeType(id->AsType()));
+	else
+		SetType(id->Type()->Ref());
 
 	EventHandler* h = event_registry->Lookup(id->Name());
 	if ( h )
@@ -287,6 +292,9 @@ Expr* NameExpr::MakeLvalue()
 	if ( id->AsType() )
 		ExprError("Type name is not an lvalue");
 
+	if ( id->IsConst() && ! in_const_init )
+		ExprError("const is not a modifiable lvalue");
+
 	return new RefExpr(this);
 	}
 
@@ -337,9 +345,11 @@ bool NameExpr::DoSerialize(SerialInfo* info) const
 
 	// Write out just the name of the function if requested.
 	if ( info->globals_as_names && id->IsGlobal() )
-		return SERIALIZE('n') && SERIALIZE(id->Name());
+		return SERIALIZE('n') && SERIALIZE(id->Name()) &&
+		       SERIALIZE(in_const_init);
 	else
-		return SERIALIZE('f') && id->Serialize(info);
+		return SERIALIZE('f') && id->Serialize(info) &&
+		       SERIALIZE(in_const_init);
 	}
 
 bool NameExpr::DoUnserialize(UnserialInfo* info)
@@ -370,6 +380,9 @@ bool NameExpr::DoUnserialize(UnserialInfo* info)
 	if ( ! id )
 		return false;
 
+	if ( ! UNSERIALIZE(&in_const_init) )
+		return false;
+
 	return true;
 	}
 
@@ -2663,7 +2676,7 @@ void AssignExpr::EvalIntoAggregate(const BroType* t, Val* aggr, Frame* f) const
 	TableVal* tv = aggr->AsTableVal();
 
 	Val* index = op1->Eval(f);
-	Val* v = op2->Eval(f);
+	Val* v = check_and_promote(op2->Eval(f), t->YieldType(), 1);
 	if ( ! index || ! v )
 		return;
 
@@ -2788,22 +2801,43 @@ bool AssignExpr::DoUnserialize(UnserialInfo* info)
 	return UNSERIALIZE(&is_init);
 	}
 
-IndexExpr::IndexExpr(Expr* arg_op1, ListExpr* arg_op2)
+IndexExpr::IndexExpr(Expr* arg_op1, ListExpr* arg_op2, bool is_slice)
 : BinaryExpr(EXPR_INDEX, arg_op1, arg_op2)
 	{
 	if ( IsError() )
 		return;
 
+	if ( is_slice )
+		{
+		if ( ! IsString(op1->Type()->Tag()) )
+			ExprError("slice notation indexing only supported for strings currently");
+		}
+
+	else if ( IsString(op1->Type()->Tag()) )
+		{
+		if ( arg_op2->Exprs().length() != 1 )
+			ExprError("invalid string index expression");
+		}
+
+	if ( IsError() )
+		return;
+
 	int match_type = op1->Type()->MatchesIndex(arg_op2);
 	if ( match_type == DOES_NOT_MATCH_INDEX )
 		SetError("not an index type");
 
 	else if ( ! op1->Type()->YieldType() )
+		{
+		if ( IsString(op1->Type()->Tag()) &&
+		     match_type == MATCHES_INDEX_SCALAR )
+			SetType(base_type(TYPE_STRING));
+		else
 		// It's a set - so indexing it yields void.  We don't
 		// directly generate an error message, though, since this
 		// expression might be part of an add/delete statement,
 		// rather than yielding a value.
-		SetType(base_type(TYPE_VOID));
+			SetType(base_type(TYPE_VOID));
+		}
 
 	else if ( match_type == MATCHES_INDEX_SCALAR )
 		SetType(op1->Type()->YieldType()->Ref());
@@ -2879,6 +2913,9 @@ void IndexExpr::Delete(Frame* f)
 
 Expr* IndexExpr::MakeLvalue()
 	{
+	if ( IsString(op1->Type()->Tag()) )
+		ExprError("cannot assign to string index expression");
+
 	return new RefExpr(this);
 	}
 
@@ -2950,16 +2987,39 @@ Val* IndexExpr::Fold(Val* v1, Val* v2) const
 	if ( IsError() )
 		return 0;
 
-	if ( v1->Type()->Tag() == TYPE_VECTOR )
+	Val* v = 0;
+
+	switch ( v1->Type()->Tag() ) {
+	case TYPE_VECTOR:
+		v = v1->AsVectorVal()->Lookup(v2);
+		break;
+
+	case TYPE_TABLE:
+		v = v1->AsTableVal()->Lookup(v2);
+		break;
+
+	case TYPE_STRING:
 		{
-		Val* v = v1->AsVectorVal()->Lookup(v2);
-		// ### dangerous - this can silently fail larger operations
-		// due to a missing element
-		return v ? v->Ref() : 0;
+		const ListVal* lv = v2->AsListVal();
+		const BroString* s = v1->AsString();
+		int len = s->Len();
+		bro_int_t first = lv->Index(0)->AsInt();
+		bro_int_t last = lv->Length() > 1 ? lv->Index(1)->AsInt() : first;
+
+		if ( first < 0 )
+			first += len;
+
+		if ( last < 0 )
+			last += len;
+
+		BroString* substring = s->GetSubstring(first, last - first + 1);
+		return new StringVal(substring ? substring : new BroString(""));
 		}
 
-	TableVal* v_tbl = v1->AsTableVal();
-	Val* v = v_tbl->Lookup(v2);
+	default:
+		Error("type cannot be indexed");
+		break;
+	}
 
 	if ( v )
 		return v->Ref();
@@ -2986,14 +3046,25 @@ void IndexExpr::Assign(Frame* f, Val* v, Opcode op)
 		return;
 		}
 
-	if ( v1->Type()->Tag() == TYPE_VECTOR )
-		{
+	switch ( v1->Type()->Tag() ) {
+	case TYPE_VECTOR:
 		if ( ! v1->AsVectorVal()->Assign(v2, v, this, op) )
 			Internal("assignment failed");
-		}
+		break;
 
-	else if ( ! v1->AsTableVal()->Assign(v2, v, op) )
-		Internal("assignment failed");
+	case TYPE_TABLE:
+		if ( ! v1->AsTableVal()->Assign(v2, v, op) )
+			Internal("assignment failed");
+		break;
+
+	case TYPE_STRING:
+		Internal("assignment via string index accessor not allowed");
+		break;
+
+	default:
+		Internal("bad index expression type in assignment");
+		break;
+	}
 
 	Unref(v1);
 	Unref(v2);
@@ -3290,20 +3361,22 @@ RecordConstructorExpr::RecordConstructorExpr(ListExpr* constructor_list)
 
 Val* RecordConstructorExpr::InitVal(const BroType* t, Val* aggr) const
 	{
-	RecordVal* rv = Eval(0)->AsRecordVal();
-	RecordVal* ar = rv->CoerceTo(t->AsRecordType(), aggr);
+	Val* v = Eval(0);
 
-	if ( ar )
+	if ( v )
 		{
-		Unref(rv);
-		return ar;
+		RecordVal* rv = v->AsRecordVal();
+		RecordVal* ar = rv->CoerceTo(t->AsRecordType(), aggr);
+
+		if ( ar )
+			{
+			Unref(rv);
+			return ar;
+			}
 		}
 
-	else
-		{
-		Error("bad record initializer");
-		return 0;
-		}
+	Error("bad record initializer");
+	return 0;
 	}
 
 Val* RecordConstructorExpr::Fold(Val* v) const
@@ -3386,7 +3459,14 @@ Val* TableConstructorExpr::InitVal(const BroType* t, Val* aggr) const
 	if ( IsError() )
 		return 0;
 
-	return op->InitVal(t, aggr);
+	TableType* tt = Type()->AsTableType();
+	TableVal* tval = aggr ? aggr->AsTableVal() : new TableVal(tt, attrs);
+	const expr_list& exprs = op->AsListExpr()->Exprs();
+
+	loop_over_list(exprs, i)
+		exprs[i]->EvalIntoAggregate(t, tval, 0);
+
+	return tval;
 	}
 
 void TableConstructorExpr::ExprDescribe(ODesc* d) const
@@ -3438,7 +3518,7 @@ Val* SetConstructorExpr::Eval(Frame* f) const
 	if ( IsError() )
 		return 0;
 
-	TableVal* aggr = new TableVal(type->AsTableType(), 0);
+	TableVal* aggr = new TableVal(type->AsTableType(), attrs);
 	const expr_list& exprs = op->AsListExpr()->Exprs();
 
 	loop_over_list(exprs, i)
@@ -3456,7 +3536,26 @@ Val* SetConstructorExpr::InitVal(const BroType* t, Val* aggr) const
 	if ( IsError() )
 		return 0;
 
-	return op->InitVal(t, aggr);
+	const BroType* index_type = t->AsTableType()->Indices();
+	TableType* tt = Type()->AsTableType();
+	TableVal* tval = aggr ? aggr->AsTableVal() : new TableVal(tt, attrs);
+	const expr_list& exprs = op->AsListExpr()->Exprs();
+
+	loop_over_list(exprs, i)
+		{
+		Expr* e = exprs[i];
+		Val* element = check_and_promote(e->Eval(0), index_type, 1);
+
+		if ( ! element || ! tval->Assign(element, 0) )
+			{
+			Error(fmt("initialization type mismatch in set"), e);
+			return 0;
+			}
+
+		Unref(element);
+		}
+
+	return tval;
 	}
 
 void SetConstructorExpr::ExprDescribe(ODesc* d) const
@@ -3536,14 +3635,14 @@ Val* VectorConstructorExpr::InitVal(const BroType* t, Val* aggr) const
 	if ( IsError() )
 		return 0;
 
-	VectorVal* vec = aggr->AsVectorVal();
-	const BroType* vt = vec->Type()->AsVectorType()->YieldType();
+	VectorType* vt = Type()->AsVectorType();
+	VectorVal* vec = aggr ? aggr->AsVectorVal() : new VectorVal(vt);
 	const expr_list& exprs = op->AsListExpr()->Exprs();
 
 	loop_over_list(exprs, i)
 		{
 		Expr* e = exprs[i];
-		Val* v = check_and_promote(e->Eval(0), vt, 1);
+		Val* v = check_and_promote(e->Eval(0), t->YieldType(), 1);
 
 		if ( ! v || ! vec->Assign(i, v, e) )
 			{
@@ -4394,6 +4493,13 @@ CallExpr::CallExpr(Expr* arg_func, ListExpr* arg_args, bool in_hook)
 		return;
 		}
 
+	if ( func_type->AsFuncType()->Flavor() == FUNC_FLAVOR_HOOK && ! in_hook )
+		{
+		func->Error("hook cannot be called directly, use hook operator");
+		SetError();
+		return;
+		}
+
 	if ( ! func_type->MatchesIndex(args) )
 		SetError("argument type mismatch in function call");
 	else
@@ -4415,13 +4521,8 @@ CallExpr::CallExpr(Expr* arg_func, ListExpr* arg_args, bool in_hook)
 				break;
 
 			case FUNC_FLAVOR_HOOK:
-				// It's fine to not have a yield if it's known that the call
-				// is being done from a hook statement.
-				if ( ! in_hook )
-					{
-					Error("hook called in expression, use hook statement instead");
-					SetError();
-					}
+				Error("hook has no yield type");
+				SetError();
 				break;
 
 			default:

src/Expr.h

@@ -198,7 +198,7 @@ protected:
 
 class NameExpr : public Expr {
 public:
-	NameExpr(ID* id);
+	NameExpr(ID* id, bool const_init = false);
 	~NameExpr();
 
 	ID* Id() const		{ return id; }
@@ -220,6 +220,7 @@ protected:
 	DECLARE_SERIAL(NameExpr);
 
 	ID* id;
+	bool in_const_init;
 };
 
 class ConstExpr : public Expr {
@@ -645,7 +646,7 @@ protected:
 
 class IndexExpr : public BinaryExpr {
 public:
-	IndexExpr(Expr* op1, ListExpr* op2);
+	IndexExpr(Expr* op1, ListExpr* op2, bool is_slice = false);
 
 	int CanAdd() const;
 	int CanDel() const;
@@ -747,6 +748,8 @@ public:
 	TableConstructorExpr(ListExpr* constructor_list, attr_list* attrs);
 	~TableConstructorExpr()	{ Unref(attrs); }
 
+	Attributes* Attrs() { return attrs; }
+
 	Val* Eval(Frame* f) const;
 
 protected:
@@ -767,6 +770,8 @@ public:
 	SetConstructorExpr(ListExpr* constructor_list, attr_list* attrs);
 	~SetConstructorExpr()	{ Unref(attrs); }
 
+	Attributes* Attrs() { return attrs; }
+
 	Val* Eval(Frame* f) const;
 
 protected:

src/File.h

@@ -56,7 +56,7 @@ public:
 	// Rotates the logfile. Returns rotate_info.
 	RecordVal* Rotate();
 
-	// Set &rotate_interval, &rotate_size, &postprocessor,
+	// Set &rotate_interval, &rotate_size,
 	// and &raw_output attributes.
 	void SetAttrs(Attributes* attrs);
 

src/Func.cc

@@ -288,7 +288,8 @@ Val* BroFunc::Call(val_list* args, Frame* parent) const
 		assert(Flavor() == FUNC_FLAVOR_EVENT || Flavor() == FUNC_FLAVOR_HOOK);
 		loop_over_list(*args, i)
 			Unref((*args)[i]);
-		return 0 ;
+
+		return Flavor() == FUNC_FLAVOR_HOOK ? new Val(true, TYPE_BOOL) : 0;
 		}
 
 	SegmentProfiler(segment_logger, location);
@@ -349,16 +350,31 @@ Val* BroFunc::Call(val_list* args, Frame* parent) const
 			break;
 			}
 
-		if ( flow == FLOW_BREAK && Flavor() == FUNC_FLAVOR_HOOK )
+		if ( Flavor() == FUNC_FLAVOR_HOOK )
 			{
-			// short-circuit execution of remaining hook handler bodies
-			break;
+			// Ignore any return values of hook bodies, final return value
+			// depends on whether a body returns as a result of break statement.
+			Unref(result);
+			result = 0;
+
+			if ( flow == FLOW_BREAK )
+				{
+				// Short-circuit execution of remaining hook handler bodies.
+				result = new Val(false, TYPE_BOOL);
+				break;
+				}
 			}
 		}
 
+	if ( Flavor() == FUNC_FLAVOR_HOOK )
+		{
+		if ( ! result )
+			result = new Val(true, TYPE_BOOL);
+		}
+
 	// Warn if the function returns something, but we returned from
 	// the function without an explicit return, or without a value.
-	if ( FType()->YieldType() && FType()->YieldType()->Tag() != TYPE_VOID &&
+	else if ( FType()->YieldType() && FType()->YieldType()->Tag() != TYPE_VOID &&
 	     (flow != FLOW_RETURN /* we fell off the end */ ||
 	      ! result /* explicit return with no result */) &&
 	     ! f->HasDelayed() )

src/GTPv1.cc

@@ -0,0 +1,31 @@
+#include "GTPv1.h"
+
+GTPv1_Analyzer::GTPv1_Analyzer(Connection* conn)
+: Analyzer(AnalyzerTag::GTPv1, conn)
+	{
+	interp = new binpac::GTPv1::GTPv1_Conn(this);
+	}
+
+GTPv1_Analyzer::~GTPv1_Analyzer()
+	{
+	delete interp;
+	}
+
+void GTPv1_Analyzer::Done()
+	{
+	Analyzer::Done();
+	Event(udp_session_done);
+	}
+
+void GTPv1_Analyzer::DeliverPacket(int len, const u_char* data, bool orig, int seq, const IP_Hdr* ip, int caplen)
+	{
+	Analyzer::DeliverPacket(len, data, orig, seq, ip, caplen);
+	try
+		{
+		interp->NewData(orig, data, data + len);
+		}
+	catch ( const binpac::Exception& e )
+		{
+		ProtocolViolation(fmt("Binpac exception: %s", e.c_msg()));
+		}
+	}

src/GTPv1.h

@@ -0,0 +1,29 @@
+#ifndef GTPv1_h
+#define GTPv1_h
+
+#include "gtpv1_pac.h"
+
+class GTPv1_Analyzer : public Analyzer {
+public:
+	GTPv1_Analyzer(Connection* conn);
+	virtual ~GTPv1_Analyzer();
+
+	virtual void Done();
+	virtual void DeliverPacket(int len, const u_char* data, bool orig,
+					int seq, const IP_Hdr* ip, int caplen);
+
+	static Analyzer* InstantiateAnalyzer(Connection* conn)
+		{ return new GTPv1_Analyzer(conn); }
+
+	static bool Available()
+		{ return BifConst::Tunnel::enable_gtpv1 &&
+		         BifConst::Tunnel::max_depth > 0; }
+
+protected:
+	friend class AnalyzerTimer;
+	void ExpireTimer(double t);
+
+	binpac::GTPv1::GTPv1_Conn* interp;
+};
+
+#endif

src/HTTP.cc

@@ -1116,38 +1116,39 @@ const char* HTTP_Analyzer::PrefixWordMatch(const char* line,
 	return line;
 	}
 
+static bool is_HTTP_token_char(char c)
+	{
+	return c > 31 && c != 127 &&      // CTL per RFC 2616.
+	        c != ' ' && c != '\t' &&  // Separators.
+		c != '(' && c != ')' && c != '<' && c != '>' && c != '@' &&
+		c != ',' && c != ';' && c != ':' && c != '\\' && c != '"' &&
+		c != '/' && c != '[' && c != ']' && c != '?' && c != '=' &&
+		c != '{' && c != '}';
+	}
+
+static const char* get_HTTP_token(const char* s, const char* e)
+	{
+	while ( s < e && is_HTTP_token_char(*s) )
+		++s;
+
+	return s;
+	}
+
+
 int HTTP_Analyzer::HTTP_RequestLine(const char* line, const char* end_of_line)
 	{
 	const char* rest = 0;
-	static const char* http_methods[] = {
-		"GET", "POST", "HEAD",
+	const char* end_of_method = get_HTTP_token(line, end_of_line);
 
-		"OPTIONS", "PUT", "DELETE", "TRACE", "CONNECT",
+	if ( end_of_method == line )
+		goto error;
 
-		// HTTP methods for distributed authoring.
-		"PROPFIND", "PROPPATCH", "MKCOL", "DELETE", "PUT",
-		"COPY", "MOVE", "LOCK", "UNLOCK",
-		"POLL", "REPORT", "SUBSCRIBE", "BMOVE",
+	rest = skip_whitespace(end_of_method, end_of_line);
 
-		"SEARCH",
+	if ( rest == end_of_method )
+		goto error;
 
-		0,
-	};
-
-	int i;
-	for ( i = 0; http_methods[i]; ++i )
-		if ( (rest = PrefixWordMatch(line, end_of_line, http_methods[i])) != 0 )
-			break;
-
-	if ( ! http_methods[i] )
-		{
-		// Weird("HTTP_unknown_method");
-		if ( RequestExpected() )
-			HTTP_Event("unknown_HTTP_method", new_string_val(line, end_of_line));
-		return 0;
-		}
-
-	request_method = new StringVal(http_methods[i]);
+	request_method = new StringVal(end_of_method - line, line);
 
 	if ( ! ParseRequest(rest, end_of_line) )
 		reporter->InternalError("HTTP ParseRequest failed");
@@ -1157,6 +1158,10 @@ int HTTP_Analyzer::HTTP_RequestLine(const char* line, const char* end_of_line)
 			unescaped_URI->AsString()->Len(), true, true, true, true);
 
 	return 1;
+
+error:
+	reporter->Weird(Conn(), "bad_HTTP_request");
+	return 0;
 	}
 
 int HTTP_Analyzer::ParseRequest(const char* line, const char* end_of_line)

src/NetVar.cc

@@ -5,6 +5,7 @@
 #include "Var.h"
 #include "NetVar.h"
 
+RecordType* gtpv1_hdr_type;
 RecordType* conn_id;
 RecordType* endpoint;
 RecordType* endpoint_stats;
@@ -308,6 +309,7 @@ void init_net_var()
 #include "input.bif.netvar_init"
 #include "reporter.bif.netvar_init"
 
+	gtpv1_hdr_type = internal_type("gtpv1_hdr")->AsRecordType();
 	conn_id = internal_type("conn_id")->AsRecordType();
 	endpoint = internal_type("endpoint")->AsRecordType();
 	endpoint_stats = internal_type("endpoint_stats")->AsRecordType();

src/NetVar.h

@@ -8,6 +8,7 @@
 #include "EventRegistry.h"
 #include "Stats.h"
 
+extern RecordType* gtpv1_hdr_type;
 extern RecordType* conn_id;
 extern RecordType* endpoint;
 extern RecordType* endpoint_stats;

src/OpaqueVal.cc

@@ -0,0 +1,501 @@
+#include "OpaqueVal.h"
+#include "Reporter.h"
+#include "Serializer.h"
+
+bool HashVal::IsValid() const
+	{
+	return valid;
+	}
+
+bool HashVal::Init()
+	{
+	if ( valid )
+		return false;
+
+	valid = DoInit();
+	return valid;
+	}
+
+StringVal* HashVal::Get()
+	{
+	if ( ! valid )
+		return new StringVal("");
+
+	StringVal* result = DoGet();
+	valid = false;
+	return result;
+	}
+
+bool HashVal::Feed(const void* data, size_t size)
+	{
+	if ( valid )
+		return DoFeed(data, size);
+
+	reporter->InternalError("invalid opaque hash value");
+	return false;
+	}
+
+bool HashVal::DoInit()
+	{
+	assert(! "missing implementation of DoInit()");
+	return false;
+	}
+
+bool HashVal::DoFeed(const void*, size_t)
+	{
+	assert(! "missing implementation of DoFeed()");
+	return false;
+	}
+
+StringVal* HashVal::DoGet()
+	{
+	assert(! "missing implementation of DoGet()");
+	return new StringVal("");
+	}
+
+HashVal::HashVal(OpaqueType* t) : OpaqueVal(t)
+	{
+	valid = false;
+	}
+
+IMPLEMENT_SERIAL(HashVal, SER_HASH_VAL);
+
+bool HashVal::DoSerialize(SerialInfo* info) const
+	{
+	DO_SERIALIZE(SER_HASH_VAL, OpaqueVal);
+	return SERIALIZE(valid);
+	}
+
+bool HashVal::DoUnserialize(UnserialInfo* info)
+	{
+	DO_UNSERIALIZE(OpaqueVal);
+	return UNSERIALIZE(&valid);
+	}
+
+void MD5Val::digest(val_list& vlist, u_char result[MD5_DIGEST_LENGTH])
+	{
+	MD5_CTX h;
+	md5_init(&h);
+
+	loop_over_list(vlist, i)
+		{
+		Val* v = vlist[i];
+		if ( v->Type()->Tag() == TYPE_STRING )
+			{
+			const BroString* str = v->AsString();
+			md5_update(&h, str->Bytes(), str->Len());
+			}
+		else
+			{
+			ODesc d(DESC_BINARY);
+			v->Describe(&d);
+			md5_update(&h, (const u_char *) d.Bytes(), d.Len());
+			}
+		}
+
+	md5_final(&h, result);
+	}
+
+void MD5Val::hmac(val_list& vlist,
+                  u_char key[MD5_DIGEST_LENGTH],
+                  u_char result[MD5_DIGEST_LENGTH])
+	{
+	digest(vlist, result);
+	for ( int i = 0; i < MD5_DIGEST_LENGTH; ++i )
+		result[i] ^= key[i];
+
+	MD5(result, MD5_DIGEST_LENGTH, result);
+	}
+
+bool MD5Val::DoInit()
+	{
+	assert(! IsValid());
+	md5_init(&ctx);
+	return true;
+	}
+
+bool MD5Val::DoFeed(const void* data, size_t size)
+	{
+	if ( ! IsValid() )
+		return false;
+
+	md5_update(&ctx, data, size);
+	return true;
+	}
+
+StringVal* MD5Val::DoGet()
+	{
+	if ( ! IsValid() )
+		return new StringVal("");
+
+	u_char digest[MD5_DIGEST_LENGTH];
+	md5_final(&ctx, digest);
+	return new StringVal(md5_digest_print(digest));
+	}
+
+IMPLEMENT_SERIAL(MD5Val, SER_MD5_VAL);
+
+bool MD5Val::DoSerialize(SerialInfo* info) const
+	{
+	DO_SERIALIZE(SER_MD5_VAL, HashVal);
+
+	if ( ! IsValid() )
+		return true;
+
+	if ( ! (SERIALIZE(ctx.A) &&
+		SERIALIZE(ctx.B) &&
+		SERIALIZE(ctx.C) &&
+		SERIALIZE(ctx.D) &&
+		SERIALIZE(ctx.Nl) &&
+		SERIALIZE(ctx.Nh)) )
+		return false;
+
+	for ( int i = 0; i < MD5_LBLOCK; ++i )
+		{
+		if ( ! SERIALIZE(ctx.data[i]) )
+			return false;
+		}
+
+	if ( ! SERIALIZE(ctx.num) )
+		return false;
+
+	return true;
+	}
+
+bool MD5Val::DoUnserialize(UnserialInfo* info)
+	{
+	DO_UNSERIALIZE(HashVal);
+
+	if ( ! IsValid() )
+		return true;
+
+	if ( ! (UNSERIALIZE(&ctx.A) &&
+		UNSERIALIZE(&ctx.B) &&
+		UNSERIALIZE(&ctx.C) &&
+		UNSERIALIZE(&ctx.D) &&
+		UNSERIALIZE(&ctx.Nl) &&
+		UNSERIALIZE(&ctx.Nh)) )
+		return false;
+
+	for ( int i = 0; i < MD5_LBLOCK; ++i )
+		{
+		if ( ! UNSERIALIZE(&ctx.data[i]) )
+			return false;
+		}
+
+	if ( ! UNSERIALIZE(&ctx.num) )
+		return false;
+
+	return true;
+	}
+
+void SHA1Val::digest(val_list& vlist, u_char result[SHA_DIGEST_LENGTH])
+	{
+	SHA_CTX h;
+	sha1_init(&h);
+
+	loop_over_list(vlist, i)
+		{
+		Val* v = vlist[i];
+		if ( v->Type()->Tag() == TYPE_STRING )
+			{
+			const BroString* str = v->AsString();
+			sha1_update(&h, str->Bytes(), str->Len());
+			}
+		else
+			{
+			ODesc d(DESC_BINARY);
+			v->Describe(&d);
+			sha1_update(&h, (const u_char *) d.Bytes(), d.Len());
+			}
+		}
+
+	sha1_final(&h, result);
+	}
+
+bool SHA1Val::DoInit()
+	{
+	assert(! IsValid());
+	sha1_init(&ctx);
+	return true;
+	}
+
+bool SHA1Val::DoFeed(const void* data, size_t size)
+	{
+	if ( ! IsValid() )
+		return false;
+
+	sha1_update(&ctx, data, size);
+	return true;
+	}
+
+StringVal* SHA1Val::DoGet()
+	{
+	if ( ! IsValid() )
+		return new StringVal("");
+
+	u_char digest[SHA_DIGEST_LENGTH];
+	sha1_final(&ctx, digest);
+	return new StringVal(sha1_digest_print(digest));
+	}
+
+IMPLEMENT_SERIAL(SHA1Val, SER_SHA1_VAL);
+
+bool SHA1Val::DoSerialize(SerialInfo* info) const
+	{
+	DO_SERIALIZE(SER_SHA1_VAL, HashVal);
+
+	if ( ! IsValid() )
+		return true;
+
+	if ( ! (SERIALIZE(ctx.h0) &&
+		SERIALIZE(ctx.h1) &&
+		SERIALIZE(ctx.h2) &&
+		SERIALIZE(ctx.h3) &&
+		SERIALIZE(ctx.h4) &&
+		SERIALIZE(ctx.Nl) &&
+		SERIALIZE(ctx.Nh)) )
+		return false;
+
+	for ( int i = 0; i < SHA_LBLOCK; ++i )
+		{
+		if ( ! SERIALIZE(ctx.data[i]) )
+			return false;
+		}
+
+	if ( ! SERIALIZE(ctx.num) )
+		return false;
+
+	return true;
+	}
+
+bool SHA1Val::DoUnserialize(UnserialInfo* info)
+	{
+	DO_UNSERIALIZE(HashVal);
+
+	if ( ! IsValid() )
+		return true;
+
+	if ( ! (UNSERIALIZE(&ctx.h0) &&
+		UNSERIALIZE(&ctx.h1) &&
+		UNSERIALIZE(&ctx.h2) &&
+		UNSERIALIZE(&ctx.h3) &&
+		UNSERIALIZE(&ctx.h4) &&
+		UNSERIALIZE(&ctx.Nl) &&
+		UNSERIALIZE(&ctx.Nh)) )
+		return false;
+
+	for ( int i = 0; i < SHA_LBLOCK; ++i )
+		{
+		if ( ! UNSERIALIZE(&ctx.data[i]) )
+			return false;
+		}
+
+	if ( ! UNSERIALIZE(&ctx.num) )
+		return false;
+
+	return true;
+	}
+
+void SHA256Val::digest(val_list& vlist, u_char result[SHA256_DIGEST_LENGTH])
+	{
+	SHA256_CTX h;
+	sha256_init(&h);
+
+	loop_over_list(vlist, i)
+		{
+		Val* v = vlist[i];
+		if ( v->Type()->Tag() == TYPE_STRING )
+			{
+			const BroString* str = v->AsString();
+			sha256_update(&h, str->Bytes(), str->Len());
+			}
+		else
+			{
+			ODesc d(DESC_BINARY);
+			v->Describe(&d);
+			sha256_update(&h, (const u_char *) d.Bytes(), d.Len());
+			}
+		}
+
+	sha256_final(&h, result);
+	}
+
+bool SHA256Val::DoInit()
+	{
+	assert( ! IsValid() );
+	sha256_init(&ctx);
+	return true;
+	}
+
+bool SHA256Val::DoFeed(const void* data, size_t size)
+	{
+	if ( ! IsValid() )
+		return false;
+
+	sha256_update(&ctx, data, size);
+	return true;
+	}
+
+StringVal* SHA256Val::DoGet()
+	{
+	if ( ! IsValid() )
+		return new StringVal("");
+
+	u_char digest[SHA256_DIGEST_LENGTH];
+	sha256_final(&ctx, digest);
+	return new StringVal(sha256_digest_print(digest));
+	}
+
+IMPLEMENT_SERIAL(SHA256Val, SER_SHA256_VAL);
+
+bool SHA256Val::DoSerialize(SerialInfo* info) const
+	{
+	DO_SERIALIZE(SER_SHA256_VAL, HashVal);
+
+	if ( ! IsValid() )
+		return true;
+
+	for ( int i = 0; i < 8; ++i )
+		{
+		if ( ! SERIALIZE(ctx.h[i]) )
+			return false;
+		}
+
+	if ( ! (SERIALIZE(ctx.Nl) &&
+		SERIALIZE(ctx.Nh)) )
+		return false;
+
+	for ( int i = 0; i < SHA_LBLOCK; ++i )
+		{
+		if ( ! SERIALIZE(ctx.data[i]) )
+			return false;
+		}
+
+	if ( ! (SERIALIZE(ctx.num) &&
+		SERIALIZE(ctx.md_len)) )
+	     return false;
+
+	return true;
+	}
+
+bool SHA256Val::DoUnserialize(UnserialInfo* info)
+	{
+	DO_UNSERIALIZE(HashVal);
+
+	if ( ! IsValid() )
+		return true;
+
+	for ( int i = 0; i < 8; ++i )
+		{
+		if ( ! UNSERIALIZE(&ctx.h[i]) )
+			return false;
+		}
+
+	if ( ! (UNSERIALIZE(&ctx.Nl) &&
+		UNSERIALIZE(&ctx.Nh)) )
+	     return false;
+
+	for ( int i = 0; i < SHA_LBLOCK; ++i )
+		{
+		if ( ! UNSERIALIZE(&ctx.data[i]) )
+			return false;
+		}
+
+
+	if ( ! (UNSERIALIZE(&ctx.num) &&
+		UNSERIALIZE(&ctx.md_len)) )
+		return false;
+
+	return true;
+	}
+
+
+bool EntropyVal::Feed(const void* data, size_t size)
+	{
+	state.add(data, size);
+	return true;
+	}
+
+bool EntropyVal::Get(double *r_ent, double *r_chisq, double *r_mean,
+                     double *r_montepicalc, double *r_scc)
+	{
+	state.end(r_ent, r_chisq, r_mean, r_montepicalc, r_scc);
+	return true;
+	}
+
+IMPLEMENT_SERIAL(EntropyVal, SER_ENTROPY_VAL);
+
+bool EntropyVal::DoSerialize(SerialInfo* info) const
+	{
+	DO_SERIALIZE(SER_ENTROPY_VAL, OpaqueVal);
+
+	for ( int i = 0; i < 256; ++i )
+		{
+		if ( ! SERIALIZE(state.ccount[i]) )
+			return false;
+		}
+
+	if ( ! (SERIALIZE(state.totalc) &&
+		SERIALIZE(state.mp) &&
+		SERIALIZE(state.sccfirst)) )
+		return false;
+
+	for ( int i = 0; i < RT_MONTEN; ++i )
+		{
+		if ( ! SERIALIZE(state.monte[i]) )
+			return false;
+		}
+
+	if ( ! (SERIALIZE(state.inmont) &&
+		SERIALIZE(state.mcount) &&
+		SERIALIZE(state.cexp) &&
+		SERIALIZE(state.montex) &&
+		SERIALIZE(state.montey) &&
+		SERIALIZE(state.montepi) &&
+		SERIALIZE(state.sccu0) &&
+		SERIALIZE(state.scclast) &&
+		SERIALIZE(state.scct1) &&
+		SERIALIZE(state.scct2) &&
+		SERIALIZE(state.scct3)) )
+		return false;
+
+	return true;
+	}
+
+bool EntropyVal::DoUnserialize(UnserialInfo* info)
+	{
+	DO_UNSERIALIZE(OpaqueVal);
+
+	for ( int i = 0; i < 256; ++i )
+		{
+		if ( ! UNSERIALIZE(&state.ccount[i]) )
+			return false;
+		}
+
+	if ( ! (UNSERIALIZE(&state.totalc) &&
+		UNSERIALIZE(&state.mp) &&
+		UNSERIALIZE(&state.sccfirst)) )
+		return false;
+
+	for ( int i = 0; i < RT_MONTEN; ++i )
+		{
+		if ( ! UNSERIALIZE(&state.monte[i]) )
+			return false;
+		}
+
+	if ( ! (UNSERIALIZE(&state.inmont) &&
+		UNSERIALIZE(&state.mcount) &&
+		UNSERIALIZE(&state.cexp) &&
+		UNSERIALIZE(&state.montex) &&
+		UNSERIALIZE(&state.montey) &&
+		UNSERIALIZE(&state.montepi) &&
+		UNSERIALIZE(&state.sccu0) &&
+		UNSERIALIZE(&state.scclast) &&
+		UNSERIALIZE(&state.scct1) &&
+		UNSERIALIZE(&state.scct2) &&
+		UNSERIALIZE(&state.scct3)) )
+		return false;
+
+	return true;
+	}

src/OpaqueVal.h

@@ -0,0 +1,110 @@
+// See the file "COPYING" in the main distribution directory for copyright.
+
+#ifndef OPAQUEVAL_H
+#define OPAQUEVAL_H
+
+#include "RandTest.h"
+#include "Val.h"
+#include "digest.h"
+
+class HashVal : public OpaqueVal {
+public:
+	virtual bool IsValid() const;
+	virtual bool Init();
+	virtual bool Feed(const void* data, size_t size);
+	virtual StringVal* Get();
+
+protected:
+	HashVal() { };
+	HashVal(OpaqueType* t);
+	virtual bool DoInit();
+	virtual bool DoFeed(const void* data, size_t size);
+	virtual StringVal* DoGet();
+
+	DECLARE_SERIAL(HashVal);
+
+private:
+	// This flag exists because Get() can only be called once.
+	bool valid;
+};
+
+class MD5Val : public HashVal {
+public:
+	static void digest(val_list& vlist, u_char result[MD5_DIGEST_LENGTH]);
+
+	static void hmac(val_list& vlist,
+			 u_char key[MD5_DIGEST_LENGTH],
+			 u_char result[MD5_DIGEST_LENGTH]);
+
+	MD5Val() : HashVal(new OpaqueType("md5"))	{ }
+
+protected:
+	friend class Val;
+
+	virtual bool DoInit() /* override */;
+	virtual bool DoFeed(const void* data, size_t size) /* override */;
+	virtual StringVal* DoGet() /* override */;
+
+	DECLARE_SERIAL(MD5Val);
+
+private:
+	MD5_CTX ctx;
+};
+
+class SHA1Val : public HashVal {
+public:
+	static void digest(val_list& vlist, u_char result[SHA_DIGEST_LENGTH]);
+
+	SHA1Val() : HashVal(new OpaqueType("sha1")) { }
+
+protected:
+	friend class Val;
+
+	virtual bool DoInit() /* override */;
+	virtual bool DoFeed(const void* data, size_t size) /* override */;
+	virtual StringVal* DoGet() /* override */;
+
+	DECLARE_SERIAL(SHA1Val);
+
+private:
+	SHA_CTX ctx;
+};
+
+class SHA256Val : public HashVal {
+public:
+	static void digest(val_list& vlist, u_char result[SHA256_DIGEST_LENGTH]);
+
+	SHA256Val() : HashVal(new OpaqueType("sha256"))	{ }
+
+protected:
+	friend class Val;
+
+	virtual bool DoInit() /* override */;
+	virtual bool DoFeed(const void* data, size_t size) /* override */;
+	virtual StringVal* DoGet() /* override */;
+
+	DECLARE_SERIAL(SHA256Val);
+
+private:
+	SHA256_CTX ctx;
+};
+
+class EntropyVal : public OpaqueVal {
+public:
+	EntropyVal() : OpaqueVal(new OpaqueType("entropy"))	{ }
+
+	bool Feed(const void* data, size_t size);
+	bool Get(double *r_ent, double *r_chisq, double *r_mean,
+		 double *r_montepicalc, double *r_scc);
+
+protected:
+	friend class Val;
+	EntropyVal(OpaqueType* t);
+
+	DECLARE_SERIAL(EntropyVal);
+
+private:
+	RandTest state;
+};
+
+#endif

src/RandTest.cc

@@ -12,7 +12,18 @@
 	Modified for Bro by Seth Hall - July 2010
 */
 
-#include <RandTest.h>
+#include <math.h>
+#include "RandTest.h"
+
+#define log2of10 3.32192809488736234787
+/*  RT_LOG2  --  Calculate log to the base 2  */
+static double rt_log2(double x)
+{
+    return log2of10 * log10(x);
+}
+
+// RT_INCIRC = pow(pow(256.0, (double) (RT_MONTEN / 2)) - 1, 2.0);
+#define RT_INCIRC 281474943156225.0
 
 RandTest::RandTest()
 	{
@@ -28,9 +39,9 @@ RandTest::RandTest()
 		}
 	}
 
-void RandTest::add(void *buf, int bufl)
+void RandTest::add(const void *buf, int bufl)
 	{
-	unsigned char *bp = (unsigned char*)buf;
+	const unsigned char *bp = static_cast<const unsigned char*>(buf);
 	int oc;
 
 	while (bufl-- > 0)
@@ -78,8 +89,8 @@ void RandTest::add(void *buf, int bufl)
 		}
 	}
 
-void RandTest::end(double *r_ent, double *r_chisq,
-                   double *r_mean, double *r_montepicalc, double *r_scc)
+void RandTest::end(double* r_ent, double* r_chisq,
+                   double* r_mean, double* r_montepicalc, double* r_scc)
 	{
 	int i;
 	double ent, chisq, scc, datasum;

src/RandTest.h

@@ -1,34 +1,33 @@
-#include <math.h>
+#ifndef RANDTEST_H
+#define RANDTEST_H
 
-#define log2of10 3.32192809488736234787
-/*  RT_LOG2  --  Calculate log to the base 2  */
-static double rt_log2(double x)
-{
-    return log2of10 * log10(x);
-}
+#include "util.h"
 
 #define RT_MONTEN 6  /* Bytes used as Monte Carlo
                         co-ordinates. This should be no more
                         bits than the mantissa of your "double"
                         floating point type. */
+class EntropyVal;
 
-// RT_INCIRC = pow(pow(256.0, (double) (RT_MONTEN / 2)) - 1, 2.0);
-#define RT_INCIRC 281474943156225.0
 
 class RandTest {
 	public:
 		RandTest();
-		void add(void *buf, int bufl);
-		void end(double *r_ent, double *r_chisq, double *r_mean,
-		         double *r_montepicalc, double *r_scc);
+		void add(const void* buf, int bufl);
+		void end(double* r_ent, double* r_chisq, double* r_mean,
+		         double* r_montepicalc, double* r_scc);
 
 	private:
-		long ccount[256];  /* Bins to count occurrences of values */
-		long totalc;       /* Total bytes counted */
+	  friend class EntropyVal;
+
+		int64 ccount[256];  /* Bins to count occurrences of values */
+		int64 totalc;       /* Total bytes counted */
 		int mp;
 		int sccfirst;
 		unsigned int monte[RT_MONTEN];
-		long inmont, mcount;
+		int64 inmont, mcount;
 		double cexp, montex, montey, montepi,
 		       sccu0, scclast, scct1, scct2, scct3;
-	};
+};
+
+#endif

src/RemoteSerializer.cc

@@ -545,8 +545,11 @@ RemoteSerializer::~RemoteSerializer()
 	{
 	if ( child_pid )
 		{
-		kill(child_pid, SIGKILL);
-		waitpid(child_pid, 0, 0);
+		if ( kill(child_pid, SIGKILL) < 0 )
+			reporter->Warning("warning: cannot kill child (pid %d), %s", child_pid, strerror(errno));
+
+		else if ( waitpid(child_pid, 0, 0) < 0 )
+			reporter->Warning("warning: error encountered during waitpid(%d), %s", child_pid, strerror(errno));
 		}
 
 	delete io;
@@ -1032,6 +1035,14 @@ bool RemoteSerializer::SendAllSynchronized(Peer* peer, SerialInfo* info)
 
 	for ( ; index < sync_ids.length(); ++index )
 		{
+		if ( ! sync_ids[index]->ID_Val() )
+			{
+#ifdef DEBUG
+			DBG_LOG(DBG_COMM, "Skip sync of ID with null value: %s\n",
+			        sync_ids[index]->Name());
+#endif
+			continue;
+			}
 		cont->SaveContext();
 
 		StateAccess sa(OP_ASSIGN, sync_ids[index],
@@ -3153,7 +3164,10 @@ void RemoteSerializer::FatalError(const char* msg)
 	reporter->Error("%s", msg);
 
 	closed = true;
-	kill(child_pid, SIGQUIT);
+
+	if ( kill(child_pid, SIGQUIT) < 0 )
+		reporter->Warning("warning: cannot kill child pid %d, %s", child_pid, strerror(errno));
+
 	child_pid = 0;
 	using_communication = false;
 	io->Clear();
@@ -3963,7 +3977,7 @@ bool SocketComm::Connect(Peer* peer)
 	{
 	int status;
 	addrinfo hints, *res, *res0;
-	bzero(&hints, sizeof(hints));
+        memset(&hints, 0, sizeof(hints));
 
 	hints.ai_family = PF_UNSPEC;
 	hints.ai_protocol = IPPROTO_TCP;
@@ -4095,7 +4109,7 @@ bool SocketComm::Listen()
 	{
 	int status, on = 1;
 	addrinfo hints, *res, *res0;
-	bzero(&hints, sizeof(hints));
+        memset(&hints, 0, sizeof(hints));
 
 	IPAddr listen_ip(listen_if);
 
@@ -4360,7 +4374,8 @@ void SocketComm::Kill()
 
 	CloseListenFDs();
 
-	kill(getpid(), SIGTERM);
+	if ( kill(getpid(), SIGTERM) < 0 )
+		Log(fmt("warning: cannot kill SocketComm pid %d, %s", getpid(), strerror(errno)));
 
 	while ( 1 )
 		; // loop until killed

src/SerialTypes.h

@@ -98,6 +98,12 @@ SERIAL_VAL(RECORD_VAL, 10)
 SERIAL_VAL(ENUM_VAL, 11)
 SERIAL_VAL(VECTOR_VAL, 12)
 SERIAL_VAL(MUTABLE_VAL, 13)
+SERIAL_VAL(OPAQUE_VAL, 14)
+SERIAL_VAL(HASH_VAL, 15)
+SERIAL_VAL(MD5_VAL, 16)
+SERIAL_VAL(SHA1_VAL, 17)
+SERIAL_VAL(SHA256_VAL, 18)
+SERIAL_VAL(ENTROPY_VAL, 19)
 
 #define SERIAL_EXPR(name, val) SERIAL_CONST(name, val, EXPR)
 SERIAL_EXPR(EXPR, 1)
@@ -165,7 +171,6 @@ SERIAL_STMT(EVENT_BODY_LIST, 16)
 SERIAL_STMT(INIT_STMT, 17)
 SERIAL_STMT(NULL_STMT, 18)
 SERIAL_STMT(WHEN_STMT, 19)
-SERIAL_STMT(HOOK_STMT, 20)
 
 #define SERIAL_TYPE(name, val) SERIAL_CONST(name, val, BRO_TYPE)
 SERIAL_TYPE(BRO_TYPE, 1)
@@ -179,6 +184,7 @@ SERIAL_TYPE(SUBNET_TYPE, 8)
 SERIAL_TYPE(FILE_TYPE, 9)
 SERIAL_TYPE(ENUM_TYPE, 10)
 SERIAL_TYPE(VECTOR_TYPE, 11)
+SERIAL_TYPE(OPAQUE_TYPE, 12)
 
 SERIAL_CONST2(ATTRIBUTES)
 SERIAL_CONST2(EVENT_HANDLER)

src/Stmt.cc

@@ -23,7 +23,7 @@ const char* stmt_name(BroStmtTag t)
 		"print", "event", "expr", "if", "when", "switch",
 		"for", "next", "break", "return", "add", "delete",
 		"list", "bodylist",
-		"<init>", "hook",
+		"<init>",
 		"null",
 	};
 
@@ -592,6 +592,21 @@ Case::~Case()
 
 void Case::Describe(ODesc* d) const
 	{
+	if ( ! Cases() )
+		{
+		if ( ! d->IsBinary() )
+			d->Add("default:");
+
+		d->AddCount(0);
+
+		d->PushIndent();
+		Body()->AccessStats(d);
+		Body()->Describe(d);
+		d->PopIndent();
+
+		return;
+		}
+
 	const expr_list& e = Cases()->Exprs();
 
 	if ( ! d->IsBinary() )
@@ -658,13 +673,64 @@ bool Case::DoUnserialize(UnserialInfo* info)
 	return this->s != 0;
 	}
 
-SwitchStmt::SwitchStmt(Expr* index, case_list* arg_cases) :
-	ExprStmt(STMT_SWITCH, index)
+static void int_del_func(void* v)
 	{
-	cases = arg_cases;
+	delete (int*) v;
+	}
 
-	//### need to loop over cases and make sure their type matches
-	//### the index, and they're constant and not redundant
+void SwitchStmt::Init()
+	{
+	TypeList* t = new TypeList();
+	t->Append(e->Type()->Ref());
+	comp_hash = new CompositeHash(t);
+	Unref(t);
+
+	case_label_map.SetDeleteFunc(int_del_func);
+	}
+
+SwitchStmt::SwitchStmt(Expr* index, case_list* arg_cases) :
+	ExprStmt(STMT_SWITCH, index), cases(arg_cases), default_case_idx(-1)
+	{
+	Init();
+
+	if ( ! is_atomic_type(e->Type()) )
+		e->Error("switch expression must be of an atomic type");
+
+	loop_over_list(*cases, i)
+		{
+		const Case* c = (*cases)[i];
+		const ListExpr* le = c->Cases();
+
+		if ( le )
+			{
+			if ( ! le->Type()->AsTypeList()->AllMatch(e->Type(), false) )
+				{
+				le->Error("case expression type differs from switch type", e);
+				continue;
+				}
+
+			const expr_list& exprs = le->Exprs();
+
+			loop_over_list(exprs, j)
+				{
+				if ( ! exprs[j]->IsConst() )
+					exprs[j]->Error("case label expression isn't constant");
+				else
+					{
+					if ( ! AddCaseLabelMapping(exprs[j]->ExprVal(), i) )
+						exprs[j]->Error("duplicate case label");
+					}
+				}
+			}
+
+		else
+			{
+			if ( default_case_idx != -1 )
+				c->Error("multiple default labels", (*cases)[default_case_idx]);
+			else
+				default_case_idx = i;
+			}
+		}
 	}
 
 SwitchStmt::~SwitchStmt()
@@ -673,12 +739,80 @@ SwitchStmt::~SwitchStmt()
 		Unref((*cases)[i]);
 
 	delete cases;
+	delete comp_hash;
 	}
 
-Val* SwitchStmt::DoExec(Frame* /* f */, Val* /* v */, stmt_flow_type& /* flow */) const
+bool SwitchStmt::AddCaseLabelMapping(const Val* v, int idx)
 	{
-	printf("switch statement not implemented\n");
-	return 0;
+	HashKey* hk = comp_hash->ComputeHash(v, 1);
+
+	if ( ! hk )
+		{
+		reporter->PushLocation(e->GetLocationInfo());
+		reporter->InternalError("switch expression type mismatch (%s/%s)",
+		    type_name(v->Type()->Tag()), type_name(e->Type()->Tag()));
+		}
+
+	int* label_idx = case_label_map.Lookup(hk);
+
+	if ( label_idx )
+		{
+		delete hk;
+		return false;
+		}
+
+	case_label_map.Insert(hk, new int(idx));
+	return true;
+	}
+
+int SwitchStmt::FindCaseLabelMatch(const Val* v) const
+	{
+	HashKey* hk = comp_hash->ComputeHash(v, 1);
+
+	if ( ! hk )
+		{
+		reporter->PushLocation(e->GetLocationInfo());
+		reporter->InternalError("switch expression type mismatch (%s/%s)",
+		    type_name(v->Type()->Tag()), type_name(e->Type()->Tag()));
+		}
+
+	int* label_idx = case_label_map.Lookup(hk);
+
+	delete hk;
+
+	if ( ! label_idx )
+		return default_case_idx;
+	else
+		return *label_idx;
+	}
+
+Val* SwitchStmt::DoExec(Frame* f, Val* v, stmt_flow_type& flow) const
+	{
+	Val* rval = 0;
+
+	int matching_label_idx = FindCaseLabelMatch(v);
+
+	if ( matching_label_idx == -1 )
+		return 0;
+
+	for ( int i = matching_label_idx; i < cases->length(); ++i )
+		{
+		const Case* c = (*cases)[i];
+
+		flow = FLOW_NEXT;
+		rval = c->Body()->Exec(f, flow);
+
+		if ( flow == FLOW_BREAK )
+			{
+			flow = FLOW_NEXT;
+			break;
+			}
+
+		if ( flow == FLOW_RETURN )
+			break;
+		}
+
+	return rval;
 	}
 
 Stmt* SwitchStmt::DoSimplify()
@@ -697,7 +831,13 @@ Stmt* SwitchStmt::DoSimplify()
 		}
 
 	if ( e->IsConst() )
-		{ // ### go through cases and pull out the one it matches
+		{
+		// Could possibly remove all case labels before the one
+		// that will match, but may be tricky to tell if any
+		// subsequent ones can also be removed since it depends
+		// on the evaluation of the body executing a break/return
+		// statement.  Then still need a way to bypass the lookup
+		// DoExec for it to be beneficial.
 		if ( ! optimize )
 			Warn("constant in switch");
 		}
@@ -770,6 +910,9 @@ bool SwitchStmt::DoSerialize(SerialInfo* info) const
 		if ( ! (*cases)[i]->Serialize(info) )
 			return false;
 
+	if ( ! SERIALIZE(default_case_idx) )
+		return false;
+
 	return true;
 	}
 
@@ -777,6 +920,8 @@ bool SwitchStmt::DoUnserialize(UnserialInfo* info)
 	{
 	DO_UNSERIALIZE(ExprStmt);
 
+	Init();
+
 	int len;
 	if ( ! UNSERIALIZE(&len) )
 		return false;
@@ -790,6 +935,25 @@ bool SwitchStmt::DoUnserialize(UnserialInfo* info)
 		cases->append(c);
 		}
 
+	if ( ! UNSERIALIZE(&default_case_idx) )
+		return false;
+
+	loop_over_list(*cases, i)
+		{
+		const ListExpr* le = (*cases)[i]->Cases();
+
+		if ( ! le )
+			continue;
+
+		const expr_list& exprs = le->Exprs();
+
+		loop_over_list(exprs, j)
+			{
+			if ( ! AddCaseLabelMapping(exprs[j]->ExprVal(), i) )
+				return false;
+			}
+		}
+
 	return true;
 	}
 
@@ -933,52 +1097,6 @@ bool EventStmt::DoUnserialize(UnserialInfo* info)
 	return event_expr != 0;
 	}
 
-HookStmt::HookStmt(CallExpr* arg_e) : ExprStmt(STMT_HOOK, arg_e)
-	{
-	call_expr = arg_e;
-	}
-
-Val* HookStmt::Exec(Frame* f, stmt_flow_type& flow) const
-	{
-	RegisterAccess();
-
-	Val* ret = call_expr->Eval(f);
-	Unref(ret);
-
-	flow = FLOW_NEXT;
-
-	return 0;
-	}
-
-TraversalCode HookStmt::Traverse(TraversalCallback* cb) const
-	{
-	TraversalCode tc = cb->PreStmt(this);
-	HANDLE_TC_STMT_PRE(tc);
-
-	// call expr is stored in base class's "e" field.
-	tc = e->Traverse(cb);
-	HANDLE_TC_STMT_PRE(tc);
-
-	tc = cb->PostStmt(this);
-	HANDLE_TC_STMT_POST(tc);
-	}
-
-IMPLEMENT_SERIAL(HookStmt, SER_HOOK_STMT);
-
-bool HookStmt::DoSerialize(SerialInfo* info) const
-	{
-	DO_SERIALIZE(SER_HOOK_STMT, ExprStmt);
-	return call_expr->Serialize(info);
-	}
-
-bool HookStmt::DoUnserialize(UnserialInfo* info)
-	{
-	DO_UNSERIALIZE(ExprStmt);
-
-	call_expr = (CallExpr*) Expr::Unserialize(info, EXPR_CALL);
-	return call_expr != 0;
-	}
-
 ForStmt::ForStmt(id_list* arg_loop_vars, Expr* loop_expr)
 : ExprStmt(STMT_FOR, loop_expr)
 	{
@@ -1378,7 +1496,10 @@ ReturnStmt::ReturnStmt(Expr* arg_e) : ExprStmt(STMT_RETURN, arg_e)
 		}
 
 	else if ( ! e )
-		Error("return statement needs expression");
+		{
+		if ( ft->Flavor() != FUNC_FLAVOR_HOOK )
+			Error("return statement needs expression");
+		}
 
 	else
 		(void) check_and_promote_expr(e, yt);
@@ -1990,7 +2111,6 @@ int same_stmt(const Stmt* s1, const Stmt* s2)
 	case STMT_RETURN:
 	case STMT_EXPR:
 	case STMT_EVENT:
-	case STMT_HOOK:
 		{
 		const ExprStmt* e1 = (const ExprStmt*) s1;
 		const ExprStmt* e2 = (const ExprStmt*) s2;

src/Stmt.h

@@ -17,6 +17,8 @@
 class StmtList;
 class ForStmt;
 
+declare(PDict, int);
+
 class Stmt : public BroObj {
 public:
 	BroStmtTag Tag() const	{ return tag; }
@@ -187,7 +189,8 @@ protected:
 
 class Case : public BroObj {
 public:
-	Case(ListExpr* c, Stmt* arg_s)	{ cases = c; s = arg_s; }
+	Case(ListExpr* c, Stmt* arg_s) :
+	    cases(simplify_expr_list(c,SIMPLIFY_GENERAL)), s(arg_s) { }
 	~Case();
 
 	const ListExpr* Cases() const	{ return cases; }
@@ -226,7 +229,7 @@ public:
 
 protected:
 	friend class Stmt;
-	SwitchStmt()	{ cases = 0; }
+	SwitchStmt()	{ cases = 0; default_case_idx = -1; comp_hash = 0; }
 
 	Val* DoExec(Frame* f, Val* v, stmt_flow_type& flow) const;
 	Stmt* DoSimplify();
@@ -234,7 +237,23 @@ protected:
 
 	DECLARE_SERIAL(SwitchStmt);
 
+	// Initialize composite hash and case label map.
+	void Init();
+
+	// Adds an entry in case_label_map for the given value to associate it
+	// with the given index in the cases list.  If the entry already exists,
+	// returns false, else returns true.
+	bool AddCaseLabelMapping(const Val* v, int idx);
+
+	// Returns index of a case label that's equal to the value, or
+	// default_case_idx if no case label matches (which may be -1 if there's
+	// no default label).
+	int FindCaseLabelMatch(const Val* v) const;
+
 	case_list* cases;
+	int default_case_idx;
+	CompositeHash* comp_hash;
+	PDict(int) case_label_map;
 };
 
 class AddStmt : public ExprStmt {
@@ -286,24 +305,6 @@ protected:
 	EventExpr* event_expr;
 };
 
-class HookStmt : public ExprStmt {
-public:
-	HookStmt(CallExpr* e);
-
-	Val* Exec(Frame* f, stmt_flow_type& flow) const;
-
-	TraversalCode Traverse(TraversalCallback* cb) const;
-
-protected:
-	friend class Stmt;
-
-	HookStmt() { call_expr = 0; }
-
-	DECLARE_SERIAL(HookStmt);
-
-	CallExpr* call_expr;
-};
-
 class ForStmt : public ExprStmt {
 public:
 	ForStmt(id_list* loop_vars, Expr* loop_expr);

src/StmtEnums.h

@@ -15,7 +15,7 @@ typedef enum {
 	STMT_RETURN,
 	STMT_ADD, STMT_DELETE,
 	STMT_LIST, STMT_EVENT_BODY_LIST,
-	STMT_INIT, STMT_HOOK,
+	STMT_INIT,
 	STMT_NULL
 #define NUM_STMTS (int(STMT_NULL) + 1)
 } BroStmtTag;

src/TCP.cc

@@ -382,7 +382,7 @@ void TCP_Analyzer::ProcessFIN(double t, TCP_Endpoint* endpoint,
 	endpoint->FIN_seq = base_seq - endpoint->StartSeq() + seq_len;
 	}
 
-bool TCP_Analyzer::ProcessRST(double t, TCP_Endpoint* endpoint,
+void TCP_Analyzer::ProcessRST(double t, TCP_Endpoint* endpoint,
 				const IP_Hdr* ip, uint32 base_seq,
 				int len, int& seq_len)
 	{
@@ -406,11 +406,9 @@ bool TCP_Analyzer::ProcessRST(double t, TCP_Endpoint* endpoint,
 		}
 
 	PacketWithRST();
-
-	return true;
 	}
 
-int TCP_Analyzer::ProcessFlags(double t,
+void TCP_Analyzer::ProcessFlags(double t,
 				const IP_Hdr* ip, const struct tcphdr* tp,
 				uint32 tcp_hdr_len, int len, int& seq_len,
 				TCP_Endpoint* endpoint, TCP_Endpoint* peer,
@@ -425,14 +423,11 @@ int TCP_Analyzer::ProcessFlags(double t,
 	if ( flags.FIN() )
 		ProcessFIN(t, endpoint, seq_len, base_seq);
 
-	if ( flags.RST() &&
-	     ! ProcessRST(t, endpoint, ip, base_seq, len, seq_len) )
-		return 0;
+	if ( flags.RST() )
+		ProcessRST(t, endpoint, ip, base_seq, len, seq_len);
 
 	if ( flags.ACK() )
 		ProcessACK(endpoint, peer, ack_seq, is_orig, flags);
-
-	return 1;
 	}
 
 void TCP_Analyzer::TransitionFromInactive(double t, TCP_Endpoint* endpoint,
@@ -825,10 +820,27 @@ void TCP_Analyzer::UpdateClosedState(double t, TCP_Endpoint* endpoint,
 		}
 	}
 
-void TCP_Analyzer::UpdateResetState(int len, TCP_Flags flags)
+void TCP_Analyzer::UpdateResetState(int len, TCP_Flags flags,
+                                    TCP_Endpoint* endpoint, uint32 base_seq,
+                                    uint32 last_seq)
 	{
 	if ( flags.SYN() )
+		{
 		Weird("SYN_after_reset");
+
+		if ( endpoint->prev_state == TCP_ENDPOINT_INACTIVE )
+			{
+			// Seq. numbers were initialized by a RST packet from this endpoint,
+			// but now that a SYN is seen from it, that could mean the earlier
+			// RST was spoofed/injected, so re-initialize.  This mostly just
+			// helps prevent misrepresentations of payload sizes that are based
+			// on bad initial sequence values.
+			endpoint->InitStartSeq(base_seq);
+			endpoint->InitAckSeq(base_seq);
+			endpoint->InitLastSeq(last_seq);
+			}
+		}
+
 	if ( flags.FIN() )
 		Weird("FIN_after_reset");
 
@@ -871,7 +883,7 @@ void TCP_Analyzer::UpdateStateMachine(double t,
 		break;
 
 	case TCP_ENDPOINT_RESET:
-		UpdateResetState(len, flags);
+		UpdateResetState(len, flags, endpoint, base_seq, last_seq);
 		break;
 	}
 	}
@@ -996,10 +1008,8 @@ void TCP_Analyzer::DeliverPacket(int len, const u_char* data, bool is_orig,
 
 	int seq_len = len;	// length in terms of sequence space
 
-	if ( ! ProcessFlags(t, ip, tp, tcp_hdr_len, len, seq_len,
-				endpoint, peer, base_seq, ack_seq,
-				orig_addr, is_orig, flags) )
-		return;
+	ProcessFlags(t, ip, tp, tcp_hdr_len, len, seq_len, endpoint, peer, base_seq,
+	             ack_seq, orig_addr, is_orig, flags);
 
 	uint32 last_seq = base_seq + seq_len;
 

src/TCP.h

@@ -135,13 +135,13 @@ protected:
 	void ProcessFIN(double t, TCP_Endpoint* endpoint, int& seq_len,
 			uint32 base_seq);
 
-	bool ProcessRST(double t, TCP_Endpoint* endpoint, const IP_Hdr* ip,
+	void ProcessRST(double t, TCP_Endpoint* endpoint, const IP_Hdr* ip,
 			uint32 base_seq, int len, int& seq_len);
 
 	void ProcessACK(TCP_Endpoint* endpoint, TCP_Endpoint* peer,
 			uint32 ack_seq, int is_orig, TCP_Flags flags);
 
-	int ProcessFlags(double t, const IP_Hdr* ip, const struct tcphdr* tp,
+	void ProcessFlags(double t, const IP_Hdr* ip, const struct tcphdr* tp,
 			uint32 tcp_hdr_len, int len, int& seq_len,
 			TCP_Endpoint* endpoint, TCP_Endpoint* peer,
 			uint32 base_seq, uint32 ack_seq,
@@ -186,7 +186,8 @@ protected:
 				int delta_last, TCP_Flags flags,
 				int& do_close);
 
-	void UpdateResetState(int len, TCP_Flags flags);
+	void UpdateResetState(int len, TCP_Flags flags, TCP_Endpoint* endpoint,
+				uint32 base_seq, uint32 last_seq);
 
 	void GeneratePacketEvent(TCP_Endpoint* endpoint, TCP_Endpoint* peer,
 					uint32 base_seq, uint32 ack_seq,

src/Trigger.cc

@@ -217,8 +217,15 @@ bool Trigger::Eval()
 			Name());
 
 	Unref(v);
+	v = 0;
 	stmt_flow_type flow;
-	v = body->Exec(f, flow);
+
+	try
+		{
+		v = body->Exec(f, flow);
+		}
+	catch ( InterpreterException& e )
+		{ /* Already reported. */ }
 
 	if ( is_return )
 		{
@@ -300,7 +307,14 @@ void Trigger::Timeout()
 		{
 		stmt_flow_type flow;
 		Frame* f = frame->Clone();
-		Val* v = timeout_stmts->Exec(f, flow);
+		Val* v = 0;
+
+		try
+			{
+			v = timeout_stmts->Exec(f, flow);
+			}
+		catch ( InterpreterException& e )
+			{ /* Already reported. */ }
 
 		if ( is_return )
 			{
@@ -382,7 +396,7 @@ void Trigger::Attach(Trigger *trigger)
 
 void Trigger::Cache(const CallExpr* expr, Val* v)
 	{
-	if ( disabled )
+	if ( disabled || ! v )
 		return;
 
 	ValCache::iterator i = cache.find(expr);

src/Type.cc

@@ -30,6 +30,7 @@ const char* type_name(TypeTag t)
 		"table", "union", "record", "types",
 		"func",
 		"file",
+		"opaque",
 		"vector",
 		"type",
 		"error",
@@ -96,6 +97,7 @@ BroType::BroType(TypeTag t, bool arg_base_type)
 	case TYPE_LIST:
 	case TYPE_FUNC:
 	case TYPE_FILE:
+	case TYPE_OPAQUE:
 	case TYPE_VECTOR:
 	case TYPE_TYPE:
 		internal_tag = TYPE_INTERNAL_OTHER;
@@ -114,8 +116,17 @@ BroType::~BroType()
 		delete [] type_id;
 	}
 
-int BroType::MatchesIndex(ListExpr*& /* index */) const
+int BroType::MatchesIndex(ListExpr*& index) const
 	{
+	if ( Tag() == TYPE_STRING )
+		{
+		if ( index->Exprs().length() != 1 && index->Exprs().length() != 2 )
+			return DOES_NOT_MATCH_INDEX;
+
+		if ( check_and_promote_exprs_to_type(index, ::base_type(TYPE_INT)) )
+			return MATCHES_INDEX_SCALAR;
+		}
+
 	return DOES_NOT_MATCH_INDEX;
 	}
 
@@ -1262,6 +1273,42 @@ bool FileType::DoUnserialize(UnserialInfo* info)
 	return yield != 0;
 	}
 
+OpaqueType::OpaqueType(const string& arg_name) : BroType(TYPE_OPAQUE)
+	{
+	name = arg_name;
+	}
+
+void OpaqueType::Describe(ODesc* d) const
+	{
+	if ( d->IsReadable() )
+		d->AddSP("opaque of");
+	else
+		d->Add(int(Tag()));
+
+	d->Add(name.c_str());
+	}
+
+IMPLEMENT_SERIAL(OpaqueType, SER_OPAQUE_TYPE);
+
+bool OpaqueType::DoSerialize(SerialInfo* info) const
+	{
+	DO_SERIALIZE(SER_OPAQUE_TYPE, BroType);
+	return SERIALIZE(name);
+	}
+
+bool OpaqueType::DoUnserialize(UnserialInfo* info)
+	{
+	DO_UNSERIALIZE(BroType);
+
+	char const* n;
+	if ( ! UNSERIALIZE_STR(&n, 0) )
+		return false;
+
+	name = n;
+	delete [] n;
+	return true;
+	}
+
 EnumType::EnumType(const string& arg_name)
 : BroType(TYPE_ENUM)
 	{
@@ -1716,6 +1763,13 @@ int same_type(const BroType* t1, const BroType* t2, int is_init)
 	case TYPE_FILE:
 		return same_type(t1->YieldType(), t2->YieldType(), is_init);
 
+	case TYPE_OPAQUE:
+		{
+		const OpaqueType* ot1 = (const OpaqueType*) t1;
+		const OpaqueType* ot2 = (const OpaqueType*) t2;
+		return ot1->Name() == ot2->Name() ? 1 : 0;
+		}
+
 	case TYPE_TYPE:
 		return same_type(t1, t2, is_init);
 
@@ -1805,6 +1859,7 @@ int is_assignable(BroType* t)
 
 	case TYPE_VECTOR:
 	case TYPE_FILE:
+	case TYPE_OPAQUE:
 	case TYPE_TABLE:
 	case TYPE_TYPE:
 		return 1;
@@ -2190,3 +2245,18 @@ BroType* init_type(Expr* init)
 
 	return new SetType(t->AsTypeList(), 0);
 	}
+
+bool is_atomic_type(const BroType* t)
+	{
+	switch ( t->InternalType() ) {
+	case TYPE_INTERNAL_INT:
+	case TYPE_INTERNAL_UNSIGNED:
+	case TYPE_INTERNAL_DOUBLE:
+	case TYPE_INTERNAL_STRING:
+	case TYPE_INTERNAL_ADDR:
+	case TYPE_INTERNAL_SUBNET:
+		return true;
+	default:
+		return false;
+	}
+	}

src/Type.h

@@ -29,6 +29,7 @@ typedef enum {
 	TYPE_LIST,
 	TYPE_FUNC,
 	TYPE_FILE,
+	TYPE_OPAQUE,
 	TYPE_VECTOR,
 	TYPE_TYPE,
 	TYPE_ERROR
@@ -499,6 +500,23 @@ protected:
 	BroType* yield;
 };
 
+class OpaqueType : public BroType {
+public:
+	OpaqueType(const string& name);
+	virtual ~OpaqueType() { };
+
+	const string& Name() const { return name; }
+
+	void Describe(ODesc* d) const;
+
+protected:
+	OpaqueType() { }
+
+	DECLARE_SERIAL(OpaqueType)
+
+	string name;
+};
+
 class EnumType : public BroType {
 public:
 	EnumType(const string& arg_name);
@@ -625,6 +643,9 @@ BroType* merge_type_list(ListExpr* elements);
 // Given an expression, infer its type when used for an initialization.
 extern BroType* init_type(Expr* init);
 
+// Returns true if argument is an atomic type.
+bool is_atomic_type(const BroType* t);
+
 // True if the given type tag corresponds to an integral type.
 #define IsIntegral(t)	(t == TYPE_INT || t == TYPE_COUNT || t == TYPE_COUNTER)
 

src/Val.cc

@@ -3114,9 +3114,33 @@ void VectorVal::ValDescribe(ODesc* d) const
 	d->Add("]");
 	}
 
+OpaqueVal::OpaqueVal(OpaqueType* t) : Val(t)
+	{
+	}
+
+OpaqueVal::~OpaqueVal()
+	{
+	}
+
+IMPLEMENT_SERIAL(OpaqueVal, SER_OPAQUE_VAL);
+
+bool OpaqueVal::DoSerialize(SerialInfo* info) const
+	{
+	DO_SERIALIZE(SER_OPAQUE_VAL, Val);
+	return true;
+	}
+
+bool OpaqueVal::DoUnserialize(UnserialInfo* info)
+	{
+	DO_UNSERIALIZE(Val);
+	return true;
+	}
 
 Val* check_and_promote(Val* v, const BroType* t, int is_init)
 	{
+	if ( ! v )
+		return 0;
+
 	BroType* vt = v->Type();
 
 	vt = flatten_type(vt);
@@ -3206,17 +3230,7 @@ int same_val(const Val* /* v1 */, const Val* /* v2 */)
 
 bool is_atomic_val(const Val* v)
 	{
-	switch ( v->Type()->InternalType() ) {
-	case TYPE_INTERNAL_INT:
-	case TYPE_INTERNAL_UNSIGNED:
-	case TYPE_INTERNAL_DOUBLE:
-	case TYPE_INTERNAL_STRING:
-	case TYPE_INTERNAL_ADDR:
-	case TYPE_INTERNAL_SUBNET:
-		return true;
-	default:
-		return false;
-	}
+	return is_atomic_type(v->Type());
 	}
 
 int same_atomic_val(const Val* v1, const Val* v2)

src/Val.h

@@ -1013,6 +1013,20 @@ protected:
 	VectorType* vector_type;
 };
 
+// Base class for values with types that are managed completely internally,
+// with no further script-level operators provided (other than bif
+// functions). See OpaqueVal.h for derived classes.
+class OpaqueVal : public Val {
+public:
+	OpaqueVal(OpaqueType* t);
+	virtual ~OpaqueVal();
+
+protected:
+	friend class Val;
+	OpaqueVal() { }
+
+	DECLARE_SERIAL(OpaqueVal);
+};
 
 // Checks the given value for consistency with the given type.  If an
 // exact match, returns it.  If promotable, returns the promoted version,

src/Var.cc

@@ -109,6 +109,36 @@ static void make_var(ID* id, BroType* t, init_class c, Expr* init,
 	if ( attr )
 		id->AddAttrs(new Attributes(attr, t, false));
 
+	if ( init )
+		{
+		switch ( init->Tag() ) {
+		case EXPR_TABLE_CONSTRUCTOR:
+			{
+			TableConstructorExpr* ctor = (TableConstructorExpr*) init;
+			if ( ctor->Attrs() )
+				{
+				::Ref(ctor->Attrs());
+				id->AddAttrs(ctor->Attrs());
+				}
+			}
+			break;
+
+		case EXPR_SET_CONSTRUCTOR:
+			{
+			SetConstructorExpr* ctor = (SetConstructorExpr*) init;
+			if ( ctor->Attrs() )
+				{
+				::Ref(ctor->Attrs());
+				id->AddAttrs(ctor->Attrs());
+				}
+			}
+			break;
+
+		default:
+			break;
+		}
+		}
+
 	if ( id->FindAttr(ATTR_PERSISTENT) || id->FindAttr(ATTR_SYNCHRONIZED) )
 		{
 		if ( dt == VAR_CONST )
@@ -180,7 +210,6 @@ static void make_var(ID* id, BroType* t, init_class c, Expr* init,
 		// defined.
 		Func* f = new BroFunc(id, 0, 0, 0, 0);
 		id->SetVal(new Val(f));
-		id->SetConst();
 		}
 	}
 
@@ -203,8 +232,9 @@ Stmt* add_local(ID* id, BroType* t, init_class c, Expr* init,
 
 		Ref(id);
 
+		Expr* name_expr = new NameExpr(id, dt == VAR_CONST);
 		Stmt* stmt =
-		    new ExprStmt(new AssignExpr(new NameExpr(id), init, 0, 0,
+		    new ExprStmt(new AssignExpr(name_expr, init, 0, 0,
 		        id->Attrs() ? id->Attrs()->Attrs() : 0 ));
 		stmt->SetLocationInfo(init->GetLocationInfo());
 
@@ -294,12 +324,12 @@ void add_type(ID* id, BroType* t, attr_list* attr, int /* is_event */)
 void begin_func(ID* id, const char* module_name, function_flavor flavor,
 		int is_redef, FuncType* t)
 	{
-	if ( flavor == FUNC_FLAVOR_EVENT || flavor == FUNC_FLAVOR_HOOK )
+	if ( flavor == FUNC_FLAVOR_EVENT )
 		{
 		const BroType* yt = t->YieldType();
 
 		if ( yt && yt->Tag() != TYPE_VOID )
-			id->Error("event/hook cannot yield a value", t);
+			id->Error("event cannot yield a value", t);
 
 		t->ClearYieldType(flavor);
 		}

src/bro.bif

@@ -530,82 +530,7 @@ function piped_exec%(program: string, to_write: string%): bool
 	%}
 
 %%{
-static void hash_md5_val(val_list& vlist, unsigned char digest[16])
-	{
-	MD5_CTX h;
-
-	md5_init(&h);
-	loop_over_list(vlist, i)
-		{
-		Val* v = vlist[i];
-		if ( v->Type()->Tag() == TYPE_STRING )
-			{
-			const BroString* str = v->AsString();
-			md5_update(&h, str->Bytes(), str->Len());
-			}
-		else
-			{
-			ODesc d(DESC_BINARY);
-			v->Describe(&d);
-			md5_update(&h, (const u_char *) d.Bytes(), d.Len());
-			}
-		}
-	md5_final(&h, digest);
-	}
-
-static void hmac_md5_val(val_list& vlist, unsigned char digest[16])
-	{
-	hash_md5_val(vlist, digest);
-	for ( int i = 0; i < 16; ++i )
-		digest[i] = digest[i] ^ shared_hmac_md5_key[i];
-	MD5(digest, 16, digest);
-	}
-
-static void hash_sha1_val(val_list& vlist, unsigned char digest[20])
-	{
-	SHA_CTX h;
-
-	sha1_init(&h);
-	loop_over_list(vlist, i)
-		{
-		Val* v = vlist[i];
-		if ( v->Type()->Tag() == TYPE_STRING )
-			{
-			const BroString* str = v->AsString();
-			sha1_update(&h, str->Bytes(), str->Len());
-			}
-		else
-			{
-			ODesc d(DESC_BINARY);
-			v->Describe(&d);
-			sha1_update(&h, (const u_char *) d.Bytes(), d.Len());
-			}
-		}
-	sha1_final(&h, digest);
-	}
-
-static void hash_sha256_val(val_list& vlist, unsigned char digest[32])
-	{
-	SHA256_CTX h;
-
-	sha256_init(&h);
-	loop_over_list(vlist, i)
-		{
-		Val* v = vlist[i];
-		if ( v->Type()->Tag() == TYPE_STRING )
-			{
-			const BroString* str = v->AsString();
-			sha256_update(&h, str->Bytes(), str->Len());
-			}
-		else
-			{
-			ODesc d(DESC_BINARY);
-			v->Describe(&d);
-			sha256_update(&h, (const u_char *) d.Bytes(), d.Len());
-			}
-		}
-	sha256_final(&h, digest);
-	}
+#include "OpaqueVal.h"
 %%}
 
 ## Computes the MD5 hash value of the provided list of arguments.
@@ -623,8 +548,8 @@ static void hash_sha256_val(val_list& vlist, unsigned char digest[32])
 ##      friends.
 function md5_hash%(...%): string
 	%{
-	unsigned char digest[16];
-	hash_md5_val(@ARG@, digest);
+	unsigned char digest[MD5_DIGEST_LENGTH];
+	MD5Val::digest(@ARG@, digest);
 	return new StringVal(md5_digest_print(digest));
 	%}
 
@@ -643,8 +568,8 @@ function md5_hash%(...%): string
 ##      friends.
 function sha1_hash%(...%): string
 	%{
-	unsigned char digest[20];
-	hash_sha1_val(@ARG@, digest);
+	unsigned char digest[SHA_DIGEST_LENGTH];
+	SHA1Val::digest(@ARG@, digest);
 	return new StringVal(sha1_digest_print(digest));
 	%}
 
@@ -663,8 +588,8 @@ function sha1_hash%(...%): string
 ##      friends.
 function sha256_hash%(...%): string
 	%{
-	unsigned char digest[32];
-	hash_sha256_val(@ARG@, digest);
+	unsigned char digest[SHA256_DIGEST_LENGTH];
+	SHA256Val::digest(@ARG@, digest);
 	return new StringVal(sha256_digest_print(digest));
 	%}
 
@@ -679,288 +604,183 @@ function sha256_hash%(...%): string
 ##    sha256_hash sha256_hash_init sha256_hash_update sha256_hash_finish
 function md5_hmac%(...%): string
 	%{
-	unsigned char digest[16];
-	hmac_md5_val(@ARG@, digest);
-	return new StringVal(md5_digest_print(digest));
+	unsigned char hmac[MD5_DIGEST_LENGTH];
+	MD5Val::hmac(@ARG@, shared_hmac_md5_key, hmac);
+	return new StringVal(md5_digest_print(hmac));
 	%}
 
-%%{
-static map<BroString, MD5_CTX> md5_states;
-static map<BroString, SHA_CTX> sha1_states;
-static map<BroString, SHA256_CTX> sha256_states;
-
-BroString* convert_index_to_string(Val* index)
-	{
-	ODesc d;
-	index->Describe(&d);
-	BroString* s = new BroString(1, d.TakeBytes(), d.Len());
-	s->SetUseFreeToDelete(1);
-	return s;
-	}
-%%}
-
-## Initializes MD5 state to enable incremental hash computation. After
-## initializing the MD5 state with this function, you can feed data to
-## :bro:id:`md5_hash_update` and finally need to call :bro:id:`md5_hash_finish`
-## to finish the computation and get the final hash value.
+## Constructs an MD5 handle to enable incremental hash computation. You can
+## feed data to the returned opaque value with :bro:id:`md5_hash_update` and
+## eventually need to call :bro:id:`md5_hash_finish` to finish the computation
+## and get the hash digest as result.
 ##
 ## For example, when computing incremental MD5 values of transferred files in
-## multiple concurrent HTTP connections, one would call ``md5_hash_init(c$id)``
-## once before invoking ``md5_hash_update(c$id, some_more_data)`` in the
+## multiple concurrent HTTP connections, one keeps an optional handle in the
+## HTTP session record. Then, one would call
+## ``c$http$md5_handle = md5_hash_init()`` once before invoking
+## ``md5_hash_update(c$http$md5_handle, some_more_data)`` in the
 ## :bro:id:`http_entity_data` event handler. When all data has arrived, a call
 ## to :bro:id:`md5_hash_finish` returns the final hash value.
 ## 
-## index: The unique identifier to associate with this hash computation.
+## Returns: The opaque handle associated with this hash computation.
 ##
 ## .. bro:see:: md5_hmac md5_hash md5_hash_update md5_hash_finish
 ##    sha1_hash sha1_hash_init sha1_hash_update sha1_hash_finish
 ##    sha256_hash sha256_hash_init sha256_hash_update sha256_hash_finish
-function md5_hash_init%(index: any%): bool
+function md5_hash_init%(%): opaque of md5
 	%{
-	BroString* s = convert_index_to_string(index);
-	int status = 0;
-
-	if ( md5_states.count(*s) < 1 )
-		{
-		MD5_CTX h;
-		md5_init(&h);
-		md5_states[*s] = h;
-		status = 1;
-		}
-
-	delete s;
-	return new Val(status, TYPE_BOOL);
+	HashVal* digest = new MD5Val();
+	digest->Init();
+	return digest;
 	%}
 
-## Initializes SHA1 state to enable incremental hash computation. After
-## initializing the SHA1 state with this function, you can feed data to
-## :bro:id:`sha1_hash_update` and finally need to call
-## :bro:id:`sha1_hash_finish` to finish the computation and get the final hash
-## value.
+## Constructs an SHA1 handle to enable incremental hash computation. You can
+## feed data to the returned opaque value with :bro:id:`sha1_hash_update` and
+## finally need to call :bro:id:`sha1_hash_finish` to finish the computation
+## and get the hash digest as result.
 ##
 ## For example, when computing incremental SHA1 values of transferred files in
-## multiple concurrent HTTP connections, one would call ``sha1_hash_init(c$id)``
-## once before invoking ``sha1_hash_update(c$id, some_more_data)`` in the
+## multiple concurrent HTTP connections, one keeps an optional handle in the
+## HTTP session record. Then, one would call
+## ``c$http$sha1_handle = sha1_hash_init()`` ## once before invoking
+## ``sha1_hash_update(c$http$sha1_handle, some_more_data)`` in the
 ## :bro:id:`http_entity_data` event handler. When all data has arrived, a call
 ## to :bro:id:`sha1_hash_finish` returns the final hash value.
 ##
-## index: The unique identifier to associate with this hash computation.
+## Returns: The opaque handle associated with this hash computation.
 ##
 ## .. bro:see:: md5_hmac md5_hash md5_hash_init md5_hash_update md5_hash_finish
 ##    sha1_hash sha1_hash_update sha1_hash_finish
 ##    sha256_hash sha256_hash_init sha256_hash_update sha256_hash_finish
-function sha1_hash_init%(index: any%): bool
+function sha1_hash_init%(%): opaque of sha1
 	%{
-	BroString* s = convert_index_to_string(index);
-	int status = 0;
-
-	if ( sha1_states.count(*s) < 1 )
-		{
-		SHA_CTX h;
-		sha1_init(&h);
-		sha1_states[*s] = h;
-		status = 1;
-		}
-
-	delete s;
-	return new Val(status, TYPE_BOOL);
+	HashVal* digest = new SHA1Val();
+	digest->Init();
+	return digest;
 	%}
 
-## Initializes SHA256 state to enable incremental hash computation. After
-## initializing the SHA256 state with this function, you can feed data to
-## :bro:id:`sha256_hash_update` and finally need to call
-## :bro:id:`sha256_hash_finish` to finish the computation and get the final hash
-## value.
+## Constructs an SHA256 handle to enable incremental hash computation. You can
+## feed data to the returned opaque value with :bro:id:`sha256_hash_update` and
+## finally need to call :bro:id:`sha256_hash_finish` to finish the computation
+## and get the hash digest as result.
 ##
 ## For example, when computing incremental SHA256 values of transferred files in
-## multiple concurrent HTTP connections, one would call
-## ``sha256_hash_init(c$id)`` once before invoking
-## ``sha256_hash_update(c$id, some_more_data)`` in the
+## multiple concurrent HTTP connections, one keeps an optional handle in the
+## HTTP session record. Then, one would call
+## ``c$http$sha256_handle = sha256_hash_init()`` ## once before invoking
+## ``sha256_hash_update(c$http$sha256_handle, some_more_data)`` in the
 ## :bro:id:`http_entity_data` event handler. When all data has arrived, a call
 ## to :bro:id:`sha256_hash_finish` returns the final hash value.
 ##
-## index: The unique identifier to associate with this hash computation.
+## Returns: The opaque handle associated with this hash computation.
 ##
 ## .. bro:see:: md5_hmac md5_hash md5_hash_init md5_hash_update md5_hash_finish
 ##    sha1_hash sha1_hash_init sha1_hash_update sha1_hash_finish
 ##    sha256_hash sha256_hash_update sha256_hash_finish
-function sha256_hash_init%(index: any%): bool
+function sha256_hash_init%(%): opaque of sha256
 	%{
-	BroString* s = convert_index_to_string(index);
-	int status = 0;
-
-	if ( sha256_states.count(*s) < 1 )
-		{
-		SHA256_CTX h;
-		sha256_init(&h);
-		sha256_states[*s] = h;
-		status = 1;
-		}
-
-	delete s;
-	return new Val(status, TYPE_BOOL);
+	HashVal* digest = new SHA256Val();
+	digest->Init();
+	return digest;
 	%}
 
-## Update the MD5 value associated with a given index. It is required to
+## Updates the MD5 value associated with a given index. It is required to
 ## call :bro:id:`md5_hash_init` once before calling this
 ## function.
 ##
-## index: The unique identifier to associate with this hash computation.
+## handle: The opaque handle associated with this hash computation.
 ##
 ## data: The data to add to the hash computation.
 ##
+## Returns: True on success.
+##
 ## .. bro:see:: md5_hmac md5_hash md5_hash_init md5_hash_finish
 ##    sha1_hash sha1_hash_init sha1_hash_update sha1_hash_finish
 ##    sha256_hash sha256_hash_init sha256_hash_update sha256_hash_finish
-function md5_hash_update%(index: any, data: string%): bool
+function md5_hash_update%(handle: opaque of md5, data: string%): bool
 	%{
-	BroString* s = convert_index_to_string(index);
-	int status = 0;
-
-	if ( md5_states.count(*s) > 0 )
-		{
-		md5_update(&md5_states[*s], data->Bytes(), data->Len());
-		status = 1;
-		}
-
-	delete s;
-	return new Val(status, TYPE_BOOL);
+	bool rc = static_cast<HashVal*>(handle)->Feed(data->Bytes(), data->Len());
+	return new Val(rc, TYPE_BOOL);
 	%}
 
-## Update the SHA1 value associated with a given index. It is required to
+## Updates the SHA1 value associated with a given index. It is required to
 ## call :bro:id:`sha1_hash_init` once before calling this
 ## function.
 ##
-## index: The unique identifier to associate with this hash computation.
+## handle: The opaque handle associated with this hash computation.
 ##
 ## data: The data to add to the hash computation.
 ##
+## Returns: True on success.
+##
 ## .. bro:see:: md5_hmac md5_hash md5_hash_init md5_hash_update md5_hash_finish
 ##    sha1_hash sha1_hash_init sha1_hash_finish
 ##    sha256_hash sha256_hash_init sha256_hash_update sha256_hash_finish
-function sha1_hash_update%(index: any, data: string%): bool
+function sha1_hash_update%(handle: opaque of sha1, data: string%): bool
 	%{
-	BroString* s = convert_index_to_string(index);
-	int status = 0;
-
-	if ( sha1_states.count(*s) > 0 )
-		{
-		sha1_update(&sha1_states[*s], data->Bytes(), data->Len());
-		status = 1;
-		}
-
-	delete s;
-	return new Val(status, TYPE_BOOL);
+	bool rc = static_cast<HashVal*>(handle)->Feed(data->Bytes(), data->Len());
+	return new Val(rc, TYPE_BOOL);
 	%}
 
-## Update the SHA256 value associated with a given index. It is required to
+## Updates the SHA256 value associated with a given index. It is required to
 ## call :bro:id:`sha256_hash_init` once before calling this
 ## function.
 ##
-## index: The unique identifier to associate with this hash computation.
+## handle: The opaque handle associated with this hash computation.
 ##
 ## data: The data to add to the hash computation.
 ##
+## Returns: True on success.
+##
 ## .. bro:see:: md5_hmac md5_hash md5_hash_init md5_hash_update md5_hash_finish
 ##    sha1_hash sha1_hash_init sha1_hash_update sha1_hash_finish
 ##    sha256_hash sha256_hash_init sha256_hash_finish
-function sha256_hash_update%(index: any, data: string%): bool
+function sha256_hash_update%(handle: opaque of sha256, data: string%): bool
 	%{
-	BroString* s = convert_index_to_string(index);
-	int status = 0;
-
-	if ( sha256_states.count(*s) > 0 )
-		{
-		sha256_update(&sha256_states[*s], data->Bytes(), data->Len());
-		status = 1;
-		}
-
-	delete s;
-	return new Val(status, TYPE_BOOL);
+	bool rc = static_cast<HashVal*>(handle)->Feed(data->Bytes(), data->Len());
+	return new Val(rc, TYPE_BOOL);
 	%}
 
 ## Returns the final MD5 digest of an incremental hash computation.
 ##
-## index: The unique identifier of this hash computation.
+## handle: The opaque handle associated with this hash computation.
 ##
-## Returns: The hash value associated with the computation at *index*.
+## Returns: The hash value associated with the computation of *handle*.
 ##
 ## .. bro:see:: md5_hmac md5_hash md5_hash_init md5_hash_update
 ##    sha1_hash sha1_hash_init sha1_hash_update sha1_hash_finish
 ##    sha256_hash sha256_hash_init sha256_hash_update sha256_hash_finish
-function md5_hash_finish%(index: any%): string
+function md5_hash_finish%(handle: opaque of md5%): string
 	%{
-	BroString* s = convert_index_to_string(index);
-	StringVal* printable_digest;
-
-	if ( md5_states.count(*s) > 0 )
-		{
-		unsigned char digest[16];
-		md5_final(&md5_states[*s], digest);
-		md5_states.erase(*s);
-		printable_digest = new StringVal(md5_digest_print(digest));
-		}
-	else
-		printable_digest = new StringVal("");
-
-	delete s;
-	return printable_digest;
+	return static_cast<HashVal*>(handle)->Get();
 	%}
 
 ## Returns the final SHA1 digest of an incremental hash computation.
 ##
-## index: The unique identifier of this hash computation.
+## handle: The opaque handle associated with this hash computation.
 ##
-## Returns: The hash value associated with the computation at *index*.
+## Returns: The hash value associated with the computation of *handle*.
 ##
 ## .. bro:see:: md5_hmac md5_hash md5_hash_init md5_hash_update md5_hash_finish
 ##    sha1_hash sha1_hash_init sha1_hash_update
 ##    sha256_hash sha256_hash_init sha256_hash_update sha256_hash_finish
-function sha1_hash_finish%(index: any%): string
+function sha1_hash_finish%(handle: opaque of sha1%): string
 	%{
-	BroString* s = convert_index_to_string(index);
-	StringVal* printable_digest;
-
-	if ( sha1_states.count(*s) > 0 )
-		{
-		unsigned char digest[20];
-		sha1_final(&sha1_states[*s], digest);
-		sha1_states.erase(*s);
-		printable_digest = new StringVal(sha1_digest_print(digest));
-		}
-	else
-		printable_digest = new StringVal("");
-
-	delete s;
-	return printable_digest;
+	return static_cast<HashVal*>(handle)->Get();
 	%}
 
 ## Returns the final SHA256 digest of an incremental hash computation.
 ##
-## index: The unique identifier of this hash computation.
+## handle: The opaque handle associated with this hash computation.
 ##
-## Returns: The hash value associated with the computation at *index*.
+## Returns: The hash value associated with the computation of *handle*.
 ##
 ## .. bro:see:: md5_hmac md5_hash md5_hash_init md5_hash_update md5_hash_finish
 ##    sha1_hash sha1_hash_init sha1_hash_update sha1_hash_finish
 ##    sha256_hash sha256_hash_init sha256_hash_update
-function sha256_hash_finish%(index: any%): string
+function sha256_hash_finish%(handle: opaque of sha256%): string
 	%{
-	BroString* s = convert_index_to_string(index);
-	StringVal* printable_digest;
-
-	if ( sha256_states.count(*s) > 0 )
-		{
-		unsigned char digest[32];
-		sha256_final(&sha256_states[*s], digest);
-		sha256_states.erase(*s);
-		printable_digest = new StringVal(sha256_digest_print(digest));
-		}
-	else
-		printable_digest = new StringVal("");
-
-	delete s;
-	return printable_digest;
+	return static_cast<HashVal*>(handle)->Get();
 	%}
 
 ## Generates a random number.
@@ -1058,11 +878,6 @@ function identify_data%(data: string, return_mime: bool%): string
 	return new StringVal(descr);
 	%}
 
-%%{
-#include <RandTest.h>
-static map<BroString, RandTest*> entropy_states;
-%%}
-
 ## Performs an entropy test on the given data.
 ## See http://www.fourmilab.ch/random.
 ##
@@ -1107,13 +922,11 @@ function find_entropy%(data: string%): entropy_test_result
 	%{
 	double montepi, scc, ent, mean, chisq;
 	montepi = scc = ent = mean = chisq = 0.0;
+	EntropyVal e;
+	e.Feed(data->Bytes(), data->Len());
+	e.Get(&ent, &chisq, &mean, &montepi, &scc);
+
 	RecordVal* ent_result = new RecordVal(entropy_test_result);
-	RandTest *rt = new RandTest();
-
-	rt->add((char*) data->Bytes(), data->Len());
-	rt->end(&ent, &chisq, &mean, &montepi, &scc);
-	delete rt;
-
 	ent_result->Assign(0, new Val(ent,     TYPE_DOUBLE));
 	ent_result->Assign(1, new Val(chisq,   TYPE_DOUBLE));
 	ent_result->Assign(2, new Val(mean,    TYPE_DOUBLE));
@@ -1124,85 +937,52 @@ function find_entropy%(data: string%): entropy_test_result
 
 ## Initializes data structures for incremental entropy calculation.
 ##
-## index: An arbitrary unique value per distinct computation.
-##
-## Returns: True on success.
+## Returns: An opaque handle to be used in subsequent operations.
 ##
 ## .. bro:see:: find_entropy entropy_test_add entropy_test_finish
-function entropy_test_init%(index: any%): bool
+function entropy_test_init%(%): opaque of entropy
 	%{
-	BroString* s = convert_index_to_string(index);
-	int status = 0;
-
-	if ( entropy_states.count(*s) < 1 )
-		{
-		entropy_states[*s] = new RandTest();
-		status = 1;
-		}
-
-	delete s;
-	return new Val(status, TYPE_BOOL);
+	return new EntropyVal();
 	%}
 
-## Adds data to an incremental entropy calculation. Before using this function,
-## one needs to invoke :bro:id:`entropy_test_init`.
+## Adds data to an incremental entropy calculation.
+##
+## handle: The opaque handle representing the entropy calculation state.
 ##
 ## data: The data to add to the entropy calculation.
 ##
-## index: An arbitrary unique value that identifies a particular entropy
-##        computation.
-##
 ## Returns: True on success.
 ##
 ## .. bro:see:: find_entropy entropy_test_add entropy_test_finish
-function entropy_test_add%(index: any, data: string%): bool
+function entropy_test_add%(handle: opaque of entropy, data: string%): bool
 	%{
-	BroString* s = convert_index_to_string(index);
-	int status = 0;
-
-	if ( entropy_states.count(*s) > 0 )
-		{
-		entropy_states[*s]->add((char*) data->Bytes(), data->Len());
-		status = 1;
-		}
-
-	delete s;
+	bool status = static_cast<EntropyVal*>(handle)->Feed(data->Bytes(),
+	                                                     data->Len());
 	return new Val(status, TYPE_BOOL);
 	%}
 
 ## Finishes an incremental entropy calculation. Before using this function,
-## one needs to initialize the computation with :bro:id:`entropy_test_init` and
+## one needs to obtain an opaque handle with :bro:id:`entropy_test_init` and
 ## add data to it via :bro:id:`entropy_test_add`.
 ##
-## index: An arbitrary unique value that identifies a particular entropy
-##        computation.
+## handle: The opaque handle representing the entropy calculation state.
 ##
 ## Returns: The result of the entropy test. See :bro:id:`find_entropy` for a
 ##          description of the individual components.
 ##
 ## .. bro:see:: find_entropy entropy_test_init entropy_test_add
-function entropy_test_finish%(index: any%): entropy_test_result
+function entropy_test_finish%(handle: opaque of entropy%): entropy_test_result
 	%{
-	BroString* s = convert_index_to_string(index);
 	double montepi, scc, ent, mean, chisq;
 	montepi = scc = ent = mean = chisq = 0.0;
+	static_cast<EntropyVal*>(handle)->Get(&ent, &chisq, &mean, &montepi, &scc);
+
 	RecordVal* ent_result = new RecordVal(entropy_test_result);
-
-	if ( entropy_states.count(*s) > 0 )
-		{
-		RandTest *rt = entropy_states[*s];
-		rt->end(&ent, &chisq, &mean, &montepi, &scc);
-		entropy_states.erase(*s);
-		delete rt;
-		}
-
 	ent_result->Assign(0, new Val(ent,     TYPE_DOUBLE));
 	ent_result->Assign(1, new Val(chisq,   TYPE_DOUBLE));
 	ent_result->Assign(2, new Val(mean,    TYPE_DOUBLE));
 	ent_result->Assign(3, new Val(montepi, TYPE_DOUBLE));
 	ent_result->Assign(4, new Val(scc,     TYPE_DOUBLE));
-
-	delete s;
 	return ent_result;
 	%}
 
@@ -2684,8 +2464,9 @@ function to_port%(s: string%): port
     if ( s->Len() < 10 )
         {
         char* slash;
+        errno = 0;
         port = strtol(s->CheckString(), &slash, 10);
-        if ( port )
+        if ( ! errno )
             {
             ++slash;
             if ( streq(slash, "tcp") )

src/builtin-func.l

@@ -74,7 +74,9 @@ HEX	[0-9a-fA-F]+
 "set"		return check_c_mode(TOK_SET);
 "table"		return check_c_mode(TOK_TABLE);
 "vector"	return check_c_mode(TOK_VECTOR);
-"module"    return check_c_mode(TOK_MODULE);
+"of"            return check_c_mode(TOK_OF);
+"opaque"        return check_c_mode(TOK_OPAQUE);
+"module"        return check_c_mode(TOK_MODULE);
 
 "@ARG@"		return TOK_ARG;
 "@ARGS@"	return TOK_ARGS;

src/builtin-func.y

@@ -269,15 +269,15 @@ void print_event_c_body(FILE *fp)
 
 %token TOK_LPP TOK_RPP TOK_LPB TOK_RPB TOK_LPPB TOK_RPPB TOK_VAR_ARG
 %token TOK_BOOL
-%token TOK_FUNCTION TOK_EVENT TOK_CONST TOK_ENUM
-%token TOK_TYPE TOK_RECORD TOK_SET TOK_VECTOR TOK_TABLE TOK_MODULE
+%token TOK_FUNCTION TOK_EVENT TOK_CONST TOK_ENUM TOK_OF
+%token TOK_TYPE TOK_RECORD TOK_SET TOK_VECTOR TOK_OPAQUE TOK_TABLE TOK_MODULE
 %token TOK_ARGS TOK_ARG TOK_ARGC
 %token TOK_ID TOK_ATTR TOK_CSTR TOK_LF TOK_WS TOK_COMMENT
 %token TOK_ATOM TOK_INT TOK_C_TOKEN
 
 %left ',' ':'
 
-%type <str> TOK_C_TOKEN TOK_ID TOK_CSTR TOK_WS TOK_COMMENT TOK_ATTR TOK_INT opt_ws
+%type <str> TOK_C_TOKEN TOK_ID TOK_CSTR TOK_WS TOK_COMMENT TOK_ATTR TOK_INT opt_ws type
 %type <val> TOK_ATOM TOK_BOOL
 
 %union	{
@@ -584,7 +584,17 @@ args_1:		args_1 ',' opt_ws arg opt_ws
 			{ /* empty */ }
 	;
 
-arg:		TOK_ID opt_ws ':' opt_ws TOK_ID
+// TODO: Migrate all other compound types to this rule. Once the BiF language
+// can parse all regular Bro types, we can throw out the unnecessary
+// boilerplate typedefs for addr_set, string_set, etc.
+type:
+                TOK_OPAQUE opt_ws TOK_OF opt_ws TOK_ID
+                        { $$ = concat("opaque of ", $5); }
+        |       TOK_ID
+                        { $$ = $1; }
+        ;
+
+arg:		TOK_ID opt_ws ':' opt_ws type
 			{ args.push_back(new BuiltinFuncArg($1, $5)); }
 	|	TOK_VAR_ARG
 			{
@@ -594,7 +604,7 @@ arg:		TOK_ID opt_ws ':' opt_ws TOK_ID
 			}
 	;
 
-return_type:	':' opt_ws TOK_ID opt_ws
+return_type:	':' opt_ws type opt_ws
 			{
 			BuiltinFuncArg* ret = new BuiltinFuncArg("", $3);
 			ret->PrintBro(fp_bro_init);

src/const.bif

@@ -15,8 +15,10 @@ const Tunnel::max_depth: count;
 const Tunnel::enable_ip: bool;
 const Tunnel::enable_ayiya: bool;
 const Tunnel::enable_teredo: bool;
+const Tunnel::enable_gtpv1: bool;
 const Tunnel::yielding_teredo_decapsulation: bool;
 const Tunnel::delay_teredo_confirmation: bool;
+const Tunnel::delay_gtp_confirmation: bool;
 const Tunnel::ip_tunnel_timeout: interval;
 
 const Threading::heartbeat_interval: interval;

src/event.bif

@@ -577,6 +577,19 @@ event teredo_origin_indication%(outer: connection, inner: teredo_hdr%);
 ##    it may become particularly expensive for real-time analysis.
 event teredo_bubble%(outer: connection, inner: teredo_hdr%);
 
+## Generated for GTPv1 G-PDU packets.  That is, packets with a UDP payload
+## that includes a GTP header followed by an IPv4 or IPv6 packet.
+##
+## outer: The GTP outer tunnel connection.
+##
+## inner_gtp: The GTP header.
+##
+## inner_ip: The inner IP and transport layer packet headers.
+##
+## .. note:: Since this event may be raised on a per-packet basis, handling
+##    it may become particularly expensive for real-time analysis.
+event gtpv1_g_pdu_packet%(outer: connection, inner_gtp: gtpv1_hdr, inner_ip: pkt_hdr%);
+
 ## Generated for every packet that has a non-empty transport-layer payload.
 ## This is a very low-level and expensive event that should be avoided when
 ## at all possible.  It's usually infeasible to handle when processing even
@@ -858,7 +871,7 @@ event udp_contents%(u: connection, is_orig: bool, contents: string%);
 ## Generated when a UDP session for a supported protocol has finished. Some of
 ## Bro's application-layer UDP analyzers flag the end of a session by raising
 ## this event. Currently, the analyzers for DNS, NTP, Netbios, Syslog, AYIYA,
-## and Teredo support this.
+## Teredo, and GTPv1 support this.
 ##
 ## u: The connection record for the corresponding UDP flow.
 ##

src/gtpv1-analyzer.pac

@@ -0,0 +1,161 @@
+
+connection GTPv1_Conn(bro_analyzer: BroAnalyzer)
+	{
+	upflow = GTPv1_Flow(true);
+	downflow = GTPv1_Flow(false);
+
+	%member{
+		bool valid_orig;
+		bool valid_resp;
+	%}
+
+	%init{
+		valid_orig = valid_resp = false;
+	%}
+
+	function valid(orig: bool): bool
+		%{
+		return orig ? valid_orig : valid_resp;
+		%}
+
+	function set_valid(orig: bool, val: bool): void
+		%{
+		if ( orig )
+			valid_orig = val;
+		else
+			valid_resp = val;
+		%}
+	}
+
+%code{
+inline void violate(const char* r, const BroAnalyzer& a, const bytestring& p)
+	{
+	a->ProtocolViolation(r, (const char*) p.data(), p.length());
+	}
+%}
+
+flow GTPv1_Flow(is_orig: bool)
+	{
+	datagram = GTPv1_Header withcontext(connection, this);
+
+	function process_gtpv1(pdu: GTPv1_Header): bool
+		%{
+		BroAnalyzer a = connection()->bro_analyzer();
+		Connection *c = a->Conn();
+		const EncapsulationStack* e = c->GetEncapsulation();
+
+		connection()->set_valid(is_orig(), false);
+
+		if ( e && e->Depth() >= BifConst::Tunnel::max_depth )
+			{
+			reporter->Weird(c, "tunnel_depth");
+			return false;
+			}
+
+		if ( e && e->LastType() == BifEnum::Tunnel::GTPv1 )
+			{
+			// GTP is never tunneled in GTP so, this must be a regular packet
+			violate("GTP-in-GTP", a, ${pdu.packet});
+			return false;
+			}
+
+		if ( ${pdu.version} != 1 )
+			{
+			// Only know of GTPv1 with Version == 1
+			violate("GTPv1 bad Version", a, ${pdu.packet});
+			return false;
+			}
+
+		if ( ! ${pdu.pt_flag} )
+			{
+			// Not interested in GTP'
+			return false;
+			}
+
+		if ( ${pdu.e_flag} )
+			{
+			// TODO: can't currently parse past extension headers
+			return false;
+			}
+
+		if ( ${pdu.msg_type} != 0xff )
+			{
+			// Only interested in decapsulating user plane data beyond here.
+			return false;
+			}
+
+		if ( ${pdu.packet}.length() < (int)sizeof(struct ip) )
+			{
+			violate("Truncated GTPv1", a, ${pdu.packet});
+			return false;
+			}
+
+		const struct ip* ip = (const struct ip*) ${pdu.packet}.data();
+
+		if ( ip->ip_v != 4 && ip->ip_v != 6 )
+			{
+			violate("non-IP packet in GTPv1", a, ${pdu.packet});
+			return false;
+			}
+
+		IP_Hdr* inner = 0;
+		int result = sessions->ParseIPPacket(${pdu.packet}.length(),
+		     ${pdu.packet}.data(), ip->ip_v == 6 ? IPPROTO_IPV6 : IPPROTO_IPV4,
+		     inner);
+
+		if ( result == 0 )
+			{
+			connection()->set_valid(is_orig(), true);
+
+			if ( (! BifConst::Tunnel::delay_gtp_confirmation) ||
+			     (connection()->valid(true) && connection()->valid(false)) )
+				a->ProtocolConfirmation();
+			}
+
+		else if ( result < 0 )
+			violate("Truncated GTPv1", a, ${pdu.packet});
+
+		else
+			violate("GTPv1 payload length", a, ${pdu.packet});
+
+		if ( result != 0 )
+			{
+			delete inner;
+			return false;
+			}
+
+		if ( ::gtpv1_g_pdu_packet )
+			{
+			RecordVal* rv = new RecordVal(gtpv1_hdr_type);
+
+			rv->Assign(0, new Val(${pdu.version}, TYPE_COUNT));
+			rv->Assign(1, new Val(${pdu.pt_flag}, TYPE_BOOL));
+			rv->Assign(2, new Val(${pdu.rsv}, TYPE_BOOL));
+			rv->Assign(3, new Val(${pdu.e_flag}, TYPE_BOOL));
+			rv->Assign(4, new Val(${pdu.s_flag}, TYPE_BOOL));
+			rv->Assign(5, new Val(${pdu.pn_flag}, TYPE_BOOL));
+			rv->Assign(6, new Val(${pdu.msg_type}, TYPE_COUNT));
+			rv->Assign(7, new Val(ntohs(${pdu.length}), TYPE_COUNT));
+			rv->Assign(8, new Val(ntohl(${pdu.teid}), TYPE_COUNT));
+
+			if ( ${pdu.has_opt} )
+				{
+				rv->Assign(9, new Val(ntohs(${pdu.opt_hdr.seq}), TYPE_COUNT));
+				rv->Assign(10, new Val(${pdu.opt_hdr.n_pdu}, TYPE_COUNT));
+				rv->Assign(11, new Val(${pdu.opt_hdr.next_type}, TYPE_COUNT));
+				}
+
+			BifEvent::generate_gtpv1_g_pdu_packet(a, c, rv,
+			                                      inner->BuildPktHdrVal());
+			}
+
+		EncapsulatingConn ec(c, BifEnum::Tunnel::GTPv1);
+
+		sessions->DoNextInnerPacket(network_time(), 0, inner, e, ec);
+
+		return (result == 0) ? true : false;
+		%}
+
+	};
+
+refine typeattr GTPv1_Header += &let { proc_gtpv1 = $context.flow.process_gtpv1(this); };

src/gtpv1-protocol.pac

@@ -0,0 +1,27 @@
+
+type GTPv1_Header = record {
+	flags:     uint8;
+	msg_type:  uint8;
+	length:    uint16;
+	teid:      uint32;
+	opt:       case has_opt of {
+		true  -> opt_hdr: GTPv1_Opt_Header;
+		false -> no_opt:  empty;
+	} &requires(has_opt);
+	packet:    bytestring &restofdata;
+
+} &let {
+	version:  uint8 = (flags & 0xE0) >> 5;
+	pt_flag:  bool  = flags & 0x10;
+	rsv:      bool  = flags & 0x08;
+	e_flag:   bool  = flags & 0x04;
+	s_flag:   bool  = flags & 0x02;
+	pn_flag:  bool  = flags & 0x01;
+	has_opt:  bool  = flags & 0x07;
+} &byteorder = littleendian;
+
+type GTPv1_Opt_Header = record {
+	seq:       uint16;
+	n_pdu:     uint8;
+	next_type: uint8;
+}

src/gtpv1.pac

@@ -0,0 +1,10 @@
+%include binpac.pac
+%include bro.pac
+
+analyzer GTPv1 withcontext {
+	connection:	GTPv1_Conn;
+	flow:		GTPv1_Flow;
+};
+
+%include gtpv1-protocol.pac
+%include gtpv1-analyzer.pac

src/input/readers/Ascii.cc

@@ -68,7 +68,7 @@ Ascii::Ascii(ReaderFrontend *frontend) : ReaderBackend(frontend)
 	unset_field.assign( (const char*) BifConst::InputAscii::unset_field->Bytes(),
 			    BifConst::InputAscii::unset_field->Len());
 
-	io = new AsciiInputOutput(this, set_separator, unset_field, empty_field);	
+	io = new AsciiInputOutput(this, AsciiInputOutput::SeparatorInfo(set_separator, unset_field, empty_field));	
 }
 
 Ascii::~Ascii()
@@ -213,8 +213,6 @@ bool Ascii::GetLine(string& str)
 	return false;
 	}
 
-
-
 // read the entire file and send appropriate thingies back to InputMgr
 bool Ascii::DoUpdate()
 	{
@@ -325,7 +323,7 @@ bool Ascii::DoUpdate()
 				return false;
 				}
 
-			Value* val = io->EntryToVal(stringfields[(*fit).position], (*fit).name, (*fit).type, (*fit).subtype);
+			Value* val = io->StringToVal(stringfields[(*fit).position], (*fit).name, (*fit).type, (*fit).subtype);
 			if ( val == 0 )
 				{
 				Error(Fmt("Could not convert line '%s' to Val. Ignoring line.", line.c_str()));
@@ -354,7 +352,7 @@ bool Ascii::DoUpdate()
 			// array structure.
 
 			for ( int i = 0; i < fpos; i++ )
-				delete fields[fpos];
+				delete fields[i];
 
 			delete [] fields;
 			continue;

src/input/readers/Ascii.h

@@ -7,7 +7,7 @@
 #include <vector>
 
 #include "../ReaderBackend.h"
-#include "../../AsciiInputOutput.h"
+#include "../../threading/AsciiInputOutput.h"
 
 namespace input { namespace reader {
 

src/input/readers/Benchmark.cc

@@ -26,7 +26,7 @@ Benchmark::Benchmark(ReaderFrontend *frontend) : ReaderBackend(frontend)
 	timedspread = double(BifConst::InputBenchmark::timedspread);
 	heartbeat_interval = double(BifConst::Threading::heartbeat_interval);
 
-	io = new AsciiInputOutput(this);
+	io = new AsciiInputOutput(this, AsciiInputOutput::SeparatorInfo());
 	}
 
 Benchmark::~Benchmark()

src/input/readers/Benchmark.h

@@ -4,7 +4,7 @@
 #define INPUT_READERS_BENCHMARK_H
 
 #include "../ReaderBackend.h"
-#include "../../AsciiInputOutput.h"
+#include "../../threading/AsciiInputOutput.h"
 
 namespace input { namespace reader {
 

src/logging/writers/Ascii.cc

@@ -19,7 +19,7 @@ Ascii::Ascii(WriterFrontend* frontend) : WriterBackend(frontend)
 	{
 	fd = 0;
 	ascii_done = false;
-	only_single_header_row = false;
+	tsv = false;
 
 	output_to_stdout = BifConst::LogAscii::output_to_stdout;
 	include_meta = BifConst::LogAscii::include_meta;
@@ -52,7 +52,7 @@ Ascii::Ascii(WriterFrontend* frontend) : WriterBackend(frontend)
 	desc.EnableEscaping();
 	desc.AddEscapeSequence(separator);
 
-	io = new AsciiInputOutput(this, set_separator, unset_field, empty_field);
+	io = new AsciiInputOutput(this, AsciiInputOutput::SeparatorInfo(set_separator, unset_field, empty_field));
 	}
 
 Ascii::~Ascii()
@@ -78,7 +78,7 @@ void Ascii::CloseFile(double t)
 	if ( ! fd )
 		return;
 
-	if ( include_meta && ! only_single_header_row )
+	if ( include_meta && ! tsv )
 		WriteHeaderField("close", Timestamp(0));
 
 	safe_close(fd);
@@ -108,17 +108,17 @@ bool Ascii::DoInit(const WriterInfo& info, int num_fields, const Field* const *
 
 	for ( WriterInfo::config_map::const_iterator i = info.config.begin(); i != info.config.end(); i++ )
 		{
-		if ( strcmp(i->first, "only_single_header_row") == 0 )
+		if ( strcmp(i->first, "tsv") == 0 )
 			{
 			if ( strcmp(i->second, "T") == 0 )
-				only_single_header_row = true;
+				tsv = true;
 
 			else if ( strcmp(i->second, "F") == 0 )
-				only_single_header_row = false;
+				tsv = false;
 
 			else
 				{
-				Error("invalid value for 'only_single_header_row', must be boolean (T/F)");
+				Error("invalid value for 'tsv', must be a string and either \"T\" or \"F\"");
 				return false;
 				}
 			}
@@ -141,9 +141,9 @@ bool Ascii::DoInit(const WriterInfo& info, int num_fields, const Field* const *
 			types += fields[i]->TypeName().c_str();
 			}
 
-		if ( only_single_header_row )
+		if ( tsv )
 			{
-			// A single CSV-style line is all we need.
+			// A single TSV-style line is all we need.
 			string str = names + "\n";
 			if ( ! safe_write(fd, str.c_str(), str.length()) )
 				goto write_error;

src/logging/writers/Ascii.h

@@ -6,7 +6,7 @@
 #define LOGGING_WRITER_ASCII_H
 
 #include "../WriterBackend.h"
-#include "../../AsciiInputOutput.h"
+#include "../../threading/AsciiInputOutput.h"
 
 namespace logging { namespace writer {
 
@@ -45,7 +45,7 @@ private:
 	// Options set from the script-level.
 	bool output_to_stdout;
 	bool include_meta;
-	bool only_single_header_row;
+	bool tsv;
 
 	string separator;
 	string set_separator;

src/logging/writers/ElasticSearch.cc

@@ -16,7 +16,7 @@
 #include "BroString.h"
 #include "NetVar.h"
 #include "threading/SerialTypes.h"
-#include "../../AsciiInputOutput.h"
+#include "../../threading/AsciiInputOutput.h"
 
 #include <curl/curl.h>
 #include <curl/easy.h>

src/parse.y

@@ -11,13 +11,13 @@
 %token TOK_DOUBLE TOK_ELSE TOK_ENUM TOK_EVENT TOK_EXPORT TOK_FILE TOK_FOR
 %token TOK_FUNCTION TOK_GLOBAL TOK_HOOK TOK_ID TOK_IF TOK_INT
 %token TOK_INTERVAL TOK_LIST TOK_LOCAL TOK_MODULE
-%token TOK_NEXT TOK_OF TOK_PATTERN TOK_PATTERN_TEXT
+%token TOK_NEXT TOK_OF TOK_OPAQUE TOK_PATTERN TOK_PATTERN_TEXT
 %token TOK_PORT TOK_PRINT TOK_RECORD TOK_REDEF
 %token TOK_REMOVE_FROM TOK_RETURN TOK_SCHEDULE TOK_SET
 %token TOK_STRING TOK_SUBNET TOK_SWITCH TOK_TABLE
 %token TOK_TIME TOK_TIMEOUT TOK_TIMER TOK_TYPE TOK_UNION TOK_VECTOR TOK_WHEN
 
-%token TOK_ATTR_ADD_FUNC TOK_ATTR_ATTR TOK_ATTR_ENCRYPT TOK_ATTR_DEFAULT
+%token TOK_ATTR_ADD_FUNC TOK_ATTR_ENCRYPT TOK_ATTR_DEFAULT
 %token TOK_ATTR_OPTIONAL TOK_ATTR_REDEF TOK_ATTR_ROTATE_INTERVAL
 %token TOK_ATTR_ROTATE_SIZE TOK_ATTR_DEL_FUNC TOK_ATTR_EXPIRE_FUNC
 %token TOK_ATTR_EXPIRE_CREATE TOK_ATTR_EXPIRE_READ TOK_ATTR_EXPIRE_WRITE
@@ -32,6 +32,7 @@
 
 %token TOK_NO_TEST
 
+%nonassoc TOK_HOOK
 %left ',' '|'
 %right '=' TOK_ADD_TO TOK_REMOVE_FROM
 %right '?' ':'
@@ -56,7 +57,6 @@
 %type <re> pattern
 %type <expr> expr init anonymous_function
 %type <event_expr> event
-%type <call_expr> hook
 %type <stmt> stmt stmt_list func_body for_head
 %type <type> type opt_type enum_body
 %type <func_type> func_hdr func_params
@@ -119,6 +119,7 @@ extern const char* g_curr_debug_error;
 
 #define YYLTYPE yyltype
 
+static int in_hook = 0;
 int in_init = 0;
 int in_record = 0;
 bool resolving_global_ID = false;
@@ -212,7 +213,6 @@ static std::list<std::string>* concat_opt_docs (std::list<std::string>* pre,
 	Val* val;
 	RE_Matcher* re;
 	Expr* expr;
-	CallExpr* call_expr;
 	EventExpr* event_expr;
 	Stmt* stmt;
 	ListExpr* list;
@@ -418,6 +418,14 @@ expr:
 			$$ = new IndexExpr($1, $3);
 			}
 
+	|	expr '[' expr ':' expr ']'
+			{
+			set_location(@1, @6);
+			ListExpr* le = new ListExpr($3);
+			le->Append($5);
+			$$ = new IndexExpr($1, le, true);
+			}
+
 	|	expr '$' TOK_ID
 			{
 			set_location(@1, @3);
@@ -517,7 +525,16 @@ expr:
 	|	expr '(' opt_expr_list ')'
 			{
 			set_location(@1, @4);
-			$$ = new CallExpr($1, $3);
+			$$ = new CallExpr($1, $3, in_hook > 0);
+			}
+
+	|	TOK_HOOK { ++in_hook; } expr
+			{
+			--in_hook;
+			set_location(@1, @3);
+			if ( $3->Tag() != EXPR_CALL )
+				$3->Error("not a valid hook call expression");
+			$$ = $3;
 			}
 
 	|	expr TOK_HAS_FIELD TOK_ID
@@ -875,7 +892,7 @@ type:
 	|	TOK_HOOK '(' formal_args ')'
 				{
 				set_location(@1, @3);
-				$$ = new FuncType($3, 0, FUNC_FLAVOR_HOOK);
+				$$ = new FuncType($3, base_type(TYPE_BOOL), FUNC_FLAVOR_HOOK);
 				}
 
 	|	TOK_FILE TOK_OF type
@@ -890,6 +907,12 @@ type:
 				$$ = new FileType(base_type(TYPE_STRING));
 				}
 
+	|	TOK_OPAQUE TOK_OF TOK_ID
+				{
+				set_location(@1, @3);
+				$$ = new OpaqueType($3);
+				}
+
 	|	resolve_id
 			{
 			if ( ! $1 || ! ($$ = $1->AsType()) )
@@ -1209,6 +1232,8 @@ func_hdr:
 			}
 	|	TOK_HOOK def_global_id func_params
 			{
+			$3->ClearYieldType(FUNC_FLAVOR_HOOK);
+			$3->SetYieldType(base_type(TYPE_BOOL));
 			begin_func($2, current_module.c_str(),
 				   FUNC_FLAVOR_HOOK, 0, $3);
 			$$ = $3;
@@ -1372,14 +1397,6 @@ stmt:
 			    brofiler.AddStmt($$);
 			}
 
-	|	TOK_HOOK hook ';' opt_no_test
-			{
-			set_location(@1, @4);
-			$$ = new HookStmt($2);
-			if ( ! $4 )
-			    brofiler.AddStmt($$);
-			}
-
 	|	TOK_IF '(' expr ')' stmt
 			{
 			set_location(@1, @4);
@@ -1533,14 +1550,6 @@ event:
 			}
 	;
 
-hook:
-		expr '(' opt_expr_list ')'
-			{
-			set_location(@1, @4);
-			$$ = new CallExpr($1, $3, true);
-			}
-	;
-
 case_list:
 		case_list case
 			{ $1->append($2); }

src/patricia.c

@@ -66,6 +66,9 @@ static char copyright[] =
 
 #define Delete free
 
+// From Bro for reporting memory exhaustion.
+extern void out_of_memory(const char* where);
+
 /* { from prefix.c */
 
 /* prefix_tochar
@@ -251,6 +254,9 @@ New_Prefix2 (int family, void *dest, int bitlen, prefix_t *prefix)
         default_bitlen = 128;
 	if (prefix == NULL) {
             prefix = calloc(1, sizeof (prefix_t));
+            if (prefix == NULL)
+                out_of_memory("patrica/new_prefix2: unable to allocate memory");
+
 	    dynamic_allocated++;
 	}
 	memcpy (&prefix->add.sin6, dest, 16);
@@ -260,10 +266,14 @@ New_Prefix2 (int family, void *dest, int bitlen, prefix_t *prefix)
 		if (prefix == NULL) {
 #ifndef NT
             prefix = calloc(1, sizeof (prefix4_t));
+            if (prefix == NULL)
+                out_of_memory("patrica/new_prefix2: unable to allocate memory");
 #else
 			//for some reason, compiler is getting
 			//prefix4_t size incorrect on NT
 			prefix = calloc(1, sizeof (prefix_t));
+                        if (prefix == NULL)
+                            out_of_memory("patrica/new_prefix2: unable to allocate memory");
 #endif /* NT */
 
 			dynamic_allocated++;
@@ -396,6 +406,8 @@ patricia_tree_t *
 New_Patricia (int maxbits)
 {
     patricia_tree_t *patricia = calloc(1, sizeof *patricia);
+    if (patricia == NULL)
+        out_of_memory("patrica/new_patricia: unable to allocate memory");
 
     patricia->maxbits = maxbits;
     patricia->head = NULL;
@@ -665,6 +677,9 @@ patricia_lookup (patricia_tree_t *patricia, prefix_t *prefix)
 
     if (patricia->head == NULL) {
 	node = calloc(1, sizeof *node);
+        if (node == NULL)
+            out_of_memory("patrica/patrica_lookup: unable to allocate memory");
+
 	node->bit = prefix->bitlen;
 	node->prefix = Ref_Prefix (prefix);
 	node->parent = NULL;
@@ -776,6 +791,9 @@ patricia_lookup (patricia_tree_t *patricia, prefix_t *prefix)
     }
 
     new_node = calloc(1, sizeof *new_node);
+    if (new_node == NULL)
+        out_of_memory("patrica/patrica_lookup: unable to allocate memory");
+
     new_node->bit = prefix->bitlen;
     new_node->prefix = Ref_Prefix (prefix);
     new_node->parent = NULL;
@@ -828,6 +846,9 @@ patricia_lookup (patricia_tree_t *patricia, prefix_t *prefix)
     }
     else {
         glue = calloc(1, sizeof *glue);
+        if (glue == NULL)
+            out_of_memory("patrica/patrica_lookup: unable to allocate memory");
+
         glue->bit = differ_bit;
         glue->prefix = NULL;
         glue->parent = node->parent;

src/scan.l

@@ -298,6 +298,7 @@ local	return TOK_LOCAL;
 module	return TOK_MODULE;
 next	return TOK_NEXT;
 of	return TOK_OF;
+opaque	return TOK_OPAQUE;
 pattern	return TOK_PATTERN;
 port	return TOK_PORT;
 print	return TOK_PRINT;
@@ -319,7 +320,6 @@ vector	return TOK_VECTOR;
 when	return TOK_WHEN;
 
 &add_func	return TOK_ATTR_ADD_FUNC;
-&attr		return TOK_ATTR_ATTR;
 &create_expire	return TOK_ATTR_EXPIRE_CREATE;
 &default	return TOK_ATTR_DEFAULT;
 &delete_func	return TOK_ATTR_DEL_FUNC;

src/threading/AsciiInputOutput.cc

@@ -5,26 +5,25 @@
 #include <sstream>
 #include <errno.h>
 #include "AsciiInputOutput.h"
-#include "bro_inet_ntop.h"
+#include "../bro_inet_ntop.h"
 
-AsciiInputOutput::AsciiInputOutput(threading::MsgThread* t) 
+AsciiInputOutput::AsciiInputOutput(threading::MsgThread* t, const SeparatorInfo info) 
 	{
 	thread = t;
+	this->separators = info;
 	}
 
-AsciiInputOutput::AsciiInputOutput(threading::MsgThread* t, const string & set_separator, 
+AsciiInputOutput::SeparatorInfo::SeparatorInfo(const string & set_separator, 
 				 const string & unset_field, const string & empty_field) 
 	{
-	thread = t;
 	this->set_separator = set_separator;
 	this->unset_field = unset_field;
 	this->empty_field = empty_field;
 	}
 
-AsciiInputOutput::AsciiInputOutput(threading::MsgThread* t, const string & set_separator, 
+AsciiInputOutput::SeparatorInfo::SeparatorInfo(const string & set_separator, 
 				 const string & unset_field) 
 	{
-	thread = t;
 	this->set_separator = set_separator;
 	this->unset_field = unset_field;
 	}
@@ -38,7 +37,7 @@ bool AsciiInputOutput::ValToODesc(ODesc* desc, threading::Value* val, const thre
 	{
 	if ( ! val->present )
 		{
-		desc->Add(unset_field);
+		desc->Add(separators.unset_field);
 		return true;
 		}
 
@@ -94,11 +93,11 @@ bool AsciiInputOutput::ValToODesc(ODesc* desc, threading::Value* val, const thre
 
 		if ( ! size )
 			{
-			desc->Add(empty_field);
+			desc->Add(separators.empty_field);
 			break;
 			}
 
-		if ( size == unset_field.size() && memcmp(data, unset_field.data(), size) == 0 )
+		if ( size == separators.unset_field.size() && memcmp(data, separators.unset_field.data(), size) == 0 )
 			{
 			// The value we'd write out would match exactly the
 			// place-holder we use for unset optional fields. We
@@ -124,24 +123,24 @@ bool AsciiInputOutput::ValToODesc(ODesc* desc, threading::Value* val, const thre
 		{
 		if ( ! val->val.set_val.size )
 			{
-			desc->Add(empty_field);
+			desc->Add(separators.empty_field);
 			break;
 			}
 
-		desc->AddEscapeSequence(set_separator);
+		desc->AddEscapeSequence(separators.set_separator);
 		for ( int j = 0; j < val->val.set_val.size; j++ )
 			{
 			if ( j > 0 )
-				desc->AddRaw(set_separator);
+				desc->AddRaw(separators.set_separator);
 
 			assert(field != 0);
 			if ( ! ValToODesc(desc, val->val.set_val.vals[j], field) )
 				{
-				desc->RemoveEscapeSequence(set_separator);
+				desc->RemoveEscapeSequence(separators.set_separator);
 				return false;
 				}
 			}
-		desc->RemoveEscapeSequence(set_separator);
+		desc->RemoveEscapeSequence(separators.set_separator);
 
 		break;
 		}
@@ -150,24 +149,24 @@ bool AsciiInputOutput::ValToODesc(ODesc* desc, threading::Value* val, const thre
 		{
 		if ( ! val->val.vector_val.size )
 			{
-			desc->Add(empty_field);
+			desc->Add(separators.empty_field);
 			break;
 			}
 
-		desc->AddEscapeSequence(set_separator);
+		desc->AddEscapeSequence(separators.set_separator);
 		for ( int j = 0; j < val->val.vector_val.size; j++ )
 			{
 			if ( j > 0 )
-				desc->AddRaw(set_separator);
+				desc->AddRaw(separators.set_separator);
 
 			assert(field != 0);
 			if ( ! ValToODesc(desc, val->val.vector_val.vals[j], field) )
 				{
-				desc->RemoveEscapeSequence(set_separator);
+				desc->RemoveEscapeSequence(separators.set_separator);
 				return false;
 				}
 			}
-		desc->RemoveEscapeSequence(set_separator);
+		desc->RemoveEscapeSequence(separators.set_separator);
 
 		break;
 		}
@@ -181,9 +180,9 @@ bool AsciiInputOutput::ValToODesc(ODesc* desc, threading::Value* val, const thre
 	}
 
 
-threading::Value* AsciiInputOutput::EntryToVal(string s, string name, TypeTag type, TypeTag subtype) const
+threading::Value* AsciiInputOutput::StringToVal(string s, string name, TypeTag type, TypeTag subtype) const
 	{
-	if ( s.compare(unset_field) == 0 )  // field is not set...
+	if ( s.compare(separators.unset_field) == 0 )  // field is not set...
 		return new threading::Value(type, false);
 
 	threading::Value* val = new threading::Value(type, true);
@@ -207,14 +206,14 @@ threading::Value* AsciiInputOutput::EntryToVal(string s, string name, TypeTag ty
 			{
 			thread->Error(thread->Fmt("Field: %s Invalid value for boolean: %s",
 				  name.c_str(), s.c_str()));
-			return 0;
+			goto parse_error;
 			}
 		break;
 
 	case TYPE_INT:
 		val->val.int_val = strtoll(s.c_str(), &end, 10);
 		if ( CheckNumberError(s, end) )
-			return 0;
+			goto parse_error;
 		break;
 
 	case TYPE_DOUBLE:
@@ -222,20 +221,20 @@ threading::Value* AsciiInputOutput::EntryToVal(string s, string name, TypeTag ty
 	case TYPE_INTERVAL:
 		val->val.double_val = strtod(s.c_str(), &end);
 		if ( CheckNumberError(s, end) )
-			return 0;
+			goto parse_error;
 		break;
 
 	case TYPE_COUNT:
 	case TYPE_COUNTER:
 		val->val.uint_val = strtoull(s.c_str(), &end, 10);
 		if ( CheckNumberError(s, end) )
-			return 0;
+			goto parse_error;
 		break;
 
 	case TYPE_PORT:
 		val->val.port_val.port = strtoull(s.c_str(), &end, 10);
 		if ( CheckNumberError(s, end) )
-			return 0;
+			goto parse_error;
 
 		val->val.port_val.proto = TRANSPORT_UNKNOWN;
 		break;
@@ -247,13 +246,13 @@ threading::Value* AsciiInputOutput::EntryToVal(string s, string name, TypeTag ty
 		if ( pos == s.npos )
 			{
 			thread->Error(thread->Fmt("Invalid value for subnet: %s", s.c_str()));
-			return 0;
+			goto parse_error;
 			}
 
 		uint8_t width = (uint8_t) strtol(s.substr(pos+1).c_str(), &end, 10);
 
 		if ( CheckNumberError(s, end) )
-			return 0;
+			goto parse_error;
 
 		string addr = s.substr(0, pos);
 
@@ -278,13 +277,14 @@ threading::Value* AsciiInputOutput::EntryToVal(string s, string name, TypeTag ty
 		unsigned int length = 1;
 		for ( unsigned int i = 0; i < s.size(); i++ )
 			{
-			if ( s[i] == set_separator[0] )
+			if ( s[i] == separators.set_separator[0] )
 				length++;
 			}
 
 		unsigned int pos = 0;
+		bool error = false;
 
-		if ( empty_field.size() > 0 && s.compare(empty_field) == 0 )
+		if ( separators.empty_field.size() > 0 && s.compare(separators.empty_field) == 0 )
 			length = 0;
 
 		threading::Value** lvals = new threading::Value* [length];
@@ -312,21 +312,23 @@ threading::Value* AsciiInputOutput::EntryToVal(string s, string name, TypeTag ty
 			{
 			string element;
 
-			if ( ! getline(splitstream, element, set_separator[0]) )
+			if ( ! getline(splitstream, element, separators.set_separator[0]) )
 				break;
 
 			if ( pos >= length )
 				{
 				thread->Error(thread->Fmt("Internal error while parsing set. pos %d >= length %d."
 				          " Element: %s", pos, length, element.c_str()));
+				error = true;
 				break;
 				}
 
-			threading::Value* newval = EntryToVal(element, name, subtype);
+			threading::Value* newval = StringToVal(element, name, subtype);
 			if ( newval == 0 )
 				{
-				thread->Error("Error while reading set");
-				return 0;
+				thread->Error("Error while reading set or vector");
+				error = true;
+				break;
 				}
 
 			lvals[pos] = newval;
@@ -337,22 +339,32 @@ threading::Value* AsciiInputOutput::EntryToVal(string s, string name, TypeTag ty
 		// Test if the string ends with a set_separator... or if the
 		// complete string is empty. In either of these cases we have
 		// to push an empty val on top of it.
-		if ( s.empty() || *s.rbegin() == set_separator[0] )
+		if ( ! error && (s.empty() || *s.rbegin() == separators.set_separator[0]) )
 			{
-			lvals[pos] = EntryToVal("", name, subtype);
+			lvals[pos] = StringToVal("", name, subtype);
 			if ( lvals[pos] == 0 )
 				{
 				thread->Error("Error while trying to add empty set element");
-				return 0;
+				goto parse_error;
 				}
 
 			pos++;
 			}
 
+		if ( error ) {
+			// We had an error while reading a set or a vector.
+			// Hence we have to clean up the values that have
+			// been read so far
+			for ( unsigned int i = 0; i < pos; i++ )
+				delete lvals[i];
+
+			goto parse_error;
+		}
+
 		if ( pos != length )
 			{
 			thread->Error(thread->Fmt("Internal error while parsing set: did not find all elements: %s", s.c_str()));
-			return 0;
+			goto parse_error;
 			}
 
 		break;
@@ -361,10 +373,14 @@ threading::Value* AsciiInputOutput::EntryToVal(string s, string name, TypeTag ty
 	default:
 		thread->Error(thread->Fmt("unsupported field format %d for %s", type,
 		name.c_str()));
-		return 0;
+		goto parse_error;
 	}
 
 	return val;
+
+parse_error:
+	delete val;
+	return 0;	
 	}
 
 bool AsciiInputOutput::CheckNumberError(const string& s, const char * end) const

src/threading/AsciiInputOutput.h

@@ -3,34 +3,60 @@
 #ifndef AsciiInputOutput_h
 #define AsciiInputOutput_h
 
-#include "Desc.h"
-#include "threading/MsgThread.h"
+#include "../Desc.h"
+#include "MsgThread.h"
 
 class AsciiInputOutput {
 	public: 
-		// Constructor that leaves separators, etc empty.
-		// Use if you just need functionality like StringToAddr, etc.
-		AsciiInputOutput(threading::MsgThread*);
 
-		// Constructor that defines all separators, etc.
-		// Use if you need either ValToODesc or EntryToVal.
-		AsciiInputOutput(threading::MsgThread*, const string & set_separator, 
+		/** 
+		 * A struct to pass the necessary initialization values to the AsciiInputOutput module
+		 * on startup
+		 */
+		struct SeparatorInfo 
+			{
+			//const string separator;
+			string set_separator;
+			string empty_field;
+			string unset_field;
+			string meta_prefix;
+
+			// Constructor that leaves separators, etc empty.
+			// Use if you just need functionality like StringToAddr, etc.
+			SeparatorInfo() { };
+
+			// Constructor that defines all separators, etc.
+			// Use if you need either ValToODesc or EntryToVal.
+			SeparatorInfo(const string & set_separator, 
 				 const string & unset_field, const string & empty_field);
 
-		// Constructor that defines all separators, etc, besides empty_field, which is not needed for many
-		// non-ascii-based io sources.
-		// Use if you need either ValToODesc or EntryToVal.
-		AsciiInputOutput(threading::MsgThread*, const string & set_separator, 
+			// Constructor that defines all separators, etc, besides empty_field, which is not needed for many
+			// non-ascii-based io sources.
+			// Use if you need either ValToODesc or EntryToVal.
+			SeparatorInfo(const string & set_separator, 
 				 const string & unset_field);
-		~AsciiInputOutput();
+			};
 
+		/**
+		 * Constructor
+		 *
+		 * @param t The thread that uses this class instance. Used to access thread
+		 * message passing methods
+		 *
+		 * @param info
+		 * SeparatorInfo structure defining the necessary separators
+		 */
+		AsciiInputOutput(threading::MsgThread* t, const SeparatorInfo info);
+	
+		// Destructor
+		~AsciiInputOutput();
 
 		// converts a threading value to the corresponding ascii representation
 		// returns false & logs an error with reporter in case an error occurs
 		bool ValToODesc(ODesc* desc, threading::Value* val, const threading::Field* field) const;
 
 		// convert the ascii representation of a field into a Value
-		threading::Value* EntryToVal(string s, string name, TypeTag type, TypeTag subtype = TYPE_ERROR) const;
+		threading::Value* StringToVal(string s, string name, TypeTag type, TypeTag subtype = TYPE_ERROR) const;
 
 		/** Helper method to render an IP address as a string.
 		  *
@@ -75,11 +101,7 @@ class AsciiInputOutput {
 	private:
 		bool CheckNumberError(const string& s, const char * end) const;
 
-		string separator;
-		string set_separator;
-		string empty_field;
-		string unset_field;
-		string meta_prefix;
+		SeparatorInfo separators;
 
 		threading::MsgThread* thread;
 };

src/types.bif

@@ -185,6 +185,7 @@ enum Type %{
 	AYIYA,
 	TEREDO,
 	SOCKS,
+	GTPv1,
 %}
 
 type EncapsulatingConn: record;

src/util.cc

@@ -1416,7 +1416,7 @@ void safe_close(int fd)
 		}
 	}
 
-void out_of_memory(const char* where)
+extern "C" void out_of_memory(const char* where)
 	{
 	fprintf(stderr, "out of memory in %s.\n", where);
 

src/util.h

@@ -303,7 +303,7 @@ extern bool safe_write(int fd, const char* data, int len);
 // Wraps close(2) to emit error messages and abort on unrecoverable errors.
 extern void safe_close(int fd);
 
-extern void out_of_memory(const char* where);
+extern "C" void out_of_memory(const char* where);
 
 inline void* safe_realloc(void* ptr, size_t size)
 	{

testing/btest/Baseline/bifs.strptime/.stdout

@@ -1,2 +0,0 @@
-1350604800.0
-0.0

testing/btest/Baseline/bifs.strptime/out

@@ -0,0 +1,3 @@
+warning: strptime conversion failed: fmt:%m d:1980-10-24
+1350604800.0
+0.0

testing/btest/Baseline/bifs.strptime/reporter.log

@@ -1,10 +0,0 @@
-#separator \x09
-#set_separator	,
-#empty_field	(empty)
-#unset_field	-
-#path	reporter
-#open	2012-10-19-06-06-36
-#fields	ts	level	message	location
-#types	time	enum	string	string
-0.000000	Reporter::WARNING	strptime conversion failed: fmt:%m d:1980-10-24	(empty)
-#close	2012-10-19-06-06-36

testing/btest/Baseline/bifs.to_port/out

@@ -1,6 +1,9 @@
 123/tcp
 123/udp
 123/icmp
+0/tcp
+0/udp
+0/icmp
 0/unknown
 256/tcp
 256/udp

testing/btest/Baseline/core.tcp.rst-after-syn/.stdout

@@ -0,0 +1,3 @@
+[orig_h=1.2.0.2, orig_p=2527/tcp, resp_h=1.2.0.3, resp_p=6649/tcp]
+orig:, [size=175, state=1, num_pkts=4, num_bytes_ip=395, flow_label=0]
+resp:, [size=0, state=6, num_pkts=5, num_bytes_ip=236, flow_label=0]

testing/btest/Baseline/core.tunnels.gtp.different_dl_and_ul/conn.log

@@ -0,0 +1,12 @@
+#separator \x09
+#set_separator	,
+#empty_field	(empty)
+#unset_field	-
+#path	conn
+#open	2012-10-19-17-03-55
+#fields	ts	uid	id.orig_h	id.orig_p	id.resp_h	id.resp_p	proto	service	duration	orig_bytes	resp_bytes	conn_state	local_orig	missed_bytes	history	orig_pkts	orig_ip_bytes	resp_pkts	resp_ip_bytes	tunnel_parents
+#types	time	string	addr	port	addr	port	enum	string	interval	count	count	string	bool	count	string	count	count	count	count	table[string]
+1333458850.321642	arKYeMETxOg	10.131.17.170	51803	173.199.115.168	80	tcp	http	0.257902	1138	63424	S3	-	0	ShADadf	29	2310	49	65396	UWkUyAuUGXf,k6kgXLOoSKl
+1333458850.325787	k6kgXLOoSKl	207.233.125.40	2152	167.55.105.244	2152	udp	gtpv1	0.251127	65788	0	S0	-	0	D	49	67160	0	0	(empty)
+1333458850.321642	UWkUyAuUGXf	167.55.105.244	5906	207.233.125.40	2152	udp	gtpv1	0.257902	2542	0	S0	-	0	D	29	3354	0	0	(empty)
+#close	2012-10-19-17-03-55

testing/btest/Baseline/core.tunnels.gtp.different_dl_and_ul/http.log

@@ -0,0 +1,11 @@
+#separator \x09
+#set_separator	,
+#empty_field	(empty)
+#unset_field	-
+#path	http
+#open	2012-10-19-17-03-55
+#fields	ts	uid	id.orig_h	id.orig_p	id.resp_h	id.resp_p	trans_depth	method	host	uri	referrer	user_agent	request_body_len	response_body_len	status_code	status_msg	info_code	info_msg	filename	tags	username	password	proxied	mime_type	md5	extraction_file
+#types	time	string	addr	port	addr	port	count	string	string	string	string	string	count	count	count	string	count	string	string	table[enum]	string	string	table[string]	string	string	file
+1333458850.340368	arKYeMETxOg	10.131.17.170	51803	173.199.115.168	80	1	GET	cdn.epicgameads.com	/ads/flash/728x90_nx8com.swf?clickTAG=http://www.epicgameads.com/ads/bannerclickPage.php?id=e3ubwU6IF&pd=1&adid=0&icpc=1&axid=0&uctt=1&channel=4&cac=1&t=728x90&cb=1333458879	http://www.epicgameads.com/ads/banneriframe.php?id=e3ubwU6IF&t=728x90&channel=4&cb=1333458905296	Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Trident/5.0)	0	31461	200	OK	-	-	-	(empty)	-	-	-	application/x-shockwave-flash	-	-
+1333458850.399501	arKYeMETxOg	10.131.17.170	51803	173.199.115.168	80	2	GET	cdn.epicgameads.com	/ads/flash/728x90_nx8com.swf?clickTAG=http://www.epicgameads.com/ads/bannerclickPage.php?id=e3ubwU6IF&pd=1&adid=0&icpc=1&axid=0&uctt=1&channel=0&cac=1&t=728x90&cb=1333458881	http://www.epicgameads.com/ads/banneriframe.php?id=e3ubwU6IF&t=728x90&cb=1333458920207	Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Trident/5.0)	0	31461	200	OK	-	-	-	(empty)	-	-	-	application/x-shockwave-flash	-	-
+#close	2012-10-19-17-03-55

testing/btest/Baseline/core.tunnels.gtp.different_dl_and_ul/tunnel.log

@@ -0,0 +1,13 @@
+#separator \x09
+#set_separator	,
+#empty_field	(empty)
+#unset_field	-
+#path	tunnel
+#open	2012-10-19-17-03-55
+#fields	ts	uid	id.orig_h	id.orig_p	id.resp_h	id.resp_p	tunnel_type	action
+#types	time	string	addr	port	addr	port	enum	enum
+1333458850.321642	UWkUyAuUGXf	167.55.105.244	5906	207.233.125.40	2152	Tunnel::GTPv1	Tunnel::DISCOVER
+1333458850.325787	k6kgXLOoSKl	207.233.125.40	2152	167.55.105.244	2152	Tunnel::GTPv1	Tunnel::DISCOVER
+1333458850.579544	k6kgXLOoSKl	207.233.125.40	2152	167.55.105.244	2152	Tunnel::GTPv1	Tunnel::CLOSE
+1333458850.579544	UWkUyAuUGXf	167.55.105.244	5906	207.233.125.40	2152	Tunnel::GTPv1	Tunnel::CLOSE
+#close	2012-10-19-17-03-55

testing/btest/Baseline/core.tunnels.gtp.false_gtp/conn.log

@@ -0,0 +1,10 @@
+#separator \x09
+#set_separator	,
+#empty_field	(empty)
+#unset_field	-
+#path	conn
+#open	2012-10-19-17-07-44
+#fields	ts	uid	id.orig_h	id.orig_p	id.resp_h	id.resp_p	proto	service	duration	orig_bytes	resp_bytes	conn_state	local_orig	missed_bytes	history	orig_pkts	orig_ip_bytes	resp_pkts	resp_ip_bytes	tunnel_parents
+#types	time	string	addr	port	addr	port	enum	string	interval	count	count	string	bool	count	string	count	count	count	count	table[string]
+1333458871.219794	UWkUyAuUGXf	10.131.24.6	2152	195.178.38.3	53	udp	dns	-	-	-	S0	-	0	D	1	64	0	0	(empty)
+#close	2012-10-19-17-07-44

testing/btest/Baseline/core.tunnels.gtp.false_gtp/dns.log

@@ -0,0 +1,10 @@
+#separator \x09
+#set_separator	,
+#empty_field	(empty)
+#unset_field	-
+#path	dns
+#open	2012-10-19-17-07-44
+#fields	ts	uid	id.orig_h	id.orig_p	id.resp_h	id.resp_p	proto	trans_id	query	qclass	qclass_name	qtype	qtype_name	rcode	rcode_name	AA	TC	RD	RA	Z	answers	TTLs	rejected
+#types	time	string	addr	port	addr	port	enum	count	string	count	string	count	string	count	string	bool	bool	bool	bool	count	vector[string]	vector[interval]	bool
+1333458871.219794	UWkUyAuUGXf	10.131.24.6	2152	195.178.38.3	53	udp	27595	abcd.efg.hijklm.nm	1	C_INTERNET	1	A	-	-	F	F	T	F	0	-	-	F
+#close	2012-10-19-17-07-44

testing/btest/Baseline/core.tunnels.gtp.inner_ipv6/conn.log

@@ -0,0 +1,12 @@
+#separator \x09
+#set_separator	,
+#empty_field	(empty)
+#unset_field	-
+#path	conn
+#open	2012-10-19-17-21-27
+#fields	ts	uid	id.orig_h	id.orig_p	id.resp_h	id.resp_p	proto	service	duration	orig_bytes	resp_bytes	conn_state	local_orig	missed_bytes	history	orig_pkts	orig_ip_bytes	resp_pkts	resp_ip_bytes	tunnel_parents
+#types	time	string	addr	port	addr	port	enum	string	interval	count	count	string	bool	count	string	count	count	count	count	table[string]
+1333458851.770000	arKYeMETxOg	fe80::224c:4fff:fe43:414c	1234	ff02::1:3	5355	udp	dns	-	-	-	S0	-	0	D	1	80	0	0	UWkUyAuUGXf
+1333458851.770000	UWkUyAuUGXf	118.92.124.41	2152	118.92.124.72	2152	udp	gtpv1	0.199236	152	0	S0	-	0	D	2	208	0	0	(empty)
+1333458851.969236	k6kgXLOoSKl	fe80::224c:4fff:fe43:414c	133	ff02::2	134	icmp	-	-	-	-	OTH	-	0	-	1	56	0	0	UWkUyAuUGXf
+#close	2012-10-19-17-21-27

testing/btest/Baseline/core.tunnels.gtp.inner_ipv6/tunnel.log

@@ -0,0 +1,11 @@
+#separator \x09
+#set_separator	,
+#empty_field	(empty)
+#unset_field	-
+#path	tunnel
+#open	2012-10-19-17-21-27
+#fields	ts	uid	id.orig_h	id.orig_p	id.resp_h	id.resp_p	tunnel_type	action
+#types	time	string	addr	port	addr	port	enum	enum
+1333458851.770000	UWkUyAuUGXf	118.92.124.41	2152	118.92.124.72	2152	Tunnel::GTPv1	Tunnel::DISCOVER
+1333458851.969236	UWkUyAuUGXf	118.92.124.41	2152	118.92.124.72	2152	Tunnel::GTPv1	Tunnel::CLOSE
+#close	2012-10-19-17-21-27

testing/btest/Baseline/core.tunnels.gtp.inner_teredo/conn.log

@@ -0,0 +1,26 @@
+#separator \x09
+#set_separator	,
+#empty_field	(empty)
+#unset_field	-
+#path	conn
+#open	2012-10-19-17-34-25
+#fields	ts	uid	id.orig_h	id.orig_p	id.resp_h	id.resp_p	proto	service	duration	orig_bytes	resp_bytes	conn_state	local_orig	missed_bytes	history	orig_pkts	orig_ip_bytes	resp_pkts	resp_ip_bytes	tunnel_parents
+#types	time	string	addr	port	addr	port	enum	string	interval	count	count	string	bool	count	string	count	count	count	count	table[string]
+1333458850.037956	qCaWGmzFtM5	10.131.112.102	51403	94.245.121.253	3544	udp	teredo	-	-	-	SHR	-	0	d	0	0	1	84	GSxOnSLghOa
+1333458850.040098	70MGiRM1Qf4	174.94.190.229	2152	190.104.181.57	2152	udp	gtpv1	0.003698	192	0	S0	-	0	D	2	248	0	0	(empty)
+1333458850.016620	nQcgTWjvg4c	172.24.16.121	61901	94.245.121.251	3544	udp	teredo	-	-	-	S0	-	0	D	1	80	0	0	k6kgXLOoSKl
+1333458850.029781	FrJExwHcSal	172.24.16.67	52298	94.245.121.253	3544	udp	teredo	-	-	-	S0	-	0	D	1	88	0	0	TEfuqmmG4bh
+1333458850.035456	VW0XPVINV8a	190.104.181.210	2152	190.104.181.125	2152	udp	gtpv1	0.000004	194	0	S0	-	0	D	2	250	0	0	(empty)
+1333458850.016620	k6kgXLOoSKl	174.94.190.229	2152	190.104.181.62	2152	udp	gtpv1	0.016267	88	92	SF	-	0	Dd	1	116	1	120	(empty)
+1333458850.029781	TEfuqmmG4bh	190.104.181.254	2152	190.104.181.62	2152	udp	gtpv1	0.000002	192	0	S0	-	0	D	2	248	0	0	(empty)
+1333458850.035460	iE6yhOq3SF	172.27.159.9	63912	94.245.121.254	3544	udp	teredo	-	-	-	S0	-	0	D	1	89	0	0	VW0XPVINV8a
+1333458850.037956	GSxOnSLghOa	190.104.181.57	2152	190.104.181.222	2152	udp	gtpv1	-	-	-	S0	-	0	D	1	120	0	0	(empty)
+1333458850.014199	UWkUyAuUGXf	174.94.190.213	2152	190.104.181.57	2152	udp	gtpv1	-	-	-	S0	-	0	D	1	124	0	0	(empty)
+1333458850.040098	h5DsfNtYzi1	172.24.203.81	54447	65.55.158.118	3544	udp	teredo	0.003698	120	0	S0	-	0	D	2	176	0	0	70MGiRM1Qf4
+1333458850.029783	5OKnoww6xl4	172.24.16.67	52298	65.55.158.118	3544	udp	teredo	-	-	-	S0	-	0	D	1	88	0	0	TEfuqmmG4bh
+1333458850.032887	3PKsZ2Uye21	10.131.42.160	62069	94.245.121.253	3544	udp	teredo	-	-	-	SHR	-	0	d	0	0	1	84	k6kgXLOoSKl
+1333458850.014199	arKYeMETxOg	172.24.204.200	56528	65.55.158.118	3544	udp	teredo	-	-	-	S0	-	0	D	1	88	0	0	UWkUyAuUGXf
+1333458850.035456	fRFu0wcOle6	172.27.159.9	63912	94.245.121.253	3544	udp	teredo	-	-	-	S0	-	0	D	1	89	0	0	VW0XPVINV8a
+1333458850.016620	j4u32Pc5bif	2001:0:5ef5:79fb:38b8:1695:2b37:be8e	128	2002:2571:c817::2571:c817	129	icmp	-	-	-	-	OTH	-	0	-	1	52	0	0	nQcgTWjvg4c
+1333458850.035456	qSsw6ESzHV4	fe80::ffff:ffff:fffe	133	ff02::2	134	icmp	-	0.000004	0	0	OTH	-	0	-	2	96	0	0	fRFu0wcOle6,iE6yhOq3SF
+#close	2012-10-19-17-34-25

testing/btest/Baseline/core.tunnels.gtp.inner_teredo/tunnel.log

@@ -0,0 +1,27 @@
+#separator \x09
+#set_separator	,
+#empty_field	(empty)
+#unset_field	-
+#path	tunnel
+#open	2012-10-19-17-34-25
+#fields	ts	uid	id.orig_h	id.orig_p	id.resp_h	id.resp_p	tunnel_type	action
+#types	time	string	addr	port	addr	port	enum	enum
+1333458850.014199	UWkUyAuUGXf	174.94.190.213	2152	190.104.181.57	2152	Tunnel::GTPv1	Tunnel::DISCOVER
+1333458850.016620	k6kgXLOoSKl	174.94.190.229	2152	190.104.181.62	2152	Tunnel::GTPv1	Tunnel::DISCOVER
+1333458850.016620	nQcgTWjvg4c	172.24.16.121	61901	94.245.121.251	3544	Tunnel::TEREDO	Tunnel::DISCOVER
+1333458850.029781	TEfuqmmG4bh	190.104.181.254	2152	190.104.181.62	2152	Tunnel::GTPv1	Tunnel::DISCOVER
+1333458850.035456	VW0XPVINV8a	190.104.181.210	2152	190.104.181.125	2152	Tunnel::GTPv1	Tunnel::DISCOVER
+1333458850.035456	fRFu0wcOle6	172.27.159.9	63912	94.245.121.253	3544	Tunnel::TEREDO	Tunnel::DISCOVER
+1333458850.035460	iE6yhOq3SF	172.27.159.9	63912	94.245.121.254	3544	Tunnel::TEREDO	Tunnel::DISCOVER
+1333458850.037956	GSxOnSLghOa	190.104.181.57	2152	190.104.181.222	2152	Tunnel::GTPv1	Tunnel::DISCOVER
+1333458850.040098	70MGiRM1Qf4	174.94.190.229	2152	190.104.181.57	2152	Tunnel::GTPv1	Tunnel::DISCOVER
+1333458850.043796	70MGiRM1Qf4	174.94.190.229	2152	190.104.181.57	2152	Tunnel::GTPv1	Tunnel::CLOSE
+1333458850.043796	nQcgTWjvg4c	172.24.16.121	61901	94.245.121.251	3544	Tunnel::TEREDO	Tunnel::CLOSE
+1333458850.043796	VW0XPVINV8a	190.104.181.210	2152	190.104.181.125	2152	Tunnel::GTPv1	Tunnel::CLOSE
+1333458850.043796	k6kgXLOoSKl	174.94.190.229	2152	190.104.181.62	2152	Tunnel::GTPv1	Tunnel::CLOSE
+1333458850.043796	TEfuqmmG4bh	190.104.181.254	2152	190.104.181.62	2152	Tunnel::GTPv1	Tunnel::CLOSE
+1333458850.043796	iE6yhOq3SF	172.27.159.9	63912	94.245.121.254	3544	Tunnel::TEREDO	Tunnel::CLOSE
+1333458850.043796	GSxOnSLghOa	190.104.181.57	2152	190.104.181.222	2152	Tunnel::GTPv1	Tunnel::CLOSE
+1333458850.043796	UWkUyAuUGXf	174.94.190.213	2152	190.104.181.57	2152	Tunnel::GTPv1	Tunnel::CLOSE
+1333458850.043796	fRFu0wcOle6	172.27.159.9	63912	94.245.121.253	3544	Tunnel::TEREDO	Tunnel::CLOSE
+#close	2012-10-19-17-34-25

testing/btest/Baseline/core.tunnels.gtp.non_recursive/out

@@ -0,0 +1 @@
+protocol_violation, [orig_h=74.125.216.149, orig_p=2152/udp, resp_h=10.131.138.69, resp_p=2152/udp], GTP-in-GTP [n\xd9'|\x00\x00\x01\xb6[\xf6\xdc0\xb7d\xe5\xe6\xa76\x91\xfbk\x0e\x02\xc8A\x05\xa8\xe6\xf3Gi\x80(]\xcew\x84\xae}\xd2...]

testing/btest/Baseline/core.tunnels.gtp.not_user_plane_data/conn.log

@@ -0,0 +1,11 @@
+#separator \x09
+#set_separator	,
+#empty_field	(empty)
+#unset_field	-
+#path	conn
+#open	2012-10-19-17-46-48
+#fields	ts	uid	id.orig_h	id.orig_p	id.resp_h	id.resp_p	proto	service	duration	orig_bytes	resp_bytes	conn_state	local_orig	missed_bytes	history	orig_pkts	orig_ip_bytes	resp_pkts	resp_ip_bytes	tunnel_parents
+#types	time	string	addr	port	addr	port	enum	string	interval	count	count	string	bool	count	string	count	count	count	count	table[string]
+1333458850.532814	UWkUyAuUGXf	247.56.43.90	2152	247.56.43.248	2152	udp	-	-	-	-	S0	-	0	D	1	52	0	0	(empty)
+1333458850.867091	arKYeMETxOg	247.56.43.214	2152	237.56.101.238	2152	udp	-	0.028676	12	14	SF	-	0	Dd	1	40	1	42	(empty)
+#close	2012-10-19-17-46-48

testing/btest/Baseline/core.tunnels.gtp.opt_header/conn.log

@@ -0,0 +1,11 @@
+#separator \x09
+#set_separator	,
+#empty_field	(empty)
+#unset_field	-
+#path	conn
+#open	2012-10-19-17-19-16
+#fields	ts	uid	id.orig_h	id.orig_p	id.resp_h	id.resp_p	proto	service	duration	orig_bytes	resp_bytes	conn_state	local_orig	missed_bytes	history	orig_pkts	orig_ip_bytes	resp_pkts	resp_ip_bytes	tunnel_parents
+#types	time	string	addr	port	addr	port	enum	string	interval	count	count	string	bool	count	string	count	count	count	count	table[string]
+1333458852.011535	arKYeMETxOg	10.222.10.10	44960	173.194.69.188	5228	tcp	ssl	0.573499	704	1026	S1	-	0	ShADad	17	1604	14	1762	UWkUyAuUGXf
+1333458852.011535	UWkUyAuUGXf	79.188.154.91	2152	243.149.173.198	2152	udp	gtpv1	0.573499	1740	1930	SF	-	0	Dd	17	2216	14	2322	(empty)
+#close	2012-10-19-17-19-16

testing/btest/Baseline/core.tunnels.gtp.opt_header/out

@@ -0,0 +1,31 @@
+gtpv1_packet, [version=1, pt_flag=T, rsv=F, e_flag=F, s_flag=F, pn_flag=F, msg_type=255, length=60, teid=1980578736, seq=<uninitialized>, n_pdu=<uninitialized>, next_type=<uninitialized>]
+gtpv1_packet, [version=1, pt_flag=T, rsv=F, e_flag=F, s_flag=F, pn_flag=F, msg_type=255, length=60, teid=1980578736, seq=<uninitialized>, n_pdu=<uninitialized>, next_type=<uninitialized>]
+gtpv1_packet, [version=1, pt_flag=T, rsv=F, e_flag=F, s_flag=T, pn_flag=F, msg_type=255, length=64, teid=159098, seq=0, n_pdu=0, next_type=0]
+gtpv1_packet, [version=1, pt_flag=T, rsv=F, e_flag=F, s_flag=F, pn_flag=F, msg_type=255, length=52, teid=1980578736, seq=<uninitialized>, n_pdu=<uninitialized>, next_type=<uninitialized>]
+gtpv1_packet, [version=1, pt_flag=T, rsv=F, e_flag=F, s_flag=F, pn_flag=F, msg_type=255, length=170, teid=1980578736, seq=<uninitialized>, n_pdu=<uninitialized>, next_type=<uninitialized>]
+gtpv1_packet, [version=1, pt_flag=T, rsv=F, e_flag=F, s_flag=T, pn_flag=F, msg_type=255, length=56, teid=159098, seq=1, n_pdu=0, next_type=0]
+gtpv1_packet, [version=1, pt_flag=T, rsv=F, e_flag=F, s_flag=T, pn_flag=F, msg_type=255, length=194, teid=159098, seq=2, n_pdu=0, next_type=0]
+gtpv1_packet, [version=1, pt_flag=T, rsv=F, e_flag=F, s_flag=F, pn_flag=F, msg_type=255, length=52, teid=1980578736, seq=<uninitialized>, n_pdu=<uninitialized>, next_type=<uninitialized>]
+gtpv1_packet, [version=1, pt_flag=T, rsv=F, e_flag=F, s_flag=F, pn_flag=F, msg_type=255, length=111, teid=1980578736, seq=<uninitialized>, n_pdu=<uninitialized>, next_type=<uninitialized>]
+gtpv1_packet, [version=1, pt_flag=T, rsv=F, e_flag=F, s_flag=T, pn_flag=F, msg_type=255, length=56, teid=159098, seq=3, n_pdu=0, next_type=0]
+gtpv1_packet, [version=1, pt_flag=T, rsv=F, e_flag=F, s_flag=F, pn_flag=F, msg_type=255, length=89, teid=1980578736, seq=<uninitialized>, n_pdu=<uninitialized>, next_type=<uninitialized>]
+gtpv1_packet, [version=1, pt_flag=T, rsv=F, e_flag=F, s_flag=T, pn_flag=F, msg_type=255, length=56, teid=159098, seq=4, n_pdu=0, next_type=0]
+gtpv1_packet, [version=1, pt_flag=T, rsv=F, e_flag=F, s_flag=T, pn_flag=F, msg_type=255, length=93, teid=159098, seq=5, n_pdu=0, next_type=0]
+gtpv1_packet, [version=1, pt_flag=T, rsv=F, e_flag=F, s_flag=F, pn_flag=F, msg_type=255, length=52, teid=1980578736, seq=<uninitialized>, n_pdu=<uninitialized>, next_type=<uninitialized>]
+gtpv1_packet, [version=1, pt_flag=T, rsv=F, e_flag=F, s_flag=F, pn_flag=F, msg_type=255, length=457, teid=1980578736, seq=<uninitialized>, n_pdu=<uninitialized>, next_type=<uninitialized>]
+gtpv1_packet, [version=1, pt_flag=T, rsv=F, e_flag=F, s_flag=T, pn_flag=F, msg_type=255, length=157, teid=159098, seq=6, n_pdu=0, next_type=0]
+gtpv1_packet, [version=1, pt_flag=T, rsv=F, e_flag=F, s_flag=T, pn_flag=F, msg_type=255, length=173, teid=159098, seq=7, n_pdu=0, next_type=0]
+gtpv1_packet, [version=1, pt_flag=T, rsv=F, e_flag=F, s_flag=F, pn_flag=F, msg_type=255, length=52, teid=1980578736, seq=<uninitialized>, n_pdu=<uninitialized>, next_type=<uninitialized>]
+gtpv1_packet, [version=1, pt_flag=T, rsv=F, e_flag=F, s_flag=F, pn_flag=F, msg_type=255, length=52, teid=1980578736, seq=<uninitialized>, n_pdu=<uninitialized>, next_type=<uninitialized>]
+gtpv1_packet, [version=1, pt_flag=T, rsv=F, e_flag=F, s_flag=F, pn_flag=F, msg_type=255, length=137, teid=1980578736, seq=<uninitialized>, n_pdu=<uninitialized>, next_type=<uninitialized>]
+gtpv1_packet, [version=1, pt_flag=T, rsv=F, e_flag=F, s_flag=T, pn_flag=F, msg_type=255, length=56, teid=159098, seq=8, n_pdu=0, next_type=0]
+gtpv1_packet, [version=1, pt_flag=T, rsv=F, e_flag=F, s_flag=T, pn_flag=F, msg_type=255, length=157, teid=159098, seq=9, n_pdu=0, next_type=0]
+gtpv1_packet, [version=1, pt_flag=T, rsv=F, e_flag=F, s_flag=T, pn_flag=F, msg_type=255, length=189, teid=159098, seq=10, n_pdu=0, next_type=0]
+gtpv1_packet, [version=1, pt_flag=T, rsv=F, e_flag=F, s_flag=T, pn_flag=F, msg_type=255, length=173, teid=159098, seq=11, n_pdu=0, next_type=0]
+gtpv1_packet, [version=1, pt_flag=T, rsv=F, e_flag=F, s_flag=T, pn_flag=F, msg_type=255, length=205, teid=159098, seq=12, n_pdu=0, next_type=0]
+gtpv1_packet, [version=1, pt_flag=T, rsv=F, e_flag=F, s_flag=T, pn_flag=F, msg_type=255, length=189, teid=159098, seq=13, n_pdu=0, next_type=0]
+gtpv1_packet, [version=1, pt_flag=T, rsv=F, e_flag=F, s_flag=F, pn_flag=F, msg_type=255, length=52, teid=1980578736, seq=<uninitialized>, n_pdu=<uninitialized>, next_type=<uninitialized>]
+gtpv1_packet, [version=1, pt_flag=T, rsv=F, e_flag=F, s_flag=F, pn_flag=F, msg_type=255, length=52, teid=1980578736, seq=<uninitialized>, n_pdu=<uninitialized>, next_type=<uninitialized>]
+gtpv1_packet, [version=1, pt_flag=T, rsv=F, e_flag=F, s_flag=F, pn_flag=F, msg_type=255, length=52, teid=1980578736, seq=<uninitialized>, n_pdu=<uninitialized>, next_type=<uninitialized>]
+gtpv1_packet, [version=1, pt_flag=T, rsv=F, e_flag=F, s_flag=F, pn_flag=F, msg_type=255, length=52, teid=1980578736, seq=<uninitialized>, n_pdu=<uninitialized>, next_type=<uninitialized>]
+gtpv1_packet, [version=1, pt_flag=T, rsv=F, e_flag=F, s_flag=F, pn_flag=F, msg_type=255, length=52, teid=1980578736, seq=<uninitialized>, n_pdu=<uninitialized>, next_type=<uninitialized>]

testing/btest/Baseline/core.tunnels.gtp.opt_header/tunnel.log

@@ -0,0 +1,11 @@
+#separator \x09
+#set_separator	,
+#empty_field	(empty)
+#unset_field	-
+#path	tunnel
+#open	2012-10-19-17-19-16
+#fields	ts	uid	id.orig_h	id.orig_p	id.resp_h	id.resp_p	tunnel_type	action
+#types	time	string	addr	port	addr	port	enum	enum
+1333458852.011535	UWkUyAuUGXf	79.188.154.91	2152	243.149.173.198	2152	Tunnel::GTPv1	Tunnel::DISCOVER
+1333458852.585034	UWkUyAuUGXf	79.188.154.91	2152	243.149.173.198	2152	Tunnel::GTPv1	Tunnel::CLOSE
+#close	2012-10-19-17-19-16

testing/btest/Baseline/core.tunnels.gtp.outer_ip_frag/conn.log

@@ -0,0 +1,11 @@
+#separator \x09
+#set_separator	,
+#empty_field	(empty)
+#unset_field	-
+#path	conn
+#open	2012-10-19-16-44-02
+#fields	ts	uid	id.orig_h	id.orig_p	id.resp_h	id.resp_p	proto	service	duration	orig_bytes	resp_bytes	conn_state	local_orig	missed_bytes	history	orig_pkts	orig_ip_bytes	resp_pkts	resp_ip_bytes	tunnel_parents
+#types	time	string	addr	port	addr	port	enum	string	interval	count	count	string	bool	count	string	count	count	count	count	table[string]
+1333458850.364667	arKYeMETxOg	10.131.47.185	1923	79.101.110.141	80	tcp	http	0.069783	2100	56702	SF	-	0	ShADadfF	27	3204	41	52594	UWkUyAuUGXf
+1333458850.364667	UWkUyAuUGXf	239.114.155.111	2152	63.94.149.181	2152	udp	gtpv1	0.069813	3420	52922	SF	-	0	Dd	27	4176	41	54070	(empty)
+#close	2012-10-19-16-44-02

testing/btest/Baseline/core.tunnels.gtp.outer_ip_frag/http.log

@@ -0,0 +1,10 @@
+#separator \x09
+#set_separator	,
+#empty_field	(empty)
+#unset_field	-
+#path	http
+#open	2012-10-19-16-44-02
+#fields	ts	uid	id.orig_h	id.orig_p	id.resp_h	id.resp_p	trans_depth	method	host	uri	referrer	user_agent	request_body_len	response_body_len	status_code	status_msg	info_code	info_msg	filename	tags	username	password	proxied	mime_type	md5	extraction_file
+#types	time	string	addr	port	addr	port	count	string	string	string	string	string	count	count	count	string	count	string	string	table[enum]	string	string	table[string]	string	string	file
+1333458850.375568	arKYeMETxOg	10.131.47.185	1923	79.101.110.141	80	1	GET	o-o.preferred.telekomrs-beg1.v2.lscache8.c.youtube.com	/videoplayback?upn=MTU2MDY5NzQ5OTM0NTI3NDY4NDc&sparams=algorithm,burst,cp,factor,id,ip,ipbits,itag,source,upn,expire&fexp=912300,907210&algorithm=throttle-factor&itag=34&ip=212.0.0.0&burst=40&sver=3&signature=832FB1042E20780CFCA77A4DB5EA64AC593E8627.D1166C7E8365732E52DAFD68076DAE0146E0AE01&source=youtube&expire=1333484980&key=yt1&ipbits=8&factor=1.25&cp=U0hSSFRTUl9NSkNOMl9MTVZKOjh5eEN2SG8tZF84&id=ebf1e932d4bd1286&cm2=1	http://s.ytimg.com/yt/swfbin/watch_as3-vflqrJwOA.swf	Mozilla/5.0 (Windows NT 5.1) AppleWebKit/535.11 (KHTML, like Gecko; X-SBLSP) Chrome/17.0.963.83 Safari/535.11	0	56320	206	Partial Content	-	-	-	(empty)	-	-	-	application/octet-stream	-	-
+#close	2012-10-19-16-44-02

testing/btest/Baseline/core.tunnels.gtp.outer_ip_frag/tunnel.log

@@ -0,0 +1,11 @@
+#separator \x09
+#set_separator	,
+#empty_field	(empty)
+#unset_field	-
+#path	tunnel
+#open	2012-10-19-16-44-02
+#fields	ts	uid	id.orig_h	id.orig_p	id.resp_h	id.resp_p	tunnel_type	action
+#types	time	string	addr	port	addr	port	enum	enum
+1333458850.364667	UWkUyAuUGXf	239.114.155.111	2152	63.94.149.181	2152	Tunnel::GTPv1	Tunnel::DISCOVER
+1333458850.434480	UWkUyAuUGXf	239.114.155.111	2152	63.94.149.181	2152	Tunnel::GTPv1	Tunnel::CLOSE
+#close	2012-10-19-16-44-02

testing/btest/Baseline/core.tunnels.gtp.unknown_or_too_short/dpd.log

@@ -0,0 +1,10 @@
+#separator \x09
+#set_separator	,
+#empty_field	(empty)
+#unset_field	-
+#path	dpd
+#open	2012-10-19-17-38-54
+#fields	ts	uid	id.orig_h	id.orig_p	id.resp_h	id.resp_p	proto	analyzer	failure_reason
+#types	time	string	addr	port	addr	port	enum	string	string
+1333458853.075889	UWkUyAuUGXf	173.86.159.28	2152	213.72.147.186	2152	udp	GTPV1	Truncated GTPv1 [E\x00\x05\xc8G\xea@\x00\x80\x06\xb6\x83\x0a\x83w&\xd9\x14\x9c\x04\xd9\xc2\x00P\xddh\xb4\x8f41eVP\x10\x10\xe0u\xcf\x00\x00...]
+#close	2012-10-19-17-38-54

testing/btest/Baseline/core.tunnels.gtp.unknown_or_too_short/tunnel.log

@@ -0,0 +1,11 @@
+#separator \x09
+#set_separator	,
+#empty_field	(empty)
+#unset_field	-
+#path	tunnel
+#open	2012-10-19-17-38-54
+#fields	ts	uid	id.orig_h	id.orig_p	id.resp_h	id.resp_p	tunnel_type	action
+#types	time	string	addr	port	addr	port	enum	enum
+1333458853.034734	UWkUyAuUGXf	173.86.159.28	2152	213.72.147.186	2152	Tunnel::GTPv1	Tunnel::DISCOVER
+1333458853.108391	UWkUyAuUGXf	173.86.159.28	2152	213.72.147.186	2152	Tunnel::GTPv1	Tunnel::CLOSE
+#close	2012-10-19-17-38-54

testing/btest/Baseline/core.when-interpreter-exceptions/bro..stderr

@@ -0,0 +1,5 @@
+1355264617.053514 expression error in /home/jsiwek/bro/testing/btest/.tmp/core.when-interpreter-exceptions/when-interpreter-exceptions.bro, line 32: field value missing [p$ip]
+1355264617.053514 expression error in /home/jsiwek/bro/testing/btest/.tmp/core.when-interpreter-exceptions/when-interpreter-exceptions.bro, line 53: field value missing [p$ip]
+1355264617.053514 expression error in /home/jsiwek/bro/testing/btest/.tmp/core.when-interpreter-exceptions/when-interpreter-exceptions.bro, line 43: field value missing [p$ip]
+1355264617.053514 expression error in /home/jsiwek/bro/testing/btest/.tmp/core.when-interpreter-exceptions/when-interpreter-exceptions.bro, line 16: field value missing [p$ip]
+1355264617.053514 received termination signal

testing/btest/Baseline/core.when-interpreter-exceptions/bro..stdout

@@ -0,0 +1,8 @@
+timeout g(), F
+timeout g(), T
+timeout
+g() done, no exception, T
+localhost resolved
+localhost resolved from f(), T
+localhost resolved from f(), F
+f() done, no exception, T