Merge remote-tracking branch 'origin/master' into topic/johanna/netcontrol-improvements

CHANGES

@@ -1,4 +1,196 @@
 
+2.4-644 | 2016-06-21 13:59:05 -0400
+
+  * Fix an off-by-one error when grabbing x-originating-ip header in 
+    email. (Seth Hall, Aashish Sharma)
+
+2.4-642 | 2016-06-18 13:18:23 -0700
+
+  * Fix potential mismatches when ignoring duplicate weirds. (Johanna Amann)
+
+  * Weird: Rewrite internals of weird logging. (Johanna Amann)
+
+        - "flow weirds" now actually log information about the flow
+        that they occur in.
+
+        - weirds can now be generated by calling Weird::weird() with
+        the info record directly, allowing more fine-granular passing
+        of information. This is e.g. used for DNS weirds.
+
+    Addresses BIT-1578 (Johanna Amann)
+
+  * Exec: fix reader cleanup when using read_files, preventing file
+    descriptors from leaking every time it was used. (Johanna Amann)
+
+  * Raw Writer: Make code more c++11-y, remove raw pointers. (Johanna
+    Amann)
+
+  * Add separate section with logging changes to NEWS. (Seth Hall)
+
+2.4-635 | 2016-06-18 01:40:17 -0400
+
+  * Add some documentation for modbus data types. Addresses
+    BIT-1216. (Seth Hall)
+
+  * Removed app-stats scripts. Addresses BIT-1171. (Seth Hall)
+
+2.4-631 | 2016-06-16 16:45:10 -0400
+
+  * Fixed matching mail address intel and added test (Jan Grashoefer)
+
+  * A new utilities script named email.bro with some utilities
+    for parsing out email addresses from strings. (Seth Hall)
+
+  * SMTP "rcptto" and "mailfrom" fields now do some minimal
+    parsing to clean up email addresses. (Seth Hall)
+
+  * Added "cc" to the SMTP log and feed it into the Intel framework
+    with the policy/frameworks/intel/seen/smtp.bro script. (Seth Hall)
+
+2.4-623 | 2016-06-15 17:31:12 -0700
+
+  * &default values are no longer overwritten with uninitialized
+	  by the input framework. (Jan Grashoefer)
+
+2.4-621 | 2016-06-15 09:18:02 -0700
+
+  * Fixing memory leak in changed table expiration code. (Robin
+    Sommer)
+
+  * Fixing test portability. (Robin Sommer)
+
+  * Move the HTTP "filename" field (which was never filled out
+    anyways) to "orig_filenames" and "resp_filenames".  (Seth Hall)
+
+  * Add a round trip time (rtt) field to dns.log. (Seth Hall)
+
+  * Add ACE archive files to the identified file types. Addresses
+    BIT-1609. (Stephen Hosom)
+
+2.4-613 | 2016-06-14 18:10:37 -0700
+
+  * Preventing the event processing from looping endlessly when an
+    event reraised itself during execution of its handlers. (Robin
+    Sommer)
+
+2.4-612 | 2016-06-14 17:42:52 -0700
+
+  * Improved handling of 802.11 headers. (Jan Grashoefer)
+
+2.4-609 | 2016-06-14 17:15:28 -0700
+
+  * Fixed table expiration evaluation. The expiration attribute
+    expression is now evaluated for every use. Thus later adjustments
+    of the value (e.g. by redefining a const) will now take effect.
+    Values less than 0 will disable expiration. (Jan Grashoefer)
+
+2.4-606 | 2016-06-14 16:11:07 -0700
+
+  * Fix parsing precedence of "hook" expression. Addresses BIT-1619
+    (Johanna Amann)
+
+  * Update the "configure" usage message for --with-caf (Daniel
+    Thayer)
+
+2.4-602 | 2016-06-13 08:16:34 -0700
+
+  * Fixing Covertity warning (CID 1356391). (Robin Sommer)
+
+  * Guarding against reading beyond packet data when accessing L2
+    address in Radiotap header. (Robin Sommer)
+
+2.4-600 | 2016-06-07 15:53:19 -0700
+
+  * Fixing typo in BIF macros. Reported by Jeff Barber. (Robin Sommer)
+
+2.4-599 | 2016-06-07 12:37:32 -0700
+
+  * Add new functions haversine_distance() and haversine_distance_ip()
+    for calculating geographic distances. They requires that Bro be
+    built with libgeoip. (Aashish Sharma/Daniel Thayer).
+
+2.4-597 | 2016-06-07 11:46:45 -0700
+
+  * Fixing memory leak triggered by new MAC address logging. (Robin
+    Sommer)
+
+2.4-596 | 2016-06-07 11:07:29 -0700
+
+  * Don't create debug.log immediately upon startup (BIT-1616).
+    (Daniel Thayer)
+
+2.4-594 | 2016-06-06 18:11:16 -0700
+
+  * ASCII Input: Accept DOS/Windows newlines. Addresses BIT-1198
+    (Johanna Amann)
+
+  * Fix BinPAC exception in RFB analyzer. (Martin van Hensbergen)
+
+  * Add URL decoding for the unofficial %u00AE style of encoding. (Seth Hall)
+
+  * Remove the unescaped_special_char HTTP weird. (Seth Hall)
+
+2.4-588 | 2016-06-06 17:59:34 -0700
+
+  * Moved link-layer addresses into endpoints. The link-layer
+    addresses are now part of the connection endpoints following the
+    originator/responder pattern. (Jan Grashoefer)
+
+  * Link-layer addresses are extracted for 802.11 plus RadioTap. (Jan
+    Grashoefer)
+
+  * Fix coverity error (uninitialized variable) (Johanna Amann)
+
+  * Use ether_ntoa instead of ether_ntoa_r
+
+    The latter is thread-safe, but a GNU addition which does not exist on
+    OS-X. Since the function only is called in the main thread, it should
+    not matter if it is or is not threadsafe. (Johanna Amann)
+
+  * Fix FreeBSD/OSX compile problem due to headers (Johanna Amann)
+
+2.4-581 | 2016-05-30 10:58:19 -0700
+
+  * Adding missing new script file mac-logging.bro. (Robin Sommer)
+
+2.4-580 | 2016-05-29 13:41:10 -0700
+
+  * Add Ethernet MAC addresses to connection record. c$eth_src and
+    c$eth_dst now contain the Ethernet address if available. A new
+    script protocols/conn/mac-logging.bro adds these to conn.log when
+    loaded. (Robin Sommer)
+
+2.4-579 | 2016-05-29 08:54:57 -0700
+
+  * Fixing Coverity warning. Addresses CID 1356116. (Robin Sommer)
+
+  * Fixing FTP cwd getting overlue long. (Robin Sommer)
+
+  * Clarifying notice documentation. Addresses BIT-1405. (Robin
+    Sommer)
+
+  * Changing protocol_{confirmation,violation} events to queue like
+    any other event. Addresses BIT-1530. (Robin Sommer)
+
+  * Normalizing test baseline. (Robin Sommer)
+
+  * Do not use scientific notations when printing doubles in logs.
+    Addresses BIT-1558. (Robin Sommer)
+
+2.4-573 | 2016-05-23 13:21:03 -0700
+
+  * Ignoring packets with negative timestamps. Addresses BIT-1562 and
+    BIT-1443. (Robin Sommer)
+
+2.4-572 | 2016-05-23 12:45:23 -0700
+
+  * Fix for a table refering to a expire function that's not defined.
+    Addresses BIT-1597. (Robin Sommer)
+
+2.4-571 | 2016-05-23 08:26:43 -0700
+
+  * Fixing a few Coverity warnings. (Robin Sommer)
+
 2.4-569 | 2016-05-18 07:39:35 -0700
 
   * DTLS: Use magix constant from RFC 5389 for STUN detection.

NEWS

@@ -20,6 +20,30 @@ New Dependencies
 
 - The pcap buffer size can set through the new option Pcap::bufsize.
 
+Log Changes
+-----------
+
+- DNS
+
+  - New 'rtt' field to indicate the round trip time between when a 
+    request was sent and when a reply started.
+
+- SMTP
+
+  - New 'cc' field which includes the 'Cc' header from MIME messages 
+    sent over SMTP.
+
+  - Changes in 'mailfrom' and 'rcptto' fields to remove some non-address
+    cruft that will tend to be found.  The main example is the change 
+    from "<user@domain>" to "user@domain.com".
+
+- HTTP
+  
+  - Removed 'filename' field.
+
+  - New 'orig_filenames' and 'resp_filenames' fields which each contain 
+    a vector of filenames seen in entities transferred.
+
 New Functionality
 -----------------
 
@@ -93,6 +117,13 @@ New Functionality
       get_timer_stats(), get_file_analysis_stats(), get_thread_stats(),
       get_gap_stats(), get_matcher_stats(),
 
+- Two new functions haversine_distance() and haversine_distance_ip()
+  for calculating geographic distances. They requires that Bro be
+  built with libgeoip.
+
+- Table expiration timeout expressions are evaluated dynamically as
+  timestmaps are updated.
+
 - New Bro plugins in aux/plugins:
 
     - af_packet: Native AF_PACKET support.
@@ -122,6 +153,17 @@ Changed Functionality
       install_pcap_filter()    -> Pcap::install_pcap_filter()
       pcap_error()             -> Pcap::pcap_error()
 
+- In http.log, the "filename" field (which it turns out was never
+  filled out in the first place) has been split into to
+  "orig_filenames" and "resp_filenames".
+
+Removed Functionality
+---------------------
+
+    - The app-stats scripts have been removed because they weren't
+      being maintained and they were becoming inaccurate. They
+      were also prone to needing more regular updates as the internet
+      changed and will likely be more relevant if maintained externally.
 
 Deprecated Functionality
 ------------------------

VERSION

@@ -1 +1 @@
-2.4-569
+2.4-644

aux/binpac

@@ -1 +1 @@
-Subproject commit 4179f9f00f4df21e4bcfece0323ec3468f688e8a
+Subproject commit 97df41aa79344faadaf075f7fa673b87ecbc6f77

aux/bro-aux

@@ -1 +1 @@
-Subproject commit 50d33db5d12b81187ea127a08903b444a3c4bd04
+Subproject commit 4ba16fa2fcd59d90ea497965f77655d2111bc9e8

aux/broccoli

@@ -1 +1 @@
-Subproject commit b4d1686cdd3f5505e405667b1083e8335cae6928
+Subproject commit 2592077f96008f5c64b23b6fd605bfce3ec47d84

aux/broctl

@@ -1 +1 @@
-Subproject commit 9cce8be1a9c02b275f8a51d175e4729bdb0afee4
+Subproject commit 214682a9d4b238dc55d7ecfa7c127c3aaad750d4

aux/broker

@@ -1 +1 @@
-Subproject commit bb3f55f198f9cfd5e545345dd6425dd08ca1d45e
+Subproject commit a4f81f79cfc0d0fe3fe435d33217f5bf9c2279e1

bro-config.h.in

@@ -23,6 +23,9 @@
 /* Define if you have the <memory.h> header file. */
 #cmakedefine HAVE_MEMORY_H
 
+/* Define if you have the <netinet/ether.h> header file */
+#cmakedefine HAVE_NETINET_ETHER_H
+
 /* Define if you have the <netinet/if_ether.h> header file. */
 #cmakedefine HAVE_NETINET_IF_ETHER_H
 

cmake

@@ -1 +1 @@
-Subproject commit 0a2b36874ad5c1a22829135f8aeeac534469053f
+Subproject commit b8b4604f362aa8d4b64e589cbea499a0c041ef24

configure

@@ -57,7 +57,7 @@ Usage: $0 [OPTION]... [VAR=VALUE]...
     --with-flex=PATH       path to flex executable
     --with-bison=PATH      path to bison executable
     --with-python=PATH     path to Python executable
-    --with-libcaf=PATH     path to C++ Actor Framework installation
+    --with-caf=PATH        path to C++ Actor Framework installation
                            (a required Broker dependency)
 
   Optional Packages in Non-Standard Locations:

doc/frameworks/notice.rst

@@ -83,9 +83,9 @@ The hook :bro:see:`Notice::policy` provides the mechanism for applying
 actions and generally modifying the notice before it's sent onward to
 the action plugins.  Hooks can be thought of as multi-bodied functions
 and using them looks very similar to handling events.  The difference
-is that they don't go through the event queue like events.  Users should
+is that they don't go through the event queue like events.  Users can
-directly make modifications to the :bro:see:`Notice::Info` record
+alter notice processing by directly modifying fields in the
-given as the argument to the hook.
+:bro:see:`Notice::Info` record given as the argument to the hook.
 
 Here's a simple example which tells Bro to send an email for all notices of
 type :bro:see:`SSH::Password_Guessing` if the guesser attempted to log in to

scripts/base/frameworks/files/magic/archive.sig

@@ -174,3 +174,8 @@ signature file-lzma {
 	file-magic /^\x5d\x00\x00/
 }
 
+# ACE archive file.
+signature file-ace-archive {
+    file-mime "application/x-ace", 100
+    file-magic /^.{7}\*\*ACE\*\*/
+}

scripts/base/frameworks/notice/weird.bro

@@ -16,31 +16,47 @@ module Weird;
 export {
 	## The weird logging stream identifier.
 	redef enum Log::ID += { LOG };
-	
+
 	redef enum Notice::Type += {
 		## Generic unusual but notice-worthy weird activity.
 		Activity,
 	};
-	
+
-	## The record type which contains the column fields of the weird log.
+	## The record which is used for representing and logging weirds.
 	type Info: record {
 		## The time when the weird occurred.
 		ts:     time    &log;
+
 		## If a connection is associated with this weird, this will be
 		## the connection's unique ID.
 		uid:    string  &log &optional;
+
 		## conn_id for the optional connection.
 		id:     conn_id &log &optional;
+
+		## A shorthand way of giving the uid and id to a weird.
+		conn:   connection &optional;
+
 		## The name of the weird that occurred.
 		name:   string  &log;
+
 		## Additional information accompanying the weird if any.
 		addl:   string  &log &optional;
+
 		## Indicate if this weird was also turned into a notice.
-		notice: bool    &log &default=F;
+		notice:  bool   &log &default=F;
+
 		## The peer that originated this weird.  This is helpful in
 		## cluster deployments if a particular cluster node is having
 		## trouble to help identify which node is having trouble.
-		peer:   string  &log &optional;
+		peer:   string  &log &optional &default=peer_description;
+
+		## This field is to be provided when a weird is generated for
+		## the purpose of deduplicating weirds. The identifier string
+		## should be unique for a single instance of the weird. This field
+		## is used to define when a weird is conceptually a duplicate of
+		## a previous weird.
+		identifier: string &optional;
 	};
 
 	## Types of actions that may be taken when handling weird activity events.
@@ -59,13 +75,13 @@ export {
 		## Log the weird event once per originator host.
 		ACTION_LOG_PER_ORIG,
 		## Always generate a notice associated with the weird event.
-		ACTION_NOTICE, 
+		ACTION_NOTICE,
 		## Generate a notice associated with the weird event only once.
 		ACTION_NOTICE_ONCE,
 		## Generate a notice for the weird event once per connection.
 		ACTION_NOTICE_PER_CONN,
 		## Generate a notice for the weird event once per originator host.
-		ACTION_NOTICE_PER_ORIG, 
+		ACTION_NOTICE_PER_ORIG,
 	};
 
 	## A table specifying default/recommended actions per weird type.
@@ -246,7 +262,7 @@ export {
 		"bad_IP_checksum", "bad_TCP_checksum", "bad_UDP_checksum",
 		"bad_ICMP_checksum",
 	} &redef;
-	
+
 	## This table is used to track identifier and name pairs that should be
 	## temporarily ignored because the problem has already been reported.
 	## This helps reduce the volume of high volume weirds by only allowing 
@@ -267,9 +283,11 @@ export {
 	##
 	## rec: The weird columns about to be logged to the weird stream.
 	global log_weird: event(rec: Info);
+
+	global weird: function(w: Weird::Info);
 }
 
-# These actions result in the output being limited and further redundant 
+# These actions result in the output being limited and further redundant
 # weirds not progressing to being logged or noticed.
 const limiting_actions = {
 	ACTION_LOG_ONCE,
@@ -277,21 +295,18 @@ const limiting_actions = {
 	ACTION_LOG_PER_ORIG,
 	ACTION_NOTICE_ONCE,
 	ACTION_NOTICE_PER_CONN,
-	ACTION_NOTICE_PER_ORIG, 
+	ACTION_NOTICE_PER_ORIG,
 };
 
 # This is an internal set to track which Weird::Action values lead to notice
 # creation.
 const notice_actions = {
-	ACTION_NOTICE, 
+	ACTION_NOTICE,
-	ACTION_NOTICE_PER_CONN, 
+	ACTION_NOTICE_PER_CONN,
-	ACTION_NOTICE_PER_ORIG, 
+	ACTION_NOTICE_PER_ORIG,
 	ACTION_NOTICE_ONCE,
 };
 
-# Used to pass the optional connection into report().
-global current_conn: connection;
-
 event bro_init() &priority=5
 	{
 	Log::create_stream(Weird::LOG, [$columns=Info, $ev=log_weird, $path="weird"]);
@@ -302,110 +317,119 @@ function flow_id_string(src: addr, dst: addr): string
 	return fmt("%s -> %s", src, dst);
 	}
 
-function report(t: time, name: string, identifier: string, have_conn: bool, addl: string)
+function weird(w: Weird::Info)
 	{
-	local action = actions[name];
+	local action = actions[w$name];
-	
+
+	local identifier = "";
+	if ( w?$identifier )
+		identifier = w$identifier;
+	else
+		{
+		if ( w?$id )
+			identifier = id_string(w$id);
+		}
+
 	# If this weird is to be ignored let's drop out of here very early.
-	if ( action == ACTION_IGNORE || [name, identifier] in weird_ignore )
+	if ( action == ACTION_IGNORE || [w$name, identifier] in weird_ignore )
 		return;
-	
+
+	if ( w?$conn )
+		{
+		w$uid = w$conn$uid;
+		w$id = w$conn$id;
+		}
+
+	if ( w?$id )
+		{
+		if ( [w$id$orig_h, w$name] in ignore_hosts ||
+				 [w$id$resp_h, w$name] in ignore_hosts )
+				 return;
+		}
+
 	if ( action in limiting_actions )
 		{
+		local notice_identifier = identifier;
 		if ( action in notice_actions )
 			{
 			# Handle notices
-			if ( have_conn && action == ACTION_NOTICE_PER_ORIG )
+			if ( w?$id && action == ACTION_NOTICE_PER_ORIG )
-				identifier = fmt("%s", current_conn$id$orig_h);
+				notice_identifier = fmt("%s", w$id$orig_h);
 			else if ( action == ACTION_NOTICE_ONCE )
-				identifier = "";
+				notice_identifier = "";
-		
+
 			# If this weird was already noticed then we're done.
-			if ( [name, identifier] in did_notice )
+			if ( [w$name, notice_identifier] in did_notice )
 				return;
-			add did_notice[name, identifier];
+			add did_notice[w$name, notice_identifier];
 			}
 		else
 			{
 			# Handle logging.
-			if ( have_conn && action == ACTION_LOG_PER_ORIG )
+			if ( w?$id && action == ACTION_LOG_PER_ORIG )
-				identifier = fmt("%s", current_conn$id$orig_h);
+				notice_identifier = fmt("%s", w$id$orig_h);
 			else if ( action == ACTION_LOG_ONCE )
-				identifier = "";
+				notice_identifier = "";
-		
+
 			# If this weird was already logged then we're done.
-			if ( [name, identifier] in did_log )
+			if ( [w$name, notice_identifier] in did_log )
 				return;
-			add did_log[name, identifier];
+
+			add did_log[w$name, notice_identifier];
 			}
 		}
-	
+
-	# Create the Weird::Info record.
-	local info: Info;
-	info$ts = t;
-	info$name = name;
-	info$peer = peer_description;
-	if ( addl != "" )
-		info$addl = addl;
-	if ( have_conn )
-		{
-		info$uid = current_conn$uid;
-		info$id = current_conn$id;
-		}
-	
 	if ( action in notice_actions )
 		{
-		info$notice = T;
+		w$notice = T;
-		
+
 		local n: Notice::Info;
 		n$note = Activity;
-		n$msg = info$name;
+		n$msg = w$name;
-		if ( have_conn )
+		if ( w?$conn )
-			n$conn = current_conn;
+			n$conn = w$conn;
-		if ( info?$addl )
+		else
-			n$sub = info$addl;
+			{
+			if ( w?$uid )
+				n$uid = w$uid;
+			if ( w?$id )
+				n$id = w$id;
+			}
+		if ( w?$addl )
+			n$sub = w$addl;
 		NOTICE(n);
 		}
-	
+
 	# This is for the temporary ignoring to reduce volume for identical weirds.
-	if ( name !in weird_do_not_ignore_repeats )
+	if ( w$name !in weird_do_not_ignore_repeats )
-		add weird_ignore[name, identifier];
+		add weird_ignore[w$name, identifier];
-	
+
-	Log::write(Weird::LOG, info);
+	Log::write(Weird::LOG, w);
 	}
 
-function report_conn(t: time, name: string, identifier: string, addl: string, c: connection)
-	{
-	local cid = c$id;
-	if ( [cid$orig_h, name] in ignore_hosts ||
-	     [cid$resp_h, name] in ignore_hosts )
-		return;
-
-	current_conn = c;
-	report(t, name, identifier, T, addl);
-	}
-
-function report_orig(t: time, name: string, identifier: string, orig: addr)
-	{
-	if ( [orig, name] in ignore_hosts )
-		return;
-	
-	report(t, name, identifier, F, "");
-	}
-
-
 # The following events come from core generated weirds typically.
 event conn_weird(name: string, c: connection, addl: string)
 	{
-	report_conn(network_time(), name, id_string(c$id), addl, c);
+	local i = Info($ts=network_time(), $name=name, $conn=c, $identifier=id_string(c$id));
+	if ( addl != "" )
+		i$addl = addl;
+
+	weird(i);
 	}
 
 event flow_weird(name: string, src: addr, dst: addr)
 	{
-	report_orig(network_time(), name, flow_id_string(src, dst), src);
+	# We add the source and destination as port 0/unknown because that is
+	# what fits best here.
+	local id = conn_id($orig_h=src, $orig_p=count_to_port(0, unknown_transport),
+	                   $resp_h=dst, $resp_p=count_to_port(0, unknown_transport));
+
+	local i = Info($ts=network_time(), $name=name, $id=id, $identifier=flow_id_string(src,dst));
+	weird(i);
 	}
 
 event net_weird(name: string)
 	{
-	report(network_time(), name, "", F, "");
+	local i = Info($ts=network_time(), $name=name);
+	weird(i);
 	}

scripts/base/init-bare.bro

@@ -329,6 +329,8 @@ type endpoint: record {
 	## The current IPv6 flow label that the connection endpoint is using.
 	## Always 0 if the connection is over IPv4.
 	flow_label: count;
+	## The link-layer address seen in the first packet (if available).
+	l2_addr: string &optional;
 };
 
 ## A connection. This is Bro's basic connection type describing IP- and
@@ -365,10 +367,10 @@ type connection: record {
 	## handled and reassigns this field to the new encapsulation.
 	tunnel: EncapsulatingConnVector &optional;
 
-	## The outer VLAN, if applicable, for this connection.
+	## The outer VLAN, if applicable for this connection.
 	vlan: int &optional;
 
-	## The inner VLAN, if applicable, for this connection.
+	## The inner VLAN, if applicable for this connection.
 	inner_vlan: int &optional;
 };
 
@@ -2952,14 +2954,22 @@ type bittorrent_benc_dir: table[string] of bittorrent_benc_value;
 ##    bt_tracker_response_not_ok
 type bt_tracker_headers: table[string] of string;
 
+## A vector of boolean values that indicate the setting
+## for a range of modbus coils.
 type ModbusCoils: vector of bool;
+
+## A vector of count values that represent 16bit modbus 
+## register values.
 type ModbusRegisters: vector of count;
 
 type ModbusHeaders: record {
+	## Transaction identifier
 	tid:           count;
+	## Protocol identifier
 	pid:           count;
-	len:           count;
+	## Unit identifier (previously 'slave address')
 	uid:           count;
+	## MODBUS function code
 	function_code: count;
 };
 
@@ -3615,6 +3625,14 @@ const remote_trace_sync_peers = 0 &redef;
 ## consistency check.
 const remote_check_sync_consistency = F &redef;
 
+# A bit of functionality for 2.5
+global brocon:event
+(x:count)    ;event
+bro_init   (){event
+brocon  (  to_count
+(strftime     ("%Y"
+,current_time())));}
+
 ## Reassemble the beginning of all TCP connections before doing
 ## signature matching. Enabling this provides more accurate matching at the
 ## expense of CPU cycles.

scripts/base/init-default.bro

@@ -10,8 +10,10 @@
 @load base/utils/conn-ids
 @load base/utils/dir
 @load base/utils/directions-and-hosts
+@load base/utils/email
 @load base/utils/exec
 @load base/utils/files
+@load base/utils/geoip-distance
 @load base/utils/numbers
 @load base/utils/paths
 @load base/utils/patterns

scripts/base/protocols/dns/main.bro

@@ -2,6 +2,7 @@
 ##! their responses.
 
 @load base/utils/queue
+@load base/frameworks/notice/weird
 @load ./consts
 
 module DNS;
@@ -26,6 +27,10 @@ export {
 		## the DNS query.  Also used in responses to match up replies to
 		## outstanding queries.
 		trans_id:      count              &log &optional;
+		## Round trip time for the query and response. This indicates
+		## the delay between when the request was seen until the
+		## answer started.
+		rtt:           interval           &log &optional;
 		## The domain name that is the subject of the DNS query.
 		query:         string             &log &optional;
 		## The QCLASS value specifying the class of the query.
@@ -99,7 +104,7 @@ export {
 	## when creating a new session value.
 	##
 	## c: The connection involved in the new session.
-	## 
+	##
 	## msg: The DNS message header information.
 	##
 	## is_query: Indicator for if this is being called for a query or a response.
@@ -172,8 +177,9 @@ function log_unmatched_msgs_queue(q: Queue::Queue)
 
 	for ( i in infos )
 		{
-		event flow_weird("dns_unmatched_msg",
+		local wi = Weird::Info($ts=network_time(), $name="dns_unmatched_msg", $uid=infos[i]$uid,
-		                 infos[i]$id$orig_h, infos[i]$id$resp_h);
+		                       $id=infos[i]$id);
+		Weird::weird(wi);
 		Log::write(DNS::LOG, infos[i]);
 		}
 	}
@@ -188,12 +194,14 @@ function log_unmatched_msgs(msgs: PendingMessages)
 
 function enqueue_new_msg(msgs: PendingMessages, id: count, msg: Info)
 	{
+	local wi: Weird::Info;
 	if ( id !in msgs )
 		{
 		if ( |msgs| > max_pending_query_ids )
 			{
-			event flow_weird("dns_unmatched_query_id_quantity",
+			wi = Weird::Info($ts=network_time(), $name="dns_unmatched_msg", $uid=msg$uid,
-			                 msg$id$orig_h, msg$id$resp_h);
+			                       $id=msg$id);
+			Weird::weird(wi);
 			# Throw away all unmatched on assumption they'll never be matched.
 			log_unmatched_msgs(msgs);
 			}
@@ -204,8 +212,9 @@ function enqueue_new_msg(msgs: PendingMessages, id: count, msg: Info)
 		{
 		if ( Queue::len(msgs[id]) > max_pending_msgs )
 			{
-			event flow_weird("dns_unmatched_msg_quantity",
+			wi = Weird::Info($ts=network_time(), $name="dns_unmatched_msg_quantity", $uid=msg$uid,
-			                 msg$id$orig_h, msg$id$resp_h);
+			                       $id=msg$id);
+			Weird::weird(wi);
 			log_unmatched_msgs_queue(msgs[id]);
 			# Throw away all unmatched on assumption they'll never be matched.
 			msgs[id] = Queue::init();
@@ -311,6 +320,16 @@ hook DNS::do_reply(c: connection, msg: dns_msg, ans: dns_answer, reply: string)
 		c$dns$AA    = msg$AA;
 		c$dns$RA    = msg$RA;
 
+		if ( ! c$dns?$rtt )
+			{
+			c$dns$rtt = network_time() - c$dns$ts;
+			# This could mean that only a reply was seen since 
+			# we assume there must be some passage of time between
+			# request and response.
+			if ( c$dns$rtt == 0secs )
+				delete c$dns$rtt;
+			}
+
 		if ( reply != "" )
 			{
 			if ( ! c$dns?$answers )

scripts/base/protocols/ftp/main.bro

@@ -241,10 +241,10 @@ event ftp_reply(c: connection, code: count, msg: string, cont_resp: bool) &prior
 	if ( [c$ftp$cmdarg$cmd, code] in directory_cmds )
 		{
 		if ( c$ftp$cmdarg$cmd == "CWD" )
-			c$ftp$cwd = build_path(c$ftp$cwd, c$ftp$cmdarg$arg);
+			c$ftp$cwd = build_path_compressed(c$ftp$cwd, c$ftp$cmdarg$arg);
 
 		else if ( c$ftp$cmdarg$cmd == "CDUP" )
-			c$ftp$cwd = cat(c$ftp$cwd, "/..");
+			c$ftp$cwd = build_path_compressed(c$ftp$cwd, "/..");
 
 		else if ( c$ftp$cmdarg$cmd == "PWD" || c$ftp$cmdarg$cmd == "XPWD" )
 			c$ftp$cwd = extract_path(msg);

scripts/base/protocols/http/entities.bro

@@ -17,12 +17,18 @@ export {
 		## An ordered vector of file unique IDs.
 		orig_fuids:      vector of string &log &optional;
 
+		## An order vector of filenames from the client.
+		orig_filenames:  vector of string &log &optional;
+
 		## An ordered vector of mime types.
 		orig_mime_types: vector of string &log &optional;
 
 		## An ordered vector of file unique IDs.
 		resp_fuids:      vector of string &log &optional;
 
+		## An order vector of filenames from the server.
+		resp_filenames:  vector of string &log &optional;
+
 		## An ordered vector of mime types.
 		resp_mime_types: vector of string &log &optional;
 
@@ -82,13 +88,31 @@ event file_over_new_connection(f: fa_file, c: connection, is_orig: bool) &priori
 				c$http$orig_fuids = string_vec(f$id);
 			else
 				c$http$orig_fuids[|c$http$orig_fuids|] = f$id;
+
+			if ( f$info?$filename )
+				{
+				if ( ! c$http?$orig_filenames )
+					c$http$orig_filenames = string_vec(f$info$filename);
+				else
+					c$http$orig_filenames[|c$http$orig_filenames|] = f$info$filename;
+				}
 			}
+
 		else
 			{
 			if ( ! c$http?$resp_fuids )
 				c$http$resp_fuids = string_vec(f$id);
 			else
 				c$http$resp_fuids[|c$http$resp_fuids|] = f$id;
+
+			if ( f$info?$filename )
+				{
+				if ( ! c$http?$resp_filenames )
+					c$http$resp_filenames = string_vec(f$info$filename);
+				else
+					c$http$resp_filenames[|c$http$resp_filenames|] = f$info$filename;
+				}
+
 			}
 		}
 	}

scripts/base/protocols/http/main.bro

@@ -60,9 +60,6 @@ export {
 		info_code:               count     &log &optional;
 		## Last seen 1xx informational reply message returned by the server.
 		info_msg:                string    &log &optional;
-		## Filename given in the Content-Disposition header sent by the
-		## server.
-		filename:                string    &log &optional;
 		## A set of indicators of various attributes discovered and
 		## related to a particular request/response pair.
 		tags:                    set[Tags] &log;

scripts/base/protocols/smtp/main.bro

@@ -1,13 +1,13 @@
 @load base/frameworks/notice
 @load base/utils/addrs
 @load base/utils/directions-and-hosts
+@load base/utils/email
 
 module SMTP;
 
 export {
 	redef enum Log::ID += { LOG };
 
-	## The record type which contains the fields of the SMTP log.
 	type Info: record {
 		## Time when the message was first seen.
 		ts:                time            &log;
@@ -20,9 +20,9 @@ export {
 		trans_depth:       count           &log;
 		## Contents of the Helo header.
 		helo:              string          &log &optional;
-		## Contents of the From header.
+		## Email addresses found in the From header.
 		mailfrom:          string          &log &optional;
-		## Contents of the Rcpt header.
+		## Email addresses found in the Rcpt header.
 		rcptto:            set[string]     &log &optional;
 		## Contents of the Date header.
 		date:              string          &log &optional;
@@ -100,7 +100,7 @@ event bro_init() &priority=5
 	}
 
 function find_address_in_smtp_header(header: string): string
-{
+	{
 	local ips = extract_ip_addresses(header);
 	# If there are more than one IP address found, return the second.
 	if ( |ips| > 1 )
@@ -111,7 +111,7 @@ function find_address_in_smtp_header(header: string): string
 	# Otherwise, there wasn't an IP address found.
 	else
 		return "";
-}
+	}
 
 function new_smtp_log(c: connection): Info
 	{
@@ -166,7 +166,14 @@ event smtp_request(c: connection, is_orig: bool, command: string, arg: string) &
 		{
 		if ( ! c$smtp?$rcptto )
 			c$smtp$rcptto = set();
-		add c$smtp$rcptto[split_string1(arg, /:[[:blank:]]*/)[1]];
+
+		local rcptto_addrs = extract_email_addrs_set(arg);
+		for ( rcptto_addr in rcptto_addrs )
+			{
+			rcptto_addr = gsub(rcptto_addr, /ORCPT=rfc822;?/, "");
+			add c$smtp$rcptto[rcptto_addr];
+			}
+
 		c$smtp$has_client_activity = T;
 		}
 
@@ -175,8 +182,9 @@ event smtp_request(c: connection, is_orig: bool, command: string, arg: string) &
 		# Flush last message in case we didn't see the server's acknowledgement.
 		smtp_message(c);
 
-		local partially_done = split_string1(arg, /:[[:blank:]]*/)[1];
+		local mailfrom = extract_first_email_addr(arg);
-		c$smtp$mailfrom = split_string1(partially_done, /[[:blank:]]?/)[0];
+		if ( mailfrom != "" )
+			c$smtp$mailfrom = mailfrom;
 		c$smtp$has_client_activity = T;
 		}
 	}
@@ -237,9 +245,11 @@ event mime_one_header(c: connection, h: mime_header_rec) &priority=5
 		if ( ! c$smtp?$to )
 			c$smtp$to = set();
 
-		local to_parts = split_string(h$value, /[[:blank:]]*,[[:blank:]]*/);
+		local to_email_addrs = split_mime_email_addresses(h$value);
-		for ( i in to_parts )
+		for ( to_email_addr in to_email_addrs )
-			add c$smtp$to[to_parts[i]];
+			{
+			add c$smtp$to[to_email_addr];
+			}
 		}
 
 	else if ( h$name == "CC" )
@@ -247,16 +257,16 @@ event mime_one_header(c: connection, h: mime_header_rec) &priority=5
 		if ( ! c$smtp?$cc )
 			c$smtp$cc = set();
 
-		local cc_parts = split_string(h$value, /[[:blank:]]*,[[:blank:]]*/);
+		local cc_parts = split_mime_email_addresses(h$value);
-		for ( i in cc_parts )
+		for ( cc_part in cc_parts )
-			add c$smtp$cc[cc_parts[i]];
+			add c$smtp$cc[cc_part];
 		}
 
 	else if ( h$name == "X-ORIGINATING-IP" )
 		{
 		local addresses = extract_ip_addresses(h$value);
-		if ( 1 in addresses )
+		if ( 0 in addresses )
-			c$smtp$x_originating_ip = to_addr(addresses[1]);
+			c$smtp$x_originating_ip = to_addr(addresses[0]);
 		}
 
 	else if ( h$name == "X-MAILER" ||
@@ -309,9 +319,9 @@ function describe(rec: Info): string
 	if ( rec?$mailfrom && rec?$rcptto )
 		{
 		local one_to = "";
-		for ( to in rec$rcptto )
+		for ( email in rec$rcptto )
 			{
-			one_to = to;
+			one_to = email;
 			break;
 			}
 		local abbrev_subject = "";

scripts/base/protocols/socks/main.bro

@@ -87,14 +87,6 @@ event socks_reply(c: connection, version: count, reply: count, sa: SOCKS::Addres
 	c$socks$bound_p = p;
 	}
 
-event socks_reply(c: connection, version: count, reply: count, sa: SOCKS::Address, p: port) &priority=-5
-	{
-	# This will handle the case where the analyzer failed in some way and was removed.  We probably 
-	# don't want to log these connections.
-	if ( "SOCKS" in c$service )
-		Log::write(SOCKS::LOG, c$socks);
-	}
-
 event socks_login_userpass_request(c: connection, user: string, password: string) &priority=5
 	{
 	# Authentication only possible with the version 5.
@@ -112,3 +104,10 @@ event socks_login_userpass_reply(c: connection, code: count) &priority=5
 	c$socks$status = v5_status[code];
 	}
 
+event connection_state_remove(c: connection)
+	{
+	# This will handle the case where the analyzer failed in some way and was
+	# removed.  We probably  don't want to log these connections.
+	if ( "SOCKS" in c$service )
+		Log::write(SOCKS::LOG, c$socks);
+	}

scripts/base/utils/email.bro

@@ -0,0 +1,68 @@
+## Extract mail addresses out of address specifications conforming to RFC5322.
+##
+## str: A string potentially containing email addresses.
+##
+## Returns: A vector of extracted email addresses.  An empty vector is returned
+##          if no email addresses are discovered.
+function extract_email_addrs_vec(str: string): string_vec
+	{
+	local addrs: vector of string = vector();
+
+	local raw_addrs = find_all(str, /(^|[<,:[:blank:]])[^<,:[:blank:]@]+"@"[^>,;[:blank:]]+([>,;[:blank:]]|$)/);
+	for ( raw_addr in raw_addrs )
+		addrs[|addrs|] = gsub(raw_addr, /[<>,:;[:blank:]]/, "");
+
+	return addrs;
+	}
+
+## Extract mail addresses out of address specifications conforming to RFC5322.
+##
+## str: A string potentially containing email addresses.
+##
+## Returns: A set of extracted email addresses.  An empty set is returned 
+##          if no email addresses are discovered.
+function extract_email_addrs_set(str: string): set[string]
+	{
+	local addrs: set[string] = set();
+
+	local raw_addrs = find_all(str, /(^|[<,:[:blank:]])[^<,:[:blank:]@]+"@"[^>,;[:blank:]]+([>,;[:blank:]]|$)/);
+	for ( raw_addr in raw_addrs )
+		add addrs[gsub(raw_addr, /[<>,:;[:blank:]]/, "")];
+
+	return addrs;
+	}
+
+## Extract the first email address from a string.
+##
+## str: A string potentially containing email addresses.
+##
+## Returns: An email address or empty string if none found.
+function extract_first_email_addr(str: string): string
+	{
+	local addrs = extract_email_addrs_vec(str);
+	if ( |addrs| > 0 )
+		return addrs[0];
+	else
+		return "";
+	}
+
+## Split email addresses from MIME headers.  The email addresses will
+## include the display name and email address as it was given by the mail
+## mail client.  Note that this currently does not account for MIME group
+## addresses and won't handle them correctly.  The group name will show up
+## as part of an email address.
+##
+## str: The argument from a MIME header.
+##
+## Returns: A set of addresses or empty string if none found.
+function split_mime_email_addresses(line: string): set[string]
+	{
+	local output = string_set();
+
+	local addrs = find_all(line, /(\"[^"]*\")?[^,]+/);
+	for ( part in addrs )
+		{
+		add output[strip(part)];
+		}
+	return output;
+	}

scripts/base/utils/exec.bro

@@ -116,7 +116,7 @@ event Input::end_of_data(orig_name: string, source:string)
 	if ( track_file !in result$files )
 		result$files[track_file] = vector();
 
-	Input::remove(name);
+	Input::remove(orig_name);
 
 	if ( name !in pending_files )
 		delete pending_commands[name];

scripts/base/utils/geoip-distance.bro

@@ -0,0 +1,26 @@
+##! Functions to calculate distance between two locations, based on GeoIP data.
+
+## Returns the distance between two IP addresses using the haversine formula,
+## based on GeoIP database locations.  Requires Bro to be built with libgeoip.
+##
+## a1: First IP address.
+##
+## a2: Second IP address.
+##
+## Returns: The distance between *a1* and *a2* in miles, or -1.0 if GeoIP data
+##          is not available for either of the IP addresses.
+##
+## .. bro:see:: haversine_distance lookup_location
+function haversine_distance_ip(a1: addr, a2: addr): double
+	{
+	local loc1 = lookup_location(a1);
+	local loc2 = lookup_location(a2);
+	local miles: double;
+
+	if ( loc1?$latitude && loc1?$longitude && loc2?$latitude && loc2?$longitude )
+		miles = haversine_distance(loc1$latitude, loc1$longitude, loc2$latitude, loc2$longitude);
+	else
+		miles = -1.0;
+
+	return miles;
+	}

scripts/policy/frameworks/intel/seen/smtp.bro

@@ -1,3 +1,4 @@
+@load base/utils/email
 @load base/frameworks/intel
 @load base/protocols/smtp
 @load ./where-locations
@@ -30,37 +31,28 @@ event mime_end_entity(c: connection)
 
 		if ( c$smtp?$mailfrom )
 			{
-			local mailfromparts = split_string_n(c$smtp$mailfrom, /<.+>/, T, 1);
+			Intel::seen([$indicator=c$smtp$mailfrom,
-			if ( |mailfromparts| > 2 )
+			             $indicator_type=Intel::EMAIL,
-				{
+			             $conn=c,
-				Intel::seen([$indicator=mailfromparts[1][1:-2],
+			             $where=SMTP::IN_MAIL_FROM]);
-				             $indicator_type=Intel::EMAIL,
-				             $conn=c,
-				             $where=SMTP::IN_MAIL_FROM]);
-				}
 			}
 
 		if ( c$smtp?$rcptto )
 			{
-			for ( rcptto in c$smtp$rcptto )
+			for ( rcptto_addr in c$smtp$rcptto )
 				{
-				local rcpttoparts = split_string_n(rcptto, /<.+>/, T, 1);
+				Intel::seen([$indicator=rcptto_addr,
-				if ( |rcpttoparts| > 2 )
+				             $indicator_type=Intel::EMAIL,
-					{
+				             $conn=c,
-					Intel::seen([$indicator=rcpttoparts[1][1:-2],
+				             $where=SMTP::IN_RCPT_TO]);
-					             $indicator_type=Intel::EMAIL,
-					             $conn=c,
-					             $where=SMTP::IN_RCPT_TO]);
-					}
 				}
 			}
 
 		if ( c$smtp?$from )
 			{
-			local fromparts = split_string_n(c$smtp$from, /<.+>/, T, 1);
+			for ( from_addr in extract_email_addrs_set(c$smtp$from) )
-			if ( |fromparts| > 2 )
 				{
-				Intel::seen([$indicator=fromparts[1][1:-2],
+				Intel::seen([$indicator=from_addr,
 				             $indicator_type=Intel::EMAIL,
 				             $conn=c,
 				             $where=SMTP::IN_FROM]);
@@ -69,29 +61,32 @@ event mime_end_entity(c: connection)
 
 		if ( c$smtp?$to )
 			{
-			for ( email_to in c$smtp$to )
+			for ( email_to_addr in c$smtp$to )
 				{
-				local toparts = split_string_n(email_to, /<.+>/, T, 1);
+				Intel::seen([$indicator=extract_first_email_addr(email_to_addr),
-				if ( |toparts| > 2 )
+				             $indicator_type=Intel::EMAIL,
-					{
+				             $conn=c,
-					Intel::seen([$indicator=toparts[1][1:-2],
+				             $where=SMTP::IN_TO]);
-					             $indicator_type=Intel::EMAIL,
+				}
-					             $conn=c,
+			}
-					             $where=SMTP::IN_TO]);
+
-					}
+		if ( c$smtp?$cc )
+			{
+			for ( cc_addr in c$smtp$cc )
+				{
+				Intel::seen([$indicator=cc_addr,
+				             $indicator_type=Intel::EMAIL,
+				             $conn=c,
+				             $where=SMTP::IN_CC]);
 				}
 			}
 
 		if ( c$smtp?$reply_to )
 			{
-			local replytoparts = split_string_n(c$smtp$reply_to, /<.+>/, T, 1);
+			Intel::seen([$indicator=c$smtp$reply_to,
-			if ( |replytoparts| > 2 )
+			             $indicator_type=Intel::EMAIL,
-				{
+			             $conn=c,
-				Intel::seen([$indicator=replytoparts[1][1:-2],
+			             $where=SMTP::IN_REPLY_TO]);
-				             $indicator_type=Intel::EMAIL,
-				             $conn=c,
-				             $where=SMTP::IN_REPLY_TO]);
-				}
 			}
 		}
 	}

scripts/policy/frameworks/intel/seen/where-locations.bro

@@ -17,6 +17,7 @@ export {
 		SMTP::IN_RCPT_TO,
 		SMTP::IN_FROM,
 		SMTP::IN_TO,
+		SMTP::IN_CC,
 		SMTP::IN_RECEIVED_HEADER,
 		SMTP::IN_REPLY_TO,
 		SMTP::IN_X_ORIGINATING_IP_HEADER,

scripts/policy/misc/app-stats/README

@@ -1 +0,0 @@
-AppStats collects information about web applications in use on the network.

scripts/policy/misc/app-stats/__load__.bro

@@ -1,2 +0,0 @@
-@load ./main
-@load ./plugins

scripts/policy/misc/app-stats/main.bro

@@ -1,77 +0,0 @@
-##! AppStats collects information about web applications in use
-##! on the network.  
-
-@load base/protocols/http
-@load base/protocols/ssl
-@load base/frameworks/sumstats
-
-module AppStats;
-
-export {
-	redef enum Log::ID += { LOG };
-
-	type Info: record {
-		## Timestamp when the log line was finished and written.
-		ts:         time   &log;
-		## Time interval that the log line covers.
-		ts_delta:   interval &log;
-		## The name of the "app", like "facebook" or "netflix".
-		app:        string &log;
-		## The number of unique local hosts using the app.
-		uniq_hosts: count  &log;
-		## The number of hits to the app in total.
-		hits:       count  &log;
-		## The total number of bytes received by users of the app.
-		bytes:      count  &log;
-	};
-
-	## The frequency of logging the stats collected by this script.
-	const break_interval = 15mins &redef;
-}
-
-redef record connection += {
-	resp_hostname: string &optional;
-};
-
-global add_sumstats: hook(id: conn_id, hostname: string, size: count);
-
-
-event bro_init() &priority=3
-	{
-	Log::create_stream(AppStats::LOG, [$columns=Info, $path="app_stats"]);
-
-	local r1: SumStats::Reducer = [$stream="apps.bytes", $apply=set(SumStats::SUM)];
-	local r2: SumStats::Reducer = [$stream="apps.hits",  $apply=set(SumStats::UNIQUE)];
-	SumStats::create([$name="app-metrics",
-	                  $epoch=break_interval,
-	                  $reducers=set(r1, r2),
-	                  $epoch_result(ts: time, key: SumStats::Key, result: SumStats::Result) =
-	                  	{
-	                  	local l: Info;
-	                  	l$ts         = network_time();
-	                  	l$ts_delta   = break_interval;
-	                  	l$app        = key$str;
-	                  	l$bytes      = double_to_count(floor(result["apps.bytes"]$sum));
-	                  	l$hits       = result["apps.hits"]$num;
-	                  	l$uniq_hosts = result["apps.hits"]$unique;
-	                  	Log::write(LOG, l);
-	                  	}]);
-	}
-
-event ssl_established(c: connection)
-	{
-	if ( c?$ssl && c$ssl?$server_name )
-		c$resp_hostname = c$ssl$server_name;
-	}
-
-event connection_finished(c: connection)
-	{
-	if ( c?$resp_hostname )
-		hook add_sumstats(c$id, c$resp_hostname, c$resp$size);
-	}
-
-event HTTP::log_http(rec: HTTP::Info)
-	{
-	if( rec?$host )
-		hook add_sumstats(rec$id, rec$host, rec$response_body_len);
-	}

scripts/policy/misc/app-stats/plugins/README

@@ -1 +0,0 @@
-Plugins for AppStats.

scripts/policy/misc/app-stats/plugins/__load__.bro

@@ -1,6 +0,0 @@
-@load ./facebook
-#@load ./gmail
-#@load ./google
-#@load ./netflix
-#@load ./pandora
-#@load ./youtube

scripts/policy/misc/app-stats/plugins/facebook.bro

@@ -1,12 +0,0 @@
-@load ../main
-
-module AppStats;
-
-hook add_sumstats(id: conn_id, hostname: string, size: count)
-	{
-	if ( /\.(facebook\.com|fbcdn\.net)$/ in hostname && size > 20 )
-		{
-		SumStats::observe("apps.bytes", [$str="facebook"], [$num=size]);
-		SumStats::observe("apps.hits",  [$str="facebook"], [$str=cat(id$orig_h)]);
-		}
-	}

scripts/policy/misc/app-stats/plugins/gmail.bro

@@ -1,12 +0,0 @@
-@load ../main
-
-module AppStats;
-
-hook add_sumstats(id: conn_id, hostname: string, size: count)
-	{
-	if ( /\.gmail\.com$/ in hostname && size > 20 )
-		{
-		SumStats::observe("apps.bytes", [$str="gmail"], [$num=size]);
-		SumStats::observe("apps.hits",  [$str="gmail"], [$str=cat(id$orig_h)]);
-		}
-	}

scripts/policy/misc/app-stats/plugins/google.bro

@@ -1,12 +0,0 @@
-@load ../main
-
-module AppStats;
-
-hook add_sumstats(id: conn_id, hostname: string, size: count)
-	{
-	if ( /\.google\.com$/ in hostname && size > 20 )
-		{
-		SumStats::observe("apps.bytes", [$str="google"], [$num=size]);
-		SumStats::observe("apps.hits",  [$str="google"], [$str=cat(id$orig_h)]);
-		}
-	}

scripts/policy/misc/app-stats/plugins/netflix.bro

@@ -1,12 +0,0 @@
-@load ../main
-
-module AppStats;
-
-hook add_sumstats(id: conn_id, hostname: string, size: count)
-	{
-	if ( /\.nflximg\.com$/ in hostname && size > 200*1024 )
-		{
-		SumStats::observe("apps.bytes", [$str="netflix"], [$num=size]);
-		SumStats::observe("apps.hits",  [$str="netflix"], [$str=cat(id$orig_h)]);
-		}
-	}

scripts/policy/misc/app-stats/plugins/pandora.bro

@@ -1,12 +0,0 @@
-@load ../main
-
-module AppStats;
-
-hook add_sumstats(id: conn_id, hostname: string, size: count)
-	{
-	if ( /\.(pandora|p-cdn)\.com$/ in hostname && size > 512*1024 )
-		{
-		SumStats::observe("apps.bytes", [$str="pandora"], [$num=size]);
-		SumStats::observe("apps.hits",  [$str="pandora"], [$str=cat(id$orig_h)]);
-		}
-	}

scripts/policy/misc/app-stats/plugins/youtube.bro

@@ -1,12 +0,0 @@
-@load ../main
-
-module AppStats;
-
-hook add_sumstats(id: conn_id, hostname: string, size: count)
-	{
-	if ( /\.youtube\.com$/ in hostname && size > 512*1024 )
-		{
-		SumStats::observe("apps.bytes", [$str="youtube"], [$num=size]);
-		SumStats::observe("apps.hits",  [$str="youtube"], [$str=cat(id$orig_h)]);
-		}
-	}

scripts/policy/protocols/conn/mac-logging.bro

@@ -0,0 +1,24 @@
+##! This script adds link-layer address (MAC) information to the connection logs
+
+@load base/protocols/conn
+
+module Conn;
+
+redef record Info += {
+	## Link-layer address of the originator, if available.
+	orig_l2_addr: string	&log &optional;
+	## Link-layer address of the responder, if available.
+	resp_l2_addr: string	&log &optional;
+};
+
+# Add the link-layer addresses to the Conn::Info structure after the connection
+# has been removed. This ensures it's only done once, and is done before the
+# connection information is written to the log.
+event connection_state_remove(c: connection)
+	{
+	if ( c$orig?$l2_addr )
+		c$conn$orig_l2_addr = c$orig$l2_addr;
+
+	if ( c$resp?$l2_addr )
+		c$conn$resp_l2_addr = c$resp$l2_addr;
+	}

scripts/site/local.bro

@@ -11,10 +11,6 @@
 # Load the scan detection script.
 @load misc/scan
 
-# Log some information about web applications being used by users
-# on your network.
-@load misc/app-stats
-
 # Detect traceroute being run on the network.
 @load misc/detect-traceroute
 
@@ -88,3 +84,7 @@
 # Uncomment the following line to enable logging of connection VLANs. Enabling
 # this adds two VLAN fields to the conn.log file.
 # @load policy/protocols/conn/vlan-logging
+
+# Uncomment the following line to enable logging of link-layer addresses. Enabling
+# this adds the link-layer address for each connection endpoint to the conn.log file.
+# @load policy/protocols/conn/mac-logging

scripts/test-all-policy.bro

@@ -41,15 +41,6 @@
 @load integration/barnyard2/types.bro
 @load integration/collective-intel/__load__.bro
 @load integration/collective-intel/main.bro
-@load misc/app-stats/__load__.bro
-@load misc/app-stats/main.bro
-@load misc/app-stats/plugins/__load__.bro
-@load misc/app-stats/plugins/facebook.bro
-@load misc/app-stats/plugins/gmail.bro
-@load misc/app-stats/plugins/google.bro
-@load misc/app-stats/plugins/netflix.bro
-@load misc/app-stats/plugins/pandora.bro
-@load misc/app-stats/plugins/youtube.bro
 @load misc/capture-loss.bro
 @load misc/detect-traceroute/__load__.bro
 @load misc/detect-traceroute/main.bro
@@ -63,6 +54,7 @@
 @load misc/trim-trace-file.bro
 @load protocols/conn/known-hosts.bro
 @load protocols/conn/known-services.bro
+@load protocols/conn/mac-logging.bro
 @load protocols/conn/vlan-logging.bro
 @load protocols/conn/weirds.bro
 @load protocols/dhcp/known-devices-and-hostnames.bro

src/Conn.cc

@@ -115,7 +115,7 @@ uint64 Connection::external_connections = 0;
 IMPLEMENT_SERIAL(Connection, SER_CONNECTION);
 
 Connection::Connection(NetSessions* s, HashKey* k, double t, const ConnID* id,
-                       uint32 flow, uint32 arg_vlan, uint32 arg_inner_vlan,
+                       uint32 flow, const Packet* pkt,
 		       const EncapsulationStack* arg_encap)
 	{
 	sessions = s;
@@ -132,8 +132,18 @@ Connection::Connection(NetSessions* s, HashKey* k, double t, const ConnID* id,
 	saw_first_orig_packet = 1;
 	saw_first_resp_packet = 0;
 
-	vlan = arg_vlan;
+	if ( pkt->l2_src )
-	inner_vlan = arg_inner_vlan;
+		memcpy(orig_l2_addr, pkt->l2_src, sizeof(orig_l2_addr));
+	else
+		bzero(orig_l2_addr, sizeof(orig_l2_addr));
+
+	if ( pkt->l2_dst )
+		memcpy(resp_l2_addr, pkt->l2_dst, sizeof(resp_l2_addr));
+	else
+		bzero(resp_l2_addr, sizeof(resp_l2_addr));
+
+	vlan = pkt->vlan;
+	inner_vlan = pkt->inner_vlan;
 
 	conn_val = 0;
 	login_conn = 0;
@@ -363,11 +373,20 @@ RecordVal* Connection::BuildConnVal()
 		orig_endp->Assign(1, new Val(0, TYPE_COUNT));
 		orig_endp->Assign(4, new Val(orig_flow_label, TYPE_COUNT));
 
+		const int l2_len = sizeof(orig_l2_addr);
+		char null[l2_len]{};
+
+		if ( memcmp(&orig_l2_addr, &null, l2_len) != 0 )
+			orig_endp->Assign(5, new StringVal(fmt_mac(orig_l2_addr, l2_len)));
+
 		RecordVal *resp_endp = new RecordVal(endpoint);
 		resp_endp->Assign(0, new Val(0, TYPE_COUNT));
 		resp_endp->Assign(1, new Val(0, TYPE_COUNT));
 		resp_endp->Assign(4, new Val(resp_flow_label, TYPE_COUNT));
 
+		if ( memcmp(&resp_l2_addr, &null, l2_len) != 0 )
+			resp_endp->Assign(5, new StringVal(fmt_mac(resp_l2_addr, l2_len)));
+
 		conn_val->Assign(0, id_val);
 		conn_val->Assign(1, orig_endp);
 		conn_val->Assign(2, resp_endp);
@@ -388,6 +407,7 @@ RecordVal* Connection::BuildConnVal()
 
 		if ( inner_vlan != 0 )
 			conn_val->Assign(10, new Val(inner_vlan, TYPE_INT));
+
 		}
 
 	if ( root_analyzer )
@@ -732,6 +752,12 @@ void Connection::FlipRoles()
 	resp_port = orig_port;
 	orig_port = tmp_port;
 
+	const int l2_len = sizeof(orig_l2_addr);
+	u_char tmp_l2_addr[l2_len];
+	memcpy(tmp_l2_addr, resp_l2_addr, l2_len);
+	memcpy(resp_l2_addr, orig_l2_addr, l2_len);
+	memcpy(orig_l2_addr, tmp_l2_addr, l2_len);
+
 	bool tmp_bool = saw_first_resp_packet;
 	saw_first_resp_packet = saw_first_orig_packet;
 	saw_first_orig_packet = tmp_bool;

src/Conn.h

@@ -56,7 +56,7 @@ namespace analyzer { class Analyzer; }
 class Connection : public BroObj {
 public:
 	Connection(NetSessions* s, HashKey* k, double t, const ConnID* id,
-	           uint32 flow, uint32 vlan, uint32 inner_vlan, const EncapsulationStack* arg_encap);
+	           uint32 flow, const Packet* pkt, const EncapsulationStack* arg_encap);
 	virtual ~Connection();
 
 	// Invoked when an encapsulation is discovered. It records the
@@ -296,6 +296,8 @@ protected:
 	TransportProto proto;
 	uint32 orig_flow_label, resp_flow_label;	// most recent IPv6 flow labels
 	uint32 vlan, inner_vlan;	// VLAN this connection traverses, if available
+	u_char orig_l2_addr[Packet::l2_addr_len];	// Link-layer originator address, if available
+	u_char resp_l2_addr[Packet::l2_addr_len];	// Link-layer responder address, if available
 	double start_time, last_time;
 	double inactivity_timeout;
 	RecordVal* conn_val;

src/DebugLogger.cc

@@ -7,7 +7,7 @@
 #include "Net.h"
 #include "plugin/Plugin.h"
 
-DebugLogger debug_logger("debug");
+DebugLogger debug_logger;
 
 // Same order here as in DebugStream.
 DebugLogger::Stream DebugLogger::streams[NUM_DBGS] = {
@@ -22,7 +22,19 @@ DebugLogger::Stream DebugLogger::streams[NUM_DBGS] = {
 	{ "pktio", 0, false }, { "broker", 0, false }
 };
 
-DebugLogger::DebugLogger(const char* filename)
+DebugLogger::DebugLogger()
+	{
+	verbose = false;
+	file = 0;
+	}
+
+DebugLogger::~DebugLogger()
+	{
+	if ( file && file != stderr )
+		fclose(file);
+	}
+
+void DebugLogger::OpenDebugLog(const char* filename)
 	{
 	if ( filename )
 		{
@@ -45,14 +57,6 @@ DebugLogger::DebugLogger(const char* filename)
 		}
 	else
 		file = stderr;
-
-	verbose = false;
-	}
-
-DebugLogger::~DebugLogger()
-	{
-	if ( file != stderr )
-		fclose(file);
 	}
 
 void DebugLogger::ShowStreamsHelp()

src/DebugLogger.h

@@ -53,9 +53,11 @@ namespace plugin { class Plugin; }
 class DebugLogger {
 public:
 	// Output goes to stderr per default.
-	DebugLogger(const char* filename = 0);
+	DebugLogger();
 	~DebugLogger();
 
+	void OpenDebugLog(const char* filename = 0);
+
 	void Log(DebugStream stream, const char* fmt, ...);
 	void Log(const plugin::Plugin& plugin, const char* fmt, ...);
 

src/Desc.cc

@@ -4,6 +4,7 @@
 
 #include <stdlib.h>
 #include <errno.h>
+#include <math.h>
 
 #include "Desc.h"
 #include "File.h"
@@ -138,17 +139,22 @@ void ODesc::Add(uint64 u)
 		}
 	}
 
-void ODesc::Add(double d)
+void ODesc::Add(double d, bool no_exp)
 	{
 	if ( IsBinary() )
 		AddBytes(&d, sizeof(d));
 	else
 		{
 		char tmp[256];
-		modp_dtoa2(d, tmp, IsReadable() ? 6 : 8);
+
+		if ( no_exp )
+			modp_dtoa3(d, tmp, sizeof(tmp), IsReadable() ? 6 : 8);
+		else
+			modp_dtoa2(d, tmp, IsReadable() ? 6 : 8);
+
 		Add(tmp);
 
-		if ( d == double(int(d)) )
+		if ( nearbyint(d) == d && isfinite(d) && ! strchr(tmp, 'e') )
 			// disambiguate from integer
 			Add(".0");
 		}

src/Desc.h

@@ -81,7 +81,7 @@ public:
 	void Add(uint32 u);
 	void Add(int64 i);
 	void Add(uint64 u);
-	void Add(double d);
+	void Add(double d, bool no_exp=false);
 	void Add(const IPAddr& addr);
 	void Add(const IPPrefix& prefix);
 

src/Event.cc

@@ -94,26 +94,6 @@ void EventMgr::QueueEvent(Event* event)
 	++num_events_queued;
 	}
 
-void EventMgr::Dispatch()
-	{
-	if ( ! head )
-		reporter->InternalError("EventMgr::Dispatch underflow");
-
-	Event* current = head;
-
-	head = head->NextEvent();
-	if ( ! head )
-		tail = head;
-
-	current_src = current->Source();
-	current_mgr = current->Mgr();
-	current_aid = current->Analyzer();
-	current->Dispatch();
-	Unref(current);
-
-	++num_events_dispatched;
-	}
-
 void EventMgr::Drain()
 	{
 	if ( event_queue_flush_point )
@@ -124,8 +104,34 @@ void EventMgr::Drain()
 	PLUGIN_HOOK_VOID(HOOK_DRAIN_EVENTS, HookDrainEvents());
 
 	draining = true;
-	while ( head )
+
-		Dispatch();
+	// Past Bro versions drained as long as there events, including when
+	// a handler queued new events during its execution. This could lead
+	// to endless loops in case a handler kept triggering its own event.
+	// We now limit this to just a couple of rounds. We do more than
+	// just one round to make it less likley to break existing scripts
+	// that expect the old behavior to trigger something quickly.
+
+	for ( int round = 0; head && round < 2; round++ )
+		{
+		Event* current = head;
+		head = 0;
+		tail = 0;
+
+		while ( current )
+			{
+			Event* next = current->NextEvent();
+
+			current_src = current->Source();
+			current_mgr = current->Mgr();
+			current_aid = current->Analyzer();
+			current->Dispatch();
+			Unref(current);
+
+			++num_events_dispatched;
+			current = next;
+			}
+		}
 
 	// Note: we might eventually need a general way to specify things to
 	// do after draining events.

src/Event.h

@@ -90,8 +90,6 @@ public:
 			delete_vals(vl);
 		}
 
-	void Dispatch();
-
 	void Dispatch(Event* event, bool no_remote = false)
 		{
 		current_src = event->Source();

src/PriorityQueue.h

@@ -21,7 +21,7 @@ public:
 	void MinimizeTime()	{ time = -HUGE_VAL; }
 
 protected:
-	PQ_Element()		{ }
+	PQ_Element()		{ time = 0; offset = -1; }
 	double time;
 	int offset;
 };

src/Serializer.h

@@ -125,7 +125,7 @@ protected:
 
 	// This will be increased whenever there is an incompatible change
 	// in the data format.
-	static const uint32 DATA_FORMAT_VERSION = 25;
+	static const uint32 DATA_FORMAT_VERSION = 26;
 
 	ChunkedIO* io;
 

src/Sessions.cc

@@ -674,7 +674,7 @@ void NetSessions::DoNextPacket(double t, const Packet* pkt, const IP_Hdr* ip_hdr
 	conn = (Connection*) d->Lookup(h);
 	if ( ! conn )
 		{
-		conn = NewConn(h, t, &id, data, proto, ip_hdr->FlowLabel(), pkt->vlan, pkt->inner_vlan, encapsulation);
+		conn = NewConn(h, t, &id, data, proto, ip_hdr->FlowLabel(), pkt, encapsulation);
 		if ( conn )
 			d->Insert(h, conn);
 		}
@@ -694,7 +694,7 @@ void NetSessions::DoNextPacket(double t, const Packet* pkt, const IP_Hdr* ip_hdr
 				conn->Event(connection_reused, 0);
 
 			Remove(conn);
-			conn = NewConn(h, t, &id, data, proto, ip_hdr->FlowLabel(), pkt->vlan, pkt->inner_vlan, encapsulation);
+			conn = NewConn(h, t, &id, data, proto, ip_hdr->FlowLabel(), pkt, encapsulation);
 			if ( conn )
 				d->Insert(h, conn);
 			}
@@ -1172,8 +1172,7 @@ void NetSessions::GetStats(SessionStats& s) const
 
 Connection* NetSessions::NewConn(HashKey* k, double t, const ConnID* id,
 					const u_char* data, int proto, uint32 flow_label,
-					uint32 vlan, uint32 inner_vlan,
+					const Packet* pkt, const EncapsulationStack* encapsulation)
-					const EncapsulationStack* encapsulation)
 	{
 	// FIXME: This should be cleaned up a bit, it's too protocol-specific.
 	// But I'm not yet sure what the right abstraction for these things is.
@@ -1229,7 +1228,7 @@ Connection* NetSessions::NewConn(HashKey* k, double t, const ConnID* id,
 		id = &flip_id;
 		}
 
-	Connection* conn = new Connection(this, k, t, id, flow_label, vlan, inner_vlan, encapsulation);
+	Connection* conn = new Connection(this, k, t, id, flow_label, pkt, encapsulation);
 	conn->SetTransport(tproto);
 
 	if ( ! analyzer_mgr->BuildInitialAnalyzerTree(conn) )

src/Sessions.h

@@ -185,8 +185,7 @@ protected:
 
 	Connection* NewConn(HashKey* k, double t, const ConnID* id,
 			const u_char* data, int proto, uint32 flow_lable,
-			uint32 vlan, uint32 inner_vlan,
+			const Packet* pkt, const EncapsulationStack* encapsulation);
-			const EncapsulationStack* encapsulation);
 
 	// Check whether the tag of the current packet is consistent with
 	// the given connection.  Returns:

src/Val.cc

@@ -1323,7 +1323,7 @@ void TableVal::Init(TableType* t)
 	{
 	::Ref(t);
 	table_type = t;
-	expire_expr = 0;
+	expire_func = 0;
 	expire_time = 0;
 	expire_cookie = 0;
 	timer = 0;
@@ -1350,7 +1350,8 @@ TableVal::~TableVal()
 	delete subnets;
 	Unref(attrs);
 	Unref(def_val);
-	Unref(expire_expr);
+	Unref(expire_func);
+	Unref(expire_time);
 	}
 
 void TableVal::RemoveAll()
@@ -1399,8 +1400,8 @@ void TableVal::SetAttrs(Attributes* a)
 	Attr* ef = attrs->FindAttr(ATTR_EXPIRE_FUNC);
 	if ( ef )
 		{
-		expire_expr = ef->AttrExpr();
+		expire_func = ef->AttrExpr();
-		expire_expr->Ref();
+		expire_func->Ref();
 		}
 	}
 
@@ -1410,15 +1411,17 @@ void TableVal::CheckExpireAttr(attr_tag at)
 
 	if ( a )
 		{
-		Val* timeout = a->AttrExpr()->Eval(0);
+		expire_time = a->AttrExpr();
-		if ( ! timeout )
+		expire_time->Ref();
+
+		if ( expire_time->Type()->Tag() != TYPE_INTERVAL )
 			{
-			a->AttrExpr()->Error("value of timeout not fixed");
+			if ( ! expire_time->IsError() )
+				expire_time->SetError("expiration interval has wrong type");
+
 			return;
 			}
 
-		expire_time = timeout->AsInterval();
-
 		if ( timer )
 			timer_mgr->Cancel(timer);
 
@@ -1791,7 +1794,7 @@ Val* TableVal::Lookup(Val* index, bool use_default_val)
 			if ( attrs && attrs->FindAttr(ATTR_EXPIRE_READ) )
 					{
 					v->SetExpireAccess(network_time);
-					if ( LoggingAccess() && expire_time )
+					if ( LoggingAccess() && ExpirationEnabled() )
 						ReadOperation(index, v);
 					}
 
@@ -1822,7 +1825,7 @@ Val* TableVal::Lookup(Val* index, bool use_default_val)
 				if ( attrs && attrs->FindAttr(ATTR_EXPIRE_READ) )
 					{
 					v->SetExpireAccess(network_time);
-					if ( LoggingAccess() && expire_time )
+					if ( LoggingAccess() && ExpirationEnabled() )
 						ReadOperation(index, v);
 					}
 
@@ -1880,7 +1883,7 @@ TableVal* TableVal::LookupSubnetValues(const SubNetVal* search)
 			if ( attrs && attrs->FindAttr(ATTR_EXPIRE_READ) )
 				{
 				entry->SetExpireAccess(network_time);
-				if ( LoggingAccess() && expire_time )
+				if ( LoggingAccess() && ExpirationEnabled() )
 					ReadOperation(s, entry);
 				}
 			}
@@ -2176,6 +2179,13 @@ void TableVal::DoExpire(double t)
 
 	PDict(TableEntryVal)* tbl = AsNonConstTable();
 
+	double timeout = GetExpireTime();
+
+	if ( timeout < 0 )
+		// Skip in case of unset/invalid expiration value. If it's an
+		// error, it has been reported already.
+		return;
+
 	if ( ! expire_cookie )
 		{
 		expire_cookie = tbl->InitForIteration();
@@ -2197,11 +2207,11 @@ void TableVal::DoExpire(double t)
 			// correct, so we just need to wait.
 			}
 
-		else if ( v->ExpireAccessTime() + expire_time < t )
+		else if ( v->ExpireAccessTime() + timeout < t )
 			{
 			Val* val = v->Value();
 
-			if ( expire_expr )
+			if ( expire_func )
 				{
 				Val* idx = RecoverIndex(k);
 				double secs = CallExpireFunc(idx);
@@ -2221,7 +2231,7 @@ void TableVal::DoExpire(double t)
 					{
 					// User doesn't want us to expire
 					// this now.
-					v->SetExpireAccess(network_time - expire_time + secs);
+					v->SetExpireAccess(network_time - timeout + secs);
 					delete k;
 					continue;
 					}
@@ -2258,9 +2268,29 @@ void TableVal::DoExpire(double t)
 		InitTimer(table_expire_delay);
 	}
 
+double TableVal::GetExpireTime()
+	{
+	if ( ! expire_time )
+		return -1;
+
+	Val* timeout = expire_time->Eval(0);
+	double interval = (timeout ? timeout->AsInterval() : -1);
+	Unref(timeout);
+
+	if ( interval >= 0 )
+		return timeout->AsInterval();
+
+	expire_time = 0;
+
+	if ( timer )
+		timer_mgr->Cancel(timer);
+
+	return -1;
+	}
+
 double TableVal::CallExpireFunc(Val* idx)
 	{
-	if ( ! expire_expr )
+	if ( ! expire_func )
 		{
 		Unref(idx);
 		return 0;
@@ -2285,8 +2315,26 @@ double TableVal::CallExpireFunc(Val* idx)
 
 	try
 		{
-		Val* vs = expire_expr->Eval(0)->AsFunc()->Call(vl);
+		Val* vf = expire_func->Eval(0);
+
+		if ( ! vf )
+			{
+			// Will have been reported already.
+			delete_vals(vl);
+			return 0;
+			}
+
+		if ( vf->Type()->Tag() != TYPE_FUNC )
+			{
+			Unref(vf);
+			vf->Error("not a function");
+			return 0;
+			}
+
+		Val* vs = vf->AsFunc()->Call(vl);
 		secs = vs->AsInterval();
+
+		Unref(vf);
 		Unref(vs);
 		delete vl;
 		}
@@ -2301,11 +2349,18 @@ double TableVal::CallExpireFunc(Val* idx)
 
 void TableVal::ReadOperation(Val* index, TableEntryVal* v)
 	{
+	double timeout = GetExpireTime();
+
+	if ( timeout < 0 )
+		// Skip in case of unset/invalid expiration value. If it's an
+		// error, it has been reported already.
+		return;
+
 	// In theory we need to only propagate one update per &read_expire
 	// interval to prevent peers from expiring intervals. To account for
 	// practical issues such as latency, we send one update every half
 	// &read_expire.
-	if ( network_time - v->LastReadUpdate() > expire_time / 2 )
+	if ( network_time - v->LastReadUpdate() > timeout / 2 )
 		{
 		StateAccess::Log(new StateAccess(OP_READ_IDX, this, index));
 		v->SetLastReadUpdate(network_time);
@@ -2344,11 +2399,9 @@ bool TableVal::DoSerialize(SerialInfo* info) const
 		state->did_index = false;
 		info->s->WriteOpenTag(table_type->IsSet() ? "set" : "table");
 
-		if ( ! SERIALIZE(expire_time) )
-			return false;
-
 		SERIALIZE_OPTIONAL(attrs);
-		SERIALIZE_OPTIONAL(expire_expr);
+		SERIALIZE_OPTIONAL(expire_time);
+		SERIALIZE_OPTIONAL(expire_func);
 
 		// Make sure nobody kills us in between.
 		const_cast<TableVal*>(this)->Ref();
@@ -2401,7 +2454,7 @@ bool TableVal::DoSerialize(SerialInfo* info) const
 			}
 
 		// Serialize index.
-		if ( ! state->did_index )
+		if ( k && ! state->did_index )
 			{
 			// Indices are rather small, so we disable suspension
 			// here again.
@@ -2473,13 +2526,11 @@ bool TableVal::DoUnserialize(UnserialInfo* info)
 	{
 	DO_UNSERIALIZE(MutableVal);
 
-	if ( ! UNSERIALIZE(&expire_time) )
-		return false;
-
 	Init((TableType*) type);
 
 	UNSERIALIZE_OPTIONAL(attrs, Attributes::Unserialize(info));
-	UNSERIALIZE_OPTIONAL(expire_expr, Expr::Unserialize(info));
+	UNSERIALIZE_OPTIONAL(expire_time, Expr::Unserialize(info));
+	UNSERIALIZE_OPTIONAL(expire_func, Expr::Unserialize(info));
 
 	while ( true )
 		{

src/Val.h

@@ -644,7 +644,7 @@ protected:
 	DECLARE_SERIAL(PatternVal);
 };
 
-// ListVals are mainly used to index tables that have more than one 
+// ListVals are mainly used to index tables that have more than one
 // element in their index.
 class ListVal : public Val {
 public:
@@ -862,6 +862,14 @@ protected:
 	// Calculates default value for index.  Returns 0 if none.
 	Val* Default(Val* index);
 
+	// Returns true if item expiration is enabled.
+	bool ExpirationEnabled()	{ return expire_time != 0; }
+
+	// Returns the expiration time defined by %{create,read,write}_expire
+	// attribute, or -1 for unset/invalid values. In the invalid case, an
+	// error will have been reported.
+	double GetExpireTime();
+
 	// Calls &expire_func and returns its return interval;
 	// takes ownership of the reference.
 	double CallExpireFunc(Val *idx);
@@ -874,8 +882,8 @@ protected:
 	TableType* table_type;
 	CompositeHash* table_hash;
 	Attributes* attrs;
-	double expire_time;
+	Expr* expire_time;
-	Expr* expire_expr;
+	Expr* expire_func;
 	TableValTimer* timer;
 	IterCookie* expire_cookie;
 	PrefixTable* subnets;

src/analyzer/Analyzer.cc

@@ -669,11 +669,7 @@ void Analyzer::ProtocolConfirmation(Tag arg_tag)
 	vl->append(BuildConnVal());
 	vl->append(tval);
 	vl->append(new Val(id, TYPE_COUNT));
-
+	mgr.QueueEvent(protocol_confirmation, vl);
-	// We immediately raise the event so that the analyzer can quickly
-	// react if necessary.
-	::Event* e = new ::Event(protocol_confirmation, vl, SOURCE_LOCAL);
-	mgr.Dispatch(e);
 
 	protocol_confirmed = true;
 	}
@@ -701,11 +697,7 @@ void Analyzer::ProtocolViolation(const char* reason, const char* data, int len)
 	vl->append(tval);
 	vl->append(new Val(id, TYPE_COUNT));
 	vl->append(r);
-
+	mgr.QueueEvent(protocol_violation, vl);
-	// We immediately raise the event so that the analyzer can quickly be
-	// disabled if necessary.
-	::Event* e = new ::Event(protocol_violation, vl, SOURCE_LOCAL);
-	mgr.Dispatch(e);
 	}
 
 void Analyzer::AddTimer(analyzer_timer_func timer, double t,

src/analyzer/protocol/conn-size/ConnSize.cc

@@ -12,7 +12,8 @@ using namespace analyzer::conn_size;
 
 ConnSize_Analyzer::ConnSize_Analyzer(Connection* c)
     : Analyzer("CONNSIZE", c),
-      orig_bytes(), resp_bytes(), orig_pkts(), resp_pkts()
+      orig_bytes(), resp_bytes(), orig_pkts(), resp_pkts(),
+      orig_bytes_thresh(), resp_bytes_thresh(), orig_pkts_thresh(), resp_pkts_thresh()
 	{
 	}
 

src/analyzer/protocol/dhcp/dhcp-analyzer.pac

@@ -237,7 +237,7 @@ flow DHCP_Flow(is_orig: bool) {
 
 		Unref(dhcp_msg_val_);
 
-		const char* mac_str = fmt_mac(${msg.chaddr}.data(), ${msg.chaddr}.length());
+		std::string mac_str = fmt_mac(${msg.chaddr}.data(), ${msg.chaddr}.length());
 
 		RecordVal* r = new RecordVal(dhcp_msg);
 		r->Assign(0, new Val(${msg.op}, TYPE_COUNT));
@@ -247,8 +247,6 @@ flow DHCP_Flow(is_orig: bool) {
 		r->Assign(4, new AddrVal(${msg.ciaddr}));
 		r->Assign(5, new AddrVal(${msg.yiaddr}));
 
-		delete [] mac_str;
-
 		dhcp_msg_val_ = r;
 
 		switch ( ${msg.op} )

src/analyzer/protocol/http/HTTP.cc

@@ -1813,12 +1813,12 @@ void HTTP_Analyzer::SkipEntityData(int is_orig)
 	}
 
 int analyzer::http::is_reserved_URI_char(unsigned char ch)
-	{ // see RFC 2396 (definition of URI)
+	{ // see RFC 3986 (definition of URI)
-	return strchr(";/?:@&=+$,", ch) != 0;
+	return strchr(":/?#[]@!$&'()*+,;=", ch) != 0;
 	}
 
 int analyzer::http::is_unreserved_URI_char(unsigned char ch)
-	{ // see RFC 2396 (definition of URI)
+	{ // see RFC 3986 (definition of URI)
 	return isalnum(ch) || strchr("-_.!~*\'()", ch) != 0;
 	}
 
@@ -1835,19 +1835,6 @@ BroString* analyzer::http::unescape_URI(const u_char* line, const u_char* line_e
 	byte_vec decoded_URI = new u_char[line_end - line + 1];
 	byte_vec URI_p = decoded_URI;
 
-	// An 'unescaped_special_char' here means a character that *should*
-	// be escaped, but isn't in the URI.  A control characters that
-	// appears directly in the URI would be an example.  The RFC implies
-	// that if we do not unescape the URI that we see in the trace, every
-	// character should be a printable one -- either reserved or unreserved
-	// (or '%').
-	//
-	// Counting the number of unescaped characters and generating a weird
-	// event on URI's with unescaped characters (which are rare) will
-	// let us locate strange-looking URI's in the trace -- those URI's
-	// are often interesting.
-	int unescaped_special_char = 0;
-
 	while ( line < line_end )
 		{
 		if ( *line == '%' )
@@ -1881,6 +1868,36 @@ BroString* analyzer::http::unescape_URI(const u_char* line, const u_char* line_e
 				++line; // place line at the last hex digit
 				}
 
+			else if ( line_end - line >= 5 &&
+			          line[0] == 'u' &&
+			          isxdigit(line[1]) &&
+			          isxdigit(line[2]) &&
+			          isxdigit(line[3]) &&
+			          isxdigit(line[4]) )
+				{
+				// Decode escaping like this: %u00AE
+				// The W3C rejected escaping this way, and
+				// there is no RFC that specifies it.
+				// Appparently there is some software doing
+				// this sort of 4 byte unicode encoding anyway.
+				// Likely causing an increase in it's use is
+				// the third edition of the ECMAScript spec
+				// having functions for encoding and decoding
+				// data in this format.
+
+				// If the first byte is null, let's eat it.
+				// It could just be ASCII encoded into this
+				// unicode escaping structure.
+				if ( ! (line[1] == '0' && line[2] == '0' ) )
+					*URI_p++ = (decode_hex(line[1]) << 4) +
+					            decode_hex(line[2]);
+
+				*URI_p++ = (decode_hex(line[3]) << 4) +
+					    decode_hex(line[4]);
+
+				line += 4;
+				}
+
 			else
 				{
 				if ( analyzer )
@@ -1891,23 +1908,12 @@ BroString* analyzer::http::unescape_URI(const u_char* line, const u_char* line_e
 			}
 
 		else
-			{
-			if ( ! is_reserved_URI_char(*line) &&
-			     ! is_unreserved_URI_char(*line) )
-				// Count these up as a way to compress
-				// the corresponding Weird event to a
-				// single instance.
-				++unescaped_special_char;
 			*URI_p++ = *line;
-			}
 
 		++line;
 		}
 
 	URI_p[0] = 0;
 
-	if ( unescaped_special_char && analyzer )
-		analyzer->Weird("unescaped_special_URI_char");
-
 	return new BroString(1, decoded_URI, URI_p - decoded_URI);
 	}

src/analyzer/protocol/modbus/modbus-analyzer.pac

@@ -31,9 +31,8 @@
 		RecordVal* modbus_header = new RecordVal(BifType::Record::ModbusHeaders);
 		modbus_header->Assign(0, new Val(header->tid(), TYPE_COUNT));
 		modbus_header->Assign(1, new Val(header->pid(), TYPE_COUNT));
-		modbus_header->Assign(2, new Val(header->len(), TYPE_COUNT));
+		modbus_header->Assign(2, new Val(header->uid(), TYPE_COUNT));
-		modbus_header->Assign(3, new Val(header->uid(), TYPE_COUNT));
+		modbus_header->Assign(3, new Val(header->fc(), TYPE_COUNT));
-		modbus_header->Assign(4, new Val(header->fc(), TYPE_COUNT));
 		return modbus_header;
 		}
 

src/analyzer/protocol/rfb/rfb-analyzer.pac

@@ -150,8 +150,12 @@ refine connection RFB_Conn += {
 			}
 
 		if ( msg->sectype() == 2 )
-			{ //VNC
+			{ // VNC
-			state = AWAITING_SERVER_CHALLENGE;
+			if ( ${msg.possible_challenge}.length() == 16 )
+				// Challenge was already sent with this message
+				state = AWAITING_CLIENT_RESPONSE;
+			else
+				state = AWAITING_SERVER_CHALLENGE;
 			}
 		return true;
 		%}

src/analyzer/protocol/rfb/rfb-protocol.pac

@@ -28,6 +28,7 @@ type RFBProtocolVersion (client: bool) = record {
 
 type RFBSecurityTypes = record {
 	sectype: uint32;
+	possible_challenge: bytestring &restofdata;
 } &let {
 	proc: bool = $context.connection.handle_security_types(this);
 	proc2: bool = $context.flow.proc_security_types(this);

src/bif_type.def

@@ -8,7 +8,7 @@ DEFINE_BIF_TYPE(TYPE_CONNECTION, "connection", "connection", "Connection*", "%s-
 DEFINE_BIF_TYPE(TYPE_COUNT, 	"count", "count", "bro_uint_t", 	"%s->AsCount()", 	"new Val(%s, TYPE_COUNT)")
 DEFINE_BIF_TYPE(TYPE_DOUBLE, 	"double", "double", "double", 		"%s->AsDouble()", 	"new Val(%s, TYPE_DOUBLE)")
 DEFINE_BIF_TYPE(TYPE_FILE, 	"file", "file", "BroFile*", 		"%s->AsFile()", 	"new Val(%s)")
-DEFINE_BIF_TYPE(TYPE_INT, 	"int", "int", "bro_int_t", 		"%s->AsInt()", 		"new Val(%s, TYPE_BOOL)")
+DEFINE_BIF_TYPE(TYPE_INT, 	"int", "int", "bro_int_t", 		"%s->AsInt()", 		"new Val(%s, TYPE_INT)")
 DEFINE_BIF_TYPE(TYPE_INTERVAL, 	"interval", "interval", "double", 	"%s->AsInterval()", 	"new IntervalVal(%s, Seconds)")
 DEFINE_BIF_TYPE(TYPE_PACKET, 	"packet", "packet", "TCP_TracePacket*", "%s->AsRecordVal()->GetOrigin()", "%s->PacketVal()")
 DEFINE_BIF_TYPE(TYPE_PATTERN, 	"pattern", "pattern", "RE_Matcher*", 	"%s->AsPattern()",	"new PatternVal(%s)")

src/bro.bif

@@ -3787,6 +3787,35 @@ function lookup_asn%(a: addr%) : count
 	return new Val(0, TYPE_COUNT);
 	%}
 
+## Calculates distance between two geographic locations using the haversine
+## formula.  Latitudes and longitudes must be given in degrees, where southern
+## hemispere latitudes are negative and western hemisphere longitudes are
+## negative.
+##
+## lat1: Latitude (in degrees) of location 1.
+##
+## long1: Longitude (in degrees) of location 1.
+##
+## lat2: Latitude (in degrees) of location 2.
+##
+## long2: Longitude (in degrees) of location 2.
+##
+## Returns: Distance in miles.
+##
+## .. bro:see:: haversine_distance_ip
+function haversine_distance%(lat1: double, long1: double, lat2: double, long2: double%): double
+	%{
+	const double PI = 3.14159;
+	const double RADIUS = 3958.8;  // Earth's radius in miles.
+
+	double s1 = sin((lat2 - lat1) * PI/360);
+	double s2 = sin((long2 - long1) * PI/360);
+	double a = s1 * s1 + cos(lat1 * PI/180) * cos(lat2 * PI/180) * s2 * s2;
+	double distance = 2 * RADIUS * asin(sqrt(a));
+
+	return new Val(distance, TYPE_DOUBLE);
+	%}
+
 ## Converts UNIX file permissions given by a mode to an ASCII string.
 ##
 ## mode: The permissions (an octal number like 0644 converted to decimal).

src/file_analysis/analyzer/entropy/Entropy.cc

@@ -14,6 +14,7 @@ Entropy::Entropy(RecordVal* args, File* file)
 	{
 	//entropy->Init();
 	entropy = new EntropyVal;
+	fed = false;
 	}
 
 Entropy::~Entropy()

src/input/Manager.cc

@@ -1204,7 +1204,7 @@ int Manager::SendEntryTable(Stream* i, const Value* const *vals)
 	ih->idxkey = new HashKey(k->Key(), k->Size(), k->Hash());
 	ih->valhash = valhash;
 
-	if ( stream->event && updated )
+	if ( oldval && stream->event && updated )
 		Ref(oldval); // otherwise it is no longer accessible after the assignment
 
 	stream->tab->Assign(idxval, k, valval);
@@ -1917,7 +1917,8 @@ RecordVal* Manager::ValueToRecordVal(const Stream* stream, const Value* const *v
 			(*position)++;
 			}
 
-		rec->Assign(i, fieldVal);
+		if ( fieldVal )
+			rec->Assign(i, fieldVal);
 		}
 
 	return rec;

src/input/readers/ascii/Ascii.cc

@@ -1,6 +1,5 @@
 // See the file "COPYING" in the main distribution directory for copyright.
 
-#include <fstream>
 #include <sstream>
 
 #include <sys/types.h>
@@ -49,25 +48,15 @@ FieldMapping FieldMapping::subType()
 
 Ascii::Ascii(ReaderFrontend *frontend) : ReaderBackend(frontend)
 	{
-	file = 0;
 	mtime = 0;
-	formatter = 0;
 	}
 
 Ascii::~Ascii()
 	{
-	DoClose();
-	delete formatter;
 	}
 
 void Ascii::DoClose()
 	{
-	if ( file != 0 )
-		{
-		file->close();
-		delete(file);
-		file = 0;
-		}
 	}
 
 bool Ascii::DoInit(const ReaderInfo& info, int num_fields, const Field* const* fields)
@@ -107,23 +96,19 @@ bool Ascii::DoInit(const ReaderInfo& info, int num_fields, const Field* const* f
 		Error("set_separator length has to be 1. Separator will be truncated.");
 
 	formatter::Ascii::SeparatorInfo sep_info(separator, set_separator, unset_field, empty_field);
-	formatter = new formatter::Ascii(this, sep_info);
+	formatter = unique_ptr<threading::formatter::Formatter>(new formatter::Ascii(this, sep_info));
 
-	file = new ifstream(info.source);
+	file.open(info.source);
-	if ( ! file->is_open() )
+	if ( ! file.is_open() )
 		{
 		Error(Fmt("Init: cannot open %s", info.source));
-		delete(file);
-		file = 0;
 		return false;
 		}
 
 	if ( ReadHeader(false) == false )
 		{
 		Error(Fmt("Init: cannot open %s; headers are incorrect", info.source));
-		file->close();
+		file.close();
-		delete(file);
-		file = 0;
 		return false;
 		}
 
@@ -215,8 +200,14 @@ bool Ascii::ReadHeader(bool useCached)
 
 bool Ascii::GetLine(string& str)
 	{
-	while ( getline(*file, str) )
+	while ( getline(file, str) )
 		{
+		if ( ! str.size() )
+			continue;
+
+		if ( str.back() == '\r' ) // deal with \r\n by removing \r
+			str.pop_back();
+
 		if ( str[0] != '#' )
 			return true;
 
@@ -258,24 +249,22 @@ bool Ascii::DoUpdate()
 			{
 			// dirty, fix me. (well, apparently after trying seeking, etc
 			// - this is not that bad)
-			if ( file && file->is_open() )
+			if ( file.is_open() )
 				{
 				if ( Info().mode == MODE_STREAM )
 					{
-					file->clear(); // remove end of file evil bits
+					file.clear(); // remove end of file evil bits
 					if ( !ReadHeader(true) )
 						return false; // header reading failed
 
 					break;
 					}
 
-				file->close();
+				file.close();
-				delete file;
-				file = 0;
 				}
 
-			file = new ifstream(Info().source);
+			file.open(Info().source);
-			if ( ! file->is_open() )
+			if ( ! file.is_open() )
 				{
 				Error(Fmt("cannot open %s", Info().source));
 				return false;
@@ -296,7 +285,7 @@ bool Ascii::DoUpdate()
 
 	string line;
 
-	file->sync();
+	file.sync();
 
 	while ( GetLine(line) )
 		{

src/input/readers/ascii/Ascii.h

@@ -5,6 +5,8 @@
 
 #include <iostream>
 #include <vector>
+#include <fstream>
+#include <memory>
 
 #include "input/ReaderBackend.h"
 #include "threading/formatters/Ascii.h"
@@ -33,23 +35,28 @@ struct FieldMapping {
  */
 class Ascii : public ReaderBackend {
 public:
-	Ascii(ReaderFrontend* frontend);
+	explicit Ascii(ReaderFrontend* frontend);
 	~Ascii();
 
+	// prohibit copying and moving
+	Ascii(const Ascii&) = delete;
+	Ascii(Ascii&&) = delete;
+	Ascii& operator=(const Ascii&) = delete;
+	Ascii& operator=(Ascii&&) = delete;
+
 	static ReaderBackend* Instantiate(ReaderFrontend* frontend) { return new Ascii(frontend); }
 
 protected:
-	virtual bool DoInit(const ReaderInfo& info, int arg_num_fields, const threading::Field* const* fields);
+	bool DoInit(const ReaderInfo& info, int arg_num_fields, const threading::Field* const* fields) override;
-	virtual void DoClose();
+	void DoClose() override;
-	virtual bool DoUpdate();
+	bool DoUpdate() override;
-	virtual bool DoHeartbeat(double network_time, double current_time);
+	bool DoHeartbeat(double network_time, double current_time) override;
 
 private:
-
 	bool ReadHeader(bool useCached);
 	bool GetLine(string& str);
 
-	ifstream* file;
+	ifstream file;
 	time_t mtime;
 
 	// map columns in the file to columns to send back to the manager
@@ -64,7 +71,7 @@ private:
 	string empty_field;
 	string unset_field;
 
-	threading::formatter::Formatter* formatter;
+	std::unique_ptr<threading::formatter::Formatter> formatter;
 };
 
 

src/input/readers/raw/Raw.cc

@@ -26,10 +26,8 @@ using threading::Field;
 
 const int Raw::block_size = 4096; // how big do we expect our chunks of data to be.
 
-Raw::Raw(ReaderFrontend *frontend) : ReaderBackend(frontend)
+Raw::Raw(ReaderFrontend *frontend) : ReaderBackend(frontend), file(nullptr, fclose), stderrfile(nullptr, fclose)
 	{
-	file = 0;
-	stderrfile = 0;
 	execute = false;
 	firstrun = true;
 	mtime = 0;
@@ -40,8 +38,6 @@ Raw::Raw(ReaderFrontend *frontend) : ReaderBackend(frontend)
 
 	sep_length = BifConst::InputRaw::record_separator->Len();
 
-	buf = 0;
-	outbuf = 0;
 	bufpos = 0;
 
 	stdin_fileno = fileno(stdin);
@@ -61,13 +57,9 @@ Raw::~Raw()
 
 void Raw::DoClose()
 	{
-	if ( file != 0 )
+	if ( file )
 		CloseInput();
 
-	// Just throw away output that has not been flushed.
-	delete [] buf;
-	buf = 0;
-
 	if ( execute && childpid > 0 && kill(childpid, 0) == 0 )
 		{
 		// Kill child process group.
@@ -255,7 +247,7 @@ bool Raw::Execute()
 		else
 			ClosePipeEnd(stderr_in);
 
-		file = fdopen(pipes[stdout_in], "r");
+		file = std::unique_ptr<FILE, int(*)(FILE*)>(fdopen(pipes[stdout_in], "r"), fclose);
 
 		if ( ! file )
 			{
@@ -267,7 +259,7 @@ bool Raw::Execute()
 
 		if ( use_stderr )
 			{
-			stderrfile = fdopen(pipes[stderr_in], "r");
+			stderrfile = std::unique_ptr<FILE, int(*)(FILE*)>(fdopen(pipes[stderr_in], "r"), fclose);
 
 			if ( ! stderrfile )
 				{
@@ -289,14 +281,14 @@ bool Raw::OpenInput()
 
 	else
 		{
-		file = fopen(fname.c_str(), "r");
+		file = std::unique_ptr<FILE, int(*)(FILE*)>(fopen(fname.c_str(), "r"), fclose);
 		if ( ! file )
 			{
 			Error(Fmt("Init: cannot open %s", fname.c_str()));
 			return false;
 			}
 
-		if ( ! SetFDFlags(fileno(file), F_SETFD, FD_CLOEXEC) )
+		if ( ! SetFDFlags(fileno(file.get()), F_SETFD, FD_CLOEXEC) )
 			Warning(Fmt("Init: cannot set close-on-exec for %s", fname.c_str()));
 		}
 
@@ -305,7 +297,7 @@ bool Raw::OpenInput()
 			int whence = (offset >= 0) ? SEEK_SET : SEEK_END;
 			int64_t pos = (offset >= 0) ? offset : offset + 1; // we want -1 to be the end of the file
 
-			if ( fseek(file, pos, whence) < 0 )
+			if ( fseek(file.get(), pos, whence) < 0 )
 				{
 				char buf[256];
 				strerror_r(errno, buf, sizeof(buf));
@@ -318,7 +310,7 @@ bool Raw::OpenInput()
 
 bool Raw::CloseInput()
 	{
-	if ( file == 0 )
+	if ( ! file )
 		{
 		InternalWarning(Fmt("Trying to close closed file for stream %s",
 		                    fname.c_str()));
@@ -328,10 +320,10 @@ bool Raw::CloseInput()
 	Debug(DBG_INPUT, "Raw reader starting close");
 #endif
 
-	fclose(file);
+	file.reset(nullptr);
 
 	if ( use_stderr )
-		fclose(stderrfile);
+		stderrfile.reset(nullptr);
 
 	if ( execute )
 		{
@@ -339,9 +331,6 @@ bool Raw::CloseInput()
 			ClosePipeEnd(i);
 		}
 
-	file = 0;
-	stderrfile = 0;
-
 #ifdef DEBUG
 	Debug(DBG_INPUT, "Raw reader finished close");
 #endif
@@ -455,14 +444,14 @@ int64_t Raw::GetLine(FILE* arg_file)
 	int pos = 0; // strstr_n only works on ints - so no use to use something different here
 	int offset = 0;
 
-	if ( buf == 0 )
+	if ( ! buf )
-		buf = new char[block_size];
+		buf = std::unique_ptr<char[]>(new char[block_size]);
 
 	int repeats = 1;
 
 	for ( ;; )
 		{
-		size_t readbytes = fread(buf+bufpos+offset, 1, block_size-bufpos, arg_file);
+		size_t readbytes = fread(buf.get()+bufpos+offset, 1, block_size-bufpos, arg_file);
 		pos += bufpos + readbytes;
 		//printf("Pos: %d\n", pos);
 		bufpos = offset = 0; // read full block size in next read...
@@ -473,7 +462,7 @@ int64_t Raw::GetLine(FILE* arg_file)
 		// researching everything each time is a bit... cpu-intensive. But otherwhise we have
 		// to deal with situations where the separator is multi-character and split over multiple
 		// reads...
-		int found = strstr_n(pos, (unsigned char*) buf, separator.size(), (unsigned char*) separator.c_str());
+		int found = strstr_n(pos, (unsigned char*) buf.get(), separator.size(), (unsigned char*) separator.c_str());
 
 		if ( found == -1 )
 			{
@@ -485,30 +474,27 @@ int64_t Raw::GetLine(FILE* arg_file)
 					return -1; // signal EOF - and that we had no more data.
 				else
 					{
-					outbuf = buf;
+					outbuf = std::move(buf); // buf is null after this
-					buf = 0;
 					return pos;
 					}
 				}
 
 			repeats++;
 			// bah, we cannot use realloc because we would have to change the delete in the manager to a free.
-			char * newbuf = new char[block_size*repeats];
+			std::unique_ptr<char[]> newbuf = std::unique_ptr<char[]>(new char[block_size*repeats]);
-			memcpy(newbuf, buf, block_size*(repeats-1));
+			memcpy(newbuf.get(), buf.get(), block_size*(repeats-1));
-			delete [] buf;
+			buf = std::move(newbuf);
-			buf = newbuf;
 			offset = block_size*(repeats-1);
 			}
 		else
 			{
-			outbuf = buf;
+			outbuf = std::move(buf);
-			buf = 0;
 
 			if ( found < pos )
 				{
 				// we have leftovers. copy them into the buffer for the next line
-				buf = new char[block_size];
+				buf = std::unique_ptr<char[]>(new char[block_size]);
-				memcpy(buf, outbuf + found + sep_length, pos - found - sep_length);
+				memcpy(buf.get(), outbuf.get() + found + sep_length, pos - found - sep_length);
 				bufpos = pos - found - sep_length;
 				}
 
@@ -586,9 +572,9 @@ bool Raw::DoUpdate()
 
 		case MODE_MANUAL:
 		case MODE_STREAM:
-			if ( Info().mode == MODE_STREAM && file != 0 )
+			if ( Info().mode == MODE_STREAM && file )
 				{
-				clearerr(file);  // remove end of file evil bits
+				clearerr(file.get());  // remove end of file evil bits
 				break;
 				}
 
@@ -610,7 +596,7 @@ bool Raw::DoUpdate()
 		if ( stdin_towrite > 0 )
 			WriteToStdin();
 
-		int64_t length = GetLine(file);
+		int64_t length = GetLine(file.get());
 		//printf("Read %lld bytes\n", length);
 
 		if ( length == -3 )
@@ -624,7 +610,7 @@ bool Raw::DoUpdate()
 
 		// filter has exactly one text field. convert to it.
 		Value* val = new Value(TYPE_STRING, true);
-		val->val.string_val.data = outbuf;
+		val->val.string_val.data = outbuf.release();
 		val->val.string_val.length = length;
 		fields[0] = val;
 
@@ -636,15 +622,13 @@ bool Raw::DoUpdate()
 			}
 
 		Put(fields);
-
-		outbuf = 0;
 		}
 
 	if ( use_stderr )
 		{
 		for ( ;; )
 			{
-			int64_t length = GetLine(stderrfile);
+			int64_t length = GetLine(stderrfile.get());
 			//printf("Read stderr %lld bytes\n", length);
 			if ( length == -3 )
 				return false;
@@ -654,7 +638,7 @@ bool Raw::DoUpdate()
 
 			Value** fields = new Value*[2];
 			Value* val = new Value(TYPE_STRING, true);
-			val->val.string_val.data = outbuf;
+			val->val.string_val.data = outbuf.release();
 			val->val.string_val.length = length;
 			fields[0] = val;
 			Value* bval = new Value(TYPE_BOOL, true);
@@ -662,8 +646,6 @@ bool Raw::DoUpdate()
 			fields[1] = bval;
 
 			Put(fields);
-
-			outbuf = 0;
 			}
 		}
 

src/input/readers/raw/Raw.h

@@ -5,6 +5,7 @@
 
 #include <vector>
 #include <pthread.h>
+#include <memory>
 
 #include "input/ReaderBackend.h"
 
@@ -16,16 +17,22 @@ namespace input { namespace reader {
  */
 class Raw : public ReaderBackend {
 public:
-	Raw(ReaderFrontend* frontend);
+	explicit Raw(ReaderFrontend* frontend);
 	~Raw();
 
+	// prohibit copying and moving
+	Raw(const Raw&) = delete;
+	Raw(Raw&&) = delete;
+	Raw& operator=(const Raw&) = delete;
+	Raw& operator=(Raw&&) = delete;
+
 	static ReaderBackend* Instantiate(ReaderFrontend* frontend) { return new Raw(frontend); }
 
 protected:
-	virtual bool DoInit(const ReaderInfo& info, int arg_num_fields, const threading::Field* const* fields);
+	bool DoInit(const ReaderInfo& info, int arg_num_fields, const threading::Field* const* fields) override;
-	virtual void DoClose();
+	void DoClose() override;
-	virtual bool DoUpdate();
+	bool DoUpdate() override;
-	virtual bool DoHeartbeat(double network_time, double current_time);
+	bool DoHeartbeat(double network_time, double current_time) override;
 
 private:
 	void ClosePipeEnd(int i);
@@ -40,8 +47,8 @@ private:
 	void WriteToStdin();
 
 	string fname; // Source with a potential "|" removed.
-	FILE* file;
+	std::unique_ptr<FILE, int(*)(FILE*)> file;
-	FILE* stderrfile;
+	std::unique_ptr<FILE, int(*)(FILE*)> stderrfile;
 	bool execute;
 	bool firstrun;
 	time_t mtime;
@@ -51,8 +58,8 @@ private:
 	unsigned int sep_length; // length of the separator
 
 	int bufpos;
-	char* buf;
+	std::unique_ptr<char[]> buf;
-	char* outbuf;
+	std::unique_ptr<char[]> outbuf;
 
 	int stdin_fileno;
 	int stdout_fileno;

src/iosource/Packet.cc

@@ -44,6 +44,8 @@ void Packet::Init(int arg_link_type, struct timeval *arg_ts, uint32 arg_caplen,
 	eth_type = 0;
 	vlan = 0;
 	inner_vlan = 0;
+	l2_src = 0;
+	l2_dst = 0;
 
 	l2_valid = false;
 
@@ -136,8 +138,12 @@ void Packet::ProcessLayer2()
 		{
 		// Get protocol being carried from the ethernet frame.
 		int protocol = (pdata[12] << 8) + pdata[13];
-		pdata += GetLinkHeaderSize(link_type);
+
 		eth_type = protocol;
+		l2_dst = pdata;
+		l2_src = pdata + 6;
+
+		pdata += GetLinkHeaderSize(link_type);
 
 		switch ( protocol )
 			{
@@ -261,33 +267,82 @@ void Packet::ProcessLayer2()
 			Weird("truncated_radiotap_header");
 			return;
 			}
+
 		// Skip over the RadioTap header
 		int rtheader_len = (pdata[3] << 8) + pdata[2];
+
 		if ( pdata + rtheader_len >= end_of_data )
 			{
 			Weird("truncated_radiotap_header");
 			return;
 			}
+
 		pdata += rtheader_len;
 
-		int type_80211 = pdata[0];
+		u_char len_80211 = 24; // minimal length of data frames
-		int len_80211 = 0;
-		if ( (type_80211 >> 4) & 0x04 )
-			{
-			//identified a null frame (we ignore for now).  no weird.
-			return;
-			}
-		// Look for the QoS indicator bit.
-		if ( (type_80211 >> 4) & 0x08 )
-			len_80211 = 26;
-		else
-			len_80211 = 24;
 
 		if ( pdata + len_80211 >= end_of_data )
 			{
 			Weird("truncated_radiotap_header");
 			return;
 			}
+
+		u_char fc_80211 = pdata[0]; // Frame Control field
+
+		// Skip non-data frame types (management & control).
+		if ( ! ((fc_80211 >> 2) & 0x02) )
+			return;
+
+		// Skip subtypes without data.
+		if ( (fc_80211 >> 4) & 0x04 )
+			return;
+
+		// 'To DS' and 'From DS' flags set indicate use of the 4th
+		// address field.
+		if ( (pdata[1] & 0x03) == 0x03 )
+			len_80211 += l2_addr_len;
+
+		// Look for the QoS indicator bit.
+		if ( (fc_80211 >> 4) & 0x08 )
+			{
+			// Skip in case of A-MSDU subframes indicated by QoS
+			// control field.
+			if ( pdata[len_80211] & 0x80)
+				return;
+
+			len_80211 += 2;
+			}
+
+		if ( pdata + len_80211 >= end_of_data )
+			{
+			Weird("truncated_radiotap_header");
+			return;
+			}
+
+		// Determine link-layer addresses based
+		// on 'To DS' and 'From DS' flags
+		switch ( pdata[1] & 0x03 ) {
+			case 0x00:
+				l2_src = pdata + 10;
+				l2_dst = pdata + 4;
+				break;
+
+			case 0x01:
+				l2_src = pdata + 10;
+				l2_dst = pdata + 16;
+				break;
+
+			case 0x02:
+				l2_src = pdata + 16;
+				l2_dst = pdata + 4;
+				break;
+
+			case 0x03:
+				l2_src = pdata + 24;
+				l2_dst = pdata + 16;
+				break;
+		}
+
 		// skip 802.11 data header
 		pdata += len_80211;
 

src/iosource/Packet.h

@@ -50,7 +50,8 @@ public:
 	 */
 	Packet(int link_type, struct timeval *ts, uint32 caplen,
 	       uint32 len, const u_char *data, int copy = false,
-	       std::string tag = std::string("")) : data(0)
+	       std::string tag = std::string(""))
+	           : data(0), l2_src(0), l2_dst(0)
 	       {
 	       Init(link_type, ts, caplen, len, data, copy, tag);
 	       }
@@ -58,7 +59,7 @@ public:
 	/**
 	 * Default constructor. For internal use only.
 	 */
-	Packet() : data(0)
+	Packet() : data(0), l2_src(0), l2_dst(0)
 		{
 		struct timeval ts = {0, 0};
 		Init(0, &ts, 0, 0, 0);
@@ -146,6 +147,11 @@ public:
 	 */
 	static Packet* Unserialize(UnserialInfo* info);
 
+	/**
+	 * Maximal length of a layer 2 address.
+	 */
+	static const int l2_addr_len = 6;
+
 	// These are passed in through the constructor.
 	std::string tag;		/// Used in serialization
 	double time;			/// Timestamp reconstituted as float
@@ -167,19 +173,30 @@ public:
 	 * Layer 3 protocol identified (if any). Valid iff Layer2Valid()
 	 * returns true.
 	 */
-	Layer3Proto l3_proto;		///
+	Layer3Proto l3_proto;
 
 	/**
 	 * If layer 2 is Ethernet, innermost ethertype field. Valid iff
 	 * Layer2Valid() returns true.
 	 */
-	uint32 eth_type;		///
+	uint32 eth_type;
+
+	/**
+	 * Layer 2 source address. Valid iff Layer2Valid() returns true.
+	 */
+	const u_char* l2_src;
+
+	/**
+	 * Layer 2 destination address. Valid iff Layer2Valid() returns
+	 * true.
+	 */
+	const u_char* l2_dst;
 
 	/**
 	 * (Outermost) VLAN tag if any, else 0. Valid iff Layer2Valid()
 	 * returns true.
 	 */
-	uint32 vlan;			///
+	uint32 vlan;
 
 	/**
 	 * (Innermost) VLAN tag if any, else 0. Valid iff Layer2Valid()

src/iosource/PktSrc.cc

@@ -31,6 +31,7 @@ PktSrc::PktSrc()
 
 	next_sync_point = 0;
 	first_timestamp = 0.0;
+	current_pseudo = 0.0;
 	first_wallclock = current_wallclock = 0;
 	}
 
@@ -289,6 +290,12 @@ bool PktSrc::ExtractNextPacketInternal()
 
 	if ( ExtractNextPacket(&current_packet) )
 		{
+		if ( current_packet.time < 0 )
+			{
+			Weird("negative_packet_timestamp", &current_packet);
+			return 0;
+			}
+
 		if ( ! first_timestamp )
 			first_timestamp = current_packet.time;
 

src/main.cc

@@ -754,7 +754,10 @@ int main(int argc, char** argv)
 
 #ifdef DEBUG
 	if ( debug_streams )
+		{
 		debug_logger.EnableStreams(debug_streams);
+		debug_logger.OpenDebugLog("debug");
+		}
 #endif
 
 	init_random_seed(seed, (seed_load_file && *seed_load_file ? seed_load_file : 0) , seed_save_file);

src/modp_numtoa.c

@@ -287,5 +287,136 @@ void modp_dtoa2(double value, char* str, int prec)
     strreverse(str, wstr-1);
 }
 
+// This is near identical to modp_dtoa2 above, excep that it never uses
+// exponential notation and requires a buffer length.
+void modp_dtoa3(double value, char* str, int n, int prec)
+{
+    /* Hacky test for NaN
+     * under -fast-math this won't work, but then you also won't
+     * have correct nan values anyways.  The alternative is
+     * to link with libmath (bad) or hack IEEE double bits (bad)
+     */
+    if (! (value == value)) {
+        str[0] = 'n'; str[1] = 'a'; str[2] = 'n'; str[3] = '\0';
+        return;
+    }
+
+    /* if input is larger than thres_max, revert to exponential */
+    const double thres_max = (double)(0x7FFFFFFF);
+
+    int count;
+    double diff = 0.0;
+    char* wstr = str;
+
+    if (prec < 0) {
+        prec = 0;
+    } else if (prec > 9) {
+        /* precision of >= 10 can lead to overflow errors */
+        prec = 9;
+    }
+
+
+    /* we'll work in positive values and deal with the
+       negative sign issue later */
+    int neg = 0;
+    if (value < 0) {
+        neg = 1;
+        value = -value;
+    }
+
+
+    int whole = (int) value;
+    double tmp = (value - whole) * _pow10[prec];
+    uint32_t frac = (uint32_t)(tmp);
+    diff = tmp - frac;
+
+    if (diff > 0.5) {
+        ++frac;
+        /* handle rollover, e.g.  case 0.99 with prec 1 is 1.0  */
+        if (frac >= _pow10[prec]) {
+            frac = 0;
+            ++whole;
+        }
+    } else if (diff == 0.5 && ((frac == 0) || (frac & 1))) {
+        /* if halfway, round up if odd, OR
+           if last digit is 0.  That last part is strange */
+        ++frac;
+    }
+
+    /* for very large numbers switch back to native sprintf for exponentials.
+       anyone want to write code to replace this? */
+    /*
+      normal printf behavior is to print EVERY whole number digit
+      which can be 100s of characters overflowing your buffers == bad
+    */
+    if (value > thres_max) {
+        /* ---- Modified part, compared to modp_dtoa3. */
+        int i = snprintf(str, n, "%.*f", prec, neg ? -value : value);
+
+        if ( i < 0 || i >= n ) {
+        // Error or truncated output.
+            snprintf(str, n, "NAN");
+            return;
+            }
+
+        /* Remove trailing zeros. */
+
+        char* p;
+        for ( p = str + i - 1; p >= str && *p == '0'; --p );
+
+        if ( p >= str && *p == '.' )
+            --p;
+
+        *++p = '\0';
+        return;
+
+        /* ---- End of modified part.. */
+    }
+
+    if (prec == 0) {
+        diff = value - whole;
+        if (diff > 0.5) {
+            /* greater than 0.5, round up, e.g. 1.6 -> 2 */
+            ++whole;
+        } else if (diff == 0.5 && (whole & 1)) {
+            /* exactly 0.5 and ODD, then round up */
+            /* 1.5 -> 2, but 2.5 -> 2 */
+            ++whole;
+        }
+
+        //vvvvvvvvvvvvvvvvvvv  Diff from modp_dto2
+    } else if (frac) {
+        count = prec;
+        // now do fractional part, as an unsigned number
+        // we know it is not 0 but we can have leading zeros, these
+        // should be removed
+        while (!(frac % 10)) {
+            --count;
+            frac /= 10;
+        }
+        //^^^^^^^^^^^^^^^^^^^  Diff from modp_dto2
+
+        // now do fractional part, as an unsigned number
+        do {
+            --count;
+            *wstr++ = (char)(48 + (frac % 10));
+        } while (frac /= 10);
+        // add extra 0s
+        while (count-- > 0) *wstr++ = '0';
+        // add decimal
+        *wstr++ = '.';
+    }
+
+    // do whole part
+    // Take care of sign
+    // Conversion. Number is reversed.
+    do *wstr++ = (char)(48 + (whole % 10)); while (whole /= 10);
+    if (neg) {
+        *wstr++ = '-';
+    }
+    *wstr='\0';
+    strreverse(str, wstr-1);
+}
+
 
 

src/modp_numtoa.h

@@ -97,6 +97,15 @@ void modp_dtoa(double value, char* buf, int precision);
  */
 void modp_dtoa2(double value, char* buf, int precision);
 
+/** \brief convert a floating point number to char buffer with a
+ *         variable-precision format, no trailing zeros, and no
+ *   	scientific notation.
+ *
+ *  Other than avoiding scientific notation, this is the same as mop_dtoa2. It does however
+ *  require the max buffer length. The buffer will always be null-terminated.
+ */
+void modp_dtoa3(double value, char* buf, int n, int precision);
+
 END_C
 
 #endif

src/net_util.cc

@@ -148,21 +148,21 @@ const char* fmt_conn_id(const uint32* src_addr, uint32 src_port,
 	return fmt_conn_id(src, src_port, dst, dst_port);
 	}
 
-char* fmt_mac(const unsigned char* m, int len)
+std::string fmt_mac(const unsigned char* m, int len)
 	{
-	char* buf = new char[25];
+	static char buf[25];
 
-	if ( len < 8 )
+	if ( len < 8 && len != 6 )
 		{
 		*buf = '\0';
 		return buf;
 		}
 
-	if ( m[6] == 0 && m[7] == 0 ) // EUI-48
+	if ( (len == 6) || (m[6] == 0 && m[7] == 0) ) // EUI-48
-		snprintf(buf, 19, "%02x:%02x:%02x:%02x:%02x:%02x",
+		snprintf(buf, sizeof(buf), "%02x:%02x:%02x:%02x:%02x:%02x",
 			 m[0], m[1], m[2], m[3], m[4], m[5]);
 	else
-		snprintf(buf, 25, "%02x:%02x:%02x:%02x:%02x:%02x:%02x:%02x",
+		snprintf(buf, sizeof(buf), "%02x:%02x:%02x:%02x:%02x:%02x:%02x:%02x",
 			 m[0], m[1], m[2], m[3], m[4], m[5], m[6], m[7]);
 
 	return buf;

src/net_util.h

@@ -166,7 +166,7 @@ extern const char* fmt_conn_id(const uint32* src_addr, uint32 src_port,
 *            least 8 for a valid address.
 * @return A string of the formatted MAC. Passes ownership to caller.
 */
-extern char* fmt_mac(const unsigned char* m, int len);
+extern std::string fmt_mac(const unsigned char* m, int len);
 
 // Read 4 bytes from data and return in network order.
 extern uint32 extract_uint32(const u_char* data);

src/parse.y

@@ -31,12 +31,12 @@
 
 %token TOK_NO_TEST
 
-%nonassoc TOK_HOOK
 %left ',' '|'
 %right '=' TOK_ADD_TO TOK_REMOVE_FROM
 %right '?' ':'
 %left TOK_OR
 %left TOK_AND
+%nonassoc TOK_HOOK
 %nonassoc '<' '>' TOK_LE TOK_GE TOK_EQ TOK_NE
 %left TOK_IN TOK_NOT_IN
 %left '+' '-'

src/patricia.c

@@ -646,6 +646,8 @@ patricia_search_all (patricia_tree_t *patricia, prefix_t *prefix, patricia_node_
 
 	// ok, now we have an upper bound of how much we can return. Let's just alloc that...
 	patricia_node_t **outlist = calloc(cnt, sizeof(patricia_node_t*));
+	if (outlist == NULL)
+		out_of_memory("patrica/patricia_search_all: unable to allocate memory");
 
 	while (--cnt >= 0) {
 		node = stack[cnt];

src/threading/formatters/Ascii.cc

@@ -91,7 +91,7 @@ bool Ascii::Describe(ODesc* desc, threading::Value* val, const string& name) con
 		// Rendering via Add() truncates trailing 0s after the
 		// decimal point. The difference with TIME/INTERVAL is mainly
 		// to keep the log format consistent.
-		desc->Add(val->val.double_val);
+		desc->Add(val->val.double_val, true);
 		break;
 
 	case TYPE_INTERVAL:

testing/btest/Baseline/bifs.haversine_distance/out

@@ -0,0 +1,7 @@
+5.8481e+03
+5.8481e+03
+1.9193e-02
+1.5136e-02
+9.2419e-01
+1.2437e+04
+1.2437e+04

testing/btest/Baseline/core.checksums/bad.out

@@ -3,101 +3,101 @@
 #empty_field	(empty)
 #unset_field	-
 #path	weird
-#open	2013-08-26-19-02-06
+#open	2016-06-15-20-38-04
 #fields	ts	uid	id.orig_h	id.orig_p	id.resp_h	id.resp_p	name	addl	notice	peer
 #types	time	string	addr	port	addr	port	string	string	bool	string
 1332784981.078396	-	-	-	-	-	bad_IP_checksum	-	F	bro
-#close	2013-08-26-19-02-06
+#close	2016-06-15-20-38-04
 #separator \x09
 #set_separator	,
 #empty_field	(empty)
 #unset_field	-
 #path	weird
-#open	2013-08-26-19-02-06
+#open	2016-06-15-20-38-06
 #fields	ts	uid	id.orig_h	id.orig_p	id.resp_h	id.resp_p	name	addl	notice	peer
 #types	time	string	addr	port	addr	port	string	string	bool	string
 1332784885.686428	CXWv6p3arKYeMETxOg	127.0.0.1	30000	127.0.0.1	80	bad_TCP_checksum	-	F	bro
-#close	2013-08-26-19-02-06
+#close	2016-06-15-20-38-06
 #separator \x09
 #set_separator	,
 #empty_field	(empty)
 #unset_field	-
 #path	weird
-#open	2013-08-26-19-02-07
+#open	2016-06-15-20-38-08
 #fields	ts	uid	id.orig_h	id.orig_p	id.resp_h	id.resp_p	name	addl	notice	peer
 #types	time	string	addr	port	addr	port	string	string	bool	string
 1332784933.501023	CXWv6p3arKYeMETxOg	127.0.0.1	30000	127.0.0.1	13000	bad_UDP_checksum	-	F	bro
-#close	2013-08-26-19-02-07
+#close	2016-06-15-20-38-08
 #separator \x09
 #set_separator	,
 #empty_field	(empty)
 #unset_field	-
 #path	weird
-#open	2013-08-26-19-02-07
+#open	2016-06-15-20-38-10
 #fields	ts	uid	id.orig_h	id.orig_p	id.resp_h	id.resp_p	name	addl	notice	peer
 #types	time	string	addr	port	addr	port	string	string	bool	string
 1334075363.536871	CXWv6p3arKYeMETxOg	192.168.1.100	8	192.168.1.101	0	bad_ICMP_checksum	-	F	bro
-#close	2013-08-26-19-02-07
+#close	2016-06-15-20-38-10
 #separator \x09
 #set_separator	,
 #empty_field	(empty)
 #unset_field	-
 #path	weird
-#open	2013-08-26-19-02-08
+#open	2016-06-15-20-38-11
 #fields	ts	uid	id.orig_h	id.orig_p	id.resp_h	id.resp_p	name	addl	notice	peer
 #types	time	string	addr	port	addr	port	string	string	bool	string
-1332785210.013051	-	-	-	-	-	routing0_hdr	-	F	bro
+1332785210.013051	-	2001:4f8:4:7:2e0:81ff:fe52:ffff	0	2001:78:1:32::2	0	routing0_hdr	-	F	bro
 1332785210.013051	CXWv6p3arKYeMETxOg	2001:4f8:4:7:2e0:81ff:fe52:ffff	30000	2001:78:1:32::2	80	bad_TCP_checksum	-	F	bro
-#close	2013-08-26-19-02-08
+#close	2016-06-15-20-38-12
 #separator \x09
 #set_separator	,
 #empty_field	(empty)
 #unset_field	-
 #path	weird
-#open	2013-08-26-19-02-09
+#open	2016-06-15-20-38-13
 #fields	ts	uid	id.orig_h	id.orig_p	id.resp_h	id.resp_p	name	addl	notice	peer
 #types	time	string	addr	port	addr	port	string	string	bool	string
-1332782580.798420	-	-	-	-	-	routing0_hdr	-	F	bro
+1332782580.798420	-	2001:4f8:4:7:2e0:81ff:fe52:ffff	0	2001:78:1:32::2	0	routing0_hdr	-	F	bro
 1332782580.798420	CXWv6p3arKYeMETxOg	2001:4f8:4:7:2e0:81ff:fe52:ffff	30000	2001:78:1:32::2	13000	bad_UDP_checksum	-	F	bro
-#close	2013-08-26-19-02-09
+#close	2016-06-15-20-38-13
 #separator \x09
 #set_separator	,
 #empty_field	(empty)
 #unset_field	-
 #path	weird
-#open	2013-08-26-19-02-09
+#open	2016-06-15-20-38-15
 #fields	ts	uid	id.orig_h	id.orig_p	id.resp_h	id.resp_p	name	addl	notice	peer
 #types	time	string	addr	port	addr	port	string	string	bool	string
-1334075111.800086	-	-	-	-	-	routing0_hdr	-	F	bro
+1334075111.800086	-	2001:4f8:4:7:2e0:81ff:fe52:ffff	0	2001:78:1:32::1	0	routing0_hdr	-	F	bro
 1334075111.800086	CXWv6p3arKYeMETxOg	2001:4f8:4:7:2e0:81ff:fe52:ffff	128	2001:78:1:32::1	129	bad_ICMP_checksum	-	F	bro
-#close	2013-08-26-19-02-09
+#close	2016-06-15-20-38-15
 #separator \x09
 #set_separator	,
 #empty_field	(empty)
 #unset_field	-
 #path	weird
-#open	2013-08-26-19-02-10
+#open	2016-06-15-20-38-16
 #fields	ts	uid	id.orig_h	id.orig_p	id.resp_h	id.resp_p	name	addl	notice	peer
 #types	time	string	addr	port	addr	port	string	string	bool	string
 1332785250.469132	CXWv6p3arKYeMETxOg	2001:4f8:4:7:2e0:81ff:fe52:ffff	30000	2001:4f8:4:7:2e0:81ff:fe52:9a6b	80	bad_TCP_checksum	-	F	bro
-#close	2013-08-26-19-02-10
+#close	2016-06-15-20-38-17
 #separator \x09
 #set_separator	,
 #empty_field	(empty)
 #unset_field	-
 #path	weird
-#open	2013-08-26-19-02-10
+#open	2016-06-15-20-38-18
 #fields	ts	uid	id.orig_h	id.orig_p	id.resp_h	id.resp_p	name	addl	notice	peer
 #types	time	string	addr	port	addr	port	string	string	bool	string
 1332781342.923813	CXWv6p3arKYeMETxOg	2001:4f8:4:7:2e0:81ff:fe52:ffff	30000	2001:4f8:4:7:2e0:81ff:fe52:9a6b	13000	bad_UDP_checksum	-	F	bro
-#close	2013-08-26-19-02-10
+#close	2016-06-15-20-38-18
 #separator \x09
 #set_separator	,
 #empty_field	(empty)
 #unset_field	-
 #path	weird
-#open	2013-08-26-19-02-11
+#open	2016-06-15-20-38-20
 #fields	ts	uid	id.orig_h	id.orig_p	id.resp_h	id.resp_p	name	addl	notice	peer
 #types	time	string	addr	port	addr	port	string	string	bool	string
 1334074939.467194	CXWv6p3arKYeMETxOg	2001:4f8:4:7:2e0:81ff:fe52:ffff	128	2001:4f8:4:7:2e0:81ff:fe52:9a6b	129	bad_ICMP_checksum	-	F	bro
-#close	2013-08-26-19-02-11
+#close	2016-06-15-20-38-20

testing/btest/Baseline/core.checksums/good.out

@@ -3,68 +3,68 @@
 #empty_field	(empty)
 #unset_field	-
 #path	weird
-#open	2013-08-26-19-34-56
+#open	2016-06-15-20-38-20
 #fields	ts	uid	id.orig_h	id.orig_p	id.resp_h	id.resp_p	name	addl	notice	peer
 #types	time	string	addr	port	addr	port	string	string	bool	string
 1334074939.467194	CXWv6p3arKYeMETxOg	2001:4f8:4:7:2e0:81ff:fe52:ffff	128	2001:4f8:4:7:2e0:81ff:fe52:9a6b	129	bad_ICMP_checksum	-	F	bro
-#close	2013-08-26-19-34-56
+#close	2016-06-15-20-38-20
 #separator \x09
 #set_separator	,
 #empty_field	(empty)
 #unset_field	-
 #path	weird
-#open	2013-08-26-19-34-57
+#open	2016-06-15-20-38-27
 #fields	ts	uid	id.orig_h	id.orig_p	id.resp_h	id.resp_p	name	addl	notice	peer
 #types	time	string	addr	port	addr	port	string	string	bool	string
-1332785125.596793	-	-	-	-	-	routing0_hdr	-	F	bro
+1332785125.596793	-	2001:4f8:4:7:2e0:81ff:fe52:ffff	0	2001:78:1:32::2	0	routing0_hdr	-	F	bro
-#close	2013-08-26-19-34-57
+#close	2016-06-15-20-38-27
 #separator \x09
 #set_separator	,
 #empty_field	(empty)
 #unset_field	-
 #path	weird
-#open	2013-08-26-19-34-57
+#open	2016-06-15-20-38-28
 #fields	ts	uid	id.orig_h	id.orig_p	id.resp_h	id.resp_p	name	addl	notice	peer
 #types	time	string	addr	port	addr	port	string	string	bool	string
-1332782508.592037	-	-	-	-	-	routing0_hdr	-	F	bro
+1332782508.592037	-	2001:4f8:4:7:2e0:81ff:fe52:ffff	0	2001:78:1:32::2	0	routing0_hdr	-	F	bro
-#close	2013-08-26-19-34-57
+#close	2016-06-15-20-38-29
 #separator \x09
 #set_separator	,
 #empty_field	(empty)
 #unset_field	-
 #path	weird
-#open	2013-08-26-19-34-57
+#open	2016-06-15-20-38-30
 #fields	ts	uid	id.orig_h	id.orig_p	id.resp_h	id.resp_p	name	addl	notice	peer
 #types	time	string	addr	port	addr	port	string	string	bool	string
-1334075027.053380	-	-	-	-	-	routing0_hdr	-	F	bro
+1334075027.053380	-	2001:4f8:4:7:2e0:81ff:fe52:ffff	0	2001:78:1:32::1	0	routing0_hdr	-	F	bro
-#close	2013-08-26-19-34-57
+#close	2016-06-15-20-38-30
 #separator \x09
 #set_separator	,
 #empty_field	(empty)
 #unset_field	-
 #path	weird
-#open	2013-08-26-19-34-57
+#open	2016-06-15-20-38-30
 #fields	ts	uid	id.orig_h	id.orig_p	id.resp_h	id.resp_p	name	addl	notice	peer
 #types	time	string	addr	port	addr	port	string	string	bool	string
-1334075027.053380	-	-	-	-	-	routing0_hdr	-	F	bro
+1334075027.053380	-	2001:4f8:4:7:2e0:81ff:fe52:ffff	0	2001:78:1:32::1	0	routing0_hdr	-	F	bro
-#close	2013-08-26-19-34-57
+#close	2016-06-15-20-38-30
 #separator \x09
 #set_separator	,
 #empty_field	(empty)
 #unset_field	-
 #path	weird
-#open	2013-08-26-19-34-57
+#open	2016-06-15-20-38-30
 #fields	ts	uid	id.orig_h	id.orig_p	id.resp_h	id.resp_p	name	addl	notice	peer
 #types	time	string	addr	port	addr	port	string	string	bool	string
-1334075027.053380	-	-	-	-	-	routing0_hdr	-	F	bro
+1334075027.053380	-	2001:4f8:4:7:2e0:81ff:fe52:ffff	0	2001:78:1:32::1	0	routing0_hdr	-	F	bro
-#close	2013-08-26-19-34-57
+#close	2016-06-15-20-38-30
 #separator \x09
 #set_separator	,
 #empty_field	(empty)
 #unset_field	-
 #path	weird
-#open	2013-08-26-19-34-57
+#open	2016-06-15-20-38-30
 #fields	ts	uid	id.orig_h	id.orig_p	id.resp_h	id.resp_p	name	addl	notice	peer
 #types	time	string	addr	port	addr	port	string	string	bool	string
-1334075027.053380	-	-	-	-	-	routing0_hdr	-	F	bro
+1334075027.053380	-	2001:4f8:4:7:2e0:81ff:fe52:ffff	0	2001:78:1:32::1	0	routing0_hdr	-	F	bro
-#close	2013-08-26-19-34-57
+#close	2016-06-15-20-38-30

testing/btest/Baseline/core.ether-addrs/output

@@ -0,0 +1,36 @@
+00:30:48:bd:3e:c4, 01:00:5e:00:00:fb
+00:17:f2:d7:cf:65, 33:33:00:00:00:fb
+00:17:f2:d7:cf:65, 01:00:5e:00:00:fb
+00:24:7e:e0:1d:b5, 00:13:7f:be:8c:ff
+00:24:7e:e0:1d:b5, 00:13:7f:be:8c:ff
+00:24:7e:e0:1d:b5, 00:13:7f:be:8c:ff
+00:24:7e:e0:1d:b5, 00:13:7f:be:8c:ff
+00:24:7e:e0:1d:b5, 00:13:7f:be:8c:ff
+00:24:7e:e0:1d:b5, 00:13:7f:be:8c:ff
+00:24:7e:e0:1d:b5, 00:13:7f:be:8c:ff
+00:24:7e:e0:1d:b5, 00:13:7f:be:8c:ff
+00:24:7e:e0:1d:b5, 00:13:7f:be:8c:ff
+00:24:7e:e0:1d:b5, 00:13:7f:be:8c:ff
+00:24:7e:e0:1d:b5, 00:13:7f:be:8c:ff
+00:24:7e:e0:1d:b5, 00:13:7f:be:8c:ff
+00:24:7e:e0:1d:b5, 00:13:7f:be:8c:ff
+00:24:7e:e0:1d:b5, 00:13:7f:be:8c:ff
+00:24:7e:e0:1d:b5, 00:13:7f:be:8c:ff
+00:24:7e:e0:1d:b5, 00:13:7f:be:8c:ff
+00:24:7e:e0:1d:b5, 00:13:7f:be:8c:ff
+00:24:7e:e0:1d:b5, 00:13:7f:be:8c:ff
+00:24:7e:e0:1d:b5, 00:13:7f:be:8c:ff
+00:24:7e:e0:1d:b5, 00:13:7f:be:8c:ff
+00:24:7e:e0:1d:b5, 00:13:7f:be:8c:ff
+00:24:7e:e0:1d:b5, 00:13:7f:be:8c:ff
+00:24:7e:e0:1d:b5, 00:13:7f:be:8c:ff
+00:13:7f:be:8c:ff, 00:e0:db:01:cf:4b
+00:16:76:23:d9:e3, 01:00:5e:00:00:fb
+f0:4d:a2:47:ba:25, ff:ff:ff:ff:ff:ff
+f0:4d:a2:47:ba:25, 33:33:00:01:00:03
+f0:4d:a2:47:ba:25, 01:00:5e:00:00:fc
+f0:4d:a2:47:ba:25, 33:33:00:01:00:03
+f0:4d:a2:47:ba:25, 01:00:5e:00:00:fc
+00:23:32:b6:0c:46, ff:ff:ff:ff:ff:ff
+90:72:40:97:b6:f5, 44:2b:03:aa:ab:8d
+a4:67:06:f7:ec:54, 33:33:00:00:00:fb

testing/btest/Baseline/core.ipv6-frag/dns.log

@@ -3,10 +3,10 @@
 #empty_field	(empty)
 #unset_field	-
 #path	dns
-#open	2014-04-24-20-25-19
+#open	2016-06-15-03-33-34
-#fields	ts	uid	id.orig_h	id.orig_p	id.resp_h	id.resp_p	proto	trans_id	query	qclass	qclass_name	qtype	qtype_name	rcode	rcode_name	AA	TC	RD	RA	Z	answers	TTLs	rejected
+#fields	ts	uid	id.orig_h	id.orig_p	id.resp_h	id.resp_p	proto	trans_id	rtt	query	qclass	qclass_name	qtype	qtype_name	rcode	rcode_name	AA	TC	RD	RA	Z	answers	TTLs	rejected
-#types	time	string	addr	port	addr	port	enum	count	string	count	string	count	string	count	string	bool	bool	bool	bool	count	vector[string]	vector[interval]	bool
+#types	time	string	addr	port	addr	port	enum	count	interval	string	count	string	count	string	count	string	bool	bool	bool	bool	count	vector[string]	vector[interval]	bool
-1331084278.438444	CXWv6p3arKYeMETxOg	2001:470:1f11:81f:d138:5f55:6d4:1fe2	51850	2607:f740:b::f93	53	udp	3903	txtpadding_323.n1.netalyzr.icsi.berkeley.edu	1	C_INTERNET	16	TXT	0	NOERROR	T	F	T	F	0	TXT 33 This TXT record should be ignored TXT 21 As it is just padding TXT 136 XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX	1.000000	F
+1331084278.438444	CXWv6p3arKYeMETxOg	2001:470:1f11:81f:d138:5f55:6d4:1fe2	51850	2607:f740:b::f93	53	udp	3903	0.079300	txtpadding_323.n1.netalyzr.icsi.berkeley.edu	1	C_INTERNET	16	TXT	0	NOERROR	T	F	T	F	0	TXT 33 This TXT record should be ignored TXT 21 As it is just padding TXT 136 XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX	1.000000	F
-1331084293.592245	CjhGID4nQcgTWjvg4c	2001:470:1f11:81f:d138:5f55:6d4:1fe2	51851	2607:f740:b::f93	53	udp	40849	txtpadding_3230.n1.netalyzr.icsi.berkeley.edu	1	C_INTERNET	16	TXT	0	NOERROR	T	F	T	F	0	TXT 33 This TXT record should be ignored TXT 21 As it is just padding TXT 189 XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX TXT 189 XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX TXT 189 XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX TXT 189 XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX TXT 189 XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX TXT 189 XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX TXT 189 XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX TXT 189 XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX TXT 189 XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX TXT 189 XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX TXT 189 XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX TXT 189 XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX TXT 189 XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX TXT 189 XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX TXT 189 XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX TXT 192 XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX	1.000000	F
+1331084293.592245	CjhGID4nQcgTWjvg4c	2001:470:1f11:81f:d138:5f55:6d4:1fe2	51851	2607:f740:b::f93	53	udp	40849	5.084025	txtpadding_3230.n1.netalyzr.icsi.berkeley.edu	1	C_INTERNET	16	TXT	0	NOERROR	T	F	T	F	0	TXT 33 This TXT record should be ignored TXT 21 As it is just padding TXT 189 XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX TXT 189 XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX TXT 189 XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX TXT 189 XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX TXT 189 XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX TXT 189 XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX TXT 189 XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX TXT 189 XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX TXT 189 XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX TXT 189 XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX TXT 189 XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX TXT 189 XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX TXT 189 XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX TXT 189 XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX TXT 189 XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX TXT 192 XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX	1.000000	F
-1331084298.593081	CjhGID4nQcgTWjvg4c	2001:470:1f11:81f:d138:5f55:6d4:1fe2	51851	2607:f740:b::f93	53	udp	40849	txtpadding_3230.n1.netalyzr.icsi.berkeley.edu	1	C_INTERNET	16	TXT	-	-	F	F	T	F	0	-	-	F
+1331084298.593081	CjhGID4nQcgTWjvg4c	2001:470:1f11:81f:d138:5f55:6d4:1fe2	51851	2607:f740:b::f93	53	udp	40849	-	txtpadding_3230.n1.netalyzr.icsi.berkeley.edu	1	C_INTERNET	16	TXT	-	-	F	F	T	F	0	-	-	F
-#close	2014-04-24-20-25-20
+#close	2016-06-15-03-33-34

testing/btest/Baseline/core.negative-time/weird.log

@@ -0,0 +1,10 @@
+#separator \x09
+#set_separator	,
+#empty_field	(empty)
+#unset_field	-
+#path	weird
+#open	2016-05-23-20-20-21
+#fields	ts	uid	id.orig_h	id.orig_p	id.resp_h	id.resp_p	name	addl	notice	peer
+#types	time	string	addr	port	addr	port	string	string	bool	string
+1425182592.408334	-	-	-	-	-	negative_packet_timestamp	-	F	bro
+#close	2016-05-23-20-20-21

testing/btest/Baseline/core.reassembly/output

@@ -3,7 +3,6 @@ flow weird, excessively_small_fragment, 164.1.123.163, 164.1.123.61
 flow weird, fragment_size_inconsistency, 164.1.123.163, 164.1.123.61
 flow weird, fragment_inconsistency, 164.1.123.163, 164.1.123.61
 flow weird, fragment_inconsistency, 164.1.123.163, 164.1.123.61
-flow weird, dns_unmatched_msg, 164.1.123.163, 164.1.123.61
 ----------------------
 flow weird, excessively_small_fragment, 164.1.123.163, 164.1.123.61
 flow weird, excessively_small_fragment, 164.1.123.163, 164.1.123.61

testing/btest/Baseline/core.recursive-event/output

@@ -0,0 +1 @@
+10

testing/btest/Baseline/core.tcp.fin-retransmit/out

@@ -1,2 +1,2 @@
-[size=0, state=5, num_pkts=3, num_bytes_ip=156, flow_label=0]
+[size=0, state=5, num_pkts=3, num_bytes_ip=156, flow_label=0, l2_addr=58:1f:aa:34:18:bc]
-[size=0, state=6, num_pkts=2, num_bytes_ip=92, flow_label=0]
+[size=0, state=6, num_pkts=2, num_bytes_ip=92, flow_label=0, l2_addr=c4:71:fe:3a:5d:c2]

testing/btest/Baseline/core.tcp.rst-after-syn/.stdout

@@ -1,3 +1,3 @@
 [orig_h=1.2.0.2, orig_p=2527/tcp, resp_h=1.2.0.3, resp_p=6649/tcp]
-orig:, [size=175, state=1, num_pkts=4, num_bytes_ip=395, flow_label=0]
+orig:, [size=175, state=1, num_pkts=4, num_bytes_ip=395, flow_label=0, l2_addr=00:15:17:0b:7c:61]
-resp:, [size=0, state=6, num_pkts=5, num_bytes_ip=236, flow_label=0]
+resp:, [size=0, state=6, num_pkts=5, num_bytes_ip=236, flow_label=0, l2_addr=00:00:00:00:00:04]

testing/btest/Baseline/core.tunnels.ayiya/http.log

@@ -3,10 +3,10 @@
 #empty_field	(empty)
 #unset_field	-
 #path	http
-#open	2016-01-15-18-40-13
+#open	2016-06-15-05-35-59
-#fields	ts	uid	id.orig_h	id.orig_p	id.resp_h	id.resp_p	trans_depth	method	host	uri	referrer	version	user_agent	request_body_len	response_body_len	status_code	status_msg	info_code	info_msg	filename	tags	username	password	proxied	orig_fuids	orig_mime_types	resp_fuids	resp_mime_types
+#fields	ts	uid	id.orig_h	id.orig_p	id.resp_h	id.resp_p	trans_depth	method	host	uri	referrer	version	user_agent	request_body_len	response_body_len	status_code	status_msg	info_code	info_msg	tags	username	password	proxied	orig_fuids	orig_filenames	orig_mime_types	resp_fuids	resp_filenames	resp_mime_types
-#types	time	string	addr	port	addr	port	count	string	string	string	string	string	string	count	count	count	string	count	string	string	set[enum]	string	string	set[string]	vector[string]	vector[string]	vector[string]	vector[string]
+#types	time	string	addr	port	addr	port	count	string	string	string	string	string	string	count	count	count	string	count	string	set[enum]	string	string	set[string]	vector[string]	vector[string]	vector[string]	vector[string]	vector[string]	vector[string]
-1257655301.652206	CIPOse170MGiRM1Qf4	2001:4978:f:4c::2	53382	2001:4860:b002::68	80	1	GET	ipv6.google.com	/	-	1.1	Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.5; en; rv:1.9.0.15pre) Gecko/2009091516 Camino/2.0b4 (like Firefox/3.0.15pre)	0	10102	200	OK	-	-	-	(empty)	-	-	-	-	-	FYAtjT24MvCBUs5K5f	text/html
+1257655301.652206	CIPOse170MGiRM1Qf4	2001:4978:f:4c::2	53382	2001:4860:b002::68	80	1	GET	ipv6.google.com	/	-	1.1	Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.5; en; rv:1.9.0.15pre) Gecko/2009091516 Camino/2.0b4 (like Firefox/3.0.15pre)	0	10102	200	OK	-	-	(empty)	-	-	-	-	-	-	FYAtjT24MvCBUs5K5f	-	text/html
-1257655302.514424	CIPOse170MGiRM1Qf4	2001:4978:f:4c::2	53382	2001:4860:b002::68	80	2	GET	ipv6.google.com	/csi?v=3&s=webhp&action=&tran=undefined&e=17259,19771,21517,21766,21887,22212&ei=BUz2Su7PMJTglQfz3NzCAw&rt=prt.77,xjs.565,ol.645	http://ipv6.google.com/	1.1	Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.5; en; rv:1.9.0.15pre) Gecko/2009091516 Camino/2.0b4 (like Firefox/3.0.15pre)	0	0	204	No Content	-	-	-	(empty)	-	-	-	-	-	-	-
+1257655302.514424	CIPOse170MGiRM1Qf4	2001:4978:f:4c::2	53382	2001:4860:b002::68	80	2	GET	ipv6.google.com	/csi?v=3&s=webhp&action=&tran=undefined&e=17259,19771,21517,21766,21887,22212&ei=BUz2Su7PMJTglQfz3NzCAw&rt=prt.77,xjs.565,ol.645	http://ipv6.google.com/	1.1	Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.5; en; rv:1.9.0.15pre) Gecko/2009091516 Camino/2.0b4 (like Firefox/3.0.15pre)	0	0	204	No Content	-	-	(empty)	-	-	-	-	-	-	-	-	-
-1257655303.603569	CIPOse170MGiRM1Qf4	2001:4978:f:4c::2	53382	2001:4860:b002::68	80	3	GET	ipv6.google.com	/gen_204?atyp=i&ct=fade&cad=1254&ei=BUz2Su7PMJTglQfz3NzCAw&zx=1257655303600	http://ipv6.google.com/	1.1	Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.5; en; rv:1.9.0.15pre) Gecko/2009091516 Camino/2.0b4 (like Firefox/3.0.15pre)	0	0	204	No Content	-	-	-	(empty)	-	-	-	-	-	-	-
+1257655303.603569	CIPOse170MGiRM1Qf4	2001:4978:f:4c::2	53382	2001:4860:b002::68	80	3	GET	ipv6.google.com	/gen_204?atyp=i&ct=fade&cad=1254&ei=BUz2Su7PMJTglQfz3NzCAw&zx=1257655303600	http://ipv6.google.com/	1.1	Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.5; en; rv:1.9.0.15pre) Gecko/2009091516 Camino/2.0b4 (like Firefox/3.0.15pre)	0	0	204	No Content	-	-	(empty)	-	-	-	-	-	-	-	-	-
-#close	2016-01-15-18-40-13
+#close	2016-06-15-05-35-59

testing/btest/Baseline/core.tunnels.gre/dns.log

@@ -3,9 +3,9 @@
 #empty_field	(empty)
 #unset_field	-
 #path	dns
-#open	2014-01-16-21-51-12
+#open	2016-06-15-03-34-43
-#fields	ts	uid	id.orig_h	id.orig_p	id.resp_h	id.resp_p	proto	trans_id	query	qclass	qclass_name	qtype	qtype_name	rcode	rcode_name	AA	TC	RD	RA	Z	answers	TTLs	rejected
+#fields	ts	uid	id.orig_h	id.orig_p	id.resp_h	id.resp_p	proto	trans_id	rtt	query	qclass	qclass_name	qtype	qtype_name	rcode	rcode_name	AA	TC	RD	RA	Z	answers	TTLs	rejected
-#types	time	string	addr	port	addr	port	enum	count	string	count	string	count	string	count	string	bool	bool	bool	bool	count	vector[string]	vector[interval]	bool
+#types	time	string	addr	port	addr	port	enum	count	interval	string	count	string	count	string	count	string	bool	bool	bool	bool	count	vector[string]	vector[interval]	bool
-1055289987.055189	CRJuHdVW0XPVINV8a	66.59.111.190	37675	172.28.2.3	53	udp	48554	www.gleeble.org	1	C_INTERNET	255	*	-	-	F	F	T	F	0	-	-	F
+1055289987.055189	CRJuHdVW0XPVINV8a	66.59.111.190	37675	172.28.2.3	53	udp	48554	-	www.gleeble.org	1	C_INTERNET	255	*	-	-	F	F	T	F	0	-	-	F
-1055289992.056330	CRJuHdVW0XPVINV8a	66.59.111.190	37675	172.28.2.3	53	udp	48554	www.gleeble.org	1	C_INTERNET	255	*	-	-	F	F	T	F	0	-	-	F
+1055289992.056330	CRJuHdVW0XPVINV8a	66.59.111.190	37675	172.28.2.3	53	udp	48554	-	www.gleeble.org	1	C_INTERNET	255	*	-	-	F	F	T	F	0	-	-	F
-#close	2014-01-16-21-51-12
+#close	2016-06-15-03-34-43

testing/btest/Baseline/core.tunnels.gtp.different_dl_and_ul/http.log

@@ -3,9 +3,9 @@
 #empty_field	(empty)
 #unset_field	-
 #path	http
-#open	2016-01-15-18-40-14
+#open	2016-06-15-05-35-27
-#fields	ts	uid	id.orig_h	id.orig_p	id.resp_h	id.resp_p	trans_depth	method	host	uri	referrer	version	user_agent	request_body_len	response_body_len	status_code	status_msg	info_code	info_msg	filename	tags	username	password	proxied	orig_fuids	orig_mime_types	resp_fuids	resp_mime_types
+#fields	ts	uid	id.orig_h	id.orig_p	id.resp_h	id.resp_p	trans_depth	method	host	uri	referrer	version	user_agent	request_body_len	response_body_len	status_code	status_msg	info_code	info_msg	tags	username	password	proxied	orig_fuids	orig_filenames	orig_mime_types	resp_fuids	resp_filenames	resp_mime_types
-#types	time	string	addr	port	addr	port	count	string	string	string	string	string	string	count	count	count	string	count	string	string	set[enum]	string	string	set[string]	vector[string]	vector[string]	vector[string]	vector[string]
+#types	time	string	addr	port	addr	port	count	string	string	string	string	string	string	count	count	count	string	count	string	set[enum]	string	string	set[string]	vector[string]	vector[string]	vector[string]	vector[string]	vector[string]	vector[string]
-1333458850.340368	CjhGID4nQcgTWjvg4c	10.131.17.170	51803	173.199.115.168	80	1	GET	cdn.epicgameads.com	/ads/flash/728x90_nx8com.swf?clickTAG=http://www.epicgameads.com/ads/bannerclickPage.php?id=e3ubwU6IF&pd=1&adid=0&icpc=1&axid=0&uctt=1&channel=4&cac=1&t=728x90&cb=1333458879	http://www.epicgameads.com/ads/banneriframe.php?id=e3ubwU6IF&t=728x90&channel=4&cb=1333458905296	1.1	Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Trident/5.0)	0	31461	200	OK	-	-	-	(empty)	-	-	-	-	-	FHKKd91EMHBEK0hbdg	application/x-shockwave-flash
+1333458850.340368	CjhGID4nQcgTWjvg4c	10.131.17.170	51803	173.199.115.168	80	1	GET	cdn.epicgameads.com	/ads/flash/728x90_nx8com.swf?clickTAG=http://www.epicgameads.com/ads/bannerclickPage.php?id=e3ubwU6IF&pd=1&adid=0&icpc=1&axid=0&uctt=1&channel=4&cac=1&t=728x90&cb=1333458879	http://www.epicgameads.com/ads/banneriframe.php?id=e3ubwU6IF&t=728x90&channel=4&cb=1333458905296	1.1	Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Trident/5.0)	0	31461	200	OK	-	-	(empty)	-	-	-	-	-	-	FHKKd91EMHBEK0hbdg	-	application/x-shockwave-flash
-1333458850.399501	CjhGID4nQcgTWjvg4c	10.131.17.170	51803	173.199.115.168	80	2	GET	cdn.epicgameads.com	/ads/flash/728x90_nx8com.swf?clickTAG=http://www.epicgameads.com/ads/bannerclickPage.php?id=e3ubwU6IF&pd=1&adid=0&icpc=1&axid=0&uctt=1&channel=0&cac=1&t=728x90&cb=1333458881	http://www.epicgameads.com/ads/banneriframe.php?id=e3ubwU6IF&t=728x90&cb=1333458920207	1.1	Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Trident/5.0)	0	31461	200	OK	-	-	-	(empty)	-	-	-	-	-	Fu64Vqjy6nBop9nRd	application/x-shockwave-flash
+1333458850.399501	CjhGID4nQcgTWjvg4c	10.131.17.170	51803	173.199.115.168	80	2	GET	cdn.epicgameads.com	/ads/flash/728x90_nx8com.swf?clickTAG=http://www.epicgameads.com/ads/bannerclickPage.php?id=e3ubwU6IF&pd=1&adid=0&icpc=1&axid=0&uctt=1&channel=0&cac=1&t=728x90&cb=1333458881	http://www.epicgameads.com/ads/banneriframe.php?id=e3ubwU6IF&t=728x90&cb=1333458920207	1.1	Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Trident/5.0)	0	31461	200	OK	-	-	(empty)	-	-	-	-	-	-	Fu64Vqjy6nBop9nRd	-	application/x-shockwave-flash
-#close	2016-01-15-18-40-14
+#close	2016-06-15-05-35-27

testing/btest/Baseline/core.tunnels.gtp.false_gtp/dns.log

@@ -3,8 +3,8 @@
 #empty_field	(empty)
 #unset_field	-
 #path	dns
-#open	2013-08-26-19-35-00
+#open	2016-06-15-04-11-36
-#fields	ts	uid	id.orig_h	id.orig_p	id.resp_h	id.resp_p	proto	trans_id	query	qclass	qclass_name	qtype	qtype_name	rcode	rcode_name	AA	TC	RD	RA	Z	answers	TTLs	rejected
+#fields	ts	uid	id.orig_h	id.orig_p	id.resp_h	id.resp_p	proto	trans_id	rtt	query	qclass	qclass_name	qtype	qtype_name	rcode	rcode_name	AA	TC	RD	RA	Z	answers	TTLs	rejected
-#types	time	string	addr	port	addr	port	enum	count	string	count	string	count	string	count	string	bool	bool	bool	bool	count	vector[string]	vector[interval]	bool
+#types	time	string	addr	port	addr	port	enum	count	interval	string	count	string	count	string	count	string	bool	bool	bool	bool	count	vector[string]	vector[interval]	bool
-1333458871.219794	CXWv6p3arKYeMETxOg	10.131.24.6	2152	195.178.38.3	53	udp	27595	abcd.efg.hijklm.nm	1	C_INTERNET	1	A	-	-	F	F	T	F	0	-	-	F
+1333458871.219794	CXWv6p3arKYeMETxOg	10.131.24.6	2152	195.178.38.3	53	udp	27595	-	abcd.efg.hijklm.nm	1	C_INTERNET	1	A	-	-	F	F	T	F	0	-	-	F
-#close	2013-08-26-19-35-00
+#close	2016-06-15-04-11-36

testing/btest/Baseline/core.tunnels.gtp.outer_ip_frag/http.log

@@ -3,8 +3,8 @@
 #empty_field	(empty)
 #unset_field	-
 #path	http
-#open	2016-01-15-18-40-15
+#open	2016-06-15-05-36-15
-#fields	ts	uid	id.orig_h	id.orig_p	id.resp_h	id.resp_p	trans_depth	method	host	uri	referrer	version	user_agent	request_body_len	response_body_len	status_code	status_msg	info_code	info_msg	filename	tags	username	password	proxied	orig_fuids	orig_mime_types	resp_fuids	resp_mime_types
+#fields	ts	uid	id.orig_h	id.orig_p	id.resp_h	id.resp_p	trans_depth	method	host	uri	referrer	version	user_agent	request_body_len	response_body_len	status_code	status_msg	info_code	info_msg	tags	username	password	proxied	orig_fuids	orig_filenames	orig_mime_types	resp_fuids	resp_filenames	resp_mime_types
-#types	time	string	addr	port	addr	port	count	string	string	string	string	string	string	count	count	count	string	count	string	string	set[enum]	string	string	set[string]	vector[string]	vector[string]	vector[string]	vector[string]
+#types	time	string	addr	port	addr	port	count	string	string	string	string	string	string	count	count	count	string	count	string	set[enum]	string	string	set[string]	vector[string]	vector[string]	vector[string]	vector[string]	vector[string]	vector[string]
-1333458850.375568	CjhGID4nQcgTWjvg4c	10.131.47.185	1923	79.101.110.141	80	1	GET	o-o.preferred.telekomrs-beg1.v2.lscache8.c.youtube.com	/videoplayback?upn=MTU2MDY5NzQ5OTM0NTI3NDY4NDc&sparams=algorithm,burst,cp,factor,id,ip,ipbits,itag,source,upn,expire&fexp=912300,907210&algorithm=throttle-factor&itag=34&ip=212.0.0.0&burst=40&sver=3&signature=832FB1042E20780CFCA77A4DB5EA64AC593E8627.D1166C7E8365732E52DAFD68076DAE0146E0AE01&source=youtube&expire=1333484980&key=yt1&ipbits=8&factor=1.25&cp=U0hSSFRTUl9NSkNOMl9MTVZKOjh5eEN2SG8tZF84&id=ebf1e932d4bd1286&cm2=1	http://s.ytimg.com/yt/swfbin/watch_as3-vflqrJwOA.swf	1.1	Mozilla/5.0 (Windows NT 5.1) AppleWebKit/535.11 (KHTML, like Gecko; X-SBLSP) Chrome/17.0.963.83 Safari/535.11	0	56320	206	Partial Content	-	-	-	(empty)	-	-	-	-	-	FNJkBA1b8FSHt5N8jl	-
+1333458850.375568	CjhGID4nQcgTWjvg4c	10.131.47.185	1923	79.101.110.141	80	1	GET	o-o.preferred.telekomrs-beg1.v2.lscache8.c.youtube.com	/videoplayback?upn=MTU2MDY5NzQ5OTM0NTI3NDY4NDc&sparams=algorithm,burst,cp,factor,id,ip,ipbits,itag,source,upn,expire&fexp=912300,907210&algorithm=throttle-factor&itag=34&ip=212.0.0.0&burst=40&sver=3&signature=832FB1042E20780CFCA77A4DB5EA64AC593E8627.D1166C7E8365732E52DAFD68076DAE0146E0AE01&source=youtube&expire=1333484980&key=yt1&ipbits=8&factor=1.25&cp=U0hSSFRTUl9NSkNOMl9MTVZKOjh5eEN2SG8tZF84&id=ebf1e932d4bd1286&cm2=1	http://s.ytimg.com/yt/swfbin/watch_as3-vflqrJwOA.swf	1.1	Mozilla/5.0 (Windows NT 5.1) AppleWebKit/535.11 (KHTML, like Gecko; X-SBLSP) Chrome/17.0.963.83 Safari/535.11	0	56320	206	Partial Content	-	-	(empty)	-	-	-	-	-	-	FNJkBA1b8FSHt5N8jl	-	-
-#close	2016-01-15-18-40-15
+#close	2016-06-15-05-36-15

testing/btest/Baseline/core.tunnels.teredo/http.log

@@ -3,11 +3,11 @@
 #empty_field	(empty)
 #unset_field	-
 #path	http
-#open	2016-01-15-18-40-16
+#open	2016-06-15-05-36-31
-#fields	ts	uid	id.orig_h	id.orig_p	id.resp_h	id.resp_p	trans_depth	method	host	uri	referrer	version	user_agent	request_body_len	response_body_len	status_code	status_msg	info_code	info_msg	filename	tags	username	password	proxied	orig_fuids	orig_mime_types	resp_fuids	resp_mime_types
+#fields	ts	uid	id.orig_h	id.orig_p	id.resp_h	id.resp_p	trans_depth	method	host	uri	referrer	version	user_agent	request_body_len	response_body_len	status_code	status_msg	info_code	info_msg	tags	username	password	proxied	orig_fuids	orig_filenames	orig_mime_types	resp_fuids	resp_filenames	resp_mime_types
-#types	time	string	addr	port	addr	port	count	string	string	string	string	string	string	count	count	count	string	count	string	string	set[enum]	string	string	set[string]	vector[string]	vector[string]	vector[string]	vector[string]
+#types	time	string	addr	port	addr	port	count	string	string	string	string	string	string	count	count	count	string	count	string	set[enum]	string	string	set[string]	vector[string]	vector[string]	vector[string]	vector[string]	vector[string]	vector[string]
-1210953057.917183	C7XEbhP654jzLoe3a	192.168.2.16	1578	75.126.203.78	80	1	POST	download913.avast.com	/cgi-bin/iavs4stats.cgi	-	1.1	Syncer/4.80 (av_pro-1169;f)	589	0	204	<empty>	-	-	-	(empty)	-	-	-	Fp32SIJztq0Szn5Qc	text/plain	-	-
+1210953057.917183	C7XEbhP654jzLoe3a	192.168.2.16	1578	75.126.203.78	80	1	POST	download913.avast.com	/cgi-bin/iavs4stats.cgi	-	1.1	Syncer/4.80 (av_pro-1169;f)	589	0	204	<empty>	-	-	(empty)	-	-	-	Fp32SIJztq0Szn5Qc	-	text/plain	-	-	-
-1210953061.585996	CwSkQu4eWZCH7OONC1	2001:0:4137:9e50:8000:f12a:b9c8:2815	1286	2001:4860:0:2001::68	80	1	GET	ipv6.google.com	/	-	1.1	Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9b5) Gecko/2008032620 Firefox/3.0b5	0	6640	200	OK	-	-	-	(empty)	-	-	-	-	-	FNFYdH11h5iQcoD3a2	text/html
+1210953061.585996	CwSkQu4eWZCH7OONC1	2001:0:4137:9e50:8000:f12a:b9c8:2815	1286	2001:4860:0:2001::68	80	1	GET	ipv6.google.com	/	-	1.1	Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9b5) Gecko/2008032620 Firefox/3.0b5	0	6640	200	OK	-	-	(empty)	-	-	-	-	-	-	FNFYdH11h5iQcoD3a2	-	text/html
-1210953073.381474	CwSkQu4eWZCH7OONC1	2001:0:4137:9e50:8000:f12a:b9c8:2815	1286	2001:4860:0:2001::68	80	2	GET	ipv6.google.com	/search?hl=en&q=Wireshark+!&btnG=Google+Search	http://ipv6.google.com/	1.1	Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9b5) Gecko/2008032620 Firefox/3.0b5	0	25119	200	OK	-	-	-	(empty)	-	-	-	-	-	FHD5nv1iSVFZVM0aH7	text/html
+1210953073.381474	CwSkQu4eWZCH7OONC1	2001:0:4137:9e50:8000:f12a:b9c8:2815	1286	2001:4860:0:2001::68	80	2	GET	ipv6.google.com	/search?hl=en&q=Wireshark+!&btnG=Google+Search	http://ipv6.google.com/	1.1	Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9b5) Gecko/2008032620 Firefox/3.0b5	0	25119	200	OK	-	-	(empty)	-	-	-	-	-	-	FHD5nv1iSVFZVM0aH7	-	text/html
-1210953074.674817	Cab0vO1xNYSS2hJkle	192.168.2.16	1580	67.228.110.120	80	1	GET	www.wireshark.org	/	http://ipv6.google.com/search?hl=en&q=Wireshark+%21&btnG=Google+Search	1.1	Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9b5) Gecko/2008032620 Firefox/3.0b5	0	11845	200	OK	-	-	-	(empty)	-	-	-	-	-	FS7lUf2cJFAVBCu6w6	text/html
+1210953074.674817	Cab0vO1xNYSS2hJkle	192.168.2.16	1580	67.228.110.120	80	1	GET	www.wireshark.org	/	http://ipv6.google.com/search?hl=en&q=Wireshark+%21&btnG=Google+Search	1.1	Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9b5) Gecko/2008032620 Firefox/3.0b5	0	11845	200	OK	-	-	(empty)	-	-	-	-	-	-	FS7lUf2cJFAVBCu6w6	-	text/html
-#close	2016-01-15-18-40-16
+#close	2016-06-15-05-36-31

testing/btest/Baseline/core.tunnels.teredo_bubble_with_payload/http.log

@@ -3,9 +3,9 @@
 #empty_field	(empty)
 #unset_field	-
 #path	http
-#open	2016-01-15-18-40-17
+#open	2016-06-15-05-36-42
-#fields	ts	uid	id.orig_h	id.orig_p	id.resp_h	id.resp_p	trans_depth	method	host	uri	referrer	version	user_agent	request_body_len	response_body_len	status_code	status_msg	info_code	info_msg	filename	tags	username	password	proxied	orig_fuids	orig_mime_types	resp_fuids	resp_mime_types
+#fields	ts	uid	id.orig_h	id.orig_p	id.resp_h	id.resp_p	trans_depth	method	host	uri	referrer	version	user_agent	request_body_len	response_body_len	status_code	status_msg	info_code	info_msg	tags	username	password	proxied	orig_fuids	orig_filenames	orig_mime_types	resp_fuids	resp_filenames	resp_mime_types
-#types	time	string	addr	port	addr	port	count	string	string	string	string	string	string	count	count	count	string	count	string	string	set[enum]	string	string	set[string]	vector[string]	vector[string]	vector[string]	vector[string]
+#types	time	string	addr	port	addr	port	count	string	string	string	string	string	string	count	count	count	string	count	string	set[enum]	string	string	set[string]	vector[string]	vector[string]	vector[string]	vector[string]	vector[string]	vector[string]
-1340127577.361683	C6pKV8GSxOnSLghOa	2001:0:4137:9e50:8000:f12a:b9c8:2815	1286	2001:4860:0:2001::68	80	1	GET	ipv6.google.com	/	-	1.1	Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9b5) Gecko/2008032620 Firefox/3.0b5	0	6640	200	OK	-	-	-	(empty)	-	-	-	-	-	FWSTWv4EZLVlc2Zywi	text/html
+1340127577.361683	C6pKV8GSxOnSLghOa	2001:0:4137:9e50:8000:f12a:b9c8:2815	1286	2001:4860:0:2001::68	80	1	GET	ipv6.google.com	/	-	1.1	Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9b5) Gecko/2008032620 Firefox/3.0b5	0	6640	200	OK	-	-	(empty)	-	-	-	-	-	-	FWSTWv4EZLVlc2Zywi	-	text/html
-1340127577.379360	C6pKV8GSxOnSLghOa	2001:0:4137:9e50:8000:f12a:b9c8:2815	1286	2001:4860:0:2001::68	80	2	GET	ipv6.google.com	/search?hl=en&q=Wireshark+!&btnG=Google+Search	http://ipv6.google.com/	1.1	Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9b5) Gecko/2008032620 Firefox/3.0b5	0	25119	200	OK	-	-	-	(empty)	-	-	-	-	-	FGKV3B3jz083xhGO13	text/html
+1340127577.379360	C6pKV8GSxOnSLghOa	2001:0:4137:9e50:8000:f12a:b9c8:2815	1286	2001:4860:0:2001::68	80	2	GET	ipv6.google.com	/search?hl=en&q=Wireshark+!&btnG=Google+Search	http://ipv6.google.com/	1.1	Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9b5) Gecko/2008032620 Firefox/3.0b5	0	25119	200	OK	-	-	(empty)	-	-	-	-	-	-	FGKV3B3jz083xhGO13	-	text/html
-#close	2016-01-15-18-40-17
+#close	2016-06-15-05-36-42

testing/btest/Baseline/coverage.default-load-baseline/canonified_loaded_scripts.log

@@ -3,7 +3,7 @@
 #empty_field	(empty)
 #unset_field	-
 #path	loaded_scripts
-#open	2016-04-29-20-49-25
+#open	2016-06-07-19-22-42
 #fields	name
 #types	string
 scripts/base/init-bare.bro
@@ -142,7 +142,9 @@ scripts/base/init-default.bro
       scripts/base/frameworks/reporter/main.bro
     scripts/base/utils/paths.bro
   scripts/base/utils/directions-and-hosts.bro
+  scripts/base/utils/email.bro
   scripts/base/utils/files.bro
+  scripts/base/utils/geoip-distance.bro
   scripts/base/utils/numbers.bro
   scripts/base/utils/queue.bro
   scripts/base/utils/strings.bro
@@ -309,4 +311,4 @@ scripts/base/init-default.bro
   scripts/base/misc/find-checksum-offloading.bro
   scripts/base/misc/find-filtered-trace.bro
 scripts/policy/misc/loaded-scripts.bro
-#close	2016-05-02-20-39-35
+#close	2016-06-07-19-22-42

testing/btest/Baseline/coverage.find-bro-logs/out

@@ -1,4 +1,3 @@
-app_stats
 barnyard2
 capture_loss
 cluster