Merge remote-tracking branch 'origin/master' into topic/johanna/netcontrol-improvements

CHANGES

@@ -1,4 +1,104 @@
 
+2.4-498 | 2016-04-28 11:34:52 -0700
+
+  * Rename Broker::print to Broker::send_print and Broker::event to
+    Broker::send_event to avoid using reserved keywords as function
+    names. (Daniel Thayer)
+
+  * Add script wrapper functions for Broker BIFs. This faciliates
+    documenting them through Broxygen. (Daniel Thayer)
+
+  * Extend, update, and clean up Broker tests. (Daniel Thayer)
+
+  * Intel: Allow to provide uid/fuid instead of conn/file. (Johanna
+    Amann)
+
+  * Provide file IDs for hostname matches in certificates. (Johanna
+    Amann)
+
+  * Rudimentary IMAP StartTLS analyzer. It parses certificates out of
+    IMAP connections using StartTLS. It aborts processing if StartTLS
+    is not found. (Johanna Amann)
+
+2.4-478 | 2016-04-28 09:56:24
+
+  * Fix parsing of x509 pre-y2k dates. (Johanna Amann)
+
+  * Fix small error in bif documentation. (Johanna Amann)
+
+  * Fix unknown data link type error message. (Vitaly Repin)
+
+  * Correcting spelling errors. (Jeannette Dopheide)
+
+  * Minor cleanup in ARP analyzer. (Johanna Amann)
+
+  * Fix parsing of pre-y2k dates in X509 certificates. (Johanna Amann)
+
+  * Fix small error in get_current_packet documentation. (Johanna Amann)
+
+2.4-471 | 2016-04-25 15:37:15 -0700
+
+  * Add DNS tests for huge TLLs and CAA. (Johanna Amann)
+
+  * Add DNS "CAA" RR type and event. (Mark Taylor)
+
+  * Fix DNS response parsing: TTLs are unsigned. (Mark Taylor)
+
+2.4-466 | 2016-04-22 16:25:33 -0700
+
+  * Rename BrokerStore and BrokerComm to Broker. Also split broker main.bro
+    into two scripts. (Daniel Thayer)
+
+  * Add get_current_packet_header bif. (Jan Grashoefer)
+
+2.4-457 | 2016-04-22 08:36:27 -0700
+
+  * Fix Intel framework not checking the CERT_HASH indicator type. (Johanna Amann)
+
+2.4-454 | 2016-04-14 10:06:58 -0400
+
+  * Additional mime types for file identification and a few fixes. (Seth Hall)
+
+    New file mime types:
+     - .ini files
+     - MS Registry policy files
+     - MS Registry files
+     - MS Registry format files (e.g. DESKTOP.DAT)
+     - MS Outlook PST files
+     - Apple AFPInfo files
+
+    Mime type fixes:
+     - MP3 files with ID3 tags.
+     - JSON and XML matchers were extended
+
+  * Avoid a macro name conflict on FreeBSD. (Seth Hall, Daniel Thayer)
+
+2.4-452 | 2016-04-13 01:15:20 -0400
+
+  * Add a simple file entropy analyzer. (Seth Hall)
+
+  * Analyzer and bro script for RFB/VNC protocol (Martin van Hensbergen)
+
+    This analyzer parses the Remote Frame Buffer
+    protocol, usually referred to as the 'VNC protocol'.
+
+    It supports several dialects (3.3, 3.7, 3.8) and
+    also handles the Apple Remote Desktop variant.
+
+    It will log such facts as client/server versions,
+    authentication method used, authentication result,
+    height, width and name of the shared screen.
+
+
+2.4-430 | 2016-04-07 13:36:36 -0700
+
+  * Fix regex literal in scripting documentation. (William Tom)
+
+2.4-428 | 2016-04-07 13:33:08 -0700
+
+  * Confirm protocol in SNMP/SIP only if we saw a response SNMP/SIP
+    packet. (Vlad Grigorescu)
+
 2.4-424 | 2016-03-24 13:38:47 -0700
 
   * Only load openflow/netcontrol if compiled with broker. (Johanna Amann)

NEWS

@@ -26,11 +26,24 @@ New Functionality
 - Bro now includes the NetControl framework. The framework allows for easy
   interaction of Bro with hard- and software switches, firewalls, etc.
 
+- There is a new file entropy analyzer for files.
+
+- Bro now supports the remote framebuffer protocol (RFB) that is used by
+  VNC servers for remote graphical displays.
+
 - Bro now supports the Radiotap header for 802.11 frames.
 
+- Bro now has a rudimentary IMAP analyzer examinig the initial phase
+  of the protocol. Right now the analyzer only identify STARTTLS
+  sessions, handing them over to TLS analysis. The analyzer does not
+  yet analyze any further IMAP content.
+
 - Bro now tracks VLAN IDs. To record them inside the connection log,
   load protocols/conn/vlan-logging.bro.
 
+- A new dns_CAA_reply event gives access to DNS Certification Authority
+  Authorization replies.
+
 - A new per-packet event raw_packet() provides access to layer 2
   information. Use with care, generating events per packet is
   expensive.
@@ -40,6 +53,9 @@ New Functionality
   argument that will be used for decoding errors into weird.log
   (instead of reporter.log).
 
+- A new get_current_packet_header bif returns the headers of the current
+  packet.
+
 - Two new built-in functions for handling set[subnet] and table[subnet]:
 
   - check_subnet(subnet, table) checks if a specific subnet is a member
@@ -79,6 +95,10 @@ New Functionality
 Changed Functionality
 ---------------------
 
+- The BrokerComm and BrokerStore namespaces were renamed to Broker.
+  The Broker "print" function was renamed to Broker::send_print, and
+  "event" to "Broker::send_event".
+
 - ``SSH::skip_processing_after_detection`` was removed. The functionality was
   replaced by ``SSH::disable_analyzer_after_detection``.
 

VERSION

@@ -1 +1 @@
-2.4-424
+2.4-498

aux/binpac

@@ -1 +1 @@
-Subproject commit 424d40c1e8d5888311b50c0e5a9dfc9c5f818b66
+Subproject commit edbbe445d92cc6a5c2557661195f486b784769db

aux/bro-aux

@@ -1 +1 @@
-Subproject commit 105dfe4ad6c4ae4563b21cb0466ee350f0af0d43
+Subproject commit cb771a3cf592d46643eea35d206b9f3e1a0758f7

aux/broccoli

@@ -1 +1 @@
-Subproject commit 6ded82da498d805def6aa129cd7691d3b7287c37
+Subproject commit b4d1686cdd3f5505e405667b1083e8335cae6928

aux/broctl

@@ -1 +1 @@
-Subproject commit 583f3a3ff1847cf96a87f865d5cf0f36fae9dd67
+Subproject commit 7df7878abfd864f9ae5609918c0f04f58b5f5e2d

aux/broker

@@ -1 +1 @@
-Subproject commit 6684ab5109f526fb535013760f17a4c8dff093ae
+Subproject commit bb3f55f198f9cfd5e545345dd6425dd08ca1d45e

cmake

@@ -1 +1 @@
-Subproject commit 537e45afe1006a10f73847fab5f13d28ce43fc4d
+Subproject commit 0a2b36874ad5c1a22829135f8aeeac534469053f

doc/frameworks/broker.rst

@@ -17,20 +17,20 @@ Connecting to Peers
 ===================
 
 Communication via Broker must first be turned on via
-:bro:see:`BrokerComm::enable`.
+:bro:see:`Broker::enable`.
 
-Bro can accept incoming connections by calling :bro:see:`BrokerComm::listen`
+Bro can accept incoming connections by calling :bro:see:`Broker::listen`
 and then monitor connection status updates via the
-:bro:see:`BrokerComm::incoming_connection_established` and
-:bro:see:`BrokerComm::incoming_connection_broken` events.
+:bro:see:`Broker::incoming_connection_established` and
+:bro:see:`Broker::incoming_connection_broken` events.
 
 .. btest-include:: ${DOC_ROOT}/frameworks/broker/connecting-listener.bro
 
-Bro can initiate outgoing connections by calling :bro:see:`BrokerComm::connect`
+Bro can initiate outgoing connections by calling :bro:see:`Broker::connect`
 and then monitor connection status updates via the
-:bro:see:`BrokerComm::outgoing_connection_established`,
-:bro:see:`BrokerComm::outgoing_connection_broken`, and
-:bro:see:`BrokerComm::outgoing_connection_incompatible` events.
+:bro:see:`Broker::outgoing_connection_established`,
+:bro:see:`Broker::outgoing_connection_broken`, and
+:bro:see:`Broker::outgoing_connection_incompatible` events.
 
 .. btest-include:: ${DOC_ROOT}/frameworks/broker/connecting-connector.bro
 
@@ -38,14 +38,14 @@ Remote Printing
 ===============
 
 To receive remote print messages, first use the
-:bro:see:`BrokerComm::subscribe_to_prints` function to advertise to peers a
+:bro:see:`Broker::subscribe_to_prints` function to advertise to peers a
 topic prefix of interest and then create an event handler for
-:bro:see:`BrokerComm::print_handler` to handle any print messages that are
+:bro:see:`Broker::print_handler` to handle any print messages that are
 received.
 
 .. btest-include:: ${DOC_ROOT}/frameworks/broker/printing-listener.bro
 
-To send remote print messages, just call :bro:see:`BrokerComm::print`.
+To send remote print messages, just call :bro:see:`Broker::send_print`.
 
 .. btest-include:: ${DOC_ROOT}/frameworks/broker/printing-connector.bro
 
@@ -69,14 +69,14 @@ Remote Events
 =============
 
 Receiving remote events is similar to remote prints.  Just use the
-:bro:see:`BrokerComm::subscribe_to_events` function and possibly define any
+:bro:see:`Broker::subscribe_to_events` function and possibly define any
 new events along with handlers that peers may want to send.
 
 .. btest-include:: ${DOC_ROOT}/frameworks/broker/events-listener.bro
 
 There are two different ways to send events.  The first is to call the
-:bro:see:`BrokerComm::event` function directly.  The second option is to call
-the :bro:see:`BrokerComm::auto_event` function where you specify a
+:bro:see:`Broker::send_event` function directly.  The second option is to call
+the :bro:see:`Broker::auto_event` function where you specify a
 particular event that will be automatically sent to peers whenever the
 event is called locally via the normal event invocation syntax.
 
@@ -104,14 +104,14 @@ Remote Logging
 
 .. btest-include:: ${DOC_ROOT}/frameworks/broker/testlog.bro
 
-Use the :bro:see:`BrokerComm::subscribe_to_logs` function to advertise interest
+Use the :bro:see:`Broker::subscribe_to_logs` function to advertise interest
 in logs written by peers.  The topic names that Bro uses are implicitly of the
 form "bro/log/<stream-name>".
 
 .. btest-include:: ${DOC_ROOT}/frameworks/broker/logs-listener.bro
 
 To send remote logs either redef :bro:see:`Log::enable_remote_logging` or
-use the :bro:see:`BrokerComm::enable_remote_logs` function.  The former
+use the :bro:see:`Broker::enable_remote_logs` function.  The former
 allows any log stream to be sent to peers while the latter enables remote
 logging for particular streams.
 
@@ -137,24 +137,24 @@ Tuning Access Control
 By default, endpoints do not restrict the message topics that it sends
 to peers and do not restrict what message topics and data store
 identifiers get advertised to peers.  These are the default
-:bro:see:`BrokerComm::EndpointFlags` supplied to :bro:see:`BrokerComm::enable`.
+:bro:see:`Broker::EndpointFlags` supplied to :bro:see:`Broker::enable`.
 
 If not using the ``auto_publish`` flag, one can use the
-:bro:see:`BrokerComm::publish_topic` and :bro:see:`BrokerComm::unpublish_topic`
+:bro:see:`Broker::publish_topic` and :bro:see:`Broker::unpublish_topic`
 functions to manipulate the set of message topics (must match exactly)
 that are allowed to be sent to peer endpoints.  These settings take
 precedence over the per-message ``peers`` flag supplied to functions
-that take a :bro:see:`BrokerComm::SendFlags` such as :bro:see:`BrokerComm::print`,
-:bro:see:`BrokerComm::event`, :bro:see:`BrokerComm::auto_event` or
-:bro:see:`BrokerComm::enable_remote_logs`.
+that take a :bro:see:`Broker::SendFlags` such as :bro:see:`Broker::send_print`,
+:bro:see:`Broker::send_event`, :bro:see:`Broker::auto_event` or
+:bro:see:`Broker::enable_remote_logs`.
 
 If not using the ``auto_advertise`` flag, one can use the
-:bro:see:`BrokerComm::advertise_topic` and
-:bro:see:`BrokerComm::unadvertise_topic` functions
+:bro:see:`Broker::advertise_topic` and
+:bro:see:`Broker::unadvertise_topic` functions
 to manipulate the set of topic prefixes that are allowed to be
 advertised to peers.  If an endpoint does not advertise a topic prefix, then
 the only way peers can send messages to it is via the ``unsolicited``
-flag of :bro:see:`BrokerComm::SendFlags` and choosing a topic with a matching
+flag of :bro:see:`Broker::SendFlags` and choosing a topic with a matching
 prefix (i.e. full topic may be longer than receivers prefix, just the
 prefix needs to match).
 
@@ -192,8 +192,8 @@ last modification time.
 .. btest-include:: ${DOC_ROOT}/frameworks/broker/stores-connector.bro
 
 In the above example, if a local copy of the store contents isn't
-needed, just replace the :bro:see:`BrokerStore::create_clone` call with
-:bro:see:`BrokerStore::create_frontend`.  Queries will then be made against
+needed, just replace the :bro:see:`Broker::create_clone` call with
+:bro:see:`Broker::create_frontend`.  Queries will then be made against
 the remote master store instead of the local clone.
 
 Note that all data store queries must be made within Bro's asynchronous

doc/frameworks/broker/connecting-connector.bro

@@ -1,18 +1,18 @@
 const broker_port: port = 9999/tcp &redef;
 redef exit_only_after_terminate = T;
-redef BrokerComm::endpoint_name = "connector";
+redef Broker::endpoint_name = "connector";
 
 event bro_init()
 	{
-	BrokerComm::enable();
-	BrokerComm::connect("127.0.0.1", broker_port, 1sec);
+	Broker::enable();
+	Broker::connect("127.0.0.1", broker_port, 1sec);
 	}
 
-event BrokerComm::outgoing_connection_established(peer_address: string,
+event Broker::outgoing_connection_established(peer_address: string,
                                             peer_port: port,
                                             peer_name: string)
 	{
-	print "BrokerComm::outgoing_connection_established",
+	print "Broker::outgoing_connection_established",
 		  peer_address, peer_port, peer_name;
 	terminate();
 	}

doc/frameworks/broker/connecting-listener.bro

@@ -1,20 +1,20 @@
 const broker_port: port = 9999/tcp &redef;
 redef exit_only_after_terminate = T;
-redef BrokerComm::endpoint_name = "listener";
+redef Broker::endpoint_name = "listener";
 
 event bro_init()
 	{
-	BrokerComm::enable();
-	BrokerComm::listen(broker_port, "127.0.0.1");
+	Broker::enable();
+	Broker::listen(broker_port, "127.0.0.1");
 	}
 
-event BrokerComm::incoming_connection_established(peer_name: string)
+event Broker::incoming_connection_established(peer_name: string)
 	{
-	print "BrokerComm::incoming_connection_established", peer_name;
+	print "Broker::incoming_connection_established", peer_name;
 	}
 
-event BrokerComm::incoming_connection_broken(peer_name: string)
+event Broker::incoming_connection_broken(peer_name: string)
 	{
-	print "BrokerComm::incoming_connection_broken", peer_name;
+	print "Broker::incoming_connection_broken", peer_name;
 	terminate();
 	}

doc/frameworks/broker/events-connector.bro

@@ -1,30 +1,30 @@
 const broker_port: port = 9999/tcp &redef;
 redef exit_only_after_terminate = T;
-redef BrokerComm::endpoint_name = "connector";
+redef Broker::endpoint_name = "connector";
 global my_event: event(msg: string, c: count);
 global my_auto_event: event(msg: string, c: count);
 
 event bro_init()
 	{
-	BrokerComm::enable();
-	BrokerComm::connect("127.0.0.1", broker_port, 1sec);
-	BrokerComm::auto_event("bro/event/my_auto_event", my_auto_event);
+	Broker::enable();
+	Broker::connect("127.0.0.1", broker_port, 1sec);
+	Broker::auto_event("bro/event/my_auto_event", my_auto_event);
 	}
 
-event BrokerComm::outgoing_connection_established(peer_address: string,
+event Broker::outgoing_connection_established(peer_address: string,
                                             peer_port: port,
                                             peer_name: string)
 	{
-	print "BrokerComm::outgoing_connection_established",
+	print "Broker::outgoing_connection_established",
 	      peer_address, peer_port, peer_name;
-	BrokerComm::event("bro/event/my_event", BrokerComm::event_args(my_event, "hi", 0));
+	Broker::send_event("bro/event/my_event", Broker::event_args(my_event, "hi", 0));
 	event my_auto_event("stuff", 88);
-	BrokerComm::event("bro/event/my_event", BrokerComm::event_args(my_event, "...", 1));
+	Broker::send_event("bro/event/my_event", Broker::event_args(my_event, "...", 1));
 	event my_auto_event("more stuff", 51);
-	BrokerComm::event("bro/event/my_event", BrokerComm::event_args(my_event, "bye", 2));
+	Broker::send_event("bro/event/my_event", Broker::event_args(my_event, "bye", 2));
 	}
 
-event BrokerComm::outgoing_connection_broken(peer_address: string,
+event Broker::outgoing_connection_broken(peer_address: string,
                                        peer_port: port)
 	{
 	terminate();

doc/frameworks/broker/events-listener.bro

@@ -1,20 +1,20 @@
 const broker_port: port = 9999/tcp &redef;
 redef exit_only_after_terminate = T;
-redef BrokerComm::endpoint_name = "listener";
+redef Broker::endpoint_name = "listener";
 global msg_count = 0;
 global my_event: event(msg: string, c: count);
 global my_auto_event: event(msg: string, c: count);
 
 event bro_init()
 	{
-	BrokerComm::enable();
-	BrokerComm::subscribe_to_events("bro/event/");
-	BrokerComm::listen(broker_port, "127.0.0.1");
+	Broker::enable();
+	Broker::subscribe_to_events("bro/event/");
+	Broker::listen(broker_port, "127.0.0.1");
 	}
 
-event BrokerComm::incoming_connection_established(peer_name: string)
+event Broker::incoming_connection_established(peer_name: string)
 	{
-	print "BrokerComm::incoming_connection_established", peer_name;
+	print "Broker::incoming_connection_established", peer_name;
 	}
 
 event my_event(msg: string, c: count)

doc/frameworks/broker/logs-connector.bro

@@ -2,16 +2,16 @@
 
 const broker_port: port = 9999/tcp &redef;
 redef exit_only_after_terminate = T;
-redef BrokerComm::endpoint_name = "connector";
+redef Broker::endpoint_name = "connector";
 redef Log::enable_local_logging = F;
 redef Log::enable_remote_logging = F;
 global n = 0;
 
 event bro_init()
 	{
-	BrokerComm::enable();
-	BrokerComm::enable_remote_logs(Test::LOG);
-	BrokerComm::connect("127.0.0.1", broker_port, 1sec);
+	Broker::enable();
+	Broker::enable_remote_logs(Test::LOG);
+	Broker::connect("127.0.0.1", broker_port, 1sec);
 	}
 
 event do_write()
@@ -24,16 +24,16 @@ event do_write()
 	event do_write();
 	}
 
-event BrokerComm::outgoing_connection_established(peer_address: string,
+event Broker::outgoing_connection_established(peer_address: string,
                                             peer_port: port,
                                             peer_name: string)
 	{
-	print "BrokerComm::outgoing_connection_established",
+	print "Broker::outgoing_connection_established",
 	      peer_address, peer_port, peer_name;
 	event do_write();
 	}
 
-event BrokerComm::outgoing_connection_broken(peer_address: string,
+event Broker::outgoing_connection_broken(peer_address: string,
                                        peer_port: port)
 	{
 	terminate();

doc/frameworks/broker/logs-listener.bro

@@ -2,18 +2,18 @@
 
 const broker_port: port = 9999/tcp &redef;
 redef exit_only_after_terminate = T;
-redef BrokerComm::endpoint_name = "listener";
+redef Broker::endpoint_name = "listener";
 
 event bro_init()
 	{
-	BrokerComm::enable();
-	BrokerComm::subscribe_to_logs("bro/log/Test::LOG");
-	BrokerComm::listen(broker_port, "127.0.0.1");
+	Broker::enable();
+	Broker::subscribe_to_logs("bro/log/Test::LOG");
+	Broker::listen(broker_port, "127.0.0.1");
 	}
 
-event BrokerComm::incoming_connection_established(peer_name: string)
+event Broker::incoming_connection_established(peer_name: string)
 	{
-	print "BrokerComm::incoming_connection_established", peer_name;
+	print "Broker::incoming_connection_established", peer_name;
 	}
 
 event Test::log_test(rec: Test::Info)

doc/frameworks/broker/printing-connector.bro

@@ -1,25 +1,25 @@
 const broker_port: port = 9999/tcp &redef;
 redef exit_only_after_terminate = T;
-redef BrokerComm::endpoint_name = "connector";
+redef Broker::endpoint_name = "connector";
 
 event bro_init()
 	{
-	BrokerComm::enable();
-	BrokerComm::connect("127.0.0.1", broker_port, 1sec);
+	Broker::enable();
+	Broker::connect("127.0.0.1", broker_port, 1sec);
 	}
 
-event BrokerComm::outgoing_connection_established(peer_address: string,
+event Broker::outgoing_connection_established(peer_address: string,
                                             peer_port: port,
                                             peer_name: string)
 	{
-	print "BrokerComm::outgoing_connection_established",
+	print "Broker::outgoing_connection_established",
 	      peer_address, peer_port, peer_name;
-	BrokerComm::print("bro/print/hi", "hello");
-	BrokerComm::print("bro/print/stuff", "...");
-	BrokerComm::print("bro/print/bye", "goodbye");
+	Broker::send_print("bro/print/hi", "hello");
+	Broker::send_print("bro/print/stuff", "...");
+	Broker::send_print("bro/print/bye", "goodbye");
 	}
 
-event BrokerComm::outgoing_connection_broken(peer_address: string,
+event Broker::outgoing_connection_broken(peer_address: string,
                                        peer_port: port)
 	{
 	terminate();

doc/frameworks/broker/printing-listener.bro

@@ -1,21 +1,21 @@
 const broker_port: port = 9999/tcp &redef;
 redef exit_only_after_terminate = T;
-redef BrokerComm::endpoint_name = "listener";
+redef Broker::endpoint_name = "listener";
 global msg_count = 0;
 
 event bro_init()
 	{
-	BrokerComm::enable();
-	BrokerComm::subscribe_to_prints("bro/print/");
-	BrokerComm::listen(broker_port, "127.0.0.1");
+	Broker::enable();
+	Broker::subscribe_to_prints("bro/print/");
+	Broker::listen(broker_port, "127.0.0.1");
 	}
 
-event BrokerComm::incoming_connection_established(peer_name: string)
+event Broker::incoming_connection_established(peer_name: string)
 	{
-	print "BrokerComm::incoming_connection_established", peer_name;
+	print "Broker::incoming_connection_established", peer_name;
 	}
 
-event BrokerComm::print_handler(msg: string)
+event Broker::print_handler(msg: string)
 	{
 	++msg_count;
 	print "got print message", msg;

doc/frameworks/broker/stores-connector.bro

@@ -1,42 +1,42 @@
 const broker_port: port = 9999/tcp &redef;
 redef exit_only_after_terminate = T;
 
-global h: opaque of BrokerStore::Handle;
+global h: opaque of Broker::Handle;
 
-function dv(d: BrokerComm::Data): BrokerComm::DataVector
+function dv(d: Broker::Data): Broker::DataVector
 	{
-	local rval: BrokerComm::DataVector;
+	local rval: Broker::DataVector;
 	rval[0] = d;
 	return rval;
 	}
 
 global ready: event();
 
-event BrokerComm::outgoing_connection_broken(peer_address: string,
+event Broker::outgoing_connection_broken(peer_address: string,
                                        peer_port: port)
 	{
 	terminate();
 	}
 
-event BrokerComm::outgoing_connection_established(peer_address: string,
+event Broker::outgoing_connection_established(peer_address: string,
                                             peer_port: port,
                                             peer_name: string)
 	{
 	local myset: set[string] = {"a", "b", "c"};
 	local myvec: vector of string = {"alpha", "beta", "gamma"};
-	h = BrokerStore::create_master("mystore");
-	BrokerStore::insert(h, BrokerComm::data("one"), BrokerComm::data(110));
-	BrokerStore::insert(h, BrokerComm::data("two"), BrokerComm::data(223));
-	BrokerStore::insert(h, BrokerComm::data("myset"), BrokerComm::data(myset));
-	BrokerStore::insert(h, BrokerComm::data("myvec"), BrokerComm::data(myvec));
-	BrokerStore::increment(h, BrokerComm::data("one"));
-	BrokerStore::decrement(h, BrokerComm::data("two"));
-	BrokerStore::add_to_set(h, BrokerComm::data("myset"), BrokerComm::data("d"));
-	BrokerStore::remove_from_set(h, BrokerComm::data("myset"), BrokerComm::data("b"));
-	BrokerStore::push_left(h, BrokerComm::data("myvec"), dv(BrokerComm::data("delta")));
-	BrokerStore::push_right(h, BrokerComm::data("myvec"), dv(BrokerComm::data("omega")));
+	h = Broker::create_master("mystore");
+	Broker::insert(h, Broker::data("one"), Broker::data(110));
+	Broker::insert(h, Broker::data("two"), Broker::data(223));
+	Broker::insert(h, Broker::data("myset"), Broker::data(myset));
+	Broker::insert(h, Broker::data("myvec"), Broker::data(myvec));
+	Broker::increment(h, Broker::data("one"));
+	Broker::decrement(h, Broker::data("two"));
+	Broker::add_to_set(h, Broker::data("myset"), Broker::data("d"));
+	Broker::remove_from_set(h, Broker::data("myset"), Broker::data("b"));
+	Broker::push_left(h, Broker::data("myvec"), dv(Broker::data("delta")));
+	Broker::push_right(h, Broker::data("myvec"), dv(Broker::data("omega")));
 
-	when ( local res = BrokerStore::size(h) )
+	when ( local res = Broker::size(h) )
 		{
 		print "master size", res;
 		event ready();
@@ -47,7 +47,7 @@ event BrokerComm::outgoing_connection_established(peer_address: string,
 
 event bro_init()
 	{
-	BrokerComm::enable();
-	BrokerComm::connect("127.0.0.1", broker_port, 1secs);
-	BrokerComm::auto_event("bro/event/ready", ready);
+	Broker::enable();
+	Broker::connect("127.0.0.1", broker_port, 1secs);
+	Broker::auto_event("bro/event/ready", ready);
 	}

doc/frameworks/broker/stores-listener.bro

@@ -1,13 +1,13 @@
 const broker_port: port = 9999/tcp &redef;
 redef exit_only_after_terminate = T;
 
-global h: opaque of BrokerStore::Handle;
+global h: opaque of Broker::Handle;
 global expected_key_count = 4;
 global key_count = 0;
 
 function do_lookup(key: string)
 	{
-	when ( local res = BrokerStore::lookup(h, BrokerComm::data(key)) )
+	when ( local res = Broker::lookup(h, Broker::data(key)) )
 		{
 		++key_count;
 		print "lookup", key, res;
@@ -21,15 +21,15 @@ function do_lookup(key: string)
 
 event ready()
 	{
-	h = BrokerStore::create_clone("mystore");
+	h = Broker::create_clone("mystore");
 
-	when ( local res = BrokerStore::keys(h) )
+	when ( local res = Broker::keys(h) )
 		{
 		print "clone keys", res;
-		do_lookup(BrokerComm::refine_to_string(BrokerComm::vector_lookup(res$result, 0)));
-		do_lookup(BrokerComm::refine_to_string(BrokerComm::vector_lookup(res$result, 1)));
-		do_lookup(BrokerComm::refine_to_string(BrokerComm::vector_lookup(res$result, 2)));
-		do_lookup(BrokerComm::refine_to_string(BrokerComm::vector_lookup(res$result, 3)));
+		do_lookup(Broker::refine_to_string(Broker::vector_lookup(res$result, 0)));
+		do_lookup(Broker::refine_to_string(Broker::vector_lookup(res$result, 1)));
+		do_lookup(Broker::refine_to_string(Broker::vector_lookup(res$result, 2)));
+		do_lookup(Broker::refine_to_string(Broker::vector_lookup(res$result, 3)));
 		}
 	timeout 10sec
 		{ print "timeout"; }
@@ -37,7 +37,7 @@ event ready()
 
 event bro_init()
 	{
-	BrokerComm::enable();
-	BrokerComm::subscribe_to_events("bro/event/ready");
-	BrokerComm::listen(broker_port, "127.0.0.1");
+	Broker::enable();
+	Broker::subscribe_to_events("bro/event/ready");
+	Broker::listen(broker_port, "127.0.0.1");
 	}

doc/frameworks/broker/testlog.bro

@@ -13,6 +13,6 @@ export {
 
 event bro_init() &priority=5
 	{
-	BrokerComm::enable();
+	Broker::enable();
 	Log::create_stream(Test::LOG, [$columns=Test::Info, $ev=log_test, $path="test"]);
 	}

doc/scripting/index.rst

@@ -776,7 +776,7 @@ string against which it will be tested to be on the right.
 In the sample above, two local variables are declared to hold our
 sample sentence and regular expression.  Our regular expression in
 this case will return true if the string contains either the word
-``quick`` or the word ``fox``. The ``if`` statement in the script uses
+``quick`` or the word ``lazy``. The ``if`` statement in the script uses
 embedded matching and the ``in`` operator to check for the existence
 of the pattern within the string.  If the statement resolves to true,
 :bro:id:`split` is called to break the string into separate pieces.

scripts/base/frameworks/broker/__load__.bro

@@ -1 +1,2 @@
 @load ./main
+@load ./store

scripts/base/frameworks/broker/main.bro

@@ -1,11 +1,20 @@
 ##! Various data structure definitions for use with Bro's communication system.
 
-module BrokerComm;
+module Log;
+
+export {
+    type Log::ID: enum {
+        ## Dummy place-holder.
+        UNKNOWN
+    };
+}
+
+module Broker;
 
 export {
 
 	## A name used to identify this endpoint to peers.
-	## .. bro:see:: BrokerComm::connect BrokerComm::listen
+	## .. bro:see:: Broker::connect Broker::listen
 	const endpoint_name = "" &redef;
 
 	## Change communication behavior.
@@ -32,11 +41,11 @@ export {
 
 	## Opaque communication data.
 	type Data: record {
-		d: opaque of BrokerComm::Data &optional;
+		d: opaque of Broker::Data &optional;
 	};
 
 	## Opaque communication data.
-	type DataVector: vector of BrokerComm::Data;
+	type DataVector: vector of Broker::Data;
 
 	## Opaque event communication data.
 	type EventArgs: record {
@@ -49,55 +58,315 @@ export {
 	## Opaque communication data used as a convenient way to wrap key-value
 	## pairs that comprise table entries.
 	type TableItem : record {
-		key: BrokerComm::Data;
-		val: BrokerComm::Data;
+		key: Broker::Data;
+		val: Broker::Data;
 	};
+
+	## Enable use of communication.
+	##
+	## flags: used to tune the local Broker endpoint behavior.
+	##
+	## Returns: true if communication is successfully initialized.
+	global enable: function(flags: EndpointFlags &default = EndpointFlags()): bool;
+
+	## Changes endpoint flags originally supplied to :bro:see:`Broker::enable`.
+	##
+	## flags: the new endpoint behavior flags to use.
+	##
+	## Returns: true if flags were changed.
+	global set_endpoint_flags: function(flags: EndpointFlags &default = EndpointFlags()): bool;
+
+	## Allow sending messages to peers if associated with the given topic.
+	## This has no effect if auto publication behavior is enabled via the flags
+	## supplied to :bro:see:`Broker::enable` or :bro:see:`Broker::set_endpoint_flags`.
+	##
+	## topic: a topic to allow messages to be published under.
+	##
+	## Returns: true if successful.
+	global publish_topic: function(topic: string): bool;
+
+	## Disallow sending messages to peers if associated with the given topic.
+	## This has no effect if auto publication behavior is enabled via the flags
+	## supplied to :bro:see:`Broker::enable` or :bro:see:`Broker::set_endpoint_flags`.
+	##
+	## topic: a topic to disallow messages to be published under.
+	##
+	## Returns: true if successful.
+	global unpublish_topic: function(topic: string): bool;
+
+	## Listen for remote connections.
+	##
+	## p: the TCP port to listen on.
+	##
+	## a: an address string on which to accept connections, e.g.
+	##    "127.0.0.1".  An empty string refers to @p INADDR_ANY.
+	##
+	## reuse: equivalent to behavior of SO_REUSEADDR.
+	##
+	## Returns: true if the local endpoint is now listening for connections.
+	##
+	## .. bro:see:: Broker::incoming_connection_established
+	global listen: function(p: port, a: string &default = "", reuse: bool &default = T): bool;
+
+	## Initiate a remote connection.
+	##
+	## a: an address to connect to, e.g. "localhost" or "127.0.0.1".
+	##
+	## p: the TCP port on which the remote side is listening.
+	##
+	## retry: an interval at which to retry establishing the
+	##        connection with the remote peer if it cannot be made initially, or
+	##        if it ever becomes disconnected.
+	##
+	## Returns: true if it's possible to try connecting with the peer and
+	##          it's a new peer.  The actual connection may not be established
+	##          until a later point in time.
+	##
+	## .. bro:see:: Broker::outgoing_connection_established
+	global connect: function(a: string, p: port, retry: interval): bool;
+
+	## Remove a remote connection.
+	##
+	## a: the address used in previous successful call to :bro:see:`Broker::connect`.
+	##
+	## p: the port used in previous successful call to :bro:see:`Broker::connect`.
+	##
+	## Returns: true if the arguments match a previously successful call to
+	##          :bro:see:`Broker::connect`.
+	global disconnect: function(a: string, p: port): bool;
+
+	## Print a simple message to any interested peers.  The receiver can use
+	## :bro:see:`Broker::print_handler` to handle messages.
+	##
+	## topic: a topic associated with the printed message.
+	##
+	## msg: the print message to send to peers.
+	##
+	## flags: tune the behavior of how the message is sent.
+	##
+	## Returns: true if the message is sent.
+	global send_print: function(topic: string, msg: string, flags: SendFlags &default = SendFlags()): bool;
+
+	## Register interest in all peer print messages that use a certain topic
+	## prefix. Use :bro:see:`Broker::print_handler` to handle received
+	## messages.
+	##
+	## topic_prefix: a prefix to match against remote message topics.
+	##               e.g. an empty prefix matches everything and "a" matches
+	##               "alice" and "amy" but not "bob".
+	##
+	## Returns: true if it's a new print subscription and it is now registered.
+	global subscribe_to_prints: function(topic_prefix: string): bool;
+
+	## Unregister interest in all peer print messages that use a topic prefix.
+	##
+	## topic_prefix: a prefix previously supplied to a successful call to
+	##               :bro:see:`Broker::subscribe_to_prints`.
+	##
+	## Returns: true if interest in the topic prefix is no longer advertised.
+	global unsubscribe_to_prints: function(topic_prefix: string): bool;
+
+	## Send an event to any interested peers.
+	##
+	## topic: a topic associated with the event message.
+	##
+	## args: event arguments as made by :bro:see:`Broker::event_args`.
+	##
+	## flags: tune the behavior of how the message is sent.
+	##
+	## Returns: true if the message is sent.
+	global send_event: function(topic: string, args: EventArgs, flags: SendFlags &default = SendFlags()): bool;
+
+	## Automatically send an event to any interested peers whenever it is
+	## locally dispatched (e.g. using "event my_event(...);" in a script).
+	##
+	## topic: a topic string associated with the event message.
+	##        Peers advertise interest by registering a subscription to some
+	##        prefix of this topic name.
+	##
+	## ev: a Bro event value.
+	##
+	## flags: tune the behavior of how the message is sent.
+	##
+	## Returns: true if automatic event sending is now enabled.
+	global auto_event: function(topic: string, ev: any, flags: SendFlags &default = SendFlags()): bool;
+
+	## Stop automatically sending an event to peers upon local dispatch.
+	##
+	## topic: a topic originally given to :bro:see:`Broker::auto_event`.
+	##
+	## ev: an event originally given to :bro:see:`Broker::auto_event`.
+	##
+	## Returns: true if automatic events will not occur for the topic/event
+	##          pair.
+	global auto_event_stop: function(topic: string, ev: any): bool;
+
+	## Register interest in all peer event messages that use a certain topic
+	## prefix.
+	##
+	## topic_prefix: a prefix to match against remote message topics.
+	##               e.g. an empty prefix matches everything and "a" matches
+	##               "alice" and "amy" but not "bob".
+	##
+	## Returns: true if it's a new event subscription and it is now registered.
+	global subscribe_to_events: function(topic_prefix: string): bool;
+
+	## Unregister interest in all peer event messages that use a topic prefix.
+	##
+	## topic_prefix: a prefix previously supplied to a successful call to
+	##               :bro:see:`Broker::subscribe_to_events`.
+	##
+	## Returns: true if interest in the topic prefix is no longer advertised.
+	global unsubscribe_to_events: function(topic_prefix: string): bool;
+
+	## Enable remote logs for a given log stream.
+	##
+	## id: the log stream to enable remote logs for.
+	##
+	## flags: tune the behavior of how log entry messages are sent.
+	##
+	## Returns: true if remote logs are enabled for the stream.
+	global enable_remote_logs: function(id: Log::ID, flags: SendFlags &default = SendFlags()): bool;
+
+	## Disable remote logs for a given log stream.
+	##
+	## id: the log stream to disable remote logs for.
+	##
+	## Returns: true if remote logs are disabled for the stream.
+	global disable_remote_logs: function(id: Log::ID): bool;
+
+	## Check if remote logs are enabled for a given log stream.
+	##
+	## id: the log stream to check.
+	##
+	## Returns: true if remote logs are enabled for the given stream.
+	global remote_logs_enabled: function(id: Log::ID): bool;
+
+	## Register interest in all peer log messages that use a certain topic
+	## prefix. Logs are implicitly sent with topic "bro/log/<stream-name>" and
+	## the receiving side processes them through the logging framework as usual.
+	##
+	## topic_prefix: a prefix to match against remote message topics.
+	##               e.g. an empty prefix matches everything and "a" matches
+	##               "alice" and "amy" but not "bob".
+	##
+	## Returns: true if it's a new log subscription and it is now registered.
+	global subscribe_to_logs: function(topic_prefix: string): bool;
+
+	## Unregister interest in all peer log messages that use a topic prefix.
+	## Logs are implicitly sent with topic "bro/log/<stream-name>" and the
+	## receiving side processes them through the logging framework as usual.
+	##
+	## topic_prefix: a prefix previously supplied to a successful call to
+	##               :bro:see:`Broker::subscribe_to_logs`.
+	##
+	## Returns: true if interest in the topic prefix is no longer advertised.
+	global unsubscribe_to_logs: function(topic_prefix: string): bool;
+
 }
 
-module BrokerStore;
+@load base/bif/comm.bif
+@load base/bif/messaging.bif
 
-export {
+module Broker;
 
-	## Whether a data store query could be completed or not.
-	type QueryStatus: enum {
-		SUCCESS,
-		FAILURE,
-	};
-
-	## An expiry time for a key-value pair inserted in to a data store.
-	type ExpiryTime: record {
-		## Absolute point in time at which to expire the entry.
-		absolute: time &optional;
-		## A point in time relative to the last modification time at which
-		## to expire the entry.  New modifications will delay the expiration.
-		since_last_modification: interval &optional;
-	};
-
-	## The result of a data store query.
-	type QueryResult: record {
-		## Whether the query completed or not.
-		status: BrokerStore::QueryStatus;
-		## The result of the query.  Certain queries may use a particular
-		## data type (e.g. querying store size always returns a count, but
-		## a lookup may return various data types).
-		result: BrokerComm::Data;
-	};
-
-	## Options to tune the SQLite storage backend.
-	type SQLiteOptions: record {
-		## File system path of the database.
-		path: string &default = "store.sqlite";
-	};
-
-	## Options to tune the RocksDB storage backend.
-	type RocksDBOptions: record {
-		## File system path of the database.
-		path: string &default = "store.rocksdb";
-	};
-
-	## Options to tune the particular storage backends.
-	type BackendOptions: record {
-		sqlite: SQLiteOptions &default = SQLiteOptions();
-		rocksdb: RocksDBOptions &default = RocksDBOptions();
-	};
+function enable(flags: EndpointFlags &default = EndpointFlags()) : bool
+    {
+    return __enable(flags);
     }
+
+function set_endpoint_flags(flags: EndpointFlags &default = EndpointFlags()): bool
+    {
+    return __set_endpoint_flags(flags);
+    }
+
+function publish_topic(topic: string): bool
+    {
+    return __publish_topic(topic);
+    }
+
+function unpublish_topic(topic: string): bool
+    {
+    return __unpublish_topic(topic);
+    }
+
+function listen(p: port, a: string &default = "", reuse: bool &default = T): bool
+    {
+    return __listen(p, a, reuse);
+    }
+
+function connect(a: string, p: port, retry: interval): bool
+    {
+    return __connect(a, p, retry);
+    }
+
+function disconnect(a: string, p: port): bool
+    {
+    return __disconnect(a, p);
+    }
+
+function send_print(topic: string, msg: string, flags: SendFlags &default = SendFlags()): bool
+    {
+    return __send_print(topic, msg, flags);
+    }
+
+function subscribe_to_prints(topic_prefix: string): bool
+    {
+    return __subscribe_to_prints(topic_prefix);
+    }
+
+function unsubscribe_to_prints(topic_prefix: string): bool
+    {
+    return __unsubscribe_to_prints(topic_prefix);
+    }
+
+function send_event(topic: string, args: EventArgs, flags: SendFlags &default = SendFlags()): bool
+    {
+    return __event(topic, args, flags);
+    }
+
+function auto_event(topic: string, ev: any, flags: SendFlags &default = SendFlags()): bool
+    {
+    return __auto_event(topic, ev, flags);
+    }
+
+function auto_event_stop(topic: string, ev: any): bool
+    {
+    return __auto_event_stop(topic, ev);
+    }
+
+function subscribe_to_events(topic_prefix: string): bool
+    {
+    return __subscribe_to_events(topic_prefix);
+    }
+
+function unsubscribe_to_events(topic_prefix: string): bool
+    {
+    return __unsubscribe_to_events(topic_prefix);
+    }
+
+function enable_remote_logs(id: Log::ID, flags: SendFlags &default = SendFlags()): bool
+    {
+    return __enable_remote_logs(id, flags);
+    }
+
+function disable_remote_logs(id: Log::ID): bool
+    {
+    return __disable_remote_logs(id);
+    }
+
+function remote_logs_enabled(id: Log::ID): bool
+    {
+    return __remote_logs_enabled(id);
+    }
+
+function subscribe_to_logs(topic_prefix: string): bool
+    {
+    return __subscribe_to_logs(topic_prefix);
+    }
+
+function unsubscribe_to_logs(topic_prefix: string): bool
+    {
+    return __unsubscribe_to_logs(topic_prefix);
+    }
+

scripts/base/frameworks/files/magic/audio.sig

@@ -2,7 +2,7 @@
 # MPEG v3 audio
 signature file-mpeg-audio {
 	file-mime "audio/mpeg", 20
-	file-magic /^\xff[\xe2\xe3\xf2\xf3\xf6\xf7\xfa\xfb\xfc\xfd]/
+	file-magic /^(ID3|\xff[\xe2\xe3\xf2\xf3\xf6\xf7\xfa\xfb\xfc\xfd])/
 }
 
 # MPEG v4 audio

scripts/base/frameworks/files/magic/general.sig

@@ -9,53 +9,53 @@ signature file-plaintext {
 
 signature file-json {
 	file-mime "text/json", 1
-	file-magic /^(\xef\xbb\xbf)?[\x0d\x0a[:blank:]]*\{[\x0d\x0a[:blank:]]*(["][^"]{1,}["]|[a-zA-Z][a-zA-Z0-9\\_]*)[\x0d\x0a[:blank:]]*:[\x0d\x0a[:blank:]]*(["]|\[|\{|[0-9]|true|false)/
+	file-magic /^(\xef\xbb\xbf|\xff\xfe|\xfe\xff)?[\x0d\x0a[:blank:]]*\{[\x0d\x0a[:blank:]]*(["][^"]{1,}["]|[a-zA-Z][a-zA-Z0-9\\_]*)[\x0d\x0a[:blank:]]*:[\x0d\x0a[:blank:]]*(["]|\[|\{|[0-9]|true|false)/
 }
 
 signature file-json2 {
 	file-mime "text/json", 1
-	file-magic /^(\xef\xbb\xbf)?[\x0d\x0a[:blank:]]*\[[\x0d\x0a[:blank:]]*(((["][^"]{1,}["]|[0-9]{1,}(\.[0-9]{1,})?|true|false)[\x0d\x0a[:blank:]]*,)|\{|\[)[\x0d\x0a[:blank:]]*/
+	file-magic /^(\xef\xbb\xbf|\xff\xfe|\xfe\xff)?[\x0d\x0a[:blank:]]*\[[\x0d\x0a[:blank:]]*(((["][^"]{1,}["]|[0-9]{1,}(\.[0-9]{1,})?|true|false)[\x0d\x0a[:blank:]]*,)|\{|\[)[\x0d\x0a[:blank:]]*/
 }
 
 # Match empty JSON documents.
 signature file-json3 {
 	file-mime "text/json", 0
-	file-magic /^(\xef\xbb\xbf)?[\x0d\x0a[:blank:]]*(\[\]|\{\})[\x0d\x0a[:blank:]]*$/
+	file-magic /^(\xef\xbb\xbf|\xff\xfe|\xfe\xff)?[\x0d\x0a[:blank:]]*(\[\]|\{\})[\x0d\x0a[:blank:]]*$/
 }
 
 signature file-xml {
 	file-mime "application/xml", 10
-	file-magic /^(\xef\xbb\xbf)?([\x0d\x0a[:blank:]]*(<!--.*-->)?[\x0d\x0a[:blank:]]*)*<\?xml /
+	file-magic /^(\xef\xbb\xbf|\xff\xfe|\xfe\xff)?([\x0d\x0a[:blank:]]*(<!--.*-->)?[\x0d\x0a[:blank:]]*)*\x00?<\x00?\?\x00?x\x00?m\x00?l\x00? \x00?/
 }
 
 signature file-xhtml {
 	file-mime "text/html", 100
-	file-magic /^(\xef\xbb\xbf)?([\x0d\x0a[:blank:]]*(<!--.*-->)?[\x0d\x0a[:blank:]]*)*(<\?xml .*\?>)?([\x0d\x0a[:blank:]]*(<!--.*-->)?[\x0d\x0a[:blank:]]*)*<(![dD][oO][cC][tT][yY][pP][eE] {1,}[hH][tT][mM][lL]|[hH][tT][mM][lL]|[mM][eE][tT][aA] {1,}[hH][tT][tT][pP]-[eE][qQ][uU][iI][vV])/
+	file-magic /^(\xef\xbb\xbf|\xff\xfe|\xfe\xff)?([\x0d\x0a[:blank:]]*(<!--.*-->)?[\x0d\x0a[:blank:]]*)*(<\?xml .*\?>)?([\x0d\x0a[:blank:]]*(<!--.*-->)?[\x0d\x0a[:blank:]]*)*<(![dD][oO][cC][tT][yY][pP][eE] {1,}[hH][tT][mM][lL]|[hH][tT][mM][lL]|[mM][eE][tT][aA] {1,}[hH][tT][tT][pP]-[eE][qQ][uU][iI][vV])/
 }
 
 signature file-html {
 	file-mime "text/html", 49
-	file-magic /^(\xef\xbb\xbf)?([\x0d\x0a[:blank:]]*(<!--.*-->)?[\x0d\x0a[:blank:]]*)*(<\?xml .*\?>)?([\x0d\x0a[:blank:]]*(<!--.*-->)?[\x0d\x0a[:blank:]]*)*<![dD][oO][cC][tT][yY][pP][eE] {1,}[hH][tT][mM][lL]/
+	file-magic /^(\xef\xbb\xbf|\xff\xfe|\xfe\xff)?([\x0d\x0a[:blank:]]*(<!--.*-->)?[\x0d\x0a[:blank:]]*)*(<\?xml .*\?>)?([\x0d\x0a[:blank:]]*(<!--.*-->)?[\x0d\x0a[:blank:]]*)*<![dD][oO][cC][tT][yY][pP][eE] {1,}[hH][tT][mM][lL]/
 }
 
 signature file-html2 {
 	file-mime "text/html", 20
-	file-magic /^(\xef\xbb\xbf)?([\x0d\x0a[:blank:]]*(<!--.*-->)?[\x0d\x0a[:blank:]]*)*(<\?xml .*\?>)?([\x0d\x0a[:blank:]]*(<!--.*-->)?[\x0d\x0a[:blank:]]*)*<([hH][eE][aA][dD]|[hH][tT][mM][lL]|[tT][iI][tT][lL][eE]|[bB][oO][dD][yY])/
+	file-magic /^(\xef\xbb\xbf|\xff\xfe|\xfe\xff)?([\x0d\x0a[:blank:]]*(<!--.*-->)?[\x0d\x0a[:blank:]]*)*(<\?xml .*\?>)?([\x0d\x0a[:blank:]]*(<!--.*-->)?[\x0d\x0a[:blank:]]*)*<([hH][eE][aA][dD]|[hH][tT][mM][lL]|[tT][iI][tT][lL][eE]|[bB][oO][dD][yY])/
 }
 
 signature file-rss {
 	file-mime "text/rss", 90
-	file-magic /^(\xef\xbb\xbf)?([\x0d\x0a[:blank:]]*(<!--.*-->)?[\x0d\x0a[:blank:]]*)*(<\?xml .*\?>)?([\x0d\x0a[:blank:]]*(<!--.*-->)?[\x0d\x0a[:blank:]]*)*<[rR][sS][sS]/
+	file-magic /^(\xef\xbb\xbf|\xff\xfe|\xfe\xff)?([\x0d\x0a[:blank:]]*(<!--.*-->)?[\x0d\x0a[:blank:]]*)*(<\?xml .*\?>)?([\x0d\x0a[:blank:]]*(<!--.*-->)?[\x0d\x0a[:blank:]]*)*<[rR][sS][sS]/
 }
 
 signature file-atom {
 	file-mime "text/atom", 100
-	file-magic /^(\xef\xbb\xbf)?([\x0d\x0a[:blank:]]*(<!--.*-->)?[\x0d\x0a[:blank:]]*)*(<\?xml .*\?>)?([\x0d\x0a[:blank:]]*(<!--.*-->)?[\x0d\x0a[:blank:]]*)*<([rR][sS][sS][^>]*xmlns:atom|[fF][eE][eE][dD][^>]*xmlns=["']?http:\/\/www.w3.org\/2005\/Atom["']?)/
+	file-magic /^(\xef\xbb\xbf|\xff\xfe|\xfe\xff)?([\x0d\x0a[:blank:]]*(<!--.*-->)?[\x0d\x0a[:blank:]]*)*(<\?xml .*\?>)?([\x0d\x0a[:blank:]]*(<!--.*-->)?[\x0d\x0a[:blank:]]*)*<([rR][sS][sS][^>]*xmlns:atom|[fF][eE][eE][dD][^>]*xmlns=["']?http:\/\/www.w3.org\/2005\/Atom["']?)/
 }
 
 signature file-soap {
 	file-mime "application/soap+xml", 49
-	file-magic /^(\xef\xbb\xbf)?([\x0d\x0a[:blank:]]*(<!--.*-->)?[\x0d\x0a[:blank:]]*)*(<\?xml .*\?>)?([\x0d\x0a[:blank:]]*(<!--.*-->)?[\x0d\x0a[:blank:]]*)*<[sS][oO][aA][pP](-[eE][nN][vV])?:[eE][nN][vV][eE][lL][oO][pP][eE]/
+	file-magic /^(\xef\xbb\xbf|\xff\xfe|\xfe\xff)?([\x0d\x0a[:blank:]]*(<!--.*-->)?[\x0d\x0a[:blank:]]*)*(<\?xml .*\?>)?([\x0d\x0a[:blank:]]*(<!--.*-->)?[\x0d\x0a[:blank:]]*)*<[sS][oO][aA][pP](-[eE][nN][vV])?:[eE][nN][vV][eE][lL][oO][pP][eE]/
 }
 
 signature file-cross-domain-policy {
@@ -70,7 +70,7 @@ signature file-cross-domain-policy2 {
 
 signature file-xmlrpc {
 	file-mime "application/xml-rpc", 49
-	file-magic /^(\xef\xbb\xbf)?([\x0d\x0a[:blank:]]*(<!--.*-->)?[\x0d\x0a[:blank:]]*)*(<\?xml .*\?>)?([\x0d\x0a[:blank:]]*(<!--.*-->)?[\x0d\x0a[:blank:]]*)*<[mM][eE][tT][hH][oO][dD][rR][eE][sS][pP][oO][nN][sS][eE]>/
+	file-magic /^(\xef\xbb\xbf|\xff\xfe|\xfe\xff)?([\x0d\x0a[:blank:]]*(<!--.*-->)?[\x0d\x0a[:blank:]]*)*(<\?xml .*\?>)?([\x0d\x0a[:blank:]]*(<!--.*-->)?[\x0d\x0a[:blank:]]*)*<[mM][eE][tT][hH][oO][dD][rR][eE][sS][pP][oO][nN][sS][eE]>/
 }
 
 signature file-coldfusion {
@@ -81,7 +81,13 @@ signature file-coldfusion {
 # Adobe Flash Media Manifest
 signature file-f4m {
 	file-mime "application/f4m", 49
-	file-magic /^(\xef\xbb\xbf)?([\x0d\x0a[:blank:]]*(<!--.*-->)?[\x0d\x0a[:blank:]]*)*(<\?xml .*\?>)?([\x0d\x0a[:blank:]]*(<!--.*-->)?[\x0d\x0a[:blank:]]*)*<[mM][aA][nN][iI][fF][eE][sS][tT][\x0d\x0a[:blank:]]{1,}xmlns=\"http:\/\/ns\.adobe\.com\/f4m\//
+	file-magic /^(\xef\xbb\xbf|\xff\xfe|\xfe\xff)?([\x0d\x0a[:blank:]]*(<!--.*-->)?[\x0d\x0a[:blank:]]*)*(<\?xml .*\?>)?([\x0d\x0a[:blank:]]*(<!--.*-->)?[\x0d\x0a[:blank:]]*)*<[mM][aA][nN][iI][fF][eE][sS][tT][\x0d\x0a[:blank:]]{1,}xmlns=\"http:\/\/ns\.adobe\.com\/f4m\//
+}
+
+# .ini style files
+signature file-ini {
+	file-mime "text/ini", 20
+	file-magic /^(\xef\xbb\xbf|\xff\xfe|\xfe\xff)?[\x00\x0d\x0a[:blank:]]*\[[^\x0d\x0a]+\][[:blank:]\x00]*[\x0d\x0a]/
 }
 
 # Microsoft LNK files
@@ -90,6 +96,41 @@ signature file-lnk {
 	file-magic /^\x4C\x00\x00\x00\x01\x14\x02\x00\x00\x00\x00\x00\xC0\x00\x00\x00\x00\x10\x00\x00\x00\x46/
 }
 
+# Microsoft Registry policies
+signature file-pol {
+	file-mime "application/vnd.ms-pol", 49
+	file-magic /^PReg/
+}
+
+# Old style Windows registry file
+signature file-reg {
+	file-mime "application/vnd.ms-reg", 49
+	file-magic /^REGEDIT4/
+}
+
+# Newer Windows registry file
+signature file-reg-utf16 {
+	file-mime "application/vnd.ms-reg", 49
+	file-magic /^\xFF\xFEW\x00i\x00n\x00d\x00o\x00w\x00s\x00 \x00R\x00e\x00g\x00i\x00s\x00t\x00r\x00y\x00 \x00E\x00d\x00i\x00t\x00o\x00r\x00 \x00V\x00e\x00r\x00s\x00i\x00o\x00n\x00 \x005\x00\.\x000\x000/
+}
+
+# Microsoft Registry format (typically DESKTOP.DAT)
+signature file-regf {
+	file-mime "application vnd.ms-regf", 49
+	file-magic /^\x72\x65\x67\x66/
+}
+
+# Microsoft Outlook PST files
+signature file-pst {
+	file-mime "application/vnd.ms-outlook", 49
+	file-magic /!BDN......[\x0e\x0f\x15\x17][\x00-\x02]/
+}
+
+signature file-afpinfo {
+	file-mime "application/vnd.apple-afpinfo"
+	file-magic /^AFP/
+}
+
 signature file-jar {
 	file-mime "application/java-archive", 100
 	file-magic /^PK\x03\x04.{1,200}\x14\x00..META-INF\/MANIFEST\.MF/

scripts/base/frameworks/intel/main.bro

@@ -91,9 +91,20 @@ export {
 		## connection record should go here to give context to the data.
 		conn:            connection    &optional;
 
+		## If the data was discovered within a connection, the
+		## connection uid should go here to give context to the data.
+		## If the *conn* field is provided, this will be automatically
+		## filled out.
+		uid:             string        &optional;
+
 		## If the data was discovered within a file, the file record
 		## should go here to provide context to the data.
 		f:               fa_file       &optional;
+
+		## If the data was discovered within a file, the file uid should
+		## go here to provide context to the data. If the *f* field is
+		## provided, this will be automatically filled out.
+		fuid:            string        &optional;
 	};
 
 	## Record used for the logging framework representing a positive
@@ -112,6 +123,7 @@ export {
 		## If a file was associated with this intelligence hit,
 		## this is the uid for the file.
 		fuid:           string   &log &optional;
+
 		## A mime type if the intelligence hit is related to a file.
 		## If the $f field is provided this will be automatically filled
 		## out.
@@ -283,15 +295,14 @@ event Intel::match(s: Seen, items: set[Item]) &priority=5
 
 	if ( s?$f )
 		{
+		s$fuid = s$f$id;
+
 		if ( s$f?$conns && |s$f$conns| == 1 )
 			{
 			for ( cid in s$f$conns )
 				s$conn = s$f$conns[cid];
 			}
 
-		if ( ! info?$fuid )
-			info$fuid = s$f$id;
-
 		if ( ! info?$file_mime_type && s$f?$info && s$f$info?$mime_type )
 			info$file_mime_type = s$f$info$mime_type;
 
@@ -299,12 +310,18 @@ event Intel::match(s: Seen, items: set[Item]) &priority=5
 			info$file_desc = Files::describe(s$f);
 		}
 
+	if ( s?$fuid )
+		info$fuid = s$fuid;
+
 	if ( s?$conn )
 		{
-		info$uid = s$conn$uid;
+		s$uid = s$conn$uid;
 		info$id  = s$conn$id;
 		}
 
+	if ( s?$uid )
+		info$uid = s$uid;
+
 	for ( item in items )
 		add info$sources[item$meta$source];
 

scripts/base/frameworks/netcontrol/plugins/acld.bro

@@ -241,7 +241,7 @@ function acld_add_rule_fun(p: PluginState, r: Rule) : bool
 	if ( ar$command == "" )
 		return F;
 
-	BrokerComm::event(p$acld_config$acld_topic, BrokerComm::event_args(acld_add_rule, p$acld_id, r, ar));
+	Broker::send_event(p$acld_config$acld_topic, Broker::event_args(acld_add_rule, p$acld_id, r, ar));
 	return T;
 	}
 
@@ -256,18 +256,18 @@ function acld_remove_rule_fun(p: PluginState, r: Rule) : bool
 	else
 		return F;
 
-	BrokerComm::event(p$acld_config$acld_topic, BrokerComm::event_args(acld_remove_rule, p$acld_id, r, ar));
+	Broker::send_event(p$acld_config$acld_topic, Broker::event_args(acld_remove_rule, p$acld_id, r, ar));
 	return T;
 	}
 
 function acld_init(p: PluginState)
 	{
-	BrokerComm::enable();
-	BrokerComm::connect(cat(p$acld_config$acld_host), p$acld_config$acld_port, 1sec);
-	BrokerComm::subscribe_to_events(p$acld_config$acld_topic);
+	Broker::enable();
+	Broker::connect(cat(p$acld_config$acld_host), p$acld_config$acld_port, 1sec);
+	Broker::subscribe_to_events(p$acld_config$acld_topic);
 	}
 
-event BrokerComm::outgoing_connection_established(peer_address: string, peer_port: port, peer_name: string)
+event Broker::outgoing_connection_established(peer_address: string, peer_port: port, peer_name: string)
 	{
 	if ( [peer_port, peer_address] !in netcontrol_acld_peers )
 		# ok, this one was none of ours...

scripts/base/frameworks/netcontrol/plugins/broker.bro

@@ -147,7 +147,7 @@ function broker_add_rule_fun(p: PluginState, r: Rule) : bool
 	if ( ! broker_check_rule(p, r) )
 		return F;
 
-	BrokerComm::event(p$broker_config$topic, BrokerComm::event_args(broker_add_rule, p$broker_id, r));
+	Broker::send_event(p$broker_config$topic, Broker::event_args(broker_add_rule, p$broker_id, r));
 	return T;
 	}
 
@@ -156,18 +156,18 @@ function broker_remove_rule_fun(p: PluginState, r: Rule) : bool
 	if ( ! broker_check_rule(p, r) )
 		return F;
 
-	BrokerComm::event(p$broker_config$topic, BrokerComm::event_args(broker_remove_rule, p$broker_id, r));
+	Broker::send_event(p$broker_config$topic, Broker::event_args(broker_remove_rule, p$broker_id, r));
 	return T;
 	}
 
 function broker_init(p: PluginState)
 	{
-	BrokerComm::enable();
-	BrokerComm::connect(cat(p$broker_config$host), p$broker_config$bport, 1sec);
-	BrokerComm::subscribe_to_events(p$broker_config$topic);
+	Broker::enable();
+	Broker::connect(cat(p$broker_config$host), p$broker_config$bport, 1sec);
+	Broker::subscribe_to_events(p$broker_config$topic);
 	}
 
-event BrokerComm::outgoing_connection_established(peer_address: string, peer_port: port, peer_name: string)
+event Broker::outgoing_connection_established(peer_address: string, peer_port: port, peer_name: string)
 	{
 	if ( [peer_port, peer_address] !in netcontrol_broker_peers )
 		return;

scripts/base/frameworks/openflow/plugins/broker.bro

@@ -47,26 +47,26 @@ function broker_describe(state: ControllerState): string
 
 function broker_flow_mod_fun(state: ControllerState, match: ofp_match, flow_mod: OpenFlow::ofp_flow_mod): bool
 	{
-	BrokerComm::event(state$broker_topic, BrokerComm::event_args(broker_flow_mod, state$_name, state$broker_dpid, match, flow_mod));
+	Broker::send_event(state$broker_topic, Broker::event_args(broker_flow_mod, state$_name, state$broker_dpid, match, flow_mod));
 
 	return T;
 	}
 
 function broker_flow_clear_fun(state: OpenFlow::ControllerState): bool
 	{
-	BrokerComm::event(state$broker_topic, BrokerComm::event_args(broker_flow_clear, state$_name, state$broker_dpid));
+	Broker::send_event(state$broker_topic, Broker::event_args(broker_flow_clear, state$_name, state$broker_dpid));
 
 	return T;
 	}
 
 function broker_init(state: OpenFlow::ControllerState)
 	{
-	BrokerComm::enable();
-	BrokerComm::connect(cat(state$broker_host), state$broker_port, 1sec);
-	BrokerComm::subscribe_to_events(state$broker_topic); # openflow success and failure events are directly sent back via the other plugin via broker.
+	Broker::enable();
+	Broker::connect(cat(state$broker_host), state$broker_port, 1sec);
+	Broker::subscribe_to_events(state$broker_topic); # openflow success and failure events are directly sent back via the other plugin via broker.
 	}
 
-event BrokerComm::outgoing_connection_established(peer_address: string, peer_port: port, peer_name: string)
+event Broker::outgoing_connection_established(peer_address: string, peer_port: port, peer_name: string)
 	{
 	if ( [peer_port, peer_address] !in broker_peers )
 		# ok, this one was none of ours...

scripts/base/init-bare.bro

@@ -793,71 +793,6 @@ type entropy_test_result: record {
 	serial_correlation: double;	##< Serial correlation coefficient.
 };
 
-# Prototypes of Bro built-in functions.
-@load base/bif/strings.bif
-@load base/bif/bro.bif
-@load base/bif/reporter.bif
-
-## Deprecated. This is superseded by the new logging framework.
-global log_file_name: function(tag: string): string &redef;
-
-## Deprecated. This is superseded by the new logging framework.
-global open_log_file: function(tag: string): file &redef;
-
-## Specifies a directory for Bro to store its persistent state. All globals can
-## be declared persistent via the :bro:attr:`&persistent` attribute.
-const state_dir = ".state" &redef;
-
-## Length of the delays inserted when storing state incrementally. To avoid
-## dropping packets when serializing larger volumes of persistent state to
-## disk, Bro interleaves the operation with continued packet processing.
-const state_write_delay = 0.01 secs &redef;
-
-global done_with_network = F;
-event net_done(t: time) { done_with_network = T; }
-
-function log_file_name(tag: string): string
-	{
-	local suffix = getenv("BRO_LOG_SUFFIX") == "" ? "log" : getenv("BRO_LOG_SUFFIX");
-	return fmt("%s.%s", tag, suffix);
-	}
-
-function open_log_file(tag: string): file
-	{
-	return open(log_file_name(tag));
-	}
-
-## Internal function.
-function add_interface(iold: string, inew: string): string
-	{
-	if ( iold == "" )
-		return inew;
-	else
-		return fmt("%s %s", iold, inew);
-	}
-
-## Network interfaces to listen on. Use ``redef interfaces += "eth0"`` to
-## extend.
-global interfaces = "" &add_func = add_interface;
-
-## Internal function.
-function add_signature_file(sold: string, snew: string): string
-	{
-	if ( sold == "" )
-		return snew;
-	else
-		return cat(sold, " ", snew);
-	}
-
-## Signature files to read. Use ``redef signature_files  += "foo.sig"`` to
-## extend. Signature files added this way will be searched relative to
-## ``BROPATH``.  Using the ``@load-sigs`` directive instead is preferred
-## since that can search paths relative to the current script.
-global signature_files = "" &add_func = add_signature_file;
-
-## ``p0f`` fingerprint file to use. Will be searched relative to ``BROPATH``.
-const passive_fingerprint_file = "base/misc/p0f.fp" &redef;
-
 # TCP values for :bro:see:`endpoint` *state* field.
 # todo:: these should go into an enum to make them autodoc'able.
 const TCP_INACTIVE = 0;	##< Endpoint is still inactive.
@@ -1768,6 +1703,71 @@ type gtp_delete_pdp_ctx_response_elements: record {
 	ext:   gtp_private_extension &optional;
 };
 
+# Prototypes of Bro built-in functions.
+@load base/bif/strings.bif
+@load base/bif/bro.bif
+@load base/bif/reporter.bif
+
+## Deprecated. This is superseded by the new logging framework.
+global log_file_name: function(tag: string): string &redef;
+
+## Deprecated. This is superseded by the new logging framework.
+global open_log_file: function(tag: string): file &redef;
+
+## Specifies a directory for Bro to store its persistent state. All globals can
+## be declared persistent via the :bro:attr:`&persistent` attribute.
+const state_dir = ".state" &redef;
+
+## Length of the delays inserted when storing state incrementally. To avoid
+## dropping packets when serializing larger volumes of persistent state to
+## disk, Bro interleaves the operation with continued packet processing.
+const state_write_delay = 0.01 secs &redef;
+
+global done_with_network = F;
+event net_done(t: time) { done_with_network = T; }
+
+function log_file_name(tag: string): string
+	{
+	local suffix = getenv("BRO_LOG_SUFFIX") == "" ? "log" : getenv("BRO_LOG_SUFFIX");
+	return fmt("%s.%s", tag, suffix);
+	}
+
+function open_log_file(tag: string): file
+	{
+	return open(log_file_name(tag));
+	}
+
+## Internal function.
+function add_interface(iold: string, inew: string): string
+	{
+	if ( iold == "" )
+		return inew;
+	else
+		return fmt("%s %s", iold, inew);
+	}
+
+## Network interfaces to listen on. Use ``redef interfaces += "eth0"`` to
+## extend.
+global interfaces = "" &add_func = add_interface;
+
+## Internal function.
+function add_signature_file(sold: string, snew: string): string
+	{
+	if ( sold == "" )
+		return snew;
+	else
+		return cat(sold, " ", snew);
+	}
+
+## Signature files to read. Use ``redef signature_files  += "foo.sig"`` to
+## extend. Signature files added this way will be searched relative to
+## ``BROPATH``.  Using the ``@load-sigs`` directive instead is preferred
+## since that can search paths relative to the current script.
+global signature_files = "" &add_func = add_signature_file;
+
+## ``p0f`` fingerprint file to use. Will be searched relative to ``BROPATH``.
+const passive_fingerprint_file = "base/misc/p0f.fp" &redef;
+
 ## Definition of "secondary filters". A secondary filter is a BPF filter given
 ## as index in this table. For each such filter, the corresponding event is
 ## raised for all matching packets.

scripts/base/init-default.bro

@@ -37,7 +37,7 @@
 @load base/frameworks/reporter
 @load base/frameworks/sumstats
 @load base/frameworks/tunnels
-@ifdef ( BrokerComm::enable )
+@ifdef ( Broker::enable )
 @load base/frameworks/openflow
 @load base/frameworks/netcontrol
 @endif
@@ -48,6 +48,7 @@
 @load base/protocols/dns
 @load base/protocols/ftp
 @load base/protocols/http
+@load base/protocols/imap
 @load base/protocols/irc
 @load base/protocols/krb
 @load base/protocols/modbus
@@ -55,6 +56,7 @@
 @load base/protocols/pop3
 @load base/protocols/radius
 @load base/protocols/rdp
+@load base/protocols/rfb
 @load base/protocols/sip
 @load base/protocols/snmp
 @load base/protocols/smtp

scripts/base/protocols/dns/consts.bro

@@ -26,6 +26,7 @@ export {
 		[49] = "DHCID", [99] = "SPF", [100] = "DINFO", [101] = "UID",
 		[102] = "GID", [103] = "UNSPEC", [249] = "TKEY", [250] = "TSIG",
 		[251] = "IXFR", [252] = "AXFR", [253] = "MAILB", [254] = "MAILA",
+		[257] = "CAA",
 		[32768] = "TA", [32769] = "DLV",
 		[ANY] = "*",
 	} &default = function(n: count): string { return fmt("query-%d", n); };

scripts/base/protocols/imap/README

@@ -0,0 +1,5 @@
+Support for the Internet Message Access Protocol (IMAP).
+
+Note that currently the IMAP analyzer only supports analyzing IMAP sessions
+until they do or do not switch to TLS using StartTLS. Hence, we do not get
+mails from IMAP sessions, only X509 certificates.

scripts/base/protocols/imap/__load__.bro

@@ -0,0 +1,2 @@
+@load ./main
+

scripts/base/protocols/imap/main.bro

@@ -0,0 +1,11 @@
+
+module IMAP;
+
+const ports = { 143/tcp };
+redef likely_server_ports += { ports };
+
+event bro_init() &priority=5
+	{
+	Analyzer::register_for_ports(Analyzer::ANALYZER_IMAP, ports);
+	}
+

scripts/base/protocols/rfb/README

@@ -0,0 +1 @@
+Support for Remote FrameBuffer analysis.  This includes all VNC servers.

scripts/base/protocols/rfb/__load__.bro

@@ -0,0 +1,3 @@
+# Generated by binpac_quickstart
+@load ./main
+@load-sigs ./dpd.sig

scripts/base/protocols/rfb/dpd.sig

@@ -0,0 +1,12 @@
+signature dpd_rfb_server {
+	ip-proto == tcp
+	payload /^RFB/
+	requires-reverse-signature dpd_rfb_client
+	enable "rfb"
+}
+
+signature dpd_rfb_client {
+	ip-proto == tcp
+	payload /^RFB/
+	tcp-state originator
+}

scripts/base/protocols/rfb/main.bro

@@ -0,0 +1,164 @@
+module RFB;
+
+export {
+	redef enum Log::ID += { LOG };
+
+	type Info: record {
+		## Timestamp for when the event happened.
+		ts:     time    &log;
+		## Unique ID for the connection.
+		uid:    string  &log;
+		## The connection's 4-tuple of endpoint addresses/ports.
+		id:     conn_id &log;
+
+		## Major version of the client.
+		client_major_version: string &log &optional;
+		## Minor version of the client.
+		client_minor_version: string &log &optional;
+		## Major version of the server.
+		server_major_version: string &log &optional;
+		## Major version of the client.
+		server_minor_version: string &log &optional;
+
+		## Identifier of authentication method used.
+		authentication_method: string &log &optional;
+		## Whether or not authentication was succesful.
+		auth: bool &log &optional;
+
+		## Whether the client has an exclusive or a shared session.
+		share_flag: bool &log &optional;
+		## Name of the screen that is being shared.
+		desktop_name: string &log &optional;
+		## Width of the screen that is being shared.
+		width: count &log &optional;
+		## Height of the screen that is being shared.
+		height: count &log &optional;
+
+		## Internally used value to determine if this connection
+		## has already been logged.
+		done: bool  &default=F;
+	};
+
+	global log_rfb: event(rec: Info);
+}
+
+function friendly_auth_name(auth: count): string
+	{
+	switch (auth) {
+		case 0:
+			return "Invalid";
+		case 1:
+			return "None";
+		case 2:
+			return "VNC";
+		case 16:
+			return "Tight";
+		case 17:
+			return "Ultra";
+		case 18:
+			return "TLS";
+		case 19:
+			return "VeNCrypt";
+		case 20:
+			return "GTK-VNC SASL";
+		case 21:
+			return "MD5 hash authentication";
+		case 22:
+			return "Colin Dean xvp";
+		case 30:
+			return "Apple Remote Desktop";
+	}
+	return "RealVNC";
+}
+
+redef record connection += {
+	rfb: Info &optional;
+};
+
+event bro_init() &priority=5
+	{
+	Log::create_stream(RFB::LOG, [$columns=Info, $ev=log_rfb, $path="rfb"]);
+	}
+
+function write_log(c:connection)
+	{
+	local state = c$rfb;
+	if ( state$done )
+		{
+		return;
+		}
+
+	Log::write(RFB::LOG, c$rfb);
+	c$rfb$done = T;
+	}
+
+function set_session(c: connection)
+	{
+	if ( ! c?$rfb )
+		{
+		local info: Info;
+		info$ts  = network_time();
+		info$uid = c$uid;
+		info$id  = c$id;
+
+		c$rfb = info;
+		}
+	}
+
+event rfb_event(c: connection) &priority=5
+	{
+	set_session(c);
+	}
+
+event rfb_client_version(c: connection, major_version: string, minor_version: string) &priority=5
+	{
+	set_session(c);
+	c$rfb$client_major_version = major_version;
+	c$rfb$client_minor_version = minor_version;
+	}
+
+event rfb_server_version(c: connection, major_version: string, minor_version: string) &priority=5
+	{
+	set_session(c);
+	c$rfb$server_major_version = major_version;
+	c$rfb$server_minor_version = minor_version;
+	}
+
+event rfb_authentication_type(c: connection, authtype: count) &priority=5
+	{
+	set_session(c);
+
+	c$rfb$authentication_method = friendly_auth_name(authtype);
+	}
+
+event rfb_server_parameters(c: connection, name: string, width: count, height: count) &priority=5
+	{
+	set_session(c);
+
+	c$rfb$desktop_name = name;
+	c$rfb$width = width;
+	c$rfb$height = height;
+	}
+
+event rfb_server_parameters(c: connection, name: string, width: count, height: count) &priority=-5
+	{
+	write_log(c);
+	}
+
+event rfb_auth_result(c: connection, result: bool) &priority=5
+	{
+	c$rfb$auth = !result;
+	}
+
+event rfb_share_flag(c: connection, flag: bool) &priority=5
+	{
+	c$rfb$share_flag = flag;
+	}
+
+event connection_state_remove(c: connection) &priority=-5
+	{
+	if ( c?$rfb )
+		{
+		write_log(c);
+		}
+	}

scripts/policy/frameworks/files/entropy-test-all-files.bro

@@ -0,0 +1,20 @@
+
+module Files;
+
+export {
+	redef record Files::Info += {
+		## The information density of the contents of the file, 
+		## expressed as a number of bits per character. 
+		entropy: double &log &optional;
+	};
+}
+
+event file_new(f: fa_file)
+	{
+	Files::add_analyzer(f, Files::ANALYZER_ENTROPY);
+	}
+
+event file_entropy(f: fa_file, ent: entropy_test_result)
+	{
+	f$info$entropy = ent$entropy;
+	}

scripts/policy/frameworks/intel/seen/ssl.bro

@@ -20,6 +20,7 @@ event ssl_established(c: connection)
 	if ( c$ssl$cert_chain[0]$x509?$certificate && c$ssl$cert_chain[0]$x509$certificate?$cn )
 		Intel::seen([$indicator=c$ssl$cert_chain[0]$x509$certificate$cn,
 			$indicator_type=Intel::DOMAIN,
+			$fuid=c$ssl$cert_chain_fuids[0],
 			$conn=c,
 			$where=X509::IN_CERT]);
 	}

scripts/policy/frameworks/intel/seen/x509.bro

@@ -26,3 +26,14 @@ event x509_certificate(f: fa_file, cert_ref: opaque of x509, cert: X509::Certifi
 			     $where=X509::IN_CERT]);
 		}
 	}
+
+event file_hash(f: fa_file, kind: string, hash: string)
+	{
+	if ( ! f?$info || ! f$info?$x509 || kind != "sha1" )
+		return;
+
+	Intel::seen([$indicator=hash,
+	             $indicator_type=Intel::CERT_HASH,
+	             $f=f,
+	             $where=X509::IN_CERT]);
+	}

scripts/test-all-policy.bro

@@ -29,6 +29,7 @@
 @load frameworks/intel/seen/where-locations.bro
 @load frameworks/intel/seen/x509.bro
 @load frameworks/files/detect-MHR.bro
+@load frameworks/files/entropy-test-all-files.bro
 #@load frameworks/files/extract-all-files.bro
 @load frameworks/files/hash-all-files.bro
 @load frameworks/packet-filter/shunt.bro

src/IP.cc

@@ -1,5 +1,9 @@
 // See the file "COPYING" in the main distribution directory for copyright.
 
+#include <sys/types.h>
+#include <netinet/in.h>
+#include <netinet/icmp6.h>
+
 #include "IP.h"
 #include "Type.h"
 #include "Val.h"
@@ -403,6 +407,17 @@ RecordVal* IP_Hdr::BuildPktHdrVal(RecordVal* pkt_hdr, int sindex) const
 		break;
 		}
 
+	case IPPROTO_ICMPV6:
+		{
+		const struct icmp6_hdr* icmpp = (const struct icmp6_hdr*) data;
+		RecordVal* icmp_hdr = new RecordVal(icmp_hdr_type);
+
+		icmp_hdr->Assign(0, new Val(icmpp->icmp6_type, TYPE_COUNT));
+
+		pkt_hdr->Assign(sindex + 4, icmp_hdr);
+		break;
+		}
+
 	default:
 		{
 		// This is not a protocol we understand.

src/NetVar.cc

@@ -15,6 +15,8 @@ RecordType* icmp_conn;
 RecordType* icmp_context;
 RecordType* SYN_packet;
 RecordType* pcap_packet;
+RecordType* raw_pkt_hdr_type;
+RecordType* l2_hdr_type;
 RecordType* signature_state;
 EnumType* transport_proto;
 TableType* string_set;
@@ -324,6 +326,8 @@ void init_net_var()
 	signature_state = internal_type("signature_state")->AsRecordType();
 	SYN_packet = internal_type("SYN_packet")->AsRecordType();
 	pcap_packet = internal_type("pcap_packet")->AsRecordType();
+	raw_pkt_hdr_type = internal_type("raw_pkt_hdr")->AsRecordType();
+	l2_hdr_type = internal_type("l2_hdr")->AsRecordType();
 	transport_proto = internal_type("transport_proto")->AsEnumType();
 	string_set = internal_type("string_set")->AsTableType();
 	string_array = internal_type("string_array")->AsTableType();

src/NetVar.h

@@ -19,6 +19,8 @@ extern RecordType* icmp_context;
 extern RecordType* signature_state;
 extern RecordType* SYN_packet;
 extern RecordType* pcap_packet;
+extern RecordType* raw_pkt_hdr_type;
+extern RecordType* l2_hdr_type;
 extern EnumType* transport_proto;
 extern TableType* string_set;
 extern TableType* string_array;

src/RuleCondition.cc

@@ -111,7 +111,7 @@ bool RuleConditionPayloadSize::DoMatch(Rule* rule, RuleEndpointState* state,
 		return payload_size >= val;
 
 	default:
-		reporter->InternalError("unknown comparision type");
+		reporter->InternalError("unknown comparison type");
 	}
 
 	// Should not be reached

src/RuleMatcher.cc

@@ -21,7 +21,7 @@
 //			it may fail to match. Work-around: Insert an always
 //			matching "payload" pattern (not done in snort2bro yet)
 //		  - tcp-state always evaluates to true
-//			(implemented but deactivated for comparision to Snort)
+//			(implemented but deactivated for comparison to Snort)
 
 uint32 RuleHdrTest::idcounter = 0;
 

src/Serializer.cc

@@ -437,7 +437,7 @@ bool Serializer::UnserializeCall(UnserialInfo* info)
 
 bool Serializer::UnserializeStateAccess(UnserialInfo* info)
 	{
-	SetErrorDescr("unserializing state acess");
+	SetErrorDescr("unserializing state access");
 
 	StateAccess* s = StateAccess::Unserialize(info);
 

src/StateAccess.cc

@@ -150,7 +150,7 @@ bool StateAccess::CheckOld(const char* op, ID* id, Val* index,
 
 	if ( should && is )
 		{
-		// There's no general comparision for non-atomic vals currently.
+		// There's no general comparison for non-atomic vals currently.
 		if ( ! (is_atomic_val(is) && is_atomic_val(should)) )
 			return true;
 

src/analyzer/protocol/CMakeLists.txt

@@ -16,6 +16,7 @@ add_subdirectory(gtpv1)
 add_subdirectory(http)
 add_subdirectory(icmp)
 add_subdirectory(ident)
+add_subdirectory(imap)
 add_subdirectory(interconn)
 add_subdirectory(irc)
 add_subdirectory(krb)
@@ -30,6 +31,7 @@ add_subdirectory(pia)
 add_subdirectory(pop3)
 add_subdirectory(radius)
 add_subdirectory(rdp)
+add_subdirectory(rfb)
 add_subdirectory(rpc)
 add_subdirectory(sip)
 add_subdirectory(snmp)

src/analyzer/protocol/arp/ARP.cc

@@ -10,9 +10,6 @@ using namespace analyzer::arp;
 
 ARP_Analyzer::ARP_Analyzer()
 	{
-	bad_arp = internal_handler("bad_arp");
-	arp_request = internal_handler("arp_request");
-	arp_reply = internal_handler("arp_reply");
 	}
 
 ARP_Analyzer::~ARP_Analyzer()

src/analyzer/protocol/arp/ARP.h

@@ -50,10 +50,6 @@ protected:
 	StringVal* EthAddrToStr(const u_char* addr);
 	void BadARP(const struct arp_pkthdr* hdr, const char* string);
 	void Corrupted(const char* string);
-
-	EventHandlerPtr arp_corrupted_packet;
-	EventHandlerPtr arp_request;
-	EventHandlerPtr arp_reply;
 };
 
 } } // namespace analyzer::* 

src/analyzer/protocol/dns/DNS.cc

@@ -282,6 +282,10 @@ int DNS_Interpreter::ParseAnswer(DNS_MsgInfo* msg,
 			status = ParseRR_TXT(msg, data, len, rdlength, msg_start);
 			break;
 
+		case TYPE_CAA:
+			status = ParseRR_CAA(msg, data, len, rdlength, msg_start);
+			break;
+
 		case TYPE_NBS:
 			status = ParseRR_NBS(msg, data, len, rdlength, msg_start);
 			break;
@@ -904,6 +908,51 @@ int DNS_Interpreter::ParseRR_TXT(DNS_MsgInfo* msg,
 	return rdlength == 0;
 	}
 
+int DNS_Interpreter::ParseRR_CAA(DNS_MsgInfo* msg,
+				const u_char*& data, int& len, int rdlength,
+				const u_char* msg_start)
+	{
+	if ( ! dns_CAA_reply || msg->skip_event )
+		{
+		data += rdlength;
+		len -= rdlength;
+		return 1;
+		}
+
+	unsigned int flags = ExtractShort(data, len);
+	unsigned int tagLen = flags & 0xff;
+	flags = flags >> 8;
+	rdlength -= 2;
+	if ( (int) tagLen >= rdlength )
+		{
+		analyzer->Weird("DNS_CAA_char_str_past_rdlen");
+		return 0;
+		}
+	BroString* tag = new BroString(data, tagLen, 1);
+	len -= tagLen;
+	data += tagLen;
+	rdlength -= tagLen;
+	BroString* value = new BroString(data, rdlength, 0);
+
+	len -= value->Len();
+	data += value->Len();
+	rdlength -= value->Len();
+
+	val_list* vl = new val_list;
+
+	vl->append(analyzer->BuildConnVal());
+	vl->append(msg->BuildHdrVal());
+	vl->append(msg->BuildAnswerVal());
+	vl->append(new Val(flags, TYPE_COUNT));
+	vl->append(new StringVal(tag));
+	vl->append(new StringVal(value));
+
+	analyzer->ConnectionEvent(dns_CAA_reply, vl);
+
+	return rdlength == 0;
+	}
+
+
 void DNS_Interpreter::SendReplyOrRejectEvent(DNS_MsgInfo* msg,
 						EventHandlerPtr event,
 						const u_char*& data, int& len,

src/analyzer/protocol/dns/DNS.h

@@ -56,6 +56,7 @@ typedef enum {
 	TYPE_EDNS = 41,		///< OPT pseudo-RR (RFC 2671)
 	TYPE_TKEY = 249,	///< Transaction Key (RFC 2930)
 	TYPE_TSIG = 250,	///< Transaction Signature (RFC 2845)
+	TYPE_CAA = 257,		///< Certification Authority Authorization (RFC 6844)
 
 	// The following are only valid in queries.
 	TYPE_AXFR = 252,
@@ -132,7 +133,7 @@ public:
 	StringVal* query_name;
 	RR_Type atype;
 	int aclass;	///< normally = 1, inet
-	int ttl;
+	uint32 ttl;
 
 	DNS_AnswerType answer_type;
 	int skip_event;		///< if true, don't generate corresponding events
@@ -211,6 +212,9 @@ protected:
 	int ParseRR_TXT(DNS_MsgInfo* msg,
 				const u_char*& data, int& len, int rdlength,
 				const u_char* msg_start);
+	int ParseRR_CAA(DNS_MsgInfo* msg,
+				const u_char*& data, int& len, int rdlength,
+				const u_char* msg_start);
 	int ParseRR_TSIG(DNS_MsgInfo* msg,
 				const u_char*& data, int& len, int rdlength,
 				const u_char* msg_start);

src/analyzer/protocol/dns/events.bif

@@ -378,6 +378,25 @@ event dns_MX_reply%(c: connection, msg: dns_msg, ans: dns_answer, name: string,
 ##    dns_skip_addl dns_skip_all_addl dns_skip_all_auth dns_skip_auth
 event dns_TXT_reply%(c: connection, msg: dns_msg, ans: dns_answer, strs: string_vec%);
 
+## Generated for DNS replies of type *CAA* (Certification Authority Authorization).
+## For replies with multiple answers, an individual event of the corresponding type
+## is raised for each.
+## See `RFC 6844 <https://tools.ietf.org/html/rfc6844>`__ for more details.
+##
+## c: The connection, which may be UDP or TCP depending on the type of the
+##    transport-layer session being analyzed.
+##
+## msg: The parsed DNS message header.
+##
+## ans: The type-independent part of the parsed answer record.
+##
+## flags: The flags byte of the CAA reply.
+##
+## tag: The property identifier of the CAA reply.
+##
+## value: The property value of the CAA reply.
+event dns_CAA_reply%(c: connection, msg: dns_msg, ans: dns_answer, flags: count, tag: string, value: string%);
+
 ## Generated for DNS replies of type *SRV*. For replies with multiple answers,
 ## an individual event of the corresponding type is raised for each.
 ##

src/analyzer/protocol/imap/CMakeLists.txt

@@ -0,0 +1,12 @@
+
+include(BroPlugin)
+
+include_directories(BEFORE ${CMAKE_CURRENT_SOURCE_DIR} ${CMAKE_CURRENT_BINARY_DIR})
+
+bro_plugin_begin(Bro IMAP)
+bro_plugin_cc(Plugin.cc)
+bro_plugin_cc(IMAP.cc)
+bro_plugin_bif(events.bif)
+bro_plugin_pac(imap.pac imap-analyzer.pac imap-protocol.pac)
+bro_plugin_end()
+

src/analyzer/protocol/imap/IMAP.cc

@@ -0,0 +1,85 @@
+// See the file "COPYING" in the main distribution directory for copyright.
+
+#include "IMAP.h"
+#include "analyzer/protocol/tcp/TCP_Reassembler.h"
+#include "analyzer/Manager.h"
+
+using namespace analyzer::imap;
+
+IMAP_Analyzer::IMAP_Analyzer(Connection* conn)
+	: tcp::TCP_ApplicationAnalyzer("IMAP", conn)
+	{
+	interp = new binpac::IMAP::IMAP_Conn(this);
+	had_gap = false;
+	tls_active = false;
+	}
+
+IMAP_Analyzer::~IMAP_Analyzer()
+	{
+	delete interp;
+	}
+
+void IMAP_Analyzer::Done()
+	{
+	tcp::TCP_ApplicationAnalyzer::Done();
+
+	interp->FlowEOF(true);
+	interp->FlowEOF(false);
+	}
+
+void IMAP_Analyzer::EndpointEOF(bool is_orig)
+	{
+	tcp::TCP_ApplicationAnalyzer::EndpointEOF(is_orig);
+	interp->FlowEOF(is_orig);
+	}
+
+void IMAP_Analyzer::DeliverStream(int len, const u_char* data, bool orig)
+	{
+	tcp::TCP_ApplicationAnalyzer::DeliverStream(len, data, orig);
+
+	if ( tls_active )
+		{
+		// If TLS has been initiated, forward to child and abort further
+		// processing
+		ForwardStream(len, data, orig);
+		return;
+		}
+
+	assert(TCP());
+	if ( TCP()->IsPartial() )
+		return;
+
+	if ( had_gap )
+		// If only one side had a content gap, we could still try to
+		// deliver data to the other side if the script layer can
+		// handle this.
+		return;
+
+	try
+		{
+		interp->NewData(orig, data, data + len);
+		}
+	catch ( const binpac::Exception& e )
+		{
+		ProtocolViolation(fmt("Binpac exception: %s", e.c_msg()));
+		}
+	}
+
+void IMAP_Analyzer::Undelivered(uint64 seq, int len, bool orig)
+	{
+	tcp::TCP_ApplicationAnalyzer::Undelivered(seq, len, orig);
+	had_gap = true;
+	interp->NewGap(orig, len);
+	}
+
+void IMAP_Analyzer::StartTLS()
+	{
+	// StartTLS was called. This means we saw a client starttls followed
+	// by a server proceed. From here on, everything should be a binary
+	// TLS datastream.
+	tls_active = true;
+
+	Analyzer* ssl = analyzer_mgr->InstantiateAnalyzer("SSL", Conn());
+	if ( ssl )
+		AddChildAnalyzer(ssl);
+	}

src/analyzer/protocol/imap/IMAP.h

@@ -0,0 +1,40 @@
+// See the file "COPYING" in the main distribution directory for copyright.
+
+#ifndef ANALYZER_PROTOCOL_IMAP_IMAP_H
+#define ANALYZER_PROTOCOL_IMAP_IMAP_H
+
+// for std::transform
+#include <algorithm>
+#include "analyzer/protocol/tcp/TCP.h"
+
+#include "imap_pac.h"
+
+namespace analyzer { namespace imap {
+
+class IMAP_Analyzer : public tcp::TCP_ApplicationAnalyzer {
+public:
+	IMAP_Analyzer(Connection* conn);
+	virtual ~IMAP_Analyzer();
+
+	virtual void Done();
+	virtual void DeliverStream(int len, const u_char* data, bool orig);
+	virtual void Undelivered(uint64 seq, int len, bool orig);
+
+	// Overriden from tcp::TCP_ApplicationAnalyzer.
+	virtual void EndpointEOF(bool is_orig);
+
+	void StartTLS();
+
+	static analyzer::Analyzer* Instantiate(Connection* conn)
+		{ return new IMAP_Analyzer(conn); }
+
+protected:
+	binpac::IMAP::IMAP_Conn* interp;
+	bool had_gap;
+
+	bool tls_active;
+};
+
+} } // namespace analyzer::*
+
+#endif /* ANALYZER_PROTOCOL_IMAP_IMAP_H */

src/analyzer/protocol/imap/Plugin.cc

@@ -0,0 +1,22 @@
+// See the file  in the main distribution directory for copyright.
+#include "plugin/Plugin.h"
+#include "IMAP.h"
+
+namespace plugin {
+namespace Bro_IMAP {
+
+class Plugin : public plugin::Plugin {
+public:
+	plugin::Configuration Configure()
+		{
+		AddComponent(new ::analyzer::Component("IMAP", ::analyzer::imap::IMAP_Analyzer::Instantiate));
+
+		plugin::Configuration config;
+		config.name = "Bro::IMAP";
+		config.description = "IMAP analyzer (StartTLS only)";
+		return config;
+		}
+} plugin;
+
+}
+}

src/analyzer/protocol/imap/events.bif

@@ -0,0 +1,13 @@
+## Generated when a server sends a capability list to the client,
+## after being queried using the CAPABILITY command.
+##
+## c: The connection.
+##
+## capabilities: The list of IMAP capabilities as sent by the server.
+event imap_capabilities%(c: connection, capabilities: string_vec%);
+
+## Generated when a IMAP connection goes encrypted after a successful
+## StartTLS exchange between the client and the server.
+##
+## c: The connection.
+event imap_starttls%(c: connection%);

src/analyzer/protocol/imap/imap-analyzer.pac

@@ -0,0 +1,76 @@
+refine connection IMAP_Conn += {
+
+	%member{
+		string client_starttls_id;
+	%}
+
+	%init{
+	%}
+
+	function proc_imap_token(is_orig: bool, tag: bytestring, command: bytestring): bool
+		%{
+		string commands = std_str(command);
+		std::transform(commands.begin(), commands.end(), commands.begin(), ::tolower);
+
+		string tags = std_str(tag);
+
+		//printf("imap %s %s\n", commands.c_str(), tags.c_str());
+
+		if ( !is_orig && tags == "*" && commands == "ok" )
+			bro_analyzer()->ProtocolConfirmation();
+
+		if ( is_orig && ( command == "capability" || commands == "starttls" ) )
+			bro_analyzer()->ProtocolConfirmation();
+
+		if ( command == "authenticate" || command == "login" || command == "examine" || command == "create" || command == "list" || command == "fetch" )
+			{
+			bro_analyzer()->ProtocolConfirmation();
+			// Handshake has passed the phase where we should see StartTLS. Simply skip from hereon...
+			bro_analyzer()->SetSkip(true);
+			return true;
+			}
+
+		if ( is_orig && commands == "starttls" )
+			{
+			if ( !client_starttls_id.empty() )
+				reporter->Weird(bro_analyzer()->Conn(), "IMAP: client sent duplicate StartTLS");
+
+			client_starttls_id = tags;
+			}
+
+		if ( !is_orig && !client_starttls_id.empty() && tags == client_starttls_id )
+			{
+			if ( commands == "ok" )
+				{
+				bro_analyzer()->StartTLS();
+				BifEvent::generate_imap_starttls(bro_analyzer(), bro_analyzer()->Conn());
+				}
+			else
+				reporter->Weird(bro_analyzer()->Conn(), "IMAP: server refused StartTLS");
+			}
+
+		return true;
+		%}
+
+	function proc_server_capability(capabilities: Capability[]): bool
+		%{
+		VectorVal* capv = new VectorVal(internal_type("string_vec")->AsVectorType());
+		for ( unsigned int i = 0; i< capabilities->size(); i++ )
+			{
+			const bytestring& capability = (*capabilities)[i]->cap();
+			capv->Assign(i, new StringVal(capability.length(), (const char*)capability.data()));
+			}
+
+		BifEvent::generate_imap_capabilities(bro_analyzer(), bro_analyzer()->Conn(), capv);
+		return true;
+		%}
+
+};
+
+refine typeattr ImapToken += &let {
+	proc: bool = $context.connection.proc_imap_token(is_orig, tag, command);
+};
+
+refine typeattr ServerCapability += &let {
+	proc: bool = $context.connection.proc_server_capability(capabilities);
+};

src/analyzer/protocol/imap/imap-protocol.pac

@@ -0,0 +1,70 @@
+# commands that we support parsing. The numbers do not really mean anything
+# in this case
+enum ImapCommand {
+	CMD_CAPABILITY,
+	CMD_UNKNOWN
+}
+
+type TAG = RE/[[:alnum:][:punct:]]+/;
+type CONTENT = RE/[^\r\n]*/;
+type SPACING = RE/[ ]+/;
+type OPTIONALSPACING = RE/[ ]*/;
+type NEWLINE = RE/[\r\n]+/;
+type OPTIONALNEWLINE = RE/[\r\n]*/;
+
+type IMAP_PDU(is_orig: bool) = ImapToken(is_orig)[] &until($input.length() == 0);
+
+type ImapToken(is_orig: bool) = record {
+	tag : TAG;
+	: SPACING;
+	command: TAG;
+	: OPTIONALSPACING;
+	client_or_server: case is_orig of {
+		true -> client: UnknownCommand(this) ;
+		false -> server: ServerContentText(this);
+	} &requires(pcommand) ;
+} &let {
+	pcommand: int = $context.connection.determine_command(is_orig, tag, command);
+};
+
+type ServerContentText(rec: ImapToken) = case rec.pcommand of {
+	CMD_CAPABILITY -> capability: ServerCapability(rec);
+	default -> unknown: UnknownCommand(rec);
+};
+
+type Capability = record {
+	cap: TAG;
+	: OPTIONALSPACING;
+	nl: OPTIONALNEWLINE;
+};
+
+type ServerCapability(rec: ImapToken) = record {
+	capabilities: Capability[] &until($context.connection.strlen($element.nl) > 0);
+};
+
+type UnknownCommand(rec: ImapToken) = record {
+	tagcontent: CONTENT;
+	: NEWLINE;
+};
+
+refine connection IMAP_Conn += {
+
+	function determine_command(is_orig: bool, tag: bytestring, command: bytestring): int
+		%{
+		string cmdstr = std_str(command);
+		std::transform(cmdstr.begin(), cmdstr.end(), cmdstr.begin(), ::tolower);
+		string tagstr = std_str(tag);
+
+		if ( !is_orig && cmdstr == "capability" && tag == "*" ) {
+			return CMD_CAPABILITY;
+		}
+
+		return CMD_UNKNOWN;
+		%}
+
+	function strlen(str: bytestring): int
+		%{
+		return str.length();
+		%}
+
+};

src/analyzer/protocol/imap/imap.pac

@@ -0,0 +1,37 @@
+# binpac file for the IMAP analyzer.
+# Note that we currently do not even try to parse the protocol
+# completely -- this is only supposed to be able to parse imap
+# till StartTLS does (or does not) kick in.
+
+%include binpac.pac
+%include bro.pac
+
+%extern{
+#include "events.bif.h"
+
+namespace analyzer { namespace imap { class IMAP_Analyzer; } }
+namespace binpac { namespace IMAP { class IMAP_Conn; } }
+typedef analyzer::imap::IMAP_Analyzer* IMAPAnalyzer;
+
+#include "IMAP.h"
+%}
+
+extern type IMAPAnalyzer;
+
+analyzer IMAP withcontext {
+	connection:	 IMAP_Conn;
+	flow:		 IMAP_Flow;
+};
+
+connection IMAP_Conn(bro_analyzer: IMAPAnalyzer) {
+	upflow = IMAP_Flow(true);
+	downflow = IMAP_Flow(false);
+};
+
+%include imap-protocol.pac
+
+flow IMAP_Flow(is_orig: bool) {
+	datagram = IMAP_PDU(is_orig) withcontext(connection, this);
+};
+
+%include imap-analyzer.pac

src/analyzer/protocol/rfb/CMakeLists.txt

@@ -0,0 +1,9 @@
+include(BroPlugin)
+
+include_directories(BEFORE ${CMAKE_CURRENT_SOURCE_DIR} ${CMAKE_CURRENT_BINARY_DIR})
+
+bro_plugin_begin(Bro RFB)
+	bro_plugin_cc(RFB.cc Plugin.cc)
+	bro_plugin_bif(events.bif)
+	bro_plugin_pac(rfb.pac rfb-analyzer.pac rfb-protocol.pac)
+bro_plugin_end()

src/analyzer/protocol/rfb/Plugin.cc

@@ -0,0 +1,23 @@
+#include "plugin/Plugin.h"
+
+#include "RFB.h"
+
+namespace plugin {
+namespace Bro_RFB {
+
+class Plugin : public plugin::Plugin {
+public:
+	plugin::Configuration Configure()
+		{
+		AddComponent(new ::analyzer::Component("RFB",
+		             ::analyzer::rfb::RFB_Analyzer::InstantiateAnalyzer));
+
+		plugin::Configuration config;
+		config.name = "Bro::RFB";
+		config.description = "Parser for rfb (VNC) analyzer";
+		return config;
+		}
+} plugin;
+
+}
+}

src/analyzer/protocol/rfb/RFB.cc

@@ -0,0 +1,67 @@
+#include "RFB.h"
+
+#include "analyzer/protocol/tcp/TCP_Reassembler.h"
+
+#include "Reporter.h"
+
+#include "events.bif.h"
+
+using namespace analyzer::rfb;
+
+RFB_Analyzer::RFB_Analyzer(Connection* c)
+
+: tcp::TCP_ApplicationAnalyzer("RFB", c)
+
+	{
+	interp = new binpac::RFB::RFB_Conn(this);
+	had_gap = false;
+	}
+
+RFB_Analyzer::~RFB_Analyzer()
+	{
+	delete interp;
+	}
+
+void RFB_Analyzer::Done()
+	{
+	tcp::TCP_ApplicationAnalyzer::Done();
+
+	interp->FlowEOF(true);
+	interp->FlowEOF(false);
+
+	}
+
+void RFB_Analyzer::EndpointEOF(bool is_orig)
+	{
+	tcp::TCP_ApplicationAnalyzer::EndpointEOF(is_orig);
+	interp->FlowEOF(is_orig);
+	}
+
+void RFB_Analyzer::DeliverStream(int len, const u_char* data, bool orig)
+	{
+	tcp::TCP_ApplicationAnalyzer::DeliverStream(len, data, orig);
+	assert(TCP());
+	if ( TCP()->IsPartial() )
+		return;
+
+	if ( had_gap )
+		// If only one side had a content gap, we could still try to
+		// deliver data to the other side if the script layer can handle this.
+		return;
+
+	try
+		{
+		interp->NewData(orig, data, data + len);
+		}
+	catch ( const binpac::Exception& e )
+		{
+		ProtocolViolation(fmt("Binpac exception: %s", e.c_msg()));
+		}
+	}
+
+void RFB_Analyzer::Undelivered(uint64 seq, int len, bool orig)
+	{
+	tcp::TCP_ApplicationAnalyzer::Undelivered(seq, len, orig);
+	had_gap = true;
+	interp->NewGap(orig, len);
+	}

src/analyzer/protocol/rfb/RFB.h

@@ -0,0 +1,43 @@
+#ifndef ANALYZER_PROTOCOL_RFB_RFB_H
+#define ANALYZER_PROTOCOL_RFB_RFB_H
+
+#include "events.bif.h"
+
+
+#include "analyzer/protocol/tcp/TCP.h"
+
+#include "rfb_pac.h"
+
+namespace analyzer { namespace rfb {
+
+class RFB_Analyzer
+
+: public tcp::TCP_ApplicationAnalyzer {
+
+public:
+	RFB_Analyzer(Connection* conn);
+	virtual ~RFB_Analyzer();
+
+	// Overriden from Analyzer.
+	virtual void Done();
+
+	virtual void DeliverStream(int len, const u_char* data, bool orig);
+	virtual void Undelivered(uint64 seq, int len, bool orig);
+
+	// Overriden from tcp::TCP_ApplicationAnalyzer.
+	virtual void EndpointEOF(bool is_orig);
+
+
+	static analyzer::Analyzer* InstantiateAnalyzer(Connection* conn)
+		{ return new RFB_Analyzer(conn); }
+
+protected:
+	binpac::RFB::RFB_Conn* interp;
+
+	bool had_gap;
+
+};
+
+} } // namespace analyzer::*
+
+#endif

src/analyzer/protocol/rfb/events.bif

@@ -0,0 +1,50 @@
+## Generated for RFB event
+##
+## c: The connection record for the underlying transport-layer session/flow.
+event rfb_event%(c: connection%);
+
+## Generated for RFB event authentication mechanism selection
+##
+## c: The connection record for the underlying transport-layer session/flow.
+##
+## authtype: the value of the chosen authentication mechanism
+event rfb_authentication_type%(c: connection, authtype: count%);
+
+## Generated for RFB event authentication result message
+##
+## c: The connection record for the underlying transport-layer session/flow.
+##
+## result: whether or not authentication was succesful
+event rfb_auth_result%(c: connection, result: bool%);
+
+## Generated for RFB event share flag messages
+##
+## c: The connection record for the underlying transport-layer session/flow.
+##
+## flag: whether or not the share flag was set
+event rfb_share_flag%(c: connection, flag: bool%);
+
+## Generated for RFB event client banner message
+##
+## c: The connection record for the underlying transport-layer session/flow.
+##
+## version: of the client's rfb library
+event rfb_client_version%(c: connection, major_version: string, minor_version: string%);
+
+## Generated for RFB event server banner message
+##
+## c: The connection record for the underlying transport-layer session/flow.
+##
+## version: of the server's rfb library
+event rfb_server_version%(c: connection, major_version: string, minor_version: string%);
+
+## Generated for RFB event server parameter message
+##
+## c: The connection record for the underlying transport-layer session/flow.
+##
+## name: name of the shared screen
+##
+## width: width of the shared screen
+##
+## height: height of the shared screen
+event rfb_server_parameters%(c: connection, name: string, width: count, height: count%);

src/analyzer/protocol/rfb/rfb-analyzer.pac

@@ -0,0 +1,199 @@
+refine flow RFB_Flow += {
+	function proc_rfb_message(msg: RFB_PDU): bool
+		%{
+		BifEvent::generate_rfb_event(connection()->bro_analyzer(), connection()->bro_analyzer()->Conn());
+		return true;
+		%}
+
+	function proc_rfb_version(client: bool, major: bytestring, minor: bytestring) : bool
+		%{
+		if (client)
+			{
+			BifEvent::generate_rfb_client_version(connection()->bro_analyzer(), connection()->bro_analyzer()->Conn(), bytestring_to_val(major), bytestring_to_val(minor));
+
+			connection()->bro_analyzer()->ProtocolConfirmation();
+			}
+			else
+			{
+			BifEvent::generate_rfb_server_version(connection()->bro_analyzer(), connection()->bro_analyzer()->Conn(), bytestring_to_val(major), bytestring_to_val(minor));
+			}
+		return true;
+		%}
+
+	function proc_rfb_share_flag(shared: bool) : bool
+		%{
+		BifEvent::generate_rfb_share_flag(connection()->bro_analyzer(), connection()->bro_analyzer()->Conn(), shared);
+		return true;
+		%}
+
+	function proc_security_types(msg: RFBSecurityTypes) : bool
+		%{
+		BifEvent::generate_rfb_authentication_type(connection()->bro_analyzer(), connection()->bro_analyzer()->Conn(), ${msg.sectype});
+		return true;
+		%}
+
+	function proc_security_types37(msg: RFBAuthTypeSelected) : bool
+		%{
+		BifEvent::generate_rfb_authentication_type(connection()->bro_analyzer(), connection()->bro_analyzer()->Conn(), ${msg.type});
+		return true;
+		%}
+
+	function proc_handle_server_params(msg:RFBServerInit) : bool
+		%{
+		BifEvent::generate_rfb_server_parameters(connection()->bro_analyzer(), connection()->bro_analyzer()->Conn(), bytestring_to_val(${msg.name}), ${msg.width}, ${msg.height});
+		return true;
+		%}
+
+	function proc_handle_security_result(result : uint32) : bool
+		%{
+		BifEvent::generate_rfb_auth_result(connection()->bro_analyzer(), connection()->bro_analyzer()->Conn(), result);
+		return true;
+		%}
+};
+
+refine connection RFB_Conn += {
+	%member{
+		enum states {
+			AWAITING_SERVER_BANNER = 0,
+			AWAITING_CLIENT_BANNER = 1,
+			AWAITING_SERVER_AUTH_TYPES = 2,
+			AWAITING_SERVER_CHALLENGE = 3,
+			AWAITING_CLIENT_RESPONSE = 4,
+			AWAITING_SERVER_AUTH_RESULT = 5,
+			AWAITING_CLIENT_SHARE_FLAG = 6,
+			AWAITING_SERVER_PARAMS = 7,
+			AWAITING_CLIENT_AUTH_METHOD = 8,
+			AWAITING_SERVER_ARD_CHALLENGE = 9,
+			AWAITING_CLIENT_ARD_RESPONSE = 10,
+			AWAITING_SERVER_AUTH_TYPES37 = 11,
+			AWAITING_CLIENT_AUTH_TYPE_SELECTED37 = 12,
+			RFB_MESSAGE = 13
+		};
+	%}
+
+	function get_state(client: bool) : int
+		%{
+		return state;
+		%}
+
+	function handle_banners(client: bool, msg: RFBProtocolVersion) : bool
+		%{
+		if ( client )
+			{
+			// Set protocol version on client's version
+			int minor_version = bytestring_to_int(${msg.minor_ver},10);
+			version = minor_version;
+
+			// Apple specifies minor version "889" but talks v37
+			if ( minor_version >= 7 )
+				state = AWAITING_SERVER_AUTH_TYPES37;
+			else
+				state = AWAITING_SERVER_AUTH_TYPES;
+			}
+			else
+				state = AWAITING_CLIENT_BANNER;
+
+		return true;
+		%}
+
+	function handle_ard_challenge() : bool
+		%{
+		state = AWAITING_CLIENT_ARD_RESPONSE;
+		return true;
+		%}
+
+	function handle_ard_response() : bool
+		%{
+		state = AWAITING_SERVER_AUTH_RESULT;
+		return true;
+		%}
+
+	function handle_auth_request() : bool
+		%{
+		state = AWAITING_CLIENT_RESPONSE;
+		return true;
+		%}
+
+	function handle_auth_response() : bool
+		%{
+		state = AWAITING_SERVER_AUTH_RESULT;
+		return true;
+		%}
+
+	function handle_security_result(msg: RFBSecurityResult) : bool
+		%{
+		if ( ${msg.result} == 0 )
+			{
+			state = AWAITING_CLIENT_SHARE_FLAG;
+			}
+		return true;
+		%}
+
+	function handle_client_init(msg: RFBClientInit) : bool
+		%{
+		state = AWAITING_SERVER_PARAMS;
+		return true;
+		%}
+
+	function handle_server_init(msg: RFBServerInit) : bool
+		%{
+		state = RFB_MESSAGE;
+		return true;
+		%}
+
+	function handle_security_types(msg: RFBSecurityTypes): bool
+		%{
+		if ( msg->sectype() == 0 )
+			{ // No auth
+			state = AWAITING_CLIENT_SHARE_FLAG;
+			return true;
+			}
+
+		if ( msg->sectype() == 2 )
+			{ //VNC
+			state = AWAITING_SERVER_CHALLENGE;
+			}
+		return true;
+		%}
+
+	function handle_security_types37(msg: RFBSecurityTypes37): bool
+		%{
+		if ( ${msg.count} == 0 )
+			{ // No auth
+			state = AWAITING_CLIENT_SHARE_FLAG;
+			return true;
+			}
+		state = AWAITING_CLIENT_AUTH_TYPE_SELECTED37;
+		return true;
+		%}
+
+	function handle_auth_type_selected(msg: RFBAuthTypeSelected): bool
+		%{
+		if ( ${msg.type} == 30 )
+			{ // Apple Remote Desktop
+				state = AWAITING_SERVER_ARD_CHALLENGE;
+				return true;
+			}
+
+		if ( ${msg.type} == 1 )
+			{
+				if ( version > 7 )
+					state = AWAITING_SERVER_AUTH_RESULT;
+				else
+					state = AWAITING_CLIENT_SHARE_FLAG;
+			}
+		else
+			state = AWAITING_SERVER_CHALLENGE;
+
+		return true;
+		%}
+
+	%member{
+		uint8 state = AWAITING_SERVER_BANNER;
+		int version = 0;
+	%}
+};
+
+refine typeattr RFB_PDU += &let {
+	proc: bool = $context.flow.proc_rfb_message(this);
+};

src/analyzer/protocol/rfb/rfb-protocol.pac

@@ -0,0 +1,139 @@
+enum states {
+			AWAITING_SERVER_BANNER = 0,
+			AWAITING_CLIENT_BANNER = 1,
+			AWAITING_SERVER_AUTH_TYPES = 2,
+			AWAITING_SERVER_CHALLENGE = 3,
+			AWAITING_CLIENT_RESPONSE = 4,
+			AWAITING_SERVER_AUTH_RESULT = 5,
+			AWAITING_CLIENT_SHARE_FLAG = 6,
+			AWAITING_SERVER_PARAMS = 7,
+			AWAITING_CLIENT_AUTH_METHOD = 8,
+			AWAITING_SERVER_ARD_CHALLENGE = 9,
+			AWAITING_CLIENT_ARD_RESPONSE = 10,
+			AWAITING_SERVER_AUTH_TYPES37 = 11,
+			AWAITING_CLIENT_AUTH_TYPE_SELECTED37 = 12,
+			RFB_MESSAGE = 13
+		};
+
+type RFBProtocolVersion (client: bool) = record {
+	header: "RFB ";
+	major_ver: bytestring &length=3;
+	dot: ".";
+	minor_ver: bytestring &length=3;
+	pad: uint8;
+} &let {
+	proc: bool = $context.connection.handle_banners(client, this);
+	proc2: bool = $context.flow.proc_rfb_version(client, major_ver, minor_ver);
+}
+
+type RFBSecurityTypes = record {
+	sectype: uint32;
+} &let {
+	proc: bool = $context.connection.handle_security_types(this);
+	proc2: bool = $context.flow.proc_security_types(this);
+};
+
+type RFBSecurityTypes37 = record {
+	count: uint8;
+	types: uint8[count];
+} &let {
+	proc: bool = $context.connection.handle_security_types37(this);
+};
+
+type RFBAuthTypeSelected = record {
+	type: uint8;
+} &let {
+	proc: bool = $context.connection.handle_auth_type_selected(this);
+	proc2: bool = $context.flow.proc_security_types37(this);
+};
+
+type RFBSecurityResult = record {
+	result: uint32;
+} &let {
+	proc: bool = $context.connection.handle_security_result(this);
+	proc2: bool = $context.flow.proc_handle_security_result(result);
+};
+
+type RFBSecurityResultReason = record {
+	len: uint32;
+	reason: bytestring &length=len;
+};
+
+type RFBVNCAuthenticationRequest = record {
+	challenge: bytestring &length=16;
+} &let {
+	proc: bool = $context.connection.handle_auth_request();
+};
+
+type RFBVNCAuthenticationResponse = record {
+	response: bytestring &length= 16;
+} &let {
+	proc: bool = $context.connection.handle_auth_response();
+};
+
+type RFBSecurityARDChallenge = record {
+	challenge: bytestring &restofdata;
+} &let {
+	proc: bool = $context.connection.handle_ard_challenge();
+}
+
+type RFBSecurityARDResponse = record {
+	response: bytestring &restofdata;
+} &let {
+	proc: bool = $context.connection.handle_ard_response();
+}
+
+type RFBClientInit = record {
+	shared_flag: uint8;
+} &let {
+	proc: bool = $context.connection.handle_client_init(this);
+	proc2: bool = $context.flow.proc_rfb_share_flag(shared_flag);
+}
+
+type RFBServerInit = record {
+	width: uint16;
+	height: uint16;
+	pixel_format: bytestring &length= 16;
+	len : uint32;
+	name: bytestring &length = len;
+} &let {
+	proc: bool = $context.connection.handle_server_init(this);
+	proc2: bool = $context.flow.proc_handle_server_params(this);
+};
+
+type RFB_PDU_request = record {
+	request: case state of {
+		AWAITING_CLIENT_BANNER -> version: RFBProtocolVersion(true);
+		AWAITING_CLIENT_RESPONSE -> response: RFBVNCAuthenticationResponse;
+		AWAITING_CLIENT_SHARE_FLAG -> shareflag: RFBClientInit;
+		AWAITING_CLIENT_AUTH_TYPE_SELECTED37 -> authtype: RFBAuthTypeSelected;
+		AWAITING_CLIENT_ARD_RESPONSE -> ard_response: RFBSecurityARDResponse;
+		RFB_MESSAGE -> ignore: bytestring &restofdata &transient;
+		default -> data: bytestring &restofdata &transient;
+	} &requires(state);
+	} &let {
+		state: uint8 = $context.connection.get_state(true);
+};
+
+type RFB_PDU_response = record {
+	request: case rstate of {
+		AWAITING_SERVER_BANNER -> version: RFBProtocolVersion(false);
+		AWAITING_SERVER_AUTH_TYPES -> auth_types: RFBSecurityTypes;
+		AWAITING_SERVER_AUTH_TYPES37 -> auth_types37: RFBSecurityTypes37;
+		AWAITING_SERVER_CHALLENGE -> challenge: RFBVNCAuthenticationRequest;
+		AWAITING_SERVER_AUTH_RESULT -> authresult : RFBSecurityResult;
+		AWAITING_SERVER_ARD_CHALLENGE -> ard_challenge: RFBSecurityARDChallenge;
+		AWAITING_SERVER_PARAMS -> serverinit: RFBServerInit;
+		RFB_MESSAGE -> ignore: bytestring &restofdata &transient;
+		default -> data: bytestring &restofdata &transient;
+	} &requires(rstate);
+	} &let {
+		rstate: uint8 = $context.connection.get_state(false);
+};
+
+type RFB_PDU(is_orig: bool) = record {
+	payload: case is_orig of {
+		true -> request: RFB_PDU_request;
+		false -> response: RFB_PDU_response;
+	};
+} &byteorder = bigendian;

src/analyzer/protocol/rfb/rfb.pac

@@ -0,0 +1,30 @@
+# Analyzer for Parser for rfb (VNC)
+#  - rfb-protocol.pac: describes the rfb protocol messages
+#  - rfb-analyzer.pac: describes the rfb analyzer code
+
+%include binpac.pac
+%include bro.pac
+
+%extern{
+	#include "events.bif.h"
+%}
+
+analyzer RFB withcontext {
+	connection: RFB_Conn;
+	flow:       RFB_Flow;
+};
+
+# Our connection consists of two flows, one in each direction.
+connection RFB_Conn(bro_analyzer: BroAnalyzer) {
+	upflow   = RFB_Flow(true);
+	downflow = RFB_Flow(false);
+};
+
+%include rfb-protocol.pac
+
+# Now we define the flow:
+flow RFB_Flow(is_orig: bool) {
+	datagram = RFB_PDU(is_orig) withcontext(connection, this);
+};
+
+%include rfb-analyzer.pac

src/analyzer/protocol/sip/sip-analyzer.pac

@@ -18,7 +18,6 @@ refine flow SIP_Flow += {
 
 	function proc_sip_request(method: bytestring, uri: bytestring, vers: SIP_Version): bool
 		%{
-		connection()->bro_analyzer()->ProtocolConfirmation();
 		if ( sip_request )
 			{
 			BifEvent::generate_sip_request(connection()->bro_analyzer(), connection()->bro_analyzer()->Conn(),

src/analyzer/protocol/snmp/snmp-analyzer.pac

@@ -373,10 +373,12 @@ refine connection SNMP_Conn += {
 
 	function proc_header(rec: Header): bool
 		%{
+		if ( ! ${rec.is_orig} )
+			bro_analyzer()->ProtocolConfirmation();
+ 
 		if ( rec->unknown() )
 			return false;
 
-		bro_analyzer()->ProtocolConfirmation();
 		return true;
 		%}
 

src/bro.bif

@@ -3458,6 +3458,26 @@ function get_current_packet%(%) : pcap_packet
 	return pkt;
 	%}
 
+## Function to get the raw headers of the currently processed packet.
+##
+## Returns: The :bro:type:`raw_pkt_hdr` record containing the Layer 2, 3 and
+##          4 headers of the currently processed packet.
+##
+## .. bro:see:: raw_pkt_hdr get_current_packet
+function get_current_packet_header%(%) : raw_pkt_hdr
+	%{
+	const Packet* p;
+
+	if ( current_pktsrc &&
+	     current_pktsrc->GetCurrentPacket(&p) )
+		{
+		return p->BuildPktHdrVal();
+		}
+
+	RecordVal* hdr = new RecordVal(raw_pkt_hdr_type);
+	return hdr;
+	%}
+
 ## Writes a given packet to a file.
 ##
 ## pkt: The PCAP packet.

src/broker/Data.cc

@@ -539,7 +539,7 @@ broker::util::optional<broker::data> bro_broker::val_to_data(Val* v)
 		return {rval};
 		}
 	default:
-		reporter->Error("unsupported BrokerComm::Data type: %s",
+		reporter->Error("unsupported Broker::Data type: %s",
 		                type_name(v->Type()->Tag()));
 		break;
 	}
@@ -549,7 +549,7 @@ broker::util::optional<broker::data> bro_broker::val_to_data(Val* v)
 
 RecordVal* bro_broker::make_data_val(Val* v)
 	{
-	auto rval = new RecordVal(BifType::Record::BrokerComm::Data);
+	auto rval = new RecordVal(BifType::Record::Broker::Data);
 	auto data = val_to_data(v);
 
 	if ( data )
@@ -560,7 +560,7 @@ RecordVal* bro_broker::make_data_val(Val* v)
 
 RecordVal* bro_broker::make_data_val(broker::data d)
 	{
-	auto rval = new RecordVal(BifType::Record::BrokerComm::Data);
+	auto rval = new RecordVal(BifType::Record::Broker::Data);
 	rval->Assign(0, new DataVal(move(d)));
 	return rval;
 	}
@@ -570,92 +570,92 @@ struct data_type_getter {
 
 	result_type operator()(bool a)
 		{
-		return new EnumVal(BifEnum::BrokerComm::BOOL,
-		                   BifType::Enum::BrokerComm::DataType);
+		return new EnumVal(BifEnum::Broker::BOOL,
+		                   BifType::Enum::Broker::DataType);
 		}
 
 	result_type operator()(uint64_t a)
 		{
-		return new EnumVal(BifEnum::BrokerComm::COUNT,
-		                   BifType::Enum::BrokerComm::DataType);
+		return new EnumVal(BifEnum::Broker::COUNT,
+		                   BifType::Enum::Broker::DataType);
 		}
 
 	result_type operator()(int64_t a)
 		{
-		return new EnumVal(BifEnum::BrokerComm::INT,
-		                   BifType::Enum::BrokerComm::DataType);
+		return new EnumVal(BifEnum::Broker::INT,
+		                   BifType::Enum::Broker::DataType);
 		}
 
 	result_type operator()(double a)
 		{
-		return new EnumVal(BifEnum::BrokerComm::DOUBLE,
-		                   BifType::Enum::BrokerComm::DataType);
+		return new EnumVal(BifEnum::Broker::DOUBLE,
+		                   BifType::Enum::Broker::DataType);
 		}
 
 	result_type operator()(const std::string& a)
 		{
-		return new EnumVal(BifEnum::BrokerComm::STRING,
-		                   BifType::Enum::BrokerComm::DataType);
+		return new EnumVal(BifEnum::Broker::STRING,
+		                   BifType::Enum::Broker::DataType);
 		}
 
 	result_type operator()(const broker::address& a)
 		{
-		return new EnumVal(BifEnum::BrokerComm::ADDR,
-		                   BifType::Enum::BrokerComm::DataType);
+		return new EnumVal(BifEnum::Broker::ADDR,
+		                   BifType::Enum::Broker::DataType);
 		}
 
 	result_type operator()(const broker::subnet& a)
 		{
-		return new EnumVal(BifEnum::BrokerComm::SUBNET,
-		                   BifType::Enum::BrokerComm::DataType);
+		return new EnumVal(BifEnum::Broker::SUBNET,
+		                   BifType::Enum::Broker::DataType);
 		}
 
 	result_type operator()(const broker::port& a)
 		{
-		return new EnumVal(BifEnum::BrokerComm::PORT,
-		                   BifType::Enum::BrokerComm::DataType);
+		return new EnumVal(BifEnum::Broker::PORT,
+		                   BifType::Enum::Broker::DataType);
 		}
 
 	result_type operator()(const broker::time_point& a)
 		{
-		return new EnumVal(BifEnum::BrokerComm::TIME,
-		                   BifType::Enum::BrokerComm::DataType);
+		return new EnumVal(BifEnum::Broker::TIME,
+		                   BifType::Enum::Broker::DataType);
 		}
 
 	result_type operator()(const broker::time_duration& a)
 		{
-		return new EnumVal(BifEnum::BrokerComm::INTERVAL,
-		                   BifType::Enum::BrokerComm::DataType);
+		return new EnumVal(BifEnum::Broker::INTERVAL,
+		                   BifType::Enum::Broker::DataType);
 		}
 
 	result_type operator()(const broker::enum_value& a)
 		{
-		return new EnumVal(BifEnum::BrokerComm::ENUM,
-		                   BifType::Enum::BrokerComm::DataType);
+		return new EnumVal(BifEnum::Broker::ENUM,
+		                   BifType::Enum::Broker::DataType);
 		}
 
 	result_type operator()(const broker::set& a)
 		{
-		return new EnumVal(BifEnum::BrokerComm::SET,
-		                   BifType::Enum::BrokerComm::DataType);
+		return new EnumVal(BifEnum::Broker::SET,
+		                   BifType::Enum::Broker::DataType);
 		}
 
 	result_type operator()(const broker::table& a)
 		{
-		return new EnumVal(BifEnum::BrokerComm::TABLE,
-		                   BifType::Enum::BrokerComm::DataType);
+		return new EnumVal(BifEnum::Broker::TABLE,
+		                   BifType::Enum::Broker::DataType);
 		}
 
 	result_type operator()(const broker::vector& a)
 		{
-		return new EnumVal(BifEnum::BrokerComm::VECTOR,
-		                   BifType::Enum::BrokerComm::DataType);
+		return new EnumVal(BifEnum::Broker::VECTOR,
+		                   BifType::Enum::Broker::DataType);
 		}
 
 	result_type operator()(const broker::record& a)
 		{
-		return new EnumVal(BifEnum::BrokerComm::RECORD,
-		                   BifType::Enum::BrokerComm::DataType);
+		return new EnumVal(BifEnum::Broker::RECORD,
+		                   BifType::Enum::Broker::DataType);
 		}
 };
 
@@ -670,7 +670,7 @@ broker::data& bro_broker::opaque_field_to_data(RecordVal* v, Frame* f)
 
 	if ( ! d )
 		reporter->RuntimeError(f->GetCall()->GetLocationInfo(),
-		                       "BrokerComm::Data's opaque field is not set");
+		                       "Broker::Data's opaque field is not set");
 
 	return static_cast<DataVal*>(d)->data;
 	}

src/broker/Data.h

@@ -21,25 +21,25 @@ extern OpaqueType* opaque_of_record_iterator;
 TransportProto to_bro_port_proto(broker::port::protocol tp);
 
 /**
- * Create a BrokerComm::Data value from a Bro value.
+ * Create a Broker::Data value from a Bro value.
  * @param v the Bro value to convert to a Broker data value.
- * @return a BrokerComm::Data value, where the optional field is set if the conversion
+ * @return a Broker::Data value, where the optional field is set if the conversion
  * was possible, else it is unset.
  */
 RecordVal* make_data_val(Val* v);
 
 /**
- * Create a BrokerComm::Data value from a Broker data value.
+ * Create a Broker::Data value from a Broker data value.
  * @param d the Broker value to wrap in an opaque type.
- * @return a BrokerComm::Data value that wraps the Broker value.
+ * @return a Broker::Data value that wraps the Broker value.
  */
 RecordVal* make_data_val(broker::data d);
 
 /**
- * Get the type of Broker data that BrokerComm::Data wraps.
- * @param v a BrokerComm::Data value.
+ * Get the type of Broker data that Broker::Data wraps.
+ * @param v a Broker::Data value.
  * @param frame used to get location info upon error.
- * @return a BrokerComm::DataType value.
+ * @return a Broker::DataType value.
  */
 EnumVal* get_data_type(RecordVal* v, Frame* frame);
 
@@ -141,8 +141,8 @@ struct type_name_getter {
 };
 
 /**
- * Retrieve Broker data value associated with a BrokerComm::Data Bro value.
- * @param v a BrokerComm::Data value.
+ * Retrieve Broker data value associated with a Broker::Data Bro value.
+ * @param v a Broker::Data value.
  * @param f used to get location information on error.
  * @return a reference to the wrapped Broker data value.  A runtime interpreter
  * exception is thrown if the the optional opaque value of \a v is not set.
@@ -183,9 +183,9 @@ inline T& require_data_type(RecordVal* v, TypeTag tag, Frame* f)
 	}
 
 /**
- * Convert a BrokerComm::Data Bro value to a Bro value of a given type.
+ * Convert a Broker::Data Bro value to a Bro value of a given type.
  * @tparam a type that a Broker data variant may contain.
- * @param v a BrokerComm::Data value.
+ * @param v a Broker::Data value.
  * @param tag a Bro type to convert to.
  * @param f used to get location information on error.
  * A runtime interpret exception is thrown if trying to access a type which

src/broker/Manager.cc

@@ -77,20 +77,20 @@ bool bro_broker::Manager::Enable(Val* broker_endpoint_flags)
 	if ( endpoint != nullptr )
 		return true;
 
-	auto send_flags_type = internal_type("BrokerComm::SendFlags")->AsRecordType();
+	auto send_flags_type = internal_type("Broker::SendFlags")->AsRecordType();
 	send_flags_self_idx = require_field(send_flags_type, "self");
 	send_flags_peers_idx = require_field(send_flags_type, "peers");
 	send_flags_unsolicited_idx = require_field(send_flags_type, "unsolicited");
 
 	log_id_type = internal_type("Log::ID")->AsEnumType();
 
-	bro_broker::opaque_of_data_type = new OpaqueType("BrokerComm::Data");
-	bro_broker::opaque_of_set_iterator = new OpaqueType("BrokerComm::SetIterator");
-	bro_broker::opaque_of_table_iterator = new OpaqueType("BrokerComm::TableIterator");
-	bro_broker::opaque_of_vector_iterator = new OpaqueType("BrokerComm::VectorIterator");
-	bro_broker::opaque_of_record_iterator = new OpaqueType("BrokerComm::RecordIterator");
-	bro_broker::opaque_of_store_handle = new OpaqueType("BrokerStore::Handle");
-	vector_of_data_type = new VectorType(internal_type("BrokerComm::Data")->Ref());
+	bro_broker::opaque_of_data_type = new OpaqueType("Broker::Data");
+	bro_broker::opaque_of_set_iterator = new OpaqueType("Broker::SetIterator");
+	bro_broker::opaque_of_table_iterator = new OpaqueType("Broker::TableIterator");
+	bro_broker::opaque_of_vector_iterator = new OpaqueType("Broker::VectorIterator");
+	bro_broker::opaque_of_record_iterator = new OpaqueType("Broker::RecordIterator");
+	bro_broker::opaque_of_store_handle = new OpaqueType("Broker::Handle");
+	vector_of_data_type = new VectorType(internal_type("Broker::Data")->Ref());
 
 	auto res = broker::init();
 
@@ -110,7 +110,7 @@ bool bro_broker::Manager::Enable(Val* broker_endpoint_flags)
 		}
 
 	const char* name;
-	auto name_from_script = internal_val("BrokerComm::endpoint_name")->AsString();
+	auto name_from_script = internal_val("Broker::endpoint_name")->AsString();
 
 	if ( name_from_script->Len() )
 		name = name_from_script->CheckString();
@@ -290,7 +290,7 @@ bool bro_broker::Manager::AutoEvent(string topic, Val* event, Val* flags)
 
 	if ( event->Type()->Tag() != TYPE_FUNC )
 		{
-		reporter->Error("BrokerComm::auto_event must operate on an event");
+		reporter->Error("Broker::auto_event must operate on an event");
 		return false;
 		}
 
@@ -298,7 +298,7 @@ bool bro_broker::Manager::AutoEvent(string topic, Val* event, Val* flags)
 
 	if ( event_val->Flavor() != FUNC_FLAVOR_EVENT )
 		{
-		reporter->Error("BrokerComm::auto_event must operate on an event");
+		reporter->Error("Broker::auto_event must operate on an event");
 		return false;
 		}
 
@@ -306,7 +306,7 @@ bool bro_broker::Manager::AutoEvent(string topic, Val* event, Val* flags)
 
 	if ( ! handler )
 		{
-		reporter->Error("BrokerComm::auto_event failed to lookup event '%s'",
+		reporter->Error("Broker::auto_event failed to lookup event '%s'",
 		                event_val->Name());
 		return false;
 		}
@@ -322,7 +322,7 @@ bool bro_broker::Manager::AutoEventStop(const string& topic, Val* event)
 
 	if ( event->Type()->Tag() != TYPE_FUNC )
 		{
-		reporter->Error("BrokerComm::auto_event_stop must operate on an event");
+		reporter->Error("Broker::auto_event_stop must operate on an event");
 		return false;
 		}
 
@@ -330,7 +330,7 @@ bool bro_broker::Manager::AutoEventStop(const string& topic, Val* event)
 
 	if ( event_val->Flavor() != FUNC_FLAVOR_EVENT )
 		{
-		reporter->Error("BrokerComm::auto_event_stop must operate on an event");
+		reporter->Error("Broker::auto_event_stop must operate on an event");
 		return false;
 		}
 
@@ -338,7 +338,7 @@ bool bro_broker::Manager::AutoEventStop(const string& topic, Val* event)
 
 	if ( ! handler )
 		{
-		reporter->Error("BrokerComm::auto_event_stop failed to lookup event '%s'",
+		reporter->Error("Broker::auto_event_stop failed to lookup event '%s'",
 		                event_val->Name());
 		return false;
 		}
@@ -353,7 +353,7 @@ RecordVal* bro_broker::Manager::MakeEventArgs(val_list* args)
 	if ( ! Enabled() )
 		return nullptr;
 
-	auto rval = new RecordVal(BifType::Record::BrokerComm::EventArgs);
+	auto rval = new RecordVal(BifType::Record::Broker::EventArgs);
 	auto arg_vec = new VectorVal(vector_of_data_type);
 	rval->Assign(1, arg_vec);
 	Func* func = 0;
@@ -368,7 +368,7 @@ RecordVal* bro_broker::Manager::MakeEventArgs(val_list* args)
 
 			if ( arg_val->Type()->Tag() != TYPE_FUNC )
 				{
-				reporter->Error("1st param of BrokerComm::event_args must be event");
+				reporter->Error("1st param of Broker::event_args must be event");
 				return rval;
 				}
 
@@ -376,7 +376,7 @@ RecordVal* bro_broker::Manager::MakeEventArgs(val_list* args)
 
 			if ( func->Flavor() != FUNC_FLAVOR_EVENT )
 				{
-				reporter->Error("1st param of BrokerComm::event_args must be event");
+				reporter->Error("1st param of Broker::event_args must be event");
 				return rval;
 				}
 
@@ -384,7 +384,7 @@ RecordVal* bro_broker::Manager::MakeEventArgs(val_list* args)
 
 			if ( num_args != args->length() - 1 )
 				{
-				reporter->Error("bad # of BrokerComm::event_args: got %d, expect %d",
+				reporter->Error("bad # of Broker::event_args: got %d, expect %d",
 				                args->length(), num_args + 1);
 				return rval;
 				}
@@ -398,7 +398,7 @@ RecordVal* bro_broker::Manager::MakeEventArgs(val_list* args)
 		if ( ! same_type((*args)[i]->Type(), expected_type) )
 			{
 			rval->Assign(0, 0);
-			reporter->Error("BrokerComm::event_args param %d type mismatch", i);
+			reporter->Error("Broker::event_args param %d type mismatch", i);
 			return rval;
 			}
 
@@ -408,7 +408,7 @@ RecordVal* bro_broker::Manager::MakeEventArgs(val_list* args)
 			{
 			Unref(data_val);
 			rval->Assign(0, 0);
-			reporter->Error("BrokerComm::event_args unsupported event/params");
+			reporter->Error("Broker::event_args unsupported event/params");
 			return rval;
 			}
 
@@ -584,7 +584,7 @@ struct response_converter {
 		case broker::store::query::tag::lookup:
 			// A boolean result means the key doesn't exist (if it did, then
 			// the result would contain the broker::data value, not a bool).
-			return new RecordVal(BifType::Record::BrokerComm::Data);
+			return new RecordVal(BifType::Record::Broker::Data);
 		default:
 			return bro_broker::make_data_val(broker::data{d});
 		}
@@ -639,36 +639,36 @@ void bro_broker::Manager::Process()
 		{
 		switch ( u.status ) {
 		case broker::outgoing_connection_status::tag::established:
-			if ( BrokerComm::outgoing_connection_established )
+			if ( Broker::outgoing_connection_established )
 				{
 				val_list* vl = new val_list;
 				vl->append(new StringVal(u.relation.remote_tuple().first));
 				vl->append(new PortVal(u.relation.remote_tuple().second,
 				                       TRANSPORT_TCP));
 				vl->append(new StringVal(u.peer_name));
-				mgr.QueueEvent(BrokerComm::outgoing_connection_established, vl);
+				mgr.QueueEvent(Broker::outgoing_connection_established, vl);
 				}
 			break;
 
 		case broker::outgoing_connection_status::tag::disconnected:
-			if ( BrokerComm::outgoing_connection_broken )
+			if ( Broker::outgoing_connection_broken )
 				{
 				val_list* vl = new val_list;
 				vl->append(new StringVal(u.relation.remote_tuple().first));
 				vl->append(new PortVal(u.relation.remote_tuple().second,
 				                       TRANSPORT_TCP));
-				mgr.QueueEvent(BrokerComm::outgoing_connection_broken, vl);
+				mgr.QueueEvent(Broker::outgoing_connection_broken, vl);
 				}
 			break;
 
 		case broker::outgoing_connection_status::tag::incompatible:
-			if ( BrokerComm::outgoing_connection_incompatible )
+			if ( Broker::outgoing_connection_incompatible )
 				{
 				val_list* vl = new val_list;
 				vl->append(new StringVal(u.relation.remote_tuple().first));
 				vl->append(new PortVal(u.relation.remote_tuple().second,
 				                       TRANSPORT_TCP));
-				mgr.QueueEvent(BrokerComm::outgoing_connection_incompatible, vl);
+				mgr.QueueEvent(Broker::outgoing_connection_incompatible, vl);
 				}
 			break;
 
@@ -684,20 +684,20 @@ void bro_broker::Manager::Process()
 		{
 		switch ( u.status ) {
 		case broker::incoming_connection_status::tag::established:
-			if ( BrokerComm::incoming_connection_established )
+			if ( Broker::incoming_connection_established )
 				{
 				val_list* vl = new val_list;
 				vl->append(new StringVal(u.peer_name));
-				mgr.QueueEvent(BrokerComm::incoming_connection_established, vl);
+				mgr.QueueEvent(Broker::incoming_connection_established, vl);
 				}
 			break;
 
 		case broker::incoming_connection_status::tag::disconnected:
-			if ( BrokerComm::incoming_connection_broken )
+			if ( Broker::incoming_connection_broken )
 				{
 				val_list* vl = new val_list;
 				vl->append(new StringVal(u.peer_name));
-				mgr.QueueEvent(BrokerComm::incoming_connection_broken, vl);
+				mgr.QueueEvent(Broker::incoming_connection_broken, vl);
 				}
 			break;
 
@@ -718,7 +718,7 @@ void bro_broker::Manager::Process()
 
 		ps.second.received += print_messages.size();
 
-		if ( ! BrokerComm::print_handler )
+		if ( ! Broker::print_handler )
 			continue;
 
 		for ( auto& pm : print_messages )
@@ -741,7 +741,7 @@ void bro_broker::Manager::Process()
 
 			val_list* vl = new val_list;
 			vl->append(new StringVal(move(*msg)));
-			mgr.QueueEvent(BrokerComm::print_handler, vl);
+			mgr.QueueEvent(Broker::print_handler, vl);
 			}
 		}
 

src/broker/Manager.h

@@ -63,7 +63,7 @@ public:
 	/**
 	 * Enable use of communication.
 	 * @param flags used to tune the local Broker endpoint's behavior.
-	 * See the BrokerComm::EndpointFlags record type.
+	 * See the Broker::EndpointFlags record type.
 	 * @return true if communication is successfully initialized.
 	 */
 	bool Enable(Val* flags);
@@ -122,7 +122,7 @@ public:
 	 * of this topic name.
 	 * @param msg the string to send to peers.
 	 * @param flags tune the behavior of how the message is send.
-	 * See the BrokerComm::SendFlags record type.
+	 * See the Broker::SendFlags record type.
 	 * @return true if the message is sent successfully.
 	 */
 	bool Print(std::string topic, std::string msg, Val* flags);
@@ -135,7 +135,7 @@ public:
 	 * @param msg the event to send to peers, which is the name of the event
 	 * as a string followed by all of its arguments.
 	 * @param flags tune the behavior of how the message is send.
-	 * See the BrokerComm::SendFlags record type.
+	 * See the Broker::SendFlags record type.
 	 * @return true if the message is sent successfully.
 	 */
 	bool Event(std::string topic, broker::message msg, int flags);
@@ -146,9 +146,9 @@ public:
 	 * Peers advertise interest by registering a subscription to some prefix
 	 * of this topic name.
 	 * @param args the event and its arguments to send to peers.  See the
-	 * BrokerComm::EventArgs record type.
+	 * Broker::EventArgs record type.
 	 * @param flags tune the behavior of how the message is send.
-	 * See the BrokerComm::SendFlags record type.
+	 * See the Broker::SendFlags record type.
 	 * @return true if the message is sent successfully.
 	 */
 	bool Event(std::string topic, RecordVal* args, Val* flags);
@@ -160,7 +160,7 @@ public:
 	 * @param columns the data which comprises the log entry.
 	 * @param info the record type corresponding to the log's columns.
 	 * @param flags tune the behavior of how the message is send.
-	 * See the BrokerComm::SendFlags record type.
+	 * See the Broker::SendFlags record type.
 	 * @return true if the message is sent successfully.
 	 */
 	bool Log(EnumVal* stream_id, RecordVal* columns, RecordType* info,
@@ -174,7 +174,7 @@ public:
 	 * of this topic name.
 	 * @param event a Bro event value.
 	 * @param flags tune the behavior of how the message is send.
-	 * See the BrokerComm::SendFlags record type.
+	 * See the Broker::SendFlags record type.
 	 * @return true if automatic event sending is now enabled.
 	 */
 	bool AutoEvent(std::string topic, Val* event, Val* flags);
@@ -320,7 +320,7 @@ public:
 	Stats ConsumeStatistics();
 
 	/**
-	 * Convert BrokerComm::SendFlags to int flags for use with broker::send().
+	 * Convert Broker::SendFlags to int flags for use with broker::send().
 	 */
 	static int send_flags_to_int(Val* flags);
 
@@ -335,7 +335,7 @@ private:
 	void Process() override;
 
 	const char* Tag() override
-		{ return "BrokerComm::Manager"; }
+		{ return "Broker::Manager"; }
 
 	broker::endpoint& Endpoint()
 		{ return *endpoint; }

src/broker/Store.cc

@@ -14,12 +14,12 @@ OpaqueType* bro_broker::opaque_of_store_handle;
 
 bro_broker::StoreHandleVal::StoreHandleVal(broker::store::identifier id,
                                      bro_broker::StoreType arg_type,
-                                     broker::util::optional<BifEnum::BrokerStore::BackendType> arg_back,
+                                     broker::util::optional<BifEnum::Broker::BackendType> arg_back,
                                      RecordVal* backend_options, std::chrono::duration<double> resync)
 	: OpaqueVal(opaque_of_store_handle),
 	  store(), store_type(arg_type), backend_type(arg_back)
 	{
-	using BifEnum::BrokerStore::BackendType;
+	using BifEnum::Broker::BackendType;
 	std::unique_ptr<broker::store::backend> backend;
 
 	if ( backend_type )
@@ -91,7 +91,7 @@ bro_broker::StoreHandleVal::StoreHandleVal(broker::store::identifier id,
 
 void bro_broker::StoreHandleVal::ValDescribe(ODesc* d) const
 	{
-	using BifEnum::BrokerStore::BackendType;
+	using BifEnum::Broker::BackendType;
 	d->Add("broker::store::");
 
 	switch ( store_type ) {

src/broker/Store.h

@@ -25,9 +25,9 @@ enum StoreType {
 };
 
 /**
- * Create a BrokerStore::QueryStatus value.
+ * Create a Broker::QueryStatus value.
  * @param success whether the query status should be set to success or failure.
- * @return a BrokerStore::QueryStatus value.
+ * @return a Broker::QueryStatus value.
  */
 inline EnumVal* query_status(bool success)
 	{
@@ -37,34 +37,34 @@ inline EnumVal* query_status(bool success)
 
 	if ( ! store_query_status )
 		{
-		store_query_status = internal_type("BrokerStore::QueryStatus")->AsEnumType();
-		success_val = store_query_status->Lookup("BrokerStore", "SUCCESS");
-		failure_val = store_query_status->Lookup("BrokerStore", "FAILURE");
+		store_query_status = internal_type("Broker::QueryStatus")->AsEnumType();
+		success_val = store_query_status->Lookup("Broker", "SUCCESS");
+		failure_val = store_query_status->Lookup("Broker", "FAILURE");
 		}
 
 	return new EnumVal(success ? success_val : failure_val, store_query_status);
 	}
 
 /**
- * @return a BrokerStore::QueryResult value that has a BrokerStore::QueryStatus indicating
+ * @return a Broker::QueryResult value that has a Broker::QueryStatus indicating
  * a failure.
  */
 inline RecordVal* query_result()
 	{
-	auto rval = new RecordVal(BifType::Record::BrokerStore::QueryResult);
+	auto rval = new RecordVal(BifType::Record::Broker::QueryResult);
 	rval->Assign(0, query_status(false));
-	rval->Assign(1, new RecordVal(BifType::Record::BrokerComm::Data));
+	rval->Assign(1, new RecordVal(BifType::Record::Broker::Data));
 	return rval;
 	}
 
 /**
  * @param data the result of the query.
- * @return a BrokerStore::QueryResult value that has a BrokerStore::QueryStatus indicating
+ * @return a Broker::QueryResult value that has a Broker::QueryStatus indicating
  * a success.
  */
 inline RecordVal* query_result(RecordVal* data)
 	{
-	auto rval = new RecordVal(BifType::Record::BrokerStore::QueryResult);
+	auto rval = new RecordVal(BifType::Record::Broker::QueryResult);
 	rval->Assign(0, query_status(true));
 	rval->Assign(1, data);
 	return rval;
@@ -130,7 +130,7 @@ public:
 
 	StoreHandleVal(broker::store::identifier id,
 		       bro_broker::StoreType arg_type,
-		       broker::util::optional<BifEnum::BrokerStore::BackendType> arg_back,
+		       broker::util::optional<BifEnum::Broker::BackendType> arg_back,
 		       RecordVal* backend_options,
 		       std::chrono::duration<double> resync = std::chrono::seconds(1));
 
@@ -140,7 +140,7 @@ public:
 
 	broker::store::frontend* store;
 	bro_broker::StoreType store_type;
-	broker::util::optional<BifEnum::BrokerStore::BackendType> backend_type;
+	broker::util::optional<BifEnum::Broker::BackendType> backend_type;
 
 protected:
 

src/broker/comm.bif

@@ -5,139 +5,102 @@
 #include "broker/Manager.h"
 %%}
 
-module BrokerComm;
+module Broker;
 
-type BrokerComm::EndpointFlags: record;
+type Broker::EndpointFlags: record;
 
-## Enable use of communication.
-##
-## flags: used to tune the local Broker endpoint behavior.
-##
-## Returns: true if communication is successfully initialized.
-function BrokerComm::enable%(flags: EndpointFlags &default = EndpointFlags()%): bool
+function Broker::__enable%(flags: EndpointFlags%): bool
 	%{
 	return new Val(broker_mgr->Enable(flags), TYPE_BOOL);
 	%}
 
-## Changes endpoint flags originally supplied to :bro:see:`BrokerComm::enable`.
-##
-## flags: the new endpoint behavior flags to use.
-##
-## Returns: true if flags were changed.
-function BrokerComm::set_endpoint_flags%(flags: EndpointFlags &default = EndpointFlags()%): bool
+function Broker::__set_endpoint_flags%(flags: EndpointFlags%): bool
 	%{
 	return new Val(broker_mgr->SetEndpointFlags(flags), TYPE_BOOL);
 	%}
 
-## Allow sending messages to peers if associated with the given topic.
-## This has no effect if auto publication behavior is enabled via the flags
-## supplied to :bro:see:`BrokerComm::enable` or :bro:see:`BrokerComm::set_endpoint_flags`.
-##
-## topic: a topic to allow messages to be published under.
-##
-## Returns: true if successful.
-function BrokerComm::publish_topic%(topic: string%): bool
+function Broker::__publish_topic%(topic: string%): bool
 	%{
 	return new Val(broker_mgr->PublishTopic(topic->CheckString()), TYPE_BOOL);
 	%}
 
-## Disallow sending messages to peers if associated with the given topic.
-## This has no effect if auto publication behavior is enabled via the flags
-## supplied to :bro:see:`BrokerComm::enable` or :bro:see:`BrokerComm::set_endpoint_flags`.
-##
-## topic: a topic to disallow messages to be published under.
-##
-## Returns: true if successful.
-function BrokerComm::unpublish_topic%(topic: string%): bool
+function Broker::__unpublish_topic%(topic: string%): bool
 	%{
 	return new Val(broker_mgr->UnpublishTopic(topic->CheckString()), TYPE_BOOL);
 	%}
 
 ## Allow advertising interest in the given topic to peers.
 ## This has no effect if auto advertise behavior is enabled via the flags
-## supplied to :bro:see:`BrokerComm::enable` or :bro:see:`BrokerComm::set_endpoint_flags`.
+## supplied to :bro:see:`Broker::enable` or :bro:see:`Broker::set_endpoint_flags`.
 ##
 ## topic: a topic to allow advertising interest/subscription to peers.
 ##
 ## Returns: true if successful.
-function BrokerComm::advertise_topic%(topic: string%): bool
+function Broker::advertise_topic%(topic: string%): bool
 	%{
 	return new Val(broker_mgr->AdvertiseTopic(topic->CheckString()), TYPE_BOOL);
 	%}
 
 ## Disallow advertising interest in the given topic to peers.
 ## This has no effect if auto advertise behavior is enabled via the flags
-## supplied to :bro:see:`BrokerComm::enable` or :bro:see:`BrokerComm::set_endpoint_flags`.
+## supplied to :bro:see:`Broker::enable` or :bro:see:`Broker::set_endpoint_flags`.
 ##
 ## topic: a topic to disallow advertising interest/subscription to peers.
 ##
 ## Returns: true if successful.
-function BrokerComm::unadvertise_topic%(topic: string%): bool
+function Broker::unadvertise_topic%(topic: string%): bool
 	%{
 	return new Val(broker_mgr->UnadvertiseTopic(topic->CheckString()), TYPE_BOOL);
 	%}
 
 ## Generated when a connection has been established due to a previous call
-## to :bro:see:`BrokerComm::connect`.
+## to :bro:see:`Broker::connect`.
 ##
 ## peer_address: the address used to connect to the peer.
 ##
 ## peer_port: the port used to connect to the peer.
 ##
 ## peer_name: the name by which the peer identified itself.
-event BrokerComm::outgoing_connection_established%(peer_address: string,
+event Broker::outgoing_connection_established%(peer_address: string,
                                              peer_port: port,
                                              peer_name: string%);
 
 ## Generated when a previously established connection becomes broken.
 ## Reconnection will automatically be attempted at a frequency given
-## by the original call to :bro:see:`BrokerComm::connect`.
+## by the original call to :bro:see:`Broker::connect`.
 ##
 ## peer_address: the address used to connect to the peer.
 ##
 ## peer_port: the port used to connect to the peer.
 ##
-## .. bro:see:: BrokerComm::outgoing_connection_established
-event BrokerComm::outgoing_connection_broken%(peer_address: string,
+## .. bro:see:: Broker::outgoing_connection_established
+event Broker::outgoing_connection_broken%(peer_address: string,
                                         peer_port: port%);
 
-## Generated when a connection via :bro:see:`BrokerComm::connect` has failed
+## Generated when a connection via :bro:see:`Broker::connect` has failed
 ## because the remote side is incompatible.
 ##
 ## peer_address: the address used to connect to the peer.
 ##
 ## peer_port: the port used to connect to the peer.
-event BrokerComm::outgoing_connection_incompatible%(peer_address: string,
+event Broker::outgoing_connection_incompatible%(peer_address: string,
                                               peer_port: port%);
 
 ## Generated when a peer has established a connection with this process
-## as a result of previously performing a :bro:see:`BrokerComm::listen`.
+## as a result of previously performing a :bro:see:`Broker::listen`.
 ##
 ## peer_name: the name by which the peer identified itself.
-event BrokerComm::incoming_connection_established%(peer_name: string%);
+event Broker::incoming_connection_established%(peer_name: string%);
 
 ## Generated when a peer that previously established a connection with this
 ## process becomes disconnected.
 ##
 ## peer_name: the name by which the peer identified itself.
 ##
-## .. bro:see:: BrokerComm::incoming_connection_established
-event BrokerComm::incoming_connection_broken%(peer_name: string%);
+## .. bro:see:: Broker::incoming_connection_established
+event Broker::incoming_connection_broken%(peer_name: string%);
 
-## Listen for remote connections.
-##
-## p: the TCP port to listen on.
-##
-## a: an address string on which to accept connections, e.g.
-##    "127.0.0.1".  An empty string refers to @p INADDR_ANY.
-##
-## reuse: equivalent to behavior of SO_REUSEADDR.
-##
-## Returns: true if the local endpoint is now listening for connections.
-##
-## .. bro:see:: BrokerComm::incoming_connection_established
-function BrokerComm::listen%(p: port, a: string &default = "",
-					   reuse: bool &default = T%): bool
+function Broker::__listen%(p: port, a: string, reuse: bool%): bool
 	%{
 	if ( ! p->IsTCP() )
 		{
@@ -150,22 +113,7 @@ function BrokerComm::listen%(p: port, a: string &default = "",
 	return new Val(rval, TYPE_BOOL);
 	%}
 
-## Initiate a remote connection.
-##
-## a: an address to connect to, e.g. "localhost" or "127.0.0.1".
-##
-## p: the TCP port on which the remote side is listening.
-##
-## retry: an interval at which to retry establishing the
-##        connection with the remote peer if it cannot be made initially, or
-##        if it ever becomes disconnected.
-##
-## Returns: true if it's possible to try connecting with the peer and
-##          it's a new peer.  The actual connection may not be established
-##          until a later point in time.
-##
-## .. bro:see:: BrokerComm::outgoing_connection_established
-function BrokerComm::connect%(a: string, p: port, retry: interval%): bool
+function Broker::__connect%(a: string, p: port, retry: interval%): bool
 	%{
 	if ( ! p->IsTCP() )
 		{
@@ -178,15 +126,7 @@ function BrokerComm::connect%(a: string, p: port, retry: interval%): bool
 	return new Val(rval, TYPE_BOOL);
 	%}
 
-## Remove a remote connection.
-##
-## a: the address used in previous successful call to :bro:see:`BrokerComm::connect`.
-##
-## p: the port used in previous successful call to :bro:see:`BrokerComm::connect`.
-##
-## Returns: true if the arguments match a previously successful call to
-##          :bro:see:`BrokerComm::connect`.
-function BrokerComm::disconnect%(a: string, p: port%): bool
+function Broker::__disconnect%(a: string, p: port%): bool
 	%{
 	if ( ! p->IsTCP() )
 		{

src/broker/data.bif

@@ -5,9 +5,9 @@
 #include "broker/Data.h"
 %%}
 
-module BrokerComm;
+module Broker;
 
-## Enumerates the possible types that :bro:see:`BrokerComm::Data` may be in
+## Enumerates the possible types that :bro:see:`Broker::Data` may be in
 ## terms of Bro data types.
 enum DataType %{
 	BOOL,
@@ -27,97 +27,48 @@ enum DataType %{
 	RECORD,
 %}
 
-type BrokerComm::Data: record;
+type Broker::Data: record;
 
-type BrokerComm::TableItem: record;
+type Broker::TableItem: record;
 
-## Convert any Bro value to communication data.
-##
-## d: any Bro value to attempt to convert (not all types are supported).
-##
-## Returns: the converted communication data.  The returned record's optional
-##          field will not be set if the conversion was not possible (this can
-##          happen if the Bro data type does not support being converted to
-##          communication data).
-function BrokerComm::data%(d: any%): BrokerComm::Data
+function Broker::__data%(d: any%): Broker::Data
 	%{
 	return bro_broker::make_data_val(d);
 	%}
 
-## Retrieve the type of data associated with communication data.
-##
-## d: the communication data.
-##
-## Returns: the data type associated with the communication data.
-function BrokerComm::data_type%(d: BrokerComm::Data%): BrokerComm::DataType
+function Broker::__data_type%(d: Broker::Data%): Broker::DataType
 	%{
 	return bro_broker::get_data_type(d->AsRecordVal(), frame);
 	%}
 
-## Convert communication data with a type of :bro:see:`BrokerComm::BOOL` to
-## an actual Bro value.
-##
-## d: the communication data to convert.
-##
-## Returns: the value retrieved from the communication data.
-function BrokerComm::refine_to_bool%(d: BrokerComm::Data%): bool
+function Broker::__refine_to_bool%(d: Broker::Data%): bool
 	%{
 	return bro_broker::refine<bool>(d->AsRecordVal(), TYPE_BOOL, frame);
 	%}
 
-## Convert communication data with a type of :bro:see:`BrokerComm::INT` to
-## an actual Bro value.
-##
-## d: the communication data to convert.
-##
-## Returns: the value retrieved from the communication data.
-function BrokerComm::refine_to_int%(d: BrokerComm::Data%): int
+function Broker::__refine_to_int%(d: Broker::Data%): int
 	%{
 	return bro_broker::refine<int64_t>(d->AsRecordVal(), TYPE_INT, frame);
 	%}
 
-## Convert communication data with a type of :bro:see:`BrokerComm::COUNT` to
-## an actual Bro value.
-##
-## d: the communication data to convert.
-##
-## Returns: the value retrieved from the communication data.
-function BrokerComm::refine_to_count%(d: BrokerComm::Data%): count
+function Broker::__refine_to_count%(d: Broker::Data%): count
 	%{
 	return bro_broker::refine<uint64_t>(d->AsRecordVal(), TYPE_COUNT, frame);
 	%}
 
-## Convert communication data with a type of :bro:see:`BrokerComm::DOUBLE` to
-## an actual Bro value.
-##
-## d: the communication data to convert.
-##
-## Returns: the value retrieved from the communication data.
-function BrokerComm::refine_to_double%(d: BrokerComm::Data%): double
+function Broker::__refine_to_double%(d: Broker::Data%): double
 	%{
 	return bro_broker::refine<double>(d->AsRecordVal(), TYPE_DOUBLE, frame);
 	%}
 
-## Convert communication data with a type of :bro:see:`BrokerComm::STRING` to
-## an actual Bro value.
-##
-## d: the communication data to convert.
-##
-## Returns: the value retrieved from the communication data.
-function BrokerComm::refine_to_string%(d: BrokerComm::Data%): string
+function Broker::__refine_to_string%(d: Broker::Data%): string
 	%{
 	return new StringVal(bro_broker::require_data_type<std::string>(d->AsRecordVal(),
 												              TYPE_STRING,
 															  frame));
 	%}
 
-## Convert communication data with a type of :bro:see:`BrokerComm::ADDR` to
-## an actual Bro value.
-##
-## d: the communication data to convert.
-##
-## Returns: the value retrieved from the communication data.
-function BrokerComm::refine_to_addr%(d: BrokerComm::Data%): addr
+function Broker::__refine_to_addr%(d: Broker::Data%): addr
 	%{
 	auto& a = bro_broker::require_data_type<broker::address>(d->AsRecordVal(),
 													   TYPE_ADDR, frame);
@@ -125,13 +76,7 @@ function BrokerComm::refine_to_addr%(d: BrokerComm::Data%): addr
 	return new AddrVal(IPAddr(*bits));
 	%}
 
-## Convert communication data with a type of :bro:see:`BrokerComm::SUBNET` to
-## an actual Bro value.
-##
-## d: the communication data to convert.
-##
-## Returns: the value retrieved from the communication data.
-function BrokerComm::refine_to_subnet%(d: BrokerComm::Data%): subnet
+function Broker::__refine_to_subnet%(d: Broker::Data%): subnet
 	%{
 	auto& a = bro_broker::require_data_type<broker::subnet>(d->AsRecordVal(),
 													  TYPE_SUBNET, frame);
@@ -139,71 +84,40 @@ function BrokerComm::refine_to_subnet%(d: BrokerComm::Data%): subnet
 	return new SubNetVal(IPPrefix(IPAddr(*bits), a.length()));
 	%}
 
-## Convert communication data with a type of :bro:see:`BrokerComm::PORT` to
-## an actual Bro value.
-##
-## d: the communication data to convert.
-##
-## Returns: the value retrieved from the communication data.
-function BrokerComm::refine_to_port%(d: BrokerComm::Data%): port
+function Broker::__refine_to_port%(d: Broker::Data%): port
 	%{
 	auto& a = bro_broker::require_data_type<broker::port>(d->AsRecordVal(),
 													TYPE_SUBNET, frame);
 	return new PortVal(a.number(), bro_broker::to_bro_port_proto(a.type()));
 	%}
 
-## Convert communication data with a type of :bro:see:`BrokerComm::TIME` to
-## an actual Bro value.
-##
-## d: the communication data to convert.
-##
-## Returns: the value retrieved from the communication data.
-function BrokerComm::refine_to_time%(d: BrokerComm::Data%): time
+function Broker::__refine_to_time%(d: Broker::Data%): time
 	%{
 	auto v = bro_broker::require_data_type<broker::time_point>(d->AsRecordVal(),
 														TYPE_TIME, frame).value;
 	return new Val(v, TYPE_TIME);
 	%}
 
-## Convert communication data with a type of :bro:see:`BrokerComm::INTERVAL` to
-## an actual Bro value.
-##
-## d: the communication data to convert.
-##
-## Returns: the value retrieved from the communication data.
-function BrokerComm::refine_to_interval%(d: BrokerComm::Data%): interval
+function Broker::__refine_to_interval%(d: Broker::Data%): interval
 	%{
 	auto v = bro_broker::require_data_type<broker::time_duration>(d->AsRecordVal(),
 														TYPE_TIME, frame).value;
 	return new Val(v, TYPE_INTERVAL);
 	%}
 
-## Convert communication data with a type of :bro:see:`BrokerComm::ENUM` to
-## the name of the enum value.  :bro:see:`lookup_ID` may be used to convert
-## the name to the actual enum value.
-##
-## d: the communication data to convert.
-##
-## Returns: the enum name retrieved from the communication data.
-function BrokerComm::refine_to_enum_name%(d: BrokerComm::Data%): string
+function Broker::__refine_to_enum_name%(d: Broker::Data%): string
 	%{
 	auto& v = bro_broker::require_data_type<broker::enum_value>(d->AsRecordVal(),
 														TYPE_ENUM, frame).name;
 	return new StringVal(v);
 	%}
 
-## Create communication data of type "set".
-function BrokerComm::set_create%(%): BrokerComm::Data
+function Broker::__set_create%(%): Broker::Data
 	%{
 	return bro_broker::make_data_val(broker::set());
 	%}
 
-## Remove all elements within a set.
-##
-## s: the set to clear.
-##
-## Returns: always true.
-function BrokerComm::set_clear%(s: BrokerComm::Data%): bool
+function Broker::__set_clear%(s: Broker::Data%): bool
 	%{
 	auto& v = bro_broker::require_data_type<broker::set>(s->AsRecordVal(), TYPE_TABLE,
 												   frame);
@@ -211,26 +125,14 @@ function BrokerComm::set_clear%(s: BrokerComm::Data%): bool
 	return new Val(true, TYPE_BOOL);
 	%}
 
-## Get the number of elements within a set.
-##
-## s: the set to query.
-##
-## Returns: the number of elements in the set.
-function BrokerComm::set_size%(s: BrokerComm::Data%): count
+function Broker::__set_size%(s: Broker::Data%): count
 	%{
 	auto& v = bro_broker::require_data_type<broker::set>(s->AsRecordVal(), TYPE_TABLE,
 												   frame);
 	return new Val(static_cast<uint64_t>(v.size()), TYPE_COUNT);
 	%}
 
-## Check if a set contains a particular element.
-##
-## s: the set to query.
-##
-## key: the element to check for existence.
-##
-## Returns: true if the key exists in the set.
-function BrokerComm::set_contains%(s: BrokerComm::Data, key: BrokerComm::Data%): bool
+function Broker::__set_contains%(s: Broker::Data, key: Broker::Data%): bool
 	%{
 	auto& v = bro_broker::require_data_type<broker::set>(s->AsRecordVal(), TYPE_TABLE,
 												   frame);
@@ -238,14 +140,7 @@ function BrokerComm::set_contains%(s: BrokerComm::Data, key: BrokerComm::Data%):
 	return new Val(v.find(k) != v.end(), TYPE_BOOL);
 	%}
 
-## Insert an element into a set.
-##
-## s: the set to modify.
-##
-## key: the element to insert.
-##
-## Returns: true if the key was inserted, or false if it already existed.
-function BrokerComm::set_insert%(s: BrokerComm::Data, key: BrokerComm::Data%): bool
+function Broker::__set_insert%(s: Broker::Data, key: Broker::Data%): bool
 	%{
 	auto& v = bro_broker::require_data_type<broker::set>(s->AsRecordVal(), TYPE_TABLE,
 												   frame);
@@ -253,14 +148,7 @@ function BrokerComm::set_insert%(s: BrokerComm::Data, key: BrokerComm::Data%): b
 	return new Val(v.insert(k).second, TYPE_BOOL);
 	%}
 
-## Remove an element from a set.
-##
-## s: the set to modify.
-##
-## key: the element to remove.
-##
-## Returns: true if the element existed in the set and is now removed.
-function BrokerComm::set_remove%(s: BrokerComm::Data, key: BrokerComm::Data%): bool
+function Broker::__set_remove%(s: Broker::Data, key: Broker::Data%): bool
 	%{
 	auto& v = bro_broker::require_data_type<broker::set>(s->AsRecordVal(), TYPE_TABLE,
 												   frame);
@@ -268,37 +156,18 @@ function BrokerComm::set_remove%(s: BrokerComm::Data, key: BrokerComm::Data%): b
 	return new Val(v.erase(k) > 0, TYPE_BOOL);
 	%}
 
-## Create an iterator for a set.  Note that this makes a copy of the set
-## internally to ensure the iterator is always valid.
-##
-## s: the set to iterate over.
-##
-## Returns: an iterator.
-function BrokerComm::set_iterator%(s: BrokerComm::Data%): opaque of BrokerComm::SetIterator
+function Broker::__set_iterator%(s: Broker::Data%): opaque of Broker::SetIterator
 	%{
 	return new bro_broker::SetIterator(s->AsRecordVal(), TYPE_TABLE, frame);
 	%}
 
-## Check if there are no more elements to iterate over.
-##
-## it: an iterator.
-##
-## Returns: true if there are no more elements to iterator over, i.e.
-##          the iterator is one-past-the-final-element.
-function BrokerComm::set_iterator_last%(it: opaque of BrokerComm::SetIterator%): bool
+function Broker::__set_iterator_last%(it: opaque of Broker::SetIterator%): bool
 	%{
 	auto set_it = static_cast<bro_broker::SetIterator*>(it);
 	return new Val(set_it->it == set_it->dat.end(), TYPE_BOOL);
 	%}
 
-## Advance an iterator.
-##
-## it: an iterator.
-##
-## Returns: true if the iterator, after advancing, still references an element
-##          in the collection.  False if the iterator, after advancing, is
-##          one-past-the-final-element.
-function BrokerComm::set_iterator_next%(it: opaque of BrokerComm::SetIterator%): bool
+function Broker::__set_iterator_next%(it: opaque of Broker::SetIterator%): bool
 	%{
 	auto set_it = static_cast<bro_broker::SetIterator*>(it);
 
@@ -309,15 +178,10 @@ function BrokerComm::set_iterator_next%(it: opaque of BrokerComm::SetIterator%):
 	return new Val(set_it->it != set_it->dat.end(), TYPE_BOOL);
 	%}
 
-## Retrieve the data at an iterator's current position.
-##
-## it: an iterator.
-##
-## Returns: element in the collection that the iterator currently references.
-function BrokerComm::set_iterator_value%(it: opaque of BrokerComm::SetIterator%): BrokerComm::Data
+function Broker::__set_iterator_value%(it: opaque of Broker::SetIterator%): Broker::Data
 	%{
 	auto set_it = static_cast<bro_broker::SetIterator*>(it);
-	auto rval = new RecordVal(BifType::Record::BrokerComm::Data);
+	auto rval = new RecordVal(BifType::Record::Broker::Data);
 
 	if ( set_it->it == set_it->dat.end() )
 		{
@@ -331,18 +195,12 @@ function BrokerComm::set_iterator_value%(it: opaque of BrokerComm::SetIterator%)
 	return rval;
 	%}
 
-## Create communication data of type "table".
-function BrokerComm::table_create%(%): BrokerComm::Data
+function Broker::__table_create%(%): Broker::Data
 	%{
 	return bro_broker::make_data_val(broker::table());
 	%}
 
-## Remove all elements within a table.
-##
-## t: the table to clear.
-##
-## Returns: always true.
-function BrokerComm::table_clear%(t: BrokerComm::Data%): bool
+function Broker::__table_clear%(t: Broker::Data%): bool
 	%{
 	auto& v = bro_broker::require_data_type<broker::table>(t->AsRecordVal(),
 													 TYPE_TABLE, frame);
@@ -350,26 +208,14 @@ function BrokerComm::table_clear%(t: BrokerComm::Data%): bool
 	return new Val(true, TYPE_BOOL);
 	%}
 
-## Get the number of elements within a table.
-##
-## t: the table to query.
-##
-## Returns: the number of elements in the table.
-function BrokerComm::table_size%(t: BrokerComm::Data%): count
+function Broker::__table_size%(t: Broker::Data%): count
 	%{
 	auto& v = bro_broker::require_data_type<broker::table>(t->AsRecordVal(),
 													 TYPE_TABLE, frame);
 	return new Val(static_cast<uint64_t>(v.size()), TYPE_COUNT);
 	%}
 
-## Check if a table contains a particular key.
-##
-## t: the table to query.
-##
-## key: the key to check for existence.
-##
-## Returns: true if the key exists in the table.
-function BrokerComm::table_contains%(t: BrokerComm::Data, key: BrokerComm::Data%): bool
+function Broker::__table_contains%(t: Broker::Data, key: Broker::Data%): bool
 	%{
 	auto& v = bro_broker::require_data_type<broker::table>(t->AsRecordVal(),
 													 TYPE_TABLE, frame);
@@ -377,17 +223,7 @@ function BrokerComm::table_contains%(t: BrokerComm::Data, key: BrokerComm::Data%
 	return new Val(v.find(k) != v.end(), TYPE_BOOL);
 	%}
 
-## Insert a key-value pair into a table.
-##
-## t: the table to modify.
-##
-## key: the key at which to insert the value.
-##
-## val: the value to insert.
-##
-## Returns: true if the key-value pair was inserted, or false if the key
-##          already existed in the table.
-function BrokerComm::table_insert%(t: BrokerComm::Data, key: BrokerComm::Data, val: BrokerComm::Data%): BrokerComm::Data
+function Broker::__table_insert%(t: Broker::Data, key: Broker::Data, val: Broker::Data%): Broker::Data
 	%{
 	auto& table = bro_broker::require_data_type<broker::table>(t->AsRecordVal(),
 													     TYPE_TABLE, frame);
@@ -404,19 +240,11 @@ function BrokerComm::table_insert%(t: BrokerComm::Data, key: BrokerComm::Data, v
 	catch (const std::out_of_range&)
 		{
 		table[k] = v;
-		return new RecordVal(BifType::Record::BrokerComm::Data);
+		return new RecordVal(BifType::Record::Broker::Data);
 		}
 	%}
 
-## Remove a key-value pair from a table.
-##
-## t: the table to modify.
-##
-## key: the key to remove from the table.
-##
-## Returns: the value associated with the key.  If the key did not exist, then
-##          the optional field of the returned record is not set.
-function BrokerComm::table_remove%(t: BrokerComm::Data, key: BrokerComm::Data%): BrokerComm::Data
+function Broker::__table_remove%(t: Broker::Data, key: Broker::Data%): Broker::Data
 	%{
 	auto& table = bro_broker::require_data_type<broker::table>(t->AsRecordVal(),
 													     TYPE_TABLE, frame);
@@ -424,7 +252,7 @@ function BrokerComm::table_remove%(t: BrokerComm::Data, key: BrokerComm::Data%):
 	auto it = table.find(k);
 
 	if ( it == table.end() )
-		return new RecordVal(BifType::Record::BrokerComm::Data);
+		return new RecordVal(BifType::Record::Broker::Data);
 	else
 		{
 		auto rval = bro_broker::make_data_val(move(it->second));
@@ -433,15 +261,7 @@ function BrokerComm::table_remove%(t: BrokerComm::Data, key: BrokerComm::Data%):
 		}
 	%}
 
-## Retrieve a value from a table.
-##
-## t: the table to query.
-##
-## key: the key to lookup.
-##
-## Returns: the value associated with the key.  If the key did not exist, then
-##          the optional field of the returned record is not set.
-function BrokerComm::table_lookup%(t: BrokerComm::Data, key: BrokerComm::Data%): BrokerComm::Data
+function Broker::__table_lookup%(t: Broker::Data, key: Broker::Data%): Broker::Data
 	%{
 	auto& table = bro_broker::require_data_type<broker::table>(t->AsRecordVal(),
 													     TYPE_TABLE, frame);
@@ -449,42 +269,23 @@ function BrokerComm::table_lookup%(t: BrokerComm::Data, key: BrokerComm::Data%):
 	auto it = table.find(k);
 
 	if ( it == table.end() )
-		return new RecordVal(BifType::Record::BrokerComm::Data);
+		return new RecordVal(BifType::Record::Broker::Data);
 	else
 		return bro_broker::make_data_val(it->second);
 	%}
 
-## Create an iterator for a table.  Note that this makes a copy of the table
-## internally to ensure the iterator is always valid.
-##
-## t: the table to iterate over.
-##
-## Returns: an iterator.
-function BrokerComm::table_iterator%(t: BrokerComm::Data%): opaque of BrokerComm::TableIterator
+function Broker::__table_iterator%(t: Broker::Data%): opaque of Broker::TableIterator
 	%{
 	return new bro_broker::TableIterator(t->AsRecordVal(), TYPE_TABLE, frame);
 	%}
 
-## Check if there are no more elements to iterate over.
-##
-## it: an iterator.
-##
-## Returns: true if there are no more elements to iterator over, i.e.
-##          the iterator is one-past-the-final-element.
-function BrokerComm::table_iterator_last%(it: opaque of BrokerComm::TableIterator%): bool
+function Broker::__table_iterator_last%(it: opaque of Broker::TableIterator%): bool
 	%{
 	auto ti = static_cast<bro_broker::TableIterator*>(it);
 	return new Val(ti->it == ti->dat.end(), TYPE_BOOL);
 	%}
 
-## Advance an iterator.
-##
-## it: an iterator.
-##
-## Returns: true if the iterator, after advancing, still references an element
-##          in the collection.  False if the iterator, after advancing, is
-##          one-past-the-final-element.
-function BrokerComm::table_iterator_next%(it: opaque of BrokerComm::TableIterator%): bool
+function Broker::__table_iterator_next%(it: opaque of Broker::TableIterator%): bool
 	%{
 	auto ti = static_cast<bro_broker::TableIterator*>(it);
 
@@ -495,17 +296,12 @@ function BrokerComm::table_iterator_next%(it: opaque of BrokerComm::TableIterato
 	return new Val(ti->it != ti->dat.end(), TYPE_BOOL);
 	%}
 
-## Retrieve the data at an iterator's current position.
-##
-## it: an iterator.
-##
-## Returns: element in the collection that the iterator currently references.
-function BrokerComm::table_iterator_value%(it: opaque of BrokerComm::TableIterator%): BrokerComm::TableItem
+function Broker::__table_iterator_value%(it: opaque of Broker::TableIterator%): Broker::TableItem
 	%{
 	auto ti = static_cast<bro_broker::TableIterator*>(it);
-	auto rval = new RecordVal(BifType::Record::BrokerComm::TableItem);
-	auto key_val = new RecordVal(BifType::Record::BrokerComm::Data);
-	auto val_val = new RecordVal(BifType::Record::BrokerComm::Data);
+	auto rval = new RecordVal(BifType::Record::Broker::TableItem);
+	auto key_val = new RecordVal(BifType::Record::Broker::Data);
+	auto val_val = new RecordVal(BifType::Record::Broker::Data);
 	rval->Assign(0, key_val);
 	rval->Assign(1, val_val);
 
@@ -522,18 +318,12 @@ function BrokerComm::table_iterator_value%(it: opaque of BrokerComm::TableIterat
 	return rval;
 	%}
 
-## Create communication data of type "vector".
-function BrokerComm::vector_create%(%): BrokerComm::Data
+function Broker::__vector_create%(%): Broker::Data
 	%{
 	return bro_broker::make_data_val(broker::vector());
 	%}
 
-## Remove all elements within a vector.
-##
-## v: the vector to clear.
-##
-## Returns: always true.
-function BrokerComm::vector_clear%(v: BrokerComm::Data%): bool
+function Broker::__vector_clear%(v: Broker::Data%): bool
 	%{
 	auto& vec = bro_broker::require_data_type<broker::vector>(v->AsRecordVal(),
 													    TYPE_VECTOR, frame);
@@ -541,30 +331,14 @@ function BrokerComm::vector_clear%(v: BrokerComm::Data%): bool
 	return new Val(true, TYPE_BOOL);
 	%}
 
-## Get the number of elements within a vector.
-##
-## v: the vector to query.
-##
-## Returns: the number of elements in the vector.
-function BrokerComm::vector_size%(v: BrokerComm::Data%): count
+function Broker::__vector_size%(v: Broker::Data%): count
 	%{
 	auto& vec = bro_broker::require_data_type<broker::vector>(v->AsRecordVal(),
 													    TYPE_VECTOR, frame);
 	return new Val(static_cast<uint64_t>(vec.size()), TYPE_COUNT);
 	%}
 
-## Insert an element into a vector at a particular position, possibly displacing
-## existing elements (insertion always grows the size of the vector by one).
-##
-## v: the vector to modify.
-##
-## d: the element to insert.
-##
-## idx: the index at which to insert the data.  If it is greater than the
-##      current size of the vector, the element is inserted at the end.
-##
-## Returns: always true.
-function BrokerComm::vector_insert%(v: BrokerComm::Data, d: BrokerComm::Data, idx: count%): bool
+function Broker::__vector_insert%(v: Broker::Data, d: Broker::Data, idx: count%): bool
 	%{
 	auto& vec = bro_broker::require_data_type<broker::vector>(v->AsRecordVal(),
 													    TYPE_VECTOR, frame);
@@ -574,101 +348,56 @@ function BrokerComm::vector_insert%(v: BrokerComm::Data, d: BrokerComm::Data, id
 	return new Val(true, TYPE_BOOL);
 	%}
 
-## Replace an element in a vector at a particular position.
-##
-## v: the vector to modify.
-##
-## d: the element to insert.
-##
-## idx: the index to replace.
-##
-## Returns: the value that was just evicted.  If the index was larger than any
-##          valid index, the optional field of the returned record is not set.
-function BrokerComm::vector_replace%(v: BrokerComm::Data, d: BrokerComm::Data, idx: count%): BrokerComm::Data
+function Broker::__vector_replace%(v: Broker::Data, d: Broker::Data, idx: count%): Broker::Data
 	%{
 	auto& vec = bro_broker::require_data_type<broker::vector>(v->AsRecordVal(),
 													    TYPE_VECTOR, frame);
 	auto& item = bro_broker::opaque_field_to_data(d->AsRecordVal(), frame);
 
 	if ( idx >= vec.size() )
-		return new RecordVal(BifType::Record::BrokerComm::Data);
+		return new RecordVal(BifType::Record::Broker::Data);
 
 	auto rval = bro_broker::make_data_val(move(vec[idx]));
 	vec[idx] = item;
 	return rval;
 	%}
 
-## Remove an element from a vector at a particular position.
-##
-## v: the vector to modify.
-##
-## idx: the index to remove.
-##
-## Returns: the value that was just evicted.  If the index was larger than any
-##          valid index, the optional field of the returned record is not set.
-function BrokerComm::vector_remove%(v: BrokerComm::Data, idx: count%): BrokerComm::Data
+function Broker::__vector_remove%(v: Broker::Data, idx: count%): Broker::Data
 	%{
 	auto& vec = bro_broker::require_data_type<broker::vector>(v->AsRecordVal(),
 													    TYPE_VECTOR, frame);
 
 	if ( idx >= vec.size() )
-		return new RecordVal(BifType::Record::BrokerComm::Data);
+		return new RecordVal(BifType::Record::Broker::Data);
 
 	auto rval = bro_broker::make_data_val(move(vec[idx]));
 	vec.erase(vec.begin() + idx);
 	return rval;
 	%}
 
-## Lookup an element in a vector at a particular position.
-##
-## v: the vector to query.
-##
-## idx: the index to lookup.
-##
-## Returns: the value at the index.  If the index was larger than any
-##          valid index, the optional field of the returned record is not set.
-function BrokerComm::vector_lookup%(v: BrokerComm::Data, idx: count%): BrokerComm::Data
+function Broker::__vector_lookup%(v: Broker::Data, idx: count%): Broker::Data
 	%{
 	auto& vec = bro_broker::require_data_type<broker::vector>(v->AsRecordVal(),
 													    TYPE_VECTOR, frame);
 
 	if ( idx >= vec.size() )
-		return new RecordVal(BifType::Record::BrokerComm::Data);
+		return new RecordVal(BifType::Record::Broker::Data);
 
 	return bro_broker::make_data_val(vec[idx]);
 	%}
 
-## Create an iterator for a vector.  Note that this makes a copy of the vector
-## internally to ensure the iterator is always valid.
-##
-## v: the vector to iterate over.
-##
-## Returns: an iterator.
-function BrokerComm::vector_iterator%(v: BrokerComm::Data%): opaque of BrokerComm::VectorIterator
+function Broker::__vector_iterator%(v: Broker::Data%): opaque of Broker::VectorIterator
 	%{
 	return new bro_broker::VectorIterator(v->AsRecordVal(), TYPE_VECTOR, frame);
 	%}
 
-## Check if there are no more elements to iterate over.
-##
-## it: an iterator.
-##
-## Returns: true if there are no more elements to iterator over, i.e.
-##          the iterator is one-past-the-final-element.
-function BrokerComm::vector_iterator_last%(it: opaque of BrokerComm::VectorIterator%): bool
+function Broker::__vector_iterator_last%(it: opaque of Broker::VectorIterator%): bool
 	%{
 	auto vi = static_cast<bro_broker::VectorIterator*>(it);
 	return new Val(vi->it == vi->dat.end(), TYPE_BOOL);
 	%}
 
-## Advance an iterator.
-##
-## it: an iterator.
-##
-## Returns: true if the iterator, after advancing, still references an element
-##          in the collection.  False if the iterator, after advancing, is
-##          one-past-the-final-element.
-function BrokerComm::vector_iterator_next%(it: opaque of BrokerComm::VectorIterator%): bool
+function Broker::__vector_iterator_next%(it: opaque of Broker::VectorIterator%): bool
 	%{
 	auto vi = static_cast<bro_broker::VectorIterator*>(it);
 
@@ -679,15 +408,10 @@ function BrokerComm::vector_iterator_next%(it: opaque of BrokerComm::VectorItera
 	return new Val(vi->it != vi->dat.end(), TYPE_BOOL);
 	%}
 
-## Retrieve the data at an iterator's current position.
-##
-## it: an iterator.
-##
-## Returns: element in the collection that the iterator currently references.
-function BrokerComm::vector_iterator_value%(it: opaque of BrokerComm::VectorIterator%): BrokerComm::Data
+function Broker::__vector_iterator_value%(it: opaque of Broker::VectorIterator%): Broker::Data
 	%{
 	auto vi = static_cast<bro_broker::VectorIterator*>(it);
-	auto rval = new RecordVal(BifType::Record::BrokerComm::Data);
+	auto rval = new RecordVal(BifType::Record::Broker::Data);
 
 	if ( vi->it == vi->dat.end() )
 		{
@@ -701,38 +425,19 @@ function BrokerComm::vector_iterator_value%(it: opaque of BrokerComm::VectorIter
 	return rval;
 	%}
 
-## Create communication data of type "record".
-##
-## sz: the number of fields in the record.
-##
-## Returns: record data, with all fields uninitialized.
-function BrokerComm::record_create%(sz: count%): BrokerComm::Data
+function Broker::__record_create%(sz: count%): Broker::Data
 	%{
 	return bro_broker::make_data_val(broker::record(std::vector<broker::record::field>(sz)));
 	%}
 
-## Get the number of fields within a record.
-##
-## r: the record to query.
-##
-## Returns: the number of fields in the record.
-function BrokerComm::record_size%(r: BrokerComm::Data%): count
+function Broker::__record_size%(r: Broker::Data%): count
 	%{
 	auto& v = bro_broker::require_data_type<broker::record>(r->AsRecordVal(),
 													  TYPE_RECORD, frame);
 	return new Val(static_cast<uint64_t>(v.fields.size()), TYPE_COUNT);
 	%}
 
-## Replace a field in a record at a particular position.
-##
-## r: the record to modify.
-##
-## d: the new field value to assign.
-##
-## idx: the index to replace.
-##
-## Returns: false if the index was larger than any valid index, else true.
-function BrokerComm::record_assign%(r: BrokerComm::Data, d: BrokerComm::Data, idx: count%): bool
+function Broker::__record_assign%(r: Broker::Data, d: Broker::Data, idx: count%): bool
 	%{
 	auto& v = bro_broker::require_data_type<broker::record>(r->AsRecordVal(),
 													  TYPE_RECORD, frame);
@@ -745,60 +450,32 @@ function BrokerComm::record_assign%(r: BrokerComm::Data, d: BrokerComm::Data, id
 	return new Val(true, TYPE_BOOL);
 	%}
 
-## Lookup a field in a record at a particular position.
-##
-## r: the record to query.
-##
-## idx: the index to lookup.
-##
-## Returns: the value at the index.  The optional field of the returned record
-##          may not be set if the field of the record has no value or if the
-##          index was not valid.
-function BrokerComm::record_lookup%(r: BrokerComm::Data, idx: count%): BrokerComm::Data
+function Broker::__record_lookup%(r: Broker::Data, idx: count%): Broker::Data
 	%{
 	auto& v = bro_broker::require_data_type<broker::record>(r->AsRecordVal(),
 													  TYPE_RECORD, frame);
 
 	if ( idx >= v.size() )
-		return new RecordVal(BifType::Record::BrokerComm::Data);
+		return new RecordVal(BifType::Record::Broker::Data);
 
 	if ( ! v.fields[idx] )
-		return new RecordVal(BifType::Record::BrokerComm::Data);
+		return new RecordVal(BifType::Record::Broker::Data);
 
 	return bro_broker::make_data_val(*v.fields[idx]);
 	%}
 
-## Create an iterator for a record.  Note that this makes a copy of the record
-## internally to ensure the iterator is always valid.
-##
-## r: the record to iterate over.
-##
-## Returns: an iterator.
-function BrokerComm::record_iterator%(r: BrokerComm::Data%): opaque of BrokerComm::RecordIterator
+function Broker::__record_iterator%(r: Broker::Data%): opaque of Broker::RecordIterator
 	%{
 	return new bro_broker::RecordIterator(r->AsRecordVal(), TYPE_RECORD, frame);
 	%}
 
-## Check if there are no more elements to iterate over.
-##
-## it: an iterator.
-##
-## Returns: true if there are no more elements to iterator over, i.e.
-##          the iterator is one-past-the-final-element.
-function BrokerComm::record_iterator_last%(it: opaque of BrokerComm::RecordIterator%): bool
+function Broker::__record_iterator_last%(it: opaque of Broker::RecordIterator%): bool
 	%{
 	auto ri = static_cast<bro_broker::RecordIterator*>(it);
 	return new Val(ri->it == ri->dat.fields.end(), TYPE_BOOL);
 	%}
 
-## Advance an iterator.
-##
-## it: an iterator.
-##
-## Returns: true if the iterator, after advancing, still references an element
-##          in the collection.  False if the iterator, after advancing, is
-##          one-past-the-final-element.
-function BrokerComm::record_iterator_next%(it: opaque of BrokerComm::RecordIterator%): bool
+function Broker::__record_iterator_next%(it: opaque of Broker::RecordIterator%): bool
 	%{
 	auto ri = static_cast<bro_broker::RecordIterator*>(it);
 
@@ -809,15 +486,10 @@ function BrokerComm::record_iterator_next%(it: opaque of BrokerComm::RecordItera
 	return new Val(ri->it != ri->dat.fields.end(), TYPE_BOOL);
 	%}
 
-## Retrieve the data at an iterator's current position.
-##
-## it: an iterator.
-##
-## Returns: element in the collection that the iterator currently references.
-function BrokerComm::record_iterator_value%(it: opaque of BrokerComm::RecordIterator%): BrokerComm::Data
+function Broker::__record_iterator_value%(it: opaque of Broker::RecordIterator%): Broker::Data
 	%{
 	auto ri = static_cast<bro_broker::RecordIterator*>(it);
-	auto rval = new RecordVal(BifType::Record::BrokerComm::Data);
+	auto rval = new RecordVal(BifType::Record::Broker::Data);
 
 	if ( ri->it == ri->dat.fields.end() )
 		{

src/broker/messaging.bif

@@ -6,209 +6,106 @@
 #include "logging/Manager.h"
 %%}
 
-module BrokerComm;
+module Broker;
 
-type BrokerComm::SendFlags: record;
+type Broker::SendFlags: record;
 
-type BrokerComm::EventArgs: record;
+type Broker::EventArgs: record;
 
 ## Used to handle remote print messages from peers that call
-## :bro:see:`BrokerComm::print`.
-event BrokerComm::print_handler%(msg: string%);
+## :bro:see:`Broker::send_print`.
+event Broker::print_handler%(msg: string%);
 
-## Print a simple message to any interested peers.  The receiver can use
-## :bro:see:`BrokerComm::print_handler` to handle messages.
-##
-## topic: a topic associated with the printed message.
-##
-## msg: the print message to send to peers.
-##
-## flags: tune the behavior of how the message is sent.
-##
-## Returns: true if the message is sent.
-function BrokerComm::print%(topic: string, msg: string,
-                      flags: SendFlags &default = SendFlags()%): bool
+function Broker::__send_print%(topic: string, msg: string, flags: Broker::SendFlags%): bool
 	%{
 	auto rval = broker_mgr->Print(topic->CheckString(), msg->CheckString(),
 	                            flags);
 	return new Val(rval, TYPE_BOOL);
 	%}
 
-## Register interest in all peer print messages that use a certain topic prefix.
-## Use :bro:see:`BrokerComm::print_handler` to handle received messages.
-##
-## topic_prefix: a prefix to match against remote message topics.
-##               e.g. an empty prefix matches everything and "a" matches
-##               "alice" and "amy" but not "bob".
-##
-## Returns: true if it's a new print subscription and it is now registered.
-function BrokerComm::subscribe_to_prints%(topic_prefix: string%): bool
+function Broker::__subscribe_to_prints%(topic_prefix: string%): bool
 	%{
 	auto rval = broker_mgr->SubscribeToPrints(topic_prefix->CheckString());
 	return new Val(rval, TYPE_BOOL);
 	%}
 
-## Unregister interest in all peer print messages that use a topic prefix.
-##
-## topic_prefix: a prefix previously supplied to a successful call to
-##               :bro:see:`BrokerComm::subscribe_to_prints`.
-##
-## Returns: true if interest in the topic prefix is no longer advertised.
-function BrokerComm::unsubscribe_to_prints%(topic_prefix: string%): bool
+function Broker::__unsubscribe_to_prints%(topic_prefix: string%): bool
 	%{
 	auto rval = broker_mgr->UnsubscribeToPrints(topic_prefix->CheckString());
 	return new Val(rval, TYPE_BOOL);
 	%}
 
 ## Create a data structure that may be used to send a remote event via
-## :bro:see:`BrokerComm::event`.
+## :bro:see:`Broker::send_event`.
 ##
 ## args: an event, followed by a list of argument values that may be used
 ##       to call it.
 ##
-## Returns: opaque communication data that may be used to send a remote event.
-function BrokerComm::event_args%(...%): BrokerComm::EventArgs
+## Returns: opaque communication data that may be used to send a remote
+##          event.
+function Broker::event_args%(...%): Broker::EventArgs
 	%{
 	auto rval = broker_mgr->MakeEventArgs(@ARGS@);
 	return rval;
 	%}
 
-## Send an event to any interested peers.
-##
-## topic: a topic associated with the event message.
-##
-## args: event arguments as made by :bro:see:`BrokerComm::event_args`.
-##
-## flags: tune the behavior of how the message is sent.
-##
-## Returns: true if the message is sent.
-function BrokerComm::event%(topic: string, args: BrokerComm::EventArgs,
-                      flags: SendFlags &default = SendFlags()%): bool
+function Broker::__event%(topic: string, args: Broker::EventArgs, flags: Broker::SendFlags%): bool
 	%{
 	auto rval = broker_mgr->Event(topic->CheckString(), args->AsRecordVal(),
 	                            flags);
 	return new Val(rval, TYPE_BOOL);
 	%}
 
-## Automatically send an event to any interested peers whenever it is
-## locally dispatched (e.g. using "event my_event(...);" in a script).
-##
-## topic: a topic string associated with the event message.
-##        Peers advertise interest by registering a subscription to some prefix
-##        of this topic name.
-##
-## ev: a Bro event value.
-##
-## flags: tune the behavior of how the message is sent.
-##
-## Returns: true if automatic event sending is now enabled.
-function BrokerComm::auto_event%(topic: string, ev: any,
-                           flags: SendFlags &default = SendFlags()%): bool
+function Broker::__auto_event%(topic: string, ev: any, flags: Broker::SendFlags%): bool
 	%{
 	auto rval = broker_mgr->AutoEvent(topic->CheckString(), ev, flags);
 	return new Val(rval, TYPE_BOOL);
 	%}
 
-## Stop automatically sending an event to peers upon local dispatch.
-##
-## topic: a topic originally given to :bro:see:`BrokerComm::auto_event`.
-##
-## ev: an event originally given to :bro:see:`BrokerComm::auto_event`.
-##
-## Returns: true if automatic events will not occur for the topic/event pair.
-function BrokerComm::auto_event_stop%(topic: string, ev: any%): bool
+function Broker::__auto_event_stop%(topic: string, ev: any%): bool
 	%{
 	auto rval = broker_mgr->AutoEventStop(topic->CheckString(), ev);
 	return new Val(rval, TYPE_BOOL);
 	%}
 
-## Register interest in all peer event messages that use a certain topic prefix.
-##
-## topic_prefix: a prefix to match against remote message topics.
-##               e.g. an empty prefix matches everything and "a" matches
-##               "alice" and "amy" but not "bob".
-##
-## Returns: true if it's a new event subscription and it is now registered.
-function BrokerComm::subscribe_to_events%(topic_prefix: string%): bool
+function Broker::__subscribe_to_events%(topic_prefix: string%): bool
 	%{
 	auto rval = broker_mgr->SubscribeToEvents(topic_prefix->CheckString());
 	return new Val(rval, TYPE_BOOL);
 	%}
 
-## Unregister interest in all peer event messages that use a topic prefix.
-##
-## topic_prefix: a prefix previously supplied to a successful call to
-##               :bro:see:`BrokerComm::subscribe_to_events`.
-##
-## Returns: true if interest in the topic prefix is no longer advertised.
-function BrokerComm::unsubscribe_to_events%(topic_prefix: string%): bool
+function Broker::__unsubscribe_to_events%(topic_prefix: string%): bool
 	%{
 	auto rval = broker_mgr->UnsubscribeToEvents(topic_prefix->CheckString());
 	return new Val(rval, TYPE_BOOL);
 	%}
 
-## Enable remote logs for a given log stream.
-##
-## id: the log stream to enable remote logs for.
-##
-## flags: tune the behavior of how log entry messages are sent.
-##
-## Returns: true if remote logs are enabled for the stream.
-function
-BrokerComm::enable_remote_logs%(id: Log::ID,
-                          flags: SendFlags &default = SendFlags()%): bool
+function Broker::__enable_remote_logs%(id: Log::ID, flags: Broker::SendFlags%): bool
 	%{
 	auto rval = log_mgr->EnableRemoteLogs(id->AsEnumVal(),
 	                                   bro_broker::Manager::send_flags_to_int(flags));
 	return new Val(rval, TYPE_BOOL);
 	%}
 
-## Disable remote logs for a given log stream.
-##
-## id: the log stream to disable remote logs for.
-##
-## Returns: true if remote logs are disabled for the stream.
-function BrokerComm::disable_remote_logs%(id: Log::ID%): bool
+function Broker::__disable_remote_logs%(id: Log::ID%): bool
 	%{
 	auto rval = log_mgr->DisableRemoteLogs(id->AsEnumVal());
 	return new Val(rval, TYPE_BOOL);
 	%}
 
-## Check if remote logs are enabled for a given log stream.
-##
-## id: the log stream to check.
-##
-## Returns: true if remote logs are enabled for the given stream.
-function BrokerComm::remote_logs_enabled%(id: Log::ID%): bool
+function Broker::__remote_logs_enabled%(id: Log::ID%): bool
 	%{
 	auto rval = log_mgr->RemoteLogsAreEnabled(id->AsEnumVal());
 	return new Val(rval, TYPE_BOOL);
 	%}
 
-## Register interest in all peer log messages that use a certain topic prefix.
-## Logs are implicitly sent with topic "bro/log/<stream-name>" and the
-## receiving side processes them through the logging framework as usual.
-##
-## topic_prefix: a prefix to match against remote message topics.
-##               e.g. an empty prefix matches everything and "a" matches
-##               "alice" and "amy" but not "bob".
-##
-## Returns: true if it's a new log subscription and it is now registered.
-function BrokerComm::subscribe_to_logs%(topic_prefix: string%): bool
+function Broker::__subscribe_to_logs%(topic_prefix: string%): bool
 	%{
 	auto rval = broker_mgr->SubscribeToLogs(topic_prefix->CheckString());
 	return new Val(rval, TYPE_BOOL);
 	%}
 
-## Unregister interest in all peer log messages that use a topic prefix.
-## Logs are implicitly sent with topic "bro/log/<stream-name>" and the
-## receiving side processes them through the logging framework as usual.
-##
-## topic_prefix: a prefix previously supplied to a successful call to
-##               :bro:see:`BrokerComm::subscribe_to_logs`.
-##
-## Returns: true if interest in the topic prefix is no longer advertised.
-function BrokerComm::unsubscribe_to_logs%(topic_prefix: string%): bool
+function Broker::__unsubscribe_to_logs%(topic_prefix: string%): bool
 	%{
 	auto rval = broker_mgr->UnsubscribeToLogs(topic_prefix->CheckString());
 	return new Val(rval, TYPE_BOOL);

src/broker/store.bif

@@ -8,13 +8,13 @@
 #include "Trigger.h"
 %%}
 
-module BrokerStore;
+module Broker;
 
-type BrokerStore::ExpiryTime: record;
+type Broker::ExpiryTime: record;
 
-type BrokerStore::QueryResult: record;
+type Broker::QueryResult: record;
 
-type BrokerStore::BackendOptions: record;
+type Broker::BackendOptions: record;
 
 ## Enumerates the possible storage backends.
 enum BackendType %{
@@ -23,17 +23,8 @@ enum BackendType %{
 	ROCKSDB,
 %}
 
-## Create a master data store which contains key-value pairs.
-##
-## id: a unique name for the data store.
-##
-## b: the storage backend to use.
-##
-## options: tunes how some storage backends operate.
-##
-## Returns: a handle to the data store.
-function BrokerStore::create_master%(id: string, b: BackendType &default = MEMORY,
-                               options: BackendOptions &default = BackendOptions()%): opaque of BrokerStore::Handle
+function Broker::__create_master%(id: string, b: BackendType,
+                               options: BackendOptions &default = BackendOptions()%): opaque of Broker::Handle
 	%{
 	auto id_str = id->CheckString();
 	auto type = bro_broker::StoreType::MASTER;
@@ -46,38 +37,16 @@ function BrokerStore::create_master%(id: string, b: BackendType &default = MEMOR
 		}
 
 	rval = new bro_broker::StoreHandleVal(id_str, type,
-	                                static_cast<BifEnum::BrokerStore::BackendType>(b->AsEnum()),
+	                                static_cast<BifEnum::Broker::BackendType>(b->AsEnum()),
 	                                options->AsRecordVal());
 	auto added = broker_mgr->AddStore(rval);
 	assert(added);
 	return rval;
 	%}
 
-## Create a clone of a master data store which may live with a remote peer.
-## A clone automatically synchronizes to the master by automatically receiving
-## modifications and applying them locally.  Direct modifications are not
-## possible, they must be sent through the master store, which then
-## automatically broadcasts the changes out to clones.  But queries may be made
-## directly against the local cloned copy, which may be resolved quicker than
-## reaching out to a remote master store.
-##
-## id: the unique name which identifies the master data store.
-##
-## b: the storage backend to use.
-##
-## options: tunes how some storage backends operate.
-##
-## resync: the interval at which to re-attempt synchronizing with the master
-##         store should the connection be lost.  If the clone has not yet
-##         synchronized for the first time, updates and queries queue up until
-##         the synchronization completes.  After, if the connection to the
-##         master store is lost, queries continue to use the clone's version,
-##         but updates will be lost until the master is once again available.
-##
-## Returns: a handle to the data store.
-function BrokerStore::create_clone%(id: string, b: BackendType &default = MEMORY,
+function Broker::__create_clone%(id: string, b: BackendType,
                               options: BackendOptions &default = BackendOptions(),
-                              resync: interval &default = 1sec%): opaque of BrokerStore::Handle
+                              resync: interval &default = 1sec%): opaque of Broker::Handle
 	%{
 	auto id_str = id->CheckString();
 	auto type = bro_broker::StoreType::CLONE;
@@ -90,7 +59,7 @@ function BrokerStore::create_clone%(id: string, b: BackendType &default = MEMORY
 		}
 
 	rval = new bro_broker::StoreHandleVal(id_str, type,
-	                                static_cast<BifEnum::BrokerStore::BackendType>(b->AsEnum()),
+	                                static_cast<BifEnum::Broker::BackendType>(b->AsEnum()),
 	                                options->AsRecordVal(),
 	                                std::chrono::duration<double>(resync));
 	auto added = broker_mgr->AddStore(rval);
@@ -98,13 +67,7 @@ function BrokerStore::create_clone%(id: string, b: BackendType &default = MEMORY
 	return rval;
 	%}
 
-## Create a frontend interface to an existing master data store that allows
-## querying and updating its contents.
-##
-## id: the unique name which identifies the master data store.
-##
-## Returns: a handle to the data store.
-function BrokerStore::create_frontend%(id: string%): opaque of BrokerStore::Handle
+function Broker::__create_frontend%(id: string%): opaque of Broker::Handle
 	%{
 	auto id_str = id->CheckString();
 	auto type = bro_broker::StoreType::FRONTEND;
@@ -122,13 +85,7 @@ function BrokerStore::create_frontend%(id: string%): opaque of BrokerStore::Hand
 	return rval;
 	%}
 
-## Close a data store.
-##
-## h: a data store handle.
-##
-## Returns: true if store was valid and is now closed.  The handle can no
-##          longer be used for data store operations.
-function BrokerStore::close_by_handle%(h: opaque of BrokerStore::Handle%): bool
+function Broker::__close_by_handle%(h: opaque of Broker::Handle%): bool
 	%{
 	auto handle = static_cast<bro_broker::StoreHandleVal*>(h);
 
@@ -143,20 +100,9 @@ function BrokerStore::close_by_handle%(h: opaque of BrokerStore::Handle%): bool
 # non-blocking update API #
 ###########################
 
-## Insert a key-value pair in to the store.
-##
-## h: the handle of the store to modify.
-##
-## k: the key to insert.
-##
-## v: the value to insert.
-##
-## e: the expiration time of the key-value pair.
-##
-## Returns: false if the store handle was not valid.
-function BrokerStore::insert%(h: opaque of BrokerStore::Handle,
-                        k: BrokerComm::Data, v: BrokerComm::Data,
-                        e: BrokerStore::ExpiryTime &default = BrokerStore::ExpiryTime()%): bool
+function Broker::__insert%(h: opaque of Broker::Handle,
+                        k: Broker::Data, v: Broker::Data,
+                        e: Broker::ExpiryTime &default = Broker::ExpiryTime()%): bool
 	%{
 	auto handle = static_cast<bro_broker::StoreHandleVal*>(h);
 
@@ -191,14 +137,7 @@ function BrokerStore::insert%(h: opaque of BrokerStore::Handle,
 	return new Val(true, TYPE_BOOL);
 	%}
 
-## Remove a key-value pair from the store.
-##
-## h: the handle of the store to modify.
-##
-## k: the key to remove.
-##
-## Returns: false if the store handle was not valid.
-function BrokerStore::erase%(h: opaque of BrokerStore::Handle, k: BrokerComm::Data%): bool
+function Broker::__erase%(h: opaque of Broker::Handle, k: Broker::Data%): bool
 	%{
 	auto handle = static_cast<bro_broker::StoreHandleVal*>(h);
 
@@ -210,12 +149,7 @@ function BrokerStore::erase%(h: opaque of BrokerStore::Handle, k: BrokerComm::Da
 	return new Val(true, TYPE_BOOL);
 	%}
 
-## Remove all key-value pairs from the store.
-##
-## h: the handle of the store to modify.
-##
-## Returns: false if the store handle was not valid.
-function BrokerStore::clear%(h: opaque of BrokerStore::Handle%): bool
+function Broker::__clear%(h: opaque of Broker::Handle%): bool
 	%{
 	auto handle = static_cast<bro_broker::StoreHandleVal*>(h);
 
@@ -226,18 +160,8 @@ function BrokerStore::clear%(h: opaque of BrokerStore::Handle%): bool
 	return new Val(true, TYPE_BOOL);
 	%}
 
-## Increment an integer value in a data store.
-##
-## h: the handle of the store to modify.
-##
-## k: the key whose associated value is to be modified.
-##
-## by: the amount to increment the value by.  A non-existent key will first
-##     create it with an implicit value of zero before incrementing.
-##
-## Returns: false if the store handle was not valid.
-function BrokerStore::increment%(h: opaque of BrokerStore::Handle,
-                           k: BrokerComm::Data, by: int &default = +1%): bool
+function Broker::__increment%(h: opaque of Broker::Handle,
+                           k: Broker::Data, by: int &default = +1%): bool
 	%{
 	auto handle = static_cast<bro_broker::StoreHandleVal*>(h);
 
@@ -249,18 +173,8 @@ function BrokerStore::increment%(h: opaque of BrokerStore::Handle,
 	return new Val(true, TYPE_BOOL);
 	%}
 
-## Decrement an integer value in a data store.
-##
-## h: the handle of the store to modify.
-##
-## k: the key whose associated value is to be modified.
-##
-## by: the amount to decrement the value by.  A non-existent key will first
-##     create it with an implicit value of zero before decrementing.
-##
-## Returns: false if the store handle was not valid.
-function BrokerStore::decrement%(h: opaque of BrokerStore::Handle,
-                           k: BrokerComm::Data, by: int &default = +1%): bool
+function Broker::__decrement%(h: opaque of Broker::Handle,
+                           k: Broker::Data, by: int &default = +1%): bool
 	%{
 	auto handle = static_cast<bro_broker::StoreHandleVal*>(h);
 
@@ -272,18 +186,8 @@ function BrokerStore::decrement%(h: opaque of BrokerStore::Handle,
 	return new Val(true, TYPE_BOOL);
 	%}
 
-## Add an element to a set value in a data store.
-##
-## h: the handle of the store to modify.
-##
-## k: the key whose associated value is to be modified.
-##
-## element: the element to add to the set.  A non-existent key will first
-##          create it with an implicit empty set value before modifying.
-##
-## Returns: false if the store handle was not valid.
-function BrokerStore::add_to_set%(h: opaque of BrokerStore::Handle,
-                            k: BrokerComm::Data, element: BrokerComm::Data%): bool
+function Broker::__add_to_set%(h: opaque of Broker::Handle,
+                            k: Broker::Data, element: Broker::Data%): bool
 	%{
 	auto handle = static_cast<bro_broker::StoreHandleVal*>(h);
 
@@ -296,18 +200,8 @@ function BrokerStore::add_to_set%(h: opaque of BrokerStore::Handle,
 	return new Val(true, TYPE_BOOL);
 	%}
 
-## Remove an element from a set value in a data store.
-##
-## h: the handle of the store to modify.
-##
-## k: the key whose associated value is to be modified.
-##
-## element: the element to remove from the set.  A non-existent key will
-##          implicitly create an empty set value associated with the key.
-##
-## Returns: false if the store handle was not valid.
-function BrokerStore::remove_from_set%(h: opaque of BrokerStore::Handle,
-                                 k: BrokerComm::Data, element: BrokerComm::Data%): bool
+function Broker::__remove_from_set%(h: opaque of Broker::Handle,
+                                 k: Broker::Data, element: Broker::Data%): bool
 	%{
 	auto handle = static_cast<bro_broker::StoreHandleVal*>(h);
 
@@ -320,18 +214,8 @@ function BrokerStore::remove_from_set%(h: opaque of BrokerStore::Handle,
 	return new Val(true, TYPE_BOOL);
 	%}
 
-## Add a new item to the head of a vector value in a data store.
-##
-## h: the handle of store to modify.
-##
-## k: the key whose associated value is to be modified.
-##
-## items: the element to insert in to the vector.  A non-existent key will first
-##        create an empty vector value before modifying.
-##
-## Returns: false if the store handle was not valid.
-function BrokerStore::push_left%(h: opaque of BrokerStore::Handle, k: BrokerComm::Data,
-                           items: BrokerComm::DataVector%): bool
+function Broker::__push_left%(h: opaque of Broker::Handle, k: Broker::Data,
+                           items: Broker::DataVector%): bool
 	%{
 	auto handle = static_cast<bro_broker::StoreHandleVal*>(h);
 
@@ -353,18 +237,8 @@ function BrokerStore::push_left%(h: opaque of BrokerStore::Handle, k: BrokerComm
 	return new Val(true, TYPE_BOOL);
 	%}
 
-## Add a new item to the tail of a vector value in a data store.
-##
-## h: the handle of store to modify.
-##
-## k: the key whose associated value is to be modified.
-##
-## items: the element to insert in to the vector.  A non-existent key will first
-##        create an empty vector value before modifying.
-##
-## Returns: false if the store handle was not valid.
-function BrokerStore::push_right%(h: opaque of BrokerStore::Handle, k: BrokerComm::Data,
-                            items: BrokerComm::DataVector%): bool
+function Broker::__push_right%(h: opaque of Broker::Handle, k: Broker::Data,
+                            items: Broker::DataVector%): bool
 	%{
 	auto handle = static_cast<bro_broker::StoreHandleVal*>(h);
 
@@ -401,7 +275,7 @@ static bool prepare_for_query(Val* opaque, Frame* frame,
 	if ( ! (*handle)->store )
 		{
 		reporter->PushLocation(frame->GetCall()->GetLocationInfo());
-		reporter->Error("BrokerStore query has an invalid data store");
+		reporter->Error("Broker query has an invalid data store");
 		reporter->PopLocation();
 		return false;
 		}
@@ -411,7 +285,7 @@ static bool prepare_for_query(Val* opaque, Frame* frame,
 	if ( ! trigger )
 		{
 		reporter->PushLocation(frame->GetCall()->GetLocationInfo());
-		reporter->Error("BrokerStore queries can only be called inside when-condition");
+		reporter->Error("Broker queries can only be called inside when-condition");
 		reporter->PopLocation();
 		return false;
 		}
@@ -421,7 +295,7 @@ static bool prepare_for_query(Val* opaque, Frame* frame,
 	if ( *timeout < 0 )
 		{
 		reporter->PushLocation(frame->GetCall()->GetLocationInfo());
-		reporter->Error("BrokerStore queries must specify a timeout block");
+		reporter->Error("Broker queries must specify a timeout block");
 		reporter->PopLocation();
 		return false;
 		}
@@ -437,15 +311,8 @@ static bool prepare_for_query(Val* opaque, Frame* frame,
 
 %%}
 
-## Pop the head of a data store vector value.
-##
-## h: the handle of the store to query.
-##
-## k: the key associated with the vector to modify.
-##
-## Returns: the result of the query.
-function BrokerStore::pop_left%(h: opaque of BrokerStore::Handle,
-                          k: BrokerComm::Data%): BrokerStore::QueryResult
+function Broker::__pop_left%(h: opaque of Broker::Handle,
+                          k: Broker::Data%): Broker::QueryResult
 	%{
 	if ( ! broker_mgr->Enabled() )
 		return bro_broker::query_result();
@@ -467,15 +334,8 @@ function BrokerStore::pop_left%(h: opaque of BrokerStore::Handle,
 	return 0;
 	%}
 
-## Pop the tail of a data store vector value.
-##
-## h: the handle of the store to query.
-##
-## k: the key associated with the vector to modify.
-##
-## Returns: the result of the query.
-function BrokerStore::pop_right%(h: opaque of BrokerStore::Handle,
-                           k: BrokerComm::Data%): BrokerStore::QueryResult
+function Broker::__pop_right%(h: opaque of Broker::Handle,
+                           k: Broker::Data%): Broker::QueryResult
 	%{
 	if ( ! broker_mgr->Enabled() )
 		return bro_broker::query_result();
@@ -497,15 +357,8 @@ function BrokerStore::pop_right%(h: opaque of BrokerStore::Handle,
 	return 0;
 	%}
 
-## Lookup the value associated with a key in a data store.
-##
-## h: the handle of the store to query.
-##
-## k: the key to lookup.
-##
-## Returns: the result of the query.
-function BrokerStore::lookup%(h: opaque of BrokerStore::Handle,
-                       k: BrokerComm::Data%): BrokerStore::QueryResult
+function Broker::__lookup%(h: opaque of Broker::Handle,
+                       k: Broker::Data%): Broker::QueryResult
 	%{
 	if ( ! broker_mgr->Enabled() )
 		return bro_broker::query_result();
@@ -527,15 +380,8 @@ function BrokerStore::lookup%(h: opaque of BrokerStore::Handle,
 	return 0;
 	%}
 
-## Check if a data store contains a given key.
-##
-## h: the handle of the store to query.
-##
-## k: the key to check for existence.
-##
-## Returns: the result of the query (uses :bro:see:`BrokerComm::BOOL`).
-function BrokerStore::exists%(h: opaque of BrokerStore::Handle,
-                        k: BrokerComm::Data%): BrokerStore::QueryResult
+function Broker::__exists%(h: opaque of Broker::Handle,
+                        k: Broker::Data%): Broker::QueryResult
 	%{
 	if ( ! broker_mgr->Enabled() )
 		return bro_broker::query_result();
@@ -557,12 +403,7 @@ function BrokerStore::exists%(h: opaque of BrokerStore::Handle,
 	return 0;
 	%}
 
-## Retrieve all keys in a data store.
-##
-## h: the handle of the store to query.
-##
-## Returns: the result of the query (uses :bro:see:`BrokerComm::VECTOR`).
-function BrokerStore::keys%(h: opaque of BrokerStore::Handle%): BrokerStore::QueryResult
+function Broker::__keys%(h: opaque of Broker::Handle%): Broker::QueryResult
 	%{
 	double timeout;
 	bro_broker::StoreQueryCallback* cb;
@@ -575,12 +416,7 @@ function BrokerStore::keys%(h: opaque of BrokerStore::Handle%): BrokerStore::Que
 	return 0;
 	%}
 
-## Get the number of key-value pairs in a data store.
-##
-## h: the handle of the store to query.
-##
-## Returns: the result of the query (uses :bro:see:`BrokerComm::COUNT`).
-function BrokerStore::size%(h: opaque of BrokerStore::Handle%): BrokerStore::QueryResult
+function Broker::__size%(h: opaque of Broker::Handle%): Broker::QueryResult
 	%{
 	if ( ! broker_mgr->Enabled() )
 		return bro_broker::query_result();

src/broxygen/Configuration.cc

@@ -65,7 +65,7 @@ Config::Config(const string& arg_file, const string& delim)
 		Target* target = target_factory.Create(tokens[0], tokens[2], tokens[1]);
 
 		if ( ! target )
-			reporter->FatalError("unkown Broxygen target type: %s",
+			reporter->FatalError("unknown Broxygen target type: %s",
 			                     tokens[0].c_str());
 
 		targets.push_back(target);

src/file_analysis/analyzer/CMakeLists.txt

@@ -1,4 +1,5 @@
 add_subdirectory(data_event)
+add_subdirectory(entropy)
 add_subdirectory(extract)
 add_subdirectory(hash)
 add_subdirectory(pe)

src/file_analysis/analyzer/entropy/CMakeLists.txt

@@ -0,0 +1,9 @@
+include(BroPlugin)
+
+include_directories(BEFORE ${CMAKE_CURRENT_SOURCE_DIR}
+                           ${CMAKE_CURRENT_BINARY_DIR})
+
+bro_plugin_begin(Bro FileEntropy)
+bro_plugin_cc(Entropy.cc Plugin.cc ../../Analyzer.cc)
+bro_plugin_bif(events.bif)
+bro_plugin_end()

src/file_analysis/analyzer/entropy/Entropy.cc

@@ -0,0 +1,71 @@
+// See the file "COPYING" in the main distribution directory for copyright.
+
+#include <string>
+
+#include "Entropy.h"
+#include "util.h"
+#include "Event.h"
+#include "file_analysis/Manager.h"
+
+using namespace file_analysis;
+
+Entropy::Entropy(RecordVal* args, File* file)
+    : file_analysis::Analyzer(file_mgr->GetComponentTag("ENTROPY"), args, file)
+	{
+	//entropy->Init();
+	entropy = new EntropyVal;
+	}
+
+Entropy::~Entropy()
+	{
+	Unref(entropy);
+	}
+
+file_analysis::Analyzer* Entropy::Instantiate(RecordVal* args, File* file)
+	{
+	return new Entropy(args, file);
+	}
+
+bool Entropy::DeliverStream(const u_char* data, uint64 len)
+	{
+	if ( ! fed )
+		fed = len > 0;
+
+	entropy->Feed(data, len);
+	return true;
+	}
+
+bool Entropy::EndOfFile()
+	{
+	Finalize();
+	return false;
+	}
+
+bool Entropy::Undelivered(uint64 offset, uint64 len)
+	{
+	return false;
+	}
+
+void Entropy::Finalize()
+	{
+	//if ( ! entropy->IsValid() || ! fed )
+	if ( ! fed )
+		return;
+
+	val_list* vl = new val_list();
+	vl->append(GetFile()->GetVal()->Ref());
+
+	double montepi, scc, ent, mean, chisq;
+	montepi = scc = ent = mean = chisq = 0.0;
+	entropy->Get(&ent, &chisq, &mean, &montepi, &scc);
+
+	RecordVal* ent_result = new RecordVal(entropy_test_result);
+	ent_result->Assign(0, new Val(ent,     TYPE_DOUBLE));
+	ent_result->Assign(1, new Val(chisq,   TYPE_DOUBLE));
+	ent_result->Assign(2, new Val(mean,    TYPE_DOUBLE));
+	ent_result->Assign(3, new Val(montepi, TYPE_DOUBLE));
+	ent_result->Assign(4, new Val(scc,     TYPE_DOUBLE));
+
+	vl->append(ent_result);
+	mgr.QueueEvent(file_entropy, vl);
+	}

src/file_analysis/analyzer/entropy/Entropy.h

@@ -0,0 +1,84 @@
+// See the file "COPYING" in the main distribution directory for copyright.
+
+#ifndef FILE_ANALYSIS_ENTROPY_H
+#define FILE_ANALYSIS_ENTROPY_H
+
+#include <string>
+
+#include "Val.h"
+#include "OpaqueVal.h"
+#include "File.h"
+#include "Analyzer.h"
+
+#include "events.bif.h"
+
+namespace file_analysis {
+
+/**
+ * An analyzer to produce a hash of file contents.
+ */
+class Entropy : public file_analysis::Analyzer {
+public:
+
+	/**
+	 * Destructor.
+	 */
+	virtual ~Entropy();
+
+	/**
+	 * Create a new instance of an Extract analyzer.
+	 * @param args the \c AnalyzerArgs value which represents the analyzer.
+	 * @param file the file to which the analyzer will be attached.
+	 * @return the new Extract analyzer instance or a null pointer if the
+	 *         the "extraction_file" field of \a args wasn't set.
+	 */
+	static file_analysis::Analyzer* Instantiate(RecordVal* args, File* file);
+
+	/**
+	 * Incrementally hash next chunk of file contents.
+	 * @param data pointer to start of a chunk of a file data.
+	 * @param len number of bytes in the data chunk.
+	 * @return false if the digest is in an invalid state, else true.
+	 */
+	virtual bool DeliverStream(const u_char* data, uint64 len);
+
+	/**
+	 * Finalizes the hash and raises a "file_entropy_test" event.
+	 * @return always false so analyze will be deteched from file.
+	 */
+	virtual bool EndOfFile();
+
+	/**
+	 * Missing data can't be handled, so just indicate the this analyzer should
+	 * be removed from receiving further data.  The hash will not be finalized.
+	 * @param offset byte offset in file at which missing chunk starts.
+	 * @param len number of missing bytes.
+	 * @return always false so analyzer will detach from file.
+	 */
+	virtual bool Undelivered(uint64 offset, uint64 len);
+
+protected:
+
+	/**
+	 * Constructor.
+	 * @param args the \c AnalyzerArgs value which represents the analyzer.
+	 * @param file the file to which the analyzer will be attached.
+	 * @param hv specific hash calculator object.
+	 * @param kind human readable name of the hash algorithm to use.
+	 */
+	Entropy(RecordVal* args, File* file);
+
+	/**
+	 * If some file contents have been seen, finalizes the hash of them and
+	 * raises the "file_hash" event with the results.
+	 */
+	void Finalize();
+
+private:
+	EntropyVal* entropy;
+	bool fed;
+};
+
+} // namespace file_analysis
+
+#endif

src/file_analysis/analyzer/entropy/Plugin.cc

@@ -0,0 +1,24 @@
+// See the file  in the main distribution directory for copyright.
+
+#include "plugin/Plugin.h"
+
+#include "Entropy.h"
+
+namespace plugin {
+namespace Bro_FileEntropy {
+
+class Plugin : public plugin::Plugin {
+public:
+	plugin::Configuration Configure()
+		{
+		AddComponent(new ::file_analysis::Component("ENTROPY", ::file_analysis::Entropy::Instantiate));
+
+		plugin::Configuration config;
+		config.name = "Bro::FileEntropy";
+		config.description = "Entropy test file content";
+		return config;
+		}
+} plugin;
+
+}
+}

src/file_analysis/analyzer/entropy/events.bif

@@ -0,0 +1,8 @@
+## This event is generated each time file analysis performs
+## entropy testing on a file.
+##
+## f: The file.
+##
+## ent: The results of the entropy testing.
+##
+event file_entropy%(f: fa_file, ent: entropy_test_result%);

src/file_analysis/analyzer/x509/X509.cc

@@ -543,7 +543,7 @@ double file_analysis::X509::GetTimeFromAsn1(const ASN1_TIME* atime, const char*
 			}
 
 		// year is first two digits in YY format. Buffer expects YYYY format.
-		if ( pString[0] - '0' < 50 ) // RFC 2459 4.1.2.5.1
+		if ( pString[0] < '5' ) // RFC 2459 4.1.2.5.1
 			{
 			*(pBuffer++) = '2';
 			*(pBuffer++) = '0';

src/iosource/Packet.cc

@@ -428,15 +428,6 @@ void Packet::ProcessLayer2()
 
 RecordVal* Packet::BuildPktHdrVal() const
 	{
-	static RecordType* l2_hdr_type = 0;
-	static RecordType* raw_pkt_hdr_type = 0;
-
-	if ( ! raw_pkt_hdr_type )
-		{
-		raw_pkt_hdr_type = internal_type("raw_pkt_hdr")->AsRecordType();
-		l2_hdr_type = internal_type("l2_hdr")->AsRecordType();
-		}
-
 	RecordVal* pkt_hdr = new RecordVal(raw_pkt_hdr_type);
 	RecordVal* l2_hdr = new RecordVal(l2_hdr_type);
 

src/iosource/PktSrc.cc

@@ -91,7 +91,7 @@ void PktSrc::Opened(const Properties& arg_props)
 		{
 		char buf[512];
 		safe_snprintf(buf, sizeof(buf),
-			 "unknown data link type 0x%x", props.link_type);
+			 "unknown data link type 0x%x", arg_props.link_type);
 		Error(buf);
 		Close();
 		return;

src/nb_dns.c

@@ -389,7 +389,7 @@ nb_dns_addr_request2(register struct nb_dns_info *nd, char *addrp,
 
 	default:
 		snprintf(errstr, NB_DNS_ERRSIZE,
-		    "nb_dns_addr_request2(): uknown address family %d", af);
+		    "nb_dns_addr_request2(): unknown address family %d", af);
 		return (-1);
 	}
 

testing/btest/Baseline/bifs.get_current_packet_header/output

@@ -0,0 +1 @@
+[l2=[encap=LINK_ETHERNET, len=78, cap_len=78, src=00:00:00:00:00:00, dst=ff:ff:ff:ff:ff:ff, vlan=<uninitialized>, inner_vlan=<uninitialized>, eth_type=34525, proto=L3_IPV6], ip=<uninitialized>, ip6=[class=0, flow=0, len=24, nxt=58, hlim=255, src=fe80::dead, dst=fe80::beef, exts=[]], tcp=<uninitialized>, udp=<uninitialized>, icmp=[icmp_type=135]]

testing/btest/Baseline/broker.clone_store/clone.clone.out

@@ -1,5 +1,5 @@
-clone keys, [status=BrokerStore::SUCCESS, result=[d=broker::data{[one, two, myset, myvec]}]]
-lookup, one, [status=BrokerStore::SUCCESS, result=[d=broker::data{111}]]
-lookup, myset, [status=BrokerStore::SUCCESS, result=[d=broker::data{{a, c, d}}]]
-lookup, two, [status=BrokerStore::SUCCESS, result=[d=broker::data{222}]]
-lookup, myvec, [status=BrokerStore::SUCCESS, result=[d=broker::data{[delta, alpha, beta, gamma, omega]}]]
+clone keys, [status=Broker::SUCCESS, result=[d=broker::data{[one, two, myset, myvec]}]]
+lookup, two, [status=Broker::SUCCESS, result=[d=broker::data{222}]]
+lookup, one, [status=Broker::SUCCESS, result=[d=broker::data{111}]]
+lookup, myvec, [status=Broker::SUCCESS, result=[d=broker::data{[delta, alpha, beta, gamma, omega]}]]
+lookup, myset, [status=Broker::SUCCESS, result=[d=broker::data{{a, c, d}}]]