Merge branch 'topic/robin/intel-framework-merge'

* topic/robin/intel-framework-merge: (22 commits)
  Fixing tests after intel-framework merge.
  Extracting URLs from message bodies over SMTP and sending them to Intel framework.
  Small comment updates in the Intel framework CIF support.
  Intelligence framework documentation first draft.
  Only the manager tries to read files with the input framework now.
  Initial support for Bro's Intel framework with the Collective Intelligence Framework.
  Initial API for Intel framework is complete.
  Fixed an issue with cluster data distribution.
  Updating some intel framework test baselines.
  Reworked cluster intelligence data distribution mechanism and fixed tests.
  Lots more intelligence checking in SMTP traffic.
  Added intelligence check for "Received" path checking and a bit of reshuffling.
  Added sources to the intel log.
  Fixing a problem with intel distribution on clusters.
  Updated intel framework test to include matching.
  Restructuring the scripts that feed data into the intel framework slightly.
  One test for cluster transparency of the intel framework.
  Fixed a cluster support bug.
  Intelligence framework checkpoint
  Major updates to fix the Intel framework API.
  ...

Closes #914.

CHANGES

@@ -1,4 +1,9 @@
 
+2.1-XXX | 2012-11-01 08:20:00 -0700
+
+  * Initial version of a completely reworked intelligence framework.
+    See doc/intel.rst for more information. (Seth Hall)
+
 2.1-112 | 2012-11-05 13:58:20 -0800
 
   * New base script for detecting cases of checksum offloading. 

doc/intel.rst

@@ -0,0 +1,125 @@
+Intel Framework
+===============
+
+Intro
+-----
+
+Intelligence data is critical to the process of monitoring for
+security purposes.  There is always data which will be discovered
+through the incident response process and data which is shared through
+private communities.  The goals of Bro's Intelligence Framework are to
+consume that data, make it available for matching, and provide
+infrastructure around improving performance, memory utilization, and
+generally making all of this easier.
+
+Data in the Intelligence Framework is the atomic piece of intelligence
+such as an IP address or an e-mail address along with a suite of
+metadata about it such as a freeform source field, a freeform
+descriptive field and a URL which might lead to more information about
+the specific item.  The metadata in the default scripts has been
+deliberately kept minimal so that the community can find the
+appropriate fields that need added by writing scripts which extend the
+base record using the normal record extension mechanism.
+
+Quick Start
+-----------
+
+Load the package of scripts that sends data into the Intelligence
+Framework to be checked by loading this script in local.bro::
+
+	@load policy/frameworks/intel
+
+(TODO: find some good mechanism for getting setup with good data
+quickly)
+
+Refer to the "Loading Intelligence" section below to see the format
+for Intelligence Framework text files, then load those text files with
+this line in local.bro::
+
+	redef Intel::read_files += { "/somewhere/yourdata.txt" };
+
+The data itself only needs to reside on the manager if running in a
+cluster.
+
+Architecture
+------------
+
+The Intelligence Framework can be thought of as containing three
+separate portions.  The first part is how intelligence is loaded,
+followed by the mechanism for indicating to the intelligence framework
+that a piece of data which needs to be checked has been seen, and
+thirdly the part where a positive match has been discovered.
+
+Loading Intelligence
+********************
+
+Intelligence data can only be loaded through plain text files using
+the Input Framework conventions.  Additionally, on clusters the
+manager is the only node that needs the intelligence data.  The
+intelligence framework has distribution mechanisms which will push
+data out to all of the nodes that need it.
+
+Here is an example of the intelligence data format.  Note that all
+whitespace separators are literal tabs and fields containing only a
+hyphen a considered to be null values.::
+
+	#fields	host	net	str	str_type	meta.source	meta.desc	meta.url
+	1.2.3.4	-	-	-	source1	Sending phishing email	http://source1.com/badhosts/1.2.3.4
+	-	31.131.248.0/21	-	-	spamhaus-drop	SBL154982	-	-
+	-	-	a.b.com	Intel::DOMAIN	source2	Name used for data exfiltration	-
+
+For more examples of built in `str_type` values, please refer to the
+autogenerated documentation for the intelligence framework (TODO:
+figure out how to do this link).
+
+To load the data once files are created, use the following example
+code to define files to load with your own file names of course::
+
+	redef Intel::read_files += {
+		"/somewhere/feed1.txt",
+		"/somewhere/feed2.txt",
+	};
+
+Remember, the files only need to be present on the file system of the
+manager node on cluster deployments.
+
+Seen Data
+*********
+
+When some bit of data is extracted (such as an email address in the
+"From" header in a message over SMTP), the Intelligence Framework
+needs to be informed that this data was discovered and it's presence
+should be checked within the intelligence data set.  This is
+accomplished through the Intel::seen (TODO: do a reference link)
+function.
+
+Typically users won't need to work with this function due to built in
+hook scripts that Bro ships with that will "see" data and send it into
+the intelligence framework.  A user may only need to load the entire
+package of hook scripts as a module or pick and choose specific
+scripts to load.  Keep in mind that as more data is sent into the
+intelligence framework, the CPU load consumed by Bro will increase
+depending on how many times the Intel::seen function is being called
+which is heavily traffic dependent.
+
+The full package of hook scripts that Bro ships with for sending this
+"seen" data into the intelligence framework can be loading by adding
+this line to local.bro::
+
+	@load policy/frameworks/intel
+
+Intelligence Matches
+********************
+
+Against all hopes, most networks will eventually have a hit on
+intelligence data which could indicate a possible compromise or other
+unwanted activity.  The Intelligence Framework provides an event that
+is generated whenever a match is discovered named Intel::match (TODO:
+make a link to inline docs).  Due to design restrictions placed upon
+the intelligence framework, there is no assurance as to where this
+event will be generated.  It could be generated on the worker where
+the data was seen or on the manager.  When the Intel::match event is
+handled, only the data given as event arguments to the event can be
+assured since the host where the data was seen may not be where
+Intel::match is handled.
+

doc/scripts/DocSourcesList.cmake

@@ -36,6 +36,8 @@ rest_target(${psd} base/frameworks/input/main.bro)
 rest_target(${psd} base/frameworks/input/readers/ascii.bro)
 rest_target(${psd} base/frameworks/input/readers/benchmark.bro)
 rest_target(${psd} base/frameworks/input/readers/raw.bro)
+rest_target(${psd} base/frameworks/intel/cluster.bro)
+rest_target(${psd} base/frameworks/intel/input.bro)
 rest_target(${psd} base/frameworks/intel/main.bro)
 rest_target(${psd} base/frameworks/logging/main.bro)
 rest_target(${psd} base/frameworks/logging/postprocessors/scp.bro)
@@ -103,11 +105,21 @@ rest_target(${psd} base/utils/patterns.bro)
 rest_target(${psd} base/utils/site.bro)
 rest_target(${psd} base/utils/strings.bro)
 rest_target(${psd} base/utils/thresholds.bro)
+rest_target(${psd} base/utils/urls.bro)
 rest_target(${psd} policy/frameworks/communication/listen.bro)
 rest_target(${psd} policy/frameworks/control/controllee.bro)
 rest_target(${psd} policy/frameworks/control/controller.bro)
 rest_target(${psd} policy/frameworks/dpd/detect-protocols.bro)
 rest_target(${psd} policy/frameworks/dpd/packet-segment-logging.bro)
+rest_target(${psd} policy/frameworks/intel/conn-established.bro)
+rest_target(${psd} policy/frameworks/intel/dns.bro)
+rest_target(${psd} policy/frameworks/intel/http-host-header.bro)
+rest_target(${psd} policy/frameworks/intel/http-url.bro)
+rest_target(${psd} policy/frameworks/intel/http-user-agents.bro)
+rest_target(${psd} policy/frameworks/intel/smtp-url-extraction.bro)
+rest_target(${psd} policy/frameworks/intel/smtp.bro)
+rest_target(${psd} policy/frameworks/intel/ssl.bro)
+rest_target(${psd} policy/frameworks/intel/where-locations.bro)
 rest_target(${psd} policy/frameworks/metrics/conn-example.bro)
 rest_target(${psd} policy/frameworks/metrics/http-example.bro)
 rest_target(${psd} policy/frameworks/metrics/ssl-example.bro)
@@ -115,6 +127,7 @@ rest_target(${psd} policy/frameworks/software/version-changes.bro)
 rest_target(${psd} policy/frameworks/software/vulnerable.bro)
 rest_target(${psd} policy/integration/barnyard2/main.bro)
 rest_target(${psd} policy/integration/barnyard2/types.bro)
+rest_target(${psd} policy/integration/collective-intel/main.bro)
 rest_target(${psd} policy/misc/analysis-groups.bro)
 rest_target(${psd} policy/misc/capture-loss.bro)
 rest_target(${psd} policy/misc/loaded-scripts.bro)
@@ -129,7 +142,6 @@ rest_target(${psd} policy/protocols/dns/detect-external-names.bro)
 rest_target(${psd} policy/protocols/ftp/detect.bro)
 rest_target(${psd} policy/protocols/ftp/software.bro)
 rest_target(${psd} policy/protocols/http/detect-MHR.bro)
-rest_target(${psd} policy/protocols/http/detect-intel.bro)
 rest_target(${psd} policy/protocols/http/detect-sqli.bro)
 rest_target(${psd} policy/protocols/http/detect-webapps.bro)
 rest_target(${psd} policy/protocols/http/header-names.bro)

scripts/base/frameworks/intel/__load__.bro

@@ -1 +1,11 @@
 @load ./main
+
+# The cluster framework must be loaded first.
+@load base/frameworks/cluster
+
+@if ( Cluster::is_enabled() )
+@load ./cluster
+@endif
+
+# This needs cluster support to only read on the manager.
+@load ./input

scripts/base/frameworks/intel/cluster.bro

@@ -0,0 +1,61 @@
+##! Cluster transparency support for the intelligence framework.  This is mostly oriented
+##! toward distributing intelligence information across clusters.
+
+@load base/frameworks/cluster
+@load ./input
+
+module Intel;
+
+redef record Item += {
+	## This field is used internally for cluster transparency to avoid 
+	## re-dispatching intelligence items over and over from workers.
+	first_dispatch: bool &default=T;
+};
+
+# If this process is not a manager process, we don't want the full metadata
+@if ( Cluster::local_node_type() != Cluster::MANAGER )
+redef have_full_data = F;
+@endif
+
+global cluster_new_item: event(item: Item);
+
+# Primary intelligence distribution comes from manager.
+redef Cluster::manager2worker_events += /^Intel::(cluster_new_item)$/;
+# If a worker finds intelligence and adds it, it should share it back to the manager.
+redef Cluster::worker2manager_events += /^Intel::(cluster_new_item|match_no_items)$/;
+
+@if ( Cluster::local_node_type() == Cluster::MANAGER )
+event Intel::match_no_items(s: Seen) &priority=5
+	{
+	event Intel::match(s, Intel::get_items(s));
+	}
+
+event remote_connection_handshake_done(p: event_peer)
+	{
+	# When a worker connects, send it the complete minimal data store.
+	# It will be kept up to date after this by the cluster_new_item event.
+	if ( Cluster::nodes[p$descr]$node_type == Cluster::WORKER )
+		{
+		send_id(p, "Intel::min_data_store");
+		}
+	}
+@endif
+
+event Intel::cluster_new_item(item: Intel::Item) &priority=5
+	{
+	# Ignore locally generated events to avoid event storms.
+	if ( is_remote_event() )
+		Intel::insert(item);
+	}
+
+event Intel::new_item(item: Intel::Item) &priority=5
+	{
+	# The cluster manager always rebroadcasts intelligence.
+	# Workers redistribute it if it was locally generated.
+	if ( Cluster::local_node_type() == Cluster::MANAGER ||
+	     item$first_dispatch )
+		{
+		item$first_dispatch=F;
+		event Intel::cluster_new_item(item);
+		}
+	}

scripts/base/frameworks/intel/input.bro

@@ -0,0 +1,33 @@
+@load ./main
+
+module Intel;
+
+export {
+	## Intelligence files that will be read off disk.  The files are
+	## reread everytime they are updated so updates much be atomic with
+	## "mv" instead of writing the file in place.
+	const read_files: set[string] = {} &redef;
+}
+
+event Intel::read_entry(desc: Input::EventDescription, tpe: Input::Event, item: Intel::Item)
+	{
+	Intel::insert(item);
+	}
+
+event bro_init() &priority=5
+	{
+	if ( ! Cluster::is_enabled() ||
+	     Cluster::local_node_type() == Cluster::MANAGER )
+		{
+		for ( a_file in read_files )
+			{
+			Input::add_event([$source=a_file,
+			                  $reader=Input::READER_ASCII,
+			                  $mode=Input::REREAD,
+			                  $name=cat("intel-", a_file),
+			                  $fields=Intel::Item,
+			                  $ev=Intel::read_entry]);
+			}
+		}
+	}
+

scripts/base/frameworks/intel/main.bro

@@ -1,323 +1,345 @@
 ##! The intelligence framework provides a way to store and query IP addresses,
-##! strings (with a subtype), and numeric (with a subtype) data.  Metadata 
-##! also be associated with the intelligence like tags which are arbitrary
-##! strings, time values, and longer descriptive strings.
-
-# Example string subtypes:
-#   url
-#   email
-#   domain
-#   software
-#   user_name
-#   file_name
-#   file_md5
-#   x509_md5
-
-# Example tags: 
-#   infrastructure
-#   malicious
-#   sensitive
-#   canary
-#   friend
+##! and strings (with a str_type).  Metadata can
+##! also be associated with the intelligence like for making more informed
+##! decisions about matching and handling of intelligence.
 
 @load base/frameworks/notice
 
 module Intel;
 
 export {
-	## The intel logging stream identifier.
 	redef enum Log::ID += { LOG };
 	
-	redef enum Notice::Type += {
-		## This notice should be used in all detector scripts to indicate 
-		## an intelligence based detection.
-		Detection,
+	## String data needs to be further categoried since it could represent
+	## and number of types of data.
+	type StrType: enum {
+		## A complete URL without the prefix "http://".
+		URL,
+		## User-Agent string, typically HTTP or mail message body.
+		USER_AGENT,
+		## Email address.
+		EMAIL,
+		## DNS domain name.
+		DOMAIN,
+		## A user name.
+		USER_NAME,
+		## File hash which is non-hash type specific.  It's up to the user to query
+		## for any relevant hash types.
+		FILE_HASH,
+		## Certificate SHA-1 hash.
+		CERT_HASH,
 	};
 	
-	## Record type used for logging information from the intelligence framework.
-	## Primarily for problems or oddities with inserting and querying data.  
-	## This is important since the content of the intelligence framework can 
-	## change quite dramatically during runtime and problems may be introduced 
-	## into the data.
-	type Info: record {
-		## The current network time.
-		ts:      time   &log;
-		## Represents the severity of the message. 
-		## This value should be one of: "info", "warn", "error"
-		level:   string &log;
-		## The message.
-		message: string &log;
-	};
-	
-	## Record to represent metadata associated with a single piece of
-	## intelligence.
+	## Data about an :bro:type:`Intel::Item`
 	type MetaData: record {
-		## A description for the data.
+		## An arbitrary string value representing the data source.  Typically,
+		## the convention for this field will be the source name and feed name
+		## separated by a hyphen.  For example: "source1-c&c".
+		source:      string;
+		## A freeform description for the data.
 		desc:        string      &optional;
-		## A URL where more information may be found about the intelligence.
+		## A URL for more information about the data.
 		url:         string      &optional;
-		## The time at which the data was first declared to be intelligence.
-		first_seen:  time        &optional;
-		## When this data was most recent inserted into the framework.
-		latest_seen: time        &optional;
-		## Arbitrary text tags for the data.
-		tags:        set[string];
 	};
 	
-	## Record to represent a singular piece of intelligence.
+	## Represents a piece of intelligence.
 	type Item: record {
-		## If the data is an IP address, this hold the address.
-		ip:          addr        &optional;
-		## If the data is textual, this holds the text.
+		## The IP address if the intelligence is about an IP address.
+		host:        addr           &optional;
+		## The network if the intelligence is about a CIDR block.
+		net:         subnet         &optional;
+		## The string if the intelligence is about a string.
 		str:         string         &optional;
-		## If the data is numeric, this holds the number.
-		num:         int         &optional;
-		## The subtype of the data for when either the $str or $num fields are
-		## given.  If one of those fields are given, this field must be present.
-		subtype:     string      &optional;
+		## The type of data that is in the string if the $str field is set.
+		str_type:    StrType        &optional;
 		
-		## The next five fields are temporary until a better model for 
-		## attaching metadata to an intelligence item is created.
-		desc:        string      &optional;
-		url:         string      &optional;
-		first_seen:  time        &optional;
-		latest_seen: time        &optional;
-		tags:        set[string];
-		
-		## These single string tags are throw away until pybroccoli supports sets.
-		tag1: string &optional;
-		tag2: string &optional;
-		tag3: string &optional;
+		## Metadata for the item.  Typically represents more deeply \
+		## descriptive data for a piece of intelligence.
+		meta:        MetaData;
 	};
 	
-	## Record model used for constructing queries against the intelligence 
-	## framework.
-	type QueryItem: record {
-		## If an IP address is being queried for, this field should be given.
-		ip:        addr        &optional;
-		## If a string is being queried for, this field should be given.
-		str:       string      &optional;
-		## If numeric data is being queried for, this field should be given.
-		num:       int         &optional;
-		## If either a string or number is being queried for, this field should
-		## indicate the subtype of the data.
-		subtype:   string      &optional;
-		
-		## A set of tags where if a single metadata record attached to an item
-		## has any one of the tags defined in this field, it will match.
-		or_tags:   set[string] &optional;
-		## A set of tags where a single metadata record attached to an item 
-		## must have all of the tags defined in this field.
-		and_tags:  set[string] &optional; 
-		
-		## The predicate can be given when searching for a match.  It will
-		## be tested against every :bro:type:`Intel::MetaData` item associated
-		## with the data being matched on.  If it returns T a single time, the 
-		## matcher will consider that the item has matched.  This field can
-		## be used for constructing arbitrarily complex queries that may not
-		## be possible with the $or_tags or $and_tags fields.
-		pred:      function(meta: Intel::MetaData): bool &optional;
+	## Enum to represent where data came from when it was discovered.
+	## The convenction is to prefix the name with "IN_".
+	type Where: enum {
+		## A catchall value to represent data of unknown provenance.
+		IN_ANYWHERE,
 	};
 
-	## Function to insert data into the intelligence framework.
-	## 
-	## item: The data item.
-	##
-	## Returns: T if the data was successfully inserted into the framework,
-	##          otherwise it returns F.
-	global insert: function(item: Item): bool;
+	## The $host field and combination of $str and $str_type fields are mutually 
+	## exclusive.  These records *must* represent either an IP address being
+	## seen or a string being seen.
+	type Seen: record {
+		## The IP address if the data seen is an IP address.
+		host:      addr          &log &optional;
+		## The string if the data is about a string.
+		str:       string        &log &optional;
+		## The type of data that is in the string if the $str field is set.
+		str_type:  StrType       &log &optional;
 
-	## A wrapper for the :bro:id:`Intel::insert` function.  This is primarily
-	## used as the external API for inserting data into the intelligence 
-	## using Broccoli.
-	global insert_event: event(item: Item);
+		## Where the data was discovered.
+		where:     Where         &log;
 		
-	## Function for matching data within the intelligence framework.
-	global matcher: function(item: QueryItem): bool;
+		## If the data was discovered within a connection, the 
+		## connection record should go into get to give context to the data.
+		conn:      connection    &optional;
+	};
+
+	## Record used for the logging framework representing a positive
+	## hit within the intelligence framework.
+	type Info: record {
+		## Timestamp when the data was discovered.
+		ts:       time           &log;
+
+		## If a connection was associated with this intelligence hit,
+		## this is the uid for the connection
+		uid:      string         &log &optional;
+		## If a connection was associated with this intelligence hit,
+		## this is the conn_id for the connection.
+		id:       conn_id        &log &optional;
+
+		## Where the data was seen.
+		seen:     Seen           &log;
+		## Sources which supplied data that resulted in this match.
+		sources:  set[string]    &log;
+	};
+
+	## Intelligence data manipulation functions.
+	global insert: function(item: Item);
+
+	## Function to declare discovery of a piece of data in order to check
+	## it against known intelligence for matches.
+	global seen: function(s: Seen);
+
+	## Event to represent a match in the intelligence data from data that was seen.  
+	## On clusters there is no assurance as to where this event will be generated 
+	## so do not assume that arbitrary global state beyond the given data
+	## will be available.
+	##
+	## This is the primary mechanism where a user will take actions based on data
+	## within the intelligence framework.
+	global match: event(s: Seen, items: set[Item]);
+
+	global log_intel: event(rec: Info);
 }
 
-type MetaDataStore: table[count] of MetaData;
+# Internal handler for matches with no metadata available.
+global match_no_items: event(s: Seen);
+
+# Internal events for cluster data distribution
+global new_item: event(item: Item);
+global updated_item: event(item: Item);
+
+# Optionally store metadata.  This is used internally depending on
+# if this is a cluster deployment or not.
+const have_full_data = T &redef;
+
+# The in memory data structure for holding intelligence.
 type DataStore: record {
-	ip_data:     table[addr] of MetaDataStore;
-	# The first string is the actual value and the second string is the subtype.
-	string_data: table[string, string] of MetaDataStore;
-	int_data:    table[int, string] of MetaDataStore;
+	net_data:    table[subnet] of set[MetaData];
+	string_data: table[string, StrType] of set[MetaData];
 };
-global data_store: DataStore;
+global data_store: DataStore &redef;
 
-event bro_init()
+# The in memory data structure for holding the barest matchable intelligence.
+# This is primarily for workers to do the initial quick matches and store
+# a minimal amount of data for the full match to happen on the manager.
+type MinDataStore: record {
+	net_data:    set[subnet];
+	string_data: set[string, StrType];
+};
+global min_data_store: MinDataStore &redef;
+
+
+event bro_init() &priority=5
 	{
-	Log::create_stream(Intel::LOG, [$columns=Info]);
+	Log::create_stream(LOG, [$columns=Info, $ev=log_intel]);
+	}
+
+function find(s: Seen): bool
+	{
+	if ( s?$host && 
+	     ((have_full_data && s$host in data_store$net_data) || 
+	      (s$host in min_data_store$net_data)))
+		{
+		return T;
+		}
+	else if ( s?$str && s?$str_type &&
+	          ((have_full_data && [s$str, s$str_type] in data_store$string_data) ||
+	           ([s$str, s$str_type] in min_data_store$string_data)))
+		{
+		return T;
+		}
+	else
+		{
+		return F;
+		}
+	}
+
+function get_items(s: Seen): set[Item]
+	{
+	local item: Item;
+	local return_data: set[Item] = set();
+
+	if ( ! have_full_data )
+		{
+		# A reporter warning should be generated here because this function
+		# should never be called from a host that doesn't have the full data.
+		# TODO: do a reporter warning.
+		return return_data;
+		}
+
+	if ( s?$host )
+		{
+		# See if the host is known about and it has meta values
+		if ( s$host in data_store$net_data )
+			{
+			for ( m in data_store$net_data[s$host] )
+				{
+				# TODO: the lookup should be finding all and not just most specific
+				#       and $host/$net should have the correct value.
+				item = [$host=s$host, $meta=m];
+				add return_data[item];
+				}
+			}
+		}
+	else if ( s?$str && s?$str_type )
+		{
+		# See if the string is known about and it has meta values
+		if ( [s$str, s$str_type] in data_store$string_data )
+			{
+			for ( m in data_store$string_data[s$str, s$str_type] )
+				{
+				item = [$str=s$str, $str_type=s$str_type, $meta=m];
+				add return_data[item];
+				}
+			}
+		}
+
+	return return_data;
+	}
+
+function Intel::seen(s: Seen)
+	{
+	if ( find(s) )
+		{
+		if ( have_full_data )
+			{
+			local items = get_items(s);
+			event Intel::match(s, items);
+			}
+		else
+			{
+			event Intel::match_no_items(s);
+			}
+		}
 	}
 
 
-function insert(item: Item): bool
+function has_meta(check: MetaData, metas: set[MetaData]): bool
 	{
-	local err_msg = "";
-	if ( (item?$str || item?$num) && ! item?$subtype )
-		err_msg = "You must provide a subtype to insert_sync or this item doesn't make sense.";
+	local check_hash = md5_hash(check);
+	for ( m in metas )
+		{
+		if ( check_hash == md5_hash(m) )
+			return T;
+		}
 
-	if ( err_msg == "" )
+	# The records must not be equivalent if we made it this far.
+	return F;
+	}
+
+event Intel::match(s: Seen, items: set[Item]) &priority=5
 	{
+	local empty_set: set[string] = set();
+	local info: Info = [$ts=network_time(), $seen=s, $sources=empty_set];
+
+	if ( s?$conn )
+		{
+		info$uid = s$conn$uid;
+		info$id  = s$conn$id;
+		}
+
+	for ( item in items )
+		add info$sources[item$meta$source];
+
+	Log::write(Intel::LOG, info);
+	}
+
+function insert(item: Item)
+	{
+	if ( item?$str && !item?$str_type )
+		{
+		event reporter_warning(network_time(), fmt("You must provide a str_type for strings or this item doesn't make sense.  Item: %s", item), "");
+		return;
+		}
+
 	# Create and fill out the meta data item.
-		local meta: MetaData;
-		if ( item?$first_seen )
-			meta$first_seen = item$first_seen;
-		if ( item?$latest_seen )
-			meta$latest_seen = item$latest_seen;
-		if ( item?$tags )
-			meta$tags = item$tags;
-		if ( item?$desc )
-			meta$desc = item$desc;
-		if ( item?$url )
-			meta$url = item$url;
+	local meta = item$meta;
+	local metas: set[MetaData];
 
-		
-		# This is hopefully only temporary until pybroccoli supports sets.
-		if ( item?$tag1 )
-			add item$tags[item$tag1];
-		if ( item?$tag2 )
-			add item$tags[item$tag2];
-		if ( item?$tag3 )
-			add item$tags[item$tag3];
-		
-		if ( item?$ip )
+	if ( item?$host )
 		{
-			if ( item$ip !in data_store$ip_data )
-				data_store$ip_data[item$ip] = table();
-			data_store$ip_data[item$ip][|data_store$ip_data[item$ip]|] = meta;
-			return T;
+		local host = mask_addr(item$host, is_v4_addr(item$host) ? 32 : 128);
+		if ( have_full_data )
+			{
+			if ( host !in data_store$net_data )
+				data_store$net_data[host] = set();
+
+			metas = data_store$net_data[host];
+			}
+
+		add min_data_store$net_data[host];
+		}
+	else if ( item?$net )
+		{
+		if ( have_full_data )
+			{
+			if ( item$net !in data_store$net_data )
+				data_store$net_data[item$net] = set();
+
+			metas = data_store$net_data[item$net];
+			}
+
+		add min_data_store$net_data[item$net];
 		}
 	else if ( item?$str )
 		{
-			if ( [item$str, item$subtype] !in data_store$string_data )
-				data_store$string_data[item$str, item$subtype] = table();
-			
-			data_store$string_data[item$str, item$subtype][|data_store$string_data[item$str, item$subtype]|] = meta;
-			return T;
-			}
-		else if ( item?$num )
+		if ( have_full_data )
 			{
-			if ( [item$num, item$subtype] !in data_store$int_data )
-				data_store$int_data[item$num, item$subtype] = table();
+			if ( [item$str, item$str_type] !in data_store$string_data )
+				data_store$string_data[item$str, item$str_type] = set();
 
-			data_store$int_data[item$num, item$subtype][|data_store$int_data[item$num, item$subtype]|] = meta;
-			return T;
+			metas = data_store$string_data[item$str, item$str_type];
+			}
+
+		add min_data_store$string_data[item$str, item$str_type];
+		}
+
+	local updated = F;
+	if ( have_full_data )
+		{
+		for ( m in metas )
+			{
+			if ( meta$source == m$source )
+				{
+				if ( has_meta(meta, metas) )
+					{
+					# It's the same item being inserted again.
+					return;
 					}
 				else
-			err_msg = "Failed to insert intelligence item for some unknown reason.";
+					{
+					# Same source, different metadata means updated item.
+					updated = T;
+					}
+				}
+			}
+		add metas[item$meta];
 		}
 	
-	if ( err_msg != "" )
-		Log::write(Intel::LOG, [$ts=network_time(), $level="warn", $message=fmt(err_msg)]);
-	return F;
-	}
-	
-event insert_event(item: Item)
-	{
-	insert(item);
-	}
-	
-function match_item_with_metadata(item: QueryItem, meta: MetaData): bool
-	{
-	if ( item?$and_tags )
-		{
-		local matched = T;
-		# Every tag given has to match in a single MetaData entry.
-		for ( tag in item$and_tags )
-			{
-			if ( tag !in meta$tags )
-				matched = F;
-			}
-		if ( matched )
-			return T;
-		}
-	else if ( item?$or_tags )
-		{
-		# For OR tags, only a single tag has to match.
-		for ( tag in item$or_tags )
-			{
-			if ( tag in meta$tags )
-				return T;
-			}
-		}
-	else if ( item?$pred )
-		return item$pred(meta);
-
-	# This indicates some sort of failure in the query
-	return F;
-	}
-	
-function matcher(item: QueryItem): bool
-	{
-	local err_msg = "";
-	if ( ! (item?$ip || item?$str || item?$num) )
-		err_msg = "You must supply one of the $ip, $str, or $num fields to search on";
-	else if ( (item?$or_tags || item?$and_tags) && item?$pred )
-		err_msg = "You can't match with both tags and a predicate.";
-	else if ( item?$or_tags && item?$and_tags )
-		err_msg = "You can't match with both OR'd together tags and AND'd together tags";
-	else if ( (item?$str || item?$num) && ! item?$subtype )
-		err_msg = "You must provide a subtype to matcher or this item doesn't make sense.";
-	else if ( item?$str && item?$num )
-		err_msg = "You must only provide $str or $num, not both.";
-		
-	local meta: MetaData;
-
-	if ( err_msg == "" )
-		{
-		if ( item?$ip )
-			{
-			if ( item$ip in data_store$ip_data )
-				{
-				if ( ! item?$and_tags && ! item?$or_tags && ! item?$pred )
-					return T;
-			
-				for ( i in data_store$ip_data[item$ip] )
-					{
-					meta = data_store$ip_data[item$ip][i];
-					if ( match_item_with_metadata(item, meta) )
-						return T;
-					}
-				}
-			}
-		
-		else if ( item?$str )
-			{
-			if ( [item$str, item$subtype] in data_store$string_data )
-				{
-				if ( ! item?$and_tags && ! item?$or_tags && ! item?$pred )
-					return T;
-
-				for ( i in data_store$string_data[item$str, item$subtype] )
-					{
-					meta = data_store$string_data[item$str, item$subtype][i];
-					if ( match_item_with_metadata(item, meta) )
-						return T;
-					}
-				}
-			}
-		
-		else if ( item?$num )
-			{
-			if ( [item$num, item$subtype] in data_store$int_data )
-				{
-				if ( ! item?$and_tags && ! item?$or_tags && ! item?$pred )
-					return T;
-
-				for ( i in data_store$int_data[item$num, item$subtype] )
-					{
-					meta = data_store$int_data[item$num, item$subtype][i];
-					if ( match_item_with_metadata(item, meta) )
-						return T;
-					}
-				}
-			}
+	if ( updated )
+		event Intel::updated_item(item);
 	else
-			err_msg = "Failed to query intelligence data for some unknown reason.";
+		event Intel::new_item(item);
 	}
 	
-	if ( err_msg != "" )
-		Log::write(Intel::LOG, [$ts=network_time(), $level="error", $message=fmt(err_msg)]);
-	return F;
-	}

scripts/base/init-default.bro

@@ -14,6 +14,7 @@
 @load base/utils/patterns
 @load base/utils/strings
 @load base/utils/thresholds
+@load base/utils/urls
 
 # This has some deep interplay between types and BiFs so it's 
 # loaded in base/init-bare.bro

scripts/base/utils/urls.bro

@@ -0,0 +1,25 @@
+## Functions for URL handling.
+
+## A regular expression for matching and extracting URLs.
+const url_regex = /^([a-zA-Z\-]{3,5})(:\/\/[^\/?#"'\r\n><]*)([^?#"'\r\n><]*)([^[:blank:]\r\n"'><]*|\??[^"'\r\n><]*)/ &redef;
+
+## Extracts URLs discovered in arbitrary text.
+function find_all_urls(s: string): string_set
+	{
+	return find_all(s, url_regex);
+	}
+
+## Extracts URLs discovered in arbitrary text without
+## the URL scheme included.
+function find_all_urls_without_scheme(s: string): string_set
+	{
+	local urls = find_all_urls(s);
+	local return_urls: set[string] = set();
+	for ( url in urls )
+		{
+		local no_scheme = sub(url, /^([a-zA-Z\-]{3,5})(:\/\/)/, "");
+		add return_urls[no_scheme];
+		}
+
+	return return_urls;
+	}

scripts/policy/frameworks/intel/__load__.bro

@@ -0,0 +1,8 @@
+@load ./conn-established
+@load ./dns
+@load ./http-host-header
+@load ./http-url
+@load ./http-user-agents
+@load ./ssl
+@load ./smtp
+@load ./smtp-url-extraction

scripts/policy/frameworks/intel/conn-established.bro

@@ -0,0 +1,8 @@
+@load base/frameworks/intel
+@load ./where-locations
+
+event connection_established(c: connection)
+	{
+	Intel::seen([$host=c$id$orig_h, $conn=c, $where=Conn::IN_ORIG]);
+	Intel::seen([$host=c$id$resp_h, $conn=c, $where=Conn::IN_RESP]);
+	}

scripts/policy/frameworks/intel/dns.bro

@@ -0,0 +1,10 @@
+@load base/frameworks/intel
+@load ./where-locations
+
+event dns_request(c: connection, msg: dns_msg, query: string, qtype: count, qclass: count)
+	{
+	Intel::seen([$str=query,
+	             $str_type=Intel::DOMAIN,
+	             $conn=c,
+	             $where=DNS::IN_REQUEST]);
+	}

scripts/policy/frameworks/intel/http-host-header.bro

@@ -0,0 +1,11 @@
+@load base/frameworks/intel
+@load ./where-locations
+
+event http_header(c: connection, is_orig: bool, name: string, value: string)
+	{
+	if ( is_orig && name == "HOST" )
+		Intel::seen([$str=value,
+		             $str_type=Intel::DOMAIN,
+		             $conn=c,
+		             $where=HTTP::IN_HOST_HEADER]);
+	}

scripts/policy/frameworks/intel/http-url.bro

@@ -0,0 +1,11 @@
+@load base/frameworks/intel
+@load ./where-locations
+
+event http_message_done(c: connection, is_orig: bool, stat: http_message_stat)
+	{
+	if ( is_orig && c?$http )
+		Intel::seen([$str=HTTP::build_url(c$http),
+		             $str_type=Intel::URL,
+		             $conn=c,
+		             $where=HTTP::IN_URL]);
+	}

scripts/policy/frameworks/intel/http-user-agents.bro

@@ -0,0 +1,12 @@
+@load base/frameworks/intel
+@load ./where-locations
+
+event http_header(c: connection, is_orig: bool, name: string, value: string)
+	{
+	if ( is_orig && name == "USER-AGENT" )
+		Intel::seen([$str=value,
+		             $str_type=Intel::USER_AGENT,
+		             $conn=c,
+		             $where=HTTP::IN_USER_AGENT_HEADER]);
+	}
+

scripts/policy/frameworks/intel/smtp-url-extraction.bro

@@ -0,0 +1,15 @@
+@load base/frameworks/intel
+@load base/utils/urls
+@load ./where-locations
+
+event mime_segment_data(c: connection, length: count, data: string) &priority=3
+	{
+	local urls = find_all_urls_without_scheme(data);
+	for ( url in urls )
+		{
+		Intel::seen([$str=url,
+		             $str_type=Intel::URL,
+		             $conn=c,
+		             $where=SMTP::IN_MESSAGE]);
+		}
+	}

scripts/policy/frameworks/intel/smtp.bro

@@ -0,0 +1,70 @@
+@load base/frameworks/intel
+@load ./where-locations
+
+event mime_end_entity(c: connection)
+	{
+	if ( c?$smtp )
+		{
+		if ( c$smtp?$path )
+			{
+			local path = c$smtp$path;
+			for ( i in path )
+				{
+				Intel::seen([$host=path[i],
+				             $conn=c,
+				             $where=SMTP::IN_RECEIVED_HEADER]);
+				}
+			}
+
+		if ( c$smtp?$user_agent )
+			Intel::seen([$str=c$smtp$user_agent,
+			             $str_type=Intel::USER_AGENT,
+			             $conn=c,
+			             $where=SMTP::IN_HEADER]);
+
+		if ( c$smtp?$x_originating_ip )
+			Intel::seen([$host=c$smtp$x_originating_ip,
+			             $conn=c,
+			             $where=SMTP::IN_X_ORIGINATING_IP_HEADER]);
+
+		if ( c$smtp?$mailfrom )
+			Intel::seen([$str=c$smtp$mailfrom,
+			             $str_type=Intel::EMAIL,
+			             $conn=c,
+			             $where=SMTP::IN_MAIL_FROM]);
+
+		if ( c$smtp?$rcptto )
+			{
+			for ( rcptto in c$smtp$rcptto )
+				{
+				Intel::seen([$str=rcptto,
+				             $str_type=Intel::EMAIL,
+				             $conn=c,
+				             $where=SMTP::IN_RCPT_TO]);
+				}
+			}
+
+		if ( c$smtp?$from )
+			Intel::seen([$str=c$smtp$from,
+			             $str_type=Intel::EMAIL,
+			             $conn=c,
+			             $where=SMTP::IN_FROM]);
+
+		if ( c$smtp?$to )
+			{
+			for ( email_to in c$smtp$to )
+				{
+				Intel::seen([$str=email_to,
+				             $str_type=Intel::EMAIL,
+				             $conn=c,
+				             $where=SMTP::IN_TO]);
+				}
+			}
+
+		if ( c$smtp?$reply_to )
+			Intel::seen([$str=c$smtp$reply_to,
+			             $str_type=Intel::EMAIL,
+			             $conn=c,
+			             $where=SMTP::IN_REPLY_TO]);
+		}
+	}

scripts/policy/frameworks/intel/ssl.bro

@@ -0,0 +1,33 @@
+@load base/frameworks/intel
+@load ./where-locations
+
+event x509_certificate(c: connection, is_orig: bool, cert: X509, chain_idx: count, chain_len: count, der_cert: string)
+	{
+	if ( chain_idx == 0 )
+		{
+		if ( /emailAddress=/ in cert$subject )
+			{
+			local email = sub(cert$subject, /^.*emailAddress=/, "");
+			email = sub(email, /,.*$/, "");
+			Intel::seen([$str=email,
+			             $str_type=Intel::EMAIL,
+			             $conn=c,
+			             $where=(is_orig ? SSL::IN_CLIENT_CERT : SSL::IN_SERVER_CERT)]);
+			}
+
+		Intel::seen([$str=sha1_hash(der_cert),
+		             $str_type=Intel::CERT_HASH,
+		             $conn=c,
+		             $where=(is_orig ? SSL::IN_CLIENT_CERT : SSL::IN_SERVER_CERT)]);
+		}
+	}
+
+event ssl_extension(c: connection, is_orig: bool, code: count, val: string)
+	{
+	if ( is_orig && SSL::extensions[code] == "server_name" && 
+	     c?$ssl && c$ssl?$server_name )
+		Intel::seen([$str=c$ssl$server_name,
+		             $str_type=Intel::DOMAIN,
+		             $conn=c,
+		             $where=SSL::IN_SERVER_NAME]);
+	}

scripts/policy/frameworks/intel/where-locations.bro

@@ -0,0 +1,25 @@
+@load base/frameworks/intel
+
+export {
+	redef enum Intel::Where += {
+		Conn::IN_ORIG,
+		Conn::IN_RESP,
+		DNS::IN_REQUEST,
+		DNS::IN_RESPONSE,
+		HTTP::IN_HOST_HEADER,
+		HTTP::IN_USER_AGENT_HEADER,
+		HTTP::IN_URL,
+		SMTP::IN_MAIL_FROM,
+		SMTP::IN_RCPT_TO,
+		SMTP::IN_FROM,
+		SMTP::IN_TO,
+		SMTP::IN_RECEIVED_HEADER,
+		SMTP::IN_REPLY_TO,
+		SMTP::IN_X_ORIGINATING_IP_HEADER,
+		SMTP::IN_MESSAGE,
+		SSL::IN_SERVER_CERT,
+		SSL::IN_CLIENT_CERT,
+		SSL::IN_SERVER_NAME,
+		SMTP::IN_HEADER,
+	};
+}

scripts/policy/integration/collective-intel/README

@@ -0,0 +1,6 @@
+Collective Intelligence Framework Integration
+=============================================
+
+The scripts in this module are for deeper integration with the Collective Intelligence
+Framework (CIF) since Bro's Intel framework doesn't natively behave the same as CIF nor
+does it store and maintain the same data in all cases.

scripts/policy/integration/collective-intel/__load__.bro

@@ -0,0 +1 @@
+@load ./main

scripts/policy/integration/collective-intel/main.bro

@@ -0,0 +1,13 @@
+
+module Intel;
+
+## These are some fields to add extended compatibility between Bro and the Collective
+## Intelligence Framework
+redef record Intel::MetaData += {
+	## Maps to the Impact field in the Collective Intelligence Framework.
+	cif_impact:     string &optional;
+	## Maps to the Severity field in the Collective Intelligence Framework.
+	cif_severity:   string &optional;
+	## Maps to the Confidence field in the Collective Intelligence Framework.
+	cif_confidence: double &optional;
+};

scripts/policy/protocols/http/detect-intel.bro

@@ -1,21 +0,0 @@
-##! Intelligence based HTTP detections.  Not yet working!
-
-@load base/protocols/http/main
-@load base/protocols/http/utils
-@load base/frameworks/intel/main
-
-module HTTP;
-
-event log_http(rec: Info)
-	{
-	local url = HTTP::build_url(rec);
-	local query = [$str=url, $subtype="url", $or_tags=set("malicious", "malware")];
-	if ( Intel::matcher(query) )
-		{
-		local msg = fmt("%s accessed a malicious URL from the intelligence framework", rec$id$orig_h);
-		NOTICE([$note=Intel::Detection, 
-		        $msg=msg, 
-		        $sub=HTTP::build_url_http(rec), 
-		        $id=rec$id]);
-		}
-	}

scripts/test-all-policy.bro

@@ -14,6 +14,16 @@
 # @load frameworks/control/controller.bro
 @load frameworks/dpd/detect-protocols.bro
 @load frameworks/dpd/packet-segment-logging.bro
+@load frameworks/intel/__load__.bro
+@load frameworks/intel/conn-established.bro
+@load frameworks/intel/dns.bro
+@load frameworks/intel/http-host-header.bro
+@load frameworks/intel/http-url.bro
+@load frameworks/intel/http-user-agents.bro
+@load frameworks/intel/smtp-url-extraction.bro
+@load frameworks/intel/smtp.bro
+@load frameworks/intel/ssl.bro
+@load frameworks/intel/where-locations.bro
 @load frameworks/metrics/conn-example.bro
 @load frameworks/metrics/http-example.bro
 @load frameworks/metrics/ssl-example.bro
@@ -22,6 +32,8 @@
 @load integration/barnyard2/__load__.bro
 @load integration/barnyard2/main.bro
 @load integration/barnyard2/types.bro
+@load integration/collective-intel/__load__.bro
+@load integration/collective-intel/main.bro
 @load misc/analysis-groups.bro
 @load misc/capture-loss.bro
 @load misc/loaded-scripts.bro
@@ -35,7 +47,6 @@
 @load protocols/dns/detect-external-names.bro
 @load protocols/ftp/detect.bro
 @load protocols/ftp/software.bro
-@load protocols/http/detect-intel.bro
 @load protocols/http/detect-MHR.bro
 @load protocols/http/detect-sqli.bro
 @load protocols/http/detect-webapps.bro

testing/btest/Baseline/coverage.default-load-baseline/canonified_loaded_scripts.log

@@ -40,6 +40,7 @@ scripts/base/init-default.bro
   scripts/base/utils/paths.bro
   scripts/base/utils/strings.bro
   scripts/base/utils/thresholds.bro
+  scripts/base/utils/urls.bro
   scripts/base/frameworks/notice/__load__.bro
     scripts/base/frameworks/notice/./main.bro
     scripts/base/frameworks/notice/./weird.bro
@@ -69,6 +70,7 @@ scripts/base/init-default.bro
     scripts/base/frameworks/metrics/./non-cluster.bro
   scripts/base/frameworks/intel/__load__.bro
     scripts/base/frameworks/intel/./main.bro
+    scripts/base/frameworks/intel/./input.bro
   scripts/base/frameworks/reporter/__load__.bro
     scripts/base/frameworks/reporter/./main.bro
   scripts/base/frameworks/tunnels/__load__.bro

testing/btest/Baseline/coverage.init-default/missing_loads

@@ -2,5 +2,6 @@
 -./frameworks/cluster/nodes/proxy.bro
 -./frameworks/cluster/nodes/worker.bro
 -./frameworks/cluster/setup-connections.bro
+-./frameworks/intel/cluster.bro
 -./frameworks/metrics/cluster.bro
 -./frameworks/notice/cluster.bro

testing/btest/Baseline/scripts.base.frameworks.intel.cluster-transparency/manager-1..stdout

@@ -0,0 +1,2 @@
+cluster_new_item: 123.123.123.123 inserted by worker-1 (from peer: worker-1)
+cluster_new_item: 4.3.2.1 inserted by worker-2 (from peer: worker-2)

testing/btest/Baseline/scripts.base.frameworks.intel.cluster-transparency/manager-1.intel.log

@@ -0,0 +1,10 @@
+#separator \x09
+#set_separator	,
+#empty_field	(empty)
+#unset_field	-
+#path	intel
+#open	2012-10-03-20-20-39
+#fields	ts	uid	id.orig_h	id.orig_p	id.resp_h	id.resp_p	seen.host	seen.str	seen.str_type	seen.where	sources
+#types	time	string	addr	port	addr	port	addr	string	enum	enum	table[string]
+1349295639.424940	-	-	-	-	-	123.123.123.123	-	-	Intel::IN_ANYWHERE	worker-1
+#close	2012-10-03-20-20-49

testing/btest/Baseline/scripts.base.frameworks.intel.cluster-transparency/worker-1..stdout

@@ -0,0 +1,3 @@
+cluster_new_item: 1.2.3.4 inserted by manager (from peer: manager-1)
+cluster_new_item: 123.123.123.123 inserted by worker-1 (from peer: manager-1)
+cluster_new_item: 4.3.2.1 inserted by worker-2 (from peer: manager-1)

testing/btest/Baseline/scripts.base.frameworks.intel.cluster-transparency/worker-2..stdout

@@ -0,0 +1,4 @@
+cluster_new_item: 1.2.3.4 inserted by manager (from peer: manager-1)
+cluster_new_item: 123.123.123.123 inserted by worker-1 (from peer: manager-1)
+cluster_new_item: 4.3.2.1 inserted by worker-2 (from peer: manager-1)
+Doing a lookup

testing/btest/Baseline/scripts.base.frameworks.intel.input-and-match/broproc.intel.log

@@ -0,0 +1,11 @@
+#separator \x09
+#set_separator	,
+#empty_field	(empty)
+#unset_field	-
+#path	intel
+#open	2012-10-03-20-18-05
+#fields	ts	uid	id.orig_h	id.orig_p	id.resp_h	id.resp_p	seen.host	seen.str	seen.str_type	seen.where	sources
+#types	time	string	addr	port	addr	port	addr	string	enum	enum	table[string]
+1349295485.114156	-	-	-	-	-	-	e@mail.com	Intel::EMAIL	SOMEWHERE	source1
+1349295485.114156	-	-	-	-	-	1.2.3.4	-	-	SOMEWHERE	source1
+#close	2012-10-03-20-18-05

testing/btest/Baseline/scripts.base.frameworks.intel.insert-and-matcher/out

@@ -1,3 +0,0 @@
-VALID
-VALID
-VALID

testing/btest/Baseline/scripts.base.frameworks.intel.read-file-dist-cluster/manager-1.intel.log

@@ -0,0 +1,13 @@
+#separator \x09
+#set_separator	,
+#empty_field	(empty)
+#unset_field	-
+#path	intel
+#open	2012-10-10-15-05-23
+#fields	ts	uid	id.orig_h	id.orig_p	id.resp_h	id.resp_p	seen.host	seen.str	seen.str_type	seen.where	sources
+#types	time	string	addr	port	addr	port	addr	string	enum	enum	table[string]
+1349881523.548946	-	-	-	-	-	1.2.3.4	-	-	Intel::IN_A_TEST	source1
+1349881523.548946	-	-	-	-	-	-	e@mail.com	Intel::EMAIL	Intel::IN_A_TEST	source1
+1349881524.567896	-	-	-	-	-	1.2.3.4	-	-	Intel::IN_A_TEST	source1
+1349881524.567896	-	-	-	-	-	-	e@mail.com	Intel::EMAIL	Intel::IN_A_TEST	source1
+#close	2012-10-10-15-05-24

testing/btest/scripts/base/frameworks/intel/cluster-transparency.bro

@@ -0,0 +1,80 @@
+# @TEST-SERIALIZE: comm
+#
+# @TEST-EXEC: btest-bg-run manager-1 BROPATH=$BROPATH:.. CLUSTER_NODE=manager-1 bro %INPUT
+# @TEST-EXEC: btest-bg-run worker-1  BROPATH=$BROPATH:.. CLUSTER_NODE=worker-1 bro %INPUT
+# @TEST-EXEC: btest-bg-run worker-2  BROPATH=$BROPATH:.. CLUSTER_NODE=worker-2 bro %INPUT
+# @TEST-EXEC: btest-bg-wait -k 10
+# @TEST-EXEC: btest-diff manager-1/.stdout
+# @TEST-EXEC: btest-diff manager-1/intel.log
+# @TEST-EXEC: btest-diff worker-1/.stdout
+# @TEST-EXEC: btest-diff worker-2/.stdout
+
+@TEST-START-FILE cluster-layout.bro
+redef Cluster::nodes = {
+	["manager-1"] = [$node_type=Cluster::MANAGER, $ip=127.0.0.1, $p=37757/tcp, $workers=set("worker-1", "worker-2")],
+	["worker-1"]  = [$node_type=Cluster::WORKER,  $ip=127.0.0.1, $p=37760/tcp, $manager="manager-1"],
+	["worker-2"]  = [$node_type=Cluster::WORKER,  $ip=127.0.0.1, $p=37761/tcp, $manager="manager-1"],
+};
+@TEST-END-FILE
+
+@load base/frameworks/control
+
+module Intel;
+
+redef Log::default_rotation_interval=0sec;
+
+event remote_connection_handshake_done(p: event_peer)
+	{
+	# Insert the data once both workers are connected.
+	if ( Cluster::local_node_type() == Cluster::MANAGER && Cluster::worker_count == 2 )
+		{
+		Intel::insert([$host=1.2.3.4,$meta=[$source="manager"]]);
+		}
+	}
+
+global worker2_data = 0;
+global sent_data = F;
+event Intel::cluster_new_item(item: Intel::Item)
+	{
+	if ( ! is_remote_event() )
+		return;
+
+	print fmt("cluster_new_item: %s inserted by %s (from peer: %s)", item$host, item$meta$source, get_event_peer()$descr);
+
+	if ( ! sent_data )
+		{
+		# We wait to insert data here because we can now be sure the 
+		# full cluster is constructed.
+		sent_data = T;
+		if ( Cluster::node == "worker-1" )
+			Intel::insert([$host=123.123.123.123,$meta=[$source="worker-1"]]);
+		if ( Cluster::node == "worker-2" )
+			Intel::insert([$host=4.3.2.1,$meta=[$source="worker-2"]]);
+		}
+
+	# We're forcing worker-2 to do a lookup when it has three intelligence items
+	# which were distributed over the cluster (data inserted locally is resent).
+	if ( Cluster::node == "worker-2" )
+		{
+		++worker2_data;
+		if ( worker2_data == 3 )
+			{
+			# Now that everything is inserted, see if we can match on the data inserted
+			# by worker-1.
+			print "Doing a lookup";
+			Intel::seen([$host=123.123.123.123, $where=Intel::IN_ANYWHERE]);
+			}
+		}
+	}
+
+event Intel::log_intel(rec: Intel::Info)
+	{
+	event Control::shutdown_request();
+	}
+
+event remote_connection_closed(p: event_peer)
+	{
+	# Cascading termination
+	#print fmt("disconnected from: %s", p);
+	terminate_communication();
+	}

testing/btest/scripts/base/frameworks/intel/input-and-match.bro

@@ -0,0 +1,40 @@
+# @TEST-SERIALIZE: comm
+
+# @TEST-EXEC: btest-bg-run broproc bro %INPUT
+# @TEST-EXEC: btest-bg-wait -k 5
+# @TEST-EXEC: btest-diff broproc/intel.log
+
+@TEST-START-FILE intel.dat
+#fields	host	net	str	str_type	meta.source	meta.desc	meta.url
+1.2.3.4	-	-	-	source1	this host is just plain baaad	http://some-data-distributor.com/1234
+1.2.3.4	-	-	-	source1	this host is just plain baaad	http://some-data-distributor.com/1234
+-	-	e@mail.com	Intel::EMAIL	source1	Phishing email source	http://some-data-distributor.com/100000
+@TEST-END-FILE
+
+@load frameworks/communication/listen
+
+redef Intel::read_files += { "../intel.dat" };
+redef enum Intel::Where += { SOMEWHERE };
+
+event do_it()
+	{
+	Intel::seen([$str="e@mail.com",
+	             $str_type=Intel::EMAIL,
+	             $where=SOMEWHERE]);
+
+	Intel::seen([$host=1.2.3.4,
+	             $where=SOMEWHERE]);
+	}
+
+global log_lines = 0;
+event Intel::log_intel(rec: Intel::Info)
+	{
+	++log_lines;
+	if ( log_lines == 2 )
+		terminate();
+	}
+
+event bro_init() &priority=-10
+	{
+	schedule 1sec { do_it() };
+	}

testing/btest/scripts/base/frameworks/intel/insert-and-matcher.bro

@@ -1,34 +0,0 @@
-#
-# @TEST-EXEC: bro %INPUT >out
-# @TEST-EXEC: btest-diff out
-
-event bro_init()
-	{
-	Intel::insert([$ip=1.2.3.4, $tags=set("zeustracker.abuse.ch", "malicious")]);
-	Intel::insert([$str="http://www.google.com/", $subtype="url", $tags=set("infrastructure", "google")]);
-	Intel::insert([$str="Ab439G32F...", $subtype="x509_cert", $tags=set("bad")]);
-	Intel::insert([$str="Ab439G32F...", $tags=set("bad")]);
-	}
-
-event bro_done()
-	{
-	local orig_h = 1.2.3.4;
-	
-	if ( Intel::matcher([$ip=orig_h, $and_tags=set("malicious")]) )
-		print "VALID";
-	
-	if ( Intel::matcher([$ip=orig_h, $and_tags=set("don't match")]) )
-		print "INVALID";
-	
-	if ( Intel::matcher([$ip=orig_h, $pred=function(meta: Intel::MetaData): bool { return T; } ]) )
-		print "VALID";
-		
-	if ( Intel::matcher([$ip=orig_h, $pred=function(meta: Intel::MetaData): bool { return F; } ]) )
-		print "INVALID";
-		
-	if ( Intel::matcher([$str="http://www.google.com/", $subtype="url", $tags=set("google")]) )
-		print "VALID";
-		
-	if ( Intel::matcher([$str="http://www.example.com", $subtype="url"]) )
-		print "INVALID";
-	}

testing/btest/scripts/base/frameworks/intel/read-file-dist-cluster.bro

@@ -0,0 +1,66 @@
+# @TEST-SERIALIZE: comm
+#
+# @TEST-EXEC: btest-bg-run manager-1 BROPATH=$BROPATH:.. CLUSTER_NODE=manager-1 bro %INPUT
+# @TEST-EXEC: sleep 2
+# @TEST-EXEC: btest-bg-run worker-1  BROPATH=$BROPATH:.. CLUSTER_NODE=worker-1 bro %INPUT
+# @TEST-EXEC: btest-bg-run worker-2  BROPATH=$BROPATH:.. CLUSTER_NODE=worker-2 bro %INPUT
+# @TEST-EXEC: btest-bg-wait -k 10
+# @TEST-EXEC: btest-diff manager-1/.stdout
+# @TEST-EXEC: btest-diff manager-1/intel.log
+# @TEST-EXEC: btest-diff worker-1/.stdout
+# @TEST-EXEC: btest-diff worker-2/.stdout
+
+@TEST-START-FILE cluster-layout.bro
+redef Cluster::nodes = {
+	["manager-1"] = [$node_type=Cluster::MANAGER, $ip=127.0.0.1, $p=37757/tcp, $workers=set("worker-1", "worker-2")],
+	["worker-1"]  = [$node_type=Cluster::WORKER,  $ip=127.0.0.1, $p=37760/tcp, $manager="manager-1"],
+	["worker-2"]  = [$node_type=Cluster::WORKER,  $ip=127.0.0.1, $p=37761/tcp, $manager="manager-1"],
+};
+@TEST-END-FILE
+
+@TEST-START-FILE intel.dat
+#fields	host	net	str	str_type	meta.source	meta.desc	meta.url
+1.2.3.4	-	-	-	source1	this host is just plain baaad	http://some-data-distributor.com/1234
+1.2.3.4	-	-	-	source1	this host is just plain baaad	http://some-data-distributor.com/1234
+-	-	e@mail.com	Intel::EMAIL	source1	Phishing email source	http://some-data-distributor.com/100000
+@TEST-END-FILE
+
+@load base/frameworks/control
+redef Log::default_rotation_interval=0sec;
+
+module Intel;
+
+@if ( Cluster::local_node_type() == Cluster::MANAGER )
+redef Intel::read_files += { "../intel.dat" };
+@endif
+
+redef enum Intel::Where += {
+	Intel::IN_A_TEST,
+};
+
+event do_it()
+	{
+	Intel::seen([$host=1.2.3.4, $where=Intel::IN_A_TEST]);
+	Intel::seen([$str="e@mail.com", $str_type=Intel::EMAIL, $where=Intel::IN_A_TEST]);
+	}
+
+event bro_init()
+	{
+	# Delay the workers searching for hits briefly to allow for the data distribution
+	# mechanism to distribute the data to the workers.
+	if ( Cluster::local_node_type() == Cluster::WORKER )
+		schedule 2sec { do_it() };
+	}
+
+global intel_hits=0;
+event Intel::log_intel(rec: Intel::Info)
+	{
+	++intel_hits;
+	# There should be 4 hits since each worker is "seeing" 2 things.
+	if ( intel_hits == 4 )
+		{
+		# We're delaying shutdown for a second here to make sure that no other
+		# matches happen (which would be wrong!).
+		schedule 1sec { Control::shutdown_request() };
+		}
+	}