From b5e9433b043bd354024b9945d00de22f50c26cad Mon Sep 17 00:00:00 2001 From: Daniel Thayer Date: Wed, 7 Jan 2015 00:01:35 -0600 Subject: [PATCH] Improve documentation of the Intelligence Framework Added some missing information and rearranged a few sentences so the order makes more sense. --- doc/frameworks/intel.rst | 72 +++++++++++++++++--------- scripts/base/frameworks/intel/main.bro | 3 +- 2 files changed, 49 insertions(+), 26 deletions(-) diff --git a/doc/frameworks/intel.rst b/doc/frameworks/intel.rst index f90092aac3..965fba4c14 100644 --- a/doc/frameworks/intel.rst +++ b/doc/frameworks/intel.rst @@ -14,32 +14,35 @@ consume that data, make it available for matching, and provide infrastructure around improving performance, memory utilization, and generally making all of this easier. -Data in the Intelligence Framework is the atomic piece of intelligence +Data in the Intelligence Framework is an atomic piece of intelligence such as an IP address or an e-mail address along with a suite of metadata about it such as a freeform source field, a freeform descriptive field and a URL which might lead to more information about the specific item. The metadata in the default scripts has been deliberately kept minimal so that the community can find the -appropriate fields that need added by writing scripts which extend the +appropriate fields that need to be added by writing scripts which extend the base record using the normal record extension mechanism. Quick Start ----------- -Load the package of scripts that sends data into the Intelligence -Framework to be checked by loading this script in local.bro:: - - @load policy/frameworks/intel/seen - Refer to the "Loading Intelligence" section below to see the format for Intelligence Framework text files, then load those text files with this line in local.bro:: redef Intel::read_files += { "/somewhere/yourdata.txt" }; -The data itself only needs to reside on the manager if running in a +The text files need to reside only on the manager if running in a cluster. +Add the following line to local.bro in order to load the scripts +that send "seen" data into the Intelligence Framework to be checked against +the loaded intelligence data:: + + @load policy/frameworks/intel/seen + +Intelligence data matches will be logged to the intel.log file. + Architecture ------------ @@ -58,8 +61,10 @@ manager is the only node that needs the intelligence data. The intelligence framework has distribution mechanisms which will push data out to all of the nodes that need it. -Here is an example of the intelligence data format. Note that all -whitespace field separators are literal tabs and fields containing only a +Here is an example of the intelligence data format (note that there will be +additional fields if you are using CIF intelligence data or if you are +using the policy/frameworks/intel/do_notice script). Note that all fields +must be separated by a single tab character and fields containing only a hyphen are considered to be null values. :: #fields indicator indicator_type meta.source meta.desc meta.url @@ -69,8 +74,21 @@ hyphen are considered to be null values. :: For a list of all built-in `indicator_type` values, please refer to the documentation of :bro:see:`Intel::Type`. -To load the data once files are created, use the following example -code to define files to load with your own file names of course:: +Note that if you are using data from the Collective Intelligence Framework, +then you will need to add the following line to your local.bro in order +to support additional metadata fields used by CIF:: + + @load policy/integration/collective-intel + +There is a simple mechanism to raise a Bro notice (of type Intel::Notice) +for user-specified intelligence matches. To use this feature, add the +following line to local.bro in order to support additional metadata fields +(documented in the :bro:see:`Intel::MetaData` record):: + + @load policy/frameworks/intel/do_notice + +To load the data once the files are created, use the following example +to specify which files to load (with your own file names of course):: redef Intel::read_files += { "/somewhere/feed1.txt", @@ -85,24 +103,23 @@ Seen Data When some bit of data is extracted (such as an email address in the "From" header in a message over SMTP), the Intelligence Framework -needs to be informed that this data was discovered and it's presence -should be checked within the intelligence data set. This is -accomplished through the :bro:see:`Intel::seen` function. +needs to be informed that this data was discovered so that its presence +will be checked within the loaded intelligence data. This is +accomplished through the :bro:see:`Intel::seen` function, however +typically users won't need to work with this function due to the +scripts included with Bro that will call this function. -Typically users won't need to work with this function due to built in -hook scripts that Bro ships with that will "see" data and send it into -the intelligence framework. A user may only need to load the entire -package of hook scripts as a module or pick and choose specific -scripts to load. Keep in mind that as more data is sent into the +To load all of the scripts included with Bro for sending "seen" data to +the intelligence framework, just add this line to local.bro:: + + @load policy/frameworks/intel/seen + +Alternatively, specific scripts in that directory can be loaded. +Keep in mind that as more data is sent into the intelligence framework, the CPU load consumed by Bro will increase depending on how many times the :bro:see:`Intel::seen` function is being called which is heavily traffic dependent. -The full package of hook scripts that Bro ships with for sending this -"seen" data into the intelligence framework can be loading by adding -this line to local.bro:: - - @load policy/frameworks/intel/seen Intelligence Matches ******************** @@ -111,6 +128,7 @@ Against all hopes, most networks will eventually have a hit on intelligence data which could indicate a possible compromise or other unwanted activity. The Intelligence Framework provides an event that is generated whenever a match is discovered named :bro:see:`Intel::match`. + Due to design restrictions placed upon the intelligence framework, there is no assurance as to where this event will be generated. It could be generated on the worker where @@ -119,3 +137,7 @@ handled, only the data given as event arguments to the event can be assured since the host where the data was seen may not be where ``Intel::match`` is handled. +Intelligence matches are logged to the intel.log file. For a description of +each field in that file, see the documentation for the :bro:see:`Intel::Info` +record. + diff --git a/scripts/base/frameworks/intel/main.bro b/scripts/base/frameworks/intel/main.bro index b5f305a7fc..4866766df4 100644 --- a/scripts/base/frameworks/intel/main.bro +++ b/scripts/base/frameworks/intel/main.bro @@ -67,6 +67,7 @@ export { IN_ANYWHERE, }; + ## Information about a piece of "seen" data. type Seen: record { ## The string if the data is about a string. indicator: string &log &optional; @@ -124,7 +125,7 @@ export { sources: set[string] &log &default=string_set(); }; - ## Intelligence data manipulation functions. + ## Intelligence data manipulation function. global insert: function(item: Item); ## Function to declare discovery of a piece of data in order to check