Mirror/zeek
1
0
Fork 0
mirror of https://github.com/zeek/zeek.git synced 2025-10-02 06:38:20 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 1

Improve documentation of the Intelligence Framework

Browse source 
Added some missing information and rearranged a few sentences so the order
makes more sense.
This commit is contained in:
Daniel Thayer 2015-01-07 00:01:35 -06:00
parent 593e74d4b7
commit b5e9433b04
2 changed files with 49 additions and 26 deletions
Download patch file Download diff file Expand all files Collapse all files

72
doc/frameworks/intel.rst
View file

@ -14,32 +14,35 @@ consume that data, make it available for matching, and provide
infrastructure around improving performance, memory utilization, and infrastructure around improving performance, memory utilization, and
generally making all of this easier. generally making all of this easier.
Data in the Intelligence Framework is the atomic piece of intelligence Data in the Intelligence Framework is an atomic piece of intelligence
such as an IP address or an e-mail address along with a suite of such as an IP address or an e-mail address along with a suite of
metadata about it such as a freeform source field, a freeform metadata about it such as a freeform source field, a freeform
descriptive field and a URL which might lead to more information about descriptive field and a URL which might lead to more information about
the specific item. The metadata in the default scripts has been the specific item. The metadata in the default scripts has been
deliberately kept minimal so that the community can find the deliberately kept minimal so that the community can find the
appropriate fields that need added by writing scripts which extend the appropriate fields that need to be added by writing scripts which extend the
base record using the normal record extension mechanism. base record using the normal record extension mechanism.
Quick Start Quick Start
----------- -----------
Load the package of scripts that sends data into the Intelligence
Framework to be checked by loading this script in local.bro::
@load policy/frameworks/intel/seen
Refer to the "Loading Intelligence" section below to see the format Refer to the "Loading Intelligence" section below to see the format
for Intelligence Framework text files, then load those text files with for Intelligence Framework text files, then load those text files with
this line in local.bro:: this line in local.bro::
redef Intel::read_files += { "/somewhere/yourdata.txt" }; redef Intel::read_files += { "/somewhere/yourdata.txt" };
The data itself only needs to reside on the manager if running in a The text files need to reside only on the manager if running in a
cluster. cluster.
Add the following line to local.bro in order to load the scripts
that send "seen" data into the Intelligence Framework to be checked against
the loaded intelligence data::
@load policy/frameworks/intel/seen
Intelligence data matches will be logged to the intel.log file.
Architecture Architecture
------------ ------------
@ -58,8 +61,10 @@ manager is the only node that needs the intelligence data. The
intelligence framework has distribution mechanisms which will push intelligence framework has distribution mechanisms which will push
data out to all of the nodes that need it. data out to all of the nodes that need it.
Here is an example of the intelligence data format. Note that all Here is an example of the intelligence data format (note that there will be
whitespace field separators are literal tabs and fields containing only a additional fields if you are using CIF intelligence data or if you are
using the policy/frameworks/intel/do_notice script). Note that all fields
must be separated by a single tab character and fields containing only a
hyphen are considered to be null values. :: hyphen are considered to be null values. ::
#fields indicator indicator_type meta.source meta.desc meta.url #fields indicator indicator_type meta.source meta.desc meta.url
@ -69,8 +74,21 @@ hyphen are considered to be null values. ::
For a list of all built-in `indicator_type` values, please refer to the For a list of all built-in `indicator_type` values, please refer to the
documentation of :bro:see:`Intel::Type`. documentation of :bro:see:`Intel::Type`.
To load the data once files are created, use the following example Note that if you are using data from the Collective Intelligence Framework,
code to define files to load with your own file names of course:: then you will need to add the following line to your local.bro in order
to support additional metadata fields used by CIF::
@load policy/integration/collective-intel
There is a simple mechanism to raise a Bro notice (of type Intel::Notice)
for user-specified intelligence matches. To use this feature, add the
following line to local.bro in order to support additional metadata fields
(documented in the :bro:see:`Intel::MetaData` record)::
@load policy/frameworks/intel/do_notice
To load the data once the files are created, use the following example
to specify which files to load (with your own file names of course)::
redef Intel::read_files += { redef Intel::read_files += {
"/somewhere/feed1.txt", "/somewhere/feed1.txt",
@ -85,24 +103,23 @@ Seen Data
When some bit of data is extracted (such as an email address in the When some bit of data is extracted (such as an email address in the
"From" header in a message over SMTP), the Intelligence Framework "From" header in a message over SMTP), the Intelligence Framework
needs to be informed that this data was discovered and it's presence needs to be informed that this data was discovered so that its presence
should be checked within the intelligence data set. This is will be checked within the loaded intelligence data. This is
accomplished through the :bro:see:`Intel::seen` function. accomplished through the :bro:see:`Intel::seen` function, however
typically users won't need to work with this function due to the
scripts included with Bro that will call this function.
Typically users won't need to work with this function due to built in To load all of the scripts included with Bro for sending "seen" data to
hook scripts that Bro ships with that will "see" data and send it into the intelligence framework, just add this line to local.bro::
the intelligence framework. A user may only need to load the entire
package of hook scripts as a module or pick and choose specific @load policy/frameworks/intel/seen
scripts to load. Keep in mind that as more data is sent into the
Alternatively, specific scripts in that directory can be loaded.
Keep in mind that as more data is sent into the
intelligence framework, the CPU load consumed by Bro will increase intelligence framework, the CPU load consumed by Bro will increase
depending on how many times the :bro:see:`Intel::seen` function is depending on how many times the :bro:see:`Intel::seen` function is
being called which is heavily traffic dependent. being called which is heavily traffic dependent.
The full package of hook scripts that Bro ships with for sending this
"seen" data into the intelligence framework can be loading by adding
this line to local.bro::
@load policy/frameworks/intel/seen
Intelligence Matches Intelligence Matches
******************** ********************
@ -111,6 +128,7 @@ Against all hopes, most networks will eventually have a hit on
intelligence data which could indicate a possible compromise or other intelligence data which could indicate a possible compromise or other
unwanted activity. The Intelligence Framework provides an event that unwanted activity. The Intelligence Framework provides an event that
is generated whenever a match is discovered named :bro:see:`Intel::match`. is generated whenever a match is discovered named :bro:see:`Intel::match`.
Due to design restrictions placed upon Due to design restrictions placed upon
the intelligence framework, there is no assurance as to where this the intelligence framework, there is no assurance as to where this
event will be generated. It could be generated on the worker where event will be generated. It could be generated on the worker where
@ -119,3 +137,7 @@ handled, only the data given as event arguments to the event can be
assured since the host where the data was seen may not be where assured since the host where the data was seen may not be where
``Intel::match`` is handled. ``Intel::match`` is handled.
Intelligence matches are logged to the intel.log file. For a description of
each field in that file, see the documentation for the :bro:see:`Intel::Info`
record.

3
scripts/base/frameworks/intel/main.bro
View file

@ -67,6 +67,7 @@ export {
IN_ANYWHERE, IN_ANYWHERE,
}; };
## Information about a piece of "seen" data.
type Seen: record { type Seen: record {
## The string if the data is about a string. ## The string if the data is about a string.
indicator: string &log &optional; indicator: string &log &optional;
@ -124,7 +125,7 @@ export {
sources: set[string] &log &default=string_set(); sources: set[string] &log &default=string_set();
}; };
## Intelligence data manipulation functions. ## Intelligence data manipulation function.
global insert: function(item: Item); global insert: function(item: Item);
## Function to declare discovery of a piece of data in order to check ## Function to declare discovery of a piece of data in order to check