Mirror/zeek
1
0
Fork 0
mirror of https://github.com/zeek/zeek.git synced 2025-10-05 16:18:19 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 1

Fix various documentation, mostly related to file analysis.

Browse source 
- Fix examples/references in the file analysis how-to/usage doc.

- Add Broxygen-generated docs for file analyzer plugins.

- Break FTP::Info type declaration out in to its own file to get
  rid of some circular dependencies (between s/b/p/ftp/main and
  s/b/p/ftp/utils).
This commit is contained in:
Jon Siwek 2013-07-29 15:41:34 -05:00
parent 1238e5bcf2
commit d84f6e012c
22 changed files with 227 additions and 138 deletions
Download patch file Download diff file Expand all files Collapse all files

27
doc/file-analysis.rst
View file

@ -82,9 +82,9 @@ attached, they start receiving the contents of the file as Bro extracts
it from an ongoing network connection. What they do with the file
contents is up to the particular file analyzer implementation, but
they'll typically either report further information about the file via
events (e.g. :bro:see:`FileAnalysis::ANALYZER_MD5` will report the
events (e.g. :bro:see:`Files::ANALYZER_MD5` will report the
file's MD5 checksum via :bro:see:`file_hash` once calculated) or they'll
have some side effect (e.g. :bro:see:`FileAnalysis::ANALYZER_EXTRACT`
have some side effect (e.g. :bro:see:`Files::ANALYZER_EXTRACT`
will write the contents of the file out to the local file system).
In the future there may be file analyzers that automatically attach to
@ -98,7 +98,7 @@ explicit attachment decision:
{
print "new file", f$id;
if ( f?$mime_type && f$mime_type == "text/plain" )
FileAnalysis::add_analyzer(f, [$tag=FileAnalysis::ANALYZER_MD5]);
Files::add_analyzer(f, Files::ANALYZER_MD5);
}
event file_hash(f: fa_file, kind: string, hash: string)
@ -113,26 +113,27 @@ output::
file_hash, Cx92a0ym5R8, md5, 397168fd09991a0e712254df7bc639ac
Some file analyzers might have tunable parameters that need to be
specified in the call to :bro:see:`FileAnalysis::add_analyzer`:
specified in the call to :bro:see:`Files::add_analyzer`:
.. code:: bro
event file_new(f: fa_file)
{
FileAnalysis::add_analyzer(f, [$tag=FileAnalysis::ANALYZER_EXTRACT,
$extract_filename="./myfile"]);
Files::add_analyzer(f, Files::ANALYZER_EXTRACT,
[$extract_filename="myfile"]);
}
In this case, the file extraction analyzer doesn't generate any further
events, but does have the side effect of writing out the file contents
to the local file system at the specified location of ``./myfile``. Of
course, for a network with more than a single file being transferred,
it's probably preferable to specify a different extraction path for each
file, unlike this example.
events, but does have the effect of writing out the file contents to the
local file system at the location resulting from the concatenation of
the path specified by :bro:see:`FileExtract::prefix` and the string,
``myfile``. Of course, for a network with more than a single file being
transferred, it's probably preferable to specify a different extraction
path for each file, unlike this example.
Regardless of which file analyzers end up acting on a file, general
information about the file (e.g. size, time of last data transferred,
MIME type, etc.) are logged in ``file_analysis.log``.
MIME type, etc.) are logged in ``files.log``.
Input Framework Integration
===========================
@ -150,7 +151,7 @@ a network interface it's monitoring. It only requires a call to
event file_new(f: fa_file)
{
print "new file", f$id;
FileAnalysis::add_analyzer(f, [$tag=FileAnalysis::ANALYZER_MD5]);
Files::add_analyzer(f, Files::ANALYZER_MD5);
}
event file_state_remove(f: fa_file)

1
doc/index.rst
View file

@ -47,6 +47,7 @@ Script Reference
scripts/index
scripts/builtins
scripts/proto-analyzers
scripts/file-analyzers
Other Bro Components
--------------------

18
doc/scripts/CMakeLists.txt
View file

@ -124,8 +124,10 @@ endmacro(REST_TARGET)
# Schedule Bro scripts for which to generate documentation.
include(DocSourcesList.cmake)
# This reST target is independent of a particular Bro script...
add_custom_command(OUTPUT proto-analyzers.rst
# Macro for generating reST docs that are independent of any particular Bro
# script.
macro(INDEPENDENT_REST_TARGET reST_file)
add_custom_command(OUTPUT ${reST_file}
# delete any leftover state from previous bro runs
COMMAND "${CMAKE_COMMAND}"
ARGS -E remove_directory .state
@ -137,15 +139,19 @@ add_custom_command(OUTPUT proto-analyzers.rst
COMMAND "${CMAKE_COMMAND}"
ARGS -E make_directory ${dstDir}
COMMAND "${CMAKE_COMMAND}"
ARGS -E copy proto-analyzers.rst ${dstDir}
ARGS -E copy ${reST_file} ${dstDir}
# clean up the build directory
COMMAND rm
ARGS -rf .state *.log *.rst
DEPENDS bro
WORKING_DIRECTORY ${CMAKE_BINARY_DIR}
COMMENT "[Bro] Generating reST docs for proto-analyzers.rst"
)
list(APPEND ALL_REST_OUTPUTS proto-analyzers.rst)
COMMENT "[Bro] Generating reST docs for ${reST_file}"
)
list(APPEND ALL_REST_OUTPUTS ${reST_file})
endmacro(INDEPENDENT_REST_TARGET)
independent_rest_target(proto-analyzers.rst)
independent_rest_target(file-analyzers.rst)
# create temporary list of all docs to include in the master policy/index file
file(WRITE ${MASTER_POLICY_INDEX} "${MASTER_POLICY_INDEX_TEXT}")

1
doc/scripts/DocSourcesList.cmake
View file

@ -140,6 +140,7 @@ rest_target(${psd} base/protocols/dns/consts.bro)
rest_target(${psd} base/protocols/dns/main.bro)
rest_target(${psd} base/protocols/ftp/files.bro)
rest_target(${psd} base/protocols/ftp/gridftp.bro)
rest_target(${psd} base/protocols/ftp/info.bro)
rest_target(${psd} base/protocols/ftp/main.bro)
rest_target(${psd} base/protocols/ftp/utils-commands.bro)
rest_target(${psd} base/protocols/ftp/utils.bro)

2
scripts/base/frameworks/files/main.bro
View file

@ -204,7 +204,7 @@ export {
##
## tag: Tag for the protocol analyzer having a callback being registered.
##
## reg: A :bro:see:`ProtoRegistration` record.
## reg: A :bro:see:`Files::ProtoRegistration` record.
##
## Returns: true if the protocol being registered was not previously registered.
global register_protocol: function(tag: Analyzer::Tag, reg: ProtoRegistration): bool;

17
scripts/base/init-bare.bro
View file

@ -531,22 +531,19 @@ type record_field_table: table[string] of record_field;
# dependent on the names remaining as they are now.
## Set of BPF capture filters to use for capturing, indexed by a user-definable
## ID (which must be unique). If Bro is *not* configured to examine
## :bro:id:`PacketFilter::all_packets`, all packets matching at least
## one of the filters in this table (and all in :bro:id:`restrict_filters`)
## will be analyzed.
## ID (which must be unique). If Bro is *not* configured with
## :bro:id:`PacketFilter::enable_auto_protocol_capture_filters`,
## all packets matching at least one of the filters in this table (and all in
## :bro:id:`restrict_filters`) will be analyzed.
##
## .. bro:see:: PacketFilter PacketFilter::all_packets
## .. bro:see:: PacketFilter PacketFilter::enable_auto_protocol_capture_filters
## PacketFilter::unrestricted_filter restrict_filters
global capture_filters: table[string] of string &redef;
## Set of BPF filters to restrict capturing, indexed by a user-definable ID (which
## must be unique). If Bro is *not* configured to examine
## :bro:id:`PacketFilter::all_packets`, only packets matching *all* of the
## filters in this table (and any in :bro:id:`capture_filters`) will be
## analyzed.
## must be unique).
##
## .. bro:see:: PacketFilter PacketFilter::all_packets
## .. bro:see:: PacketFilter PacketFilter::enable_auto_protocol_capture_filters
## PacketFilter::unrestricted_filter capture_filters
global restrict_filters: table[string] of string &redef;

1
scripts/base/protocols/ftp/__load__.bro
View file

@ -1,4 +1,5 @@
@load ./utils-commands
@load ./info
@load ./main
@load ./utils
@load ./files

1
scripts/base/protocols/ftp/files.bro
View file

@ -1,3 +1,4 @@
@load ./info
@load ./main
@load ./utils
@load base/utils/conn-ids

1
scripts/base/protocols/ftp/gridftp.bro
View file

@ -19,6 +19,7 @@
##! sizes are not logged, but at the benefit of saving CPU cycles that
##! otherwise go to analyzing the large (and likely benign) connections.
@load ./info
@load ./main
@load base/protocols/conn
@load base/protocols/ssl

72
scripts/base/protocols/ftp/info.bro Normal file
View file

@ -0,0 +1,72 @@
##! Defines data structures for tracking and logging FTP sessions.
module FTP;
@load ./utils-commands
export {
## This setting changes if passwords used in FTP sessions are
## captured or not.
const default_capture_password = F &redef;
## The expected endpoints of an FTP data channel.
type ExpectedDataChannel: record {
## Whether PASV mode is toggled for control channel.
passive: bool &log;
## The host that will be initiating the data connection.
orig_h: addr &log;
## The host that will be accepting the data connection.
resp_h: addr &log;
## The port at which the acceptor is listening for the data connection.
resp_p: port &log;
};
type Info: record {
## Time when the command was sent.
ts: time &log;
## Unique ID for the connection.
uid: string &log;
## The connection's 4-tuple of endpoint addresses/ports.
id: conn_id &log;
## User name for the current FTP session.
user: string &log &default="<unknown>";
## Password for the current FTP session if captured.
password: string &log &optional;
## Command given by the client.
command: string &log &optional;
## Argument for the command if one is given.
arg: string &log &optional;
## Libmagic "sniffed" file type if the command indicates a file transfer.
mime_type: string &log &optional;
## Size of the file if the command indicates a file transfer.
file_size: count &log &optional;
## Reply code from the server in response to the command.
reply_code: count &log &optional;
## Reply message from the server in response to the command.
reply_msg: string &log &optional;
## Expected FTP data channel.
data_channel: ExpectedDataChannel &log &optional;
## Current working directory that this session is in. By making
## the default value '.', we can indicate that unless something
## more concrete is discovered that the existing but unknown
## directory is ok to use.
cwd: string &default=".";
## Command that is currently waiting for a response.
cmdarg: CmdArg &optional;
## Queue for commands that have been sent but not yet responded to
## are tracked here.
pending_commands: PendingCmds;
## Indicates if the session is in active or passive mode.
passive: bool &default=F;
## Determines if the password will be captured for this request.
capture_password: bool &default=default_capture_password;
};
}

67
scripts/base/protocols/ftp/main.bro
View file

@ -3,6 +3,8 @@
##! will take on the full path that the client is at along with the requested
##! file name.
@load ./info
@load ./utils
@load ./utils-commands
@load base/utils/paths
@load base/utils/numbers
@ -20,72 +22,9 @@ export {
"EPSV"
} &redef;
## This setting changes if passwords used in FTP sessions are captured or not.
const default_capture_password = F &redef;
## User IDs that can be considered "anonymous".
const guest_ids = { "anonymous", "ftp", "ftpuser", "guest" } &redef;
## The expected endpoints of an FTP data channel.
type ExpectedDataChannel: record {
## Whether PASV mode is toggled for control channel.
passive: bool &log;
## The host that will be initiating the data connection.
orig_h: addr &log;
## The host that will be accepting the data connection.
resp_h: addr &log;
## The port at which the acceptor is listening for the data connection.
resp_p: port &log;
};
type Info: record {
## Time when the command was sent.
ts: time &log;
## Unique ID for the connection.
uid: string &log;
## The connection's 4-tuple of endpoint addresses/ports.
id: conn_id &log;
## User name for the current FTP session.
user: string &log &default="<unknown>";
## Password for the current FTP session if captured.
password: string &log &optional;
## Command given by the client.
command: string &log &optional;
## Argument for the command if one is given.
arg: string &log &optional;
## Libmagic "sniffed" file type if the command indicates a file transfer.
mime_type: string &log &optional;
## Size of the file if the command indicates a file transfer.
file_size: count &log &optional;
## Reply code from the server in response to the command.
reply_code: count &log &optional;
## Reply message from the server in response to the command.
reply_msg: string &log &optional;
## Expected FTP data channel.
data_channel: ExpectedDataChannel &log &optional;
## Current working directory that this session is in. By making
## the default value '.', we can indicate that unless something
## more concrete is discovered that the existing but unknown
## directory is ok to use.
cwd: string &default=".";
## Command that is currently waiting for a response.
cmdarg: CmdArg &optional;
## Queue for commands that have been sent but not yet responded to
## are tracked here.
pending_commands: PendingCmds;
## Indicates if the session is in active or passive mode.
passive: bool &default=F;
## Determines if the password will be captured for this request.
capture_password: bool &default=default_capture_password;
};
## This record is to hold a parsed FTP reply code. For example, for the
## 201 status code, the digits would be parsed as: x->2, y->0, z=>1.
type ReplyCode: record {
@ -102,8 +41,6 @@ export {
global log_ftp: event(rec: Info);
}
@load ./utils
# Add the state tracking information variable to the connection record
redef record connection += {
ftp: Info &optional;

3
scripts/base/protocols/ftp/utils.bro
View file

@ -1,7 +1,8 @@
##! Utilities specific for FTP processing.
@load ./main
@load ./info
@load base/utils/addrs
@load base/utils/paths
module FTP;

4
scripts/policy/frameworks/packet-filter/shunt.bro
View file

@ -34,8 +34,8 @@ export {
global current_shunted_host_pairs: function(): set[conn_id];
redef enum Notice::Type += {
## Indicative that :bro:id:`max_bpf_shunts` connections are already
## being shunted with BPF filters and no more are allowed.
## Indicative that :bro:id:`PacketFilter::max_bpf_shunts` connections
## are already being shunted with BPF filters and no more are allowed.
No_More_Conn_Shunts_Available,
## Limitations in BPF make shunting some connections with BPF impossible.

12
scripts/policy/misc/load-balancing.bro
View file

@ -12,12 +12,12 @@ export {
## Apply BPF filters to each worker in a way that causes them to
## automatically flow balance traffic between them.
AUTO_BPF,
## Load balance traffic across the workers by making each one apply
## a restrict filter to only listen to a single MAC address. This
## is a somewhat common deployment option for sites doing network
## based load balancing with MAC address rewriting and passing the
## traffic to a single interface. Multiple MAC addresses will show
## up on the same interface and need filtered to a single address.
# Load balance traffic across the workers by making each one apply
# a restrict filter to only listen to a single MAC address. This
# is a somewhat common deployment option for sites doing network
# based load balancing with MAC address rewriting and passing the
# traffic to a single interface. Multiple MAC addresses will show
# up on the same interface and need filtered to a single address.
#MAC_ADDR_BPF,
};

12
scripts/policy/tuning/defaults/packet-fragments.bro
View file

@ -1,10 +1,10 @@
## Capture TCP fragments, but not UDP (or ICMP), since those are a lot more
## common due to high-volume, fragmenting protocols such as NFS :-(.
# Capture TCP fragments, but not UDP (or ICMP), since those are a lot more
# common due to high-volume, fragmenting protocols such as NFS :-(.
## This normally isn't used because of the default open packet filter
## but we set it anyway in case the user is using a packet filter.
## Note: This was removed because the default model now is to have a wide
## open packet filter.
# This normally isn't used because of the default open packet filter
# but we set it anyway in case the user is using a packet filter.
# Note: This was removed because the default model now is to have a wide
# open packet filter.
#redef capture_filters += { ["frag"] = "(ip[6:2] & 0x3fff != 0) and tcp" };
## Shorten the fragment timeout from never expiring to expiring fragments after

65
src/BroDoc.cc
View file

@ -11,6 +11,7 @@
#include "plugin/Manager.h"
#include "analyzer/Manager.h"
#include "analyzer/Component.h"
#include "file_analysis/Manager.h"
BroDoc::BroDoc(const std::string& rel, const std::string& abs)
{
@ -479,6 +480,17 @@ static void WriteAnalyzerComponent(FILE* f, const analyzer::Component* c)
fprintf(f, ":bro:enum:`Analyzer::%s`\n\n", tag.c_str());
}
static void WriteAnalyzerComponent(FILE* f, const file_analysis::Component* c)
{
EnumType* atag = file_mgr->GetTagEnumType();
string tag = fmt("ANALYZER_%s", c->CanonicalName());
if ( atag->Lookup("Files", tag.c_str()) < 0 )
reporter->InternalError("missing analyzer tag for %s", tag.c_str());
fprintf(f, ":bro:enum:`Files::%s`\n\n", tag.c_str());
}
static void WritePluginComponents(FILE* f, const plugin::Plugin* p)
{
plugin::Plugin::component_list components = p->Components();
@ -494,6 +506,10 @@ static void WritePluginComponents(FILE* f, const plugin::Plugin* p)
WriteAnalyzerComponent(f,
dynamic_cast<const analyzer::Component*>(*it));
break;
case plugin::component::FILE_ANALYZER:
WriteAnalyzerComponent(f,
dynamic_cast<const file_analysis::Component*>(*it));
break;
case plugin::component::READER:
reporter->InternalError("docs for READER component unimplemented");
case plugin::component::WRITER:
@ -537,12 +553,13 @@ static void WritePluginBifItems(FILE* f, const plugin::Plugin* p,
}
}
static void WriteAnalyzerTagDefn(FILE* f, EnumType* e)
static void WriteAnalyzerTagDefn(FILE* f, EnumType* e, const string& module)
{
string tag_id= module + "::Tag";
e = new CommentedEnumType(e);
e->SetTypeID(copy_string("Analyzer::Tag"));
e->SetTypeID(copy_string(tag_id.c_str()));
ID* dummy_id = new ID(copy_string("Analyzer::Tag"), SCOPE_GLOBAL, true);
ID* dummy_id = new ID(copy_string(tag_id.c_str()), SCOPE_GLOBAL, true);
dummy_id->SetType(e);
dummy_id->MakeType();
@ -554,13 +571,17 @@ static void WriteAnalyzerTagDefn(FILE* f, EnumType* e)
bdo.WriteReST(f);
}
static bool IsAnalyzerPlugin(const plugin::Plugin* p)
static bool ComponentsMatch(const plugin::Plugin* p, plugin::component::Type t,
bool match_empty = false)
{
plugin::Plugin::component_list components = p->Components();
plugin::Plugin::component_list::const_iterator it;
if ( components.empty() )
return match_empty;
for ( it = components.begin(); it != components.end(); ++it )
if ( (*it)->Type() != plugin::component::ANALYZER )
if ( (*it)->Type() != t )
return false;
return true;
@ -573,14 +594,44 @@ void CreateProtoAnalyzerDoc(const char* filename)
fprintf(f, "Protocol Analyzer Reference\n");
fprintf(f, "===========================\n\n");
WriteAnalyzerTagDefn(f, analyzer_mgr->GetTagEnumType());
WriteAnalyzerTagDefn(f, analyzer_mgr->GetTagEnumType(), "Analyzer");
plugin::Manager::plugin_list plugins = plugin_mgr->Plugins();
plugin::Manager::plugin_list::const_iterator it;
for ( it = plugins.begin(); it != plugins.end(); ++it )
{
if ( ! IsAnalyzerPlugin(*it) )
if ( ! ComponentsMatch(*it, plugin::component::ANALYZER, true) )
continue;
WritePluginSectionHeading(f, *it);
WritePluginComponents(f, *it);
WritePluginBifItems(f, *it, plugin::BifItem::CONSTANT,
"Options/Constants");
WritePluginBifItems(f, *it, plugin::BifItem::GLOBAL, "Globals");
WritePluginBifItems(f, *it, plugin::BifItem::TYPE, "Types");
WritePluginBifItems(f, *it, plugin::BifItem::EVENT, "Events");
WritePluginBifItems(f, *it, plugin::BifItem::FUNCTION, "Functions");
}
fclose(f);
}
void CreateFileAnalyzerDoc(const char* filename)
{
FILE* f = fopen(filename, "w");
fprintf(f, "File Analyzer Reference\n");
fprintf(f, "===========================\n\n");
WriteAnalyzerTagDefn(f, file_mgr->GetTagEnumType(), "Files");
plugin::Manager::plugin_list plugins = plugin_mgr->Plugins();
plugin::Manager::plugin_list::const_iterator it;
for ( it = plugins.begin(); it != plugins.end(); ++it )
{
if ( ! ComponentsMatch(*it, plugin::component::FILE_ANALYZER) )
continue;
WritePluginSectionHeading(f, *it);

6
src/BroDoc.h
View file

@ -413,4 +413,10 @@ private:
*/
void CreateProtoAnalyzerDoc(const char* filename);
/**
* Writes out plugin index documentation for all file analyzer plugins.
* @param filename the name of the file to write.
*/
void CreateFileAnalyzerDoc(const char* filename);
#endif

5
src/file_analysis/Manager.cc
View file

@ -394,3 +394,8 @@ const char* Manager::GetAnalyzerName(int tag) const
return it->second->CanonicalName();
}
EnumType* Manager::GetTagEnumType()
{
return tag_enum_type;
}

6
src/file_analysis/Manager.h
View file

@ -214,6 +214,12 @@ public:
*/
const char* GetAnalyzerName(int tag) const;
/**
* Returns the enum type that corresponds to the script-level type
* \c Files::Tag.
*/
EnumType* GetTagEnumType();
protected:
friend class FileTimer;

1
src/main.cc
View file

@ -872,6 +872,7 @@ int main(int argc, char** argv)
if ( generate_documentation )
{
CreateProtoAnalyzerDoc("proto-analyzers.rst");
CreateFileAnalyzerDoc("file-analyzers.rst");
std::list<BroDoc*>::iterator it;

5
testing/btest/Baseline/coverage.default-load-baseline/canonified_loaded_scripts.log
View file

@ -3,7 +3,7 @@
#empty_field (empty)
#unset_field -
#path loaded_scripts
#open 2013-07-23-05-48-10
#open 2013-07-29-20-08-38
#fields name
#types string
scripts/base/init-bare.bro
@ -156,6 +156,7 @@ scripts/base/init-default.bro
scripts/base/protocols/dns/main.bro
scripts/base/protocols/ftp/__load__.bro
scripts/base/protocols/ftp/utils-commands.bro
scripts/base/protocols/ftp/info.bro
scripts/base/protocols/ftp/main.bro
scripts/base/protocols/ftp/utils.bro
scripts/base/protocols/ftp/files.bro
@ -196,4 +197,4 @@ scripts/base/init-default.bro
scripts/base/files/extract/main.bro
scripts/base/misc/find-checksum-offloading.bro
scripts/policy/misc/loaded-scripts.bro
#close 2013-07-23-05-48-10
#close 2013-07-29-20-08-38