Merge remote-tracking branch 'origin/master' into topic/robin/plugins

Conflicts:
	aux/bro-aux
	aux/broctl
	src/DPM.cc
This commit is contained in:
Robin Sommer 2013-05-30 17:43:50 -07:00
commit c049c758c3
25 changed files with 198 additions and 103 deletions

38
CHANGES
View file

@ -1,4 +1,42 @@
2.1-664 | 2013-05-28 21:37:46 -0700
* Dangling pointer fix. Addresses #1004. (Jon Siwek)
2.1-659 | 2013-05-24 17:24:18 -0700
* Fix broken/missing documentation. (Jon Siwek)
* Fixing test that would fail without ES/curl support. (Robin
Sommer)
2.1-656 | 2013-05-17 15:58:07 -0700
* Fix mutex lock problem for writers. (Bernhard Amann)
2.1-654 | 2013-05-17 13:49:52 -0700
* Tweaks to sqlite3 configuration to address threading issues.
(Bernhard Amann)
2.1-651 | 2013-05-17 13:37:16 -0700
* Fix uninitialized DPM member. (Jon Siwek)
* Fix issue with transaction ID reuse in a single DNS connection. (Seth Hall)
* New function added to the queue.bro script to support peeking at
the new gettable item in the queue without removing it. (Seth Hall)
2.1-647 | 2013-05-17 07:47:14 -0700
* Fixing Broxygen generation to have BROMAGIC set. (Robin Sommer)
* Fix for 'fchmod undeclared here' on FreeBSD. (Robin Sommer)
* CMake policy fix to avoid errors with older versions. (Robin
Sommer)
2.1-641 | 2013-05-15 18:15:09 -0700
* Test update. (Robin Sommer)

View file

@ -1 +1 @@
2.1-641
2.1-664

View file

@ -89,8 +89,7 @@ Note the fields that are set for the filter:
are generated by taking the stream's ID and munging it slightly.
:bro:enum:`Conn::LOG` is converted into ``conn``,
:bro:enum:`PacketFilter::LOG` is converted into
``packet_filter``, and :bro:enum:`Notice::POLICY_LOG` is
converted into ``notice_policy``.
``packet_filter``.
``include``
A set limiting the fields to the ones given. The names

View file

@ -86,21 +86,21 @@ directly make modifications to the :bro:see:`Notice::Info` record
given as the argument to the hook.
Here's a simple example which tells Bro to send an email for all notices of
type :bro:see:`SSH::Login` if the server is 10.0.0.1:
type :bro:see:`SSH::Password_Guessing` if the server is 10.0.0.1:
.. code:: bro
hook Notice::policy(n: Notice::Info)
{
if ( n$note == SSH::Login && n$id$resp_h == 10.0.0.1 )
if ( n$note == SSH::Password_Guessing && n$id$resp_h == 10.0.0.1 )
add n$actions[Notice::ACTION_EMAIL];
}
.. note::
Keep in mind that the semantics of the SSH::Login notice are
such that it is only raised when Bro heuristically detects a successful
login. No apparently failed logins will raise this notice.
Keep in mind that the semantics of the SSH::Password_Guessing notice are
such that it is only raised when Bro heuristically detects a failed
login.
Hooks can also have priorities applied to order their execution like events
with a default priority of 0. Greater values are executed first. Setting
@ -110,7 +110,7 @@ a hook body to run before default hook bodies might look like this:
hook Notice::policy(n: Notice::Info) &priority=5
{
if ( n$note == SSH::Login && n$id$resp_h == 10.0.0.1 )
if ( n$note == SSH::Password_Guessing && n$id$resp_h == 10.0.0.1 )
add n$actions[Notice::ACTION_EMAIL];
}
@ -173,16 +173,16 @@ Raising Notices
A script should raise a notice for any occurrence that a user may want
to be notified about or take action on. For example, whenever the base
SSH analysis scripts sees an SSH session where it is heuristically
guessed to be a successful login, it raises a Notice of the type
:bro:see:`SSH::Login`. The code in the base SSH analysis script looks
like this:
SSH analysis scripts sees enough failed logins to a given host, it
raises a notice of the type :bro:see:`SSH::Password_Guessing`. The code
in the base SSH analysis script which raises the notice looks like this:
.. code:: bro
NOTICE([$note=SSH::Login,
$msg="Heuristically detected successful SSH login.",
$conn=c]);
NOTICE([$note=Password_Guessing,
$msg=fmt("%s appears to be guessing SSH passwords (seen in %d connections).", key$host, r$num),
$src=key$host,
$identifier=cat(key$host)]);
:bro:see:`NOTICE` is a normal function in the global namespace which
wraps a function within the ``Notice`` namespace. It takes a single

View file

@ -107,7 +107,7 @@ macro(REST_TARGET srcDir broInput)
COMMAND "${CMAKE_COMMAND}"
ARGS -E remove_directory .state
# generate the reST documentation using bro
COMMAND BROPATH=${BROPATH}:${srcDir} ${CMAKE_BINARY_DIR}/src/bro
COMMAND BROPATH=${BROPATH}:${srcDir} BROMAGIC=${CMAKE_SOURCE_DIR}/magic ${CMAKE_BINARY_DIR}/src/bro
ARGS -b -Z ${broInput} || (rm -rf .state *.log *.rst && exit 1)
# move generated doc into a new directory tree that
# defines the final structure of documents

View file

@ -402,6 +402,31 @@ The Bro scripting language supports the following built-in types.
if ( r?$s )
...
.. bro:type:: opaque
A data type whose actual representation/implementation is
intentionally hidden, but whose values may be passed to certain
functions that can actually access the internal/hidden resources.
Opaque types are differentiated from each other by qualifying them
like ``opaque of md5`` or ``opaque of sha1``. Any valid identifier
can be used as the type qualifier.
An example use of this type is the set of built-in functions which
perform hashing:
.. code:: bro
local handle: opaque of md5 = md5_hash_init();
md5_hash_update(handle, "test");
md5_hash_update(handle, "testing");
print md5_hash_finish(handle);
Here the opaque type is used to provide a handle to a particular
resource which is calculating an MD5 checksum incrementally over
time, but the details of that resource aren't relevant, it's only
necessary to have a handle as a way of identifying it and
distinguishing it from other such resources.
.. bro:type:: file
Bro supports writing to files, but not reading from them. For

View file

@ -195,7 +195,7 @@ export {
##
## Returns: True if a new stream was successfully removed.
##
## .. bro:see:: Log:create_stream
## .. bro:see:: Log::create_stream
global remove_stream: function(id: ID) : bool;
## Enables a previously disabled logging stream. Disabled streams

View file

@ -431,9 +431,6 @@ hook Notice::notice(n: Notice::Info) &priority=-5
}
}
## This determines if a notice is being suppressed. It is only used
## internally as part of the mechanics for the global :bro:id:`NOTICE`
## function.
function is_being_suppressed(n: Notice::Info): bool
{
if ( n?$identifier && [n$note, n$identifier] in suppressing )

View file

@ -99,7 +99,7 @@ export {
reducers: set[Reducer];
## Provide a function to calculate a value from the
## :bro:see:`Result` structure which will be used
## :bro:see:`SumStats::Result` structure which will be used
## for thresholding.
## This is required if a $threshold value is given.
threshold_val: function(key: SumStats::Key, result: SumStats::Result): count &optional;

View file

@ -16,7 +16,8 @@ export {
redef record ResultVal += {
## This is the queue where elements are maintained. Use the
## :bro:see:`SumStats::get_elements` function to get a vector of the current element values.
## :bro:see:`SumStats::get_last` function to get a vector of
## the current element values.
last_elements: Queue::Queue &optional;
};

View file

@ -1,6 +1,7 @@
##! Base DNS analysis script which tracks and logs DNS queries along with
##! their responses.
@load base/utils/queue
@load ./consts
module DNS;
@ -73,19 +74,6 @@ export {
total_replies: count &optional;
};
## A record type which tracks the status of DNS queries for a given
## :bro:type:`connection`.
type State: record {
## Indexed by query id, returns Info record corresponding to
## query/response which haven't completed yet.
pending: table[count] of Info &optional;
## This is the list of DNS responses that have completed based on the
## number of responses declared and the number received. The contents
## of the set are transaction IDs.
finished_answers: set[count] &optional;
};
## An event that can be handled to access the :bro:type:`DNS::Info`
## record as it is sent to the logging framework.
global log_dns: event(rec: Info);
@ -102,8 +90,33 @@ export {
##
## reply: The specific response information according to RR type/class.
global do_reply: event(c: connection, msg: dns_msg, ans: dns_answer, reply: string);
## A hook that is called whenever a session is being set.
## This can be used if additional initialization logic needs to happen
## when creating a new session value.
##
## c: The connection involved in the new session
##
## msg: The DNS message header information.
##
## is_query: Indicator for if this is being called for a query or a response.
global set_session: hook(c: connection, msg: dns_msg, is_query: bool);
## A record type which tracks the status of DNS queries for a given
## :bro:type:`connection`.
type State: record {
## Indexed by query id, returns Info record corresponding to
## query/response which haven't completed yet.
pending: table[count] of Queue::Queue;
## This is the list of DNS responses that have completed based on the
## number of responses declared and the number received. The contents
## of the set are transaction IDs.
finished_answers: set[count];
};
}
redef record connection += {
dns: Info &optional;
dns_state: State &optional;
@ -128,14 +141,6 @@ event bro_init() &priority=5
function new_session(c: connection, trans_id: count): Info
{
if ( ! c?$dns_state )
{
local state: State;
state$pending=table();
state$finished_answers=set();
c$dns_state = state;
}
local info: Info;
info$ts = network_time();
info$id = c$id;
@ -145,18 +150,37 @@ function new_session(c: connection, trans_id: count): Info
return info;
}
function set_session(c: connection, msg: dns_msg, is_query: bool)
hook set_session(c: connection, msg: dns_msg, is_query: bool) &priority=5
{
if ( ! c?$dns_state || msg$id !in c$dns_state$pending )
if ( ! c?$dns_state )
{
c$dns_state$pending[msg$id] = new_session(c, msg$id);
# Try deleting this transaction id from the set of finished answers.
# Sometimes hosts will reuse ports and transaction ids and this should
# be considered to be a legit scenario (although bad practice).
delete c$dns_state$finished_answers[msg$id];
local state: State;
c$dns_state = state;
}
c$dns = c$dns_state$pending[msg$id];
if ( msg$id !in c$dns_state$pending )
c$dns_state$pending[msg$id] = Queue::init();
local info: Info;
# If this is either a query or this is the reply but
# no Info records are in the queue (we missed the query?)
# we need to create an Info record and put it in the queue.
if ( is_query ||
Queue::len(c$dns_state$pending[msg$id]) == 0 )
{
info = new_session(c, msg$id);
Queue::put(c$dns_state$pending[msg$id], info);
}
if ( is_query )
# If this is a query, assign the newly created info variable
# so that the world looks correct to anything else handling
# this query.
c$dns = info;
else
# Peek at the next item in the queue for this trans_id and
# assign it to c$dns since this is a response.
c$dns = Queue::peek(c$dns_state$pending[msg$id]);
if ( ! is_query )
{
@ -184,7 +208,7 @@ function set_session(c: connection, msg: dns_msg, is_query: bool)
event dns_message(c: connection, is_orig: bool, msg: dns_msg, len: count) &priority=5
{
set_session(c, msg, is_orig);
hook set_session(c, msg, is_orig);
}
event DNS::do_reply(c: connection, msg: dns_msg, ans: dns_answer, reply: string) &priority=5
@ -194,9 +218,6 @@ event DNS::do_reply(c: connection, msg: dns_msg, ans: dns_answer, reply: string)
c$dns$AA = msg$AA;
c$dns$RA = msg$RA;
if ( msg$id in c$dns_state$finished_answers )
event conn_weird("dns_reply_seen_after_done", c, "");
if ( reply != "" )
{
if ( ! c$dns?$answers )
@ -211,7 +232,6 @@ event DNS::do_reply(c: connection, msg: dns_msg, ans: dns_answer, reply: string)
if ( c$dns?$answers && c$dns?$total_answers &&
|c$dns$answers| == c$dns$total_answers )
{
add c$dns_state$finished_answers[c$dns$trans_id];
# Indicate this request/reply pair is ready to be logged.
c$dns$ready = T;
}
@ -224,7 +244,7 @@ event DNS::do_reply(c: connection, msg: dns_msg, ans: dns_answer, reply: string)
{
Log::write(DNS::LOG, c$dns);
# This record is logged and no longer pending.
delete c$dns_state$pending[c$dns$trans_id];
Queue::get(c$dns_state$pending[c$dns$trans_id]);
delete c$dns;
}
}
@ -237,6 +257,7 @@ event dns_request(c: connection, msg: dns_msg, query: string, qtype: count, qcla
c$dns$qclass_name = classes[qclass];
c$dns$qtype = qtype;
c$dns$qtype_name = query_types[qtype];
c$dns$Z = msg$Z;
# Decode netbios name queries
# Note: I'm ignoring the name type for now. Not sure if this should be
@ -244,8 +265,6 @@ event dns_request(c: connection, msg: dns_msg, query: string, qtype: count, qcla
if ( c$id$resp_p == 137/udp )
query = decode_netbios_name(query);
c$dns$query = query;
c$dns$Z = msg$Z;
}
event dns_A_reply(c: connection, msg: dns_msg, ans: dns_answer, a: addr) &priority=5
@ -333,6 +352,13 @@ event connection_state_remove(c: connection) &priority=-5
# If Bro is expiring state, we should go ahead and log all unlogged
# request/response pairs now.
for ( trans_id in c$dns_state$pending )
Log::write(DNS::LOG, c$dns_state$pending[trans_id]);
{
local infos: vector of Info;
Queue::get_vector(c$dns_state$pending[trans_id], infos);
for ( i in infos )
{
Log::write(DNS::LOG, infos[i]);
}
}
}

View file

@ -16,25 +16,32 @@ export {
## Initialize a queue record structure.
##
## s: A :bro:record:`Settings` record configuring the queue.
## s: A record which configures the queue.
##
## Returns: An opaque queue record.
global init: function(s: Settings): Queue;
global init: function(s: Settings &default=[]): Queue;
## Put a string onto the beginning of a queue.
## Put a value onto the beginning of a queue.
##
## q: The queue to put the value into.
##
## val: The value to insert into the queue.
global put: function(q: Queue, val: any);
## Get a string from the end of a queue.
## Get a value from the end of a queue.
##
## q: The queue to get the string from.
## q: The queue to get the value from.
##
## Returns: The value gotten from the queue.
global get: function(q: Queue): any;
## Peek at the value at the end of the queue without removing it.
##
## q: The queue to get the value from.
##
## Returns: The value at the end of the queue.
global peek: function(q: Queue): any;
## Merge two queue's together. If any settings are applied
## to the queues, the settings from q1 are used for the new
## merged queue.
@ -103,6 +110,11 @@ function get(q: Queue): any
return ret;
}
function peek(q: Queue): any
{
return q$vals[q$bottom];
}
function merge(q1: Queue, q2: Queue): Queue
{
local ret = init(q1$settings);

View file

@ -32,8 +32,8 @@ export {
const icmp_time_exceeded_threshold = 3 &redef;
## Interval at which to watch for the
## :bro:id:`ICMPTimeExceeded::icmp_time_exceeded_threshold` variable to be crossed.
## At the end of each interval the counter is reset.
## :bro:id:`Traceroute::icmp_time_exceeded_threshold` variable to be
## crossed. At the end of each interval the counter is reset.
const icmp_time_exceeded_interval = 3min &redef;
## The log record for the traceroute log.

View file

@ -13,17 +13,18 @@ module Scan;
export {
redef enum Notice::Type += {
## Address scans detect that a host appears to be scanning some number of
## destinations on a single port. This notice is generated when more than
## :bro:id:`addr_scan_threshold` unique hosts are seen over the previous
## :bro:id:`addr_scan_interval` time range.
## Address scans detect that a host appears to be scanning some number
## of destinations on a single port. This notice is generated when more
## than :bro:id:`Scan::addr_scan_threshold` unique hosts are seen over
## the previous :bro:id:`Scan::addr_scan_interval` time range.
Address_Scan,
## Port scans detect that an attacking host appears to be scanning a
## single victim host on several ports. This notice is generated when
## an attacking host attempts to connect to :bro:id:`port_scan_threshold`
## an attacking host attempts to connect to
## :bro:id:`Scan::port_scan_threshold`
## unique ports on a single host over the previous
## :bro:id:`port_scan_interval` time range.
## :bro:id:`Scan::port_scan_interval` time range.
Port_Scan,
};

View file

@ -1,3 +1,6 @@
# define SQLITE_THREADSAFE 2
# define SQLITE_DEFAULT_MEMSTATUS 0
/******************************************************************************
** This file is an amalgamation of many separate C source files from SQLite
** version 3.7.16.2. By combining all the individual C code files into this

View file

@ -1162,12 +1162,12 @@ Connection* NetSessions::NewConn(HashKey* k, double t, const ConnID* id,
if ( ! WantConnection(src_h, dst_h, tproto, flags, flip) )
return 0;
ConnID flip_id = *id;
if ( flip )
{
// Make a guess that we're seeing the tail half of
// an analyzable connection.
ConnID flip_id = *id;
const IPAddr ta = flip_id.src_addr;
flip_id.src_addr = flip_id.dst_addr;
flip_id.dst_addr = ta;

View file

@ -2635,7 +2635,7 @@ function bytestring_to_hexstr%(bytestring: string%): string
##
## Returns: The encoded version of *s*.
##
## .. bro:see:: encode_base64_custom, decode_base64
## .. bro:see:: encode_base64_custom decode_base64
function encode_base64%(s: string%): string
%{
BroString* t = encode_base64(s->AsString());
@ -2658,7 +2658,7 @@ function encode_base64%(s: string%): string
##
## Returns: The encoded version of *s*.
##
## .. bro:see:: encode_base64, decode_base64_custom
## .. bro:see:: encode_base64 decode_base64_custom
function encode_base64_custom%(s: string, a: string%): string
%{
BroString* t = encode_base64(s->AsString(), a->AsString());
@ -2677,7 +2677,7 @@ function encode_base64_custom%(s: string, a: string%): string
##
## Returns: The decoded version of *s*.
##
## .. bro:see:: decode_base64_custom, encode_base64
## .. bro:see:: decode_base64_custom encode_base64
function decode_base64%(s: string%): string
%{
BroString* t = decode_base64(s->AsString());
@ -2700,7 +2700,7 @@ function decode_base64%(s: string%): string
##
## Returns: The decoded version of *s*.
##
## .. bro:see:: decode_base64, encode_base64_custom
## .. bro:see:: decode_base64 encode_base64_custom
function decode_base64_custom%(s: string, a: string%): string
%{
BroString* t = decode_base64(s->AsString(), a->AsString());

View file

@ -936,6 +936,7 @@ event file_gap%(f: fa_file, offset: count, len: count%);
## This event is generated each time file analysis is ending for a given file.
##
## f: The file.
##
## .. bro:see:: file_new file_over_new_connection file_timeout file_gap
event file_state_remove%(f: fa_file%);

View file

@ -1271,9 +1271,8 @@ bool Manager::Flush(EnumVal* id)
return true;
}
void Manager::FlushBuffers()
void Manager::Terminate()
{
// Flush out cached entries in Frontend
for ( vector<Stream *>::iterator s = streams.begin(); s != streams.end(); ++s )
{
if ( ! *s )
@ -1281,15 +1280,10 @@ void Manager::FlushBuffers()
for ( Stream::WriterMap::iterator i = (*s)->writers.begin();
i != (*s)->writers.end(); i++ )
i->second->writer->FlushWriteBuffer();
i->second->writer->Stop();
}
}
void Manager::Terminate()
{
FlushBuffers();
}
// Timer which on dispatching rotates the filter.
class RotationTimer : public Timer {
public:

View file

@ -149,12 +149,6 @@ public:
*/
bool Flush(EnumVal* id);
/**
* Flushes all buffers that are currently held by writer frontends
* out to the threads. Does not call the thread flush operation.
*/
void FlushBuffers();
/**
* Signals the manager to shutdown at Bro's termination.
*/

View file

@ -35,13 +35,17 @@ SQLite::SQLite(WriterFrontend* frontend) : WriterBackend(frontend)
db = 0;
io = new AsciiFormatter(this, AsciiFormatter::SeparatorInfo(set_separator, unset_field, empty_field));
st = 0;
}
SQLite::~SQLite()
{
if ( db != 0 )
{
sqlite3_close(db);
sqlite3_finalize(st);
if ( ! sqlite3_close(db) )
Error("Sqlite could not close connection");
db = 0;
}

View file

@ -113,8 +113,9 @@ private:
inline static void safe_lock(pthread_mutex_t* mutex)
{
if ( pthread_mutex_lock(mutex) != 0 )
reporter->FatalErrorWithCore("cannot lock mutex");
int res = pthread_mutex_lock(mutex);
if ( res != 0 )
reporter->FatalErrorWithCore("cannot lock mutex: %d(%s)", res, strerror(res));
}
inline static void safe_unlock(pthread_mutex_t* mutex)

View file

@ -3,9 +3,10 @@
#empty_field (empty)
#unset_field -
#path dns
#open 2012-10-05-17-47-27
#open 2013-05-17-14-28-17
#fields ts uid id.orig_h id.orig_p id.resp_h id.resp_p proto trans_id query qclass qclass_name qtype qtype_name rcode rcode_name AA TC RD RA Z answers TTLs rejected
#types time string addr port addr port enum count string count string count string count string bool bool bool bool count vector[string] vector[interval] bool
1331084278.438444 UWkUyAuUGXf 2001:470:1f11:81f:d138:5f55:6d4:1fe2 51850 2607:f740:b::f93 53 udp 3903 txtpadding_323.n1.netalyzr.icsi.berkeley.edu 1 C_INTERNET 16 TXT 0 NOERROR T F T F 0 This TXT record should be ignored 1.000000 F
1331084293.592245 arKYeMETxOg 2001:470:1f11:81f:d138:5f55:6d4:1fe2 51851 2607:f740:b::f93 53 udp 40849 txtpadding_3230.n1.netalyzr.icsi.berkeley.edu 1 C_INTERNET 16 TXT 0 NOERROR T F T F 0 This TXT record should be ignored 1.000000 F
#close 2012-10-05-17-47-27
1331084298.593081 arKYeMETxOg 2001:470:1f11:81f:d138:5f55:6d4:1fe2 51851 2607:f740:b::f93 53 udp 40849 txtpadding_3230.n1.netalyzr.icsi.berkeley.edu 1 C_INTERNET 16 TXT - - F F T F 0 - - F
#close 2013-05-17-14-28-17

View file

@ -1 +0,0 @@
error: unknown writer type requested

View file

@ -11,5 +11,4 @@
# @TEST-EXEC: test -d $DIST/scripts
# @TEST-EXEC: for script in `find $DIST/scripts/ -name \*\.bro -not -path '*/site/*'`; do echo "=== $script" >>allerrors; if echo "$script" | egrep -q 'communication/listen|controllee'; then rm -rf load_attempt .bgprocs; btest-bg-run load_attempt bro -b $script; btest-bg-wait -k 2; cat load_attempt/.stderr >>allerrors; else bro -b $script 2>>allerrors; fi done || exit 0
# @TEST-EXEC: cat allerrors | grep -v "received termination signal" | grep -v '===' | sort | uniq > unique_errors
# @TEST-EXEC: if [ $(grep -c LibCURL_INCLUDE_DIR-NOTFOUND $BUILD/CMakeCache.txt) -ne 0 ]; then cp unique_errors unique_errors_no_elasticsearch; fi
# @TEST-EXEC: if [ $(grep -c LibCURL_INCLUDE_DIR-NOTFOUND $BUILD/CMakeCache.txt) -ne 0 ]; then btest-diff unique_errors_no_elasticsearch; else btest-diff unique_errors; fi
# @TEST-EXEC: btest-diff unique_errors