Add more logging framework documentation.

This commit is contained in:
Jon Siwek 2011-12-09 14:30:21 -06:00
parent 6d3b29b0ec
commit 1f57827e54
6 changed files with 272 additions and 59 deletions

View file

@ -43,13 +43,14 @@ Basics
======
The data fields that a stream records are defined by a record type
specified when it is created. Let's look at the script generating
Bro's connection summaries as an example,
``base/protocols/conn/main.bro``. It defines a record ``Conn::Info``
that lists all the fields that go into ``conn.log``, each marked with
a ``&log`` attribute indicating that it is part of the information
written out. To write a log record, the script then passes an instance
of ``Conn::Info`` to the logging framework's ``Log::write`` function.
specified when it is created. Let's look at the script generating Bro's
connection summaries as an example,
:doc:`scripts/base/protocols/conn/main`. It defines a record
:bro:type:`Conn::Info` that lists all the fields that go into
``conn.log``, each marked with a ``&log`` attribute indicating that it
is part of the information written out. To write a log record, the
script then passes an instance of :bro:type:`Conn::Info` to the logging
framework's :bro:id:`Log::write` function.
By default, each stream automatically gets a filter named ``default``
that generates the normal output by recording all record fields into a
@ -66,7 +67,7 @@ To create new a new output file for an existing stream, you can add a
new filter. A filter can, e.g., restrict the set of fields being
logged:
.. code:: bro:
.. code:: bro
event bro_init()
{
@ -85,14 +86,15 @@ Note the fields that are set for the filter:
``path``
The filename for the output file, without any extension (which
may be automatically added by the writer). Default path values
are generated by taking the stream's ID and munging it
slightly. ``Conn::LOG`` is converted into ``conn``,
``PacketFilter::LOG`` is converted into ``packet_filter``, and
``Notice::POLICY_LOG`` is converted into ``notice_policy``.
are generated by taking the stream's ID and munging it slightly.
:bro:enum:`Conn::LOG` is converted into ``conn``,
:bro:enum:`PacketFilter::LOG` is converted into
``packet_filter``, and :bro:enum:`Notice::POLICY_LOG` is
converted into ``notice_policy``.
``include``
A set limiting the fields to the ones given. The names
correspond to those in the ``Conn::LOG`` record, with
correspond to those in the :bro:type:`Conn::Info` record, with
sub-records unrolled by concatenating fields (separated with
dots).
@ -158,10 +160,10 @@ further for example to log information by subnets or even by IP
address. Be careful, however, as it is easy to create many files very
quickly ...
.. sidebar:
.. sidebar:: A More Generic Path Function
The show ``split_log`` method has one draw-back: it can be used
only with the ``Conn::Log`` stream as the record type is hardcoded
The ``split_log`` method has one draw-back: it can be used
only with the :bro:enum:`Conn::Log` stream as the record type is hardcoded
into its argument list. However, Bro allows to do a more generic
variant:
@ -201,8 +203,8 @@ Extending
You can add further fields to a log stream by extending the record
type that defines its content. Let's say we want to add a boolean
field ``is_private`` to ``Conn::Info`` that indicates whether the
originator IP address is part of the RFC1918 space:
field ``is_private`` to :bro:type:`Conn::Info` that indicates whether the
originator IP address is part of the :rfc:`1918` space:
.. code:: bro
@ -234,10 +236,10 @@ Notes:
- For extending logs this way, one needs a bit of knowledge about how
the script that creates the log stream is organizing its state
keeping. Most of the standard Bro scripts attach their log state to
the ``connection`` record where it can then be accessed, just as the
``c$conn`` above. For example, the HTTP analysis adds a field ``http
: HTTP::Info`` to the ``connection`` record. See the script
reference for more information.
the :bro:type:`connection` record where it can then be accessed, just
as the ``c$conn`` above. For example, the HTTP analysis adds a field
``http`` of type :bro:type:`HTTP::Info` to the :bro:type:`connection`
record. See the script reference for more information.
- When extending records as shown above, the new fields must always be
declared either with a ``&default`` value or as ``&optional``.
@ -251,8 +253,8 @@ Sometimes it is helpful to do additional analysis of the information
being logged. For these cases, a stream can specify an event that will
be generated every time a log record is written to it. All of Bro's
default log streams define such an event. For example, the connection
log stream raises the event ``Conn::log_conn(rec: Conn::Info)``: You
could use that for example for flagging when an a connection to
log stream raises the event :bro:id:`Conn::log_conn`. You
could use that for example for flagging when a connection to
specific destination exceeds a certain duration:
.. code:: bro
@ -279,11 +281,32 @@ real-time.
Rotation
--------
By default, no log rotation occurs, but it's globally controllable for all
filters by redefining the :bro:id:`Log::default_rotation_interval` option:
.. code:: bro
redef Log::default_rotation_interval = 1 hr;
Or specifically for certain :bro:type:`Log::Filter` instances by setting
their ``interv`` field. Here's an example of changing just the
:bro:enum:`Conn::LOG` stream's default filter rotation.
.. code:: bro
event bro_init()
{
local f = Log::get_filter(Conn::LOG, "default");
f$interv = 1 min;
Log::remove_filter(Conn::LOG, "default");
Log::add_filter(Conn::LOG, f);
}
ASCII Writer Configuration
--------------------------
The ASCII writer has a number of options for customizing the format of
its output, see XXX.bro.
its output, see :doc:`scripts/base/frameworks/logging/writers/ascii`.
Adding Streams
==============
@ -321,8 +344,8 @@ example for the ``Foo`` module:
Log::create_stream(Foo::LOG, [$columns=Info, $ev=log_foo]);
}
You can also the state to the ``connection`` record to make it easily
accessible across event handlers:
You can also add the state to the :bro:type:`connection` record to make
it easily accessible across event handlers:
.. code:: bro
@ -330,7 +353,7 @@ accessible across event handlers:
foo: Info &optional;
}
Now you can use the ``Log::write`` method to output log records and
Now you can use the :bro:id:`Log::write` method to output log records and
save the logged ``Foo::Info`` record into the connection record:
.. code:: bro
@ -343,9 +366,9 @@ save the logged ``Foo::Info`` record into the connection record:
}
See the existing scripts for how to work with such a new connection
field. A simple example is ``base/protocols/syslog/main.bro``.
field. A simple example is :doc:`scripts/base/protocols/syslog/main`.
When you are developing scripts that add data to the ``connection``
When you are developing scripts that add data to the :bro:type:`connection`
record, care must be given to when and how long data is stored.
Normally data saved to the connection record will remain there for the
duration of the connection and from a practical perspective it's not

View file

@ -410,6 +410,10 @@ The Bro scripting language supports the following built-in types.
Writing to files like this for logging usually isn't recommend, for better
logging support see :doc:`/logging`.
.. bro:type:: func
See :bro:type:`function`.
.. bro:type:: function
Function types in Bro are declared using::

View file

@ -1,16 +1,16 @@
##! The Bro logging interface.
##!
##! See XXX for a introduction to Bro's logging framework.
##! See :doc:`/logging` for a introduction to Bro's logging framework.
module Log;
# Log::ID and Log::Writer are defined in bro.init due to circular dependencies.
# Log::ID and Log::Writer are defined in types.bif due to circular dependencies.
export {
## If true, is local logging is by default enabled for all filters.
## If true, local logging is by default enabled for all filters.
const enable_local_logging = T &redef;
## If true, is remote logging is by default enabled for all filters.
## If true, remote logging is by default enabled for all filters.
const enable_remote_logging = T &redef;
## Default writer to use if a filter does not specify
@ -23,21 +23,24 @@ export {
columns: any;
## Event that will be raised once for each log entry.
## The event receives a single same parameter, an instance of type ``columns``.
## The event receives a single same parameter, an instance of type
## ``columns``.
ev: any &optional;
};
## Default function for building the path values for log filters if not
## speficied otherwise by a filter. The default implementation uses ``id``
## Builds the default path values for log filters if not otherwise
## specified by a filter. The default implementation uses *id*
## to derive a name.
##
## id: The log stream.
## id: The ID associated with the log stream.
##
## path: A suggested path value, which may be either the filter's
## ``path`` if defined, else a previous result from the function.
## If no ``path`` is defined for the filter, then the first call
## to the function will contain an empty string.
##
## rec: An instance of the streams's ``columns`` type with its
## fields set to the values to logged.
## fields set to the values to be logged.
##
## Returns: The path to be used for the filter.
global default_path_func: function(id: ID, path: string, rec: any) : string &redef;
@ -46,7 +49,7 @@ export {
## Information passed into rotation callback functions.
type RotationInfo: record {
writer: Writer; ##< Writer.
writer: Writer; ##< The :bro:type:`Log::Writer` being used.
fname: string; ##< Full name of the rotated file.
path: string; ##< Original path value.
open: time; ##< Time when opened.
@ -57,25 +60,26 @@ export {
## Default rotation interval. Zero disables rotation.
const default_rotation_interval = 0secs &redef;
## Default naming format for timestamps embedded into filenames. Uses a strftime() style.
## Default naming format for timestamps embedded into filenames.
## Uses a ``strftime()`` style.
const default_rotation_date_format = "%Y-%m-%d-%H-%M-%S" &redef;
## Default shell command to run on rotated files. Empty for none.
const default_rotation_postprocessor_cmd = "" &redef;
## Specifies the default postprocessor function per writer type. Entries in this
## table are initialized by each writer type.
## Specifies the default postprocessor function per writer type.
## Entries in this table are initialized by each writer type.
const default_rotation_postprocessors: table[Writer] of function(info: RotationInfo) : bool &redef;
## Filter customizing logging.
## A filter type describes how to customize logging streams.
type Filter: record {
## Descriptive name to reference this filter.
name: string;
## The writer to use.
## The logging writer implementation to use.
writer: Writer &default=default_writer;
## Predicate indicating whether a log entry should be recorded.
## Indicates whether a log entry should be recorded.
## If not given, all entries are recorded.
##
## rec: An instance of the streams's ``columns`` type with its
@ -101,13 +105,15 @@ export {
## easy to flood the disk by returning a new string for each
## connection ...
##
## id: The log stream.
## id: The ID associated with the log stream.
##
## path: A suggested path value, which may be either the filter's
## ``path`` if defined, else a previous result from the function.
## If no ``path`` is defined for the filter, then the first call
## to the function will contain an empty string.
##
## rec: An instance of the streams's ``columns`` type with its
## fields set to the values to logged.
## fields set to the values to be logged.
##
## Returns: The path to be used for the filter.
path_func: function(id: ID, path: string, rec: any): string &optional;
@ -129,27 +135,183 @@ export {
## Rotation interval.
interv: interval &default=default_rotation_interval;
## Callback function to trigger for rotated files. If not set,
## the default comes out of default_rotation_postprocessors.
## Callback function to trigger for rotated files. If not set, the
## default comes out of :bro:id:`Log::default_rotation_postprocessors`.
postprocessor: function(info: RotationInfo) : bool &optional;
};
## Sentinel value for indicating that a filter was not found when looked up.
const no_filter: Filter = [$name="<not found>"]; # Sentinel.
const no_filter: Filter = [$name="<not found>"];
# TODO: Document.
## Creates a new logging stream with the default filter.
##
## id: The ID enum to be associated with the new logging stream.
##
## stream: A record defining the content that the new stream will log.
##
## Returns: True if a new logging stream was successfully created and
## a default filter added to it.
##
## .. bro:see:: Log::add_default_filter Log::remove_default_filter
global create_stream: function(id: ID, stream: Stream) : bool;
## Enables a previously disabled logging stream. Disabled streams
## will not be written to until they are enabled again. New streams
## are enabled by default.
##
## id: The ID associated with the logging stream to enable.
##
## Returns: True if the stream is re-enabled or was not previously disabled.
##
## .. bro:see:: Log::disable_stream
global enable_stream: function(id: ID) : bool;
## Disables a currently enabled logging stream. Disabled streams
## will not be written to until they are enabled again. New streams
## are enabled by default.
##
## id: The ID associated with the logging stream to disable.
##
## Returns: True if the stream is now disabled or was already disabled.
##
## .. bro:see:: Log::enable_stream
global disable_stream: function(id: ID) : bool;
## Adds a custom filter to an existing logging stream. If a filter
## with a matching ``name`` field already exists for the stream, it
## is removed when the new filter is successfully added.
##
## id: The ID associated with the logging stream to filter.
##
## filter: A record describing the desired logging parameters.
##
## Returns: True if the filter was sucessfully added, false if
## the filter was not added or the *filter* argument was not
## the correct type.
##
## .. bro:see:: Log::remove_filter Log::add_default_filter
## Log::remove_default_filter
global add_filter: function(id: ID, filter: Filter) : bool;
## Removes a filter from an existing logging stream.
##
## id: The ID associated with the logging stream from which to
## remove a filter.
##
## name: A string to match against the ``name`` field of a
## :bro:type:`Log::Filter` for identification purposes.
##
## Returns: True if the logging stream's filter was removed or
## if no filter associated with *name* was found.
##
## .. bro:see:: Log::remove_filter Log::add_default_filter
## Log::remove_default_filter
global remove_filter: function(id: ID, name: string) : bool;
global get_filter: function(id: ID, name: string) : Filter; # Returns no_filter if not found.
## Gets a filter associated with an existing logging stream.
##
## id: The ID associated with a logging stream from which to
## obtain one of its filters.
##
## name: A string to match against the ``name`` field of a
## :bro:type:`Log::Filter` for identification purposes.
##
## Returns: A filter attached to the logging stream *id* matching
## *name* or, if no matches are found returns the
## :bro:id:`Log::no_filter` sentinel value.
##
## .. bro:see:: Log::add_filter Log::remove_filter Log::add_default_filter
## Log::remove_default_filter
global get_filter: function(id: ID, name: string) : Filter;
## Writes a new log line/entry to a logging stream.
##
## id: The ID associated with a logging stream to be written to.
##
## columns: A record value describing the values of each field/column
## to write to the log stream.
##
## Returns: True if the stream was found and no error occurred in writing
## to it or if the stream was disabled and nothing was written.
## False if the stream was was not found, or the *columns*
## argument did not match what the stream was initially defined
## to handle, or one of the stream's filters has an invalid
## ``path_func``.
##
## .. bro:see: Log::enable_stream Log::disable_stream
global write: function(id: ID, columns: any) : bool;
## Sets the buffering status for all the writers of a given logging stream.
## A given writer implementation may or may not support buffering and if it
## doesn't then toggling buffering with this function has no effect.
##
## id: The ID associated with a logging stream for which to
## enable/disable buffering.
##
## buffered: Whether to enable or disable log buffering.
##
## Returns: True if buffering status was set, false if the logging stream
## does not exist.
##
## .. bro:see:: Log::flush
global set_buf: function(id: ID, buffered: bool): bool;
## Flushes any currently buffered output for all the writers of a given
## logging stream.
##
## id: The ID associated with a logging stream for which to flush buffered
## data.
##
## Returns: True if all writers of a log stream were signalled to flush
## buffered data or if the logging stream is disabled,
## false if the logging stream does not exist.
##
## .. bro:see:: Log::set_buf Log::enable_stream Log::disable_stream
global flush: function(id: ID): bool;
## Adds a default :bro:type:`Log::Filter` record with ``name`` field
## set as "default" to a given logging stream.
##
## id: The ID associated with a logging stream for which to add a default
## filter.
##
## Returns: The status of a call to :bro:id:`Log::add_filter` using a
## default :bro:type:`Log::Filter` argument with ``name`` field
## set to "default".
##
## .. bro:see:: Log::add_filter Log::remove_filter
## Log::remove_default_filter
global add_default_filter: function(id: ID) : bool;
## Removes the :bro:type:`Log::Filter` with ``name`` field equal to
## "default".
##
## id: The ID associated with a logging stream from which to remove the
## default filter.
##
## Returns: The status of a call to :bro:id:`Log::remove_filter` using
## "default" as the argument.
##
## .. bro:see:: Log::add_filter Log::remove_filter Log::add_default_filter
global remove_default_filter: function(id: ID) : bool;
## Runs a command given by :bro:id:`Log::default_rotation_postprocessor_cmd`
## on a rotated file. Meant to be called from postprocessor functions
## that are added to :bro:id:`Log::default_rotation_postprocessors`.
##
## info: A record holding meta-information about the log being rotated.
##
## npath: The new path of the file (after already being rotated/processed
## by writer-specific postprocessor as defined in
## :bro:id:`Log::default_rotation_postprocessors`.
##
## Returns: True when :bro:id:`Log::default_rotation_postprocessor_cmd`
## is empty or the system command given by it has been invoked
## to postprocess a rotated log file.
##
## .. bro:see:: Log::default_rotation_date_format
## Log::default_rotation_postprocessor_cmd
## Log::default_rotation_postprocessors
global run_rotation_postprocessor_cmd: function(info: RotationInfo, npath: string) : bool;
}

View file

@ -1,29 +1,51 @@
##! This script defines a postprocessing function that can be applied
##! to a logging filter in order to automatically SCP (secure copy)
##! a log stream (or a subset of it) to a remote host at configurable
##! rotation time intervals.
##! rotation time intervals. Generally, to use this functionality
##! you must handle the :bro:id:`bro_init` event and do the following
##! in your handler:
##!
##! 1) Create a new :bro:type:`Log::Filter` record that defines a name/path,
##! rotation interval, and set the ``postprocessor`` to
##! :bro:id:`Log::scp_postprocessor`.
##! 2) Add the filter to a logging stream using :bro:id:`Log::add_filter`.
##! 3) Add a table entry to :bro:id:`Log::scp_destinations` for the filter's
##! writer/path pair which defines a set of :bro:type:`Log::SCPDestination`
##! records.
module Log;
export {
## This postprocessor SCP's the rotated-log to all the remote hosts
## Secure-copies the rotated-log to all the remote hosts
## defined in :bro:id:`Log::scp_destinations` and then deletes
## the local copy of the rotated-log. It's not active when
## reading from trace files.
##
## info: A record holding meta-information about the log file to be
## postprocessed.
##
## Returns: True if secure-copy system command was initiated or
## if no destination was configured for the log as described
## by *info*.
global scp_postprocessor: function(info: Log::RotationInfo): bool;
## A container that describes the remote destination for the SCP command
## argument as ``user@host:path``.
type SCPDestination: record {
## The remote user to log in as. A trust mechanism should be
## pre-established.
user: string;
## The remote host to which to transfer logs.
host: string;
## The path/directory on the remote host to send logs.
path: string;
};
## A table indexed by a particular log writer and filter path, that yields
## a set remote destinations. The :bro:id:`Log::scp_postprocessor`
## function queries this table upon log rotation and performs a secure
## copy of the rotated-log to each destination in the set.
## copy of the rotated-log to each destination in the set. This
## table can be modified at run-time.
global scp_destinations: table[Writer, string] of set[SCPDestination];
}

View file

@ -1,4 +1,5 @@
##! Interface for the ascii log writer.
##! Interface for the ASCII log writer. Redefinable options are available
##! to tweak the output format of ASCII logs.
module LogAscii;
@ -7,7 +8,8 @@ export {
## into files. This is primarily for debugging purposes.
const output_to_stdout = F &redef;
## If true, include a header line with column names.
## If true, include a header line with column names and description
## of the other ASCII logging options that were used.
const include_header = T &redef;
## Prefix for the header line if included.

View file

@ -1,4 +1,4 @@
# Internal functions and types used by the logging framework.
##! Internal functions and types used by the logging framework.
module Log;