Mirror/zeek
1
0
Fork 0
mirror of https://github.com/zeek/zeek.git synced 2025-10-02 06:38:20 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 1

Merge remote-tracking branch 'origin/topic/documentation'

Browse source 
I'm merging in the documentation branch to avoid that it keeps getting
out of sync. We still need to work on this a bit more, I'll summarize
in a mail.

* origin/topic/documentation: (68 commits)
  Going over initial parts of the scripting overview.
  Adding lines in scripting/index.txt. No other change.
  Updating submodule.
  Pass over the Using Bro section.
  Applying doc updates from BIT-1063. (Anthony Verez)
  Breaking lines in using/*.
  More doc reorg, and a light pass over the first 3 sections.
  Starting to put a new structure in place.
  Updating submodule(s).
  Updating submodule(s).
  Adding some temporary testing/demonstration to front page.
  Switching btest to topic/robin/parts branch.
  Basic cross-referencing UIDs between files, btests, and baselines.
  A truly embarassing number of spelling corrections.
  Using redirection with bro-cut.  Include initial btests for this document.
  Corrected a mis-typed RST include.
  Initial start for "Using Bro".
  Spelling corrections.
  Include Notice Policy shortcuts in the Scripting User Manual.
  Notice::policy hooks and tests.
  ...

Conflicts:
	src/3rdparty
This commit is contained in:
Robin Sommer 2013-08-31 15:20:20 -07:00
commit ac66db286a
309 changed files with 11468 additions and 1312 deletions
Download patch file Download diff file Expand all files Collapse all files

5
CHANGES
View file

@ -1,4 +1,9 @@
2.1-1306 | 2013-08-31 16:06:05 -0700
* Reorganized and signifcantly extended documentation. This includes
two new chapters contributed by Scott Runnels.
2.1-1216 | 2013-08-31 10:39:40 -0700

313
INSTALL
View file

@ -1,314 +1,3 @@
.. _CMake: http://www.cmake.org
.. _SWIG: http://www.swig.org
.. _Xcode: https://developer.apple.com/xcode/
.. _MacPorts: http://www.macports.org
.. _Fink: http://www.finkproject.org
.. _Homebrew: http://mxcl.github.com/homebrew
.. _bro downloads page: http://bro.org/download/index.html
==============
Installing Bro
==============
See doc/install/install.rst for installation instructions.
Bro can be downloaded in either pre-built binary package or
source code forms.
Prerequisites
=============
Bro requires the following libraries and tools to be installed
before you begin:
* Libpcap http://www.tcpdump.org
* OpenSSL libraries http://www.openssl.org
* BIND8 library
* Libmagic
* Libz
* Bash (for BroControl)
To build Bro from source, the following additional dependencies are required:
* CMake 2.6.3 or greater http://www.cmake.org
* SWIG http://www.swig.org
* Bison (GNU Parser Generator)
* Flex (Fast Lexical Analyzer)
* Libpcap headers http://www.tcpdump.org
* OpenSSL headers http://www.openssl.org
* libmagic headers
* zlib headers
* Perl
Bro can make use of some optional libraries and tools if they are found at
build time:
* LibGeoIP (for geo-locating IP addresses)
* gperftools (tcmalloc is used to improve memory and CPU usage)
* sendmail (for BroControl)
* ipsumdump (for trace-summary) http://www.cs.ucla.edu/~kohler/ipsumdump
* Ruby executable, library, and headers (for Broccoli Ruby bindings)
Installing From Pre-Built Binary Release Packages
=================================================
See the `bro downloads page`_ for currently supported/targeted platforms.
* RPM
.. console::
sudo yum localinstall Bro-*.rpm
* DEB
.. console::
sudo gdebi Bro-*.deb
* MacOS Disk Image with Installer
Just open the ``Bro-*.dmg`` and then run the ``.pkg`` installer.
Everything installed by the package will go into ``/opt/bro``.
The primary install prefix for binary packages is ``/opt/bro``.
Non-MacOS packages that include BroControl also put variable/runtime
data (e.g. Bro logs) in ``/var/opt/bro``.
Installing From Source
======================
Required Dependencies
~~~~~~~~~~~~~~~~~~~~~
The following dependencies are required to build Bro:
* RPM/RedHat-based Linux:
.. console::
sudo yum install cmake make gcc gcc-c++ flex bison libpcap-devel openssl-devel python-devel swig zlib-devel file-devel
* DEB/Debian-based Linux:
.. console::
sudo apt-get install cmake make gcc g++ flex bison libpcap-dev libssl-dev python-dev swig zlib1g-dev libmagic-dev
* FreeBSD
Most required dependencies should come with a minimal FreeBSD install
except for the following.
.. console::
sudo pkg_add -r bash cmake swig bison python
Note that ``bash`` needs to be in ``PATH``, which by default it is
not. The FreeBSD package installs the binary into
``/usr/local/bin``.
* Mac OS X
Compiling source code on Macs requires first downloading Xcode_,
then going through its "Preferences..." -> "Downloads" menus to
install the "Command Line Tools" component.
Lion (10.7) and Mountain Lion (10.8) come with all required
dependencies except for CMake_, SWIG_, and ``libmagic``.
Distributions of these dependencies can likely be obtained from your
preferred Mac OS X package management system (e.g. MacPorts_, Fink_,
or Homebrew_).
Specifically for MacPorts, the ``swig``, ``swig-ruby``, ``swig-python``
and ``file`` packages provide the required dependencies.
Optional Dependencies
~~~~~~~~~~~~~~~~~~~~~
Bro can use libGeoIP for geo-locating IP addresses, and sendmail for
sending emails.
* RedHat Enterprise Linux:
.. console::
sudo yum install geoip-devel sendmail
* CentOS Linux:
.. console::
sudo yum install GeoIP-devel sendmail
* DEB/Debian-based Linux:
.. console::
sudo apt-get install libgeoip-dev sendmail
* Ports-based FreeBSD
.. console::
sudo pkg_add -r GeoIP
sendmail is typically already available.
* Mac OS X
Vanilla OS X installations don't ship with libGeoIP, but
if installed from your preferred package management system (e.g. MacPorts,
Fink, or Homebrew), they should be automatically detected and Bro will
compile against them.
Additional steps may be needed to :doc:`get the right GeoIP database <geoip>`.
Compiling Bro Source Code
~~~~~~~~~~~~~~~~~~~~~~~~~
Bro releases are bundled into source packages for convenience and
available from the `bro downloads page`_.
Alternatively, the latest Bro development version can be obtained through git
repositories hosted at `git.bro.org <http://git.bro.org>`_. See
our `git development documentation
<http://bro.org/development/process.html>`_ for comprehensive
information on Bro's use of git revision control, but the short story
for downloading the full source code experience for Bro via git is:
.. console::
git clone --recursive git://git.bro.org/bro
.. note:: If you choose to clone the ``bro`` repository non-recursively for
a "minimal Bro experience", be aware that compiling it depends on
BinPAC, which has its own ``binpac`` repository. Either install it
first or initialize/update the cloned ``bro`` repository's
``aux/binpac`` submodule.
The typical way to build and install from source is (for more options,
run ``./configure --help``):
.. console::
./configure
make
make install
The default installation path is ``/usr/local/bro``, which would typically
require root privileges when doing the ``make install``. A different
installation path can be chosen by specifying the ``--prefix`` option.
Note that ``/usr`` and ``/opt/bro`` are the
standard prefixes for binary Bro packages to be installed, so those are
typically not good choices unless you are creating such a package.
Depending on the Bro package you downloaded, there may be auxiliary
tools and libraries available in the ``aux/`` directory. Some of them
will be automatically built and installed along with Bro. There are
``--disable-*`` options that can be given to the configure script to
turn off unwanted auxiliary projects that would otherwise be installed
automatically. Finally, use ``make install-aux`` to install some of
the other programs that are in the ``aux/bro-aux`` directory.
OpenBSD users, please see our FAQ at
http://www.bro.org/documentation/faq.html if you are having
problems installing Bro.
Upgrading From a Previous Version of Bro
========================================
If you're doing an upgrade install (rather than a fresh install),
there's two suggested approaches: either install Bro using the same
installation prefix directory as before, or pick a new prefix and copy
local customizations over.
Re-Use Previous Install Prefix
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
If you choose to configure and install Bro with the same prefix
directory as before, local customization and configuration to files in
``$prefix/share/bro/site`` and ``$prefix/etc`` won't be overwritten
(``$prefix`` indicating the root of where Bro was installed). Also, logs
generated at run-time won't be touched by the upgrade. (But making
a backup of local changes before upgrading is still recommended.)
After upgrading, remember to check ``$prefix/share/bro/site`` and
``$prefix/etc`` for ``.example`` files, which indicate the
distribution's version of the file differs from the local one, which may
include local changes. Review the differences, and make adjustments
as necessary (for differences that aren't the result of a local change,
use the new version's).
Pick a New Install prefix
~~~~~~~~~~~~~~~~~~~~~~~~~
If you want to install the newer version in a different prefix
directory than before, you can just copy local customization and
configuration files from ``$prefix/share/bro/site`` and ``$prefix/etc``
to the new location (``$prefix`` indicating the root of where Bro was
originally installed). Make sure to review the files for difference
before copying and make adjustments as necessary (for differences that
aren't the result of a local change, use the new version's). Of
particular note, the copied version of ``$prefix/etc/broctl.cfg`` is
likely to need changes to the ``SpoolDir`` and ``LogDir`` settings.
Configure the Run-Time Environment
==================================
Just remember that you may need to adjust your ``PATH`` environment variable
according to the platform/shell/package you're using. For example:
Bourne-Shell Syntax:
.. console::
export PATH=/usr/local/bro/bin:$PATH
C-Shell Syntax:
.. console::
setenv PATH /usr/local/bro/bin:$PATH
Or substitute ``/opt/bro/bin`` instead if you installed from a binary package.
Running Bro
===========
Bro is a complex program and it takes a bit of time to get familiar
with it. A good place for newcomers to start is the Quick Start Guide
at http://www.bro.org/documentation/quickstart.html.
For developers that wish to run Bro directly from the ``build/``
directory (i.e., without performing ``make install``), they will have
to first adjust ``BROPATH`` to look for scripts inside the build
directory. Sourcing either ``build/bro-path-dev.sh`` or
``build/bro-path-dev.csh`` as appropriate for the current shell
accomplishes this and also augments your ``PATH`` so you can use the
Bro binary directly::
./configure
make
source build/bro-path-dev.sh
bro <options>

352
NEWS
View file

@ -1,17 +1,14 @@
Release Notes
=============
This document summarizes the most important changes in the current Bro
release. For a complete list of changes, see the ``CHANGES`` file
(note that submodules, such as BroControl and Broccoli, come with
their own CHANGES.)
Bro 2.2
-------
Bro 2.2 (Work In Progress)
==========================
New Functionality
~~~~~~~~~~~~~~~~~
-----------------
- GPRS Tunnelling Protocol (GTPv1) decapsulation.
@ -147,8 +144,28 @@ New Functionality
base/utils/active-http.bro for providing an interface for querying
remote web servers.
- Summary statistics framework. [Extend]
- A number of new applications build on top of the summary statistics
framework:
* Scan detection: Detectors for port and address scans return. See
policy/misc/scan.bro.
* Tracerouter detector: policy/misc/detect-traceroute
* Web application detection/measurement: policy/misc/app-metrics.bro
* FTP brute-forcing detector: policy/protocols/ftp/detect-bruteforcing.bro
* HTTP-based SQL injection detector: policy/protocols/http/detect-sqli.bro
(existed before, but now ported to the new framework)
* SSH brute-forcing detector feeding the intelligence framework:
policy/protocols/ssh/detect-bruteforcing.bro
Changed Functionality
~~~~~~~~~~~~~~~~~~~~~
---------------------
- We removed the following, already deprecated, functionality:
@ -231,11 +248,12 @@ Changed Functionality
- We removed the BitTorrent DPD signatures pending further updates to
that analyzer.
Bro 2.1
-------
=======
New Functionality
~~~~~~~~~~~~~~~~~
-----------------
- Bro now comes with extensive IPv6 support. Past versions offered
only basic IPv6 functionality that was rarely used in practice as it
@ -314,30 +332,9 @@ New Functionality
outputs. We do not yet recommend them for production (but welcome
feedback!)
- Summary statistics framework. [Extend]
- A number of new applications build on top of the summary statistics
framework:
* Scan detection: Detectors for port and address scans return. See
policy/misc/scan.bro.
* Tracerouter detector: policy/misc/detect-traceroute
* Web application detection/measurement: policy/misc/app-metrics.bro
* FTP brute-forcing detector: policy/protocols/ftp/detect-bruteforcing.bro
* HTTP-based SQL injection detector: policy/protocols/http/detect-sqli.bro
(existed before, but now ported to the new framework)
* SSH brute-forcing detector feeding the intelligence framework:
policy/protocols/ssh/detect-bruteforcing.bro
Changed Functionality
~~~~~~~~~~~~~~~~~~~~~
---------------------
The following summarizes the most important differences in existing
functionality. Note that this list is not complete, see CHANGES for
@ -371,7 +368,10 @@ the full set.
soon. With that, "match" and "using" are no longer reserved keywords.
- The syntax for IPv6 literals changed from "2607:f8b0:4009:802::1012"
to "[2607:f8b0:4009:802::1012]".
to "[2607:f8b0:4009:802::1012]". When an IP address variable or IP
address literal is enclosed in pipes (for example,
``|[fe80::db15]|``) the result is now the size of the address in
bits (32 for IPv4 and 128 for IPv6).
- Bro now spawns threads for doing its logging. From a user's
perspective not much should change, except that the OS may now show
@ -411,60 +411,274 @@ the full set.
- The ASCII writers "header_*" options have been renamed to "meta_*"
(because there's now also a footer).
- Some built-in functions have been removed: "addr_to_count" (use
"addr_to_counts" instead), "bro_has_ipv6" (this is no longer
relevant because Bro now always supports IPv6), "active_connection"
(use "connection_exists" instead), and "connection_record" (use
"lookup_connection" instead).
- The "NFS3::mode2string" built-in function has been renamed to
"file_mode".
- Some built-in functions have been changed: "exit" (now takes the
exit code as a parameter), "to_port" (now takes a string as
parameter instead of a count and transport protocol, but
"count_to_port" is still available), "connect" (now takes an
additional string parameter specifying the zone of a non-global IPv6
address), and "listen" (now takes three additional parameters to
enable listening on IPv6 addresses).
- Some Bro script variables have been renamed:
"LogAscii::header_prefix" has been renamed to
"LogAscii::meta_prefix", "LogAscii::include_header" has been renamed
to "LogAscii::include_meta".
- Some Bro script variables have been removed: "tunnel_port",
"parse_udp_tunnels", "use_connection_compressor",
"cc_handle_resets", "cc_handle_only_syns", and
"cc_instantiate_on_data".
- A couple events have changed: the "icmp_redirect" event now includes
the target and destination addresses and any Neighbor Discovery
options in the message, and the last parameter of the
"dns_AAAA_reply" event has been removed because it was unused.
- The format of the ASCII log files has changed very slightly. Two
new lines are automatically added, one to record the time when the
log was opened, and the other to record the time when the log was
closed.
- In BroControl, the option (in broctl.cfg) "CFlowAddr" was renamed to
"CFlowAddress".
Bro 2.0
-------
=======
As the version number jump suggests, Bro 2.0 is a major upgrade and
lots of things have changed. We have assembled a separate upgrade
guide with the most important changes compared to Bro 1.5 at
http://www.bro.org/documentation/upgrade.html. You can find
the offline version of that document in ``doc/upgrade.rst.``.
As the version number jump from 1.5 suggests, Bro 2.0 is a major
upgrade and lots of things have changed. Most importantly, we have
rewritten almost all of Bro's default scripts from scratch, using
quite different structure now and focusing more on operational
deployment. The result is a system that works much better "out of the
box", even without much initial site-specific configuration. The
down-side is that 1.x configurations will need to be adapted to work
with the new version. The two rules of thumb are:
Compared to the earlier 2.0 Beta version, the major changes in the
final release are:
(1) If you have written your own Bro scripts
that do not depend on any of the standard scripts formerly
found in ``policy/``, they will most likely just keep working
(although you might want to adapt them to use some of the new
features, like the new logging framework; see below).
* The default scripts now come with complete reference
documentation. See
http://www.bro.org/documentation/index.html.
(2) If you have custom code that depends on specifics of 1.x
default scripts (including most configuration tuning), that is
unlikely to work with 2.x. We recommend to start by using just
the new scripts first, and then port over any customizations
incrementally as necessary (they may be much easier to do now,
or even unnecessary). Send mail to the Bro user mailing list
if you need help.
* libz and libmagic are now required dependencies.
Below we summarize changes from 1.x to 2.x in more detail. This list
isn't complete, see the :download:`CHANGES <CHANGES>` file in the
distribution for the full story.
* Reduced snaplen default from 65535 to old default of 8192. The
large value was introducing performance problems on many
systems.
Script Organization
-------------------
* Replaced the --snaplen/-l command line option with a
scripting-layer option called "snaplen". The new option can also
be redefined on the command line, e.g. ``bro -i eth0
snaplen=65535``.
In versions before 2.0, Bro scripts were all maintained in a flat
directory called ``policy/`` in the source tree. This directory is now
renamed to ``scripts/`` and contains major subdirectories ``base/``,
``policy/``, and ``site/``, each of which may also be subdivided
further.
* Reintroduced the BRO_LOG_SUFFIX environment variable that the
ASCII logger now respects to add a suffix to the log files it
creates.
The contents of the new ``scripts/`` directory, like the old/flat
``policy/`` still gets installed under the ``share/bro``
subdirectory of the installation prefix path just like previous
versions. For example, if Bro was compiled like ``./configure
--prefix=/usr/local/bro && make && make install``, then the script
hierarchy can be found in ``/usr/local/bro/share/bro``.
* The ASCII logs now include further header information, and
fields set to an empty value are now logged as ``(empty)`` by
default (instead of ``-``, which is already used for fields that
are not set at all).
The main
subdirectories of that hierarchy are as follows:
* Some NOTICES were renamed, and the signatures of some SSL events
have changed.
- ``base/`` contains all scripts that are loaded by Bro by default
(unless the ``-b`` command line option is used to run Bro in a
minimal configuration). Note that is a major conceptual change:
rather than not loading anything by default, Bro now uses an
extensive set of default scripts out of the box.
* bro-cut got some new capabilities:
The scripts under this directory generally either accumulate/log
useful state/protocol information for monitored traffic, configure a
default/recommended mode of operation, or provide extra Bro
scripting-layer functionality that has no significant performance cost.
- If no field names are given on the command line, we now pass
through all fields.
- ``policy/`` contains all scripts that a user will need to explicitly
tell Bro to load. These are scripts that implement
functionality/analysis that not all users may want to use and may have
more significant performance costs. For a new installation, you
should go through these and see what appears useful to load.
- New options -u/-U for time output in UTC.
- ``site/`` remains a directory that can be used to store locally
developed scripts. It now comes with some preinstalled example
scripts that contain recommended default configurations going beyond
the ``base/`` setup. E.g. ``local.bro`` loads extra scripts from
``policy/`` and does extra tuning. These files can be customized in
place without being overwritten by upgrades/reinstalls, unlike
scripts in other directories.
- New option -F to give output field separator.
With version 2.0, the default ``BROPATH`` is set to automatically
search for scripts in ``policy/``, ``site/`` and their parent
directory, but **not** ``base/``. Generally, everything under
``base/`` is loaded automatically, but for users of the ``-b`` option,
it's important to know that loading a script in that directory
requires the extra ``base/`` path qualification. For example, the
following two scripts:
* Broccoli supports more types internally, allowing to send
complex records.
* ``$PREFIX/share/bro/base/protocols/ssl/main.bro``
* ``$PREFIX/share/bro/policy/protocols/ssl/validate-certs.bro``
* Many smaller bug fixes, portability improvements, and general
polishing across all modules.
are referenced from another Bro script like:
.. code:: bro
@load base/protocols/ssl/main
@load protocols/ssl/validate-certs
Notice how ``policy/`` can be omitted as a convenience in the second
case. ``@load`` can now also use relative path, e.g., ``@load
../main``.
Logging Framework
-----------------
- The logs generated by scripts that ship with Bro are entirely redone
to use a standardized, machine parsable format via the new logging
framework. Generally, the log content has been restructured towards
making it more directly useful to operations. Also, several
analyzers have been significantly extended and thus now log more
information. Take a look at ``ssl.log``.
* A particular format change that may be useful to note is that the
``conn.log`` ``service`` field is derived from DPD instead of
well-known ports (while that was already possible in 1.5, it was
not the default).
* Also, ``conn.log`` now reports raw number of packets/bytes per
endpoint.
- The new logging framework makes it possible to extend, customize,
and filter logs very easily. See the :doc:`logging framework <logging>`
for more information on usage.
- A common pattern found in the new scripts is to store logging stream
records for protocols inside the ``connection`` records so that
state can be collected until enough is seen to log a coherent unit
of information regarding the activity of that connection. This
state is now frequently seen/accessible in event handlers, for
example, like ``c$<protocol>`` where ``<protocol>`` is replaced by
the name of the protocol. This field is added to the ``connection``
record by ``redef``'ing it in a
``base/protocols/<protocol>/main.bro`` script.
- The logging code has been rewritten internally, with script-level
interface and output backend now clearly separated. While ASCII
logging is still the default, we will add further output types in
the future (binary format, direct database logging).
Notice Framework
----------------
The way users interact with "notices" has changed significantly in
order to make it easier to define a site policy and more extensible
for adding customized actions. See the :doc:`notice framework <notice>`.
New Default Settings
--------------------
- Dynamic Protocol Detection (DPD) is now enabled/loaded by default.
- The default packet filter now examines all packets instead of
dynamically building a filter based on which protocol analysis scripts
are loaded. See ``PacketFilter::all_packets`` for how to revert to old
behavior.
API Changes
-----------
- The ``@prefixes`` directive works differently now.
Any added prefixes are now searched for and loaded *after* all input
files have been parsed. After all input files are parsed, Bro
searches ``BROPATH`` for prefixed, flattened versions of all of the
parsed input files. For example, if ``lcl`` is in ``@prefixes``, and
``site.bro`` is loaded, then a file named ``lcl.site.bro`` that's in
``BROPATH`` would end up being automatically loaded as well. Packages
work similarly, e.g. loading ``protocols/http`` means a file named
``lcl.protocols.http.bro`` in ``BROPATH`` gets loaded automatically.
- The ``make_addr`` BIF now returns a ``subnet`` versus an ``addr``
Variable Naming
---------------
- ``Module`` is more widely used for namespacing. E.g. the new
``site.bro`` exports the ``local_nets`` identifier (among other
things) into the ``Site`` module.
- Identifiers may have been renamed to conform to new `scripting
conventions
<http://www.bro.org/development/script-conventions.html>`_
Removed Functionality
---------------------
We have remove a bunch of functionality that was rarely used and/or
had not been maintained for a while already:
- The ``net`` script data type.
- The ``alarm`` statement; use the notice framework instead.
- Trace rewriting.
- DFA state expiration in regexp engine.
- Active mapping.
- Native DAG support (may come back eventually)
- ClamAV support.
- The connection compressor is now disabled by default, and will
be removed in the future.
BroControl Changes
------------------
BroControl looks pretty much similar to the version coming with Bro 1.x,
but has been cleaned up and streamlined significantly internally.
BroControl has a new ``process`` command to process a trace on disk
offline using a similar configuration to what BroControl installs for
live analysis.
BroControl now has an extensive plugin interface for adding new
commands and options. Note that this is still considered experimental.
We have removed the ``analysis`` command, and BroControl currently
does not send daily alarm summaries anymore (this may be restored
later).
Development Infrastructure
--------------------------
Bro development has moved from using SVN to Git for revision control.
Users that want to use the latest Bro development snapshot by checking it out
from the source repositories should see the `development process
<http://www.bro.org/development/process.html>`_. Note that all the various
sub-components now reside in their own repositories. However, the
top-level Bro repository includes them as git submodules so it's easy
to check them all out simultaneously.
Bro now uses `CMake <http://www.cmake.org>`_ for its build system so
that is a new required dependency when building from source.
Bro now comes with a growing suite of regression tests in
``testing/``.

2
VERSION
View file

@ -1 +1 @@
2.1-1216
2.1-1306

2
aux/btest

@ -1 +1 @@
Subproject commit 9726c5b982ee0e3e730a15dcd65f49bdee3fe458
Subproject commit 55e2f6c5fce254e822ff98cfd117e40f89c4c30c

1
doc/CHANGES
View file

@ -1 +0,0 @@
../CHANGES

14
doc/CMakeLists.txt
View file

@ -21,17 +21,17 @@ add_subdirectory(scripts)
add_custom_target(broxygen
# copy the template documentation to the build directory
# to give as input for sphinx
COMMAND "${CMAKE_COMMAND}" -E copy_directory
${DOC_SOURCE_DIR}
COMMAND rsync -r --copy-links --times
${DOC_SOURCE_DIR}/
${DOC_SOURCE_WORKDIR}
# copy generated policy script documentation into the
# working copy of the template documentation
COMMAND "${CMAKE_COMMAND}" -E copy_directory
${RST_OUTPUT_DIR}
COMMAND rsync -r --copy-links --times
${RST_OUTPUT_DIR}/
${DOC_SOURCE_WORKDIR}/scripts
# append to the master index of all policy scripts
COMMAND cat ${MASTER_POLICY_INDEX} >>
${DOC_SOURCE_WORKDIR}/scripts/index.rst
${DOC_SOURCE_WORKDIR}/scripts/scripts.rst
# append to the master index of all policy packages
COMMAND cat ${MASTER_PACKAGE_INDEX} >>
${DOC_SOURCE_WORKDIR}/scripts/packages.rst
@ -58,7 +58,7 @@ add_custom_target(broxygen
# SOURCES just adds stuff to IDE projects as a convenience
SOURCES ${DOC_SOURCES})
# The "sphinxclean" target removes just the Sphinx input/output directories
# The "broxygenclean" target removes just the Sphinx input/output directories
# from the build directory.
add_custom_target(broxygenclean
COMMAND "${CMAKE_COMMAND}" -E remove_directory
@ -67,7 +67,7 @@ add_custom_target(broxygenclean
${DOC_OUTPUT_DIR}
VERBATIM)
add_dependencies(broxygen broxygenclean restdoc)
add_dependencies(broxygen restdoc)
add_custom_target(doc)
add_custom_target(docclean)

1
doc/INSTALL.rst
View file

@ -1 +0,0 @@
../INSTALL

5
doc/LICENSE Normal file
View file

@ -0,0 +1,5 @@
This work is licensed under the Creative Commons
Attribution-NonCommercial 3.0 Unported License. To view a copy of this
license, visit http://creativecommons.org/licenses/by-nc/3.0/ or send
a letter to Creative Commons, 444 Castro Street, Suite 900, Mountain
View, California, 94041, USA.

38
doc/bin/group_index_generator.py
View file

@ -8,9 +8,11 @@
# 2nd argument is the directory containing ${group}_files lists of
# scripts that belong to the group and ${group}_doc_names lists of
# document names that can be supplied to a reST :doc: role
# 3rd argument is a directory in which write a ${group}.rst file (will
# append to existing file) that contains reST style references to
# script docs along with summary text contained in original script
# 3rd argument is a directory in which write a ${group}.rst file that contains
# reST style references to script docs along with summary text contained
# in original script. If ${group} ends with "index", then the file
# is always clobbered by this script, but for other unique group names,
# this script will append to existing files.
import sys
import os
@ -20,9 +22,28 @@ group_list = sys.argv[1]
file_manifest_dir = sys.argv[2]
output_dir = sys.argv[3]
def make_group_file_index(dir_name, group_name):
group_file = os.path.join(dir_name, group_name + ".rst")
if not os.path.exists(group_file):
if not os.path.exists(os.path.dirname(group_file)):
os.makedirs(os.path.dirname(group_file))
if group_name.endswith("index"):
with open(group_file, 'w') as f_group_file:
f_group_file.write(":orphan:\n\n")
title = "Package Index: %s\n" % os.path.dirname(group_name)
f_group_file.write(title);
for n in range(len(title)):
f_group_file.write("=")
f_group_file.write("\n");
return group_file
with open(group_list, 'r') as f_group_list:
for group in f_group_list.read().splitlines():
#print group
group_file = make_group_file_index(output_dir, group)
file_manifest = os.path.join(file_manifest_dir, group + "_files")
doc_manifest = os.path.join(file_manifest_dir, group + "_doc_names")
src_files = []
@ -44,17 +65,6 @@ with open(group_list, 'r') as f_group_list:
if sum_pos != -1:
summary_comments.append(line[(sum_pos+3):])
#print summary_comments
group_file = os.path.join(output_dir, group + ".rst")
if not os.path.exists(group_file):
if not os.path.exists(os.path.dirname(group_file)):
os.makedirs(os.path.dirname(group_file))
with open(group_file, 'w') as f_group_file:
f_group_file.write(":orphan:\n\n")
title = "Package Index: %s\n" % os.path.dirname(group)
f_group_file.write(title);
for n in range(len(title)):
f_group_file.write("=")
f_group_file.write("\n");
with open(group_file, 'a') as f_group_file:
f_group_file.write("\n:doc:`/scripts/%s`\n" % doc_names[i])

8
doc/cluster.rst → doc/cluster/index.rst
View file

@ -1,5 +1,7 @@
Bro Cluster
===========
========================
Setting up a Bro Cluster
========================
Intro
------
@ -11,7 +13,7 @@ Architecture
The figure below illustrates the main components of a Bro cluster.
.. image:: images/deployment.png
.. image:: /images/deployment.png
Tap
***

1
doc/components/binpac/README.rst
View file

@ -1 +0,0 @@
../../../aux/binpac/README

68
doc/components/binpac/README.rst Normal file
View file

@ -0,0 +1,68 @@
.. -*- mode: rst-mode -*-
..
.. Version number is filled in automatically.
.. |version| replace:: 0.34-3
======
BinPAC
======
.. rst-class:: opening
BinPAC is a high level language for describing protocol parsers and
generates C++ code. It is currently maintained and distributed with the
Bro Network Security Monitor distribution, however, the generated parsers
may be used with other programs besides Bro.
Download
--------
You can find the latest BinPAC release for download at
http://www.bro.org/download.
BinPAC's git repository is located at `git://git.bro.org/binpac.git
<git://git.bro.org/binpac.git>`__. You can browse the repository
`here <http://git.bro.org/binpac.git>`__.
This document describes BinPAC |version|. See the ``CHANGES``
file for version history.
Prerequisites
-------------
BinPAC relies on the following libraries and tools, which need to be
installed before you begin:
* Flex (Fast Lexical Analyzer)
Flex is already installed on most systems, so with luck you can
skip having to install it yourself.
* Bison (GNU Parser Generator)
Bison is also already installed on many system.
* CMake 2.6.3 or greater
CMake is a cross-platform, open-source build system, typically
not installed by default. See http://www.cmake.org for more
information regarding CMake and the installation steps below for
how to use it to build this distribution. CMake generates native
Makefiles that depend on GNU Make by default
Installation
------------
To build and install into ``/usr/local``::
./configure
cd build
make
make install
This will perform an out-of-source build into the build directory using
the default build options and then install the binpac binary into
``/usr/local/bin``.
You can specify a different installation directory with::
./configure --prefix=<dir>
Run ``./configure --help`` for more options.

1
doc/components/bro-aux/README.rst
View file

@ -1 +0,0 @@
../../../aux/bro-aux/README

70
doc/components/bro-aux/README.rst Normal file
View file

@ -0,0 +1,70 @@
.. -*- mode: rst; -*-
..
.. Version number is filled in automatically.
.. |version| replace:: 0.26-5
======================
Bro Auxiliary Programs
======================
.. contents::
:Version: |version|
Handy auxiliary programs related to the use of the Bro Network Security
Monitor (http://www.bro.org).
Note that some files that were formerly distributed with Bro as part
of the aux/ tree are now maintained separately. See the
http://www.bro.org/download for their download locations.
adtrace
=======
Makefile and source for the adtrace utility. This program is used
in conjunction with the localnetMAC.pl perl script to compute the
network address that compose the internal and extern nets that bro
is monitoring. This program when run by itself just reads a pcap
(tcpdump) file and writes out the src MAC, dst MAC, src IP, dst
IP for each packet seen in the file. This output is processed by
the localnetMAC.pl script during 'make install'.
devel-tools
===========
A set of scripts used commonly for Bro development.
extract-conn-by-uid:
Extracts a connection from a trace file based
on its UID found in Bro's conn.log
gen-mozilla-ca-list.rb
Generates list of Mozilla SSL root certificates in
a format readable by Bro.
update-changes
A script to maintain the CHANGES and VERSION files.
git-show-fastpath
Show commits to the fastpath branch not yet merged into master.
cpu-bench-with-trace
Run a number of Bro benchmarks on a trace file.
nftools
=======
Utilities for dealing with Bro's custom file format for storing
NetFlow records. nfcollector reads NetFlow data from a socket and
writes it in Bro's format. ftwire2bro reads NetFlow "wire" format
(e.g., as generated by a 'flow-export' directive) and writes it in
Bro's format.
rst
===
Makefile and source for the rst utility. "rst" can be invoked by
a Bro script to terminate an established TCP connection by forging
RST tear-down packets. See terminate_connection() in conn.bro.

1
doc/components/broccoli-python/README.rst
View file

@ -1 +0,0 @@
../../../aux/broccoli/bindings/broccoli-python/README

231
doc/components/broccoli-python/README.rst Normal file
View file

@ -0,0 +1,231 @@
.. -*- mode: rst-mode -*-
..
.. Version number is filled in automatically.
.. |version| replace:: 0.54
============================
Python Bindings for Broccoli
============================
.. rst-class:: opening
This Python module provides bindings for Broccoli, Bro's client
communication library. In general, the bindings provide the same
functionality as Broccoli's C API.
.. contents::
Download
--------
You can find the latest Broccoli-Python release for download at
http://www.bro.org/download.
Broccoli-Python's git repository is located at `git://git.bro.org/broccoli-python.git
<git://git.bro.org/broccoli-python.git>`__. You can browse the repository
`here <http://git.bro.org/broccoli-python.git>`__.
This document describes Broccoli-Python |version|. See the ``CHANGES``
file for version history.
Installation
------------
Installation of the Python module is pretty straight-forward. After
Broccoli itself has been installed, it follows the standard installation
process for Python modules::
python setup.py install
Try the following to test the installation. If you do not see any
error message, everything should be fine::
python -c "import broccoli"
Usage
-----
The following examples demonstrate how to send and receive Bro
events in Python.
The main challenge when using Broccoli from Python is dealing with
the data types of Bro event parameters as there is no one-to-one
mapping between Bro's types and Python's types. The Python modules
automatically maps between those types which both systems provide
(such as strings) and provides a set of wrapper classes for Bro
types which do not have a direct Python equivalent (such as IP
addresses).
Connecting to Bro
~~~~~~~~~~~~~~~~~
The following code sets up a connection from Python to a remote Bro
instance (or another Broccoli) and provides a connection handle for
further communication::
from broccoli import *
bc = Connection("127.0.0.1:47758")
An ``IOError`` will be raised if the connection cannot be established.
Sending Events
~~~~~~~~~~~~~~
Once you have a connection handle ``bc`` set up as shown above, you can
start sending events::
bc.send("foo", 5, "attack!")
This sends an event called ``foo`` with two parameters, ``5`` and
``attack!``. Broccoli operates asynchronously, i.e., events scheduled
with ``send()`` are not always sent out immediately but might be
queued for later transmission. To ensure that all events get out
(and incoming events are processed, see below), you need to call
``bc.processInput()`` regularly.
Data Types
~~~~~~~~~~
In the example above, the types of the event parameters are
automatically derived from the corresponding Python types: the first
parameter (``5``) has the Bro type ``int`` and the second one
(``attack!``) has Bro type ``string``.
For types which do not have a Python equivalent, the ``broccoli``
module provides wrapper classes which have the same names as the
corresponding Bro types. For example, to send an event called ``bar``
with one ``addr`` argument and one ``count`` argument, you can write::
bc.send("bar", addr("192.168.1.1"), count(42))
The following table summarizes the available atomic types and their
usage.
======== =========== ===========================
Bro Type Python Type Example
======== =========== ===========================
addr ``addr("192.168.1.1")``
bool bool ``True``
count ``count(42)``
double float ``3.14``
enum Type currently not supported
int int ``5``
interval ``interval(60)``
net Type currently not supported
port ``port("80/tcp")``
string string ``"attack!"``
subnet ``subnet("192.168.1.0/24")``
time ``time(1111111111.0)``
======== =========== ===========================
The ``broccoli`` module also supports sending Bro records as event
parameters. To send a record, you first define a record type. For
example, a Bro record type::
type my_record: record {
a: int;
b: addr;
c: subnet;
};
turns into Python as::
my_record = record_type("a", "b", "c")
As the example shows, Python only needs to know the attribute names
but not their types. The types are derived automatically in the same
way as discussed above for atomic event parameters.
Now you can instantiate a record instance of the newly defined type
and send it out::
rec = record(my_record)
rec.a = 5
rec.b = addr("192.168.1.1")
rec.c = subnet("192.168.1.0/24")
bc.send("my_event", rec)
.. note:: The Python module does not support nested records at this time.
Receiving Events
~~~~~~~~~~~~~~~~
To receive events, you define a callback function having the same
name as the event and mark it with the ``event`` decorator::
@event
def foo(arg1, arg2):
print arg1, arg2
Once you start calling ``bc.processInput()`` regularly (see above),
each received ``foo`` event will trigger the callback function.
By default, the event's arguments are always passed in with built-in
Python types. For Bro types which do not have a direct Python
equivalent (see table above), a substitute built-in type is used
which corresponds to the type the wrapper class' constructor expects
(see the examples in the table). For example, Bro type ``addr`` is
passed in as a string and Bro type ``time`` is passed in as a float.
Alternatively, you can define a _typed_ prototype for the event. If you
do so, arguments will first be type-checked and then passed to the
call-back with the specified type (which means instances of the
wrapper classes for non-Python types). Example::
@event(count, addr)
def bar(arg1, arg2):
print arg1, arg2
Here, ``arg1`` will be an instance of the ``count`` wrapper class and
``arg2`` will be an instance of the ``addr`` wrapper class.
Protoyping works similarly with built-in Python types::
@event(int, string):
def foo(arg1, arg2):
print arg1, arg2
In general, the prototype specifies the types in which the callback
wants to receive the arguments. This actually provides support for
simple type casts as some types support conversion to into something
different. If for instance the event source sends an event with a
single port argument, ``@event(port)`` will pass the port as an
instance of the ``port`` wrapper class; ``@event(string)`` will pass it
as a string (e.g., ``"80/tcp"``); and ``@event(int)`` will pass it as an
integer without protocol information (e.g., just ``80``). If an
argument cannot be converted into the specified type, a ``TypeError``
will be raised.
To receive an event with a record parameter, the record type first
needs to be defined, as described above. Then the type can be used
with the ``@event`` decorator in the same way as atomic types::
my_record = record_type("a", "b", "c")
@event(my_record)
def my_event(rec):
print rec.a, rec.b, rec.c
Helper Functions
----------------
The ``broccoli`` module provides one helper function: ``current_time()``
returns the current time as a float which, if necessary, can be
wrapped into a ``time`` parameter (i.e., ``time(current_time()``)
Examples
--------
There are some example scripts in the ``tests/`` subdirectory of the
``broccoli-python`` repository
`here <http://git.bro.org/broccoli-python.git/tree/HEAD:/tests>`_:
- ``broping.py`` is a (simplified) Python version of Broccoli's test program
``broping``. Start Bro with ``broping.bro``.
- ``broping-record.py`` is a Python version of Broccoli's ``broping``
for records. Start Bro with ``broping-record.bro``.
- ``test.py`` is a very ugly but comprehensive regression test and part of
the communication test-suite. Start Bro with ``test.bro``.

1
doc/components/broccoli-ruby/README.rst
View file

@ -1 +0,0 @@
../../../aux/broccoli/bindings/broccoli-ruby/README

67
doc/components/broccoli-ruby/README.rst Normal file
View file

@ -0,0 +1,67 @@
.. -*- mode: rst-mode -*-
..
.. Version number is filled in automatically.
.. |version| replace:: 1.54
===============================================
Ruby Bindings for Broccoli
===============================================
.. rst-class:: opening
This is the broccoli-ruby extension for Ruby which provides access
to the Broccoli API. Broccoli is a library for
communicating with the Bro Intrusion Detection System.
Download
========
You can find the latest Broccoli-Ruby release for download at
http://www.bro.org/download.
Broccoli-Ruby's git repository is located at `git://git.bro.org/broccoli-ruby.git
<git://git.bro.org/broccoli-ruby.git>`__. You can browse the repository
`here <http://git.bro.org/broccoli-ruby.git>`__.
This document describes Broccoli-Ruby |version|. See the ``CHANGES``
file for version history.
Installation
============
To install the extension:
1. Make sure that the ``broccoli-config`` binary is in your path.
(``export PATH=/usr/local/bro/bin:$PATH``)
2. Run ``sudo ruby setup.rb``.
To install the extension as a gem (suggested):
1. Install `rubygems <http://rubygems.org>`_.
2. Make sure that the ``broccoli-config`` binary is in your path.
(``export PATH=/usr/local/bro/bin:$PATH``)
3. Run, ``sudo gem install rbroccoli``.
Usage
=====
There aren't really any useful docs yet. Your best bet currently is
to read through the examples.
One thing I should mention however is that I haven't done any optimization
yet. You may find that if you write code that is going to be sending or
receiving extremely large numbers of events, that it won't run fast enough and
will begin to fall behind the Bro server. The dns_requests.rb example is
a good performance test if your Bro server is sitting on a network with many
dns lookups.
Contact
=======
If you have a question/comment/patch, see the Bro `contact page
<http://www.bro.org/contact/index.html>`_.

1
doc/components/broccoli/README.rst
View file

@ -1 +0,0 @@
../../../aux/broccoli/README

141
doc/components/broccoli/README.rst Normal file
View file

@ -0,0 +1,141 @@
.. -*- mode: rst-mode -*-
..
.. Version number is filled in automatically.
.. |version| replace:: 1.92-9
===============================================
Broccoli: The Bro Client Communications Library
===============================================
.. rst-class:: opening
Broccoli is the "Bro client communications library". It allows you
to create client sensors for the Bro intrusion detection system.
Broccoli can speak a good subset of the Bro communication protocol,
in particular, it can receive Bro IDs, send and receive Bro events,
and send and receive event requests to/from peering Bros. You can
currently create and receive values of pure types like integers,
counters, timestamps, IP addresses, port numbers, booleans, and
strings.
Download
--------
You can find the latest Broccoli release for download at
http://www.bro.org/download.
Broccoli's git repository is located at
`git://git.bro.org/broccoli <git://git.bro.org/broccoli>`_. You
can browse the repository `here <http://git.bro.org/broccoli>`_.
This document describes Broccoli |version|. See the ``CHANGES``
file for version history.
Installation
------------
The Broccoli library has been tested on Linux, the BSDs, and Solaris.
A Windows build has not currently been tried but is part of our future
plans. If you succeed in building Broccoli on other platforms, let us
know!
Prerequisites
-------------
Broccoli relies on the following libraries and tools, which need to be
installed before you begin:
Flex (Fast Lexical Analyzer)
Flex is already installed on most systems, so with luck you
can skip having to install it yourself.
Bison (GNU Parser Generator)
This comes with many systems, but if you get errors compiling
parse.y, you will need to install it.
OpenSSL headers and libraries
For encrypted communication. These are likely installed,
though some platforms may require installation of a 'devel'
package for the headers.
CMake 2.6.3 or greater
CMake is a cross-platform, open-source build system, typically
not installed by default. See http://www.cmake.org for more
information regarding CMake and the installation steps below
for how to use it to build this distribution. CMake generates
native Makefiles that depend on GNU Make by default.
Broccoli can also make use of some optional libraries if they are found at
installation time:
Libpcap headers and libraries
Network traffic capture library
Installation
------------
To build and install into ``/usr/local``::
./configure
make
make install
This will perform an out-of-source build into the build directory using the
default build options and then install libraries into ``/usr/local/lib``.
You can specify a different installation directory with::
./configure --prefix=<dir>
Or control the python bindings install destination more precisely with::
./configure --python-install-dir=<dir>
Run ``./configure --help`` for more options.
Further notable configure options:
``--enable-debug``
This one enables lots of debugging output. Be sure to disable
this when using the library in a production environment! The
output could easily end up in undersired places when the stdout
of the program you've instrumented is used in other ways.
``--with-configfile=FILE``
Broccoli can read key/value pairs from a config file. By default
it is located in the etc directory of the installation root
(exception: when using ``--prefix=/usr``, ``/etc`` is used
instead of /usr/etc). The default config file name is
broccoli.conf. Using ``--with-configfile``, you can override the
location and name of the config file.
To use the library in other programs & configure scripts, use the
``broccoli-config`` script. It gives you the necessary configuration flags
and linker flags for your system, see ``--cflags`` and ``--libs``.
The API is contained in broccoli.h and pretty well documented. A few
usage examples can be found in the test directory, in particular, the
``broping`` tool can be used to test event transmission and reception. Have
a look at the policy file ``broping.bro`` for the events that need to be
defined at the peering Bro. Try ``broping -h`` for a look at the available
options.
Broccoli knows two kinds of version numbers: the release version number
(as in "broccoli-x.y.tar.gz", or as shipped with Bro) and the shared
library API version number (as in libbroccoli.so.3.0.0). The former
relates to changes in the tree, the latter to compatibility changes in
the API.
Comments, feedback and patches are appreciated; please check the `Bro
website <http://www.bro.org/community>`_.
Documentation
-------------
Please see the `Broccoli User Manual <./broccoli-manual.html>`_ and
the `Broccoli API Reference <../../broccoli-api/index.html>`_.

1
doc/components/broccoli/broccoli-manual.rst
View file

@ -1 +0,0 @@
../../../aux/broccoli/doc/broccoli-manual.rst

1355
doc/components/broccoli/broccoli-manual.rst Normal file
View file

File diff suppressed because it is too large Load diff

1
doc/components/broctl/README.rst
View file

@ -1 +0,0 @@
../../../aux/broctl/doc/broctl.rst

1913
doc/components/broctl/README.rst Normal file
View file

File diff suppressed because it is too large Load diff

1
doc/components/btest/README.rst
View file

@ -1 +0,0 @@
../../../aux/btest/README

843
doc/components/btest/README.rst Normal file
View file

@ -0,0 +1,843 @@
.. -*- mode: rst-mode -*-
..
.. Version number is filled in automatically.
.. |version| replace:: 0.4-14
============================================
BTest - A Simple Driver for Basic Unit Tests
============================================
.. rst-class:: opening
The ``btest`` is a simple framework for writing unit tests. Freely
borrowing some ideas from other packages, it's main objective is to
provide an easy-to-use, straightforward driver for a suite of
shell-based tests. Each test consists of a set of command lines that
will be executed, and success is determined based on their exit
codes. ``btest`` comes with some additional tools that can be used
within such tests to compare output against a previously established
baseline.
.. contents::
Download
========
You can find the latest BTest release for download at
http://www.bro.org/download.
BTest's git repository is located at `git://git.bro.org/btest.git
<git://git.bro.org/btest.git>`__. You can browse the repository
`here <http://git.bro.org/btest.git>`__.
This document describes BTest |version|. See the ``CHANGES``
file for version history.
Installation
============
Installation is simple and standard::
tar xzvf btest-*.tar.gz
cd btest-*
python setup.py install
This will install a few scripts: ``btest`` is the main driver program,
and there are a number of further helper scripts that we discuss below
(including ``btest-diff``, which is a tool for comparing output to a
previously established baseline).
Writing a Simple Test
=====================
In the most simple case, ``btest`` simply executes a set of command
lines, each of which must be prefixed with ``@TEST-EXEC:``
::
> cat examples/t1
@TEST-EXEC: echo "Foo" | grep -q Foo
@TEST-EXEC: test -d .
> btest examples/t1
examples.t1 ... ok
The test passes as both command lines return success. If one of them
didn't, that would be reported::
> cat examples/t2
@TEST-EXEC: echo "Foo" | grep -q Foo
@TEST-EXEC: test -d DOESNOTEXIST
> btest examples/t2
examples.t2 ... failed
Usually you will just run all tests found in a directory::
> btest examples
examples.t1 ... ok
examples.t2 ... failed
1 test failed
Why do we need the ``@TEST-EXEC:`` prefixes? Because the file
containing the test can simultaneously act as *its input*. Let's
say we want to verify a shell script::
> cat examples/t3.sh
# @TEST-EXEC: sh %INPUT
ls /etc | grep -q passwd
> btest examples/t3.sh
examples.t3 ... ok
Here, ``btest`` is executing (something similar to) ``sh
examples/t3.sh``, and then checks the return value as usual. The
example also shows that the ``@TEST-EXEC`` prefix can appear
anywhere, in particular inside the comment section of another
language.
Now, let's say we want to check the output of a program, making sure
that it matches what we expect. For that, we first add a command
line to the test that produces the output we want to check, and then
run ``btest-diff`` to make sure it matches a previously recorded
baseline. ``btest-diff`` is itself just a script that returns
success if the output is as expected, and failure otherwise. In the
following example, we use an awk script as a fancy way to print all
file names starting with a dot in the user's home directory. We
write that list into a file called ``dots`` and then check whether
its content matches what we know from last time::
> cat examples/t4.awk
# @TEST-EXEC: ls -a $HOME | awk -f %INPUT >dots
# @TEST-EXEC: btest-diff dots
/^\.+/ { print $1 }
Note that each test gets its own little sandbox directory when run,
so by creating a file like ``dots``, you aren't cluttering up
anything.
The first time we run this test, we need to record a baseline::
> btest -U examples/t4.awk
Now, ``btest-diff`` has remembered what the ``dots`` file should
look like::
> btest examples/t4.awk
examples.t4 ... ok
> touch ~/.NEWDOTFILE
> btest examples/t4.awk
examples.t4 ... failed
1 test failed
If we want to see what exactly the unexpected change is that was
introduced to ``dots``, there's a *diff* mode for that::
> btest -d examples/t4.awk
examples.t4 ... failed
% 'btest-diff dots' failed unexpectedly (exit code 1)
% cat .diag
== File ===============================
[... current dots file ...]
== Diff ===============================
--- /Users/robin/work/binpacpp/btest/Baseline/examples.t4/dots
2010-10-28 20:11:11.000000000 -0700
+++ dots 2010-10-28 20:12:30.000000000 -0700
@@ -4,6 +4,7 @@
.CFUserTextEncoding
.DS_Store
.MacOSX
+.NEWDOTFILE
.Rhistory
.Trash
.Xauthority
=======================================
% cat .stderr
[... if any of the commands had printed something to stderr, that would follow here ...]
Once we delete the new file, we are fine again::
> rm ~/.NEWDOTFILE
> btest -d examples/t4.awk
examples.t4 ... ok
That's already the main functionality that the ``btest`` package
provides. In the following, we describe a number of further options
extending/modifying this basic approach.
Reference
=========
Command Line Usage
------------------
``btest`` must be started with a list of tests and/or directories
given on the command line. In the latter case, the default is to
recursively scan the directories and assume all files found to be
tests to perform. It is however possible to exclude certain files by
specifying a suitable `configuration file`_.
``btest`` returns exit code 0 if all tests have successfully passed,
and 1 otherwise.
``btest`` accepts the following options:
-a ALTERNATIVE, --alternative=ALTERNATIVE
Activates an alternative_ configuration defined in the
configuration file. This option can be given multiple times to
run tests with several alternatives. If ``ALTERNATIVE`` is ``-``
that refers to running with the standard setup, which can be used
to run tests both with and without alterantives by giving both.
-b, --brief
Does not output *anything* for tests which pass. If all tests
pass, there will not be any output at all.
-c CONFIG, --config=CONFIG
Specifies an alternative `configuration file`_ to use. If not
specified, the default is to use a file called ``btest.cfg``
if found in the current directory.
-d, --diagnostics
Reports diagnostics for all failed tests. The diagnostics
include the command line that failed, its output to standard
error, and potential additional information recorded by the
command line for diagnostic purposes (see `@TEST-EXEC`_
below). In the case of ``btest-diff``, the latter is the
``diff`` between baseline and actual output.
-D, --diagnostics-all
Reports diagnostics for all tests, including those which pass.
-f DIAGFILE, --file-diagnostics=DIAGFILE
Writes diagnostics for all failed tests into the given file.
If the file already exists, it will be overwritten.
-g GROUPS, --group=GROUPS
Runs only tests assigned to the given test groups, see
`@TEST-GROUP`_. Multiple groups can be given as a
comma-separated list. Specifying ``-`` as a group name selects
all tests that do not belong to any group.
-j [THREADS], --jobs[=THREADS]
Runs up to the given number of tests in parallel. If no number
is given, BTest substitutes the number of available CPU cores
as reported by the OS.
By default, BTest assumes that all tests can be executed
concurrently without further constraints. One can however
ensure serialization of subsets by assigning them to the same
serialization set, see `@TEST-SERIALIZE`_.
-q, --quiet
Suppress information output other than about failed tests.
If all tests pass, there will not be any output at all.
-r, --rerun
Runs only tests that failed last time. After each execution
(except when updating baselines), BTest generates a state file
that records the tests that have failed. Using this option on
the next run then reads that file back in and limits execution
to those tests found in there.
-t, --tmp-keep
Does not delete any temporary files created for running the
tests (including their outputs). By default, the temporary
files for a test will be located in ``.tmp/<test>/``, where
``<test>`` is the relative path of the test file with all slashes
replaced with dots and the file extension removed (e.g., the files
for ``example/t3.sh`` will be in ``.tmp/example.t3``).
-U, --update-baseline
Records a new baseline for all ``btest-diff`` commands found
in any of the specified tests. To do this, all tests are run
as normal except that when ``btest-diff`` is executed, it
does not compute a diff but instead considers the given file
to be authoritative and records it as the version to compare
with in future runs.
-u, --update-interactive
Each time a ``btest-diff`` command fails in any tests that are
run, btest will stop and ask whether or not the user wants to
record a new baseline.
-v, --verbose
Shows all test command lines as they are executed.
-w, --wait
Interactively waits for ``<enter>`` after showing diagnostics
for a test.
-x FILE, --xml=FILE
Records test results in JUnit XML format to the given file.
If the file exists already, it is overwritten.
.. _configuration file:
Configuration
-------------
Specifics of ``btest``'s execution can be tuned with a configuration
file, which by default is ``btest.cfg`` if that's found in the
current directory. It can alternatively be specified with the
``--config`` command line option. The configuration file is
"INI-style", and an example comes with the distribution, see
``btest.cfg.example``. A configuration file has one main section,
``btest``, that defines most options; as well as an optional section
for defining `environment variables`_ and further optional sections
for defining alternatives_.
Note that all paths specified in the configuration file are relative
to ``btest``'s *base directory*. The base directory is either the
one where the configuration file is located if such is given/found,
or the current working directory if not. When setting values for
configuration options, the absolute path to the base directory is
available by using the macro ``%(testbase)s`` (the weird syntax is
due to Python's ``ConfigParser`` module).
Furthermore, all values can use standard "backtick-syntax" to
include the output of external commands (e.g., xyz=`\echo test\`).
Note that the backtick expansion is performed after any ``%(..)``
have already been replaced (including within the backticks).
Options
~~~~~~~
The following options can be set in the ``btest`` section of the
configuration file:
``TestDirs``
A space-separated list of directories to search for tests. If
defined, one doesn't need to specify any tests on the command
line.
``TmpDir``
A directory where to create temporary files when running tests.
By default, this is set to ``%(testbase)s/.tmp``.
``BaselineDir``
A directory where to store the baseline files for ``btest-diff``.
By default, this is set to ``%(testbase)s/Baseline``.
``IgnoreDirs``
A space-separated list of relative directory names to ignore
when scanning test directories recursively. Default is empty.
``IgnoreFiles``
A space-separated list of filename globs matching files to
ignore when scanning given test directories recursively.
Default is empty.
``StateFile``
The name of the state file to record the names of failing tests. Default is
``.btest.failed.dat``.
``Finalizer``
An executable that will be executed each time any test has
successfully run. It runs in the same directory as the test itself
and receives the name of the test as its parameter. The return
value indicates whether the test should indeed be considered
successful. By default, there's no finalizer set.
.. _environment variables:
Environment Variables
~~~~~~~~~~~~~~~~~~~~~
A special section ``environment`` defines environment variables that
will be propagated to all tests::
[environment]
CFLAGS=-O3
PATH=%(testbase)s/bin:%(default_path)s
Note how ``PATH`` can be adjusted to include local scripts: the
example above prefixes it with a local ``bin/`` directory inside the
base directory, using the predefined ``default_path`` macro to refer
to the ``PATH`` as it is set by default.
Furthermore, by setting ``PATH`` to include the ``btest``
distribution directory, one could skip the installation of the
``btest`` package.
.. _alternative:
Alternatives
~~~~~~~~~~~~
BTest can run a set of tests with different settings than it would
normally use by specifying an *alternative* configuration. Currently,
three things can be adjusted:
- Further environment variables can be set that will then be
available to all the commands that a test executes.
- *Filters* can modify an input file before a test uses it.
- *Substitutions* can modify command lines executed as part of a
test.
We discuss the three separately in the following. All of them are
defined by adding sections ``[<type>-<name>]`` where ``<type>``
corresponds to the type of adjustment being made and ``<name>`` is the
name of the alternative. Once at least one section is defined for a
name, that alternative can be enabled by BTest's ``--alternative``
flag.
Environment Variables
^^^^^^^^^^^^^^^^^^^^^
An alternative can add further environment variables by defining an
``[environment-<name>]`` section:
[environment-myalternative]
CFLAGS=-O3
Running ``btest`` with ``--alternative=myalternative`` will now make
the ``CFLAGS`` environment variable available to all commands
executed.
.. _filters:
Filters
^^^^^^^
Filters are a transparent way to adapt the input to a specific test
command before it is executed. A filter is defined by adding a section
``[filter-<name>]`` to the configuration file. This section must have
exactly one entry, and the name of that entry is interpreted as the
name of a command whose input is to be filtered. The value of that
entry is the name of a filter script that will be run with two
arguments representing input and output files, respectively. Example::
[filter-myalternative]
cat=%(testbase)s/bin/filter-cat
Once the filter is activated by running ``btest`` with
``--alternative=myalternative``, every time a ``@TEST-EXEC: cat
%INPUT`` is found, ``btest`` will first execute (something similar to)
``%(testbase)s/bin/filter-cat %INPUT out.tmp``, and then subsequently
``cat out.tmp`` (i.e., the original command but with the filtered
output). In the simplest case, the filter could be a no-op in the
form ``cp $1 $2``.
.. note::
There are a few limitations to the filter concept currently:
* Filters are *always* fed with ``%INPUT`` as their first
argument. We should add a way to filter other files as well.
* Filtered commands are only recognized if they are directly
starting the command line. For example, ``@TEST-EXEC: ls | cat
>outout`` would not trigger the example filter above.
* Filters are only executed for ``@TEST-EXEC``, not for
``@TEST-EXEC-FAIL``.
.. _substitution:
Substitutions
^^^^^^^^^^^^^^
Substitutions are similar to filters, yet they do not adapt the input
but the command line being executed. A substitution is defined by
adding a section ``[substitution-<name>]`` to the configuration file.
For each entry in this section, the entry's name specifies the
command that is to be replaced with something else given as its value.
Example::
[substitution-myalternative]
gcc=gcc -O2
Once the substitution is activated by running ``btest`` with
``--alternative=myalternative``, every time a ``@TEST-EXEC`` executes
``gcc``, that is replaced with ``gcc -O2``. The replacement is simple
string substitution so it works not only with commands but anything
found on the command line; it however only replaces full words, not
subparts of words.
Writing Tests
-------------
``btest`` scans a test file for lines containing keywords that
trigger certain functionality. Currently, the following keywords are
supported:
.. _@TEST-EXEC:
``@TEST-EXEC: <cmdline>``
Executes the given command line and aborts the test if it
returns an error code other than zero. The ``<cmdline>`` is
passed to the shell and thus can be a pipeline, use redirection,
and any environment variables specified in ``<cmdline>`` will be
expanded, etc.
When running a test, the current working directory for all
command lines will be set to a temporary sandbox (and will be
deleted later).
There are two macros that can be used in ``<cmdline>``:
``%INPUT`` will be replaced with the full pathname of the file defining
the test; and ``%DIR`` will be replaced with the directory where
the test file is located. The latter can be used to reference
further files also located there.
In addition to environment variables defined in the
configuration file, there are further ones that are passed into
the commands:
``TEST_DIAGNOSTICS``
A file where further diagnostic information can be saved
in case a command fails. ``--diagnostics`` will show
this file. (This is also where ``btest-diff`` stores its
diff.)
``TEST_MODE``
This is normally set to ``TEST``, but will be ``UPDATE``
if ``btest`` is run with ``--update-baseline``, or
``UPDATE_INTERACTIVE`` if run with ``--update-interactive``.
``TEST_BASELINE``
The name of a directory where the command can save permanent
information across ``btest`` runs. (This is where
``btest-diff`` stores its baseline in ``UPDATE`` mode.)
``TEST_NAME``
The name of the currently executing test.
``TEST_VERBOSE``
The path of a file where the test can record further
information about its execution that will be included with
btest's ``--verbose`` output. This is for further tracking
the execution of commands and should generally generate
output that follows a line-based structure.
.. note::
If a command returns the special exit code 100, the test is
considered failed, however subsequent test commands are still
run. ``btest-diff`` uses this special exit code to indicate that
no baseline has yet been established.
If a command returns the special exit code 200, the test is
considered failed and all further test executions are aborted.
``@TEST-EXEC-FAIL: <cmdline>``
Like ``@TEST-EXEC``, except that this expects the command to
*fail*, i.e., the test is aborted when the return code is zero.
``@TEST-REQUIRES: <cmdline>``
Defines a condition that must be met for the test to be executed.
The given command line will be run before any of the actual test
commands, and it must return success for the test to continue. If
it does not return success, the rest of the test will be skipped
but doing so will not be considered a failure of the test. This allows to
write conditional tests that may not always make sense to run, depending
on whether external constraints are satisfied or not (say, whether
a particular library is available). Multiple requirements may be
specified and then all must be met for the test to continue.
``@TEST-ALTERNATIVE: <alternative>`` Runs this test only for the given
alternative (see alternative_). If ``<alternatives>`` is
``default``, the test executes when BTest runs with no alternative
given (which however is the default anyways).
``@TEST-NOT-ALTERNATIVE: <alternative>`` Ignores this test for the
given alternative (see alternative_). If ``<alternative>`` is
``default``, the test is ignored if BTest runs with no alternative
given.
``@TEST-COPY-FILE: <file>``
Copy the given file into the test's directory before the test is
run. If ``<file>`` is a relative path, it's interpreted relative
to the BTest's base directory. Environment variables in ``<file>``
will be replaced if enclosed in ``${..}``. This command can be
given multiple times.
``@TEST-START-NEXT``
This is a short-cut for defining multiple test inputs in the
same file, all executing with the same command lines. When
``@TEST-START-NEXT`` is encountered, the test file is initially
considered to end at that point, and all ``@TEST-EXEC-*`` are
run with an ``%INPUT`` truncated accordingly. Afterwards, a
*new* ``%INPUT`` is created with everything *following* the
``@TEST-START-NEXT`` marker, and the *same* commands are run
again (further ``@TEST-EXEC-*`` will be ignored). The effect is
that a single file can actually define two tests, and the
``btest`` output will enumerate them::
> cat examples/t5.sh
# @TEST-EXEC: cat %INPUT | wc -c >output
# @TEST-EXEC: btest-diff output
This is the first test input in this file.
# @TEST-START-NEXT
... and the second.
> ./btest -D examples/t5.sh
examples.t5 ... ok
% cat .diag
== File ===============================
119
[...]
examples.t5-2 ... ok
% cat .diag
== File ===============================
22
[...]
Multiple ``@TEST-START-NEXT`` can be used to create more than
two tests per file.
``@TEST-START-FILE <file>``
This is used to include an additional input file for a test
right inside the test file. All lines following the keyword will
be written into the given file (and removed from the test's
`%INPUT`) until a terminating ``@TEST-END-FILE`` is found.
Example::
> cat examples/t6.sh
# @TEST-EXEC: awk -f %INPUT <foo.dat >output
# @TEST-EXEC: btest-diff output
{ lines += 1; }
END { print lines; }
@TEST-START-FILE foo.dat
1
2
3
@TEST-END-FILE
> btest -D examples/t6.sh
examples.t6 ... ok
% cat .diag
== File ===============================
3
Multiple such files can be defined within a single test.
Note that this is only one way to use further input files.
Another is to store a file in the same directory as the test
itself, making sure it's ignored via ``IgnoreFiles``, and then
refer to it via ``%DIR/<name>``.
.. _@TEST-GROUP:
``@TEST-GROUP: <group>``
Assigns the test to a group of name ``<group>``. By using option
``-g`` one can limit execution to all tests that belong to a given
group (or a set of groups).
.. _@TEST-SERIALIZE:
``@TEST-SERIALIZE: <set>``
When using option ``-j`` to parallelize execution, all tests that
specify the same serialization set are guaranteed to run
sequentially. ``<set>`` is an arbitrary user-chosen string.
Canonifying Diffs
=================
``btest-diff`` has the capability to filter its input through an
additional script before it compares the current version with the
baseline. This can be useful if certain elements in an output are
*expected* to change (e.g., timestamps). The filter can then
remove/replace these with something consistent. To enable such
canonification, set the environment variable
``TEST_DIFF_CANONIFIER`` to a script reading the original version
from stdin and writing the canonified version to stdout. Note that
both baseline and current output are passed through the filter
before their differences are computed.
Running Processes in the Background
===================================
Sometimes processes need to be spawned in the background for a test,
in particular if multiple processes need to cooperate in some fashion.
``btest`` comes with two helper scripts to make life easier in such a
situation:
``btest-bg-run <tag> <cmdline>``
This is a script that runs ``<cmdline>`` in the background, i.e.,
it's like using ``cmdline &`` in a shell script. Test execution
continues immediately with the next command. Note that the spawned
command is *not* run in the current directory, but instead in a
newly created sub-directory called ``<tag>``. This allows
spawning multiple instances of the same process without needing to
worry about conflicting outputs. If you want to access a command's
output later, like with ``btest-diff``, use ``<tag>/foo.log`` to
access it.
``btest-bg-wait [-k] <timeout>``
This script waits for all processes previously spawned via
``btest-bg-run`` to finish. If any of them exits with a non-zero
return code, ``btest-bg-wait`` does so as well, indicating a
failed test. ``<timeout>`` is mandatory and gives the maximum
number of seconds to wait for any of the processes to terminate.
If any process hasn't done so when the timeout expires, it will be
killed and the test is considered to be failed as long as ``-k``
is not given. If ``-k`` is given, pending processes are still
killed but the test continues normally, i.e., non-termination is
not considered a failure in this case. This script also collects
the processes' stdout and stderr outputs for diagnostics output.
Integration with Sphinx
=======================
``btest`` comes with a new directive for the documentation framework
`Sphinx <http://sphinx.pocoo.org>`_. The directive allows to write a
test directly inside a Sphinx document, and then to include output
from the test's command into the generated documentation. The same
tests can also run externally and will catch if any changes to the
included content occur. The following walks through setting this up.
Configuration
-------------
First, you need to tell Sphinx a base directory for the ``btest``
configuration as well as a directory in there where to store tests
it extracts from the Sphinx documentation. Typically, you'd just
create a new subdirectory ``tests`` in the Sphinx project for the
``btest`` setup and then store the tests in there in, e.g.,
``doc/``::
cd <sphinx-root>
mkdir tests
mkdir tests/doc
Then add the following to your Sphinx ``conf.py``::
extensions += ["btest-sphinx"]
btest_base="tests" # Relative to Sphinx-root.
btest_tests="doc" # Relative to btest_base.
Next, a finalizer to ``btest.cfg``::
[btest]
...
Finalizer=btest-diff-rst
Finally, create a ``btest.cfg`` in ``tests/`` as usual and add
``doc/`` to the ``TestDirs`` option.
Including a Test into a Sphinx Document
---------------------------------------
The ``btest`` extension provides a new directive to include a test
inside a Sphinx document::
.. btest:: <test-name>
<test content>
Here, ``<test-name>`` is a custom name for the test; it will be
stored in ``btest_tests`` under that name. ``<test content>`` is just
a standard test as you would normally put into one of the
``TestDirs``. Example::
.. btest:: just-a-test
@TEST-EXEC: expr 2 + 2
When you now run Sphinx, it will (1) store the test content into
``tests/doc/just-a-test`` (assuming the above path layout), and (2)
execute the test by running ``btest`` on it. You can then run
``btest`` manually in ``tests/`` as well and it will execute the test
just as it would in a standard setup. If a test fails when Sphinx runs
it, there will be a corresponding error and include the diagnostic output
into the document.
By default, nothing else will be included into the generated
documentation, i.e., the above test will just turn into an empty text
block. However, ``btest`` comes with a set of scripts that you can use
to specify content to be included. As a simple example,
``btest-rst-cmd <cmdline>`` will execute a command and (if it
succeeds) include both the command line and the standard output into
the documentation. Example::
.. btest:: another-test
@TEST-EXEC: btest-rst-cmd echo Hello, world!
When running Sphinx, this will render as:
.. code::
# echo Hello, world!
Hello world!
When running ``btest`` manually in ``tests/``, the ``Finalizer`` we
added to ``btest.cfg`` (see above) compares the generated reST code
with a previously established baseline, just like ``btest-diff`` does
with files. To establish the initial baseline, run ``btest -u``, like
you would with ``btest-diff``.
Scripts
-------
The following Sphinx support scripts come with ``btest``:
``btest-rst-cmd [options] <cmdline>``
By default, this executes ``<cmdline>`` and includes both the
command line itself and its standard output into the generated
documentation. See above for an example.
This script provides the following options:
-c ALTERNATIVE_CMDLINE
Show ``ALTERNATIVE_CMDLINE`` in the generated
documentation instead of the one actually executed. (It
still runs the ``<cmdline>`` given outside the option.)
-d
Do not actually execute ``<cmdline>``; just format it for
the generated documentation and include no further output.
-f FILTER_CMD
Pipe the command line's output through ``FILTER_CMD``
before including. If ``-r`` is given, it filters the
file's content instead of stdout.
-o
Do not include the executed command into the generated
documentation, just its output.
-r FILE
Insert ``FILE`` into output instead of stdout.
``btest-rst-include <file>``
Includes ``<file>`` inside a code block.
``btest-rst-pipe <cmdline>``
Executes ``<cmdline>``, includes its standard output inside a code
block. Note that this script does not include the command line
itself into the code block, just the output.
.. note::
All these scripts can be run directly from the command line to show
the reST code they generate.
.. note::
``btest-rst-cmd`` can do everything the other scripts provide if
you give it the right options. In fact, the other scripts are
provided just for convenience and leverage ``btest-rst-cmd``
internally.
License
=======
btest is open-source under a BSD licence.

1
doc/components/capstats/README.rst
View file

@ -1 +0,0 @@
../../../aux/broctl/aux/capstats/README

107
doc/components/capstats/README.rst Normal file
View file

@ -0,0 +1,107 @@
.. -*- mode: rst-mode -*-
..
.. Version number is filled in automatically.
.. |version| replace:: 0.18
===============================================
capstats - A tool to get some NIC statistics.
===============================================
.. rst-class:: opening
capstats is a small tool to collect statistics on the
current load of a network interface, using either `libpcap
<http://www.tcpdump.org>`_ or the native interface for `Endace's
<http:///www.endace.com>`_. It reports statistics per time interval
and/or for the tool's total run-time.
Download
--------
You can find the latest capstats release for download at
http://www.bro.org/download.
Capstats's git repository is located at `git://git.bro.org/capstats.git
<git://git.bro.org/capstats.git>`__. You can browse the repository
`here <http://git.bro.org/capstats.git>`__.
This document describes capstats |version|. See the ``CHANGES``
file for version history.
Output
------
Here's an example output with output in one-second intervals until
``CTRL-C`` is hit:
.. console::
>capstats -i nve0 -I 1
1186620936.890567 pkts=12747 kpps=12.6 kbytes=10807 mbps=87.5 nic_pkts=12822 nic_drops=0 u=960 t=11705 i=58 o=24 nonip=0
1186620937.901490 pkts=13558 kpps=13.4 kbytes=11329 mbps=91.8 nic_pkts=13613 nic_drops=0 u=1795 t=24339 i=119 o=52 nonip=0
1186620938.912399 pkts=14771 kpps=14.6 kbytes=13659 mbps=110.7 nic_pkts=14781 nic_drops=0 u=2626 t=38154 i=185 o=111 nonip=0
1186620939.012446 pkts=1332 kpps=13.3 kbytes=1129 mbps=92.6 nic_pkts=1367 nic_drops=0 u=2715 t=39387 i=194 o=112 nonip=0
=== Total
1186620939.012483 pkts=42408 kpps=13.5 kbytes=36925 mbps=96.5 nic_pkts=1 nic_drops=0 u=2715 t=39387 i=194 o=112 nonip=0
Each line starts with a timestamp and the other fields are:
:pkts:
Absolute number of packets seen by ``capstats`` during interval.
:kpps:
Number of packets per second.
:kbytes:
Absolute number of KBytes during interval.
:mbps:
Mbits/sec.
:nic_pkts:
Number of packets as reported by ``libpcap``'s ``pcap_stats()`` (may not match _pkts_)
:nic_drops:
Number of packet drops as reported by ``libpcap``'s ``pcap_stats()``.
:u:
Number of UDP packets.
:t:
Number of TCP packets.
:i:
Number of ICMP packets.
:nonip:
Number of non-IP packets.
Options
-------
A list of all options::
capstats [Options] -i interface
-i| --interface <interface> Listen on interface
-d| --dag Use native DAG API
-f| --filter <filter> BPF filter
-I| --interval <secs> Stats logging interval
-l| --syslog Use syslog rather than print to stderr
-n| --number <count> Stop after outputting <number> intervals
-N| --select Use select() for live pcap (for testing only)
-p| --payload <n> Verifies that packets' payloads consist
entirely of bytes of the given value.
-q| --quiet <count> Suppress output, exit code indicates >= count
packets received.
-S| --size <size> Verify packets to have given <size>
-s| --snaplen <size> Use pcap snaplen <size>
-v| --version Print version and exit
-w| --write <filename> Write packets to file
Installation
------------
``capstats`` has been tested on Linux, FreeBSD, and MacOS. Please see
the ``INSTALL`` file for installation instructions.

28
doc/components/index.rst Normal file
View file

@ -0,0 +1,28 @@
=============
Subcomponents
=============
The following are snapshots of documentation for components that come
with this version of Bro (|version|). Since they can also be used
independently, see the `download page
<http://bro.org/download/index.html>`_ for documentation of any
current, independent component releases.
.. toctree::
:maxdepth: 1
BinPAC - A protocol parser generator <binpac/README>
Broccoli - The Bro Client Communication Library (README) <broccoli/README>
Broccoli - User Manual <broccoli/broccoli-manual>
Broccoli Python Bindings <broccoli-python/README>
Broccoli Ruby Bindings <broccoli-ruby/README>
BroControl - Interactive Bro management shell <broctl/README>
Bro-Aux - Small auxiliary tools for Bro <bro-aux/README>
BTest - A unit testing framework <btest/README>
Capstats - Command-line packet statistic tool <capstats/README>
PySubnetTree - Python module for CIDR lookups<pysubnettree/README>
trace-summary - Script for generating break-downs of network traffic <trace-summary/README>
The `Broccoli API Reference <broccoli-api/index.html>`_ may also be of
interest.

1
doc/components/pysubnettree/README.rst
View file

@ -1 +0,0 @@
../../../aux/broctl/aux/pysubnettree/README

98
doc/components/pysubnettree/README.rst Normal file
View file

@ -0,0 +1,98 @@
.. -*- mode: rst-mode -*-
..
.. Version number is filled in automatically.
.. |version| replace:: 0.19-9
===============================================
PySubnetTree - A Python Module for CIDR Lookups
===============================================
.. rst-class:: opening
The PySubnetTree package provides a Python data structure
``SubnetTree`` which maps subnets given in `CIDR
<http://tools.ietf.org/html/rfc4632>`_ notation (incl.
corresponding IPv6 versions) to Python objects. Lookups are
performed by longest-prefix matching.
Download
--------
You can find the latest PySubnetTree release for download at
http://www.bro.org/download.
PySubnetTree's git repository is located at `git://git.bro.org/pysubnettree.git
<git://git.bro.org/pysubnettree.git>`__. You can browse the repository
`here <http://git.bro.org/pysubnettree.git>`__.
This document describes PySubnetTree |version|. See the ``CHANGES``
file for version history.
Example
-------
A simple example which associates CIDR prefixes with strings::
>>> import SubnetTree
>>> t = SubnetTree.SubnetTree()
>>> t["10.1.0.0/16"] = "Network 1"
>>> t["10.1.42.0/24"] = "Network 1, Subnet 42"
>>> t["10.2.0.0/16"] = "Network 2"
>>> print t["10.1.42.1"]
Network 1, Subnet 42
>>> print t["10.1.43.1"]
Network 1
>>> print "10.1.42.1" in t
True
>>> print "10.1.43.1" in t
True
>>> print "10.20.1.1" in t
False
>>> print t["10.20.1.1"]
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "SubnetTree.py", line 67, in __getitem__
def __getitem__(*args): return _SubnetTree.SubnetTree___getitem__(*args)
KeyError: '10.20.1.1'
By default, CIDR prefixes and IP addresses are given as strings.
Alternatively, a ``SubnetTree`` object can be switched into *binary
mode*, in which single addresses are passed in the form of packed
binary strings as, e.g., returned by `socket.inet_aton
<http://docs.python.org/lib/module-socket.html#l2h-3657>`_::
>>> t.get_binary_lookup_mode()
False
>>> t.set_binary_lookup_mode(True)
>>> t.binary_lookup_mode()
True
>>> import socket
>>> print t[socket.inet_aton("10.1.42.1")]
Network 1, Subnet 42
A SubnetTree also provides methods ``insert(prefix,object=None)`` for insertion
of prefixes (``object`` can be skipped to use the tree like a set), and
``remove(prefix)`` for removing entries (``remove`` performs an _exact_ match
rather than longest-prefix).
Internally, the CIDR prefixes of a ``SubnetTree`` are managed by a
Patricia tree data structure and lookups are therefore efficient
even with a large number of prefixes.
PySubnetTree comes with a BSD license.
Prerequisites
-------------
This package requires Python 2.4 or newer.
Installation
------------
Installation is pretty simple::
> python setup.py install

1
doc/components/trace-summary/README.rst
View file

@ -1 +0,0 @@
../../../aux/broctl/aux/trace-summary/README

154
doc/components/trace-summary/README.rst Normal file
View file

@ -0,0 +1,154 @@
.. -*- mode: rst-mode -*-
..
.. Version number is filled in automatically.
.. |version| replace:: 0.8
====================================================
trace-summary - Generating network traffic summaries
====================================================
.. rst-class:: opening
``trace-summary`` is a Python script that generates break-downs of
network traffic, including lists of the top hosts, protocols,
ports, etc. Optionally, it can generate output separately for
incoming vs. outgoing traffic, per subnet, and per time-interval.
Download
--------
You can find the latest trace-summary release for download at
http://www.bro.org/download.
trace-summary's git repository is located at `git://git.bro.org/trace-summary.git
<git://git.bro.org/trace-summary.git>`__. You can browse the repository
`here <http://git.bro.org/trace-summary.git>`__.
This document describes trace-summary |version|. See the ``CHANGES``
file for version history.
Overview
--------
The ``trace-summary`` script reads both packet traces in `libpcap
<http://www.tcpdump.org>`_ format and connection logs produced by the
`Bro <http://www.bro.org>`_ network intrusion detection system
(for the latter, it supports both 1.x and 2.x output formats).
Here are two example outputs in the most basic form (note that IP
addresses are 'anonymized'). The first is from a packet trace and the
second from a Bro connection log::
>== Total === 2005-01-06-14-23-33 - 2005-01-06-15-23-43
- Bytes 918.3m - Payload 846.3m - Pkts 1.8m - Frags 0.9% - MBit/s 1.9 -
Ports | Sources | Destinations | Protocols |
80 33.8% | 131.243.89.214 8.5% | 131.243.89.214 7.7% | 6 76.0% |
22 16.7% | 128.3.2.102 6.2% | 128.3.2.102 5.4% | 17 23.3% |
11001 12.4% | 204.116.120.26 4.8% | 131.243.89.4 4.8% | 1 0.5% |
2049 10.7% | 128.3.161.32 3.6% | 131.243.88.227 3.6% | |
1023 10.6% | 131.243.89.4 3.5% | 204.116.120.26 3.4% | |
993 8.2% | 128.3.164.194 2.7% | 131.243.89.64 3.1% | |
1049 8.1% | 128.3.164.15 2.4% | 128.3.164.229 2.9% | |
524 6.6% | 128.55.82.146 2.4% | 131.243.89.155 2.5% | |
33305 4.5% | 131.243.88.227 2.3% | 128.3.161.32 2.3% | |
1085 3.7% | 131.243.89.155 2.3% | 128.55.82.146 2.1% | |
>== Total === 2005-01-06-14-23-33 - 2005-01-06-15-23-42
- Connections 43.4k - Payload 398.4m -
Ports | Sources | Destinations | Services | Protocols | States |
80 21.7% | 207.240.215.71 3.0% | 239.255.255.253 8.0% | other 51.0% | 17 55.8% | S0 46.2% |
427 13.0% | 131.243.91.71 2.2% | 131.243.91.255 4.0% | http 21.7% | 6 36.4% | SF 30.1% |
443 3.8% | 128.3.161.76 1.7% | 131.243.89.138 2.1% | i-echo 7.3% | 1 7.7% | OTH 7.8% |
138 3.7% | 131.243.90.138 1.6% | 255.255.255.255 1.7% | https 3.8% | | RSTO 5.8% |
515 2.4% | 131.243.88.159 1.6% | 128.3.97.204 1.5% | nb-dgm 3.7% | | SHR 4.4% |
11001 2.3% | 131.243.88.202 1.4% | 131.243.88.107 1.1% | printer 2.4% | | REJ 3.0% |
53 1.9% | 131.243.89.250 1.4% | 117.72.94.10 1.1% | dns 1.9% | | S1 1.0% |
161 1.6% | 131.243.89.80 1.3% | 131.243.88.64 1.1% | snmp 1.6% | | RSTR 0.9% |
137 1.4% | 131.243.90.52 1.3% | 131.243.88.159 1.1% | nb-ns 1.4% | | SH 0.3% |
2222 1.1% | 128.3.161.252 1.2% | 131.243.91.92 1.1% | ntp 1.0% | | RSTRH 0.2% |
Prerequisites
-------------
* This script requires Python 2.4 or newer.
* The `pysubnettree
<http://www.bro.org/documentation/pysubnettree.html>`_ Python
module.
* Eddie Kohler's `ipsumdump <http://www.cs.ucla.edu/~kohler/ipsumdump>`_
if using ``trace-summary`` with packet traces (versus Bro connection logs)
Installation
------------
Simply copy the script into some directory which is in your ``PATH``.
Usage
-----
The general usage is::
trace-summary [options] [input-file]
Per default, it assumes the ``input-file`` to be a ``libpcap`` trace
file. If it is a Bro connection log, use ``-c``. If ``input-file`` is
not given, the script reads from stdin. It writes its output to
stdout.
Options
~~~~~~~
The most important options are summarized
below. Run ``trace-summary --help`` to see the full list including
some more esoteric ones.
:-c:
Input is a Bro connection log instead of a ``libpcap`` trace
file.
:-b:
Counts all percentages in bytes rather than number of
packets/connections.
:-E <file>:
Gives a file which contains a list of networks to ignore for the
analysis. The file must contain one network per line, where each
network is of the CIDR form ``a.b.c.d/mask`` (including the
corresponding syntax for IPv6 prefixes, e.g., ``1:2:3:4::/64``).
Empty lines and lines starting with a "#" are ignored.
:-i <duration>:
Creates totals for each time interval of the given length
(default is seconds; add "``m``" for minutes and "``h``" for
hours). Use ``-v`` if you also want to see the breakdowns for
each interval.
:-l <file>:
Generates separate summaries for incoming and outgoing traffic.
``<file>`` is a file which contains a list of networks to be
considered local. Format as for ``-E``.
:-n <n>:
Show top n entries in each break-down. Default is 10.
:-r:
Resolves hostnames in the output.
:-s <n>:
Gives the sample factor if the input has been sampled.
:-S <n>:
Sample input with the given factor; less accurate but faster and
saves memory.
:-m:
Does skip memory-expensive statistics.
:-v:
Generates full break-downs for each time interval. Requires
``-i``.

26
doc/conf.py.in
View file

@ -12,11 +12,28 @@
import sys, os
extensions = []
# If extensions (or modules to document with autodoc) are in another directory,
# add these directories to sys.path here. If the directory is relative to the
# documentation root, use os.path.abspath to make it absolute, like shown here.
sys.path.insert(0, os.path.abspath('sphinx-sources/ext'))
# ----- Begin of BTest configuration. -----
btest = os.path.abspath("@CMAKE_SOURCE_DIR@/aux/btest")
brocut = os.path.abspath("@CMAKE_SOURCE_DIR@/aux/bro-aux/bro-cut")
bro = os.path.abspath("@CMAKE_SOURCE_DIR@/build/src")
os.environ["PATH"] += (":%s:%s/sphinx:%s:%s" % (btest, btest, bro, brocut))
sys.path.append(os.path.join(btest, "sphinx"))
extensions += ["btest-sphinx"]
btest_base="@CMAKE_SOURCE_DIR@/testing/btest"
btest_tests="doc/sphinx"
# ----- End of BTest configuration. -----
# -- General configuration -----------------------------------------------------
# If your documentation needs a minimal Sphinx version, state it here.
@ -24,7 +41,10 @@ sys.path.insert(0, os.path.abspath('sphinx-sources/ext'))
# Add any Sphinx extension module names here, as strings. They can be extensions
# coming with Sphinx (named 'sphinx.ext.*') or your custom ones.
extensions = ['bro', 'rst_directive', 'sphinx.ext.todo', 'adapt-toc']
extensions += ['bro', 'rst_directive', 'sphinx.ext.todo', 'adapt-toc']
os.environ["BRO_SRC_ROOT"] = "@CMAKE_SOURCE_DIR@"
os.environ["DOC_ROOT"] = "@CMAKE_SOURCE_DIR@/doc"
# Add any paths that contain templates here, relative to this directory.
templates_path = ['sphinx-sources/_templates', 'sphinx-sources/_static']
@ -40,7 +60,7 @@ master_doc = 'index'
# General information about the project.
project = u'Bro'
copyright = u'2012, The Bro Project'
copyright = u'2013, The Bro Project'
# The version info for the project you're documenting, acts as replacement for
# |version| and |release|, also used in various other places throughout the
@ -63,7 +83,7 @@ today_fmt = '%B %d, %Y'
# List of patterns, relative to source directory, that match files and
# directories to ignore when looking for source files.
exclude_patterns = []
exclude_patterns = [".#*"]
# The reST default role (used for this markup: `text`) to use for all documents.
#default_role = None

180
doc/faq.rst
View file

@ -1,180 +0,0 @@
==========================
Frequently Asked Questions
==========================
.. raw:: html
<div class="faq">
.. contents::
Installation and Configuration
==============================
How can I tune my operating system for best capture performance?
----------------------------------------------------------------
Here are some pointers to more information:
* Fabian Schneider's research on `high performance packet capture
<http://www.net.t-labs.tu-berlin.de/research/hppc>`_
* `NSMWiki <http://nsmwiki.org/Main_Page>`_ has page on
*Collecting Data*.
* An `IMC 2010 paper
<http://conferences.sigcomm.org/imc/2010/papers/p206.pdf>`_ by
Lothar Braun et. al evaluates packet capture performance on
commodity hardware
Are there any gotchas regarding interface configuration for live capture? Or why might I be seeing abnormally large packets much greater than interface MTU?
-------------------------------------------------------------------------------------------------------------------------------------------------------------
Some NICs offload the reassembly of traffic into "superpackets" so that
fewer packets are then passed up the stack (e.g. "TCP segmentation
offload", or "generic segmentation offload"). The result is that the
capturing application will observe packets much larger than the MTU size
of the interface they were captured from and may also interfere with the
maximum packet capture length, ``snaplen``, so it's a good idea to disable
an interface's offloading features.
You can use the ``ethtool`` program on Linux to view and disable
offloading features of an interface. See this page for more explicit
directions:
http://securityonion.blogspot.com/2011/10/when-is-full-packet-capture-not-full.html
What does an error message like ``internal error: NB-DNS error`` mean?
----------------------------------------------------------------------
That often means that DNS is not set up correctly on the system
running Bro. Try verifying from the command line that DNS lookups
work, e.g., ``host www.google.com``.
I am using OpenBSD and having problems installing Bro?
------------------------------------------------------
One potential issue is that the top-level Makefile may not work with
OpenBSD's default make program, in which case you can either install
the ``gmake`` package and use it instead or first change into the
``build/`` directory before doing either ``make`` or ``make install``
such that the CMake-generated Makefile's are used directly.
Generally, please note that we do not regularly test OpenBSD builds.
We appreciate any patches that improve Bro's support for this
platform.
How do BroControl options affect Bro script variables?
------------------------------------------------------
Some (but not all) BroControl options override a corresponding Bro script variable.
For example, setting the BroControl option "LogRotationInterval" will override
the value of the Bro script variable "Log::default_rotation_interval".
See the :doc:`BroControl Documentation <components/broctl/README>` to find out
which BroControl options override Bro script variables, and for more discussion
on site-specific customization.
Usage
=====
How can I identify backscatter?
-------------------------------
Identifying backscatter via connections labeled as ``OTH`` is not a reliable
means to detect backscatter. Backscatter is however visible by interpreting
the contents of the ``history`` field in the ``conn.log`` file. The basic idea
is to watch for connections that never had an initial ``SYN`` but started
instead with a ``SYN-ACK`` or ``RST`` (though this latter generally is just
discarded). Here are some history fields which provide backscatter examples:
``hAFf``, ``r``. Refer to the conn protocol analysis scripts to interpret the
individual character meanings in the history field.
Is there help for understanding Bro's resource consumption?
-----------------------------------------------------------
There are two scripts that collect statistics on resource usage:
``misc/stats.bro`` and ``misc/profiling.bro``. The former is quite
lightweight, while the latter should only be used for debugging.
How can I capture packets as an unprivileged user?
--------------------------------------------------
Normally, unprivileged users cannot capture packets from a network interface,
which means they would not be able to use Bro to read/analyze live traffic.
However, there are operating system specific ways to enable packet capture
permission for non-root users, which is worth doing in the context of using
Bro to monitor live traffic.
With Linux Capabilities
^^^^^^^^^^^^^^^^^^^^^^^
Fully implemented since Linux kernel 2.6.24, capabilities are a way of
parceling superuser privileges into distinct units. Attach capabilities
required to capture packets to the ``bro`` executable file like this:
.. console::
sudo setcap cap_net_raw,cap_net_admin=eip /path/to/bro
Now any unprivileged user should have the capability to capture packets
using Bro provided that they have the traditional file permissions to
read/execute the ``bro`` binary.
With BPF Devices
^^^^^^^^^^^^^^^^
Systems using Berkeley Packet Filter (BPF) (e.g. FreeBSD & Mac OS X)
can allow users with read access to a BPF device to capture packets from
it using libpcap.
* Example of manually changing BPF device permissions to allow users in
the ``admin`` group to capture packets:
.. console::
sudo chgrp admin /dev/bpf*
sudo chmod g+r /dev/bpf*
* Example of configuring devfs to set permissions of BPF devices, adding
entries to ``/etc/devfs.conf`` to grant ``admin`` group permission to
capture packets:
.. console::
sudo sh -c 'echo "own bpf root:admin" >> /etc/devfs.conf'
sudo sh -c 'echo "perm bpf 0640" >> /etc/devfs.conf'
sudo service devfs restart
.. note:: As of Mac OS X 10.6, the BPF device is on devfs, but the used version
of devfs isn't capable of setting the device permissions. The permissions
can be changed manually, but they will not survive a reboot.
Why isn't Bro producing the logs I expect? (A Note About Checksums)
-------------------------------------------------------------------
Normally, Bro's event engine will discard packets which don't have valid
checksums. This can be a problem if one wants to analyze locally
generated/captured traffic on a system that offloads checksumming to the
network adapter. In that case, all transmitted/captured packets will have
bad checksums because they haven't yet been calculated by the NIC, thus
such packets will not undergo analysis defined in Bro policy scripts as they
normally would. Bad checksums in traces may also be a result of some packet
alteration tools.
Bro has two options to workaround such situations and ignore bad checksums:
1) The ``-C`` command line option to ``bro``.
2) An option called ``ignore_checksums`` that can be redefined at the
policy script layer (e.g. in your ``$PREFIX/share/bro/site/local.bro``):
.. code:: bro
redef ignore_checksums = T;
The other alternative is to disable checksum offloading for your
network adapter, but this is not always possible or desirable.
.. raw:: html
</div>

0
doc/file-analysis.rst → doc/frameworks/file-analysis.rst
View file

2
doc/geoip.rst → doc/frameworks/geoip.rst
View file

@ -1,4 +1,6 @@
.. _geolocation:
===========
GeoLocation
===========

16
doc/frameworks/index.rst Normal file
View file

@ -0,0 +1,16 @@
==========
Frameworks
==========
.. toctree::
:maxdepth: 1
file-analysis
geoip
input
intel
logging
notice
signatures

7
doc/input.rst → doc/frameworks/input.rst
View file

@ -1,6 +1,7 @@
==============================================
Loading Data into Bro with the Input Framework
==============================================
===============
Input Framework
===============
.. rst-class:: opening

6
doc/intel.rst → doc/frameworks/intel.rst
View file

@ -1,5 +1,7 @@
Intel Framework
===============
======================
Intelligence Framework
======================
Intro
-----

0
doc/logging-dataseries.rst → doc/frameworks/logging-dataseries.rst
View file

0
doc/logging-elasticsearch.rst → doc/frameworks/logging-elasticsearch.rst
View file

12
doc/logging.rst → doc/frameworks/logging.rst
View file

@ -1,6 +1,9 @@
==========================
Customizing Bro's Logging
==========================
.. _framework-logging:
=================
Logging Framework
=================
.. rst-class:: opening
@ -89,7 +92,8 @@ Note the fields that are set for the filter:
are generated by taking the stream's ID and munging it slightly.
:bro:enum:`Conn::LOG` is converted into ``conn``,
:bro:enum:`PacketFilter::LOG` is converted into
``packet_filter``.
``packet_filter``, and :bro:enum:`Notice::POLICY_LOG` is
converted into ``notice_policy``.
``include``
A set limiting the fields to the ones given. The names

10
doc/notice.rst → doc/frameworks/notice.rst
View file

@ -98,9 +98,9 @@ type :bro:see:`SSH::Password_Guessing` if the server is 10.0.0.1:
.. note::
Keep in mind that the semantics of the SSH::Password_Guessing notice are
such that it is only raised when Bro heuristically detects a failed
login.
Keep in mind that the semantics of the :bro:see:`SSH::Password_Guessing`
notice are such that it is only raised when Bro heuristically detects
a failed login.
Hooks can also have priorities applied to order their execution like events
with a default priority of 0. Greater values are executed first. Setting
@ -339,7 +339,7 @@ included below.
hook Notice::policy(n: Notice::Info)
{
if ( n?$conn && n$conn?$http && n$conn$http?$host )
n$email_body_sections[|email_body_sections|] = fmt("HTTP host header: %s", n$conn$http$host);
n$email_body_sections[|n$email_body_sections|] = fmt("HTTP host header: %s", n$conn$http$host);
}
@ -348,7 +348,7 @@ Cluster Considerations
As a user/developer of Bro, the main cluster concern with the notice framework
is understanding what runs where. When a notice is generated on a worker, the
worker checks to see if the notice shoudl be suppressed based on information
worker checks to see if the notice should be suppressed based on information
locally maintained in the worker process. If it's not being
suppressed, the worker forwards the notice directly to the manager and does no more
local processing. The manager then runs the :bro:see:`Notice::policy` hook and

6
doc/signatures.rst → doc/frameworks/signatures.rst
View file

@ -1,7 +1,7 @@
==========
Signatures
==========
===================
Signature Framework
===================
.. rst-class:: opening

93
doc/index.rst
View file

@ -1,92 +1,25 @@
.. Bro documentation master file
=================
Bro Documentation
=================
Guides
------
.. toctree::
:maxdepth: 1
:maxdepth: 2
INSTALL
upgrade
quickstart
faq
reporting-problems
Frameworks
----------
.. toctree::
:maxdepth: 1
notice
logging
input
file-analysis
cluster
signatures
How-Tos
-------
.. toctree::
:maxdepth: 1
geoip
Script Reference
----------------
.. toctree::
:maxdepth: 1
scripts/packages
scripts/index
scripts/builtins
scripts/proto-analyzers
scripts/file-analyzers
Other Bro Components
--------------------
The following are snapshots of documentation for components that come
with this version of Bro (|version|). Since they can also be used
independently, see the `download page
<http://bro.org/download/index.html>`_ for documentation of any
current, independent component releases.
.. toctree::
:maxdepth: 1
BinPAC - A protocol parser generator <components/binpac/README>
Broccoli - The Bro Client Communication Library (README) <components/broccoli/README>
Broccoli - User Manual <components/broccoli/broccoli-manual>
Broccoli Python Bindings <components/broccoli-python/README>
Broccoli Ruby Bindings <components/broccoli-ruby/README>
BroControl - Interactive Bro management shell <components/broctl/README>
Bro-Aux - Small auxiliary tools for Bro <components/bro-aux/README>
BTest - A unit testing framework <components/btest/README>
Capstats - Command-line packet statistic tool <components/capstats/README>
PySubnetTree - Python module for CIDR lookups<components/pysubnettree/README>
trace-summary - Script for generating break-downs of network traffic <components/trace-summary/README>
The `Broccoli API Reference <broccoli-api/index.html>`_ may also be of
interest.
Other Indices and References
----------------------------
intro/index.rst
install/index.rst
quickstart/index.rst
using/index.rst
scripting/index.rst
frameworks/index.rst
cluster/index.rst
scripts/index.rst
components/index.rst
* `Notice Index <bro-noticeindex.html>`_ (TODO: Move to reference
section, but can't figure out how to include it into toctree)
* :ref:`General Index <genindex>`
* `Notice Index <bro-noticeindex.html>`_
* :ref:`search`
Internal References
-------------------
.. toctree::
:maxdepth: 1
scripts/internal

1
doc/install/CHANGES-binpac.txt Symbolic link
View file

@ -0,0 +1 @@
../../aux/binpac/CHANGES

1
doc/install/CHANGES-bro-aux.txt Symbolic link
View file

@ -0,0 +1 @@
../../aux/bro-aux/CHANGES

1
doc/install/CHANGES-bro.txt Symbolic link
View file

@ -0,0 +1 @@
../../CHANGES

1
doc/install/CHANGES-broccoli-python.txt Symbolic link
View file

@ -0,0 +1 @@
../../aux/broccoli/bindings/broccoli-python/CHANGES

1
doc/install/CHANGES-broccoli-ruby.txt Symbolic link
View file

@ -0,0 +1 @@
../../aux/broccoli/bindings/broccoli-ruby/CHANGES

1
doc/install/CHANGES-broccoli.txt Symbolic link
View file

@ -0,0 +1 @@
../../aux/broccoli/CHANGES

1
doc/install/CHANGES-broctl.txt Symbolic link
View file

@ -0,0 +1 @@
../../aux/broctl/CHANGES

1
doc/install/CHANGES-btest.txt Symbolic link
View file

@ -0,0 +1 @@
../../aux/btest/CHANGES

1
doc/install/CHANGES-capstats.txt Symbolic link
View file

@ -0,0 +1 @@
../../aux/broctl/aux/capstats/CHANGES

1
doc/install/CHANGES-pysubnettree.txt Symbolic link
View file

@ -0,0 +1 @@
../../aux/broctl/aux/pysubnettree/CHANGES

1
doc/install/CHANGES-trace-summary.txt Symbolic link
View file

@ -0,0 +1 @@
../../aux/broctl/aux/trace-summary/CHANGES

1
doc/install/NEWS.rst Symbolic link
View file

@ -0,0 +1 @@
../../NEWS

75
doc/install/changes.rst Normal file
View file

@ -0,0 +1,75 @@
========================
Detailed Version History
========================
.. contents::
---
Bro
---
.. literalinclude:: CHANGES-bro.txt
----------
BroControl
----------
.. literalinclude:: CHANGES-broctl.txt
--------
Broccoli
--------
.. literalinclude:: CHANGES-broccoli.txt
---------------
Broccoli Python
---------------
.. literalinclude:: CHANGES-broccoli-python.txt
-------------
Broccoli Ruby
-------------
.. literalinclude:: CHANGES-broccoli-ruby.txt
--------
Capstats
--------
.. literalinclude:: CHANGES-capstats.txt
-------------
Trace-Summary
-------------
.. literalinclude:: CHANGES-trace-summary.txt
------
BinPAC
------
.. literalinclude:: CHANGES-binpac.txt
-------
Bro-Aux
-------
.. literalinclude:: CHANGES-bro-aux.txt
-----
BTest
-----
.. literalinclude:: CHANGES-btest.txt
------------
PySubnetTree
------------
.. literalinclude:: CHANGES-pysubnettree.txt

43
doc/install/guidelines.rst Normal file
View file

@ -0,0 +1,43 @@
.. _upgrade-guidelines:
==================
General Guidelines
==================
If you're doing an upgrade install (rather than a fresh install),
there's two suggested approaches: either install Bro using the same
installation prefix directory as before, or pick a new prefix and copy
local customizations over. In the following we summarize general
guidelines for upgrading, see the `Release Notes <release-notes>`_ for
version-specific information.
Re-Using Previous Install Prefix
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
If you choose to configure and install Bro with the same prefix
directory as before, local customization and configuration to files in
``$prefix/share/bro/site`` and ``$prefix/etc`` won't be overwritten
(``$prefix`` indicating the root of where Bro was installed). Also, logs
generated at run-time won't be touched by the upgrade. (But making
a backup of local changes before upgrading is still recommended.)
After upgrading, remember to check ``$prefix/share/bro/site`` and
``$prefix/etc`` for ``.example`` files, which indicate the
distribution's version of the file differs from the local one, which may
include local changes. Review the differences, and make adjustments
as necessary (for differences that aren't the result of a local change,
use the new version's).
Using a New Install prefix
~~~~~~~~~~~~~~~~~~~~~~~~~~
If you want to install the newer version in a different prefix
directory than before, you can just copy local customization and
configuration files from ``$prefix/share/bro/site`` and ``$prefix/etc``
to the new location (``$prefix`` indicating the root of where Bro was
originally installed). Make sure to review the files for difference
before copying and make adjustments as necessary (for differences that
aren't the result of a local change, use the new version's). Of
particular note, the copied version of ``$prefix/etc/broctl.cfg`` is
likely to need changes to the ``SpoolDir`` and ``LogDir`` settings.

12
doc/install/index.rst Normal file
View file

@ -0,0 +1,12 @@
.. _installation:
============
Installation
============
.. toctree::
:maxdepth: 2
install
upgrade

241
doc/install/install.rst Normal file
View file

@ -0,0 +1,241 @@
.. _CMake: http://www.cmake.org
.. _SWIG: http://www.swig.org
.. _Xcode: https://developer.apple.com/xcode/
.. _MacPorts: http://www.macports.org
.. _Fink: http://www.finkproject.org
.. _Homebrew: http://mxcl.github.com/homebrew
.. _bro downloads page: http://bro.org/download/index.html
.. _installing-bro:
==============
Installing Bro
==============
.. contents::
Prerequisites
=============
Before installing Bro, you'll need to ensure that some dependencies
are in place.
Required Dependencies
---------------------
Bro requires the following libraries and tools to be installed
before you begin:
* Libpcap (http://www.tcpdump.org)
* OpenSSL libraries (http://www.openssl.org)
* BIND8 library
* Libmagic
* Libz
* Bash (for BroControl)
To build Bro from source, the following additional dependencies are required:
* CMake 2.6.3 or greater (http://www.cmake.org)
* SWIG (http://www.swig.org)
* Bison (GNU Parser Generator)
* Flex (Fast Lexical Analyzer)
* Libpcap headers (http://www.tcpdump.org)
* OpenSSL headers (http://www.openssl.org)
* libmagic headers
* zlib headers
* Perl
To install the required dependencies, you can use:
* RPM/RedHat-based Linux:
.. console::
sudo yum install cmake make gcc gcc-c++ flex bison libpcap-devel openssl-devel python-devel swig zlib-devel file-devel
* DEB/Debian-based Linux:
.. console::
sudo apt-get install cmake make gcc g++ flex bison libpcap-dev libssl-dev python-dev swig zlib1g-dev libmagic-dev
* FreeBSD:
Most required dependencies should come with a minimal FreeBSD install
except for the following.
.. console::
sudo pkg_add -r bash cmake swig bison python
Note that ``bash`` needs to be in ``PATH``, which by default it is
not. The FreeBSD package installs the binary into
``/usr/local/bin``.
* Mac OS X:
Compiling source code on Macs requires first downloading Xcode_,
then going through its "Preferences..." -> "Downloads" menus to
install the "Command Line Tools" component.
Lion (10.7) and Mountain Lion (10.8) come with all required
dependencies except for CMake_, SWIG_, and ``libmagic``.
Distributions of these dependencies can likely be obtained from your
preferred Mac OS X package management system (e.g. MacPorts_, Fink_,
or Homebrew_).
Specifically for MacPorts, the ``swig``, ``swig-ruby``, ``swig-python``
and ``file`` packages provide the required dependencies.
Optional Dependencies
---------------------
Bro can make use of some optional libraries and tools if they are found at
build time:
* LibGeoIP (for geo-locating IP addresses)
* gperftools (tcmalloc is used to improve memory and CPU usage)
* ipsumdump (for trace-summary; http://www.cs.ucla.edu/~kohler/ipsumdump)
* Ruby executable, library, and headers (for Broccoli Ruby bindings)
LibGeoIP is probably the most interesting and can be easily installed
on most platforms:
* RedHat Enterprise Linux:
.. console::
sudo yum install geoip-devel sendmail
* CentOS Linux:
.. console::
sudo yum install GeoIP-devel sendmail
* DEB/Debian-based Linux:
.. console::
sudo apt-get install libgeoip-dev sendmail
* FreeBSD using ports:
.. console::
sudo pkg_add -r GeoIP
* Mac OS X:
Vanilla OS X installations don't ship with libGeoIP, but if
installed from your preferred package management system (e.g.
MacPorts, Fink, or Homebrew), they should be automatically detected
and Bro will compile against them.
Additional steps may be needed to :ref:`get the right GeoIP database
<geolocation>`.
Installing Bro
==============
Bro can be downloaded in either pre-built binary package or source
code forms.
Using Pre-Built Binary Release Packages
=======================================
See the `bro downloads page`_ for currently supported/targeted
platforms for binary releases.
* RPM
.. console::
sudo yum localinstall Bro-*.rpm
* DEB
.. console::
sudo gdebi Bro-*.deb
* MacOS Disk Image with Installer
Just open the ``Bro-*.dmg`` and then run the ``.pkg`` installer.
Everything installed by the package will go into ``/opt/bro``.
The primary install prefix for binary packages is ``/opt/bro``.
Non-MacOS packages that include BroControl also put variable/runtime
data (e.g. Bro logs) in ``/var/opt/bro``.
Installing From Source
==========================
Bro releases are bundled into source packages for convenience and
available from the `bro downloads page`_. Alternatively, the latest
Bro development version can be obtained through git repositories
hosted at ``git.bro.org``. See our `git development documentation
<http://bro.org/development/process.html>`_ for comprehensive
information on Bro's use of git revision control, but the short story
for downloading the full source code experience for Bro via git is:
.. console::
git clone --recursive git://git.bro.org/bro
.. note:: If you choose to clone the ``bro`` repository
non-recursively for a "minimal Bro experience", be aware that
compiling it depends on several of the other submodules as well.
The typical way to build and install from source is (for more options,
run ``./configure --help``):
.. console::
./configure
make
make install
The default installation path is ``/usr/local/bro``, which would typically
require root privileges when doing the ``make install``. A different
installation path can be chosen by specifying the ``--prefix`` option.
Note that ``/usr`` and ``/opt/bro`` are the
standard prefixes for binary Bro packages to be installed, so those are
typically not good choices unless you are creating such a package.
Depending on the Bro package you downloaded, there may be auxiliary
tools and libraries available in the ``aux/`` directory. Some of them
will be automatically built and installed along with Bro. There are
``--disable-*`` options that can be given to the configure script to
turn off unwanted auxiliary projects that would otherwise be installed
automatically. Finally, use ``make install-aux`` to install some of
the other programs that are in the ``aux/bro-aux`` directory.
OpenBSD users, please see our at `FAQ
<http://www.bro.org/documentation/faq.html>`_ if you are having
problems installing Bro.
Configure the Run-Time Environment
==================================
Just remember that you may need to adjust your ``PATH`` environment variable
according to the platform/shell/package you're using. For example:
Bourne-Shell Syntax:
.. console::
export PATH=/usr/local/bro/bin:$PATH
C-Shell Syntax:
.. console::
setenv PATH /usr/local/bro/bin:$PATH
Or substitute ``/opt/bro/bin`` instead if you installed from a binary package.

13
doc/install/release-notes.rst Normal file
View file

@ -0,0 +1,13 @@
.. _release-notes:
=============
Release Notes
=============
.. contents::
.. include:: NEWS.rst

10
doc/install/upgrade.rst Normal file
View file

@ -0,0 +1,10 @@
=============
Upgrading Bro
=============
.. toctree::
guidelines
release-notes
changes

BIN
doc/intro/architecture.png Normal file
View file

Binary file not shown.
Side by side

After

Width:  |  Height:  |  Size: 136 KiB

BIN
doc/intro/bro-eyes.png Normal file
View file

Binary file not shown.
Side by side

After

Width:  |  Height:  |  Size: 45 KiB

BIN
doc/intro/history.png Normal file
View file

Binary file not shown.
Side by side

After

Width:  |  Height:  |  Size: 159 KiB

247
doc/intro/index.rst Normal file
View file

@ -0,0 +1,247 @@
============
Introduction
============
.. contents::
Overview
--------
Bro is a passive, open-source network traffic analyzer. It is
primarily a security monitor that inspects all traffic on a link in
depth for signs of suspicious activity. More generally, however,
Bro supports a wide range of traffic analysis tasks even outside of
the security domain, including performance measurements and helping
with trouble-shooting.
The most immediate benefit that a site gains from deploying Bro is an
extensive set of *log files* that record a network's activity in
high-level terms. These logs include not only a comprehensive record
of every connection seen on the wire, but also application-layer
transcripts such as, e.g., all HTTP sessions with their requested
URIs, key headers, MIME types, and server responses; DNS requests with
replies; SSL certificates; key content of SMTP sessions; and much
more. By default, Bro writes all this information into well-structured
tab-separated log files suitable for post-processing with external
software. Users can however also chose from a set of alternative
output formats and backends to interface directly with, e.g., external
databases.
In addition to the logs, Bro comes with built-in functionality for a
range of analysis and detection tasks, including extracting files from
HTTP sessions, detecting malware by interfacing to external
registries, reporting vulnerable versions of software seen on the
network, identifying popular web applications, detecting SSH
brute-forcing, validating SSL certificate chains, and much more.
However, the key to understanding Bro lies in realizing that even
though the system comes with such powerful functionality out of the
box, fundamentally it represents a *platform* for traffic analyses
that's fully customizable and extensible: Bro provides users with a
domain-specific, Turing-complete *scripting language* for expressing
arbitrary analysis tasks. Conceptually, you can think of Bro as a
"domain-specific Python" (or Perl): just like Python, the system comes
with a large set of pre-built functionality (the "standard library"),
yet you are not limited to what the system ships with but can put Bro
to use in novel ways by writing your own code. Indeed, all of Bro's
default analyses, including all the logging, is the result of such
scripts; there's no specific analysis hard-coded into the core of
system.
Bro runs on commodity hardware and hence provides a low-cost
alternative to expensive proprietary solutions. Despite the price tag,
however, Bro actually goes far beyond the capabilities of other
network monitoring tools, which typically remain limited to a small
set of hard-coded analysis tasks. We emphasize in particular that Bro
is *not* a classic signature-based intrusion detection system (IDS).
While it supports such standard functionality as well, Bro's scripting
language indeed facilitates a much broader spectrum of very different
approaches to finding malicious activity, including semantic misuse
detection, anomaly detection, and behavioral analysis.
A large variety of sites deploy Bro operationally for protecting their
cyberinfrastructure, including many universities, research labs,
supercomputing centers, open-science communities, and major
corporations. Bro specifically targets high-speed, high-volume network
monitoring, and an increasing number of sites are now using the system
to monitor their 10GE networks, with some already moving on to 100GE
links. Bro accommodates such high-performance settings by supporting
scalable load-balancing: large sites typically run "Bro Clusters" in
which a high-speed frontend load-balancer distributes the traffic
across an appropriate number of backend PCs, all running dedicated Bro
instances on their individual traffic slices. A central manager system
coordinates the process, synchronizing state across the backends and
providing the operators with a central management interface for
configuration and access to aggregated logs. Bro's integrated
management framework, BroControl, supports such cluster setups
out-of-the-box.
Features
--------
Bro supports a wide range of analyses through its scripting language.
Yet even without further customization it comes with a powerful set of
features.
- Deployment
* Runs on commodity hardware on standard UNIX-style systems
(including Linux, FreeBSD, and MacOS).
* Fully passive traffic analysis off a network tap or monitoring
port.
* Standard libpcap interface for capturing packets.
* Real-time and offline analysis.
* Cluster-support for large-scale deployments.
* Unified management framework for operating both standalone and
cluster setups.
* Open-source under a BSD license.
- Analysis
* Comprehensive logging of activity for offline analysis and
forensics.
* Port-independent analysis of application-layer protocols.
* Support for many application-layer protocols (including DNS,
FTP, HTTP, IRC, SMTP, SSH, SSL).
* Analysis of file content exchanged over application-layer
protocols, including MD5/SHA1 computation for fingerprinting.
* Comprehensive IPv6 support.
* Tunnel detection and analysis (including Ayiya, Teredo, GTPv1).
Bro decapsulates the tunnels and then proceeds to analyze their
content as if no tunnel was in place.
* Extensive sanity checks during protocol analysis.
* Support for IDS-style pattern matching.
- Scripting Language
* Turing-complete language for expression arbitrary analysis
tasks.
* Event-based programming model.
* Domain-specific data types such as IP addresses (transparently
handling both IPv4 and IPv6), port numbers, and timers.
* Extensive support for tracking and managing network state over
time.
- Interfacing
* Default output to well-structured ASCII logs.
* Alternative backends for ElasticSearch and DataSeries. Further
database interfaces in preparation.
* Real-time integration of external input into analyses. Live
database input in preparation.
* External C library for exchanging Bro events with external
programs. Comes with Perl, Python, and Ruby bindings.
* Ability to trigger arbitrary external processes from within
the scripting language.
History
-------
.. figure:: history.png
:width: 600
:align: center
:alt: Bro History Timeline
:target: history.png
Timeline of Bro's History (click to enlarge).
Bro's history goes back much further than many people realize. `Vern
Paxson <http://www.icir.org/vern>`_ designed and implemented the
initial version almost two decades ago.
Vern began work on the code in 1995 as a researcher at the `Lawrence
Berkeley National Laboratory (LBNL) <http://www.lbl.gov>`_. Berkeley
Lab began operational deployment in 1996, and the USENIX Security
Symposium published the original Bro paper in 1998 (later refined in a
subsequent `journal publication <http://www.icir.org/vern/papers/bro-CN99.pdf>`_).
In 2003, the
`National Science Foundation (NSF) <http://www.nsf.gov>`_ began
supporting research and advanced development on Bro at the
`International Computer Science Institute (ICSI)
<http://www.icsi.berkeley.edu>`_, where Vern now leads the `Networking
and Security group <http://www.icir.org>`_. Over the years, a growing
team of ICSI researchers and students kept adding novel functionality
to Bro, while LBNL continued its support with funding from the
`Department of Energy (DOE) <http://www.doe.gov>`_.
Much of Bro's capabilities originate in academic research projects,
with results often published at top-tier conferences. However, the key
to Bro's success was its ability to bridge the traditional gap between
academia and operations from early on, which provided the research
with crucial grounding to ensure that developed approaches stand up to
the challenges of the real world. Yet, with Bro's operational user
community growing over time, the research-centric development model
eventually became a bottleneck to the system's evolution: research
grants do not tend to support the more mundane parts of software
development and maintenance, even though those prove crucial for the
end-user experience. While Bro's capabilities always went beyond those
of traditional systems, a successful deployment used to require
significant technical expertise, typically with a large upfront
investment in tackling Bro's steep learning curve. In 2010, NSF set
out to address this gap by awarding ICSI a grant dedicated solely to
Bro development out of its SDCI program.
With that support in place, the `National Center for
Supercomputing Applications (NCSA) <http://www.ncsa.illinois.edu>`_
joined the team as a core partner, and the Bro Project began to
completely overhaul many of the user-visible parts of the system for
the 2.0 release. Since that version came out, Bro has experienced an
tremendous growth in new deployments across a diverse range of
settings, and the Bro team is now working to build on this success by
further advancing the system's capabilities to address the challenges
of future networks.
Architecture
------------
.. figure:: architecture.png
:width: 400
:align: center
:alt: Bro Architecture
:target: {{docroot}}/images/architecture.png
Bro's internal architecture.
Architecturally, Bro is layered into two major components. Its *event
engine* (or *core*) reduces the incoming packet stream into a series
of higher-level *events*. These events reflect network activity in
policy-neutral terms, i.e., they describe *what* has been seen, but
not *why*, or whether it is significant. For example, every HTTP
request on the wire turns into a corresponding ``http_request`` event
that carries with it the involved IP addresses and ports, the URI
being requested, and the HTTP version in use. The event however does
not convey any further *interpretation*, e.g., of whether that URI
corresponds to a known malware site.
Such semantics are instead derived by Bro's second main component, the
*script interpreter*, which executes a set of *event handlers* written
in Bro's custom scripting language. These scripts can express a site's
security policy, i.e., what actions to take when the monitor detects
different types of activity. More generally they can derive any
desired properties and statistics from the input traffic. Bro's
language comes with extensive domain-specific types and support
functionality; and, crucially, allows scripts to maintain state
over time, enabling them to track and correlate the evolution of what
they observe across connection and host boundaries. Bro scripts can
generate real-time alerts and also execute arbitrary external programs
on demand, e.g., to trigger an active response to an attack.

95
doc/quickstart.rst → doc/quickstart/index.rst
View file

@ -1,29 +1,21 @@
.. _quickstart:
=================
Quick Start Guide
=================
.. rst-class:: opening
The short story for getting Bro up and running in a simple configuration
for analysis of either live traffic from a network interface or a packet
capture trace file.
.. contents::
Installation
============
Bro works on most modern, Unix-based systems and requires no custom
hardware. It can be downloaded in either pre-built binary package or
source code forms. See :doc:`Installing Bro <INSTALL>` for instructions
on how to install Bro.
source code forms. See :ref:`installing-bro` for instructions on how to
install Bro. Below, ``$PREFIX`` is used to reference the Bro
installation root directory, which by default is ``/usr/local/`` if
you install from source.
.. note:: Below, ``$PREFIX`` is used to reference the Bro installation
root directory.
Using BroControl
================
Managing Bro with BroControl
============================
BroControl is an interactive shell for easily operating/managing Bro
installations on a single system or even across multiple systems in a
@ -68,9 +60,10 @@ policy and output the results in ``$PREFIX/logs``.
.. note:: The user starting BroControl needs permission to capture
network traffic. If you are not root, you may need to grant further
privileges to the account you're using; see the :doc:`FAQ <faq>`.
Also, if it looks like Bro is not seeing any traffic, check out
the FAQ entry on checksum offloading.
privileges to the account you're using; see the `FAQ
<http://www.bro.org/documentation/faq.html>`_. Also, if it looks
like Bro is not seeing any traffic, check out the FAQ entry on
checksum offloading.
You can leave it running for now, but to stop this Bro instance you would do:
@ -115,20 +108,18 @@ columns (shortened for brevity) show a request to the root of Bro website::
Some logs are worth explicit mention:
``weird.log``
Contains unusual/exceptional activity that can indicate
malformed connections, traffic that doesn't conform to a particular
protocol, malfunctioning/misconfigured hardware, or even an attacker
attempting to avoid/confuse a sensor. Without context, it's hard to
judge whether this category of activity is interesting and so that is
left up to the user to configure.
``conn.log``
Contains an entry for every connection seen on the wire, with
basic properties such as time and duration, originator and
responder IP addresses, services and ports, payload size, and
much more. This log provides a comprehensive record of the
network's activity.
``notice.log``
Identifies specific activity that Bro recognizes as
potentially interesting, odd, or bad. In Bro-speak, such
activity is called a "notice".
By default, ``BroControl`` regularly takes all the logs from
``$PREFIX/logs/current`` and archives/compresses them to a directory
named by date, e.g. ``$PREFIX/logs/2011-10-06``. The frequency at
@ -270,14 +261,11 @@ that only takes the email action for SSH logins to a defined set of servers:
192.168.1.102,
} &redef;
redef Notice::policy += {
[$action = Notice::ACTION_EMAIL,
$pred(n: Notice::Info) =
{
return n$note == SSH::Login && n$id$resp_h in watched_servers;
}
]
};
hook Notice::policy(n: Notice::Info)
{
if ( n$note == SSH::SUCCESSFUL_LOGIN && n$id$resp_h in watched_servers )
add n$actions[Notice::ACTION_EMAIL];
}
You'll just have to trust the syntax for now, but what we've done is
first declare our own variable to hold a set of watched addresses,
@ -302,21 +290,26 @@ tweak the most basic options. Here's some suggestions on what to explore next:
* We only looked at how to change options declared in the notice framework,
there's many more options to look at in other script packages.
* Continue reading with :ref:`using-bro` chapter which goes into more
depth on working with Bro; then look at :ref:`writing-scripts` for
learning how to start writing your own scripts.
* Look at the scripts in ``$PREFIX/share/bro/policy`` for further ones
you may want to load.
you may want to load; you can browse their documentation at the
:ref:`overview of script packages <script-packages>`.
* Reading the code of scripts that ship with Bro is also a great way to gain
understanding of the language and how you can start writing your own custom
analysis.
* Review the :doc:`FAQ <faq>`.
further understanding of the language and how scripts tend to be
structured.
* Review the `FAQ <http://www.bro.org/documentation/faq.html>`_.
* Continue reading below for another mini-tutorial on using Bro as a standalone
command-line utility.
Bro, the Command-Line Utility
Bro as a Command-Line Utility
=============================
If you prefer not to use BroControl (e.g. don't need its automation and
management features), here's how to directly control Bro for your analysis
activities.
If you prefer not to use BroControl (e.g. don't need its automation
and management features), here's how to directly control Bro for your
analysis activities from the command line for both live traffic and
offline working from traces.
Monitoring Live Traffic
-----------------------
@ -420,3 +413,19 @@ are "load this script if it hasn't already been loaded".
to be searched for scripts. See the default search path by doing
``bro --help``.
Running Bro Without Installing
------------------------------
For developers that wish to run Bro directly from the ``build/``
directory (i.e., without performing ``make install``), they will have
to first adjust ``BROPATH`` and ``BROMAGIC`` to look for scripts and
additional files inside the build directory. Sourcing either
``build/bro-path-dev.sh`` or ``build/bro-path-dev.csh`` as appropriate
for the current shell accomplishes this and also augments your
``PATH`` so you can use the Bro binary directly::
./configure
make
source build/bro-path-dev.sh
bro <options>

194
doc/reporting-problems.rst
View file

@ -1,194 +0,0 @@
Reporting Problems
==================
.. rst-class:: opening
Here we summarize some steps to follow when you see Bro doing
something it shouldn't. To provide help, it is often crucial for
us to have a way of reliably reproducing the effect you're seeing.
Unfortunately, reproducing problems can be rather tricky with Bro
because more often than not, they occur only in either very rare
situations or only after Bro has been running for some time. In
particular, getting a small trace showing a specific effect can be
a real problem. In the following, we'll summarize some strategies
to this end.
Reporting Problems
------------------
Generally, when you encounter a problem with Bro, the best thing to do
is opening a new ticket in `Bro's issue tracker
<http://tracker.bro.org/>`__ and include information on how to
reproduce the issue. Ideally, your ticket should come with the
following:
* The Bro version you're using (if working directly from the git
repository, the branch and revision number.)
* The output you're seeing along with a description of what you'd expect
Bro to do instead.
* A *small* trace in `libpcap format <http://www.tcpdump.org>`__
demonstrating the effect (assuming the problem doesn't happen right
at startup already).
* The exact command-line you're using to run Bro with that trace. If
you can, please try to run the Bro binary directly from the command
line rather than using BroControl.
* Any non-standard scripts you're using (but please only those really
necessary; just a small code snippet triggering the problem would
be perfect).
* If you encounter a crash, information from the core dump, such as
the stack backtrace, can be very helpful. See below for more on
this.
How Do I Get a Trace File?
--------------------------
As Bro is usually running live, coming up with a small trace file that
reproduces a problem can turn out to be quite a challenge. Often it
works best to start with a large trace that triggers the problem,
and then successively thin it out as much as possible.
To get to the initial large trace, here are a few things you can try:
* Capture a trace with `tcpdump <http://www.tcpdump.org/>`__, either
on the same interface Bro is running on, or on another host where
you can generate traffic of the kind likely triggering the problem
(e.g., if you're seeing problems with the HTTP analyzer, record some
of your Web browsing on your desktop.) When using tcpdump, don't
forget to record *complete* packets (``tcpdump -s 0 ...``). You can
reduce the amount of traffic captured by using a suitable BPF filter
(e.g., for HTTP only, try ``port 80``).
* Bro's command-line option ``-w <trace>`` records all packets it
processes into the given file. You can then later run Bro
offline on this trace and it will process the packets in the same
way as it did live. This is particularly helpful with problems that
only occur after Bro has already been running for some time. For
example, sometimes a crash may be triggered by a particular kind of
traffic only occurring rarely. Running Bro live with ``-w`` and
then, after the crash, offline on the recorded trace might, with a
little bit of luck, reproduce the problem reliably. However, be
careful with ``-w``: it can result in huge trace files, quickly
filling up your disk. (One way to mitigate the space issues is to
periodically delete the trace file by configuring
``rotate-logs.bro`` accordingly. BroControl does that for you if you
set its ``SaveTraces`` option.)
* Finally, you can try running Bro on a publically available trace
file, such as `anonymized FTP traffic <http://www-nrg.ee.lbl.gov
/anonymized-traces.html>`__, `headers-only enterprise traffic
<http://www.icir.org/enterprise-tracing/Overview.html>`__, or
`Defcon traffic <http://cctf.shmoo.com/>`__. Some of these
particularly stress certain components of Bro (e.g., the Defcon
traces contain tons of scans).
Once you have a trace that demonstrates the effect, you will often
notice that it's pretty big, in particular if recorded from the link
you're monitoring. Therefore, the next step is to shrink its size as
much as possible. Here are a few things you can try to this end:
* Very often, a single connection is able to demonstrate the problem.
If you can identify which one it is (e.g., from one of Bro's
``*.log`` files) you can extract the connection's packets from the
trace using tcpdump by filtering for the corresponding 4-tuple of
addresses and ports:
.. console::
> tcpdump -r large.trace -w small.trace host <ip1> and port <port1> and host <ip2> and port <port2>
* If you can't reduce the problem to a connection, try to identify
either a host pair or a single host triggering it, and filter down
the trace accordingly.
* You can try to extract a smaller time slice from the trace using
`TCPslice <http://www.tcpdump.org/related.html>`__. For example, to
extract the first 100 seconds from the trace:
.. console::
# Test comment
> tcpslice +100 <in >out
Alternatively, tcpdump extracts the first ``n`` packets with its
option ``-c <n>``.
Getting More Information After a Crash
--------------------------------------
If Bro crashes, a *core dump* can be very helpful to nail down the
problem. Examining a core is not for the faint of heart but can reveal
extremely useful information.
First, you should configure Bro with the option ``--enable-debug`` and
recompile; this will disable all compiler optimizations and thus make
the core dump more useful (don't expect great performance with this
version though; compiling Bro without optimization has a noticeable
impact on its CPU usage.). Then enable core dumps if you haven't
already (e.g., ``ulimit -c unlimited`` if you're using bash).
Once Bro has crashed, start gdb with the Bro binary and the file
containing the core dump. (Alternatively, you can also run Bro
directly inside gdb instead of working from a core file.) The first
helpful information to include with your tracker ticket is a stack
backtrace, which you get with gdb's ``bt`` command:
.. console::
> gdb bro core
[...]
> bt
If the crash occurs inside Bro's script interpreter, the next thing to
do is identifying the line of script code processed just before the
abnormal termination. Look for methods in the stack backtrace which
belong to any of the script interpreter's classes. Roughly speaking,
these are all classes with names ending in ``Expr``, ``Stmt``, or
``Val``. Then climb up the stack with ``up`` until you reach the first
of these methods. The object to which ``this`` is pointing will have a
``Location`` object, which in turn contains the file name and line
number of the corresponding piece of script code. Continuing the
example from above, here's how to get that information:
.. console::
[in gdb]
> up
> ...
> up
> print this->location->filename
> print this->location->first_line
If the crash occurs while processing input packets but you cannot
directly tell which connection is responsible (and thus not extract
its packets from the trace as suggested above), try getting the
4-tuple of the connection currently being processed from the core dump
by again examining the stack backtrace, this time looking for methods
belonging to the ``Connection`` class. That class has members
``orig_addr``/``resp_addr`` and ``orig_port``/``resp_port`` storing
(pointers to) the IP addresses and ports respectively:
.. console::
[in gdb]
> up
> ...
> up
> printf "%08x:%04x %08x:%04x\n", *this->orig_addr, this->orig_port, *this->resp_addr, this->resp_port
Note that these values are stored in `network byte order
<http://en.wikipedia.org/wiki/Endianness#Endianness_in_networking>`__
so you will need to flip the bytes around if you are on a low-endian
machine (which is why the above example prints them in hex). For
example, if an IP address prints as ``0100007f`` , that's 127.0.0.1 .

6
doc/scripting/connection_record_01.bro Normal file
View file

@ -0,0 +1,6 @@
@load base/protocols/conn
event connection_state_remove(c: connection)
{
print c;
}

7
doc/scripting/connection_record_02.bro Normal file
View file

@ -0,0 +1,7 @@
@load base/protocols/conn
@load base/protocols/dns
event connection_state_remove(c: connection)
{
print c;
}

22
doc/scripting/data_struct_record_01.bro Normal file
View file

@ -0,0 +1,22 @@
type Service: record {
name: string;
ports: set[port];
rfc: count;
};
function print_service(serv: Service): string
{
print fmt("Service: %s(RFC%d)",serv$name, serv$rfc);
for ( p in serv$ports )
print fmt(" port: %s", p);
}
event bro_init()
{
local dns: Service = [$name="dns", $ports=set(53/udp, 53/tcp), $rfc=1035];
local http: Service = [$name="http", $ports=set(80/tcp, 8080/tcp), $rfc=2616];
print_service(dns);
print_service(http);
}

41
doc/scripting/data_struct_record_02.bro Normal file
View file

@ -0,0 +1,41 @@
type Service: record {
name: string;
ports: set[port];
rfc: count;
};
type System: record {
name: string;
services: set[Service];
};
function print_service(serv: Service): string
{
print fmt(" Service: %s(RFC%d)",serv$name, serv$rfc);
for ( p in serv$ports )
print fmt(" port: %s", p);
}
function print_system(sys: System): string
{
print fmt("System: %s", sys$name);
for ( s in sys$services )
print_service(s);
}
event bro_init()
{
local server01: System;
server01$name = "morlock";
add server01$services[[ $name="dns", $ports=set(53/udp, 53/tcp), $rfc=1035]];
add server01$services[[ $name="http", $ports=set(80/tcp, 8080/tcp), $rfc=2616]];
print_system(server01);
# local dns: Service = [ $name="dns", $ports=set(53/udp, 53/tcp), $rfc=1035];
# local http: Service = [ $name="http", $ports=set(80/tcp, 8080/tcp), $rfc=2616];
# print_service(dns);
# print_service(http);
}

22
doc/scripting/data_struct_set_declaration.bro Normal file
View file

@ -0,0 +1,22 @@
event bro_init()
{
local ssl_ports: set[port];
local non_ssl_ports = set( 23/tcp, 80/tcp, 143/tcp, 25/tcp );
# SSH
add ssl_ports[22/tcp];
# HTTPS
add ssl_ports[443/tcp];
# IMAPS
add ssl_ports[993/tcp];
# Check for SMTPS
if ( 587/tcp !in ssl_ports )
add ssl_ports[587/tcp];
for ( i in ssl_ports )
print fmt("SSL Port: %s", i);
for ( i in non_ssl_ports )
print fmt("Non-SSL Port: %s", i);
}

13
doc/scripting/data_struct_table_complex.bro Normal file
View file

@ -0,0 +1,13 @@
event bro_init()
{
local samurai_flicks: table[string, string, count, string] of string;
samurai_flicks["Kihachi Okamoto", "Toho", 1968, "Tatsuya Nakadai"] = "Kiru";
samurai_flicks["Hideo Gosha", "Fuji", 1969, "Tatsuya Nakadai"] = "Goyokin";
samurai_flicks["Masaki Kobayashi", "Shochiku Eiga", 1962, "Tatsuya Nakadai" ] = "Harakiri";
samurai_flicks["Yoji Yamada", "Eisei Gekijo", 2002, "Hiroyuki Sanada" ] = "Tasogare Seibei";
for ( [d, s, y, a] in samurai_flicks )
print fmt("%s was released in %d by %s studios, directed by %s and starring %s", samurai_flicks[d, s, y, a], y, s, d, a);
}

13
doc/scripting/data_struct_table_declaration.bro Normal file
View file

@ -0,0 +1,13 @@
event bro_init()
{
local ssl_services: table[string] of port;
ssl_services = table(["SSH"] = 22/tcp, ["HTTPS"] = 443/tcp);
ssl_services["IMAPS"] = 993/tcp;
if ( "SMTPS" !in ssl_services )
ssl_services["SMTPS"] = 587/tcp;
for ( k in ssl_services )
print fmt("Service Name: %s - Common Port: %s", k, ssl_services[k]);
}

7
doc/scripting/data_struct_vector.bro Normal file
View file

@ -0,0 +1,7 @@
event bro_init()
{
local v: vector of count = vector(1, 2, 3, 4);
local w = vector(1, 2, 3, 4);
print v;
print w;
}

15
doc/scripting/data_struct_vector_declaration.bro Normal file
View file

@ -0,0 +1,15 @@
event bro_init()
{
local v1: vector of count;
local v2 = vector(1, 2, 3, 4);
v1[|v1|] = 1;
v1[|v1|] = 2;
v1[|v1|] = 3;
v1[|v1|] = 4;
print fmt("contents of v1: %s", v1);
print fmt("length of v1: %d", |v1|);
print fmt("contents of v1: %s", v2);
print fmt("length of v2: %d", |v2|);
}

7
doc/scripting/data_struct_vector_iter.bro Normal file
View file

@ -0,0 +1,7 @@
event bro_init()
{
local addr_vector: vector of addr = vector(1.2.3.4, 2.3.4.5, 3.4.5.6);
for (i in addr_vector)
print mask_addr(addr_vector[i], 18);
}

9
doc/scripting/data_type_const.bro Normal file
View file

@ -0,0 +1,9 @@
const port_list: table[port] of string &redef;
redef port_list += { [6666/tcp] = "IRC"};
redef port_list += { [80/tcp] = "WWW" };
event bro_init()
{
print port_list;
}

4
doc/scripting/data_type_const_simple.bro Normal file
View file

@ -0,0 +1,4 @@
@load base/protocols/http
redef HTTP::default_capture_password = T;

9
doc/scripting/data_type_declaration.bro Normal file
View file

@ -0,0 +1,9 @@
event bro_init()
{
local a: int;
a = 10;
local b = 10;
if ( a == b )
print fmt("A: %d, B: %d", a, b);
}

18
doc/scripting/data_type_interval.bro Normal file
View file

@ -0,0 +1,18 @@
# Store the time the previous connection was established.
global last_connection_time: time;
# boolean value to indicate whether we have seen a previous connection.
global connection_seen: bool = F;
event connection_established(c: connection)
{
local net_time: time = network_time();
print fmt("%s: New connection established from %s to %s", strftime("%Y/%M/%d %H:%m:%S", net_time), c$id$orig_h, c$id$resp_h);
if ( connection_seen )
print fmt(" Time since last connection: %s", net_time - last_connection_time);
last_connection_time = net_time;
connection_seen = T;
}

11
doc/scripting/data_type_local.bro Normal file
View file

@ -0,0 +1,11 @@
function add_two(i: count): count
{
local added_two = i+2;
print fmt("i + 2 = %d", added_two);
return added_two;
}
event bro_init()
{
local test = add_two(10);
}

13
doc/scripting/data_type_pattern_01.bro Normal file
View file

@ -0,0 +1,13 @@
event bro_init()
{
local test_string = "The quick brown fox jumped over the lazy dog.";
local test_pattern = /quick|lazy/;
if ( test_pattern in test_string )
{
local results = split(test_string, test_pattern);
print results[1];
print results[2];
print results[3];
}
}

10
doc/scripting/data_type_pattern_02.bro Normal file
View file

@ -0,0 +1,10 @@
event bro_init()
{
local test_string = "equality";
local test_pattern = /equal/;
print fmt("%s and %s %s equal", test_string, test_pattern, test_pattern == test_string ? "are" : "are not");
test_pattern = /equality/;
print fmt("%s and %s %s equal", test_string, test_pattern, test_pattern == test_string ? "are" : "are not");
}

15
doc/scripting/data_type_subnets.bro Normal file
View file

@ -0,0 +1,15 @@
event bro_init()
{
local subnets = vector(172.16.0.0/20, 172.16.16.0/20, 172.16.32.0/20, 172.16.48.0/20);
local addresses = vector(172.16.4.56, 172.16.47.254, 172.16.22.45, 172.16.1.1);
for ( a in addresses )
{
for ( s in subnets )
{
if ( addresses[a] in subnets[s] )
print fmt("%s belongs to subnet %s", addresses[a], subnets[s]);
}
}
}

4
doc/scripting/data_type_time.bro Normal file
View file

@ -0,0 +1,4 @@
event connection_established(c: connection)
{
print fmt("%s: New connection established from %s to %s\n", strftime("%Y/%M/%d %H:%m:%S", network_time()), c$id$orig_h, c$id$resp_h);
}

19
doc/scripting/framework_logging_factorial_01.bro Normal file
View file

@ -0,0 +1,19 @@
module Factor;
function factorial(n: count): count
{
if ( n == 0 )
return 1;
else
return ( n * factorial(n - 1) );
}
event bro_init()
{
local numbers: vector of count = vector(1, 2, 3, 4, 5, 6, 7, 8, 9, 10);
for ( n in numbers )
print fmt("%d", factorial(numbers[n]));
}

32
doc/scripting/framework_logging_factorial_02.bro Normal file
View file

@ -0,0 +1,32 @@
module Factor;
export {
redef enum Log::ID += { LOG };
type Info: record {
num: count &log;
factorial_num: count &log;
};
}
function factorial(n: count): count
{
if ( n == 0 )
return 1;
else
return ( n * factorial(n - 1) );
}
event bro_init()
{
Log::create_stream(LOG, [$columns=Info]);
}
event bro_done()
{
local numbers: vector of count = vector(1, 2, 3, 4, 5, 6, 7, 8, 9, 10);
for ( n in numbers )
Log::write( Factor::LOG, [$num=numbers[n],
$factorial_num=factorial(numbers[n])]);
}

45
doc/scripting/framework_logging_factorial_03.bro Normal file
View file

@ -0,0 +1,45 @@
module Factor;
export {
redef enum Log::ID += { LOG };
type Info: record {
num: count &log;
factorial_num: count &log;
};
}
function factorial(n: count): count
{
if ( n == 0 )
return 1;
else
return (n * factorial(n - 1));
}
event bro_done()
{
local numbers: vector of count = vector(1, 2, 3, 4, 5, 6, 7, 8, 9, 10);
for ( n in numbers )
Log::write( Factor::LOG, [$num=numbers[n],
$factorial_num=factorial(numbers[n])]);
}
function mod5(id: Log::ID, path: string, rec: Factor::Info) : string
{
if ( rec$factorial_num % 5 == 0 )
return "factor-mod5";
else
return "factor-non5";
}
event bro_init()
{
Log::create_stream(LOG, [$columns=Info]);
local filter: Log::Filter = [$name="split-mod5s", $path_func=mod5];
Log::add_filter(Factor::LOG, filter);
Log::remove_filter(Factor::LOG, "default");
}

50
doc/scripting/framework_logging_factorial_04.bro Normal file
View file

@ -0,0 +1,50 @@
module Factor;
export {
redef enum Log::ID += { LOG };
type Info: record {
num: count &log;
factorial_num: count &log;
};
global log_factor: event(rec: Info);
}
function factorial(n: count): count
{
if ( n == 0 )
return 1;
else
return (n * factorial(n - 1));
}
event bro_init()
{
Log::create_stream(LOG, [$columns=Info, $ev=log_factor]);
}
event bro_done()
{
local numbers: vector of count = vector(1, 2, 3, 4, 5, 6, 7, 8, 9, 10);
for ( n in numbers )
Log::write( Factor::LOG, [$num=numbers[n],
$factorial_num=factorial(numbers[n])]);
}
function mod5(id: Log::ID, path: string, rec: Factor::Info) : string
{
if ( rec$factorial_num % 5 == 0 )
return "factor-mod5";
else
return "factor-non5";
}
event bro_init()
{
local filter: Log::Filter = [$name="split-mod5s", $path_func=mod5];
Log::add_filter(Factor::LOG, filter);
Log::remove_filter(Factor::LOG, "default");
}

7
doc/scripting/framework_notice_hook_01.bro Normal file
View file

@ -0,0 +1,7 @@
@load policy/protocols/ssh/interesting-hostnames.bro
hook Notice::policy(n: Notice::Info)
{
if ( n$note == SSH::Interesting_Hostname_Login )
add n$actions[Notice::ACTION_EMAIL];
}

7
doc/scripting/framework_notice_hook_suppression_01.bro Normal file
View file

@ -0,0 +1,7 @@
@load policy/protocols/ssl/expiring-certs.bro
hook Notice::policy(n: Notice::Info)
{
if ( n$note == SSL::Certificate_Expires_Soon )
n$suppress_for = 12hrs;
}

8
doc/scripting/framework_notice_shortcuts_01.bro Normal file
View file

@ -0,0 +1,8 @@
@load policy/protocols/ssh/interesting-hostnames.bro
@load base/protocols/ssh/
redef Notice::emailed_types += {
SSH::Interesting_Hostname_Login,
SSH::Login
};

7
doc/scripting/framework_notice_shortcuts_02.bro Normal file
View file

@ -0,0 +1,7 @@
@load policy/protocols/ssh/interesting-hostnames.bro
@load base/protocols/ssh/
redef Notice::type_suppression_intervals += {
[SSH::Interesting_Hostname_Login] = 1day,
[SSH::Login] = 12hrs,
};

1313
doc/scripting/index.rst Normal file
View file

File diff suppressed because it is too large Load diff

4
doc/scripting/using_bro_sandbox_01 Normal file
View file

@ -0,0 +1,4 @@
# @TEST-EXEC: bro -r ${TRACES}/wikipedia.trace
# @TEST-EXEC: btest-diff conn.log
# @TEST-EXEC: btest-diff http.log

Some files were not shown because too many files have changed in this diff Show more