Mirror/zeek
1
0
Fork 0
mirror of https://github.com/zeek/zeek.git synced 2025-10-02 06:38:20 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 1

More doc reorg, and a light pass over the first 3 sections.

Browse source
This commit is contained in:
Robin Sommer 2013-08-19 22:08:30 -07:00
parent 534d4934b7
commit 867e4b52d8
34 changed files with 995 additions and 737 deletions
Download patch file Download diff file Expand all files Collapse all files

313
INSTALL
View file

@ -1,314 +1,3 @@
.. _CMake: http://www.cmake.org
.. _SWIG: http://www.swig.org
.. _Xcode: https://developer.apple.com/xcode/
.. _MacPorts: http://www.macports.org
.. _Fink: http://www.finkproject.org
.. _Homebrew: http://mxcl.github.com/homebrew
.. _bro downloads page: http://bro.org/download/index.html
============== See doc/install/install.rst for more installation instructions.
Installing Bro
==============
Bro can be downloaded in either pre-built binary package or
source code forms.
Prerequisites
=============
Bro requires the following libraries and tools to be installed
before you begin:
* Libpcap http://www.tcpdump.org
* OpenSSL libraries http://www.openssl.org
* BIND8 library
* Libmagic
* Libz
* Bash (for BroControl)
To build Bro from source, the following additional dependencies are required:
* CMake 2.6.3 or greater http://www.cmake.org
* SWIG http://www.swig.org
* Bison (GNU Parser Generator)
* Flex (Fast Lexical Analyzer)
* Libpcap headers http://www.tcpdump.org
* OpenSSL headers http://www.openssl.org
* libmagic headers
* zlib headers
* Perl
Bro can make use of some optional libraries and tools if they are found at
build time:
* LibGeoIP (for geo-locating IP addresses)
* gperftools (tcmalloc is used to improve memory and CPU usage)
* sendmail (for BroControl)
* ipsumdump (for trace-summary) http://www.cs.ucla.edu/~kohler/ipsumdump
* Ruby executable, library, and headers (for Broccoli Ruby bindings)
Installing From Pre-Built Binary Release Packages
=================================================
See the `bro downloads page`_ for currently supported/targeted platforms.
* RPM
.. console::
sudo yum localinstall Bro-*.rpm
* DEB
.. console::
sudo gdebi Bro-*.deb
* MacOS Disk Image with Installer
Just open the ``Bro-*.dmg`` and then run the ``.pkg`` installer.
Everything installed by the package will go into ``/opt/bro``.
The primary install prefix for binary packages is ``/opt/bro``.
Non-MacOS packages that include BroControl also put variable/runtime
data (e.g. Bro logs) in ``/var/opt/bro``.
Installing From Source
======================
Required Dependencies
~~~~~~~~~~~~~~~~~~~~~
The following dependencies are required to build Bro:
* RPM/RedHat-based Linux:
.. console::
sudo yum install cmake make gcc gcc-c++ flex bison libpcap-devel openssl-devel python-devel swig zlib-devel file-devel
* DEB/Debian-based Linux:
.. console::
sudo apt-get install cmake make gcc g++ flex bison libpcap-dev libssl-dev python-dev swig zlib1g-dev libmagic-dev
* FreeBSD
Most required dependencies should come with a minimal FreeBSD install
except for the following.
.. console::
sudo pkg_add -r bash cmake swig bison python
Note that ``bash`` needs to be in ``PATH``, which by default it is
not. The FreeBSD package installs the binary into
``/usr/local/bin``.
* Mac OS X
Compiling source code on Macs requires first downloading Xcode_,
then going through its "Preferences..." -> "Downloads" menus to
install the "Command Line Tools" component.
Lion (10.7) and Mountain Lion (10.8) come with all required
dependencies except for CMake_, SWIG_, and ``libmagic``.
Distributions of these dependencies can likely be obtained from your
preferred Mac OS X package management system (e.g. MacPorts_, Fink_,
or Homebrew_).
Specifically for MacPorts, the ``swig``, ``swig-ruby``, ``swig-python``
and ``file`` packages provide the required dependencies.
Optional Dependencies
~~~~~~~~~~~~~~~~~~~~~
Bro can use libGeoIP for geo-locating IP addresses, and sendmail for
sending emails.
* RedHat Enterprise Linux:
.. console::
sudo yum install geoip-devel sendmail
* CentOS Linux:
.. console::
sudo yum install GeoIP-devel sendmail
* DEB/Debian-based Linux:
.. console::
sudo apt-get install libgeoip-dev sendmail
* Ports-based FreeBSD
.. console::
sudo pkg_add -r GeoIP
sendmail is typically already available.
* Mac OS X
Vanilla OS X installations don't ship with libGeoIP, but
if installed from your preferred package management system (e.g. MacPorts,
Fink, or Homebrew), they should be automatically detected and Bro will
compile against them.
Additional steps may be needed to :doc:`get the right GeoIP database <geoip>`.
Compiling Bro Source Code
~~~~~~~~~~~~~~~~~~~~~~~~~
Bro releases are bundled into source packages for convenience and
available from the `bro downloads page`_.
Alternatively, the latest Bro development version can be obtained through git
repositories hosted at `git.bro.org <http://git.bro.org>`_. See
our `git development documentation
<http://bro.org/development/process.html>`_ for comprehensive
information on Bro's use of git revision control, but the short story
for downloading the full source code experience for Bro via git is:
.. console::
git clone --recursive git://git.bro.org/bro
.. note:: If you choose to clone the ``bro`` repository non-recursively for
a "minimal Bro experience", be aware that compiling it depends on
BinPAC, which has its own ``binpac`` repository. Either install it
first or initialize/update the cloned ``bro`` repository's
``aux/binpac`` submodule.
The typical way to build and install from source is (for more options,
run ``./configure --help``):
.. console::
./configure
make
make install
The default installation path is ``/usr/local/bro``, which would typically
require root privileges when doing the ``make install``. A different
installation path can be chosen by specifying the ``--prefix`` option.
Note that ``/usr`` and ``/opt/bro`` are the
standard prefixes for binary Bro packages to be installed, so those are
typically not good choices unless you are creating such a package.
Depending on the Bro package you downloaded, there may be auxiliary
tools and libraries available in the ``aux/`` directory. Some of them
will be automatically built and installed along with Bro. There are
``--disable-*`` options that can be given to the configure script to
turn off unwanted auxiliary projects that would otherwise be installed
automatically. Finally, use ``make install-aux`` to install some of
the other programs that are in the ``aux/bro-aux`` directory.
OpenBSD users, please see our FAQ at
http://www.bro.org/documentation/faq.html if you are having
problems installing Bro.
Upgrading From a Previous Version of Bro
========================================
If you're doing an upgrade install (rather than a fresh install),
there's two suggested approaches: either install Bro using the same
installation prefix directory as before, or pick a new prefix and copy
local customizations over.
Re-Use Previous Install Prefix
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
If you choose to configure and install Bro with the same prefix
directory as before, local customization and configuration to files in
``$prefix/share/bro/site`` and ``$prefix/etc`` won't be overwritten
(``$prefix`` indicating the root of where Bro was installed). Also, logs
generated at run-time won't be touched by the upgrade. (But making
a backup of local changes before upgrading is still recommended.)
After upgrading, remember to check ``$prefix/share/bro/site`` and
``$prefix/etc`` for ``.example`` files, which indicate the
distribution's version of the file differs from the local one, which may
include local changes. Review the differences, and make adjustments
as necessary (for differences that aren't the result of a local change,
use the new version's).
Pick a New Install prefix
~~~~~~~~~~~~~~~~~~~~~~~~~
If you want to install the newer version in a different prefix
directory than before, you can just copy local customization and
configuration files from ``$prefix/share/bro/site`` and ``$prefix/etc``
to the new location (``$prefix`` indicating the root of where Bro was
originally installed). Make sure to review the files for difference
before copying and make adjustments as necessary (for differences that
aren't the result of a local change, use the new version's). Of
particular note, the copied version of ``$prefix/etc/broctl.cfg`` is
likely to need changes to the ``SpoolDir`` and ``LogDir`` settings.
Configure the Run-Time Environment
==================================
Just remember that you may need to adjust your ``PATH`` environment variable
according to the platform/shell/package you're using. For example:
Bourne-Shell Syntax:
.. console::
export PATH=/usr/local/bro/bin:$PATH
C-Shell Syntax:
.. console::
setenv PATH /usr/local/bro/bin:$PATH
Or substitute ``/opt/bro/bin`` instead if you installed from a binary package.
Running Bro
===========
Bro is a complex program and it takes a bit of time to get familiar
with it. A good place for newcomers to start is the Quick Start Guide
at http://www.bro.org/documentation/quickstart.html.
For developers that wish to run Bro directly from the ``build/``
directory (i.e., without performing ``make install``), they will have
to first adjust ``BROPATH`` to look for scripts inside the build
directory. Sourcing either ``build/bro-path-dev.sh`` or
``build/bro-path-dev.csh`` as appropriate for the current shell
accomplishes this and also augments your ``PATH`` so you can use the
Bro binary directly::
./configure
make
source build/bro-path-dev.sh
bro <options>

352
NEWS
View file

@ -1,17 +1,14 @@
Release Notes
=============
This document summarizes the most important changes in the current Bro This document summarizes the most important changes in the current Bro
release. For a complete list of changes, see the ``CHANGES`` file release. For a complete list of changes, see the ``CHANGES`` file
(note that submodules, such as BroControl and Broccoli, come with (note that submodules, such as BroControl and Broccoli, come with
their own CHANGES.) their own CHANGES.)
Bro 2.2 Bro 2.2 (Work In Progress)
------- ==========================
New Functionality New Functionality
~~~~~~~~~~~~~~~~~ -----------------
- GPRS Tunnelling Protocol (GTPv1) decapsulation. - GPRS Tunnelling Protocol (GTPv1) decapsulation.
@ -147,8 +144,28 @@ New Functionality
base/utils/active-http.bro for providing an interface for querying base/utils/active-http.bro for providing an interface for querying
remote web servers. remote web servers.
- Summary statistics framework. [Extend]
- A number of new applications build on top of the summary statistics
framework:
* Scan detection: Detectors for port and address scans return. See
policy/misc/scan.bro.
* Tracerouter detector: policy/misc/detect-traceroute
* Web application detection/measurement: policy/misc/app-metrics.bro
* FTP brute-forcing detector: policy/protocols/ftp/detect-bruteforcing.bro
* HTTP-based SQL injection detector: policy/protocols/http/detect-sqli.bro
(existed before, but now ported to the new framework)
* SSH brute-forcing detector feeding the intelligence framework:
policy/protocols/ssh/detect-bruteforcing.bro
Changed Functionality Changed Functionality
~~~~~~~~~~~~~~~~~~~~~ ---------------------
- We removed the following, already deprecated, functionality: - We removed the following, already deprecated, functionality:
@ -231,11 +248,12 @@ Changed Functionality
- We removed the BitTorrent DPD signatures pending further updates to - We removed the BitTorrent DPD signatures pending further updates to
that analyzer. that analyzer.
Bro 2.1 Bro 2.1
------- =======
New Functionality New Functionality
~~~~~~~~~~~~~~~~~ -----------------
- Bro now comes with extensive IPv6 support. Past versions offered - Bro now comes with extensive IPv6 support. Past versions offered
only basic IPv6 functionality that was rarely used in practice as it only basic IPv6 functionality that was rarely used in practice as it
@ -314,30 +332,9 @@ New Functionality
outputs. We do not yet recommend them for production (but welcome outputs. We do not yet recommend them for production (but welcome
feedback!) feedback!)
- Summary statistics framework. [Extend]
- A number of new applications build on top of the summary statistics
framework:
* Scan detection: Detectors for port and address scans return. See
policy/misc/scan.bro.
* Tracerouter detector: policy/misc/detect-traceroute
* Web application detection/measurement: policy/misc/app-metrics.bro
* FTP brute-forcing detector: policy/protocols/ftp/detect-bruteforcing.bro
* HTTP-based SQL injection detector: policy/protocols/http/detect-sqli.bro
(existed before, but now ported to the new framework)
* SSH brute-forcing detector feeding the intelligence framework:
policy/protocols/ssh/detect-bruteforcing.bro
Changed Functionality Changed Functionality
~~~~~~~~~~~~~~~~~~~~~ ---------------------
The following summarizes the most important differences in existing The following summarizes the most important differences in existing
functionality. Note that this list is not complete, see CHANGES for functionality. Note that this list is not complete, see CHANGES for
@ -371,7 +368,10 @@ the full set.
soon. With that, "match" and "using" are no longer reserved keywords. soon. With that, "match" and "using" are no longer reserved keywords.
- The syntax for IPv6 literals changed from "2607:f8b0:4009:802::1012" - The syntax for IPv6 literals changed from "2607:f8b0:4009:802::1012"
to "[2607:f8b0:4009:802::1012]". to "[2607:f8b0:4009:802::1012]". When an IP address variable or IP
address literal is enclosed in pipes (for example,
``|[fe80::db15]|``) the result is now the size of the address in
bits (32 for IPv4 and 128 for IPv6).
- Bro now spawns threads for doing its logging. From a user's - Bro now spawns threads for doing its logging. From a user's
perspective not much should change, except that the OS may now show perspective not much should change, except that the OS may now show
@ -411,60 +411,274 @@ the full set.
- The ASCII writers "header_*" options have been renamed to "meta_*" - The ASCII writers "header_*" options have been renamed to "meta_*"
(because there's now also a footer). (because there's now also a footer).
- Some built-in functions have been removed: "addr_to_count" (use
"addr_to_counts" instead), "bro_has_ipv6" (this is no longer
relevant because Bro now always supports IPv6), "active_connection"
(use "connection_exists" instead), and "connection_record" (use
"lookup_connection" instead).
- The "NFS3::mode2string" built-in function has been renamed to
"file_mode".
- Some built-in functions have been changed: "exit" (now takes the
exit code as a parameter), "to_port" (now takes a string as
parameter instead of a count and transport protocol, but
"count_to_port" is still available), "connect" (now takes an
additional string parameter specifying the zone of a non-global IPv6
address), and "listen" (now takes three additional parameters to
enable listening on IPv6 addresses).
- Some Bro script variables have been renamed:
"LogAscii::header_prefix" has been renamed to
"LogAscii::meta_prefix", "LogAscii::include_header" has been renamed
to "LogAscii::include_meta".
- Some Bro script variables have been removed: "tunnel_port",
"parse_udp_tunnels", "use_connection_compressor",
"cc_handle_resets", "cc_handle_only_syns", and
"cc_instantiate_on_data".
- A couple events have changed: the "icmp_redirect" event now includes
the target and destination addresses and any Neighbor Discovery
options in the message, and the last parameter of the
"dns_AAAA_reply" event has been removed because it was unused.
- The format of the ASCII log files has changed very slightly. Two
new lines are automatically added, one to record the time when the
log was opened, and the other to record the time when the log was
closed.
- In BroControl, the option (in broctl.cfg) "CFlowAddr" was renamed to
"CFlowAddress".
Bro 2.0 Bro 2.0
------- =======
As the version number jump suggests, Bro 2.0 is a major upgrade and As the version number jump from 1.5 suggests, Bro 2.0 is a major
lots of things have changed. We have assembled a separate upgrade upgrade and lots of things have changed. Most importantly, we have
guide with the most important changes compared to Bro 1.5 at rewritten almost all of Bro's default scripts from scratch, using
http://www.bro.org/documentation/upgrade.html. You can find quite different structure now and focusing more on operational
the offline version of that document in ``doc/upgrade.rst.``. deployment. The result is a system that works much better "out of the
box", even without much initial site-specific configuration. The
down-side is that 1.x configurations will need to be adapted to work
with the new version. The two rules of thumb are:
Compared to the earlier 2.0 Beta version, the major changes in the (1) If you have written your own Bro scripts
final release are: that do not depend on any of the standard scripts formerly
found in ``policy/``, they will most likely just keep working
(although you might want to adapt them to use some of the new
features, like the new logging framework; see below).
* The default scripts now come with complete reference (2) If you have custom code that depends on specifics of 1.x
documentation. See default scripts (including most configuration tuning), that is
http://www.bro.org/documentation/index.html. unlikely to work with 2.x. We recommend to start by using just
the new scripts first, and then port over any customizations
incrementally as necessary (they may be much easier to do now,
or even unnecessary). Send mail to the Bro user mailing list
if you need help.
* libz and libmagic are now required dependencies. Below we summarize changes from 1.x to 2.x in more detail. This list
isn't complete, see the :download:`CHANGES <CHANGES>` file in the
distribution for the full story.
* Reduced snaplen default from 65535 to old default of 8192. The Script Organization
large value was introducing performance problems on many -------------------
systems.
* Replaced the --snaplen/-l command line option with a In versions before 2.0, Bro scripts were all maintained in a flat
scripting-layer option called "snaplen". The new option can also directory called ``policy/`` in the source tree. This directory is now
be redefined on the command line, e.g. ``bro -i eth0 renamed to ``scripts/`` and contains major subdirectories ``base/``,
snaplen=65535``. ``policy/``, and ``site/``, each of which may also be subdivided
further.
* Reintroduced the BRO_LOG_SUFFIX environment variable that the The contents of the new ``scripts/`` directory, like the old/flat
ASCII logger now respects to add a suffix to the log files it ``policy/`` still gets installed under the ``share/bro``
creates. subdirectory of the installation prefix path just like previous
versions. For example, if Bro was compiled like ``./configure
--prefix=/usr/local/bro && make && make install``, then the script
hierarchy can be found in ``/usr/local/bro/share/bro``.
* The ASCII logs now include further header information, and The main
fields set to an empty value are now logged as ``(empty)`` by subdirectories of that hierarchy are as follows:
default (instead of ``-``, which is already used for fields that
are not set at all).
* Some NOTICES were renamed, and the signatures of some SSL events - ``base/`` contains all scripts that are loaded by Bro by default
have changed. (unless the ``-b`` command line option is used to run Bro in a
minimal configuration). Note that is a major conceptual change:
rather than not loading anything by default, Bro now uses an
extensive set of default scripts out of the box.
* bro-cut got some new capabilities: The scripts under this directory generally either accumulate/log
useful state/protocol information for monitored traffic, configure a
default/recommended mode of operation, or provide extra Bro
scripting-layer functionality that has no significant performance cost.
- If no field names are given on the command line, we now pass - ``policy/`` contains all scripts that a user will need to explicitly
through all fields. tell Bro to load. These are scripts that implement
functionality/analysis that not all users may want to use and may have
more significant performance costs. For a new installation, you
should go through these and see what appears useful to load.
- New options -u/-U for time output in UTC. - ``site/`` remains a directory that can be used to store locally
developed scripts. It now comes with some preinstalled example
scripts that contain recommended default configurations going beyond
the ``base/`` setup. E.g. ``local.bro`` loads extra scripts from
``policy/`` and does extra tuning. These files can be customized in
place without being overwritten by upgrades/reinstalls, unlike
scripts in other directories.
- New option -F to give output field separator. With version 2.0, the default ``BROPATH`` is set to automatically
search for scripts in ``policy/``, ``site/`` and their parent
directory, but **not** ``base/``. Generally, everything under
``base/`` is loaded automatically, but for users of the ``-b`` option,
it's important to know that loading a script in that directory
requires the extra ``base/`` path qualification. For example, the
following two scripts:
* Broccoli supports more types internally, allowing to send * ``$PREFIX/share/bro/base/protocols/ssl/main.bro``
complex records. * ``$PREFIX/share/bro/policy/protocols/ssl/validate-certs.bro``
* Many smaller bug fixes, portability improvements, and general are referenced from another Bro script like:
polishing across all modules.
.. code:: bro
@load base/protocols/ssl/main
@load protocols/ssl/validate-certs
Notice how ``policy/`` can be omitted as a convenience in the second
case. ``@load`` can now also use relative path, e.g., ``@load
../main``.
Logging Framework
-----------------
- The logs generated by scripts that ship with Bro are entirely redone
to use a standardized, machine parsable format via the new logging
framework. Generally, the log content has been restructured towards
making it more directly useful to operations. Also, several
analyzers have been significantly extended and thus now log more
information. Take a look at ``ssl.log``.
* A particular format change that may be useful to note is that the
``conn.log`` ``service`` field is derived from DPD instead of
well-known ports (while that was already possible in 1.5, it was
not the default).
* Also, ``conn.log`` now reports raw number of packets/bytes per
endpoint.
- The new logging framework makes it possible to extend, customize,
and filter logs very easily. See the :doc:`logging framework <logging>`
for more information on usage.
- A common pattern found in the new scripts is to store logging stream
records for protocols inside the ``connection`` records so that
state can be collected until enough is seen to log a coherent unit
of information regarding the activity of that connection. This
state is now frequently seen/accessible in event handlers, for
example, like ``c$<protocol>`` where ``<protocol>`` is replaced by
the name of the protocol. This field is added to the ``connection``
record by ``redef``'ing it in a
``base/protocols/<protocol>/main.bro`` script.
- The logging code has been rewritten internally, with script-level
interface and output backend now clearly separated. While ASCII
logging is still the default, we will add further output types in
the future (binary format, direct database logging).
Notice Framework
----------------
The way users interact with "notices" has changed significantly in
order to make it easier to define a site policy and more extensible
for adding customized actions. See the :doc:`notice framework <notice>`.
New Default Settings
--------------------
- Dynamic Protocol Detection (DPD) is now enabled/loaded by default.
- The default packet filter now examines all packets instead of
dynamically building a filter based on which protocol analysis scripts
are loaded. See ``PacketFilter::all_packets`` for how to revert to old
behavior.
API Changes
-----------
- The ``@prefixes`` directive works differently now.
Any added prefixes are now searched for and loaded *after* all input
files have been parsed. After all input files are parsed, Bro
searches ``BROPATH`` for prefixed, flattened versions of all of the
parsed input files. For example, if ``lcl`` is in ``@prefixes``, and
``site.bro`` is loaded, then a file named ``lcl.site.bro`` that's in
``BROPATH`` would end up being automatically loaded as well. Packages
work similarly, e.g. loading ``protocols/http`` means a file named
``lcl.protocols.http.bro`` in ``BROPATH`` gets loaded automatically.
- The ``make_addr`` BIF now returns a ``subnet`` versus an ``addr``
Variable Naming
---------------
- ``Module`` is more widely used for namespacing. E.g. the new
``site.bro`` exports the ``local_nets`` identifier (among other
things) into the ``Site`` module.
- Identifiers may have been renamed to conform to new `scripting
conventions
<http://www.bro.org/development/script-conventions.html>`_
Removed Functionality
---------------------
We have remove a bunch of functionality that was rarely used and/or
had not been maintained for a while already:
- The ``net`` script data type.
- The ``alarm`` statement; use the notice framework instead.
- Trace rewriting.
- DFA state expiration in regexp engine.
- Active mapping.
- Native DAG support (may come back eventually)
- ClamAV support.
- The connection compressor is now disabled by default, and will
be removed in the future.
BroControl Changes
------------------
BroControl looks pretty much similar to the version coming with Bro 1.x,
but has been cleaned up and streamlined significantly internally.
BroControl has a new ``process`` command to process a trace on disk
offline using a similar configuration to what BroControl installs for
live analysis.
BroControl now has an extensive plugin interface for adding new
commands and options. Note that this is still considered experimental.
We have removed the ``analysis`` command, and BroControl currently
does not send daily alarm summaries anymore (this may be restored
later).
Development Infrastructure
--------------------------
Bro development has moved from using SVN to Git for revision control.
Users that want to use the latest Bro development snapshot by checking it out
from the source repositories should see the `development process
<http://www.bro.org/development/process.html>`_. Note that all the various
sub-components now reside in their own repositories. However, the
top-level Bro repository includes them as git submodules so it's easy
to check them all out simultaneously.
Bro now uses `CMake <http://www.cmake.org>`_ for its build system so
that is a new required dependency when building from source.
Bro now comes with a growing suite of regression tests in
``testing/``.

2
doc/frameworks/geoip.rst
View file

@ -1,4 +1,6 @@
.. _geolocation:
=========== ===========
GeoLocation GeoLocation
=========== ===========

7
doc/index.rst
View file

@ -9,6 +9,8 @@ Bro Documentation
:maxdepth: 2 :maxdepth: 2
intro/index.rst intro/index.rst
quickstart/index.rst
install/index.rst
using/index.rst using/index.rst
scripting/index.rst scripting/index.rst
frameworks/index.rst frameworks/index.rst
@ -20,3 +22,8 @@ Bro Documentation
section, but can't figure out how to include it into toctree) section, but can't figure out how to include it into toctree)
* :ref:`General Index <genindex>` * :ref:`General Index <genindex>`
* :ref:`search` * :ref:`search`
install
upgrade
changes
reporting-problems

1
doc/install/CHANGES-binpac.txt Symbolic link
View file

@ -0,0 +1 @@
../../aux/binpac/CHANGES

1
doc/install/CHANGES-bro-aux.txt Symbolic link
View file

@ -0,0 +1 @@
../../aux/bro-aux/CHANGES

1
doc/install/CHANGES-bro.txt Symbolic link
View file

@ -0,0 +1 @@
../../CHANGES

1
doc/install/CHANGES-broccoli-python.txt Symbolic link
View file

@ -0,0 +1 @@
../../aux/broccoli/bindings/broccoli-python/CHANGES

1
doc/install/CHANGES-broccoli-ruby.txt Symbolic link
View file

@ -0,0 +1 @@
../../aux/broccoli/bindings/broccoli-ruby/CHANGES

1
doc/install/CHANGES-broccoli.txt Symbolic link
View file

@ -0,0 +1 @@
../../aux/broccoli/CHANGES

1
doc/install/CHANGES-broctl.txt Symbolic link
View file

@ -0,0 +1 @@
../../aux/broctl/CHANGES

1
doc/install/CHANGES-btest.txt Symbolic link
View file

@ -0,0 +1 @@
../../aux/btest/CHANGES

1
doc/install/CHANGES-capstats.txt Symbolic link
View file

@ -0,0 +1 @@
../../aux/broctl/aux/capstats/CHANGES

1
doc/install/CHANGES-pysubnettree.txt Symbolic link
View file

@ -0,0 +1 @@
../../aux/broctl/aux/pysubnettree/CHANGES

1
doc/install/CHANGES-trace-summary.txt Symbolic link
View file

@ -0,0 +1 @@
../../aux/broctl/aux/trace-summary/CHANGES

1
doc/install/NEWS.rst Symbolic link
View file

@ -0,0 +1 @@
../../NEWS

75
doc/install/changes.rst Normal file
View file

@ -0,0 +1,75 @@
===========================================
Comprehensive Version History (aka CHANGES)
===========================================
.. contents::
---
Bro
---
.. literalinclude:: CHANGES-bro.txt
----------
BroControl
----------
.. literalinclude:: CHANGES-broctl.txt
--------
Broccoli
--------
.. literalinclude:: CHANGES-broccoli.txt
---------------
Broccoli Python
---------------
.. literalinclude:: CHANGES-broccoli-python.txt
-------------
Broccoli Ruby
-------------
.. literalinclude:: CHANGES-broccoli-ruby.txt
--------
Capstats
--------
.. literalinclude:: CHANGES-capstats.txt
-------------
Trace-Summary
-------------
.. literalinclude:: CHANGES-trace-summary.txt
------
BinPAC
------
.. literalinclude:: CHANGES-binpac.txt
-------
Bro-Aux
-------
.. literalinclude:: CHANGES-bro-aux.txt
-----
BTest
-----
.. literalinclude:: CHANGES-btest.txt
------------
PySubnetTree
------------
.. literalinclude:: CHANGES-pysubnettree.txt

43
doc/install/guidelines.rst Normal file
View file

@ -0,0 +1,43 @@
.. _upgrade-guidelines:
==================
General Guidelines
==================
If you're doing an upgrade install (rather than a fresh install),
there's two suggested approaches: either install Bro using the same
installation prefix directory as before, or pick a new prefix and copy
local customizations over. In the following we summarize general
guidelines for upgrading, see the `Release Notes <release-notes>`_ for
version-specific information.
Re-Using Previous Install Prefix
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
If you choose to configure and install Bro with the same prefix
directory as before, local customization and configuration to files in
``$prefix/share/bro/site`` and ``$prefix/etc`` won't be overwritten
(``$prefix`` indicating the root of where Bro was installed). Also, logs
generated at run-time won't be touched by the upgrade. (But making
a backup of local changes before upgrading is still recommended.)
After upgrading, remember to check ``$prefix/share/bro/site`` and
``$prefix/etc`` for ``.example`` files, which indicate the
distribution's version of the file differs from the local one, which may
include local changes. Review the differences, and make adjustments
as necessary (for differences that aren't the result of a local change,
use the new version's).
Using a New Install prefix
~~~~~~~~~~~~~~~~~~~~~~~~~~
If you want to install the newer version in a different prefix
directory than before, you can just copy local customization and
configuration files from ``$prefix/share/bro/site`` and ``$prefix/etc``
to the new location (``$prefix`` indicating the root of where Bro was
originally installed). Make sure to review the files for difference
before copying and make adjustments as necessary (for differences that
aren't the result of a local change, use the new version's). Of
particular note, the copied version of ``$prefix/etc/broctl.cfg`` is
likely to need changes to the ``SpoolDir`` and ``LogDir`` settings.

13
doc/install/index.rst Normal file
View file

@ -0,0 +1,13 @@
.. _installation:
============
Installation
============
.. toctree::
:maxdepth: 2
install
upgrade
reporting-problems

241
doc/install/install.rst Normal file
View file

@ -0,0 +1,241 @@
.. _CMake: http://www.cmake.org
.. _SWIG: http://www.swig.org
.. _Xcode: https://developer.apple.com/xcode/
.. _MacPorts: http://www.macports.org
.. _Fink: http://www.finkproject.org
.. _Homebrew: http://mxcl.github.com/homebrew
.. _bro downloads page: http://bro.org/download/index.html
.. _installing-bro:
==============
Installing Bro
==============
.. contents::
Prerequisites
=============
Before installing Bro, you'll need to ensure that some dependencies
are in place.
Required Dependencies
---------------------
Bro requires the following libraries and tools to be installed
before you begin:
* Libpcap (http://www.tcpdump.org)
* OpenSSL libraries (http://www.openssl.org)
* BIND8 library
* Libmagic
* Libz
* Bash (for BroControl)
To build Bro from source, the following additional dependencies are required:
* CMake 2.6.3 or greater (http://www.cmake.org)
* SWIG (http://www.swig.org)
* Bison (GNU Parser Generator)
* Flex (Fast Lexical Analyzer)
* Libpcap headers (http://www.tcpdump.org)
* OpenSSL headers (http://www.openssl.org)
* libmagic headers
* zlib headers
* Perl
To install the required dependencies, you can use:
* RPM/RedHat-based Linux:
.. console::
sudo yum install cmake make gcc gcc-c++ flex bison libpcap-devel openssl-devel python-devel swig zlib-devel file-devel
* DEB/Debian-based Linux:
.. console::
sudo apt-get install cmake make gcc g++ flex bison libpcap-dev libssl-dev python-dev swig zlib1g-dev libmagic-dev
* FreeBSD:
Most required dependencies should come with a minimal FreeBSD install
except for the following.
.. console::
sudo pkg_add -r bash cmake swig bison python
Note that ``bash`` needs to be in ``PATH``, which by default it is
not. The FreeBSD package installs the binary into
``/usr/local/bin``.
* Mac OS X:
Compiling source code on Macs requires first downloading Xcode_,
then going through its "Preferences..." -> "Downloads" menus to
install the "Command Line Tools" component.
Lion (10.7) and Mountain Lion (10.8) come with all required
dependencies except for CMake_, SWIG_, and ``libmagic``.
Distributions of these dependencies can likely be obtained from your
preferred Mac OS X package management system (e.g. MacPorts_, Fink_,
or Homebrew_).
Specifically for MacPorts, the ``swig``, ``swig-ruby``, ``swig-python``
and ``file`` packages provide the required dependencies.
Optional Dependencies
---------------------
Bro can make use of some optional libraries and tools if they are found at
build time:
* LibGeoIP (for geo-locating IP addresses)
* gperftools (tcmalloc is used to improve memory and CPU usage)
* ipsumdump (for trace-summary; http://www.cs.ucla.edu/~kohler/ipsumdump)
* Ruby executable, library, and headers (for Broccoli Ruby bindings)
LibGeoIP is probably the most interesting and can be easily installed
on most platforms:
* RedHat Enterprise Linux:
.. console::
sudo yum install geoip-devel sendmail
* CentOS Linux:
.. console::
sudo yum install GeoIP-devel sendmail
* DEB/Debian-based Linux:
.. console::
sudo apt-get install libgeoip-dev sendmail
* FreeBSD using ports:
.. console::
sudo pkg_add -r GeoIP
* Mac OS X:
Vanilla OS X installations don't ship with libGeoIP, but if
installed from your preferred package management system (e.g.
MacPorts, Fink, or Homebrew), they should be automatically detected
and Bro will compile against them.
Additional steps may be needed to :ref:`get the right GeoIP database
<geolocation>`.
Installing Bro
==============
Bro can be downloaded in either pre-built binary package or source
code forms.
Using Pre-Built Binary Release Packages
=======================================
See the `bro downloads page`_ for currently supported/targeted
platforms for binary releases.
* RPM
.. console::
sudo yum localinstall Bro-*.rpm
* DEB
.. console::
sudo gdebi Bro-*.deb
* MacOS Disk Image with Installer
Just open the ``Bro-*.dmg`` and then run the ``.pkg`` installer.
Everything installed by the package will go into ``/opt/bro``.
The primary install prefix for binary packages is ``/opt/bro``.
Non-MacOS packages that include BroControl also put variable/runtime
data (e.g. Bro logs) in ``/var/opt/bro``.
Installing From Source
==========================
Bro releases are bundled into source packages for convenience and
available from the `bro downloads page`_. Alternatively, the latest
Bro development version can be obtained through git repositories
hosted at ``git.bro.org``. See our `git development documentation
<http://bro.org/development/process.html>`_ for comprehensive
information on Bro's use of git revision control, but the short story
for downloading the full source code experience for Bro via git is:
.. console::
git clone --recursive git://git.bro.org/bro
.. note:: If you choose to clone the ``bro`` repository
non-recursively for a "minimal Bro experience", be aware that
compiling it depends on several of the other submodules as well.
The typical way to build and install from source is (for more options,
run ``./configure --help``):
.. console::
./configure
make
make install
The default installation path is ``/usr/local/bro``, which would typically
require root privileges when doing the ``make install``. A different
installation path can be chosen by specifying the ``--prefix`` option.
Note that ``/usr`` and ``/opt/bro`` are the
standard prefixes for binary Bro packages to be installed, so those are
typically not good choices unless you are creating such a package.
Depending on the Bro package you downloaded, there may be auxiliary
tools and libraries available in the ``aux/`` directory. Some of them
will be automatically built and installed along with Bro. There are
``--disable-*`` options that can be given to the configure script to
turn off unwanted auxiliary projects that would otherwise be installed
automatically. Finally, use ``make install-aux`` to install some of
the other programs that are in the ``aux/bro-aux`` directory.
OpenBSD users, please see our at `FAQ
<http://www.bro.org/documentation/faq.html>`_ if you are having
problems installing Bro.
Configure the Run-Time Environment
==================================
Just remember that you may need to adjust your ``PATH`` environment variable
according to the platform/shell/package you're using. For example:
Bourne-Shell Syntax:
.. console::
export PATH=/usr/local/bro/bin:$PATH
C-Shell Syntax:
.. console::
setenv PATH /usr/local/bro/bin:$PATH
Or substitute ``/opt/bro/bin`` instead if you installed from a binary package.

13
doc/install/release-notes.rst Normal file
View file

@ -0,0 +1,13 @@
.. _release-notes:
=============
Release Notes
=============
.. contents::
.. include:: NEWS.rst

4
doc/intro/reporting-problems.rst → doc/install/reporting-problems.rst
View file

@ -1,6 +1,6 @@
Reporting Problems Reporting Problems (TODO: Move to www)
================== ======================================
.. rst-class:: opening .. rst-class:: opening

10
doc/install/upgrade.rst Normal file
View file

@ -0,0 +1,10 @@
================================
Upgrading From Previous Versions
================================
.. toctree::
guidelines
release-notes
changes

BIN
doc/intro/architecture.png Normal file
View file

Binary file not shown.
Side by side

After

Width:  |  Height:  |  Size: 136 KiB

BIN
doc/intro/bro-eyes.png Normal file
View file

Binary file not shown.
Side by side

After

Width:  |  Height:  |  Size: 45 KiB

BIN
doc/intro/history.png Normal file
View file

Binary file not shown.
Side by side

After

Width:  |  Height:  |  Size: 159 KiB

248
doc/intro/index.rst
View file

@ -3,13 +3,245 @@
Introduction Introduction
============ ============
.. toctree:: .. contents::
:maxdepth: 2
overview Overview
quickstart --------
install
upgrade
changes
reporting-problems
Bro is a passive, open-source network traffic analyzer. It is
primarily a security monitor that inspects all traffic on a link in
depth for signs of suspicious activity. More generally, however,
Bro supports a wide range of traffic analysis tasks even outside of
the security domain, including performance measurements and helping
with trouble-shooting.
The most immediate benefit that a site gains from deploying Bro is an
extensive set of *log files* that record a network's activity in
high-level terms. These logs include not only a comprehensive record
of every connection seen on the wire, but also application-layer
transcripts such as, e.g., all HTTP sessions with their requested
URIs, key headers, MIME types, and server responses; DNS requests with
replies; SSL certificates; key content of SMTP sessions; and much
more. By default, Bro writes all this information into well-structured
tab-separated log files suitable for post-processing with external
software. Users can however also chose from a set of alternative
output formats and backends to interface directly with, e.g., external
databases.
In addition to the logs, Bro comes with built-in functionality for a
range of analysis and detection tasks, including extracting files from
HTTP sessions, detecting malware by interfacing to external
registries, reporting vulnerable versions of software seen on the
network, identifying popular web applications, detecting SSH
brute-forcing, validating SSL certificate chains, and much more.
However, the key to understanding Bro lies in realizing that even
though the system comes with such powerful functionality out of the
box, fundamentally it represents a *platform* for traffic analyses
that's fully customizable and extensible: Bro provides users with a
domain-specific, Turing-complete *scripting language* for expressing
arbitrary analysis tasks. Conceptually, you can think of Bro as a
"domain-specific Python" (or Perl): just like Python, the system comes
with a large set of pre-built functionality (the "standard library"),
yet you are not limited to what the system ships with but can put Bro
to use in novel ways by writing your own code. Indeed, all of Bro's
default analyses, including all the logging, is the result of such
scripts; there's no specific analysis hard-coded into the core of
system.
Bro runs on commodity hardware and hence provides a low-cost
alternative to expensive proprietary solutions. Despite the price tag,
however, Bro actually goes far beyond the capabilities of other
network monitoring tools, which typically remain limited to a small
set of hard-coded analysis tasks. We emphasize in particular that Bro
is *not* a classic signature-based intrusion detection system (IDS).
While it supports such standard functionality as well, Bro's scripting
language indeed facilitates a much broader spectrum of very different
approaches to finding malicious activity, including semantic misuse
detection, anomaly detection, and behavioral analysis.
A large variety of sites deploy Bro operationally for protecting their
cyberinfrastructure, including many universities, research labs,
supercomputing centers, open-science communities, and major
corporations. Bro specifically targets high-speed, high-volume network
monitoring, and an increasing number of sites are now using the system
to monitor their 10GE networks, with some already moving on to 100GE
links. Bro accommodates such high-performance settings by supporting
scalable load-balancing: large sites typically run "Bro Clusters" in
which a high-speed frontend load-balancer distributes the traffic
across an appropriate number of backend PCs, all running dedicated Bro
instances on their individual traffic slices. A central manager system
coordinates the process, synchronizing state across the backends and
providing the operators with a central management interface for
configuration and access to aggregated logs. Bro's integrated
management framework, BroControl, supports such cluster setups
out-of-the-box.
Features
--------
Bro supports a wide range of analyses through its scripting language.
Yet even without further customization it comes with a powerful set of
features.
- Deployment
* Runs on commodity hardware on standard UNIX-style systems
(including Linux, FreeBSD, and MacOS).
* Fully passive traffic analysis off a network tap or monitoring
port.
* Standard libpcap interface for capturing packets.
* Real-time and offline analysis.
* Cluster-support for large-scale deployments.
* Unified management framework for operating both standalone and
cluster setups.
* Open-source under a BSD license.
- Analysis
* Comprehensive logging of activity for offline analysis and
forensics.
* Port-independent analysis of application-layer protocols.
* Support for many application-layer protocols (including DNS,
FTP, HTTP, IRC, SMTP, SSH, SSL).
* Analysis of file content exchanged over application-layer
protocols, including MD5/SHA1 computation for fingerprinting.
* Comprehensive IPv6 support.
* Tunnel detection and analysis (including Ayiya, Teredo, GTPv1).
Bro decapsulates the tunnels and then proceeds to analyze their
content as if no tunnel was in place.
* Extensive sanity checks during protocol analysis.
* Support for IDS-style pattern matching.
- Scripting Language
* Turing-complete language for expression arbitrary analysis
tasks.
* Event-based programming model.
* Domain-specific data types such as IP addresses (transparently
handling both IPv4 and IPv6), port numbers, and timers.
* Extensive support for tracking and managing network state over
time.
- Interfacing
* Default output to well-structured ASCII logs.
* Alternative backends for ElasticSearch and DataSeries. Further
database interfaces in preparation.
* Real-time integration of external input into analyses. Live
database input in preparation.
* External C library for exchanging Bro events with external
programs. Comes with Perl, Python, and Ruby bindings.
* Ability to trigger arbitrary external processes from within
the scripting language.
History
-------
.. figure:: history.png
:width: 600
:align: center
:alt: Bro History Timeline
:target: history.png
Timeline of Bro's History (click to enlarge).
Bro's history goes back much further than many people realize. `Vern
Paxson <http://www.icir.org/vern>`_ designed and implemented the
initial version almost two decades ago.
Vern began work on the code in 1995 as a researcher at the `Lawrence
Berkeley National Laboratory (LBNL) <http://www.lbl.gov>`_. Berkeley
Lab began operational deployment in 1996, and the USENIX Security
Symposium published the original Bro paper in 1998 (later refined in a
subsequent `journal publication <http://www.icir.org/vern/papers/bro-CN99.pdf>`_).
In 2003, the
`National Science Foundation (NSF) <http://www.nsf.gov>`_ began
supporting research and advanced development on Bro at the
`International Computer Science Institute (ICSI)
<http://www.icsi.berkeley.edu>`_, where Vern now leads the `Networking
and Security group <http://www.icir.org>`_. Over the years, a growing
team of ICSI researchers and students kept adding novel functionality
to Bro, while LBNL continued its support with funding from the
`Department of Energy (DOE) <http://www.doe.gov>`_.
Much of Bro's capabilities originate in academic research projects,
with results often published at top-tier conferences. However, the key
to Bro's success was its ability to bridge the traditional gap between
academia and operations from early on, which provided the research
with crucial grounding to ensure that developed approaches stand up to
the challenges of the real world. Yet, with Bro's operational user
community growing over time, the research-centric development model
eventually became a bottleneck to the system's evolution: research
grants do not tend to support the more mundane parts of software
development and maintenance, even though those prove crucial for the
end-user experience. While Bro's capabilities always went beyond those
of traditional systems, a successful deployment used to require
significant technical expertise, typically with a large upfront
investment in tackling Bro's steep learning curve. In 2010, NSF set
out to address this gap by awarding ICSI a grant dedicated solely to
Bro development out of its SDCI program.
With that support in place, the `National Center for
Supercomputing Applications (NCSA) <http://www.ncsa.illinois.edu>`_
joined the team as a core partner, and the Bro Project began to
completely overhaul many of the user-visible parts of the system for
the 2.0 release. Since that version came out, Bro has experienced an
tremendous growth in new deployments across a diverse range of
settings, and the Bro team is now working to build on this success by
further advancing the system's capabilities to address the challenges
of future networks.
Architecture
------------
.. figure:: architecture.png
:width: 400
:align: center
:alt: Bro Architecture
:target: {{docroot}}/images/architecture.png
Bro's internal architecture.
Architecturally, Bro is layered into two major components. Its *event
engine* (or *core*) reduces the incoming packet stream into a series
of higher-level *events*. These events reflect network activity in
policy-neutral terms, i.e., they describe *what* has been seen, but
not *why*, or whether it is significant. For example, every HTTP
request on the wire turns into a corresponding ``http_request`` event
that carries with it the involved IP addresses and ports, the URI
being requested, and the HTTP version in use. The event however does
not convey any further *interpretation*, e.g., of whether that URI
corresponds to a known malware site.
Such semantics are instead derived by Bro's second main component, the
*script interpreter*, which executes a set of *event handlers* written
in Bro's custom scripting language. These scripts can express a site's
security policy, i.e., what actions to take when the monitor detects
different types of activity. More generally they can derive any
desired properties and statistics from the input traffic. Bro's
language comes with extensive domain-specific types and support
functionality; and, crucially, allows scripts to maintain state
over time, enabling them to track and correlate the evolution of what
they observe across connection and host boundaries. Bro scripts can
generate real-time alerts and also execute arbitrary external programs
on demand, e.g., to trigger an active response to an attack.

1
doc/intro/install.rst
View file

@ -1 +0,0 @@
../../INSTALL

5
doc/intro/overview.rst
View file

@ -1,5 +0,0 @@
==================
Overview (Missing)
==================

308
doc/intro/upgrade.rst
View file

@ -1,308 +0,0 @@
==========================================
Upgrading From the Previous Version of Bro
==========================================
.. rst-class:: opening
This guide details specific differences between Bro versions
that may be important for users to know as they work on updating
their Bro deployment/configuration to the later version.
.. contents::
Upgrading From Bro 2.0 to 2.1
=============================
In Bro 2.1, IPv6 is enabled by default. Therefore, when building Bro from
source, the "--enable-brov6" configure option has been removed because it
is no longer relevant.
Other configure changes include renaming the "--enable-perftools" option
to "--enable-perftools-debug" to indicate that the option is only relevant
for debugging the heap. One other change involves what happens when
tcmalloc (part of Google perftools) is found at configure time. On Linux,
it will automatically be linked with Bro, but on other platforms you
need to use the "--enable-perftools" option to enable linking to tcmalloc.
There are a couple of changes to the Bro scripting language to better
support IPv6. First, IPv6 literals appearing in a Bro script must now be
enclosed in square brackets (for example, ``[fe80::db15]``). For subnet
literals, the slash "/" appears after the closing square bracket (for
example, ``[fe80:1234::]/32``). Second, when an IP address variable or IP
address literal is enclosed in pipes (for example, ``|[fe80::db15]|``) the
result is now the size of the address in bits (32 for IPv4 and 128 for IPv6).
In the Bro scripting language, "match" and "using" are no longer reserved
keywords.
Some built-in functions have been removed: "addr_to_count" (use
"addr_to_counts" instead), "bro_has_ipv6" (this is no longer relevant
because Bro now always supports IPv6), "active_connection" (use
"connection_exists" instead), and "connection_record" (use "lookup_connection"
instead).
The "NFS3::mode2string" built-in function has been renamed to "file_mode".
Some built-in functions have been changed: "exit" (now takes the exit code
as a parameter), "to_port" (now takes a string as parameter instead
of a count and transport protocol, but "count_to_port" is still available),
"connect" (now takes an additional string parameter specifying the zone of
a non-global IPv6 address), and "listen" (now takes three additional
parameters to enable listening on IPv6 addresses).
Some Bro script variables have been renamed: "LogAscii::header_prefix"
has been renamed to "LogAscii::meta_prefix", "LogAscii::include_header"
has been renamed to "LogAscii::include_meta".
Some Bro script variables have been removed: "tunnel_port",
"parse_udp_tunnels", "use_connection_compressor", "cc_handle_resets",
"cc_handle_only_syns", and "cc_instantiate_on_data".
A couple events have changed: the "icmp_redirect" event now includes
the target and destination addresses and any Neighbor Discovery options
in the message, and the last parameter of the "dns_AAAA_reply" event has
been removed because it was unused.
The format of the ASCII log files has changed very slightly. Two new lines
are automatically added, one to record the time when the log was opened,
and the other to record the time when the log was closed.
In BroControl, the option (in broctl.cfg) "CFlowAddr" was renamed
to "CFlowAddress".
Upgrading From Bro 1.5 to 2.0
=============================
As the version number jump suggests, Bro 2.0 is a major upgrade and
lots of things have changed. Most importantly, we have rewritten
almost all of Bro's default scripts from scratch, using quite
different structure now and focusing more on operational deployment.
The result is a system that works much better "out of the box", even
without much initial site-specific configuration. The down-side is
that 1.x configurations will need to be adapted to work with the new
version. The two rules of thumb are:
(1) If you have written your own Bro scripts
that do not depend on any of the standard scripts formerly
found in ``policy/``, they will most likely just keep working
(although you might want to adapt them to use some of the new
features, like the new logging framework; see below).
(2) If you have custom code that depends on specifics of 1.x
default scripts (including most configuration tuning), that is
unlikely to work with 2.x. We recommend to start by using just
the new scripts first, and then port over any customizations
incrementally as necessary (they may be much easier to do now,
or even unnecessary). Send mail to the Bro user mailing list
if you need help.
Below we summarize changes from 1.x to 2.x in more detail. This list
isn't complete, see the :download:`CHANGES <CHANGES>` file in the
distribution for the full story.
Default Scripts
===============
Organization
------------
In versions before 2.0, Bro scripts were all maintained in a flat
directory called ``policy/`` in the source tree. This directory is now
renamed to ``scripts/`` and contains major subdirectories ``base/``,
``policy/``, and ``site/``, each of which may also be subdivided
further.
The contents of the new ``scripts/`` directory, like the old/flat
``policy/`` still gets installed under the ``share/bro``
subdirectory of the installation prefix path just like previous
versions. For example, if Bro was compiled like ``./configure
--prefix=/usr/local/bro && make && make install``, then the script
hierarchy can be found in ``/usr/local/bro/share/bro``.
The main
subdirectories of that hierarchy are as follows:
- ``base/`` contains all scripts that are loaded by Bro by default
(unless the ``-b`` command line option is used to run Bro in a
minimal configuration). Note that is a major conceptual change:
rather than not loading anything by default, Bro now uses an
extensive set of default scripts out of the box.
The scripts under this directory generally either accumulate/log
useful state/protocol information for monitored traffic, configure a
default/recommended mode of operation, or provide extra Bro
scripting-layer functionality that has no significant performance cost.
- ``policy/`` contains all scripts that a user will need to explicitly
tell Bro to load. These are scripts that implement
functionality/analysis that not all users may want to use and may have
more significant performance costs. For a new installation, you
should go through these and see what appears useful to load.
- ``site/`` remains a directory that can be used to store locally
developed scripts. It now comes with some preinstalled example
scripts that contain recommended default configurations going beyond
the ``base/`` setup. E.g. ``local.bro`` loads extra scripts from
``policy/`` and does extra tuning. These files can be customized in
place without being overwritten by upgrades/reinstalls, unlike
scripts in other directories.
With version 2.0, the default ``BROPATH`` is set to automatically
search for scripts in ``policy/``, ``site/`` and their parent
directory, but **not** ``base/``. Generally, everything under
``base/`` is loaded automatically, but for users of the ``-b`` option,
it's important to know that loading a script in that directory
requires the extra ``base/`` path qualification. For example, the
following two scripts:
* ``$PREFIX/share/bro/base/protocols/ssl/main.bro``
* ``$PREFIX/share/bro/policy/protocols/ssl/validate-certs.bro``
are referenced from another Bro script like:
.. code:: bro
@load base/protocols/ssl/main
@load protocols/ssl/validate-certs
Notice how ``policy/`` can be omitted as a convenience in the second
case. ``@load`` can now also use relative path, e.g., ``@load
../main``.
Logging Framework
-----------------
- The logs generated by scripts that ship with Bro are entirely redone
to use a standardized, machine parsable format via the new logging
framework. Generally, the log content has been restructured towards
making it more directly useful to operations. Also, several
analyzers have been significantly extended and thus now log more
information. Take a look at ``ssl.log``.
* A particular format change that may be useful to note is that the
``conn.log`` ``service`` field is derived from DPD instead of
well-known ports (while that was already possible in 1.5, it was
not the default).
* Also, ``conn.log`` now reports raw number of packets/bytes per
endpoint.
- The new logging framework makes it possible to extend, customize,
and filter logs very easily. See the :doc:`logging framework <logging>`
for more information on usage.
- A common pattern found in the new scripts is to store logging stream
records for protocols inside the ``connection`` records so that
state can be collected until enough is seen to log a coherent unit
of information regarding the activity of that connection. This
state is now frequently seen/accessible in event handlers, for
example, like ``c$<protocol>`` where ``<protocol>`` is replaced by
the name of the protocol. This field is added to the ``connection``
record by ``redef``'ing it in a
``base/protocols/<protocol>/main.bro`` script.
- The logging code has been rewritten internally, with script-level
interface and output backend now clearly separated. While ASCII
logging is still the default, we will add further output types in
the future (binary format, direct database logging).
Notice Framework
----------------
The way users interact with "notices" has changed significantly in
order to make it easier to define a site policy and more extensible
for adding customized actions. See the :doc:`notice framework <notice>`.
New Default Settings
--------------------
- Dynamic Protocol Detection (DPD) is now enabled/loaded by default.
- The default packet filter now examines all packets instead of
dynamically building a filter based on which protocol analysis scripts
are loaded. See ``PacketFilter::all_packets`` for how to revert to old
behavior.
API Changes
-----------
- The ``@prefixes`` directive works differently now.
Any added prefixes are now searched for and loaded *after* all input
files have been parsed. After all input files are parsed, Bro
searches ``BROPATH`` for prefixed, flattened versions of all of the
parsed input files. For example, if ``lcl`` is in ``@prefixes``, and
``site.bro`` is loaded, then a file named ``lcl.site.bro`` that's in
``BROPATH`` would end up being automatically loaded as well. Packages
work similarly, e.g. loading ``protocols/http`` means a file named
``lcl.protocols.http.bro`` in ``BROPATH`` gets loaded automatically.
- The ``make_addr`` BIF now returns a ``subnet`` versus an ``addr``
Variable Naming
---------------
- ``Module`` is more widely used for namespacing. E.g. the new
``site.bro`` exports the ``local_nets`` identifier (among other
things) into the ``Site`` module.
- Identifiers may have been renamed to conform to new `scripting
conventions
<http://www.bro.org/development/script-conventions.html>`_
BroControl
==========
BroControl looks pretty much similar to the version coming with Bro 1.x,
but has been cleaned up and streamlined significantly internally.
BroControl has a new ``process`` command to process a trace on disk
offline using a similar configuration to what BroControl installs for
live analysis.
BroControl now has an extensive plugin interface for adding new
commands and options. Note that this is still considered experimental.
We have removed the ``analysis`` command, and BroControl currently
does not send daily alarm summaries anymore (this may be restored
later).
Removed Functionality
=====================
We have remove a bunch of functionality that was rarely used and/or
had not been maintained for a while already:
- The ``net`` script data type.
- The ``alarm`` statement; use the notice framework instead.
- Trace rewriting.
- DFA state expiration in regexp engine.
- Active mapping.
- Native DAG support (may come back eventually)
- ClamAV support.
- The connection compressor is now disabled by default, and will
be removed in the future.
Development Infrastructure
==========================
Bro development has moved from using SVN to Git for revision control.
Users that want to use the latest Bro development snapshot by checking it out
from the source repositories should see the `development process
<http://www.bro.org/development/process.html>`_. Note that all the various
sub-components now reside in their own repositories. However, the
top-level Bro repository includes them as git submodules so it's easy
to check them all out simultaneously.
Bro now uses `CMake <http://www.cmake.org>`_ for its build system so
that is a new required dependency when building from source.
Bro now comes with a growing suite of regression tests in
``testing/``.

79
doc/intro/quickstart.rst → doc/quickstart/index.rst
View file

@ -1,14 +1,10 @@
.. _quickstart:
================= =================
Quick Start Guide Quick Start Guide
================= =================
.. rst-class:: opening
The short story for getting Bro up and running in a simple configuration
for analysis of either live traffic from a network interface or a packet
capture trace file.
.. contents:: .. contents::
Installation Installation
@ -16,14 +12,13 @@ Installation
Bro works on most modern, Unix-based systems and requires no custom Bro works on most modern, Unix-based systems and requires no custom
hardware. It can be downloaded in either pre-built binary package or hardware. It can be downloaded in either pre-built binary package or
source code forms. See :doc:`Installing Bro <INSTALL>` for instructions source code forms. See :ref:`installing-bro` for instructions on how to
on how to install Bro. install Bro. Below, ``$PREFIX`` is used to reference the Bro
installation root directory, which by default is ``/usr/local/`` if
you install from source.
.. note:: Below, ``$PREFIX`` is used to reference the Bro installation Managing Bro with BroControl
root directory. ============================
Using BroControl
================
BroControl is an interactive shell for easily operating/managing Bro BroControl is an interactive shell for easily operating/managing Bro
installations on a single system or even across multiple systems in a installations on a single system or even across multiple systems in a
@ -68,9 +63,10 @@ policy and output the results in ``$PREFIX/logs``.
.. note:: The user starting BroControl needs permission to capture .. note:: The user starting BroControl needs permission to capture
network traffic. If you are not root, you may need to grant further network traffic. If you are not root, you may need to grant further
privileges to the account you're using; see the :doc:`FAQ <faq>`. privileges to the account you're using; see the `FAQ
Also, if it looks like Bro is not seeing any traffic, check out <http://www.bro.org/documentation/faq.html>`_. Also, if it looks
the FAQ entry on checksum offloading. like Bro is not seeing any traffic, check out the FAQ entry on
checksum offloading.
You can leave it running for now, but to stop this Bro instance you would do: You can leave it running for now, but to stop this Bro instance you would do:
@ -115,20 +111,18 @@ columns (shortened for brevity) show a request to the root of Bro website::
Some logs are worth explicit mention: Some logs are worth explicit mention:
``weird.log`` ``conn.log``
Contains unusual/exceptional activity that can indicate Contains an entry for every connection seen on the wire, with
malformed connections, traffic that doesn't conform to a particular basic properties such as time and duration, originator and
protocol, malfunctioning/misconfigured hardware, or even an attacker responder IP addresses, services and ports, payload size, and
attempting to avoid/confuse a sensor. Without context, it's hard to much more. This log provides a comprehensive record of the
judge whether this category of activity is interesting and so that is network's activity.
left up to the user to configure.
``notice.log`` ``notice.log``
Identifies specific activity that Bro recognizes as Identifies specific activity that Bro recognizes as
potentially interesting, odd, or bad. In Bro-speak, such potentially interesting, odd, or bad. In Bro-speak, such
activity is called a "notice". activity is called a "notice".
By default, ``BroControl`` regularly takes all the logs from By default, ``BroControl`` regularly takes all the logs from
``$PREFIX/logs/current`` and archives/compresses them to a directory ``$PREFIX/logs/current`` and archives/compresses them to a directory
named by date, e.g. ``$PREFIX/logs/2011-10-06``. The frequency at named by date, e.g. ``$PREFIX/logs/2011-10-06``. The frequency at
@ -302,21 +296,26 @@ tweak the most basic options. Here's some suggestions on what to explore next:
* We only looked at how to change options declared in the notice framework, * We only looked at how to change options declared in the notice framework,
there's many more options to look at in other script packages. there's many more options to look at in other script packages.
* Continue reading with :ref:`using-bro` chapter which goes into more
depth on working with Bro; then look at :ref:`writing-scripts` for
learning how to start writing your own scripts.
* Look at the scripts in ``$PREFIX/share/bro/policy`` for further ones * Look at the scripts in ``$PREFIX/share/bro/policy`` for further ones
you may want to load. you may want to load; you can browse their documentation at the
:ref:`overview of script packages <script-packages>`.
* Reading the code of scripts that ship with Bro is also a great way to gain * Reading the code of scripts that ship with Bro is also a great way to gain
understanding of the language and how you can start writing your own custom further understanding of the language and how scripts tend to be
analysis. structured.
* Review the :doc:`FAQ <faq>`. * Review the `FAQ <http://www.bro.org/documentation/faq.html>`_.
* Continue reading below for another mini-tutorial on using Bro as a standalone * Continue reading below for another mini-tutorial on using Bro as a standalone
command-line utility. command-line utility.
Bro, the Command-Line Utility Bro as a Command-Line Utility
============================= =============================
If you prefer not to use BroControl (e.g. don't need its automation and If you prefer not to use BroControl (e.g. don't need its automation
management features), here's how to directly control Bro for your analysis and management features), here's how to directly control Bro for your
activities. analysis activities from the command line for both live traffic and
offline working from traces.
Monitoring Live Traffic Monitoring Live Traffic
----------------------- -----------------------
@ -420,3 +419,19 @@ are "load this script if it hasn't already been loaded".
to be searched for scripts. See the default search path by doing to be searched for scripts. See the default search path by doing
``bro --help``. ``bro --help``.
Running Bro Without Installing
------------------------------
For developers that wish to run Bro directly from the ``build/``
directory (i.e., without performing ``make install``), they will have
to first adjust ``BROPATH`` and ``BROMAGIC`` to look for scripts and
additional files inside the build directory. Sourcing either
``build/bro-path-dev.sh`` or ``build/bro-path-dev.csh`` as appropriate
for the current shell accomplishes this and also augments your
``PATH`` so you can use the Bro binary directly::
./configure
make
source build/bro-path-dev.sh
bro <options>

2
doc/scripting/index.rst
View file

@ -1,4 +1,6 @@
.. _writing-scripts:
=================== ===================
Writing Bro Scripts Writing Bro Scripts
=================== ===================

2
doc/scripts/packages.rst
View file

@ -1,5 +1,7 @@
.. This is a stub doc to which broxygen appends during the build process .. This is a stub doc to which broxygen appends during the build process
.. _script-packages:
Bro Script Packages Bro Script Packages
=================== ===================

2
doc/using/index.rst
View file

@ -1,4 +1,6 @@
.. _using-bro:
========= =========
Using Bro Using Bro
========= =========