zeek/doc/intro/upgrade.rst

==========================================
Upgrading From the Previous Version of Bro
==========================================

.. rst-class:: opening

   This guide details specific differences between Bro versions
   that may be important for users to know as they work on updating
   their Bro deployment/configuration to the later version.

.. contents::


Upgrading From Bro 2.0 to 2.1
=============================

In Bro 2.1, IPv6 is enabled by default.  Therefore, when building Bro from
source, the "--enable-brov6" configure option has been removed because it
is no longer relevant.  

Other configure changes include renaming the "--enable-perftools" option
to "--enable-perftools-debug" to indicate that the option is only relevant
for debugging the heap.  One other change involves what happens when
tcmalloc (part of Google perftools) is found at configure time.  On Linux,
it will automatically be linked with Bro, but on other platforms you
need to use the "--enable-perftools" option to enable linking to tcmalloc.

There are a couple of changes to the Bro scripting language to better
support IPv6. First, IPv6 literals appearing in a Bro script must now be
enclosed in square brackets (for example, ``[fe80::db15]``). For subnet
literals, the slash "/" appears after the closing square bracket (for
example, ``[fe80:1234::]/32``). Second, when an IP address variable or IP
address literal is enclosed in pipes (for example, ``|[fe80::db15]|``) the
result is now the size of the address in bits (32 for IPv4 and 128 for IPv6).

In the Bro scripting language, "match" and "using" are no longer reserved
keywords.

Some built-in functions have been removed: "addr_to_count" (use
"addr_to_counts" instead), "bro_has_ipv6" (this is no longer relevant
because Bro now always supports IPv6), "active_connection" (use
"connection_exists" instead), and "connection_record" (use "lookup_connection"
instead).

The "NFS3::mode2string" built-in function has been renamed to "file_mode".

Some built-in functions have been changed:  "exit" (now takes the exit code
as a parameter), "to_port" (now takes a string as parameter instead
of a count and transport protocol, but "count_to_port" is still available),
"connect" (now takes an additional string parameter specifying the zone of
a non-global IPv6 address), and "listen" (now takes three additional
parameters to enable listening on IPv6 addresses).

Some Bro script variables have been renamed:  "LogAscii::header_prefix"
has been renamed to "LogAscii::meta_prefix", "LogAscii::include_header"
has been renamed to "LogAscii::include_meta".

Some Bro script variables have been removed: "tunnel_port",
"parse_udp_tunnels", "use_connection_compressor", "cc_handle_resets",
"cc_handle_only_syns", and "cc_instantiate_on_data".

A couple events have changed:  the "icmp_redirect" event now includes
the target and destination addresses and any Neighbor Discovery options
in the message, and the last parameter of the "dns_AAAA_reply" event has
been removed because it was unused.

The format of the ASCII log files has changed very slightly.  Two new lines
are automatically added, one to record the time when the log was opened,
and the other to record the time when the log was closed.

In BroControl, the option (in broctl.cfg) "CFlowAddr" was renamed
to "CFlowAddress".


Upgrading From Bro 1.5 to 2.0
=============================

As the version number jump suggests, Bro 2.0 is a major upgrade and
lots of things have changed. Most importantly, we have rewritten
almost all of Bro's default scripts from scratch, using quite
different structure now and focusing more on operational deployment.
The result is a system that works much better "out of the box", even
without much initial site-specific configuration. The down-side is
that 1.x configurations will need to be adapted to work with the new
version. The two rules of thumb are:

    (1) If you have written your own Bro scripts
        that do not depend on any of the standard scripts formerly
        found in ``policy/``, they will most likely just keep working
        (although you might want to adapt them to use some of the new
        features, like the new logging framework; see below).

    (2) If you have custom code that depends on specifics of 1.x
        default scripts (including most configuration tuning), that is
        unlikely to work with 2.x. We recommend to start by using just
        the new scripts first, and then port over any customizations
        incrementally as necessary (they may be much easier to do now,
        or even unnecessary). Send mail to the Bro user mailing list
        if you need help.

Below we summarize changes from 1.x to 2.x in more detail. This list
isn't complete, see the :download:`CHANGES <CHANGES>` file in the
distribution for the full story. 

Default Scripts
===============

Organization
------------

In versions before 2.0, Bro scripts were all maintained in a flat
directory called ``policy/`` in the source tree.  This directory is now
renamed to ``scripts/`` and contains major subdirectories ``base/``,
``policy/``, and ``site/``, each of which may also be subdivided
further.

The contents of the new ``scripts/`` directory, like the old/flat
``policy/`` still gets installed under the ``share/bro``
subdirectory of the installation prefix path just like previous
versions.  For example, if Bro was compiled like ``./configure
--prefix=/usr/local/bro && make && make install``, then the script
hierarchy can be found in ``/usr/local/bro/share/bro``.

The main
subdirectories of that hierarchy are as follows:

- ``base/`` contains all scripts that are loaded by Bro by default
  (unless the ``-b`` command line option is used to run Bro in a
  minimal configuration). Note that is a major conceptual change:
  rather than not loading anything by default, Bro now uses an
  extensive set of default scripts out of the box.

  The scripts under this directory generally either accumulate/log
  useful state/protocol information for monitored traffic, configure a
  default/recommended mode of operation, or provide extra Bro
  scripting-layer functionality that has no significant performance cost.

- ``policy/`` contains all scripts that a user will need to explicitly
  tell Bro to load.  These are scripts that implement
  functionality/analysis that not all users may want to use and may have
  more significant performance costs. For a new installation, you
  should go through these and see what appears useful to load.

- ``site/`` remains a directory that can be used to store locally 
  developed scripts. It now comes with some preinstalled example
  scripts that contain recommended default configurations going beyond
  the ``base/`` setup. E.g. ``local.bro`` loads extra scripts from
  ``policy/`` and does extra tuning. These files can be customized in
  place without being overwritten by upgrades/reinstalls, unlike
  scripts in other directories.

With version 2.0, the default ``BROPATH`` is set to automatically
search for scripts in ``policy/``, ``site/`` and their parent
directory, but **not** ``base/``.  Generally, everything under
``base/`` is loaded automatically, but for users of the ``-b`` option,
it's important to know that loading a script in that directory
requires the extra ``base/`` path qualification.  For example, the
following two scripts:

* ``$PREFIX/share/bro/base/protocols/ssl/main.bro``
* ``$PREFIX/share/bro/policy/protocols/ssl/validate-certs.bro``

are referenced from another Bro script like:

.. code:: bro

    @load base/protocols/ssl/main
    @load protocols/ssl/validate-certs

Notice how ``policy/`` can be omitted as a convenience in the second
case. ``@load`` can now also use relative path, e.g., ``@load
../main``.


Logging Framework
-----------------

- The logs generated by scripts that ship with Bro are entirely redone
  to use a standardized, machine parsable format via the new logging
  framework. Generally, the log content has been restructured towards
  making it more directly useful to operations. Also, several
  analyzers have been significantly extended and thus now log more
  information. Take a look at ``ssl.log``.

  * A particular format change that may be useful to note is that the
    ``conn.log`` ``service`` field is derived from DPD instead of
    well-known ports (while that was already possible in 1.5, it was
    not the default).

  * Also, ``conn.log`` now reports raw number of packets/bytes per
    endpoint.

- The new logging framework makes it possible to extend, customize,
  and filter logs very easily. See the :doc:`logging framework <logging>`
  for more information on usage.

- A common pattern found in the new scripts is to store logging stream
  records for protocols inside the ``connection`` records so that
  state can be collected until enough is seen to log a coherent unit
  of information regarding the activity of that connection.  This
  state is now frequently seen/accessible in event handlers, for
  example, like ``c$<protocol>`` where ``<protocol>`` is replaced by
  the name of the protocol.  This field is added to the ``connection``
  record by ``redef``'ing it in a
  ``base/protocols/<protocol>/main.bro`` script.

- The logging code has been rewritten internally, with script-level
  interface and output backend now clearly separated. While ASCII
  logging is still the default, we will add further output types in
  the future (binary format, direct database logging).


Notice Framework
----------------

The way users interact with "notices" has changed significantly in
order to make it easier to define a site policy and more extensible
for adding customized actions. See the :doc:`notice framework <notice>`.


New Default Settings
--------------------

- Dynamic Protocol Detection (DPD) is now enabled/loaded by default.

- The default packet filter now examines all packets instead of
  dynamically building a filter based on which protocol analysis scripts
  are loaded. See ``PacketFilter::all_packets`` for how to revert to old
  behavior.

API Changes
-----------

- The ``@prefixes`` directive works differently now.
  Any added prefixes are now searched for and loaded *after* all input
  files have been parsed.  After all input files are parsed, Bro
  searches ``BROPATH`` for prefixed, flattened versions of all of the
  parsed input files.  For example, if ``lcl`` is in ``@prefixes``, and
  ``site.bro`` is loaded, then a file named ``lcl.site.bro`` that's in
  ``BROPATH`` would end up being automatically loaded as well.  Packages
  work similarly, e.g. loading ``protocols/http`` means a file named
  ``lcl.protocols.http.bro`` in ``BROPATH`` gets loaded automatically.

- The ``make_addr`` BIF now returns a ``subnet`` versus an ``addr``


Variable Naming
---------------

- ``Module`` is more widely used for namespacing. E.g. the new
  ``site.bro`` exports the ``local_nets`` identifier (among other
  things) into the ``Site`` module.

- Identifiers may have been renamed to conform to new `scripting
  conventions
  <http://www.bro.org/development/script-conventions.html>`_


BroControl
==========

BroControl looks pretty much similar to the version coming with Bro 1.x,
but has been cleaned up and streamlined significantly internally.

BroControl has a new ``process`` command to process a trace on disk
offline using a similar configuration to what BroControl installs for
live analysis.

BroControl now has an extensive plugin interface for adding new
commands and options. Note that this is still considered experimental.

We have removed the ``analysis`` command, and BroControl currently
does not send daily alarm summaries anymore (this may be restored
later).

Removed Functionality
=====================

We have remove a bunch of functionality that was rarely used and/or
had not been maintained for a while already:

    - The ``net`` script data type.
    - The ``alarm`` statement; use the notice framework instead.
    - Trace rewriting.
    - DFA state expiration in regexp engine.
    - Active mapping.
    - Native DAG support (may come back eventually)
    - ClamAV support.
    - The connection compressor is now disabled by default, and will
      be removed in the future. 

Development Infrastructure
==========================

Bro development has moved from using SVN to Git for revision control.
Users that want to use the latest Bro development snapshot by checking it out
from the source repositories should see the `development process
<http://www.bro.org/development/process.html>`_. Note that all the various
sub-components now reside in their own repositories. However, the
top-level Bro repository includes them as git submodules so it's easy
to check them all out simultaneously.

Bro now uses `CMake <http://www.cmake.org>`_ for its build system so
that is a new required dependency when building from source.

Bro now comes with a growing suite of regression tests in
``testing/``.