========================================== Upgrading From the Previous Version of Bro ========================================== .. rst-class:: opening This guide details specific differences between Bro versions that may be important for users to know as they work on updating their Bro deployment/configuration to the later version. .. contents:: Upgrading From Bro 2.0 to 2.1 ============================= In Bro 2.1, IPv6 is enabled by default. Therefore, when building Bro from source, the "--enable-brov6" configure option has been removed because it is no longer relevant. Other configure changes include renaming the "--enable-perftools" option to "--enable-perftools-debug" to indicate that the option is only relevant for debugging the heap. One other change involves what happens when tcmalloc (part of Google perftools) is found at configure time. On Linux, it will automatically be linked with Bro, but on other platforms you need to use the "--enable-perftools" option to enable linking to tcmalloc. There are a couple of changes to the Bro scripting language to better support IPv6. First, IPv6 literals appearing in a Bro script must now be enclosed in square brackets (for example, ``[fe80::db15]``). For subnet literals, the slash "/" appears after the closing square bracket (for example, ``[fe80:1234::]/32``). Second, when an IP address variable or IP address literal is enclosed in pipes (for example, ``|[fe80::db15]|``) the result is now the size of the address in bits (32 for IPv4 and 128 for IPv6). In the Bro scripting language, "match" and "using" are no longer reserved keywords. Some built-in functions have been removed: "addr_to_count" (use "addr_to_counts" instead), "bro_has_ipv6" (this is no longer relevant because Bro now always supports IPv6), "active_connection" (use "connection_exists" instead), and "connection_record" (use "lookup_connection" instead). The "NFS3::mode2string" built-in function has been renamed to "file_mode". Some built-in functions have been changed: "exit" (now takes the exit code as a parameter), "to_port" (now takes a string as parameter instead of a count and transport protocol, but "count_to_port" is still available), "connect" (now takes an additional string parameter specifying the zone of a non-global IPv6 address), and "listen" (now takes three additional parameters to enable listening on IPv6 addresses). Some Bro script variables have been renamed: "LogAscii::header_prefix" has been renamed to "LogAscii::meta_prefix", "LogAscii::include_header" has been renamed to "LogAscii::include_meta". Some Bro script variables have been removed: "tunnel_port", "parse_udp_tunnels", "use_connection_compressor", "cc_handle_resets", "cc_handle_only_syns", and "cc_instantiate_on_data". A couple events have changed: the "icmp_redirect" event now includes the target and destination addresses and any Neighbor Discovery options in the message, and the last parameter of the "dns_AAAA_reply" event has been removed because it was unused. The format of the ASCII log files has changed very slightly. Two new lines are automatically added, one to record the time when the log was opened, and the other to record the time when the log was closed. In BroControl, the option (in broctl.cfg) "CFlowAddr" was renamed to "CFlowAddress". Upgrading From Bro 1.5 to 2.0 ============================= As the version number jump suggests, Bro 2.0 is a major upgrade and lots of things have changed. Most importantly, we have rewritten almost all of Bro's default scripts from scratch, using quite different structure now and focusing more on operational deployment. The result is a system that works much better "out of the box", even without much initial site-specific configuration. The down-side is that 1.x configurations will need to be adapted to work with the new version. The two rules of thumb are: (1) If you have written your own Bro scripts that do not depend on any of the standard scripts formerly found in ``policy/``, they will most likely just keep working (although you might want to adapt them to use some of the new features, like the new logging framework; see below). (2) If you have custom code that depends on specifics of 1.x default scripts (including most configuration tuning), that is unlikely to work with 2.x. We recommend to start by using just the new scripts first, and then port over any customizations incrementally as necessary (they may be much easier to do now, or even unnecessary). Send mail to the Bro user mailing list if you need help. Below we summarize changes from 1.x to 2.x in more detail. This list isn't complete, see the :download:`CHANGES ` file in the distribution for the full story. Default Scripts =============== Organization ------------ In versions before 2.0, Bro scripts were all maintained in a flat directory called ``policy/`` in the source tree. This directory is now renamed to ``scripts/`` and contains major subdirectories ``base/``, ``policy/``, and ``site/``, each of which may also be subdivided further. The contents of the new ``scripts/`` directory, like the old/flat ``policy/`` still gets installed under the ``share/bro`` subdirectory of the installation prefix path just like previous versions. For example, if Bro was compiled like ``./configure --prefix=/usr/local/bro && make && make install``, then the script hierarchy can be found in ``/usr/local/bro/share/bro``. The main subdirectories of that hierarchy are as follows: - ``base/`` contains all scripts that are loaded by Bro by default (unless the ``-b`` command line option is used to run Bro in a minimal configuration). Note that is a major conceptual change: rather than not loading anything by default, Bro now uses an extensive set of default scripts out of the box. The scripts under this directory generally either accumulate/log useful state/protocol information for monitored traffic, configure a default/recommended mode of operation, or provide extra Bro scripting-layer functionality that has no significant performance cost. - ``policy/`` contains all scripts that a user will need to explicitly tell Bro to load. These are scripts that implement functionality/analysis that not all users may want to use and may have more significant performance costs. For a new installation, you should go through these and see what appears useful to load. - ``site/`` remains a directory that can be used to store locally developed scripts. It now comes with some preinstalled example scripts that contain recommended default configurations going beyond the ``base/`` setup. E.g. ``local.bro`` loads extra scripts from ``policy/`` and does extra tuning. These files can be customized in place without being overwritten by upgrades/reinstalls, unlike scripts in other directories. With version 2.0, the default ``BROPATH`` is set to automatically search for scripts in ``policy/``, ``site/`` and their parent directory, but **not** ``base/``. Generally, everything under ``base/`` is loaded automatically, but for users of the ``-b`` option, it's important to know that loading a script in that directory requires the extra ``base/`` path qualification. For example, the following two scripts: * ``$PREFIX/share/bro/base/protocols/ssl/main.bro`` * ``$PREFIX/share/bro/policy/protocols/ssl/validate-certs.bro`` are referenced from another Bro script like: .. code:: bro @load base/protocols/ssl/main @load protocols/ssl/validate-certs Notice how ``policy/`` can be omitted as a convenience in the second case. ``@load`` can now also use relative path, e.g., ``@load ../main``. Logging Framework ----------------- - The logs generated by scripts that ship with Bro are entirely redone to use a standardized, machine parsable format via the new logging framework. Generally, the log content has been restructured towards making it more directly useful to operations. Also, several analyzers have been significantly extended and thus now log more information. Take a look at ``ssl.log``. * A particular format change that may be useful to note is that the ``conn.log`` ``service`` field is derived from DPD instead of well-known ports (while that was already possible in 1.5, it was not the default). * Also, ``conn.log`` now reports raw number of packets/bytes per endpoint. - The new logging framework makes it possible to extend, customize, and filter logs very easily. See the :doc:`logging framework ` for more information on usage. - A common pattern found in the new scripts is to store logging stream records for protocols inside the ``connection`` records so that state can be collected until enough is seen to log a coherent unit of information regarding the activity of that connection. This state is now frequently seen/accessible in event handlers, for example, like ``c$`` where ```` is replaced by the name of the protocol. This field is added to the ``connection`` record by ``redef``'ing it in a ``base/protocols//main.bro`` script. - The logging code has been rewritten internally, with script-level interface and output backend now clearly separated. While ASCII logging is still the default, we will add further output types in the future (binary format, direct database logging). Notice Framework ---------------- The way users interact with "notices" has changed significantly in order to make it easier to define a site policy and more extensible for adding customized actions. See the :doc:`notice framework `. New Default Settings -------------------- - Dynamic Protocol Detection (DPD) is now enabled/loaded by default. - The default packet filter now examines all packets instead of dynamically building a filter based on which protocol analysis scripts are loaded. See ``PacketFilter::all_packets`` for how to revert to old behavior. API Changes ----------- - The ``@prefixes`` directive works differently now. Any added prefixes are now searched for and loaded *after* all input files have been parsed. After all input files are parsed, Bro searches ``BROPATH`` for prefixed, flattened versions of all of the parsed input files. For example, if ``lcl`` is in ``@prefixes``, and ``site.bro`` is loaded, then a file named ``lcl.site.bro`` that's in ``BROPATH`` would end up being automatically loaded as well. Packages work similarly, e.g. loading ``protocols/http`` means a file named ``lcl.protocols.http.bro`` in ``BROPATH`` gets loaded automatically. - The ``make_addr`` BIF now returns a ``subnet`` versus an ``addr`` Variable Naming --------------- - ``Module`` is more widely used for namespacing. E.g. the new ``site.bro`` exports the ``local_nets`` identifier (among other things) into the ``Site`` module. - Identifiers may have been renamed to conform to new `scripting conventions `_ BroControl ========== BroControl looks pretty much similar to the version coming with Bro 1.x, but has been cleaned up and streamlined significantly internally. BroControl has a new ``process`` command to process a trace on disk offline using a similar configuration to what BroControl installs for live analysis. BroControl now has an extensive plugin interface for adding new commands and options. Note that this is still considered experimental. We have removed the ``analysis`` command, and BroControl currently does not send daily alarm summaries anymore (this may be restored later). Removed Functionality ===================== We have remove a bunch of functionality that was rarely used and/or had not been maintained for a while already: - The ``net`` script data type. - The ``alarm`` statement; use the notice framework instead. - Trace rewriting. - DFA state expiration in regexp engine. - Active mapping. - Native DAG support (may come back eventually) - ClamAV support. - The connection compressor is now disabled by default, and will be removed in the future. Development Infrastructure ========================== Bro development has moved from using SVN to Git for revision control. Users that want to use the latest Bro development snapshot by checking it out from the source repositories should see the `development process `_. Note that all the various sub-components now reside in their own repositories. However, the top-level Bro repository includes them as git submodules so it's easy to check them all out simultaneously. Bro now uses `CMake `_ for its build system so that is a new required dependency when building from source. Bro now comes with a growing suite of regression tests in ``testing/``.