Merge remote-tracking branch 'origin/topic/dnthayer/doc-improvements-2.4'

Lots of good stuff! Thanks for catchign the plugin doc inconsistencies!

* origin/topic/dnthayer/doc-improvements-2.4:
  Add missing documentation on the "Bro Package Index" page
  More improvements to the Logging Framework doc
  Fix documentation typo
  Update the "Log Files" documentation
  Add links in the logging framework doc
  Add a link to the bro-plugins documentation
  Update bro man page
  Update script language reference documentation
  Fix typos in the "writing bro plugins" doc
  Fix a "make doc" warning
  Improve logging framework doc
  Add link to broctl doc from the quickstart doc
  Update install documentation and fix some typos
  Minor improvements to logging framework documentation
  Correct a minor typo in the docs

CHANGES

@@ -1,4 +1,9 @@
 
+2.4-beta-32 | 2015-06-02 09:43:31 -0700
+
+  * A larger set of documentation updates, fixes, and extentions.
+    (Daniel Thayer)
+
 2.4-beta-14 | 2015-06-02 09:16:44 -0700
 
   * Add memleak btest for attachments over SMTP. (Vlad Grigorescu)

VERSION

@@ -1 +1 @@
-2.4-beta-14
+2.4-beta-32

aux/plugins

@@ -1 +1 @@
-Subproject commit 475beed1e9688f572ee60c196e07c0fa72e1ed9f
+Subproject commit 64f85b5086fe9feb4cbcaaa8712db61e709836d3

doc/components/bro-plugins/README.rst

@@ -0,0 +1 @@
+../../../aux/plugins/README

doc/components/bro-plugins/dataseries/README.rst

@@ -0,0 +1 @@
+../../../../aux/plugins/dataseries/README

doc/components/bro-plugins/elasticsearch/README.rst

@@ -0,0 +1 @@
+../../../../aux/plugins/elasticsearch/README

doc/components/bro-plugins/netmap/README.rst

@@ -0,0 +1 @@
+../../../../aux/plugins/netmap/README

doc/components/index.rst

@@ -21,6 +21,7 @@ current, independent component releases.
    Broker - User Manual <broker/broker-manual.rst>
    BroControl - Interactive Bro management shell <broctl/README>
    Bro-Aux - Small auxiliary tools for Bro <bro-aux/README>
+   Bro-Plugins - A collection of plugins for Bro <bro-plugins/README>
    BTest - A unit testing framework <btest/README>
    Capstats - Command-line packet statistic tool <capstats/README>
    PySubnetTree - Python module for CIDR lookups<pysubnettree/README>

doc/devel/plugins.rst

@@ -3,7 +3,7 @@
 Writing Bro Plugins
 ===================
 
-Bro internally provides plugin API that enables extending
+Bro internally provides a plugin API that enables extending
 the system dynamically, without modifying the core code base. That way
 custom code remains self-contained and can be maintained, compiled,
 and installed independently. Currently, plugins can add the following
@@ -32,7 +32,7 @@ Quick Start
 ===========
 
 Writing a basic plugin is quite straight-forward as long as one
-follows a few conventions. In the following we walk a simple example
+follows a few conventions. In the following we create a simple example
 plugin that adds a new built-in function (bif) to Bro: we'll add 
 ``rot13(s: string) : string``, a function that rotates every character
 in a string by 13 places.
@@ -81,7 +81,7 @@ The syntax of this file is just like any other ``*.bif`` file; we
 won't go into it here.
 
 Now we can already compile our plugin, we just need to tell the
-configure script that ``init-plugin`` put in place where the Bro
+configure script (that ``init-plugin`` created) where the Bro
 source tree is located (Bro needs to have been built there first)::
 
     # cd rot13-plugin
@@ -99,7 +99,7 @@ option::
     # export BRO_PLUGIN_PATH=/path/to/rot13-plugin/build
     # bro -N
     [...]
-    Plugin: Demo::Rot13 - <Insert brief description of plugin> (dynamic, version 1)
+    Demo::Rot13 - <Insert description> (dynamic, version 0.1)
     [...]
 
 That looks quite good, except for the dummy description that we should
@@ -108,28 +108,30 @@ is about.  We do this by editing the ``config.description`` line in
 ``src/Plugin.cc``, like this::
 
     [...]
-    plugin::Configuration Configure()
+    plugin::Configuration Plugin::Configure()
         {
         plugin::Configuration config;
         config.name = "Demo::Rot13";
         config.description = "Caesar cipher rotating a string's characters by 13 places.";
-        config.version.major = 1;
+        config.version.major = 0;
-        config.version.minor = 0;
+        config.version.minor = 1;
         return config;
         }
     [...]
 
+Now rebuild and verify that the description is visible::
+
     # make
     [...]
     # bro -N | grep Rot13
-    Plugin: Demo::Rot13 - Caesar cipher rotating a string's characters by 13 places. (dynamic, version 1)
+    Demo::Rot13 - Caesar cipher rotating a string's characters by 13 places. (dynamic, version 0.1)
 
-Better. Bro can also show us what exactly the plugin provides with the
+Bro can also show us what exactly the plugin provides with the
 more verbose option ``-NN``::
 
     # bro -NN
     [...]
-    Plugin: Demo::Rot13 - Caesar cipher rotating a string's characters by 13 places. (dynamic, version 1)
+    Demo::Rot13 - Caesar cipher rotating a string's characters by 13 places. (dynamic, version 0.1)
         [Function] Demo::rot13
     [...]
 
@@ -157,10 +159,12 @@ The installed version went into
 ``<bro-install-prefix>/lib/bro/plugins/Demo_Rot13``.
 
 One can distribute the plugin independently of Bro for others to use.
-To distribute in source form, just remove the ``build/`` (``make
+To distribute in source form, just remove the ``build/`` directory
-distclean`` does that) and then tar up the whole ``rot13-plugin/``
+(``make distclean`` does that) and then tar up the whole ``rot13-plugin/``
 directory. Others then follow the same process as above after
-unpacking. To distribute the plugin in binary form, the build process
+unpacking.
+
+To distribute the plugin in binary form, the build process
 conveniently creates a corresponding tarball in ``build/dist/``. In
 this case, it's called ``Demo_Rot13-0.1.tar.gz``, with the version
 number coming out of the ``VERSION`` file that ``init-plugin`` put
@@ -169,14 +173,14 @@ plugin, but no further source files. Optionally, one can include
 further files by specifying them in the plugin's ``CMakeLists.txt``
 through the ``bro_plugin_dist_files`` macro; the skeleton does that
 for ``README``, ``VERSION``, ``CHANGES``, and ``COPYING``. To use the
-plugin through the binary tarball, just unpack it and point
+plugin through the binary tarball, just unpack it into
-``BRO_PLUGIN_PATH`` there; or copy it into
+``<bro-install-prefix>/lib/bro/plugins/``.  Alternatively, if you unpack
-``<bro-install-prefix>/lib/bro/plugins/`` directly.
+it in another location, then you need to point ``BRO_PLUGIN_PATH`` there.
 
 Before distributing your plugin, you should edit some of the meta
 files that ``init-plugin`` puts in place. Edit ``README`` and
 ``VERSION``, and update ``CHANGES`` when you make changes. Also put a
-license file in place as ``COPYING``; if BSD is fine, you find a
+license file in place as ``COPYING``; if BSD is fine, you will find a
 template in ``COPYING.edit-me``.
 
 Plugin Directory Layout
@@ -193,7 +197,7 @@ directory. With the skeleton, ``<base>`` corresponds to ``build/``.
     must exist, and its content must consist of a single line with the
     qualified name of the plugin (e.g., "Demo::Rot13").
 
-``<base>/lib/<plugin-name>-<os>-<arch>.so``
+``<base>/lib/<plugin-name>.<os>-<arch>.so``
     The shared library containing the plugin's compiled code. Bro will
     load this in dynamically at run-time if OS and architecture match
     the current platform.
@@ -215,8 +219,8 @@ directory. With the skeleton, ``<base>`` corresponds to ``build/``.
 Any other files in ``<base>`` are ignored by Bro.
 
 By convention, a plugin should put its custom scripts into sub folders
-of ``scripts/``, i.e., ``scripts/<script-namespace>/<script>.bro`` to
+of ``scripts/``, i.e., ``scripts/<plugin-namespace>/<plugin-name>/<script>.bro``
-avoid conflicts. As usual, you can then put a ``__load__.bro`` in
+to avoid conflicts. As usual, you can then put a ``__load__.bro`` in
 there as well so that, e.g., ``@load Demo/Rot13`` could load a whole
 module in the form of multiple individual scripts.
 
@@ -242,7 +246,8 @@ as well as the ``__bro_plugin__`` magic file and any further
 distribution files specified in ``CMakeLists.txt`` (e.g., README,
 VERSION). You can find a full list of files installed in
 ``build/MANIFEST``. Behind the scenes, ``make install`` really just
-copies over the binary tarball in ``build/dist``. 
+unpacks the binary tarball from ``build/dist`` into the destination
+directory.
 
 ``init-plugin`` will never overwrite existing files. If its target
 directory already exists, it will by default decline to do anything.
@@ -369,18 +374,19 @@ Testing Plugins
 ===============
 
 A plugin should come with a test suite to exercise its functionality.
-The ``init-plugin`` script puts in place a basic </btest/README> setup
+The ``init-plugin`` script puts in place a basic
+:doc:`BTest <../../components/btest/README>` setup
 to start with. Initially, it comes with a single test that just checks
 that Bro loads the plugin correctly. It won't have a baseline yet, so
 let's get that in place::
 
     # cd tests
     # btest -d
-    [  0%] plugin.loading ... failed
+    [  0%] rot13.show-plugin ... failed
     % 'btest-diff output' failed unexpectedly (exit code 100)
     % cat .diag
     == File ===============================
-    Demo::Rot13 - Caesar cipher rotating a string's characters by 13 places. (dynamic, version 1.0)
+    Demo::Rot13 - Caesar cipher rotating a string's characters by 13 places. (dynamic, version 0.1)
         [Function] Demo::rot13
 
     == Error ===============================
@@ -413,8 +419,8 @@ correctly::
 
 Check the output::
 
-    # btest -d plugin/rot13.bro
+    # btest -d rot13/bif-rot13.bro
-    [  0%] plugin.rot13 ... failed
+    [  0%] rot13.bif-rot13 ... failed
     % 'btest-diff output' failed unexpectedly (exit code 100)
     % cat .diag
     == File ===============================
@@ -429,7 +435,7 @@ Check the output::
 
 Install the baseline::
 
-    # btest -U plugin/rot13.bro
+    # btest -U rot13/bif-rot13.bro
     all 1 tests successful
 
 Run the test-suite::
@@ -457,7 +463,7 @@ your plugin's debugging output with ``-B plugin-<name>``, where
 ``<name>`` is the name of the plugin as returned by its
 ``Configure()`` method, yet with the namespace-separator ``::``
 replaced with a simple dash. Example: If the plugin is called
-``Bro::Demo``, use ``-B plugin-Bro-Demo``. As usual, the debugging
+``Demo::Rot13``, use ``-B plugin-Demo-Rot13``. As usual, the debugging
 output will be recorded to ``debug.log`` if Bro's compiled in debug
 mode.
 

doc/frameworks/logging-input-sqlite.rst

@@ -67,8 +67,8 @@ that are present in the ASCII log files::
     'id.orig_p' integer,
     ...
 
-Note that the ASCII ``conn.log`` will still be created. To disable the ASCII writer for a
+Note that the ASCII ``conn.log`` will still be created. To prevent this file
-log stream, you can remove the default filter:
+from being created, you can remove the default filter:
 
 .. code:: bro
 

doc/frameworks/logging.rst

@@ -19,195 +19,144 @@ Terminology
 
 Bro's logging interface is built around three main abstractions:
 
-    Log streams
+    Streams
-        A stream corresponds to a single log. It defines the set of
+        A log stream corresponds to a single log. It defines the set of
-        fields that a log consists of with their names and fields.
+        fields that a log consists of with their names and types.
-        Examples are the ``conn`` for recording connection summaries,
+        Examples are the ``conn`` stream for recording connection summaries,
         and the ``http`` stream for recording HTTP activity.
 
     Filters
         Each stream has a set of filters attached to it that determine
         what information gets written out. By default, each stream has
-        one default filter that just logs everything directly to disk
+        one default filter that just logs everything directly to disk.
-        with an automatically generated file name. However, further
+        However, additional filters can be added to record only a subset
-        filters can be added to record only a subset, split a stream
+        of the log records, write to different outputs, or set a custom
-        into different outputs, or to even duplicate the log to
+        rotation interval.  If all filters are removed from a stream,
-        multiple outputs. If all filters are removed from a stream,
+        then output is disabled for that stream.
-        all output is disabled.
 
     Writers
-        A writer defines the actual output format for the information
+        Each filter has a writer.  A writer defines the actual output
-        being logged. At the moment, Bro comes with only one type of
+        format for the information being logged. The default writer is
-        writer, which produces tab separated ASCII files. In the
+        the ASCII writer, which produces tab-separated ASCII files. Other
-        future we will add further writers, like for binary output and
+        writers are available, like for binary output or direct logging
-        direct logging into a database.
+        into a database.
 
-Basics
+There are several different ways to customize Bro's logging: you can create
-======
+a new log stream, you can extend an existing log with new fields, you
+can apply filters to an existing log stream, or you can customize the output
+format by setting log writer options.  All of these approaches are
+described in this document.
 
-The data fields that a stream records are defined by a record type
+Streams
-specified when it is created. Let's look at the script generating Bro's
+=======
-connection summaries as an example,
-:doc:`/scripts/base/protocols/conn/main.bro`. It defines a record
-:bro:type:`Conn::Info` that lists all the fields that go into
-``conn.log``, each marked with a ``&log`` attribute indicating that it
-is part of the information written out. To write a log record, the
-script then passes an instance of :bro:type:`Conn::Info` to the logging
-framework's :bro:id:`Log::write` function.
 
-By default, each stream automatically gets a filter named ``default``
+In order to log data to a new log stream, all of the following needs to be
-that generates the normal output by recording all record fields into a
+done:
-single output file.
 
-In the following, we summarize ways in which the logging can be
+- A :bro:type:`record` type must be defined which consists of all the
-customized. We continue using the connection summaries as our example
+  fields that will be logged (by convention, the name of this record type is
-to work with.
+  usually "Info").
+- A log stream ID (an :bro:type:`enum` with type name "Log::ID") must be
+  defined that uniquely identifies the new log stream.
+- A log stream must be created using the :bro:id:`Log::create_stream` function.
+- When the data to be logged becomes available, the :bro:id:`Log::write`
+  function must be called.
 
-Filtering
+In the following example, we create a new module "Foo" which creates
----------
+a new log stream.
-
-To create a new output file for an existing stream, you can add a
-new filter. A filter can, e.g., restrict the set of fields being
-logged:
 
 .. code:: bro
 
-    event bro_init()
+    module Foo;
+
+    export {
+        # Create an ID for our new stream. By convention, this is
+        # called "LOG".
+        redef enum Log::ID += { LOG };
+
+        # Define the record type that will contain the data to log.
+        type Info: record {
+            ts: time        &log;
+            id: conn_id     &log;
+            service: string &log &optional;
+            missed_bytes: count &log &default=0;
+        };
+    }
+
+    # Optionally, we can add a new field to the connection record so that
+    # the data we are logging (our "Info" record) will be easily
+    # accessible in a variety of event handlers.
+    redef record connection += {
+        # By convention, the name of this new field is the lowercase name
+        # of the module.
+        foo: Info &optional;
+    };
+
+    # This event is handled at a priority higher than zero so that if
+    # users modify this stream in another script, they can do so at the
+    # default priority of zero.
+    event bro_init() &priority=5
         {
-        # Add a new filter to the Conn::LOG stream that logs only
+        # Create the stream. This adds a default filter automatically.
-        # timestamp and originator address.
+        Log::create_stream(Foo::LOG, [$columns=Info, $path="foo"]);
-        local filter: Log::Filter = [$name="orig-only", $path="origs", $include=set("ts", "id.orig_h")];
-        Log::add_filter(Conn::LOG, filter);
         }
 
-Note the fields that are set for the filter:
+In the definition of the "Info" record above, notice that each field has the
+:bro:attr:`&log` attribute.  Without this attribute, a field will not appear in
+the log output. Also notice one field has the :bro:attr:`&optional` attribute.
+This indicates that the field might not be assigned any value before the
+log record is written.  Finally, a field with the :bro:attr:`&default`
+attribute has a default value assigned to it automatically.
 
-    ``name``
+At this point, the only thing missing is a call to the :bro:id:`Log::write`
-        A mandatory name for the filter that can later be used
+function to send data to the logging framework.  The actual event handler
-        to manipulate it further.
+where this should take place will depend on where your data becomes available.
-
+In this example, the :bro:id:`connection_established` event provides our data,
-    ``path``
+and we also store a copy of the data being logged into the
-        The filename for the output file, without any extension (which
+:bro:type:`connection` record:
-        may be automatically added by the writer). Default path values
-        are generated by taking the stream's ID and munging it slightly.
-        :bro:enum:`Conn::LOG` is converted into ``conn``,
-        :bro:enum:`PacketFilter::LOG` is converted into
-        ``packet_filter``, and :bro:enum:`Known::CERTS_LOG` is
-        converted into ``known_certs``.
-
-    ``include``
-        A set limiting the fields to the ones given. The names
-        correspond to those in the :bro:type:`Conn::Info` record, with
-        sub-records unrolled by concatenating fields (separated with
-        dots).
-
-Using the code above, you will now get a new log file ``origs.log``
-that looks like this::
-
-    #separator \x09
-    #path   origs
-    #fields ts      id.orig_h
-    #types  time    addr
-    1128727430.350788       141.42.64.125
-    1128727435.450898       141.42.64.125
-
-If you want to make this the only log file for the stream, you can
-remove the default filter (which, conveniently, has the name
-``default``):
 
 .. code:: bro
 
-    event bro_init()
+    event connection_established(c: connection)
         {
-        # Remove the filter called "default".
+        local rec: Foo::Info = [$ts=network_time(), $id=c$id];
-        Log::remove_filter(Conn::LOG, "default");
+
+        # Store a copy of the data in the connection record so other
+        # event handlers can access it.
+        c$foo = rec;
+
+        Log::write(Foo::LOG, rec);
         }
 
-An alternate approach to "turning off" a log is to completely disable
+If you run Bro with this script, a new log file ``foo.log`` will be created.
-the stream:
+Although we only specified four fields in the "Info" record above, the
+log output will actually contain seven fields because one of the fields
+(the one named "id") is itself a record type.  Since a :bro:type:`conn_id`
+record has four fields, then each of these fields is a separate column in
+the log output.  Note that the way that such fields are named in the log
+output differs slightly from the way we would refer to the same field
+in a Bro script (each dollar sign is replaced with a period).  For example,
+to access the first field of a ``conn_id`` in a Bro script we would use
+the notation ``id$orig_h``, but that field is named ``id.orig_h``
+in the log output.
 
-.. code:: bro
+When you are developing scripts that add data to the :bro:type:`connection`
+record, care must be given to when and how long data is stored.
+Normally data saved to the connection record will remain there for the
+duration of the connection and from a practical perspective it's not
+uncommon to need to delete that data before the end of the connection.
 
-    event bro_init()
-        {
-        Log::disable_stream(Conn::LOG);
-        }
 
-If you want to skip only some fields but keep the rest, there is a
+Add Fields to a Log
-corresponding ``exclude`` filter attribute that you can use instead of
+-------------------
-``include`` to list only the ones you are not interested in.
 
-A filter can also determine output paths *dynamically* based on the
+You can add additional fields to a log by extending the record
-record being logged. That allows, e.g., to record local and remote
+type that defines its content, and setting a value for the new fields
-connections into separate files. To do this, you define a function
+before each log record is written.
-that returns the desired path:
 
-.. code:: bro
+Let's say we want to add a boolean field ``is_private`` to
-
+:bro:type:`Conn::Info` that indicates whether the originator IP address
-    function split_log(id: Log::ID, path: string, rec: Conn::Info) : string
+is part of the :rfc:`1918` space:
-        {
-        # Return "conn-local" if originator is a local IP, otherwise "conn-remote".
-        local lr = Site::is_local_addr(rec$id$orig_h) ? "local" : "remote";
-        return fmt("%s-%s", path, lr);
-        }
-
-    event bro_init()
-        {
-        local filter: Log::Filter = [$name="conn-split", $path_func=split_log, $include=set("ts", "id.orig_h")];
-        Log::add_filter(Conn::LOG, filter);
-        }
-
-Running this will now produce two files, ``local.log`` and
-``remote.log``, with the corresponding entries. One could extend this
-further for example to log information by subnets or even by IP
-address. Be careful, however, as it is easy to create many files very
-quickly ...
-
-.. sidebar:: A More Generic Path Function
-
-    The ``split_log`` method has one draw-back: it can be used
-    only with the :bro:enum:`Conn::LOG` stream as the record type is hardcoded
-    into its argument list. However, Bro allows to do a more generic
-    variant:
-
-    .. code:: bro
-
-        function split_log(id: Log::ID, path: string, rec: record { id: conn_id; } ) : string
-            {
-            return Site::is_local_addr(rec$id$orig_h) ? "local" : "remote";
-            }
-
-    This function can be used with all log streams that have records
-    containing an ``id: conn_id`` field.
-
-While so far we have seen how to customize the columns being logged,
-you can also control which records are written out by providing a
-predicate that will be called for each log record:
-
-.. code:: bro
-
-    function http_only(rec: Conn::Info) : bool
-        {
-        # Record only connections with successfully analyzed HTTP traffic
-        return rec$service == "http";
-        }
-
-    event bro_init()
-        {
-        local filter: Log::Filter = [$name="http-only", $path="conn-http", $pred=http_only];
-        Log::add_filter(Conn::LOG, filter);
-        }
-
-This will result in a log file ``conn-http.log`` that contains only
-traffic detected and analyzed as HTTP traffic.
-
-Extending
----------
-
-You can add further fields to a log stream by extending the record
-type that defines its content. Let's say we want to add a boolean
-field ``is_private`` to :bro:type:`Conn::Info` that indicates whether the
-originator IP address is part of the :rfc:`1918` space:
 
 .. code:: bro
 
@@ -218,9 +167,21 @@ originator IP address is part of the :rfc:`1918` space:
         is_private: bool &default=F &log;
     };
 
+As this example shows, when extending a log stream's "Info" record, each
+new field must always be declared either with a ``&default`` value or
+as ``&optional``.  Furthermore, you need to add the ``&log`` attribute
+or otherwise the field won't appear in the log file.
 
-Now we need to set the field. A connection's summary is generated at
+Now we need to set the field.  Although the details vary depending on which
-the time its state is removed from memory. We can add another handler
+log is being extended, in general it is important to choose a suitable event
+in which to set the additional fields because we need to make sure that
+the fields are set before the log record is written.  Sometimes the right
+choice is the same event which writes the log record, but at a higher
+priority (in order to ensure that the event handler that sets the additional
+fields is executed before the event handler that writes the log record).
+
+In this example, since a connection's summary is generated at
+the time its state is removed from memory, we can add another handler
 at that time that sets our field correctly:
 
 .. code:: bro
@@ -232,31 +193,58 @@ at that time that sets our field correctly:
         }
 
 Now ``conn.log`` will show a new field ``is_private`` of type
-``bool``.
+``bool``.  If you look at the Bro script which defines the connection
+log stream :doc:`/scripts/base/protocols/conn/main.bro`, you will see
+that ``Log::write`` gets called in an event handler for the
+same event as used in this example to set the additional fields, but at a
+lower priority than the one used in this example (i.e., the log record gets
+written after we assign the ``is_private`` field).
 
-Notes:
+For extending logs this way, one needs a bit of knowledge about how
+the script that creates the log stream is organizing its state
+keeping. Most of the standard Bro scripts attach their log state to
+the :bro:type:`connection` record where it can then be accessed, just
+like ``c$conn`` above. For example, the HTTP analysis adds a field
+``http`` of type :bro:type:`HTTP::Info` to the :bro:type:`connection`
+record.
 
-- For extending logs this way, one needs a bit of knowledge about how
-  the script that creates the log stream is organizing its state
-  keeping. Most of the standard Bro scripts attach their log state to
-  the :bro:type:`connection` record where it can then be accessed, just
-  as the ``c$conn`` above. For example, the HTTP analysis adds a field
-  ``http`` of type :bro:type:`HTTP::Info` to the :bro:type:`connection`
-  record. See the script reference for more information.
 
-- When extending records as shown above, the new fields must always be
+Define a Logging Event
-  declared either with a ``&default`` value or as ``&optional``.
+----------------------
-  Furthermore, you need to add the ``&log`` attribute or otherwise the
-  field won't appear in the output.
-
-Hooking into the Logging
-------------------------
 
 Sometimes it is helpful to do additional analysis of the information
 being logged. For these cases, a stream can specify an event that will
-be generated every time a log record is written to it. All of Bro's
+be generated every time a log record is written to it.  To do this, we
-default log streams define such an event. For example, the connection
+need to modify the example module shown above to look something like this:
-log stream raises the event :bro:id:`Conn::log_conn`. You
+
+.. code:: bro
+
+    module Foo;
+
+    export {
+        redef enum Log::ID += { LOG };
+
+        type Info: record {
+            ts: time     &log;
+            id: conn_id  &log;
+            service: string &log &optional;
+            missed_bytes: count &log &default=0;
+        };
+
+        # Define a logging event. By convention, this is called
+        # "log_<stream>".
+        global log_foo: event(rec: Info);
+    }
+
+    event bro_init() &priority=5
+        {
+        # Specify the "log_foo" event here in order for Bro to raise it.
+        Log::create_stream(Foo::LOG, [$columns=Info, $ev=log_foo,
+                           $path="foo"]);
+        }
+
+All of Bro's default log streams define such an event. For example, the
+connection log stream raises the event :bro:id:`Conn::log_conn`. You
 could use that for example for flagging when a connection to a
 specific destination exceeds a certain duration:
 
@@ -270,7 +258,7 @@ specific destination exceeds a certain duration:
 
     event Conn::log_conn(rec: Conn::Info)
         {
-        if ( rec$duration > 5mins )
+        if ( rec?$duration && rec$duration > 5mins )
             NOTICE([$note=Long_Conn_Found,
                     $msg=fmt("unusually long conn to %s", rec$id$resp_h),
                     $id=rec$id]);
@@ -281,15 +269,196 @@ externally with Perl scripts. Much of what such an external script
 would do later offline, one may instead do directly inside of Bro in
 real-time.
 
-Rotation
+Disable a Stream
---------
+----------------
 
-By default, no log rotation occurs, but it's globally controllable for all
+One way to "turn off" a log is to completely disable the stream.  For
-filters by redefining the :bro:id:`Log::default_rotation_interval` option:
+example, the following example will prevent the conn.log from being written:
 
 .. code:: bro
 
-    redef Log::default_rotation_interval = 1 hr;
+    event bro_init()
+        {
+        Log::disable_stream(Conn::LOG);
+        }
+
+Note that this must run after the stream is created, so the priority
+of this event handler must be lower than the priority of the event handler
+where the stream was created.
+
+
+Filters
+=======
+
+A stream has one or more filters attached to it (a stream without any filters
+will not produce any log output).  When a stream is created, it automatically
+gets a default filter attached to it.  This default filter can be removed
+or replaced, or other filters can be added to the stream.  This is accomplished
+by using either the :bro:id:`Log::add_filter` or :bro:id:`Log::remove_filter`
+function.  This section shows how to use filters to do such tasks as
+rename a log file, split the output into multiple files, control which
+records are written, and set a custom rotation interval.
+
+Rename Log File
+---------------
+
+Normally, the log filename for a given log stream is determined when the
+stream is created, unless you explicitly specify a different one by adding
+a filter.
+
+The easiest way to change a log filename is to simply replace the
+default log filter with a new filter that specifies a value for the "path"
+field.  In this example, "conn.log" will be changed to "myconn.log":
+
+.. code:: bro
+
+    event bro_init()
+        {
+        # Replace default filter for the Conn::LOG stream in order to
+        # change the log filename.
+
+        local f = Log::get_filter(Conn::LOG, "default");
+        f$path = "myconn";
+        Log::add_filter(Conn::LOG, f);
+        }
+
+Keep in mind that the "path" field of a log filter never contains the
+filename extension.  The extension will be determined later by the log writer.
+
+Add a New Log File
+------------------
+
+Normally, a log stream writes to only one log file.  However, you can
+add filters so that the stream writes to multiple files.  This is useful
+if you want to restrict the set of fields being logged to the new file.
+
+In this example, a new filter is added to the Conn::LOG stream that writes
+two fields to a new log file:
+
+.. code:: bro
+
+    event bro_init()
+        {
+        # Add a new filter to the Conn::LOG stream that logs only
+        # timestamp and originator address.
+
+        local filter: Log::Filter = [$name="orig-only", $path="origs",
+                                     $include=set("ts", "id.orig_h")];
+        Log::add_filter(Conn::LOG, filter);
+        }
+
+
+Notice how the "include" filter attribute specifies a set that limits the
+fields to the ones given. The names correspond to those in the
+:bro:type:`Conn::Info` record (however, because the "id" field is itself a
+record, we can specify an individual field of "id" by the dot notation
+shown in the example).
+
+Using the code above, in addition to the regular ``conn.log``, you will
+now also get a new log file ``origs.log`` that looks like the regular
+``conn.log``, but will have only the fields specified in the "include"
+filter attribute.
+
+If you want to skip only some fields but keep the rest, there is a
+corresponding ``exclude`` filter attribute that you can use instead of
+``include`` to list only the ones you are not interested in.
+
+If you want to make this the only log file for the stream, you can
+remove the default filter:
+
+.. code:: bro
+
+    event bro_init()
+        {
+        # Remove the filter called "default".
+        Log::remove_filter(Conn::LOG, "default");
+        }
+
+Determine Log Path Dynamically
+------------------------------
+
+Instead of using the "path" filter attribute, a filter can determine
+output paths *dynamically* based on the record being logged. That
+allows, e.g., to record local and remote connections into separate
+files. To do this, you define a function that returns the desired path,
+and use the "path_func" filter attribute:
+
+.. code:: bro
+
+    # Note: if using BroControl then you don't need to redef local_nets.
+    redef Site::local_nets = { 192.168.0.0/16 };
+
+    function myfunc(id: Log::ID, path: string, rec: Conn::Info) : string
+        {
+        # Return "conn-local" if originator is a local IP, otherwise
+        # return "conn-remote".
+        local r = Site::is_local_addr(rec$id$orig_h) ? "local" : "remote";
+        return fmt("%s-%s", path, r);
+        }
+
+    event bro_init()
+        {
+        local filter: Log::Filter = [$name="conn-split",
+                 $path_func=myfunc, $include=set("ts", "id.orig_h")];
+        Log::add_filter(Conn::LOG, filter);
+        }
+
+Running this will now produce two new files, ``conn-local.log`` and
+``conn-remote.log``, with the corresponding entries (for this example to work,
+the ``Site::local_nets`` must specify your local network). One could extend
+this further for example to log information by subnets or even by IP
+address. Be careful, however, as it is easy to create many files very
+quickly.
+
+The ``myfunc`` function has one drawback: it can be used
+only with the :bro:enum:`Conn::LOG` stream as the record type is hardcoded
+into its argument list. However, Bro allows to do a more generic
+variant:
+
+.. code:: bro
+
+    function myfunc(id: Log::ID, path: string,
+                    rec: record { id: conn_id; } ) : string
+        {
+        local r = Site::is_local_addr(rec$id$orig_h) ? "local" : "remote";
+        return fmt("%s-%s", path, r);
+        }
+
+This function can be used with all log streams that have records
+containing an ``id: conn_id`` field.
+
+Filter Log Records
+------------------
+
+We have seen how to customize the columns being logged, but
+you can also control which records are written out by providing a
+predicate that will be called for each log record:
+
+.. code:: bro
+
+    function http_only(rec: Conn::Info) : bool
+        {
+        # Record only connections with successfully analyzed HTTP traffic
+        return rec?$service && rec$service == "http";
+        }
+
+    event bro_init()
+        {
+        local filter: Log::Filter = [$name="http-only", $path="conn-http",
+                                     $pred=http_only];
+        Log::add_filter(Conn::LOG, filter);
+        }
+
+This will result in a new log file ``conn-http.log`` that contains only
+the log records from ``conn.log`` that are analyzed as HTTP traffic.
+
+Rotation
+--------
+
+The log rotation interval is globally controllable for all
+filters by redefining the :bro:id:`Log::default_rotation_interval` option
+(note that when using BroControl, this option is set automatically via
+the BroControl configuration).
 
 Or specifically for certain :bro:type:`Log::Filter` instances by setting
 their ``interv`` field.  Here's an example of changing just the
@@ -301,90 +470,73 @@ their ``interv`` field.  Here's an example of changing just the
         {
         local f = Log::get_filter(Conn::LOG, "default");
         f$interv = 1 min;
-        Log::remove_filter(Conn::LOG, "default");
         Log::add_filter(Conn::LOG, f);
         }
 
-ASCII Writer Configuration
+Writers
---------------------------
+=======
 
-The ASCII writer has a number of options for customizing the format of
+Each filter has a writer.  If you do not specify a writer when adding a
-its output, see :doc:`/scripts/base/frameworks/logging/writers/ascii.bro`.
+filter to a stream, then the ASCII writer is the default.
 
-Adding Streams
+There are two ways to specify a non-default writer.  To change the default
-==============
+writer for all log filters, just redefine the :bro:id:`Log::default_writer`
+option.  Alternatively, you can specify the writer to use on a per-filter
+basis by setting a value for the filter's "writer" field.  Consult the
+documentation of the writer to use to see if there are other options that are
+needed.
 
-It's easy to create a new log stream for custom scripts. Here's an
+ASCII Writer
-example for the ``Foo`` module:
+------------
+
+By default, the ASCII writer outputs log files that begin with several
+lines of metadata, followed by the actual log output.  The metadata
+describes the format of the log file, the "path" of the log (i.e., the log
+filename without file extension), and also specifies the time that the log
+was created and the time when Bro finished writing to it.
+The ASCII writer has a number of options for customizing the format of its
+output, see :doc:`/scripts/base/frameworks/logging/writers/ascii.bro`.
+If you change the output format options, then be careful to check whether
+your postprocessing scripts can still recognize your log files.
+
+Some writer options are global (i.e., they affect all log filters using
+that log writer).  For example, to change the output format of all ASCII
+logs to JSON format:
 
 .. code:: bro
 
-    module Foo;
+    redef LogAscii::use_json = T;
 
-    export {
+Some writer options are filter-specific (i.e., they affect only the filters
-        # Create an ID for our new stream. By convention, this is
+that explicitly specify the option).  For example, to change the output
-        # called "LOG".
+format of the ``conn.log`` only:
-        redef enum Log::ID += { LOG };
 
-        # Define the fields. By convention, the type is called "Info".
+.. code:: bro
-        type Info: record {
-            ts: time     &log;
-            id: conn_id  &log;
-        };
 
-        # Define a hook event. By convention, this is called
+    event bro_init()
-        # "log_<stream>".
-        global log_foo: event(rec: Info);
-
-    }
-
-    # This event should be handled at a higher priority so that when
-    # users modify your stream later and they do it at priority 0,
-    # their code runs after this.
-    event bro_init() &priority=5
         {
-        # Create the stream. This also adds a default filter automatically.
+        local f = Log::get_filter(Conn::LOG, "default");
-        Log::create_stream(Foo::LOG, [$columns=Info, $ev=log_foo, $path="foo"]);
+        # Use tab-separated-value mode
+        f$config = table(["tsv"] = "T");
+        Log::add_filter(Conn::LOG, f);
         }
 
-You can also add the state to the :bro:type:`connection` record to make
-it easily accessible across event handlers:
-
-.. code:: bro
-
-    redef record connection += {
-        foo: Info &optional;
-        }
-
-Now you can use the :bro:id:`Log::write` method to output log records and
-save the logged ``Foo::Info`` record into the connection record:
-
-.. code:: bro
-
-    event connection_established(c: connection)
-        {
-        local rec: Foo::Info = [$ts=network_time(), $id=c$id];
-        c$foo = rec;
-        Log::write(Foo::LOG, rec);
-        }
-
-See the existing scripts for how to work with such a new connection
-field. A simple example is :doc:`/scripts/base/protocols/syslog/main.bro`.
-
-When you are developing scripts that add data to the :bro:type:`connection`
-record, care must be given to when and how long data is stored.
-Normally data saved to the connection record will remain there for the
-duration of the connection and from a practical perspective it's not
-uncommon to need to delete that data before the end of the connection.
 
 Other Writers
 -------------
 
-Bro supports the following built-in output formats other than ASCII:
+Bro supports the following additional built-in output formats:
 
 .. toctree::
    :maxdepth: 1
 
    logging-input-sqlite
 
-Further formats are available as external plugins.
+Additional writers are available as external plugins:
+
+.. toctree::
+   :maxdepth: 1
+
+   ../components/bro-plugins/dataseries/README
+   ../components/bro-plugins/elasticsearch/README
+

doc/install/guidelines.rst

@@ -8,10 +8,12 @@ How to Upgrade
 If you're doing an upgrade install (rather than a fresh install),
 there's two suggested approaches: either install Bro using the same
 installation prefix directory as before, or pick a new prefix and copy
-local customizations over.  Regardless of which approach you choose,
+local customizations over.
-if you are using BroControl, then after upgrading Bro you will need to
+
-run "broctl check" (to verify that your new configuration is OK)
+Regardless of which approach you choose, if you are using BroControl, then
-and "broctl install" to complete the upgrade process.
+before doing the upgrade you should stop all running Bro processes with the
+"broctl stop" command.  After the upgrade is complete then you will need
+to run "broctl deploy".
 
 In the following we summarize general guidelines for upgrading, see
 the :ref:`release-notes` for version-specific information.

doc/install/install.rst

@@ -46,8 +46,7 @@ To build Bro from source, the following additional dependencies are required:
     * zlib headers
     * Perl
 
-To install the required dependencies, you can use (when done, make sure
+To install the required dependencies, you can use:
-that ``bash`` and ``python`` are in your ``PATH``):
 
 * RPM/RedHat-based Linux:
 
@@ -68,13 +67,17 @@ that ``bash`` and ``python`` are in your ``PATH``):
 
   .. console::
 
-      sudo pkg_add -r bash cmake swig bison python perl py27-sqlite3
+      sudo pkg install bash cmake swig bison python perl py27-sqlite3
+
+  Note that in older versions of FreeBSD, you might have to use the
+  "pkg_add -r" command instead of "pkg install".
 
 * Mac OS X:
 
-  Compiling source code on Macs requires first downloading Xcode_,
+  Compiling source code on Macs requires first installing Xcode_ (in older
-  then going through its "Preferences..." -> "Downloads" menus to
+  versions of Xcode, you would then need to go through its
-  install the "Command Line Tools" component.
+  "Preferences..." -> "Downloads" menus to install the "Command Line Tools"
+  component).
 
   OS X comes with all required dependencies except for CMake_ and SWIG_.
   Distributions of these dependencies can likely be obtained from your
@@ -94,7 +97,6 @@ build time:
     * curl (used by a Bro script that implements active HTTP)
     * gperftools (tcmalloc is used to improve memory and CPU usage)
     * ipsumdump (for trace-summary; http://www.cs.ucla.edu/~kohler/ipsumdump)
-    * Ruby executable, library, and headers (for Broccoli Ruby bindings)
 
 LibGeoIP is probably the most interesting and can be installed
 on most platforms by following the instructions for :ref:`installing
@@ -119,8 +121,8 @@ platforms for binary releases and for installation instructions.
 
   Linux based binary installations are usually performed by adding
   information about the Bro packages to the respective system packaging
-  tool. Theen the usual system utilities such as ``apt``, ``yum``
+  tool. Then the usual system utilities such as ``apt``, ``yum``
-  or ``zyppper`` are used to perforn the installation. By default,
+  or ``zypper`` are used to perform the installation. By default,
   installations of binary packages will go into ``/opt/bro``.
 
 * MacOS Disk Image with Installer
@@ -131,7 +133,7 @@ platforms for binary releases and for installation instructions.
 The primary install prefix for binary packages is ``/opt/bro``.
 
 Installing from Source
-==========================
+======================
 
 Bro releases are bundled into source packages for convenience and are
 available on the `bro downloads page`_. Alternatively, the latest

doc/quickstart/index.rst

@@ -24,9 +24,10 @@ Managing Bro with BroControl
 BroControl is an interactive shell for easily operating/managing Bro
 installations on a single system or even across multiple systems in a
 traffic-monitoring cluster.  This section explains how to use BroControl
-to manage a stand-alone Bro installation.  For instructions on how to
+to manage a stand-alone Bro installation.  For a complete reference on
-configure a Bro cluster, see the :doc:`Cluster Configuration
+BroControl, see the :doc:`BroControl <../components/broctl/README>`
-<../configuration/index>` documentation.
+documentation.  For instructions on how to configure a Bro cluster,
+see the :doc:`Cluster Configuration <../configuration/index>` documentation.
 
 A Minimal Starting Configuration
 --------------------------------

doc/script-reference/attributes.rst

@@ -173,14 +173,20 @@ Here is a more detailed explanation of each attribute:
 
     Rotates a file after a specified interval.
 
+    Note: This attribute is deprecated and will be removed in a future release.
+
 .. bro:attr:: &rotate_size
 
     Rotates a file after it has reached a given size in bytes.
 
+    Note: This attribute is deprecated and will be removed in a future release.
+
 .. bro:attr:: &encrypt
 
     Encrypts files right before writing them to disk.
 
+    Note: This attribute is deprecated and will be removed in a future release.
+
 .. bro:attr:: &raw_output
 
     Opens a file in raw mode, i.e., non-ASCII characters are not
@@ -229,5 +235,4 @@ Here is a more detailed explanation of each attribute:
 
     The associated identifier is marked as deprecated and will be
     removed in a future version of Bro.  Look in the NEWS file for more
-    explanation and/or instructions to migrate code that uses deprecated
+    instructions to migrate code that uses deprecated functionality.
-    functionality.

doc/script-reference/directives.rst

@@ -58,6 +58,23 @@ executed.  Directives are evaluated before script execution begins.
     for that script are ignored).
 
 
+.. bro:keyword:: @load-plugin
+
+    Activate a dynamic plugin with the specified plugin name.  The specified
+    plugin must be located in Bro's plugin search path.  Example::
+
+        @load-plugin Demo::Rot13
+
+    By default, Bro will automatically activate all dynamic plugins found
+    in the plugin search path (the search path can be changed by setting
+    the environment variable BRO_PLUGIN_PATH to a colon-separated list of
+    directories). However, in bare mode ("bro -b"), dynamic plugins can be
+    activated only by using "@load-plugin", or by specifying the full
+    plugin name on the Bro command-line (e.g., "bro Demo::Rot13"), or by
+    setting the environment variable BRO_PLUGIN_ACTIVATE to a
+    comma-separated list of plugin names.
+
+
 .. bro:keyword:: @load-sigs
 
     This works similarly to "@load", except that in this case the filename

doc/script-reference/log-files.rst

@@ -26,13 +26,21 @@ Network Protocols
 +----------------------------+---------------------------------------+---------------------------------+
 | irc.log                    | IRC commands and responses            | :bro:type:`IRC::Info`           |
 +----------------------------+---------------------------------------+---------------------------------+
+| kerberos.log               | Kerberos                              | :bro:type:`KRB::Info`           |
++----------------------------+---------------------------------------+---------------------------------+
 | modbus.log                 | Modbus commands and responses         | :bro:type:`Modbus::Info`        |
 +----------------------------+---------------------------------------+---------------------------------+
 | modbus_register_change.log | Tracks changes to Modbus holding      | :bro:type:`Modbus::MemmapInfo`  |
 |                            | registers                             |                                 |
 +----------------------------+---------------------------------------+---------------------------------+
+| mysql.log                  | MySQL                                 | :bro:type:`MySQL::Info`         |
++----------------------------+---------------------------------------+---------------------------------+
 | radius.log                 | RADIUS authentication attempts        | :bro:type:`RADIUS::Info`        |
 +----------------------------+---------------------------------------+---------------------------------+
+| rdp.log                    | RDP                                   | :bro:type:`RDP::Info`           |
++----------------------------+---------------------------------------+---------------------------------+
+| sip.log                    | SIP                                   | :bro:type:`SIP::Info`           |
++----------------------------+---------------------------------------+---------------------------------+
 | smtp.log                   | SMTP transactions                     | :bro:type:`SMTP::Info`          |
 +----------------------------+---------------------------------------+---------------------------------+
 | snmp.log                   | SNMP messages                         | :bro:type:`SNMP::Info`          |
@@ -56,6 +64,8 @@ Files
 +============================+=======================================+=================================+
 | files.log                  | File analysis results                 | :bro:type:`Files::Info`         |
 +----------------------------+---------------------------------------+---------------------------------+
+| pe.log                     | Portable Executable (PE)              | :bro:type:`PE::Info`            |
++----------------------------+---------------------------------------+---------------------------------+
 | x509.log                   | X.509 certificate info                | :bro:type:`X509::Info`          |
 +----------------------------+---------------------------------------+---------------------------------+
 

doc/script-reference/statements.rst

@@ -258,8 +258,8 @@ Here are the statements that the Bro scripting language supports.
 
 .. bro:keyword:: break
 
-    The "break" statement is used to break out of a :bro:keyword:`switch` or
+    The "break" statement is used to break out of a :bro:keyword:`switch`,
-    :bro:keyword:`for` statement.
+    :bro:keyword:`for`, or :bro:keyword:`while` statement.
 
 
 .. bro:keyword:: delete
@@ -379,10 +379,10 @@ Here are the statements that the Bro scripting language supports.
 
 .. bro:keyword:: next
 
-    The "next" statement can only appear within a :bro:keyword:`for` loop.
+    The "next" statement can only appear within a :bro:keyword:`for` or
-    It causes execution to skip to the next iteration.
+    :bro:keyword:`while` loop.  It causes execution to skip to the next
+    iteration.
 
-    For an example, see the :bro:keyword:`for` statement.
 
 .. bro:keyword:: print
 
@@ -571,7 +571,7 @@ Here are the statements that the Bro scripting language supports.
 
 .. bro:keyword:: while
 
-    A "while" loop iterates over a body statement as long a given
+    A "while" loop iterates over a body statement as long as a given
     condition remains true.
 
     A :bro:keyword:`break` statement can be used at any time to immediately
@@ -609,8 +609,8 @@ Here are the statements that the Bro scripting language supports.
     (outside of the braces) of a compound statement.
 
     A compound statement is required in order to execute more than one
-    statement in the body of a :bro:keyword:`for`, :bro:keyword:`if`, or
+    statement in the body of a :bro:keyword:`for`, :bro:keyword:`while`,
-    :bro:keyword:`when` statement.
+    :bro:keyword:`if`, or :bro:keyword:`when` statement.
 
     Example::
 

man/bro.8

@@ -51,12 +51,6 @@ add given prefix to policy file resolution
 \fB\-r\fR,\ \-\-readfile <readfile>
 read from given tcpdump file
 .TP
-\fB\-y\fR,\ \-\-flowfile <file>[=<ident>]
-read from given flow file
-.TP
-\fB\-Y\fR,\ \-\-netflow <ip>:<prt>[=<id>]
-read flow from socket
-.TP
 \fB\-s\fR,\ \-\-rulefile <rulefile>
 read rules from given file
 .TP
@@ -78,27 +72,21 @@ run the specified policy file analysis
 \fB\-C\fR,\ \-\-no\-checksums
 ignore checksums
 .TP
-\fB\-D\fR,\ \-\-dfa\-size <size>
-DFA state cache size
-.TP
 \fB\-F\fR,\ \-\-force\-dns
 force DNS
 .TP
 \fB\-I\fR,\ \-\-print\-id <ID name>
 print out given ID
 .TP
+\fB\-J\fR,\ \-\-set\-seed <seed>
+set the random number seed
+.TP
 \fB\-K\fR,\ \-\-md5\-hashkey <hashkey>
 set key for MD5\-keyed hashing
 .TP
-\fB\-L\fR,\ \-\-rule\-benchmark
-benchmark for rules
-.TP
 \fB\-N\fR,\ \-\-print\-plugins
 print available plugins and exit (\fB\-NN\fR for verbose)
 .TP
-\fB\-O\fR,\ \-\-optimize
-optimize policy script
-.TP
 \fB\-P\fR,\ \-\-prime\-dns
 prime DNS
 .TP
@@ -120,7 +108,7 @@ Record process status in file
 \fB\-W\fR,\ \-\-watchdog
 activate watchdog timer
 .TP
-\fB\-X\fR,\ \-\-broxygen
+\fB\-X\fR,\ \-\-broxygen <cfgfile>
 generate documentation based on config file
 .TP
 \fB\-\-pseudo\-realtime[=\fR<speedup>]
@@ -131,6 +119,19 @@ load seeds from given file
 .TP
 \fB\-\-save\-seeds\fR <file>
 save seeds to given file
+.TP
+The following option is available only when Bro is built with the \-\-enable\-debug configure option:
+.TP
+\fB\-B\fR,\ \-\-debug <dbgstreams>
+Enable debugging output for selected streams ('-B help' for help)
+.TP
+The following options are available only when Bro is built with gperftools support (use the \-\-enable\-perftools and \-\-enable\-perftools\-debug configure options):
+.TP
+\fB\-m\fR,\ \-\-mem-leaks
+show leaks
+.TP
+\fB\-M\fR,\ \-\-mem-profile
+record heap
 .SH ENVIRONMENT
 .TP
 .B BROPATH

scripts/base/files/pe/README

@@ -0,0 +1 @@
+Support for Portable Executable (PE) file analysis.

scripts/base/frameworks/broker/README

@@ -0,0 +1,2 @@
+The Broker communication framework facilitates connecting to remote Bro
+instances to share state and transfer events.

scripts/base/frameworks/logging/main.bro

@@ -6,9 +6,10 @@
 module Log;
 
 export {
-	## Type that defines an ID unique to each log stream. Scripts creating new log
+	## Type that defines an ID unique to each log stream. Scripts creating new
-	## streams need to redef this enum to add their own specific log ID. The log ID
+	## log streams need to redef this enum to add their own specific log ID.
-	## implicitly determines the default name of the generated log file.
+	## The log ID implicitly determines the default name of the generated log
+	## file.
 	type Log::ID: enum {
 		## Dummy place-holder.
 		UNKNOWN
@@ -20,25 +21,24 @@ export {
 	## If true, remote logging is by default enabled for all filters.
 	const enable_remote_logging = T &redef;
 
-	## Default writer to use if a filter does not specify
+	## Default writer to use if a filter does not specify anything else.
-	## anything else.
 	const default_writer = WRITER_ASCII &redef;
 
-	## Default separator between fields for logwriters.
+	## Default separator to use between fields.
-	## Can be overwritten by individual writers.
+	## Individual writers can use a different value.
 	const separator = "\t" &redef;
 
-	## Separator between set elements.
+	## Default separator to use between elements of a set.
-	## Can be overwritten by individual writers.
+	## Individual writers can use a different value.
 	const set_separator = "," &redef;
 
-	## String to use for empty fields. This should be different from
+	## Default string to use for empty fields. This should be different
-	## *unset_field* to make the output unambiguous.
+	## from *unset_field* to make the output unambiguous.
-	## Can be overwritten by individual writers.
+	## Individual writers can use a different value.
 	const empty_field = "(empty)" &redef;
 
-	## String to use for an unset &optional field.
+	## Default string to use for an unset &optional field.
-	## Can be overwritten by individual writers.
+	## Individual writers can use a different value.
 	const unset_field = "-" &redef;
 
 	## Type defining the content of a logging stream.
@@ -69,7 +69,7 @@ export {
 	##       If no ``path`` is defined for the filter, then the first call
 	##       to the function will contain an empty string.
 	##
-	## rec: An instance of the streams's ``columns`` type with its
+	## rec: An instance of the stream's ``columns`` type with its
 	##      fields set to the values to be logged.
 	##
 	## Returns: The path to be used for the filter.
@@ -87,7 +87,8 @@ export {
 		terminating: bool;	##< True if rotation occured due to Bro shutting down.
 	};
 
-	## Default rotation interval. Zero disables rotation.
+	## Default rotation interval to use for filters that do not specify
+	## an interval. Zero disables rotation.
 	##
 	## Note that this is overridden by the BroControl LogRotationInterval
 	## option.
@@ -122,8 +123,8 @@ export {
 		## Indicates whether a log entry should be recorded.
 		## If not given, all entries are recorded.
 		##
-		## rec: An instance of the streams's ``columns`` type with its
+		## rec: An instance of the stream's ``columns`` type with its
-		##      fields set to the values to logged.
+		##      fields set to the values to be logged.
 		##
 		## Returns: True if the entry is to be recorded.
 		pred: function(rec: any): bool &optional;
@@ -131,10 +132,10 @@ export {
 		## Output path for recording entries matching this
 		## filter.
 		##
-		## The specific interpretation of the string is up to
+		## The specific interpretation of the string is up to the
-		## the used writer, and may for example be the destination
+		## logging writer, and may for example be the destination
 		## file name. Generally, filenames are expected to be given
-		## without any extensions; writers will add appropiate
+		## without any extensions; writers will add appropriate
 		## extensions automatically.
 		##
 		## If this path is found to conflict with another filter's
@@ -151,7 +152,7 @@ export {
 		## easy to flood the disk by returning a new string for each
 		## connection.  Upon adding a filter to a stream, if neither
 		## ``path`` nor ``path_func`` is explicitly set by them, then
-		## :bro:see:`default_path_func` is used.
+		## :bro:see:`Log::default_path_func` is used.
 		##
 		## id: The ID associated with the log stream.
 		##
@@ -161,7 +162,7 @@ export {
 		##       then the first call to the function will contain an
 		##       empty string.
 		##
-		## rec: An instance of the streams's ``columns`` type with its
+		## rec: An instance of the stream's ``columns`` type with its
 		##      fields set to the values to be logged.
 		##
 		## Returns: The path to be used for the filter, which will be
@@ -185,7 +186,7 @@ export {
 		## If true, entries are passed on to remote peers.
 		log_remote: bool &default=enable_remote_logging;
 
-		## Rotation interval.
+		## Rotation interval. Zero disables rotation.
 		interv: interval &default=default_rotation_interval;
 
 		## Callback function to trigger for rotated files. If not set, the
@@ -215,9 +216,9 @@ export {
 
 	## Removes a logging stream completely, stopping all the threads.
 	##
-	## id: The ID enum to be associated with the new logging stream.
+	## id: The ID associated with the logging stream.
 	##
-	## Returns: True if a new stream was successfully removed.
+	## Returns: True if the stream was successfully removed.
 	##
 	## .. bro:see:: Log::create_stream
 	global remove_stream: function(id: ID) : bool;

scripts/base/frameworks/logging/writers/ascii.bro

@@ -1,15 +1,15 @@
 ##! Interface for the ASCII log writer.  Redefinable options are available
 ##! to tweak the output format of ASCII logs.
 ##!
-##! The ASCII writer supports currently one writer-specific filter option via
+##! The ASCII writer currently supports one writer-specific per-filter config
-##! ``config``: setting ``tsv`` to the string ``T`` turns the output into
+##! option: setting ``tsv`` to the string ``T`` turns the output into
 ##! "tab-separated-value" mode where only a single header row with the column
 ##! names is printed out as meta information, with no "# fields" prepended; no
-##! other meta data gets included in that mode.
+##! other meta data gets included in that mode.  Example filter using this::
 ##!
-##! Example filter using this::
+##!    local f: Log::Filter = [$name = "my-filter",
-##!
+##!                            $writer = Log::WRITER_ASCII,
-##!    local my_filter: Log::Filter = [$name = "my-filter", $writer = Log::WRITER_ASCII, $config = table(["tsv"] = "T")];
+##!                            $config = table(["tsv"] = "T")];
 ##!
 
 module LogAscii;
@@ -29,6 +29,8 @@ export {
 	## Format of timestamps when writing out JSON. By default, the JSON
 	## formatter will use double values for timestamps which represent the
 	## number of seconds from the UNIX epoch.
+	##
+	## This option is also available as a per-filter ``$config`` option.
 	const json_timestamps: JSON::TimestampFormat = JSON::TS_EPOCH &redef;
 
 	## If true, include lines with log meta information such as column names

scripts/base/frameworks/logging/writers/sqlite.bro

@@ -19,7 +19,7 @@ export {
 	const unset_field = Log::unset_field &redef;
 
 	## String to use for empty fields. This should be different from
-        ## *unset_field* to make the output unambiguous.
+	## *unset_field* to make the output unambiguous.
 	const empty_field = Log::empty_field &redef;
 }
 

scripts/base/protocols/krb/README

@@ -0,0 +1 @@
+Support for Kerberos protocol analysis.

scripts/base/protocols/krb/main.bro

@@ -1,4 +1,5 @@
-##! Implements base functionality for KRB analysis. Generates the krb.log file.
+##! Implements base functionality for KRB analysis. Generates the kerberos.log
+##! file.
 
 module KRB;
 

scripts/base/protocols/mysql/README

@@ -0,0 +1 @@
+Support for MySQL protocol analysis.

scripts/base/protocols/radius/README

@@ -0,0 +1 @@
+Support for RADIUS protocol analysis.

scripts/base/protocols/rdp/README

@@ -0,0 +1 @@
+Support for Remote Desktop Protocol (RDP) analysis.

scripts/base/protocols/sip/README

@@ -0,0 +1 @@
+Support for Session Initiation Protocol (SIP) analysis.

scripts/base/protocols/ssh/README

@@ -0,0 +1 @@
+Support for SSH protocol analysis.

src/strings.bif

@@ -700,7 +700,7 @@ function split_n%(str: string, re: pattern,
 ##
 ## Returns: An array of strings where, if *incl_sep* is true, each two
 ##          successive elements correspond to a substring in *str* of the part
-##          not matching *re* (event-indexed) and the part that matches *re*
+##          not matching *re* (even-indexed) and the part that matches *re*
 ##          (odd-indexed).
 ##
 ## .. bro:see:: split_string split_string1 split_string_all str_split