Distribution cleanup and documentation setupt tweaks.

COPYING

@@ -1,11 +1,12 @@
-Copyright (c) 1995-2010, The Regents of the University of California,
+Copyright (c) 1995-2011, The Regents of the University of California
-through Lawrence Berkeley National Laboratory. All rights reserved.
+through the Lawrence Berkeley National Laboratory and the
+International Computer Science Institute. All rights reserved.
 
 Redistribution and use in source and binary forms, with or without
 modification, are permitted provided that the following conditions are met:
 
-(1) Redistributions of source code must retain the above copyright notice,
+(1) Redistributions of source code must retain the above copyright
-    this list of conditions and the following disclaimer.
+    notice, this list of conditions and the following disclaimer.
 
 (2) Redistributions in binary form must reproduce the above copyright
     notice, this list of conditions and the following disclaimer in the
@@ -29,20 +30,5 @@ CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
 ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
 POSSIBILITY OF SUCH DAMAGE.
 
-Note that some files in the Bro distribution carry their own copyright
+Note that some files in the distribution may carry their own copyright
-notices.  The above applies to the Bro scripts in policy/ (other than as
+notices.
-noted below) and the source files in src/, other than:
-
-	policy/sigs/p0fsyn.osf
-	src/H3.h
-	src/OSFinger.cc
-	src/OSFinger.h
-	src/bsd-getopt-long.c
-	src/bsd-getopt-long.h
-	src/md5.c
-	src/md5.h
-	src/patricia.c
-	src/patricia.h
-
-In addition, other components, such as the build system, may have
-separate copyrights.

aux/binpac

@@ -1 +1 @@
-Subproject commit 952724eb355aa7d4ba50623e97a666dd7d1173a8
+Subproject commit 479f10b9d9abeb3024d358935691ed8bf9bb8038

aux/bro-aux

@@ -1 +1 @@
-Subproject commit 86c604193543e0fa85f7edeb132436f3d1b33ac7
+Subproject commit 94401ea11f78665973bcf833c2dd28c165d02bc6

aux/broccoli

@@ -1 +1 @@
-Subproject commit 042ce05428035b6bdd4863b9d316b7fd4bb5252e
+Subproject commit 31c3d1d9efa706969f0cd98223444c5f064a6fa7

aux/broctl

@@ -1 +1 @@
-Subproject commit 807a54fb264e0e2d00aa2267efd39738658ef320
+Subproject commit 8434c3f57772dab896f8c7c9c205a523dddf3f75

aux/btest

@@ -1 +1 @@
-Subproject commit 3c0b0e9a91060a7a453a5d6fb72ed1fd9071fda9
+Subproject commit 87c95911be41b36ca85f6d15dd86e127e9048c63

doc/.gitignore

@@ -0,0 +1 @@
+html

doc/Makefile

@@ -0,0 +1,7 @@
+
+all:
+	test -d html || mkdir html
+	for i in *.rst; do echo "$$i ..."; ./bin/rst2html.py $$i >html/`echo $$i | sed 's/rst$$/html/g'`; done
+
+clean:
+	rm -rf html

doc/README

@@ -1 +1,38 @@
-TODO
+
+Documentation
+=============
+
+This directory contains Bro documentation in reStructured text format
+(see http://docutils.sourceforge.net/rst.html).
+
+Please note that for now these files are primarily intended for use on
+http://www.bro-ids.org. While the Bro build process will render local
+versions into ``build/doc/`` (if docutils is found), the resulting
+HTML is very minimalistic and some features are not supported. In
+particular, some links will be broken.
+
+
+Notes for Writing Documentation
+-------------------------------
+
+* If you want to refer to a Bro script that's part of the
+  distribution, use {{'`foo.bro
+  <{{autodoc_bro_scripts}}/path/to/foo.html>`_'}}. For example,
+  ``{{'{{autodoc_bro_scripts}}/scripts/base/frameworks/notice/main.html}}'}}``.
+
+* If you want to refer to a page on the Bro web site, use the
+  ``docroot`` macro (e.g.,
+  ``{{'href="{{docroot}}/download/index.html"'}}). Make sure to
+  include the ``index.html`` for the main pages, just as in the
+  example.
+
+* If you want to refer to page inside this directory, use a relative
+  path with HTML extension. (e.g., ``href="quickstart.html``).
+
+Guidelines
+----------
+
+TODO.
+
+
+

doc/bin/rst2html.py

@@ -0,0 +1,62 @@
+#!/usr/bin/env python
+#
+# Derived from docutils standard rst2html.py.
+#
+# $Id: rst2html.py 4564 2006-05-21 20:44:42Z wiemann $
+# Author: David Goodger <goodger@python.org>
+# Copyright: This module has been placed in the public domain.
+#
+#
+# Extension: we add to dummy directorives "code" and "console" to be
+# compatible with Bro's web site setup.
+
+try:
+    import locale
+    locale.setlocale(locale.LC_ALL, '')
+except:
+    pass
+
+import textwrap
+
+from docutils.core import publish_cmdline, default_description
+
+from docutils import nodes
+from docutils.parsers.rst import directives, Directive
+from docutils.parsers.rst.directives.body import LineBlock
+
+class Literal(Directive):
+    #max_line_length = 68
+    max_line_length = 0
+
+    required_arguments = 0
+    optional_arguments = 1
+    final_argument_whitespace = True
+    has_content = True
+
+    def wrapped_content(self):
+        content = []
+
+        if Literal.max_line_length:
+            for line in self.content:
+                content += textwrap.wrap(line, Literal.max_line_length, subsequent_indent="      ")
+        else:
+            content = self.content
+
+        return u'\n'.join(content)
+
+    def run(self):
+        self.assert_has_content()
+        content = self.wrapped_content()
+        literal = nodes.literal_block(content, content)
+        return [literal]
+
+directives.register_directive('code', Literal)
+directives.register_directive('console', Literal)
+
+description = ('Generates (X)HTML documents from standalone reStructuredText '
+               'sources.  ' + default_description)
+
+publish_cmdline(writer_name='html', description=description)
+
+
+

doc/cluster.rst

@@ -9,11 +9,21 @@ Bro is not multithreaded, so once the limitations of a single processor core are
 Architecture
 ---------------
 
-There are 4 main components to a Bro cluster which are described in detail below.
+The figure below illustrates the main components of a Bro cluster.
+
+.. {{git_pull('bro:doc/deployment.png')}}
+
+.. image:: deployment.png
+
+Tap
+***
+This is a mechanism that splits the packet stream in order to make a copy
+available for inspection. Examples include the monitoring port on a switch and
+an optical splitter for fiber networks.
 
 Frontend 
 ********
-This is a discrete hardware device or on-host technique that will split your traffic into many streams or flows.  The Bro binary does not do this job.  There are numerous ways to accomplish this task, some of which are described below in the Frontend <link to that section> section.
+This is a discrete hardware device or on-host technique that will split your traffic into many streams or flows.  The Bro binary does not do this job.  There are numerous ways to accomplish this task, some of which are described below in `Frontend Options`_.
 
 Manager
 *******
@@ -50,7 +60,7 @@ Discrete hardware flow balancers
 cPacket
 ^^^^^^^
 
-If you are monitoring one or more 10G physical interfaces, the recommended solution is to use either a cFlow or cVu device from cPacket (is this too much? i don’t want to recommend gigamon with all of the problems we’ve had various places and i don’t know enough about VSS monitoring’s offerings to make a judgement.  i don’t want to recommend netoptics director hardware either, from what i understand it doesn’t fit the use case very well).  These devices will perform layer-2 load balancing by rewriting the destination ethernet MAC address to cause each packet associated with a particular flow to have the same destination MAC.  The packets can then be passed directly to a monitoring host or onward to a commodity switch to split the traffic out to multiple 1G interfaces for the workers.  This can ultimately greatly reduce costs since workers can use relatively inexpensive 1G interfaces.
+If you are monitoring one or more 10G physical interfaces, the recommended solution is to use either a cFlow or cVu device from cPacket because they are currently being used very successfully at a number of sites.  These devices will perform layer-2 load balancing by rewriting the destination ethernet MAC address to cause each packet associated with a particular flow to have the same destination MAC.  The packets can then be passed directly to a monitoring host where each worker has a BPF filter to limit it's visibility to only that stream of flows or onward to a commodity switch to split the traffic out to multiple 1G interfaces for the workers.  This can ultimately greatly reduce costs since workers can use relatively inexpensive 1G interfaces.
 
 OpenFlow Switches
 ^^^^^^^^^^^^^^^^^

doc/geoip.rst

@@ -0,0 +1,102 @@
+
+===========
+GeoLocation
+===========
+
+.. class:: opening
+
+    During the process of creating policy scripts the need may arise
+    to find the geographic location for an IP address. Bro has support
+    for the `GeoIP library <http://www.maxmind.com/app/c>`__ at the
+    policy script level beginning with release 1.3 to account for this
+    need.
+
+.. contents::
+
+GeoIPLite Database Installation
+------------------------------------
+
+A country database for GeoIPLite is included when you do the C API
+install, but for Bro, we are using the city database which includes
+cities and regions in addition to countries.
+
+`Download <http://www.maxmind.com/app/geolitecity>`__ the geolitecity
+binary database and follow the directions to install it.
+
+FreeBSD Quick Install
+---------------------
+
+.. console::
+
+    pkg_add -r GeoIP
+    wget http://geolite.maxmind.com/download/geoip/database/GeoLiteCity.dat.gz
+    gunzip GeoLiteCity.dat.gz
+    mv GeoLiteCity.dat /usr/local/share/GeoIP/GeoIPCity.dat
+    
+    # Set your environment correctly before running Bro's configure script
+    export CFLAGS=-I/usr/local/include
+    export LDFLAGS=-L/usr/local/lib
+
+
+CentOS Quick Install
+--------------------
+
+.. console::
+
+    yum install GeoIP-devel
+    
+    wget http://geolite.maxmind.com/download/geoip/database/GeoLiteCity.dat.gz
+    gunzip GeoLiteCity.dat.gz
+    mkdir -p /var/lib/GeoIP/
+    mv GeoLiteCity.dat /var/lib/GeoIP/GeoIPCity.dat
+    
+    # Set your environment correctly before running Bro's configure script
+    export CFLAGS=-I/usr/local/include
+    export LDFLAGS=-L/usr/local/lib
+
+
+Usage
+-----
+
+There is a single built in function that provides the GeoIP
+functionality:
+
+.. code:: bro
+
+    function lookup_location(a:addr): geo_location
+
+There is also the ``geo_location`` data structure that is returned
+from the ``lookup_location`` function:
+
+.. code:: bro
+
+    type geo_location: record {
+      country_code: string;
+      region: string;
+      city: string;
+      latitude: double;
+      longitude: double;
+    };
+
+
+Example
+-------
+
+To write a line in a log file for every ftp connection from hosts in
+Ohio, this is now very easy:
+
+.. code:: bro
+
+    global ftp_location_log: file = open_log_file("ftp-location");
+    
+    event ftp_reply(c: connection, code: count, msg: string, cont_resp: bool)
+    {
+      local client = c$id$orig_h;
+      local loc = lookup_location(client);
+      if (loc$region == "OH" && loc$country_code == "US")
+      {
+        print ftp_location_log, fmt("FTP Connection from:%s (%s,%s,%s)", client, loc$city, loc$region, loc$country_code); 
+      }
+    }
+
+

doc/index.rst

@@ -0,0 +1,50 @@
+
+
+Bro Documentation
+=================
+
+`Getting Started <{{git('bro:doc/quickstart.rst')}}>`_
+    A quick introduction into using Bro 2.x.
+
+`Bro 1.5 to 2.0 Upgrade Guide <{{git('bro:doc/upgrade.rst')}}>`_
+    Guidelines and notes about upgrading from Bro 1.5 to 2.x. Lots of
+    things have changed, so make sure to read this when upgrading.
+
+`BroControl <{{git('broctl:doc/broctl.rst')}}>`_
+    An interactive console for managing Bro installations.
+
+`Script Reference <{{autodoc_bro_scripts}}/index.html>`_ 
+    A complete reference of all policy scripts shipped with Bro.
+
+`FAQ <{{docroot}}/documentation/faq.html>`_
+    A list with frequently asked questions.
+
+`How to Report a Problem <{{docroot}}/reporting-problems.html>`_
+    Some advice for when you see Bro doing something you believe it
+    shouldn't.
+
+Frameworks
+----------
+
+Bro comes with a number of frameworks, some of which are described in
+more detail here:
+
+`Reporting <{{git('bro:doc/notice.rst')}}>`_
+    The notice/alarm framework.
+
+`Logging <{{git('bro:doc/logging.rst')}}>`_
+    Customizing and extensing Bro's logging.
+
+`Cluster <{{git('bro:doc/cluster.rst')}}>`_
+    Setting up a Bro Cluster when a single box can't handle the traffic anymore.
+        
+`Signatures <{{git('bro:doc/signatures.rst')}}>`_ 
+    Bro has support for traditional NIDS signatures as well.
+
+How-Tos
+-------
+
+We also collect more specific How-Tos on specific topics:
+
+`Using GeoIP in Bro scripts <{{git('bro:doc/geoip.rst')}}>`_
+    Installation and usage of the the GeoIP library.

doc/notice.rst

@@ -0,0 +1,5 @@
+
+The Notice Framework
+====================
+
+TODO.

doc/quickstart.rst

@@ -26,14 +26,9 @@ source code forms.
 Pre-Built Binary Release Packages
 ---------------------------------
 
-See the `downloads page <http://www.bro-ids.org/download/index.html>`_ for currently
+See the `downloads page <{{docroot}}/download/index.html>`_ for currently
 supported/targeted platforms.
 
-The primary install prefix for binary packages is ``/opt/bro``.
-
-Non-MacOS packages that include BroControl also put variable/runtime data
-(e.g. bro logs) in ``/var/opt/bro``.
-
 * RPM
 
 .. console::
@@ -51,9 +46,9 @@ Non-MacOS packages that include BroControl also put variable/runtime data
   Just open the ``Bro-all-*.dmg`` and then run the ``.pkg`` installer.
   Everything installed by the package will go into ``/opt/bro``.
 
-* FreeBSD
+The primary install prefix for binary packages is ``/opt/bro``.
-
+Non-MacOS packages that include BroControl also put variable/runtime
-  TODO: ports will eventually be available.
+data (e.g. Bro logs) in ``/var/opt/bro``.
 
 Building From Source
 --------------------
@@ -113,12 +108,12 @@ and sendmail for sending emails.
 
 * Ports-based FreeBSD
 
-  (libz, libmagic, and sendmail are typically already available)
-
 .. console::
 
    > sudo pkg_add -r GeoIP
 
+  libz, libmagic, and sendmail are typically already available.
+
 * Mac OS X
 
   Vanilla OS X installations don't ship with libmagic or libGeoIP, but
@@ -127,17 +122,17 @@ and sendmail for sending emails.
   against them.
 
 Additional steps may be needed to `get the right GeoIP database
-<http://www.bro-ids.org/documentation/geoip.html>`_.
+<geoip.html>`_.
 
 Compiling Bro Source Code
 ~~~~~~~~~~~~~~~~~~~~~~~~~
 
 Bro releases are bundled into source packages for convenience and
-available from the `downloads page <http://www.bro-ids.org/download/index.html>`_.
+available from the `downloads page <{{docroot}}/download/index.html>`_.
 
 The latest Bro development versions are obtainable through git repositories
-hosted at `{{cfg_git_url}} <{{cfg_git_url}}>`_.  See our `git development
+hosted at `git.bro-ids.org <http://git.bro-ids.org>`_.  See our `git development
-documentation <http://www.bro-ids.org/development/process.html>`_ for comprehensive
+documentation <{{docroot}}/development/process.html>`_ for comprehensive
 information on Bro's use of git revision control, but the short story for
 downloading the full source code experience for Bro via git is:
 
@@ -195,15 +190,15 @@ traffic-monitoring cluster.
 .. note:: Below, ``$PREFIX``, is used to reference the Bro installation
    root directory.
 
-Minimal Starting Config
+A Minimal Starting Configuration
------------------------
+--------------------------------
 
-The basic configuration changes to make for a minimal BroControl installation
+These are the basic configuration changes to make for a minimal BroControl installation
 that will manage a single Bro instance on the ``localhost``:
 
 1) In ``$PREFIX/etc/node.cfg``, set the right interface to monitor.
 2) In ``$PREFIX/etc/networks.cfg``, comment out the default settings and add
-   the networking that Bro will consider local to the monitored environment.
+   the networks that Bro will consider local to the monitored environment.
 3) In ``$PREFIX/etc/broctl.cfg``, change the ``MailTo`` email address to a
    desired recipient and the ``LogRotationInterval`` to a desired log
    archival frequency.
@@ -232,9 +227,12 @@ can view the details with the ``diag`` command.  If started successfully,
 the Bro instance will begin analyzing traffic according to a default
 policy and output the results in ``$PREFIX/logs``.
 
-.. note:: The `FAQ <http://www.bro-ids.org/documentation/faq.html>`_ entries about
+.. note:: The user starting BroControl needs permission to capture
-   capturing as an unprivileged user and checksum offloading are particularly
+   network traffic. If you are not root, you may need to grant further
-   relevant at this point.
+   privileges to the account you're using; see the `FAQ
+   <{{docroot}}/documentation/faq.html>`_. Also, if it
+   looks like Bro is not seeing any traffic, check out the FAQ entry
+   checksum offloading.
 
 You can leave it running for now, but to stop this Bro instance you would do:
 
@@ -242,26 +240,32 @@ You can leave it running for now, but to stop this Bro instance you would do:
 
    [BroControl] > stop
 
+We also recommend to insert the following entry into `crontab`:
+
+.. console::
+
+      0-59/5 * * * * $PREFIX/bin/broctl cron
+
+This will perform a number of regular housekeeping tasks, including
+verifying that the process is still running (and restarting if not in
+case of any abnormal termination).
+
 Browsing Log Files
 ------------------
 
-By default, logs are written in human-readable (ASCII) format and data
+By default, logs are written out in human-readable (ASCII) format and
-is organized into columns (tab-delimited). Logs that are part of the
+data is organized into columns (tab-delimited). Logs that are part of
-current rotation interval are accumulated in ``$PREFIX/logs/current/``
+the current rotation interval are accumulated in
-(if Bro is not running, then there will not be any log files in this
+``$PREFIX/logs/current/`` (if Bro is not running, the directory will
-directory).  For example, the ``http.log`` contains the results of
+be empty). For example, the ``http.log`` contains the results of Bro
-analysis performed by scripts in ``$PREFIX/share/bro/``\ **base**\
+HTTP protocol analysis. Here are the first few columns of
-``/protocols/http/`` or ``$PREFIX/share/bro/``\ **policy**\
+``http.log``::
-``/protocols/http/`` (both contain code that may contribute to what ends
-up in the log).
-
-Here's the first few columns of ``http.log``::
 
     # ts          uid          orig_h        orig_p  resp_h         resp_p
     1311627961.8  HSH4uV8KVJg  192.168.1.100 52303   192.150.187.43 80
 
 Logs that deal with analysis of a network protocol will often start like this:
-a timestamp, a connection identifier (UID), and a connection 4-tuple
+a timestamp, a unique connection identifier (UID), and a connection 4-tuple
 (originator host/port and responder host/port).  The UID can be used to
 identify all logged activity (possibly across multiple log files) associated
 with a given connection 4-tuple over its lifetime.
@@ -275,24 +279,24 @@ columns (shortened for brevity) show a request to the root of Bro website::
 
 Some logs are worth explicit mention:
 
-``weird.log`` contains unusual/exceptional activity that can indicate
+    ``weird.log``
+        Contains unusual/exceptional activity that can indicate
         malformed connections, traffic that doesn't conform to a particular
         protocol, malfunctioning/misconfigured hardware, or even an attacker
-attempting to avoid/confuse a sensor.  Without context, it's hard to judge 
+        attempting to avoid/confuse a sensor.  Without context, it's hard to
-whether this category of activity is interesting and so that is left up to
+        judge whether this category of activity is interesting and so that is
-the user to configure.
+        left up to the user to configure.
 
-``notice.log`` identifies specific activity that Bro recognizes as
+    ``notice.log``
-potentially interesting, odd, or bad.
+        Identifies specific activity that Bro recognizes as
+        potentially interesting, odd, or bad. In Bro-speak, such
+        activity is called a "notice".
 
-``alarm.log`` is just a filtered version of ``notice.log``, containing
-only notices for which the user has taught Bro to recognize as
-interesting/bad.
 
 By default, ``BroControl`` regularly takes all the logs from
-``$PREFIX/logs/current``, and archives/compresses them to a directory
+``$PREFIX/logs/current`` and archives/compresses them to a directory
-named by date, e.g. ``$PREFIX/logs/2011-10-06``.  The frequency
+named by date, e.g. ``$PREFIX/logs/2011-10-06``.  The frequency at
-at which this is done can be configured via the ``LogRotationInterval``
+which this is done can be configured via the ``LogRotationInterval``
 option in ``$PREFIX/etc/broctl.cfg``.
 
 Deployment Customization
@@ -329,23 +333,22 @@ let's do a quick intro to Bro scripting.
 Bro Scripts
 ~~~~~~~~~~~
 
-Bro ships with many pre-written scripts that are highly customizable to
+Bro ships with many pre-written scripts that are highly customizable
-support traffic analysis for your specific environment.  By default,
+to support traffic analysis for your specific environment.  By
-these will be installed into ``$PREFIX/share/bro`` and can be identified
+default, these will be installed into ``$PREFIX/share/bro`` and can be
-by the use of a ``.bro`` file name extension.  These files should
+identified by the use of a ``.bro`` file name extension.  These files
-**never** be edited directly as changes will be lost when upgrading to
+should **never** be edited directly as changes will be lost when
-newer versions of Bro.  The exception to this rule is that any ``.bro``
+upgrading to newer versions of Bro.  The exception to this rule is the
-file in ``$PREFIX/share/bro/site`` can be modified without fear of being
+directory ``$PREFIX/share/bro/site`` where local site-specific files
-clobbered later.  If desired, the ``site`` directory can also be used to
+can be put without fear of being clobbered later. The other main
-store new scripts.  The other main script directories under
+script directories under ``$PREFIX/share/bro`` are ``base`` and
-``$PREFIX/share/bro`` are ``base`` and ``policy``.  By default, Bro
+``policy``.  By default, Bro automatically loads all scripts under
-automatically loads all scripts under ``base`` (unless the ``-b``
+``base`` (unless the ``-b`` command line option is supplied), which
-command line option is supplied), which deal either with collecting
+deal either with collecting basic/useful state about network
-basic/useful state about network activities or providing
+activities or providing frameworks/utilities that extend Bro's
-frameworks/utilities that extend Bro's functionality without any
+functionality without any performance cost.  Scripts under the
-performance cost.  Scripts under the ``policy`` directory may be more
+``policy`` directory may be more situational or costly, and so users
-situational or costly, and so users must explicitly choose if
+must explicitly choose if they want to load them.
-they want to load them.
 
 The main entry point for the default analysis configuration of a standalone
 Bro instance managed by BroControl is the ``$PREFIX/share/bro/site/local.bro``
@@ -355,8 +358,9 @@ we have to figure out what to add.
 Redefining Script Option Variables
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
-Many simple customizations just require you to redefine (using the ``redef``
+Many simple customizations just require you to redefine a variable
-operator) a variable from a standard Bro script with your own value.
+from a standard Bro script with your own value, using Bro's ``redef``
+operator.
 
 The typical way a standard Bro script advertises tweak-able options to users
 is by defining variables with the ``&redef`` attribute and ``const`` qualifier. 
@@ -364,9 +368,10 @@ A redefineable constant might seem strange, but what that really means is that
 the variable's value may not change at run-time, but whose initial value can be
 modified via the ``redef`` operator at parse-time.
 
-So let's continue on our path to modify the behavior for the two SSL and SSH
+So let's continue on our path to modify the behavior for the two SSL
-notices.  Looking at
+and SSH notices.  Looking at
-`$PREFIX/share/bro/base/frameworks/notice/main.bro <{{ git('base.frameworks.notice.main.bro.txt', 'master:bro/scripts/base/frameworks/notice/main.bro') }}>`_,
+`$PREFIX/share/bro/base/frameworks/notice/main.bro
+<{{autodoc_bro_scripts}}/scripts/base/frameworks/notice/main.html>`_,
 we see that it advertises:
 
 .. code:: bro
@@ -441,15 +446,16 @@ that only takes the email action for SSH logins to a defined set of servers:
     };
 
 You'll just have to trust the syntax for now, but what we've done is first
-first declare our own variable to hold a set watched addresses,
+first declare our own variable to hold a set of watched addresses,
-``watched_servers``, then added a record to the policy that will generate
+``watched_servers``; then added a record to the policy that will generate
 an email on the condition that the predicate function evaluates to true, which
 is whenever the notice type is an SSH login and the responding host stored
 inside the ``Info`` record's connection field is in the set of watched servers.
 
-.. note:: record field member access is done with the '$' character instead
+.. note:: record field member access is done with the '$' character
-   of a '.' as might be expected in order to avoid ambiguity with the builtin
+   instead of a '.' as might be expected from other languages, in
-   address type's use of '.' in IPv4 dotted decimal representations.
+   order to avoid ambiguity with the builtin address type's use of '.'
+   in IPv4 dotted decimal representations.
 
 Remember, to finalize that configuration change perform the ``check``,
 ``install``, ``restart`` commands in that order inside the BroControl shell.
@@ -462,11 +468,13 @@ tweak the most basic options.  Here's some suggestions on what to explore next:
 
 * We only looked at how to change options declared in the notice framework,
   there's many more options to look at in other script packages.
+* Look at the scripts in ``$PREFIX/share/bro/policy`` for further ones
+  you may want to load.
 * Reading the code of scripts that ship with Bro is also a great way to gain
   understanding of the language and how you can start writing your own custom
   analysis.
-* Review the `FAQ <http://www.bro-ids.org/documentation/faq.html>`_.
+* Review the `FAQ <{{docroot}}/documentation/faq.html>`_.
-* Check out more `documentation <http://www.bro-ids.org/documentation/index.html>`_.
+* Check out more `documentation <{{docroot}}/documentation/index.html>`_.
 * Continue reading below for another mini-tutorial on using Bro as a standalone
   command-line utility.
 
@@ -492,10 +500,27 @@ that's available.
 
 Bro will output log files into the working directory.
 
-.. note:: The `FAQ <http://www.bro-ids.org/documentation/faq.html>`_ entries about
+.. note:: The `FAQ <{{docroot}}/documentation/faq.html>`_ entries about
    capturing as an unprivileged user and checksum offloading are particularly
    relevant at this point.
 
+To use the site-specific ``local.bro`` script, just add it to the
+command-line:
+
+.. console::
+
+   > bro -i en0 local
+
+This will cause Bro to print a warning about lacking the
+``Site::local_nets`` variable being configured. You can supply this
+information at the command line like this (supply your "local" subnets
+in place of the example subnets):
+
+.. console::
+
+  > bro -r mypackets.trace local "Site::local_nets += { 1.2.3.0/24, 5.6.7.0/24 }"
+
+
 Reading Packet Capture (pcap) Files
 -----------------------------------
 
@@ -519,21 +544,13 @@ and tell Bro to perform all the default analysis on the capture which primarily
 
 Bro will output log files into the working directory.
 
-If you are interested in more detection, you can load the ``local``
+If you are interested in more detection, you can again load the ``local``
 script that we include as a suggested configuration:
 
 .. console::
 
   > bro -r mypackets.trace local
 
-This will cause Bro to print a warning about lacking the
-``Site::local_nets`` variable being configured. You can supply this
-information at the command line like this (supply your "local" subnets
-in place of the example subnets):
-
-.. console::
-
-  > bro -r mypackets.trace local "Site::local_nets += { 1.2.3.0/24, 5.6.7.0/24 }"
 
 Telling Bro Which Scripts to Load
 ---------------------------------

doc/scripts/example.rst

@@ -0,0 +1,291 @@
+.. Automatically generated.  Do not edit.
+
+example.bro
+===========
+
+:download:`Original Source File <example.bro>`
+
+Overview
+--------
+This is an example script that demonstrates how to document.  Comments
+of the form ``##!`` are for the script summary.  The contents of
+these comments are transferred directly into the auto-generated
+`reStructuredText <http://docutils.sourceforge.net/rst.html>`_
+(reST) document's summary section.
+
+.. tip:: You can embed directives and roles within ``##``-stylized comments.
+
+:Imports: :doc:`policy/frameworks/software/vulnerable </scripts/policy/frameworks/software/vulnerable>`
+
+Summary
+~~~~~~~
+Options
+#######
+============================================================================ ======================================
+:bro:id:`Example::an_option`: :bro:type:`set` :bro:attr:`&redef`             add documentation for "an_option" here
+
+:bro:id:`Example::option_with_init`: :bro:type:`interval` :bro:attr:`&redef`
+============================================================================ ======================================
+
+State Variables
+###############
+=========================================================================== =======================================
+:bro:id:`Example::a_var`: :bro:type:`bool`                                  put some documentation for "a_var" here
+
+:bro:id:`Example::var_with_attr`: :bro:type:`count` :bro:attr:`&persistent`
+
+:bro:id:`Example::var_without_explicit_type`: :bro:type:`string`
+=========================================================================== =======================================
+
+Types
+#####
+====================================================== ==========================================================
+:bro:type:`Example::SimpleEnum`: :bro:type:`enum`      documentation for "SimpleEnum"
+                                                       goes here.
+
+:bro:type:`Example::SimpleRecord`: :bro:type:`record`  general documentation for a type "SimpleRecord"
+                                                       goes here.
+
+:bro:type:`Example::ComplexRecord`: :bro:type:`record` general documentation for a type "ComplexRecord" goes here
+
+:bro:type:`Example::Info`: :bro:type:`record`          An example record to be used with a logging stream.
+====================================================== ==========================================================
+
+Events
+######
+================================================= =============================================================
+:bro:id:`Example::an_event`: :bro:type:`event`    Summarize "an_event" here.
+
+:bro:id:`Example::log_example`: :bro:type:`event` This is a declaration of an example event that can be used in
+                                                  logging streams and is raised once for each log entry.
+================================================= =============================================================
+
+Functions
+#########
+=============================================== =======================================
+:bro:id:`Example::a_function`: :bro:type:`func` Summarize purpose of "a_function" here.
+=============================================== =======================================
+
+Redefinitions
+#############
+===================================================== ========================================
+:bro:type:`Log::ID`: :bro:type:`enum`
+
+:bro:type:`Example::SimpleEnum`: :bro:type:`enum`     document the "SimpleEnum" redef here
+
+:bro:type:`Example::SimpleRecord`: :bro:type:`record` document the record extension redef here
+===================================================== ========================================
+
+Namespaces
+~~~~~~~~~~
+.. bro:namespace:: Example
+
+Notices
+~~~~~~~
+:bro:type:`Notice::Type`
+
+   :Type: :bro:type:`enum`
+
+      .. bro:enum:: Example::Notice_One Notice::Type
+
+         any number of this type of comment
+         will document "Notice_One"
+
+      .. bro:enum:: Example::Notice_Two Notice::Type
+
+         any number of this type of comment
+         will document "Notice_Two"
+
+      .. bro:enum:: Example::Notice_Three Notice::Type
+
+      .. bro:enum:: Example::Notice_Four Notice::Type
+
+Public Interface
+----------------
+Options
+~~~~~~~
+.. bro:id:: Example::an_option
+
+   :Type: :bro:type:`set` [:bro:type:`addr`, :bro:type:`addr`, :bro:type:`string`]
+   :Attributes: :bro:attr:`&redef`
+   :Default: ``{}``
+
+   add documentation for "an_option" here
+
+.. bro:id:: Example::option_with_init
+
+   :Type: :bro:type:`interval`
+   :Attributes: :bro:attr:`&redef`
+   :Default: ``10.0 msecs``
+
+State Variables
+~~~~~~~~~~~~~~~
+.. bro:id:: Example::a_var
+
+   :Type: :bro:type:`bool`
+
+   put some documentation for "a_var" here
+
+.. bro:id:: Example::var_with_attr
+
+   :Type: :bro:type:`count`
+   :Attributes: :bro:attr:`&persistent`
+
+.. bro:id:: Example::var_without_explicit_type
+
+   :Type: :bro:type:`string`
+   :Default: ``"this works"``
+
+Types
+~~~~~
+.. bro:type:: Example::SimpleEnum
+
+   :Type: :bro:type:`enum`
+
+      .. bro:enum:: Example::ONE Example::SimpleEnum
+
+         and more specific info for "ONE"
+         can span multiple lines
+
+      .. bro:enum:: Example::TWO Example::SimpleEnum
+
+         or more info like this for "TWO"
+         can span multiple lines
+
+      .. bro:enum:: Example::THREE Example::SimpleEnum
+
+   documentation for "SimpleEnum"
+   goes here.
+
+.. bro:type:: Example::SimpleRecord
+
+   :Type: :bro:type:`record`
+
+      field1: :bro:type:`count`
+         counts something
+
+      field2: :bro:type:`bool`
+         toggles something
+
+   general documentation for a type "SimpleRecord"
+   goes here.
+
+.. bro:type:: Example::ComplexRecord
+
+   :Type: :bro:type:`record`
+
+      field1: :bro:type:`count`
+         counts something
+
+      field2: :bro:type:`bool`
+         toggles something
+
+      field3: :bro:type:`Example::SimpleRecord`
+
+      msg: :bro:type:`string` :bro:attr:`&default` = ``"blah"`` :bro:attr:`&optional`
+         attributes are self-documenting
+
+   general documentation for a type "ComplexRecord" goes here
+
+.. bro:type:: Example::Info
+
+   :Type: :bro:type:`record`
+
+      ts: :bro:type:`time` :bro:attr:`&log`
+
+      uid: :bro:type:`string` :bro:attr:`&log`
+
+      status: :bro:type:`count` :bro:attr:`&log` :bro:attr:`&optional`
+
+   An example record to be used with a logging stream.
+
+Events
+~~~~~~
+.. bro:id:: Example::an_event
+
+   :Type: :bro:type:`event` (name: :bro:type:`string`)
+
+   Summarize "an_event" here.
+   Give more details about "an_event" here.
+   
+   :param name: describe the argument here
+
+.. bro:id:: Example::log_example
+
+   :Type: :bro:type:`event` (rec: :bro:type:`Example::Info`)
+
+   This is a declaration of an example event that can be used in
+   logging streams and is raised once for each log entry.
+
+Functions
+~~~~~~~~~
+.. bro:id:: Example::a_function
+
+   :Type: :bro:type:`function` (tag: :bro:type:`string`, msg: :bro:type:`string`) : :bro:type:`string`
+
+   Summarize purpose of "a_function" here.
+   Give more details about "a_function" here.
+   Separating the documentation of the params/return values with
+   empty comments is optional, but improves readability of script.
+   
+   
+   :param tag: function arguments can be described
+        like this
+   
+   :param msg: another param
+   
+   
+   :returns: describe the return type here
+
+Redefinitions
+~~~~~~~~~~~~~
+:bro:type:`Log::ID`
+
+   :Type: :bro:type:`enum`
+
+      .. bro:enum:: Example::LOG Log::ID
+
+:bro:type:`Example::SimpleEnum`
+
+   :Type: :bro:type:`enum`
+
+      .. bro:enum:: Example::FOUR Example::SimpleEnum
+
+         and some documentation for "FOUR"
+
+      .. bro:enum:: Example::FIVE Example::SimpleEnum
+
+         also "FIVE" for good measure
+
+   document the "SimpleEnum" redef here
+
+:bro:type:`Example::SimpleRecord`
+
+   :Type: :bro:type:`record`
+
+      field_ext: :bro:type:`string` :bro:attr:`&optional`
+         document the extending field here
+         (or here)
+
+   document the record extension redef here
+
+Port Analysis
+-------------
+:ref:`More Information <common_port_analysis_doc>`
+
+SSL::
+
+    [ports={
+        443/tcp,
+        562/tcp
+    }]
+
+Packet Filter
+-------------
+:ref:`More Information <common_packet_filter_doc>`
+
+Filters added::
+
+    [ssl] = tcp port 443,
+    [nntps] = tcp port 562
+

doc/upgrade.rst

@@ -5,22 +5,12 @@ Upgrading From Bro 1.5 to 2.0
 
 .. class:: opening
 
-   This guide details differences between Bro version 1.5 and 2.0 that
+   This guide details differences between Bro versions 1.5 and 2.0
-   may be important for users to know as they work on updating their
+   that may be important for users to know as they work on updating
-   Bro deployment/configuration to the later version.
+   their Bro deployment/configuration to the later version.
 
 .. contents::
 
-New Development Process
-=======================
-
-Bro development has moved from using SVN to Git for revision control.
-Users that like to use the latest Bro developments by checking it out
-from the source repositories should see the `development process
-<http://www.bro-ids.org/development/process.html>`_
-
-Bro now uses `CMake <http://www.cmake.org>`_ for its build system so
-that is a new required dependency when building from source.
 
 New Script Organization/Hierarchy
 =================================
@@ -28,22 +18,29 @@ New Script Organization/Hierarchy
 In versions before 2.0, Bro scripts were all maintained in a flat
 directory called ``policy/`` in the source tree.  This directory is now
 renamed to ``scripts/`` and contains major subdirectories ``base/``,
-``policy/``, and ``site/``, each of which may also be subdivided further
+``policy/``, and ``site/``, each of which may also be subdivided
+further.
 
 The contents of the new ``scripts/`` directory, like the old/flat
 ``policy/`` still gets installed under under the ``share/bro``
 subdirectory of the installation prefix path just like previous
 versions.  For example, if Bro was compiled like ``./configure
 --prefix=/usr/local/bro && make && make install``, then the script
-hierarchy can be found in ``/usr/local/bro/share/bro``.  And main
+hierarchy can be found in ``/usr/local/bro/share/bro``.
+
+THe main
 subdirectories of that hierarchy are as follows:
 
 - ``base/`` contains all scripts that are loaded by Bro by default
-  (unless the ``-b`` command line option is used to run Bro in a minimal
+  (unless the ``-b`` command line option is used to run Bro in a
-  configuration).  Scripts under this directory generally either provide
+  minimal configuration). Note that is a major conceptual change:
-  extra Bro scripting-layer functionality that has no performance cost,
+  rather than not loading anything by default, Bro now uses an
-  configure a default/recommended mode of operation, or accumulate/log
+  extensive set of default scripts out of the box.
-  useful state/protocol information for monitored traffic.
+
+  The scripts under this directory generally either accumulate/log
+  useful state/protocol information for monitored traffic, configure a
+  default/recommended mode of operation, or provide extra Bro
+  scripting-layer functionality that has no significant performance cost.
 
 - ``policy/`` contains all scripts that a user will need to explicitly
   tell Bro to load.  These are scripts that implement
@@ -51,24 +48,24 @@ subdirectories of that hierarchy are as follows:
   more significant performance costs.
 
 - ``site/`` remains a directory that can be used to store locally 
-  developed scripts, but now contains some extra scripts that may
+  developed scripts, but now contains some extra scripts that contain
-  contain some recommended default configurations.  E.g. ``local.bro``
+  some recommended default configurations.  E.g. ``local.bro`` loads
-  is loads extra scripts from ``policy/`` and does extra tuning.
+  extra scripts from ``policy/`` and does extra tuning. These files
-  These files can also be customized in place without being overwritten
+  can also be customized in place without being overwritten by
-  by upgrades/reinstalls, unlike scripts in other directories.
+  upgrades/reinstalls, unlike scripts in other directories.
 
-Now, with version 2.0, the default/builtin ``BROPATH`` automatically
+Now, with version 2.0, the default ``BROPATH`` automatically will
-will search for scripts in only ``policy/``, ``site/`` and their parent
+search for scripts in ``policy/``, ``site/`` and their parent
-directory, but **not** ``base/``.  Generally, everything under ``base/``
+directory, but **not** ``base/``.  Generally, everything under
-is loaded automatically, but for users of the ``-b``, option, scripts
+``base/`` is loaded automatically, but for users of the ``-b`` option
-it's important to know that loading a script in that directory requires
+(which prevents this), it's important to know that loading a
-the extra ``base/`` path qualification.  For example, the following two
+script in that directory requires the extra ``base/`` path
-scripts:
+qualification.  For example, the following two scripts:
 
 * ``$PREFIX/share/bro/base/protocols/ssl/main.bro``
 * ``$PREFIX/share/bro/policy/protocols/ssl/validate-certs.bro``
 
-Are referenced from another Bro script like:
+are referenced from another Bro script like:
 
 .. code:: bro
 
@@ -78,6 +75,34 @@ Are referenced from another Bro script like:
 Notice how ``policy/`` can be omitted as a convenience in the second
 case.
 
+Logging Framework
+-----------------
+
+- The logs generated by scripts that ship with Bro are entirely redone
+  to use a standardized format via the new logging framework and
+  generally the content has changed towards making the logs even more
+  useful.
+
+  * A particular format change that may be useful to note is that the
+    ``conn.log`` ``service`` field is derived from DPD instead of
+    well-known ports (while that was already possible in 1.5, it was
+    not the default).
+
+- A common pattern found in the new scripts is to store logging stream
+  records for protocols inside the ``connection`` records so that
+  state can be collected until enough is seen to log a coherent unit
+  of information regarding the activity of that connection.  This
+  state is now frequently seen/accessible in event handlers, for
+  example, like ``c$<protocol>`` where ``<protocol>`` is replaced by
+  the name of the protocol.  This field is added to the ``connection``
+  record by ``redef``'ing it in a
+  ``base/protocols/<protocol>/main.bro`` script.
+
+- The new logging framework also makes it possible to extend and
+  filter logs. See `the logging framework
+  <{{git('bro:doc/logging.rst')}}>`_ more information on usage.
+
+
 Scripting-Layer API Changes
 ===========================
 
@@ -94,7 +119,7 @@ Scripting-Layer API Changes
 
 - The ``make_addr`` BIF now returns a ``subnet`` versus an ``addr``
 
-- The ``net`` type has been removed
+- The ``net`` type has been removed.
 
 
 New Default Settings
@@ -117,33 +142,9 @@ Variable Naming
   ``site.bro`` exports the ``local_nets`` identifier (among other
   things) into the ``Site`` module.
 
-- Identifiers may have been renamed to conform to `scripting
+- Identifiers may have been renamed to conform to new `scripting
   conventions
-  <http://www.bro-ids.org/development/script-conventions.html>`_
+  <{{docroot}}/development/script-conventions.html>`_
-
-Logging Framework
------------------
-
-- The logs generated by scripts that ship with Bro are entirely redone
-  to use a standardized format via the new logging framework and
-  generally the content has changed towards making the logs even more
-  useful.
-
-  * a particular format change that may be useful to note is that the
-    ``conn.log`` ``service`` field is derived from DPD instead of
-    well-known ports
-
-- A common pattern found in the new scripts is to store logging
-  stream records for protocols inside ``connection`` records so that
-  state can be collected until enough is seen to log a coherent unit
-  of information regarding the activity of that connection.  This state
-  is now frequently seen/accessible in event handlers, for example, like
-  ``c$<protocol>`` where ``<protocol>`` is replaced by the name of the
-  protocol.  This field is added to the ``connection`` record by
-  ``redef``'ing it in a ``base/protocols/<protocol>/main.bro`` script.
-
-- The new logging framework also makes it possible to extend and
-  filter logs. See `<logging.rst>`_.
 
 Communication Framework
 -----------------------
@@ -161,10 +162,19 @@ Communication Framework
 Notice Framework
 ----------------
 
-The way users interact with "notices" has changed significantly in order
+The way users interact with "notices" has changed significantly in
-to make it easier to define a site policy and more extensible for adding
+order to make it easier to define a site policy and more extensible
-customized actions.
+for adding customized actions. See the `the notice framework
+<{{git('bro:doc/notice.rst')}}>`_.
 
-TODO: we need new notice documentation with examples to link from
+
-here. The `old notice documentation <notices.html>`_ can be used as a
+New Development Infrastructure
-starting point.
+==============================
+
+Bro development has moved from using SVN to Git for revision control.
+Users that like to use the latest Bro developments by checking it out
+from the source repositories should see the `development process
+<{{docroot}}/development/process.html>`_
+
+Bro now uses `CMake <http://www.cmake.org>`_ for its build system so
+that is a new required dependency when building from source.

pkg/make-src-packages

@@ -12,12 +12,16 @@ cp -R ${SOURCE} ${TMP}/Bro-${BRO_V}
 ( cd ${TMP} && find . -name .git\* | xargs rm -rf )
 ( cd ${TMP} && find . -name \*.swp | xargs rm -rf )
 ( cd ${TMP} && find . -type d -name build | xargs rm -rf )
-( cd ${TMP} && tar -czf ${BUILD}/Bro-all-${BRO_V}.tar.gz Bro-${BRO_V} )
+( cd ${TMP} && tar -czf ${BUILD}/Bro-all-${BRO_V}.tar.gz Bro-${BRO_V} && echo "Package: ${BUILD}/Bro-all-${BRO_V}.tar.gz" )
+
+# TODO: Remove? -Robin
 ( cd ${TMP}/Bro-${BRO_V}/aux && mv broccoli Broccoli-${BROCCOLI_V} && \
         tar -czf ${BUILD}/Broccoli-${BROCCOLI_V}.tar.gz Broccoli-${BROCCOLI_V} )
 ( cd ${TMP}/Bro-${BRO_V}/aux && mv broctl Broctl-${BROCTL_V} && \
         tar -czf ${BUILD}/Broctl-${BROCTL_V}.tar.gz Broctl-${BROCTL_V} )
 ( cd ${TMP}/Bro-${BRO_V}/aux && rm -rf Broctl* Broccoli* )
-( cd ${TMP} && tar -czf ${BUILD}/Bro-${BRO_V}.tar.gz Bro-${BRO_V} )
+
+( cd ${TMP} && tar -czf ${BUILD}/Bro-${BRO_V}.tar.gz Bro-${BRO_V} && echo "Package: ${BUILD}/Bro-${BRO_V}.tar.gz" )
+
 rm -rf ${TMP}
 echo "Distribution source tarballs have been compiled in ${BUILD}"