Merge remote-tracking branch 'origin/rtd-test'

* origin/rtd-test:
  Disable RTD pdf format due to exceeded capacity
  Add RTD pdf format
  Add RTD yaml config file
  Remove some Bro usages in main TOC entries
  Remove "contents" Sphinx directive usages
  Add a `make livehtml` target
  Use sourcecode Sphinx directive more widely
  Use Sphinx RTD theme for user manual
  Replace some code-block Sphinx directives
  Remove unused Sphinx extensions
  Remove broxygen Sphinx integration
  Remove Sphinx btest integrations and tests
  Fix a Sphinx deprecation

These are all changes required to build documentation from a static
Sphinx tree (e.g. on Read the Docs)

doc/scripts/base/frameworks/analyzer/__load__.bro.rst

@@ -0,0 +1,14 @@
+:tocdepth: 3
+
+base/frameworks/analyzer/__load__.bro
+=====================================
+
+
+:Imports: :doc:`base/frameworks/analyzer/main.bro </scripts/base/frameworks/analyzer/main.bro>`
+
+Summary
+~~~~~~~
+
+Detailed Interface
+~~~~~~~~~~~~~~~~~~
+

doc/scripts/base/frameworks/analyzer/index.rst

@@ -0,0 +1,26 @@
+:orphan:
+
+Package: base/frameworks/analyzer
+=================================
+
+The analyzer framework allows to dynamically enable or disable Bro's
+protocol analyzers, as well as to manage the well-known ports which
+automatically activate a particular analyzer for new connections.
+
+:doc:`/scripts/base/frameworks/analyzer/__load__.bro`
+
+
+:doc:`/scripts/base/frameworks/analyzer/main.bro`
+
+   Framework for managing Bro's protocol analyzers.
+   
+   The analyzer framework allows to dynamically enable or disable analyzers, as
+   well as to manage the well-known ports which automatically activate a
+   particular analyzer for new connections.
+   
+   Protocol analyzers are identified by unique tags of type
+   :bro:type:`Analyzer::Tag`, such as :bro:enum:`Analyzer::ANALYZER_HTTP`.
+   These tags are defined internally by
+   the analyzers themselves, and documented in their analyzer-specific
+   description along with the events that they generate.
+

doc/scripts/base/frameworks/analyzer/main.bro.rst

@@ -0,0 +1,246 @@
+:tocdepth: 3
+
+base/frameworks/analyzer/main.bro
+=================================
+.. bro:namespace:: Analyzer
+
+Framework for managing Bro's protocol analyzers.
+
+The analyzer framework allows to dynamically enable or disable analyzers, as
+well as to manage the well-known ports which automatically activate a
+particular analyzer for new connections.
+
+Protocol analyzers are identified by unique tags of type
+:bro:type:`Analyzer::Tag`, such as :bro:enum:`Analyzer::ANALYZER_HTTP`.
+These tags are defined internally by
+the analyzers themselves, and documented in their analyzer-specific
+description along with the events that they generate.
+
+:Namespace: Analyzer
+:Imports: :doc:`base/bif/analyzer.bif.bro </scripts/base/bif/analyzer.bif.bro>`, :doc:`base/frameworks/packet-filter/utils.bro </scripts/base/frameworks/packet-filter/utils.bro>`
+
+Summary
+~~~~~~~
+State Variables
+###############
+========================================================================== ===================================================================
+:bro:id:`Analyzer::disable_all`: :bro:type:`bool` :bro:attr:`&redef`       If true, all available analyzers are initially disabled at startup.
+:bro:id:`Analyzer::disabled_analyzers`: :bro:type:`set` :bro:attr:`&redef` A set of analyzers to disable by default at startup.
+========================================================================== ===================================================================
+
+Functions
+#########
+============================================================== =======================================================================
+:bro:id:`Analyzer::all_registered_ports`: :bro:type:`function` Returns a table of all ports-to-analyzer mappings currently registered.
+:bro:id:`Analyzer::analyzer_to_bpf`: :bro:type:`function`      Automatically creates a BPF filter for the specified protocol based
+                                                               on the data supplied for the protocol through the
+                                                               :bro:see:`Analyzer::register_for_ports` function.
+:bro:id:`Analyzer::disable_analyzer`: :bro:type:`function`     Disables an analyzer.
+:bro:id:`Analyzer::enable_analyzer`: :bro:type:`function`      Enables an analyzer.
+:bro:id:`Analyzer::get_bpf`: :bro:type:`function`              Create a BPF filter which matches all of the ports defined
+                                                               by the various protocol analysis scripts as "registered ports"
+                                                               for the protocol.
+:bro:id:`Analyzer::get_tag`: :bro:type:`function`              Translates an analyzer's name to a tag enum value.
+:bro:id:`Analyzer::name`: :bro:type:`function`                 Translates an analyzer type to a string with the analyzer's name.
+:bro:id:`Analyzer::register_for_port`: :bro:type:`function`    Registers an individual well-known port for an analyzer.
+:bro:id:`Analyzer::register_for_ports`: :bro:type:`function`   Registers a set of well-known ports for an analyzer.
+:bro:id:`Analyzer::registered_ports`: :bro:type:`function`     Returns a set of all well-known ports currently registered for a
+                                                               specific analyzer.
+:bro:id:`Analyzer::schedule_analyzer`: :bro:type:`function`    Schedules an analyzer for a future connection originating from a
+                                                               given IP address and port.
+============================================================== =======================================================================
+
+
+Detailed Interface
+~~~~~~~~~~~~~~~~~~
+State Variables
+###############
+.. bro:id:: Analyzer::disable_all
+
+   :Type: :bro:type:`bool`
+   :Attributes: :bro:attr:`&redef`
+   :Default: ``F``
+
+   If true, all available analyzers are initially disabled at startup.
+   One can then selectively enable them with
+   :bro:id:`Analyzer::enable_analyzer`.
+
+.. bro:id:: Analyzer::disabled_analyzers
+
+   :Type: :bro:type:`set` [:bro:type:`Analyzer::Tag`]
+   :Attributes: :bro:attr:`&redef`
+   :Default:
+
+   ::
+
+      {
+         Analyzer::ANALYZER_BACKDOOR,
+         Analyzer::ANALYZER_INTERCONN,
+         Analyzer::ANALYZER_TCPSTATS,
+         Analyzer::ANALYZER_STEPPINGSTONE
+      }
+
+   A set of analyzers to disable by default at startup. The default set
+   contains legacy analyzers that are no longer supported.
+
+Functions
+#########
+.. bro:id:: Analyzer::all_registered_ports
+
+   :Type: :bro:type:`function` () : :bro:type:`table` [:bro:type:`Analyzer::Tag`] of :bro:type:`set` [:bro:type:`port`]
+
+   Returns a table of all ports-to-analyzer mappings currently registered.
+   
+
+   :returns: A table mapping each analyzer to the set of ports
+            registered for it.
+
+.. bro:id:: Analyzer::analyzer_to_bpf
+
+   :Type: :bro:type:`function` (tag: :bro:type:`Analyzer::Tag`) : :bro:type:`string`
+
+   Automatically creates a BPF filter for the specified protocol based
+   on the data supplied for the protocol through the
+   :bro:see:`Analyzer::register_for_ports` function.
+   
+
+   :tag: The analyzer tag.
+   
+
+   :returns: BPF filter string.
+
+.. bro:id:: Analyzer::disable_analyzer
+
+   :Type: :bro:type:`function` (tag: :bro:type:`Analyzer::Tag`) : :bro:type:`bool`
+
+   Disables an analyzer. Once disabled, the analyzer will not be used
+   further for analysis of future connections.
+   
+
+   :tag: The tag of the analyzer to disable.
+   
+
+   :returns: True if the analyzer was successfully disabled.
+
+.. bro:id:: Analyzer::enable_analyzer
+
+   :Type: :bro:type:`function` (tag: :bro:type:`Analyzer::Tag`) : :bro:type:`bool`
+
+   Enables an analyzer. Once enabled, the analyzer may be used for analysis
+   of future connections as decided by Bro's dynamic protocol detection.
+   
+
+   :tag: The tag of the analyzer to enable.
+   
+
+   :returns: True if the analyzer was successfully enabled.
+
+.. bro:id:: Analyzer::get_bpf
+
+   :Type: :bro:type:`function` () : :bro:type:`string`
+
+   Create a BPF filter which matches all of the ports defined
+   by the various protocol analysis scripts as "registered ports"
+   for the protocol.
+
+.. bro:id:: Analyzer::get_tag
+
+   :Type: :bro:type:`function` (name: :bro:type:`string`) : :bro:type:`Analyzer::Tag`
+
+   Translates an analyzer's name to a tag enum value.
+   
+
+   :name: The analyzer name.
+   
+
+   :returns: The analyzer tag corresponding to the name.
+
+.. bro:id:: Analyzer::name
+
+   :Type: :bro:type:`function` (atype: :bro:type:`Analyzer::Tag`) : :bro:type:`string`
+
+   Translates an analyzer type to a string with the analyzer's name.
+   
+
+   :tag: The analyzer tag.
+   
+
+   :returns: The analyzer name corresponding to the tag.
+
+.. bro:id:: Analyzer::register_for_port
+
+   :Type: :bro:type:`function` (tag: :bro:type:`Analyzer::Tag`, p: :bro:type:`port`) : :bro:type:`bool`
+
+   Registers an individual well-known port for an analyzer. If a future
+   connection on this port is seen, the analyzer will be automatically
+   assigned to parsing it. The function *adds* to all ports already
+   registered, it doesn't replace them.
+   
+
+   :tag: The tag of the analyzer.
+   
+
+   :p: The well-known port to associate with the analyzer.
+   
+
+   :returns: True if the port was successfully registered.
+
+.. bro:id:: Analyzer::register_for_ports
+
+   :Type: :bro:type:`function` (tag: :bro:type:`Analyzer::Tag`, ports: :bro:type:`set` [:bro:type:`port`]) : :bro:type:`bool`
+
+   Registers a set of well-known ports for an analyzer. If a future
+   connection on one of these ports is seen, the analyzer will be
+   automatically assigned to parsing it. The function *adds* to all ports
+   already registered, it doesn't replace them.
+   
+
+   :tag: The tag of the analyzer.
+   
+
+   :ports: The set of well-known ports to associate with the analyzer.
+   
+
+   :returns: True if the ports were successfully registered.
+
+.. bro:id:: Analyzer::registered_ports
+
+   :Type: :bro:type:`function` (tag: :bro:type:`Analyzer::Tag`) : :bro:type:`set` [:bro:type:`port`]
+
+   Returns a set of all well-known ports currently registered for a
+   specific analyzer.
+   
+
+   :tag: The tag of the analyzer.
+   
+
+   :returns: The set of ports.
+
+.. bro:id:: Analyzer::schedule_analyzer
+
+   :Type: :bro:type:`function` (orig: :bro:type:`addr`, resp: :bro:type:`addr`, resp_p: :bro:type:`port`, analyzer: :bro:type:`Analyzer::Tag`, tout: :bro:type:`interval`) : :bro:type:`bool`
+
+   Schedules an analyzer for a future connection originating from a
+   given IP address and port.
+   
+
+   :orig: The IP address originating a connection in the future.
+         0.0.0.0 can be used as a wildcard to match any originator address.
+   
+
+   :resp: The IP address responding to a connection from *orig*.
+   
+
+   :resp_p: The destination port at *resp*.
+   
+
+   :analyzer: The analyzer ID.
+   
+
+   :tout: A timeout interval after which the scheduling request will be
+         discarded if the connection has not yet been seen.
+   
+
+   :returns: True if successful.
+
+

doc/scripts/base/frameworks/broker/__load__.bro.rst

@@ -0,0 +1,14 @@
+:tocdepth: 3
+
+base/frameworks/broker/__load__.bro
+===================================
+
+
+:Imports: :doc:`base/frameworks/broker/log.bro </scripts/base/frameworks/broker/log.bro>`, :doc:`base/frameworks/broker/main.bro </scripts/base/frameworks/broker/main.bro>`, :doc:`base/frameworks/broker/store.bro </scripts/base/frameworks/broker/store.bro>`
+
+Summary
+~~~~~~~
+
+Detailed Interface
+~~~~~~~~~~~~~~~~~~
+

doc/scripts/base/frameworks/broker/index.rst

@@ -0,0 +1,22 @@
+:orphan:
+
+Package: base/frameworks/broker
+===============================
+
+The Broker communication framework facilitates connecting to remote Bro
+instances to share state and transfer events.
+
+:doc:`/scripts/base/frameworks/broker/__load__.bro`
+
+
+:doc:`/scripts/base/frameworks/broker/main.bro`
+
+   The Broker-based communication API and its various options.
+
+:doc:`/scripts/base/frameworks/broker/store.bro`
+
+   The Broker-based data store API and its various options.
+
+:doc:`/scripts/base/frameworks/broker/log.bro`
+
+

doc/scripts/base/frameworks/broker/log.bro.rst

@@ -0,0 +1,67 @@
+:tocdepth: 3
+
+base/frameworks/broker/log.bro
+==============================
+.. bro:namespace:: Broker
+
+
+:Namespace: Broker
+:Imports: :doc:`base/frameworks/broker/main.bro </scripts/base/frameworks/broker/main.bro>`
+
+Summary
+~~~~~~~
+Types
+#####
+============================================ =============================================================
+:bro:type:`Broker::Info`: :bro:type:`record` A record type containing the column fields of the Broker log.
+:bro:type:`Broker::Type`: :bro:type:`enum`   The type of a Broker activity being logged.
+============================================ =============================================================
+
+Redefinitions
+#############
+===================================== =====================================
+:bro:type:`Log::ID`: :bro:type:`enum` The Broker logging stream identifier.
+===================================== =====================================
+
+
+Detailed Interface
+~~~~~~~~~~~~~~~~~~
+Types
+#####
+.. bro:type:: Broker::Info
+
+   :Type: :bro:type:`record`
+
+      ts: :bro:type:`time` :bro:attr:`&log`
+         The network time at which a Broker event occurred.
+
+      ty: :bro:type:`Broker::Type` :bro:attr:`&log`
+         The type of the Broker event.
+
+      ev: :bro:type:`string` :bro:attr:`&log`
+         The event being logged.
+
+      peer: :bro:type:`Broker::NetworkInfo` :bro:attr:`&log` :bro:attr:`&optional`
+         The peer (if any) with which a Broker event is
+         concerned.
+
+      message: :bro:type:`string` :bro:attr:`&log` :bro:attr:`&optional`
+         An optional message describing the Broker event in more detail
+
+   A record type containing the column fields of the Broker log.
+
+.. bro:type:: Broker::Type
+
+   :Type: :bro:type:`enum`
+
+      .. bro:enum:: Broker::STATUS Broker::Type
+
+         An informational status update.
+
+      .. bro:enum:: Broker::ERROR Broker::Type
+
+         An error situation.
+
+   The type of a Broker activity being logged.
+
+

doc/scripts/base/frameworks/broker/main.bro.rst

@@ -0,0 +1,696 @@
+:tocdepth: 3
+
+base/frameworks/broker/main.bro
+===============================
+.. bro:namespace:: Broker
+
+The Broker-based communication API and its various options.
+
+:Namespace: Broker
+:Imports: :doc:`base/bif/comm.bif.bro </scripts/base/bif/comm.bif.bro>`, :doc:`base/bif/messaging.bif.bro </scripts/base/bif/messaging.bif.bro>`
+
+Summary
+~~~~~~~
+Redefinable Options
+###################
+================================================================================= ======================================================================
+:bro:id:`Broker::aggressive_interval`: :bro:type:`count` :bro:attr:`&redef`       Frequency of work-stealing polling attempts for Broker/CAF threads
+                                                                                  in "aggressive" mode.
+:bro:id:`Broker::aggressive_polls`: :bro:type:`count` :bro:attr:`&redef`          Number of work-stealing polling attempts for Broker/CAF threads
+                                                                                  in "aggressive" mode.
+:bro:id:`Broker::congestion_queue_size`: :bro:type:`count` :bro:attr:`&redef`     The number of buffered messages at the Broker/CAF layer after which
+                                                                                  a subscriber considers themselves congested (i.e.
+:bro:id:`Broker::default_connect_retry`: :bro:type:`interval` :bro:attr:`&redef`  Default interval to retry connecting to a peer if it cannot be made to
+                                                                                  work initially, or if it ever becomes disconnected.
+:bro:id:`Broker::default_listen_address`: :bro:type:`string` :bro:attr:`&redef`   Default address on which to listen.
+:bro:id:`Broker::default_listen_retry`: :bro:type:`interval` :bro:attr:`&redef`   Default interval to retry listening on a port if it's currently in
+                                                                                  use already.
+:bro:id:`Broker::default_log_topic_prefix`: :bro:type:`string` :bro:attr:`&redef` The default topic prefix where logs will be published.
+:bro:id:`Broker::default_port`: :bro:type:`port` :bro:attr:`&redef`               Default port for Broker communication.
+:bro:id:`Broker::disable_ssl`: :bro:type:`bool` :bro:attr:`&redef`                If true, do not use SSL for network connections.
+:bro:id:`Broker::forward_messages`: :bro:type:`bool` :bro:attr:`&redef`           Forward all received messages to subscribing peers.
+:bro:id:`Broker::max_threads`: :bro:type:`count` :bro:attr:`&redef`               Max number of threads to use for Broker/CAF functionality.
+:bro:id:`Broker::moderate_interval`: :bro:type:`count` :bro:attr:`&redef`         Frequency of work-stealing polling attempts for Broker/CAF threads
+                                                                                  in "moderate" mode.
+:bro:id:`Broker::moderate_polls`: :bro:type:`count` :bro:attr:`&redef`            Number of work-stealing polling attempts for Broker/CAF threads
+                                                                                  in "moderate" mode.
+:bro:id:`Broker::moderate_sleep`: :bro:type:`interval` :bro:attr:`&redef`         Interval of time for under-utilized Broker/CAF threads to sleep
+                                                                                  when in "moderate" mode.
+:bro:id:`Broker::relaxed_interval`: :bro:type:`count` :bro:attr:`&redef`          Frequency of work-stealing polling attempts for Broker/CAF threads
+                                                                                  in "relaxed" mode.
+:bro:id:`Broker::relaxed_sleep`: :bro:type:`interval` :bro:attr:`&redef`          Interval of time for under-utilized Broker/CAF threads to sleep
+                                                                                  when in "relaxed" mode.
+:bro:id:`Broker::ssl_cafile`: :bro:type:`string` :bro:attr:`&redef`               Path to a file containing concatenated trusted certificates 
+                                                                                  in PEM format.
+:bro:id:`Broker::ssl_capath`: :bro:type:`string` :bro:attr:`&redef`               Path to an OpenSSL-style directory of trusted certificates.
+:bro:id:`Broker::ssl_certificate`: :bro:type:`string` :bro:attr:`&redef`          Path to a file containing a X.509 certificate for this
+                                                                                  node in PEM format.
+:bro:id:`Broker::ssl_keyfile`: :bro:type:`string` :bro:attr:`&redef`              Path to the file containing the private key for this node's
+                                                                                  certificate.
+:bro:id:`Broker::ssl_passphrase`: :bro:type:`string` :bro:attr:`&redef`           Passphrase to decrypt the private key specified by
+                                                                                  :bro:see:`Broker::ssl_keyfile`.
+================================================================================= ======================================================================
+
+Types
+#####
+==================================================== ====================================================================
+:bro:type:`Broker::Data`: :bro:type:`record`         Opaque communication data.
+:bro:type:`Broker::DataVector`: :bro:type:`vector`   Opaque communication data sequence.
+:bro:type:`Broker::EndpointInfo`: :bro:type:`record` 
+:bro:type:`Broker::ErrorCode`: :bro:type:`enum`      Enumerates the possible error types.
+:bro:type:`Broker::Event`: :bro:type:`record`        Opaque event communication data.
+:bro:type:`Broker::NetworkInfo`: :bro:type:`record`  
+:bro:type:`Broker::PeerInfo`: :bro:type:`record`     
+:bro:type:`Broker::PeerInfos`: :bro:type:`vector`    
+:bro:type:`Broker::PeerStatus`: :bro:type:`enum`     The possible states of a peer endpoint.
+:bro:type:`Broker::TableItem`: :bro:type:`record`    Opaque communication data used as a convenient way to wrap key-value
+                                                     pairs that comprise table entries.
+==================================================== ====================================================================
+
+Functions
+#########
+==================================================================== =======================================================================
+:bro:id:`Broker::auto_publish`: :bro:type:`function`                 Automatically send an event to any interested peers whenever it is
+                                                                     locally dispatched.
+:bro:id:`Broker::auto_unpublish`: :bro:type:`function`               Stop automatically sending an event to peers upon local dispatch.
+:bro:id:`Broker::default_log_topic`: :bro:type:`function`            The default implementation for :bro:see:`Broker::log_topic`.
+:bro:id:`Broker::flush_logs`: :bro:type:`function`                   Sends all pending log messages to remote peers.
+:bro:id:`Broker::forward`: :bro:type:`function`                      Register a topic prefix subscription for events that should only be
+                                                                     forwarded to any subscribing peers and not raise any event handlers
+                                                                     on the receiving/forwarding node.
+:bro:id:`Broker::listen`: :bro:type:`function`                       Listen for remote connections.
+:bro:id:`Broker::log_topic`: :bro:type:`function` :bro:attr:`&redef` A function that will be called for each log entry to determine what
+                                                                     broker topic string will be used for sending it to peers.
+:bro:id:`Broker::node_id`: :bro:type:`function`                      Get a unique identifier for the local broker endpoint.
+:bro:id:`Broker::peer`: :bro:type:`function`                         Initiate a remote connection.
+:bro:id:`Broker::peers`: :bro:type:`function`                        Get a list of all peer connections.
+:bro:id:`Broker::publish_id`: :bro:type:`function`                   Publishes the value of an identifier to a given topic.
+:bro:id:`Broker::subscribe`: :bro:type:`function`                    Register interest in all peer event messages that use a certain topic
+                                                                     prefix.
+:bro:id:`Broker::unpeer`: :bro:type:`function`                       Remove a remote connection.
+:bro:id:`Broker::unsubscribe`: :bro:type:`function`                  Unregister interest in all peer event messages that use a topic prefix.
+==================================================================== =======================================================================
+
+
+Detailed Interface
+~~~~~~~~~~~~~~~~~~
+Redefinable Options
+###################
+.. bro:id:: Broker::aggressive_interval
+
+   :Type: :bro:type:`count`
+   :Attributes: :bro:attr:`&redef`
+   :Default: ``4``
+
+   Frequency of work-stealing polling attempts for Broker/CAF threads
+   in "aggressive" mode.
+
+.. bro:id:: Broker::aggressive_polls
+
+   :Type: :bro:type:`count`
+   :Attributes: :bro:attr:`&redef`
+   :Default: ``5``
+
+   Number of work-stealing polling attempts for Broker/CAF threads
+   in "aggressive" mode.
+
+.. bro:id:: Broker::congestion_queue_size
+
+   :Type: :bro:type:`count`
+   :Attributes: :bro:attr:`&redef`
+   :Default: ``200``
+
+   The number of buffered messages at the Broker/CAF layer after which
+   a subscriber considers themselves congested (i.e. tune the congestion
+   control mechanisms).
+
+.. bro:id:: Broker::default_connect_retry
+
+   :Type: :bro:type:`interval`
+   :Attributes: :bro:attr:`&redef`
+   :Default: ``30.0 secs``
+
+   Default interval to retry connecting to a peer if it cannot be made to
+   work initially, or if it ever becomes disconnected.  Use of the
+   BRO_DEFAULT_CONNECT_RETRY environment variable (set as number of
+   seconds) will override this option and also any values given to
+   :bro:see:`Broker::peer`.
+
+.. bro:id:: Broker::default_listen_address
+
+   :Type: :bro:type:`string`
+   :Attributes: :bro:attr:`&redef`
+   :Default: ``""``
+
+   Default address on which to listen.
+   
+   .. bro:see:: Broker::listen
+
+.. bro:id:: Broker::default_listen_retry
+
+   :Type: :bro:type:`interval`
+   :Attributes: :bro:attr:`&redef`
+   :Default: ``30.0 secs``
+
+   Default interval to retry listening on a port if it's currently in
+   use already.  Use of the BRO_DEFAULT_LISTEN_RETRY environment variable
+   (set as a number of seconds) will override this option and also
+   any values given to :bro:see:`Broker::listen`.
+
+.. bro:id:: Broker::default_log_topic_prefix
+
+   :Type: :bro:type:`string`
+   :Attributes: :bro:attr:`&redef`
+   :Default: ``"bro/logs/"``
+
+   The default topic prefix where logs will be published.  The log's stream
+   id is appended when writing to a particular stream.
+
+.. bro:id:: Broker::default_port
+
+   :Type: :bro:type:`port`
+   :Attributes: :bro:attr:`&redef`
+   :Default: ``9999/tcp``
+
+   Default port for Broker communication. Where not specified
+   otherwise, this is the port to connect to and listen on.
+
+.. bro:id:: Broker::disable_ssl
+
+   :Type: :bro:type:`bool`
+   :Attributes: :bro:attr:`&redef`
+   :Default: ``F``
+
+   If true, do not use SSL for network connections. By default, SSL will
+   even be used if no certificates / CAs have been configured. In that case
+   (which is the default) the communication will be encrypted, but not
+   authenticated.
+
+.. bro:id:: Broker::forward_messages
+
+   :Type: :bro:type:`bool`
+   :Attributes: :bro:attr:`&redef`
+   :Default: ``F``
+
+   Forward all received messages to subscribing peers.
+
+.. bro:id:: Broker::max_threads
+
+   :Type: :bro:type:`count`
+   :Attributes: :bro:attr:`&redef`
+   :Default: ``1``
+
+   Max number of threads to use for Broker/CAF functionality.  The
+   BRO_BROKER_MAX_THREADS environment variable overrides this setting.
+
+.. bro:id:: Broker::moderate_interval
+
+   :Type: :bro:type:`count`
+   :Attributes: :bro:attr:`&redef`
+   :Default: ``2``
+
+   Frequency of work-stealing polling attempts for Broker/CAF threads
+   in "moderate" mode.
+
+.. bro:id:: Broker::moderate_polls
+
+   :Type: :bro:type:`count`
+   :Attributes: :bro:attr:`&redef`
+   :Default: ``5``
+
+   Number of work-stealing polling attempts for Broker/CAF threads
+   in "moderate" mode.
+
+.. bro:id:: Broker::moderate_sleep
+
+   :Type: :bro:type:`interval`
+   :Attributes: :bro:attr:`&redef`
+   :Default: ``16.0 msecs``
+
+   Interval of time for under-utilized Broker/CAF threads to sleep
+   when in "moderate" mode.
+
+.. bro:id:: Broker::relaxed_interval
+
+   :Type: :bro:type:`count`
+   :Attributes: :bro:attr:`&redef`
+   :Default: ``1``
+
+   Frequency of work-stealing polling attempts for Broker/CAF threads
+   in "relaxed" mode.
+
+.. bro:id:: Broker::relaxed_sleep
+
+   :Type: :bro:type:`interval`
+   :Attributes: :bro:attr:`&redef`
+   :Default: ``64.0 msecs``
+
+   Interval of time for under-utilized Broker/CAF threads to sleep
+   when in "relaxed" mode.
+
+.. bro:id:: Broker::ssl_cafile
+
+   :Type: :bro:type:`string`
+   :Attributes: :bro:attr:`&redef`
+   :Default: ``""``
+
+   Path to a file containing concatenated trusted certificates 
+   in PEM format. If set, Bro will require valid certificates for
+   all peers.
+
+.. bro:id:: Broker::ssl_capath
+
+   :Type: :bro:type:`string`
+   :Attributes: :bro:attr:`&redef`
+   :Default: ``""``
+
+   Path to an OpenSSL-style directory of trusted certificates.
+   If set, Bro will require valid certificates for
+   all peers.
+
+.. bro:id:: Broker::ssl_certificate
+
+   :Type: :bro:type:`string`
+   :Attributes: :bro:attr:`&redef`
+   :Default: ``""``
+
+   Path to a file containing a X.509 certificate for this
+   node in PEM format. If set, Bro will require valid certificates for
+   all peers.
+
+.. bro:id:: Broker::ssl_keyfile
+
+   :Type: :bro:type:`string`
+   :Attributes: :bro:attr:`&redef`
+   :Default: ``""``
+
+   Path to the file containing the private key for this node's
+   certificate. If set, Bro will require valid certificates for
+   all peers.
+
+.. bro:id:: Broker::ssl_passphrase
+
+   :Type: :bro:type:`string`
+   :Attributes: :bro:attr:`&redef`
+   :Default: ``""``
+
+   Passphrase to decrypt the private key specified by
+   :bro:see:`Broker::ssl_keyfile`. If set, Bro will require valid
+   certificates for all peers.
+
+Types
+#####
+.. bro:type:: Broker::Data
+
+   :Type: :bro:type:`record`
+
+      data: :bro:type:`opaque` of Broker::Data :bro:attr:`&optional`
+
+   Opaque communication data.
+
+.. bro:type:: Broker::DataVector
+
+   :Type: :bro:type:`vector` of :bro:type:`Broker::Data`
+
+   Opaque communication data sequence.
+
+.. bro:type:: Broker::EndpointInfo
+
+   :Type: :bro:type:`record`
+
+      id: :bro:type:`string`
+         A unique identifier of the node.
+
+      network: :bro:type:`Broker::NetworkInfo` :bro:attr:`&optional`
+         Network-level information.
+
+
+.. bro:type:: Broker::ErrorCode
+
+   :Type: :bro:type:`enum`
+
+      .. bro:enum:: Broker::UNSPECIFIED Broker::ErrorCode
+
+         The unspecified default error code.
+
+      .. bro:enum:: Broker::PEER_INCOMPATIBLE Broker::ErrorCode
+
+         Version incompatibility.
+
+      .. bro:enum:: Broker::PEER_INVALID Broker::ErrorCode
+
+         Referenced peer does not exist.
+
+      .. bro:enum:: Broker::PEER_UNAVAILABLE Broker::ErrorCode
+
+         Remote peer not listening.
+
+      .. bro:enum:: Broker::PEER_TIMEOUT Broker::ErrorCode
+
+         A peering request timed out.
+
+      .. bro:enum:: Broker::MASTER_EXISTS Broker::ErrorCode
+
+         Master with given name already exists.
+
+      .. bro:enum:: Broker::NO_SUCH_MASTER Broker::ErrorCode
+
+         Master with given name does not exist.
+
+      .. bro:enum:: Broker::NO_SUCH_KEY Broker::ErrorCode
+
+         The given data store key does not exist.
+
+      .. bro:enum:: Broker::REQUEST_TIMEOUT Broker::ErrorCode
+
+         The store operation timed out.
+
+      .. bro:enum:: Broker::TYPE_CLASH Broker::ErrorCode
+
+         The operation expected a different type than provided.
+
+      .. bro:enum:: Broker::INVALID_DATA Broker::ErrorCode
+
+         The data value cannot be used to carry out the desired operation.
+
+      .. bro:enum:: Broker::BACKEND_FAILURE Broker::ErrorCode
+
+         The storage backend failed to execute the operation.
+
+      .. bro:enum:: Broker::STALE_DATA Broker::ErrorCode
+
+         The storage backend failed to execute the operation.
+
+      .. bro:enum:: Broker::CAF_ERROR Broker::ErrorCode
+
+         Catch-all for a CAF-level problem.
+
+   Enumerates the possible error types. 
+
+.. bro:type:: Broker::Event
+
+   :Type: :bro:type:`record`
+
+      name: :bro:type:`string` :bro:attr:`&optional`
+         The name of the event.  Not set if invalid event or arguments.
+
+      args: :bro:type:`Broker::DataVector`
+         The arguments to the event.
+
+   Opaque event communication data.
+
+.. bro:type:: Broker::NetworkInfo
+
+   :Type: :bro:type:`record`
+
+      address: :bro:type:`string` :bro:attr:`&log`
+         The IP address or hostname where the endpoint listens.
+
+      bound_port: :bro:type:`port` :bro:attr:`&log`
+         The port where the endpoint is bound to.
+
+
+.. bro:type:: Broker::PeerInfo
+
+   :Type: :bro:type:`record`
+
+      peer: :bro:type:`Broker::EndpointInfo`
+
+      status: :bro:type:`Broker::PeerStatus`
+
+
+.. bro:type:: Broker::PeerInfos
+
+   :Type: :bro:type:`vector` of :bro:type:`Broker::PeerInfo`
+
+
+.. bro:type:: Broker::PeerStatus
+
+   :Type: :bro:type:`enum`
+
+      .. bro:enum:: Broker::INITIALIZING Broker::PeerStatus
+
+         The peering process is initiated.
+
+      .. bro:enum:: Broker::CONNECTING Broker::PeerStatus
+
+         Connection establishment in process.
+
+      .. bro:enum:: Broker::CONNECTED Broker::PeerStatus
+
+         Connection established, peering pending.
+
+      .. bro:enum:: Broker::PEERED Broker::PeerStatus
+
+         Successfully peered.
+
+      .. bro:enum:: Broker::DISCONNECTED Broker::PeerStatus
+
+         Connection to remote peer lost.
+
+      .. bro:enum:: Broker::RECONNECTING Broker::PeerStatus
+
+         Reconnecting to peer after a lost connection.
+
+   The possible states of a peer endpoint.
+
+.. bro:type:: Broker::TableItem
+
+   :Type: :bro:type:`record`
+
+      key: :bro:type:`Broker::Data`
+
+      val: :bro:type:`Broker::Data`
+
+   Opaque communication data used as a convenient way to wrap key-value
+   pairs that comprise table entries.
+
+Functions
+#########
+.. bro:id:: Broker::auto_publish
+
+   :Type: :bro:type:`function` (topic: :bro:type:`string`, ev: :bro:type:`any`) : :bro:type:`bool`
+
+   Automatically send an event to any interested peers whenever it is
+   locally dispatched. (For example, using "event my_event(...);" in a
+   script.)
+   
+
+   :topic: a topic string associated with the event message.
+          Peers advertise interest by registering a subscription to some
+          prefix of this topic name.
+   
+
+   :ev: a Bro event value.
+   
+
+   :returns: true if automatic event sending is now enabled.
+
+.. bro:id:: Broker::auto_unpublish
+
+   :Type: :bro:type:`function` (topic: :bro:type:`string`, ev: :bro:type:`any`) : :bro:type:`bool`
+
+   Stop automatically sending an event to peers upon local dispatch.
+   
+
+   :topic: a topic originally given to :bro:see:`Broker::auto_publish`.
+   
+
+   :ev: an event originally given to :bro:see:`Broker::auto_publish`.
+   
+
+   :returns: true if automatic events will not occur for the topic/event
+            pair.
+
+.. bro:id:: Broker::default_log_topic
+
+   :Type: :bro:type:`function` (id: :bro:type:`Log::ID`, path: :bro:type:`string`) : :bro:type:`string`
+
+   The default implementation for :bro:see:`Broker::log_topic`.
+
+.. bro:id:: Broker::flush_logs
+
+   :Type: :bro:type:`function` () : :bro:type:`count`
+
+   Sends all pending log messages to remote peers.  This normally
+   doesn't need to be used except for test cases that are time-sensitive.
+
+.. bro:id:: Broker::forward
+
+   :Type: :bro:type:`function` (topic_prefix: :bro:type:`string`) : :bro:type:`bool`
+
+   Register a topic prefix subscription for events that should only be
+   forwarded to any subscribing peers and not raise any event handlers
+   on the receiving/forwarding node.  i.e. it's the same as
+   :bro:see:`Broker::subscribe` except matching events are not raised
+   on the receiver, just forwarded.  Use :bro:see:`Broker::unsubscribe`
+   with the same argument to undo this operation.
+   
+
+   :topic_prefix: a prefix to match against remote message topics.
+                 e.g. an empty prefix matches everything and "a" matches
+                 "alice" and "amy" but not "bob".
+   
+
+   :returns: true if a new event forwarding/subscription is now registered.
+
+.. bro:id:: Broker::listen
+
+   :Type: :bro:type:`function` (a: :bro:type:`string` :bro:attr:`&default` = :bro:see:`Broker::default_listen_address` :bro:attr:`&optional`, p: :bro:type:`port` :bro:attr:`&default` = :bro:see:`Broker::default_port` :bro:attr:`&optional`, retry: :bro:type:`interval` :bro:attr:`&default` = :bro:see:`Broker::default_listen_retry` :bro:attr:`&optional`) : :bro:type:`port`
+
+   Listen for remote connections.
+   
+
+   :a: an address string on which to accept connections, e.g.
+      "127.0.0.1".  An empty string refers to INADDR_ANY.
+   
+
+   :p: the TCP port to listen on. The value 0 means that the OS should choose
+      the next available free port.
+   
+
+   :retry: If non-zero, retries listening in regular intervals if the port cannot be
+          acquired immediately. 0 disables retries.  If the
+          BRO_DEFAULT_LISTEN_RETRY environment variable is set (as number
+          of seconds), it overrides any value given here.
+   
+
+   :returns: the bound port or 0/? on failure.
+   
+   .. bro:see:: Broker::status
+
+.. bro:id:: Broker::log_topic
+
+   :Type: :bro:type:`function` (id: :bro:type:`Log::ID`, path: :bro:type:`string`) : :bro:type:`string`
+   :Attributes: :bro:attr:`&redef`
+
+   A function that will be called for each log entry to determine what
+   broker topic string will be used for sending it to peers.  The
+   default implementation will return a value based on
+   :bro:see:`Broker::default_log_topic_prefix`.
+   
+
+   :id: the ID associated with the log stream entry that will be sent.
+   
+
+   :path: the path to which the log stream entry will be output.
+   
+
+   :returns: a string representing the broker topic to which the log
+            will be sent.
+
+.. bro:id:: Broker::node_id
+
+   :Type: :bro:type:`function` () : :bro:type:`string`
+
+   Get a unique identifier for the local broker endpoint.
+   
+
+   :returns: a unique identifier for the local broker endpoint.
+
+.. bro:id:: Broker::peer
+
+   :Type: :bro:type:`function` (a: :bro:type:`string`, p: :bro:type:`port` :bro:attr:`&default` = :bro:see:`Broker::default_port` :bro:attr:`&optional`, retry: :bro:type:`interval` :bro:attr:`&default` = :bro:see:`Broker::default_connect_retry` :bro:attr:`&optional`) : :bro:type:`bool`
+
+   Initiate a remote connection.
+   
+
+   :a: an address to connect to, e.g. "localhost" or "127.0.0.1".
+   
+
+   :p: the TCP port on which the remote side is listening.
+   
+
+   :retry: an interval at which to retry establishing the
+          connection with the remote peer if it cannot be made initially, or
+          if it ever becomes disconnected.  If the
+          BRO_DEFAULT_CONNECT_RETRY environment variable is set (as number
+          of seconds), it overrides any value given here.
+   
+
+   :returns: true if it's possible to try connecting with the peer and
+            it's a new peer. The actual connection may not be established
+            until a later point in time.
+   
+   .. bro:see:: Broker::status
+
+.. bro:id:: Broker::peers
+
+   :Type: :bro:type:`function` () : :bro:type:`vector` of :bro:type:`Broker::PeerInfo`
+
+   Get a list of all peer connections.
+   
+
+   :returns: a list of all peer connections.
+
+.. bro:id:: Broker::publish_id
+
+   :Type: :bro:type:`function` (topic: :bro:type:`string`, id: :bro:type:`string`) : :bro:type:`bool`
+
+   Publishes the value of an identifier to a given topic.  The subscribers
+   will update their local value for that identifier on receipt.
+   
+
+   :topic: a topic associated with the message.
+   
+
+   :id: the identifier to publish.
+   
+
+   :returns: true if the message is sent.
+
+.. bro:id:: Broker::subscribe
+
+   :Type: :bro:type:`function` (topic_prefix: :bro:type:`string`) : :bro:type:`bool`
+
+   Register interest in all peer event messages that use a certain topic
+   prefix.  Note that subscriptions may not be altered immediately after
+   calling (except during :bro:see:`bro_init`).
+   
+
+   :topic_prefix: a prefix to match against remote message topics.
+                 e.g. an empty prefix matches everything and "a" matches
+                 "alice" and "amy" but not "bob".
+   
+
+   :returns: true if it's a new event subscription and it is now registered.
+
+.. bro:id:: Broker::unpeer
+
+   :Type: :bro:type:`function` (a: :bro:type:`string`, p: :bro:type:`port`) : :bro:type:`bool`
+
+   Remove a remote connection.
+   
+   Note that this does not terminate the connection to the peer, it
+   just means that we won't exchange any further information with it
+   unless peering resumes later.
+   
+
+   :a: the address used in previous successful call to :bro:see:`Broker::peer`.
+   
+
+   :p: the port used in previous successful call to :bro:see:`Broker::peer`.
+   
+
+   :returns: true if the arguments match a previously successful call to
+            :bro:see:`Broker::peer`.
+   
+
+   :TODO: We do not have a function yet to terminate a connection.
+
+.. bro:id:: Broker::unsubscribe
+
+   :Type: :bro:type:`function` (topic_prefix: :bro:type:`string`) : :bro:type:`bool`
+
+   Unregister interest in all peer event messages that use a topic prefix.
+   Note that subscriptions may not be altered immediately after calling
+   (except during :bro:see:`bro_init`).
+   
+
+   :topic_prefix: a prefix previously supplied to a successful call to
+                 :bro:see:`Broker::subscribe` or :bro:see:`Broker::forward`.
+   
+
+   :returns: true if interest in the topic prefix is no longer advertised.
+
+

doc/scripts/base/frameworks/cluster/__load__.bro.rst

@@ -0,0 +1,14 @@
+:tocdepth: 3
+
+base/frameworks/cluster/__load__.bro
+====================================
+
+
+:Imports: :doc:`base/frameworks/cluster/main.bro </scripts/base/frameworks/cluster/main.bro>`, :doc:`base/frameworks/cluster/pools.bro </scripts/base/frameworks/cluster/pools.bro>`
+
+Summary
+~~~~~~~
+
+Detailed Interface
+~~~~~~~~~~~~~~~~~~
+

doc/scripts/base/frameworks/cluster/index.rst

@@ -0,0 +1,26 @@
+:orphan:
+
+Package: base/frameworks/cluster
+================================
+
+The cluster framework provides for establishing and controlling a cluster
+of Bro instances.
+
+:doc:`/scripts/base/frameworks/cluster/__load__.bro`
+
+
+:doc:`/scripts/base/frameworks/cluster/main.bro`
+
+   A framework for establishing and controlling a cluster of Bro instances.
+   In order to use the cluster framework, a script named
+   ``cluster-layout.bro`` must exist somewhere in Bro's script search path
+   which has a cluster definition of the :bro:id:`Cluster::nodes` variable.
+   The ``CLUSTER_NODE`` environment variable or :bro:id:`Cluster::node`
+   must also be sent and the cluster framework loaded as a package like
+   ``@load base/frameworks/cluster``.
+
+:doc:`/scripts/base/frameworks/cluster/pools.bro`
+
+   Defines an interface for managing pools of cluster nodes.  Pools are
+   a useful way to distribute work or data among nodes within a cluster.
+

doc/scripts/base/frameworks/cluster/main.bro.rst

@@ -0,0 +1,529 @@
+:tocdepth: 3
+
+base/frameworks/cluster/main.bro
+================================
+.. bro:namespace:: Cluster
+
+A framework for establishing and controlling a cluster of Bro instances.
+In order to use the cluster framework, a script named
+``cluster-layout.bro`` must exist somewhere in Bro's script search path
+which has a cluster definition of the :bro:id:`Cluster::nodes` variable.
+The ``CLUSTER_NODE`` environment variable or :bro:id:`Cluster::node`
+must also be sent and the cluster framework loaded as a package like
+``@load base/frameworks/cluster``.
+
+:Namespace: Cluster
+:Imports: :doc:`base/frameworks/broker </scripts/base/frameworks/broker/index>`, :doc:`base/frameworks/control </scripts/base/frameworks/control/index>`
+
+Summary
+~~~~~~~
+Redefinable Options
+###################
+================================================================================================= ==============================================================================
+:bro:id:`Cluster::default_backend`: :bro:type:`Broker::BackendType` :bro:attr:`&redef`            The type of data store backend that will be used for all data stores if
+                                                                                                  no other has already been specified by the user in :bro:see:`Cluster::stores`.
+:bro:id:`Cluster::default_master_node`: :bro:type:`string` :bro:attr:`&redef`                     Name of the node on which master data stores will be created if no other
+                                                                                                  has already been specified by the user in :bro:see:`Cluster::stores`.
+:bro:id:`Cluster::default_persistent_backend`: :bro:type:`Broker::BackendType` :bro:attr:`&redef` The type of persistent data store backend that will be used for all data
+                                                                                                  stores if no other has already been specified by the user in
+                                                                                                  :bro:see:`Cluster::stores`.
+:bro:id:`Cluster::default_store_dir`: :bro:type:`string` :bro:attr:`&redef`                       Setting a default dir will, for persistent backends that have not
+                                                                                                  been given an explicit file path via :bro:see:`Cluster::stores`,
+                                                                                                  automatically create a path within this dir that is based on the name of
+                                                                                                  the data store.
+:bro:id:`Cluster::enable_round_robin_logging`: :bro:type:`bool` :bro:attr:`&redef`                Whether to distribute log messages among available logging nodes.
+:bro:id:`Cluster::logger_topic`: :bro:type:`string` :bro:attr:`&redef`                            The topic name used for exchanging messages that are relevant to
+                                                                                                  logger nodes in a cluster.
+:bro:id:`Cluster::manager_is_logger`: :bro:type:`bool` :bro:attr:`&redef`                         Indicates whether or not the manager will act as the logger and receive
+                                                                                                  logs.
+:bro:id:`Cluster::manager_topic`: :bro:type:`string` :bro:attr:`&redef`                           The topic name used for exchanging messages that are relevant to
+                                                                                                  manager nodes in a cluster.
+:bro:id:`Cluster::node`: :bro:type:`string` :bro:attr:`&redef`                                    This is usually supplied on the command line for each instance
+                                                                                                  of the cluster that is started up.
+:bro:id:`Cluster::node_topic_prefix`: :bro:type:`string` :bro:attr:`&redef`                       The topic prefix used for exchanging messages that are relevant to
+                                                                                                  a named node in a cluster.
+:bro:id:`Cluster::nodeid_topic_prefix`: :bro:type:`string` :bro:attr:`&redef`                     The topic prefix used for exchanging messages that are relevant to
+                                                                                                  a unique node in a cluster.
+:bro:id:`Cluster::nodes`: :bro:type:`table` :bro:attr:`&redef`                                    The cluster layout definition.
+:bro:id:`Cluster::proxy_topic`: :bro:type:`string` :bro:attr:`&redef`                             The topic name used for exchanging messages that are relevant to
+                                                                                                  proxy nodes in a cluster.
+:bro:id:`Cluster::retry_interval`: :bro:type:`interval` :bro:attr:`&redef`                        Interval for retrying failed connections between cluster nodes.
+:bro:id:`Cluster::time_machine_topic`: :bro:type:`string` :bro:attr:`&redef`                      The topic name used for exchanging messages that are relevant to
+                                                                                                  time machine nodes in a cluster.
+:bro:id:`Cluster::worker_topic`: :bro:type:`string` :bro:attr:`&redef`                            The topic name used for exchanging messages that are relevant to
+                                                                                                  worker nodes in a cluster.
+================================================================================================= ==============================================================================
+
+State Variables
+###############
+================================================================================================================================================================================================================================================================================================================================================================== ======================================================================
+:bro:id:`Cluster::stores`: :bro:type:`table` :bro:attr:`&default` = ``[name=<uninitialized>, store=<uninitialized>, master_node=, master=F, backend=Broker::MEMORY, options=[sqlite=[path=], rocksdb=[path=]], clone_resync_interval=10.0 secs, clone_stale_interval=5.0 mins, clone_mutation_buffer_interval=2.0 mins]`` :bro:attr:`&optional` :bro:attr:`&redef` A table of cluster-enabled data stores that have been created, indexed
+                                                                                                                                                                                                                                                                                                                                                                   by their name.
+:bro:id:`Cluster::worker_count`: :bro:type:`count`                                                                                                                                                                                                                                                                                                                 This gives the value for the number of workers currently connected to,
+                                                                                                                                                                                                                                                                                                                                                                   and it's maintained internally by the cluster framework.
+================================================================================================================================================================================================================================================================================================================================================================== ======================================================================
+
+Types
+#####
+============================================================== ====================================================================
+:bro:type:`Cluster::Info`: :bro:type:`record` :bro:attr:`&log` The record type which contains the column fields of the cluster log.
+:bro:type:`Cluster::Node`: :bro:type:`record`                  Record type to indicate a node in a cluster.
+:bro:type:`Cluster::NodeType`: :bro:type:`enum`                Types of nodes that are allowed to participate in the cluster
+                                                               configuration.
+:bro:type:`Cluster::StoreInfo`: :bro:type:`record`             Information regarding a cluster-enabled data store.
+============================================================== ====================================================================
+
+Redefinitions
+#############
+===================================== ======================================
+:bro:type:`Log::ID`: :bro:type:`enum` The cluster logging stream identifier.
+===================================== ======================================
+
+Events
+######
+=============================================== =======================================================================
+:bro:id:`Cluster::hello`: :bro:type:`event`     When using broker-enabled cluster framework, nodes broadcast this event
+                                                to exchange their user-defined name along with a string that uniquely
+                                                identifies it for the duration of its lifetime.
+:bro:id:`Cluster::node_down`: :bro:type:`event` When using broker-enabled cluster framework, this event will be emitted
+                                                locally whenever a connected cluster node becomes disconnected.
+:bro:id:`Cluster::node_up`: :bro:type:`event`   When using broker-enabled cluster framework, this event will be emitted
+                                                locally whenever a cluster node connects or reconnects.
+=============================================== =======================================================================
+
+Functions
+#########
+======================================================== ===================================================================
+:bro:id:`Cluster::create_store`: :bro:type:`function`    Sets up a cluster-enabled data store.
+:bro:id:`Cluster::is_enabled`: :bro:type:`function`      This function can be called at any time to determine if the cluster
+                                                         framework is being enabled for this run.
+:bro:id:`Cluster::local_node_type`: :bro:type:`function` This function can be called at any time to determine what type of
+                                                         cluster node the current Bro instance is going to be acting as.
+:bro:id:`Cluster::log`: :bro:type:`function`             Write a message to the cluster logging stream.
+:bro:id:`Cluster::node_topic`: :bro:type:`function`      Retrieve the topic associated with a specific node in the cluster.
+:bro:id:`Cluster::nodeid_topic`: :bro:type:`function`    Retrieve the topic associated with a specific node in the cluster.
+======================================================== ===================================================================
+
+
+Detailed Interface
+~~~~~~~~~~~~~~~~~~
+Redefinable Options
+###################
+.. bro:id:: Cluster::default_backend
+
+   :Type: :bro:type:`Broker::BackendType`
+   :Attributes: :bro:attr:`&redef`
+   :Default: ``Broker::MEMORY``
+
+   The type of data store backend that will be used for all data stores if
+   no other has already been specified by the user in :bro:see:`Cluster::stores`.
+
+.. bro:id:: Cluster::default_master_node
+
+   :Type: :bro:type:`string`
+   :Attributes: :bro:attr:`&redef`
+   :Default: ``""``
+
+   Name of the node on which master data stores will be created if no other
+   has already been specified by the user in :bro:see:`Cluster::stores`.
+   An empty value means "use whatever name corresponds to the manager
+   node".
+
+.. bro:id:: Cluster::default_persistent_backend
+
+   :Type: :bro:type:`Broker::BackendType`
+   :Attributes: :bro:attr:`&redef`
+   :Default: ``Broker::SQLITE``
+
+   The type of persistent data store backend that will be used for all data
+   stores if no other has already been specified by the user in
+   :bro:see:`Cluster::stores`.  This will be used when script authors call
+   :bro:see:`Cluster::create_store` with the *persistent* argument set true.
+
+.. bro:id:: Cluster::default_store_dir
+
+   :Type: :bro:type:`string`
+   :Attributes: :bro:attr:`&redef`
+   :Default: ``""``
+
+   Setting a default dir will, for persistent backends that have not
+   been given an explicit file path via :bro:see:`Cluster::stores`,
+   automatically create a path within this dir that is based on the name of
+   the data store.
+
+.. bro:id:: Cluster::enable_round_robin_logging
+
+   :Type: :bro:type:`bool`
+   :Attributes: :bro:attr:`&redef`
+   :Default: ``T``
+
+   Whether to distribute log messages among available logging nodes.
+
+.. bro:id:: Cluster::logger_topic
+
+   :Type: :bro:type:`string`
+   :Attributes: :bro:attr:`&redef`
+   :Default: ``"bro/cluster/logger"``
+
+   The topic name used for exchanging messages that are relevant to
+   logger nodes in a cluster.  Used with broker-enabled cluster communication.
+
+.. bro:id:: Cluster::manager_is_logger
+
+   :Type: :bro:type:`bool`
+   :Attributes: :bro:attr:`&redef`
+   :Default: ``T``
+
+   Indicates whether or not the manager will act as the logger and receive
+   logs.  This value should be set in the cluster-layout.bro script (the
+   value should be true only if no logger is specified in Cluster::nodes).
+   Note that BroControl handles this automatically.
+
+.. bro:id:: Cluster::manager_topic
+
+   :Type: :bro:type:`string`
+   :Attributes: :bro:attr:`&redef`
+   :Default: ``"bro/cluster/manager"``
+
+   The topic name used for exchanging messages that are relevant to
+   manager nodes in a cluster.  Used with broker-enabled cluster communication.
+
+.. bro:id:: Cluster::node
+
+   :Type: :bro:type:`string`
+   :Attributes: :bro:attr:`&redef`
+   :Default: ``""``
+
+   This is usually supplied on the command line for each instance
+   of the cluster that is started up.
+
+.. bro:id:: Cluster::node_topic_prefix
+
+   :Type: :bro:type:`string`
+   :Attributes: :bro:attr:`&redef`
+   :Default: ``"bro/cluster/node/"``
+
+   The topic prefix used for exchanging messages that are relevant to
+   a named node in a cluster.  Used with broker-enabled cluster communication.
+
+.. bro:id:: Cluster::nodeid_topic_prefix
+
+   :Type: :bro:type:`string`
+   :Attributes: :bro:attr:`&redef`
+   :Default: ``"bro/cluster/nodeid/"``
+
+   The topic prefix used for exchanging messages that are relevant to
+   a unique node in a cluster.  Used with broker-enabled cluster communication.
+
+.. bro:id:: Cluster::nodes
+
+   :Type: :bro:type:`table` [:bro:type:`string`] of :bro:type:`Cluster::Node`
+   :Attributes: :bro:attr:`&redef`
+   :Default: ``{}``
+
+   The cluster layout definition.  This should be placed into a filter
+   named cluster-layout.bro somewhere in the BROPATH.  It will be
+   automatically loaded if the CLUSTER_NODE environment variable is set.
+   Note that BroControl handles all of this automatically.
+   The table is typically indexed by node names/labels (e.g. "manager"
+   or "worker-1").
+
+.. bro:id:: Cluster::proxy_topic
+
+   :Type: :bro:type:`string`
+   :Attributes: :bro:attr:`&redef`
+   :Default: ``"bro/cluster/proxy"``
+
+   The topic name used for exchanging messages that are relevant to
+   proxy nodes in a cluster.  Used with broker-enabled cluster communication.
+
+.. bro:id:: Cluster::retry_interval
+
+   :Type: :bro:type:`interval`
+   :Attributes: :bro:attr:`&redef`
+   :Default: ``1.0 min``
+
+   Interval for retrying failed connections between cluster nodes.
+   If set, the BRO_DEFAULT_CONNECT_RETRY (given in number of seconds)
+   overrides this option.
+
+.. bro:id:: Cluster::time_machine_topic
+
+   :Type: :bro:type:`string`
+   :Attributes: :bro:attr:`&redef`
+   :Default: ``"bro/cluster/time_machine"``
+
+   The topic name used for exchanging messages that are relevant to
+   time machine nodes in a cluster.  Used with broker-enabled cluster communication.
+
+.. bro:id:: Cluster::worker_topic
+
+   :Type: :bro:type:`string`
+   :Attributes: :bro:attr:`&redef`
+   :Default: ``"bro/cluster/worker"``
+
+   The topic name used for exchanging messages that are relevant to
+   worker nodes in a cluster.  Used with broker-enabled cluster communication.
+
+State Variables
+###############
+.. bro:id:: Cluster::stores
+
+   :Type: :bro:type:`table` [:bro:type:`string`] of :bro:type:`Cluster::StoreInfo`
+   :Attributes: :bro:attr:`&default` = ``[name=<uninitialized>, store=<uninitialized>, master_node=, master=F, backend=Broker::MEMORY, options=[sqlite=[path=], rocksdb=[path=]], clone_resync_interval=10.0 secs, clone_stale_interval=5.0 mins, clone_mutation_buffer_interval=2.0 mins]`` :bro:attr:`&optional` :bro:attr:`&redef`
+   :Default: ``{}``
+
+   A table of cluster-enabled data stores that have been created, indexed
+   by their name.  This table will be populated automatically by
+   :bro:see:`Cluster::create_store`, but if you need to customize
+   the options related to a particular data store, you may redef this
+   table.  Calls to :bro:see:`Cluster::create_store` will first check
+   the table for an entry of the same name and, if found, will use the
+   predefined options there when setting up the store.
+
+.. bro:id:: Cluster::worker_count
+
+   :Type: :bro:type:`count`
+   :Default: ``0``
+
+   This gives the value for the number of workers currently connected to,
+   and it's maintained internally by the cluster framework.  It's
+   primarily intended for use by managers to find out how many workers
+   should be responding to requests.
+
+Types
+#####
+.. bro:type:: Cluster::Info
+
+   :Type: :bro:type:`record`
+
+      ts: :bro:type:`time` :bro:attr:`&log`
+         The time at which a cluster message was generated.
+
+      node: :bro:type:`string` :bro:attr:`&log`
+         The name of the node that is creating the log record.
+
+      message: :bro:type:`string` :bro:attr:`&log`
+         A message indicating information about the cluster's operation.
+   :Attributes: :bro:attr:`&log`
+
+   The record type which contains the column fields of the cluster log.
+
+.. bro:type:: Cluster::Node
+
+   :Type: :bro:type:`record`
+
+      node_type: :bro:type:`Cluster::NodeType`
+         Identifies the type of cluster node in this node's configuration.
+
+      ip: :bro:type:`addr`
+         The IP address of the cluster node.
+
+      zone_id: :bro:type:`string` :bro:attr:`&default` = ``""`` :bro:attr:`&optional`
+         If the *ip* field is a non-global IPv6 address, this field
+         can specify a particular :rfc:`4007` ``zone_id``.
+
+      p: :bro:type:`port`
+         The port that this node will listen on for peer connections.
+
+      interface: :bro:type:`string` :bro:attr:`&optional`
+         Identifier for the interface a worker is sniffing.
+
+      manager: :bro:type:`string` :bro:attr:`&optional`
+         Name of the manager node this node uses.  For workers and proxies.
+
+      time_machine: :bro:type:`string` :bro:attr:`&optional`
+         Name of a time machine node with which this node connects.
+
+      id: :bro:type:`string` :bro:attr:`&optional`
+         A unique identifier assigned to the node by the broker framework.
+         This field is only set while a node is connected.
+
+      lb_filter: :bro:type:`string` :bro:attr:`&optional`
+         (present if :doc:`/scripts/policy/misc/load-balancing.bro` is loaded)
+
+         A BPF filter for load balancing traffic sniffed on a single
+         interface across a number of processes.  In normal uses, this
+         will be assigned dynamically by the manager and installed by
+         the workers.
+
+   Record type to indicate a node in a cluster.
+
+.. bro:type:: Cluster::NodeType
+
+   :Type: :bro:type:`enum`
+
+      .. bro:enum:: Cluster::NONE Cluster::NodeType
+
+         A dummy node type indicating the local node is not operating
+         within a cluster.
+
+      .. bro:enum:: Cluster::CONTROL Cluster::NodeType
+
+         A node type which is allowed to view/manipulate the configuration
+         of other nodes in the cluster.
+
+      .. bro:enum:: Cluster::LOGGER Cluster::NodeType
+
+         A node type responsible for log management.
+
+      .. bro:enum:: Cluster::MANAGER Cluster::NodeType
+
+         A node type responsible for policy management.
+
+      .. bro:enum:: Cluster::PROXY Cluster::NodeType
+
+         A node type for relaying worker node communication and synchronizing
+         worker node state.
+
+      .. bro:enum:: Cluster::WORKER Cluster::NodeType
+
+         The node type doing all the actual traffic analysis.
+
+      .. bro:enum:: Cluster::TIME_MACHINE Cluster::NodeType
+
+         A node acting as a traffic recorder using the
+         `Time Machine <https://www.zeek.org/community/time-machine.html>`_
+         software.
+
+   Types of nodes that are allowed to participate in the cluster
+   configuration.
+
+.. bro:type:: Cluster::StoreInfo
+
+   :Type: :bro:type:`record`
+
+      name: :bro:type:`string` :bro:attr:`&optional`
+         The name of the data store.
+
+      store: :bro:type:`opaque` of Broker::Store :bro:attr:`&optional`
+         The store handle.
+
+      master_node: :bro:type:`string` :bro:attr:`&default` = :bro:see:`Cluster::default_master_node` :bro:attr:`&optional`
+         The name of the cluster node on which the master version of the data
+         store resides.
+
+      master: :bro:type:`bool` :bro:attr:`&default` = ``F`` :bro:attr:`&optional`
+         Whether the data store is the master version or a clone.
+
+      backend: :bro:type:`Broker::BackendType` :bro:attr:`&default` = :bro:see:`Cluster::default_backend` :bro:attr:`&optional`
+         The type of backend used for storing data.
+
+      options: :bro:type:`Broker::BackendOptions` :bro:attr:`&default` = ``[sqlite=[path=], rocksdb=[path=]]`` :bro:attr:`&optional`
+         Parameters used for configuring the backend.
+
+      clone_resync_interval: :bro:type:`interval` :bro:attr:`&default` = :bro:see:`Broker::default_clone_resync_interval` :bro:attr:`&optional`
+         A resync/reconnect interval to pass through to
+         :bro:see:`Broker::create_clone`.
+
+      clone_stale_interval: :bro:type:`interval` :bro:attr:`&default` = :bro:see:`Broker::default_clone_stale_interval` :bro:attr:`&optional`
+         A staleness duration to pass through to
+         :bro:see:`Broker::create_clone`.
+
+      clone_mutation_buffer_interval: :bro:type:`interval` :bro:attr:`&default` = :bro:see:`Broker::default_clone_mutation_buffer_interval` :bro:attr:`&optional`
+         A mutation buffer interval to pass through to
+         :bro:see:`Broker::create_clone`.
+
+   Information regarding a cluster-enabled data store.
+
+Events
+######
+.. bro:id:: Cluster::hello
+
+   :Type: :bro:type:`event` (name: :bro:type:`string`, id: :bro:type:`string`)
+
+   When using broker-enabled cluster framework, nodes broadcast this event
+   to exchange their user-defined name along with a string that uniquely
+   identifies it for the duration of its lifetime.  This string may change
+   if the node dies and has to reconnect later.
+
+.. bro:id:: Cluster::node_down
+
+   :Type: :bro:type:`event` (name: :bro:type:`string`, id: :bro:type:`string`)
+
+   When using broker-enabled cluster framework, this event will be emitted
+   locally whenever a connected cluster node becomes disconnected.
+
+.. bro:id:: Cluster::node_up
+
+   :Type: :bro:type:`event` (name: :bro:type:`string`, id: :bro:type:`string`)
+
+   When using broker-enabled cluster framework, this event will be emitted
+   locally whenever a cluster node connects or reconnects.
+
+Functions
+#########
+.. bro:id:: Cluster::create_store
+
+   :Type: :bro:type:`function` (name: :bro:type:`string`, persistent: :bro:type:`bool` :bro:attr:`&default` = ``F`` :bro:attr:`&optional`) : :bro:type:`Cluster::StoreInfo`
+
+   Sets up a cluster-enabled data store.  They will also still properly
+   function for uses that are not operating a cluster.
+   
+
+   :name: the name of the data store to create.
+   
+
+   :persistent: whether the data store must be persistent.
+   
+
+   :returns: the store's information.  For master stores, the store will be
+            ready to use immediately.  For clones, the store field will not
+            be set until the node containing the master store has connected.
+
+.. bro:id:: Cluster::is_enabled
+
+   :Type: :bro:type:`function` () : :bro:type:`bool`
+
+   This function can be called at any time to determine if the cluster
+   framework is being enabled for this run.
+   
+
+   :returns: True if :bro:id:`Cluster::node` has been set.
+
+.. bro:id:: Cluster::local_node_type
+
+   :Type: :bro:type:`function` () : :bro:type:`Cluster::NodeType`
+
+   This function can be called at any time to determine what type of
+   cluster node the current Bro instance is going to be acting as.
+   If :bro:id:`Cluster::is_enabled` returns false, then
+   :bro:enum:`Cluster::NONE` is returned.
+   
+
+   :returns: The :bro:type:`Cluster::NodeType` the calling node acts as.
+
+.. bro:id:: Cluster::log
+
+   :Type: :bro:type:`function` (msg: :bro:type:`string`) : :bro:type:`void`
+
+   Write a message to the cluster logging stream.
+
+.. bro:id:: Cluster::node_topic
+
+   :Type: :bro:type:`function` (name: :bro:type:`string`) : :bro:type:`string`
+
+   Retrieve the topic associated with a specific node in the cluster.
+   
+
+   :name: the name of the cluster node (e.g. "manager").
+   
+
+   :returns: a topic string that may used to send a message exclusively to
+            a given cluster node.
+
+.. bro:id:: Cluster::nodeid_topic
+
+   :Type: :bro:type:`function` (id: :bro:type:`string`) : :bro:type:`string`
+
+   Retrieve the topic associated with a specific node in the cluster.
+   
+
+   :id: the id of the cluster node (from :bro:see:`Broker::EndpointInfo`
+       or :bro:see:`Broker::node_id`.
+   
+
+   :returns: a topic string that may used to send a message exclusively to
+            a given cluster node.
+
+

doc/scripts/base/frameworks/cluster/pools.bro.rst

@@ -0,0 +1,291 @@
+:tocdepth: 3
+
+base/frameworks/cluster/pools.bro
+=================================
+.. bro:namespace:: Cluster
+
+Defines an interface for managing pools of cluster nodes.  Pools are
+a useful way to distribute work or data among nodes within a cluster.
+
+:Namespace: Cluster
+:Imports: :doc:`base/frameworks/cluster/main.bro </scripts/base/frameworks/cluster/main.bro>`, :doc:`base/utils/hash_hrw.bro </scripts/base/utils/hash_hrw.bro>`
+
+Summary
+~~~~~~~
+State Variables
+###############
+===================================================================================== ======================================================
+:bro:id:`Cluster::logger_pool`: :bro:type:`Cluster::Pool`                             A pool containing all the logger nodes of a cluster.
+:bro:id:`Cluster::logger_pool_spec`: :bro:type:`Cluster::PoolSpec` :bro:attr:`&redef` The specification for :bro:see:`Cluster::logger_pool`.
+:bro:id:`Cluster::proxy_pool`: :bro:type:`Cluster::Pool`                              A pool containing all the proxy nodes of a cluster.
+:bro:id:`Cluster::proxy_pool_spec`: :bro:type:`Cluster::PoolSpec` :bro:attr:`&redef`  The specification for :bro:see:`Cluster::proxy_pool`.
+:bro:id:`Cluster::worker_pool`: :bro:type:`Cluster::Pool`                             A pool containing all the worker nodes of a cluster.
+:bro:id:`Cluster::worker_pool_spec`: :bro:type:`Cluster::PoolSpec` :bro:attr:`&redef` The specification for :bro:see:`Cluster::worker_pool`.
+===================================================================================== ======================================================
+
+Types
+#####
+======================================================= ===========================================================
+:bro:type:`Cluster::PoolNode`: :bro:type:`record`       Store state of a cluster within the context of a work pool.
+:bro:type:`Cluster::PoolNodeTable`: :bro:type:`table`   
+:bro:type:`Cluster::PoolSpec`: :bro:type:`record`       A pool specification.
+:bro:type:`Cluster::RoundRobinTable`: :bro:type:`table` 
+======================================================= ===========================================================
+
+Functions
+#########
+====================================================== ======================================================================
+:bro:id:`Cluster::hrw_topic`: :bro:type:`function`     Retrieve the topic associated with the node mapped via Rendezvous hash
+                                                       of an arbitrary key.
+:bro:id:`Cluster::register_pool`: :bro:type:`function` Registers and initializes a pool.
+:bro:id:`Cluster::rr_log_topic`: :bro:type:`function`  Distributes log message topics among logger nodes via round-robin.
+:bro:id:`Cluster::rr_topic`: :bro:type:`function`      Retrieve the topic associated with the node in a round-robin fashion.
+====================================================== ======================================================================
+
+
+Detailed Interface
+~~~~~~~~~~~~~~~~~~
+State Variables
+###############
+.. bro:id:: Cluster::logger_pool
+
+   :Type: :bro:type:`Cluster::Pool`
+   :Default:
+
+   ::
+
+      {
+         spec=[topic="", node_type=Cluster::PROXY, max_nodes=<uninitialized>, exclusive=F]
+         nodes={
+
+         }
+         node_list=[]
+         hrw_pool=[sites={
+
+         }]
+         rr_key_seq={
+
+         }
+         alive_count=0
+      }
+
+   A pool containing all the logger nodes of a cluster.
+   The pool's node membership/availability is automatically
+   maintained by the cluster framework.
+
+.. bro:id:: Cluster::logger_pool_spec
+
+   :Type: :bro:type:`Cluster::PoolSpec`
+   :Attributes: :bro:attr:`&redef`
+   :Default:
+
+   ::
+
+      {
+         topic="bro/cluster/pool/logger"
+         node_type=Cluster::LOGGER
+         max_nodes=<uninitialized>
+         exclusive=F
+      }
+
+   The specification for :bro:see:`Cluster::logger_pool`.
+
+.. bro:id:: Cluster::proxy_pool
+
+   :Type: :bro:type:`Cluster::Pool`
+   :Default:
+
+   ::
+
+      {
+         spec=[topic="", node_type=Cluster::PROXY, max_nodes=<uninitialized>, exclusive=F]
+         nodes={
+
+         }
+         node_list=[]
+         hrw_pool=[sites={
+
+         }]
+         rr_key_seq={
+
+         }
+         alive_count=0
+      }
+
+   A pool containing all the proxy nodes of a cluster.
+   The pool's node membership/availability is automatically
+   maintained by the cluster framework.
+
+.. bro:id:: Cluster::proxy_pool_spec
+
+   :Type: :bro:type:`Cluster::PoolSpec`
+   :Attributes: :bro:attr:`&redef`
+   :Default:
+
+   ::
+
+      {
+         topic="bro/cluster/pool/proxy"
+         node_type=Cluster::PROXY
+         max_nodes=<uninitialized>
+         exclusive=F
+      }
+
+   The specification for :bro:see:`Cluster::proxy_pool`.
+
+.. bro:id:: Cluster::worker_pool
+
+   :Type: :bro:type:`Cluster::Pool`
+   :Default:
+
+   ::
+
+      {
+         spec=[topic="", node_type=Cluster::PROXY, max_nodes=<uninitialized>, exclusive=F]
+         nodes={
+
+         }
+         node_list=[]
+         hrw_pool=[sites={
+
+         }]
+         rr_key_seq={
+
+         }
+         alive_count=0
+      }
+
+   A pool containing all the worker nodes of a cluster.
+   The pool's node membership/availability is automatically
+   maintained by the cluster framework.
+
+.. bro:id:: Cluster::worker_pool_spec
+
+   :Type: :bro:type:`Cluster::PoolSpec`
+   :Attributes: :bro:attr:`&redef`
+   :Default:
+
+   ::
+
+      {
+         topic="bro/cluster/pool/worker"
+         node_type=Cluster::WORKER
+         max_nodes=<uninitialized>
+         exclusive=F
+      }
+
+   The specification for :bro:see:`Cluster::worker_pool`.
+
+Types
+#####
+.. bro:type:: Cluster::PoolNode
+
+   :Type: :bro:type:`record`
+
+      name: :bro:type:`string`
+         The node name (e.g. "manager").
+
+      alias: :bro:type:`string`
+         An alias of *name* used to prevent hashing collisions when creating
+         *site_id*.
+
+      site_id: :bro:type:`count`
+         A 32-bit unique identifier for the pool node, derived from name/alias.
+
+      alive: :bro:type:`bool` :bro:attr:`&default` = ``F`` :bro:attr:`&optional`
+         Whether the node is currently alive and can receive work.
+
+   Store state of a cluster within the context of a work pool.
+
+.. bro:type:: Cluster::PoolNodeTable
+
+   :Type: :bro:type:`table` [:bro:type:`string`] of :bro:type:`Cluster::PoolNode`
+
+
+.. bro:type:: Cluster::PoolSpec
+
+   :Type: :bro:type:`record`
+
+      topic: :bro:type:`string` :bro:attr:`&default` = ``""`` :bro:attr:`&optional`
+         A topic string that can be used to reach all nodes within a pool.
+
+      node_type: :bro:type:`Cluster::NodeType` :bro:attr:`&default` = ``Cluster::PROXY`` :bro:attr:`&optional`
+         The type of nodes that are contained within the pool.
+
+      max_nodes: :bro:type:`count` :bro:attr:`&optional`
+         The maximum number of nodes that may belong to the pool.
+         If not set, then all available nodes will be added to the pool,
+         else the cluster framework will automatically limit the pool
+         membership according to the threshhold.
+
+      exclusive: :bro:type:`bool` :bro:attr:`&default` = ``F`` :bro:attr:`&optional`
+         Whether the pool requires exclusive access to nodes.  If true,
+         then *max_nodes* nodes will not be assigned to any other pool.
+         When using this flag, *max_nodes* must also be set.
+
+   A pool specification.
+
+.. bro:type:: Cluster::RoundRobinTable
+
+   :Type: :bro:type:`table` [:bro:type:`string`] of :bro:type:`int`
+
+
+Functions
+#########
+.. bro:id:: Cluster::hrw_topic
+
+   :Type: :bro:type:`function` (pool: :bro:type:`Cluster::Pool`, key: :bro:type:`any`) : :bro:type:`string`
+
+   Retrieve the topic associated with the node mapped via Rendezvous hash
+   of an arbitrary key.
+   
+
+   :pool: the pool of nodes to consider.
+   
+
+   :key: data used for input to the hashing function that will uniformly
+        distribute keys among available nodes.
+   
+
+   :returns: a topic string associated with a cluster node that is alive
+            or an empty string if nothing is alive.
+
+.. bro:id:: Cluster::register_pool
+
+   :Type: :bro:type:`function` (spec: :bro:type:`Cluster::PoolSpec`) : :bro:type:`Cluster::Pool`
+
+   Registers and initializes a pool.
+
+.. bro:id:: Cluster::rr_log_topic
+
+   :Type: :bro:type:`function` (id: :bro:type:`Log::ID`, path: :bro:type:`string`) : :bro:type:`string`
+
+   Distributes log message topics among logger nodes via round-robin.
+   This will be automatically assigned to :bro:see:`Broker::log_topic`
+   if :bro:see:`Cluster::enable_round_robin_logging` is enabled.
+   If no logger nodes are active, then this will return the value
+   of :bro:see:`Broker::default_log_topic`.
+
+.. bro:id:: Cluster::rr_topic
+
+   :Type: :bro:type:`function` (pool: :bro:type:`Cluster::Pool`, key: :bro:type:`string` :bro:attr:`&default` = ``""`` :bro:attr:`&optional`) : :bro:type:`string`
+
+   Retrieve the topic associated with the node in a round-robin fashion.
+   
+
+   :pool: the pool of nodes to consider.
+   
+
+   :key: an arbitrary string to identify the purpose for which you're
+        requesting the topic.  e.g. consider using a name-spaced key
+        like "Intel::cluster_rr_key" if you need to guarantee that
+        a group of messages get distributed in a well-defined pattern
+        without other messages being interleaved within the round-robin.
+        Usually sharing the default key is fine for load-balancing
+        purposes.
+   
+
+   :returns: a topic string associated with a cluster node that is alive,
+            or an empty string if nothing is alive.
+
+

doc/scripts/base/frameworks/config/__load__.bro.rst

@@ -0,0 +1,14 @@
+:tocdepth: 3
+
+base/frameworks/config/__load__.bro
+===================================
+
+
+:Imports: :doc:`base/frameworks/config/input.bro </scripts/base/frameworks/config/input.bro>`, :doc:`base/frameworks/config/main.bro </scripts/base/frameworks/config/main.bro>`, :doc:`base/frameworks/config/weird.bro </scripts/base/frameworks/config/weird.bro>`
+
+Summary
+~~~~~~~
+
+Detailed Interface
+~~~~~~~~~~~~~~~~~~
+

doc/scripts/base/frameworks/config/index.rst

@@ -0,0 +1,25 @@
+:orphan:
+
+Package: base/frameworks/config
+===============================
+
+The configuration framework provides a way to change the Bro configuration
+in "option" values at run-time.
+
+:doc:`/scripts/base/frameworks/config/__load__.bro`
+
+
+:doc:`/scripts/base/frameworks/config/main.bro`
+
+   The configuration framework provides a way to change Bro options
+   (as specified by the "option" keyword) at runtime. It also logs runtime
+   changes to options to config.log.
+
+:doc:`/scripts/base/frameworks/config/input.bro`
+
+   File input for the configuration framework using the input framework.
+
+:doc:`/scripts/base/frameworks/config/weird.bro`
+
+   This script sets up the config framework change handlers for weirds.
+

doc/scripts/base/frameworks/config/input.bro.rst

@@ -0,0 +1,54 @@
+:tocdepth: 3
+
+base/frameworks/config/input.bro
+================================
+.. bro:namespace:: Config
+
+File input for the configuration framework using the input framework.
+
+:Namespace: Config
+:Imports: :doc:`base/frameworks/cluster </scripts/base/frameworks/cluster/index>`, :doc:`base/frameworks/config/main.bro </scripts/base/frameworks/config/main.bro>`
+
+Summary
+~~~~~~~
+Redefinable Options
+###################
+================================================================== ===============================================
+:bro:id:`Config::config_files`: :bro:type:`set` :bro:attr:`&redef` Configuration files that will be read off disk.
+================================================================== ===============================================
+
+Functions
+#########
+=================================================== ===================================================================
+:bro:id:`Config::read_config`: :bro:type:`function` Read specified configuration file and apply values; updates to file
+                                                    are not tracked.
+=================================================== ===================================================================
+
+
+Detailed Interface
+~~~~~~~~~~~~~~~~~~
+Redefinable Options
+###################
+.. bro:id:: Config::config_files
+
+   :Type: :bro:type:`set` [:bro:type:`string`]
+   :Attributes: :bro:attr:`&redef`
+   :Default: ``{}``
+
+   Configuration files that will be read off disk. Files are reread
+   every time they are updated so updates should be atomic with "mv"
+   instead of writing the file in place.
+   
+   If the same configuration option is defined in several files with
+   different values, behavior is unspecified.
+
+Functions
+#########
+.. bro:id:: Config::read_config
+
+   :Type: :bro:type:`function` (filename: :bro:type:`string`) : :bro:type:`void`
+
+   Read specified configuration file and apply values; updates to file
+   are not tracked.
+
+

doc/scripts/base/frameworks/config/main.bro.rst

@@ -0,0 +1,101 @@
+:tocdepth: 3
+
+base/frameworks/config/main.bro
+===============================
+.. bro:namespace:: Config
+
+The configuration framework provides a way to change Bro options
+(as specified by the "option" keyword) at runtime. It also logs runtime
+changes to options to config.log.
+
+:Namespace: Config
+:Imports: :doc:`base/frameworks/cluster </scripts/base/frameworks/cluster/index>`
+
+Summary
+~~~~~~~
+Types
+#####
+============================================ ==================================
+:bro:type:`Config::Info`: :bro:type:`record` Represents the data in config.log.
+============================================ ==================================
+
+Redefinitions
+#############
+===================================== =====================================
+:bro:type:`Log::ID`: :bro:type:`enum` The config logging stream identifier.
+===================================== =====================================
+
+Events
+######
+=============================================== ================================================================
+:bro:id:`Config::log_config`: :bro:type:`event` Event that can be handled to access the :bro:type:`Config::Info`
+                                                record as it is sent on to the logging framework.
+=============================================== ================================================================
+
+Functions
+#########
+================================================= ==================================================================
+:bro:id:`Config::set_value`: :bro:type:`function` This function is the config framework layer around the lower-level
+                                                  :bro:see:`Option::set` call.
+================================================= ==================================================================
+
+
+Detailed Interface
+~~~~~~~~~~~~~~~~~~
+Types
+#####
+.. bro:type:: Config::Info
+
+   :Type: :bro:type:`record`
+
+      ts: :bro:type:`time` :bro:attr:`&log`
+         Timestamp at which the configuration change occured.
+
+      id: :bro:type:`string` :bro:attr:`&log`
+         ID of the value that was changed.
+
+      old_value: :bro:type:`string` :bro:attr:`&log`
+         Value before the change.
+
+      new_value: :bro:type:`string` :bro:attr:`&log`
+         Value after the change.
+
+      location: :bro:type:`string` :bro:attr:`&optional` :bro:attr:`&log`
+         Optional location that triggered the change.
+
+   Represents the data in config.log.
+
+Events
+######
+.. bro:id:: Config::log_config
+
+   :Type: :bro:type:`event` (rec: :bro:type:`Config::Info`)
+
+   Event that can be handled to access the :bro:type:`Config::Info`
+   record as it is sent on to the logging framework.
+
+Functions
+#########
+.. bro:id:: Config::set_value
+
+   :Type: :bro:type:`function` (ID: :bro:type:`string`, val: :bro:type:`any`, location: :bro:type:`string` :bro:attr:`&default` = ``""`` :bro:attr:`&optional` :bro:attr:`&optional`) : :bro:type:`bool`
+
+   This function is the config framework layer around the lower-level
+   :bro:see:`Option::set` call. Config::set_value will set the configuration
+   value for all nodes in the cluster, no matter where it was called. Note
+   that :bro:see:`Option::set` does not distribute configuration changes
+   to other nodes.
+   
+
+   :ID: The ID of the option to update.
+   
+
+   :val: The new value of the option.
+   
+
+   :location: Optional parameter detailing where this change originated from.
+   
+
+   :returns: true on success, false when an error occurs.
+
+

doc/scripts/base/frameworks/config/weird.bro.rst

@@ -0,0 +1,17 @@
+:tocdepth: 3
+
+base/frameworks/config/weird.bro
+================================
+.. bro:namespace:: Config
+
+This script sets up the config framework change handlers for weirds.
+
+:Namespace: Config
+:Imports: :doc:`base/frameworks/config/main.bro </scripts/base/frameworks/config/main.bro>`
+
+Summary
+~~~~~~~
+
+Detailed Interface
+~~~~~~~~~~~~~~~~~~
+

doc/scripts/base/frameworks/control/__load__.bro.rst

@@ -0,0 +1,14 @@
+:tocdepth: 3
+
+base/frameworks/control/__load__.bro
+====================================
+
+
+:Imports: :doc:`base/frameworks/control/main.bro </scripts/base/frameworks/control/main.bro>`
+
+Summary
+~~~~~~~
+
+Detailed Interface
+~~~~~~~~~~~~~~~~~~
+

doc/scripts/base/frameworks/control/index.rst

@@ -0,0 +1,18 @@
+:orphan:
+
+Package: base/frameworks/control
+================================
+
+The control framework provides the foundation for providing "commands"
+that can be taken remotely at runtime to modify a running Bro instance
+or collect information from the running instance.
+
+:doc:`/scripts/base/frameworks/control/__load__.bro`
+
+
+:doc:`/scripts/base/frameworks/control/main.bro`
+
+   The control framework provides the foundation for providing "commands"
+   that can be taken remotely at runtime to modify a running Bro instance
+   or collect information from the running instance.
+

doc/scripts/base/frameworks/control/main.bro.rst

@@ -0,0 +1,218 @@
+:tocdepth: 3
+
+base/frameworks/control/main.bro
+================================
+.. bro:namespace:: Control
+
+The control framework provides the foundation for providing "commands"
+that can be taken remotely at runtime to modify a running Bro instance
+or collect information from the running instance.
+
+:Namespace: Control
+
+Summary
+~~~~~~~
+Redefinable Options
+###################
+========================================================================= ================================================================
+:bro:id:`Control::arg`: :bro:type:`string` :bro:attr:`&redef`             This can be used by commands that take an argument.
+:bro:id:`Control::cmd`: :bro:type:`string` :bro:attr:`&redef`             The command that is being done.
+:bro:id:`Control::commands`: :bro:type:`set` :bro:attr:`&redef`           The commands that can currently be given on the command line for
+                                                                          remote control.
+:bro:id:`Control::controllee_listen`: :bro:type:`bool` :bro:attr:`&redef` Whether the controllee should call :bro:see:`Broker::listen`.
+:bro:id:`Control::host`: :bro:type:`addr` :bro:attr:`&redef`              The address of the host that will be controlled.
+:bro:id:`Control::host_port`: :bro:type:`port` :bro:attr:`&redef`         The port of the host that will be controlled.
+:bro:id:`Control::zone_id`: :bro:type:`string` :bro:attr:`&redef`         If :bro:id:`Control::host` is a non-global IPv6 address and
+                                                                          requires a specific :rfc:`4007` ``zone_id``, it can be set here.
+========================================================================= ================================================================
+
+Constants
+#########
+=================================================== =================================================================
+:bro:id:`Control::ignore_ids`: :bro:type:`set`      Variable IDs that are to be ignored by the update process.
+:bro:id:`Control::topic_prefix`: :bro:type:`string` The topic prefix used for exchanging control messages via Broker.
+=================================================== =================================================================
+
+Events
+######
+=================================================================== ====================================================================
+:bro:id:`Control::configuration_update`: :bro:type:`event`          This event is a wrapper and alias for the
+                                                                    :bro:id:`Control::configuration_update_request` event.
+:bro:id:`Control::configuration_update_request`: :bro:type:`event`  Inform the remote Bro instance that it's configuration may have been
+                                                                    updated.
+:bro:id:`Control::configuration_update_response`: :bro:type:`event` Message in response to a configuration update request.
+:bro:id:`Control::id_value_request`: :bro:type:`event`              Event for requesting the value of an ID (a variable).
+:bro:id:`Control::id_value_response`: :bro:type:`event`             Event for returning the value of an ID after an
+                                                                    :bro:id:`Control::id_value_request` event.
+:bro:id:`Control::net_stats_request`: :bro:type:`event`             Requests the current net_stats.
+:bro:id:`Control::net_stats_response`: :bro:type:`event`            Returns the current net_stats.
+:bro:id:`Control::peer_status_request`: :bro:type:`event`           Requests the current communication status.
+:bro:id:`Control::peer_status_response`: :bro:type:`event`          Returns the current communication status.
+:bro:id:`Control::shutdown_request`: :bro:type:`event`              Requests that the Bro instance begins shutting down.
+:bro:id:`Control::shutdown_response`: :bro:type:`event`             Message in response to a shutdown request.
+=================================================================== ====================================================================
+
+
+Detailed Interface
+~~~~~~~~~~~~~~~~~~
+Redefinable Options
+###################
+.. bro:id:: Control::arg
+
+   :Type: :bro:type:`string`
+   :Attributes: :bro:attr:`&redef`
+   :Default: ``""``
+
+   This can be used by commands that take an argument.
+
+.. bro:id:: Control::cmd
+
+   :Type: :bro:type:`string`
+   :Attributes: :bro:attr:`&redef`
+   :Default: ``""``
+
+   The command that is being done.  It's typically set on the
+   command line.
+
+.. bro:id:: Control::commands
+
+   :Type: :bro:type:`set` [:bro:type:`string`]
+   :Attributes: :bro:attr:`&redef`
+   :Default:
+
+   ::
+
+      {
+         "shutdown",
+         "id_value",
+         "net_stats",
+         "peer_status",
+         "configuration_update"
+      }
+
+   The commands that can currently be given on the command line for
+   remote control.
+
+.. bro:id:: Control::controllee_listen
+
+   :Type: :bro:type:`bool`
+   :Attributes: :bro:attr:`&redef`
+   :Default: ``T``
+
+   Whether the controllee should call :bro:see:`Broker::listen`.
+   In a cluster, this isn't needed since the setup process calls it.
+
+.. bro:id:: Control::host
+
+   :Type: :bro:type:`addr`
+   :Attributes: :bro:attr:`&redef`
+   :Default: ``0.0.0.0``
+
+   The address of the host that will be controlled.
+
+.. bro:id:: Control::host_port
+
+   :Type: :bro:type:`port`
+   :Attributes: :bro:attr:`&redef`
+   :Default: ``0/tcp``
+
+   The port of the host that will be controlled.
+
+.. bro:id:: Control::zone_id
+
+   :Type: :bro:type:`string`
+   :Attributes: :bro:attr:`&redef`
+   :Default: ``""``
+
+   If :bro:id:`Control::host` is a non-global IPv6 address and
+   requires a specific :rfc:`4007` ``zone_id``, it can be set here.
+
+Constants
+#########
+.. bro:id:: Control::ignore_ids
+
+   :Type: :bro:type:`set` [:bro:type:`string`]
+   :Default: ``{}``
+
+   Variable IDs that are to be ignored by the update process.
+
+.. bro:id:: Control::topic_prefix
+
+   :Type: :bro:type:`string`
+   :Default: ``"bro/control"``
+
+   The topic prefix used for exchanging control messages via Broker.
+
+Events
+######
+.. bro:id:: Control::configuration_update
+
+   :Type: :bro:type:`event` ()
+
+   This event is a wrapper and alias for the
+   :bro:id:`Control::configuration_update_request` event.
+   This event is also a primary hooking point for the control framework.
+
+.. bro:id:: Control::configuration_update_request
+
+   :Type: :bro:type:`event` ()
+
+   Inform the remote Bro instance that it's configuration may have been
+   updated.
+
+.. bro:id:: Control::configuration_update_response
+
+   :Type: :bro:type:`event` ()
+
+   Message in response to a configuration update request.
+
+.. bro:id:: Control::id_value_request
+
+   :Type: :bro:type:`event` (id: :bro:type:`string`)
+
+   Event for requesting the value of an ID (a variable).
+
+.. bro:id:: Control::id_value_response
+
+   :Type: :bro:type:`event` (id: :bro:type:`string`, val: :bro:type:`string`)
+
+   Event for returning the value of an ID after an
+   :bro:id:`Control::id_value_request` event.
+
+.. bro:id:: Control::net_stats_request
+
+   :Type: :bro:type:`event` ()
+
+   Requests the current net_stats.
+
+.. bro:id:: Control::net_stats_response
+
+   :Type: :bro:type:`event` (s: :bro:type:`string`)
+
+   Returns the current net_stats.
+
+.. bro:id:: Control::peer_status_request
+
+   :Type: :bro:type:`event` ()
+
+   Requests the current communication status.
+
+.. bro:id:: Control::peer_status_response
+
+   :Type: :bro:type:`event` (s: :bro:type:`string`)
+
+   Returns the current communication status.
+
+.. bro:id:: Control::shutdown_request
+
+   :Type: :bro:type:`event` ()
+
+   Requests that the Bro instance begins shutting down.
+
+.. bro:id:: Control::shutdown_response
+
+   :Type: :bro:type:`event` ()
+
+   Message in response to a shutdown request.
+
+

doc/scripts/base/frameworks/dpd/__load__.bro.rst

@@ -0,0 +1,14 @@
+:tocdepth: 3
+
+base/frameworks/dpd/__load__.bro
+================================
+
+
+:Imports: :doc:`base/frameworks/dpd/main.bro </scripts/base/frameworks/dpd/main.bro>`
+
+Summary
+~~~~~~~
+
+Detailed Interface
+~~~~~~~~~~~~~~~~~~
+

doc/scripts/base/frameworks/dpd/index.rst

@@ -0,0 +1,16 @@
+:orphan:
+
+Package: base/frameworks/dpd
+============================
+
+The DPD (dynamic protocol detection) activates port-independent protocol
+detection and selectively disables analyzers if protocol violations occur.
+
+:doc:`/scripts/base/frameworks/dpd/__load__.bro`
+
+
+:doc:`/scripts/base/frameworks/dpd/main.bro`
+
+   Activates port-independent protocol detection and selectively disables
+   analyzers if protocol violations occur.
+

doc/scripts/base/frameworks/dpd/main.bro.rst

@@ -0,0 +1,99 @@
+:tocdepth: 3
+
+base/frameworks/dpd/main.bro
+============================
+.. bro:namespace:: DPD
+
+Activates port-independent protocol detection and selectively disables
+analyzers if protocol violations occur.
+
+:Namespace: DPD
+
+Summary
+~~~~~~~
+Runtime Options
+###############
+============================================================================ ===============================================================
+:bro:id:`DPD::ignore_violations`: :bro:type:`set` :bro:attr:`&redef`         Analyzers which you don't want to throw 
+:bro:id:`DPD::ignore_violations_after`: :bro:type:`count` :bro:attr:`&redef` Ignore violations which go this many bytes into the connection.
+============================================================================ ===============================================================
+
+Types
+#####
+========================================= ======================================================================
+:bro:type:`DPD::Info`: :bro:type:`record` The record type defining the columns to log in the DPD logging stream.
+========================================= ======================================================================
+
+Redefinitions
+#############
+========================================== ======================================
+:bro:type:`Log::ID`: :bro:type:`enum`      Add the DPD logging stream identifier.
+:bro:type:`connection`: :bro:type:`record` 
+========================================== ======================================
+
+
+Detailed Interface
+~~~~~~~~~~~~~~~~~~
+Runtime Options
+###############
+.. bro:id:: DPD::ignore_violations
+
+   :Type: :bro:type:`set` [:bro:type:`Analyzer::Tag`]
+   :Attributes: :bro:attr:`&redef`
+   :Default:
+
+   ::
+
+      {
+         Analyzer::ANALYZER_DCE_RPC,
+         Analyzer::ANALYZER_NTLM
+      }
+
+   Analyzers which you don't want to throw 
+
+.. bro:id:: DPD::ignore_violations_after
+
+   :Type: :bro:type:`count`
+   :Attributes: :bro:attr:`&redef`
+   :Default: ``10240``
+
+   Ignore violations which go this many bytes into the connection.
+   Set to 0 to never ignore protocol violations.
+
+Types
+#####
+.. bro:type:: DPD::Info
+
+   :Type: :bro:type:`record`
+
+      ts: :bro:type:`time` :bro:attr:`&log`
+         Timestamp for when protocol analysis failed.
+
+      uid: :bro:type:`string` :bro:attr:`&log`
+         Connection unique ID.
+
+      id: :bro:type:`conn_id` :bro:attr:`&log`
+         Connection ID containing the 4-tuple which identifies endpoints.
+
+      proto: :bro:type:`transport_proto` :bro:attr:`&log`
+         Transport protocol for the violation.
+
+      analyzer: :bro:type:`string` :bro:attr:`&log`
+         The analyzer that generated the violation.
+
+      failure_reason: :bro:type:`string` :bro:attr:`&log`
+         The textual reason for the analysis failure.
+
+      disabled_aids: :bro:type:`set` [:bro:type:`count`]
+         Disabled analyzer IDs.  This is only for internal tracking
+         so as to not attempt to disable analyzers multiple times.
+
+      packet_segment: :bro:type:`string` :bro:attr:`&optional` :bro:attr:`&log`
+         (present if :doc:`/scripts/policy/frameworks/dpd/packet-segment-logging.bro` is loaded)
+
+         A chunk of the payload that most likely resulted in the
+         protocol violation.
+
+   The record type defining the columns to log in the DPD logging stream.
+
+

doc/scripts/base/frameworks/files/__load__.bro.rst

@@ -0,0 +1,14 @@
+:tocdepth: 3
+
+base/frameworks/files/__load__.bro
+==================================
+
+
+:Imports: :doc:`base/frameworks/files/magic </scripts/base/frameworks/files/magic/index>`, :doc:`base/frameworks/files/main.bro </scripts/base/frameworks/files/main.bro>`
+
+Summary
+~~~~~~~
+
+Detailed Interface
+~~~~~~~~~~~~~~~~~~
+

doc/scripts/base/frameworks/files/index.rst

@@ -0,0 +1,20 @@
+:orphan:
+
+Package: base/frameworks/files
+==============================
+
+The file analysis framework provides an interface for driving the analysis
+of files, possibly independent of any network protocol over which they're
+transported.
+
+:doc:`/scripts/base/frameworks/files/__load__.bro`
+
+
+:doc:`/scripts/base/frameworks/files/main.bro`
+
+   An interface for driving the analysis of files, possibly independent of
+   any network protocol over which they're transported.
+
+:doc:`/scripts/base/frameworks/files/magic/__load__.bro`
+
+

doc/scripts/base/frameworks/files/magic/__load__.bro.rst

@@ -0,0 +1,13 @@
+:tocdepth: 3
+
+base/frameworks/files/magic/__load__.bro
+========================================
+
+
+
+Summary
+~~~~~~~
+
+Detailed Interface
+~~~~~~~~~~~~~~~~~~
+

doc/scripts/base/frameworks/files/magic/index.rst

@@ -0,0 +1,9 @@
+:orphan:
+
+Package: base/frameworks/files/magic
+====================================
+
+
+:doc:`/scripts/base/frameworks/files/magic/__load__.bro`
+
+

doc/scripts/base/frameworks/files/main.bro.rst

@@ -0,0 +1,580 @@
+:tocdepth: 3
+
+base/frameworks/files/main.bro
+==============================
+.. bro:namespace:: Files
+
+An interface for driving the analysis of files, possibly independent of
+any network protocol over which they're transported.
+
+:Namespace: Files
+:Imports: :doc:`base/bif/file_analysis.bif.bro </scripts/base/bif/file_analysis.bif.bro>`, :doc:`base/frameworks/analyzer </scripts/base/frameworks/analyzer/index>`, :doc:`base/frameworks/logging </scripts/base/frameworks/logging/index>`, :doc:`base/utils/site.bro </scripts/base/utils/site.bro>`
+
+Summary
+~~~~~~~
+Runtime Options
+###############
+======================================================================== ========================================
+:bro:id:`Files::enable_reassembler`: :bro:type:`bool` :bro:attr:`&redef` The default setting for file reassembly.
+======================================================================== ========================================
+
+Redefinable Options
+###################
+======================================================================================== ================================================================
+:bro:id:`Files::analyze_by_mime_type_automatically`: :bro:type:`bool` :bro:attr:`&redef` Decide if you want to automatically attached analyzers to 
+                                                                                         files based on the detected mime type of the file.
+:bro:id:`Files::disable`: :bro:type:`table` :bro:attr:`&redef`                           A table that can be used to disable file analysis completely for
+                                                                                         any files transferred over given network protocol analyzers.
+:bro:id:`Files::reassembly_buffer_size`: :bro:type:`count` :bro:attr:`&redef`            The default per-file reassembly buffer size.
+:bro:id:`Files::salt`: :bro:type:`string` :bro:attr:`&redef`                             The salt concatenated to unique file handle strings generated by
+                                                                                         :bro:see:`get_file_handle` before hashing them in to a file id
+                                                                                         (the *id* field of :bro:see:`fa_file`).
+======================================================================================== ================================================================
+
+Types
+#####
+====================================================================== ==============================================================
+:bro:type:`Files::AnalyzerArgs`: :bro:type:`record` :bro:attr:`&redef` A structure which parameterizes a type of file analysis.
+:bro:type:`Files::Info`: :bro:type:`record` :bro:attr:`&redef`         Contains all metadata related to the analysis of a given file.
+:bro:type:`Files::ProtoRegistration`: :bro:type:`record`               
+====================================================================== ==============================================================
+
+Redefinitions
+#############
+========================================================== =
+:bro:type:`Log::ID`: :bro:type:`enum`                      
+:bro:type:`fa_file`: :bro:type:`record` :bro:attr:`&redef` 
+========================================================== =
+
+Events
+######
+============================================= ====================================================================
+:bro:id:`Files::log_files`: :bro:type:`event` Event that can be handled to access the Info record as it is sent on
+                                              to the logging framework.
+============================================= ====================================================================
+
+Functions
+#########
+===================================================================== =============================================================================
+:bro:id:`Files::add_analyzer`: :bro:type:`function`                   Adds an analyzer to the analysis of a given file.
+:bro:id:`Files::all_registered_mime_types`: :bro:type:`function`      Returns a table of all MIME-type-to-analyzer mappings currently registered.
+:bro:id:`Files::analyzer_name`: :bro:type:`function`                  Translates a file analyzer enum value to a string with the
+                                                                      analyzer's name.
+:bro:id:`Files::describe`: :bro:type:`function`                       Provides a text description regarding metadata of the file.
+:bro:id:`Files::disable_reassembly`: :bro:type:`function`             Disables the file reassembler on this file.
+:bro:id:`Files::enable_reassembly`: :bro:type:`function`              Allows the file reassembler to be used if it's necessary because the
+                                                                      file is transferred out of order.
+:bro:id:`Files::file_exists`: :bro:type:`function`                    Lookup to see if a particular file id exists and is still valid.
+:bro:id:`Files::lookup_file`: :bro:type:`function`                    Lookup an :bro:see:`fa_file` record with the file id.
+:bro:id:`Files::register_analyzer_add_callback`: :bro:type:`function` Register a callback for file analyzers to use if they need to do some
+                                                                      manipulation when they are being added to a file before the core code
+                                                                      takes over.
+:bro:id:`Files::register_for_mime_type`: :bro:type:`function`         Registers a MIME type for an analyzer.
+:bro:id:`Files::register_for_mime_types`: :bro:type:`function`        Registers a set of MIME types for an analyzer.
+:bro:id:`Files::register_protocol`: :bro:type:`function`              Register callbacks for protocols that work with the Files framework.
+:bro:id:`Files::registered_mime_types`: :bro:type:`function`          Returns a set of all MIME types currently registered for a specific analyzer.
+:bro:id:`Files::remove_analyzer`: :bro:type:`function`                Removes an analyzer from the analysis of a given file.
+:bro:id:`Files::set_reassembly_buffer_size`: :bro:type:`function`     Set the maximum size the reassembly buffer is allowed to grow
+                                                                      for the given file.
+:bro:id:`Files::set_timeout_interval`: :bro:type:`function`           Sets the *timeout_interval* field of :bro:see:`fa_file`, which is
+                                                                      used to determine the length of inactivity that is allowed for a file
+                                                                      before internal state related to it is cleaned up.
+:bro:id:`Files::stop`: :bro:type:`function`                           Stops/ignores any further analysis of a given file.
+===================================================================== =============================================================================
+
+
+Detailed Interface
+~~~~~~~~~~~~~~~~~~
+Runtime Options
+###############
+.. bro:id:: Files::enable_reassembler
+
+   :Type: :bro:type:`bool`
+   :Attributes: :bro:attr:`&redef`
+   :Default: ``T``
+
+   The default setting for file reassembly.
+
+Redefinable Options
+###################
+.. bro:id:: Files::analyze_by_mime_type_automatically
+
+   :Type: :bro:type:`bool`
+   :Attributes: :bro:attr:`&redef`
+   :Default: ``T``
+
+   Decide if you want to automatically attached analyzers to 
+   files based on the detected mime type of the file.
+
+.. bro:id:: Files::disable
+
+   :Type: :bro:type:`table` [:bro:type:`Files::Tag`] of :bro:type:`bool`
+   :Attributes: :bro:attr:`&redef`
+   :Default: ``{}``
+
+   A table that can be used to disable file analysis completely for
+   any files transferred over given network protocol analyzers.
+
+.. bro:id:: Files::reassembly_buffer_size
+
+   :Type: :bro:type:`count`
+   :Attributes: :bro:attr:`&redef`
+   :Default: ``524288``
+
+   The default per-file reassembly buffer size.
+
+.. bro:id:: Files::salt
+
+   :Type: :bro:type:`string`
+   :Attributes: :bro:attr:`&redef`
+   :Default: ``"I recommend changing this."``
+
+   The salt concatenated to unique file handle strings generated by
+   :bro:see:`get_file_handle` before hashing them in to a file id
+   (the *id* field of :bro:see:`fa_file`).
+   Provided to help mitigate the possibility of manipulating parts of
+   network connections that factor in to the file handle in order to
+   generate two handles that would hash to the same file id.
+
+Types
+#####
+.. bro:type:: Files::AnalyzerArgs
+
+   :Type: :bro:type:`record`
+
+      chunk_event: :bro:type:`event` (f: :bro:type:`fa_file`, data: :bro:type:`string`, off: :bro:type:`count`) :bro:attr:`&optional`
+         An event which will be generated for all new file contents,
+         chunk-wise.  Used when *tag* (in the
+         :bro:see:`Files::add_analyzer` function) is
+         :bro:see:`Files::ANALYZER_DATA_EVENT`.
+
+      stream_event: :bro:type:`event` (f: :bro:type:`fa_file`, data: :bro:type:`string`) :bro:attr:`&optional`
+         An event which will be generated for all new file contents,
+         stream-wise.  Used when *tag* is
+         :bro:see:`Files::ANALYZER_DATA_EVENT`.
+
+      extract_filename: :bro:type:`string` :bro:attr:`&optional`
+         (present if :doc:`/scripts/base/files/extract/main.bro` is loaded)
+
+         The local filename to which to write an extracted file.
+         This field is used in the core by the extraction plugin
+         to know where to write the file to.  If not specified, then
+         a filename in the format "extract-<source>-<id>" is
+         automatically assigned (using the *source* and *id*
+         fields of :bro:see:`fa_file`).
+
+      extract_limit: :bro:type:`count` :bro:attr:`&default` = :bro:see:`FileExtract::default_limit` :bro:attr:`&optional`
+         (present if :doc:`/scripts/base/files/extract/main.bro` is loaded)
+
+         The maximum allowed file size in bytes of *extract_filename*.
+         Once reached, a :bro:see:`file_extraction_limit` event is
+         raised and the analyzer will be removed unless
+         :bro:see:`FileExtract::set_limit` is called to increase the
+         limit.  A value of zero means "no limit".
+   :Attributes: :bro:attr:`&redef`
+
+   A structure which parameterizes a type of file analysis.
+
+.. bro:type:: Files::Info
+
+   :Type: :bro:type:`record`
+
+      ts: :bro:type:`time` :bro:attr:`&log`
+         The time when the file was first seen.
+
+      fuid: :bro:type:`string` :bro:attr:`&log`
+         An identifier associated with a single file.
+
+      tx_hosts: :bro:type:`set` [:bro:type:`addr`] :bro:attr:`&default` = ``{  }`` :bro:attr:`&optional` :bro:attr:`&log`
+         If this file was transferred over a network
+         connection this should show the host or hosts that
+         the data sourced from.
+
+      rx_hosts: :bro:type:`set` [:bro:type:`addr`] :bro:attr:`&default` = ``{  }`` :bro:attr:`&optional` :bro:attr:`&log`
+         If this file was transferred over a network
+         connection this should show the host or hosts that
+         the data traveled to.
+
+      conn_uids: :bro:type:`set` [:bro:type:`string`] :bro:attr:`&default` = ``{  }`` :bro:attr:`&optional` :bro:attr:`&log`
+         Connection UIDs over which the file was transferred.
+
+      source: :bro:type:`string` :bro:attr:`&log` :bro:attr:`&optional`
+         An identification of the source of the file data.  E.g. it
+         may be a network protocol over which it was transferred, or a
+         local file path which was read, or some other input source.
+
+      depth: :bro:type:`count` :bro:attr:`&default` = ``0`` :bro:attr:`&optional` :bro:attr:`&log`
+         A value to represent the depth of this file in relation
+         to its source.  In SMTP, it is the depth of the MIME
+         attachment on the message.  In HTTP, it is the depth of the
+         request within the TCP connection.
+
+      analyzers: :bro:type:`set` [:bro:type:`string`] :bro:attr:`&default` = ``{  }`` :bro:attr:`&optional` :bro:attr:`&log`
+         A set of analysis types done during the file analysis.
+
+      mime_type: :bro:type:`string` :bro:attr:`&log` :bro:attr:`&optional`
+         A mime type provided by the strongest file magic signature
+         match against the *bof_buffer* field of :bro:see:`fa_file`,
+         or in the cases where no buffering of the beginning of file
+         occurs, an initial guess of the mime type based on the first
+         data seen.
+
+      filename: :bro:type:`string` :bro:attr:`&log` :bro:attr:`&optional`
+         A filename for the file if one is available from the source
+         for the file.  These will frequently come from
+         "Content-Disposition" headers in network protocols.
+
+      duration: :bro:type:`interval` :bro:attr:`&log` :bro:attr:`&default` = ``0 secs`` :bro:attr:`&optional`
+         The duration the file was analyzed for.
+
+      local_orig: :bro:type:`bool` :bro:attr:`&log` :bro:attr:`&optional`
+         If the source of this file is a network connection, this field
+         indicates if the data originated from the local network or not as
+         determined by the configured :bro:see:`Site::local_nets`.
+
+      is_orig: :bro:type:`bool` :bro:attr:`&log` :bro:attr:`&optional`
+         If the source of this file is a network connection, this field
+         indicates if the file is being sent by the originator of the
+         connection or the responder.
+
+      seen_bytes: :bro:type:`count` :bro:attr:`&log` :bro:attr:`&default` = ``0`` :bro:attr:`&optional`
+         Number of bytes provided to the file analysis engine for the file.
+
+      total_bytes: :bro:type:`count` :bro:attr:`&log` :bro:attr:`&optional`
+         Total number of bytes that are supposed to comprise the full file.
+
+      missing_bytes: :bro:type:`count` :bro:attr:`&log` :bro:attr:`&default` = ``0`` :bro:attr:`&optional`
+         The number of bytes in the file stream that were completely missed
+         during the process of analysis e.g. due to dropped packets.
+
+      overflow_bytes: :bro:type:`count` :bro:attr:`&log` :bro:attr:`&default` = ``0`` :bro:attr:`&optional`
+         The number of bytes in the file stream that were not delivered to
+         stream file analyzers.  This could be overlapping bytes or 
+         bytes that couldn't be reassembled.
+
+      timedout: :bro:type:`bool` :bro:attr:`&log` :bro:attr:`&default` = ``F`` :bro:attr:`&optional`
+         Whether the file analysis timed out at least once for the file.
+
+      parent_fuid: :bro:type:`string` :bro:attr:`&log` :bro:attr:`&optional`
+         Identifier associated with a container file from which this one was
+         extracted as part of the file analysis.
+
+      md5: :bro:type:`string` :bro:attr:`&log` :bro:attr:`&optional`
+         (present if :doc:`/scripts/base/files/hash/main.bro` is loaded)
+
+         An MD5 digest of the file contents.
+
+      sha1: :bro:type:`string` :bro:attr:`&log` :bro:attr:`&optional`
+         (present if :doc:`/scripts/base/files/hash/main.bro` is loaded)
+
+         A SHA1 digest of the file contents.
+
+      sha256: :bro:type:`string` :bro:attr:`&log` :bro:attr:`&optional`
+         (present if :doc:`/scripts/base/files/hash/main.bro` is loaded)
+
+         A SHA256 digest of the file contents.
+
+      x509: :bro:type:`X509::Info` :bro:attr:`&optional`
+         (present if :doc:`/scripts/base/files/x509/main.bro` is loaded)
+
+         Information about X509 certificates. This is used to keep
+         certificate information until all events have been received.
+
+      extracted: :bro:type:`string` :bro:attr:`&optional` :bro:attr:`&log`
+         (present if :doc:`/scripts/base/files/extract/main.bro` is loaded)
+
+         Local filename of extracted file.
+
+      extracted_cutoff: :bro:type:`bool` :bro:attr:`&optional` :bro:attr:`&log`
+         (present if :doc:`/scripts/base/files/extract/main.bro` is loaded)
+
+         Set to true if the file being extracted was cut off
+         so the whole file was not logged.
+
+      extracted_size: :bro:type:`count` :bro:attr:`&optional` :bro:attr:`&log`
+         (present if :doc:`/scripts/base/files/extract/main.bro` is loaded)
+
+         The number of bytes extracted to disk.
+
+      entropy: :bro:type:`double` :bro:attr:`&log` :bro:attr:`&optional`
+         (present if :doc:`/scripts/policy/frameworks/files/entropy-test-all-files.bro` is loaded)
+
+         The information density of the contents of the file, 
+         expressed as a number of bits per character. 
+   :Attributes: :bro:attr:`&redef`
+
+   Contains all metadata related to the analysis of a given file.
+   For the most part, fields here are derived from ones of the same name
+   in :bro:see:`fa_file`.
+
+.. bro:type:: Files::ProtoRegistration
+
+   :Type: :bro:type:`record`
+
+      get_file_handle: :bro:type:`function` (c: :bro:type:`connection`, is_orig: :bro:type:`bool`) : :bro:type:`string`
+         A callback to generate a file handle on demand when
+         one is needed by the core.
+
+      describe: :bro:type:`function` (f: :bro:type:`fa_file`) : :bro:type:`string` :bro:attr:`&default` = :bro:type:`function` :bro:attr:`&optional`
+         A callback to "describe" a file.  In the case of an HTTP
+         transfer the most obvious description would be the URL.
+         It's like an extremely compressed version of the normal log.
+
+
+Events
+######
+.. bro:id:: Files::log_files
+
+   :Type: :bro:type:`event` (rec: :bro:type:`Files::Info`)
+
+   Event that can be handled to access the Info record as it is sent on
+   to the logging framework.
+
+Functions
+#########
+.. bro:id:: Files::add_analyzer
+
+   :Type: :bro:type:`function` (f: :bro:type:`fa_file`, tag: :bro:type:`Files::Tag`, args: :bro:type:`Files::AnalyzerArgs` :bro:attr:`&default` = ``[chunk_event=<uninitialized>, stream_event=<uninitialized>, extract_filename=<uninitialized>, extract_limit=104857600]`` :bro:attr:`&optional`) : :bro:type:`bool`
+
+   Adds an analyzer to the analysis of a given file.
+   
+
+   :f: the file.
+   
+
+   :tag: the analyzer type.
+   
+
+   :args: any parameters the analyzer takes.
+   
+
+   :returns: true if the analyzer will be added, or false if analysis
+            for the file isn't currently active or the *args*
+            were invalid for the analyzer type.
+
+.. bro:id:: Files::all_registered_mime_types
+
+   :Type: :bro:type:`function` () : :bro:type:`table` [:bro:type:`Files::Tag`] of :bro:type:`set` [:bro:type:`string`]
+
+   Returns a table of all MIME-type-to-analyzer mappings currently registered.
+   
+
+   :returns: A table mapping each analyzer to the set of MIME types
+            registered for it.
+
+.. bro:id:: Files::analyzer_name
+
+   :Type: :bro:type:`function` (tag: :bro:type:`Files::Tag`) : :bro:type:`string`
+
+   Translates a file analyzer enum value to a string with the
+   analyzer's name.
+   
+
+   :tag: The analyzer tag.
+   
+
+   :returns: The analyzer name corresponding to the tag.
+
+.. bro:id:: Files::describe
+
+   :Type: :bro:type:`function` (f: :bro:type:`fa_file`) : :bro:type:`string`
+
+   Provides a text description regarding metadata of the file.
+   For example, with HTTP it would return a URL.
+   
+
+   :f: The file to be described.
+   
+
+   :returns: a text description regarding metadata of the file.
+
+.. bro:id:: Files::disable_reassembly
+
+   :Type: :bro:type:`function` (f: :bro:type:`fa_file`) : :bro:type:`void`
+
+   Disables the file reassembler on this file.  If the file is not 
+   transferred out of order this will have no effect.
+   
+
+   :f: the file.
+
+.. bro:id:: Files::enable_reassembly
+
+   :Type: :bro:type:`function` (f: :bro:type:`fa_file`) : :bro:type:`void`
+
+   Allows the file reassembler to be used if it's necessary because the
+   file is transferred out of order.
+   
+
+   :f: the file.
+
+.. bro:id:: Files::file_exists
+
+   :Type: :bro:type:`function` (fuid: :bro:type:`string`) : :bro:type:`bool`
+
+   Lookup to see if a particular file id exists and is still valid.
+   
+
+   :fuid: the file id.
+   
+
+   :returns: T if the file uid is known.
+
+.. bro:id:: Files::lookup_file
+
+   :Type: :bro:type:`function` (fuid: :bro:type:`string`) : :bro:type:`fa_file`
+
+   Lookup an :bro:see:`fa_file` record with the file id.
+   
+
+   :fuid: the file id.
+   
+
+   :returns: the associated :bro:see:`fa_file` record.
+
+.. bro:id:: Files::register_analyzer_add_callback
+
+   :Type: :bro:type:`function` (tag: :bro:type:`Files::Tag`, callback: :bro:type:`function` (f: :bro:type:`fa_file`, args: :bro:type:`Files::AnalyzerArgs`) : :bro:type:`void`) : :bro:type:`void`
+
+   Register a callback for file analyzers to use if they need to do some
+   manipulation when they are being added to a file before the core code
+   takes over.  This is unlikely to be interesting for users and should
+   only be called by file analyzer authors but is *not required*.
+   
+
+   :tag: Tag for the file analyzer.
+   
+
+   :callback: Function to execute when the given file analyzer is being added.
+
+.. bro:id:: Files::register_for_mime_type
+
+   :Type: :bro:type:`function` (tag: :bro:type:`Files::Tag`, mt: :bro:type:`string`) : :bro:type:`bool`
+
+   Registers a MIME type for an analyzer. If a future file with this type is seen,
+   the analyzer will be automatically assigned to parsing it. The function *adds*
+   to all MIME types already registered, it doesn't replace them.
+   
+
+   :tag: The tag of the analyzer.
+   
+
+   :mt: The MIME type in the form "foo/bar" (case-insensitive).
+   
+
+   :returns: True if the MIME type was successfully registered.
+
+.. bro:id:: Files::register_for_mime_types
+
+   :Type: :bro:type:`function` (tag: :bro:type:`Files::Tag`, mime_types: :bro:type:`set` [:bro:type:`string`]) : :bro:type:`bool`
+
+   Registers a set of MIME types for an analyzer. If a future connection on one of
+   these types is seen, the analyzer will be automatically assigned to parsing it.
+   The function *adds* to all MIME types already registered, it doesn't replace
+   them.
+   
+
+   :tag: The tag of the analyzer.
+   
+
+   :mts: The set of MIME types, each in the form "foo/bar" (case-insensitive).
+   
+
+   :returns: True if the MIME types were successfully registered.
+
+.. bro:id:: Files::register_protocol
+
+   :Type: :bro:type:`function` (tag: :bro:type:`Analyzer::Tag`, reg: :bro:type:`Files::ProtoRegistration`) : :bro:type:`bool`
+
+   Register callbacks for protocols that work with the Files framework.
+   The callbacks must uniquely identify a file and each protocol can 
+   only have a single callback registered for it.
+   
+
+   :tag: Tag for the protocol analyzer having a callback being registered.
+   
+
+   :reg: A :bro:see:`Files::ProtoRegistration` record.
+   
+
+   :returns: true if the protocol being registered was not previously registered.
+
+.. bro:id:: Files::registered_mime_types
+
+   :Type: :bro:type:`function` (tag: :bro:type:`Files::Tag`) : :bro:type:`set` [:bro:type:`string`]
+
+   Returns a set of all MIME types currently registered for a specific analyzer.
+   
+
+   :tag: The tag of the analyzer.
+   
+
+   :returns: The set of MIME types.
+
+.. bro:id:: Files::remove_analyzer
+
+   :Type: :bro:type:`function` (f: :bro:type:`fa_file`, tag: :bro:type:`Files::Tag`, args: :bro:type:`Files::AnalyzerArgs` :bro:attr:`&default` = ``[chunk_event=<uninitialized>, stream_event=<uninitialized>, extract_filename=<uninitialized>, extract_limit=104857600]`` :bro:attr:`&optional`) : :bro:type:`bool`
+
+   Removes an analyzer from the analysis of a given file.
+   
+
+   :f: the file.
+   
+
+   :tag: the analyzer type.
+   
+
+   :args: the analyzer (type and args) to remove.
+   
+
+   :returns: true if the analyzer will be removed, or false if analysis
+            for the file isn't currently active.
+
+.. bro:id:: Files::set_reassembly_buffer_size
+
+   :Type: :bro:type:`function` (f: :bro:type:`fa_file`, max: :bro:type:`count`) : :bro:type:`void`
+
+   Set the maximum size the reassembly buffer is allowed to grow
+   for the given file.
+   
+
+   :f: the file.
+   
+
+   :max: Maximum allowed size of the reassembly buffer.
+
+.. bro:id:: Files::set_timeout_interval
+
+   :Type: :bro:type:`function` (f: :bro:type:`fa_file`, t: :bro:type:`interval`) : :bro:type:`bool`
+
+   Sets the *timeout_interval* field of :bro:see:`fa_file`, which is
+   used to determine the length of inactivity that is allowed for a file
+   before internal state related to it is cleaned up.  When used within
+   a :bro:see:`file_timeout` handler, the analysis will delay timing out
+   again for the period specified by *t*.
+   
+
+   :f: the file.
+   
+
+   :t: the amount of time the file can remain inactive before discarding.
+   
+
+   :returns: true if the timeout interval was set, or false if analysis
+            for the file isn't currently active.
+
+.. bro:id:: Files::stop
+
+   :Type: :bro:type:`function` (f: :bro:type:`fa_file`) : :bro:type:`bool`
+
+   Stops/ignores any further analysis of a given file.
+   
+
+   :f: the file.
+   
+
+   :returns: true if analysis for the given file will be ignored for the
+            rest of its contents, or false if analysis for the file
+            isn't currently active.
+
+

doc/scripts/base/frameworks/input/__load__.bro.rst

@@ -0,0 +1,14 @@
+:tocdepth: 3
+
+base/frameworks/input/__load__.bro
+==================================
+
+
+:Imports: :doc:`base/frameworks/input/main.bro </scripts/base/frameworks/input/main.bro>`, :doc:`base/frameworks/input/readers/ascii.bro </scripts/base/frameworks/input/readers/ascii.bro>`, :doc:`base/frameworks/input/readers/benchmark.bro </scripts/base/frameworks/input/readers/benchmark.bro>`, :doc:`base/frameworks/input/readers/binary.bro </scripts/base/frameworks/input/readers/binary.bro>`, :doc:`base/frameworks/input/readers/config.bro </scripts/base/frameworks/input/readers/config.bro>`, :doc:`base/frameworks/input/readers/raw.bro </scripts/base/frameworks/input/readers/raw.bro>`, :doc:`base/frameworks/input/readers/sqlite.bro </scripts/base/frameworks/input/readers/sqlite.bro>`
+
+Summary
+~~~~~~~
+
+Detailed Interface
+~~~~~~~~~~~~~~~~~~
+

doc/scripts/base/frameworks/input/index.rst

@@ -0,0 +1,50 @@
+:orphan:
+
+Package: base/frameworks/input
+==============================
+
+The input framework provides a way to read previously stored data either as
+an event stream or into a Bro table.
+
+:doc:`/scripts/base/frameworks/input/__load__.bro`
+
+
+:doc:`/scripts/base/frameworks/input/main.bro`
+
+   The input framework provides a way to read previously stored data either
+   as an event stream or into a Bro table.
+
+:doc:`/scripts/base/frameworks/input/readers/ascii.bro`
+
+   Interface for the ascii input reader.
+   
+   The defaults are set to match Bro's ASCII output.
+
+:doc:`/scripts/base/frameworks/input/readers/raw.bro`
+
+   Interface for the raw input reader.
+
+:doc:`/scripts/base/frameworks/input/readers/benchmark.bro`
+
+   Interface for the benchmark input reader.
+
+:doc:`/scripts/base/frameworks/input/readers/binary.bro`
+
+   Interface for the binary input reader.
+
+:doc:`/scripts/base/frameworks/input/readers/config.bro`
+
+   Interface for the config input reader.
+
+:doc:`/scripts/base/frameworks/input/readers/sqlite.bro`
+
+   Interface for the SQLite input reader. Redefinable options are available
+   to tweak the input format of the SQLite reader.
+   
+   See :doc:`/frameworks/logging-input-sqlite` for an introduction on how to
+   use the SQLite reader.
+   
+   When using the SQLite reader, you have to specify the SQL query that returns
+   the desired data by setting ``query`` in the ``config`` table. See the
+   introduction mentioned above for an example.
+

doc/scripts/base/frameworks/input/main.bro.rst

@@ -0,0 +1,418 @@
+:tocdepth: 3
+
+base/frameworks/input/main.bro
+==============================
+.. bro:namespace:: Input
+
+The input framework provides a way to read previously stored data either
+as an event stream or into a Bro table.
+
+:Namespace: Input
+:Imports: :doc:`base/bif/input.bif.bro </scripts/base/bif/input.bif.bro>`
+
+Summary
+~~~~~~~
+Runtime Options
+###############
+============================================================================= ==============================
+:bro:id:`Input::default_mode`: :bro:type:`Input::Mode` :bro:attr:`&redef`     The default reader mode used.
+:bro:id:`Input::default_reader`: :bro:type:`Input::Reader` :bro:attr:`&redef` The default input reader used.
+============================================================================= ==============================
+
+Redefinable Options
+###################
+============================================================================== =========================================================
+:bro:id:`Input::accept_unsupported_types`: :bro:type:`bool` :bro:attr:`&redef` Flag that controls if the input framework accepts records
+                                                                               that contain types that are not supported (at the moment
+                                                                               file and function).
+:bro:id:`Input::empty_field`: :bro:type:`string` :bro:attr:`&redef`            String to use for empty fields.
+:bro:id:`Input::separator`: :bro:type:`string` :bro:attr:`&redef`              Separator between fields.
+:bro:id:`Input::set_separator`: :bro:type:`string` :bro:attr:`&redef`          Separator between set elements.
+:bro:id:`Input::unset_field`: :bro:type:`string` :bro:attr:`&redef`            String to use for an unset &optional field.
+============================================================================== =========================================================
+
+Types
+#####
+========================================================== ===================================================================
+:bro:type:`Input::AnalysisDescription`: :bro:type:`record` A file analysis input stream type used to forward input data to the
+                                                           file analysis framework.
+:bro:type:`Input::Event`: :bro:type:`enum`                 
+:bro:type:`Input::EventDescription`: :bro:type:`record`    An event input stream type used to send input data to a Bro event.
+:bro:type:`Input::Mode`: :bro:type:`enum`                  Type that defines the input stream read mode.
+:bro:type:`Input::TableDescription`: :bro:type:`record`    A table input stream type used to send data to a Bro table.
+:bro:type:`Input::Reader`: :bro:type:`enum`                
+========================================================== ===================================================================
+
+Events
+######
+=============================================== ====================================================================
+:bro:id:`Input::end_of_data`: :bro:type:`event` Event that is called when the end of a data source has been reached,
+                                                including after an update.
+=============================================== ====================================================================
+
+Functions
+#########
+=================================================== ============================================================
+:bro:id:`Input::add_analysis`: :bro:type:`function` Create a new file analysis input stream from a given source.
+:bro:id:`Input::add_event`: :bro:type:`function`    Create a new event input stream from a given source.
+:bro:id:`Input::add_table`: :bro:type:`function`    Create a new table input stream from a given source.
+:bro:id:`Input::force_update`: :bro:type:`function` Forces the current input to be checked for changes.
+:bro:id:`Input::remove`: :bro:type:`function`       Remove an input stream.
+=================================================== ============================================================
+
+
+Detailed Interface
+~~~~~~~~~~~~~~~~~~
+Runtime Options
+###############
+.. bro:id:: Input::default_mode
+
+   :Type: :bro:type:`Input::Mode`
+   :Attributes: :bro:attr:`&redef`
+   :Default: ``Input::MANUAL``
+
+   The default reader mode used. Defaults to `MANUAL`.
+
+.. bro:id:: Input::default_reader
+
+   :Type: :bro:type:`Input::Reader`
+   :Attributes: :bro:attr:`&redef`
+   :Default: ``Input::READER_ASCII``
+
+   The default input reader used. Defaults to `READER_ASCII`.
+
+Redefinable Options
+###################
+.. bro:id:: Input::accept_unsupported_types
+
+   :Type: :bro:type:`bool`
+   :Attributes: :bro:attr:`&redef`
+   :Default: ``F``
+
+   Flag that controls if the input framework accepts records
+   that contain types that are not supported (at the moment
+   file and function). If true, the input framework will
+   warn in these cases, but continue. If false, it will
+   abort. Defaults to false (abort).
+
+.. bro:id:: Input::empty_field
+
+   :Type: :bro:type:`string`
+   :Attributes: :bro:attr:`&redef`
+   :Default: ``"(empty)"``
+
+   String to use for empty fields.
+   Individual readers can use a different value.
+
+.. bro:id:: Input::separator
+
+   :Type: :bro:type:`string`
+   :Attributes: :bro:attr:`&redef`
+   :Default: ``"\x09"``
+
+   Separator between fields.
+   Please note that the separator has to be exactly one character long.
+   Individual readers can use a different value.
+
+.. bro:id:: Input::set_separator
+
+   :Type: :bro:type:`string`
+   :Attributes: :bro:attr:`&redef`
+   :Default: ``","``
+
+   Separator between set elements.
+   Please note that the separator has to be exactly one character long.
+   Individual readers can use a different value.
+
+.. bro:id:: Input::unset_field
+
+   :Type: :bro:type:`string`
+   :Attributes: :bro:attr:`&redef`
+   :Default: ``"-"``
+
+   String to use for an unset &optional field.
+   Individual readers can use a different value.
+
+Types
+#####
+.. bro:type:: Input::AnalysisDescription
+
+   :Type: :bro:type:`record`
+
+      source: :bro:type:`string`
+         String that allows the reader to find the source.
+         For `READER_ASCII`, this is the filename.
+
+      reader: :bro:type:`Input::Reader` :bro:attr:`&default` = ``Input::READER_BINARY`` :bro:attr:`&optional`
+         Reader to use for this stream.  Compatible readers must be
+         able to accept a filter of a single string type (i.e.
+         they read a byte stream).
+
+      mode: :bro:type:`Input::Mode` :bro:attr:`&default` = :bro:see:`Input::default_mode` :bro:attr:`&optional`
+         Read mode to use for this stream.
+
+      name: :bro:type:`string`
+         Descriptive name that uniquely identifies the input source.
+         Can be used to remove a stream at a later time.
+         This will also be used for the unique *source* field of
+         :bro:see:`fa_file`.  Most of the time, the best choice for this
+         field will be the same value as the *source* field.
+
+      config: :bro:type:`table` [:bro:type:`string`] of :bro:type:`string` :bro:attr:`&default` = ``{  }`` :bro:attr:`&optional`
+         A key/value table that will be passed to the reader.
+         Interpretation of the values is left to the reader, but
+         usually they will be used for configuration purposes.
+
+   A file analysis input stream type used to forward input data to the
+   file analysis framework.
+
+.. bro:type:: Input::Event
+
+   :Type: :bro:type:`enum`
+
+      .. bro:enum:: Input::EVENT_NEW Input::Event
+
+         New data has been imported.
+
+      .. bro:enum:: Input::EVENT_CHANGED Input::Event
+
+         Existing data has been changed.
+
+      .. bro:enum:: Input::EVENT_REMOVED Input::Event
+
+         Previously existing data has been removed.
+
+
+.. bro:type:: Input::EventDescription
+
+   :Type: :bro:type:`record`
+
+      source: :bro:type:`string`
+         String that allows the reader to find the source.
+         For `READER_ASCII`, this is the filename.
+
+      reader: :bro:type:`Input::Reader` :bro:attr:`&default` = :bro:see:`Input::default_reader` :bro:attr:`&optional`
+         Reader to use for this stream.
+
+      mode: :bro:type:`Input::Mode` :bro:attr:`&default` = :bro:see:`Input::default_mode` :bro:attr:`&optional`
+         Read mode to use for this stream.
+
+      name: :bro:type:`string`
+         Descriptive name. Used to remove a stream at a later time.
+
+      fields: :bro:type:`any`
+         Record type describing the fields to be retrieved from the input
+         source.
+
+      want_record: :bro:type:`bool` :bro:attr:`&default` = ``T`` :bro:attr:`&optional`
+         If this is false, the event receives each value in *fields* as a
+         separate argument.
+         If this is set to true (default), the event receives all fields in
+         a single record value.
+
+      ev: :bro:type:`any`
+         The event that is raised each time a new line is received from the
+         reader. The event will receive an Input::EventDescription record
+         as the first argument, an Input::Event enum as the second
+         argument, and the fields (as specified in *fields*) as the following
+         arguments (this will either be a single record value containing
+         all fields, or each field value as a separate argument).
+
+      error_ev: :bro:type:`any` :bro:attr:`&optional`
+         Error event that is raised when an information, warning or error
+         is raised by the input stream. If the level is error, the stream will automatically
+         be closed.
+         The event receives the Input::EventDescription as the first argument, the
+         message as the second argument and the Reporter::Level as the third argument.
+         
+         The event is raised like it had been declared as follows:
+         error_ev: function(desc: EventDescription, message: string, level: Reporter::Level) &optional;
+         The actual declaration uses the ``any`` type because of deficiencies of the Bro type system.
+
+      config: :bro:type:`table` [:bro:type:`string`] of :bro:type:`string` :bro:attr:`&default` = ``{  }`` :bro:attr:`&optional`
+         A key/value table that will be passed to the reader.
+         Interpretation of the values is left to the reader, but
+         usually they will be used for configuration purposes.
+
+   An event input stream type used to send input data to a Bro event.
+
+.. bro:type:: Input::Mode
+
+   :Type: :bro:type:`enum`
+
+      .. bro:enum:: Input::MANUAL Input::Mode
+
+         Do not automatically reread the file after it has been read.
+
+      .. bro:enum:: Input::REREAD Input::Mode
+
+         Reread the entire file each time a change is found.
+
+      .. bro:enum:: Input::STREAM Input::Mode
+
+         Read data from end of file each time new data is appended.
+
+   Type that defines the input stream read mode.
+
+.. bro:type:: Input::TableDescription
+
+   :Type: :bro:type:`record`
+
+      source: :bro:type:`string`
+         String that allows the reader to find the source of the data.
+         For `READER_ASCII`, this is the filename.
+
+      reader: :bro:type:`Input::Reader` :bro:attr:`&default` = :bro:see:`Input::default_reader` :bro:attr:`&optional`
+         Reader to use for this stream.
+
+      mode: :bro:type:`Input::Mode` :bro:attr:`&default` = :bro:see:`Input::default_mode` :bro:attr:`&optional`
+         Read mode to use for this stream.
+
+      name: :bro:type:`string`
+         Name of the input stream.  This is used by some functions to
+         manipulate the stream.
+
+      destination: :bro:type:`any`
+         Table which will receive the data read by the input framework.
+
+      idx: :bro:type:`any`
+         Record that defines the values used as the index of the table.
+
+      val: :bro:type:`any` :bro:attr:`&optional`
+         Record that defines the values used as the elements of the table.
+         If this is undefined, then *destination* must be a set.
+
+      want_record: :bro:type:`bool` :bro:attr:`&default` = ``T`` :bro:attr:`&optional`
+         Defines if the value of the table is a record (default), or a single
+         value. When this is set to false, then *val* can only contain one
+         element.
+
+      ev: :bro:type:`any` :bro:attr:`&optional`
+         The event that is raised each time a value is added to, changed in,
+         or removed from the table. The event will receive an
+         Input::TableDescription as the first argument, an Input::Event
+         enum as the second argument, the *idx* record as the third argument
+         and the value (record) as the fourth argument.
+
+      pred: :bro:type:`function` (typ: :bro:type:`Input::Event`, left: :bro:type:`any`, right: :bro:type:`any`) : :bro:type:`bool` :bro:attr:`&optional`
+         Predicate function that can decide if an insertion, update or removal
+         should really be executed. Parameters have same meaning as for the
+         event.
+         If true is returned, the update is performed. If false is returned,
+         it is skipped.
+
+      error_ev: :bro:type:`any` :bro:attr:`&optional`
+         Error event that is raised when an information, warning or error
+         is raised by the input stream. If the level is error, the stream will automatically
+         be closed.
+         The event receives the Input::TableDescription as the first argument, the
+         message as the second argument and the Reporter::Level as the third argument.
+         
+         The event is raised like if it had been declared as follows:
+         error_ev: function(desc: TableDescription, message: string, level: Reporter::Level) &optional;
+         The actual declaration uses the ``any`` type because of deficiencies of the Bro type system.
+
+      config: :bro:type:`table` [:bro:type:`string`] of :bro:type:`string` :bro:attr:`&default` = ``{  }`` :bro:attr:`&optional`
+         A key/value table that will be passed to the reader.
+         Interpretation of the values is left to the reader, but
+         usually they will be used for configuration purposes.
+
+   A table input stream type used to send data to a Bro table.
+
+.. bro:type:: Input::Reader
+
+   :Type: :bro:type:`enum`
+
+      .. bro:enum:: Input::READER_ASCII Input::Reader
+
+      .. bro:enum:: Input::READER_BENCHMARK Input::Reader
+
+      .. bro:enum:: Input::READER_BINARY Input::Reader
+
+      .. bro:enum:: Input::READER_CONFIG Input::Reader
+
+      .. bro:enum:: Input::READER_RAW Input::Reader
+
+      .. bro:enum:: Input::READER_SQLITE Input::Reader
+
+
+Events
+######
+.. bro:id:: Input::end_of_data
+
+   :Type: :bro:type:`event` (name: :bro:type:`string`, source: :bro:type:`string`)
+
+   Event that is called when the end of a data source has been reached,
+   including after an update.
+   
+
+   :name: Name of the input stream.
+   
+
+   :source: String that identifies the data source (such as the filename).
+
+Functions
+#########
+.. bro:id:: Input::add_analysis
+
+   :Type: :bro:type:`function` (description: :bro:type:`Input::AnalysisDescription`) : :bro:type:`bool`
+
+   Create a new file analysis input stream from a given source.  Data read
+   from the source is automatically forwarded to the file analysis
+   framework.
+   
+
+   :description: A record describing the source.
+   
+
+   :returns: true on success.
+
+.. bro:id:: Input::add_event
+
+   :Type: :bro:type:`function` (description: :bro:type:`Input::EventDescription`) : :bro:type:`bool`
+
+   Create a new event input stream from a given source.
+   
+
+   :description: `EventDescription` record describing the source.
+   
+
+   :returns: true on success.
+
+.. bro:id:: Input::add_table
+
+   :Type: :bro:type:`function` (description: :bro:type:`Input::TableDescription`) : :bro:type:`bool`
+
+   Create a new table input stream from a given source.
+   
+
+   :description: `TableDescription` record describing the source.
+   
+
+   :returns: true on success.
+
+.. bro:id:: Input::force_update
+
+   :Type: :bro:type:`function` (id: :bro:type:`string`) : :bro:type:`bool`
+
+   Forces the current input to be checked for changes.
+   
+
+   :id: string value identifying the stream.
+   
+
+   :returns: true on success and false if the named stream was not found.
+
+.. bro:id:: Input::remove
+
+   :Type: :bro:type:`function` (id: :bro:type:`string`) : :bro:type:`bool`
+
+   Remove an input stream.
+   
+
+   :id: string value identifying the stream to be removed.
+   
+
+   :returns: true on success and false if the named stream was not found.
+
+

doc/scripts/base/frameworks/input/readers/ascii.bro.rst

@@ -0,0 +1,104 @@
+:tocdepth: 3
+
+base/frameworks/input/readers/ascii.bro
+=======================================
+.. bro:namespace:: InputAscii
+
+Interface for the ascii input reader.
+
+The defaults are set to match Bro's ASCII output.
+
+:Namespace: InputAscii
+
+Summary
+~~~~~~~
+Redefinable Options
+###################
+================================================================================ ===========================================
+:bro:id:`InputAscii::empty_field`: :bro:type:`string` :bro:attr:`&redef`         String to use for empty fields.
+:bro:id:`InputAscii::fail_on_file_problem`: :bro:type:`bool` :bro:attr:`&redef`  Fail on file read problems.
+:bro:id:`InputAscii::fail_on_invalid_lines`: :bro:type:`bool` :bro:attr:`&redef` Fail on invalid lines.
+:bro:id:`InputAscii::separator`: :bro:type:`string` :bro:attr:`&redef`           Separator between fields.
+:bro:id:`InputAscii::set_separator`: :bro:type:`string` :bro:attr:`&redef`       Separator between set and vector elements.
+:bro:id:`InputAscii::unset_field`: :bro:type:`string` :bro:attr:`&redef`         String to use for an unset &optional field.
+================================================================================ ===========================================
+
+
+Detailed Interface
+~~~~~~~~~~~~~~~~~~
+Redefinable Options
+###################
+.. bro:id:: InputAscii::empty_field
+
+   :Type: :bro:type:`string`
+   :Attributes: :bro:attr:`&redef`
+   :Default: ``"(empty)"``
+
+   String to use for empty fields.
+
+.. bro:id:: InputAscii::fail_on_file_problem
+
+   :Type: :bro:type:`bool`
+   :Attributes: :bro:attr:`&redef`
+   :Default: ``F``
+
+   Fail on file read problems. If set to true, the ascii
+   input reader will fail when encountering any problems
+   while reading a file different from invalid lines.
+   Examples of such problems are permission problems, or
+   missing files.
+   When set to false, these problems will be ignored. This
+   has an especially big effect for the REREAD mode, which will
+   seamlessly recover from read errors when a file is
+   only temporarily inaccessible. For MANUAL or STREAM files,
+   errors will most likely still be fatal since no automatic
+   re-reading of the file is attempted.
+   Individual readers can use a different value using
+   the $config table.
+   fail_on_file_problem = T was the default behavior
+   until Bro 2.6.
+
+.. bro:id:: InputAscii::fail_on_invalid_lines
+
+   :Type: :bro:type:`bool`
+   :Attributes: :bro:attr:`&redef`
+   :Default: ``F``
+
+   Fail on invalid lines. If set to false, the ascii
+   input reader will jump over invalid lines, reporting
+   warnings in reporter.log. If set to true, errors in
+   input lines will be handled as fatal errors for the
+   reader thread; reading will abort immediately and
+   an error will be logged to reporter.log.
+   Individual readers can use a different value using
+   the $config table.
+   fail_on_invalid_lines = T was the default behavior
+   until Bro 2.6.
+
+.. bro:id:: InputAscii::separator
+
+   :Type: :bro:type:`string`
+   :Attributes: :bro:attr:`&redef`
+   :Default: ``"\x09"``
+
+   Separator between fields.
+   Please note that the separator has to be exactly one character long.
+
+.. bro:id:: InputAscii::set_separator
+
+   :Type: :bro:type:`string`
+   :Attributes: :bro:attr:`&redef`
+   :Default: ``","``
+
+   Separator between set and vector elements.
+   Please note that the separator has to be exactly one character long.
+
+.. bro:id:: InputAscii::unset_field
+
+   :Type: :bro:type:`string`
+   :Attributes: :bro:attr:`&redef`
+   :Default: ``"-"``
+
+   String to use for an unset &optional field.
+
+

doc/scripts/base/frameworks/input/readers/benchmark.bro.rst

@@ -0,0 +1,77 @@
+:tocdepth: 3
+
+base/frameworks/input/readers/benchmark.bro
+===========================================
+.. bro:namespace:: InputBenchmark
+
+Interface for the benchmark input reader.
+
+:Namespace: InputBenchmark
+
+Summary
+~~~~~~~
+Redefinable Options
+###################
+============================================================================ =========================================================
+:bro:id:`InputBenchmark::addfactor`: :bro:type:`count` :bro:attr:`&redef`    Addition factor for each heartbeat.
+:bro:id:`InputBenchmark::autospread`: :bro:type:`double` :bro:attr:`&redef`  Spreading where usleep = 1000000 / autospread * num_lines
+:bro:id:`InputBenchmark::factor`: :bro:type:`double` :bro:attr:`&redef`      Multiplication factor for each second.
+:bro:id:`InputBenchmark::spread`: :bro:type:`count` :bro:attr:`&redef`       Spread factor between lines.
+:bro:id:`InputBenchmark::stopspreadat`: :bro:type:`count` :bro:attr:`&redef` Stop spreading at x lines per heartbeat.
+:bro:id:`InputBenchmark::timedspread`: :bro:type:`double` :bro:attr:`&redef` 1 -> enable timed spreading.
+============================================================================ =========================================================
+
+
+Detailed Interface
+~~~~~~~~~~~~~~~~~~
+Redefinable Options
+###################
+.. bro:id:: InputBenchmark::addfactor
+
+   :Type: :bro:type:`count`
+   :Attributes: :bro:attr:`&redef`
+   :Default: ``0``
+
+   Addition factor for each heartbeat.
+
+.. bro:id:: InputBenchmark::autospread
+
+   :Type: :bro:type:`double`
+   :Attributes: :bro:attr:`&redef`
+   :Default: ``0.0``
+
+   Spreading where usleep = 1000000 / autospread * num_lines
+
+.. bro:id:: InputBenchmark::factor
+
+   :Type: :bro:type:`double`
+   :Attributes: :bro:attr:`&redef`
+   :Default: ``1.0``
+
+   Multiplication factor for each second.
+
+.. bro:id:: InputBenchmark::spread
+
+   :Type: :bro:type:`count`
+   :Attributes: :bro:attr:`&redef`
+   :Default: ``0``
+
+   Spread factor between lines.
+
+.. bro:id:: InputBenchmark::stopspreadat
+
+   :Type: :bro:type:`count`
+   :Attributes: :bro:attr:`&redef`
+   :Default: ``0``
+
+   Stop spreading at x lines per heartbeat.
+
+.. bro:id:: InputBenchmark::timedspread
+
+   :Type: :bro:type:`double`
+   :Attributes: :bro:attr:`&redef`
+   :Default: ``0.0``
+
+   1 -> enable timed spreading.
+
+

doc/scripts/base/frameworks/input/readers/binary.bro.rst

@@ -0,0 +1,32 @@
+:tocdepth: 3
+
+base/frameworks/input/readers/binary.bro
+========================================
+.. bro:namespace:: InputBinary
+
+Interface for the binary input reader.
+
+:Namespace: InputBinary
+
+Summary
+~~~~~~~
+Redefinable Options
+###################
+======================================================================= ==========================================================
+:bro:id:`InputBinary::chunk_size`: :bro:type:`count` :bro:attr:`&redef` Size of data chunks to read from the input file at a time.
+======================================================================= ==========================================================
+
+
+Detailed Interface
+~~~~~~~~~~~~~~~~~~
+Redefinable Options
+###################
+.. bro:id:: InputBinary::chunk_size
+
+   :Type: :bro:type:`count`
+   :Attributes: :bro:attr:`&redef`
+   :Default: ``1024``
+
+   Size of data chunks to read from the input file at a time.
+
+

doc/scripts/base/frameworks/input/readers/config.bro.rst

@@ -0,0 +1,95 @@
+:tocdepth: 3
+
+base/frameworks/input/readers/config.bro
+========================================
+.. bro:namespace:: InputConfig
+
+Interface for the config input reader.
+
+:Namespace: InputConfig
+
+Summary
+~~~~~~~
+Redefinable Options
+###################
+================================================================================ ==========================================
+:bro:id:`InputConfig::empty_field`: :bro:type:`string` :bro:attr:`&redef`        String to use for empty fields.
+:bro:id:`InputConfig::fail_on_file_problem`: :bro:type:`bool` :bro:attr:`&redef` Fail on file read problems.
+:bro:id:`InputConfig::set_separator`: :bro:type:`string` :bro:attr:`&redef`      Separator between set and vector elements.
+================================================================================ ==========================================
+
+Events
+######
+=================================================== ==============================================================
+:bro:id:`InputConfig::new_value`: :bro:type:`event` Event that is called when a config option is added or changes.
+=================================================== ==============================================================
+
+
+Detailed Interface
+~~~~~~~~~~~~~~~~~~
+Redefinable Options
+###################
+.. bro:id:: InputConfig::empty_field
+
+   :Type: :bro:type:`string`
+   :Attributes: :bro:attr:`&redef`
+   :Default: ``""``
+
+   String to use for empty fields.
+   By default this is the empty string, meaning that an empty input field
+   will result in an empty set.
+
+.. bro:id:: InputConfig::fail_on_file_problem
+
+   :Type: :bro:type:`bool`
+   :Attributes: :bro:attr:`&redef`
+   :Default: ``F``
+
+   Fail on file read problems. If set to true, the config
+   input reader will fail when encountering any problems
+   while reading a file different from invalid lines.
+   Examples of such problems are permission problems, or
+   missing files.
+   When set to false, these problems will be ignored. This
+   has an especially big effect for the REREAD mode, which will
+   seamlessly recover from read errors when a file is
+   only temporarily inaccessible. For MANUAL or STREAM files,
+   errors will most likely still be fatal since no automatic
+   re-reading of the file is attempted.
+   Individual readers can use a different value using
+   the $config table.
+
+.. bro:id:: InputConfig::set_separator
+
+   :Type: :bro:type:`string`
+   :Attributes: :bro:attr:`&redef`
+   :Default: ``","``
+
+   Separator between set and vector elements.
+   Please note that the separator has to be exactly one character long.
+
+Events
+######
+.. bro:id:: InputConfig::new_value
+
+   :Type: :bro:type:`event` (name: :bro:type:`string`, source: :bro:type:`string`, id: :bro:type:`string`, value: :bro:type:`any`)
+
+   Event that is called when a config option is added or changes.
+   
+   Note - this does not track the reason for a change (new, changed),
+   and also does not track removals. If you need this, combine the event
+   with a table reader.
+   
+
+   :name: Name of the input stream.
+   
+
+   :source: Source of the input stream.
+   
+
+   :id: ID of the configuration option being set.
+   
+
+   :value: New value of the configuration option being set.
+
+

doc/scripts/base/frameworks/input/readers/raw.bro.rst

@@ -0,0 +1,58 @@
+:tocdepth: 3
+
+base/frameworks/input/readers/raw.bro
+=====================================
+.. bro:namespace:: InputRaw
+
+Interface for the raw input reader.
+
+:Namespace: InputRaw
+
+Summary
+~~~~~~~
+Redefinable Options
+###################
+=========================================================================== ================================
+:bro:id:`InputRaw::record_separator`: :bro:type:`string` :bro:attr:`&redef` Separator between input records.
+=========================================================================== ================================
+
+Events
+######
+======================================================= ====================================================================
+:bro:id:`InputRaw::process_finished`: :bro:type:`event` Event that is called when a process created by the raw reader exits.
+======================================================= ====================================================================
+
+
+Detailed Interface
+~~~~~~~~~~~~~~~~~~
+Redefinable Options
+###################
+.. bro:id:: InputRaw::record_separator
+
+   :Type: :bro:type:`string`
+   :Attributes: :bro:attr:`&redef`
+   :Default: ``"\x0a"``
+
+   Separator between input records.
+   Please note that the separator has to be exactly one character long.
+
+Events
+######
+.. bro:id:: InputRaw::process_finished
+
+   :Type: :bro:type:`event` (name: :bro:type:`string`, source: :bro:type:`string`, exit_code: :bro:type:`count`, signal_exit: :bro:type:`bool`)
+
+   Event that is called when a process created by the raw reader exits.
+   
+
+   :name: name of the input stream.
+
+   :source: source of the input stream.
+
+   :exit_code: exit code of the program, or number of the signal that forced
+              the program to exit.
+
+   :signal_exit: false when program exited normally, true when program was
+                forced to exit by a signal.
+
+

doc/scripts/base/frameworks/input/readers/sqlite.bro.rst

@@ -0,0 +1,59 @@
+:tocdepth: 3
+
+base/frameworks/input/readers/sqlite.bro
+========================================
+.. bro:namespace:: InputSQLite
+
+Interface for the SQLite input reader. Redefinable options are available
+to tweak the input format of the SQLite reader.
+
+See :doc:`/frameworks/logging-input-sqlite` for an introduction on how to
+use the SQLite reader.
+
+When using the SQLite reader, you have to specify the SQL query that returns
+the desired data by setting ``query`` in the ``config`` table. See the
+introduction mentioned above for an example.
+
+:Namespace: InputSQLite
+
+Summary
+~~~~~~~
+Redefinable Options
+###################
+=========================================================================== ===========================================
+:bro:id:`InputSQLite::empty_field`: :bro:type:`string` :bro:attr:`&redef`   String to use for empty fields.
+:bro:id:`InputSQLite::set_separator`: :bro:type:`string` :bro:attr:`&redef` Separator between set elements.
+:bro:id:`InputSQLite::unset_field`: :bro:type:`string` :bro:attr:`&redef`   String to use for an unset &optional field.
+=========================================================================== ===========================================
+
+
+Detailed Interface
+~~~~~~~~~~~~~~~~~~
+Redefinable Options
+###################
+.. bro:id:: InputSQLite::empty_field
+
+   :Type: :bro:type:`string`
+   :Attributes: :bro:attr:`&redef`
+   :Default: ``"(empty)"``
+
+   String to use for empty fields.
+
+.. bro:id:: InputSQLite::set_separator
+
+   :Type: :bro:type:`string`
+   :Attributes: :bro:attr:`&redef`
+   :Default: ``","``
+
+   Separator between set elements.
+   Please note that the separator has to be exactly one character long.
+
+.. bro:id:: InputSQLite::unset_field
+
+   :Type: :bro:type:`string`
+   :Attributes: :bro:attr:`&redef`
+   :Default: ``"-"``
+
+   String to use for an unset &optional field.
+
+

doc/scripts/base/frameworks/intel/__load__.bro.rst

@@ -0,0 +1,14 @@
+:tocdepth: 3
+
+base/frameworks/intel/__load__.bro
+==================================
+
+
+:Imports: :doc:`base/frameworks/cluster </scripts/base/frameworks/cluster/index>`, :doc:`base/frameworks/intel/files.bro </scripts/base/frameworks/intel/files.bro>`, :doc:`base/frameworks/intel/input.bro </scripts/base/frameworks/intel/input.bro>`, :doc:`base/frameworks/intel/main.bro </scripts/base/frameworks/intel/main.bro>`
+
+Summary
+~~~~~~~
+
+Detailed Interface
+~~~~~~~~~~~~~~~~~~
+

doc/scripts/base/frameworks/intel/files.bro.rst

@@ -0,0 +1,27 @@
+:tocdepth: 3
+
+base/frameworks/intel/files.bro
+===============================
+.. bro:namespace:: Intel
+
+File analysis framework integration for the intelligence framework. This
+script manages file information in intelligence framework data structures.
+
+:Namespace: Intel
+:Imports: :doc:`base/frameworks/intel/main.bro </scripts/base/frameworks/intel/main.bro>`
+
+Summary
+~~~~~~~
+Redefinitions
+#############
+=========================================== =============================================================
+:bro:type:`Intel::Info`: :bro:type:`record` Record used for the logging framework representing a positive
+                                            hit within the intelligence framework.
+:bro:type:`Intel::Seen`: :bro:type:`record` Information about a piece of "seen" data.
+:bro:type:`Intel::Type`: :bro:type:`enum`   Enum type to represent various types of intelligence data.
+=========================================== =============================================================
+
+
+Detailed Interface
+~~~~~~~~~~~~~~~~~~
+

doc/scripts/base/frameworks/intel/index.rst

@@ -0,0 +1,29 @@
+:orphan:
+
+Package: base/frameworks/intel
+==============================
+
+The intelligence framework provides a way to store and query intelligence
+data (such as IP addresses or strings). Metadata can also be associated
+with the intelligence.
+
+:doc:`/scripts/base/frameworks/intel/__load__.bro`
+
+
+:doc:`/scripts/base/frameworks/intel/main.bro`
+
+   The intelligence framework provides a way to store and query intelligence
+   data (e.g. IP addresses, URLs and hashes). The intelligence items can be
+   associated with metadata to allow informed decisions about matching and
+   handling.
+
+:doc:`/scripts/base/frameworks/intel/files.bro`
+
+   File analysis framework integration for the intelligence framework. This
+   script manages file information in intelligence framework data structures.
+
+:doc:`/scripts/base/frameworks/intel/input.bro`
+
+   Input handling for the intelligence framework. This script implements the
+   import of intelligence data from files using the input framework.
+

doc/scripts/base/frameworks/intel/input.bro.rst

@@ -0,0 +1,36 @@
+:tocdepth: 3
+
+base/frameworks/intel/input.bro
+===============================
+.. bro:namespace:: Intel
+
+Input handling for the intelligence framework. This script implements the
+import of intelligence data from files using the input framework.
+
+:Namespace: Intel
+:Imports: :doc:`base/frameworks/intel/main.bro </scripts/base/frameworks/intel/main.bro>`
+
+Summary
+~~~~~~~
+Redefinable Options
+###################
+=============================================================== ==============================================
+:bro:id:`Intel::read_files`: :bro:type:`set` :bro:attr:`&redef` Intelligence files that will be read off disk.
+=============================================================== ==============================================
+
+
+Detailed Interface
+~~~~~~~~~~~~~~~~~~
+Redefinable Options
+###################
+.. bro:id:: Intel::read_files
+
+   :Type: :bro:type:`set` [:bro:type:`string`]
+   :Attributes: :bro:attr:`&redef`
+   :Default: ``{}``
+
+   Intelligence files that will be read off disk. The files are
+   reread every time they are updated so updates must be atomic
+   with "mv" instead of writing the file in place.
+
+

doc/scripts/base/frameworks/intel/main.bro.rst

@@ -0,0 +1,531 @@
+:tocdepth: 3
+
+base/frameworks/intel/main.bro
+==============================
+.. bro:namespace:: Intel
+
+The intelligence framework provides a way to store and query intelligence
+data (e.g. IP addresses, URLs and hashes). The intelligence items can be
+associated with metadata to allow informed decisions about matching and
+handling.
+
+:Namespace: Intel
+:Imports: :doc:`base/frameworks/notice </scripts/base/frameworks/notice/index>`
+
+Summary
+~~~~~~~
+Redefinable Options
+###################
+========================================================================= ==============================================
+:bro:id:`Intel::item_expiration`: :bro:type:`interval` :bro:attr:`&redef` The expiration timeout for intelligence items.
+========================================================================= ==============================================
+
+Types
+#####
+=============================================== ==============================================================
+:bro:type:`Intel::Info`: :bro:type:`record`     Record used for the logging framework representing a positive
+                                                hit within the intelligence framework.
+:bro:type:`Intel::Item`: :bro:type:`record`     Represents a piece of intelligence.
+:bro:type:`Intel::MetaData`: :bro:type:`record` Data about an :bro:type:`Intel::Item`.
+:bro:type:`Intel::Seen`: :bro:type:`record`     Information about a piece of "seen" data.
+:bro:type:`Intel::Type`: :bro:type:`enum`       Enum type to represent various types of intelligence data.
+:bro:type:`Intel::TypeSet`: :bro:type:`set`     Set of intelligence data types.
+:bro:type:`Intel::Where`: :bro:type:`enum`      Enum to represent where data came from when it was discovered.
+=============================================== ==============================================================
+
+Redefinitions
+#############
+===================================== =
+:bro:type:`Log::ID`: :bro:type:`enum` 
+===================================== =
+
+Events
+######
+============================================= ==================================================================
+:bro:id:`Intel::log_intel`: :bro:type:`event` 
+:bro:id:`Intel::match`: :bro:type:`event`     Event to represent a match in the intelligence data from data that
+                                              was seen.
+============================================= ==================================================================
+
+Hooks
+#####
+=============================================== ===================================================================
+:bro:id:`Intel::extend_match`: :bro:type:`hook` This hook can be used to influence the logging of intelligence hits
+                                                (e.g.
+:bro:id:`Intel::item_expired`: :bro:type:`hook` This hook can be used to handle expiration of intelligence items.
+=============================================== ===================================================================
+
+Functions
+#########
+============================================= ==================================================================
+:bro:id:`Intel::insert`: :bro:type:`function` Function to insert intelligence data.
+:bro:id:`Intel::remove`: :bro:type:`function` Function to remove intelligence data.
+:bro:id:`Intel::seen`: :bro:type:`function`   Function to declare discovery of a piece of data in order to check
+                                              it against known intelligence for matches.
+============================================= ==================================================================
+
+
+Detailed Interface
+~~~~~~~~~~~~~~~~~~
+Redefinable Options
+###################
+.. bro:id:: Intel::item_expiration
+
+   :Type: :bro:type:`interval`
+   :Attributes: :bro:attr:`&redef`
+   :Default: ``10.0 mins``
+
+   The expiration timeout for intelligence items. Once an item expires, the
+   :bro:id:`Intel::item_expired` hook is called. Reinsertion of an item 
+   resets the timeout. A negative value disables expiration of intelligence 
+   items.
+
+Types
+#####
+.. bro:type:: Intel::Info
+
+   :Type: :bro:type:`record`
+
+      ts: :bro:type:`time` :bro:attr:`&log`
+         Timestamp when the data was discovered.
+
+      uid: :bro:type:`string` :bro:attr:`&log` :bro:attr:`&optional`
+         If a connection was associated with this intelligence hit,
+         this is the uid for the connection
+
+      id: :bro:type:`conn_id` :bro:attr:`&log` :bro:attr:`&optional`
+         If a connection was associated with this intelligence hit,
+         this is the conn_id for the connection.
+
+      seen: :bro:type:`Intel::Seen` :bro:attr:`&log`
+         Where the data was seen.
+
+      matched: :bro:type:`Intel::TypeSet` :bro:attr:`&log`
+         Which indicator types matched.
+
+      sources: :bro:type:`set` [:bro:type:`string`] :bro:attr:`&log` :bro:attr:`&default` = ``{  }`` :bro:attr:`&optional`
+         Sources which supplied data that resulted in this match.
+
+      fuid: :bro:type:`string` :bro:attr:`&log` :bro:attr:`&optional`
+         (present if :doc:`/scripts/base/frameworks/intel/files.bro` is loaded)
+
+         If a file was associated with this intelligence hit,
+         this is the uid for the file.
+
+      file_mime_type: :bro:type:`string` :bro:attr:`&log` :bro:attr:`&optional`
+         (present if :doc:`/scripts/base/frameworks/intel/files.bro` is loaded)
+
+         A mime type if the intelligence hit is related to a file.
+         If the $f field is provided this will be automatically filled
+         out.
+
+      file_desc: :bro:type:`string` :bro:attr:`&log` :bro:attr:`&optional`
+         (present if :doc:`/scripts/base/frameworks/intel/files.bro` is loaded)
+
+         Frequently files can be "described" to give a bit more context.
+         If the $f field is provided this field will be automatically
+         filled out.
+
+   Record used for the logging framework representing a positive
+   hit within the intelligence framework.
+
+.. bro:type:: Intel::Item
+
+   :Type: :bro:type:`record`
+
+      indicator: :bro:type:`string`
+         The intelligence indicator.
+
+      indicator_type: :bro:type:`Intel::Type`
+         The type of data that the indicator field represents.
+
+      meta: :bro:type:`Intel::MetaData`
+         Metadata for the item. Typically represents more deeply
+         descriptive data for a piece of intelligence.
+
+   Represents a piece of intelligence.
+
+.. bro:type:: Intel::MetaData
+
+   :Type: :bro:type:`record`
+
+      source: :bro:type:`string`
+         An arbitrary string value representing the data source. This
+         value is used as unique key to identify a metadata record in
+         the scope of a single intelligence item.
+
+      desc: :bro:type:`string` :bro:attr:`&optional`
+         A freeform description for the data.
+
+      url: :bro:type:`string` :bro:attr:`&optional`
+         A URL for more information about the data.
+
+      do_notice: :bro:type:`bool` :bro:attr:`&default` = ``F`` :bro:attr:`&optional`
+         (present if :doc:`/scripts/policy/frameworks/intel/do_notice.bro` is loaded)
+
+         A boolean value to allow the data itself to represent
+         if the indicator that this metadata is attached to
+         is notice worthy.
+
+      if_in: :bro:type:`Intel::Where` :bro:attr:`&optional`
+         (present if :doc:`/scripts/policy/frameworks/intel/do_notice.bro` is loaded)
+
+         Restrictions on when notices are created to only create
+         them if the *do_notice* field is T and the notice was
+         seen in the indicated location.
+
+      whitelist: :bro:type:`bool` :bro:attr:`&default` = ``F`` :bro:attr:`&optional`
+         (present if :doc:`/scripts/policy/frameworks/intel/whitelist.bro` is loaded)
+
+         A boolean value to indicate whether the item is whitelisted.
+
+      cif_impact: :bro:type:`string` :bro:attr:`&optional`
+         (present if :doc:`/scripts/policy/integration/collective-intel/main.bro` is loaded)
+
+         Maps to the Impact field in the Collective Intelligence Framework.
+
+      cif_severity: :bro:type:`string` :bro:attr:`&optional`
+         (present if :doc:`/scripts/policy/integration/collective-intel/main.bro` is loaded)
+
+         Maps to the Severity field in the Collective Intelligence Framework.
+
+      cif_confidence: :bro:type:`double` :bro:attr:`&optional`
+         (present if :doc:`/scripts/policy/integration/collective-intel/main.bro` is loaded)
+
+         Maps to the Confidence field in the Collective Intelligence Framework.
+
+   Data about an :bro:type:`Intel::Item`.
+
+.. bro:type:: Intel::Seen
+
+   :Type: :bro:type:`record`
+
+      indicator: :bro:type:`string` :bro:attr:`&log` :bro:attr:`&optional`
+         The string if the data is about a string.
+
+      indicator_type: :bro:type:`Intel::Type` :bro:attr:`&log` :bro:attr:`&optional`
+         The type of data that the indicator represents.
+
+      host: :bro:type:`addr` :bro:attr:`&optional`
+         If the indicator type was :bro:enum:`Intel::ADDR`, then this
+         field will be present.
+
+      where: :bro:type:`Intel::Where` :bro:attr:`&log`
+         Where the data was discovered.
+
+      node: :bro:type:`string` :bro:attr:`&optional` :bro:attr:`&log`
+         The name of the node where the match was discovered.
+
+      conn: :bro:type:`connection` :bro:attr:`&optional`
+         If the data was discovered within a connection, the
+         connection record should go here to give context to the data.
+
+      uid: :bro:type:`string` :bro:attr:`&optional`
+         If the data was discovered within a connection, the
+         connection uid should go here to give context to the data.
+         If the *conn* field is provided, this will be automatically
+         filled out.
+
+      f: :bro:type:`fa_file` :bro:attr:`&optional`
+         (present if :doc:`/scripts/base/frameworks/intel/files.bro` is loaded)
+
+         If the data was discovered within a file, the file record
+         should go here to provide context to the data.
+
+      fuid: :bro:type:`string` :bro:attr:`&optional`
+         (present if :doc:`/scripts/base/frameworks/intel/files.bro` is loaded)
+
+         If the data was discovered within a file, the file uid should
+         go here to provide context to the data. If the file record *f*
+         is provided, this will be automatically filled out.
+
+   Information about a piece of "seen" data.
+
+.. bro:type:: Intel::Type
+
+   :Type: :bro:type:`enum`
+
+      .. bro:enum:: Intel::ADDR Intel::Type
+
+         An IP address.
+
+      .. bro:enum:: Intel::SUBNET Intel::Type
+
+         A subnet in CIDR notation.
+
+      .. bro:enum:: Intel::URL Intel::Type
+
+         A complete URL without the prefix ``"http://"``.
+
+      .. bro:enum:: Intel::SOFTWARE Intel::Type
+
+         Software name.
+
+      .. bro:enum:: Intel::EMAIL Intel::Type
+
+         Email address.
+
+      .. bro:enum:: Intel::DOMAIN Intel::Type
+
+         DNS domain name.
+
+      .. bro:enum:: Intel::USER_NAME Intel::Type
+
+         A user name.
+
+      .. bro:enum:: Intel::CERT_HASH Intel::Type
+
+         Certificate SHA-1 hash.
+
+      .. bro:enum:: Intel::PUBKEY_HASH Intel::Type
+
+         Public key MD5 hash. (SSH server host keys are a good example.)
+
+      .. bro:enum:: Intel::FILE_HASH Intel::Type
+
+         (present if :doc:`/scripts/base/frameworks/intel/files.bro` is loaded)
+
+
+         File hash which is non-hash type specific.  It's up to the
+         user to query for any relevant hash types.
+
+      .. bro:enum:: Intel::FILE_NAME Intel::Type
+
+         (present if :doc:`/scripts/base/frameworks/intel/files.bro` is loaded)
+
+
+         File name.  Typically with protocols with definite
+         indications of a file name.
+
+   Enum type to represent various types of intelligence data.
+
+.. bro:type:: Intel::TypeSet
+
+   :Type: :bro:type:`set` [:bro:type:`Intel::Type`]
+
+   Set of intelligence data types.
+
+.. bro:type:: Intel::Where
+
+   :Type: :bro:type:`enum`
+
+      .. bro:enum:: Intel::IN_ANYWHERE Intel::Where
+
+         A catchall value to represent data of unknown provenance.
+
+      .. bro:enum:: Conn::IN_ORIG Intel::Where
+
+         (present if :doc:`/scripts/policy/frameworks/intel/seen/where-locations.bro` is loaded)
+
+
+      .. bro:enum:: Conn::IN_RESP Intel::Where
+
+         (present if :doc:`/scripts/policy/frameworks/intel/seen/where-locations.bro` is loaded)
+
+
+      .. bro:enum:: Files::IN_HASH Intel::Where
+
+         (present if :doc:`/scripts/policy/frameworks/intel/seen/where-locations.bro` is loaded)
+
+
+      .. bro:enum:: Files::IN_NAME Intel::Where
+
+         (present if :doc:`/scripts/policy/frameworks/intel/seen/where-locations.bro` is loaded)
+
+
+      .. bro:enum:: DNS::IN_REQUEST Intel::Where
+
+         (present if :doc:`/scripts/policy/frameworks/intel/seen/where-locations.bro` is loaded)
+
+
+      .. bro:enum:: DNS::IN_RESPONSE Intel::Where
+
+         (present if :doc:`/scripts/policy/frameworks/intel/seen/where-locations.bro` is loaded)
+
+
+      .. bro:enum:: HTTP::IN_HOST_HEADER Intel::Where
+
+         (present if :doc:`/scripts/policy/frameworks/intel/seen/where-locations.bro` is loaded)
+
+
+      .. bro:enum:: HTTP::IN_REFERRER_HEADER Intel::Where
+
+         (present if :doc:`/scripts/policy/frameworks/intel/seen/where-locations.bro` is loaded)
+
+
+      .. bro:enum:: HTTP::IN_USER_AGENT_HEADER Intel::Where
+
+         (present if :doc:`/scripts/policy/frameworks/intel/seen/where-locations.bro` is loaded)
+
+
+      .. bro:enum:: HTTP::IN_X_FORWARDED_FOR_HEADER Intel::Where
+
+         (present if :doc:`/scripts/policy/frameworks/intel/seen/where-locations.bro` is loaded)
+
+
+      .. bro:enum:: HTTP::IN_URL Intel::Where
+
+         (present if :doc:`/scripts/policy/frameworks/intel/seen/where-locations.bro` is loaded)
+
+
+      .. bro:enum:: SMTP::IN_MAIL_FROM Intel::Where
+
+         (present if :doc:`/scripts/policy/frameworks/intel/seen/where-locations.bro` is loaded)
+
+
+      .. bro:enum:: SMTP::IN_RCPT_TO Intel::Where
+
+         (present if :doc:`/scripts/policy/frameworks/intel/seen/where-locations.bro` is loaded)
+
+
+      .. bro:enum:: SMTP::IN_FROM Intel::Where
+
+         (present if :doc:`/scripts/policy/frameworks/intel/seen/where-locations.bro` is loaded)
+
+
+      .. bro:enum:: SMTP::IN_TO Intel::Where
+
+         (present if :doc:`/scripts/policy/frameworks/intel/seen/where-locations.bro` is loaded)
+
+
+      .. bro:enum:: SMTP::IN_CC Intel::Where
+
+         (present if :doc:`/scripts/policy/frameworks/intel/seen/where-locations.bro` is loaded)
+
+
+      .. bro:enum:: SMTP::IN_RECEIVED_HEADER Intel::Where
+
+         (present if :doc:`/scripts/policy/frameworks/intel/seen/where-locations.bro` is loaded)
+
+
+      .. bro:enum:: SMTP::IN_REPLY_TO Intel::Where
+
+         (present if :doc:`/scripts/policy/frameworks/intel/seen/where-locations.bro` is loaded)
+
+
+      .. bro:enum:: SMTP::IN_X_ORIGINATING_IP_HEADER Intel::Where
+
+         (present if :doc:`/scripts/policy/frameworks/intel/seen/where-locations.bro` is loaded)
+
+
+      .. bro:enum:: SMTP::IN_MESSAGE Intel::Where
+
+         (present if :doc:`/scripts/policy/frameworks/intel/seen/where-locations.bro` is loaded)
+
+
+      .. bro:enum:: SSH::IN_SERVER_HOST_KEY Intel::Where
+
+         (present if :doc:`/scripts/policy/frameworks/intel/seen/where-locations.bro` is loaded)
+
+
+      .. bro:enum:: SSL::IN_SERVER_NAME Intel::Where
+
+         (present if :doc:`/scripts/policy/frameworks/intel/seen/where-locations.bro` is loaded)
+
+
+      .. bro:enum:: SMTP::IN_HEADER Intel::Where
+
+         (present if :doc:`/scripts/policy/frameworks/intel/seen/where-locations.bro` is loaded)
+
+
+      .. bro:enum:: X509::IN_CERT Intel::Where
+
+         (present if :doc:`/scripts/policy/frameworks/intel/seen/where-locations.bro` is loaded)
+
+
+      .. bro:enum:: SSH::SUCCESSFUL_LOGIN Intel::Where
+
+         (present if :doc:`/scripts/policy/protocols/ssh/detect-bruteforcing.bro` is loaded)
+
+
+         An indicator of the login for the intel framework.
+
+   Enum to represent where data came from when it was discovered.
+   The convention is to prefix the name with ``IN_``.
+
+Events
+######
+.. bro:id:: Intel::log_intel
+
+   :Type: :bro:type:`event` (rec: :bro:type:`Intel::Info`)
+
+
+.. bro:id:: Intel::match
+
+   :Type: :bro:type:`event` (s: :bro:type:`Intel::Seen`, items: :bro:type:`set` [:bro:type:`Intel::Item`])
+
+   Event to represent a match in the intelligence data from data that
+   was seen. On clusters there is no assurance as to when this event
+   will be generated so do not assume that arbitrary global state beyond
+   the given data will be available.
+   
+   This is the primary mechanism where a user may take actions based on
+   data provided by the intelligence framework.
+
+Hooks
+#####
+.. bro:id:: Intel::extend_match
+
+   :Type: :bro:type:`hook` (info: :bro:type:`Intel::Info`, s: :bro:type:`Intel::Seen`, items: :bro:type:`set` [:bro:type:`Intel::Item`]) : :bro:type:`bool`
+
+   This hook can be used to influence the logging of intelligence hits
+   (e.g. by adding data to the Info record). The default information is
+   added with a priority of 5.
+   
+
+   :info: The Info record that will be logged.
+   
+
+   :s: Information about the data seen.
+   
+
+   :items: The intel items that match the seen data.
+   
+   In case the hook execution is terminated using break, the match will
+   not be logged.
+
+.. bro:id:: Intel::item_expired
+
+   :Type: :bro:type:`hook` (indicator: :bro:type:`string`, indicator_type: :bro:type:`Intel::Type`, metas: :bro:type:`set` [:bro:type:`Intel::MetaData`]) : :bro:type:`bool`
+
+   This hook can be used to handle expiration of intelligence items.
+   
+
+   :indicator: The indicator of the expired item.
+   
+
+   :indicator_type: The indicator type of the expired item.
+   
+
+   :metas: The set of metadata describing the expired item.
+   
+   If all hook handlers are executed, the expiration timeout will be reset.
+   Otherwise, if one of the handlers terminates using break, the item will
+   be removed.
+
+Functions
+#########
+.. bro:id:: Intel::insert
+
+   :Type: :bro:type:`function` (item: :bro:type:`Intel::Item`) : :bro:type:`void`
+
+   Function to insert intelligence data. If the indicator is already
+   present, the associated metadata will be added to the indicator. If
+   the indicator already contains a metadata record from the same source,
+   the existing metadata record will be updated.
+
+.. bro:id:: Intel::remove
+
+   :Type: :bro:type:`function` (item: :bro:type:`Intel::Item`, purge_indicator: :bro:type:`bool` :bro:attr:`&default` = ``F`` :bro:attr:`&optional`) : :bro:type:`void`
+
+   Function to remove intelligence data. If purge_indicator is set, the
+   given metadata is ignored and the indicator is removed completely.
+
+.. bro:id:: Intel::seen
+
+   :Type: :bro:type:`function` (s: :bro:type:`Intel::Seen`) : :bro:type:`void`
+
+   Function to declare discovery of a piece of data in order to check
+   it against known intelligence for matches.
+
+

doc/scripts/base/frameworks/logging/__load__.bro.rst

@@ -0,0 +1,14 @@
+:tocdepth: 3
+
+base/frameworks/logging/__load__.bro
+====================================
+
+
+:Imports: :doc:`base/frameworks/logging/main.bro </scripts/base/frameworks/logging/main.bro>`, :doc:`base/frameworks/logging/postprocessors </scripts/base/frameworks/logging/postprocessors/index>`, :doc:`base/frameworks/logging/writers/ascii.bro </scripts/base/frameworks/logging/writers/ascii.bro>`, :doc:`base/frameworks/logging/writers/none.bro </scripts/base/frameworks/logging/writers/none.bro>`, :doc:`base/frameworks/logging/writers/sqlite.bro </scripts/base/frameworks/logging/writers/sqlite.bro>`
+
+Summary
+~~~~~~~
+
+Detailed Interface
+~~~~~~~~~~~~~~~~~~
+

doc/scripts/base/frameworks/logging/index.rst

@@ -0,0 +1,87 @@
+:orphan:
+
+Package: base/frameworks/logging
+================================
+
+The logging framework provides a flexible key-value based logging interface.
+
+:doc:`/scripts/base/frameworks/logging/__load__.bro`
+
+
+:doc:`/scripts/base/frameworks/logging/main.bro`
+
+   The Bro logging interface.
+   
+   See :doc:`/frameworks/logging` for an introduction to Bro's
+   logging framework.
+
+:doc:`/scripts/base/frameworks/logging/postprocessors/__load__.bro`
+
+
+:doc:`/scripts/base/frameworks/logging/postprocessors/scp.bro`
+
+   This script defines a postprocessing function that can be applied
+   to a logging filter in order to automatically SCP (secure copy)
+   a log stream (or a subset of it) to a remote host at configurable
+   rotation time intervals.  Generally, to use this functionality
+   you must handle the :bro:id:`bro_init` event and do the following
+   in your handler:
+   
+   1) Create a new :bro:type:`Log::Filter` record that defines a name/path,
+      rotation interval, and set the ``postprocessor`` to
+      :bro:id:`Log::scp_postprocessor`.
+   2) Add the filter to a logging stream using :bro:id:`Log::add_filter`.
+   3) Add a table entry to :bro:id:`Log::scp_destinations` for the filter's
+      writer/path pair which defines a set of :bro:type:`Log::SCPDestination`
+      records.
+
+:doc:`/scripts/base/frameworks/logging/postprocessors/sftp.bro`
+
+   This script defines a postprocessing function that can be applied
+   to a logging filter in order to automatically SFTP
+   a log stream (or a subset of it) to a remote host at configurable
+   rotation time intervals.  Generally, to use this functionality
+   you must handle the :bro:id:`bro_init` event and do the following
+   in your handler:
+   
+   1) Create a new :bro:type:`Log::Filter` record that defines a name/path,
+      rotation interval, and set the ``postprocessor`` to
+      :bro:id:`Log::sftp_postprocessor`.
+   2) Add the filter to a logging stream using :bro:id:`Log::add_filter`.
+   3) Add a table entry to :bro:id:`Log::sftp_destinations` for the filter's
+      writer/path pair which defines a set of :bro:type:`Log::SFTPDestination`
+      records.
+
+:doc:`/scripts/base/frameworks/logging/writers/ascii.bro`
+
+   Interface for the ASCII log writer.  Redefinable options are available
+   to tweak the output format of ASCII logs.
+   
+   The ASCII writer currently supports one writer-specific per-filter config
+   option: setting ``tsv`` to the string ``T`` turns the output into
+   "tab-separated-value" mode where only a single header row with the column
+   names is printed out as meta information, with no "# fields" prepended; no
+   other meta data gets included in that mode.  Example filter using this::
+   
+      local f: Log::Filter = [$name = "my-filter",
+                              $writer = Log::WRITER_ASCII,
+                              $config = table(["tsv"] = "T")];
+   
+
+:doc:`/scripts/base/frameworks/logging/writers/sqlite.bro`
+
+   Interface for the SQLite log writer. Redefinable options are available
+   to tweak the output format of the SQLite reader.
+   
+   See :doc:`/frameworks/logging-input-sqlite` for an introduction on how to
+   use the SQLite log writer.
+   
+   The SQL writer currently supports one writer-specific filter option via
+   ``config``: setting ``tablename`` sets the name of the table that is used
+   or created in the SQLite database. An example for this is given in the
+   introduction mentioned above.
+
+:doc:`/scripts/base/frameworks/logging/writers/none.bro`
+
+   Interface for the None log writer. This writer is mainly for debugging.
+

doc/scripts/base/frameworks/logging/postprocessors/__load__.bro.rst

@@ -0,0 +1,14 @@
+:tocdepth: 3
+
+base/frameworks/logging/postprocessors/__load__.bro
+===================================================
+
+
+:Imports: :doc:`base/frameworks/logging/postprocessors/scp.bro </scripts/base/frameworks/logging/postprocessors/scp.bro>`, :doc:`base/frameworks/logging/postprocessors/sftp.bro </scripts/base/frameworks/logging/postprocessors/sftp.bro>`
+
+Summary
+~~~~~~~
+
+Detailed Interface
+~~~~~~~~~~~~~~~~~~
+

doc/scripts/base/frameworks/logging/postprocessors/index.rst

@@ -0,0 +1,44 @@
+:orphan:
+
+Package: base/frameworks/logging/postprocessors
+===============================================
+
+Support for postprocessors in the logging framework.
+
+:doc:`/scripts/base/frameworks/logging/postprocessors/__load__.bro`
+
+
+:doc:`/scripts/base/frameworks/logging/postprocessors/scp.bro`
+
+   This script defines a postprocessing function that can be applied
+   to a logging filter in order to automatically SCP (secure copy)
+   a log stream (or a subset of it) to a remote host at configurable
+   rotation time intervals.  Generally, to use this functionality
+   you must handle the :bro:id:`bro_init` event and do the following
+   in your handler:
+   
+   1) Create a new :bro:type:`Log::Filter` record that defines a name/path,
+      rotation interval, and set the ``postprocessor`` to
+      :bro:id:`Log::scp_postprocessor`.
+   2) Add the filter to a logging stream using :bro:id:`Log::add_filter`.
+   3) Add a table entry to :bro:id:`Log::scp_destinations` for the filter's
+      writer/path pair which defines a set of :bro:type:`Log::SCPDestination`
+      records.
+
+:doc:`/scripts/base/frameworks/logging/postprocessors/sftp.bro`
+
+   This script defines a postprocessing function that can be applied
+   to a logging filter in order to automatically SFTP
+   a log stream (or a subset of it) to a remote host at configurable
+   rotation time intervals.  Generally, to use this functionality
+   you must handle the :bro:id:`bro_init` event and do the following
+   in your handler:
+   
+   1) Create a new :bro:type:`Log::Filter` record that defines a name/path,
+      rotation interval, and set the ``postprocessor`` to
+      :bro:id:`Log::sftp_postprocessor`.
+   2) Add the filter to a logging stream using :bro:id:`Log::add_filter`.
+   3) Add a table entry to :bro:id:`Log::sftp_destinations` for the filter's
+      writer/path pair which defines a set of :bro:type:`Log::SFTPDestination`
+      records.
+

doc/scripts/base/frameworks/logging/postprocessors/scp.bro.rst

@@ -0,0 +1,121 @@
+:tocdepth: 3
+
+base/frameworks/logging/postprocessors/scp.bro
+==============================================
+.. bro:namespace:: Log
+
+This script defines a postprocessing function that can be applied
+to a logging filter in order to automatically SCP (secure copy)
+a log stream (or a subset of it) to a remote host at configurable
+rotation time intervals.  Generally, to use this functionality
+you must handle the :bro:id:`bro_init` event and do the following
+in your handler:
+
+1) Create a new :bro:type:`Log::Filter` record that defines a name/path,
+   rotation interval, and set the ``postprocessor`` to
+   :bro:id:`Log::scp_postprocessor`.
+2) Add the filter to a logging stream using :bro:id:`Log::add_filter`.
+3) Add a table entry to :bro:id:`Log::scp_destinations` for the filter's
+   writer/path pair which defines a set of :bro:type:`Log::SCPDestination`
+   records.
+
+:Namespace: Log
+
+Summary
+~~~~~~~
+Redefinable Options
+###################
+============================================================================== ================================================================
+:bro:id:`Log::scp_rotation_date_format`: :bro:type:`string` :bro:attr:`&redef` Default naming format for timestamps embedded into log filenames
+                                                                               that use the SCP rotator.
+============================================================================== ================================================================
+
+State Variables
+###############
+================================================== =======================================================================
+:bro:id:`Log::scp_destinations`: :bro:type:`table` A table indexed by a particular log writer and filter path, that yields
+                                                   a set of remote destinations.
+================================================== =======================================================================
+
+Types
+#####
+=================================================== =====================================================================
+:bro:type:`Log::SCPDestination`: :bro:type:`record` A container that describes the remote destination for the SCP command
+                                                    argument as ``user@host:path``.
+=================================================== =====================================================================
+
+Functions
+#########
+====================================================== ===========================================================
+:bro:id:`Log::scp_postprocessor`: :bro:type:`function` Secure-copies the rotated log to all the remote hosts
+                                                       defined in :bro:id:`Log::scp_destinations` and then deletes
+                                                       the local copy of the rotated log.
+====================================================== ===========================================================
+
+
+Detailed Interface
+~~~~~~~~~~~~~~~~~~
+Redefinable Options
+###################
+.. bro:id:: Log::scp_rotation_date_format
+
+   :Type: :bro:type:`string`
+   :Attributes: :bro:attr:`&redef`
+   :Default: ``"%Y-%m-%d-%H-%M-%S"``
+
+   Default naming format for timestamps embedded into log filenames
+   that use the SCP rotator.
+
+State Variables
+###############
+.. bro:id:: Log::scp_destinations
+
+   :Type: :bro:type:`table` [:bro:type:`Log::Writer`, :bro:type:`string`] of :bro:type:`set` [:bro:type:`Log::SCPDestination`]
+   :Default: ``{}``
+
+   A table indexed by a particular log writer and filter path, that yields
+   a set of remote destinations.  The :bro:id:`Log::scp_postprocessor`
+   function queries this table upon log rotation and performs a secure
+   copy of the rotated log to each destination in the set.  This
+   table can be modified at run-time.
+
+Types
+#####
+.. bro:type:: Log::SCPDestination
+
+   :Type: :bro:type:`record`
+
+      user: :bro:type:`string`
+         The remote user to log in as.  A trust mechanism should be
+         pre-established.
+
+      host: :bro:type:`string`
+         The remote host to which to transfer logs.
+
+      path: :bro:type:`string`
+         The path/directory on the remote host to send logs.
+
+   A container that describes the remote destination for the SCP command
+   argument as ``user@host:path``.
+
+Functions
+#########
+.. bro:id:: Log::scp_postprocessor
+
+   :Type: :bro:type:`function` (info: :bro:type:`Log::RotationInfo`) : :bro:type:`bool`
+
+   Secure-copies the rotated log to all the remote hosts
+   defined in :bro:id:`Log::scp_destinations` and then deletes
+   the local copy of the rotated log.  It's not active when
+   reading from trace files.
+   
+
+   :info: A record holding meta-information about the log file to be
+         postprocessed.
+   
+
+   :returns: True if secure-copy system command was initiated or
+            if no destination was configured for the log as described
+            by *info*.
+
+

doc/scripts/base/frameworks/logging/postprocessors/sftp.bro.rst

@@ -0,0 +1,124 @@
+:tocdepth: 3
+
+base/frameworks/logging/postprocessors/sftp.bro
+===============================================
+.. bro:namespace:: Log
+
+This script defines a postprocessing function that can be applied
+to a logging filter in order to automatically SFTP
+a log stream (or a subset of it) to a remote host at configurable
+rotation time intervals.  Generally, to use this functionality
+you must handle the :bro:id:`bro_init` event and do the following
+in your handler:
+
+1) Create a new :bro:type:`Log::Filter` record that defines a name/path,
+   rotation interval, and set the ``postprocessor`` to
+   :bro:id:`Log::sftp_postprocessor`.
+2) Add the filter to a logging stream using :bro:id:`Log::add_filter`.
+3) Add a table entry to :bro:id:`Log::sftp_destinations` for the filter's
+   writer/path pair which defines a set of :bro:type:`Log::SFTPDestination`
+   records.
+
+:Namespace: Log
+
+Summary
+~~~~~~~
+Redefinable Options
+###################
+=============================================================================== ================================================================
+:bro:id:`Log::sftp_rotation_date_format`: :bro:type:`string` :bro:attr:`&redef` Default naming format for timestamps embedded into log filenames
+                                                                                that use the SFTP rotator.
+=============================================================================== ================================================================
+
+State Variables
+###############
+=================================================== =======================================================================
+:bro:id:`Log::sftp_destinations`: :bro:type:`table` A table indexed by a particular log writer and filter path, that yields
+                                                    a set of remote destinations.
+=================================================== =======================================================================
+
+Types
+#####
+==================================================== =======================================================================
+:bro:type:`Log::SFTPDestination`: :bro:type:`record` A container that describes the remote destination for the SFTP command,
+                                                     comprised of the username, host, and path at which to upload the file.
+==================================================== =======================================================================
+
+Functions
+#########
+======================================================= ============================================================
+:bro:id:`Log::sftp_postprocessor`: :bro:type:`function` Securely transfers the rotated log to all the remote hosts
+                                                        defined in :bro:id:`Log::sftp_destinations` and then deletes
+                                                        the local copy of the rotated log.
+======================================================= ============================================================
+
+
+Detailed Interface
+~~~~~~~~~~~~~~~~~~
+Redefinable Options
+###################
+.. bro:id:: Log::sftp_rotation_date_format
+
+   :Type: :bro:type:`string`
+   :Attributes: :bro:attr:`&redef`
+   :Default: ``"%Y-%m-%d-%H-%M-%S"``
+
+   Default naming format for timestamps embedded into log filenames
+   that use the SFTP rotator.
+
+State Variables
+###############
+.. bro:id:: Log::sftp_destinations
+
+   :Type: :bro:type:`table` [:bro:type:`Log::Writer`, :bro:type:`string`] of :bro:type:`set` [:bro:type:`Log::SFTPDestination`]
+   :Default: ``{}``
+
+   A table indexed by a particular log writer and filter path, that yields
+   a set of remote destinations.  The :bro:id:`Log::sftp_postprocessor`
+   function queries this table upon log rotation and performs a secure
+   transfer of the rotated log to each destination in the set.  This
+   table can be modified at run-time.
+
+Types
+#####
+.. bro:type:: Log::SFTPDestination
+
+   :Type: :bro:type:`record`
+
+      user: :bro:type:`string`
+         The remote user to log in as.  A trust mechanism should be
+         pre-established.
+
+      host: :bro:type:`string`
+         The remote host to which to transfer logs.
+
+      host_port: :bro:type:`count` :bro:attr:`&default` = ``22`` :bro:attr:`&optional`
+         The port to connect to. Defaults to 22
+
+      path: :bro:type:`string`
+         The path/directory on the remote host to send logs.
+
+   A container that describes the remote destination for the SFTP command,
+   comprised of the username, host, and path at which to upload the file.
+
+Functions
+#########
+.. bro:id:: Log::sftp_postprocessor
+
+   :Type: :bro:type:`function` (info: :bro:type:`Log::RotationInfo`) : :bro:type:`bool`
+
+   Securely transfers the rotated log to all the remote hosts
+   defined in :bro:id:`Log::sftp_destinations` and then deletes
+   the local copy of the rotated log.  It's not active when
+   reading from trace files.
+   
+
+   :info: A record holding meta-information about the log file to be
+         postprocessed.
+   
+
+   :returns: True if sftp system command was initiated or
+            if no destination was configured for the log as described
+            by *info*.
+
+

doc/scripts/base/frameworks/logging/writers/ascii.bro.rst

@@ -0,0 +1,162 @@
+:tocdepth: 3
+
+base/frameworks/logging/writers/ascii.bro
+=========================================
+.. bro:namespace:: LogAscii
+
+Interface for the ASCII log writer.  Redefinable options are available
+to tweak the output format of ASCII logs.
+
+The ASCII writer currently supports one writer-specific per-filter config
+option: setting ``tsv`` to the string ``T`` turns the output into
+"tab-separated-value" mode where only a single header row with the column
+names is printed out as meta information, with no "# fields" prepended; no
+other meta data gets included in that mode.  Example filter using this::
+
+   local f: Log::Filter = [$name = "my-filter",
+                           $writer = Log::WRITER_ASCII,
+                           $config = table(["tsv"] = "T")];
+
+
+:Namespace: LogAscii
+
+Summary
+~~~~~~~
+Redefinable Options
+###################
+========================================================================================= =====================================================================
+:bro:id:`LogAscii::empty_field`: :bro:type:`string` :bro:attr:`&redef`                    String to use for empty fields.
+:bro:id:`LogAscii::gzip_level`: :bro:type:`count` :bro:attr:`&redef`                      Define the gzip level to compress the logs.
+:bro:id:`LogAscii::include_meta`: :bro:type:`bool` :bro:attr:`&redef`                     If true, include lines with log meta information such as column names
+                                                                                          with types, the values of ASCII logging options that are in use, and
+                                                                                          the time when the file was opened and closed (the latter at the end).
+:bro:id:`LogAscii::json_timestamps`: :bro:type:`JSON::TimestampFormat` :bro:attr:`&redef` Format of timestamps when writing out JSON.
+:bro:id:`LogAscii::meta_prefix`: :bro:type:`string` :bro:attr:`&redef`                    Prefix for lines with meta information.
+:bro:id:`LogAscii::output_to_stdout`: :bro:type:`bool` :bro:attr:`&redef`                 If true, output everything to stdout rather than
+                                                                                          into files.
+:bro:id:`LogAscii::separator`: :bro:type:`string` :bro:attr:`&redef`                      Separator between fields.
+:bro:id:`LogAscii::set_separator`: :bro:type:`string` :bro:attr:`&redef`                  Separator between set elements.
+:bro:id:`LogAscii::unset_field`: :bro:type:`string` :bro:attr:`&redef`                    String to use for an unset &optional field.
+:bro:id:`LogAscii::use_json`: :bro:type:`bool` :bro:attr:`&redef`                         If true, the default will be to write logs in a JSON format.
+========================================================================================= =====================================================================
+
+Redefinitions
+#############
+==================================================================================== =
+:bro:id:`Log::default_rotation_postprocessors`: :bro:type:`table` :bro:attr:`&redef` 
+==================================================================================== =
+
+
+Detailed Interface
+~~~~~~~~~~~~~~~~~~
+Redefinable Options
+###################
+.. bro:id:: LogAscii::empty_field
+
+   :Type: :bro:type:`string`
+   :Attributes: :bro:attr:`&redef`
+   :Default: ``"(empty)"``
+
+   String to use for empty fields. This should be different from
+   *unset_field* to make the output unambiguous.
+   
+   This option is also available as a per-filter ``$config`` option.
+
+.. bro:id:: LogAscii::gzip_level
+
+   :Type: :bro:type:`count`
+   :Attributes: :bro:attr:`&redef`
+   :Default: ``0``
+
+   Define the gzip level to compress the logs.  If 0, then no gzip
+   compression is performed. Enabling compression also changes
+   the log file name extension to include ".gz".
+   
+   This option is also available as a per-filter ``$config`` option.
+
+.. bro:id:: LogAscii::include_meta
+
+   :Type: :bro:type:`bool`
+   :Attributes: :bro:attr:`&redef`
+   :Default: ``T``
+
+   If true, include lines with log meta information such as column names
+   with types, the values of ASCII logging options that are in use, and
+   the time when the file was opened and closed (the latter at the end).
+   
+   If writing in JSON format, this is implicitly disabled.
+
+.. bro:id:: LogAscii::json_timestamps
+
+   :Type: :bro:type:`JSON::TimestampFormat`
+   :Attributes: :bro:attr:`&redef`
+   :Default: ``JSON::TS_EPOCH``
+
+   Format of timestamps when writing out JSON. By default, the JSON
+   formatter will use double values for timestamps which represent the
+   number of seconds from the UNIX epoch.
+   
+   This option is also available as a per-filter ``$config`` option.
+
+.. bro:id:: LogAscii::meta_prefix
+
+   :Type: :bro:type:`string`
+   :Attributes: :bro:attr:`&redef`
+   :Default: ``"#"``
+
+   Prefix for lines with meta information.
+   
+   This option is also available as a per-filter ``$config`` option.
+
+.. bro:id:: LogAscii::output_to_stdout
+
+   :Type: :bro:type:`bool`
+   :Attributes: :bro:attr:`&redef`
+   :Default: ``F``
+
+   If true, output everything to stdout rather than
+   into files. This is primarily for debugging purposes.
+   
+   This option is also available as a per-filter ``$config`` option.
+
+.. bro:id:: LogAscii::separator
+
+   :Type: :bro:type:`string`
+   :Attributes: :bro:attr:`&redef`
+   :Default: ``"\x09"``
+
+   Separator between fields.
+   
+   This option is also available as a per-filter ``$config`` option.
+
+.. bro:id:: LogAscii::set_separator
+
+   :Type: :bro:type:`string`
+   :Attributes: :bro:attr:`&redef`
+   :Default: ``","``
+
+   Separator between set elements.
+   
+   This option is also available as a per-filter ``$config`` option.
+
+.. bro:id:: LogAscii::unset_field
+
+   :Type: :bro:type:`string`
+   :Attributes: :bro:attr:`&redef`
+   :Default: ``"-"``
+
+   String to use for an unset &optional field.
+   
+   This option is also available as a per-filter ``$config`` option.
+
+.. bro:id:: LogAscii::use_json
+
+   :Type: :bro:type:`bool`
+   :Attributes: :bro:attr:`&redef`
+   :Default: ``T``
+
+   If true, the default will be to write logs in a JSON format.
+   
+   This option is also available as a per-filter ``$config`` option.
+
+

doc/scripts/base/frameworks/logging/writers/none.bro.rst

@@ -0,0 +1,40 @@
+:tocdepth: 3
+
+base/frameworks/logging/writers/none.bro
+========================================
+.. bro:namespace:: LogNone
+
+Interface for the None log writer. This writer is mainly for debugging.
+
+:Namespace: LogNone
+
+Summary
+~~~~~~~
+Redefinable Options
+###################
+============================================================= ============================================================
+:bro:id:`LogNone::debug`: :bro:type:`bool` :bro:attr:`&redef` If true, output debugging output that can be useful for unit
+                                                              testing the logging framework.
+============================================================= ============================================================
+
+Redefinitions
+#############
+==================================================================================== =
+:bro:id:`Log::default_rotation_postprocessors`: :bro:type:`table` :bro:attr:`&redef` 
+==================================================================================== =
+
+
+Detailed Interface
+~~~~~~~~~~~~~~~~~~
+Redefinable Options
+###################
+.. bro:id:: LogNone::debug
+
+   :Type: :bro:type:`bool`
+   :Attributes: :bro:attr:`&redef`
+   :Default: ``F``
+
+   If true, output debugging output that can be useful for unit
+   testing the logging framework.
+
+

doc/scripts/base/frameworks/logging/writers/sqlite.bro.rst

@@ -0,0 +1,60 @@
+:tocdepth: 3
+
+base/frameworks/logging/writers/sqlite.bro
+==========================================
+.. bro:namespace:: LogSQLite
+
+Interface for the SQLite log writer. Redefinable options are available
+to tweak the output format of the SQLite reader.
+
+See :doc:`/frameworks/logging-input-sqlite` for an introduction on how to
+use the SQLite log writer.
+
+The SQL writer currently supports one writer-specific filter option via
+``config``: setting ``tablename`` sets the name of the table that is used
+or created in the SQLite database. An example for this is given in the
+introduction mentioned above.
+
+:Namespace: LogSQLite
+
+Summary
+~~~~~~~
+Redefinable Options
+###################
+========================================================================= ===========================================
+:bro:id:`LogSQLite::empty_field`: :bro:type:`string` :bro:attr:`&redef`   String to use for empty fields.
+:bro:id:`LogSQLite::set_separator`: :bro:type:`string` :bro:attr:`&redef` Separator between set elements.
+:bro:id:`LogSQLite::unset_field`: :bro:type:`string` :bro:attr:`&redef`   String to use for an unset &optional field.
+========================================================================= ===========================================
+
+
+Detailed Interface
+~~~~~~~~~~~~~~~~~~
+Redefinable Options
+###################
+.. bro:id:: LogSQLite::empty_field
+
+   :Type: :bro:type:`string`
+   :Attributes: :bro:attr:`&redef`
+   :Default: ``"(empty)"``
+
+   String to use for empty fields. This should be different from
+   *unset_field* to make the output unambiguous.
+
+.. bro:id:: LogSQLite::set_separator
+
+   :Type: :bro:type:`string`
+   :Attributes: :bro:attr:`&redef`
+   :Default: ``","``
+
+   Separator between set elements.
+
+.. bro:id:: LogSQLite::unset_field
+
+   :Type: :bro:type:`string`
+   :Attributes: :bro:attr:`&redef`
+   :Default: ``"-"``
+
+   String to use for an unset &optional field.
+
+

doc/scripts/base/frameworks/netcontrol/__load__.bro.rst

@@ -0,0 +1,14 @@
+:tocdepth: 3
+
+base/frameworks/netcontrol/__load__.bro
+=======================================
+
+
+:Imports: :doc:`base/frameworks/cluster </scripts/base/frameworks/cluster/index>`, :doc:`base/frameworks/netcontrol/catch-and-release.bro </scripts/base/frameworks/netcontrol/catch-and-release.bro>`, :doc:`base/frameworks/netcontrol/drop.bro </scripts/base/frameworks/netcontrol/drop.bro>`, :doc:`base/frameworks/netcontrol/main.bro </scripts/base/frameworks/netcontrol/main.bro>`, :doc:`base/frameworks/netcontrol/non-cluster.bro </scripts/base/frameworks/netcontrol/non-cluster.bro>`, :doc:`base/frameworks/netcontrol/plugins </scripts/base/frameworks/netcontrol/plugins/index>`, :doc:`base/frameworks/netcontrol/shunt.bro </scripts/base/frameworks/netcontrol/shunt.bro>`, :doc:`base/frameworks/netcontrol/types.bro </scripts/base/frameworks/netcontrol/types.bro>`
+
+Summary
+~~~~~~~
+
+Detailed Interface
+~~~~~~~~~~~~~~~~~~
+

doc/scripts/base/frameworks/netcontrol/catch-and-release.bro.rst

@@ -0,0 +1,343 @@
+:tocdepth: 3
+
+base/frameworks/netcontrol/catch-and-release.bro
+================================================
+.. bro:namespace:: NetControl
+
+Implementation of catch-and-release functionality for NetControl.
+
+:Namespace: NetControl
+:Imports: :doc:`base/frameworks/cluster </scripts/base/frameworks/cluster/index>`, :doc:`base/frameworks/netcontrol/drop.bro </scripts/base/frameworks/netcontrol/drop.bro>`, :doc:`base/frameworks/netcontrol/main.bro </scripts/base/frameworks/netcontrol/main.bro>`
+
+Summary
+~~~~~~~
+Runtime Options
+###############
+==================================================================================================== ====================================================================================
+:bro:id:`NetControl::catch_release_warn_blocked_ip_encountered`: :bro:type:`bool` :bro:attr:`&redef` If true, catch and release warns if packets of an IP address are still seen after it
+                                                                                                     should have been blocked.
+==================================================================================================== ====================================================================================
+
+Redefinable Options
+###################
+==================================================================================== =====================================================================================
+:bro:id:`NetControl::catch_release_intervals`: :bro:type:`vector` :bro:attr:`&redef` Time intervals for which subsequent drops of the same IP take
+                                                                                     effect.
+:bro:id:`NetControl::watch_connections`: :bro:type:`bool` :bro:attr:`&redef`         If true, catch_release_seen is called on the connection originator in new_connection,
+                                                                                     connection_established, partial_connection, connection_attempt, connection_rejected,
+                                                                                     connection_reset and connection_pending
+==================================================================================== =====================================================================================
+
+Types
+#####
+============================================================= =========================================================================
+:bro:type:`NetControl::BlockInfo`: :bro:type:`record`         This record is used for storing information about current blocks that are
+                                                              part of catch and release.
+:bro:type:`NetControl::CatchReleaseActions`: :bro:type:`enum` The enum that contains the different kinds of messages that are logged by
+                                                              catch and release.
+:bro:type:`NetControl::CatchReleaseInfo`: :bro:type:`record`  The record type that is used for representing and logging
+============================================================= =========================================================================
+
+Redefinitions
+#############
+===================================== =
+:bro:type:`Log::ID`: :bro:type:`enum` 
+===================================== =
+
+Events
+######
+===================================================================== ===================================================================================
+:bro:id:`NetControl::catch_release_add`: :bro:type:`event`            
+:bro:id:`NetControl::catch_release_block_delete`: :bro:type:`event`   
+:bro:id:`NetControl::catch_release_block_new`: :bro:type:`event`      
+:bro:id:`NetControl::catch_release_delete`: :bro:type:`event`         
+:bro:id:`NetControl::catch_release_encountered`: :bro:type:`event`    
+:bro:id:`NetControl::catch_release_forgotten`: :bro:type:`event`      Event is raised when catch and release cases management of an IP address because no
+                                                                      activity was seen within the watch_until period.
+:bro:id:`NetControl::log_netcontrol_catch_release`: :bro:type:`event` Event that can be handled to access the :bro:type:`NetControl::CatchReleaseInfo`
+                                                                      record as it is sent on to the logging framework.
+===================================================================== ===================================================================================
+
+Functions
+#########
+========================================================================= ======================================================================================================
+:bro:id:`NetControl::catch_release_seen`: :bro:type:`function`            This function can be called to notify the catch and release script that activity by
+                                                                          an IP address was seen.
+:bro:id:`NetControl::drop_address_catch_release`: :bro:type:`function`    Stops all packets involving an IP address from being forwarded.
+:bro:id:`NetControl::get_catch_release_info`: :bro:type:`function`        Get the :bro:see:`NetControl::BlockInfo` record for an address currently blocked by catch and release.
+:bro:id:`NetControl::unblock_address_catch_release`: :bro:type:`function` Removes an address from being watched with catch and release.
+========================================================================= ======================================================================================================
+
+
+Detailed Interface
+~~~~~~~~~~~~~~~~~~
+Runtime Options
+###############
+.. bro:id:: NetControl::catch_release_warn_blocked_ip_encountered
+
+   :Type: :bro:type:`bool`
+   :Attributes: :bro:attr:`&redef`
+   :Default: ``F``
+
+   If true, catch and release warns if packets of an IP address are still seen after it
+   should have been blocked.
+
+Redefinable Options
+###################
+.. bro:id:: NetControl::catch_release_intervals
+
+   :Type: :bro:type:`vector` of :bro:type:`interval`
+   :Attributes: :bro:attr:`&redef`
+   :Default:
+
+   ::
+
+      [10.0 mins, 1.0 hr, 1.0 day, 7.0 days]
+
+   Time intervals for which subsequent drops of the same IP take
+   effect.
+
+.. bro:id:: NetControl::watch_connections
+
+   :Type: :bro:type:`bool`
+   :Attributes: :bro:attr:`&redef`
+   :Default: ``T``
+
+   If true, catch_release_seen is called on the connection originator in new_connection,
+   connection_established, partial_connection, connection_attempt, connection_rejected,
+   connection_reset and connection_pending
+
+Types
+#####
+.. bro:type:: NetControl::BlockInfo
+
+   :Type: :bro:type:`record`
+
+      block_until: :bro:type:`time` :bro:attr:`&optional`
+         Absolute time indicating until when a block is inserted using NetControl.
+
+      watch_until: :bro:type:`time`
+         Absolute time indicating until when an IP address is watched to reblock it.
+
+      num_reblocked: :bro:type:`count` :bro:attr:`&default` = ``0`` :bro:attr:`&optional`
+         Number of times an IP address was reblocked.
+
+      current_interval: :bro:type:`count`
+         Number indicating at which catch and release interval we currently are.
+
+      current_block_id: :bro:type:`string`
+         ID of the inserted block, if any.
+
+      location: :bro:type:`string` :bro:attr:`&optional`
+         User specified string.
+
+   This record is used for storing information about current blocks that are
+   part of catch and release.
+
+.. bro:type:: NetControl::CatchReleaseActions
+
+   :Type: :bro:type:`enum`
+
+      .. bro:enum:: NetControl::INFO NetControl::CatchReleaseActions
+
+         Log lines marked with info are purely informational; no action was taken.
+
+      .. bro:enum:: NetControl::ADDED NetControl::CatchReleaseActions
+
+         A rule for the specified IP address already existed in NetControl (outside
+         of catch-and-release). Catch and release did not add a new rule, but is now
+         watching the IP address and will add a new rule after the current rule expires.
+
+      .. bro:enum:: NetControl::DROP NetControl::CatchReleaseActions
+
+         (present if :doc:`/scripts/base/frameworks/netcontrol/types.bro` is loaded)
+
+
+         Stop forwarding all packets matching the entity.
+         
+         No additional arguments.
+
+      .. bro:enum:: NetControl::DROPPED NetControl::CatchReleaseActions
+
+         A drop was requested by catch and release.
+         An address was successfully blocked by catch and release.
+
+      .. bro:enum:: NetControl::UNBLOCK NetControl::CatchReleaseActions
+
+         An address was unblocked after the timeout expired.
+
+      .. bro:enum:: NetControl::FORGOTTEN NetControl::CatchReleaseActions
+
+         An address was forgotten because it did not reappear within the `watch_until` interval.
+
+      .. bro:enum:: NetControl::SEEN_AGAIN NetControl::CatchReleaseActions
+
+         A watched IP address was seen again; catch and release will re-block it.
+
+   The enum that contains the different kinds of messages that are logged by
+   catch and release.
+
+.. bro:type:: NetControl::CatchReleaseInfo
+
+   :Type: :bro:type:`record`
+
+      ts: :bro:type:`time` :bro:attr:`&log`
+         The absolute time indicating when the action for this log-line occured.
+
+      rule_id: :bro:type:`string` :bro:attr:`&log` :bro:attr:`&optional`
+         The rule id that this log line refers to.
+
+      ip: :bro:type:`addr` :bro:attr:`&log`
+         The IP address that this line refers to.
+
+      action: :bro:type:`NetControl::CatchReleaseActions` :bro:attr:`&log`
+         The action that was taken in this log-line.
+
+      block_interval: :bro:type:`interval` :bro:attr:`&log` :bro:attr:`&optional`
+         The current block_interaval (for how long the address is blocked).
+
+      watch_interval: :bro:type:`interval` :bro:attr:`&log` :bro:attr:`&optional`
+         The current watch_interval (for how long the address will be watched and re-block if it reappears).
+
+      blocked_until: :bro:type:`time` :bro:attr:`&log` :bro:attr:`&optional`
+         The absolute time until which the address is blocked.
+
+      watched_until: :bro:type:`time` :bro:attr:`&log` :bro:attr:`&optional`
+         The absolute time until which the address will be monitored.
+
+      num_blocked: :bro:type:`count` :bro:attr:`&log` :bro:attr:`&optional`
+         Number of times that this address was blocked in the current cycle.
+
+      location: :bro:type:`string` :bro:attr:`&log` :bro:attr:`&optional`
+         The user specified location string.
+
+      message: :bro:type:`string` :bro:attr:`&log` :bro:attr:`&optional`
+         Additional informational string by the catch and release framework about this log-line.
+
+   The record type that is used for representing and logging
+
+Events
+######
+.. bro:id:: NetControl::catch_release_add
+
+   :Type: :bro:type:`event` (a: :bro:type:`addr`, location: :bro:type:`string`)
+
+
+.. bro:id:: NetControl::catch_release_block_delete
+
+   :Type: :bro:type:`event` (a: :bro:type:`addr`)
+
+
+.. bro:id:: NetControl::catch_release_block_new
+
+   :Type: :bro:type:`event` (a: :bro:type:`addr`, b: :bro:type:`NetControl::BlockInfo`)
+
+
+.. bro:id:: NetControl::catch_release_delete
+
+   :Type: :bro:type:`event` (a: :bro:type:`addr`, reason: :bro:type:`string`)
+
+
+.. bro:id:: NetControl::catch_release_encountered
+
+   :Type: :bro:type:`event` (a: :bro:type:`addr`)
+
+
+.. bro:id:: NetControl::catch_release_forgotten
+
+   :Type: :bro:type:`event` (a: :bro:type:`addr`, bi: :bro:type:`NetControl::BlockInfo`)
+
+   Event is raised when catch and release cases management of an IP address because no
+   activity was seen within the watch_until period.
+   
+
+   :a: The address that is no longer being managed.
+   
+
+   :bi: The :bro:see:`NetControl::BlockInfo` record containing information about the block.
+
+.. bro:id:: NetControl::log_netcontrol_catch_release
+
+   :Type: :bro:type:`event` (rec: :bro:type:`NetControl::CatchReleaseInfo`)
+
+   Event that can be handled to access the :bro:type:`NetControl::CatchReleaseInfo`
+   record as it is sent on to the logging framework.
+
+Functions
+#########
+.. bro:id:: NetControl::catch_release_seen
+
+   :Type: :bro:type:`function` (a: :bro:type:`addr`) : :bro:type:`void`
+
+   This function can be called to notify the catch and release script that activity by
+   an IP address was seen. If the respective IP address is currently monitored by catch and
+   release and not blocked, the block will be reinstated. See the documentation of watch_new_connection
+   which events the catch and release functionality usually monitors for activity.
+   
+
+   :a: The address that was seen and should be re-dropped if it is being watched.
+
+.. bro:id:: NetControl::drop_address_catch_release
+
+   :Type: :bro:type:`function` (a: :bro:type:`addr`, location: :bro:type:`string` :bro:attr:`&default` = ``""`` :bro:attr:`&optional`) : :bro:type:`NetControl::BlockInfo`
+
+   Stops all packets involving an IP address from being forwarded. This function
+   uses catch-and-release functionality, where the IP address is only dropped for
+   a short amount of time that is incremented steadily when the IP is encountered
+   again.
+   
+   In cluster mode, this function works on workers as well as the manager. On managers,
+   the returned :bro:see:`NetControl::BlockInfo` record will not contain the block ID,
+   which will be assigned on the manager.
+   
+
+   :a: The address to be dropped.
+   
+
+   :t: How long to drop it, with 0 being indefinitely.
+   
+
+   :location: An optional string describing where the drop was triggered.
+   
+
+   :returns: The :bro:see:`NetControl::BlockInfo` record containing information about
+            the inserted block.
+
+.. bro:id:: NetControl::get_catch_release_info
+
+   :Type: :bro:type:`function` (a: :bro:type:`addr`) : :bro:type:`NetControl::BlockInfo`
+
+   Get the :bro:see:`NetControl::BlockInfo` record for an address currently blocked by catch and release.
+   If the address is unknown to catch and release, the watch_until time will be set to 0.
+   
+   In cluster mode, this function works on the manager and workers. On workers, the data will
+   lag slightly behind the manager; if you add a block, it will not be instantly available via
+   this function.
+   
+
+   :a: The address to get information about.
+   
+
+   :returns: The :bro:see:`NetControl::BlockInfo` record containing information about
+            the inserted block.
+
+.. bro:id:: NetControl::unblock_address_catch_release
+
+   :Type: :bro:type:`function` (a: :bro:type:`addr`, reason: :bro:type:`string` :bro:attr:`&default` = ``""`` :bro:attr:`&optional`) : :bro:type:`bool`
+
+   Removes an address from being watched with catch and release. Returns true if the
+   address was found and removed; returns false if it was unknown to catch and release.
+   
+   If the address is currently blocked, and the block was inserted by catch and release,
+   the block is removed.
+   
+
+   :a: The address to be unblocked.
+   
+
+   :reason: A reason for the unblock.
+   
+
+   :returns: True if the address was unblocked.
+
+

doc/scripts/base/frameworks/netcontrol/drop.bro.rst

@@ -0,0 +1,140 @@
+:tocdepth: 3
+
+base/frameworks/netcontrol/drop.bro
+===================================
+.. bro:namespace:: NetControl
+
+Implementation of the drop functionality for NetControl.
+
+:Namespace: NetControl
+:Imports: :doc:`base/frameworks/netcontrol/main.bro </scripts/base/frameworks/netcontrol/main.bro>`
+
+Summary
+~~~~~~~
+Types
+#####
+==================================================== =
+:bro:type:`NetControl::DropInfo`: :bro:type:`record` 
+==================================================== =
+
+Redefinitions
+#############
+===================================== =
+:bro:type:`Log::ID`: :bro:type:`enum` 
+===================================== =
+
+Events
+######
+============================================================ =========================================================================
+:bro:id:`NetControl::log_netcontrol_drop`: :bro:type:`event` Event that can be handled to access the :bro:type:`NetControl::ShuntInfo`
+                                                             record as it is sent on to the logging framework.
+============================================================ =========================================================================
+
+Hooks
+#####
+======================================================== =======================================================================
+:bro:id:`NetControl::drop_rule_policy`: :bro:type:`hook` Hook that allows the modification of rules passed to drop_* before they
+                                                         are passed on.
+======================================================== =======================================================================
+
+Functions
+#########
+=========================================================== ======================================================================
+:bro:id:`NetControl::drop_address`: :bro:type:`function`    Stops all packets involving an IP address from being forwarded.
+:bro:id:`NetControl::drop_connection`: :bro:type:`function` Stops all packets involving a connection address from being forwarded.
+=========================================================== ======================================================================
+
+
+Detailed Interface
+~~~~~~~~~~~~~~~~~~
+Types
+#####
+.. bro:type:: NetControl::DropInfo
+
+   :Type: :bro:type:`record`
+
+      ts: :bro:type:`time` :bro:attr:`&log`
+         Time at which the recorded activity occurred.
+
+      rule_id: :bro:type:`string` :bro:attr:`&log`
+         ID of the rule; unique during each Bro run.
+
+      orig_h: :bro:type:`addr` :bro:attr:`&log`
+         The originator's IP address.
+
+      orig_p: :bro:type:`port` :bro:attr:`&log` :bro:attr:`&optional`
+         The originator's port number.
+
+      resp_h: :bro:type:`addr` :bro:attr:`&log` :bro:attr:`&optional`
+         The responder's IP address.
+
+      resp_p: :bro:type:`port` :bro:attr:`&log` :bro:attr:`&optional`
+         The responder's port number.
+
+      expire: :bro:type:`interval` :bro:attr:`&log`
+         Expiry time of the shunt.
+
+      location: :bro:type:`string` :bro:attr:`&log` :bro:attr:`&optional`
+         Location where the underlying action was triggered.
+
+
+Events
+######
+.. bro:id:: NetControl::log_netcontrol_drop
+
+   :Type: :bro:type:`event` (rec: :bro:type:`NetControl::DropInfo`)
+
+   Event that can be handled to access the :bro:type:`NetControl::ShuntInfo`
+   record as it is sent on to the logging framework.
+
+Hooks
+#####
+.. bro:id:: NetControl::drop_rule_policy
+
+   :Type: :bro:type:`hook` (r: :bro:type:`NetControl::Rule`) : :bro:type:`bool`
+
+   Hook that allows the modification of rules passed to drop_* before they
+   are passed on. If one of the hooks uses break, the rule is ignored.
+   
+
+   :r: The rule to be added.
+
+Functions
+#########
+.. bro:id:: NetControl::drop_address
+
+   :Type: :bro:type:`function` (a: :bro:type:`addr`, t: :bro:type:`interval`, location: :bro:type:`string` :bro:attr:`&default` = ``""`` :bro:attr:`&optional`) : :bro:type:`string`
+
+   Stops all packets involving an IP address from being forwarded.
+   
+
+   :a: The address to be dropped.
+   
+
+   :t: How long to drop it, with 0 being indefinitely.
+   
+
+   :location: An optional string describing where the drop was triggered.
+   
+
+   :returns: The id of the inserted rule on success and zero on failure.
+
+.. bro:id:: NetControl::drop_connection
+
+   :Type: :bro:type:`function` (c: :bro:type:`conn_id`, t: :bro:type:`interval`, location: :bro:type:`string` :bro:attr:`&default` = ``""`` :bro:attr:`&optional`) : :bro:type:`string`
+
+   Stops all packets involving a connection address from being forwarded.
+   
+
+   :c: The connection to be dropped.
+   
+
+   :t: How long to drop it, with 0 being indefinitely.
+   
+
+   :location: An optional string describing where the drop was triggered.
+   
+
+   :returns: The id of the inserted rule on success and zero on failure.
+
+

doc/scripts/base/frameworks/netcontrol/index.rst

@@ -0,0 +1,81 @@
+:orphan:
+
+Package: base/frameworks/netcontrol
+===================================
+
+The NetControl framework provides a way for Bro to interact with networking
+hard- and software, e.g. for dropping and shunting IP addresses/connections,
+etc.
+
+:doc:`/scripts/base/frameworks/netcontrol/__load__.bro`
+
+
+:doc:`/scripts/base/frameworks/netcontrol/types.bro`
+
+   This file defines the types that are used by the NetControl framework.
+   
+   The most important type defined in this file is :bro:see:`NetControl::Rule`,
+   which is used to describe all rules that can be expressed by the NetControl framework. 
+
+:doc:`/scripts/base/frameworks/netcontrol/main.bro`
+
+   Bro's NetControl framework.
+   
+   This plugin-based framework allows to control the traffic that Bro monitors
+   as well as, if having access to the forwarding path, the traffic the network
+   forwards. By default, the framework lets everything through, to both Bro
+   itself as well as on the network. Scripts can then add rules to impose
+   restrictions on entities, such as specific connections or IP addresses.
+   
+   This framework has two APIs: a high-level and low-level. The high-level API
+   provides convenience functions for a set of common operations. The
+   low-level API provides full flexibility.
+
+:doc:`/scripts/base/frameworks/netcontrol/plugin.bro`
+
+   This file defines the plugin interface for NetControl.
+
+:doc:`/scripts/base/frameworks/netcontrol/plugins/__load__.bro`
+
+
+:doc:`/scripts/base/frameworks/netcontrol/plugins/debug.bro`
+
+   Debugging plugin for the NetControl framework, providing insight into
+   executed operations.
+
+:doc:`/scripts/base/frameworks/netcontrol/plugins/openflow.bro`
+
+   OpenFlow plugin for the NetControl framework.
+
+:doc:`/scripts/base/frameworks/netcontrol/plugins/packetfilter.bro`
+
+   NetControl plugin for the process-level PacketFilter that comes with
+   Bro. Since the PacketFilter in Bro is quite limited in scope
+   and can only add/remove filters for addresses, this is quite
+   limited in scope at the moment. 
+
+:doc:`/scripts/base/frameworks/netcontrol/plugins/broker.bro`
+
+   Broker plugin for the NetControl framework. Sends the raw data structures
+   used in NetControl on to Broker to allow for easy handling, e.g., of
+   command-line scripts.
+
+:doc:`/scripts/base/frameworks/netcontrol/plugins/acld.bro`
+
+   Acld plugin for the netcontrol framework.
+
+:doc:`/scripts/base/frameworks/netcontrol/drop.bro`
+
+   Implementation of the drop functionality for NetControl.
+
+:doc:`/scripts/base/frameworks/netcontrol/shunt.bro`
+
+   Implementation of the shunt functionality for NetControl.
+
+:doc:`/scripts/base/frameworks/netcontrol/catch-and-release.bro`
+
+   Implementation of catch-and-release functionality for NetControl.
+
+:doc:`/scripts/base/frameworks/netcontrol/non-cluster.bro`
+
+

doc/scripts/base/frameworks/netcontrol/main.bro.rst

@@ -0,0 +1,544 @@
+:tocdepth: 3
+
+base/frameworks/netcontrol/main.bro
+===================================
+.. bro:namespace:: NetControl
+
+Bro's NetControl framework.
+
+This plugin-based framework allows to control the traffic that Bro monitors
+as well as, if having access to the forwarding path, the traffic the network
+forwards. By default, the framework lets everything through, to both Bro
+itself as well as on the network. Scripts can then add rules to impose
+restrictions on entities, such as specific connections or IP addresses.
+
+This framework has two APIs: a high-level and low-level. The high-level API
+provides convenience functions for a set of common operations. The
+low-level API provides full flexibility.
+
+:Namespace: NetControl
+:Imports: :doc:`base/frameworks/netcontrol/plugin.bro </scripts/base/frameworks/netcontrol/plugin.bro>`, :doc:`base/frameworks/netcontrol/types.bro </scripts/base/frameworks/netcontrol/types.bro>`
+
+Summary
+~~~~~~~
+Types
+#####
+====================================================== =================================================================
+:bro:type:`NetControl::Info`: :bro:type:`record`       The record type defining the column fields of the NetControl log.
+:bro:type:`NetControl::InfoCategory`: :bro:type:`enum` Type of an entry in the NetControl log.
+:bro:type:`NetControl::InfoState`: :bro:type:`enum`    State of an entry in the NetControl log.
+====================================================== =================================================================
+
+Redefinitions
+#############
+================================================ ==========================================
+:bro:type:`Log::ID`: :bro:type:`enum`            The framework's logging stream identifier.
+:bro:type:`NetControl::Rule`: :bro:type:`record` 
+================================================ ==========================================
+
+Events
+######
+======================================================= ===========================================================================
+:bro:id:`NetControl::init`: :bro:type:`event`           Event that is used to initialize plugins.
+:bro:id:`NetControl::init_done`: :bro:type:`event`      Event that is raised once all plugins activated in ``NetControl::init``
+                                                        have finished their initialization.
+:bro:id:`NetControl::log_netcontrol`: :bro:type:`event` Event that can be handled to access the :bro:type:`NetControl::Info`
+                                                        record as it is sent on to the logging framework.
+:bro:id:`NetControl::rule_added`: :bro:type:`event`     Confirms that a rule was put in place by a plugin.
+:bro:id:`NetControl::rule_destroyed`: :bro:type:`event` This event is raised when a rule is deleted from the NetControl framework,
+                                                        because it is no longer in use.
+:bro:id:`NetControl::rule_error`: :bro:type:`event`     Reports an error when operating on a rule.
+:bro:id:`NetControl::rule_exists`: :bro:type:`event`    Signals that a rule that was supposed to be put in place was already
+                                                        existing at the specified plugin.
+:bro:id:`NetControl::rule_new`: :bro:type:`event`       This event is raised when a new rule is created by the NetControl framework
+                                                        due to a call to add_rule.
+:bro:id:`NetControl::rule_removed`: :bro:type:`event`   Reports that a plugin reports a rule was removed due to a
+                                                        remove_rule function call.
+:bro:id:`NetControl::rule_timeout`: :bro:type:`event`   Reports that a rule was removed from a plugin due to a timeout.
+======================================================= ===========================================================================
+
+Hooks
+#####
+=================================================== =========================================================================
+:bro:id:`NetControl::rule_policy`: :bro:type:`hook` Hook that allows the modification of rules passed to add_rule before they
+                                                    are passed on to the plugins.
+=================================================== =========================================================================
+
+Functions
+#########
+============================================================= ==============================================================================================
+:bro:id:`NetControl::activate`: :bro:type:`function`          Activates a plugin.
+:bro:id:`NetControl::add_rule`: :bro:type:`function`          Installs a rule.
+:bro:id:`NetControl::clear`: :bro:type:`function`             Flushes all state by calling :bro:see:`NetControl::remove_rule` on all currently active rules.
+:bro:id:`NetControl::delete_rule`: :bro:type:`function`       Deletes a rule without removing it from the backends to which it has been
+                                                              added before.
+:bro:id:`NetControl::find_rules_addr`: :bro:type:`function`   Searches all rules affecting a certain IP address.
+:bro:id:`NetControl::find_rules_subnet`: :bro:type:`function` Searches all rules affecting a certain subnet.
+:bro:id:`NetControl::plugin_activated`: :bro:type:`function`  Function called by plugins once they finished their activation.
+:bro:id:`NetControl::quarantine_host`: :bro:type:`function`   Quarantines a host.
+:bro:id:`NetControl::redirect_flow`: :bro:type:`function`     Redirects a uni-directional flow to another port.
+:bro:id:`NetControl::remove_rule`: :bro:type:`function`       Removes a rule.
+:bro:id:`NetControl::whitelist_address`: :bro:type:`function` Allows all traffic involving a specific IP address to be forwarded.
+:bro:id:`NetControl::whitelist_subnet`: :bro:type:`function`  Allows all traffic involving a specific IP subnet to be forwarded.
+============================================================= ==============================================================================================
+
+
+Detailed Interface
+~~~~~~~~~~~~~~~~~~
+Types
+#####
+.. bro:type:: NetControl::Info
+
+   :Type: :bro:type:`record`
+
+      ts: :bro:type:`time` :bro:attr:`&log`
+         Time at which the recorded activity occurred.
+
+      rule_id: :bro:type:`string` :bro:attr:`&log` :bro:attr:`&optional`
+         ID of the rule; unique during each Bro run.
+
+      category: :bro:type:`NetControl::InfoCategory` :bro:attr:`&log` :bro:attr:`&optional`
+         Type of the log entry.
+
+      cmd: :bro:type:`string` :bro:attr:`&log` :bro:attr:`&optional`
+         The command the log entry is about.
+
+      state: :bro:type:`NetControl::InfoState` :bro:attr:`&log` :bro:attr:`&optional`
+         State the log entry reflects.
+
+      action: :bro:type:`string` :bro:attr:`&log` :bro:attr:`&optional`
+         String describing an action the entry is about.
+
+      target: :bro:type:`NetControl::TargetType` :bro:attr:`&log` :bro:attr:`&optional`
+         The target type of the action.
+
+      entity_type: :bro:type:`string` :bro:attr:`&log` :bro:attr:`&optional`
+         Type of the entity the log entry is about.
+
+      entity: :bro:type:`string` :bro:attr:`&log` :bro:attr:`&optional`
+         String describing the entity the log entry is about.
+
+      mod: :bro:type:`string` :bro:attr:`&log` :bro:attr:`&optional`
+         String describing the optional modification of the entry (e.h. redirect)
+
+      msg: :bro:type:`string` :bro:attr:`&log` :bro:attr:`&optional`
+         String with an additional message.
+
+      priority: :bro:type:`int` :bro:attr:`&log` :bro:attr:`&optional`
+         Number describing the priority of the log entry.
+
+      expire: :bro:type:`interval` :bro:attr:`&log` :bro:attr:`&optional`
+         Expiry time of the log entry.
+
+      location: :bro:type:`string` :bro:attr:`&log` :bro:attr:`&optional`
+         Location where the underlying action was triggered.
+
+      plugin: :bro:type:`string` :bro:attr:`&log` :bro:attr:`&optional`
+         Plugin triggering the log entry.
+
+   The record type defining the column fields of the NetControl log.
+
+.. bro:type:: NetControl::InfoCategory
+
+   :Type: :bro:type:`enum`
+
+      .. bro:enum:: NetControl::MESSAGE NetControl::InfoCategory
+
+         A log entry reflecting a framework message.
+
+      .. bro:enum:: NetControl::ERROR NetControl::InfoCategory
+
+         A log entry reflecting a framework message.
+
+      .. bro:enum:: NetControl::RULE NetControl::InfoCategory
+
+         A log entry about a rule.
+
+   Type of an entry in the NetControl log.
+
+.. bro:type:: NetControl::InfoState
+
+   :Type: :bro:type:`enum`
+
+      .. bro:enum:: NetControl::REQUESTED NetControl::InfoState
+
+         The request to add/remove a rule was sent to the respective backend.
+
+      .. bro:enum:: NetControl::SUCCEEDED NetControl::InfoState
+
+         A rule was successfully added by a backend.
+
+      .. bro:enum:: NetControl::EXISTS NetControl::InfoState
+
+         A backend reported that a rule was already existing.
+
+      .. bro:enum:: NetControl::FAILED NetControl::InfoState
+
+         A rule addition failed.
+
+      .. bro:enum:: NetControl::REMOVED NetControl::InfoState
+
+         A rule was successfully removed by a backend.
+
+      .. bro:enum:: NetControl::TIMEOUT NetControl::InfoState
+
+         A rule timeout was triggered by the NetControl framework or a backend.
+
+   State of an entry in the NetControl log.
+
+Events
+######
+.. bro:id:: NetControl::init
+
+   :Type: :bro:type:`event` ()
+
+   Event that is used to initialize plugins. Place all plugin initialization
+   related functionality in this event.
+
+.. bro:id:: NetControl::init_done
+
+   :Type: :bro:type:`event` ()
+
+   Event that is raised once all plugins activated in ``NetControl::init``
+   have finished their initialization.
+
+.. bro:id:: NetControl::log_netcontrol
+
+   :Type: :bro:type:`event` (rec: :bro:type:`NetControl::Info`)
+
+   Event that can be handled to access the :bro:type:`NetControl::Info`
+   record as it is sent on to the logging framework.
+
+.. bro:id:: NetControl::rule_added
+
+   :Type: :bro:type:`event` (r: :bro:type:`NetControl::Rule`, p: :bro:type:`NetControl::PluginState`, msg: :bro:type:`string` :bro:attr:`&default` = ``""`` :bro:attr:`&optional`)
+
+   Confirms that a rule was put in place by a plugin.
+   
+
+   :r: The rule now in place.
+   
+
+   :p: The state for the plugin that put it into place.
+   
+
+   :msg: An optional informational message by the plugin.
+
+.. bro:id:: NetControl::rule_destroyed
+
+   :Type: :bro:type:`event` (r: :bro:type:`NetControl::Rule`)
+
+   This event is raised when a rule is deleted from the NetControl framework,
+   because it is no longer in use. This can be caused by the fact that a rule
+   was removed by all plugins to which it was added, by the fact that it timed out
+   or due to rule errors.
+   
+   To get the cause of a rule remove, catch the rule_removed, rule_timeout and
+   rule_error events.
+
+.. bro:id:: NetControl::rule_error
+
+   :Type: :bro:type:`event` (r: :bro:type:`NetControl::Rule`, p: :bro:type:`NetControl::PluginState`, msg: :bro:type:`string` :bro:attr:`&default` = ``""`` :bro:attr:`&optional`)
+
+   Reports an error when operating on a rule.
+   
+
+   :r: The rule that encountered an error.
+   
+
+   :p: The state for the plugin that reported the error.
+   
+
+   :msg: An optional informational message by the plugin.
+
+.. bro:id:: NetControl::rule_exists
+
+   :Type: :bro:type:`event` (r: :bro:type:`NetControl::Rule`, p: :bro:type:`NetControl::PluginState`, msg: :bro:type:`string` :bro:attr:`&default` = ``""`` :bro:attr:`&optional`)
+
+   Signals that a rule that was supposed to be put in place was already
+   existing at the specified plugin. Rules that already have been existing
+   continue to be tracked like normal, but no timeout calls will be sent
+   to the specified plugins. Removal of the rule from the hardware can
+   still be forced by manually issuing a remove_rule call.
+   
+
+   :r: The rule that was already in place.
+   
+
+   :p: The plugin that reported that the rule already was in place.
+   
+
+   :msg: An optional informational message by the plugin.
+
+.. bro:id:: NetControl::rule_new
+
+   :Type: :bro:type:`event` (r: :bro:type:`NetControl::Rule`)
+
+   This event is raised when a new rule is created by the NetControl framework
+   due to a call to add_rule. From this moment, until the rule_destroyed event
+   is raised, the rule is tracked internally by the NetControl framework.
+   
+   Note that this event does not mean that a rule was successfully added by
+   any backend; it just means that the rule has been accepted and addition
+   to the specified backend is queued. To get information when rules are actually
+   installed by the hardware, use the rule_added, rule_exists, rule_removed, rule_timeout
+   and rule_error events.
+
+.. bro:id:: NetControl::rule_removed
+
+   :Type: :bro:type:`event` (r: :bro:type:`NetControl::Rule`, p: :bro:type:`NetControl::PluginState`, msg: :bro:type:`string` :bro:attr:`&default` = ``""`` :bro:attr:`&optional`)
+
+   Reports that a plugin reports a rule was removed due to a
+   remove_rule function call.
+   
+
+   :r: The rule now removed.
+   
+
+   :p: The state for the plugin that had the rule in place and now
+      removed it.
+   
+
+   :msg: An optional informational message by the plugin.
+
+.. bro:id:: NetControl::rule_timeout
+
+   :Type: :bro:type:`event` (r: :bro:type:`NetControl::Rule`, i: :bro:type:`NetControl::FlowInfo`, p: :bro:type:`NetControl::PluginState`)
+
+   Reports that a rule was removed from a plugin due to a timeout.
+   
+
+   :r: The rule now removed.
+   
+
+   :i: Additional flow information, if supported by the protocol.
+   
+
+   :p: The state for the plugin that had the rule in place and now
+      removed it.
+   
+
+   :msg: An optional informational message by the plugin.
+
+Hooks
+#####
+.. bro:id:: NetControl::rule_policy
+
+   :Type: :bro:type:`hook` (r: :bro:type:`NetControl::Rule`) : :bro:type:`bool`
+
+   Hook that allows the modification of rules passed to add_rule before they
+   are passed on to the plugins. If one of the hooks uses break, the rule is
+   ignored and not passed on to any plugin.
+   
+
+   :r: The rule to be added.
+
+Functions
+#########
+.. bro:id:: NetControl::activate
+
+   :Type: :bro:type:`function` (p: :bro:type:`NetControl::PluginState`, priority: :bro:type:`int`) : :bro:type:`void`
+
+   Activates a plugin.
+   
+
+   :p: The plugin to activate.
+   
+
+   :priority: The higher the priority, the earlier this plugin will be checked
+             whether it supports an operation, relative to other plugins.
+
+.. bro:id:: NetControl::add_rule
+
+   :Type: :bro:type:`function` (r: :bro:type:`NetControl::Rule`) : :bro:type:`string`
+
+   Installs a rule.
+   
+
+   :r: The rule to install.
+   
+
+   :returns: If successful, returns an ID string unique to the rule that can
+            later be used to refer to it. If unsuccessful, returns an empty
+            string. The ID is also assigned to ``r$id``. Note that
+            "successful" means "a plugin knew how to handle the rule", it
+            doesn't necessarily mean that it was indeed successfully put in
+            place, because that might happen asynchronously and thus fail
+            only later.
+
+.. bro:id:: NetControl::clear
+
+   :Type: :bro:type:`function` () : :bro:type:`void`
+
+   Flushes all state by calling :bro:see:`NetControl::remove_rule` on all currently active rules.
+
+.. bro:id:: NetControl::delete_rule
+
+   :Type: :bro:type:`function` (id: :bro:type:`string`, reason: :bro:type:`string` :bro:attr:`&default` = ``""`` :bro:attr:`&optional`) : :bro:type:`bool`
+
+   Deletes a rule without removing it from the backends to which it has been
+   added before. This means that no messages will be sent to the switches to which
+   the rule has been added; if it is not removed from them by a separate mechanism,
+   it will stay installed and not be removed later.
+   
+
+   :id: The rule to delete, specified as the ID returned by :bro:see:`NetControl::add_rule`.
+   
+
+   :reason: Optional string argument giving information on why the rule was deleted.
+   
+
+   :returns: True if removal is successful, or sent to manager.
+            False if the rule could not be found.
+
+.. bro:id:: NetControl::find_rules_addr
+
+   :Type: :bro:type:`function` (ip: :bro:type:`addr`) : :bro:type:`vector` of :bro:type:`NetControl::Rule`
+
+   Searches all rules affecting a certain IP address.
+   
+   This function works on both the manager and workers of a cluster. Note that on
+   the worker, the internal rule variables (starting with _) will not reflect the
+   current state.
+   
+
+   :ip: The ip address to search for.
+   
+
+   :returns: vector of all rules affecting the IP address.
+
+.. bro:id:: NetControl::find_rules_subnet
+
+   :Type: :bro:type:`function` (sn: :bro:type:`subnet`) : :bro:type:`vector` of :bro:type:`NetControl::Rule`
+
+   Searches all rules affecting a certain subnet.
+   
+   A rule affects a subnet, if it covers the whole subnet. Note especially that
+   this function will not reveal all rules that are covered by a subnet.
+   
+   For example, a search for 192.168.17.0/8 will reveal a rule that exists for
+   192.168.0.0/16, since this rule affects the subnet. However, it will not reveal
+   a more specific rule for 192.168.17.1/32, which does not directy affect the whole
+   subnet.
+   
+   This function works on both the manager and workers of a cluster. Note that on
+   the worker, the internal rule variables (starting with _) will not reflect the
+   current state.
+   
+
+   :sn: The subnet to search for.
+   
+
+   :returns: vector of all rules affecting the subnet.
+
+.. bro:id:: NetControl::plugin_activated
+
+   :Type: :bro:type:`function` (p: :bro:type:`NetControl::PluginState`) : :bro:type:`void`
+
+   Function called by plugins once they finished their activation. After all
+   plugins defined in bro_init finished to activate, rules will start to be sent
+   to the plugins. Rules that scripts try to set before the backends are ready
+   will be discarded.
+
+.. bro:id:: NetControl::quarantine_host
+
+   :Type: :bro:type:`function` (infected: :bro:type:`addr`, dns: :bro:type:`addr`, quarantine: :bro:type:`addr`, t: :bro:type:`interval`, location: :bro:type:`string` :bro:attr:`&default` = ``""`` :bro:attr:`&optional`) : :bro:type:`vector` of :bro:type:`string`
+
+   Quarantines a host. This requires a special quarantine server, which runs a HTTP server explaining
+   the quarantine and a DNS server which resolves all requests to the quarantine server. DNS queries
+   from the host to the network DNS server will be rewritten and will be sent to the quarantine server
+   instead. Only http communication infected to quarantinehost is allowed. All other network communication
+   is blocked.
+   
+
+   :infected: the host to quarantine.
+   
+
+   :dns: the network dns server.
+   
+
+   :quarantine: the quarantine server running a dns and a web server.
+   
+
+   :t: how long to leave the quarantine in place.
+   
+
+   :returns: Vector of inserted rules on success, empty list on failure.
+
+.. bro:id:: NetControl::redirect_flow
+
+   :Type: :bro:type:`function` (f: :bro:type:`flow_id`, out_port: :bro:type:`count`, t: :bro:type:`interval`, location: :bro:type:`string` :bro:attr:`&default` = ``""`` :bro:attr:`&optional`) : :bro:type:`string`
+
+   Redirects a uni-directional flow to another port.
+   
+
+   :f: The flow to redirect.
+   
+
+   :out_port: Port to redirect the flow to.
+   
+
+   :t: How long to leave the redirect in place, with 0 being indefinitely.
+   
+
+   :location: An optional string describing where the redirect was triggered.
+   
+
+   :returns: The id of the inserted rule on success and zero on failure.
+
+.. bro:id:: NetControl::remove_rule
+
+   :Type: :bro:type:`function` (id: :bro:type:`string`, reason: :bro:type:`string` :bro:attr:`&default` = ``""`` :bro:attr:`&optional`) : :bro:type:`bool`
+
+   Removes a rule.
+   
+
+   :id: The rule to remove, specified as the ID returned by :bro:see:`NetControl::add_rule`.
+   
+
+   :reason: Optional string argument giving information on why the rule was removed.
+   
+
+   :returns: True if successful, the relevant plugin indicated that it knew
+            how to handle the removal. Note that again "success" means the
+            plugin accepted the removal. It might still fail to put it
+            into effect, as that might happen asynchronously and thus go
+            wrong at that point.
+
+.. bro:id:: NetControl::whitelist_address
+
+   :Type: :bro:type:`function` (a: :bro:type:`addr`, t: :bro:type:`interval`, location: :bro:type:`string` :bro:attr:`&default` = ``""`` :bro:attr:`&optional`) : :bro:type:`string`
+
+   Allows all traffic involving a specific IP address to be forwarded.
+   
+
+   :a: The address to be whitelisted.
+   
+
+   :t: How long to whitelist it, with 0 being indefinitely.
+   
+
+   :location: An optional string describing whitelist was triddered.
+   
+
+   :returns: The id of the inserted rule on success and zero on failure.
+
+.. bro:id:: NetControl::whitelist_subnet
+
+   :Type: :bro:type:`function` (s: :bro:type:`subnet`, t: :bro:type:`interval`, location: :bro:type:`string` :bro:attr:`&default` = ``""`` :bro:attr:`&optional`) : :bro:type:`string`
+
+   Allows all traffic involving a specific IP subnet to be forwarded.
+   
+
+   :s: The subnet to be whitelisted.
+   
+
+   :t: How long to whitelist it, with 0 being indefinitely.
+   
+
+   :location: An optional string describing whitelist was triddered.
+   
+
+   :returns: The id of the inserted rule on success and zero on failure.
+
+

doc/scripts/base/frameworks/netcontrol/non-cluster.bro.rst

@@ -0,0 +1,16 @@
+:tocdepth: 3
+
+base/frameworks/netcontrol/non-cluster.bro
+==========================================
+.. bro:namespace:: NetControl
+
+
+:Namespace: NetControl
+:Imports: :doc:`base/frameworks/netcontrol/main.bro </scripts/base/frameworks/netcontrol/main.bro>`
+
+Summary
+~~~~~~~
+
+Detailed Interface
+~~~~~~~~~~~~~~~~~~
+

doc/scripts/base/frameworks/netcontrol/plugin.bro.rst

@@ -0,0 +1,137 @@
+:tocdepth: 3
+
+base/frameworks/netcontrol/plugin.bro
+=====================================
+.. bro:namespace:: NetControl
+
+This file defines the plugin interface for NetControl.
+
+:Namespace: NetControl
+:Imports: :doc:`base/frameworks/netcontrol/types.bro </scripts/base/frameworks/netcontrol/types.bro>`
+
+Summary
+~~~~~~~
+Types
+#####
+======================================================= =====================================================
+:bro:type:`NetControl::Plugin`: :bro:type:`record`      Definition of a plugin.
+:bro:type:`NetControl::PluginState`: :bro:type:`record` This record keeps the per instance state of a plugin.
+======================================================= =====================================================
+
+Redefinitions
+#############
+======================================================= ========================================================================
+:bro:type:`NetControl::PluginState`: :bro:type:`record` Table for a plugin to store instance-specific configuration information.
+======================================================= ========================================================================
+
+
+Detailed Interface
+~~~~~~~~~~~~~~~~~~
+Types
+#####
+.. bro:type:: NetControl::Plugin
+
+   :Type: :bro:type:`record`
+
+      name: :bro:type:`function` (state: :bro:type:`NetControl::PluginState`) : :bro:type:`string`
+         Returns a descriptive name of the plugin instance, suitable for use in logging
+         messages. Note that this function is not optional.
+
+      can_expire: :bro:type:`bool`
+         If true, plugin can expire rules itself. If false, the NetControl
+         framework will manage rule expiration. 
+
+      init: :bro:type:`function` (state: :bro:type:`NetControl::PluginState`) : :bro:type:`void` :bro:attr:`&optional`
+         One-time initialization function called when plugin gets registered, and
+         before any other methods are called.
+         
+         If this function is provided, NetControl assumes that the plugin has to
+         perform, potentially lengthy, initialization before the plugin will become
+         active. In this case, the plugin has to call ``NetControl::plugin_activated``,
+         once initialization finishes.
+
+      done: :bro:type:`function` (state: :bro:type:`NetControl::PluginState`) : :bro:type:`void` :bro:attr:`&optional`
+         One-time finalization function called when a plugin is shutdown; no further
+         functions will be called afterwords.
+
+      add_rule: :bro:type:`function` (state: :bro:type:`NetControl::PluginState`, r: :bro:type:`NetControl::Rule`) : :bro:type:`bool` :bro:attr:`&optional`
+         Implements the add_rule() operation. If the plugin accepts the rule,
+         it returns true, false otherwise. The rule will already have its
+         ``id`` field set, which the plugin may use for identification
+         purposes.
+
+      remove_rule: :bro:type:`function` (state: :bro:type:`NetControl::PluginState`, r: :bro:type:`NetControl::Rule`, reason: :bro:type:`string`) : :bro:type:`bool` :bro:attr:`&optional`
+         Implements the remove_rule() operation. This will only be called for
+         rules that the plugin has previously accepted with add_rule(). The
+         ``id`` field will match that of the add_rule() call.  Generally,
+         a plugin that accepts an add_rule() should also accept the
+         remove_rule().
+
+   Definition of a plugin.
+   
+   Generally a plugin needs to implement only what it can support.  By
+   returning failure, it indicates that it can't support something and
+   the framework will then try another plugin, if available; or inform the
+   that the operation failed. If a function isn't implemented by a plugin,
+   that's considered an implicit failure to support the operation.
+   
+   If plugin accepts a rule operation, it *must* generate one of the reporting
+   events ``rule_{added,remove,error}`` to signal if it indeed worked out;
+   this is separate from accepting the operation because often a plugin
+   will only know later (i.e., asynchronously) if that was an error for
+   something it thought it could handle.
+
+.. bro:type:: NetControl::PluginState
+
+   :Type: :bro:type:`record`
+
+      config: :bro:type:`table` [:bro:type:`string`] of :bro:type:`string` :bro:attr:`&default` = ``{  }`` :bro:attr:`&optional`
+         Table for a plugin to store custom, instance-specific state.
+
+      _id: :bro:type:`count` :bro:attr:`&optional`
+         Unique plugin identifier -- used for backlookup of plugins from Rules. Set internally.
+
+      _priority: :bro:type:`int` :bro:attr:`&default` = ``0`` :bro:attr:`&optional`
+         Set internally.
+
+      _activated: :bro:type:`bool` :bro:attr:`&default` = ``F`` :bro:attr:`&optional`
+         Set internally. Signifies if the plugin has returned that it has activated successfully.
+
+      plugin: :bro:type:`NetControl::Plugin` :bro:attr:`&optional`
+         The plugin that the state belongs to. (Defined separately
+         because of cyclic type dependency.)
+
+      of_controller: :bro:type:`OpenFlow::Controller` :bro:attr:`&optional`
+         (present if :doc:`/scripts/base/frameworks/netcontrol/plugins/openflow.bro` is loaded)
+
+         OpenFlow controller for NetControl OpenFlow plugin.
+
+      of_config: :bro:type:`NetControl::OfConfig` :bro:attr:`&optional`
+         (present if :doc:`/scripts/base/frameworks/netcontrol/plugins/openflow.bro` is loaded)
+
+         OpenFlow configuration record that is passed on initialization.
+
+      broker_config: :bro:type:`NetControl::BrokerConfig` :bro:attr:`&optional`
+         (present if :doc:`/scripts/base/frameworks/netcontrol/plugins/broker.bro` is loaded)
+
+         OpenFlow controller for NetControl Broker plugin.
+
+      broker_id: :bro:type:`count` :bro:attr:`&optional`
+         (present if :doc:`/scripts/base/frameworks/netcontrol/plugins/broker.bro` is loaded)
+
+         The ID of this broker instance - for the mapping to PluginStates.
+
+      acld_config: :bro:type:`NetControl::AcldConfig` :bro:attr:`&optional`
+         (present if :doc:`/scripts/base/frameworks/netcontrol/plugins/acld.bro` is loaded)
+
+
+      acld_id: :bro:type:`count` :bro:attr:`&optional`
+         (present if :doc:`/scripts/base/frameworks/netcontrol/plugins/acld.bro` is loaded)
+
+         The ID of this acld instance - for the mapping to PluginStates.
+
+   This record keeps the per instance state of a plugin.
+   
+   Individual plugins commonly extend this record to suit their needs.
+
+

doc/scripts/base/frameworks/netcontrol/plugins/__load__.bro.rst

@@ -0,0 +1,14 @@
+:tocdepth: 3
+
+base/frameworks/netcontrol/plugins/__load__.bro
+===============================================
+
+
+:Imports: :doc:`base/frameworks/netcontrol/plugins/acld.bro </scripts/base/frameworks/netcontrol/plugins/acld.bro>`, :doc:`base/frameworks/netcontrol/plugins/broker.bro </scripts/base/frameworks/netcontrol/plugins/broker.bro>`, :doc:`base/frameworks/netcontrol/plugins/debug.bro </scripts/base/frameworks/netcontrol/plugins/debug.bro>`, :doc:`base/frameworks/netcontrol/plugins/openflow.bro </scripts/base/frameworks/netcontrol/plugins/openflow.bro>`, :doc:`base/frameworks/netcontrol/plugins/packetfilter.bro </scripts/base/frameworks/netcontrol/plugins/packetfilter.bro>`
+
+Summary
+~~~~~~~
+
+Detailed Interface
+~~~~~~~~~~~~~~~~~~
+

doc/scripts/base/frameworks/netcontrol/plugins/acld.bro.rst

@@ -0,0 +1,162 @@
+:tocdepth: 3
+
+base/frameworks/netcontrol/plugins/acld.bro
+===========================================
+.. bro:namespace:: NetControl
+
+Acld plugin for the netcontrol framework.
+
+:Namespace: NetControl
+:Imports: :doc:`base/frameworks/broker </scripts/base/frameworks/broker/index>`, :doc:`base/frameworks/netcontrol/main.bro </scripts/base/frameworks/netcontrol/main.bro>`, :doc:`base/frameworks/netcontrol/plugin.bro </scripts/base/frameworks/netcontrol/plugin.bro>`
+
+Summary
+~~~~~~~
+Types
+#####
+====================================================== =
+:bro:type:`NetControl::AclRule`: :bro:type:`record`    
+:bro:type:`NetControl::AcldConfig`: :bro:type:`record` 
+====================================================== =
+
+Redefinitions
+#############
+======================================================= =
+:bro:type:`NetControl::PluginState`: :bro:type:`record` 
+======================================================= =
+
+Events
+######
+========================================================== =======================================
+:bro:id:`NetControl::acld_add_rule`: :bro:type:`event`     Events that are sent from us to Broker.
+:bro:id:`NetControl::acld_remove_rule`: :bro:type:`event`  
+:bro:id:`NetControl::acld_rule_added`: :bro:type:`event`   Events that are sent from Broker to us.
+:bro:id:`NetControl::acld_rule_error`: :bro:type:`event`   
+:bro:id:`NetControl::acld_rule_exists`: :bro:type:`event`  
+:bro:id:`NetControl::acld_rule_removed`: :bro:type:`event` 
+========================================================== =======================================
+
+Hooks
+#####
+======================================================== ==============================================================
+:bro:id:`NetControl::acld_rule_policy`: :bro:type:`hook` Hook that is called after a rule is converted to an acld rule.
+======================================================== ==============================================================
+
+Functions
+#########
+======================================================= =============================
+:bro:id:`NetControl::create_acld`: :bro:type:`function` Instantiates the acld plugin.
+======================================================= =============================
+
+
+Detailed Interface
+~~~~~~~~~~~~~~~~~~
+Types
+#####
+.. bro:type:: NetControl::AclRule
+
+   :Type: :bro:type:`record`
+
+      command: :bro:type:`string`
+
+      cookie: :bro:type:`count`
+
+      arg: :bro:type:`string`
+
+      comment: :bro:type:`string` :bro:attr:`&optional`
+
+
+.. bro:type:: NetControl::AcldConfig
+
+   :Type: :bro:type:`record`
+
+      acld_topic: :bro:type:`string`
+         The acld topic to send events to.
+
+      acld_host: :bro:type:`addr`
+         Broker host to connect to.
+
+      acld_port: :bro:type:`port`
+         Broker port to connect to.
+
+      monitor: :bro:type:`bool` :bro:attr:`&default` = ``F`` :bro:attr:`&optional`
+         Do we accept rules for the monitor path? Default false.
+
+      forward: :bro:type:`bool` :bro:attr:`&default` = ``T`` :bro:attr:`&optional`
+         Do we accept rules for the forward path? Default true.
+
+      check_pred: :bro:type:`function` (p: :bro:type:`NetControl::PluginState`, r: :bro:type:`NetControl::Rule`) : :bro:type:`bool` :bro:attr:`&optional`
+         Predicate that is called on rule insertion or removal.
+         
+
+         :p: Current plugin state.
+         
+
+         :r: The rule to be inserted or removed.
+         
+
+         :returns: T if the rule can be handled by the current backend, F otherwise.
+
+
+Events
+######
+.. bro:id:: NetControl::acld_add_rule
+
+   :Type: :bro:type:`event` (id: :bro:type:`count`, r: :bro:type:`NetControl::Rule`, ar: :bro:type:`NetControl::AclRule`)
+
+   Events that are sent from us to Broker.
+
+.. bro:id:: NetControl::acld_remove_rule
+
+   :Type: :bro:type:`event` (id: :bro:type:`count`, r: :bro:type:`NetControl::Rule`, ar: :bro:type:`NetControl::AclRule`)
+
+
+.. bro:id:: NetControl::acld_rule_added
+
+   :Type: :bro:type:`event` (id: :bro:type:`count`, r: :bro:type:`NetControl::Rule`, msg: :bro:type:`string`)
+
+   Events that are sent from Broker to us.
+
+.. bro:id:: NetControl::acld_rule_error
+
+   :Type: :bro:type:`event` (id: :bro:type:`count`, r: :bro:type:`NetControl::Rule`, msg: :bro:type:`string`)
+
+
+.. bro:id:: NetControl::acld_rule_exists
+
+   :Type: :bro:type:`event` (id: :bro:type:`count`, r: :bro:type:`NetControl::Rule`, msg: :bro:type:`string`)
+
+
+.. bro:id:: NetControl::acld_rule_removed
+
+   :Type: :bro:type:`event` (id: :bro:type:`count`, r: :bro:type:`NetControl::Rule`, msg: :bro:type:`string`)
+
+
+Hooks
+#####
+.. bro:id:: NetControl::acld_rule_policy
+
+   :Type: :bro:type:`hook` (p: :bro:type:`NetControl::PluginState`, r: :bro:type:`NetControl::Rule`, ar: :bro:type:`NetControl::AclRule`) : :bro:type:`bool`
+
+   Hook that is called after a rule is converted to an acld rule.
+   The hook may modify the rule before it is sent to acld.
+   Setting the acld command to F will cause the rule to be rejected
+   by the plugin.
+   
+
+   :p: Current plugin state.
+   
+
+   :r: The rule to be inserted or removed.
+   
+
+   :ar: The acld rule to be inserted or removed.
+
+Functions
+#########
+.. bro:id:: NetControl::create_acld
+
+   :Type: :bro:type:`function` (config: :bro:type:`NetControl::AcldConfig`) : :bro:type:`NetControl::PluginState`
+
+   Instantiates the acld plugin.
+
+

doc/scripts/base/frameworks/netcontrol/plugins/broker.bro.rst

@@ -0,0 +1,129 @@
+:tocdepth: 3
+
+base/frameworks/netcontrol/plugins/broker.bro
+=============================================
+.. bro:namespace:: NetControl
+
+Broker plugin for the NetControl framework. Sends the raw data structures
+used in NetControl on to Broker to allow for easy handling, e.g., of
+command-line scripts.
+
+:Namespace: NetControl
+:Imports: :doc:`base/frameworks/broker </scripts/base/frameworks/broker/index>`, :doc:`base/frameworks/netcontrol/main.bro </scripts/base/frameworks/netcontrol/main.bro>`, :doc:`base/frameworks/netcontrol/plugin.bro </scripts/base/frameworks/netcontrol/plugin.bro>`
+
+Summary
+~~~~~~~
+Types
+#####
+======================================================== ===============================================================================================
+:bro:type:`NetControl::BrokerConfig`: :bro:type:`record` This record specifies the configuration that is passed to :bro:see:`NetControl::create_broker`.
+======================================================== ===============================================================================================
+
+Redefinitions
+#############
+======================================================= =
+:bro:type:`NetControl::PluginState`: :bro:type:`record` 
+======================================================= =
+
+Events
+######
+============================================================ =
+:bro:id:`NetControl::broker_add_rule`: :bro:type:`event`     
+:bro:id:`NetControl::broker_remove_rule`: :bro:type:`event`  
+:bro:id:`NetControl::broker_rule_added`: :bro:type:`event`   
+:bro:id:`NetControl::broker_rule_error`: :bro:type:`event`   
+:bro:id:`NetControl::broker_rule_exists`: :bro:type:`event`  
+:bro:id:`NetControl::broker_rule_removed`: :bro:type:`event` 
+:bro:id:`NetControl::broker_rule_timeout`: :bro:type:`event` 
+============================================================ =
+
+Functions
+#########
+========================================================= ===============================
+:bro:id:`NetControl::create_broker`: :bro:type:`function` Instantiates the broker plugin.
+========================================================= ===============================
+
+
+Detailed Interface
+~~~~~~~~~~~~~~~~~~
+Types
+#####
+.. bro:type:: NetControl::BrokerConfig
+
+   :Type: :bro:type:`record`
+
+      topic: :bro:type:`string` :bro:attr:`&optional`
+         The broker topic to send events to.
+
+      host: :bro:type:`addr` :bro:attr:`&optional`
+         Broker host to connect to.
+
+      bport: :bro:type:`port` :bro:attr:`&optional`
+         Broker port to connect to.
+
+      monitor: :bro:type:`bool` :bro:attr:`&default` = ``T`` :bro:attr:`&optional`
+         Do we accept rules for the monitor path? Default true.
+
+      forward: :bro:type:`bool` :bro:attr:`&default` = ``T`` :bro:attr:`&optional`
+         Do we accept rules for the forward path? Default true.
+
+      check_pred: :bro:type:`function` (p: :bro:type:`NetControl::PluginState`, r: :bro:type:`NetControl::Rule`) : :bro:type:`bool` :bro:attr:`&optional`
+         Predicate that is called on rule insertion or removal.
+         
+
+         :p: Current plugin state.
+         
+
+         :r: The rule to be inserted or removed.
+         
+
+         :returns: T if the rule can be handled by the current backend, F otherwise.
+
+   This record specifies the configuration that is passed to :bro:see:`NetControl::create_broker`.
+
+Events
+######
+.. bro:id:: NetControl::broker_add_rule
+
+   :Type: :bro:type:`event` (id: :bro:type:`count`, r: :bro:type:`NetControl::Rule`)
+
+
+.. bro:id:: NetControl::broker_remove_rule
+
+   :Type: :bro:type:`event` (id: :bro:type:`count`, r: :bro:type:`NetControl::Rule`, reason: :bro:type:`string`)
+
+
+.. bro:id:: NetControl::broker_rule_added
+
+   :Type: :bro:type:`event` (id: :bro:type:`count`, r: :bro:type:`NetControl::Rule`, msg: :bro:type:`string`)
+
+
+.. bro:id:: NetControl::broker_rule_error
+
+   :Type: :bro:type:`event` (id: :bro:type:`count`, r: :bro:type:`NetControl::Rule`, msg: :bro:type:`string`)
+
+
+.. bro:id:: NetControl::broker_rule_exists
+
+   :Type: :bro:type:`event` (id: :bro:type:`count`, r: :bro:type:`NetControl::Rule`, msg: :bro:type:`string`)
+
+
+.. bro:id:: NetControl::broker_rule_removed
+
+   :Type: :bro:type:`event` (id: :bro:type:`count`, r: :bro:type:`NetControl::Rule`, msg: :bro:type:`string`)
+
+
+.. bro:id:: NetControl::broker_rule_timeout
+
+   :Type: :bro:type:`event` (id: :bro:type:`count`, r: :bro:type:`NetControl::Rule`, i: :bro:type:`NetControl::FlowInfo`)
+
+
+Functions
+#########
+.. bro:id:: NetControl::create_broker
+
+   :Type: :bro:type:`function` (config: :bro:type:`NetControl::BrokerConfig`, can_expire: :bro:type:`bool`) : :bro:type:`NetControl::PluginState`
+
+   Instantiates the broker plugin.
+
+

doc/scripts/base/frameworks/netcontrol/plugins/debug.bro.rst

@@ -0,0 +1,37 @@
+:tocdepth: 3
+
+base/frameworks/netcontrol/plugins/debug.bro
+============================================
+.. bro:namespace:: NetControl
+
+Debugging plugin for the NetControl framework, providing insight into
+executed operations.
+
+:Namespace: NetControl
+:Imports: :doc:`base/frameworks/netcontrol/main.bro </scripts/base/frameworks/netcontrol/main.bro>`, :doc:`base/frameworks/netcontrol/plugin.bro </scripts/base/frameworks/netcontrol/plugin.bro>`
+
+Summary
+~~~~~~~
+Functions
+#########
+======================================================== =========================================================
+:bro:id:`NetControl::create_debug`: :bro:type:`function` Instantiates a debug plugin for the NetControl framework.
+======================================================== =========================================================
+
+
+Detailed Interface
+~~~~~~~~~~~~~~~~~~
+Functions
+#########
+.. bro:id:: NetControl::create_debug
+
+   :Type: :bro:type:`function` (do_something: :bro:type:`bool`) : :bro:type:`NetControl::PluginState`
+
+   Instantiates a debug plugin for the NetControl framework. The debug
+   plugin simply logs the operations it receives.
+   
+
+   :do_something: If true, the plugin will claim it supports all operations; if
+                 false, it will indicate it doesn't support any.
+
+

doc/scripts/base/frameworks/netcontrol/plugins/index.rst

@@ -0,0 +1,36 @@
+:orphan:
+
+Package: base/frameworks/netcontrol/plugins
+===========================================
+
+Plugins for the NetControl framework.
+
+:doc:`/scripts/base/frameworks/netcontrol/plugins/__load__.bro`
+
+
+:doc:`/scripts/base/frameworks/netcontrol/plugins/debug.bro`
+
+   Debugging plugin for the NetControl framework, providing insight into
+   executed operations.
+
+:doc:`/scripts/base/frameworks/netcontrol/plugins/openflow.bro`
+
+   OpenFlow plugin for the NetControl framework.
+
+:doc:`/scripts/base/frameworks/netcontrol/plugins/packetfilter.bro`
+
+   NetControl plugin for the process-level PacketFilter that comes with
+   Bro. Since the PacketFilter in Bro is quite limited in scope
+   and can only add/remove filters for addresses, this is quite
+   limited in scope at the moment. 
+
+:doc:`/scripts/base/frameworks/netcontrol/plugins/broker.bro`
+
+   Broker plugin for the NetControl framework. Sends the raw data structures
+   used in NetControl on to Broker to allow for easy handling, e.g., of
+   command-line scripts.
+
+:doc:`/scripts/base/frameworks/netcontrol/plugins/acld.bro`
+
+   Acld plugin for the netcontrol framework.
+

doc/scripts/base/frameworks/netcontrol/plugins/openflow.bro.rst

@@ -0,0 +1,158 @@
+:tocdepth: 3
+
+base/frameworks/netcontrol/plugins/openflow.bro
+===============================================
+.. bro:namespace:: NetControl
+
+OpenFlow plugin for the NetControl framework.
+
+:Namespace: NetControl
+:Imports: :doc:`base/frameworks/netcontrol/main.bro </scripts/base/frameworks/netcontrol/main.bro>`, :doc:`base/frameworks/netcontrol/plugin.bro </scripts/base/frameworks/netcontrol/plugin.bro>`, :doc:`base/frameworks/openflow </scripts/base/frameworks/openflow/index>`
+
+Summary
+~~~~~~~
+Redefinable Options
+###################
+======================================================================================= ===============================================================================
+:bro:id:`NetControl::openflow_flow_timeout`: :bro:type:`interval` :bro:attr:`&redef`    The time interval after we consider a flow timed out.
+:bro:id:`NetControl::openflow_message_timeout`: :bro:type:`interval` :bro:attr:`&redef` The time interval after which an openflow message is considered to be timed out
+                                                                                        and we delete it from our internal tracking.
+======================================================================================= ===============================================================================
+
+Types
+#####
+==================================================== =================================================================================================
+:bro:type:`NetControl::OfConfig`: :bro:type:`record` This record specifies the configuration that is passed to :bro:see:`NetControl::create_openflow`.
+:bro:type:`NetControl::OfTable`: :bro:type:`record`  
+==================================================== =================================================================================================
+
+Redefinitions
+#############
+======================================================= =
+:bro:type:`NetControl::PluginState`: :bro:type:`record` 
+======================================================= =
+
+Functions
+#########
+=========================================================== =============================================================
+:bro:id:`NetControl::create_openflow`: :bro:type:`function` Instantiates an openflow plugin for the NetControl framework.
+=========================================================== =============================================================
+
+
+Detailed Interface
+~~~~~~~~~~~~~~~~~~
+Redefinable Options
+###################
+.. bro:id:: NetControl::openflow_flow_timeout
+
+   :Type: :bro:type:`interval`
+   :Attributes: :bro:attr:`&redef`
+   :Default: ``1.0 day``
+
+   The time interval after we consider a flow timed out. This should be fairly high (or
+   even disabled) if you expect a lot of long flows. However, one also will have state
+   buildup for quite a while if keeping this around...
+
+.. bro:id:: NetControl::openflow_message_timeout
+
+   :Type: :bro:type:`interval`
+   :Attributes: :bro:attr:`&redef`
+   :Default: ``20.0 secs``
+
+   The time interval after which an openflow message is considered to be timed out
+   and we delete it from our internal tracking.
+
+Types
+#####
+.. bro:type:: NetControl::OfConfig
+
+   :Type: :bro:type:`record`
+
+      monitor: :bro:type:`bool` :bro:attr:`&default` = ``T`` :bro:attr:`&optional`
+         Accept rules that target the monitor path.
+
+      forward: :bro:type:`bool` :bro:attr:`&default` = ``T`` :bro:attr:`&optional`
+         Accept rules that target the forward path.
+
+      idle_timeout: :bro:type:`count` :bro:attr:`&default` = ``0`` :bro:attr:`&optional`
+         Default OpenFlow idle timeout.
+
+      table_id: :bro:type:`count` :bro:attr:`&optional`
+         Default OpenFlow table ID.
+
+      priority_offset: :bro:type:`int` :bro:attr:`&default` = ``0`` :bro:attr:`&optional`
+         Add this to all rule priorities. Can be useful if you want the openflow priorities be offset from the netcontrol priorities without having to write a filter function.
+
+      check_pred: :bro:type:`function` (p: :bro:type:`NetControl::PluginState`, r: :bro:type:`NetControl::Rule`) : :bro:type:`bool` :bro:attr:`&optional`
+         Predicate that is called on rule insertion or removal.
+         
+
+         :p: Current plugin state.
+         
+
+         :r: The rule to be inserted or removed.
+         
+
+         :returns: T if the rule can be handled by the current backend, F otherwise.
+
+      match_pred: :bro:type:`function` (p: :bro:type:`NetControl::PluginState`, e: :bro:type:`NetControl::Entity`, m: :bro:type:`vector` of :bro:type:`OpenFlow::ofp_match`) : :bro:type:`vector` of :bro:type:`OpenFlow::ofp_match` :bro:attr:`&optional`
+         This predicate is called each time an OpenFlow match record is created.
+         The predicate can modify the match structure before it is sent on to the
+         device.
+         
+
+         :p: Current plugin state.
+         
+
+         :r: The rule to be inserted or removed.
+         
+
+         :m: The openflow match structures that were generated for this rules.
+         
+
+         :returns: The modified OpenFlow match structures that will be used in place of the structures passed in m.
+
+      flow_mod_pred: :bro:type:`function` (p: :bro:type:`NetControl::PluginState`, r: :bro:type:`NetControl::Rule`, m: :bro:type:`OpenFlow::ofp_flow_mod`) : :bro:type:`OpenFlow::ofp_flow_mod` :bro:attr:`&optional`
+         This predicate is called before a FlowMod message is sent to the OpenFlow
+         device. It can modify the FlowMod message before it is passed on.
+         
+
+         :p: Current plugin state.
+         
+
+         :r: The rule to be inserted or removed.
+         
+
+         :m: The OpenFlow FlowMod message.
+         
+
+         :returns: The modified FlowMod message that is used in lieu of m.
+
+   This record specifies the configuration that is passed to :bro:see:`NetControl::create_openflow`.
+
+.. bro:type:: NetControl::OfTable
+
+   :Type: :bro:type:`record`
+
+      p: :bro:type:`NetControl::PluginState`
+
+      r: :bro:type:`NetControl::Rule`
+
+      c: :bro:type:`count` :bro:attr:`&default` = ``0`` :bro:attr:`&optional`
+
+      packet_count: :bro:type:`count` :bro:attr:`&default` = ``0`` :bro:attr:`&optional`
+
+      byte_count: :bro:type:`count` :bro:attr:`&default` = ``0`` :bro:attr:`&optional`
+
+      duration_sec: :bro:type:`double` :bro:attr:`&default` = ``0.0`` :bro:attr:`&optional`
+
+
+Functions
+#########
+.. bro:id:: NetControl::create_openflow
+
+   :Type: :bro:type:`function` (controller: :bro:type:`OpenFlow::Controller`, config: :bro:type:`NetControl::OfConfig` :bro:attr:`&default` = ``[]`` :bro:attr:`&optional`) : :bro:type:`NetControl::PluginState`
+
+   Instantiates an openflow plugin for the NetControl framework.
+
+

doc/scripts/base/frameworks/netcontrol/plugins/packetfilter.bro.rst

@@ -0,0 +1,34 @@
+:tocdepth: 3
+
+base/frameworks/netcontrol/plugins/packetfilter.bro
+===================================================
+.. bro:namespace:: NetControl
+
+NetControl plugin for the process-level PacketFilter that comes with
+Bro. Since the PacketFilter in Bro is quite limited in scope
+and can only add/remove filters for addresses, this is quite
+limited in scope at the moment. 
+
+:Namespace: NetControl
+:Imports: :doc:`base/frameworks/netcontrol/plugin.bro </scripts/base/frameworks/netcontrol/plugin.bro>`
+
+Summary
+~~~~~~~
+Functions
+#########
+=============================================================== =====================================
+:bro:id:`NetControl::create_packetfilter`: :bro:type:`function` Instantiates the packetfilter plugin.
+=============================================================== =====================================
+
+
+Detailed Interface
+~~~~~~~~~~~~~~~~~~
+Functions
+#########
+.. bro:id:: NetControl::create_packetfilter
+
+   :Type: :bro:type:`function` () : :bro:type:`NetControl::PluginState`
+
+   Instantiates the packetfilter plugin.
+
+

doc/scripts/base/frameworks/netcontrol/shunt.bro.rst

@@ -0,0 +1,93 @@
+:tocdepth: 3
+
+base/frameworks/netcontrol/shunt.bro
+====================================
+.. bro:namespace:: NetControl
+
+Implementation of the shunt functionality for NetControl.
+
+:Namespace: NetControl
+:Imports: :doc:`base/frameworks/netcontrol/main.bro </scripts/base/frameworks/netcontrol/main.bro>`
+
+Summary
+~~~~~~~
+Types
+#####
+===================================================== =
+:bro:type:`NetControl::ShuntInfo`: :bro:type:`record` 
+===================================================== =
+
+Redefinitions
+#############
+===================================== =
+:bro:type:`Log::ID`: :bro:type:`enum` 
+===================================== =
+
+Events
+######
+============================================================= =========================================================================
+:bro:id:`NetControl::log_netcontrol_shunt`: :bro:type:`event` Event that can be handled to access the :bro:type:`NetControl::ShuntInfo`
+                                                              record as it is sent on to the logging framework.
+============================================================= =========================================================================
+
+Functions
+#########
+====================================================== =========================================================
+:bro:id:`NetControl::shunt_flow`: :bro:type:`function` Stops forwarding a uni-directional flow's packets to Bro.
+====================================================== =========================================================
+
+
+Detailed Interface
+~~~~~~~~~~~~~~~~~~
+Types
+#####
+.. bro:type:: NetControl::ShuntInfo
+
+   :Type: :bro:type:`record`
+
+      ts: :bro:type:`time` :bro:attr:`&log`
+         Time at which the recorded activity occurred.
+
+      rule_id: :bro:type:`string` :bro:attr:`&log`
+         ID of the rule; unique during each Bro run.
+
+      f: :bro:type:`flow_id` :bro:attr:`&log`
+         Flow ID of the shunted flow.
+
+      expire: :bro:type:`interval` :bro:attr:`&log`
+         Expiry time of the shunt.
+
+      location: :bro:type:`string` :bro:attr:`&log` :bro:attr:`&optional`
+         Location where the underlying action was triggered.
+
+
+Events
+######
+.. bro:id:: NetControl::log_netcontrol_shunt
+
+   :Type: :bro:type:`event` (rec: :bro:type:`NetControl::ShuntInfo`)
+
+   Event that can be handled to access the :bro:type:`NetControl::ShuntInfo`
+   record as it is sent on to the logging framework.
+
+Functions
+#########
+.. bro:id:: NetControl::shunt_flow
+
+   :Type: :bro:type:`function` (f: :bro:type:`flow_id`, t: :bro:type:`interval`, location: :bro:type:`string` :bro:attr:`&default` = ``""`` :bro:attr:`&optional`) : :bro:type:`string`
+
+   Stops forwarding a uni-directional flow's packets to Bro.
+   
+
+   :f: The flow to shunt.
+   
+
+   :t: How long to leave the shunt in place, with 0 being indefinitely.
+   
+
+   :location: An optional string describing where the shunt was triggered.
+   
+
+   :returns: The id of the inserted rule on success and zero on failure.
+
+

doc/scripts/base/frameworks/netcontrol/types.bro.rst

@@ -0,0 +1,296 @@
+:tocdepth: 3
+
+base/frameworks/netcontrol/types.bro
+====================================
+.. bro:namespace:: NetControl
+
+This file defines the types that are used by the NetControl framework.
+
+The most important type defined in this file is :bro:see:`NetControl::Rule`,
+which is used to describe all rules that can be expressed by the NetControl framework. 
+
+:Namespace: NetControl
+
+Summary
+~~~~~~~
+Runtime Options
+###############
+========================================================================== ======================================================
+:bro:id:`NetControl::default_priority`: :bro:type:`int` :bro:attr:`&redef` The default priority that is used when creating rules.
+========================================================================== ======================================================
+
+Redefinable Options
+###################
+============================================================================ ====================================================================================
+:bro:id:`NetControl::whitelist_priority`: :bro:type:`int` :bro:attr:`&redef` The default priority that is used when using the high-level functions to
+                                                                             push whitelist entries to the backends (:bro:see:`NetControl::whitelist_address` and
+                                                                             :bro:see:`NetControl::whitelist_subnet`).
+============================================================================ ====================================================================================
+
+Types
+#####
+==================================================== ====================================================================================================
+:bro:type:`NetControl::Entity`: :bro:type:`record`   Type defining the entity a rule is operating on.
+:bro:type:`NetControl::EntityType`: :bro:type:`enum` Type defining the entity that a rule applies to.
+:bro:type:`NetControl::Flow`: :bro:type:`record`     Flow is used in :bro:type:`NetControl::Entity` together with :bro:enum:`NetControl::FLOW` to specify
+                                                     a uni-directional flow that a rule applies to.
+:bro:type:`NetControl::FlowInfo`: :bro:type:`record` Information of a flow that can be provided by switches when the flow times out.
+:bro:type:`NetControl::FlowMod`: :bro:type:`record`  Type for defining a flow modification action.
+:bro:type:`NetControl::Rule`: :bro:type:`record`     A rule for the framework to put in place.
+:bro:type:`NetControl::RuleType`: :bro:type:`enum`   Type of rules that the framework supports.
+:bro:type:`NetControl::TargetType`: :bro:type:`enum` Type defining the target of a rule.
+==================================================== ====================================================================================================
+
+
+Detailed Interface
+~~~~~~~~~~~~~~~~~~
+Runtime Options
+###############
+.. bro:id:: NetControl::default_priority
+
+   :Type: :bro:type:`int`
+   :Attributes: :bro:attr:`&redef`
+   :Default: ``0``
+
+   The default priority that is used when creating rules.
+
+Redefinable Options
+###################
+.. bro:id:: NetControl::whitelist_priority
+
+   :Type: :bro:type:`int`
+   :Attributes: :bro:attr:`&redef`
+   :Default: ``5``
+
+   The default priority that is used when using the high-level functions to
+   push whitelist entries to the backends (:bro:see:`NetControl::whitelist_address` and
+   :bro:see:`NetControl::whitelist_subnet`).
+   
+   Note that this priority is not automatically used when manually creating rules
+   that have a :bro:see:`NetControl::RuleType` of :bro:enum:`NetControl::WHITELIST`.
+
+Types
+#####
+.. bro:type:: NetControl::Entity
+
+   :Type: :bro:type:`record`
+
+      ty: :bro:type:`NetControl::EntityType`
+         Type of entity.
+
+      conn: :bro:type:`conn_id` :bro:attr:`&optional`
+         Used with :bro:enum:`NetControl::CONNECTION`.
+
+      flow: :bro:type:`NetControl::Flow` :bro:attr:`&optional`
+         Used with :bro:enum:`NetControl::FLOW`.
+
+      ip: :bro:type:`subnet` :bro:attr:`&optional`
+         Used with :bro:enum:`NetControl::ADDRESS` to specifiy a CIDR subnet.
+
+      mac: :bro:type:`string` :bro:attr:`&optional`
+         Used with :bro:enum:`NetControl::MAC`.
+
+   Type defining the entity a rule is operating on.
+
+.. bro:type:: NetControl::EntityType
+
+   :Type: :bro:type:`enum`
+
+      .. bro:enum:: NetControl::ADDRESS NetControl::EntityType
+
+         Activity involving a specific IP address.
+
+      .. bro:enum:: NetControl::CONNECTION NetControl::EntityType
+
+         Activity involving all of a bi-directional connection's activity.
+
+      .. bro:enum:: NetControl::FLOW NetControl::EntityType
+
+         Activity involving a uni-directional flow's activity. Can contain wildcards.
+
+      .. bro:enum:: NetControl::MAC NetControl::EntityType
+
+         Activity involving a MAC address.
+
+   Type defining the entity that a rule applies to.
+
+.. bro:type:: NetControl::Flow
+
+   :Type: :bro:type:`record`
+
+      src_h: :bro:type:`subnet` :bro:attr:`&optional`
+         The source IP address/subnet.
+
+      src_p: :bro:type:`port` :bro:attr:`&optional`
+         The source port number.
+
+      dst_h: :bro:type:`subnet` :bro:attr:`&optional`
+         The destination IP address/subnet.
+
+      dst_p: :bro:type:`port` :bro:attr:`&optional`
+         The destination port number.
+
+      src_m: :bro:type:`string` :bro:attr:`&optional`
+         The source MAC address.
+
+      dst_m: :bro:type:`string` :bro:attr:`&optional`
+         The destination MAC address.
+
+   Flow is used in :bro:type:`NetControl::Entity` together with :bro:enum:`NetControl::FLOW` to specify
+   a uni-directional flow that a rule applies to.
+   
+   If optional fields are not set, they are interpreted as wildcarded.
+
+.. bro:type:: NetControl::FlowInfo
+
+   :Type: :bro:type:`record`
+
+      duration: :bro:type:`interval` :bro:attr:`&optional`
+         Total duration of the rule.
+
+      packet_count: :bro:type:`count` :bro:attr:`&optional`
+         Number of packets exchanged over connections matched by the rule.
+
+      byte_count: :bro:type:`count` :bro:attr:`&optional`
+         Total bytes exchanged over connections matched by the rule.
+
+   Information of a flow that can be provided by switches when the flow times out.
+   Currently this is heavily influenced by the data that OpenFlow returns by default.
+   That being said - their design makes sense and this is probably the data one
+   can expect to be available.
+
+.. bro:type:: NetControl::FlowMod
+
+   :Type: :bro:type:`record`
+
+      src_h: :bro:type:`addr` :bro:attr:`&optional`
+         The source IP address.
+
+      src_p: :bro:type:`count` :bro:attr:`&optional`
+         The source port number.
+
+      dst_h: :bro:type:`addr` :bro:attr:`&optional`
+         The destination IP address.
+
+      dst_p: :bro:type:`count` :bro:attr:`&optional`
+         The destination port number.
+
+      src_m: :bro:type:`string` :bro:attr:`&optional`
+         The source MAC address.
+
+      dst_m: :bro:type:`string` :bro:attr:`&optional`
+         The destination MAC address.
+
+      redirect_port: :bro:type:`count` :bro:attr:`&optional`
+
+   Type for defining a flow modification action.
+
+.. bro:type:: NetControl::Rule
+
+   :Type: :bro:type:`record`
+
+      ty: :bro:type:`NetControl::RuleType`
+         Type of rule.
+
+      target: :bro:type:`NetControl::TargetType`
+         Where to apply rule.
+
+      entity: :bro:type:`NetControl::Entity`
+         Entity to apply rule to.
+
+      expire: :bro:type:`interval` :bro:attr:`&optional`
+         Timeout after which to expire the rule.
+
+      priority: :bro:type:`int` :bro:attr:`&default` = :bro:see:`NetControl::default_priority` :bro:attr:`&optional`
+         Priority if multiple rules match an entity (larger value is higher priority).
+
+      location: :bro:type:`string` :bro:attr:`&optional`
+         Optional string describing where/what installed the rule.
+
+      out_port: :bro:type:`count` :bro:attr:`&optional`
+         Argument for :bro:enum:`NetControl::REDIRECT` rules.
+
+      mod: :bro:type:`NetControl::FlowMod` :bro:attr:`&optional`
+         Argument for :bro:enum:`NetControl::MODIFY` rules.
+
+      id: :bro:type:`string` :bro:attr:`&default` = ``""`` :bro:attr:`&optional`
+         Internally determined unique ID for this rule. Will be set when added.
+
+      cid: :bro:type:`count` :bro:attr:`&default` = ``0`` :bro:attr:`&optional`
+         Internally determined unique numeric ID for this rule. Set when added.
+
+      _plugin_ids: :bro:type:`set` [:bro:type:`count`] :bro:attr:`&default` = ``{  }`` :bro:attr:`&optional`
+         (present if :doc:`/scripts/base/frameworks/netcontrol/main.bro` is loaded)
+
+         Internally set to the plugins handling the rule.
+
+      _active_plugin_ids: :bro:type:`set` [:bro:type:`count`] :bro:attr:`&default` = ``{  }`` :bro:attr:`&optional`
+         (present if :doc:`/scripts/base/frameworks/netcontrol/main.bro` is loaded)
+
+         Internally set to the plugins on which the rule is currently active.
+
+      _no_expire_plugins: :bro:type:`set` [:bro:type:`count`] :bro:attr:`&default` = ``{  }`` :bro:attr:`&optional`
+         (present if :doc:`/scripts/base/frameworks/netcontrol/main.bro` is loaded)
+
+         Internally set to plugins where the rule should not be removed upon timeout.
+
+      _added: :bro:type:`bool` :bro:attr:`&default` = ``F`` :bro:attr:`&optional`
+         (present if :doc:`/scripts/base/frameworks/netcontrol/main.bro` is loaded)
+
+         Track if the rule was added successfully by all responsible plugins.
+
+   A rule for the framework to put in place. Of all rules currently in
+   place, the first match will be taken, sorted by priority. All
+   further rules will be ignored.
+
+.. bro:type:: NetControl::RuleType
+
+   :Type: :bro:type:`enum`
+
+      .. bro:enum:: NetControl::DROP NetControl::RuleType
+
+         Stop forwarding all packets matching the entity.
+         
+         No additional arguments.
+
+      .. bro:enum:: NetControl::MODIFY NetControl::RuleType
+
+         Modify all packets matching entity. The packets
+         will be modified according to the `mod` entry of
+         the rule.
+         
+
+      .. bro:enum:: NetControl::REDIRECT NetControl::RuleType
+
+         Redirect all packets matching entity to a different switch port,
+         given in the `out_port` argument of the rule.
+         
+
+      .. bro:enum:: NetControl::WHITELIST NetControl::RuleType
+
+         Whitelists all packets of an entity, meaning no restrictions will be applied.
+         While whitelisting is the default if no rule matches, this type can be
+         used to override lower-priority rules that would otherwise take effect for the
+         entity.
+
+   Type of rules that the framework supports. Each type lists the extra
+   :bro:type:`NetControl::Rule` fields it uses, if any.
+   
+   Plugins may extend this type to define their own.
+
+.. bro:type:: NetControl::TargetType
+
+   :Type: :bro:type:`enum`
+
+      .. bro:enum:: NetControl::FORWARD NetControl::TargetType
+
+      .. bro:enum:: NetControl::MONITOR NetControl::TargetType
+
+   Type defining the target of a rule.
+   
+   Rules can either be applied to the forward path, affecting all network traffic, or
+   on the monitor path, only affecting the traffic that is sent to Bro. The second
+   is mostly used for shunting, which allows Bro to tell the networking hardware that
+   it wants to no longer see traffic that it identified as benign.
+
+

doc/scripts/base/frameworks/notice/__load__.bro.rst

@@ -0,0 +1,14 @@
+:tocdepth: 3
+
+base/frameworks/notice/__load__.bro
+===================================
+
+
+:Imports: :doc:`base/frameworks/notice/actions/add-geodata.bro </scripts/base/frameworks/notice/actions/add-geodata.bro>`, :doc:`base/frameworks/notice/actions/drop.bro </scripts/base/frameworks/notice/actions/drop.bro>`, :doc:`base/frameworks/notice/actions/email_admin.bro </scripts/base/frameworks/notice/actions/email_admin.bro>`, :doc:`base/frameworks/notice/actions/page.bro </scripts/base/frameworks/notice/actions/page.bro>`, :doc:`base/frameworks/notice/actions/pp-alarms.bro </scripts/base/frameworks/notice/actions/pp-alarms.bro>`, :doc:`base/frameworks/notice/main.bro </scripts/base/frameworks/notice/main.bro>`, :doc:`base/frameworks/notice/weird.bro </scripts/base/frameworks/notice/weird.bro>`
+
+Summary
+~~~~~~~
+
+Detailed Interface
+~~~~~~~~~~~~~~~~~~
+

doc/scripts/base/frameworks/notice/actions/add-geodata.bro.rst

@@ -0,0 +1,45 @@
+:tocdepth: 3
+
+base/frameworks/notice/actions/add-geodata.bro
+==============================================
+.. bro:namespace:: Notice
+
+This script adds geographic location data to notices for the "remote"
+host in a connection.  It does make the assumption that one of the 
+addresses in a connection is "local" and one is "remote" which is 
+probably a safe assumption to make in most cases.  If both addresses
+are remote, it will use the $src address.
+
+:Namespace: Notice
+:Imports: :doc:`base/frameworks/notice </scripts/base/frameworks/notice/index>`, :doc:`base/frameworks/notice/main.bro </scripts/base/frameworks/notice/main.bro>`, :doc:`base/utils/site.bro </scripts/base/utils/site.bro>`
+
+Summary
+~~~~~~~
+Runtime Options
+###############
+=========================================================================== ===============================================================
+:bro:id:`Notice::lookup_location_types`: :bro:type:`set` :bro:attr:`&redef` Notice types which should have the "remote" location looked up.
+=========================================================================== ===============================================================
+
+Redefinitions
+#############
+============================================ =
+:bro:type:`Notice::Action`: :bro:type:`enum` 
+:bro:type:`Notice::Info`: :bro:type:`record` 
+============================================ =
+
+
+Detailed Interface
+~~~~~~~~~~~~~~~~~~
+Runtime Options
+###############
+.. bro:id:: Notice::lookup_location_types
+
+   :Type: :bro:type:`set` [:bro:type:`Notice::Type`]
+   :Attributes: :bro:attr:`&redef`
+   :Default: ``{}``
+
+   Notice types which should have the "remote" location looked up.
+   If GeoIP support is not built in, this does nothing.
+
+

doc/scripts/base/frameworks/notice/actions/drop.bro.rst

@@ -0,0 +1,25 @@
+:tocdepth: 3
+
+base/frameworks/notice/actions/drop.bro
+=======================================
+.. bro:namespace:: Notice
+
+This script extends the built in notice code to implement the IP address
+dropping functionality.
+
+:Namespace: Notice
+:Imports: :doc:`base/frameworks/netcontrol </scripts/base/frameworks/netcontrol/index>`, :doc:`base/frameworks/notice/main.bro </scripts/base/frameworks/notice/main.bro>`
+
+Summary
+~~~~~~~
+Redefinitions
+#############
+============================================ =
+:bro:type:`Notice::Action`: :bro:type:`enum` 
+:bro:type:`Notice::Info`: :bro:type:`record` 
+============================================ =
+
+
+Detailed Interface
+~~~~~~~~~~~~~~~~~~
+

doc/scripts/base/frameworks/notice/actions/email_admin.bro.rst

@@ -0,0 +1,26 @@
+:tocdepth: 3
+
+base/frameworks/notice/actions/email_admin.bro
+==============================================
+.. bro:namespace:: Notice
+
+Adds a new notice action type which can be used to email notices
+to the administrators of a particular address space as set by
+:bro:id:`Site::local_admins` if the notice contains a source
+or destination address that lies within their space.
+
+:Namespace: Notice
+:Imports: :doc:`base/frameworks/notice/main.bro </scripts/base/frameworks/notice/main.bro>`, :doc:`base/utils/site.bro </scripts/base/utils/site.bro>`
+
+Summary
+~~~~~~~
+Redefinitions
+#############
+============================================ =
+:bro:type:`Notice::Action`: :bro:type:`enum` 
+============================================ =
+
+
+Detailed Interface
+~~~~~~~~~~~~~~~~~~
+

doc/scripts/base/frameworks/notice/actions/page.bro.rst

@@ -0,0 +1,41 @@
+:tocdepth: 3
+
+base/frameworks/notice/actions/page.bro
+=======================================
+.. bro:namespace:: Notice
+
+Allows configuration of a pager email address to which notices can be sent.
+
+:Namespace: Notice
+:Imports: :doc:`base/frameworks/notice/main.bro </scripts/base/frameworks/notice/main.bro>`
+
+Summary
+~~~~~~~
+Runtime Options
+###############
+======================================================================= ======================================================================
+:bro:id:`Notice::mail_page_dest`: :bro:type:`string` :bro:attr:`&redef` Email address to send notices with the :bro:enum:`Notice::ACTION_PAGE`
+                                                                        action.
+======================================================================= ======================================================================
+
+Redefinitions
+#############
+============================================ =
+:bro:type:`Notice::Action`: :bro:type:`enum` 
+============================================ =
+
+
+Detailed Interface
+~~~~~~~~~~~~~~~~~~
+Runtime Options
+###############
+.. bro:id:: Notice::mail_page_dest
+
+   :Type: :bro:type:`string`
+   :Attributes: :bro:attr:`&redef`
+   :Default: ``""``
+
+   Email address to send notices with the :bro:enum:`Notice::ACTION_PAGE`
+   action.
+
+

doc/scripts/base/frameworks/notice/actions/pp-alarms.bro.rst

@@ -0,0 +1,92 @@
+:tocdepth: 3
+
+base/frameworks/notice/actions/pp-alarms.bro
+============================================
+.. bro:namespace:: Notice
+
+Notice extension that mails out a pretty-printed version of alarm.log
+in regular intervals, formatted for better human readability. If activated,
+that replaces the default summary mail having the raw log output.
+
+:Namespace: Notice
+:Imports: :doc:`base/frameworks/cluster </scripts/base/frameworks/cluster/index>`, :doc:`base/frameworks/notice/main.bro </scripts/base/frameworks/notice/main.bro>`
+
+Summary
+~~~~~~~
+Redefinable Options
+###################
+================================================================================= ==============================================
+:bro:id:`Notice::mail_dest_pretty_printed`: :bro:type:`string` :bro:attr:`&redef` Address to send the pretty-printed reports to.
+:bro:id:`Notice::pretty_print_alarms`: :bro:type:`bool` :bro:attr:`&redef`        Activate pretty-printed alarm summaries.
+================================================================================= ==============================================
+
+State Variables
+###############
+============================================================================ ==================================================================
+:bro:id:`Notice::flag_nets`: :bro:type:`set` :bro:attr:`&redef`              If an address from one of these networks is reported, we mark
+                                                                             the entry with an additional quote symbol (i.e., ">").
+:bro:id:`Notice::force_email_summaries`: :bro:type:`bool` :bro:attr:`&redef` Force generating mail file, even if reading from traces or no mail
+                                                                             destination is defined.
+============================================================================ ==================================================================
+
+Functions
+#########
+============================================================================= =====================================
+:bro:id:`Notice::pretty_print_alarm`: :bro:type:`function` :bro:attr:`&redef` Function that renders a single alarm.
+============================================================================= =====================================
+
+
+Detailed Interface
+~~~~~~~~~~~~~~~~~~
+Redefinable Options
+###################
+.. bro:id:: Notice::mail_dest_pretty_printed
+
+   :Type: :bro:type:`string`
+   :Attributes: :bro:attr:`&redef`
+   :Default: ``""``
+
+   Address to send the pretty-printed reports to. Default if not set is
+   :bro:id:`Notice::mail_dest`.
+   
+   Note that this is overridden by the BroControl MailAlarmsTo option.
+
+.. bro:id:: Notice::pretty_print_alarms
+
+   :Type: :bro:type:`bool`
+   :Attributes: :bro:attr:`&redef`
+   :Default: ``T``
+
+   Activate pretty-printed alarm summaries.
+
+State Variables
+###############
+.. bro:id:: Notice::flag_nets
+
+   :Type: :bro:type:`set` [:bro:type:`subnet`]
+   :Attributes: :bro:attr:`&redef`
+   :Default: ``{}``
+
+   If an address from one of these networks is reported, we mark
+   the entry with an additional quote symbol (i.e., ">"). Many MUAs
+   then highlight such lines differently.
+
+.. bro:id:: Notice::force_email_summaries
+
+   :Type: :bro:type:`bool`
+   :Attributes: :bro:attr:`&redef`
+   :Default: ``F``
+
+   Force generating mail file, even if reading from traces or no mail
+   destination is defined. This is mainly for testing.
+
+Functions
+#########
+.. bro:id:: Notice::pretty_print_alarm
+
+   :Type: :bro:type:`function` (out: :bro:type:`file`, n: :bro:type:`Notice::Info`) : :bro:type:`void`
+   :Attributes: :bro:attr:`&redef`
+
+   Function that renders a single alarm. Can be overridden.
+
+

doc/scripts/base/frameworks/notice/index.rst

@@ -0,0 +1,62 @@
+:orphan:
+
+Package: base/frameworks/notice
+===============================
+
+The notice framework enables Bro to "notice" things which are odd or
+potentially bad, leaving it to the local configuration to define which
+of them are actionable.  This decoupling of detection and reporting allows
+Bro to be customized to the different needs that sites have.
+
+:doc:`/scripts/base/frameworks/notice/__load__.bro`
+
+
+:doc:`/scripts/base/frameworks/notice/main.bro`
+
+   This is the notice framework which enables Bro to "notice" things which
+   are odd or potentially bad.  Decisions of the meaning of various notices
+   need to be done per site because Bro does not ship with assumptions about
+   what is bad activity for sites.  More extensive documentation about using
+   the notice framework can be found in :doc:`/frameworks/notice`.
+
+:doc:`/scripts/base/frameworks/notice/weird.bro`
+
+   This script provides a default set of actions to take for "weird activity"
+   events generated from Bro's event engine.  Weird activity is defined as
+   unusual or exceptional activity that can indicate malformed connections,
+   traffic that doesn't conform to a particular protocol, malfunctioning
+   or misconfigured hardware, or even an attacker attempting to avoid/confuse
+   a sensor.  Without context, it's hard to judge whether a particular
+   category of weird activity is interesting, but this script provides
+   a starting point for the user.
+
+:doc:`/scripts/base/frameworks/notice/actions/drop.bro`
+
+   This script extends the built in notice code to implement the IP address
+   dropping functionality.
+
+:doc:`/scripts/base/frameworks/notice/actions/email_admin.bro`
+
+   Adds a new notice action type which can be used to email notices
+   to the administrators of a particular address space as set by
+   :bro:id:`Site::local_admins` if the notice contains a source
+   or destination address that lies within their space.
+
+:doc:`/scripts/base/frameworks/notice/actions/page.bro`
+
+   Allows configuration of a pager email address to which notices can be sent.
+
+:doc:`/scripts/base/frameworks/notice/actions/add-geodata.bro`
+
+   This script adds geographic location data to notices for the "remote"
+   host in a connection.  It does make the assumption that one of the 
+   addresses in a connection is "local" and one is "remote" which is 
+   probably a safe assumption to make in most cases.  If both addresses
+   are remote, it will use the $src address.
+
+:doc:`/scripts/base/frameworks/notice/actions/pp-alarms.bro`
+
+   Notice extension that mails out a pretty-printed version of alarm.log
+   in regular intervals, formatted for better human readability. If activated,
+   that replaces the default summary mail having the raw log output.
+

doc/scripts/base/frameworks/notice/weird.bro.rst

@@ -0,0 +1,412 @@
+:tocdepth: 3
+
+base/frameworks/notice/weird.bro
+================================
+.. bro:namespace:: Weird
+
+This script provides a default set of actions to take for "weird activity"
+events generated from Bro's event engine.  Weird activity is defined as
+unusual or exceptional activity that can indicate malformed connections,
+traffic that doesn't conform to a particular protocol, malfunctioning
+or misconfigured hardware, or even an attacker attempting to avoid/confuse
+a sensor.  Without context, it's hard to judge whether a particular
+category of weird activity is interesting, but this script provides
+a starting point for the user.
+
+:Namespace: Weird
+:Imports: :doc:`base/frameworks/notice/main.bro </scripts/base/frameworks/notice/main.bro>`, :doc:`base/utils/conn-ids.bro </scripts/base/utils/conn-ids.bro>`, :doc:`base/utils/site.bro </scripts/base/utils/site.bro>`
+
+Summary
+~~~~~~~
+Runtime Options
+###############
+================================================================================ ==============================================================
+:bro:id:`Weird::ignore_hosts`: :bro:type:`set` :bro:attr:`&redef`                To completely ignore a specific weird for a host, add the host
+                                                                                 and weird name into this set.
+:bro:id:`Weird::weird_do_not_ignore_repeats`: :bro:type:`set` :bro:attr:`&redef` Don't ignore repeats for weirds in this set.
+================================================================================ ==============================================================
+
+Redefinable Options
+###################
+================================================================================================================================= ==============================================================
+:bro:id:`Weird::actions`: :bro:type:`table` :bro:attr:`&default` = ``Weird::ACTION_LOG`` :bro:attr:`&optional` :bro:attr:`&redef` A table specifying default/recommended actions per weird type.
+================================================================================================================================= ==============================================================
+
+State Variables
+###############
+============================================================================================================ ====================================================================
+:bro:id:`Weird::did_log`: :bro:type:`set` :bro:attr:`&create_expire` = ``1.0 day`` :bro:attr:`&redef`        A state set which tracks unique weirds solely by name to reduce
+                                                                                                             duplicate logging.
+:bro:id:`Weird::did_notice`: :bro:type:`set` :bro:attr:`&create_expire` = ``1.0 day`` :bro:attr:`&redef`     A state set which tracks unique weirds solely by name to reduce
+                                                                                                             duplicate notices from being raised.
+:bro:id:`Weird::weird_ignore`: :bro:type:`set` :bro:attr:`&create_expire` = ``10.0 mins`` :bro:attr:`&redef` This table is used to track identifier and name pairs that should be
+                                                                                                             temporarily ignored because the problem has already been reported.
+============================================================================================================ ====================================================================
+
+Types
+#####
+=========================================== =======================================================================
+:bro:type:`Weird::Action`: :bro:type:`enum` Types of actions that may be taken when handling weird activity events.
+:bro:type:`Weird::Info`: :bro:type:`record` The record which is used for representing and logging weirds.
+=========================================== =======================================================================
+
+Redefinitions
+#############
+========================================== ====================================
+:bro:type:`Log::ID`: :bro:type:`enum`      The weird logging stream identifier.
+:bro:type:`Notice::Type`: :bro:type:`enum` 
+========================================== ====================================
+
+Events
+######
+============================================= ==============================================================
+:bro:id:`Weird::log_weird`: :bro:type:`event` Handlers of this event are invoked once per write to the weird
+                                              logging stream before the data is actually written.
+============================================= ==============================================================
+
+Functions
+#########
+============================================ =
+:bro:id:`Weird::weird`: :bro:type:`function` 
+============================================ =
+
+
+Detailed Interface
+~~~~~~~~~~~~~~~~~~
+Runtime Options
+###############
+.. bro:id:: Weird::ignore_hosts
+
+   :Type: :bro:type:`set` [:bro:type:`addr`, :bro:type:`string`]
+   :Attributes: :bro:attr:`&redef`
+   :Default: ``{}``
+
+   To completely ignore a specific weird for a host, add the host
+   and weird name into this set.
+
+.. bro:id:: Weird::weird_do_not_ignore_repeats
+
+   :Type: :bro:type:`set` [:bro:type:`string`]
+   :Attributes: :bro:attr:`&redef`
+   :Default:
+
+   ::
+
+      {
+         "bad_ICMP_checksum",
+         "bad_TCP_checksum",
+         "bad_IP_checksum",
+         "bad_UDP_checksum"
+      }
+
+   Don't ignore repeats for weirds in this set.  For example,
+   it's handy keeping track of clustered checksum errors.
+
+Redefinable Options
+###################
+.. bro:id:: Weird::actions
+
+   :Type: :bro:type:`table` [:bro:type:`string`] of :bro:type:`Weird::Action`
+   :Attributes: :bro:attr:`&default` = ``Weird::ACTION_LOG`` :bro:attr:`&optional` :bro:attr:`&redef`
+   :Default:
+
+   ::
+
+      {
+         ["DNS_AAAA_neg_length"] = Weird::ACTION_LOG,
+         ["partial_ftp_request"] = Weird::ACTION_LOG,
+         ["repeated_SYN_reply_wo_ack"] = Weird::ACTION_LOG,
+         ["bad_UDP_checksum"] = Weird::ACTION_LOG_PER_ORIG,
+         ["line_terminated_with_single_LF"] = Weird::ACTION_LOG,
+         ["truncated_IP"] = Weird::ACTION_LOG,
+         ["DNS_truncated_len_lt_hdr_len"] = Weird::ACTION_LOG,
+         ["excessive_data_without_further_acks"] = Weird::ACTION_LOG,
+         ["pop3_malformed_auth_plain"] = Weird::ACTION_LOG,
+         ["excess_netbios_hdr_len"] = Weird::ACTION_LOG,
+         ["irc_invalid_whois_channel_line"] = Weird::ACTION_LOG,
+         ["bad_RPC"] = Weird::ACTION_LOG_PER_ORIG,
+         ["unknown_netbios_type"] = Weird::ACTION_LOG,
+         ["HTTP_chunked_transfer_for_multipart_message"] = Weird::ACTION_LOG,
+         ["RST_storm"] = Weird::ACTION_LOG,
+         ["bad_IP_checksum"] = Weird::ACTION_LOG_PER_ORIG,
+         ["excessively_small_fragment"] = Weird::ACTION_LOG_PER_ORIG,
+         ["bad_rsh_prolog"] = Weird::ACTION_LOG,
+         ["pop3_client_sending_server_commands"] = Weird::ACTION_LOG,
+         ["unexpected_multiple_HTTP_requests"] = Weird::ACTION_LOG,
+         ["irc_invalid_topic_reply"] = Weird::ACTION_LOG,
+         ["irc_invalid_squery_message_format"] = Weird::ACTION_LOG,
+         ["bad_SYN_ack"] = Weird::ACTION_LOG,
+         ["contentline_size_exceeded"] = Weird::ACTION_LOG,
+         ["above_hole_data_without_any_acks"] = Weird::ACTION_LOG,
+         ["bad_HTTP_reply"] = Weird::ACTION_LOG,
+         ["DNS_RR_length_mismatch"] = Weird::ACTION_LOG,
+         ["SMB_parsing_error"] = Weird::ACTION_LOG,
+         ["multiple_HTTP_request_elements"] = Weird::ACTION_LOG,
+         ["FIN_after_reset"] = Weird::ACTION_IGNORE,
+         ["SYN_after_partial"] = Weird::ACTION_NOTICE_PER_ORIG,
+         ["baroque_SYN"] = Weird::ACTION_LOG,
+         ["DNS_label_forward_compress_offset"] = Weird::ACTION_LOG_PER_ORIG,
+         ["connection_originator_SYN_ack"] = Weird::ACTION_LOG_PER_ORIG,
+         ["irc_invalid_dcc_message_format"] = Weird::ACTION_LOG,
+         ["unmatched_HTTP_reply"] = Weird::ACTION_LOG,
+         ["unpaired_RPC_response"] = Weird::ACTION_LOG,
+         ["SYN_inside_connection"] = Weird::ACTION_LOG,
+         ["irc_invalid_who_message_format"] = Weird::ACTION_LOG,
+         ["irc_invalid_reply_number"] = Weird::ACTION_LOG,
+         ["pop3_client_command_unknown"] = Weird::ACTION_LOG,
+         ["bad_ICMP_checksum"] = Weird::ACTION_LOG_PER_ORIG,
+         ["DNS_RR_unknown_type"] = Weird::ACTION_LOG,
+         ["excessively_large_fragment"] = Weird::ACTION_LOG,
+         ["DNS_label_len_gt_name_len"] = Weird::ACTION_LOG_PER_ORIG,
+         ["DNS_label_len_gt_pkt"] = Weird::ACTION_LOG_PER_ORIG,
+         ["partial_ident_request"] = Weird::ACTION_LOG,
+         ["excess_RPC"] = Weird::ACTION_LOG_PER_ORIG,
+         ["line_terminated_with_single_CR"] = Weird::ACTION_LOG,
+         ["unknown_HTTP_method"] = Weird::ACTION_LOG,
+         ["bad_ident_request"] = Weird::ACTION_LOG,
+         ["crud_trailing_HTTP_request"] = Weird::ACTION_LOG,
+         ["irc_invalid_whois_operator_line"] = Weird::ACTION_LOG,
+         ["unexpected_server_HTTP_data"] = Weird::ACTION_LOG,
+         ["irc_invalid_njoin_line"] = Weird::ACTION_LOG,
+         ["irc_invalid_mode_message_format"] = Weird::ACTION_LOG,
+         ["pop3_bad_base64_encoding"] = Weird::ACTION_LOG,
+         ["responder_RPC_call"] = Weird::ACTION_LOG_PER_ORIG,
+         ["fragment_size_inconsistency"] = Weird::ACTION_LOG_PER_ORIG,
+         ["successful_RPC_reply_to_invalid_request"] = Weird::ACTION_NOTICE_PER_ORIG,
+         ["irc_line_too_short"] = Weird::ACTION_LOG,
+         ["irc_invalid_kick_message_format"] = Weird::ACTION_LOG,
+         ["repeated_SYN_with_ack"] = Weird::ACTION_LOG,
+         ["partial_finger_request"] = Weird::ACTION_LOG,
+         ["irc_invalid_join_line"] = Weird::ACTION_LOG,
+         ["premature_connection_reuse"] = Weird::ACTION_LOG,
+         ["netbios_raw_session_msg"] = Weird::ACTION_LOG,
+         ["incompletely_captured_fragment"] = Weird::ACTION_LOG,
+         ["malformed_ssh_version"] = Weird::ACTION_LOG,
+         ["netbios_client_session_reply"] = Weird::ACTION_LOG,
+         ["bad_TCP_header_len"] = Weird::ACTION_LOG,
+         ["unescaped_%_in_URI"] = Weird::ACTION_LOG,
+         ["netbios_server_session_request"] = Weird::ACTION_LOG,
+         ["irc_too_many_invalid"] = Weird::ACTION_LOG,
+         ["irc_invalid_names_line"] = Weird::ACTION_LOG,
+         ["RPC_rexmit_inconsistency"] = Weird::ACTION_LOG,
+         ["smb_andx_command_failed_to_parse"] = Weird::ACTION_LOG,
+         ["irc_invalid_invite_message_format"] = Weird::ACTION_LOG,
+         ["spontaneous_FIN"] = Weird::ACTION_IGNORE,
+         ["DNS_truncated_quest_too_short"] = Weird::ACTION_LOG,
+         ["SSL_many_server_names"] = Weird::ACTION_LOG,
+         ["FIN_storm"] = Weird::ACTION_NOTICE_PER_ORIG,
+         ["data_before_established"] = Weird::ACTION_LOG,
+         ["SYN_after_reset"] = Weird::ACTION_LOG,
+         ["double_%_in_URI"] = Weird::ACTION_LOG,
+         ["DNS_truncated_ans_too_short"] = Weird::ACTION_LOG,
+         ["DNS_Conn_count_too_large"] = Weird::ACTION_LOG,
+         ["data_after_reset"] = Weird::ACTION_LOG,
+         ["RPC_underflow"] = Weird::ACTION_LOG,
+         ["unexpected_client_HTTP_data"] = Weird::ACTION_LOG,
+         ["originator_RPC_reply"] = Weird::ACTION_LOG_PER_ORIG,
+         ["DNS_label_too_long"] = Weird::ACTION_LOG_PER_ORIG,
+         ["SYN_with_data"] = Weird::ACTION_LOG_PER_ORIG,
+         ["RST_with_data"] = Weird::ACTION_LOG,
+         ["bad_HTTP_version"] = Weird::ACTION_LOG,
+         ["pending_data_when_closed"] = Weird::ACTION_LOG,
+         ["rlogin_text_after_rejected"] = Weird::ACTION_LOG,
+         ["FIN_advanced_last_seq"] = Weird::ACTION_LOG,
+         ["transaction_subcmd_missing"] = Weird::ACTION_LOG,
+         ["fragment_protocol_inconsistency"] = Weird::ACTION_LOG,
+         ["invalid_irc_global_users_reply"] = Weird::ACTION_LOG,
+         ["ident_request_addendum"] = Weird::ACTION_LOG,
+         ["window_recision"] = Weird::ACTION_LOG,
+         ["spontaneous_RST"] = Weird::ACTION_IGNORE,
+         ["truncated_header"] = Weird::ACTION_LOG,
+         ["UDP_datagram_length_mismatch"] = Weird::ACTION_LOG_PER_ORIG,
+         ["fragment_with_DF"] = Weird::ACTION_LOG,
+         ["SYN_after_close"] = Weird::ACTION_LOG,
+         ["SYN_seq_jump"] = Weird::ACTION_LOG,
+         ["irc_invalid_notice_message_format"] = Weird::ACTION_LOG,
+         ["irc_invalid_command"] = Weird::ACTION_LOG,
+         ["DNS_NAME_too_long"] = Weird::ACTION_LOG,
+         ["inflate_failed"] = Weird::ACTION_LOG,
+         ["base64_illegal_encoding"] = Weird::ACTION_LOG,
+         ["internally_truncated_header"] = Weird::ACTION_LOG,
+         ["pop3_server_sending_client_commands"] = Weird::ACTION_LOG,
+         ["irc_invalid_who_line"] = Weird::ACTION_LOG,
+         ["irc_invalid_privmsg_message_format"] = Weird::ACTION_LOG,
+         ["pop3_server_command_unknown"] = Weird::ACTION_LOG,
+         ["fragment_overlap"] = Weird::ACTION_LOG_PER_ORIG,
+         ["bad_rlogin_prolog"] = Weird::ACTION_LOG,
+         ["bad_ident_port"] = Weird::ACTION_LOG,
+         ["irc_invalid_line"] = Weird::ACTION_LOG,
+         ["HTTP_overlapping_messages"] = Weird::ACTION_LOG,
+         ["simultaneous_open"] = Weird::ACTION_LOG_PER_CONN,
+         ["unsolicited_SYN_response"] = Weird::ACTION_IGNORE,
+         ["DNS_RR_bad_length"] = Weird::ACTION_LOG,
+         ["TCP_christmas"] = Weird::ACTION_LOG,
+         ["inappropriate_FIN"] = Weird::ACTION_LOG,
+         ["irc_invalid_oper_message_format"] = Weird::ACTION_LOG,
+         ["no_smb_session_using_parsesambamsg"] = Weird::ACTION_LOG,
+         ["illegal_%_at_end_of_URI"] = Weird::ACTION_LOG,
+         ["active_connection_reuse"] = Weird::ACTION_LOG,
+         ["bad_TCP_checksum"] = Weird::ACTION_LOG_PER_ORIG,
+         ["fragment_inconsistency"] = Weird::ACTION_LOG_PER_ORIG,
+         ["malformed_ssh_identification"] = Weird::ACTION_LOG,
+         ["DNS_truncated_RR_rdlength_lt_len"] = Weird::ACTION_LOG,
+         ["possible_split_routing"] = Weird::ACTION_LOG,
+         ["irc_line_size_exceeded"] = Weird::ACTION_LOG,
+         ["bad_RPC_program"] = Weird::ACTION_LOG,
+         ["bad_ident_reply"] = Weird::ACTION_LOG,
+         ["HTTP_bad_chunk_size"] = Weird::ACTION_LOG,
+         ["unescaped_special_URI_char"] = Weird::ACTION_LOG,
+         ["HTTP_version_mismatch"] = Weird::ACTION_LOG,
+         ["irc_invalid_whois_message_format"] = Weird::ACTION_LOG,
+         ["rsh_text_after_rejected"] = Weird::ACTION_LOG,
+         ["partial_RPC"] = Weird::ACTION_LOG_PER_ORIG,
+         ["truncated_ARP"] = Weird::ACTION_LOG,
+         ["truncated_NTP"] = Weird::ACTION_LOG,
+         ["irc_invalid_whois_user_line"] = Weird::ACTION_LOG,
+         ["NUL_in_line"] = Weird::ACTION_LOG,
+         ["deficit_netbios_hdr_len"] = Weird::ACTION_LOG
+      }
+
+   A table specifying default/recommended actions per weird type.
+
+State Variables
+###############
+.. bro:id:: Weird::did_log
+
+   :Type: :bro:type:`set` [:bro:type:`string`, :bro:type:`string`]
+   :Attributes: :bro:attr:`&create_expire` = ``1.0 day`` :bro:attr:`&redef`
+   :Default: ``{}``
+
+   A state set which tracks unique weirds solely by name to reduce
+   duplicate logging.  This is deliberately not synchronized because it
+   could cause overload during storms.
+
+.. bro:id:: Weird::did_notice
+
+   :Type: :bro:type:`set` [:bro:type:`string`, :bro:type:`string`]
+   :Attributes: :bro:attr:`&create_expire` = ``1.0 day`` :bro:attr:`&redef`
+   :Default: ``{}``
+
+   A state set which tracks unique weirds solely by name to reduce
+   duplicate notices from being raised.
+
+.. bro:id:: Weird::weird_ignore
+
+   :Type: :bro:type:`set` [:bro:type:`string`, :bro:type:`string`]
+   :Attributes: :bro:attr:`&create_expire` = ``10.0 mins`` :bro:attr:`&redef`
+   :Default: ``{}``
+
+   This table is used to track identifier and name pairs that should be
+   temporarily ignored because the problem has already been reported.
+   This helps reduce the volume of high volume weirds by only allowing 
+   a unique weird every ``create_expire`` interval.
+
+Types
+#####
+.. bro:type:: Weird::Action
+
+   :Type: :bro:type:`enum`
+
+      .. bro:enum:: Weird::ACTION_UNSPECIFIED Weird::Action
+
+         A dummy action indicating the user does not care what
+         internal decision is made regarding a given type of weird.
+
+      .. bro:enum:: Weird::ACTION_IGNORE Weird::Action
+
+         No action is to be taken.
+
+      .. bro:enum:: Weird::ACTION_LOG Weird::Action
+
+         Log the weird event every time it occurs.
+
+      .. bro:enum:: Weird::ACTION_LOG_ONCE Weird::Action
+
+         Log the weird event only once.
+
+      .. bro:enum:: Weird::ACTION_LOG_PER_CONN Weird::Action
+
+         Log the weird event once per connection.
+
+      .. bro:enum:: Weird::ACTION_LOG_PER_ORIG Weird::Action
+
+         Log the weird event once per originator host.
+
+      .. bro:enum:: Weird::ACTION_NOTICE Weird::Action
+
+         Always generate a notice associated with the weird event.
+
+      .. bro:enum:: Weird::ACTION_NOTICE_ONCE Weird::Action
+
+         Generate a notice associated with the weird event only once.
+
+      .. bro:enum:: Weird::ACTION_NOTICE_PER_CONN Weird::Action
+
+         Generate a notice for the weird event once per connection.
+
+      .. bro:enum:: Weird::ACTION_NOTICE_PER_ORIG Weird::Action
+
+         Generate a notice for the weird event once per originator host.
+
+   Types of actions that may be taken when handling weird activity events.
+
+.. bro:type:: Weird::Info
+
+   :Type: :bro:type:`record`
+
+      ts: :bro:type:`time` :bro:attr:`&log`
+         The time when the weird occurred.
+
+      uid: :bro:type:`string` :bro:attr:`&log` :bro:attr:`&optional`
+         If a connection is associated with this weird, this will be
+         the connection's unique ID.
+
+      id: :bro:type:`conn_id` :bro:attr:`&log` :bro:attr:`&optional`
+         conn_id for the optional connection.
+
+      conn: :bro:type:`connection` :bro:attr:`&optional`
+         A shorthand way of giving the uid and id to a weird.
+
+      name: :bro:type:`string` :bro:attr:`&log`
+         The name of the weird that occurred.
+
+      addl: :bro:type:`string` :bro:attr:`&log` :bro:attr:`&optional`
+         Additional information accompanying the weird if any.
+
+      notice: :bro:type:`bool` :bro:attr:`&log` :bro:attr:`&default` = ``F`` :bro:attr:`&optional`
+         Indicate if this weird was also turned into a notice.
+
+      peer: :bro:type:`string` :bro:attr:`&log` :bro:attr:`&optional` :bro:attr:`&default` = :bro:see:`peer_description`
+         The peer that originated this weird.  This is helpful in
+         cluster deployments if a particular cluster node is having
+         trouble to help identify which node is having trouble.
+
+      identifier: :bro:type:`string` :bro:attr:`&optional`
+         This field is to be provided when a weird is generated for
+         the purpose of deduplicating weirds. The identifier string
+         should be unique for a single instance of the weird. This field
+         is used to define when a weird is conceptually a duplicate of
+         a previous weird.
+
+   The record which is used for representing and logging weirds.
+
+Events
+######
+.. bro:id:: Weird::log_weird
+
+   :Type: :bro:type:`event` (rec: :bro:type:`Weird::Info`)
+
+   Handlers of this event are invoked once per write to the weird
+   logging stream before the data is actually written.
+   
+
+   :rec: The weird columns about to be logged to the weird stream.
+
+Functions
+#########
+.. bro:id:: Weird::weird
+
+   :Type: :bro:type:`function` (w: :bro:type:`Weird::Info`) : :bro:type:`void`
+
+
+

doc/scripts/base/frameworks/openflow/__load__.bro.rst

@@ -0,0 +1,14 @@
+:tocdepth: 3
+
+base/frameworks/openflow/__load__.bro
+=====================================
+
+
+:Imports: :doc:`base/frameworks/cluster </scripts/base/frameworks/cluster/index>`, :doc:`base/frameworks/openflow/consts.bro </scripts/base/frameworks/openflow/consts.bro>`, :doc:`base/frameworks/openflow/main.bro </scripts/base/frameworks/openflow/main.bro>`, :doc:`base/frameworks/openflow/non-cluster.bro </scripts/base/frameworks/openflow/non-cluster.bro>`, :doc:`base/frameworks/openflow/plugins </scripts/base/frameworks/openflow/plugins/index>`, :doc:`base/frameworks/openflow/types.bro </scripts/base/frameworks/openflow/types.bro>`
+
+Summary
+~~~~~~~
+
+Detailed Interface
+~~~~~~~~~~~~~~~~~~
+

doc/scripts/base/frameworks/openflow/consts.bro.rst

@@ -0,0 +1,564 @@
+:tocdepth: 3
+
+base/frameworks/openflow/consts.bro
+===================================
+.. bro:namespace:: OpenFlow
+
+Constants used by the OpenFlow framework.
+
+:Namespace: OpenFlow
+
+Summary
+~~~~~~~
+Constants
+#########
+============================================================= ======================================================================
+:bro:id:`OpenFlow::ETH_APPLETALK`: :bro:type:`count`          
+:bro:id:`OpenFlow::ETH_APPLETALK_ARP`: :bro:type:`count`      
+:bro:id:`OpenFlow::ETH_ARP`: :bro:type:`count`                
+:bro:id:`OpenFlow::ETH_EAP_OVER_LAN`: :bro:type:`count`       
+:bro:id:`OpenFlow::ETH_ETHER_FLOW_CONTROL`: :bro:type:`count` 
+:bro:id:`OpenFlow::ETH_IPX`: :bro:type:`count`                
+:bro:id:`OpenFlow::ETH_IPX_OLD`: :bro:type:`count`            
+:bro:id:`OpenFlow::ETH_IPv4`: :bro:type:`count`               
+:bro:id:`OpenFlow::ETH_IPv6`: :bro:type:`count`               
+:bro:id:`OpenFlow::ETH_JUMBO_FRAMES`: :bro:type:`count`       
+:bro:id:`OpenFlow::ETH_MAC_SECURITY`: :bro:type:`count`       
+:bro:id:`OpenFlow::ETH_MPLS_MULTICAST`: :bro:type:`count`     
+:bro:id:`OpenFlow::ETH_MPLS_UNICAST`: :bro:type:`count`       
+:bro:id:`OpenFlow::ETH_PPPOE_DISCOVERY`: :bro:type:`count`    
+:bro:id:`OpenFlow::ETH_PPPOE_SESSION`: :bro:type:`count`      
+:bro:id:`OpenFlow::ETH_PROVIDER_BRIDING`: :bro:type:`count`   
+:bro:id:`OpenFlow::ETH_QINQ`: :bro:type:`count`               
+:bro:id:`OpenFlow::ETH_RARP`: :bro:type:`count`               
+:bro:id:`OpenFlow::ETH_VLAN`: :bro:type:`count`               
+:bro:id:`OpenFlow::ETH_WOL`: :bro:type:`count`                
+:bro:id:`OpenFlow::INVALID_COOKIE`: :bro:type:`count`         Return value for a cookie from a flow
+                                                              which is not added, modified or deleted
+                                                              from the bro openflow framework.
+:bro:id:`OpenFlow::IP_CBT`: :bro:type:`count`                 
+:bro:id:`OpenFlow::IP_EGP`: :bro:type:`count`                 
+:bro:id:`OpenFlow::IP_ETHERIP`: :bro:type:`count`             
+:bro:id:`OpenFlow::IP_FC`: :bro:type:`count`                  
+:bro:id:`OpenFlow::IP_GGP`: :bro:type:`count`                 
+:bro:id:`OpenFlow::IP_GRE`: :bro:type:`count`                 
+:bro:id:`OpenFlow::IP_HOPOPT`: :bro:type:`count`              
+:bro:id:`OpenFlow::IP_ICMP`: :bro:type:`count`                
+:bro:id:`OpenFlow::IP_IGMP`: :bro:type:`count`                
+:bro:id:`OpenFlow::IP_IGP`: :bro:type:`count`                 
+:bro:id:`OpenFlow::IP_IPIP`: :bro:type:`count`                
+:bro:id:`OpenFlow::IP_IPv6`: :bro:type:`count`                
+:bro:id:`OpenFlow::IP_ISIS`: :bro:type:`count`                
+:bro:id:`OpenFlow::IP_L2TP`: :bro:type:`count`                
+:bro:id:`OpenFlow::IP_MPLS`: :bro:type:`count`                
+:bro:id:`OpenFlow::IP_MTP`: :bro:type:`count`                 
+:bro:id:`OpenFlow::IP_OSPF`: :bro:type:`count`                
+:bro:id:`OpenFlow::IP_RDP`: :bro:type:`count`                 
+:bro:id:`OpenFlow::IP_RSVP`: :bro:type:`count`                
+:bro:id:`OpenFlow::IP_ST`: :bro:type:`count`                  
+:bro:id:`OpenFlow::IP_TCP`: :bro:type:`count`                 
+:bro:id:`OpenFlow::IP_UDP`: :bro:type:`count`                 
+:bro:id:`OpenFlow::OFPFF_CHECK_OVERLAP`: :bro:type:`count`    Check for overlapping entries first.
+:bro:id:`OpenFlow::OFPFF_EMERG`: :bro:type:`count`            Remark this is for emergency.
+:bro:id:`OpenFlow::OFPFF_SEND_FLOW_REM`: :bro:type:`count`    Send flow removed message when flow
+                                                              expires or is deleted.
+:bro:id:`OpenFlow::OFPP_ALL`: :bro:type:`count`               All physical ports except input port.
+:bro:id:`OpenFlow::OFPP_ANY`: :bro:type:`count`               Wildcard port used only for flow mod (delete) and flow stats requests.
+:bro:id:`OpenFlow::OFPP_CONTROLLER`: :bro:type:`count`        Send to controller.
+:bro:id:`OpenFlow::OFPP_FLOOD`: :bro:type:`count`             All physical ports except input port and
+                                                              those disabled by STP.
+:bro:id:`OpenFlow::OFPP_IN_PORT`: :bro:type:`count`           Send the packet out the input port.
+:bro:id:`OpenFlow::OFPP_LOCAL`: :bro:type:`count`             Local openflow "port".
+:bro:id:`OpenFlow::OFPP_NORMAL`: :bro:type:`count`            Process with normal L2/L3 switching.
+:bro:id:`OpenFlow::OFPP_TABLE`: :bro:type:`count`             Perform actions in flow table.
+:bro:id:`OpenFlow::OFPTT_ALL`: :bro:type:`count`              
+:bro:id:`OpenFlow::OFP_NO_BUFFER`: :bro:type:`count`          
+============================================================= ======================================================================
+
+Types
+#####
+============================================================ ======================================
+:bro:type:`OpenFlow::ofp_action_type`: :bro:type:`enum`      Openflow action_type definitions.
+:bro:type:`OpenFlow::ofp_config_flags`: :bro:type:`enum`     Openflow config flag definitions.
+:bro:type:`OpenFlow::ofp_flow_mod_command`: :bro:type:`enum` Openflow flow_mod_command definitions.
+============================================================ ======================================
+
+
+Detailed Interface
+~~~~~~~~~~~~~~~~~~
+Constants
+#########
+.. bro:id:: OpenFlow::ETH_APPLETALK
+
+   :Type: :bro:type:`count`
+   :Default: ``32923``
+
+
+.. bro:id:: OpenFlow::ETH_APPLETALK_ARP
+
+   :Type: :bro:type:`count`
+   :Default: ``33011``
+
+
+.. bro:id:: OpenFlow::ETH_ARP
+
+   :Type: :bro:type:`count`
+   :Default: ``2054``
+
+
+.. bro:id:: OpenFlow::ETH_EAP_OVER_LAN
+
+   :Type: :bro:type:`count`
+   :Default: ``34958``
+
+
+.. bro:id:: OpenFlow::ETH_ETHER_FLOW_CONTROL
+
+   :Type: :bro:type:`count`
+   :Default: ``34824``
+
+
+.. bro:id:: OpenFlow::ETH_IPX
+
+   :Type: :bro:type:`count`
+   :Default: ``33080``
+
+
+.. bro:id:: OpenFlow::ETH_IPX_OLD
+
+   :Type: :bro:type:`count`
+   :Default: ``33079``
+
+
+.. bro:id:: OpenFlow::ETH_IPv4
+
+   :Type: :bro:type:`count`
+   :Default: ``2048``
+
+
+.. bro:id:: OpenFlow::ETH_IPv6
+
+   :Type: :bro:type:`count`
+   :Default: ``34525``
+
+
+.. bro:id:: OpenFlow::ETH_JUMBO_FRAMES
+
+   :Type: :bro:type:`count`
+   :Default: ``34928``
+
+
+.. bro:id:: OpenFlow::ETH_MAC_SECURITY
+
+   :Type: :bro:type:`count`
+   :Default: ``35045``
+
+
+.. bro:id:: OpenFlow::ETH_MPLS_MULTICAST
+
+   :Type: :bro:type:`count`
+   :Default: ``34888``
+
+
+.. bro:id:: OpenFlow::ETH_MPLS_UNICAST
+
+   :Type: :bro:type:`count`
+   :Default: ``34887``
+
+
+.. bro:id:: OpenFlow::ETH_PPPOE_DISCOVERY
+
+   :Type: :bro:type:`count`
+   :Default: ``34915``
+
+
+.. bro:id:: OpenFlow::ETH_PPPOE_SESSION
+
+   :Type: :bro:type:`count`
+   :Default: ``34916``
+
+
+.. bro:id:: OpenFlow::ETH_PROVIDER_BRIDING
+
+   :Type: :bro:type:`count`
+   :Default: ``34984``
+
+
+.. bro:id:: OpenFlow::ETH_QINQ
+
+   :Type: :bro:type:`count`
+   :Default: ``37120``
+
+
+.. bro:id:: OpenFlow::ETH_RARP
+
+   :Type: :bro:type:`count`
+   :Default: ``32821``
+
+
+.. bro:id:: OpenFlow::ETH_VLAN
+
+   :Type: :bro:type:`count`
+   :Default: ``33024``
+
+
+.. bro:id:: OpenFlow::ETH_WOL
+
+   :Type: :bro:type:`count`
+   :Default: ``2114``
+
+
+.. bro:id:: OpenFlow::INVALID_COOKIE
+
+   :Type: :bro:type:`count`
+   :Default: ``18446744073709551615``
+
+   Return value for a cookie from a flow
+   which is not added, modified or deleted
+   from the bro openflow framework.
+
+.. bro:id:: OpenFlow::IP_CBT
+
+   :Type: :bro:type:`count`
+   :Default: ``7``
+
+
+.. bro:id:: OpenFlow::IP_EGP
+
+   :Type: :bro:type:`count`
+   :Default: ``8``
+
+
+.. bro:id:: OpenFlow::IP_ETHERIP
+
+   :Type: :bro:type:`count`
+   :Default: ``97``
+
+
+.. bro:id:: OpenFlow::IP_FC
+
+   :Type: :bro:type:`count`
+   :Default: ``133``
+
+
+.. bro:id:: OpenFlow::IP_GGP
+
+   :Type: :bro:type:`count`
+   :Default: ``3``
+
+
+.. bro:id:: OpenFlow::IP_GRE
+
+   :Type: :bro:type:`count`
+   :Default: ``47``
+
+
+.. bro:id:: OpenFlow::IP_HOPOPT
+
+   :Type: :bro:type:`count`
+   :Default: ``0``
+
+
+.. bro:id:: OpenFlow::IP_ICMP
+
+   :Type: :bro:type:`count`
+   :Default: ``1``
+
+
+.. bro:id:: OpenFlow::IP_IGMP
+
+   :Type: :bro:type:`count`
+   :Default: ``2``
+
+
+.. bro:id:: OpenFlow::IP_IGP
+
+   :Type: :bro:type:`count`
+   :Default: ``9``
+
+
+.. bro:id:: OpenFlow::IP_IPIP
+
+   :Type: :bro:type:`count`
+   :Default: ``4``
+
+
+.. bro:id:: OpenFlow::IP_IPv6
+
+   :Type: :bro:type:`count`
+   :Default: ``41``
+
+
+.. bro:id:: OpenFlow::IP_ISIS
+
+   :Type: :bro:type:`count`
+   :Default: ``124``
+
+
+.. bro:id:: OpenFlow::IP_L2TP
+
+   :Type: :bro:type:`count`
+   :Default: ``115``
+
+
+.. bro:id:: OpenFlow::IP_MPLS
+
+   :Type: :bro:type:`count`
+   :Default: ``137``
+
+
+.. bro:id:: OpenFlow::IP_MTP
+
+   :Type: :bro:type:`count`
+   :Default: ``92``
+
+
+.. bro:id:: OpenFlow::IP_OSPF
+
+   :Type: :bro:type:`count`
+   :Default: ``89``
+
+
+.. bro:id:: OpenFlow::IP_RDP
+
+   :Type: :bro:type:`count`
+   :Default: ``27``
+
+
+.. bro:id:: OpenFlow::IP_RSVP
+
+   :Type: :bro:type:`count`
+   :Default: ``46``
+
+
+.. bro:id:: OpenFlow::IP_ST
+
+   :Type: :bro:type:`count`
+   :Default: ``5``
+
+
+.. bro:id:: OpenFlow::IP_TCP
+
+   :Type: :bro:type:`count`
+   :Default: ``6``
+
+
+.. bro:id:: OpenFlow::IP_UDP
+
+   :Type: :bro:type:`count`
+   :Default: ``17``
+
+
+.. bro:id:: OpenFlow::OFPFF_CHECK_OVERLAP
+
+   :Type: :bro:type:`count`
+   :Default: ``2``
+
+   Check for overlapping entries first.
+
+.. bro:id:: OpenFlow::OFPFF_EMERG
+
+   :Type: :bro:type:`count`
+   :Default: ``4``
+
+   Remark this is for emergency.
+   Flows added with this are only used
+   when the controller is disconnected.
+
+.. bro:id:: OpenFlow::OFPFF_SEND_FLOW_REM
+
+   :Type: :bro:type:`count`
+   :Default: ``1``
+
+   Send flow removed message when flow
+   expires or is deleted.
+
+.. bro:id:: OpenFlow::OFPP_ALL
+
+   :Type: :bro:type:`count`
+   :Default: ``4294967292``
+
+   All physical ports except input port.
+
+.. bro:id:: OpenFlow::OFPP_ANY
+
+   :Type: :bro:type:`count`
+   :Default: ``4294967295``
+
+   Wildcard port used only for flow mod (delete) and flow stats requests.
+
+.. bro:id:: OpenFlow::OFPP_CONTROLLER
+
+   :Type: :bro:type:`count`
+   :Default: ``4294967293``
+
+   Send to controller.
+
+.. bro:id:: OpenFlow::OFPP_FLOOD
+
+   :Type: :bro:type:`count`
+   :Default: ``4294967291``
+
+   All physical ports except input port and
+   those disabled by STP.
+
+.. bro:id:: OpenFlow::OFPP_IN_PORT
+
+   :Type: :bro:type:`count`
+   :Default: ``4294967288``
+
+   Send the packet out the input port. This
+   virual port must be explicitly used in
+   order to send back out of the input port.
+
+.. bro:id:: OpenFlow::OFPP_LOCAL
+
+   :Type: :bro:type:`count`
+   :Default: ``4294967294``
+
+   Local openflow "port".
+
+.. bro:id:: OpenFlow::OFPP_NORMAL
+
+   :Type: :bro:type:`count`
+   :Default: ``4294967290``
+
+   Process with normal L2/L3 switching.
+
+.. bro:id:: OpenFlow::OFPP_TABLE
+
+   :Type: :bro:type:`count`
+   :Default: ``4294967289``
+
+   Perform actions in flow table.
+   NB: This can only be the destination port
+   for packet-out messages.
+
+.. bro:id:: OpenFlow::OFPTT_ALL
+
+   :Type: :bro:type:`count`
+   :Default: ``255``
+
+
+.. bro:id:: OpenFlow::OFP_NO_BUFFER
+
+   :Type: :bro:type:`count`
+   :Default: ``4294967295``
+
+
+Types
+#####
+.. bro:type:: OpenFlow::ofp_action_type
+
+   :Type: :bro:type:`enum`
+
+      .. bro:enum:: OpenFlow::OFPAT_OUTPUT OpenFlow::ofp_action_type
+
+         Output to switch port.
+
+      .. bro:enum:: OpenFlow::OFPAT_SET_VLAN_VID OpenFlow::ofp_action_type
+
+         Set the 802.1q VLAN id.
+
+      .. bro:enum:: OpenFlow::OFPAT_SET_VLAN_PCP OpenFlow::ofp_action_type
+
+         Set the 802.1q priority.
+
+      .. bro:enum:: OpenFlow::OFPAT_STRIP_VLAN OpenFlow::ofp_action_type
+
+         Strip the 802.1q header.
+
+      .. bro:enum:: OpenFlow::OFPAT_SET_DL_SRC OpenFlow::ofp_action_type
+
+         Ethernet source address.
+
+      .. bro:enum:: OpenFlow::OFPAT_SET_DL_DST OpenFlow::ofp_action_type
+
+         Ethernet destination address.
+
+      .. bro:enum:: OpenFlow::OFPAT_SET_NW_SRC OpenFlow::ofp_action_type
+
+         IP source address.
+
+      .. bro:enum:: OpenFlow::OFPAT_SET_NW_DST OpenFlow::ofp_action_type
+
+         IP destination address.
+
+      .. bro:enum:: OpenFlow::OFPAT_SET_NW_TOS OpenFlow::ofp_action_type
+
+         IP ToS (DSCP field, 6 bits).
+
+      .. bro:enum:: OpenFlow::OFPAT_SET_TP_SRC OpenFlow::ofp_action_type
+
+         TCP/UDP source port.
+
+      .. bro:enum:: OpenFlow::OFPAT_SET_TP_DST OpenFlow::ofp_action_type
+
+         TCP/UDP destination port.
+
+      .. bro:enum:: OpenFlow::OFPAT_ENQUEUE OpenFlow::ofp_action_type
+
+         Output to queue.
+
+      .. bro:enum:: OpenFlow::OFPAT_VENDOR OpenFlow::ofp_action_type
+
+         Vendor specific.
+
+   Openflow action_type definitions.
+   
+   The openflow action type defines
+   what actions openflow can take
+   to modify a packet
+
+.. bro:type:: OpenFlow::ofp_config_flags
+
+   :Type: :bro:type:`enum`
+
+      .. bro:enum:: OpenFlow::OFPC_FRAG_NORMAL OpenFlow::ofp_config_flags
+
+         No special handling for fragments.
+
+      .. bro:enum:: OpenFlow::OFPC_FRAG_DROP OpenFlow::ofp_config_flags
+
+         Drop fragments.
+
+      .. bro:enum:: OpenFlow::OFPC_FRAG_REASM OpenFlow::ofp_config_flags
+
+         Reassemble (only if OFPC_IP_REASM set).
+
+      .. bro:enum:: OpenFlow::OFPC_FRAG_MASK OpenFlow::ofp_config_flags
+
+   Openflow config flag definitions.
+   
+   TODO: describe
+
+.. bro:type:: OpenFlow::ofp_flow_mod_command
+
+   :Type: :bro:type:`enum`
+
+      .. bro:enum:: OpenFlow::OFPFC_ADD OpenFlow::ofp_flow_mod_command
+
+         New flow.
+
+      .. bro:enum:: OpenFlow::OFPFC_MODIFY OpenFlow::ofp_flow_mod_command
+
+         Modify all matching flows.
+
+      .. bro:enum:: OpenFlow::OFPFC_MODIFY_STRICT OpenFlow::ofp_flow_mod_command
+
+         Modify entry strictly matching wildcards.
+
+      .. bro:enum:: OpenFlow::OFPFC_DELETE OpenFlow::ofp_flow_mod_command
+
+         Delete all matching flows.
+
+      .. bro:enum:: OpenFlow::OFPFC_DELETE_STRICT OpenFlow::ofp_flow_mod_command
+
+         Strictly matching wildcards and priority.
+
+   Openflow flow_mod_command definitions.
+   
+   The openflow flow_mod_command describes
+   of what kind an action is.
+
+

doc/scripts/base/frameworks/openflow/index.rst

@@ -0,0 +1,50 @@
+:orphan:
+
+Package: base/frameworks/openflow
+=================================
+
+The OpenFlow framework exposes the data structures and functions
+necessary to interface to OpenFlow capable hardware.
+
+:doc:`/scripts/base/frameworks/openflow/__load__.bro`
+
+
+:doc:`/scripts/base/frameworks/openflow/consts.bro`
+
+   Constants used by the OpenFlow framework.
+
+:doc:`/scripts/base/frameworks/openflow/types.bro`
+
+   Types used by the OpenFlow framework.
+
+:doc:`/scripts/base/frameworks/openflow/main.bro`
+
+   Bro's OpenFlow control framework.
+   
+   This plugin-based framework allows to control OpenFlow capable
+   switches by implementing communication to an OpenFlow controller
+   via plugins. The framework has to be instantiated via the new function
+   in one of the plugins. This framework only offers very low-level
+   functionality; if you want to use OpenFlow capable switches, e.g.,
+   for shunting, please look at the NetControl framework, which provides higher
+   level functions and can use the OpenFlow framework as a backend.
+
+:doc:`/scripts/base/frameworks/openflow/plugins/__load__.bro`
+
+
+:doc:`/scripts/base/frameworks/openflow/plugins/ryu.bro`
+
+   OpenFlow plugin for the Ryu controller.
+
+:doc:`/scripts/base/frameworks/openflow/plugins/log.bro`
+
+   OpenFlow plugin that outputs flow-modification commands
+   to a Bro log file.
+
+:doc:`/scripts/base/frameworks/openflow/plugins/broker.bro`
+
+   OpenFlow plugin for interfacing to controllers via Broker.
+
+:doc:`/scripts/base/frameworks/openflow/non-cluster.bro`
+
+

doc/scripts/base/frameworks/openflow/main.bro.rst

@@ -0,0 +1,265 @@
+:tocdepth: 3
+
+base/frameworks/openflow/main.bro
+=================================
+.. bro:namespace:: OpenFlow
+
+Bro's OpenFlow control framework.
+
+This plugin-based framework allows to control OpenFlow capable
+switches by implementing communication to an OpenFlow controller
+via plugins. The framework has to be instantiated via the new function
+in one of the plugins. This framework only offers very low-level
+functionality; if you want to use OpenFlow capable switches, e.g.,
+for shunting, please look at the NetControl framework, which provides higher
+level functions and can use the OpenFlow framework as a backend.
+
+:Namespace: OpenFlow
+:Imports: :doc:`base/frameworks/openflow/consts.bro </scripts/base/frameworks/openflow/consts.bro>`, :doc:`base/frameworks/openflow/types.bro </scripts/base/frameworks/openflow/types.bro>`
+
+Summary
+~~~~~~~
+Events
+######
+=========================================================== =============================================================================================
+:bro:id:`OpenFlow::controller_activated`: :bro:type:`event` Event that is raised once a controller finishes initialization
+                                                            and is completely activated.
+:bro:id:`OpenFlow::flow_mod_failure`: :bro:type:`event`     Reports an error while installing a flow Rule.
+:bro:id:`OpenFlow::flow_mod_success`: :bro:type:`event`     Event confirming successful modification of a flow rule.
+:bro:id:`OpenFlow::flow_removed`: :bro:type:`event`         Reports that a flow was removed by the switch because of either the hard or the idle timeout.
+=========================================================== =============================================================================================
+
+Functions
+#########
+=============================================================== =====================================================================
+:bro:id:`OpenFlow::controller_init_done`: :bro:type:`function`  Function to signal that a controller finished activation and is
+                                                                ready to use.
+:bro:id:`OpenFlow::flow_clear`: :bro:type:`function`            Clear the current flow table of the controller.
+:bro:id:`OpenFlow::flow_mod`: :bro:type:`function`              Global flow_mod function.
+:bro:id:`OpenFlow::generate_cookie`: :bro:type:`function`       Function to generate a new cookie using our group id.
+:bro:id:`OpenFlow::get_cookie_gid`: :bro:type:`function`        Function to get the group id out of a given cookie.
+:bro:id:`OpenFlow::get_cookie_uid`: :bro:type:`function`        Function to get the unique id out of a given cookie.
+:bro:id:`OpenFlow::lookup_controller`: :bro:type:`function`     Function to lookup a controller instance by name.
+:bro:id:`OpenFlow::match_conn`: :bro:type:`function`            Convert a conn_id record into an ofp_match record that can be used to
+                                                                create match objects for OpenFlow.
+:bro:id:`OpenFlow::register_controller`: :bro:type:`function`   Function to register a controller instance.
+:bro:id:`OpenFlow::unregister_controller`: :bro:type:`function` Function to unregister a controller instance.
+=============================================================== =====================================================================
+
+
+Detailed Interface
+~~~~~~~~~~~~~~~~~~
+Events
+######
+.. bro:id:: OpenFlow::controller_activated
+
+   :Type: :bro:type:`event` (name: :bro:type:`string`, controller: :bro:type:`OpenFlow::Controller`)
+
+   Event that is raised once a controller finishes initialization
+   and is completely activated.
+
+   :name: Unique name of this controller instance.
+   
+
+   :controller: The controller that finished activation.
+
+.. bro:id:: OpenFlow::flow_mod_failure
+
+   :Type: :bro:type:`event` (name: :bro:type:`string`, match: :bro:type:`OpenFlow::ofp_match`, flow_mod: :bro:type:`OpenFlow::ofp_flow_mod`, msg: :bro:type:`string` :bro:attr:`&default` = ``""`` :bro:attr:`&optional`)
+
+   Reports an error while installing a flow Rule.
+   
+
+   :name: The unique name of the OpenFlow controller from which this event originated.
+   
+
+   :match: The ofp_match record which describes the flow to match.
+   
+
+   :flow_mod: The openflow flow_mod record which describes the action to take.
+   
+
+   :msg: Message to describe the event.
+
+.. bro:id:: OpenFlow::flow_mod_success
+
+   :Type: :bro:type:`event` (name: :bro:type:`string`, match: :bro:type:`OpenFlow::ofp_match`, flow_mod: :bro:type:`OpenFlow::ofp_flow_mod`, msg: :bro:type:`string` :bro:attr:`&default` = ``""`` :bro:attr:`&optional`)
+
+   Event confirming successful modification of a flow rule.
+   
+
+   :name: The unique name of the OpenFlow controller from which this event originated.
+   
+
+   :match: The ofp_match record which describes the flow to match.
+   
+
+   :flow_mod: The openflow flow_mod record which describes the action to take.
+   
+
+   :msg: An optional informational message by the plugin.
+
+.. bro:id:: OpenFlow::flow_removed
+
+   :Type: :bro:type:`event` (name: :bro:type:`string`, match: :bro:type:`OpenFlow::ofp_match`, cookie: :bro:type:`count`, priority: :bro:type:`count`, reason: :bro:type:`count`, duration_sec: :bro:type:`count`, idle_timeout: :bro:type:`count`, packet_count: :bro:type:`count`, byte_count: :bro:type:`count`)
+
+   Reports that a flow was removed by the switch because of either the hard or the idle timeout.
+   This message is only generated by controllers that indicate that they support flow removal
+   in supports_flow_removed.
+   
+
+   :name: The unique name of the OpenFlow controller from which this event originated.
+   
+
+   :match: The ofp_match record which was used to create the flow.
+   
+
+   :cookie: The cookie that was specified when creating the flow.
+   
+
+   :priority: The priority that was specified when creating the flow.
+   
+
+   :reason: The reason for flow removal (OFPRR_*).
+   
+
+   :duration_sec: Duration of the flow in seconds.
+   
+
+   :packet_count: Packet count of the flow.
+   
+
+   :byte_count: Byte count of the flow.
+
+Functions
+#########
+.. bro:id:: OpenFlow::controller_init_done
+
+   :Type: :bro:type:`function` (controller: :bro:type:`OpenFlow::Controller`) : :bro:type:`void`
+
+   Function to signal that a controller finished activation and is
+   ready to use. Will throw the ``OpenFlow::controller_activated``
+   event.
+
+.. bro:id:: OpenFlow::flow_clear
+
+   :Type: :bro:type:`function` (controller: :bro:type:`OpenFlow::Controller`) : :bro:type:`bool`
+
+   Clear the current flow table of the controller.
+   
+
+   :controller: The controller which should execute the flow modification.
+   
+
+   :returns: F on error or if the plugin does not support the operation, T when the operation was queued.
+
+.. bro:id:: OpenFlow::flow_mod
+
+   :Type: :bro:type:`function` (controller: :bro:type:`OpenFlow::Controller`, match: :bro:type:`OpenFlow::ofp_match`, flow_mod: :bro:type:`OpenFlow::ofp_flow_mod`) : :bro:type:`bool`
+
+   Global flow_mod function.
+   
+
+   :controller: The controller which should execute the flow modification.
+   
+
+   :match: The ofp_match record which describes the flow to match.
+   
+
+   :flow_mod: The openflow flow_mod record which describes the action to take.
+   
+
+   :returns: F on error or if the plugin does not support the operation, T when the operation was queued.
+
+.. bro:id:: OpenFlow::generate_cookie
+
+   :Type: :bro:type:`function` (cookie: :bro:type:`count` :bro:attr:`&default` = ``0`` :bro:attr:`&optional`) : :bro:type:`count`
+
+   Function to generate a new cookie using our group id.
+   
+
+   :cookie: The openflow match cookie.
+   
+
+   :returns: The cookie group id.
+
+.. bro:id:: OpenFlow::get_cookie_gid
+
+   :Type: :bro:type:`function` (cookie: :bro:type:`count`) : :bro:type:`count`
+
+   Function to get the group id out of a given cookie.
+   
+
+   :cookie: The openflow match cookie.
+   
+
+   :returns: The cookie group id.
+
+.. bro:id:: OpenFlow::get_cookie_uid
+
+   :Type: :bro:type:`function` (cookie: :bro:type:`count`) : :bro:type:`count`
+
+   Function to get the unique id out of a given cookie.
+   
+
+   :cookie: The openflow match cookie.
+   
+
+   :returns: The cookie unique id.
+
+.. bro:id:: OpenFlow::lookup_controller
+
+   :Type: :bro:type:`function` (name: :bro:type:`string`) : :bro:type:`vector` of :bro:type:`OpenFlow::Controller`
+
+   Function to lookup a controller instance by name.
+   
+
+   :name: Unique name of the controller to look up.
+   
+
+   :returns: One element vector with controller, if found. Empty vector otherwise.
+
+.. bro:id:: OpenFlow::match_conn
+
+   :Type: :bro:type:`function` (id: :bro:type:`conn_id`, reverse: :bro:type:`bool` :bro:attr:`&default` = ``F`` :bro:attr:`&optional`) : :bro:type:`OpenFlow::ofp_match`
+
+   Convert a conn_id record into an ofp_match record that can be used to
+   create match objects for OpenFlow.
+   
+
+   :id: The conn_id record that describes the record.
+   
+
+   :reverse: Reverse the sources and destinations when creating the match record (default F).
+   
+
+   :returns: ofp_match object for the conn_id record.
+
+.. bro:id:: OpenFlow::register_controller
+
+   :Type: :bro:type:`function` (tpe: :bro:type:`OpenFlow::Plugin`, name: :bro:type:`string`, controller: :bro:type:`OpenFlow::Controller`) : :bro:type:`void`
+
+   Function to register a controller instance. This function
+   is called automatically by the plugin _new functions.
+   
+
+   :tpe: Type of this plugin.
+   
+
+   :name: Unique name of this controller instance.
+   
+
+   :controller: The controller to register.
+
+.. bro:id:: OpenFlow::unregister_controller
+
+   :Type: :bro:type:`function` (controller: :bro:type:`OpenFlow::Controller`) : :bro:type:`void`
+
+   Function to unregister a controller instance. This function
+   should be called when a specific controller should no longer
+   be used.
+   
+
+   :controller: The controller to unregister.
+
+

doc/scripts/base/frameworks/openflow/non-cluster.bro.rst

@@ -0,0 +1,16 @@
+:tocdepth: 3
+
+base/frameworks/openflow/non-cluster.bro
+========================================
+.. bro:namespace:: OpenFlow
+
+
+:Namespace: OpenFlow
+:Imports: :doc:`base/frameworks/openflow/main.bro </scripts/base/frameworks/openflow/main.bro>`
+
+Summary
+~~~~~~~
+
+Detailed Interface
+~~~~~~~~~~~~~~~~~~
+

doc/scripts/base/frameworks/openflow/plugins/__load__.bro.rst

@@ -0,0 +1,14 @@
+:tocdepth: 3
+
+base/frameworks/openflow/plugins/__load__.bro
+=============================================
+
+
+:Imports: :doc:`base/frameworks/openflow/plugins/broker.bro </scripts/base/frameworks/openflow/plugins/broker.bro>`, :doc:`base/frameworks/openflow/plugins/log.bro </scripts/base/frameworks/openflow/plugins/log.bro>`, :doc:`base/frameworks/openflow/plugins/ryu.bro </scripts/base/frameworks/openflow/plugins/ryu.bro>`
+
+Summary
+~~~~~~~
+
+Detailed Interface
+~~~~~~~~~~~~~~~~~~
+

doc/scripts/base/frameworks/openflow/plugins/broker.bro.rst

@@ -0,0 +1,72 @@
+:tocdepth: 3
+
+base/frameworks/openflow/plugins/broker.bro
+===========================================
+.. bro:namespace:: OpenFlow
+
+OpenFlow plugin for interfacing to controllers via Broker.
+
+:Namespace: OpenFlow
+:Imports: :doc:`base/frameworks/broker </scripts/base/frameworks/broker/index>`, :doc:`base/frameworks/openflow </scripts/base/frameworks/openflow/index>`
+
+Summary
+~~~~~~~
+Redefinitions
+#############
+============================================================================ =
+:bro:type:`OpenFlow::ControllerState`: :bro:type:`record` :bro:attr:`&redef` 
+:bro:type:`OpenFlow::Plugin`: :bro:type:`enum`                               
+============================================================================ =
+
+Events
+######
+======================================================== =
+:bro:id:`OpenFlow::broker_flow_clear`: :bro:type:`event` 
+:bro:id:`OpenFlow::broker_flow_mod`: :bro:type:`event`   
+======================================================== =
+
+Functions
+#########
+==================================================== ==============================
+:bro:id:`OpenFlow::broker_new`: :bro:type:`function` Broker controller constructor.
+==================================================== ==============================
+
+
+Detailed Interface
+~~~~~~~~~~~~~~~~~~
+Events
+######
+.. bro:id:: OpenFlow::broker_flow_clear
+
+   :Type: :bro:type:`event` (name: :bro:type:`string`, dpid: :bro:type:`count`)
+
+
+.. bro:id:: OpenFlow::broker_flow_mod
+
+   :Type: :bro:type:`event` (name: :bro:type:`string`, dpid: :bro:type:`count`, match: :bro:type:`OpenFlow::ofp_match`, flow_mod: :bro:type:`OpenFlow::ofp_flow_mod`)
+
+
+Functions
+#########
+.. bro:id:: OpenFlow::broker_new
+
+   :Type: :bro:type:`function` (name: :bro:type:`string`, host: :bro:type:`addr`, host_port: :bro:type:`port`, topic: :bro:type:`string`, dpid: :bro:type:`count`) : :bro:type:`OpenFlow::Controller`
+
+   Broker controller constructor.
+   
+
+   :host: Controller ip.
+   
+
+   :host_port: Controller listen port.
+   
+
+   :topic: Broker topic to send messages to.
+   
+
+   :dpid: OpenFlow switch datapath id.
+   
+
+   :returns: OpenFlow::Controller record.
+
+

doc/scripts/base/frameworks/openflow/plugins/index.rst

@@ -0,0 +1,23 @@
+:orphan:
+
+Package: base/frameworks/openflow/plugins
+=========================================
+
+Plugins for the OpenFlow framework.
+
+:doc:`/scripts/base/frameworks/openflow/plugins/__load__.bro`
+
+
+:doc:`/scripts/base/frameworks/openflow/plugins/ryu.bro`
+
+   OpenFlow plugin for the Ryu controller.
+
+:doc:`/scripts/base/frameworks/openflow/plugins/log.bro`
+
+   OpenFlow plugin that outputs flow-modification commands
+   to a Bro log file.
+
+:doc:`/scripts/base/frameworks/openflow/plugins/broker.bro`
+
+   OpenFlow plugin for interfacing to controllers via Broker.
+

doc/scripts/base/frameworks/openflow/plugins/log.bro.rst

@@ -0,0 +1,91 @@
+:tocdepth: 3
+
+base/frameworks/openflow/plugins/log.bro
+========================================
+.. bro:namespace:: OpenFlow
+
+OpenFlow plugin that outputs flow-modification commands
+to a Bro log file.
+
+:Namespace: OpenFlow
+:Imports: :doc:`base/frameworks/logging </scripts/base/frameworks/logging/index>`, :doc:`base/frameworks/openflow </scripts/base/frameworks/openflow/index>`
+
+Summary
+~~~~~~~
+Types
+#####
+============================================== =================================================================
+:bro:type:`OpenFlow::Info`: :bro:type:`record` The record type which contains column fields of the OpenFlow log.
+============================================== =================================================================
+
+Redefinitions
+#############
+============================================================================ =
+:bro:type:`Log::ID`: :bro:type:`enum`                                        
+:bro:type:`OpenFlow::ControllerState`: :bro:type:`record` :bro:attr:`&redef` 
+:bro:type:`OpenFlow::Plugin`: :bro:type:`enum`                               
+============================================================================ =
+
+Events
+######
+=================================================== ==================================================================
+:bro:id:`OpenFlow::log_openflow`: :bro:type:`event` Event that can be handled to access the :bro:type:`OpenFlow::Info`
+                                                    record as it is sent on to the logging framework.
+=================================================== ==================================================================
+
+Functions
+#########
+================================================= ===========================
+:bro:id:`OpenFlow::log_new`: :bro:type:`function` Log controller constructor.
+================================================= ===========================
+
+
+Detailed Interface
+~~~~~~~~~~~~~~~~~~
+Types
+#####
+.. bro:type:: OpenFlow::Info
+
+   :Type: :bro:type:`record`
+
+      ts: :bro:type:`time` :bro:attr:`&log`
+         Network time.
+
+      dpid: :bro:type:`count` :bro:attr:`&log`
+         OpenFlow switch datapath id.
+
+      match: :bro:type:`OpenFlow::ofp_match` :bro:attr:`&log`
+         OpenFlow match fields.
+
+      flow_mod: :bro:type:`OpenFlow::ofp_flow_mod` :bro:attr:`&log`
+         OpenFlow modify flow entry message.
+
+   The record type which contains column fields of the OpenFlow log.
+
+Events
+######
+.. bro:id:: OpenFlow::log_openflow
+
+   :Type: :bro:type:`event` (rec: :bro:type:`OpenFlow::Info`)
+
+   Event that can be handled to access the :bro:type:`OpenFlow::Info`
+   record as it is sent on to the logging framework.
+
+Functions
+#########
+.. bro:id:: OpenFlow::log_new
+
+   :Type: :bro:type:`function` (dpid: :bro:type:`count`, success_event: :bro:type:`bool` :bro:attr:`&default` = ``T`` :bro:attr:`&optional`) : :bro:type:`OpenFlow::Controller`
+
+   Log controller constructor.
+   
+
+   :dpid: OpenFlow switch datapath id.
+   
+
+   :success_event: If true, flow_mod_success is raised for each logged line.
+   
+
+   :returns: OpenFlow::Controller record.
+
+

doc/scripts/base/frameworks/openflow/plugins/ryu.bro.rst

@@ -0,0 +1,50 @@
+:tocdepth: 3
+
+base/frameworks/openflow/plugins/ryu.bro
+========================================
+.. bro:namespace:: OpenFlow
+
+OpenFlow plugin for the Ryu controller.
+
+:Namespace: OpenFlow
+:Imports: :doc:`base/frameworks/openflow </scripts/base/frameworks/openflow/index>`, :doc:`base/utils/active-http.bro </scripts/base/utils/active-http.bro>`, :doc:`base/utils/exec.bro </scripts/base/utils/exec.bro>`, :doc:`base/utils/json.bro </scripts/base/utils/json.bro>`
+
+Summary
+~~~~~~~
+Redefinitions
+#############
+============================================================================ =
+:bro:type:`OpenFlow::ControllerState`: :bro:type:`record` :bro:attr:`&redef` 
+:bro:type:`OpenFlow::Plugin`: :bro:type:`enum`                               
+============================================================================ =
+
+Functions
+#########
+================================================= ===========================
+:bro:id:`OpenFlow::ryu_new`: :bro:type:`function` Ryu controller constructor.
+================================================= ===========================
+
+
+Detailed Interface
+~~~~~~~~~~~~~~~~~~
+Functions
+#########
+.. bro:id:: OpenFlow::ryu_new
+
+   :Type: :bro:type:`function` (host: :bro:type:`addr`, host_port: :bro:type:`count`, dpid: :bro:type:`count`) : :bro:type:`OpenFlow::Controller`
+
+   Ryu controller constructor.
+   
+
+   :host: Controller ip.
+   
+
+   :host_port: Controller listen port.
+   
+
+   :dpid: OpenFlow switch datapath id.
+   
+
+   :returns: OpenFlow::Controller record.
+
+

doc/scripts/base/frameworks/openflow/types.bro.rst

@@ -0,0 +1,264 @@
+:tocdepth: 3
+
+base/frameworks/openflow/types.bro
+==================================
+.. bro:namespace:: OpenFlow
+
+Types used by the OpenFlow framework.
+
+:Namespace: OpenFlow
+:Imports: :doc:`base/frameworks/openflow/consts.bro </scripts/base/frameworks/openflow/consts.bro>`
+
+Summary
+~~~~~~~
+Types
+#####
+============================================================================ ===============================================================
+:bro:type:`OpenFlow::Controller`: :bro:type:`record`                         Controller record representing an openflow controller.
+:bro:type:`OpenFlow::ControllerState`: :bro:type:`record` :bro:attr:`&redef` Controller related state.
+:bro:type:`OpenFlow::Plugin`: :bro:type:`enum`                               Available openflow plugins.
+:bro:type:`OpenFlow::ofp_flow_action`: :bro:type:`record` :bro:attr:`&log`   The actions that can be taken in a flow.
+:bro:type:`OpenFlow::ofp_flow_mod`: :bro:type:`record` :bro:attr:`&log`      Openflow flow_mod definition, describing the action to perform.
+:bro:type:`OpenFlow::ofp_match`: :bro:type:`record` :bro:attr:`&log`         Openflow match definition.
+============================================================================ ===============================================================
+
+
+Detailed Interface
+~~~~~~~~~~~~~~~~~~
+Types
+#####
+.. bro:type:: OpenFlow::Controller
+
+   :Type: :bro:type:`record`
+
+      state: :bro:type:`OpenFlow::ControllerState`
+         Controller related state.
+
+      supports_flow_removed: :bro:type:`bool`
+         Does the controller support the flow_removed event?
+
+      describe: :bro:type:`function` (state: :bro:type:`OpenFlow::ControllerState`) : :bro:type:`string`
+         Function that describes the controller. Has to be implemented.
+
+      init: :bro:type:`function` (state: :bro:type:`OpenFlow::ControllerState`) : :bro:type:`void` :bro:attr:`&optional`
+         One-time initialization function. If defined, controller_init_done has to be called once initialization finishes.
+
+      destroy: :bro:type:`function` (state: :bro:type:`OpenFlow::ControllerState`) : :bro:type:`void` :bro:attr:`&optional`
+         One-time destruction function.
+
+      flow_mod: :bro:type:`function` (state: :bro:type:`OpenFlow::ControllerState`, match: :bro:type:`OpenFlow::ofp_match`, flow_mod: :bro:type:`OpenFlow::ofp_flow_mod`) : :bro:type:`bool` :bro:attr:`&optional`
+         flow_mod function.
+
+      flow_clear: :bro:type:`function` (state: :bro:type:`OpenFlow::ControllerState`) : :bro:type:`bool` :bro:attr:`&optional`
+         flow_clear function.
+
+   Controller record representing an openflow controller.
+
+.. bro:type:: OpenFlow::ControllerState
+
+   :Type: :bro:type:`record`
+
+      _plugin: :bro:type:`OpenFlow::Plugin` :bro:attr:`&optional`
+         Internally set to the type of plugin used.
+
+      _name: :bro:type:`string` :bro:attr:`&optional`
+         Internally set to the unique name of the controller.
+
+      _activated: :bro:type:`bool` :bro:attr:`&default` = ``F`` :bro:attr:`&optional`
+         Internally set to true once the controller is activated.
+
+      ryu_host: :bro:type:`addr` :bro:attr:`&optional`
+         (present if :doc:`/scripts/base/frameworks/openflow/plugins/ryu.bro` is loaded)
+
+         Controller ip.
+
+      ryu_port: :bro:type:`count` :bro:attr:`&optional`
+         (present if :doc:`/scripts/base/frameworks/openflow/plugins/ryu.bro` is loaded)
+
+         Controller listen port.
+
+      ryu_dpid: :bro:type:`count` :bro:attr:`&optional`
+         (present if :doc:`/scripts/base/frameworks/openflow/plugins/ryu.bro` is loaded)
+
+         OpenFlow switch datapath id.
+
+      ryu_debug: :bro:type:`bool` :bro:attr:`&default` = ``F`` :bro:attr:`&optional`
+         (present if :doc:`/scripts/base/frameworks/openflow/plugins/ryu.bro` is loaded)
+
+         Enable debug mode - output JSON to stdout; do not perform actions.
+
+      log_dpid: :bro:type:`count` :bro:attr:`&optional`
+         (present if :doc:`/scripts/base/frameworks/openflow/plugins/log.bro` is loaded)
+
+         OpenFlow switch datapath id.
+
+      log_success_event: :bro:type:`bool` :bro:attr:`&optional`
+         (present if :doc:`/scripts/base/frameworks/openflow/plugins/log.bro` is loaded)
+
+         Raise or do not raise success event.
+
+      broker_host: :bro:type:`addr` :bro:attr:`&optional`
+         (present if :doc:`/scripts/base/frameworks/openflow/plugins/broker.bro` is loaded)
+
+         Controller ip.
+
+      broker_port: :bro:type:`port` :bro:attr:`&optional`
+         (present if :doc:`/scripts/base/frameworks/openflow/plugins/broker.bro` is loaded)
+
+         Controller listen port.
+
+      broker_dpid: :bro:type:`count` :bro:attr:`&optional`
+         (present if :doc:`/scripts/base/frameworks/openflow/plugins/broker.bro` is loaded)
+
+         OpenFlow switch datapath id.
+
+      broker_topic: :bro:type:`string` :bro:attr:`&optional`
+         (present if :doc:`/scripts/base/frameworks/openflow/plugins/broker.bro` is loaded)
+
+         Topic to send events for this controller to.
+   :Attributes: :bro:attr:`&redef`
+
+   Controller related state.
+   Can be redefined by plugins to
+   add state.
+
+.. bro:type:: OpenFlow::Plugin
+
+   :Type: :bro:type:`enum`
+
+      .. bro:enum:: OpenFlow::INVALID OpenFlow::Plugin
+
+         Internal placeholder plugin.
+
+      .. bro:enum:: OpenFlow::RYU OpenFlow::Plugin
+
+         (present if :doc:`/scripts/base/frameworks/openflow/plugins/ryu.bro` is loaded)
+
+
+      .. bro:enum:: OpenFlow::OFLOG OpenFlow::Plugin
+
+         (present if :doc:`/scripts/base/frameworks/openflow/plugins/log.bro` is loaded)
+
+
+      .. bro:enum:: OpenFlow::BROKER OpenFlow::Plugin
+
+         (present if :doc:`/scripts/base/frameworks/openflow/plugins/broker.bro` is loaded)
+
+
+   Available openflow plugins.
+
+.. bro:type:: OpenFlow::ofp_flow_action
+
+   :Type: :bro:type:`record`
+
+      out_ports: :bro:type:`vector` of :bro:type:`count` :bro:attr:`&default` = ``[]`` :bro:attr:`&optional` :bro:attr:`&log`
+         Output ports to send data to.
+
+      vlan_vid: :bro:type:`count` :bro:attr:`&optional` :bro:attr:`&log`
+         Set vlan vid to this value.
+
+      vlan_pcp: :bro:type:`count` :bro:attr:`&optional` :bro:attr:`&log`
+         Set vlan priority to this value.
+
+      vlan_strip: :bro:type:`bool` :bro:attr:`&default` = ``F`` :bro:attr:`&optional` :bro:attr:`&log`
+         Strip vlan tag.
+
+      dl_src: :bro:type:`string` :bro:attr:`&optional` :bro:attr:`&log`
+         Set ethernet source address.
+
+      dl_dst: :bro:type:`string` :bro:attr:`&optional` :bro:attr:`&log`
+         Set ethernet destination address.
+
+      nw_tos: :bro:type:`count` :bro:attr:`&optional` :bro:attr:`&log`
+         Set ip tos to this value.
+
+      nw_src: :bro:type:`addr` :bro:attr:`&optional` :bro:attr:`&log`
+         Set source to this ip.
+
+      nw_dst: :bro:type:`addr` :bro:attr:`&optional` :bro:attr:`&log`
+         Set destination to this ip.
+
+      tp_src: :bro:type:`count` :bro:attr:`&optional` :bro:attr:`&log`
+         Set tcp/udp source port.
+
+      tp_dst: :bro:type:`count` :bro:attr:`&optional` :bro:attr:`&log`
+         Set tcp/udp destination port.
+   :Attributes: :bro:attr:`&log`
+
+   The actions that can be taken in a flow.
+   (Separate record to make ofp_flow_mod less crowded)
+
+.. bro:type:: OpenFlow::ofp_flow_mod
+
+   :Type: :bro:type:`record`
+
+      cookie: :bro:type:`count` :bro:attr:`&log`
+         Opaque controller-issued identifier.
+
+      table_id: :bro:type:`count` :bro:attr:`&optional` :bro:attr:`&log`
+         Table to put the flow in. OFPTT_ALL can be used for delete,
+         to delete flows from all matching tables.
+
+      command: :bro:type:`OpenFlow::ofp_flow_mod_command` :bro:attr:`&log`
+         One of OFPFC_*.
+
+      idle_timeout: :bro:type:`count` :bro:attr:`&default` = ``0`` :bro:attr:`&optional` :bro:attr:`&log`
+         Idle time before discarding (seconds).
+
+      hard_timeout: :bro:type:`count` :bro:attr:`&default` = ``0`` :bro:attr:`&optional` :bro:attr:`&log`
+         Max time before discarding (seconds).
+
+      priority: :bro:type:`count` :bro:attr:`&default` = ``0`` :bro:attr:`&optional` :bro:attr:`&log`
+         Priority level of flow entry.
+
+      out_port: :bro:type:`count` :bro:attr:`&optional` :bro:attr:`&log`
+         For OFPFC_DELETE* commands, require matching entried to include
+         this as an output port/group. OFPP_ANY/OFPG_ANY means no restrictions.
+
+      out_group: :bro:type:`count` :bro:attr:`&optional` :bro:attr:`&log`
+
+      flags: :bro:type:`count` :bro:attr:`&default` = ``0`` :bro:attr:`&optional` :bro:attr:`&log`
+         Bitmap of the OFPFF_* flags
+
+      actions: :bro:type:`OpenFlow::ofp_flow_action` :bro:attr:`&default` = ``[out_ports=[], vlan_vid=<uninitialized>, vlan_pcp=<uninitialized>, vlan_strip=F, dl_src=<uninitialized>, dl_dst=<uninitialized>, nw_tos=<uninitialized>, nw_src=<uninitialized>, nw_dst=<uninitialized>, tp_src=<uninitialized>, tp_dst=<uninitialized>]`` :bro:attr:`&optional` :bro:attr:`&log`
+         Actions to take on match
+   :Attributes: :bro:attr:`&log`
+
+   Openflow flow_mod definition, describing the action to perform.
+
+.. bro:type:: OpenFlow::ofp_match
+
+   :Type: :bro:type:`record`
+
+      in_port: :bro:type:`count` :bro:attr:`&optional` :bro:attr:`&log`
+
+      dl_src: :bro:type:`string` :bro:attr:`&optional` :bro:attr:`&log`
+
+      dl_dst: :bro:type:`string` :bro:attr:`&optional` :bro:attr:`&log`
+
+      dl_vlan: :bro:type:`count` :bro:attr:`&optional` :bro:attr:`&log`
+
+      dl_vlan_pcp: :bro:type:`count` :bro:attr:`&optional` :bro:attr:`&log`
+
+      dl_type: :bro:type:`count` :bro:attr:`&optional` :bro:attr:`&log`
+
+      nw_tos: :bro:type:`count` :bro:attr:`&optional` :bro:attr:`&log`
+
+      nw_proto: :bro:type:`count` :bro:attr:`&optional` :bro:attr:`&log`
+
+      nw_src: :bro:type:`subnet` :bro:attr:`&optional` :bro:attr:`&log`
+
+      nw_dst: :bro:type:`subnet` :bro:attr:`&optional` :bro:attr:`&log`
+
+      tp_src: :bro:type:`count` :bro:attr:`&optional` :bro:attr:`&log`
+
+      tp_dst: :bro:type:`count` :bro:attr:`&optional` :bro:attr:`&log`
+   :Attributes: :bro:attr:`&log`
+
+   Openflow match definition.
+   
+   The openflow match record describes
+   which packets match to a specific
+   rule in a flow table.
+
+

doc/scripts/base/frameworks/packet-filter/__load__.bro.rst

@@ -0,0 +1,14 @@
+:tocdepth: 3
+
+base/frameworks/packet-filter/__load__.bro
+==========================================
+
+
+:Imports: :doc:`base/frameworks/cluster </scripts/base/frameworks/cluster/index>`, :doc:`base/frameworks/packet-filter/main.bro </scripts/base/frameworks/packet-filter/main.bro>`, :doc:`base/frameworks/packet-filter/netstats.bro </scripts/base/frameworks/packet-filter/netstats.bro>`, :doc:`base/frameworks/packet-filter/utils.bro </scripts/base/frameworks/packet-filter/utils.bro>`
+
+Summary
+~~~~~~~
+
+Detailed Interface
+~~~~~~~~~~~~~~~~~~
+

doc/scripts/base/frameworks/packet-filter/index.rst

@@ -0,0 +1,27 @@
+:orphan:
+
+Package: base/frameworks/packet-filter
+======================================
+
+The packet filter framework supports how Bro sets its BPF capture filter.
+
+:doc:`/scripts/base/frameworks/packet-filter/utils.bro`
+
+
+:doc:`/scripts/base/frameworks/packet-filter/__load__.bro`
+
+
+:doc:`/scripts/base/frameworks/packet-filter/main.bro`
+
+   This script supports how Bro sets its BPF capture filter.  By default
+   Bro sets a capture filter that allows all traffic.  If a filter
+   is set on the command line, that filter takes precedence over the default
+   open filter and all filters defined in Bro scripts with the
+   :bro:id:`capture_filters` and :bro:id:`restrict_filters` variables.
+
+:doc:`/scripts/base/frameworks/packet-filter/netstats.bro`
+
+   This script reports on packet loss from the various packet sources.
+   When Bro is reading input from trace files, this script will not
+   report any packet loss statistics.
+

doc/scripts/base/frameworks/packet-filter/main.bro.rst

@@ -0,0 +1,223 @@
+:tocdepth: 3
+
+base/frameworks/packet-filter/main.bro
+======================================
+.. bro:namespace:: PacketFilter
+
+This script supports how Bro sets its BPF capture filter.  By default
+Bro sets a capture filter that allows all traffic.  If a filter
+is set on the command line, that filter takes precedence over the default
+open filter and all filters defined in Bro scripts with the
+:bro:id:`capture_filters` and :bro:id:`restrict_filters` variables.
+
+:Namespace: PacketFilter
+:Imports: :doc:`base/frameworks/analyzer </scripts/base/frameworks/analyzer/index>`, :doc:`base/frameworks/notice </scripts/base/frameworks/notice/index>`, :doc:`base/frameworks/packet-filter/utils.bro </scripts/base/frameworks/packet-filter/utils.bro>`
+
+Summary
+~~~~~~~
+Redefinable Options
+###################
+================================================================================================= ===============================================================================
+:bro:id:`PacketFilter::default_capture_filter`: :bro:type:`string` :bro:attr:`&redef`             The BPF filter that is used by default to define what traffic should
+                                                                                                  be captured.
+:bro:id:`PacketFilter::enable_auto_protocol_capture_filters`: :bro:type:`bool` :bro:attr:`&redef` Enables the old filtering approach of "only watch common ports for
+                                                                                                  analyzed protocols".
+:bro:id:`PacketFilter::max_filter_compile_time`: :bro:type:`interval` :bro:attr:`&redef`          The maximum amount of time that you'd like to allow for BPF filters to compile.
+:bro:id:`PacketFilter::restricted_filter`: :bro:type:`string` :bro:attr:`&redef`                  Filter string which is unconditionally and'ed to the beginning of
+                                                                                                  every dynamically built filter.
+:bro:id:`PacketFilter::unrestricted_filter`: :bro:type:`string` :bro:attr:`&redef`                Filter string which is unconditionally or'ed to the beginning of
+                                                                                                  every dynamically built filter.
+================================================================================================= ===============================================================================
+
+State Variables
+###############
+========================================================== ===================================================================
+:bro:id:`PacketFilter::current_filter`: :bro:type:`string` This is where the default packet filter is stored and it should not
+                                                           normally be modified by users.
+========================================================== ===================================================================
+
+Types
+#####
+========================================================== ==================================================================
+:bro:type:`PacketFilter::FilterPlugin`: :bro:type:`record` A data structure to represent filter generating plugins.
+:bro:type:`PacketFilter::Info`: :bro:type:`record`         The record type defining columns to be logged in the packet filter
+                                                           logging stream.
+========================================================== ==================================================================
+
+Redefinitions
+#############
+========================================== =================================================
+:bro:type:`Log::ID`: :bro:type:`enum`      Add the packet filter logging stream.
+:bro:type:`Notice::Type`: :bro:type:`enum` Add notice types related to packet filter errors.
+:bro:type:`PcapFilterID`: :bro:type:`enum` 
+========================================== =================================================
+
+Functions
+#########
+==================================================================== ======================================================================
+:bro:id:`PacketFilter::exclude`: :bro:type:`function`                Install a BPF filter to exclude some traffic.
+:bro:id:`PacketFilter::exclude_for`: :bro:type:`function`            Install a temporary filter to traffic which should not be passed
+                                                                     through the BPF filter.
+:bro:id:`PacketFilter::install`: :bro:type:`function`                Call this function to build and install a new dynamically built
+                                                                     packet filter.
+:bro:id:`PacketFilter::register_filter_plugin`: :bro:type:`function` API function to register a new plugin for dynamic restriction filters.
+==================================================================== ======================================================================
+
+
+Detailed Interface
+~~~~~~~~~~~~~~~~~~
+Redefinable Options
+###################
+.. bro:id:: PacketFilter::default_capture_filter
+
+   :Type: :bro:type:`string`
+   :Attributes: :bro:attr:`&redef`
+   :Default: ``"ip or not ip"``
+
+   The BPF filter that is used by default to define what traffic should
+   be captured.  Filters defined in :bro:id:`restrict_filters` will
+   still be applied to reduce the captured traffic.
+
+.. bro:id:: PacketFilter::enable_auto_protocol_capture_filters
+
+   :Type: :bro:type:`bool`
+   :Attributes: :bro:attr:`&redef`
+   :Default: ``F``
+
+   Enables the old filtering approach of "only watch common ports for
+   analyzed protocols".
+   
+   Unless you know what you are doing, leave this set to F.
+
+.. bro:id:: PacketFilter::max_filter_compile_time
+
+   :Type: :bro:type:`interval`
+   :Attributes: :bro:attr:`&redef`
+   :Default: ``100.0 msecs``
+
+   The maximum amount of time that you'd like to allow for BPF filters to compile.
+   If this time is exceeded, compensation measures may be taken by the framework
+   to reduce the filter size.  This threshold being crossed also results
+   in the :bro:see:`PacketFilter::Too_Long_To_Compile_Filter` notice.
+
+.. bro:id:: PacketFilter::restricted_filter
+
+   :Type: :bro:type:`string`
+   :Attributes: :bro:attr:`&redef`
+   :Default: ``""``
+
+   Filter string which is unconditionally and'ed to the beginning of
+   every dynamically built filter.  This is mostly used when a custom
+   filter is being used but MPLS or VLAN tags are on the traffic.
+
+.. bro:id:: PacketFilter::unrestricted_filter
+
+   :Type: :bro:type:`string`
+   :Attributes: :bro:attr:`&redef`
+   :Default: ``""``
+
+   Filter string which is unconditionally or'ed to the beginning of
+   every dynamically built filter.
+
+State Variables
+###############
+.. bro:id:: PacketFilter::current_filter
+
+   :Type: :bro:type:`string`
+   :Default: ``"<not set yet>"``
+
+   This is where the default packet filter is stored and it should not
+   normally be modified by users.
+
+Types
+#####
+.. bro:type:: PacketFilter::FilterPlugin
+
+   :Type: :bro:type:`record`
+
+      func: :bro:type:`function` () : :bro:type:`void`
+         A function that is directly called when generating the complete filter.
+
+   A data structure to represent filter generating plugins.
+
+.. bro:type:: PacketFilter::Info
+
+   :Type: :bro:type:`record`
+
+      ts: :bro:type:`time` :bro:attr:`&log`
+         The time at which the packet filter installation attempt was made.
+
+      node: :bro:type:`string` :bro:attr:`&log` :bro:attr:`&optional`
+         This is a string representation of the node that applied this
+         packet filter.  It's mostly useful in the context of
+         dynamically changing filters on clusters.
+
+      filter: :bro:type:`string` :bro:attr:`&log`
+         The packet filter that is being set.
+
+      init: :bro:type:`bool` :bro:attr:`&log` :bro:attr:`&default` = ``F`` :bro:attr:`&optional`
+         Indicate if this is the filter set during initialization.
+
+      success: :bro:type:`bool` :bro:attr:`&log` :bro:attr:`&default` = ``T`` :bro:attr:`&optional`
+         Indicate if the filter was applied successfully.
+
+   The record type defining columns to be logged in the packet filter
+   logging stream.
+
+Functions
+#########
+.. bro:id:: PacketFilter::exclude
+
+   :Type: :bro:type:`function` (filter_id: :bro:type:`string`, filter: :bro:type:`string`) : :bro:type:`bool`
+
+   Install a BPF filter to exclude some traffic.  The filter should
+   positively match what is to be excluded, it will be wrapped in
+   a "not".
+   
+
+   :filter_id: An arbitrary string that can be used to identify
+              the filter.
+   
+
+   :filter: A BPF expression of traffic that should be excluded.
+   
+
+   :returns: A boolean value to indicate if the filter was successfully
+            installed or not.
+
+.. bro:id:: PacketFilter::exclude_for
+
+   :Type: :bro:type:`function` (filter_id: :bro:type:`string`, filter: :bro:type:`string`, span: :bro:type:`interval`) : :bro:type:`bool`
+
+   Install a temporary filter to traffic which should not be passed
+   through the BPF filter.  The filter should match the traffic you
+   don't want to see (it will be wrapped in a "not" condition).
+   
+
+   :filter_id: An arbitrary string that can be used to identify
+              the filter.
+   
+
+   :filter: A BPF expression of traffic that should be excluded.
+   
+
+   :length: The duration for which this filter should be put in place.
+   
+
+   :returns: A boolean value to indicate if the filter was successfully
+            installed or not.
+
+.. bro:id:: PacketFilter::install
+
+   :Type: :bro:type:`function` () : :bro:type:`bool`
+
+   Call this function to build and install a new dynamically built
+   packet filter.
+
+.. bro:id:: PacketFilter::register_filter_plugin
+
+   :Type: :bro:type:`function` (fp: :bro:type:`PacketFilter::FilterPlugin`) : :bro:type:`void`
+
+   API function to register a new plugin for dynamic restriction filters.
+
+

doc/scripts/base/frameworks/packet-filter/netstats.bro.rst

@@ -0,0 +1,40 @@
+:tocdepth: 3
+
+base/frameworks/packet-filter/netstats.bro
+==========================================
+.. bro:namespace:: PacketFilter
+
+This script reports on packet loss from the various packet sources.
+When Bro is reading input from trace files, this script will not
+report any packet loss statistics.
+
+:Namespace: PacketFilter
+:Imports: :doc:`base/frameworks/notice </scripts/base/frameworks/notice/index>`
+
+Summary
+~~~~~~~
+Constants
+#########
+======================================================================= ==============================================================
+:bro:id:`PacketFilter::stats_collection_interval`: :bro:type:`interval` This is the interval between individual statistics collection.
+======================================================================= ==============================================================
+
+Redefinitions
+#############
+========================================== =
+:bro:type:`Notice::Type`: :bro:type:`enum` 
+========================================== =
+
+
+Detailed Interface
+~~~~~~~~~~~~~~~~~~
+Constants
+#########
+.. bro:id:: PacketFilter::stats_collection_interval
+
+   :Type: :bro:type:`interval`
+   :Default: ``5.0 mins``
+
+   This is the interval between individual statistics collection.
+
+

doc/scripts/base/frameworks/packet-filter/utils.bro.rst

@@ -0,0 +1,73 @@
+:tocdepth: 3
+
+base/frameworks/packet-filter/utils.bro
+=======================================
+.. bro:namespace:: PacketFilter
+
+
+:Namespace: PacketFilter
+
+Summary
+~~~~~~~
+Functions
+#########
+============================================================= ==================================================================
+:bro:id:`PacketFilter::combine_filters`: :bro:type:`function` Combines two valid BPF filter strings with a string based operator
+                                                              to form a new filter.
+:bro:id:`PacketFilter::port_to_bpf`: :bro:type:`function`     Takes a :bro:type:`port` and returns a BPF expression which will
+                                                              match the port.
+:bro:id:`PacketFilter::sampling_filter`: :bro:type:`function` Create a BPF filter to sample IPv4 and IPv6 traffic.
+============================================================= ==================================================================
+
+
+Detailed Interface
+~~~~~~~~~~~~~~~~~~
+Functions
+#########
+.. bro:id:: PacketFilter::combine_filters
+
+   :Type: :bro:type:`function` (lfilter: :bro:type:`string`, op: :bro:type:`string`, rfilter: :bro:type:`string`) : :bro:type:`string`
+
+   Combines two valid BPF filter strings with a string based operator
+   to form a new filter.
+   
+
+   :lfilter: Filter which will go on the left side.
+   
+
+   :op: Operation being applied (typically "or" or "and").
+   
+
+   :rfilter: Filter which will go on the right side.
+   
+
+   :returns: A new string representing the two filters combined with
+            the operator.  Either filter being an empty string will
+            still result in a valid filter.
+
+.. bro:id:: PacketFilter::port_to_bpf
+
+   :Type: :bro:type:`function` (p: :bro:type:`port`) : :bro:type:`string`
+
+   Takes a :bro:type:`port` and returns a BPF expression which will
+   match the port.
+   
+
+   :p: The port.
+   
+
+   :returns: A valid BPF filter string for matching the port.
+
+.. bro:id:: PacketFilter::sampling_filter
+
+   :Type: :bro:type:`function` (num_parts: :bro:type:`count`, this_part: :bro:type:`count`) : :bro:type:`string`
+
+   Create a BPF filter to sample IPv4 and IPv6 traffic.
+   
+
+   :num_parts: The number of parts the traffic should be split into.
+   
+
+   :this_part: The part of the traffic this filter will accept (0-based).
+
+

doc/scripts/base/frameworks/reporter/__load__.bro.rst

@@ -0,0 +1,14 @@
+:tocdepth: 3
+
+base/frameworks/reporter/__load__.bro
+=====================================
+
+
+:Imports: :doc:`base/frameworks/reporter/main.bro </scripts/base/frameworks/reporter/main.bro>`
+
+Summary
+~~~~~~~
+
+Detailed Interface
+~~~~~~~~~~~~~~~~~~
+

doc/scripts/base/frameworks/reporter/index.rst

@@ -0,0 +1,26 @@
+:orphan:
+
+Package: base/frameworks/reporter
+=================================
+
+This framework is intended to create an output and filtering path for
+internally generated messages/warnings/errors.
+
+:doc:`/scripts/base/frameworks/reporter/__load__.bro`
+
+
+:doc:`/scripts/base/frameworks/reporter/main.bro`
+
+   This framework is intended to create an output and filtering path for
+   internal messages/warnings/errors.  It should typically be loaded to
+   log such messages to a file in a standard way.  For the options to
+   toggle whether messages are additionally written to STDERR, see
+   :bro:see:`Reporter::info_to_stderr`,
+   :bro:see:`Reporter::warnings_to_stderr`, and
+   :bro:see:`Reporter::errors_to_stderr`.
+   
+   Note that this framework deals with the handling of internally generated
+   reporter messages, for the interface
+   into actually creating reporter messages from the scripting layer, use
+   the built-in functions in :doc:`/scripts/base/bif/reporter.bif.bro`.
+

doc/scripts/base/frameworks/reporter/main.bro.rst

@@ -0,0 +1,64 @@
+:tocdepth: 3
+
+base/frameworks/reporter/main.bro
+=================================
+.. bro:namespace:: Reporter
+
+This framework is intended to create an output and filtering path for
+internal messages/warnings/errors.  It should typically be loaded to
+log such messages to a file in a standard way.  For the options to
+toggle whether messages are additionally written to STDERR, see
+:bro:see:`Reporter::info_to_stderr`,
+:bro:see:`Reporter::warnings_to_stderr`, and
+:bro:see:`Reporter::errors_to_stderr`.
+
+Note that this framework deals with the handling of internally generated
+reporter messages, for the interface
+into actually creating reporter messages from the scripting layer, use
+the built-in functions in :doc:`/scripts/base/bif/reporter.bif.bro`.
+
+:Namespace: Reporter
+
+Summary
+~~~~~~~
+Types
+#####
+============================================== =====================================================================
+:bro:type:`Reporter::Info`: :bro:type:`record` The record type which contains the column fields of the reporter log.
+============================================== =====================================================================
+
+Redefinitions
+#############
+===================================== =======================================
+:bro:type:`Log::ID`: :bro:type:`enum` The reporter logging stream identifier.
+===================================== =======================================
+
+
+Detailed Interface
+~~~~~~~~~~~~~~~~~~
+Types
+#####
+.. bro:type:: Reporter::Info
+
+   :Type: :bro:type:`record`
+
+      ts: :bro:type:`time` :bro:attr:`&log`
+         The network time at which the reporter event was generated.
+
+      level: :bro:type:`Reporter::Level` :bro:attr:`&log`
+         The severity of the reporter message. Levels are INFO for informational
+         messages, not needing specific attention; WARNING for warning of a potential
+         problem, and ERROR for a non-fatal error that should be addressed, but doesn't
+         terminate program execution.
+
+      message: :bro:type:`string` :bro:attr:`&log`
+         An info/warning/error message that could have either been
+         generated from the internal Bro core or at the scripting-layer.
+
+      location: :bro:type:`string` :bro:attr:`&log` :bro:attr:`&optional`
+         This is the location in a Bro script where the message originated.
+         Not all reporter messages will have locations in them though.
+
+   The record type which contains the column fields of the reporter log.
+
+

doc/scripts/base/frameworks/signatures/__load__.bro.rst

@@ -0,0 +1,14 @@
+:tocdepth: 3
+
+base/frameworks/signatures/__load__.bro
+=======================================
+
+
+:Imports: :doc:`base/frameworks/signatures/main.bro </scripts/base/frameworks/signatures/main.bro>`
+
+Summary
+~~~~~~~
+
+Detailed Interface
+~~~~~~~~~~~~~~~~~~
+

doc/scripts/base/frameworks/signatures/index.rst

@@ -0,0 +1,19 @@
+:orphan:
+
+Package: base/frameworks/signatures
+===================================
+
+The signature framework provides for doing low-level pattern matching.  While
+signatures are not Bro's preferred detection tool, they sometimes come in
+handy and are closer to what many people are familiar with from using
+other NIDS.
+
+:doc:`/scripts/base/frameworks/signatures/__load__.bro`
+
+
+:doc:`/scripts/base/frameworks/signatures/main.bro`
+
+   Script level signature support.  See the
+   :doc:`signature documentation </frameworks/signatures>` for more
+   information about Bro's signature engine.
+

doc/scripts/base/frameworks/signatures/main.bro.rst

@@ -0,0 +1,272 @@
+:tocdepth: 3
+
+base/frameworks/signatures/main.bro
+===================================
+.. bro:namespace:: Signatures
+
+Script level signature support.  See the
+:doc:`signature documentation </frameworks/signatures>` for more
+information about Bro's signature engine.
+
+:Namespace: Signatures
+:Imports: :doc:`base/frameworks/notice </scripts/base/frameworks/notice/index>`
+
+Summary
+~~~~~~~
+Runtime Options
+###############
+=============================================================================== ===================================================================
+:bro:id:`Signatures::ignored_ids`: :bro:type:`pattern` :bro:attr:`&redef`       Signature IDs that should always be ignored.
+:bro:id:`Signatures::summary_interval`: :bro:type:`interval` :bro:attr:`&redef` The interval between when :bro:enum:`Signatures::Signature_Summary`
+                                                                                notices are generated.
+=============================================================================== ===================================================================
+
+Redefinable Options
+###################
+========================================================================================================================================== ====================================================================
+:bro:id:`Signatures::actions`: :bro:type:`table` :bro:attr:`&redef` :bro:attr:`&default` = ``Signatures::SIG_ALARM`` :bro:attr:`&optional` Actions for a signature.
+:bro:id:`Signatures::count_thresholds`: :bro:type:`set` :bro:attr:`&redef`                                                                 Generate a notice if a :bro:enum:`Signatures::SIG_COUNT_PER_RESP`
+                                                                                                                                           signature is triggered as often as given by one of these thresholds.
+:bro:id:`Signatures::horiz_scan_thresholds`: :bro:type:`set` :bro:attr:`&redef`                                                            Generate a notice if, for a pair [orig, signature], the number of
+                                                                                                                                           different responders has reached one of the thresholds.
+:bro:id:`Signatures::vert_scan_thresholds`: :bro:type:`set` :bro:attr:`&redef`                                                             Generate a notice if, for a pair [orig, resp], the number of
+                                                                                                                                           different signature matches has reached one of the thresholds.
+========================================================================================================================================== ====================================================================
+
+Types
+#####
+================================================ ======================================================================
+:bro:type:`Signatures::Action`: :bro:type:`enum` These are the default actions you can apply to signature matches.
+:bro:type:`Signatures::Info`: :bro:type:`record` The record type which contains the column fields of the signature log.
+================================================ ======================================================================
+
+Redefinitions
+#############
+========================================== ===========================================
+:bro:type:`Log::ID`: :bro:type:`enum`      The signature logging stream identifier.
+:bro:type:`Notice::Type`: :bro:type:`enum` Add various signature-related notice types.
+========================================== ===========================================
+
+Events
+######
+====================================================== =================================================================
+:bro:id:`Signatures::log_signature`: :bro:type:`event` This event can be handled to access/alter data about to be logged
+                                                       to the signature logging stream.
+====================================================== =================================================================
+
+
+Detailed Interface
+~~~~~~~~~~~~~~~~~~
+Runtime Options
+###############
+.. bro:id:: Signatures::ignored_ids
+
+   :Type: :bro:type:`pattern`
+   :Attributes: :bro:attr:`&redef`
+   :Default:
+
+   ::
+
+      /(^?(^?(^webapp-)$?)$?)|(^?((^?(^?(traceroute-detector.*)$?)$?)|(^?(^?(NO_DEFAULT_MATCHES)$?)$?))$?)/
+
+   Signature IDs that should always be ignored.
+
+.. bro:id:: Signatures::summary_interval
+
+   :Type: :bro:type:`interval`
+   :Attributes: :bro:attr:`&redef`
+   :Default: ``1.0 day``
+
+   The interval between when :bro:enum:`Signatures::Signature_Summary`
+   notices are generated.
+
+Redefinable Options
+###################
+.. bro:id:: Signatures::actions
+
+   :Type: :bro:type:`table` [:bro:type:`string`] of :bro:type:`Signatures::Action`
+   :Attributes: :bro:attr:`&redef` :bro:attr:`&default` = ``Signatures::SIG_ALARM`` :bro:attr:`&optional`
+   :Default:
+
+   ::
+
+      {
+         ["unspecified"] = Signatures::SIG_IGNORE
+      }
+
+   Actions for a signature.  
+
+.. bro:id:: Signatures::count_thresholds
+
+   :Type: :bro:type:`set` [:bro:type:`count`]
+   :Attributes: :bro:attr:`&redef`
+   :Default:
+
+   ::
+
+      {
+         500,
+         1000,
+         1000000,
+         5,
+         100,
+         50,
+         10000,
+         10
+      }
+
+   Generate a notice if a :bro:enum:`Signatures::SIG_COUNT_PER_RESP`
+   signature is triggered as often as given by one of these thresholds.
+
+.. bro:id:: Signatures::horiz_scan_thresholds
+
+   :Type: :bro:type:`set` [:bro:type:`count`]
+   :Attributes: :bro:attr:`&redef`
+   :Default:
+
+   ::
+
+      {
+         500,
+         1000,
+         5,
+         100,
+         50,
+         10
+      }
+
+   Generate a notice if, for a pair [orig, signature], the number of
+   different responders has reached one of the thresholds.
+
+.. bro:id:: Signatures::vert_scan_thresholds
+
+   :Type: :bro:type:`set` [:bro:type:`count`]
+   :Attributes: :bro:attr:`&redef`
+   :Default:
+
+   ::
+
+      {
+         500,
+         1000,
+         5,
+         100,
+         50,
+         10
+      }
+
+   Generate a notice if, for a pair [orig, resp], the number of
+   different signature matches has reached one of the thresholds.
+
+Types
+#####
+.. bro:type:: Signatures::Action
+
+   :Type: :bro:type:`enum`
+
+      .. bro:enum:: Signatures::SIG_IGNORE Signatures::Action
+
+         Ignore this signature completely (even for scan detection).
+         Don't write to the signatures logging stream.
+
+      .. bro:enum:: Signatures::SIG_QUIET Signatures::Action
+
+         Process through the various aggregate techniques, but don't
+         report individually and don't write to the signatures logging
+         stream.
+
+      .. bro:enum:: Signatures::SIG_LOG Signatures::Action
+
+         Generate a notice.
+
+      .. bro:enum:: Signatures::SIG_FILE_BUT_NO_SCAN Signatures::Action
+
+         The same as :bro:enum:`Signatures::SIG_LOG`, but ignore for
+         aggregate/scan processing.
+
+      .. bro:enum:: Signatures::SIG_ALARM Signatures::Action
+
+         Generate a notice and set it to be alarmed upon.
+
+      .. bro:enum:: Signatures::SIG_ALARM_PER_ORIG Signatures::Action
+
+         Alarm once per originator.
+
+      .. bro:enum:: Signatures::SIG_ALARM_ONCE Signatures::Action
+
+         Alarm once and then never again.
+
+      .. bro:enum:: Signatures::SIG_COUNT_PER_RESP Signatures::Action
+
+         Count signatures per responder host and alarm with the 
+         :bro:enum:`Signatures::Count_Signature` notice if a threshold
+         defined by :bro:id:`Signatures::count_thresholds` is reached.
+
+      .. bro:enum:: Signatures::SIG_SUMMARY Signatures::Action
+
+         Don't alarm, but generate per-orig summary.
+
+   These are the default actions you can apply to signature matches.
+   All of them write the signature record to the logging stream unless
+   declared otherwise.
+
+.. bro:type:: Signatures::Info
+
+   :Type: :bro:type:`record`
+
+      ts: :bro:type:`time` :bro:attr:`&log`
+         The network time at which a signature matching type of event
+         to be logged has occurred.
+
+      uid: :bro:type:`string` :bro:attr:`&log` :bro:attr:`&optional`
+         A unique identifier of the connection which triggered the
+         signature match event.
+
+      src_addr: :bro:type:`addr` :bro:attr:`&log` :bro:attr:`&optional`
+         The host which triggered the signature match event.
+
+      src_port: :bro:type:`port` :bro:attr:`&log` :bro:attr:`&optional`
+         The host port on which the signature-matching activity
+         occurred.
+
+      dst_addr: :bro:type:`addr` :bro:attr:`&log` :bro:attr:`&optional`
+         The destination host which was sent the payload that
+         triggered the signature match.
+
+      dst_port: :bro:type:`port` :bro:attr:`&log` :bro:attr:`&optional`
+         The destination host port which was sent the payload that
+         triggered the signature match.
+
+      note: :bro:type:`Notice::Type` :bro:attr:`&log`
+         Notice associated with signature event.
+
+      sig_id: :bro:type:`string` :bro:attr:`&log` :bro:attr:`&optional`
+         The name of the signature that matched.
+
+      event_msg: :bro:type:`string` :bro:attr:`&log` :bro:attr:`&optional`
+         A more descriptive message of the signature-matching event.
+
+      sub_msg: :bro:type:`string` :bro:attr:`&log` :bro:attr:`&optional`
+         Extracted payload data or extra message.
+
+      sig_count: :bro:type:`count` :bro:attr:`&log` :bro:attr:`&optional`
+         Number of sigs, usually from summary count.
+
+      host_count: :bro:type:`count` :bro:attr:`&log` :bro:attr:`&optional`
+         Number of hosts, from a summary count.
+
+   The record type which contains the column fields of the signature log.
+
+Events
+######
+.. bro:id:: Signatures::log_signature
+
+   :Type: :bro:type:`event` (rec: :bro:type:`Signatures::Info`)
+
+   This event can be handled to access/alter data about to be logged
+   to the signature logging stream.
+   
+
+   :rec: The record of signature data about to be logged.
+
+

doc/scripts/base/frameworks/software/__load__.bro.rst

@@ -0,0 +1,14 @@
+:tocdepth: 3
+
+base/frameworks/software/__load__.bro
+=====================================
+
+
+:Imports: :doc:`base/frameworks/software/main.bro </scripts/base/frameworks/software/main.bro>`
+
+Summary
+~~~~~~~
+
+Detailed Interface
+~~~~~~~~~~~~~~~~~~
+