Remove broxygen Sphinx integration

The broxygen-generated files now live in the git repo, have tests
that check that they are up-to-date, and a script to re-generate
them on-demand.

doc/scripts/base/frameworks/analyzer/__load__.bro.rst

@@ -0,0 +1,14 @@
+:tocdepth: 3
+
+base/frameworks/analyzer/__load__.bro
+=====================================
+
+
+:Imports: :doc:`base/frameworks/analyzer/main.bro </scripts/base/frameworks/analyzer/main.bro>`
+
+Summary
+~~~~~~~
+
+Detailed Interface
+~~~~~~~~~~~~~~~~~~
+

doc/scripts/base/frameworks/analyzer/index.rst

@@ -0,0 +1,26 @@
+:orphan:
+
+Package: base/frameworks/analyzer
+=================================
+
+The analyzer framework allows to dynamically enable or disable Bro's
+protocol analyzers, as well as to manage the well-known ports which
+automatically activate a particular analyzer for new connections.
+
+:doc:`/scripts/base/frameworks/analyzer/__load__.bro`
+
+
+:doc:`/scripts/base/frameworks/analyzer/main.bro`
+
+   Framework for managing Bro's protocol analyzers.
+   
+   The analyzer framework allows to dynamically enable or disable analyzers, as
+   well as to manage the well-known ports which automatically activate a
+   particular analyzer for new connections.
+   
+   Protocol analyzers are identified by unique tags of type
+   :bro:type:`Analyzer::Tag`, such as :bro:enum:`Analyzer::ANALYZER_HTTP`.
+   These tags are defined internally by
+   the analyzers themselves, and documented in their analyzer-specific
+   description along with the events that they generate.
+

doc/scripts/base/frameworks/analyzer/main.bro.rst

@@ -0,0 +1,246 @@
+:tocdepth: 3
+
+base/frameworks/analyzer/main.bro
+=================================
+.. bro:namespace:: Analyzer
+
+Framework for managing Bro's protocol analyzers.
+
+The analyzer framework allows to dynamically enable or disable analyzers, as
+well as to manage the well-known ports which automatically activate a
+particular analyzer for new connections.
+
+Protocol analyzers are identified by unique tags of type
+:bro:type:`Analyzer::Tag`, such as :bro:enum:`Analyzer::ANALYZER_HTTP`.
+These tags are defined internally by
+the analyzers themselves, and documented in their analyzer-specific
+description along with the events that they generate.
+
+:Namespace: Analyzer
+:Imports: :doc:`base/bif/analyzer.bif.bro </scripts/base/bif/analyzer.bif.bro>`, :doc:`base/frameworks/packet-filter/utils.bro </scripts/base/frameworks/packet-filter/utils.bro>`
+
+Summary
+~~~~~~~
+State Variables
+###############
+========================================================================== ===================================================================
+:bro:id:`Analyzer::disable_all`: :bro:type:`bool` :bro:attr:`&redef`       If true, all available analyzers are initially disabled at startup.
+:bro:id:`Analyzer::disabled_analyzers`: :bro:type:`set` :bro:attr:`&redef` A set of analyzers to disable by default at startup.
+========================================================================== ===================================================================
+
+Functions
+#########
+============================================================== =======================================================================
+:bro:id:`Analyzer::all_registered_ports`: :bro:type:`function` Returns a table of all ports-to-analyzer mappings currently registered.
+:bro:id:`Analyzer::analyzer_to_bpf`: :bro:type:`function`      Automatically creates a BPF filter for the specified protocol based
+                                                               on the data supplied for the protocol through the
+                                                               :bro:see:`Analyzer::register_for_ports` function.
+:bro:id:`Analyzer::disable_analyzer`: :bro:type:`function`     Disables an analyzer.
+:bro:id:`Analyzer::enable_analyzer`: :bro:type:`function`      Enables an analyzer.
+:bro:id:`Analyzer::get_bpf`: :bro:type:`function`              Create a BPF filter which matches all of the ports defined
+                                                               by the various protocol analysis scripts as "registered ports"
+                                                               for the protocol.
+:bro:id:`Analyzer::get_tag`: :bro:type:`function`              Translates an analyzer's name to a tag enum value.
+:bro:id:`Analyzer::name`: :bro:type:`function`                 Translates an analyzer type to a string with the analyzer's name.
+:bro:id:`Analyzer::register_for_port`: :bro:type:`function`    Registers an individual well-known port for an analyzer.
+:bro:id:`Analyzer::register_for_ports`: :bro:type:`function`   Registers a set of well-known ports for an analyzer.
+:bro:id:`Analyzer::registered_ports`: :bro:type:`function`     Returns a set of all well-known ports currently registered for a
+                                                               specific analyzer.
+:bro:id:`Analyzer::schedule_analyzer`: :bro:type:`function`    Schedules an analyzer for a future connection originating from a
+                                                               given IP address and port.
+============================================================== =======================================================================
+
+
+Detailed Interface
+~~~~~~~~~~~~~~~~~~
+State Variables
+###############
+.. bro:id:: Analyzer::disable_all
+
+   :Type: :bro:type:`bool`
+   :Attributes: :bro:attr:`&redef`
+   :Default: ``F``
+
+   If true, all available analyzers are initially disabled at startup.
+   One can then selectively enable them with
+   :bro:id:`Analyzer::enable_analyzer`.
+
+.. bro:id:: Analyzer::disabled_analyzers
+
+   :Type: :bro:type:`set` [:bro:type:`Analyzer::Tag`]
+   :Attributes: :bro:attr:`&redef`
+   :Default:
+
+   ::
+
+      {
+         Analyzer::ANALYZER_BACKDOOR,
+         Analyzer::ANALYZER_INTERCONN,
+         Analyzer::ANALYZER_TCPSTATS,
+         Analyzer::ANALYZER_STEPPINGSTONE
+      }
+
+   A set of analyzers to disable by default at startup. The default set
+   contains legacy analyzers that are no longer supported.
+
+Functions
+#########
+.. bro:id:: Analyzer::all_registered_ports
+
+   :Type: :bro:type:`function` () : :bro:type:`table` [:bro:type:`Analyzer::Tag`] of :bro:type:`set` [:bro:type:`port`]
+
+   Returns a table of all ports-to-analyzer mappings currently registered.
+   
+
+   :returns: A table mapping each analyzer to the set of ports
+            registered for it.
+
+.. bro:id:: Analyzer::analyzer_to_bpf
+
+   :Type: :bro:type:`function` (tag: :bro:type:`Analyzer::Tag`) : :bro:type:`string`
+
+   Automatically creates a BPF filter for the specified protocol based
+   on the data supplied for the protocol through the
+   :bro:see:`Analyzer::register_for_ports` function.
+   
+
+   :tag: The analyzer tag.
+   
+
+   :returns: BPF filter string.
+
+.. bro:id:: Analyzer::disable_analyzer
+
+   :Type: :bro:type:`function` (tag: :bro:type:`Analyzer::Tag`) : :bro:type:`bool`
+
+   Disables an analyzer. Once disabled, the analyzer will not be used
+   further for analysis of future connections.
+   
+
+   :tag: The tag of the analyzer to disable.
+   
+
+   :returns: True if the analyzer was successfully disabled.
+
+.. bro:id:: Analyzer::enable_analyzer
+
+   :Type: :bro:type:`function` (tag: :bro:type:`Analyzer::Tag`) : :bro:type:`bool`
+
+   Enables an analyzer. Once enabled, the analyzer may be used for analysis
+   of future connections as decided by Bro's dynamic protocol detection.
+   
+
+   :tag: The tag of the analyzer to enable.
+   
+
+   :returns: True if the analyzer was successfully enabled.
+
+.. bro:id:: Analyzer::get_bpf
+
+   :Type: :bro:type:`function` () : :bro:type:`string`
+
+   Create a BPF filter which matches all of the ports defined
+   by the various protocol analysis scripts as "registered ports"
+   for the protocol.
+
+.. bro:id:: Analyzer::get_tag
+
+   :Type: :bro:type:`function` (name: :bro:type:`string`) : :bro:type:`Analyzer::Tag`
+
+   Translates an analyzer's name to a tag enum value.
+   
+
+   :name: The analyzer name.
+   
+
+   :returns: The analyzer tag corresponding to the name.
+
+.. bro:id:: Analyzer::name
+
+   :Type: :bro:type:`function` (atype: :bro:type:`Analyzer::Tag`) : :bro:type:`string`
+
+   Translates an analyzer type to a string with the analyzer's name.
+   
+
+   :tag: The analyzer tag.
+   
+
+   :returns: The analyzer name corresponding to the tag.
+
+.. bro:id:: Analyzer::register_for_port
+
+   :Type: :bro:type:`function` (tag: :bro:type:`Analyzer::Tag`, p: :bro:type:`port`) : :bro:type:`bool`
+
+   Registers an individual well-known port for an analyzer. If a future
+   connection on this port is seen, the analyzer will be automatically
+   assigned to parsing it. The function *adds* to all ports already
+   registered, it doesn't replace them.
+   
+
+   :tag: The tag of the analyzer.
+   
+
+   :p: The well-known port to associate with the analyzer.
+   
+
+   :returns: True if the port was successfully registered.
+
+.. bro:id:: Analyzer::register_for_ports
+
+   :Type: :bro:type:`function` (tag: :bro:type:`Analyzer::Tag`, ports: :bro:type:`set` [:bro:type:`port`]) : :bro:type:`bool`
+
+   Registers a set of well-known ports for an analyzer. If a future
+   connection on one of these ports is seen, the analyzer will be
+   automatically assigned to parsing it. The function *adds* to all ports
+   already registered, it doesn't replace them.
+   
+
+   :tag: The tag of the analyzer.
+   
+
+   :ports: The set of well-known ports to associate with the analyzer.
+   
+
+   :returns: True if the ports were successfully registered.
+
+.. bro:id:: Analyzer::registered_ports
+
+   :Type: :bro:type:`function` (tag: :bro:type:`Analyzer::Tag`) : :bro:type:`set` [:bro:type:`port`]
+
+   Returns a set of all well-known ports currently registered for a
+   specific analyzer.
+   
+
+   :tag: The tag of the analyzer.
+   
+
+   :returns: The set of ports.
+
+.. bro:id:: Analyzer::schedule_analyzer
+
+   :Type: :bro:type:`function` (orig: :bro:type:`addr`, resp: :bro:type:`addr`, resp_p: :bro:type:`port`, analyzer: :bro:type:`Analyzer::Tag`, tout: :bro:type:`interval`) : :bro:type:`bool`
+
+   Schedules an analyzer for a future connection originating from a
+   given IP address and port.
+   
+
+   :orig: The IP address originating a connection in the future.
+         0.0.0.0 can be used as a wildcard to match any originator address.
+   
+
+   :resp: The IP address responding to a connection from *orig*.
+   
+
+   :resp_p: The destination port at *resp*.
+   
+
+   :analyzer: The analyzer ID.
+   
+
+   :tout: A timeout interval after which the scheduling request will be
+         discarded if the connection has not yet been seen.
+   
+
+   :returns: True if successful.
+
+

doc/scripts/base/frameworks/broker/__load__.bro.rst

@@ -0,0 +1,14 @@
+:tocdepth: 3
+
+base/frameworks/broker/__load__.bro
+===================================
+
+
+:Imports: :doc:`base/frameworks/broker/log.bro </scripts/base/frameworks/broker/log.bro>`, :doc:`base/frameworks/broker/main.bro </scripts/base/frameworks/broker/main.bro>`, :doc:`base/frameworks/broker/store.bro </scripts/base/frameworks/broker/store.bro>`
+
+Summary
+~~~~~~~
+
+Detailed Interface
+~~~~~~~~~~~~~~~~~~
+

doc/scripts/base/frameworks/broker/index.rst

@@ -0,0 +1,22 @@
+:orphan:
+
+Package: base/frameworks/broker
+===============================
+
+The Broker communication framework facilitates connecting to remote Bro
+instances to share state and transfer events.
+
+:doc:`/scripts/base/frameworks/broker/__load__.bro`
+
+
+:doc:`/scripts/base/frameworks/broker/main.bro`
+
+   The Broker-based communication API and its various options.
+
+:doc:`/scripts/base/frameworks/broker/store.bro`
+
+   The Broker-based data store API and its various options.
+
+:doc:`/scripts/base/frameworks/broker/log.bro`
+
+

doc/scripts/base/frameworks/broker/log.bro.rst

@@ -0,0 +1,67 @@
+:tocdepth: 3
+
+base/frameworks/broker/log.bro
+==============================
+.. bro:namespace:: Broker
+
+
+:Namespace: Broker
+:Imports: :doc:`base/frameworks/broker/main.bro </scripts/base/frameworks/broker/main.bro>`
+
+Summary
+~~~~~~~
+Types
+#####
+============================================ =============================================================
+:bro:type:`Broker::Info`: :bro:type:`record` A record type containing the column fields of the Broker log.
+:bro:type:`Broker::Type`: :bro:type:`enum`   The type of a Broker activity being logged.
+============================================ =============================================================
+
+Redefinitions
+#############
+===================================== =====================================
+:bro:type:`Log::ID`: :bro:type:`enum` The Broker logging stream identifier.
+===================================== =====================================
+
+
+Detailed Interface
+~~~~~~~~~~~~~~~~~~
+Types
+#####
+.. bro:type:: Broker::Info
+
+   :Type: :bro:type:`record`
+
+      ts: :bro:type:`time` :bro:attr:`&log`
+         The network time at which a Broker event occurred.
+
+      ty: :bro:type:`Broker::Type` :bro:attr:`&log`
+         The type of the Broker event.
+
+      ev: :bro:type:`string` :bro:attr:`&log`
+         The event being logged.
+
+      peer: :bro:type:`Broker::NetworkInfo` :bro:attr:`&log` :bro:attr:`&optional`
+         The peer (if any) with which a Broker event is
+         concerned.
+
+      message: :bro:type:`string` :bro:attr:`&log` :bro:attr:`&optional`
+         An optional message describing the Broker event in more detail
+
+   A record type containing the column fields of the Broker log.
+
+.. bro:type:: Broker::Type
+
+   :Type: :bro:type:`enum`
+
+      .. bro:enum:: Broker::STATUS Broker::Type
+
+         An informational status update.
+
+      .. bro:enum:: Broker::ERROR Broker::Type
+
+         An error situation.
+
+   The type of a Broker activity being logged.
+
+

doc/scripts/base/frameworks/broker/main.bro.rst

@@ -0,0 +1,696 @@
+:tocdepth: 3
+
+base/frameworks/broker/main.bro
+===============================
+.. bro:namespace:: Broker
+
+The Broker-based communication API and its various options.
+
+:Namespace: Broker
+:Imports: :doc:`base/bif/comm.bif.bro </scripts/base/bif/comm.bif.bro>`, :doc:`base/bif/messaging.bif.bro </scripts/base/bif/messaging.bif.bro>`
+
+Summary
+~~~~~~~
+Redefinable Options
+###################
+================================================================================= ======================================================================
+:bro:id:`Broker::aggressive_interval`: :bro:type:`count` :bro:attr:`&redef`       Frequency of work-stealing polling attempts for Broker/CAF threads
+                                                                                  in "aggressive" mode.
+:bro:id:`Broker::aggressive_polls`: :bro:type:`count` :bro:attr:`&redef`          Number of work-stealing polling attempts for Broker/CAF threads
+                                                                                  in "aggressive" mode.
+:bro:id:`Broker::congestion_queue_size`: :bro:type:`count` :bro:attr:`&redef`     The number of buffered messages at the Broker/CAF layer after which
+                                                                                  a subscriber considers themselves congested (i.e.
+:bro:id:`Broker::default_connect_retry`: :bro:type:`interval` :bro:attr:`&redef`  Default interval to retry connecting to a peer if it cannot be made to
+                                                                                  work initially, or if it ever becomes disconnected.
+:bro:id:`Broker::default_listen_address`: :bro:type:`string` :bro:attr:`&redef`   Default address on which to listen.
+:bro:id:`Broker::default_listen_retry`: :bro:type:`interval` :bro:attr:`&redef`   Default interval to retry listening on a port if it's currently in
+                                                                                  use already.
+:bro:id:`Broker::default_log_topic_prefix`: :bro:type:`string` :bro:attr:`&redef` The default topic prefix where logs will be published.
+:bro:id:`Broker::default_port`: :bro:type:`port` :bro:attr:`&redef`               Default port for Broker communication.
+:bro:id:`Broker::disable_ssl`: :bro:type:`bool` :bro:attr:`&redef`                If true, do not use SSL for network connections.
+:bro:id:`Broker::forward_messages`: :bro:type:`bool` :bro:attr:`&redef`           Forward all received messages to subscribing peers.
+:bro:id:`Broker::max_threads`: :bro:type:`count` :bro:attr:`&redef`               Max number of threads to use for Broker/CAF functionality.
+:bro:id:`Broker::moderate_interval`: :bro:type:`count` :bro:attr:`&redef`         Frequency of work-stealing polling attempts for Broker/CAF threads
+                                                                                  in "moderate" mode.
+:bro:id:`Broker::moderate_polls`: :bro:type:`count` :bro:attr:`&redef`            Number of work-stealing polling attempts for Broker/CAF threads
+                                                                                  in "moderate" mode.
+:bro:id:`Broker::moderate_sleep`: :bro:type:`interval` :bro:attr:`&redef`         Interval of time for under-utilized Broker/CAF threads to sleep
+                                                                                  when in "moderate" mode.
+:bro:id:`Broker::relaxed_interval`: :bro:type:`count` :bro:attr:`&redef`          Frequency of work-stealing polling attempts for Broker/CAF threads
+                                                                                  in "relaxed" mode.
+:bro:id:`Broker::relaxed_sleep`: :bro:type:`interval` :bro:attr:`&redef`          Interval of time for under-utilized Broker/CAF threads to sleep
+                                                                                  when in "relaxed" mode.
+:bro:id:`Broker::ssl_cafile`: :bro:type:`string` :bro:attr:`&redef`               Path to a file containing concatenated trusted certificates 
+                                                                                  in PEM format.
+:bro:id:`Broker::ssl_capath`: :bro:type:`string` :bro:attr:`&redef`               Path to an OpenSSL-style directory of trusted certificates.
+:bro:id:`Broker::ssl_certificate`: :bro:type:`string` :bro:attr:`&redef`          Path to a file containing a X.509 certificate for this
+                                                                                  node in PEM format.
+:bro:id:`Broker::ssl_keyfile`: :bro:type:`string` :bro:attr:`&redef`              Path to the file containing the private key for this node's
+                                                                                  certificate.
+:bro:id:`Broker::ssl_passphrase`: :bro:type:`string` :bro:attr:`&redef`           Passphrase to decrypt the private key specified by
+                                                                                  :bro:see:`Broker::ssl_keyfile`.
+================================================================================= ======================================================================
+
+Types
+#####
+==================================================== ====================================================================
+:bro:type:`Broker::Data`: :bro:type:`record`         Opaque communication data.
+:bro:type:`Broker::DataVector`: :bro:type:`vector`   Opaque communication data sequence.
+:bro:type:`Broker::EndpointInfo`: :bro:type:`record` 
+:bro:type:`Broker::ErrorCode`: :bro:type:`enum`      Enumerates the possible error types.
+:bro:type:`Broker::Event`: :bro:type:`record`        Opaque event communication data.
+:bro:type:`Broker::NetworkInfo`: :bro:type:`record`  
+:bro:type:`Broker::PeerInfo`: :bro:type:`record`     
+:bro:type:`Broker::PeerInfos`: :bro:type:`vector`    
+:bro:type:`Broker::PeerStatus`: :bro:type:`enum`     The possible states of a peer endpoint.
+:bro:type:`Broker::TableItem`: :bro:type:`record`    Opaque communication data used as a convenient way to wrap key-value
+                                                     pairs that comprise table entries.
+==================================================== ====================================================================
+
+Functions
+#########
+==================================================================== =======================================================================
+:bro:id:`Broker::auto_publish`: :bro:type:`function`                 Automatically send an event to any interested peers whenever it is
+                                                                     locally dispatched.
+:bro:id:`Broker::auto_unpublish`: :bro:type:`function`               Stop automatically sending an event to peers upon local dispatch.
+:bro:id:`Broker::default_log_topic`: :bro:type:`function`            The default implementation for :bro:see:`Broker::log_topic`.
+:bro:id:`Broker::flush_logs`: :bro:type:`function`                   Sends all pending log messages to remote peers.
+:bro:id:`Broker::forward`: :bro:type:`function`                      Register a topic prefix subscription for events that should only be
+                                                                     forwarded to any subscribing peers and not raise any event handlers
+                                                                     on the receiving/forwarding node.
+:bro:id:`Broker::listen`: :bro:type:`function`                       Listen for remote connections.
+:bro:id:`Broker::log_topic`: :bro:type:`function` :bro:attr:`&redef` A function that will be called for each log entry to determine what
+                                                                     broker topic string will be used for sending it to peers.
+:bro:id:`Broker::node_id`: :bro:type:`function`                      Get a unique identifier for the local broker endpoint.
+:bro:id:`Broker::peer`: :bro:type:`function`                         Initiate a remote connection.
+:bro:id:`Broker::peers`: :bro:type:`function`                        Get a list of all peer connections.
+:bro:id:`Broker::publish_id`: :bro:type:`function`                   Publishes the value of an identifier to a given topic.
+:bro:id:`Broker::subscribe`: :bro:type:`function`                    Register interest in all peer event messages that use a certain topic
+                                                                     prefix.
+:bro:id:`Broker::unpeer`: :bro:type:`function`                       Remove a remote connection.
+:bro:id:`Broker::unsubscribe`: :bro:type:`function`                  Unregister interest in all peer event messages that use a topic prefix.
+==================================================================== =======================================================================
+
+
+Detailed Interface
+~~~~~~~~~~~~~~~~~~
+Redefinable Options
+###################
+.. bro:id:: Broker::aggressive_interval
+
+   :Type: :bro:type:`count`
+   :Attributes: :bro:attr:`&redef`
+   :Default: ``4``
+
+   Frequency of work-stealing polling attempts for Broker/CAF threads
+   in "aggressive" mode.
+
+.. bro:id:: Broker::aggressive_polls
+
+   :Type: :bro:type:`count`
+   :Attributes: :bro:attr:`&redef`
+   :Default: ``5``
+
+   Number of work-stealing polling attempts for Broker/CAF threads
+   in "aggressive" mode.
+
+.. bro:id:: Broker::congestion_queue_size
+
+   :Type: :bro:type:`count`
+   :Attributes: :bro:attr:`&redef`
+   :Default: ``200``
+
+   The number of buffered messages at the Broker/CAF layer after which
+   a subscriber considers themselves congested (i.e. tune the congestion
+   control mechanisms).
+
+.. bro:id:: Broker::default_connect_retry
+
+   :Type: :bro:type:`interval`
+   :Attributes: :bro:attr:`&redef`
+   :Default: ``30.0 secs``
+
+   Default interval to retry connecting to a peer if it cannot be made to
+   work initially, or if it ever becomes disconnected.  Use of the
+   BRO_DEFAULT_CONNECT_RETRY environment variable (set as number of
+   seconds) will override this option and also any values given to
+   :bro:see:`Broker::peer`.
+
+.. bro:id:: Broker::default_listen_address
+
+   :Type: :bro:type:`string`
+   :Attributes: :bro:attr:`&redef`
+   :Default: ``""``
+
+   Default address on which to listen.
+   
+   .. bro:see:: Broker::listen
+
+.. bro:id:: Broker::default_listen_retry
+
+   :Type: :bro:type:`interval`
+   :Attributes: :bro:attr:`&redef`
+   :Default: ``30.0 secs``
+
+   Default interval to retry listening on a port if it's currently in
+   use already.  Use of the BRO_DEFAULT_LISTEN_RETRY environment variable
+   (set as a number of seconds) will override this option and also
+   any values given to :bro:see:`Broker::listen`.
+
+.. bro:id:: Broker::default_log_topic_prefix
+
+   :Type: :bro:type:`string`
+   :Attributes: :bro:attr:`&redef`
+   :Default: ``"bro/logs/"``
+
+   The default topic prefix where logs will be published.  The log's stream
+   id is appended when writing to a particular stream.
+
+.. bro:id:: Broker::default_port
+
+   :Type: :bro:type:`port`
+   :Attributes: :bro:attr:`&redef`
+   :Default: ``9999/tcp``
+
+   Default port for Broker communication. Where not specified
+   otherwise, this is the port to connect to and listen on.
+
+.. bro:id:: Broker::disable_ssl
+
+   :Type: :bro:type:`bool`
+   :Attributes: :bro:attr:`&redef`
+   :Default: ``F``
+
+   If true, do not use SSL for network connections. By default, SSL will
+   even be used if no certificates / CAs have been configured. In that case
+   (which is the default) the communication will be encrypted, but not
+   authenticated.
+
+.. bro:id:: Broker::forward_messages
+
+   :Type: :bro:type:`bool`
+   :Attributes: :bro:attr:`&redef`
+   :Default: ``F``
+
+   Forward all received messages to subscribing peers.
+
+.. bro:id:: Broker::max_threads
+
+   :Type: :bro:type:`count`
+   :Attributes: :bro:attr:`&redef`
+   :Default: ``1``
+
+   Max number of threads to use for Broker/CAF functionality.  The
+   BRO_BROKER_MAX_THREADS environment variable overrides this setting.
+
+.. bro:id:: Broker::moderate_interval
+
+   :Type: :bro:type:`count`
+   :Attributes: :bro:attr:`&redef`
+   :Default: ``2``
+
+   Frequency of work-stealing polling attempts for Broker/CAF threads
+   in "moderate" mode.
+
+.. bro:id:: Broker::moderate_polls
+
+   :Type: :bro:type:`count`
+   :Attributes: :bro:attr:`&redef`
+   :Default: ``5``
+
+   Number of work-stealing polling attempts for Broker/CAF threads
+   in "moderate" mode.
+
+.. bro:id:: Broker::moderate_sleep
+
+   :Type: :bro:type:`interval`
+   :Attributes: :bro:attr:`&redef`
+   :Default: ``16.0 msecs``
+
+   Interval of time for under-utilized Broker/CAF threads to sleep
+   when in "moderate" mode.
+
+.. bro:id:: Broker::relaxed_interval
+
+   :Type: :bro:type:`count`
+   :Attributes: :bro:attr:`&redef`
+   :Default: ``1``
+
+   Frequency of work-stealing polling attempts for Broker/CAF threads
+   in "relaxed" mode.
+
+.. bro:id:: Broker::relaxed_sleep
+
+   :Type: :bro:type:`interval`
+   :Attributes: :bro:attr:`&redef`
+   :Default: ``64.0 msecs``
+
+   Interval of time for under-utilized Broker/CAF threads to sleep
+   when in "relaxed" mode.
+
+.. bro:id:: Broker::ssl_cafile
+
+   :Type: :bro:type:`string`
+   :Attributes: :bro:attr:`&redef`
+   :Default: ``""``
+
+   Path to a file containing concatenated trusted certificates 
+   in PEM format. If set, Bro will require valid certificates for
+   all peers.
+
+.. bro:id:: Broker::ssl_capath
+
+   :Type: :bro:type:`string`
+   :Attributes: :bro:attr:`&redef`
+   :Default: ``""``
+
+   Path to an OpenSSL-style directory of trusted certificates.
+   If set, Bro will require valid certificates for
+   all peers.
+
+.. bro:id:: Broker::ssl_certificate
+
+   :Type: :bro:type:`string`
+   :Attributes: :bro:attr:`&redef`
+   :Default: ``""``
+
+   Path to a file containing a X.509 certificate for this
+   node in PEM format. If set, Bro will require valid certificates for
+   all peers.
+
+.. bro:id:: Broker::ssl_keyfile
+
+   :Type: :bro:type:`string`
+   :Attributes: :bro:attr:`&redef`
+   :Default: ``""``
+
+   Path to the file containing the private key for this node's
+   certificate. If set, Bro will require valid certificates for
+   all peers.
+
+.. bro:id:: Broker::ssl_passphrase
+
+   :Type: :bro:type:`string`
+   :Attributes: :bro:attr:`&redef`
+   :Default: ``""``
+
+   Passphrase to decrypt the private key specified by
+   :bro:see:`Broker::ssl_keyfile`. If set, Bro will require valid
+   certificates for all peers.
+
+Types
+#####
+.. bro:type:: Broker::Data
+
+   :Type: :bro:type:`record`
+
+      data: :bro:type:`opaque` of Broker::Data :bro:attr:`&optional`
+
+   Opaque communication data.
+
+.. bro:type:: Broker::DataVector
+
+   :Type: :bro:type:`vector` of :bro:type:`Broker::Data`
+
+   Opaque communication data sequence.
+
+.. bro:type:: Broker::EndpointInfo
+
+   :Type: :bro:type:`record`
+
+      id: :bro:type:`string`
+         A unique identifier of the node.
+
+      network: :bro:type:`Broker::NetworkInfo` :bro:attr:`&optional`
+         Network-level information.
+
+
+.. bro:type:: Broker::ErrorCode
+
+   :Type: :bro:type:`enum`
+
+      .. bro:enum:: Broker::UNSPECIFIED Broker::ErrorCode
+
+         The unspecified default error code.
+
+      .. bro:enum:: Broker::PEER_INCOMPATIBLE Broker::ErrorCode
+
+         Version incompatibility.
+
+      .. bro:enum:: Broker::PEER_INVALID Broker::ErrorCode
+
+         Referenced peer does not exist.
+
+      .. bro:enum:: Broker::PEER_UNAVAILABLE Broker::ErrorCode
+
+         Remote peer not listening.
+
+      .. bro:enum:: Broker::PEER_TIMEOUT Broker::ErrorCode
+
+         A peering request timed out.
+
+      .. bro:enum:: Broker::MASTER_EXISTS Broker::ErrorCode
+
+         Master with given name already exists.
+
+      .. bro:enum:: Broker::NO_SUCH_MASTER Broker::ErrorCode
+
+         Master with given name does not exist.
+
+      .. bro:enum:: Broker::NO_SUCH_KEY Broker::ErrorCode
+
+         The given data store key does not exist.
+
+      .. bro:enum:: Broker::REQUEST_TIMEOUT Broker::ErrorCode
+
+         The store operation timed out.
+
+      .. bro:enum:: Broker::TYPE_CLASH Broker::ErrorCode
+
+         The operation expected a different type than provided.
+
+      .. bro:enum:: Broker::INVALID_DATA Broker::ErrorCode
+
+         The data value cannot be used to carry out the desired operation.
+
+      .. bro:enum:: Broker::BACKEND_FAILURE Broker::ErrorCode
+
+         The storage backend failed to execute the operation.
+
+      .. bro:enum:: Broker::STALE_DATA Broker::ErrorCode
+
+         The storage backend failed to execute the operation.
+
+      .. bro:enum:: Broker::CAF_ERROR Broker::ErrorCode
+
+         Catch-all for a CAF-level problem.
+
+   Enumerates the possible error types. 
+
+.. bro:type:: Broker::Event
+
+   :Type: :bro:type:`record`
+
+      name: :bro:type:`string` :bro:attr:`&optional`
+         The name of the event.  Not set if invalid event or arguments.
+
+      args: :bro:type:`Broker::DataVector`
+         The arguments to the event.
+
+   Opaque event communication data.
+
+.. bro:type:: Broker::NetworkInfo
+
+   :Type: :bro:type:`record`
+
+      address: :bro:type:`string` :bro:attr:`&log`
+         The IP address or hostname where the endpoint listens.
+
+      bound_port: :bro:type:`port` :bro:attr:`&log`
+         The port where the endpoint is bound to.
+
+
+.. bro:type:: Broker::PeerInfo
+
+   :Type: :bro:type:`record`
+
+      peer: :bro:type:`Broker::EndpointInfo`
+
+      status: :bro:type:`Broker::PeerStatus`
+
+
+.. bro:type:: Broker::PeerInfos
+
+   :Type: :bro:type:`vector` of :bro:type:`Broker::PeerInfo`
+
+
+.. bro:type:: Broker::PeerStatus
+
+   :Type: :bro:type:`enum`
+
+      .. bro:enum:: Broker::INITIALIZING Broker::PeerStatus
+
+         The peering process is initiated.
+
+      .. bro:enum:: Broker::CONNECTING Broker::PeerStatus
+
+         Connection establishment in process.
+
+      .. bro:enum:: Broker::CONNECTED Broker::PeerStatus
+
+         Connection established, peering pending.
+
+      .. bro:enum:: Broker::PEERED Broker::PeerStatus
+
+         Successfully peered.
+
+      .. bro:enum:: Broker::DISCONNECTED Broker::PeerStatus
+
+         Connection to remote peer lost.
+
+      .. bro:enum:: Broker::RECONNECTING Broker::PeerStatus
+
+         Reconnecting to peer after a lost connection.
+
+   The possible states of a peer endpoint.
+
+.. bro:type:: Broker::TableItem
+
+   :Type: :bro:type:`record`
+
+      key: :bro:type:`Broker::Data`
+
+      val: :bro:type:`Broker::Data`
+
+   Opaque communication data used as a convenient way to wrap key-value
+   pairs that comprise table entries.
+
+Functions
+#########
+.. bro:id:: Broker::auto_publish
+
+   :Type: :bro:type:`function` (topic: :bro:type:`string`, ev: :bro:type:`any`) : :bro:type:`bool`
+
+   Automatically send an event to any interested peers whenever it is
+   locally dispatched. (For example, using "event my_event(...);" in a
+   script.)
+   
+
+   :topic: a topic string associated with the event message.
+          Peers advertise interest by registering a subscription to some
+          prefix of this topic name.
+   
+
+   :ev: a Bro event value.
+   
+
+   :returns: true if automatic event sending is now enabled.
+
+.. bro:id:: Broker::auto_unpublish
+
+   :Type: :bro:type:`function` (topic: :bro:type:`string`, ev: :bro:type:`any`) : :bro:type:`bool`
+
+   Stop automatically sending an event to peers upon local dispatch.
+   
+
+   :topic: a topic originally given to :bro:see:`Broker::auto_publish`.
+   
+
+   :ev: an event originally given to :bro:see:`Broker::auto_publish`.
+   
+
+   :returns: true if automatic events will not occur for the topic/event
+            pair.
+
+.. bro:id:: Broker::default_log_topic
+
+   :Type: :bro:type:`function` (id: :bro:type:`Log::ID`, path: :bro:type:`string`) : :bro:type:`string`
+
+   The default implementation for :bro:see:`Broker::log_topic`.
+
+.. bro:id:: Broker::flush_logs
+
+   :Type: :bro:type:`function` () : :bro:type:`count`
+
+   Sends all pending log messages to remote peers.  This normally
+   doesn't need to be used except for test cases that are time-sensitive.
+
+.. bro:id:: Broker::forward
+
+   :Type: :bro:type:`function` (topic_prefix: :bro:type:`string`) : :bro:type:`bool`
+
+   Register a topic prefix subscription for events that should only be
+   forwarded to any subscribing peers and not raise any event handlers
+   on the receiving/forwarding node.  i.e. it's the same as
+   :bro:see:`Broker::subscribe` except matching events are not raised
+   on the receiver, just forwarded.  Use :bro:see:`Broker::unsubscribe`
+   with the same argument to undo this operation.
+   
+
+   :topic_prefix: a prefix to match against remote message topics.
+                 e.g. an empty prefix matches everything and "a" matches
+                 "alice" and "amy" but not "bob".
+   
+
+   :returns: true if a new event forwarding/subscription is now registered.
+
+.. bro:id:: Broker::listen
+
+   :Type: :bro:type:`function` (a: :bro:type:`string` :bro:attr:`&default` = :bro:see:`Broker::default_listen_address` :bro:attr:`&optional`, p: :bro:type:`port` :bro:attr:`&default` = :bro:see:`Broker::default_port` :bro:attr:`&optional`, retry: :bro:type:`interval` :bro:attr:`&default` = :bro:see:`Broker::default_listen_retry` :bro:attr:`&optional`) : :bro:type:`port`
+
+   Listen for remote connections.
+   
+
+   :a: an address string on which to accept connections, e.g.
+      "127.0.0.1".  An empty string refers to INADDR_ANY.
+   
+
+   :p: the TCP port to listen on. The value 0 means that the OS should choose
+      the next available free port.
+   
+
+   :retry: If non-zero, retries listening in regular intervals if the port cannot be
+          acquired immediately. 0 disables retries.  If the
+          BRO_DEFAULT_LISTEN_RETRY environment variable is set (as number
+          of seconds), it overrides any value given here.
+   
+
+   :returns: the bound port or 0/? on failure.
+   
+   .. bro:see:: Broker::status
+
+.. bro:id:: Broker::log_topic
+
+   :Type: :bro:type:`function` (id: :bro:type:`Log::ID`, path: :bro:type:`string`) : :bro:type:`string`
+   :Attributes: :bro:attr:`&redef`
+
+   A function that will be called for each log entry to determine what
+   broker topic string will be used for sending it to peers.  The
+   default implementation will return a value based on
+   :bro:see:`Broker::default_log_topic_prefix`.
+   
+
+   :id: the ID associated with the log stream entry that will be sent.
+   
+
+   :path: the path to which the log stream entry will be output.
+   
+
+   :returns: a string representing the broker topic to which the log
+            will be sent.
+
+.. bro:id:: Broker::node_id
+
+   :Type: :bro:type:`function` () : :bro:type:`string`
+
+   Get a unique identifier for the local broker endpoint.
+   
+
+   :returns: a unique identifier for the local broker endpoint.
+
+.. bro:id:: Broker::peer
+
+   :Type: :bro:type:`function` (a: :bro:type:`string`, p: :bro:type:`port` :bro:attr:`&default` = :bro:see:`Broker::default_port` :bro:attr:`&optional`, retry: :bro:type:`interval` :bro:attr:`&default` = :bro:see:`Broker::default_connect_retry` :bro:attr:`&optional`) : :bro:type:`bool`
+
+   Initiate a remote connection.
+   
+
+   :a: an address to connect to, e.g. "localhost" or "127.0.0.1".
+   
+
+   :p: the TCP port on which the remote side is listening.
+   
+
+   :retry: an interval at which to retry establishing the
+          connection with the remote peer if it cannot be made initially, or
+          if it ever becomes disconnected.  If the
+          BRO_DEFAULT_CONNECT_RETRY environment variable is set (as number
+          of seconds), it overrides any value given here.
+   
+
+   :returns: true if it's possible to try connecting with the peer and
+            it's a new peer. The actual connection may not be established
+            until a later point in time.
+   
+   .. bro:see:: Broker::status
+
+.. bro:id:: Broker::peers
+
+   :Type: :bro:type:`function` () : :bro:type:`vector` of :bro:type:`Broker::PeerInfo`
+
+   Get a list of all peer connections.
+   
+
+   :returns: a list of all peer connections.
+
+.. bro:id:: Broker::publish_id
+
+   :Type: :bro:type:`function` (topic: :bro:type:`string`, id: :bro:type:`string`) : :bro:type:`bool`
+
+   Publishes the value of an identifier to a given topic.  The subscribers
+   will update their local value for that identifier on receipt.
+   
+
+   :topic: a topic associated with the message.
+   
+
+   :id: the identifier to publish.
+   
+
+   :returns: true if the message is sent.
+
+.. bro:id:: Broker::subscribe
+
+   :Type: :bro:type:`function` (topic_prefix: :bro:type:`string`) : :bro:type:`bool`
+
+   Register interest in all peer event messages that use a certain topic
+   prefix.  Note that subscriptions may not be altered immediately after
+   calling (except during :bro:see:`bro_init`).
+   
+
+   :topic_prefix: a prefix to match against remote message topics.
+                 e.g. an empty prefix matches everything and "a" matches
+                 "alice" and "amy" but not "bob".
+   
+
+   :returns: true if it's a new event subscription and it is now registered.
+
+.. bro:id:: Broker::unpeer
+
+   :Type: :bro:type:`function` (a: :bro:type:`string`, p: :bro:type:`port`) : :bro:type:`bool`
+
+   Remove a remote connection.
+   
+   Note that this does not terminate the connection to the peer, it
+   just means that we won't exchange any further information with it
+   unless peering resumes later.
+   
+
+   :a: the address used in previous successful call to :bro:see:`Broker::peer`.
+   
+
+   :p: the port used in previous successful call to :bro:see:`Broker::peer`.
+   
+
+   :returns: true if the arguments match a previously successful call to
+            :bro:see:`Broker::peer`.
+   
+
+   :TODO: We do not have a function yet to terminate a connection.
+
+.. bro:id:: Broker::unsubscribe
+
+   :Type: :bro:type:`function` (topic_prefix: :bro:type:`string`) : :bro:type:`bool`
+
+   Unregister interest in all peer event messages that use a topic prefix.
+   Note that subscriptions may not be altered immediately after calling
+   (except during :bro:see:`bro_init`).
+   
+
+   :topic_prefix: a prefix previously supplied to a successful call to
+                 :bro:see:`Broker::subscribe` or :bro:see:`Broker::forward`.
+   
+
+   :returns: true if interest in the topic prefix is no longer advertised.
+
+

doc/scripts/base/frameworks/cluster/__load__.bro.rst

@@ -0,0 +1,14 @@
+:tocdepth: 3
+
+base/frameworks/cluster/__load__.bro
+====================================
+
+
+:Imports: :doc:`base/frameworks/cluster/main.bro </scripts/base/frameworks/cluster/main.bro>`, :doc:`base/frameworks/cluster/pools.bro </scripts/base/frameworks/cluster/pools.bro>`
+
+Summary
+~~~~~~~
+
+Detailed Interface
+~~~~~~~~~~~~~~~~~~
+

doc/scripts/base/frameworks/cluster/index.rst

@@ -0,0 +1,26 @@
+:orphan:
+
+Package: base/frameworks/cluster
+================================
+
+The cluster framework provides for establishing and controlling a cluster
+of Bro instances.
+
+:doc:`/scripts/base/frameworks/cluster/__load__.bro`
+
+
+:doc:`/scripts/base/frameworks/cluster/main.bro`
+
+   A framework for establishing and controlling a cluster of Bro instances.
+   In order to use the cluster framework, a script named
+   ``cluster-layout.bro`` must exist somewhere in Bro's script search path
+   which has a cluster definition of the :bro:id:`Cluster::nodes` variable.
+   The ``CLUSTER_NODE`` environment variable or :bro:id:`Cluster::node`
+   must also be sent and the cluster framework loaded as a package like
+   ``@load base/frameworks/cluster``.
+
+:doc:`/scripts/base/frameworks/cluster/pools.bro`
+
+   Defines an interface for managing pools of cluster nodes.  Pools are
+   a useful way to distribute work or data among nodes within a cluster.
+

doc/scripts/base/frameworks/cluster/main.bro.rst

@@ -0,0 +1,529 @@
+:tocdepth: 3
+
+base/frameworks/cluster/main.bro
+================================
+.. bro:namespace:: Cluster
+
+A framework for establishing and controlling a cluster of Bro instances.
+In order to use the cluster framework, a script named
+``cluster-layout.bro`` must exist somewhere in Bro's script search path
+which has a cluster definition of the :bro:id:`Cluster::nodes` variable.
+The ``CLUSTER_NODE`` environment variable or :bro:id:`Cluster::node`
+must also be sent and the cluster framework loaded as a package like
+``@load base/frameworks/cluster``.
+
+:Namespace: Cluster
+:Imports: :doc:`base/frameworks/broker </scripts/base/frameworks/broker/index>`, :doc:`base/frameworks/control </scripts/base/frameworks/control/index>`
+
+Summary
+~~~~~~~
+Redefinable Options
+###################
+================================================================================================= ==============================================================================
+:bro:id:`Cluster::default_backend`: :bro:type:`Broker::BackendType` :bro:attr:`&redef`            The type of data store backend that will be used for all data stores if
+                                                                                                  no other has already been specified by the user in :bro:see:`Cluster::stores`.
+:bro:id:`Cluster::default_master_node`: :bro:type:`string` :bro:attr:`&redef`                     Name of the node on which master data stores will be created if no other
+                                                                                                  has already been specified by the user in :bro:see:`Cluster::stores`.
+:bro:id:`Cluster::default_persistent_backend`: :bro:type:`Broker::BackendType` :bro:attr:`&redef` The type of persistent data store backend that will be used for all data
+                                                                                                  stores if no other has already been specified by the user in
+                                                                                                  :bro:see:`Cluster::stores`.
+:bro:id:`Cluster::default_store_dir`: :bro:type:`string` :bro:attr:`&redef`                       Setting a default dir will, for persistent backends that have not
+                                                                                                  been given an explicit file path via :bro:see:`Cluster::stores`,
+                                                                                                  automatically create a path within this dir that is based on the name of
+                                                                                                  the data store.
+:bro:id:`Cluster::enable_round_robin_logging`: :bro:type:`bool` :bro:attr:`&redef`                Whether to distribute log messages among available logging nodes.
+:bro:id:`Cluster::logger_topic`: :bro:type:`string` :bro:attr:`&redef`                            The topic name used for exchanging messages that are relevant to
+                                                                                                  logger nodes in a cluster.
+:bro:id:`Cluster::manager_is_logger`: :bro:type:`bool` :bro:attr:`&redef`                         Indicates whether or not the manager will act as the logger and receive
+                                                                                                  logs.
+:bro:id:`Cluster::manager_topic`: :bro:type:`string` :bro:attr:`&redef`                           The topic name used for exchanging messages that are relevant to
+                                                                                                  manager nodes in a cluster.
+:bro:id:`Cluster::node`: :bro:type:`string` :bro:attr:`&redef`                                    This is usually supplied on the command line for each instance
+                                                                                                  of the cluster that is started up.
+:bro:id:`Cluster::node_topic_prefix`: :bro:type:`string` :bro:attr:`&redef`                       The topic prefix used for exchanging messages that are relevant to
+                                                                                                  a named node in a cluster.
+:bro:id:`Cluster::nodeid_topic_prefix`: :bro:type:`string` :bro:attr:`&redef`                     The topic prefix used for exchanging messages that are relevant to
+                                                                                                  a unique node in a cluster.
+:bro:id:`Cluster::nodes`: :bro:type:`table` :bro:attr:`&redef`                                    The cluster layout definition.
+:bro:id:`Cluster::proxy_topic`: :bro:type:`string` :bro:attr:`&redef`                             The topic name used for exchanging messages that are relevant to
+                                                                                                  proxy nodes in a cluster.
+:bro:id:`Cluster::retry_interval`: :bro:type:`interval` :bro:attr:`&redef`                        Interval for retrying failed connections between cluster nodes.
+:bro:id:`Cluster::time_machine_topic`: :bro:type:`string` :bro:attr:`&redef`                      The topic name used for exchanging messages that are relevant to
+                                                                                                  time machine nodes in a cluster.
+:bro:id:`Cluster::worker_topic`: :bro:type:`string` :bro:attr:`&redef`                            The topic name used for exchanging messages that are relevant to
+                                                                                                  worker nodes in a cluster.
+================================================================================================= ==============================================================================
+
+State Variables
+###############
+================================================================================================================================================================================================================================================================================================================================================================== ======================================================================
+:bro:id:`Cluster::stores`: :bro:type:`table` :bro:attr:`&default` = ``[name=<uninitialized>, store=<uninitialized>, master_node=, master=F, backend=Broker::MEMORY, options=[sqlite=[path=], rocksdb=[path=]], clone_resync_interval=10.0 secs, clone_stale_interval=5.0 mins, clone_mutation_buffer_interval=2.0 mins]`` :bro:attr:`&optional` :bro:attr:`&redef` A table of cluster-enabled data stores that have been created, indexed
+                                                                                                                                                                                                                                                                                                                                                                   by their name.
+:bro:id:`Cluster::worker_count`: :bro:type:`count`                                                                                                                                                                                                                                                                                                                 This gives the value for the number of workers currently connected to,
+                                                                                                                                                                                                                                                                                                                                                                   and it's maintained internally by the cluster framework.
+================================================================================================================================================================================================================================================================================================================================================================== ======================================================================
+
+Types
+#####
+============================================================== ====================================================================
+:bro:type:`Cluster::Info`: :bro:type:`record` :bro:attr:`&log` The record type which contains the column fields of the cluster log.
+:bro:type:`Cluster::Node`: :bro:type:`record`                  Record type to indicate a node in a cluster.
+:bro:type:`Cluster::NodeType`: :bro:type:`enum`                Types of nodes that are allowed to participate in the cluster
+                                                               configuration.
+:bro:type:`Cluster::StoreInfo`: :bro:type:`record`             Information regarding a cluster-enabled data store.
+============================================================== ====================================================================
+
+Redefinitions
+#############
+===================================== ======================================
+:bro:type:`Log::ID`: :bro:type:`enum` The cluster logging stream identifier.
+===================================== ======================================
+
+Events
+######
+=============================================== =======================================================================
+:bro:id:`Cluster::hello`: :bro:type:`event`     When using broker-enabled cluster framework, nodes broadcast this event
+                                                to exchange their user-defined name along with a string that uniquely
+                                                identifies it for the duration of its lifetime.
+:bro:id:`Cluster::node_down`: :bro:type:`event` When using broker-enabled cluster framework, this event will be emitted
+                                                locally whenever a connected cluster node becomes disconnected.
+:bro:id:`Cluster::node_up`: :bro:type:`event`   When using broker-enabled cluster framework, this event will be emitted
+                                                locally whenever a cluster node connects or reconnects.
+=============================================== =======================================================================
+
+Functions
+#########
+======================================================== ===================================================================
+:bro:id:`Cluster::create_store`: :bro:type:`function`    Sets up a cluster-enabled data store.
+:bro:id:`Cluster::is_enabled`: :bro:type:`function`      This function can be called at any time to determine if the cluster
+                                                         framework is being enabled for this run.
+:bro:id:`Cluster::local_node_type`: :bro:type:`function` This function can be called at any time to determine what type of
+                                                         cluster node the current Bro instance is going to be acting as.
+:bro:id:`Cluster::log`: :bro:type:`function`             Write a message to the cluster logging stream.
+:bro:id:`Cluster::node_topic`: :bro:type:`function`      Retrieve the topic associated with a specific node in the cluster.
+:bro:id:`Cluster::nodeid_topic`: :bro:type:`function`    Retrieve the topic associated with a specific node in the cluster.
+======================================================== ===================================================================
+
+
+Detailed Interface
+~~~~~~~~~~~~~~~~~~
+Redefinable Options
+###################
+.. bro:id:: Cluster::default_backend
+
+   :Type: :bro:type:`Broker::BackendType`
+   :Attributes: :bro:attr:`&redef`
+   :Default: ``Broker::MEMORY``
+
+   The type of data store backend that will be used for all data stores if
+   no other has already been specified by the user in :bro:see:`Cluster::stores`.
+
+.. bro:id:: Cluster::default_master_node
+
+   :Type: :bro:type:`string`
+   :Attributes: :bro:attr:`&redef`
+   :Default: ``""``
+
+   Name of the node on which master data stores will be created if no other
+   has already been specified by the user in :bro:see:`Cluster::stores`.
+   An empty value means "use whatever name corresponds to the manager
+   node".
+
+.. bro:id:: Cluster::default_persistent_backend
+
+   :Type: :bro:type:`Broker::BackendType`
+   :Attributes: :bro:attr:`&redef`
+   :Default: ``Broker::SQLITE``
+
+   The type of persistent data store backend that will be used for all data
+   stores if no other has already been specified by the user in
+   :bro:see:`Cluster::stores`.  This will be used when script authors call
+   :bro:see:`Cluster::create_store` with the *persistent* argument set true.
+
+.. bro:id:: Cluster::default_store_dir
+
+   :Type: :bro:type:`string`
+   :Attributes: :bro:attr:`&redef`
+   :Default: ``""``
+
+   Setting a default dir will, for persistent backends that have not
+   been given an explicit file path via :bro:see:`Cluster::stores`,
+   automatically create a path within this dir that is based on the name of
+   the data store.
+
+.. bro:id:: Cluster::enable_round_robin_logging
+
+   :Type: :bro:type:`bool`
+   :Attributes: :bro:attr:`&redef`
+   :Default: ``T``
+
+   Whether to distribute log messages among available logging nodes.
+
+.. bro:id:: Cluster::logger_topic
+
+   :Type: :bro:type:`string`
+   :Attributes: :bro:attr:`&redef`
+   :Default: ``"bro/cluster/logger"``
+
+   The topic name used for exchanging messages that are relevant to
+   logger nodes in a cluster.  Used with broker-enabled cluster communication.
+
+.. bro:id:: Cluster::manager_is_logger
+
+   :Type: :bro:type:`bool`
+   :Attributes: :bro:attr:`&redef`
+   :Default: ``T``
+
+   Indicates whether or not the manager will act as the logger and receive
+   logs.  This value should be set in the cluster-layout.bro script (the
+   value should be true only if no logger is specified in Cluster::nodes).
+   Note that BroControl handles this automatically.
+
+.. bro:id:: Cluster::manager_topic
+
+   :Type: :bro:type:`string`
+   :Attributes: :bro:attr:`&redef`
+   :Default: ``"bro/cluster/manager"``
+
+   The topic name used for exchanging messages that are relevant to
+   manager nodes in a cluster.  Used with broker-enabled cluster communication.
+
+.. bro:id:: Cluster::node
+
+   :Type: :bro:type:`string`
+   :Attributes: :bro:attr:`&redef`
+   :Default: ``""``
+
+   This is usually supplied on the command line for each instance
+   of the cluster that is started up.
+
+.. bro:id:: Cluster::node_topic_prefix
+
+   :Type: :bro:type:`string`
+   :Attributes: :bro:attr:`&redef`
+   :Default: ``"bro/cluster/node/"``
+
+   The topic prefix used for exchanging messages that are relevant to
+   a named node in a cluster.  Used with broker-enabled cluster communication.
+
+.. bro:id:: Cluster::nodeid_topic_prefix
+
+   :Type: :bro:type:`string`
+   :Attributes: :bro:attr:`&redef`
+   :Default: ``"bro/cluster/nodeid/"``
+
+   The topic prefix used for exchanging messages that are relevant to
+   a unique node in a cluster.  Used with broker-enabled cluster communication.
+
+.. bro:id:: Cluster::nodes
+
+   :Type: :bro:type:`table` [:bro:type:`string`] of :bro:type:`Cluster::Node`
+   :Attributes: :bro:attr:`&redef`
+   :Default: ``{}``
+
+   The cluster layout definition.  This should be placed into a filter
+   named cluster-layout.bro somewhere in the BROPATH.  It will be
+   automatically loaded if the CLUSTER_NODE environment variable is set.
+   Note that BroControl handles all of this automatically.
+   The table is typically indexed by node names/labels (e.g. "manager"
+   or "worker-1").
+
+.. bro:id:: Cluster::proxy_topic
+
+   :Type: :bro:type:`string`
+   :Attributes: :bro:attr:`&redef`
+   :Default: ``"bro/cluster/proxy"``
+
+   The topic name used for exchanging messages that are relevant to
+   proxy nodes in a cluster.  Used with broker-enabled cluster communication.
+
+.. bro:id:: Cluster::retry_interval
+
+   :Type: :bro:type:`interval`
+   :Attributes: :bro:attr:`&redef`
+   :Default: ``1.0 min``
+
+   Interval for retrying failed connections between cluster nodes.
+   If set, the BRO_DEFAULT_CONNECT_RETRY (given in number of seconds)
+   overrides this option.
+
+.. bro:id:: Cluster::time_machine_topic
+
+   :Type: :bro:type:`string`
+   :Attributes: :bro:attr:`&redef`
+   :Default: ``"bro/cluster/time_machine"``
+
+   The topic name used for exchanging messages that are relevant to
+   time machine nodes in a cluster.  Used with broker-enabled cluster communication.
+
+.. bro:id:: Cluster::worker_topic
+
+   :Type: :bro:type:`string`
+   :Attributes: :bro:attr:`&redef`
+   :Default: ``"bro/cluster/worker"``
+
+   The topic name used for exchanging messages that are relevant to
+   worker nodes in a cluster.  Used with broker-enabled cluster communication.
+
+State Variables
+###############
+.. bro:id:: Cluster::stores
+
+   :Type: :bro:type:`table` [:bro:type:`string`] of :bro:type:`Cluster::StoreInfo`
+   :Attributes: :bro:attr:`&default` = ``[name=<uninitialized>, store=<uninitialized>, master_node=, master=F, backend=Broker::MEMORY, options=[sqlite=[path=], rocksdb=[path=]], clone_resync_interval=10.0 secs, clone_stale_interval=5.0 mins, clone_mutation_buffer_interval=2.0 mins]`` :bro:attr:`&optional` :bro:attr:`&redef`
+   :Default: ``{}``
+
+   A table of cluster-enabled data stores that have been created, indexed
+   by their name.  This table will be populated automatically by
+   :bro:see:`Cluster::create_store`, but if you need to customize
+   the options related to a particular data store, you may redef this
+   table.  Calls to :bro:see:`Cluster::create_store` will first check
+   the table for an entry of the same name and, if found, will use the
+   predefined options there when setting up the store.
+
+.. bro:id:: Cluster::worker_count
+
+   :Type: :bro:type:`count`
+   :Default: ``0``
+
+   This gives the value for the number of workers currently connected to,
+   and it's maintained internally by the cluster framework.  It's
+   primarily intended for use by managers to find out how many workers
+   should be responding to requests.
+
+Types
+#####
+.. bro:type:: Cluster::Info
+
+   :Type: :bro:type:`record`
+
+      ts: :bro:type:`time` :bro:attr:`&log`
+         The time at which a cluster message was generated.
+
+      node: :bro:type:`string` :bro:attr:`&log`
+         The name of the node that is creating the log record.
+
+      message: :bro:type:`string` :bro:attr:`&log`
+         A message indicating information about the cluster's operation.
+   :Attributes: :bro:attr:`&log`
+
+   The record type which contains the column fields of the cluster log.
+
+.. bro:type:: Cluster::Node
+
+   :Type: :bro:type:`record`
+
+      node_type: :bro:type:`Cluster::NodeType`
+         Identifies the type of cluster node in this node's configuration.
+
+      ip: :bro:type:`addr`
+         The IP address of the cluster node.
+
+      zone_id: :bro:type:`string` :bro:attr:`&default` = ``""`` :bro:attr:`&optional`
+         If the *ip* field is a non-global IPv6 address, this field
+         can specify a particular :rfc:`4007` ``zone_id``.
+
+      p: :bro:type:`port`
+         The port that this node will listen on for peer connections.
+
+      interface: :bro:type:`string` :bro:attr:`&optional`
+         Identifier for the interface a worker is sniffing.
+
+      manager: :bro:type:`string` :bro:attr:`&optional`
+         Name of the manager node this node uses.  For workers and proxies.
+
+      time_machine: :bro:type:`string` :bro:attr:`&optional`
+         Name of a time machine node with which this node connects.
+
+      id: :bro:type:`string` :bro:attr:`&optional`
+         A unique identifier assigned to the node by the broker framework.
+         This field is only set while a node is connected.
+
+      lb_filter: :bro:type:`string` :bro:attr:`&optional`
+         (present if :doc:`/scripts/policy/misc/load-balancing.bro` is loaded)
+
+         A BPF filter for load balancing traffic sniffed on a single
+         interface across a number of processes.  In normal uses, this
+         will be assigned dynamically by the manager and installed by
+         the workers.
+
+   Record type to indicate a node in a cluster.
+
+.. bro:type:: Cluster::NodeType
+
+   :Type: :bro:type:`enum`
+
+      .. bro:enum:: Cluster::NONE Cluster::NodeType
+
+         A dummy node type indicating the local node is not operating
+         within a cluster.
+
+      .. bro:enum:: Cluster::CONTROL Cluster::NodeType
+
+         A node type which is allowed to view/manipulate the configuration
+         of other nodes in the cluster.
+
+      .. bro:enum:: Cluster::LOGGER Cluster::NodeType
+
+         A node type responsible for log management.
+
+      .. bro:enum:: Cluster::MANAGER Cluster::NodeType
+
+         A node type responsible for policy management.
+
+      .. bro:enum:: Cluster::PROXY Cluster::NodeType
+
+         A node type for relaying worker node communication and synchronizing
+         worker node state.
+
+      .. bro:enum:: Cluster::WORKER Cluster::NodeType
+
+         The node type doing all the actual traffic analysis.
+
+      .. bro:enum:: Cluster::TIME_MACHINE Cluster::NodeType
+
+         A node acting as a traffic recorder using the
+         `Time Machine <http://bro.org/community/time-machine.html>`_
+         software.
+
+   Types of nodes that are allowed to participate in the cluster
+   configuration.
+
+.. bro:type:: Cluster::StoreInfo
+
+   :Type: :bro:type:`record`
+
+      name: :bro:type:`string` :bro:attr:`&optional`
+         The name of the data store.
+
+      store: :bro:type:`opaque` of Broker::Store :bro:attr:`&optional`
+         The store handle.
+
+      master_node: :bro:type:`string` :bro:attr:`&default` = :bro:see:`Cluster::default_master_node` :bro:attr:`&optional`
+         The name of the cluster node on which the master version of the data
+         store resides.
+
+      master: :bro:type:`bool` :bro:attr:`&default` = ``F`` :bro:attr:`&optional`
+         Whether the data store is the master version or a clone.
+
+      backend: :bro:type:`Broker::BackendType` :bro:attr:`&default` = :bro:see:`Cluster::default_backend` :bro:attr:`&optional`
+         The type of backend used for storing data.
+
+      options: :bro:type:`Broker::BackendOptions` :bro:attr:`&default` = ``[sqlite=[path=], rocksdb=[path=]]`` :bro:attr:`&optional`
+         Parameters used for configuring the backend.
+
+      clone_resync_interval: :bro:type:`interval` :bro:attr:`&default` = :bro:see:`Broker::default_clone_resync_interval` :bro:attr:`&optional`
+         A resync/reconnect interval to pass through to
+         :bro:see:`Broker::create_clone`.
+
+      clone_stale_interval: :bro:type:`interval` :bro:attr:`&default` = :bro:see:`Broker::default_clone_stale_interval` :bro:attr:`&optional`
+         A staleness duration to pass through to
+         :bro:see:`Broker::create_clone`.
+
+      clone_mutation_buffer_interval: :bro:type:`interval` :bro:attr:`&default` = :bro:see:`Broker::default_clone_mutation_buffer_interval` :bro:attr:`&optional`
+         A mutation buffer interval to pass through to
+         :bro:see:`Broker::create_clone`.
+
+   Information regarding a cluster-enabled data store.
+
+Events
+######
+.. bro:id:: Cluster::hello
+
+   :Type: :bro:type:`event` (name: :bro:type:`string`, id: :bro:type:`string`)
+
+   When using broker-enabled cluster framework, nodes broadcast this event
+   to exchange their user-defined name along with a string that uniquely
+   identifies it for the duration of its lifetime.  This string may change
+   if the node dies and has to reconnect later.
+
+.. bro:id:: Cluster::node_down
+
+   :Type: :bro:type:`event` (name: :bro:type:`string`, id: :bro:type:`string`)
+
+   When using broker-enabled cluster framework, this event will be emitted
+   locally whenever a connected cluster node becomes disconnected.
+
+.. bro:id:: Cluster::node_up
+
+   :Type: :bro:type:`event` (name: :bro:type:`string`, id: :bro:type:`string`)
+
+   When using broker-enabled cluster framework, this event will be emitted
+   locally whenever a cluster node connects or reconnects.
+
+Functions
+#########
+.. bro:id:: Cluster::create_store
+
+   :Type: :bro:type:`function` (name: :bro:type:`string`, persistent: :bro:type:`bool` :bro:attr:`&default` = ``F`` :bro:attr:`&optional`) : :bro:type:`Cluster::StoreInfo`
+
+   Sets up a cluster-enabled data store.  They will also still properly
+   function for uses that are not operating a cluster.
+   
+
+   :name: the name of the data store to create.
+   
+
+   :persistent: whether the data store must be persistent.
+   
+
+   :returns: the store's information.  For master stores, the store will be
+            ready to use immediately.  For clones, the store field will not
+            be set until the node containing the master store has connected.
+
+.. bro:id:: Cluster::is_enabled
+
+   :Type: :bro:type:`function` () : :bro:type:`bool`
+
+   This function can be called at any time to determine if the cluster
+   framework is being enabled for this run.
+   
+
+   :returns: True if :bro:id:`Cluster::node` has been set.
+
+.. bro:id:: Cluster::local_node_type
+
+   :Type: :bro:type:`function` () : :bro:type:`Cluster::NodeType`
+
+   This function can be called at any time to determine what type of
+   cluster node the current Bro instance is going to be acting as.
+   If :bro:id:`Cluster::is_enabled` returns false, then
+   :bro:enum:`Cluster::NONE` is returned.
+   
+
+   :returns: The :bro:type:`Cluster::NodeType` the calling node acts as.
+
+.. bro:id:: Cluster::log
+
+   :Type: :bro:type:`function` (msg: :bro:type:`string`) : :bro:type:`void`
+
+   Write a message to the cluster logging stream.
+
+.. bro:id:: Cluster::node_topic
+
+   :Type: :bro:type:`function` (name: :bro:type:`string`) : :bro:type:`string`
+
+   Retrieve the topic associated with a specific node in the cluster.
+   
+
+   :name: the name of the cluster node (e.g. "manager").
+   
+
+   :returns: a topic string that may used to send a message exclusively to
+            a given cluster node.
+
+.. bro:id:: Cluster::nodeid_topic
+
+   :Type: :bro:type:`function` (id: :bro:type:`string`) : :bro:type:`string`
+
+   Retrieve the topic associated with a specific node in the cluster.
+   
+
+   :id: the id of the cluster node (from :bro:see:`Broker::EndpointInfo`
+       or :bro:see:`Broker::node_id`.
+   
+
+   :returns: a topic string that may used to send a message exclusively to
+            a given cluster node.
+
+

doc/scripts/base/frameworks/cluster/pools.bro.rst

@@ -0,0 +1,291 @@
+:tocdepth: 3
+
+base/frameworks/cluster/pools.bro
+=================================
+.. bro:namespace:: Cluster
+
+Defines an interface for managing pools of cluster nodes.  Pools are
+a useful way to distribute work or data among nodes within a cluster.
+
+:Namespace: Cluster
+:Imports: :doc:`base/frameworks/cluster/main.bro </scripts/base/frameworks/cluster/main.bro>`, :doc:`base/utils/hash_hrw.bro </scripts/base/utils/hash_hrw.bro>`
+
+Summary
+~~~~~~~
+State Variables
+###############
+===================================================================================== ======================================================
+:bro:id:`Cluster::logger_pool`: :bro:type:`Cluster::Pool`                             A pool containing all the logger nodes of a cluster.
+:bro:id:`Cluster::logger_pool_spec`: :bro:type:`Cluster::PoolSpec` :bro:attr:`&redef` The specification for :bro:see:`Cluster::logger_pool`.
+:bro:id:`Cluster::proxy_pool`: :bro:type:`Cluster::Pool`                              A pool containing all the proxy nodes of a cluster.
+:bro:id:`Cluster::proxy_pool_spec`: :bro:type:`Cluster::PoolSpec` :bro:attr:`&redef`  The specification for :bro:see:`Cluster::proxy_pool`.
+:bro:id:`Cluster::worker_pool`: :bro:type:`Cluster::Pool`                             A pool containing all the worker nodes of a cluster.
+:bro:id:`Cluster::worker_pool_spec`: :bro:type:`Cluster::PoolSpec` :bro:attr:`&redef` The specification for :bro:see:`Cluster::worker_pool`.
+===================================================================================== ======================================================
+
+Types
+#####
+======================================================= ===========================================================
+:bro:type:`Cluster::PoolNode`: :bro:type:`record`       Store state of a cluster within the context of a work pool.
+:bro:type:`Cluster::PoolNodeTable`: :bro:type:`table`   
+:bro:type:`Cluster::PoolSpec`: :bro:type:`record`       A pool specification.
+:bro:type:`Cluster::RoundRobinTable`: :bro:type:`table` 
+======================================================= ===========================================================
+
+Functions
+#########
+====================================================== ======================================================================
+:bro:id:`Cluster::hrw_topic`: :bro:type:`function`     Retrieve the topic associated with the node mapped via Rendezvous hash
+                                                       of an arbitrary key.
+:bro:id:`Cluster::register_pool`: :bro:type:`function` Registers and initializes a pool.
+:bro:id:`Cluster::rr_log_topic`: :bro:type:`function`  Distributes log message topics among logger nodes via round-robin.
+:bro:id:`Cluster::rr_topic`: :bro:type:`function`      Retrieve the topic associated with the node in a round-robin fashion.
+====================================================== ======================================================================
+
+
+Detailed Interface
+~~~~~~~~~~~~~~~~~~
+State Variables
+###############
+.. bro:id:: Cluster::logger_pool
+
+   :Type: :bro:type:`Cluster::Pool`
+   :Default:
+
+   ::
+
+      {
+         spec=[topic="", node_type=Cluster::PROXY, max_nodes=<uninitialized>, exclusive=F]
+         nodes={
+
+         }
+         node_list=[]
+         hrw_pool=[sites={
+
+         }]
+         rr_key_seq={
+
+         }
+         alive_count=0
+      }
+
+   A pool containing all the logger nodes of a cluster.
+   The pool's node membership/availability is automatically
+   maintained by the cluster framework.
+
+.. bro:id:: Cluster::logger_pool_spec
+
+   :Type: :bro:type:`Cluster::PoolSpec`
+   :Attributes: :bro:attr:`&redef`
+   :Default:
+
+   ::
+
+      {
+         topic="bro/cluster/pool/logger"
+         node_type=Cluster::LOGGER
+         max_nodes=<uninitialized>
+         exclusive=F
+      }
+
+   The specification for :bro:see:`Cluster::logger_pool`.
+
+.. bro:id:: Cluster::proxy_pool
+
+   :Type: :bro:type:`Cluster::Pool`
+   :Default:
+
+   ::
+
+      {
+         spec=[topic="", node_type=Cluster::PROXY, max_nodes=<uninitialized>, exclusive=F]
+         nodes={
+
+         }
+         node_list=[]
+         hrw_pool=[sites={
+
+         }]
+         rr_key_seq={
+
+         }
+         alive_count=0
+      }
+
+   A pool containing all the proxy nodes of a cluster.
+   The pool's node membership/availability is automatically
+   maintained by the cluster framework.
+
+.. bro:id:: Cluster::proxy_pool_spec
+
+   :Type: :bro:type:`Cluster::PoolSpec`
+   :Attributes: :bro:attr:`&redef`
+   :Default:
+
+   ::
+
+      {
+         topic="bro/cluster/pool/proxy"
+         node_type=Cluster::PROXY
+         max_nodes=<uninitialized>
+         exclusive=F
+      }
+
+   The specification for :bro:see:`Cluster::proxy_pool`.
+
+.. bro:id:: Cluster::worker_pool
+
+   :Type: :bro:type:`Cluster::Pool`
+   :Default:
+
+   ::
+
+      {
+         spec=[topic="", node_type=Cluster::PROXY, max_nodes=<uninitialized>, exclusive=F]
+         nodes={
+
+         }
+         node_list=[]
+         hrw_pool=[sites={
+
+         }]
+         rr_key_seq={
+
+         }
+         alive_count=0
+      }
+
+   A pool containing all the worker nodes of a cluster.
+   The pool's node membership/availability is automatically
+   maintained by the cluster framework.
+
+.. bro:id:: Cluster::worker_pool_spec
+
+   :Type: :bro:type:`Cluster::PoolSpec`
+   :Attributes: :bro:attr:`&redef`
+   :Default:
+
+   ::
+
+      {
+         topic="bro/cluster/pool/worker"
+         node_type=Cluster::WORKER
+         max_nodes=<uninitialized>
+         exclusive=F
+      }
+
+   The specification for :bro:see:`Cluster::worker_pool`.
+
+Types
+#####
+.. bro:type:: Cluster::PoolNode
+
+   :Type: :bro:type:`record`
+
+      name: :bro:type:`string`
+         The node name (e.g. "manager").
+
+      alias: :bro:type:`string`
+         An alias of *name* used to prevent hashing collisions when creating
+         *site_id*.
+
+      site_id: :bro:type:`count`
+         A 32-bit unique identifier for the pool node, derived from name/alias.
+
+      alive: :bro:type:`bool` :bro:attr:`&default` = ``F`` :bro:attr:`&optional`
+         Whether the node is currently alive and can receive work.
+
+   Store state of a cluster within the context of a work pool.
+
+.. bro:type:: Cluster::PoolNodeTable
+
+   :Type: :bro:type:`table` [:bro:type:`string`] of :bro:type:`Cluster::PoolNode`
+
+
+.. bro:type:: Cluster::PoolSpec
+
+   :Type: :bro:type:`record`
+
+      topic: :bro:type:`string` :bro:attr:`&default` = ``""`` :bro:attr:`&optional`
+         A topic string that can be used to reach all nodes within a pool.
+
+      node_type: :bro:type:`Cluster::NodeType` :bro:attr:`&default` = ``Cluster::PROXY`` :bro:attr:`&optional`
+         The type of nodes that are contained within the pool.
+
+      max_nodes: :bro:type:`count` :bro:attr:`&optional`
+         The maximum number of nodes that may belong to the pool.
+         If not set, then all available nodes will be added to the pool,
+         else the cluster framework will automatically limit the pool
+         membership according to the threshhold.
+
+      exclusive: :bro:type:`bool` :bro:attr:`&default` = ``F`` :bro:attr:`&optional`
+         Whether the pool requires exclusive access to nodes.  If true,
+         then *max_nodes* nodes will not be assigned to any other pool.
+         When using this flag, *max_nodes* must also be set.
+
+   A pool specification.
+
+.. bro:type:: Cluster::RoundRobinTable
+
+   :Type: :bro:type:`table` [:bro:type:`string`] of :bro:type:`int`
+
+
+Functions
+#########
+.. bro:id:: Cluster::hrw_topic
+
+   :Type: :bro:type:`function` (pool: :bro:type:`Cluster::Pool`, key: :bro:type:`any`) : :bro:type:`string`
+
+   Retrieve the topic associated with the node mapped via Rendezvous hash
+   of an arbitrary key.
+   
+
+   :pool: the pool of nodes to consider.
+   
+
+   :key: data used for input to the hashing function that will uniformly
+        distribute keys among available nodes.
+   
+
+   :returns: a topic string associated with a cluster node that is alive
+            or an empty string if nothing is alive.
+
+.. bro:id:: Cluster::register_pool
+
+   :Type: :bro:type:`function` (spec: :bro:type:`Cluster::PoolSpec`) : :bro:type:`Cluster::Pool`
+
+   Registers and initializes a pool.
+
+.. bro:id:: Cluster::rr_log_topic
+
+   :Type: :bro:type:`function` (id: :bro:type:`Log::ID`, path: :bro:type:`string`) : :bro:type:`string`
+
+   Distributes log message topics among logger nodes via round-robin.
+   This will be automatically assigned to :bro:see:`Broker::log_topic`
+   if :bro:see:`Cluster::enable_round_robin_logging` is enabled.
+   If no logger nodes are active, then this will return the value
+   of :bro:see:`Broker::default_log_topic`.
+
+.. bro:id:: Cluster::rr_topic
+
+   :Type: :bro:type:`function` (pool: :bro:type:`Cluster::Pool`, key: :bro:type:`string` :bro:attr:`&default` = ``""`` :bro:attr:`&optional`) : :bro:type:`string`
+
+   Retrieve the topic associated with the node in a round-robin fashion.
+   
+
+   :pool: the pool of nodes to consider.
+   
+
+   :key: an arbitrary string to identify the purpose for which you're
+        requesting the topic.  e.g. consider using a name-spaced key
+        like "Intel::cluster_rr_key" if you need to guarantee that
+        a group of messages get distributed in a well-defined pattern
+        without other messages being interleaved within the round-robin.
+        Usually sharing the default key is fine for load-balancing
+        purposes.
+   
+
+   :returns: a topic string associated with a cluster node that is alive,
+            or an empty string if nothing is alive.
+
+

doc/scripts/base/frameworks/config/__load__.bro.rst

@@ -0,0 +1,14 @@
+:tocdepth: 3
+
+base/frameworks/config/__load__.bro
+===================================
+
+
+:Imports: :doc:`base/frameworks/config/input.bro </scripts/base/frameworks/config/input.bro>`, :doc:`base/frameworks/config/main.bro </scripts/base/frameworks/config/main.bro>`, :doc:`base/frameworks/config/weird.bro </scripts/base/frameworks/config/weird.bro>`
+
+Summary
+~~~~~~~
+
+Detailed Interface
+~~~~~~~~~~~~~~~~~~
+

doc/scripts/base/frameworks/config/index.rst

@@ -0,0 +1,25 @@
+:orphan:
+
+Package: base/frameworks/config
+===============================
+
+The configuration framework provides a way to change the Bro configuration
+in "option" values at run-time.
+
+:doc:`/scripts/base/frameworks/config/__load__.bro`
+
+
+:doc:`/scripts/base/frameworks/config/main.bro`
+
+   The configuration framework provides a way to change Bro options
+   (as specified by the "option" keyword) at runtime. It also logs runtime
+   changes to options to config.log.
+
+:doc:`/scripts/base/frameworks/config/input.bro`
+
+   File input for the configuration framework using the input framework.
+
+:doc:`/scripts/base/frameworks/config/weird.bro`
+
+   This script sets up the config framework change handlers for weirds.
+

doc/scripts/base/frameworks/config/input.bro.rst

@@ -0,0 +1,54 @@
+:tocdepth: 3
+
+base/frameworks/config/input.bro
+================================
+.. bro:namespace:: Config
+
+File input for the configuration framework using the input framework.
+
+:Namespace: Config
+:Imports: :doc:`base/frameworks/cluster </scripts/base/frameworks/cluster/index>`, :doc:`base/frameworks/config/main.bro </scripts/base/frameworks/config/main.bro>`
+
+Summary
+~~~~~~~
+Redefinable Options
+###################
+================================================================== ===============================================
+:bro:id:`Config::config_files`: :bro:type:`set` :bro:attr:`&redef` Configuration files that will be read off disk.
+================================================================== ===============================================
+
+Functions
+#########
+=================================================== ===================================================================
+:bro:id:`Config::read_config`: :bro:type:`function` Read specified configuration file and apply values; updates to file
+                                                    are not tracked.
+=================================================== ===================================================================
+
+
+Detailed Interface
+~~~~~~~~~~~~~~~~~~
+Redefinable Options
+###################
+.. bro:id:: Config::config_files
+
+   :Type: :bro:type:`set` [:bro:type:`string`]
+   :Attributes: :bro:attr:`&redef`
+   :Default: ``{}``
+
+   Configuration files that will be read off disk. Files are reread
+   every time they are updated so updates should be atomic with "mv"
+   instead of writing the file in place.
+   
+   If the same configuration option is defined in several files with
+   different values, behavior is unspecified.
+
+Functions
+#########
+.. bro:id:: Config::read_config
+
+   :Type: :bro:type:`function` (filename: :bro:type:`string`) : :bro:type:`void`
+
+   Read specified configuration file and apply values; updates to file
+   are not tracked.
+
+

doc/scripts/base/frameworks/config/main.bro.rst

@@ -0,0 +1,101 @@
+:tocdepth: 3
+
+base/frameworks/config/main.bro
+===============================
+.. bro:namespace:: Config
+
+The configuration framework provides a way to change Bro options
+(as specified by the "option" keyword) at runtime. It also logs runtime
+changes to options to config.log.
+
+:Namespace: Config
+:Imports: :doc:`base/frameworks/cluster </scripts/base/frameworks/cluster/index>`
+
+Summary
+~~~~~~~
+Types
+#####
+============================================ ==================================
+:bro:type:`Config::Info`: :bro:type:`record` Represents the data in config.log.
+============================================ ==================================
+
+Redefinitions
+#############
+===================================== =====================================
+:bro:type:`Log::ID`: :bro:type:`enum` The config logging stream identifier.
+===================================== =====================================
+
+Events
+######
+=============================================== ================================================================
+:bro:id:`Config::log_config`: :bro:type:`event` Event that can be handled to access the :bro:type:`Config::Info`
+                                                record as it is sent on to the logging framework.
+=============================================== ================================================================
+
+Functions
+#########
+================================================= ==================================================================
+:bro:id:`Config::set_value`: :bro:type:`function` This function is the config framework layer around the lower-level
+                                                  :bro:see:`Option::set` call.
+================================================= ==================================================================
+
+
+Detailed Interface
+~~~~~~~~~~~~~~~~~~
+Types
+#####
+.. bro:type:: Config::Info
+
+   :Type: :bro:type:`record`
+
+      ts: :bro:type:`time` :bro:attr:`&log`
+         Timestamp at which the configuration change occured.
+
+      id: :bro:type:`string` :bro:attr:`&log`
+         ID of the value that was changed.
+
+      old_value: :bro:type:`string` :bro:attr:`&log`
+         Value before the change.
+
+      new_value: :bro:type:`string` :bro:attr:`&log`
+         Value after the change.
+
+      location: :bro:type:`string` :bro:attr:`&optional` :bro:attr:`&log`
+         Optional location that triggered the change.
+
+   Represents the data in config.log.
+
+Events
+######
+.. bro:id:: Config::log_config
+
+   :Type: :bro:type:`event` (rec: :bro:type:`Config::Info`)
+
+   Event that can be handled to access the :bro:type:`Config::Info`
+   record as it is sent on to the logging framework.
+
+Functions
+#########
+.. bro:id:: Config::set_value
+
+   :Type: :bro:type:`function` (ID: :bro:type:`string`, val: :bro:type:`any`, location: :bro:type:`string` :bro:attr:`&default` = ``""`` :bro:attr:`&optional` :bro:attr:`&optional`) : :bro:type:`bool`
+
+   This function is the config framework layer around the lower-level
+   :bro:see:`Option::set` call. Config::set_value will set the configuration
+   value for all nodes in the cluster, no matter where it was called. Note
+   that :bro:see:`Option::set` does not distribute configuration changes
+   to other nodes.
+   
+
+   :ID: The ID of the option to update.
+   
+
+   :val: The new value of the option.
+   
+
+   :location: Optional parameter detailing where this change originated from.
+   
+
+   :returns: true on success, false when an error occurs.
+
+

doc/scripts/base/frameworks/config/weird.bro.rst

@@ -0,0 +1,17 @@
+:tocdepth: 3
+
+base/frameworks/config/weird.bro
+================================
+.. bro:namespace:: Config
+
+This script sets up the config framework change handlers for weirds.
+
+:Namespace: Config
+:Imports: :doc:`base/frameworks/config/main.bro </scripts/base/frameworks/config/main.bro>`
+
+Summary
+~~~~~~~
+
+Detailed Interface
+~~~~~~~~~~~~~~~~~~
+

doc/scripts/base/frameworks/control/__load__.bro.rst

@@ -0,0 +1,14 @@
+:tocdepth: 3
+
+base/frameworks/control/__load__.bro
+====================================
+
+
+:Imports: :doc:`base/frameworks/control/main.bro </scripts/base/frameworks/control/main.bro>`
+
+Summary
+~~~~~~~
+
+Detailed Interface
+~~~~~~~~~~~~~~~~~~
+

doc/scripts/base/frameworks/control/index.rst

@@ -0,0 +1,18 @@
+:orphan:
+
+Package: base/frameworks/control
+================================
+
+The control framework provides the foundation for providing "commands"
+that can be taken remotely at runtime to modify a running Bro instance
+or collect information from the running instance.
+
+:doc:`/scripts/base/frameworks/control/__load__.bro`
+
+
+:doc:`/scripts/base/frameworks/control/main.bro`
+
+   The control framework provides the foundation for providing "commands"
+   that can be taken remotely at runtime to modify a running Bro instance
+   or collect information from the running instance.
+

doc/scripts/base/frameworks/control/main.bro.rst

@@ -0,0 +1,218 @@
+:tocdepth: 3
+
+base/frameworks/control/main.bro
+================================
+.. bro:namespace:: Control
+
+The control framework provides the foundation for providing "commands"
+that can be taken remotely at runtime to modify a running Bro instance
+or collect information from the running instance.
+
+:Namespace: Control
+
+Summary
+~~~~~~~
+Redefinable Options
+###################
+========================================================================= ================================================================
+:bro:id:`Control::arg`: :bro:type:`string` :bro:attr:`&redef`             This can be used by commands that take an argument.
+:bro:id:`Control::cmd`: :bro:type:`string` :bro:attr:`&redef`             The command that is being done.
+:bro:id:`Control::commands`: :bro:type:`set` :bro:attr:`&redef`           The commands that can currently be given on the command line for
+                                                                          remote control.
+:bro:id:`Control::controllee_listen`: :bro:type:`bool` :bro:attr:`&redef` Whether the controllee should call :bro:see:`Broker::listen`.
+:bro:id:`Control::host`: :bro:type:`addr` :bro:attr:`&redef`              The address of the host that will be controlled.
+:bro:id:`Control::host_port`: :bro:type:`port` :bro:attr:`&redef`         The port of the host that will be controlled.
+:bro:id:`Control::zone_id`: :bro:type:`string` :bro:attr:`&redef`         If :bro:id:`Control::host` is a non-global IPv6 address and
+                                                                          requires a specific :rfc:`4007` ``zone_id``, it can be set here.
+========================================================================= ================================================================
+
+Constants
+#########
+=================================================== =================================================================
+:bro:id:`Control::ignore_ids`: :bro:type:`set`      Variable IDs that are to be ignored by the update process.
+:bro:id:`Control::topic_prefix`: :bro:type:`string` The topic prefix used for exchanging control messages via Broker.
+=================================================== =================================================================
+
+Events
+######
+=================================================================== ====================================================================
+:bro:id:`Control::configuration_update`: :bro:type:`event`          This event is a wrapper and alias for the
+                                                                    :bro:id:`Control::configuration_update_request` event.
+:bro:id:`Control::configuration_update_request`: :bro:type:`event`  Inform the remote Bro instance that it's configuration may have been
+                                                                    updated.
+:bro:id:`Control::configuration_update_response`: :bro:type:`event` Message in response to a configuration update request.
+:bro:id:`Control::id_value_request`: :bro:type:`event`              Event for requesting the value of an ID (a variable).
+:bro:id:`Control::id_value_response`: :bro:type:`event`             Event for returning the value of an ID after an
+                                                                    :bro:id:`Control::id_value_request` event.
+:bro:id:`Control::net_stats_request`: :bro:type:`event`             Requests the current net_stats.
+:bro:id:`Control::net_stats_response`: :bro:type:`event`            Returns the current net_stats.
+:bro:id:`Control::peer_status_request`: :bro:type:`event`           Requests the current communication status.
+:bro:id:`Control::peer_status_response`: :bro:type:`event`          Returns the current communication status.
+:bro:id:`Control::shutdown_request`: :bro:type:`event`              Requests that the Bro instance begins shutting down.
+:bro:id:`Control::shutdown_response`: :bro:type:`event`             Message in response to a shutdown request.
+=================================================================== ====================================================================
+
+
+Detailed Interface
+~~~~~~~~~~~~~~~~~~
+Redefinable Options
+###################
+.. bro:id:: Control::arg
+
+   :Type: :bro:type:`string`
+   :Attributes: :bro:attr:`&redef`
+   :Default: ``""``
+
+   This can be used by commands that take an argument.
+
+.. bro:id:: Control::cmd
+
+   :Type: :bro:type:`string`
+   :Attributes: :bro:attr:`&redef`
+   :Default: ``""``
+
+   The command that is being done.  It's typically set on the
+   command line.
+
+.. bro:id:: Control::commands
+
+   :Type: :bro:type:`set` [:bro:type:`string`]
+   :Attributes: :bro:attr:`&redef`
+   :Default:
+
+   ::
+
+      {
+         "shutdown",
+         "id_value",
+         "net_stats",
+         "peer_status",
+         "configuration_update"
+      }
+
+   The commands that can currently be given on the command line for
+   remote control.
+
+.. bro:id:: Control::controllee_listen
+
+   :Type: :bro:type:`bool`
+   :Attributes: :bro:attr:`&redef`
+   :Default: ``T``
+
+   Whether the controllee should call :bro:see:`Broker::listen`.
+   In a cluster, this isn't needed since the setup process calls it.
+
+.. bro:id:: Control::host
+
+   :Type: :bro:type:`addr`
+   :Attributes: :bro:attr:`&redef`
+   :Default: ``0.0.0.0``
+
+   The address of the host that will be controlled.
+
+.. bro:id:: Control::host_port
+
+   :Type: :bro:type:`port`
+   :Attributes: :bro:attr:`&redef`
+   :Default: ``0/tcp``
+
+   The port of the host that will be controlled.
+
+.. bro:id:: Control::zone_id
+
+   :Type: :bro:type:`string`
+   :Attributes: :bro:attr:`&redef`
+   :Default: ``""``
+
+   If :bro:id:`Control::host` is a non-global IPv6 address and
+   requires a specific :rfc:`4007` ``zone_id``, it can be set here.
+
+Constants
+#########
+.. bro:id:: Control::ignore_ids
+
+   :Type: :bro:type:`set` [:bro:type:`string`]
+   :Default: ``{}``
+
+   Variable IDs that are to be ignored by the update process.
+
+.. bro:id:: Control::topic_prefix
+
+   :Type: :bro:type:`string`
+   :Default: ``"bro/control"``
+
+   The topic prefix used for exchanging control messages via Broker.
+
+Events
+######
+.. bro:id:: Control::configuration_update
+
+   :Type: :bro:type:`event` ()
+
+   This event is a wrapper and alias for the
+   :bro:id:`Control::configuration_update_request` event.
+   This event is also a primary hooking point for the control framework.
+
+.. bro:id:: Control::configuration_update_request
+
+   :Type: :bro:type:`event` ()
+
+   Inform the remote Bro instance that it's configuration may have been
+   updated.
+
+.. bro:id:: Control::configuration_update_response
+
+   :Type: :bro:type:`event` ()
+
+   Message in response to a configuration update request.
+
+.. bro:id:: Control::id_value_request
+
+   :Type: :bro:type:`event` (id: :bro:type:`string`)
+
+   Event for requesting the value of an ID (a variable).
+
+.. bro:id:: Control::id_value_response
+
+   :Type: :bro:type:`event` (id: :bro:type:`string`, val: :bro:type:`string`)
+
+   Event for returning the value of an ID after an
+   :bro:id:`Control::id_value_request` event.
+
+.. bro:id:: Control::net_stats_request
+
+   :Type: :bro:type:`event` ()
+
+   Requests the current net_stats.
+
+.. bro:id:: Control::net_stats_response
+
+   :Type: :bro:type:`event` (s: :bro:type:`string`)
+
+   Returns the current net_stats.
+
+.. bro:id:: Control::peer_status_request
+
+   :Type: :bro:type:`event` ()
+
+   Requests the current communication status.
+
+.. bro:id:: Control::peer_status_response
+
+   :Type: :bro:type:`event` (s: :bro:type:`string`)
+
+   Returns the current communication status.
+
+.. bro:id:: Control::shutdown_request
+
+   :Type: :bro:type:`event` ()
+
+   Requests that the Bro instance begins shutting down.
+
+.. bro:id:: Control::shutdown_response
+
+   :Type: :bro:type:`event` ()
+
+   Message in response to a shutdown request.
+
+

doc/scripts/base/frameworks/dpd/__load__.bro.rst

@@ -0,0 +1,14 @@
+:tocdepth: 3
+
+base/frameworks/dpd/__load__.bro
+================================
+
+
+:Imports: :doc:`base/frameworks/dpd/main.bro </scripts/base/frameworks/dpd/main.bro>`
+
+Summary
+~~~~~~~
+
+Detailed Interface
+~~~~~~~~~~~~~~~~~~
+

doc/scripts/base/frameworks/dpd/index.rst

@@ -0,0 +1,16 @@
+:orphan:
+
+Package: base/frameworks/dpd
+============================
+
+The DPD (dynamic protocol detection) activates port-independent protocol
+detection and selectively disables analyzers if protocol violations occur.
+
+:doc:`/scripts/base/frameworks/dpd/__load__.bro`
+
+
+:doc:`/scripts/base/frameworks/dpd/main.bro`
+
+   Activates port-independent protocol detection and selectively disables
+   analyzers if protocol violations occur.
+

doc/scripts/base/frameworks/dpd/main.bro.rst

@@ -0,0 +1,99 @@
+:tocdepth: 3
+
+base/frameworks/dpd/main.bro
+============================
+.. bro:namespace:: DPD
+
+Activates port-independent protocol detection and selectively disables
+analyzers if protocol violations occur.
+
+:Namespace: DPD
+
+Summary
+~~~~~~~
+Runtime Options
+###############
+============================================================================ ===============================================================
+:bro:id:`DPD::ignore_violations`: :bro:type:`set` :bro:attr:`&redef`         Analyzers which you don't want to throw 
+:bro:id:`DPD::ignore_violations_after`: :bro:type:`count` :bro:attr:`&redef` Ignore violations which go this many bytes into the connection.
+============================================================================ ===============================================================
+
+Types
+#####
+========================================= ======================================================================
+:bro:type:`DPD::Info`: :bro:type:`record` The record type defining the columns to log in the DPD logging stream.
+========================================= ======================================================================
+
+Redefinitions
+#############
+========================================== ======================================
+:bro:type:`Log::ID`: :bro:type:`enum`      Add the DPD logging stream identifier.
+:bro:type:`connection`: :bro:type:`record` 
+========================================== ======================================
+
+
+Detailed Interface
+~~~~~~~~~~~~~~~~~~
+Runtime Options
+###############
+.. bro:id:: DPD::ignore_violations
+
+   :Type: :bro:type:`set` [:bro:type:`Analyzer::Tag`]
+   :Attributes: :bro:attr:`&redef`
+   :Default:
+
+   ::
+
+      {
+         Analyzer::ANALYZER_DCE_RPC,
+         Analyzer::ANALYZER_NTLM
+      }
+
+   Analyzers which you don't want to throw 
+
+.. bro:id:: DPD::ignore_violations_after
+
+   :Type: :bro:type:`count`
+   :Attributes: :bro:attr:`&redef`
+   :Default: ``10240``
+
+   Ignore violations which go this many bytes into the connection.
+   Set to 0 to never ignore protocol violations.
+
+Types
+#####
+.. bro:type:: DPD::Info
+
+   :Type: :bro:type:`record`
+
+      ts: :bro:type:`time` :bro:attr:`&log`
+         Timestamp for when protocol analysis failed.
+
+      uid: :bro:type:`string` :bro:attr:`&log`
+         Connection unique ID.
+
+      id: :bro:type:`conn_id` :bro:attr:`&log`
+         Connection ID containing the 4-tuple which identifies endpoints.
+
+      proto: :bro:type:`transport_proto` :bro:attr:`&log`
+         Transport protocol for the violation.
+
+      analyzer: :bro:type:`string` :bro:attr:`&log`
+         The analyzer that generated the violation.
+
+      failure_reason: :bro:type:`string` :bro:attr:`&log`
+         The textual reason for the analysis failure.
+
+      disabled_aids: :bro:type:`set` [:bro:type:`count`]
+         Disabled analyzer IDs.  This is only for internal tracking
+         so as to not attempt to disable analyzers multiple times.
+
+      packet_segment: :bro:type:`string` :bro:attr:`&optional` :bro:attr:`&log`
+         (present if :doc:`/scripts/policy/frameworks/dpd/packet-segment-logging.bro` is loaded)
+
+         A chunk of the payload that most likely resulted in the
+         protocol violation.
+
+   The record type defining the columns to log in the DPD logging stream.
+
+

doc/scripts/base/frameworks/files/__load__.bro.rst

@@ -0,0 +1,14 @@
+:tocdepth: 3
+
+base/frameworks/files/__load__.bro
+==================================
+
+
+:Imports: :doc:`base/frameworks/files/magic </scripts/base/frameworks/files/magic/index>`, :doc:`base/frameworks/files/main.bro </scripts/base/frameworks/files/main.bro>`
+
+Summary
+~~~~~~~
+
+Detailed Interface
+~~~~~~~~~~~~~~~~~~
+

doc/scripts/base/frameworks/files/index.rst

@@ -0,0 +1,20 @@
+:orphan:
+
+Package: base/frameworks/files
+==============================
+
+The file analysis framework provides an interface for driving the analysis
+of files, possibly independent of any network protocol over which they're
+transported.
+
+:doc:`/scripts/base/frameworks/files/__load__.bro`
+
+
+:doc:`/scripts/base/frameworks/files/main.bro`
+
+   An interface for driving the analysis of files, possibly independent of
+   any network protocol over which they're transported.
+
+:doc:`/scripts/base/frameworks/files/magic/__load__.bro`
+
+

doc/scripts/base/frameworks/files/magic/__load__.bro.rst

@@ -0,0 +1,13 @@
+:tocdepth: 3
+
+base/frameworks/files/magic/__load__.bro
+========================================
+
+
+
+Summary
+~~~~~~~
+
+Detailed Interface
+~~~~~~~~~~~~~~~~~~
+

doc/scripts/base/frameworks/files/magic/index.rst

@@ -0,0 +1,9 @@
+:orphan:
+
+Package: base/frameworks/files/magic
+====================================
+
+
+:doc:`/scripts/base/frameworks/files/magic/__load__.bro`
+
+

doc/scripts/base/frameworks/files/main.bro.rst

@@ -0,0 +1,580 @@
+:tocdepth: 3
+
+base/frameworks/files/main.bro
+==============================
+.. bro:namespace:: Files
+
+An interface for driving the analysis of files, possibly independent of
+any network protocol over which they're transported.
+
+:Namespace: Files
+:Imports: :doc:`base/bif/file_analysis.bif.bro </scripts/base/bif/file_analysis.bif.bro>`, :doc:`base/frameworks/analyzer </scripts/base/frameworks/analyzer/index>`, :doc:`base/frameworks/logging </scripts/base/frameworks/logging/index>`, :doc:`base/utils/site.bro </scripts/base/utils/site.bro>`
+
+Summary
+~~~~~~~
+Runtime Options
+###############
+======================================================================== ========================================
+:bro:id:`Files::enable_reassembler`: :bro:type:`bool` :bro:attr:`&redef` The default setting for file reassembly.
+======================================================================== ========================================
+
+Redefinable Options
+###################
+======================================================================================== ================================================================
+:bro:id:`Files::analyze_by_mime_type_automatically`: :bro:type:`bool` :bro:attr:`&redef` Decide if you want to automatically attached analyzers to 
+                                                                                         files based on the detected mime type of the file.
+:bro:id:`Files::disable`: :bro:type:`table` :bro:attr:`&redef`                           A table that can be used to disable file analysis completely for
+                                                                                         any files transferred over given network protocol analyzers.
+:bro:id:`Files::reassembly_buffer_size`: :bro:type:`count` :bro:attr:`&redef`            The default per-file reassembly buffer size.
+:bro:id:`Files::salt`: :bro:type:`string` :bro:attr:`&redef`                             The salt concatenated to unique file handle strings generated by
+                                                                                         :bro:see:`get_file_handle` before hashing them in to a file id
+                                                                                         (the *id* field of :bro:see:`fa_file`).
+======================================================================================== ================================================================
+
+Types
+#####
+====================================================================== ==============================================================
+:bro:type:`Files::AnalyzerArgs`: :bro:type:`record` :bro:attr:`&redef` A structure which parameterizes a type of file analysis.
+:bro:type:`Files::Info`: :bro:type:`record` :bro:attr:`&redef`         Contains all metadata related to the analysis of a given file.
+:bro:type:`Files::ProtoRegistration`: :bro:type:`record`               
+====================================================================== ==============================================================
+
+Redefinitions
+#############
+========================================================== =
+:bro:type:`Log::ID`: :bro:type:`enum`                      
+:bro:type:`fa_file`: :bro:type:`record` :bro:attr:`&redef` 
+========================================================== =
+
+Events
+######
+============================================= ====================================================================
+:bro:id:`Files::log_files`: :bro:type:`event` Event that can be handled to access the Info record as it is sent on
+                                              to the logging framework.
+============================================= ====================================================================
+
+Functions
+#########
+===================================================================== =============================================================================
+:bro:id:`Files::add_analyzer`: :bro:type:`function`                   Adds an analyzer to the analysis of a given file.
+:bro:id:`Files::all_registered_mime_types`: :bro:type:`function`      Returns a table of all MIME-type-to-analyzer mappings currently registered.
+:bro:id:`Files::analyzer_name`: :bro:type:`function`                  Translates a file analyzer enum value to a string with the
+                                                                      analyzer's name.
+:bro:id:`Files::describe`: :bro:type:`function`                       Provides a text description regarding metadata of the file.
+:bro:id:`Files::disable_reassembly`: :bro:type:`function`             Disables the file reassembler on this file.
+:bro:id:`Files::enable_reassembly`: :bro:type:`function`              Allows the file reassembler to be used if it's necessary because the
+                                                                      file is transferred out of order.
+:bro:id:`Files::file_exists`: :bro:type:`function`                    Lookup to see if a particular file id exists and is still valid.
+:bro:id:`Files::lookup_file`: :bro:type:`function`                    Lookup an :bro:see:`fa_file` record with the file id.
+:bro:id:`Files::register_analyzer_add_callback`: :bro:type:`function` Register a callback for file analyzers to use if they need to do some
+                                                                      manipulation when they are being added to a file before the core code
+                                                                      takes over.
+:bro:id:`Files::register_for_mime_type`: :bro:type:`function`         Registers a MIME type for an analyzer.
+:bro:id:`Files::register_for_mime_types`: :bro:type:`function`        Registers a set of MIME types for an analyzer.
+:bro:id:`Files::register_protocol`: :bro:type:`function`              Register callbacks for protocols that work with the Files framework.
+:bro:id:`Files::registered_mime_types`: :bro:type:`function`          Returns a set of all MIME types currently registered for a specific analyzer.
+:bro:id:`Files::remove_analyzer`: :bro:type:`function`                Removes an analyzer from the analysis of a given file.
+:bro:id:`Files::set_reassembly_buffer_size`: :bro:type:`function`     Set the maximum size the reassembly buffer is allowed to grow
+                                                                      for the given file.
+:bro:id:`Files::set_timeout_interval`: :bro:type:`function`           Sets the *timeout_interval* field of :bro:see:`fa_file`, which is
+                                                                      used to determine the length of inactivity that is allowed for a file
+                                                                      before internal state related to it is cleaned up.
+:bro:id:`Files::stop`: :bro:type:`function`                           Stops/ignores any further analysis of a given file.
+===================================================================== =============================================================================
+
+
+Detailed Interface
+~~~~~~~~~~~~~~~~~~
+Runtime Options
+###############
+.. bro:id:: Files::enable_reassembler
+
+   :Type: :bro:type:`bool`
+   :Attributes: :bro:attr:`&redef`
+   :Default: ``T``
+
+   The default setting for file reassembly.
+
+Redefinable Options
+###################
+.. bro:id:: Files::analyze_by_mime_type_automatically
+
+   :Type: :bro:type:`bool`
+   :Attributes: :bro:attr:`&redef`
+   :Default: ``T``
+
+   Decide if you want to automatically attached analyzers to 
+   files based on the detected mime type of the file.
+
+.. bro:id:: Files::disable
+
+   :Type: :bro:type:`table` [:bro:type:`Files::Tag`] of :bro:type:`bool`
+   :Attributes: :bro:attr:`&redef`
+   :Default: ``{}``
+
+   A table that can be used to disable file analysis completely for
+   any files transferred over given network protocol analyzers.
+
+.. bro:id:: Files::reassembly_buffer_size
+
+   :Type: :bro:type:`count`
+   :Attributes: :bro:attr:`&redef`
+   :Default: ``524288``
+
+   The default per-file reassembly buffer size.
+
+.. bro:id:: Files::salt
+
+   :Type: :bro:type:`string`
+   :Attributes: :bro:attr:`&redef`
+   :Default: ``"I recommend changing this."``
+
+   The salt concatenated to unique file handle strings generated by
+   :bro:see:`get_file_handle` before hashing them in to a file id
+   (the *id* field of :bro:see:`fa_file`).
+   Provided to help mitigate the possibility of manipulating parts of
+   network connections that factor in to the file handle in order to
+   generate two handles that would hash to the same file id.
+
+Types
+#####
+.. bro:type:: Files::AnalyzerArgs
+
+   :Type: :bro:type:`record`
+
+      chunk_event: :bro:type:`event` (f: :bro:type:`fa_file`, data: :bro:type:`string`, off: :bro:type:`count`) :bro:attr:`&optional`
+         An event which will be generated for all new file contents,
+         chunk-wise.  Used when *tag* (in the
+         :bro:see:`Files::add_analyzer` function) is
+         :bro:see:`Files::ANALYZER_DATA_EVENT`.
+
+      stream_event: :bro:type:`event` (f: :bro:type:`fa_file`, data: :bro:type:`string`) :bro:attr:`&optional`
+         An event which will be generated for all new file contents,
+         stream-wise.  Used when *tag* is
+         :bro:see:`Files::ANALYZER_DATA_EVENT`.
+
+      extract_filename: :bro:type:`string` :bro:attr:`&optional`
+         (present if :doc:`/scripts/base/files/extract/main.bro` is loaded)
+
+         The local filename to which to write an extracted file.
+         This field is used in the core by the extraction plugin
+         to know where to write the file to.  If not specified, then
+         a filename in the format "extract-<source>-<id>" is
+         automatically assigned (using the *source* and *id*
+         fields of :bro:see:`fa_file`).
+
+      extract_limit: :bro:type:`count` :bro:attr:`&default` = :bro:see:`FileExtract::default_limit` :bro:attr:`&optional`
+         (present if :doc:`/scripts/base/files/extract/main.bro` is loaded)
+
+         The maximum allowed file size in bytes of *extract_filename*.
+         Once reached, a :bro:see:`file_extraction_limit` event is
+         raised and the analyzer will be removed unless
+         :bro:see:`FileExtract::set_limit` is called to increase the
+         limit.  A value of zero means "no limit".
+   :Attributes: :bro:attr:`&redef`
+
+   A structure which parameterizes a type of file analysis.
+
+.. bro:type:: Files::Info
+
+   :Type: :bro:type:`record`
+
+      ts: :bro:type:`time` :bro:attr:`&log`
+         The time when the file was first seen.
+
+      fuid: :bro:type:`string` :bro:attr:`&log`
+         An identifier associated with a single file.
+
+      tx_hosts: :bro:type:`set` [:bro:type:`addr`] :bro:attr:`&default` = ``{  }`` :bro:attr:`&optional` :bro:attr:`&log`
+         If this file was transferred over a network
+         connection this should show the host or hosts that
+         the data sourced from.
+
+      rx_hosts: :bro:type:`set` [:bro:type:`addr`] :bro:attr:`&default` = ``{  }`` :bro:attr:`&optional` :bro:attr:`&log`
+         If this file was transferred over a network
+         connection this should show the host or hosts that
+         the data traveled to.
+
+      conn_uids: :bro:type:`set` [:bro:type:`string`] :bro:attr:`&default` = ``{  }`` :bro:attr:`&optional` :bro:attr:`&log`
+         Connection UIDs over which the file was transferred.
+
+      source: :bro:type:`string` :bro:attr:`&log` :bro:attr:`&optional`
+         An identification of the source of the file data.  E.g. it
+         may be a network protocol over which it was transferred, or a
+         local file path which was read, or some other input source.
+
+      depth: :bro:type:`count` :bro:attr:`&default` = ``0`` :bro:attr:`&optional` :bro:attr:`&log`
+         A value to represent the depth of this file in relation
+         to its source.  In SMTP, it is the depth of the MIME
+         attachment on the message.  In HTTP, it is the depth of the
+         request within the TCP connection.
+
+      analyzers: :bro:type:`set` [:bro:type:`string`] :bro:attr:`&default` = ``{  }`` :bro:attr:`&optional` :bro:attr:`&log`
+         A set of analysis types done during the file analysis.
+
+      mime_type: :bro:type:`string` :bro:attr:`&log` :bro:attr:`&optional`
+         A mime type provided by the strongest file magic signature
+         match against the *bof_buffer* field of :bro:see:`fa_file`,
+         or in the cases where no buffering of the beginning of file
+         occurs, an initial guess of the mime type based on the first
+         data seen.
+
+      filename: :bro:type:`string` :bro:attr:`&log` :bro:attr:`&optional`
+         A filename for the file if one is available from the source
+         for the file.  These will frequently come from
+         "Content-Disposition" headers in network protocols.
+
+      duration: :bro:type:`interval` :bro:attr:`&log` :bro:attr:`&default` = ``0 secs`` :bro:attr:`&optional`
+         The duration the file was analyzed for.
+
+      local_orig: :bro:type:`bool` :bro:attr:`&log` :bro:attr:`&optional`
+         If the source of this file is a network connection, this field
+         indicates if the data originated from the local network or not as
+         determined by the configured :bro:see:`Site::local_nets`.
+
+      is_orig: :bro:type:`bool` :bro:attr:`&log` :bro:attr:`&optional`
+         If the source of this file is a network connection, this field
+         indicates if the file is being sent by the originator of the
+         connection or the responder.
+
+      seen_bytes: :bro:type:`count` :bro:attr:`&log` :bro:attr:`&default` = ``0`` :bro:attr:`&optional`
+         Number of bytes provided to the file analysis engine for the file.
+
+      total_bytes: :bro:type:`count` :bro:attr:`&log` :bro:attr:`&optional`
+         Total number of bytes that are supposed to comprise the full file.
+
+      missing_bytes: :bro:type:`count` :bro:attr:`&log` :bro:attr:`&default` = ``0`` :bro:attr:`&optional`
+         The number of bytes in the file stream that were completely missed
+         during the process of analysis e.g. due to dropped packets.
+
+      overflow_bytes: :bro:type:`count` :bro:attr:`&log` :bro:attr:`&default` = ``0`` :bro:attr:`&optional`
+         The number of bytes in the file stream that were not delivered to
+         stream file analyzers.  This could be overlapping bytes or 
+         bytes that couldn't be reassembled.
+
+      timedout: :bro:type:`bool` :bro:attr:`&log` :bro:attr:`&default` = ``F`` :bro:attr:`&optional`
+         Whether the file analysis timed out at least once for the file.
+
+      parent_fuid: :bro:type:`string` :bro:attr:`&log` :bro:attr:`&optional`
+         Identifier associated with a container file from which this one was
+         extracted as part of the file analysis.
+
+      md5: :bro:type:`string` :bro:attr:`&log` :bro:attr:`&optional`
+         (present if :doc:`/scripts/base/files/hash/main.bro` is loaded)
+
+         An MD5 digest of the file contents.
+
+      sha1: :bro:type:`string` :bro:attr:`&log` :bro:attr:`&optional`
+         (present if :doc:`/scripts/base/files/hash/main.bro` is loaded)
+
+         A SHA1 digest of the file contents.
+
+      sha256: :bro:type:`string` :bro:attr:`&log` :bro:attr:`&optional`
+         (present if :doc:`/scripts/base/files/hash/main.bro` is loaded)
+
+         A SHA256 digest of the file contents.
+
+      x509: :bro:type:`X509::Info` :bro:attr:`&optional`
+         (present if :doc:`/scripts/base/files/x509/main.bro` is loaded)
+
+         Information about X509 certificates. This is used to keep
+         certificate information until all events have been received.
+
+      extracted: :bro:type:`string` :bro:attr:`&optional` :bro:attr:`&log`
+         (present if :doc:`/scripts/base/files/extract/main.bro` is loaded)
+
+         Local filename of extracted file.
+
+      extracted_cutoff: :bro:type:`bool` :bro:attr:`&optional` :bro:attr:`&log`
+         (present if :doc:`/scripts/base/files/extract/main.bro` is loaded)
+
+         Set to true if the file being extracted was cut off
+         so the whole file was not logged.
+
+      extracted_size: :bro:type:`count` :bro:attr:`&optional` :bro:attr:`&log`
+         (present if :doc:`/scripts/base/files/extract/main.bro` is loaded)
+
+         The number of bytes extracted to disk.
+
+      entropy: :bro:type:`double` :bro:attr:`&log` :bro:attr:`&optional`
+         (present if :doc:`/scripts/policy/frameworks/files/entropy-test-all-files.bro` is loaded)
+
+         The information density of the contents of the file, 
+         expressed as a number of bits per character. 
+   :Attributes: :bro:attr:`&redef`
+
+   Contains all metadata related to the analysis of a given file.
+   For the most part, fields here are derived from ones of the same name
+   in :bro:see:`fa_file`.
+
+.. bro:type:: Files::ProtoRegistration
+
+   :Type: :bro:type:`record`
+
+      get_file_handle: :bro:type:`function` (c: :bro:type:`connection`, is_orig: :bro:type:`bool`) : :bro:type:`string`
+         A callback to generate a file handle on demand when
+         one is needed by the core.
+
+      describe: :bro:type:`function` (f: :bro:type:`fa_file`) : :bro:type:`string` :bro:attr:`&default` = :bro:type:`function` :bro:attr:`&optional`
+         A callback to "describe" a file.  In the case of an HTTP
+         transfer the most obvious description would be the URL.
+         It's like an extremely compressed version of the normal log.
+
+
+Events
+######
+.. bro:id:: Files::log_files
+
+   :Type: :bro:type:`event` (rec: :bro:type:`Files::Info`)
+
+   Event that can be handled to access the Info record as it is sent on
+   to the logging framework.
+
+Functions
+#########
+.. bro:id:: Files::add_analyzer
+
+   :Type: :bro:type:`function` (f: :bro:type:`fa_file`, tag: :bro:type:`Files::Tag`, args: :bro:type:`Files::AnalyzerArgs` :bro:attr:`&default` = ``[chunk_event=<uninitialized>, stream_event=<uninitialized>, extract_filename=<uninitialized>, extract_limit=104857600]`` :bro:attr:`&optional`) : :bro:type:`bool`
+
+   Adds an analyzer to the analysis of a given file.
+   
+
+   :f: the file.
+   
+
+   :tag: the analyzer type.
+   
+
+   :args: any parameters the analyzer takes.
+   
+
+   :returns: true if the analyzer will be added, or false if analysis
+            for the file isn't currently active or the *args*
+            were invalid for the analyzer type.
+
+.. bro:id:: Files::all_registered_mime_types
+
+   :Type: :bro:type:`function` () : :bro:type:`table` [:bro:type:`Files::Tag`] of :bro:type:`set` [:bro:type:`string`]
+
+   Returns a table of all MIME-type-to-analyzer mappings currently registered.
+   
+
+   :returns: A table mapping each analyzer to the set of MIME types
+            registered for it.
+
+.. bro:id:: Files::analyzer_name
+
+   :Type: :bro:type:`function` (tag: :bro:type:`Files::Tag`) : :bro:type:`string`
+
+   Translates a file analyzer enum value to a string with the
+   analyzer's name.
+   
+
+   :tag: The analyzer tag.
+   
+
+   :returns: The analyzer name corresponding to the tag.
+
+.. bro:id:: Files::describe
+
+   :Type: :bro:type:`function` (f: :bro:type:`fa_file`) : :bro:type:`string`
+
+   Provides a text description regarding metadata of the file.
+   For example, with HTTP it would return a URL.
+   
+
+   :f: The file to be described.
+   
+
+   :returns: a text description regarding metadata of the file.
+
+.. bro:id:: Files::disable_reassembly
+
+   :Type: :bro:type:`function` (f: :bro:type:`fa_file`) : :bro:type:`void`
+
+   Disables the file reassembler on this file.  If the file is not 
+   transferred out of order this will have no effect.
+   
+
+   :f: the file.
+
+.. bro:id:: Files::enable_reassembly
+
+   :Type: :bro:type:`function` (f: :bro:type:`fa_file`) : :bro:type:`void`
+
+   Allows the file reassembler to be used if it's necessary because the
+   file is transferred out of order.
+   
+
+   :f: the file.
+
+.. bro:id:: Files::file_exists
+
+   :Type: :bro:type:`function` (fuid: :bro:type:`string`) : :bro:type:`bool`
+
+   Lookup to see if a particular file id exists and is still valid.
+   
+
+   :fuid: the file id.
+   
+
+   :returns: T if the file uid is known.
+
+.. bro:id:: Files::lookup_file
+
+   :Type: :bro:type:`function` (fuid: :bro:type:`string`) : :bro:type:`fa_file`
+
+   Lookup an :bro:see:`fa_file` record with the file id.
+   
+
+   :fuid: the file id.
+   
+
+   :returns: the associated :bro:see:`fa_file` record.
+
+.. bro:id:: Files::register_analyzer_add_callback
+
+   :Type: :bro:type:`function` (tag: :bro:type:`Files::Tag`, callback: :bro:type:`function` (f: :bro:type:`fa_file`, args: :bro:type:`Files::AnalyzerArgs`) : :bro:type:`void`) : :bro:type:`void`
+
+   Register a callback for file analyzers to use if they need to do some
+   manipulation when they are being added to a file before the core code
+   takes over.  This is unlikely to be interesting for users and should
+   only be called by file analyzer authors but is *not required*.
+   
+
+   :tag: Tag for the file analyzer.
+   
+
+   :callback: Function to execute when the given file analyzer is being added.
+
+.. bro:id:: Files::register_for_mime_type
+
+   :Type: :bro:type:`function` (tag: :bro:type:`Files::Tag`, mt: :bro:type:`string`) : :bro:type:`bool`
+
+   Registers a MIME type for an analyzer. If a future file with this type is seen,
+   the analyzer will be automatically assigned to parsing it. The function *adds*
+   to all MIME types already registered, it doesn't replace them.
+   
+
+   :tag: The tag of the analyzer.
+   
+
+   :mt: The MIME type in the form "foo/bar" (case-insensitive).
+   
+
+   :returns: True if the MIME type was successfully registered.
+
+.. bro:id:: Files::register_for_mime_types
+
+   :Type: :bro:type:`function` (tag: :bro:type:`Files::Tag`, mime_types: :bro:type:`set` [:bro:type:`string`]) : :bro:type:`bool`
+
+   Registers a set of MIME types for an analyzer. If a future connection on one of
+   these types is seen, the analyzer will be automatically assigned to parsing it.
+   The function *adds* to all MIME types already registered, it doesn't replace
+   them.
+   
+
+   :tag: The tag of the analyzer.
+   
+
+   :mts: The set of MIME types, each in the form "foo/bar" (case-insensitive).
+   
+
+   :returns: True if the MIME types were successfully registered.
+
+.. bro:id:: Files::register_protocol
+
+   :Type: :bro:type:`function` (tag: :bro:type:`Analyzer::Tag`, reg: :bro:type:`Files::ProtoRegistration`) : :bro:type:`bool`
+
+   Register callbacks for protocols that work with the Files framework.
+   The callbacks must uniquely identify a file and each protocol can 
+   only have a single callback registered for it.
+   
+
+   :tag: Tag for the protocol analyzer having a callback being registered.
+   
+
+   :reg: A :bro:see:`Files::ProtoRegistration` record.
+   
+
+   :returns: true if the protocol being registered was not previously registered.
+
+.. bro:id:: Files::registered_mime_types
+
+   :Type: :bro:type:`function` (tag: :bro:type:`Files::Tag`) : :bro:type:`set` [:bro:type:`string`]
+
+   Returns a set of all MIME types currently registered for a specific analyzer.
+   
+
+   :tag: The tag of the analyzer.
+   
+
+   :returns: The set of MIME types.
+
+.. bro:id:: Files::remove_analyzer
+
+   :Type: :bro:type:`function` (f: :bro:type:`fa_file`, tag: :bro:type:`Files::Tag`, args: :bro:type:`Files::AnalyzerArgs` :bro:attr:`&default` = ``[chunk_event=<uninitialized>, stream_event=<uninitialized>, extract_filename=<uninitialized>, extract_limit=104857600]`` :bro:attr:`&optional`) : :bro:type:`bool`
+
+   Removes an analyzer from the analysis of a given file.
+   
+
+   :f: the file.
+   
+
+   :tag: the analyzer type.
+   
+
+   :args: the analyzer (type and args) to remove.
+   
+
+   :returns: true if the analyzer will be removed, or false if analysis
+            for the file isn't currently active.
+
+.. bro:id:: Files::set_reassembly_buffer_size
+
+   :Type: :bro:type:`function` (f: :bro:type:`fa_file`, max: :bro:type:`count`) : :bro:type:`void`
+
+   Set the maximum size the reassembly buffer is allowed to grow
+   for the given file.
+   
+
+   :f: the file.
+   
+
+   :max: Maximum allowed size of the reassembly buffer.
+
+.. bro:id:: Files::set_timeout_interval
+
+   :Type: :bro:type:`function` (f: :bro:type:`fa_file`, t: :bro:type:`interval`) : :bro:type:`bool`
+
+   Sets the *timeout_interval* field of :bro:see:`fa_file`, which is
+   used to determine the length of inactivity that is allowed for a file
+   before internal state related to it is cleaned up.  When used within
+   a :bro:see:`file_timeout` handler, the analysis will delay timing out
+   again for the period specified by *t*.
+   
+
+   :f: the file.
+   
+
+   :t: the amount of time the file can remain inactive before discarding.
+   
+
+   :returns: true if the timeout interval was set, or false if analysis
+            for the file isn't currently active.
+
+.. bro:id:: Files::stop
+
+   :Type: :bro:type:`function` (f: :bro:type:`fa_file`) : :bro:type:`bool`
+
+   Stops/ignores any further analysis of a given file.
+   
+
+   :f: the file.
+   
+
+   :returns: true if analysis for the given file will be ignored for the
+            rest of its contents, or false if analysis for the file
+            isn't currently active.
+
+

doc/scripts/base/frameworks/input/__load__.bro.rst

@@ -0,0 +1,14 @@
+:tocdepth: 3
+
+base/frameworks/input/__load__.bro
+==================================
+
+
+:Imports: :doc:`base/frameworks/input/main.bro </scripts/base/frameworks/input/main.bro>`, :doc:`base/frameworks/input/readers/ascii.bro </scripts/base/frameworks/input/readers/ascii.bro>`, :doc:`base/frameworks/input/readers/benchmark.bro </scripts/base/frameworks/input/readers/benchmark.bro>`, :doc:`base/frameworks/input/readers/binary.bro </scripts/base/frameworks/input/readers/binary.bro>`, :doc:`base/frameworks/input/readers/config.bro </scripts/base/frameworks/input/readers/config.bro>`, :doc:`base/frameworks/input/readers/raw.bro </scripts/base/frameworks/input/readers/raw.bro>`, :doc:`base/frameworks/input/readers/sqlite.bro </scripts/base/frameworks/input/readers/sqlite.bro>`
+
+Summary
+~~~~~~~
+
+Detailed Interface
+~~~~~~~~~~~~~~~~~~
+

doc/scripts/base/frameworks/input/index.rst

@@ -0,0 +1,50 @@
+:orphan:
+
+Package: base/frameworks/input
+==============================
+
+The input framework provides a way to read previously stored data either as
+an event stream or into a Bro table.
+
+:doc:`/scripts/base/frameworks/input/__load__.bro`
+
+
+:doc:`/scripts/base/frameworks/input/main.bro`
+
+   The input framework provides a way to read previously stored data either
+   as an event stream or into a Bro table.
+
+:doc:`/scripts/base/frameworks/input/readers/ascii.bro`
+
+   Interface for the ascii input reader.
+   
+   The defaults are set to match Bro's ASCII output.
+
+:doc:`/scripts/base/frameworks/input/readers/raw.bro`
+
+   Interface for the raw input reader.
+
+:doc:`/scripts/base/frameworks/input/readers/benchmark.bro`
+
+   Interface for the benchmark input reader.
+
+:doc:`/scripts/base/frameworks/input/readers/binary.bro`
+
+   Interface for the binary input reader.
+
+:doc:`/scripts/base/frameworks/input/readers/config.bro`
+
+   Interface for the config input reader.
+
+:doc:`/scripts/base/frameworks/input/readers/sqlite.bro`
+
+   Interface for the SQLite input reader. Redefinable options are available
+   to tweak the input format of the SQLite reader.
+   
+   See :doc:`/frameworks/logging-input-sqlite` for an introduction on how to
+   use the SQLite reader.
+   
+   When using the SQLite reader, you have to specify the SQL query that returns
+   the desired data by setting ``query`` in the ``config`` table. See the
+   introduction mentioned above for an example.
+

doc/scripts/base/frameworks/input/main.bro.rst

@@ -0,0 +1,418 @@
+:tocdepth: 3
+
+base/frameworks/input/main.bro
+==============================
+.. bro:namespace:: Input
+
+The input framework provides a way to read previously stored data either
+as an event stream or into a Bro table.
+
+:Namespace: Input
+:Imports: :doc:`base/bif/input.bif.bro </scripts/base/bif/input.bif.bro>`
+
+Summary
+~~~~~~~
+Runtime Options
+###############
+============================================================================= ==============================
+:bro:id:`Input::default_mode`: :bro:type:`Input::Mode` :bro:attr:`&redef`     The default reader mode used.
+:bro:id:`Input::default_reader`: :bro:type:`Input::Reader` :bro:attr:`&redef` The default input reader used.
+============================================================================= ==============================
+
+Redefinable Options
+###################
+============================================================================== =========================================================
+:bro:id:`Input::accept_unsupported_types`: :bro:type:`bool` :bro:attr:`&redef` Flag that controls if the input framework accepts records
+                                                                               that contain types that are not supported (at the moment
+                                                                               file and function).
+:bro:id:`Input::empty_field`: :bro:type:`string` :bro:attr:`&redef`            String to use for empty fields.
+:bro:id:`Input::separator`: :bro:type:`string` :bro:attr:`&redef`              Separator between fields.
+:bro:id:`Input::set_separator`: :bro:type:`string` :bro:attr:`&redef`          Separator between set elements.
+:bro:id:`Input::unset_field`: :bro:type:`string` :bro:attr:`&redef`            String to use for an unset &optional field.
+============================================================================== =========================================================
+
+Types
+#####
+========================================================== ===================================================================
+:bro:type:`Input::AnalysisDescription`: :bro:type:`record` A file analysis input stream type used to forward input data to the
+                                                           file analysis framework.
+:bro:type:`Input::Event`: :bro:type:`enum`                 
+:bro:type:`Input::EventDescription`: :bro:type:`record`    An event input stream type used to send input data to a Bro event.
+:bro:type:`Input::Mode`: :bro:type:`enum`                  Type that defines the input stream read mode.
+:bro:type:`Input::TableDescription`: :bro:type:`record`    A table input stream type used to send data to a Bro table.
+:bro:type:`Input::Reader`: :bro:type:`enum`                
+========================================================== ===================================================================
+
+Events
+######
+=============================================== ====================================================================
+:bro:id:`Input::end_of_data`: :bro:type:`event` Event that is called when the end of a data source has been reached,
+                                                including after an update.
+=============================================== ====================================================================
+
+Functions
+#########
+=================================================== ============================================================
+:bro:id:`Input::add_analysis`: :bro:type:`function` Create a new file analysis input stream from a given source.
+:bro:id:`Input::add_event`: :bro:type:`function`    Create a new event input stream from a given source.
+:bro:id:`Input::add_table`: :bro:type:`function`    Create a new table input stream from a given source.
+:bro:id:`Input::force_update`: :bro:type:`function` Forces the current input to be checked for changes.
+:bro:id:`Input::remove`: :bro:type:`function`       Remove an input stream.
+=================================================== ============================================================
+
+
+Detailed Interface
+~~~~~~~~~~~~~~~~~~
+Runtime Options
+###############
+.. bro:id:: Input::default_mode
+
+   :Type: :bro:type:`Input::Mode`
+   :Attributes: :bro:attr:`&redef`
+   :Default: ``Input::MANUAL``
+
+   The default reader mode used. Defaults to `MANUAL`.
+
+.. bro:id:: Input::default_reader
+
+   :Type: :bro:type:`Input::Reader`
+   :Attributes: :bro:attr:`&redef`
+   :Default: ``Input::READER_ASCII``
+
+   The default input reader used. Defaults to `READER_ASCII`.
+
+Redefinable Options
+###################
+.. bro:id:: Input::accept_unsupported_types
+
+   :Type: :bro:type:`bool`
+   :Attributes: :bro:attr:`&redef`
+   :Default: ``F``
+
+   Flag that controls if the input framework accepts records
+   that contain types that are not supported (at the moment
+   file and function). If true, the input framework will
+   warn in these cases, but continue. If false, it will
+   abort. Defaults to false (abort).
+
+.. bro:id:: Input::empty_field
+
+   :Type: :bro:type:`string`
+   :Attributes: :bro:attr:`&redef`
+   :Default: ``"(empty)"``
+
+   String to use for empty fields.
+   Individual readers can use a different value.
+
+.. bro:id:: Input::separator
+
+   :Type: :bro:type:`string`
+   :Attributes: :bro:attr:`&redef`
+   :Default: ``"\x09"``
+
+   Separator between fields.
+   Please note that the separator has to be exactly one character long.
+   Individual readers can use a different value.
+
+.. bro:id:: Input::set_separator
+
+   :Type: :bro:type:`string`
+   :Attributes: :bro:attr:`&redef`
+   :Default: ``","``
+
+   Separator between set elements.
+   Please note that the separator has to be exactly one character long.
+   Individual readers can use a different value.
+
+.. bro:id:: Input::unset_field
+
+   :Type: :bro:type:`string`
+   :Attributes: :bro:attr:`&redef`
+   :Default: ``"-"``
+
+   String to use for an unset &optional field.
+   Individual readers can use a different value.
+
+Types
+#####
+.. bro:type:: Input::AnalysisDescription
+
+   :Type: :bro:type:`record`
+
+      source: :bro:type:`string`
+         String that allows the reader to find the source.
+         For `READER_ASCII`, this is the filename.
+
+      reader: :bro:type:`Input::Reader` :bro:attr:`&default` = ``Input::READER_BINARY`` :bro:attr:`&optional`
+         Reader to use for this stream.  Compatible readers must be
+         able to accept a filter of a single string type (i.e.
+         they read a byte stream).
+
+      mode: :bro:type:`Input::Mode` :bro:attr:`&default` = :bro:see:`Input::default_mode` :bro:attr:`&optional`
+         Read mode to use for this stream.
+
+      name: :bro:type:`string`
+         Descriptive name that uniquely identifies the input source.
+         Can be used to remove a stream at a later time.
+         This will also be used for the unique *source* field of
+         :bro:see:`fa_file`.  Most of the time, the best choice for this
+         field will be the same value as the *source* field.
+
+      config: :bro:type:`table` [:bro:type:`string`] of :bro:type:`string` :bro:attr:`&default` = ``{  }`` :bro:attr:`&optional`
+         A key/value table that will be passed to the reader.
+         Interpretation of the values is left to the reader, but
+         usually they will be used for configuration purposes.
+
+   A file analysis input stream type used to forward input data to the
+   file analysis framework.
+
+.. bro:type:: Input::Event
+
+   :Type: :bro:type:`enum`
+
+      .. bro:enum:: Input::EVENT_NEW Input::Event
+
+         New data has been imported.
+
+      .. bro:enum:: Input::EVENT_CHANGED Input::Event
+
+         Existing data has been changed.
+
+      .. bro:enum:: Input::EVENT_REMOVED Input::Event
+
+         Previously existing data has been removed.
+
+
+.. bro:type:: Input::EventDescription
+
+   :Type: :bro:type:`record`
+
+      source: :bro:type:`string`
+         String that allows the reader to find the source.
+         For `READER_ASCII`, this is the filename.
+
+      reader: :bro:type:`Input::Reader` :bro:attr:`&default` = :bro:see:`Input::default_reader` :bro:attr:`&optional`
+         Reader to use for this stream.
+
+      mode: :bro:type:`Input::Mode` :bro:attr:`&default` = :bro:see:`Input::default_mode` :bro:attr:`&optional`
+         Read mode to use for this stream.
+
+      name: :bro:type:`string`
+         Descriptive name. Used to remove a stream at a later time.
+
+      fields: :bro:type:`any`
+         Record type describing the fields to be retrieved from the input
+         source.
+
+      want_record: :bro:type:`bool` :bro:attr:`&default` = ``T`` :bro:attr:`&optional`
+         If this is false, the event receives each value in *fields* as a
+         separate argument.
+         If this is set to true (default), the event receives all fields in
+         a single record value.
+
+      ev: :bro:type:`any`
+         The event that is raised each time a new line is received from the
+         reader. The event will receive an Input::EventDescription record
+         as the first argument, an Input::Event enum as the second
+         argument, and the fields (as specified in *fields*) as the following
+         arguments (this will either be a single record value containing
+         all fields, or each field value as a separate argument).
+
+      error_ev: :bro:type:`any` :bro:attr:`&optional`
+         Error event that is raised when an information, warning or error
+         is raised by the input stream. If the level is error, the stream will automatically
+         be closed.
+         The event receives the Input::EventDescription as the first argument, the
+         message as the second argument and the Reporter::Level as the third argument.
+         
+         The event is raised like it had been declared as follows:
+         error_ev: function(desc: EventDescription, message: string, level: Reporter::Level) &optional;
+         The actual declaration uses the ``any`` type because of deficiencies of the Bro type system.
+
+      config: :bro:type:`table` [:bro:type:`string`] of :bro:type:`string` :bro:attr:`&default` = ``{  }`` :bro:attr:`&optional`
+         A key/value table that will be passed to the reader.
+         Interpretation of the values is left to the reader, but
+         usually they will be used for configuration purposes.
+
+   An event input stream type used to send input data to a Bro event.
+
+.. bro:type:: Input::Mode
+
+   :Type: :bro:type:`enum`
+
+      .. bro:enum:: Input::MANUAL Input::Mode
+
+         Do not automatically reread the file after it has been read.
+
+      .. bro:enum:: Input::REREAD Input::Mode
+
+         Reread the entire file each time a change is found.
+
+      .. bro:enum:: Input::STREAM Input::Mode
+
+         Read data from end of file each time new data is appended.
+
+   Type that defines the input stream read mode.
+
+.. bro:type:: Input::TableDescription
+
+   :Type: :bro:type:`record`
+
+      source: :bro:type:`string`
+         String that allows the reader to find the source of the data.
+         For `READER_ASCII`, this is the filename.
+
+      reader: :bro:type:`Input::Reader` :bro:attr:`&default` = :bro:see:`Input::default_reader` :bro:attr:`&optional`
+         Reader to use for this stream.
+
+      mode: :bro:type:`Input::Mode` :bro:attr:`&default` = :bro:see:`Input::default_mode` :bro:attr:`&optional`
+         Read mode to use for this stream.
+
+      name: :bro:type:`string`
+         Name of the input stream.  This is used by some functions to
+         manipulate the stream.
+
+      destination: :bro:type:`any`
+         Table which will receive the data read by the input framework.
+
+      idx: :bro:type:`any`
+         Record that defines the values used as the index of the table.
+
+      val: :bro:type:`any` :bro:attr:`&optional`
+         Record that defines the values used as the elements of the table.
+         If this is undefined, then *destination* must be a set.
+
+      want_record: :bro:type:`bool` :bro:attr:`&default` = ``T`` :bro:attr:`&optional`
+         Defines if the value of the table is a record (default), or a single
+         value. When this is set to false, then *val* can only contain one
+         element.
+
+      ev: :bro:type:`any` :bro:attr:`&optional`
+         The event that is raised each time a value is added to, changed in,
+         or removed from the table. The event will receive an
+         Input::TableDescription as the first argument, an Input::Event
+         enum as the second argument, the *idx* record as the third argument
+         and the value (record) as the fourth argument.
+
+      pred: :bro:type:`function` (typ: :bro:type:`Input::Event`, left: :bro:type:`any`, right: :bro:type:`any`) : :bro:type:`bool` :bro:attr:`&optional`
+         Predicate function that can decide if an insertion, update or removal
+         should really be executed. Parameters have same meaning as for the
+         event.
+         If true is returned, the update is performed. If false is returned,
+         it is skipped.
+
+      error_ev: :bro:type:`any` :bro:attr:`&optional`
+         Error event that is raised when an information, warning or error
+         is raised by the input stream. If the level is error, the stream will automatically
+         be closed.
+         The event receives the Input::TableDescription as the first argument, the
+         message as the second argument and the Reporter::Level as the third argument.
+         
+         The event is raised like if it had been declared as follows:
+         error_ev: function(desc: TableDescription, message: string, level: Reporter::Level) &optional;
+         The actual declaration uses the ``any`` type because of deficiencies of the Bro type system.
+
+      config: :bro:type:`table` [:bro:type:`string`] of :bro:type:`string` :bro:attr:`&default` = ``{  }`` :bro:attr:`&optional`
+         A key/value table that will be passed to the reader.
+         Interpretation of the values is left to the reader, but
+         usually they will be used for configuration purposes.
+
+   A table input stream type used to send data to a Bro table.
+
+.. bro:type:: Input::Reader
+
+   :Type: :bro:type:`enum`
+
+      .. bro:enum:: Input::READER_ASCII Input::Reader
+
+      .. bro:enum:: Input::READER_BENCHMARK Input::Reader
+
+      .. bro:enum:: Input::READER_BINARY Input::Reader
+
+      .. bro:enum:: Input::READER_CONFIG Input::Reader
+
+      .. bro:enum:: Input::READER_RAW Input::Reader
+
+      .. bro:enum:: Input::READER_SQLITE Input::Reader
+
+
+Events
+######
+.. bro:id:: Input::end_of_data
+
+   :Type: :bro:type:`event` (name: :bro:type:`string`, source: :bro:type:`string`)
+
+   Event that is called when the end of a data source has been reached,
+   including after an update.
+   
+
+   :name: Name of the input stream.
+   
+
+   :source: String that identifies the data source (such as the filename).
+
+Functions
+#########
+.. bro:id:: Input::add_analysis
+
+   :Type: :bro:type:`function` (description: :bro:type:`Input::AnalysisDescription`) : :bro:type:`bool`
+
+   Create a new file analysis input stream from a given source.  Data read
+   from the source is automatically forwarded to the file analysis
+   framework.
+   
+
+   :description: A record describing the source.
+   
+
+   :returns: true on success.
+
+.. bro:id:: Input::add_event
+
+   :Type: :bro:type:`function` (description: :bro:type:`Input::EventDescription`) : :bro:type:`bool`
+
+   Create a new event input stream from a given source.
+   
+
+   :description: `EventDescription` record describing the source.
+   
+
+   :returns: true on success.
+
+.. bro:id:: Input::add_table
+
+   :Type: :bro:type:`function` (description: :bro:type:`Input::TableDescription`) : :bro:type:`bool`
+
+   Create a new table input stream from a given source.
+   
+
+   :description: `TableDescription` record describing the source.
+   
+
+   :returns: true on success.
+
+.. bro:id:: Input::force_update
+
+   :Type: :bro:type:`function` (id: :bro:type:`string`) : :bro:type:`bool`
+
+   Forces the current input to be checked for changes.
+   
+
+   :id: string value identifying the stream.
+   
+
+   :returns: true on success and false if the named stream was not found.
+
+.. bro:id:: Input::remove
+
+   :Type: :bro:type:`function` (id: :bro:type:`string`) : :bro:type:`bool`
+
+   Remove an input stream.
+   
+
+   :id: string value identifying the stream to be removed.
+   
+
+   :returns: true on success and false if the named stream was not found.
+
+

doc/scripts/base/frameworks/input/readers/ascii.bro.rst

@@ -0,0 +1,104 @@
+:tocdepth: 3
+
+base/frameworks/input/readers/ascii.bro
+=======================================
+.. bro:namespace:: InputAscii
+
+Interface for the ascii input reader.
+
+The defaults are set to match Bro's ASCII output.
+
+:Namespace: InputAscii
+
+Summary
+~~~~~~~
+Redefinable Options
+###################
+================================================================================ ===========================================
+:bro:id:`InputAscii::empty_field`: :bro:type:`string` :bro:attr:`&redef`         String to use for empty fields.
+:bro:id:`InputAscii::fail_on_file_problem`: :bro:type:`bool` :bro:attr:`&redef`  Fail on file read problems.
+:bro:id:`InputAscii::fail_on_invalid_lines`: :bro:type:`bool` :bro:attr:`&redef` Fail on invalid lines.
+:bro:id:`InputAscii::separator`: :bro:type:`string` :bro:attr:`&redef`           Separator between fields.
+:bro:id:`InputAscii::set_separator`: :bro:type:`string` :bro:attr:`&redef`       Separator between set and vector elements.
+:bro:id:`InputAscii::unset_field`: :bro:type:`string` :bro:attr:`&redef`         String to use for an unset &optional field.
+================================================================================ ===========================================
+
+
+Detailed Interface
+~~~~~~~~~~~~~~~~~~
+Redefinable Options
+###################
+.. bro:id:: InputAscii::empty_field
+
+   :Type: :bro:type:`string`
+   :Attributes: :bro:attr:`&redef`
+   :Default: ``"(empty)"``
+
+   String to use for empty fields.
+
+.. bro:id:: InputAscii::fail_on_file_problem
+
+   :Type: :bro:type:`bool`
+   :Attributes: :bro:attr:`&redef`
+   :Default: ``F``
+
+   Fail on file read problems. If set to true, the ascii
+   input reader will fail when encountering any problems
+   while reading a file different from invalid lines.
+   Examples of such problems are permission problems, or
+   missing files.
+   When set to false, these problems will be ignored. This
+   has an especially big effect for the REREAD mode, which will
+   seamlessly recover from read errors when a file is
+   only temporarily inaccessible. For MANUAL or STREAM files,
+   errors will most likely still be fatal since no automatic
+   re-reading of the file is attempted.
+   Individual readers can use a different value using
+   the $config table.
+   fail_on_file_problem = T was the default behavior
+   until Bro 2.6.
+
+.. bro:id:: InputAscii::fail_on_invalid_lines
+
+   :Type: :bro:type:`bool`
+   :Attributes: :bro:attr:`&redef`
+   :Default: ``F``
+
+   Fail on invalid lines. If set to false, the ascii
+   input reader will jump over invalid lines, reporting
+   warnings in reporter.log. If set to true, errors in
+   input lines will be handled as fatal errors for the
+   reader thread; reading will abort immediately and
+   an error will be logged to reporter.log.
+   Individual readers can use a different value using
+   the $config table.
+   fail_on_invalid_lines = T was the default behavior
+   until Bro 2.6.
+
+.. bro:id:: InputAscii::separator
+
+   :Type: :bro:type:`string`
+   :Attributes: :bro:attr:`&redef`
+   :Default: ``"\x09"``
+
+   Separator between fields.
+   Please note that the separator has to be exactly one character long.
+
+.. bro:id:: InputAscii::set_separator
+
+   :Type: :bro:type:`string`
+   :Attributes: :bro:attr:`&redef`
+   :Default: ``","``
+
+   Separator between set and vector elements.
+   Please note that the separator has to be exactly one character long.
+
+.. bro:id:: InputAscii::unset_field
+
+   :Type: :bro:type:`string`
+   :Attributes: :bro:attr:`&redef`
+   :Default: ``"-"``
+
+   String to use for an unset &optional field.
+
+

doc/scripts/base/frameworks/input/readers/benchmark.bro.rst

@@ -0,0 +1,77 @@
+:tocdepth: 3
+
+base/frameworks/input/readers/benchmark.bro
+===========================================
+.. bro:namespace:: InputBenchmark
+
+Interface for the benchmark input reader.
+
+:Namespace: InputBenchmark
+
+Summary
+~~~~~~~
+Redefinable Options
+###################
+============================================================================ =========================================================
+:bro:id:`InputBenchmark::addfactor`: :bro:type:`count` :bro:attr:`&redef`    Addition factor for each heartbeat.
+:bro:id:`InputBenchmark::autospread`: :bro:type:`double` :bro:attr:`&redef`  Spreading where usleep = 1000000 / autospread * num_lines
+:bro:id:`InputBenchmark::factor`: :bro:type:`double` :bro:attr:`&redef`      Multiplication factor for each second.
+:bro:id:`InputBenchmark::spread`: :bro:type:`count` :bro:attr:`&redef`       Spread factor between lines.
+:bro:id:`InputBenchmark::stopspreadat`: :bro:type:`count` :bro:attr:`&redef` Stop spreading at x lines per heartbeat.
+:bro:id:`InputBenchmark::timedspread`: :bro:type:`double` :bro:attr:`&redef` 1 -> enable timed spreading.
+============================================================================ =========================================================
+
+
+Detailed Interface
+~~~~~~~~~~~~~~~~~~
+Redefinable Options
+###################
+.. bro:id:: InputBenchmark::addfactor
+
+   :Type: :bro:type:`count`
+   :Attributes: :bro:attr:`&redef`
+   :Default: ``0``
+
+   Addition factor for each heartbeat.
+
+.. bro:id:: InputBenchmark::autospread
+
+   :Type: :bro:type:`double`
+   :Attributes: :bro:attr:`&redef`
+   :Default: ``0.0``
+
+   Spreading where usleep = 1000000 / autospread * num_lines
+
+.. bro:id:: InputBenchmark::factor
+
+   :Type: :bro:type:`double`
+   :Attributes: :bro:attr:`&redef`
+   :Default: ``1.0``
+
+   Multiplication factor for each second.
+
+.. bro:id:: InputBenchmark::spread
+
+   :Type: :bro:type:`count`
+   :Attributes: :bro:attr:`&redef`
+   :Default: ``0``
+
+   Spread factor between lines.
+
+.. bro:id:: InputBenchmark::stopspreadat
+
+   :Type: :bro:type:`count`
+   :Attributes: :bro:attr:`&redef`
+   :Default: ``0``
+
+   Stop spreading at x lines per heartbeat.
+
+.. bro:id:: InputBenchmark::timedspread
+
+   :Type: :bro:type:`double`
+   :Attributes: :bro:attr:`&redef`
+   :Default: ``0.0``
+
+   1 -> enable timed spreading.
+
+

doc/scripts/base/frameworks/input/readers/binary.bro.rst

@@ -0,0 +1,32 @@
+:tocdepth: 3
+
+base/frameworks/input/readers/binary.bro
+========================================
+.. bro:namespace:: InputBinary
+
+Interface for the binary input reader.
+
+:Namespace: InputBinary
+
+Summary
+~~~~~~~
+Redefinable Options
+###################
+======================================================================= ==========================================================
+:bro:id:`InputBinary::chunk_size`: :bro:type:`count` :bro:attr:`&redef` Size of data chunks to read from the input file at a time.
+======================================================================= ==========================================================
+
+
+Detailed Interface
+~~~~~~~~~~~~~~~~~~
+Redefinable Options
+###################
+.. bro:id:: InputBinary::chunk_size
+
+   :Type: :bro:type:`count`
+   :Attributes: :bro:attr:`&redef`
+   :Default: ``1024``
+
+   Size of data chunks to read from the input file at a time.
+
+

doc/scripts/base/frameworks/input/readers/config.bro.rst

@@ -0,0 +1,95 @@
+:tocdepth: 3
+
+base/frameworks/input/readers/config.bro
+========================================
+.. bro:namespace:: InputConfig
+
+Interface for the config input reader.
+
+:Namespace: InputConfig
+
+Summary
+~~~~~~~
+Redefinable Options
+###################
+================================================================================ ==========================================
+:bro:id:`InputConfig::empty_field`: :bro:type:`string` :bro:attr:`&redef`        String to use for empty fields.
+:bro:id:`InputConfig::fail_on_file_problem`: :bro:type:`bool` :bro:attr:`&redef` Fail on file read problems.
+:bro:id:`InputConfig::set_separator`: :bro:type:`string` :bro:attr:`&redef`      Separator between set and vector elements.
+================================================================================ ==========================================
+
+Events
+######
+=================================================== ==============================================================
+:bro:id:`InputConfig::new_value`: :bro:type:`event` Event that is called when a config option is added or changes.
+=================================================== ==============================================================
+
+
+Detailed Interface
+~~~~~~~~~~~~~~~~~~
+Redefinable Options
+###################
+.. bro:id:: InputConfig::empty_field
+
+   :Type: :bro:type:`string`
+   :Attributes: :bro:attr:`&redef`
+   :Default: ``""``
+
+   String to use for empty fields.
+   By default this is the empty string, meaning that an empty input field
+   will result in an empty set.
+
+.. bro:id:: InputConfig::fail_on_file_problem
+
+   :Type: :bro:type:`bool`
+   :Attributes: :bro:attr:`&redef`
+   :Default: ``F``
+
+   Fail on file read problems. If set to true, the config
+   input reader will fail when encountering any problems
+   while reading a file different from invalid lines.
+   Examples of such problems are permission problems, or
+   missing files.
+   When set to false, these problems will be ignored. This
+   has an especially big effect for the REREAD mode, which will
+   seamlessly recover from read errors when a file is
+   only temporarily inaccessible. For MANUAL or STREAM files,
+   errors will most likely still be fatal since no automatic
+   re-reading of the file is attempted.
+   Individual readers can use a different value using
+   the $config table.
+
+.. bro:id:: InputConfig::set_separator
+
+   :Type: :bro:type:`string`
+   :Attributes: :bro:attr:`&redef`
+   :Default: ``","``
+
+   Separator between set and vector elements.
+   Please note that the separator has to be exactly one character long.
+
+Events
+######
+.. bro:id:: InputConfig::new_value
+
+   :Type: :bro:type:`event` (name: :bro:type:`string`, source: :bro:type:`string`, id: :bro:type:`string`, value: :bro:type:`any`)
+
+   Event that is called when a config option is added or changes.
+   
+   Note - this does not track the reason for a change (new, changed),
+   and also does not track removals. If you need this, combine the event
+   with a table reader.
+   
+
+   :name: Name of the input stream.
+   
+
+   :source: Source of the input stream.
+   
+
+   :id: ID of the configuration option being set.
+   
+
+   :value: New value of the configuration option being set.
+
+

doc/scripts/base/frameworks/input/readers/raw.bro.rst

@@ -0,0 +1,58 @@
+:tocdepth: 3
+
+base/frameworks/input/readers/raw.bro
+=====================================
+.. bro:namespace:: InputRaw
+
+Interface for the raw input reader.
+
+:Namespace: InputRaw
+
+Summary
+~~~~~~~
+Redefinable Options
+###################
+=========================================================================== ================================
+:bro:id:`InputRaw::record_separator`: :bro:type:`string` :bro:attr:`&redef` Separator between input records.
+=========================================================================== ================================
+
+Events
+######
+======================================================= ====================================================================
+:bro:id:`InputRaw::process_finished`: :bro:type:`event` Event that is called when a process created by the raw reader exits.
+======================================================= ====================================================================
+
+
+Detailed Interface
+~~~~~~~~~~~~~~~~~~
+Redefinable Options
+###################
+.. bro:id:: InputRaw::record_separator
+
+   :Type: :bro:type:`string`
+   :Attributes: :bro:attr:`&redef`
+   :Default: ``"\x0a"``
+
+   Separator between input records.
+   Please note that the separator has to be exactly one character long.
+
+Events
+######
+.. bro:id:: InputRaw::process_finished
+
+   :Type: :bro:type:`event` (name: :bro:type:`string`, source: :bro:type:`string`, exit_code: :bro:type:`count`, signal_exit: :bro:type:`bool`)
+
+   Event that is called when a process created by the raw reader exits.
+   
+
+   :name: name of the input stream.
+
+   :source: source of the input stream.
+
+   :exit_code: exit code of the program, or number of the signal that forced
+              the program to exit.
+
+   :signal_exit: false when program exited normally, true when program was
+                forced to exit by a signal.
+
+

doc/scripts/base/frameworks/input/readers/sqlite.bro.rst

@@ -0,0 +1,59 @@
+:tocdepth: 3
+
+base/frameworks/input/readers/sqlite.bro
+========================================
+.. bro:namespace:: InputSQLite
+
+Interface for the SQLite input reader. Redefinable options are available
+to tweak the input format of the SQLite reader.
+
+See :doc:`/frameworks/logging-input-sqlite` for an introduction on how to
+use the SQLite reader.
+
+When using the SQLite reader, you have to specify the SQL query that returns
+the desired data by setting ``query`` in the ``config`` table. See the
+introduction mentioned above for an example.
+
+:Namespace: InputSQLite
+
+Summary
+~~~~~~~
+Redefinable Options
+###################
+=========================================================================== ===========================================
+:bro:id:`InputSQLite::empty_field`: :bro:type:`string` :bro:attr:`&redef`   String to use for empty fields.
+:bro:id:`InputSQLite::set_separator`: :bro:type:`string` :bro:attr:`&redef` Separator between set elements.
+:bro:id:`InputSQLite::unset_field`: :bro:type:`string` :bro:attr:`&redef`   String to use for an unset &optional field.
+=========================================================================== ===========================================
+
+
+Detailed Interface
+~~~~~~~~~~~~~~~~~~
+Redefinable Options
+###################
+.. bro:id:: InputSQLite::empty_field
+
+   :Type: :bro:type:`string`
+   :Attributes: :bro:attr:`&redef`
+   :Default: ``"(empty)"``
+
+   String to use for empty fields.
+
+.. bro:id:: InputSQLite::set_separator
+
+   :Type: :bro:type:`string`
+   :Attributes: :bro:attr:`&redef`
+   :Default: ``","``
+
+   Separator between set elements.
+   Please note that the separator has to be exactly one character long.
+
+.. bro:id:: InputSQLite::unset_field
+
+   :Type: :bro:type:`string`
+   :Attributes: :bro:attr:`&redef`
+   :Default: ``"-"``
+
+   String to use for an unset &optional field.
+
+

doc/scripts/base/frameworks/intel/__load__.bro.rst

@@ -0,0 +1,14 @@
+:tocdepth: 3
+
+base/frameworks/intel/__load__.bro
+==================================
+
+
+:Imports: :doc:`base/frameworks/cluster </scripts/base/frameworks/cluster/index>`, :doc:`base/frameworks/intel/files.bro </scripts/base/frameworks/intel/files.bro>`, :doc:`base/frameworks/intel/input.bro </scripts/base/frameworks/intel/input.bro>`, :doc:`base/frameworks/intel/main.bro </scripts/base/frameworks/intel/main.bro>`
+
+Summary
+~~~~~~~
+
+Detailed Interface
+~~~~~~~~~~~~~~~~~~
+

doc/scripts/base/frameworks/intel/files.bro.rst

@@ -0,0 +1,27 @@
+:tocdepth: 3
+
+base/frameworks/intel/files.bro
+===============================
+.. bro:namespace:: Intel
+
+File analysis framework integration for the intelligence framework. This
+script manages file information in intelligence framework data structures.
+
+:Namespace: Intel
+:Imports: :doc:`base/frameworks/intel/main.bro </scripts/base/frameworks/intel/main.bro>`
+
+Summary
+~~~~~~~
+Redefinitions
+#############
+=========================================== =============================================================
+:bro:type:`Intel::Info`: :bro:type:`record` Record used for the logging framework representing a positive
+                                            hit within the intelligence framework.
+:bro:type:`Intel::Seen`: :bro:type:`record` Information about a piece of "seen" data.
+:bro:type:`Intel::Type`: :bro:type:`enum`   Enum type to represent various types of intelligence data.
+=========================================== =============================================================
+
+
+Detailed Interface
+~~~~~~~~~~~~~~~~~~
+

doc/scripts/base/frameworks/intel/index.rst

@@ -0,0 +1,29 @@
+:orphan:
+
+Package: base/frameworks/intel
+==============================
+
+The intelligence framework provides a way to store and query intelligence
+data (such as IP addresses or strings). Metadata can also be associated
+with the intelligence.
+
+:doc:`/scripts/base/frameworks/intel/__load__.bro`
+
+
+:doc:`/scripts/base/frameworks/intel/main.bro`
+
+   The intelligence framework provides a way to store and query intelligence
+   data (e.g. IP addresses, URLs and hashes). The intelligence items can be
+   associated with metadata to allow informed decisions about matching and
+   handling.
+
+:doc:`/scripts/base/frameworks/intel/files.bro`
+
+   File analysis framework integration for the intelligence framework. This
+   script manages file information in intelligence framework data structures.
+
+:doc:`/scripts/base/frameworks/intel/input.bro`
+
+   Input handling for the intelligence framework. This script implements the
+   import of intelligence data from files using the input framework.
+

doc/scripts/base/frameworks/intel/input.bro.rst

@@ -0,0 +1,36 @@
+:tocdepth: 3
+
+base/frameworks/intel/input.bro
+===============================
+.. bro:namespace:: Intel
+
+Input handling for the intelligence framework. This script implements the
+import of intelligence data from files using the input framework.
+
+:Namespace: Intel
+:Imports: :doc:`base/frameworks/intel/main.bro </scripts/base/frameworks/intel/main.bro>`
+
+Summary
+~~~~~~~
+Redefinable Options
+###################
+=============================================================== ==============================================
+:bro:id:`Intel::read_files`: :bro:type:`set` :bro:attr:`&redef` Intelligence files that will be read off disk.
+=============================================================== ==============================================
+
+
+Detailed Interface
+~~~~~~~~~~~~~~~~~~
+Redefinable Options
+###################
+.. bro:id:: Intel::read_files
+
+   :Type: :bro:type:`set` [:bro:type:`string`]
+   :Attributes: :bro:attr:`&redef`
+   :Default: ``{}``
+
+   Intelligence files that will be read off disk. The files are
+   reread every time they are updated so updates must be atomic
+   with "mv" instead of writing the file in place.
+
+

doc/scripts/base/frameworks/intel/main.bro.rst

@@ -0,0 +1,531 @@
+:tocdepth: 3
+
+base/frameworks/intel/main.bro
+==============================
+.. bro:namespace:: Intel
+
+The intelligence framework provides a way to store and query intelligence
+data (e.g. IP addresses, URLs and hashes). The intelligence items can be
+associated with metadata to allow informed decisions about matching and
+handling.
+
+:Namespace: Intel
+:Imports: :doc:`base/frameworks/notice </scripts/base/frameworks/notice/index>`
+
+Summary
+~~~~~~~
+Redefinable Options
+###################
+========================================================================= ==============================================
+:bro:id:`Intel::item_expiration`: :bro:type:`interval` :bro:attr:`&redef` The expiration timeout for intelligence items.
+========================================================================= ==============================================
+
+Types
+#####
+=============================================== ==============================================================
+:bro:type:`Intel::Info`: :bro:type:`record`     Record used for the logging framework representing a positive
+                                                hit within the intelligence framework.
+:bro:type:`Intel::Item`: :bro:type:`record`     Represents a piece of intelligence.
+:bro:type:`Intel::MetaData`: :bro:type:`record` Data about an :bro:type:`Intel::Item`.
+:bro:type:`Intel::Seen`: :bro:type:`record`     Information about a piece of "seen" data.
+:bro:type:`Intel::Type`: :bro:type:`enum`       Enum type to represent various types of intelligence data.
+:bro:type:`Intel::TypeSet`: :bro:type:`set`     Set of intelligence data types.
+:bro:type:`Intel::Where`: :bro:type:`enum`      Enum to represent where data came from when it was discovered.
+=============================================== ==============================================================
+
+Redefinitions
+#############
+===================================== =
+:bro:type:`Log::ID`: :bro:type:`enum` 
+===================================== =
+
+Events
+######
+============================================= ==================================================================
+:bro:id:`Intel::log_intel`: :bro:type:`event` 
+:bro:id:`Intel::match`: :bro:type:`event`     Event to represent a match in the intelligence data from data that
+                                              was seen.
+============================================= ==================================================================
+
+Hooks
+#####
+=============================================== ===================================================================
+:bro:id:`Intel::extend_match`: :bro:type:`hook` This hook can be used to influence the logging of intelligence hits
+                                                (e.g.
+:bro:id:`Intel::item_expired`: :bro:type:`hook` This hook can be used to handle expiration of intelligence items.
+=============================================== ===================================================================
+
+Functions
+#########
+============================================= ==================================================================
+:bro:id:`Intel::insert`: :bro:type:`function` Function to insert intelligence data.
+:bro:id:`Intel::remove`: :bro:type:`function` Function to remove intelligence data.
+:bro:id:`Intel::seen`: :bro:type:`function`   Function to declare discovery of a piece of data in order to check
+                                              it against known intelligence for matches.
+============================================= ==================================================================
+
+
+Detailed Interface
+~~~~~~~~~~~~~~~~~~
+Redefinable Options
+###################
+.. bro:id:: Intel::item_expiration
+
+   :Type: :bro:type:`interval`
+   :Attributes: :bro:attr:`&redef`
+   :Default: ``10.0 mins``
+
+   The expiration timeout for intelligence items. Once an item expires, the
+   :bro:id:`Intel::item_expired` hook is called. Reinsertion of an item 
+   resets the timeout. A negative value disables expiration of intelligence 
+   items.
+
+Types
+#####
+.. bro:type:: Intel::Info
+
+   :Type: :bro:type:`record`
+
+      ts: :bro:type:`time` :bro:attr:`&log`
+         Timestamp when the data was discovered.
+
+      uid: :bro:type:`string` :bro:attr:`&log` :bro:attr:`&optional`
+         If a connection was associated with this intelligence hit,
+         this is the uid for the connection
+
+      id: :bro:type:`conn_id` :bro:attr:`&log` :bro:attr:`&optional`
+         If a connection was associated with this intelligence hit,
+         this is the conn_id for the connection.
+
+      seen: :bro:type:`Intel::Seen` :bro:attr:`&log`
+         Where the data was seen.
+
+      matched: :bro:type:`Intel::TypeSet` :bro:attr:`&log`
+         Which indicator types matched.
+
+      sources: :bro:type:`set` [:bro:type:`string`] :bro:attr:`&log` :bro:attr:`&default` = ``{  }`` :bro:attr:`&optional`
+         Sources which supplied data that resulted in this match.
+
+      fuid: :bro:type:`string` :bro:attr:`&log` :bro:attr:`&optional`
+         (present if :doc:`/scripts/base/frameworks/intel/files.bro` is loaded)
+
+         If a file was associated with this intelligence hit,
+         this is the uid for the file.
+
+      file_mime_type: :bro:type:`string` :bro:attr:`&log` :bro:attr:`&optional`
+         (present if :doc:`/scripts/base/frameworks/intel/files.bro` is loaded)
+
+         A mime type if the intelligence hit is related to a file.
+         If the $f field is provided this will be automatically filled
+         out.
+
+      file_desc: :bro:type:`string` :bro:attr:`&log` :bro:attr:`&optional`
+         (present if :doc:`/scripts/base/frameworks/intel/files.bro` is loaded)
+
+         Frequently files can be "described" to give a bit more context.
+         If the $f field is provided this field will be automatically
+         filled out.
+
+   Record used for the logging framework representing a positive
+   hit within the intelligence framework.
+
+.. bro:type:: Intel::Item
+
+   :Type: :bro:type:`record`
+
+      indicator: :bro:type:`string`
+         The intelligence indicator.
+
+      indicator_type: :bro:type:`Intel::Type`
+         The type of data that the indicator field represents.
+
+      meta: :bro:type:`Intel::MetaData`
+         Metadata for the item. Typically represents more deeply
+         descriptive data for a piece of intelligence.
+
+   Represents a piece of intelligence.
+
+.. bro:type:: Intel::MetaData
+
+   :Type: :bro:type:`record`
+
+      source: :bro:type:`string`
+         An arbitrary string value representing the data source. This
+         value is used as unique key to identify a metadata record in
+         the scope of a single intelligence item.
+
+      desc: :bro:type:`string` :bro:attr:`&optional`
+         A freeform description for the data.
+
+      url: :bro:type:`string` :bro:attr:`&optional`
+         A URL for more information about the data.
+
+      do_notice: :bro:type:`bool` :bro:attr:`&default` = ``F`` :bro:attr:`&optional`
+         (present if :doc:`/scripts/policy/frameworks/intel/do_notice.bro` is loaded)
+
+         A boolean value to allow the data itself to represent
+         if the indicator that this metadata is attached to
+         is notice worthy.
+
+      if_in: :bro:type:`Intel::Where` :bro:attr:`&optional`
+         (present if :doc:`/scripts/policy/frameworks/intel/do_notice.bro` is loaded)
+
+         Restrictions on when notices are created to only create
+         them if the *do_notice* field is T and the notice was
+         seen in the indicated location.
+
+      whitelist: :bro:type:`bool` :bro:attr:`&default` = ``F`` :bro:attr:`&optional`
+         (present if :doc:`/scripts/policy/frameworks/intel/whitelist.bro` is loaded)
+
+         A boolean value to indicate whether the item is whitelisted.
+
+      cif_impact: :bro:type:`string` :bro:attr:`&optional`
+         (present if :doc:`/scripts/policy/integration/collective-intel/main.bro` is loaded)
+
+         Maps to the Impact field in the Collective Intelligence Framework.
+
+      cif_severity: :bro:type:`string` :bro:attr:`&optional`
+         (present if :doc:`/scripts/policy/integration/collective-intel/main.bro` is loaded)
+
+         Maps to the Severity field in the Collective Intelligence Framework.
+
+      cif_confidence: :bro:type:`double` :bro:attr:`&optional`
+         (present if :doc:`/scripts/policy/integration/collective-intel/main.bro` is loaded)
+
+         Maps to the Confidence field in the Collective Intelligence Framework.
+
+   Data about an :bro:type:`Intel::Item`.
+
+.. bro:type:: Intel::Seen
+
+   :Type: :bro:type:`record`
+
+      indicator: :bro:type:`string` :bro:attr:`&log` :bro:attr:`&optional`
+         The string if the data is about a string.
+
+      indicator_type: :bro:type:`Intel::Type` :bro:attr:`&log` :bro:attr:`&optional`
+         The type of data that the indicator represents.
+
+      host: :bro:type:`addr` :bro:attr:`&optional`
+         If the indicator type was :bro:enum:`Intel::ADDR`, then this
+         field will be present.
+
+      where: :bro:type:`Intel::Where` :bro:attr:`&log`
+         Where the data was discovered.
+
+      node: :bro:type:`string` :bro:attr:`&optional` :bro:attr:`&log`
+         The name of the node where the match was discovered.
+
+      conn: :bro:type:`connection` :bro:attr:`&optional`
+         If the data was discovered within a connection, the
+         connection record should go here to give context to the data.
+
+      uid: :bro:type:`string` :bro:attr:`&optional`
+         If the data was discovered within a connection, the
+         connection uid should go here to give context to the data.
+         If the *conn* field is provided, this will be automatically
+         filled out.
+
+      f: :bro:type:`fa_file` :bro:attr:`&optional`
+         (present if :doc:`/scripts/base/frameworks/intel/files.bro` is loaded)
+
+         If the data was discovered within a file, the file record
+         should go here to provide context to the data.
+
+      fuid: :bro:type:`string` :bro:attr:`&optional`
+         (present if :doc:`/scripts/base/frameworks/intel/files.bro` is loaded)
+
+         If the data was discovered within a file, the file uid should
+         go here to provide context to the data. If the file record *f*
+         is provided, this will be automatically filled out.
+
+   Information about a piece of "seen" data.
+
+.. bro:type:: Intel::Type
+
+   :Type: :bro:type:`enum`
+
+      .. bro:enum:: Intel::ADDR Intel::Type
+
+         An IP address.
+
+      .. bro:enum:: Intel::SUBNET Intel::Type
+
+         A subnet in CIDR notation.
+
+      .. bro:enum:: Intel::URL Intel::Type
+
+         A complete URL without the prefix ``"http://"``.
+
+      .. bro:enum:: Intel::SOFTWARE Intel::Type
+
+         Software name.
+
+      .. bro:enum:: Intel::EMAIL Intel::Type
+
+         Email address.
+
+      .. bro:enum:: Intel::DOMAIN Intel::Type
+
+         DNS domain name.
+
+      .. bro:enum:: Intel::USER_NAME Intel::Type
+
+         A user name.
+
+      .. bro:enum:: Intel::CERT_HASH Intel::Type
+
+         Certificate SHA-1 hash.
+
+      .. bro:enum:: Intel::PUBKEY_HASH Intel::Type
+
+         Public key MD5 hash. (SSH server host keys are a good example.)
+
+      .. bro:enum:: Intel::FILE_HASH Intel::Type
+
+         (present if :doc:`/scripts/base/frameworks/intel/files.bro` is loaded)
+
+
+         File hash which is non-hash type specific.  It's up to the
+         user to query for any relevant hash types.
+
+      .. bro:enum:: Intel::FILE_NAME Intel::Type
+
+         (present if :doc:`/scripts/base/frameworks/intel/files.bro` is loaded)
+
+
+         File name.  Typically with protocols with definite
+         indications of a file name.
+
+   Enum type to represent various types of intelligence data.
+
+.. bro:type:: Intel::TypeSet
+
+   :Type: :bro:type:`set` [:bro:type:`Intel::Type`]
+
+   Set of intelligence data types.
+
+.. bro:type:: Intel::Where
+
+   :Type: :bro:type:`enum`
+
+      .. bro:enum:: Intel::IN_ANYWHERE Intel::Where
+
+         A catchall value to represent data of unknown provenance.
+
+      .. bro:enum:: Conn::IN_ORIG Intel::Where
+
+         (present if :doc:`/scripts/policy/frameworks/intel/seen/where-locations.bro` is loaded)
+
+
+      .. bro:enum:: Conn::IN_RESP Intel::Where
+
+         (present if :doc:`/scripts/policy/frameworks/intel/seen/where-locations.bro` is loaded)
+
+
+      .. bro:enum:: Files::IN_HASH Intel::Where
+
+         (present if :doc:`/scripts/policy/frameworks/intel/seen/where-locations.bro` is loaded)
+
+
+      .. bro:enum:: Files::IN_NAME Intel::Where
+
+         (present if :doc:`/scripts/policy/frameworks/intel/seen/where-locations.bro` is loaded)
+
+
+      .. bro:enum:: DNS::IN_REQUEST Intel::Where
+
+         (present if :doc:`/scripts/policy/frameworks/intel/seen/where-locations.bro` is loaded)
+
+
+      .. bro:enum:: DNS::IN_RESPONSE Intel::Where
+
+         (present if :doc:`/scripts/policy/frameworks/intel/seen/where-locations.bro` is loaded)
+
+
+      .. bro:enum:: HTTP::IN_HOST_HEADER Intel::Where
+
+         (present if :doc:`/scripts/policy/frameworks/intel/seen/where-locations.bro` is loaded)
+
+
+      .. bro:enum:: HTTP::IN_REFERRER_HEADER Intel::Where
+
+         (present if :doc:`/scripts/policy/frameworks/intel/seen/where-locations.bro` is loaded)
+
+
+      .. bro:enum:: HTTP::IN_USER_AGENT_HEADER Intel::Where
+
+         (present if :doc:`/scripts/policy/frameworks/intel/seen/where-locations.bro` is loaded)
+
+
+      .. bro:enum:: HTTP::IN_X_FORWARDED_FOR_HEADER Intel::Where
+
+         (present if :doc:`/scripts/policy/frameworks/intel/seen/where-locations.bro` is loaded)
+
+
+      .. bro:enum:: HTTP::IN_URL Intel::Where
+
+         (present if :doc:`/scripts/policy/frameworks/intel/seen/where-locations.bro` is loaded)
+
+
+      .. bro:enum:: SMTP::IN_MAIL_FROM Intel::Where
+
+         (present if :doc:`/scripts/policy/frameworks/intel/seen/where-locations.bro` is loaded)
+
+
+      .. bro:enum:: SMTP::IN_RCPT_TO Intel::Where
+
+         (present if :doc:`/scripts/policy/frameworks/intel/seen/where-locations.bro` is loaded)
+
+
+      .. bro:enum:: SMTP::IN_FROM Intel::Where
+
+         (present if :doc:`/scripts/policy/frameworks/intel/seen/where-locations.bro` is loaded)
+
+
+      .. bro:enum:: SMTP::IN_TO Intel::Where
+
+         (present if :doc:`/scripts/policy/frameworks/intel/seen/where-locations.bro` is loaded)
+
+
+      .. bro:enum:: SMTP::IN_CC Intel::Where
+
+         (present if :doc:`/scripts/policy/frameworks/intel/seen/where-locations.bro` is loaded)
+
+
+      .. bro:enum:: SMTP::IN_RECEIVED_HEADER Intel::Where
+
+         (present if :doc:`/scripts/policy/frameworks/intel/seen/where-locations.bro` is loaded)
+
+
+      .. bro:enum:: SMTP::IN_REPLY_TO Intel::Where
+
+         (present if :doc:`/scripts/policy/frameworks/intel/seen/where-locations.bro` is loaded)
+
+
+      .. bro:enum:: SMTP::IN_X_ORIGINATING_IP_HEADER Intel::Where
+
+         (present if :doc:`/scripts/policy/frameworks/intel/seen/where-locations.bro` is loaded)
+
+
+      .. bro:enum:: SMTP::IN_MESSAGE Intel::Where
+
+         (present if :doc:`/scripts/policy/frameworks/intel/seen/where-locations.bro` is loaded)
+
+
+      .. bro:enum:: SSH::IN_SERVER_HOST_KEY Intel::Where
+
+         (present if :doc:`/scripts/policy/frameworks/intel/seen/where-locations.bro` is loaded)
+
+
+      .. bro:enum:: SSL::IN_SERVER_NAME Intel::Where
+
+         (present if :doc:`/scripts/policy/frameworks/intel/seen/where-locations.bro` is loaded)
+
+
+      .. bro:enum:: SMTP::IN_HEADER Intel::Where
+
+         (present if :doc:`/scripts/policy/frameworks/intel/seen/where-locations.bro` is loaded)
+
+
+      .. bro:enum:: X509::IN_CERT Intel::Where
+
+         (present if :doc:`/scripts/policy/frameworks/intel/seen/where-locations.bro` is loaded)
+
+
+      .. bro:enum:: SSH::SUCCESSFUL_LOGIN Intel::Where
+
+         (present if :doc:`/scripts/policy/protocols/ssh/detect-bruteforcing.bro` is loaded)
+
+
+         An indicator of the login for the intel framework.
+
+   Enum to represent where data came from when it was discovered.
+   The convention is to prefix the name with ``IN_``.
+
+Events
+######
+.. bro:id:: Intel::log_intel
+
+   :Type: :bro:type:`event` (rec: :bro:type:`Intel::Info`)
+
+
+.. bro:id:: Intel::match
+
+   :Type: :bro:type:`event` (s: :bro:type:`Intel::Seen`, items: :bro:type:`set` [:bro:type:`Intel::Item`])
+
+   Event to represent a match in the intelligence data from data that
+   was seen. On clusters there is no assurance as to when this event
+   will be generated so do not assume that arbitrary global state beyond
+   the given data will be available.
+   
+   This is the primary mechanism where a user may take actions based on
+   data provided by the intelligence framework.
+
+Hooks
+#####
+.. bro:id:: Intel::extend_match
+
+   :Type: :bro:type:`hook` (info: :bro:type:`Intel::Info`, s: :bro:type:`Intel::Seen`, items: :bro:type:`set` [:bro:type:`Intel::Item`]) : :bro:type:`bool`
+
+   This hook can be used to influence the logging of intelligence hits
+   (e.g. by adding data to the Info record). The default information is
+   added with a priority of 5.
+   
+
+   :info: The Info record that will be logged.
+   
+
+   :s: Information about the data seen.
+   
+
+   :items: The intel items that match the seen data.
+   
+   In case the hook execution is terminated using break, the match will
+   not be logged.
+
+.. bro:id:: Intel::item_expired
+
+   :Type: :bro:type:`hook` (indicator: :bro:type:`string`, indicator_type: :bro:type:`Intel::Type`, metas: :bro:type:`set` [:bro:type:`Intel::MetaData`]) : :bro:type:`bool`
+
+   This hook can be used to handle expiration of intelligence items.
+   
+
+   :indicator: The indicator of the expired item.
+   
+
+   :indicator_type: The indicator type of the expired item.
+   
+
+   :metas: The set of metadata describing the expired item.
+   
+   If all hook handlers are executed, the expiration timeout will be reset.
+   Otherwise, if one of the handlers terminates using break, the item will
+   be removed.
+
+Functions
+#########
+.. bro:id:: Intel::insert
+
+   :Type: :bro:type:`function` (item: :bro:type:`Intel::Item`) : :bro:type:`void`
+
+   Function to insert intelligence data. If the indicator is already
+   present, the associated metadata will be added to the indicator. If
+   the indicator already contains a metadata record from the same source,
+   the existing metadata record will be updated.
+
+.. bro:id:: Intel::remove
+
+   :Type: :bro:type:`function` (item: :bro:type:`Intel::Item`, purge_indicator: :bro:type:`bool` :bro:attr:`&default` = ``F`` :bro:attr:`&optional`) : :bro:type:`void`
+
+   Function to remove intelligence data. If purge_indicator is set, the
+   given metadata is ignored and the indicator is removed completely.
+
+.. bro:id:: Intel::seen
+
+   :Type: :bro:type:`function` (s: :bro:type:`Intel::Seen`) : :bro:type:`void`
+
+   Function to declare discovery of a piece of data in order to check
+   it against known intelligence for matches.
+
+

doc/scripts/base/frameworks/logging/__load__.bro.rst

@@ -0,0 +1,14 @@
+:tocdepth: 3
+
+base/frameworks/logging/__load__.bro
+====================================
+
+
+:Imports: :doc:`base/frameworks/logging/main.bro </scripts/base/frameworks/logging/main.bro>`, :doc:`base/frameworks/logging/postprocessors </scripts/base/frameworks/logging/postprocessors/index>`, :doc:`base/frameworks/logging/writers/ascii.bro </scripts/base/frameworks/logging/writers/ascii.bro>`, :doc:`base/frameworks/logging/writers/none.bro </scripts/base/frameworks/logging/writers/none.bro>`, :doc:`base/frameworks/logging/writers/sqlite.bro </scripts/base/frameworks/logging/writers/sqlite.bro>`
+
+Summary
+~~~~~~~
+
+Detailed Interface
+~~~~~~~~~~~~~~~~~~
+

doc/scripts/base/frameworks/logging/index.rst

@@ -0,0 +1,87 @@
+:orphan:
+
+Package: base/frameworks/logging
+================================
+
+The logging framework provides a flexible key-value based logging interface.
+
+:doc:`/scripts/base/frameworks/logging/__load__.bro`
+
+
+:doc:`/scripts/base/frameworks/logging/main.bro`
+
+   The Bro logging interface.
+   
+   See :doc:`/frameworks/logging` for an introduction to Bro's
+   logging framework.
+
+:doc:`/scripts/base/frameworks/logging/postprocessors/__load__.bro`
+
+
+:doc:`/scripts/base/frameworks/logging/postprocessors/scp.bro`
+
+   This script defines a postprocessing function that can be applied
+   to a logging filter in order to automatically SCP (secure copy)
+   a log stream (or a subset of it) to a remote host at configurable
+   rotation time intervals.  Generally, to use this functionality
+   you must handle the :bro:id:`bro_init` event and do the following
+   in your handler:
+   
+   1) Create a new :bro:type:`Log::Filter` record that defines a name/path,
+      rotation interval, and set the ``postprocessor`` to
+      :bro:id:`Log::scp_postprocessor`.
+   2) Add the filter to a logging stream using :bro:id:`Log::add_filter`.
+   3) Add a table entry to :bro:id:`Log::scp_destinations` for the filter's
+      writer/path pair which defines a set of :bro:type:`Log::SCPDestination`
+      records.
+
+:doc:`/scripts/base/frameworks/logging/postprocessors/sftp.bro`
+
+   This script defines a postprocessing function that can be applied
+   to a logging filter in order to automatically SFTP
+   a log stream (or a subset of it) to a remote host at configurable
+   rotation time intervals.  Generally, to use this functionality
+   you must handle the :bro:id:`bro_init` event and do the following
+   in your handler:
+   
+   1) Create a new :bro:type:`Log::Filter` record that defines a name/path,
+      rotation interval, and set the ``postprocessor`` to
+      :bro:id:`Log::sftp_postprocessor`.
+   2) Add the filter to a logging stream using :bro:id:`Log::add_filter`.
+   3) Add a table entry to :bro:id:`Log::sftp_destinations` for the filter's
+      writer/path pair which defines a set of :bro:type:`Log::SFTPDestination`
+      records.
+
+:doc:`/scripts/base/frameworks/logging/writers/ascii.bro`
+
+   Interface for the ASCII log writer.  Redefinable options are available
+   to tweak the output format of ASCII logs.
+   
+   The ASCII writer currently supports one writer-specific per-filter config
+   option: setting ``tsv`` to the string ``T`` turns the output into
+   "tab-separated-value" mode where only a single header row with the column
+   names is printed out as meta information, with no "# fields" prepended; no
+   other meta data gets included in that mode.  Example filter using this::
+   
+      local f: Log::Filter = [$name = "my-filter",
+                              $writer = Log::WRITER_ASCII,
+                              $config = table(["tsv"] = "T")];
+   
+
+:doc:`/scripts/base/frameworks/logging/writers/sqlite.bro`
+
+   Interface for the SQLite log writer. Redefinable options are available
+   to tweak the output format of the SQLite reader.
+   
+   See :doc:`/frameworks/logging-input-sqlite` for an introduction on how to
+   use the SQLite log writer.
+   
+   The SQL writer currently supports one writer-specific filter option via
+   ``config``: setting ``tablename`` sets the name of the table that is used
+   or created in the SQLite database. An example for this is given in the
+   introduction mentioned above.
+
+:doc:`/scripts/base/frameworks/logging/writers/none.bro`
+
+   Interface for the None log writer. This writer is mainly for debugging.
+

doc/scripts/base/frameworks/logging/postprocessors/__load__.bro.rst

@@ -0,0 +1,14 @@
+:tocdepth: 3
+
+base/frameworks/logging/postprocessors/__load__.bro
+===================================================
+
+
+:Imports: :doc:`base/frameworks/logging/postprocessors/scp.bro </scripts/base/frameworks/logging/postprocessors/scp.bro>`, :doc:`base/frameworks/logging/postprocessors/sftp.bro </scripts/base/frameworks/logging/postprocessors/sftp.bro>`
+
+Summary
+~~~~~~~
+
+Detailed Interface
+~~~~~~~~~~~~~~~~~~
+

doc/scripts/base/frameworks/logging/postprocessors/index.rst

@@ -0,0 +1,44 @@
+:orphan:
+
+Package: base/frameworks/logging/postprocessors
+===============================================
+
+Support for postprocessors in the logging framework.
+
+:doc:`/scripts/base/frameworks/logging/postprocessors/__load__.bro`
+
+
+:doc:`/scripts/base/frameworks/logging/postprocessors/scp.bro`
+
+   This script defines a postprocessing function that can be applied
+   to a logging filter in order to automatically SCP (secure copy)
+   a log stream (or a subset of it) to a remote host at configurable
+   rotation time intervals.  Generally, to use this functionality
+   you must handle the :bro:id:`bro_init` event and do the following
+   in your handler:
+   
+   1) Create a new :bro:type:`Log::Filter` record that defines a name/path,
+      rotation interval, and set the ``postprocessor`` to
+      :bro:id:`Log::scp_postprocessor`.
+   2) Add the filter to a logging stream using :bro:id:`Log::add_filter`.
+   3) Add a table entry to :bro:id:`Log::scp_destinations` for the filter's
+      writer/path pair which defines a set of :bro:type:`Log::SCPDestination`
+      records.
+
+:doc:`/scripts/base/frameworks/logging/postprocessors/sftp.bro`
+
+   This script defines a postprocessing function that can be applied
+   to a logging filter in order to automatically SFTP
+   a log stream (or a subset of it) to a remote host at configurable
+   rotation time intervals.  Generally, to use this functionality
+   you must handle the :bro:id:`bro_init` event and do the following
+   in your handler:
+   
+   1) Create a new :bro:type:`Log::Filter` record that defines a name/path,
+      rotation interval, and set the ``postprocessor`` to
+      :bro:id:`Log::sftp_postprocessor`.
+   2) Add the filter to a logging stream using :bro:id:`Log::add_filter`.
+   3) Add a table entry to :bro:id:`Log::sftp_destinations` for the filter's
+      writer/path pair which defines a set of :bro:type:`Log::SFTPDestination`
+      records.
+

doc/scripts/base/frameworks/logging/postprocessors/scp.bro.rst

@@ -0,0 +1,121 @@
+:tocdepth: 3
+
+base/frameworks/logging/postprocessors/scp.bro
+==============================================
+.. bro:namespace:: Log
+
+This script defines a postprocessing function that can be applied
+to a logging filter in order to automatically SCP (secure copy)
+a log stream (or a subset of it) to a remote host at configurable
+rotation time intervals.  Generally, to use this functionality
+you must handle the :bro:id:`bro_init` event and do the following
+in your handler:
+
+1) Create a new :bro:type:`Log::Filter` record that defines a name/path,
+   rotation interval, and set the ``postprocessor`` to
+   :bro:id:`Log::scp_postprocessor`.
+2) Add the filter to a logging stream using :bro:id:`Log::add_filter`.
+3) Add a table entry to :bro:id:`Log::scp_destinations` for the filter's
+   writer/path pair which defines a set of :bro:type:`Log::SCPDestination`
+   records.
+
+:Namespace: Log
+
+Summary
+~~~~~~~
+Redefinable Options
+###################
+============================================================================== ================================================================
+:bro:id:`Log::scp_rotation_date_format`: :bro:type:`string` :bro:attr:`&redef` Default naming format for timestamps embedded into log filenames
+                                                                               that use the SCP rotator.
+============================================================================== ================================================================
+
+State Variables
+###############
+================================================== =======================================================================
+:bro:id:`Log::scp_destinations`: :bro:type:`table` A table indexed by a particular log writer and filter path, that yields
+                                                   a set of remote destinations.
+================================================== =======================================================================
+
+Types
+#####
+=================================================== =====================================================================
+:bro:type:`Log::SCPDestination`: :bro:type:`record` A container that describes the remote destination for the SCP command
+                                                    argument as ``user@host:path``.
+=================================================== =====================================================================
+
+Functions
+#########
+====================================================== ===========================================================
+:bro:id:`Log::scp_postprocessor`: :bro:type:`function` Secure-copies the rotated log to all the remote hosts
+                                                       defined in :bro:id:`Log::scp_destinations` and then deletes
+                                                       the local copy of the rotated log.
+====================================================== ===========================================================
+
+
+Detailed Interface
+~~~~~~~~~~~~~~~~~~
+Redefinable Options
+###################
+.. bro:id:: Log::scp_rotation_date_format
+
+   :Type: :bro:type:`string`
+   :Attributes: :bro:attr:`&redef`
+   :Default: ``"%Y-%m-%d-%H-%M-%S"``
+
+   Default naming format for timestamps embedded into log filenames
+   that use the SCP rotator.
+
+State Variables
+###############
+.. bro:id:: Log::scp_destinations
+
+   :Type: :bro:type:`table` [:bro:type:`Log::Writer`, :bro:type:`string`] of :bro:type:`set` [:bro:type:`Log::SCPDestination`]
+   :Default: ``{}``
+
+   A table indexed by a particular log writer and filter path, that yields
+   a set of remote destinations.  The :bro:id:`Log::scp_postprocessor`
+   function queries this table upon log rotation and performs a secure
+   copy of the rotated log to each destination in the set.  This
+   table can be modified at run-time.
+
+Types
+#####
+.. bro:type:: Log::SCPDestination
+
+   :Type: :bro:type:`record`
+
+      user: :bro:type:`string`
+         The remote user to log in as.  A trust mechanism should be
+         pre-established.
+
+      host: :bro:type:`string`
+         The remote host to which to transfer logs.
+
+      path: :bro:type:`string`
+         The path/directory on the remote host to send logs.
+
+   A container that describes the remote destination for the SCP command
+   argument as ``user@host:path``.
+
+Functions
+#########
+.. bro:id:: Log::scp_postprocessor
+
+   :Type: :bro:type:`function` (info: :bro:type:`Log::RotationInfo`) : :bro:type:`bool`
+
+   Secure-copies the rotated log to all the remote hosts
+   defined in :bro:id:`Log::scp_destinations` and then deletes
+   the local copy of the rotated log.  It's not active when
+   reading from trace files.
+   
+
+   :info: A record holding meta-information about the log file to be
+         postprocessed.
+   
+
+   :returns: True if secure-copy system command was initiated or
+            if no destination was configured for the log as described
+            by *info*.
+
+

doc/scripts/base/frameworks/logging/postprocessors/sftp.bro.rst

@@ -0,0 +1,124 @@
+:tocdepth: 3
+
+base/frameworks/logging/postprocessors/sftp.bro
+===============================================
+.. bro:namespace:: Log
+
+This script defines a postprocessing function that can be applied
+to a logging filter in order to automatically SFTP
+a log stream (or a subset of it) to a remote host at configurable
+rotation time intervals.  Generally, to use this functionality
+you must handle the :bro:id:`bro_init` event and do the following
+in your handler:
+
+1) Create a new :bro:type:`Log::Filter` record that defines a name/path,
+   rotation interval, and set the ``postprocessor`` to
+   :bro:id:`Log::sftp_postprocessor`.
+2) Add the filter to a logging stream using :bro:id:`Log::add_filter`.
+3) Add a table entry to :bro:id:`Log::sftp_destinations` for the filter's
+   writer/path pair which defines a set of :bro:type:`Log::SFTPDestination`
+   records.
+
+:Namespace: Log
+
+Summary
+~~~~~~~
+Redefinable Options
+###################
+=============================================================================== ================================================================
+:bro:id:`Log::sftp_rotation_date_format`: :bro:type:`string` :bro:attr:`&redef` Default naming format for timestamps embedded into log filenames
+                                                                                that use the SFTP rotator.
+=============================================================================== ================================================================
+
+State Variables
+###############
+=================================================== =======================================================================
+:bro:id:`Log::sftp_destinations`: :bro:type:`table` A table indexed by a particular log writer and filter path, that yields
+                                                    a set of remote destinations.
+=================================================== =======================================================================
+
+Types
+#####
+==================================================== =======================================================================
+:bro:type:`Log::SFTPDestination`: :bro:type:`record` A container that describes the remote destination for the SFTP command,
+                                                     comprised of the username, host, and path at which to upload the file.
+==================================================== =======================================================================
+
+Functions
+#########
+======================================================= ============================================================
+:bro:id:`Log::sftp_postprocessor`: :bro:type:`function` Securely transfers the rotated log to all the remote hosts
+                                                        defined in :bro:id:`Log::sftp_destinations` and then deletes
+                                                        the local copy of the rotated log.
+======================================================= ============================================================
+
+
+Detailed Interface
+~~~~~~~~~~~~~~~~~~
+Redefinable Options
+###################
+.. bro:id:: Log::sftp_rotation_date_format
+
+   :Type: :bro:type:`string`
+   :Attributes: :bro:attr:`&redef`
+   :Default: ``"%Y-%m-%d-%H-%M-%S"``
+
+   Default naming format for timestamps embedded into log filenames
+   that use the SFTP rotator.
+
+State Variables
+###############
+.. bro:id:: Log::sftp_destinations
+
+   :Type: :bro:type:`table` [:bro:type:`Log::Writer`, :bro:type:`string`] of :bro:type:`set` [:bro:type:`Log::SFTPDestination`]
+   :Default: ``{}``
+
+   A table indexed by a particular log writer and filter path, that yields
+   a set of remote destinations.  The :bro:id:`Log::sftp_postprocessor`
+   function queries this table upon log rotation and performs a secure
+   transfer of the rotated log to each destination in the set.  This
+   table can be modified at run-time.
+
+Types
+#####
+.. bro:type:: Log::SFTPDestination
+
+   :Type: :bro:type:`record`
+
+      user: :bro:type:`string`
+         The remote user to log in as.  A trust mechanism should be
+         pre-established.
+
+      host: :bro:type:`string`
+         The remote host to which to transfer logs.
+
+      host_port: :bro:type:`count` :bro:attr:`&default` = ``22`` :bro:attr:`&optional`
+         The port to connect to. Defaults to 22
+
+      path: :bro:type:`string`
+         The path/directory on the remote host to send logs.
+
+   A container that describes the remote destination for the SFTP command,
+   comprised of the username, host, and path at which to upload the file.
+
+Functions
+#########
+.. bro:id:: Log::sftp_postprocessor
+
+   :Type: :bro:type:`function` (info: :bro:type:`Log::RotationInfo`) : :bro:type:`bool`
+
+   Securely transfers the rotated log to all the remote hosts
+   defined in :bro:id:`Log::sftp_destinations` and then deletes
+   the local copy of the rotated log.  It's not active when
+   reading from trace files.
+   
+
+   :info: A record holding meta-information about the log file to be
+         postprocessed.
+   
+
+   :returns: True if sftp system command was initiated or
+            if no destination was configured for the log as described
+            by *info*.
+
+

doc/scripts/base/frameworks/logging/writers/ascii.bro.rst

@@ -0,0 +1,162 @@
+:tocdepth: 3
+
+base/frameworks/logging/writers/ascii.bro
+=========================================
+.. bro:namespace:: LogAscii
+
+Interface for the ASCII log writer.  Redefinable options are available
+to tweak the output format of ASCII logs.
+
+The ASCII writer currently supports one writer-specific per-filter config
+option: setting ``tsv`` to the string ``T`` turns the output into
+"tab-separated-value" mode where only a single header row with the column
+names is printed out as meta information, with no "# fields" prepended; no
+other meta data gets included in that mode.  Example filter using this::
+
+   local f: Log::Filter = [$name = "my-filter",
+                           $writer = Log::WRITER_ASCII,
+                           $config = table(["tsv"] = "T")];
+
+
+:Namespace: LogAscii
+
+Summary
+~~~~~~~
+Redefinable Options
+###################
+========================================================================================= =====================================================================
+:bro:id:`LogAscii::empty_field`: :bro:type:`string` :bro:attr:`&redef`                    String to use for empty fields.
+:bro:id:`LogAscii::gzip_level`: :bro:type:`count` :bro:attr:`&redef`                      Define the gzip level to compress the logs.
+:bro:id:`LogAscii::include_meta`: :bro:type:`bool` :bro:attr:`&redef`                     If true, include lines with log meta information such as column names
+                                                                                          with types, the values of ASCII logging options that are in use, and
+                                                                                          the time when the file was opened and closed (the latter at the end).
+:bro:id:`LogAscii::json_timestamps`: :bro:type:`JSON::TimestampFormat` :bro:attr:`&redef` Format of timestamps when writing out JSON.
+:bro:id:`LogAscii::meta_prefix`: :bro:type:`string` :bro:attr:`&redef`                    Prefix for lines with meta information.
+:bro:id:`LogAscii::output_to_stdout`: :bro:type:`bool` :bro:attr:`&redef`                 If true, output everything to stdout rather than
+                                                                                          into files.
+:bro:id:`LogAscii::separator`: :bro:type:`string` :bro:attr:`&redef`                      Separator between fields.
+:bro:id:`LogAscii::set_separator`: :bro:type:`string` :bro:attr:`&redef`                  Separator between set elements.
+:bro:id:`LogAscii::unset_field`: :bro:type:`string` :bro:attr:`&redef`                    String to use for an unset &optional field.
+:bro:id:`LogAscii::use_json`: :bro:type:`bool` :bro:attr:`&redef`                         If true, the default will be to write logs in a JSON format.
+========================================================================================= =====================================================================
+
+Redefinitions
+#############
+==================================================================================== =
+:bro:id:`Log::default_rotation_postprocessors`: :bro:type:`table` :bro:attr:`&redef` 
+==================================================================================== =
+
+
+Detailed Interface
+~~~~~~~~~~~~~~~~~~
+Redefinable Options
+###################
+.. bro:id:: LogAscii::empty_field
+
+   :Type: :bro:type:`string`
+   :Attributes: :bro:attr:`&redef`
+   :Default: ``"(empty)"``
+
+   String to use for empty fields. This should be different from
+   *unset_field* to make the output unambiguous.
+   
+   This option is also available as a per-filter ``$config`` option.
+
+.. bro:id:: LogAscii::gzip_level
+
+   :Type: :bro:type:`count`
+   :Attributes: :bro:attr:`&redef`
+   :Default: ``0``
+
+   Define the gzip level to compress the logs.  If 0, then no gzip
+   compression is performed. Enabling compression also changes
+   the log file name extension to include ".gz".
+   
+   This option is also available as a per-filter ``$config`` option.
+
+.. bro:id:: LogAscii::include_meta
+
+   :Type: :bro:type:`bool`
+   :Attributes: :bro:attr:`&redef`
+   :Default: ``T``
+
+   If true, include lines with log meta information such as column names
+   with types, the values of ASCII logging options that are in use, and
+   the time when the file was opened and closed (the latter at the end).
+   
+   If writing in JSON format, this is implicitly disabled.
+
+.. bro:id:: LogAscii::json_timestamps
+
+   :Type: :bro:type:`JSON::TimestampFormat`
+   :Attributes: :bro:attr:`&redef`
+   :Default: ``JSON::TS_EPOCH``
+
+   Format of timestamps when writing out JSON. By default, the JSON
+   formatter will use double values for timestamps which represent the
+   number of seconds from the UNIX epoch.
+   
+   This option is also available as a per-filter ``$config`` option.
+
+.. bro:id:: LogAscii::meta_prefix
+
+   :Type: :bro:type:`string`
+   :Attributes: :bro:attr:`&redef`
+   :Default: ``"#"``
+
+   Prefix for lines with meta information.
+   
+   This option is also available as a per-filter ``$config`` option.
+
+.. bro:id:: LogAscii::output_to_stdout
+
+   :Type: :bro:type:`bool`
+   :Attributes: :bro:attr:`&redef`
+   :Default: ``F``
+
+   If true, output everything to stdout rather than
+   into files. This is primarily for debugging purposes.
+   
+   This option is also available as a per-filter ``$config`` option.
+
+.. bro:id:: LogAscii::separator
+
+   :Type: :bro:type:`string`
+   :Attributes: :bro:attr:`&redef`
+   :Default: ``"\x09"``
+
+   Separator between fields.
+   
+   This option is also available as a per-filter ``$config`` option.
+
+.. bro:id:: LogAscii::set_separator
+
+   :Type: :bro:type:`string`
+   :Attributes: :bro:attr:`&redef`
+   :Default: ``","``
+
+   Separator between set elements.
+   
+   This option is also available as a per-filter ``$config`` option.
+
+.. bro:id:: LogAscii::unset_field
+
+   :Type: :bro:type:`string`
+   :Attributes: :bro:attr:`&redef`
+   :Default: ``"-"``
+
+   String to use for an unset &optional field.
+   
+   This option is also available as a per-filter ``$config`` option.
+
+.. bro:id:: LogAscii::use_json
+
+   :Type: :bro:type:`bool`
+   :Attributes: :bro:attr:`&redef`
+   :Default: ``T``
+
+   If true, the default will be to write logs in a JSON format.
+   
+   This option is also available as a per-filter ``$config`` option.
+
+

doc/scripts/base/frameworks/logging/writers/none.bro.rst

@@ -0,0 +1,40 @@
+:tocdepth: 3
+
+base/frameworks/logging/writers/none.bro
+========================================
+.. bro:namespace:: LogNone
+
+Interface for the None log writer. This writer is mainly for debugging.
+
+:Namespace: LogNone
+
+Summary
+~~~~~~~
+Redefinable Options
+###################
+============================================================= ============================================================
+:bro:id:`LogNone::debug`: :bro:type:`bool` :bro:attr:`&redef` If true, output debugging output that can be useful for unit
+                                                              testing the logging framework.
+============================================================= ============================================================
+
+Redefinitions
+#############
+==================================================================================== =
+:bro:id:`Log::default_rotation_postprocessors`: :bro:type:`table` :bro:attr:`&redef` 
+==================================================================================== =
+
+
+Detailed Interface
+~~~~~~~~~~~~~~~~~~
+Redefinable Options
+###################
+.. bro:id:: LogNone::debug
+
+   :Type: :bro:type:`bool`
+   :Attributes: :bro:attr:`&redef`
+   :Default: ``F``
+
+   If true, output debugging output that can be useful for unit
+   testing the logging framework.
+
+

doc/scripts/base/frameworks/logging/writers/sqlite.bro.rst

@@ -0,0 +1,60 @@
+:tocdepth: 3
+
+base/frameworks/logging/writers/sqlite.bro
+==========================================
+.. bro:namespace:: LogSQLite
+
+Interface for the SQLite log writer. Redefinable options are available
+to tweak the output format of the SQLite reader.
+
+See :doc:`/frameworks/logging-input-sqlite` for an introduction on how to
+use the SQLite log writer.
+
+The SQL writer currently supports one writer-specific filter option via
+``config``: setting ``tablename`` sets the name of the table that is used
+or created in the SQLite database. An example for this is given in the
+introduction mentioned above.
+
+:Namespace: LogSQLite
+
+Summary
+~~~~~~~
+Redefinable Options
+###################
+========================================================================= ===========================================
+:bro:id:`LogSQLite::empty_field`: :bro:type:`string` :bro:attr:`&redef`   String to use for empty fields.
+:bro:id:`LogSQLite::set_separator`: :bro:type:`string` :bro:attr:`&redef` Separator between set elements.
+:bro:id:`LogSQLite::unset_field`: :bro:type:`string` :bro:attr:`&redef`   String to use for an unset &optional field.
+========================================================================= ===========================================
+
+
+Detailed Interface
+~~~~~~~~~~~~~~~~~~
+Redefinable Options
+###################
+.. bro:id:: LogSQLite::empty_field
+
+   :Type: :bro:type:`string`
+   :Attributes: :bro:attr:`&redef`
+   :Default: ``"(empty)"``
+
+   String to use for empty fields. This should be different from
+   *unset_field* to make the output unambiguous.
+
+.. bro:id:: LogSQLite::set_separator
+
+   :Type: :bro:type:`string`
+   :Attributes: :bro:attr:`&redef`
+   :Default: ``","``
+
+   Separator between set elements.
+
+.. bro:id:: LogSQLite::unset_field
+
+   :Type: :bro:type:`string`
+   :Attributes: :bro:attr:`&redef`
+   :Default: ``"-"``
+
+   String to use for an unset &optional field.
+
+

doc/scripts/base/frameworks/netcontrol/__load__.bro.rst

@@ -0,0 +1,14 @@
+:tocdepth: 3
+
+base/frameworks/netcontrol/__load__.bro
+=======================================
+
+
+:Imports: :doc:`base/frameworks/cluster </scripts/base/frameworks/cluster/index>`, :doc:`base/frameworks/netcontrol/catch-and-release.bro </scripts/base/frameworks/netcontrol/catch-and-release.bro>`, :doc:`base/frameworks/netcontrol/drop.bro </scripts/base/frameworks/netcontrol/drop.bro>`, :doc:`base/frameworks/netcontrol/main.bro </scripts/base/frameworks/netcontrol/main.bro>`, :doc:`base/frameworks/netcontrol/non-cluster.bro </scripts/base/frameworks/netcontrol/non-cluster.bro>`, :doc:`base/frameworks/netcontrol/plugins </scripts/base/frameworks/netcontrol/plugins/index>`, :doc:`base/frameworks/netcontrol/shunt.bro </scripts/base/frameworks/netcontrol/shunt.bro>`, :doc:`base/frameworks/netcontrol/types.bro </scripts/base/frameworks/netcontrol/types.bro>`
+
+Summary
+~~~~~~~
+
+Detailed Interface
+~~~~~~~~~~~~~~~~~~
+

doc/scripts/base/frameworks/netcontrol/catch-and-release.bro.rst

@@ -0,0 +1,343 @@
+:tocdepth: 3
+
+base/frameworks/netcontrol/catch-and-release.bro
+================================================
+.. bro:namespace:: NetControl
+
+Implementation of catch-and-release functionality for NetControl.
+
+:Namespace: NetControl
+:Imports: :doc:`base/frameworks/cluster </scripts/base/frameworks/cluster/index>`, :doc:`base/frameworks/netcontrol/drop.bro </scripts/base/frameworks/netcontrol/drop.bro>`, :doc:`base/frameworks/netcontrol/main.bro </scripts/base/frameworks/netcontrol/main.bro>`
+
+Summary
+~~~~~~~
+Runtime Options
+###############
+==================================================================================================== ====================================================================================
+:bro:id:`NetControl::catch_release_warn_blocked_ip_encountered`: :bro:type:`bool` :bro:attr:`&redef` If true, catch and release warns if packets of an IP address are still seen after it
+                                                                                                     should have been blocked.
+==================================================================================================== ====================================================================================
+
+Redefinable Options
+###################
+==================================================================================== =====================================================================================
+:bro:id:`NetControl::catch_release_intervals`: :bro:type:`vector` :bro:attr:`&redef` Time intervals for which subsequent drops of the same IP take
+                                                                                     effect.
+:bro:id:`NetControl::watch_connections`: :bro:type:`bool` :bro:attr:`&redef`         If true, catch_release_seen is called on the connection originator in new_connection,
+                                                                                     connection_established, partial_connection, connection_attempt, connection_rejected,
+                                                                                     connection_reset and connection_pending
+==================================================================================== =====================================================================================
+
+Types
+#####
+============================================================= =========================================================================
+:bro:type:`NetControl::BlockInfo`: :bro:type:`record`         This record is used for storing information about current blocks that are
+                                                              part of catch and release.
+:bro:type:`NetControl::CatchReleaseActions`: :bro:type:`enum` The enum that contains the different kinds of messages that are logged by
+                                                              catch and release.
+:bro:type:`NetControl::CatchReleaseInfo`: :bro:type:`record`  The record type that is used for representing and logging
+============================================================= =========================================================================
+
+Redefinitions
+#############
+===================================== =
+:bro:type:`Log::ID`: :bro:type:`enum` 
+===================================== =
+
+Events
+######
+===================================================================== ===================================================================================
+:bro:id:`NetControl::catch_release_add`: :bro:type:`event`            
+:bro:id:`NetControl::catch_release_block_delete`: :bro:type:`event`   
+:bro:id:`NetControl::catch_release_block_new`: :bro:type:`event`      
+:bro:id:`NetControl::catch_release_delete`: :bro:type:`event`         
+:bro:id:`NetControl::catch_release_encountered`: :bro:type:`event`    
+:bro:id:`NetControl::catch_release_forgotten`: :bro:type:`event`      Event is raised when catch and release cases management of an IP address because no
+                                                                      activity was seen within the watch_until period.
+:bro:id:`NetControl::log_netcontrol_catch_release`: :bro:type:`event` Event that can be handled to access the :bro:type:`NetControl::CatchReleaseInfo`
+                                                                      record as it is sent on to the logging framework.
+===================================================================== ===================================================================================
+
+Functions
+#########
+========================================================================= ======================================================================================================
+:bro:id:`NetControl::catch_release_seen`: :bro:type:`function`            This function can be called to notify the catch and release script that activity by
+                                                                          an IP address was seen.
+:bro:id:`NetControl::drop_address_catch_release`: :bro:type:`function`    Stops all packets involving an IP address from being forwarded.
+:bro:id:`NetControl::get_catch_release_info`: :bro:type:`function`        Get the :bro:see:`NetControl::BlockInfo` record for an address currently blocked by catch and release.
+:bro:id:`NetControl::unblock_address_catch_release`: :bro:type:`function` Removes an address from being watched with catch and release.
+========================================================================= ======================================================================================================
+
+
+Detailed Interface
+~~~~~~~~~~~~~~~~~~
+Runtime Options
+###############
+.. bro:id:: NetControl::catch_release_warn_blocked_ip_encountered
+
+   :Type: :bro:type:`bool`
+   :Attributes: :bro:attr:`&redef`
+   :Default: ``F``
+
+   If true, catch and release warns if packets of an IP address are still seen after it
+   should have been blocked.
+
+Redefinable Options
+###################
+.. bro:id:: NetControl::catch_release_intervals
+
+   :Type: :bro:type:`vector` of :bro:type:`interval`
+   :Attributes: :bro:attr:`&redef`
+   :Default:
+
+   ::
+
+      [10.0 mins, 1.0 hr, 1.0 day, 7.0 days]
+
+   Time intervals for which subsequent drops of the same IP take
+   effect.
+
+.. bro:id:: NetControl::watch_connections
+
+   :Type: :bro:type:`bool`
+   :Attributes: :bro:attr:`&redef`
+   :Default: ``T``
+
+   If true, catch_release_seen is called on the connection originator in new_connection,
+   connection_established, partial_connection, connection_attempt, connection_rejected,
+   connection_reset and connection_pending
+
+Types
+#####
+.. bro:type:: NetControl::BlockInfo
+
+   :Type: :bro:type:`record`
+
+      block_until: :bro:type:`time` :bro:attr:`&optional`
+         Absolute time indicating until when a block is inserted using NetControl.
+
+      watch_until: :bro:type:`time`
+         Absolute time indicating until when an IP address is watched to reblock it.
+
+      num_reblocked: :bro:type:`count` :bro:attr:`&default` = ``0`` :bro:attr:`&optional`
+         Number of times an IP address was reblocked.
+
+      current_interval: :bro:type:`count`
+         Number indicating at which catch and release interval we currently are.
+
+      current_block_id: :bro:type:`string`
+         ID of the inserted block, if any.
+
+      location: :bro:type:`string` :bro:attr:`&optional`
+         User specified string.
+
+   This record is used for storing information about current blocks that are
+   part of catch and release.
+
+.. bro:type:: NetControl::CatchReleaseActions
+
+   :Type: :bro:type:`enum`
+
+      .. bro:enum:: NetControl::INFO NetControl::CatchReleaseActions
+
+         Log lines marked with info are purely informational; no action was taken.
+
+      .. bro:enum:: NetControl::ADDED NetControl::CatchReleaseActions
+
+         A rule for the specified IP address already existed in NetControl (outside
+         of catch-and-release). Catch and release did not add a new rule, but is now
+         watching the IP address and will add a new rule after the current rule expires.
+
+      .. bro:enum:: NetControl::DROP NetControl::CatchReleaseActions
+
+         (present if :doc:`/scripts/base/frameworks/netcontrol/types.bro` is loaded)
+
+
+         Stop forwarding all packets matching the entity.
+         
+         No additional arguments.
+
+      .. bro:enum:: NetControl::DROPPED NetControl::CatchReleaseActions
+
+         A drop was requested by catch and release.
+         An address was successfully blocked by catch and release.
+
+      .. bro:enum:: NetControl::UNBLOCK NetControl::CatchReleaseActions
+
+         An address was unblocked after the timeout expired.
+
+      .. bro:enum:: NetControl::FORGOTTEN NetControl::CatchReleaseActions
+
+         An address was forgotten because it did not reappear within the `watch_until` interval.
+
+      .. bro:enum:: NetControl::SEEN_AGAIN NetControl::CatchReleaseActions
+
+         A watched IP address was seen again; catch and release will re-block it.
+
+   The enum that contains the different kinds of messages that are logged by
+   catch and release.
+
+.. bro:type:: NetControl::CatchReleaseInfo
+
+   :Type: :bro:type:`record`
+
+      ts: :bro:type:`time` :bro:attr:`&log`
+         The absolute time indicating when the action for this log-line occured.
+
+      rule_id: :bro:type:`string` :bro:attr:`&log` :bro:attr:`&optional`
+         The rule id that this log line refers to.
+
+      ip: :bro:type:`addr` :bro:attr:`&log`
+         The IP address that this line refers to.
+
+      action: :bro:type:`NetControl::CatchReleaseActions` :bro:attr:`&log`
+         The action that was taken in this log-line.
+
+      block_interval: :bro:type:`interval` :bro:attr:`&log` :bro:attr:`&optional`
+         The current block_interaval (for how long the address is blocked).
+
+      watch_interval: :bro:type:`interval` :bro:attr:`&log` :bro:attr:`&optional`
+         The current watch_interval (for how long the address will be watched and re-block if it reappears).
+
+      blocked_until: :bro:type:`time` :bro:attr:`&log` :bro:attr:`&optional`
+         The absolute time until which the address is blocked.
+
+      watched_until: :bro:type:`time` :bro:attr:`&log` :bro:attr:`&optional`
+         The absolute time until which the address will be monitored.
+
+      num_blocked: :bro:type:`count` :bro:attr:`&log` :bro:attr:`&optional`
+         Number of times that this address was blocked in the current cycle.
+
+      location: :bro:type:`string` :bro:attr:`&log` :bro:attr:`&optional`
+         The user specified location string.
+
+      message: :bro:type:`string` :bro:attr:`&log` :bro:attr:`&optional`
+         Additional informational string by the catch and release framework about this log-line.
+
+   The record type that is used for representing and logging
+
+Events
+######
+.. bro:id:: NetControl::catch_release_add
+
+   :Type: :bro:type:`event` (a: :bro:type:`addr`, location: :bro:type:`string`)
+
+
+.. bro:id:: NetControl::catch_release_block_delete
+
+   :Type: :bro:type:`event` (a: :bro:type:`addr`)
+
+
+.. bro:id:: NetControl::catch_release_block_new
+
+   :Type: :bro:type:`event` (a: :bro:type:`addr`, b: :bro:type:`NetControl::BlockInfo`)
+
+
+.. bro:id:: NetControl::catch_release_delete
+
+   :Type: :bro:type:`event` (a: :bro:type:`addr`, reason: :bro:type:`string`)
+
+
+.. bro:id:: NetControl::catch_release_encountered
+
+   :Type: :bro:type:`event` (a: :bro:type:`addr`)
+
+
+.. bro:id:: NetControl::catch_release_forgotten
+
+   :Type: :bro:type:`event` (a: :bro:type:`addr`, bi: :bro:type:`NetControl::BlockInfo`)
+
+   Event is raised when catch and release cases management of an IP address because no
+   activity was seen within the watch_until period.
+   
+
+   :a: The address that is no longer being managed.
+   
+
+   :bi: The :bro:see:`NetControl::BlockInfo` record containing information about the block.
+
+.. bro:id:: NetControl::log_netcontrol_catch_release
+
+   :Type: :bro:type:`event` (rec: :bro:type:`NetControl::CatchReleaseInfo`)
+
+   Event that can be handled to access the :bro:type:`NetControl::CatchReleaseInfo`
+   record as it is sent on to the logging framework.
+
+Functions
+#########
+.. bro:id:: NetControl::catch_release_seen
+
+   :Type: :bro:type:`function` (a: :bro:type:`addr`) : :bro:type:`void`
+
+   This function can be called to notify the catch and release script that activity by
+   an IP address was seen. If the respective IP address is currently monitored by catch and
+   release and not blocked, the block will be reinstated. See the documentation of watch_new_connection
+   which events the catch and release functionality usually monitors for activity.
+   
+
+   :a: The address that was seen and should be re-dropped if it is being watched.
+
+.. bro:id:: NetControl::drop_address_catch_release
+
+   :Type: :bro:type:`function` (a: :bro:type:`addr`, location: :bro:type:`string` :bro:attr:`&default` = ``""`` :bro:attr:`&optional`) : :bro:type:`NetControl::BlockInfo`
+
+   Stops all packets involving an IP address from being forwarded. This function
+   uses catch-and-release functionality, where the IP address is only dropped for
+   a short amount of time that is incremented steadily when the IP is encountered
+   again.
+   
+   In cluster mode, this function works on workers as well as the manager. On managers,
+   the returned :bro:see:`NetControl::BlockInfo` record will not contain the block ID,
+   which will be assigned on the manager.
+   
+
+   :a: The address to be dropped.
+   
+
+   :t: How long to drop it, with 0 being indefinitely.
+   
+
+   :location: An optional string describing where the drop was triggered.
+   
+
+   :returns: The :bro:see:`NetControl::BlockInfo` record containing information about
+            the inserted block.
+
+.. bro:id:: NetControl::get_catch_release_info
+
+   :Type: :bro:type:`function` (a: :bro:type:`addr`) : :bro:type:`NetControl::BlockInfo`
+
+   Get the :bro:see:`NetControl::BlockInfo` record for an address currently blocked by catch and release.
+   If the address is unknown to catch and release, the watch_until time will be set to 0.
+   
+   In cluster mode, this function works on the manager and workers. On workers, the data will
+   lag slightly behind the manager; if you add a block, it will not be instantly available via
+   this function.
+   
+
+   :a: The address to get information about.
+   
+
+   :returns: The :bro:see:`NetControl::BlockInfo` record containing information about
+            the inserted block.
+
+.. bro:id:: NetControl::unblock_address_catch_release
+
+   :Type: :bro:type:`function` (a: :bro:type:`addr`, reason: :bro:type:`string` :bro:attr:`&default` = ``""`` :bro:attr:`&optional`) : :bro:type:`bool`
+
+   Removes an address from being watched with catch and release. Returns true if the
+   address was found and removed; returns false if it was unknown to catch and release.
+   
+   If the address is currently blocked, and the block was inserted by catch and release,
+   the block is removed.
+   
+
+   :a: The address to be unblocked.
+   
+
+   :reason: A reason for the unblock.
+   
+
+   :returns: True if the address was unblocked.
+
+

doc/scripts/base/frameworks/netcontrol/drop.bro.rst

@@ -0,0 +1,140 @@
+:tocdepth: 3
+
+base/frameworks/netcontrol/drop.bro
+===================================
+.. bro:namespace:: NetControl
+
+Implementation of the drop functionality for NetControl.
+
+:Namespace: NetControl
+:Imports: :doc:`base/frameworks/netcontrol/main.bro </scripts/base/frameworks/netcontrol/main.bro>`
+
+Summary
+~~~~~~~
+Types
+#####
+==================================================== =
+:bro:type:`NetControl::DropInfo`: :bro:type:`record` 
+==================================================== =
+
+Redefinitions
+#############
+===================================== =
+:bro:type:`Log::ID`: :bro:type:`enum` 
+===================================== =
+
+Events
+######
+============================================================ =========================================================================
+:bro:id:`NetControl::log_netcontrol_drop`: :bro:type:`event` Event that can be handled to access the :bro:type:`NetControl::ShuntInfo`
+                                                             record as it is sent on to the logging framework.
+============================================================ =========================================================================
+
+Hooks
+#####
+======================================================== =======================================================================
+:bro:id:`NetControl::drop_rule_policy`: :bro:type:`hook` Hook that allows the modification of rules passed to drop_* before they
+                                                         are passed on.
+======================================================== =======================================================================
+
+Functions
+#########
+=========================================================== ======================================================================
+:bro:id:`NetControl::drop_address`: :bro:type:`function`    Stops all packets involving an IP address from being forwarded.
+:bro:id:`NetControl::drop_connection`: :bro:type:`function` Stops all packets involving a connection address from being forwarded.
+=========================================================== ======================================================================
+
+
+Detailed Interface
+~~~~~~~~~~~~~~~~~~
+Types
+#####
+.. bro:type:: NetControl::DropInfo
+
+   :Type: :bro:type:`record`
+
+      ts: :bro:type:`time` :bro:attr:`&log`
+         Time at which the recorded activity occurred.
+
+      rule_id: :bro:type:`string` :bro:attr:`&log`
+         ID of the rule; unique during each Bro run.
+
+      orig_h: :bro:type:`addr` :bro:attr:`&log`
+         The originator's IP address.
+
+      orig_p: :bro:type:`port` :bro:attr:`&log` :bro:attr:`&optional`
+         The originator's port number.
+
+      resp_h: :bro:type:`addr` :bro:attr:`&log` :bro:attr:`&optional`
+         The responder's IP address.
+
+      resp_p: :bro:type:`port` :bro:attr:`&log` :bro:attr:`&optional`
+         The responder's port number.
+
+      expire: :bro:type:`interval` :bro:attr:`&log`
+         Expiry time of the shunt.
+
+      location: :bro:type:`string` :bro:attr:`&log` :bro:attr:`&optional`
+         Location where the underlying action was triggered.
+
+
+Events
+######
+.. bro:id:: NetControl::log_netcontrol_drop
+
+   :Type: :bro:type:`event` (rec: :bro:type:`NetControl::DropInfo`)
+
+   Event that can be handled to access the :bro:type:`NetControl::ShuntInfo`
+   record as it is sent on to the logging framework.
+
+Hooks
+#####
+.. bro:id:: NetControl::drop_rule_policy
+
+   :Type: :bro:type:`hook` (r: :bro:type:`NetControl::Rule`) : :bro:type:`bool`
+
+   Hook that allows the modification of rules passed to drop_* before they
+   are passed on. If one of the hooks uses break, the rule is ignored.
+   
+
+   :r: The rule to be added.
+
+Functions
+#########
+.. bro:id:: NetControl::drop_address
+
+   :Type: :bro:type:`function` (a: :bro:type:`addr`, t: :bro:type:`interval`, location: :bro:type:`string` :bro:attr:`&default` = ``""`` :bro:attr:`&optional`) : :bro:type:`string`
+
+   Stops all packets involving an IP address from being forwarded.
+   
+
+   :a: The address to be dropped.
+   
+
+   :t: How long to drop it, with 0 being indefinitely.
+   
+
+   :location: An optional string describing where the drop was triggered.
+   
+
+   :returns: The id of the inserted rule on success and zero on failure.
+
+.. bro:id:: NetControl::drop_connection
+
+   :Type: :bro:type:`function` (c: :bro:type:`conn_id`, t: :bro:type:`interval`, location: :bro:type:`string` :bro:attr:`&default` = ``""`` :bro:attr:`&optional`) : :bro:type:`string`
+
+   Stops all packets involving a connection address from being forwarded.
+   
+
+   :c: The connection to be dropped.
+   
+
+   :t: How long to drop it, with 0 being indefinitely.
+   
+
+   :location: An optional string describing where the drop was triggered.
+   
+
+   :returns: The id of the inserted rule on success and zero on failure.
+
+

doc/scripts/base/frameworks/netcontrol/index.rst

@@ -0,0 +1,81 @@
+:orphan:
+
+Package: base/frameworks/netcontrol
+===================================
+
+The NetControl framework provides a way for Bro to interact with networking
+hard- and software, e.g. for dropping and shunting IP addresses/connections,
+etc.
+
+:doc:`/scripts/base/frameworks/netcontrol/__load__.bro`
+
+
+:doc:`/scripts/base/frameworks/netcontrol/types.bro`
+
+   This file defines the types that are used by the NetControl framework.
+   
+   The most important type defined in this file is :bro:see:`NetControl::Rule`,
+   which is used to describe all rules that can be expressed by the NetControl framework. 
+
+:doc:`/scripts/base/frameworks/netcontrol/main.bro`
+
+   Bro's NetControl framework.
+   
+   This plugin-based framework allows to control the traffic that Bro monitors
+   as well as, if having access to the forwarding path, the traffic the network
+   forwards. By default, the framework lets everything through, to both Bro
+   itself as well as on the network. Scripts can then add rules to impose
+   restrictions on entities, such as specific connections or IP addresses.
+   
+   This framework has two APIs: a high-level and low-level. The high-level API
+   provides convenience functions for a set of common operations. The
+   low-level API provides full flexibility.
+
+:doc:`/scripts/base/frameworks/netcontrol/plugin.bro`
+
+   This file defines the plugin interface for NetControl.
+
+:doc:`/scripts/base/frameworks/netcontrol/plugins/__load__.bro`
+
+
+:doc:`/scripts/base/frameworks/netcontrol/plugins/debug.bro`
+
+   Debugging plugin for the NetControl framework, providing insight into
+   executed operations.
+
+:doc:`/scripts/base/frameworks/netcontrol/plugins/openflow.bro`
+
+   OpenFlow plugin for the NetControl framework.
+
+:doc:`/scripts/base/frameworks/netcontrol/plugins/packetfilter.bro`
+
+   NetControl plugin for the process-level PacketFilter that comes with
+   Bro. Since the PacketFilter in Bro is quite limited in scope
+   and can only add/remove filters for addresses, this is quite
+   limited in scope at the moment. 
+
+:doc:`/scripts/base/frameworks/netcontrol/plugins/broker.bro`
+
+   Broker plugin for the NetControl framework. Sends the raw data structures
+   used in NetControl on to Broker to allow for easy handling, e.g., of
+   command-line scripts.
+
+:doc:`/scripts/base/frameworks/netcontrol/plugins/acld.bro`
+
+   Acld plugin for the netcontrol framework.
+
+:doc:`/scripts/base/frameworks/netcontrol/drop.bro`
+
+   Implementation of the drop functionality for NetControl.
+
+:doc:`/scripts/base/frameworks/netcontrol/shunt.bro`
+
+   Implementation of the shunt functionality for NetControl.
+
+:doc:`/scripts/base/frameworks/netcontrol/catch-and-release.bro`
+
+   Implementation of catch-and-release functionality for NetControl.
+
+:doc:`/scripts/base/frameworks/netcontrol/non-cluster.bro`
+
+

doc/scripts/base/frameworks/netcontrol/main.bro.rst

@@ -0,0 +1,544 @@
+:tocdepth: 3
+
+base/frameworks/netcontrol/main.bro
+===================================
+.. bro:namespace:: NetControl
+
+Bro's NetControl framework.
+
+This plugin-based framework allows to control the traffic that Bro monitors
+as well as, if having access to the forwarding path, the traffic the network
+forwards. By default, the framework lets everything through, to both Bro
+itself as well as on the network. Scripts can then add rules to impose
+restrictions on entities, such as specific connections or IP addresses.
+
+This framework has two APIs: a high-level and low-level. The high-level API
+provides convenience functions for a set of common operations. The
+low-level API provides full flexibility.
+
+:Namespace: NetControl
+:Imports: :doc:`base/frameworks/netcontrol/plugin.bro </scripts/base/frameworks/netcontrol/plugin.bro>`, :doc:`base/frameworks/netcontrol/types.bro </scripts/base/frameworks/netcontrol/types.bro>`
+
+Summary
+~~~~~~~
+Types
+#####
+====================================================== =================================================================
+:bro:type:`NetControl::Info`: :bro:type:`record`       The record type defining the column fields of the NetControl log.
+:bro:type:`NetControl::InfoCategory`: :bro:type:`enum` Type of an entry in the NetControl log.
+:bro:type:`NetControl::InfoState`: :bro:type:`enum`    State of an entry in the NetControl log.
+====================================================== =================================================================
+
+Redefinitions
+#############
+================================================ ==========================================
+:bro:type:`Log::ID`: :bro:type:`enum`            The framework's logging stream identifier.
+:bro:type:`NetControl::Rule`: :bro:type:`record` 
+================================================ ==========================================
+
+Events
+######
+======================================================= ===========================================================================
+:bro:id:`NetControl::init`: :bro:type:`event`           Event that is used to initialize plugins.
+:bro:id:`NetControl::init_done`: :bro:type:`event`      Event that is raised once all plugins activated in ``NetControl::init``
+                                                        have finished their initialization.
+:bro:id:`NetControl::log_netcontrol`: :bro:type:`event` Event that can be handled to access the :bro:type:`NetControl::Info`
+                                                        record as it is sent on to the logging framework.
+:bro:id:`NetControl::rule_added`: :bro:type:`event`     Confirms that a rule was put in place by a plugin.
+:bro:id:`NetControl::rule_destroyed`: :bro:type:`event` This event is raised when a rule is deleted from the NetControl framework,
+                                                        because it is no longer in use.
+:bro:id:`NetControl::rule_error`: :bro:type:`event`     Reports an error when operating on a rule.
+:bro:id:`NetControl::rule_exists`: :bro:type:`event`    Signals that a rule that was supposed to be put in place was already
+                                                        existing at the specified plugin.
+:bro:id:`NetControl::rule_new`: :bro:type:`event`       This event is raised when a new rule is created by the NetControl framework
+                                                        due to a call to add_rule.
+:bro:id:`NetControl::rule_removed`: :bro:type:`event`   Reports that a plugin reports a rule was removed due to a
+                                                        remove_rule function call.
+:bro:id:`NetControl::rule_timeout`: :bro:type:`event`   Reports that a rule was removed from a plugin due to a timeout.
+======================================================= ===========================================================================
+
+Hooks
+#####
+=================================================== =========================================================================
+:bro:id:`NetControl::rule_policy`: :bro:type:`hook` Hook that allows the modification of rules passed to add_rule before they
+                                                    are passed on to the plugins.
+=================================================== =========================================================================
+
+Functions
+#########
+============================================================= ==============================================================================================
+:bro:id:`NetControl::activate`: :bro:type:`function`          Activates a plugin.
+:bro:id:`NetControl::add_rule`: :bro:type:`function`          Installs a rule.
+:bro:id:`NetControl::clear`: :bro:type:`function`             Flushes all state by calling :bro:see:`NetControl::remove_rule` on all currently active rules.
+:bro:id:`NetControl::delete_rule`: :bro:type:`function`       Deletes a rule without removing it from the backends to which it has been
+                                                              added before.
+:bro:id:`NetControl::find_rules_addr`: :bro:type:`function`   Searches all rules affecting a certain IP address.
+:bro:id:`NetControl::find_rules_subnet`: :bro:type:`function` Searches all rules affecting a certain subnet.
+:bro:id:`NetControl::plugin_activated`: :bro:type:`function`  Function called by plugins once they finished their activation.
+:bro:id:`NetControl::quarantine_host`: :bro:type:`function`   Quarantines a host.
+:bro:id:`NetControl::redirect_flow`: :bro:type:`function`     Redirects a uni-directional flow to another port.
+:bro:id:`NetControl::remove_rule`: :bro:type:`function`       Removes a rule.
+:bro:id:`NetControl::whitelist_address`: :bro:type:`function` Allows all traffic involving a specific IP address to be forwarded.
+:bro:id:`NetControl::whitelist_subnet`: :bro:type:`function`  Allows all traffic involving a specific IP subnet to be forwarded.
+============================================================= ==============================================================================================
+
+
+Detailed Interface
+~~~~~~~~~~~~~~~~~~
+Types
+#####
+.. bro:type:: NetControl::Info
+
+   :Type: :bro:type:`record`
+
+      ts: :bro:type:`time` :bro:attr:`&log`
+         Time at which the recorded activity occurred.
+
+      rule_id: :bro:type:`string` :bro:attr:`&log` :bro:attr:`&optional`
+         ID of the rule; unique during each Bro run.
+
+      category: :bro:type:`NetControl::InfoCategory` :bro:attr:`&log` :bro:attr:`&optional`
+         Type of the log entry.
+
+      cmd: :bro:type:`string` :bro:attr:`&log` :bro:attr:`&optional`
+         The command the log entry is about.
+
+      state: :bro:type:`NetControl::InfoState` :bro:attr:`&log` :bro:attr:`&optional`
+         State the log entry reflects.
+
+      action: :bro:type:`string` :bro:attr:`&log` :bro:attr:`&optional`
+         String describing an action the entry is about.
+
+      target: :bro:type:`NetControl::TargetType` :bro:attr:`&log` :bro:attr:`&optional`
+         The target type of the action.
+
+      entity_type: :bro:type:`string` :bro:attr:`&log` :bro:attr:`&optional`
+         Type of the entity the log entry is about.
+
+      entity: :bro:type:`string` :bro:attr:`&log` :bro:attr:`&optional`
+         String describing the entity the log entry is about.
+
+      mod: :bro:type:`string` :bro:attr:`&log` :bro:attr:`&optional`
+         String describing the optional modification of the entry (e.h. redirect)
+
+      msg: :bro:type:`string` :bro:attr:`&log` :bro:attr:`&optional`
+         String with an additional message.
+
+      priority: :bro:type:`int` :bro:attr:`&log` :bro:attr:`&optional`
+         Number describing the priority of the log entry.
+
+      expire: :bro:type:`interval` :bro:attr:`&log` :bro:attr:`&optional`
+         Expiry time of the log entry.
+
+      location: :bro:type:`string` :bro:attr:`&log` :bro:attr:`&optional`
+         Location where the underlying action was triggered.
+
+      plugin: :bro:type:`string` :bro:attr:`&log` :bro:attr:`&optional`
+         Plugin triggering the log entry.
+
+   The record type defining the column fields of the NetControl log.
+
+.. bro:type:: NetControl::InfoCategory
+
+   :Type: :bro:type:`enum`
+
+      .. bro:enum:: NetControl::MESSAGE NetControl::InfoCategory
+
+         A log entry reflecting a framework message.
+
+      .. bro:enum:: NetControl::ERROR NetControl::InfoCategory
+
+         A log entry reflecting a framework message.
+
+      .. bro:enum:: NetControl::RULE NetControl::InfoCategory
+
+         A log entry about a rule.
+
+   Type of an entry in the NetControl log.
+
+.. bro:type:: NetControl::InfoState
+
+   :Type: :bro:type:`enum`
+
+      .. bro:enum:: NetControl::REQUESTED NetControl::InfoState
+
+         The request to add/remove a rule was sent to the respective backend.
+
+      .. bro:enum:: NetControl::SUCCEEDED NetControl::InfoState
+
+         A rule was successfully added by a backend.
+
+      .. bro:enum:: NetControl::EXISTS NetControl::InfoState
+
+         A backend reported that a rule was already existing.
+
+      .. bro:enum:: NetControl::FAILED NetControl::InfoState
+
+         A rule addition failed.
+
+      .. bro:enum:: NetControl::REMOVED NetControl::InfoState
+
+         A rule was successfully removed by a backend.
+
+      .. bro:enum:: NetControl::TIMEOUT NetControl::InfoState
+
+         A rule timeout was triggered by the NetControl framework or a backend.
+
+   State of an entry in the NetControl log.
+
+Events
+######
+.. bro:id:: NetControl::init
+
+   :Type: :bro:type:`event` ()
+
+   Event that is used to initialize plugins. Place all plugin initialization
+   related functionality in this event.
+
+.. bro:id:: NetControl::init_done
+
+   :Type: :bro:type:`event` ()
+
+   Event that is raised once all plugins activated in ``NetControl::init``
+   have finished their initialization.
+
+.. bro:id:: NetControl::log_netcontrol
+
+   :Type: :bro:type:`event` (rec: :bro:type:`NetControl::Info`)
+
+   Event that can be handled to access the :bro:type:`NetControl::Info`
+   record as it is sent on to the logging framework.
+
+.. bro:id:: NetControl::rule_added
+
+   :Type: :bro:type:`event` (r: :bro:type:`NetControl::Rule`, p: :bro:type:`NetControl::PluginState`, msg: :bro:type:`string` :bro:attr:`&default` = ``""`` :bro:attr:`&optional`)
+
+   Confirms that a rule was put in place by a plugin.
+   
+
+   :r: The rule now in place.
+   
+
+   :p: The state for the plugin that put it into place.
+   
+
+   :msg: An optional informational message by the plugin.
+
+.. bro:id:: NetControl::rule_destroyed
+
+   :Type: :bro:type:`event` (r: :bro:type:`NetControl::Rule`)
+
+   This event is raised when a rule is deleted from the NetControl framework,
+   because it is no longer in use. This can be caused by the fact that a rule
+   was removed by all plugins to which it was added, by the fact that it timed out
+   or due to rule errors.
+   
+   To get the cause of a rule remove, catch the rule_removed, rule_timeout and
+   rule_error events.
+
+.. bro:id:: NetControl::rule_error
+
+   :Type: :bro:type:`event` (r: :bro:type:`NetControl::Rule`, p: :bro:type:`NetControl::PluginState`, msg: :bro:type:`string` :bro:attr:`&default` = ``""`` :bro:attr:`&optional`)
+
+   Reports an error when operating on a rule.
+   
+
+   :r: The rule that encountered an error.
+   
+
+   :p: The state for the plugin that reported the error.
+   
+
+   :msg: An optional informational message by the plugin.
+
+.. bro:id:: NetControl::rule_exists
+
+   :Type: :bro:type:`event` (r: :bro:type:`NetControl::Rule`, p: :bro:type:`NetControl::PluginState`, msg: :bro:type:`string` :bro:attr:`&default` = ``""`` :bro:attr:`&optional`)
+
+   Signals that a rule that was supposed to be put in place was already
+   existing at the specified plugin. Rules that already have been existing
+   continue to be tracked like normal, but no timeout calls will be sent
+   to the specified plugins. Removal of the rule from the hardware can
+   still be forced by manually issuing a remove_rule call.
+   
+
+   :r: The rule that was already in place.
+   
+
+   :p: The plugin that reported that the rule already was in place.
+   
+
+   :msg: An optional informational message by the plugin.
+
+.. bro:id:: NetControl::rule_new
+
+   :Type: :bro:type:`event` (r: :bro:type:`NetControl::Rule`)
+
+   This event is raised when a new rule is created by the NetControl framework
+   due to a call to add_rule. From this moment, until the rule_destroyed event
+   is raised, the rule is tracked internally by the NetControl framework.
+   
+   Note that this event does not mean that a rule was successfully added by
+   any backend; it just means that the rule has been accepted and addition
+   to the specified backend is queued. To get information when rules are actually
+   installed by the hardware, use the rule_added, rule_exists, rule_removed, rule_timeout
+   and rule_error events.
+
+.. bro:id:: NetControl::rule_removed
+
+   :Type: :bro:type:`event` (r: :bro:type:`NetControl::Rule`, p: :bro:type:`NetControl::PluginState`, msg: :bro:type:`string` :bro:attr:`&default` = ``""`` :bro:attr:`&optional`)
+
+   Reports that a plugin reports a rule was removed due to a
+   remove_rule function call.
+   
+
+   :r: The rule now removed.
+   
+
+   :p: The state for the plugin that had the rule in place and now
+      removed it.
+   
+
+   :msg: An optional informational message by the plugin.
+
+.. bro:id:: NetControl::rule_timeout
+
+   :Type: :bro:type:`event` (r: :bro:type:`NetControl::Rule`, i: :bro:type:`NetControl::FlowInfo`, p: :bro:type:`NetControl::PluginState`)
+
+   Reports that a rule was removed from a plugin due to a timeout.
+   
+
+   :r: The rule now removed.
+   
+
+   :i: Additional flow information, if supported by the protocol.
+   
+
+   :p: The state for the plugin that had the rule in place and now
+      removed it.
+   
+
+   :msg: An optional informational message by the plugin.
+
+Hooks
+#####
+.. bro:id:: NetControl::rule_policy
+
+   :Type: :bro:type:`hook` (r: :bro:type:`NetControl::Rule`) : :bro:type:`bool`
+
+   Hook that allows the modification of rules passed to add_rule before they
+   are passed on to the plugins. If one of the hooks uses break, the rule is
+   ignored and not passed on to any plugin.
+   
+
+   :r: The rule to be added.
+
+Functions
+#########
+.. bro:id:: NetControl::activate
+
+   :Type: :bro:type:`function` (p: :bro:type:`NetControl::PluginState`, priority: :bro:type:`int`) : :bro:type:`void`
+
+   Activates a plugin.
+   
+
+   :p: The plugin to activate.
+   
+
+   :priority: The higher the priority, the earlier this plugin will be checked
+             whether it supports an operation, relative to other plugins.
+
+.. bro:id:: NetControl::add_rule
+
+   :Type: :bro:type:`function` (r: :bro:type:`NetControl::Rule`) : :bro:type:`string`
+
+   Installs a rule.
+   
+
+   :r: The rule to install.
+   
+
+   :returns: If successful, returns an ID string unique to the rule that can
+            later be used to refer to it. If unsuccessful, returns an empty
+            string. The ID is also assigned to ``r$id``. Note that
+            "successful" means "a plugin knew how to handle the rule", it
+            doesn't necessarily mean that it was indeed successfully put in
+            place, because that might happen asynchronously and thus fail
+            only later.
+
+.. bro:id:: NetControl::clear
+
+   :Type: :bro:type:`function` () : :bro:type:`void`
+
+   Flushes all state by calling :bro:see:`NetControl::remove_rule` on all currently active rules.
+
+.. bro:id:: NetControl::delete_rule
+
+   :Type: :bro:type:`function` (id: :bro:type:`string`, reason: :bro:type:`string` :bro:attr:`&default` = ``""`` :bro:attr:`&optional`) : :bro:type:`bool`
+
+   Deletes a rule without removing it from the backends to which it has been
+   added before. This means that no messages will be sent to the switches to which
+   the rule has been added; if it is not removed from them by a separate mechanism,
+   it will stay installed and not be removed later.
+   
+
+   :id: The rule to delete, specified as the ID returned by :bro:see:`NetControl::add_rule`.
+   
+
+   :reason: Optional string argument giving information on why the rule was deleted.
+   
+
+   :returns: True if removal is successful, or sent to manager.
+            False if the rule could not be found.
+
+.. bro:id:: NetControl::find_rules_addr
+
+   :Type: :bro:type:`function` (ip: :bro:type:`addr`) : :bro:type:`vector` of :bro:type:`NetControl::Rule`
+
+   Searches all rules affecting a certain IP address.
+   
+   This function works on both the manager and workers of a cluster. Note that on
+   the worker, the internal rule variables (starting with _) will not reflect the
+   current state.
+   
+
+   :ip: The ip address to search for.
+   
+
+   :returns: vector of all rules affecting the IP address.
+
+.. bro:id:: NetControl::find_rules_subnet
+
+   :Type: :bro:type:`function` (sn: :bro:type:`subnet`) : :bro:type:`vector` of :bro:type:`NetControl::Rule`
+
+   Searches all rules affecting a certain subnet.
+   
+   A rule affects a subnet, if it covers the whole subnet. Note especially that
+   this function will not reveal all rules that are covered by a subnet.
+   
+   For example, a search for 192.168.17.0/8 will reveal a rule that exists for
+   192.168.0.0/16, since this rule affects the subnet. However, it will not reveal
+   a more specific rule for 192.168.17.1/32, which does not directy affect the whole
+   subnet.
+   
+   This function works on both the manager and workers of a cluster. Note that on
+   the worker, the internal rule variables (starting with _) will not reflect the
+   current state.
+   
+
+   :sn: The subnet to search for.
+   
+
+   :returns: vector of all rules affecting the subnet.
+
+.. bro:id:: NetControl::plugin_activated
+
+   :Type: :bro:type:`function` (p: :bro:type:`NetControl::PluginState`) : :bro:type:`void`
+
+   Function called by plugins once they finished their activation. After all
+   plugins defined in bro_init finished to activate, rules will start to be sent
+   to the plugins. Rules that scripts try to set before the backends are ready
+   will be discarded.
+
+.. bro:id:: NetControl::quarantine_host
+
+   :Type: :bro:type:`function` (infected: :bro:type:`addr`, dns: :bro:type:`addr`, quarantine: :bro:type:`addr`, t: :bro:type:`interval`, location: :bro:type:`string` :bro:attr:`&default` = ``""`` :bro:attr:`&optional`) : :bro:type:`vector` of :bro:type:`string`
+
+   Quarantines a host. This requires a special quarantine server, which runs a HTTP server explaining
+   the quarantine and a DNS server which resolves all requests to the quarantine server. DNS queries
+   from the host to the network DNS server will be rewritten and will be sent to the quarantine server
+   instead. Only http communication infected to quarantinehost is allowed. All other network communication
+   is blocked.
+   
+
+   :infected: the host to quarantine.
+   
+
+   :dns: the network dns server.
+   
+
+   :quarantine: the quarantine server running a dns and a web server.
+   
+
+   :t: how long to leave the quarantine in place.
+   
+
+   :returns: Vector of inserted rules on success, empty list on failure.
+
+.. bro:id:: NetControl::redirect_flow
+
+   :Type: :bro:type:`function` (f: :bro:type:`flow_id`, out_port: :bro:type:`count`, t: :bro:type:`interval`, location: :bro:type:`string` :bro:attr:`&default` = ``""`` :bro:attr:`&optional`) : :bro:type:`string`
+
+   Redirects a uni-directional flow to another port.
+   
+
+   :f: The flow to redirect.
+   
+
+   :out_port: Port to redirect the flow to.
+   
+
+   :t: How long to leave the redirect in place, with 0 being indefinitely.
+   
+
+   :location: An optional string describing where the redirect was triggered.
+   
+
+   :returns: The id of the inserted rule on success and zero on failure.
+
+.. bro:id:: NetControl::remove_rule
+
+   :Type: :bro:type:`function` (id: :bro:type:`string`, reason: :bro:type:`string` :bro:attr:`&default` = ``""`` :bro:attr:`&optional`) : :bro:type:`bool`
+
+   Removes a rule.
+   
+
+   :id: The rule to remove, specified as the ID returned by :bro:see:`NetControl::add_rule`.
+   
+
+   :reason: Optional string argument giving information on why the rule was removed.
+   
+
+   :returns: True if successful, the relevant plugin indicated that it knew
+            how to handle the removal. Note that again "success" means the
+            plugin accepted the removal. It might still fail to put it
+            into effect, as that might happen asynchronously and thus go
+            wrong at that point.
+
+.. bro:id:: NetControl::whitelist_address
+
+   :Type: :bro:type:`function` (a: :bro:type:`addr`, t: :bro:type:`interval`, location: :bro:type:`string` :bro:attr:`&default` = ``""`` :bro:attr:`&optional`) : :bro:type:`string`
+
+   Allows all traffic involving a specific IP address to be forwarded.
+   
+
+   :a: The address to be whitelisted.
+   
+
+   :t: How long to whitelist it, with 0 being indefinitely.
+   
+
+   :location: An optional string describing whitelist was triddered.
+   
+
+   :returns: The id of the inserted rule on success and zero on failure.
+
+.. bro:id:: NetControl::whitelist_subnet
+
+   :Type: :bro:type:`function` (s: :bro:type:`subnet`, t: :bro:type:`interval`, location: :bro:type:`string` :bro:attr:`&default` = ``""`` :bro:attr:`&optional`) : :bro:type:`string`
+
+   Allows all traffic involving a specific IP subnet to be forwarded.
+   
+
+   :s: The subnet to be whitelisted.
+   
+
+   :t: How long to whitelist it, with 0 being indefinitely.
+   
+
+   :location: An optional string describing whitelist was triddered.
+   
+
+   :returns: The id of the inserted rule on success and zero on failure.
+
+

doc/scripts/base/frameworks/netcontrol/non-cluster.bro.rst

@@ -0,0 +1,16 @@
+:tocdepth: 3
+
+base/frameworks/netcontrol/non-cluster.bro
+==========================================
+.. bro:namespace:: NetControl
+
+
+:Namespace: NetControl
+:Imports: :doc:`base/frameworks/netcontrol/main.bro </scripts/base/frameworks/netcontrol/main.bro>`
+
+Summary
+~~~~~~~
+
+Detailed Interface
+~~~~~~~~~~~~~~~~~~
+

doc/scripts/base/frameworks/netcontrol/plugin.bro.rst

@@ -0,0 +1,137 @@
+:tocdepth: 3
+
+base/frameworks/netcontrol/plugin.bro
+=====================================
+.. bro:namespace:: NetControl
+
+This file defines the plugin interface for NetControl.
+
+:Namespace: NetControl
+:Imports: :doc:`base/frameworks/netcontrol/types.bro </scripts/base/frameworks/netcontrol/types.bro>`
+
+Summary
+~~~~~~~
+Types
+#####
+======================================================= =====================================================
+:bro:type:`NetControl::Plugin`: :bro:type:`record`      Definition of a plugin.
+:bro:type:`NetControl::PluginState`: :bro:type:`record` This record keeps the per instance state of a plugin.
+======================================================= =====================================================
+
+Redefinitions
+#############
+======================================================= ========================================================================
+:bro:type:`NetControl::PluginState`: :bro:type:`record` Table for a plugin to store instance-specific configuration information.
+======================================================= ========================================================================
+
+
+Detailed Interface
+~~~~~~~~~~~~~~~~~~
+Types
+#####
+.. bro:type:: NetControl::Plugin
+
+   :Type: :bro:type:`record`
+
+      name: :bro:type:`function` (state: :bro:type:`NetControl::PluginState`) : :bro:type:`string`
+         Returns a descriptive name of the plugin instance, suitable for use in logging
+         messages. Note that this function is not optional.
+
+      can_expire: :bro:type:`bool`
+         If true, plugin can expire rules itself. If false, the NetControl
+         framework will manage rule expiration. 
+
+      init: :bro:type:`function` (state: :bro:type:`NetControl::PluginState`) : :bro:type:`void` :bro:attr:`&optional`
+         One-time initialization function called when plugin gets registered, and
+         before any other methods are called.
+         
+         If this function is provided, NetControl assumes that the plugin has to
+         perform, potentially lengthy, initialization before the plugin will become
+         active. In this case, the plugin has to call ``NetControl::plugin_activated``,
+         once initialization finishes.
+
+      done: :bro:type:`function` (state: :bro:type:`NetControl::PluginState`) : :bro:type:`void` :bro:attr:`&optional`
+         One-time finalization function called when a plugin is shutdown; no further
+         functions will be called afterwords.
+
+      add_rule: :bro:type:`function` (state: :bro:type:`NetControl::PluginState`, r: :bro:type:`NetControl::Rule`) : :bro:type:`bool` :bro:attr:`&optional`
+         Implements the add_rule() operation. If the plugin accepts the rule,
+         it returns true, false otherwise. The rule will already have its
+         ``id`` field set, which the plugin may use for identification
+         purposes.
+
+      remove_rule: :bro:type:`function` (state: :bro:type:`NetControl::PluginState`, r: :bro:type:`NetControl::Rule`, reason: :bro:type:`string`) : :bro:type:`bool` :bro:attr:`&optional`
+         Implements the remove_rule() operation. This will only be called for
+         rules that the plugin has previously accepted with add_rule(). The
+         ``id`` field will match that of the add_rule() call.  Generally,
+         a plugin that accepts an add_rule() should also accept the
+         remove_rule().
+
+   Definition of a plugin.
+   
+   Generally a plugin needs to implement only what it can support.  By
+   returning failure, it indicates that it can't support something and
+   the framework will then try another plugin, if available; or inform the
+   that the operation failed. If a function isn't implemented by a plugin,
+   that's considered an implicit failure to support the operation.
+   
+   If plugin accepts a rule operation, it *must* generate one of the reporting
+   events ``rule_{added,remove,error}`` to signal if it indeed worked out;
+   this is separate from accepting the operation because often a plugin
+   will only know later (i.e., asynchronously) if that was an error for
+   something it thought it could handle.
+
+.. bro:type:: NetControl::PluginState
+
+   :Type: :bro:type:`record`
+
+      config: :bro:type:`table` [:bro:type:`string`] of :bro:type:`string` :bro:attr:`&default` = ``{  }`` :bro:attr:`&optional`
+         Table for a plugin to store custom, instance-specific state.
+
+      _id: :bro:type:`count` :bro:attr:`&optional`
+         Unique plugin identifier -- used for backlookup of plugins from Rules. Set internally.
+
+      _priority: :bro:type:`int` :bro:attr:`&default` = ``0`` :bro:attr:`&optional`
+         Set internally.
+
+      _activated: :bro:type:`bool` :bro:attr:`&default` = ``F`` :bro:attr:`&optional`
+         Set internally. Signifies if the plugin has returned that it has activated successfully.
+
+      plugin: :bro:type:`NetControl::Plugin` :bro:attr:`&optional`
+         The plugin that the state belongs to. (Defined separately
+         because of cyclic type dependency.)
+
+      of_controller: :bro:type:`OpenFlow::Controller` :bro:attr:`&optional`
+         (present if :doc:`/scripts/base/frameworks/netcontrol/plugins/openflow.bro` is loaded)
+
+         OpenFlow controller for NetControl OpenFlow plugin.
+
+      of_config: :bro:type:`NetControl::OfConfig` :bro:attr:`&optional`
+         (present if :doc:`/scripts/base/frameworks/netcontrol/plugins/openflow.bro` is loaded)
+
+         OpenFlow configuration record that is passed on initialization.
+
+      broker_config: :bro:type:`NetControl::BrokerConfig` :bro:attr:`&optional`
+         (present if :doc:`/scripts/base/frameworks/netcontrol/plugins/broker.bro` is loaded)
+
+         OpenFlow controller for NetControl Broker plugin.
+
+      broker_id: :bro:type:`count` :bro:attr:`&optional`
+         (present if :doc:`/scripts/base/frameworks/netcontrol/plugins/broker.bro` is loaded)
+
+         The ID of this broker instance - for the mapping to PluginStates.
+
+      acld_config: :bro:type:`NetControl::AcldConfig` :bro:attr:`&optional`
+         (present if :doc:`/scripts/base/frameworks/netcontrol/plugins/acld.bro` is loaded)
+
+
+      acld_id: :bro:type:`count` :bro:attr:`&optional`
+         (present if :doc:`/scripts/base/frameworks/netcontrol/plugins/acld.bro` is loaded)
+
+         The ID of this acld instance - for the mapping to PluginStates.
+
+   This record keeps the per instance state of a plugin.
+   
+   Individual plugins commonly extend this record to suit their needs.
+
+

doc/scripts/base/frameworks/netcontrol/plugins/__load__.bro.rst

@@ -0,0 +1,14 @@
+:tocdepth: 3
+
+base/frameworks/netcontrol/plugins/__load__.bro
+===============================================
+
+
+:Imports: :doc:`base/frameworks/netcontrol/plugins/acld.bro </scripts/base/frameworks/netcontrol/plugins/acld.bro>`, :doc:`base/frameworks/netcontrol/plugins/broker.bro </scripts/base/frameworks/netcontrol/plugins/broker.bro>`, :doc:`base/frameworks/netcontrol/plugins/debug.bro </scripts/base/frameworks/netcontrol/plugins/debug.bro>`, :doc:`base/frameworks/netcontrol/plugins/openflow.bro </scripts/base/frameworks/netcontrol/plugins/openflow.bro>`, :doc:`base/frameworks/netcontrol/plugins/packetfilter.bro </scripts/base/frameworks/netcontrol/plugins/packetfilter.bro>`
+
+Summary
+~~~~~~~
+
+Detailed Interface
+~~~~~~~~~~~~~~~~~~
+

doc/scripts/base/frameworks/netcontrol/plugins/acld.bro.rst

@@ -0,0 +1,162 @@
+:tocdepth: 3
+
+base/frameworks/netcontrol/plugins/acld.bro
+===========================================
+.. bro:namespace:: NetControl
+
+Acld plugin for the netcontrol framework.
+
+:Namespace: NetControl
+:Imports: :doc:`base/frameworks/broker </scripts/base/frameworks/broker/index>`, :doc:`base/frameworks/netcontrol/main.bro </scripts/base/frameworks/netcontrol/main.bro>`, :doc:`base/frameworks/netcontrol/plugin.bro </scripts/base/frameworks/netcontrol/plugin.bro>`
+
+Summary
+~~~~~~~
+Types
+#####
+====================================================== =
+:bro:type:`NetControl::AclRule`: :bro:type:`record`    
+:bro:type:`NetControl::AcldConfig`: :bro:type:`record` 
+====================================================== =
+
+Redefinitions
+#############
+======================================================= =
+:bro:type:`NetControl::PluginState`: :bro:type:`record` 
+======================================================= =
+
+Events
+######
+========================================================== =======================================
+:bro:id:`NetControl::acld_add_rule`: :bro:type:`event`     Events that are sent from us to Broker.
+:bro:id:`NetControl::acld_remove_rule`: :bro:type:`event`  
+:bro:id:`NetControl::acld_rule_added`: :bro:type:`event`   Events that are sent from Broker to us.
+:bro:id:`NetControl::acld_rule_error`: :bro:type:`event`   
+:bro:id:`NetControl::acld_rule_exists`: :bro:type:`event`  
+:bro:id:`NetControl::acld_rule_removed`: :bro:type:`event` 
+========================================================== =======================================
+
+Hooks
+#####
+======================================================== ==============================================================
+:bro:id:`NetControl::acld_rule_policy`: :bro:type:`hook` Hook that is called after a rule is converted to an acld rule.
+======================================================== ==============================================================
+
+Functions
+#########
+======================================================= =============================
+:bro:id:`NetControl::create_acld`: :bro:type:`function` Instantiates the acld plugin.
+======================================================= =============================
+
+
+Detailed Interface
+~~~~~~~~~~~~~~~~~~
+Types
+#####
+.. bro:type:: NetControl::AclRule
+
+   :Type: :bro:type:`record`
+
+      command: :bro:type:`string`
+
+      cookie: :bro:type:`count`
+
+      arg: :bro:type:`string`
+
+      comment: :bro:type:`string` :bro:attr:`&optional`
+
+
+.. bro:type:: NetControl::AcldConfig
+
+   :Type: :bro:type:`record`
+
+      acld_topic: :bro:type:`string`
+         The acld topic to send events to.
+
+      acld_host: :bro:type:`addr`
+         Broker host to connect to.
+
+      acld_port: :bro:type:`port`
+         Broker port to connect to.
+
+      monitor: :bro:type:`bool` :bro:attr:`&default` = ``F`` :bro:attr:`&optional`
+         Do we accept rules for the monitor path? Default false.
+
+      forward: :bro:type:`bool` :bro:attr:`&default` = ``T`` :bro:attr:`&optional`
+         Do we accept rules for the forward path? Default true.
+
+      check_pred: :bro:type:`function` (p: :bro:type:`NetControl::PluginState`, r: :bro:type:`NetControl::Rule`) : :bro:type:`bool` :bro:attr:`&optional`
+         Predicate that is called on rule insertion or removal.
+         
+
+         :p: Current plugin state.
+         
+
+         :r: The rule to be inserted or removed.
+         
+
+         :returns: T if the rule can be handled by the current backend, F otherwise.
+
+
+Events
+######
+.. bro:id:: NetControl::acld_add_rule
+
+   :Type: :bro:type:`event` (id: :bro:type:`count`, r: :bro:type:`NetControl::Rule`, ar: :bro:type:`NetControl::AclRule`)
+
+   Events that are sent from us to Broker.
+
+.. bro:id:: NetControl::acld_remove_rule
+
+   :Type: :bro:type:`event` (id: :bro:type:`count`, r: :bro:type:`NetControl::Rule`, ar: :bro:type:`NetControl::AclRule`)
+
+
+.. bro:id:: NetControl::acld_rule_added
+
+   :Type: :bro:type:`event` (id: :bro:type:`count`, r: :bro:type:`NetControl::Rule`, msg: :bro:type:`string`)
+
+   Events that are sent from Broker to us.
+
+.. bro:id:: NetControl::acld_rule_error
+
+   :Type: :bro:type:`event` (id: :bro:type:`count`, r: :bro:type:`NetControl::Rule`, msg: :bro:type:`string`)
+
+
+.. bro:id:: NetControl::acld_rule_exists
+
+   :Type: :bro:type:`event` (id: :bro:type:`count`, r: :bro:type:`NetControl::Rule`, msg: :bro:type:`string`)
+
+
+.. bro:id:: NetControl::acld_rule_removed
+
+   :Type: :bro:type:`event` (id: :bro:type:`count`, r: :bro:type:`NetControl::Rule`, msg: :bro:type:`string`)
+
+
+Hooks
+#####
+.. bro:id:: NetControl::acld_rule_policy
+
+   :Type: :bro:type:`hook` (p: :bro:type:`NetControl::PluginState`, r: :bro:type:`NetControl::Rule`, ar: :bro:type:`NetControl::AclRule`) : :bro:type:`bool`
+
+   Hook that is called after a rule is converted to an acld rule.
+   The hook may modify the rule before it is sent to acld.
+   Setting the acld command to F will cause the rule to be rejected
+   by the plugin.
+   
+
+   :p: Current plugin state.
+   
+
+   :r: The rule to be inserted or removed.
+   
+
+   :ar: The acld rule to be inserted or removed.
+
+Functions
+#########
+.. bro:id:: NetControl::create_acld
+
+   :Type: :bro:type:`function` (config: :bro:type:`NetControl::AcldConfig`) : :bro:type:`NetControl::PluginState`
+
+   Instantiates the acld plugin.
+
+

doc/scripts/base/frameworks/netcontrol/plugins/broker.bro.rst

@@ -0,0 +1,129 @@
+:tocdepth: 3
+
+base/frameworks/netcontrol/plugins/broker.bro
+=============================================
+.. bro:namespace:: NetControl
+
+Broker plugin for the NetControl framework. Sends the raw data structures
+used in NetControl on to Broker to allow for easy handling, e.g., of
+command-line scripts.
+
+:Namespace: NetControl
+:Imports: :doc:`base/frameworks/broker </scripts/base/frameworks/broker/index>`, :doc:`base/frameworks/netcontrol/main.bro </scripts/base/frameworks/netcontrol/main.bro>`, :doc:`base/frameworks/netcontrol/plugin.bro </scripts/base/frameworks/netcontrol/plugin.bro>`
+
+Summary
+~~~~~~~
+Types
+#####
+======================================================== ===============================================================================================
+:bro:type:`NetControl::BrokerConfig`: :bro:type:`record` This record specifies the configuration that is passed to :bro:see:`NetControl::create_broker`.
+======================================================== ===============================================================================================
+
+Redefinitions
+#############
+======================================================= =
+:bro:type:`NetControl::PluginState`: :bro:type:`record` 
+======================================================= =
+
+Events
+######
+============================================================ =
+:bro:id:`NetControl::broker_add_rule`: :bro:type:`event`     
+:bro:id:`NetControl::broker_remove_rule`: :bro:type:`event`  
+:bro:id:`NetControl::broker_rule_added`: :bro:type:`event`   
+:bro:id:`NetControl::broker_rule_error`: :bro:type:`event`   
+:bro:id:`NetControl::broker_rule_exists`: :bro:type:`event`  
+:bro:id:`NetControl::broker_rule_removed`: :bro:type:`event` 
+:bro:id:`NetControl::broker_rule_timeout`: :bro:type:`event` 
+============================================================ =
+
+Functions
+#########
+========================================================= ===============================
+:bro:id:`NetControl::create_broker`: :bro:type:`function` Instantiates the broker plugin.
+========================================================= ===============================
+
+
+Detailed Interface
+~~~~~~~~~~~~~~~~~~
+Types
+#####
+.. bro:type:: NetControl::BrokerConfig
+
+   :Type: :bro:type:`record`
+
+      topic: :bro:type:`string` :bro:attr:`&optional`
+         The broker topic to send events to.
+
+      host: :bro:type:`addr` :bro:attr:`&optional`
+         Broker host to connect to.
+
+      bport: :bro:type:`port` :bro:attr:`&optional`
+         Broker port to connect to.
+
+      monitor: :bro:type:`bool` :bro:attr:`&default` = ``T`` :bro:attr:`&optional`
+         Do we accept rules for the monitor path? Default true.
+
+      forward: :bro:type:`bool` :bro:attr:`&default` = ``T`` :bro:attr:`&optional`
+         Do we accept rules for the forward path? Default true.
+
+      check_pred: :bro:type:`function` (p: :bro:type:`NetControl::PluginState`, r: :bro:type:`NetControl::Rule`) : :bro:type:`bool` :bro:attr:`&optional`
+         Predicate that is called on rule insertion or removal.
+         
+
+         :p: Current plugin state.
+         
+
+         :r: The rule to be inserted or removed.
+         
+
+         :returns: T if the rule can be handled by the current backend, F otherwise.
+
+   This record specifies the configuration that is passed to :bro:see:`NetControl::create_broker`.
+
+Events
+######
+.. bro:id:: NetControl::broker_add_rule
+
+   :Type: :bro:type:`event` (id: :bro:type:`count`, r: :bro:type:`NetControl::Rule`)
+
+
+.. bro:id:: NetControl::broker_remove_rule
+
+   :Type: :bro:type:`event` (id: :bro:type:`count`, r: :bro:type:`NetControl::Rule`, reason: :bro:type:`string`)
+
+
+.. bro:id:: NetControl::broker_rule_added
+
+   :Type: :bro:type:`event` (id: :bro:type:`count`, r: :bro:type:`NetControl::Rule`, msg: :bro:type:`string`)
+
+
+.. bro:id:: NetControl::broker_rule_error
+
+   :Type: :bro:type:`event` (id: :bro:type:`count`, r: :bro:type:`NetControl::Rule`, msg: :bro:type:`string`)
+
+
+.. bro:id:: NetControl::broker_rule_exists
+
+   :Type: :bro:type:`event` (id: :bro:type:`count`, r: :bro:type:`NetControl::Rule`, msg: :bro:type:`string`)
+
+
+.. bro:id:: NetControl::broker_rule_removed
+
+   :Type: :bro:type:`event` (id: :bro:type:`count`, r: :bro:type:`NetControl::Rule`, msg: :bro:type:`string`)
+
+
+.. bro:id:: NetControl::broker_rule_timeout
+
+   :Type: :bro:type:`event` (id: :bro:type:`count`, r: :bro:type:`NetControl::Rule`, i: :bro:type:`NetControl::FlowInfo`)
+
+
+Functions
+#########
+.. bro:id:: NetControl::create_broker
+
+   :Type: :bro:type:`function` (config: :bro:type:`NetControl::BrokerConfig`, can_expire: :bro:type:`bool`) : :bro:type:`NetControl::PluginState`
+
+   Instantiates the broker plugin.
+
+

doc/scripts/base/frameworks/netcontrol/plugins/debug.bro.rst

@@ -0,0 +1,37 @@
+:tocdepth: 3
+
+base/frameworks/netcontrol/plugins/debug.bro
+============================================
+.. bro:namespace:: NetControl
+
+Debugging plugin for the NetControl framework, providing insight into
+executed operations.
+
+:Namespace: NetControl
+:Imports: :doc:`base/frameworks/netcontrol/main.bro </scripts/base/frameworks/netcontrol/main.bro>`, :doc:`base/frameworks/netcontrol/plugin.bro </scripts/base/frameworks/netcontrol/plugin.bro>`
+
+Summary
+~~~~~~~
+Functions
+#########
+======================================================== =========================================================
+:bro:id:`NetControl::create_debug`: :bro:type:`function` Instantiates a debug plugin for the NetControl framework.
+======================================================== =========================================================
+
+
+Detailed Interface
+~~~~~~~~~~~~~~~~~~
+Functions
+#########
+.. bro:id:: NetControl::create_debug
+
+   :Type: :bro:type:`function` (do_something: :bro:type:`bool`) : :bro:type:`NetControl::PluginState`
+
+   Instantiates a debug plugin for the NetControl framework. The debug
+   plugin simply logs the operations it receives.
+   
+
+   :do_something: If true, the plugin will claim it supports all operations; if
+                 false, it will indicate it doesn't support any.
+
+

doc/scripts/base/frameworks/netcontrol/plugins/index.rst

@@ -0,0 +1,36 @@
+:orphan:
+
+Package: base/frameworks/netcontrol/plugins
+===========================================
+
+Plugins for the NetControl framework.
+
+:doc:`/scripts/base/frameworks/netcontrol/plugins/__load__.bro`
+
+
+:doc:`/scripts/base/frameworks/netcontrol/plugins/debug.bro`
+
+   Debugging plugin for the NetControl framework, providing insight into
+   executed operations.
+
+:doc:`/scripts/base/frameworks/netcontrol/plugins/openflow.bro`
+
+   OpenFlow plugin for the NetControl framework.
+
+:doc:`/scripts/base/frameworks/netcontrol/plugins/packetfilter.bro`
+
+   NetControl plugin for the process-level PacketFilter that comes with
+   Bro. Since the PacketFilter in Bro is quite limited in scope
+   and can only add/remove filters for addresses, this is quite
+   limited in scope at the moment. 
+
+:doc:`/scripts/base/frameworks/netcontrol/plugins/broker.bro`
+
+   Broker plugin for the NetControl framework. Sends the raw data structures
+   used in NetControl on to Broker to allow for easy handling, e.g., of
+   command-line scripts.
+
+:doc:`/scripts/base/frameworks/netcontrol/plugins/acld.bro`
+
+   Acld plugin for the netcontrol framework.
+

doc/scripts/base/frameworks/netcontrol/plugins/openflow.bro.rst

@@ -0,0 +1,158 @@
+:tocdepth: 3
+
+base/frameworks/netcontrol/plugins/openflow.bro
+===============================================
+.. bro:namespace:: NetControl
+
+OpenFlow plugin for the NetControl framework.
+
+:Namespace: NetControl
+:Imports: :doc:`base/frameworks/netcontrol/main.bro </scripts/base/frameworks/netcontrol/main.bro>`, :doc:`base/frameworks/netcontrol/plugin.bro </scripts/base/frameworks/netcontrol/plugin.bro>`, :doc:`base/frameworks/openflow </scripts/base/frameworks/openflow/index>`
+
+Summary
+~~~~~~~
+Redefinable Options
+###################
+======================================================================================= ===============================================================================
+:bro:id:`NetControl::openflow_flow_timeout`: :bro:type:`interval` :bro:attr:`&redef`    The time interval after we consider a flow timed out.
+:bro:id:`NetControl::openflow_message_timeout`: :bro:type:`interval` :bro:attr:`&redef` The time interval after which an openflow message is considered to be timed out
+                                                                                        and we delete it from our internal tracking.
+======================================================================================= ===============================================================================
+
+Types
+#####
+==================================================== =================================================================================================
+:bro:type:`NetControl::OfConfig`: :bro:type:`record` This record specifies the configuration that is passed to :bro:see:`NetControl::create_openflow`.
+:bro:type:`NetControl::OfTable`: :bro:type:`record`  
+==================================================== =================================================================================================
+
+Redefinitions
+#############
+======================================================= =
+:bro:type:`NetControl::PluginState`: :bro:type:`record` 
+======================================================= =
+
+Functions
+#########
+=========================================================== =============================================================
+:bro:id:`NetControl::create_openflow`: :bro:type:`function` Instantiates an openflow plugin for the NetControl framework.
+=========================================================== =============================================================
+
+
+Detailed Interface
+~~~~~~~~~~~~~~~~~~
+Redefinable Options
+###################
+.. bro:id:: NetControl::openflow_flow_timeout
+
+   :Type: :bro:type:`interval`
+   :Attributes: :bro:attr:`&redef`
+   :Default: ``1.0 day``
+
+   The time interval after we consider a flow timed out. This should be fairly high (or
+   even disabled) if you expect a lot of long flows. However, one also will have state
+   buildup for quite a while if keeping this around...
+
+.. bro:id:: NetControl::openflow_message_timeout
+
+   :Type: :bro:type:`interval`
+   :Attributes: :bro:attr:`&redef`
+   :Default: ``20.0 secs``
+
+   The time interval after which an openflow message is considered to be timed out
+   and we delete it from our internal tracking.
+
+Types
+#####
+.. bro:type:: NetControl::OfConfig
+
+   :Type: :bro:type:`record`
+
+      monitor: :bro:type:`bool` :bro:attr:`&default` = ``T`` :bro:attr:`&optional`
+         Accept rules that target the monitor path.
+
+      forward: :bro:type:`bool` :bro:attr:`&default` = ``T`` :bro:attr:`&optional`
+         Accept rules that target the forward path.
+
+      idle_timeout: :bro:type:`count` :bro:attr:`&default` = ``0`` :bro:attr:`&optional`
+         Default OpenFlow idle timeout.
+
+      table_id: :bro:type:`count` :bro:attr:`&optional`
+         Default OpenFlow table ID.
+
+      priority_offset: :bro:type:`int` :bro:attr:`&default` = ``0`` :bro:attr:`&optional`
+         Add this to all rule priorities. Can be useful if you want the openflow priorities be offset from the netcontrol priorities without having to write a filter function.
+
+      check_pred: :bro:type:`function` (p: :bro:type:`NetControl::PluginState`, r: :bro:type:`NetControl::Rule`) : :bro:type:`bool` :bro:attr:`&optional`
+         Predicate that is called on rule insertion or removal.
+         
+
+         :p: Current plugin state.
+         
+
+         :r: The rule to be inserted or removed.
+         
+
+         :returns: T if the rule can be handled by the current backend, F otherwise.
+
+      match_pred: :bro:type:`function` (p: :bro:type:`NetControl::PluginState`, e: :bro:type:`NetControl::Entity`, m: :bro:type:`vector` of :bro:type:`OpenFlow::ofp_match`) : :bro:type:`vector` of :bro:type:`OpenFlow::ofp_match` :bro:attr:`&optional`
+         This predicate is called each time an OpenFlow match record is created.
+         The predicate can modify the match structure before it is sent on to the
+         device.
+         
+
+         :p: Current plugin state.
+         
+
+         :r: The rule to be inserted or removed.
+         
+
+         :m: The openflow match structures that were generated for this rules.
+         
+
+         :returns: The modified OpenFlow match structures that will be used in place of the structures passed in m.
+
+      flow_mod_pred: :bro:type:`function` (p: :bro:type:`NetControl::PluginState`, r: :bro:type:`NetControl::Rule`, m: :bro:type:`OpenFlow::ofp_flow_mod`) : :bro:type:`OpenFlow::ofp_flow_mod` :bro:attr:`&optional`
+         This predicate is called before a FlowMod message is sent to the OpenFlow
+         device. It can modify the FlowMod message before it is passed on.
+         
+
+         :p: Current plugin state.
+         
+
+         :r: The rule to be inserted or removed.
+         
+
+         :m: The OpenFlow FlowMod message.
+         
+
+         :returns: The modified FlowMod message that is used in lieu of m.
+
+   This record specifies the configuration that is passed to :bro:see:`NetControl::create_openflow`.
+
+.. bro:type:: NetControl::OfTable
+
+   :Type: :bro:type:`record`
+
+      p: :bro:type:`NetControl::PluginState`
+
+      r: :bro:type:`NetControl::Rule`
+
+      c: :bro:type:`count` :bro:attr:`&default` = ``0`` :bro:attr:`&optional`
+
+      packet_count: :bro:type:`count` :bro:attr:`&default` = ``0`` :bro:attr:`&optional`
+
+      byte_count: :bro:type:`count` :bro:attr:`&default` = ``0`` :bro:attr:`&optional`
+
+      duration_sec: :bro:type:`double` :bro:attr:`&default` = ``0.0`` :bro:attr:`&optional`
+
+
+Functions
+#########
+.. bro:id:: NetControl::create_openflow
+
+   :Type: :bro:type:`function` (controller: :bro:type:`OpenFlow::Controller`, config: :bro:type:`NetControl::OfConfig` :bro:attr:`&default` = ``[]`` :bro:attr:`&optional`) : :bro:type:`NetControl::PluginState`
+
+   Instantiates an openflow plugin for the NetControl framework.
+
+

doc/scripts/base/frameworks/netcontrol/plugins/packetfilter.bro.rst

@@ -0,0 +1,34 @@
+:tocdepth: 3
+
+base/frameworks/netcontrol/plugins/packetfilter.bro
+===================================================
+.. bro:namespace:: NetControl
+
+NetControl plugin for the process-level PacketFilter that comes with
+Bro. Since the PacketFilter in Bro is quite limited in scope
+and can only add/remove filters for addresses, this is quite
+limited in scope at the moment. 
+
+:Namespace: NetControl
+:Imports: :doc:`base/frameworks/netcontrol/plugin.bro </scripts/base/frameworks/netcontrol/plugin.bro>`
+
+Summary
+~~~~~~~
+Functions
+#########
+=============================================================== =====================================
+:bro:id:`NetControl::create_packetfilter`: :bro:type:`function` Instantiates the packetfilter plugin.
+=============================================================== =====================================
+
+
+Detailed Interface
+~~~~~~~~~~~~~~~~~~
+Functions
+#########
+.. bro:id:: NetControl::create_packetfilter
+
+   :Type: :bro:type:`function` () : :bro:type:`NetControl::PluginState`
+
+   Instantiates the packetfilter plugin.
+
+

doc/scripts/base/frameworks/netcontrol/shunt.bro.rst

@@ -0,0 +1,93 @@
+:tocdepth: 3
+
+base/frameworks/netcontrol/shunt.bro
+====================================
+.. bro:namespace:: NetControl
+
+Implementation of the shunt functionality for NetControl.
+
+:Namespace: NetControl
+:Imports: :doc:`base/frameworks/netcontrol/main.bro </scripts/base/frameworks/netcontrol/main.bro>`
+
+Summary
+~~~~~~~
+Types
+#####
+===================================================== =
+:bro:type:`NetControl::ShuntInfo`: :bro:type:`record` 
+===================================================== =
+
+Redefinitions
+#############
+===================================== =
+:bro:type:`Log::ID`: :bro:type:`enum` 
+===================================== =
+
+Events
+######
+============================================================= =========================================================================
+:bro:id:`NetControl::log_netcontrol_shunt`: :bro:type:`event` Event that can be handled to access the :bro:type:`NetControl::ShuntInfo`
+                                                              record as it is sent on to the logging framework.
+============================================================= =========================================================================
+
+Functions
+#########
+====================================================== =========================================================
+:bro:id:`NetControl::shunt_flow`: :bro:type:`function` Stops forwarding a uni-directional flow's packets to Bro.
+====================================================== =========================================================
+
+
+Detailed Interface
+~~~~~~~~~~~~~~~~~~
+Types
+#####
+.. bro:type:: NetControl::ShuntInfo
+
+   :Type: :bro:type:`record`
+
+      ts: :bro:type:`time` :bro:attr:`&log`
+         Time at which the recorded activity occurred.
+
+      rule_id: :bro:type:`string` :bro:attr:`&log`
+         ID of the rule; unique during each Bro run.
+
+      f: :bro:type:`flow_id` :bro:attr:`&log`
+         Flow ID of the shunted flow.
+
+      expire: :bro:type:`interval` :bro:attr:`&log`
+         Expiry time of the shunt.
+
+      location: :bro:type:`string` :bro:attr:`&log` :bro:attr:`&optional`
+         Location where the underlying action was triggered.
+
+
+Events
+######
+.. bro:id:: NetControl::log_netcontrol_shunt
+
+   :Type: :bro:type:`event` (rec: :bro:type:`NetControl::ShuntInfo`)
+
+   Event that can be handled to access the :bro:type:`NetControl::ShuntInfo`
+   record as it is sent on to the logging framework.
+
+Functions
+#########
+.. bro:id:: NetControl::shunt_flow
+
+   :Type: :bro:type:`function` (f: :bro:type:`flow_id`, t: :bro:type:`interval`, location: :bro:type:`string` :bro:attr:`&default` = ``""`` :bro:attr:`&optional`) : :bro:type:`string`
+
+   Stops forwarding a uni-directional flow's packets to Bro.
+   
+
+   :f: The flow to shunt.
+   
+
+   :t: How long to leave the shunt in place, with 0 being indefinitely.
+   
+
+   :location: An optional string describing where the shunt was triggered.
+   
+
+   :returns: The id of the inserted rule on success and zero on failure.
+
+

doc/scripts/base/frameworks/netcontrol/types.bro.rst

@@ -0,0 +1,296 @@
+:tocdepth: 3
+
+base/frameworks/netcontrol/types.bro
+====================================
+.. bro:namespace:: NetControl
+
+This file defines the types that are used by the NetControl framework.
+
+The most important type defined in this file is :bro:see:`NetControl::Rule`,
+which is used to describe all rules that can be expressed by the NetControl framework. 
+
+:Namespace: NetControl
+
+Summary
+~~~~~~~
+Runtime Options
+###############
+========================================================================== ======================================================
+:bro:id:`NetControl::default_priority`: :bro:type:`int` :bro:attr:`&redef` The default priority that is used when creating rules.
+========================================================================== ======================================================
+
+Redefinable Options
+###################
+============================================================================ ====================================================================================
+:bro:id:`NetControl::whitelist_priority`: :bro:type:`int` :bro:attr:`&redef` The default priority that is used when using the high-level functions to
+                                                                             push whitelist entries to the backends (:bro:see:`NetControl::whitelist_address` and
+                                                                             :bro:see:`NetControl::whitelist_subnet`).
+============================================================================ ====================================================================================
+
+Types
+#####
+==================================================== ====================================================================================================
+:bro:type:`NetControl::Entity`: :bro:type:`record`   Type defining the entity a rule is operating on.
+:bro:type:`NetControl::EntityType`: :bro:type:`enum` Type defining the entity that a rule applies to.
+:bro:type:`NetControl::Flow`: :bro:type:`record`     Flow is used in :bro:type:`NetControl::Entity` together with :bro:enum:`NetControl::FLOW` to specify
+                                                     a uni-directional flow that a rule applies to.
+:bro:type:`NetControl::FlowInfo`: :bro:type:`record` Information of a flow that can be provided by switches when the flow times out.
+:bro:type:`NetControl::FlowMod`: :bro:type:`record`  Type for defining a flow modification action.
+:bro:type:`NetControl::Rule`: :bro:type:`record`     A rule for the framework to put in place.
+:bro:type:`NetControl::RuleType`: :bro:type:`enum`   Type of rules that the framework supports.
+:bro:type:`NetControl::TargetType`: :bro:type:`enum` Type defining the target of a rule.
+==================================================== ====================================================================================================
+
+
+Detailed Interface
+~~~~~~~~~~~~~~~~~~
+Runtime Options
+###############
+.. bro:id:: NetControl::default_priority
+
+   :Type: :bro:type:`int`
+   :Attributes: :bro:attr:`&redef`
+   :Default: ``0``
+
+   The default priority that is used when creating rules.
+
+Redefinable Options
+###################
+.. bro:id:: NetControl::whitelist_priority
+
+   :Type: :bro:type:`int`
+   :Attributes: :bro:attr:`&redef`
+   :Default: ``5``
+
+   The default priority that is used when using the high-level functions to
+   push whitelist entries to the backends (:bro:see:`NetControl::whitelist_address` and
+   :bro:see:`NetControl::whitelist_subnet`).
+   
+   Note that this priority is not automatically used when manually creating rules
+   that have a :bro:see:`NetControl::RuleType` of :bro:enum:`NetControl::WHITELIST`.
+
+Types
+#####
+.. bro:type:: NetControl::Entity
+
+   :Type: :bro:type:`record`
+
+      ty: :bro:type:`NetControl::EntityType`
+         Type of entity.
+
+      conn: :bro:type:`conn_id` :bro:attr:`&optional`
+         Used with :bro:enum:`NetControl::CONNECTION`.
+
+      flow: :bro:type:`NetControl::Flow` :bro:attr:`&optional`
+         Used with :bro:enum:`NetControl::FLOW`.
+
+      ip: :bro:type:`subnet` :bro:attr:`&optional`
+         Used with :bro:enum:`NetControl::ADDRESS` to specifiy a CIDR subnet.
+
+      mac: :bro:type:`string` :bro:attr:`&optional`
+         Used with :bro:enum:`NetControl::MAC`.
+
+   Type defining the entity a rule is operating on.
+
+.. bro:type:: NetControl::EntityType
+
+   :Type: :bro:type:`enum`
+
+      .. bro:enum:: NetControl::ADDRESS NetControl::EntityType
+
+         Activity involving a specific IP address.
+
+      .. bro:enum:: NetControl::CONNECTION NetControl::EntityType
+
+         Activity involving all of a bi-directional connection's activity.
+
+      .. bro:enum:: NetControl::FLOW NetControl::EntityType
+
+         Activity involving a uni-directional flow's activity. Can contain wildcards.
+
+      .. bro:enum:: NetControl::MAC NetControl::EntityType
+
+         Activity involving a MAC address.
+
+   Type defining the entity that a rule applies to.
+
+.. bro:type:: NetControl::Flow
+
+   :Type: :bro:type:`record`
+
+      src_h: :bro:type:`subnet` :bro:attr:`&optional`
+         The source IP address/subnet.
+
+      src_p: :bro:type:`port` :bro:attr:`&optional`
+         The source port number.
+
+      dst_h: :bro:type:`subnet` :bro:attr:`&optional`
+         The destination IP address/subnet.
+
+      dst_p: :bro:type:`port` :bro:attr:`&optional`
+         The destination port number.
+
+      src_m: :bro:type:`string` :bro:attr:`&optional`
+         The source MAC address.
+
+      dst_m: :bro:type:`string` :bro:attr:`&optional`
+         The destination MAC address.
+
+   Flow is used in :bro:type:`NetControl::Entity` together with :bro:enum:`NetControl::FLOW` to specify
+   a uni-directional flow that a rule applies to.
+   
+   If optional fields are not set, they are interpreted as wildcarded.
+
+.. bro:type:: NetControl::FlowInfo
+
+   :Type: :bro:type:`record`
+
+      duration: :bro:type:`interval` :bro:attr:`&optional`
+         Total duration of the rule.
+
+      packet_count: :bro:type:`count` :bro:attr:`&optional`
+         Number of packets exchanged over connections matched by the rule.
+
+      byte_count: :bro:type:`count` :bro:attr:`&optional`
+         Total bytes exchanged over connections matched by the rule.
+
+   Information of a flow that can be provided by switches when the flow times out.
+   Currently this is heavily influenced by the data that OpenFlow returns by default.
+   That being said - their design makes sense and this is probably the data one
+   can expect to be available.
+
+.. bro:type:: NetControl::FlowMod
+
+   :Type: :bro:type:`record`
+
+      src_h: :bro:type:`addr` :bro:attr:`&optional`
+         The source IP address.
+
+      src_p: :bro:type:`count` :bro:attr:`&optional`
+         The source port number.
+
+      dst_h: :bro:type:`addr` :bro:attr:`&optional`
+         The destination IP address.
+
+      dst_p: :bro:type:`count` :bro:attr:`&optional`
+         The destination port number.
+
+      src_m: :bro:type:`string` :bro:attr:`&optional`
+         The source MAC address.
+
+      dst_m: :bro:type:`string` :bro:attr:`&optional`
+         The destination MAC address.
+
+      redirect_port: :bro:type:`count` :bro:attr:`&optional`
+
+   Type for defining a flow modification action.
+
+.. bro:type:: NetControl::Rule
+
+   :Type: :bro:type:`record`
+
+      ty: :bro:type:`NetControl::RuleType`
+         Type of rule.
+
+      target: :bro:type:`NetControl::TargetType`
+         Where to apply rule.
+
+      entity: :bro:type:`NetControl::Entity`
+         Entity to apply rule to.
+
+      expire: :bro:type:`interval` :bro:attr:`&optional`
+         Timeout after which to expire the rule.
+
+      priority: :bro:type:`int` :bro:attr:`&default` = :bro:see:`NetControl::default_priority` :bro:attr:`&optional`
+         Priority if multiple rules match an entity (larger value is higher priority).
+
+      location: :bro:type:`string` :bro:attr:`&optional`
+         Optional string describing where/what installed the rule.
+
+      out_port: :bro:type:`count` :bro:attr:`&optional`
+         Argument for :bro:enum:`NetControl::REDIRECT` rules.
+
+      mod: :bro:type:`NetControl::FlowMod` :bro:attr:`&optional`
+         Argument for :bro:enum:`NetControl::MODIFY` rules.
+
+      id: :bro:type:`string` :bro:attr:`&default` = ``""`` :bro:attr:`&optional`
+         Internally determined unique ID for this rule. Will be set when added.
+
+      cid: :bro:type:`count` :bro:attr:`&default` = ``0`` :bro:attr:`&optional`
+         Internally determined unique numeric ID for this rule. Set when added.
+
+      _plugin_ids: :bro:type:`set` [:bro:type:`count`] :bro:attr:`&default` = ``{  }`` :bro:attr:`&optional`
+         (present if :doc:`/scripts/base/frameworks/netcontrol/main.bro` is loaded)
+
+         Internally set to the plugins handling the rule.
+
+      _active_plugin_ids: :bro:type:`set` [:bro:type:`count`] :bro:attr:`&default` = ``{  }`` :bro:attr:`&optional`
+         (present if :doc:`/scripts/base/frameworks/netcontrol/main.bro` is loaded)
+
+         Internally set to the plugins on which the rule is currently active.
+
+      _no_expire_plugins: :bro:type:`set` [:bro:type:`count`] :bro:attr:`&default` = ``{  }`` :bro:attr:`&optional`
+         (present if :doc:`/scripts/base/frameworks/netcontrol/main.bro` is loaded)
+
+         Internally set to plugins where the rule should not be removed upon timeout.
+
+      _added: :bro:type:`bool` :bro:attr:`&default` = ``F`` :bro:attr:`&optional`
+         (present if :doc:`/scripts/base/frameworks/netcontrol/main.bro` is loaded)
+
+         Track if the rule was added successfully by all responsible plugins.
+
+   A rule for the framework to put in place. Of all rules currently in
+   place, the first match will be taken, sorted by priority. All
+   further rules will be ignored.
+
+.. bro:type:: NetControl::RuleType
+
+   :Type: :bro:type:`enum`
+
+      .. bro:enum:: NetControl::DROP NetControl::RuleType
+
+         Stop forwarding all packets matching the entity.
+         
+         No additional arguments.
+
+      .. bro:enum:: NetControl::MODIFY NetControl::RuleType
+
+         Modify all packets matching entity. The packets
+         will be modified according to the `mod` entry of
+         the rule.
+         
+
+      .. bro:enum:: NetControl::REDIRECT NetControl::RuleType
+
+         Redirect all packets matching entity to a different switch port,
+         given in the `out_port` argument of the rule.
+         
+
+      .. bro:enum:: NetControl::WHITELIST NetControl::RuleType
+
+         Whitelists all packets of an entity, meaning no restrictions will be applied.
+         While whitelisting is the default if no rule matches, this type can be
+         used to override lower-priority rules that would otherwise take effect for the
+         entity.
+
+   Type of rules that the framework supports. Each type lists the extra
+   :bro:type:`NetControl::Rule` fields it uses, if any.
+   
+   Plugins may extend this type to define their own.
+
+.. bro:type:: NetControl::TargetType
+
+   :Type: :bro:type:`enum`
+
+      .. bro:enum:: NetControl::FORWARD NetControl::TargetType
+
+      .. bro:enum:: NetControl::MONITOR NetControl::TargetType
+
+   Type defining the target of a rule.
+   
+   Rules can either be applied to the forward path, affecting all network traffic, or
+   on the monitor path, only affecting the traffic that is sent to Bro. The second
+   is mostly used for shunting, which allows Bro to tell the networking hardware that
+   it wants to no longer see traffic that it identified as benign.
+
+

doc/scripts/base/frameworks/notice/__load__.bro.rst

@@ -0,0 +1,14 @@
+:tocdepth: 3
+
+base/frameworks/notice/__load__.bro
+===================================
+
+
+:Imports: :doc:`base/frameworks/notice/actions/add-geodata.bro </scripts/base/frameworks/notice/actions/add-geodata.bro>`, :doc:`base/frameworks/notice/actions/drop.bro </scripts/base/frameworks/notice/actions/drop.bro>`, :doc:`base/frameworks/notice/actions/email_admin.bro </scripts/base/frameworks/notice/actions/email_admin.bro>`, :doc:`base/frameworks/notice/actions/page.bro </scripts/base/frameworks/notice/actions/page.bro>`, :doc:`base/frameworks/notice/actions/pp-alarms.bro </scripts/base/frameworks/notice/actions/pp-alarms.bro>`, :doc:`base/frameworks/notice/main.bro </scripts/base/frameworks/notice/main.bro>`, :doc:`base/frameworks/notice/weird.bro </scripts/base/frameworks/notice/weird.bro>`
+
+Summary
+~~~~~~~
+
+Detailed Interface
+~~~~~~~~~~~~~~~~~~
+

doc/scripts/base/frameworks/notice/actions/add-geodata.bro.rst

@@ -0,0 +1,45 @@
+:tocdepth: 3
+
+base/frameworks/notice/actions/add-geodata.bro
+==============================================
+.. bro:namespace:: Notice
+
+This script adds geographic location data to notices for the "remote"
+host in a connection.  It does make the assumption that one of the 
+addresses in a connection is "local" and one is "remote" which is 
+probably a safe assumption to make in most cases.  If both addresses
+are remote, it will use the $src address.
+
+:Namespace: Notice
+:Imports: :doc:`base/frameworks/notice </scripts/base/frameworks/notice/index>`, :doc:`base/frameworks/notice/main.bro </scripts/base/frameworks/notice/main.bro>`, :doc:`base/utils/site.bro </scripts/base/utils/site.bro>`
+
+Summary
+~~~~~~~
+Runtime Options
+###############
+=========================================================================== ===============================================================
+:bro:id:`Notice::lookup_location_types`: :bro:type:`set` :bro:attr:`&redef` Notice types which should have the "remote" location looked up.
+=========================================================================== ===============================================================
+
+Redefinitions
+#############
+============================================ =
+:bro:type:`Notice::Action`: :bro:type:`enum` 
+:bro:type:`Notice::Info`: :bro:type:`record` 
+============================================ =
+
+
+Detailed Interface
+~~~~~~~~~~~~~~~~~~
+Runtime Options
+###############
+.. bro:id:: Notice::lookup_location_types
+
+   :Type: :bro:type:`set` [:bro:type:`Notice::Type`]
+   :Attributes: :bro:attr:`&redef`
+   :Default: ``{}``
+
+   Notice types which should have the "remote" location looked up.
+   If GeoIP support is not built in, this does nothing.
+
+

doc/scripts/base/frameworks/notice/actions/drop.bro.rst

@@ -0,0 +1,25 @@
+:tocdepth: 3
+
+base/frameworks/notice/actions/drop.bro
+=======================================
+.. bro:namespace:: Notice
+
+This script extends the built in notice code to implement the IP address
+dropping functionality.
+
+:Namespace: Notice
+:Imports: :doc:`base/frameworks/netcontrol </scripts/base/frameworks/netcontrol/index>`, :doc:`base/frameworks/notice/main.bro </scripts/base/frameworks/notice/main.bro>`
+
+Summary
+~~~~~~~
+Redefinitions
+#############
+============================================ =
+:bro:type:`Notice::Action`: :bro:type:`enum` 
+:bro:type:`Notice::Info`: :bro:type:`record` 
+============================================ =
+
+
+Detailed Interface
+~~~~~~~~~~~~~~~~~~
+

doc/scripts/base/frameworks/notice/actions/email_admin.bro.rst

@@ -0,0 +1,26 @@
+:tocdepth: 3
+
+base/frameworks/notice/actions/email_admin.bro
+==============================================
+.. bro:namespace:: Notice
+
+Adds a new notice action type which can be used to email notices
+to the administrators of a particular address space as set by
+:bro:id:`Site::local_admins` if the notice contains a source
+or destination address that lies within their space.
+
+:Namespace: Notice
+:Imports: :doc:`base/frameworks/notice/main.bro </scripts/base/frameworks/notice/main.bro>`, :doc:`base/utils/site.bro </scripts/base/utils/site.bro>`
+
+Summary
+~~~~~~~
+Redefinitions
+#############
+============================================ =
+:bro:type:`Notice::Action`: :bro:type:`enum` 
+============================================ =
+
+
+Detailed Interface
+~~~~~~~~~~~~~~~~~~
+

doc/scripts/base/frameworks/notice/actions/page.bro.rst

@@ -0,0 +1,41 @@
+:tocdepth: 3
+
+base/frameworks/notice/actions/page.bro
+=======================================
+.. bro:namespace:: Notice
+
+Allows configuration of a pager email address to which notices can be sent.
+
+:Namespace: Notice
+:Imports: :doc:`base/frameworks/notice/main.bro </scripts/base/frameworks/notice/main.bro>`
+
+Summary
+~~~~~~~
+Runtime Options
+###############
+======================================================================= ======================================================================
+:bro:id:`Notice::mail_page_dest`: :bro:type:`string` :bro:attr:`&redef` Email address to send notices with the :bro:enum:`Notice::ACTION_PAGE`
+                                                                        action.
+======================================================================= ======================================================================
+
+Redefinitions
+#############
+============================================ =
+:bro:type:`Notice::Action`: :bro:type:`enum` 
+============================================ =
+
+
+Detailed Interface
+~~~~~~~~~~~~~~~~~~
+Runtime Options
+###############
+.. bro:id:: Notice::mail_page_dest
+
+   :Type: :bro:type:`string`
+   :Attributes: :bro:attr:`&redef`
+   :Default: ``""``
+
+   Email address to send notices with the :bro:enum:`Notice::ACTION_PAGE`
+   action.
+
+

doc/scripts/base/frameworks/notice/actions/pp-alarms.bro.rst

@@ -0,0 +1,92 @@
+:tocdepth: 3
+
+base/frameworks/notice/actions/pp-alarms.bro
+============================================
+.. bro:namespace:: Notice
+
+Notice extension that mails out a pretty-printed version of alarm.log
+in regular intervals, formatted for better human readability. If activated,
+that replaces the default summary mail having the raw log output.
+
+:Namespace: Notice
+:Imports: :doc:`base/frameworks/cluster </scripts/base/frameworks/cluster/index>`, :doc:`base/frameworks/notice/main.bro </scripts/base/frameworks/notice/main.bro>`
+
+Summary
+~~~~~~~
+Redefinable Options
+###################
+================================================================================= ==============================================
+:bro:id:`Notice::mail_dest_pretty_printed`: :bro:type:`string` :bro:attr:`&redef` Address to send the pretty-printed reports to.
+:bro:id:`Notice::pretty_print_alarms`: :bro:type:`bool` :bro:attr:`&redef`        Activate pretty-printed alarm summaries.
+================================================================================= ==============================================
+
+State Variables
+###############
+============================================================================ ==================================================================
+:bro:id:`Notice::flag_nets`: :bro:type:`set` :bro:attr:`&redef`              If an address from one of these networks is reported, we mark
+                                                                             the entry with an additional quote symbol (i.e., ">").
+:bro:id:`Notice::force_email_summaries`: :bro:type:`bool` :bro:attr:`&redef` Force generating mail file, even if reading from traces or no mail
+                                                                             destination is defined.
+============================================================================ ==================================================================
+
+Functions
+#########
+============================================================================= =====================================
+:bro:id:`Notice::pretty_print_alarm`: :bro:type:`function` :bro:attr:`&redef` Function that renders a single alarm.
+============================================================================= =====================================
+
+
+Detailed Interface
+~~~~~~~~~~~~~~~~~~
+Redefinable Options
+###################
+.. bro:id:: Notice::mail_dest_pretty_printed
+
+   :Type: :bro:type:`string`
+   :Attributes: :bro:attr:`&redef`
+   :Default: ``""``
+
+   Address to send the pretty-printed reports to. Default if not set is
+   :bro:id:`Notice::mail_dest`.
+   
+   Note that this is overridden by the BroControl MailAlarmsTo option.
+
+.. bro:id:: Notice::pretty_print_alarms
+
+   :Type: :bro:type:`bool`
+   :Attributes: :bro:attr:`&redef`
+   :Default: ``T``
+
+   Activate pretty-printed alarm summaries.
+
+State Variables
+###############
+.. bro:id:: Notice::flag_nets
+
+   :Type: :bro:type:`set` [:bro:type:`subnet`]
+   :Attributes: :bro:attr:`&redef`
+   :Default: ``{}``
+
+   If an address from one of these networks is reported, we mark
+   the entry with an additional quote symbol (i.e., ">"). Many MUAs
+   then highlight such lines differently.
+
+.. bro:id:: Notice::force_email_summaries
+
+   :Type: :bro:type:`bool`
+   :Attributes: :bro:attr:`&redef`
+   :Default: ``F``
+
+   Force generating mail file, even if reading from traces or no mail
+   destination is defined. This is mainly for testing.
+
+Functions
+#########
+.. bro:id:: Notice::pretty_print_alarm
+
+   :Type: :bro:type:`function` (out: :bro:type:`file`, n: :bro:type:`Notice::Info`) : :bro:type:`void`
+   :Attributes: :bro:attr:`&redef`
+
+   Function that renders a single alarm. Can be overridden.
+
+

doc/scripts/base/frameworks/notice/index.rst

@@ -0,0 +1,62 @@
+:orphan:
+
+Package: base/frameworks/notice
+===============================
+
+The notice framework enables Bro to "notice" things which are odd or
+potentially bad, leaving it to the local configuration to define which
+of them are actionable.  This decoupling of detection and reporting allows
+Bro to be customized to the different needs that sites have.
+
+:doc:`/scripts/base/frameworks/notice/__load__.bro`
+
+
+:doc:`/scripts/base/frameworks/notice/main.bro`
+
+   This is the notice framework which enables Bro to "notice" things which
+   are odd or potentially bad.  Decisions of the meaning of various notices
+   need to be done per site because Bro does not ship with assumptions about
+   what is bad activity for sites.  More extensive documentation about using
+   the notice framework can be found in :doc:`/frameworks/notice`.
+
+:doc:`/scripts/base/frameworks/notice/weird.bro`
+
+   This script provides a default set of actions to take for "weird activity"
+   events generated from Bro's event engine.  Weird activity is defined as
+   unusual or exceptional activity that can indicate malformed connections,
+   traffic that doesn't conform to a particular protocol, malfunctioning
+   or misconfigured hardware, or even an attacker attempting to avoid/confuse
+   a sensor.  Without context, it's hard to judge whether a particular
+   category of weird activity is interesting, but this script provides
+   a starting point for the user.
+
+:doc:`/scripts/base/frameworks/notice/actions/drop.bro`
+
+   This script extends the built in notice code to implement the IP address
+   dropping functionality.
+
+:doc:`/scripts/base/frameworks/notice/actions/email_admin.bro`
+
+   Adds a new notice action type which can be used to email notices
+   to the administrators of a particular address space as set by
+   :bro:id:`Site::local_admins` if the notice contains a source
+   or destination address that lies within their space.
+
+:doc:`/scripts/base/frameworks/notice/actions/page.bro`
+
+   Allows configuration of a pager email address to which notices can be sent.
+
+:doc:`/scripts/base/frameworks/notice/actions/add-geodata.bro`
+
+   This script adds geographic location data to notices for the "remote"
+   host in a connection.  It does make the assumption that one of the 
+   addresses in a connection is "local" and one is "remote" which is 
+   probably a safe assumption to make in most cases.  If both addresses
+   are remote, it will use the $src address.
+
+:doc:`/scripts/base/frameworks/notice/actions/pp-alarms.bro`
+
+   Notice extension that mails out a pretty-printed version of alarm.log
+   in regular intervals, formatted for better human readability. If activated,
+   that replaces the default summary mail having the raw log output.
+

doc/scripts/base/frameworks/notice/weird.bro.rst

@@ -0,0 +1,412 @@
+:tocdepth: 3
+
+base/frameworks/notice/weird.bro
+================================
+.. bro:namespace:: Weird
+
+This script provides a default set of actions to take for "weird activity"
+events generated from Bro's event engine.  Weird activity is defined as
+unusual or exceptional activity that can indicate malformed connections,
+traffic that doesn't conform to a particular protocol, malfunctioning
+or misconfigured hardware, or even an attacker attempting to avoid/confuse
+a sensor.  Without context, it's hard to judge whether a particular
+category of weird activity is interesting, but this script provides
+a starting point for the user.
+
+:Namespace: Weird
+:Imports: :doc:`base/frameworks/notice/main.bro </scripts/base/frameworks/notice/main.bro>`, :doc:`base/utils/conn-ids.bro </scripts/base/utils/conn-ids.bro>`, :doc:`base/utils/site.bro </scripts/base/utils/site.bro>`
+
+Summary
+~~~~~~~
+Runtime Options
+###############
+================================================================================ ==============================================================
+:bro:id:`Weird::ignore_hosts`: :bro:type:`set` :bro:attr:`&redef`                To completely ignore a specific weird for a host, add the host
+                                                                                 and weird name into this set.
+:bro:id:`Weird::weird_do_not_ignore_repeats`: :bro:type:`set` :bro:attr:`&redef` Don't ignore repeats for weirds in this set.
+================================================================================ ==============================================================
+
+Redefinable Options
+###################
+================================================================================================================================= ==============================================================
+:bro:id:`Weird::actions`: :bro:type:`table` :bro:attr:`&default` = ``Weird::ACTION_LOG`` :bro:attr:`&optional` :bro:attr:`&redef` A table specifying default/recommended actions per weird type.
+================================================================================================================================= ==============================================================
+
+State Variables
+###############
+============================================================================================================ ====================================================================
+:bro:id:`Weird::did_log`: :bro:type:`set` :bro:attr:`&create_expire` = ``1.0 day`` :bro:attr:`&redef`        A state set which tracks unique weirds solely by name to reduce
+                                                                                                             duplicate logging.
+:bro:id:`Weird::did_notice`: :bro:type:`set` :bro:attr:`&create_expire` = ``1.0 day`` :bro:attr:`&redef`     A state set which tracks unique weirds solely by name to reduce
+                                                                                                             duplicate notices from being raised.
+:bro:id:`Weird::weird_ignore`: :bro:type:`set` :bro:attr:`&create_expire` = ``10.0 mins`` :bro:attr:`&redef` This table is used to track identifier and name pairs that should be
+                                                                                                             temporarily ignored because the problem has already been reported.
+============================================================================================================ ====================================================================
+
+Types
+#####
+=========================================== =======================================================================
+:bro:type:`Weird::Action`: :bro:type:`enum` Types of actions that may be taken when handling weird activity events.
+:bro:type:`Weird::Info`: :bro:type:`record` The record which is used for representing and logging weirds.
+=========================================== =======================================================================
+
+Redefinitions
+#############
+========================================== ====================================
+:bro:type:`Log::ID`: :bro:type:`enum`      The weird logging stream identifier.
+:bro:type:`Notice::Type`: :bro:type:`enum` 
+========================================== ====================================
+
+Events
+######
+============================================= ==============================================================
+:bro:id:`Weird::log_weird`: :bro:type:`event` Handlers of this event are invoked once per write to the weird
+                                              logging stream before the data is actually written.
+============================================= ==============================================================
+
+Functions
+#########
+============================================ =
+:bro:id:`Weird::weird`: :bro:type:`function` 
+============================================ =
+
+
+Detailed Interface
+~~~~~~~~~~~~~~~~~~
+Runtime Options
+###############
+.. bro:id:: Weird::ignore_hosts
+
+   :Type: :bro:type:`set` [:bro:type:`addr`, :bro:type:`string`]
+   :Attributes: :bro:attr:`&redef`
+   :Default: ``{}``
+
+   To completely ignore a specific weird for a host, add the host
+   and weird name into this set.
+
+.. bro:id:: Weird::weird_do_not_ignore_repeats
+
+   :Type: :bro:type:`set` [:bro:type:`string`]
+   :Attributes: :bro:attr:`&redef`
+   :Default:
+
+   ::
+
+      {
+         "bad_ICMP_checksum",
+         "bad_TCP_checksum",
+         "bad_IP_checksum",
+         "bad_UDP_checksum"
+      }
+
+   Don't ignore repeats for weirds in this set.  For example,
+   it's handy keeping track of clustered checksum errors.
+
+Redefinable Options
+###################
+.. bro:id:: Weird::actions
+
+   :Type: :bro:type:`table` [:bro:type:`string`] of :bro:type:`Weird::Action`
+   :Attributes: :bro:attr:`&default` = ``Weird::ACTION_LOG`` :bro:attr:`&optional` :bro:attr:`&redef`
+   :Default:
+
+   ::
+
+      {
+         ["DNS_AAAA_neg_length"] = Weird::ACTION_LOG,
+         ["partial_ftp_request"] = Weird::ACTION_LOG,
+         ["repeated_SYN_reply_wo_ack"] = Weird::ACTION_LOG,
+         ["bad_UDP_checksum"] = Weird::ACTION_LOG_PER_ORIG,
+         ["line_terminated_with_single_LF"] = Weird::ACTION_LOG,
+         ["truncated_IP"] = Weird::ACTION_LOG,
+         ["DNS_truncated_len_lt_hdr_len"] = Weird::ACTION_LOG,
+         ["excessive_data_without_further_acks"] = Weird::ACTION_LOG,
+         ["pop3_malformed_auth_plain"] = Weird::ACTION_LOG,
+         ["excess_netbios_hdr_len"] = Weird::ACTION_LOG,
+         ["irc_invalid_whois_channel_line"] = Weird::ACTION_LOG,
+         ["bad_RPC"] = Weird::ACTION_LOG_PER_ORIG,
+         ["unknown_netbios_type"] = Weird::ACTION_LOG,
+         ["HTTP_chunked_transfer_for_multipart_message"] = Weird::ACTION_LOG,
+         ["RST_storm"] = Weird::ACTION_LOG,
+         ["bad_IP_checksum"] = Weird::ACTION_LOG_PER_ORIG,
+         ["excessively_small_fragment"] = Weird::ACTION_LOG_PER_ORIG,
+         ["bad_rsh_prolog"] = Weird::ACTION_LOG,
+         ["pop3_client_sending_server_commands"] = Weird::ACTION_LOG,
+         ["unexpected_multiple_HTTP_requests"] = Weird::ACTION_LOG,
+         ["irc_invalid_topic_reply"] = Weird::ACTION_LOG,
+         ["irc_invalid_squery_message_format"] = Weird::ACTION_LOG,
+         ["bad_SYN_ack"] = Weird::ACTION_LOG,
+         ["contentline_size_exceeded"] = Weird::ACTION_LOG,
+         ["above_hole_data_without_any_acks"] = Weird::ACTION_LOG,
+         ["bad_HTTP_reply"] = Weird::ACTION_LOG,
+         ["DNS_RR_length_mismatch"] = Weird::ACTION_LOG,
+         ["SMB_parsing_error"] = Weird::ACTION_LOG,
+         ["multiple_HTTP_request_elements"] = Weird::ACTION_LOG,
+         ["FIN_after_reset"] = Weird::ACTION_IGNORE,
+         ["SYN_after_partial"] = Weird::ACTION_NOTICE_PER_ORIG,
+         ["baroque_SYN"] = Weird::ACTION_LOG,
+         ["DNS_label_forward_compress_offset"] = Weird::ACTION_LOG_PER_ORIG,
+         ["connection_originator_SYN_ack"] = Weird::ACTION_LOG_PER_ORIG,
+         ["irc_invalid_dcc_message_format"] = Weird::ACTION_LOG,
+         ["unmatched_HTTP_reply"] = Weird::ACTION_LOG,
+         ["unpaired_RPC_response"] = Weird::ACTION_LOG,
+         ["SYN_inside_connection"] = Weird::ACTION_LOG,
+         ["irc_invalid_who_message_format"] = Weird::ACTION_LOG,
+         ["irc_invalid_reply_number"] = Weird::ACTION_LOG,
+         ["pop3_client_command_unknown"] = Weird::ACTION_LOG,
+         ["bad_ICMP_checksum"] = Weird::ACTION_LOG_PER_ORIG,
+         ["DNS_RR_unknown_type"] = Weird::ACTION_LOG,
+         ["excessively_large_fragment"] = Weird::ACTION_LOG,
+         ["DNS_label_len_gt_name_len"] = Weird::ACTION_LOG_PER_ORIG,
+         ["DNS_label_len_gt_pkt"] = Weird::ACTION_LOG_PER_ORIG,
+         ["partial_ident_request"] = Weird::ACTION_LOG,
+         ["excess_RPC"] = Weird::ACTION_LOG_PER_ORIG,
+         ["line_terminated_with_single_CR"] = Weird::ACTION_LOG,
+         ["unknown_HTTP_method"] = Weird::ACTION_LOG,
+         ["bad_ident_request"] = Weird::ACTION_LOG,
+         ["crud_trailing_HTTP_request"] = Weird::ACTION_LOG,
+         ["irc_invalid_whois_operator_line"] = Weird::ACTION_LOG,
+         ["unexpected_server_HTTP_data"] = Weird::ACTION_LOG,
+         ["irc_invalid_njoin_line"] = Weird::ACTION_LOG,
+         ["irc_invalid_mode_message_format"] = Weird::ACTION_LOG,
+         ["pop3_bad_base64_encoding"] = Weird::ACTION_LOG,
+         ["responder_RPC_call"] = Weird::ACTION_LOG_PER_ORIG,
+         ["fragment_size_inconsistency"] = Weird::ACTION_LOG_PER_ORIG,
+         ["successful_RPC_reply_to_invalid_request"] = Weird::ACTION_NOTICE_PER_ORIG,
+         ["irc_line_too_short"] = Weird::ACTION_LOG,
+         ["irc_invalid_kick_message_format"] = Weird::ACTION_LOG,
+         ["repeated_SYN_with_ack"] = Weird::ACTION_LOG,
+         ["partial_finger_request"] = Weird::ACTION_LOG,
+         ["irc_invalid_join_line"] = Weird::ACTION_LOG,
+         ["premature_connection_reuse"] = Weird::ACTION_LOG,
+         ["netbios_raw_session_msg"] = Weird::ACTION_LOG,
+         ["incompletely_captured_fragment"] = Weird::ACTION_LOG,
+         ["malformed_ssh_version"] = Weird::ACTION_LOG,
+         ["netbios_client_session_reply"] = Weird::ACTION_LOG,
+         ["bad_TCP_header_len"] = Weird::ACTION_LOG,
+         ["unescaped_%_in_URI"] = Weird::ACTION_LOG,
+         ["netbios_server_session_request"] = Weird::ACTION_LOG,
+         ["irc_too_many_invalid"] = Weird::ACTION_LOG,
+         ["irc_invalid_names_line"] = Weird::ACTION_LOG,
+         ["RPC_rexmit_inconsistency"] = Weird::ACTION_LOG,
+         ["smb_andx_command_failed_to_parse"] = Weird::ACTION_LOG,
+         ["irc_invalid_invite_message_format"] = Weird::ACTION_LOG,
+         ["spontaneous_FIN"] = Weird::ACTION_IGNORE,
+         ["DNS_truncated_quest_too_short"] = Weird::ACTION_LOG,
+         ["SSL_many_server_names"] = Weird::ACTION_LOG,
+         ["FIN_storm"] = Weird::ACTION_NOTICE_PER_ORIG,
+         ["data_before_established"] = Weird::ACTION_LOG,
+         ["SYN_after_reset"] = Weird::ACTION_LOG,
+         ["double_%_in_URI"] = Weird::ACTION_LOG,
+         ["DNS_truncated_ans_too_short"] = Weird::ACTION_LOG,
+         ["DNS_Conn_count_too_large"] = Weird::ACTION_LOG,
+         ["data_after_reset"] = Weird::ACTION_LOG,
+         ["RPC_underflow"] = Weird::ACTION_LOG,
+         ["unexpected_client_HTTP_data"] = Weird::ACTION_LOG,
+         ["originator_RPC_reply"] = Weird::ACTION_LOG_PER_ORIG,
+         ["DNS_label_too_long"] = Weird::ACTION_LOG_PER_ORIG,
+         ["SYN_with_data"] = Weird::ACTION_LOG_PER_ORIG,
+         ["RST_with_data"] = Weird::ACTION_LOG,
+         ["bad_HTTP_version"] = Weird::ACTION_LOG,
+         ["pending_data_when_closed"] = Weird::ACTION_LOG,
+         ["rlogin_text_after_rejected"] = Weird::ACTION_LOG,
+         ["FIN_advanced_last_seq"] = Weird::ACTION_LOG,
+         ["transaction_subcmd_missing"] = Weird::ACTION_LOG,
+         ["fragment_protocol_inconsistency"] = Weird::ACTION_LOG,
+         ["invalid_irc_global_users_reply"] = Weird::ACTION_LOG,
+         ["ident_request_addendum"] = Weird::ACTION_LOG,
+         ["window_recision"] = Weird::ACTION_LOG,
+         ["spontaneous_RST"] = Weird::ACTION_IGNORE,
+         ["truncated_header"] = Weird::ACTION_LOG,
+         ["UDP_datagram_length_mismatch"] = Weird::ACTION_LOG_PER_ORIG,
+         ["fragment_with_DF"] = Weird::ACTION_LOG,
+         ["SYN_after_close"] = Weird::ACTION_LOG,
+         ["SYN_seq_jump"] = Weird::ACTION_LOG,
+         ["irc_invalid_notice_message_format"] = Weird::ACTION_LOG,
+         ["irc_invalid_command"] = Weird::ACTION_LOG,
+         ["DNS_NAME_too_long"] = Weird::ACTION_LOG,
+         ["inflate_failed"] = Weird::ACTION_LOG,
+         ["base64_illegal_encoding"] = Weird::ACTION_LOG,
+         ["internally_truncated_header"] = Weird::ACTION_LOG,
+         ["pop3_server_sending_client_commands"] = Weird::ACTION_LOG,
+         ["irc_invalid_who_line"] = Weird::ACTION_LOG,
+         ["irc_invalid_privmsg_message_format"] = Weird::ACTION_LOG,
+         ["pop3_server_command_unknown"] = Weird::ACTION_LOG,
+         ["fragment_overlap"] = Weird::ACTION_LOG_PER_ORIG,
+         ["bad_rlogin_prolog"] = Weird::ACTION_LOG,
+         ["bad_ident_port"] = Weird::ACTION_LOG,
+         ["irc_invalid_line"] = Weird::ACTION_LOG,
+         ["HTTP_overlapping_messages"] = Weird::ACTION_LOG,
+         ["simultaneous_open"] = Weird::ACTION_LOG_PER_CONN,
+         ["unsolicited_SYN_response"] = Weird::ACTION_IGNORE,
+         ["DNS_RR_bad_length"] = Weird::ACTION_LOG,
+         ["TCP_christmas"] = Weird::ACTION_LOG,
+         ["inappropriate_FIN"] = Weird::ACTION_LOG,
+         ["irc_invalid_oper_message_format"] = Weird::ACTION_LOG,
+         ["no_smb_session_using_parsesambamsg"] = Weird::ACTION_LOG,
+         ["illegal_%_at_end_of_URI"] = Weird::ACTION_LOG,
+         ["active_connection_reuse"] = Weird::ACTION_LOG,
+         ["bad_TCP_checksum"] = Weird::ACTION_LOG_PER_ORIG,
+         ["fragment_inconsistency"] = Weird::ACTION_LOG_PER_ORIG,
+         ["malformed_ssh_identification"] = Weird::ACTION_LOG,
+         ["DNS_truncated_RR_rdlength_lt_len"] = Weird::ACTION_LOG,
+         ["possible_split_routing"] = Weird::ACTION_LOG,
+         ["irc_line_size_exceeded"] = Weird::ACTION_LOG,
+         ["bad_RPC_program"] = Weird::ACTION_LOG,
+         ["bad_ident_reply"] = Weird::ACTION_LOG,
+         ["HTTP_bad_chunk_size"] = Weird::ACTION_LOG,
+         ["unescaped_special_URI_char"] = Weird::ACTION_LOG,
+         ["HTTP_version_mismatch"] = Weird::ACTION_LOG,
+         ["irc_invalid_whois_message_format"] = Weird::ACTION_LOG,
+         ["rsh_text_after_rejected"] = Weird::ACTION_LOG,
+         ["partial_RPC"] = Weird::ACTION_LOG_PER_ORIG,
+         ["truncated_ARP"] = Weird::ACTION_LOG,
+         ["truncated_NTP"] = Weird::ACTION_LOG,
+         ["irc_invalid_whois_user_line"] = Weird::ACTION_LOG,
+         ["NUL_in_line"] = Weird::ACTION_LOG,
+         ["deficit_netbios_hdr_len"] = Weird::ACTION_LOG
+      }
+
+   A table specifying default/recommended actions per weird type.
+
+State Variables
+###############
+.. bro:id:: Weird::did_log
+
+   :Type: :bro:type:`set` [:bro:type:`string`, :bro:type:`string`]
+   :Attributes: :bro:attr:`&create_expire` = ``1.0 day`` :bro:attr:`&redef`
+   :Default: ``{}``
+
+   A state set which tracks unique weirds solely by name to reduce
+   duplicate logging.  This is deliberately not synchronized because it
+   could cause overload during storms.
+
+.. bro:id:: Weird::did_notice
+
+   :Type: :bro:type:`set` [:bro:type:`string`, :bro:type:`string`]
+   :Attributes: :bro:attr:`&create_expire` = ``1.0 day`` :bro:attr:`&redef`
+   :Default: ``{}``
+
+   A state set which tracks unique weirds solely by name to reduce
+   duplicate notices from being raised.
+
+.. bro:id:: Weird::weird_ignore
+
+   :Type: :bro:type:`set` [:bro:type:`string`, :bro:type:`string`]
+   :Attributes: :bro:attr:`&create_expire` = ``10.0 mins`` :bro:attr:`&redef`
+   :Default: ``{}``
+
+   This table is used to track identifier and name pairs that should be
+   temporarily ignored because the problem has already been reported.
+   This helps reduce the volume of high volume weirds by only allowing 
+   a unique weird every ``create_expire`` interval.
+
+Types
+#####
+.. bro:type:: Weird::Action
+
+   :Type: :bro:type:`enum`
+
+      .. bro:enum:: Weird::ACTION_UNSPECIFIED Weird::Action
+
+         A dummy action indicating the user does not care what
+         internal decision is made regarding a given type of weird.
+
+      .. bro:enum:: Weird::ACTION_IGNORE Weird::Action
+
+         No action is to be taken.
+
+      .. bro:enum:: Weird::ACTION_LOG Weird::Action
+
+         Log the weird event every time it occurs.
+
+      .. bro:enum:: Weird::ACTION_LOG_ONCE Weird::Action
+
+         Log the weird event only once.
+
+      .. bro:enum:: Weird::ACTION_LOG_PER_CONN Weird::Action
+
+         Log the weird event once per connection.
+
+      .. bro:enum:: Weird::ACTION_LOG_PER_ORIG Weird::Action
+
+         Log the weird event once per originator host.
+
+      .. bro:enum:: Weird::ACTION_NOTICE Weird::Action
+
+         Always generate a notice associated with the weird event.
+
+      .. bro:enum:: Weird::ACTION_NOTICE_ONCE Weird::Action
+
+         Generate a notice associated with the weird event only once.
+
+      .. bro:enum:: Weird::ACTION_NOTICE_PER_CONN Weird::Action
+
+         Generate a notice for the weird event once per connection.
+
+      .. bro:enum:: Weird::ACTION_NOTICE_PER_ORIG Weird::Action
+
+         Generate a notice for the weird event once per originator host.
+
+   Types of actions that may be taken when handling weird activity events.
+
+.. bro:type:: Weird::Info
+
+   :Type: :bro:type:`record`
+
+      ts: :bro:type:`time` :bro:attr:`&log`
+         The time when the weird occurred.
+
+      uid: :bro:type:`string` :bro:attr:`&log` :bro:attr:`&optional`
+         If a connection is associated with this weird, this will be
+         the connection's unique ID.
+
+      id: :bro:type:`conn_id` :bro:attr:`&log` :bro:attr:`&optional`
+         conn_id for the optional connection.
+
+      conn: :bro:type:`connection` :bro:attr:`&optional`
+         A shorthand way of giving the uid and id to a weird.
+
+      name: :bro:type:`string` :bro:attr:`&log`
+         The name of the weird that occurred.
+
+      addl: :bro:type:`string` :bro:attr:`&log` :bro:attr:`&optional`
+         Additional information accompanying the weird if any.
+
+      notice: :bro:type:`bool` :bro:attr:`&log` :bro:attr:`&default` = ``F`` :bro:attr:`&optional`
+         Indicate if this weird was also turned into a notice.
+
+      peer: :bro:type:`string` :bro:attr:`&log` :bro:attr:`&optional` :bro:attr:`&default` = :bro:see:`peer_description`
+         The peer that originated this weird.  This is helpful in
+         cluster deployments if a particular cluster node is having
+         trouble to help identify which node is having trouble.
+
+      identifier: :bro:type:`string` :bro:attr:`&optional`
+         This field is to be provided when a weird is generated for
+         the purpose of deduplicating weirds. The identifier string
+         should be unique for a single instance of the weird. This field
+         is used to define when a weird is conceptually a duplicate of
+         a previous weird.
+
+   The record which is used for representing and logging weirds.
+
+Events
+######
+.. bro:id:: Weird::log_weird
+
+   :Type: :bro:type:`event` (rec: :bro:type:`Weird::Info`)
+
+   Handlers of this event are invoked once per write to the weird
+   logging stream before the data is actually written.
+   
+
+   :rec: The weird columns about to be logged to the weird stream.
+
+Functions
+#########
+.. bro:id:: Weird::weird
+
+   :Type: :bro:type:`function` (w: :bro:type:`Weird::Info`) : :bro:type:`void`
+
+
+

doc/scripts/base/frameworks/openflow/__load__.bro.rst

@@ -0,0 +1,14 @@
+:tocdepth: 3
+
+base/frameworks/openflow/__load__.bro
+=====================================
+
+
+:Imports: :doc:`base/frameworks/cluster </scripts/base/frameworks/cluster/index>`, :doc:`base/frameworks/openflow/consts.bro </scripts/base/frameworks/openflow/consts.bro>`, :doc:`base/frameworks/openflow/main.bro </scripts/base/frameworks/openflow/main.bro>`, :doc:`base/frameworks/openflow/non-cluster.bro </scripts/base/frameworks/openflow/non-cluster.bro>`, :doc:`base/frameworks/openflow/plugins </scripts/base/frameworks/openflow/plugins/index>`, :doc:`base/frameworks/openflow/types.bro </scripts/base/frameworks/openflow/types.bro>`
+
+Summary
+~~~~~~~
+
+Detailed Interface
+~~~~~~~~~~~~~~~~~~
+

doc/scripts/base/frameworks/openflow/consts.bro.rst

@@ -0,0 +1,564 @@
+:tocdepth: 3
+
+base/frameworks/openflow/consts.bro
+===================================
+.. bro:namespace:: OpenFlow
+
+Constants used by the OpenFlow framework.
+
+:Namespace: OpenFlow
+
+Summary
+~~~~~~~
+Constants
+#########
+============================================================= ======================================================================
+:bro:id:`OpenFlow::ETH_APPLETALK`: :bro:type:`count`          
+:bro:id:`OpenFlow::ETH_APPLETALK_ARP`: :bro:type:`count`      
+:bro:id:`OpenFlow::ETH_ARP`: :bro:type:`count`                
+:bro:id:`OpenFlow::ETH_EAP_OVER_LAN`: :bro:type:`count`       
+:bro:id:`OpenFlow::ETH_ETHER_FLOW_CONTROL`: :bro:type:`count` 
+:bro:id:`OpenFlow::ETH_IPX`: :bro:type:`count`                
+:bro:id:`OpenFlow::ETH_IPX_OLD`: :bro:type:`count`            
+:bro:id:`OpenFlow::ETH_IPv4`: :bro:type:`count`               
+:bro:id:`OpenFlow::ETH_IPv6`: :bro:type:`count`               
+:bro:id:`OpenFlow::ETH_JUMBO_FRAMES`: :bro:type:`count`       
+:bro:id:`OpenFlow::ETH_MAC_SECURITY`: :bro:type:`count`       
+:bro:id:`OpenFlow::ETH_MPLS_MULTICAST`: :bro:type:`count`     
+:bro:id:`OpenFlow::ETH_MPLS_UNICAST`: :bro:type:`count`       
+:bro:id:`OpenFlow::ETH_PPPOE_DISCOVERY`: :bro:type:`count`    
+:bro:id:`OpenFlow::ETH_PPPOE_SESSION`: :bro:type:`count`      
+:bro:id:`OpenFlow::ETH_PROVIDER_BRIDING`: :bro:type:`count`   
+:bro:id:`OpenFlow::ETH_QINQ`: :bro:type:`count`               
+:bro:id:`OpenFlow::ETH_RARP`: :bro:type:`count`               
+:bro:id:`OpenFlow::ETH_VLAN`: :bro:type:`count`               
+:bro:id:`OpenFlow::ETH_WOL`: :bro:type:`count`                
+:bro:id:`OpenFlow::INVALID_COOKIE`: :bro:type:`count`         Return value for a cookie from a flow
+                                                              which is not added, modified or deleted
+                                                              from the bro openflow framework.
+:bro:id:`OpenFlow::IP_CBT`: :bro:type:`count`                 
+:bro:id:`OpenFlow::IP_EGP`: :bro:type:`count`                 
+:bro:id:`OpenFlow::IP_ETHERIP`: :bro:type:`count`             
+:bro:id:`OpenFlow::IP_FC`: :bro:type:`count`                  
+:bro:id:`OpenFlow::IP_GGP`: :bro:type:`count`                 
+:bro:id:`OpenFlow::IP_GRE`: :bro:type:`count`                 
+:bro:id:`OpenFlow::IP_HOPOPT`: :bro:type:`count`              
+:bro:id:`OpenFlow::IP_ICMP`: :bro:type:`count`                
+:bro:id:`OpenFlow::IP_IGMP`: :bro:type:`count`                
+:bro:id:`OpenFlow::IP_IGP`: :bro:type:`count`                 
+:bro:id:`OpenFlow::IP_IPIP`: :bro:type:`count`                
+:bro:id:`OpenFlow::IP_IPv6`: :bro:type:`count`                
+:bro:id:`OpenFlow::IP_ISIS`: :bro:type:`count`                
+:bro:id:`OpenFlow::IP_L2TP`: :bro:type:`count`                
+:bro:id:`OpenFlow::IP_MPLS`: :bro:type:`count`                
+:bro:id:`OpenFlow::IP_MTP`: :bro:type:`count`                 
+:bro:id:`OpenFlow::IP_OSPF`: :bro:type:`count`                
+:bro:id:`OpenFlow::IP_RDP`: :bro:type:`count`                 
+:bro:id:`OpenFlow::IP_RSVP`: :bro:type:`count`                
+:bro:id:`OpenFlow::IP_ST`: :bro:type:`count`                  
+:bro:id:`OpenFlow::IP_TCP`: :bro:type:`count`                 
+:bro:id:`OpenFlow::IP_UDP`: :bro:type:`count`                 
+:bro:id:`OpenFlow::OFPFF_CHECK_OVERLAP`: :bro:type:`count`    Check for overlapping entries first.
+:bro:id:`OpenFlow::OFPFF_EMERG`: :bro:type:`count`            Remark this is for emergency.
+:bro:id:`OpenFlow::OFPFF_SEND_FLOW_REM`: :bro:type:`count`    Send flow removed message when flow
+                                                              expires or is deleted.
+:bro:id:`OpenFlow::OFPP_ALL`: :bro:type:`count`               All physical ports except input port.
+:bro:id:`OpenFlow::OFPP_ANY`: :bro:type:`count`               Wildcard port used only for flow mod (delete) and flow stats requests.
+:bro:id:`OpenFlow::OFPP_CONTROLLER`: :bro:type:`count`        Send to controller.
+:bro:id:`OpenFlow::OFPP_FLOOD`: :bro:type:`count`             All physical ports except input port and
+                                                              those disabled by STP.
+:bro:id:`OpenFlow::OFPP_IN_PORT`: :bro:type:`count`           Send the packet out the input port.
+:bro:id:`OpenFlow::OFPP_LOCAL`: :bro:type:`count`             Local openflow "port".
+:bro:id:`OpenFlow::OFPP_NORMAL`: :bro:type:`count`            Process with normal L2/L3 switching.
+:bro:id:`OpenFlow::OFPP_TABLE`: :bro:type:`count`             Perform actions in flow table.
+:bro:id:`OpenFlow::OFPTT_ALL`: :bro:type:`count`              
+:bro:id:`OpenFlow::OFP_NO_BUFFER`: :bro:type:`count`          
+============================================================= ======================================================================
+
+Types
+#####
+============================================================ ======================================
+:bro:type:`OpenFlow::ofp_action_type`: :bro:type:`enum`      Openflow action_type definitions.
+:bro:type:`OpenFlow::ofp_config_flags`: :bro:type:`enum`     Openflow config flag definitions.
+:bro:type:`OpenFlow::ofp_flow_mod_command`: :bro:type:`enum` Openflow flow_mod_command definitions.
+============================================================ ======================================
+
+
+Detailed Interface
+~~~~~~~~~~~~~~~~~~
+Constants
+#########
+.. bro:id:: OpenFlow::ETH_APPLETALK
+
+   :Type: :bro:type:`count`
+   :Default: ``32923``
+
+
+.. bro:id:: OpenFlow::ETH_APPLETALK_ARP
+
+   :Type: :bro:type:`count`
+   :Default: ``33011``
+
+
+.. bro:id:: OpenFlow::ETH_ARP
+
+   :Type: :bro:type:`count`
+   :Default: ``2054``
+
+
+.. bro:id:: OpenFlow::ETH_EAP_OVER_LAN
+
+   :Type: :bro:type:`count`
+   :Default: ``34958``
+
+
+.. bro:id:: OpenFlow::ETH_ETHER_FLOW_CONTROL
+
+   :Type: :bro:type:`count`
+   :Default: ``34824``
+
+
+.. bro:id:: OpenFlow::ETH_IPX
+
+   :Type: :bro:type:`count`
+   :Default: ``33080``
+
+
+.. bro:id:: OpenFlow::ETH_IPX_OLD
+
+   :Type: :bro:type:`count`
+   :Default: ``33079``
+
+
+.. bro:id:: OpenFlow::ETH_IPv4
+
+   :Type: :bro:type:`count`
+   :Default: ``2048``
+
+
+.. bro:id:: OpenFlow::ETH_IPv6
+
+   :Type: :bro:type:`count`
+   :Default: ``34525``
+
+
+.. bro:id:: OpenFlow::ETH_JUMBO_FRAMES
+
+   :Type: :bro:type:`count`
+   :Default: ``34928``
+
+
+.. bro:id:: OpenFlow::ETH_MAC_SECURITY
+
+   :Type: :bro:type:`count`
+   :Default: ``35045``
+
+
+.. bro:id:: OpenFlow::ETH_MPLS_MULTICAST
+
+   :Type: :bro:type:`count`
+   :Default: ``34888``
+
+
+.. bro:id:: OpenFlow::ETH_MPLS_UNICAST
+
+   :Type: :bro:type:`count`
+   :Default: ``34887``
+
+
+.. bro:id:: OpenFlow::ETH_PPPOE_DISCOVERY
+
+   :Type: :bro:type:`count`
+   :Default: ``34915``
+
+
+.. bro:id:: OpenFlow::ETH_PPPOE_SESSION
+
+   :Type: :bro:type:`count`
+   :Default: ``34916``
+
+
+.. bro:id:: OpenFlow::ETH_PROVIDER_BRIDING
+
+   :Type: :bro:type:`count`
+   :Default: ``34984``
+
+
+.. bro:id:: OpenFlow::ETH_QINQ
+
+   :Type: :bro:type:`count`
+   :Default: ``37120``
+
+
+.. bro:id:: OpenFlow::ETH_RARP
+
+   :Type: :bro:type:`count`
+   :Default: ``32821``
+
+
+.. bro:id:: OpenFlow::ETH_VLAN
+
+   :Type: :bro:type:`count`
+   :Default: ``33024``
+
+
+.. bro:id:: OpenFlow::ETH_WOL
+
+   :Type: :bro:type:`count`
+   :Default: ``2114``
+
+
+.. bro:id:: OpenFlow::INVALID_COOKIE
+
+   :Type: :bro:type:`count`
+   :Default: ``18446744073709551615``
+
+   Return value for a cookie from a flow
+   which is not added, modified or deleted
+   from the bro openflow framework.
+
+.. bro:id:: OpenFlow::IP_CBT
+
+   :Type: :bro:type:`count`
+   :Default: ``7``
+
+
+.. bro:id:: OpenFlow::IP_EGP
+
+   :Type: :bro:type:`count`
+   :Default: ``8``
+
+
+.. bro:id:: OpenFlow::IP_ETHERIP
+
+   :Type: :bro:type:`count`
+   :Default: ``97``
+
+
+.. bro:id:: OpenFlow::IP_FC
+
+   :Type: :bro:type:`count`
+   :Default: ``133``
+
+
+.. bro:id:: OpenFlow::IP_GGP
+
+   :Type: :bro:type:`count`
+   :Default: ``3``
+
+
+.. bro:id:: OpenFlow::IP_GRE
+
+   :Type: :bro:type:`count`
+   :Default: ``47``
+
+
+.. bro:id:: OpenFlow::IP_HOPOPT
+
+   :Type: :bro:type:`count`
+   :Default: ``0``
+
+
+.. bro:id:: OpenFlow::IP_ICMP
+
+   :Type: :bro:type:`count`
+   :Default: ``1``
+
+
+.. bro:id:: OpenFlow::IP_IGMP
+
+   :Type: :bro:type:`count`
+   :Default: ``2``
+
+
+.. bro:id:: OpenFlow::IP_IGP
+
+   :Type: :bro:type:`count`
+   :Default: ``9``
+
+
+.. bro:id:: OpenFlow::IP_IPIP
+
+   :Type: :bro:type:`count`
+   :Default: ``4``
+
+
+.. bro:id:: OpenFlow::IP_IPv6
+
+   :Type: :bro:type:`count`
+   :Default: ``41``
+
+
+.. bro:id:: OpenFlow::IP_ISIS
+
+   :Type: :bro:type:`count`
+   :Default: ``124``
+
+
+.. bro:id:: OpenFlow::IP_L2TP
+
+   :Type: :bro:type:`count`
+   :Default: ``115``
+
+
+.. bro:id:: OpenFlow::IP_MPLS
+
+   :Type: :bro:type:`count`
+   :Default: ``137``
+
+
+.. bro:id:: OpenFlow::IP_MTP
+
+   :Type: :bro:type:`count`
+   :Default: ``92``
+
+
+.. bro:id:: OpenFlow::IP_OSPF
+
+   :Type: :bro:type:`count`
+   :Default: ``89``
+
+
+.. bro:id:: OpenFlow::IP_RDP
+
+   :Type: :bro:type:`count`
+   :Default: ``27``
+
+
+.. bro:id:: OpenFlow::IP_RSVP
+
+   :Type: :bro:type:`count`
+   :Default: ``46``
+
+
+.. bro:id:: OpenFlow::IP_ST
+
+   :Type: :bro:type:`count`
+   :Default: ``5``
+
+
+.. bro:id:: OpenFlow::IP_TCP
+
+   :Type: :bro:type:`count`
+   :Default: ``6``
+
+
+.. bro:id:: OpenFlow::IP_UDP
+
+   :Type: :bro:type:`count`
+   :Default: ``17``
+
+
+.. bro:id:: OpenFlow::OFPFF_CHECK_OVERLAP
+
+   :Type: :bro:type:`count`
+   :Default: ``2``
+
+   Check for overlapping entries first.
+
+.. bro:id:: OpenFlow::OFPFF_EMERG
+
+   :Type: :bro:type:`count`
+   :Default: ``4``
+
+   Remark this is for emergency.
+   Flows added with this are only used
+   when the controller is disconnected.
+
+.. bro:id:: OpenFlow::OFPFF_SEND_FLOW_REM
+
+   :Type: :bro:type:`count`
+   :Default: ``1``
+
+   Send flow removed message when flow
+   expires or is deleted.
+
+.. bro:id:: OpenFlow::OFPP_ALL
+
+   :Type: :bro:type:`count`
+   :Default: ``4294967292``
+
+   All physical ports except input port.
+
+.. bro:id:: OpenFlow::OFPP_ANY
+
+   :Type: :bro:type:`count`
+   :Default: ``4294967295``
+
+   Wildcard port used only for flow mod (delete) and flow stats requests.
+
+.. bro:id:: OpenFlow::OFPP_CONTROLLER
+
+   :Type: :bro:type:`count`
+   :Default: ``4294967293``
+
+   Send to controller.
+
+.. bro:id:: OpenFlow::OFPP_FLOOD
+
+   :Type: :bro:type:`count`
+   :Default: ``4294967291``
+
+   All physical ports except input port and
+   those disabled by STP.
+
+.. bro:id:: OpenFlow::OFPP_IN_PORT
+
+   :Type: :bro:type:`count`
+   :Default: ``4294967288``
+
+   Send the packet out the input port. This
+   virual port must be explicitly used in
+   order to send back out of the input port.
+
+.. bro:id:: OpenFlow::OFPP_LOCAL
+
+   :Type: :bro:type:`count`
+   :Default: ``4294967294``
+
+   Local openflow "port".
+
+.. bro:id:: OpenFlow::OFPP_NORMAL
+
+   :Type: :bro:type:`count`
+   :Default: ``4294967290``
+
+   Process with normal L2/L3 switching.
+
+.. bro:id:: OpenFlow::OFPP_TABLE
+
+   :Type: :bro:type:`count`
+   :Default: ``4294967289``
+
+   Perform actions in flow table.
+   NB: This can only be the destination port
+   for packet-out messages.
+
+.. bro:id:: OpenFlow::OFPTT_ALL
+
+   :Type: :bro:type:`count`
+   :Default: ``255``
+
+
+.. bro:id:: OpenFlow::OFP_NO_BUFFER
+
+   :Type: :bro:type:`count`
+   :Default: ``4294967295``
+
+
+Types
+#####
+.. bro:type:: OpenFlow::ofp_action_type
+
+   :Type: :bro:type:`enum`
+
+      .. bro:enum:: OpenFlow::OFPAT_OUTPUT OpenFlow::ofp_action_type
+
+         Output to switch port.
+
+      .. bro:enum:: OpenFlow::OFPAT_SET_VLAN_VID OpenFlow::ofp_action_type
+
+         Set the 802.1q VLAN id.
+
+      .. bro:enum:: OpenFlow::OFPAT_SET_VLAN_PCP OpenFlow::ofp_action_type
+
+         Set the 802.1q priority.
+
+      .. bro:enum:: OpenFlow::OFPAT_STRIP_VLAN OpenFlow::ofp_action_type
+
+         Strip the 802.1q header.
+
+      .. bro:enum:: OpenFlow::OFPAT_SET_DL_SRC OpenFlow::ofp_action_type
+
+         Ethernet source address.
+
+      .. bro:enum:: OpenFlow::OFPAT_SET_DL_DST OpenFlow::ofp_action_type
+
+         Ethernet destination address.
+
+      .. bro:enum:: OpenFlow::OFPAT_SET_NW_SRC OpenFlow::ofp_action_type
+
+         IP source address.
+
+      .. bro:enum:: OpenFlow::OFPAT_SET_NW_DST OpenFlow::ofp_action_type
+
+         IP destination address.
+
+      .. bro:enum:: OpenFlow::OFPAT_SET_NW_TOS OpenFlow::ofp_action_type
+
+         IP ToS (DSCP field, 6 bits).
+
+      .. bro:enum:: OpenFlow::OFPAT_SET_TP_SRC OpenFlow::ofp_action_type
+
+         TCP/UDP source port.
+
+      .. bro:enum:: OpenFlow::OFPAT_SET_TP_DST OpenFlow::ofp_action_type
+
+         TCP/UDP destination port.
+
+      .. bro:enum:: OpenFlow::OFPAT_ENQUEUE OpenFlow::ofp_action_type
+
+         Output to queue.
+
+      .. bro:enum:: OpenFlow::OFPAT_VENDOR OpenFlow::ofp_action_type
+
+         Vendor specific.
+
+   Openflow action_type definitions.
+   
+   The openflow action type defines
+   what actions openflow can take
+   to modify a packet
+
+.. bro:type:: OpenFlow::ofp_config_flags
+
+   :Type: :bro:type:`enum`
+
+      .. bro:enum:: OpenFlow::OFPC_FRAG_NORMAL OpenFlow::ofp_config_flags
+
+         No special handling for fragments.
+
+      .. bro:enum:: OpenFlow::OFPC_FRAG_DROP OpenFlow::ofp_config_flags
+
+         Drop fragments.
+
+      .. bro:enum:: OpenFlow::OFPC_FRAG_REASM OpenFlow::ofp_config_flags
+
+         Reassemble (only if OFPC_IP_REASM set).
+
+      .. bro:enum:: OpenFlow::OFPC_FRAG_MASK OpenFlow::ofp_config_flags
+
+   Openflow config flag definitions.
+   
+   TODO: describe
+
+.. bro:type:: OpenFlow::ofp_flow_mod_command
+
+   :Type: :bro:type:`enum`
+
+      .. bro:enum:: OpenFlow::OFPFC_ADD OpenFlow::ofp_flow_mod_command
+
+         New flow.
+
+      .. bro:enum:: OpenFlow::OFPFC_MODIFY OpenFlow::ofp_flow_mod_command
+
+         Modify all matching flows.
+
+      .. bro:enum:: OpenFlow::OFPFC_MODIFY_STRICT OpenFlow::ofp_flow_mod_command
+
+         Modify entry strictly matching wildcards.
+
+      .. bro:enum:: OpenFlow::OFPFC_DELETE OpenFlow::ofp_flow_mod_command
+
+         Delete all matching flows.
+
+      .. bro:enum:: OpenFlow::OFPFC_DELETE_STRICT OpenFlow::ofp_flow_mod_command
+
+         Strictly matching wildcards and priority.
+
+   Openflow flow_mod_command definitions.
+   
+   The openflow flow_mod_command describes
+   of what kind an action is.
+
+

doc/scripts/base/frameworks/openflow/index.rst

@@ -0,0 +1,50 @@
+:orphan:
+
+Package: base/frameworks/openflow
+=================================
+
+The OpenFlow framework exposes the data structures and functions
+necessary to interface to OpenFlow capable hardware.
+
+:doc:`/scripts/base/frameworks/openflow/__load__.bro`
+
+
+:doc:`/scripts/base/frameworks/openflow/consts.bro`
+
+   Constants used by the OpenFlow framework.
+
+:doc:`/scripts/base/frameworks/openflow/types.bro`
+
+   Types used by the OpenFlow framework.
+
+:doc:`/scripts/base/frameworks/openflow/main.bro`
+
+   Bro's OpenFlow control framework.
+   
+   This plugin-based framework allows to control OpenFlow capable
+   switches by implementing communication to an OpenFlow controller
+   via plugins. The framework has to be instantiated via the new function
+   in one of the plugins. This framework only offers very low-level
+   functionality; if you want to use OpenFlow capable switches, e.g.,
+   for shunting, please look at the NetControl framework, which provides higher
+   level functions and can use the OpenFlow framework as a backend.
+
+:doc:`/scripts/base/frameworks/openflow/plugins/__load__.bro`
+
+
+:doc:`/scripts/base/frameworks/openflow/plugins/ryu.bro`
+
+   OpenFlow plugin for the Ryu controller.
+
+:doc:`/scripts/base/frameworks/openflow/plugins/log.bro`
+
+   OpenFlow plugin that outputs flow-modification commands
+   to a Bro log file.
+
+:doc:`/scripts/base/frameworks/openflow/plugins/broker.bro`
+
+   OpenFlow plugin for interfacing to controllers via Broker.
+
+:doc:`/scripts/base/frameworks/openflow/non-cluster.bro`
+
+

doc/scripts/base/frameworks/openflow/main.bro.rst

@@ -0,0 +1,265 @@
+:tocdepth: 3
+
+base/frameworks/openflow/main.bro
+=================================
+.. bro:namespace:: OpenFlow
+
+Bro's OpenFlow control framework.
+
+This plugin-based framework allows to control OpenFlow capable
+switches by implementing communication to an OpenFlow controller
+via plugins. The framework has to be instantiated via the new function
+in one of the plugins. This framework only offers very low-level
+functionality; if you want to use OpenFlow capable switches, e.g.,
+for shunting, please look at the NetControl framework, which provides higher
+level functions and can use the OpenFlow framework as a backend.
+
+:Namespace: OpenFlow
+:Imports: :doc:`base/frameworks/openflow/consts.bro </scripts/base/frameworks/openflow/consts.bro>`, :doc:`base/frameworks/openflow/types.bro </scripts/base/frameworks/openflow/types.bro>`
+
+Summary
+~~~~~~~
+Events
+######
+=========================================================== =============================================================================================
+:bro:id:`OpenFlow::controller_activated`: :bro:type:`event` Event that is raised once a controller finishes initialization
+                                                            and is completely activated.
+:bro:id:`OpenFlow::flow_mod_failure`: :bro:type:`event`     Reports an error while installing a flow Rule.
+:bro:id:`OpenFlow::flow_mod_success`: :bro:type:`event`     Event confirming successful modification of a flow rule.
+:bro:id:`OpenFlow::flow_removed`: :bro:type:`event`         Reports that a flow was removed by the switch because of either the hard or the idle timeout.
+=========================================================== =============================================================================================
+
+Functions
+#########
+=============================================================== =====================================================================
+:bro:id:`OpenFlow::controller_init_done`: :bro:type:`function`  Function to signal that a controller finished activation and is
+                                                                ready to use.
+:bro:id:`OpenFlow::flow_clear`: :bro:type:`function`            Clear the current flow table of the controller.
+:bro:id:`OpenFlow::flow_mod`: :bro:type:`function`              Global flow_mod function.
+:bro:id:`OpenFlow::generate_cookie`: :bro:type:`function`       Function to generate a new cookie using our group id.
+:bro:id:`OpenFlow::get_cookie_gid`: :bro:type:`function`        Function to get the group id out of a given cookie.
+:bro:id:`OpenFlow::get_cookie_uid`: :bro:type:`function`        Function to get the unique id out of a given cookie.
+:bro:id:`OpenFlow::lookup_controller`: :bro:type:`function`     Function to lookup a controller instance by name.
+:bro:id:`OpenFlow::match_conn`: :bro:type:`function`            Convert a conn_id record into an ofp_match record that can be used to
+                                                                create match objects for OpenFlow.
+:bro:id:`OpenFlow::register_controller`: :bro:type:`function`   Function to register a controller instance.
+:bro:id:`OpenFlow::unregister_controller`: :bro:type:`function` Function to unregister a controller instance.
+=============================================================== =====================================================================
+
+
+Detailed Interface
+~~~~~~~~~~~~~~~~~~
+Events
+######
+.. bro:id:: OpenFlow::controller_activated
+
+   :Type: :bro:type:`event` (name: :bro:type:`string`, controller: :bro:type:`OpenFlow::Controller`)
+
+   Event that is raised once a controller finishes initialization
+   and is completely activated.
+
+   :name: Unique name of this controller instance.
+   
+
+   :controller: The controller that finished activation.
+
+.. bro:id:: OpenFlow::flow_mod_failure
+
+   :Type: :bro:type:`event` (name: :bro:type:`string`, match: :bro:type:`OpenFlow::ofp_match`, flow_mod: :bro:type:`OpenFlow::ofp_flow_mod`, msg: :bro:type:`string` :bro:attr:`&default` = ``""`` :bro:attr:`&optional`)
+
+   Reports an error while installing a flow Rule.
+   
+
+   :name: The unique name of the OpenFlow controller from which this event originated.
+   
+
+   :match: The ofp_match record which describes the flow to match.
+   
+
+   :flow_mod: The openflow flow_mod record which describes the action to take.
+   
+
+   :msg: Message to describe the event.
+
+.. bro:id:: OpenFlow::flow_mod_success
+
+   :Type: :bro:type:`event` (name: :bro:type:`string`, match: :bro:type:`OpenFlow::ofp_match`, flow_mod: :bro:type:`OpenFlow::ofp_flow_mod`, msg: :bro:type:`string` :bro:attr:`&default` = ``""`` :bro:attr:`&optional`)
+
+   Event confirming successful modification of a flow rule.
+   
+
+   :name: The unique name of the OpenFlow controller from which this event originated.
+   
+
+   :match: The ofp_match record which describes the flow to match.
+   
+
+   :flow_mod: The openflow flow_mod record which describes the action to take.
+   
+
+   :msg: An optional informational message by the plugin.
+
+.. bro:id:: OpenFlow::flow_removed
+
+   :Type: :bro:type:`event` (name: :bro:type:`string`, match: :bro:type:`OpenFlow::ofp_match`, cookie: :bro:type:`count`, priority: :bro:type:`count`, reason: :bro:type:`count`, duration_sec: :bro:type:`count`, idle_timeout: :bro:type:`count`, packet_count: :bro:type:`count`, byte_count: :bro:type:`count`)
+
+   Reports that a flow was removed by the switch because of either the hard or the idle timeout.
+   This message is only generated by controllers that indicate that they support flow removal
+   in supports_flow_removed.
+   
+
+   :name: The unique name of the OpenFlow controller from which this event originated.
+   
+
+   :match: The ofp_match record which was used to create the flow.
+   
+
+   :cookie: The cookie that was specified when creating the flow.
+   
+
+   :priority: The priority that was specified when creating the flow.
+   
+
+   :reason: The reason for flow removal (OFPRR_*).
+   
+
+   :duration_sec: Duration of the flow in seconds.
+   
+
+   :packet_count: Packet count of the flow.
+   
+
+   :byte_count: Byte count of the flow.
+
+Functions
+#########
+.. bro:id:: OpenFlow::controller_init_done
+
+   :Type: :bro:type:`function` (controller: :bro:type:`OpenFlow::Controller`) : :bro:type:`void`
+
+   Function to signal that a controller finished activation and is
+   ready to use. Will throw the ``OpenFlow::controller_activated``
+   event.
+
+.. bro:id:: OpenFlow::flow_clear
+
+   :Type: :bro:type:`function` (controller: :bro:type:`OpenFlow::Controller`) : :bro:type:`bool`
+
+   Clear the current flow table of the controller.
+   
+
+   :controller: The controller which should execute the flow modification.
+   
+
+   :returns: F on error or if the plugin does not support the operation, T when the operation was queued.
+
+.. bro:id:: OpenFlow::flow_mod
+
+   :Type: :bro:type:`function` (controller: :bro:type:`OpenFlow::Controller`, match: :bro:type:`OpenFlow::ofp_match`, flow_mod: :bro:type:`OpenFlow::ofp_flow_mod`) : :bro:type:`bool`
+
+   Global flow_mod function.
+   
+
+   :controller: The controller which should execute the flow modification.
+   
+
+   :match: The ofp_match record which describes the flow to match.
+   
+
+   :flow_mod: The openflow flow_mod record which describes the action to take.
+   
+
+   :returns: F on error or if the plugin does not support the operation, T when the operation was queued.
+
+.. bro:id:: OpenFlow::generate_cookie
+
+   :Type: :bro:type:`function` (cookie: :bro:type:`count` :bro:attr:`&default` = ``0`` :bro:attr:`&optional`) : :bro:type:`count`
+
+   Function to generate a new cookie using our group id.
+   
+
+   :cookie: The openflow match cookie.
+   
+
+   :returns: The cookie group id.
+
+.. bro:id:: OpenFlow::get_cookie_gid
+
+   :Type: :bro:type:`function` (cookie: :bro:type:`count`) : :bro:type:`count`
+
+   Function to get the group id out of a given cookie.
+   
+
+   :cookie: The openflow match cookie.
+   
+
+   :returns: The cookie group id.
+
+.. bro:id:: OpenFlow::get_cookie_uid
+
+   :Type: :bro:type:`function` (cookie: :bro:type:`count`) : :bro:type:`count`
+
+   Function to get the unique id out of a given cookie.
+   
+
+   :cookie: The openflow match cookie.
+   
+
+   :returns: The cookie unique id.
+
+.. bro:id:: OpenFlow::lookup_controller
+
+   :Type: :bro:type:`function` (name: :bro:type:`string`) : :bro:type:`vector` of :bro:type:`OpenFlow::Controller`
+
+   Function to lookup a controller instance by name.
+   
+
+   :name: Unique name of the controller to look up.
+   
+
+   :returns: One element vector with controller, if found. Empty vector otherwise.
+
+.. bro:id:: OpenFlow::match_conn
+
+   :Type: :bro:type:`function` (id: :bro:type:`conn_id`, reverse: :bro:type:`bool` :bro:attr:`&default` = ``F`` :bro:attr:`&optional`) : :bro:type:`OpenFlow::ofp_match`
+
+   Convert a conn_id record into an ofp_match record that can be used to
+   create match objects for OpenFlow.
+   
+
+   :id: The conn_id record that describes the record.
+   
+
+   :reverse: Reverse the sources and destinations when creating the match record (default F).
+   
+
+   :returns: ofp_match object for the conn_id record.
+
+.. bro:id:: OpenFlow::register_controller
+
+   :Type: :bro:type:`function` (tpe: :bro:type:`OpenFlow::Plugin`, name: :bro:type:`string`, controller: :bro:type:`OpenFlow::Controller`) : :bro:type:`void`
+
+   Function to register a controller instance. This function
+   is called automatically by the plugin _new functions.
+   
+
+   :tpe: Type of this plugin.
+   
+
+   :name: Unique name of this controller instance.
+   
+
+   :controller: The controller to register.
+
+.. bro:id:: OpenFlow::unregister_controller
+
+   :Type: :bro:type:`function` (controller: :bro:type:`OpenFlow::Controller`) : :bro:type:`void`
+
+   Function to unregister a controller instance. This function
+   should be called when a specific controller should no longer
+   be used.
+   
+
+   :controller: The controller to unregister.
+
+

doc/scripts/base/frameworks/openflow/non-cluster.bro.rst

@@ -0,0 +1,16 @@
+:tocdepth: 3
+
+base/frameworks/openflow/non-cluster.bro
+========================================
+.. bro:namespace:: OpenFlow
+
+
+:Namespace: OpenFlow
+:Imports: :doc:`base/frameworks/openflow/main.bro </scripts/base/frameworks/openflow/main.bro>`
+
+Summary
+~~~~~~~
+
+Detailed Interface
+~~~~~~~~~~~~~~~~~~
+

doc/scripts/base/frameworks/openflow/plugins/__load__.bro.rst

@@ -0,0 +1,14 @@
+:tocdepth: 3
+
+base/frameworks/openflow/plugins/__load__.bro
+=============================================
+
+
+:Imports: :doc:`base/frameworks/openflow/plugins/broker.bro </scripts/base/frameworks/openflow/plugins/broker.bro>`, :doc:`base/frameworks/openflow/plugins/log.bro </scripts/base/frameworks/openflow/plugins/log.bro>`, :doc:`base/frameworks/openflow/plugins/ryu.bro </scripts/base/frameworks/openflow/plugins/ryu.bro>`
+
+Summary
+~~~~~~~
+
+Detailed Interface
+~~~~~~~~~~~~~~~~~~
+

doc/scripts/base/frameworks/openflow/plugins/broker.bro.rst

@@ -0,0 +1,72 @@
+:tocdepth: 3
+
+base/frameworks/openflow/plugins/broker.bro
+===========================================
+.. bro:namespace:: OpenFlow
+
+OpenFlow plugin for interfacing to controllers via Broker.
+
+:Namespace: OpenFlow
+:Imports: :doc:`base/frameworks/broker </scripts/base/frameworks/broker/index>`, :doc:`base/frameworks/openflow </scripts/base/frameworks/openflow/index>`
+
+Summary
+~~~~~~~
+Redefinitions
+#############
+============================================================================ =
+:bro:type:`OpenFlow::ControllerState`: :bro:type:`record` :bro:attr:`&redef` 
+:bro:type:`OpenFlow::Plugin`: :bro:type:`enum`                               
+============================================================================ =
+
+Events
+######
+======================================================== =
+:bro:id:`OpenFlow::broker_flow_clear`: :bro:type:`event` 
+:bro:id:`OpenFlow::broker_flow_mod`: :bro:type:`event`   
+======================================================== =
+
+Functions
+#########
+==================================================== ==============================
+:bro:id:`OpenFlow::broker_new`: :bro:type:`function` Broker controller constructor.
+==================================================== ==============================
+
+
+Detailed Interface
+~~~~~~~~~~~~~~~~~~
+Events
+######
+.. bro:id:: OpenFlow::broker_flow_clear
+
+   :Type: :bro:type:`event` (name: :bro:type:`string`, dpid: :bro:type:`count`)
+
+
+.. bro:id:: OpenFlow::broker_flow_mod
+
+   :Type: :bro:type:`event` (name: :bro:type:`string`, dpid: :bro:type:`count`, match: :bro:type:`OpenFlow::ofp_match`, flow_mod: :bro:type:`OpenFlow::ofp_flow_mod`)
+
+
+Functions
+#########
+.. bro:id:: OpenFlow::broker_new
+
+   :Type: :bro:type:`function` (name: :bro:type:`string`, host: :bro:type:`addr`, host_port: :bro:type:`port`, topic: :bro:type:`string`, dpid: :bro:type:`count`) : :bro:type:`OpenFlow::Controller`
+
+   Broker controller constructor.
+   
+
+   :host: Controller ip.
+   
+
+   :host_port: Controller listen port.
+   
+
+   :topic: Broker topic to send messages to.
+   
+
+   :dpid: OpenFlow switch datapath id.
+   
+
+   :returns: OpenFlow::Controller record.
+
+

doc/scripts/base/frameworks/openflow/plugins/index.rst

@@ -0,0 +1,23 @@
+:orphan:
+
+Package: base/frameworks/openflow/plugins
+=========================================
+
+Plugins for the OpenFlow framework.
+
+:doc:`/scripts/base/frameworks/openflow/plugins/__load__.bro`
+
+
+:doc:`/scripts/base/frameworks/openflow/plugins/ryu.bro`
+
+   OpenFlow plugin for the Ryu controller.
+
+:doc:`/scripts/base/frameworks/openflow/plugins/log.bro`
+
+   OpenFlow plugin that outputs flow-modification commands
+   to a Bro log file.
+
+:doc:`/scripts/base/frameworks/openflow/plugins/broker.bro`
+
+   OpenFlow plugin for interfacing to controllers via Broker.
+

doc/scripts/base/frameworks/openflow/plugins/log.bro.rst

@@ -0,0 +1,91 @@
+:tocdepth: 3
+
+base/frameworks/openflow/plugins/log.bro
+========================================
+.. bro:namespace:: OpenFlow
+
+OpenFlow plugin that outputs flow-modification commands
+to a Bro log file.
+
+:Namespace: OpenFlow
+:Imports: :doc:`base/frameworks/logging </scripts/base/frameworks/logging/index>`, :doc:`base/frameworks/openflow </scripts/base/frameworks/openflow/index>`
+
+Summary
+~~~~~~~
+Types
+#####
+============================================== =================================================================
+:bro:type:`OpenFlow::Info`: :bro:type:`record` The record type which contains column fields of the OpenFlow log.
+============================================== =================================================================
+
+Redefinitions
+#############
+============================================================================ =
+:bro:type:`Log::ID`: :bro:type:`enum`                                        
+:bro:type:`OpenFlow::ControllerState`: :bro:type:`record` :bro:attr:`&redef` 
+:bro:type:`OpenFlow::Plugin`: :bro:type:`enum`                               
+============================================================================ =
+
+Events
+######
+=================================================== ==================================================================
+:bro:id:`OpenFlow::log_openflow`: :bro:type:`event` Event that can be handled to access the :bro:type:`OpenFlow::Info`
+                                                    record as it is sent on to the logging framework.
+=================================================== ==================================================================
+
+Functions
+#########
+================================================= ===========================
+:bro:id:`OpenFlow::log_new`: :bro:type:`function` Log controller constructor.
+================================================= ===========================
+
+
+Detailed Interface
+~~~~~~~~~~~~~~~~~~
+Types
+#####
+.. bro:type:: OpenFlow::Info
+
+   :Type: :bro:type:`record`
+
+      ts: :bro:type:`time` :bro:attr:`&log`
+         Network time.
+
+      dpid: :bro:type:`count` :bro:attr:`&log`
+         OpenFlow switch datapath id.
+
+      match: :bro:type:`OpenFlow::ofp_match` :bro:attr:`&log`
+         OpenFlow match fields.
+
+      flow_mod: :bro:type:`OpenFlow::ofp_flow_mod` :bro:attr:`&log`
+         OpenFlow modify flow entry message.
+
+   The record type which contains column fields of the OpenFlow log.
+
+Events
+######
+.. bro:id:: OpenFlow::log_openflow
+
+   :Type: :bro:type:`event` (rec: :bro:type:`OpenFlow::Info`)
+
+   Event that can be handled to access the :bro:type:`OpenFlow::Info`
+   record as it is sent on to the logging framework.
+
+Functions
+#########
+.. bro:id:: OpenFlow::log_new
+
+   :Type: :bro:type:`function` (dpid: :bro:type:`count`, success_event: :bro:type:`bool` :bro:attr:`&default` = ``T`` :bro:attr:`&optional`) : :bro:type:`OpenFlow::Controller`
+
+   Log controller constructor.
+   
+
+   :dpid: OpenFlow switch datapath id.
+   
+
+   :success_event: If true, flow_mod_success is raised for each logged line.
+   
+
+   :returns: OpenFlow::Controller record.
+
+

doc/scripts/base/frameworks/openflow/plugins/ryu.bro.rst

@@ -0,0 +1,50 @@
+:tocdepth: 3
+
+base/frameworks/openflow/plugins/ryu.bro
+========================================
+.. bro:namespace:: OpenFlow
+
+OpenFlow plugin for the Ryu controller.
+
+:Namespace: OpenFlow
+:Imports: :doc:`base/frameworks/openflow </scripts/base/frameworks/openflow/index>`, :doc:`base/utils/active-http.bro </scripts/base/utils/active-http.bro>`, :doc:`base/utils/exec.bro </scripts/base/utils/exec.bro>`, :doc:`base/utils/json.bro </scripts/base/utils/json.bro>`
+
+Summary
+~~~~~~~
+Redefinitions
+#############
+============================================================================ =
+:bro:type:`OpenFlow::ControllerState`: :bro:type:`record` :bro:attr:`&redef` 
+:bro:type:`OpenFlow::Plugin`: :bro:type:`enum`                               
+============================================================================ =
+
+Functions
+#########
+================================================= ===========================
+:bro:id:`OpenFlow::ryu_new`: :bro:type:`function` Ryu controller constructor.
+================================================= ===========================
+
+
+Detailed Interface
+~~~~~~~~~~~~~~~~~~
+Functions
+#########
+.. bro:id:: OpenFlow::ryu_new
+
+   :Type: :bro:type:`function` (host: :bro:type:`addr`, host_port: :bro:type:`count`, dpid: :bro:type:`count`) : :bro:type:`OpenFlow::Controller`
+
+   Ryu controller constructor.
+   
+
+   :host: Controller ip.
+   
+
+   :host_port: Controller listen port.
+   
+
+   :dpid: OpenFlow switch datapath id.
+   
+
+   :returns: OpenFlow::Controller record.
+
+

doc/scripts/base/frameworks/openflow/types.bro.rst

@@ -0,0 +1,264 @@
+:tocdepth: 3
+
+base/frameworks/openflow/types.bro
+==================================
+.. bro:namespace:: OpenFlow
+
+Types used by the OpenFlow framework.
+
+:Namespace: OpenFlow
+:Imports: :doc:`base/frameworks/openflow/consts.bro </scripts/base/frameworks/openflow/consts.bro>`
+
+Summary
+~~~~~~~
+Types
+#####
+============================================================================ ===============================================================
+:bro:type:`OpenFlow::Controller`: :bro:type:`record`                         Controller record representing an openflow controller.
+:bro:type:`OpenFlow::ControllerState`: :bro:type:`record` :bro:attr:`&redef` Controller related state.
+:bro:type:`OpenFlow::Plugin`: :bro:type:`enum`                               Available openflow plugins.
+:bro:type:`OpenFlow::ofp_flow_action`: :bro:type:`record` :bro:attr:`&log`   The actions that can be taken in a flow.
+:bro:type:`OpenFlow::ofp_flow_mod`: :bro:type:`record` :bro:attr:`&log`      Openflow flow_mod definition, describing the action to perform.
+:bro:type:`OpenFlow::ofp_match`: :bro:type:`record` :bro:attr:`&log`         Openflow match definition.
+============================================================================ ===============================================================
+
+
+Detailed Interface
+~~~~~~~~~~~~~~~~~~
+Types
+#####
+.. bro:type:: OpenFlow::Controller
+
+   :Type: :bro:type:`record`
+
+      state: :bro:type:`OpenFlow::ControllerState`
+         Controller related state.
+
+      supports_flow_removed: :bro:type:`bool`
+         Does the controller support the flow_removed event?
+
+      describe: :bro:type:`function` (state: :bro:type:`OpenFlow::ControllerState`) : :bro:type:`string`
+         Function that describes the controller. Has to be implemented.
+
+      init: :bro:type:`function` (state: :bro:type:`OpenFlow::ControllerState`) : :bro:type:`void` :bro:attr:`&optional`
+         One-time initialization function. If defined, controller_init_done has to be called once initialization finishes.
+
+      destroy: :bro:type:`function` (state: :bro:type:`OpenFlow::ControllerState`) : :bro:type:`void` :bro:attr:`&optional`
+         One-time destruction function.
+
+      flow_mod: :bro:type:`function` (state: :bro:type:`OpenFlow::ControllerState`, match: :bro:type:`OpenFlow::ofp_match`, flow_mod: :bro:type:`OpenFlow::ofp_flow_mod`) : :bro:type:`bool` :bro:attr:`&optional`
+         flow_mod function.
+
+      flow_clear: :bro:type:`function` (state: :bro:type:`OpenFlow::ControllerState`) : :bro:type:`bool` :bro:attr:`&optional`
+         flow_clear function.
+
+   Controller record representing an openflow controller.
+
+.. bro:type:: OpenFlow::ControllerState
+
+   :Type: :bro:type:`record`
+
+      _plugin: :bro:type:`OpenFlow::Plugin` :bro:attr:`&optional`
+         Internally set to the type of plugin used.
+
+      _name: :bro:type:`string` :bro:attr:`&optional`
+         Internally set to the unique name of the controller.
+
+      _activated: :bro:type:`bool` :bro:attr:`&default` = ``F`` :bro:attr:`&optional`
+         Internally set to true once the controller is activated.
+
+      ryu_host: :bro:type:`addr` :bro:attr:`&optional`
+         (present if :doc:`/scripts/base/frameworks/openflow/plugins/ryu.bro` is loaded)
+
+         Controller ip.
+
+      ryu_port: :bro:type:`count` :bro:attr:`&optional`
+         (present if :doc:`/scripts/base/frameworks/openflow/plugins/ryu.bro` is loaded)
+
+         Controller listen port.
+
+      ryu_dpid: :bro:type:`count` :bro:attr:`&optional`
+         (present if :doc:`/scripts/base/frameworks/openflow/plugins/ryu.bro` is loaded)
+
+         OpenFlow switch datapath id.
+
+      ryu_debug: :bro:type:`bool` :bro:attr:`&default` = ``F`` :bro:attr:`&optional`
+         (present if :doc:`/scripts/base/frameworks/openflow/plugins/ryu.bro` is loaded)
+
+         Enable debug mode - output JSON to stdout; do not perform actions.
+
+      log_dpid: :bro:type:`count` :bro:attr:`&optional`
+         (present if :doc:`/scripts/base/frameworks/openflow/plugins/log.bro` is loaded)
+
+         OpenFlow switch datapath id.
+
+      log_success_event: :bro:type:`bool` :bro:attr:`&optional`
+         (present if :doc:`/scripts/base/frameworks/openflow/plugins/log.bro` is loaded)
+
+         Raise or do not raise success event.
+
+      broker_host: :bro:type:`addr` :bro:attr:`&optional`
+         (present if :doc:`/scripts/base/frameworks/openflow/plugins/broker.bro` is loaded)
+
+         Controller ip.
+
+      broker_port: :bro:type:`port` :bro:attr:`&optional`
+         (present if :doc:`/scripts/base/frameworks/openflow/plugins/broker.bro` is loaded)
+
+         Controller listen port.
+
+      broker_dpid: :bro:type:`count` :bro:attr:`&optional`
+         (present if :doc:`/scripts/base/frameworks/openflow/plugins/broker.bro` is loaded)
+
+         OpenFlow switch datapath id.
+
+      broker_topic: :bro:type:`string` :bro:attr:`&optional`
+         (present if :doc:`/scripts/base/frameworks/openflow/plugins/broker.bro` is loaded)
+
+         Topic to send events for this controller to.
+   :Attributes: :bro:attr:`&redef`
+
+   Controller related state.
+   Can be redefined by plugins to
+   add state.
+
+.. bro:type:: OpenFlow::Plugin
+
+   :Type: :bro:type:`enum`
+
+      .. bro:enum:: OpenFlow::INVALID OpenFlow::Plugin
+
+         Internal placeholder plugin.
+
+      .. bro:enum:: OpenFlow::RYU OpenFlow::Plugin
+
+         (present if :doc:`/scripts/base/frameworks/openflow/plugins/ryu.bro` is loaded)
+
+
+      .. bro:enum:: OpenFlow::OFLOG OpenFlow::Plugin
+
+         (present if :doc:`/scripts/base/frameworks/openflow/plugins/log.bro` is loaded)
+
+
+      .. bro:enum:: OpenFlow::BROKER OpenFlow::Plugin
+
+         (present if :doc:`/scripts/base/frameworks/openflow/plugins/broker.bro` is loaded)
+
+
+   Available openflow plugins.
+
+.. bro:type:: OpenFlow::ofp_flow_action
+
+   :Type: :bro:type:`record`
+
+      out_ports: :bro:type:`vector` of :bro:type:`count` :bro:attr:`&default` = ``[]`` :bro:attr:`&optional` :bro:attr:`&log`
+         Output ports to send data to.
+
+      vlan_vid: :bro:type:`count` :bro:attr:`&optional` :bro:attr:`&log`
+         Set vlan vid to this value.
+
+      vlan_pcp: :bro:type:`count` :bro:attr:`&optional` :bro:attr:`&log`
+         Set vlan priority to this value.
+
+      vlan_strip: :bro:type:`bool` :bro:attr:`&default` = ``F`` :bro:attr:`&optional` :bro:attr:`&log`
+         Strip vlan tag.
+
+      dl_src: :bro:type:`string` :bro:attr:`&optional` :bro:attr:`&log`
+         Set ethernet source address.
+
+      dl_dst: :bro:type:`string` :bro:attr:`&optional` :bro:attr:`&log`
+         Set ethernet destination address.
+
+      nw_tos: :bro:type:`count` :bro:attr:`&optional` :bro:attr:`&log`
+         Set ip tos to this value.
+
+      nw_src: :bro:type:`addr` :bro:attr:`&optional` :bro:attr:`&log`
+         Set source to this ip.
+
+      nw_dst: :bro:type:`addr` :bro:attr:`&optional` :bro:attr:`&log`
+         Set destination to this ip.
+
+      tp_src: :bro:type:`count` :bro:attr:`&optional` :bro:attr:`&log`
+         Set tcp/udp source port.
+
+      tp_dst: :bro:type:`count` :bro:attr:`&optional` :bro:attr:`&log`
+         Set tcp/udp destination port.
+   :Attributes: :bro:attr:`&log`
+
+   The actions that can be taken in a flow.
+   (Separate record to make ofp_flow_mod less crowded)
+
+.. bro:type:: OpenFlow::ofp_flow_mod
+
+   :Type: :bro:type:`record`
+
+      cookie: :bro:type:`count` :bro:attr:`&log`
+         Opaque controller-issued identifier.
+
+      table_id: :bro:type:`count` :bro:attr:`&optional` :bro:attr:`&log`
+         Table to put the flow in. OFPTT_ALL can be used for delete,
+         to delete flows from all matching tables.
+
+      command: :bro:type:`OpenFlow::ofp_flow_mod_command` :bro:attr:`&log`
+         One of OFPFC_*.
+
+      idle_timeout: :bro:type:`count` :bro:attr:`&default` = ``0`` :bro:attr:`&optional` :bro:attr:`&log`
+         Idle time before discarding (seconds).
+
+      hard_timeout: :bro:type:`count` :bro:attr:`&default` = ``0`` :bro:attr:`&optional` :bro:attr:`&log`
+         Max time before discarding (seconds).
+
+      priority: :bro:type:`count` :bro:attr:`&default` = ``0`` :bro:attr:`&optional` :bro:attr:`&log`
+         Priority level of flow entry.
+
+      out_port: :bro:type:`count` :bro:attr:`&optional` :bro:attr:`&log`
+         For OFPFC_DELETE* commands, require matching entried to include
+         this as an output port/group. OFPP_ANY/OFPG_ANY means no restrictions.
+
+      out_group: :bro:type:`count` :bro:attr:`&optional` :bro:attr:`&log`
+
+      flags: :bro:type:`count` :bro:attr:`&default` = ``0`` :bro:attr:`&optional` :bro:attr:`&log`
+         Bitmap of the OFPFF_* flags
+
+      actions: :bro:type:`OpenFlow::ofp_flow_action` :bro:attr:`&default` = ``[out_ports=[], vlan_vid=<uninitialized>, vlan_pcp=<uninitialized>, vlan_strip=F, dl_src=<uninitialized>, dl_dst=<uninitialized>, nw_tos=<uninitialized>, nw_src=<uninitialized>, nw_dst=<uninitialized>, tp_src=<uninitialized>, tp_dst=<uninitialized>]`` :bro:attr:`&optional` :bro:attr:`&log`
+         Actions to take on match
+   :Attributes: :bro:attr:`&log`
+
+   Openflow flow_mod definition, describing the action to perform.
+
+.. bro:type:: OpenFlow::ofp_match
+
+   :Type: :bro:type:`record`
+
+      in_port: :bro:type:`count` :bro:attr:`&optional` :bro:attr:`&log`
+
+      dl_src: :bro:type:`string` :bro:attr:`&optional` :bro:attr:`&log`
+
+      dl_dst: :bro:type:`string` :bro:attr:`&optional` :bro:attr:`&log`
+
+      dl_vlan: :bro:type:`count` :bro:attr:`&optional` :bro:attr:`&log`
+
+      dl_vlan_pcp: :bro:type:`count` :bro:attr:`&optional` :bro:attr:`&log`
+
+      dl_type: :bro:type:`count` :bro:attr:`&optional` :bro:attr:`&log`
+
+      nw_tos: :bro:type:`count` :bro:attr:`&optional` :bro:attr:`&log`
+
+      nw_proto: :bro:type:`count` :bro:attr:`&optional` :bro:attr:`&log`
+
+      nw_src: :bro:type:`subnet` :bro:attr:`&optional` :bro:attr:`&log`
+
+      nw_dst: :bro:type:`subnet` :bro:attr:`&optional` :bro:attr:`&log`
+
+      tp_src: :bro:type:`count` :bro:attr:`&optional` :bro:attr:`&log`
+
+      tp_dst: :bro:type:`count` :bro:attr:`&optional` :bro:attr:`&log`
+   :Attributes: :bro:attr:`&log`
+
+   Openflow match definition.
+   
+   The openflow match record describes
+   which packets match to a specific
+   rule in a flow table.
+
+

doc/scripts/base/frameworks/packet-filter/__load__.bro.rst

@@ -0,0 +1,14 @@
+:tocdepth: 3
+
+base/frameworks/packet-filter/__load__.bro
+==========================================
+
+
+:Imports: :doc:`base/frameworks/cluster </scripts/base/frameworks/cluster/index>`, :doc:`base/frameworks/packet-filter/main.bro </scripts/base/frameworks/packet-filter/main.bro>`, :doc:`base/frameworks/packet-filter/netstats.bro </scripts/base/frameworks/packet-filter/netstats.bro>`, :doc:`base/frameworks/packet-filter/utils.bro </scripts/base/frameworks/packet-filter/utils.bro>`
+
+Summary
+~~~~~~~
+
+Detailed Interface
+~~~~~~~~~~~~~~~~~~
+

doc/scripts/base/frameworks/packet-filter/index.rst

@@ -0,0 +1,27 @@
+:orphan:
+
+Package: base/frameworks/packet-filter
+======================================
+
+The packet filter framework supports how Bro sets its BPF capture filter.
+
+:doc:`/scripts/base/frameworks/packet-filter/utils.bro`
+
+
+:doc:`/scripts/base/frameworks/packet-filter/__load__.bro`
+
+
+:doc:`/scripts/base/frameworks/packet-filter/main.bro`
+
+   This script supports how Bro sets its BPF capture filter.  By default
+   Bro sets a capture filter that allows all traffic.  If a filter
+   is set on the command line, that filter takes precedence over the default
+   open filter and all filters defined in Bro scripts with the
+   :bro:id:`capture_filters` and :bro:id:`restrict_filters` variables.
+
+:doc:`/scripts/base/frameworks/packet-filter/netstats.bro`
+
+   This script reports on packet loss from the various packet sources.
+   When Bro is reading input from trace files, this script will not
+   report any packet loss statistics.
+

doc/scripts/base/frameworks/packet-filter/main.bro.rst

@@ -0,0 +1,223 @@
+:tocdepth: 3
+
+base/frameworks/packet-filter/main.bro
+======================================
+.. bro:namespace:: PacketFilter
+
+This script supports how Bro sets its BPF capture filter.  By default
+Bro sets a capture filter that allows all traffic.  If a filter
+is set on the command line, that filter takes precedence over the default
+open filter and all filters defined in Bro scripts with the
+:bro:id:`capture_filters` and :bro:id:`restrict_filters` variables.
+
+:Namespace: PacketFilter
+:Imports: :doc:`base/frameworks/analyzer </scripts/base/frameworks/analyzer/index>`, :doc:`base/frameworks/notice </scripts/base/frameworks/notice/index>`, :doc:`base/frameworks/packet-filter/utils.bro </scripts/base/frameworks/packet-filter/utils.bro>`
+
+Summary
+~~~~~~~
+Redefinable Options
+###################
+================================================================================================= ===============================================================================
+:bro:id:`PacketFilter::default_capture_filter`: :bro:type:`string` :bro:attr:`&redef`             The BPF filter that is used by default to define what traffic should
+                                                                                                  be captured.
+:bro:id:`PacketFilter::enable_auto_protocol_capture_filters`: :bro:type:`bool` :bro:attr:`&redef` Enables the old filtering approach of "only watch common ports for
+                                                                                                  analyzed protocols".
+:bro:id:`PacketFilter::max_filter_compile_time`: :bro:type:`interval` :bro:attr:`&redef`          The maximum amount of time that you'd like to allow for BPF filters to compile.
+:bro:id:`PacketFilter::restricted_filter`: :bro:type:`string` :bro:attr:`&redef`                  Filter string which is unconditionally and'ed to the beginning of
+                                                                                                  every dynamically built filter.
+:bro:id:`PacketFilter::unrestricted_filter`: :bro:type:`string` :bro:attr:`&redef`                Filter string which is unconditionally or'ed to the beginning of
+                                                                                                  every dynamically built filter.
+================================================================================================= ===============================================================================
+
+State Variables
+###############
+========================================================== ===================================================================
+:bro:id:`PacketFilter::current_filter`: :bro:type:`string` This is where the default packet filter is stored and it should not
+                                                           normally be modified by users.
+========================================================== ===================================================================
+
+Types
+#####
+========================================================== ==================================================================
+:bro:type:`PacketFilter::FilterPlugin`: :bro:type:`record` A data structure to represent filter generating plugins.
+:bro:type:`PacketFilter::Info`: :bro:type:`record`         The record type defining columns to be logged in the packet filter
+                                                           logging stream.
+========================================================== ==================================================================
+
+Redefinitions
+#############
+========================================== =================================================
+:bro:type:`Log::ID`: :bro:type:`enum`      Add the packet filter logging stream.
+:bro:type:`Notice::Type`: :bro:type:`enum` Add notice types related to packet filter errors.
+:bro:type:`PcapFilterID`: :bro:type:`enum` 
+========================================== =================================================
+
+Functions
+#########
+==================================================================== ======================================================================
+:bro:id:`PacketFilter::exclude`: :bro:type:`function`                Install a BPF filter to exclude some traffic.
+:bro:id:`PacketFilter::exclude_for`: :bro:type:`function`            Install a temporary filter to traffic which should not be passed
+                                                                     through the BPF filter.
+:bro:id:`PacketFilter::install`: :bro:type:`function`                Call this function to build and install a new dynamically built
+                                                                     packet filter.
+:bro:id:`PacketFilter::register_filter_plugin`: :bro:type:`function` API function to register a new plugin for dynamic restriction filters.
+==================================================================== ======================================================================
+
+
+Detailed Interface
+~~~~~~~~~~~~~~~~~~
+Redefinable Options
+###################
+.. bro:id:: PacketFilter::default_capture_filter
+
+   :Type: :bro:type:`string`
+   :Attributes: :bro:attr:`&redef`
+   :Default: ``"ip or not ip"``
+
+   The BPF filter that is used by default to define what traffic should
+   be captured.  Filters defined in :bro:id:`restrict_filters` will
+   still be applied to reduce the captured traffic.
+
+.. bro:id:: PacketFilter::enable_auto_protocol_capture_filters
+
+   :Type: :bro:type:`bool`
+   :Attributes: :bro:attr:`&redef`
+   :Default: ``F``
+
+   Enables the old filtering approach of "only watch common ports for
+   analyzed protocols".
+   
+   Unless you know what you are doing, leave this set to F.
+
+.. bro:id:: PacketFilter::max_filter_compile_time
+
+   :Type: :bro:type:`interval`
+   :Attributes: :bro:attr:`&redef`
+   :Default: ``100.0 msecs``
+
+   The maximum amount of time that you'd like to allow for BPF filters to compile.
+   If this time is exceeded, compensation measures may be taken by the framework
+   to reduce the filter size.  This threshold being crossed also results
+   in the :bro:see:`PacketFilter::Too_Long_To_Compile_Filter` notice.
+
+.. bro:id:: PacketFilter::restricted_filter
+
+   :Type: :bro:type:`string`
+   :Attributes: :bro:attr:`&redef`
+   :Default: ``""``
+
+   Filter string which is unconditionally and'ed to the beginning of
+   every dynamically built filter.  This is mostly used when a custom
+   filter is being used but MPLS or VLAN tags are on the traffic.
+
+.. bro:id:: PacketFilter::unrestricted_filter
+
+   :Type: :bro:type:`string`
+   :Attributes: :bro:attr:`&redef`
+   :Default: ``""``
+
+   Filter string which is unconditionally or'ed to the beginning of
+   every dynamically built filter.
+
+State Variables
+###############
+.. bro:id:: PacketFilter::current_filter
+
+   :Type: :bro:type:`string`
+   :Default: ``"<not set yet>"``
+
+   This is where the default packet filter is stored and it should not
+   normally be modified by users.
+
+Types
+#####
+.. bro:type:: PacketFilter::FilterPlugin
+
+   :Type: :bro:type:`record`
+
+      func: :bro:type:`function` () : :bro:type:`void`
+         A function that is directly called when generating the complete filter.
+
+   A data structure to represent filter generating plugins.
+
+.. bro:type:: PacketFilter::Info
+
+   :Type: :bro:type:`record`
+
+      ts: :bro:type:`time` :bro:attr:`&log`
+         The time at which the packet filter installation attempt was made.
+
+      node: :bro:type:`string` :bro:attr:`&log` :bro:attr:`&optional`
+         This is a string representation of the node that applied this
+         packet filter.  It's mostly useful in the context of
+         dynamically changing filters on clusters.
+
+      filter: :bro:type:`string` :bro:attr:`&log`
+         The packet filter that is being set.
+
+      init: :bro:type:`bool` :bro:attr:`&log` :bro:attr:`&default` = ``F`` :bro:attr:`&optional`
+         Indicate if this is the filter set during initialization.
+
+      success: :bro:type:`bool` :bro:attr:`&log` :bro:attr:`&default` = ``T`` :bro:attr:`&optional`
+         Indicate if the filter was applied successfully.
+
+   The record type defining columns to be logged in the packet filter
+   logging stream.
+
+Functions
+#########
+.. bro:id:: PacketFilter::exclude
+
+   :Type: :bro:type:`function` (filter_id: :bro:type:`string`, filter: :bro:type:`string`) : :bro:type:`bool`
+
+   Install a BPF filter to exclude some traffic.  The filter should
+   positively match what is to be excluded, it will be wrapped in
+   a "not".
+   
+
+   :filter_id: An arbitrary string that can be used to identify
+              the filter.
+   
+
+   :filter: A BPF expression of traffic that should be excluded.
+   
+
+   :returns: A boolean value to indicate if the filter was successfully
+            installed or not.
+
+.. bro:id:: PacketFilter::exclude_for
+
+   :Type: :bro:type:`function` (filter_id: :bro:type:`string`, filter: :bro:type:`string`, span: :bro:type:`interval`) : :bro:type:`bool`
+
+   Install a temporary filter to traffic which should not be passed
+   through the BPF filter.  The filter should match the traffic you
+   don't want to see (it will be wrapped in a "not" condition).
+   
+
+   :filter_id: An arbitrary string that can be used to identify
+              the filter.
+   
+
+   :filter: A BPF expression of traffic that should be excluded.
+   
+
+   :length: The duration for which this filter should be put in place.
+   
+
+   :returns: A boolean value to indicate if the filter was successfully
+            installed or not.
+
+.. bro:id:: PacketFilter::install
+
+   :Type: :bro:type:`function` () : :bro:type:`bool`
+
+   Call this function to build and install a new dynamically built
+   packet filter.
+
+.. bro:id:: PacketFilter::register_filter_plugin
+
+   :Type: :bro:type:`function` (fp: :bro:type:`PacketFilter::FilterPlugin`) : :bro:type:`void`
+
+   API function to register a new plugin for dynamic restriction filters.
+
+

doc/scripts/base/frameworks/packet-filter/netstats.bro.rst

@@ -0,0 +1,40 @@
+:tocdepth: 3
+
+base/frameworks/packet-filter/netstats.bro
+==========================================
+.. bro:namespace:: PacketFilter
+
+This script reports on packet loss from the various packet sources.
+When Bro is reading input from trace files, this script will not
+report any packet loss statistics.
+
+:Namespace: PacketFilter
+:Imports: :doc:`base/frameworks/notice </scripts/base/frameworks/notice/index>`
+
+Summary
+~~~~~~~
+Constants
+#########
+======================================================================= ==============================================================
+:bro:id:`PacketFilter::stats_collection_interval`: :bro:type:`interval` This is the interval between individual statistics collection.
+======================================================================= ==============================================================
+
+Redefinitions
+#############
+========================================== =
+:bro:type:`Notice::Type`: :bro:type:`enum` 
+========================================== =
+
+
+Detailed Interface
+~~~~~~~~~~~~~~~~~~
+Constants
+#########
+.. bro:id:: PacketFilter::stats_collection_interval
+
+   :Type: :bro:type:`interval`
+   :Default: ``5.0 mins``
+
+   This is the interval between individual statistics collection.
+
+

doc/scripts/base/frameworks/packet-filter/utils.bro.rst

@@ -0,0 +1,73 @@
+:tocdepth: 3
+
+base/frameworks/packet-filter/utils.bro
+=======================================
+.. bro:namespace:: PacketFilter
+
+
+:Namespace: PacketFilter
+
+Summary
+~~~~~~~
+Functions
+#########
+============================================================= ==================================================================
+:bro:id:`PacketFilter::combine_filters`: :bro:type:`function` Combines two valid BPF filter strings with a string based operator
+                                                              to form a new filter.
+:bro:id:`PacketFilter::port_to_bpf`: :bro:type:`function`     Takes a :bro:type:`port` and returns a BPF expression which will
+                                                              match the port.
+:bro:id:`PacketFilter::sampling_filter`: :bro:type:`function` Create a BPF filter to sample IPv4 and IPv6 traffic.
+============================================================= ==================================================================
+
+
+Detailed Interface
+~~~~~~~~~~~~~~~~~~
+Functions
+#########
+.. bro:id:: PacketFilter::combine_filters
+
+   :Type: :bro:type:`function` (lfilter: :bro:type:`string`, op: :bro:type:`string`, rfilter: :bro:type:`string`) : :bro:type:`string`
+
+   Combines two valid BPF filter strings with a string based operator
+   to form a new filter.
+   
+
+   :lfilter: Filter which will go on the left side.
+   
+
+   :op: Operation being applied (typically "or" or "and").
+   
+
+   :rfilter: Filter which will go on the right side.
+   
+
+   :returns: A new string representing the two filters combined with
+            the operator.  Either filter being an empty string will
+            still result in a valid filter.
+
+.. bro:id:: PacketFilter::port_to_bpf
+
+   :Type: :bro:type:`function` (p: :bro:type:`port`) : :bro:type:`string`
+
+   Takes a :bro:type:`port` and returns a BPF expression which will
+   match the port.
+   
+
+   :p: The port.
+   
+
+   :returns: A valid BPF filter string for matching the port.
+
+.. bro:id:: PacketFilter::sampling_filter
+
+   :Type: :bro:type:`function` (num_parts: :bro:type:`count`, this_part: :bro:type:`count`) : :bro:type:`string`
+
+   Create a BPF filter to sample IPv4 and IPv6 traffic.
+   
+
+   :num_parts: The number of parts the traffic should be split into.
+   
+
+   :this_part: The part of the traffic this filter will accept (0-based).
+
+

doc/scripts/base/frameworks/reporter/__load__.bro.rst

@@ -0,0 +1,14 @@
+:tocdepth: 3
+
+base/frameworks/reporter/__load__.bro
+=====================================
+
+
+:Imports: :doc:`base/frameworks/reporter/main.bro </scripts/base/frameworks/reporter/main.bro>`
+
+Summary
+~~~~~~~
+
+Detailed Interface
+~~~~~~~~~~~~~~~~~~
+

doc/scripts/base/frameworks/reporter/index.rst

@@ -0,0 +1,26 @@
+:orphan:
+
+Package: base/frameworks/reporter
+=================================
+
+This framework is intended to create an output and filtering path for
+internally generated messages/warnings/errors.
+
+:doc:`/scripts/base/frameworks/reporter/__load__.bro`
+
+
+:doc:`/scripts/base/frameworks/reporter/main.bro`
+
+   This framework is intended to create an output and filtering path for
+   internal messages/warnings/errors.  It should typically be loaded to
+   log such messages to a file in a standard way.  For the options to
+   toggle whether messages are additionally written to STDERR, see
+   :bro:see:`Reporter::info_to_stderr`,
+   :bro:see:`Reporter::warnings_to_stderr`, and
+   :bro:see:`Reporter::errors_to_stderr`.
+   
+   Note that this framework deals with the handling of internally generated
+   reporter messages, for the interface
+   into actually creating reporter messages from the scripting layer, use
+   the built-in functions in :doc:`/scripts/base/bif/reporter.bif.bro`.
+

doc/scripts/base/frameworks/reporter/main.bro.rst

@@ -0,0 +1,64 @@
+:tocdepth: 3
+
+base/frameworks/reporter/main.bro
+=================================
+.. bro:namespace:: Reporter
+
+This framework is intended to create an output and filtering path for
+internal messages/warnings/errors.  It should typically be loaded to
+log such messages to a file in a standard way.  For the options to
+toggle whether messages are additionally written to STDERR, see
+:bro:see:`Reporter::info_to_stderr`,
+:bro:see:`Reporter::warnings_to_stderr`, and
+:bro:see:`Reporter::errors_to_stderr`.
+
+Note that this framework deals with the handling of internally generated
+reporter messages, for the interface
+into actually creating reporter messages from the scripting layer, use
+the built-in functions in :doc:`/scripts/base/bif/reporter.bif.bro`.
+
+:Namespace: Reporter
+
+Summary
+~~~~~~~
+Types
+#####
+============================================== =====================================================================
+:bro:type:`Reporter::Info`: :bro:type:`record` The record type which contains the column fields of the reporter log.
+============================================== =====================================================================
+
+Redefinitions
+#############
+===================================== =======================================
+:bro:type:`Log::ID`: :bro:type:`enum` The reporter logging stream identifier.
+===================================== =======================================
+
+
+Detailed Interface
+~~~~~~~~~~~~~~~~~~
+Types
+#####
+.. bro:type:: Reporter::Info
+
+   :Type: :bro:type:`record`
+
+      ts: :bro:type:`time` :bro:attr:`&log`
+         The network time at which the reporter event was generated.
+
+      level: :bro:type:`Reporter::Level` :bro:attr:`&log`
+         The severity of the reporter message. Levels are INFO for informational
+         messages, not needing specific attention; WARNING for warning of a potential
+         problem, and ERROR for a non-fatal error that should be addressed, but doesn't
+         terminate program execution.
+
+      message: :bro:type:`string` :bro:attr:`&log`
+         An info/warning/error message that could have either been
+         generated from the internal Bro core or at the scripting-layer.
+
+      location: :bro:type:`string` :bro:attr:`&log` :bro:attr:`&optional`
+         This is the location in a Bro script where the message originated.
+         Not all reporter messages will have locations in them though.
+
+   The record type which contains the column fields of the reporter log.
+
+

doc/scripts/base/frameworks/signatures/__load__.bro.rst

@@ -0,0 +1,14 @@
+:tocdepth: 3
+
+base/frameworks/signatures/__load__.bro
+=======================================
+
+
+:Imports: :doc:`base/frameworks/signatures/main.bro </scripts/base/frameworks/signatures/main.bro>`
+
+Summary
+~~~~~~~
+
+Detailed Interface
+~~~~~~~~~~~~~~~~~~
+

doc/scripts/base/frameworks/signatures/index.rst

@@ -0,0 +1,19 @@
+:orphan:
+
+Package: base/frameworks/signatures
+===================================
+
+The signature framework provides for doing low-level pattern matching.  While
+signatures are not Bro's preferred detection tool, they sometimes come in
+handy and are closer to what many people are familiar with from using
+other NIDS.
+
+:doc:`/scripts/base/frameworks/signatures/__load__.bro`
+
+
+:doc:`/scripts/base/frameworks/signatures/main.bro`
+
+   Script level signature support.  See the
+   :doc:`signature documentation </frameworks/signatures>` for more
+   information about Bro's signature engine.
+

doc/scripts/base/frameworks/signatures/main.bro.rst

@@ -0,0 +1,272 @@
+:tocdepth: 3
+
+base/frameworks/signatures/main.bro
+===================================
+.. bro:namespace:: Signatures
+
+Script level signature support.  See the
+:doc:`signature documentation </frameworks/signatures>` for more
+information about Bro's signature engine.
+
+:Namespace: Signatures
+:Imports: :doc:`base/frameworks/notice </scripts/base/frameworks/notice/index>`
+
+Summary
+~~~~~~~
+Runtime Options
+###############
+=============================================================================== ===================================================================
+:bro:id:`Signatures::ignored_ids`: :bro:type:`pattern` :bro:attr:`&redef`       Signature IDs that should always be ignored.
+:bro:id:`Signatures::summary_interval`: :bro:type:`interval` :bro:attr:`&redef` The interval between when :bro:enum:`Signatures::Signature_Summary`
+                                                                                notices are generated.
+=============================================================================== ===================================================================
+
+Redefinable Options
+###################
+========================================================================================================================================== ====================================================================
+:bro:id:`Signatures::actions`: :bro:type:`table` :bro:attr:`&redef` :bro:attr:`&default` = ``Signatures::SIG_ALARM`` :bro:attr:`&optional` Actions for a signature.
+:bro:id:`Signatures::count_thresholds`: :bro:type:`set` :bro:attr:`&redef`                                                                 Generate a notice if a :bro:enum:`Signatures::SIG_COUNT_PER_RESP`
+                                                                                                                                           signature is triggered as often as given by one of these thresholds.
+:bro:id:`Signatures::horiz_scan_thresholds`: :bro:type:`set` :bro:attr:`&redef`                                                            Generate a notice if, for a pair [orig, signature], the number of
+                                                                                                                                           different responders has reached one of the thresholds.
+:bro:id:`Signatures::vert_scan_thresholds`: :bro:type:`set` :bro:attr:`&redef`                                                             Generate a notice if, for a pair [orig, resp], the number of
+                                                                                                                                           different signature matches has reached one of the thresholds.
+========================================================================================================================================== ====================================================================
+
+Types
+#####
+================================================ ======================================================================
+:bro:type:`Signatures::Action`: :bro:type:`enum` These are the default actions you can apply to signature matches.
+:bro:type:`Signatures::Info`: :bro:type:`record` The record type which contains the column fields of the signature log.
+================================================ ======================================================================
+
+Redefinitions
+#############
+========================================== ===========================================
+:bro:type:`Log::ID`: :bro:type:`enum`      The signature logging stream identifier.
+:bro:type:`Notice::Type`: :bro:type:`enum` Add various signature-related notice types.
+========================================== ===========================================
+
+Events
+######
+====================================================== =================================================================
+:bro:id:`Signatures::log_signature`: :bro:type:`event` This event can be handled to access/alter data about to be logged
+                                                       to the signature logging stream.
+====================================================== =================================================================
+
+
+Detailed Interface
+~~~~~~~~~~~~~~~~~~
+Runtime Options
+###############
+.. bro:id:: Signatures::ignored_ids
+
+   :Type: :bro:type:`pattern`
+   :Attributes: :bro:attr:`&redef`
+   :Default:
+
+   ::
+
+      /(^?(^?(^webapp-)$?)$?)|(^?((^?(^?(traceroute-detector.*)$?)$?)|(^?(^?(NO_DEFAULT_MATCHES)$?)$?))$?)/
+
+   Signature IDs that should always be ignored.
+
+.. bro:id:: Signatures::summary_interval
+
+   :Type: :bro:type:`interval`
+   :Attributes: :bro:attr:`&redef`
+   :Default: ``1.0 day``
+
+   The interval between when :bro:enum:`Signatures::Signature_Summary`
+   notices are generated.
+
+Redefinable Options
+###################
+.. bro:id:: Signatures::actions
+
+   :Type: :bro:type:`table` [:bro:type:`string`] of :bro:type:`Signatures::Action`
+   :Attributes: :bro:attr:`&redef` :bro:attr:`&default` = ``Signatures::SIG_ALARM`` :bro:attr:`&optional`
+   :Default:
+
+   ::
+
+      {
+         ["unspecified"] = Signatures::SIG_IGNORE
+      }
+
+   Actions for a signature.  
+
+.. bro:id:: Signatures::count_thresholds
+
+   :Type: :bro:type:`set` [:bro:type:`count`]
+   :Attributes: :bro:attr:`&redef`
+   :Default:
+
+   ::
+
+      {
+         500,
+         1000,
+         1000000,
+         5,
+         100,
+         50,
+         10000,
+         10
+      }
+
+   Generate a notice if a :bro:enum:`Signatures::SIG_COUNT_PER_RESP`
+   signature is triggered as often as given by one of these thresholds.
+
+.. bro:id:: Signatures::horiz_scan_thresholds
+
+   :Type: :bro:type:`set` [:bro:type:`count`]
+   :Attributes: :bro:attr:`&redef`
+   :Default:
+
+   ::
+
+      {
+         500,
+         1000,
+         5,
+         100,
+         50,
+         10
+      }
+
+   Generate a notice if, for a pair [orig, signature], the number of
+   different responders has reached one of the thresholds.
+
+.. bro:id:: Signatures::vert_scan_thresholds
+
+   :Type: :bro:type:`set` [:bro:type:`count`]
+   :Attributes: :bro:attr:`&redef`
+   :Default:
+
+   ::
+
+      {
+         500,
+         1000,
+         5,
+         100,
+         50,
+         10
+      }
+
+   Generate a notice if, for a pair [orig, resp], the number of
+   different signature matches has reached one of the thresholds.
+
+Types
+#####
+.. bro:type:: Signatures::Action
+
+   :Type: :bro:type:`enum`
+
+      .. bro:enum:: Signatures::SIG_IGNORE Signatures::Action
+
+         Ignore this signature completely (even for scan detection).
+         Don't write to the signatures logging stream.
+
+      .. bro:enum:: Signatures::SIG_QUIET Signatures::Action
+
+         Process through the various aggregate techniques, but don't
+         report individually and don't write to the signatures logging
+         stream.
+
+      .. bro:enum:: Signatures::SIG_LOG Signatures::Action
+
+         Generate a notice.
+
+      .. bro:enum:: Signatures::SIG_FILE_BUT_NO_SCAN Signatures::Action
+
+         The same as :bro:enum:`Signatures::SIG_LOG`, but ignore for
+         aggregate/scan processing.
+
+      .. bro:enum:: Signatures::SIG_ALARM Signatures::Action
+
+         Generate a notice and set it to be alarmed upon.
+
+      .. bro:enum:: Signatures::SIG_ALARM_PER_ORIG Signatures::Action
+
+         Alarm once per originator.
+
+      .. bro:enum:: Signatures::SIG_ALARM_ONCE Signatures::Action
+
+         Alarm once and then never again.
+
+      .. bro:enum:: Signatures::SIG_COUNT_PER_RESP Signatures::Action
+
+         Count signatures per responder host and alarm with the 
+         :bro:enum:`Signatures::Count_Signature` notice if a threshold
+         defined by :bro:id:`Signatures::count_thresholds` is reached.
+
+      .. bro:enum:: Signatures::SIG_SUMMARY Signatures::Action
+
+         Don't alarm, but generate per-orig summary.
+
+   These are the default actions you can apply to signature matches.
+   All of them write the signature record to the logging stream unless
+   declared otherwise.
+
+.. bro:type:: Signatures::Info
+
+   :Type: :bro:type:`record`
+
+      ts: :bro:type:`time` :bro:attr:`&log`
+         The network time at which a signature matching type of event
+         to be logged has occurred.
+
+      uid: :bro:type:`string` :bro:attr:`&log` :bro:attr:`&optional`
+         A unique identifier of the connection which triggered the
+         signature match event.
+
+      src_addr: :bro:type:`addr` :bro:attr:`&log` :bro:attr:`&optional`
+         The host which triggered the signature match event.
+
+      src_port: :bro:type:`port` :bro:attr:`&log` :bro:attr:`&optional`
+         The host port on which the signature-matching activity
+         occurred.
+
+      dst_addr: :bro:type:`addr` :bro:attr:`&log` :bro:attr:`&optional`
+         The destination host which was sent the payload that
+         triggered the signature match.
+
+      dst_port: :bro:type:`port` :bro:attr:`&log` :bro:attr:`&optional`
+         The destination host port which was sent the payload that
+         triggered the signature match.
+
+      note: :bro:type:`Notice::Type` :bro:attr:`&log`
+         Notice associated with signature event.
+
+      sig_id: :bro:type:`string` :bro:attr:`&log` :bro:attr:`&optional`
+         The name of the signature that matched.
+
+      event_msg: :bro:type:`string` :bro:attr:`&log` :bro:attr:`&optional`
+         A more descriptive message of the signature-matching event.
+
+      sub_msg: :bro:type:`string` :bro:attr:`&log` :bro:attr:`&optional`
+         Extracted payload data or extra message.
+
+      sig_count: :bro:type:`count` :bro:attr:`&log` :bro:attr:`&optional`
+         Number of sigs, usually from summary count.
+
+      host_count: :bro:type:`count` :bro:attr:`&log` :bro:attr:`&optional`
+         Number of hosts, from a summary count.
+
+   The record type which contains the column fields of the signature log.
+
+Events
+######
+.. bro:id:: Signatures::log_signature
+
+   :Type: :bro:type:`event` (rec: :bro:type:`Signatures::Info`)
+
+   This event can be handled to access/alter data about to be logged
+   to the signature logging stream.
+   
+
+   :rec: The record of signature data about to be logged.
+
+

doc/scripts/base/frameworks/software/__load__.bro.rst

@@ -0,0 +1,14 @@
+:tocdepth: 3
+
+base/frameworks/software/__load__.bro
+=====================================
+
+
+:Imports: :doc:`base/frameworks/software/main.bro </scripts/base/frameworks/software/main.bro>`
+
+Summary
+~~~~~~~
+
+Detailed Interface
+~~~~~~~~~~~~~~~~~~
+