Documentation improvements/fixes

CHANGES

@@ -1,4 +1,8 @@
 
+2.5-599 | 2018-05-23 16:50:12 -0500
+
+  * Documentation improvements/fixes (Corelight)
+
 2.5-598 | 2018-05-22 15:05:24 -0500
 
   * Fixes for MySQL and SMB protocol parsers (Corelight)

VERSION

@@ -1 +1 @@
-2.5-598
+2.5-599

aux/broker

@@ -1 +1 @@
-Subproject commit 39ff1195bd22f412fd91073c5df3014a2e03c4e9
+Subproject commit b29708e14f33c87c4c9dbfe1ebe7c7c1482c9f6b

doc/components/index.rst

@@ -20,6 +20,3 @@ current, independent component releases.
    Capstats - Command-line packet statistic tool <capstats/README>
    PySubnetTree - Python module for CIDR lookups<pysubnettree/README>
    trace-summary - Script for generating break-downs of network traffic <trace-summary/README>
-
-The `Broker User Manual <../broker-manual/index.html>`_ may also be of
-interest.

doc/frameworks/broker.rst

@@ -74,11 +74,6 @@ Notable / Specific Script API Changes
 - :bro:see:`Software::tracked` is now partitioned among proxy nodes
   instead of synchronized in its entirety to all nodes.
 
-- ``Known::known_devices`` is renamed to :bro:see:`Known::device_store`
-  and implemented via the new Broker data store interface.
-  Also use :bro:see:`Known::device_found` instead of updating the
-  store directly directly.
-
 - ``Known::known_hosts`` is renamed to :bro:see:`Known::host_store` and
   implemented via the new Broker data store interface.
 
@@ -116,13 +111,18 @@ Some general suggestions as to the purpose/utilization of each node type:
 
 - Proxies: serve as intermediaries for data storage and work/calculation
   offloading.  Good for helping offload work or data in a scalable and
-  distributed way.  i.e. since any given worker is connected to all
+  distributed way.  Since any given worker is connected to all
   proxies and can agree on an "arbitrary key -> proxy node" mapping
   (more on that later), you can partition work or data amongst them in a
   uniform manner.  e.g. you might choose to use proxies as a method of
   sharing non-persistent state or as a "second pass" analysis for any
   work that you don't want interferring with the workers' capacity to
-  keep up with capturing and parsing packets.
+  keep up with capturing and parsing packets.  Note that the default scripts
+  that come with Bro don't utilize proxies themselves, so if you are coming
+  from a previous BroControl deployment, you may want to try reducing down
+  to a single proxy node.  If you come to have custom/community scripts
+  that utilize proxies, that would be the time to start considering scaling
+  up the number of proxies to meet demands.
 
 - Manager: this node will be good at performing decisions that require a
   global view of things since it is in a centralized location, connected
@@ -179,10 +179,12 @@ Data Partitioning
 -----------------
 
 New data partitioning strategies are available using the API in
-:doc:`/scripts/base/frameworks/cluster/pools.bro`.
+:doc:`/scripts/base/frameworks/cluster/pools.bro`.  Using that API, developers
+of custom Bro scripts can define a custom pool of nodes that best fits the
+needs of their script.
 
 One example strategy is to use Highest Random Weight (HRW) hashing to
-partition data tables amongst proxy nodes.  e.g. using
+partition data tables amongst the pool of all proxy nodes.  e.g. using
 :bro:see:`Cluster::publish_hrw`.  This could allow clusters to
 be scaled more easily than the approach of "the entire data set gets
 synchronized to all nodes" as the solution to memory limitations becomes
@@ -195,6 +197,13 @@ crashing, etc.) cause a temporary gap in the total data set until
 workers start hashing keys to a new proxy node that is still alive,
 causing data to now be located and updated there.
 
+If the developer of a script expects its workload to be particularly
+intensive, wants to ensure that their operations get exclusive
+access to nodes, or otherwise set containts on the number of nodes within
+a pool utilized by their script, then the :bro:see:`Cluster::PoolSpec`
+structure will allow them to that while still allowing users of that script
+to override the default suggestions made by the original developer.
+
 Broker Framework Examples
 =========================
 

doc/index.rst

@@ -45,8 +45,6 @@ Reference Section
    script-reference/index.rst
    components/index.rst
 
-`Broker User Manual <../broker-manual/index.html>`_
-
 Development
 ===========
 

doc/install/index.rst

@@ -10,3 +10,4 @@ Installation
 
    install
    upgrade
+   cross-compiling

doc/script-reference/log-files.rst

@@ -130,9 +130,6 @@ Network Observations
 +============================+=======================================+=================================+
 | known_certs.log            | SSL certificates                      | :bro:type:`Known::CertsInfo`    |
 +----------------------------+---------------------------------------+---------------------------------+
-| known_devices.log          | MAC addresses of devices on the       | :bro:type:`Known::DevicesInfo`  |
-|                            | network                               |                                 |
-+----------------------------+---------------------------------------+---------------------------------+
 | known_hosts.log            | Hosts that have completed TCP         | :bro:type:`Known::HostsInfo`    |
 |                            | handshakes                            |                                 |
 +----------------------------+---------------------------------------+---------------------------------+

doc/scripting/index.rst

@@ -171,7 +171,7 @@ write scripts for Bro but for understanding Bro itself.
 Gaining familiarity with the specific events generated by Bro is a big
 step towards building a mind set for working with Bro scripts.  The
 majority of events generated by Bro are defined in the
-built-in-function files or ``.bif`` files which also act as the basis for
+built-in-function (``*.bif``) files which also act as the basis for
 online event documentation.  These in-line comments are compiled into
 an online documentation system using Broxygen.  Whether starting a
 script from scratch or reading and maintaining someone else's script,
@@ -212,11 +212,11 @@ later.
 While Bro is capable of packet level processing, its strengths lay in
 the context of a connection between an originator and a responder.  As
 such, there are events defined for the primary parts of the connection
-life-cycle as you'll see from the small selection of
-connection-related events below.  
+life-cycle such as the following:
 
-.. btest-include:: ${BRO_SRC_ROOT}/build/scripts/base/bif/event.bif.bro
-   :lines: 69-72,88,106-109,129,132-137,148
+* :bro:see:`new_connection`
+* :bro:see:`connection_timeout`
+* :bro:see:`connection_state_remove`
 
 Of the events listed, the event that will give us the best insight
 into the connection record data type will be

testing/btest/Baseline/doc.sphinx.include-build_scripts_base_bif_plugins_Bro_DNS_events_bif_bro/output

@@ -0,0 +1,30 @@
+# @TEST-EXEC: cat %INPUT >output && btest-diff output
+
+Bro_DNS.events.bif.bro
+
+## Generated for DNS requests. For requests with multiple queries, this event
+## is raised once for each.
+##
+## See `Wikipedia <http://en.wikipedia.org/wiki/Domain_Name_System>`__ for more
+## information about the DNS protocol. Bro analyzes both UDP and TCP DNS
+## sessions.
+##
+## c: The connection, which may be UDP or TCP depending on the type of the
+##    transport-layer session being analyzed.
+##
+## msg: The parsed DNS message header.
+##
+## query: The queried name.
+##
+## qtype: The queried resource record type.
+##
+## qclass: The queried resource record class.
+##
+## .. bro:see:: dns_AAAA_reply dns_A_reply dns_CNAME_reply dns_EDNS_addl
+##    dns_HINFO_reply dns_MX_reply dns_NS_reply dns_PTR_reply dns_SOA_reply
+##    dns_SRV_reply dns_TSIG_addl dns_TXT_reply dns_WKS_reply dns_end
+##    dns_full_request dns_mapping_altered dns_mapping_lost_name dns_mapping_new_name
+##    dns_mapping_unverified dns_mapping_valid dns_message dns_query_reply
+##    dns_rejected non_dns_request dns_max_queries dns_session_timeout dns_skip_addl
+##    dns_skip_all_addl dns_skip_all_auth dns_skip_auth
+global dns_request: event(c: connection , msg: dns_msg , query: string , qtype: count , qclass: count );

testing/btest/doc/sphinx/include-build_scripts_base_bif_plugins_Bro_DNS_events_bif_bro.btest

@@ -0,0 +1,30 @@
+# @TEST-EXEC: cat %INPUT >output && btest-diff output
+
+Bro_DNS.events.bif.bro
+
+## Generated for DNS requests. For requests with multiple queries, this event
+## is raised once for each.
+##
+## See `Wikipedia <http://en.wikipedia.org/wiki/Domain_Name_System>`__ for more
+## information about the DNS protocol. Bro analyzes both UDP and TCP DNS
+## sessions.
+##
+## c: The connection, which may be UDP or TCP depending on the type of the
+##    transport-layer session being analyzed.
+##
+## msg: The parsed DNS message header.
+##
+## query: The queried name.
+##
+## qtype: The queried resource record type.
+##
+## qclass: The queried resource record class.
+##
+## .. bro:see:: dns_AAAA_reply dns_A_reply dns_CNAME_reply dns_EDNS_addl
+##    dns_HINFO_reply dns_MX_reply dns_NS_reply dns_PTR_reply dns_SOA_reply
+##    dns_SRV_reply dns_TSIG_addl dns_TXT_reply dns_WKS_reply dns_end
+##    dns_full_request dns_mapping_altered dns_mapping_lost_name dns_mapping_new_name
+##    dns_mapping_unverified dns_mapping_valid dns_message dns_query_reply
+##    dns_rejected non_dns_request dns_max_queries dns_session_timeout dns_skip_addl
+##    dns_skip_all_addl dns_skip_all_auth dns_skip_auth
+global dns_request: event(c: connection , msg: dns_msg , query: string , qtype: count , qclass: count );