diff --git a/CHANGES b/CHANGES index 826e5b2f6e..d2578bcf01 100644 --- a/CHANGES +++ b/CHANGES @@ -1,4 +1,8 @@ +2.5-599 | 2018-05-23 16:50:12 -0500 + + * Documentation improvements/fixes (Corelight) + 2.5-598 | 2018-05-22 15:05:24 -0500 * Fixes for MySQL and SMB protocol parsers (Corelight) diff --git a/VERSION b/VERSION index f281a4fb84..5b206a3bed 100644 --- a/VERSION +++ b/VERSION @@ -1 +1 @@ -2.5-598 +2.5-599 diff --git a/aux/broker b/aux/broker index 39ff1195bd..b29708e14f 160000 --- a/aux/broker +++ b/aux/broker @@ -1 +1 @@ -Subproject commit 39ff1195bd22f412fd91073c5df3014a2e03c4e9 +Subproject commit b29708e14f33c87c4c9dbfe1ebe7c7c1482c9f6b diff --git a/doc/components/index.rst b/doc/components/index.rst index 2a6624480b..2f69b3ef54 100644 --- a/doc/components/index.rst +++ b/doc/components/index.rst @@ -20,6 +20,3 @@ current, independent component releases. Capstats - Command-line packet statistic tool PySubnetTree - Python module for CIDR lookups trace-summary - Script for generating break-downs of network traffic - -The `Broker User Manual <../broker-manual/index.html>`_ may also be of -interest. diff --git a/doc/frameworks/broker.rst b/doc/frameworks/broker.rst index c2904881c7..23b1f91478 100644 --- a/doc/frameworks/broker.rst +++ b/doc/frameworks/broker.rst @@ -74,11 +74,6 @@ Notable / Specific Script API Changes - :bro:see:`Software::tracked` is now partitioned among proxy nodes instead of synchronized in its entirety to all nodes. -- ``Known::known_devices`` is renamed to :bro:see:`Known::device_store` - and implemented via the new Broker data store interface. - Also use :bro:see:`Known::device_found` instead of updating the - store directly directly. - - ``Known::known_hosts`` is renamed to :bro:see:`Known::host_store` and implemented via the new Broker data store interface. @@ -116,13 +111,18 @@ Some general suggestions as to the purpose/utilization of each node type: - Proxies: serve as intermediaries for data storage and work/calculation offloading. Good for helping offload work or data in a scalable and - distributed way. i.e. since any given worker is connected to all + distributed way. Since any given worker is connected to all proxies and can agree on an "arbitrary key -> proxy node" mapping (more on that later), you can partition work or data amongst them in a uniform manner. e.g. you might choose to use proxies as a method of sharing non-persistent state or as a "second pass" analysis for any work that you don't want interferring with the workers' capacity to - keep up with capturing and parsing packets. + keep up with capturing and parsing packets. Note that the default scripts + that come with Bro don't utilize proxies themselves, so if you are coming + from a previous BroControl deployment, you may want to try reducing down + to a single proxy node. If you come to have custom/community scripts + that utilize proxies, that would be the time to start considering scaling + up the number of proxies to meet demands. - Manager: this node will be good at performing decisions that require a global view of things since it is in a centralized location, connected @@ -179,10 +179,12 @@ Data Partitioning ----------------- New data partitioning strategies are available using the API in -:doc:`/scripts/base/frameworks/cluster/pools.bro`. +:doc:`/scripts/base/frameworks/cluster/pools.bro`. Using that API, developers +of custom Bro scripts can define a custom pool of nodes that best fits the +needs of their script. One example strategy is to use Highest Random Weight (HRW) hashing to -partition data tables amongst proxy nodes. e.g. using +partition data tables amongst the pool of all proxy nodes. e.g. using :bro:see:`Cluster::publish_hrw`. This could allow clusters to be scaled more easily than the approach of "the entire data set gets synchronized to all nodes" as the solution to memory limitations becomes @@ -195,6 +197,13 @@ crashing, etc.) cause a temporary gap in the total data set until workers start hashing keys to a new proxy node that is still alive, causing data to now be located and updated there. +If the developer of a script expects its workload to be particularly +intensive, wants to ensure that their operations get exclusive +access to nodes, or otherwise set containts on the number of nodes within +a pool utilized by their script, then the :bro:see:`Cluster::PoolSpec` +structure will allow them to that while still allowing users of that script +to override the default suggestions made by the original developer. + Broker Framework Examples ========================= diff --git a/doc/index.rst b/doc/index.rst index 172ae6f171..22fb8cbe1a 100644 --- a/doc/index.rst +++ b/doc/index.rst @@ -45,8 +45,6 @@ Reference Section script-reference/index.rst components/index.rst -`Broker User Manual <../broker-manual/index.html>`_ - Development =========== diff --git a/doc/install/index.rst b/doc/install/index.rst index 71d2534921..2689b9adea 100644 --- a/doc/install/index.rst +++ b/doc/install/index.rst @@ -10,3 +10,4 @@ Installation install upgrade + cross-compiling diff --git a/doc/script-reference/log-files.rst b/doc/script-reference/log-files.rst index 285eb20a7c..8f7d87d89a 100644 --- a/doc/script-reference/log-files.rst +++ b/doc/script-reference/log-files.rst @@ -130,9 +130,6 @@ Network Observations +============================+=======================================+=================================+ | known_certs.log | SSL certificates | :bro:type:`Known::CertsInfo` | +----------------------------+---------------------------------------+---------------------------------+ -| known_devices.log | MAC addresses of devices on the | :bro:type:`Known::DevicesInfo` | -| | network | | -+----------------------------+---------------------------------------+---------------------------------+ | known_hosts.log | Hosts that have completed TCP | :bro:type:`Known::HostsInfo` | | | handshakes | | +----------------------------+---------------------------------------+---------------------------------+ diff --git a/doc/scripting/index.rst b/doc/scripting/index.rst index 1a410cef12..54ae83bf81 100644 --- a/doc/scripting/index.rst +++ b/doc/scripting/index.rst @@ -171,7 +171,7 @@ write scripts for Bro but for understanding Bro itself. Gaining familiarity with the specific events generated by Bro is a big step towards building a mind set for working with Bro scripts. The majority of events generated by Bro are defined in the -built-in-function files or ``.bif`` files which also act as the basis for +built-in-function (``*.bif``) files which also act as the basis for online event documentation. These in-line comments are compiled into an online documentation system using Broxygen. Whether starting a script from scratch or reading and maintaining someone else's script, @@ -212,11 +212,11 @@ later. While Bro is capable of packet level processing, its strengths lay in the context of a connection between an originator and a responder. As such, there are events defined for the primary parts of the connection -life-cycle as you'll see from the small selection of -connection-related events below. +life-cycle such as the following: -.. btest-include:: ${BRO_SRC_ROOT}/build/scripts/base/bif/event.bif.bro - :lines: 69-72,88,106-109,129,132-137,148 +* :bro:see:`new_connection` +* :bro:see:`connection_timeout` +* :bro:see:`connection_state_remove` Of the events listed, the event that will give us the best insight into the connection record data type will be diff --git a/testing/btest/Baseline/doc.sphinx.include-build_scripts_base_bif_plugins_Bro_DNS_events_bif_bro/output b/testing/btest/Baseline/doc.sphinx.include-build_scripts_base_bif_plugins_Bro_DNS_events_bif_bro/output new file mode 100644 index 0000000000..6e15ece5e0 --- /dev/null +++ b/testing/btest/Baseline/doc.sphinx.include-build_scripts_base_bif_plugins_Bro_DNS_events_bif_bro/output @@ -0,0 +1,30 @@ +# @TEST-EXEC: cat %INPUT >output && btest-diff output + +Bro_DNS.events.bif.bro + +## Generated for DNS requests. For requests with multiple queries, this event +## is raised once for each. +## +## See `Wikipedia `__ for more +## information about the DNS protocol. Bro analyzes both UDP and TCP DNS +## sessions. +## +## c: The connection, which may be UDP or TCP depending on the type of the +## transport-layer session being analyzed. +## +## msg: The parsed DNS message header. +## +## query: The queried name. +## +## qtype: The queried resource record type. +## +## qclass: The queried resource record class. +## +## .. bro:see:: dns_AAAA_reply dns_A_reply dns_CNAME_reply dns_EDNS_addl +## dns_HINFO_reply dns_MX_reply dns_NS_reply dns_PTR_reply dns_SOA_reply +## dns_SRV_reply dns_TSIG_addl dns_TXT_reply dns_WKS_reply dns_end +## dns_full_request dns_mapping_altered dns_mapping_lost_name dns_mapping_new_name +## dns_mapping_unverified dns_mapping_valid dns_message dns_query_reply +## dns_rejected non_dns_request dns_max_queries dns_session_timeout dns_skip_addl +## dns_skip_all_addl dns_skip_all_auth dns_skip_auth +global dns_request: event(c: connection , msg: dns_msg , query: string , qtype: count , qclass: count ); diff --git a/testing/btest/doc/sphinx/include-build_scripts_base_bif_plugins_Bro_DNS_events_bif_bro.btest b/testing/btest/doc/sphinx/include-build_scripts_base_bif_plugins_Bro_DNS_events_bif_bro.btest new file mode 100644 index 0000000000..6e15ece5e0 --- /dev/null +++ b/testing/btest/doc/sphinx/include-build_scripts_base_bif_plugins_Bro_DNS_events_bif_bro.btest @@ -0,0 +1,30 @@ +# @TEST-EXEC: cat %INPUT >output && btest-diff output + +Bro_DNS.events.bif.bro + +## Generated for DNS requests. For requests with multiple queries, this event +## is raised once for each. +## +## See `Wikipedia `__ for more +## information about the DNS protocol. Bro analyzes both UDP and TCP DNS +## sessions. +## +## c: The connection, which may be UDP or TCP depending on the type of the +## transport-layer session being analyzed. +## +## msg: The parsed DNS message header. +## +## query: The queried name. +## +## qtype: The queried resource record type. +## +## qclass: The queried resource record class. +## +## .. bro:see:: dns_AAAA_reply dns_A_reply dns_CNAME_reply dns_EDNS_addl +## dns_HINFO_reply dns_MX_reply dns_NS_reply dns_PTR_reply dns_SOA_reply +## dns_SRV_reply dns_TSIG_addl dns_TXT_reply dns_WKS_reply dns_end +## dns_full_request dns_mapping_altered dns_mapping_lost_name dns_mapping_new_name +## dns_mapping_unverified dns_mapping_valid dns_message dns_query_reply +## dns_rejected non_dns_request dns_max_queries dns_session_timeout dns_skip_addl +## dns_skip_all_addl dns_skip_all_auth dns_skip_auth +global dns_request: event(c: connection , msg: dns_msg , query: string , qtype: count , qclass: count );