Merge remote-tracking branch 'origin/topic/dnthayer/ticket1947'

* origin/topic/dnthayer/ticket1947:
  Fix some typos and formatting in NEWS
  Add pattern operators to the documentation of operators
  Fix minor typos in broker reference documentation
  Fix a broken link and some typos in broker documentation
  Fix reST formatting in documentation of "count" type
  Add documentation for some new Bro features

CHANGES

@@ -1,4 +1,12 @@
 
+2.5-719 | 2018-06-27 20:02:52 -0500
+
+  * Fix some typos and formatting in NEWS and other documentation
+    (Daniel Thayer)
+
+  * Add documentation for type-based switch statment, as/is operators,
+    bitwise operators, and pattern operators (Daniel Thayer)
+
 2.5-711 | 2018-06-27 19:11:58 -0500
 
   * Prevent double-wrapping Broker::Data in published event args (Corelight)

NEWS

@@ -18,7 +18,7 @@ New Functionality
   supersedes the old "communication" framework, which is now
   deprecated. The "cluster" and "control" frameworks have been ported
   to Broker; same for BroControl. For more about the new Broker
-  framework, see doc/frameworks/broker.rst (there's also guide there
+  framework, see doc/frameworks/broker.rst (there's also a guide there
   for porting existing Bro scripts to Broker). For more about Broker
   itself, including its API for external applications, see
   aux/broker/doc.
@@ -258,11 +258,11 @@ New Functionality
 Changed Functionality
 ---------------------
 
-  - All communication is now handled through Broker, requiring changes
+- All communication is now handled through Broker, requiring changes
-    to existing scripts to port them over to the new API. The Broker
+  to existing scripts to port them over to the new API. The Broker
-    framework documentation comes with a porting guide.
+  framework documentation comes with a porting guide.
 
-  - The DHCP analyzer and its script-layer interface have been rewritten.
+- The DHCP analyzer and its script-layer interface have been rewritten.
 
   - Supports more DHCP options than before.
 
@@ -272,9 +272,6 @@ Changed Functionality
   - Removed the policy/protocols/dhcp/known-devices-and-hostnames.bro
     script since it's generally less relevant now with the updated log.
 
-  - Removed policy/misc/known-devices.bro script and thus
-    known_devices.log will no longer be created.
-
   - Removed the base/protocols/dhcp/utils.bro script and thus the
     'reverse_ip' function.
 
@@ -290,6 +287,9 @@ Changed Functionality
     - dhcp_release
     - dhcp_inform
 
+- Removed policy/misc/known-devices.bro script and thus
+  known_devices.log will no longer be created.
+
 - The --with-binpac= configure option has changed to mean "path
   to the binpac executable" instead of "path to binpac installation root".
 
@@ -315,7 +315,7 @@ Changed Functionality
 
 - The 'tunnel_parents' field of conn.log is now marked &optional, so, for
   the default configuration of logs, this field will show "-" instead of
-  "(empty)" for connections that lack any tunelling.
+  "(empty)" for connections that lack any tunneling.
 
 - SMB event argument changes:
 
@@ -335,14 +335,13 @@ Removed Functionality
 ---------------------
 
 - We no longer maintain any Bro plugins as part of the Bro
-  distribution. Most of the plugins that used be in aux/plugins have
+  distribution. Most of the plugins that used to be in aux/plugins have
   been moved over to use the Bro Package Manager instead. See
   https://github.com/bro/packages for a list of Bro packages currently
   available.
 
 - BroControl: The option 'IPv6Comm' and 'ZoneID' options are no longer
-  available (though Broker should be able to handle IPv6
+  available (though Broker should be able to handle IPv6 automatically).
-  automatically).
 
 Deprecated Functionality
 ------------------------

VERSION

@@ -1 +1 @@
-2.5-711
+2.5-719

aux/broker

@@ -1 +1 @@
-Subproject commit 6d5b3c785c3c1e517c4c7596beda23e411edeff2
+Subproject commit c0faf99fb0215f410a987b502cfba9f7037906aa

doc/frameworks/broker.rst

@@ -55,7 +55,7 @@ General Porting Tips
   you manually take action to set up the old communication system.
   To aid in porting, such usages will default to raising a fatal error
   unless you explicitly acknowledge that such usages of the old system
-  are ok.  Set the :bro:see:`old_comm_usage_is_ok`` flag in this case.
+  are ok.  Set the :bro:see:`old_comm_usage_is_ok` flag in this case.
 
 - Instead of using e.g. ``Cluster::manager2worker_events`` (and all
   permutations for every node type), what you'd now use is either 
@@ -67,7 +67,7 @@ General Porting Tips
 - Instead of using the ``send_id`` BIF, use :bro:see:`Broker::publish_id`.
 
 - Use :bro:see:`terminate` instead of :bro:see:`terminate_communication`.
-  The later refers to the old communication system and no longer effects
+  The latter refers to the old communication system and no longer affects
   the new Broker-based system.
 
 - For replacing :bro:see:`remote_connection_established` and
@@ -123,7 +123,7 @@ Some general suggestions as to the purpose/utilization of each node type:
   (more on that later), you can partition work or data amongst them in a
   uniform manner.  e.g. you might choose to use proxies as a method of
   sharing non-persistent state or as a "second pass" analysis for any
-  work that you don't want interferring with the workers' capacity to
+  work that you don't want interfering with the workers' capacity to
   keep up with capturing and parsing packets.  Note that the default scripts
   that come with Bro don't utilize proxies themselves, so if you are coming
   from a previous BroControl deployment, you may want to try reducing down
@@ -149,7 +149,7 @@ Data Management/Sharing Strategies
 
 There's maybe no single, best approach or pattern to use when you need a
 Bro script to store or share long-term state and data.  The two
-approaches that were previously used were either using ``&synchronized``
+approaches that were previously used were either using the ``&synchronized``
 attribute on tables/sets or by explicitly sending events to specific
 nodes on which you wanted data to be stored.  The former is no longer
 possible, though there are several new possibilities that the new
@@ -206,16 +206,16 @@ causing data to now be located and updated there.
 
 If the developer of a script expects its workload to be particularly
 intensive, wants to ensure that their operations get exclusive
-access to nodes, or otherwise set containts on the number of nodes within
+access to nodes, or otherwise set constraints on the number of nodes within
 a pool utilized by their script, then the :bro:see:`Cluster::PoolSpec`
-structure will allow them to that while still allowing users of that script
+structure will allow them to do that while still allowing users of that script
 to override the default suggestions made by the original developer.
 
 Broker Framework Examples
 =========================
 
 The broker framework provides basic facilities for connecting Bro instances
-to eachother and exchanging messages, like events or logs.
+to each other and exchanging messages, like events or logs.
 
 See :doc:`/scripts/base/frameworks/broker/main.bro` for an overview
 of the main Broker API.
@@ -320,7 +320,7 @@ the event are not called.
 The second option is to call the :bro:see:`Broker::auto_publish` function where
 you specify a particular event that will be automatically sent to peers
 whenever the event is called locally via the normal event invocation syntax.
-When auto-publishing events, local event handler for the event are called
+When auto-publishing events, local event handlers for the event are called
 in addition to sending the event to any subscribed peers.
 
 .. btest-include:: ${DOC_ROOT}/frameworks/broker/events-connector.bro
@@ -541,7 +541,7 @@ did before.  Instead of using :bro:see:`Broker::publish` we use different
         # though now we have a choice of which proxy to use.  If we
         # want to distribute the work associated with relaying uniformly,
         # we can use a round-robin strategy.  The key used here is simply
-        # used by the cluster framework internally to keep track of the
+        # used by the cluster framework internally to keep track of
         # which node is up next in the round-robin.
         Cluster::relay_rr(Cluster::proxy_pool, "example_key",
                           Cluster::worker_topic, worker_to_workers,

doc/script-reference/operators.rst

@@ -85,6 +85,25 @@ Arithmetic operators
 |                              |             | of elements.                  |
 +------------------------------+-------------+-------------------------------+
 
+Bitwise operators
+-----------------
+
+The bitwise operators work with operands of type :bro:type:`count` or
+``vector of count``, but the bitwise complement operator works with ``count``
+only.
+
++------------------------------+-------------+
+| Name                         | Syntax      |
++==============================+=============+
+| Bitwise AND                  | *a* & *b*   |
++------------------------------+-------------+
+| Bitwise OR                   | *a* | *b*   |
++------------------------------+-------------+
+| Bitwise XOR                  | *a* ^ *b*   |
++------------------------------+-------------+
+| Bitwise complement           | ~ *a*       |
++------------------------------+-------------+
+
 
 Assignment operators
 --------------------
@@ -122,6 +141,73 @@ field name must be in the declaration of the record type.
 +------------------------------+-------------+-------------------------------+
 
 
+Pattern operators
+-----------------
+
+In the table below, *p* is a pattern, and *s* is a string.
+
++------------------------------+-------------+-------------------------------+
+| Name                         | Syntax      | Notes                         |
++==============================+=============+===============================+
+| Exact matching               | *p* == *s*  | Evaluates to a boolean,       |
+|                              |             | indicating if the entire      |
+|                              |             | string exactly matches the    |
+|                              |             | pattern.                      |
++------------------------------+-------------+-------------------------------+
+| Embedded matching            | *p* in *s*  | Evaluates to a boolean,       |
+|                              |             | indicating if pattern is      |
+|                              |             | found somewhere in the string.|
++------------------------------+-------------+-------------------------------+
+| Conjunction                  | *p1* & *p2* | Evaluates to a pattern that   |
+|                              |             | represents matching p1        |
+|                              |             | followed by p2.               |
++------------------------------+-------------+-------------------------------+
+| Disjunction                  | *p1* | *p2* | Evaluates to a pattern that   |
+|                              |             | represents matching p1 or p2. |
++------------------------------+-------------+-------------------------------+
+
+
+Type casting
+------------
+
+The "as" operator performs type casting and the "is" operator checks if a
+type cast is supported or not.  For both operators, the first operand is a
+value and the second operand is the name of a Bro script type (either built-in
+or user-defined).
+
++------------------------------+-------------+-------------------------------+
+| Name                         | Syntax      | Notes                         |
++==============================+=============+===============================+
+| Type cast                    | *v* as *t*  | Cast value "v" into type "t". |
+|                              |             | Evaluates to the value casted |
+|                              |             | to the specified type.        |
+|                              |             | If this is not a supported    |
+|                              |             | cast, then a runtime error is |
+|                              |             | triggered.                    |
++------------------------------+-------------+-------------------------------+
+| Check if a cast is supported | *v* is *t*  | Evaluates to boolean. If true,|
+|                              |             | then "v as t" would succeed.  |
++------------------------------+-------------+-------------------------------+
+
+Only the following kinds of type casts are supported currently:
+
+- Broker values (i.e., :bro:see:`Broker::Data` values returned from
+  functions such as :bro:id:`Broker::data`) can be casted to their
+  corresponding Bro script types.
+- A value of declared type "any" can be casted to its actual underlying type.
+- All values can be casted to their declared types (i.e., this is a no-op).
+
+The function in this example tries to cast a value to a string::
+
+    function example(a: any)
+        {
+        local s: string;
+
+        if ( a is string )
+            s = (a as string);
+        }
+
+
 Other operators
 ---------------
 

doc/script-reference/statements.rst

@@ -571,6 +571,42 @@ Here are the statements that the Bro scripting language supports.
     do not indicate the presence of a `compound statement`_), and that no
     semicolon is needed at the end of a "switch" statement.
 
+    There is an alternative form of the switch statement that supports
+    switching by type rather than value.  This form of the switch statement
+    uses type-based versions of "case":
+
+    - "case type t: ...": Take branch if the value of the switch expression
+      could be casted to type t (where "t" is the name of a Bro script type,
+      either built-in or user-defined).
+
+    - "case type t as x: ...": Same as above, but the casted value is
+      available through ID "x".
+
+    Multiple types can be listed per branch, separated by commas (the "type"
+    keyword must be repeated for each type in the list).
+
+    Example::
+
+        function example(v: any)
+            {
+            switch (v) {
+            case type count as c:
+                    print "It's a count", c;
+                    break;
+
+            case type bool, type addr:
+                    print "It's a bool or address";
+                    break;
+            }
+            }
+
+    Note that a single switch statement switches either by type or by value,
+    but not both.
+
+    Also note that the type-based switch statement will trigger a runtime
+    error if any cast in any "case" is an unsupported cast (see the
+    documentation of the type casting operator "as").
+
 
 .. bro:keyword:: when
 

doc/script-reference/types.rst

@@ -92,7 +92,7 @@ Here is a more detailed description of each type:
     "int".
 
     In addition, "count" types support bitwise operations.  You can use
-    ``&``, ``|``, and ``^`` for bitwise ``and'', ``or'', and ``xor''.  You
+    ``&``, ``|``, and ``^`` for bitwise ``and``, ``or``, and ``xor``.  You
     can also use ``~`` for bitwise (one's) complement.
 
 .. bro:type:: double

scripts/base/frameworks/broker/main.bro

@@ -20,19 +20,19 @@ export {
 	## initially, or if it ever becomes disconnected.
 	const default_connect_retry = 30sec &redef;
 
-	## If false, do not use SSL for network connections. By default, SSL will even
+	## If true, do not use SSL for network connections. By default, SSL will
-	## be used if no certificates / CAs have been configured. In that case
+	## even be used if no certificates / CAs have been configured. In that case
 	## (which is the default) the communication will be encrypted, but not
 	## authenticated.
 	const disable_ssl = F &redef;
 
 	## Path to a file containing concatenated trusted certificates 
-	## in PEM format. If set, Bro will require valid certificates forx
+	## in PEM format. If set, Bro will require valid certificates for
 	## all peers.
 	const ssl_cafile = "" &redef;
 
 	## Path to an OpenSSL-style directory of trusted certificates.
-	## If set, Bro will require valid certificates forx
+	## If set, Bro will require valid certificates for
 	## all peers.
 	const ssl_capath = "" &redef;
 
@@ -96,9 +96,9 @@ export {
 		PEER_INVALID = 3,
 		## Remote peer not listening.
 		PEER_UNAVAILABLE = 4,
-		## An peering request timed out.
+		## A peering request timed out.
 		PEER_TIMEOUT = 5,
-		## Master with given name already exist.
+		## Master with given name already exists.
 		MASTER_EXISTS = 6,
 		## Master with given name does not exist.
 		NO_SUCH_MASTER = 7,
@@ -106,7 +106,7 @@ export {
 		NO_SUCH_KEY = 8,
 		## The store operation timed out.
 		REQUEST_TIMEOUT = 9,
-		## The operation expected a different type than provided
+		## The operation expected a different type than provided.
 		TYPE_CLASH = 10,
 		## The data value cannot be used to carry out the desired operation.
 		INVALID_DATA = 11,
@@ -181,7 +181,7 @@ export {
 	## Listen for remote connections.
 	##
 	## a: an address string on which to accept connections, e.g.
-	##    "127.0.0.1".  An empty string refers to @p INADDR_ANY.
+	##    "127.0.0.1".  An empty string refers to INADDR_ANY.
 	##
 	## p: the TCP port to listen on. The value 0 means that the OS should choose
 	##    the next available free port.
@@ -229,9 +229,13 @@ export {
 	## TODO: We do not have a function yet to terminate a connection.
 	global unpeer: function(a: string, p: port): bool;
 
+	## Get a list of all peer connections.
+	##
 	## Returns: a list of all peer connections.
 	global peers: function(): vector of PeerInfo;
 
+	## Get a unique identifier for the local broker endpoint.
+	##
 	## Returns: a unique identifier for the local broker endpoint.
 	global node_id: function(): string;
 
@@ -268,7 +272,8 @@ export {
 	global unsubscribe: function(topic_prefix: string): bool;
 
 	## Automatically send an event to any interested peers whenever it is
-	## locally dispatched (e.g. using "event my_event(...);" in a script).
+	## locally dispatched. (For example, using "event my_event(...);" in a
+	## script.)
 	##
 	## topic: a topic string associated with the event message.
 	##        Peers advertise interest by registering a subscription to some

scripts/base/frameworks/broker/store.bro

@@ -59,7 +59,7 @@ export {
 	## Options to tune the RocksDB storage backend.
 	type RocksDBOptions: record {
 		## File system path of the database.
-		## If left empty, will be derived from the name of the store.
+		## If left empty, will be derived from the name of the store,
 		## and use the '.rocksdb' file suffix.
 		path: string &default = "";
 	};
@@ -125,9 +125,13 @@ export {
 	##          longer be used for data store operations.
 	global close: function(h: opaque of Broker::Store): bool;
 
-	## Returns: whether a store is closed or not.
+	## Check if a store is closed or not.
+	##
+	## Returns: true if the store is closed.
 	global is_closed: function(h: opaque of Broker::Store): bool;
 
+	## Get the name of a store.
+	##
 	## Returns: the name of the store.
 	global store_name: function(h: opaque of Broker::Store): string;
 

scripts/base/frameworks/cluster/pools.bro

@@ -1,5 +1,5 @@
 ##! Defines an interface for managing pools of cluster nodes.  Pools are
-##! are useful way to distribute work or data among nodes within a cluster.
+##! a useful way to distribute work or data among nodes within a cluster.
 
 @load ./main
 @load base/utils/hash_hrw
@@ -7,7 +7,7 @@
 module Cluster;
 
 export {
-	## Store state of a cluster within within the context of a work pool.
+	## Store state of a cluster within the context of a work pool.
 	type PoolNode: record {
 		## The node name (e.g. "manager").
 		name: string;

src/broker/messaging.bif

@@ -304,7 +304,7 @@ type Cluster::Pool: record;
 ## Publishes an event to a node within a pool according to Round-Robin
 ## distribution strategy.
 ##
-## pool: the pool of nodes that are eligible to receive the revent
+## pool: the pool of nodes that are eligible to receive the event.
 ##
 ## key: an arbitrary string to identify the purpose for which you're
 ##      distributing the event.  e.g. consider using namespacing of your
@@ -350,7 +350,7 @@ function Cluster::publish_rr%(pool: Pool, key: string, ...%): bool
 ## node, then automatically forwards it to its peers with a different topic.
 ## The event is relayed at most a single hop.
 ##
-## pool: the pool of nodes that are eligible to receive the event
+## pool: the pool of nodes that are eligible to receive the event.
 ##
 ## key: an arbitrary string to identify the purpose for which you're
 ##      distributing the event.  e.g. consider using namespacing of your
@@ -423,7 +423,7 @@ function Cluster::relay_rr%(pool: Pool, key: any, ...%): bool
 ## Publishes an event to a node within a pool according to Rendezvous
 ## (Highest Random Weight) hashing strategy.
 ##
-## pool: the pool of nodes that are eligible to receive the revent
+## pool: the pool of nodes that are eligible to receive the event.
 ##
 ## key: data used for input to the hashing function that will uniformly
 ##      distribute keys among available nodes.
@@ -467,7 +467,7 @@ function Cluster::publish_hrw%(pool: Pool, key: any, ...%): bool
 ## strategy.  The receiving nodes then automatically forwards it to its peers
 ## with a different topic.  The event is relayed at most a single hop.
 ##
-## pool: the pool of nodes that are eligible to receive the revent
+## pool: the pool of nodes that are eligible to receive the event.
 ##
 ## key: data used for input to the hashing function that will uniformly
 ##      distribute keys among available nodes.