From fef351b9c146cc366603c6739991c84a97723be5 Mon Sep 17 00:00:00 2001 From: Daniel Thayer Date: Tue, 26 Jun 2018 14:38:24 -0500 Subject: [PATCH 1/6] Add documentation for some new Bro features Add documentation for the type-based "switch" statement, the "as" operator, the "is" operator, and bitwise operators. --- doc/script-reference/operators.rst | 61 +++++++++++++++++++++++++++++ doc/script-reference/statements.rst | 36 +++++++++++++++++ 2 files changed, 97 insertions(+) diff --git a/doc/script-reference/operators.rst b/doc/script-reference/operators.rst index 9442102b52..e9683f3e97 100644 --- a/doc/script-reference/operators.rst +++ b/doc/script-reference/operators.rst @@ -85,6 +85,25 @@ Arithmetic operators | | | of elements. | +------------------------------+-------------+-------------------------------+ +Bitwise operators +----------------- + +The bitwise operators work with operands of type :bro:type:`count` or +``vector of count``, but the bitwise complement operator works with ``count`` +only. + ++------------------------------+-------------+ +| Name | Syntax | ++==============================+=============+ +| Bitwise AND | *a* & *b* | ++------------------------------+-------------+ +| Bitwise OR | *a* | *b* | ++------------------------------+-------------+ +| Bitwise XOR | *a* ^ *b* | ++------------------------------+-------------+ +| Bitwise complement | ~ *a* | ++------------------------------+-------------+ + Assignment operators -------------------- @@ -122,6 +141,48 @@ field name must be in the declaration of the record type. +------------------------------+-------------+-------------------------------+ +Type casting +------------ + +The "as" operator performs type casting and the "is" operator checks if a +type cast is supported or not. For both operators, the first operand is a +value and the second operand is the name of a Bro script type (either built-in +or user-defined). + ++------------------------------+-------------+-------------------------------+ +| Name | Syntax | Notes | ++==============================+=============+===============================+ +| Type cast | *v* as *t* | Cast value "v" into type "t". | +| | | Evaluates to the value casted | +| | | to the specified type. | +| | | If this is not a supported | +| | | cast, then a runtime error is | +| | | triggered. | ++------------------------------+-------------+-------------------------------+ +| Check if a cast is supported | *v* is *t* | Evaluates to boolean. If true,| +| | | then "v as t" would succeed. | ++------------------------------+-------------+-------------------------------+ + +Only the following kinds of type casts are supported currently: + +- Broker values (i.e., values returned from functions such as + :bro:id:`Broker::data`) can be casted to their corresponding Bro script + types. +- A value of declared type "any" can be casted to its actual underlying type. +- All values can be casted to their declared types (i.e., this is a no-op). + +The function in this example tries to cast a value to a string:: + + function example(a: any) + { + local s: string; + + if ( a is string ) + s = (a as string); + + } + + Other operators --------------- diff --git a/doc/script-reference/statements.rst b/doc/script-reference/statements.rst index 963d4dc7a2..9e061d4df7 100644 --- a/doc/script-reference/statements.rst +++ b/doc/script-reference/statements.rst @@ -571,6 +571,42 @@ Here are the statements that the Bro scripting language supports. do not indicate the presence of a `compound statement`_), and that no semicolon is needed at the end of a "switch" statement. + There is an alternative form of the switch statement that supports + switching by type rather than value. This form of the switch statement + uses type-based versions of "case": + + - "case type t: ...": Take branch if the value of the switch expression + could be casted to type t (where "t" is the name of a Bro script type, + either built-in or user-defined). + + - "case type t as x: ...": Same as above, but the casted value is + available through ID "x". + + Multiple types can be listed per branch, separated by commas (the "type" + keyword must be repeated for each type in the list). + + Example:: + + function example(v: any) + { + switch (v) { + case type count as c: + print "It's a count", c; + break; + + case type bool, type addr: + print "It's a bool or address"; + break; + } + } + + Note that a single switch statement switches either by type or by value, + but not both. + + Also note that the type-based switch statement will trigger a runtime + error if any cast in any "case" is an unsupported cast (see the + documentation of the type casting operator "as"). + .. bro:keyword:: when From 884d3d2abd601e9ba788629939d875cc9de77614 Mon Sep 17 00:00:00 2001 From: Daniel Thayer Date: Tue, 26 Jun 2018 14:43:41 -0500 Subject: [PATCH 2/6] Fix reST formatting in documentation of "count" type --- doc/script-reference/types.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc/script-reference/types.rst b/doc/script-reference/types.rst index 44dcbbdfb8..de5fa689f9 100644 --- a/doc/script-reference/types.rst +++ b/doc/script-reference/types.rst @@ -92,7 +92,7 @@ Here is a more detailed description of each type: "int". In addition, "count" types support bitwise operations. You can use - ``&``, ``|``, and ``^`` for bitwise ``and'', ``or'', and ``xor''. You + ``&``, ``|``, and ``^`` for bitwise ``and``, ``or``, and ``xor``. You can also use ``~`` for bitwise (one's) complement. .. bro:type:: double From 57128af3ab3bfc3f71ccccfb8d0acfb4a2650646 Mon Sep 17 00:00:00 2001 From: Daniel Thayer Date: Tue, 26 Jun 2018 15:34:57 -0500 Subject: [PATCH 3/6] Fix a broken link and some typos in broker documentation --- doc/frameworks/broker.rst | 18 +++++++++--------- 1 file changed, 9 insertions(+), 9 deletions(-) diff --git a/doc/frameworks/broker.rst b/doc/frameworks/broker.rst index 6943a0a698..e050ec6479 100644 --- a/doc/frameworks/broker.rst +++ b/doc/frameworks/broker.rst @@ -55,7 +55,7 @@ General Porting Tips you manually take action to set up the old communication system. To aid in porting, such usages will default to raising a fatal error unless you explicitly acknowledge that such usages of the old system - are ok. Set the :bro:see:`old_comm_usage_is_ok`` flag in this case. + are ok. Set the :bro:see:`old_comm_usage_is_ok` flag in this case. - Instead of using e.g. ``Cluster::manager2worker_events`` (and all permutations for every node type), what you'd now use is either @@ -67,7 +67,7 @@ General Porting Tips - Instead of using the ``send_id`` BIF, use :bro:see:`Broker::publish_id`. - Use :bro:see:`terminate` instead of :bro:see:`terminate_communication`. - The later refers to the old communication system and no longer effects + The latter refers to the old communication system and no longer affects the new Broker-based system. - For replacing :bro:see:`remote_connection_established` and @@ -123,7 +123,7 @@ Some general suggestions as to the purpose/utilization of each node type: (more on that later), you can partition work or data amongst them in a uniform manner. e.g. you might choose to use proxies as a method of sharing non-persistent state or as a "second pass" analysis for any - work that you don't want interferring with the workers' capacity to + work that you don't want interfering with the workers' capacity to keep up with capturing and parsing packets. Note that the default scripts that come with Bro don't utilize proxies themselves, so if you are coming from a previous BroControl deployment, you may want to try reducing down @@ -149,7 +149,7 @@ Data Management/Sharing Strategies There's maybe no single, best approach or pattern to use when you need a Bro script to store or share long-term state and data. The two -approaches that were previously used were either using ``&synchronized`` +approaches that were previously used were either using the ``&synchronized`` attribute on tables/sets or by explicitly sending events to specific nodes on which you wanted data to be stored. The former is no longer possible, though there are several new possibilities that the new @@ -206,16 +206,16 @@ causing data to now be located and updated there. If the developer of a script expects its workload to be particularly intensive, wants to ensure that their operations get exclusive -access to nodes, or otherwise set containts on the number of nodes within +access to nodes, or otherwise set constraints on the number of nodes within a pool utilized by their script, then the :bro:see:`Cluster::PoolSpec` -structure will allow them to that while still allowing users of that script +structure will allow them to do that while still allowing users of that script to override the default suggestions made by the original developer. Broker Framework Examples ========================= The broker framework provides basic facilities for connecting Bro instances -to eachother and exchanging messages, like events or logs. +to each other and exchanging messages, like events or logs. See :doc:`/scripts/base/frameworks/broker/main.bro` for an overview of the main Broker API. @@ -320,7 +320,7 @@ the event are not called. The second option is to call the :bro:see:`Broker::auto_publish` function where you specify a particular event that will be automatically sent to peers whenever the event is called locally via the normal event invocation syntax. -When auto-publishing events, local event handler for the event are called +When auto-publishing events, local event handlers for the event are called in addition to sending the event to any subscribed peers. .. btest-include:: ${DOC_ROOT}/frameworks/broker/events-connector.bro @@ -541,7 +541,7 @@ did before. Instead of using :bro:see:`Broker::publish` we use different # though now we have a choice of which proxy to use. If we # want to distribute the work associated with relaying uniformly, # we can use a round-robin strategy. The key used here is simply - # used by the cluster framework internally to keep track of the + # used by the cluster framework internally to keep track of # which node is up next in the round-robin. Cluster::relay_rr(Cluster::proxy_pool, "example_key", Cluster::worker_topic, worker_to_workers, From ceefb6edafe669d90b5abd03b18962cba6ccb875 Mon Sep 17 00:00:00 2001 From: Daniel Thayer Date: Wed, 27 Jun 2018 13:00:09 -0500 Subject: [PATCH 4/6] Fix minor typos in broker reference documentation --- scripts/base/frameworks/broker/main.bro | 23 ++++++++++++++--------- scripts/base/frameworks/broker/store.bro | 8 ++++++-- scripts/base/frameworks/cluster/pools.bro | 4 ++-- src/broker/messaging.bif | 8 ++++---- 4 files changed, 26 insertions(+), 17 deletions(-) diff --git a/scripts/base/frameworks/broker/main.bro b/scripts/base/frameworks/broker/main.bro index 23a701c3ef..cab205dd85 100644 --- a/scripts/base/frameworks/broker/main.bro +++ b/scripts/base/frameworks/broker/main.bro @@ -20,19 +20,19 @@ export { ## initially, or if it ever becomes disconnected. const default_connect_retry = 30sec &redef; - ## If false, do not use SSL for network connections. By default, SSL will even - ## be used if no certificates / CAs have been configured. In that case + ## If true, do not use SSL for network connections. By default, SSL will + ## even be used if no certificates / CAs have been configured. In that case ## (which is the default) the communication will be encrypted, but not ## authenticated. const disable_ssl = F &redef; ## Path to a file containing concatenated trusted certificates - ## in PEM format. If set, Bro will require valid certificates forx + ## in PEM format. If set, Bro will require valid certificates for ## all peers. const ssl_cafile = "" &redef; ## Path to an OpenSSL-style directory of trusted certificates. - ## If set, Bro will require valid certificates forx + ## If set, Bro will require valid certificates for ## all peers. const ssl_capath = "" &redef; @@ -96,9 +96,9 @@ export { PEER_INVALID = 3, ## Remote peer not listening. PEER_UNAVAILABLE = 4, - ## An peering request timed out. + ## A peering request timed out. PEER_TIMEOUT = 5, - ## Master with given name already exist. + ## Master with given name already exists. MASTER_EXISTS = 6, ## Master with given name does not exist. NO_SUCH_MASTER = 7, @@ -106,7 +106,7 @@ export { NO_SUCH_KEY = 8, ## The store operation timed out. REQUEST_TIMEOUT = 9, - ## The operation expected a different type than provided + ## The operation expected a different type than provided. TYPE_CLASH = 10, ## The data value cannot be used to carry out the desired operation. INVALID_DATA = 11, @@ -181,7 +181,7 @@ export { ## Listen for remote connections. ## ## a: an address string on which to accept connections, e.g. - ## "127.0.0.1". An empty string refers to @p INADDR_ANY. + ## "127.0.0.1". An empty string refers to INADDR_ANY. ## ## p: the TCP port to listen on. The value 0 means that the OS should choose ## the next available free port. @@ -229,9 +229,13 @@ export { ## TODO: We do not have a function yet to terminate a connection. global unpeer: function(a: string, p: port): bool; + ## Get a list of all peer connections. + ## ## Returns: a list of all peer connections. global peers: function(): vector of PeerInfo; + ## Get a unique identifier for the local broker endpoint. + ## ## Returns: a unique identifier for the local broker endpoint. global node_id: function(): string; @@ -268,7 +272,8 @@ export { global unsubscribe: function(topic_prefix: string): bool; ## Automatically send an event to any interested peers whenever it is - ## locally dispatched (e.g. using "event my_event(...);" in a script). + ## locally dispatched. (For example, using "event my_event(...);" in a + ## script.) ## ## topic: a topic string associated with the event message. ## Peers advertise interest by registering a subscription to some diff --git a/scripts/base/frameworks/broker/store.bro b/scripts/base/frameworks/broker/store.bro index 75b7d69b35..2e216afa93 100644 --- a/scripts/base/frameworks/broker/store.bro +++ b/scripts/base/frameworks/broker/store.bro @@ -59,7 +59,7 @@ export { ## Options to tune the RocksDB storage backend. type RocksDBOptions: record { ## File system path of the database. - ## If left empty, will be derived from the name of the store. + ## If left empty, will be derived from the name of the store, ## and use the '.rocksdb' file suffix. path: string &default = ""; }; @@ -125,9 +125,13 @@ export { ## longer be used for data store operations. global close: function(h: opaque of Broker::Store): bool; - ## Returns: whether a store is closed or not. + ## Check if a store is closed or not. + ## + ## Returns: true if the store is closed. global is_closed: function(h: opaque of Broker::Store): bool; + ## Get the name of a store. + ## ## Returns: the name of the store. global store_name: function(h: opaque of Broker::Store): string; diff --git a/scripts/base/frameworks/cluster/pools.bro b/scripts/base/frameworks/cluster/pools.bro index 69b6c7128e..fb45594adc 100644 --- a/scripts/base/frameworks/cluster/pools.bro +++ b/scripts/base/frameworks/cluster/pools.bro @@ -1,5 +1,5 @@ ##! Defines an interface for managing pools of cluster nodes. Pools are -##! are useful way to distribute work or data among nodes within a cluster. +##! a useful way to distribute work or data among nodes within a cluster. @load ./main @load base/utils/hash_hrw @@ -7,7 +7,7 @@ module Cluster; export { - ## Store state of a cluster within within the context of a work pool. + ## Store state of a cluster within the context of a work pool. type PoolNode: record { ## The node name (e.g. "manager"). name: string; diff --git a/src/broker/messaging.bif b/src/broker/messaging.bif index c7b16dba72..bc0d03a629 100644 --- a/src/broker/messaging.bif +++ b/src/broker/messaging.bif @@ -304,7 +304,7 @@ type Cluster::Pool: record; ## Publishes an event to a node within a pool according to Round-Robin ## distribution strategy. ## -## pool: the pool of nodes that are eligible to receive the revent +## pool: the pool of nodes that are eligible to receive the event. ## ## key: an arbitrary string to identify the purpose for which you're ## distributing the event. e.g. consider using namespacing of your @@ -350,7 +350,7 @@ function Cluster::publish_rr%(pool: Pool, key: string, ...%): bool ## node, then automatically forwards it to its peers with a different topic. ## The event is relayed at most a single hop. ## -## pool: the pool of nodes that are eligible to receive the event +## pool: the pool of nodes that are eligible to receive the event. ## ## key: an arbitrary string to identify the purpose for which you're ## distributing the event. e.g. consider using namespacing of your @@ -423,7 +423,7 @@ function Cluster::relay_rr%(pool: Pool, key: any, ...%): bool ## Publishes an event to a node within a pool according to Rendezvous ## (Highest Random Weight) hashing strategy. ## -## pool: the pool of nodes that are eligible to receive the revent +## pool: the pool of nodes that are eligible to receive the event. ## ## key: data used for input to the hashing function that will uniformly ## distribute keys among available nodes. @@ -467,7 +467,7 @@ function Cluster::publish_hrw%(pool: Pool, key: any, ...%): bool ## strategy. The receiving nodes then automatically forwards it to its peers ## with a different topic. The event is relayed at most a single hop. ## -## pool: the pool of nodes that are eligible to receive the revent +## pool: the pool of nodes that are eligible to receive the event. ## ## key: data used for input to the hashing function that will uniformly ## distribute keys among available nodes. From bd74b4525b1cf038331fc77e49e892bcae73c15e Mon Sep 17 00:00:00 2001 From: Daniel Thayer Date: Wed, 27 Jun 2018 14:00:32 -0500 Subject: [PATCH 5/6] Add pattern operators to the documentation of operators --- doc/script-reference/operators.rst | 26 ++++++++++++++++++++++++++ 1 file changed, 26 insertions(+) diff --git a/doc/script-reference/operators.rst b/doc/script-reference/operators.rst index e9683f3e97..b7d6e71b73 100644 --- a/doc/script-reference/operators.rst +++ b/doc/script-reference/operators.rst @@ -141,6 +141,32 @@ field name must be in the declaration of the record type. +------------------------------+-------------+-------------------------------+ +Pattern operators +----------------- + +In the table below, *p* is a pattern, and *s* is a string. + ++------------------------------+-------------+-------------------------------+ +| Name | Syntax | Notes | ++==============================+=============+===============================+ +| Exact matching | *p* == *s* | Evaluates to a boolean, | +| | | indicating if the entire | +| | | string exactly matches the | +| | | pattern. | ++------------------------------+-------------+-------------------------------+ +| Embedded matching | *p* in *s* | Evaluates to a boolean, | +| | | indicating if pattern is | +| | | found somewhere in the string.| ++------------------------------+-------------+-------------------------------+ +| Conjunction | *p1* & *p2* | Evaluates to a pattern that | +| | | represents matching p1 | +| | | followed by p2. | ++------------------------------+-------------+-------------------------------+ +| Disjunction | *p1* | *p2* | Evaluates to a pattern that | +| | | represents matching p1 or p2. | ++------------------------------+-------------+-------------------------------+ + + Type casting ------------ From 8849e214caeb0dbae9fe110a8f13f3e33e665a14 Mon Sep 17 00:00:00 2001 From: Daniel Thayer Date: Wed, 27 Jun 2018 14:33:07 -0500 Subject: [PATCH 6/6] Fix some typos and formatting in NEWS --- NEWS | 23 +++++++++++------------ 1 file changed, 11 insertions(+), 12 deletions(-) diff --git a/NEWS b/NEWS index a4f233a116..b6d88aa82f 100644 --- a/NEWS +++ b/NEWS @@ -18,7 +18,7 @@ New Functionality supersedes the old "communication" framework, which is now deprecated. The "cluster" and "control" frameworks have been ported to Broker; same for BroControl. For more about the new Broker - framework, see doc/frameworks/broker.rst (there's also guide there + framework, see doc/frameworks/broker.rst (there's also a guide there for porting existing Bro scripts to Broker). For more about Broker itself, including its API for external applications, see aux/broker/doc. @@ -258,11 +258,11 @@ New Functionality Changed Functionality --------------------- - - All communication is now handled through Broker, requiring changes - to existing scripts to port them over to the new API. The Broker - framework documentation comes with a porting guide. +- All communication is now handled through Broker, requiring changes + to existing scripts to port them over to the new API. The Broker + framework documentation comes with a porting guide. - - The DHCP analyzer and its script-layer interface have been rewritten. +- The DHCP analyzer and its script-layer interface have been rewritten. - Supports more DHCP options than before. @@ -272,9 +272,6 @@ Changed Functionality - Removed the policy/protocols/dhcp/known-devices-and-hostnames.bro script since it's generally less relevant now with the updated log. - - Removed policy/misc/known-devices.bro script and thus - known_devices.log will no longer be created. - - Removed the base/protocols/dhcp/utils.bro script and thus the 'reverse_ip' function. @@ -290,6 +287,9 @@ Changed Functionality - dhcp_release - dhcp_inform +- Removed policy/misc/known-devices.bro script and thus + known_devices.log will no longer be created. + - The --with-binpac= configure option has changed to mean "path to the binpac executable" instead of "path to binpac installation root". @@ -315,7 +315,7 @@ Changed Functionality - The 'tunnel_parents' field of conn.log is now marked &optional, so, for the default configuration of logs, this field will show "-" instead of - "(empty)" for connections that lack any tunelling. + "(empty)" for connections that lack any tunneling. - SMB event argument changes: @@ -335,14 +335,13 @@ Removed Functionality --------------------- - We no longer maintain any Bro plugins as part of the Bro - distribution. Most of the plugins that used be in aux/plugins have + distribution. Most of the plugins that used to be in aux/plugins have been moved over to use the Bro Package Manager instead. See https://github.com/bro/packages for a list of Bro packages currently available. - BroControl: The option 'IPv6Comm' and 'ZoneID' options are no longer - available (though Broker should be able to handle IPv6 - automatically). + available (though Broker should be able to handle IPv6 automatically). Deprecated Functionality ------------------------