Mirror/zeek
1
0
Fork 0
mirror of https://github.com/zeek/zeek.git synced 2025-10-02 14:48:21 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 2

Merge remote-tracking branch 'origin/topic/dnthayer/ticket1947'

Browse source 
* origin/topic/dnthayer/ticket1947:
  Fix some typos and formatting in NEWS
  Add pattern operators to the documentation of operators
  Fix minor typos in broker reference documentation
  Fix a broken link and some typos in broker documentation
  Fix reST formatting in documentation of "count" type
  Add documentation for some new Bro features
This commit is contained in:
Jon Siwek 2018-06-27 19:55:54 -05:00
commit e2d5ca5f95
12 changed files with 179 additions and 41 deletions
Download patch file Download diff file Expand all files Collapse all files

8
CHANGES
View file

@ -1,4 +1,12 @@
2.5-719 | 2018-06-27 20:02:52 -0500
* Fix some typos and formatting in NEWS and other documentation
(Daniel Thayer)
* Add documentation for type-based switch statment, as/is operators,
bitwise operators, and pattern operators (Daniel Thayer)
2.5-711 | 2018-06-27 19:11:58 -0500
* Prevent double-wrapping Broker::Data in published event args (Corelight)

19
NEWS
View file

@ -18,7 +18,7 @@ New Functionality
supersedes the old "communication" framework, which is now
deprecated. The "cluster" and "control" frameworks have been ported
to Broker; same for BroControl. For more about the new Broker
framework, see doc/frameworks/broker.rst (there's also guide there
framework, see doc/frameworks/broker.rst (there's also a guide there
for porting existing Bro scripts to Broker). For more about Broker
itself, including its API for external applications, see
aux/broker/doc.
@ -258,11 +258,11 @@ New Functionality
Changed Functionality
---------------------
- All communication is now handled through Broker, requiring changes
- All communication is now handled through Broker, requiring changes
to existing scripts to port them over to the new API. The Broker
framework documentation comes with a porting guide.
- The DHCP analyzer and its script-layer interface have been rewritten.
- The DHCP analyzer and its script-layer interface have been rewritten.
- Supports more DHCP options than before.
@ -272,9 +272,6 @@ Changed Functionality
- Removed the policy/protocols/dhcp/known-devices-and-hostnames.bro
script since it's generally less relevant now with the updated log.
- Removed policy/misc/known-devices.bro script and thus
known_devices.log will no longer be created.
- Removed the base/protocols/dhcp/utils.bro script and thus the
'reverse_ip' function.
@ -290,6 +287,9 @@ Changed Functionality
- dhcp_release
- dhcp_inform
- Removed policy/misc/known-devices.bro script and thus
known_devices.log will no longer be created.
- The --with-binpac= configure option has changed to mean "path
to the binpac executable" instead of "path to binpac installation root".
@ -315,7 +315,7 @@ Changed Functionality
- The 'tunnel_parents' field of conn.log is now marked &optional, so, for
the default configuration of logs, this field will show "-" instead of
"(empty)" for connections that lack any tunelling.
"(empty)" for connections that lack any tunneling.
- SMB event argument changes:
@ -335,14 +335,13 @@ Removed Functionality
---------------------
- We no longer maintain any Bro plugins as part of the Bro
distribution. Most of the plugins that used be in aux/plugins have
distribution. Most of the plugins that used to be in aux/plugins have
been moved over to use the Bro Package Manager instead. See
https://github.com/bro/packages for a list of Bro packages currently
available.
- BroControl: The option 'IPv6Comm' and 'ZoneID' options are no longer
available (though Broker should be able to handle IPv6
automatically).
available (though Broker should be able to handle IPv6 automatically).
Deprecated Functionality
------------------------

2
VERSION
View file

@ -1 +1 @@
2.5-711
2.5-719

2
aux/broker

@ -1 +1 @@
Subproject commit 6d5b3c785c3c1e517c4c7596beda23e411edeff2
Subproject commit c0faf99fb0215f410a987b502cfba9f7037906aa

18
doc/frameworks/broker.rst
View file

@ -55,7 +55,7 @@ General Porting Tips
you manually take action to set up the old communication system.
To aid in porting, such usages will default to raising a fatal error
unless you explicitly acknowledge that such usages of the old system
are ok. Set the :bro:see:`old_comm_usage_is_ok`` flag in this case.
are ok. Set the :bro:see:`old_comm_usage_is_ok` flag in this case.
- Instead of using e.g. ``Cluster::manager2worker_events`` (and all
permutations for every node type), what you'd now use is either
@ -67,7 +67,7 @@ General Porting Tips
- Instead of using the ``send_id`` BIF, use :bro:see:`Broker::publish_id`.
- Use :bro:see:`terminate` instead of :bro:see:`terminate_communication`.
The later refers to the old communication system and no longer effects
The latter refers to the old communication system and no longer affects
the new Broker-based system.
- For replacing :bro:see:`remote_connection_established` and
@ -123,7 +123,7 @@ Some general suggestions as to the purpose/utilization of each node type:
(more on that later), you can partition work or data amongst them in a
uniform manner. e.g. you might choose to use proxies as a method of
sharing non-persistent state or as a "second pass" analysis for any
work that you don't want interferring with the workers' capacity to
work that you don't want interfering with the workers' capacity to
keep up with capturing and parsing packets. Note that the default scripts
that come with Bro don't utilize proxies themselves, so if you are coming
from a previous BroControl deployment, you may want to try reducing down
@ -149,7 +149,7 @@ Data Management/Sharing Strategies
There's maybe no single, best approach or pattern to use when you need a
Bro script to store or share long-term state and data. The two
approaches that were previously used were either using ``&synchronized``
approaches that were previously used were either using the ``&synchronized``
attribute on tables/sets or by explicitly sending events to specific
nodes on which you wanted data to be stored. The former is no longer
possible, though there are several new possibilities that the new
@ -206,16 +206,16 @@ causing data to now be located and updated there.
If the developer of a script expects its workload to be particularly
intensive, wants to ensure that their operations get exclusive
access to nodes, or otherwise set containts on the number of nodes within
access to nodes, or otherwise set constraints on the number of nodes within
a pool utilized by their script, then the :bro:see:`Cluster::PoolSpec`
structure will allow them to that while still allowing users of that script
structure will allow them to do that while still allowing users of that script
to override the default suggestions made by the original developer.
Broker Framework Examples
=========================
The broker framework provides basic facilities for connecting Bro instances
to eachother and exchanging messages, like events or logs.
to each other and exchanging messages, like events or logs.
See :doc:`/scripts/base/frameworks/broker/main.bro` for an overview
of the main Broker API.
@ -320,7 +320,7 @@ the event are not called.
The second option is to call the :bro:see:`Broker::auto_publish` function where
you specify a particular event that will be automatically sent to peers
whenever the event is called locally via the normal event invocation syntax.
When auto-publishing events, local event handler for the event are called
When auto-publishing events, local event handlers for the event are called
in addition to sending the event to any subscribed peers.
.. btest-include:: ${DOC_ROOT}/frameworks/broker/events-connector.bro
@ -541,7 +541,7 @@ did before. Instead of using :bro:see:`Broker::publish` we use different
# though now we have a choice of which proxy to use. If we
# want to distribute the work associated with relaying uniformly,
# we can use a round-robin strategy. The key used here is simply
# used by the cluster framework internally to keep track of the
# used by the cluster framework internally to keep track of
# which node is up next in the round-robin.
Cluster::relay_rr(Cluster::proxy_pool, "example_key",
Cluster::worker_topic, worker_to_workers,

86
doc/script-reference/operators.rst
View file

@ -85,6 +85,25 @@ Arithmetic operators
| | | of elements. |
+------------------------------+-------------+-------------------------------+
Bitwise operators
-----------------
The bitwise operators work with operands of type :bro:type:`count` or
``vector of count``, but the bitwise complement operator works with ``count``
only.
+------------------------------+-------------+
| Name | Syntax |
+==============================+=============+
| Bitwise AND | *a* & *b* |
+------------------------------+-------------+
| Bitwise OR | *a* | *b* |
+------------------------------+-------------+
| Bitwise XOR | *a* ^ *b* |
+------------------------------+-------------+
| Bitwise complement | ~ *a* |
+------------------------------+-------------+
Assignment operators
--------------------
@ -122,6 +141,73 @@ field name must be in the declaration of the record type.
+------------------------------+-------------+-------------------------------+
Pattern operators
-----------------
In the table below, *p* is a pattern, and *s* is a string.
+------------------------------+-------------+-------------------------------+
| Name | Syntax | Notes |
+==============================+=============+===============================+
| Exact matching | *p* == *s* | Evaluates to a boolean, |
| | | indicating if the entire |
| | | string exactly matches the |
| | | pattern. |
+------------------------------+-------------+-------------------------------+
| Embedded matching | *p* in *s* | Evaluates to a boolean, |
| | | indicating if pattern is |
| | | found somewhere in the string.|
+------------------------------+-------------+-------------------------------+
| Conjunction | *p1* & *p2* | Evaluates to a pattern that |
| | | represents matching p1 |
| | | followed by p2. |
+------------------------------+-------------+-------------------------------+
| Disjunction | *p1* | *p2* | Evaluates to a pattern that |
| | | represents matching p1 or p2. |
+------------------------------+-------------+-------------------------------+
Type casting
------------
The "as" operator performs type casting and the "is" operator checks if a
type cast is supported or not. For both operators, the first operand is a
value and the second operand is the name of a Bro script type (either built-in
or user-defined).
+------------------------------+-------------+-------------------------------+
| Name | Syntax | Notes |
+==============================+=============+===============================+
| Type cast | *v* as *t* | Cast value "v" into type "t". |
| | | Evaluates to the value casted |
| | | to the specified type. |
| | | If this is not a supported |
| | | cast, then a runtime error is |
| | | triggered. |
+------------------------------+-------------+-------------------------------+
| Check if a cast is supported | *v* is *t* | Evaluates to boolean. If true,|
| | | then "v as t" would succeed. |
+------------------------------+-------------+-------------------------------+
Only the following kinds of type casts are supported currently:
- Broker values (i.e., :bro:see:`Broker::Data` values returned from
functions such as :bro:id:`Broker::data`) can be casted to their
corresponding Bro script types.
- A value of declared type "any" can be casted to its actual underlying type.
- All values can be casted to their declared types (i.e., this is a no-op).
The function in this example tries to cast a value to a string::
function example(a: any)
{
local s: string;
if ( a is string )
s = (a as string);
}
Other operators
---------------

36
doc/script-reference/statements.rst
View file

@ -571,6 +571,42 @@ Here are the statements that the Bro scripting language supports.
do not indicate the presence of a `compound statement`_), and that no
semicolon is needed at the end of a "switch" statement.
There is an alternative form of the switch statement that supports
switching by type rather than value. This form of the switch statement
uses type-based versions of "case":
- "case type t: ...": Take branch if the value of the switch expression
could be casted to type t (where "t" is the name of a Bro script type,
either built-in or user-defined).
- "case type t as x: ...": Same as above, but the casted value is
available through ID "x".
Multiple types can be listed per branch, separated by commas (the "type"
keyword must be repeated for each type in the list).
Example::
function example(v: any)
{
switch (v) {
case type count as c:
print "It's a count", c;
break;
case type bool, type addr:
print "It's a bool or address";
break;
}
}
Note that a single switch statement switches either by type or by value,
but not both.
Also note that the type-based switch statement will trigger a runtime
error if any cast in any "case" is an unsupported cast (see the
documentation of the type casting operator "as").
.. bro:keyword:: when

2
doc/script-reference/types.rst
View file

@ -92,7 +92,7 @@ Here is a more detailed description of each type:
"int".
In addition, "count" types support bitwise operations. You can use
``&``, ``|``, and ``^`` for bitwise ``and'', ``or'', and ``xor''. You
``&``, ``|``, and ``^`` for bitwise ``and``, ``or``, and ``xor``. You
can also use ``~`` for bitwise (one's) complement.
.. bro:type:: double

23
scripts/base/frameworks/broker/main.bro
View file

@ -20,19 +20,19 @@ export {
## initially, or if it ever becomes disconnected.
const default_connect_retry = 30sec &redef;
## If false, do not use SSL for network connections. By default, SSL will even
## be used if no certificates / CAs have been configured. In that case
## If true, do not use SSL for network connections. By default, SSL will
## even be used if no certificates / CAs have been configured. In that case
## (which is the default) the communication will be encrypted, but not
## authenticated.
const disable_ssl = F &redef;
## Path to a file containing concatenated trusted certificates
## in PEM format. If set, Bro will require valid certificates forx
## in PEM format. If set, Bro will require valid certificates for
## all peers.
const ssl_cafile = "" &redef;
## Path to an OpenSSL-style directory of trusted certificates.
## If set, Bro will require valid certificates forx
## If set, Bro will require valid certificates for
## all peers.
const ssl_capath = "" &redef;
@ -96,9 +96,9 @@ export {
PEER_INVALID = 3,
## Remote peer not listening.
PEER_UNAVAILABLE = 4,
## An peering request timed out.
## A peering request timed out.
PEER_TIMEOUT = 5,
## Master with given name already exist.
## Master with given name already exists.
MASTER_EXISTS = 6,
## Master with given name does not exist.
NO_SUCH_MASTER = 7,
@ -106,7 +106,7 @@ export {
NO_SUCH_KEY = 8,
## The store operation timed out.
REQUEST_TIMEOUT = 9,
## The operation expected a different type than provided
## The operation expected a different type than provided.
TYPE_CLASH = 10,
## The data value cannot be used to carry out the desired operation.
INVALID_DATA = 11,
@ -181,7 +181,7 @@ export {
## Listen for remote connections.
##
## a: an address string on which to accept connections, e.g.
## "127.0.0.1". An empty string refers to @p INADDR_ANY.
## "127.0.0.1". An empty string refers to INADDR_ANY.
##
## p: the TCP port to listen on. The value 0 means that the OS should choose
## the next available free port.
@ -229,9 +229,13 @@ export {
## TODO: We do not have a function yet to terminate a connection.
global unpeer: function(a: string, p: port): bool;
## Get a list of all peer connections.
##
## Returns: a list of all peer connections.
global peers: function(): vector of PeerInfo;
## Get a unique identifier for the local broker endpoint.
##
## Returns: a unique identifier for the local broker endpoint.
global node_id: function(): string;
@ -268,7 +272,8 @@ export {
global unsubscribe: function(topic_prefix: string): bool;
## Automatically send an event to any interested peers whenever it is
## locally dispatched (e.g. using "event my_event(...);" in a script).
## locally dispatched. (For example, using "event my_event(...);" in a
## script.)
##
## topic: a topic string associated with the event message.
## Peers advertise interest by registering a subscription to some

8
scripts/base/frameworks/broker/store.bro
View file

@ -59,7 +59,7 @@ export {
## Options to tune the RocksDB storage backend.
type RocksDBOptions: record {
## File system path of the database.
## If left empty, will be derived from the name of the store.
## If left empty, will be derived from the name of the store,
## and use the '.rocksdb' file suffix.
path: string &default = "";
};
@ -125,9 +125,13 @@ export {
## longer be used for data store operations.
global close: function(h: opaque of Broker::Store): bool;
## Returns: whether a store is closed or not.
## Check if a store is closed or not.
##
## Returns: true if the store is closed.
global is_closed: function(h: opaque of Broker::Store): bool;
## Get the name of a store.
##
## Returns: the name of the store.
global store_name: function(h: opaque of Broker::Store): string;

4
scripts/base/frameworks/cluster/pools.bro
View file

@ -1,5 +1,5 @@
##! Defines an interface for managing pools of cluster nodes. Pools are
##! are useful way to distribute work or data among nodes within a cluster.
##! a useful way to distribute work or data among nodes within a cluster.
@load ./main
@load base/utils/hash_hrw
@ -7,7 +7,7 @@
module Cluster;
export {
## Store state of a cluster within within the context of a work pool.
## Store state of a cluster within the context of a work pool.
type PoolNode: record {
## The node name (e.g. "manager").
name: string;

8
src/broker/messaging.bif
View file

@ -304,7 +304,7 @@ type Cluster::Pool: record;
## Publishes an event to a node within a pool according to Round-Robin
## distribution strategy.
##
## pool: the pool of nodes that are eligible to receive the revent
## pool: the pool of nodes that are eligible to receive the event.
##
## key: an arbitrary string to identify the purpose for which you're
## distributing the event. e.g. consider using namespacing of your
@ -350,7 +350,7 @@ function Cluster::publish_rr%(pool: Pool, key: string, ...%): bool
## node, then automatically forwards it to its peers with a different topic.
## The event is relayed at most a single hop.
##
## pool: the pool of nodes that are eligible to receive the event
## pool: the pool of nodes that are eligible to receive the event.
##
## key: an arbitrary string to identify the purpose for which you're
## distributing the event. e.g. consider using namespacing of your
@ -423,7 +423,7 @@ function Cluster::relay_rr%(pool: Pool, key: any, ...%): bool
## Publishes an event to a node within a pool according to Rendezvous
## (Highest Random Weight) hashing strategy.
##
## pool: the pool of nodes that are eligible to receive the revent
## pool: the pool of nodes that are eligible to receive the event.
##
## key: data used for input to the hashing function that will uniformly
## distribute keys among available nodes.
@ -467,7 +467,7 @@ function Cluster::publish_hrw%(pool: Pool, key: any, ...%): bool
## strategy. The receiving nodes then automatically forwards it to its peers
## with a different topic. The event is relayed at most a single hop.
##
## pool: the pool of nodes that are eligible to receive the revent
## pool: the pool of nodes that are eligible to receive the event.
##
## key: data used for input to the hashing function that will uniformly
## distribute keys among available nodes.