Commit graph

35 commits

Author SHA1 Message Date
Christian Kreibich
46a11ec37d Add Cluster::nodeid_to_node() helper function
This translates backend-specific node identifiers (like Broker IDs) to
cluster nodes and their names, if available.
2024-12-06 15:18:05 -08:00
Christian Kreibich
e81856a4af No need to namespace Cluster:: functions in their own namespace 2024-12-06 15:18:05 -08:00
Arne Welzel
ef04a199c8 cluster: Add Cluster scoped bifs
... and a broker based test using Cluster::publish() and
Cluster::subscribe().
2024-11-26 12:58:23 +01:00
Arne Welzel
fb23a06f6f cluster/Backend: Interface for cluster backends 2024-11-22 10:43:50 +01:00
Arne Welzel
6bb7b9d726 scripts/base/cluster: Move active node management into node_down()
With the idea of an alternative cluster backend, we should
not maintain Cluster state within low-level Broker events.
2024-09-27 15:32:09 +02:00
Tim Wojtulewicz
4e9d843cec Remove deprecated Cluster::Node::interface field 2024-08-07 11:58:22 -07:00
Tim Wojtulewicz
a716903f3a Remove deprecated time machine settings 2024-08-07 11:58:21 -07:00
Tim Wojtulewicz
e93e4cc26d Add a services.json endpoint for Prometheus service discovery 2024-05-31 13:30:31 -07:00
Arne Welzel
cd24acdfc8 time machine: Mark leftovers for removal in v7.1
I suspect we could just drop these directly, but lets follow the
deprecation cycle.
2023-11-07 16:06:16 +01:00
Arne Welzel
d88b147ac9 cluster: Deprecate the Cluster::Node$interface field
This field isn't required by a worker and it's certainly not used by a
worker to listen on that specific interface. It also isn't required to
be set consistently and its use in-tree limited to the old load-balancing
script.

There's a bif called packet_source() which on a worker will provide
information about the actually used packet source.

Relates to zeek/zeek#2877.
2023-11-07 16:06:16 +01:00
Tim Wojtulewicz
4229af6820 Remove deprecations tagged for v6.1 2023-06-14 10:07:22 -07:00
Arne Welzel
eef7acc1e9 cluster/main: Remove extra @if ( Cluster::is_enabled() )
These have been discussed in the context of "@if &analyze" [1] and
am much in favor for not disabling/removing ~100 lines (more than
fits on a single terminal) out from the middle of a file. There's no
performance impact for having these handlers enabled unconditionally.
Also, any future work on "@if &analyze" will look at them again which
we could also skip.

This also reverts back to the behavior where the Cluster::LOG stream
is created even in non cluster setups like in previous Zeek versions.
As long as no one writes to it there's essentially no difference. If
someone does write to Cluster::LOG, I'd argue not black holing these
messages is better. Schema generators using Log::active_streams will
continue to discover Cluster::LOG even if they run in non-cluster
mode.

https://github.com/zeek/zeek/pull/3062#discussion_r1200498905
2023-06-06 15:20:10 +02:00
Tim Wojtulewicz
5a3abbe364 Revert "Merge remote-tracking branch 'origin/topic/vern/at-if-analyze'"
This reverts commit 4e797ddbbc, reversing
changes made to 3ac28ba5a2.
2023-05-31 09:20:33 +02:00
Vern Paxson
890010915a change base scripts to use run-time if's or @if ... &analyze 2023-05-19 13:26:27 -07:00
Arne Welzel
c2a07476cc Merge remote-tracking branch 'jgras/topic/jgras/cluster-active-node-count-fix'
* jgras/topic/jgras/cluster-active-node-count-fix:
  Fix get_active_node_count for node types not present.

Changed over to explicit existence check instead to avoid the set()
creation upon missed lookups.
2023-05-17 10:37:00 +02:00
Jan Grashoefer
e4f654c14c Fix get_active_node_count for node types not present. 2023-05-16 17:47:50 +02:00
Jan Grashoefer
88c86cc7d4 Add hook into cluster connection setup. 2023-04-21 19:04:52 +02:00
Jan Grashoefer
c7626d797f Add broadcast_topics set.
This set contains the topics to reach all cluster nodes. Due to broker's
forwarding mechanism, we cannot define a single broadcast topic, as it
would create routing loops.
2023-04-21 19:04:52 +02:00
Jan Grashoefer
3db8bb4a44 Generalize Cluster::worker_count. 2023-04-21 19:04:39 +02:00
Robin Sommer
3a9320dab3
Merge remote-tracking branch 'origin/topic/awelzel/2528-cluster-layout-content-warning'
* origin/topic/awelzel/2528-cluster-layout-content-warning:
  cluster: Add warning about cluster-layout.zeek content
2022-11-07 11:28:57 +01:00
Arne Welzel
28336709b8 cluster: Add warning about cluster-layout.zeek content
Relates to #2528, #991.
2022-11-03 14:02:43 +01:00
Jon Siwek
e82824b638 Fix various broken links in script documentation 2021-01-28 17:46:58 -08:00
Jon Siwek
321a027d07 Remove unusable/broken RocksDB code and options
The Broker RockSDB data store backend was previously unusable
and broken, so all code and options related to it are now removed.
2021-01-11 11:12:59 -08:00
Seth Hall
cd330c801d
Apply suggestions from code review
Co-authored-by: Jon Siwek <jsiwek@corelight.com>
2020-10-13 16:48:15 -04:00
Seth Hall
e78386d6e5
Update scripts/base/frameworks/cluster/main.zeek
Co-authored-by: Jon Siwek <jsiwek@corelight.com>
2020-10-13 16:46:26 -04:00
Seth Hall
cf8671d078 Make defining a port number for hosts in a cluster that only connect outbound optional 2020-10-12 10:46:28 -04:00
Christian Kreibich
1bd658da8f Support for log filter policy hooks
This adds a "policy" hook into the logging framework's streams and
filters to replace the existing log filter predicates. The hook
signature is as follows:

    hook(rec: any, id: Log::ID, filter: Log::Filter);

The logging manager invokes hooks on each log record. Hooks can veto
log records via a break, and modify them if necessary. Log filters
inherit the stream-level hook, but can override or remove the hook as
needed.

The distribution's existing log streams now come with pre-defined
hooks that users can add handlers to. Their name is standardized as
"log_policy" by convention, with additional suffixes when a module
provides multiple streams. The following adds a handler to the Conn
module's default log policy hook:

    hook Conn::log_policy(rec: Conn::Info, id: Log::ID, filter: Log::Filter)
            {
            if ( some_veto_reason(rec) )
                break;
            }

By default, this handler will get invoked for any log filter
associated with the Conn::LOG stream.

The existing predicates are deprecated for removal in 4.1 but continue
to work.
2020-09-30 12:32:45 -07:00
Jon Siwek
ce9183a2ed Fix Broker topics used to uniquely identify cluster nodes
Node-specific topic prefix subscriptions/publications now add a trailing
slash like "zeek/cluster/node/<name>/".  Without the trailing slash,
messages attempting to target "proxy-10" may also be sent to "proxy-1"
since subscription matching is prefix-based.
2020-04-10 14:36:00 -07:00
Jon Siwek
29f386e388 Implement minimal supervised cluster configuration
More aspects of the cluster configuration to get fleshed out later,
but a basic cluster like one would use for a live deployment
can now be instantiated and run under supervision.  The new
clusterized-pcap-processing supervisor mode is also not done yet.
2019-10-23 17:37:53 -07:00
Jon Siwek
1ce0fcce49 GH-387: update Broker topic names to use "zeek/" prefix 2019-05-29 15:56:37 -07:00
Daniel Thayer
1a74516db1 Rename all BRO-prefixed environment variables
For backward compatibility when reading values, we first check
the ZEEK-prefixed value, and if not set, then check the corresponding
BRO-prefixed value.
2019-05-22 00:12:31 -05:00
Daniel Thayer
be182aac83 More bro-to-zeek renaming in scripts and other files 2019-05-16 02:36:41 -05:00
Jon Siwek
aebcb1415d GH-234: rename Broxygen to Zeexygen along with roles/directives
* All "Broxygen" usages have been replaced in
  code, documentation, filenames, etc.

* Sphinx roles/directives like ":bro:see" are now ":zeek:see"

* The "--broxygen" command-line option is now "--zeexygen"
2019-04-22 19:45:50 -07:00
Jon Siwek
a994be9eeb Merge remote-tracking branch 'origin/topic/seth/zeek_init'
* origin/topic/seth/zeek_init:
  Some more testing fixes.
  Update docs and tests for bro_(init|done) -> zeek_(init|done)
  Implement the zeek_init handler.
2019-04-19 11:24:29 -07:00
Daniel Thayer
18bd74454b Rename all scripts to have ".zeek" file extension 2019-04-11 21:12:40 -05:00
Renamed from scripts/base/frameworks/cluster/main.bro (Browse further)