Mirror/zeek
1
0
Fork 0
mirror of https://github.com/zeek/zeek.git synced 2025-10-17 05:58:20 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 2

Merge branch 'topic/timw/prometheus-cpp-3'

Browse source 
* topic/timw/prometheus-cpp-3: (35 commits)
  Update docs submodule [nomail] [skip ci]
  Add type aliases for instrument and family shared_ptrs
  Update NEWS for double and is_sum changes
  Remove is_sum arguments from counters and gauges
  Change all instruments to only handle doubles
  Add comment to telemetry::Manager::InitPostScript
  Remove all of the ZEEK_METRICS_ environment variables
  Fix header comments in scripts/policy/frameworks/telemetry/prometheus.zeek
  Change all prometheus #includes to use angle brackets
  Update zeekctl submodule for metrics_port node.cfg option
  Regenerate docs [nomail]
  Remove the is_sum argument from BIF histogram creation methods
  Update NEWS for Telemetry rework
  Remove Telemetry::metrics_export_prefixes option
  Validate that label names are constant in non-zeek metrics
  Avoid calling Collect() in counter/gauge Value() methods if not needed
  Fix some determinism issues with btests
  Temporarily disable the scripts/base/frameworks/telemetry/internal-metrics btest
  Fix the scripts.policy.frameworks.telemetry.prometheus btest to use the service discovery endpoint
  Btest updates due to recent changes
  ...
This commit is contained in:
Tim Wojtulewicz 2024-05-31 14:18:15 -07:00
commit 29d66ec197
93 changed files with 1929 additions and 2411 deletions
Download patch file Download diff file Expand all files Collapse all files

3
.gitmodules vendored
View file

@ -73,3 +73,6 @@
[submodule "auxil/vcpkg"]
path = auxil/vcpkg
url = https://github.com/microsoft/vcpkg
[submodule "auxil/prometheus-cpp"]
path = auxil/prometheus-cpp
url = https://github.com/jupp0r/prometheus-cpp

68
CHANGES
View file

@ -1,3 +1,71 @@
7.0.0-dev.338 | 2024-05-31 14:18:15 -0700
* Add type aliases for instrument and family shared_ptrs (Tim Wojtulewicz, Corelight)
* Update NEWS for double and is_sum changes (Tim Wojtulewicz, Corelight)
* Remove is_sum arguments from counters and gauges (Tim Wojtulewicz, Corelight)
* Change all instruments to only handle doubles (Tim Wojtulewicz, Corelight)
* Add comment to telemetry::Manager::InitPostScript (Tim Wojtulewicz, Corelight)
* Remove all of the ZEEK_METRICS_ environment variables (Tim Wojtulewicz, Corelight)
* Fix header comments in scripts/policy/frameworks/telemetry/prometheus.zeek (Tim Wojtulewicz, Corelight)
* Change all prometheus #includes to use angle brackets (Tim Wojtulewicz, Corelight)
* Update zeekctl submodule for metrics_port node.cfg option (Tim Wojtulewicz, Corelight)
* Remove the is_sum argument from BIF histogram creation methods (Tim Wojtulewicz, Corelight)
* Update NEWS for Telemetry rework (Tim Wojtulewicz, Corelight)
* Remove Telemetry::metrics_export_prefixes option (Tim Wojtulewicz, Corelight)
* Validate that label names are constant in non-zeek metrics (Tim Wojtulewicz, Corelight)
* Avoid calling Collect() in counter/gauge Value() methods if not needed (Tim Wojtulewicz, Corelight)
* Fix some determinism issues with btests (Tim Wojtulewicz, Corelight)
* Temporarily disable the scripts/base/frameworks/telemetry/internal-metrics btest (Tim Wojtulewicz, Corelight)
* Fix the scripts.policy.frameworks.telemetry.prometheus btest to use the service discovery endpoint (Tim Wojtulewicz, Corelight)
* Btest updates due to recent changes (Tim Wojtulewicz, Corelight)
* Update telemetry log policy due to the fact that unit will not be filled in anymore (Tim Wojtulewicz, Corelight)
* Rework everything to access the prometheus-cpp objects more directly (Tim Wojtulewicz, Corelight)
* Move telemetry opaques out of OpaqueVal.h (Tim Wojtulewicz, Corelight)
* Add GetRegistry() method for retrieving access to prometheus-cpp registry (Tim Wojtulewicz, Corelight)
* Remove Broker metrics configuration values and methods (Tim Wojtulewicz, Corelight)
* Add support for callbacks for gauges/counters, restore process stat metrics (Tim Wojtulewicz, Corelight)
* Add a services.json endpoint for Prometheus service discovery (Tim Wojtulewicz, Corelight)
* Don't include endpoint label/value when collecting metrics internally (Tim Wojtulewicz, Corelight)
* Remove everything related to aggregation (Tim Wojtulewicz, Corelight)
* Make all options const-redefs, remove all of the change handler code (Tim Wojtulewicz, Corelight)
* Move base types from telemetry framework to init-bare (Tim Wojtulewicz, Corelight)
* Update test baselines due to underscore changes (Tim Wojtulewicz, Corelight)
* Convert telemetry code to use prometheus-cpp (Tim Wojtulewicz, Corelight)
* Add necessary script-land changes (Tim Wojtulewicz, Corelight)
* Add prometheus-cpp submodule, FindPrometheusCpp cmake module (Tim Wojtulewicz, Corelight)
7.0.0-dev.302 | 2024-05-31 09:37:21 -0700
* More precise error reporting for the disable_analyzer() BiF (Christian Kreibich, Corelight)

12
CMakeLists.txt
View file

@ -234,6 +234,11 @@ if (ZEEK_STANDALONE)
endif ()
# Tell zeek_target_link_libraries to add library dependencies as PRIVATE.
set(zeek_exe_access PRIVATE)
if (${CMAKE_SYSTEM_NAME} MATCHES "FreeBSD")
target_link_libraries(zeek_exe PRIVATE /usr/lib/libutil.so)
target_link_libraries(zeek_exe PRIVATE procstat)
endif ()
else ()
add_library(zeek_lib STATIC)
endif ()
@ -248,6 +253,12 @@ if (TARGET zeek_lib)
install(TARGETS zeek_lib LIBRARY DESTINATION lib)
# Tell zeek_target_link_libraries to add library dependencies as PRIVATE.
set(zeek_lib_access PRIVATE)
if (${CMAKE_SYSTEM_NAME} MATCHES "FreeBSD")
target_link_libraries(zeek_lib PRIVATE /usr/lib/libutil.so)
target_link_libraries(zeek_lib PRIVATE procstat)
endif ()
endif ()
# When building our fuzzers, we also need one extra top-level target that
@ -1113,6 +1124,7 @@ include(GetArchitecture)
include(FindCAres)
include(FindKqueue)
include(FindPrometheusCpp)
include_directories(BEFORE "auxil/out_ptr/include")
if ((OPENSSL_VERSION VERSION_EQUAL "1.1.0") OR (OPENSSL_VERSION VERSION_GREATER "1.1.0"))

42
NEWS
View file

@ -9,6 +9,48 @@ Zeek 7.0.0
Breaking Changes
----------------
- The Telemetry framework has had a major rework, and includes a number of
breaking changes. The biggest change is a move towards a Prometheus-first
model. This removes the internal aggregation of metrics from nodes onto the
manager node, replacing it with a Prometheus service discovery endpoint. The
usage of this endpoint is described in the updated documentation for the
Telemetry framework. Using this endpoint also requires adding a new
``metrics_port`` field to each node in the cluster configuration, denoting
what port to connect to for each node.
All of the metrics-related script-level options, type, and methods have been
moved to the Telemetry framework:
* Option ``Broker::metrics_port` is now ``Telemetry::metrics_port``
* Option ``Broker::metrics_export_endpoint_name`` is now ``Telemetry::metrics_endpoint_name``
The following options have been removed:
* ``Broker::metrics_export_interval``
* ``Broker::metrics_export_topic``
* ``Broker::metrics_import_topics``
* ``Broker::metrics_export_prefixes``
The ``unit`` field has been removed from the telemetry log.
All of the ``BROKER_METRICS_*`` environment variables have been removed. Use
the equivalent script-level variables for setting configuration settings. For
cluster settings, a new ``metrics_port`` option was added to the ``node.cfg``
options and can be used to set the port number. The cluster layout record
defined in ``cluster-layout.zeek`` has the same option, and it can be
automatically populated by ``zeekctl`` during the ``install`` or ``deploy``
commands.
The instruments that previously supported ``count`` in scripts and ``int64_t``
in C++ were removed in favor of only providing ``double`` versions. Prometheus
only handles ``double`` underneath the Zeek code, so it makes sense to only
support it directly in Zeek as well. This also simplifies the code
significantly.
The ``is_sum`` argument has been removed from the constructors/creation
methods for all of the instruments. This again follows how Prometheus works,
where ``counter`` instruments are always considered sums and ``gauge``
instruments are not. ``Histogram`` instruments don't have the concept of
summing.
New Functionality
-----------------

2
VERSION
View file

@ -1 +1 @@
7.0.0-dev.302
7.0.0-dev.338

1
auxil/prometheus-cpp Submodule

@ -0,0 +1 @@
Subproject commit cdb357ad556c9ba96cbfa90fed2940fedf101673

2
auxil/zeekctl

@ -1 +1 @@
Subproject commit 801abac20f836c7623b2f256c7fe02bb13dc79ea
Subproject commit 4dad935e9c995b7ae2f0a4e7677892fcfb988cf0

1
ci/alpine/Dockerfile
View file

@ -16,6 +16,7 @@ RUN apk add --no-cache \
musl-fts-dev \
g++ \
git \
jq \
libpcap-dev \
linux-headers \
make \

1
ci/centos-stream-8/Dockerfile
View file

@ -16,6 +16,7 @@ RUN dnf -y install \
gcc \
gcc-c++ \
git \
jq \
libpcap-devel \
make \
openssl \

1
ci/centos-stream-9/Dockerfile
View file

@ -27,6 +27,7 @@ RUN dnf -y --nobest install \
gcc \
gcc-c++ \
git \
jq \
libpcap-devel \
make \
openssl \

1
ci/debian-11/Dockerfile
View file

@ -16,6 +16,7 @@ RUN apt-get update && apt-get -y install \
g++ \
gcc \
git \
jq \
libkrb5-dev \
libnode-dev \
libpcap-dev \

1
ci/debian-12/Dockerfile
View file

@ -16,6 +16,7 @@ RUN apt-get update && apt-get -y install \
g++ \
gcc \
git \
jq \
libkrb5-dev \
libnode-dev \
libpcap-dev \

1
ci/fedora-38/Dockerfile
View file

@ -14,6 +14,7 @@ RUN dnf -y install \
gcc \
gcc-c++ \
git \
jq \
libpcap-devel \
make \
nodejs-devel \

1
ci/fedora-39/Dockerfile
View file

@ -14,6 +14,7 @@ RUN dnf -y install \
gcc \
gcc-c++ \
git \
jq \
libpcap-devel \
make \
nodejs-devel \

2
ci/freebsd/prepare.sh
View file

@ -6,7 +6,7 @@ set -e
set -x
env ASSUME_ALWAYS_YES=YES pkg bootstrap
pkg install -y bash git cmake swig bison python3 base64 flex ccache
pkg install -y bash git cmake swig bison python3 base64 flex ccache jq
pkg upgrade -y curl
pyver=$(python3 -c 'import sys; print(f"py{sys.version_info[0]}{sys.version_info[1]}")')
pkg install -y $pyver-sqlite3

1
ci/opensuse-leap-15.5/Dockerfile
View file

@ -16,6 +16,7 @@ RUN zypper addrepo https://download.opensuse.org/repositories/openSUSE:Leap:15.5
gcc12-c++ \
git \
gzip \
jq \
libopenssl-devel \
libpcap-devel \
make \

1
ci/opensuse-leap-15.6/Dockerfile
View file

@ -16,6 +16,7 @@ RUN zypper addrepo https://download.opensuse.org/repositories/openSUSE:Leap:15.6
gcc12-c++ \
git \
gzip \
jq \
libopenssl-devel \
libpcap-devel \
make \

1
ci/opensuse-tumbleweed/Dockerfile
View file

@ -22,6 +22,7 @@ RUN zypper refresh \
gcc-c++ \
git \
gzip \
jq \
libopenssl-devel \
libpcap-devel \
make \

1
ci/ubuntu-20.04/Dockerfile
View file

@ -17,6 +17,7 @@ RUN apt-get update && apt-get -y install \
g++ \
gcc \
git \
jq \
lcov \
libkrb5-dev \
libmaxminddb-dev \

1
ci/ubuntu-22.04/Dockerfile
View file

@ -17,6 +17,7 @@ RUN apt-get update && apt-get -y install \
g++ \
gcc \
git \
jq \
lcov \
libkrb5-dev \
libmaxminddb-dev \

1
ci/ubuntu-24.04/Dockerfile
View file

@ -19,6 +19,7 @@ RUN apt-get update && apt-get -y install \
g++ \
gcc \
git \
jq \
lcov \
libkrb5-dev \
libmaxminddb-dev \

2
cmake

@ -1 +1 @@
Subproject commit 71ac5ebe5ff61eb30c8f7b2973faacc6f16f88f2
Subproject commit 34cf738d60a95d5ecf001de2c6e259578be4fc56

2
doc

@ -1 +1 @@
Subproject commit e84d6035ab3f2432abca617b2fbde4b39a5ca778
Subproject commit a369cee890d88a106216915c0202ddb581e39974

94
scripts/base/frameworks/broker/main.zeek
View file

@ -135,45 +135,6 @@ export {
## done reading the pcap.
option peer_counts_as_iosource = T;
## Port for Broker's metric exporter. Setting this to a valid TCP port causes
## Broker to make metrics available to Prometheus scrapers via HTTP. Zeek
## overrides any value provided in zeek_init or earlier at startup if the
## environment variable BROKER_METRICS_PORT is defined.
const metrics_port = 0/unknown &redef;
## Frequency for publishing scraped metrics to the target topic. Zeek
## overrides any value provided in zeek_init or earlier at startup if the
## environment variable BROKER_METRICS_EXPORT_INTERVAL is defined.
option metrics_export_interval = 1 sec;
## Target topic for the metrics. Setting a non-empty string starts the
## periodic publishing of local metrics. Zeek overrides any value provided in
## zeek_init or earlier at startup if the environment variable
## BROKER_METRICS_EXPORT_TOPIC is defined.
option metrics_export_topic = "";
## Topics for the Prometheus exporter for collecting metrics from other
## peers in the network and including them in the output. Has no effect when
## not exporting the metrics to Prometheus.
##
## Zeek overrides any value provided in zeek_init or earlier at startup if
## the environment variable BROKER_METRICS_IMPORT_TOPICS is defined.
option metrics_import_topics: vector of string = vector();
## ID for the metrics exporter. When setting a target topic for the
## exporter, Broker sets this option to the suffix of the new topic *unless*
## the ID is a non-empty string. Since setting a topic starts the periodic
## publishing of events, we recommend setting the ID always first or avoid
## setting it at all if the topic suffix serves as a good-enough ID. Zeek
## overrides any value provided in zeek_init or earlier at startup if the
## environment variable BROKER_METRICS_ENDPOINT_NAME is defined.
option metrics_export_endpoint_name = "";
## Selects prefixes from the local metrics. Only metrics with prefixes
## listed in this variable are included when publishing local metrics.
## Setting an empty vector selects *all* metrics.
option metrics_export_prefixes: vector of string = vector();
## The default topic prefix where logs will be published. The log's stream
## id is appended when writing to a particular stream.
const default_log_topic_prefix = "zeek/logs/" &redef;
@ -458,64 +419,9 @@ event Broker::log_flush() &priority=10
schedule Broker::log_batch_interval { Broker::log_flush() };
}
function update_metrics_export_interval(id: string, val: interval): interval
{
Broker::__set_metrics_export_interval(val);
return val;
}
function update_metrics_export_topic(id: string, val: string): string
{
Broker::__set_metrics_export_topic(val);
return val;
}
function update_metrics_import_topics(id: string, topics: vector of string): vector of string
{
Broker::__set_metrics_import_topics(topics);
return topics;
}
function update_metrics_export_endpoint_name(id: string, val: string): string
{
Broker::__set_metrics_export_endpoint_name(val);
return val;
}
function update_metrics_export_prefixes(id: string, filter: vector of string): vector of string
{
Broker::__set_metrics_export_prefixes(filter);
return filter;
}
event zeek_init()
{
schedule Broker::log_batch_interval { Broker::log_flush() };
# interval
update_metrics_export_interval("Broker::metrics_export_interval",
Broker::metrics_export_interval);
Option::set_change_handler("Broker::metrics_export_interval",
update_metrics_export_interval);
# topic
update_metrics_export_topic("Broker::metrics_export_topic",
Broker::metrics_export_topic);
Option::set_change_handler("Broker::metrics_export_topic",
update_metrics_export_topic);
# import topics
update_metrics_import_topics("Broker::metrics_import_topics",
Broker::metrics_import_topics);
Option::set_change_handler("Broker::metrics_import_topics",
update_metrics_import_topics);
# endpoint name
update_metrics_export_endpoint_name("Broker::metrics_export_endpoint_name",
Broker::metrics_export_endpoint_name);
Option::set_change_handler("Broker::metrics_export_endpoint_name",
update_metrics_export_endpoint_name);
# prefixes
update_metrics_export_prefixes("Broker::metrics_export_prefixes",
Broker::metrics_export_prefixes);
Option::set_change_handler("Broker::metrics_export_prefixes",
update_metrics_export_prefixes);
}
event retry_listen(a: string, p: port, retry: interval)

26
scripts/base/frameworks/cluster/main.zeek
View file

@ -196,6 +196,10 @@ export {
## A unique identifier assigned to the node by the broker framework.
## This field is only set while a node is connected.
id: string &optional;
## The port used to expose metrics to Prometheus. Setting this in a cluster
## configuration will override the setting for Telemetry::metrics_port for
## the node.
metrics_port: port &optional;
};
## Record to represent a cluster node including its name.
@ -218,6 +222,14 @@ export {
## Returns: The :zeek:type:`Cluster::NodeType` the calling node acts as.
global local_node_type: function(): NodeType;
## This function can be called at any time to determine the configured
## metrics port for Prometheus being used by current Zeek instance. If
## :zeek:id:`Cluster::is_enabled` returns false or the node isn't found,
## ``0/unknown`` is returned.
##
## Returns: The metrics port used by the calling node.
global local_node_metrics_port: function(): port;
## The cluster layout definition. This should be placed into a filter
## named cluster-layout.zeek somewhere in the ZEEKPATH. It will be
## automatically loaded if the CLUSTER_NODE environment variable is set.
@ -338,6 +350,20 @@ function local_node_type(): NodeType
return nodes[node]$node_type;
}
function local_node_metrics_port(): port
{
if ( ! is_enabled() )
return 0/unknown;
if ( node !in nodes )
return 0/unknown;
if ( ! nodes[node]?$metrics_port )
return 0/unknown;
return nodes[node]$metrics_port;
}
function node_topic(name: string): string
{
return node_topic_prefix + name + "/";

173
scripts/base/frameworks/telemetry/main.zeek
View file

@ -2,10 +2,10 @@
##! the lower-level telemetry.bif functions.
##!
##! Metrics will be exposed through a Prometheus HTTP endpoint when
##! enabled by setting :zeek:see:`Broker::metrics_port` or using the
##! `BROKER_METRICS_PORT` environment variable.
##! enabled by setting :zeek:see:`Telemetry::metrics_port`.
@load base/misc/version
@load base/frameworks/telemetry/options
module Telemetry;
@ -13,56 +13,13 @@ export {
## Alias for a vector of label values.
type labels_vector: vector of string;
## Type that captures options used to create metrics.
type MetricOpts: record {
## The prefix (namespace) of the metric.
prefix: string;
## The human-readable name of the metric.
name: string;
## The unit of the metric. Use the pseudo-unit "1" if this is a unit-less metric.
unit: string;
## Documentation for this metric.
help_text: string;
## The label names (also called dimensions) of the metric. When
## instantiating or working with concrete metrics, corresponding
## label values have to be provided.
labels: vector of string &default=vector();
## Whether the metric represents something that is accumulating.
## Defaults to ``T`` for counters and ``F`` for gauges and
## histograms.
is_total: bool &optional;
## When creating a :zeek:see:`Telemetry::HistogramFamily`,
## describes the number and bounds of the individual buckets.
bounds: vector of double &optional;
## The same meaning as *bounds*, but as :zeek:type:`count`.
## Only set in the return value of
## :zeek:see:`Telemetry::collect_histogram_metrics`.
## for histograms when the underlying type is ``int64_t``,
## otherwise ignored.
count_bounds: vector of count &optional;
## Describes the underlying metric type.
## Only set in the return value of
## :zeek:see:`Telemetry::collect_metrics` or
## :zeek:see:`Telemetry::collect_histogram_metrics`,
## otherwise ignored.
metric_type: MetricType &optional;
};
## Type representing a family of counters with uninitialized label values.
##
## To create concrete :zeek:see:`Telemetry::Counter` instances, use
## :zeek:see:`Telemetry::counter_with`. To modify counters directly
## use :zeek:see:`Telemetry::counter_family_inc`.
type CounterFamily: record {
__family: opaque of dbl_counter_metric_family;
__family: opaque of counter_metric_family;
__labels: vector of string;
};
@ -75,7 +32,7 @@ export {
## per :zeek:see:`Log::Stream` or number connections broken down
## by protocol and service.
type Counter: record {
__metric: opaque of dbl_counter_metric;
__metric: opaque of counter_metric;
};
## Register a counter family.
@ -146,7 +103,7 @@ export {
## :zeek:see:`Telemetry::gauge_family_inc` or
## :zeek:see:`Telemetry::gauge_family_set` directly.
type GaugeFamily: record {
__family: opaque of dbl_gauge_metric_family;
__family: opaque of gauge_metric_family;
__labels: vector of string;
};
@ -158,7 +115,7 @@ export {
## or footprints of long-lived values as determined by
## :zeek:see:`val_footprint`.
type Gauge: record {
__metric: opaque of dbl_gauge_metric;
__metric: opaque of gauge_metric;
};
## Register a gauge family.
@ -247,14 +204,14 @@ export {
## :zeek:see:`Telemetry::histogram_with` or use
## :zeek:see:`Telemetry::histogram_family_observe` directly.
type HistogramFamily: record {
__family: opaque of dbl_histogram_metric_family;
__family: opaque of histogram_metric_family;
__labels: vector of string;
};
## Type representing a histogram metric with initialized label values.
## Use :zeek:see:`Telemetry::histogram_observe` to make observations.
type Histogram: record {
__metric: opaque of dbl_histogram_metric;
__metric: opaque of histogram_metric;
};
## Register a histogram family.
@ -305,62 +262,6 @@ export {
## Interval at which the :zeek:see:`Telemetry::sync` hook is invoked.
option sync_interval = 10sec;
## Type of elements returned by the :zeek:see:`Telemetry::collect_metrics` function.
type Metric: record {
## A :zeek:see:`Telemetry::MetricOpts` record describing this metric.
opts: MetricOpts;
## The label values associated with this metric, if any.
labels: vector of string;
## The value of gauge or counter cast to a double
## independent of the underlying data type.
## This value is set for all counter and gauge metrics,
## it is unset for histograms.
value: double &optional;
## The value of the underlying gauge or counter as a double
## if the underlying metric type uses ``int64_t``.
## Only counters and gauges created with the C++ API may
## have this value set.
count_value: count &optional;
};
## Type of elements returned by the :zeek:see:`Telemetry::collect_histogram_metrics` function.
type HistogramMetric: record {
## A :zeek:see:`Telemetry::MetricOpts` record describing this histogram.
opts: MetricOpts;
## The label values associated with this histogram, if any.
labels: vector of string;
## Individual counters for each of the buckets as
## described by the *bounds* field in *opts*;
values: vector of double;
## If the underlying data type of the histogram is ``int64_t``,
## this vector will hold the values as counts, otherwise it
## is unset. Only histograms created with the C++ API have
## may have this value set.
count_values: vector of count &optional;
## The number of observations made for this histogram.
observations: double;
## The sum of all observations for this histogram.
sum: double;
## If the underlying data type of the histogram is ``int64_t``,
## the number of observations as :zeek:type:`count`, otherwise
## unset.
count_observations: count &optional;
## If the underlying data type of the histogram is ``int64_t``,
## the sum of all observations as :zeek:type:`count`, otherwise
## unset.
count_sum: count &optional;
};
## Collect all counter and gauge metrics matching the given *name* and *prefix*.
##
## For histogram metrics, use the :zeek:see:`Telemetry::collect_histogram_metrics`.
@ -391,13 +292,12 @@ function make_labels(keys: vector of string, values: labels_vector): table[strin
function register_counter_family(opts: MetricOpts): CounterFamily
{
local f = Telemetry::__dbl_counter_family(
local f = Telemetry::__counter_family(
opts$prefix,
opts$name,
opts$labels,
opts$help_text,
opts$unit,
opts?$is_total ? opts$is_total : T
opts$unit
);
return CounterFamily($__family=f, $__labels=opts$labels);
}
@ -406,7 +306,7 @@ function register_counter_family(opts: MetricOpts): CounterFamily
global error_counter_cf = register_counter_family([
$prefix="zeek",
$name="telemetry_counter_usage_error",
$unit="1",
$unit="",
$help_text="This counter is returned when label usage for counters is wrong. Check reporter.log if non-zero."
]);
@ -419,24 +319,24 @@ function counter_with(cf: CounterFamily, label_values: labels_vector): Counter
}
local labels = make_labels(cf$__labels, label_values);
local m = Telemetry::__dbl_counter_metric_get_or_add(cf$__family, labels);
local m = Telemetry::__counter_metric_get_or_add(cf$__family, labels);
return Counter($__metric=m);
}
function counter_inc(c: Counter, amount: double): bool
{
return Telemetry::__dbl_counter_inc(c$__metric, amount);
return Telemetry::__counter_inc(c$__metric, amount);
}
function counter_set(c: Counter, value: double): bool
{
local cur_value: double = Telemetry::__dbl_counter_value(c$__metric);
local cur_value: double = Telemetry::__counter_value(c$__metric);
if (value < cur_value)
{
Reporter::error(fmt("Attempted to set lower counter value=%s cur_value=%s", value, cur_value));
return F;
}
return Telemetry::__dbl_counter_inc(c$__metric, value - cur_value);
return Telemetry::__counter_inc(c$__metric, value - cur_value);
}
function counter_family_inc(cf: CounterFamily, label_values: labels_vector, amount: double): bool
@ -451,13 +351,12 @@ function counter_family_set(cf: CounterFamily, label_values: labels_vector, valu
function register_gauge_family(opts: MetricOpts): GaugeFamily
{
local f = Telemetry::__dbl_gauge_family(
local f = Telemetry::__gauge_family(
opts$prefix,
opts$name,
opts$labels,
opts$help_text,
opts$unit,
opts?$is_total ? opts$is_total : F
opts$unit
);
return GaugeFamily($__family=f, $__labels=opts$labels);
}
@ -466,7 +365,7 @@ function register_gauge_family(opts: MetricOpts): GaugeFamily
global error_gauge_cf = register_gauge_family([
$prefix="zeek",
$name="telemetry_gauge_usage_error",
$unit="1",
$unit="",
$help_text="This gauge is returned when label usage for gauges is wrong. Check reporter.log if non-zero."
]);
@ -478,29 +377,29 @@ function gauge_with(gf: GaugeFamily, label_values: labels_vector): Gauge
return gauge_with(error_gauge_cf);
}
local labels = make_labels(gf$__labels, label_values);
local m = Telemetry::__dbl_gauge_metric_get_or_add(gf$__family, labels);
local m = Telemetry::__gauge_metric_get_or_add(gf$__family, labels);
return Gauge($__metric=m);
}
function gauge_inc(g: Gauge, amount: double &default=1.0): bool
{
return Telemetry::__dbl_gauge_inc(g$__metric, amount);
return Telemetry::__gauge_inc(g$__metric, amount);
}
function gauge_dec(g: Gauge, amount: double &default=1.0): bool
{
return Telemetry::__dbl_gauge_dec(g$__metric, amount);
return Telemetry::__gauge_dec(g$__metric, amount);
}
function gauge_set(g: Gauge, value: double): bool
{
# Telemetry currently does not implement __dbl_gauge_set(), do
# Telemetry currently does not implement __gauge_set(), do
# it by hand here.
local cur_value: double = Telemetry::__dbl_gauge_value(g$__metric);
local cur_value: double = Telemetry::__gauge_value(g$__metric);
if (value > cur_value)
return Telemetry::__dbl_gauge_inc(g$__metric, value - cur_value);
return Telemetry::__gauge_inc(g$__metric, value - cur_value);
return Telemetry::__dbl_gauge_dec(g$__metric, cur_value - value);
return Telemetry::__gauge_dec(g$__metric, cur_value - value);
}
function gauge_family_inc(gf: GaugeFamily, label_values: labels_vector, value: double): bool
@ -520,14 +419,13 @@ function gauge_family_set(gf: GaugeFamily, label_values: labels_vector, value: d
function register_histogram_family(opts: MetricOpts): HistogramFamily
{
local f = Telemetry::__dbl_histogram_family(
local f = Telemetry::__histogram_family(
opts$prefix,
opts$name,
opts$labels,
opts$bounds,
opts$help_text,
opts$unit,
opts?$is_total ? opts$is_total : F
opts$unit
);
return HistogramFamily($__family=f, $__labels=opts$labels);
}
@ -536,7 +434,7 @@ function register_histogram_family(opts: MetricOpts): HistogramFamily
global error_histogram_hf = register_histogram_family([
$prefix="zeek",
$name="telemetry_histogram_usage_error",
$unit="1",
$unit="",
$help_text="This histogram is returned when label usage for histograms is wrong. Check reporter.log if non-zero.",
$bounds=vector(1.0)
]);
@ -550,16 +448,16 @@ function histogram_with(hf: HistogramFamily, label_values: labels_vector): Histo
}
local labels = make_labels(hf$__labels, label_values);
local m = Telemetry::__dbl_histogram_metric_get_or_add(hf$__family, labels);
local m = Telemetry::__histogram_metric_get_or_add(hf$__family, labels);
return Histogram($__metric=m);
}
function histogram_observe(h: Histogram, measurement: double): bool
{
return Telemetry::__dbl_histogram_observe(h$__metric, measurement);
return Telemetry::__histogram_observe(h$__metric, measurement);
}
function histogram_family_observe(hf: HistogramFamily, label_values: labels_vector, measurement: double): bool
function histogram_family_observe(hf: HistogramFamily, label_values: labels_vector, measurement: double): bool
{
return histogram_observe(histogram_with(hf, label_values), measurement);
}
@ -580,16 +478,11 @@ event run_sync_hook()
schedule sync_interval { run_sync_hook() };
}
event zeek_init()
{
schedule sync_interval { run_sync_hook() };
}
# Expose the Zeek version as Prometheus style info metric
global version_gauge_family = Telemetry::register_gauge_family([
$prefix="zeek",
$name="version_info",
$unit="1",
$unit="",
$help_text="The Zeek version",
$labels=vector("version_number", "major", "minor", "patch", "commit",
"beta", "debug","version_string")
@ -597,6 +490,8 @@ global version_gauge_family = Telemetry::register_gauge_family([
event zeek_init()
{
schedule sync_interval { run_sync_hook() };
local v = Version::info;
local labels = vector(cat(v$version_number),
cat(v$major), cat(v$minor), cat (v$patch),

17
scripts/base/frameworks/telemetry/options.zeek Normal file
View file

@ -0,0 +1,17 @@
module Telemetry;
# This file contains the options for the Telemetry framework. These are kept
# separately so that they can be loaded in bare mode without loading all of
# the rest of the framework. This allows things like the plugins.hooks test
# to see the options without needing the rest.
export {
## Port used to make metric data available to Prometheus scrapers via
## HTTP.
const metrics_port = 0/unknown &redef;
## ID for the metrics exporter. This is used as the 'endpoint' label
## value when exporting data to Prometheus. In a cluster setup, this
## defaults to the name of the node in the cluster configuration.
const metrics_endpoint_name = "" &redef;
}

97
scripts/base/init-bare.zeek
View file

@ -5771,6 +5771,100 @@ export {
const flowbuffer_contract_threshold = 2 * 1024 * 1024 &redef;
}
module Telemetry;
export {
type MetricType: enum {
COUNTER,
GAUGE,
HISTOGRAM,
};
## Type that captures options used to create metrics.
type MetricOpts: record {
## The prefix (namespace) of the metric. Zeek uses the ``zeek``
## prefix for any internal metrics and the ``process`` prefix
## for any metrics involving process state (CPU, memory, etc).
prefix: string;
## The human-readable name of the metric. This is set to the
## full prefixed name including the unit when returned from
## :zeek:see:`Telemetry::collect_metrics` or
## :zeek:see:`Telemetry::collect_histogram_metrics`.
name: string;
## The unit of the metric. Leave this unset for a unit-less
## metric. Will be unset when returned from
## :zeek:see:`Telemetry::collect_metrics` or
## :zeek:see:`Telemetry::collect_histogram_metrics`.
unit: string &optional;
## Documentation for this metric.
help_text: string &optional;
## The label names (also called dimensions) of the metric. When
## instantiating or working with concrete metrics, corresponding
## label values have to be provided. Examples of a label might
## be the protocol a general observation applies to, the
## directionality in a traffic flow, or protocol-specific
## context like a particular message type.
labels: vector of string &default=vector();
## Whether the metric represents something that is accumulating.
## Defaults to ``T`` for counters and ``F`` for gauges and
## histograms.
is_total: bool &optional;
## When creating a :zeek:see:`Telemetry::HistogramFamily`,
## describes the number and bounds of the individual buckets.
bounds: vector of double &optional;
## Describes the underlying metric type.
## Only set in the return value of
## :zeek:see:`Telemetry::collect_metrics` or
## :zeek:see:`Telemetry::collect_histogram_metrics`,
## otherwise ignored.
metric_type: MetricType &optional;
};
## Metrics returned by the :zeek:see:`Telemetry::collect_metrics` function.
type Metric: record {
## A :zeek:see:`Telemetry::MetricOpts` record describing this metric.
opts: MetricOpts;
## The label values associated with this metric, if any.
labels: vector of string;
## The value of gauge or counter cast to a double
## independent of the underlying data type.
## This value is set for all counter and gauge metrics,
## it is unset for histograms.
value: double &optional;
};
## Histograms returned by the :zeek:see:`Telemetry::collect_histogram_metrics` function.
type HistogramMetric: record {
## A :zeek:see:`Telemetry::MetricOpts` record describing this histogram.
opts: MetricOpts;
## The label values associated with this histogram, if any.
labels: vector of string;
## Individual counters for each of the buckets as
## described by the *bounds* field in *opts*;
values: vector of double;
## The number of observations made for this histogram.
observations: double;
## The sum of all observations for this histogram.
sum: double;
};
type MetricVector : vector of Metric;
type HistogramMetricVector : vector of HistogramMetric;
}
module GLOBAL;
## Seed for hashes computed internally for probabilistic data structures. Using
@ -5823,6 +5917,9 @@ const io_poll_interval_default = 100 &redef;
## .. zeek:see:: io_poll_interval_default
const io_poll_interval_live = 10 &redef;
## Whether Zeek is being run under test. This can be used to alter functionality
## while testing, but should be used sparingly.
const running_under_test: bool = F &redef;
global done_with_network = F;
event net_done(t: time)

1
scripts/base/init-frameworks-and-bifs.zeek
View file

@ -11,6 +11,7 @@
@load base/frameworks/config
@load base/frameworks/analyzer
@load base/frameworks/files
@load base/frameworks/telemetry/options
@load base/bif

55
scripts/policy/frameworks/telemetry/log.zeek
View file

@ -39,9 +39,6 @@ export {
## The name of the metric.
name: string &log;
## The unit of this metric, or unset if unit-less.
unit: string &log &optional;
## The names of the individual labels.
labels: vector of string &log;
@ -66,9 +63,6 @@ export {
## The name of the metric.
name: string &log;
## The unit of this metric, or unset if unit-less.
unit: string &log &optional;
## The names of the individual labels.
labels: vector of string &log;
@ -104,27 +98,38 @@ export {
function do_log()
{
local ts = network_time();
local metrics = Telemetry::collect_metrics();
## TODO: this is potentially slow, since it requires looping over all of the metrics for each
## prefix, and then doing it again for all of the histograms multiple times.
local metrics : vector of Telemetry::Metric;
if ( |log_prefixes| > 0 )
{
for ( prefix in log_prefixes )
{
metrics += Telemetry::collect_metrics(prefix, "*");
}
}
else
{
metrics = Telemetry::collect_metrics();
}
for ( i in metrics )
{
local m = metrics[i];
# Histograms don't have single values, skip over them.
if ( m$opts$metric_type == DOUBLE_HISTOGRAM || m$opts$metric_type == INT_HISTOGRAM )
next;
if ( |log_prefixes| > 0 && m$opts$prefix !in log_prefixes )
if ( m$opts$metric_type == HISTOGRAM )
next;
# Render the metric_type as a short string. Unknown
# shouldn't really happen, but lets have a fallback.
local metric_type = "unknown";
switch ( m$opts$metric_type ) {
case DOUBLE_COUNTER, INT_COUNTER:
case COUNTER:
metric_type = "counter";
break;
case DOUBLE_GAUGE, INT_GAUGE:
case GAUGE:
metric_type = "gauge";
break;
}
@ -138,22 +143,29 @@ function do_log()
$label_values=m$labels,
$value=m$value);
if ( m$opts$unit != "1" )
rec$unit = m$opts$unit;
Log::write(LOG, rec);
}
# Logging of histograms.
ts = network_time();
local histogram_metrics = Telemetry::collect_histogram_metrics();
local histogram_metrics : vector of Telemetry::HistogramMetric;
if ( |log_prefixes| > 0 )
{
for ( prefix in log_prefixes )
{
histogram_metrics += Telemetry::collect_histogram_metrics(prefix, "*");
}
}
else
{
histogram_metrics = Telemetry::collect_histogram_metrics();
}
for ( i in histogram_metrics )
{
local hm = histogram_metrics[i];
if ( |log_prefixes| > 0 && hm$opts$prefix !in log_prefixes )
next;
local hrec = HistogramInfo($ts=ts,
$peer=peer_description,
$prefix=hm$opts$prefix,
@ -165,9 +177,6 @@ function do_log()
$sum=hm$sum,
$observations=hm$observations);
if ( hm$opts$unit != "1" )
hrec$unit = hm$opts$unit;
Log::write(LOG_HISTOGRAM, hrec);
}
}

31
scripts/policy/frameworks/telemetry/prometheus.zeek
View file

@ -1,28 +1,23 @@
##! In a cluster configuration, open port 9911 on the manager for
##! Prometheus exposition and import all metrics from the
##! `zeek/cluster/metrics/...` topic.
##! In a cluster configuration, open the port number for metrics
##! from the cluster node configuration for exporting data to
##! Prometheus.
##!
##! For customization or disabling, redef the involved Broker options again.
##! Specifically, to disable listening on port 9911, set
##! :zeek:see:`Broker::metrics_port` to `0/unknown` again.
##!
##! Note that in large clusters, metrics import may cause significant
##! communication overhead as well as load on the manager.
##! For customization or disabling, redef the involved Telemetry options
##! again. Specifically, to disable listening on port 9911, set
##! :zeek:see:`Telemetry::metrics_port` to `0/unknown` again.
##!
##! The manager node will also provide a ``/services.json`` endpoint
##! for the HTTP Service Discovery system in Prometheus to use for
##! configuration. This endpoint will include information for all of
##! the other nodes in the cluster.
@load base/frameworks/cluster
@if ( Cluster::is_enabled() )
# Use Cluster::node as "endpoint" label
redef Broker::metrics_export_endpoint_name = Cluster::node;
redef Telemetry::metrics_endpoint_name = Cluster::node;
# The manager opens port 9911 and imports metrics from all nodes by default.
@if ( Cluster::local_node_type() == Cluster::MANAGER )
redef Broker::metrics_port = 9911/tcp;
redef Broker::metrics_import_topics = vector("zeek/cluster/metrics/");
@else
redef Broker::metrics_export_topic = "zeek/cluster/metrics/";
@if ( Cluster::local_node_metrics_port() != 0/unknown )
redef Telemetry::metrics_port = Cluster::local_node_metrics_port();
@endif
@endif

10
scripts/policy/misc/stats.zeek
View file

@ -91,35 +91,35 @@ export {
global bytes_received_cf = Telemetry::register_counter_family([
$prefix="zeek",
$name="net-received-bytes",
$unit="1",
$unit="",
$help_text="Total number of bytes received",
]);
global packets_received_cf = Telemetry::register_counter_family([
$prefix="zeek",
$name="net-received-packets",
$unit="1",
$unit="",
$help_text="Total number of packets received",
]);
global packets_dropped_cf = Telemetry::register_counter_family([
$prefix="zeek",
$name="net-dropped-packets",
$unit="1",
$unit="",
$help_text="Total number of packets dropped",
]);
global link_packets_cf = Telemetry::register_counter_family([
$prefix="zeek",
$name="net-link-packets",
$unit="1",
$unit="",
$help_text="Total number of packets on the packet source link before filtering",
]);
global packets_filtered_cf = Telemetry::register_counter_family([
$prefix="zeek",
$name="net-filtered-packets",
$unit="1",
$unit="",
$help_text="Total number of packets filtered",
]);

6
src/EventHandler.cc
View file

@ -47,9 +47,9 @@ void EventHandler::Call(Args* vl, bool no_remote, double ts) {
if ( ! call_count ) {
static auto eh_invocations_family =
telemetry_mgr->CounterFamily("zeek", "event-handler-invocations", {"name"},
"Number of times the given event handler was called", "1", true);
"Number of times the given event handler was called");
call_count = eh_invocations_family.GetOrAdd({{"name", name}});
call_count = eh_invocations_family->GetOrAdd({{"name", name}});
}
call_count->Inc();
@ -113,4 +113,6 @@ void EventHandler::NewEvent(Args* vl) {
event_mgr.Dispatch(ev);
}
uint64_t EventHandler::CallCount() const { return call_count ? call_count->Value() : 0; }
} // namespace zeek

10
src/EventHandler.h
View file

@ -9,7 +9,6 @@
#include "zeek/Type.h"
#include "zeek/ZeekArgs.h"
#include "zeek/ZeekList.h"
#include "zeek/telemetry/Counter.h"
namespace zeek {
@ -17,6 +16,10 @@ namespace run_state {
extern double network_time;
} // namespace run_state
namespace telemetry {
class Counter;
}
class Func;
using FuncPtr = IntrusivePtr<Func>;
@ -60,7 +63,8 @@ public:
void SetGenerateAlways(bool arg_generate_always = true) { generate_always = arg_generate_always; }
bool GenerateAlways() const { return generate_always; }
uint64_t CallCount() const { return call_count ? call_count->Value() : 0; }
// Returns the number of times this EventHandler has been called since startup.
uint64_t CallCount() const;
private:
void NewEvent(zeek::Args* vl); // Raise new_event() meta event.
@ -74,7 +78,7 @@ private:
bool generate_always;
// Initialize this lazy, so we don't expose metrics for 0 values.
std::optional<zeek::telemetry::IntCounter> call_count;
std::shared_ptr<zeek::telemetry::Counter> call_count;
std::unordered_set<std::string> auto_publish;
};

40
src/OpaqueVal.cc
View file

@ -1092,44 +1092,4 @@ ValPtr ParaglobVal::DoClone(CloneState* state) {
}
}
std::optional<BrokerData> TelemetryVal::DoSerializeData() const { return std::nullopt; }
bool TelemetryVal::DoUnserializeData(BrokerDataView) { return false; }
TelemetryVal::TelemetryVal(telemetry::IntCounter) : OpaqueVal(int_counter_metric_type) {}
TelemetryVal::TelemetryVal(telemetry::IntCounterFamily) : OpaqueVal(int_counter_metric_family_type) {}
TelemetryVal::TelemetryVal(telemetry::DblCounter) : OpaqueVal(dbl_counter_metric_type) {}
TelemetryVal::TelemetryVal(telemetry::DblCounterFamily) : OpaqueVal(dbl_counter_metric_family_type) {}
TelemetryVal::TelemetryVal(telemetry::IntGauge) : OpaqueVal(int_gauge_metric_type) {}
TelemetryVal::TelemetryVal(telemetry::IntGaugeFamily) : OpaqueVal(int_gauge_metric_family_type) {}
TelemetryVal::TelemetryVal(telemetry::DblGauge) : OpaqueVal(dbl_gauge_metric_type) {}
TelemetryVal::TelemetryVal(telemetry::DblGaugeFamily) : OpaqueVal(dbl_gauge_metric_family_type) {}
TelemetryVal::TelemetryVal(telemetry::IntHistogram) : OpaqueVal(int_histogram_metric_type) {}
TelemetryVal::TelemetryVal(telemetry::IntHistogramFamily) : OpaqueVal(int_histogram_metric_family_type) {}
TelemetryVal::TelemetryVal(telemetry::DblHistogram) : OpaqueVal(dbl_histogram_metric_type) {}
TelemetryVal::TelemetryVal(telemetry::DblHistogramFamily) : OpaqueVal(dbl_histogram_metric_family_type) {}
IMPLEMENT_OPAQUE_VALUE(IntCounterMetricVal)
IMPLEMENT_OPAQUE_VALUE(IntCounterMetricFamilyVal)
IMPLEMENT_OPAQUE_VALUE(DblCounterMetricVal)
IMPLEMENT_OPAQUE_VALUE(DblCounterMetricFamilyVal)
IMPLEMENT_OPAQUE_VALUE(IntGaugeMetricVal)
IMPLEMENT_OPAQUE_VALUE(IntGaugeMetricFamilyVal)
IMPLEMENT_OPAQUE_VALUE(DblGaugeMetricVal)
IMPLEMENT_OPAQUE_VALUE(DblGaugeMetricFamilyVal)
IMPLEMENT_OPAQUE_VALUE(IntHistogramMetricVal)
IMPLEMENT_OPAQUE_VALUE(IntHistogramMetricFamilyVal)
IMPLEMENT_OPAQUE_VALUE(DblHistogramMetricVal)
IMPLEMENT_OPAQUE_VALUE(DblHistogramMetricFamilyVal)
} // namespace zeek

58
src/OpaqueVal.h
View file

@ -443,62 +443,4 @@ private:
std::unique_ptr<paraglob::Paraglob> internal_paraglob;
};
/**
* Base class for metric handles. Handle types are not serializable.
*/
class TelemetryVal : public OpaqueVal {
protected:
explicit TelemetryVal(telemetry::IntCounter);
explicit TelemetryVal(telemetry::IntCounterFamily);
explicit TelemetryVal(telemetry::DblCounter);
explicit TelemetryVal(telemetry::DblCounterFamily);
explicit TelemetryVal(telemetry::IntGauge);
explicit TelemetryVal(telemetry::IntGaugeFamily);
explicit TelemetryVal(telemetry::DblGauge);
explicit TelemetryVal(telemetry::DblGaugeFamily);
explicit TelemetryVal(telemetry::IntHistogram);
explicit TelemetryVal(telemetry::IntHistogramFamily);
explicit TelemetryVal(telemetry::DblHistogram);
explicit TelemetryVal(telemetry::DblHistogramFamily);
std::optional<BrokerData> DoSerializeData() const override;
bool DoUnserializeData(BrokerDataView data) override;
};
template<class Handle>
class TelemetryValImpl : public TelemetryVal {
public:
using HandleType = Handle;
explicit TelemetryValImpl(Handle hdl) : TelemetryVal(hdl), hdl(hdl) {}
Handle GetHandle() const noexcept { return hdl; }
static zeek::OpaqueValPtr OpaqueInstantiate() {
reporter->Error("TelemetryValImpl::OpaqueInstantiate is unsupported");
return nullptr;
}
protected:
ValPtr DoClone(CloneState*) override { return make_intrusive<TelemetryValImpl>(hdl); }
const char* OpaqueName() const override { return Handle::OpaqueName; }
private:
Handle hdl;
};
using IntCounterMetricVal = TelemetryValImpl<telemetry::IntCounter>;
using IntCounterMetricFamilyVal = TelemetryValImpl<telemetry::IntCounterFamily>;
using DblCounterMetricVal = TelemetryValImpl<telemetry::DblCounter>;
using DblCounterMetricFamilyVal = TelemetryValImpl<telemetry::DblCounterFamily>;
using IntGaugeMetricVal = TelemetryValImpl<telemetry::IntGauge>;
using IntGaugeMetricFamilyVal = TelemetryValImpl<telemetry::IntGaugeFamily>;
using DblGaugeMetricVal = TelemetryValImpl<telemetry::DblGauge>;
using DblGaugeMetricFamilyVal = TelemetryValImpl<telemetry::DblGaugeFamily>;
using IntHistogramMetricVal = TelemetryValImpl<telemetry::IntHistogram>;
using IntHistogramMetricFamilyVal = TelemetryValImpl<telemetry::IntHistogramFamily>;
using DblHistogramMetricVal = TelemetryValImpl<telemetry::DblHistogram>;
using DblHistogramMetricFamilyVal = TelemetryValImpl<telemetry::DblHistogramFamily>;
} // namespace zeek

18
src/Type.h
View file

@ -1042,15 +1042,9 @@ extern zeek::OpaqueTypePtr bloomfilter_type;
extern zeek::OpaqueTypePtr x509_opaque_type;
extern zeek::OpaqueTypePtr ocsp_resp_opaque_type;
extern zeek::OpaqueTypePtr paraglob_type;
extern zeek::OpaqueTypePtr int_counter_metric_type;
extern zeek::OpaqueTypePtr int_counter_metric_family_type;
extern zeek::OpaqueTypePtr dbl_counter_metric_type;
extern zeek::OpaqueTypePtr dbl_counter_metric_family_type;
extern zeek::OpaqueTypePtr int_gauge_metric_type;
extern zeek::OpaqueTypePtr int_gauge_metric_family_type;
extern zeek::OpaqueTypePtr dbl_gauge_metric_type;
extern zeek::OpaqueTypePtr dbl_gauge_metric_family_type;
extern zeek::OpaqueTypePtr int_histogram_metric_type;
extern zeek::OpaqueTypePtr int_histogram_metric_family_type;
extern zeek::OpaqueTypePtr dbl_histogram_metric_type;
extern zeek::OpaqueTypePtr dbl_histogram_metric_family_type;
extern zeek::OpaqueTypePtr counter_metric_type;
extern zeek::OpaqueTypePtr counter_metric_family_type;
extern zeek::OpaqueTypePtr gauge_metric_type;
extern zeek::OpaqueTypePtr gauge_metric_family_type;
extern zeek::OpaqueTypePtr histogram_metric_type;
extern zeek::OpaqueTypePtr histogram_metric_family_type;

100
src/broker/Manager.cc
View file

@ -331,82 +331,6 @@ void Manager::InitPostScript() {
config.set("caf.work-stealing.moderate-steal-interval", get_option("Broker::moderate_interval")->AsCount());
config.set("caf.work-stealing.relaxed-steal-interval", get_option("Broker::relaxed_interval")->AsCount());
// Before launching Broker, we check whether the configuration contains
// values for the metric_exporter_* options. If Broker already has picked up
// values from environment variables (or config files) then we write then
// back. Otherwise, we forward user-defined values from script land (but
// ignore defaults).
WITH_OPT_MAPPING("broker.metrics.port", "Broker::metrics_port") {
if ( auto port = opt.broker_read<uint16_t>() ) {
opt.zeek_write(broker::port{*port, broker::port::protocol::tcp});
}
else {
auto ptr = opt.zeek_read()->AsPortVal();
if ( ptr->IsTCP() )
opt.broker_write(ptr->Port());
}
}
WITH_OPT_MAPPING("broker.metrics.export.interval", "Broker::metrics_export_interval") {
if ( auto ts = opt.broker_read<broker::timespan>() ) {
opt.zeek_write(*ts);
}
else {
using std::chrono::duration_cast;
auto val = opt.zeek_read()->AsInterval();
auto frac_ts = broker::fractional_seconds{val};
if ( frac_ts.count() > 0.0 )
opt.broker_write(duration_cast<broker::timespan>(frac_ts));
}
}
WITH_OPT_MAPPING("broker.metrics.export.topic", "Broker::metrics_export_topic") {
if ( auto str = opt.broker_read<std::string>() ) {
opt.zeek_write(*str);
}
else {
auto ptr = opt.zeek_read()->AsStringVal();
if ( ptr->Len() > 0 )
opt.broker_write(ptr->ToStdString());
}
}
WITH_OPT_MAPPING("broker.metrics.endpoint-name", "Broker::metrics_export_endpoint_name") {
if ( auto str = opt.broker_read<std::string>() ) {
opt.zeek_write(*str);
}
else {
auto ptr = opt.zeek_read()->AsStringVal();
if ( ptr->Len() > 0 )
opt.broker_write(ptr->ToStdString());
}
}
WITH_OPT_MAPPING("broker.metrics.export.prefixes", "Broker::metrics_export_prefixes") {
if ( auto str = opt.broker_read<std::vector<std::string>>() ) {
opt.zeek_write(*str);
}
else {
auto ptr = opt.zeek_read()->AsVectorVal();
std::vector<std::string> str_ls;
for ( unsigned index = 0; index < ptr->Size(); ++index )
str_ls.emplace_back(ptr->StringValAt(index)->ToStdString());
opt.broker_write(std::move(str_ls));
}
}
WITH_OPT_MAPPING("broker.metrics.import.topics", "Broker::metrics_import_topics") {
if ( auto topics = opt.broker_read<std::vector<std::string>>() ) {
opt.zeek_write(*topics);
}
else {
auto ptr = opt.zeek_read()->AsVectorVal();
std::vector<std::string> str_ls;
for ( unsigned index = 0; index < ptr->Size(); ++index )
str_ls.emplace_back(ptr->StringValAt(index)->ToStdString());
opt.broker_write(std::move(str_ls));
}
}
auto cqs = get_option("Broker::congestion_queue_size")->AsCount();
bstate = std::make_shared<BrokerState>(std::move(config), cqs);
@ -415,8 +339,6 @@ void Manager::InitPostScript() {
bstate->subscriber.add_topic(broker::topic::store_events(), true);
telemetry_mgr->InitPostBrokerSetup(bstate->endpoint);
InitializeBrokerStoreForwarding();
}
@ -1814,26 +1736,4 @@ void Manager::PrepareForwarding(const std::string& name) {
DBG_LOG(DBG_BROKER, "Resolved table forward for data store %s", name.c_str());
}
void Manager::SetMetricsExportInterval(double value) {
using dbl_seconds = std::chrono::duration<double>;
auto ts = std::chrono::duration_cast<broker::timespan>(dbl_seconds{value});
bstate->endpoint.metrics_exporter().set_interval(ts);
}
void Manager::SetMetricsExportTopic(std::string value) {
bstate->endpoint.metrics_exporter().set_target(std::move(value));
}
void Manager::SetMetricsImportTopics(std::vector<std::string> topics) {
bstate->endpoint.metrics_exporter().set_import_topics(std::move(topics));
}
void Manager::SetMetricsExportEndpointName(std::string value) {
bstate->endpoint.metrics_exporter().set_id(std::move(value));
}
void Manager::SetMetricsExportPrefixes(std::vector<std::string> filter) {
bstate->endpoint.metrics_exporter().set_prefixes(std::move(filter));
}
} // namespace zeek::Broker

36
src/broker/Manager.h
View file

@ -377,42 +377,6 @@ public:
~ScriptScopeGuard() { --script_scope; }
};
/**
* Changes the frequency for publishing scraped metrics to the target topic.
* Passing a zero-length interval has no effect.
* @param value Interval between two scrapes in seconds.
*/
void SetMetricsExportInterval(double value);
/**
* Sets a new target topic for the metrics. Passing an empty string has no
* effect.
* @param value The new topic for publishing local metrics to.
*/
void SetMetricsExportTopic(std::string value);
/**
* Sets the import topics for a node importing metrics.
*
* @param topics List of topics from which to import metrics.
*/
void SetMetricsImportTopics(std::vector<std::string> topics);
/**
* Sets a new ID for the metrics exporter. Passing an empty string has no
* effect.
* @param value The new ID of the exporter in published metrics.
*/
void SetMetricsExportEndpointName(std::string value);
/**
* Sets a prefix selection for the metrics exporter. An empty vector selects
* *all* metrics.
* @param filter List of selected metric prefixes or an empty vector for
* selecting all metrics.
*/
void SetMetricsExportPrefixes(std::vector<std::string> filter);
private:
// Process events used for Broker store backed zeek tables
void ProcessStoreEvent(broker::data msg);

56
src/broker/comm.bif
View file

@ -178,59 +178,3 @@ function Broker::__node_id%(%): string
zeek::Broker::Manager::ScriptScopeGuard ssg;
return zeek::make_intrusive<zeek::StringVal>(broker_mgr->NodeID());
%}
function Broker::__set_metrics_export_interval%(value: interval%): bool
%{
// This BIF may run prior to broker::Manager::InitPostScript. In this case,
// broker_mgr is still null but we can safely ignore this event because the
// Manager is going to initialize Broker using the most recent value of the
// corresponding option.
zeek::Broker::Manager::ScriptScopeGuard ssg;
if ( broker_mgr )
broker_mgr->SetMetricsExportInterval(value);
return zeek::val_mgr->True();
%}
function Broker::__set_metrics_export_topic%(value: string%): bool
%{
zeek::Broker::Manager::ScriptScopeGuard ssg;
if ( broker_mgr )
broker_mgr->SetMetricsExportTopic(value->ToStdString());
return zeek::val_mgr->True();
%}
function Broker::__set_metrics_import_topics%(filter: string_vec%): bool
%{
zeek::Broker::Manager::ScriptScopeGuard ssg;
if ( broker_mgr )
{
std::vector<std::string> slist;
auto* vval = filter->AsVectorVal();
for ( unsigned index = 0; index < vval->Size(); ++index )
slist.emplace_back(vval->StringValAt(index)->ToStdString());
broker_mgr->SetMetricsImportTopics(std::move(slist));
}
return zeek::val_mgr->True();
%}
function Broker::__set_metrics_export_endpoint_name%(value: string%): bool
%{
zeek::Broker::Manager::ScriptScopeGuard ssg;
if ( broker_mgr )
broker_mgr->SetMetricsExportEndpointName(value->ToStdString());
return zeek::val_mgr->True();
%}
function Broker::__set_metrics_export_prefixes%(filter: string_vec%): bool
%{
zeek::Broker::Manager::ScriptScopeGuard ssg;
if ( broker_mgr )
{
std::vector<std::string> slist;
auto* vval = filter->AsVectorVal();
for ( unsigned index = 0; index < vval->Size(); ++index )
slist.emplace_back(vval->StringValAt(index)->ToStdString());
broker_mgr->SetMetricsExportPrefixes(std::move(slist));
}
return zeek::val_mgr->True();
%}

18
src/logging/Manager.cc
View file

@ -223,9 +223,9 @@ struct Manager::WriterInfo {
bool hook_initialized = false;
string instantiating_filter;
telemetry::IntCounter total_writes;
std::shared_ptr<telemetry::Counter> total_writes;
WriterInfo(telemetry::IntCounter total_writes) : total_writes(total_writes) {}
WriterInfo(std::shared_ptr<telemetry::Counter> total_writes) : total_writes(std::move(total_writes)) {}
};
struct Manager::Stream {
@ -244,7 +244,7 @@ struct Manager::Stream {
bool enable_remote = false;
std::optional<telemetry::IntCounter> total_writes; // Initialized on first write.
std::shared_ptr<telemetry::Counter> total_writes; // Initialized on first write.
// State about delayed writes for this Stream.
detail::DelayQueue delay_queue;
@ -418,14 +418,12 @@ void Manager::Stream::DispatchDelayExpiredTimer(double t, bool is_expire) {
Manager::Manager()
: plugin::ComponentManager<logging::Component>("Log", "Writer"),
total_log_stream_writes_family(telemetry_mgr->CounterFamily("zeek", "log-stream-writes", {"module", "stream"},
"Total number of log writes for the given stream.",
"1", true)),
"Total number of log writes for the given stream.")),
total_log_writer_writes_family(
telemetry_mgr
->CounterFamily("zeek", "log-writer-writes", {"writer", "module", "stream", "filter-name", "path"},
"Total number of log writes passed to a concrete log writer not vetoed by stream or "
"filter policies.",
"1", true)) {
"filter policies.")) {
rotations_pending = 0;
}
@ -947,7 +945,7 @@ bool Manager::Write(EnumVal* id, RecordVal* columns_arg) {
if ( ! stream->total_writes ) {
std::string module_name = zeek::detail::extract_module_name(stream->name.c_str());
std::initializer_list<telemetry::LabelView> labels{{"module", module_name}, {"stream", stream->name}};
stream->total_writes = total_log_stream_writes_family.GetOrAdd(labels);
stream->total_writes = total_log_stream_writes_family->GetOrAdd(labels);
}
stream->total_writes->Inc();
@ -1173,7 +1171,7 @@ bool Manager::WriteToFilters(const Manager::Stream* stream, zeek::RecordValPtr c
}
assert(w != stream->writers.end());
w->second->total_writes.Inc();
w->second->total_writes->Inc();
// Write takes ownership of vals.
assert(writer);
@ -1616,7 +1614,7 @@ WriterFrontend* Manager::CreateWriter(EnumVal* id, EnumVal* writer, WriterBacken
{"filter-name", instantiating_filter},
{"path", info->path}};
WriterInfo* winfo = new WriterInfo(zeek::log_mgr->total_log_writer_writes_family.GetOrAdd(labels));
WriterInfo* winfo = new WriterInfo(zeek::log_mgr->total_log_writer_writes_family->GetOrAdd(labels));
winfo->type = writer->Ref()->AsEnumVal();
winfo->writer = nullptr;
winfo->open_time = run_state::network_time;

4
src/logging/Manager.h
View file

@ -403,8 +403,8 @@ private:
FuncPtr rotation_format_func;
FuncPtr log_stream_policy_hook;
telemetry::IntCounterFamily total_log_stream_writes_family;
telemetry::IntCounterFamily total_log_writer_writes_family;
std::shared_ptr<telemetry::CounterFamily> total_log_stream_writes_family;
std::shared_ptr<telemetry::CounterFamily> total_log_writer_writes_family;
zeek_uint_t last_delay_token = 0;
std::vector<detail::WriteContext> active_writes;

39
src/script_opt/FuncInfo.cc
View file

@ -151,32 +151,19 @@ static std::unordered_map<std::string, unsigned int> func_attrs = {
{"Supervisor::__stem_pid", ATTR_NO_SCRIPT_SIDE_EFFECTS},
{"Telemetry::__collect_histogram_metrics", ATTR_NO_SCRIPT_SIDE_EFFECTS},
{"Telemetry::__collect_metrics", ATTR_NO_SCRIPT_SIDE_EFFECTS},
{"Telemetry::__dbl_counter_family", ATTR_NO_SCRIPT_SIDE_EFFECTS},
{"Telemetry::__dbl_counter_inc", ATTR_NO_SCRIPT_SIDE_EFFECTS},
{"Telemetry::__dbl_counter_metric_get_or_add", ATTR_NO_SCRIPT_SIDE_EFFECTS},
{"Telemetry::__dbl_counter_value", ATTR_NO_SCRIPT_SIDE_EFFECTS},
{"Telemetry::__dbl_gauge_dec", ATTR_NO_SCRIPT_SIDE_EFFECTS},
{"Telemetry::__dbl_gauge_family", ATTR_NO_SCRIPT_SIDE_EFFECTS},
{"Telemetry::__dbl_gauge_inc", ATTR_NO_SCRIPT_SIDE_EFFECTS},
{"Telemetry::__dbl_gauge_metric_get_or_add", ATTR_NO_SCRIPT_SIDE_EFFECTS},
{"Telemetry::__dbl_gauge_value", ATTR_NO_SCRIPT_SIDE_EFFECTS},
{"Telemetry::__dbl_histogram_family", ATTR_NO_SCRIPT_SIDE_EFFECTS},
{"Telemetry::__dbl_histogram_metric_get_or_add", ATTR_NO_SCRIPT_SIDE_EFFECTS},
{"Telemetry::__dbl_histogram_observe", ATTR_NO_SCRIPT_SIDE_EFFECTS},
{"Telemetry::__dbl_histogram_sum", ATTR_NO_SCRIPT_SIDE_EFFECTS},
{"Telemetry::__int_counter_family", ATTR_NO_SCRIPT_SIDE_EFFECTS},
{"Telemetry::__int_counter_inc", ATTR_NO_SCRIPT_SIDE_EFFECTS},
{"Telemetry::__int_counter_metric_get_or_add", ATTR_NO_SCRIPT_SIDE_EFFECTS},
{"Telemetry::__int_counter_value", ATTR_NO_SCRIPT_SIDE_EFFECTS},
{"Telemetry::__int_gauge_dec", ATTR_NO_SCRIPT_SIDE_EFFECTS},
{"Telemetry::__int_gauge_family", ATTR_NO_SCRIPT_SIDE_EFFECTS},
{"Telemetry::__int_gauge_inc", ATTR_NO_SCRIPT_SIDE_EFFECTS},
{"Telemetry::__int_gauge_metric_get_or_add", ATTR_NO_SCRIPT_SIDE_EFFECTS},
{"Telemetry::__int_gauge_value", ATTR_NO_SCRIPT_SIDE_EFFECTS},
{"Telemetry::__int_histogram_family", ATTR_NO_SCRIPT_SIDE_EFFECTS},
{"Telemetry::__int_histogram_metric_get_or_add", ATTR_NO_SCRIPT_SIDE_EFFECTS},
{"Telemetry::__int_histogram_observe", ATTR_NO_SCRIPT_SIDE_EFFECTS},
{"Telemetry::__int_histogram_sum", ATTR_NO_SCRIPT_SIDE_EFFECTS},
{"Telemetry::__counter_family", ATTR_NO_SCRIPT_SIDE_EFFECTS},
{"Telemetry::__counter_inc", ATTR_NO_SCRIPT_SIDE_EFFECTS},
{"Telemetry::__counter_metric_get_or_add", ATTR_NO_SCRIPT_SIDE_EFFECTS},
{"Telemetry::__counter_value", ATTR_NO_SCRIPT_SIDE_EFFECTS},
{"Telemetry::__gauge_dec", ATTR_NO_SCRIPT_SIDE_EFFECTS},
{"Telemetry::__gauge_family", ATTR_NO_SCRIPT_SIDE_EFFECTS},
{"Telemetry::__gauge_inc", ATTR_NO_SCRIPT_SIDE_EFFECTS},
{"Telemetry::__gauge_metric_get_or_add", ATTR_NO_SCRIPT_SIDE_EFFECTS},
{"Telemetry::__gauge_value", ATTR_NO_SCRIPT_SIDE_EFFECTS},
{"Telemetry::__histogram_family", ATTR_NO_SCRIPT_SIDE_EFFECTS},
{"Telemetry::__histogram_metric_get_or_add", ATTR_NO_SCRIPT_SIDE_EFFECTS},
{"Telemetry::__histogram_observe", ATTR_NO_SCRIPT_SIDE_EFFECTS},
{"Telemetry::__histogram_sum", ATTR_NO_SCRIPT_SIDE_EFFECTS},
{"WebSocket::__configure_analyzer", ATTR_NO_SCRIPT_SIDE_EFFECTS},
{"__init_primary_bifs", ATTR_NO_SCRIPT_SIDE_EFFECTS},
{"__init_secondary_bifs", ATTR_NO_SCRIPT_SIDE_EFFECTS},

38
src/session/Manager.cc
View file

@ -32,23 +32,23 @@ namespace detail {
class ProtocolStats {
public:
struct Protocol {
telemetry::IntGauge active;
telemetry::IntCounter total;
std::shared_ptr<telemetry::Gauge> active;
std::shared_ptr<telemetry::Counter> total;
ssize_t max = 0;
Protocol(telemetry::IntGaugeFamily active_family, telemetry::IntCounterFamily total_family,
std::string protocol)
: active(active_family.GetOrAdd({{"protocol", protocol}})),
total(total_family.GetOrAdd({{"protocol", protocol}})) {}
Protocol(const std::shared_ptr<telemetry::GaugeFamily>& active_family,
const std::shared_ptr<telemetry::CounterFamily>& total_family, std::string protocol)
: active(active_family->GetOrAdd({{"protocol", protocol}})),
total(total_family->GetOrAdd({{"protocol", protocol}})) {}
};
using ProtocolMap = std::map<std::string, Protocol>;
ProtocolMap::iterator InitCounters(const std::string& protocol) {
telemetry::IntGaugeFamily active_family =
auto active_family =
telemetry_mgr->GaugeFamily("zeek", "active-sessions", {"protocol"}, "Active Zeek Sessions");
telemetry::IntCounterFamily total_family =
telemetry_mgr->CounterFamily("zeek", "total-sessions", {"protocol"}, "Total number of sessions", "1", true);
auto total_family =
telemetry_mgr->CounterFamily("zeek", "total-sessions", {"protocol"}, "Total number of sessions");
auto [it, inserted] = entries.insert({protocol, Protocol{active_family, total_family, protocol}});
@ -116,7 +116,7 @@ void Manager::Remove(Session* s) {
else {
Connection* c = static_cast<Connection*>(s);
if ( auto* stat_block = stats->GetCounters(c->TransportIdentifier()) )
stat_block->active.Dec();
stat_block->active->Dec();
}
// Mark that the session isn't in the table so that in case the
@ -190,18 +190,18 @@ void Manager::Clear() {
void Manager::GetStats(Stats& s) {
auto* tcp_stats = stats->GetCounters("tcp");
s.max_TCP_conns = tcp_stats->max;
s.num_TCP_conns = tcp_stats->active.Value();
s.cumulative_TCP_conns = tcp_stats->total.Value();
s.num_TCP_conns = tcp_stats->active->Value();
s.cumulative_TCP_conns = tcp_stats->total->Value();
auto* udp_stats = stats->GetCounters("udp");
s.max_UDP_conns = udp_stats->max;
s.num_UDP_conns = udp_stats->active.Value();
s.cumulative_UDP_conns = udp_stats->total.Value();
s.num_UDP_conns = udp_stats->active->Value();
s.cumulative_UDP_conns = udp_stats->total->Value();
auto* icmp_stats = stats->GetCounters("icmp");
s.max_ICMP_conns = icmp_stats->max;
s.num_ICMP_conns = icmp_stats->active.Value();
s.cumulative_ICMP_conns = icmp_stats->total.Value();
s.num_ICMP_conns = icmp_stats->active->Value();
s.cumulative_ICMP_conns = icmp_stats->total->Value();
s.num_fragments = zeek::detail::fragment_mgr->Size();
s.max_fragments = zeek::detail::fragment_mgr->MaxFragments();
@ -238,10 +238,10 @@ void Manager::InsertSession(detail::Key key, Session* session) {
std::string protocol = session->TransportIdentifier();
if ( auto* stat_block = stats->GetCounters(protocol) ) {
stat_block->active.Inc();
stat_block->total.Inc();
stat_block->active->Inc();
stat_block->total->Inc();
if ( stat_block->active.Value() > stat_block->max )
if ( stat_block->active->Value() > stat_block->max )
stat_block->max++;
}
}

19
src/telemetry/CMakeLists.txt
View file

@ -1 +1,18 @@
zeek_add_subdir_library(telemetry SOURCES Manager.cc BIFS telemetry.bif)
zeek_add_subdir_library(
telemetry
SOURCES
Counter.cc
Gauge.cc
Histogram.cc
Manager.cc
Opaques.cc
ProcessStats.cc
Utils.cc
BIFS
telemetry.bif)
# We don't need to include the civetweb headers across the whole project, only
# here in the telemetry framework.
target_include_directories(
zeek_telemetry_obj BEFORE
PUBLIC ${PROJECT_SOURCE_DIR}/auxil/prometheus-cpp/3rdparty/civetweb/include)

41
src/telemetry/Counter.cc Normal file
View file

@ -0,0 +1,41 @@
#include "zeek/telemetry/Counter.h"
using namespace zeek::telemetry;
Counter::Counter(FamilyType* family, const prometheus::Labels& labels, prometheus::CollectCallbackPtr callback) noexcept
: handle(family->Add(labels)), labels(labels) {
if ( callback ) {
handle.AddCollectCallback(callback);
has_callback = true;
}
}
double Counter::Value() const noexcept {
if ( has_callback ) {
// Use Collect() here instead of Value() to correctly handle metrics with
// callbacks.
auto metric = handle.Collect();
return metric.counter.value;
}
return handle.Value();
}
std::shared_ptr<Counter> CounterFamily::GetOrAdd(Span<const LabelView> labels,
prometheus::CollectCallbackPtr callback) {
prometheus::Labels p_labels = detail::BuildPrometheusLabels(labels);
auto check = [&](const std::shared_ptr<Counter>& counter) { return counter->CompareLabels(p_labels); };
if ( auto it = std::find_if(counters.begin(), counters.end(), check); it != counters.end() )
return *it;
auto counter = std::make_shared<Counter>(family, p_labels, callback);
counters.push_back(counter);
return counter;
}
std::shared_ptr<Counter> CounterFamily::GetOrAdd(std::initializer_list<LabelView> labels,
prometheus::CollectCallbackPtr callback) {
return GetOrAdd(Span{labels.begin(), labels.size()}, callback);
}

199
src/telemetry/Counter.h
View file

@ -2,215 +2,92 @@
#pragma once
#include <prometheus/counter.h>
#include <prometheus/family.h>
#include <cstdint>
#include <initializer_list>
#include <type_traits>
#include <memory>
#include "zeek/Span.h"
#include "zeek/telemetry/MetricFamily.h"
#include "broker/telemetry/fwd.hh"
#include "zeek/telemetry/Utils.h"
#include "zeek/telemetry/telemetry.bif.h"
namespace zeek::telemetry {
class DblCounterFamily;
class IntCounterFamily;
class Manager;
/**
* A handle to a metric that represents an integer value that can only go up.
* A handle to a metric that can only go up.
*/
class IntCounter {
class Counter {
public:
friend class IntCounterFamily;
static inline const char* OpaqueName = "CounterMetricVal";
static inline const char* OpaqueName = "IntCounterMetricVal";
using Handle = prometheus::Counter;
using FamilyType = prometheus::Family<Handle>;
IntCounter() = delete;
IntCounter(const IntCounter&) noexcept = default;
IntCounter& operator=(const IntCounter&) noexcept = default;
explicit Counter(FamilyType* family, const prometheus::Labels& labels,
prometheus::CollectCallbackPtr callback = nullptr) noexcept;
/**
* Increments the value by 1.
*/
void Inc() noexcept { broker::telemetry::inc(hdl); }
void Inc() noexcept { Inc(1); }
/**
* Increments the value by @p amount.
* @pre `amount >= 0`
*/
void Inc(int64_t amount) noexcept { broker::telemetry::inc(hdl, amount); }
void Inc(double amount) noexcept { handle.Increment(amount); }
/**
* Increments the value by 1.
* @return The new value.
*/
int64_t operator++() noexcept { return broker::telemetry::inc(hdl); }
double operator++() noexcept {
Inc(1);
return Value();
}
/**
* @return The current value.
*/
int64_t Value() const noexcept { return broker::telemetry::value(hdl); }
double Value() const noexcept;
/**
* @return Whether @c this and @p other refer to the same counter.
*/
constexpr bool IsSameAs(const IntCounter& other) const noexcept { return hdl == other.hdl; }
bool operator==(const Counter& rhs) const noexcept { return &handle == &rhs.handle; }
bool operator!=(const Counter& rhs) const noexcept { return &handle != &rhs.handle; }
bool CompareLabels(const prometheus::Labels& lbls) const { return labels == lbls; }
private:
using Handle = broker::telemetry::int_counter_hdl*;
explicit IntCounter(Handle hdl) noexcept : hdl(hdl) {}
Handle hdl;
Handle& handle;
prometheus::Labels labels;
bool has_callback = false;
};
/**
* Checks whether two @ref IntCounter handles are identical.
* @return Whether @p lhs and @p rhs refer to the same object.
* @note compare their @c value instead to check for equality.
*/
constexpr bool operator==(const IntCounter& lhs, const IntCounter& rhs) noexcept { return lhs.IsSameAs(rhs); }
using CounterPtr = std::shared_ptr<Counter>;
/// @relates IntCounter
constexpr bool operator!=(const IntCounter& lhs, const IntCounter& rhs) noexcept { return ! (lhs == rhs); }
/**
* Manages a collection of IntCounter metrics.
*/
class IntCounterFamily : public MetricFamily {
class CounterFamily : public MetricFamily, public std::enable_shared_from_this<CounterFamily> {
public:
friend class Manager;
static inline const char* OpaqueName = "CounterMetricFamilyVal";
static inline const char* OpaqueName = "IntCounterMetricFamilyVal";
using InstanceType = IntCounter;
IntCounterFamily(const IntCounterFamily&) noexcept = default;
IntCounterFamily& operator=(const IntCounterFamily&) noexcept = default;
CounterFamily(prometheus::Family<prometheus::Counter>* family, Span<const std::string_view> labels)
: MetricFamily(labels), family(family) {}
/**
* Returns the metrics handle for given labels, creating a new instance
* lazily if necessary.
*/
IntCounter GetOrAdd(Span<const LabelView> labels) { return IntCounter{int_counter_get_or_add(hdl, labels)}; }
CounterPtr GetOrAdd(Span<const LabelView> labels, prometheus::CollectCallbackPtr callback = nullptr);
/**
* @copydoc GetOrAdd
*/
IntCounter GetOrAdd(std::initializer_list<LabelView> labels) {
return GetOrAdd(Span{labels.begin(), labels.size()});
}
CounterPtr GetOrAdd(std::initializer_list<LabelView> labels, prometheus::CollectCallbackPtr callback = nullptr);
zeek_int_t MetricType() const noexcept override { return BifEnum::Telemetry::MetricType::COUNTER; }
private:
using Handle = broker::telemetry::int_counter_family_hdl*;
explicit IntCounterFamily(Handle hdl) : MetricFamily(upcast(hdl)) {}
prometheus::Family<prometheus::Counter>* family;
std::vector<CounterPtr> counters;
};
/**
* A handle to a metric that represents a floating point value that can only go
* up.
*/
class DblCounter {
public:
friend class DblCounterFamily;
static inline const char* OpaqueName = "DblCounterMetricVal";
DblCounter() = delete;
DblCounter(const DblCounter&) noexcept = default;
DblCounter& operator=(const DblCounter&) noexcept = default;
/**
* Increments the value by 1.
*/
void Inc() noexcept { broker::telemetry::inc(hdl); }
/**
* Increments the value by @p amount.
* @pre `amount >= 0`
*/
void Inc(double amount) noexcept { broker::telemetry::inc(hdl, amount); }
/**
* @return The current value.
*/
double Value() const noexcept { return broker::telemetry::value(hdl); }
/**
* @return Whether @c this and @p other refer to the same counter.
*/
constexpr bool IsSameAs(const DblCounter& other) const noexcept { return hdl == other.hdl; }
private:
using Handle = broker::telemetry::dbl_counter_hdl*;
explicit DblCounter(Handle hdl) noexcept : hdl(hdl) {}
Handle hdl;
};
/**
* Checks whether two @ref DblCounter handles are identical.
* @return Whether @p lhs and @p rhs refer to the same object.
* @note compare their @c value instead to check for equality.
*/
constexpr bool operator==(const DblCounter& lhs, const DblCounter& rhs) noexcept { return lhs.IsSameAs(rhs); }
/// @relates DblCounter
constexpr bool operator!=(const DblCounter& lhs, const DblCounter& rhs) noexcept { return ! (lhs == rhs); }
/**
* Manages a collection of DblCounter metrics.
*/
class DblCounterFamily : public MetricFamily {
public:
friend class Manager;
static inline const char* OpaqueName = "DblCounterMetricFamilyVal";
using InstanceType = DblCounter;
DblCounterFamily(const DblCounterFamily&) noexcept = default;
DblCounterFamily& operator=(const DblCounterFamily&) noexcept = default;
/**
* Returns the metrics handle for given labels, creating a new instance
* lazily if necessary.
*/
DblCounter GetOrAdd(Span<const LabelView> labels) { return DblCounter{dbl_counter_get_or_add(hdl, labels)}; }
/**
* @copydoc GetOrAdd
*/
DblCounter GetOrAdd(std::initializer_list<LabelView> labels) {
return GetOrAdd(Span{labels.begin(), labels.size()});
}
private:
using Handle = broker::telemetry::dbl_counter_family_hdl*;
explicit DblCounterFamily(Handle hdl) : MetricFamily(upcast(hdl)) {}
};
namespace detail {
template<class T>
struct CounterOracle {
static_assert(std::is_same<T, int64_t>::value, "Counter<T> only supports int64_t and double");
using type = IntCounter;
};
template<>
struct CounterOracle<double> {
using type = DblCounter;
};
} // namespace detail
template<class T>
using Counter = typename detail::CounterOracle<T>::type;
using CounterFamilyPtr = std::shared_ptr<CounterFamily>;
} // namespace zeek::telemetry

41
src/telemetry/Gauge.cc Normal file
View file

@ -0,0 +1,41 @@
#include "zeek/telemetry/Gauge.h"
using namespace zeek::telemetry;
double Gauge::Value() const noexcept {
if ( has_callback ) {
// Use Collect() here instead of Value() to correctly handle metrics
// with callbacks.
auto metric = handle.Collect();
return metric.gauge.value;
}
return handle.Value();
}
Gauge::Gauge(FamilyType* family, const prometheus::Labels& labels, prometheus::CollectCallbackPtr callback) noexcept
: handle(family->Add(labels)), labels(labels) {
if ( callback ) {
handle.AddCollectCallback(callback);
has_callback = true;
}
}
std::shared_ptr<Gauge> GaugeFamily::GetOrAdd(Span<const LabelView> labels, prometheus::CollectCallbackPtr callback) {
prometheus::Labels p_labels = detail::BuildPrometheusLabels(labels);
auto check = [&](const std::shared_ptr<Gauge>& gauge) { return gauge->CompareLabels(p_labels); };
if ( auto it = std::find_if(gauges.begin(), gauges.end(), check); it != gauges.end() )
return *it;
auto gauge = std::make_shared<Gauge>(family, p_labels, callback);
gauges.push_back(gauge);
return gauge;
}
std::shared_ptr<Gauge> GaugeFamily::GetOrAdd(std::initializer_list<LabelView> labels,
prometheus::CollectCallbackPtr callback) {
return GetOrAdd(Span{labels.begin(), labels.size()}, callback);
}

216
src/telemetry/Gauge.h
View file

@ -2,236 +2,110 @@
#pragma once
#include <prometheus/family.h>
#include <prometheus/gauge.h>
#include <cstdint>
#include <initializer_list>
#include <type_traits>
#include <memory>
#include "zeek/Span.h"
#include "zeek/telemetry/MetricFamily.h"
#include "broker/telemetry/fwd.hh"
#include "zeek/telemetry/Utils.h"
#include "zeek/telemetry/telemetry.bif.h"
namespace zeek::telemetry {
class DblGaugeFamily;
class IntGaugeFamily;
class Manager;
/**
* A handle to a metric that represents an integer value. Gauges are more
* permissive than counters and also allow decrementing the value.
* A handle to a metric that can count up and down.
*/
class IntGauge {
class Gauge {
public:
friend class IntGaugeFamily;
static inline const char* OpaqueName = "GaugeMetricVal";
static inline const char* OpaqueName = "IntGaugeMetricVal";
using Handle = prometheus::Gauge;
using FamilyType = prometheus::Family<Handle>;
IntGauge() = delete;
IntGauge(const IntGauge&) noexcept = default;
IntGauge& operator=(const IntGauge&) noexcept = default;
explicit Gauge(FamilyType* family, const prometheus::Labels& labels,
prometheus::CollectCallbackPtr callback = nullptr) noexcept;
/**
* Increments the value by 1.
*/
void Inc() noexcept { broker::telemetry::inc(hdl); }
void Inc() noexcept { Inc(1); }
/**
* Increments the value by @p amount.
*/
void Inc(int64_t amount) noexcept { broker::telemetry::inc(hdl, amount); }
void Inc(double amount) noexcept { handle.Increment(amount); }
/**
* Increments the value by 1.
* @return The new value.
*/
int64_t operator++() noexcept { return broker::telemetry::inc(hdl); }
double operator++() noexcept {
Inc(1);
return Value();
}
/**
* Decrements the value by 1.
*/
void Dec() noexcept { broker::telemetry::dec(hdl); }
void Dec() noexcept { Dec(1); }
/**
* Decrements the value by @p amount.
*/
void Dec(int64_t amount) noexcept { broker::telemetry::dec(hdl, amount); }
void Dec(double amount) noexcept { handle.Decrement(amount); }
/**
* Decrements the value by 1.
* @return The new value.
*/
int64_t operator--() noexcept { return broker::telemetry::dec(hdl); }
double operator--() noexcept {
Dec(1);
return Value();
}
/**
* @return The current value.
*/
int64_t Value() const noexcept { return broker::telemetry::value(hdl); }
double Value() const noexcept;
/**
* @return Whether @c this and @p other refer to the same counter.
*/
constexpr bool IsSameAs(const IntGauge& other) const noexcept { return hdl == other.hdl; }
bool operator==(const Gauge& rhs) const noexcept { return &handle == &rhs.handle; }
bool operator!=(const Gauge& rhs) const noexcept { return &handle != &rhs.handle; }
bool CompareLabels(const prometheus::Labels& lbls) const { return labels == lbls; }
private:
using Handle = broker::telemetry::int_gauge_hdl*;
explicit IntGauge(Handle hdl) noexcept : hdl(hdl) {}
Handle hdl;
Handle& handle;
prometheus::Labels labels;
bool has_callback = false;
};
/**
* Checks whether two @ref IntGauge handles are identical.
* @return Whether @p lhs and @p rhs refer to the same object.
* @note compare their @c value instead to check for equality.
*/
constexpr bool operator==(const IntGauge& lhs, const IntGauge& rhs) noexcept { return lhs.IsSameAs(rhs); }
using GaugePtr = std::shared_ptr<Gauge>;
/// @relates IntGauge
constexpr bool operator!=(const IntGauge& lhs, const IntGauge& rhs) noexcept { return ! (lhs == rhs); }
/**
* Manages a collection of IntGauge metrics.
*/
class IntGaugeFamily : public MetricFamily {
class GaugeFamily : public MetricFamily, public std::enable_shared_from_this<GaugeFamily> {
public:
friend class Manager;
static inline const char* OpaqueName = "IntGaugeMetricFamilyVal";
using InstanceType = IntGauge;
IntGaugeFamily(const IntGaugeFamily&) noexcept = default;
IntGaugeFamily& operator=(const IntGaugeFamily&) noexcept = default;
static inline const char* OpaqueName = "GaugeMetricFamilyVal";
/**
* Returns the metrics handle for given labels, creating a new instance
* lazily if necessary.
*/
IntGauge GetOrAdd(Span<const LabelView> labels) { return IntGauge{int_gauge_get_or_add(hdl, labels)}; }
GaugePtr GetOrAdd(Span<const LabelView> labels, prometheus::CollectCallbackPtr callback = nullptr);
/**
* @copydoc GetOrAdd
*/
IntGauge GetOrAdd(std::initializer_list<LabelView> labels) { return GetOrAdd(Span{labels.begin(), labels.size()}); }
GaugePtr GetOrAdd(std::initializer_list<LabelView> labels, prometheus::CollectCallbackPtr callback = nullptr);
zeek_int_t MetricType() const noexcept override { return BifEnum::Telemetry::MetricType::GAUGE; }
GaugeFamily(prometheus::Family<prometheus::Gauge>* family, Span<const std::string_view> labels)
: MetricFamily(labels), family(family) {}
private:
using Handle = broker::telemetry::int_gauge_family_hdl*;
explicit IntGaugeFamily(Handle hdl) : MetricFamily(upcast(hdl)) {}
prometheus::Family<prometheus::Gauge>* family;
std::vector<GaugePtr> gauges;
};
/**
* A handle to a metric that represents a floating point value. Gauges are more
* permissive than counters and also allow decrementing the value.
*/
class DblGauge {
public:
friend class DblGaugeFamily;
static inline const char* OpaqueName = "DblGaugeMetricVal";
DblGauge() = delete;
DblGauge(const DblGauge&) noexcept = default;
DblGauge& operator=(const DblGauge&) noexcept = default;
/**
* Increments the value by 1.
*/
void Inc() noexcept { broker::telemetry::inc(hdl); }
/**
* Increments the value by @p amount.
*/
void Inc(double amount) noexcept { broker::telemetry::inc(hdl, amount); }
/**
* Increments the value by 1.
*/
void Dec() noexcept { broker::telemetry::dec(hdl); }
/**
* Increments the value by @p amount.
*/
void Dec(double amount) noexcept { broker::telemetry::dec(hdl, amount); }
/**
* @return The current value.
*/
double Value() const noexcept { return broker::telemetry::value(hdl); }
/**
* @return Whether @c this and @p other refer to the same counter.
*/
constexpr bool IsSameAs(const DblGauge& other) const noexcept { return hdl == other.hdl; }
private:
using Handle = broker::telemetry::dbl_gauge_hdl*;
explicit DblGauge(Handle hdl) noexcept : hdl(hdl) {}
Handle hdl;
};
/**
* Checks whether two @ref DblGauge handles are identical.
* @return Whether @p lhs and @p rhs refer to the same object.
* @note compare their @c value instead to check for equality.
*/
constexpr bool operator==(const DblGauge& lhs, const DblGauge& rhs) noexcept { return lhs.IsSameAs(rhs); }
/// @relates DblGauge
constexpr bool operator!=(const DblGauge& lhs, const DblGauge& rhs) noexcept { return ! (lhs == rhs); }
/**
* Manages a collection of DblGauge metrics.
*/
class DblGaugeFamily : public MetricFamily {
public:
friend class Manager;
static inline const char* OpaqueName = "DblGaugeMetricFamilyVal";
using InstanceType = DblGauge;
DblGaugeFamily(const DblGaugeFamily&) noexcept = default;
DblGaugeFamily& operator=(const DblGaugeFamily&) noexcept = default;
/**
* Returns the metrics handle for given labels, creating a new instance
* lazily if necessary.
*/
DblGauge GetOrAdd(Span<const LabelView> labels) { return DblGauge{dbl_gauge_get_or_add(hdl, labels)}; }
/**
* @copydoc GetOrAdd
*/
DblGauge GetOrAdd(std::initializer_list<LabelView> labels) { return GetOrAdd(Span{labels.begin(), labels.size()}); }
private:
using Handle = broker::telemetry::dbl_gauge_family_hdl*;
explicit DblGaugeFamily(Handle hdl) : MetricFamily(upcast(hdl)) {}
};
namespace detail {
template<class T>
struct GaugeOracle {
static_assert(std::is_same<T, int64_t>::value, "Gauge<T> only supports int64_t and double");
using type = IntGauge;
};
template<>
struct GaugeOracle<double> {
using type = DblGauge;
};
} // namespace detail
template<class T>
using Gauge = typename detail::GaugeOracle<T>::type;
using GaugeFamilyPtr = std::shared_ptr<GaugeFamily>;
} // namespace zeek::telemetry

38
src/telemetry/Histogram.cc Normal file
View file

@ -0,0 +1,38 @@
#include "zeek/telemetry/Histogram.h"
using namespace zeek::telemetry;
double Histogram::Sum() const noexcept {
auto metric = handle.Collect();
return static_cast<double>(metric.histogram.sample_sum);
}
Histogram::Histogram(FamilyType* family, const prometheus::Labels& labels,
prometheus::Histogram::BucketBoundaries bounds) noexcept
: handle(family->Add(labels, std::move(bounds))), labels(labels) {}
std::shared_ptr<Histogram> HistogramFamily::GetOrAdd(Span<const LabelView> labels) {
prometheus::Labels p_labels = detail::BuildPrometheusLabels(labels);
auto check = [&](const std::shared_ptr<Histogram>& histo) { return histo->CompareLabels(p_labels); };
if ( auto it = std::find_if(histograms.begin(), histograms.end(), check); it != histograms.end() )
return *it;
auto histogram = std::make_shared<Histogram>(family, p_labels, boundaries);
histograms.push_back(histogram);
return histogram;
}
/**
* @copydoc GetOrAdd
*/
std::shared_ptr<Histogram> HistogramFamily::GetOrAdd(std::initializer_list<LabelView> labels) {
return GetOrAdd(Span{labels.begin(), labels.size()});
}
HistogramFamily::HistogramFamily(prometheus::Family<prometheus::Histogram>* family, Span<const double> bounds,
Span<const std::string_view> labels)
: MetricFamily(labels), family(family) {
std::copy(bounds.begin(), bounds.end(), std::back_inserter(boundaries));
}

204
src/telemetry/Histogram.h
View file

@ -2,218 +2,76 @@
#pragma once
#include <prometheus/family.h>
#include <prometheus/histogram.h>
#include <cstdint>
#include <initializer_list>
#include <type_traits>
#include <memory>
#include "zeek/Span.h"
#include "zeek/telemetry/MetricFamily.h"
#include "broker/telemetry/fwd.hh"
#include "zeek/telemetry/Utils.h"
#include "zeek/telemetry/telemetry.bif.h"
namespace zeek::telemetry {
class DblHistogramFamily;
class IntHistogramFamily;
class Manager;
/**
* A handle to a metric that represents an aggregable distribution of observed
* measurements with integer precision. Sorts individual measurements into
* configurable buckets.
*/
class IntHistogram {
class Histogram {
public:
friend class IntHistogramFamily;
static inline const char* OpaqueName = "HistogramMetricVal";
static inline const char* OpaqueName = "IntHistogramMetricVal";
using Handle = prometheus::Histogram;
using FamilyType = prometheus::Family<Handle>;
IntHistogram() = delete;
IntHistogram(const IntHistogram&) noexcept = default;
IntHistogram& operator=(const IntHistogram&) noexcept = default;
explicit Histogram(FamilyType* family, const prometheus::Labels& labels,
prometheus::Histogram::BucketBoundaries bounds) noexcept;
/**
* Increments all buckets with an upper bound less than or equal to @p value
* by one and adds @p value to the total sum of all observed values.
*/
void Observe(int64_t value) noexcept { return broker::telemetry::observe(hdl, value); }
void Observe(double value) noexcept { handle.Observe(value); }
/// @return The sum of all observed values.
int64_t Sum() const noexcept { return broker::telemetry::sum(hdl); }
double Sum() const noexcept;
/// @return The number of buckets, including the implicit "infinite" bucket.
size_t NumBuckets() const noexcept { return broker::telemetry::num_buckets(hdl); }
bool operator==(const Histogram& rhs) const noexcept { return &handle == &rhs.handle; }
bool operator!=(const Histogram& rhs) const noexcept { return &handle != &rhs.handle; }
/// @return The number of observations in the bucket at @p index.
/// @pre index < NumBuckets()
int64_t CountAt(size_t index) const noexcept { return broker::telemetry::count_at(hdl, index); }
/// @return The upper bound of the bucket at @p index.
/// @pre index < NumBuckets()
int64_t UpperBoundAt(size_t index) const noexcept { return broker::telemetry::upper_bound_at(hdl, index); }
/**
* @return Whether @c this and @p other refer to the same histogram.
*/
constexpr bool IsSameAs(const IntHistogram& other) const noexcept { return hdl == other.hdl; }
bool CompareLabels(const prometheus::Labels& lbls) const { return labels == lbls; }
private:
using Handle = broker::telemetry::int_histogram_hdl*;
explicit IntHistogram(Handle hdl) noexcept : hdl(hdl) {}
Handle hdl;
Handle& handle;
prometheus::Labels labels;
};
/**
* Checks whether two @ref IntHistogram handles are identical.
* @return Whether @p lhs and @p rhs refer to the same object.
*/
constexpr bool operator==(const IntHistogram& lhs, const IntHistogram& rhs) noexcept { return lhs.IsSameAs(rhs); }
using HistogramPtr = std::shared_ptr<Histogram>;
/// @relates IntHistogram
constexpr bool operator!=(const IntHistogram& lhs, const IntHistogram& rhs) noexcept { return ! (lhs == rhs); }
/**
* Manages a collection of IntHistogram metrics.
*/
class IntHistogramFamily : public MetricFamily {
class HistogramFamily : public MetricFamily, public std::enable_shared_from_this<HistogramFamily> {
public:
friend class Manager;
static inline const char* OpaqueName = "HistogramMetricFamilyVal";
static inline const char* OpaqueName = "IntHistogramMetricFamilyVal";
using InstanceType = IntHistogram;
IntHistogramFamily(const IntHistogramFamily&) noexcept = default;
IntHistogramFamily& operator=(const IntHistogramFamily&) noexcept = default;
HistogramFamily(prometheus::Family<prometheus::Histogram>* family, Span<const double> bounds,
Span<const std::string_view> labels);
/**
* Returns the metrics handle for given labels, creating a new instance
* lazily if necessary.
*/
IntHistogram GetOrAdd(Span<const LabelView> labels) { return IntHistogram{int_histogram_get_or_add(hdl, labels)}; }
HistogramPtr GetOrAdd(Span<const LabelView> labels);
/**
* @copydoc GetOrAdd
*/
IntHistogram GetOrAdd(std::initializer_list<LabelView> labels) {
return GetOrAdd(Span{labels.begin(), labels.size()});
}
HistogramPtr GetOrAdd(std::initializer_list<LabelView> labels);
zeek_int_t MetricType() const noexcept override { return BifEnum::Telemetry::MetricType::HISTOGRAM; }
private:
using Handle = broker::telemetry::int_histogram_family_hdl*;
explicit IntHistogramFamily(Handle hdl) : MetricFamily(upcast(hdl)) {}
prometheus::Family<prometheus::Histogram>* family;
prometheus::Histogram::BucketBoundaries boundaries;
std::vector<HistogramPtr> histograms;
};
/**
* A handle to a metric that represents an aggregable distribution of observed
* measurements with floating point precision. Sorts individual measurements
* into configurable buckets.
*/
class DblHistogram {
public:
friend class DblHistogramFamily;
static inline const char* OpaqueName = "DblHistogramMetricVal";
DblHistogram() = delete;
DblHistogram(const DblHistogram&) noexcept = default;
DblHistogram& operator=(const DblHistogram&) noexcept = default;
/**
* Increments all buckets with an upper bound less than or equal to @p value
* by one and adds @p value to the total sum of all observed values.
*/
void Observe(double value) noexcept { broker::telemetry::observe(hdl, value); }
/// @return The sum of all observed values.
double Sum() const noexcept { return broker::telemetry::sum(hdl); }
/// @return The number of buckets, including the implicit "infinite" bucket.
size_t NumBuckets() const noexcept { return broker::telemetry::num_buckets(hdl); }
/// @return The number of observations in the bucket at @p index.
/// @pre index < NumBuckets()
int64_t CountAt(size_t index) const noexcept { return broker::telemetry::count_at(hdl, index); }
/// @return The upper bound of the bucket at @p index.
/// @pre index < NumBuckets()
double UpperBoundAt(size_t index) const noexcept { return broker::telemetry::upper_bound_at(hdl, index); }
/**
* @return Whether @c this and @p other refer to the same histogram.
*/
constexpr bool IsSameAs(const DblHistogram& other) const noexcept { return hdl == other.hdl; }
private:
using Handle = broker::telemetry::dbl_histogram_hdl*;
explicit DblHistogram(Handle hdl) noexcept : hdl(hdl) {}
Handle hdl;
};
/**
* Checks whether two @ref DblHistogram handles are identical.
* @return Whether @p lhs and @p rhs refer to the same object.
*/
constexpr bool operator==(const DblHistogram& lhs, const DblHistogram& rhs) noexcept { return lhs.IsSameAs(rhs); }
/// @relates DblHistogram
constexpr bool operator!=(const DblHistogram& lhs, const DblHistogram& rhs) noexcept { return ! (lhs == rhs); }
/**
* Manages a collection of DblHistogram metrics.
*/
class DblHistogramFamily : public MetricFamily {
public:
friend class Manager;
static inline const char* OpaqueName = "DblHistogramMetricFamilyVal";
using InstanceType = DblHistogram;
DblHistogramFamily(const DblHistogramFamily&) noexcept = default;
DblHistogramFamily& operator=(const DblHistogramFamily&) noexcept = default;
/**
* Returns the metrics handle for given labels, creating a new instance
* lazily if necessary.
*/
DblHistogram GetOrAdd(Span<const LabelView> labels) { return DblHistogram{dbl_histogram_get_or_add(hdl, labels)}; }
/**
* @copydoc GetOrAdd
*/
DblHistogram GetOrAdd(std::initializer_list<LabelView> labels) {
return GetOrAdd(Span{labels.begin(), labels.size()});
}
private:
using Handle = broker::telemetry::dbl_histogram_family_hdl*;
explicit DblHistogramFamily(Handle hdl) : MetricFamily(upcast(hdl)) {}
};
namespace detail {
template<class T>
struct HistogramOracle {
static_assert(std::is_same<T, int64_t>::value, "Histogram<T> only supports int64_t and double");
using type = IntHistogram;
};
template<>
struct HistogramOracle<double> {
using type = DblHistogram;
};
} // namespace detail
template<class T>
using Histogram = typename detail::HistogramOracle<T>::type;
using HistogramFamilyPtr = std::shared_ptr<HistogramFamily>;
} // namespace zeek::telemetry

893
src/telemetry/Manager.cc
View file

File diff suppressed because it is too large Load diff

341
src/telemetry/Manager.h
View file

@ -2,11 +2,13 @@
#pragma once
#include <prometheus/exposer.h>
#include <prometheus/registry.h>
#include <condition_variable>
#include <cstdint>
#include <initializer_list>
#include <memory>
#include <string_view>
#include <unordered_map>
#include <variant>
#include <vector>
#include "zeek/IntrusivePtr.h"
@ -14,149 +16,50 @@
#include "zeek/telemetry/Counter.h"
#include "zeek/telemetry/Gauge.h"
#include "zeek/telemetry/Histogram.h"
#include "broker/telemetry/fwd.hh"
namespace broker {
class endpoint;
}
#include "zeek/telemetry/ProcessStats.h"
#include "zeek/telemetry/Utils.h"
namespace zeek {
class RecordVal;
using RecordValPtr = IntrusivePtr<RecordVal>;
} // namespace zeek
namespace zeek::Broker {
class Manager;
}
namespace zeek::telemetry {
/**
* Manages a collection of metric families.
*/
class Manager {
class Manager final {
public:
friend class Broker::Manager;
Manager();
Manager(const Manager&) = delete;
Manager& operator=(const Manager&) = delete;
virtual ~Manager() = default;
~Manager() = default;
/**
* Initialization of the manager. This is called late during Zeek's
* initialization after any scripts are processed.
* initialization after any scripts are processed. Sets up the Prometheus
* server and the process stats metrics, and on a manager node will enable
* the Prometheus service discovery endpoint.
*/
virtual void InitPostScript();
void InitPostScript();
/**
* Supported metric types.
*/
enum class MetricType { Counter, Gauge, Histogram };
/**
* Captures information about counter and gauge metrics.
*/
struct CollectedValueMetric {
/**
* Constructor.
* @param metric_type The type of this metric.
* @param family Broker layer family handle for this metric.
* @param label_values The string values for each of the metric's labels.
* @param value The metric's current value.
*/
CollectedValueMetric(MetricType metric_type, const broker::telemetry::metric_family_hdl* family,
std::vector<std::string_view> label_values, std::variant<double, int64_t> value)
: metric_type(metric_type), family(family), label_values(std::move(label_values)), value(value) {}
/**
* @return A script layer Telemetry::Metric record for this metric.
*/
zeek::RecordValPtr AsMetricRecord() const;
enum MetricType metric_type;
const broker::telemetry::metric_family_hdl* family;
std::vector<std::string_view> label_values;
std::variant<double, int64_t> value;
};
/**
* Captures information about histogram metrics.
*/
struct CollectedHistogramMetric {
/**
* Helper struct representing a single bucket of a histogram.
* @tparam T The data type used by the histogram (double or int64_t).
*/
template<class T>
struct Bucket {
Bucket(T count, T upper_bound) : count(count), upper_bound(upper_bound) {}
T count;
T upper_bound;
};
/**
* Helper struct representing a histogram as sum and buckets.
* @tparam T The data type used by the histogram (double or int64_t).
*/
template<class T>
struct HistogramData {
T sum;
std::vector<Bucket<T>> buckets;
};
using DblHistogramData = HistogramData<double>;
using IntHistogramData = HistogramData<int64_t>;
/**
* Constructor.
* @param family Broker layer family handle for this metric.
* @param label_values The string values for each of the metric's labels.
* @param histogram The histogram's data (sum and individual buckets).
*/
CollectedHistogramMetric(const broker::telemetry::metric_family_hdl* family,
std::vector<std::string_view> label_values,
std::variant<DblHistogramData, IntHistogramData> histogram)
: family(family), label_values(std::move(label_values)), histogram(std::move(histogram)) {}
const broker::telemetry::metric_family_hdl* family;
std::vector<std::string_view> label_values;
std::variant<DblHistogramData, IntHistogramData> histogram;
/**
* @return A script layer Telemetry::HistogramMetric record for this histogram.
*/
zeek::RecordValPtr AsHistogramMetricRecord() const;
};
/**
* @return A script layer Telemetry::MetricOpts record for the given metric family.
* @param metric_typ The type of metric.
* @param family Broker layer family handle for the family.
* @tparam T The underlying data type (double or int64_t)
*/
template<typename T>
zeek::RecordValPtr GetMetricOptsRecord(MetricType metric_type, const broker::telemetry::metric_family_hdl* family);
/**
* @return All counter and gauge metrics and their values matching prefix and name.
* @return A VectorVal containing all counter and gauge metrics and their values matching prefix and name.
* @param prefix The prefix pattern to use for filtering. Supports globbing.
* @param name The name pattern to use for filtering. Supports globbing.
*/
std::vector<CollectedValueMetric> CollectMetrics(std::string_view prefix, std::string_view name);
ValPtr CollectMetrics(std::string_view prefix, std::string_view name);
/**
* @return All histogram metrics and their data matching prefix and name.
* @return A VectorVal containing all histogram metrics and their values matching prefix and name.
* @param prefix The prefix pattern to use for filtering. Supports globbing.
* @param name The name pattern to use for filtering. Supports globbing.
*/
std::vector<CollectedHistogramMetric> CollectHistogramMetrics(std::string_view prefix, std::string_view name);
ValPtr CollectHistogramMetrics(std::string_view prefix, std::string_view name);
/**
* @return A counter metric family. Creates the family lazily if necessary.
@ -165,30 +68,14 @@ public:
* @param labels Names for all label dimensions of the metric.
* @param helptext Short explanation of the metric.
* @param unit Unit of measurement.
* @param is_sum Indicates whether this metric accumulates something, where
* only the total value is of interest.
*/
template<class ValueType = int64_t>
auto CounterFamily(std::string_view prefix, std::string_view name, Span<const std::string_view> labels,
std::string_view helptext, std::string_view unit = "1", bool is_sum = false) {
if constexpr ( std::is_same<ValueType, int64_t>::value ) {
auto fam = int_counter_fam(Ptr(), prefix, name, labels, helptext, unit, is_sum);
return IntCounterFamily{fam};
}
else {
static_assert(std::is_same<ValueType, double>::value, "metrics only support int64_t and double values");
auto fam = dbl_counter_fam(Ptr(), prefix, name, labels, helptext, unit, is_sum);
return DblCounterFamily{fam};
}
}
CounterFamilyPtr CounterFamily(std::string_view prefix, std::string_view name, Span<const std::string_view> labels,
std::string_view helptext, std::string_view unit = "");
/// @copydoc CounterFamily
template<class ValueType = int64_t>
auto CounterFamily(std::string_view prefix, std::string_view name, std::initializer_list<std::string_view> labels,
std::string_view helptext, std::string_view unit = "1", bool is_sum = false) {
auto lbl_span = Span{labels.begin(), labels.size()};
return CounterFamily<ValueType>(prefix, name, lbl_span, helptext, unit, is_sum);
}
CounterFamilyPtr CounterFamily(std::string_view prefix, std::string_view name,
std::initializer_list<std::string_view> labels, std::string_view helptext,
std::string_view unit = "");
/**
* Accesses a counter instance. Creates the hosting metric family as well
@ -198,26 +85,17 @@ public:
* @param labels Values for all label dimensions of the metric.
* @param helptext Short explanation of the metric.
* @param unit Unit of measurement.
* @param is_sum Indicates whether this metric accumulates something, where
* only the total value is of interest.
* @param callback Passing a callback method will enable asynchronous mode. The callback method will be called by
* the metrics subsystem whenever data is requested.
*/
template<class ValueType = int64_t>
Counter<ValueType> CounterInstance(std::string_view prefix, std::string_view name, Span<const LabelView> labels,
std::string_view helptext, std::string_view unit = "1", bool is_sum = false) {
return WithLabelNames(labels, [&, this](auto labelNames) {
auto family = CounterFamily<ValueType>(prefix, name, labelNames, helptext, unit, is_sum);
return family.getOrAdd(labels);
});
}
CounterPtr CounterInstance(std::string_view prefix, std::string_view name, Span<const LabelView> labels,
std::string_view helptext, std::string_view unit = "",
prometheus::CollectCallbackPtr callback = nullptr);
/// @copydoc counterInstance
template<class ValueType = int64_t>
Counter<ValueType> CounterInstance(std::string_view prefix, std::string_view name,
std::initializer_list<LabelView> labels, std::string_view helptext,
std::string_view unit = "1", bool is_sum = false) {
auto lbl_span = Span{labels.begin(), labels.size()};
return CounterInstance(prefix, name, lbl_span, helptext, unit, is_sum);
}
CounterPtr CounterInstance(std::string_view prefix, std::string_view name, std::initializer_list<LabelView> labels,
std::string_view helptext, std::string_view unit = "",
prometheus::CollectCallbackPtr callback = nullptr);
/**
* @return A gauge metric family. Creates the family lazily if necessary.
@ -226,30 +104,14 @@ public:
* @param labels Names for all label dimensions of the metric.
* @param helptext Short explanation of the metric.
* @param unit Unit of measurement.
* @param is_sum Indicates whether this metric accumulates something, where
* only the total value is of interest.
*/
template<class ValueType = int64_t>
auto GaugeFamily(std::string_view prefix, std::string_view name, Span<const std::string_view> labels,
std::string_view helptext, std::string_view unit = "1", bool is_sum = false) {
if constexpr ( std::is_same<ValueType, int64_t>::value ) {
auto fam = int_gauge_fam(Ptr(), prefix, name, labels, helptext, unit, is_sum);
return IntGaugeFamily{fam};
}
else {
static_assert(std::is_same<ValueType, double>::value, "metrics only support int64_t and double values");
auto fam = dbl_gauge_fam(Ptr(), prefix, name, labels, helptext, unit, is_sum);
return DblGaugeFamily{fam};
}
}
GaugeFamilyPtr GaugeFamily(std::string_view prefix, std::string_view name, Span<const std::string_view> labels,
std::string_view helptext, std::string_view unit = "");
/// @copydoc GaugeFamily
template<class ValueType = int64_t>
auto GaugeFamily(std::string_view prefix, std::string_view name, std::initializer_list<std::string_view> labels,
std::string_view helptext, std::string_view unit = "1", bool is_sum = false) {
auto lbl_span = Span{labels.begin(), labels.size()};
return GaugeFamily<ValueType>(prefix, name, lbl_span, helptext, unit, is_sum);
}
GaugeFamilyPtr GaugeFamily(std::string_view prefix, std::string_view name,
std::initializer_list<std::string_view> labels, std::string_view helptext,
std::string_view unit = "");
/**
* Accesses a gauge instance. Creates the hosting metric family as well
@ -259,26 +121,17 @@ public:
* @param labels Values for all label dimensions of the metric.
* @param helptext Short explanation of the metric.
* @param unit Unit of measurement.
* @param is_sum Indicates whether this metric accumulates something, where
* only the total value is of interest.
* @param callback Passing a callback method will enable asynchronous mode. The callback method will be called by
* the metrics subsystem whenever data is requested.
*/
template<class ValueType = int64_t>
Gauge<ValueType> GaugeInstance(std::string_view prefix, std::string_view name, Span<const LabelView> labels,
std::string_view helptext, std::string_view unit = "1", bool is_sum = false) {
return WithLabelNames(labels, [&, this](auto labelNames) {
auto family = GaugeFamily<ValueType>(prefix, name, labelNames, helptext, unit, is_sum);
return family.getOrAdd(labels);
});
}
GaugePtr GaugeInstance(std::string_view prefix, std::string_view name, Span<const LabelView> labels,
std::string_view helptext, std::string_view unit = "",
prometheus::CollectCallbackPtr callback = nullptr);
/// @copydoc GaugeInstance
template<class ValueType = int64_t>
Gauge<ValueType> GaugeInstance(std::string_view prefix, std::string_view name,
std::initializer_list<LabelView> labels, std::string_view helptext,
std::string_view unit = "1", bool is_sum = false) {
auto lbl_span = Span{labels.begin(), labels.size()};
return GaugeInstance(prefix, name, lbl_span, helptext, unit, is_sum);
}
GaugePtr GaugeInstance(std::string_view prefix, std::string_view name, std::initializer_list<LabelView> labels,
std::string_view helptext, std::string_view unit = "",
prometheus::CollectCallbackPtr callback = nullptr);
// Forces the compiler to use the type `Span<const T>` instead of trying to
// match parameters to a `span`.
@ -300,41 +153,23 @@ public:
* reserved.
* @param name The human-readable name of the metric, e.g., `requests`.
* @param labels Names for all label dimensions of the metric.
* @param default_upper_bounds Upper bounds for the metric buckets.
* @param bounds Upper bounds for the metric buckets.
* @param helptext Short explanation of the metric.
* @param unit Unit of measurement. Please use base units such as `bytes` or
* `seconds` (prefer lowercase). The pseudo-unit `1` identifies
* dimensionless counts.
* @param is_sum Setting this to `true` indicates that this metric adds
* something up to a total, where only the total value is of
* interest. For example, the total number of HTTP requests.
* @note The first call wins when calling this function multiple times with
* different bucket settings. Users may also override
* @p default_upper_bounds via run-time configuration.
* @p bounds via run-time configuration.
*/
template<class ValueType = int64_t>
auto HistogramFamily(std::string_view prefix, std::string_view name, Span<const std::string_view> labels,
ConstSpan<ValueType> default_upper_bounds, std::string_view helptext,
std::string_view unit = "1", bool is_sum = false) {
if constexpr ( std::is_same<ValueType, int64_t>::value ) {
auto fam = int_histogram_fam(Ptr(), prefix, name, labels, default_upper_bounds, helptext, unit, is_sum);
return IntHistogramFamily{fam};
}
else {
static_assert(std::is_same<ValueType, double>::value, "metrics only support int64_t and double values");
auto fam = dbl_histogram_fam(Ptr(), prefix, name, labels, default_upper_bounds, helptext, unit, is_sum);
return DblHistogramFamily{fam};
}
}
HistogramFamilyPtr HistogramFamily(std::string_view prefix, std::string_view name,
Span<const std::string_view> labels, ConstSpan<double> bounds,
std::string_view helptext, std::string_view unit = "");
/// @copydoc HistogramFamily
template<class ValueType = int64_t>
auto HistogramFamily(std::string_view prefix, std::string_view name, std::initializer_list<std::string_view> labels,
ConstSpan<ValueType> default_upper_bounds, std::string_view helptext,
std::string_view unit = "1", bool is_sum = false) {
auto lbl_span = Span{labels.begin(), labels.size()};
return HistogramFamily<ValueType>(prefix, name, lbl_span, default_upper_bounds, helptext, unit, is_sum);
}
HistogramFamilyPtr HistogramFamily(std::string_view prefix, std::string_view name,
std::initializer_list<std::string_view> labels, ConstSpan<double> bounds,
std::string_view helptext, std::string_view unit = "");
/**
* Returns a histogram. Creates the family lazily if necessary.
@ -344,42 +179,39 @@ public:
* reserved.
* @param name The human-readable name of the metric, e.g., `requests`.
* @param labels Names for all label dimensions of the metric.
* @param default_upper_bounds Upper bounds for the metric buckets.
* @param bounds Upper bounds for the metric buckets.
* @param helptext Short explanation of the metric.
* @param unit Unit of measurement. Please use base units such as `bytes` or
* `seconds` (prefer lowercase). The pseudo-unit `1` identifies
* dimensionless counts.
* @param is_sum Setting this to `true` indicates that this metric adds
* something up to a total, where only the total value is of
* interest. For example, the total number of HTTP requests.
* @note The first call wins when calling this function multiple times with
* different bucket settings. Users may also override
* @p default_upper_bounds via run-time configuration.
* @p bounds via run-time configuration.
*/
template<class ValueType = int64_t>
Histogram<ValueType> HistogramInstance(std::string_view prefix, std::string_view name, Span<const LabelView> labels,
ConstSpan<ValueType> default_upper_bounds, std::string_view helptext,
std::string_view unit = "1", bool is_sum = false) {
return WithLabelNames(labels, [&, this](auto labelNames) {
auto family =
HistogramFamily<ValueType>(prefix, name, labelNames, default_upper_bounds, helptext, unit, is_sum);
return family.getOrAdd(labels);
});
}
HistogramPtr HistogramInstance(std::string_view prefix, std::string_view name, Span<const LabelView> labels,
ConstSpan<double> bounds, std::string_view helptext, std::string_view unit = "");
/// @copdoc HistogramInstance
template<class ValueType = int64_t>
Histogram<ValueType> HistogramInstance(std::string_view prefix, std::string_view name,
std::initializer_list<LabelView> labels,
ConstSpan<ValueType> default_upper_bounds, std::string_view helptext,
std::string_view unit = "1", bool is_sum = false) {
auto lbls = Span{labels.begin(), labels.size()};
return HistogramInstance(prefix, name, lbls, default_upper_bounds, helptext, unit, is_sum);
}
HistogramPtr HistogramInstance(std::string_view prefix, std::string_view name,
std::initializer_list<LabelView> labels, std::initializer_list<double> bounds,
std::string_view helptext, std::string_view unit = "");
/**
* @return A JSON description of the cluster configuration for reporting
* to Prometheus for service discovery requests.
*/
std::string GetClusterJson() const;
/**
* @return The pointer to the prometheus-cpp registry used by the telemetry
* manager. This is public so that third parties (such as broker) can add
* elements to it directly.
*/
std::shared_ptr<prometheus::Registry> GetRegistry() const { return prometheus_registry; }
protected:
template<class F>
static void WithLabelNames(Span<const LabelView> xs, F continuation) {
static auto WithLabelNames(Span<const LabelView> xs, F continuation) {
if ( xs.size() <= 10 ) {
std::string_view buf[10];
for ( size_t index = 0; index < xs.size(); ++index )
@ -390,29 +222,36 @@ protected:
else {
std::vector<std::string_view> buf;
for ( auto x : xs )
buf.emplace_back(x.first, x.second);
buf.emplace_back(x.first);
return continuation(Span{buf});
}
}
broker::telemetry::metric_registry_impl* Ptr() { return pimpl.get(); }
// Connects all the dots after the Broker Manager constructed the endpoint
// for this Zeek instance. Called from Broker::Manager::InitPostScript().
void InitPostBrokerSetup(broker::endpoint&);
IntrusivePtr<broker::telemetry::metric_registry_impl> pimpl;
private:
// Caching of metric_family_hdl instances to their Zeek record representation.
std::unordered_map<const broker::telemetry::metric_family_hdl*, zeek::RecordValPtr> metric_opts_cache;
RecordValPtr GetMetricOptsRecord(const prometheus::MetricFamily& metric_family);
std::map<std::string, std::shared_ptr<MetricFamily>> families;
std::map<std::string, RecordValPtr> opts_records;
detail::process_stats current_process_stats;
double process_stats_last_updated = 0.0;
GaugePtr rss_gauge;
GaugePtr vms_gauge;
GaugePtr cpu_gauge;
GaugePtr fds_gauge;
std::string endpoint_name;
std::vector<std::string> export_prefixes;
std::shared_ptr<prometheus::Registry> prometheus_registry;
std::unique_ptr<prometheus::Exposer> prometheus_exposer;
};
} // namespace zeek::telemetry
namespace zeek {
extern telemetry::Manager* telemetry_mgr;
} // namespace zeek

62
src/telemetry/MetricFamily.h
View file

@ -3,74 +3,32 @@
#pragma once
#include <string>
#include <string_view>
#include <utility>
#include <vector>
#include "zeek/Span.h"
#include "broker/telemetry/metric_family.hh"
#include "zeek/util.h"
namespace zeek::telemetry {
/**
* A key-value pair for a single label dimension.
*/
using LabelView = std::pair<std::string_view, std::string_view>;
/**
* Manages a collection (family) of metrics. All members of the family share
* the same prefix (namespace), name, and label dimensions.
*/
class MetricFamily {
public:
MetricFamily() = delete;
MetricFamily(const MetricFamily&) noexcept = default;
MetricFamily& operator=(const MetricFamily&) noexcept = default;
virtual ~MetricFamily() = default;
/**
* @return The prefix (namespace) this family belongs to. Builtin metrics
* of Zeek return @c zeek. Custom metrics, e.g., created in a
* script, may use a prefix that represents the application/script
* or protocol (e.g. @c http) name.
*/
std::string_view Prefix() const noexcept { return broker::telemetry::prefix(hdl); }
virtual zeek_int_t MetricType() const = 0;
/**
* @return The human-readable name of the metric, e.g.,
* @p open-connections.
*/
std::string_view Name() const noexcept { return broker::telemetry::name(hdl); }
/**
* @return The names for all label dimensions.
*/
Span<const std::string> LabelNames() const noexcept { return broker::telemetry::label_names(hdl); }
/**
* @return A short explanation of the metric.
*/
std::string_view Helptext() const noexcept { return broker::telemetry::helptext(hdl); }
/**
* @return The unit of measurement, preferably a base unit such as
* @c bytes or @c seconds. Dimensionless counts return the
* pseudo-unit @c 1.
*/
std::string_view Unit() const noexcept { return broker::telemetry::unit(hdl); }
/**
* @return Whether metrics of this family accumulate values, where only the
* total value is of interest. For example, the total number of
* HTTP requests.
*/
bool IsSum() const noexcept { return broker::telemetry::is_sum(hdl); }
std::vector<std::string> LabelNames() const { return label_names; }
protected:
using Handle = broker::telemetry::metric_family_hdl*;
MetricFamily(Span<const std::string_view> labels) {
for ( const auto& lbl : labels )
label_names.emplace_back(lbl);
}
explicit MetricFamily(Handle hdl) : hdl(hdl) {}
Handle hdl;
std::vector<std::string> label_names;
};
} // namespace zeek::telemetry

22
src/telemetry/Opaques.cc Normal file
View file

@ -0,0 +1,22 @@
#include "Opaques.h"
using namespace zeek;
TelemetryVal::TelemetryVal(std::shared_ptr<telemetry::Counter>) : OpaqueVal(counter_metric_type) {}
TelemetryVal::TelemetryVal(std::shared_ptr<telemetry::CounterFamily>) : OpaqueVal(counter_metric_family_type) {}
TelemetryVal::TelemetryVal(std::shared_ptr<telemetry::Gauge>) : OpaqueVal(gauge_metric_type) {}
TelemetryVal::TelemetryVal(std::shared_ptr<telemetry::GaugeFamily>) : OpaqueVal(gauge_metric_family_type) {}
TelemetryVal::TelemetryVal(std::shared_ptr<telemetry::Histogram>) : OpaqueVal(histogram_metric_type) {}
TelemetryVal::TelemetryVal(std::shared_ptr<telemetry::HistogramFamily>) : OpaqueVal(histogram_metric_family_type) {}
IMPLEMENT_OPAQUE_VALUE(CounterMetricVal)
IMPLEMENT_OPAQUE_VALUE(CounterMetricFamilyVal)
IMPLEMENT_OPAQUE_VALUE(GaugeMetricVal)
IMPLEMENT_OPAQUE_VALUE(GaugeMetricFamilyVal)
IMPLEMENT_OPAQUE_VALUE(HistogramMetricVal)
IMPLEMENT_OPAQUE_VALUE(HistogramMetricFamilyVal)

59
src/telemetry/Opaques.h Normal file
View file

@ -0,0 +1,59 @@
// See the file "COPYING" in the main distribution directory for copyright.
#pragma once
#include <prometheus/counter.h>
#include <prometheus/family.h>
#include "zeek/OpaqueVal.h"
#include "zeek/broker/Data.h"
namespace zeek {
/**
* Base class for metric handles. Handle types are not serializable.
*/
class TelemetryVal : public OpaqueVal {
protected:
explicit TelemetryVal(std::shared_ptr<telemetry::Counter>);
explicit TelemetryVal(std::shared_ptr<telemetry::CounterFamily>);
explicit TelemetryVal(std::shared_ptr<telemetry::Gauge>);
explicit TelemetryVal(std::shared_ptr<telemetry::GaugeFamily>);
explicit TelemetryVal(std::shared_ptr<telemetry::Histogram>);
explicit TelemetryVal(std::shared_ptr<telemetry::HistogramFamily>);
std::optional<BrokerData> DoSerializeData() const override { return std::nullopt; }
bool DoUnserializeData(BrokerDataView data) override { return false; }
};
template<class Handle>
class TelemetryValImpl : public TelemetryVal {
public:
using HandleType = std::shared_ptr<Handle>;
explicit TelemetryValImpl(HandleType hdl) : TelemetryVal(hdl), hdl(hdl) {}
HandleType GetHandle() const noexcept { return hdl; }
static zeek::OpaqueValPtr OpaqueInstantiate() {
reporter->Error("TelemetryValImpl::OpaqueInstantiate is unsupported");
return nullptr;
}
protected:
ValPtr DoClone(CloneState*) override { return make_intrusive<TelemetryValImpl>(hdl); }
const char* OpaqueName() const override { return Handle::OpaqueName; }
private:
HandleType hdl;
};
using CounterMetricVal = TelemetryValImpl<telemetry::Counter>;
using CounterMetricFamilyVal = TelemetryValImpl<telemetry::CounterFamily>;
using GaugeMetricVal = TelemetryValImpl<telemetry::Gauge>;
using GaugeMetricFamilyVal = TelemetryValImpl<telemetry::GaugeFamily>;
using HistogramMetricVal = TelemetryValImpl<telemetry::Histogram>;
using HistogramMetricFamilyVal = TelemetryValImpl<telemetry::HistogramFamily>;
} // namespace zeek

199
src/telemetry/ProcessStats.cc Normal file
View file

@ -0,0 +1,199 @@
#include "zeek/telemetry/ProcessStats.h"
#include "zeek/util.h"
#ifdef __APPLE__
#include <libproc.h>
#include <mach/mach.h>
#include <mach/task.h>
#include <math.h>
#include <stdlib.h>
#include <sys/resource.h>
#include <unistd.h>
namespace zeek::telemetry::detail {
process_stats get_process_stats() {
process_stats result;
// Fetch memory usage.
{
mach_task_basic_info info;
mach_msg_type_number_t count = MACH_TASK_BASIC_INFO_COUNT;
if ( task_info(mach_task_self(), MACH_TASK_BASIC_INFO, reinterpret_cast<task_info_t>(&info), &count) ==
KERN_SUCCESS ) {
result.rss = static_cast<int64_t>(info.resident_size);
result.vms = static_cast<int64_t>(info.virtual_size);
}
}
// Fetch CPU time.
{
task_thread_times_info info;
mach_msg_type_number_t count = TASK_THREAD_TIMES_INFO_COUNT;
if ( task_info(mach_task_self(), TASK_THREAD_TIMES_INFO, reinterpret_cast<task_info_t>(&info), &count) ==
KERN_SUCCESS ) {
// Round to milliseconds.
result.cpu += info.user_time.seconds;
result.cpu += ceil(info.user_time.microseconds / 1000.0) / 1000.0;
result.cpu += info.system_time.seconds;
result.cpu += ceil(info.system_time.microseconds / 1000.0) / 1000.0;
}
}
// Fetch open file handles.
{
// proc_pidinfo is undocumented, but this is what lsof also uses.
auto suggested_buf_size = proc_pidinfo(getpid(), PROC_PIDLISTFDS, 0, nullptr, 0);
if ( suggested_buf_size > 0 ) {
auto buf_size = suggested_buf_size;
auto buf = malloc(buf_size); // TODO: could be thread-local
auto res = proc_pidinfo(getpid(), PROC_PIDLISTFDS, 0, buf, buf_size);
free(buf);
if ( res > 0 )
result.fds = static_cast<int64_t>(res / sizeof(proc_fdinfo));
}
}
return result;
}
} // namespace zeek::telemetry::detail
#elif defined(HAVE_LINUX)
#include <atomic>
std::atomic<long> global_ticks_per_second;
std::atomic<long> global_page_size;
namespace zeek::telemetry::detail {
/// Caches the result from a `sysconf` call in a cache variable to avoid
/// frequent syscalls. Sets `cache_var` to -1 in case of an error. Initially,
/// `cache_var` must be 0 and we assume a successful syscall would always return
/// some value > 0. If `cache_var` is > 0 then this function simply returns the
/// cached value directly.
bool load_system_setting(std::atomic<long>& cache_var, long& var, int name, [[maybe_unused]] const char* pretty_name) {
var = cache_var.load();
switch ( var ) {
case -1: return false;
case 0:
var = sysconf(name);
if ( var <= 0 ) {
var = -1;
cache_var = var;
return false;
}
else {
cache_var = var;
return true;
}
default: return true;
}
}
#define TRY_LOAD(varname, confname) load_system_setting(global_##varname, varname, confname, #confname)
process_stats get_process_stats() {
process_stats result;
long ticks_per_second = 0;
long page_size = 0;
if ( ! TRY_LOAD(ticks_per_second, _SC_CLK_TCK) || ! TRY_LOAD(page_size, _SC_PAGE_SIZE) )
return result;
if ( auto f = fopen("/proc/self/stat", "r") ) {
unsigned long utime_ticks = 0;
unsigned long stime_ticks = 0;
unsigned long vmsize_bytes = 0;
unsigned long rss_pages = 0;
auto rd = fscanf(f,
"%*d " // 1. PID
"%*s " // 2. Executable
"%*c " // 3. State
"%*d " // 4. Parent PID
"%*d " // 5. Process group ID
"%*d " // 6. Session ID
"%*d " // 7. Controlling terminal
"%*d " // 8. Foreground process group ID
"%*u " // 9. Flags
"%*u " // 10. Number of minor faults
"%*u " // 11. Number of minor faults of waited-for children
"%*u " // 12. Number of major faults
"%*u " // 13. Number of major faults of waited-for children
"%lu " // 14. CPU user time in ticks
"%lu " // 15. CPU kernel time in ticks
"%*d " // 16. CPU user time of waited-for children
"%*d " // 17. CPU kernel time of waited-for children
"%*d " // 18. Priority
"%*d " // 19. Nice value
"%*d " // 20. Num threads
"%*d " // 21. Obsolete since 2.6
"%*u " // 22. Time the process started after system boot
"%lu " // 23. Virtual memory size in bytes
"%ld", // 24. Resident set size in pages
&utime_ticks, &stime_ticks, &vmsize_bytes, &rss_pages);
fclose(f);
if ( rd != 4 )
return result;
result.rss = rss_pages * page_size;
result.vms = vmsize_bytes;
result.cpu = static_cast<double>(utime_ticks + stime_ticks) / ticks_per_second;
zeek::filesystem::path fd_path{"/proc/self/fd"};
result.fds =
std::distance(zeek::filesystem::directory_iterator{fd_path}, zeek::filesystem::directory_iterator{});
}
return result;
}
} // namespace zeek::telemetry::detail
#elif defined(__FreeBSD__)
// Force these includes into a specific order so that the libraries can find
// all of the required types.
// clang-format off
#include <sys/queue.h>
#include <sys/sysctl.h>
#include <sys/types.h>
#include <sys/user.h>
#include <unistd.h>
#include <libprocstat.h>
#include <libutil.h>
// clang-format on
namespace zeek::telemetry::detail {
process_stats get_process_stats() {
process_stats result;
struct kinfo_proc* kp = kinfo_getproc(getpid());
if ( kp ) {
result.vms = kp->ki_size;
result.rss = kp->ki_rssize * getpagesize();
result.cpu = static_cast<double>(kp->ki_runtime) / 1000000.0;
struct procstat* procstat = procstat_open_sysctl();
struct filestat_list* files = procstat_getfiles(procstat, kp, 0);
struct filestat* file = nullptr;
// Use one of the looping methods from sys/queue.h instead of
// implementing this by hand.
STAILQ_FOREACH(file, files, next)
result.fds++;
procstat_freeprocs(procstat, kp);
procstat_close(procstat);
}
return result;
}
} // namespace zeek::telemetry::detail
#endif

23
src/telemetry/ProcessStats.h Normal file
View file

@ -0,0 +1,23 @@
#pragma once
#include "zeek/zeek-config.h"
#include <cstdint>
namespace zeek::telemetry::detail {
struct process_stats {
int64_t rss = 0;
int64_t vms = 0;
double cpu = 0.0;
int64_t fds = 0;
};
#if defined(__APPLE__) || defined(HAVE_LINUX) || defined(__FreeBSD__)
#define HAVE_PROCESS_STAT_METRICS
process_stats get_process_stats();
#endif
} // namespace zeek::telemetry::detail

11
src/telemetry/Timer.h
View file

@ -1,3 +1,4 @@
// See the file "COPYING" in the main distribution directory for copyright.
#pragma once
@ -15,7 +16,7 @@ class [[nodiscard]] Timer {
public:
using Clock = std::chrono::steady_clock;
explicit Timer(DblHistogram h) : h_(h) { start_ = Clock::now(); }
explicit Timer(std::shared_ptr<Histogram> h) : h_(std::move(h)) { start_ = Clock::now(); }
Timer(const Timer&) = delete;
@ -30,14 +31,14 @@ public:
auto Started() const noexcept { return start_; }
/// Calls `h.Observe` with the time passed since `start`.
static void Observe(DblHistogram h, Clock::time_point start) {
using DblSec = std::chrono::duration<double>;
static void Observe(const std::shared_ptr<Histogram>& h, Clock::time_point start) {
using Sec = std::chrono::duration<double>;
if ( auto end = Clock::now(); end > start )
h.Observe(std::chrono::duration_cast<DblSec>(end - start).count());
h->Observe(std::chrono::duration_cast<Sec>(end - start).count());
}
private:
DblHistogram h_;
std::shared_ptr<Histogram> h_;
Clock::time_point start_;
};

60
src/telemetry/Utils.cc Normal file
View file

@ -0,0 +1,60 @@
#include "Utils.h"
#include "zeek/ID.h"
#include "zeek/Reporter.h"
#include "zeek/Val.h"
#include "zeek/telemetry/telemetry.bif.h"
#include "zeek/util.h"
using namespace zeek;
namespace zeek::telemetry::detail {
std::string BuildFullPrometheusName(std::string_view prefix, std::string_view name, std::string_view unit,
bool is_sum) {
if ( prefix.empty() || name.empty() )
reporter->FatalError("Telemetry metric families must have a non-zero-length prefix and name");
std::string fn = util::fmt("%s_%s", prefix.data(), name.data());
std::for_each(fn.begin(), fn.end(), [](char& c) {
if ( ! std::isalnum(c) )
c = '_';
});
// Suffixes of full metric names of _total are reserved by Prometheus. Disallow their use here.
if ( util::ends_with(fn, "_total") )
reporter->FatalError("Metric names cannot end with '_total'");
else if ( unit == "total" || util::ends_with(unit, "_total") )
reporter->FatalError("Metric units cannot end with '_total'");
// We were previously using "1" to mean "no unit value" for whatever reason, so we have to handle that now
// to mean the same thing.
if ( ! unit.empty() && unit != "1" )
fn.append("_").append(unit);
if ( is_sum )
fn.append("_total");
return fn;
}
prometheus::Labels BuildPrometheusLabels(Span<const LabelView> labels) {
prometheus::Labels p_labels;
bool found_endpoint = false;
for ( const auto& lbl : labels ) {
p_labels.emplace(util::strreplace(std::string{lbl.first}, "-", "_"), lbl.second);
if ( lbl.first == "endpoint" )
found_endpoint = true;
}
if ( ! found_endpoint ) {
auto endpoint = id::find_val("Telemetry::metrics_endpoint_name")->AsStringVal();
if ( endpoint && endpoint->Len() > 0 )
p_labels.emplace("endpoint", endpoint->ToStdString());
}
return p_labels;
}
} // namespace zeek::telemetry::detail

29
src/telemetry/Utils.h Normal file
View file

@ -0,0 +1,29 @@
#pragma once
#include <prometheus/family.h>
#include <prometheus/labels.h>
#include <string_view>
#include "zeek/Span.h"
#include "zeek/Val.h"
namespace zeek::telemetry {
using LabelView = std::pair<std::string_view, std::string_view>;
namespace detail {
/**
* Builds a set of labels for prometheus based on a set of labels from
* Zeek. This adds an 'endpoint' label if it's missing from the set.
*/
prometheus::Labels BuildPrometheusLabels(Span<const LabelView> labels);
/**
* Builds a full metric name for Prometheus from prefix, name, and unit values.
*/
std::string BuildFullPrometheusName(std::string_view prefix, std::string_view name, std::string_view unit,
bool is_sum = false);
} // namespace detail
} // namespace zeek::telemetry

365
src/telemetry/telemetry.bif
View file

@ -3,19 +3,18 @@
module Telemetry;
enum MetricType %{
DOUBLE_COUNTER,
INT_COUNTER,
DOUBLE_GAUGE,
INT_GAUGE,
DOUBLE_HISTOGRAM,
INT_HISTOGRAM,
COUNTER,
GAUGE,
HISTOGRAM,
%}
%%{
#include "zeek/telemetry/Counter.h"
#include "zeek/telemetry/Gauge.h"
#include "zeek/telemetry/Histogram.h"
#include "zeek/telemetry/Manager.h"
#include "zeek/telemetry/Opaques.h"
namespace {
@ -71,7 +70,7 @@ std::vector<zeek::telemetry::LabelView> sv_tbl(zeek::TableVal* xs)
return result;
}
bool is_valid(zeek::Span<const zeek::telemetry::LabelView> labels,
bool labels_valid(zeek::Span<const zeek::telemetry::LabelView> labels,
zeek::Span<const std::string> label_names)
{
auto key_in_label_names = [keys{label_names}](auto x)
@ -106,407 +105,199 @@ auto to_std_vec(zeek::Val* xs)
}
%%}
# -- IntCounter ----------------------------------------------------------------
# -- Counter ----------------------------------------------------------------
function Telemetry::__int_counter_family%(prefix: string,
function Telemetry::__counter_family%(prefix: string,
name: string,
labels: string_vec,
helptext: string &default = "Zeek Script Metric",
unit: string &default = "1",
is_sum: bool &default = F%): opaque of int_counter_metric_family
unit: string &default = ""%): opaque of counter_metric_family
%{
auto lbl_vec = sv_vec(labels->AsVectorVal());
auto hdl = telemetry_mgr->CounterFamily(sv(prefix), sv(name), lbl_vec,
sv(helptext), sv(unit), is_sum);
return zeek::make_intrusive<IntCounterMetricFamilyVal>(hdl);
sv(helptext), sv(unit));
return zeek::make_intrusive<CounterMetricFamilyVal>(hdl);
%}
function Telemetry::__int_counter_metric_get_or_add%(family: opaque of int_counter_metric_family,
labels: table_string_of_string%): opaque of int_counter_metric
function Telemetry::__counter_metric_get_or_add%(family: opaque of counter_metric_family,
labels: table_string_of_string%): opaque of counter_metric
%{
using ResultType = zeek::IntrusivePtr<IntCounterMetricFamilyVal>;
if ( auto ptr = dynamic_cast<zeek::IntCounterMetricFamilyVal*>(family) )
using ResultType = zeek::IntrusivePtr<CounterMetricFamilyVal>;
if ( auto ptr = dynamic_cast<zeek::CounterMetricFamilyVal*>(family) )
{
auto hdl = ptr->GetHandle();
auto lbl_map = sv_tbl(labels->AsTableVal());
if ( is_valid(lbl_map, hdl.LabelNames()) )
if ( labels_valid(lbl_map, hdl->LabelNames()) )
{
auto res = hdl.GetOrAdd(lbl_map);
return zeek::make_intrusive<IntCounterMetricVal>(res);
auto res = hdl->GetOrAdd(lbl_map);
return zeek::make_intrusive<CounterMetricVal>(res);
}
else
{
zeek::reporter->Error("Telemetry::int_counter_metric_get_or_add: invalid label dimensions.");
zeek::reporter->Error("Telemetry::counter_metric_get_or_add: invalid label dimensions.");
return ResultType{nullptr};
}
}
else
{
zeek::reporter->Error("Telemetry::int_counter_metric_get_or_add: invalid handle.");
zeek::reporter->Error("Telemetry::counter_metric_get_or_add: invalid handle.");
return ResultType{nullptr};
}
%}
function Telemetry::__int_counter_inc%(val: opaque of int_counter_metric,
amount: int &default = 1%): bool
function Telemetry::__counter_inc%(val: opaque of counter_metric,
amount: double &default = 1.0%): bool
%{
return with<IntCounterMetricVal>(val, "Telemetry::int_counter_inc: invalid handle.", [amount](auto hdl) { hdl.Inc(amount); });
return with<CounterMetricVal>(val, "Telemetry::counter_inc: invalid handle.", [amount](auto hdl) { hdl->Inc(amount); });
%}
function Telemetry::__int_counter_value%(val: opaque of int_counter_metric%): int
function Telemetry::__counter_value%(val: opaque of counter_metric%): double
%{
if ( auto ptr = dynamic_cast<zeek::IntCounterMetricVal*>(val) )
if ( auto ptr = dynamic_cast<zeek::CounterMetricVal*>(val) )
{
return zeek::val_mgr->Int(ptr->GetHandle().Value());
return zeek::make_intrusive<zeek::DoubleVal>(ptr->GetHandle()->Value());
}
else
{
zeek::reporter->Error("Telemetry::int_counter_value: invalid handle.");
return zeek::val_mgr->Int(0);
zeek::reporter->Error("Telemetry::counter_value: invalid handle.");
return zeek::make_intrusive<zeek::DoubleVal>(0);
}
%}
# -- DblCounter ----------------------------------------------------------------
# -- Gauge ------------------------------------------------------------------
function Telemetry::__dbl_counter_family%(prefix: string,
name: string,
labels: string_vec,
helptext: string &default = "Zeek Script Metric",
unit: string &default = "1",
is_sum: bool &default = F%): opaque of dbl_counter_metric_family
%{
auto lbl_vec = sv_vec(labels->AsVectorVal());
auto hdl = telemetry_mgr->CounterFamily<double>(sv(prefix), sv(name), lbl_vec,
sv(helptext), sv(unit), is_sum);
return zeek::make_intrusive<DblCounterMetricFamilyVal>(hdl);
%}
function Telemetry::__dbl_counter_metric_get_or_add%(family: opaque of dbl_counter_metric_family,
labels: table_string_of_string%): opaque of dbl_counter_metric
%{
using ResultType = zeek::IntrusivePtr<DblCounterMetricFamilyVal>;
if ( auto ptr = dynamic_cast<zeek::DblCounterMetricFamilyVal*>(family) )
{
auto hdl = ptr->GetHandle();
auto lbl_map = sv_tbl(labels->AsTableVal());
if ( is_valid(lbl_map, hdl.LabelNames()) )
{
auto res = hdl.GetOrAdd(lbl_map);
return zeek::make_intrusive<DblCounterMetricVal>(res);
}
else
{
zeek::reporter->Error("Telemetry::dbl_counter_metric_get_or_add: invalid label dimensions.");
return ResultType{nullptr};
}
}
else
{
zeek::reporter->Error("Telemetry::dbl_counter_metric_get_or_add: invalid handle.");
return ResultType{nullptr};
}
%}
function Telemetry::__dbl_counter_inc%(val: opaque of dbl_counter_metric,
amount: double &default = 1.0%): bool
%{
return with<DblCounterMetricVal>(val, "Telemetry::dbl_counter_inc: invalid handle.", [amount](auto hdl) { hdl.Inc(amount); });
%}
function Telemetry::__dbl_counter_value%(val: opaque of dbl_counter_metric%): double
%{
if ( auto ptr = dynamic_cast<zeek::DblCounterMetricVal*>(val) )
{
return zeek::make_intrusive<DoubleVal>(ptr->GetHandle().Value());
}
else
{
zeek::reporter->Error("Telemetry::dbl_counter_value: invalid handle.");
return zeek::make_intrusive<DoubleVal>(0.0);
}
%}
# -- IntGauge ------------------------------------------------------------------
function Telemetry::__int_gauge_family%(prefix: string,
function Telemetry::__gauge_family%(prefix: string,
name: string,
labels: string_vec,
helptext: string &default = "Zeek Script Metric",
unit: string &default = "1",
is_sum: bool &default = F%): opaque of int_gauge_metric_family
unit: string &default = ""%): opaque of gauge_metric_family
%{
auto lbl_vec = sv_vec(labels->AsVectorVal());
auto hdl = telemetry_mgr->GaugeFamily(sv(prefix), sv(name), lbl_vec,
sv(helptext), sv(unit), is_sum);
return zeek::make_intrusive<IntGaugeMetricFamilyVal>(hdl);
sv(helptext), sv(unit));
return zeek::make_intrusive<GaugeMetricFamilyVal>(hdl);
%}
function Telemetry::__int_gauge_metric_get_or_add%(family: opaque of int_gauge_metric_family,
labels: table_string_of_string%): opaque of int_gauge_metric
function Telemetry::__gauge_metric_get_or_add%(family: opaque of gauge_metric_family,
labels: table_string_of_string%): opaque of gauge_metric
%{
using ResultType = zeek::IntrusivePtr<IntGaugeMetricFamilyVal>;
if ( auto ptr = dynamic_cast<zeek::IntGaugeMetricFamilyVal*>(family) )
using ResultType = zeek::IntrusivePtr<GaugeMetricFamilyVal>;
if ( auto ptr = dynamic_cast<zeek::GaugeMetricFamilyVal*>(family) )
{
auto hdl = ptr->GetHandle();
auto lbl_map = sv_tbl(labels->AsTableVal());
if ( is_valid(lbl_map, hdl.LabelNames()) )
if ( labels_valid(lbl_map, hdl->LabelNames()) )
{
auto res = hdl.GetOrAdd(lbl_map);
return zeek::make_intrusive<IntGaugeMetricVal>(res);
auto res = hdl->GetOrAdd(lbl_map);
return zeek::make_intrusive<GaugeMetricVal>(res);
}
else
{
zeek::reporter->Error("Telemetry::int_gauge_metric_get_or_add: invalid label dimensions.");
zeek::reporter->Error("Telemetry::gauge_metric_get_or_add: invalid label dimensions.");
return ResultType{nullptr};
}
}
else
{
zeek::reporter->Error("Telemetry::int_gauge_metric_get_or_add: invalid handle.");
zeek::reporter->Error("Telemetry::gauge_metric_get_or_add: invalid handle.");
return ResultType{nullptr};
}
%}
function Telemetry::__int_gauge_inc%(val: opaque of int_gauge_metric,
amount: int &default = 1%): bool
%{
return with<IntGaugeMetricVal>(val, "Telemetry::int_gauge_inc: invalid handle.", [amount](auto hdl) { hdl.Inc(amount); });
%}
function Telemetry::__int_gauge_dec%(val: opaque of int_gauge_metric,
amount: int &default = 1%): bool
%{
return with<IntGaugeMetricVal>(val, "Telemetry::int_gauge_dec: invalid handle.", [amount](auto hdl) { hdl.Dec(amount); });
%}
function Telemetry::__int_gauge_value%(val: opaque of int_gauge_metric%): int
%{
if ( auto ptr = dynamic_cast<zeek::IntGaugeMetricVal*>(val) )
{
return zeek::val_mgr->Int(ptr->GetHandle().Value());
}
else
{
zeek::reporter->Error("Telemetry::int_gauge_value: invalid handle.");
return zeek::val_mgr->Int(0);
}
%}
# -- DblGauge ------------------------------------------------------------------
function Telemetry::__dbl_gauge_family%(prefix: string,
name: string,
labels: string_vec,
helptext: string &default = "Zeek Script Metric",
unit: string &default = "1",
is_sum: bool &default = F%): opaque of dbl_gauge_metric_family
%{
auto lbl_vec = sv_vec(labels->AsVectorVal());
auto hdl = telemetry_mgr->GaugeFamily<double>(sv(prefix), sv(name), lbl_vec,
sv(helptext), sv(unit), is_sum);
return zeek::make_intrusive<DblGaugeMetricFamilyVal>(hdl);
%}
function Telemetry::__dbl_gauge_metric_get_or_add%(family: opaque of dbl_gauge_metric_family,
labels: table_string_of_string%): opaque of dbl_gauge_metric
%{
using ResultType = zeek::IntrusivePtr<DblGaugeMetricFamilyVal>;
if ( auto ptr = dynamic_cast<zeek::DblGaugeMetricFamilyVal*>(family) )
{
auto hdl = ptr->GetHandle();
auto lbl_map = sv_tbl(labels->AsTableVal());
if ( is_valid(lbl_map, hdl.LabelNames()) )
{
auto res = hdl.GetOrAdd(lbl_map);
return zeek::make_intrusive<DblGaugeMetricVal>(res);
}
else
{
zeek::reporter->Error("Telemetry::dbl_gauge_metric_get_or_add: invalid label dimensions.");
return ResultType{nullptr};
}
}
else
{
zeek::reporter->Error("Telemetry::dbl_gauge_metric_get_or_add: invalid handle.");
return ResultType{nullptr};
}
%}
function Telemetry::__dbl_gauge_inc%(val: opaque of dbl_gauge_metric,
function Telemetry::__gauge_inc%(val: opaque of gauge_metric,
amount: double &default = 1.0%): bool
%{
return with<DblGaugeMetricVal>(val, "Telemetry::dbl_gauge_inc: invalid handle.", [amount](auto hdl) { hdl.Inc(amount); });
return with<GaugeMetricVal>(val, "Telemetry::gauge_inc: invalid handle.", [amount](auto hdl) { hdl->Inc(amount); });
%}
function Telemetry::__dbl_gauge_dec%(val: opaque of dbl_gauge_metric,
function Telemetry::__gauge_dec%(val: opaque of gauge_metric,
amount: double &default = 1.0%): bool
%{
return with<DblGaugeMetricVal>(val, "Telemetry::dbl_gauge_dec: invalid handle.", [amount](auto hdl) { hdl.Dec(amount); });
return with<GaugeMetricVal>(val, "Telemetry::gauge_dec: invalid handle.", [amount](auto hdl) { hdl->Dec(amount); });
%}
function Telemetry::__dbl_gauge_value%(val: opaque of dbl_gauge_metric%): double
function Telemetry::__gauge_value%(val: opaque of gauge_metric%): double
%{
if ( auto ptr = dynamic_cast<zeek::DblGaugeMetricVal*>(val) )
if ( auto ptr = dynamic_cast<zeek::GaugeMetricVal*>(val) )
{
return zeek::make_intrusive<DoubleVal>(ptr->GetHandle().Value());
return zeek::make_intrusive<zeek::DoubleVal>(ptr->GetHandle()->Value());
}
else
{
zeek::reporter->Error("Telemetry::dbl_gauge_value: invalid handle.");
return zeek::make_intrusive<DoubleVal>(0.0);
zeek::reporter->Error("Telemetry::gauge_value: invalid handle.");
return zeek::make_intrusive<zeek::DoubleVal>(0.0);
}
%}
# -- IntHistogram --------------------------------------------------------------
# -- Histogram --------------------------------------------------------------
function Telemetry::__int_histogram_family%(prefix: string,
name: string,
labels: string_vec,
bounds: int_vec,
helptext: string &default = "Zeek Script Metric",
unit: string &default = "1",
is_sum: bool &default = F%): opaque of int_histogram_metric_family
%{
auto lbl_vec = sv_vec(labels->AsVectorVal());
auto std_bounds = to_std_vec<int64_t>(bounds);
auto hdl = telemetry_mgr->HistogramFamily(sv(prefix), sv(name), lbl_vec,
std_bounds, sv(helptext),
sv(unit), is_sum);
return zeek::make_intrusive<IntHistogramMetricFamilyVal>(hdl);
%}
function Telemetry::__int_histogram_metric_get_or_add%(family: opaque of int_histogram_metric_family,
labels: table_string_of_string%): opaque of int_histogram_metric
%{
using ResultType = zeek::IntrusivePtr<IntHistogramMetricFamilyVal>;
if ( auto ptr = dynamic_cast<zeek::IntHistogramMetricFamilyVal*>(family) )
{
auto hdl = ptr->GetHandle();
auto lbl_map = sv_tbl(labels->AsTableVal());
if ( is_valid(lbl_map, hdl.LabelNames()) )
{
auto res = hdl.GetOrAdd(lbl_map);
return zeek::make_intrusive<IntHistogramMetricVal>(res);
}
else
{
zeek::reporter->Error("Telemetry::int_histogram_metric_get_or_add: invalid label dimensions.");
return ResultType{nullptr};
}
}
else
{
zeek::reporter->Error("Telemetry::int_histogram_metric_get_or_add: invalid handle.");
return ResultType{nullptr};
}
%}
function Telemetry::__int_histogram_observe%(val: opaque of int_histogram_metric,
measurement: int%): bool
%{
return with<IntHistogramMetricVal>(val, "Telemetry::int_histogram_inc: invalid handle.", [measurement](auto hdl) { hdl.Observe(measurement); });
%}
function Telemetry::__int_histogram_sum%(val: opaque of int_histogram_metric%): int
%{
if ( auto ptr = dynamic_cast<zeek::IntHistogramMetricVal*>(val) )
{
return zeek::val_mgr->Int(ptr->GetHandle().Sum());
}
else
{
zeek::reporter->Error("Telemetry::int_histogram_sum: invalid handle.");
return zeek::val_mgr->Int(0);
}
%}
# -- DblHistogram --------------------------------------------------------------
function Telemetry::__dbl_histogram_family%(prefix: string,
function Telemetry::__histogram_family%(prefix: string,
name: string,
labels: string_vec,
bounds: double_vec,
helptext: string &default = "Zeek Script Metric",
unit: string &default = "1",
is_sum: bool &default = F%): opaque of dbl_histogram_metric_family
unit: string &default = ""%): opaque of histogram_metric_family
%{
auto lbl_vec = sv_vec(labels->AsVectorVal());
auto std_bounds = to_std_vec<double>(bounds);
auto hdl = telemetry_mgr->HistogramFamily<double>(sv(prefix), sv(name),
lbl_vec, std_bounds,
sv(helptext), sv(unit),
is_sum);
return zeek::make_intrusive<DblHistogramMetricFamilyVal>(hdl);
auto hdl = telemetry_mgr->HistogramFamily(sv(prefix), sv(name), lbl_vec,
std_bounds, sv(helptext),
sv(unit));
return zeek::make_intrusive<HistogramMetricFamilyVal>(hdl);
%}
function Telemetry::__dbl_histogram_metric_get_or_add%(family: opaque of dbl_histogram_metric_family,
labels: table_string_of_string%): opaque of dbl_histogram_metric
function Telemetry::__histogram_metric_get_or_add%(family: opaque of histogram_metric_family,
labels: table_string_of_string%): opaque of histogram_metric
%{
using ResultType = zeek::IntrusivePtr<DblHistogramMetricFamilyVal>;
if ( auto ptr = dynamic_cast<zeek::DblHistogramMetricFamilyVal*>(family) )
using ResultType = zeek::IntrusivePtr<HistogramMetricFamilyVal>;
if ( auto ptr = dynamic_cast<zeek::HistogramMetricFamilyVal*>(family) )
{
auto hdl = ptr->GetHandle();
auto lbl_map = sv_tbl(labels->AsTableVal());
if ( is_valid(lbl_map, hdl.LabelNames()) )
if ( labels_valid(lbl_map, hdl->LabelNames()) )
{
auto res = hdl.GetOrAdd(lbl_map);
return zeek::make_intrusive<DblHistogramMetricVal>(res);
auto res = hdl->GetOrAdd(lbl_map);
return zeek::make_intrusive<HistogramMetricVal>(res);
}
else
{
zeek::reporter->Error("Telemetry::dbl_histogram_metric_get_or_add: invalid label dimensions.");
zeek::reporter->Error("Telemetry::histogram_metric_get_or_add: invalid label dimensions.");
return ResultType{nullptr};
}
}
else
{
zeek::reporter->Error("Telemetry::dbl_histogram_metric_get_or_add: invalid handle.");
zeek::reporter->Error("Telemetry::histogram_metric_get_or_add: invalid handle.");
return ResultType{nullptr};
}
%}
function Telemetry::__dbl_histogram_observe%(val: opaque of dbl_histogram_metric,
measurement: double%): bool
function Telemetry::__histogram_observe%(val: opaque of histogram_metric,
measurement: double%): bool
%{
return with<DblHistogramMetricVal>(val, "Telemetry::dbl_histogram_inc: invalid handle.", [measurement](auto hdl) { hdl.Observe(measurement); });
return with<HistogramMetricVal>(val, "Telemetry::histogram_inc: invalid handle.",
[measurement](auto hdl) { hdl->Observe(measurement); });
%}
function Telemetry::__dbl_histogram_sum%(val: opaque of dbl_histogram_metric%): double
function Telemetry::__histogram_sum%(val: opaque of histogram_metric%): double
%{
if ( auto ptr = dynamic_cast<zeek::DblHistogramMetricVal*>(val) )
if ( auto ptr = dynamic_cast<zeek::HistogramMetricVal*>(val) )
{
return zeek::make_intrusive<DoubleVal>(ptr->GetHandle().Sum());
return zeek::make_intrusive<zeek::DoubleVal>(ptr->GetHandle()->Sum());
}
else
{
zeek::reporter->Error("Telemetry::dbl_histogram_sum: invalid handle.");
return zeek::make_intrusive<DoubleVal>(0.0);
zeek::reporter->Error("Telemetry::histogram_sum: invalid handle.");
return make_intrusive<zeek::DoubleVal>(0.0);
}
%}
function Telemetry::__collect_metrics%(prefix: string, name: string%): any_vec
%{
auto metrics = telemetry_mgr->CollectMetrics(sv(prefix), sv(name));
static auto metrics_vector_type = zeek::id::find_type<VectorType>("any_vec");
auto vec = zeek::make_intrusive<VectorVal>(metrics_vector_type);
for ( const auto& m : metrics )
vec->Append(m.AsMetricRecord());
return vec;
return telemetry_mgr->CollectMetrics(sv(prefix), sv(name));
%}
function Telemetry::__collect_histogram_metrics%(prefix: string, name: string%): any_vec
%{
auto metrics = telemetry_mgr->CollectHistogramMetrics(sv(prefix), sv(name));
static auto metrics_vector_type = zeek::id::find_type<VectorType>("any_vec");
auto vec = zeek::make_intrusive<VectorVal>(metrics_vector_type);
for ( const auto& m : metrics )
vec->Append(m.AsHistogramMetricRecord());
return vec;
return telemetry_mgr->CollectHistogramMetrics(sv(prefix), sv(name));
%}

36
src/zeek-setup.cc
View file

@ -205,18 +205,12 @@ zeek::OpaqueTypePtr bloomfilter_type;
zeek::OpaqueTypePtr x509_opaque_type;
zeek::OpaqueTypePtr ocsp_resp_opaque_type;
zeek::OpaqueTypePtr paraglob_type;
zeek::OpaqueTypePtr int_counter_metric_type;
zeek::OpaqueTypePtr int_counter_metric_family_type;
zeek::OpaqueTypePtr dbl_counter_metric_type;
zeek::OpaqueTypePtr dbl_counter_metric_family_type;
zeek::OpaqueTypePtr int_gauge_metric_type;
zeek::OpaqueTypePtr int_gauge_metric_family_type;
zeek::OpaqueTypePtr dbl_gauge_metric_type;
zeek::OpaqueTypePtr dbl_gauge_metric_family_type;
zeek::OpaqueTypePtr int_histogram_metric_type;
zeek::OpaqueTypePtr int_histogram_metric_family_type;
zeek::OpaqueTypePtr dbl_histogram_metric_type;
zeek::OpaqueTypePtr dbl_histogram_metric_family_type;
zeek::OpaqueTypePtr counter_metric_type;
zeek::OpaqueTypePtr counter_metric_family_type;
zeek::OpaqueTypePtr gauge_metric_type;
zeek::OpaqueTypePtr gauge_metric_family_type;
zeek::OpaqueTypePtr histogram_metric_type;
zeek::OpaqueTypePtr histogram_metric_family_type;
// Keep copy of command line
int zeek::detail::zeek_argc;
@ -705,18 +699,12 @@ SetupResult setup(int argc, char** argv, Options* zopts) {
x509_opaque_type = make_intrusive<OpaqueType>("x509");
ocsp_resp_opaque_type = make_intrusive<OpaqueType>("ocsp_resp");
paraglob_type = make_intrusive<OpaqueType>("paraglob");
int_counter_metric_type = make_intrusive<OpaqueType>("int_counter_metric");
int_counter_metric_family_type = make_intrusive<OpaqueType>("int_counter_metric_family");
dbl_counter_metric_type = make_intrusive<OpaqueType>("dbl_counter_metric");
dbl_counter_metric_family_type = make_intrusive<OpaqueType>("dbl_counter_metric_family");
int_gauge_metric_type = make_intrusive<OpaqueType>("int_gauge_metric");
int_gauge_metric_family_type = make_intrusive<OpaqueType>("int_gauge_metric_family");
dbl_gauge_metric_type = make_intrusive<OpaqueType>("dbl_gauge_metric");
dbl_gauge_metric_family_type = make_intrusive<OpaqueType>("dbl_gauge_metric_family");
int_histogram_metric_type = make_intrusive<OpaqueType>("int_histogram_metric");
int_histogram_metric_family_type = make_intrusive<OpaqueType>("int_histogram_metric_family");
dbl_histogram_metric_type = make_intrusive<OpaqueType>("dbl_histogram_metric");
dbl_histogram_metric_family_type = make_intrusive<OpaqueType>("dbl_histogram_metric_family");
counter_metric_type = make_intrusive<OpaqueType>("counter_metric");
counter_metric_family_type = make_intrusive<OpaqueType>("counter_metric_family");
gauge_metric_type = make_intrusive<OpaqueType>("gauge_metric");
gauge_metric_family_type = make_intrusive<OpaqueType>("gauge_metric_family");
histogram_metric_type = make_intrusive<OpaqueType>("histogram_metric");
histogram_metric_family_type = make_intrusive<OpaqueType>("histogram_metric_family");
log_delay_token_type = make_intrusive<OpaqueType>("LogDelayToken");
// After spinning up Broker, we have background threads running now. If

1
testing/btest/Baseline/coverage.bare-load-baseline/canonified_loaded_scripts.log
View file

@ -144,6 +144,7 @@ scripts/base/init-frameworks-and-bifs.zeek
scripts/base/utils/site.zeek
scripts/base/utils/patterns.zeek
scripts/base/frameworks/files/magic/__load__.zeek
scripts/base/frameworks/telemetry/options.zeek
build/scripts/base/bif/__load__.zeek
build/scripts/base/bif/telemetry.bif.zeek
build/scripts/base/bif/zeekygen.bif.zeek

1
testing/btest/Baseline/coverage.default-load-baseline/canonified_loaded_scripts.log
View file

@ -144,6 +144,7 @@ scripts/base/init-frameworks-and-bifs.zeek
scripts/base/utils/site.zeek
scripts/base/utils/patterns.zeek
scripts/base/frameworks/files/magic/__load__.zeek
scripts/base/frameworks/telemetry/options.zeek
build/scripts/base/bif/__load__.zeek
build/scripts/base/bif/telemetry.bif.zeek
build/scripts/base/bif/zeekygen.bif.zeek

10
testing/btest/Baseline/language.init-integration/out
View file

@ -13,15 +13,15 @@ init_key2 in state2: 1
[worker-1] = [node_type=Cluster::WORKER, ip=127.0.0.1, p=5/udp, manager=manager-1]
}
{
[worker-4] = [node_type=Cluster::WORKER, ip=2.3.4.5, zone_id=, p=13/udp, interface=<uninitialized>, manager=<uninitialized>, time_machine=<uninitialized>, id=<uninitialized>]
[worker-4] = [node_type=Cluster::WORKER, ip=2.3.4.5, zone_id=, p=13/udp, interface=<uninitialized>, manager=<uninitialized>, time_machine=<uninitialized>, id=<uninitialized>, metrics_port=<uninitialized>]
}
{
[worker-4] = [node_type=Cluster::WORKER, ip=2.3.4.5, zone_id=, p=13/udp, interface=<uninitialized>, manager=<uninitialized>, time_machine=<uninitialized>, id=<uninitialized>],
[worker-5] = [node_type=Cluster::WORKER, ip=3.4.5.6, zone_id=, p=15/tcp, interface=<uninitialized>, manager=<uninitialized>, time_machine=<uninitialized>, id=<uninitialized>]
[worker-4] = [node_type=Cluster::WORKER, ip=2.3.4.5, zone_id=, p=13/udp, interface=<uninitialized>, manager=<uninitialized>, time_machine=<uninitialized>, id=<uninitialized>, metrics_port=<uninitialized>],
[worker-5] = [node_type=Cluster::WORKER, ip=3.4.5.6, zone_id=, p=15/tcp, interface=<uninitialized>, manager=<uninitialized>, time_machine=<uninitialized>, id=<uninitialized>, metrics_port=<uninitialized>]
}
{
[worker-4] = [node_type=Cluster::WORKER, ip=2.3.4.5, zone_id=, p=13/udp, interface=<uninitialized>, manager=<uninitialized>, time_machine=<uninitialized>, id=<uninitialized>],
[worker-6] = [node_type=Cluster::WORKER, ip=4.5.6.7, zone_id=, p=17/udp, interface=<uninitialized>, manager=<uninitialized>, time_machine=<uninitialized>, id=<uninitialized>]
[worker-4] = [node_type=Cluster::WORKER, ip=2.3.4.5, zone_id=, p=13/udp, interface=<uninitialized>, manager=<uninitialized>, time_machine=<uninitialized>, id=<uninitialized>, metrics_port=<uninitialized>],
[worker-6] = [node_type=Cluster::WORKER, ip=4.5.6.7, zone_id=, p=17/udp, interface=<uninitialized>, manager=<uninitialized>, time_machine=<uninitialized>, id=<uninitialized>, metrics_port=<uninitialized>]
}
{
[3.0, 4]

66
testing/btest/Baseline/plugins.hooks/output
View file

@ -18,18 +18,8 @@
0.000000 MetaHookPost CallFunction(Analyzer::register_for_port, <frame>, (Analyzer::ANALYZER_HTTP, 81/tcp)) -> <no result>
0.000000 MetaHookPost CallFunction(Analyzer::register_for_port, <frame>, (Analyzer::ANALYZER_HTTP, 8888/tcp)) -> <no result>
0.000000 MetaHookPost CallFunction(Analyzer::register_for_ports, <frame>, (Analyzer::ANALYZER_HTTP, {80<...>/tcp})) -> <no result>
0.000000 MetaHookPost CallFunction(Broker::__set_metrics_export_endpoint_name, <frame>, ()) -> <no result>
0.000000 MetaHookPost CallFunction(Broker::__set_metrics_export_interval, <frame>, (1.0 sec)) -> <no result>
0.000000 MetaHookPost CallFunction(Broker::__set_metrics_export_prefixes, <frame>, ([])) -> <no result>
0.000000 MetaHookPost CallFunction(Broker::__set_metrics_export_topic, <frame>, ()) -> <no result>
0.000000 MetaHookPost CallFunction(Broker::__set_metrics_import_topics, <frame>, ([])) -> <no result>
0.000000 MetaHookPost CallFunction(Broker::__subscribe, <frame>, (zeek/supervisor)) -> <no result>
0.000000 MetaHookPost CallFunction(Broker::subscribe, <frame>, (zeek/supervisor)) -> <no result>
0.000000 MetaHookPost CallFunction(Broker::update_metrics_export_endpoint_name, <frame>, (Broker::metrics_export_endpoint_name, )) -> <no result>
0.000000 MetaHookPost CallFunction(Broker::update_metrics_export_interval, <frame>, (Broker::metrics_export_interval, 1.0 sec)) -> <no result>
0.000000 MetaHookPost CallFunction(Broker::update_metrics_export_prefixes, <frame>, (Broker::metrics_export_prefixes, [])) -> <no result>
0.000000 MetaHookPost CallFunction(Broker::update_metrics_export_topic, <frame>, (Broker::metrics_export_topic, )) -> <no result>
0.000000 MetaHookPost CallFunction(Broker::update_metrics_import_topics, <frame>, (Broker::metrics_import_topics, [])) -> <no result>
0.000000 MetaHookPost CallFunction(Cluster::is_enabled, <frame>, ()) -> <no result>
0.000000 MetaHookPost CallFunction(Cluster::is_enabled, <null>, ()) -> <no result>
0.000000 MetaHookPost CallFunction(Cluster::register_pool, <frame>, ([topic=zeek<...>/logger, node_type=Cluster::LOGGER, max_nodes=<uninitialized>, exclusive=F])) -> <no result>
@ -118,16 +108,6 @@
0.000000 MetaHookPost CallFunction(Option::set_change_handler, <frame>, (Analyzer::Logging::ignore_analyzers, Config::config_option_changed{ if (<skip-config-log> == Config::location) return (Config::new_value)Config::log = Config::Info($ts=network_time(), $id=Config::ID, $old_value=Config::format_value(lookup_ID(Config::ID)), $new_value=Config::format_value(Config::new_value))if ( != Config::location) Config::log$location = Config::locationLog::write(Config::LOG, to_any_coerce Config::log)return (Config::new_value)}, -100)) -> <no result>
0.000000 MetaHookPost CallFunction(Option::set_change_handler, <frame>, (Analyzer::Logging::include_confirmations, Config::config_option_changed{ if (<skip-config-log> == Config::location) return (Config::new_value)Config::log = Config::Info($ts=network_time(), $id=Config::ID, $old_value=Config::format_value(lookup_ID(Config::ID)), $new_value=Config::format_value(Config::new_value))if ( != Config::location) Config::log$location = Config::locationLog::write(Config::LOG, to_any_coerce Config::log)return (Config::new_value)}, -100)) -> <no result>
0.000000 MetaHookPost CallFunction(Option::set_change_handler, <frame>, (Analyzer::Logging::include_confirmations, lambda_<15261139872714441626>{ if (Analyzer::Logging::new_value) enable_event_group(Analyzer::Logging::include_confirmations)elsedisable_event_group(Analyzer::Logging::include_confirmations)return (Analyzer::Logging::new_value)}, 0)) -> <no result>
0.000000 MetaHookPost CallFunction(Option::set_change_handler, <frame>, (Broker::metrics_export_endpoint_name, Broker::update_metrics_export_endpoint_name{ Broker::__set_metrics_export_endpoint_name(Broker::val)return (Broker::val)}, 0)) -> <no result>
0.000000 MetaHookPost CallFunction(Option::set_change_handler, <frame>, (Broker::metrics_export_endpoint_name, Config::config_option_changed{ if (<skip-config-log> == Config::location) return (Config::new_value)Config::log = Config::Info($ts=network_time(), $id=Config::ID, $old_value=Config::format_value(lookup_ID(Config::ID)), $new_value=Config::format_value(Config::new_value))if ( != Config::location) Config::log$location = Config::locationLog::write(Config::LOG, to_any_coerce Config::log)return (Config::new_value)}, -100)) -> <no result>
0.000000 MetaHookPost CallFunction(Option::set_change_handler, <frame>, (Broker::metrics_export_interval, Broker::update_metrics_export_interval{ Broker::__set_metrics_export_interval(Broker::val)return (Broker::val)}, 0)) -> <no result>
0.000000 MetaHookPost CallFunction(Option::set_change_handler, <frame>, (Broker::metrics_export_interval, Config::config_option_changed{ if (<skip-config-log> == Config::location) return (Config::new_value)Config::log = Config::Info($ts=network_time(), $id=Config::ID, $old_value=Config::format_value(lookup_ID(Config::ID)), $new_value=Config::format_value(Config::new_value))if ( != Config::location) Config::log$location = Config::locationLog::write(Config::LOG, to_any_coerce Config::log)return (Config::new_value)}, -100)) -> <no result>
0.000000 MetaHookPost CallFunction(Option::set_change_handler, <frame>, (Broker::metrics_export_prefixes, Broker::update_metrics_export_prefixes{ Broker::__set_metrics_export_prefixes(Broker::filter)return (Broker::filter)}, 0)) -> <no result>
0.000000 MetaHookPost CallFunction(Option::set_change_handler, <frame>, (Broker::metrics_export_prefixes, Config::config_option_changed{ if (<skip-config-log> == Config::location) return (Config::new_value)Config::log = Config::Info($ts=network_time(), $id=Config::ID, $old_value=Config::format_value(lookup_ID(Config::ID)), $new_value=Config::format_value(Config::new_value))if ( != Config::location) Config::log$location = Config::locationLog::write(Config::LOG, to_any_coerce Config::log)return (Config::new_value)}, -100)) -> <no result>
0.000000 MetaHookPost CallFunction(Option::set_change_handler, <frame>, (Broker::metrics_export_topic, Broker::update_metrics_export_topic{ Broker::__set_metrics_export_topic(Broker::val)return (Broker::val)}, 0)) -> <no result>
0.000000 MetaHookPost CallFunction(Option::set_change_handler, <frame>, (Broker::metrics_export_topic, Config::config_option_changed{ if (<skip-config-log> == Config::location) return (Config::new_value)Config::log = Config::Info($ts=network_time(), $id=Config::ID, $old_value=Config::format_value(lookup_ID(Config::ID)), $new_value=Config::format_value(Config::new_value))if ( != Config::location) Config::log$location = Config::locationLog::write(Config::LOG, to_any_coerce Config::log)return (Config::new_value)}, -100)) -> <no result>
0.000000 MetaHookPost CallFunction(Option::set_change_handler, <frame>, (Broker::metrics_import_topics, Broker::update_metrics_import_topics{ Broker::__set_metrics_import_topics(Broker::topics)return (Broker::topics)}, 0)) -> <no result>
0.000000 MetaHookPost CallFunction(Option::set_change_handler, <frame>, (Broker::metrics_import_topics, Config::config_option_changed{ if (<skip-config-log> == Config::location) return (Config::new_value)Config::log = Config::Info($ts=network_time(), $id=Config::ID, $old_value=Config::format_value(lookup_ID(Config::ID)), $new_value=Config::format_value(Config::new_value))if ( != Config::location) Config::log$location = Config::locationLog::write(Config::LOG, to_any_coerce Config::log)return (Config::new_value)}, -100)) -> <no result>
0.000000 MetaHookPost CallFunction(Option::set_change_handler, <frame>, (Broker::peer_counts_as_iosource, Config::config_option_changed{ if (<skip-config-log> == Config::location) return (Config::new_value)Config::log = Config::Info($ts=network_time(), $id=Config::ID, $old_value=Config::format_value(lookup_ID(Config::ID)), $new_value=Config::format_value(Config::new_value))if ( != Config::location) Config::log$location = Config::locationLog::write(Config::LOG, to_any_coerce Config::log)return (Config::new_value)}, -100)) -> <no result>
0.000000 MetaHookPost CallFunction(Option::set_change_handler, <frame>, (Conn::analyzer_inactivity_timeouts, Config::config_option_changed{ if (<skip-config-log> == Config::location) return (Config::new_value)Config::log = Config::Info($ts=network_time(), $id=Config::ID, $old_value=Config::format_value(lookup_ID(Config::ID)), $new_value=Config::format_value(Config::new_value))if ( != Config::location) Config::log$location = Config::locationLog::write(Config::LOG, to_any_coerce Config::log)return (Config::new_value)}, -100)) -> <no result>
0.000000 MetaHookPost CallFunction(Option::set_change_handler, <frame>, (Conn::default_extract, Config::config_option_changed{ if (<skip-config-log> == Config::location) return (Config::new_value)Config::log = Config::Info($ts=network_time(), $id=Config::ID, $old_value=Config::format_value(lookup_ID(Config::ID)), $new_value=Config::format_value(Config::new_value))if ( != Config::location) Config::log$location = Config::locationLog::write(Config::LOG, to_any_coerce Config::log)return (Config::new_value)}, -100)) -> <no result>
@ -594,6 +574,7 @@
0.000000 MetaHookPost LoadFile(0, base<...>/null, <...>/null) -> -1
0.000000 MetaHookPost LoadFile(0, base<...>/numbers, <...>/numbers.zeek) -> -1
0.000000 MetaHookPost LoadFile(0, base<...>/option.bif, <...>/option.bif.zeek) -> -1
0.000000 MetaHookPost LoadFile(0, base<...>/options, <...>/options.zeek) -> -1
0.000000 MetaHookPost LoadFile(0, base<...>/packet_analysis.bif, <...>/packet_analysis.bif.zeek) -> -1
0.000000 MetaHookPost LoadFile(0, base<...>/pbb, <...>/pbb) -> -1
0.000000 MetaHookPost LoadFile(0, base<...>/plugins, <...>/plugins) -> -1
@ -887,6 +868,7 @@
0.000000 MetaHookPost LoadFileExtended(0, base<...>/null, <...>/null) -> (-1, <no content>)
0.000000 MetaHookPost LoadFileExtended(0, base<...>/numbers, <...>/numbers.zeek) -> (-1, <no content>)
0.000000 MetaHookPost LoadFileExtended(0, base<...>/option.bif, <...>/option.bif.zeek) -> (-1, <no content>)
0.000000 MetaHookPost LoadFileExtended(0, base<...>/options, <...>/options.zeek) -> (-1, <no content>)
0.000000 MetaHookPost LoadFileExtended(0, base<...>/packet_analysis.bif, <...>/packet_analysis.bif.zeek) -> (-1, <no content>)
0.000000 MetaHookPost LoadFileExtended(0, base<...>/pbb, <...>/pbb) -> (-1, <no content>)
0.000000 MetaHookPost LoadFileExtended(0, base<...>/plugins, <...>/plugins) -> (-1, <no content>)
@ -952,18 +934,8 @@
0.000000 MetaHookPre CallFunction(Analyzer::register_for_port, <frame>, (Analyzer::ANALYZER_HTTP, 81/tcp))
0.000000 MetaHookPre CallFunction(Analyzer::register_for_port, <frame>, (Analyzer::ANALYZER_HTTP, 8888/tcp))
0.000000 MetaHookPre CallFunction(Analyzer::register_for_ports, <frame>, (Analyzer::ANALYZER_HTTP, {80<...>/tcp}))
0.000000 MetaHookPre CallFunction(Broker::__set_metrics_export_endpoint_name, <frame>, ())
0.000000 MetaHookPre CallFunction(Broker::__set_metrics_export_interval, <frame>, (1.0 sec))
0.000000 MetaHookPre CallFunction(Broker::__set_metrics_export_prefixes, <frame>, ([]))
0.000000 MetaHookPre CallFunction(Broker::__set_metrics_export_topic, <frame>, ())
0.000000 MetaHookPre CallFunction(Broker::__set_metrics_import_topics, <frame>, ([]))
0.000000 MetaHookPre CallFunction(Broker::__subscribe, <frame>, (zeek/supervisor))
0.000000 MetaHookPre CallFunction(Broker::subscribe, <frame>, (zeek/supervisor))
0.000000 MetaHookPre CallFunction(Broker::update_metrics_export_endpoint_name, <frame>, (Broker::metrics_export_endpoint_name, ))
0.000000 MetaHookPre CallFunction(Broker::update_metrics_export_interval, <frame>, (Broker::metrics_export_interval, 1.0 sec))
0.000000 MetaHookPre CallFunction(Broker::update_metrics_export_prefixes, <frame>, (Broker::metrics_export_prefixes, []))
0.000000 MetaHookPre CallFunction(Broker::update_metrics_export_topic, <frame>, (Broker::metrics_export_topic, ))
0.000000 MetaHookPre CallFunction(Broker::update_metrics_import_topics, <frame>, (Broker::metrics_import_topics, []))
0.000000 MetaHookPre CallFunction(Cluster::is_enabled, <frame>, ())
0.000000 MetaHookPre CallFunction(Cluster::is_enabled, <null>, ())
0.000000 MetaHookPre CallFunction(Cluster::register_pool, <frame>, ([topic=zeek<...>/logger, node_type=Cluster::LOGGER, max_nodes=<uninitialized>, exclusive=F]))
@ -1052,16 +1024,6 @@
0.000000 MetaHookPre CallFunction(Option::set_change_handler, <frame>, (Analyzer::Logging::ignore_analyzers, Config::config_option_changed{ if (<skip-config-log> == Config::location) return (Config::new_value)Config::log = Config::Info($ts=network_time(), $id=Config::ID, $old_value=Config::format_value(lookup_ID(Config::ID)), $new_value=Config::format_value(Config::new_value))if ( != Config::location) Config::log$location = Config::locationLog::write(Config::LOG, to_any_coerce Config::log)return (Config::new_value)}, -100))
0.000000 MetaHookPre CallFunction(Option::set_change_handler, <frame>, (Analyzer::Logging::include_confirmations, Config::config_option_changed{ if (<skip-config-log> == Config::location) return (Config::new_value)Config::log = Config::Info($ts=network_time(), $id=Config::ID, $old_value=Config::format_value(lookup_ID(Config::ID)), $new_value=Config::format_value(Config::new_value))if ( != Config::location) Config::log$location = Config::locationLog::write(Config::LOG, to_any_coerce Config::log)return (Config::new_value)}, -100))
0.000000 MetaHookPre CallFunction(Option::set_change_handler, <frame>, (Analyzer::Logging::include_confirmations, lambda_<15261139872714441626>{ if (Analyzer::Logging::new_value) enable_event_group(Analyzer::Logging::include_confirmations)elsedisable_event_group(Analyzer::Logging::include_confirmations)return (Analyzer::Logging::new_value)}, 0))
0.000000 MetaHookPre CallFunction(Option::set_change_handler, <frame>, (Broker::metrics_export_endpoint_name, Broker::update_metrics_export_endpoint_name{ Broker::__set_metrics_export_endpoint_name(Broker::val)return (Broker::val)}, 0))
0.000000 MetaHookPre CallFunction(Option::set_change_handler, <frame>, (Broker::metrics_export_endpoint_name, Config::config_option_changed{ if (<skip-config-log> == Config::location) return (Config::new_value)Config::log = Config::Info($ts=network_time(), $id=Config::ID, $old_value=Config::format_value(lookup_ID(Config::ID)), $new_value=Config::format_value(Config::new_value))if ( != Config::location) Config::log$location = Config::locationLog::write(Config::LOG, to_any_coerce Config::log)return (Config::new_value)}, -100))
0.000000 MetaHookPre CallFunction(Option::set_change_handler, <frame>, (Broker::metrics_export_interval, Broker::update_metrics_export_interval{ Broker::__set_metrics_export_interval(Broker::val)return (Broker::val)}, 0))
0.000000 MetaHookPre CallFunction(Option::set_change_handler, <frame>, (Broker::metrics_export_interval, Config::config_option_changed{ if (<skip-config-log> == Config::location) return (Config::new_value)Config::log = Config::Info($ts=network_time(), $id=Config::ID, $old_value=Config::format_value(lookup_ID(Config::ID)), $new_value=Config::format_value(Config::new_value))if ( != Config::location) Config::log$location = Config::locationLog::write(Config::LOG, to_any_coerce Config::log)return (Config::new_value)}, -100))
0.000000 MetaHookPre CallFunction(Option::set_change_handler, <frame>, (Broker::metrics_export_prefixes, Broker::update_metrics_export_prefixes{ Broker::__set_metrics_export_prefixes(Broker::filter)return (Broker::filter)}, 0))
0.000000 MetaHookPre CallFunction(Option::set_change_handler, <frame>, (Broker::metrics_export_prefixes, Config::config_option_changed{ if (<skip-config-log> == Config::location) return (Config::new_value)Config::log = Config::Info($ts=network_time(), $id=Config::ID, $old_value=Config::format_value(lookup_ID(Config::ID)), $new_value=Config::format_value(Config::new_value))if ( != Config::location) Config::log$location = Config::locationLog::write(Config::LOG, to_any_coerce Config::log)return (Config::new_value)}, -100))
0.000000 MetaHookPre CallFunction(Option::set_change_handler, <frame>, (Broker::metrics_export_topic, Broker::update_metrics_export_topic{ Broker::__set_metrics_export_topic(Broker::val)return (Broker::val)}, 0))
0.000000 MetaHookPre CallFunction(Option::set_change_handler, <frame>, (Broker::metrics_export_topic, Config::config_option_changed{ if (<skip-config-log> == Config::location) return (Config::new_value)Config::log = Config::Info($ts=network_time(), $id=Config::ID, $old_value=Config::format_value(lookup_ID(Config::ID)), $new_value=Config::format_value(Config::new_value))if ( != Config::location) Config::log$location = Config::locationLog::write(Config::LOG, to_any_coerce Config::log)return (Config::new_value)}, -100))
0.000000 MetaHookPre CallFunction(Option::set_change_handler, <frame>, (Broker::metrics_import_topics, Broker::update_metrics_import_topics{ Broker::__set_metrics_import_topics(Broker::topics)return (Broker::topics)}, 0))
0.000000 MetaHookPre CallFunction(Option::set_change_handler, <frame>, (Broker::metrics_import_topics, Config::config_option_changed{ if (<skip-config-log> == Config::location) return (Config::new_value)Config::log = Config::Info($ts=network_time(), $id=Config::ID, $old_value=Config::format_value(lookup_ID(Config::ID)), $new_value=Config::format_value(Config::new_value))if ( != Config::location) Config::log$location = Config::locationLog::write(Config::LOG, to_any_coerce Config::log)return (Config::new_value)}, -100))
0.000000 MetaHookPre CallFunction(Option::set_change_handler, <frame>, (Broker::peer_counts_as_iosource, Config::config_option_changed{ if (<skip-config-log> == Config::location) return (Config::new_value)Config::log = Config::Info($ts=network_time(), $id=Config::ID, $old_value=Config::format_value(lookup_ID(Config::ID)), $new_value=Config::format_value(Config::new_value))if ( != Config::location) Config::log$location = Config::locationLog::write(Config::LOG, to_any_coerce Config::log)return (Config::new_value)}, -100))
0.000000 MetaHookPre CallFunction(Option::set_change_handler, <frame>, (Conn::analyzer_inactivity_timeouts, Config::config_option_changed{ if (<skip-config-log> == Config::location) return (Config::new_value)Config::log = Config::Info($ts=network_time(), $id=Config::ID, $old_value=Config::format_value(lookup_ID(Config::ID)), $new_value=Config::format_value(Config::new_value))if ( != Config::location) Config::log$location = Config::locationLog::write(Config::LOG, to_any_coerce Config::log)return (Config::new_value)}, -100))
0.000000 MetaHookPre CallFunction(Option::set_change_handler, <frame>, (Conn::default_extract, Config::config_option_changed{ if (<skip-config-log> == Config::location) return (Config::new_value)Config::log = Config::Info($ts=network_time(), $id=Config::ID, $old_value=Config::format_value(lookup_ID(Config::ID)), $new_value=Config::format_value(Config::new_value))if ( != Config::location) Config::log$location = Config::locationLog::write(Config::LOG, to_any_coerce Config::log)return (Config::new_value)}, -100))
@ -1528,6 +1490,7 @@
0.000000 MetaHookPre LoadFile(0, base<...>/null, <...>/null)
0.000000 MetaHookPre LoadFile(0, base<...>/numbers, <...>/numbers.zeek)
0.000000 MetaHookPre LoadFile(0, base<...>/option.bif, <...>/option.bif.zeek)
0.000000 MetaHookPre LoadFile(0, base<...>/options, <...>/options.zeek)
0.000000 MetaHookPre LoadFile(0, base<...>/packet_analysis.bif, <...>/packet_analysis.bif.zeek)
0.000000 MetaHookPre LoadFile(0, base<...>/pbb, <...>/pbb)
0.000000 MetaHookPre LoadFile(0, base<...>/plugins, <...>/plugins)
@ -1821,6 +1784,7 @@
0.000000 MetaHookPre LoadFileExtended(0, base<...>/null, <...>/null)
0.000000 MetaHookPre LoadFileExtended(0, base<...>/numbers, <...>/numbers.zeek)
0.000000 MetaHookPre LoadFileExtended(0, base<...>/option.bif, <...>/option.bif.zeek)
0.000000 MetaHookPre LoadFileExtended(0, base<...>/options, <...>/options.zeek)
0.000000 MetaHookPre LoadFileExtended(0, base<...>/packet_analysis.bif, <...>/packet_analysis.bif.zeek)
0.000000 MetaHookPre LoadFileExtended(0, base<...>/pbb, <...>/pbb)
0.000000 MetaHookPre LoadFileExtended(0, base<...>/plugins, <...>/plugins)
@ -1886,18 +1850,8 @@
0.000000 | HookCallFunction Analyzer::register_for_port(Analyzer::ANALYZER_HTTP, 81/tcp)
0.000000 | HookCallFunction Analyzer::register_for_port(Analyzer::ANALYZER_HTTP, 8888/tcp)
0.000000 | HookCallFunction Analyzer::register_for_ports(Analyzer::ANALYZER_HTTP, {80<...>/tcp})
0.000000 | HookCallFunction Broker::__set_metrics_export_endpoint_name()
0.000000 | HookCallFunction Broker::__set_metrics_export_interval(1.0 sec)
0.000000 | HookCallFunction Broker::__set_metrics_export_prefixes([])
0.000000 | HookCallFunction Broker::__set_metrics_export_topic()
0.000000 | HookCallFunction Broker::__set_metrics_import_topics([])
0.000000 | HookCallFunction Broker::__subscribe(zeek/supervisor)
0.000000 | HookCallFunction Broker::subscribe(zeek/supervisor)
0.000000 | HookCallFunction Broker::update_metrics_export_endpoint_name(Broker::metrics_export_endpoint_name, )
0.000000 | HookCallFunction Broker::update_metrics_export_interval(Broker::metrics_export_interval, 1.0 sec)
0.000000 | HookCallFunction Broker::update_metrics_export_prefixes(Broker::metrics_export_prefixes, [])
0.000000 | HookCallFunction Broker::update_metrics_export_topic(Broker::metrics_export_topic, )
0.000000 | HookCallFunction Broker::update_metrics_import_topics(Broker::metrics_import_topics, [])
0.000000 | HookCallFunction Cluster::is_enabled()
0.000000 | HookCallFunction Cluster::register_pool([topic=zeek<...>/logger, node_type=Cluster::LOGGER, max_nodes=<uninitialized>, exclusive=F])
0.000000 | HookCallFunction Cluster::register_pool([topic=zeek<...>/proxy, node_type=Cluster::PROXY, max_nodes=<uninitialized>, exclusive=F])
@ -1985,16 +1939,6 @@
0.000000 | HookCallFunction Option::set_change_handler(Analyzer::Logging::ignore_analyzers, Config::config_option_changed{ if (<skip-config-log> == Config::location) return (Config::new_value)Config::log = Config::Info($ts=network_time(), $id=Config::ID, $old_value=Config::format_value(lookup_ID(Config::ID)), $new_value=Config::format_value(Config::new_value))if ( != Config::location) Config::log$location = Config::locationLog::write(Config::LOG, to_any_coerce Config::log)return (Config::new_value)}, -100)
0.000000 | HookCallFunction Option::set_change_handler(Analyzer::Logging::include_confirmations, Config::config_option_changed{ if (<skip-config-log> == Config::location) return (Config::new_value)Config::log = Config::Info($ts=network_time(), $id=Config::ID, $old_value=Config::format_value(lookup_ID(Config::ID)), $new_value=Config::format_value(Config::new_value))if ( != Config::location) Config::log$location = Config::locationLog::write(Config::LOG, to_any_coerce Config::log)return (Config::new_value)}, -100)
0.000000 | HookCallFunction Option::set_change_handler(Analyzer::Logging::include_confirmations, lambda_<15261139872714441626>{ if (Analyzer::Logging::new_value) enable_event_group(Analyzer::Logging::include_confirmations)elsedisable_event_group(Analyzer::Logging::include_confirmations)return (Analyzer::Logging::new_value)}, 0)
0.000000 | HookCallFunction Option::set_change_handler(Broker::metrics_export_endpoint_name, Broker::update_metrics_export_endpoint_name{ Broker::__set_metrics_export_endpoint_name(Broker::val)return (Broker::val)}, 0)
0.000000 | HookCallFunction Option::set_change_handler(Broker::metrics_export_endpoint_name, Config::config_option_changed{ if (<skip-config-log> == Config::location) return (Config::new_value)Config::log = Config::Info($ts=network_time(), $id=Config::ID, $old_value=Config::format_value(lookup_ID(Config::ID)), $new_value=Config::format_value(Config::new_value))if ( != Config::location) Config::log$location = Config::locationLog::write(Config::LOG, to_any_coerce Config::log)return (Config::new_value)}, -100)
0.000000 | HookCallFunction Option::set_change_handler(Broker::metrics_export_interval, Broker::update_metrics_export_interval{ Broker::__set_metrics_export_interval(Broker::val)return (Broker::val)}, 0)
0.000000 | HookCallFunction Option::set_change_handler(Broker::metrics_export_interval, Config::config_option_changed{ if (<skip-config-log> == Config::location) return (Config::new_value)Config::log = Config::Info($ts=network_time(), $id=Config::ID, $old_value=Config::format_value(lookup_ID(Config::ID)), $new_value=Config::format_value(Config::new_value))if ( != Config::location) Config::log$location = Config::locationLog::write(Config::LOG, to_any_coerce Config::log)return (Config::new_value)}, -100)
0.000000 | HookCallFunction Option::set_change_handler(Broker::metrics_export_prefixes, Broker::update_metrics_export_prefixes{ Broker::__set_metrics_export_prefixes(Broker::filter)return (Broker::filter)}, 0)
0.000000 | HookCallFunction Option::set_change_handler(Broker::metrics_export_prefixes, Config::config_option_changed{ if (<skip-config-log> == Config::location) return (Config::new_value)Config::log = Config::Info($ts=network_time(), $id=Config::ID, $old_value=Config::format_value(lookup_ID(Config::ID)), $new_value=Config::format_value(Config::new_value))if ( != Config::location) Config::log$location = Config::locationLog::write(Config::LOG, to_any_coerce Config::log)return (Config::new_value)}, -100)
0.000000 | HookCallFunction Option::set_change_handler(Broker::metrics_export_topic, Broker::update_metrics_export_topic{ Broker::__set_metrics_export_topic(Broker::val)return (Broker::val)}, 0)
0.000000 | HookCallFunction Option::set_change_handler(Broker::metrics_export_topic, Config::config_option_changed{ if (<skip-config-log> == Config::location) return (Config::new_value)Config::log = Config::Info($ts=network_time(), $id=Config::ID, $old_value=Config::format_value(lookup_ID(Config::ID)), $new_value=Config::format_value(Config::new_value))if ( != Config::location) Config::log$location = Config::locationLog::write(Config::LOG, to_any_coerce Config::log)return (Config::new_value)}, -100)
0.000000 | HookCallFunction Option::set_change_handler(Broker::metrics_import_topics, Broker::update_metrics_import_topics{ Broker::__set_metrics_import_topics(Broker::topics)return (Broker::topics)}, 0)
0.000000 | HookCallFunction Option::set_change_handler(Broker::metrics_import_topics, Config::config_option_changed{ if (<skip-config-log> == Config::location) return (Config::new_value)Config::log = Config::Info($ts=network_time(), $id=Config::ID, $old_value=Config::format_value(lookup_ID(Config::ID)), $new_value=Config::format_value(Config::new_value))if ( != Config::location) Config::log$location = Config::locationLog::write(Config::LOG, to_any_coerce Config::log)return (Config::new_value)}, -100)
0.000000 | HookCallFunction Option::set_change_handler(Broker::peer_counts_as_iosource, Config::config_option_changed{ if (<skip-config-log> == Config::location) return (Config::new_value)Config::log = Config::Info($ts=network_time(), $id=Config::ID, $old_value=Config::format_value(lookup_ID(Config::ID)), $new_value=Config::format_value(Config::new_value))if ( != Config::location) Config::log$location = Config::locationLog::write(Config::LOG, to_any_coerce Config::log)return (Config::new_value)}, -100)
0.000000 | HookCallFunction Option::set_change_handler(Conn::analyzer_inactivity_timeouts, Config::config_option_changed{ if (<skip-config-log> == Config::location) return (Config::new_value)Config::log = Config::Info($ts=network_time(), $id=Config::ID, $old_value=Config::format_value(lookup_ID(Config::ID)), $new_value=Config::format_value(Config::new_value))if ( != Config::location) Config::log$location = Config::locationLog::write(Config::LOG, to_any_coerce Config::log)return (Config::new_value)}, -100)
0.000000 | HookCallFunction Option::set_change_handler(Conn::default_extract, Config::config_option_changed{ if (<skip-config-log> == Config::location) return (Config::new_value)Config::log = Config::Info($ts=network_time(), $id=Config::ID, $old_value=Config::format_value(lookup_ID(Config::ID)), $new_value=Config::format_value(Config::new_value))if ( != Config::location) Config::log$location = Config::locationLog::write(Config::LOG, to_any_coerce Config::log)return (Config::new_value)}, -100)
@ -2473,6 +2417,7 @@
0.000000 | HookLoadFile base<...>/null <...>/null
0.000000 | HookLoadFile base<...>/numbers <...>/numbers.zeek
0.000000 | HookLoadFile base<...>/option.bif <...>/option.bif.zeek
0.000000 | HookLoadFile base<...>/options <...>/options.zeek
0.000000 | HookLoadFile base<...>/packet_analysis.bif <...>/packet_analysis.bif.zeek
0.000000 | HookLoadFile base<...>/pbb <...>/pbb
0.000000 | HookLoadFile base<...>/plugins <...>/plugins
@ -2766,6 +2711,7 @@
0.000000 | HookLoadFileExtended base<...>/null <...>/null
0.000000 | HookLoadFileExtended base<...>/numbers <...>/numbers.zeek
0.000000 | HookLoadFileExtended base<...>/option.bif <...>/option.bif.zeek
0.000000 | HookLoadFileExtended base<...>/options <...>/options.zeek
0.000000 | HookLoadFileExtended base<...>/packet_analysis.bif <...>/packet_analysis.bif.zeek
0.000000 | HookLoadFileExtended base<...>/pbb <...>/pbb
0.000000 | HookLoadFileExtended base<...>/plugins <...>/plugins

16
testing/btest/Baseline/scripts.base.frameworks.logging.telemetry/telemetry.log
View file

@ -5,12 +5,12 @@
#unset_field -
#path telemetry
#open XXXX-XX-XX-XX-XX-XX
#fields ts peer metric_type prefix name unit labels label_values value
#types time string string string string string vector[string] vector[string] double
XXXXXXXXXX.XXXXXX zeek counter zeek log-stream-writes - module,stream HTTP,HTTP::LOG 14.0
XXXXXXXXXX.XXXXXX zeek counter zeek log-stream-writes - module,stream DNS,DNS::LOG 34.0
XXXXXXXXXX.XXXXXX zeek counter zeek log-stream-writes - module,stream Conn,Conn::LOG 34.0
XXXXXXXXXX.XXXXXX zeek counter zeek log-writer-writes - filter-name,module,path,stream,writer default,HTTP,http,HTTP::LOG,Log::WRITER_ASCII 10.0
XXXXXXXXXX.XXXXXX zeek counter zeek log-writer-writes - filter-name,module,path,stream,writer default,DNS,dns,DNS::LOG,Log::WRITER_ASCII 23.0
XXXXXXXXXX.XXXXXX zeek counter zeek log-writer-writes - filter-name,module,path,stream,writer default,Conn,conn,Conn::LOG,Log::WRITER_ASCII 30.0
#fields ts peer metric_type prefix name labels label_values value
#types time string string string string vector[string] vector[string] double
XXXXXXXXXX.XXXXXX zeek counter zeek zeek_log_stream_writes_total module,stream Conn,Conn::LOG 34.0
XXXXXXXXXX.XXXXXX zeek counter zeek zeek_log_stream_writes_total module,stream DNS,DNS::LOG 34.0
XXXXXXXXXX.XXXXXX zeek counter zeek zeek_log_stream_writes_total module,stream HTTP,HTTP::LOG 14.0
XXXXXXXXXX.XXXXXX zeek counter zeek zeek_log_writer_writes_total writer,module,stream,filter-name,path default,Conn,conn,Conn::LOG,Log::WRITER_ASCII 30.0
XXXXXXXXXX.XXXXXX zeek counter zeek zeek_log_writer_writes_total writer,module,stream,filter-name,path default,DNS,dns,DNS::LOG,Log::WRITER_ASCII 23.0
XXXXXXXXXX.XXXXXX zeek counter zeek zeek_log_writer_writes_total writer,module,stream,filter-name,path default,HTTP,http,HTTP::LOG,Log::WRITER_ASCII 10.0
#close XXXX-XX-XX-XX-XX-XX

34
testing/btest/Baseline/scripts.base.frameworks.telemetry.basic/out
View file

@ -1,25 +1,23 @@
### BTest baseline data generated by btest-diff. Do not edit. Use "btest -U/-u" to update. Requires BTest >= 0.63.
### zeek_session_metrics |2|
Telemetry::INT_GAUGE, zeek, active-sessions, [protocol], [tcp], 500.0
count_value, 500
Telemetry::INT_COUNTER, zeek, total-sessions, [protocol], [tcp], 500.0
count_value, 500
Telemetry::COUNTER, zeek, zeek_total_sessions_total, [protocol], [tcp], 500.0
Telemetry::GAUGE, zeek, zeek_active_sessions, [protocol], [tcp], 500.0
### bt* metrics |5|
Telemetry::DOUBLE_COUNTER, btest, a_test, [x, y], [a, b], 1.0
Telemetry::DOUBLE_COUNTER, btest, a_test, [x, y], [a, c], 2.0
Telemetry::DOUBLE_COUNTER, btest, b_test, [x, y], [a, b], 10.0
Telemetry::DOUBLE_COUNTER, btest, b_test, [x, y], [a, c], 20.0
Telemetry::DOUBLE_COUNTER, btest, c_test, [x, y], [a, b], 200.0
Telemetry::COUNTER, btest, btest_a_test_total, [x, y], [a, b], 1.0
Telemetry::COUNTER, btest, btest_b_test_total, [x, y], [a, b], 10.0
Telemetry::COUNTER, btest, btest_c_test_total, [x, y], [a, b], 200.0
Telemetry::COUNTER, btest, btest_a_test_total, [x, y], [a, c], 2.0
Telemetry::COUNTER, btest, btest_b_test_total, [x, y], [a, c], 20.0
### btest_a_metrics |2|
Telemetry::DOUBLE_COUNTER, btest, a_test, [x, y], [a, b], 1.0
Telemetry::DOUBLE_COUNTER, btest, a_test, [x, y], [a, c], 2.0
Telemetry::COUNTER, btest, btest_a_test_total, [x, y], [a, b], 1.0
Telemetry::COUNTER, btest, btest_a_test_total, [x, y], [a, c], 2.0
### btest_b_metrics |2|
Telemetry::DOUBLE_COUNTER, btest, b_test, [x, y], [a, b], 10.0
Telemetry::DOUBLE_COUNTER, btest, b_test, [x, y], [a, c], 20.0
Telemetry::COUNTER, btest, btest_b_test_total, [x, y], [a, b], 10.0
Telemetry::COUNTER, btest, btest_b_test_total, [x, y], [a, c], 20.0
### system_metrics |3|
Telemetry::DOUBLE_GAUGE, system, sensor_temperature, [name], [cpu0], 43.0
Telemetry::DOUBLE_GAUGE, system, sensor_temperature, [name], [cpu1], 44.1
Telemetry::DOUBLE_GAUGE, system, sensor_temperature, [name], [cpu3], 42.2
Telemetry::GAUGE, system, system_sensor_temperature_celsius, [name], [cpu0], 43.0
Telemetry::GAUGE, system, system_sensor_temperature_celsius, [name], [cpu1], 44.1
Telemetry::GAUGE, system, system_sensor_temperature_celsius, [name], [cpu3], 42.2
### btest_histogram_metrics |2|
Telemetry::DOUBLE_HISTOGRAM, btest, sample_histogram, [1.0, 2.0, 3.0, 4.0, 5.0, inf], [dim], [a], [2.0, 2.0, 0.0, 0.0, 0.0, 1.0], 11.5, 5.0
Telemetry::DOUBLE_HISTOGRAM, btest, sample_histogram, [1.0, 2.0, 3.0, 4.0, 5.0, inf], [dim], [b], [1.0, 0.0, 0.0, 0.0, 0.0, 1.0], 7.5, 2.0
Telemetry::HISTOGRAM, btest, btest_sample_histogram, [1.0, 2.0, 3.0, 4.0, 5.0, inf], [dim], [a], [2.0, 2.0, 0.0, 0.0, 0.0, 1.0], 11.5, 5.0
Telemetry::HISTOGRAM, btest, btest_sample_histogram, [1.0, 2.0, 3.0, 4.0, 5.0, inf], [dim], [b], [1.0, 0.0, 0.0, 0.0, 0.0, 1.0], 7.5, 2.0

4
testing/btest/Baseline/scripts.base.frameworks.telemetry.conn-duration-histogram/out
View file

@ -1,11 +1,11 @@
### BTest baseline data generated by btest-diff. Do not edit. Use "btest -U/-u" to update. Requires BTest >= 0.63.
Telemetry::DOUBLE_HISTOGRAM, zeek, connection_duration
Telemetry::HISTOGRAM, zeek, zeek_connection_duration_seconds
[]
[]
[2.0, 3.0, 4.0, 5.0, 6.0, 10.0, inf]
[0.0, 322.0, 90.0, 5.0, 76.0, 7.0, 0.0]
500.0, 1650.264644
Telemetry::DOUBLE_HISTOGRAM, zeek, realistic_connection_duration
Telemetry::HISTOGRAM, zeek, zeek_realistic_connection_duration_seconds
[proto]
[tcp]
[0.1, 1.0, 10.0, 30.0, 60.0, 120.0, 300.0, 900.0, 1800.0, inf]

6
testing/btest/Baseline/scripts.base.frameworks.telemetry.event-handler-invocations/out
View file

@ -1,4 +1,4 @@
### BTest baseline data generated by btest-diff. Do not edit. Use "btest -U/-u" to update. Requires BTest >= 0.63.
zeek, event-handler-invocations, [zeek_init], 1
zeek, event-handler-invocations, [connection_state_remove], 500
zeek, event-handler-invocations, [zeek_done], 1
zeek, zeek_event_handler_invocations_total, [connection_state_remove], 500.0
zeek, zeek_event_handler_invocations_total, [zeek_done], 1.0
zeek, zeek_event_handler_invocations_total, [zeek_init], 1.0

8
testing/btest/Baseline/scripts.base.utils.json/output
View file

@ -42,7 +42,7 @@ XXXXXXXXXX.XXXXXX
{"10.1.1.1":[1,2],"10.2.2.2":[3,5]}
{"1":{"s":"test"}}
{"opaque_type":"TopkVal"}
{"_metric":{"opaque_type":"DblGaugeMetricVal"}}
{"_family":{"opaque_type":"DblGaugeMetricFamilyVal"},"_labels":["dim_1"]}
{"_metric":{"opaque_type":"DblCounterMetricVal"}}
{"_family":{"opaque_type":"DblCounterMetricFamilyVal"},"_labels":["dim_1"]}
{"_metric":{"opaque_type":"GaugeMetricVal"}}
{"_family":{"opaque_type":"GaugeMetricFamilyVal"},"_labels":["dim_1"]}
{"_metric":{"opaque_type":"CounterMetricVal"}}
{"_family":{"opaque_type":"CounterMetricFamilyVal"},"_labels":["dim_1"]}

6
testing/btest/Baseline/scripts.policy.frameworks.telemetry.log-prefixes/telemetry.log
View file

@ -5,7 +5,7 @@
#unset_field -
#path telemetry
#open XXXX-XX-XX-XX-XX-XX
#fields ts peer metric_type prefix name unit labels label_values value
#types time string string string string string vector[string] vector[string] double
XXXXXXXXXX.XXXXXX zeek counter btest connections - proto tcp 500.0
#fields ts peer metric_type prefix name labels label_values value
#types time string string string string vector[string] vector[string] double
XXXXXXXXXX.XXXXXX zeek counter btest btest_connections_total proto tcp 500.0
#close XXXX-XX-XX-XX-XX-XX

8
testing/btest/Baseline/scripts.policy.frameworks.telemetry.log-prefixes/telemetry_histogram.log
View file

@ -5,8 +5,8 @@
#unset_field -
#path telemetry_histogram
#open XXXX-XX-XX-XX-XX-XX
#fields ts peer prefix name unit labels label_values bounds values sum observations
#types time string string string string vector[string] vector[string] vector[double] vector[double] double double
XXXXXXXXXX.XXXXXX zeek btest connection_duration seconds (empty) (empty) 2.0,3.0,4.0,5.0,6.0,10.0,inf 0.0,0.0,0.0,0.0,0.0,0.0,0.0 0.0 0.0
XXXXXXXXXX.XXXXXX zeek btest connection_duration seconds (empty) (empty) 2.0,3.0,4.0,5.0,6.0,10.0,inf 0.0,322.0,90.0,5.0,76.0,7.0,0.0 1650.264644 500.0
#fields ts peer prefix name labels label_values bounds values sum observations
#types time string string string vector[string] vector[string] vector[double] vector[double] double double
XXXXXXXXXX.XXXXXX zeek btest btest_connection_duration_seconds (empty) (empty) 2.0,3.0,4.0,5.0,6.0,10.0,inf 0.0,0.0,0.0,0.0,0.0,0.0,0.0 0.0 0.0
XXXXXXXXXX.XXXXXX zeek btest btest_connection_duration_seconds (empty) (empty) 2.0,3.0,4.0,5.0,6.0,10.0,inf 0.0,322.0,90.0,5.0,76.0,7.0,0.0 1650.264644 500.0
#close XXXX-XX-XX-XX-XX-XX

8
testing/btest/Baseline/scripts.policy.frameworks.telemetry.log/telemetry.log.filtered
View file

@ -1,5 +1,5 @@
### BTest baseline data generated by btest-diff. Do not edit. Use "btest -U/-u" to update. Requires BTest >= 0.63.
XXXXXXXXXX.XXXXXX zeek gauge zeek active-sessions - protocol tcp 1.0
XXXXXXXXXX.XXXXXX zeek counter zeek total-sessions - protocol tcp 1.0
XXXXXXXXXX.XXXXXX zeek gauge zeek active-sessions - protocol tcp 500.0
XXXXXXXXXX.XXXXXX zeek counter zeek total-sessions - protocol tcp 500.0
XXXXXXXXXX.XXXXXX zeek counter zeek zeek_total_sessions_total protocol tcp 1.0
XXXXXXXXXX.XXXXXX zeek gauge zeek zeek_active_sessions protocol tcp 1.0
XXXXXXXXXX.XXXXXX zeek counter zeek zeek_total_sessions_total protocol tcp 500.0
XXXXXXXXXX.XXXXXX zeek gauge zeek zeek_active_sessions protocol tcp 500.0

4
testing/btest/Baseline/scripts.policy.frameworks.telemetry.log/telemetry_histogram.log.filtered
View file

@ -1,3 +1,3 @@
### BTest baseline data generated by btest-diff. Do not edit. Use "btest -U/-u" to update. Requires BTest >= 0.63.
XXXXXXXXXX.XXXXXX zeek zeek connection_duration seconds (empty) (empty) 2.0,3.0,4.0,5.0,6.0,10.0,inf 0.0,0.0,0.0,0.0,0.0,0.0,0.0 0.0 0.0
XXXXXXXXXX.XXXXXX zeek zeek connection_duration seconds (empty) (empty) 2.0,3.0,4.0,5.0,6.0,10.0,inf 0.0,322.0,90.0,5.0,76.0,7.0,0.0 1650.264644 500.0
XXXXXXXXXX.XXXXXX zeek zeek zeek_connection_duration_seconds (empty) (empty) 2.0,3.0,4.0,5.0,6.0,10.0,inf 0.0,0.0,0.0,0.0,0.0,0.0,0.0 0.0 0.0
XXXXXXXXXX.XXXXXX zeek zeek zeek_connection_duration_seconds (empty) (empty) 2.0,3.0,4.0,5.0,6.0,10.0,inf 0.0,322.0,90.0,5.0,76.0,7.0,0.0 1650.264644 500.0

3
testing/btest/Baseline/scripts.policy.frameworks.telemetry.prometheus/manager-1..stdout
View file

@ -1,3 +0,0 @@
### BTest baseline data generated by btest-diff. Do not edit. Use "btest -U/-u" to update. Requires BTest >= 0.63.
manager-1, original Broker::metrics_port, 9911/tcp
[endpoint="logger-1", endpoint="manager-1", endpoint="proxy-1", endpoint="worker-1"]

5
testing/btest/Baseline/scripts.policy.frameworks.telemetry.prometheus/manager-1.services.out Normal file
View file

@ -0,0 +1,5 @@
### BTest baseline data generated by btest-diff. Do not edit. Use "btest -U/-u" to update. Requires BTest >= 0.63.
endpoint="manager-1"
endpoint="logger-1"
endpoint="proxy-1"
endpoint="worker-1"

2
testing/btest/Baseline/supervisor.large-cluster/zeek.bare-1.node.out
View file

@ -1,5 +1,5 @@
### BTest baseline data generated by btest-diff. Do not edit. Use "btest -U/-u" to update. Requires BTest >= 0.63.
supervised node zeek_init()
1024, cluster_nodes!
[node_type=Cluster::WORKER, ip=127.0.0.1, zone_id=, p=0/tcp, interface=<uninitialized>, manager=<uninitialized>, time_machine=<uninitialized>, id=<uninitialized>]
[node_type=Cluster::WORKER, ip=127.0.0.1, zone_id=, p=0/tcp, interface=<uninitialized>, manager=<uninitialized>, time_machine=<uninitialized>, id=<uninitialized>, metrics_port=<uninitialized>]
supervised node zeek_done()

2
testing/btest/Baseline/supervisor.large-cluster/zeek.bare-32.node.out
View file

@ -1,5 +1,5 @@
### BTest baseline data generated by btest-diff. Do not edit. Use "btest -U/-u" to update. Requires BTest >= 0.63.
supervised node zeek_init()
1024, cluster_nodes!
[node_type=Cluster::WORKER, ip=127.0.0.1, zone_id=, p=0/tcp, interface=<uninitialized>, manager=<uninitialized>, time_machine=<uninitialized>, id=<uninitialized>]
[node_type=Cluster::WORKER, ip=127.0.0.1, zone_id=, p=0/tcp, interface=<uninitialized>, manager=<uninitialized>, time_machine=<uninitialized>, id=<uninitialized>, metrics_port=<uninitialized>]
supervised node zeek_done()

6
testing/btest/Baseline/telemetry.counter/output
View file

@ -1,5 +1,3 @@
### BTest baseline data generated by btest-diff. Do not edit. Use "btest -U/-u" to update. Requires BTest >= 0.63.
cnt1_bar: 1
cnt2_bar: 42
cnt3_bar: 1.000000
cnt4_bar: 42.000000
cnt1_bar: 1.000000
cnt2_bar: 42.000000

6
testing/btest/Baseline/telemetry.gauge/output
View file

@ -1,5 +1,3 @@
### BTest baseline data generated by btest-diff. Do not edit. Use "btest -U/-u" to update. Requires BTest >= 0.63.
gg1_bar: 1
gg2_bar: 23
gg3_bar: 1.000000
gg4_bar: 23.000000
gg1_bar: 1.000000
gg2_bar: 23.000000

6
testing/btest/Baseline/telemetry.histogram/output
View file

@ -1,5 +1,3 @@
### BTest baseline data generated by btest-diff. Do not edit. Use "btest -U/-u" to update. Requires BTest >= 0.63.
hst1_bar: 12
hst2_bar: 31337
hst3_bar: 6.000000
hst4_bar: 64.000000
hst1_bar: 6.000000
hst2_bar: 64.000000

2
testing/btest/scripts/base/frameworks/logging/telemetry.zeek
View file

@ -28,7 +28,7 @@ hook Log::log_stream_policy(rec: any, id: Log::ID)
hook Telemetry::log_policy(rec: Telemetry::Info, id: Log::ID, filter: Log::Filter)
{
if ( rec$prefix != "zeek" || /^log-/ !in rec$name )
if ( rec$prefix != "zeek" || /^zeek_log_/ !in rec$name )
break;
if ( /HTTP|DNS|Conn/ !in cat(rec$label_values) )

13
testing/btest/scripts/base/frameworks/telemetry/basic.zeek
View file

@ -8,10 +8,12 @@
@load base/frameworks/telemetry
redef running_under_test = T;
global btest_a_cf = Telemetry::register_counter_family([
$prefix="btest",
$name="a_test",
$unit="1",
$unit="",
$help_text="A btest metric",
$labels=vector("x", "y")
]);
@ -19,7 +21,7 @@ global btest_a_cf = Telemetry::register_counter_family([
global btest_b_cf = Telemetry::register_counter_family([
$prefix="btest",
$name="b_test",
$unit="1",
$unit="",
$help_text="Another btest metric",
$labels=vector("x", "y")
]);
@ -27,7 +29,7 @@ global btest_b_cf = Telemetry::register_counter_family([
global btest_c_cf = Telemetry::register_counter_family([
$prefix="btest",
$name="c_test",
$unit="1",
$unit="",
$help_text="The last btest metric",
$labels=vector("x", "y")
]);
@ -43,7 +45,7 @@ global system_sensor_temp_gf = Telemetry::register_gauge_family([
global btest_sample_histogram_hf = Telemetry::register_histogram_family([
$prefix="btest",
$name="sample_histogram",
$unit="1",
$unit="",
$help_text="A sample histogram that is not returned by Telemetry::collect_metrics",
$bounds=vector(1.0, 2.0, 3.0, 4.0, 5.0),
$labels=vector("dim")
@ -56,9 +58,6 @@ function print_metrics(what: string, metrics: vector of Telemetry::Metric)
{
local m = metrics[i];
print m$opts$metric_type, m$opts$prefix, m$opts$name, m$opts$labels, m$labels, m$value;
if (m?$count_value)
print "count_value", m$count_value;
}
}

5
testing/btest/scripts/base/frameworks/telemetry/event-handler-invocations.zeek
View file

@ -9,13 +9,14 @@
@load base/frameworks/telemetry
redef running_under_test = T;
event zeek_done() &priority=-100
{
local ms = Telemetry::collect_metrics("zeek", "event-handler-invocations");
local ms = Telemetry::collect_metrics("zeek", "event_handler_invocations");
for ( _, m in ms )
{
if ( /zeek_.*|connection_.*/ in cat(m$labels))
print m$opts$prefix, m$opts$name, m$labels, m$count_value;
print m$opts$prefix, m$opts$name, m$labels, m$value;
}
}

1
testing/btest/scripts/base/frameworks/telemetry/internal-metrics.zeek
View file

@ -1,6 +1,7 @@
# @TEST-DOC: Query some internal broker/caf related metrics as they use the int64_t versions, too.
# Note compilable to C++ due to globals being initialized to a record that
# has an opaque type as a field.
# @TEST-KNOWN-FAILURE: Implementation for prometheus-cpp missing in broker
# @TEST-REQUIRES: test "${ZEEK_USE_CPP}" != "1"
# @TEST-EXEC: zcat <$TRACES/echo-connections.pcap.gz | zeek -b -Cr - %INPUT > out
# @TEST-EXEC: btest-diff out

4
testing/btest/scripts/base/utils/json.test
View file

@ -139,7 +139,7 @@ event zeek_init()
local gauge_family = Telemetry::register_gauge_family([
$prefix="btest",
$name="btest_testing_gauge",
$unit="1",
$unit="",
$help_text="Btest testing",
$labels=vector("dim_1"),
]);
@ -150,7 +150,7 @@ event zeek_init()
local counter_family = Telemetry::register_counter_family([
$prefix="btest",
$name="btest_testing_counter",
$unit="1",
$unit="",
$help_text="Btest testing",
$labels=vector("dim_1"),
]);

2
testing/btest/scripts/policy/frameworks/telemetry/log-prefixes.zeek
View file

@ -11,7 +11,7 @@ redef Telemetry::log_prefixes = {"btest"};
global connections_by_proto_cf = Telemetry::register_counter_family([
$prefix="btest",
$name="connections",
$unit="1",
$unit="",
$help_text="Total number of monitored connections",
$labels=vector("proto")
]);

88
testing/btest/scripts/policy/frameworks/telemetry/prometheus.zeek
View file

@ -1,4 +1,4 @@
# @TEST-DOC: Query the Prometheus endpoint on 9911 and smoke check that zeek_version_info{...} is contained in the response for all cluster nodes.
# @TEST-DOC: Query the Prometheus endpoint and smoke check that zeek_version_info{...} is contained in the response for all cluster nodes.
# Note compilable to C++ due to globals being initialized to a record that
# has an opaque type as a field.
# @TEST-REQUIRES: test "${ZEEK_USE_CPP}" != "1"
@ -7,8 +7,10 @@
# @TEST-PORT: BROKER_PORT2
# @TEST-PORT: BROKER_PORT3
# @TEST-PORT: BROKER_PORT4
# @TEST-PORT: BROKER_PORT4
# @TEST-PORT: BROKER_TEST_METRICS_PORT
# @TEST-PORT: METRICS_PORT1
# @TEST-PORT: METRICS_PORT2
# @TEST-PORT: METRICS_PORT3
# @TEST-PORT: METRICS_PORT4
#
# @TEST-REQUIRES: which curl
# @TEST-EXEC: zeek --parse-only %INPUT
@ -16,73 +18,89 @@
# @TEST-EXEC: btest-bg-run logger-1 ZEEKPATH=$ZEEKPATH:.. CLUSTER_NODE=logger-1 zeek -b %INPUT
# @TEST-EXEC: btest-bg-run proxy-1 ZEEKPATH=$ZEEKPATH:.. CLUSTER_NODE=proxy-1 zeek -b %INPUT
# @TEST-EXEC: btest-bg-run worker-1 ZEEKPATH=$ZEEKPATH:.. CLUSTER_NODE=worker-1 zeek -b %INPUT
# @TEST-EXEC: btest-bg-wait 10
# @TEST-EXEC: btest-diff manager-1/.stdout
# @TEST-EXEC: btest-bg-wait 30
# @TEST-EXEC: btest-diff manager-1/services.out
@TEST-START-FILE cluster-layout.zeek
redef Cluster::nodes = {
["manager-1"] = [$node_type=Cluster::MANAGER, $ip=127.0.0.1, $p=to_port(getenv("BROKER_PORT1"))],
["logger-1"] = [$node_type=Cluster::LOGGER, $ip=127.0.0.1, $p=to_port(getenv("BROKER_PORT2")), $manager="manager-1"],
["proxy-1"] = [$node_type=Cluster::PROXY, $ip=127.0.0.1, $p=to_port(getenv("BROKER_PORT3")), $manager="manager-1"],
["worker-1"] = [$node_type=Cluster::WORKER, $ip=127.0.0.1, $p=to_port(getenv("BROKER_PORT4")), $manager="manager-1"],
["manager-1"] = [$node_type=Cluster::MANAGER, $ip=127.0.0.1, $p=to_port(getenv("BROKER_PORT1")), $metrics_port=to_port(getenv("METRICS_PORT1"))],
["logger-1"] = [$node_type=Cluster::LOGGER, $ip=127.0.0.1, $p=to_port(getenv("BROKER_PORT2")), $manager="manager-1", $metrics_port=to_port(getenv("METRICS_PORT2"))],
["proxy-1"] = [$node_type=Cluster::PROXY, $ip=127.0.0.1, $p=to_port(getenv("BROKER_PORT3")), $manager="manager-1", $metrics_port=to_port(getenv("METRICS_PORT3"))],
["worker-1"] = [$node_type=Cluster::WORKER, $ip=127.0.0.1, $p=to_port(getenv("BROKER_PORT4")), $manager="manager-1", $metrics_port=to_port(getenv("METRICS_PORT4"))],
};
@TEST-END-FILE
@TEST-START-FILE request-services.sh
#! /usr/bin/env bash
# This script makes repeat curl requests to find all of the metrics data from the
# hosts listed in the services output from the manager, and outputs it all into a
# single file.
services_url=$1
output_file=$2
services_data=$(curl -s -m 5 ${services_url})
for host in $(echo ${services_data} | jq -r '.[0].targets[]' | sort); do
metrics=$(curl -m 5 --trace trace-${host}.out http://${host}/metrics)
if [ $? -eq 0 ] ; then
version_info=$(echo ${metrics} | grep -Eo "zeek_version_info\{[^}]+\}" | grep -o 'endpoint="[^"]*"')
echo ${version_info} >> ${output_file};
else
echo "Failed to request data from ${host}" >> ${output_file}
fi
done
@TEST-END-FILE
@load policy/frameworks/cluster/experimental
@load policy/frameworks/telemetry/prometheus
@load base/frameworks/telemetry
@load base/utils/active-http
# Query the Prometheus endpoint using ActiveHTTP for testing, oh my.
# So the cluster nodes don't terminate right away.
#redef exit_only_after_terminate=T;
@if ( Cluster::node == "manager-1" )
@load base/utils/exec
# Query the Prometheus endpoint using curl for testing, oh my.
event run_test()
{
local url = fmt("http://localhost:%s/metrics", port_to_count(Broker::metrics_port));
when [url] ( local response = ActiveHTTP::request([$url=url]) )
local services_url = fmt("http://localhost:%s/services.json", port_to_count(Telemetry::metrics_port));
local req_cmd = fmt("sh ../request-services.sh %s %s", services_url, "services.out");
when [req_cmd] ( local result = Exec::run([$cmd=req_cmd]) )
{
if ( response$code != 200 )
if ( result$exit_code != 0 )
{
print fmt("ERROR: %s", response);
# This is bad.
print "ERROR: Failed to request service information";
exit(1);
}
# Grumble grumble, ActiveHTTP actually joins away the \n characters
# from the response. Not sure how that's helpful. We simply
# grep out the zeek_version_info{...} endpoint="..." pieces and
# expect one for each node to exist as a smoke test.
local version_infos = find_all(response$body, /zeek_version_info\{[^}]+\}/, 0);
local endpoints: vector of string;
for ( info in version_infos )
for ( ep in find_all(info, /endpoint=\"[^"]+\"/))
endpoints += ep;
print sort(endpoints, strcmp);
terminate();
}
timeout 10sec
{
# This is bad.
print "ERROR: HTTP request timeout";
print "ERROR: Timed out requesting service information";
exit(1);
}
}
@if ( Cluster::node == "manager-1" )
# Use a dynamic metrics port for testing to avoid colliding on 9911/tcp
# when running tests in parallel.
global orig_metrics_port = Broker::metrics_port;
redef Broker::metrics_port = to_port(getenv("BROKER_TEST_METRICS_PORT"));
event zeek_init()
{
print Cluster::node, "original Broker::metrics_port", orig_metrics_port;
print Cluster::node, "Telemetry::metrics_port from cluster config", Telemetry::metrics_port;
}
event Cluster::Experimental::cluster_started()
{
# Run the test once all nodes are up and metrics_export_interval
# has passed at least once.
schedule 2 * Broker::metrics_export_interval { run_test() };
# Run the test once all nodes are up
schedule 2 secs { run_test() };
}
@endif

27
testing/btest/telemetry/counter.zeek
View file

@ -3,25 +3,16 @@
# @TEST-EXEC: zeek -b %INPUT >output
# @TEST-EXEC: btest-diff output
global cnt1 = Telemetry::__int_counter_family("cnt1", "bar", vector("dim1", "dim2"));
global cnt2 = Telemetry::__int_counter_family("cnt2", "bar", vector());
global cnt3 = Telemetry::__dbl_counter_family("cnt3", "bar", vector("dim1", "dim2"));
global cnt4 = Telemetry::__dbl_counter_family("cnt4", "bar", vector());
global cnt1 = Telemetry::__counter_family("cnt1", "bar", vector("dim1", "dim2"));
global cnt2 = Telemetry::__counter_family("cnt2", "bar", vector());
event zeek_init()
{
local cnt1_bar = Telemetry::__int_counter_metric_get_or_add(cnt1, table(["dim1"] = "val1", ["dim2"] = "val2"));
Telemetry::__int_counter_inc(cnt1_bar);
local cnt2_bar = Telemetry::__int_counter_metric_get_or_add(cnt2, table());
Telemetry::__int_counter_inc(cnt2_bar);
Telemetry::__int_counter_inc(cnt2_bar, 41);
print fmt("cnt1_bar: %d", Telemetry::__int_counter_value(cnt1_bar));
print fmt("cnt2_bar: %d", Telemetry::__int_counter_value(cnt2_bar));
local cnt3_bar = Telemetry::__dbl_counter_metric_get_or_add(cnt3, table(["dim1"] = "val1", ["dim2"] = "val2"));
Telemetry::__dbl_counter_inc(cnt3_bar);
local cnt4_bar = Telemetry::__dbl_counter_metric_get_or_add(cnt4, table());
Telemetry::__dbl_counter_inc(cnt4_bar);
Telemetry::__dbl_counter_inc(cnt4_bar, 41.0);
print fmt("cnt3_bar: %f", Telemetry::__dbl_counter_value(cnt3_bar));
print fmt("cnt4_bar: %f", Telemetry::__dbl_counter_value(cnt4_bar));
local cnt1_bar = Telemetry::__counter_metric_get_or_add(cnt1, table(["dim1"] = "val1", ["dim2"] = "val2"));
Telemetry::__counter_inc(cnt1_bar);
local cnt2_bar = Telemetry::__counter_metric_get_or_add(cnt2, table());
Telemetry::__counter_inc(cnt2_bar);
Telemetry::__counter_inc(cnt2_bar, 41);
print fmt("cnt1_bar: %f", Telemetry::__counter_value(cnt1_bar));
print fmt("cnt2_bar: %f", Telemetry::__counter_value(cnt2_bar));
}

34
testing/btest/telemetry/gauge.zeek
View file

@ -3,30 +3,18 @@
# @TEST-EXEC: zeek -b %INPUT >output
# @TEST-EXEC: btest-diff output
global gg1 = Telemetry::__int_gauge_family("gg1", "bar", vector("dim1", "dim2"));
global gg2 = Telemetry::__int_gauge_family("gg2", "bar", vector());
global gg3 = Telemetry::__dbl_gauge_family("gg3", "bar", vector("dim1", "dim2"));
global gg4 = Telemetry::__dbl_gauge_family("gg4", "bar", vector());
global gg1 = Telemetry::__gauge_family("gg1", "bar", vector("dim1", "dim2"));
global gg2 = Telemetry::__gauge_family("gg2", "bar", vector());
event zeek_init()
{
local gg1_bar = Telemetry::__int_gauge_metric_get_or_add(gg1, table(["dim1"] = "val1", ["dim2"] = "val2"));
Telemetry::__int_gauge_inc(gg1_bar);
local gg2_bar = Telemetry::__int_gauge_metric_get_or_add(gg2, table());
Telemetry::__int_gauge_inc(gg2_bar);
Telemetry::__int_gauge_inc(gg2_bar, 41);
Telemetry::__int_gauge_dec(gg2_bar);
Telemetry::__int_gauge_dec(gg2_bar, 18);
print fmt("gg1_bar: %d", Telemetry::__int_gauge_value(gg1_bar));
print fmt("gg2_bar: %d", Telemetry::__int_gauge_value(gg2_bar));
local gg3_bar = Telemetry::__dbl_gauge_metric_get_or_add(gg3, table(["dim1"] = "val1", ["dim2"] = "val2"));
Telemetry::__dbl_gauge_inc(gg3_bar);
local gg4_bar = Telemetry::__dbl_gauge_metric_get_or_add(gg4, table());
Telemetry::__dbl_gauge_inc(gg4_bar);
Telemetry::__dbl_gauge_inc(gg4_bar, 41.0);
Telemetry::__dbl_gauge_dec(gg4_bar);
Telemetry::__dbl_gauge_dec(gg4_bar, 18.0);
print fmt("gg3_bar: %f", Telemetry::__dbl_gauge_value(gg3_bar));
print fmt("gg4_bar: %f", Telemetry::__dbl_gauge_value(gg4_bar));
local gg1_bar = Telemetry::__gauge_metric_get_or_add(gg1, table(["dim1"] = "val1", ["dim2"] = "val2"));
Telemetry::__gauge_inc(gg1_bar);
local gg2_bar = Telemetry::__gauge_metric_get_or_add(gg2, table());
Telemetry::__gauge_inc(gg2_bar);
Telemetry::__gauge_inc(gg2_bar, 41.0);
Telemetry::__gauge_dec(gg2_bar);
Telemetry::__gauge_dec(gg2_bar, 18.0);
print fmt("gg1_bar: %f", Telemetry::__gauge_value(gg1_bar));
print fmt("gg2_bar: %f", Telemetry::__gauge_value(gg2_bar));
}

28
testing/btest/telemetry/histogram.zeek
View file

@ -3,28 +3,18 @@
# @TEST-EXEC: zeek -b %INPUT >output
# @TEST-EXEC: btest-diff output
const int_bounds = vector(+10, +20);
const dbl_bounds = vector(10.0, 20.0);
global hst1 = Telemetry::__int_histogram_family("hst1", "bar", vector("dim1", "dim2"), int_bounds);
global hst2 = Telemetry::__int_histogram_family("hst2", "bar", vector(), int_bounds);
global hst3 = Telemetry::__dbl_histogram_family("hst3", "bar", vector("dim1", "dim2"), dbl_bounds);
global hst4 = Telemetry::__dbl_histogram_family("hst4", "bar", vector(), dbl_bounds);
global hst1 = Telemetry::__histogram_family("hst1", "bar", vector("dim1", "dim2"), dbl_bounds);
global hst2 = Telemetry::__histogram_family("hst2", "bar", vector(), dbl_bounds);
event zeek_init()
{
local hst1_bar = Telemetry::__int_histogram_metric_get_or_add(hst1, table(["dim1"] = "val1", ["dim2"] = "val2"));
Telemetry::__int_histogram_observe(hst1_bar, 1);
Telemetry::__int_histogram_observe(hst1_bar, 11);
local hst2_bar = Telemetry::__int_histogram_metric_get_or_add(hst2, table());
Telemetry::__int_histogram_observe(hst2_bar, 31337);
print fmt("hst1_bar: %d", Telemetry::__int_histogram_sum(hst1_bar));
print fmt("hst2_bar: %d", Telemetry::__int_histogram_sum(hst2_bar));
local hst3_bar = Telemetry::__dbl_histogram_metric_get_or_add(hst3, table(["dim1"] = "val1", ["dim2"] = "val2"));
Telemetry::__dbl_histogram_observe(hst3_bar, 2.0);
Telemetry::__dbl_histogram_observe(hst3_bar, 4.0);
local hst4_bar = Telemetry::__dbl_histogram_metric_get_or_add(hst4, table());
Telemetry::__dbl_histogram_observe(hst4_bar, 64.0);
print fmt("hst3_bar: %f", Telemetry::__dbl_histogram_sum(hst3_bar));
print fmt("hst4_bar: %f", Telemetry::__dbl_histogram_sum(hst4_bar));
local hst1_bar = Telemetry::__histogram_metric_get_or_add(hst1, table(["dim1"] = "val1", ["dim2"] = "val2"));
Telemetry::__histogram_observe(hst1_bar, 2.0);
Telemetry::__histogram_observe(hst1_bar, 4.0);
local hst2_bar = Telemetry::__histogram_metric_get_or_add(hst2, table());
Telemetry::__histogram_observe(hst2_bar, 64.0);
print fmt("hst1_bar: %f", Telemetry::__histogram_sum(hst1_bar));
print fmt("hst2_bar: %f", Telemetry::__histogram_sum(hst2_bar));
}