mirror of
https://github.com/zeek/zeek.git
synced 2025-10-02 06:38:20 +00:00
Merge branch 'zeek:master' into master
This commit is contained in:
Fatema BW
GitHub
commit
d9632631ce
243 changed files with 44421 additions and 20773 deletions
263
CHANGES
263
CHANGES
|
|
@ -1,3 +1,266 @@
|
|||
5.1.0-dev.386 | 2022-08-11 11:56:55 -0700
|
||||
|
||||
* Add a field to Modbus/TCP log to indicate the Modbus PDU type (Michael Torres)
|
||||
|
||||
Add the `pdu_type` field to Modbus over TCP logs to indicate whether the Modbus
|
||||
message was a request or a response. Due to the client/server nature of Modbus
|
||||
over TCP/IP, all messages from the TCP session originator are requests, while
|
||||
all messages from the TCP session responder are responses.
|
||||
|
||||
Adding this information to the default log surfaces protocol metadata in a way
|
||||
that doesn't require users to understand the Modbus over TCP protocol.
|
||||
|
||||
* Add modbus transaction and unit ids to logs (Michael Torres)
|
||||
|
||||
Add transaction IDs and unit IDs to default modbus over TCP/IP logs.
|
||||
Update the relevant testing baselines to account for the extra fields.
|
||||
|
||||
* Enable modbus logging for requests (Michael Torres)
|
||||
|
||||
5.1.0-dev.382 | 2022-08-11 10:41:08 -0700
|
||||
|
||||
* UPDATED: improving email address splitting for common comma case (TheAvgJojo)
|
||||
|
||||
5.1.0-dev.380 | 2022-08-11 10:30:33 -0700
|
||||
|
||||
* Fix a crash related to a broken IPv6 chain (Tim Wojtulewicz)
|
||||
|
||||
* Add a couple of ICMP files to corpus for packet-fuzzer (Tim Wojtulewicz, Corelight)
|
||||
|
||||
* Trick event handlers into returning that they exist during fuzzing (Tim Wojtulewicz, Corelight)
|
||||
|
||||
* Add http, ftp, imap, and smtp fuzzers and corpora (Tim Wojtulewicz, Corelight)
|
||||
|
||||
* Add section to fuzzer README about generating corpus from pcaps (Tim Wojtulewicz, Corelight)
|
||||
|
||||
* Rename fuzzers/README to README.rst so github renders it (Tim Wojtulewicz, Corelight)
|
||||
|
||||
5.1.0-dev.373 | 2022-08-11 08:49:22 -0700
|
||||
|
||||
* Add table_keys function (AmazingPP)
|
||||
|
||||
* Add table_values function (AmazingPP)
|
||||
|
||||
5.1.0-dev.368 | 2022-08-11 16:01:32 +0200
|
||||
|
||||
* GH-1678: Introduce global disabling_analyzer() hook to veto disable_analyzer() (Arne Welzel, Corelight)
|
||||
|
||||
This hook can be used to coordinate disabling an analyzer for a
|
||||
given connection. The contract is simple: Any script can veto a
|
||||
disable_analyzer() call by breaking from this hook. The decision
|
||||
is local to the script taking into account any state attached to
|
||||
the connection object or script specific state stored elsewhere. A
|
||||
script breaking from the hook takes over the responsibility to
|
||||
call disable_analyzer() at a later point when it finds the
|
||||
condition due to which it vetoed fulfilled (which may be never).
|
||||
|
||||
Signature:
|
||||
|
||||
disabling_analyzer: hook(c: connection, atype: AllAnalyzers::Tag, aid: count);
|
||||
|
||||
Example use-cases are keeping the SSL analyzer enabled for
|
||||
finger-printing until a certain amount of bytes or packets have
|
||||
been transferred or similarly the connection duration exceed a
|
||||
certain threshold.
|
||||
|
||||
Other example use-cases might be keeping analyzers for SSH, RDP or SSL
|
||||
enabled for connections from specific subnets.
|
||||
|
||||
* ssl: Only delete `c$ssl$analyzer_id` when disabling the analyzer
|
||||
was successful (Arne Welzel, Corelight)
|
||||
|
||||
5.1.0-dev.364 | 2022-08-11 11:59:00 +0200
|
||||
|
||||
* GH-2000: Support redef'ing the &log attribute of record fields (Arne Welzel, Corelight)
|
||||
|
||||
Add new syntax for adding and removing attributes from record fields:
|
||||
|
||||
redef RecordType$field_name += { &log };
|
||||
redef RecordType$field_name -= { &log };
|
||||
|
||||
For now this only allowed for the &log attribute as the semantics are clear.
|
||||
For &default and &optional the semantics aren't obvious and no use-cases have
|
||||
been identified where those would make sense to change.
|
||||
|
||||
This enables a mechanism to add potentially interesting fields to the typical
|
||||
Info records in base scripts, but letting users opt-into actually including
|
||||
them into their log. At the same time, users that find specific fields in a
|
||||
standard log uninteresting can opt-out without using `Log::Filter$exclude`
|
||||
which can be difficult to use correctly. Patching or forking external packages
|
||||
to remove columns from a log can also be avoided with this mechanism.
|
||||
|
||||
Closes #2000.
|
||||
|
||||
* GH-2262: telemetry: Remove singleton BIFs and the C++ pieces (Arne Welzel, Corelight)
|
||||
|
||||
The low-level singleton Telemetry BIFs have been removed with the that there
|
||||
haven't been any users. Singleton metrics can be instantiated by providing
|
||||
an empty label vector instead and aren't in any way a special concept.
|
||||
|
||||
Closes #2262.
|
||||
|
||||
5.1.0-dev.360 | 2022-08-11 11:52:26 +0200
|
||||
|
||||
* telemetry: Switch to histogram family bucket and bound accessors
|
||||
(Arne Welzel, Corelight)
|
||||
|
||||
5.1.0-dev.357 | 2022-08-10 14:16:33 -0700
|
||||
|
||||
* Management framework: minor bugfixes and tweaks (Christian Kreibich, Corelight)
|
||||
|
||||
- bump cluster testsuite
|
||||
- log node set in dispatch requests cleanly
|
||||
- log additional node events
|
||||
- upon deployment, make agent log multiple node results
|
||||
- fix early return condition for get-id-value
|
||||
|
||||
5.1.0-dev.351 | 2022-08-09 09:50:13 -0700
|
||||
|
||||
* Fix module-scoped type definitions that conflict with existing global ones (Tim Wojtulewicz, Corelight)
|
||||
|
||||
5.1.0-dev.349 | 2022-08-08 11:09:04 -0700
|
||||
|
||||
* lower priority for change handlers (Yacin Nadji, Corelight)
|
||||
|
||||
* split update_zones_regex into two functions (Yacin Nadji, Corelight)
|
||||
|
||||
5.1.0-dev.344 | 2022-08-08 11:03:24 -0700
|
||||
|
||||
* Squelch the zeekygen warnings for command line (AmazingPP)
|
||||
|
||||
5.1.0-dev.341 | 2022-08-05 16:21:13 +0000
|
||||
|
||||
* Added test case with back-to-back notices (Annie Bryan)
|
||||
|
||||
* Fix notice suppression atomicity bug (Annie Bryan)
|
||||
|
||||
5.1.0-dev.337 | 2022-08-05 16:19:59 +0000
|
||||
|
||||
* GH-2034: Store module names and use them in lookups for ifdef (Tim Wojtulewicz, Corelight)
|
||||
|
||||
5.1.0-dev.335 | 2022-08-05 16:50:18 +0200
|
||||
|
||||
* Introduce telemetry framework (Arne Welzel, Corelight)
|
||||
|
||||
Adds base/frameworks/telemetry with wrappers around telemetry.bif
|
||||
and updates telemetry/Manager to support collecting metrics from
|
||||
script land.
|
||||
|
||||
Adds policy/frameworks/telemetry/log for logging of metrics data
|
||||
into a new telemetry.log and telemetry_histogram.log and add into
|
||||
local.zeek by default.
|
||||
|
||||
5.1.0-dev.331 | 2022-08-03 10:27:28 -0700
|
||||
|
||||
* input/config: Use thread-safe Fmt() rather than util::fmt() for regex formatting (Arne Welzel, Corelight)
|
||||
|
||||
Calling util::fmt() from DoUpdate() of a thread is not safe as it is
|
||||
using a statically allocated buffer and other threads or the main
|
||||
thread may concurrently modify this buffer.
|
||||
|
||||
This was found by observing the scripts.base.frameworks.config.several-files
|
||||
failing once in a blue moon (1/250 sometimes 1/1000 runs) with messages like
|
||||
"Failed to compile regex: Parenthesis is not closed.":
|
||||
|
||||
scripts.base.frameworks.config.several-files ...
|
||||
> btest-bg-run zeek zeek -b %INPUT
|
||||
> btest-bg-wait 10
|
||||
... scripts.base.frameworks.config.several-files failed
|
||||
% 'btest-bg-wait 10' failed unexpectedly (exit code 1)
|
||||
% cat .stderr
|
||||
The following processes did not terminate:
|
||||
zeek -b /home/awelzel/corelight-oss/zeek/testing/btest/.tmp/scripts.base.frameworks.config.several-files/several-files.zeek
|
||||
-----------
|
||||
<<< [3667265] zeek -b /home/awelzel/corelight-oss/zeek/testing/btest/.tmp/scripts.base.frameworks.config.several-files/several-files.zeek
|
||||
error: ../configfile1/Input::READER_CONFIG: Failed to compile regex: Parenthesis is not closed.
|
||||
received termination signal
|
||||
>>>
|
||||
|
||||
5.1.0-dev.329 | 2022-08-03 10:26:45 -0700
|
||||
|
||||
* testing/missing-enum-value: redef exit_only_after_terminate=T (Arne Welzel, Corelight)
|
||||
|
||||
Seems this was the intention all along as the corresponding terminate()
|
||||
call is there.
|
||||
|
||||
* option.bif: Short-circuit option changes when terminating (Arne Welzel, Corelight)
|
||||
|
||||
Due to the asynchronous behavior of the input framework and broker
|
||||
communication, change handlers were previously called even after
|
||||
zeek_done() event processing completed and also broker shutdown.
|
||||
|
||||
Accessing broker store handles within change handlers this late
|
||||
triggered invalid Broker store handle messages:
|
||||
|
||||
error in ././my_option_store.zeek, line 13: invalid Broker store handle (Broker::put(Test::store, to_any_coercemy_option, to_any_coerceTest::new_value, 0 secs) and broker::store::{})
|
||||
|
||||
Fixes #2010
|
||||
|
||||
5.1.0-dev.326 | 2022-08-03 09:56:37 -0700
|
||||
|
||||
* updates for gen-C++ maintenance, including skipping some inappropriate tests (Vern Paxson, Corelight)
|
||||
|
||||
* fix for profiling "when" statements (Vern Paxson, Corelight)
|
||||
|
||||
* gen-C++ support for vector bit-shift operations (Vern Paxson, Corelight)
|
||||
|
||||
* corrected wording in some btest comments (Vern Paxson, Corelight)
|
||||
|
||||
* make gen-C++ maintenance scripts directly executable (Vern Paxson, Corelight)
|
||||
|
||||
* ZAM support for bit-shifting (Vern Paxson, Corelight)
|
||||
|
||||
* don't allow deprecated-style mixing of vectors and scaling for shifting (Vern Paxson, Corelight)
|
||||
leverage restrictions placed on shifting (RHS is always unsigned)
|
||||
split deprecated vector operations into separate test, with separate ZAM baseline
|
||||
|
||||
* ZAM fix for vector "in" operator (Vern Paxson, Corelight)
|
||||
|
||||
* ensure that language tests pay attention to .stderr (Vern Paxson, Corelight)
|
||||
|
||||
* fix vector tests, including checking for errors (Vern Paxson, Corelight)
|
||||
|
||||
5.1.0-dev.312 | 2022-08-02 12:37:51 -0700
|
||||
|
||||
* Update plugins.hooks baseline with new DHCP options (peter.cullen, Corelight)
|
||||
|
||||
* Prevent large dhcp log entries (Peter Cullen, Corelight)
|
||||
|
||||
A flood of DHCP traffic can result if very large log entries consisting
|
||||
of many uids and/or msg_types. Such large log entries can disrupt a SIEM
|
||||
ingestion pipeline. This change forcing a log entry to be written when
|
||||
the number of uids or the number of msg_Types exceed a certain value.
|
||||
The values are treated as options for easy configuration.
|
||||
|
||||
5.1.0-dev.309 | 2022-08-02 11:33:22 -0700
|
||||
|
||||
* Add support for /s modifier to RE matcher and parser (Tim Wojtulewicz, Corelight)
|
||||
|
||||
* Code cleanup in RE_Matcher code (Tim Wojtulewicz, Corelight)
|
||||
|
||||
- Use std::string in Specific_RE_Matcher instead of char*
|
||||
- Change a couple of ints-as-bools to bools
|
||||
|
||||
* Add basic unit tests for RE_Matcher (Tim Wojtulewicz, Corelight)
|
||||
|
||||
* Add /s modifier to parser for patterns (Tim Wojtulewicz, Corelight)
|
||||
|
||||
* Update gen-zam submodule [nomail] (Tim Wojtulewicz, Corelight)
|
||||
|
||||
5.1.0-dev.303 | 2022-08-01 09:56:45 -0700
|
||||
|
||||
* GH-1344: Give better warning when using a type that doesn't exist (Tim Wojtulewicz, Corelight)
|
||||
|
||||
5.1.0-dev.301 | 2022-07-29 12:10:20 -0700
|
||||
|
||||
* Add btest for vector bit-shift operators (Tim Wojtulewicz, Corelight)
|
||||
|
||||
* Handle error cases for bit-shift operators more cleanly (Tim Wojtulewicz, Corelight)
|
||||
|
||||
5.1.0-dev.298 | 2022-07-29 12:00:57 -0700
|
||||
|
||||
* Update 3rdparty submodule to get patricia reformat (Tim Wojtulewicz, Corelight)
|
||||
|
||||
5.1.0-dev.295 | 2022-07-28 11:21:59 -0700
|
||||
|
||||
* Re-enable sending coverage data to Coveralls via Cirrus (Tim Wojtulewicz, Corelight)
|
||||
|
|
|
|||
47
NEWS
47
NEWS
|
|
@ -21,9 +21,52 @@ Breaking Changes
|
|||
- The Dictionary and PDict classes are now C++ templates. This may cause
|
||||
plugin/package builds to fail due to needing to modify uses of them to match.
|
||||
|
||||
- The low-level singleton Telemetry BIFs have been removed with the assumption that
|
||||
there haven't been any users. Singleton metrics can be instantiated by providing
|
||||
an empty label vector instead and aren't in any way a special concept.
|
||||
|
||||
New Functionality
|
||||
-----------------
|
||||
|
||||
- Added support for the /s regular expression modifier. Using this modifier in
|
||||
patterns in Zeek scripts will cause the '.' character to also match newline
|
||||
characters.
|
||||
|
||||
- Added a new telemetry framework for providing high-level access to Zeek's
|
||||
metric subsystem. This framework allows script writers to use different
|
||||
metric types (counters, gauges and histograms) for tracking metrics without
|
||||
using lower-level BiFs from ``telemetry.bif``. Additionally, metrics can
|
||||
now be accessed from script land using ``Telemetry::collect_metrics() and
|
||||
``Telemetry::collect_histogram_metrics()``.
|
||||
|
||||
The framework is located in ``base/frameworks/telemetry``.
|
||||
|
||||
In addition to the Prometheus endpoint for metrics export that has existed
|
||||
since Zeek 4.1, two new log streams, ``telemetry.log`` and ``telemetry_histogram.log``,
|
||||
can be enabled by loading ``policy/frameworks/telemetry/log``. This policy
|
||||
script is included in ``local.zeek`` by default.
|
||||
|
||||
For further details on the framework and examples, please refer to the
|
||||
Zeek documentation.
|
||||
|
||||
- Allow redef'ing the ``&log`` attribute of record fields:
|
||||
|
||||
redef Notice::Info$email_dest -= { &log };
|
||||
|
||||
While the syntax allows for any attribute, only ``&log`` is supported. The
|
||||
semantics for other record field attributes are not easy to grasp and there
|
||||
were no obvious use-cases identified.
|
||||
|
||||
- Introduce a global ``disabling_analyzer()`` hook to allow vetoing calls
|
||||
to ``disable_analyzer()``.
|
||||
|
||||
The contract is simple: Any script can veto a ``disable_analyzer()`` call by
|
||||
breaking from this hook. The decision is local to the script taking into
|
||||
account any state attached to the connection or state stored elsewhere.
|
||||
A script breaking from the hook takes over responsibility to call
|
||||
``disable_analyzer()`` at a later point when it finds the condition due
|
||||
to which it vetoed fulfilled (which may be never).
|
||||
|
||||
Changed Functionality
|
||||
---------------------
|
||||
|
||||
|
|
@ -41,6 +84,10 @@ Changed Functionality
|
|||
- The default logging directory is now set globally across all log
|
||||
writers through ``Log::default_logdir``.
|
||||
|
||||
- Calling `Option::set()` when Zeek is terminating is now a noop and returns `F`.
|
||||
This prevents callbacks into script-land through change handlers when parts
|
||||
of the environment have already been torn down.
|
||||
|
||||
Deprecated Functionality
|
||||
------------------------
|
||||
|
||||
|
|
|
|||
2
VERSION
2
VERSION
|
|
@ -1 +1 @@
|
|||
5.1.0-dev.295
|
||||
5.1.0-dev.386
|
||||
|
|
|
|||
|
|
@ -1 +1 @@
|
|||
Subproject commit dc7b266e8289472096828ad51547237f49a91841
|
||||
Subproject commit 8d03476847411b0ab7b9c47ea9eaf7b1535d08c2
|
||||
|
|
@ -1 +1 @@
|
|||
Subproject commit 5b0855905f7c7937a38cfe3141544fb015192b5e
|
||||
Subproject commit d4e0fc662a8f55632c8ec84550d8e4a80460d35d
|
||||
2
doc
2
doc
|
|
@ -1 +1 @@
|
|||
Subproject commit 4550b7fbe963d39f31d0c50f7d6764d4ec95091d
|
||||
Subproject commit 497bcfbcc547b78f5ceafeb3860b12a0bf6f2375
|
||||
|
|
@ -538,6 +538,7 @@ hook Notice::notice(n: Notice::Info) &priority=-5
|
|||
n$suppress_for != 0secs )
|
||||
{
|
||||
event Notice::begin_suppression(n$ts, n$suppress_for, n$note, n$identifier);
|
||||
suppressing[n$note, n$identifier] = n$ts + n$suppress_for;
|
||||
@if ( Cluster::is_enabled() && Cluster::local_node_type() != Cluster::MANAGER )
|
||||
event Notice::manager_begin_suppression(n$ts, n$suppress_for, n$note, n$identifier);
|
||||
@endif
|
||||
|
|
|
|||
1
scripts/base/frameworks/telemetry/__load__.zeek
Normal file
1
scripts/base/frameworks/telemetry/__load__.zeek
Normal file
|
|
@ -0,0 +1 @@
|
|||
@load ./main
|
||||
609
scripts/base/frameworks/telemetry/main.zeek
Normal file
609
scripts/base/frameworks/telemetry/main.zeek
Normal file
|
|
@ -0,0 +1,609 @@
|
|||
##! Module for recording and querying metrics. This modules wraps
|
||||
##! the lower-level telemetry.bif functions.
|
||||
##!
|
||||
##! Metrics will be exposed through a Prometheus HTTP endpoint when
|
||||
##! enabled by setting :zeek:see:`Broker::metrics_port` or using the
|
||||
##! `BROKER_METRICS_PORT` environment variable.
|
||||
|
||||
@load base/misc/version
|
||||
|
||||
module Telemetry;
|
||||
|
||||
export {
|
||||
## Alias for a vector of label values.
|
||||
type labels_vector: vector of string;
|
||||
|
||||
## Type that captures options used to create metrics.
|
||||
type MetricOpts: record {
|
||||
## The prefix (namespace) of the metric.
|
||||
prefix: string;
|
||||
|
||||
## The human-readable name of the metric.
|
||||
name: string;
|
||||
|
||||
## The unit of the metric. Use the pseudo-unit "1" if this is a unit-less metric.
|
||||
unit: string;
|
||||
|
||||
## Documentation for this metric.
|
||||
help_text: string;
|
||||
|
||||
## The label names (also called dimensions) of the metric. When
|
||||
## instantiating or working with concrete metrics, corresponding
|
||||
## label values have to be provided.
|
||||
labels: vector of string &default=vector();
|
||||
|
||||
## Whether the metric represents something that is accumulating.
|
||||
## Defaults to ``T`` for counters and ``F`` for gauges and
|
||||
## histograms.
|
||||
is_total: bool &optional;
|
||||
|
||||
## When creating a :zeek:see:`Telemetry::HistogramFamily`,
|
||||
## describes the number and bounds of the individual buckets.
|
||||
bounds: vector of double &optional;
|
||||
|
||||
## The same meaning as *bounds*, but as :zeek:type:`count`.
|
||||
## Only set in the return value of
|
||||
## :zeek:see:`Telemetry::collect_histogram_metrics`.
|
||||
## for histograms when the underlying type is ``int64_t``,
|
||||
## otherwise ignored.
|
||||
count_bounds: vector of count &optional;
|
||||
|
||||
## Describes the underlying metric type.
|
||||
## Only set in the return value of
|
||||
## :zeek:see:`Telemetry::collect_metrics` or
|
||||
## :zeek:see:`Telemetry::collect_histogram_metrics`,
|
||||
## otherwise ignored.
|
||||
metric_type: MetricType &optional;
|
||||
};
|
||||
|
||||
## Type representing a family of counters with uninitialized label values.
|
||||
##
|
||||
## To create concrete :zeek:see:`Telemetry::Counter` instances, use
|
||||
## :zeek:see:`Telemetry::counter_with`. To modify counters directly
|
||||
## use :zeek:see:`Telemetry::counter_family_inc`.
|
||||
type CounterFamily: record {
|
||||
__family: opaque of dbl_counter_metric_family;
|
||||
__labels: vector of string;
|
||||
};
|
||||
|
||||
## Type representing a counter metric with initialized label values.
|
||||
##
|
||||
## Counter metrics only ever go up and reset when the process
|
||||
## restarts. Use :zeek:see:`Telemetry::counter_inc` or
|
||||
## :zeek:see:`Telemetry::counter_set` to modify counters.
|
||||
## An example for a counter is the number of log writes
|
||||
## per :zeek:see:`Log::Stream` or number connections broken down
|
||||
## by protocol and service.
|
||||
type Counter: record {
|
||||
__metric: opaque of dbl_counter_metric;
|
||||
};
|
||||
|
||||
## Register a counter family.
|
||||
global register_counter_family: function(opts: MetricOpts): CounterFamily;
|
||||
|
||||
## Get a :zeek:see:`Telemetry::Counter` instance given family and label values.
|
||||
global counter_with: function(cf: CounterFamily,
|
||||
label_values: labels_vector &default=vector()): Counter;
|
||||
|
||||
## Increment a :zeek:see:`Telemetry::Counter` by `amount`.
|
||||
## Using a negative `amount` is an error.
|
||||
##
|
||||
## c: The counter instance.
|
||||
##
|
||||
## amount: The amount by which to increment the counter.
|
||||
##
|
||||
## Returns: True if the counter was incremented successfully.
|
||||
global counter_inc: function(c: Counter, amount: double &default=1.0): bool;
|
||||
|
||||
## Helper to set a :zeek:see:`Telemetry::Counter` to the given `value`.
|
||||
## This can be useful for mirroring counter metrics in an
|
||||
## :zeek:see:`Telemetry::sync` hook implementation.
|
||||
## Setting a value that is less than the current value of the
|
||||
## metric is an error and will be ignored.
|
||||
##
|
||||
## c: The counter instance.
|
||||
##
|
||||
## value: The value to set the counter to.
|
||||
##
|
||||
## Returns: True if the counter value was set successfully.
|
||||
global counter_set: function(c: Counter, value: double): bool;
|
||||
|
||||
## Increment a :zeek:see:`Telemetry::Counter` through the :zeek:see:`Telemetry::CounterFamily`.
|
||||
## This is a short-cut for :zeek:see:`Telemetry::counter_inc`.
|
||||
## Using a negative amount is an error.
|
||||
##
|
||||
## cf: The counter family to use.
|
||||
##
|
||||
## label_values: The label values to use for the counter.
|
||||
##
|
||||
## amount: The amount by which to increment the counter.
|
||||
##
|
||||
## Returns: True if the counter was incremented successfully.
|
||||
global counter_family_inc: function(cf: CounterFamily,
|
||||
label_values: labels_vector &default=vector(),
|
||||
amount: double &default=1.0): bool;
|
||||
|
||||
## Set a :zeek:see:`Telemetry::Counter` through the :zeek:see:`Telemetry::CounterFamily`.
|
||||
## This is a short-cut for :zeek:see:`Telemetry::counter_set`.
|
||||
## Setting a value that is less than the current value of the
|
||||
## metric is an error and will be ignored.
|
||||
##
|
||||
## cf: The counter family to use.
|
||||
##
|
||||
## label_values: The label values to use for the counter.
|
||||
##
|
||||
## value: The value to set the counter to.
|
||||
##
|
||||
## Returns: True if the counter value was set successfully.
|
||||
global counter_family_set: function(cf: CounterFamily,
|
||||
label_values: labels_vector,
|
||||
value: double): bool;
|
||||
|
||||
## Type representing a family of gauges with uninitialized label values.
|
||||
##
|
||||
## Create concrete :zeek:see:`Telemetry::Gauge` instances with
|
||||
## :zeek:see:`Telemetry::gauge_with`, or use
|
||||
## :zeek:see:`Telemetry::gauge_family_inc` or
|
||||
## :zeek:see:`Telemetry::gauge_family_set` directly.
|
||||
type GaugeFamily: record {
|
||||
__family: opaque of dbl_gauge_metric_family;
|
||||
__labels: vector of string;
|
||||
};
|
||||
|
||||
## Type representing a gauge metric with initialized label values.
|
||||
##
|
||||
## Use :zeek:see:`Telemetry::gauge_inc`, :zeek:see:`Telemetry::gauge_dec`,
|
||||
## or :zeek:see:`Telemetry::gauge_set` to modify the gauge.
|
||||
## Example for gauges are process memory usage, table sizes
|
||||
## or footprints of long-lived values as determined by
|
||||
## :zeek:see:`val_footprint`.
|
||||
type Gauge: record {
|
||||
__metric: opaque of dbl_gauge_metric;
|
||||
};
|
||||
|
||||
## Register a gauge family.
|
||||
global register_gauge_family: function(opts: MetricOpts): GaugeFamily;
|
||||
|
||||
|
||||
## Get a :zeek:see:`Telemetry::Gauge` instance given family and label values.
|
||||
global gauge_with: function(gf: GaugeFamily,
|
||||
label_values: labels_vector &default=vector()): Gauge;
|
||||
|
||||
## Increment a :zeek:see:`Telemetry::Gauge` by `amount`.
|
||||
##
|
||||
## g: The gauge instance.
|
||||
##
|
||||
## amount: The amount by which to increment the gauge.
|
||||
##
|
||||
## Returns: True if the gauge was incremented successfully.
|
||||
global gauge_inc: function(g: Gauge, amount: double &default=1.0): bool;
|
||||
|
||||
## Decrement a :zeek:see:`Telemetry::Gauge` by `amount`.
|
||||
##
|
||||
## g: The gauge instance.
|
||||
##
|
||||
## amount: The amount by which to decrement the gauge.
|
||||
##
|
||||
## Returns: True if the gauge was incremented successfully.
|
||||
global gauge_dec: function(g: Gauge, amount: double &default=1.0): bool;
|
||||
|
||||
## Helper to set a :zeek:see:`Telemetry::Gauge` to the given `value`.
|
||||
##
|
||||
## g: The gauge instance.
|
||||
##
|
||||
## value: The value to set the gauge to.
|
||||
##
|
||||
## Returns: True if the gauge value was set successfully.
|
||||
global gauge_set: function(g: Gauge, value: double): bool;
|
||||
|
||||
## Increment a :zeek:see:`Telemetry::Gauge` by the given `amount` through
|
||||
## the :zeek:see:`Telemetry::GaugeFamily`.
|
||||
## This is a short-cut for :zeek:see:`Telemetry::gauge_inc`.
|
||||
## Using a negative amount is an error.
|
||||
##
|
||||
## gf: The gauge family to use.
|
||||
##
|
||||
## label_values: The label values to use for the gauge.
|
||||
##
|
||||
## amount: The amount by which to increment the gauge.
|
||||
##
|
||||
## Returns: True if the gauge was incremented successfully.
|
||||
global gauge_family_inc: function(gf: GaugeFamily,
|
||||
label_values: labels_vector &default=vector(),
|
||||
amount: double &default=1.0): bool;
|
||||
|
||||
## Decrement a :zeek:see:`Telemetry::Gauge` by the given `amount` through
|
||||
## the :zeek:see:`Telemetry::GaugeFamily`.
|
||||
## This is a short-cut for :zeek:see:`Telemetry::gauge_dec`.
|
||||
##
|
||||
## gf: The gauge family to use.
|
||||
##
|
||||
## label_values: The label values to use for the gauge.
|
||||
##
|
||||
## amount: The amount by which to increment the gauge.
|
||||
##
|
||||
## Returns: True if the gauge was incremented successfully.
|
||||
global gauge_family_dec: function(gf: GaugeFamily,
|
||||
label_values: labels_vector &default=vector(),
|
||||
amount: double &default=1.0): bool;
|
||||
|
||||
## Set a :zeek:see:`Telemetry::Gauge` to the given `value` through
|
||||
## the :zeek:see:`Telemetry::GaugeFamily`.
|
||||
## This is a short-cut for :zeek:see:`Telemetry::gauge_set`.
|
||||
##
|
||||
## gf: The gauge family to use.
|
||||
##
|
||||
## label_values: The label values to use for the gauge.
|
||||
##
|
||||
## value: The value to set the gauge to.
|
||||
##
|
||||
## Returns: True if the gauge value was set successfully.
|
||||
global gauge_family_set: function(g: GaugeFamily,
|
||||
label_values: labels_vector,
|
||||
value: double): bool;
|
||||
|
||||
## Type representing a family of histograms with uninitialized label values.
|
||||
## Create concrete :zeek:see:`Telemetry::Histogram` instances with
|
||||
## :zeek:see:`Telemetry::histogram_with` or use
|
||||
## :zeek:see:`Telemetry::histogram_family_observe` directly.
|
||||
type HistogramFamily: record {
|
||||
__family: opaque of dbl_histogram_metric_family;
|
||||
__labels: vector of string;
|
||||
};
|
||||
|
||||
## Type representing a histogram metric with initialized label values.
|
||||
## Use :zeek:see:`Telemetry::histogram_observe` to make observations.
|
||||
type Histogram: record {
|
||||
__metric: opaque of dbl_histogram_metric;
|
||||
};
|
||||
|
||||
## Register a histogram family.
|
||||
global register_histogram_family: function(opts: MetricOpts): HistogramFamily;
|
||||
|
||||
## Get a :zeek:see:`Telemetry::Histogram` instance given family and label values.
|
||||
global histogram_with: function(hf: HistogramFamily,
|
||||
label_values: labels_vector &default=vector()): Histogram;
|
||||
|
||||
## Observe a measurement for a :zeek:see:`Telemetry::Histogram`.
|
||||
##
|
||||
## h: The histogram instance.
|
||||
##
|
||||
## measurement: The value for this observations.
|
||||
##
|
||||
## Returns: True if measurement was observed successfully.
|
||||
global histogram_observe: function(h: Histogram, measurement: double): bool;
|
||||
|
||||
## Observe a measurement for a :zeek:see:`Telemetry::Histogram` through
|
||||
## the :zeek:see:`Telemetry::HistogramFamily`.
|
||||
## This is a short-cut for :zeek:see:`Telemetry::histogram_observe`.
|
||||
##
|
||||
## hf: The histogram family to use.
|
||||
##
|
||||
## label_values: The label values to use for the histogram.
|
||||
##
|
||||
## measurement: The value for this observations.
|
||||
##
|
||||
## Returns: True if measurement was observed successfully.
|
||||
global histogram_family_observe: function(hf: HistogramFamily,
|
||||
label_values: labels_vector,
|
||||
measurement: double): bool;
|
||||
|
||||
## Telemetry sync hook.
|
||||
##
|
||||
## This hook is invoked every :zeek:see:`Telemetry::sync_interval`
|
||||
## for script writers to synchronize or mirror metrics with the
|
||||
## telemetry subsystem. For example, when tracking table or value
|
||||
## footprints with gauges, the value in question can be set on an actual
|
||||
## :zeek:see:`Telemetry::Gauge` instance during execution of this hook.
|
||||
##
|
||||
## Implementations should be lightweight, this hook may be called
|
||||
## multiple times per minute. The interval can increased by changing
|
||||
## :zeek:see:`Telemetry::sync_interval` at the cost of delaying
|
||||
## metric updates and thereby reducing granularity.
|
||||
global sync: hook();
|
||||
|
||||
## Interval at which the :zeek:see:`Telemetry::sync` hook is invoked.
|
||||
option sync_interval = 10sec;
|
||||
|
||||
## Type of elements returned by the :zeek:see:`Telemetry::collect_metrics` function.
|
||||
type Metric: record {
|
||||
## A :zeek:see:`Telemetry::MetricOpts` record describing this metric.
|
||||
opts: MetricOpts;
|
||||
|
||||
## The label values associated with this metric, if any.
|
||||
labels: vector of string;
|
||||
|
||||
## The value of gauge or counter cast to a double
|
||||
## independent of the underlying data type.
|
||||
## This value is set for all counter and gauge metrics,
|
||||
## it is unset for histograms.
|
||||
value: double &optional;
|
||||
|
||||
## The value of the underlying gauge or counter as a double
|
||||
## if the underlying metric type uses ``int64_t``.
|
||||
## Only counters and gauges created with the C++ API may
|
||||
## have this value set.
|
||||
count_value: count &optional;
|
||||
};
|
||||
|
||||
## Type of elements returned by the :zeek:see:`Telemetry::collect_histogram_metrics` function.
|
||||
type HistogramMetric: record {
|
||||
## A :zeek:see:`Telemetry::MetricOpts` record describing this histogram.
|
||||
opts: MetricOpts;
|
||||
|
||||
## The label values associated with this histogram, if any.
|
||||
labels: vector of string;
|
||||
|
||||
## Individual counters for each of the buckets as
|
||||
## described by the *bounds* field in *opts*;
|
||||
values: vector of double;
|
||||
|
||||
## If the underlying data type of the histogram is ``int64_t``,
|
||||
## this vector will hold the values as counts, otherwise it
|
||||
## is unset. Only histograms created with the C++ API have
|
||||
## may have this value set.
|
||||
count_values: vector of count &optional;
|
||||
|
||||
## The number of observations made for this histogram.
|
||||
observations: double;
|
||||
|
||||
## The sum of all observations for this histogram.
|
||||
sum: double;
|
||||
|
||||
## If the underlying data type of the histogram is ``int64_t``,
|
||||
## the number of observations as :zeek:type:`count`, otherwise
|
||||
## unset.
|
||||
count_observations: count &optional;
|
||||
|
||||
## If the underlying data type of the histogram is ``int64_t``,
|
||||
## the sum of all observations as :zeek:type:`count`, otherwise
|
||||
## unset.
|
||||
count_sum: count &optional;
|
||||
};
|
||||
|
||||
## Collect all counter and gauge metrics matching the given *name* and *prefix*.
|
||||
##
|
||||
## For histogram metrics, use the :zeek:see:`Telemetry::collect_histogram_metrics`.
|
||||
##
|
||||
## The *prefix* and *name* parameters support globbing. By default,
|
||||
## all counters and gauges are returned.
|
||||
global collect_metrics: function(prefix: string &default="*",
|
||||
name: string &default="*"): vector of Metric;
|
||||
|
||||
## Collect all histograms and their observations matching the given
|
||||
## *prefix* and *name*.
|
||||
##
|
||||
## The *prefix* and *name* parameters support globbing. By default,
|
||||
## all histogram metrics are returned.
|
||||
global collect_histogram_metrics: function(prefix: string &default="*",
|
||||
name: string &default="*"): vector of HistogramMetric;
|
||||
}
|
||||
|
||||
## Internal helper to create the labels table.
|
||||
function make_labels(keys: vector of string, values: labels_vector): table[string] of string
|
||||
{
|
||||
local labels: table[string] of string;
|
||||
for ( i in keys )
|
||||
labels[keys[i]] = values[i];
|
||||
|
||||
return labels;
|
||||
}
|
||||
|
||||
function register_counter_family(opts: MetricOpts): CounterFamily
|
||||
{
|
||||
local f = Telemetry::__dbl_counter_family(
|
||||
opts$prefix,
|
||||
opts$name,
|
||||
opts$labels,
|
||||
opts$help_text,
|
||||
opts$unit,
|
||||
opts?$is_total ? opts$is_total : T
|
||||
);
|
||||
return CounterFamily($__family=f, $__labels=opts$labels);
|
||||
}
|
||||
|
||||
# Fallback Counter returned when there are issues with the labels.
|
||||
global error_counter_cf = register_counter_family([
|
||||
$prefix="zeek",
|
||||
$name="telemetry_counter_usage_error",
|
||||
$unit="1",
|
||||
$help_text="This counter is returned when label usage for counters is wrong. Check reporter.log if non-zero."
|
||||
]);
|
||||
|
||||
function counter_with(cf: CounterFamily, label_values: labels_vector): Counter
|
||||
{
|
||||
if ( |cf$__labels| != |label_values| )
|
||||
{
|
||||
Reporter::error(fmt("Invalid label values expected %s, have %s", |cf$__labels|, |label_values|));
|
||||
return counter_with(error_counter_cf);
|
||||
}
|
||||
|
||||
local labels = make_labels(cf$__labels, label_values);
|
||||
local m = Telemetry::__dbl_counter_metric_get_or_add(cf$__family, labels);
|
||||
return Counter($__metric=m);
|
||||
}
|
||||
|
||||
function counter_inc(c: Counter, amount: double): bool
|
||||
{
|
||||
return Telemetry::__dbl_counter_inc(c$__metric, amount);
|
||||
}
|
||||
|
||||
function counter_set(c: Counter, value: double): bool
|
||||
{
|
||||
local cur_value: double = Telemetry::__dbl_counter_value(c$__metric);
|
||||
if (value < cur_value)
|
||||
{
|
||||
Reporter::error(fmt("Attempted to set lower counter value=%s cur_value=%s", value, cur_value));
|
||||
return F;
|
||||
}
|
||||
return Telemetry::__dbl_counter_inc(c$__metric, value - cur_value);
|
||||
}
|
||||
|
||||
function counter_family_inc(cf: CounterFamily, label_values: labels_vector, amount: double): bool
|
||||
{
|
||||
return counter_inc(counter_with(cf, label_values), amount);
|
||||
}
|
||||
|
||||
function counter_family_set(cf: CounterFamily, label_values: labels_vector, value: double): bool
|
||||
{
|
||||
return counter_set(counter_with(cf, label_values), value);
|
||||
}
|
||||
|
||||
function register_gauge_family(opts: MetricOpts): GaugeFamily
|
||||
{
|
||||
local f = Telemetry::__dbl_gauge_family(
|
||||
opts$prefix,
|
||||
opts$name,
|
||||
opts$labels,
|
||||
opts$help_text,
|
||||
opts$unit,
|
||||
opts?$is_total ? opts$is_total : F
|
||||
);
|
||||
return GaugeFamily($__family=f, $__labels=opts$labels);
|
||||
}
|
||||
|
||||
# Fallback Gauge returned when there are issues with the label usage.
|
||||
global error_gauge_cf = register_gauge_family([
|
||||
$prefix="zeek",
|
||||
$name="telemetry_gauge_usage_error",
|
||||
$unit="1",
|
||||
$help_text="This gauge is returned when label usage for gauges is wrong. Check reporter.log if non-zero."
|
||||
]);
|
||||
|
||||
function gauge_with(gf: GaugeFamily, label_values: labels_vector): Gauge
|
||||
{
|
||||
if ( |gf$__labels| != |label_values| )
|
||||
{
|
||||
Reporter::error(fmt("Invalid label values expected %s, have %s", |gf$__labels|, |label_values|));
|
||||
return gauge_with(error_gauge_cf);
|
||||
}
|
||||
local labels = make_labels(gf$__labels, label_values);
|
||||
local m = Telemetry::__dbl_gauge_metric_get_or_add(gf$__family, labels);
|
||||
return Gauge($__metric=m);
|
||||
}
|
||||
|
||||
function gauge_inc(g: Gauge, amount: double &default=1.0): bool
|
||||
{
|
||||
return Telemetry::__dbl_gauge_inc(g$__metric, amount);
|
||||
}
|
||||
|
||||
function gauge_dec(g: Gauge, amount: double &default=1.0): bool
|
||||
{
|
||||
return Telemetry::__dbl_gauge_dec(g$__metric, amount);
|
||||
}
|
||||
|
||||
function gauge_set(g: Gauge, value: double): bool
|
||||
{
|
||||
# Telemetry currently does not implement __dbl_gauge_set(), do
|
||||
# it by hand here.
|
||||
local cur_value: double = Telemetry::__dbl_gauge_value(g$__metric);
|
||||
if (value > cur_value)
|
||||
return Telemetry::__dbl_gauge_inc(g$__metric, value - cur_value);
|
||||
|
||||
return Telemetry::__dbl_gauge_dec(g$__metric, cur_value - value);
|
||||
}
|
||||
|
||||
function gauge_family_inc(gf: GaugeFamily, label_values: labels_vector, value: double): bool
|
||||
{
|
||||
return gauge_inc(gauge_with(gf, label_values), value);
|
||||
}
|
||||
|
||||
function gauge_family_dec(gf: GaugeFamily, label_values: labels_vector, value: double): bool
|
||||
{
|
||||
return gauge_dec(gauge_with(gf, label_values), value);
|
||||
}
|
||||
|
||||
function gauge_family_set(gf: GaugeFamily, label_values: labels_vector, value: double): bool
|
||||
{
|
||||
return gauge_set(gauge_with(gf, label_values), value);
|
||||
}
|
||||
|
||||
function register_histogram_family(opts: MetricOpts): HistogramFamily
|
||||
{
|
||||
local f = Telemetry::__dbl_histogram_family(
|
||||
opts$prefix,
|
||||
opts$name,
|
||||
opts$labels,
|
||||
opts$bounds,
|
||||
opts$help_text,
|
||||
opts$unit,
|
||||
opts?$is_total ? opts$is_total : F
|
||||
);
|
||||
return HistogramFamily($__family=f, $__labels=opts$labels);
|
||||
}
|
||||
|
||||
# Fallback Histogram when there are issues with the labels.
|
||||
global error_histogram_hf = register_histogram_family([
|
||||
$prefix="zeek",
|
||||
$name="telemetry_histogram_usage_error",
|
||||
$unit="1",
|
||||
$help_text="This histogram is returned when label usage for histograms is wrong. Check reporter.log if non-zero.",
|
||||
$bounds=vector(1.0)
|
||||
]);
|
||||
|
||||
function histogram_with(hf: HistogramFamily, label_values: labels_vector): Histogram
|
||||
{
|
||||
if ( |hf$__labels| != |label_values| )
|
||||
{
|
||||
Reporter::error(fmt("Invalid label values expected %s, have %s", |hf$__labels|, |label_values|));
|
||||
return histogram_with(error_histogram_hf);
|
||||
}
|
||||
|
||||
local labels = make_labels(hf$__labels, label_values);
|
||||
local m = Telemetry::__dbl_histogram_metric_get_or_add(hf$__family, labels);
|
||||
return Histogram($__metric=m);
|
||||
}
|
||||
|
||||
function histogram_observe(h: Histogram, measurement: double): bool
|
||||
{
|
||||
return Telemetry::__dbl_histogram_observe(h$__metric, measurement);
|
||||
}
|
||||
|
||||
function histogram_family_observe(hf: HistogramFamily, label_values: labels_vector, measurement: double): bool
|
||||
{
|
||||
return histogram_observe(histogram_with(hf, label_values), measurement);
|
||||
}
|
||||
|
||||
function collect_metrics(prefix: string, name: string): vector of Metric
|
||||
{
|
||||
return Telemetry::__collect_metrics(prefix, name);
|
||||
}
|
||||
|
||||
function collect_histogram_metrics(prefix: string, name: string): vector of HistogramMetric
|
||||
{
|
||||
return Telemetry::__collect_histogram_metrics(prefix, name);
|
||||
}
|
||||
|
||||
event run_sync_hook()
|
||||
{
|
||||
hook Telemetry::sync();
|
||||
schedule sync_interval { run_sync_hook() };
|
||||
}
|
||||
|
||||
event zeek_init()
|
||||
{
|
||||
schedule sync_interval { run_sync_hook() };
|
||||
}
|
||||
|
||||
# Expose the Zeek version as Prometheus style info metric
|
||||
global version_gauge_family = Telemetry::register_gauge_family([
|
||||
$prefix="zeek",
|
||||
$name="version_info",
|
||||
$unit="1",
|
||||
$help_text="The Zeek version",
|
||||
$labels=vector("version_number", "major", "minor", "patch", "commit",
|
||||
"beta", "debug","version_string")
|
||||
]);
|
||||
|
||||
event zeek_init()
|
||||
{
|
||||
local v = Version::info;
|
||||
local labels = vector(cat(v$version_number),
|
||||
cat(v$major), cat(v$minor), cat (v$patch),
|
||||
cat(v$commit),
|
||||
v$beta ? "true" : "false",
|
||||
v$debug ? "true" : "false",
|
||||
v$version_string);
|
||||
|
||||
Telemetry::gauge_family_set(version_gauge_family, labels, 1.0);
|
||||
}
|
||||
|
|
@ -588,6 +588,23 @@ type fa_metadata: record {
|
|||
inferred: bool &default=T;
|
||||
};
|
||||
|
||||
## A hook taking a connection, analyzer tag and analyzer id that can be
|
||||
## used to veto disabling analyzers. Specifically, an analyzer can be prevented
|
||||
## from being disabled by using a :zeek:see:`break` statement within the hook.
|
||||
## This hook is invoked synchronously during a :zeek:see:`disable_analyzer` call.
|
||||
##
|
||||
## Scripts implementing this hook should have other logic that will eventually
|
||||
## disable the analyzer for the given connection. That is, if a script vetoes
|
||||
## disabling an analyzer, it takes responsibility for a later call to
|
||||
## :zeek:see:`disable_analyzer`, which may be never.
|
||||
##
|
||||
## c: The connection
|
||||
##
|
||||
## atype: The type / tag of the analyzer being disabled.
|
||||
##
|
||||
## aid: The analyzer ID.
|
||||
type disabling_analyzer: hook(c: connection, atype: AllAnalyzers::Tag, aid: count);
|
||||
|
||||
## Fields of a SYN packet.
|
||||
##
|
||||
## .. zeek:see:: connection_SYN_packet
|
||||
|
|
|
|||
|
|
@ -43,6 +43,7 @@
|
|||
@load base/frameworks/tunnels
|
||||
@load base/frameworks/openflow
|
||||
@load base/frameworks/netcontrol
|
||||
@load base/frameworks/telemetry
|
||||
|
||||
@load base/protocols/conn
|
||||
@load base/protocols/dce-rpc
|
||||
|
|
|
|||
|
|
@ -91,6 +91,12 @@ export {
|
|||
## transaction narrative.
|
||||
option DHCP::max_txid_watch_time = 30secs;
|
||||
|
||||
## The maximum number of uids allowed in a single log entry.
|
||||
option DHCP::max_uids_per_log_entry = 10;
|
||||
|
||||
## The maximum number of msg_types allowed in a single log entry.
|
||||
option DHCP::max_msg_types_per_log_entry = 50;
|
||||
|
||||
## This event is used internally to distribute data around clusters
|
||||
## since DHCP doesn't follow the normal "connection" model used by
|
||||
## most protocols. It can also be handled to extend the DHCP log.
|
||||
|
|
@ -266,6 +272,13 @@ event DHCP::aggregate_msgs(ts: time, id: conn_id, uid: string, is_orig: bool, ms
|
|||
if ( options?$lease )
|
||||
log_info$lease_time = options$lease;
|
||||
}
|
||||
|
||||
# Write log entry if |uids| or |msg_types| becomes too large
|
||||
if ( |log_info$uids| >= max_uids_per_log_entry || |log_info$msg_types| >= max_msg_types_per_log_entry )
|
||||
{
|
||||
Log::write(LOG, log_info);
|
||||
delete join_data[msg$xid];
|
||||
}
|
||||
}
|
||||
@endif
|
||||
|
||||
|
|
|
|||
|
|
@ -16,8 +16,14 @@ export {
|
|||
uid: string &log;
|
||||
## Identifier for the connection.
|
||||
id: conn_id &log;
|
||||
## Modbus transaction ID
|
||||
tid: count &log &optional;
|
||||
## The terminal unit identifier for the message
|
||||
unit: count &log &optional;
|
||||
## The name of the function message that was sent.
|
||||
func: string &log &optional;
|
||||
## Whether this PDU was a response ("RESP") or request ("REQ")
|
||||
pdu_type: string &log &optional;
|
||||
## The exception if the response was a failure.
|
||||
exception: string &log &optional;
|
||||
};
|
||||
|
|
@ -48,14 +54,18 @@ event modbus_message(c: connection, headers: ModbusHeaders, is_orig: bool) &prio
|
|||
}
|
||||
|
||||
c$modbus$ts = network_time();
|
||||
c$modbus$tid = headers$tid;
|
||||
c$modbus$unit = headers$uid;
|
||||
c$modbus$func = function_codes[headers$function_code];
|
||||
## If this message is from the TCP originator, it is a request. Otherwise,
|
||||
## it is a response.
|
||||
c$modbus$pdu_type = is_orig ? "REQ" : "RESP";
|
||||
}
|
||||
|
||||
event modbus_message(c: connection, headers: ModbusHeaders, is_orig: bool) &priority=-5
|
||||
{
|
||||
# Only log upon replies.
|
||||
# Also, don't log now if this is an exception (log in the exception event handler)
|
||||
if ( ! is_orig && ( headers$function_code <= 0x81 || headers$function_code >= 0x98 ) )
|
||||
# Don't log now if this is an exception (log in the exception event handler)
|
||||
if ( headers$function_code <= 0x81 || headers$function_code >= 0x98 )
|
||||
Log::write(LOG, c$modbus);
|
||||
}
|
||||
|
||||
|
|
|
|||
|
|
@ -255,11 +255,9 @@ function finish(c: connection, remove_analyzer: bool)
|
|||
{
|
||||
log_record(c$ssl);
|
||||
if ( remove_analyzer && disable_analyzer_after_detection && c?$ssl && c$ssl?$analyzer_id )
|
||||
{
|
||||
disable_analyzer(c$id, c$ssl$analyzer_id);
|
||||
if ( disable_analyzer(c$id, c$ssl$analyzer_id) )
|
||||
delete c$ssl$analyzer_id;
|
||||
}
|
||||
}
|
||||
|
||||
event ssl_client_hello(c: connection, version: count, record_version: count, possible_ts: time, client_random: string, session_id: string, ciphers: index_vec, comp_methods: index_vec) &priority=5
|
||||
{
|
||||
|
|
|
|||
|
|
@ -58,7 +58,7 @@ function extract_first_email_addr(str: string): string
|
|||
function split_mime_email_addresses(line: string): set[string]
|
||||
{
|
||||
local output = string_set();
|
||||
local addrs = find_all(line, /(\"[^"]*\")?[^,]+/);
|
||||
local addrs = find_all(line, /(\"[^"]*\")?[^,]+@[^,]+/);
|
||||
for ( part in addrs )
|
||||
{
|
||||
add output[strip(part)];
|
||||
|
|
|
|||
|
|
@ -234,13 +234,38 @@ function get_emails(a: addr): string
|
|||
return fmt_email_string(find_all_emails(a));
|
||||
}
|
||||
|
||||
event zeek_init() &priority=10
|
||||
function update_local_nets_table(id: string, new_value: set[subnet]): set[subnet]
|
||||
{
|
||||
# Create the local_nets mapping table.
|
||||
for ( cidr in new_value )
|
||||
local_nets_table[cidr] = cidr;
|
||||
return new_value;
|
||||
}
|
||||
|
||||
function update_local_zones_regex(id: string, new_value: set[string]): set[string]
|
||||
{
|
||||
# Double backslashes are needed due to string parsing.
|
||||
local_dns_suffix_regex = set_to_regex(local_zones, "(^\\.?|\\.)(~~)$");
|
||||
local_dns_neighbor_suffix_regex = set_to_regex(neighbor_zones, "(^\\.?|\\.)(~~)$");
|
||||
|
||||
# Create the local_nets mapping table.
|
||||
for ( cidr in Site::local_nets )
|
||||
local_nets_table[cidr] = cidr;
|
||||
local_dns_suffix_regex = set_to_regex(new_value, "(^\\.?|\\.)(~~)$");
|
||||
return new_value;
|
||||
}
|
||||
|
||||
function update_neighbor_zones_regex(id: string, new_value: set[string]): set[string]
|
||||
{
|
||||
local_dns_neighbor_suffix_regex = set_to_regex(new_value, "(^\\.?|\\.)(~~)$");
|
||||
return new_value;
|
||||
}
|
||||
|
||||
event zeek_init() &priority=10
|
||||
{
|
||||
# Have these run with a lower priority so we account for additions/removals
|
||||
# from user created change handlers.
|
||||
Option::set_change_handler("Site::local_nets", update_local_nets_table, -5);
|
||||
Option::set_change_handler("Site::local_zones", update_local_zones_regex, -5);
|
||||
Option::set_change_handler("Site::neighbor_zones", update_neighbor_zones_regex, -5);
|
||||
|
||||
# Use change handler to initialize local_nets mapping table and zones
|
||||
# regexes.
|
||||
update_local_nets_table("Site::local_nets", Site::local_nets);
|
||||
update_local_zones_regex("Site::local_zones", Site::local_zones);
|
||||
update_neighbor_zones_regex("Site::neighbor_zones", Site::neighbor_zones);
|
||||
}
|
||||
|
|
|
|||
|
|
@ -204,7 +204,7 @@ function send_deploy_response(req: Management::Request::Request)
|
|||
}
|
||||
|
||||
Management::Log::info(fmt("tx Management::Agent::API::deploy_response %s",
|
||||
Management::result_to_string(res)));
|
||||
Management::result_vec_to_string(req$results)));
|
||||
Broker::publish(agent_topic(),
|
||||
Management::Agent::API::deploy_response, req$id, req$results);
|
||||
|
||||
|
|
@ -262,6 +262,8 @@ event Management::Agent::Runtime::trigger_log_archival(run_archival: bool)
|
|||
|
||||
event Management::Supervisor::API::notify_node_exit(node: string, outputs: Management::NodeOutputs)
|
||||
{
|
||||
Management::Log::info(fmt("rx Management::Supervisor::API::notify_node_exit %s", node));
|
||||
|
||||
if ( node in g_nodes )
|
||||
g_outputs[node] = outputs;
|
||||
}
|
||||
|
|
@ -753,7 +755,8 @@ event Management::Node::API::node_dispatch_response(reqid: string, result: Manag
|
|||
|
||||
event Management::Agent::API::node_dispatch_request(reqid: string, action: vector of string, nodes: set[string])
|
||||
{
|
||||
Management::Log::info(fmt("rx Management::Agent::API::node_dispatch_request %s %s %s", reqid, action, nodes));
|
||||
Management::Log::info(fmt("rx Management::Agent::API::node_dispatch_request %s %s %s",
|
||||
reqid, action, Management::Util::set_to_vector(nodes)));
|
||||
|
||||
local node: string;
|
||||
local cluster_nodes: set[string];
|
||||
|
|
|
|||
|
|
@ -1174,11 +1174,11 @@ event Management::Controller::API::get_id_value_request(reqid: string, id: strin
|
|||
|
||||
local res: Management::Result;
|
||||
|
||||
# Special case: if we have no instances, respond right away.
|
||||
if ( |g_instances_known| == 0 )
|
||||
# Special case: if we have no deployed cluster, respond right away.
|
||||
if ( |g_instances| == 0 )
|
||||
{
|
||||
Management::Log::info(fmt("tx Management::Controller::API::get_id_value_response %s", reqid));
|
||||
res = Management::Result($reqid=reqid, $success=F, $error="no instances connected");
|
||||
res = Management::Result($reqid=reqid, $success=F, $error="no cluster deployed");
|
||||
Broker::publish(Management::Controller::topic,
|
||||
Management::Controller::API::get_id_value_response,
|
||||
reqid, vector(res));
|
||||
|
|
|
|||
|
|
@ -57,7 +57,8 @@ global g_dispatch_table: table[string] of DispatchCallback = {
|
|||
|
||||
event Management::Node::API::node_dispatch_request(reqid: string, action: vector of string, nodes: set[string])
|
||||
{
|
||||
Management::Log::info(fmt("rx Management::Node::API::node_dispatch_request %s %s %s", reqid, action, nodes));
|
||||
Management::Log::info(fmt("rx Management::Node::API::node_dispatch_request %s %s %s",
|
||||
reqid, action, Management::Util::set_to_vector(nodes)));
|
||||
|
||||
if ( |nodes| > 0 && Cluster::node !in nodes )
|
||||
{
|
||||
|
|
@ -102,8 +103,11 @@ event Broker::peer_added(peer: Broker::EndpointInfo, msg: string)
|
|||
# If this is the agent peering, notify it that we're ready
|
||||
if ( peer$network$address == epi$network$address &&
|
||||
peer$network$bound_port == epi$network$bound_port )
|
||||
{
|
||||
Management::Log::info(fmt("tx Management::Node::API::notify_node_hello %s", Cluster::node));
|
||||
Broker::publish(node_topic, Management::Node::API::notify_node_hello, Cluster::node);
|
||||
}
|
||||
}
|
||||
|
||||
event zeek_init()
|
||||
{
|
||||
|
|
|
|||
199
scripts/policy/frameworks/telemetry/log.zeek
Normal file
199
scripts/policy/frameworks/telemetry/log.zeek
Normal file
|
|
@ -0,0 +1,199 @@
|
|||
##! Implementation of a telemetry.log and telemetry_histogram.log file
|
||||
##! using metrics accessible via the Telemetry module.
|
||||
|
||||
@load base/frameworks/telemetry
|
||||
|
||||
module Telemetry;
|
||||
|
||||
export {
|
||||
redef enum Log::ID += { LOG, LOG_HISTOGRAM };
|
||||
|
||||
## How often metrics are reported.
|
||||
option log_interval = 60sec;
|
||||
|
||||
## Only metrics with prefixes in this set will be included in the
|
||||
## `telemetry.log` and `telemetry_histogram.log` files by default.
|
||||
## Setting this option to an empty set includes all prefixes.
|
||||
##
|
||||
## For more fine-grained customization, setting this option to an
|
||||
## empty set and implementing the :zeek:see:`Telemetry::log_policy`
|
||||
## and :zeek:see:`Telemetry::log_policy_histogram` hooks to filter
|
||||
## individual records is recommended.
|
||||
option log_prefixes: set[string] = {"process", "zeek"};
|
||||
|
||||
## Record type used for logging counter and gauge metrics.
|
||||
type Info: record {
|
||||
## Timestamp of reporting.
|
||||
ts: time &log;
|
||||
|
||||
## Peer that generated this log.
|
||||
peer: string &log;
|
||||
|
||||
## Contains the value "counter" or "gauge" depending on
|
||||
## the underlying metric type.
|
||||
metric_type: string &log;
|
||||
|
||||
## The prefix (namespace) of the metric.
|
||||
prefix: string &log;
|
||||
|
||||
## The name of the metric.
|
||||
name: string &log;
|
||||
|
||||
## The unit of this metric, or unset if unit-less.
|
||||
unit: string &log &optional;
|
||||
|
||||
## The names of the individual labels.
|
||||
labels: vector of string &log;
|
||||
|
||||
## The values of the labels as listed in ``labels``.
|
||||
label_values: vector of string &log;
|
||||
|
||||
## The value of this metric.
|
||||
value: double &log;
|
||||
};
|
||||
|
||||
## Record type used for logging histogram metrics.
|
||||
type HistogramInfo: record {
|
||||
## Timestamp of reporting.
|
||||
ts: time &log;
|
||||
|
||||
## Peer that generated this log.
|
||||
peer: string &log;
|
||||
|
||||
## The prefix (namespace) of the metric.
|
||||
prefix: string &log;
|
||||
|
||||
## The name of the metric.
|
||||
name: string &log;
|
||||
|
||||
## The unit of this metric, or unset if unit-less.
|
||||
unit: string &log &optional;
|
||||
|
||||
## The names of the individual labels.
|
||||
labels: vector of string &log;
|
||||
|
||||
## The values of the labels as listed in ``labels``.
|
||||
label_values: vector of string &log;
|
||||
|
||||
## The bounds of the individual buckets
|
||||
bounds: vector of double &log;
|
||||
|
||||
## The number of observations within each individual bucket.
|
||||
values: vector of double &log;
|
||||
|
||||
## The sum over all observations
|
||||
sum: double &log;
|
||||
|
||||
## The total number of observations.
|
||||
observations: double &log;
|
||||
};
|
||||
|
||||
## A default logging policy hook for the stream.
|
||||
global log_policy: Log::PolicyHook;
|
||||
|
||||
## A default logging policy hook for the histogram stream.
|
||||
global log_policy_histogram: Log::PolicyHook;
|
||||
|
||||
## Event triggered for every record in the stream.
|
||||
global log_telemetry: event(rec: Info);
|
||||
|
||||
## Event triggered for every record in the histogram stream.
|
||||
global log_telemetry_histogram: event(rec: HistogramInfo);
|
||||
}
|
||||
|
||||
function do_log()
|
||||
{
|
||||
local ts = network_time();
|
||||
local metrics = Telemetry::collect_metrics();
|
||||
|
||||
for ( i in metrics )
|
||||
{
|
||||
local m = metrics[i];
|
||||
|
||||
# Histograms don't have single values, skip over them.
|
||||
if ( m$opts$metric_type == DOUBLE_HISTOGRAM || m$opts$metric_type == INT_HISTOGRAM )
|
||||
next;
|
||||
|
||||
if ( |log_prefixes| > 0 && m$opts$prefix !in log_prefixes )
|
||||
next;
|
||||
|
||||
# Render the metric_type as a short string. Unknown
|
||||
# shouldn't really happen, but lets have a fallback.
|
||||
local metric_type = "unknown";
|
||||
switch ( m$opts$metric_type ) {
|
||||
case DOUBLE_COUNTER, INT_COUNTER:
|
||||
metric_type = "counter";
|
||||
break;
|
||||
case DOUBLE_GAUGE, INT_GAUGE:
|
||||
metric_type = "gauge";
|
||||
break;
|
||||
}
|
||||
|
||||
local rec = Info($ts=ts,
|
||||
$peer=peer_description,
|
||||
$metric_type=metric_type,
|
||||
$prefix=m$opts$prefix,
|
||||
$name=m$opts$name,
|
||||
$labels=m$opts$labels,
|
||||
$label_values=m$labels,
|
||||
$value=m$value);
|
||||
|
||||
if ( m$opts$unit != "1" )
|
||||
rec$unit = m$opts$unit;
|
||||
|
||||
Log::write(LOG, rec);
|
||||
}
|
||||
|
||||
# Logging of histograms.
|
||||
ts = network_time();
|
||||
local histogram_metrics = Telemetry::collect_histogram_metrics();
|
||||
for ( i in histogram_metrics )
|
||||
{
|
||||
local hm = histogram_metrics[i];
|
||||
|
||||
if ( |log_prefixes| > 0 && hm$opts$prefix !in log_prefixes )
|
||||
next;
|
||||
|
||||
local hrec = HistogramInfo($ts=ts,
|
||||
$peer=peer_description,
|
||||
$prefix=hm$opts$prefix,
|
||||
$name=hm$opts$name,
|
||||
$labels=hm$opts$labels,
|
||||
$label_values=hm$labels,
|
||||
$bounds=hm$opts$bounds,
|
||||
$values=hm$values,
|
||||
$sum=hm$sum,
|
||||
$observations=hm$observations);
|
||||
|
||||
if ( hm$opts$unit != "1" )
|
||||
hrec$unit = hm$opts$unit;
|
||||
|
||||
Log::write(LOG_HISTOGRAM, hrec);
|
||||
}
|
||||
}
|
||||
|
||||
event Telemetry::log()
|
||||
{
|
||||
# We explicitly log once during zeek_done(), so short-circuit
|
||||
# here when we're already in the process of shutting down.
|
||||
if ( zeek_is_terminating() )
|
||||
return;
|
||||
|
||||
do_log();
|
||||
schedule log_interval { Telemetry::log() };
|
||||
}
|
||||
|
||||
event zeek_init() &priority=5
|
||||
{
|
||||
Log::create_stream(LOG, [$columns=Info, $ev=log_telemetry, $path="telemetry", $policy=log_policy]);
|
||||
Log::create_stream(LOG_HISTOGRAM, [$columns=HistogramInfo, $ev=log_telemetry_histogram, $path="telemetry_histogram", $policy=log_policy_histogram]);
|
||||
|
||||
schedule log_interval { Telemetry::log() };
|
||||
}
|
||||
|
||||
# Log late during zeek_done() once more. Any metric updates
|
||||
# afterwards won't be visible in the log.
|
||||
event zeek_done() &priority=-1000
|
||||
{
|
||||
do_log();
|
||||
}
|
||||
|
|
@ -89,6 +89,10 @@ redef digest_salt = "Please change this value.";
|
|||
# Extend email alerting to include hostnames
|
||||
@load policy/frameworks/notice/extend-email/hostnames
|
||||
|
||||
# Enable logging of telemetry data into telemetry.log and
|
||||
# telemetry_histogram.log.
|
||||
@load frameworks/telemetry/log
|
||||
|
||||
# Uncomment the following line to enable detection of the heartbleed attack. Enabling
|
||||
# this might impact performance a bit.
|
||||
# @load policy/protocols/ssl/heartbleed
|
||||
|
|
|
|||
|
|
@ -71,6 +71,7 @@
|
|||
@load frameworks/software/version-changes.zeek
|
||||
@load frameworks/software/vulnerable.zeek
|
||||
@load frameworks/software/windows-version-detection.zeek
|
||||
@load frameworks/telemetry/log.zeek
|
||||
@load integration/barnyard2/__load__.zeek
|
||||
@load integration/barnyard2/main.zeek
|
||||
@load integration/barnyard2/types.zeek
|
||||
|
|
|
|||
|
|
@ -1 +1 @@
|
|||
Subproject commit a60ffe6a41c0f335f1dcf22ec2e6bf246b9ab0ac
|
||||
Subproject commit 423e2df85c449c9575d17514722e4ad7eb6ea53d
|
||||
|
|
@ -136,4 +136,14 @@ void EventRegistry::SetErrorHandler(std::string_view name)
|
|||
std::string(name).c_str());
|
||||
}
|
||||
|
||||
void EventRegistry::ActivateAllHandlers()
|
||||
{
|
||||
auto event_names = AllHandlers();
|
||||
for ( const auto& name : event_names )
|
||||
{
|
||||
if ( auto event = Lookup(name) )
|
||||
event->SetGenerateAlways();
|
||||
}
|
||||
}
|
||||
|
||||
} // namespace zeek
|
||||
|
|
|
|||
|
|
@ -60,6 +60,17 @@ public:
|
|||
|
||||
void PrintDebug();
|
||||
|
||||
/**
|
||||
* Marks all event handlers as active.
|
||||
*
|
||||
* By default, zeek does not generate (raise) events that have not handled by
|
||||
* any scripts. This means that these events will be invisible to a lot of other
|
||||
* event handlers - and will not raise :zeek:id:`new_event`. Calling this
|
||||
* function will cause all event handlers to be raised. This is likely only
|
||||
* useful for debugging and fuzzing, and likely causes reduced performance.
|
||||
*/
|
||||
void ActivateAllHandlers();
|
||||
|
||||
private:
|
||||
std::map<std::string, std::unique_ptr<EventHandler>, std::less<>> handlers;
|
||||
// Tracks whether a given event handler was registered in a
|
||||
|
|
|
|||
38
src/Expr.cc
38
src/Expr.cc
|
|
@ -960,11 +960,34 @@ ValPtr BinaryExpr::Fold(Val* v1, Val* v2) const
|
|||
DO_UINT_FOLD(^);
|
||||
break;
|
||||
case EXPR_LSHIFT:
|
||||
DO_INT_FOLD(<<);
|
||||
{
|
||||
if ( is_integral )
|
||||
{
|
||||
if ( i1 < 0 )
|
||||
RuntimeError("left shifting a negative number is undefined");
|
||||
|
||||
i3 = i1 << static_cast<zeek_uint_t>(i2);
|
||||
}
|
||||
|
||||
else if ( is_unsigned )
|
||||
u3 = u1 << u2;
|
||||
|
||||
else
|
||||
RuntimeErrorWithCallStack("bad type in BinaryExpr::Fold");
|
||||
break;
|
||||
}
|
||||
case EXPR_RSHIFT:
|
||||
DO_INT_FOLD(>>);
|
||||
{
|
||||
if ( is_integral )
|
||||
i3 = i1 >> static_cast<zeek_uint_t>(i2);
|
||||
|
||||
else if ( is_unsigned )
|
||||
u3 = u1 >> u2;
|
||||
|
||||
else
|
||||
RuntimeErrorWithCallStack("bad type in BinaryExpr::Fold");
|
||||
break;
|
||||
}
|
||||
|
||||
case EXPR_AND_AND:
|
||||
DO_INT_FOLD(&&);
|
||||
|
|
@ -2166,6 +2189,9 @@ BitExpr::BitExpr(ExprTag arg_tag, ExprPtr arg_op1, ExprPtr arg_op2)
|
|||
|
||||
if ( tag == EXPR_LSHIFT || tag == EXPR_RSHIFT )
|
||||
{
|
||||
if ( (is_vector(op1) || is_vector(op2)) && ! (is_vector(op1) && is_vector(op2)) )
|
||||
ExprError("cannot mix vectors and scalars for shift operations");
|
||||
|
||||
if ( IsIntegral(bt1) && bt2 == TYPE_COUNT )
|
||||
{
|
||||
if ( is_vector(op1) || is_vector(op2) )
|
||||
|
|
@ -2173,13 +2199,19 @@ BitExpr::BitExpr(ExprTag arg_tag, ExprPtr arg_op1, ExprPtr arg_op2)
|
|||
else
|
||||
SetType(base_type(bt1));
|
||||
}
|
||||
|
||||
else if ( IsIntegral(bt1) && bt2 == TYPE_INT )
|
||||
ExprError("requires \"count\" right operand");
|
||||
|
||||
else
|
||||
ExprError("requires integral operands");
|
||||
|
||||
return; // because following scalar check isn't apt
|
||||
}
|
||||
|
||||
else if ( (bt1 == TYPE_COUNT) && (bt2 == TYPE_COUNT) )
|
||||
CheckScalarAggOp();
|
||||
|
||||
if ( (bt1 == TYPE_COUNT) && (bt2 == TYPE_COUNT) )
|
||||
{
|
||||
if ( is_vector(op1) || is_vector(op2) )
|
||||
SetType(make_intrusive<VectorType>(base_type(TYPE_COUNT)));
|
||||
|
|
|
|||
|
|
@ -39,6 +39,10 @@ static VectorValPtr BuildOptionsVal(const u_char* data, int len)
|
|||
{
|
||||
// PadN or other option
|
||||
uint16_t off = 2 * sizeof(uint8_t);
|
||||
|
||||
if ( len < opt->ip6o_len + off )
|
||||
break;
|
||||
|
||||
rv->Assign(1, opt->ip6o_len);
|
||||
rv->Assign(2, new String(data + off, opt->ip6o_len, true));
|
||||
data += opt->ip6o_len + off;
|
||||
|
|
|
|||
188
src/RE.cc
188
src/RE.cc
|
|
@ -7,6 +7,7 @@
|
|||
#include <cstdlib>
|
||||
#include <utility>
|
||||
|
||||
#include "zeek/3rdparty/doctest.h"
|
||||
#include "zeek/CCL.h"
|
||||
#include "zeek/DFA.h"
|
||||
#include "zeek/EquivClass.h"
|
||||
|
|
@ -16,7 +17,8 @@
|
|||
zeek::detail::CCL* zeek::detail::curr_ccl = nullptr;
|
||||
zeek::detail::Specific_RE_Matcher* zeek::detail::rem = nullptr;
|
||||
zeek::detail::NFA_Machine* zeek::detail::nfa = nullptr;
|
||||
int zeek::detail::case_insensitive = 0;
|
||||
bool zeek::detail::case_insensitive = false;
|
||||
bool zeek::detail::re_single_line = false;
|
||||
|
||||
extern int RE_parse(void);
|
||||
extern void RE_set_input(const char* str);
|
||||
|
|
@ -27,13 +29,11 @@ namespace zeek
|
|||
namespace detail
|
||||
{
|
||||
|
||||
Specific_RE_Matcher::Specific_RE_Matcher(match_type arg_mt, int arg_multiline)
|
||||
: equiv_class(NUM_SYM)
|
||||
Specific_RE_Matcher::Specific_RE_Matcher(match_type arg_mt, bool arg_multiline)
|
||||
: mt(arg_mt), multiline(arg_multiline), equiv_class(NUM_SYM)
|
||||
{
|
||||
mt = arg_mt;
|
||||
multiline = arg_multiline;
|
||||
any_ccl = nullptr;
|
||||
pattern_text = nullptr;
|
||||
single_line_ccl = nullptr;
|
||||
dfa = nullptr;
|
||||
ecs = nullptr;
|
||||
accepted = new AcceptingSet();
|
||||
|
|
@ -45,14 +45,25 @@ Specific_RE_Matcher::~Specific_RE_Matcher()
|
|||
delete ccl_list[i];
|
||||
|
||||
Unref(dfa);
|
||||
delete[] pattern_text;
|
||||
delete accepted;
|
||||
}
|
||||
|
||||
CCL* Specific_RE_Matcher::AnyCCL()
|
||||
CCL* Specific_RE_Matcher::AnyCCL(bool single_line_mode)
|
||||
{
|
||||
if ( single_line_mode )
|
||||
{
|
||||
if ( ! single_line_ccl )
|
||||
{
|
||||
single_line_ccl = new CCL();
|
||||
single_line_ccl->Negate();
|
||||
EC()->CCL_Use(single_line_ccl);
|
||||
}
|
||||
|
||||
return single_line_ccl;
|
||||
}
|
||||
|
||||
if ( ! any_ccl )
|
||||
{ // Create the '.' character class.
|
||||
{
|
||||
any_ccl = new CCL();
|
||||
if ( ! multiline )
|
||||
any_ccl->Add('\n');
|
||||
|
|
@ -89,51 +100,38 @@ void Specific_RE_Matcher::AddExactPat(const char* new_pat)
|
|||
|
||||
void Specific_RE_Matcher::AddPat(const char* new_pat, const char* orig_fmt, const char* app_fmt)
|
||||
{
|
||||
int n = strlen(new_pat);
|
||||
|
||||
if ( pattern_text )
|
||||
n += strlen(pattern_text) + strlen(app_fmt);
|
||||
if ( ! pattern_text.empty() )
|
||||
pattern_text = util::fmt(app_fmt, pattern_text.c_str(), new_pat);
|
||||
else
|
||||
n += strlen(orig_fmt);
|
||||
|
||||
char* s = new char[n + 5 /* slop */];
|
||||
|
||||
if ( pattern_text )
|
||||
sprintf(s, app_fmt, pattern_text, new_pat);
|
||||
else
|
||||
sprintf(s, orig_fmt, new_pat);
|
||||
|
||||
delete[] pattern_text;
|
||||
pattern_text = s;
|
||||
pattern_text = util::fmt(orig_fmt, new_pat);
|
||||
}
|
||||
|
||||
void Specific_RE_Matcher::MakeCaseInsensitive()
|
||||
{
|
||||
const char fmt[] = "(?i:%s)";
|
||||
int n = strlen(pattern_text) + strlen(fmt);
|
||||
pattern_text = util::fmt(fmt, pattern_text.c_str());
|
||||
}
|
||||
|
||||
char* s = new char[n + 5 /* slop */];
|
||||
|
||||
snprintf(s, n + 5, fmt, pattern_text);
|
||||
|
||||
delete[] pattern_text;
|
||||
pattern_text = s;
|
||||
void Specific_RE_Matcher::MakeSingleLine()
|
||||
{
|
||||
const char fmt[] = "(?s:%s)";
|
||||
pattern_text = util::fmt(fmt, pattern_text.c_str());
|
||||
}
|
||||
|
||||
bool Specific_RE_Matcher::Compile(bool lazy)
|
||||
{
|
||||
if ( ! pattern_text )
|
||||
if ( pattern_text.empty() )
|
||||
return false;
|
||||
|
||||
rem = this;
|
||||
RE_set_input(pattern_text);
|
||||
RE_set_input(pattern_text.c_str());
|
||||
|
||||
int parse_status = RE_parse();
|
||||
RE_done_with_scan();
|
||||
|
||||
if ( parse_status )
|
||||
{
|
||||
reporter->Error("error compiling pattern /%s/", pattern_text);
|
||||
reporter->Error("error compiling pattern /%s/", pattern_text.c_str());
|
||||
Unref(nfa);
|
||||
nfa = nullptr;
|
||||
return false;
|
||||
|
|
@ -416,13 +414,10 @@ static RE_Matcher* matcher_merge(const RE_Matcher* re1, const RE_Matcher* re2, c
|
|||
const char* text1 = re1->PatternText();
|
||||
const char* text2 = re2->PatternText();
|
||||
|
||||
int n = strlen(text1) + strlen(text2) + strlen(merge_op) + 32 /* slop */;
|
||||
size_t n = strlen(text1) + strlen(text2) + strlen(merge_op) + 32 /* slop */;
|
||||
|
||||
char* merge_text = new char[n];
|
||||
snprintf(merge_text, n, "(%s)%s(%s)", text1, merge_op, text2);
|
||||
|
||||
RE_Matcher* merge = new RE_Matcher(merge_text);
|
||||
delete[] merge_text;
|
||||
std::string merge_text = util::fmt("(%s)%s(%s)", text1, merge_op, text2);
|
||||
RE_Matcher* merge = new RE_Matcher(merge_text.c_str());
|
||||
|
||||
merge->Compile();
|
||||
|
||||
|
|
@ -483,9 +478,122 @@ void RE_Matcher::MakeCaseInsensitive()
|
|||
is_case_insensitive = true;
|
||||
}
|
||||
|
||||
void RE_Matcher::MakeSingleLine()
|
||||
{
|
||||
re_anywhere->MakeSingleLine();
|
||||
re_exact->MakeSingleLine();
|
||||
|
||||
is_single_line = true;
|
||||
}
|
||||
|
||||
bool RE_Matcher::Compile(bool lazy)
|
||||
{
|
||||
return re_anywhere->Compile(lazy) && re_exact->Compile(lazy);
|
||||
}
|
||||
|
||||
TEST_SUITE("re_matcher")
|
||||
{
|
||||
|
||||
TEST_CASE("simple_pattern")
|
||||
{
|
||||
RE_Matcher match("[0-9]+");
|
||||
match.Compile();
|
||||
CHECK(strcmp(match.OrigText(), "[0-9]+") == 0);
|
||||
CHECK(strcmp(match.PatternText(), "^?([0-9]+)$?") == 0);
|
||||
CHECK(strcmp(match.AnywherePatternText(), "^?(.|\\n)*([0-9]+)") == 0);
|
||||
|
||||
CHECK(match.MatchExactly("12345"));
|
||||
CHECK_FALSE(match.MatchExactly("a12345"));
|
||||
|
||||
// The documentation for MatchAnywhere says that it returns the
|
||||
// "index just beyond where the first match occurs", which I would
|
||||
// think means *after* the match. This is returning the position
|
||||
// where the match starts though.
|
||||
CHECK(match.MatchAnywhere("a1234bcd") == 2);
|
||||
CHECK(match.MatchAnywhere("abcd") == 0);
|
||||
}
|
||||
|
||||
TEST_CASE("case_insensitive_mode")
|
||||
{
|
||||
RE_Matcher match("[a-z]+");
|
||||
match.MakeCaseInsensitive();
|
||||
match.Compile();
|
||||
CHECK(strcmp(match.PatternText(), "(?i:^?([a-z]+)$?)") == 0);
|
||||
|
||||
CHECK(match.MatchExactly("abcDEF"));
|
||||
}
|
||||
|
||||
TEST_CASE("multi_pattern")
|
||||
{
|
||||
RE_Matcher match("[0-9]+");
|
||||
match.AddPat("[a-z]+");
|
||||
match.Compile();
|
||||
|
||||
CHECK(strcmp(match.PatternText(), "(^?([0-9]+)$?)|(^?([a-z]+)$?)") == 0);
|
||||
|
||||
CHECK(match.MatchExactly("abc"));
|
||||
CHECK(match.MatchExactly("123"));
|
||||
CHECK_FALSE(match.MatchExactly("abc123"));
|
||||
}
|
||||
|
||||
TEST_CASE("modes_multi_pattern")
|
||||
{
|
||||
RE_Matcher match("[a-m]+");
|
||||
match.MakeCaseInsensitive();
|
||||
|
||||
match.AddPat("[n-z]+");
|
||||
match.Compile();
|
||||
|
||||
CHECK(strcmp(match.PatternText(), "((?i:^?([a-m]+)$?))|(^?([n-z]+)$?)") == 0);
|
||||
CHECK(match.MatchExactly("aBc"));
|
||||
CHECK(match.MatchExactly("nop"));
|
||||
CHECK_FALSE(match.MatchExactly("NoP"));
|
||||
}
|
||||
|
||||
TEST_CASE("single_line_mode")
|
||||
{
|
||||
RE_Matcher match(".*");
|
||||
match.MakeSingleLine();
|
||||
match.Compile();
|
||||
|
||||
CHECK(strcmp(match.PatternText(), "(?s:^?(.*)$?)") == 0);
|
||||
CHECK(match.MatchExactly("abc\ndef"));
|
||||
|
||||
RE_Matcher match2("fOO.*bAR");
|
||||
match2.MakeSingleLine();
|
||||
match2.Compile();
|
||||
|
||||
CHECK(strcmp(match2.PatternText(), "(?s:^?(fOO.*bAR)$?)") == 0);
|
||||
CHECK(match.MatchExactly("fOOab\ncdbAR"));
|
||||
|
||||
RE_Matcher match3("b.r");
|
||||
match3.MakeSingleLine();
|
||||
match3.Compile();
|
||||
CHECK(match3.MatchExactly("bar"));
|
||||
CHECK(match3.MatchExactly("b\nr"));
|
||||
|
||||
RE_Matcher match4("a.c");
|
||||
match4.MakeSingleLine();
|
||||
match4.AddPat("def");
|
||||
match4.Compile();
|
||||
CHECK(match4.MatchExactly("abc"));
|
||||
CHECK(match4.MatchExactly("a\nc"));
|
||||
}
|
||||
|
||||
TEST_CASE("disjunction")
|
||||
{
|
||||
RE_Matcher match1("a.c");
|
||||
match1.MakeSingleLine();
|
||||
match1.Compile();
|
||||
RE_Matcher match2("def");
|
||||
match2.Compile();
|
||||
auto dj = detail::RE_Matcher_disjunction(&match1, &match2);
|
||||
CHECK(dj->MatchExactly("abc"));
|
||||
CHECK(dj->MatchExactly("a.c"));
|
||||
CHECK(dj->MatchExactly("a\nc"));
|
||||
CHECK(dj->MatchExactly("def"));
|
||||
delete dj;
|
||||
}
|
||||
}
|
||||
|
||||
} // namespace zeek
|
||||
|
|
|
|||
26
src/RE.h
26
src/RE.h
|
|
@ -32,7 +32,8 @@ class DFA_State;
|
|||
class Specific_RE_Matcher;
|
||||
class CCL;
|
||||
|
||||
extern int case_insensitive;
|
||||
extern bool case_insensitive;
|
||||
extern bool re_single_line;
|
||||
extern CCL* curr_ccl;
|
||||
extern NFA_Machine* nfa;
|
||||
extern Specific_RE_Matcher* rem;
|
||||
|
|
@ -59,14 +60,15 @@ enum match_type
|
|||
class Specific_RE_Matcher
|
||||
{
|
||||
public:
|
||||
explicit Specific_RE_Matcher(match_type mt, int multiline = 0);
|
||||
explicit Specific_RE_Matcher(match_type mt, bool multiline = false);
|
||||
~Specific_RE_Matcher();
|
||||
|
||||
void AddPat(const char* pat);
|
||||
|
||||
void MakeCaseInsensitive();
|
||||
void MakeSingleLine();
|
||||
|
||||
void SetPat(const char* pat) { pattern_text = util::copy_string(pat); }
|
||||
void SetPat(const char* pat) { pattern_text = pat; }
|
||||
|
||||
bool Compile(bool lazy = false);
|
||||
|
||||
|
|
@ -90,7 +92,7 @@ public:
|
|||
return nullptr;
|
||||
}
|
||||
CCL* LookupCCL(int index) { return ccl_list[index]; }
|
||||
CCL* AnyCCL();
|
||||
CCL* AnyCCL(bool single_line_mode = false);
|
||||
|
||||
void ConvertCCLs();
|
||||
|
||||
|
|
@ -117,7 +119,7 @@ public:
|
|||
|
||||
EquivClass* EC() { return &equiv_class; }
|
||||
|
||||
const char* PatternText() const { return pattern_text; }
|
||||
const char* PatternText() const { return pattern_text.c_str(); }
|
||||
|
||||
DFA_Machine* DFA() const { return dfa; }
|
||||
|
||||
|
|
@ -135,17 +137,21 @@ protected:
|
|||
bool MatchAll(const u_char* bv, int n);
|
||||
|
||||
match_type mt;
|
||||
int multiline;
|
||||
char* pattern_text;
|
||||
bool multiline;
|
||||
|
||||
std::string pattern_text;
|
||||
|
||||
std::map<std::string, std::string> defs;
|
||||
std::map<std::string, CCL*> ccl_dict;
|
||||
std::vector<char> modifiers;
|
||||
PList<CCL> ccl_list;
|
||||
EquivClass equiv_class;
|
||||
int* ecs;
|
||||
DFA_Machine* dfa;
|
||||
CCL* any_ccl;
|
||||
AcceptingSet* accepted;
|
||||
|
||||
CCL* any_ccl;
|
||||
CCL* single_line_ccl;
|
||||
};
|
||||
|
||||
class RE_Match_State
|
||||
|
|
@ -205,6 +211,9 @@ public:
|
|||
void MakeCaseInsensitive();
|
||||
bool IsCaseInsensitive() const { return is_case_insensitive; }
|
||||
|
||||
void MakeSingleLine();
|
||||
bool IsSingleLine() const { return is_single_line; }
|
||||
|
||||
bool Compile(bool lazy = false);
|
||||
|
||||
// Returns true if s exactly matches the pattern, false otherwise.
|
||||
|
|
@ -240,6 +249,7 @@ protected:
|
|||
detail::Specific_RE_Matcher* re_exact;
|
||||
|
||||
bool is_case_insensitive = false;
|
||||
bool is_single_line = false;
|
||||
};
|
||||
|
||||
} // namespace zeek
|
||||
|
|
|
|||
|
|
@ -526,7 +526,7 @@ void RuleMatcher::BuildPatternSets(RuleHdrTest::pattern_set_list* dst, const str
|
|||
if ( group_exprs.length() > sig_max_group_size || i == exprs.length() )
|
||||
{
|
||||
RuleHdrTest::PatternSet* set = new RuleHdrTest::PatternSet;
|
||||
set->re = new Specific_RE_Matcher(MATCH_EXACTLY, 1);
|
||||
set->re = new Specific_RE_Matcher(MATCH_EXACTLY, true);
|
||||
set->re->CompileSet(group_exprs, group_ids);
|
||||
set->patterns = group_exprs;
|
||||
set->ids = group_ids;
|
||||
|
|
|
|||
|
|
@ -2185,6 +2185,12 @@ TraversalCode WhenStmt::Traverse(TraversalCallback* cb) const
|
|||
}
|
||||
}
|
||||
|
||||
if ( wi->TimeoutExpr() )
|
||||
{
|
||||
tc = wi->TimeoutExpr()->Traverse(cb);
|
||||
HANDLE_TC_STMT_PRE(tc);
|
||||
}
|
||||
|
||||
tc = cb->PostStmt(this);
|
||||
HANDLE_TC_STMT_POST(tc);
|
||||
}
|
||||
|
|
|
|||
|
|
@ -932,6 +932,8 @@ public:
|
|||
|
||||
const PDict<TableEntryVal>* Get() const { return table_val; }
|
||||
|
||||
const detail::CompositeHash* GetTableHash() const { return table_hash; }
|
||||
|
||||
// Returns the size of the table.
|
||||
int Size() const;
|
||||
int RecursiveSize() const;
|
||||
|
|
@ -1593,10 +1595,8 @@ public:
|
|||
ValPtr ValAt(unsigned int index) const { return At(index); }
|
||||
|
||||
bool Has(unsigned int index) const
|
||||
// Version to use once std::optional implementation is merged.
|
||||
// { return index < vector_val->size() && vector_val[index]; }
|
||||
{
|
||||
return At(index) != nullptr;
|
||||
return index < vector_val->size() && (*vector_val)[index];
|
||||
}
|
||||
|
||||
/**
|
||||
|
|
|
|||
11
src/Var.cc
11
src/Var.cc
|
|
@ -429,14 +429,25 @@ void add_type(ID* id, TypePtr t, std::unique_ptr<std::vector<AttrPtr>> attr)
|
|||
{
|
||||
std::string new_type_name = id->Name();
|
||||
std::string old_type_name = t->GetName();
|
||||
|
||||
TypePtr tnew;
|
||||
|
||||
if ( (t->Tag() == TYPE_RECORD || t->Tag() == TYPE_ENUM) && old_type_name.empty() )
|
||||
// An extensible type (record/enum) being declared for first time.
|
||||
tnew = std::move(t);
|
||||
else
|
||||
{
|
||||
// If the old type is an error or the old type doesn't exist, then return
|
||||
// an error instead of trying to clone it.
|
||||
if ( t->Tag() == TYPE_ERROR && t->InternalType() == TYPE_INTERNAL_ERROR )
|
||||
{
|
||||
reporter->Error("Error trying to create alias to nonexistent type");
|
||||
return;
|
||||
}
|
||||
|
||||
// Clone the type to preserve type name aliasing.
|
||||
tnew = t->ShallowClone();
|
||||
}
|
||||
|
||||
Type::RegisterAlias(new_type_name, tnew);
|
||||
|
||||
|
|
|
|||
|
|
@ -81,3 +81,7 @@ target_link_libraries(zeek_fuzzer_shared
|
|||
add_fuzz_target(dns)
|
||||
add_fuzz_target(pop3)
|
||||
add_fuzz_target(packet)
|
||||
add_fuzz_target(http)
|
||||
add_fuzz_target(imap)
|
||||
add_fuzz_target(smtp)
|
||||
add_fuzz_target(ftp)
|
||||
|
|
|
|||
|
|
@ -23,6 +23,9 @@ First configure and build for fuzzing (with libFuzzer) and code coverage::
|
|||
variable may be changed to use another flag or direct path to fuzzing engine
|
||||
library to link against.
|
||||
|
||||
Text/Dictionary-based Corpus
|
||||
````````````````````````````
|
||||
|
||||
Now start fuzzing to generate an initial corpus (this uses the POP3 fuzzer as
|
||||
an example)::
|
||||
|
||||
|
|
@ -54,6 +57,24 @@ commit)::
|
|||
|
||||
zip -j ../src/fuzzers/pop3-corpus.zip min-corpus/*
|
||||
|
||||
pcap-based Corpus
|
||||
`````````````````
|
||||
|
||||
A corpus can also be generated from representative pcp files using the
|
||||
``pcap-to-pkt`` application from pcap_simplify_. The fuzzers only handle a
|
||||
single connection at a time, so pcap files with multiple connections will
|
||||
need to be split using ``PcapSplitter`` from PcapPlusPlus_ or something
|
||||
similar. Once the file has been split, the individual connections can be
|
||||
converted into separate pkt files. The ``http`` fuzzer is a good example
|
||||
of a fuzzer using such files. The corpus for that fuzzer was initially
|
||||
generated from a subset of the pcap files located in ``testing/btest/Traces/http``.
|
||||
|
||||
.. _pcap_simplify: https://github.com/JustinAzoff/pcap_simplify
|
||||
.. _PcapPlusPlus: https://github.com/seladb/PcapPlusPlus
|
||||
|
||||
The converted pkt files can then be zipped as in the text-based section
|
||||
above.
|
||||
|
||||
Example Build: Run Standalone Fuzz Targets
|
||||
------------------------------------------
|
||||
|
||||
BIN
src/fuzzers/ftp-corpus.zip
Normal file
BIN
src/fuzzers/ftp-corpus.zip
Normal file
Binary file not shown.
78
src/fuzzers/ftp-fuzzer.cc
Normal file
78
src/fuzzers/ftp-fuzzer.cc
Normal file
|
|
@ -0,0 +1,78 @@
|
|||
#include <binpac.h>
|
||||
|
||||
#include "zeek/Conn.h"
|
||||
#include "zeek/RunState.h"
|
||||
#include "zeek/analyzer/Analyzer.h"
|
||||
#include "zeek/analyzer/Manager.h"
|
||||
#include "zeek/analyzer/protocol/pia/PIA.h"
|
||||
#include "zeek/analyzer/protocol/tcp/TCP.h"
|
||||
#include "zeek/fuzzers/FuzzBuffer.h"
|
||||
#include "zeek/fuzzers/fuzzer-setup.h"
|
||||
#include "zeek/packet_analysis/protocol/tcp/TCPSessionAdapter.h"
|
||||
#include "zeek/session/Manager.h"
|
||||
|
||||
static constexpr auto ZEEK_FUZZ_ANALYZER = "ftp";
|
||||
|
||||
static zeek::Connection* add_connection()
|
||||
{
|
||||
static constexpr double network_time_start = 1439471031;
|
||||
zeek::run_state::detail::update_network_time(network_time_start);
|
||||
|
||||
zeek::Packet p;
|
||||
zeek::ConnTuple conn_id;
|
||||
conn_id.src_addr = zeek::IPAddr("1.2.3.4");
|
||||
conn_id.dst_addr = zeek::IPAddr("5.6.7.8");
|
||||
conn_id.src_port = htons(23132);
|
||||
conn_id.dst_port = htons(80);
|
||||
conn_id.is_one_way = false;
|
||||
conn_id.proto = TRANSPORT_TCP;
|
||||
zeek::detail::ConnKey key(conn_id);
|
||||
zeek::Connection* conn = new zeek::Connection(key, network_time_start, &conn_id, 1, &p);
|
||||
conn->SetTransport(TRANSPORT_TCP);
|
||||
zeek::session_mgr->Insert(conn);
|
||||
return conn;
|
||||
}
|
||||
|
||||
static zeek::analyzer::Analyzer* add_analyzer(zeek::Connection* conn)
|
||||
{
|
||||
auto* tcp = new zeek::packet_analysis::TCP::TCPSessionAdapter(conn);
|
||||
auto* pia = new zeek::analyzer::pia::PIA_TCP(conn);
|
||||
auto a = zeek::analyzer_mgr->InstantiateAnalyzer(ZEEK_FUZZ_ANALYZER, conn);
|
||||
tcp->AddChildAnalyzer(a);
|
||||
tcp->AddChildAnalyzer(pia->AsAnalyzer());
|
||||
conn->SetSessionAdapter(tcp, pia);
|
||||
return a;
|
||||
}
|
||||
|
||||
extern "C" int LLVMFuzzerTestOneInput(const uint8_t* data, size_t size)
|
||||
{
|
||||
zeek::detail::FuzzBuffer fb{data, size};
|
||||
|
||||
if ( ! fb.Valid() )
|
||||
return 0;
|
||||
|
||||
auto conn = add_connection();
|
||||
auto a = add_analyzer(conn);
|
||||
|
||||
for ( ;; )
|
||||
{
|
||||
auto chunk = fb.Next();
|
||||
|
||||
if ( ! chunk )
|
||||
break;
|
||||
|
||||
try
|
||||
{
|
||||
a->DeliverStream(chunk->size, chunk->data.get(), chunk->is_orig);
|
||||
}
|
||||
catch ( const binpac::Exception& e )
|
||||
{
|
||||
}
|
||||
|
||||
chunk = {};
|
||||
zeek::event_mgr.Drain();
|
||||
}
|
||||
|
||||
zeek::detail::fuzzer_cleanup_one_input();
|
||||
return 0;
|
||||
}
|
||||
|
|
@ -4,6 +4,7 @@
|
|||
#include <cstdlib>
|
||||
|
||||
#include "zeek/Event.h"
|
||||
#include "zeek/EventRegistry.h"
|
||||
#include "zeek/broker/Manager.h"
|
||||
#include "zeek/file_analysis/Manager.h"
|
||||
#include "zeek/session/Manager.h"
|
||||
|
|
@ -41,6 +42,11 @@ extern "C" int LLVMFuzzerInitialize(int* argc, char*** argv)
|
|||
if ( zeek::detail::setup(*argc, *argv, &options).code )
|
||||
abort();
|
||||
|
||||
// We have to trick the event handlers into returning true that they exist here
|
||||
// even if they don't, because otherwise we lose a bit of coverage where if
|
||||
// statements return false that would otherwise not.
|
||||
zeek::event_registry->ActivateAllHandlers();
|
||||
|
||||
return 0;
|
||||
}
|
||||
|
||||
|
|
|
|||
BIN
src/fuzzers/http-corpus.zip
Normal file
BIN
src/fuzzers/http-corpus.zip
Normal file
Binary file not shown.
78
src/fuzzers/http-fuzzer.cc
Normal file
78
src/fuzzers/http-fuzzer.cc
Normal file
|
|
@ -0,0 +1,78 @@
|
|||
#include <binpac.h>
|
||||
|
||||
#include "zeek/Conn.h"
|
||||
#include "zeek/RunState.h"
|
||||
#include "zeek/analyzer/Analyzer.h"
|
||||
#include "zeek/analyzer/Manager.h"
|
||||
#include "zeek/analyzer/protocol/pia/PIA.h"
|
||||
#include "zeek/analyzer/protocol/tcp/TCP.h"
|
||||
#include "zeek/fuzzers/FuzzBuffer.h"
|
||||
#include "zeek/fuzzers/fuzzer-setup.h"
|
||||
#include "zeek/packet_analysis/protocol/tcp/TCPSessionAdapter.h"
|
||||
#include "zeek/session/Manager.h"
|
||||
|
||||
static constexpr auto ZEEK_FUZZ_ANALYZER = "http";
|
||||
|
||||
static zeek::Connection* add_connection()
|
||||
{
|
||||
static constexpr double network_time_start = 1439471031;
|
||||
zeek::run_state::detail::update_network_time(network_time_start);
|
||||
|
||||
zeek::Packet p;
|
||||
zeek::ConnTuple conn_id;
|
||||
conn_id.src_addr = zeek::IPAddr("1.2.3.4");
|
||||
conn_id.dst_addr = zeek::IPAddr("5.6.7.8");
|
||||
conn_id.src_port = htons(23132);
|
||||
conn_id.dst_port = htons(80);
|
||||
conn_id.is_one_way = false;
|
||||
conn_id.proto = TRANSPORT_TCP;
|
||||
zeek::detail::ConnKey key(conn_id);
|
||||
zeek::Connection* conn = new zeek::Connection(key, network_time_start, &conn_id, 1, &p);
|
||||
conn->SetTransport(TRANSPORT_TCP);
|
||||
zeek::session_mgr->Insert(conn);
|
||||
return conn;
|
||||
}
|
||||
|
||||
static zeek::analyzer::Analyzer* add_analyzer(zeek::Connection* conn)
|
||||
{
|
||||
auto* tcp = new zeek::packet_analysis::TCP::TCPSessionAdapter(conn);
|
||||
auto* pia = new zeek::analyzer::pia::PIA_TCP(conn);
|
||||
auto a = zeek::analyzer_mgr->InstantiateAnalyzer(ZEEK_FUZZ_ANALYZER, conn);
|
||||
tcp->AddChildAnalyzer(a);
|
||||
tcp->AddChildAnalyzer(pia->AsAnalyzer());
|
||||
conn->SetSessionAdapter(tcp, pia);
|
||||
return a;
|
||||
}
|
||||
|
||||
extern "C" int LLVMFuzzerTestOneInput(const uint8_t* data, size_t size)
|
||||
{
|
||||
zeek::detail::FuzzBuffer fb{data, size};
|
||||
|
||||
if ( ! fb.Valid() )
|
||||
return 0;
|
||||
|
||||
auto conn = add_connection();
|
||||
auto a = add_analyzer(conn);
|
||||
|
||||
for ( ;; )
|
||||
{
|
||||
auto chunk = fb.Next();
|
||||
|
||||
if ( ! chunk )
|
||||
break;
|
||||
|
||||
try
|
||||
{
|
||||
a->DeliverStream(chunk->size, chunk->data.get(), chunk->is_orig);
|
||||
}
|
||||
catch ( const binpac::Exception& e )
|
||||
{
|
||||
}
|
||||
|
||||
chunk = {};
|
||||
zeek::event_mgr.Drain();
|
||||
}
|
||||
|
||||
zeek::detail::fuzzer_cleanup_one_input();
|
||||
return 0;
|
||||
}
|
||||
BIN
src/fuzzers/imap-corpus.zip
Normal file
BIN
src/fuzzers/imap-corpus.zip
Normal file
Binary file not shown.
78
src/fuzzers/imap-fuzzer.cc
Normal file
78
src/fuzzers/imap-fuzzer.cc
Normal file
|
|
@ -0,0 +1,78 @@
|
|||
#include <binpac.h>
|
||||
|
||||
#include "zeek/Conn.h"
|
||||
#include "zeek/RunState.h"
|
||||
#include "zeek/analyzer/Analyzer.h"
|
||||
#include "zeek/analyzer/Manager.h"
|
||||
#include "zeek/analyzer/protocol/pia/PIA.h"
|
||||
#include "zeek/analyzer/protocol/tcp/TCP.h"
|
||||
#include "zeek/fuzzers/FuzzBuffer.h"
|
||||
#include "zeek/fuzzers/fuzzer-setup.h"
|
||||
#include "zeek/packet_analysis/protocol/tcp/TCPSessionAdapter.h"
|
||||
#include "zeek/session/Manager.h"
|
||||
|
||||
static constexpr auto ZEEK_FUZZ_ANALYZER = "imap";
|
||||
|
||||
static zeek::Connection* add_connection()
|
||||
{
|
||||
static constexpr double network_time_start = 1439471031;
|
||||
zeek::run_state::detail::update_network_time(network_time_start);
|
||||
|
||||
zeek::Packet p;
|
||||
zeek::ConnTuple conn_id;
|
||||
conn_id.src_addr = zeek::IPAddr("1.2.3.4");
|
||||
conn_id.dst_addr = zeek::IPAddr("5.6.7.8");
|
||||
conn_id.src_port = htons(23132);
|
||||
conn_id.dst_port = htons(80);
|
||||
conn_id.is_one_way = false;
|
||||
conn_id.proto = TRANSPORT_TCP;
|
||||
zeek::detail::ConnKey key(conn_id);
|
||||
zeek::Connection* conn = new zeek::Connection(key, network_time_start, &conn_id, 1, &p);
|
||||
conn->SetTransport(TRANSPORT_TCP);
|
||||
zeek::session_mgr->Insert(conn);
|
||||
return conn;
|
||||
}
|
||||
|
||||
static zeek::analyzer::Analyzer* add_analyzer(zeek::Connection* conn)
|
||||
{
|
||||
auto* tcp = new zeek::packet_analysis::TCP::TCPSessionAdapter(conn);
|
||||
auto* pia = new zeek::analyzer::pia::PIA_TCP(conn);
|
||||
auto a = zeek::analyzer_mgr->InstantiateAnalyzer(ZEEK_FUZZ_ANALYZER, conn);
|
||||
tcp->AddChildAnalyzer(a);
|
||||
tcp->AddChildAnalyzer(pia->AsAnalyzer());
|
||||
conn->SetSessionAdapter(tcp, pia);
|
||||
return a;
|
||||
}
|
||||
|
||||
extern "C" int LLVMFuzzerTestOneInput(const uint8_t* data, size_t size)
|
||||
{
|
||||
zeek::detail::FuzzBuffer fb{data, size};
|
||||
|
||||
if ( ! fb.Valid() )
|
||||
return 0;
|
||||
|
||||
auto conn = add_connection();
|
||||
auto a = add_analyzer(conn);
|
||||
|
||||
for ( ;; )
|
||||
{
|
||||
auto chunk = fb.Next();
|
||||
|
||||
if ( ! chunk )
|
||||
break;
|
||||
|
||||
try
|
||||
{
|
||||
a->DeliverStream(chunk->size, chunk->data.get(), chunk->is_orig);
|
||||
}
|
||||
catch ( const binpac::Exception& e )
|
||||
{
|
||||
}
|
||||
|
||||
chunk = {};
|
||||
zeek::event_mgr.Drain();
|
||||
}
|
||||
|
||||
zeek::detail::fuzzer_cleanup_one_input();
|
||||
return 0;
|
||||
}
|
||||
Binary file not shown.
BIN
src/fuzzers/smtp-corpus.zip
Normal file
BIN
src/fuzzers/smtp-corpus.zip
Normal file
Binary file not shown.
78
src/fuzzers/smtp-fuzzer.cc
Normal file
78
src/fuzzers/smtp-fuzzer.cc
Normal file
|
|
@ -0,0 +1,78 @@
|
|||
#include <binpac.h>
|
||||
|
||||
#include "zeek/Conn.h"
|
||||
#include "zeek/RunState.h"
|
||||
#include "zeek/analyzer/Analyzer.h"
|
||||
#include "zeek/analyzer/Manager.h"
|
||||
#include "zeek/analyzer/protocol/pia/PIA.h"
|
||||
#include "zeek/analyzer/protocol/tcp/TCP.h"
|
||||
#include "zeek/fuzzers/FuzzBuffer.h"
|
||||
#include "zeek/fuzzers/fuzzer-setup.h"
|
||||
#include "zeek/packet_analysis/protocol/tcp/TCPSessionAdapter.h"
|
||||
#include "zeek/session/Manager.h"
|
||||
|
||||
static constexpr auto ZEEK_FUZZ_ANALYZER = "smtp";
|
||||
|
||||
static zeek::Connection* add_connection()
|
||||
{
|
||||
static constexpr double network_time_start = 1439471031;
|
||||
zeek::run_state::detail::update_network_time(network_time_start);
|
||||
|
||||
zeek::Packet p;
|
||||
zeek::ConnTuple conn_id;
|
||||
conn_id.src_addr = zeek::IPAddr("1.2.3.4");
|
||||
conn_id.dst_addr = zeek::IPAddr("5.6.7.8");
|
||||
conn_id.src_port = htons(23132);
|
||||
conn_id.dst_port = htons(80);
|
||||
conn_id.is_one_way = false;
|
||||
conn_id.proto = TRANSPORT_TCP;
|
||||
zeek::detail::ConnKey key(conn_id);
|
||||
zeek::Connection* conn = new zeek::Connection(key, network_time_start, &conn_id, 1, &p);
|
||||
conn->SetTransport(TRANSPORT_TCP);
|
||||
zeek::session_mgr->Insert(conn);
|
||||
return conn;
|
||||
}
|
||||
|
||||
static zeek::analyzer::Analyzer* add_analyzer(zeek::Connection* conn)
|
||||
{
|
||||
auto* tcp = new zeek::packet_analysis::TCP::TCPSessionAdapter(conn);
|
||||
auto* pia = new zeek::analyzer::pia::PIA_TCP(conn);
|
||||
auto a = zeek::analyzer_mgr->InstantiateAnalyzer(ZEEK_FUZZ_ANALYZER, conn);
|
||||
tcp->AddChildAnalyzer(a);
|
||||
tcp->AddChildAnalyzer(pia->AsAnalyzer());
|
||||
conn->SetSessionAdapter(tcp, pia);
|
||||
return a;
|
||||
}
|
||||
|
||||
extern "C" int LLVMFuzzerTestOneInput(const uint8_t* data, size_t size)
|
||||
{
|
||||
zeek::detail::FuzzBuffer fb{data, size};
|
||||
|
||||
if ( ! fb.Valid() )
|
||||
return 0;
|
||||
|
||||
auto conn = add_connection();
|
||||
auto a = add_analyzer(conn);
|
||||
|
||||
for ( ;; )
|
||||
{
|
||||
auto chunk = fb.Next();
|
||||
|
||||
if ( ! chunk )
|
||||
break;
|
||||
|
||||
try
|
||||
{
|
||||
a->DeliverStream(chunk->size, chunk->data.get(), chunk->is_orig);
|
||||
}
|
||||
catch ( const binpac::Exception& e )
|
||||
{
|
||||
}
|
||||
|
||||
chunk = {};
|
||||
zeek::event_mgr.Drain();
|
||||
}
|
||||
|
||||
zeek::detail::fuzzer_cleanup_one_input();
|
||||
return 0;
|
||||
}
|
||||
|
|
@ -184,7 +184,7 @@ bool Config::DoUpdate()
|
|||
std::regex re;
|
||||
try
|
||||
{
|
||||
std::string re_str = util::fmt(
|
||||
std::string re_str = Fmt(
|
||||
"^([^[:blank:]]+)[[:blank:]]+([^[:blank:]](.*[^[:blank:]%c])?)?[[:blank:]%c]*$",
|
||||
set_separator[0], set_separator[0]);
|
||||
re.assign(re_str, std::regex::extended);
|
||||
|
|
|
|||
|
|
@ -10,6 +10,11 @@ module Option;
|
|||
static bool call_option_handlers_and_set_value(zeek::StringVal* name, const zeek::detail::IDPtr& i,
|
||||
zeek::ValPtr val, zeek::StringVal* location)
|
||||
{
|
||||
// when shutting down, don't call back into script land handlers as
|
||||
// these may use resources that have already been shutdown.
|
||||
if ( zeek::run_state::terminating )
|
||||
return false;
|
||||
|
||||
if ( i->HasOptionHandlers() )
|
||||
{
|
||||
for ( auto handler_function : i->GetOptionHandlers() )
|
||||
|
|
|
|||
|
|
@ -145,7 +145,7 @@ TeredoAnalyzer::TeredoAnalyzer() : zeek::packet_analysis::Analyzer("TEREDO")
|
|||
// }
|
||||
|
||||
pattern_re = std::make_unique<zeek::detail::Specific_RE_Matcher>(zeek::detail::MATCH_EXACTLY,
|
||||
1);
|
||||
true);
|
||||
pattern_re->AddPat("^(\\x00\\x00)|(\\x00\\x01)|([\\x60-\\x6f].{7}((\\x20\\x01\\x00\\x00)).{28})"
|
||||
"|([\\x60-\\x6f].{23}((\\x20\\x01\\x00\\x00))).{12}");
|
||||
pattern_re->Compile();
|
||||
|
|
|
|||
82
src/parse.y
82
src/parse.y
|
|
@ -54,7 +54,7 @@
|
|||
%left '$' '[' ']' '(' ')' TOK_HAS_FIELD TOK_HAS_ATTR
|
||||
%nonassoc TOK_AS TOK_IS
|
||||
|
||||
%type <b> opt_no_test opt_no_test_block TOK_PATTERN_END opt_deep when_flavor
|
||||
%type <b> opt_no_test opt_no_test_block opt_deep when_flavor
|
||||
%type <str> TOK_ID TOK_PATTERN_TEXT
|
||||
%type <id> local_id global_id def_global_id event_id global_or_event_id resolve_id begin_lambda case_type
|
||||
%type <id_l> local_id_list case_type_list
|
||||
|
|
@ -77,6 +77,7 @@
|
|||
%type <capture> capture
|
||||
%type <captures> capture_list opt_captures when_captures
|
||||
%type <when_clause> when_head when_start when_clause
|
||||
%type <re_modes> TOK_PATTERN_END
|
||||
|
||||
%{
|
||||
#include <cstdlib>
|
||||
|
|
@ -155,6 +156,8 @@ static int func_hdr_cond_epoch = 0;
|
|||
EnumType* cur_enum_type = nullptr;
|
||||
static ID* cur_decl_type_id = nullptr;
|
||||
|
||||
std::set<std::string> module_names;
|
||||
|
||||
static void parse_new_enum(void)
|
||||
{
|
||||
// Starting a new enum definition.
|
||||
|
|
@ -186,6 +189,56 @@ static void parse_redef_enum(ID* id)
|
|||
}
|
||||
}
|
||||
|
||||
static void parse_redef_record_field(ID* id, const char* field, InitClass ic,
|
||||
std::unique_ptr<std::vector<AttrPtr>> attrs)
|
||||
{
|
||||
if ( ! id->GetType() )
|
||||
{
|
||||
reporter->FatalError("unknown record identifier \"%s\"", id->Name());
|
||||
return;
|
||||
}
|
||||
|
||||
auto t = id->GetType();
|
||||
if ( ! t || t->Tag() != TYPE_RECORD )
|
||||
{
|
||||
reporter->FatalError("identifier \"%s\" has type \"%s\", expected \"record\"",
|
||||
id->Name(), type_name(t->Tag()));
|
||||
return;
|
||||
}
|
||||
|
||||
auto rt = t->AsRecordType();
|
||||
auto idx = rt->FieldOffset(field);
|
||||
if ( idx < 0 )
|
||||
{
|
||||
reporter->FatalError("field \"%s\" not in record \"%s\"", field, id->Name());
|
||||
return;
|
||||
}
|
||||
|
||||
auto decl = rt->FieldDecl(idx);
|
||||
if ( ! decl->attrs )
|
||||
if ( ic == INIT_EXTRA )
|
||||
decl->attrs = make_intrusive<detail::Attributes>(decl->type,
|
||||
true /* in_record */,
|
||||
false /* is_global */);
|
||||
|
||||
for ( const auto& attr : *attrs )
|
||||
{
|
||||
// At this point, only support &log redef'ing.
|
||||
if ( attr->Tag() != ATTR_LOG )
|
||||
{
|
||||
reporter->FatalError("Can only redef \"&log\" attributes of record fields");
|
||||
return;
|
||||
}
|
||||
|
||||
if ( ic == INIT_EXTRA )
|
||||
decl->attrs->AddAttr(attr, true /* is_redef */);
|
||||
else
|
||||
// Removing attributes is a noop if they don't exist.
|
||||
if ( decl->attrs )
|
||||
decl->attrs->RemoveAttr(attr->Tag());
|
||||
}
|
||||
}
|
||||
|
||||
static void extend_record(ID* id, std::unique_ptr<type_decl_list> fields,
|
||||
std::unique_ptr<std::vector<AttrPtr>> attrs)
|
||||
{
|
||||
|
|
@ -324,6 +377,11 @@ static StmtPtr build_local(ID* id, Type* t, InitClass ic, Expr* e,
|
|||
zeek::FuncType::Capture* capture;
|
||||
zeek::FuncType::CaptureList* captures;
|
||||
zeek::detail::WhenInfo* when_clause;
|
||||
struct
|
||||
{
|
||||
bool ignore_case;
|
||||
bool single_line;
|
||||
} re_modes;
|
||||
}
|
||||
|
||||
%%
|
||||
|
|
@ -912,9 +970,12 @@ expr:
|
|||
auto* re = new RE_Matcher($3);
|
||||
delete [] $3;
|
||||
|
||||
if ( $4 )
|
||||
if ( $4.ignore_case )
|
||||
re->MakeCaseInsensitive();
|
||||
|
||||
if ( $4.single_line )
|
||||
re->MakeSingleLine();
|
||||
|
||||
re->Compile();
|
||||
$$ = new ConstExpr(make_intrusive<PatternVal>(re));
|
||||
}
|
||||
|
|
@ -1260,6 +1321,7 @@ decl:
|
|||
TOK_MODULE TOK_ID ';'
|
||||
{
|
||||
current_module = $2;
|
||||
module_names.insert($2);
|
||||
zeekygen_mgr->ModuleUsage(::filename, current_module);
|
||||
}
|
||||
|
||||
|
|
@ -1294,6 +1356,20 @@ decl:
|
|||
// Zeekygen already grabbed new enum IDs as the type created them.
|
||||
}
|
||||
|
||||
| TOK_REDEF TOK_RECORD global_id '$' TOK_ID
|
||||
{ cur_decl_type_id = $3; zeekygen_mgr->Redef($3, ::filename, INIT_EXTRA); }
|
||||
TOK_ADD_TO '{' attr_list '}' ';'
|
||||
{
|
||||
cur_decl_type_id = 0;
|
||||
parse_redef_record_field($3, $5, INIT_EXTRA, std::unique_ptr<std::vector<AttrPtr>>($9));
|
||||
}
|
||||
| TOK_REDEF TOK_RECORD global_id '$' TOK_ID
|
||||
{ cur_decl_type_id = $3; zeekygen_mgr->Redef($3, ::filename, INIT_REMOVE); }
|
||||
TOK_REMOVE_FROM '{' attr_list '}' ';'
|
||||
{
|
||||
cur_decl_type_id = 0;
|
||||
parse_redef_record_field($3, $5, INIT_REMOVE, std::unique_ptr<std::vector<AttrPtr>>($9));
|
||||
}
|
||||
| TOK_REDEF TOK_RECORD global_id
|
||||
{ cur_decl_type_id = $3; zeekygen_mgr->Redef($3, ::filename); }
|
||||
TOK_ADD_TO '{'
|
||||
|
|
@ -1311,7 +1387,7 @@ decl:
|
|||
std::unique_ptr<std::vector<AttrPtr>>($11));
|
||||
}
|
||||
|
||||
| TOK_TYPE global_id ':'
|
||||
| TOK_TYPE def_global_id ':'
|
||||
{ cur_decl_type_id = $2; zeekygen_mgr->StartType({NewRef{}, $2}); }
|
||||
type opt_attr ';'
|
||||
{
|
||||
|
|
|
|||
|
|
@ -20,7 +20,7 @@ namespace zeek::detail {
|
|||
void yyerror(const char msg[]);
|
||||
%}
|
||||
|
||||
%token TOK_CHAR TOK_NUMBER TOK_CCL TOK_CCE TOK_CASE_INSENSITIVE
|
||||
%token TOK_CHAR TOK_NUMBER TOK_CCL TOK_CCE TOK_CASE_INSENSITIVE TOK_SINGLE_LINE
|
||||
|
||||
%union {
|
||||
int int_val;
|
||||
|
|
@ -112,7 +112,8 @@ singleton : singleton '*'
|
|||
|
||||
| '.'
|
||||
{
|
||||
$$ = new zeek::detail::NFA_Machine(new zeek::detail::NFA_State(zeek::detail::rem->AnyCCL()));
|
||||
$$ = new zeek::detail::NFA_Machine(new zeek::detail::NFA_State(
|
||||
zeek::detail::rem->AnyCCL(zeek::detail::re_single_line)));
|
||||
}
|
||||
|
||||
| full_ccl
|
||||
|
|
@ -132,7 +133,10 @@ singleton : singleton '*'
|
|||
{ $$ = $2; }
|
||||
|
||||
| TOK_CASE_INSENSITIVE re ')'
|
||||
{ $$ = $2; zeek::detail::case_insensitive = 0; }
|
||||
{ $$ = $2; zeek::detail::case_insensitive = false; }
|
||||
|
||||
| TOK_SINGLE_LINE re ')'
|
||||
{ $$ = $2; zeek::detail::re_single_line = false; }
|
||||
|
||||
| TOK_CHAR
|
||||
{
|
||||
|
|
|
|||
|
|
@ -115,7 +115,8 @@ CCL_EXPR ("[:"[[:alpha:]]+":]")
|
|||
}
|
||||
}
|
||||
|
||||
"(?i:" zeek::detail::case_insensitive = 1; return TOK_CASE_INSENSITIVE;
|
||||
"(?i:" zeek::detail::case_insensitive = true; return TOK_CASE_INSENSITIVE;
|
||||
"(?s:" zeek::detail::re_single_line = true; return TOK_SINGLE_LINE;
|
||||
|
||||
[a-zA-Z] {
|
||||
if ( zeek::detail::case_insensitive )
|
||||
|
|
|
|||
37
src/scan.l
37
src/scan.l
|
|
@ -80,6 +80,8 @@ const char* last_tok_filename = 0;
|
|||
const char* last_last_tok_filename = 0;
|
||||
char last_tok[128];
|
||||
|
||||
extern std::set<std::string> module_names;
|
||||
|
||||
#define YY_USER_ACTION strncpy(last_tok, yytext, sizeof(last_tok) - 1); \
|
||||
last_last_tok_filename = last_tok_filename; \
|
||||
last_tok_filename = ::filename;
|
||||
|
|
@ -562,13 +564,28 @@ F RET_CONST(zeek::val_mgr->False()->Ref())
|
|||
|
||||
<RE>"/" {
|
||||
BEGIN(INITIAL);
|
||||
yylval.b = false;
|
||||
yylval.re_modes.ignore_case = false;
|
||||
yylval.re_modes.single_line = false;
|
||||
return TOK_PATTERN_END;
|
||||
}
|
||||
|
||||
<RE>"/i" {
|
||||
<RE>(\/[is]{0,2}) {
|
||||
BEGIN(INITIAL);
|
||||
yylval.b = true;
|
||||
|
||||
if ( strlen(yytext) == 2 )
|
||||
{
|
||||
yylval.re_modes.ignore_case = (yytext[1] == 'i');
|
||||
yylval.re_modes.single_line = (yytext[1] == 's');
|
||||
}
|
||||
else
|
||||
{
|
||||
if ( yytext[1] == yytext[2] )
|
||||
zeek::reporter->Error("pattern has duplicate mode %c", yytext[1]);
|
||||
|
||||
yylval.re_modes.ignore_case = (yytext[1] == 'i' || yytext[2] == 'i');
|
||||
yylval.re_modes.single_line = (yytext[1] == 's' || yytext[2] == 's');
|
||||
}
|
||||
|
||||
return TOK_PATTERN_END;
|
||||
}
|
||||
|
||||
|
|
@ -781,7 +798,12 @@ void do_atifdef(const char* id)
|
|||
|
||||
const auto& i = zeek::detail::lookup_ID(id, zeek::detail::current_module.c_str());
|
||||
|
||||
if ( ! i )
|
||||
// If the ID didn't exist, look to see if this is a module name instead.
|
||||
bool found = (i != nullptr);
|
||||
if ( ! found )
|
||||
found = (module_names.count(id) != 0);
|
||||
|
||||
if ( ! found )
|
||||
begin_ignoring();
|
||||
}
|
||||
|
||||
|
|
@ -791,7 +813,12 @@ void do_atifndef(const char *id)
|
|||
|
||||
const auto& i = zeek::detail::lookup_ID(id, zeek::detail::current_module.c_str());
|
||||
|
||||
if ( i )
|
||||
// If the ID didn't exist, look to see if this is a module name instead.
|
||||
bool found = (i != nullptr);
|
||||
if ( ! found )
|
||||
found = (module_names.count(id) != 0);
|
||||
|
||||
if ( found )
|
||||
begin_ignoring();
|
||||
}
|
||||
|
||||
|
|
|
|||
|
|
@ -170,6 +170,8 @@ VEC_OP2(or, |, )
|
|||
VEC_OP2(xor, ^, )
|
||||
VEC_OP2(andand, &&, )
|
||||
VEC_OP2(oror, ||, )
|
||||
VEC_OP2(lshift, <<, )
|
||||
VEC_OP2(rshift, >>, )
|
||||
|
||||
// A version of VEC_OP2 that instead supports relational operations, so
|
||||
// the result type is always vector-of-bool.
|
||||
|
|
|
|||
|
|
@ -45,6 +45,8 @@ extern VectorValPtr vec_op_or__CPP(const VectorValPtr& v1, const VectorValPtr& v
|
|||
extern VectorValPtr vec_op_xor__CPP(const VectorValPtr& v1, const VectorValPtr& v2);
|
||||
extern VectorValPtr vec_op_andand__CPP(const VectorValPtr& v1, const VectorValPtr& v2);
|
||||
extern VectorValPtr vec_op_oror__CPP(const VectorValPtr& v1, const VectorValPtr& v2);
|
||||
extern VectorValPtr vec_op_lshift__CPP(const VectorValPtr& v1, const VectorValPtr& v2);
|
||||
extern VectorValPtr vec_op_rshift__CPP(const VectorValPtr& v1, const VectorValPtr& v2);
|
||||
|
||||
// Vector relational operations.
|
||||
extern VectorValPtr vec_op_lt__CPP(const VectorValPtr& v1, const VectorValPtr& v2);
|
||||
|
|
|
|||
|
|
@ -495,7 +495,7 @@ void CPPCompile::GenForOverTable(const ExprPtr& tbl, const IDPtr& value_var,
|
|||
StartBlock();
|
||||
|
||||
Emit("auto k__CPP = lve__CPP.GetHashKey();");
|
||||
Emit("auto* current_tev__CPP = lve__CPP.GetValue<TableEntryVal*>();");
|
||||
Emit("auto* current_tev__CPP = lve__CPP.value;");
|
||||
Emit("auto ind_lv__CPP = tv__CPP->RecreateIndex(*k__CPP);");
|
||||
|
||||
if ( value_var )
|
||||
|
|
|
|||
|
|
@ -15,7 +15,7 @@ The maintenance workflow:
|
|||
to check in updates to the list of how the compiler currently fares
|
||||
on various btests (see end of this doc):
|
||||
|
||||
Thu May 12 12:54:10 PDT 2022
|
||||
Mon Aug 1 16:39:05 PDT 2022
|
||||
|
||||
2. Run "find-test-files.sh" to generate a list (to stdout) of all of the
|
||||
possible Zeek source files found in the test suite.
|
||||
|
|
@ -57,17 +57,27 @@ These BTests won't successfully run due to the indicated issue:
|
|||
command-line-error - a deliberate command-line error
|
||||
complex-to-debug - hard-to-figure-out failure
|
||||
deprecated - uses features deprecated for -O C++
|
||||
error-handling - behavior in face of an error differs
|
||||
needs-plugin - requires knowing how to build an associated plugin
|
||||
no-script - there's no actual script to compile
|
||||
ZAM - meant specifically for -O ZAM
|
||||
|
||||
Consider migrating these to have @TEST-REQUIRES clauses so we don't have
|
||||
to maintain this list.
|
||||
|
||||
../testing/btest/core/negative-time.test no-script
|
||||
../testing/btest/core/pcap/dumper.zeek no-script
|
||||
../testing/btest/core/pcap/input-error.zeek command-line-error
|
||||
../testing/btest/core/proc-status-file.zeek no-script
|
||||
../testing/btest/core/scalar-vector.zeek deprecated
|
||||
../testing/btest/language/at-if-event.zeek @if
|
||||
../testing/btest/language/at-if.zeek @if
|
||||
../testing/btest/language/at-ifdef.zeek @if
|
||||
../testing/btest/language/at-ifndef.zeek @if
|
||||
../testing/btest/language/incr-vec-expr.test deprecated
|
||||
../testing/btest/language/uninitialized-local2.zeek error-handling
|
||||
../testing/btest/language/vector-deprecated.zeek deprecated
|
||||
../testing/btest/language/vector-in-operator.zeek
|
||||
../testing/btest/language/vector-in-operator.zeek deprecated
|
||||
../testing/btest/language/when-aggregates.zeek bad-when
|
||||
../testing/btest/opt/opt-files.zeek ZAM
|
||||
|
|
@ -76,6 +86,7 @@ These BTests won't successfully run due to the indicated issue:
|
|||
../testing/btest/opt/opt-func.zeek ZAM
|
||||
../testing/btest/opt/opt-func2.zeek ZAM
|
||||
../testing/btest/opt/opt-func3.zeek ZAM
|
||||
../testing/btest/plugins/packet-protocol.zeek needs-plugin
|
||||
../testing/btest/scripts/base/protocols/dhcp/dhcp-ack-msg-types.zeek no-script
|
||||
../testing/btest/scripts/base/protocols/dhcp/dhcp-all-msg-types.zeek no-script
|
||||
../testing/btest/scripts/base/protocols/dhcp/dhcp-discover-msg-types.zeek no-script
|
||||
|
|
|
|||
17
src/script_opt/CPP/maint/check-CPP-gen.sh
Normal file → Executable file
17
src/script_opt/CPP/maint/check-CPP-gen.sh
Normal file → Executable file
|
|
@ -1,8 +1,19 @@
|
|||
#! /bin/sh
|
||||
|
||||
out=out.$(echo $1 | sed 's,\.\./,,;s,/,#,g')
|
||||
abbr=$(echo $1 | sed 's,\.\./,,;s,/,#,g')
|
||||
out=CPP-test/out.$abbr
|
||||
gen_out=CPP-test/gen.$abbr
|
||||
|
||||
(
|
||||
/bin/echo -n $1" "
|
||||
(src/zeek -O gen-C++ --optimize-files=testing/btest --optimize-func="<global-stmts>" $1 >&/dev/null && echo "success") || echo "fail"
|
||||
) >CPP-test/$out 2>&1
|
||||
if ! src/zeek -O gen-C++ --optimize-files=testing/btest $1 >&$gen_out 2>&1; then
|
||||
echo "fail"
|
||||
exit 1
|
||||
fi
|
||||
if grep -E -q 'deprecated|skipping|cannot compile|no matching functions' $gen_out; then
|
||||
echo "fail"
|
||||
exit 1
|
||||
fi
|
||||
echo "success"
|
||||
exit 0
|
||||
) >$out 2>&1
|
||||
|
|
|
|||
0
src/script_opt/CPP/maint/check-zeek.sh
Normal file → Executable file
0
src/script_opt/CPP/maint/check-zeek.sh
Normal file → Executable file
13
src/script_opt/CPP/maint/do-CPP-btest.sh
Normal file → Executable file
13
src/script_opt/CPP/maint/do-CPP-btest.sh
Normal file → Executable file
|
|
@ -1,11 +1,22 @@
|
|||
#! /bin/sh
|
||||
|
||||
rm -f CPP-gen.cc
|
||||
|
||||
cp zeek.HOLD src/zeek || (
|
||||
echo Need to create clean zeek.HOLD
|
||||
exit 1
|
||||
) || exit 1
|
||||
|
||||
if [ "$1" == "-U" ]; then
|
||||
btest_opt=-U
|
||||
shift
|
||||
elif [ "$1" == "-d" ]; then
|
||||
btest_opt=-d
|
||||
shift
|
||||
else
|
||||
btest_opt=-d
|
||||
fi
|
||||
|
||||
base=$(echo $1 | sed 's,\.\./,,;s,/,#,g')
|
||||
rel_test=$(echo $1 | sed 's,.*testing/btest/,,')
|
||||
|
||||
|
|
@ -26,5 +37,5 @@ ninja
|
|||
|
||||
(
|
||||
cd ../testing/btest
|
||||
../../auxil/btest/btest -a cpp -d -f ../../build/CPP-test/diag.$base $rel_test
|
||||
../../auxil/btest/btest -a cpp $btest_opt -f ../../build/CPP-test/diag.$base $rel_test
|
||||
)
|
||||
|
|
|
|||
0
src/script_opt/CPP/maint/find-test-files.sh
Normal file → Executable file
0
src/script_opt/CPP/maint/find-test-files.sh
Normal file → Executable file
|
|
@ -543,6 +543,19 @@ op-type U
|
|||
vector
|
||||
eval $1 ^ $2
|
||||
|
||||
binary-expr-op Lshift
|
||||
op-type I U
|
||||
vector
|
||||
eval-type I if ( $1 < 0 )
|
||||
ZAM_run_time_error(z.loc, "left shifting a negative number is undefined");
|
||||
$$ = $1 << $2;
|
||||
eval $1 << $2
|
||||
|
||||
binary-expr-op Rshift
|
||||
op-type I U
|
||||
vector
|
||||
eval $1 >> $2
|
||||
|
||||
########## Relationals ##########
|
||||
|
||||
rel-expr-op LT
|
||||
|
|
@ -833,13 +846,13 @@ internal-op Val-Is-In-Vector
|
|||
type VVV
|
||||
eval auto& vec = frame[z.v3].vector_val;
|
||||
auto ind = frame[z.v2].int_val;
|
||||
frame[z.v1].int_val = ind >= 0 && static_cast<zeek_uint_t>(ind) < vec->Size();
|
||||
frame[z.v1].int_val = vec->Has(ind);
|
||||
|
||||
internal-op Const-Is-In-Vector
|
||||
type VCV
|
||||
eval auto& vec = frame[z.v2].vector_val;
|
||||
auto ind = z.c.int_val;
|
||||
frame[z.v1].int_val = ind >= 0 && static_cast<zeek_uint_t>(ind) < vec->Size();
|
||||
frame[z.v1].int_val = vec->Has(ind);
|
||||
|
||||
expr-op Cond
|
||||
type VVVV
|
||||
|
|
|
|||
|
|
@ -12,4 +12,4 @@ set(telemetry_SRCS
|
|||
bif_target(telemetry.bif)
|
||||
|
||||
bro_add_subdir_library(telemetry ${telemetry_SRCS})
|
||||
|
||||
add_dependencies(bro_telemetry generate_outputs)
|
||||
|
|
|
|||
|
|
@ -2,11 +2,15 @@
|
|||
|
||||
#include "zeek/telemetry/Manager.h"
|
||||
|
||||
#include <fnmatch.h>
|
||||
#include <thread>
|
||||
#include <variant>
|
||||
|
||||
#include "zeek/3rdparty/doctest.h"
|
||||
#include "zeek/ID.h"
|
||||
#include "zeek/broker/Manager.h"
|
||||
#include "zeek/telemetry/Timer.h"
|
||||
#include "zeek/telemetry/telemetry.bif.h"
|
||||
|
||||
#include "broker/telemetry/metric_registry.hh"
|
||||
|
||||
|
|
@ -15,6 +19,33 @@ namespace
|
|||
using NativeManager = broker::telemetry::metric_registry;
|
||||
using NativeManagerImpl = broker::telemetry::metric_registry_impl;
|
||||
using NativeManagerImplPtr = zeek::IntrusivePtr<NativeManagerImpl>;
|
||||
using DoubleValPtr = zeek::IntrusivePtr<zeek::DoubleVal>;
|
||||
|
||||
std::vector<std::string_view> extract_label_values(broker::telemetry::const_label_list labels)
|
||||
{
|
||||
auto get_value = [](const auto& label)
|
||||
{
|
||||
return label.second;
|
||||
};
|
||||
std::vector<std::string_view> v;
|
||||
std::transform(labels.begin(), labels.end(), std::back_inserter(v), get_value);
|
||||
return v;
|
||||
}
|
||||
|
||||
// Convert an int64_t or double to a DoubleValPtr. int64_t is casted.
|
||||
template <typename T> DoubleValPtr as_double_val(T val)
|
||||
{
|
||||
if constexpr ( std::is_same_v<T, int64_t> )
|
||||
{
|
||||
return zeek::make_intrusive<zeek::DoubleVal>(static_cast<double>(val));
|
||||
}
|
||||
else
|
||||
{
|
||||
static_assert(std::is_same_v<T, double>);
|
||||
return zeek::make_intrusive<zeek::DoubleVal>(val);
|
||||
}
|
||||
};
|
||||
|
||||
}
|
||||
|
||||
namespace zeek::telemetry
|
||||
|
|
@ -38,6 +69,430 @@ void Manager::InitPostBrokerSetup(broker::endpoint& ep)
|
|||
pimpl.swap(ptr);
|
||||
}
|
||||
|
||||
// -- collect metric stuff -----------------------------------------------------
|
||||
|
||||
template <typename T>
|
||||
zeek::RecordValPtr Manager::GetMetricOptsRecord(Manager::MetricType metric_type,
|
||||
const broker::telemetry::metric_family_hdl* family)
|
||||
{
|
||||
static auto string_vec_type = zeek::id::find_type<zeek::VectorType>("string_vec");
|
||||
static auto double_vec_type = zeek::id::find_type<zeek::VectorType>("double_vec");
|
||||
static auto count_vec_type = zeek::id::find_type<zeek::VectorType>("index_vec");
|
||||
static auto metric_opts_type = zeek::id::find_type<zeek::RecordType>("Telemetry::MetricOpts");
|
||||
|
||||
static auto prefix_idx = metric_opts_type->FieldOffset("prefix");
|
||||
static auto name_idx = metric_opts_type->FieldOffset("name");
|
||||
static auto help_text_idx = metric_opts_type->FieldOffset("help_text");
|
||||
static auto unit_idx = metric_opts_type->FieldOffset("unit");
|
||||
static auto is_total_idx = metric_opts_type->FieldOffset("is_total");
|
||||
static auto labels_idx = metric_opts_type->FieldOffset("labels");
|
||||
static auto bounds_idx = metric_opts_type->FieldOffset("bounds");
|
||||
static auto metric_type_idx = metric_opts_type->FieldOffset("metric_type");
|
||||
|
||||
if ( const auto& it = metric_opts_cache.find(family); it != metric_opts_cache.end() )
|
||||
return it->second;
|
||||
|
||||
auto r = make_intrusive<zeek::RecordVal>(metric_opts_type);
|
||||
r->Assign(prefix_idx, make_intrusive<zeek::StringVal>(broker::telemetry::prefix(family)));
|
||||
r->Assign(name_idx, make_intrusive<zeek::StringVal>(broker::telemetry::name(family)));
|
||||
r->Assign(help_text_idx, make_intrusive<zeek::StringVal>(broker::telemetry::helptext(family)));
|
||||
r->Assign(unit_idx, make_intrusive<zeek::StringVal>(broker::telemetry::unit(family)));
|
||||
r->Assign(is_total_idx, val_mgr->Bool(broker::telemetry::is_sum(family)));
|
||||
|
||||
auto label_names_vec = make_intrusive<zeek::VectorVal>(string_vec_type);
|
||||
for ( const auto& l : broker::telemetry::label_names(family) )
|
||||
label_names_vec->Append(make_intrusive<StringVal>(l));
|
||||
|
||||
r->Assign(labels_idx, label_names_vec);
|
||||
|
||||
// This is mapping Manager.h enums to bif values depending on
|
||||
// the template type and whether this is a counter, gauge or
|
||||
// histogram.
|
||||
zeek_int_t metric_type_int = -1;
|
||||
if constexpr ( std::is_same_v<T, double> )
|
||||
{
|
||||
switch ( metric_type )
|
||||
{
|
||||
case MetricType::Counter:
|
||||
metric_type_int = BifEnum::Telemetry::MetricType::DOUBLE_COUNTER;
|
||||
break;
|
||||
case MetricType::Gauge:
|
||||
metric_type_int = BifEnum::Telemetry::MetricType::DOUBLE_GAUGE;
|
||||
break;
|
||||
case MetricType::Histogram:
|
||||
metric_type_int = BifEnum::Telemetry::MetricType::DOUBLE_HISTOGRAM;
|
||||
break;
|
||||
}
|
||||
}
|
||||
else
|
||||
{
|
||||
switch ( metric_type )
|
||||
{
|
||||
case MetricType::Counter:
|
||||
metric_type_int = BifEnum::Telemetry::MetricType::INT_COUNTER;
|
||||
break;
|
||||
case MetricType::Gauge:
|
||||
metric_type_int = BifEnum::Telemetry::MetricType::INT_GAUGE;
|
||||
break;
|
||||
case MetricType::Histogram:
|
||||
metric_type_int = BifEnum::Telemetry::MetricType::INT_HISTOGRAM;
|
||||
break;
|
||||
}
|
||||
}
|
||||
|
||||
if ( metric_type_int < 0 )
|
||||
reporter->FatalError("Unable to lookup metric type %d", int(metric_type));
|
||||
|
||||
r->Assign(metric_type_idx,
|
||||
zeek::BifType::Enum::Telemetry::MetricType->GetEnumVal(metric_type_int));
|
||||
|
||||
// Add bounds and optionally count_bounds into the MetricOpts record.
|
||||
static auto opts_rt = zeek::id::find_type<zeek::RecordType>("Telemetry::MetricOpts");
|
||||
static auto opts_rt_idx_bounds = opts_rt->FieldOffset("bounds");
|
||||
static auto opts_rt_idx_count_bounds = opts_rt->FieldOffset("count_bounds");
|
||||
|
||||
if ( metric_type == MetricType::Histogram )
|
||||
{
|
||||
auto add_double_bounds = [](auto& r, const auto* histogram_family)
|
||||
{
|
||||
size_t buckets = broker::telemetry::num_buckets(histogram_family);
|
||||
auto bounds_vec = make_intrusive<zeek::VectorVal>(double_vec_type);
|
||||
for ( size_t i = 0; i < buckets; i++ )
|
||||
bounds_vec->Append(
|
||||
as_double_val(broker::telemetry::upper_bound_at(histogram_family, i)));
|
||||
|
||||
r->Assign(opts_rt_idx_bounds, bounds_vec);
|
||||
};
|
||||
|
||||
if constexpr ( std::is_same_v<T, int64_t> )
|
||||
{
|
||||
auto histogram_family = broker::telemetry::as_int_histogram_family(family);
|
||||
add_double_bounds(r, histogram_family);
|
||||
|
||||
// Add count_bounds to int64_t histograms
|
||||
size_t buckets = broker::telemetry::num_buckets(histogram_family);
|
||||
auto count_bounds_vec = make_intrusive<zeek::VectorVal>(count_vec_type);
|
||||
for ( size_t i = 0; i < buckets; i++ )
|
||||
count_bounds_vec->Append(
|
||||
val_mgr->Count(broker::telemetry::upper_bound_at(histogram_family, i)));
|
||||
|
||||
r->Assign(opts_rt_idx_count_bounds, count_bounds_vec);
|
||||
}
|
||||
else
|
||||
{
|
||||
static_assert(std::is_same_v<T, double>);
|
||||
add_double_bounds(r, broker::telemetry::as_dbl_histogram_family(family));
|
||||
}
|
||||
}
|
||||
|
||||
metric_opts_cache.insert({family, r});
|
||||
|
||||
return r;
|
||||
}
|
||||
|
||||
zeek::RecordValPtr Manager::CollectedValueMetric::AsMetricRecord() const
|
||||
{
|
||||
static auto string_vec_type = zeek::id::find_type<zeek::VectorType>("string_vec");
|
||||
static auto metric_record_type = zeek::id::find_type<zeek::RecordType>("Telemetry::Metric");
|
||||
static auto opts_idx = metric_record_type->FieldOffset("opts");
|
||||
static auto labels_idx = metric_record_type->FieldOffset("labels");
|
||||
static auto value_idx = metric_record_type->FieldOffset("value");
|
||||
static auto count_value_idx = metric_record_type->FieldOffset("count_value");
|
||||
|
||||
auto r = make_intrusive<zeek::RecordVal>(metric_record_type);
|
||||
|
||||
auto label_values_vec = make_intrusive<zeek::VectorVal>(string_vec_type);
|
||||
for ( const auto& l : label_values )
|
||||
label_values_vec->Append(make_intrusive<StringVal>(l));
|
||||
|
||||
r->Assign(labels_idx, label_values_vec);
|
||||
|
||||
auto fn = [&](auto val)
|
||||
{
|
||||
using val_t = decltype(val);
|
||||
auto opts_record = telemetry_mgr->GetMetricOptsRecord<val_t>(metric_type, family);
|
||||
r->Assign(opts_idx, opts_record);
|
||||
r->Assign(value_idx, as_double_val(val));
|
||||
if constexpr ( std::is_same_v<val_t, int64_t> )
|
||||
r->Assign(count_value_idx, val_mgr->Count(val));
|
||||
};
|
||||
|
||||
std::visit(fn, value);
|
||||
|
||||
return r;
|
||||
}
|
||||
|
||||
zeek::RecordValPtr Manager::CollectedHistogramMetric::AsHistogramMetricRecord() const
|
||||
{
|
||||
static auto string_vec_type = zeek::id::find_type<zeek::VectorType>("string_vec");
|
||||
static auto double_vec_type = zeek::id::find_type<zeek::VectorType>("double_vec");
|
||||
static auto count_vec_type = zeek::id::find_type<zeek::VectorType>("index_vec");
|
||||
static auto histogram_metric_type = zeek::id::find_type<zeek::RecordType>(
|
||||
"Telemetry::HistogramMetric");
|
||||
static auto opts_idx = histogram_metric_type->FieldOffset("opts");
|
||||
static auto labels_idx = histogram_metric_type->FieldOffset("labels");
|
||||
static auto values_idx = histogram_metric_type->FieldOffset("values");
|
||||
static auto count_values_idx = histogram_metric_type->FieldOffset("count_values");
|
||||
static auto observations_idx = histogram_metric_type->FieldOffset("observations");
|
||||
static auto sum_idx = histogram_metric_type->FieldOffset("sum");
|
||||
static auto count_observations_idx = histogram_metric_type->FieldOffset("count_observations");
|
||||
static auto count_sum_idx = histogram_metric_type->FieldOffset("count_sum");
|
||||
|
||||
auto r = make_intrusive<zeek::RecordVal>(histogram_metric_type);
|
||||
|
||||
auto label_values_vec = make_intrusive<zeek::VectorVal>(string_vec_type);
|
||||
for ( const auto& l : label_values )
|
||||
label_values_vec->Append(make_intrusive<StringVal>(l));
|
||||
|
||||
r->Assign(labels_idx, label_values_vec);
|
||||
|
||||
auto fn = [&](const auto& histogram_data)
|
||||
{
|
||||
using val_t = std::decay_t<decltype(histogram_data.sum)>;
|
||||
auto opts_record = telemetry_mgr->GetMetricOptsRecord<val_t>(MetricType::Histogram, family);
|
||||
r->Assign(opts_idx, opts_record);
|
||||
|
||||
val_t observations = 0;
|
||||
auto values_vec = make_intrusive<zeek::VectorVal>(double_vec_type);
|
||||
auto count_values_vec = make_intrusive<zeek::VectorVal>(count_vec_type);
|
||||
|
||||
for ( const auto& b : histogram_data.buckets )
|
||||
{
|
||||
observations += b.count;
|
||||
values_vec->Append(as_double_val(b.count));
|
||||
if constexpr ( std::is_same_v<val_t, int64_t> )
|
||||
count_values_vec->Append(val_mgr->Count(b.count));
|
||||
}
|
||||
|
||||
r->Assign(values_idx, values_vec);
|
||||
r->Assign(sum_idx, as_double_val(histogram_data.sum));
|
||||
r->Assign(observations_idx, as_double_val(observations));
|
||||
|
||||
// Add extra fields just for int64_t based histograms with type count
|
||||
if constexpr ( std::is_same_v<val_t, int64_t> )
|
||||
{
|
||||
r->Assign(count_values_idx, count_values_vec);
|
||||
r->Assign(count_sum_idx, val_mgr->Count(histogram_data.sum));
|
||||
r->Assign(count_observations_idx, val_mgr->Count(observations));
|
||||
}
|
||||
};
|
||||
|
||||
std::visit(fn, histogram);
|
||||
|
||||
return r;
|
||||
}
|
||||
|
||||
/**
|
||||
* Encapsulate matching of prefix and name against a broker::telemetry::metric_family_hdl
|
||||
*/
|
||||
class MetricFamilyMatcher
|
||||
{
|
||||
public:
|
||||
MetricFamilyMatcher(std::string_view prefix, std::string_view name)
|
||||
: prefix_pattern(prefix), name_pattern(name)
|
||||
{
|
||||
}
|
||||
|
||||
/**
|
||||
* @return true if the given family's prefix and name match, else false;
|
||||
*/
|
||||
bool operator()(const broker::telemetry::metric_family_hdl* family)
|
||||
{
|
||||
auto prefix = std::string{broker::telemetry::prefix(family)};
|
||||
auto name = std::string{broker::telemetry::name(family)};
|
||||
|
||||
return fnmatch(prefix_pattern.c_str(), prefix.c_str(), 0) != FNM_NOMATCH &&
|
||||
fnmatch(name_pattern.c_str(), name.c_str(), 0) != FNM_NOMATCH;
|
||||
}
|
||||
|
||||
private:
|
||||
std::string prefix_pattern;
|
||||
std::string name_pattern;
|
||||
};
|
||||
|
||||
/**
|
||||
* A collector implementation for counters and gauges.
|
||||
*/
|
||||
class MetricsCollector : public broker::telemetry::metrics_collector
|
||||
{
|
||||
using MetricType = Manager::MetricType;
|
||||
|
||||
public:
|
||||
MetricsCollector(std::string_view prefix, std::string_view name) : matches(prefix, name) { }
|
||||
|
||||
void operator()(const broker::telemetry::metric_family_hdl* family,
|
||||
const broker::telemetry::dbl_counter_hdl* counter,
|
||||
broker::telemetry::const_label_list labels)
|
||||
{
|
||||
if ( matches(family) )
|
||||
metrics.emplace_back(MetricType::Counter, family, extract_label_values(labels),
|
||||
broker::telemetry::value(counter));
|
||||
}
|
||||
|
||||
void operator()(const broker::telemetry::metric_family_hdl* family,
|
||||
const broker::telemetry::int_counter_hdl* counter,
|
||||
broker::telemetry::const_label_list labels)
|
||||
{
|
||||
if ( matches(family) )
|
||||
metrics.emplace_back(MetricType::Counter, family, extract_label_values(labels),
|
||||
broker::telemetry::value(counter));
|
||||
}
|
||||
|
||||
void operator()(const broker::telemetry::metric_family_hdl* family,
|
||||
const broker::telemetry::dbl_gauge_hdl* gauge,
|
||||
broker::telemetry::const_label_list labels)
|
||||
{
|
||||
if ( matches(family) )
|
||||
metrics.emplace_back(MetricType::Gauge, family, extract_label_values(labels),
|
||||
broker::telemetry::value(gauge));
|
||||
}
|
||||
|
||||
void operator()(const broker::telemetry::metric_family_hdl* family,
|
||||
const broker::telemetry::int_gauge_hdl* gauge,
|
||||
broker::telemetry::const_label_list labels)
|
||||
{
|
||||
if ( matches(family) )
|
||||
metrics.emplace_back(MetricType::Gauge, family, extract_label_values(labels),
|
||||
broker::telemetry::value(gauge));
|
||||
}
|
||||
|
||||
void operator()(const broker::telemetry::metric_family_hdl* family,
|
||||
const broker::telemetry::dbl_histogram_hdl* histogram,
|
||||
broker::telemetry::const_label_list labels)
|
||||
{
|
||||
// Ignored
|
||||
}
|
||||
|
||||
void operator()(const broker::telemetry::metric_family_hdl* family,
|
||||
const broker::telemetry::int_histogram_hdl* histogram,
|
||||
broker::telemetry::const_label_list labels)
|
||||
{
|
||||
// Ignored
|
||||
}
|
||||
|
||||
std::vector<Manager::CollectedValueMetric>& GetResult() { return metrics; }
|
||||
|
||||
private:
|
||||
MetricFamilyMatcher matches;
|
||||
std::vector<Manager::CollectedValueMetric> metrics;
|
||||
};
|
||||
|
||||
std::vector<Manager::CollectedValueMetric> Manager::CollectMetrics(std::string_view prefix,
|
||||
std::string_view name)
|
||||
{
|
||||
auto collector = MetricsCollector(prefix, name);
|
||||
|
||||
pimpl->collect(collector);
|
||||
|
||||
return std::move(collector.GetResult());
|
||||
}
|
||||
|
||||
/**
|
||||
* A collector implementation for histograms.
|
||||
*/
|
||||
class HistogramMetricsCollector : public broker::telemetry::metrics_collector
|
||||
{
|
||||
using MetricType = Manager::MetricType;
|
||||
|
||||
public:
|
||||
HistogramMetricsCollector(std::string_view prefix, std::string_view name)
|
||||
: matches(prefix, name)
|
||||
{
|
||||
}
|
||||
|
||||
void operator()(const broker::telemetry::metric_family_hdl* family,
|
||||
const broker::telemetry::dbl_counter_hdl* counter,
|
||||
broker::telemetry::const_label_list labels)
|
||||
{
|
||||
// Ignored
|
||||
}
|
||||
|
||||
void operator()(const broker::telemetry::metric_family_hdl* family,
|
||||
const broker::telemetry::int_counter_hdl* counter,
|
||||
broker::telemetry::const_label_list labels)
|
||||
{
|
||||
// Ignored
|
||||
}
|
||||
|
||||
void operator()(const broker::telemetry::metric_family_hdl* family,
|
||||
const broker::telemetry::dbl_gauge_hdl* gauge,
|
||||
broker::telemetry::const_label_list labels)
|
||||
{
|
||||
// Ignored
|
||||
}
|
||||
|
||||
void operator()(const broker::telemetry::metric_family_hdl* family,
|
||||
const broker::telemetry::int_gauge_hdl* gauge,
|
||||
broker::telemetry::const_label_list labels)
|
||||
{
|
||||
// Ignored
|
||||
}
|
||||
|
||||
void operator()(const broker::telemetry::metric_family_hdl* family,
|
||||
const broker::telemetry::dbl_histogram_hdl* histogram,
|
||||
broker::telemetry::const_label_list labels)
|
||||
{
|
||||
if ( ! matches(family) )
|
||||
return;
|
||||
|
||||
size_t num_buckets = broker::telemetry::num_buckets(histogram);
|
||||
|
||||
Manager::CollectedHistogramMetric::DblHistogramData histogram_data;
|
||||
histogram_data.buckets.reserve(num_buckets);
|
||||
|
||||
for ( size_t i = 0; i < num_buckets; i++ )
|
||||
{
|
||||
double c = broker::telemetry::count_at(histogram, i);
|
||||
double ub = broker::telemetry::upper_bound_at(histogram, i);
|
||||
histogram_data.buckets.emplace_back(c, ub);
|
||||
}
|
||||
|
||||
histogram_data.sum = broker::telemetry::sum(histogram);
|
||||
|
||||
metrics.emplace_back(family, extract_label_values(labels), std::move(histogram_data));
|
||||
}
|
||||
|
||||
void operator()(const broker::telemetry::metric_family_hdl* family,
|
||||
const broker::telemetry::int_histogram_hdl* histogram,
|
||||
broker::telemetry::const_label_list labels)
|
||||
{
|
||||
if ( ! matches(family) )
|
||||
return;
|
||||
|
||||
size_t num_buckets = broker::telemetry::num_buckets(histogram);
|
||||
|
||||
Manager::CollectedHistogramMetric::IntHistogramData histogram_data;
|
||||
histogram_data.buckets.reserve(num_buckets);
|
||||
|
||||
for ( size_t i = 0; i < num_buckets; i++ )
|
||||
{
|
||||
int64_t c = broker::telemetry::count_at(histogram, i);
|
||||
int64_t ub = broker::telemetry::upper_bound_at(histogram, i);
|
||||
histogram_data.buckets.emplace_back(c, ub);
|
||||
}
|
||||
|
||||
histogram_data.sum = broker::telemetry::sum(histogram);
|
||||
|
||||
metrics.emplace_back(family, extract_label_values(labels), std::move(histogram_data));
|
||||
}
|
||||
|
||||
std::vector<Manager::CollectedHistogramMetric>& GetResult() { return metrics; }
|
||||
|
||||
private:
|
||||
MetricFamilyMatcher matches;
|
||||
std::vector<Manager::CollectedHistogramMetric> metrics;
|
||||
};
|
||||
|
||||
std::vector<Manager::CollectedHistogramMetric>
|
||||
Manager::CollectHistogramMetrics(std::string_view prefix, std::string_view name)
|
||||
{
|
||||
auto collector = HistogramMetricsCollector(prefix, name);
|
||||
|
||||
pimpl->collect(collector);
|
||||
|
||||
return std::move(collector.GetResult());
|
||||
}
|
||||
|
||||
} // namespace zeek::telemetry
|
||||
|
||||
// -- unit tests ---------------------------------------------------------------
|
||||
|
|
@ -58,60 +513,6 @@ template <class T> auto toVector(zeek::Span<T> xs)
|
|||
|
||||
} // namespace
|
||||
|
||||
SCENARIO("telemetry managers provide access to counter singletons")
|
||||
{
|
||||
GIVEN("a telemetry manager")
|
||||
{
|
||||
Manager mgr;
|
||||
WHEN("retrieving an IntCounter singleton")
|
||||
{
|
||||
auto first = mgr.CounterSingleton("zeek", "int-count", "test");
|
||||
THEN("its initial value is zero") { CHECK_EQ(first.Value(), 0); }
|
||||
AND_THEN("calling Inc() or operator++ changes the value")
|
||||
{
|
||||
first.Inc();
|
||||
CHECK_EQ(first.Value(), 1);
|
||||
first.Inc(2);
|
||||
CHECK_EQ(first.Value(), 3);
|
||||
CHECK_EQ(++first, 4);
|
||||
CHECK_EQ(first.Value(), 4);
|
||||
}
|
||||
AND_THEN("calling counterSingleton again for the same name returns the same handle")
|
||||
{
|
||||
auto second = mgr.CounterSingleton("zeek", "int-count", "test");
|
||||
CHECK_EQ(first, second);
|
||||
}
|
||||
AND_THEN("calling counterSingleton for a different name returns another handle")
|
||||
{
|
||||
auto third = mgr.CounterSingleton("zeek", "int-count-2", "test");
|
||||
CHECK_NE(first, third);
|
||||
}
|
||||
}
|
||||
WHEN("retrieving a DblCounter singleton")
|
||||
{
|
||||
auto first = mgr.CounterSingleton<double>("zeek", "dbl-count", "test");
|
||||
THEN("its initial value is zero") { CHECK_EQ(first.Value(), 0.0); }
|
||||
AND_THEN("calling Inc() changes the value")
|
||||
{
|
||||
first.Inc();
|
||||
CHECK_EQ(first.Value(), 1.0);
|
||||
first.Inc(3.0);
|
||||
CHECK_EQ(first.Value(), 4.0);
|
||||
}
|
||||
AND_THEN("calling counterSingleton again for the same name returns the same handle")
|
||||
{
|
||||
auto second = mgr.CounterSingleton<double>("zeek", "dbl-count", "test");
|
||||
CHECK_EQ(first, second);
|
||||
}
|
||||
AND_THEN("calling counterSingleton for a different name returns another handle")
|
||||
{
|
||||
auto third = mgr.CounterSingleton<double>("zeek", "dbl-count-2", "test");
|
||||
CHECK_NE(first, third);
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
SCENARIO("telemetry managers provide access to counter families")
|
||||
{
|
||||
GIVEN("a telemetry manager")
|
||||
|
|
@ -171,70 +572,6 @@ SCENARIO("telemetry managers provide access to counter families")
|
|||
}
|
||||
}
|
||||
|
||||
SCENARIO("telemetry managers provide access to gauge singletons")
|
||||
{
|
||||
GIVEN("a telemetry manager")
|
||||
{
|
||||
Manager mgr;
|
||||
WHEN("retrieving an IntGauge singleton")
|
||||
{
|
||||
auto first = mgr.GaugeSingleton("zeek", "int-gauge", "test");
|
||||
THEN("its initial value is zero") { CHECK_EQ(first.Value(), 0); }
|
||||
AND_THEN("calling Inc(), Dec(), operator++ or operator-- changes the value")
|
||||
{
|
||||
first.Inc();
|
||||
CHECK_EQ(first.Value(), 1);
|
||||
first.Inc(2);
|
||||
CHECK_EQ(first.Value(), 3);
|
||||
first.Dec();
|
||||
CHECK_EQ(first.Value(), 2);
|
||||
CHECK_EQ(++first, 3);
|
||||
CHECK_EQ(first.Value(), 3);
|
||||
CHECK_EQ(--first, 2);
|
||||
CHECK_EQ(first.Value(), 2);
|
||||
first.Dec(2);
|
||||
CHECK_EQ(first.Value(), 0);
|
||||
}
|
||||
AND_THEN("calling gaugeSingleton again for the same name returns the same handle")
|
||||
{
|
||||
auto second = mgr.GaugeSingleton("zeek", "int-gauge", "test");
|
||||
CHECK_EQ(first, second);
|
||||
}
|
||||
AND_THEN("calling gaugeSingleton for a different name returns another handle")
|
||||
{
|
||||
auto third = mgr.GaugeSingleton("zeek", "int-gauge-2", "test");
|
||||
CHECK_NE(first, third);
|
||||
}
|
||||
}
|
||||
WHEN("retrieving a DblGauge singleton")
|
||||
{
|
||||
auto first = mgr.GaugeSingleton<double>("zeek", "dbl-gauge", "test");
|
||||
THEN("its initial value is zero") { CHECK_EQ(first.Value(), 0.0); }
|
||||
AND_THEN("calling Inc() or Dec() changes the value")
|
||||
{
|
||||
first.Inc();
|
||||
CHECK_EQ(first.Value(), 1.0);
|
||||
first.Inc(3.0);
|
||||
CHECK_EQ(first.Value(), 4.0);
|
||||
first.Dec(2.0);
|
||||
CHECK_EQ(first.Value(), 2.0);
|
||||
first.Dec();
|
||||
CHECK_EQ(first.Value(), 1.0);
|
||||
}
|
||||
AND_THEN("calling gaugeSingleton again for the same name returns the same handle")
|
||||
{
|
||||
auto second = mgr.GaugeSingleton<double>("zeek", "dbl-gauge", "test");
|
||||
CHECK_EQ(first, second);
|
||||
}
|
||||
AND_THEN("calling gaugeSingleton for a different name returns another handle")
|
||||
{
|
||||
auto third = mgr.GaugeSingleton<double>("zeek", "dbl-gauge-2", "test");
|
||||
CHECK_NE(first, third);
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
SCENARIO("telemetry managers provide access to gauge families")
|
||||
{
|
||||
GIVEN("a telemetry manager")
|
||||
|
|
@ -294,92 +631,6 @@ SCENARIO("telemetry managers provide access to gauge families")
|
|||
}
|
||||
}
|
||||
|
||||
SCENARIO("telemetry managers provide access to histogram singletons")
|
||||
{
|
||||
GIVEN("a telemetry manager")
|
||||
{
|
||||
Manager mgr;
|
||||
WHEN("retrieving an IntHistogram singleton")
|
||||
{
|
||||
const auto max_int = std::numeric_limits<int64_t>::max();
|
||||
int64_t buckets[] = {10, 20};
|
||||
auto first = mgr.HistogramSingleton("zeek", "int-hist", buckets, "test");
|
||||
THEN("it initially has no observations")
|
||||
{
|
||||
REQUIRE_EQ(first.NumBuckets(), 3u);
|
||||
CHECK_EQ(first.Sum(), 0);
|
||||
CHECK_EQ(first.CountAt(0), 0);
|
||||
CHECK_EQ(first.CountAt(1), 0);
|
||||
CHECK_EQ(first.CountAt(2), 0);
|
||||
CHECK_EQ(first.UpperBoundAt(0), 10);
|
||||
CHECK_EQ(first.UpperBoundAt(1), 20);
|
||||
CHECK_EQ(first.UpperBoundAt(2), max_int);
|
||||
}
|
||||
AND_THEN("calling Observe() increments bucket counters")
|
||||
{
|
||||
first.Observe(1);
|
||||
first.Observe(9);
|
||||
first.Observe(10);
|
||||
first.Observe(11);
|
||||
first.Observe(19);
|
||||
first.Observe(20);
|
||||
first.Observe(21);
|
||||
CHECK_EQ(first.Sum(), 91);
|
||||
CHECK_EQ(first.CountAt(0), 3);
|
||||
CHECK_EQ(first.CountAt(1), 3);
|
||||
CHECK_EQ(first.CountAt(2), 1);
|
||||
}
|
||||
AND_THEN("calling HistogramSingleton again for the same name returns the same handle")
|
||||
{
|
||||
auto second = mgr.HistogramSingleton("zeek", "int-hist", buckets, "test");
|
||||
CHECK_EQ(first, second);
|
||||
}
|
||||
AND_THEN("calling HistogramSingleton for a different name returns another handle")
|
||||
{
|
||||
auto third = mgr.HistogramSingleton("zeek", "int-hist-2", buckets, "test");
|
||||
CHECK_NE(first, third);
|
||||
}
|
||||
}
|
||||
WHEN("retrieving a DblHistogram singleton")
|
||||
{
|
||||
double buckets[] = {10.0, 20.0};
|
||||
auto first = mgr.HistogramSingleton<double>("zeek", "dbl-count", buckets, "test");
|
||||
THEN("it initially has no observations")
|
||||
{
|
||||
REQUIRE_EQ(first.NumBuckets(), 3u);
|
||||
CHECK_EQ(first.Sum(), 0.0);
|
||||
CHECK_EQ(first.CountAt(0), 0);
|
||||
CHECK_EQ(first.CountAt(1), 0);
|
||||
CHECK_EQ(first.CountAt(2), 0);
|
||||
CHECK_EQ(first.UpperBoundAt(0), 10.0);
|
||||
CHECK_EQ(first.UpperBoundAt(1), 20.0);
|
||||
}
|
||||
AND_THEN("calling Observe() increments bucket counters")
|
||||
{
|
||||
first.Observe(2.0);
|
||||
first.Observe(4.0);
|
||||
first.Observe(8.0);
|
||||
first.Observe(16.0);
|
||||
first.Observe(32.0);
|
||||
CHECK_EQ(first.Sum(), 62.0);
|
||||
CHECK_EQ(first.CountAt(0), 3);
|
||||
CHECK_EQ(first.CountAt(1), 1);
|
||||
CHECK_EQ(first.CountAt(2), 1);
|
||||
}
|
||||
AND_THEN("calling histogramSingleton again for the same name returns the same handle")
|
||||
{
|
||||
auto second = mgr.HistogramSingleton<double>("zeek", "dbl-count", buckets, "test");
|
||||
CHECK_EQ(first, second);
|
||||
}
|
||||
AND_THEN("calling histogramSingleton for a different name returns another handle")
|
||||
{
|
||||
auto third = mgr.HistogramSingleton<double>("zeek", "dbl-count-2", buckets, "test");
|
||||
CHECK_NE(first, third);
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
SCENARIO("telemetry managers provide access to histogram families")
|
||||
{
|
||||
GIVEN("a telemetry manager")
|
||||
|
|
|
|||
|
|
@ -5,6 +5,8 @@
|
|||
#include <cstdint>
|
||||
#include <initializer_list>
|
||||
#include <string_view>
|
||||
#include <unordered_map>
|
||||
#include <variant>
|
||||
#include <vector>
|
||||
|
||||
#include "zeek/IntrusivePtr.h"
|
||||
|
|
@ -20,6 +22,12 @@ namespace broker
|
|||
class endpoint;
|
||||
}
|
||||
|
||||
namespace zeek
|
||||
{
|
||||
class RecordVal;
|
||||
using RecordValPtr = IntrusivePtr<RecordVal>;
|
||||
}
|
||||
|
||||
namespace zeek::Broker
|
||||
{
|
||||
class Manager;
|
||||
|
|
@ -50,6 +58,128 @@ public:
|
|||
*/
|
||||
virtual void InitPostScript();
|
||||
|
||||
/**
|
||||
* Supported metric types.
|
||||
*/
|
||||
enum class MetricType
|
||||
{
|
||||
Counter,
|
||||
Gauge,
|
||||
Histogram
|
||||
};
|
||||
|
||||
/**
|
||||
* Captures information about counter and gauge metrics.
|
||||
*/
|
||||
struct CollectedValueMetric
|
||||
{
|
||||
/**
|
||||
* Constructor.
|
||||
* @param metric_type The type of this metric.
|
||||
* @param family Broker layer family handle for this metric.
|
||||
* @param label_values The string values for each of the metric's labels.
|
||||
* @param value The metric's current value.
|
||||
*/
|
||||
CollectedValueMetric(MetricType metric_type,
|
||||
const broker::telemetry::metric_family_hdl* family,
|
||||
std::vector<std::string_view> label_values,
|
||||
std::variant<double, int64_t> value)
|
||||
: metric_type(metric_type), family(family), label_values(std::move(label_values)),
|
||||
value(value)
|
||||
{
|
||||
}
|
||||
|
||||
/**
|
||||
* @return A script layer Telemetry::Metric record for this metric.
|
||||
*/
|
||||
zeek::RecordValPtr AsMetricRecord() const;
|
||||
|
||||
enum MetricType metric_type;
|
||||
const broker::telemetry::metric_family_hdl* family;
|
||||
std::vector<std::string_view> label_values;
|
||||
std::variant<double, int64_t> value;
|
||||
};
|
||||
|
||||
/**
|
||||
* Captures information about histogram metrics.
|
||||
*/
|
||||
struct CollectedHistogramMetric
|
||||
{
|
||||
/**
|
||||
* Helper struct representing a single bucket of a histogram.
|
||||
* @tparam T The data type used by the histogram (double or int64_t).
|
||||
*/
|
||||
template <class T> struct Bucket
|
||||
{
|
||||
Bucket(T count, T upper_bound) : count(count), upper_bound(upper_bound) { }
|
||||
|
||||
T count;
|
||||
T upper_bound;
|
||||
};
|
||||
|
||||
/**
|
||||
* Helper struct representing a histogram as sum and buckets.
|
||||
* @tparam T The data type used by the histogram (double or int64_t).
|
||||
*/
|
||||
template <class T> struct HistogramData
|
||||
{
|
||||
T sum;
|
||||
std::vector<Bucket<T>> buckets;
|
||||
};
|
||||
|
||||
using DblHistogramData = HistogramData<double>;
|
||||
using IntHistogramData = HistogramData<int64_t>;
|
||||
|
||||
/**
|
||||
* Constructor.
|
||||
* @param family Broker layer family handle for this metric.
|
||||
* @param label_values The string values for each of the metric's labels.
|
||||
* @param histogram The histogram's data (sum and individual buckets).
|
||||
*/
|
||||
CollectedHistogramMetric(const broker::telemetry::metric_family_hdl* family,
|
||||
std::vector<std::string_view> label_values,
|
||||
std::variant<DblHistogramData, IntHistogramData> histogram)
|
||||
|
||||
: family(family), label_values(std::move(label_values)), histogram(std::move(histogram))
|
||||
{
|
||||
}
|
||||
|
||||
const broker::telemetry::metric_family_hdl* family;
|
||||
std::vector<std::string_view> label_values;
|
||||
std::variant<DblHistogramData, IntHistogramData> histogram;
|
||||
|
||||
/**
|
||||
* @return A script layer Telemetry::HistogramMetric record for this histogram.
|
||||
*/
|
||||
zeek::RecordValPtr AsHistogramMetricRecord() const;
|
||||
};
|
||||
|
||||
/**
|
||||
* @return A script layer Telemetry::MetricOpts record for the given metric family.
|
||||
* @param metric_typ The type of metric.
|
||||
* @param family Broker layer family handle for the family.
|
||||
* @tparam T The underlying data type (double or int64_t)
|
||||
*/
|
||||
template <typename T>
|
||||
zeek::RecordValPtr GetMetricOptsRecord(MetricType metric_type,
|
||||
const broker::telemetry::metric_family_hdl* family);
|
||||
|
||||
/**
|
||||
* @return All counter and gauge metrics and their values matching prefix and name.
|
||||
* @param prefix The prefix pattern to use for filtering. Supports globbing.
|
||||
* @param name The name pattern to use for filtering. Supports globbing.
|
||||
*/
|
||||
std::vector<CollectedValueMetric> CollectMetrics(std::string_view prefix,
|
||||
std::string_view name);
|
||||
|
||||
/**
|
||||
* @return All histogram metrics and their data matching prefix and name.
|
||||
* @param prefix The prefix pattern to use for filtering. Supports globbing.
|
||||
* @param name The name pattern to use for filtering. Supports globbing.
|
||||
*/
|
||||
std::vector<CollectedHistogramMetric> CollectHistogramMetrics(std::string_view prefix,
|
||||
std::string_view name);
|
||||
|
||||
/**
|
||||
* @return A counter metric family. Creates the family lazily if necessary.
|
||||
* @param prefix The prefix (namespace) this family belongs to.
|
||||
|
|
@ -67,14 +197,14 @@ public:
|
|||
{
|
||||
if constexpr ( std::is_same<ValueType, int64_t>::value )
|
||||
{
|
||||
auto fam = int_counter_fam(ptr(), prefix, name, labels, helptext, unit, is_sum);
|
||||
auto fam = int_counter_fam(Ptr(), prefix, name, labels, helptext, unit, is_sum);
|
||||
return IntCounterFamily{fam};
|
||||
}
|
||||
else
|
||||
{
|
||||
static_assert(std::is_same<ValueType, double>::value,
|
||||
"metrics only support int64_t and double values");
|
||||
auto fam = dbl_counter_fam(ptr(), prefix, name, labels, helptext, unit, is_sum);
|
||||
auto fam = dbl_counter_fam(Ptr(), prefix, name, labels, helptext, unit, is_sum);
|
||||
return DblCounterFamily{fam};
|
||||
}
|
||||
}
|
||||
|
|
@ -125,27 +255,6 @@ public:
|
|||
return CounterInstance(prefix, name, lbl_span, helptext, unit, is_sum);
|
||||
}
|
||||
|
||||
/**
|
||||
* Accesses a counter singleton, i.e., a counter that belongs to a family
|
||||
* without label dimensions (which thus only has a single member). Creates
|
||||
* the hosting metric family as well as the counter lazily if necessary.
|
||||
* @param prefix The prefix (namespace) this family belongs to.
|
||||
* @param name The human-readable name of the metric, e.g., `requests`.
|
||||
* @param helptext Short explanation of the metric.
|
||||
* @param unit Unit of measurement.
|
||||
* @param is_sum Indicates whether this metric accumulates something, where
|
||||
* only the total value is of interest.
|
||||
*/
|
||||
template <class ValueType = int64_t>
|
||||
Counter<ValueType> CounterSingleton(std::string_view prefix, std::string_view name,
|
||||
std::string_view helptext, std::string_view unit = "1",
|
||||
bool is_sum = false)
|
||||
{
|
||||
auto labels = Span<const std::string_view>{};
|
||||
auto fam = CounterFamily<ValueType>(prefix, name, labels, helptext, unit, is_sum);
|
||||
return fam.GetOrAdd({});
|
||||
}
|
||||
|
||||
/**
|
||||
* @return A gauge metric family. Creates the family lazily if necessary.
|
||||
* @param prefix The prefix (namespace) this family belongs to.
|
||||
|
|
@ -163,14 +272,14 @@ public:
|
|||
{
|
||||
if constexpr ( std::is_same<ValueType, int64_t>::value )
|
||||
{
|
||||
auto fam = int_gauge_fam(ptr(), prefix, name, labels, helptext, unit, is_sum);
|
||||
auto fam = int_gauge_fam(Ptr(), prefix, name, labels, helptext, unit, is_sum);
|
||||
return IntGaugeFamily{fam};
|
||||
}
|
||||
else
|
||||
{
|
||||
static_assert(std::is_same<ValueType, double>::value,
|
||||
"metrics only support int64_t and double values");
|
||||
auto fam = dbl_gauge_fam(ptr(), prefix, name, labels, helptext, unit, is_sum);
|
||||
auto fam = dbl_gauge_fam(Ptr(), prefix, name, labels, helptext, unit, is_sum);
|
||||
return DblGaugeFamily{fam};
|
||||
}
|
||||
}
|
||||
|
|
@ -221,27 +330,6 @@ public:
|
|||
return GaugeInstance(prefix, name, lbl_span, helptext, unit, is_sum);
|
||||
}
|
||||
|
||||
/**
|
||||
* Accesses a gauge singleton, i.e., a gauge that belongs to a family
|
||||
* without label dimensions (which thus only has a single member). Creates
|
||||
* the hosting metric family as well as the gauge lazily if necessary.
|
||||
* @param prefix The prefix (namespace) this family belongs to.
|
||||
* @param name The human-readable name of the metric, e.g., `requests`.
|
||||
* @param helptext Short explanation of the metric.
|
||||
* @param unit Unit of measurement.
|
||||
* @param is_sum Indicates whether this metric accumulates something, where
|
||||
* only the total value is of interest.
|
||||
*/
|
||||
template <class ValueType = int64_t>
|
||||
Gauge<ValueType> GaugeSingleton(std::string_view prefix, std::string_view name,
|
||||
std::string_view helptext, std::string_view unit = "1",
|
||||
bool is_sum = false)
|
||||
{
|
||||
auto labels = Span<const std::string_view>{};
|
||||
auto fam = GaugeFamily<ValueType>(prefix, name, labels, helptext, unit, is_sum);
|
||||
return fam.GetOrAdd({});
|
||||
}
|
||||
|
||||
// Forces the compiler to use the type `Span<const T>` instead of trying to
|
||||
// match paremeters to a `span`.
|
||||
template <class T> struct ConstSpanOracle
|
||||
|
|
@ -281,7 +369,7 @@ public:
|
|||
{
|
||||
if constexpr ( std::is_same<ValueType, int64_t>::value )
|
||||
{
|
||||
auto fam = int_histogram_fam(ptr(), prefix, name, labels, default_upper_bounds,
|
||||
auto fam = int_histogram_fam(Ptr(), prefix, name, labels, default_upper_bounds,
|
||||
helptext, unit, is_sum);
|
||||
return IntHistogramFamily{fam};
|
||||
}
|
||||
|
|
@ -289,7 +377,7 @@ public:
|
|||
{
|
||||
static_assert(std::is_same<ValueType, double>::value,
|
||||
"metrics only support int64_t and double values");
|
||||
auto fam = dbl_histogram_fam(ptr(), prefix, name, labels, default_upper_bounds,
|
||||
auto fam = dbl_histogram_fam(Ptr(), prefix, name, labels, default_upper_bounds,
|
||||
helptext, unit, is_sum);
|
||||
return DblHistogramFamily{fam};
|
||||
}
|
||||
|
|
@ -355,39 +443,6 @@ public:
|
|||
return HistogramInstance(prefix, name, lbls, default_upper_bounds, helptext, unit, is_sum);
|
||||
}
|
||||
|
||||
/**
|
||||
* Returns a histogram metric singleton, i.e., the single instance of a
|
||||
* family without label dimensions. Creates all objects lazily if necessary,
|
||||
* but fails if the full name already belongs to a different family.
|
||||
* @param prefix The prefix (namespace) this family belongs to. Usually the
|
||||
* application or protocol name, e.g., `http`. The prefix `caf`
|
||||
* as well as prefixes starting with an underscore are
|
||||
* reserved.
|
||||
* @param name The human-readable name of the metric, e.g., `requests`.
|
||||
* @param default_upper_bounds Upper bounds for the metric buckets.
|
||||
* @param helptext Short explanation of the metric.
|
||||
* @param unit Unit of measurement. Please use base units such as `bytes` or
|
||||
* `seconds` (prefer lowercase). The pseudo-unit `1` identifies
|
||||
* dimensionless counts.
|
||||
* @param is_sum Setting this to `true` indicates that this metric adds
|
||||
* something up to a total, where only the total value is of
|
||||
* interest. For example, the total number of HTTP requests.
|
||||
* @note The first call wins when calling this function multiple times with
|
||||
* different bucket settings. Users may also override
|
||||
* @p default_upper_bounds via run-time configuration.
|
||||
*/
|
||||
template <class ValueType = int64_t>
|
||||
Histogram<ValueType> HistogramSingleton(std::string_view prefix, std::string_view name,
|
||||
ConstSpan<ValueType> default_upper_bounds,
|
||||
std::string_view helptext, std::string_view unit = "1",
|
||||
bool is_sum = false)
|
||||
{
|
||||
auto lbls = Span<const std::string_view>{};
|
||||
auto fam = HistogramFamily<ValueType>(prefix, name, lbls, default_upper_bounds, helptext,
|
||||
unit, is_sum);
|
||||
return fam.GetOrAdd({});
|
||||
}
|
||||
|
||||
protected:
|
||||
template <class F> static void WithLabelNames(Span<const LabelView> xs, F continuation)
|
||||
{
|
||||
|
|
@ -396,6 +451,7 @@ protected:
|
|||
std::string_view buf[10];
|
||||
for ( size_t index = 0; index < xs.size(); ++index )
|
||||
buf[index] = xs[index].first;
|
||||
|
||||
return continuation(Span{buf, xs.size()});
|
||||
}
|
||||
else
|
||||
|
|
@ -403,17 +459,23 @@ protected:
|
|||
std::vector<std::string_view> buf;
|
||||
for ( auto x : xs )
|
||||
buf.emplace_back(x.first, x.second);
|
||||
|
||||
return continuation(Span{buf});
|
||||
}
|
||||
}
|
||||
|
||||
broker::telemetry::metric_registry_impl* ptr() { return pimpl.get(); }
|
||||
broker::telemetry::metric_registry_impl* Ptr() { return pimpl.get(); }
|
||||
|
||||
// Connects all the dots after the Broker Manager constructed the endpoint
|
||||
// for this Zeek instance. Called from Broker::Manager::InitPostScript().
|
||||
void InitPostBrokerSetup(broker::endpoint&);
|
||||
|
||||
IntrusivePtr<broker::telemetry::metric_registry_impl> pimpl;
|
||||
|
||||
private:
|
||||
// Caching of metric_family_hdl instances to their Zeek record representation.
|
||||
std::unordered_map<const broker::telemetry::metric_family_hdl*, zeek::RecordValPtr>
|
||||
metric_opts_cache;
|
||||
};
|
||||
|
||||
} // namespace zeek::telemetry
|
||||
|
|
|
|||
|
|
@ -1,8 +1,16 @@
|
|||
|
||||
##! Functions for accessing counter metrics from script land.
|
||||
|
||||
module Telemetry;
|
||||
|
||||
enum MetricType %{
|
||||
DOUBLE_COUNTER,
|
||||
INT_COUNTER,
|
||||
DOUBLE_GAUGE,
|
||||
INT_GAUGE,
|
||||
DOUBLE_HISTOGRAM,
|
||||
INT_HISTOGRAM,
|
||||
%}
|
||||
|
||||
%%{
|
||||
#include "zeek/telemetry/Counter.h"
|
||||
#include "zeek/telemetry/Gauge.h"
|
||||
|
|
@ -139,17 +147,6 @@ function Telemetry::__int_counter_metric_get_or_add%(family: opaque of int_count
|
|||
}
|
||||
%}
|
||||
|
||||
function Telemetry::__int_counter_singleton%(prefix: string,
|
||||
name: string,
|
||||
helptext: string &default = "Zeek Script Metric",
|
||||
unit: string &default = "1",
|
||||
is_sum: bool &default = F%): opaque of int_counter_metric
|
||||
%{
|
||||
auto hdl = telemetry_mgr->CounterSingleton(sv(prefix), sv(name),
|
||||
sv(helptext), sv(unit), is_sum);
|
||||
return zeek::make_intrusive<IntCounterMetricVal>(hdl);
|
||||
%}
|
||||
|
||||
function Telemetry::__int_counter_inc%(val: opaque of int_counter_metric,
|
||||
amount: int &default = 1%): bool
|
||||
%{
|
||||
|
|
@ -210,17 +207,6 @@ function Telemetry::__dbl_counter_metric_get_or_add%(family: opaque of dbl_count
|
|||
}
|
||||
%}
|
||||
|
||||
function Telemetry::__dbl_counter_singleton%(prefix: string,
|
||||
name: string,
|
||||
helptext: string &default = "Zeek Script Metric",
|
||||
unit: string &default = "1",
|
||||
is_sum: bool &default = F%): opaque of dbl_counter_metric
|
||||
%{
|
||||
auto hdl = telemetry_mgr->CounterSingleton<double>(sv(prefix), sv(name),
|
||||
sv(helptext), sv(unit), is_sum);
|
||||
return zeek::make_intrusive<DblCounterMetricVal>(hdl);
|
||||
%}
|
||||
|
||||
function Telemetry::__dbl_counter_inc%(val: opaque of dbl_counter_metric,
|
||||
amount: double &default = 1.0%): bool
|
||||
%{
|
||||
|
|
@ -281,17 +267,6 @@ function Telemetry::__int_gauge_metric_get_or_add%(family: opaque of int_gauge_m
|
|||
}
|
||||
%}
|
||||
|
||||
function Telemetry::__int_gauge_singleton%(prefix: string,
|
||||
name: string,
|
||||
helptext: string &default = "Zeek Script Metric",
|
||||
unit: string &default = "1",
|
||||
is_sum: bool &default = F%): opaque of int_gauge_metric
|
||||
%{
|
||||
auto hdl = telemetry_mgr->GaugeSingleton(sv(prefix), sv(name),
|
||||
sv(helptext), sv(unit), is_sum);
|
||||
return zeek::make_intrusive<IntGaugeMetricVal>(hdl);
|
||||
%}
|
||||
|
||||
function Telemetry::__int_gauge_inc%(val: opaque of int_gauge_metric,
|
||||
amount: int &default = 1%): bool
|
||||
%{
|
||||
|
|
@ -358,17 +333,6 @@ function Telemetry::__dbl_gauge_metric_get_or_add%(family: opaque of dbl_gauge_m
|
|||
}
|
||||
%}
|
||||
|
||||
function Telemetry::__dbl_gauge_singleton%(prefix: string,
|
||||
name: string,
|
||||
helptext: string &default = "Zeek Script Metric",
|
||||
unit: string &default = "1",
|
||||
is_sum: bool &default = F%): opaque of dbl_gauge_metric
|
||||
%{
|
||||
auto hdl = telemetry_mgr->GaugeSingleton<double>(sv(prefix), sv(name),
|
||||
sv(helptext), sv(unit), is_sum);
|
||||
return zeek::make_intrusive<DblGaugeMetricVal>(hdl);
|
||||
%}
|
||||
|
||||
function Telemetry::__dbl_gauge_inc%(val: opaque of dbl_gauge_metric,
|
||||
amount: double &default = 1.0%): bool
|
||||
%{
|
||||
|
|
@ -438,20 +402,6 @@ function Telemetry::__int_histogram_metric_get_or_add%(family: opaque of int_his
|
|||
}
|
||||
%}
|
||||
|
||||
function Telemetry::__int_histogram_singleton%(prefix: string,
|
||||
name: string,
|
||||
bounds: int_vec,
|
||||
helptext: string &default = "Zeek Script Metric",
|
||||
unit: string &default = "1",
|
||||
is_sum: bool &default = F%): opaque of int_histogram_metric
|
||||
%{
|
||||
auto std_bounds = to_std_vec<int64_t>(bounds);
|
||||
auto hdl = telemetry_mgr->HistogramSingleton(sv(prefix), sv(name),
|
||||
std_bounds, sv(helptext),
|
||||
sv(unit), is_sum);
|
||||
return zeek::make_intrusive<IntHistogramMetricVal>(hdl);
|
||||
%}
|
||||
|
||||
function Telemetry::__int_histogram_observe%(val: opaque of int_histogram_metric,
|
||||
measurement: int%): bool
|
||||
%{
|
||||
|
|
@ -516,20 +466,6 @@ function Telemetry::__dbl_histogram_metric_get_or_add%(family: opaque of dbl_his
|
|||
}
|
||||
%}
|
||||
|
||||
function Telemetry::__dbl_histogram_singleton%(prefix: string,
|
||||
name: string,
|
||||
bounds: double_vec,
|
||||
helptext: string &default = "Zeek Script Metric",
|
||||
unit: string &default = "1",
|
||||
is_sum: bool &default = F%): opaque of dbl_histogram_metric
|
||||
%{
|
||||
auto std_bounds = to_std_vec<double>(bounds);
|
||||
auto hdl = telemetry_mgr->HistogramSingleton<double>(sv(prefix), sv(name),
|
||||
std_bounds, sv(helptext),
|
||||
sv(unit), is_sum);
|
||||
return zeek::make_intrusive<DblHistogramMetricVal>(hdl);
|
||||
%}
|
||||
|
||||
function Telemetry::__dbl_histogram_observe%(val: opaque of dbl_histogram_metric,
|
||||
measurement: double%): bool
|
||||
%{
|
||||
|
|
@ -548,3 +484,29 @@ function Telemetry::__dbl_histogram_sum%(val: opaque of dbl_histogram_metric%):
|
|||
return zeek::make_intrusive<DoubleVal>(0.0);
|
||||
}
|
||||
%}
|
||||
|
||||
function Telemetry::__collect_metrics%(prefix: string, name: string%): any_vec
|
||||
%{
|
||||
auto metrics = telemetry_mgr->CollectMetrics(sv(prefix), sv(name));
|
||||
|
||||
static auto metrics_vector_type = zeek::id::find_type<VectorType>("any_vec");
|
||||
auto vec = zeek::make_intrusive<VectorVal>(metrics_vector_type);
|
||||
|
||||
for ( const auto& m : metrics )
|
||||
vec->Append(m.AsMetricRecord());
|
||||
|
||||
return vec;
|
||||
%}
|
||||
|
||||
function Telemetry::__collect_histogram_metrics%(prefix: string, name: string%): any_vec
|
||||
%{
|
||||
auto metrics = telemetry_mgr->CollectHistogramMetrics(sv(prefix), sv(name));
|
||||
|
||||
static auto metrics_vector_type = zeek::id::find_type<VectorType>("any_vec");
|
||||
auto vec = zeek::make_intrusive<VectorVal>(metrics_vector_type);
|
||||
|
||||
for ( const auto& m : metrics )
|
||||
vec->Append(m.AsHistogramMetricRecord());
|
||||
|
||||
return vec;
|
||||
%}
|
||||
|
|
|
|||
73
src/zeek.bif
73
src/zeek.bif
|
|
@ -28,6 +28,7 @@
|
|||
#include "zeek/IntrusivePtr.h"
|
||||
#include "zeek/input.h"
|
||||
#include "zeek/Hash.h"
|
||||
#include "zeek/CompHash.h"
|
||||
#include "zeek/packet_analysis/Manager.h"
|
||||
|
||||
using namespace std;
|
||||
|
|
@ -1178,6 +1179,58 @@ function clear_table%(v: any%): any
|
|||
return nullptr;
|
||||
%}
|
||||
|
||||
## Gets all values from a table.
|
||||
##
|
||||
## t: The :zeek:type:`table`
|
||||
##
|
||||
## Returns: A ``vector of T`` of all the values in t.
|
||||
##
|
||||
## .. zeek:see:: table_keys
|
||||
function table_values%(t: any%): any_vec
|
||||
%{
|
||||
if ( ! t->GetType()->IsTable() )
|
||||
{
|
||||
zeek::emit_builtin_error("table_values() requires a table argument");
|
||||
return nullptr;
|
||||
}
|
||||
|
||||
auto vt = zeek::make_intrusive<zeek::VectorType>(t->GetType()->AsTableType()->Yield());
|
||||
auto vv = zeek::make_intrusive<zeek::VectorVal>(std::move(vt));
|
||||
for ( const auto& iter : *t->AsTable() )
|
||||
{
|
||||
vv->Append(iter.value->GetVal());
|
||||
}
|
||||
|
||||
return vv;
|
||||
%}
|
||||
|
||||
## Gets all keys from a table.
|
||||
##
|
||||
## t: The :zeek:type:`table`
|
||||
##
|
||||
## Returns: A ``set of T`` of all the keys in t.
|
||||
##
|
||||
## .. zeek:see:: table_values
|
||||
function table_keys%(t: any%): any
|
||||
%{
|
||||
if ( ! t->GetType()->IsTable() )
|
||||
{
|
||||
zeek::emit_builtin_error("table_keys() requires a table argument");
|
||||
return nullptr;
|
||||
}
|
||||
|
||||
auto tl = t->GetType()->AsTableType()->GetIndices();
|
||||
auto st = zeek::make_intrusive<zeek::SetType>(std::move(tl), nullptr);
|
||||
auto sv = zeek::make_intrusive<zeek::TableVal>(std::move(st));
|
||||
auto th = t->AsTableVal()->GetTableHash();
|
||||
for ( const auto& iter : *t->AsTable() )
|
||||
{
|
||||
sv->Assign(th->RecoverVals(*iter.GetHashKey()), nullptr);
|
||||
}
|
||||
|
||||
return sv;
|
||||
%}
|
||||
|
||||
## Gets all subnets that contain a given subnet from a set/table[subnet].
|
||||
##
|
||||
## search: the subnet to search for.
|
||||
|
|
@ -4590,6 +4643,15 @@ function disable_analyzer%(cid: conn_id, aid: count, err_if_no_conn: bool &defau
|
|||
return zeek::val_mgr->False();
|
||||
}
|
||||
|
||||
static auto disabling_analyzer_hook = id::find_func("disabling_analyzer");
|
||||
if ( disabling_analyzer_hook )
|
||||
{
|
||||
auto hook_rval = disabling_analyzer_hook->Invoke(c->GetVal(), a->GetAnalyzerTag().AsVal(),
|
||||
zeek::val_mgr->Count(aid));
|
||||
if ( hook_rval && ! hook_rval->AsBool() )
|
||||
return zeek::val_mgr->False();
|
||||
}
|
||||
|
||||
if ( prevent )
|
||||
a->Parent()->PreventChildren(a->GetAnalyzerTag());
|
||||
|
||||
|
|
@ -5304,16 +5366,7 @@ function match_signatures%(c: connection, pattern_type: int, s: string,
|
|||
## only useful for debugging and causes reduced performance.
|
||||
function generate_all_events%(%) : bool
|
||||
%{
|
||||
auto event_names = event_registry->AllHandlers();
|
||||
for ( const auto& name: event_names )
|
||||
{
|
||||
auto event = event_registry->Lookup(name);
|
||||
if ( event == nullptr )
|
||||
continue;
|
||||
|
||||
event->SetGenerateAlways();
|
||||
}
|
||||
|
||||
event_registry->ActivateAllHandlers();
|
||||
return zeek::val_mgr->True();
|
||||
%}
|
||||
|
||||
|
|
|
|||
|
|
@ -157,6 +157,10 @@ void Manager::ScriptDependency(const string& path, const string& dep)
|
|||
if ( disabled )
|
||||
return;
|
||||
|
||||
if ( path == "<command line>" )
|
||||
// This is a @load directive on the command line.
|
||||
return;
|
||||
|
||||
if ( dep.empty() )
|
||||
{
|
||||
DbgAndWarn(util::fmt("Empty Zeekygen script doc dependency: %s", path.c_str()));
|
||||
|
|
@ -188,6 +192,10 @@ void Manager::ModuleUsage(const string& path, const string& module)
|
|||
if ( disabled )
|
||||
return;
|
||||
|
||||
if ( path == "<command line>" )
|
||||
// This is a moudle defined on the command line.
|
||||
return;
|
||||
|
||||
string name = normalize_script_path(path);
|
||||
ScriptInfo* script_info = scripts.GetInfo(name);
|
||||
|
||||
|
|
|
|||
|
|
@ -1,8 +1,10 @@
|
|||
### BTest baseline data generated by btest-diff. Do not edit. Use "btest -U/-u" to update. Requires BTest >= 0.63.
|
||||
|
||||
--- Backtrace ---
|
||||
#0: zeek_init()
|
||||
|
||||
--- Backtrace ---
|
||||
|
||||
| #0: zeek_init() |
|
||||
|
||||
--- Backtrace ---
|
||||
#0: zeek_init()
|
||||
|
|
|
|||
7
testing/btest/Baseline.cpp/bifs.disable_analyzer/out
Normal file
7
testing/btest/Baseline.cpp/bifs.disable_analyzer/out
Normal file
|
|
@ -0,0 +1,7 @@
|
|||
### BTest baseline data generated by btest-diff. Do not edit. Use "btest -U/-u" to update. Requires BTest >= 0.63.
|
||||
proto confirm, AllAnalyzers::ANALYZER_ANALYZER_HTTP
|
||||
http_request, GET, /style/enhanced.css
|
||||
T
|
||||
total http messages, {
|
||||
[[orig_h=192.168.1.104, orig_p=1673/tcp, resp_h=63.245.209.11, resp_p=80/tcp]] = 1
|
||||
}
|
||||
|
|
@ -5,16 +5,16 @@ type aliases for 'MyRec val': MyRec MyRecAlias
|
|||
type aliases for 'MyRecAlias val': MyRec MyRecAlias
|
||||
type aliases for 'MyRec type': MyRec MyRecAlias
|
||||
type aliases for 'MyRecalias type': MyRec MyRecAlias
|
||||
type aliases for 'MyString val': MyString AnotherString
|
||||
type aliases for 'MyString val': it's just a 'string'
|
||||
type aliases for 'MyString type': MyString AnotherString
|
||||
type aliases for 'MyOtherString type': MyOtherString
|
||||
type aliases for 'AnotherString type': MyString AnotherString
|
||||
type aliases for 'string literal value': MyString AnotherString
|
||||
type aliases for 'string literal value': it's just a 'string'
|
||||
type aliases for 'count literal value': it's just a 'count'
|
||||
type aliases for 'MyTable value': MyTable2 MyTable3 MyTable MyTable4
|
||||
type aliases for 'MyTable2 value': MyTable2 MyTable3 MyTable MyTable4
|
||||
type aliases for 'MyTable3 value': MyTable2 MyTable3 MyTable MyTable4
|
||||
type aliases for 'MyTable4 value': MyTable2 MyTable3 MyTable MyTable4
|
||||
type aliases for 'MyTable value': it's just a 'table[count] of string'
|
||||
type aliases for 'MyTable2 value': it's just a 'table[count] of string'
|
||||
type aliases for 'MyTable3 value': it's just a 'table[count] of string'
|
||||
type aliases for 'MyTable4 value': it's just a 'table[count] of string'
|
||||
type aliases for 'MyTable type': MyTable2 MyTable3 MyTable MyTable4
|
||||
type aliases for 'MyTable2 type': MyTable2 MyTable3 MyTable MyTable4
|
||||
type aliases for 'MyTable3 type': MyTable2 MyTable3 MyTable MyTable4
|
||||
|
|
|
|||
|
|
@ -0,0 +1,11 @@
|
|||
### BTest baseline data generated by btest-diff. Do not edit. Use "btest -U/-u" to update. Requires BTest >= 0.63.
|
||||
error in <...>/main.zeek, lines 498-499: Failed to attach master store backend_failure: (Option::set_change_handler(Broker::metrics_export_prefixes, to_any_coerceBroker::update_metrics_export_prefixes, (coerce 0 to int)))
|
||||
error in <...>/main.zeek, lines 498-499: Could not create Broker master store '../fail' (Option::set_change_handler(Broker::metrics_export_prefixes, to_any_coerceBroker::update_metrics_export_prefixes, (coerce 0 to int)))
|
||||
error in <no location>: invalid Broker store handle (broker::store::{})
|
||||
error in <no location>: invalid Broker store handle (broker::store::{})
|
||||
error in <no location>: invalid Broker store handle (broker::store::{})
|
||||
error in <no location>: invalid Broker store handle (broker::store::{})
|
||||
error in <no location>: invalid Broker store handle (broker::store::{})
|
||||
error in <no location>: invalid Broker store handle (broker::store::{})
|
||||
error in <no location>: invalid Broker store handle (broker::store::{})
|
||||
received termination signal
|
||||
|
|
@ -0,0 +1,21 @@
|
|||
### BTest baseline data generated by btest-diff. Do not edit. Use "btest -U/-u" to update. Requires BTest >= 0.63.
|
||||
T
|
||||
F
|
||||
F
|
||||
F
|
||||
m1 keys result: [status=Broker::FAILURE, result=[data=<uninitialized>]]
|
||||
m2 keys result: [status=Broker::SUCCESS, result=[data=broker::data{{}}]]
|
||||
c2 keys result: [status=Broker::SUCCESS, result=[data=broker::data{{}}]]
|
||||
T
|
||||
F
|
||||
F
|
||||
F
|
||||
T
|
||||
T
|
||||
T
|
||||
T
|
||||
m1 keys result: [status=Broker::FAILURE, result=[data=<uninitialized>]]
|
||||
c1 keys result: [status=Broker::FAILURE, result=[data=<uninitialized>]]
|
||||
m2 keys result: [status=Broker::FAILURE, result=[data=<uninitialized>]]
|
||||
c2 keys result: [status=Broker::FAILURE, result=[data=<uninitialized>]]
|
||||
c1 timeout
|
||||
|
|
@ -0,0 +1 @@
|
|||
### BTest baseline data generated by btest-diff. Do not edit. Use "btest -U/-u" to update. Requires BTest >= 0.63.
|
||||
|
|
@ -1,7 +1,7 @@
|
|||
### BTest baseline data generated by btest-diff. Do not edit. Use "btest -U/-u" to update. Requires BTest >= 0.63.
|
||||
reporter_info|init test-info||XXXXXXXXXX.XXXXXX
|
||||
reporter_warning|init test-warning||XXXXXXXXXX.XXXXXX
|
||||
reporter_error|init test-error||XXXXXXXXXX.XXXXXX
|
||||
reporter_info|init test-info|<...>/main.zeek, lines 498-499|XXXXXXXXXX.XXXXXX
|
||||
reporter_warning|init test-warning|<...>/main.zeek, lines 498-499|XXXXXXXXXX.XXXXXX
|
||||
reporter_error|init test-error|<...>/main.zeek, lines 498-499|XXXXXXXXXX.XXXXXX
|
||||
reporter_info|done test-info||XXXXXXXXXX.XXXXXX
|
||||
reporter_warning|done test-warning||XXXXXXXXXX.XXXXXX
|
||||
reporter_error|done test-error||XXXXXXXXXX.XXXXXX
|
||||
|
|
|
|||
|
|
@ -2,9 +2,9 @@
|
|||
pre test-info
|
||||
warning: pre test-warning
|
||||
error: pre test-error
|
||||
init test-info
|
||||
warning: init test-warning
|
||||
error: init test-error
|
||||
<...>/main.zeek, lines 498-499: init test-info
|
||||
warning in <...>/main.zeek, lines 498-499: init test-warning
|
||||
error in <...>/main.zeek, lines 498-499: init test-error
|
||||
done test-info
|
||||
warning: done test-warning
|
||||
error: done test-error
|
||||
|
|
|
|||
|
|
@ -0,0 +1,15 @@
|
|||
### BTest baseline data generated by btest-diff. Do not edit. Use "btest -U/-u" to update. Requires BTest >= 0.63.
|
||||
### NOTE: This file has been sorted with diff-sort.
|
||||
[f(F)]
|
||||
[f(T)]
|
||||
[zeek_init()]
|
||||
f() done, no exception, T
|
||||
g() done, no exception, T
|
||||
received termination signal
|
||||
runtime error in compiled code: field value missing
|
||||
runtime error in compiled code: field value missing
|
||||
runtime error in compiled code: field value missing
|
||||
runtime error in compiled code: field value missing
|
||||
timeout
|
||||
timeout g(), F
|
||||
timeout g(), T
|
||||
|
|
@ -9,5 +9,5 @@ orig=/^?(.*PATTERN.*)$?/ (pattern) clone=/^?(.*PATTERN.*)$?/ (pattern) same_obje
|
|||
orig=2,5,3,4,1 (set[count]) clone=2,5,3,4,1 (set[count]) equal=T same_object=F (ok)
|
||||
orig=[1, 2, 3, 4, 5] (vector of count) clone=[1, 2, 3, 4, 5] (vector of count) equal=T same_object=F (ok)
|
||||
orig=a=va;b=vb (table[string] of string) clone=a=va;b=vb (table[string] of string) equal=T same_object=F (ok)
|
||||
orig=ENUMME (enum MyEnum) clone=ENUMME (enum MyEnum) equal=T same_object=F (FAIL1)
|
||||
orig=[s1=s1, s2=s2, i1=[a=a], i2=[a=a], donotset=<uninitialized>, def=5] (record { s1:string; s2:string; i1:record { a:string; }; i2:record { a:string; } &optional; donotset:record { a:string; } &optional; def:count &default=5, &optional; }) clone=[s1=s1, s2=s2, i1=[a=a], i2=[a=a], donotset=<uninitialized>, def=5] (record { s1:string; s2:string; i1:record { a:string; }; i2:record { a:string; } &optional; donotset:record { a:string; } &optional; def:count &default=5, &optional; }) equal=T same_object=F (ok)
|
||||
orig=ENUMME (MyEnum) clone=ENUMME (MyEnum) equal=T same_object=F (FAIL1)
|
||||
orig=[s1=s1, s2=s2, i1=[a=a], i2=[a=a], donotset=<uninitialized>, def=5] (TestRecord) clone=[s1=s1, s2=s2, i1=[a=a], i2=[a=a], donotset=<uninitialized>, def=5] (TestRecord) equal=T same_object=F (ok)
|
||||
|
|
|
|||
|
|
@ -1,6 +1,6 @@
|
|||
### BTest baseline data generated by btest-diff. Do not edit. Use "btest -U/-u" to update. Requires BTest >= 0.63.
|
||||
runtime error in compiled code: vector index assignment failed for invalid type 'myrec', value: [a=T, b=hi, c=<uninitialized>]
|
||||
runtime error in compiled code: vector index assignment failed for invalid type 'myrec', value: [a=T, b=hi, c=<uninitialized>]
|
||||
runtime error in compiled code: vector index assignment failed for invalid type 'myrec', value: [a=T, b=hi, c=<uninitialized>]
|
||||
runtime error in compiled code: vector index assignment failed for invalid type 'myrec', value: [a=T, b=hi, c=<uninitialized>]
|
||||
runtime error in compiled code: vector index assignment failed for invalid type 'myrec', value: [a=T, b=hi, c=<uninitialized>]
|
||||
runtime error in <...>/queue.zeek, line 152: vector index assignment failed for invalid type 'myrec', value: [a=T, b=hi, c=<uninitialized>], expression: Queue::ret[Queue::j], call stack:
|
||||
#0 Queue::get_vector([initialized=T, vals={[2] = test,[3] = [a=T, b=hi, c=<uninitialized>],[5] = 3,[0] = hello,[6] = jkl;,[4] = asdf,[1] = goodbye}, settings=[max_len=<uninitialized>], top=7, bottom=0, size=0], [hello, goodbye, test]) at <...>/main.zeek:498
|
||||
#1 zeek_init()
|
||||
|
||||
|
||||
|
|
|
|||
|
|
@ -1,11 +1,14 @@
|
|||
### BTest baseline data generated by btest-diff. Do not edit. Use "btest -U/-u" to update. Requires BTest >= 0.63.
|
||||
| Hook Some Info
|
||||
| Hook error An Error
|
||||
| Hook error An Error that does not show up in the log
|
||||
Reporter::Hook - Exercise Reporter Hook (dynamic, version 1.0.0)
|
||||
Implements Reporter (priority 0)
|
||||
|
||||
| Hook Some Info <...>/main.zeek, lines 498-499
|
||||
| Hook error An Error <...>/main.zeek, lines 498-499
|
||||
| Hook error An Error that does not show up in the log <...>/main.zeek, lines 498-499
|
||||
| Hook runtime error in compiled code field value missing
|
||||
| Hook warning A warning
|
||||
Some Info
|
||||
error: An Error
|
||||
error: An Error that does not show up in the log
|
||||
| Hook warning A warning <...>/main.zeek, lines 498-499
|
||||
<...>/main.zeek, lines 498-499: Some Info
|
||||
error in <...>/main.zeek, lines 498-499: An Error
|
||||
error in <...>/main.zeek, lines 498-499: An Error that does not show up in the log
|
||||
runtime error in compiled code: field value missing
|
||||
warning: A warning
|
||||
warning in <...>/main.zeek, lines 498-499: A warning
|
||||
|
|
|
|||
|
|
@ -7,8 +7,8 @@
|
|||
#open XXXX-XX-XX-XX-XX-XX
|
||||
#fields ts level message location
|
||||
#types time enum string string
|
||||
XXXXXXXXXX.XXXXXX Reporter::INFO Some Info (empty)
|
||||
XXXXXXXXXX.XXXXXX Reporter::WARNING A warning (empty)
|
||||
XXXXXXXXXX.XXXXXX Reporter::ERROR An Error (empty)
|
||||
XXXXXXXXXX.XXXXXX Reporter::INFO Some Info <...>/main.zeek, lines 498-499
|
||||
XXXXXXXXXX.XXXXXX Reporter::WARNING A warning <...>/main.zeek, lines 498-499
|
||||
XXXXXXXXXX.XXXXXX Reporter::ERROR An Error <...>/main.zeek, lines 498-499
|
||||
XXXXXXXXXX.XXXXXX Reporter::ERROR field value missing (empty)
|
||||
#close XXXX-XX-XX-XX-XX-XX
|
||||
|
|
|
|||
|
|
@ -0,0 +1,7 @@
|
|||
### BTest baseline data generated by btest-diff. Do not edit. Use "btest -U/-u" to update. Requires BTest >= 0.63.
|
||||
all, AllAnalyzers::ANALYZER_ANALYZER_DNS
|
||||
analyzer, Analyzer::ANALYZER_DNS
|
||||
all, AllAnalyzers::PACKETANALYZER_ANALYZER_UDP
|
||||
packet analyzer, PacketAnalyzer::ANALYZER_UDP
|
||||
all, AllAnalyzers::FILES_ANALYZER_X509
|
||||
file analyzer, Files::ANALYZER_X509
|
||||
|
|
@ -1,2 +1,3 @@
|
|||
### BTest baseline data generated by btest-diff. Do not edit. Use "btest -U/-u" to update. Requires BTest >= 0.63.
|
||||
error: file ID asdf not a known file
|
||||
warning: non-void function returning without a value: Files::lookup_file
|
||||
|
|
|
|||
|
|
@ -0,0 +1,7 @@
|
|||
### BTest baseline data generated by btest-diff. Do not edit. Use "btest -U/-u" to update. Requires BTest >= 0.63.
|
||||
warning in /Users/vern/warehouse/zeek/zeek-master.29Jul22/testing/btest/.tmp/language.vector-deprecated/vector-deprecated.zeek, line 18: mixing vector and scalar operands is deprecated (vector) (string)
|
||||
warning in /Users/vern/warehouse/zeek/zeek-master.29Jul22/testing/btest/.tmp/language.vector-deprecated/vector-deprecated.zeek, line 21: mixing vector and scalar operands is deprecated (string) (vector)
|
||||
warning in /Users/vern/warehouse/zeek/zeek-master.29Jul22/testing/btest/.tmp/language.vector-deprecated/vector-deprecated.zeek, line 24: mixing vector and scalar operands is deprecated (string) (vector)
|
||||
error: deprecated mixed vector/scalar operation not supported for ZAM compiling
|
||||
error: deprecated mixed vector/scalar operation not supported for ZAM compiling
|
||||
error: deprecated mixed vector/scalar operation not supported for ZAM compiling
|
||||
|
|
@ -0,0 +1,4 @@
|
|||
### BTest baseline data generated by btest-diff. Do not edit. Use "btest -U/-u" to update. Requires BTest >= 0.63.
|
||||
+ operator [string] (PASS)
|
||||
== operator [string] (PASS)
|
||||
== operator [string] (PASS)
|
||||
16
testing/btest/Baseline/bifs.disable_analyzer-hook/out
Normal file
16
testing/btest/Baseline/bifs.disable_analyzer-hook/out
Normal file
|
|
@ -0,0 +1,16 @@
|
|||
### BTest baseline data generated by btest-diff. Do not edit. Use "btest -U/-u" to update. Requires BTest >= 0.63.
|
||||
proto confirm, Analyzer::ANALYZER_HTTP
|
||||
http_request, GET, /style/enhanced.css
|
||||
preventing disable_analyzer, [orig_h=192.168.1.104, orig_p=1673/tcp, resp_h=63.245.209.11, resp_p=80/tcp], Analyzer::ANALYZER_HTTP, 3, 1
|
||||
F
|
||||
http_reply, 200
|
||||
http_request, GET, /script/urchin.js
|
||||
preventing disable_analyzer, [orig_h=192.168.1.104, orig_p=1673/tcp, resp_h=63.245.209.11, resp_p=80/tcp], Analyzer::ANALYZER_HTTP, 3, 3
|
||||
F
|
||||
http_reply, 200
|
||||
http_request, GET, /images/template/screen/bullet_utility.png
|
||||
allowing disable_analyzer, [orig_h=192.168.1.104, orig_p=1673/tcp, resp_h=63.245.209.11, resp_p=80/tcp], Analyzer::ANALYZER_HTTP, 3, 5
|
||||
T
|
||||
total http messages, {
|
||||
[[orig_h=192.168.1.104, orig_p=1673/tcp, resp_h=63.245.209.11, resp_p=80/tcp]] = 5
|
||||
}
|
||||
13
testing/btest/Baseline/bifs.table_keys/out
Normal file
13
testing/btest/Baseline/bifs.table_keys/out
Normal file
|
|
@ -0,0 +1,13 @@
|
|||
### BTest baseline data generated by btest-diff. Do not edit. Use "btest -U/-u" to update. Requires BTest >= 0.63.
|
||||
{
|
||||
http,
|
||||
https
|
||||
}
|
||||
{
|
||||
[a=<uninitialized>, b=7],
|
||||
[a=10, b=5]
|
||||
}
|
||||
{
|
||||
[1/tcp, test, T] ,
|
||||
[2/tcp, example, F]
|
||||
}
|
||||
3
testing/btest/Baseline/bifs.table_values/out
Normal file
3
testing/btest/Baseline/bifs.table_values/out
Normal file
|
|
@ -0,0 +1,3 @@
|
|||
### BTest baseline data generated by btest-diff. Do not edit. Use "btest -U/-u" to update. Requires BTest >= 0.63.
|
||||
[example, test]
|
||||
[80/tcp, http, 443/tcp, https, 21/tcp, ftp, 23/tcp, telnet]
|
||||
7
testing/btest/Baseline/core.option-zeek-done/.stdout
Normal file
7
testing/btest/Baseline/core.option-zeek-done/.stdout
Normal file
|
|
@ -0,0 +1,7 @@
|
|||
### BTest baseline data generated by btest-diff. Do not edit. Use "btest -U/-u" to update. Requires BTest >= 0.63.
|
||||
Initial value, T
|
||||
Value of testbool changed from T to F (zeek_is_terminating=F)
|
||||
Next value, F
|
||||
Next changed, T
|
||||
Final value, F
|
||||
Final changed, F
|
||||
|
|
@ -0,0 +1,2 @@
|
|||
### BTest baseline data generated by btest-diff. Do not edit. Use "btest -U/-u" to update. Requires BTest >= 0.63.
|
||||
fatal error in <...>/record-field-redef-errors.zeek, line 1: field "no_such_field" not in record "M::Info"
|
||||
|
|
@ -0,0 +1,2 @@
|
|||
### BTest baseline data generated by btest-diff. Do not edit. Use "btest -U/-u" to update. Requires BTest >= 0.63.
|
||||
fatal error in <...>/record-field-redef-errors.zeek, line 1: field "no_such_field" not in record "M::Info"
|
||||
|
|
@ -0,0 +1,2 @@
|
|||
### BTest baseline data generated by btest-diff. Do not edit. Use "btest -U/-u" to update. Requires BTest >= 0.63.
|
||||
error in <...>/record-field-redef-errors.zeek, line 2: syntax error, at or near "&"
|
||||
|
|
@ -0,0 +1,2 @@
|
|||
### BTest baseline data generated by btest-diff. Do not edit. Use "btest -U/-u" to update. Requires BTest >= 0.63.
|
||||
error in <...>/record-field-redef-errors.zeek, line 2: syntax error, at or near ";"
|
||||
|
|
@ -0,0 +1,2 @@
|
|||
### BTest baseline data generated by btest-diff. Do not edit. Use "btest -U/-u" to update. Requires BTest >= 0.63.
|
||||
error in <...>/record-field-redef-errors.zeek, line 2: syntax error, at or near "["
|
||||
|
|
@ -0,0 +1,2 @@
|
|||
### BTest baseline data generated by btest-diff. Do not edit. Use "btest -U/-u" to update. Requires BTest >= 0.63.
|
||||
fatal error in <...>/record-field-redef-errors.zeek, line 2: Can only redef "&log" attributes of record fields
|
||||
|
|
@ -0,0 +1,2 @@
|
|||
### BTest baseline data generated by btest-diff. Do not edit. Use "btest -U/-u" to update. Requires BTest >= 0.63.
|
||||
fatal error in <...>/record-field-redef-errors.zeek, line 2: Can only redef "&log" attributes of record fields
|
||||
|
|
@ -0,0 +1,2 @@
|
|||
### BTest baseline data generated by btest-diff. Do not edit. Use "btest -U/-u" to update. Requires BTest >= 0.63.
|
||||
fatal error in <...>/record-field-redef-errors.zeek, line 2: Can only redef "&log" attributes of record fields
|
||||
|
|
@ -0,0 +1,2 @@
|
|||
### BTest baseline data generated by btest-diff. Do not edit. Use "btest -U/-u" to update. Requires BTest >= 0.63.
|
||||
fatal error in <...>/record-field-redef-errors.zeek, line 2: identifier "M::ErrCode" has type "enum", expected "record"
|
||||
|
|
@ -0,0 +1,2 @@
|
|||
### BTest baseline data generated by btest-diff. Do not edit. Use "btest -U/-u" to update. Requires BTest >= 0.63.
|
||||
fatal error in <...>/record-field-redef-errors.zeek, line 1: unknown record identifier "M::Unknown"
|
||||
|
|
@ -0,0 +1,2 @@
|
|||
### BTest baseline data generated by btest-diff. Do not edit. Use "btest -U/-u" to update. Requires BTest >= 0.63.
|
||||
fatal error in <...>/record-field-redef-errors.zeek, line 1: unknown record identifier "M::Unknown"
|
||||
|
|
@ -0,0 +1,2 @@
|
|||
### BTest baseline data generated by btest-diff. Do not edit. Use "btest -U/-u" to update. Requires BTest >= 0.63.
|
||||
error in <...>/record-field-redef-errors.zeek, line 6: syntax error, at or near "&log"
|
||||
Some files were not shown because too many files have changed in this diff Show more
Loading…
Add table
Add a link
Reference in a new issue