Merge branch 'zeek:master' into master

CHANGES

@@ -1,3 +1,266 @@
+5.1.0-dev.386 | 2022-08-11 11:56:55 -0700
+
+  * Add a field to Modbus/TCP log to indicate the Modbus PDU type (Michael Torres)
+
+    Add the `pdu_type` field to Modbus over TCP logs to indicate whether the Modbus
+    message was a request or a response. Due to the client/server nature of Modbus
+    over TCP/IP, all messages from the TCP session originator are requests, while
+    all messages from the TCP session responder are responses.
+
+    Adding this information to the default log surfaces protocol metadata in a way
+    that doesn't require users to understand the Modbus over TCP protocol.
+
+  * Add modbus transaction and unit ids to logs (Michael Torres)
+
+    Add transaction IDs and unit IDs to default modbus over TCP/IP logs.
+    Update the relevant testing baselines to account for the extra fields.
+
+  * Enable modbus logging for requests (Michael Torres)
+
+5.1.0-dev.382 | 2022-08-11 10:41:08 -0700
+
+  * UPDATED: improving email address splitting for common comma case (TheAvgJojo)
+
+5.1.0-dev.380 | 2022-08-11 10:30:33 -0700
+
+  * Fix a crash related to a broken IPv6 chain (Tim Wojtulewicz)
+
+  * Add a couple of ICMP files to corpus for packet-fuzzer (Tim Wojtulewicz, Corelight)
+
+  * Trick event handlers into returning that they exist during fuzzing (Tim Wojtulewicz, Corelight)
+
+  * Add http, ftp, imap, and smtp fuzzers and corpora (Tim Wojtulewicz, Corelight)
+
+  * Add section to fuzzer README about generating corpus from pcaps (Tim Wojtulewicz, Corelight)
+
+  * Rename fuzzers/README to README.rst so github renders it (Tim Wojtulewicz, Corelight)
+
+5.1.0-dev.373 | 2022-08-11 08:49:22 -0700
+
+  * Add table_keys function (AmazingPP)
+
+  * Add table_values function (AmazingPP)
+
+5.1.0-dev.368 | 2022-08-11 16:01:32 +0200
+
+  * GH-1678: Introduce global disabling_analyzer() hook to veto disable_analyzer() (Arne Welzel, Corelight)
+
+    This hook can be used to coordinate disabling an analyzer for a
+    given connection. The contract is simple: Any script can veto a
+    disable_analyzer() call by breaking from this hook. The decision
+    is local to the script taking into account any state attached to
+    the connection object or script specific state stored elsewhere. A
+    script breaking from the hook takes over the responsibility to
+    call disable_analyzer() at a later point when it finds the
+    condition due to which it vetoed fulfilled (which may be never).
+
+    Signature:
+
+        disabling_analyzer: hook(c: connection, atype: AllAnalyzers::Tag, aid: count);
+
+    Example use-cases are keeping the SSL analyzer enabled for
+    finger-printing until a certain amount of bytes or packets have
+    been transferred or similarly the connection duration exceed a
+    certain threshold.
+
+    Other example use-cases might be keeping analyzers for SSH, RDP or SSL
+    enabled for connections from specific subnets.
+
+  * ssl: Only delete `c$ssl$analyzer_id` when disabling the analyzer
+    was successful (Arne Welzel, Corelight)
+
+5.1.0-dev.364 | 2022-08-11 11:59:00 +0200
+
+  * GH-2000: Support redef'ing the &log attribute of record fields (Arne Welzel, Corelight)
+
+    Add new syntax for adding and removing attributes from record fields:
+
+        redef RecordType$field_name += { &log };
+        redef RecordType$field_name -= { &log };
+
+    For now this only allowed for the &log attribute as the semantics are clear.
+    For &default and &optional the semantics aren't obvious and no use-cases have
+    been identified where those would make sense to change.
+
+    This enables a mechanism to add potentially interesting fields to the typical
+    Info records in base scripts, but letting users opt-into actually including
+    them into their log. At the same time, users that find specific fields in a
+    standard log uninteresting can opt-out without using `Log::Filter$exclude`
+    which can be difficult to use correctly. Patching or forking external packages
+    to remove columns from a log can also be avoided with this mechanism.
+
+    Closes #2000.
+
+  * GH-2262: telemetry: Remove singleton BIFs and the C++ pieces (Arne Welzel, Corelight)
+
+    The low-level singleton Telemetry BIFs have been removed with the that there
+    haven't been any users. Singleton metrics can be instantiated by providing
+    an empty label vector instead and aren't in any way a special concept.
+
+    Closes #2262.
+
+5.1.0-dev.360 | 2022-08-11 11:52:26 +0200
+
+  * telemetry: Switch to histogram family bucket and bound accessors
+    (Arne Welzel, Corelight)
+
+5.1.0-dev.357 | 2022-08-10 14:16:33 -0700
+
+  * Management framework: minor bugfixes and tweaks (Christian Kreibich, Corelight)
+
+    - bump cluster testsuite
+    - log node set in dispatch requests cleanly
+    - log additional node events
+    - upon deployment, make agent log multiple node results
+    - fix early return condition for get-id-value
+
+5.1.0-dev.351 | 2022-08-09 09:50:13 -0700
+
+  * Fix module-scoped type definitions that conflict with existing global ones (Tim Wojtulewicz, Corelight)
+
+5.1.0-dev.349 | 2022-08-08 11:09:04 -0700
+
+  * lower priority for change handlers (Yacin Nadji, Corelight)
+
+  * split update_zones_regex into two functions (Yacin Nadji, Corelight)
+
+5.1.0-dev.344 | 2022-08-08 11:03:24 -0700
+
+  * Squelch the zeekygen warnings for command line (AmazingPP)
+
+5.1.0-dev.341 | 2022-08-05 16:21:13 +0000
+
+  * Added test case with back-to-back notices (Annie Bryan)
+
+  * Fix notice suppression atomicity bug (Annie Bryan)
+
+5.1.0-dev.337 | 2022-08-05 16:19:59 +0000
+
+  * GH-2034: Store module names and use them in lookups for ifdef (Tim Wojtulewicz, Corelight)
+
+5.1.0-dev.335 | 2022-08-05 16:50:18 +0200
+
+  * Introduce telemetry framework (Arne Welzel, Corelight)
+
+    Adds base/frameworks/telemetry with wrappers around telemetry.bif
+    and updates telemetry/Manager to support collecting metrics from
+    script land.
+
+    Adds policy/frameworks/telemetry/log for logging of metrics data
+    into a new telemetry.log and telemetry_histogram.log and add into
+    local.zeek by default.
+
+5.1.0-dev.331 | 2022-08-03 10:27:28 -0700
+
+  * input/config: Use thread-safe Fmt() rather than util::fmt() for regex formatting (Arne Welzel, Corelight)
+
+    Calling util::fmt() from DoUpdate() of a thread is not safe as it is
+    using a statically allocated buffer and other threads or the main
+    thread may concurrently modify this buffer.
+
+    This was found by observing the scripts.base.frameworks.config.several-files
+    failing once in a blue moon (1/250 sometimes 1/1000 runs) with messages like
+    "Failed to compile regex: Parenthesis is not closed.":
+
+        scripts.base.frameworks.config.several-files ...
+          > btest-bg-run zeek zeek -b %INPUT
+          > btest-bg-wait 10
+        ... scripts.base.frameworks.config.several-files failed
+          % 'btest-bg-wait 10' failed unexpectedly (exit code 1)
+          % cat .stderr
+          The following processes did not terminate:
+          zeek -b /home/awelzel/corelight-oss/zeek/testing/btest/.tmp/scripts.base.frameworks.config.several-files/several-files.zeek
+          -----------
+          <<< [3667265] zeek -b /home/awelzel/corelight-oss/zeek/testing/btest/.tmp/scripts.base.frameworks.config.several-files/several-files.zeek
+          error: ../configfile1/Input::READER_CONFIG: Failed to compile regex: Parenthesis is not closed.
+          received termination signal
+          >>>
+
+5.1.0-dev.329 | 2022-08-03 10:26:45 -0700
+
+  * testing/missing-enum-value: redef exit_only_after_terminate=T (Arne Welzel, Corelight)
+
+    Seems this was the intention all along as the corresponding terminate()
+    call is there.
+
+  * option.bif: Short-circuit option changes when terminating (Arne Welzel, Corelight)
+
+    Due to the asynchronous behavior of the input framework and broker
+    communication, change handlers were previously called even after
+    zeek_done() event processing completed and also broker shutdown.
+
+    Accessing broker store handles within change handlers this late
+    triggered invalid Broker store handle messages:
+
+        error in ././my_option_store.zeek, line 13: invalid Broker store handle (Broker::put(Test::store, to_any_coercemy_option, to_any_coerceTest::new_value, 0 secs) and broker::store::{})
+
+    Fixes #2010
+
+5.1.0-dev.326 | 2022-08-03 09:56:37 -0700
+
+  * updates for gen-C++ maintenance, including skipping some inappropriate tests (Vern Paxson, Corelight)
+
+  * fix for profiling "when" statements (Vern Paxson, Corelight)
+
+  * gen-C++ support for vector bit-shift operations (Vern Paxson, Corelight)
+
+  * corrected wording in some btest comments (Vern Paxson, Corelight)
+
+  * make gen-C++ maintenance scripts directly executable (Vern Paxson, Corelight)
+
+  * ZAM support for bit-shifting (Vern Paxson, Corelight)
+
+  * don't allow deprecated-style mixing of vectors and scaling for shifting (Vern Paxson, Corelight)
+    leverage restrictions placed on shifting (RHS is always unsigned)
+    split deprecated vector operations into separate test, with separate ZAM baseline
+
+  * ZAM fix for vector "in" operator (Vern Paxson, Corelight)
+
+  * ensure that language tests pay attention to .stderr (Vern Paxson, Corelight)
+
+  * fix vector tests, including checking for errors (Vern Paxson, Corelight)
+
+5.1.0-dev.312 | 2022-08-02 12:37:51 -0700
+
+  * Update plugins.hooks baseline with new DHCP options (peter.cullen, Corelight)
+
+  * Prevent large dhcp log entries (Peter Cullen, Corelight)
+
+    A flood of DHCP traffic can result if very large log entries consisting
+    of many uids and/or msg_types. Such large log entries can disrupt a SIEM
+    ingestion pipeline. This change forcing a log entry to be written when
+    the number of uids or the number of msg_Types exceed a certain value.
+    The values are treated as options for easy configuration.
+
+5.1.0-dev.309 | 2022-08-02 11:33:22 -0700
+
+  * Add support for /s modifier to RE matcher and parser (Tim Wojtulewicz, Corelight)
+
+  * Code cleanup in RE_Matcher code (Tim Wojtulewicz, Corelight)
+
+    - Use std::string in Specific_RE_Matcher instead of char*
+    - Change a couple of ints-as-bools to bools
+
+  * Add basic unit tests for RE_Matcher (Tim Wojtulewicz, Corelight)
+
+  * Add /s modifier to parser for patterns (Tim Wojtulewicz, Corelight)
+
+  * Update gen-zam submodule [nomail] (Tim Wojtulewicz, Corelight)
+
+5.1.0-dev.303 | 2022-08-01 09:56:45 -0700
+
+  * GH-1344: Give better warning when using a type that doesn't exist (Tim Wojtulewicz, Corelight)
+
+5.1.0-dev.301 | 2022-07-29 12:10:20 -0700
+
+  * Add btest for vector bit-shift operators (Tim Wojtulewicz, Corelight)
+
+  * Handle error cases for bit-shift operators more cleanly (Tim Wojtulewicz, Corelight)
+
+5.1.0-dev.298 | 2022-07-29 12:00:57 -0700
+
+  * Update 3rdparty submodule to get patricia reformat (Tim Wojtulewicz, Corelight)
+
 5.1.0-dev.295 | 2022-07-28 11:21:59 -0700
 
   * Re-enable sending coverage data to Coveralls via Cirrus (Tim Wojtulewicz, Corelight)

NEWS

@@ -21,9 +21,52 @@ Breaking Changes
 - The Dictionary and PDict classes are now C++ templates. This may cause
   plugin/package builds to fail due to needing to modify uses of them to match.
 
+- The low-level singleton Telemetry BIFs have been removed with the assumption that
+  there haven't been any users. Singleton metrics can be instantiated by providing
+  an empty label vector instead and aren't in any way a special concept.
+
 New Functionality
 -----------------
 
+- Added support for the /s regular expression modifier. Using this modifier in
+  patterns in Zeek scripts will cause the '.' character to also match newline
+  characters.
+
+- Added a new telemetry framework for providing high-level access to Zeek's
+  metric subsystem. This framework allows script writers to use different
+  metric types (counters, gauges and histograms) for tracking metrics without
+  using lower-level BiFs from ``telemetry.bif``. Additionally, metrics can
+  now be accessed from script land using ``Telemetry::collect_metrics() and
+  ``Telemetry::collect_histogram_metrics()``.
+
+  The framework is located in ``base/frameworks/telemetry``.
+
+  In addition to the Prometheus endpoint for metrics export that has existed
+  since Zeek 4.1, two new log streams, ``telemetry.log`` and ``telemetry_histogram.log``,
+  can be enabled by loading ``policy/frameworks/telemetry/log``. This policy
+  script is included in ``local.zeek`` by default.
+
+  For further details on the framework and examples, please refer to the
+  Zeek documentation.
+
+- Allow redef'ing the ``&log`` attribute of record fields:
+
+	redef Notice::Info$email_dest -= { &log };
+
+  While the syntax allows for any attribute, only ``&log`` is supported. The
+  semantics for other record field attributes are not easy to grasp and there
+  were no obvious use-cases identified.
+
+- Introduce a global ``disabling_analyzer()`` hook to allow vetoing calls
+  to ``disable_analyzer()``.
+
+  The contract is simple: Any script can veto a ``disable_analyzer()`` call by
+  breaking from this hook. The decision is local to the script taking into
+  account any state attached to the connection or state stored elsewhere.
+  A script breaking from the hook takes over responsibility to call
+  ``disable_analyzer()`` at a later point when it finds the condition due
+  to which it vetoed fulfilled (which may be never).
+
 Changed Functionality
 ---------------------
 
@@ -41,6 +84,10 @@ Changed Functionality
 - The default logging directory is now set globally across all log
   writers through ``Log::default_logdir``.
 
+- Calling `Option::set()` when Zeek is terminating is now a noop and returns `F`.
+  This prevents callbacks into script-land through change handlers when parts
+  of the environment have already been torn down.
+
 Deprecated Functionality
 ------------------------
 

VERSION

@@ -1 +1 @@
-5.1.0-dev.295
+5.1.0-dev.386

auxil/broker

@@ -1 +1 @@
-Subproject commit dc7b266e8289472096828ad51547237f49a91841
+Subproject commit 8d03476847411b0ab7b9c47ea9eaf7b1535d08c2

auxil/gen-zam

@@ -1 +1 @@
-Subproject commit 5b0855905f7c7937a38cfe3141544fb015192b5e
+Subproject commit d4e0fc662a8f55632c8ec84550d8e4a80460d35d

doc

@@ -1 +1 @@
-Subproject commit 4550b7fbe963d39f31d0c50f7d6764d4ec95091d
+Subproject commit 497bcfbcc547b78f5ceafeb3860b12a0bf6f2375

scripts/base/frameworks/notice/main.zeek

@@ -538,6 +538,7 @@ hook Notice::notice(n: Notice::Info) &priority=-5
 	     n$suppress_for != 0secs )
 		{
 		event Notice::begin_suppression(n$ts, n$suppress_for, n$note, n$identifier);
+		suppressing[n$note, n$identifier] = n$ts + n$suppress_for;
 @if ( Cluster::is_enabled() && Cluster::local_node_type() != Cluster::MANAGER )
 		event Notice::manager_begin_suppression(n$ts, n$suppress_for, n$note, n$identifier);
 @endif

scripts/base/frameworks/telemetry/__load__.zeek

@@ -0,0 +1 @@
+@load ./main

scripts/base/frameworks/telemetry/main.zeek

@@ -0,0 +1,609 @@
+##! Module for recording and querying metrics. This modules wraps
+##! the lower-level telemetry.bif functions.
+##!
+##! Metrics will be exposed through a Prometheus HTTP endpoint when
+##! enabled by setting :zeek:see:`Broker::metrics_port` or using the
+##! `BROKER_METRICS_PORT` environment variable.
+
+@load base/misc/version
+
+module Telemetry;
+
+export {
+	## Alias for a vector of label values.
+	type labels_vector: vector of string;
+
+	## Type that captures options used to create metrics.
+	type MetricOpts: record {
+		## The prefix (namespace) of the metric.
+		prefix: string;
+
+		## The human-readable name of the metric.
+		name: string;
+
+		## The unit of the metric. Use the pseudo-unit "1" if this is a unit-less metric.
+		unit: string;
+
+		## Documentation for this metric.
+		help_text: string;
+
+		## The label names (also called dimensions) of the metric. When
+		## instantiating or working with concrete metrics, corresponding
+		## label values have to be provided.
+		labels: vector of string &default=vector();
+
+		## Whether the metric represents something that is accumulating.
+		## Defaults to ``T`` for counters and ``F`` for gauges and
+		## histograms.
+		is_total: bool &optional;
+
+		## When creating a :zeek:see:`Telemetry::HistogramFamily`,
+		## describes the number and bounds of the individual buckets.
+		bounds: vector of double &optional;
+
+		## The same meaning as *bounds*, but as :zeek:type:`count`.
+		## Only set in the return value of
+		## :zeek:see:`Telemetry::collect_histogram_metrics`.
+		## for histograms when the underlying type is ``int64_t``,
+		## otherwise ignored.
+		count_bounds: vector of count &optional;
+
+		## Describes the underlying metric type.
+		## Only set in the return value of
+		## :zeek:see:`Telemetry::collect_metrics` or
+		## :zeek:see:`Telemetry::collect_histogram_metrics`,
+		## otherwise ignored.
+		metric_type: MetricType &optional;
+	};
+
+	## Type representing a family of counters with uninitialized label values.
+	##
+	## To create concrete :zeek:see:`Telemetry::Counter` instances, use
+	## :zeek:see:`Telemetry::counter_with`. To modify counters directly
+	## use :zeek:see:`Telemetry::counter_family_inc`.
+	type CounterFamily: record {
+		__family: opaque of dbl_counter_metric_family;
+		__labels: vector of string;
+	};
+
+	## Type representing a counter metric with initialized label values.
+	##
+	## Counter metrics only ever go up and reset when the process
+	## restarts. Use :zeek:see:`Telemetry::counter_inc` or
+	## :zeek:see:`Telemetry::counter_set` to modify counters.
+	## An example for a counter is the number of log writes
+	## per :zeek:see:`Log::Stream` or number connections broken down
+	## by protocol and service.
+	type Counter: record {
+		__metric: opaque of dbl_counter_metric;
+	};
+
+	## Register a counter family.
+	global register_counter_family: function(opts: MetricOpts): CounterFamily;
+
+	## Get a :zeek:see:`Telemetry::Counter` instance given family and label values.
+	global counter_with: function(cf: CounterFamily,
+	                              label_values: labels_vector &default=vector()): Counter;
+
+	## Increment a :zeek:see:`Telemetry::Counter` by `amount`.
+	## Using a negative `amount` is an error.
+	##
+	## c: The counter instance.
+	##
+	## amount: The amount by which to increment the counter.
+	##
+	## Returns: True if the counter was incremented successfully.
+	global counter_inc: function(c: Counter, amount: double &default=1.0): bool;
+
+	## Helper to set a :zeek:see:`Telemetry::Counter` to the given `value`.
+	## This can be useful for mirroring counter metrics in an
+	## :zeek:see:`Telemetry::sync` hook implementation.
+	## Setting a value that is less than the current value of the
+	## metric is an error and will be ignored.
+	##
+	## c: The counter instance.
+	##
+	## value: The value to set the counter to.
+	##
+	## Returns: True if the counter value was set successfully.
+	global counter_set: function(c: Counter, value: double): bool;
+
+	## Increment a :zeek:see:`Telemetry::Counter` through the :zeek:see:`Telemetry::CounterFamily`.
+	## This is a short-cut for :zeek:see:`Telemetry::counter_inc`.
+	## Using a negative amount is an error.
+	##
+	## cf: The counter family to use.
+	##
+	## label_values: The label values to use for the counter.
+	##
+	## amount: The amount by which to increment the counter.
+	##
+	## Returns: True if the counter was incremented successfully.
+	global counter_family_inc: function(cf: CounterFamily,
+	                                    label_values: labels_vector &default=vector(),
+	                                    amount: double &default=1.0): bool;
+
+	## Set a :zeek:see:`Telemetry::Counter` through the :zeek:see:`Telemetry::CounterFamily`.
+	## This is a short-cut for :zeek:see:`Telemetry::counter_set`.
+	## Setting a value that is less than the current value of the
+	## metric is an error and will be ignored.
+	##
+	## cf: The counter family to use.
+	##
+	## label_values: The label values to use for the counter.
+	##
+	## value: The value to set the counter to.
+	##
+	## Returns: True if the counter value was set successfully.
+	global counter_family_set: function(cf: CounterFamily,
+	                                    label_values: labels_vector,
+	                                    value: double): bool;
+
+	## Type representing a family of gauges with uninitialized label values.
+	##
+	## Create concrete :zeek:see:`Telemetry::Gauge` instances with
+	## :zeek:see:`Telemetry::gauge_with`, or use
+	## :zeek:see:`Telemetry::gauge_family_inc` or
+	## :zeek:see:`Telemetry::gauge_family_set` directly.
+	type GaugeFamily: record {
+		__family: opaque of dbl_gauge_metric_family;
+		__labels: vector of string;
+	};
+
+	## Type representing a gauge metric with initialized label values.
+	##
+	## Use :zeek:see:`Telemetry::gauge_inc`, :zeek:see:`Telemetry::gauge_dec`,
+	## or :zeek:see:`Telemetry::gauge_set` to modify the gauge.
+	## Example for gauges are process memory usage, table sizes
+	## or footprints of long-lived values as determined by
+	## :zeek:see:`val_footprint`.
+	type Gauge: record {
+		__metric: opaque of dbl_gauge_metric;
+	};
+
+	## Register a gauge family.
+	global register_gauge_family: function(opts: MetricOpts): GaugeFamily;
+
+
+	## Get a :zeek:see:`Telemetry::Gauge` instance given family and label values.
+	global gauge_with: function(gf: GaugeFamily,
+	                            label_values: labels_vector &default=vector()): Gauge;
+
+	## Increment a :zeek:see:`Telemetry::Gauge` by `amount`.
+	##
+	## g: The gauge instance.
+	##
+	## amount: The amount by which to increment the gauge.
+	##
+	## Returns: True if the gauge was incremented successfully.
+	global gauge_inc: function(g: Gauge, amount: double &default=1.0): bool;
+
+	## Decrement a :zeek:see:`Telemetry::Gauge` by `amount`.
+	##
+	## g: The gauge instance.
+	##
+	## amount: The amount by which to decrement the gauge.
+	##
+	## Returns: True if the gauge was incremented successfully.
+	global gauge_dec: function(g: Gauge, amount: double &default=1.0): bool;
+
+	## Helper to set a :zeek:see:`Telemetry::Gauge` to the given `value`.
+	##
+	## g: The gauge instance.
+	##
+	## value: The value to set the gauge to.
+	##
+	## Returns: True if the gauge value was set successfully.
+	global gauge_set: function(g: Gauge, value: double): bool;
+
+	## Increment a :zeek:see:`Telemetry::Gauge` by the given `amount` through
+	## the :zeek:see:`Telemetry::GaugeFamily`.
+	## This is a short-cut for :zeek:see:`Telemetry::gauge_inc`.
+	## Using a negative amount is an error.
+	##
+	## gf: The gauge family to use.
+	##
+	## label_values: The label values to use for the gauge.
+	##
+	## amount: The amount by which to increment the gauge.
+	##
+	## Returns: True if the gauge was incremented successfully.
+	global gauge_family_inc: function(gf: GaugeFamily,
+	                                  label_values: labels_vector &default=vector(),
+	                                  amount: double &default=1.0): bool;
+
+	## Decrement a :zeek:see:`Telemetry::Gauge` by the given `amount` through
+	## the :zeek:see:`Telemetry::GaugeFamily`.
+	## This is a short-cut for :zeek:see:`Telemetry::gauge_dec`.
+	##
+	## gf: The gauge family to use.
+	##
+	## label_values: The label values to use for the gauge.
+	##
+	## amount: The amount by which to increment the gauge.
+	##
+	## Returns: True if the gauge was incremented successfully.
+	global gauge_family_dec: function(gf: GaugeFamily,
+	                                  label_values: labels_vector &default=vector(),
+	                                  amount: double &default=1.0): bool;
+
+	## Set a :zeek:see:`Telemetry::Gauge` to the given `value` through
+	## the :zeek:see:`Telemetry::GaugeFamily`.
+	## This is a short-cut for :zeek:see:`Telemetry::gauge_set`.
+	##
+	## gf: The gauge family to use.
+	##
+	## label_values: The label values to use for the gauge.
+	##
+	## value: The value to set the gauge to.
+	##
+	## Returns: True if the gauge value was set successfully.
+	global gauge_family_set: function(g: GaugeFamily,
+	                                  label_values: labels_vector,
+	                                  value: double): bool;
+
+	## Type representing a family of histograms with uninitialized label values.
+	## Create concrete :zeek:see:`Telemetry::Histogram` instances with
+	## :zeek:see:`Telemetry::histogram_with` or use
+	## :zeek:see:`Telemetry::histogram_family_observe` directly.
+	type HistogramFamily: record {
+		__family: opaque of dbl_histogram_metric_family;
+		__labels: vector of string;
+	};
+
+	## Type representing a histogram metric with initialized label values.
+	## Use :zeek:see:`Telemetry::histogram_observe` to make observations.
+	type Histogram: record {
+		__metric: opaque of dbl_histogram_metric;
+	};
+
+	## Register a histogram family.
+	global register_histogram_family: function(opts: MetricOpts): HistogramFamily;
+
+	## Get a :zeek:see:`Telemetry::Histogram` instance given family and label values.
+	global histogram_with: function(hf: HistogramFamily,
+	                                label_values: labels_vector &default=vector()): Histogram;
+
+	## Observe a measurement for a :zeek:see:`Telemetry::Histogram`.
+	##
+	## h: The histogram instance.
+	##
+	## measurement: The value for this observations.
+	##
+	## Returns: True if measurement was observed successfully.
+	global histogram_observe: function(h: Histogram, measurement: double): bool;
+
+	## Observe a measurement for a :zeek:see:`Telemetry::Histogram` through
+	## the :zeek:see:`Telemetry::HistogramFamily`.
+	## This is a short-cut for :zeek:see:`Telemetry::histogram_observe`.
+	##
+	## hf: The histogram family to use.
+	##
+	## label_values: The label values to use for the histogram.
+	##
+	## measurement: The value for this observations.
+	##
+	## Returns: True if measurement was observed successfully.
+	global histogram_family_observe: function(hf: HistogramFamily,
+	                                          label_values: labels_vector,
+	                                          measurement: double): bool;
+
+	## Telemetry sync hook.
+	##
+	## This hook is invoked every :zeek:see:`Telemetry::sync_interval`
+	## for script writers to synchronize or mirror metrics with the
+	## telemetry subsystem. For example, when tracking table or value
+	## footprints with gauges, the value in question can be set on an actual
+	## :zeek:see:`Telemetry::Gauge` instance during execution of this hook.
+	##
+	## Implementations should be lightweight, this hook may be called
+	## multiple times per minute. The interval can increased by changing
+	## :zeek:see:`Telemetry::sync_interval` at the cost of delaying
+	## metric updates and thereby reducing granularity.
+	global sync: hook();
+
+	## Interval at which the :zeek:see:`Telemetry::sync` hook is invoked.
+	option sync_interval = 10sec;
+
+	## Type of elements returned by the :zeek:see:`Telemetry::collect_metrics` function.
+	type Metric: record {
+		## A :zeek:see:`Telemetry::MetricOpts` record describing this metric.
+		opts: MetricOpts;
+
+		## The label values associated with this metric, if any.
+		labels: vector of string;
+
+		## The value of gauge or counter cast to a double
+		## independent of the underlying data type.
+		## This value is set for all counter and gauge metrics,
+		## it is unset for histograms.
+		value: double &optional;
+
+		## The value of the underlying gauge or counter as a double
+		## if the underlying metric type uses ``int64_t``.
+		## Only counters and gauges created with the C++ API may
+		## have this value set.
+		count_value: count &optional;
+	};
+
+	## Type of elements returned by the :zeek:see:`Telemetry::collect_histogram_metrics` function.
+	type HistogramMetric: record {
+		## A :zeek:see:`Telemetry::MetricOpts` record describing this histogram.
+		opts: MetricOpts;
+
+		## The label values associated with this histogram, if any.
+		labels: vector of string;
+
+		## Individual counters for each of the buckets as
+		## described by the *bounds* field in *opts*;
+		values: vector of double;
+
+		## If the underlying data type of the histogram is ``int64_t``,
+		## this vector will hold the values as counts, otherwise it
+		## is unset. Only histograms created with the C++ API have
+		## may have this value set.
+		count_values: vector of count &optional;
+
+		## The number of observations made for this histogram.
+		observations: double;
+
+		## The sum of all observations for this histogram.
+		sum: double;
+
+		## If the underlying data type of the histogram is ``int64_t``,
+		## the number of observations as :zeek:type:`count`, otherwise
+		## unset.
+		count_observations: count &optional;
+
+		## If the underlying data type of the histogram is ``int64_t``,
+		## the sum of all observations as :zeek:type:`count`, otherwise
+		## unset.
+		count_sum: count &optional;
+	};
+
+	## Collect all counter and gauge metrics matching the given *name* and *prefix*.
+	##
+	## For histogram metrics, use the :zeek:see:`Telemetry::collect_histogram_metrics`.
+	##
+	## The *prefix* and *name* parameters support globbing. By default,
+	## all counters and gauges are returned.
+	global collect_metrics: function(prefix: string &default="*",
+	                                 name: string &default="*"): vector of Metric;
+
+	## Collect all histograms and their observations matching the given
+	## *prefix* and *name*.
+	##
+	## The *prefix* and *name* parameters support globbing. By default,
+	## all histogram metrics are returned.
+	global collect_histogram_metrics: function(prefix: string &default="*",
+	                                           name: string &default="*"): vector of HistogramMetric;
+}
+
+## Internal helper to create the labels table.
+function make_labels(keys: vector of string, values: labels_vector): table[string] of string
+	{
+	local labels: table[string] of string;
+	for ( i in keys )
+		labels[keys[i]] = values[i];
+
+	return labels;
+	}
+
+function register_counter_family(opts: MetricOpts): CounterFamily
+	{
+	local f = Telemetry::__dbl_counter_family(
+		opts$prefix,
+		opts$name,
+		opts$labels,
+		opts$help_text,
+		opts$unit,
+		opts?$is_total ? opts$is_total : T
+	);
+	return CounterFamily($__family=f, $__labels=opts$labels);
+	}
+
+# Fallback Counter returned when there are issues with the labels.
+global error_counter_cf = register_counter_family([
+	$prefix="zeek",
+	$name="telemetry_counter_usage_error",
+	$unit="1",
+	$help_text="This counter is returned when label usage for counters is wrong. Check reporter.log if non-zero."
+]);
+
+function counter_with(cf: CounterFamily, label_values: labels_vector): Counter
+	{
+	if ( |cf$__labels| != |label_values| )
+		{
+		Reporter::error(fmt("Invalid label values expected %s, have %s", |cf$__labels|, |label_values|));
+		return counter_with(error_counter_cf);
+		}
+
+	local labels = make_labels(cf$__labels, label_values);
+	local m = Telemetry::__dbl_counter_metric_get_or_add(cf$__family, labels);
+	return Counter($__metric=m);
+	}
+
+function counter_inc(c: Counter, amount: double): bool
+	{
+	return Telemetry::__dbl_counter_inc(c$__metric, amount);
+	}
+
+function counter_set(c: Counter, value: double): bool
+	{
+	local cur_value: double = Telemetry::__dbl_counter_value(c$__metric);
+	if (value < cur_value)
+		{
+		Reporter::error(fmt("Attempted to set lower counter value=%s cur_value=%s", value, cur_value));
+		return F;
+		}
+	return Telemetry::__dbl_counter_inc(c$__metric, value - cur_value);
+	}
+
+function counter_family_inc(cf: CounterFamily, label_values: labels_vector, amount: double): bool
+	{
+	return counter_inc(counter_with(cf, label_values), amount);
+	}
+
+function counter_family_set(cf: CounterFamily, label_values: labels_vector, value: double): bool
+	{
+	return counter_set(counter_with(cf, label_values), value);
+	}
+
+function register_gauge_family(opts: MetricOpts): GaugeFamily
+	{
+	local f = Telemetry::__dbl_gauge_family(
+		opts$prefix,
+		opts$name,
+		opts$labels,
+		opts$help_text,
+		opts$unit,
+		opts?$is_total ? opts$is_total : F
+	);
+	return GaugeFamily($__family=f, $__labels=opts$labels);
+	}
+
+# Fallback Gauge returned when there are issues with the label usage.
+global error_gauge_cf = register_gauge_family([
+	$prefix="zeek",
+	$name="telemetry_gauge_usage_error",
+	$unit="1",
+	$help_text="This gauge is returned when label usage for gauges is wrong. Check reporter.log if non-zero."
+]);
+
+function gauge_with(gf: GaugeFamily, label_values: labels_vector): Gauge
+	{
+	if ( |gf$__labels| != |label_values| )
+		{
+		Reporter::error(fmt("Invalid label values expected %s, have %s", |gf$__labels|, |label_values|));
+		return gauge_with(error_gauge_cf);
+		}
+	local labels = make_labels(gf$__labels, label_values);
+	local m = Telemetry::__dbl_gauge_metric_get_or_add(gf$__family, labels);
+	return Gauge($__metric=m);
+	}
+
+function gauge_inc(g: Gauge, amount: double &default=1.0): bool
+	{
+	return Telemetry::__dbl_gauge_inc(g$__metric, amount);
+	}
+
+function gauge_dec(g: Gauge, amount: double &default=1.0): bool
+	{
+	return Telemetry::__dbl_gauge_dec(g$__metric, amount);
+	}
+
+function gauge_set(g: Gauge, value: double): bool
+	{
+	# Telemetry currently does not implement __dbl_gauge_set(), do
+	# it by hand here.
+	local cur_value: double = Telemetry::__dbl_gauge_value(g$__metric);
+	if (value > cur_value)
+		return Telemetry::__dbl_gauge_inc(g$__metric, value - cur_value);
+
+	return Telemetry::__dbl_gauge_dec(g$__metric, cur_value - value);
+	}
+
+function gauge_family_inc(gf: GaugeFamily, label_values: labels_vector, value: double): bool
+	{
+	return gauge_inc(gauge_with(gf, label_values), value);
+	}
+
+function gauge_family_dec(gf: GaugeFamily, label_values: labels_vector, value: double): bool
+	{
+	return gauge_dec(gauge_with(gf, label_values), value);
+	}
+
+function gauge_family_set(gf: GaugeFamily, label_values: labels_vector, value: double): bool
+	{
+	return gauge_set(gauge_with(gf, label_values), value);
+	}
+
+function register_histogram_family(opts: MetricOpts): HistogramFamily
+	{
+	local f = Telemetry::__dbl_histogram_family(
+		opts$prefix,
+		opts$name,
+		opts$labels,
+		opts$bounds,
+		opts$help_text,
+		opts$unit,
+		opts?$is_total ? opts$is_total : F
+	);
+	return HistogramFamily($__family=f, $__labels=opts$labels);
+	}
+
+# Fallback Histogram when there are issues with the labels.
+global error_histogram_hf = register_histogram_family([
+	$prefix="zeek",
+	$name="telemetry_histogram_usage_error",
+	$unit="1",
+	$help_text="This histogram is returned when label usage for histograms is wrong. Check reporter.log if non-zero.",
+	$bounds=vector(1.0)
+]);
+
+function histogram_with(hf: HistogramFamily, label_values: labels_vector): Histogram
+	{
+	if ( |hf$__labels| != |label_values| )
+		{
+		Reporter::error(fmt("Invalid label values expected %s, have %s", |hf$__labels|, |label_values|));
+		return histogram_with(error_histogram_hf);
+		}
+
+	local labels = make_labels(hf$__labels, label_values);
+	local m = Telemetry::__dbl_histogram_metric_get_or_add(hf$__family, labels);
+	return Histogram($__metric=m);
+	}
+
+function histogram_observe(h: Histogram, measurement: double): bool
+	{
+	return Telemetry::__dbl_histogram_observe(h$__metric, measurement);
+	}
+
+function histogram_family_observe(hf: HistogramFamily,  label_values: labels_vector, measurement: double): bool
+	{
+	return histogram_observe(histogram_with(hf, label_values), measurement);
+	}
+
+function collect_metrics(prefix: string, name: string): vector of Metric
+	{
+	return Telemetry::__collect_metrics(prefix, name);
+	}
+
+function collect_histogram_metrics(prefix: string, name: string): vector of HistogramMetric
+	{
+	return Telemetry::__collect_histogram_metrics(prefix, name);
+	}
+
+event run_sync_hook()
+	{
+	hook Telemetry::sync();
+	schedule sync_interval { run_sync_hook() };
+	}
+
+event zeek_init()
+	{
+	schedule sync_interval { run_sync_hook() };
+	}
+
+# Expose the Zeek version as Prometheus style info metric
+global version_gauge_family = Telemetry::register_gauge_family([
+	$prefix="zeek",
+	$name="version_info",
+	$unit="1",
+	$help_text="The Zeek version",
+	$labels=vector("version_number", "major", "minor", "patch", "commit",
+                       "beta", "debug","version_string")
+]);
+
+event zeek_init()
+	{
+	local v = Version::info;
+	local labels = vector(cat(v$version_number),
+	                      cat(v$major), cat(v$minor), cat (v$patch),
+	                      cat(v$commit),
+	                      v$beta ? "true" : "false",
+	                      v$debug ? "true" : "false",
+	                      v$version_string);
+
+	Telemetry::gauge_family_set(version_gauge_family, labels, 1.0);
+	}

scripts/base/init-bare.zeek

@@ -588,6 +588,23 @@ type fa_metadata: record {
 	inferred: bool &default=T;
 };
 
+## A hook taking a connection, analyzer tag and analyzer id that can be
+## used to veto disabling analyzers. Specifically, an analyzer can be prevented
+## from being disabled by using a :zeek:see:`break` statement within the hook.
+## This hook is invoked synchronously during a :zeek:see:`disable_analyzer` call.
+##
+## Scripts implementing this hook should have other logic that will eventually
+## disable the analyzer for the given connection. That is, if a script vetoes
+## disabling an analyzer, it takes responsibility for a later call to
+## :zeek:see:`disable_analyzer`, which may be never.
+##
+## c: The connection
+##
+## atype: The type / tag of the analyzer being disabled.
+##
+## aid: The analyzer ID.
+type disabling_analyzer: hook(c: connection, atype: AllAnalyzers::Tag, aid: count);
+
 ## Fields of a SYN packet.
 ##
 ## .. zeek:see:: connection_SYN_packet

scripts/base/init-default.zeek

@@ -43,6 +43,7 @@
 @load base/frameworks/tunnels
 @load base/frameworks/openflow
 @load base/frameworks/netcontrol
+@load base/frameworks/telemetry
 
 @load base/protocols/conn
 @load base/protocols/dce-rpc

scripts/base/protocols/dhcp/main.zeek

@@ -91,6 +91,12 @@ export {
 	## transaction narrative.
 	option DHCP::max_txid_watch_time = 30secs;
 
+	## The maximum number of uids allowed in a single log entry.
+	option DHCP::max_uids_per_log_entry = 10;
+
+	## The maximum number of msg_types allowed in a single log entry.
+	option DHCP::max_msg_types_per_log_entry = 50;
+
 	## This event is used internally to distribute data around clusters
 	## since DHCP doesn't follow the normal "connection" model used by
 	## most protocols. It can also be handled to extend the DHCP log.
@@ -266,6 +272,13 @@ event DHCP::aggregate_msgs(ts: time, id: conn_id, uid: string, is_orig: bool, ms
 		if ( options?$lease )
 			log_info$lease_time = options$lease;
 		}
+
+	# Write log entry if |uids| or |msg_types| becomes too large
+	if ( |log_info$uids| >= max_uids_per_log_entry || |log_info$msg_types| >= max_msg_types_per_log_entry )
+		{
+		Log::write(LOG, log_info);
+		delete join_data[msg$xid];
+		}
 	}
 @endif
 

scripts/base/protocols/modbus/main.zeek

@@ -16,8 +16,14 @@ export {
 		uid:       string         &log;
 		## Identifier for the connection.
 		id:        conn_id        &log;
+		## Modbus transaction ID
+		tid:	   count         &log &optional;
+		## The terminal unit identifier for the message
+		unit:	   count         &log &optional;
 		## The name of the function message that was sent.
 		func:      string         &log &optional;
+		## Whether this PDU was a response ("RESP") or request ("REQ")
+		pdu_type:  string         &log &optional;
 		## The exception if the response was a failure.
 		exception: string         &log &optional;
 	};
@@ -48,14 +54,18 @@ event modbus_message(c: connection, headers: ModbusHeaders, is_orig: bool) &prio
 		}
 
 	c$modbus$ts   = network_time();
+	c$modbus$tid = headers$tid;
+	c$modbus$unit = headers$uid;
 	c$modbus$func = function_codes[headers$function_code];
+	## If this message is from the TCP originator, it is a request. Otherwise,
+	## it is a response.
+	c$modbus$pdu_type = is_orig ? "REQ" : "RESP";
 	}
 
 event modbus_message(c: connection, headers: ModbusHeaders, is_orig: bool) &priority=-5
 	{
-	# Only log upon replies.
+	# Don't log now if this is an exception (log in the exception event handler)
-	# Also, don't log now if this is an exception (log in the exception event handler)
+	if ( headers$function_code <= 0x81 || headers$function_code >= 0x98 )
-	if ( ! is_orig && ( headers$function_code <= 0x81 || headers$function_code >= 0x98 ) )
 		Log::write(LOG, c$modbus);
 	}
 

scripts/base/protocols/ssl/main.zeek

@@ -255,10 +255,8 @@ function finish(c: connection, remove_analyzer: bool)
 	{
 	log_record(c$ssl);
 	if ( remove_analyzer && disable_analyzer_after_detection && c?$ssl && c$ssl?$analyzer_id )
-		{
+		if ( disable_analyzer(c$id, c$ssl$analyzer_id) )
-		disable_analyzer(c$id, c$ssl$analyzer_id);
+			delete c$ssl$analyzer_id;
-		delete c$ssl$analyzer_id;
-		}
 	}
 
 event ssl_client_hello(c: connection, version: count, record_version: count, possible_ts: time, client_random: string, session_id: string, ciphers: index_vec, comp_methods: index_vec) &priority=5

scripts/base/utils/email.zeek

@@ -58,7 +58,7 @@ function extract_first_email_addr(str: string): string
 function split_mime_email_addresses(line: string): set[string]
 	{
 	local output = string_set();
-	local addrs = find_all(line, /(\"[^"]*\")?[^,]+/);
+	local addrs = find_all(line, /(\"[^"]*\")?[^,]+@[^,]+/);
 	for ( part in addrs )
 		{
 		add output[strip(part)];

scripts/base/utils/site.zeek

@@ -234,13 +234,38 @@ function get_emails(a: addr): string
 	return fmt_email_string(find_all_emails(a));
 	}
 
-event zeek_init() &priority=10
+function update_local_nets_table(id: string, new_value: set[subnet]): set[subnet]
+	{
+	# Create the local_nets mapping table.
+	for ( cidr in new_value )
+		local_nets_table[cidr] = cidr;
+	return new_value;
+	}
+
+function update_local_zones_regex(id: string, new_value: set[string]): set[string]
 	{
 	# Double backslashes are needed due to string parsing.
-	local_dns_suffix_regex = set_to_regex(local_zones, "(^\\.?|\\.)(~~)$");
+	local_dns_suffix_regex = set_to_regex(new_value, "(^\\.?|\\.)(~~)$");
-	local_dns_neighbor_suffix_regex = set_to_regex(neighbor_zones, "(^\\.?|\\.)(~~)$");
+	return new_value;
-
+	}
-	# Create the local_nets mapping table.
+
-	for ( cidr in Site::local_nets )
+function update_neighbor_zones_regex(id: string, new_value: set[string]): set[string]
-		local_nets_table[cidr] = cidr;
+	{
+	local_dns_neighbor_suffix_regex = set_to_regex(new_value, "(^\\.?|\\.)(~~)$");
+	return new_value;
+	}
+
+event zeek_init() &priority=10
+	{
+	# Have these run with a lower priority so we account for additions/removals
+	# from user created change handlers.
+	Option::set_change_handler("Site::local_nets", update_local_nets_table, -5);
+	Option::set_change_handler("Site::local_zones", update_local_zones_regex, -5);
+	Option::set_change_handler("Site::neighbor_zones", update_neighbor_zones_regex, -5);
+
+	# Use change handler to initialize local_nets mapping table and zones
+	# regexes.
+	update_local_nets_table("Site::local_nets", Site::local_nets);
+	update_local_zones_regex("Site::local_zones", Site::local_zones);
+	update_neighbor_zones_regex("Site::neighbor_zones", Site::neighbor_zones);
 	}

scripts/policy/frameworks/management/agent/main.zeek

@@ -204,7 +204,7 @@ function send_deploy_response(req: Management::Request::Request)
 		}
 
 	Management::Log::info(fmt("tx Management::Agent::API::deploy_response %s",
-	    Management::result_to_string(res)));
+	    Management::result_vec_to_string(req$results)));
 	Broker::publish(agent_topic(),
 	    Management::Agent::API::deploy_response, req$id, req$results);
 
@@ -262,6 +262,8 @@ event Management::Agent::Runtime::trigger_log_archival(run_archival: bool)
 
 event Management::Supervisor::API::notify_node_exit(node: string, outputs: Management::NodeOutputs)
 	{
+	Management::Log::info(fmt("rx Management::Supervisor::API::notify_node_exit %s", node));
+
 	if ( node in g_nodes )
 		g_outputs[node] = outputs;
 	}
@@ -753,7 +755,8 @@ event Management::Node::API::node_dispatch_response(reqid: string, result: Manag
 
 event Management::Agent::API::node_dispatch_request(reqid: string, action: vector of string, nodes: set[string])
 	{
-	Management::Log::info(fmt("rx Management::Agent::API::node_dispatch_request %s %s %s", reqid, action, nodes));
+	Management::Log::info(fmt("rx Management::Agent::API::node_dispatch_request %s %s %s",
+	    reqid, action, Management::Util::set_to_vector(nodes)));
 
 	local node: string;
 	local cluster_nodes: set[string];

scripts/policy/frameworks/management/controller/main.zeek

@@ -1174,11 +1174,11 @@ event Management::Controller::API::get_id_value_request(reqid: string, id: strin
 
 	local res: Management::Result;
 
-	# Special case: if we have no instances, respond right away.
+	# Special case: if we have no deployed cluster, respond right away.
-	if ( |g_instances_known| == 0 )
+	if ( |g_instances| == 0 )
 		{
 		Management::Log::info(fmt("tx Management::Controller::API::get_id_value_response %s", reqid));
-		res = Management::Result($reqid=reqid, $success=F, $error="no instances connected");
+		res = Management::Result($reqid=reqid, $success=F, $error="no cluster deployed");
 		Broker::publish(Management::Controller::topic,
 		    Management::Controller::API::get_id_value_response,
 		    reqid, vector(res));

scripts/policy/frameworks/management/node/main.zeek

@@ -57,7 +57,8 @@ global g_dispatch_table: table[string] of DispatchCallback = {
 
 event Management::Node::API::node_dispatch_request(reqid: string, action: vector of string, nodes: set[string])
 	{
-	Management::Log::info(fmt("rx Management::Node::API::node_dispatch_request %s %s %s", reqid, action, nodes));
+	Management::Log::info(fmt("rx Management::Node::API::node_dispatch_request %s %s %s",
+	    reqid, action, Management::Util::set_to_vector(nodes)));
 
 	if ( |nodes| > 0 && Cluster::node !in nodes )
 		{
@@ -102,7 +103,10 @@ event Broker::peer_added(peer: Broker::EndpointInfo, msg: string)
 	# If this is the agent peering, notify it that we're ready
 	if ( peer$network$address == epi$network$address &&
 	     peer$network$bound_port == epi$network$bound_port )
+		{
+		Management::Log::info(fmt("tx Management::Node::API::notify_node_hello %s", Cluster::node));
 		Broker::publish(node_topic, Management::Node::API::notify_node_hello, Cluster::node);
+		}
 	}
 
 event zeek_init()

scripts/policy/frameworks/telemetry/log.zeek

@@ -0,0 +1,199 @@
+##! Implementation of a telemetry.log and telemetry_histogram.log file
+##! using metrics accessible via the Telemetry module.
+
+@load base/frameworks/telemetry
+
+module Telemetry;
+
+export {
+	redef enum Log::ID += { LOG, LOG_HISTOGRAM };
+
+	## How often metrics are reported.
+	option log_interval = 60sec;
+
+	## Only metrics with prefixes in this set will be included in the
+	## `telemetry.log` and `telemetry_histogram.log` files by default.
+	## Setting this option to an empty set includes all prefixes.
+	##
+	## For more fine-grained customization, setting this option to an
+	## empty set and implementing the :zeek:see:`Telemetry::log_policy`
+	## and :zeek:see:`Telemetry::log_policy_histogram` hooks to filter
+	## individual records is recommended.
+	option log_prefixes: set[string] = {"process", "zeek"};
+
+	## Record type used for logging counter and gauge metrics.
+	type Info: record {
+		## Timestamp of reporting.
+		ts: time &log;
+
+		## Peer that generated this log.
+		peer: string &log;
+
+		## Contains the value "counter" or "gauge" depending on
+		## the underlying metric type.
+		metric_type: string &log;
+
+		## The prefix (namespace) of the metric.
+		prefix: string &log;
+
+		## The name of the metric.
+		name: string &log;
+
+		## The unit of this metric, or unset if unit-less.
+		unit: string &log &optional;
+
+		## The names of the individual labels.
+		labels: vector of string &log;
+
+		## The values of the labels as listed in ``labels``.
+		label_values: vector of string &log;
+
+		## The value of this metric.
+		value: double &log;
+	};
+
+	## Record type used for logging histogram metrics.
+	type HistogramInfo: record {
+		## Timestamp of reporting.
+		ts: time &log;
+
+		## Peer that generated this log.
+		peer: string &log;
+
+		## The prefix (namespace) of the metric.
+		prefix: string &log;
+
+		## The name of the metric.
+		name: string &log;
+
+		## The unit of this metric, or unset if unit-less.
+		unit: string &log &optional;
+
+		## The names of the individual labels.
+		labels: vector of string &log;
+
+		## The values of the labels as listed in ``labels``.
+		label_values: vector of string &log;
+
+		## The bounds of the individual buckets
+		bounds: vector of double &log;
+
+		## The number of observations within each individual bucket.
+		values: vector of double &log;
+
+		## The sum over all observations
+		sum: double &log;
+
+		## The total number of observations.
+		observations: double &log;
+	};
+
+	## A default logging policy hook for the stream.
+	global log_policy: Log::PolicyHook;
+
+	## A default logging policy hook for the histogram stream.
+	global log_policy_histogram: Log::PolicyHook;
+
+	## Event triggered for every record in the stream.
+	global log_telemetry: event(rec: Info);
+
+	## Event triggered for every record in the histogram stream.
+	global log_telemetry_histogram: event(rec: HistogramInfo);
+}
+
+function do_log()
+	{
+	local ts = network_time();
+	local metrics = Telemetry::collect_metrics();
+
+	for ( i in metrics )
+		{
+		local m = metrics[i];
+
+		# Histograms don't have single values, skip over them.
+		if ( m$opts$metric_type == DOUBLE_HISTOGRAM || m$opts$metric_type == INT_HISTOGRAM )
+			next;
+
+		if ( |log_prefixes| > 0 && m$opts$prefix !in log_prefixes )
+			next;
+
+		# Render the metric_type as a short string. Unknown
+		# shouldn't really happen, but lets have a fallback.
+		local metric_type = "unknown";
+		switch ( m$opts$metric_type ) {
+			case DOUBLE_COUNTER, INT_COUNTER:
+				metric_type = "counter";
+				break;
+			case DOUBLE_GAUGE, INT_GAUGE:
+				metric_type = "gauge";
+				break;
+		}
+
+		local rec = Info($ts=ts,
+		                 $peer=peer_description,
+		                 $metric_type=metric_type,
+		                 $prefix=m$opts$prefix,
+		                 $name=m$opts$name,
+		                 $labels=m$opts$labels,
+		                 $label_values=m$labels,
+		                 $value=m$value);
+
+		if ( m$opts$unit != "1" )
+			rec$unit = m$opts$unit;
+
+		Log::write(LOG, rec);
+		}
+
+	# Logging of histograms.
+	ts = network_time();
+	local histogram_metrics = Telemetry::collect_histogram_metrics();
+	for ( i in histogram_metrics )
+		{
+		local hm = histogram_metrics[i];
+
+		if ( |log_prefixes| > 0 && hm$opts$prefix !in log_prefixes )
+			next;
+
+		local hrec = HistogramInfo($ts=ts,
+		                           $peer=peer_description,
+		                           $prefix=hm$opts$prefix,
+		                           $name=hm$opts$name,
+		                           $labels=hm$opts$labels,
+		                           $label_values=hm$labels,
+		                           $bounds=hm$opts$bounds,
+		                           $values=hm$values,
+		                           $sum=hm$sum,
+		                           $observations=hm$observations);
+
+		if ( hm$opts$unit != "1" )
+			hrec$unit = hm$opts$unit;
+
+		Log::write(LOG_HISTOGRAM, hrec);
+		}
+	}
+
+event Telemetry::log()
+	{
+	# We explicitly log once during zeek_done(), so short-circuit
+	# here when we're already in the process of shutting down.
+	if ( zeek_is_terminating() )
+		return;
+
+	do_log();
+	schedule log_interval { Telemetry::log() };
+	}
+
+event zeek_init() &priority=5
+	{
+	Log::create_stream(LOG, [$columns=Info, $ev=log_telemetry, $path="telemetry", $policy=log_policy]);
+	Log::create_stream(LOG_HISTOGRAM, [$columns=HistogramInfo, $ev=log_telemetry_histogram, $path="telemetry_histogram", $policy=log_policy_histogram]);
+
+	schedule log_interval { Telemetry::log() };
+	}
+
+# Log late during zeek_done() once more. Any metric updates
+# afterwards won't be visible in the log.
+event zeek_done() &priority=-1000
+	{
+	do_log();
+	}

scripts/site/local.zeek

@@ -89,6 +89,10 @@ redef digest_salt = "Please change this value.";
 # Extend email alerting to include hostnames
 @load policy/frameworks/notice/extend-email/hostnames
 
+# Enable logging of telemetry data into telemetry.log and
+# telemetry_histogram.log.
+@load frameworks/telemetry/log
+
 # Uncomment the following line to enable detection of the heartbleed attack. Enabling
 # this might impact performance a bit.
 # @load policy/protocols/ssl/heartbleed

scripts/test-all-policy.zeek

@@ -71,6 +71,7 @@
 @load frameworks/software/version-changes.zeek
 @load frameworks/software/vulnerable.zeek
 @load frameworks/software/windows-version-detection.zeek
+@load frameworks/telemetry/log.zeek
 @load integration/barnyard2/__load__.zeek
 @load integration/barnyard2/main.zeek
 @load integration/barnyard2/types.zeek

src/3rdparty

@@ -1 +1 @@
-Subproject commit a60ffe6a41c0f335f1dcf22ec2e6bf246b9ab0ac
+Subproject commit 423e2df85c449c9575d17514722e4ad7eb6ea53d

src/EventRegistry.cc

@@ -136,4 +136,14 @@ void EventRegistry::SetErrorHandler(std::string_view name)
 	                          std::string(name).c_str());
 	}
 
+void EventRegistry::ActivateAllHandlers()
+	{
+	auto event_names = AllHandlers();
+	for ( const auto& name : event_names )
+		{
+		if ( auto event = Lookup(name) )
+			event->SetGenerateAlways();
+		}
+	}
+
 	} // namespace zeek

src/EventRegistry.h

@@ -60,6 +60,17 @@ public:
 
 	void PrintDebug();
 
+	/**
+	 * Marks all event handlers as active.
+	 *
+	 * By default, zeek does not generate (raise) events that have not handled by
+	 * any scripts. This means that these events will be invisible to a lot of other
+	 * event handlers - and will not raise :zeek:id:`new_event`. Calling this
+	 * function will cause all event handlers to be raised. This is likely only
+	 * useful for debugging and fuzzing, and likely causes reduced performance.
+	 */
+	void ActivateAllHandlers();
+
 private:
 	std::map<std::string, std::unique_ptr<EventHandler>, std::less<>> handlers;
 	// Tracks whether a given event handler was registered in a

src/Expr.cc

@@ -960,11 +960,34 @@ ValPtr BinaryExpr::Fold(Val* v1, Val* v2) const
 			DO_UINT_FOLD(^);
 			break;
 		case EXPR_LSHIFT:
-			DO_INT_FOLD(<<);
+			{
+			if ( is_integral )
+				{
+				if ( i1 < 0 )
+					RuntimeError("left shifting a negative number is undefined");
+
+				i3 = i1 << static_cast<zeek_uint_t>(i2);
+				}
+
+			else if ( is_unsigned )
+				u3 = u1 << u2;
+
+			else
+				RuntimeErrorWithCallStack("bad type in BinaryExpr::Fold");
 			break;
+			}
 		case EXPR_RSHIFT:
-			DO_INT_FOLD(>>);
+			{
+			if ( is_integral )
+				i3 = i1 >> static_cast<zeek_uint_t>(i2);
+
+			else if ( is_unsigned )
+				u3 = u1 >> u2;
+
+			else
+				RuntimeErrorWithCallStack("bad type in BinaryExpr::Fold");
 			break;
+			}
 
 		case EXPR_AND_AND:
 			DO_INT_FOLD(&&);
@@ -2166,6 +2189,9 @@ BitExpr::BitExpr(ExprTag arg_tag, ExprPtr arg_op1, ExprPtr arg_op2)
 
 	if ( tag == EXPR_LSHIFT || tag == EXPR_RSHIFT )
 		{
+		if ( (is_vector(op1) || is_vector(op2)) && ! (is_vector(op1) && is_vector(op2)) )
+			ExprError("cannot mix vectors and scalars for shift operations");
+
 		if ( IsIntegral(bt1) && bt2 == TYPE_COUNT )
 			{
 			if ( is_vector(op1) || is_vector(op2) )
@@ -2173,13 +2199,19 @@ BitExpr::BitExpr(ExprTag arg_tag, ExprPtr arg_op1, ExprPtr arg_op2)
 			else
 				SetType(base_type(bt1));
 			}
+
 		else if ( IsIntegral(bt1) && bt2 == TYPE_INT )
 			ExprError("requires \"count\" right operand");
+
 		else
 			ExprError("requires integral operands");
+
+		return; // because following scalar check isn't apt
 		}
 
-	else if ( (bt1 == TYPE_COUNT) && (bt2 == TYPE_COUNT) )
+	CheckScalarAggOp();
+
+	if ( (bt1 == TYPE_COUNT) && (bt2 == TYPE_COUNT) )
 		{
 		if ( is_vector(op1) || is_vector(op2) )
 			SetType(make_intrusive<VectorType>(base_type(TYPE_COUNT)));

src/IP.cc

@@ -39,6 +39,10 @@ static VectorValPtr BuildOptionsVal(const u_char* data, int len)
 			{
 			// PadN or other option
 			uint16_t off = 2 * sizeof(uint8_t);
+
+			if ( len < opt->ip6o_len + off )
+				break;
+
 			rv->Assign(1, opt->ip6o_len);
 			rv->Assign(2, new String(data + off, opt->ip6o_len, true));
 			data += opt->ip6o_len + off;

src/RE.cc

@@ -7,6 +7,7 @@
 #include <cstdlib>
 #include <utility>
 
+#include "zeek/3rdparty/doctest.h"
 #include "zeek/CCL.h"
 #include "zeek/DFA.h"
 #include "zeek/EquivClass.h"
@@ -16,7 +17,8 @@
 zeek::detail::CCL* zeek::detail::curr_ccl = nullptr;
 zeek::detail::Specific_RE_Matcher* zeek::detail::rem = nullptr;
 zeek::detail::NFA_Machine* zeek::detail::nfa = nullptr;
-int zeek::detail::case_insensitive = 0;
+bool zeek::detail::case_insensitive = false;
+bool zeek::detail::re_single_line = false;
 
 extern int RE_parse(void);
 extern void RE_set_input(const char* str);
@@ -27,13 +29,11 @@ namespace zeek
 namespace detail
 	{
 
-Specific_RE_Matcher::Specific_RE_Matcher(match_type arg_mt, int arg_multiline)
+Specific_RE_Matcher::Specific_RE_Matcher(match_type arg_mt, bool arg_multiline)
-	: equiv_class(NUM_SYM)
+	: mt(arg_mt), multiline(arg_multiline), equiv_class(NUM_SYM)
 	{
-	mt = arg_mt;
-	multiline = arg_multiline;
 	any_ccl = nullptr;
-	pattern_text = nullptr;
+	single_line_ccl = nullptr;
 	dfa = nullptr;
 	ecs = nullptr;
 	accepted = new AcceptingSet();
@@ -45,14 +45,25 @@ Specific_RE_Matcher::~Specific_RE_Matcher()
 		delete ccl_list[i];
 
 	Unref(dfa);
-	delete[] pattern_text;
 	delete accepted;
 	}
 
-CCL* Specific_RE_Matcher::AnyCCL()
+CCL* Specific_RE_Matcher::AnyCCL(bool single_line_mode)
 	{
+	if ( single_line_mode )
+		{
+		if ( ! single_line_ccl )
+			{
+			single_line_ccl = new CCL();
+			single_line_ccl->Negate();
+			EC()->CCL_Use(single_line_ccl);
+			}
+
+		return single_line_ccl;
+		}
+
 	if ( ! any_ccl )
-		{ // Create the '.' character class.
+		{
 		any_ccl = new CCL();
 		if ( ! multiline )
 			any_ccl->Add('\n');
@@ -89,51 +100,38 @@ void Specific_RE_Matcher::AddExactPat(const char* new_pat)
 
 void Specific_RE_Matcher::AddPat(const char* new_pat, const char* orig_fmt, const char* app_fmt)
 	{
-	int n = strlen(new_pat);
+	if ( ! pattern_text.empty() )
-
+		pattern_text = util::fmt(app_fmt, pattern_text.c_str(), new_pat);
-	if ( pattern_text )
-		n += strlen(pattern_text) + strlen(app_fmt);
 	else
-		n += strlen(orig_fmt);
+		pattern_text = util::fmt(orig_fmt, new_pat);
-
-	char* s = new char[n + 5 /* slop */];
-
-	if ( pattern_text )
-		sprintf(s, app_fmt, pattern_text, new_pat);
-	else
-		sprintf(s, orig_fmt, new_pat);
-
-	delete[] pattern_text;
-	pattern_text = s;
 	}
 
 void Specific_RE_Matcher::MakeCaseInsensitive()
 	{
 	const char fmt[] = "(?i:%s)";
-	int n = strlen(pattern_text) + strlen(fmt);
+	pattern_text = util::fmt(fmt, pattern_text.c_str());
+	}
 
-	char* s = new char[n + 5 /* slop */];
+void Specific_RE_Matcher::MakeSingleLine()
-
+	{
-	snprintf(s, n + 5, fmt, pattern_text);
+	const char fmt[] = "(?s:%s)";
-
+	pattern_text = util::fmt(fmt, pattern_text.c_str());
-	delete[] pattern_text;
-	pattern_text = s;
 	}
 
 bool Specific_RE_Matcher::Compile(bool lazy)
 	{
-	if ( ! pattern_text )
+	if ( pattern_text.empty() )
 		return false;
 
 	rem = this;
-	RE_set_input(pattern_text);
+	RE_set_input(pattern_text.c_str());
 
 	int parse_status = RE_parse();
 	RE_done_with_scan();
 
 	if ( parse_status )
 		{
-		reporter->Error("error compiling pattern /%s/", pattern_text);
+		reporter->Error("error compiling pattern /%s/", pattern_text.c_str());
 		Unref(nfa);
 		nfa = nullptr;
 		return false;
@@ -416,13 +414,10 @@ static RE_Matcher* matcher_merge(const RE_Matcher* re1, const RE_Matcher* re2, c
 	const char* text1 = re1->PatternText();
 	const char* text2 = re2->PatternText();
 
-	int n = strlen(text1) + strlen(text2) + strlen(merge_op) + 32 /* slop */;
+	size_t n = strlen(text1) + strlen(text2) + strlen(merge_op) + 32 /* slop */;
 
-	char* merge_text = new char[n];
+	std::string merge_text = util::fmt("(%s)%s(%s)", text1, merge_op, text2);
-	snprintf(merge_text, n, "(%s)%s(%s)", text1, merge_op, text2);
+	RE_Matcher* merge = new RE_Matcher(merge_text.c_str());
-
-	RE_Matcher* merge = new RE_Matcher(merge_text);
-	delete[] merge_text;
 
 	merge->Compile();
 
@@ -483,9 +478,122 @@ void RE_Matcher::MakeCaseInsensitive()
 	is_case_insensitive = true;
 	}
 
+void RE_Matcher::MakeSingleLine()
+	{
+	re_anywhere->MakeSingleLine();
+	re_exact->MakeSingleLine();
+
+	is_single_line = true;
+	}
+
 bool RE_Matcher::Compile(bool lazy)
 	{
 	return re_anywhere->Compile(lazy) && re_exact->Compile(lazy);
 	}
 
+TEST_SUITE("re_matcher")
+	{
+
+	TEST_CASE("simple_pattern")
+		{
+		RE_Matcher match("[0-9]+");
+		match.Compile();
+		CHECK(strcmp(match.OrigText(), "[0-9]+") == 0);
+		CHECK(strcmp(match.PatternText(), "^?([0-9]+)$?") == 0);
+		CHECK(strcmp(match.AnywherePatternText(), "^?(.|\\n)*([0-9]+)") == 0);
+
+		CHECK(match.MatchExactly("12345"));
+		CHECK_FALSE(match.MatchExactly("a12345"));
+
+		// The documentation for MatchAnywhere says that it returns the
+		// "index just beyond where the first match occurs", which I would
+		// think means *after* the match. This is returning the position
+		// where the match starts though.
+		CHECK(match.MatchAnywhere("a1234bcd") == 2);
+		CHECK(match.MatchAnywhere("abcd") == 0);
+		}
+
+	TEST_CASE("case_insensitive_mode")
+		{
+		RE_Matcher match("[a-z]+");
+		match.MakeCaseInsensitive();
+		match.Compile();
+		CHECK(strcmp(match.PatternText(), "(?i:^?([a-z]+)$?)") == 0);
+
+		CHECK(match.MatchExactly("abcDEF"));
+		}
+
+	TEST_CASE("multi_pattern")
+		{
+		RE_Matcher match("[0-9]+");
+		match.AddPat("[a-z]+");
+		match.Compile();
+
+		CHECK(strcmp(match.PatternText(), "(^?([0-9]+)$?)|(^?([a-z]+)$?)") == 0);
+
+		CHECK(match.MatchExactly("abc"));
+		CHECK(match.MatchExactly("123"));
+		CHECK_FALSE(match.MatchExactly("abc123"));
+		}
+
+	TEST_CASE("modes_multi_pattern")
+		{
+		RE_Matcher match("[a-m]+");
+		match.MakeCaseInsensitive();
+
+		match.AddPat("[n-z]+");
+		match.Compile();
+
+		CHECK(strcmp(match.PatternText(), "((?i:^?([a-m]+)$?))|(^?([n-z]+)$?)") == 0);
+		CHECK(match.MatchExactly("aBc"));
+		CHECK(match.MatchExactly("nop"));
+		CHECK_FALSE(match.MatchExactly("NoP"));
+		}
+
+	TEST_CASE("single_line_mode")
+		{
+		RE_Matcher match(".*");
+		match.MakeSingleLine();
+		match.Compile();
+
+		CHECK(strcmp(match.PatternText(), "(?s:^?(.*)$?)") == 0);
+		CHECK(match.MatchExactly("abc\ndef"));
+
+		RE_Matcher match2("fOO.*bAR");
+		match2.MakeSingleLine();
+		match2.Compile();
+
+		CHECK(strcmp(match2.PatternText(), "(?s:^?(fOO.*bAR)$?)") == 0);
+		CHECK(match.MatchExactly("fOOab\ncdbAR"));
+
+		RE_Matcher match3("b.r");
+		match3.MakeSingleLine();
+		match3.Compile();
+		CHECK(match3.MatchExactly("bar"));
+		CHECK(match3.MatchExactly("b\nr"));
+
+		RE_Matcher match4("a.c");
+		match4.MakeSingleLine();
+		match4.AddPat("def");
+		match4.Compile();
+		CHECK(match4.MatchExactly("abc"));
+		CHECK(match4.MatchExactly("a\nc"));
+		}
+
+	TEST_CASE("disjunction")
+		{
+		RE_Matcher match1("a.c");
+		match1.MakeSingleLine();
+		match1.Compile();
+		RE_Matcher match2("def");
+		match2.Compile();
+		auto dj = detail::RE_Matcher_disjunction(&match1, &match2);
+		CHECK(dj->MatchExactly("abc"));
+		CHECK(dj->MatchExactly("a.c"));
+		CHECK(dj->MatchExactly("a\nc"));
+		CHECK(dj->MatchExactly("def"));
+		delete dj;
+		}
+	}
+
 	} // namespace zeek

src/RE.h

@@ -32,7 +32,8 @@ class DFA_State;
 class Specific_RE_Matcher;
 class CCL;
 
-extern int case_insensitive;
+extern bool case_insensitive;
+extern bool re_single_line;
 extern CCL* curr_ccl;
 extern NFA_Machine* nfa;
 extern Specific_RE_Matcher* rem;
@@ -59,14 +60,15 @@ enum match_type
 class Specific_RE_Matcher
 	{
 public:
-	explicit Specific_RE_Matcher(match_type mt, int multiline = 0);
+	explicit Specific_RE_Matcher(match_type mt, bool multiline = false);
 	~Specific_RE_Matcher();
 
 	void AddPat(const char* pat);
 
 	void MakeCaseInsensitive();
+	void MakeSingleLine();
 
-	void SetPat(const char* pat) { pattern_text = util::copy_string(pat); }
+	void SetPat(const char* pat) { pattern_text = pat; }
 
 	bool Compile(bool lazy = false);
 
@@ -90,7 +92,7 @@ public:
 		return nullptr;
 		}
 	CCL* LookupCCL(int index) { return ccl_list[index]; }
-	CCL* AnyCCL();
+	CCL* AnyCCL(bool single_line_mode = false);
 
 	void ConvertCCLs();
 
@@ -117,7 +119,7 @@ public:
 
 	EquivClass* EC() { return &equiv_class; }
 
-	const char* PatternText() const { return pattern_text; }
+	const char* PatternText() const { return pattern_text.c_str(); }
 
 	DFA_Machine* DFA() const { return dfa; }
 
@@ -135,17 +137,21 @@ protected:
 	bool MatchAll(const u_char* bv, int n);
 
 	match_type mt;
-	int multiline;
+	bool multiline;
-	char* pattern_text;
+
+	std::string pattern_text;
 
 	std::map<std::string, std::string> defs;
 	std::map<std::string, CCL*> ccl_dict;
+	std::vector<char> modifiers;
 	PList<CCL> ccl_list;
 	EquivClass equiv_class;
 	int* ecs;
 	DFA_Machine* dfa;
-	CCL* any_ccl;
 	AcceptingSet* accepted;
+
+	CCL* any_ccl;
+	CCL* single_line_ccl;
 	};
 
 class RE_Match_State
@@ -205,6 +211,9 @@ public:
 	void MakeCaseInsensitive();
 	bool IsCaseInsensitive() const { return is_case_insensitive; }
 
+	void MakeSingleLine();
+	bool IsSingleLine() const { return is_single_line; }
+
 	bool Compile(bool lazy = false);
 
 	// Returns true if s exactly matches the pattern, false otherwise.
@@ -240,6 +249,7 @@ protected:
 	detail::Specific_RE_Matcher* re_exact;
 
 	bool is_case_insensitive = false;
+	bool is_single_line = false;
 	};
 
 	} // namespace zeek

src/RuleMatcher.cc

@@ -526,7 +526,7 @@ void RuleMatcher::BuildPatternSets(RuleHdrTest::pattern_set_list* dst, const str
 		if ( group_exprs.length() > sig_max_group_size || i == exprs.length() )
 			{
 			RuleHdrTest::PatternSet* set = new RuleHdrTest::PatternSet;
-			set->re = new Specific_RE_Matcher(MATCH_EXACTLY, 1);
+			set->re = new Specific_RE_Matcher(MATCH_EXACTLY, true);
 			set->re->CompileSet(group_exprs, group_ids);
 			set->patterns = group_exprs;
 			set->ids = group_ids;

src/Stmt.cc

@@ -2185,6 +2185,12 @@ TraversalCode WhenStmt::Traverse(TraversalCallback* cb) const
 			}
 		}
 
+	if ( wi->TimeoutExpr() )
+		{
+		tc = wi->TimeoutExpr()->Traverse(cb);
+		HANDLE_TC_STMT_PRE(tc);
+		}
+
 	tc = cb->PostStmt(this);
 	HANDLE_TC_STMT_POST(tc);
 	}

src/Val.h

@@ -932,6 +932,8 @@ public:
 
 	const PDict<TableEntryVal>* Get() const { return table_val; }
 
+	const detail::CompositeHash* GetTableHash() const { return table_hash; }
+
 	// Returns the size of the table.
 	int Size() const;
 	int RecursiveSize() const;
@@ -1593,10 +1595,8 @@ public:
 	ValPtr ValAt(unsigned int index) const { return At(index); }
 
 	bool Has(unsigned int index) const
-		// Version to use once std::optional implementation is merged.
-		// { return index < vector_val->size() && vector_val[index]; }
 		{
-		return At(index) != nullptr;
+		return index < vector_val->size() && (*vector_val)[index];
 		}
 
 	/**

src/Var.cc

@@ -429,14 +429,25 @@ void add_type(ID* id, TypePtr t, std::unique_ptr<std::vector<AttrPtr>> attr)
 	{
 	std::string new_type_name = id->Name();
 	std::string old_type_name = t->GetName();
+
 	TypePtr tnew;
 
 	if ( (t->Tag() == TYPE_RECORD || t->Tag() == TYPE_ENUM) && old_type_name.empty() )
 		// An extensible type (record/enum) being declared for first time.
 		tnew = std::move(t);
 	else
+		{
+		// If the old type is an error or the old type doesn't exist, then return
+		// an error instead of trying to clone it.
+		if ( t->Tag() == TYPE_ERROR && t->InternalType() == TYPE_INTERNAL_ERROR )
+			{
+			reporter->Error("Error trying to create alias to nonexistent type");
+			return;
+			}
+
 		// Clone the type to preserve type name aliasing.
 		tnew = t->ShallowClone();
+		}
 
 	Type::RegisterAlias(new_type_name, tnew);
 

src/fuzzers/CMakeLists.txt

@@ -81,3 +81,7 @@ target_link_libraries(zeek_fuzzer_shared
 add_fuzz_target(dns)
 add_fuzz_target(pop3)
 add_fuzz_target(packet)
+add_fuzz_target(http)
+add_fuzz_target(imap)
+add_fuzz_target(smtp)
+add_fuzz_target(ftp)

src/fuzzers/README.rst

@@ -23,6 +23,9 @@ First configure and build for fuzzing (with libFuzzer) and code coverage::
    variable may be changed to use another flag or direct path to fuzzing engine
    library to link against.
 
+Text/Dictionary-based Corpus
+````````````````````````````
+
 Now start fuzzing to generate an initial corpus (this uses the POP3 fuzzer as
 an example)::
 
@@ -54,6 +57,24 @@ commit)::
 
     zip -j ../src/fuzzers/pop3-corpus.zip min-corpus/*
 
+pcap-based Corpus
+`````````````````
+
+A corpus can also be generated from representative pcp files using the
+``pcap-to-pkt`` application from pcap_simplify_. The fuzzers only handle a
+single connection at a time, so pcap files with multiple connections will
+need to be split using ``PcapSplitter`` from PcapPlusPlus_ or something
+similar. Once the file has been split, the individual connections can be
+converted into separate pkt files. The ``http`` fuzzer is a good example
+of a fuzzer using such files. The corpus for that fuzzer was initially
+generated from a subset of the pcap files located in ``testing/btest/Traces/http``.
+
+.. _pcap_simplify: https://github.com/JustinAzoff/pcap_simplify
+.. _PcapPlusPlus: https://github.com/seladb/PcapPlusPlus
+
+The converted pkt files can then be zipped as in the text-based section
+above.
+
 Example Build: Run Standalone Fuzz Targets
 ------------------------------------------
 

src/fuzzers/ftp-fuzzer.cc

@@ -0,0 +1,78 @@
+#include <binpac.h>
+
+#include "zeek/Conn.h"
+#include "zeek/RunState.h"
+#include "zeek/analyzer/Analyzer.h"
+#include "zeek/analyzer/Manager.h"
+#include "zeek/analyzer/protocol/pia/PIA.h"
+#include "zeek/analyzer/protocol/tcp/TCP.h"
+#include "zeek/fuzzers/FuzzBuffer.h"
+#include "zeek/fuzzers/fuzzer-setup.h"
+#include "zeek/packet_analysis/protocol/tcp/TCPSessionAdapter.h"
+#include "zeek/session/Manager.h"
+
+static constexpr auto ZEEK_FUZZ_ANALYZER = "ftp";
+
+static zeek::Connection* add_connection()
+	{
+	static constexpr double network_time_start = 1439471031;
+	zeek::run_state::detail::update_network_time(network_time_start);
+
+	zeek::Packet p;
+	zeek::ConnTuple conn_id;
+	conn_id.src_addr = zeek::IPAddr("1.2.3.4");
+	conn_id.dst_addr = zeek::IPAddr("5.6.7.8");
+	conn_id.src_port = htons(23132);
+	conn_id.dst_port = htons(80);
+	conn_id.is_one_way = false;
+	conn_id.proto = TRANSPORT_TCP;
+	zeek::detail::ConnKey key(conn_id);
+	zeek::Connection* conn = new zeek::Connection(key, network_time_start, &conn_id, 1, &p);
+	conn->SetTransport(TRANSPORT_TCP);
+	zeek::session_mgr->Insert(conn);
+	return conn;
+	}
+
+static zeek::analyzer::Analyzer* add_analyzer(zeek::Connection* conn)
+	{
+	auto* tcp = new zeek::packet_analysis::TCP::TCPSessionAdapter(conn);
+	auto* pia = new zeek::analyzer::pia::PIA_TCP(conn);
+	auto a = zeek::analyzer_mgr->InstantiateAnalyzer(ZEEK_FUZZ_ANALYZER, conn);
+	tcp->AddChildAnalyzer(a);
+	tcp->AddChildAnalyzer(pia->AsAnalyzer());
+	conn->SetSessionAdapter(tcp, pia);
+	return a;
+	}
+
+extern "C" int LLVMFuzzerTestOneInput(const uint8_t* data, size_t size)
+	{
+	zeek::detail::FuzzBuffer fb{data, size};
+
+	if ( ! fb.Valid() )
+		return 0;
+
+	auto conn = add_connection();
+	auto a = add_analyzer(conn);
+
+	for ( ;; )
+		{
+		auto chunk = fb.Next();
+
+		if ( ! chunk )
+			break;
+
+		try
+			{
+			a->DeliverStream(chunk->size, chunk->data.get(), chunk->is_orig);
+			}
+		catch ( const binpac::Exception& e )
+			{
+			}
+
+		chunk = {};
+		zeek::event_mgr.Drain();
+		}
+
+	zeek::detail::fuzzer_cleanup_one_input();
+	return 0;
+	}

src/fuzzers/fuzzer-setup.h

@@ -4,6 +4,7 @@
 #include <cstdlib>
 
 #include "zeek/Event.h"
+#include "zeek/EventRegistry.h"
 #include "zeek/broker/Manager.h"
 #include "zeek/file_analysis/Manager.h"
 #include "zeek/session/Manager.h"
@@ -41,6 +42,11 @@ extern "C" int LLVMFuzzerInitialize(int* argc, char*** argv)
 	if ( zeek::detail::setup(*argc, *argv, &options).code )
 		abort();
 
+	// We have to trick the event handlers into returning true that they exist here
+	// even if they don't, because otherwise we lose a bit of coverage where if
+	// statements return false that would otherwise not.
+	zeek::event_registry->ActivateAllHandlers();
+
 	return 0;
 	}
 

src/fuzzers/http-fuzzer.cc

@@ -0,0 +1,78 @@
+#include <binpac.h>
+
+#include "zeek/Conn.h"
+#include "zeek/RunState.h"
+#include "zeek/analyzer/Analyzer.h"
+#include "zeek/analyzer/Manager.h"
+#include "zeek/analyzer/protocol/pia/PIA.h"
+#include "zeek/analyzer/protocol/tcp/TCP.h"
+#include "zeek/fuzzers/FuzzBuffer.h"
+#include "zeek/fuzzers/fuzzer-setup.h"
+#include "zeek/packet_analysis/protocol/tcp/TCPSessionAdapter.h"
+#include "zeek/session/Manager.h"
+
+static constexpr auto ZEEK_FUZZ_ANALYZER = "http";
+
+static zeek::Connection* add_connection()
+	{
+	static constexpr double network_time_start = 1439471031;
+	zeek::run_state::detail::update_network_time(network_time_start);
+
+	zeek::Packet p;
+	zeek::ConnTuple conn_id;
+	conn_id.src_addr = zeek::IPAddr("1.2.3.4");
+	conn_id.dst_addr = zeek::IPAddr("5.6.7.8");
+	conn_id.src_port = htons(23132);
+	conn_id.dst_port = htons(80);
+	conn_id.is_one_way = false;
+	conn_id.proto = TRANSPORT_TCP;
+	zeek::detail::ConnKey key(conn_id);
+	zeek::Connection* conn = new zeek::Connection(key, network_time_start, &conn_id, 1, &p);
+	conn->SetTransport(TRANSPORT_TCP);
+	zeek::session_mgr->Insert(conn);
+	return conn;
+	}
+
+static zeek::analyzer::Analyzer* add_analyzer(zeek::Connection* conn)
+	{
+	auto* tcp = new zeek::packet_analysis::TCP::TCPSessionAdapter(conn);
+	auto* pia = new zeek::analyzer::pia::PIA_TCP(conn);
+	auto a = zeek::analyzer_mgr->InstantiateAnalyzer(ZEEK_FUZZ_ANALYZER, conn);
+	tcp->AddChildAnalyzer(a);
+	tcp->AddChildAnalyzer(pia->AsAnalyzer());
+	conn->SetSessionAdapter(tcp, pia);
+	return a;
+	}
+
+extern "C" int LLVMFuzzerTestOneInput(const uint8_t* data, size_t size)
+	{
+	zeek::detail::FuzzBuffer fb{data, size};
+
+	if ( ! fb.Valid() )
+		return 0;
+
+	auto conn = add_connection();
+	auto a = add_analyzer(conn);
+
+	for ( ;; )
+		{
+		auto chunk = fb.Next();
+
+		if ( ! chunk )
+			break;
+
+		try
+			{
+			a->DeliverStream(chunk->size, chunk->data.get(), chunk->is_orig);
+			}
+		catch ( const binpac::Exception& e )
+			{
+			}
+
+		chunk = {};
+		zeek::event_mgr.Drain();
+		}
+
+	zeek::detail::fuzzer_cleanup_one_input();
+	return 0;
+	}

src/fuzzers/imap-fuzzer.cc

@@ -0,0 +1,78 @@
+#include <binpac.h>
+
+#include "zeek/Conn.h"
+#include "zeek/RunState.h"
+#include "zeek/analyzer/Analyzer.h"
+#include "zeek/analyzer/Manager.h"
+#include "zeek/analyzer/protocol/pia/PIA.h"
+#include "zeek/analyzer/protocol/tcp/TCP.h"
+#include "zeek/fuzzers/FuzzBuffer.h"
+#include "zeek/fuzzers/fuzzer-setup.h"
+#include "zeek/packet_analysis/protocol/tcp/TCPSessionAdapter.h"
+#include "zeek/session/Manager.h"
+
+static constexpr auto ZEEK_FUZZ_ANALYZER = "imap";
+
+static zeek::Connection* add_connection()
+	{
+	static constexpr double network_time_start = 1439471031;
+	zeek::run_state::detail::update_network_time(network_time_start);
+
+	zeek::Packet p;
+	zeek::ConnTuple conn_id;
+	conn_id.src_addr = zeek::IPAddr("1.2.3.4");
+	conn_id.dst_addr = zeek::IPAddr("5.6.7.8");
+	conn_id.src_port = htons(23132);
+	conn_id.dst_port = htons(80);
+	conn_id.is_one_way = false;
+	conn_id.proto = TRANSPORT_TCP;
+	zeek::detail::ConnKey key(conn_id);
+	zeek::Connection* conn = new zeek::Connection(key, network_time_start, &conn_id, 1, &p);
+	conn->SetTransport(TRANSPORT_TCP);
+	zeek::session_mgr->Insert(conn);
+	return conn;
+	}
+
+static zeek::analyzer::Analyzer* add_analyzer(zeek::Connection* conn)
+	{
+	auto* tcp = new zeek::packet_analysis::TCP::TCPSessionAdapter(conn);
+	auto* pia = new zeek::analyzer::pia::PIA_TCP(conn);
+	auto a = zeek::analyzer_mgr->InstantiateAnalyzer(ZEEK_FUZZ_ANALYZER, conn);
+	tcp->AddChildAnalyzer(a);
+	tcp->AddChildAnalyzer(pia->AsAnalyzer());
+	conn->SetSessionAdapter(tcp, pia);
+	return a;
+	}
+
+extern "C" int LLVMFuzzerTestOneInput(const uint8_t* data, size_t size)
+	{
+	zeek::detail::FuzzBuffer fb{data, size};
+
+	if ( ! fb.Valid() )
+		return 0;
+
+	auto conn = add_connection();
+	auto a = add_analyzer(conn);
+
+	for ( ;; )
+		{
+		auto chunk = fb.Next();
+
+		if ( ! chunk )
+			break;
+
+		try
+			{
+			a->DeliverStream(chunk->size, chunk->data.get(), chunk->is_orig);
+			}
+		catch ( const binpac::Exception& e )
+			{
+			}
+
+		chunk = {};
+		zeek::event_mgr.Drain();
+		}
+
+	zeek::detail::fuzzer_cleanup_one_input();
+	return 0;
+	}

src/fuzzers/smtp-fuzzer.cc

@@ -0,0 +1,78 @@
+#include <binpac.h>
+
+#include "zeek/Conn.h"
+#include "zeek/RunState.h"
+#include "zeek/analyzer/Analyzer.h"
+#include "zeek/analyzer/Manager.h"
+#include "zeek/analyzer/protocol/pia/PIA.h"
+#include "zeek/analyzer/protocol/tcp/TCP.h"
+#include "zeek/fuzzers/FuzzBuffer.h"
+#include "zeek/fuzzers/fuzzer-setup.h"
+#include "zeek/packet_analysis/protocol/tcp/TCPSessionAdapter.h"
+#include "zeek/session/Manager.h"
+
+static constexpr auto ZEEK_FUZZ_ANALYZER = "smtp";
+
+static zeek::Connection* add_connection()
+	{
+	static constexpr double network_time_start = 1439471031;
+	zeek::run_state::detail::update_network_time(network_time_start);
+
+	zeek::Packet p;
+	zeek::ConnTuple conn_id;
+	conn_id.src_addr = zeek::IPAddr("1.2.3.4");
+	conn_id.dst_addr = zeek::IPAddr("5.6.7.8");
+	conn_id.src_port = htons(23132);
+	conn_id.dst_port = htons(80);
+	conn_id.is_one_way = false;
+	conn_id.proto = TRANSPORT_TCP;
+	zeek::detail::ConnKey key(conn_id);
+	zeek::Connection* conn = new zeek::Connection(key, network_time_start, &conn_id, 1, &p);
+	conn->SetTransport(TRANSPORT_TCP);
+	zeek::session_mgr->Insert(conn);
+	return conn;
+	}
+
+static zeek::analyzer::Analyzer* add_analyzer(zeek::Connection* conn)
+	{
+	auto* tcp = new zeek::packet_analysis::TCP::TCPSessionAdapter(conn);
+	auto* pia = new zeek::analyzer::pia::PIA_TCP(conn);
+	auto a = zeek::analyzer_mgr->InstantiateAnalyzer(ZEEK_FUZZ_ANALYZER, conn);
+	tcp->AddChildAnalyzer(a);
+	tcp->AddChildAnalyzer(pia->AsAnalyzer());
+	conn->SetSessionAdapter(tcp, pia);
+	return a;
+	}
+
+extern "C" int LLVMFuzzerTestOneInput(const uint8_t* data, size_t size)
+	{
+	zeek::detail::FuzzBuffer fb{data, size};
+
+	if ( ! fb.Valid() )
+		return 0;
+
+	auto conn = add_connection();
+	auto a = add_analyzer(conn);
+
+	for ( ;; )
+		{
+		auto chunk = fb.Next();
+
+		if ( ! chunk )
+			break;
+
+		try
+			{
+			a->DeliverStream(chunk->size, chunk->data.get(), chunk->is_orig);
+			}
+		catch ( const binpac::Exception& e )
+			{
+			}
+
+		chunk = {};
+		zeek::event_mgr.Drain();
+		}
+
+	zeek::detail::fuzzer_cleanup_one_input();
+	return 0;
+	}

src/input/readers/config/Config.cc

@@ -184,7 +184,7 @@ bool Config::DoUpdate()
 	std::regex re;
 	try
 		{
-		std::string re_str = util::fmt(
+		std::string re_str = Fmt(
 			"^([^[:blank:]]+)[[:blank:]]+([^[:blank:]](.*[^[:blank:]%c])?)?[[:blank:]%c]*$",
 			set_separator[0], set_separator[0]);
 		re.assign(re_str, std::regex::extended);

src/option.bif

@@ -10,6 +10,11 @@ module Option;
 static bool call_option_handlers_and_set_value(zeek::StringVal* name, const zeek::detail::IDPtr& i,
                                                zeek::ValPtr val, zeek::StringVal* location)
 	{
+	// when shutting down, don't call back into script land handlers as
+	// these may use resources that have already been shutdown.
+	if ( zeek::run_state::terminating )
+		return false;
+
 	if ( i->HasOptionHandlers() )
 		{
 		for ( auto handler_function : i->GetOptionHandlers() )

src/packet_analysis/protocol/teredo/Teredo.cc

@@ -145,7 +145,7 @@ TeredoAnalyzer::TeredoAnalyzer() : zeek::packet_analysis::Analyzer("TEREDO")
 	// 	}
 
 	pattern_re = std::make_unique<zeek::detail::Specific_RE_Matcher>(zeek::detail::MATCH_EXACTLY,
-	                                                                 1);
+	                                                                 true);
 	pattern_re->AddPat("^(\\x00\\x00)|(\\x00\\x01)|([\\x60-\\x6f].{7}((\\x20\\x01\\x00\\x00)).{28})"
 	                   "|([\\x60-\\x6f].{23}((\\x20\\x01\\x00\\x00))).{12}");
 	pattern_re->Compile();

src/parse.y

@@ -54,7 +54,7 @@
 %left '$' '[' ']' '(' ')' TOK_HAS_FIELD TOK_HAS_ATTR
 %nonassoc TOK_AS TOK_IS
 
-%type <b> opt_no_test opt_no_test_block TOK_PATTERN_END opt_deep when_flavor
+%type <b> opt_no_test opt_no_test_block opt_deep when_flavor
 %type <str> TOK_ID TOK_PATTERN_TEXT
 %type <id> local_id global_id def_global_id event_id global_or_event_id resolve_id begin_lambda case_type
 %type <id_l> local_id_list case_type_list
@@ -77,6 +77,7 @@
 %type <capture> capture
 %type <captures> capture_list opt_captures when_captures
 %type <when_clause> when_head when_start when_clause
+%type <re_modes> TOK_PATTERN_END
 
 %{
 #include <cstdlib>
@@ -155,6 +156,8 @@ static int func_hdr_cond_epoch = 0;
 EnumType* cur_enum_type = nullptr;
 static ID* cur_decl_type_id = nullptr;
 
+std::set<std::string> module_names;
+
 static void parse_new_enum(void)
 	{
 	// Starting a new enum definition.
@@ -186,6 +189,56 @@ static void parse_redef_enum(ID* id)
 		}
 	}
 
+static void parse_redef_record_field(ID* id, const char* field, InitClass ic,
+                                     std::unique_ptr<std::vector<AttrPtr>> attrs)
+	{
+	if ( ! id->GetType() )
+		{
+		reporter->FatalError("unknown record identifier \"%s\"", id->Name());
+		return;
+		}
+
+	auto t = id->GetType();
+	if ( ! t || t->Tag() != TYPE_RECORD )
+		{
+		reporter->FatalError("identifier \"%s\" has type \"%s\", expected \"record\"",
+		                     id->Name(), type_name(t->Tag()));
+		return;
+		}
+
+	auto rt = t->AsRecordType();
+	auto idx = rt->FieldOffset(field);
+	if ( idx < 0 )
+		{
+		reporter->FatalError("field \"%s\" not in record \"%s\"", field, id->Name());
+		return;
+		}
+
+	auto decl = rt->FieldDecl(idx);
+	if ( ! decl->attrs )
+		if ( ic == INIT_EXTRA )
+			decl->attrs = make_intrusive<detail::Attributes>(decl->type,
+															 true /* in_record */,
+															 false /* is_global */);
+
+	for ( const auto& attr : *attrs )
+		{
+		// At this point, only support &log redef'ing.
+		if ( attr->Tag() != ATTR_LOG )
+			{
+				reporter->FatalError("Can only redef \"&log\" attributes of record fields");
+				return;
+			}
+
+		if ( ic == INIT_EXTRA )
+			decl->attrs->AddAttr(attr, true /* is_redef */);
+		else
+			// Removing attributes is a noop if they don't exist.
+			if ( decl->attrs )
+				decl->attrs->RemoveAttr(attr->Tag());
+		}
+	}
+
 static void extend_record(ID* id, std::unique_ptr<type_decl_list> fields,
                           std::unique_ptr<std::vector<AttrPtr>> attrs)
 	{
@@ -324,6 +377,11 @@ static StmtPtr build_local(ID* id, Type* t, InitClass ic, Expr* e,
 	zeek::FuncType::Capture* capture;
 	zeek::FuncType::CaptureList* captures;
 	zeek::detail::WhenInfo* when_clause;
+	struct
+		{
+		bool ignore_case;
+		bool single_line;
+		} re_modes;
 }
 
 %%
@@ -912,9 +970,12 @@ expr:
 			auto* re = new RE_Matcher($3);
 			delete [] $3;
 
-			if ( $4 )
+			if ( $4.ignore_case )
 				re->MakeCaseInsensitive();
 
+			if ( $4.single_line )
+				re->MakeSingleLine();
+
 			re->Compile();
 			$$ = new ConstExpr(make_intrusive<PatternVal>(re));
 			}
@@ -1260,6 +1321,7 @@ decl:
 		TOK_MODULE TOK_ID ';'
 			{
 			current_module = $2;
+			module_names.insert($2);
 			zeekygen_mgr->ModuleUsage(::filename, current_module);
 			}
 
@@ -1294,6 +1356,20 @@ decl:
 			// Zeekygen already grabbed new enum IDs as the type created them.
 			}
 
+	|	TOK_REDEF TOK_RECORD  global_id '$' TOK_ID
+			{ cur_decl_type_id = $3; zeekygen_mgr->Redef($3, ::filename, INIT_EXTRA); }
+		TOK_ADD_TO '{' attr_list '}' ';'
+			{
+			cur_decl_type_id = 0;
+			parse_redef_record_field($3, $5, INIT_EXTRA, std::unique_ptr<std::vector<AttrPtr>>($9));
+			}
+	|	TOK_REDEF TOK_RECORD  global_id '$' TOK_ID
+			{ cur_decl_type_id = $3; zeekygen_mgr->Redef($3, ::filename, INIT_REMOVE); }
+		TOK_REMOVE_FROM '{' attr_list '}' ';'
+			{
+			cur_decl_type_id = 0;
+			parse_redef_record_field($3, $5, INIT_REMOVE, std::unique_ptr<std::vector<AttrPtr>>($9));
+			}
 	|	TOK_REDEF TOK_RECORD global_id
 			{ cur_decl_type_id = $3; zeekygen_mgr->Redef($3, ::filename); }
 		TOK_ADD_TO '{'
@@ -1311,7 +1387,7 @@ decl:
 				              std::unique_ptr<std::vector<AttrPtr>>($11));
 			}
 
-	|	TOK_TYPE global_id ':'
+	|	TOK_TYPE def_global_id ':'
 			{ cur_decl_type_id = $2; zeekygen_mgr->StartType({NewRef{}, $2});  }
 		type opt_attr ';'
 			{

src/re-parse.y

@@ -20,7 +20,7 @@ namespace zeek::detail {
 void yyerror(const char msg[]);
 %}
 
-%token TOK_CHAR TOK_NUMBER TOK_CCL TOK_CCE TOK_CASE_INSENSITIVE
+%token TOK_CHAR TOK_NUMBER TOK_CCL TOK_CCE TOK_CASE_INSENSITIVE TOK_SINGLE_LINE
 
 %union {
 	int int_val;
@@ -112,7 +112,8 @@ singleton	:  singleton '*'
 
 		|  '.'
 			{
-			$$ = new zeek::detail::NFA_Machine(new zeek::detail::NFA_State(zeek::detail::rem->AnyCCL()));
+			$$ = new zeek::detail::NFA_Machine(new zeek::detail::NFA_State(
+                zeek::detail::rem->AnyCCL(zeek::detail::re_single_line)));
 			}
 
 		|  full_ccl
@@ -132,7 +133,10 @@ singleton	:  singleton '*'
 			{ $$ = $2; }
 
 		|  TOK_CASE_INSENSITIVE re ')'
-			{ $$ = $2; zeek::detail::case_insensitive = 0; }
+			{ $$ = $2; zeek::detail::case_insensitive = false; }
+
+		|  TOK_SINGLE_LINE re ')'
+			{ $$ = $2; zeek::detail::re_single_line = false; }
 
 		|  TOK_CHAR
 			{

src/re-scan.l

@@ -115,7 +115,8 @@ CCL_EXPR	("[:"[[:alpha:]]+":]")
 				}
 			}
 
-	"(?i:"		zeek::detail::case_insensitive = 1; return TOK_CASE_INSENSITIVE;
+	"(?i:"		zeek::detail::case_insensitive = true; return TOK_CASE_INSENSITIVE;
+	"(?s:"		zeek::detail::re_single_line = true; return TOK_SINGLE_LINE;
 
 	[a-zA-Z]	{
 			if ( zeek::detail::case_insensitive )

src/scan.l

@@ -80,6 +80,8 @@ const char* last_tok_filename = 0;
 const char* last_last_tok_filename = 0;
 char last_tok[128];
 
+extern std::set<std::string> module_names;
+
 #define YY_USER_ACTION	strncpy(last_tok, yytext, sizeof(last_tok) - 1); \
                         last_last_tok_filename = last_tok_filename; \
                         last_tok_filename = ::filename;
@@ -562,13 +564,28 @@ F	RET_CONST(zeek::val_mgr->False()->Ref())
 
 <RE>"/"	{
 	BEGIN(INITIAL);
-	yylval.b = false;
+	yylval.re_modes.ignore_case = false;
+	yylval.re_modes.single_line = false;
 	return TOK_PATTERN_END;
 	}
 
-<RE>"/i"	{
+<RE>(\/[is]{0,2})	{
 	BEGIN(INITIAL);
-	yylval.b = true;
+
+	if ( strlen(yytext) == 2 )
+		{
+		yylval.re_modes.ignore_case = (yytext[1] == 'i');
+		yylval.re_modes.single_line = (yytext[1] == 's');
+		}
+	else
+		{
+		if ( yytext[1] == yytext[2] )
+			zeek::reporter->Error("pattern has duplicate mode %c", yytext[1]);
+
+		yylval.re_modes.ignore_case = (yytext[1] == 'i' || yytext[2] == 'i');
+		yylval.re_modes.single_line = (yytext[1] == 's' || yytext[2] == 's');
+		}
+
 	return TOK_PATTERN_END;
 	}
 
@@ -781,7 +798,12 @@ void do_atifdef(const char* id)
 
 	const auto& i = zeek::detail::lookup_ID(id, zeek::detail::current_module.c_str());
 
-	if ( ! i )
+	// If the ID didn't exist, look to see if this is a module name instead.
+	bool found = (i != nullptr);
+	if ( ! found )
+		found = (module_names.count(id) != 0);
+
+	if ( ! found )
 		begin_ignoring();
 	}
 
@@ -791,7 +813,12 @@ void do_atifndef(const char *id)
 
 	const auto& i = zeek::detail::lookup_ID(id, zeek::detail::current_module.c_str());
 
-	if ( i )
+	// If the ID didn't exist, look to see if this is a module name instead.
+	bool found = (i != nullptr);
+	if ( ! found )
+		found = (module_names.count(id) != 0);
+
+	if ( found )
 		begin_ignoring();
 	}
 

src/script_opt/CPP/RuntimeVec.cc

@@ -170,6 +170,8 @@ VEC_OP2(or, |, )
 VEC_OP2(xor, ^, )
 VEC_OP2(andand, &&, )
 VEC_OP2(oror, ||, )
+VEC_OP2(lshift, <<, )
+VEC_OP2(rshift, >>, )
 
 // A version of VEC_OP2 that instead supports relational operations, so
 // the result type is always vector-of-bool.

src/script_opt/CPP/RuntimeVec.h

@@ -45,6 +45,8 @@ extern VectorValPtr vec_op_or__CPP(const VectorValPtr& v1, const VectorValPtr& v
 extern VectorValPtr vec_op_xor__CPP(const VectorValPtr& v1, const VectorValPtr& v2);
 extern VectorValPtr vec_op_andand__CPP(const VectorValPtr& v1, const VectorValPtr& v2);
 extern VectorValPtr vec_op_oror__CPP(const VectorValPtr& v1, const VectorValPtr& v2);
+extern VectorValPtr vec_op_lshift__CPP(const VectorValPtr& v1, const VectorValPtr& v2);
+extern VectorValPtr vec_op_rshift__CPP(const VectorValPtr& v1, const VectorValPtr& v2);
 
 // Vector relational operations.
 extern VectorValPtr vec_op_lt__CPP(const VectorValPtr& v1, const VectorValPtr& v2);

src/script_opt/CPP/Stmts.cc

@@ -495,7 +495,7 @@ void CPPCompile::GenForOverTable(const ExprPtr& tbl, const IDPtr& value_var,
 	StartBlock();
 
 	Emit("auto k__CPP = lve__CPP.GetHashKey();");
-	Emit("auto* current_tev__CPP = lve__CPP.GetValue<TableEntryVal*>();");
+	Emit("auto* current_tev__CPP = lve__CPP.value;");
 	Emit("auto ind_lv__CPP = tv__CPP->RecreateIndex(*k__CPP);");
 
 	if ( value_var )

src/script_opt/CPP/maint/README

@@ -15,7 +15,7 @@ The maintenance workflow:
     to check in updates to the list of how the compiler currently fares
     on various btests (see end of this doc):
 
-	Thu May 12 12:54:10 PDT 2022
+	Mon Aug  1 16:39:05 PDT 2022
 
 2.  Run "find-test-files.sh" to generate a list (to stdout) of all of the
     possible Zeek source files found in the test suite.
@@ -57,17 +57,27 @@ These BTests won't successfully run due to the indicated issue:
 	command-line-error - a deliberate command-line error
 	complex-to-debug - hard-to-figure-out failure
 	deprecated - uses features deprecated for -O C++
+	error-handling - behavior in face of an error differs
+	needs-plugin - requires knowing how to build an associated plugin
 	no-script - there's no actual script to compile
 	ZAM - meant specifically for -O ZAM
 
+Consider migrating these to have @TEST-REQUIRES clauses so we don't have
+to maintain this list.
+
 ../testing/btest/core/negative-time.test no-script
 ../testing/btest/core/pcap/dumper.zeek no-script
 ../testing/btest/core/pcap/input-error.zeek command-line-error
 ../testing/btest/core/proc-status-file.zeek no-script
+../testing/btest/core/scalar-vector.zeek deprecated
 ../testing/btest/language/at-if-event.zeek @if
 ../testing/btest/language/at-if.zeek @if
 ../testing/btest/language/at-ifdef.zeek @if
 ../testing/btest/language/at-ifndef.zeek @if
+../testing/btest/language/incr-vec-expr.test deprecated
+../testing/btest/language/uninitialized-local2.zeek error-handling
+../testing/btest/language/vector-deprecated.zeek deprecated
+../testing/btest/language/vector-in-operator.zeek
 ../testing/btest/language/vector-in-operator.zeek deprecated
 ../testing/btest/language/when-aggregates.zeek bad-when
 ../testing/btest/opt/opt-files.zeek ZAM
@@ -76,6 +86,7 @@ These BTests won't successfully run due to the indicated issue:
 ../testing/btest/opt/opt-func.zeek ZAM
 ../testing/btest/opt/opt-func2.zeek ZAM
 ../testing/btest/opt/opt-func3.zeek ZAM
+../testing/btest/plugins/packet-protocol.zeek needs-plugin
 ../testing/btest/scripts/base/protocols/dhcp/dhcp-ack-msg-types.zeek no-script
 ../testing/btest/scripts/base/protocols/dhcp/dhcp-all-msg-types.zeek no-script
 ../testing/btest/scripts/base/protocols/dhcp/dhcp-discover-msg-types.zeek no-script

src/script_opt/CPP/maint/check-CPP-gen.sh

@@ -1,8 +1,19 @@
 #! /bin/sh
 
-out=out.$(echo $1 | sed 's,\.\./,,;s,/,#,g')
+abbr=$(echo $1 | sed 's,\.\./,,;s,/,#,g')
+out=CPP-test/out.$abbr
+gen_out=CPP-test/gen.$abbr
 
 (
     /bin/echo -n $1" "
-    (src/zeek -O gen-C++ --optimize-files=testing/btest --optimize-func="<global-stmts>" $1 >&/dev/null && echo "success") || echo "fail"
+    if ! src/zeek -O gen-C++ --optimize-files=testing/btest $1 >&$gen_out 2>&1; then
-) >CPP-test/$out 2>&1
+        echo "fail"
+        exit 1
+    fi
+    if grep -E -q 'deprecated|skipping|cannot compile|no matching functions' $gen_out; then
+        echo "fail"
+        exit 1
+    fi
+    echo "success"
+    exit 0
+) >$out 2>&1

src/script_opt/CPP/maint/do-CPP-btest.sh

@@ -1,11 +1,22 @@
 #! /bin/sh
 
 rm -f CPP-gen.cc
+
 cp zeek.HOLD src/zeek || (
     echo Need to create clean zeek.HOLD
     exit 1
 ) || exit 1
 
+if [ "$1" == "-U" ]; then
+    btest_opt=-U
+    shift
+elif [ "$1" == "-d" ]; then
+    btest_opt=-d
+    shift
+else
+    btest_opt=-d
+fi
+
 base=$(echo $1 | sed 's,\.\./,,;s,/,#,g')
 rel_test=$(echo $1 | sed 's,.*testing/btest/,,')
 
@@ -26,5 +37,5 @@ ninja
 
 (
     cd ../testing/btest
-    ../../auxil/btest/btest -a cpp -d -f ../../build/CPP-test/diag.$base $rel_test
+    ../../auxil/btest/btest -a cpp $btest_opt -f ../../build/CPP-test/diag.$base $rel_test
 )

src/script_opt/ZAM/Ops.in

@@ -543,6 +543,19 @@ op-type U
 vector
 eval $1 ^ $2
 
+binary-expr-op Lshift
+op-type I U
+vector
+eval-type I	if ( $1 < 0 )
+			ZAM_run_time_error(z.loc, "left shifting a negative number is undefined");
+		$$ = $1 << $2;
+eval $1 << $2
+
+binary-expr-op Rshift
+op-type I U
+vector
+eval $1 >> $2
+
 ########## Relationals ##########
 
 rel-expr-op LT
@@ -833,13 +846,13 @@ internal-op Val-Is-In-Vector
 type VVV
 eval	auto& vec = frame[z.v3].vector_val;
 	auto ind = frame[z.v2].int_val;
-	frame[z.v1].int_val = ind >= 0 && static_cast<zeek_uint_t>(ind) < vec->Size();
+	frame[z.v1].int_val = vec->Has(ind);
 
 internal-op Const-Is-In-Vector
 type VCV
 eval	auto& vec = frame[z.v2].vector_val;
 	auto ind = z.c.int_val;
-	frame[z.v1].int_val = ind >= 0 && static_cast<zeek_uint_t>(ind) < vec->Size();
+	frame[z.v1].int_val = vec->Has(ind);
 
 expr-op Cond
 type VVVV

src/telemetry/CMakeLists.txt

@@ -12,4 +12,4 @@ set(telemetry_SRCS
 bif_target(telemetry.bif)
 
 bro_add_subdir_library(telemetry ${telemetry_SRCS})
-
+add_dependencies(bro_telemetry generate_outputs)

src/telemetry/Manager.cc

@@ -2,11 +2,15 @@
 
 #include "zeek/telemetry/Manager.h"
 
+#include <fnmatch.h>
 #include <thread>
+#include <variant>
 
 #include "zeek/3rdparty/doctest.h"
+#include "zeek/ID.h"
 #include "zeek/broker/Manager.h"
 #include "zeek/telemetry/Timer.h"
+#include "zeek/telemetry/telemetry.bif.h"
 
 #include "broker/telemetry/metric_registry.hh"
 
@@ -15,6 +19,33 @@ namespace
 using NativeManager = broker::telemetry::metric_registry;
 using NativeManagerImpl = broker::telemetry::metric_registry_impl;
 using NativeManagerImplPtr = zeek::IntrusivePtr<NativeManagerImpl>;
+using DoubleValPtr = zeek::IntrusivePtr<zeek::DoubleVal>;
+
+std::vector<std::string_view> extract_label_values(broker::telemetry::const_label_list labels)
+	{
+	auto get_value = [](const auto& label)
+	{
+		return label.second;
+	};
+	std::vector<std::string_view> v;
+	std::transform(labels.begin(), labels.end(), std::back_inserter(v), get_value);
+	return v;
+	}
+
+// Convert an int64_t or double to a DoubleValPtr. int64_t is casted.
+template <typename T> DoubleValPtr as_double_val(T val)
+	{
+	if constexpr ( std::is_same_v<T, int64_t> )
+		{
+		return zeek::make_intrusive<zeek::DoubleVal>(static_cast<double>(val));
+		}
+	else
+		{
+		static_assert(std::is_same_v<T, double>);
+		return zeek::make_intrusive<zeek::DoubleVal>(val);
+		}
+	};
+
 	}
 
 namespace zeek::telemetry
@@ -38,6 +69,430 @@ void Manager::InitPostBrokerSetup(broker::endpoint& ep)
 	pimpl.swap(ptr);
 	}
 
+// -- collect metric stuff -----------------------------------------------------
+
+template <typename T>
+zeek::RecordValPtr Manager::GetMetricOptsRecord(Manager::MetricType metric_type,
+                                                const broker::telemetry::metric_family_hdl* family)
+	{
+	static auto string_vec_type = zeek::id::find_type<zeek::VectorType>("string_vec");
+	static auto double_vec_type = zeek::id::find_type<zeek::VectorType>("double_vec");
+	static auto count_vec_type = zeek::id::find_type<zeek::VectorType>("index_vec");
+	static auto metric_opts_type = zeek::id::find_type<zeek::RecordType>("Telemetry::MetricOpts");
+
+	static auto prefix_idx = metric_opts_type->FieldOffset("prefix");
+	static auto name_idx = metric_opts_type->FieldOffset("name");
+	static auto help_text_idx = metric_opts_type->FieldOffset("help_text");
+	static auto unit_idx = metric_opts_type->FieldOffset("unit");
+	static auto is_total_idx = metric_opts_type->FieldOffset("is_total");
+	static auto labels_idx = metric_opts_type->FieldOffset("labels");
+	static auto bounds_idx = metric_opts_type->FieldOffset("bounds");
+	static auto metric_type_idx = metric_opts_type->FieldOffset("metric_type");
+
+	if ( const auto& it = metric_opts_cache.find(family); it != metric_opts_cache.end() )
+		return it->second;
+
+	auto r = make_intrusive<zeek::RecordVal>(metric_opts_type);
+	r->Assign(prefix_idx, make_intrusive<zeek::StringVal>(broker::telemetry::prefix(family)));
+	r->Assign(name_idx, make_intrusive<zeek::StringVal>(broker::telemetry::name(family)));
+	r->Assign(help_text_idx, make_intrusive<zeek::StringVal>(broker::telemetry::helptext(family)));
+	r->Assign(unit_idx, make_intrusive<zeek::StringVal>(broker::telemetry::unit(family)));
+	r->Assign(is_total_idx, val_mgr->Bool(broker::telemetry::is_sum(family)));
+
+	auto label_names_vec = make_intrusive<zeek::VectorVal>(string_vec_type);
+	for ( const auto& l : broker::telemetry::label_names(family) )
+		label_names_vec->Append(make_intrusive<StringVal>(l));
+
+	r->Assign(labels_idx, label_names_vec);
+
+	// This is mapping Manager.h enums to bif values depending on
+	// the template type and whether this is a counter, gauge or
+	// histogram.
+	zeek_int_t metric_type_int = -1;
+	if constexpr ( std::is_same_v<T, double> )
+		{
+		switch ( metric_type )
+			{
+			case MetricType::Counter:
+				metric_type_int = BifEnum::Telemetry::MetricType::DOUBLE_COUNTER;
+				break;
+			case MetricType::Gauge:
+				metric_type_int = BifEnum::Telemetry::MetricType::DOUBLE_GAUGE;
+				break;
+			case MetricType::Histogram:
+				metric_type_int = BifEnum::Telemetry::MetricType::DOUBLE_HISTOGRAM;
+				break;
+			}
+		}
+	else
+		{
+		switch ( metric_type )
+			{
+			case MetricType::Counter:
+				metric_type_int = BifEnum::Telemetry::MetricType::INT_COUNTER;
+				break;
+			case MetricType::Gauge:
+				metric_type_int = BifEnum::Telemetry::MetricType::INT_GAUGE;
+				break;
+			case MetricType::Histogram:
+				metric_type_int = BifEnum::Telemetry::MetricType::INT_HISTOGRAM;
+				break;
+			}
+		}
+
+	if ( metric_type_int < 0 )
+		reporter->FatalError("Unable to lookup metric type %d", int(metric_type));
+
+	r->Assign(metric_type_idx,
+	          zeek::BifType::Enum::Telemetry::MetricType->GetEnumVal(metric_type_int));
+
+	// Add bounds and optionally count_bounds into the MetricOpts record.
+	static auto opts_rt = zeek::id::find_type<zeek::RecordType>("Telemetry::MetricOpts");
+	static auto opts_rt_idx_bounds = opts_rt->FieldOffset("bounds");
+	static auto opts_rt_idx_count_bounds = opts_rt->FieldOffset("count_bounds");
+
+	if ( metric_type == MetricType::Histogram )
+		{
+		auto add_double_bounds = [](auto& r, const auto* histogram_family)
+		{
+			size_t buckets = broker::telemetry::num_buckets(histogram_family);
+			auto bounds_vec = make_intrusive<zeek::VectorVal>(double_vec_type);
+			for ( size_t i = 0; i < buckets; i++ )
+				bounds_vec->Append(
+					as_double_val(broker::telemetry::upper_bound_at(histogram_family, i)));
+
+			r->Assign(opts_rt_idx_bounds, bounds_vec);
+		};
+
+		if constexpr ( std::is_same_v<T, int64_t> )
+			{
+			auto histogram_family = broker::telemetry::as_int_histogram_family(family);
+			add_double_bounds(r, histogram_family);
+
+			// Add count_bounds to int64_t histograms
+			size_t buckets = broker::telemetry::num_buckets(histogram_family);
+			auto count_bounds_vec = make_intrusive<zeek::VectorVal>(count_vec_type);
+			for ( size_t i = 0; i < buckets; i++ )
+				count_bounds_vec->Append(
+					val_mgr->Count(broker::telemetry::upper_bound_at(histogram_family, i)));
+
+			r->Assign(opts_rt_idx_count_bounds, count_bounds_vec);
+			}
+		else
+			{
+			static_assert(std::is_same_v<T, double>);
+			add_double_bounds(r, broker::telemetry::as_dbl_histogram_family(family));
+			}
+		}
+
+	metric_opts_cache.insert({family, r});
+
+	return r;
+	}
+
+zeek::RecordValPtr Manager::CollectedValueMetric::AsMetricRecord() const
+	{
+	static auto string_vec_type = zeek::id::find_type<zeek::VectorType>("string_vec");
+	static auto metric_record_type = zeek::id::find_type<zeek::RecordType>("Telemetry::Metric");
+	static auto opts_idx = metric_record_type->FieldOffset("opts");
+	static auto labels_idx = metric_record_type->FieldOffset("labels");
+	static auto value_idx = metric_record_type->FieldOffset("value");
+	static auto count_value_idx = metric_record_type->FieldOffset("count_value");
+
+	auto r = make_intrusive<zeek::RecordVal>(metric_record_type);
+
+	auto label_values_vec = make_intrusive<zeek::VectorVal>(string_vec_type);
+	for ( const auto& l : label_values )
+		label_values_vec->Append(make_intrusive<StringVal>(l));
+
+	r->Assign(labels_idx, label_values_vec);
+
+	auto fn = [&](auto val)
+	{
+		using val_t = decltype(val);
+		auto opts_record = telemetry_mgr->GetMetricOptsRecord<val_t>(metric_type, family);
+		r->Assign(opts_idx, opts_record);
+		r->Assign(value_idx, as_double_val(val));
+		if constexpr ( std::is_same_v<val_t, int64_t> )
+			r->Assign(count_value_idx, val_mgr->Count(val));
+	};
+
+	std::visit(fn, value);
+
+	return r;
+	}
+
+zeek::RecordValPtr Manager::CollectedHistogramMetric::AsHistogramMetricRecord() const
+	{
+	static auto string_vec_type = zeek::id::find_type<zeek::VectorType>("string_vec");
+	static auto double_vec_type = zeek::id::find_type<zeek::VectorType>("double_vec");
+	static auto count_vec_type = zeek::id::find_type<zeek::VectorType>("index_vec");
+	static auto histogram_metric_type = zeek::id::find_type<zeek::RecordType>(
+		"Telemetry::HistogramMetric");
+	static auto opts_idx = histogram_metric_type->FieldOffset("opts");
+	static auto labels_idx = histogram_metric_type->FieldOffset("labels");
+	static auto values_idx = histogram_metric_type->FieldOffset("values");
+	static auto count_values_idx = histogram_metric_type->FieldOffset("count_values");
+	static auto observations_idx = histogram_metric_type->FieldOffset("observations");
+	static auto sum_idx = histogram_metric_type->FieldOffset("sum");
+	static auto count_observations_idx = histogram_metric_type->FieldOffset("count_observations");
+	static auto count_sum_idx = histogram_metric_type->FieldOffset("count_sum");
+
+	auto r = make_intrusive<zeek::RecordVal>(histogram_metric_type);
+
+	auto label_values_vec = make_intrusive<zeek::VectorVal>(string_vec_type);
+	for ( const auto& l : label_values )
+		label_values_vec->Append(make_intrusive<StringVal>(l));
+
+	r->Assign(labels_idx, label_values_vec);
+
+	auto fn = [&](const auto& histogram_data)
+	{
+		using val_t = std::decay_t<decltype(histogram_data.sum)>;
+		auto opts_record = telemetry_mgr->GetMetricOptsRecord<val_t>(MetricType::Histogram, family);
+		r->Assign(opts_idx, opts_record);
+
+		val_t observations = 0;
+		auto values_vec = make_intrusive<zeek::VectorVal>(double_vec_type);
+		auto count_values_vec = make_intrusive<zeek::VectorVal>(count_vec_type);
+
+		for ( const auto& b : histogram_data.buckets )
+			{
+			observations += b.count;
+			values_vec->Append(as_double_val(b.count));
+			if constexpr ( std::is_same_v<val_t, int64_t> )
+				count_values_vec->Append(val_mgr->Count(b.count));
+			}
+
+		r->Assign(values_idx, values_vec);
+		r->Assign(sum_idx, as_double_val(histogram_data.sum));
+		r->Assign(observations_idx, as_double_val(observations));
+
+		// Add extra fields just for int64_t based histograms with type count
+		if constexpr ( std::is_same_v<val_t, int64_t> )
+			{
+			r->Assign(count_values_idx, count_values_vec);
+			r->Assign(count_sum_idx, val_mgr->Count(histogram_data.sum));
+			r->Assign(count_observations_idx, val_mgr->Count(observations));
+			}
+	};
+
+	std::visit(fn, histogram);
+
+	return r;
+	}
+
+/**
+ * Encapsulate matching of prefix and name against a broker::telemetry::metric_family_hdl
+ */
+class MetricFamilyMatcher
+	{
+public:
+	MetricFamilyMatcher(std::string_view prefix, std::string_view name)
+		: prefix_pattern(prefix), name_pattern(name)
+		{
+		}
+
+	/**
+	 * @return true if the given family's prefix and name match, else false;
+	 */
+	bool operator()(const broker::telemetry::metric_family_hdl* family)
+		{
+		auto prefix = std::string{broker::telemetry::prefix(family)};
+		auto name = std::string{broker::telemetry::name(family)};
+
+		return fnmatch(prefix_pattern.c_str(), prefix.c_str(), 0) != FNM_NOMATCH &&
+		       fnmatch(name_pattern.c_str(), name.c_str(), 0) != FNM_NOMATCH;
+		}
+
+private:
+	std::string prefix_pattern;
+	std::string name_pattern;
+	};
+
+/**
+ * A collector implementation for counters and gauges.
+ */
+class MetricsCollector : public broker::telemetry::metrics_collector
+	{
+	using MetricType = Manager::MetricType;
+
+public:
+	MetricsCollector(std::string_view prefix, std::string_view name) : matches(prefix, name) { }
+
+	void operator()(const broker::telemetry::metric_family_hdl* family,
+	                const broker::telemetry::dbl_counter_hdl* counter,
+	                broker::telemetry::const_label_list labels)
+		{
+		if ( matches(family) )
+			metrics.emplace_back(MetricType::Counter, family, extract_label_values(labels),
+			                     broker::telemetry::value(counter));
+		}
+
+	void operator()(const broker::telemetry::metric_family_hdl* family,
+	                const broker::telemetry::int_counter_hdl* counter,
+	                broker::telemetry::const_label_list labels)
+		{
+		if ( matches(family) )
+			metrics.emplace_back(MetricType::Counter, family, extract_label_values(labels),
+			                     broker::telemetry::value(counter));
+		}
+
+	void operator()(const broker::telemetry::metric_family_hdl* family,
+	                const broker::telemetry::dbl_gauge_hdl* gauge,
+	                broker::telemetry::const_label_list labels)
+		{
+		if ( matches(family) )
+			metrics.emplace_back(MetricType::Gauge, family, extract_label_values(labels),
+			                     broker::telemetry::value(gauge));
+		}
+
+	void operator()(const broker::telemetry::metric_family_hdl* family,
+	                const broker::telemetry::int_gauge_hdl* gauge,
+	                broker::telemetry::const_label_list labels)
+		{
+		if ( matches(family) )
+			metrics.emplace_back(MetricType::Gauge, family, extract_label_values(labels),
+			                     broker::telemetry::value(gauge));
+		}
+
+	void operator()(const broker::telemetry::metric_family_hdl* family,
+	                const broker::telemetry::dbl_histogram_hdl* histogram,
+	                broker::telemetry::const_label_list labels)
+		{
+		// Ignored
+		}
+
+	void operator()(const broker::telemetry::metric_family_hdl* family,
+	                const broker::telemetry::int_histogram_hdl* histogram,
+	                broker::telemetry::const_label_list labels)
+		{
+		// Ignored
+		}
+
+	std::vector<Manager::CollectedValueMetric>& GetResult() { return metrics; }
+
+private:
+	MetricFamilyMatcher matches;
+	std::vector<Manager::CollectedValueMetric> metrics;
+	};
+
+std::vector<Manager::CollectedValueMetric> Manager::CollectMetrics(std::string_view prefix,
+                                                                   std::string_view name)
+	{
+	auto collector = MetricsCollector(prefix, name);
+
+	pimpl->collect(collector);
+
+	return std::move(collector.GetResult());
+	}
+
+/**
+ * A collector implementation for histograms.
+ */
+class HistogramMetricsCollector : public broker::telemetry::metrics_collector
+	{
+	using MetricType = Manager::MetricType;
+
+public:
+	HistogramMetricsCollector(std::string_view prefix, std::string_view name)
+		: matches(prefix, name)
+		{
+		}
+
+	void operator()(const broker::telemetry::metric_family_hdl* family,
+	                const broker::telemetry::dbl_counter_hdl* counter,
+	                broker::telemetry::const_label_list labels)
+		{
+		// Ignored
+		}
+
+	void operator()(const broker::telemetry::metric_family_hdl* family,
+	                const broker::telemetry::int_counter_hdl* counter,
+	                broker::telemetry::const_label_list labels)
+		{
+		// Ignored
+		}
+
+	void operator()(const broker::telemetry::metric_family_hdl* family,
+	                const broker::telemetry::dbl_gauge_hdl* gauge,
+	                broker::telemetry::const_label_list labels)
+		{
+		// Ignored
+		}
+
+	void operator()(const broker::telemetry::metric_family_hdl* family,
+	                const broker::telemetry::int_gauge_hdl* gauge,
+	                broker::telemetry::const_label_list labels)
+		{
+		// Ignored
+		}
+
+	void operator()(const broker::telemetry::metric_family_hdl* family,
+	                const broker::telemetry::dbl_histogram_hdl* histogram,
+	                broker::telemetry::const_label_list labels)
+		{
+		if ( ! matches(family) )
+			return;
+
+		size_t num_buckets = broker::telemetry::num_buckets(histogram);
+
+		Manager::CollectedHistogramMetric::DblHistogramData histogram_data;
+		histogram_data.buckets.reserve(num_buckets);
+
+		for ( size_t i = 0; i < num_buckets; i++ )
+			{
+			double c = broker::telemetry::count_at(histogram, i);
+			double ub = broker::telemetry::upper_bound_at(histogram, i);
+			histogram_data.buckets.emplace_back(c, ub);
+			}
+
+		histogram_data.sum = broker::telemetry::sum(histogram);
+
+		metrics.emplace_back(family, extract_label_values(labels), std::move(histogram_data));
+		}
+
+	void operator()(const broker::telemetry::metric_family_hdl* family,
+	                const broker::telemetry::int_histogram_hdl* histogram,
+	                broker::telemetry::const_label_list labels)
+		{
+		if ( ! matches(family) )
+			return;
+
+		size_t num_buckets = broker::telemetry::num_buckets(histogram);
+
+		Manager::CollectedHistogramMetric::IntHistogramData histogram_data;
+		histogram_data.buckets.reserve(num_buckets);
+
+		for ( size_t i = 0; i < num_buckets; i++ )
+			{
+			int64_t c = broker::telemetry::count_at(histogram, i);
+			int64_t ub = broker::telemetry::upper_bound_at(histogram, i);
+			histogram_data.buckets.emplace_back(c, ub);
+			}
+
+		histogram_data.sum = broker::telemetry::sum(histogram);
+
+		metrics.emplace_back(family, extract_label_values(labels), std::move(histogram_data));
+		}
+
+	std::vector<Manager::CollectedHistogramMetric>& GetResult() { return metrics; }
+
+private:
+	MetricFamilyMatcher matches;
+	std::vector<Manager::CollectedHistogramMetric> metrics;
+	};
+
+std::vector<Manager::CollectedHistogramMetric>
+Manager::CollectHistogramMetrics(std::string_view prefix, std::string_view name)
+	{
+	auto collector = HistogramMetricsCollector(prefix, name);
+
+	pimpl->collect(collector);
+
+	return std::move(collector.GetResult());
+	}
+
 	} // namespace zeek::telemetry
 
 // -- unit tests ---------------------------------------------------------------
@@ -58,60 +513,6 @@ template <class T> auto toVector(zeek::Span<T> xs)
 
 	} // namespace
 
-SCENARIO("telemetry managers provide access to counter singletons")
-	{
-	GIVEN("a telemetry manager")
-		{
-		Manager mgr;
-		WHEN("retrieving an IntCounter singleton")
-			{
-			auto first = mgr.CounterSingleton("zeek", "int-count", "test");
-			THEN("its initial value is zero") { CHECK_EQ(first.Value(), 0); }
-			AND_THEN("calling Inc() or operator++ changes the value")
-				{
-				first.Inc();
-				CHECK_EQ(first.Value(), 1);
-				first.Inc(2);
-				CHECK_EQ(first.Value(), 3);
-				CHECK_EQ(++first, 4);
-				CHECK_EQ(first.Value(), 4);
-				}
-			AND_THEN("calling counterSingleton again for the same name returns the same handle")
-				{
-				auto second = mgr.CounterSingleton("zeek", "int-count", "test");
-				CHECK_EQ(first, second);
-				}
-			AND_THEN("calling counterSingleton for a different name returns another handle")
-				{
-				auto third = mgr.CounterSingleton("zeek", "int-count-2", "test");
-				CHECK_NE(first, third);
-				}
-			}
-		WHEN("retrieving a DblCounter singleton")
-			{
-			auto first = mgr.CounterSingleton<double>("zeek", "dbl-count", "test");
-			THEN("its initial value is zero") { CHECK_EQ(first.Value(), 0.0); }
-			AND_THEN("calling Inc() changes the value")
-				{
-				first.Inc();
-				CHECK_EQ(first.Value(), 1.0);
-				first.Inc(3.0);
-				CHECK_EQ(first.Value(), 4.0);
-				}
-			AND_THEN("calling counterSingleton again for the same name returns the same handle")
-				{
-				auto second = mgr.CounterSingleton<double>("zeek", "dbl-count", "test");
-				CHECK_EQ(first, second);
-				}
-			AND_THEN("calling counterSingleton for a different name returns another handle")
-				{
-				auto third = mgr.CounterSingleton<double>("zeek", "dbl-count-2", "test");
-				CHECK_NE(first, third);
-				}
-			}
-		}
-	}
-
 SCENARIO("telemetry managers provide access to counter families")
 	{
 	GIVEN("a telemetry manager")
@@ -171,70 +572,6 @@ SCENARIO("telemetry managers provide access to counter families")
 		}
 	}
 
-SCENARIO("telemetry managers provide access to gauge singletons")
-	{
-	GIVEN("a telemetry manager")
-		{
-		Manager mgr;
-		WHEN("retrieving an IntGauge singleton")
-			{
-			auto first = mgr.GaugeSingleton("zeek", "int-gauge", "test");
-			THEN("its initial value is zero") { CHECK_EQ(first.Value(), 0); }
-			AND_THEN("calling Inc(), Dec(), operator++ or operator-- changes the value")
-				{
-				first.Inc();
-				CHECK_EQ(first.Value(), 1);
-				first.Inc(2);
-				CHECK_EQ(first.Value(), 3);
-				first.Dec();
-				CHECK_EQ(first.Value(), 2);
-				CHECK_EQ(++first, 3);
-				CHECK_EQ(first.Value(), 3);
-				CHECK_EQ(--first, 2);
-				CHECK_EQ(first.Value(), 2);
-				first.Dec(2);
-				CHECK_EQ(first.Value(), 0);
-				}
-			AND_THEN("calling gaugeSingleton again for the same name returns the same handle")
-				{
-				auto second = mgr.GaugeSingleton("zeek", "int-gauge", "test");
-				CHECK_EQ(first, second);
-				}
-			AND_THEN("calling gaugeSingleton for a different name returns another handle")
-				{
-				auto third = mgr.GaugeSingleton("zeek", "int-gauge-2", "test");
-				CHECK_NE(first, third);
-				}
-			}
-		WHEN("retrieving a DblGauge singleton")
-			{
-			auto first = mgr.GaugeSingleton<double>("zeek", "dbl-gauge", "test");
-			THEN("its initial value is zero") { CHECK_EQ(first.Value(), 0.0); }
-			AND_THEN("calling Inc() or Dec() changes the value")
-				{
-				first.Inc();
-				CHECK_EQ(first.Value(), 1.0);
-				first.Inc(3.0);
-				CHECK_EQ(first.Value(), 4.0);
-				first.Dec(2.0);
-				CHECK_EQ(first.Value(), 2.0);
-				first.Dec();
-				CHECK_EQ(first.Value(), 1.0);
-				}
-			AND_THEN("calling gaugeSingleton again for the same name returns the same handle")
-				{
-				auto second = mgr.GaugeSingleton<double>("zeek", "dbl-gauge", "test");
-				CHECK_EQ(first, second);
-				}
-			AND_THEN("calling gaugeSingleton for a different name returns another handle")
-				{
-				auto third = mgr.GaugeSingleton<double>("zeek", "dbl-gauge-2", "test");
-				CHECK_NE(first, third);
-				}
-			}
-		}
-	}
-
 SCENARIO("telemetry managers provide access to gauge families")
 	{
 	GIVEN("a telemetry manager")
@@ -294,92 +631,6 @@ SCENARIO("telemetry managers provide access to gauge families")
 		}
 	}
 
-SCENARIO("telemetry managers provide access to histogram singletons")
-	{
-	GIVEN("a telemetry manager")
-		{
-		Manager mgr;
-		WHEN("retrieving an IntHistogram singleton")
-			{
-			const auto max_int = std::numeric_limits<int64_t>::max();
-			int64_t buckets[] = {10, 20};
-			auto first = mgr.HistogramSingleton("zeek", "int-hist", buckets, "test");
-			THEN("it initially has no observations")
-				{
-				REQUIRE_EQ(first.NumBuckets(), 3u);
-				CHECK_EQ(first.Sum(), 0);
-				CHECK_EQ(first.CountAt(0), 0);
-				CHECK_EQ(first.CountAt(1), 0);
-				CHECK_EQ(first.CountAt(2), 0);
-				CHECK_EQ(first.UpperBoundAt(0), 10);
-				CHECK_EQ(first.UpperBoundAt(1), 20);
-				CHECK_EQ(first.UpperBoundAt(2), max_int);
-				}
-			AND_THEN("calling Observe() increments bucket counters")
-				{
-				first.Observe(1);
-				first.Observe(9);
-				first.Observe(10);
-				first.Observe(11);
-				first.Observe(19);
-				first.Observe(20);
-				first.Observe(21);
-				CHECK_EQ(first.Sum(), 91);
-				CHECK_EQ(first.CountAt(0), 3);
-				CHECK_EQ(first.CountAt(1), 3);
-				CHECK_EQ(first.CountAt(2), 1);
-				}
-			AND_THEN("calling HistogramSingleton again for the same name returns the same handle")
-				{
-				auto second = mgr.HistogramSingleton("zeek", "int-hist", buckets, "test");
-				CHECK_EQ(first, second);
-				}
-			AND_THEN("calling HistogramSingleton for a different name returns another handle")
-				{
-				auto third = mgr.HistogramSingleton("zeek", "int-hist-2", buckets, "test");
-				CHECK_NE(first, third);
-				}
-			}
-		WHEN("retrieving a DblHistogram singleton")
-			{
-			double buckets[] = {10.0, 20.0};
-			auto first = mgr.HistogramSingleton<double>("zeek", "dbl-count", buckets, "test");
-			THEN("it initially has no observations")
-				{
-				REQUIRE_EQ(first.NumBuckets(), 3u);
-				CHECK_EQ(first.Sum(), 0.0);
-				CHECK_EQ(first.CountAt(0), 0);
-				CHECK_EQ(first.CountAt(1), 0);
-				CHECK_EQ(first.CountAt(2), 0);
-				CHECK_EQ(first.UpperBoundAt(0), 10.0);
-				CHECK_EQ(first.UpperBoundAt(1), 20.0);
-				}
-			AND_THEN("calling Observe() increments bucket counters")
-				{
-				first.Observe(2.0);
-				first.Observe(4.0);
-				first.Observe(8.0);
-				first.Observe(16.0);
-				first.Observe(32.0);
-				CHECK_EQ(first.Sum(), 62.0);
-				CHECK_EQ(first.CountAt(0), 3);
-				CHECK_EQ(first.CountAt(1), 1);
-				CHECK_EQ(first.CountAt(2), 1);
-				}
-			AND_THEN("calling histogramSingleton again for the same name returns the same handle")
-				{
-				auto second = mgr.HistogramSingleton<double>("zeek", "dbl-count", buckets, "test");
-				CHECK_EQ(first, second);
-				}
-			AND_THEN("calling histogramSingleton for a different name returns another handle")
-				{
-				auto third = mgr.HistogramSingleton<double>("zeek", "dbl-count-2", buckets, "test");
-				CHECK_NE(first, third);
-				}
-			}
-		}
-	}
-
 SCENARIO("telemetry managers provide access to histogram families")
 	{
 	GIVEN("a telemetry manager")

src/telemetry/Manager.h

@@ -5,6 +5,8 @@
 #include <cstdint>
 #include <initializer_list>
 #include <string_view>
+#include <unordered_map>
+#include <variant>
 #include <vector>
 
 #include "zeek/IntrusivePtr.h"
@@ -20,6 +22,12 @@ namespace broker
 class endpoint;
 	}
 
+namespace zeek
+	{
+class RecordVal;
+using RecordValPtr = IntrusivePtr<RecordVal>;
+	}
+
 namespace zeek::Broker
 	{
 class Manager;
@@ -50,6 +58,128 @@ public:
 	 */
 	virtual void InitPostScript();
 
+	/**
+	 * Supported metric types.
+	 */
+	enum class MetricType
+		{
+		Counter,
+		Gauge,
+		Histogram
+		};
+
+	/**
+	 * Captures information about counter and gauge metrics.
+	 */
+	struct CollectedValueMetric
+		{
+		/**
+		 * Constructor.
+		 * @param metric_type The type of this metric.
+		 * @param family Broker layer family handle for this metric.
+		 * @param label_values The string values for each of the metric's labels.
+		 * @param value The metric's current value.
+		 */
+		CollectedValueMetric(MetricType metric_type,
+		                     const broker::telemetry::metric_family_hdl* family,
+		                     std::vector<std::string_view> label_values,
+		                     std::variant<double, int64_t> value)
+			: metric_type(metric_type), family(family), label_values(std::move(label_values)),
+			  value(value)
+			{
+			}
+
+		/**
+		 * @return A script layer Telemetry::Metric record for this metric.
+		 */
+		zeek::RecordValPtr AsMetricRecord() const;
+
+		enum MetricType metric_type;
+		const broker::telemetry::metric_family_hdl* family;
+		std::vector<std::string_view> label_values;
+		std::variant<double, int64_t> value;
+		};
+
+	/**
+	 * Captures information about histogram metrics.
+	 */
+	struct CollectedHistogramMetric
+		{
+		/**
+		 * Helper struct representing a single bucket of a histogram.
+		 * @tparam T The data type used by the histogram (double or int64_t).
+		 */
+		template <class T> struct Bucket
+			{
+			Bucket(T count, T upper_bound) : count(count), upper_bound(upper_bound) { }
+
+			T count;
+			T upper_bound;
+			};
+
+		/**
+		 * Helper struct representing a histogram as sum and buckets.
+		 * @tparam T The data type used by the histogram (double or int64_t).
+		 */
+		template <class T> struct HistogramData
+			{
+			T sum;
+			std::vector<Bucket<T>> buckets;
+			};
+
+		using DblHistogramData = HistogramData<double>;
+		using IntHistogramData = HistogramData<int64_t>;
+
+		/**
+		 * Constructor.
+		 * @param family Broker layer family handle for this metric.
+		 * @param label_values The string values for each of the metric's labels.
+		 * @param histogram The histogram's data (sum and individual buckets).
+		 */
+		CollectedHistogramMetric(const broker::telemetry::metric_family_hdl* family,
+		                         std::vector<std::string_view> label_values,
+		                         std::variant<DblHistogramData, IntHistogramData> histogram)
+
+			: family(family), label_values(std::move(label_values)), histogram(std::move(histogram))
+			{
+			}
+
+		const broker::telemetry::metric_family_hdl* family;
+		std::vector<std::string_view> label_values;
+		std::variant<DblHistogramData, IntHistogramData> histogram;
+
+		/**
+		 * @return A script layer Telemetry::HistogramMetric record for this histogram.
+		 */
+		zeek::RecordValPtr AsHistogramMetricRecord() const;
+		};
+
+	/**
+	 * @return A script layer Telemetry::MetricOpts record for the given metric family.
+	 * @param metric_typ The type of metric.
+	 * @param family Broker layer family handle for the family.
+	 * @tparam T The underlying data type (double or int64_t)
+	 */
+	template <typename T>
+	zeek::RecordValPtr GetMetricOptsRecord(MetricType metric_type,
+	                                       const broker::telemetry::metric_family_hdl* family);
+
+	/**
+	 * @return All counter and gauge metrics and their values matching prefix and name.
+	 * @param prefix The prefix pattern to use for filtering. Supports globbing.
+	 * @param name The name pattern to use for filtering. Supports globbing.
+	 */
+	std::vector<CollectedValueMetric> CollectMetrics(std::string_view prefix,
+	                                                 std::string_view name);
+
+	/**
+	 * @return All histogram metrics and their data matching prefix and name.
+	 * @param prefix The prefix pattern to use for filtering. Supports globbing.
+	 * @param name The name pattern to use for filtering. Supports globbing.
+	 */
+	std::vector<CollectedHistogramMetric> CollectHistogramMetrics(std::string_view prefix,
+	                                                              std::string_view name);
+
 	/**
 	 * @return A counter metric family. Creates the family lazily if necessary.
 	 * @param prefix The prefix (namespace) this family belongs to.
@@ -67,14 +197,14 @@ public:
 		{
 		if constexpr ( std::is_same<ValueType, int64_t>::value )
 			{
-			auto fam = int_counter_fam(ptr(), prefix, name, labels, helptext, unit, is_sum);
+			auto fam = int_counter_fam(Ptr(), prefix, name, labels, helptext, unit, is_sum);
 			return IntCounterFamily{fam};
 			}
 		else
 			{
 			static_assert(std::is_same<ValueType, double>::value,
 			              "metrics only support int64_t and double values");
-			auto fam = dbl_counter_fam(ptr(), prefix, name, labels, helptext, unit, is_sum);
+			auto fam = dbl_counter_fam(Ptr(), prefix, name, labels, helptext, unit, is_sum);
 			return DblCounterFamily{fam};
 			}
 		}
@@ -125,27 +255,6 @@ public:
 		return CounterInstance(prefix, name, lbl_span, helptext, unit, is_sum);
 		}
 
-	/**
-	 * Accesses a counter singleton, i.e., a counter that belongs to a family
-	 * without label dimensions (which thus only has a single member). Creates
-	 * the hosting metric family as well as the counter lazily if necessary.
-	 * @param prefix The prefix (namespace) this family belongs to.
-	 * @param name The human-readable name of the metric, e.g., `requests`.
-	 * @param helptext Short explanation of the metric.
-	 * @param unit Unit of measurement.
-	 * @param is_sum Indicates whether this metric accumulates something, where
-	 *               only the total value is of interest.
-	 */
-	template <class ValueType = int64_t>
-	Counter<ValueType> CounterSingleton(std::string_view prefix, std::string_view name,
-	                                    std::string_view helptext, std::string_view unit = "1",
-	                                    bool is_sum = false)
-		{
-		auto labels = Span<const std::string_view>{};
-		auto fam = CounterFamily<ValueType>(prefix, name, labels, helptext, unit, is_sum);
-		return fam.GetOrAdd({});
-		}
-
 	/**
 	 * @return A gauge metric family. Creates the family lazily if necessary.
 	 * @param prefix The prefix (namespace) this family belongs to.
@@ -163,14 +272,14 @@ public:
 		{
 		if constexpr ( std::is_same<ValueType, int64_t>::value )
 			{
-			auto fam = int_gauge_fam(ptr(), prefix, name, labels, helptext, unit, is_sum);
+			auto fam = int_gauge_fam(Ptr(), prefix, name, labels, helptext, unit, is_sum);
 			return IntGaugeFamily{fam};
 			}
 		else
 			{
 			static_assert(std::is_same<ValueType, double>::value,
 			              "metrics only support int64_t and double values");
-			auto fam = dbl_gauge_fam(ptr(), prefix, name, labels, helptext, unit, is_sum);
+			auto fam = dbl_gauge_fam(Ptr(), prefix, name, labels, helptext, unit, is_sum);
 			return DblGaugeFamily{fam};
 			}
 		}
@@ -221,27 +330,6 @@ public:
 		return GaugeInstance(prefix, name, lbl_span, helptext, unit, is_sum);
 		}
 
-	/**
-	 * Accesses a gauge singleton, i.e., a gauge that belongs to a family
-	 * without label dimensions (which thus only has a single member). Creates
-	 * the hosting metric family as well as the gauge lazily if necessary.
-	 * @param prefix The prefix (namespace) this family belongs to.
-	 * @param name The human-readable name of the metric, e.g., `requests`.
-	 * @param helptext Short explanation of the metric.
-	 * @param unit Unit of measurement.
-	 * @param is_sum Indicates whether this metric accumulates something, where
-	 *               only the total value is of interest.
-	 */
-	template <class ValueType = int64_t>
-	Gauge<ValueType> GaugeSingleton(std::string_view prefix, std::string_view name,
-	                                std::string_view helptext, std::string_view unit = "1",
-	                                bool is_sum = false)
-		{
-		auto labels = Span<const std::string_view>{};
-		auto fam = GaugeFamily<ValueType>(prefix, name, labels, helptext, unit, is_sum);
-		return fam.GetOrAdd({});
-		}
-
 	// Forces the compiler to use the type `Span<const T>` instead of trying to
 	// match paremeters to a `span`.
 	template <class T> struct ConstSpanOracle
@@ -281,7 +369,7 @@ public:
 		{
 		if constexpr ( std::is_same<ValueType, int64_t>::value )
 			{
-			auto fam = int_histogram_fam(ptr(), prefix, name, labels, default_upper_bounds,
+			auto fam = int_histogram_fam(Ptr(), prefix, name, labels, default_upper_bounds,
 			                             helptext, unit, is_sum);
 			return IntHistogramFamily{fam};
 			}
@@ -289,7 +377,7 @@ public:
 			{
 			static_assert(std::is_same<ValueType, double>::value,
 			              "metrics only support int64_t and double values");
-			auto fam = dbl_histogram_fam(ptr(), prefix, name, labels, default_upper_bounds,
+			auto fam = dbl_histogram_fam(Ptr(), prefix, name, labels, default_upper_bounds,
 			                             helptext, unit, is_sum);
 			return DblHistogramFamily{fam};
 			}
@@ -355,39 +443,6 @@ public:
 		return HistogramInstance(prefix, name, lbls, default_upper_bounds, helptext, unit, is_sum);
 		}
 
-	/**
-	 * Returns a histogram metric singleton, i.e., the single instance of a
-	 * family without label dimensions. Creates all objects lazily if necessary,
-	 * but fails if the full name already belongs to a different family.
-	 * @param prefix The prefix (namespace) this family belongs to. Usually the
-	 *               application or protocol name, e.g., `http`. The prefix `caf`
-	 *               as well as prefixes starting with an underscore are
-	 *               reserved.
-	 * @param name The human-readable name of the metric, e.g., `requests`.
-	 * @param default_upper_bounds Upper bounds for the metric buckets.
-	 * @param helptext Short explanation of the metric.
-	 * @param unit Unit of measurement. Please use base units such as `bytes` or
-	 *             `seconds` (prefer lowercase). The pseudo-unit `1` identifies
-	 *             dimensionless counts.
-	 * @param is_sum Setting this to `true` indicates that this metric adds
-	 *               something up to a total, where only the total value is of
-	 *               interest. For example, the total number of HTTP requests.
-	 * @note The first call wins when calling this function multiple times with
-	 *       different bucket settings. Users may also override
-	 *       @p default_upper_bounds via run-time configuration.
-	 */
-	template <class ValueType = int64_t>
-	Histogram<ValueType> HistogramSingleton(std::string_view prefix, std::string_view name,
-	                                        ConstSpan<ValueType> default_upper_bounds,
-	                                        std::string_view helptext, std::string_view unit = "1",
-	                                        bool is_sum = false)
-		{
-		auto lbls = Span<const std::string_view>{};
-		auto fam = HistogramFamily<ValueType>(prefix, name, lbls, default_upper_bounds, helptext,
-		                                      unit, is_sum);
-		return fam.GetOrAdd({});
-		}
-
 protected:
 	template <class F> static void WithLabelNames(Span<const LabelView> xs, F continuation)
 		{
@@ -396,6 +451,7 @@ protected:
 			std::string_view buf[10];
 			for ( size_t index = 0; index < xs.size(); ++index )
 				buf[index] = xs[index].first;
+
 			return continuation(Span{buf, xs.size()});
 			}
 		else
@@ -403,17 +459,23 @@ protected:
 			std::vector<std::string_view> buf;
 			for ( auto x : xs )
 				buf.emplace_back(x.first, x.second);
+
 			return continuation(Span{buf});
 			}
 		}
 
-	broker::telemetry::metric_registry_impl* ptr() { return pimpl.get(); }
+	broker::telemetry::metric_registry_impl* Ptr() { return pimpl.get(); }
 
 	// Connects all the dots after the Broker Manager constructed the endpoint
 	// for this Zeek instance. Called from Broker::Manager::InitPostScript().
 	void InitPostBrokerSetup(broker::endpoint&);
 
 	IntrusivePtr<broker::telemetry::metric_registry_impl> pimpl;
+
+private:
+	// Caching of metric_family_hdl instances to their Zeek record representation.
+	std::unordered_map<const broker::telemetry::metric_family_hdl*, zeek::RecordValPtr>
+		metric_opts_cache;
 	};
 
 	} // namespace zeek::telemetry

src/telemetry/telemetry.bif

@@ -1,8 +1,16 @@
-
 ##! Functions for accessing counter metrics from script land.
 
 module Telemetry;
 
+enum MetricType %{
+	DOUBLE_COUNTER,
+	INT_COUNTER,
+	DOUBLE_GAUGE,
+	INT_GAUGE,
+	DOUBLE_HISTOGRAM,
+	INT_HISTOGRAM,
+%}
+
 %%{
 #include "zeek/telemetry/Counter.h"
 #include "zeek/telemetry/Gauge.h"
@@ -139,17 +147,6 @@ function Telemetry::__int_counter_metric_get_or_add%(family: opaque of int_count
 		}
 	%}
 
-function Telemetry::__int_counter_singleton%(prefix: string,
-                                             name: string,
-                                             helptext: string &default = "Zeek Script Metric",
-                                             unit: string &default = "1",
-                                             is_sum: bool &default = F%): opaque of int_counter_metric
-	%{
-	auto hdl = telemetry_mgr->CounterSingleton(sv(prefix), sv(name),
-	                                           sv(helptext), sv(unit), is_sum);
-	return zeek::make_intrusive<IntCounterMetricVal>(hdl);
-	%}
-
 function Telemetry::__int_counter_inc%(val: opaque of int_counter_metric,
                                        amount: int &default = 1%): bool
 	%{
@@ -210,17 +207,6 @@ function Telemetry::__dbl_counter_metric_get_or_add%(family: opaque of dbl_count
 		}
 	%}
 
-function Telemetry::__dbl_counter_singleton%(prefix: string,
-                                             name: string,
-                                             helptext: string &default = "Zeek Script Metric",
-                                             unit: string &default = "1",
-                                             is_sum: bool &default = F%): opaque of dbl_counter_metric
-	%{
-	auto hdl = telemetry_mgr->CounterSingleton<double>(sv(prefix), sv(name),
-	                                                   sv(helptext), sv(unit), is_sum);
-	return zeek::make_intrusive<DblCounterMetricVal>(hdl);
-	%}
-
 function Telemetry::__dbl_counter_inc%(val: opaque of dbl_counter_metric,
                                        amount: double &default = 1.0%): bool
 	%{
@@ -281,17 +267,6 @@ function Telemetry::__int_gauge_metric_get_or_add%(family: opaque of int_gauge_m
 		}
 	%}
 
-function Telemetry::__int_gauge_singleton%(prefix: string,
-                                           name: string,
-                                           helptext: string &default = "Zeek Script Metric",
-                                           unit: string &default = "1",
-                                           is_sum: bool &default = F%): opaque of int_gauge_metric
-	%{
-	auto hdl = telemetry_mgr->GaugeSingleton(sv(prefix), sv(name),
-	                                         sv(helptext), sv(unit), is_sum);
-	return zeek::make_intrusive<IntGaugeMetricVal>(hdl);
-	%}
-
 function Telemetry::__int_gauge_inc%(val: opaque of int_gauge_metric,
                                      amount: int &default = 1%): bool
 	%{
@@ -358,17 +333,6 @@ function Telemetry::__dbl_gauge_metric_get_or_add%(family: opaque of dbl_gauge_m
 		}
 	%}
 
-function Telemetry::__dbl_gauge_singleton%(prefix: string,
-                                           name: string,
-                                           helptext: string &default = "Zeek Script Metric",
-                                           unit: string &default = "1",
-                                           is_sum: bool &default = F%): opaque of dbl_gauge_metric
-	%{
-	auto hdl = telemetry_mgr->GaugeSingleton<double>(sv(prefix), sv(name),
-	                                                 sv(helptext), sv(unit), is_sum);
-	return zeek::make_intrusive<DblGaugeMetricVal>(hdl);
-	%}
-
 function Telemetry::__dbl_gauge_inc%(val: opaque of dbl_gauge_metric,
                                      amount: double &default = 1.0%): bool
 	%{
@@ -438,20 +402,6 @@ function Telemetry::__int_histogram_metric_get_or_add%(family: opaque of int_his
 		}
 	%}
 
-function Telemetry::__int_histogram_singleton%(prefix: string,
-                                               name: string,
-                                               bounds: int_vec,
-                                               helptext: string &default = "Zeek Script Metric",
-                                               unit: string &default = "1",
-                                               is_sum: bool &default = F%): opaque of int_histogram_metric
-	%{
-	auto std_bounds = to_std_vec<int64_t>(bounds);
-	auto hdl = telemetry_mgr->HistogramSingleton(sv(prefix), sv(name),
-	                                             std_bounds, sv(helptext),
-	                                             sv(unit), is_sum);
-	return zeek::make_intrusive<IntHistogramMetricVal>(hdl);
-	%}
-
 function Telemetry::__int_histogram_observe%(val: opaque of int_histogram_metric,
                                              measurement: int%): bool
 	%{
@@ -516,20 +466,6 @@ function Telemetry::__dbl_histogram_metric_get_or_add%(family: opaque of dbl_his
 		}
 	%}
 
-function Telemetry::__dbl_histogram_singleton%(prefix: string,
-                                               name: string,
-                                               bounds: double_vec,
-                                               helptext: string &default = "Zeek Script Metric",
-                                               unit: string &default = "1",
-                                               is_sum: bool &default = F%): opaque of dbl_histogram_metric
-	%{
-	auto std_bounds = to_std_vec<double>(bounds);
-	auto hdl = telemetry_mgr->HistogramSingleton<double>(sv(prefix), sv(name),
-	                                                     std_bounds, sv(helptext),
-	                                                     sv(unit), is_sum);
-	return zeek::make_intrusive<DblHistogramMetricVal>(hdl);
-	%}
-
 function Telemetry::__dbl_histogram_observe%(val: opaque of dbl_histogram_metric,
                                              measurement: double%): bool
 	%{
@@ -548,3 +484,29 @@ function Telemetry::__dbl_histogram_sum%(val: opaque of dbl_histogram_metric%):
 		return zeek::make_intrusive<DoubleVal>(0.0);
 		}
 	%}
+
+function Telemetry::__collect_metrics%(prefix: string, name: string%): any_vec
+	%{
+	auto metrics = telemetry_mgr->CollectMetrics(sv(prefix), sv(name));
+
+	static auto metrics_vector_type = zeek::id::find_type<VectorType>("any_vec");
+	auto vec = zeek::make_intrusive<VectorVal>(metrics_vector_type);
+
+	for ( const auto& m : metrics )
+		vec->Append(m.AsMetricRecord());
+
+	return vec;
+	%}
+
+function Telemetry::__collect_histogram_metrics%(prefix: string, name: string%): any_vec
+	%{
+	auto metrics = telemetry_mgr->CollectHistogramMetrics(sv(prefix), sv(name));
+
+	static auto metrics_vector_type = zeek::id::find_type<VectorType>("any_vec");
+	auto vec = zeek::make_intrusive<VectorVal>(metrics_vector_type);
+
+	for ( const auto& m : metrics )
+		vec->Append(m.AsHistogramMetricRecord());
+
+	return vec;
+	%}

src/zeek.bif

@@ -28,6 +28,7 @@
 #include "zeek/IntrusivePtr.h"
 #include "zeek/input.h"
 #include "zeek/Hash.h"
+#include "zeek/CompHash.h"
 #include "zeek/packet_analysis/Manager.h"
 
 using namespace std;
@@ -1178,6 +1179,58 @@ function clear_table%(v: any%): any
 	return nullptr;
 	%}
 
+## Gets all values from a table.
+##
+## t: The :zeek:type:`table`
+##
+## Returns: A ``vector of T`` of all the values in t.
+##
+## .. zeek:see:: table_keys
+function table_values%(t: any%): any_vec
+	%{
+	if ( ! t->GetType()->IsTable() )
+		{
+		zeek::emit_builtin_error("table_values() requires a table argument");
+		return nullptr;
+		}
+
+	auto vt = zeek::make_intrusive<zeek::VectorType>(t->GetType()->AsTableType()->Yield());
+	auto vv = zeek::make_intrusive<zeek::VectorVal>(std::move(vt));
+	for ( const auto& iter : *t->AsTable() )
+		{
+		vv->Append(iter.value->GetVal());
+		}
+
+	return vv;
+	%}
+
+## Gets all keys from a table.
+##
+## t: The :zeek:type:`table`
+##
+## Returns: A ``set of T`` of all the keys in t.
+##
+## .. zeek:see:: table_values
+function table_keys%(t: any%): any
+	%{
+	if ( ! t->GetType()->IsTable() )
+		{
+		zeek::emit_builtin_error("table_keys() requires a table argument");
+		return nullptr;
+		}
+
+	auto tl = t->GetType()->AsTableType()->GetIndices();
+	auto st = zeek::make_intrusive<zeek::SetType>(std::move(tl), nullptr);
+	auto sv = zeek::make_intrusive<zeek::TableVal>(std::move(st));
+	auto th = t->AsTableVal()->GetTableHash();
+	for ( const auto& iter : *t->AsTable() )
+		{
+		sv->Assign(th->RecoverVals(*iter.GetHashKey()), nullptr);
+		}
+
+	return sv;
+	%}
+
 ## Gets all subnets that contain a given subnet from a set/table[subnet].
 ##
 ## search: the subnet to search for.
@@ -4590,6 +4643,15 @@ function disable_analyzer%(cid: conn_id, aid: count, err_if_no_conn: bool &defau
 		return zeek::val_mgr->False();
 		}
 
+	static auto disabling_analyzer_hook = id::find_func("disabling_analyzer");
+	if ( disabling_analyzer_hook )
+		{
+		auto hook_rval = disabling_analyzer_hook->Invoke(c->GetVal(), a->GetAnalyzerTag().AsVal(),
+		                                                 zeek::val_mgr->Count(aid));
+		if ( hook_rval && ! hook_rval->AsBool() )
+			return zeek::val_mgr->False();
+		}
+
 	if ( prevent )
 		a->Parent()->PreventChildren(a->GetAnalyzerTag());
 
@@ -5304,16 +5366,7 @@ function match_signatures%(c: connection, pattern_type: int, s: string,
 ## only useful for debugging and causes reduced performance.
 function generate_all_events%(%) : bool
 	%{
-	auto event_names = event_registry->AllHandlers();
+	event_registry->ActivateAllHandlers();
-	for ( const auto& name: event_names )
-		{
-		auto event = event_registry->Lookup(name);
-		if ( event == nullptr )
-			continue;
-
-		event->SetGenerateAlways();
-		}
-
 	return zeek::val_mgr->True();
 	%}
 

src/zeekygen/Manager.cc

@@ -157,6 +157,10 @@ void Manager::ScriptDependency(const string& path, const string& dep)
 	if ( disabled )
 		return;
 
+	if ( path == "<command line>" )
+		// This is a @load directive on the command line.
+		return;
+
 	if ( dep.empty() )
 		{
 		DbgAndWarn(util::fmt("Empty Zeekygen script doc dependency: %s", path.c_str()));
@@ -188,6 +192,10 @@ void Manager::ModuleUsage(const string& path, const string& module)
 	if ( disabled )
 		return;
 
+	if ( path == "<command line>" )
+		// This is a moudle defined on the command line.
+		return;
+
 	string name = normalize_script_path(path);
 	ScriptInfo* script_info = scripts.GetInfo(name);
 

testing/btest/Baseline.cpp/bifs.backtrace/out

@@ -1,8 +1,10 @@
 ### BTest baseline data generated by btest-diff. Do not edit. Use "btest -U/-u" to update. Requires BTest >= 0.63.
 
 --- Backtrace ---
+#0: zeek_init()
 
 --- Backtrace ---
-
+| #0: zeek_init() |
 
 --- Backtrace ---
+#0: zeek_init()

testing/btest/Baseline.cpp/bifs.disable_analyzer/out

@@ -0,0 +1,7 @@
+### BTest baseline data generated by btest-diff. Do not edit. Use "btest -U/-u" to update. Requires BTest >= 0.63.
+proto confirm, AllAnalyzers::ANALYZER_ANALYZER_HTTP
+http_request, GET, /style/enhanced.css
+T
+total http messages, {
+[[orig_h=192.168.1.104, orig_p=1673/tcp, resp_h=63.245.209.11, resp_p=80/tcp]] = 1
+}

testing/btest/Baseline.cpp/bifs.type_aliases/out

@@ -5,16 +5,16 @@ type aliases for 'MyRec val': MyRec MyRecAlias
 type aliases for 'MyRecAlias val': MyRec MyRecAlias
 type aliases for 'MyRec type': MyRec MyRecAlias
 type aliases for 'MyRecalias type': MyRec MyRecAlias
-type aliases for 'MyString val': MyString AnotherString
+type aliases for 'MyString val': it's just a 'string'
 type aliases for 'MyString type': MyString AnotherString
 type aliases for 'MyOtherString type': MyOtherString
 type aliases for 'AnotherString type': MyString AnotherString
-type aliases for 'string literal value': MyString AnotherString
+type aliases for 'string literal value': it's just a 'string'
 type aliases for 'count literal value': it's just a 'count'
-type aliases for 'MyTable value': MyTable2 MyTable3 MyTable MyTable4
+type aliases for 'MyTable value': it's just a 'table[count] of string'
-type aliases for 'MyTable2 value': MyTable2 MyTable3 MyTable MyTable4
+type aliases for 'MyTable2 value': it's just a 'table[count] of string'
-type aliases for 'MyTable3 value': MyTable2 MyTable3 MyTable MyTable4
+type aliases for 'MyTable3 value': it's just a 'table[count] of string'
-type aliases for 'MyTable4 value': MyTable2 MyTable3 MyTable MyTable4
+type aliases for 'MyTable4 value': it's just a 'table[count] of string'
 type aliases for 'MyTable type': MyTable2 MyTable3 MyTable MyTable4
 type aliases for 'MyTable2 type': MyTable2 MyTable3 MyTable MyTable4
 type aliases for 'MyTable3 type': MyTable2 MyTable3 MyTable MyTable4

testing/btest/Baseline.cpp/broker.store.create-failure/zeek.err

@@ -0,0 +1,11 @@
+### BTest baseline data generated by btest-diff. Do not edit. Use "btest -U/-u" to update. Requires BTest >= 0.63.
+error in <...>/main.zeek, lines 498-499: Failed to attach master store backend_failure: (Option::set_change_handler(Broker::metrics_export_prefixes, to_any_coerceBroker::update_metrics_export_prefixes, (coerce 0 to int)))
+error in <...>/main.zeek, lines 498-499: Could not create Broker master store '../fail' (Option::set_change_handler(Broker::metrics_export_prefixes, to_any_coerceBroker::update_metrics_export_prefixes, (coerce 0 to int)))
+error in <no location>: invalid Broker store handle (broker::store::{})
+error in <no location>: invalid Broker store handle (broker::store::{})
+error in <no location>: invalid Broker store handle (broker::store::{})
+error in <no location>: invalid Broker store handle (broker::store::{})
+error in <no location>: invalid Broker store handle (broker::store::{})
+error in <no location>: invalid Broker store handle (broker::store::{})
+error in <no location>: invalid Broker store handle (broker::store::{})
+received termination signal

testing/btest/Baseline.cpp/broker.store.create-failure/zeek.out

@@ -0,0 +1,21 @@
+### BTest baseline data generated by btest-diff. Do not edit. Use "btest -U/-u" to update. Requires BTest >= 0.63.
+T
+F
+F
+F
+m1 keys result: [status=Broker::FAILURE, result=[data=<uninitialized>]]
+m2 keys result: [status=Broker::SUCCESS, result=[data=broker::data{{}}]]
+c2 keys result: [status=Broker::SUCCESS, result=[data=broker::data{{}}]]
+T
+F
+F
+F
+T
+T
+T
+T
+m1 keys result: [status=Broker::FAILURE, result=[data=<uninitialized>]]
+c1 keys result: [status=Broker::FAILURE, result=[data=<uninitialized>]]
+m2 keys result: [status=Broker::FAILURE, result=[data=<uninitialized>]]
+c2 keys result: [status=Broker::FAILURE, result=[data=<uninitialized>]]
+c1 timeout

testing/btest/Baseline.cpp/core.parse-only-usage-issues/out

@@ -0,0 +1 @@
+### BTest baseline data generated by btest-diff. Do not edit. Use "btest -U/-u" to update. Requires BTest >= 0.63.

testing/btest/Baseline.cpp/core.reporter/logger-test.log

@@ -1,7 +1,7 @@
 ### BTest baseline data generated by btest-diff. Do not edit. Use "btest -U/-u" to update. Requires BTest >= 0.63.
-reporter_info|init test-info||XXXXXXXXXX.XXXXXX
+reporter_info|init test-info|<...>/main.zeek, lines 498-499|XXXXXXXXXX.XXXXXX
-reporter_warning|init test-warning||XXXXXXXXXX.XXXXXX
+reporter_warning|init test-warning|<...>/main.zeek, lines 498-499|XXXXXXXXXX.XXXXXX
-reporter_error|init test-error||XXXXXXXXXX.XXXXXX
+reporter_error|init test-error|<...>/main.zeek, lines 498-499|XXXXXXXXXX.XXXXXX
 reporter_info|done test-info||XXXXXXXXXX.XXXXXX
 reporter_warning|done test-warning||XXXXXXXXXX.XXXXXX
 reporter_error|done test-error||XXXXXXXXXX.XXXXXX

testing/btest/Baseline.cpp/core.reporter/output

@@ -2,9 +2,9 @@
 pre test-info
 warning: pre test-warning
 error: pre test-error
-init test-info
+<...>/main.zeek, lines 498-499: init test-info
-warning: init test-warning
+warning in <...>/main.zeek, lines 498-499: init test-warning
-error: init test-error
+error in <...>/main.zeek, lines 498-499: init test-error
 done test-info
 warning: done test-warning
 error: done test-error

testing/btest/Baseline.cpp/core.when-interpreter-exceptions/zeek.output

@@ -0,0 +1,15 @@
+### BTest baseline data generated by btest-diff. Do not edit. Use "btest -U/-u" to update. Requires BTest >= 0.63.
+### NOTE: This file has been sorted with diff-sort.
+[f(F)]
+[f(T)]
+[zeek_init()]
+f() done, no exception, T
+g() done, no exception, T
+received termination signal
+runtime error in compiled code: field value missing
+runtime error in compiled code: field value missing
+runtime error in compiled code: field value missing
+runtime error in compiled code: field value missing
+timeout
+timeout g(), F
+timeout g(), T

testing/btest/Baseline.cpp/language.copy-all-types/out

@@ -9,5 +9,5 @@ orig=/^?(.*PATTERN.*)$?/ (pattern) clone=/^?(.*PATTERN.*)$?/ (pattern) same_obje
 orig=2,5,3,4,1 (set[count]) clone=2,5,3,4,1 (set[count]) equal=T same_object=F (ok)
 orig=[1, 2, 3, 4, 5] (vector of count) clone=[1, 2, 3, 4, 5] (vector of count) equal=T same_object=F (ok)
 orig=a=va;b=vb (table[string] of string) clone=a=va;b=vb (table[string] of string) equal=T same_object=F (ok)
-orig=ENUMME (enum MyEnum) clone=ENUMME (enum MyEnum) equal=T same_object=F (FAIL1)
+orig=ENUMME (MyEnum) clone=ENUMME (MyEnum) equal=T same_object=F (FAIL1)
-orig=[s1=s1, s2=s2, i1=[a=a], i2=[a=a], donotset=<uninitialized>, def=5] (record { s1:string; s2:string; i1:record { a:string; }; i2:record { a:string; } &optional; donotset:record { a:string; } &optional; def:count &default=5, &optional; }) clone=[s1=s1, s2=s2, i1=[a=a], i2=[a=a], donotset=<uninitialized>, def=5] (record { s1:string; s2:string; i1:record { a:string; }; i2:record { a:string; } &optional; donotset:record { a:string; } &optional; def:count &default=5, &optional; }) equal=T same_object=F (ok)
+orig=[s1=s1, s2=s2, i1=[a=a], i2=[a=a], donotset=<uninitialized>, def=5] (TestRecord) clone=[s1=s1, s2=s2, i1=[a=a], i2=[a=a], donotset=<uninitialized>, def=5] (TestRecord) equal=T same_object=F (ok)

testing/btest/Baseline.cpp/language.index-assignment-invalid/out

@@ -1,6 +1,6 @@
 ### BTest baseline data generated by btest-diff. Do not edit. Use "btest -U/-u" to update. Requires BTest >= 0.63.
-runtime error in compiled code: vector index assignment failed for invalid type 'myrec', value: [a=T, b=hi, c=<uninitialized>]
+runtime error in <...>/queue.zeek, line 152: vector index assignment failed for invalid type 'myrec', value: [a=T, b=hi, c=<uninitialized>], expression: Queue::ret[Queue::j], call stack: 
-runtime error in compiled code: vector index assignment failed for invalid type 'myrec', value: [a=T, b=hi, c=<uninitialized>]
+ #0 Queue::get_vector([initialized=T, vals={[2] = test,[3] = [a=T, b=hi, c=<uninitialized>],[5] = 3,[0] = hello,[6] = jkl;,[4] = asdf,[1] = goodbye}, settings=[max_len=<uninitialized>], top=7, bottom=0, size=0], [hello, goodbye, test]) at <...>/main.zeek:498 
-runtime error in compiled code: vector index assignment failed for invalid type 'myrec', value: [a=T, b=hi, c=<uninitialized>]
+ #1 zeek_init() 
-runtime error in compiled code: vector index assignment failed for invalid type 'myrec', value: [a=T, b=hi, c=<uninitialized>]
+
-runtime error in compiled code: vector index assignment failed for invalid type 'myrec', value: [a=T, b=hi, c=<uninitialized>]
+

testing/btest/Baseline.cpp/plugins.reporter-hook/output

@@ -1,11 +1,14 @@
 ### BTest baseline data generated by btest-diff. Do not edit. Use "btest -U/-u" to update. Requires BTest >= 0.63.
- | Hook  Some Info 
+Reporter::Hook - Exercise Reporter Hook (dynamic, version 1.0.0)
- | Hook error An Error 
+    Implements Reporter (priority 0)
- | Hook error An Error that does not show up in the log 
+
+ | Hook  Some Info <...>/main.zeek, lines 498-499
+ | Hook error An Error <...>/main.zeek, lines 498-499
+ | Hook error An Error that does not show up in the log <...>/main.zeek, lines 498-499
  | Hook runtime error in compiled code field value missing 
- | Hook warning A warning 
+ | Hook warning A warning <...>/main.zeek, lines 498-499
-Some Info
+<...>/main.zeek, lines 498-499: Some Info
-error: An Error
+error in <...>/main.zeek, lines 498-499: An Error
-error: An Error that does not show up in the log
+error in <...>/main.zeek, lines 498-499: An Error that does not show up in the log
 runtime error in compiled code: field value missing
-warning: A warning
+warning in <...>/main.zeek, lines 498-499: A warning

testing/btest/Baseline.cpp/plugins.reporter-hook/reporter.log

@@ -7,8 +7,8 @@
 #open XXXX-XX-XX-XX-XX-XX
 #fields	ts	level	message	location
 #types	time	enum	string	string
-XXXXXXXXXX.XXXXXX	Reporter::INFO	Some Info	(empty)
+XXXXXXXXXX.XXXXXX	Reporter::INFO	Some Info	<...>/main.zeek, lines 498-499
-XXXXXXXXXX.XXXXXX	Reporter::WARNING	A warning	(empty)
+XXXXXXXXXX.XXXXXX	Reporter::WARNING	A warning	<...>/main.zeek, lines 498-499
-XXXXXXXXXX.XXXXXX	Reporter::ERROR	An Error	(empty)
+XXXXXXXXXX.XXXXXX	Reporter::ERROR	An Error	<...>/main.zeek, lines 498-499
 XXXXXXXXXX.XXXXXX	Reporter::ERROR	field value missing	(empty)
 #close XXXX-XX-XX-XX-XX-XX

testing/btest/Baseline.cpp/scripts.base.frameworks.analyzer.tags/output

@@ -0,0 +1,7 @@
+### BTest baseline data generated by btest-diff. Do not edit. Use "btest -U/-u" to update. Requires BTest >= 0.63.
+all, AllAnalyzers::ANALYZER_ANALYZER_DNS
+analyzer, Analyzer::ANALYZER_DNS
+all, AllAnalyzers::PACKETANALYZER_ANALYZER_UDP
+packet analyzer, PacketAnalyzer::ANALYZER_UDP
+all, AllAnalyzers::FILES_ANALYZER_X509
+file analyzer, Files::ANALYZER_X509

testing/btest/Baseline.cpp/scripts.base.frameworks.file-analysis.bifs.file_exists_lookup_file/err

@@ -1,2 +1,3 @@
 ### BTest baseline data generated by btest-diff. Do not edit. Use "btest -U/-u" to update. Requires BTest >= 0.63.
 error: file ID asdf not a known file
+warning: non-void function returning without a value: Files::lookup_file

testing/btest/Baseline.zam/language.vector-deprecated/.stderr

@@ -0,0 +1,7 @@
+### BTest baseline data generated by btest-diff. Do not edit. Use "btest -U/-u" to update. Requires BTest >= 0.63.
+warning in /Users/vern/warehouse/zeek/zeek-master.29Jul22/testing/btest/.tmp/language.vector-deprecated/vector-deprecated.zeek, line 18: mixing vector and scalar operands is deprecated (vector) (string)
+warning in /Users/vern/warehouse/zeek/zeek-master.29Jul22/testing/btest/.tmp/language.vector-deprecated/vector-deprecated.zeek, line 21: mixing vector and scalar operands is deprecated (string) (vector)
+warning in /Users/vern/warehouse/zeek/zeek-master.29Jul22/testing/btest/.tmp/language.vector-deprecated/vector-deprecated.zeek, line 24: mixing vector and scalar operands is deprecated (string) (vector)
+error: deprecated mixed vector/scalar operation not supported for ZAM compiling
+error: deprecated mixed vector/scalar operation not supported for ZAM compiling
+error: deprecated mixed vector/scalar operation not supported for ZAM compiling

testing/btest/Baseline.zam/language.vector-deprecated/out

@@ -0,0 +1,4 @@
+### BTest baseline data generated by btest-diff. Do not edit. Use "btest -U/-u" to update. Requires BTest >= 0.63.
++ operator [string] (PASS)
+== operator [string] (PASS)
+== operator [string] (PASS)

testing/btest/Baseline/bifs.disable_analyzer-hook/out

@@ -0,0 +1,16 @@
+### BTest baseline data generated by btest-diff. Do not edit. Use "btest -U/-u" to update. Requires BTest >= 0.63.
+proto confirm, Analyzer::ANALYZER_HTTP
+http_request, GET, /style/enhanced.css
+preventing disable_analyzer, [orig_h=192.168.1.104, orig_p=1673/tcp, resp_h=63.245.209.11, resp_p=80/tcp], Analyzer::ANALYZER_HTTP, 3, 1
+F
+http_reply, 200
+http_request, GET, /script/urchin.js
+preventing disable_analyzer, [orig_h=192.168.1.104, orig_p=1673/tcp, resp_h=63.245.209.11, resp_p=80/tcp], Analyzer::ANALYZER_HTTP, 3, 3
+F
+http_reply, 200
+http_request, GET, /images/template/screen/bullet_utility.png
+allowing disable_analyzer, [orig_h=192.168.1.104, orig_p=1673/tcp, resp_h=63.245.209.11, resp_p=80/tcp], Analyzer::ANALYZER_HTTP, 3, 5
+T
+total http messages, {
+[[orig_h=192.168.1.104, orig_p=1673/tcp, resp_h=63.245.209.11, resp_p=80/tcp]] = 5
+}

testing/btest/Baseline/bifs.table_keys/out

@@ -0,0 +1,13 @@
+### BTest baseline data generated by btest-diff. Do not edit. Use "btest -U/-u" to update. Requires BTest >= 0.63.
+{
+http,
+https
+}
+{
+[a=<uninitialized>, b=7],
+[a=10, b=5]
+}
+{
+[1/tcp, test, T] ,
+[2/tcp, example, F] 
+}

testing/btest/Baseline/bifs.table_values/out

@@ -0,0 +1,3 @@
+### BTest baseline data generated by btest-diff. Do not edit. Use "btest -U/-u" to update. Requires BTest >= 0.63.
+[example, test]
+[80/tcp, http, 443/tcp, https, 21/tcp, ftp, 23/tcp, telnet]

testing/btest/Baseline/core.option-zeek-done/.stdout

@@ -0,0 +1,7 @@
+### BTest baseline data generated by btest-diff. Do not edit. Use "btest -U/-u" to update. Requires BTest >= 0.63.
+Initial value, T
+Value of testbool changed from T to F (zeek_is_terminating=F)
+Next value, F
+Next changed, T
+Final value, F
+Final changed, F

testing/btest/Baseline/core.record-field-redef-errors-10/.stderr

@@ -0,0 +1,2 @@
+### BTest baseline data generated by btest-diff. Do not edit. Use "btest -U/-u" to update. Requires BTest >= 0.63.
+fatal error in <...>/record-field-redef-errors.zeek, line 1: field "no_such_field" not in record "M::Info"

testing/btest/Baseline/core.record-field-redef-errors-11/.stderr

@@ -0,0 +1,2 @@
+### BTest baseline data generated by btest-diff. Do not edit. Use "btest -U/-u" to update. Requires BTest >= 0.63.
+fatal error in <...>/record-field-redef-errors.zeek, line 1: field "no_such_field" not in record "M::Info"

testing/btest/Baseline/core.record-field-redef-errors-12/.stderr

@@ -0,0 +1,2 @@
+### BTest baseline data generated by btest-diff. Do not edit. Use "btest -U/-u" to update. Requires BTest >= 0.63.
+error in <...>/record-field-redef-errors.zeek, line 2: syntax error, at or near "&"

testing/btest/Baseline/core.record-field-redef-errors-2/.stderr

@@ -0,0 +1,2 @@
+### BTest baseline data generated by btest-diff. Do not edit. Use "btest -U/-u" to update. Requires BTest >= 0.63.
+error in <...>/record-field-redef-errors.zeek, line 2: syntax error, at or near ";"

testing/btest/Baseline/core.record-field-redef-errors-3/.stderr

@@ -0,0 +1,2 @@
+### BTest baseline data generated by btest-diff. Do not edit. Use "btest -U/-u" to update. Requires BTest >= 0.63.
+error in <...>/record-field-redef-errors.zeek, line 2: syntax error, at or near "["

testing/btest/Baseline/core.record-field-redef-errors-4/.stderr

@@ -0,0 +1,2 @@
+### BTest baseline data generated by btest-diff. Do not edit. Use "btest -U/-u" to update. Requires BTest >= 0.63.
+fatal error in <...>/record-field-redef-errors.zeek, line 2: Can only redef "&log" attributes of record fields

testing/btest/Baseline/core.record-field-redef-errors-5/.stderr

@@ -0,0 +1,2 @@
+### BTest baseline data generated by btest-diff. Do not edit. Use "btest -U/-u" to update. Requires BTest >= 0.63.
+fatal error in <...>/record-field-redef-errors.zeek, line 2: Can only redef "&log" attributes of record fields

testing/btest/Baseline/core.record-field-redef-errors-6/.stderr

@@ -0,0 +1,2 @@
+### BTest baseline data generated by btest-diff. Do not edit. Use "btest -U/-u" to update. Requires BTest >= 0.63.
+fatal error in <...>/record-field-redef-errors.zeek, line 2: Can only redef "&log" attributes of record fields

testing/btest/Baseline/core.record-field-redef-errors-7/.stderr

@@ -0,0 +1,2 @@
+### BTest baseline data generated by btest-diff. Do not edit. Use "btest -U/-u" to update. Requires BTest >= 0.63.
+fatal error in <...>/record-field-redef-errors.zeek, line 2: identifier "M::ErrCode" has type "enum", expected "record"

testing/btest/Baseline/core.record-field-redef-errors-8/.stderr

@@ -0,0 +1,2 @@
+### BTest baseline data generated by btest-diff. Do not edit. Use "btest -U/-u" to update. Requires BTest >= 0.63.
+fatal error in <...>/record-field-redef-errors.zeek, line 1: unknown record identifier "M::Unknown"

testing/btest/Baseline/core.record-field-redef-errors-9/.stderr

@@ -0,0 +1,2 @@
+### BTest baseline data generated by btest-diff. Do not edit. Use "btest -U/-u" to update. Requires BTest >= 0.63.
+fatal error in <...>/record-field-redef-errors.zeek, line 1: unknown record identifier "M::Unknown"

testing/btest/Baseline/core.record-field-redef-errors/.stderr

@@ -0,0 +1,2 @@
+### BTest baseline data generated by btest-diff. Do not edit. Use "btest -U/-u" to update. Requires BTest >= 0.63.
+error in <...>/record-field-redef-errors.zeek, line 6: syntax error, at or near "&log"