Adds base/frameworks/telemetry with wrappers around telemetry.bif
and updates telemetry/Manager to support collecting metrics from
script land.
Add policy/frameworks/telemetry/log for logging of metrics data
into a new telemetry.log and telemetry_histogram.log and add into
local.zeek by default.
* origin/topic/vern/bit-shift-fixes:
btest portability fix address review comment about shifting corner-case
canonicalize filenames for new vector deprecation btest
updates for gen-C++ maintenance, including skipping some inappropriate tests
fix for profiling "when" statements
gen-C++ support for vector bit-shift operations
corrected wording in some btest comments
make gen-C++ maintenance scripts directly executable
ZAM support for bit-shifting
don't allow deprecated-style mixing of vectors and scaling for shifting leverage restrictions placed on shifting (RHS is always unsigned) split deprecated vector operations into separate test, with separate ZAM baseline
ZAM fix for vector "in" operator
ensure that language tests pay attention to .stderr
fix vector tests, including checking for errors
Calling util::fmt() from DoUpdate() of a thread is not safe as it is
using a statically allocated buffer and other threads or the main
thread may concurrently modify this buffer.
This was found by observing the scripts.base.frameworks.config.several-files
failing once in a blue moon (1/250 sometimes 1/1000 runs) with messages like
"Failed to compile regex: Parenthesis is not closed.":
scripts.base.frameworks.config.several-files ...
> btest-bg-run zeek zeek -b %INPUT
> btest-bg-wait 10
... scripts.base.frameworks.config.several-files failed
% 'btest-bg-wait 10' failed unexpectedly (exit code 1)
% cat .stderr
The following processes did not terminate:
zeek -b /home/awelzel/corelight-oss/zeek/testing/btest/.tmp/scripts.base.frameworks.config.several-files/several-files.zeek
-----------
<<< [3667265] zeek -b /home/awelzel/corelight-oss/zeek/testing/btest/.tmp/scripts.base.frameworks.config.several-files/several-files.zeek
error: ../configfile1/Input::READER_CONFIG: Failed to compile regex: Parenthesis is not closed.
received termination signal
>>>
* origin/topic/timw/1129-slash-s-patterns:
Add support for /s modifier to RE matcher and parser
Code cleanup in RE_Matcher code
Add basic unit tests for RE_Matcher
Add /s modifier to parser for patterns
Due to the asynchronous behavior of the input framework and broker
communication, change handlers were previously called even after
zeek_done() event processing completed and also broker shutdown.
Accessing broker store handles within change handlers this late
triggered invalid Broker store handle messages:
error in ././my_option_store.zeek, line 13: invalid Broker store handle (Broker::put(Test::store, to_any_coercemy_option, to_any_coerceTest::new_value, 0 secs) and broker::store::{})
Fixes#2010
A flood of DHCP traffic can result if very large log entries consisting
of many uids and/or msg_types. Such large log entries can disrupt a SIEM
ingestion pipeline. This change forcing a log entry to be written when
the number of uids or the number of msg_Types exceed a certain value.
The values are treated as options for easy configuration.
