In the past, we used a default canonifier, which removes everything that
looks like a timestamp from log files. The goal of this is to prevent
logs from changing, e.g., due to local system times ending up in log
files.
This, however, also has the side-effect of removing information that is
parsed from protocols which probably should be part of our tests.
There is at least one test (1999 certificates) where the entire test
output was essentially removed by the canonifier.
GH-4521 was similarly masked by this.
This commit changes the default canonifier, so that only the first
timestamp in a line is removed. This should skip timestamps that are
likely to change while keeping timestamps that are parsed
from protocol information.
A pass has been made over the tests, with some additional adjustments
for cases which require the old canonifier.
There are some cases in which we probably could go further and not
remove timestamps at all - that, however, seems like a follow-up
project.
To address review feedback in GH-4362: rename analyzer-failed-log.zeek
to loggig.zeek, analyzer-debug-log.zeek to debug-logging.zeek and
dpd-log.zeek to deprecated-dpd-log.zeek.
Includes respective test, NEWS, etc updates.
The main part of this commit are changes in tests. A lot of the tests
that previously relied on analyzer.log or dpd.log now use the new
analyzer-failed.log.
I verified all the changes and, as far as I can tell, everything
behaves as it should. This includes the external test baselines.
This change also enables logging of file and packet analyzer to
analyzer_failed.log and fixes some small behavior issues.
The analyzer_failed event is no longer raised when the removal of an
analyzer is vetoed.
If an analyzer is no longer active when an analyzer violation is raised,
currently the analyzer_failed event is raised. This can, e.g., happen
when an analyzer error happens at the very end of the connection. This
makes the behavior more similar to what happened in the past, and also
intuitively seems to make sense.
A bug introduced in the failed service logging was fixed.
This commit builds on top of GH-4183 and adds IPv6 support for
policy/protocols/dns/detect-external-names.
Additionally it adds a test-case for this file testing it with mDNS
queries.
This reverts the call to update-crypto-policies in the Fedora 41 image
and instead sets OPENSSL_ENABLE_SHA1_SIGNATURES in the individual tests.
This allows RHEL 10 or Fedora 41 users to run the tests in question
without needing to fiddle with system settings.
Fixes#4035
This wasn't possible before #3028 was fixed, but now it's safe to set
the value in new_connection() and allow other users access to the
field much earlier. We do not have to deal with connection_flipped()
because the community-id hash is symmetric.
remove instance of plus sign to account for real plus in sql
account for spaces encoding to plus signs in sqli regex detection
add test cases for sqli space to plus
account for spaces encoding to plus signs in sqli regex detection
forgot semicolon
account for spaces encoding to plus signs in sqli regex detection
Adding a metric for the network time value itself should make it
possible to observe it stopping or growing slowly as compared to
realtime when Zeek isn't able to keep up.
Also, modify the telemetry/log.zeek test to include misc/stats and
log at a higher frequency with a more interesting pcap.
* origin/master: (60 commits)
Update gen-zam submodule [nomail] [skip ci]
Update doc submodule [nomail] [skip ci]
Remove unused wrapper packet analyzer
Add DNS TKEY event
ScriptOpt: Ensure global statements have non-null scope
simpler and more robust identification of function parameters for AST profiling
fixes to limit AST traversal in the face of recursive types
address some script optimization compiler warnings under Linux
fix for -O C++ construction of variable names that use multiple module namespaces
fix for script optimization of "opaque" values that are run-time constants
fix for script optimization of nested switch statements
script optimization fix for complex "in" expressions in conditionals
updates to typos allow-list reflecting ZAM regularization changes
BTest updates for ZAM regularization changes
convert new ZAM operations to use typed operands
complete migration of ZAM to use only public ZVal methods
"-O validate-ZAM" option to validate generated ZAM instructions
internal option to suppress control-flow optimization
exposing some functionality for greater flexibility in structuring run-time execution
rework ZAM compilation of type switches to leverage value switches
...
With this commit, the entire Zeek test suite passes using spicy TLS.
Tests that either use a SSLv2 handshake, or DTLS are skipped, as the
parser currently does not support either.
Similarly, tests that rely on behavior we cannot replicate (baseline,
hooks, exact error messages) are passed. Other than that, all the
TLS-based tests pass with 100% the exact same baseline results.
This necessitated a couple of small tweaks to the spicy file - the
testcases uncovered several small problems.
This commit also enables cirrus tests for Spicy SSL/TLS.
The analyzer now detects partial connections at the beginning of a
connection - and will skip them. This makes behavior more similar to the
binpac analyzer.
The decryption test is skipped.
And some minor refacoring.
With Cluster::Node$metrics_port being optional, there's not really
a need for the extra script. New rule, if a metrics_port is set, the
node will attempt to listen on it.
Users can still redef Telemetry::metrics_port *after*
base/frameworks/telemetry was loaded to change the port defined
in cluster-layout.zeek.
Catch-and-release logs now include the plugin that is responsible for an
action. Furthermore, the catch-and-release log also includes instances
where a rule already existed, and where an error occurred during an
operation.
While it seems interesting functionality, this hasn't been documented,
maintained or knowingly leveraged for many years.
There are various other approaches today, too:
* We track the number of event handler invocations regardless of
profiling. It's possible to approximate a load_sample event by
comparing the result of two get_event_stats() calls. Or, visualize
the corresponding counters in a Prometheus setup to get an idea of
event/s broken down by event names.
* HookCallFunction() allows to intercept script execution, including
measuring the time execution takes.
* The global call_stack and g_frame_stack can be used from plugins
(and even external processes) to walk the Zeek script stack at certain
points to implement a sampling profiler.
* USDT probes or more plugin hooks will likely be preferred over Zeek
builtin functionality in the future.
Relates to #3458
This field isn't required by a worker and it's certainly not used by a
worker to listen on that specific interface. It also isn't required to
be set consistently and its use in-tree limited to the old load-balancing
script.
There's a bif called packet_source() which on a worker will provide
information about the actually used packet source.
Relates to zeek/zeek#2877.
The dump-events baseline changes are pure noise and have spurred confusion
for internal and external contributors. For example, adding new
analyzers have perturbed orderings of sets holding analyzer tags.
Running in non-bare mode, the baselines change almost whenever any of the
record types attached to connections change in the default scripts. This
causes continuous and seemingly little useful updates to the baselines.
This change switches the test to run in bare mode and explicitly loads
just base/protocols/conn and base/protocols/smtp. The primary intention
of the test should be testing the functionality of the misc/dump-events
script, not the raised events of all loaded default scripts (for that the
used PCAP is too narrow).
Protocol specific scripts that do want to leverage misc/dump-events for
baseline creation of their or their analyzer's events can add additional
specific tests with suitable PCAP files.
* origin/topic/jazoff/gh-3268:
Fix check for emailed notices
Changes: Added a test-case printing email_delay_tokens to compare email vs
non-email notice types. Previously, both notice types would have email
delay tokens at that point in the flow.
