ssl-log-ext had a bug that caused data present in the SSL connection to
not be logged in some cases. Specifically, the script relied on the base
ssl script to initialize some data structures; however, this means that
protocol messages that arrive before a message is handled by the base
ssl script are not logged.
This commit changes the ssl-log-ext script to also initialize the data
structures; now messages are correctly included in the log in all cases.
The diffs produced by telemetry.log when introducing a weird or
removing/adding protocol specific logs is overwhelming and distracting
without providing value. Exclude telemetry.log similar to how we already
exclude stats.log.
Some more targeted telemetry.log tests exists in the normal testing/btest
suite and that appears more sensible.
* topic/awelzel/3235-dont-flip-broadcasts:
testing: Bump external test suite
dhcp: Handle is_orig=T for connections from server to 255.255.255.255
IPBasedAnalyzer: Don't flip connections when destination is broadcast
This change makes the community-id script that adds the community id to
notice.log automatically load the main script if this was not already
loaded.
In the past, the script just did not perform any action if the main
script was not loaded.
This change also makes the notice script respect the seed/base64
settings that were set in the main script.
Fixes GH-3242
* origin/topic/vern/zam-feature-complete: (23 commits)
documentation updates
updates to ZAM-specific BTest baseline
Address PR review feedback on zam-feature-complete
Updates to BTest baselines due to previous BTest tweaks
tweaks to BTests for ZAM feature completeness; new ZAM-only btest
removal of unused functionality and some follow-on simplifications
feature completeness for ZAM
-O gen-C++ tweaks to be compatible with ZAM changes
ZAM support for "when" statements
ZAM changes intermixed with lambda and "when" support
WhenStmt/WhenInfo restructuring in support of ZAM "when" statements
ZAM support for lambdas
ZAM internals have a notion of "captures" as global-like variables
AST profiling enhnacements in support of script optimization for lambdas/whens
refinements to LambdaExpr's to provide flexibility, support for ZVal captures
support in ScriptFunc class for ZVal-oriented vector of captures
simplifications to the Frame class now that it no longer has to support old-style captures
use Ingredients directly for constructing functions
the "Capture" struct is now a class
more debugging information when dumping script optimization data structures
...
* origin/topic/awelzel/add-community-id:
testing/external: Bump hashes for community_id addition
NEWS: Add entry for Community ID
policy: Import zeek-community-id scripts into protocols/conn frameworks/notice
Add community_id_v1() based on corelight/zeek-community-id
There was a misunderstanding whether to include them by default in
the dns.log, so remove them again.
There had also been a discussion and quirk that AD of a request would
always be overwritten by reply in the dns.log unless the reply is
missing. For now, let users extend dns.log themselves for what best
fits their requirements, rather than adding these flags by default.
Add a btest to print AD and CD flags for smoke testing still.
* topic/christian/gh-2761-default-localnets:
Update NEWS to cover new Site::local_nets behavior.
Bump external testsuites to reflect updated baselines
Add btests for new Site::local_nets behavior
Update plugins.hooks baseline to reflect added config framework activity
Update scripts.policy.misc.dump-events baseline
Update btest baselines to reflect the use of local address ranges.
Tighten local-nets filtering in the logging framework's path-func-column-demote test
Fix scripts.policy.frameworks.intel.removal test given address locality info
Treat private address space as site-local by default
Provide a mechanism to suppress logging of internal config framework activity
* 'dnssec-flag-parse' of github.com:micrictor/zeek-codespace:
Update external testing commit hash for DNS flag changes
Parse DNSSEC AD and CD bits
Updated dump-events baseline which seemed unrelated.
This commit introduces parsing of the CertificateRequest message in the
TLS handshake. It introduces a new event ssl_certificate_request, as
well as a new function parse_distinguished_name, which can be used to
parse part of the ssl_certificate_request event parameters.
This commit also introduces a new policy script, which appends
information about the CAs a TLS server requests in the
CertificateRequest message, if it sends it.
* origin/topic/awelzel/2572-event-handler-stats:
Bump external baselines
external/testing-setup: Less telemetry.log volume
EventHandler: Use telemetry framework for EventHandler.call_count
* origin/topic/awelzel/analyzer-log:
btest/net-control: Use different expiration times for rules
analyzer: Add analyzer.log for logging violations/confirmations
By default this only logs all the violations, regardless of the
confirmation state (for which there's still dpd.log). It includes
packet, protocol and file analyzers.
This uses options, change handlers and event groups for toggling
the functionality at runtime.
Closes#2031
I removed `deprecated-txhosts-rxhosts-connuids.zeek` from
`local.zeek`, seems preferable not to have a script-to-go-away in the
standard configuration for new users. Also tweaked `NEWS` just a tiny
bit.
* origin/topic/awelzel/files-log-unrolling:
files.log: Unroll and introduce uid and id fields
This is a script-only change that unrolls File::Info records into
multiple files.log entries if the same file was seen over different
connections by single worker. Consequently, the File::Info record
gets the commonly used uid and id fields added. These fields are
optional for File::Info - a file may be analyzed without relation
to a network connection (e.g by using Input::add_analysis()).
The existing tx_hosts, rx_hosts and conn_uids fields of Files::Info
are not meaningful after this change and removed by default. Therefore,
files.log will have them removed, too.
The tx_hosts, rx_hosts and conn_uids fields can be revived by using the
policy script frameworks/files/deprecated-txhosts-rxhosts-connuids.zeek
included in the distribution. However, with v6.1 this script will be
removed.
Adds base/frameworks/telemetry with wrappers around telemetry.bif
and updates telemetry/Manager to support collecting metrics from
script land.
Add policy/frameworks/telemetry/log for logging of metrics data
into a new telemetry.log and telemetry_histogram.log and add into
local.zeek by default.
* origin/topic/seth/tsv-logs-utf8-by-default:
Fix mis-usage of string::append that leads to an overflow
Use json_escape_utf8 for all utf8 data in ODesc
Switch the TSV Zeek logs to be UTF8 by default.
