The diffs produced by telemetry.log when introducing a weird or
removing/adding protocol specific logs is overwhelming and distracting
without providing value. Exclude telemetry.log similar to how we already
exclude stats.log.
Some more targeted telemetry.log tests exists in the normal testing/btest
suite and that appears more sensible.
This change makes the community-id script that adds the community id to
notice.log automatically load the main script if this was not already
loaded.
In the past, the script just did not perform any action if the main
script was not loaded.
This change also makes the notice script respect the seed/base64
settings that were set in the main script.
Fixes GH-3242
* origin/topic/vern/zam-feature-complete: (23 commits)
documentation updates
updates to ZAM-specific BTest baseline
Address PR review feedback on zam-feature-complete
Updates to BTest baselines due to previous BTest tweaks
tweaks to BTests for ZAM feature completeness; new ZAM-only btest
removal of unused functionality and some follow-on simplifications
feature completeness for ZAM
-O gen-C++ tweaks to be compatible with ZAM changes
ZAM support for "when" statements
ZAM changes intermixed with lambda and "when" support
WhenStmt/WhenInfo restructuring in support of ZAM "when" statements
ZAM support for lambdas
ZAM internals have a notion of "captures" as global-like variables
AST profiling enhnacements in support of script optimization for lambdas/whens
refinements to LambdaExpr's to provide flexibility, support for ZVal captures
support in ScriptFunc class for ZVal-oriented vector of captures
simplifications to the Frame class now that it no longer has to support old-style captures
use Ingredients directly for constructing functions
the "Capture" struct is now a class
more debugging information when dumping script optimization data structures
...
* origin/topic/timw/2914-truncated-tunnel-plus-things:
Address wire/capture length feedback
packet_analysis/TCP: Do not use untrusted len for DeliverPacket()
Add regression test using pcap from GH-2683
Add btest to test Geneve->VXLAN->Truncated inner packet
IP: Update packet->len with accumulated fragment size
UDP: Forward any remaining data (also empty) to session-analysis
IPTunnel: Compute inner wire length based on cap_len differences.
IP: fix weird name to not be ipv6 specific
UDP: don't validate checksum if caplen < len
PIA: Modernize how struct initialization is done
* origin/topic/awelzel/add-community-id:
testing/external: Bump hashes for community_id addition
NEWS: Add entry for Community ID
policy: Import zeek-community-id scripts into protocols/conn frameworks/notice
Add community_id_v1() based on corelight/zeek-community-id
There was a misunderstanding whether to include them by default in
the dns.log, so remove them again.
There had also been a discussion and quirk that AD of a request would
always be overwritten by reply in the dns.log unless the reply is
missing. For now, let users extend dns.log themselves for what best
fits their requirements, rather than adding these flags by default.
Add a btest to print AD and CD flags for smoke testing still.
* topic/christian/gh-2761-default-localnets:
Update NEWS to cover new Site::local_nets behavior.
Bump external testsuites to reflect updated baselines
Add btests for new Site::local_nets behavior
Update plugins.hooks baseline to reflect added config framework activity
Update scripts.policy.misc.dump-events baseline
Update btest baselines to reflect the use of local address ranges.
Tighten local-nets filtering in the logging framework's path-func-column-demote test
Fix scripts.policy.frameworks.intel.removal test given address locality info
Treat private address space as site-local by default
Provide a mechanism to suppress logging of internal config framework activity
* 'dnssec-flag-parse' of github.com:micrictor/zeek-codespace:
Update external testing commit hash for DNS flag changes
Parse DNSSEC AD and CD bits
Updated dump-events baseline which seemed unrelated.
Mostly, treat HTTP0.9 completely separate. Because we're doing raw
delivery of a body directly, fake enough (connection_close=1, and finish
headers manually) so that the MIME infrastructure thinks it is seeing a
body.
This deals better with the body due to accounting for the first line. Also
it avoids the content line analyzer to strip CRLF/LF and the analyzer
then adding CRLF unconditionally by fully bypassing the content line
analyzer.
Concretely, the vlan-mpls test case contains a HTTP response with LF only,
but the previous implementation would use CRLF, accounting for two many bytes.
Same for the http.no-version test which would previously report a body
length of 280 and now is at 323 (which agrees with wireshark).
Further, the mime_type detection for the http-09 test case works because
it's now seeing the full body.
Drawback: We don't extract headers when a server actually replies with
a HTTP/1.1 message, but grrr, something needs to give I guess.
This commit introduces parsing of the CertificateRequest message in the
TLS handshake. It introduces a new event ssl_certificate_request, as
well as a new function parse_distinguished_name, which can be used to
parse part of the ssl_certificate_request event parameters.
This commit also introduces a new policy script, which appends
information about the CAs a TLS server requests in the
CertificateRequest message, if it sends it.
* origin/topic/awelzel/2572-event-handler-stats:
Bump external baselines
external/testing-setup: Less telemetry.log volume
EventHandler: Use telemetry framework for EventHandler.call_count
The user and password fields are replicated to each of the ftp.log
entries. Using a very large username (100s of KBs) allows to bloat
the log without actually sending much traffic. Further, limit the
arg and reply_msg columns to large, but not unbounded values.
* origin/topic/awelzel/arm64-container-follow-ups-2:
Bump private testsuite for char handling fixes
Base64: report byte as positive integer
netbios_decode: use unsigned char for result
analyzer/http: Do not assume char is signed
cirrus: Run tests in a Debian 11 container, too
As initial examples, this branch ports the Syslog and Finger analyzers
over. We leave the old analyzers in place for now and activate them
iff we compile without any Spicy.
Needs `zeek-spicy-infra` branches in `spicy/`, `spicy-plugin/`,
`CMake/`, and `zeek/zeek-testing-private`.
Note that the analyzer events remain associated with the Spicy plugin
for now: that's where they will show up with `-NN`, and also inside
the Zeekygen documentation.
We switch CMake over to linking the runtime library into the plugin,
vs. at the top-level through object libraries.
* origin/topic/awelzel/analyzer-log:
btest/net-control: Use different expiration times for rules
analyzer: Add analyzer.log for logging violations/confirmations
By default this only logs all the violations, regardless of the
confirmation state (for which there's still dpd.log). It includes
packet, protocol and file analyzers.
This uses options, change handlers and event groups for toggling
the functionality at runtime.
Closes#2031
I removed `deprecated-txhosts-rxhosts-connuids.zeek` from
`local.zeek`, seems preferable not to have a script-to-go-away in the
standard configuration for new users. Also tweaked `NEWS` just a tiny
bit.
* origin/topic/awelzel/files-log-unrolling:
files.log: Unroll and introduce uid and id fields
This is a script-only change that unrolls File::Info records into
multiple files.log entries if the same file was seen over different
connections by single worker. Consequently, the File::Info record
gets the commonly used uid and id fields added. These fields are
optional for File::Info - a file may be analyzed without relation
to a network connection (e.g by using Input::add_analysis()).
The existing tx_hosts, rx_hosts and conn_uids fields of Files::Info
are not meaningful after this change and removed by default. Therefore,
files.log will have them removed, too.
The tx_hosts, rx_hosts and conn_uids fields can be revived by using the
policy script frameworks/files/deprecated-txhosts-rxhosts-connuids.zeek
included in the distribution. However, with v6.1 this script will be
removed.
Adds base/frameworks/telemetry with wrappers around telemetry.bif
and updates telemetry/Manager to support collecting metrics from
script land.
Add policy/frameworks/telemetry/log for logging of metrics data
into a new telemetry.log and telemetry_histogram.log and add into
local.zeek by default.
