As initial examples, this branch ports the Syslog and Finger analyzers
over. We leave the old analyzers in place for now and activate them
iff we compile without any Spicy.
Needs `zeek-spicy-infra` branches in `spicy/`, `spicy-plugin/`,
`CMake/`, and `zeek/zeek-testing-private`.
Note that the analyzer events remain associated with the Spicy plugin
for now: that's where they will show up with `-NN`, and also inside
the Zeekygen documentation.
We switch CMake over to linking the runtime library into the plugin,
vs. at the top-level through object libraries.
By default this only logs all the violations, regardless of the
confirmation state (for which there's still dpd.log). It includes
packet, protocol and file analyzers.
This uses options, change handlers and event groups for toggling
the functionality at runtime.
Closes#2031
This commit adds an optional event_groups field to the Logging::Stream record
to associated event groups with logging streams.
This can be used to disable all event groups of a logging stream when it is
disabled. It does require making an explicit connection between the
logging stream and the involved groups, however.
This allows to enable/disable file analyzers through the same interfaces
as packet and protocol analyzers, specifically Analyzer::disable_analyzer
could be interesting.
This adds machinery to the packet_analysis manager for disabling
and enabling packet analyzers and implements two low-level bifs
to use it.
Extend Analyzer::enable_analyzer() and Analyzer::disable_analyzer()
to transparently work with packet analyzers, too. This also allows
to add packet analyzers to Analyzer::disabled_analyzers.
Introduce two new events for analyzer confirmation and analyzer violation
reporting. The current analyzer_confirmation and analyzer_violation
events assume connection objects and analyzer ids are available which
is not always the case. We're already passing aid=0 for packet analyzers
and there's not currently a way to report violations from file analyzers
using analyzer_violation, for example.
These new events use an extensible Info record approach so that additional
(optional) information can be added later without changing the signature.
It would allow for per analyzer extensions to the info records to pass
analyzer specific info to script land. It's not clear that this would be
a good idea, however.
The previous analyzer_confirmation and analyzer_violation events
continue to exist, but are deprecated and will be removed with Zeek 6.1.
...the only known cases where the `-` for `connection$service` was
handled is to skip/ignore these analyzers.
Slight suspicion that join_string_set() should maybe become a bif
now determine_service() runs once for each connection.
Closes#2388
* Because frameworks/analyzer is loaded via init-frameworks-and-bifs the
dpd functionality (really just dpd.log and disabling of analyzers) is
now enabled even in bare mode.
* Not sure we need to keep frameworks/base/dpd/__load__.zeek around
or can just remove it right away.
While reviewing/understanding the analyzer setup, it didn't seem like
GTPv1 implements packet_analysis::Analyzer::DetectProtocol(), so
should not register it for protocol_detection either.
Alternatively, maybe DetectProtocol() should've been implemented in
which case maybe this should be an issue?
* simeonmiteff/master:
Pull changes from zeek/cmake fork
Skip test based on preprocessor flag set by cmake
Set flag for libpcap without DLT_LINUX_SLL2
Force event order in core/init-error btest
Update some coverage baselines
Update plugins/hooks baseline
Add support for DLT_LINUX_SLL2 PCAP link-type
This is a script-only change that unrolls File::Info records into
multiple files.log entries if the same file was seen over different
connections by single worker. Consequently, the File::Info record
gets the commonly used uid and id fields added. These fields are
optional for File::Info - a file may be analyzed without relation
to a network connection (e.g by using Input::add_analysis()).
The existing tx_hosts, rx_hosts and conn_uids fields of Files::Info
are not meaningful after this change and removed by default. Therefore,
files.log will have them removed, too.
The tx_hosts, rx_hosts and conn_uids fields can be revived by using the
policy script frameworks/files/deprecated-txhosts-rxhosts-connuids.zeek
included in the distribution. However, with v6.1 this script will be
removed.
* ynadji/topic/yacin/2319-add-change-handler-to-site:
update plugins.hooks baseline
lower priority for change handlers
split update_zones_regex into two functions
GH-2319: Add change handlers to Site
Adds base/frameworks/telemetry with wrappers around telemetry.bif
and updates telemetry/Manager to support collecting metrics from
script land.
Add policy/frameworks/telemetry/log for logging of metrics data
into a new telemetry.log and telemetry_histogram.log and add into
local.zeek by default.
This was excercising hooks to validate that they were called. For that
it uses pretty verbose logging. Since version numbers are not stable and
hard to canonify this script already unloads the version module (there
is e.g., code in `base/misc/version` which splits the version strings
and works on single components which might appear like pretty ordinary
floating point numbers in the log).
This test however worked under the assumption that nothing in bare mode
loads `base/misc/version` which is hard to guarantee, especially
considering that plugins can be embedded and might load that module
themself.
With this patch we now make the logging slightly less verbose so that
function call arguments are not logged anymore.
* origin/topic/vern/script-profiling:
tidy up after generating profile
test suite updates for refined script coverage, use of new BiF to speed startup
fix for coverage reporting for functions that use "when" statements
new global_options() BiF to speed up startup, plus a micro-preen
hooks for new --profile-scripts option
classes for managing script profiles
address some holes in script coverage
fix for script coverage missing on-exit activity
memory management fixes for loggers
make curr_CPU_time() broadly available rather than just isolated to ZAM
We used to attempt to remove any port specification before recording
HTTP host headers in logs. Doing so would (1) remove potentially useful
information, (2) not match what the documentation seemed to suggest, and
(3) fail for IP6 addresses containing colons.
We now record the original HOST header as is.
Addresses #1844.
These allow packet analyzers to register ports as identifiers to forward from
parent analyzers, while also adding those ports to the now-global
Analyzer::ports table at the same time.
The new hooks works similar to the existing `HookLoadFile` but,
additionally, allows the plugin to return a string that contains the
code to be used for the file being loaded. If the plugin does so, the
content of any actual file on disk will be ignored (in fact, there
doesn't even need to be a file on disk in that case). This works for
both Zeek scripts and signatures.
There's a new test that covers the new functionality, testing loading
both scripts and signatures from memory. I also manually tested that the
debugger integration works, but I don't see much of a way to add a
regression test for that part.
We keep the existing hook as well for backwards compatibility. We could
decide to deprecate it, but not sure that buys us much, so left that
out.
Closes#1757.
This (1) fixes an issue where signature files supplied on the command
line wouldn't pass through the hooks, and (2) prepares for allowing
hooks to supply the content of a signature file directly.
This could lead to a noticeable (single-percent) performance
improvement.
Most of the functionality for this is in the packet analyzers that now
cache ignore_chesksums_nets.
Based on a patch by Arne Welzel (Corelight).
* origin/topic/johanna/gh-859:
Add X509/SSL changes to NEWS
X509: add check if function succeeds
GH-1634: Address feedback
Small indentation fixes in ssl-log-ext.zeek
Fix memory leak in x509_check_cert_hostname bif
Small bugfix and updates for external test hashes (SSL/X509)
Baseline updates for recent SSL changes.
Add ability to check if hostname is valid for a specific cert
Add ssl_history field to ssl.log
Add policy script suppressing certificate events
Add new ssl-log-ext policy script
Deprecate extract-certs-pem.zeek and add log-certs-base64.zeek
Implement X509 certificate log caching
Deprecate ICSI SSL notary script.
Change SSL and X.509 logging format
Enable OCSP logging by default.
Split the code that handles X509 event hashing into its own file
Closes GH-859
This addresses the need for a central hook on any log write, which
wasn't previously doable without a lot of effort. The log manager
invokes the new Log::log_stream_policy hook prior to any filter-specific
hooks. Like filter-level hooks, it may veto a log write. Even when
it does, filter-level hooks still get invoked, but cannot "un-veto".
Includes test cases.
