This started with reverting commit 52cd02173d
and then rewriting it to be per handler rather than handler identifier
and adding support for hooks as well as adding implicit module groups.
Mostly: Do not instantiate variables within for loops and allow
reusing differently typed blanks which previously wasn't possible.
This may be missing some corner-cases, but the added tests seem
to work as expected and nothing else fell apart it seems.
In his ZeekWeek 2022 presentation, @stevesmoot mentioned that he had a
difficult time looking up enum names when all he had was a string
naming the type.
Add support to enum_names() to transparently lookup the type if a string
is provided. This is similar in how record_fields() behaves when being
passed a string.
This has come up a few times and the motivation is mainly better "first timer"
experience with Zeek. Concretely, if one wants to run a Zeek cluster with
multiple workers and reasonable load balancing on Linux, AF_PACKET is a decent
start. Without AF_PACKET support being built into Zeek, however, a new user's
next experience is that of setting up a development environment in order
to compile an external plugin (think compiler, kernel headers, zkg, ...).
Only to get what could be termed basic functionality.
This is using the ZEEK_INCLUDE_PLUGINS infrastructure. I've used the all
upper case spelling of AF_PACKET in the help output because it seems everyone
else references/writes it like that. I think we should also write it
like that in the docs.
This allows to enable/disable file analyzers through the same interfaces
as packet and protocol analyzers, specifically Analyzer::disable_analyzer
could be interesting.
This adds machinery to the packet_analysis manager for disabling
and enabling packet analyzers and implements two low-level bifs
to use it.
Extend Analyzer::enable_analyzer() and Analyzer::disable_analyzer()
to transparently work with packet analyzers, too. This also allows
to add packet analyzers to Analyzer::disabled_analyzers.
Add a test parsing a malformed PE file showing that analyzer_violation_info
is raised with the fa_file object set.
It could be interesting to pass through an optional connection if one
exists, but access is provided through f$conns, too.
Introduce two new events for analyzer confirmation and analyzer violation
reporting. The current analyzer_confirmation and analyzer_violation
events assume connection objects and analyzer ids are available which
is not always the case. We're already passing aid=0 for packet analyzers
and there's not currently a way to report violations from file analyzers
using analyzer_violation, for example.
These new events use an extensible Info record approach so that additional
(optional) information can be added later without changing the signature.
It would allow for per analyzer extensions to the info records to pass
analyzer specific info to script land. It's not clear that this would be
a good idea, however.
The previous analyzer_confirmation and analyzer_violation events
continue to exist, but are deprecated and will be removed with Zeek 6.1.
...the only known cases where the `-` for `connection$service` was
handled is to skip/ignore these analyzers.
Slight suspicion that join_string_set() should maybe become a bif
now determine_service() runs once for each connection.
Closes#2388
* topic/christian/supervisor-pre-user-scripts:
Update Management framework to new Supervisor::NodeConfig script fields
Suppress deprecation warnings for the Supervisor's own NodeConfig::script access
Expand Supervisor to support loading additional scripts before user scripts
Avoid the issue outlined in #2289 where the @if or @else is taken as the
statement of an `if`, `for` or `while` by rejecting such constructs.
Effectively this means the following scripts are now rejected:
# Print's "cond true" with Zeek 5.0 even though the `if ( F )`
# should be in effect.
if ( F )
@if ( T )
print "cond true";
@else
print "cond false";
@endif
or
# Print's "hello" once with Zeek 5.0
local v = vector( 1, 2, 3 );
for ( i in v )
@if ( T )
print("hello")
@endif
To make above work as intended, additional braces can be used.
if ( T )
{
@if ( cond )
print "cond true";
@else
print "cond false";
@endif
}
for ( i in v )
{
@if ( T )
print("hello")
@endif
}
I ran into wanting to iterate over just the values of a vector and wondering
whether that could just work.
This adds support for the following, where v will be value of vec[i].
local vec = vector("zero", "one", "two");
for ( i, v in vec )
print i, v;
I removed `deprecated-txhosts-rxhosts-connuids.zeek` from
`local.zeek`, seems preferable not to have a script-to-go-away in the
standard configuration for new users. Also tweaked `NEWS` just a tiny
bit.
* origin/topic/awelzel/files-log-unrolling:
files.log: Unroll and introduce uid and id fields
This is a script-only change that unrolls File::Info records into
multiple files.log entries if the same file was seen over different
connections by single worker. Consequently, the File::Info record
gets the commonly used uid and id fields added. These fields are
optional for File::Info - a file may be analyzed without relation
to a network connection (e.g by using Input::add_analysis()).
The existing tx_hosts, rx_hosts and conn_uids fields of Files::Info
are not meaningful after this change and removed by default. Therefore,
files.log will have them removed, too.
The tx_hosts, rx_hosts and conn_uids fields can be revived by using the
policy script frameworks/files/deprecated-txhosts-rxhosts-connuids.zeek
included in the distribution. However, with v6.1 this script will be
removed.
* origin/topic/awelzel/1678-disabling-analyzer-hook:
Add NEWS entry and zeekygen-smithing for disabling_analyzer()
Introduce global disabling_analyzer() hook to veto disable_analyzer()
ssl: Only delete c$ssl$analyzer_id when disabling the analyzer was successful
Add new syntax for adding and removing attributes from record fields:
redef RecordType$field_name += { &log };
redef RecordType$field_name -= { &log };
For now this only allowed for the &log attribute as the semantics are clear.
For &default and &optional the semantics aren't obvious and no use-cases have
been identified where those would make sense to change.
This enables a mechanism to add potentially interesting fields to the typical
Info records in base scripts, but letting users opt-into actually including
them into their log. At the same time, users that find specific fields in a
standard log uninteresting can opt-out without using `Log::Filter$exclude`
which can be difficult to use correctly. Patching or forking external packages
to remove columns from a log can also be avoided with this mechanism.
Closes#2000.
The low-level singleton Telemetry BIFs have been removed with the that there
haven't been any users. Singleton metrics can be instantiated by providing
an empty label vector instead and aren't in any way a special concept.
Closes#2262.
Adds base/frameworks/telemetry with wrappers around telemetry.bif
and updates telemetry/Manager to support collecting metrics from
script land.
Add policy/frameworks/telemetry/log for logging of metrics data
into a new telemetry.log and telemetry_histogram.log and add into
local.zeek by default.
Due to the asynchronous behavior of the input framework and broker
communication, change handlers were previously called even after
zeek_done() event processing completed and also broker shutdown.
Accessing broker store handles within change handlers this late
triggered invalid Broker store handle messages:
error in ././my_option_store.zeek, line 13: invalid Broker store handle (Broker::put(Test::store, to_any_coercemy_option, to_any_coerceTest::new_value, 0 secs) and broker::store::{})
Fixes#2010
* origin/topic/awelzel/2120-logdir-leftover:
sqlite default-logdir test: Remove ls ./logs baseline
logging/sqlite: Recognize Log::default_logdir and place files there if set
logging: Introduce Log::default_logdir deprecate LogAscii::logdir and per writer logdir
logging/ascii: Fix .shadow paths when using LogAscii::logdir
* 'update' of https://github.com/leres/zeek:
Return false on error from the other place we call fstat()
Address concerns raised by @0xxon; avoid the new code path when reading from a pipe and return false if fstat() fails after sucessfully opening the file (unlikely).
Tweak some new comments
Remove child program check, it's probably wrong given the test failures it causes
Conform to style police
Only set mtime and ino in Raw::OpenInput() do this for MODE_STREAM and avoid breaking MODE_REREAD
Implement tail -F semantics for input framework MODE_STREAM Open /dev/null if the file is missing during init and wait for file to be created Collect initial ino, dev, and mtime when first opening the file Detect if the file has been replaced and open the new version
Only set mtime and ino in Raw::OpenInput() do this for MODE_STREAM and avoid breaking MODE_REREAD
Implement tail -F semantics for input framework MODE_STREAM Open /dev/null if the file is missing during init and wait for file to be created Collect initial ino, dev, and mtime when first opening the file Detect if the file has been replaced and open the new version
