Adds base/frameworks/telemetry with wrappers around telemetry.bif
and updates telemetry/Manager to support collecting metrics from
script land.
Add policy/frameworks/telemetry/log for logging of metrics data
into a new telemetry.log and telemetry_histogram.log and add into
local.zeek by default.
A flood of DHCP traffic can result if very large log entries consisting
of many uids and/or msg_types. Such large log entries can disrupt a SIEM
ingestion pipeline. This change forcing a log entry to be written when
the number of uids or the number of msg_Types exceed a certain value.
The values are treated as options for easy configuration.
When setting up the DPD info we previously would get the
`transport_proto` for the connection with `get_conn_transport_proto`.
This function takes a `conn_id` and would fail fatally if the connection
for the given ID was unknown. It seems it was possible to run into such
scenarios when the `analyzer_violation` event was processed after the
connection had been cleaned up.
We now get the `transport_proto` directly from the ports in the
`connection` passed into `analyzer_violation` via
`get_port_transport_proto` which cannot fail.
When a CREATE request contains the FILE_DELETE_ON_CLOSE option and
the subsequent CREATE response indicates success, we now raise the
smb2_file_delete event to log a delete action in smb_files.log and
also give users a way to handle this scenario.
The provided pcap was generated locally by recording a smbtorture run
of the smb2.delete-on-close-perms test case.
Placed the create_options into the CmdInfo record for potential
exposure in smb_cmd.log (wasn't sure how that would look so left it
for the future).
Fixes#2276.
* origin/topic/awelzel/2120-logdir-leftover:
sqlite default-logdir test: Remove ls ./logs baseline
logging/sqlite: Recognize Log::default_logdir and place files there if set
logging: Introduce Log::default_logdir deprecate LogAscii::logdir and per writer logdir
logging/ascii: Fix .shadow paths when using LogAscii::logdir
Also modify FormatRotationPath to keep rotated logs within
Log::default_logdir unless the rotation function explicitly
set dir, e.g. by when the user redef'ed default_rotation_interval.
This PR changes the way in which the SSL analyzer tracks the direction
of connections. So far, the SSL analyzer assumed that the originator of
a connection would send the client hello (and other associated
client-side events), and that the responder would be the SSL servers.
In some circumstances this is not true, and the initiator of a
connection is the server, with the responder being the client. So far
this confused some of the internal statekeeping logic and could lead to
mis-parsing of extensions.
This reversal of roles can happen in DTLS, if a connection uses STUN -
and potentially in some StartTLS protocols.
This PR tracks the direction of a TLS connection using the hello
request, client hello and server hello handshake messages. Furthermore,
it changes the SSL events from providing is_orig to providing is_client,
where is_client is true for the client_side of a connection. Since the
argument positioning in the event has not changed, old scripts will
continue to work seamlessly - the new semantics are what everyone
writing SSL scripts will have expected in any case.
There is a new event that is raised when a connection is flipped. A
weird is raised if a flip happens repeatedly.
Addresses GH-2198.
* zeek-as-org/as-org:
Mark lookup_asn() BIF as deprecated in v6.1
Define geo_autonomous_system record type
Add lookup_autonomous_system() BIF that returns AS number and org
* topic/christian/gh-2134-fix-intel-test-races:
Expand scripts.base.frameworks.intel.cluster-transparency test
Fix races in scripts.base.frameworks.intel.cluster-transparency-with-proxy test
Add Intel::send_store_on_node_up boolean to control min_data_store delivery
This exposes Broker's new WebSocket support in Zeek. To enable it,
call `Broker::listen_websocket()`. Zeek will then start listening on
port 9997 for incoming WebSocket connections.
See the Broker documentation for a description of the message format
expected over these WebSocket connections.
This adds a redefinable const to the internals of the Intel framework, to allow
suppression of the manager sending its current min_data_store when a worker
connects. This feature is desirable for nodes that check in "late" to bring them
up to speed, but during testing it introduces nondeterminism.
The Supervisor generates this event every time it receives a status update from
the stem, meaning a node got created or re-created. A corresponding
SupervisorControl::node_status event relays the same information for users
interacting with the Supervisor over Broker.
When passing an empty string as a directory, the function would produce
filenames starting with a slash even when the given file_name is not an absolute
path. Defaulting to the root directory is likely never intended and might
conveivably be dangerous. The middle "/" is now skipped also if dir is an empty
string.
* origin/topic/vern/script-profiling:
tidy up after generating profile
test suite updates for refined script coverage, use of new BiF to speed startup
fix for coverage reporting for functions that use "when" statements
new global_options() BiF to speed up startup, plus a micro-preen
hooks for new --profile-scripts option
classes for managing script profiles
address some holes in script coverage
fix for script coverage missing on-exit activity
memory management fixes for loggers
make curr_CPU_time() broadly available rather than just isolated to ZAM
I needed to figure out which exact algorithm we use for our
probabilistic top-k measurements. It turns out that we do not mention
this in our source tree at all so far.
* origin/topic/vern/table-attr-fixes:
updates for btests - new cases to check, new baselines
updates for btests - new cases to check, new baselines
fix for ill-formed (complex) &default function
type-checking for use of empty table constructors in expressions
catch empty constructors used for type inference suppress repeated error messages
factoring to make checking of &default attributes externally accessible
bug fix for empty table constructors with &default attributes (plus a typo)
This commit changes DPD matching for TLS connections. A one-sided match
is enough to enable DPD now.
This commit also removes DPD for SSLv2 connections. SSLv2 connections do
basically no longer happen in the wild. SSLv2 is also really finnicky to
identify correctly - there is very little data required to match it, and
basically all matches today will be false positives. If DPD for SSLv2 is
still desired, the optional signature in policy/protocols/ssl/dpd-v2.sig
can be loaded.
Fixes GH-1952
* origin/topic/vern/when-lambda:
explicitly provide the frame for evaluating a "when" timeout expression
attempt to make "when" btest deterministic
tests for new "when" semantics/errors
update existing test suite usage of "when" statements to include captures
update uses of "when" in base scripts to include captures
captures for "when" statements update Triggers to IntrusivePtr's and simpler AST traversal introduce IDSet type, migrate associated "ID*" types to "const ID*"
logic (other than in profiling) for assignments that yield separate values
option for internal use to mark a function type as allowing non-expression returns
removed some now-obsolete profiling functionality
minor commenting clarifications
We used to attempt to remove any port specification before recording
HTTP host headers in logs. Doing so would (1) remove potentially useful
information, (2) not match what the documentation seemed to suggest, and
(3) fail for IP6 addresses containing colons.
We now record the original HOST header as is.
Addresses #1844.
