Mirror/zeek
1
0
Fork 0
mirror of https://github.com/zeek/zeek.git synced 2025-10-06 16:48:19 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 2

Merge remote-tracking branch 'origin/master' into topic/johanna/table-on-change

Browse source
This commit is contained in:
Johanna Amann 2020-01-31 10:06:52 -08:00
commit 7166cb7373
146 changed files with 4562 additions and 1180 deletions
Download patch file Download diff file Expand all files Collapse all files

2
.cirrus.yml
View file

@ -94,7 +94,7 @@ ubuntu16_task:
# image is available.
macos_task:
osx_instance:
image: mojave-base
image: catalina-base
prepare_script: ./ci/macos/prepare.sh
<< : *CI_TEMPLATE
env:

124
CHANGES
View file

@ -1,4 +1,128 @@
3.1.0-dev.482 | 2020-01-30 19:16:03 -0800
* Update NEWS for naming changes. (Robin Sommer, Corelight)
* Remove test for legacy plugin. (Robin Sommer, Corelight)
We no longer support creating new plugins using the old naming
scheme.
* Remove legancy symlinks in aux/. (Robin Sommer, Corelight)
That's aux/bro-aux and aux/broctl.
* Add warnings when loading scripts ending in ".bro", or using legacy environment variables. (Robin Sommer, Corelight)
* Fix missing rename to .zeek-dns-cache. (Robin Sommer, Corelight)
* No longer symlink local.zeek to local.bro. (Robin Sommer, Corelight)
* Update notice user agent. (Robin Sommer, Corelight)
* Remove old_comm_usage_is_ok. (Robin Sommer, Corelight)
* Remove bro-config.h.in and bro-path-dev.in. (Robin Sommer, Corelight)
* Change Bro wrapper script to now abort when old executable names are still used. (Robin Sommer, Corelight)
* Remove APIs that were explicitly deprecated to be removed in 3.1. (Robin Sommer, Corelight)
Special handling for bro_{init,done,script_loaded} events: if still
used, they cause Zeek to abort at startup.
3.1.0-dev.469 | 2020-01-30 08:44:15 -0700
* Fix leaked FDs in redirecting supervised node stdout/stderr (Jon Siwek, Corelight)
* Add checks for failed fcntl calls (Jon Siwek, Corelight)
* Set Pipe file descriptor flags correctly (Jon Siwek, Corelight)
3.1.0-dev.465 | 2020-01-29 11:45:15 -0800
* Scope: fix memory leak by removing duplicate copy_string() call (Max Kellermann)
Note this is a parse-time memory leak, not a run-time leak.
* util, nb_dns: fix off-by-one bugs in strncpy() calls (Max Kellermann)
Fortunately, these bugs had no effect because the following lines
overwrote the last character with a null byte.
* Type, util: add `constexpr` to static variables (Max Kellermann)
* Net: remove unused variable (Max Kellermann)
3.1.0-dev.460 | 2020-01-29 11:16:15 -0800
* Skip file analysis for zero-length SSL/TLS data (Jon Siwek, Corelight)
3.1.0-dev.458 | 2020-01-29 12:53:32 +0000
* Add a new supervisor framework that enables Zeek to operate
clusters of processes itself without any external help. (Jon
Siwek, Corelight)
The Supervisor framework provides an entirely new deployment mode
for Zeek, one that supervises a set of Zeek processes that are
meant to be persistent. A Supervisor automatically revives any
process that dies or exits prematurely and also arranges for an
ordered shutdown of the entire process tree upon its own
termination. This Supervisor mode for Zeek provides the basic
foundation for process configuration/management that could be used
to deploy a Zeek cluster similar to what ZeekControl does, but is
also simpler to integrate as a standard system service.
This mode is still experimental and will evolve over time. The
command-line argument of ``-j`` toggles Zeek to run in "Supervisor
mode" to allow for creation and management of child processes. If
you're going to test this, please note that you will need some
custom script code to configure the processes you want Zeek to
run. See the documentation for more information:
https://docs.zeek.org/en/stable/frameworks/supervisor.html
3.1.0-dev.408 | 2020-01-28 17:56:02 -0800
* Update Cirrus CI config to use macOS Catalina (Jon Siwek, Corelight)
3.1.0-dev.406 | 2020-01-28 12:42:25 -0800
* util: use getrandom() on Linux if available (Max Kellermann)
Unlike /dev/urandom, getrandom() doesn't need a file descriptor and
works when there is no /dev. It requires Linux 3.17 and glibc 2.25,
but there is a fallback to the old code.
3.1.0-dev.403 | 2020-01-24 15:15:04 -0800
* Fixed decompose_uri() errors on URIs with empty port component (Frerich Raabe)
3.1.0-dev.398 | 2020-01-23 18:08:43 -0800
* Skip check for outdated docs in Cirrus CI for PRs (Jon Siwek, Corelight)
3.1.0-dev.397 | 2020-01-23 15:30:33 -0800
* Use consistent zeek_init priority for Log::create_stream calls
Typically in base scripts, Log::create_stream() is called in zeek_init()
handler with &priority=5 such that it will have already been created
in the default zeek_init() &priority=0. (Jon Siwek, Corelight)
3.1.0-dev.395 | 2020-01-23 13:50:42 +0000
* Update Broker submodule.
3.1.0-dev.394 | 2020-01-23 13:18:54 +0000
* Improve creation of SMB transaction data strings. (Jon Siwek,
Corelight)
3.1.0-dev.392 | 2020-01-23 13:14:28 +0000
* Improve FTP word/whitespace handling. (Jon Siwek, Corelight)
3.1.0-dev.389 | 2020-01-18 10:49:15 +0000
* GHI-595: Convert from nlohmann/json to RapidJSON for performance

3
CMakeLists.txt
View file

@ -379,11 +379,8 @@ endif ()
string(TOLOWER ${CMAKE_BUILD_TYPE} CMAKE_BUILD_TYPE_LOWER)
configure_file(${CMAKE_CURRENT_SOURCE_DIR}/zeek-config.h.in
${CMAKE_CURRENT_BINARY_DIR}/zeek-config.h)
configure_file(${CMAKE_CURRENT_SOURCE_DIR}/bro-config.h.in
${CMAKE_CURRENT_BINARY_DIR}/bro-config.h)
include_directories(${CMAKE_CURRENT_BINARY_DIR})
install(FILES ${CMAKE_CURRENT_BINARY_DIR}/zeek-config.h DESTINATION include/zeek)
install(FILES ${CMAKE_CURRENT_BINARY_DIR}/bro-config.h DESTINATION include/zeek)
if ( CAF_ROOT_DIR )
set(ZEEK_CONFIG_CAF_ROOT_DIR ${CAF_ROOT_DIR})

35
NEWS
View file

@ -9,6 +9,27 @@ Zeek 3.1.0
New Functionality
-----------------
- Add a new supervisor framework that enables Zeek to operate clusters
of processes itself without any external help.
The Supervisor framework provides an entirely new deployment mode
for Zeek, one that supervises a set of Zeek processes that are meant
to be persistent. A Supervisor automatically revives any process
that dies or exits prematurely and also arranges for an ordered
shutdown of the entire process tree upon its own termination. This
Supervisor mode for Zeek provides the basic foundation for process
configuration/management that could be used to deploy a Zeek cluster
similar to what ZeekControl does, but is also simpler to integrate
as a standard system service.
This mode is still experimental and will evolve over time. The
command-line argument of ``-j`` toggles Zeek to run in "Supervisor
mode" to allow for creation and management of child processes. If
you're going to test this, please note that you will need some
custom script code to configure the processes you want Zeek to run.
See the documentation for more information:
https://docs.zeek.org/en/stable/frameworks/supervisor.html
- Add a new option, ``dpd_late_match_stop``, which can be used in conjuction
with the option ``dpd_match_only_beginning`` and the new event
``protocol_late_match`` to help annotate the conn.log with a field
@ -29,6 +50,15 @@ New Functionality
Changed Functionality
---------------------
- The backwards-compability wrappers & work-arounds introduced in 3.0
for the "Bro to Zeek rename" have either changed their operation, or in some
cases been removed. Generally, anything that reported a
naming-related warning in 3.0 now aborts with a corresponding error
message. In cases where 3.0 silently continued to accept old names,
3.1 now reports warnings. Most importantly, that's loading of
scripts with ".bro" endings, which are now flagged and should be
renamed.
- The key type of ``Known::service_store`` has changed to
``Known::AddrPortServTriplet`` and ``Known::services`` is now a table
instead of just a set.
@ -67,6 +97,11 @@ Changed Functionality
previously used. Output from the formatters remains nearly
identical.
- The ``decompose_uri`` function no longer raises an error when parsing
URIs with an empty port number (e.g. ``http://example.org:/``). Instead,
the ``portnum`` component of the returned ``URI`` value is left
uninitialized.
Removed Functionality
---------------------

2
VERSION
View file

@ -1 +1 @@
3.1.0-dev.389
3.1.0-dev.482

1
aux/bro-aux
View file

@ -1 +0,0 @@
zeek-aux

1
aux/broctl
View file

@ -1 +0,0 @@
zeekctl

2
aux/broker

@ -1 +1 @@
Subproject commit 2854e3e30810673fd6b9200321fa4ada364c559b
Subproject commit 89a43997d6715fca4e2242a3b88b3aa1c075a340

2
aux/netcontrol-connectors

@ -1 +1 @@
Subproject commit 6bb6709e755ecd2b930ff4a3ddd68f16d2b52cba
Subproject commit ba7b6c81a17b578fd4723e903c0a8b112b43e1b1

2
aux/zeek-aux

@ -1 +1 @@
Subproject commit 0790f420148806c1380fc7e0e0a4278c7970753c
Subproject commit 5b98e96695c334b6c2929fb96666d58eb22f9b66

2
aux/zeekctl

@ -1 +1 @@
Subproject commit 6933b86e60f22f7a39ac1a8adbee4867902ce02e
Subproject commit f8941cd0ba171d9ea2b2e01724597e5e790ffb16

5
bro-config.h.in
View file

@ -1,5 +0,0 @@
#pragma once
#pragma message("Warning: bro-config.h is deprecated, use zeek-config.h instead")
#include "zeek-config.h"

1
bro-path-dev.in
View file

@ -1 +0,0 @@
zeek-path-dev.in

2
doc

@ -1 +1 @@
Subproject commit 7192dbedf3ca9ce49294057262074f0e888177f3
Subproject commit 566174d004c14d061fcf7c03e4829f20d46aaad8

20
scripts/CMakeLists.txt
View file

@ -7,26 +7,6 @@ install(DIRECTORY ./ DESTINATION ${ZEEK_SCRIPT_INSTALL_PATH} FILES_MATCHING
PATTERN "*.fp"
)
if ( NOT BINARY_PACKAGING_MODE )
# If the user has a local.bro file from a previous installation, prefer to
# symlink local.zeek to it to avoid breaking their custom configuration --
# because ZeekControl will now prefer to load local.zeek rather than local.bro
# and we're about to install a default version of local.zeek.
set(_local_bro_dst ${ZEEK_SCRIPT_INSTALL_PATH}/site/local.bro)
set(_local_zeek_dst ${ZEEK_SCRIPT_INSTALL_PATH}/site/local.zeek)
install(CODE "
if ( \"\$ENV{DESTDIR}\" STREQUAL \"\" )
if ( EXISTS \"${_local_bro_dst}\" AND NOT EXISTS \"${_local_zeek_dst}\" )
message(STATUS \"WARNING: installed ${_local_zeek_dst} as symlink to ${_local_bro_dst}\")
execute_process(COMMAND \"${CMAKE_COMMAND}\" -E create_symlink
\"${_local_bro_dst}\" \"${_local_zeek_dst}\")
endif ()
endif ()
")
endif ()
# Install local script as a config file since it's meant to be modified directly.
InstallPackageConfigFile(
${CMAKE_CURRENT_SOURCE_DIR}/site/local.zeek

5
scripts/base/frameworks/cluster/__load__.zeek
View file

@ -18,7 +18,12 @@ redef Broker::log_topic = Cluster::rr_log_topic;
# Loading the cluster framework requires that a script by this name exists
# somewhere in the ZEEKPATH. The only thing in the file should be the
# cluster definition in the :zeek:id:`Cluster::nodes` variable.
@if ( ! Supervisor::__init_cluster() )
# When running a supervised cluster, Cluster::nodes is instead populated
# from the internal C++-layer directly via the above BIF.
@load cluster-layout
@endif
@if ( Cluster::node in Cluster::nodes )

8
scripts/base/frameworks/cluster/main.zeek
View file

@ -287,7 +287,13 @@ function is_enabled(): bool
function local_node_type(): NodeType
{
return is_enabled() ? nodes[node]$node_type : NONE;
if ( ! is_enabled() )
return NONE;
if ( node !in nodes )
return NONE;
return nodes[node]$node_type;
}
function node_topic(name: string): string

3
scripts/base/frameworks/cluster/nodes/logger.zeek
View file

@ -23,4 +23,7 @@ redef Log::default_rotation_interval = 1 hrs;
redef Log::default_mail_alarms_interval = 24 hrs;
## Use the cluster's archive logging script.
@if ( ! Supervisor::is_supervised() )
redef Log::default_rotation_postprocessor_cmd = "archive-log";
@endif

2
scripts/base/frameworks/notice/main.zeek
View file

@ -402,7 +402,7 @@ function email_headers(subject_desc: string, dest: string): string
"From: ", mail_from, "\n",
"Subject: ", mail_subject_prefix, " ", subject_desc, "\n",
"To: ", dest, "\n",
"User-Agent: Bro-IDS/", zeek_version(), "\n");
"User-Agent: Zeek/", zeek_version(), "\n");
if ( reply_to != "" )
header_text = string_cat(header_text, "Reply-To: ", reply_to, "\n");
return header_text;

2
scripts/base/frameworks/signatures/main.zeek
View file

@ -140,7 +140,7 @@ global count_per_orig: table[addr, string] of count
global did_sig_log: set[string] &read_expire = 1 hr;
event zeek_init()
event zeek_init() &priority=5
{
Log::create_stream(Signatures::LOG, [$columns=Info, $ev=log_signature, $path="signatures"]);
}

3
scripts/base/frameworks/supervisor/__load__.zeek Normal file
View file

@ -0,0 +1,3 @@
@load ./api
@load ./control
@load ./main

120
scripts/base/frameworks/supervisor/api.zeek Normal file
View file

@ -0,0 +1,120 @@
##! The Zeek process supervision API.
##! This API was introduced in Zeek 3.1.0 and considered unstable until 4.0.0.
##! That is, it may change in various incompatible ways without warning or
##! deprecation until the stable 4.0.0 release.
module Supervisor;
export {
## The role a supervised-node will play in Zeek's Cluster Framework.
type ClusterRole: enum {
NONE,
LOGGER,
MANAGER,
PROXY,
WORKER,
};
## Describes configuration of a supervised-node within Zeek's Cluster
## Framework.
type ClusterEndpoint: record {
## The role a supervised-node will play in Zeek's Cluster Framework.
role: ClusterRole;
## The host/IP at which the cluster node runs.
host: addr;
## The TCP port at which the cluster node listens for connections.
p: port;
## The interface name from which the node will read/analyze packets.
## Typically used by worker nodes.
interface: string &optional;
};
## Configuration options that influence behavior of a supervised Zeek node.
type NodeConfig: record {
## The name of the supervised node. These are unique within a given
## supervised process tree and typically human-readable.
name: string;
## The interface name from which the node will read/analyze packets.
interface: string &optional;
## The working directory that the node should use.
directory: string &optional;
## The filename/path to which the node's stdout will be redirected.
stdout_file: string &optional;
## The filename/path to which the node's stderr will be redirected.
stderr_file: string &optional;
## Additional script filenames/paths that the node should load.
scripts: vector of string &default = vector();
## A cpu/core number to which the node will try to pin itself.
cpu_affinity: int &optional;
## The Cluster Layout definition. Each node in the Cluster Framework
## knows about the full, static cluster topology to which it belongs.
## Entries use node names for keys. The Supervisor framework will
## automatically translate this table into the right Cluster Framework
## configuration when spawning supervised-nodes. E.g. it will
## populate the both the CLUSTER_NODE environment variable and
## :zeek:see:`Cluster::nodes` table.
cluster: table[string] of ClusterEndpoint &default=table();
};
## The current status of a supervised node.
type NodeStatus: record {
## The desired node configuration.
node: NodeConfig;
## The current or last known process ID of the node. This may not
## be initialized if the process has not yet started.
pid: int &optional;
};
## The current status of a set of supervised nodes.
type Status: record {
## The status of supervised nodes, keyed by node names.
nodes: table[string] of NodeStatus;
};
## Create a new supervised node process.
## It's an error to call this from a process other than a Supervisor.
##
## node: the desired configuration for the new supervised node process.
##
## Returns: an empty string on success or description of the error/failure.
global create: function(node: NodeConfig): string;
## Retrieve current status of a supervised node process.
## It's an error to call this from a process other than a Supervisor.
##
## node: the name of the node to get the status of or an empty string
## to mean "all nodes".
##
## Returns: the current status of a set of nodes.
global status: function(node: string &default=""): Status;
## Restart a supervised node process by destroying (killing) and
## re-recreating it.
## It's an error to call this from a process other than a Supervisor.
##
## node: the name of the node to restart or an empty string to mean
## "all nodes".
##
## Returns: true on success.
global restart: function(node: string &default=""): bool;
## Destroy and remove a supervised node process.
## It's an error to call this from a process other than a Supervisor.
##
## node: the name of the node to destroy or an empty string to mean
## "all nodes".
##
## Returns: true on success.
global destroy: function(node: string &default=""): bool;
## Returns: true if this is the Supervisor process.
global is_supervisor: function(): bool;
## Returns: true if this is a supervised node process.
global is_supervised: function(): bool;
## Returns: the node configuration if this is a supervised node.
## It's an error to call this function from a process other than
## a supervised one.
global node: function(): NodeConfig;
}

89
scripts/base/frameworks/supervisor/control.zeek Normal file
View file

@ -0,0 +1,89 @@
##! The Zeek process supervision (remote) control API. This defines a Broker topic
##! prefix and events that can be used to control an external Zeek supervisor process.
##! This API was introduced in Zeek 3.1.0 and considered unstable until 4.0.0.
##! That is, it may change in various incompatible ways without warning or
##! deprecation until the stable 4.0.0 release.
@load ./api
module SupervisorControl;
export {
## The Broker topic prefix to use when subscribing to Supervisor API
## requests and when publishing Supervisor API responses. If you are
## publishing Supervisor requests, this is also the prefix string to use
## for their topic names.
const topic_prefix = "zeek/supervisor" &redef;
## Send a request to a remote Supervisor process to create a node.
##
## reqid: an arbitrary string that will be directly echoed in the response
##
## node: the desired configuration for the new supervised node process.
global SupervisorControl::create_request: event(reqid: string, node: Supervisor::NodeConfig);
## Handle a response from a Supervisor process that received
## :zeek:see:`SupervisorControl::create_request`.
##
## reqid: an arbitrary string matching the value in the original request.
##
## result: the return value of the remote call to
## :zeek:see:`Supervisor::create`.
global SupervisorControl::create_response: event(reqid: string, result: string);
## Send a request to a remote Supervisor process to retrieve node status.
##
## reqid: an arbitrary string that will be directly echoed in the response
##
## node: the name of the node to get status of or empty string to mean "all
## nodes".
global SupervisorControl::status_request: event(reqid: string, node: string);
## Handle a response from a Supervisor process that received
## :zeek:see:`SupervisorControl::status_request`.
##
## reqid: an arbitrary string matching the value in the original request.
##
## result: the return value of the remote call to
## :zeek:see:`Supervisor::status`.
global SupervisorControl::status_response: event(reqid: string, result: Supervisor::Status);
## Send a request to a remote Supervisor process to restart a node.
##
## reqid: an arbitrary string that will be directly echoed in the response
##
## node: the name of the node to restart or empty string to mean "all
## nodes".
global SupervisorControl::restart_request: event(reqid: string, node: string);
## Handle a response from a Supervisor process that received
## :zeek:see:`SupervisorControl::restart_request`.
##
## reqid: an arbitrary string matching the value in the original request.
##
## result: the return value of the remote call to
## :zeek:see:`Supervisor::restart`.
global SupervisorControl::restart_response: event(reqid: string, result: bool);
## Send a request to a remote Supervisor process to destroy a node.
##
## reqid: an arbitrary string that will be directly echoed in the response
##
## node: the name of the node to destory or empty string to mean "all
## nodes".
global SupervisorControl::destroy_request: event(reqid: string, node: string);
## Handle a response from a Supervisor process that received
## :zeek:see:`SupervisorControl::destroy_request`.
##
## reqid: an arbitrary string matching the value in the original request.
##
## result: the return value of the remote call to
## :zeek:see:`Supervisor::destroy`.
global SupervisorControl::destroy_response: event(reqid: string, result: bool);
## Send a request to a remote Supervisor to stop and shutdown its
## process tree. There is no response to this message as the Supervisor
## simply terminates on receipt.
global SupervisorControl::stop_request: event();
}

94
scripts/base/frameworks/supervisor/main.zeek Normal file
View file

@ -0,0 +1,94 @@
##! Implements Zeek process supervision API and default behavior for its
##! associated (remote) control events.
@load ./api
@load ./control
@load base/frameworks/broker
function Supervisor::status(node: string): Supervisor::Status
{
return Supervisor::__status(node);
}
function Supervisor::create(node: Supervisor::NodeConfig): string
{
return Supervisor::__create(node);
}
function Supervisor::destroy(node: string): bool
{
return Supervisor::__destroy(node);
}
function Supervisor::restart(node: string): bool
{
return Supervisor::__restart(node);
}
function Supervisor::is_supervisor(): bool
{
return Supervisor::__is_supervisor();
}
function Supervisor::is_supervised(): bool
{
return Supervisor::__is_supervised();
}
function Supervisor::node(): Supervisor::NodeConfig
{
return Supervisor::__node();
}
event zeek_init() &priority=10
{
Broker::subscribe(SupervisorControl::topic_prefix);
}
event SupervisorControl::stop_request()
{
if ( ! Supervisor::is_supervisor() )
return;
terminate();
}
event SupervisorControl::status_request(reqid: string, node: string)
{
if ( ! Supervisor::is_supervisor() )
return;
local res = Supervisor::status(node);
local topic = SupervisorControl::topic_prefix + fmt("/status_response/%s", reqid);
Broker::publish(topic, SupervisorControl::status_response, reqid, res);
}
event SupervisorControl::create_request(reqid: string, node: Supervisor::NodeConfig)
{
if ( ! Supervisor::is_supervisor() )
return;
local res = Supervisor::create(node);
local topic = SupervisorControl::topic_prefix + fmt("/create_response/%s", reqid);
Broker::publish(topic, SupervisorControl::create_response, reqid, res);
}
event SupervisorControl::destroy_request(reqid: string, node: string)
{
if ( ! Supervisor::is_supervisor() )
return;
local res = Supervisor::destroy(node);
local topic = SupervisorControl::topic_prefix + fmt("/destroy_response/%s", reqid);
Broker::publish(topic, SupervisorControl::destroy_response, reqid, res);
}
event SupervisorControl::restart_request(reqid: string, node: string)
{
if ( ! Supervisor::is_supervisor() )
return;
local res = Supervisor::restart(node);
local topic = SupervisorControl::topic_prefix + fmt("/restart_response/%s", reqid);
Broker::publish(topic, SupervisorControl::restart_response, reqid, res);
}

9
scripts/base/init-bare.zeek
View file

@ -1835,6 +1835,8 @@ type gtp_delete_pdp_ctx_response_elements: record {
@load base/bif/reporter.bif
@load base/bif/strings.bif
@load base/bif/option.bif
@load base/frameworks/supervisor/api
@load base/bif/supervisor.bif
global done_with_network = F;
event net_done(t: time) { done_with_network = T; }
@ -5228,10 +5230,3 @@ const global_hash_seed: string = "" &redef;
## files. The larger the value, the more confidence in UID uniqueness.
## The maximum is currently 128 bits.
const bits_per_uid: count = 96 &redef;
## Whether usage of the old communication system is considered an error or
## not. The default Zeek configuration no longer works with the non-Broker
## communication system unless you have manually taken action to initialize
## and set up the old comm. system. Deprecation warnings are still emitted
## when setting this flag, but they will not result in a fatal error.
const old_comm_usage_is_ok: bool = F &redef;

1
scripts/base/init-frameworks-and-bifs.zeek
View file

@ -5,6 +5,7 @@
# the separate file).
@load base/frameworks/logging
@load base/frameworks/broker
@load base/frameworks/supervisor
@load base/frameworks/input
@load base/frameworks/analyzer
@load base/frameworks/files

2
scripts/base/utils/json.zeek
View file

@ -1,2 +0,0 @@
## This file is deprecated in favor of to_json in zeek.bif
@deprecated="Remove in 3.1. to_json is now always available as a built-in function."

3
scripts/base/utils/urls.zeek
View file

@ -119,8 +119,11 @@ function decompose_uri(uri: string): URI
# Parse location and port.
parts = split_string1(s, /:/);
u$netlocation = parts[0];
if ( parts[1] != "" )
{
u$portnum = to_count(parts[1]);
}
}
else
{
u$netlocation = s;

2
scripts/policy/files/x509/log-ocsp.zeek
View file

@ -39,7 +39,7 @@ export {
global log_ocsp: event(rec: Info);
}
event zeek_init()
event zeek_init() &priority=5
{
Log::create_stream(LOG, [$columns=Info, $ev=log_ocsp, $path="ocsp"]);
Files::register_for_mime_type(Files::ANALYZER_OCSP_REPLY, "application/ocsp-response");

2
scripts/policy/protocols/conn/known-hosts.zeek
View file

@ -145,7 +145,7 @@ event Known::host_found(info: HostsInfo)
event known_host_add(info);
}
event zeek_init()
event zeek_init() &priority=5
{
Log::create_stream(Known::HOSTS_LOG, [$columns=HostsInfo, $ev=log_known_hosts, $path="known_hosts"]);
}

9
src/CMakeLists.txt
View file

@ -107,6 +107,11 @@ set(BIF_SRCS
strings.bif
reporter.bif
option.bif
# Note: the supervisor BIF file is treated like other top-level BIFs
# instead of contained in its own subdirectory CMake logic because
# subdirectory BIFs are treated differently and don't support being called
# *during* parsing (e.g. within an @if directive).
supervisor/supervisor.bif
)
foreach (bift ${BIF_SRCS})
@ -203,6 +208,7 @@ set(MAIN_SRCS
net_util.cc
util.cc
module_util.cc
zeek-affinity.cc
Anon.cc
Attr.cc
Base64.cc
@ -245,6 +251,7 @@ set(MAIN_SRCS
NetVar.cc
Obj.cc
OpaqueVal.cc
Options.cc
PacketFilter.cc
Pipe.cc
PolicyFile.cc
@ -284,6 +291,8 @@ set(MAIN_SRCS
modp_numtoa.c
siphash24.c
supervisor/Supervisor.cc
threading/BasicThread.cc
threading/Formatter.cc
threading/Manager.cc

2
src/DNS_Mgr.cc
View file

@ -467,7 +467,7 @@ void DNS_Mgr::InitPostScript()
const char* cache_dir = dir ? dir : ".";
cache_name = new char[strlen(cache_dir) + 64];
sprintf(cache_name, "%s/%s", cache_dir, ".bro-dns-cache");
sprintf(cache_name, "%s/%s", cache_dir, ".zeek-dns-cache");
LoadCache(fopen(cache_name, "r"));
}

3
src/DebugLogger.cc
View file

@ -19,7 +19,8 @@ DebugLogger::Stream DebugLogger::streams[NUM_DBGS] = {
{ "threading", 0, false }, { "file_analysis", 0, false },
{ "plugins", 0, false }, { "zeekygen", 0, false },
{ "pktio", 0, false }, { "broker", 0, false },
{ "scripts", 0, false}
{ "scripts", 0, false},
{ "supervisor", 0, false}
};
DebugLogger::DebugLogger()

1
src/DebugLogger.h
View file

@ -29,6 +29,7 @@ enum DebugStream {
DBG_PKTIO, // Packet sources and dumpers.
DBG_BROKER, // Broker communication
DBG_SCRIPTS, // Script initialization
DBG_SUPERVISOR, // Process supervisor
NUM_DBGS // Has to be last
};

26
src/Flare.cc
View file

@ -13,14 +13,24 @@ Flare::Flare()
{
}
static void bad_pipe_op(const char* which)
static void bad_pipe_op(const char* which, bool signal_safe)
{
if ( signal_safe )
abort();
char buf[256];
bro_strerror_r(errno, buf, sizeof(buf));
if ( reporter )
reporter->FatalErrorWithCore("unexpected pipe %s failure: %s", which, buf);
else
{
fprintf(stderr, "unexpected pipe %s failure: %s", which, buf);
abort();
}
}
void Flare::Fire()
void Flare::Fire(bool signal_safe)
{
char tmp = 0;
@ -42,15 +52,16 @@ void Flare::Fire()
// Interrupted: try again.
continue;
bad_pipe_op("write");
bad_pipe_op("write", signal_safe);
}
// No error, but didn't write a byte: try again.
}
}
void Flare::Extinguish()
int Flare::Extinguish(bool signal_safe)
{
int rval = 0;
char tmp[256];
for ( ; ; )
@ -58,8 +69,11 @@ void Flare::Extinguish()
int n = read(pipe.ReadFD(), &tmp, sizeof(tmp));
if ( n >= 0 )
{
rval += n;
// Pipe may not be empty yet: try again.
continue;
}
if ( errno == EAGAIN )
// Success: pipe is now empty.
@ -69,6 +83,8 @@ void Flare::Extinguish()
// Interrupted: try again.
continue;
bad_pipe_op("read");
bad_pipe_op("read", signal_safe);
}
return rval;
}

10
src/Flare.h
View file

@ -26,13 +26,19 @@ public:
/**
* Put the object in the "ready" state.
* @param signal_safe whether to skip error-reporting functionality that
* is not async-signal-safe (errors still abort the process regardless)
*/
void Fire();
void Fire(bool signal_safe = false);
/**
* Take the object out of the "ready" state.
* @param signal_safe whether to skip error-reporting functionality that
* is not async-signal-safe (errors still abort the process regardless)
* @return number of bytes read from the pipe, corresponds to the number
* of times Fire() was called.
*/
void Extinguish();
int Extinguish(bool signal_safe = false);
private:
Pipe pipe;

3
src/Func.cc
View file

@ -743,12 +743,14 @@ void builtin_error(const char* msg, BroObj* arg)
#include "reporter.bif.func_h"
#include "strings.bif.func_h"
#include "option.bif.func_h"
#include "supervisor.bif.func_h"
#include "zeek.bif.func_def"
#include "stats.bif.func_def"
#include "reporter.bif.func_def"
#include "strings.bif.func_def"
#include "option.bif.func_def"
#include "supervisor.bif.func_def"
#include "__all__.bif.cc" // Autogenerated for compiling in the bif_target() code.
#include "__all__.bif.register.cc" // Autogenerated for compiling in the bif_target() code.
@ -776,6 +778,7 @@ void init_builtin_funcs()
#include "reporter.bif.func_init"
#include "strings.bif.func_init"
#include "option.bif.func_init"
#include "supervisor.bif.func_init"
did_builtin_init = true;
}

22
src/List.h
View file

@ -154,12 +154,6 @@ public:
return max_entries;
}
ZEEK_DEPRECATED("Remove in v3.1: Use std::sort instead")
void sort(list_cmp_func cmp_func)
{
qsort(entries, num_entries, sizeof(T), cmp_func);
}
int MemoryAllocation() const
{ return padded_sizeof(*this) + pad_size(max_entries * sizeof(T)); }
@ -189,12 +183,6 @@ public:
T& front() { return entries[0]; }
T& back() { return entries[num_entries-1]; }
ZEEK_DEPRECATED("Remove in v3.1: Use push_front instead")
void insert(const T& a) // add at head of list
{
push_front(a);
}
// The append method is maintained for historical/compatibility reasons.
// (It's commonly used in the event generation API)
void append(const T& a) // add to end of list
@ -229,16 +217,6 @@ public:
return old_ent;
}
ZEEK_DEPRECATED("Remove in v3.1: Use back()/pop_back() instead")
T get() // return and remove ent at end of list
{
assert(num_entries > 0);
return entries[--num_entries];
}
ZEEK_DEPRECATED("Remove in v3.1: Use back() instead")
T& last() { return back(); }
// Return 0 if ent is not in the list, ent otherwise.
bool is_member(const T& a) const
{

35
src/Net.cc
View file

@ -145,40 +145,40 @@ void net_update_time(double new_network_time)
PLUGIN_HOOK_VOID(HOOK_UPDATE_NETWORK_TIME, HookUpdateNetworkTime(new_network_time));
}
void net_init(name_list& interfaces, name_list& readfiles,
const char* writefile, int do_watchdog)
void net_init(const std::vector<std::string>& interfaces,
const std::vector<std::string>& pcap_input_files,
const std::optional<std::string>& pcap_output_file,
bool do_watchdog)
{
if ( readfiles.length() > 0 )
if ( ! pcap_input_files.empty() )
{
reading_live = pseudo_realtime > 0.0;
reading_traces = 1;
for ( int i = 0; i < readfiles.length(); ++i )
for ( const auto& pif : pcap_input_files )
{
iosource::PktSrc* ps = iosource_mgr->OpenPktSrc(readfiles[i], false);
iosource::PktSrc* ps = iosource_mgr->OpenPktSrc(pif, false);
assert(ps);
if ( ! ps->IsOpen() )
reporter->FatalError("problem with trace file %s (%s)",
readfiles[i],
ps->ErrorMsg());
pif.data(), ps->ErrorMsg());
}
}
else if ( interfaces.length() > 0 )
else if ( ! interfaces.empty() )
{
reading_live = 1;
reading_traces = 0;
for ( int i = 0; i < interfaces.length(); ++i )
for ( const auto& iface : interfaces )
{
iosource::PktSrc* ps = iosource_mgr->OpenPktSrc(interfaces[i], true);
iosource::PktSrc* ps = iosource_mgr->OpenPktSrc(iface, true);
assert(ps);
if ( ! ps->IsOpen() )
reporter->FatalError("problem with interface %s (%s)",
interfaces[i],
ps->ErrorMsg());
iface.data(), ps->ErrorMsg());
}
}
@ -189,8 +189,9 @@ void net_init(name_list& interfaces, name_list& readfiles,
// a timer.
reading_traces = reading_live = 0;
if ( writefile )
if ( pcap_output_file )
{
const char* writefile = pcap_output_file->data();
pkt_dumper = iosource_mgr->OpenPktDumper(writefile, false);
assert(pkt_dumper);
@ -361,13 +362,11 @@ void net_run()
current_dispatched = 0;
current_iosrc = 0;
// Should we put the signal handling into an IOSource?
extern void termination_signal();
if ( signal_val == SIGTERM || signal_val == SIGINT )
// We received a signal while processing the
// current packet and its related events.
termination_signal();
// Should we put the signal handling into an IOSource?
zeek_terminate_loop("received termination signal");
if ( ! reading_traces )
// Check whether we have timers scheduled for
@ -456,8 +455,6 @@ void net_delete()
int _processing_suspended = 0;
static double suspend_start = 0;
void net_suspend_processing()
{
if ( _processing_suspended == 0 )

14
src/Net.h
View file

@ -2,6 +2,10 @@
#pragma once
#include <vector>
#include <string>
#include <optional>
#include "net_util.h"
#include "util.h"
#include "List.h"
@ -10,8 +14,10 @@
#include "iosource/PktSrc.h"
#include "iosource/PktDumper.h"
extern void net_init(name_list& interfaces, name_list& readfiles,
const char* writefile, int do_watchdog);
extern void net_init(const std::vector<std::string>& interfaces,
const std::vector<std::string>& pcap_input_files,
const std::optional<std::string>& pcap_output_file,
bool do_watchdog);
extern void net_run();
extern void net_get_final_stats();
extern void net_finish(int drain_events);
@ -20,7 +26,7 @@ extern void net_update_time(double new_network_time);
extern void net_packet_dispatch(double t, const Packet* pkt,
iosource::PktSrc* src_ps);
extern void expire_timers(iosource::PktSrc* src_ps = 0);
extern void termination_signal();
extern void zeek_terminate_loop(const char* reason);
// Functions to temporarily suspend processing of live input (network packets
// and remote events/state). Turning this is on is sure to lead to data loss!
@ -76,8 +82,6 @@ extern iosource::IOSource* current_iosrc;
extern iosource::PktDumper* pkt_dumper; // where to save packets
extern char* writefile;
// Script file we have already scanned (or are in the process of scanning).
// They are identified by inode number.
struct ScannedFile {

2
src/NetVar.cc
View file

@ -196,6 +196,7 @@ bro_uint_t bits_per_uid;
#include "types.bif.netvar_def"
#include "event.bif.netvar_def"
#include "reporter.bif.netvar_def"
#include "supervisor.bif.netvar_def"
void init_event_handlers()
{
@ -240,6 +241,7 @@ void init_net_var()
#include "const.bif.netvar_init"
#include "types.bif.netvar_init"
#include "reporter.bif.netvar_init"
#include "supervisor.bif.netvar_init"
conn_id = internal_type("conn_id")->AsRecordType();
endpoint = internal_type("endpoint")->AsRecordType();

1
src/NetVar.h
View file

@ -203,3 +203,4 @@ extern void init_net_var();
#include "types.bif.netvar_h"
#include "event.bif.netvar_h"
#include "reporter.bif.netvar_h"
#include "supervisor.bif.netvar_h"

460
src/Options.cc Normal file
View file

@ -0,0 +1,460 @@
// See the file "COPYING" in the main distribution directory for copyright.
#include <unistd.h>
#include "zeek-config.h"
#ifdef HAVE_GETOPT_H
#include <getopt.h>
#endif
#include "bsd-getopt-long.h"
#include "logging/writers/ascii/Ascii.h"
#include "Options.h"
void zeek::Options::filter_supervisor_options()
{
pcap_filter = {};
interfaces = {};
pcap_files = {};
signature_files = {};
pcap_output_file = {};
}
void zeek::Options::filter_supervised_node_options()
{
auto og = *this;
*this = {};
debug_log_streams = og.debug_log_streams;
debug_script_tracing_file = og.debug_script_tracing_file;
script_code_to_exec = og.script_code_to_exec;
script_prefixes = og.script_prefixes;
signature_re_level = og.signature_re_level;
ignore_checksums = og.ignore_checksums;
use_watchdog = og.use_watchdog;
pseudo_realtime = og.pseudo_realtime;
dns_mode = og.dns_mode;
bare_mode = og.bare_mode;
perftools_check_leaks = og.perftools_check_leaks;
perftools_profile = og.perftools_profile;
pcap_filter = og.pcap_filter;
signature_files = og.signature_files;
// TODO: These are likely to be handled in a node-specific or
// use-case-specific way. e.g. interfaces is already handled for the
// "cluster" use-case, but don't have supervised-pcap-reading
// functionality yet.
/* interfaces = og.interfaces; */
/* pcap_files = og.pcap_files; */
pcap_output_file = og.pcap_output_file;
random_seed_input_file = og.random_seed_input_file;
random_seed_output_file = og.random_seed_output_file;
process_status_file = og.process_status_file;
plugins_to_load = og.plugins_to_load;
scripts_to_load = og.scripts_to_load;
script_options_to_set = og.script_options_to_set;
}
bool zeek::fake_dns()
{
return zeekenv("ZEEK_DNS_FAKE");
}
extern const char* zeek_version();
void zeek::usage(const char* prog, int code)
{
fprintf(stderr, "zeek version %s\n", zeek_version());
fprintf(stderr, "usage: %s [options] [file ...]\n", prog);
fprintf(stderr, "usage: %s --test [doctest-options] -- [options] [file ...]\n", prog);
fprintf(stderr, " <file> | Zeek script file, or read stdin\n");
fprintf(stderr, " -a|--parse-only | exit immediately after parsing scripts\n");
fprintf(stderr, " -b|--bare-mode | don't load scripts from the base/ directory\n");
fprintf(stderr, " -d|--debug-script | activate Zeek script debugging\n");
fprintf(stderr, " -e|--exec <zeek code> | augment loaded scripts by given code\n");
fprintf(stderr, " -f|--filter <filter> | tcpdump filter\n");
fprintf(stderr, " -h|--help | command line help\n");
fprintf(stderr, " -i|--iface <interface> | read from given interface\n");
fprintf(stderr, " -p|--prefix <prefix> | add given prefix to Zeek script file resolution\n");
fprintf(stderr, " -r|--readfile <readfile> | read from given tcpdump file\n");
fprintf(stderr, " -s|--rulefile <rulefile> | read rules from given file\n");
fprintf(stderr, " -t|--tracefile <tracefile> | activate execution tracing\n");
fprintf(stderr, " -v|--version | print version and exit\n");
fprintf(stderr, " -w|--writefile <writefile> | write to given tcpdump file\n");
#ifdef DEBUG
fprintf(stderr, " -B|--debug <dbgstreams> | Enable debugging output for selected streams ('-B help' for help)\n");
#endif
fprintf(stderr, " -C|--no-checksums | ignore checksums\n");
fprintf(stderr, " -F|--force-dns | force DNS\n");
fprintf(stderr, " -G|--load-seeds <file> | load seeds from given file\n");
fprintf(stderr, " -H|--save-seeds <file> | save seeds to given file\n");
fprintf(stderr, " -I|--print-id <ID name> | print out given ID\n");
fprintf(stderr, " -N|--print-plugins | print available plugins and exit (-NN for verbose)\n");
fprintf(stderr, " -P|--prime-dns | prime DNS\n");
fprintf(stderr, " -Q|--time | print execution time summary to stderr\n");
fprintf(stderr, " -S|--debug-rules | enable rule debugging\n");
fprintf(stderr, " -T|--re-level <level> | set 'RE_level' for rules\n");
fprintf(stderr, " -U|--status-file <file> | Record process status in file\n");
fprintf(stderr, " -W|--watchdog | activate watchdog timer\n");
fprintf(stderr, " -X|--zeekygen <cfgfile> | generate documentation based on config file\n");
#ifdef USE_PERFTOOLS_DEBUG
fprintf(stderr, " -m|--mem-leaks | show leaks [perftools]\n");
fprintf(stderr, " -M|--mem-profile | record heap [perftools]\n");
#endif
fprintf(stderr, " --pseudo-realtime[=<speedup>] | enable pseudo-realtime for performance evaluation (default 1)\n");
fprintf(stderr, " -j|--jobs | enable supervisor mode\n");
#ifdef USE_IDMEF
fprintf(stderr, " -n|--idmef-dtd <idmef-msg.dtd> | specify path to IDMEF DTD file\n");
#endif
fprintf(stderr, " --test | run unit tests ('--test -h' for help, only when compiling with ENABLE_ZEEK_UNIT_TESTS)\n");
fprintf(stderr, " $ZEEKPATH | file search path (%s)\n", bro_path().c_str());
fprintf(stderr, " $ZEEK_PLUGIN_PATH | plugin search path (%s)\n", bro_plugin_path());
fprintf(stderr, " $ZEEK_PLUGIN_ACTIVATE | plugins to always activate (%s)\n", bro_plugin_activate());
fprintf(stderr, " $ZEEK_PREFIXES | prefix list (%s)\n", bro_prefixes().c_str());
fprintf(stderr, " $ZEEK_DNS_FAKE | disable DNS lookups (%s)\n", zeek::fake_dns() ? "on" : "off");
fprintf(stderr, " $ZEEK_SEED_FILE | file to load seeds from (not set)\n");
fprintf(stderr, " $ZEEK_LOG_SUFFIX | ASCII log file extension (.%s)\n", logging::writer::Ascii::LogExt().c_str());
fprintf(stderr, " $ZEEK_PROFILER_FILE | Output file for script execution statistics (not set)\n");
fprintf(stderr, " $ZEEK_DISABLE_ZEEKYGEN | Disable Zeekygen documentation support (%s)\n", zeekenv("ZEEK_DISABLE_ZEEKYGEN") ? "set" : "not set");
fprintf(stderr, " $ZEEK_DNS_RESOLVER | IPv4/IPv6 address of DNS resolver to use (%s)\n", zeekenv("ZEEK_DNS_RESOLVER") ? zeekenv("ZEEK_DNS_RESOLVER") : "not set, will use first IPv4 address from /etc/resolv.conf");
fprintf(stderr, " $ZEEK_DEBUG_LOG_STDERR | Use stderr for debug logs generated via the -B flag");
fprintf(stderr, "\n");
exit(code);
}
zeek::Options zeek::parse_cmdline(int argc, char** argv)
{
zeek::Options rval;
// When running unit tests, the first argument on the command line must be
// --test, followed by doctest options. Optionally, users can use "--" as
// separator to pass Zeek options afterwards:
//
// zeek --test [doctest-options] -- [zeek-options]
// Just locally filtering out the args for Zeek usage from doctest args.
std::vector<std::string> zeek_args;
if ( argc > 1 && strcmp(argv[1], "--test") == 0 )
{
#ifdef DOCTEST_CONFIG_DISABLE
fprintf(stderr, "ERROR: C++ unit tests are disabled for this build.\n"
" Please re-compile with ENABLE_ZEEK_UNIT_TESTS "
"to run the C++ unit tests.\n");
usage(argv[0], 1);
#endif
auto is_separator = [](const char* cstr)
{
return strcmp(cstr, "--") == 0;
};
auto first = argv;
auto last = argv + argc;
auto separator = std::find_if(first, last, is_separator);
zeek_args.emplace_back(argv[0]);
if ( separator != last )
{
auto first_zeek_arg = std::next(separator);
for ( auto i = first_zeek_arg; i != last; ++i )
zeek_args.emplace_back(*i);
}
rval.run_unit_tests = true;
for ( auto i = 0; i < std::distance(first, separator); ++i )
rval.doctest_args.emplace_back(argv[i]);
}
else
{
for ( auto i = 0; i < argc; ++i )
zeek_args.emplace_back(argv[i]);
}
constexpr struct option long_opts[] = {
{"parse-only", no_argument, 0, 'a'},
{"bare-mode", no_argument, 0, 'b'},
{"debug-script", no_argument, 0, 'd'},
{"exec", required_argument, 0, 'e'},
{"filter", required_argument, 0, 'f'},
{"help", no_argument, 0, 'h'},
{"iface", required_argument, 0, 'i'},
{"zeekygen", required_argument, 0, 'X'},
{"prefix", required_argument, 0, 'p'},
{"readfile", required_argument, 0, 'r'},
{"rulefile", required_argument, 0, 's'},
{"tracefile", required_argument, 0, 't'},
{"writefile", required_argument, 0, 'w'},
{"version", no_argument, 0, 'v'},
{"no-checksums", no_argument, 0, 'C'},
{"force-dns", no_argument, 0, 'F'},
{"load-seeds", required_argument, 0, 'G'},
{"save-seeds", required_argument, 0, 'H'},
{"print-plugins", no_argument, 0, 'N'},
{"prime-dns", no_argument, 0, 'P'},
{"time", no_argument, 0, 'Q'},
{"debug-rules", no_argument, 0, 'S'},
{"re-level", required_argument, 0, 'T'},
{"watchdog", no_argument, 0, 'W'},
{"print-id", required_argument, 0, 'I'},
{"status-file", required_argument, 0, 'U'},
#ifdef DEBUG
{"debug", required_argument, 0, 'B'},
#endif
#ifdef USE_IDMEF
{"idmef-dtd", required_argument, 0, 'n'},
#endif
#ifdef USE_PERFTOOLS_DEBUG
{"mem-leaks", no_argument, 0, 'm'},
{"mem-profile", no_argument, 0, 'M'},
#endif
{"pseudo-realtime", optional_argument, 0, 'E'},
{"jobs", optional_argument, 0, 'j'},
{"test", no_argument, 0, '#'},
{0, 0, 0, 0},
};
char opts[256];
safe_strncpy(opts, "B:e:f:G:H:I:i:j::n:p:r:s:T:t:U:w:X:CFNPQSWabdhv",
sizeof(opts));
#ifdef USE_PERFTOOLS_DEBUG
strncat(opts, "mM", 2);
#endif
int op;
int long_optsind;
opterr = 0;
// getopt may permute the array, so need yet another array
auto zargs = std::make_unique<char*[]>(zeek_args.size());
for ( auto i = 0u; i < zeek_args.size(); ++i )
zargs[i] = zeek_args[i].data();
while ( (op = getopt_long(zeek_args.size(), zargs.get(), opts, long_opts, &long_optsind)) != EOF )
switch ( op ) {
case 'a':
rval.parse_only = true;
break;
case 'b':
rval.bare_mode = true;
break;
case 'd':
rval.debug_scripts = true;
break;
case 'e':
rval.script_code_to_exec = optarg;
break;
case 'f':
rval.pcap_filter = optarg;
break;
case 'h':
rval.print_usage = true;
break;
case 'i':
if ( ! rval.pcap_files.empty() )
{
fprintf(stderr, "Using -i is not allowed when reading pcap files");
exit(1);
}
rval.interfaces.emplace_back(optarg);
break;
case 'j':
rval.supervisor_mode = true;
if ( optarg )
{
// TODO: for supervised offline pcap reading, the argument is
// expected to be number of workers like "-j 4" or possibly a
// list of worker/proxy/logger counts like "-j 4,2,1"
}
break;
case 'p':
rval.script_prefixes.emplace_back(optarg);
break;
case 'r':
if ( ! rval.interfaces.empty() )
{
fprintf(stderr, "Using -r is not allowed when reading a live interface");
exit(1);
}
rval.pcap_files.emplace_back(optarg);
break;
case 's':
rval.signature_files.emplace_back(optarg);
break;
case 't':
rval.debug_script_tracing_file = optarg;
break;
case 'v':
rval.print_version = true;
break;
case 'w':
rval.pcap_output_file = optarg;
break;
case 'B':
rval.debug_log_streams = optarg;
break;
case 'C':
rval.ignore_checksums = true;
break;
case 'E':
rval.pseudo_realtime = 1.0;
if ( optarg )
rval.pseudo_realtime = atof(optarg);
break;
case 'F':
if ( rval.dns_mode != DNS_DEFAULT )
usage(zargs[0], 1);
rval.dns_mode = DNS_FORCE;
break;
case 'G':
rval.random_seed_input_file = optarg;
break;
case 'H':
rval.random_seed_output_file = optarg;
break;
case 'I':
rval.identifier_to_print = optarg;
break;
case 'N':
++rval.print_plugins;
break;
case 'P':
if ( rval.dns_mode != DNS_DEFAULT )
usage(zargs[0], 1);
rval.dns_mode = DNS_PRIME;
break;
case 'Q':
rval.print_execution_time = true;
break;
case 'S':
rval.print_signature_debug_info = true;
break;
case 'T':
rval.signature_re_level = atoi(optarg);
break;
case 'U':
rval.process_status_file = optarg;
break;
case 'W':
rval.use_watchdog = true;
break;
case 'X':
rval.zeekygen_config_file = optarg;
break;
#ifdef USE_PERFTOOLS_DEBUG
case 'm':
rval.perftools_check_leaks = 1;
break;
case 'M':
rval.perftools_profile = 1;
break;
#endif
#ifdef USE_IDMEF
case 'n':
rval.libidmef_dtd_path = optarg;
break;
#endif
case '#':
fprintf(stderr, "ERROR: --test only allowed as first argument.\n");
usage(zargs[0], 1);
break;
case 0:
// This happens for long options that don't have
// a short-option equivalent.
break;
case '?':
default:
usage(zargs[0], 1);
break;
}
// Process remaining arguments. X=Y arguments indicate script
// variable/parameter assignments. X::Y arguments indicate plugins to
// activate/query. The remainder are treated as scripts to load.
while ( optind < static_cast<int>(zeek_args.size()) )
{
if ( strchr(zargs[optind], '=') )
rval.script_options_to_set.emplace_back(zargs[optind++]);
else if ( strstr(zargs[optind], "::") )
rval.plugins_to_load.emplace(zargs[optind++]);
else
rval.scripts_to_load.emplace_back(zargs[optind++]);
}
auto canonify_script_path = [](std::string* path)
{
if ( path->empty() )
return;
*path = normalize_path(*path);
if ( (*path)[0] == '/' || (*path)[0] == '~' )
// Absolute path
return;
if ( (*path)[0] != '.' )
{
// Look up file in ZEEKPATH
auto res = find_script_file(*path, bro_path());
if ( res.empty() )
{
fprintf(stderr, "failed to locate script: %s\n", path->data());
exit(1);
}
*path = res;
if ( (*path)[0] == '/' || (*path)[0] == '~' )
// Now an absolute path
return;
}
// Need to translate relative path to absolute.
char cwd[PATH_MAX];
if ( ! getcwd(cwd, sizeof(cwd)) )
{
fprintf(stderr, "failed to get current directory: %s\n",
strerror(errno));
exit(1);
}
*path = std::string(cwd) + "/" + *path;
};
if ( rval.supervisor_mode )
{
// Translate any relative paths supplied to supervisor into absolute
// paths for use by supervised nodes since they have the option to
// operate out of a different working directory.
for ( auto& s : rval.scripts_to_load )
canonify_script_path(&s);
}
return rval;
}

97
src/Options.h Normal file
View file

@ -0,0 +1,97 @@
// See the file "COPYING" in the main distribution directory for copyright.
#pragma once
#include <optional>
#include <string>
#include <vector>
#include "DNS_Mgr.h"
namespace zeek {
/**
* Options that define general Zeek processing behavior, usually determined
* from command-line arguments.
*/
struct Options {
/**
* Unset options that aren't meant to be used by the supervisor, but may
* make sense for supervised nodes to inherit (as opposed to flagging
* as an error an exiting outright if used in supervisor-mode).
*/
void filter_supervisor_options();
/**
* Inherit certain options set in the original supervisor parent process
* and discard the rest.
*/
void filter_supervised_node_options();
bool print_version = false;
bool print_usage = false;
bool print_execution_time = false;
bool print_signature_debug_info = false;
int print_plugins = 0;
std::optional<std::string> debug_log_streams;
std::optional<std::string> debug_script_tracing_file;
std::optional<std::string> identifier_to_print;
std::optional<std::string> script_code_to_exec;
std::vector<std::string> script_prefixes = { "" }; // "" = "no prefix"
int signature_re_level = 4;
bool ignore_checksums = false;
bool use_watchdog = false;
double pseudo_realtime = 0;
DNS_MgrMode dns_mode = DNS_DEFAULT;
bool supervisor_mode = false;
bool parse_only = false;
bool bare_mode = false;
bool debug_scripts = false;
bool perftools_check_leaks = false;
bool perftools_profile = false;
bool run_unit_tests = false;
std::vector<std::string> doctest_args;
std::optional<std::string> pcap_filter;
std::vector<std::string> interfaces;
std::vector<std::string> pcap_files;
std::vector<std::string> signature_files;
std::optional<std::string> pcap_output_file;
std::optional<std::string> random_seed_input_file;
std::optional<std::string> random_seed_output_file;
std::optional<std::string> process_status_file;
std::optional<std::string> zeekygen_config_file;
std::string libidmef_dtd_file = "idmef-message.dtd";
std::set<std::string> plugins_to_load;
std::vector<std::string> scripts_to_load;
std::vector<std::string> script_options_to_set;
};
/**
* Parse Zeek command-line arguments.
* @param argc argument count (same semantics as arguments to main())
* @param argv argument strings (same semantics as arguments to main())
* @return the parsed command-line options
*/
zeek::Options parse_cmdline(int argc, char** argv);
/**
* Print command-line Zeek usage information and exit.
* @param prog the name/path of the Zeek command-line invocation
* @code the exit code to use
*/
void usage(const char* prog, int code = 1);
/**
* @return true if zeek is running a "fake" DNS resolver, else false.
*/
bool fake_dns();
} // namespace zeek

105
src/Pipe.cc
View file

@ -13,24 +13,68 @@ static void pipe_fail(int eno)
{
char tmp[256];
bro_strerror_r(eno, tmp, sizeof(tmp));
if ( reporter )
reporter->FatalError("Pipe failure: %s", tmp);
else
fprintf(stderr, "Pipe failure: %s", tmp);
}
static void set_flags(int fd, int flags)
static int set_flags(int fd, int flags)
{
auto rval = fcntl(fd, F_GETFD);
if ( rval == -1 )
pipe_fail(errno);
if ( flags )
if ( fcntl(fd, F_SETFD, fcntl(fd, F_GETFD) | flags) == -1 )
{
rval |= flags;
if ( fcntl(fd, F_SETFD, rval) == -1 )
pipe_fail(errno);
}
static void set_status_flags(int fd, int flags)
return rval;
}
static int unset_flags(int fd, int flags)
{
auto rval = fcntl(fd, F_GETFD);
if ( rval == -1 )
pipe_fail(errno);
if ( flags )
if ( fcntl(fd, F_SETFL, fcntl(fd, F_GETFL) | flags) == -1 )
{
rval &= ~flags;
if ( fcntl(fd, F_SETFD, rval) == -1 )
pipe_fail(errno);
}
static int dup_or_fail(int fd, int flags)
return rval;
}
static int set_status_flags(int fd, int flags)
{
auto rval = fcntl(fd, F_GETFL);
if ( rval == -1 )
pipe_fail(errno);
if ( flags )
{
rval |= flags;
if ( fcntl(fd, F_SETFL, rval) == -1 )
pipe_fail(errno);
}
return rval;
}
static int dup_or_fail(int fd, int flags, int status_flags)
{
int rval = dup(fd);
@ -38,22 +82,41 @@ static int dup_or_fail(int fd, int flags)
pipe_fail(errno);
set_flags(fd, flags);
set_status_flags(fd, status_flags);
return rval;
}
Pipe::Pipe(int flags0, int flags1, int status_flags0, int status_flags1)
Pipe::Pipe(int flags0, int flags1, int status_flags0, int status_flags1,
int* arg_fds)
{
if ( arg_fds )
{
fds[0] = arg_fds[0];
fds[1] = arg_fds[1];
}
else
{
// pipe2 can set flags atomically, but not yet available everywhere.
if ( ::pipe(fds) )
pipe_fail(errno);
}
flags[0] = flags0;
flags[1] = flags1;
flags[0] = set_flags(fds[0], flags0);
flags[1] = set_flags(fds[1], flags1);
status_flags[0] = set_status_flags(fds[0], status_flags0);
status_flags[1] = set_status_flags(fds[1], status_flags1);
}
set_flags(fds[0], flags[0]);
set_flags(fds[1], flags[1]);
set_status_flags(fds[0], status_flags0);
set_status_flags(fds[1], status_flags1);
void Pipe::SetFlags(int arg_flags)
{
flags[0] = set_flags(fds[0], arg_flags);
flags[1] = set_flags(fds[1], arg_flags);
}
void Pipe::UnsetFlags(int arg_flags)
{
flags[0] = unset_flags(fds[0], arg_flags);
flags[1] = unset_flags(fds[1], arg_flags);
}
Pipe::~Pipe()
@ -64,10 +127,12 @@ Pipe::~Pipe()
Pipe::Pipe(const Pipe& other)
{
fds[0] = dup_or_fail(other.fds[0], other.flags[0]);
fds[1] = dup_or_fail(other.fds[1], other.flags[1]);
fds[0] = dup_or_fail(other.fds[0], other.flags[0], other.status_flags[0]);
fds[1] = dup_or_fail(other.fds[1], other.flags[1], other.status_flags[1]);
flags[0] = other.flags[0];
flags[1] = other.flags[1];
status_flags[0] = other.status_flags[0];
status_flags[1] = other.status_flags[1];
}
Pipe& Pipe::operator=(const Pipe& other)
@ -77,9 +142,17 @@ Pipe& Pipe::operator=(const Pipe& other)
close(fds[0]);
close(fds[1]);
fds[0] = dup_or_fail(other.fds[0], other.flags[0]);
fds[1] = dup_or_fail(other.fds[1], other.flags[1]);
fds[0] = dup_or_fail(other.fds[0], other.flags[0], other.status_flags[0]);
fds[1] = dup_or_fail(other.fds[1], other.flags[1], other.status_flags[1]);
flags[0] = other.flags[0];
flags[1] = other.flags[1];
status_flags[0] = other.status_flags[0];
status_flags[1] = other.status_flags[1];
return *this;
}
PipePair::PipePair(int flags, int status_flags, int* fds)
: pipes{Pipe(flags, flags, status_flags, status_flags, fds ? fds + 0 : nullptr),
Pipe(flags, flags, status_flags, status_flags, fds ? fds + 2 : nullptr)}
{
}

88
src/Pipe.h
View file

@ -13,9 +13,12 @@ public:
* @param flags1 file descriptor flags to set on write end of pipe.
* @param status_flags0 descriptor status flags to set on read end of pipe.
* @param status_flags1 descriptor status flags to set on write end of pipe.
* @param fds may be supplied to open an existing file descriptors rather
* than create ones from a new pipe. Should point to memory containing
* two consecutive file descriptors, the "read" one and then the "write" one.
*/
explicit Pipe(int flags0 = 0, int flags1 = 0, int status_flags0 = 0,
int status_flags1 = 0);
int status_flags1 = 0, int* fds = nullptr);
/**
* Close the pair of file descriptors owned by the object.
@ -45,9 +48,92 @@ public:
int WriteFD() const
{ return fds[1]; }
/**
* Sets the given file descriptor flags for both the read and write end
* of the pipe.
*/
void SetFlags(int flags);
/**
* Unsets the given file descriptor flags for both the read and write end
* of the pipe.
*/
void UnsetFlags(int flags);
private:
int fds[2];
int flags[2];
int status_flags[2];
};
/**
* A pair of pipes that can be used for bi-directinoal IPC.
*/
class PipePair {
public:
/**
* Create a pair of pipes
* @param flags file descriptor flags to set on pipes
* @status_flags descriptor status flags to set on pipes
* @fds may be supplied to open existing file descriptors rather
* than create ones from a new pair of pipes. Should point to memory
* containing four consecutive file descriptors, "read" end and "write" end
* of the first pipe followed by the "read" end and "write" end of the
* second pipe.
*/
PipePair(int flags, int status_flags, int* fds = nullptr);
/**
* @return the pipe used for receiving input
*/
Pipe& In()
{ return pipes[swapped]; }
/**
* @return the pipe used for sending output
*/
Pipe& Out()
{ return pipes[!swapped]; }
/**
* @return the pipe used for receiving input
*/
const Pipe& In() const
{ return pipes[swapped]; }
/**
* @return the pipe used for sending output
*/
const Pipe& Out() const
{ return pipes[!swapped]; }
/**
* @return a file descriptor that may used for receiving messages by
* polling/reading it.
*/
int InFD() const
{ return In().ReadFD(); }
/**
* @return a file descriptor that may be used for sending messages by
* writing to it.
*/
int OutFD() const
{ return Out().WriteFD(); }
/**
* Swaps the meaning of the pipes in the pair. E.g. call this after
* fork()'ing so that the child process uses the right pipe for
* reading/writing.
*/
void Swap()
{ swapped = ! swapped; }
private:
Pipe pipes[2];
bool swapped = false;
};
} // namespace bro

10
src/RuleMatcher.cc
View file

@ -231,7 +231,7 @@ void RuleMatcher::Delete(RuleHdrTest* node)
delete node;
}
bool RuleMatcher::ReadFiles(const name_list& files)
bool RuleMatcher::ReadFiles(const std::vector<std::string>& files)
{
#ifdef USE_PERFTOOLS_DEBUG
HeapLeakChecker::Disabler disabler;
@ -239,18 +239,18 @@ bool RuleMatcher::ReadFiles(const name_list& files)
parse_error = false;
for ( int i = 0; i < files.length(); ++i )
for ( const auto& f : files )
{
rules_in = open_file(find_file(files[i], bro_path(), ".sig"));
rules_in = open_file(find_file(f, bro_path(), ".sig"));
if ( ! rules_in )
{
reporter->Error("Can't open signature file %s", files[i]);
reporter->Error("Can't open signature file %s", f.data());
return false;
}
rules_line_number = 0;
current_rule_file = files[i];
current_rule_file = f.data();
rules_parse();
fclose(rules_in);
}

2
src/RuleMatcher.h
View file

@ -221,7 +221,7 @@ public:
~RuleMatcher();
// Parse the given files and built up data structures.
bool ReadFiles(const name_list& files);
bool ReadFiles(const std::vector<std::string>& files);
/**
* Inititialize a state object for matching file magic signatures.

2
src/Scope.cc
View file

@ -59,7 +59,7 @@ Scope::~Scope()
ID* Scope::GenerateTemporary(const char* name)
{
return new ID(copy_string(name), SCOPE_FUNCTION, false);
return new ID(name, SCOPE_FUNCTION, false);
}
id_list* Scope::GetInits()

1
src/Timer.cc
View file

@ -37,6 +37,7 @@ const char* TimerNames[] = {
"TCPConnectionPartialClose",
"TCPConnectionResetTimer",
"TriggerTimer",
"ParentProcessIDCheck",
"TimerMgrExpireTimer",
};

1
src/Timer.h
View file

@ -41,6 +41,7 @@ enum TimerType : uint8_t {
TIMER_TCP_PARTIAL_CLOSE,
TIMER_TCP_RESET,
TIMER_TRIGGER,
TIMER_PPID_CHECK,
TIMER_TIMERMGR_EXPIRE,
};
const int NUM_TIMER_TYPES = int(TIMER_TIMERMGR_EXPIRE) + 1;

2
src/Type.cc
View file

@ -19,7 +19,7 @@ BroType::TypeAliasMap BroType::type_aliases;
// Note: This function must be thread-safe.
const char* type_name(TypeTag t)
{
static const char* type_names[int(NUM_TYPES)] = {
static constexpr const char* type_names[int(NUM_TYPES)] = {
"void", // 0
"bool", // 1
"int", // 2

29
src/Val.cc
View file

@ -719,16 +719,6 @@ void IntervalVal::ValDescribe(ODesc* d) const
}
}
PortVal* PortManager::Get(uint32_t port_num) const
{
return val_mgr->GetPort(port_num);
}
PortVal* PortManager::Get(uint32_t port_num, TransportProto port_type) const
{
return val_mgr->GetPort(port_num, port_type);
}
uint32_t PortVal::Mask(uint32_t port_num, TransportProto port_type)
{
// Note, for ICMP one-way connections:
@ -760,25 +750,8 @@ uint32_t PortVal::Mask(uint32_t port_num, TransportProto port_type)
return port_num;
}
PortVal::PortVal(uint32_t p, TransportProto port_type) : Val(TYPE_PORT)
{
auto port_num = PortVal::Mask(p, port_type);
val.uint_val = static_cast<bro_uint_t>(port_num);
}
PortVal::PortVal(uint32_t p, bool unused) : Val(TYPE_PORT)
{
val.uint_val = static_cast<bro_uint_t>(p);
}
PortVal::PortVal(uint32_t p) : Val(TYPE_PORT)
{
if ( p >= 65536 * NUM_PORT_SPACES )
{
InternalWarning("bad port number");
p = 0;
}
val.uint_val = static_cast<bro_uint_t>(p);
}
@ -3305,7 +3278,7 @@ ValManager::ValManager()
auto port_type = (TransportProto)i;
for ( auto j = 0u; j < arr.size(); ++j )
arr[j] = new PortVal(PortVal::Mask(j, port_type), true);
arr[j] = new PortVal(PortVal::Mask(j, port_type));
}
}

84
src/Val.h
View file

@ -85,56 +85,6 @@ typedef union {
class Val : public BroObj {
public:
ZEEK_DEPRECATED("Remove in v3.1: use val_mgr->GetBool, GetFalse/GetTrue, GetInt, or GetCount instead")
Val(bool b, TypeTag t)
{
val.int_val = b;
type = base_type(t);
#ifdef DEBUG
bound_id = 0;
#endif
}
ZEEK_DEPRECATED("Remove in v3.1: use val_mgr->GetBool, GetFalse/GetTrue, GetInt, or GetCount instead")
Val(int32_t i, TypeTag t)
{
val.int_val = bro_int_t(i);
type = base_type(t);
#ifdef DEBUG
bound_id = 0;
#endif
}
ZEEK_DEPRECATED("Remove in v3.1: use val_mgr->GetBool, GetFalse/GetTrue, GetInt, or GetCount instead")
Val(uint32_t u, TypeTag t)
{
val.uint_val = bro_uint_t(u);
type = base_type(t);
#ifdef DEBUG
bound_id = 0;
#endif
}
ZEEK_DEPRECATED("Remove in v3.1: use val_mgr->GetBool, GetFalse/GetTrue, GetInt, or GetCount instead")
Val(int64_t i, TypeTag t)
{
val.int_val = i;
type = base_type(t);
#ifdef DEBUG
bound_id = 0;
#endif
}
ZEEK_DEPRECATED("Remove in v3.1: use val_mgr->GetBool, GetFalse/GetTrue, GetInt, or GetCount instead")
Val(uint64_t u, TypeTag t)
{
val.uint_val = u;
type = base_type(t);
#ifdef DEBUG
bound_id = 0;
#endif
}
Val(double d, TypeTag t)
{
val.double_val = d;
@ -429,23 +379,6 @@ protected:
};
class PortManager {
public:
// Port number given in host order.
ZEEK_DEPRECATED("Remove in v3.1: use val_mgr->GetPort() instead")
PortVal* Get(uint32_t port_num, TransportProto port_type) const;
// Host-order port number already masked with port space protocol mask.
ZEEK_DEPRECATED("Remove in v3.1: use val_mgr->GetPort() instead")
PortVal* Get(uint32_t port_num) const;
// Returns a masked port number
ZEEK_DEPRECATED("Remove in v3.1: use PortVal::Mask() instead")
uint32_t Mask(uint32_t port_num, TransportProto port_type) const;
};
extern PortManager* port_mgr;
// Holds pre-allocated Val objects for those where it's more optimal to
// re-use existing ones rather than allocate anew.
class ValManager {
@ -521,14 +454,6 @@ protected:
class PortVal : public Val {
public:
// Port number given in host order.
ZEEK_DEPRECATED("Remove in v3.1: use val_mgr->GetPort() instead")
PortVal(uint32_t p, TransportProto port_type);
// Host-order port number already masked with port space protocol mask.
ZEEK_DEPRECATED("Remove in v3.1: use val_mgr->GetPort() instead")
explicit PortVal(uint32_t p);
Val* SizeVal() const override { return val_mgr->GetInt(val.uint_val); }
// Returns the port number in host order (not including the mask).
@ -559,7 +484,7 @@ protected:
friend class Val;
friend class ValManager;
PortVal() {}
PortVal(uint32_t p, bool unused);
PortVal(uint32_t p);
void ValDescribe(ODesc* d) const override;
Val* DoClone(CloneState* state) override;
@ -1014,13 +939,6 @@ protected:
class EnumVal : public Val {
public:
ZEEK_DEPRECATED("Remove in v3.1: use t->GetVal(i) instead")
EnumVal(int i, EnumType* t) : Val(t)
{
val.int_val = i;
}
Val* SizeVal() const override { return val_mgr->GetInt(val.int_val); }
protected:

2
src/analyzer/protocol/ftp/FTP.cc
View file

@ -83,7 +83,7 @@ void FTP_Analyzer::DeliverStream(int length, const u_char* data, bool orig)
StringVal* cmd_str;
line = skip_whitespace(line, end_of_line);
get_word(length, line, cmd_len, cmd);
get_word(end_of_line - line, line, cmd_len, cmd);
line = skip_whitespace(line + cmd_len, end_of_line);
if ( cmd_len == 0 )

5
src/analyzer/protocol/rfb/events.bif
View file

@ -1,8 +1,3 @@
## Generated for RFB event
##
## c: The connection record for the underlying transport-layer session/flow.
event rfb_event%(c: connection%) &deprecated="Remove in v3.1: This event never served a real purpose and will be removed. Please use the other rfb events instead.";
## Generated for RFB event authentication mechanism selection
##
## c: The connection record for the underlying transport-layer session/flow.

11
src/analyzer/protocol/rfb/rfb-analyzer.pac
View file

@ -1,11 +1,4 @@
refine flow RFB_Flow += {
function proc_rfb_message(msg: RFB_PDU): bool
%{
if ( rfb_event )
BifEvent::generate_rfb_event(connection()->bro_analyzer(), connection()->bro_analyzer()->Conn());
return true;
%}
function proc_rfb_version(client: bool, major: bytestring, minor: bytestring) : bool
%{
if (client)
@ -371,7 +364,3 @@ refine connection RFB_Conn += {
return true;
%}
};
refine typeattr RFB_PDU += &let {
proc: bool = $context.flow.proc_rfb_message(this);
};

69
src/analyzer/protocol/smb/smb1-com-transaction.pac
View file

@ -4,10 +4,29 @@ enum Trans_subcommands {
NT_TRANSACT_CREATE2 = 0x0009,
};
%code{
StringVal* SMB_Conn::transaction_data_to_val(SMB1_transaction_data* payload)
{
switch ( payload->trans_type() ) {
case SMB_PIPE:
return bytestring_to_val(payload->pipe_data());
case SMB_UNKNOWN:
return bytestring_to_val(payload->unknown());
default:
return bytestring_to_val(payload->data());
}
assert(false);
return val_mgr->GetEmptyString();
}
%}
refine connection SMB_Conn += {
%member{
map<uint16, bool> is_file_a_pipe;
static StringVal* transaction_data_to_val(SMB1_transaction_data* payload);
%}
function get_is_file_a_pipe(id: uint16): bool
@ -37,32 +56,11 @@ refine connection SMB_Conn += {
StringVal* parameters = new StringVal(${val.parameters}.length(),
(const char*)${val.parameters}.data());
StringVal* payload_str = nullptr;
SMB1_transaction_data* payload = nullptr;
if ( ${val.data_count} > 0 )
{
payload = ${val.data};
}
if ( payload )
{
switch ( payload->trans_type() ) {
case SMB_PIPE:
payload_str = new StringVal(${val.data_count}, (const char*)${val.data.pipe_data}.data());
break;
case SMB_UNKNOWN:
payload_str = new StringVal(${val.data_count}, (const char*)${val.data.unknown}.data());
break;
default:
payload_str = new StringVal(${val.data_count}, (const char*)${val.data.data}.data());
break;
}
}
if ( ! payload_str )
{
payload_str = transaction_data_to_val(${val.data});
else
payload_str = val_mgr->GetEmptyString();
}
BifEvent::generate_smb1_transaction_request(bro_analyzer(),
bro_analyzer()->Conn(),
@ -83,32 +81,11 @@ refine connection SMB_Conn += {
StringVal* parameters = new StringVal(${val.parameters}.length(),
(const char*)${val.parameters}.data());
StringVal* payload_str = nullptr;
SMB1_transaction_data* payload = nullptr;
if ( ${val.data_count} > 0 )
{
payload = ${val.data[0]};
}
if ( payload )
{
switch ( payload->trans_type() ) {
case SMB_PIPE:
payload_str = new StringVal(${val.data_count}, (const char*)${val.data[0].pipe_data}.data());
break;
case SMB_UNKNOWN:
payload_str = new StringVal(${val.data_count}, (const char*)${val.data[0].unknown}.data());
break;
default:
payload_str = new StringVal(${val.data_count}, (const char*)${val.data[0].data}.data());
break;
}
}
if ( ! payload_str )
{
payload_str = transaction_data_to_val(${val.data[0]});
else
payload_str = val_mgr->GetEmptyString();
}
BifEvent::generate_smb1_transaction_response(bro_analyzer(),
bro_analyzer()->Conn(),

6
src/analyzer/protocol/ssl/proc-certificate.pac
View file

@ -16,6 +16,12 @@
{
const bytestring& cert = (*certificates)[i];
if ( cert.length() <= 0 )
{
reporter->Weird(bro_analyzer()->Conn(), "zero_length_certificate");
continue;
}
ODesc file_handle;
file_handle.Add(common.Description());
file_handle.Add(i);

6
src/analyzer/protocol/ssl/tls-handshake-analyzer.pac
View file

@ -303,7 +303,7 @@ refine connection Handshake_Conn += {
common.AddRaw("F");
bro_analyzer()->Conn()->IDString(&common);
if ( status_type == 1 ) // ocsp
if ( status_type == 1 && response.length() > 0 ) // ocsp
{
ODesc file_handle;
file_handle.Add(common.Description());
@ -323,6 +323,10 @@ refine connection Handshake_Conn += {
file_mgr->EndOfFile(file_id);
}
else if ( response.length() == 0 )
{
reporter->Weird(bro_analyzer()->Conn(), "SSL_zero_length_stapled_OCSP_message");
}
return true;
%}

9
src/event.bif
View file

@ -48,9 +48,6 @@
##
event zeek_init%(%);
## Deprecated synonym for :zeek:see:`zeek_init`.
event bro_init%(%) &deprecated="Remove in v3.1: use zeek_init";
## Generated at Zeek termination time. The event engine generates this event when
## Zeek is about to terminate, either due to having exhausted reading its input
## trace file(s), receiving a termination signal, or because Zeek was run without
@ -64,9 +61,6 @@ event bro_init%(%) &deprecated="Remove in v3.1: use zeek_init";
## is not generated.
event zeek_done%(%);
## Deprecated synonym for :zeek:see:`zeek_done`.
event bro_done%(%) &deprecated="Remove in v3.1: use zeek_done";
## Generated for every new connection. This event is raised with the first
## packet of a previously unknown connection. Zeek uses a flow-based definition
## of "connection" here that includes not only TCP sessions but also UDP and
@ -666,9 +660,6 @@ event reporter_error%(t: time, msg: string, location: string%) &error_handler;
## recursively for each ``@load``.
event zeek_script_loaded%(path: string, level: count%);
## Deprecated synonym for :zeek:see:`zeek_script_loaded`.
event bro_script_loaded%(path: string, level: count%) &deprecated="Remove in v3.1: use zeek_script_loaded";
## Generated each time Zeek's script interpreter opens a file. This event is
## triggered only for files opened via :zeek:id:`open`, and in particular not for
## normal log files as created by log writers.

4
src/input.h
View file

@ -37,8 +37,8 @@ extern int bro_argc;
extern char** bro_argv;
extern const char* prog;
extern name_list prefixes; // -p flag
extern char* command_line_policy; // -e flag
extern std::vector<std::string> zeek_script_prefixes; // -p flag
extern const char* command_line_policy; // -e flag
extern std::vector<std::string> params;
class Stmt;

625
src/main.cc
View file

@ -7,10 +7,9 @@
#include <unistd.h>
#include <signal.h>
#include <string.h>
#include <sys/types.h>
#include <list>
#ifdef HAVE_GETOPT_H
#include <getopt.h>
#endif
#include <optional>
#ifdef USE_IDMEF
extern "C" {
@ -21,7 +20,7 @@ extern "C" {
#include <openssl/ssl.h>
#include <openssl/err.h>
#include "bsd-getopt-long.h"
#include "Options.h"
#include "input.h"
#include "DNS_Mgr.h"
#include "Frame.h"
@ -43,10 +42,10 @@ extern "C" {
#include "Brofiler.h"
#include "Traverse.h"
#include "supervisor/Supervisor.h"
#include "threading/Manager.h"
#include "input/Manager.h"
#include "logging/Manager.h"
#include "logging/writers/ascii/Ascii.h"
#include "input/readers/raw/Raw.h"
#include "analyzer/Manager.h"
#include "analyzer/Tag.h"
@ -84,7 +83,6 @@ int perftools_profile = 0;
DNS_Mgr* dns_mgr;
TimerMgr* timer_mgr;
ValManager* val_mgr = 0;
PortManager* port_mgr = 0;
logging::Manager* log_mgr = 0;
threading::Manager* thread_mgr = 0;
input::Manager* input_mgr = 0;
@ -94,10 +92,9 @@ file_analysis::Manager* file_mgr = 0;
zeekygen::Manager* zeekygen_mgr = 0;
iosource::Manager* iosource_mgr = 0;
bro_broker::Manager* broker_mgr = 0;
zeek::Supervisor* zeek::supervisor_mgr = 0;
const char* prog;
char* writefile = 0;
name_list prefixes;
std::vector<std::string> zeek_script_prefixes;
Stmt* stmts;
EventHandlerPtr net_done = 0;
RuleMatcher* rule_matcher = 0;
@ -107,10 +104,10 @@ ProfileLogger* segment_logger = 0;
SampleLogger* sample_logger = 0;
int signal_val = 0;
extern char version[];
char* command_line_policy = 0;
const char* command_line_policy = 0;
vector<string> params;
set<string> requested_plugins;
char* proc_status_file = 0;
const char* proc_status_file = 0;
OpaqueType* md5_type = 0;
OpaqueType* sha1_type = 0;
@ -145,74 +142,15 @@ const char* zeek_version()
#endif
}
bool bro_dns_fake()
static std::vector<const char*> to_cargs(const std::vector<std::string>& args)
{
return zeekenv("ZEEK_DNS_FAKE");
}
std::vector<const char*> rval;
rval.reserve(args.size());
void usage(int code = 1)
{
fprintf(stderr, "zeek version %s\n", zeek_version());
for ( const auto& arg : args )
rval.emplace_back(arg.data());
fprintf(stderr, "usage: %s [options] [file ...]\n", prog);
fprintf(stderr, "usage: %s --test [doctest-options] -- [options] [file ...]\n", prog);
fprintf(stderr, " <file> | policy file, or read stdin\n");
fprintf(stderr, " -a|--parse-only | exit immediately after parsing scripts\n");
fprintf(stderr, " -b|--bare-mode | don't load scripts from the base/ directory\n");
fprintf(stderr, " -d|--debug-policy | activate policy file debugging\n");
fprintf(stderr, " -e|--exec <zeek code> | augment loaded policies by given code\n");
fprintf(stderr, " -f|--filter <filter> | tcpdump filter\n");
fprintf(stderr, " -h|--help | command line help\n");
fprintf(stderr, " -i|--iface <interface> | read from given interface\n");
fprintf(stderr, " -p|--prefix <prefix> | add given prefix to policy file resolution\n");
fprintf(stderr, " -r|--readfile <readfile> | read from given tcpdump file\n");
fprintf(stderr, " -s|--rulefile <rulefile> | read rules from given file\n");
fprintf(stderr, " -t|--tracefile <tracefile> | activate execution tracing\n");
fprintf(stderr, " -v|--version | print version and exit\n");
fprintf(stderr, " -w|--writefile <writefile> | write to given tcpdump file\n");
#ifdef DEBUG
fprintf(stderr, " -B|--debug <dbgstreams> | Enable debugging output for selected streams ('-B help' for help)\n");
#endif
fprintf(stderr, " -C|--no-checksums | ignore checksums\n");
fprintf(stderr, " -F|--force-dns | force DNS\n");
fprintf(stderr, " -G|--load-seeds <file> | load seeds from given file\n");
fprintf(stderr, " -H|--save-seeds <file> | save seeds to given file\n");
fprintf(stderr, " -I|--print-id <ID name> | print out given ID\n");
fprintf(stderr, " -N|--print-plugins | print available plugins and exit (-NN for verbose)\n");
fprintf(stderr, " -P|--prime-dns | prime DNS\n");
fprintf(stderr, " -Q|--time | print execution time summary to stderr\n");
fprintf(stderr, " -S|--debug-rules | enable rule debugging\n");
fprintf(stderr, " -T|--re-level <level> | set 'RE_level' for rules\n");
fprintf(stderr, " -U|--status-file <file> | Record process status in file\n");
fprintf(stderr, " -W|--watchdog | activate watchdog timer\n");
fprintf(stderr, " -X|--zeekygen <cfgfile> | generate documentation based on config file\n");
#ifdef USE_PERFTOOLS_DEBUG
fprintf(stderr, " -m|--mem-leaks | show leaks [perftools]\n");
fprintf(stderr, " -M|--mem-profile | record heap [perftools]\n");
#endif
fprintf(stderr, " --pseudo-realtime[=<speedup>] | enable pseudo-realtime for performance evaluation (default 1)\n");
#ifdef USE_IDMEF
fprintf(stderr, " -n|--idmef-dtd <idmef-msg.dtd> | specify path to IDMEF DTD file\n");
#endif
fprintf(stderr, " --test | run unit tests ('--test -h' for help, only when compiling with ENABLE_ZEEK_UNIT_TESTS)\n");
fprintf(stderr, " $ZEEKPATH | file search path (%s)\n", bro_path().c_str());
fprintf(stderr, " $ZEEK_PLUGIN_PATH | plugin search path (%s)\n", bro_plugin_path());
fprintf(stderr, " $ZEEK_PLUGIN_ACTIVATE | plugins to always activate (%s)\n", bro_plugin_activate());
fprintf(stderr, " $ZEEK_PREFIXES | prefix list (%s)\n", bro_prefixes().c_str());
fprintf(stderr, " $ZEEK_DNS_FAKE | disable DNS lookups (%s)\n", bro_dns_fake() ? "on" : "off");
fprintf(stderr, " $ZEEK_SEED_FILE | file to load seeds from (not set)\n");
fprintf(stderr, " $ZEEK_LOG_SUFFIX | ASCII log file extension (.%s)\n", logging::writer::Ascii::LogExt().c_str());
fprintf(stderr, " $ZEEK_PROFILER_FILE | Output file for script execution statistics (not set)\n");
fprintf(stderr, " $ZEEK_DISABLE_ZEEKYGEN | Disable Zeekygen documentation support (%s)\n", zeekenv("ZEEK_DISABLE_ZEEKYGEN") ? "set" : "not set");
fprintf(stderr, " $ZEEK_DNS_RESOLVER | IPv4/IPv6 address of DNS resolver to use (%s)\n", zeekenv("ZEEK_DNS_RESOLVER") ? zeekenv("ZEEK_DNS_RESOLVER") : "not set, will use first IPv4 address from /etc/resolv.conf");
fprintf(stderr, "\n");
exit(code);
return rval;
}
bool show_plugins(int level)
@ -359,21 +297,21 @@ void terminate_bro()
delete analyzer_mgr;
delete file_mgr;
// broker_mgr is deleted via iosource_mgr
// supervisor is deleted via iosource_mgr
delete iosource_mgr;
delete log_mgr;
delete reporter;
delete plugin_mgr;
delete val_mgr;
delete port_mgr;
reporter = 0;
}
void termination_signal()
void zeek_terminate_loop(const char* reason)
{
set_processing_status("TERMINATING", "termination_signal");
set_processing_status("TERMINATING", reason);
reporter->Info("%s", reason);
reporter->Info("received termination signal");
net_get_final_stats();
done_with_network();
net_delete();
@ -407,327 +345,141 @@ static void bro_new_handler()
out_of_memory("new");
}
static std::vector<std::string> get_script_signature_files()
{
std::vector<std::string> rval;
// Parse rule files defined on the script level.
char* script_signature_files =
copy_string(internal_val("signature_files")->AsString()->CheckString());
char* tmp = script_signature_files;
char* s;
while ( (s = strsep(&tmp, " \t")) )
if ( *s )
rval.emplace_back(s);
delete [] script_signature_files;
return rval;
}
static std::string get_exe_path(const std::string& invocation)
{
if ( invocation.empty() )
return "";
if ( invocation[0] == '/' || invocation[0] == '~' )
// Absolute path
return invocation;
if ( invocation.find('/') != std::string::npos )
{
// Relative path
char cwd[PATH_MAX];
if ( ! getcwd(cwd, sizeof(cwd)) )
{
fprintf(stderr, "failed to get current directory: %s\n",
strerror(errno));
exit(1);
}
return std::string(cwd) + "/" + invocation;
}
auto path = getenv("PATH");
if ( ! path )
return "";
return find_file(invocation, path);
}
int main(int argc, char** argv)
{
ZEEK_LSAN_DISABLE();
std::set_new_handler(bro_new_handler);
// When running unit tests, the first argument on the command line must be
// --test, followed by doctest options. Optionally, users can use "--" as
// separator to pass Zeek options afterwards:
//
// zeek --test [doctest-options] -- [zeek-options]
auto zeek_exe_path = get_exe_path(argv[0]);
if ( argc > 1 && strcmp(argv[1], "--test") == 0 )
if ( zeek_exe_path.empty() )
{
// Deal with CLI arguments (copy Zeek part over to bro_argv).
auto is_separator = [](const char* cstr)
{
return strcmp(cstr, "--") == 0;
};
auto first = argv;
auto last = argv + argc;
auto separator = std::find_if(first, last, is_separator);
if ( separator == last )
{
bro_argc = 1;
bro_argv = new char*[1];
bro_argv[0] = copy_string(argv[0]);
}
else
{
auto first_zeek_arg = std::next(separator);
bro_argc = 1 + std::distance(first_zeek_arg, last);
bro_argv = new char*[bro_argc];
bro_argv[0] = copy_string(argv[0]);
auto bro_argv_iter = std::next(bro_argv);
for ( auto i = first_zeek_arg; i != last; ++i )
*bro_argv_iter++ = copy_string(*i);
fprintf(stderr, "failed to get path to executable '%s'", argv[0]);
exit(1);
}
#ifdef DOCTEST_CONFIG_DISABLE
fprintf(stderr, "ERROR: C++ unit tests are disabled for this build.\n"
" Please re-compile with ENABLE_ZEEK_UNIT_TESTS "
"to run the C++ unit tests.\n");
return EXIT_FAILURE;
#else
doctest::Context context;
context.applyCommandLine(std::distance(first, separator), argv);
ZEEK_LSAN_ENABLE();
return context.run();
#endif
}
else
{
bro_argc = argc;
bro_argv = new char* [argc];
for ( int i = 0; i < argc; i++ )
bro_argv[i] = copy_string(argv[i]);
auto options = zeek::parse_cmdline(argc, argv);
if ( options.print_usage )
zeek::usage(argv[0], 0);
if ( options.print_version )
{
fprintf(stdout, "%s version %s\n", argv[0], zeek_version());
exit(0);
}
if ( options.run_unit_tests )
{
doctest::Context context;
auto dargs = to_cargs(options.doctest_args);
context.applyCommandLine(dargs.size(), dargs.data());
ZEEK_LSAN_ENABLE();
return context.run();
}
auto stem_state = zeek::Supervisor::CreateStem(options.supervisor_mode);
if ( zeek::Supervisor::ThisNode() )
zeek::Supervisor::ThisNode()->Init(&options);
double time_start = current_time(true);
brofiler.ReadStats();
name_list interfaces;
name_list read_files;
name_list rule_files;
char* id_name = 0;
auto dns_type = options.dns_mode;
char* seed_load_file = zeekenv("ZEEK_SEED_FILE");
char* seed_save_file = 0;
char* user_pcap_filter = 0;
char* debug_streams = 0;
int parse_only = false;
int bare_mode = false;
int do_watchdog = 0;
int override_ignore_checksums = 0;
int rule_debug = 0;
int RE_level = 4;
int print_plugins = 0;
int time_bro = 0;
static struct option long_opts[] = {
{"parse-only", no_argument, 0, 'a'},
{"bare-mode", no_argument, 0, 'b'},
{"debug-policy", no_argument, 0, 'd'},
{"exec", required_argument, 0, 'e'},
{"filter", required_argument, 0, 'f'},
{"help", no_argument, 0, 'h'},
{"iface", required_argument, 0, 'i'},
{"zeekygen", required_argument, 0, 'X'},
{"prefix", required_argument, 0, 'p'},
{"readfile", required_argument, 0, 'r'},
{"rulefile", required_argument, 0, 's'},
{"tracefile", required_argument, 0, 't'},
{"writefile", required_argument, 0, 'w'},
{"version", no_argument, 0, 'v'},
{"no-checksums", no_argument, 0, 'C'},
{"force-dns", no_argument, 0, 'F'},
{"load-seeds", required_argument, 0, 'G'},
{"save-seeds", required_argument, 0, 'H'},
{"print-plugins", no_argument, 0, 'N'},
{"prime-dns", no_argument, 0, 'P'},
{"time", no_argument, 0, 'Q'},
{"debug-rules", no_argument, 0, 'S'},
{"re-level", required_argument, 0, 'T'},
{"watchdog", no_argument, 0, 'W'},
{"print-id", required_argument, 0, 'I'},
{"status-file", required_argument, 0, 'U'},
#ifdef DEBUG
{"debug", required_argument, 0, 'B'},
#endif
#ifdef USE_IDMEF
{"idmef-dtd", required_argument, 0, 'n'},
#endif
#ifdef USE_PERFTOOLS_DEBUG
{"mem-leaks", no_argument, 0, 'm'},
{"mem-profile", no_argument, 0, 'M'},
#endif
{"pseudo-realtime", optional_argument, 0, 'E'},
{"test", no_argument, 0, '#'},
{0, 0, 0, 0},
};
enum DNS_MgrMode dns_type = DNS_DEFAULT;
dns_type = bro_dns_fake() ? DNS_FAKE : DNS_DEFAULT;
if ( dns_type == DNS_DEFAULT && zeek::fake_dns() )
dns_type = DNS_FAKE;
RETSIGTYPE (*oldhandler)(int);
prog = argv[0];
zeek_script_prefixes = options.script_prefixes;
auto zeek_prefixes = zeekenv("ZEEK_PREFIXES");
prefixes.push_back(strdup("")); // "" = "no prefix"
if ( zeek_prefixes )
tokenize_string(zeek_prefixes, ":", &zeek_script_prefixes);
char* p = zeekenv("ZEEK_PREFIXES");
if ( p )
add_to_name_list(p, ':', prefixes);
string zeekygen_config;
#ifdef USE_IDMEF
string libidmef_dtd_path = "idmef-message.dtd";
#endif
extern char* optarg;
extern int optind, opterr;
int long_optsind;
opterr = 0;
char opts[256];
safe_strncpy(opts, "B:e:f:G:H:I:i:n:p:r:s:T:t:U:w:X:CFNPQSWabdhv",
sizeof(opts));
pseudo_realtime = options.pseudo_realtime;
#ifdef USE_PERFTOOLS_DEBUG
strncat(opts, "mM", 2);
perftools_leaks = options.perftools_check_leaks;
perftools_profile = options.perftools_profile;
#endif
int op;
while ( (op = getopt_long(bro_argc, bro_argv, opts, long_opts, &long_optsind)) != EOF )
switch ( op ) {
case 'a':
parse_only = true;
break;
case 'b':
bare_mode = true;
break;
case 'd':
fprintf(stderr, "Policy file debugging ON.\n");
g_policy_debug = true;
break;
case 'e':
command_line_policy = optarg;
break;
case 'f':
user_pcap_filter = optarg;
break;
case 'h':
usage(0);
break;
case 'i':
interfaces.push_back(optarg);
break;
case 'p':
prefixes.push_back(optarg);
break;
case 'r':
read_files.push_back(optarg);
break;
case 's':
rule_files.push_back(optarg);
break;
case 't':
g_trace_state.SetTraceFile(optarg);
g_trace_state.TraceOn();
break;
case 'v':
fprintf(stdout, "%s version %s\n", prog, zeek_version());
exit(0);
break;
case 'w':
writefile = optarg;
break;
case 'B':
debug_streams = optarg;
break;
case 'C':
override_ignore_checksums = 1;
break;
case 'E':
pseudo_realtime = 1.0;
if ( optarg )
pseudo_realtime = atof(optarg);
break;
case 'F':
if ( dns_type != DNS_DEFAULT )
usage(1);
dns_type = DNS_FORCE;
break;
case 'G':
seed_load_file = optarg;
break;
case 'H':
seed_save_file = optarg;
break;
case 'I':
id_name = optarg;
break;
case 'N':
++print_plugins;
break;
case 'P':
if ( dns_type != DNS_DEFAULT )
usage(1);
dns_type = DNS_PRIME;
break;
case 'Q':
time_bro = 1;
break;
case 'S':
rule_debug = 1;
break;
case 'T':
RE_level = atoi(optarg);
break;
case 'U':
proc_status_file = optarg;
break;
case 'W':
do_watchdog = 1;
break;
case 'X':
zeekygen_config = optarg;
break;
#ifdef USE_PERFTOOLS_DEBUG
case 'm':
perftools_leaks = 1;
break;
case 'M':
perftools_profile = 1;
break;
#endif
#ifdef USE_IDMEF
case 'n':
fprintf(stderr, "Using IDMEF XML DTD from %s\n", optarg);
libidmef_dtd_path = optarg;
break;
#endif
case '#':
fprintf(stderr, "ERROR: --test only allowed as first argument.\n");
usage(1);
break;
case 0:
// This happens for long options that don't have
// a short-option equivalent.
break;
case '?':
default:
usage(1);
break;
if ( options.debug_scripts )
{
g_policy_debug = options.debug_scripts;
fprintf(stderr, "Zeek script debugging ON.\n");
}
if ( interfaces.length() > 0 && read_files.length() > 0 )
usage(1);
if ( options.script_code_to_exec )
command_line_policy = options.script_code_to_exec->data();
if ( options.debug_script_tracing_file )
{
g_trace_state.SetTraceFile(options.debug_script_tracing_file->data());
g_trace_state.TraceOn();
}
if ( options.process_status_file )
proc_status_file = options.process_status_file->data();
atexit(atexit_handler);
set_processing_status("INITIALIZING", "main");
@ -735,20 +487,38 @@ int main(int argc, char** argv)
bro_start_time = current_time(true);
val_mgr = new ValManager();
port_mgr = new PortManager();
reporter = new Reporter();
thread_mgr = new threading::Manager();
plugin_mgr = new plugin::Manager();
#ifdef DEBUG
if ( debug_streams )
if ( options.debug_log_streams )
{
debug_logger.EnableStreams(debug_streams);
debug_logger.EnableStreams(options.debug_log_streams->data());
if ( getenv("ZEEK_DEBUG_LOG_STDERR") )
debug_logger.OpenDebugLog(nullptr);
else
debug_logger.OpenDebugLog("debug");
}
#endif
init_random_seed((seed_load_file && *seed_load_file ? seed_load_file : 0) , seed_save_file);
if ( options.supervisor_mode )
{
zeek::Supervisor::Config cfg = {};
cfg.zeek_exe_path = zeek_exe_path;
options.filter_supervisor_options();
zeek::supervisor_mgr = new zeek::Supervisor(std::move(cfg),
std::move(*stem_state));
}
const char* seed_load_file = zeekenv("ZEEK_SEED_FILE");
if ( options.random_seed_input_file )
seed_load_file = options.random_seed_input_file->data();
init_random_seed((seed_load_file && *seed_load_file ? seed_load_file : 0),
options.random_seed_output_file ? options.random_seed_output_file->data() : 0);
// DEBUG_MSG("HMAC key: %s\n", md5_digest_print(shared_hmac_md5_key));
init_hash_function();
@ -767,9 +537,9 @@ int main(int argc, char** argv)
reporter->Error("Failed to initialize sqlite3: %s", sqlite3_errstr(r));
#ifdef USE_IDMEF
char* libidmef_dtd_path_cstr = new char[libidmef_dtd_path.length() + 1];
safe_strncpy(libidmef_dtd_path_cstr, libidmef_dtd_path.c_str(),
libidmef_dtd_path.length());
char* libidmef_dtd_path_cstr = new char[options.libidmef_dtd_file.size() + 1];
safe_strncpy(libidmef_dtd_path_cstr, options.libidmef_dtd_file.data(),
options.libidmef_dtd_file.size());
globalsInit(libidmef_dtd_path_cstr); // Init LIBIDMEF globals
createCurrentDoc("1.0"); // Set a global XML document
#endif
@ -777,34 +547,34 @@ int main(int argc, char** argv)
timer_mgr = new PQ_TimerMgr("<GLOBAL>");
// timer_mgr = new CQ_TimerMgr();
zeekygen_mgr = new zeekygen::Manager(zeekygen_config, bro_argv[0]);
auto zeekygen_cfg = options.zeekygen_config_file.value_or("");
zeekygen_mgr = new zeekygen::Manager(zeekygen_cfg, bro_argv[0]);
add_essential_input_file("base/init-bare.zeek");
add_essential_input_file("base/init-frameworks-and-bifs.zeek");
if ( ! bare_mode )
if ( ! options.bare_mode )
add_input_file("base/init-default.zeek");
plugin_mgr->SearchDynamicPlugins(bro_plugin_path());
if ( optind == bro_argc &&
read_files.length() == 0 &&
interfaces.length() == 0 &&
! id_name && ! command_line_policy && ! print_plugins )
if ( options.plugins_to_load.empty() && options.scripts_to_load.empty() &&
options.script_options_to_set.empty() &&
options.pcap_files.size() == 0 &&
options.interfaces.size() == 0 &&
! options.identifier_to_print &&
! command_line_policy && ! options.print_plugins &&
! options.supervisor_mode && ! zeek::Supervisor::ThisNode() )
add_input_file("-");
// Process remaining arguments. X=Y arguments indicate script
// variable/parameter assignments. X::Y arguments indicate plugins to
// activate/query. The remainder are treated as scripts to load.
while ( optind < bro_argc )
{
if ( strchr(bro_argv[optind], '=') )
params.push_back(bro_argv[optind++]);
else if ( strstr(bro_argv[optind], "::") )
requested_plugins.insert(bro_argv[optind++]);
else
add_input_file(bro_argv[optind++]);
}
for ( const auto& script_option : options.script_options_to_set )
params.push_back(script_option);
for ( const auto& plugin : options.plugins_to_load )
requested_plugins.insert(plugin);
for ( const auto& script : options.scripts_to_load )
add_input_file(script.data());
push_scope(nullptr, nullptr);
@ -821,7 +591,7 @@ int main(int argc, char** argv)
log_mgr = new logging::Manager();
input_mgr = new input::Manager();
file_mgr = new file_analysis::Manager();
broker_mgr = new bro_broker::Manager(read_files.length() > 0);
broker_mgr = new bro_broker::Manager(! options.pcap_files.empty());
plugin_mgr->InitPreScript();
analyzer_mgr->InitPreScript();
@ -840,7 +610,7 @@ int main(int argc, char** argv)
if ( missing_plugin )
reporter->FatalError("Failed to activate requested dynamic plugin(s).");
plugin_mgr->ActivateDynamicPlugins(! bare_mode);
plugin_mgr->ActivateDynamicPlugins(! options.bare_mode);
init_event_handlers();
@ -896,9 +666,9 @@ int main(int argc, char** argv)
zeekygen_mgr->InitPostScript();
broker_mgr->InitPostScript();
if ( print_plugins )
if ( options.print_plugins )
{
bool success = show_plugins(print_plugins);
bool success = show_plugins(options.print_plugins);
exit(success ? 0 : 1);
}
@ -906,7 +676,7 @@ int main(int argc, char** argv)
file_mgr->InitPostScript();
dns_mgr->InitPostScript();
if ( parse_only )
if ( options.parse_only )
{
int rc = (reporter->Errors() > 0 ? 1 : 0);
exit(rc);
@ -925,52 +695,48 @@ int main(int argc, char** argv)
reporter->InitOptions();
zeekygen_mgr->GenerateDocs();
if ( user_pcap_filter )
if ( options.pcap_filter )
{
ID* id = global_scope()->Lookup("cmd_line_bpf_filter");
if ( ! id )
reporter->InternalError("global cmd_line_bpf_filter not defined");
id->SetVal(new StringVal(user_pcap_filter));
id->SetVal(new StringVal(*options.pcap_filter));
}
// Parse rule files defined on the script level.
char* script_rule_files =
copy_string(internal_val("signature_files")->AsString()->CheckString());
auto all_signature_files = options.signature_files;
char* tmp = script_rule_files;
char* s;
while ( (s = strsep(&tmp, " \t")) )
if ( *s )
rule_files.push_back(s);
// Append signature files defined in "signature_files" script option
for ( auto&& sf : get_script_signature_files() )
all_signature_files.emplace_back(std::move(sf));
// Append signature files defined in @load-sigs
for ( size_t i = 0; i < sig_files.size(); ++i )
rule_files.push_back(copy_string(sig_files[i].c_str()));
for ( const auto& sf : sig_files )
all_signature_files.emplace_back(sf);
if ( rule_files.length() > 0 )
if ( ! all_signature_files.empty() )
{
rule_matcher = new RuleMatcher(RE_level);
if ( ! rule_matcher->ReadFiles(rule_files) )
rule_matcher = new RuleMatcher(options.signature_re_level);
if ( ! rule_matcher->ReadFiles(all_signature_files) )
{
delete dns_mgr;
exit(1);
}
if ( rule_debug )
if ( options.print_signature_debug_info )
rule_matcher->PrintDebug();
file_mgr->InitMagic();
}
delete [] script_rule_files;
if ( g_policy_debug )
// ### Add support for debug command file.
dbg_init_debugger(0);
if ( read_files.length() == 0 && interfaces.length() == 0 )
auto all_interfaces = options.interfaces;
if ( options.pcap_files.empty() && options.interfaces.empty() )
{
Val* interfaces_val = internal_val("interfaces");
if ( interfaces_val )
@ -979,14 +745,15 @@ int main(int argc, char** argv)
interfaces_val->AsString()->Render();
if ( interfaces_str[0] != '\0' )
add_to_name_list(interfaces_str, ' ', interfaces);
tokenize_string(interfaces_str, " ", &all_interfaces);
delete [] interfaces_str;
}
}
if ( dns_type != DNS_PRIME )
net_init(interfaces, read_files, writefile, do_watchdog);
net_init(all_interfaces, options.pcap_files,
options.pcap_output_file, options.use_watchdog);
net_done = internal_handler("net_done");
@ -1015,11 +782,11 @@ int main(int argc, char** argv)
}
// Print the ID.
if ( id_name )
if ( options.identifier_to_print )
{
ID* id = global_scope()->Lookup(id_name);
ID* id = global_scope()->Lookup(*options.identifier_to_print);
if ( ! id )
reporter->FatalError("No such ID: %s\n", id_name);
reporter->FatalError("No such ID: %s\n", options.identifier_to_print->data());
ODesc desc;
desc.SetQuotes(true);
@ -1082,7 +849,7 @@ int main(int argc, char** argv)
g_frame_stack.pop_back();
}
if ( override_ignore_checksums )
if ( options.ignore_checksums )
ignore_checksums = 1;
if ( zeek_script_loaded )
@ -1116,6 +883,9 @@ int main(int argc, char** argv)
iosource_mgr->Register(thread_mgr, true);
if ( zeek::supervisor_mgr )
iosource_mgr->Register(zeek::supervisor_mgr);
if ( iosource_mgr->Size() > 0 ||
have_pending_timers ||
BifConst::exit_only_after_terminate )
@ -1135,12 +905,15 @@ int main(int argc, char** argv)
#endif
if ( zeek::Supervisor::ThisNode() )
timer_mgr->Add(new zeek::ParentProcessCheckTimer(1, 1));
double time_net_start = current_time(true);;
uint64_t mem_net_start_total;
uint64_t mem_net_start_malloced;
if ( time_bro )
if ( options.print_execution_time )
{
get_memory_usage(&mem_net_start_total, &mem_net_start_malloced);
@ -1158,7 +931,7 @@ int main(int argc, char** argv)
uint64_t mem_net_done_total;
uint64_t mem_net_done_malloced;
if ( time_bro )
if ( options.print_execution_time )
{
get_memory_usage(&mem_net_done_total, &mem_net_done_malloced);

2
src/nb_dns.c
View file

@ -411,7 +411,7 @@ _nb_dns_mkquery(register struct nb_dns_info *nd, register const char *name,
return (-1);
}
memset(ne, 0, sizeof(*ne));
strncpy(ne->name, name, sizeof(ne->name));
strncpy(ne->name, name, sizeof(ne->name) - 1);
ne->name[sizeof(ne->name) - 1] = '\0';
ne->qtype = qtype;
ne->atype = atype;

14
src/parse.y
View file

@ -1164,14 +1164,12 @@ func_hdr:
}
| TOK_EVENT event_id func_params opt_attr
{
// Gracefully handle the deprecation of bro_init, bro_done,
// and bro_script_loaded
if ( streq("bro_init", $2->Name()) )
$2 = global_scope()->Lookup("zeek_init");
else if ( streq("bro_done", $2->Name()) )
$2 = global_scope()->Lookup("zeek_done");
else if ( streq("bro_script_loaded", $2->Name()) )
$2 = global_scope()->Lookup("zeek_script_loaded");
const char* name = $2->Name();
if ( streq("bro_init", name) || streq("bro_done", name) || streq("bro_script_loaded", name) )
{
auto base = std::string(name).substr(4);
reporter->Error(fmt("event %s() is no longer available, use zeek_%s() instead", name, base.c_str()));
}
begin_func($2, current_module.c_str(),
FUNC_FLAVOR_EVENT, 0, $3, $4);

3
src/plugin/Manager.cc
View file

@ -189,6 +189,7 @@ bool Manager::ActivateDynamicPluginInternal(const std::string& name, bool ok_if_
if ( is_file(init) )
{
DBG_LOG(DBG_PLUGINS, " Loading %s", init.c_str());
warn_if_legacy_script(init);
scripts_to_load.push_back(init);
break;
}
@ -202,6 +203,7 @@ bool Manager::ActivateDynamicPluginInternal(const std::string& name, bool ok_if_
if ( is_file(init) )
{
DBG_LOG(DBG_PLUGINS, " Loading %s", init.c_str());
warn_if_legacy_script(init);
scripts_to_load.push_back(init);
break;
}
@ -214,6 +216,7 @@ bool Manager::ActivateDynamicPluginInternal(const std::string& name, bool ok_if_
if ( is_file(init) )
{
DBG_LOG(DBG_PLUGINS, " Loading %s", init.c_str());
warn_if_legacy_script(init);
scripts_to_load.push_back(init);
break;
}

11
src/scan.l
View file

@ -435,10 +435,9 @@ when return TOK_WHEN;
pref = skip_whitespace(pref + 1); // Skip over '='.
if ( ! append )
while ( prefixes.length() > 1 ) // don't delete "" prefix
delete prefixes.remove_nth(1);
zeek_script_prefixes = { "" }; // don't delete the "" prefix
add_to_name_list(pref, ':', prefixes);
tokenize_string(pref, ":", &zeek_script_prefixes);
}
@if return TOK_ATIF;
@ -943,14 +942,14 @@ int yywrap()
it->prefixes_checked = true;
// Prefixes are pushed onto a stack, so iterate backwards.
for ( int i = prefixes.length() - 1; i >= 0; --i )
for ( int i = zeek_script_prefixes.size() - 1; i >= 0; --i )
{
// Don't look at empty prefixes.
if ( ! prefixes[i][0] )
if ( ! zeek_script_prefixes[i][0] )
continue;
string canon = without_bropath_component(it->name);
string flat = flatten_script_name(canon, prefixes[i]);
string flat = flatten_script_name(canon, zeek_script_prefixes[i]);
string path = find_relative_script_file(flat);
if ( ! path.empty() )

41
src/strings.bif
View file

@ -935,47 +935,6 @@ function safe_shell_quote%(source: string%): string
return new StringVal(new BroString(1, dst, j));
%}
## Takes a string and escapes characters that would allow execution of
## commands at the shell level. Must be used before including strings in
## :zeek:id:`system` or similar calls. This function is deprecated, use
## :zeek:see:`safe_shell_quote` as a replacement. The difference is that
## :zeek:see:`safe_shell_quote` automatically returns a value that is
## wrapped in double-quotes, which is required to correctly and fully
## escape any characters that might be interpreted by the shell.
##
## source: The string to escape.
##
## Returns: A shell-escaped version of *source*.
##
## .. zeek:see:: system safe_shell_quote
function str_shell_escape%(source: string%): string &deprecated="Remove in v3.1: use safe_shell_quote"
%{
unsigned j = 0;
const u_char* src = source->Bytes();
unsigned n = source->Len();
byte_vec dst = new u_char[n * 2 + 1];
for ( unsigned i = 0; i < n; ++i )
{
switch ( src[i] ) {
case '`': case '"': case '\\': case '$':
// case '|': case '&': case ';': case '(': case ')': case '<':
// case '>': case '\'': case '*': case '?': case '[': case ']':
// case '!': case '#': case '{': case '}':
dst[j++] = '\\';
break;
default:
break;
}
dst[j++] = src[i];
}
dst[j] = '\0';
return new StringVal(new BroString(1, dst, j));
%}
## Finds all occurrences of a pattern in a string.
##
## str: The string to inspect.

1413
src/supervisor/Supervisor.cc Normal file
View file

File diff suppressed because it is too large Load diff

414
src/supervisor/Supervisor.h Normal file
View file

@ -0,0 +1,414 @@
// See the file "COPYING" in the main distribution directory for copyright.
#pragma once
#include <sys/types.h>
#include <optional>
#include <cstdint>
#include <string>
#include <string_view>
#include <vector>
#include <utility>
#include <memory>
#include <chrono>
#include <map>
#include "iosource/IOSource.h"
#include "Timer.h"
#include "Pipe.h"
#include "Flare.h"
#include "NetVar.h"
#include "IntrusivePtr.h"
#include "Options.h"
namespace zeek {
/**
* A Supervisor object manages a tree of persistent Zeek processes. If any
* child process dies it will be re-created with its original configuration.
* The Supervisor process itself actually only manages a single child process,
* called the Stem process. That Stem is created via a fork() just after the
* command-line arguments have been parsed. The Stem process is used as the
* baseline image for spawning and supervising further Zeek child nodes since
* it has the purest global state without having to risk an exec() using an
* on-disk binary that's changed in the meantime from the original Supervisor's
* version of the Zeek binary. However, if the Stem process itself dies
* prematurely, the Supervisor will have to fork() and exec() to revive it (and
* then the revived Stem will re-spawn its own children). Any node in the tree
* will self-terminate if it detects its parent has died and that detection is
* done via polling for change in parent process ID.
*/
class Supervisor : public iosource::IOSource {
public:
/**
* Configuration options that change Supervisor behavior.
*/
struct Config {
/**
* The filesystem path of the Zeek binary/executable. This is used
* if the Stem process ever dies and we need to fork() and exec() to
* re-create it.
*/
std::string zeek_exe_path;
};
/**
* Configuration options that influence how a Supervised Zeek node
* integrates into the normal Zeek Cluster Framework.
*/
struct ClusterEndpoint {
/**
* The node's role within the cluster. E.g. manager, logger, worker.
*/
BifEnum::Supervisor::ClusterRole role;
/**
* The host/IP at which the cluster node is listening for connections.
*/
std::string host;
/**
* The TCP port number at which the cluster node listens for connections.
*/
int port;
/**
* The interface name from which the node read/analyze packets.
* Typically used by worker nodes.
*/
std::optional<std::string> interface;
};
/**
* Configuration options that influence behavior of a Supervised Zeek node.
*/
struct NodeConfig {
/**
* Create configuration from script-layer record value.
* @param node_val the script-layer record value to convert.
*/
static NodeConfig FromRecord(const RecordVal* node_val);
/**
* Create configuration from JSON representation.
* @param json the JSON string to convert.
*/
static NodeConfig FromJSON(std::string_view json);
/**
* Convert this object into JSON respresentation.
* @return the JSON string representing the node config.
*/
std::string ToJSON() const;
/**
* Convert his object into script-layer record value.
* @return the script-layer record value representing the node config.
*/
IntrusivePtr<RecordVal> ToRecord() const;
/**
* The name of the supervised Zeek node. These are unique within
* a given supervised process tree and typically human-readable.
*/
std::string name;
/**
* The interface name from which the node should read/analyze packets.
*/
std::optional<std::string> interface;
/**
* The working directory that should be used by the node.
*/
std::optional<std::string> directory;
/**
* The filename/path to which the node's stdout will be redirected.
*/
std::optional<std::string> stdout_file;
/**
* The filename/path to which the node's stderr will be redirected.
*/
std::optional<std::string> stderr_file;
/**
* A cpu/core number to which the node will try to pin itself.
*/
std::optional<int> cpu_affinity;
/**
* Additional script filename/paths that the node should load.
*/
std::vector<std::string> scripts;
/**
* The Cluster Layout definition. Each node in the Cluster Framework
* knows about the full, static cluster topology to which it belongs.
* Entries in the map use node names for keys.
*/
std::map<std::string, ClusterEndpoint> cluster;
};
/**
* State which defines a Supervised node's understanding of itself.
*/
struct SupervisedNode {
/**
* Initialize the Supervised node within the Zeek Cluster Framework.
* This function populates the "Cluster::nodes" script-layer variable
* that otherwise is expected to be populated by a
* "cluster-layout.zeek" script in other context (e.g. ZeekCtl
* generates that cluster layout).
* @return true if the supervised node is using the Cluster Framework
* else false.
*/
bool InitCluster() const;
/**
* Initialize the Supervised node.
* @param options the Zeek options to extend/modify as appropriate
* for the node's configuration.
*/
void Init(zeek::Options* options) const;
/**
* The node's configuration options.
*/
NodeConfig config;
/**
* The process ID of the supervised node's parent process (i.e. the PID
* of the Stem process).
*/
pid_t parent_pid;
};
/**
* The state of a supervised node from the Supervisor's perspective.
*/
struct Node {
/**
* Convert the node into script-layer Supervisor::NodeStatus record
* representation.
*/
IntrusivePtr<RecordVal> ToRecord() const;
/**
* @return the name of the node.
*/
const std::string& Name() const
{ return config.name; }
/**
* Create a new node state from a given configuration.
* @param arg_config the configuration to use for the node.
*/
Node(NodeConfig arg_config) : config(std::move(arg_config))
{ }
/**
* The desired configuration for the node.
*/
NodeConfig config;
/**
* Process ID of the node (positive/non-zero are valid/live PIDs).
*/
pid_t pid = 0;
/**
* Whether the node is voluntarily marked for termination by the
* Supervisor.
*/
bool killed = false;
/**
* The last exit status of the node.
*/
int exit_status = 0;
/**
* The last signal which terminated the node.
*/
int signal_number = 0;
/**
* Number of process revival attempts made after the node first died
* prematurely.
*/
int revival_attempts = 0;
/**
* How many seconds to wait until the next revival attempt for the node.
*/
int revival_delay = 1;
/**
* The time at which the node's process was last spawned.
*/
std::chrono::time_point<std::chrono::steady_clock> spawn_time;
};
/**
* State used to initalialize the Stem process.
*/
struct StemState {
/**
* Bidirectional pipes that allow the Supervisor and Stem to talk.
*/
std::unique_ptr<bro::PipePair> pipe;
/**
* The Stem's parent process ID (i.e. PID of the Supervisor).
*/
pid_t parent_pid = 0;
/**
* The Stem's process ID.
*/
pid_t pid = 0;
};
/**
* Create and run the Stem process if necessary.
* @param supervisor_mode whether Zeek was invoked with the supervisor
* mode specified as command-line argument/option.
* @return state that defines the Stem process if called from the
* Supervisor process. The Stem process itself will not return from this,
* function but a node it spawns via fork() will return from it and
* information about it is available in ThisNode().
*/
static std::optional<StemState> CreateStem(bool supervisor_mode);
/**
* @return the state which describes what a supervised node should know
* about itself if this is a supervised process. If called from a process
* that is not supervised, this returns an "empty" object.
*/
static const std::optional<SupervisedNode>& ThisNode()
{ return supervised_node; }
using NodeMap = std::map<std::string, Node, std::less<>>;
/**
* Create a new Supervisor object.
* @param stem_state information about the Stem process that was already
* created via CreateStem()
*/
Supervisor(Config cfg, StemState stem_state);
/**
* Destruction also cleanly shuts down the entire supervised process tree.
*/
~Supervisor();
/**
* @return the process ID of the Stem.
*/
pid_t StemPID() const
{ return stem_pid; }
/**
* @return the state of currently supervised processes. The map uses
* node names for keys.
*/
const NodeMap& Nodes()
{ return nodes; }
/**
* Retrieve current status of a supervised node.
* @param node_name the name of the node for which to retrieve status
* or an empty string to mean "all nodes".
* @return script-layer Supervisor::Status record value describing the
* status of a node or set of nodes.
*/
RecordVal* Status(std::string_view node_name);
/**
* Create a new supervised node.
* @param node the script-layer Supervisor::NodeConfig value that
* describes the desired node configuration
* @return an empty string on success or description of the error/failure
*/
std::string Create(const RecordVal* node);
/**
* Create a new supervised node.
* @param node the desired node configuration
* @return an empty string on success or description of the error/failure
*/
std::string Create(const Supervisor::NodeConfig& node);
/**
* Destroys and removes a supervised node.
* @param node_name the name of the node to destroy or an empty string
* to mean "all nodes"
* @return true on success
*/
bool Destroy(std::string_view node_name);
/**
* Restart a supervised node process (by destroying and re-recreating).
* @param node_name the name of the node to restart or an empty string
* to mean "all nodes"
* @return true on success
*/
bool Restart(std::string_view node_name);
/**
* Not meant for public use. For use in a signal handler to tell the
* Supervisor a child process (i.e. the Stem) potentially died.
*/
void ObserveChildSignal(int signo);
private:
// IOSource interface overrides:
void GetFds(iosource::FD_Set* read, iosource::FD_Set* write,
iosource::FD_Set* except) override;
double NextTimestamp(double* local_network_time) override;
void Process() override;
size_t ProcessMessages();
void HandleChildSignal();
void ReapStem();
const char* Tag() override
{ return "zeek::Supervisor"; }
/**
* Run the Stem process. The Stem process will receive instructions from
* the Supervisor to manipulate the process hierarchy and it's in charge
* of directly monitoring for whether any nodes die premature and need
* to be revived.
* @param pipe bidirectional pipes that allow the Supervisor and Stem
* process to communicate.
* @param pid the Stem's parent process ID (i.e. the PID of the Supervisor)
* @return state which describes what a supervised node should know about
* itself. I.e. this function only returns from a fork()'d child process.
*/
static SupervisedNode RunStem(StemState stem_state);
static std::optional<SupervisedNode> supervised_node;
Config config;
pid_t stem_pid;
std::unique_ptr<bro::PipePair> stem_pipe;
int last_signal = -1;
bro::Flare signal_flare;
NodeMap nodes;
std::string msg_buffer;
};
/**
* A timer used by supervised processes to periodically check whether their
* parent (supervisor) process has died. If it has died, the supervised
* process self-terminates.
*/
class ParentProcessCheckTimer : public Timer {
public:
/**
* Create a timer to check for parent process death.
* @param t the time at which to trigger the timer's check.
* @param interval number of seconds to wait before checking again.
*/
ParentProcessCheckTimer(double t, double interval);
protected:
void Dispatch(double t, int is_expire) override;
double interval;
};
extern Supervisor* supervisor_mgr;
} // namespace zeek

112
src/supervisor/supervisor.bif Normal file
View file

@ -0,0 +1,112 @@
##! The BIFs that define the Zeek supervisor control interface.
%%{
#include "supervisor/Supervisor.h"
%%}
module Supervisor;
enum ClusterRole %{
NONE,
LOGGER,
MANAGER,
PROXY,
WORKER,
%}
type Supervisor::ClusterEndpoint: record;
type Supervisor::Status: record;
type Supervisor::NodeConfig: record;
type Supervisor::NodeStatus: record;
function Supervisor::__status%(node: string%): Supervisor::Status
%{
if ( ! zeek::supervisor_mgr )
{
builtin_error("supervisor mode not enabled");
return new RecordVal(BifType::Record::Supervisor::Status);
}
return zeek::supervisor_mgr->Status(node->CheckString());
%}
function Supervisor::__create%(node: Supervisor::NodeConfig%): string
%{
if ( ! zeek::supervisor_mgr )
{
builtin_error("supervisor mode not enabled");
return new StringVal("supervisor mode not enabled");
}
auto rval = zeek::supervisor_mgr->Create(node->AsRecordVal());
return new StringVal(rval);
%}
function Supervisor::__destroy%(node: string%): bool
%{
if ( ! zeek::supervisor_mgr )
{
builtin_error("supervisor mode not enabled");
return val_mgr->GetBool(false);
}
auto rval = zeek::supervisor_mgr->Destroy(node->CheckString());
return val_mgr->GetBool(rval);
%}
function Supervisor::__restart%(node: string%): bool
%{
if ( ! zeek::supervisor_mgr )
{
builtin_error("supervisor mode not enabled");
return val_mgr->GetBool(false);
}
auto rval = zeek::supervisor_mgr->Restart(node->CheckString());
return val_mgr->GetBool(rval);
%}
function Supervisor::__init_cluster%(%): bool
%{
if ( zeek::Supervisor::ThisNode() )
return val_mgr->GetBool(zeek::Supervisor::ThisNode()->InitCluster());
return val_mgr->GetBool(false);
%}
function Supervisor::__is_supervised%(%): bool
%{
return val_mgr->GetBool(zeek::Supervisor::ThisNode().has_value());
%}
function Supervisor::__node%(%): Supervisor::NodeConfig
%{
if ( ! zeek::Supervisor::ThisNode() )
{
builtin_error("not a supervised process");
auto rt = BifType::Record::Supervisor::NodeConfig;
auto rval = make_intrusive<RecordVal>(rt);
rval->Assign(rt->FieldOffset("name"), new StringVal("<invalid>"));
return rval.detach();
}
auto rval = zeek::Supervisor::ThisNode()->config.ToRecord();
return rval.detach();
%}
function Supervisor::__is_supervisor%(%): bool
%{
return val_mgr->GetBool(zeek::supervisor_mgr != nullptr);
%}
function Supervisor::__stem_pid%(%): int
%{
if ( zeek::supervisor_mgr )
return val_mgr->GetInt(zeek::supervisor_mgr->StemPID());
if ( zeek::Supervisor::ThisNode() )
return val_mgr->GetInt(zeek::Supervisor::ThisNode()->parent_pid);
builtin_error("supervisor mode not enabled and not a supervised node");
return val_mgr->GetInt(-1);
%}

24
src/threading/BasicThread.cc
View file

@ -1,18 +1,11 @@
#include <signal.h>
#include <pthread.h>
#include "zeek-config.h"
#include "BasicThread.h"
#include "Manager.h"
#include "pthread.h"
#ifdef HAVE_LINUX
#include <sys/prctl.h>
#endif
#ifdef __FreeBSD__
#include <pthread_np.h>
#endif
#include "util.h"
using namespace threading;
@ -54,18 +47,7 @@ void BasicThread::SetName(const char* arg_name)
void BasicThread::SetOSName(const char* arg_name)
{
static_assert(std::is_same<std::thread::native_handle_type, pthread_t>::value, "libstdc++ doesn't use pthread_t");
#ifdef HAVE_LINUX
prctl(PR_SET_NAME, arg_name, 0, 0, 0);
#endif
#ifdef __APPLE__
pthread_setname_np(arg_name);
#endif
#ifdef __FreeBSD__
pthread_set_name_np(thread.native_handle(), arg_name);
#endif
zeek::set_thread_name(arg_name, thread.native_handle());
}
const char* BasicThread::Fmt(const char* format, ...)

169
src/util.cc
View file

@ -55,6 +55,43 @@
#include "3rdparty/doctest.h"
#ifdef __linux__
#if __has_include(<sys/random.h>)
#define HAVE_GETRANDOM
#include <sys/random.h>
#endif
#endif
static bool starts_with(const std::string_view& s, const std::string& beginning)
{
if ( beginning.size() > s.size() )
return false;
return std::equal(beginning.begin(), beginning.end(), s.begin());
}
TEST_CASE("util starts_with")
{
CHECK(starts_with("abcde", "ab") == true);
CHECK(starts_with("abcde", "de") == false);
CHECK(starts_with("abcde", "abcedf") == false);
}
static bool ends_with(const std::string_view& s, const std::string& ending)
{
if ( ending.size() > s.size() )
return false;
return std::equal(ending.rbegin(), ending.rend(), s.rbegin());
}
TEST_CASE("util ends_with")
{
CHECK(ends_with("abcde", "de") == true);
CHECK(ends_with("abcde", "fg") == false);
CHECK(ends_with("abcde", "abcedf") == false);
}
TEST_CASE("util extract_ip")
{
CHECK(extract_ip("[1.2.3.4]") == "1.2.3.4");
@ -605,7 +642,7 @@ TEST_CASE("util uitoa_n")
char* uitoa_n(uint64_t value, char* str, int n, int base, const char* prefix)
{
static char dig[] = "0123456789abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ";
static constexpr char dig[] = "0123456789abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ";
assert(n);
@ -616,7 +653,7 @@ char* uitoa_n(uint64_t value, char* str, int n, int base, const char* prefix)
if ( prefix )
{
strncpy(str, prefix, n);
strncpy(str, prefix, n-1);
str[n-1] = '\0';
i += strlen(prefix);
}
@ -1035,6 +1072,14 @@ void init_random_seed(const char* read_file, const char* write_file)
seeds_done = true;
}
#ifdef HAVE_GETRANDOM
if ( ! seeds_done )
{
ssize_t nbytes = getrandom(buf, sizeof(buf), 0);
seeds_done = nbytes == ssize_t(sizeof(buf));
}
#endif
if ( ! seeds_done )
{
// Gather up some entropy.
@ -1216,7 +1261,7 @@ string bro_prefixes()
{
string rval;
for ( const auto& prefix : prefixes )
for ( const auto& prefix : zeek_script_prefixes )
{
if ( ! rval.empty() )
rval.append(":");
@ -1234,6 +1279,15 @@ TEST_CASE("util is_package_loader")
const array<string, 2> script_extensions = {".zeek", ".bro"};
void warn_if_legacy_script(const std::string_view& filename)
{
if ( ends_with(filename, ".bro") )
{
std::string x(filename);
reporter->Warning("Loading script '%s' with legacy extension, support for '.bro' will be removed in Zeek v4.1", x.c_str());
}
}
bool is_package_loader(const string& path)
{
string filename(std::move(SafeBasename(path).result));
@ -1241,8 +1295,11 @@ bool is_package_loader(const string& path)
for ( const string& ext : script_extensions )
{
if ( filename == "__load__" + ext )
{
warn_if_legacy_script(filename);
return true;
}
}
return false;
}
@ -1279,6 +1336,7 @@ FILE* open_package(string& path, const string& mode)
string p = path + ext;
if ( can_read(p) )
{
warn_if_legacy_script(path);
path.append(ext);
return open_file(path, mode);
}
@ -1444,17 +1502,22 @@ TEST_CASE("util tokenize_string")
}
vector<string>* tokenize_string(string input, const string& delim,
vector<string>* rval)
vector<string>* rval, int limit)
{
if ( ! rval )
rval = new vector<string>();
size_t n;
auto found = 0;
while ( (n = input.find(delim)) != string::npos )
{
++found;
rval->push_back(input.substr(0, n));
input.erase(0, n + 1);
if ( limit && found == limit )
break;
}
rval->push_back(input);
@ -1467,6 +1530,25 @@ TEST_CASE("util normalize_path")
CHECK(normalize_path("/1/./2/3") == "/1/2/3");
CHECK(normalize_path("/1/2/../3") == "/1/3");
CHECK(normalize_path("1/2/3/") == "1/2/3");
CHECK(normalize_path("1/2//3///") == "1/2/3");
CHECK(normalize_path("~/zeek/testing") == "~/zeek/testing");
CHECK(normalize_path("~jon/zeek/testing") == "~jon/zeek/testing");
CHECK(normalize_path("~jon/./zeek/testing") == "~jon/zeek/testing");
CHECK(normalize_path("~/zeek/testing/../././.") == "~/zeek");
CHECK(normalize_path("./zeek") == "./zeek");
CHECK(normalize_path("../zeek") == "../zeek");
CHECK(normalize_path("../zeek/testing/..") == "../zeek");
CHECK(normalize_path("./zeek/..") == ".");
CHECK(normalize_path("./zeek/../..") == "..");
CHECK(normalize_path("./zeek/../../..") == "../..");
CHECK(normalize_path("./..") == "..");
CHECK(normalize_path("../..") == "../..");
CHECK(normalize_path("/..") == "/..");
CHECK(normalize_path("~/..") == "~/..");
CHECK(normalize_path("/../..") == "/../..");
CHECK(normalize_path("~/../..") == "~/../..");
CHECK(normalize_path("zeek/..") == "");
CHECK(normalize_path("zeek/../..") == "..");
}
string normalize_path(const string& path)
@ -1488,12 +1570,32 @@ string normalize_path(const string& path)
if ( *it == "." && it != components.begin() )
final_components.pop_back();
else if ( *it == ".." && final_components[0] != ".." )
else if ( *it == ".." )
{
auto cur_idx = final_components.size() - 1;
if ( cur_idx != 0 )
{
auto last_idx = cur_idx - 1;
auto& last_component = final_components[last_idx];
if ( last_component == "/" || last_component == "~" ||
last_component == ".." )
continue;
if ( last_component == "." )
{
last_component = "..";
final_components.pop_back();
}
else
{
final_components.pop_back();
final_components.pop_back();
}
}
}
}
for ( it = final_components.begin(); it != final_components.end(); ++it )
{
@ -1589,21 +1691,6 @@ string find_file(const string& filename, const string& path_set,
return string();
}
static bool ends_with(const std::string& s, const std::string& ending)
{
if ( ending.size() > s.size() )
return false;
return std::equal(ending.rbegin(), ending.rend(), s.rbegin());
}
TEST_CASE("util ends_with")
{
CHECK(ends_with("abcde", "de") == true);
CHECK(ends_with("abcde", "fg") == false);
CHECK(ends_with("abcde", "abcedf") == false);
}
string find_script_file(const string& filename, const string& path_set)
{
vector<string> paths;
@ -1616,11 +1703,16 @@ string find_script_file(const string& filename, const string& path_set)
string f = find_file_in_path(filename, paths[n], ext);
if ( ! f.empty() )
{
warn_if_legacy_script(f);
return f;
}
}
if ( ends_with(filename, ".bro") )
{
warn_if_legacy_script(filename);
// We were looking for a file explicitly ending in .bro and didn't
// find it, so fall back to one ending in .zeek, if it exists.
auto fallback = string(filename.data(), filename.size() - 4) + ".zeek";
@ -1749,7 +1841,7 @@ void terminate_processing()
}
extern const char* proc_status_file;
void _set_processing_status(const char* status)
void set_processing_status(const char* status, const char* reason)
{
if ( ! proc_status_file )
return;
@ -1776,20 +1868,27 @@ void _set_processing_status(const char* status)
return;
}
int len = strlen(status);
auto write_str = [](int fd, const char* s)
{
int len = strlen(s);
while ( len )
{
int n = write(fd, status, len);
int n = write(fd, s, len);
if ( n < 0 && errno != EINTR && errno != EAGAIN )
// Ignore errors, as they're too difficult to
// safely report here.
break;
status += n;
s += n;
len -= n;
}
};
write_str(fd, status);
write_str(fd, " [");
write_str(fd, reason);
write_str(fd, "]\n");
safe_close(fd);
errno = old_errno;
@ -2171,7 +2270,12 @@ char* zeekenv(const char* name)
if ( it == legacy_vars.end() )
return rval;
return getenv(it->second);
auto val = getenv(it->second);
if ( val && starts_with(it->second, "BRO_") )
reporter->Warning("Using legacy environment variable %s, support will be removed in Zeek v4.1; use %s instead", it->second, name);
return val;
}
static string json_escape_byte(char c)
@ -2299,3 +2403,18 @@ string json_escape_utf8(const string& val)
return result;
}
void zeek::set_thread_name(const char* name, pthread_t tid)
{
#ifdef HAVE_LINUX
prctl(PR_SET_NAME, name, 0, 0, 0);
#endif
#ifdef __APPLE__
pthread_setname_np(name);
#endif
#ifdef __FreeBSD__
pthread_set_name_np(tid, name);
#endif
}

31
src/util.h
View file

@ -60,6 +60,15 @@ extern HeapLeakChecker* heap_checker;
#endif
#include <stdint.h>
#include <pthread.h>
#ifdef HAVE_LINUX
#include <sys/prctl.h>
#endif
#ifdef __FreeBSD__
#include <pthread_np.h>
#endif
ZEEK_DEPRECATED("Remove in v4.1. Use uint64_t instead.")
typedef uint64_t uint64;
@ -138,7 +147,7 @@ inline std::string get_escaped_string(const std::string& str, bool escape_all)
std::vector<std::string>* tokenize_string(std::string input,
const std::string& delim,
std::vector<std::string>* rval = 0);
std::vector<std::string>* rval = 0, int limit = 0);
extern char* copy_string(const char* s);
extern int streq(const char* s1, const char* s2);
@ -265,6 +274,9 @@ extern std::string bro_prefixes();
extern const std::array<std::string, 2> script_extensions;
/** Prints a warning if the filename ends in .bro. */
void warn_if_legacy_script(const std::string_view& filename);
bool is_package_loader(const std::string& path);
extern void add_to_bro_path(const std::string& dir);
@ -402,9 +414,7 @@ void terminate_processing();
// Sets the current status of the Bro process to the given string.
// If the option --status-file has been set, this is written into
// the the corresponding file. Otherwise, the function is a no-op.
#define set_processing_status(status, location) \
_set_processing_status(status " [" location "]\n");
void _set_processing_status(const char* status);
void set_processing_status(const char* status, const char* reason);
// Current timestamp, from a networking perspective, not a wall-clock
// perspective. In particular, if we're reading from a savefile this
@ -505,7 +515,7 @@ inline void* safe_malloc(size_t size)
inline char* safe_strncpy(char* dest, const char* src, size_t n)
{
char* result = strncpy(dest, src, n);
char* result = strncpy(dest, src, n-1);
dest[n-1] = '\0';
return result;
}
@ -572,3 +582,14 @@ char* zeekenv(const char* name);
* @return the escaped string
*/
std::string json_escape_utf8(const std::string& val);
namespace zeek {
/**
* Set the process/thread name. May not be supported on all OSs.
* @param name new name for the process/thread. OS limitations typically
* truncate the name to 15 bytes maximum.
* @param tid handle of thread whose name shall change
*/
void set_thread_name(const char* name, pthread_t tid = pthread_self());
} // namespace zeek

57
src/zeek-affinity.cc Normal file
View file

@ -0,0 +1,57 @@
// See the file "COPYING" in the main distribution directory for copyright.
// This is all in its own source file primarily because the Linux
// implementation uses the _GNU_SOURCE feature test macro which must be
// defined before including any header file and lumping this together with
// other util functions makes that requirement less apparent and less
// self-contained.
#if defined(__linux__)
#if !defined(_GNU_SOURCE)
#define _GNU_SOURCE
#endif
#include <sched.h>
namespace zeek {
bool set_affinity(int core_number)
{
cpu_set_t cpus;
CPU_ZERO(&cpus);
CPU_SET(core_number, &cpus);
auto res = sched_setaffinity(0, sizeof(cpus), &cpus);
return res == 0;
}
} // namespace zeek
#elif defined(__FreeBSD__)
#include <sys/param.h>
#include <sys/cpuset.h>
namespace zeek {
bool set_affinity(int core_number)
{
cpuset_t cpus;
CPU_ZERO(&cpus);
CPU_SET(core_number, &cpus);
auto res = cpuset_setaffinity(CPU_LEVEL_WHICH, CPU_WHICH_PID, -1,
sizeof(cpus), &cpus);
return res == 0;
}
} // namespace zeek
#else
#include <cerrno>
namespace zeek {
bool set_affinity(int core_number)
{
errno = ENOTSUP;
return false;
}
} // namespace zeek
#endif

17
src/zeek-affinity.h Normal file
View file

@ -0,0 +1,17 @@
// See the file "COPYING" in the main distribution directory for copyright.
#pragma once
namespace zeek {
/**
* Set the process affinity to a given CPU. Currently only supported on
* Linux and FreeBSD.
* @param core_number the core to which this process should set its affinity.
* Cores are typically numbered 0..N.
* @return true if the affinity is successfully set and false if not with
* errno additionally being set to indicate the reason.
*/
bool set_affinity(int core_number);
} // namespace zeek

20
src/zeek.bif
View file

@ -1820,15 +1820,6 @@ function getpid%(%) : count
extern const char* zeek_version();
%%}
## Returns the Zeek version string. This function is deprecated, use
## :zeek:see:`zeek_version` instead.
##
## Returns: Zeek's version, e.g., 2.0-beta-47-debug.
function bro_version%(%): string &deprecated="Remove in v3.1: use zeek_version"
%{
return new StringVal(zeek_version());
%}
## Returns the Zeek version string.
##
## Returns: Zeek's version, e.g., 2.0-beta-47-debug.
@ -2130,17 +2121,6 @@ function dump_rule_stats%(f: file%): bool
return val_mgr->GetBool(1);
%}
## Checks if Zeek is terminating. This function is deprecated, use
## :zeek:see:`zeek_is_terminating` instead.
##
## Returns: True if Zeek is in the process of shutting down.
##
## .. zeek:see:: terminate
function bro_is_terminating%(%): bool &deprecated="Remove in v3.1: use zeek_is_terminating"
%{
return val_mgr->GetBool(terminating);
%}
## Checks if Zeek is terminating.
##
## Returns: True if Zeek is in the process of shutting down.

4
testing/btest/Baseline/bifs.str_shell_escape/out
View file

@ -1,4 +0,0 @@
24
echo ${TEST} > "my file"
27
echo \${TEST} > \"my file\"

2
testing/btest/Baseline/broker.ssl_auth_failure/send.send.out
View file

@ -1 +1 @@
sender error: code=Broker::PEER_UNAVAILABLE msg=(invalid-node, *<endpoint addr:port>, "remote endpoint unavailable")
sender error: code=Broker::PEER_UNAVAILABLE msg=(invalid-node, *<endpoint addr:port>, "remote endpoint unavailable: system_error(disconnect_during_handshake)")

7
testing/btest/Baseline/core.check-unused-event-handlers/.stderr
View file

@ -1,3 +1,8 @@
warning in <params>, line 1: event handler never invoked: this_is_never_used
warning in <params>, line 1: event handler never invoked: InputConfig::new_value
warning in <params>, line 1: event handler never invoked: InputRaw::process_finished
warning in <params>, line 1: event handler never invoked: SupervisorControl::create_request
warning in <params>, line 1: event handler never invoked: SupervisorControl::destroy_request
warning in <params>, line 1: event handler never invoked: SupervisorControl::restart_request
warning in <params>, line 1: event handler never invoked: SupervisorControl::status_request
warning in <params>, line 1: event handler never invoked: SupervisorControl::stop_request
warning in <params>, line 1: event handler never invoked: this_is_never_used

2
testing/btest/Baseline/core.load-explicit-bro-suffix-fallback/out
View file

@ -1 +1,3 @@
warning in /home/robin/bro/master/testing/btest/.tmp/core.load-explicit-bro-suffix-fallback/load-explicit-bro-suffix-fallback.zeek, line 5: Loading script 'foo.bro' with legacy extension, support for '.bro' will be removed in Zeek v4.1
warning in /home/robin/bro/master/testing/btest/.tmp/core.load-explicit-bro-suffix-fallback/load-explicit-bro-suffix-fallback.zeek, line 5: Loading script 'foo.bro' with legacy extension, support for '.bro' will be removed in Zeek v4.1
loaded foo.zeek

9
testing/btest/Baseline/coverage.bare-load-baseline/canonified_loaded_scripts.log
View file

@ -3,7 +3,7 @@
#empty_field (empty)
#unset_field -
#path loaded_scripts
#open 2019-07-29-19-05-26
#open 2019-10-15-01-48-24
#fields name
#types string
scripts/base/init-bare.zeek
@ -14,6 +14,8 @@ scripts/base/init-bare.zeek
build/scripts/base/bif/reporter.bif.zeek
build/scripts/base/bif/strings.bif.zeek
build/scripts/base/bif/option.bif.zeek
scripts/base/frameworks/supervisor/api.zeek
build/scripts/base/bif/supervisor.bif.zeek
build/scripts/base/bif/plugins/Zeek_SNMP.types.bif.zeek
build/scripts/base/bif/plugins/Zeek_KRB.types.bif.zeek
build/scripts/base/bif/event.bif.zeek
@ -35,6 +37,9 @@ scripts/base/init-frameworks-and-bifs.zeek
build/scripts/base/bif/data.bif.zeek
build/scripts/base/bif/store.bif.zeek
scripts/base/frameworks/broker/log.zeek
scripts/base/frameworks/supervisor/__load__.zeek
scripts/base/frameworks/supervisor/control.zeek
scripts/base/frameworks/supervisor/main.zeek
scripts/base/frameworks/input/__load__.zeek
scripts/base/frameworks/input/main.zeek
build/scripts/base/bif/input.bif.zeek
@ -182,4 +187,4 @@ scripts/base/init-frameworks-and-bifs.zeek
build/scripts/base/bif/plugins/Zeek_SQLiteWriter.sqlite.bif.zeek
scripts/policy/misc/loaded-scripts.zeek
scripts/base/utils/paths.zeek
#close 2019-07-29-19-05-26
#close 2019-10-15-01-48-24

1
testing/btest/Baseline/coverage.bare-mode-errors/errors
View file

@ -1 +0,0 @@
warning in /Users/tim/Desktop/projects/zeek/testing/btest/../../scripts//base/utils/json.zeek, line 2: deprecated script loaded from command line arguments ="Remove in 3.1. to_json is now always available as a built-in function."

9
testing/btest/Baseline/coverage.default-load-baseline/canonified_loaded_scripts.log
View file

@ -3,7 +3,7 @@
#empty_field (empty)
#unset_field -
#path loaded_scripts
#open 2019-08-06-00-02-39
#open 2019-10-15-01-48-24
#fields name
#types string
scripts/base/init-bare.zeek
@ -14,6 +14,8 @@ scripts/base/init-bare.zeek
build/scripts/base/bif/reporter.bif.zeek
build/scripts/base/bif/strings.bif.zeek
build/scripts/base/bif/option.bif.zeek
scripts/base/frameworks/supervisor/api.zeek
build/scripts/base/bif/supervisor.bif.zeek
build/scripts/base/bif/plugins/Zeek_SNMP.types.bif.zeek
build/scripts/base/bif/plugins/Zeek_KRB.types.bif.zeek
build/scripts/base/bif/event.bif.zeek
@ -35,6 +37,9 @@ scripts/base/init-frameworks-and-bifs.zeek
build/scripts/base/bif/data.bif.zeek
build/scripts/base/bif/store.bif.zeek
scripts/base/frameworks/broker/log.zeek
scripts/base/frameworks/supervisor/__load__.zeek
scripts/base/frameworks/supervisor/control.zeek
scripts/base/frameworks/supervisor/main.zeek
scripts/base/frameworks/input/__load__.zeek
scripts/base/frameworks/input/main.zeek
build/scripts/base/bif/input.bif.zeek
@ -376,4 +381,4 @@ scripts/base/init-default.zeek
scripts/base/misc/find-filtered-trace.zeek
scripts/base/misc/version.zeek
scripts/policy/misc/loaded-scripts.zeek
#close 2019-08-06-00-02-39
#close 2019-10-15-01-48-25

1
testing/btest/Baseline/coverage.init-default/missing_loads
View file

@ -8,4 +8,3 @@
-./frameworks/openflow/cluster.zeek
-./frameworks/packet-filter/cluster.zeek
-./frameworks/sumstats/cluster.zeek
-./utils/json.zeek

2
testing/btest/Baseline/language.bro_init/out Normal file
View file

@ -0,0 +1,2 @@
error in /Users/jsiwek/pro/zeek/zeek/testing/btest/.tmp/language.bro_init/bro_init.zeek, line 5: event bro_init() is no longer available, use zeek_init() instead
error in /Users/jsiwek/pro/zeek/zeek/testing/btest/.tmp/language.bro_init/bro_init.zeek, line 10: event bro_done() is no longer available, use zeek_done() instead

1
testing/btest/Baseline/language.bro_script_loaded/out Normal file
View file

@ -0,0 +1 @@
error in /Users/jsiwek/pro/zeek/zeek/testing/btest/.tmp/language.bro_script_loaded/bro_script_loaded.zeek, line 5: event bro_script_loaded() is no longer available, use zeek_script_loaded() instead

6
testing/btest/Baseline/language.zeek_init/out
View file

@ -1,8 +1,6 @@
zeek_init at priority 10!
bro_init at priority 5!
zeek_init at priority 0!
bro_init at priority -10!
zeek_init at priority -10!
zeek_done at priority 10!
bro_done at priority 5!
zeek_done at priority 0!
bro_done at priority -10!
zeek_done at priority -10!

2
testing/btest/Baseline/language.zeek_script_loaded/out
View file

@ -1,4 +1,2 @@
zeek_script_loaded priority 10
bro_script_loaded priority 5
zeek_script_loaded priority 0
bro_script_loaded priority -10

38
testing/btest/Baseline/plugins.hooks/output
View file

@ -157,6 +157,8 @@
0.000000 MetaHookPost CallFunction(Analyzer::register_for_ports, <frame>, (Analyzer::ANALYZER_TEREDO, {3544/udp})) -> <no result>
0.000000 MetaHookPost CallFunction(Analyzer::register_for_ports, <frame>, (Analyzer::ANALYZER_VXLAN, {4789/udp})) -> <no result>
0.000000 MetaHookPost CallFunction(Analyzer::register_for_ports, <frame>, (Analyzer::ANALYZER_XMPP, {5222<...>/tcp})) -> <no result>
0.000000 MetaHookPost CallFunction(Broker::__subscribe, <frame>, (zeek/supervisor)) -> <no result>
0.000000 MetaHookPost CallFunction(Broker::subscribe, <frame>, (zeek/supervisor)) -> <no result>
0.000000 MetaHookPost CallFunction(Cluster::is_enabled, <frame>, ()) -> <no result>
0.000000 MetaHookPost CallFunction(Cluster::is_enabled, <null>, ()) -> <no result>
0.000000 MetaHookPost CallFunction(Cluster::local_node_type, <null>, ()) -> <no result>
@ -274,7 +276,7 @@
0.000000 MetaHookPost CallFunction(Log::__create_stream, <frame>, (Weird::LOG, [columns=Weird::Info, ev=Weird::log_weird, path=weird])) -> <no result>
0.000000 MetaHookPost CallFunction(Log::__create_stream, <frame>, (X509::LOG, [columns=X509::Info, ev=X509::log_x509, path=x509])) -> <no result>
0.000000 MetaHookPost CallFunction(Log::__create_stream, <frame>, (mysql::LOG, [columns=MySQL::Info, ev=MySQL::log_mysql, path=mysql])) -> <no result>
0.000000 MetaHookPost CallFunction(Log::__write, <frame>, (PacketFilter::LOG, [ts=1573527265.694831, node=zeek, filter=ip or not ip, init=T, success=T])) -> <no result>
0.000000 MetaHookPost CallFunction(Log::__write, <frame>, (PacketFilter::LOG, [ts=1579727603.636084, node=zeek, filter=ip or not ip, init=T, success=T])) -> <no result>
0.000000 MetaHookPost CallFunction(Log::add_default_filter, <frame>, (Broker::LOG)) -> <no result>
0.000000 MetaHookPost CallFunction(Log::add_default_filter, <frame>, (Cluster::LOG)) -> <no result>
0.000000 MetaHookPost CallFunction(Log::add_default_filter, <frame>, (Config::LOG)) -> <no result>
@ -455,7 +457,7 @@
0.000000 MetaHookPost CallFunction(Log::create_stream, <frame>, (Weird::LOG, [columns=Weird::Info, ev=Weird::log_weird, path=weird])) -> <no result>
0.000000 MetaHookPost CallFunction(Log::create_stream, <frame>, (X509::LOG, [columns=X509::Info, ev=X509::log_x509, path=x509])) -> <no result>
0.000000 MetaHookPost CallFunction(Log::create_stream, <frame>, (mysql::LOG, [columns=MySQL::Info, ev=MySQL::log_mysql, path=mysql])) -> <no result>
0.000000 MetaHookPost CallFunction(Log::write, <frame>, (PacketFilter::LOG, [ts=1573527265.694831, node=zeek, filter=ip or not ip, init=T, success=T])) -> <no result>
0.000000 MetaHookPost CallFunction(Log::write, <frame>, (PacketFilter::LOG, [ts=1579727603.636084, node=zeek, filter=ip or not ip, init=T, success=T])) -> <no result>
0.000000 MetaHookPost CallFunction(NetControl::check_plugins, <frame>, ()) -> <no result>
0.000000 MetaHookPost CallFunction(NetControl::init, <null>, ()) -> <no result>
0.000000 MetaHookPost CallFunction(Notice::want_pp, <frame>, ()) -> <no result>
@ -694,6 +696,7 @@
0.000000 MetaHookPost LoadFile(0, .<...>/add-geodata.zeek) -> -1
0.000000 MetaHookPost LoadFile(0, .<...>/addrs.zeek) -> -1
0.000000 MetaHookPost LoadFile(0, .<...>/analyzer.bif.zeek) -> -1
0.000000 MetaHookPost LoadFile(0, .<...>/api.zeek) -> -1
0.000000 MetaHookPost LoadFile(0, .<...>/ascii.zeek) -> -1
0.000000 MetaHookPost LoadFile(0, .<...>/average.zeek) -> -1
0.000000 MetaHookPost LoadFile(0, .<...>/benchmark.zeek) -> -1
@ -708,6 +711,7 @@
0.000000 MetaHookPost LoadFile(0, .<...>/const.bif.zeek) -> -1
0.000000 MetaHookPost LoadFile(0, .<...>/consts.zeek) -> -1
0.000000 MetaHookPost LoadFile(0, .<...>/contents.zeek) -> -1
0.000000 MetaHookPost LoadFile(0, .<...>/control.zeek) -> -1
0.000000 MetaHookPost LoadFile(0, .<...>/ct-list.zeek) -> -1
0.000000 MetaHookPost LoadFile(0, .<...>/data.bif.zeek) -> -1
0.000000 MetaHookPost LoadFile(0, .<...>/dcc-send.zeek) -> -1
@ -767,6 +771,7 @@
0.000000 MetaHookPost LoadFile(0, .<...>/store.zeek) -> -1
0.000000 MetaHookPost LoadFile(0, .<...>/strings.bif.zeek) -> -1
0.000000 MetaHookPost LoadFile(0, .<...>/sum.zeek) -> -1
0.000000 MetaHookPost LoadFile(0, .<...>/supervisor.bif.zeek) -> -1
0.000000 MetaHookPost LoadFile(0, .<...>/thresholds.zeek) -> -1
0.000000 MetaHookPost LoadFile(0, .<...>/top-k.bif.zeek) -> -1
0.000000 MetaHookPost LoadFile(0, .<...>/topk.zeek) -> -1
@ -788,6 +793,7 @@
0.000000 MetaHookPost LoadFile(0, base<...>/addrs.zeek) -> -1
0.000000 MetaHookPost LoadFile(0, base<...>/analyzer) -> -1
0.000000 MetaHookPost LoadFile(0, base<...>/analyzer.bif.zeek) -> -1
0.000000 MetaHookPost LoadFile(0, base<...>/api.zeek) -> -1
0.000000 MetaHookPost LoadFile(0, base<...>/bif) -> -1
0.000000 MetaHookPost LoadFile(0, base<...>/broker) -> -1
0.000000 MetaHookPost LoadFile(0, base<...>/cluster) -> -1
@ -868,6 +874,8 @@
0.000000 MetaHookPost LoadFile(0, base<...>/strings.bif.zeek) -> -1
0.000000 MetaHookPost LoadFile(0, base<...>/strings.zeek) -> -1
0.000000 MetaHookPost LoadFile(0, base<...>/sumstats) -> -1
0.000000 MetaHookPost LoadFile(0, base<...>/supervisor) -> -1
0.000000 MetaHookPost LoadFile(0, base<...>/supervisor.bif.zeek) -> -1
0.000000 MetaHookPost LoadFile(0, base<...>/syslog) -> -1
0.000000 MetaHookPost LoadFile(0, base<...>/thresholds.zeek) -> -1
0.000000 MetaHookPost LoadFile(0, base<...>/time.zeek) -> -1
@ -1053,6 +1061,8 @@
0.000000 MetaHookPre CallFunction(Analyzer::register_for_ports, <frame>, (Analyzer::ANALYZER_TEREDO, {3544/udp}))
0.000000 MetaHookPre CallFunction(Analyzer::register_for_ports, <frame>, (Analyzer::ANALYZER_VXLAN, {4789/udp}))
0.000000 MetaHookPre CallFunction(Analyzer::register_for_ports, <frame>, (Analyzer::ANALYZER_XMPP, {5222<...>/tcp}))
0.000000 MetaHookPre CallFunction(Broker::__subscribe, <frame>, (zeek/supervisor))
0.000000 MetaHookPre CallFunction(Broker::subscribe, <frame>, (zeek/supervisor))
0.000000 MetaHookPre CallFunction(Cluster::is_enabled, <frame>, ())
0.000000 MetaHookPre CallFunction(Cluster::is_enabled, <null>, ())
0.000000 MetaHookPre CallFunction(Cluster::local_node_type, <null>, ())
@ -1170,7 +1180,7 @@
0.000000 MetaHookPre CallFunction(Log::__create_stream, <frame>, (Weird::LOG, [columns=Weird::Info, ev=Weird::log_weird, path=weird]))
0.000000 MetaHookPre CallFunction(Log::__create_stream, <frame>, (X509::LOG, [columns=X509::Info, ev=X509::log_x509, path=x509]))
0.000000 MetaHookPre CallFunction(Log::__create_stream, <frame>, (mysql::LOG, [columns=MySQL::Info, ev=MySQL::log_mysql, path=mysql]))
0.000000 MetaHookPre CallFunction(Log::__write, <frame>, (PacketFilter::LOG, [ts=1573527265.694831, node=zeek, filter=ip or not ip, init=T, success=T]))
0.000000 MetaHookPre CallFunction(Log::__write, <frame>, (PacketFilter::LOG, [ts=1579727603.636084, node=zeek, filter=ip or not ip, init=T, success=T]))
0.000000 MetaHookPre CallFunction(Log::add_default_filter, <frame>, (Broker::LOG))
0.000000 MetaHookPre CallFunction(Log::add_default_filter, <frame>, (Cluster::LOG))
0.000000 MetaHookPre CallFunction(Log::add_default_filter, <frame>, (Config::LOG))
@ -1351,7 +1361,7 @@
0.000000 MetaHookPre CallFunction(Log::create_stream, <frame>, (Weird::LOG, [columns=Weird::Info, ev=Weird::log_weird, path=weird]))
0.000000 MetaHookPre CallFunction(Log::create_stream, <frame>, (X509::LOG, [columns=X509::Info, ev=X509::log_x509, path=x509]))
0.000000 MetaHookPre CallFunction(Log::create_stream, <frame>, (mysql::LOG, [columns=MySQL::Info, ev=MySQL::log_mysql, path=mysql]))
0.000000 MetaHookPre CallFunction(Log::write, <frame>, (PacketFilter::LOG, [ts=1573527265.694831, node=zeek, filter=ip or not ip, init=T, success=T]))
0.000000 MetaHookPre CallFunction(Log::write, <frame>, (PacketFilter::LOG, [ts=1579727603.636084, node=zeek, filter=ip or not ip, init=T, success=T]))
0.000000 MetaHookPre CallFunction(NetControl::check_plugins, <frame>, ())
0.000000 MetaHookPre CallFunction(NetControl::init, <null>, ())
0.000000 MetaHookPre CallFunction(Notice::want_pp, <frame>, ())
@ -1590,6 +1600,7 @@
0.000000 MetaHookPre LoadFile(0, .<...>/add-geodata.zeek)
0.000000 MetaHookPre LoadFile(0, .<...>/addrs.zeek)
0.000000 MetaHookPre LoadFile(0, .<...>/analyzer.bif.zeek)
0.000000 MetaHookPre LoadFile(0, .<...>/api.zeek)
0.000000 MetaHookPre LoadFile(0, .<...>/ascii.zeek)
0.000000 MetaHookPre LoadFile(0, .<...>/average.zeek)
0.000000 MetaHookPre LoadFile(0, .<...>/benchmark.zeek)
@ -1604,6 +1615,7 @@
0.000000 MetaHookPre LoadFile(0, .<...>/const.bif.zeek)
0.000000 MetaHookPre LoadFile(0, .<...>/consts.zeek)
0.000000 MetaHookPre LoadFile(0, .<...>/contents.zeek)
0.000000 MetaHookPre LoadFile(0, .<...>/control.zeek)
0.000000 MetaHookPre LoadFile(0, .<...>/ct-list.zeek)
0.000000 MetaHookPre LoadFile(0, .<...>/data.bif.zeek)
0.000000 MetaHookPre LoadFile(0, .<...>/dcc-send.zeek)
@ -1663,6 +1675,7 @@
0.000000 MetaHookPre LoadFile(0, .<...>/store.zeek)
0.000000 MetaHookPre LoadFile(0, .<...>/strings.bif.zeek)
0.000000 MetaHookPre LoadFile(0, .<...>/sum.zeek)
0.000000 MetaHookPre LoadFile(0, .<...>/supervisor.bif.zeek)
0.000000 MetaHookPre LoadFile(0, .<...>/thresholds.zeek)
0.000000 MetaHookPre LoadFile(0, .<...>/top-k.bif.zeek)
0.000000 MetaHookPre LoadFile(0, .<...>/topk.zeek)
@ -1684,6 +1697,7 @@
0.000000 MetaHookPre LoadFile(0, base<...>/addrs.zeek)
0.000000 MetaHookPre LoadFile(0, base<...>/analyzer)
0.000000 MetaHookPre LoadFile(0, base<...>/analyzer.bif.zeek)
0.000000 MetaHookPre LoadFile(0, base<...>/api.zeek)
0.000000 MetaHookPre LoadFile(0, base<...>/bif)
0.000000 MetaHookPre LoadFile(0, base<...>/broker)
0.000000 MetaHookPre LoadFile(0, base<...>/cluster)
@ -1764,6 +1778,8 @@
0.000000 MetaHookPre LoadFile(0, base<...>/strings.bif.zeek)
0.000000 MetaHookPre LoadFile(0, base<...>/strings.zeek)
0.000000 MetaHookPre LoadFile(0, base<...>/sumstats)
0.000000 MetaHookPre LoadFile(0, base<...>/supervisor)
0.000000 MetaHookPre LoadFile(0, base<...>/supervisor.bif.zeek)
0.000000 MetaHookPre LoadFile(0, base<...>/syslog)
0.000000 MetaHookPre LoadFile(0, base<...>/thresholds.zeek)
0.000000 MetaHookPre LoadFile(0, base<...>/time.zeek)
@ -1949,6 +1965,8 @@
0.000000 | HookCallFunction Analyzer::register_for_ports(Analyzer::ANALYZER_TEREDO, {3544/udp})
0.000000 | HookCallFunction Analyzer::register_for_ports(Analyzer::ANALYZER_VXLAN, {4789/udp})
0.000000 | HookCallFunction Analyzer::register_for_ports(Analyzer::ANALYZER_XMPP, {5222<...>/tcp})
0.000000 | HookCallFunction Broker::__subscribe(zeek/supervisor)
0.000000 | HookCallFunction Broker::subscribe(zeek/supervisor)
0.000000 | HookCallFunction Cluster::is_enabled()
0.000000 | HookCallFunction Cluster::local_node_type()
0.000000 | HookCallFunction Cluster::register_pool([topic=zeek<...>/logger, node_type=Cluster::LOGGER, max_nodes=<uninitialized>, exclusive=F])
@ -2065,7 +2083,7 @@
0.000000 | HookCallFunction Log::__create_stream(Weird::LOG, [columns=Weird::Info, ev=Weird::log_weird, path=weird])
0.000000 | HookCallFunction Log::__create_stream(X509::LOG, [columns=X509::Info, ev=X509::log_x509, path=x509])
0.000000 | HookCallFunction Log::__create_stream(mysql::LOG, [columns=MySQL::Info, ev=MySQL::log_mysql, path=mysql])
0.000000 | HookCallFunction Log::__write(PacketFilter::LOG, [ts=1573527265.694831, node=zeek, filter=ip or not ip, init=T, success=T])
0.000000 | HookCallFunction Log::__write(PacketFilter::LOG, [ts=1579727603.636084, node=zeek, filter=ip or not ip, init=T, success=T])
0.000000 | HookCallFunction Log::add_default_filter(Broker::LOG)
0.000000 | HookCallFunction Log::add_default_filter(Cluster::LOG)
0.000000 | HookCallFunction Log::add_default_filter(Config::LOG)
@ -2246,7 +2264,7 @@
0.000000 | HookCallFunction Log::create_stream(Weird::LOG, [columns=Weird::Info, ev=Weird::log_weird, path=weird])
0.000000 | HookCallFunction Log::create_stream(X509::LOG, [columns=X509::Info, ev=X509::log_x509, path=x509])
0.000000 | HookCallFunction Log::create_stream(mysql::LOG, [columns=MySQL::Info, ev=MySQL::log_mysql, path=mysql])
0.000000 | HookCallFunction Log::write(PacketFilter::LOG, [ts=1573527265.694831, node=zeek, filter=ip or not ip, init=T, success=T])
0.000000 | HookCallFunction Log::write(PacketFilter::LOG, [ts=1579727603.636084, node=zeek, filter=ip or not ip, init=T, success=T])
0.000000 | HookCallFunction NetControl::check_plugins()
0.000000 | HookCallFunction NetControl::init()
0.000000 | HookCallFunction Notice::want_pp()
@ -2485,6 +2503,7 @@
0.000000 | HookLoadFile .<...>/add-geodata.zeek
0.000000 | HookLoadFile .<...>/addrs.zeek
0.000000 | HookLoadFile .<...>/analyzer.bif.zeek
0.000000 | HookLoadFile .<...>/api.zeek
0.000000 | HookLoadFile .<...>/archive.sig
0.000000 | HookLoadFile .<...>/ascii.zeek
0.000000 | HookLoadFile .<...>/audio.sig
@ -2501,6 +2520,7 @@
0.000000 | HookLoadFile .<...>/const.bif.zeek
0.000000 | HookLoadFile .<...>/consts.zeek
0.000000 | HookLoadFile .<...>/contents.zeek
0.000000 | HookLoadFile .<...>/control.zeek
0.000000 | HookLoadFile .<...>/ct-list.zeek
0.000000 | HookLoadFile .<...>/data.bif.zeek
0.000000 | HookLoadFile .<...>/dcc-send.zeek
@ -2566,6 +2586,7 @@
0.000000 | HookLoadFile .<...>/store.zeek
0.000000 | HookLoadFile .<...>/strings.bif.zeek
0.000000 | HookLoadFile .<...>/sum.zeek
0.000000 | HookLoadFile .<...>/supervisor.bif.zeek
0.000000 | HookLoadFile .<...>/thresholds.zeek
0.000000 | HookLoadFile .<...>/top-k.bif.zeek
0.000000 | HookLoadFile .<...>/topk.zeek
@ -2588,6 +2609,7 @@
0.000000 | HookLoadFile base<...>/addrs.zeek
0.000000 | HookLoadFile base<...>/analyzer
0.000000 | HookLoadFile base<...>/analyzer.bif.zeek
0.000000 | HookLoadFile base<...>/api.zeek
0.000000 | HookLoadFile base<...>/bif
0.000000 | HookLoadFile base<...>/broker
0.000000 | HookLoadFile base<...>/cluster
@ -2668,6 +2690,8 @@
0.000000 | HookLoadFile base<...>/strings.bif.zeek
0.000000 | HookLoadFile base<...>/strings.zeek
0.000000 | HookLoadFile base<...>/sumstats
0.000000 | HookLoadFile base<...>/supervisor
0.000000 | HookLoadFile base<...>/supervisor.bif.zeek
0.000000 | HookLoadFile base<...>/syslog
0.000000 | HookLoadFile base<...>/thresholds.zeek
0.000000 | HookLoadFile base<...>/time.zeek
@ -2681,7 +2705,7 @@
0.000000 | HookLoadFile base<...>/xmpp
0.000000 | HookLoadFile base<...>/zeek.bif.zeek
0.000000 | HookLogInit packet_filter 1/1 {ts (time), node (string), filter (string), init (bool), success (bool)}
0.000000 | HookLogWrite packet_filter [ts=1573527265.694831, node=zeek, filter=ip or not ip, init=T, success=T]
0.000000 | HookLogWrite packet_filter [ts=1579727603.636084, node=zeek, filter=ip or not ip, init=T, success=T]
0.000000 | HookQueueEvent NetControl::init()
0.000000 | HookQueueEvent filter_change_tracking()
0.000000 | HookQueueEvent zeek_init()

6
testing/btest/Baseline/plugins.legacy/output
View file

@ -1,6 +0,0 @@
Demo::Foo - A Foo test analyzer (dynamic, version 1.0.0)
[Analyzer] Foo (ANALYZER_FOO, enabled)
[Event] foo_message
===
foo_message, [orig_h=::1, orig_p=37927/tcp, resp_h=::1, resp_p=4242/tcp], Hello, Foo!\x0a

1
testing/btest/Baseline/scripts.base.utils.urls/output
View file

@ -6,6 +6,7 @@
[scheme=http, netlocation=hyphen-example.com, portnum=<uninitialized>, path=/index.asp, file_name=index.asp, file_base=index, file_ext=asp, params={
[q] = 123
}]
[scheme=git, netlocation=git.kernel.org, portnum=<uninitialized>, path=/pub/scm/linux/, file_name=<uninitialized>, file_base=<uninitialized>, file_ext=<uninitialized>, params=<uninitialized>]
[scheme=<uninitialized>, netlocation=dfasjdfasdfasdf, portnum=<uninitialized>, path=/, file_name=<uninitialized>, file_base=<uninitialized>, file_ext=<uninitialized>, params={
}]

2
testing/btest/Baseline/supervisor.config-cluster/zeek.logger-1.stdout Normal file
View file

@ -0,0 +1,2 @@
supervised node zeek_init(), logger-1, Cluster::LOGGER
supervised node zeek_done(), logger-1, logger-1

2
testing/btest/Baseline/supervisor.config-cluster/zeek.manager.stdout Normal file
View file

@ -0,0 +1,2 @@
supervised node zeek_init(), manager, Cluster::MANAGER
supervised node zeek_done(), manager, manager

2
testing/btest/Baseline/supervisor.config-cluster/zeek.proxy-1.stdout Normal file
View file

@ -0,0 +1,2 @@
supervised node zeek_init(), proxy-1, Cluster::PROXY
supervised node zeek_done(), proxy-1, proxy-1

3
testing/btest/Baseline/supervisor.config-cluster/zeek.supervisor.out Normal file
View file

@ -0,0 +1,3 @@
supervisor zeek_init()
shutting down
supervisor zeek_done()

2
testing/btest/Baseline/supervisor.config-cluster/zeek.worker-1.stdout Normal file
View file

@ -0,0 +1,2 @@
supervised node zeek_init(), worker-1, Cluster::WORKER
supervised node zeek_done(), worker-1, worker-1

2
testing/btest/Baseline/supervisor.config-directory/zeek.qux.node.out Normal file
View file

@ -0,0 +1,2 @@
supervised node zeek_init()
supervised node zeek_done()

3
testing/btest/Baseline/supervisor.config-directory/zeek.supervisor.out Normal file
View file

@ -0,0 +1,3 @@
supervisor zeek_init()
destroying node
supervisor zeek_done()

3
testing/btest/Baseline/supervisor.config-output-redirect/zeek.qux.grault.stderr Normal file
View file

@ -0,0 +1,3 @@
(stderr) supervised node zeek_init()
received termination signal
(stderr) supervised node zeek_done()

Some files were not shown because too many files have changed in this diff Show more