Commit graph

1132 commits

Author SHA1 Message Date
Arne Welzel
d2314d2666 files.log: Unroll and introduce uid and id fields
This is a script-only change that unrolls File::Info records into
multiple files.log entries if the same file was seen over different
connections by single worker. Consequently, the File::Info record
gets the commonly used uid and id fields added. These fields are
optional for File::Info - a file may be analyzed without relation
to a network connection (e.g by using Input::add_analysis()).

The existing tx_hosts, rx_hosts and conn_uids fields of Files::Info
are not meaningful after this change and removed by default. Therefore,
files.log will have them removed, too.

The tx_hosts, rx_hosts and conn_uids fields can be revived by using the
policy script frameworks/files/deprecated-txhosts-rxhosts-connuids.zeek
included in the distribution. However, with v6.1 this script will be
removed.
2022-08-16 17:22:20 +02:00
Tim Wojtulewicz
9524963da6 Merge remote-tracking branch 'anniebryan/notice-suppression-bug-fix'
* anniebryan/notice-suppression-bug-fix:
  Added test case with back-to-back notices
  Fix notice suppression atomicity bug
2022-08-05 16:21:29 +00:00
Robin Sommer
686e740bbe
Merge remote-tracking branch 'origin/topic/awelzel/telemetry-script-land-v0'
* origin/topic/awelzel/telemetry-script-land-v0:
  Introduce telemetry framework
  Bump broker submodule to master.
2022-08-05 17:14:23 +02:00
Arne Welzel
3fe930dbf2 Introduce telemetry framework
Adds base/frameworks/telemetry with wrappers around telemetry.bif
and updates telemetry/Manager to support collecting metrics from
script land.

Add policy/frameworks/telemetry/log for logging of metrics data
into a new telemetry.log and telemetry_histogram.log and add into
local.zeek by default.
2022-08-05 11:43:44 +02:00
Annie Bryan
82ee17be8d Fix notice suppression atomicity bug 2022-08-03 16:30:57 +02:00
Benjamin Bannier
396fb89504 Remove use of fallible get_conn_transport_proto in analyzer_violation.
When setting up the DPD info we previously would get the
`transport_proto` for the connection with `get_conn_transport_proto`.
This function takes a `conn_id` and would fail fatally if the connection
for the given ID was unknown. It seems it was possible to run into such
scenarios when the `analyzer_violation` event was processed after the
connection had been cleaned up.

We now get the `transport_proto` directly from the ports in the
`connection` passed into `analyzer_violation` via
`get_port_transport_proto` which cannot fail.
2022-07-19 12:20:45 +02:00
Robin Sommer
691b099de1 Merge remote-tracking branch 'origin/topic/awelzel/2120-logdir-leftover'
* origin/topic/awelzel/2120-logdir-leftover:
  sqlite default-logdir test: Remove ls ./logs baseline
  logging/sqlite: Recognize Log::default_logdir and place files there if set
  logging: Introduce Log::default_logdir deprecate LogAscii::logdir and per writer logdir
  logging/ascii: Fix .shadow paths when using LogAscii::logdir
2022-07-07 08:06:13 +02:00
Arne Welzel
aaa47a709c logging: Introduce Log::default_logdir deprecate LogAscii::logdir and per writer logdir
Also modify FormatRotationPath to keep rotated logs within
Log::default_logdir unless the rotation function explicitly
set dir, e.g. by when the user redef'ed default_rotation_interval.
2022-07-06 18:54:29 +02:00
Tim Wojtulewicz
fb16ce3711 Remove other general deprecations 2022-06-30 19:17:13 +00:00
Christian Kreibich
edef3736fb Additional &is_used tags in the Netcontrol and Sumstats frameworks
When running a cluster, these functions only get called in select node types and
could trigger no-caller warnings on stderr.
2022-06-02 22:57:07 -07:00
Christian Kreibich
1cebdd569d Merge branch 'topic/christian/gh-2134-fix-intel-test-races'
* topic/christian/gh-2134-fix-intel-test-races:
  Expand scripts.base.frameworks.intel.cluster-transparency test
  Fix races in scripts.base.frameworks.intel.cluster-transparency-with-proxy test
  Add Intel::send_store_on_node_up boolean to control min_data_store delivery
2022-06-02 12:20:06 -07:00
Robin Sommer
d99f041ac5
Add WebSocket support for exchanging events with external clients.
This exposes Broker's new WebSocket support in Zeek. To enable it,
call `Broker::listen_websocket()`. Zeek will then start listening on
port 9997 for incoming WebSocket connections.

See the Broker documentation for a description of the message format
expected over these WebSocket connections.
2022-06-02 10:31:52 +02:00
Christian Kreibich
892a3a8452 Add Intel::send_store_on_node_up boolean to control min_data_store delivery
This adds a redefinable const to the internals of the Intel framework, to allow
suppression of the manager sending its current min_data_store when a worker
connects. This feature is desirable for nodes that check in "late" to bring them
up to speed, but during testing it introduces nondeterminism.
2022-06-01 17:45:19 -07:00
Christian Kreibich
93bed5a261 Merge branch 'topic/christian/node-status-notification'
* topic/christian/node-status-notification:
  Add Supervisor::node_status notification event
2022-05-31 12:53:18 -07:00
Christian Kreibich
14188fc7a7 Add Supervisor::node_status notification event
The Supervisor generates this event every time it receives a status update from
the stem, meaning a node got created or re-created. A corresponding
SupervisorControl::node_status event relays the same information for users
interacting with the Supervisor over Broker.
2022-05-30 21:36:35 -07:00
Vern Paxson
07cf5cb089 deprecation messages for unused base script functions 2022-05-27 14:36:30 -07:00
Vern Paxson
6dc711c39e annotate orphan base script components with &deprecated 2022-05-26 17:39:17 -07:00
Vern Paxson
9b8ac44169 annotate base scripts with &is_used as needed 2022-05-26 17:39:17 -07:00
Tim Wojtulewicz
f8bc23d3e1 Propagate BPF_Program error message to script land 2022-05-25 09:41:35 -07:00
Tim Wojtulewicz
8b0263cb39 Merge remote-tracking branch 'origin/topic/vern/script-profiling'
* origin/topic/vern/script-profiling:
  tidy up after generating profile
  test suite updates for refined script coverage, use of new BiF to speed startup
  fix for coverage reporting for functions that use "when" statements
  new global_options() BiF to speed up startup, plus a micro-preen
  hooks for new --profile-scripts option
  classes for managing script profiles
  address some holes in script coverage
  fix for script coverage missing on-exit activity
  memory management fixes for loggers
  make curr_CPU_time() broadly available rather than just isolated to ZAM
2022-05-11 12:56:41 -07:00
Johanna Amann
3a7b127a71 Merge remote-tracking branch 'origin/topic/johanna/topk-cite'
* origin/topic/johanna/topk-cite:
  Add exact name of the Top-k algorithm.
2022-05-11 18:45:48 +01:00
Johanna Amann
25c33d2a29 Add exact name of the Top-k algorithm.
I needed to figure out which exact algorithm we use for our
probabilistic top-k measurements. It turns out that we do not mention
this in our source tree at all so far.
2022-05-11 13:21:26 +01:00
Vern Paxson
fab4905fc2 new global_options() BiF to speed up startup, plus a micro-preen 2022-05-03 11:13:15 -07:00
Vern Paxson
f62c04c9eb tweaks to base scripts revealed by switch to expression-based initialization 2022-03-23 15:36:35 -07:00
Christian Kreibich
6a46d1584c Add capture to a Sumstats when-statement
This resolves a deprecation warning that currently triggers when running
Sumstats in clusterized Zeek.
2022-02-04 12:54:34 -08:00
Vern Paxson
98cd3f2213 update uses of "when" in base scripts to include captures 2022-01-07 14:53:33 -08:00
Vern Paxson
72a59bf828 removed unused script variable 2021-12-14 12:49:27 -08:00
Tim Wojtulewicz
8816547964 Fix types for Analyzer::register_for_port(s) to be the same 2021-12-10 17:48:19 +00:00
Christian Kreibich
1aaed1cc2e Add LogAscii::json_include_unset_fields flag to control unset field rendering
The flag controls whether JSON rendering includes unset &optional log fields
(F, the default), or includes them with a null value (T).
2021-12-08 17:29:07 -08:00
Tim Wojtulewicz
e82a78616b Update NEWS and some minor fixes for docs/zeekygen 2021-11-23 19:39:36 -07:00
Tim Wojtulewicz
2044fbe53b Add GTPv1 packet analyzer, disable old analyzer 2021-11-23 19:36:50 -07:00
Tim Wojtulewicz
dc0ecf9811 Add Teredo packet analyzer, disable old analyzer 2021-11-23 19:36:50 -07:00
Tim Wojtulewicz
05574ecce1 Add VXLAN packet analyzer, disable old analyzer 2021-11-23 19:36:50 -07:00
Tim Wojtulewicz
cbb0bcd49c Add Geneve packet analyzer, disable old analyzer 2021-11-23 19:36:50 -07:00
Tim Wojtulewicz
7e40094f2c Add AYIYA packet analyzer, disable old analyzer 2021-11-23 19:36:50 -07:00
Tim Wojtulewicz
44e0760e96 Add PacketAnalyzer::register_for_port(s) functions
These allow packet analyzers to register ports as identifiers to forward from
parent analyzers, while also adding those ports to the now-global
Analyzer::ports table at the same time.
2021-11-23 19:36:50 -07:00
Tim Wojtulewicz
612212568a Add analyzer_confirmation and analyzer_violation events 2021-11-23 19:36:50 -07:00
Tim Wojtulewicz
a7d3cb48ef Add concept of "parent" tag namespaces
This allows us to create an EnumType that groups all of the analyzer
tag values into a single type, while still having the existing types
that split them up. We can then use this for certain events that benefit
from taking all of the tag types at once.
2021-11-23 19:36:49 -07:00
Tim Wojtulewicz
a6378531db Remove trailing whitespace from script files 2021-10-20 09:57:09 -07:00
Tim Wojtulewicz
0a0ed65306 Merge remote-tracking branch 'origin/topic/robin/gh-54-sanitize'
* origin/topic/robin/gh-54-sanitize:
  Sanitize log files names before they go into system().
2021-09-22 12:17:05 -07:00
Robin Sommer
74680bf4e6 Merge remote-tracking branch 'origin/topic/justin/software-framework-parse-cache'
* origin/topic/justin/software-framework-parse-cache:
  Restore behavior of Software::register event
  Optimzie software framework version parsing
2021-09-21 18:00:46 +02:00
Justin Azoff
ef5fb790ef Restore behavior of Software::register event
Use an intermediary event to ensure that software versions are parsed
before calling Software::register.
2021-09-20 14:38:47 -04:00
Seth Hall
a4ceb98bf8 Switch the TSV Zeek logs to be UTF8 by default.
There is a paired zeek-testing branch for some updates there.
2021-09-07 09:16:53 -07:00
Justin Azoff
3bf8c8ceb6 Optimzie software framework version parsing
Add a small cache in front of the parse method.  This cache should
reduce most of the calls to parse, and ultimately save memory because
redundant versions of the parsed strings will not be created in memory.

Move the parsing itself to the proxies where the caching can be more
efficient.
2021-09-05 13:18:51 -04:00
Christian Kreibich
90b7c6961e Simplify the supervisor's listen() on default address/port 2021-08-18 12:35:49 -07:00
Christian Kreibich
281eef398f Merge branch 'topic/vlad/doc_tweaks'
* topic/vlad/doc_tweaks:
  docs: escape * in documentation
  Update the mail_dest documentation with the new email_dest field
2021-07-09 12:02:32 -07:00
Christian Kreibich
8db985ea78 Merge branch 'topic/christian/cluster-controller'
* topic/christian/cluster-controller:
  Add a cluster controller testcase for agent-controller checkin
  Add zeek-client via new submodule
  Update baselines affected by cluster controller changes
  Introduce cluster controller and cluster agent scripting
  Establish a separate init script when using the supervisor
  Add optional bare-mode boolean flag to Supervisor's node configuration
  Add support for making the supervisor listen for requests
  Add support for setting environment variables via supervisor
2021-07-08 16:51:11 -07:00
Christian Kreibich
6eddaed62d Fix reference in the logging framework docs re. Log::StreamPolicyHook 2021-07-08 15:39:46 -07:00
Vlad Grigorescu
10159a3fdf Update the mail_dest documentation with the new email_dest field 2021-07-08 16:01:21 -05:00
Christian Kreibich
7bee79b400 Add optional bare-mode boolean flag to Supervisor's node configuration
When omitted, the node inherits the Supervisor's bare-mode
status. When true/false, the new Zeek node will enable/disable bare
mode, respectively. It continues to load any scripts passed at the
command line and in the additional scripts list already provided in
the node configuration.

Includes testcase.
2021-07-08 13:12:53 -07:00