Commit graph

17949 commits

Author SHA1 Message Date
Arne Welzel
f7425b805d EventHandler: Deprecate Call(args, no_remote, ts), add Call(args)
The ts parameter was only added to Call() for the Broker::auto_publish()
functionality and propagating the network timestamp. By now, the auto-publish
functionality is deprecated, so it'd be good to cleanup that signature. There
won't be any need for no_remote in the future either.

Allow users to just use Call() instead.
2025-03-27 19:23:47 +01:00
Arne Welzel
b535f03382 EventHandler: Header cleanup 2025-03-27 19:23:47 +01:00
Arne Welzel
7dadbb0c1d EventMgr: Do not cache current event attributes
Avoid proliferation of various members on EventMgr by storing the
pointer of the current event instead.

This subtly changes the behavior of some builtin functions as they would
have returned the prior event's data when executed outside of event
draining (e.g. C++ level hook invocations), but I think that's actually
for the better.
2025-03-27 19:23:30 +01:00
Arne Welzel
bef923ebeb EventMgr: Drop src_val
This is a left over and hasn't been used since a while.
2025-03-27 11:16:31 +01:00
Arne Welzel
277e6d4129 Event: Header cleanup 2025-03-27 11:16:29 +01:00
Arne Welzel
acab4236e2 Merge remote-tracking branch 'origin/topic/awelzel/zeromq-no-fprintf-at-overload'
* origin/topic/awelzel/zeromq-no-fprintf-at-overload:
  cluster/zeromq: Improve XPUB stall behavior, add a metric
2025-03-26 14:24:02 +01:00
Arne Welzel
bfffc8dac8 cluster/zeromq: Improve XPUB stall behavior, add a metric
Instead of fprintf, track the number of occurrences via a metric and
change the sleep loop to a blocking send instead.
2025-03-26 14:23:09 +01:00
Arne Welzel
cd9b37e4a5 Merge remote-tracking branch 'origin/topic/awelzel/4309-telemetry-collect-calbacks'
* origin/topic/awelzel/4309-telemetry-collect-calbacks:
  telemetry: Run callbacks at collect time
2025-03-26 14:08:40 +01:00
Arne Welzel
c3c6ee5a2b telemetry: Run callbacks at collect time
Calling collect_metrics() from a script would not invoke metric
callbacks, resulting in most of the process metrics to be zero
when a Zeek process isn't scraped via Prometheus.

Fixes #4309
2025-03-26 12:07:27 +01:00
Arne Welzel
33d7e5a7bf Merge remote-tracking branch 'origin/topic/awelzel/zeromq-unsubscription-visibility'
* origin/topic/awelzel/zeromq-unsubscription-visibility:
  cluster/OnLoop: Add metric for queue stalling instead of fprintf
  btest/cluster/websocket: ZeroMQ backend test
  cluster/OnLoop: Switch to condition variable
  cluster/zeromq: Fix node_topic() and nodeid_topic()
  cluster/websocket: Fix null deref at WebSocket server shutdown
  btest/cluster/websocket: Harden multi-client tests
  cluster/zeromq: Fix unsubscription visibility
2025-03-24 19:46:20 +01:00
Arne Welzel
52143a5712 cluster/OnLoop: Add metric for queue stalling instead of fprintf 2025-03-24 19:44:08 +01:00
Arne Welzel
50b26fcea8 btest/cluster/websocket: ZeroMQ backend test
This test ensures that WebSocket clients connected to the same node see
each other's messages.
2025-03-24 18:36:52 +01:00
Arne Welzel
387237e9c2 cluster/OnLoop: Switch to condition variable
The busy polling wasn't clever and usually resulted in delays. For now,
switch to mutex/condition variable and log an error if the timeouts are
immense.
2025-03-24 18:36:45 +01:00
Arne Welzel
2963c49f27 cluster/zeromq: Fix node_topic() and nodeid_topic()
Due to prefix matching, worker-1's node_topic() also matched worker-10,
worker-11, etc. Suffix the node topic with a `.`. The original implementation
came from NATS, where subjects are separated by `.`.

Adapt nodeid_topic() for consistency.
2025-03-24 18:36:26 +01:00
Arne Welzel
26441e0c24 cluster/websocket: Fix null deref at WebSocket server shutdown
WebSocket clients that connected with the wrong URL do not have
a backend attached. If a dispatcher is terminated while these
clients are still connected, a null deref would happen.

This was found while running all cluster/websocket tests in a loop
for a long time, tickling a segfault during the bad-url test.
2025-03-24 18:36:26 +01:00
Arne Welzel
888af244b2 btest/cluster/websocket: Harden multi-client tests
These test were very sensible to the speed at which ZeroMQ distributes
subscriptions in the cluster and showed to be unreliably when testing with
zeek/btest#113.

The main fix here is to have individual WebSocket clients subscribe to unique
topics, e.g /test/client-0 and /test/client-1, instead of just a shared topic.

This ensures the WebSocket handshake completes only when they observed their
own subscriptions and not prematurely when observing the shared topic.

This seems mainly relevant for tests: In the real world one shouldn't
rely on subscription visibility - you miss messages if you're too late
to the party.
2025-03-24 18:36:26 +01:00
Arne Welzel
3885871e7d cluster/zeromq: Fix unsubscription visibility
When two workers connect to zeek.cluster.worker, the central ZeroMQ
proxy would not propagate unsubscription information to other nodes
once they both left. Set ZMQ_XPUB_VERBOSER on the proxies XPUB socket
for visibility.
2025-03-24 18:36:16 +01:00
zeek-bot
6045c8ee64 Update doc submodule [nomail] [skip ci] 2025-03-22 00:15:29 +00:00
Tim Wojtulewicz
bc38dbcc99 Merge remote-tracking branch 'origin/topic/timw/storage-framework-followup'
* origin/topic/timw/storage-framework-followup:
  Redis: Handle other errors from requests, fix KEY_EXISTS for put operations
  SQLite: handle existing keys when overwrite=F correctly
  Remove unnecessary type aliases from storage btests
  Avoid thread-leak in scripts.base.frameworks.file-analysis.bifs.enable-disable btest
  Fix data-race with calling DBG_LOG from a separate thread
  Fix data-race with ReturnCode objects in Sqlite::DoExpire
  Fix data race with calling run_state::network_time from a separate thread
  Add NEWS entry for Storage, reduce CHANGES spam
  Fix Coverity findings in the SQLite backend
  Remove inclusion of non-existent expected-lite path during dynamic plugin builds
  Squash code from OperationResultCallback into ResultCallback
  Add hiredis to generate-docs workflow to enable Redis backend
2025-03-21 11:57:17 -07:00
Tim Wojtulewicz
855c530b64 Redis: Handle other errors from requests, fix KEY_EXISTS for put operations 2025-03-21 11:56:27 -07:00
Tim Wojtulewicz
3d7fcfb428 SQLite: handle existing keys when overwrite=F correctly 2025-03-21 11:56:27 -07:00
Tim Wojtulewicz
ba9cf1e4db Remove unnecessary type aliases from storage btests 2025-03-21 11:56:27 -07:00
Tim Wojtulewicz
d5ebaf476d Avoid thread-leak in scripts.base.frameworks.file-analysis.bifs.enable-disable btest
This btest uses the exit() BIF to shut down, which immediately calls
::exit() and kills Zeek without doing any shutdown. This will sometimes
leave the thread running the storage manager, which causes TSan to
complain about a thread leak. Switch to use the terminate() BIF instead
which cleanly shuts down all of Zeek.
2025-03-21 11:56:27 -07:00
Tim Wojtulewicz
78267c382c Fix data-race with calling DBG_LOG from a separate thread 2025-03-21 11:56:27 -07:00
Tim Wojtulewicz
a67e138d4e Fix data-race with ReturnCode objects in Sqlite::DoExpire 2025-03-21 11:56:27 -07:00
Tim Wojtulewicz
8b9fe48f13 Fix data race with calling run_state::network_time from a separate thread 2025-03-21 11:56:27 -07:00
Tim Wojtulewicz
458bf762f1 Add NEWS entry for Storage, reduce CHANGES spam 2025-03-21 11:56:27 -07:00
Tim Wojtulewicz
0929cf507f Fix Coverity findings in the SQLite backend 2025-03-21 11:56:27 -07:00
Tim Wojtulewicz
e505135507 Remove inclusion of non-existent expected-lite path during dynamic plugin builds 2025-03-21 11:55:12 -07:00
Tim Wojtulewicz
e5b06367f7 Squash code from OperationResultCallback into ResultCallback 2025-03-21 11:44:39 -07:00
Tim Wojtulewicz
cd66b32428 Add hiredis to generate-docs workflow to enable Redis backend 2025-03-21 11:44:39 -07:00
Arne Welzel
0a6429d447 Merge remote-tracking branch 'origin/topic/awelzel/btest-bg-run-sleep-0'
* origin/topic/awelzel/btest-bg-run-sleep-0:
  testing/btest: Remove btest-bg-run sleep 1
2025-03-21 09:45:15 +01:00
Arne Welzel
32d7cec549 testing/btest: Remove btest-bg-run sleep 1
Bump to the latest btest master version and set BTEST_BG_RUN_SLEEP
environment variable and to 0. This makes btest-bg-run return
immediately instead of delaying by 1 second.
2025-03-20 10:11:12 +01:00
zeek-bot
1c598df53b Update doc submodule [nomail] [skip ci] 2025-03-20 00:14:12 +00:00
Tim Wojtulewicz
0b590c575e Merge remote-tracking branch 'origin/topic/awelzel/allow-terminate-processing-from-non-main-thread'
* origin/topic/awelzel/allow-terminate-processing-from-non-main-thread:
  util: Fix terminate_processing() called from non-main thread
2025-03-19 12:47:48 -07:00
Tim Wojtulewicz
38484b6dc6 Merge remote-tracking branch 'origin/topic/vern/CPP-maint.Mar25'
* origin/topic/vern/CPP-maint.Mar25:
  minor baseline updates for -O gen-C++
  -O gen-C++ support for pattern-to-pattern comparison
  -O gen-C++ support for pattern vector comparisons
  -O gen-C++ support for type expressions
2025-03-19 12:46:52 -07:00
Vern Paxson
fb43bc63cd minor baseline updates for -O gen-C++ 2025-03-19 12:46:15 -07:00
Tim Wojtulewicz
ba28dfe87e Merge remote-tracking branch 'origin/topic/timw/run-macos-ci-tests-as-root'
* origin/topic/timw/run-macos-ci-tests-as-root:
  CI: Run btests on macOS under sudo
2025-03-19 11:58:55 -07:00
Arne Welzel
f4d7243365 Merge remote-tracking branch 'origin/topic/awelzel/bump-zeekjs-0-16-0'
* origin/topic/awelzel/bump-zeekjs-0-16-0:
  Bump zeekjs to v0.16.0
2025-03-19 19:57:16 +01:00
Tim Wojtulewicz
d989d1fc0e CI: Run btests on macOS under sudo 2025-03-19 10:56:45 -07:00
Christian Kreibich
637baefd2c Merge branch 'topic/christian/is_valid_subnet'
* topic/christian/is_valid_subnet:
  Minor typo/markup fixes in NEWS.
  Add is_valid_subnet BiF
  Add btests for has_valid_octets() and normalize_mac()
2025-03-19 10:11:02 -07:00
Johanna Amann
7626039635 Merge remote-tracking branch 'origin/topic/johanna/ssh-server-banners-can-be-wild'
* origin/topic/johanna/ssh-server-banners-can-be-wild:
  SSH analyzer - tiny aesthetic fixes
  SSH: make banner parsing more robust
  SSH: split banner into client/server parts
2025-03-19 15:31:35 +00:00
Johanna Amann
19a070080f SSH analyzer - tiny aesthetic fixes
No functional changes. Addresses feedback of GH-4289
2025-03-19 10:38:03 +00:00
Arne Welzel
5f9018879d Bump zeekjs to v0.16.0
8edcd88 version: 0.16.0
    9ec0b25 Executor: Allow compilation with GCC 11.2
    2760b4f Node.js: Install uv_timer to imitate being alive
2025-03-19 09:45:47 +01:00
Christian Kreibich
283b140e6a Minor typo/markup fixes in NEWS. 2025-03-18 20:25:49 -07:00
Yacin Nadji
8b83c2995a Add is_valid_subnet BiF
Also includes consistency tweak for is_valid_ip() plus test cases.
2025-03-18 20:25:32 -07:00
Christian Kreibich
7ba9609992 Add btests for has_valid_octets() and normalize_mac()
These functions reside in base/utils/addrs.zeek and were not tested. We don't
use has_valid_octets() anywhere right now, normalize_mac() is used in the RADIUS
analyzer, and addr_to_uri() is used in the signatures/dpd.zeek tests.
2025-03-18 20:21:43 -07:00
zeek-bot
d7cc409fd5 Update doc submodule [nomail] [skip ci] 2025-03-19 00:36:02 +00:00
Tim Wojtulewicz
cb5e3d0054 Merge remote-tracking branch 'origin/topic/timw/vntag-in-vlan'
* origin/topic/timw/vntag-in-vlan:
  Add analyzer registration from VLAN to VNTAG
2025-03-18 11:52:19 -07:00
Tim Wojtulewicz
43faea880b Add analyzer registration from VLAN to VNTAG 2025-03-18 11:51:27 -07:00