Commit graph

5258 commits

Author SHA1 Message Date
Tim Wojtulewicz
30d2642272 Merge remote-tracking branch 'origin/topic/timw/4281-unconditionally-forward-out-of-snap'
* origin/topic/timw/4281-unconditionally-forward-out-of-snap:
  Make SNAP analyzer use both OUI and protocol for forwarding
  Change packet analyzer identifiers to be 64-bit
2025-04-08 10:59:15 -07:00
Arne Welzel
cb610bdea2 btest/plugin: Add plugin testing enum identifiers
This plugin has a generic name, but for now just tests the API around
enum types and enum vals at InitPostScript() time.
2025-04-04 15:36:42 +02:00
Robin Sommer
000ed528dc
Spicy: Rework code for converting Spicy values to Zeek values.
The logic was template-based so far, which wasn't great because: (1)
conceptually, it models the Spicy types at the wrong layer (C++ rather
than HILTI types), and (2) stopped working with some recent Spicy
updates (which we have temporarily reverted in the meantime to keep
Zeek working).

The new code is based on HILTI's runtime type information and the
corresponding introspection API, pretty much like `spicy-dump` works
as well. This is the recommended approach for working with HILTI
values, and generally much cleaner.

This is on top of https://github.com/zeek/zeek/pull/4300.
2025-04-02 17:45:19 +02:00
Robin Sommer
94ddd7f411
Spicy: Port over to Spicy's new tuple representation.
Includes a fix for supporting CMake 4.0.
2025-04-02 14:14:26 +02:00
Arne Welzel
363fbd9e28 Merge remote-tracking branch 'origin/topic/awelzel/fix-canon-name-builtin-plugins'
* origin/topic/awelzel/fix-canon-name-builtin-plugins:
  CI: Upgrade zeekctl task to Debian 12
  Bump cmake submodule
  CI: Add external plugins to builtin plugins task
2025-03-31 18:59:47 +02:00
Arne Welzel
14697ea6ba Merge remote-tracking branch 'origin/topic/neverlord/broker-logging'
* origin/topic/neverlord/broker-logging:
  Integrate review feedback
  Hook into Broker logs via its new API
2025-03-31 18:53:43 +02:00
Arne Welzel
c45625b909 CI: Add external plugins to builtin plugins task
This clones a few external plugins to include them in a build. Except
for zeek-kafka, the others are maintained by the Zeek project.

Closes #3535
2025-03-31 14:19:18 +02:00
Benjamin Bannier
b1ddf9e476 Use one-based index in warning message
This was surfacing a zero-based counter to users which was confusing.
2025-03-28 14:09:11 +01:00
Tim Wojtulewicz
32ae8f4eaa Make storage events take a tag for the backend instead of a string 2025-03-27 16:12:24 -07:00
Tim Wojtulewicz
85701e4514 Fix handling of timeout conditions from storage backends 2025-03-27 13:52:11 -07:00
Tim Wojtulewicz
046f32a6df Reformat plugin.storage btest to be more consistent with other storage tests 2025-03-27 13:52:11 -07:00
Arne Welzel
c3c6ee5a2b telemetry: Run callbacks at collect time
Calling collect_metrics() from a script would not invoke metric
callbacks, resulting in most of the process metrics to be zero
when a Zeek process isn't scraped via Prometheus.

Fixes #4309
2025-03-26 12:07:27 +01:00
Tim Wojtulewicz
195b87b873 Make SNAP analyzer use both OUI and protocol for forwarding 2025-03-24 15:20:50 -07:00
Arne Welzel
50b26fcea8 btest/cluster/websocket: ZeroMQ backend test
This test ensures that WebSocket clients connected to the same node see
each other's messages.
2025-03-24 18:36:52 +01:00
Arne Welzel
2963c49f27 cluster/zeromq: Fix node_topic() and nodeid_topic()
Due to prefix matching, worker-1's node_topic() also matched worker-10,
worker-11, etc. Suffix the node topic with a `.`. The original implementation
came from NATS, where subjects are separated by `.`.

Adapt nodeid_topic() for consistency.
2025-03-24 18:36:26 +01:00
Arne Welzel
888af244b2 btest/cluster/websocket: Harden multi-client tests
These test were very sensible to the speed at which ZeroMQ distributes
subscriptions in the cluster and showed to be unreliably when testing with
zeek/btest#113.

The main fix here is to have individual WebSocket clients subscribe to unique
topics, e.g /test/client-0 and /test/client-1, instead of just a shared topic.

This ensures the WebSocket handshake completes only when they observed their
own subscriptions and not prematurely when observing the shared topic.

This seems mainly relevant for tests: In the real world one shouldn't
rely on subscription visibility - you miss messages if you're too late
to the party.
2025-03-24 18:36:26 +01:00
Arne Welzel
3885871e7d cluster/zeromq: Fix unsubscription visibility
When two workers connect to zeek.cluster.worker, the central ZeroMQ
proxy would not propagate unsubscription information to other nodes
once they both left. Set ZMQ_XPUB_VERBOSER on the proxies XPUB socket
for visibility.
2025-03-24 18:36:16 +01:00
Tim Wojtulewicz
855c530b64 Redis: Handle other errors from requests, fix KEY_EXISTS for put operations 2025-03-21 11:56:27 -07:00
Tim Wojtulewicz
3d7fcfb428 SQLite: handle existing keys when overwrite=F correctly 2025-03-21 11:56:27 -07:00
Tim Wojtulewicz
ba9cf1e4db Remove unnecessary type aliases from storage btests 2025-03-21 11:56:27 -07:00
Tim Wojtulewicz
d5ebaf476d Avoid thread-leak in scripts.base.frameworks.file-analysis.bifs.enable-disable btest
This btest uses the exit() BIF to shut down, which immediately calls
::exit() and kills Zeek without doing any shutdown. This will sometimes
leave the thread running the storage manager, which causes TSan to
complain about a thread leak. Switch to use the terminate() BIF instead
which cleanly shuts down all of Zeek.
2025-03-21 11:56:27 -07:00
Tim Wojtulewicz
e5b06367f7 Squash code from OperationResultCallback into ResultCallback 2025-03-21 11:44:39 -07:00
Arne Welzel
32d7cec549 testing/btest: Remove btest-bg-run sleep 1
Bump to the latest btest master version and set BTEST_BG_RUN_SLEEP
environment variable and to 0. This makes btest-bg-run return
immediately instead of delaying by 1 second.
2025-03-20 10:11:12 +01:00
Tim Wojtulewicz
38484b6dc6 Merge remote-tracking branch 'origin/topic/vern/CPP-maint.Mar25'
* origin/topic/vern/CPP-maint.Mar25:
  minor baseline updates for -O gen-C++
  -O gen-C++ support for pattern-to-pattern comparison
  -O gen-C++ support for pattern vector comparisons
  -O gen-C++ support for type expressions
2025-03-19 12:46:52 -07:00
Vern Paxson
fb43bc63cd minor baseline updates for -O gen-C++ 2025-03-19 12:46:15 -07:00
Christian Kreibich
637baefd2c Merge branch 'topic/christian/is_valid_subnet'
* topic/christian/is_valid_subnet:
  Minor typo/markup fixes in NEWS.
  Add is_valid_subnet BiF
  Add btests for has_valid_octets() and normalize_mac()
2025-03-19 10:11:02 -07:00
Johanna Amann
7626039635 Merge remote-tracking branch 'origin/topic/johanna/ssh-server-banners-can-be-wild'
* origin/topic/johanna/ssh-server-banners-can-be-wild:
  SSH analyzer - tiny aesthetic fixes
  SSH: make banner parsing more robust
  SSH: split banner into client/server parts
2025-03-19 15:31:35 +00:00
Yacin Nadji
8b83c2995a Add is_valid_subnet BiF
Also includes consistency tweak for is_valid_ip() plus test cases.
2025-03-18 20:25:32 -07:00
Christian Kreibich
7ba9609992 Add btests for has_valid_octets() and normalize_mac()
These functions reside in base/utils/addrs.zeek and were not tested. We don't
use has_valid_octets() anywhere right now, normalize_mac() is used in the RADIUS
analyzer, and addr_to_uri() is used in the signatures/dpd.zeek tests.
2025-03-18 20:21:43 -07:00
Tim Wojtulewicz
43faea880b Add analyzer registration from VLAN to VNTAG 2025-03-18 11:51:27 -07:00
Tim Wojtulewicz
c7015e8250 Split storage.bif file into events/sync/async, add more comments 2025-03-18 10:20:34 -07:00
Tim Wojtulewicz
d0741c8001 Allow sync methods to be called from when conditions, add related btest 2025-03-18 10:20:34 -07:00
Tim Wojtulewicz
b067a6e588 Redis: Fix sync erase, add btest for it 2025-03-18 10:20:34 -07:00
Tim Wojtulewicz
e6f1eea1b7 Remove default argument for callbacks, reorder function arguments 2025-03-18 10:20:34 -07:00
Tim Wojtulewicz
cc7b2dc890 Implement Storage::backend_opened and Storage::backend_lost events 2025-03-18 10:20:34 -07:00
Tim Wojtulewicz
a99a13dc4c SQLite: expand expiration test 2025-03-18 10:20:33 -07:00
Tim Wojtulewicz
cca1d4f988 Redis: Fix thread-contention issues with Expire(), add more tests 2025-03-18 10:20:33 -07:00
Tim Wojtulewicz
b81e876ec8 Change how redis-server is run during btests, removing redis.conf 2025-03-18 10:20:33 -07:00
Tim Wojtulewicz
9ed3e33f97 Completely rework return values from storage operations 2025-03-18 10:20:33 -07:00
Tim Wojtulewicz
8ddda016ff Update some btests due to timing changes 2025-03-18 10:20:33 -07:00
Tim Wojtulewicz
e766af7322 Split sync/async handling into the BIF methods 2025-03-18 10:20:33 -07:00
Tim Wojtulewicz
c247de8ec3 Redis: Rework everything to only use async mode 2025-03-18 10:20:33 -07:00
Tim Wojtulewicz
a485b1d237 Make backend options a record, move actual options to be sub-records 2025-03-18 10:20:33 -07:00
Tim Wojtulewicz
28951dccf1 Split sync and async into separate script-land namespaces 2025-03-18 10:20:33 -07:00
Tim Wojtulewicz
42ad5bbf7d Add btest that uses a Redis backend in a cluster 2025-03-18 09:32:34 -07:00
Tim Wojtulewicz
f1a7376e0a Return generic result for get operations that includes error messages 2025-03-18 09:32:34 -07:00
Tim Wojtulewicz
4695060d75 Allow opening and closing backends to be async 2025-03-18 09:32:34 -07:00
Tim Wojtulewicz
ea87c773cd Redis: Support non-native expiration when reading traces 2025-03-18 09:32:34 -07:00
Tim Wojtulewicz
08bebaa426 Redis: Add btests for the redis backend 2025-03-18 09:32:34 -07:00
Tim Wojtulewicz
6289eb8e15 SQLite: Fix some issues with expiration, including in the btest 2025-03-18 09:32:34 -07:00