Commit graph

5268 commits

Author SHA1 Message Date
Tim Wojtulewicz
cb1ef47a31 Add STORAGE_ prefixes for backends and serializers 2025-04-14 10:11:13 -07:00
Tim Wojtulewicz
1169fcf2a2 Move byte_buffer types from cluster and storage into util 2025-04-14 10:11:13 -07:00
Tim Wojtulewicz
88786a28a2 Add JSON storage serializer, use with existing backends/tests 2025-04-14 10:11:13 -07:00
Arne Welzel
c2e039f14d Merge remote-tracking branch 'origin/topic/awelzel/generic-metadata-pre-work'
* origin/topic/awelzel/generic-metadata-pre-work:
  cluster/Backend: Add name and lookup component tag
  cluster/Event: Hide members behind accessors
  cluster/PublishEvent:: Make event non-const
  broker/Manager: Re-use broker serializer for conversion
  EventMgr: Add Dispatch() with handler and args
  plugin/Manager: Fix MetaHookPre and MetaHookPost using HOOK_CALL_FUNCTION
2025-04-13 17:16:46 +02:00
Jan Grashoefer
3858a2920e Prevent event timestamps set to future
For scheduled events, the event timestamp is the intended timestamp. If
we force timer expiration, the timestamp might be in the future. Today,
this happens on shutdown. This change guarantees that event timestamps
are never set beyond network time.
2025-04-11 13:06:33 +02:00
Arne Welzel
0e027fa4e3 EventMgr: Add Dispatch() with handler and args
Allow users to call event_mgr.Dispatch(handler, args) instead of
constructing the Event instance themselves. Deprecate the old API
and replace users.

There's a subtle change that net_done() may be propagated via
auto_publish() now, but that still needs opt-in from script land
and likely no one did that, or else they'd expected to have it
work anyhow.
2025-04-11 09:58:07 +02:00
Arne Welzel
6d97d5526a plugin/Manager: Fix MetaHookPre and MetaHookPost using HOOK_CALL_FUNCTION 2025-04-11 09:58:07 +02:00
Arne Welzel
2f6c31df89 btest/files/x509/files: Sort analyzers in baseline
This test is very sensitive to new enum values that perturb the final
order of the output. Listing the commits that modify that file in the
past clearly shows changes to the baseline happen when a new component
with an enum value is introduced.

    40f04d4ccf broker: Add WebSocketShim backend
    51836d08ae protocol: Add StreamEvent analyzer
    fdde1e9841 cluster/serializer: Add binary-serialization-format
    6fb73aa9da broker: Add shim plugin adding a backend component
    85ca59484b postgresql: Initial parser implementation
    efc2681152 WebSocket: Introduce new analyzer and log
    ...

That's just distraction. Point fix the test to sort the analyzers before
writing the final log line.

I've previously attempted this in a more complete fashion in PR #3365
(issue #3364), but seems this direct fix might just make more sense.
2025-04-10 15:02:44 +02:00
Arne Welzel
c0f2ba468d Merge remote-tracking branch 'origin/topic/awelzel/4312-paper-over-flakiness-through-sleep'
* origin/topic/awelzel/4312-paper-over-flakiness-through-sleep:
  btest/brokerstore-attr-persistence-clone: Add BTEST_BG_RUN_SLEEP=1
2025-04-09 18:11:42 +02:00
Arne Welzel
fa7ffaf8c1 btest/brokerstore-attr-persistence-clone: Add BTEST_BG_RUN_SLEEP=1
This test has become flaky after #4295. It hasn't been failing regularly
enough locally to make it approachable. Annotate with
BTEST_BG_RUN_SLEEP=1 to revert to pre #4295 behavior, hopefully
fixing the flakiness as a side-effect.
2025-04-09 18:10:35 +02:00
Tim Wojtulewicz
30d2642272 Merge remote-tracking branch 'origin/topic/timw/4281-unconditionally-forward-out-of-snap'
* origin/topic/timw/4281-unconditionally-forward-out-of-snap:
  Make SNAP analyzer use both OUI and protocol for forwarding
  Change packet analyzer identifiers to be 64-bit
2025-04-08 10:59:15 -07:00
Arne Welzel
cb610bdea2 btest/plugin: Add plugin testing enum identifiers
This plugin has a generic name, but for now just tests the API around
enum types and enum vals at InitPostScript() time.
2025-04-04 15:36:42 +02:00
Robin Sommer
000ed528dc
Spicy: Rework code for converting Spicy values to Zeek values.
The logic was template-based so far, which wasn't great because: (1)
conceptually, it models the Spicy types at the wrong layer (C++ rather
than HILTI types), and (2) stopped working with some recent Spicy
updates (which we have temporarily reverted in the meantime to keep
Zeek working).

The new code is based on HILTI's runtime type information and the
corresponding introspection API, pretty much like `spicy-dump` works
as well. This is the recommended approach for working with HILTI
values, and generally much cleaner.

This is on top of https://github.com/zeek/zeek/pull/4300.
2025-04-02 17:45:19 +02:00
Robin Sommer
94ddd7f411
Spicy: Port over to Spicy's new tuple representation.
Includes a fix for supporting CMake 4.0.
2025-04-02 14:14:26 +02:00
Arne Welzel
363fbd9e28 Merge remote-tracking branch 'origin/topic/awelzel/fix-canon-name-builtin-plugins'
* origin/topic/awelzel/fix-canon-name-builtin-plugins:
  CI: Upgrade zeekctl task to Debian 12
  Bump cmake submodule
  CI: Add external plugins to builtin plugins task
2025-03-31 18:59:47 +02:00
Arne Welzel
14697ea6ba Merge remote-tracking branch 'origin/topic/neverlord/broker-logging'
* origin/topic/neverlord/broker-logging:
  Integrate review feedback
  Hook into Broker logs via its new API
2025-03-31 18:53:43 +02:00
Arne Welzel
c45625b909 CI: Add external plugins to builtin plugins task
This clones a few external plugins to include them in a build. Except
for zeek-kafka, the others are maintained by the Zeek project.

Closes #3535
2025-03-31 14:19:18 +02:00
Benjamin Bannier
b1ddf9e476 Use one-based index in warning message
This was surfacing a zero-based counter to users which was confusing.
2025-03-28 14:09:11 +01:00
Tim Wojtulewicz
32ae8f4eaa Make storage events take a tag for the backend instead of a string 2025-03-27 16:12:24 -07:00
Tim Wojtulewicz
85701e4514 Fix handling of timeout conditions from storage backends 2025-03-27 13:52:11 -07:00
Tim Wojtulewicz
046f32a6df Reformat plugin.storage btest to be more consistent with other storage tests 2025-03-27 13:52:11 -07:00
Arne Welzel
c3c6ee5a2b telemetry: Run callbacks at collect time
Calling collect_metrics() from a script would not invoke metric
callbacks, resulting in most of the process metrics to be zero
when a Zeek process isn't scraped via Prometheus.

Fixes #4309
2025-03-26 12:07:27 +01:00
Tim Wojtulewicz
195b87b873 Make SNAP analyzer use both OUI and protocol for forwarding 2025-03-24 15:20:50 -07:00
Arne Welzel
50b26fcea8 btest/cluster/websocket: ZeroMQ backend test
This test ensures that WebSocket clients connected to the same node see
each other's messages.
2025-03-24 18:36:52 +01:00
Arne Welzel
2963c49f27 cluster/zeromq: Fix node_topic() and nodeid_topic()
Due to prefix matching, worker-1's node_topic() also matched worker-10,
worker-11, etc. Suffix the node topic with a `.`. The original implementation
came from NATS, where subjects are separated by `.`.

Adapt nodeid_topic() for consistency.
2025-03-24 18:36:26 +01:00
Arne Welzel
888af244b2 btest/cluster/websocket: Harden multi-client tests
These test were very sensible to the speed at which ZeroMQ distributes
subscriptions in the cluster and showed to be unreliably when testing with
zeek/btest#113.

The main fix here is to have individual WebSocket clients subscribe to unique
topics, e.g /test/client-0 and /test/client-1, instead of just a shared topic.

This ensures the WebSocket handshake completes only when they observed their
own subscriptions and not prematurely when observing the shared topic.

This seems mainly relevant for tests: In the real world one shouldn't
rely on subscription visibility - you miss messages if you're too late
to the party.
2025-03-24 18:36:26 +01:00
Arne Welzel
3885871e7d cluster/zeromq: Fix unsubscription visibility
When two workers connect to zeek.cluster.worker, the central ZeroMQ
proxy would not propagate unsubscription information to other nodes
once they both left. Set ZMQ_XPUB_VERBOSER on the proxies XPUB socket
for visibility.
2025-03-24 18:36:16 +01:00
Tim Wojtulewicz
855c530b64 Redis: Handle other errors from requests, fix KEY_EXISTS for put operations 2025-03-21 11:56:27 -07:00
Tim Wojtulewicz
3d7fcfb428 SQLite: handle existing keys when overwrite=F correctly 2025-03-21 11:56:27 -07:00
Tim Wojtulewicz
ba9cf1e4db Remove unnecessary type aliases from storage btests 2025-03-21 11:56:27 -07:00
Tim Wojtulewicz
d5ebaf476d Avoid thread-leak in scripts.base.frameworks.file-analysis.bifs.enable-disable btest
This btest uses the exit() BIF to shut down, which immediately calls
::exit() and kills Zeek without doing any shutdown. This will sometimes
leave the thread running the storage manager, which causes TSan to
complain about a thread leak. Switch to use the terminate() BIF instead
which cleanly shuts down all of Zeek.
2025-03-21 11:56:27 -07:00
Tim Wojtulewicz
e5b06367f7 Squash code from OperationResultCallback into ResultCallback 2025-03-21 11:44:39 -07:00
Arne Welzel
32d7cec549 testing/btest: Remove btest-bg-run sleep 1
Bump to the latest btest master version and set BTEST_BG_RUN_SLEEP
environment variable and to 0. This makes btest-bg-run return
immediately instead of delaying by 1 second.
2025-03-20 10:11:12 +01:00
Tim Wojtulewicz
38484b6dc6 Merge remote-tracking branch 'origin/topic/vern/CPP-maint.Mar25'
* origin/topic/vern/CPP-maint.Mar25:
  minor baseline updates for -O gen-C++
  -O gen-C++ support for pattern-to-pattern comparison
  -O gen-C++ support for pattern vector comparisons
  -O gen-C++ support for type expressions
2025-03-19 12:46:52 -07:00
Vern Paxson
fb43bc63cd minor baseline updates for -O gen-C++ 2025-03-19 12:46:15 -07:00
Christian Kreibich
637baefd2c Merge branch 'topic/christian/is_valid_subnet'
* topic/christian/is_valid_subnet:
  Minor typo/markup fixes in NEWS.
  Add is_valid_subnet BiF
  Add btests for has_valid_octets() and normalize_mac()
2025-03-19 10:11:02 -07:00
Johanna Amann
7626039635 Merge remote-tracking branch 'origin/topic/johanna/ssh-server-banners-can-be-wild'
* origin/topic/johanna/ssh-server-banners-can-be-wild:
  SSH analyzer - tiny aesthetic fixes
  SSH: make banner parsing more robust
  SSH: split banner into client/server parts
2025-03-19 15:31:35 +00:00
Yacin Nadji
8b83c2995a Add is_valid_subnet BiF
Also includes consistency tweak for is_valid_ip() plus test cases.
2025-03-18 20:25:32 -07:00
Christian Kreibich
7ba9609992 Add btests for has_valid_octets() and normalize_mac()
These functions reside in base/utils/addrs.zeek and were not tested. We don't
use has_valid_octets() anywhere right now, normalize_mac() is used in the RADIUS
analyzer, and addr_to_uri() is used in the signatures/dpd.zeek tests.
2025-03-18 20:21:43 -07:00
Tim Wojtulewicz
43faea880b Add analyzer registration from VLAN to VNTAG 2025-03-18 11:51:27 -07:00
Tim Wojtulewicz
c7015e8250 Split storage.bif file into events/sync/async, add more comments 2025-03-18 10:20:34 -07:00
Tim Wojtulewicz
d0741c8001 Allow sync methods to be called from when conditions, add related btest 2025-03-18 10:20:34 -07:00
Tim Wojtulewicz
b067a6e588 Redis: Fix sync erase, add btest for it 2025-03-18 10:20:34 -07:00
Tim Wojtulewicz
e6f1eea1b7 Remove default argument for callbacks, reorder function arguments 2025-03-18 10:20:34 -07:00
Tim Wojtulewicz
cc7b2dc890 Implement Storage::backend_opened and Storage::backend_lost events 2025-03-18 10:20:34 -07:00
Tim Wojtulewicz
a99a13dc4c SQLite: expand expiration test 2025-03-18 10:20:33 -07:00
Tim Wojtulewicz
cca1d4f988 Redis: Fix thread-contention issues with Expire(), add more tests 2025-03-18 10:20:33 -07:00
Tim Wojtulewicz
b81e876ec8 Change how redis-server is run during btests, removing redis.conf 2025-03-18 10:20:33 -07:00
Tim Wojtulewicz
9ed3e33f97 Completely rework return values from storage operations 2025-03-18 10:20:33 -07:00
Tim Wojtulewicz
8ddda016ff Update some btests due to timing changes 2025-03-18 10:20:33 -07:00