Mirror/zeek
1
0
Fork 0
mirror of https://github.com/zeek/zeek.git synced 2025-10-02 06:38:20 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 1
18139 commits 387 branches 195 tags 255 MiB
Commit graph

18139 commits

This branch
This branch
All branches
Author SHA1 Message Date
Arne Welzel
cf0669ade3 fixup! cluster/websocket: Automatic WebSocket client topic subscription 2025-04-24 15:02:57 +02:00
Arne Welzel
8edec9885a cluster/websocket: Automatic WebSocket client topic subscription 
Subscribe every WebSocket client to a unique topic, by default under

    zeek/cluster/websocket/client/<identifier>/

Add tests that verify that WebSocket clients receive messages on these topics
even if they didn't explicitly pass them in their handshake message.

This is somewhere between feature and bug fix. It aids the ZeroMQ backend
implementation: A WebSocket client that doesn't provide any subscriptions
and immediately starts publishing would discard events until receiving
other nodes subscriptions from the central XPUB/XSUB proxy. ZeroMQ does sender
side topic filtering. When using subscriptions, the client waits until
its own subscriptions are returned from the central XPUB/XSUB proxy,
thereby also learning about other node's subscriptions.

Also, make the no-subscriptions.zeek test use 32 clients sequentially to
trigger potential issues more quickly.
 2025-04-24 14:14:22 +02:00
Arne Welzel
69a1ad2c3d Merge remote-tracking branch 'origin/topic/awelzel/cluster-fix-tsan-zeromq-do-terminate' 
* origin/topic/awelzel/cluster-fix-tsan-zeromq-do-terminate:
  NEWS: Add entry about WebSocket client events
  btest/cluster: Testing cleanup
  cluster/websocket: Raise websocket_client_lost() after terminate
  cluster/ThreadedBackend: Invoke onloop->Process() during DoTerminate()
  cluster/ThreadedBackend: Remove Process()
  zeromq: Call super class DoTerminate() after stopping thread
 2025-04-24 14:04:11 +02:00
Arne Welzel
7513d0ef1b NEWS: Add entry about WebSocket client events 2025-04-24 09:50:04 +02:00
Arne Welzel
2a6beae50b btest/cluster: Testing cleanup 2025-04-24 09:35:53 +02:00
Arne Welzel
63723cd9e3 cluster/websocket: Raise websocket_client_lost() after terminate 
Just in case events are created during backend->Terminate(). These
should come before the Cluster::websocket_client_lost() event.
 2025-04-24 09:35:20 +02:00
Arne Welzel
eb2eaee284 cluster/ThreadedBackend: Invoke onloop->Process() during DoTerminate() 
Also, document how to use ThreadedBackend's DoTerminate()
 2025-04-24 09:35:20 +02:00
Arne Welzel
85a2694cb2 cluster/ThreadedBackend: Remove Process() 
This must have been left-over from before OnLoopProcess existed. It
wasn't called or used anymore.
 2025-04-24 09:35:20 +02:00
Arne Welzel
63d31d7d9f zeromq: Call super class DoTerminate() after stopping thread 
The internal ZeroMQ thread would call QueueForProcessing() thereby
accessing the onloop member. As ThreadedBackend::DoTerminate() unsets it,
this was a) reported as a data race by TSAN and b) potentially caused
missed events that were still to be queued.
 2025-04-24 09:35:20 +02:00
Arne Welzel
79c4fdb237 Merge remote-tracking branch 'origin/topic/awelzel/websocket-empty-subscriptions' 
* origin/topic/awelzel/websocket-empty-subscriptions:
  cluster/websocket: Short-circuit clients without subscriptions
  cluster/websocket: Factor out active subscription handling
 2025-04-24 08:17:46 +02:00
Arne Welzel
23f0370e91 cluster/websocket: Short-circuit clients without subscriptions 2025-04-24 08:14:56 +02:00
Arne Welzel
47206d6a8a cluster/websocket: Factor out active subscription handling 2025-04-24 08:14:56 +02:00
zeek-bot
dee6f1421a Update doc submodule [nomail] [skip ci] 2025-04-24 00:26:49 +00:00
Tim Wojtulewicz
8054547712 Merge remote-tracking branch 'origin/topic/timw/no-islower-before-toupper' 
* origin/topic/timw/no-islower-before-toupper:
  Statically lookup field offsets for connection values in UDP and ICMP analyzers
  Skip calling islower before toupper
 2025-04-23 12:15:03 -07:00
Tim Wojtulewicz
c2d8bc0620 Statically lookup field offsets for connection values in UDP and ICMP analyzers 2025-04-23 12:14:02 -07:00
Tim Wojtulewicz
928b648f93 Skip calling islower before toupper 2025-04-23 12:12:36 -07:00
Tim Wojtulewicz
a26f380fff Merge remote-tracking branch 'origin/topic/timw/update-all-submodules' 
* origin/topic/timw/update-all-submodules:
  Updating submodule(s) [nomail]
 2025-04-23 10:40:47 -07:00
Tim Wojtulewicz
023d69f929 Updating submodule(s) [nomail] 2025-04-23 10:40:07 -07:00
Arne Welzel
c27d74b2f9 Merge remote-tracking branch 'origin/topic/awelzel/copying-3rdparty-update' 
* origin/topic/awelzel/copying-3rdparty-update:
  COPYING-3rdparty: Add some missing submodules [skip ci]
 2025-04-23 17:18:40 +02:00
Tim Wojtulewicz
985d6b06fe Merge remote-tracking branch 'origin/topic/robin/bump-spicy' 
* origin/topic/robin/bump-spicy:
  Bump Spicy.
 2025-04-23 08:10:52 -07:00
Tim Wojtulewicz
ef943a29ec Merge remote-tracking branch 'origin/topic/neverlord/broker-store-crash' 
* origin/topic/neverlord/broker-store-crash:
  Fix crash related to Broker stores
 2025-04-23 08:08:11 -07:00
Dominik Charousset
b23869a2cd Fix crash related to Broker stores 2025-04-23 08:07:35 -07:00
Arne Welzel
68bc6111ed Merge remote-tracking branch 'origin/topic/awelzel/fix-listen-websocket-ubsan-port-count-confusion' 
* origin/topic/awelzel/fix-listen-websocket-ubsan-port-count-confusion:
  cluster/cluster.bif: Fix CountVal casted to PortVal
 2025-04-23 15:55:59 +02:00
Arne Welzel
dc428b2da2 cluster/cluster.bif: Fix CountVal casted to PortVal 
Thanks UBSAN!
 2025-04-23 15:48:15 +02:00
Arne Welzel
2b7b42e4df Merge remote-tracking branch 'origin/topic/awelzel/no-more-ubuntu-20.04' 
* origin/topic/awelzel/no-more-ubuntu-20.04:
  CI: No more Ubuntu 20.04 [skip ci]
 2025-04-23 15:08:06 +02:00
Arne Welzel
99f36ce01a CI: No more Ubuntu 20.04 [skip ci] 
Ubuntu 20.04's default Python doesn't deal well with the type annotations
used in btest/Files/wstest.py. Given that it's about to be EOL, just remove it.
 2025-04-23 15:05:23 +02:00
Arne Welzel
f6fae00122 COPYING-3rdparty: Add some missing submodules [skip ci] 2025-04-23 14:57:51 +02:00
Arne Welzel
3b80630996 Merge remote-tracking branch 'origin/topic/awelzel/cluster-broker-hub-websocket-support' 
* origin/topic/awelzel/cluster-broker-hub-websocket-support: (24 commits)
  Add NEWS entry for WebSocket functionality
  btest/cluster: Add broker logging test for sanity
  broker/WebSocketShim: Add tests for endpoint, hubs and subscribers
  broker: Deprecate NodeID(), use SetNodeId()
  cluster/Backend: Do not inline Init()
  cluster: Add Backend::SetNodeId()
  cluster/websocket: Make websocket dispatcher queue size configurable
  cluster/zeromq: Attempt publish during termination
  broker/main: Reference Cluster::publish() for auto_publish() deprecation
  broker/main: Deprecate Broker::listen_websocket()
  cluster/websocket: Remove comment about broker
  btest/cluster/websocket: Add tests using broker
  btest/cluster/websocket: Move ZeroMQ test and use wstest.py
  btest/files: Introduce wstest.py
  cluster/websocket: Special case broker backend for shim usage
  broker: Add WebSocketShim backend
  broker/Manager: Add MakeHub() and ReleaseHub()
  Bump broker submodule
  cluster/Backend: Add ProcessError()
  cluster/Backend: Rename EnqueueLocalEvent() to ProcessLocalEvent()
  ...
 2025-04-23 14:29:33 +02:00
Arne Welzel
74f171801c Add NEWS entry for WebSocket functionality 2025-04-23 14:27:43 +02:00
Arne Welzel
f2e60fdaff btest/cluster: Add broker logging test for sanity 
Not very related to the PR, but created to help provoke an issue
with the broker changes.
 2025-04-23 14:27:43 +02:00
Arne Welzel
7acedd18d0 broker/WebSocketShim: Add tests for endpoint, hubs and subscribers 
These are really testing broker functionality, mostly added to ensure
the behavior is as expected by the WebSocketShim functionality.
 2025-04-23 14:27:43 +02:00
Arne Welzel
cb243e2d28 broker: Deprecate NodeID(), use SetNodeId() 2025-04-23 14:27:43 +02:00
Arne Welzel
bfb033622f cluster/Backend: Do not inline Init() 2025-04-23 14:27:43 +02:00
Arne Welzel
a90a41a8b5 cluster: Add Backend::SetNodeId() 2025-04-23 14:27:43 +02:00
Arne Welzel
011029addc cluster/websocket: Make websocket dispatcher queue size configurable 
Limit the number WebSocket events queued from external clients to
dispatcher instances to produce back pressure to the clients if
Zeek's IO loop is overloaded.
 2025-04-23 14:27:43 +02:00
Arne Welzel
6bd624d9b2 cluster/zeromq: Attempt publish during termination 
Explicitly notify the internal thread about the shutdown via the
inproc socket pair. This ensures that the internal thread processes
all previous messages on the inproc socket before terminating.

This fixes the scenario where a backend is created, a few messages published
and then immediately terminated as can be done with WebSocket clients.
Previously, some of the messages published might have still been in the
inproc socket's queue and were simply discarded.

Adds the same test for Broker and ZeroMQ backends.
 2025-04-23 14:27:43 +02:00
Arne Welzel
ab25e5d24b broker/main: Reference Cluster::publish() for auto_publish() deprecation 
In hindsight, this is the better thing to do and with Zeek 7.2 we should
be confident enough that it'll work.
 2025-04-23 14:27:43 +02:00
Arne Welzel
a7423104e1 broker/main: Deprecate Broker::listen_websocket() 
Optimistically deprecate Broker::listen_websocket() and promote
Cluster::listen_websocket() instead.
 2025-04-23 14:27:43 +02:00
Arne Welzel
b9569cb06f cluster/websocket: Remove comment about broker 2025-04-23 14:27:43 +02:00
Arne Welzel
0c8f52664d btest/cluster/websocket: Add tests using broker 
Add tests to verify Cluster::listen_websocket() with the Broker backend
is functional.
 2025-04-23 14:27:43 +02:00
Arne Welzel
3319615c65 btest/cluster/websocket: Move ZeroMQ test and use wstest.py 
Adapt the test to be the same as Broker, to have "expected" behavior.
 2025-04-23 14:27:43 +02:00
Arne Welzel
1191f6b66d btest/files: Introduce wstest.py 
This adds a minimal helper library for reusing some of the code to
test WebSocket client access to Zeek using Python.
 2025-04-23 14:27:43 +02:00
Arne Welzel
193350483e cluster/websocket: Special case broker backend for shim usage 
When Cluster::backend is configured with CLUSTER_BACKEND_BROKER, switch
WebSocketClients to CLUSTER_BACKEND_BROKER_WEBSOCKET_SHIM instead.

Instead of the special case, we could also add something to Backend
called NewWebSocketBackend(), but if it only affects broker, I think
the special case is okay for now.
 2025-04-23 14:27:43 +02:00
Arne Welzel
76c508f001 broker: Add WebSocketShim backend 
This adds a cluster backend implementation using broker's hub primitive
to connect WebSocket clients with the local broker endpoint for pub/sub
functionality.
 2025-04-23 14:27:43 +02:00
Arne Welzel
591e3400d4 broker/Manager: Add MakeHub() and ReleaseHub() 
These are used by WebSocket clients to create broker::hub instances
 2025-04-23 14:27:43 +02:00
Arne Welzel
c9d7418a23 Bump broker submodule 2025-04-23 14:27:43 +02:00
Arne Welzel
3d3b7a0759 cluster/Backend: Add ProcessError() 
Allow backends to pass errors to a strategy. Locally, these raise
Cluster::Backend::error() events that are logged to the reporter
as errors.
 2025-04-23 14:19:08 +02:00
Arne Welzel
fcc0f45c57 cluster/Backend: Rename EnqueueLocalEvent() to ProcessLocalEvent() 2025-04-23 14:19:08 +02:00
Arne Welzel
f8ef5addaa cluster/Backend: Rename HandleRemoteEvent() to ProcessEvent() 
...also add Backend::ProcessEvent() for backends to trigger event
processing without needing to use ProcessEventMessage().
 2025-04-23 14:19:08 +02:00
Arne Welzel
390a4fc1bb cluster/websocket: Fix websocket_client_added id value 2025-04-23 14:19:08 +02:00