18140 commits

This branch

This branch

All branches

Author	SHA1	Message	Date
Arne Welzel	b0ecc131d0	Merge remote-tracking branch 'origin/topic/awelzel/comment-out-broker-websocket-shim-two-endpoint-tests' ... * origin/topic/awelzel/comment-out-broker-websocket-shim-two-endpoint-tests: broker/WebSocketShim/tests: Comment out two endpoint tests broker/WebSocketShim/tests: Replace hard-coded timeout values with vars	2025-04-25 09:03:14 +02:00
Arne Welzel	63a75c26c4	broker/WebSocketShim/tests: Comment out two endpoint tests ... Running the remote tests on a loaded system results in timeouts, even after bumping the tiemouts to 10 seconds. Comment them out for now.	2025-04-24 19:19:58 +02:00
Arne Welzel	8030ecf893	broker/WebSocketShim/tests: Replace hard-coded timeout values with vars	2025-04-24 19:19:58 +02:00
Arne Welzel	69a1ad2c3d	Merge remote-tracking branch 'origin/topic/awelzel/cluster-fix-tsan-zeromq-do-terminate' ... * origin/topic/awelzel/cluster-fix-tsan-zeromq-do-terminate: NEWS: Add entry about WebSocket client events btest/cluster: Testing cleanup cluster/websocket: Raise websocket_client_lost() after terminate cluster/ThreadedBackend: Invoke onloop->Process() during DoTerminate() cluster/ThreadedBackend: Remove Process() zeromq: Call super class DoTerminate() after stopping thread	2025-04-24 14:04:11 +02:00
Arne Welzel	7513d0ef1b	NEWS: Add entry about WebSocket client events	2025-04-24 09:50:04 +02:00
Arne Welzel	2a6beae50b	btest/cluster: Testing cleanup	2025-04-24 09:35:53 +02:00
Arne Welzel	63723cd9e3	cluster/websocket: Raise websocket_client_lost() after terminate ... Just in case events are created during backend->Terminate(). These should come before the Cluster::websocket_client_lost() event.	2025-04-24 09:35:20 +02:00
Arne Welzel	eb2eaee284	cluster/ThreadedBackend: Invoke onloop->Process() during DoTerminate() ... Also, document how to use ThreadedBackend's DoTerminate()	2025-04-24 09:35:20 +02:00
Arne Welzel	85a2694cb2	cluster/ThreadedBackend: Remove Process() ... This must have been left-over from before OnLoopProcess existed. It wasn't called or used anymore.	2025-04-24 09:35:20 +02:00
Arne Welzel	63d31d7d9f	zeromq: Call super class DoTerminate() after stopping thread ... The internal ZeroMQ thread would call QueueForProcessing() thereby accessing the onloop member. As ThreadedBackend::DoTerminate() unsets it, this was a) reported as a data race by TSAN and b) potentially caused missed events that were still to be queued.	2025-04-24 09:35:20 +02:00
Arne Welzel	79c4fdb237	Merge remote-tracking branch 'origin/topic/awelzel/websocket-empty-subscriptions' ... * origin/topic/awelzel/websocket-empty-subscriptions: cluster/websocket: Short-circuit clients without subscriptions cluster/websocket: Factor out active subscription handling	2025-04-24 08:17:46 +02:00
Arne Welzel	23f0370e91	cluster/websocket: Short-circuit clients without subscriptions	2025-04-24 08:14:56 +02:00
Arne Welzel	47206d6a8a	cluster/websocket: Factor out active subscription handling	2025-04-24 08:14:56 +02:00
zeek-bot	dee6f1421a	Update doc submodule [nomail] [skip ci]	2025-04-24 00:26:49 +00:00
Tim Wojtulewicz	8054547712	Merge remote-tracking branch 'origin/topic/timw/no-islower-before-toupper' ... * origin/topic/timw/no-islower-before-toupper: Statically lookup field offsets for connection values in UDP and ICMP analyzers Skip calling islower before toupper	2025-04-23 12:15:03 -07:00
Tim Wojtulewicz	c2d8bc0620	Statically lookup field offsets for connection values in UDP and ICMP analyzers	2025-04-23 12:14:02 -07:00
Tim Wojtulewicz	928b648f93	Skip calling islower before toupper	2025-04-23 12:12:36 -07:00
Tim Wojtulewicz	a26f380fff	Merge remote-tracking branch 'origin/topic/timw/update-all-submodules' ... * origin/topic/timw/update-all-submodules: Updating submodule(s) [nomail]	2025-04-23 10:40:47 -07:00
Tim Wojtulewicz	023d69f929	Updating submodule(s) [nomail]	2025-04-23 10:40:07 -07:00
Arne Welzel	c27d74b2f9	Merge remote-tracking branch 'origin/topic/awelzel/copying-3rdparty-update' ... * origin/topic/awelzel/copying-3rdparty-update: COPYING-3rdparty: Add some missing submodules [skip ci]	2025-04-23 17:18:40 +02:00
Tim Wojtulewicz	985d6b06fe	Merge remote-tracking branch 'origin/topic/robin/bump-spicy' ... * origin/topic/robin/bump-spicy: Bump Spicy.	2025-04-23 08:10:52 -07:00
Tim Wojtulewicz	ef943a29ec	Merge remote-tracking branch 'origin/topic/neverlord/broker-store-crash' ... * origin/topic/neverlord/broker-store-crash: Fix crash related to Broker stores	2025-04-23 08:08:11 -07:00
Dominik Charousset	b23869a2cd	Fix crash related to Broker stores	2025-04-23 08:07:35 -07:00
Arne Welzel	68bc6111ed	Merge remote-tracking branch 'origin/topic/awelzel/fix-listen-websocket-ubsan-port-count-confusion' ... * origin/topic/awelzel/fix-listen-websocket-ubsan-port-count-confusion: cluster/cluster.bif: Fix CountVal casted to PortVal	2025-04-23 15:55:59 +02:00
Arne Welzel	dc428b2da2	cluster/cluster.bif: Fix CountVal casted to PortVal ... Thanks UBSAN!	2025-04-23 15:48:15 +02:00
Arne Welzel	2b7b42e4df	Merge remote-tracking branch 'origin/topic/awelzel/no-more-ubuntu-20.04' ... * origin/topic/awelzel/no-more-ubuntu-20.04: CI: No more Ubuntu 20.04 [skip ci]	2025-04-23 15:08:06 +02:00
Arne Welzel	99f36ce01a	CI: No more Ubuntu 20.04 [skip ci] ... Ubuntu 20.04's default Python doesn't deal well with the type annotations used in btest/Files/wstest.py. Given that it's about to be EOL, just remove it.	2025-04-23 15:05:23 +02:00
Arne Welzel	f6fae00122	COPYING-3rdparty: Add some missing submodules [skip ci]	2025-04-23 14:57:51 +02:00
Arne Welzel	3b80630996	Merge remote-tracking branch 'origin/topic/awelzel/cluster-broker-hub-websocket-support' ... * origin/topic/awelzel/cluster-broker-hub-websocket-support: (24 commits) Add NEWS entry for WebSocket functionality btest/cluster: Add broker logging test for sanity broker/WebSocketShim: Add tests for endpoint, hubs and subscribers broker: Deprecate NodeID(), use SetNodeId() cluster/Backend: Do not inline Init() cluster: Add Backend::SetNodeId() cluster/websocket: Make websocket dispatcher queue size configurable cluster/zeromq: Attempt publish during termination broker/main: Reference Cluster::publish() for auto_publish() deprecation broker/main: Deprecate Broker::listen_websocket() cluster/websocket: Remove comment about broker btest/cluster/websocket: Add tests using broker btest/cluster/websocket: Move ZeroMQ test and use wstest.py btest/files: Introduce wstest.py cluster/websocket: Special case broker backend for shim usage broker: Add WebSocketShim backend broker/Manager: Add MakeHub() and ReleaseHub() Bump broker submodule cluster/Backend: Add ProcessError() cluster/Backend: Rename EnqueueLocalEvent() to ProcessLocalEvent() ...	2025-04-23 14:29:33 +02:00
Arne Welzel	74f171801c	Add NEWS entry for WebSocket functionality	2025-04-23 14:27:43 +02:00
Arne Welzel	f2e60fdaff	btest/cluster: Add broker logging test for sanity ... Not very related to the PR, but created to help provoke an issue with the broker changes.	2025-04-23 14:27:43 +02:00
Arne Welzel	7acedd18d0	broker/WebSocketShim: Add tests for endpoint, hubs and subscribers ... These are really testing broker functionality, mostly added to ensure the behavior is as expected by the WebSocketShim functionality.	2025-04-23 14:27:43 +02:00
Arne Welzel	cb243e2d28	broker: Deprecate NodeID(), use SetNodeId()	2025-04-23 14:27:43 +02:00
Arne Welzel	bfb033622f	cluster/Backend: Do not inline Init()	2025-04-23 14:27:43 +02:00
Arne Welzel	a90a41a8b5	cluster: Add Backend::SetNodeId()	2025-04-23 14:27:43 +02:00
Arne Welzel	011029addc	cluster/websocket: Make websocket dispatcher queue size configurable ... Limit the number WebSocket events queued from external clients to dispatcher instances to produce back pressure to the clients if Zeek's IO loop is overloaded.	2025-04-23 14:27:43 +02:00
Arne Welzel	6bd624d9b2	cluster/zeromq: Attempt publish during termination ... Explicitly notify the internal thread about the shutdown via the inproc socket pair. This ensures that the internal thread processes all previous messages on the inproc socket before terminating. This fixes the scenario where a backend is created, a few messages published and then immediately terminated as can be done with WebSocket clients. Previously, some of the messages published might have still been in the inproc socket's queue and were simply discarded. Adds the same test for Broker and ZeroMQ backends.	2025-04-23 14:27:43 +02:00
Arne Welzel	ab25e5d24b	broker/main: Reference Cluster::publish() for auto_publish() deprecation ... In hindsight, this is the better thing to do and with Zeek 7.2 we should be confident enough that it'll work.	2025-04-23 14:27:43 +02:00
Arne Welzel	a7423104e1	broker/main: Deprecate Broker::listen_websocket() ... Optimistically deprecate Broker::listen_websocket() and promote Cluster::listen_websocket() instead.	2025-04-23 14:27:43 +02:00
Arne Welzel	b9569cb06f	cluster/websocket: Remove comment about broker	2025-04-23 14:27:43 +02:00
Arne Welzel	0c8f52664d	btest/cluster/websocket: Add tests using broker ... Add tests to verify Cluster::listen_websocket() with the Broker backend is functional.	2025-04-23 14:27:43 +02:00
Arne Welzel	3319615c65	btest/cluster/websocket: Move ZeroMQ test and use wstest.py ... Adapt the test to be the same as Broker, to have "expected" behavior.	2025-04-23 14:27:43 +02:00
Arne Welzel	1191f6b66d	btest/files: Introduce wstest.py ... This adds a minimal helper library for reusing some of the code to test WebSocket client access to Zeek using Python.	2025-04-23 14:27:43 +02:00
Arne Welzel	193350483e	cluster/websocket: Special case broker backend for shim usage ... When Cluster::backend is configured with CLUSTER_BACKEND_BROKER, switch WebSocketClients to CLUSTER_BACKEND_BROKER_WEBSOCKET_SHIM instead. Instead of the special case, we could also add something to Backend called NewWebSocketBackend(), but if it only affects broker, I think the special case is okay for now.	2025-04-23 14:27:43 +02:00
Arne Welzel	76c508f001	broker: Add WebSocketShim backend ... This adds a cluster backend implementation using broker's hub primitive to connect WebSocket clients with the local broker endpoint for pub/sub functionality.	2025-04-23 14:27:43 +02:00
Arne Welzel	591e3400d4	broker/Manager: Add MakeHub() and ReleaseHub() ... These are used by WebSocket clients to create broker::hub instances	2025-04-23 14:27:43 +02:00
Arne Welzel	c9d7418a23	Bump broker submodule	2025-04-23 14:27:43 +02:00
Arne Welzel	3d3b7a0759	cluster/Backend: Add ProcessError() ... Allow backends to pass errors to a strategy. Locally, these raise Cluster::Backend::error() events that are logged to the reporter as errors.	2025-04-23 14:19:08 +02:00
Arne Welzel	fcc0f45c57	cluster/Backend: Rename EnqueueLocalEvent() to ProcessLocalEvent()	2025-04-23 14:19:08 +02:00
Arne Welzel	f8ef5addaa	cluster/Backend: Rename HandleRemoteEvent() to ProcessEvent() ... ...also add Backend::ProcessEvent() for backends to trigger event processing without needing to use ProcessEventMessage().	2025-04-23 14:19:08 +02:00