Mirror/zeek
1
0
Fork 0
mirror of https://github.com/zeek/zeek.git synced 2025-10-02 06:38:20 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 1
18149 commits 387 branches 195 tags 255 MiB
Commit graph

18149 commits

This branch
This branch
All branches
Author SHA1 Message Date
Tim Wojtulewicz
c4613cf573 Merge remote-tracking branch 'origin/topic/timw/storage-framework-script-docs-updates' 
* origin/topic/timw/storage-framework-script-docs-updates:
  Minor changes to storage framework script docs
 2025-04-25 06:40:54 -07:00
Evan Typanski
154ee7720e Merge remote-tracking branch 'origin/topic/etyp/spicy-bump' 
* origin/topic/etyp/spicy-bump:
  Bump Spicy
 2025-04-25 08:41:02 -04:00
Evan Typanski
e98aae8b5f Bump Spicy 2025-04-25 13:07:02 +02:00
Arne Welzel
a852ecf913 Merge remote-tracking branch 'origin/topic/awelzel/backend-ready-callback-logic' 
* origin/topic/awelzel/backend-ready-callback-logic:
  btest/cluster/websocket: Move no-subscriptions test
  cluster/websocket: Leverage ReadyToPublishCallback()
  cluster/zeromq: Implement DoReadyToPublishCallback()
  cluster/Backend: Add ReadyToPublishCallback() API
 2025-04-25 10:06:36 +00:00
Arne Welzel
43a1bab960 btest/cluster/websocket: Move no-subscriptions test 
...and also add one for broker.
 2025-04-25 10:01:23 +00:00
Arne Welzel
2cd2a2b8a6 cluster/websocket: Leverage ReadyToPublishCallback() 
Change WebSocket client handling to return only when the ready to
publish callback has been invoked.
 2025-04-25 09:57:06 +00:00
Arne Welzel
643b926625 cluster/zeromq: Implement DoReadyToPublishCallback() 
The ZeroMQ heuristic for "ready to publish" is to create an unique and
ephemeral subscription using the XSUB socket and observe it arrive on the
XPUB socket. At this point, visibility into other node's subscriptions
is provided.
 2025-04-25 09:57:06 +00:00
Arne Welzel
e7a876da35 cluster/Backend: Add ReadyToPublishCallback() API 
Provide a mechanism to allow a cluster backend report when it is ready
for publish operations. This is primarily useful for ZeroMQ which has
sender-side filtering and is only really ready for publishing when it
has learned about subscriptions from other nodes.
 2025-04-25 09:57:06 +00:00
Arne Welzel
b0ecc131d0 Merge remote-tracking branch 'origin/topic/awelzel/comment-out-broker-websocket-shim-two-endpoint-tests' 
* origin/topic/awelzel/comment-out-broker-websocket-shim-two-endpoint-tests:
  broker/WebSocketShim/tests: Comment out two endpoint tests
  broker/WebSocketShim/tests: Replace hard-coded timeout values with vars
 2025-04-25 09:03:14 +02:00
Tim Wojtulewicz
3ab83a3f74 Minor changes to storage framework script docs 2025-04-24 11:11:08 -07:00
Arne Welzel
63a75c26c4 broker/WebSocketShim/tests: Comment out two endpoint tests 
Running the remote tests on a loaded system results in timeouts, even
after bumping the tiemouts to 10 seconds. Comment them out for now.
 2025-04-24 19:19:58 +02:00
Arne Welzel
8030ecf893 broker/WebSocketShim/tests: Replace hard-coded timeout values with vars 2025-04-24 19:19:58 +02:00
Arne Welzel
69a1ad2c3d Merge remote-tracking branch 'origin/topic/awelzel/cluster-fix-tsan-zeromq-do-terminate' 
* origin/topic/awelzel/cluster-fix-tsan-zeromq-do-terminate:
  NEWS: Add entry about WebSocket client events
  btest/cluster: Testing cleanup
  cluster/websocket: Raise websocket_client_lost() after terminate
  cluster/ThreadedBackend: Invoke onloop->Process() during DoTerminate()
  cluster/ThreadedBackend: Remove Process()
  zeromq: Call super class DoTerminate() after stopping thread
 2025-04-24 14:04:11 +02:00
Arne Welzel
7513d0ef1b NEWS: Add entry about WebSocket client events 2025-04-24 09:50:04 +02:00
Arne Welzel
2a6beae50b btest/cluster: Testing cleanup 2025-04-24 09:35:53 +02:00
Arne Welzel
63723cd9e3 cluster/websocket: Raise websocket_client_lost() after terminate 
Just in case events are created during backend->Terminate(). These
should come before the Cluster::websocket_client_lost() event.
 2025-04-24 09:35:20 +02:00
Arne Welzel
eb2eaee284 cluster/ThreadedBackend: Invoke onloop->Process() during DoTerminate() 
Also, document how to use ThreadedBackend's DoTerminate()
 2025-04-24 09:35:20 +02:00
Arne Welzel
85a2694cb2 cluster/ThreadedBackend: Remove Process() 
This must have been left-over from before OnLoopProcess existed. It
wasn't called or used anymore.
 2025-04-24 09:35:20 +02:00
Arne Welzel
63d31d7d9f zeromq: Call super class DoTerminate() after stopping thread 
The internal ZeroMQ thread would call QueueForProcessing() thereby
accessing the onloop member. As ThreadedBackend::DoTerminate() unsets it,
this was a) reported as a data race by TSAN and b) potentially caused
missed events that were still to be queued.
 2025-04-24 09:35:20 +02:00
Arne Welzel
79c4fdb237 Merge remote-tracking branch 'origin/topic/awelzel/websocket-empty-subscriptions' 
* origin/topic/awelzel/websocket-empty-subscriptions:
  cluster/websocket: Short-circuit clients without subscriptions
  cluster/websocket: Factor out active subscription handling
 2025-04-24 08:17:46 +02:00
Arne Welzel
23f0370e91 cluster/websocket: Short-circuit clients without subscriptions 2025-04-24 08:14:56 +02:00
Arne Welzel
47206d6a8a cluster/websocket: Factor out active subscription handling 2025-04-24 08:14:56 +02:00
zeek-bot
dee6f1421a Update doc submodule [nomail] [skip ci] 2025-04-24 00:26:49 +00:00
Tim Wojtulewicz
8054547712 Merge remote-tracking branch 'origin/topic/timw/no-islower-before-toupper' 
* origin/topic/timw/no-islower-before-toupper:
  Statically lookup field offsets for connection values in UDP and ICMP analyzers
  Skip calling islower before toupper
 2025-04-23 12:15:03 -07:00
Tim Wojtulewicz
c2d8bc0620 Statically lookup field offsets for connection values in UDP and ICMP analyzers 2025-04-23 12:14:02 -07:00
Tim Wojtulewicz
928b648f93 Skip calling islower before toupper 2025-04-23 12:12:36 -07:00
Tim Wojtulewicz
a26f380fff Merge remote-tracking branch 'origin/topic/timw/update-all-submodules' 
* origin/topic/timw/update-all-submodules:
  Updating submodule(s) [nomail]
 2025-04-23 10:40:47 -07:00
Tim Wojtulewicz
023d69f929 Updating submodule(s) [nomail] 2025-04-23 10:40:07 -07:00
Arne Welzel
c27d74b2f9 Merge remote-tracking branch 'origin/topic/awelzel/copying-3rdparty-update' 
* origin/topic/awelzel/copying-3rdparty-update:
  COPYING-3rdparty: Add some missing submodules [skip ci]
 2025-04-23 17:18:40 +02:00
Tim Wojtulewicz
985d6b06fe Merge remote-tracking branch 'origin/topic/robin/bump-spicy' 
* origin/topic/robin/bump-spicy:
  Bump Spicy.
 2025-04-23 08:10:52 -07:00
Tim Wojtulewicz
ef943a29ec Merge remote-tracking branch 'origin/topic/neverlord/broker-store-crash' 
* origin/topic/neverlord/broker-store-crash:
  Fix crash related to Broker stores
 2025-04-23 08:08:11 -07:00
Dominik Charousset
b23869a2cd Fix crash related to Broker stores 2025-04-23 08:07:35 -07:00
Arne Welzel
68bc6111ed Merge remote-tracking branch 'origin/topic/awelzel/fix-listen-websocket-ubsan-port-count-confusion' 
* origin/topic/awelzel/fix-listen-websocket-ubsan-port-count-confusion:
  cluster/cluster.bif: Fix CountVal casted to PortVal
 2025-04-23 15:55:59 +02:00
Arne Welzel
dc428b2da2 cluster/cluster.bif: Fix CountVal casted to PortVal 
Thanks UBSAN!
 2025-04-23 15:48:15 +02:00
Arne Welzel
2b7b42e4df Merge remote-tracking branch 'origin/topic/awelzel/no-more-ubuntu-20.04' 
* origin/topic/awelzel/no-more-ubuntu-20.04:
  CI: No more Ubuntu 20.04 [skip ci]
 2025-04-23 15:08:06 +02:00
Arne Welzel
99f36ce01a CI: No more Ubuntu 20.04 [skip ci] 
Ubuntu 20.04's default Python doesn't deal well with the type annotations
used in btest/Files/wstest.py. Given that it's about to be EOL, just remove it.
 2025-04-23 15:05:23 +02:00
Arne Welzel
f6fae00122 COPYING-3rdparty: Add some missing submodules [skip ci] 2025-04-23 14:57:51 +02:00
Arne Welzel
3b80630996 Merge remote-tracking branch 'origin/topic/awelzel/cluster-broker-hub-websocket-support' 
* origin/topic/awelzel/cluster-broker-hub-websocket-support: (24 commits)
  Add NEWS entry for WebSocket functionality
  btest/cluster: Add broker logging test for sanity
  broker/WebSocketShim: Add tests for endpoint, hubs and subscribers
  broker: Deprecate NodeID(), use SetNodeId()
  cluster/Backend: Do not inline Init()
  cluster: Add Backend::SetNodeId()
  cluster/websocket: Make websocket dispatcher queue size configurable
  cluster/zeromq: Attempt publish during termination
  broker/main: Reference Cluster::publish() for auto_publish() deprecation
  broker/main: Deprecate Broker::listen_websocket()
  cluster/websocket: Remove comment about broker
  btest/cluster/websocket: Add tests using broker
  btest/cluster/websocket: Move ZeroMQ test and use wstest.py
  btest/files: Introduce wstest.py
  cluster/websocket: Special case broker backend for shim usage
  broker: Add WebSocketShim backend
  broker/Manager: Add MakeHub() and ReleaseHub()
  Bump broker submodule
  cluster/Backend: Add ProcessError()
  cluster/Backend: Rename EnqueueLocalEvent() to ProcessLocalEvent()
  ...
 2025-04-23 14:29:33 +02:00
Arne Welzel
74f171801c Add NEWS entry for WebSocket functionality 2025-04-23 14:27:43 +02:00
Arne Welzel
f2e60fdaff btest/cluster: Add broker logging test for sanity 
Not very related to the PR, but created to help provoke an issue
with the broker changes.
 2025-04-23 14:27:43 +02:00
Arne Welzel
7acedd18d0 broker/WebSocketShim: Add tests for endpoint, hubs and subscribers 
These are really testing broker functionality, mostly added to ensure
the behavior is as expected by the WebSocketShim functionality.
 2025-04-23 14:27:43 +02:00
Arne Welzel
cb243e2d28 broker: Deprecate NodeID(), use SetNodeId() 2025-04-23 14:27:43 +02:00
Arne Welzel
bfb033622f cluster/Backend: Do not inline Init() 2025-04-23 14:27:43 +02:00
Arne Welzel
a90a41a8b5 cluster: Add Backend::SetNodeId() 2025-04-23 14:27:43 +02:00
Arne Welzel
011029addc cluster/websocket: Make websocket dispatcher queue size configurable 
Limit the number WebSocket events queued from external clients to
dispatcher instances to produce back pressure to the clients if
Zeek's IO loop is overloaded.
 2025-04-23 14:27:43 +02:00
Arne Welzel
6bd624d9b2 cluster/zeromq: Attempt publish during termination 
Explicitly notify the internal thread about the shutdown via the
inproc socket pair. This ensures that the internal thread processes
all previous messages on the inproc socket before terminating.

This fixes the scenario where a backend is created, a few messages published
and then immediately terminated as can be done with WebSocket clients.
Previously, some of the messages published might have still been in the
inproc socket's queue and were simply discarded.

Adds the same test for Broker and ZeroMQ backends.
 2025-04-23 14:27:43 +02:00
Arne Welzel
ab25e5d24b broker/main: Reference Cluster::publish() for auto_publish() deprecation 
In hindsight, this is the better thing to do and with Zeek 7.2 we should
be confident enough that it'll work.
 2025-04-23 14:27:43 +02:00
Arne Welzel
a7423104e1 broker/main: Deprecate Broker::listen_websocket() 
Optimistically deprecate Broker::listen_websocket() and promote
Cluster::listen_websocket() instead.
 2025-04-23 14:27:43 +02:00
Arne Welzel
b9569cb06f cluster/websocket: Remove comment about broker 2025-04-23 14:27:43 +02:00
Arne Welzel
0c8f52664d btest/cluster/websocket: Add tests using broker 
Add tests to verify Cluster::listen_websocket() with the Broker backend
is functional.
 2025-04-23 14:27:43 +02:00