Calling collect_metrics() from a script would not invoke metric
callbacks, resulting in most of the process metrics to be zero
when a Zeek process isn't scraped via Prometheus.
Fixes#4309
The busy polling wasn't clever and usually resulted in delays. For now,
switch to mutex/condition variable and log an error if the timeouts are
immense.
Due to prefix matching, worker-1's node_topic() also matched worker-10,
worker-11, etc. Suffix the node topic with a `.`. The original implementation
came from NATS, where subjects are separated by `.`.
Adapt nodeid_topic() for consistency.
WebSocket clients that connected with the wrong URL do not have
a backend attached. If a dispatcher is terminated while these
clients are still connected, a null deref would happen.
This was found while running all cluster/websocket tests in a loop
for a long time, tickling a segfault during the bad-url test.
These test were very sensible to the speed at which ZeroMQ distributes
subscriptions in the cluster and showed to be unreliably when testing with
zeek/btest#113.
The main fix here is to have individual WebSocket clients subscribe to unique
topics, e.g /test/client-0 and /test/client-1, instead of just a shared topic.
This ensures the WebSocket handshake completes only when they observed their
own subscriptions and not prematurely when observing the shared topic.
This seems mainly relevant for tests: In the real world one shouldn't
rely on subscription visibility - you miss messages if you're too late
to the party.
When two workers connect to zeek.cluster.worker, the central ZeroMQ
proxy would not propagate unsubscription information to other nodes
once they both left. Set ZMQ_XPUB_VERBOSER on the proxies XPUB socket
for visibility.
* origin/topic/timw/storage-framework-followup:
Redis: Handle other errors from requests, fix KEY_EXISTS for put operations
SQLite: handle existing keys when overwrite=F correctly
Remove unnecessary type aliases from storage btests
Avoid thread-leak in scripts.base.frameworks.file-analysis.bifs.enable-disable btest
Fix data-race with calling DBG_LOG from a separate thread
Fix data-race with ReturnCode objects in Sqlite::DoExpire
Fix data race with calling run_state::network_time from a separate thread
Add NEWS entry for Storage, reduce CHANGES spam
Fix Coverity findings in the SQLite backend
Remove inclusion of non-existent expected-lite path during dynamic plugin builds
Squash code from OperationResultCallback into ResultCallback
Add hiredis to generate-docs workflow to enable Redis backend
This btest uses the exit() BIF to shut down, which immediately calls
::exit() and kills Zeek without doing any shutdown. This will sometimes
leave the thread running the storage manager, which causes TSan to
complain about a thread leak. Switch to use the terminate() BIF instead
which cleanly shuts down all of Zeek.
Bump to the latest btest master version and set BTEST_BG_RUN_SLEEP
environment variable and to 0. This makes btest-bg-run return
immediately instead of delaying by 1 second.
* origin/topic/vern/CPP-maint.Mar25:
minor baseline updates for -O gen-C++
-O gen-C++ support for pattern-to-pattern comparison
-O gen-C++ support for pattern vector comparisons
-O gen-C++ support for type expressions
* origin/topic/johanna/ssh-server-banners-can-be-wild:
SSH analyzer - tiny aesthetic fixes
SSH: make banner parsing more robust
SSH: split banner into client/server parts
These functions reside in base/utils/addrs.zeek and were not tested. We don't
use has_valid_octets() anywhere right now, normalize_mac() is used in the RADIUS
analyzer, and addr_to_uri() is used in the signatures/dpd.zeek tests.
* origin/topic/timw/storage-framework: (52 commits)
Update docs submodule [nomail]
Cleanup/update comments across the storage C++ files
Split storage.bif file into events/sync/async, add more comments
Update comments in script files, run zeek-format on all of them
Allow sync methods to be called from when conditions, add related btest
Redis: Handle disconnection correctly via callback
Redis: Fix sync erase, add btest for it
Remove default argument for callbacks, reorder function arguments
Remove file-local expire_running variable
Pass network time down to Expire()
Add IN_PROGRESS return code, handle for async backends
Store sqlite3_stmts directly instead of looking up from a map
Reduce code duplication in storage.bif
Add OperationResult::MakeVal, use it to reduce some code duplication
Rearrange visibility of Backend methods, add DoPoll/DoExpire, add return comments
Implement Storage::backend_opened and Storage::backend_lost events
SQLite: expand expiration test
SQLite: Handle other return values from sqlite3_step
Redis: Fix thread-contention issues with Expire(), add more tests
Change how redis-server is run during btests, removing redis.conf
...
