Commit graph

17954 commits

Author SHA1 Message Date
Tim Wojtulewicz
e95784db16 SQLite: Store/lookup prepared statements instead of recreating 2025-03-18 09:32:34 -07:00
Tim Wojtulewicz
9d1eef3fbc Add basic SQLite storage backend 2025-03-18 09:32:34 -07:00
Tim Wojtulewicz
7ad6a05f5b Add infrastructure for asynchronous storage operations 2025-03-18 09:32:34 -07:00
Tim Wojtulewicz
d07d27453a Add infrastructure for automated expiration of storage entries
This is used for backends that don't support expiration natively.
2025-03-18 09:32:34 -07:00
Tim Wojtulewicz
8dee733a7d Change args to Storage::put to be a record
The number of args being passed to the put() methods was getting to be
fairly long, with more on the horizon. Changing to a record means simplifying
things a little bit.
2025-03-18 09:32:34 -07:00
Tim Wojtulewicz
69d940533d Pass key/value types for validation when opening backends 2025-03-18 09:32:34 -07:00
Tim Wojtulewicz
2ea0f3e70a Lay out initial parts for the Storage framework
This includes a manager, component manager, BIF and script code, and
parts to support new storage backend plugins.
2025-03-18 09:32:34 -07:00
Tim Wojtulewicz
3d6e7c85b0 DebugLogger: add stream for storage 2025-03-18 09:32:34 -07:00
Tim Wojtulewicz
e2b9e81c53 plugin: Add component enum for storage backends 2025-03-18 09:32:34 -07:00
Tim Wojtulewicz
aeedd25cff Add martinmoene/expected-lite as a submodule 2025-03-18 09:32:34 -07:00
Johanna Amann
6023c8b906 SSH: make banner parsing more robust
This change revamps SSH banner parsing.  The previous behavior was both
a bit too strict in some regards, and too permissive in other.

Specifically, clients are now required to send a line starting with
"SSH-" as the first line.  This is in line with the RFC, as well with
observed behavior. This also prevents the creation of `ssh.log` for
non-SSH traffic on port 22.

For the server side, we now accept text before the SSH banner. This
previously led to a protocol violation but is allowed by the spec.

New tests are added to cover these cases.
2025-03-18 16:19:33 +00:00
Robin Sommer
3d25328204
Merge remote-tracking branch 'origin/topic/robin/gh-4250-vec-allocator'
* origin/topic/robin/gh-4250-vec-allocator:
  Spicy: Fix passing vectors to Zeek.
2025-03-18 09:08:50 +01:00
Robin Sommer
993e1a6562
Merge remote-tracking branch 'origin/topic/robin/gh-zeek-4238-spicy-bump'
* origin/topic/robin/gh-zeek-4238-spicy-bump:
  Bump `cmake` submodule.
  Bump Spicy.
  [Spicy] Add missing `to_string` overload.
2025-03-18 09:07:16 +01:00
zeek-bot
189fd0b071 Update doc submodule [nomail] [skip ci] 2025-03-18 00:14:00 +00:00
Tim Wojtulewicz
43f108bb71 Merge remote-tracking branch 'origin/topic/timw/ci-macos-sequoia'
* origin/topic/timw/ci-macos-sequoia:
  ci/init-external-repo.sh: Use regex to match macos cirrus task
  CI: Change macOS runner to Sequoia
2025-03-17 11:55:27 -07:00
Arne Welzel
919176873e ci/init-external-repo.sh: Use regex to match macos cirrus task 2025-03-17 10:48:59 +01:00
Tim Wojtulewicz
46edc829cf CI: Change macOS runner to Sequoia 2025-03-14 18:29:52 -07:00
zeek-bot
8a4041b776 Update doc submodule [nomail] [skip ci] 2025-03-13 00:13:10 +00:00
Johanna Amann
629f2bd03a SSH: split banner into client/server parts
This is prepatatory work and should not lead to functional changes.
Client and server banners can be quite different in practice.
2025-03-12 15:06:31 +00:00
Robin Sommer
33fba2476e
Bump cmake submodule. 2025-03-12 14:34:02 +01:00
Robin Sommer
16ff7f69c2
Bump Spicy.
This includes a fix to adapt to Spicy-side changes.
2025-03-12 13:59:32 +01:00
Robin Sommer
859ec00bd9
[Spicy] Add missing to_string overload.
Spicy now requires this.
2025-03-12 13:59:30 +01:00
Arne Welzel
1b655836be Merge remote-tracking branch 'origin/topic/awelzel/ci-generate-docs-run-pre-commit'
* origin/topic/awelzel/ci-generate-docs-run-pre-commit:
  github/generate-docs: Run pre-commit
  cluster/backends/zeromq: Fix rst link in docs
2025-03-12 13:11:18 +01:00
Arne Welzel
e489b322e5 github/generate-docs: Run pre-commit
...it's a bit unfortunate that we cannot use the action for this. It's
because ./doc is a submodule and not the main repo.
2025-03-12 12:06:49 +01:00
Arne Welzel
cc0c48423d cluster/backends/zeromq: Fix rst link in docs 2025-03-12 10:11:25 +01:00
zeek-bot
f15281ab64 Update doc submodule [nomail] [skip ci] 2025-03-12 00:15:42 +00:00
Arne Welzel
bb58148c64 Merge remote-tracking branch 'origin/topic/awelzel/4136-cluster-websocket-support'
* origin/topic/awelzel/4136-cluster-websocket-support:
  ci/opensuse-tumpleweed: Bust cache
  ci/macos/prepare: Install python@3 explicitly
  cluster/websocket: Implement WebSocket server
  cluster/websocket: Add IXWebsocket submodule
  ci/alpine: Install openssl package for testing
  ci: Install websockets from pip for all distros
  auxil/libunistd: Bump for ssize_t typedef
  auxil/broker: Bump to latest master version
  cluster/zeromq: Catch log_push.send() exception
  cluster/zeromq: Catch exceptions as const zmq::error_t&
  cluster/zeromq: No assert on inproc handling
  cluster/zeromq: Support configuring IO threads for proxy thread
  cluster/zeromq: Move variable lookups from DoInit() to DoInitPostScript()
  cluster/zeromq: Handle EINTR at shutdown
  cluster/zeromq: Queue one message at a time
  cluster/Backend: Queue a single message only
  cluster/zeromq: Adapt for OnLoopProcess changes
  cluster/ThreadedBackend: Switch to OnLoopProcess
  cluster/OnLoop: Introduce helper template class
  serializer/broker: Expose to_broker_event() and to_zeek_event()
2025-03-11 10:51:09 +01:00
Arne Welzel
81acc4509d ci/opensuse-tumpleweed: Bust cache
Got a build failure because the old container images had python3-devel
for Python3.11 or something older, but then prepare.sh would install
Python 3.13 and Zeek's configure failing due to trying to find the
devel headers from python313-devel which wasn't installed by prepare.sh
2025-03-11 10:18:20 +01:00
Arne Welzel
70f5430e7c ci/macos/prepare: Install python@3 explicitly
It seems Homebrew's Python 3.12 doesn't install default symlinks or
python3 symlinks[1]. I believe this results in prepare.sh using the
system's Python rather than Homebrew's. Install python@3 explicitly
to put the symlinks in place.

[1] https://stackoverflow.com/a/77655631
2025-03-11 09:51:01 +01:00
zeek-bot
a62b7ffe4d Update doc submodule [nomail] [skip ci] 2025-03-11 00:14:11 +00:00
Arne Welzel
db28ff04e2 Merge remote-tracking branch 'origin/topic/awelzel/javascript-separate-thread'
* origin/topic/awelzel/javascript-separate-thread:
  btest/javascript: Add file_sniff() and file_state_remove() test
  Bump zeekjs to v0.15.0
2025-03-10 19:42:36 +01:00
Arne Welzel
e2b65acc6d btest/javascript: Add file_sniff() and file_state_remove() test
Using file_state_remove() or file_sniff() would previously crash with the
spicy-zip file analyzer. We don't have spicy-zip here, so it's not a
proper reproducer, but still makes sense to test these events.
2025-03-10 19:15:06 +01:00
Arne Welzel
cf154e02dd Bump zeekjs to v0.15.0
f5ff7c4 version: 0.15.0
    2a48143 docker/fedora: Bust cache and ensure sqlite-libs is updated
    05e5f57 Executor: Unblock SIGTERM in thread
    46496b9 Instance: Remove unsued variable
    c3b3c86 Instance: Offload JavaScript invocations to executor thread
    09e311f Introduce Executor helper

Mainly, run JavaScript on a separate thread for interoperability with
Spicy fiber stacks.

Closes #4239
2025-03-10 19:14:45 +01:00
Arne Welzel
6032741868 cluster/websocket: Implement WebSocket server 2025-03-10 17:07:30 +01:00
Arne Welzel
1e757b2b59 cluster/websocket: Add IXWebsocket submodule 2025-03-10 17:07:30 +01:00
Arne Welzel
0b49eac057 ci/alpine: Install openssl package for testing 2025-03-10 17:07:30 +01:00
Arne Welzel
9f768d1896 ci: Install websockets from pip for all distros
The cluster/websocket tests were developed against websockets 14.2,
but Ubuntu and Alpine ship too old versions. Switch to installing
the latest version from pip instead, so we don't need to bother making
tests compatible with very old Python packages shipped by distributions.
2025-03-10 17:07:30 +01:00
Arne Welzel
91eb2786be auxil/libunistd: Bump for ssize_t typedef 2025-03-10 17:07:30 +01:00
Arne Welzel
3a0216728c auxil/broker: Bump to latest master version 2025-03-10 17:07:30 +01:00
Arne Welzel
eb1f9f9a42 cluster/zeromq: Catch log_push.send() exception 2025-03-10 17:07:30 +01:00
Arne Welzel
b82dcfafa4 cluster/zeromq: Catch exceptions as const zmq::error_t& 2025-03-10 17:07:30 +01:00
Arne Welzel
8a1abfa8ef cluster/zeromq: No assert on inproc handling
This might happen if we didn't succeed in completely sending a multipart
message and stop early.
2025-03-10 17:07:30 +01:00
Arne Welzel
aad512c616 cluster/zeromq: Support configuring IO threads for proxy thread 2025-03-10 17:07:30 +01:00
Arne Welzel
ba7b605a97 cluster/zeromq: Move variable lookups from DoInit() to DoInitPostScript() 2025-03-10 17:07:30 +01:00
Arne Welzel
540d9da5ef cluster/zeromq: Handle EINTR at shutdown
Read ::signal_val and early exit a DoPublish() in case termination
happened while blocked in inproc.send()
2025-03-10 17:07:30 +01:00
Arne Welzel
94ec3af2b0 cluster/zeromq: Queue one message at a time
Queueing multiple messages can easily overload the IO loop without
creating any backpressure.
2025-03-10 17:07:30 +01:00
Arne Welzel
09ccb2e250 cluster/Backend: Queue a single message only
The ZeroMQ backend would accumulate multiple messages and enqueue them
all at once. However, as this could potentially result in huge batches
of events being queued into the event loop at once, switch to a one
message at a time model. If there's too many messages queued already,
OnLoop::QueueForProcessing() will block the ZeroMQ thread until
there's room available again.
2025-03-10 17:07:30 +01:00
Arne Welzel
827eccb732 cluster/zeromq: Adapt for OnLoopProcess changes 2025-03-10 17:07:30 +01:00
Arne Welzel
23405194a0 cluster/ThreadedBackend: Switch to OnLoopProcess 2025-03-10 17:07:30 +01:00
Arne Welzel
5dee77e6f2 cluster/OnLoop: Introduce helper template class 2025-03-10 17:07:30 +01:00