Tim Wojtulewicz
b81e876ec8
Change how redis-server is run during btests, removing redis.conf
2025-03-18 10:20:33 -07:00
Tim Wojtulewicz
9ed3e33f97
Completely rework return values from storage operations
2025-03-18 10:20:33 -07:00
Tim Wojtulewicz
8ddda016ff
Update some btests due to timing changes
2025-03-18 10:20:33 -07:00
Tim Wojtulewicz
e766af7322
Split sync/async handling into the BIF methods
2025-03-18 10:20:33 -07:00
Tim Wojtulewicz
c247de8ec3
Redis: Rework everything to only use async mode
2025-03-18 10:20:33 -07:00
Tim Wojtulewicz
40f60f26b3
Run expiration on a separate thread
2025-03-18 10:20:33 -07:00
Tim Wojtulewicz
cad48cebd4
Pass network-time-based expiration time to backends instead of an interval
2025-03-18 10:20:33 -07:00
Tim Wojtulewicz
a485b1d237
Make backend options a record, move actual options to be sub-records
2025-03-18 10:20:33 -07:00
Tim Wojtulewicz
64f3969434
Always register backend for expiration, check for open during loop
2025-03-18 10:20:33 -07:00
Tim Wojtulewicz
28951dccf1
Split sync and async into separate script-land namespaces
2025-03-18 10:20:33 -07:00
Tim Wojtulewicz
e8074c40d4
Remove Backend::SupportsAsync
2025-03-18 09:32:34 -07:00
Tim Wojtulewicz
42ad5bbf7d
Add btest that uses a Redis backend in a cluster
2025-03-18 09:32:34 -07:00
Tim Wojtulewicz
f1a7376e0a
Return generic result for get operations that includes error messages
2025-03-18 09:32:34 -07:00
Tim Wojtulewicz
4695060d75
Allow opening and closing backends to be async
2025-03-18 09:32:34 -07:00
Tim Wojtulewicz
ea87c773cd
Redis: Support non-native expiration when reading traces
2025-03-18 09:32:34 -07:00
Tim Wojtulewicz
08bebaa426
Redis: Add btests for the redis backend
2025-03-18 09:32:34 -07:00
Tim Wojtulewicz
52d94b781a
Redis: Force storage sync mode when reading pcaps, default to async mode
2025-03-18 09:32:34 -07:00
Tim Wojtulewicz
31e146b16d
Redis: Add new backend
2025-03-18 09:32:34 -07:00
Tim Wojtulewicz
6289eb8e15
SQLite: Fix some issues with expiration, including in the btest
2025-03-18 09:32:34 -07:00
Tim Wojtulewicz
6bc5f70236
SQLite: Add additional btests, which also cover general storage functionality
...
- New erase/overwrite tests
- Change existing sqlite-basic test to use async
- Test passing bad keys to validate backend type checking
- New test for compound keys and values
2025-03-18 09:32:34 -07:00
Tim Wojtulewicz
b2bcb19b22
SQLite: Add pragma integrity_check
2025-03-18 09:32:34 -07:00
Tim Wojtulewicz
3e8ff836aa
SQLite: Add tuning options to configuration
2025-03-18 09:32:34 -07:00
Tim Wojtulewicz
ec49f5d550
SQLite: Handle automated expiration
2025-03-18 09:32:34 -07:00
Tim Wojtulewicz
e95784db16
SQLite: Store/lookup prepared statements instead of recreating
2025-03-18 09:32:34 -07:00
Tim Wojtulewicz
9d1eef3fbc
Add basic SQLite storage backend
2025-03-18 09:32:34 -07:00
Tim Wojtulewicz
7ad6a05f5b
Add infrastructure for asynchronous storage operations
2025-03-18 09:32:34 -07:00
Tim Wojtulewicz
d07d27453a
Add infrastructure for automated expiration of storage entries
...
This is used for backends that don't support expiration natively.
2025-03-18 09:32:34 -07:00
Tim Wojtulewicz
8dee733a7d
Change args to Storage::put to be a record
...
The number of args being passed to the put() methods was getting to be
fairly long, with more on the horizon. Changing to a record means simplifying
things a little bit.
2025-03-18 09:32:34 -07:00
Tim Wojtulewicz
69d940533d
Pass key/value types for validation when opening backends
2025-03-18 09:32:34 -07:00
Tim Wojtulewicz
2ea0f3e70a
Lay out initial parts for the Storage framework
...
This includes a manager, component manager, BIF and script code, and
parts to support new storage backend plugins.
2025-03-18 09:32:34 -07:00
Tim Wojtulewicz
3d6e7c85b0
DebugLogger: add stream for storage
2025-03-18 09:32:34 -07:00
Tim Wojtulewicz
e2b9e81c53
plugin: Add component enum for storage backends
2025-03-18 09:32:34 -07:00
Tim Wojtulewicz
aeedd25cff
Add martinmoene/expected-lite as a submodule
2025-03-18 09:32:34 -07:00
Robin Sommer
3d25328204
Merge remote-tracking branch 'origin/topic/robin/gh-4250-vec-allocator'
...
* origin/topic/robin/gh-4250-vec-allocator:
Spicy: Fix passing vectors to Zeek.
2025-03-18 09:08:50 +01:00
Robin Sommer
993e1a6562
Merge remote-tracking branch 'origin/topic/robin/gh-zeek-4238-spicy-bump'
...
* origin/topic/robin/gh-zeek-4238-spicy-bump:
Bump `cmake` submodule.
Bump Spicy.
[Spicy] Add missing `to_string` overload.
2025-03-18 09:07:16 +01:00
zeek-bot
189fd0b071
Update doc submodule [nomail] [skip ci]
2025-03-18 00:14:00 +00:00
Tim Wojtulewicz
43f108bb71
Merge remote-tracking branch 'origin/topic/timw/ci-macos-sequoia'
...
* origin/topic/timw/ci-macos-sequoia:
ci/init-external-repo.sh: Use regex to match macos cirrus task
CI: Change macOS runner to Sequoia
2025-03-17 11:55:27 -07:00
Arne Welzel
919176873e
ci/init-external-repo.sh: Use regex to match macos cirrus task
2025-03-17 10:48:59 +01:00
Tim Wojtulewicz
46edc829cf
CI: Change macOS runner to Sequoia
2025-03-14 18:29:52 -07:00
zeek-bot
8a4041b776
Update doc submodule [nomail] [skip ci]
2025-03-13 00:13:10 +00:00
Robin Sommer
33fba2476e
Bump cmake submodule.
2025-03-12 14:34:02 +01:00
Robin Sommer
16ff7f69c2
Bump Spicy.
...
This includes a fix to adapt to Spicy-side changes.
2025-03-12 13:59:32 +01:00
Robin Sommer
859ec00bd9
[Spicy] Add missing to_string overload.
...
Spicy now requires this.
2025-03-12 13:59:30 +01:00
Arne Welzel
1b655836be
Merge remote-tracking branch 'origin/topic/awelzel/ci-generate-docs-run-pre-commit'
...
* origin/topic/awelzel/ci-generate-docs-run-pre-commit:
github/generate-docs: Run pre-commit
cluster/backends/zeromq: Fix rst link in docs
2025-03-12 13:11:18 +01:00
Arne Welzel
e489b322e5
github/generate-docs: Run pre-commit
...
...it's a bit unfortunate that we cannot use the action for this. It's
because ./doc is a submodule and not the main repo.
2025-03-12 12:06:49 +01:00
Arne Welzel
cc0c48423d
cluster/backends/zeromq: Fix rst link in docs
2025-03-12 10:11:25 +01:00
zeek-bot
f15281ab64
Update doc submodule [nomail] [skip ci]
2025-03-12 00:15:42 +00:00
Arne Welzel
bb58148c64
Merge remote-tracking branch 'origin/topic/awelzel/4136-cluster-websocket-support'
...
* origin/topic/awelzel/4136-cluster-websocket-support:
ci/opensuse-tumpleweed: Bust cache
ci/macos/prepare: Install python@3 explicitly
cluster/websocket: Implement WebSocket server
cluster/websocket: Add IXWebsocket submodule
ci/alpine: Install openssl package for testing
ci: Install websockets from pip for all distros
auxil/libunistd: Bump for ssize_t typedef
auxil/broker: Bump to latest master version
cluster/zeromq: Catch log_push.send() exception
cluster/zeromq: Catch exceptions as const zmq::error_t&
cluster/zeromq: No assert on inproc handling
cluster/zeromq: Support configuring IO threads for proxy thread
cluster/zeromq: Move variable lookups from DoInit() to DoInitPostScript()
cluster/zeromq: Handle EINTR at shutdown
cluster/zeromq: Queue one message at a time
cluster/Backend: Queue a single message only
cluster/zeromq: Adapt for OnLoopProcess changes
cluster/ThreadedBackend: Switch to OnLoopProcess
cluster/OnLoop: Introduce helper template class
serializer/broker: Expose to_broker_event() and to_zeek_event()
2025-03-11 10:51:09 +01:00
Arne Welzel
81acc4509d
ci/opensuse-tumpleweed: Bust cache
...
Got a build failure because the old container images had python3-devel
for Python3.11 or something older, but then prepare.sh would install
Python 3.13 and Zeek's configure failing due to trying to find the
devel headers from python313-devel which wasn't installed by prepare.sh
2025-03-11 10:18:20 +01:00
Arne Welzel
70f5430e7c
ci/macos/prepare: Install python@3 explicitly
...
It seems Homebrew's Python 3.12 doesn't install default symlinks or
python3 symlinks[1]. I believe this results in prepare.sh using the
system's Python rather than Homebrew's. Install python@3 explicitly
to put the symlinks in place.
[1] https://stackoverflow.com/a/77655631
2025-03-11 09:51:01 +01:00
