Commit graph

5427 commits

Author SHA1 Message Date
Tim Wojtulewicz
d0741c8001 Allow sync methods to be called from when conditions, add related btest 2025-03-18 10:20:34 -07:00
Tim Wojtulewicz
b067a6e588 Redis: Fix sync erase, add btest for it 2025-03-18 10:20:34 -07:00
Tim Wojtulewicz
e6f1eea1b7 Remove default argument for callbacks, reorder function arguments 2025-03-18 10:20:34 -07:00
Tim Wojtulewicz
cc7b2dc890 Implement Storage::backend_opened and Storage::backend_lost events 2025-03-18 10:20:34 -07:00
Tim Wojtulewicz
a99a13dc4c SQLite: expand expiration test 2025-03-18 10:20:33 -07:00
Tim Wojtulewicz
cca1d4f988 Redis: Fix thread-contention issues with Expire(), add more tests 2025-03-18 10:20:33 -07:00
Tim Wojtulewicz
b81e876ec8 Change how redis-server is run during btests, removing redis.conf 2025-03-18 10:20:33 -07:00
Tim Wojtulewicz
9ed3e33f97 Completely rework return values from storage operations 2025-03-18 10:20:33 -07:00
Tim Wojtulewicz
8ddda016ff Update some btests due to timing changes 2025-03-18 10:20:33 -07:00
Tim Wojtulewicz
e766af7322 Split sync/async handling into the BIF methods 2025-03-18 10:20:33 -07:00
Tim Wojtulewicz
c247de8ec3 Redis: Rework everything to only use async mode 2025-03-18 10:20:33 -07:00
Tim Wojtulewicz
a485b1d237 Make backend options a record, move actual options to be sub-records 2025-03-18 10:20:33 -07:00
Tim Wojtulewicz
28951dccf1 Split sync and async into separate script-land namespaces 2025-03-18 10:20:33 -07:00
Tim Wojtulewicz
42ad5bbf7d Add btest that uses a Redis backend in a cluster 2025-03-18 09:32:34 -07:00
Tim Wojtulewicz
f1a7376e0a Return generic result for get operations that includes error messages 2025-03-18 09:32:34 -07:00
Tim Wojtulewicz
4695060d75 Allow opening and closing backends to be async 2025-03-18 09:32:34 -07:00
Tim Wojtulewicz
ea87c773cd Redis: Support non-native expiration when reading traces 2025-03-18 09:32:34 -07:00
Tim Wojtulewicz
08bebaa426 Redis: Add btests for the redis backend 2025-03-18 09:32:34 -07:00
Tim Wojtulewicz
6289eb8e15 SQLite: Fix some issues with expiration, including in the btest 2025-03-18 09:32:34 -07:00
Tim Wojtulewicz
6bc5f70236 SQLite: Add additional btests, which also cover general storage functionality
- New erase/overwrite tests
- Change existing sqlite-basic test to use async
- Test passing bad keys to validate backend type checking
- New test for compound keys and values
2025-03-18 09:32:34 -07:00
Tim Wojtulewicz
ec49f5d550 SQLite: Handle automated expiration 2025-03-18 09:32:34 -07:00
Tim Wojtulewicz
e95784db16 SQLite: Store/lookup prepared statements instead of recreating 2025-03-18 09:32:34 -07:00
Tim Wojtulewicz
9d1eef3fbc Add basic SQLite storage backend 2025-03-18 09:32:34 -07:00
Tim Wojtulewicz
7ad6a05f5b Add infrastructure for asynchronous storage operations 2025-03-18 09:32:34 -07:00
Tim Wojtulewicz
d07d27453a Add infrastructure for automated expiration of storage entries
This is used for backends that don't support expiration natively.
2025-03-18 09:32:34 -07:00
Tim Wojtulewicz
8dee733a7d Change args to Storage::put to be a record
The number of args being passed to the put() methods was getting to be
fairly long, with more on the horizon. Changing to a record means simplifying
things a little bit.
2025-03-18 09:32:34 -07:00
Tim Wojtulewicz
69d940533d Pass key/value types for validation when opening backends 2025-03-18 09:32:34 -07:00
Tim Wojtulewicz
2ea0f3e70a Lay out initial parts for the Storage framework
This includes a manager, component manager, BIF and script code, and
parts to support new storage backend plugins.
2025-03-18 09:32:34 -07:00
Johanna Amann
6023c8b906 SSH: make banner parsing more robust
This change revamps SSH banner parsing.  The previous behavior was both
a bit too strict in some regards, and too permissive in other.

Specifically, clients are now required to send a line starting with
"SSH-" as the first line.  This is in line with the RFC, as well with
observed behavior. This also prevents the creation of `ssh.log` for
non-SSH traffic on port 22.

For the server side, we now accept text before the SSH banner. This
previously led to a protocol violation but is allowed by the spec.

New tests are added to cover these cases.
2025-03-18 16:19:33 +00:00
Robin Sommer
3d25328204
Merge remote-tracking branch 'origin/topic/robin/gh-4250-vec-allocator'
* origin/topic/robin/gh-4250-vec-allocator:
  Spicy: Fix passing vectors to Zeek.
2025-03-18 09:08:50 +01:00
Arne Welzel
bb58148c64 Merge remote-tracking branch 'origin/topic/awelzel/4136-cluster-websocket-support'
* origin/topic/awelzel/4136-cluster-websocket-support:
  ci/opensuse-tumpleweed: Bust cache
  ci/macos/prepare: Install python@3 explicitly
  cluster/websocket: Implement WebSocket server
  cluster/websocket: Add IXWebsocket submodule
  ci/alpine: Install openssl package for testing
  ci: Install websockets from pip for all distros
  auxil/libunistd: Bump for ssize_t typedef
  auxil/broker: Bump to latest master version
  cluster/zeromq: Catch log_push.send() exception
  cluster/zeromq: Catch exceptions as const zmq::error_t&
  cluster/zeromq: No assert on inproc handling
  cluster/zeromq: Support configuring IO threads for proxy thread
  cluster/zeromq: Move variable lookups from DoInit() to DoInitPostScript()
  cluster/zeromq: Handle EINTR at shutdown
  cluster/zeromq: Queue one message at a time
  cluster/Backend: Queue a single message only
  cluster/zeromq: Adapt for OnLoopProcess changes
  cluster/ThreadedBackend: Switch to OnLoopProcess
  cluster/OnLoop: Introduce helper template class
  serializer/broker: Expose to_broker_event() and to_zeek_event()
2025-03-11 10:51:09 +01:00
Arne Welzel
e2b65acc6d btest/javascript: Add file_sniff() and file_state_remove() test
Using file_state_remove() or file_sniff() would previously crash with the
spicy-zip file analyzer. We don't have spicy-zip here, so it's not a
proper reproducer, but still makes sense to test these events.
2025-03-10 19:15:06 +01:00
Arne Welzel
6032741868 cluster/websocket: Implement WebSocket server 2025-03-10 17:07:30 +01:00
Evan Typanski
4237239325 Fix printing type constants like double
Fixes #4268
2025-03-07 11:54:04 -05:00
Tim Wojtulewicz
81f7e77801 Merge remote-tracking branch 'origin/topic/timw/4266-tunnel-ip-proto'
* origin/topic/timw/4266-tunnel-ip-proto:
  Set ip_proto when creating new tunnel encapsulations
2025-03-06 16:42:03 -07:00
Tim Wojtulewicz
479c4be08b Set ip_proto when creating new tunnel encapsulations 2025-03-06 09:25:03 -07:00
Johanna Amann
6fcd02b3a4 Merge remote-tracking branch 'origin/topic/johanna/gh-4251-discussion'
* origin/topic/johanna/gh-4251-discussion:
  Add two protocol mismatch testcases
  Remove violating t panalyzer from services field again
2025-03-06 13:12:51 +00:00
Arne Welzel
3b7bbb2651 ZAM: bif-tracking updates for blocking_lookup_hostname() 2025-03-05 19:23:08 +01:00
Arne Welzel
715c309b03 scan.l: Deprecate DNS resolutions of hostname literals
This also skips DNS lookups when running with zeek --parse-only.

Closes #4216 #4219
2025-03-05 19:23:08 +01:00
Arne Welzel
376913b509 zeek.bif: Introduce blocking_lookup_hostname()
As a replacement for host literal DNS resolutions.
2025-03-05 19:22:57 +01:00
Arne Welzel
7eec3859fa btest/dns_mgr: Update run-dnsmasq, use --host-record
Using dnsmasq --address covers an entire domain, so dns.example.com
AAAA queries would also return IPv6 addresses for example.com

Instead, this was always meant to use host entries.
2025-03-05 12:39:15 +01:00
Johanna Amann
2daf692c95 Add two protocol mismatch testcases
These traces contain different protocols being used by originator/responder.

Traces from GH-4251
2025-03-04 15:38:20 +00:00
Johanna Amann
b8c135d7cb Remove violating analyzer from services field again
This reverts some of the recent DPD changes; specifically violations
trigger removal from the services field, again, by default.

Discussion in GH-4521
2025-03-04 15:10:49 +00:00
Johanna Amann
dfa011a3ee Merge remote-tracking branch 'origin/topic/johanna/gh-4237'
* origin/topic/johanna/gh-4237:
  RDP: cookie is optional
2025-03-04 13:41:18 +00:00
Johanna Amann
72ea7fc06c RDP: cookie is optional
Fixes GH-4237
2025-03-04 13:38:01 +00:00
Arne Welzel
5cb25fd2e8 testing/external: Fix commit-hash.zeek-testing-private
update-changes mishap :-(
2025-03-04 14:36:02 +01:00
Arne Welzel
548a12813a Merge branch 'master' of https://github.com/markoverholser/zeek
* 'master' of https://github.com/markoverholser/zeek:
  Update zeek.bif documentation for to_count and to_int
2025-03-04 12:07:22 +01:00
Arne Welzel
3bec357ace Merge remote-tracking branch 'origin/topic/timw/bifs-tocount-zam-baseline'
* origin/topic/timw/bifs-tocount-zam-baseline:
  CI: Add release-build ZAM task for baseline checks during PRs
  Fix language/assert btest baselines for to_count/to_int changes
  Update the bifs.to_count baseline for ZAM
2025-03-04 12:06:24 +01:00
Arne Welzel
ab99f8e233 scan.l: Fix @load-plugin scripts loading
For a plugin loaded via @load-plugin, create a YY_BUFFER_STATE holding
the required loads for the implicitly loaded files. In loaded scripts,
this generated file will show up with a path of the shared object file
of the plugin with the __preload__.zeek and __load__.zeek files loaded
by it.

Closes #2311
2025-03-04 09:35:00 +01:00
Robin Sommer
b77faa765a
Spicy: Fix passing vectors to Zeek.
We missed the allocator argument for the C++-side conversion
template, which let vector's of certain types not compile.

Closes #4250.
2025-03-04 09:24:03 +01:00