Commit graph

5205 commits

Author SHA1 Message Date
Tim Wojtulewicz
ec49f5d550 SQLite: Handle automated expiration 2025-03-18 09:32:34 -07:00
Tim Wojtulewicz
e95784db16 SQLite: Store/lookup prepared statements instead of recreating 2025-03-18 09:32:34 -07:00
Tim Wojtulewicz
9d1eef3fbc Add basic SQLite storage backend 2025-03-18 09:32:34 -07:00
Tim Wojtulewicz
7ad6a05f5b Add infrastructure for asynchronous storage operations 2025-03-18 09:32:34 -07:00
Tim Wojtulewicz
d07d27453a Add infrastructure for automated expiration of storage entries
This is used for backends that don't support expiration natively.
2025-03-18 09:32:34 -07:00
Tim Wojtulewicz
8dee733a7d Change args to Storage::put to be a record
The number of args being passed to the put() methods was getting to be
fairly long, with more on the horizon. Changing to a record means simplifying
things a little bit.
2025-03-18 09:32:34 -07:00
Tim Wojtulewicz
69d940533d Pass key/value types for validation when opening backends 2025-03-18 09:32:34 -07:00
Tim Wojtulewicz
2ea0f3e70a Lay out initial parts for the Storage framework
This includes a manager, component manager, BIF and script code, and
parts to support new storage backend plugins.
2025-03-18 09:32:34 -07:00
Robin Sommer
3d25328204
Merge remote-tracking branch 'origin/topic/robin/gh-4250-vec-allocator'
* origin/topic/robin/gh-4250-vec-allocator:
  Spicy: Fix passing vectors to Zeek.
2025-03-18 09:08:50 +01:00
Arne Welzel
bb58148c64 Merge remote-tracking branch 'origin/topic/awelzel/4136-cluster-websocket-support'
* origin/topic/awelzel/4136-cluster-websocket-support:
  ci/opensuse-tumpleweed: Bust cache
  ci/macos/prepare: Install python@3 explicitly
  cluster/websocket: Implement WebSocket server
  cluster/websocket: Add IXWebsocket submodule
  ci/alpine: Install openssl package for testing
  ci: Install websockets from pip for all distros
  auxil/libunistd: Bump for ssize_t typedef
  auxil/broker: Bump to latest master version
  cluster/zeromq: Catch log_push.send() exception
  cluster/zeromq: Catch exceptions as const zmq::error_t&
  cluster/zeromq: No assert on inproc handling
  cluster/zeromq: Support configuring IO threads for proxy thread
  cluster/zeromq: Move variable lookups from DoInit() to DoInitPostScript()
  cluster/zeromq: Handle EINTR at shutdown
  cluster/zeromq: Queue one message at a time
  cluster/Backend: Queue a single message only
  cluster/zeromq: Adapt for OnLoopProcess changes
  cluster/ThreadedBackend: Switch to OnLoopProcess
  cluster/OnLoop: Introduce helper template class
  serializer/broker: Expose to_broker_event() and to_zeek_event()
2025-03-11 10:51:09 +01:00
Arne Welzel
e2b65acc6d btest/javascript: Add file_sniff() and file_state_remove() test
Using file_state_remove() or file_sniff() would previously crash with the
spicy-zip file analyzer. We don't have spicy-zip here, so it's not a
proper reproducer, but still makes sense to test these events.
2025-03-10 19:15:06 +01:00
Arne Welzel
6032741868 cluster/websocket: Implement WebSocket server 2025-03-10 17:07:30 +01:00
Evan Typanski
4237239325 Fix printing type constants like double
Fixes #4268
2025-03-07 11:54:04 -05:00
Tim Wojtulewicz
81f7e77801 Merge remote-tracking branch 'origin/topic/timw/4266-tunnel-ip-proto'
* origin/topic/timw/4266-tunnel-ip-proto:
  Set ip_proto when creating new tunnel encapsulations
2025-03-06 16:42:03 -07:00
Tim Wojtulewicz
479c4be08b Set ip_proto when creating new tunnel encapsulations 2025-03-06 09:25:03 -07:00
Johanna Amann
6fcd02b3a4 Merge remote-tracking branch 'origin/topic/johanna/gh-4251-discussion'
* origin/topic/johanna/gh-4251-discussion:
  Add two protocol mismatch testcases
  Remove violating t panalyzer from services field again
2025-03-06 13:12:51 +00:00
Arne Welzel
3b7bbb2651 ZAM: bif-tracking updates for blocking_lookup_hostname() 2025-03-05 19:23:08 +01:00
Arne Welzel
715c309b03 scan.l: Deprecate DNS resolutions of hostname literals
This also skips DNS lookups when running with zeek --parse-only.

Closes #4216 #4219
2025-03-05 19:23:08 +01:00
Arne Welzel
376913b509 zeek.bif: Introduce blocking_lookup_hostname()
As a replacement for host literal DNS resolutions.
2025-03-05 19:22:57 +01:00
Arne Welzel
7eec3859fa btest/dns_mgr: Update run-dnsmasq, use --host-record
Using dnsmasq --address covers an entire domain, so dns.example.com
AAAA queries would also return IPv6 addresses for example.com

Instead, this was always meant to use host entries.
2025-03-05 12:39:15 +01:00
Johanna Amann
2daf692c95 Add two protocol mismatch testcases
These traces contain different protocols being used by originator/responder.

Traces from GH-4251
2025-03-04 15:38:20 +00:00
Johanna Amann
b8c135d7cb Remove violating analyzer from services field again
This reverts some of the recent DPD changes; specifically violations
trigger removal from the services field, again, by default.

Discussion in GH-4521
2025-03-04 15:10:49 +00:00
Johanna Amann
dfa011a3ee Merge remote-tracking branch 'origin/topic/johanna/gh-4237'
* origin/topic/johanna/gh-4237:
  RDP: cookie is optional
2025-03-04 13:41:18 +00:00
Johanna Amann
72ea7fc06c RDP: cookie is optional
Fixes GH-4237
2025-03-04 13:38:01 +00:00
Arne Welzel
5cb25fd2e8 testing/external: Fix commit-hash.zeek-testing-private
update-changes mishap :-(
2025-03-04 14:36:02 +01:00
Arne Welzel
548a12813a Merge branch 'master' of https://github.com/markoverholser/zeek
* 'master' of https://github.com/markoverholser/zeek:
  Update zeek.bif documentation for to_count and to_int
2025-03-04 12:07:22 +01:00
Arne Welzel
3bec357ace Merge remote-tracking branch 'origin/topic/timw/bifs-tocount-zam-baseline'
* origin/topic/timw/bifs-tocount-zam-baseline:
  CI: Add release-build ZAM task for baseline checks during PRs
  Fix language/assert btest baselines for to_count/to_int changes
  Update the bifs.to_count baseline for ZAM
2025-03-04 12:06:24 +01:00
Arne Welzel
ab99f8e233 scan.l: Fix @load-plugin scripts loading
For a plugin loaded via @load-plugin, create a YY_BUFFER_STATE holding
the required loads for the implicitly loaded files. In loaded scripts,
this generated file will show up with a path of the shared object file
of the plugin with the __preload__.zeek and __load__.zeek files loaded
by it.

Closes #2311
2025-03-04 09:35:00 +01:00
Robin Sommer
b77faa765a
Spicy: Fix passing vectors to Zeek.
We missed the allocator argument for the C++-side conversion
template, which let vector's of certain types not compile.

Closes #4250.
2025-03-04 09:24:03 +01:00
Tim Wojtulewicz
6de81160b6 Fix language/assert btest baselines for to_count/to_int changes 2025-02-28 20:21:57 -07:00
Tim Wojtulewicz
eb015801bd Update the bifs.to_count baseline for ZAM 2025-02-28 20:21:57 -07:00
Tim Wojtulewicz
66e3232dcc Merge remote-tracking branch 'markoverholser/master'
* markoverholser/master:
  Fix incorrect syntax for static_cast in to_count that I introduced in a previous merge
  Update baselines after changes to to_count and to_int
  fix to_count in zeek.bif to resolve conflict from bbanier's adjustment to static casting of the return from strtoull
  Fix tests in to_count to reflect appropriate function; add tests for 0x-formatted hex values for to_count and to_int
  update BIFs to_int() and to_count() to accept optional 'base' argument; allows more more exotic conversions from hex, octal, binary
  Fixes #4076; update BIFs to_int() and to_count() to accept optional 'base' argument for more easy conversions of strings representing (for example) hexadecimal, octal, or binary numbers
2025-02-24 11:38:11 -07:00
Mark Overholser
362f083351 Update baselines after changes to to_count and to_int 2025-02-23 16:34:20 -06:00
Mark Overholser
8fce9c34f8 Fix tests in to_count to reflect appropriate function; add tests for 0x-formatted hex values for to_count and to_int 2025-02-23 16:19:12 -06:00
Arne Welzel
776c003033 PacketAnalyzer::Geneve: Add get_options()
Allow to extract Geneve options on-demand, for example during a
new_connection() event.
2025-02-22 12:19:42 -08:00
Tim Wojtulewicz
caeae0b777 Fix bifs.to_count and bifs.to_int btests under ZAM 2025-02-20 17:24:50 -07:00
Benjamin Bannier
d421a19691 Also trim trailing spaces in to_count/to_int inputs
Previously we would already trim leading spaces in inputs to `to_count`
and `to_int`, effectively by just passing the behavior of the low-level
functions used in their implementations to the user. While this was
useful it was also inconsistent in that we did not allow trailing
spaces which we enable with this patch.
2025-02-18 14:15:32 +01:00
Benjamin Bannier
55533e12d4 Align error handling in to_int with existing behavior of to_count
Previously `to_int` would silently ignore invalid inputs and simply
return `0` while `to_count` would return an error; this patch changes
`to_int` to behave like `to_count`.

This introduces a breaking change in that `to_int` now raises an error
for trailing spaces (but still accepts leading spaces) where it
previously would have silently accepted it. This is consistent with
the behavior of `to_count`, but one could also argue that both of
these should only accept properly trimmed input; I did not go that route
since that would introduce breaking changes for both these functions
instead of for just one of them.
2025-02-18 14:03:09 +01:00
Benjamin Bannier
742f17fb15 Baseline handling of leading/trailing spaces in to_count/to_int.
Currently `to_count` reports an error for trailing spaces (but not for
leading ones) while `to_int` silently accepts them. This patch adds
baselines capture the current behavior.
2025-02-18 14:01:53 +01:00
Benjamin Bannier
4c4cd5984c Add error messages to to_count/to_int baselines
This captures error messages produced by `to_count`, but will also
baseline future error messages from `to_int` once we introduce them.
2025-02-18 14:01:32 +01:00
Mark Overholser
df348ff2f2 update BIFs to_int() and to_count() to accept optional 'base' argument; allows more more exotic conversions from hex, octal, binary 2025-02-18 06:49:55 -06:00
Mark Overholser
0b81bf5ad3 Fixes #4076; update BIFs to_int() and to_count() to accept optional 'base' argument for more easy conversions of strings representing (for example) hexadecimal, octal, or binary numbers 2025-02-15 07:53:38 -06:00
Evan Typanski
7160e074f6 Do not export locals
Fixes #4227
2025-02-12 13:49:12 -08:00
Johanna Amann
eda2c30452 DPD changes - ZAM baseline updates
See GH-4200 for related PR
2025-02-07 09:04:02 +00:00
Johanna Amann
fc233fd8d0 Merge remote-tracking branch 'origin/topic/johanna/dpd-changes'
* origin/topic/johanna/dpd-changes:
  DPD: failed services logging alignment
  DPD: update test baselines; change options for external tests.
  DPD: change policy script for service violation logging; add NEWS
  DPD changes - small script fixes and renames.
  Update public and private test suite for DPD changes.
  Allow to track service violations in conn.log.
  Make conn.log service field ordered
  DPD: change handling of pre-confirmation violations, remove max_violations
  DPD: log analyzers that have confirmed
  IRC analyzer - make protocol confirmation more robust.
2025-02-07 07:35:30 +00:00
Johanna Amann
0e9cf7b019 DPD: update test baselines; change options for external tests.
As services are sorted by default now, this disables the canonifier that
sorts the service field for the external baseline.

This also adds the tracking of disabled services in the service field
via DPD::track_removed_services_in_connection - to make such changes
more visible, and check that the feature works as desired.
2025-02-06 18:56:30 +00:00
Johanna Amann
0fa1ecce8f DPD: change policy script for service violation logging; add NEWS
This commit renames the `service_violation` column that can be added via
a policy script to `failed_service`. This expresses the intent of it
better - the column contains services that failed and were removed after
confirmation.

Furthermore, the script is fixed so it actually does this - before it
would sometimes add services to the list that were not actually removed.
In the course of this, the type of the column was changed from a vector
to an ordered set.

Due to the column rename, the policy script itself is also renamed.

Also adds a NEWS entry for the DPD changes.
2025-02-06 18:56:30 +00:00
Christian Kreibich
1f3d13a371 Downgrade internal errors to reporter warnings in file/directory BiFs
This allows Zeek execution to continue gracefully in the presence of such
errors, particularly at zeek_init() time. Includes a tweak to expand the
bifs.directory_operations test to check continuation after errors.

Resolves #3595.
2025-02-05 17:49:37 -08:00
Johanna Amann
e3493bc110 DPD changes - small script fixes and renames.
This addresses review feedback of GH-4200. No functional changes.
2025-02-05 13:55:43 +00:00
Arne Welzel
9c5c0f40e1 cluster/zeromq: Fix Unsubscribe() bug caused by \x00 prefix 2025-02-05 10:39:56 +01:00