Commit graph

17381 commits

Author SHA1 Message Date
Arne Welzel
28453ca1a5 fixup! HTTP: Support HTTP::upgrade_content_type_analyzers 2024-12-09 10:31:49 +01:00
Arne Welzel
e5c0a597d0 HTTP: Support HTTP::upgrade_content_type_analyzers
This allows to select the Upgrade analyzer based on the tuple
of [Upgrade, Content-Type] as motivated by the Docker API trace
that sets a very specific Content-Type in the HTTP response.

Closes #4068
2024-12-06 18:49:51 +01:00
Arne Welzel
2b3f9bc345 init-bare/HTTP: Fix client to server Upgrade header 2024-12-06 18:49:26 +01:00
Arne Welzel
27517ee4cb Merge remote-tracking branch 'origin/topic/awelzel/4068-http-upgrade-content-type'
* origin/topic/awelzel/4068-http-upgrade-content-type:
  btest/http: Demo StreamEvent analyzer with HTTP::upgrade_analyzers
  protocol: Add StreamEvent analyzer
2024-12-06 17:33:52 +01:00
Arne Welzel
079ae460a7 btest/http: Demo StreamEvent analyzer with HTTP::upgrade_analyzers
Relates to #4068
2024-12-06 16:12:40 +01:00
Arne Welzel
51836d08ae protocol: Add StreamEvent analyzer
This analyzer can be used to transport raw stream data for a given
connection to the script layer. For example, adding this analyzer into
the HTTP::upgrade_analyzer or using it to configure a child WebSocket
analyzer allows to get access to the raw stream data in script land
when no more appropriate protocol analyzer is available.
2024-12-06 16:12:40 +01:00
Arne Welzel
2f27db6542 Merge remote-tracking branch 'origin/topic/vern/zam-assert-fix'
* origin/topic/vern/zam-assert-fix:
  fix for ZAM "assert" statements potentially evaluating invalid expressions
2024-12-06 10:22:24 +01:00
zeek-bot
a1776540f4 Update doc submodule [nomail] [skip ci] 2024-12-06 00:22:21 +00:00
Vern Paxson
e6e31bc25a fix for ZAM "assert" statements potentially evaluating invalid expressions 2024-12-05 14:29:06 -08:00
Tim Wojtulewicz
682789a8e9 Merge remote-tracking branch 'origin/topic/awelzel/deprecate-tcp-reassembler-ports'
* origin/topic/awelzel/deprecate-tcp-reassembler-ports:
  init-bare: Deprecate tcp_reassembler_ports
2024-12-05 13:48:57 -07:00
Arne Welzel
93a3a11d36 Merge remote-tracking branch 'origin/topic/vern/script-opt-keep-asserts'
* origin/topic/vern/script-opt-keep-asserts:
  ZAM documentation updates for asserts and event handler run-time errors
  BTest updates for ZAM support of (optionally) keeping "assert" statements
  command-line options for controlling script optimization: keeping asserts, avoiding event handler coalescence
  ZAM support for option to not coalesce event handlers
  ZAM support for keeping "assert" statements
  internal support for script optimization options for keeping asserts, not consolidating event handlers
  ZAM operations to support asserts
  simplified "assert" by not trying to catch messages that themselves have errors

Fixed some TEST-REQUIRES "${ZEEK_ZAM}" == "1" to use "=" instead to
be /bin/sh compatible.
2024-12-05 21:45:07 +01:00
Arne Welzel
9efa12b055 Merge remote-tracking branch 'origin/topic/vern/ZAM-empty-hook-opt'
* origin/topic/vern/ZAM-empty-hook-opt:
  ZAM optimization now removes hook calls to hooks without any bodies
2024-12-05 19:12:59 +01:00
Tim Wojtulewicz
488ac7ddf0 Merge remote-tracking branch 'origin/topic/vern/zam-pattern-comparison'
* origin/topic/vern/zam-pattern-comparison:
  update of BTest that tracks number of (and validates) ZAM operations
  ZAM support for pattern equality/inequality operations
  expanded ZAM operations for bit-shifting to allow for int/count shift values
  added type coercion for bit-shifting expressions
2024-12-05 11:09:15 -07:00
Arne Welzel
e773fe0506 init-bare: Deprecate tcp_reassembler_ports
...and remove from NetVar, seems left-over.
2024-12-05 16:18:11 +01:00
Arne Welzel
6482a9a8c2 Merge remote-tracking branch 'origin/topic/awelzel/ctu-sme-11-vm-win7ad'
* origin/topic/awelzel/ctu-sme-11-vm-win7ad:
  zeek-testing: Add Experiment-VM-Microsoft-Windows7AD-1 PCAP and baselines
2024-12-05 14:52:14 +01:00
zeek-bot
55929e5acd Update doc submodule [nomail] [skip ci] 2024-12-05 00:19:43 +00:00
Vern Paxson
678db505fb update of BTest that tracks number of (and validates) ZAM operations 2024-12-04 15:26:13 -07:00
Vern Paxson
f300fb6cf0 ZAM support for pattern equality/inequality operations 2024-12-04 15:25:31 -07:00
Vern Paxson
7d25835ca8 expanded ZAM operations for bit-shifting to allow for int/count shift values 2024-12-04 15:24:11 -07:00
Vern Paxson
01fe72d9fc added type coercion for bit-shifting expressions 2024-12-04 15:23:44 -07:00
Arne Welzel
df5a6a1ea0 zeek-testing: Add Experiment-VM-Microsoft-Windows7AD-1 PCAP and baselines
The pcap comes from the following dataset [1]:

    CTU-SME-11: a labeled dataset with real benign and malicious network
    traffic mimicking a small medium-size enterprise environment

[1] https://zenodo.org/records/7958259
2024-12-04 20:12:30 +01:00
Arne Welzel
a44cafafe7 Merge remote-tracking branch 'origin/topic/awelzel/pluggable-cluster-backends-part3'
* origin/topic/awelzel/pluggable-cluster-backends-part3:
  init-bare/zeek-setup: Groundwork for instantiating cluster backends
  cluster/serializer: Add binary-serialization-format
  logging/WriterFrontend: Add logic for non-broker cluster backends
  logging/WriterBackend: Include logging/Types.h
  logging/Manager: Implement new WriteBatchFromRemote()
  logging/WriterFrontend: Add LogWriteHeader as member
  logging: Add filter_name to WriterInfo
2024-12-04 13:47:49 +01:00
Arne Welzel
b3080f2652 init-bare/zeek-setup: Groundwork for instantiating cluster backends 2024-12-04 12:40:35 +01:00
Arne Welzel
fdde1e9841 cluster/serializer: Add binary-serialization-format
This is a serializer for log records that is using SerialTypes
for serializing and un-serializing rather. Essentially, this is
similar to what broker does except for the envelope.
2024-12-04 12:40:35 +01:00
Arne Welzel
10d93eff36 logging/WriterFrontend: Add logic for non-broker cluster backends
If cluster::backend isn't broker_mgr, use the WriterFrontend's buffering
logic and send a whole batch of log writes during FlushWriteBuffer().
This is a different path than broker's own logging logic.

Preferably we adapt broker to a model where it isn't
buffering either.
2024-12-04 12:40:35 +01:00
Arne Welzel
15ea8a3be9 logging/WriterBackend: Include logging/Types.h 2024-12-04 12:40:35 +01:00
Arne Welzel
ab1d48c95a logging/Manager: Implement new WriteBatchFromRemote() 2024-12-04 12:40:35 +01:00
Arne Welzel
3dd1f8d38a logging/WriterFrontend: Add LogWriteHeader as member
The header captures the enum values as well as the fields
2024-12-04 12:37:22 +01:00
Arne Welzel
f5d4526eac logging: Add filter_name to WriterInfo
...with this change, it'll be possible to identify WriterFrontend's
based on (stream, filter_name, path) pairs in addition to (stream,
writer, path) pairs.
2024-12-04 12:37:22 +01:00
Arne Welzel
2c52e533b2 Merge remote-tracking branch 'origin/topic/awelzel/spicy-type-convert-asan-options'
* origin/topic/awelzel/spicy-type-convert-asan-options:
  btest/spicy.type-converter: Drop ASAN_OPTIONS from test
2024-12-04 11:14:10 +01:00
Arne Welzel
a2cf0d874a btest/spicy.type-converter: Drop ASAN_OPTIONS from test
...doesn't seem it's needed anymore.

Fixes #4077
2024-12-04 11:11:47 +01:00
Arne Welzel
067c40a545 Merge remote-tracking branch 'origin/topic/neverlord/broker-cleanup'
* origin/topic/neverlord/broker-cleanup:
  Remove obsolete c_str_safe utility
  Remove obsolete Broker compatibility layer
2024-12-04 11:08:08 +01:00
zeek-bot
2ef57aaea1 Update doc submodule [nomail] [skip ci] 2024-12-04 00:23:36 +00:00
Vern Paxson
0fda589a19 ZAM optimization now removes hook calls to hooks without any bodies 2024-12-03 12:47:20 -07:00
Vern Paxson
0775719342 ZAM documentation updates for asserts and event handler run-time errors 2024-12-03 10:59:47 -07:00
Vern Paxson
47a555da87 BTest updates for ZAM support of (optionally) keeping "assert" statements 2024-12-03 10:40:41 -07:00
Vern Paxson
aa9c0a99b0 command-line options for controlling script optimization: keeping asserts,
avoiding event handler coalescence
2024-12-03 10:38:31 -07:00
Vern Paxson
c523684cc4 ZAM support for option to not coalesce event handlers 2024-12-03 10:38:05 -07:00
Vern Paxson
908e8a3a27 ZAM support for keeping "assert" statements 2024-12-03 10:37:38 -07:00
Vern Paxson
a328185a8f internal support for script optimization options for keeping asserts, not
consolidating event handlers
2024-12-03 10:35:35 -07:00
Vern Paxson
8b1790019a ZAM operations to support asserts 2024-12-03 10:34:56 -07:00
Tim Wojtulewicz
deb2acaeda Merge remote-tracking branch 'origin/topic/timw/json-interval-conversion'
* origin/topic/timw/json-interval-conversion:
  Add interval_as_double argument to control how intervals are converted to JSON
  Add btest for round-trip JSON conversion
  Allow comparing two PatternVals
  Handle conversion between data from Val::ToJSON and ValFromJSON better
2024-12-03 10:10:37 -07:00
Dominik Charousset
feeb06f7cf Remove obsolete c_str_safe utility
The old `c_str_safe` utility function allowed Zeek to operator on
`broker::data` and `broker::variant`. The former grants access to actual
`std::string` objects while the latter only provides access to fields
via `std::string_view`. Since the Zeek formatting functions need null
terminated strings, we need to copy the characters into a
null-terminated container first.

After removing support for `broker::data` and `broker::variant` from the
same code paths, we can drop `c_str_safe` and always do the copying
(since we are always dealing with `broker::variant` now).
2024-12-03 17:26:23 +01:00
Tim Wojtulewicz
43e3de5c79 Add interval_as_double argument to control how intervals are converted to JSON 2024-12-03 09:26:08 -07:00
Tim Wojtulewicz
8ff10e4d21 Add btest for round-trip JSON conversion 2024-12-03 09:26:08 -07:00
Tim Wojtulewicz
7670a6f589 Allow comparing two PatternVals 2024-12-03 09:26:08 -07:00
Tim Wojtulewicz
9f5cd54e78 Handle conversion between data from Val::ToJSON and ValFromJSON better 2024-12-03 09:26:08 -07:00
Dominik Charousset
470f545b37 Remove obsolete Broker compatibility layer
Since the transition to broker::variant has been long finalized, there
is no more need to be able to go back to a pre-variant version of
Broker. Hence, we can drop various utilities that allow Zeek to run with
older Broker releases.
2024-12-03 17:21:19 +01:00
Vern Paxson
05e913db1b simplified "assert" by not trying to catch messages that themselves have errors 2024-12-02 10:37:10 -08:00
Arne Welzel
d2ecdc2a11 Merge remote-tracking branch 'origin/topic/vern/ZAM-cluster-bifs'
* origin/topic/vern/ZAM-cluster-bifs:
  added new Cluster:: BiFs to script optimization tracking
2024-12-02 13:43:36 +01:00