Commit graph

18601 commits

Author SHA1 Message Date
zeek-bot
9b1f3b5838 Update doc submodule [nomail] [skip ci] 2023-11-08 00:36:48 +00:00
Arne Welzel
cef0210799 Merge remote-tracking branch 'origin/topic/awelzel/deprecate-things-for-7.1'
* origin/topic/awelzel/deprecate-things-for-7.1:
  Bump zeekctl
  EventHandler: Deprecate SetUsed() and Used() as well.
  EventRegistry: Deprecate UsedHandlers() and UnusedHandlers()
  time machine: Mark leftovers for removal in v7.1
  policy/misc/load-balancing: Deprecate script
  cluster: Deprecate the Cluster::Node$interface field
2023-11-07 19:42:56 +01:00
Arne Welzel
1fd5814c7f Bump zeekctl 2023-11-07 16:06:35 +01:00
Arne Welzel
ff34a4aa7f EventHandler: Deprecate SetUsed() and Used() as well.
Seems the latter isn't used outside of the functions that were deprecated
in the previous commit and with UsageAnalyzer not making use of this
information unclear why we should keep it around.

Relates to #3187.
2023-11-07 16:06:35 +01:00
Arne Welzel
398122206e EventRegistry: Deprecate UsedHandlers() and UnusedHandlers()
and check_for_unused_event_handlers: UsageAnalyzer is more thorough
and the previous ones weren't extended to work with &is_used and
should probably be considered superseded by the UsageAnalyzer even
if that currently does not provide a public API and just prints
out deprecation warnings.

I'm also tempted to deprecate SetUsed() and Used() of EventHandler
for the same reason.

Closes #3187.
2023-11-07 16:06:17 +01:00
Arne Welzel
cd24acdfc8 time machine: Mark leftovers for removal in v7.1
I suspect we could just drop these directly, but lets follow the
deprecation cycle.
2023-11-07 16:06:16 +01:00
Arne Welzel
384453346d policy/misc/load-balancing: Deprecate script 2023-11-07 16:06:16 +01:00
Arne Welzel
d88b147ac9 cluster: Deprecate the Cluster::Node$interface field
This field isn't required by a worker and it's certainly not used by a
worker to listen on that specific interface. It also isn't required to
be set consistently and its use in-tree limited to the old load-balancing
script.

There's a bif called packet_source() which on a worker will provide
information about the actually used packet source.

Relates to zeek/zeek#2877.
2023-11-07 16:06:16 +01:00
Arne Welzel
46b7e9d7be Merge remote-tracking branch 'origin/topic/timw/update-bifcl-binpac'
* origin/topic/timw/update-bifcl-binpac:
  Update binpac and bifcl submodules [nomail]
2023-11-07 10:03:43 +01:00
Arne Welzel
7dca9d6836 Merge remote-tracking branch 'origin/topic/timw/macos-sonoma-build'
* origin/topic/timw/macos-sonoma-build:
  CI: Use other base64 options on macOS Sonoma too
  CI: Add macOS Sonoma build, remove macOS Monterey build
2023-11-07 10:01:50 +01:00
Arne Welzel
8a13155a41 Merge branch 'topic/xb-anssi/http_signature_body_end_match' of https://github.com/xb-anssi/zeek
* 'topic/xb-anssi/http_signature_body_end_match' of https://github.com/xb-anssi/zeek:
  Let signature framework match HTTP body end
  Test how the signature framework matches HTTP body
2023-11-07 09:58:59 +01:00
Arne Welzel
768153452a Merge remote-tracking branch 'origin/topic/neverlord/copy-string'
* origin/topic/neverlord/copy-string:
  Fix unsafe and inefficient uses of copy_string
2023-11-07 09:57:12 +01:00
Tim Wojtulewicz
90d188b63f Update binpac and bifcl submodules [nomail] 2023-11-06 13:42:22 -07:00
Tim Wojtulewicz
b1e63ffd26 CI: Use other base64 options on macOS Sonoma too 2023-11-06 11:29:44 -07:00
Tim Wojtulewicz
35ca1e88c7 CI: Add macOS Sonoma build, remove macOS Monterey build 2023-11-06 11:09:44 -07:00
Tim Wojtulewicz
f2608680ce Merge remote-tracking branch 'origin/topic/timw/update-c-ares'
* origin/topic/timw/update-c-ares:
  Update c-ares in Windows conanfile
  Upgrade c-ares submodule to 1.21.0
2023-11-06 09:30:01 -07:00
Johanna Amann
32d27b1b3f Spicy TLS: Re-enable starttls for RDP 2023-11-06 13:32:27 +00:00
Johanna Amann
c682e5723b Spicy TLS: rename the analyzer to SSL
Naming the analyzer different than the old one was a mistake that
required unnecessary code changes; keeping the old name makes things
like StartTLS in other protocol work without additional code changes.
2023-11-06 12:54:45 +00:00
Johanna Amann
283d900758 Merge remote-tracking branch 'origin/master' into topic/johanna/spicy-tls
* origin/master:
  Update doc submodule [nomail] [skip ci]
  build_inner_connection: Use the outer packet's timestamp
  build_inner_connection: Avoid one extra Init()
  packet_analysis: Do not run DetectProtocol() on disabled analyzers
  packet_analysis/Dispatcher: Do not index table twice
  packet_analysis: Avoid shared_ptr copying for analyzer lookups
  Update doc submodule [nomail] [skip ci]
  Update doc submodule [nomail] [skip ci]
  SSL: Add new extension types and ECH test
  Update `.git-blame-ignore-revs`
  Format JSON with clang-format
  Bump pre-commit hooks
  Reformat Zeek in Spicy style
2023-11-06 11:34:48 +00:00
Johanna Amann
36741c2fbf Reformat 2023-11-06 09:42:46 +00:00
zeek-bot
9f7a390adc Update doc submodule [nomail] [skip ci] 2023-11-04 00:19:31 +00:00
Tim Wojtulewicz
4c9721e8bf Update c-ares in Windows conanfile 2023-11-03 11:11:15 -07:00
Tim Wojtulewicz
e6fc3b98c6 Upgrade c-ares submodule to 1.21.0 2023-11-03 10:50:22 -07:00
Dominik Charousset
c500370563 Avoid OpenSSL header dependencies 2023-11-03 15:54:46 +01:00
xb-anssi
9e61bfd010
Let signature framework match HTTP body end
The HTTP analyzer never tells the signature framework when the body of a
request or a response ends, so any signature regex ending in a '$' used
in an 'http-request-body' or in an 'http-reply-body' condition will
never match.

This made it impossible to write a signature which could distinguish an
HTTP body consisting only of something from an HTTP body prefixed by
that same something.

- Fix:

The fix notifies the signature framework on EndOfData() that there will
be no further data to match for this body by giving it an empty buffer
of length 0 with the eol parameter set to true and all others set to
false. This lets it reach the '$' state in its DFA, and doesn't affect
other documented HTTP match behaviours.

- Limitation:

Since the signature framework doesn't appear to keep previously consumed
data on hand, any match of an http-*-body condition whose patterns ends
with a '$' will lead to an empty data parameter being passed to the
signature_match() event because the body data is no longer available
when EndOfData() happens.

Due to segmentation there is anyway no guarantee the data parameter
would have held the entire match even without the '$', since the data
parameter only receives the last chunk of data which completed the match
condition, as can be seen on prefix matches in the btest cases where the
matching data spans multiple segments (the event gives 'B' and not
'AB'), so this is only an extreme case of partial data being given to
that event.
2023-11-03 15:28:24 +01:00
xb-anssi
c8103dd963
Test how the signature framework matches HTTP body
This adds a signatures/http-body-match btest to verify how the signature
framework matches HTTP body in requests and responses.

It currently fails because the 'http-request-body' and 'http-reply-body'
clauses never match anything when there is a '$' in their regular
expressions.

The other pattern clauses such as the 'payload' clause do not suffer
from that restriction and it is not documented as a limitation of HTTP
body pattern clauses either, so it is probably a bug.

The "http-body-match" btest shows that without a fix any signatures
which ends with a '$' in a http-request-body or http-reply-body rule
will never raise a signature_match() event, and that signatures which do
not end with a '$' cannot distinguish an HTTP body prefixed by the
matching pattern (ex: ABCD) from an HTTP body consisting entirely of the
matching pattern (ex: AB).

Test cases by source port:
- 13579:
  - GET without body, plain res body (CD, only)
- 13578:
  - GET without body, plain res body (CDEF, prefix)
- 24680:
  - POST plain req body (AB, only), plain res body (CD, only)
- 24681:
  - POST plain req body (ABCD, prefix), plain res body (CDEF, prefix)
- 24682:
  - POST gzipped req body (AB, only), gzipped res body (CD, only)
  - POST plain req body (CD, only), plain res body (EF, only)
- 33210:
  - POST multipart plain req body (AB;CD;EF, prefix)
  - plain res body (CD, only)
- 33211:
  - POST multipart plain req body (ABCD;EF, prefix)
  - plain res body (CDEF, prefix)
- 34527:
  - POST chunked gzipped req body (AB, only)
  - chunked gzipped res body (CD, only)
- 34528:
  - POST chunked gzipped req body (ABCD, prefix)
  - chunked gzipped res body (CDEF, prefix)

The tests with source ports 24680, 24682 and 34527 should
match the signature http_request_body_AB_only and the signature
http_request_body_AB_prefix, but they only match the latter.

The tests with source ports 13579, 24680, 24682, 33210 and 34527 should
match the signature http_response_body_CD_only and the signature
http_response_body_CD_prefix, but they only match the latter.

The tests with source ports 24680, 24681, 33210 and 33211 show how the
http_request_body_AB_then_CD signature with two http-request-body
conditions match either on one or multiple requests (documented
behaviour).

The test cases with other source ports show where the
http_request_body_AB_only and http_response_body_CD_only signatures
should not match because their bodies include more than the searched
patterns.
2023-11-03 15:28:15 +01:00
Dominik Charousset
cebb85b1e8 Fix unsafe and inefficient uses of copy_string
Add a new overload to `copy_string` that takes the input characters plus
size. The new overload avoids inefficient scanning of the input for the
null terminator in cases where we know the size beforehand. Furthermore,
this overload *must* be used when dealing with input character sequences
that may have no null terminator, e.g., when the input is from a
`std::string_view` object.
2023-11-03 15:25:38 +01:00
Arne Welzel
4eb1b71d1b Merge remote-tracking branch 'origin/topic/awelzel/3379-shared-ptr-and-micro-optimizations'
* origin/topic/awelzel/3379-shared-ptr-and-micro-optimizations:
  build_inner_connection: Use the outer packet's timestamp
  build_inner_connection: Avoid one extra Init()
  packet_analysis: Do not run DetectProtocol() on disabled analyzers
  packet_analysis/Dispatcher: Do not index table twice
  packet_analysis: Avoid shared_ptr copying for analyzer lookups
2023-11-01 12:04:47 +01:00
Arne Welzel
0c19e6fc54 build_inner_connection: Use the outer packet's timestamp
Don't construct the timeval based on run_state, just use the timestamp
of the outer packet to avoid the extra int/double conversions required.
2023-11-01 12:03:55 +01:00
Arne Welzel
d08e347e5e build_inner_connection: Avoid one extra Init()
Packet::Init() is not so cheap as one might think: It computes a
timestamp from { 0, 0 } using double division. Just avoid this
by not initializing an empty Packet.
2023-11-01 12:03:55 +01:00
Arne Welzel
ec4ad2e80d packet_analysis: Do not run DetectProtocol() on disabled analyzers
This came up when disabling the TEREDO analyzer but still seeing its
DetectProtocol() method prominently in flame graphs.
2023-11-01 12:03:55 +01:00
Arne Welzel
9bebd93c06 packet_analysis/Dispatcher: Do not index table twice
It's okay to return the nullptr that's in the table, no need to check
for != nullptr before dereferencing again.
2023-11-01 12:03:55 +01:00
Arne Welzel
e56ef0fd9b packet_analysis: Avoid shared_ptr copying for analyzer lookups
For deeply encapsulated connections (think AWS traffic mirroring format
like IP,UDP,GENEVE,IP,UDP,VXLAN,ETH,IP,TCP), the Dispatcher::Lookup()
method is fairly visible in profiles when running in bare mode.

This changes the Analyzer::Lookup() and Dispatcher::Lookup() return value
breaking the API in favor of the performance improvement.

Relates to zeek/zeek#3379.
2023-11-01 12:03:02 +01:00
zeek-bot
a821604505 Update doc submodule [nomail] [skip ci] 2023-11-01 00:24:38 +00:00
Johanna Amann
3257c0e216 Merge remote-tracking branch 'origin/topic/johanna/its-time-to-add-more-tls-extension-types'
* origin/topic/johanna/its-time-to-add-more-tls-extension-types:
  SSL: Add new extension types and ECH test
2023-10-31 16:19:57 +00:00
zeek-bot
f1073fbc5f Update doc submodule [nomail] [skip ci] 2023-10-31 00:46:55 +00:00
Tim Wojtulewicz
d7cb384545 Merge remote-tracking branch 'origin/topic/bbannier/use-spicy-formatting-style'
* origin/topic/bbannier/use-spicy-formatting-style:
  Update `.git-blame-ignore-revs`
  Format JSON with clang-format
  Bump pre-commit hooks
  Reformat Zeek in Spicy style
2023-10-30 12:48:37 -07:00
Johanna Amann
ff27eb5a69 SSL: Add new extension types and ECH test
This commit adds a multitude of new extension types that were added in
the last few years; it also adds grease values to extensions, curves,
and ciphersuites.

Furthermore, it adds a test that contains a encrypted-client-hello
key-exchange (which uses several extension types that we do not have in
our baseline so far).
2023-10-30 14:19:16 +00:00
Johanna Amann
0afe94154d Merge remote-tracking branch 'origin/master' into topic/johanna/spicy-tls
* origin/master: (386 commits)
  Normalize version strings in test
  Update doc submodule [nomail] [skip ci]
  Update external testing baseline hashes
  fuzzers: Add DTLS fuzzer
  generic-analyzer-fuzzer: Support NextPacket() fuzzing
  Require `truncate` for a test using it
  Bump outdated baseline
  Fix tests so they work both with GNU and BSD tools
  Install libmaxminddb in macOS CI
  Bump auxil/spicy to latest release
  Supervisor: Handle EAGAIN error on stem pipe
  fuzzer-setup: Allow customization without recompiling
  ssl: Prevent unbounded ssl_history growth
  ssl: Cap number of alerts parsed from SSL record
  subdir-btest: Allow setting build_dir
  Update doc submodule [nomail] [skip ci]
  CI: Pass -A flag to btest for cluster-testing builds
  Update doc submodule [nomail] [skip ci]
  Update baselines
  ftp: Do not base seq on number of pending commands
  ...
2023-10-30 12:28:40 +00:00
Benjamin Bannier
552c65a881 Merge remote-tracking branch 'origin/topic/bbannier/issue-3421' 2023-10-30 11:47:43 +01:00
Benjamin Bannier
49d7e638df Normalize version strings in test
Closes #3421.
2023-10-30 10:42:12 +01:00
Benjamin Bannier
09cd3745af Update .git-blame-ignore-revs 2023-10-30 09:42:39 +01:00
Benjamin Bannier
e6256446dd Format JSON with clang-format
We do not activate support for JavaScript at this time since most of our
JavaScript code is in BTest files to test zeekjs, but these files also
contain other languages which leads to largely misformated files.
2023-10-30 09:41:13 +01:00
Benjamin Bannier
26d04fd9fc Bump pre-commit hooks 2023-10-30 09:41:12 +01:00
Benjamin Bannier
f5a76c1aed Reformat Zeek in Spicy style
This largely copies over Spicy's `.clang-format` configuration file. The
one place where we deviate is header include order since Zeek depends on
headers being included in a certain order.
2023-10-30 09:40:55 +01:00
zeek-bot
7b8e7ed72c Update doc submodule [nomail] [skip ci] 2023-10-28 00:26:57 +00:00
Tim Wojtulewicz
fd84b793c6 Update external testing baseline hashes 2023-10-27 11:26:09 -07:00
Tim Wojtulewicz
7852463b2f Merge remote-tracking branch 'security/topic/awelzel/195-ieee802-heap-buffer-overflow'
* security/topic/awelzel/195-ieee802-heap-buffer-overflow:
  IEEE802_11: Fix buffer-overflow due to not accounting for the amsdu header
2023-10-27 11:04:33 -07:00
Tim Wojtulewicz
091c849abe Merge remote-tracking branch 'security/topic/awelzel/200-pop-fuzzer-timeout'
* security/topic/awelzel/200-pop-fuzzer-timeout:
  ssl: Prevent unbounded ssl_history growth
  ssl: Cap number of alerts parsed from SSL record
2023-10-27 11:04:03 -07:00
Tim Wojtulewicz
d9534f687a Merge remote-tracking branch 'security/topic/awelzel/196-ftp-timeout-smaller-fix'
* security/topic/awelzel/196-ftp-timeout-smaller-fix:
  Update baselines
  ftp: Do not base seq on number of pending commands
2023-10-27 11:03:54 -07:00