Commit graph

16116 commits

Author SHA1 Message Date
Tim Wojtulewicz
a3af4a4b51 Merge branch 'topic/timw/more-string-view-usage'
* topic/timw/more-string-view-usage:
  Change to use ToStdStringView() in a few other BIFs
  Convert remove_prefix/suffix BIFs to use std::string_view
  Rework starts_with BIF similarly to ends_with changes in 1649e3e7cc
2024-01-23 10:41:37 -07:00
Tim Wojtulewicz
012acb17cc Change to use ToStdStringView() in a few other BIFs 2024-01-23 10:38:56 -07:00
Tim Wojtulewicz
c77f8cc898 Convert remove_prefix/suffix BIFs to use std::string_view 2024-01-23 10:38:56 -07:00
Tim Wojtulewicz
94ad676db0 Rework starts_with BIF similarly to ends_with changes in 1649e3e7cc 2024-01-23 10:38:56 -07:00
Arne Welzel
822ca99e80 Merge remote-tracking branch 'origin/topic/awelzel/3424-http-upgrade-websocket-v1'
* origin/topic/awelzel/3424-http-upgrade-websocket-v1:
  websocket: Handle breaking from WebSocket::configure_analyzer()
  websocket: Address review feedback for BinPac code
  fuzzers: Add WebSocket fuzzer
  websocket: Fix crash for fragmented messages
  websocket: Verify Sec-WebSocket-Key/Accept headers and review feedback
  btest/websocket: Test for coalesced reply-ping
  HTTP/CONNECT: Also weird on extra data in reply
  HTTP/Upgrade: Weird when more data is available
  ContentLine: Add GetDeliverStreamRemainingLength() accessor
  HTTP: Drain event queue after instantiating upgrade analyzer
  btest/http: Explain switching-protocols test change as comment
  WebSocket: Introduce new analyzer and log
  HTTP: Add mechanism to instantiate Upgrade analyzer
2024-01-23 18:17:50 +01:00
zeek-bot
dddb8052d2 Update doc submodule [nomail] [skip ci] 2024-01-23 00:20:55 +00:00
Arne Welzel
9654226075 websocket: Handle breaking from WebSocket::configure_analyzer()
...and various nits from the review.
2024-01-22 18:54:41 +01:00
Arne Welzel
015a7c5fbc websocket: Address review feedback for BinPac code
* Rename mask_ to masking_key_
* Fold FrameHeaderFixed into FrameHeader directly
* Drop WebSocket_FramePayloadUnmask type

Thanks a bunch @ckreibich!
2024-01-22 18:54:41 +01:00
Arne Welzel
1775b01b58 fuzzers: Add WebSocket fuzzer
It immediately found an issue with &transient, but fairly stable thereafter.

This is a separate fuzzer implementation as there's a custom Configure()
call for the analyzer as well as disabling all other analyzers so we
don't fuzz unrelated protocols.
2024-01-22 18:54:38 +01:00
Arne Welzel
5eb380d74a websocket: Fix crash for fragmented messages
The &transient attribute does not work well with $element as that won't
be available within &until anymore apparently.

Found after a few seconds building out the fuzzer.
2024-01-22 18:54:38 +01:00
Arne Welzel
e17655be61 websocket: Verify Sec-WebSocket-Key/Accept headers and review feedback
Don't log them, they are random and arbitrary in the normal case. Users
can do the following to log them if wanted.

    redef += WebSocket::Info$client_key += { &log };
    redef += WebSocket::Info$server_accept += { &log };
2024-01-22 18:54:38 +01:00
Arne Welzel
a6c1d12206 btest/websocket: Test for coalesced reply-ping
Add a constructed PCAP where the HTTP/websocket server send a WebSocket
ping message directly with the packet of the HTTP reply. Ensure this is
interpreted the same as if the WebSocket message is in a separate packet
following the HTTP reply.

For the server side this should work, for the client side we'd need to
synchronize suspend parsing the client side as we currently cannot quite
know whether it's a pipelined HTTP request following, or upgraded protocol
data and we don't have "suspend parsing" functionality here.
2024-01-22 18:54:38 +01:00
Arne Welzel
4d81389df0 HTTP/CONNECT: Also weird on extra data in reply 2024-01-22 18:54:38 +01:00
Arne Welzel
de836ab528 HTTP/Upgrade: Weird when more data is available
After an HTTP upgrade to another protocol, create a weird if the packet
that contains the HTTP reply *also* contains some additional data
belonging to the upgraded to protocol already.
2024-01-22 18:54:38 +01:00
Arne Welzel
2b9776adca ContentLine: Add GetDeliverStreamRemainingLength() accessor
Helper to get information from the ContentLine analyzer about
bytes still pending to be delivered. In certain cases this can
be a signal for weirdness.
2024-01-22 18:54:38 +01:00
Arne Welzel
7967ef993b HTTP: Drain event queue after instantiating upgrade analyzer
With configurability through script-land comes the draw back
that we actually need to execute event handlers in the middle
of the parsing process: This might not be the best model, but
the script-side configurability it enables is kind of nice.

This explicit call only matters here when the HTTP reply is
directly followed by some WebSocket message data within the
same network packet, otherwise the queue is drained once the
packet has been completely processed anyhow.
2024-01-22 18:54:38 +01:00
Arne Welzel
37521f58e5 btest/http: Explain switching-protocols test change as comment
DPD enables HTTP based on the content of the WebSocket frames. However,
it's not HTTP, the protocol is x-kaazing-handshake and the server sends
some form of status/acknowledge to the client first, so the HTTP and the
HTTP analyzer receives that as the first bytes of the response and
bails, oh well.
2024-01-22 18:54:38 +01:00
Arne Welzel
efc2681152 WebSocket: Introduce new analyzer and log
This adds a new WebSocket analyzer that is enabled with the HTTP upgrade
mechanism introduced previously. It is a first implementation in BinPac with
manual chunking of frame payload. Configuration of the analyzer is sketched
via the new websocket_handshake() event and a configuration BiF called
WebSocket::__configure_analyzer(). In short, script land collects WebSocket
related HTTP headers and can forward these to the analyzer to change its
parsing behavior at websocket_handshake() time. For now, however, there's
no actual logic that would change behavior based on agreed upon extensions
exchanged via HTTP headers (e.g. frame compression). WebSocket::Configure()
simply attaches a PIA_TCP analyzer to the WebSocket analyzer for dynamic
protocol detection (or a custom analyzer if set). The added pcaps show this
in action for tunneled ssh, http and https using wstunnel. One test pcap is
Broker's WebSocket traffic from our own test suite, the other is the
Jupyter websocket traffic from the ticket/discussion.

This commit further adds a basic websocket.log that aggregates the WebSocket
specific headers (Sec-WebSocket-*) headers into a single log.

Closes #3424
2024-01-22 18:54:38 +01:00
Arne Welzel
8ebd054abc HTTP: Add mechanism to instantiate Upgrade analyzer
When a HTTP upgrade request/reply is detected, lookup an analyzer tag
from HTTP::upgrade_analyzers, or if nothing is found, attach PIA_TCP.
2024-01-22 18:54:38 +01:00
Tim Wojtulewicz
b5f9e5a3b1 Merge remote-tracking branch 'origin/topic/timw/remove-bifreturnval'
* origin/topic/timw/remove-bifreturnval:
  Make BIFs just return ValPtr directly instead of BifReturnVal
2024-01-22 10:36:30 -07:00
Tim Wojtulewicz
2aaaab4dad Make BIFs just return ValPtr directly instead of BifReturnVal 2024-01-22 10:35:34 -07:00
Tim Wojtulewicz
5602546f2e Merge remote-tracking branch 'origin/topic/timw/stop-supporting-bro-topics'
* origin/topic/timw/stop-supporting-bro-topics:
  Drop support for subscribing to bro/ topics
2024-01-22 10:01:23 -07:00
Tim Wojtulewicz
13fde341d2 Merge remote-tracking branch 'security/topic/awelzel/topic/awelzel/208-http-mime-nested-v2'
* security/topic/awelzel/topic/awelzel/208-http-mime-nested-v2:
  MIME: Cap nested MIME analysis depth to 100
2024-01-21 19:31:14 -07:00
zeek-bot
6bfdc06243 Update doc submodule [nomail] [skip ci] 2024-01-20 00:11:27 +00:00
Arne Welzel
029c44c789 Merge remote-tracking branch 'origin/topic/awelzel/smtp-bdat-follow-up-2'
* origin/topic/awelzel/smtp-bdat-follow-up-2:
  SMTP/BDAT: Use strtoull and bail on UULONG_MAX values
  SMTP/BDAT: Fix int/int64_t/uint64_t confusion
  SMTP: Reject BDAT chunks larger than int64_t's max value
2024-01-19 21:19:05 +01:00
Christian Kreibich
832ce9f9a1 Merge branch 'topic/christian/more-feature-tests'
* topic/christian/more-feature-tests:
  Default to setting ZEEK_HAVE_JAVASCRIPT=no in CMakeLists.txt
  Show --disable-javascript in `configure --help`
  Show AF_PACKET support status in cmake output, and sort features
  Add feature tests for AF_PACKET, GeoIP, and JavaScript to zeek-config
2024-01-19 10:26:23 -08:00
Arne Welzel
e1ed709243 SMTP/BDAT: Use strtoull and bail on UULONG_MAX values 2024-01-19 13:24:07 +01:00
Arne Welzel
c23d605286 SMTP/BDAT: Fix int/int64_t/uint64_t confusion
The BDAT analyzer should be supporting uint64_t sized chunks reasonably well,
but the ContentLine analyzer does not, And also, I totally got types for
RemainingChunkSize() and in DeliverStream() wrong, resulting in overflows
and segfaults when very large chunk sizes were used.

Tickled by OSS-Fuzz. Actually running the fuzzer locally only took a
few minutes to find the crash, too. Embarrassing.
2024-01-19 13:05:26 +01:00
Arne Welzel
0318ddbee9 SMTP: Reject BDAT chunks larger than int64_t's max value
The ContentLine analyzer does not support uint64_t.
2024-01-19 13:05:24 +01:00
Benjamin Bannier
638e8a0519 Merge branch 'topic/bbannier/issue-3177' 2024-01-19 12:27:59 +01:00
Benjamin Bannier
9aba561e44 Make sure Spicy symbols are available.
The Spicy/HILTI runtime libraries are modelled as object libraries and
linked into the `spicy` and `hilti` libraries. These libraries are then
linked into `zeek_objs` which is another object library linked into the
`zeek` executable (target `zeek_exe`). On some platforms this transitive
link of object libraries is broken with `--binary-package` which causes
creation of static archives, even with our minimum required CMake
version which already contains many fixes for object libraries.

With this patch we now explicitly linked the Spicy/HILTI runtime
libraries into `zeek_exe`.

Closes #3177.
2024-01-19 12:26:44 +01:00
Christian Kreibich
6d10082cc3 Merge branch 'topic/christian/a-couple-of-nits'
* topic/christian/a-couple-of-nits:
  Fix a zeek_init -> zeek_done confusion in a docstring [skip ci]
  Fix typo in docstring [skip ci]
2024-01-18 16:17:21 -08:00
Christian Kreibich
3d95bd08ed Fix a zeek_init -> zeek_done confusion in a docstring [skip ci]
Thanks to Siavash Tahmureszadeh for flagging this one.
2024-01-18 16:14:33 -08:00
Christian Kreibich
ae2fd8f171 Fix typo in docstring [skip ci] 2024-01-18 16:14:27 -08:00
Christian Kreibich
2947bcda56 Default to setting ZEEK_HAVE_JAVASCRIPT=no in CMakeLists.txt
This still overrides it as appropriate when building with JS support, but shows
"JavaScript: no" in the configure output also when using
--disable-javascript, where it previously showed no output.
2024-01-18 15:52:19 -08:00
Christian Kreibich
b97907daed Show --disable-javascript in configure --help
This is already supported and just missing from the help output.
2024-01-18 15:52:19 -08:00
Christian Kreibich
e8007f04d9 Show AF_PACKET support status in cmake output, and sort features 2024-01-18 15:52:19 -08:00
Christian Kreibich
8ae560e7c6 Add feature tests for AF_PACKET, GeoIP, and JavaScript to zeek-config 2024-01-18 15:52:15 -08:00
zeek-bot
3b7749b8de Update doc submodule [nomail] [skip ci] 2024-01-18 00:13:18 +00:00
Arne Welzel
2a858d252e MIME: Cap nested MIME analysis depth to 100
OSS-Fuzz managed to produce a MIME multipart message construction with
thousands of nested entities (or that's what Zeek makes out of it anyhow).
Prevent such deep analysis by capping at a nesting depth of 100,
preventing unnecessary resource usage. A new weird named exceeded_mime_max_depth
is reported when this limit is reached.

This change reduces the runtime of the OSS-Fuzz reproducer from ~45 seconds
to ~2.5 seconds.

The test PCAP was produced from a Python script using the email package
and sending the rendered version via POST to a HTTP server.

Closes #208
2024-01-17 10:18:13 -07:00
Tim Wojtulewicz
cea7c473ac Merge remote-tracking branch 'origin/topic/timw/security-darwin-builds'
* origin/topic/timw/security-darwin-builds:
  Enable darwin builds for zeek-security repo
2024-01-17 10:00:21 -07:00
Tim Wojtulewicz
c4edd8410e Enable darwin builds for zeek-security repo 2024-01-17 09:59:49 -07:00
zeek-bot
bf3e4c5291 Update doc submodule [nomail] [skip ci] 2024-01-17 00:12:17 +00:00
Tim Wojtulewicz
9033018509 Drop support for subscribing to bro/ topics 2024-01-16 16:16:01 -07:00
Tim Wojtulewicz
1649e3e7cc Merge remote-tracking branch 'origin/topic/timw/ends-with-rework'
* origin/topic/timw/ends-with-rework:
  Squeeze a bit more performance out of the ends_with bif
2024-01-16 12:07:25 -07:00
Tim Wojtulewicz
b962bd30ce Squeeze a bit more performance out of the ends_with bif 2024-01-16 12:06:08 -07:00
Arne Welzel
378f380b71 Merge remote-tracking branch 'origin/topic/awelzel/smtp-bdat-follow-up'
* origin/topic/awelzel/smtp-bdat-follow-up:
  SMTP: No state update for bad BDAT commands
  SMTP/BDAT: Harden BDAT argument parsing a bit
2024-01-16 18:04:51 +01:00
Johanna Amann
273731e1ce Merge branch 'topic/johanna/fix-logging-of-ssl-log-ext-in-some-cases'
* topic/johanna/fix-logging-of-ssl-log-ext-in-some-cases:
  Fix ssl-log-ext omitting data in some cases
2024-01-16 13:10:35 +00:00
Arne Welzel
ae2a5c83a4 SMTP: No state update for bad BDAT commands
OSS-Fuzz found that providing an invalid BDAT line would tickle an
assert in UpdateState(). The BDAT state was never initialized, but
within UpdateState() that was expected.

This also removes the AnalyzerViolation() call for bad BDAT commands
and instead raises a weird. The SMTP analyzer is very lax and not triggering
the violation allows to parse the server's response to such an invalid
command.

PCAP files produced by a custom Python SMTP client against Postfix.
2024-01-15 18:25:41 +01:00
Arne Welzel
5ad11e00e3 SMTP/BDAT: Harden BDAT argument parsing a bit
If the size is followed by something and it's not " LAST", treat it
as an error.
2024-01-15 18:25:38 +01:00