Commit graph

16773 commits

Author SHA1 Message Date
Arne Welzel
bd8c5b7bbb Merge remote-tracking branch 'upstream/topic/awelzel/fix-btest-spicy-replaces-conflicts'
* upstream/topic/awelzel/fix-btest-spicy-replaces-conflicts:
  btest/spicy: Make replaces-conflicts trigger replaces code path
2024-08-06 12:03:18 +02:00
Arne Welzel
bae15230bb btest/spicy: Make replaces-conflicts trigger replaces code path
The current test attempts to instantiate two spicy::SSH_1 protocol
analyzers in the .evt file. The intention likely was to use two
distinct protocol analyzer both trying to replace the builtin SSH
analyzer.

Coincidentally, fixing this happens to workaround TSAN errors tickled
by the FatalError() call while loading the .hlto with two identically
named analyzers.

    $ cat .tmp/spicy.replaces-conflicts/output
    error: redefinition of protocol analyzer spicy::SSH_1
    ThreadSanitizer: main thread finished with ignores enabled
      One of the following ignores was not ended (in order of probability)
      Ignore was enabled at:
        #0 __llvm_gcov_init __linker___d192e45c25d5ee23-484d3e0fc2caf5b4.cc (ssh.hlto+0x34036) (BuildId: 091934ca4da885e7)
        #1 __llvm_gcov_init __linker___d192e45c25d5ee23-484d3e0fc2caf5b4.cc (ssh.hlto+0x34036) (BuildId: 091934ca4da885e7)
        ...

I was tempted to replace FatalError() with Error() and rely on
zeek-setup.cc's early exiting on any reporter errors, but this
seems easier for now.

Relates to #3865.
2024-08-06 11:52:52 +02:00
Tim Wojtulewicz
25f65a705f Updating CHANGES and VERSION. 2024-08-05 14:40:31 -07:00
Tim Wojtulewicz
b483fad3d0 Merge remote-tracking branch 'origin/topic/timw/3081-prof-log-metrics'
* origin/topic/timw/3081-prof-log-metrics:
  Remove some unnecessary #includes
  Avoid capturing 'this' for callback in telemetry::Manager
  Make telemetry metrics out of MsgThread statistics
  Split cpu time metric into user/system components like prof.log
  Move broker statistics to be telemetry metrics
  Add timer counts as telemetry metrics
  Move thread manager stats to telemetry metric
  Add extra metrics to session_mgr
  Move dns_mgr stats to telemetry instruments
  Move trigger stats to telemetry instruments
  Move pulling of global state inside 'expensive' check for stats
2024-08-05 14:39:56 -07:00
Tim Wojtulewicz
1325e16a0d Remove some unnecessary #includes 2024-08-05 13:27:30 -07:00
Tim Wojtulewicz
7a1eb78b67 Avoid capturing 'this' for callback in telemetry::Manager 2024-08-05 13:27:30 -07:00
Tim Wojtulewicz
73f71e652d Make telemetry metrics out of MsgThread statistics 2024-08-05 13:27:30 -07:00
Tim Wojtulewicz
a6843067e9 Split cpu time metric into user/system components like prof.log
The total can be calculated from the two parts via Prometheus/Grafana
if desired, so it's more informative to pass them as separate parts.
2024-08-05 13:27:30 -07:00
Tim Wojtulewicz
206f5cd522 Move broker statistics to be telemetry metrics 2024-08-05 13:27:30 -07:00
Tim Wojtulewicz
44860676a2 Add timer counts as telemetry metrics 2024-08-05 13:21:44 -07:00
Tim Wojtulewicz
4face43462 Move thread manager stats to telemetry metric 2024-08-05 13:21:44 -07:00
Tim Wojtulewicz
a81f6ab9a6 Add extra metrics to session_mgr
- Sessions killed by activity
- Current number of sessions across all types
2024-08-05 13:21:44 -07:00
Tim Wojtulewicz
d1f7999f61 Move dns_mgr stats to telemetry instruments 2024-08-02 21:27:26 -07:00
Tim Wojtulewicz
8b4af06484 Move trigger stats to telemetry instruments 2024-08-02 21:27:26 -07:00
Tim Wojtulewicz
77c05357b5 Move pulling of global state inside 'expensive' check for stats 2024-08-02 21:27:26 -07:00
zeek-bot
8acc5ae15e Update doc submodule [nomail] [skip ci] 2024-08-03 00:12:18 +00:00
Tim Wojtulewicz
3c3853dc7d Merge remote-tracking branch 'origin/topic/timw/telemetry-threading'
* origin/topic/timw/telemetry-threading:
  Process metric callbacks from the main-loop thread
2024-08-02 15:50:47 -07:00
Tim Wojtulewicz
7ac7ce1d2b Process metric callbacks from the main-loop thread
This avoids the callbacks from being processed on the worker thread
spawned by Civetweb. It fixes data race issues with lookups involving
global variables, amongst other threading issues.
2024-08-02 15:30:47 -07:00
Arne Welzel
9d9cc51e9d Merge remote-tracking branch 'origin/topic/timw/use-more-memory-for-freebsd-builds'
* origin/topic/timw/use-more-memory-for-freebsd-builds:
  CI: Use 16GB of memory for FreeBSD builds
2024-08-01 11:14:46 +02:00
Tim Wojtulewicz
af65c29a3e CI: Use 16GB of memory for FreeBSD builds 2024-07-31 10:19:11 -07:00
Arne Welzel
ed4acd2437 Update doc submodule [nomail] [skip ci] 2024-07-31 12:22:08 +02:00
zeek-bot
e9adb8e462 Update doc submodule [nomail] [skip ci] 2024-07-30 00:10:33 +00:00
Tim Wojtulewicz
4c0c7581c8 Merge remote-tracking branch 'origin/topic/bbannier/bump-spicy'
* origin/topic/bbannier/bump-spicy:
  Bump auxil/spicy to latest development snapshot
2024-07-26 10:12:04 -07:00
Arne Welzel
152bbbd680 Merge remote-tracking branch 'origin/topic/awelzel/3853-ldap-spnego-ntlmssp'
* origin/topic/awelzel/3853-ldap-spnego-ntlmssp:
  ldap: Recognize SASL+SPNEGO+NTLMSSP
2024-07-26 15:14:52 +02:00
Arne Welzel
02e3d30227 ldap: Recognize SASL+SPNEGO+NTLMSSP
The ctu-sme-11-win7ad-1-ldap-tcp-50041.pcap file was harvested
from the CTU-SME-11 (Experiment-VM-Microsoft-Windows7AD-1) dataset
at https://zenodo.org/records/7958259 (DOI 10.5281/zenodo.7958258).

Closes #3853
2024-07-26 14:37:36 +02:00
Benjamin Bannier
666341fcbb Bump auxil/spicy to latest development snapshot
This in particular pulls in a fix for zeek/spicy#1808.
2024-07-26 13:28:31 +02:00
zeek-bot
7afb9b2afb Update doc submodule [nomail] [skip ci] 2024-07-26 00:10:24 +00:00
Tim Wojtulewicz
a40da34915 Merge remote-tracking branch 'botovq/opaque_x509_algor'
* botovq/opaque_x509_algor:
  Use accessor to reach into X509_ALGOR
2024-07-25 09:19:31 -07:00
Arne Welzel
93a430ce15 Merge remote-tracking branch 'origin/topic/awelzel/mysql-amazing-pp-issue-2716-additions'
* origin/topic/awelzel/mysql-amazing-pp-issue-2716-additions:
  mysql: Simplify length computation
  mysql: Improve date and time parsing
  btest/mysql: Clean query-attr.pcapng
  mysql: Support non-string query attributes
  btest/mysql: Add pcap with non-string query attributes
  mysql: Introduce mysql_ssl_request event
  mysql: Fix EOFIfLegacyThenResultSet
  mysql: Add data parameter to mysql_auth_plugin
  mysql: Add mysql_auth_plugin, mysql_auth_more_data and mysql_auth_switch_request events
  mysql: AuthSwitchRequest: &enforce a 0xfe / 254 status
  mysql: Make auth_plugin_ a std::string
  mysql: Fix auth_plugin_data_part2 length computation
  Refactored connection phase state handling
  Add support for "auth switch" and "query attrs"
  Add support for parsing the "caching_sha2_password" auth plugin
2024-07-25 12:18:31 +02:00
Arne Welzel
2e1e57033d mysql: Simplify length computation
Thanks Tim!
2024-07-25 11:53:41 +02:00
Fupeng Zhao
4adea7978c mysql: Improve date and time parsing 2024-07-25 11:53:41 +02:00
Arne Welzel
7b8bbc6d00 btest/mysql: Clean query-attr.pcapng
It contained some unrelated IntelliJ traffic.
2024-07-25 11:53:41 +02:00
Arne Welzel
2894ae38d0 mysql: Support non-string query attributes
The query attributes aren't exposed to script layer right now, but this
should at least parse over them once encountered and some fixups.
2024-07-25 11:53:41 +02:00
Arne Welzel
93f5813be3 btest/mysql: Add pcap with non-string query attributes
Pcap was generated as follows. Doesn't seem wireshark even parses
this properly right now.

    with common.get_connection() as c:
        with c.cursor() as cur:
            date1 = datetime.date(1987, 10, 18)
            datetime1 = datetime.datetime(1990, 9, 26, 12, 13, 14)
            cur.add_attribute("number1", 42)
            cur.add_attribute("string1", "a string")
            cur.add_attribute("date1", date1)
            cur.add_attribute("datetime1", datetime1)
            cur.execute("SELECT version()")
            result = cur.fetchall()
            print("result", result)
2024-07-25 11:53:41 +02:00
Arne Welzel
0a1568f1a1 mysql: Introduce mysql_ssl_request event
This should've been added with fa48c885 for completion. Do it now.
The MySQL spec calls it SSLRequest packet, so keep SSL in the name for
consistency.
2024-07-25 11:53:41 +02:00
Arne Welzel
6ea1045245 mysql: Fix EOFIfLegacyThenResultSet
Only expect a result next if an EOF was consumed.
2024-07-25 11:53:41 +02:00
Arne Welzel
320923418c mysql: Add data parameter to mysql_auth_plugin
This may contain salt from the server or a hashed password from the client.
2024-07-25 11:53:41 +02:00
Arne Welzel
40f1c2cb6d mysql: Add mysql_auth_plugin, mysql_auth_more_data and mysql_auth_switch_request events
Remove caching_sha2_password parsing/state from the analyzer and implement
the generic events. If we actually want to peak into the authentication
mechanism, we could write a separate analyzer for it. For now, treat it
as opaque values that are exposed to script land.

The added tests show the --get-server-public-key in use where
mysql_auth_more_data contains an RSA public key.
2024-07-25 11:53:41 +02:00
Arne Welzel
8a92945b06 mysql: AuthSwitchRequest: &enforce a 0xfe / 254 status 2024-07-25 11:53:41 +02:00
Arne Welzel
48e76f38cb mysql: Make auth_plugin_ a std::string 2024-07-25 11:53:40 +02:00
Arne Welzel
e98b80d140 mysql: Fix auth_plugin_data_part2 length computation 2024-07-25 11:53:40 +02:00
Fupeng Zhao
c82756bda4 Refactored connection phase state handling
Added `ConnectionExpected` enum for expected packet types during the connection phase.
2024-07-25 11:53:40 +02:00
Fupeng Zhao
e8bdf149f2 Add support for "auth switch" and "query attrs"
Also fix the issue where Resultset could not correctly distinguish between EOF_Packet and OK_Packet.
2024-07-25 11:53:40 +02:00
Fupeng Zhao
9cb618c718 Add support for parsing the "caching_sha2_password" auth plugin 2024-07-25 11:53:35 +02:00
Tim Wojtulewicz
6527850487 Merge remote-tracking branch 'origin/topic/timw/format-binpac-output-code'
* origin/topic/timw/format-binpac-output-code:
  Update binpac submodule to better format output code [nomail]
2024-07-24 15:29:08 -07:00
Tim Wojtulewicz
c4907c3f77 Update binpac submodule to better format output code [nomail] 2024-07-24 13:33:54 -07:00
Tim Wojtulewicz
bd208f4c54 Merge remote-tracking branch 'origin/topic/timw/smb2-ioctl-errors'
* origin/topic/timw/smb2-ioctl-errors:
  Update 7.0 NEWS with blurb about multi-PDU parsing causing increased load [nomail] [skip ci]
  Fix handling of zero-length SMB2 error responses
2024-07-24 13:26:07 -07:00
Tim Wojtulewicz
73fd12a76e Update 7.0 NEWS with blurb about multi-PDU parsing causing increased load [nomail] [skip ci] 2024-07-24 13:20:47 -07:00
Tim Wojtulewicz
2844d54f67 Fix handling of zero-length SMB2 error responses 2024-07-24 12:44:46 -07:00
Arne Welzel
52bcc92e40 Merge remote-tracking branch 'origin/topic/vern/script-opt-maint.Aug24'
* origin/topic/vern/script-opt-maint.Aug24:
  minor optimization of boolean comparisons
  fix & regression test for GH-3839 (spurious warnings for "when" constructs)
2024-07-24 11:18:18 +02:00