Mirror/zeek
1
0
Fork 0
mirror of https://github.com/zeek/zeek.git synced 2025-10-02 06:38:20 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 1
18258 commits 387 branches 195 tags 255 MiB
Commit graph

18258 commits

This branch
This branch
All branches
Author SHA1 Message Date
Arne Welzel
c687a24503 site/local: Switch to detect-sql-injection 2025-05-20 16:24:28 +02:00
Vern Paxson
dcd14f7a16 Add a revised script for detecting HTTP SQL injection, deprecate original 2025-05-20 16:24:20 +02:00
Arne Welzel
6b6c3dbbb7 Merge remote-tracking branch 'origin/topic/justin/file_analysis_speedup' 
* origin/topic/justin/file_analysis_speedup:
  speed up file analysis, remove IncrementByteCount
 2025-05-09 17:25:55 +02:00
Justin Azoff
7f350587b0 speed up file analysis, remove IncrementByteCount 
Avoid creating and recreating count objects for each chunk of file
analyzed.  This replaces counts inside of records with c++ uint64_ts.

On a pcap containing a 100GB file download this gives a 9% speedup

    Benchmark 1 (3 runs): zeek-master/bin/zeek -Cr http_100g_zeroes.pcap tuning/json-logs frameworks/files/hash-all-files
      measurement          mean ± σ            min … max           outliers         delta
      wall_time           102s  ± 1.23s      101s  …  103s           0 ( 0%)        0%
      peak_rss            108MB ±  632KB     107MB …  109MB          0 ( 0%)        0%
      cpu_cycles          381G  ±  862M      380G  …  382G           0 ( 0%)        0%
      instructions        663G  ± 5.16M      663G  …  663G           0 ( 0%)        0%
      cache_references   1.03G  ±  109M      927M  … 1.15G           0 ( 0%)        0%
      cache_misses       12.3M  ±  587K     11.7M  … 12.9M           0 ( 0%)        0%
      branch_misses      1.23G  ± 2.10M     1.22G  … 1.23G           0 ( 0%)        0%
    Benchmark 2 (3 runs): zeek-file_analysis_speedup/bin/zeek -Cr http_100g_zeroes.pcap tuning/json-logs frameworks/files/hash-all-files
      measurement          mean ± σ            min … max           outliers         delta
      wall_time          92.9s  ± 1.85s     91.8s  … 95.1s           0 ( 0%)        ⚡-  9.0% ±  3.5%
      peak_rss            108MB ±  393KB     108MB …  109MB          0 ( 0%)          +  0.1% ±  1.1%
      cpu_cycles          341G  ±  695M      341G  …  342G           0 ( 0%)        ⚡- 10.4% ±  0.5%
      instructions        605G  ±  626M      605G  …  606G           0 ( 0%)        ⚡-  8.7% ±  0.2%
      cache_references    831M  ± 16.9M      813M  …  846M           0 ( 0%)        ⚡- 19.6% ± 17.2%
      cache_misses       12.4M  ± 1.48M     11.4M  … 14.1M           0 ( 0%)          +  0.3% ± 20.8%
      branch_misses      1.02G  ± 3.45M     1.02G  … 1.02G           0 ( 0%)        ⚡- 16.8% ±  0.5%
 2025-05-09 10:50:04 -04:00
Arne Welzel
20ada619c5 Updating submodules binpac, broker and zeek-aux [nomail] 2025-05-09 09:52:20 +02:00
zeek-bot
b629b6bfe6 Update doc submodule [nomail] [skip ci] 2025-05-09 00:23:30 +00:00
Tim Wojtulewicz
957098af1c Merge remote-tracking branch 'origin/topic/bbannier/bump-spicy' 
* origin/topic/bbannier/bump-spicy:
  Bump auxil/spicy to latest development snapshot
 2025-05-08 15:14:37 -07:00
Christian Kreibich
8d79429555 Merge branch 'topic/christian/fix-broker-peering-overflows-metric' 
* topic/christian/fix-broker-peering-overflows-metric:
  Bugfix: accurately track Broker buffer overflows w/ multiple peerings
 2025-05-08 15:07:39 -07:00
Tim Wojtulewicz
2c17c85f55 Merge branch 'topic/timw/switch-broker-error-to-error' 
* topic/timw/switch-broker-error-to-error:
  Downgrade broker clone FatalError to an Error
 2025-05-08 13:47:06 -07:00
Tim Wojtulewicz
e00942c021 Downgrade broker clone FatalError to an Error 2025-05-08 13:46:21 -07:00
Benjamin Bannier
abbc4f23b8 Bump auxil/spicy to latest development snapshot 2025-05-08 21:54:15 +02:00
Arne Welzel
7583651bec Merge remote-tracking branch 'origin/topic/awelzel/ci-fedora-42' 
* origin/topic/awelzel/ci-fedora-42:
  probabilistic/BitVector: Add include <cstdint>
  Bump spicy to fix build with GCC 15.1
  CI: Drop fedora-40
  CI: Add fedora-42
 2025-05-08 13:19:30 +02:00
Arne Welzel
f80abe9d45 probabilistic/BitVector: Add include <cstdint> 2025-05-08 10:37:32 +02:00
Arne Welzel
43c20f41da Bump spicy to fix build with GCC 15.1 2025-05-08 10:37:32 +02:00
Arne Welzel
368a38a050 CI: Drop fedora-40 
Still 3 weeks to go, but not quite seeing why we should support 3
different Fedora releases with Zeek 7.2.
 2025-05-08 10:37:32 +02:00
Arne Welzel
aacc484543 CI: Add fedora-42 
This ships with gcc 15.1.1 and Python 3.13.3
 2025-05-08 10:37:29 +02:00
Arne Welzel
60efc7741d Merge branch 'gcc15' of https://github.com/i2z1/zeek 
* 'gcc15' of https://github.com/i2z1/zeek:
  Include cstdint header to support uint8_t/uint64_t in GCC 15+
 2025-05-08 09:25:28 +02:00
Christian Kreibich
738ce1c235 Bugfix: accurately track Broker buffer overflows w/ multiple peerings 
When a node restarts or a peering between two nodes starts over for other
reasons, the internal tracking in the Broker manager resets its state (since
it's per-peering), and thus the message overflow counter. The script layer was
unaware of this, and threw errors when trying to reset the corresponding counter
metric down to zero at sync time.

We now track past buffer overflows via a separate epoch table, using Broker peer
ID comparisons to identify new peerings, and set the counter to the sum of past
and current overflows.

I considered just making this a gauge, but it seems more helpful to be able to
look at a counter to see whether any messages have ever been dropped over the
lifetime of the node process.

As an aside, this now also avoids repeatedly creating the labels vector,
re-using the same one for each metric.

Thanks to @pbcullen for identifying this one!
 2025-05-07 17:27:38 -07:00
zeek-bot
42365b85f0 Update doc submodule [nomail] [skip ci] 2025-05-08 00:14:52 +00:00
Tim Wojtulewicz
1dfbd33929 Merge remote-tracking branch 'origin/topic/timw/storage-timeout-option' 
* origin/topic/timw/storage-timeout-option:
  Move options to redis backend options instead of module-level options
  Add operation_timeout and command_timeout storage backend options
 2025-05-07 15:39:35 -07:00
Tim Wojtulewicz
fd10dd015f Move options to redis backend options instead of module-level options 2025-05-07 15:38:58 -07:00
Tim Wojtulewicz
824b91216f Add operation_timeout and command_timeout storage backend options 2025-05-07 15:38:58 -07:00
Tim Wojtulewicz
507974a1d8 Merge remote-tracking branch 'origin/topic/christian/news-fix' 
* origin/topic/christian/news-fix:
  Fix Broker metrics naming bugs that snuck in w/ last-minute renaming [skip ci]
 2025-05-07 14:09:52 -07:00
i2z1
c0da0642b1
Include cstdint header to support uint8_t/uint64_t in GCC 15+ 2025-05-07 22:42:01 +03:00
Christian Kreibich
6657e4efbd Fix Broker metrics naming bugs that snuck in w/ last-minute renaming [skip ci] 2025-05-07 12:32:43 -07:00
Tim Wojtulewicz
6f8924596f Merge remote-tracking branch 'origin/topic/johanna/fix-failed-service-logging' 
* origin/topic/johanna/fix-failed-service-logging:
  Fix policy/protocols/conn/failed-service-logging.zeek
 2025-05-07 10:29:54 -07:00
Tim Wojtulewicz
8096388904 Update opt.ZAM-bif-tracking baseline 2025-05-07 09:12:56 -07:00
Arne Welzel
8089f5bed4 Merge remote-tracking branch 'origin/topic/awelzel/more-terminate-while-queueing-hardening' 
* origin/topic/awelzel/more-terminate-while-queueing-hardening:
  btest/cluster/generic/publish-any: Apply Christian's fix from broker/publish-any
  wstest/terminate-while-queueing: Patch close_socket()
 2025-05-07 17:24:04 +02:00
Arne Welzel
3ec3205074 btest/cluster/generic/publish-any: Apply Christian's fix from broker/publish-any 2025-05-07 17:18:01 +02:00
Tim Wojtulewicz
d7b31f5b44 Merge remote-tracking branch 'origin/topic/timw/storage-is-connected-bif' 
* origin/topic/timw/storage-is-connected-bif:
  Add Storage::is_connected BIF
  Move cast_handle from storage-async.bif to BackendHandleVal static method
 2025-05-07 08:16:59 -07:00
Tim Wojtulewicz
58ee8d3c5c Add Storage::is_connected BIF 2025-05-07 08:13:16 -07:00
Tim Wojtulewicz
cbc41e298d Move cast_handle from storage-async.bif to BackendHandleVal static method 2025-05-07 08:13:16 -07:00
Arne Welzel
82731992d9 wstest/terminate-while-queueing: Patch close_socket() 
I believe there's a bug/usage issue in the websockets library
where during send(), EOF is detected and stored, but the receiving
thread is then discarding the last received frame. Avoid the bug
by replacing the close_socket() implementation of the websockets
library just for that test and leave detecting the EOF condition
to the receiving thread.
 2025-05-07 16:33:54 +02:00
Arne Welzel
135acc7c6d Merge remote-tracking branch 'origin/topic/awelzel/fix-flaky-terminate-while-queueing' 
* origin/topic/awelzel/fix-flaky-terminate-while-queueing:
  cluster/websocket: Stop and wait for reply thread during Terminate()
 2025-05-07 13:22:13 +02:00
Arne Welzel
ca02316671 cluster/websocket: Stop and wait for reply thread during Terminate() 
The terminate-while-queueing test added for #4428 failed spuriously
indicating that sometimes WebSocket clients receive code 1000 instead of 1001.
This happens if the ixwebsocket server is shutdown before the reply thread had a
chance to process queued close messages.

Fix by signaling and waiting for the dispatcher's reply thread to terminate
before returning from Terminate().
 2025-05-07 12:45:01 +02:00
zeek-bot
ac1230fcbe Update doc submodule [nomail] [skip ci] 2025-05-07 00:15:52 +00:00
Tim Wojtulewicz
1da414ec3d Update paraglob submodule [nomail] 2025-05-06 13:19:49 -07:00
Johanna Amann
f293d5a852 Fix policy/protocols/conn/failed-service-logging.zeek 
In GH-4422 it was pointed out that the protocols/conn/failed-service-logging.zeek
policy script only works when
`DPD::track_removed_services_in_connection=T` is set.

This was caused by a logic error in the script. This commit fixes this
logic error and introduces an additional test that checks that
failed-service-logging works even when the option is not set to true.
 2025-05-06 13:37:12 +01:00
Arne Welzel
3be7a9ce91 Merge remote-tracking branch 'origin/topic/awelzel/double-commented-btest-lines' 
* origin/topic/awelzel/double-commented-btest-lines:
  testing/btest: Fix double commented @TEST- lines
 2025-05-06 14:21:03 +02:00
Arne Welzel
4afb0ffeeb Merge remote-tracking branch 'origin/topic/awelzel/4420-simeon-breaking-websockets' 
* origin/topic/awelzel/4420-simeon-breaking-websockets:
  Websocket: Close onloop during Terminate()
  OnLoop: notify_all() instead of notify_one()
 2025-05-06 14:20:38 +02:00
Arne Welzel
bb06af601f Websocket: Close onloop during Terminate() 
Terminate() is called when Zeek shuts down. If WebSocket client threads
were blocked in QueueForProcessing() due to reaching queue limits, these
previously would not exit QueueForProcessing() and instead block
indefinitely, resulting in the ixwebsocket library blocking and its
garbage collection thread running at 100%. Not great.

Closing the onloop instance will unblock the WebSocket client threads
for a timely shutdown.

Closes #4420
 2025-05-06 14:19:08 +02:00
Arne Welzel
0e327a0c12 testing/btest: Fix double commented @TEST- lines 
sed -i 's/^# # @/# @/g'
 2025-05-06 14:06:29 +02:00
Arne Welzel
6ebec6dde7 OnLoop: notify_all() instead of notify_one() 
There might be more than one thread blocked waiting for room in the
queue, ensure they all wake up when shutting down.
 2025-05-06 12:23:08 +02:00
zeek-bot
e986caddf0 Update doc submodule [nomail] [skip ci] 2025-05-06 00:16:07 +00:00
Tim Wojtulewicz
0393e4b84a Merge remote-tracking branch 'XueSongTap/master' 
* XueSongTap/master:
  Add baseline for find_first test, update comments, and reorder function imports
  Add find_first string function
 2025-05-05 13:40:40 -07:00
Tim Wojtulewicz
f9aa9a430d Merge remote-tracking branch 'origin/topic/timw/storage-check-redis-server-version' 
* origin/topic/timw/storage-check-redis-server-version:
  Add commands to the static methods for the Redis implementation
  Redis: Check server version when connecting
 2025-05-05 11:12:09 -07:00
Tim Wojtulewicz
dda319fd53 Add commands to the static methods for the Redis implementation 2025-05-05 11:10:06 -07:00
Tim Wojtulewicz
ecd603516f Redis: Check server version when connecting 2025-05-05 10:49:37 -07:00
Tim Wojtulewicz
58d71d2fa3 Merge remote-tracking branch 'origin/topic/timw/storage-redis-expire-string-view' 
* origin/topic/timw/storage-redis-expire-string-view:
  Use std::string_view in Redis::DoExpire to avoid copies
 2025-05-05 10:44:51 -07:00
Tim Wojtulewicz
df9b396ee6 Use std::string_view in Redis::DoExpire to avoid copies 2025-05-05 10:44:06 -07:00