Commit graph

16273 commits

Author SHA1 Message Date
Arne Welzel
3f7881a57b segment_profiling: Remove SegmentProfiler and load_sample event
While it seems interesting functionality, this hasn't been documented,
maintained or knowingly leveraged for many years.

There are various other approaches today, too:

* We track the number of event handler invocations regardless of
  profiling. It's possible to approximate a load_sample event by
  comparing the result of two get_event_stats() calls. Or, visualize
  the corresponding counters in a Prometheus setup to get an idea of
  event/s broken down by event names.

* HookCallFunction() allows to intercept script execution, including
  measuring the time execution takes.

* The global call_stack and g_frame_stack can be used from plugins
  (and even external processes) to walk the Zeek script stack at certain
  points to implement a sampling profiler.

* USDT probes or more plugin hooks will likely be preferred over Zeek
  builtin functionality in the future.

Relates to #3458
2024-01-03 11:55:54 +01:00
Johanna Amann
df37cadbe8 Merge remote-tracking branch 'origin/topic/neverlord/data-to-threading-field'
* origin/topic/neverlord/data-to-threading-field:
  Avoid extra copies in threading_field_to_data
2024-01-03 09:44:35 +00:00
zeek-bot
190753ab45 Update doc submodule [nomail] [skip ci] 2024-01-03 00:12:44 +00:00
Tim Wojtulewicz
5740dbcf20 Merge remote-tracking branch 'origin/topic/awelzel/smb-unbounded-recent-files-growth'
* origin/topic/awelzel/smb-unbounded-recent-files-growth:
  smb: Fix &read_expire not in effect due to &default=string_set() usage
2024-01-02 11:13:36 -07:00
Dominik Charousset
12a45c2ff1 Avoid extra copies in threading_field_to_data
Passing the `broker::data` paramter for `threading_field_to_data` by
value forces copies. However, the function only needs const access to
its parameter. Hence, taking the parameter as const-ref is the logical
choice and avoids unnecessary copies of individual data fields.
2023-12-22 11:58:14 +01:00
Arne Welzel
fea8ee2260 smb: Fix &read_expire not in effect due to &default=string_set() usage
The SMB::State$recent_files field is meant to have expiring entries.
However, due to usage of &default=string_set(), the &read_expire
attribute is not respected causing unbounded state growth. Replace
&default=string_set() with &default=set().

Thanks to ya-sato on Slack for reporting!

Related: zeek/zeek-docs#179, #3513.
2023-12-17 15:02:05 +01:00
Arne Welzel
7a9a40f822 Merge remote-tracking branch 'origin/topic/vern/table-create-opt'
* origin/topic/vern/table-create-opt:
  TableType: Convert table_hash unique_ptr
  streamlining of constructing script-level tables
2023-12-16 17:57:58 +01:00
Arne Welzel
be0a22fec2 TableType: Convert table_hash unique_ptr
Cannot remove the destructor as otherwise the compiler attempts to create
its implementation in Type.h where CompositeHash isn't a complete type
yet and std::unique_ptr's delete fails to be instantiated.
2023-12-16 17:33:48 +01:00
Vern Paxson
a11ee9038b streamlining of constructing script-level tables 2023-12-16 17:33:46 +01:00
Tim Wojtulewicz
d1d9b9a1be Merge remote-tracking branch 'origin/topic/neverlord/broker-format'
* origin/topic/neverlord/broker-format:
  Update broker submodule
2023-12-15 14:59:42 -07:00
Dominik Charousset
a9954b363f Update broker submodule 2023-12-15 14:59:04 -07:00
Tim Wojtulewicz
43edd3c945 Merge remote-tracking branch 'stevesmoot/master'
* stevesmoot/master:
  improve search-ability in  zeek.spicy
2023-12-15 13:10:48 -07:00
Arne Welzel
83caf6108e Merge remote-tracking branch 'origin/topic/awelzel/3494-no-more-btest-script-coverage-locally'
* origin/topic/awelzel/3494-no-more-btest-script-coverage-locally:
  ci/btest: Remove ZEEK_PROFILER_FILE from btest.cfg, set in ci/test.sh explicitly
  ci: Remove ZEEK_CI_DISABLE_SCRIPT_PROFILING logic
2023-12-15 18:58:25 +01:00
Arne Welzel
43fec1f2c8 ci/btest: Remove ZEEK_PROFILER_FILE from btest.cfg, set in ci/test.sh explicitly
The produced coverage files are of little use in current local workflows
and usually just end-up taking up disk space. ZEEK_PROFILER_FILE can be
set explicitly if there's a one-off need to produce these locally, too.
2023-12-15 15:23:52 +01:00
Arne Welzel
344369f169 ci: Remove ZEEK_CI_DISABLE_SCRIPT_PROFILING logic
To my knowledge this doesn't make a huge difference anymore and locally
I'm only using ASAN builds. It's not actually too slow.
2023-12-15 15:23:52 +01:00
Christian Kreibich
36ae384a9c Updating CHANGES and VERSION. 2023-12-14 20:17:24 -08:00
Christian Kreibich
2c4395c504 Bump binpac, zeek-archiver, and zeek-aux submodules.
These only pull in CI updates, no other changes.
2023-12-14 20:03:07 -08:00
Christian Kreibich
98e70d3cfc Merge branch 'topic/christian/ci-updates'
* topic/christian/ci-updates:
  CI: Move Debian variations from 11 to 12
  CI: Bump Ubuntu 23.04 to 23.10
  CI: Drop openSUSE Leap 15.4, about to EOL.
  CI: FreeBSD 14 is out now, 12 is about to EOL.
  CI: distro EOL comment tweaks
  CI: drop Fedora 37, add Fedora 39
2023-12-14 19:56:07 -08:00
Christian Kreibich
ba6fe9afe9 Updating CHANGES and VERSION. 2023-12-14 19:55:24 -08:00
zeek-bot
f6964cc6fc Update doc submodule [nomail] [skip ci] 2023-12-15 00:11:59 +00:00
Christian Kreibich
4f6c25bb91 CI: Move Debian variations from 11 to 12
This switches the ARM and static builds from 11 to 12. x86 and ARM now build by
default, and 11 only with fullci. 10 remains as-is, because we like to torture
ourselves.
2023-12-14 15:43:56 -08:00
Christian Kreibich
866d4b4127 CI: Bump Ubuntu 23.04 to 23.10 2023-12-14 15:35:13 -08:00
Smoot
a36ef3c6b0
Merge pull request #1 from stevesmoot/stevesmoot-doc++
improve search-ability in  zeek.spicy
2023-12-14 16:25:36 -05:00
Smoot
9414abe3f8
improve search-ability in zeek.spicy 2023-12-14 16:24:32 -05:00
Arne Welzel
0f56758d08 Merge remote-tracking branch 'origin/topic/awelzel/less-clunky-signature-event'
* origin/topic/awelzel/less-clunky-signature-event:
  rule-parse: Remove [event_name] syntax, deprecate msg as identifier
2023-12-14 10:28:35 +01:00
Arne Welzel
f3470843d6 rule-parse: Remove [event_name] syntax, deprecate msg as identifier
As suggested by Robin. Thanks.
2023-12-14 10:16:35 +01:00
Christian Kreibich
2b35ff67a1 Bump zeekctl submodule to pull in Python 3.12 ConfigParser fix. 2023-12-13 22:32:36 -08:00
Christian Kreibich
ddd56b9ad7 CI: Drop openSUSE Leap 15.4, about to EOL. 2023-12-13 22:03:25 -08:00
Christian Kreibich
18b8e6d6bc CI: FreeBSD 14 is out now, 12 is about to EOL. 2023-12-13 22:01:53 -08:00
Christian Kreibich
eb1c4bf019 CI: distro EOL comment tweaks 2023-12-13 17:45:36 -08:00
Christian Kreibich
ee8cc77050 CI: drop Fedora 37, add Fedora 39
This makes 39 the ony to run at all times, and 38 one to run only with fullci.
2023-12-13 17:45:13 -08:00
Arne Welzel
0b5126f650 Merge remote-tracking branch 'origin/topic/vern/script-opt-maint.Dec23'
* origin/topic/vern/script-opt-maint.Dec23:
  recent BTests that should be skipped when using -O gen-C++
  expanded ZAM maintenance notes & support scripts
  script optimization tracking of functions called by event engine or indirectly
  memory-handling fixes for information associated with low-level ZAM instructions
  fix for -O C++ lambda functions reporting errors/warnings
  revert problems with profiling attributes introduced by recent script-opt PR
  script optimization fixes for pattern tables
  regularized (some) types of pointers used in script optimization
  splitting off script optimization CSE into its own source files
  some very minor tidying of script optimization code/documentation
  fix for Trigger's whose termination leads to deleting other Trigger's
  bug fix for delayed logging
2023-12-12 09:52:15 +01:00
Vern Paxson
56cf3170c2 recent BTests that should be skipped when using -O gen-C++ 2023-12-12 09:45:19 +01:00
Vern Paxson
eab2f8a49f expanded ZAM maintenance notes & support scripts 2023-12-12 09:45:19 +01:00
Vern Paxson
dd91790f1e script optimization tracking of functions called by event engine or indirectly 2023-12-12 09:45:19 +01:00
Vern Paxson
e9b990254a memory-handling fixes for information associated with low-level ZAM instructions 2023-12-12 09:45:19 +01:00
Vern Paxson
e2646f9752 fix for -O C++ lambda functions reporting errors/warnings 2023-12-12 09:45:19 +01:00
Vern Paxson
6e0ce17c22 revert problems with profiling attributes introduced by recent script-opt PR 2023-12-12 09:45:19 +01:00
Vern Paxson
f573036099 script optimization fixes for pattern tables 2023-12-12 09:45:19 +01:00
Vern Paxson
709d410fcd regularized (some) types of pointers used in script optimization 2023-12-12 09:45:19 +01:00
Vern Paxson
dd389c0380 splitting off script optimization CSE into its own source files 2023-12-12 09:45:19 +01:00
Vern Paxson
a824bb1e4b some very minor tidying of script optimization code/documentation 2023-12-12 09:45:19 +01:00
Vern Paxson
8e0a62bfec fix for Trigger's whose termination leads to deleting other Trigger's 2023-12-12 09:45:19 +01:00
Vern Paxson
ead4b681aa bug fix for delayed logging 2023-12-12 09:45:19 +01:00
Benjamin Bannier
84b2e493a1 Merge branch 'topic/bbannier/bump-spicy' 2023-12-11 13:11:33 +01:00
Benjamin Bannier
3ddbd87115 Bump auxil/spicy to latest development snapshot 2023-12-11 12:11:11 +01:00
zeek-bot
9e9c2e2f32 Update doc submodule [nomail] [skip ci] 2023-12-09 00:46:12 +00:00
Tim Wojtulewicz
fd254dece5 Merge remote-tracking branch 'origin/topic/bbannier/bump-spicy'
* origin/topic/bbannier/bump-spicy:
  Audit creation of temporary strings in Spicy bindings
  Bump auxil/spicy to latest development snapshot
2023-12-08 11:54:14 -07:00
Tim Wojtulewicz
9e8a738a8a Merge remote-tracking branch 'origin/topic/vern/CSE-opt'
* origin/topic/vern/CSE-opt:
  incorporate latest version of gen-zam to correctly generate indirect calls
  added sub-directory for tracking ZAM maintenance issues
  BTest to stress-test AST optimizer's assessment of side effects
  reworked AST optimizers analysis of side effects during aggregate operations & calls
  script optimization support for tracking information associated with BiFs/functions
  fix for AST analysis of inlined functions
  improved AST optimizer's analysis of variable usage in inlined functions
  new method for Stmt nodes to report whether they could execute a "return"
  bug fixes for indirect function calls when using ZAM
  minor fixes for script optimization, exporting of attr_name, script layout tweak
2023-12-08 10:01:23 -07:00
Benjamin Bannier
dd320f7e9f Audit creation of temporary strings in Spicy bindings 2023-12-08 16:50:33 +01:00