Commit graph

18601 commits

Author SHA1 Message Date
Benjamin Bannier
3ddbd87115 Bump auxil/spicy to latest development snapshot 2023-12-11 12:11:11 +01:00
zeek-bot
9e9c2e2f32 Update doc submodule [nomail] [skip ci] 2023-12-09 00:46:12 +00:00
Tim Wojtulewicz
fd254dece5 Merge remote-tracking branch 'origin/topic/bbannier/bump-spicy'
* origin/topic/bbannier/bump-spicy:
  Audit creation of temporary strings in Spicy bindings
  Bump auxil/spicy to latest development snapshot
2023-12-08 11:54:14 -07:00
Tim Wojtulewicz
9e8a738a8a Merge remote-tracking branch 'origin/topic/vern/CSE-opt'
* origin/topic/vern/CSE-opt:
  incorporate latest version of gen-zam to correctly generate indirect calls
  added sub-directory for tracking ZAM maintenance issues
  BTest to stress-test AST optimizer's assessment of side effects
  reworked AST optimizers analysis of side effects during aggregate operations & calls
  script optimization support for tracking information associated with BiFs/functions
  fix for AST analysis of inlined functions
  improved AST optimizer's analysis of variable usage in inlined functions
  new method for Stmt nodes to report whether they could execute a "return"
  bug fixes for indirect function calls when using ZAM
  minor fixes for script optimization, exporting of attr_name, script layout tweak
2023-12-08 10:01:23 -07:00
Benjamin Bannier
dd320f7e9f Audit creation of temporary strings in Spicy bindings 2023-12-08 16:50:33 +01:00
Benjamin Bannier
20310f9392 Bump auxil/spicy to latest development snapshot 2023-12-08 16:50:33 +01:00
Vern Paxson
51a343c813 incorporate latest version of gen-zam to correctly generate indirect calls 2023-12-06 12:26:46 -08:00
Vern Paxson
d43d6dd09d added sub-directory for tracking ZAM maintenance issues 2023-12-06 12:26:46 -08:00
Vern Paxson
6f0495b6ea BTest to stress-test AST optimizer's assessment of side effects 2023-12-06 12:26:46 -08:00
Vern Paxson
740a087765 reworked AST optimizers analysis of side effects during aggregate operations & calls 2023-12-06 12:26:27 -08:00
Arne Welzel
e7b8d064af Merge remote-tracking branch 'origin/topic/awelzel/no-global-libkqueue-includes'
* origin/topic/awelzel/no-global-libkqueue-includes:
  Bump cmake submodule
  iosource: Specify libkqueue dependency for subdir lib
2023-12-06 18:42:43 +01:00
Arne Welzel
8d52dbe4e5 Update zeek-aux submodule 2023-12-06 18:41:09 +01:00
Arne Welzel
a8792a16b0 Bump cmake submodule 2023-12-06 16:00:08 +01:00
Arne Welzel
4e20a484ea iosource: Specify libkqueue dependency for subdir lib
Since a while my build has been spilling the following warnings:

    [18/1687] Building C object auxil/c-ares/src/lib/CMakeFiles/c-ares.dir/ares__addrinfo2hostent.c.o
    cc1: warning: zeek/prod-build/libkqueue-build/include: No such file or directory [-Wmissing-include-dirs]

My take is that FindKqueue extends the include directories globally and
tickles this warning because c-ares is built first. Grepping around,
<sys/event.h> is only included in iosource/Manager.cc, so we should
be able to reduce the exposure just to the iosource subdir.
2023-12-06 16:00:00 +01:00
Arne Welzel
d1e7c8b81c Merge remote-tracking branch 'origin/topic/awelzel/spicy-rt-raise-event-vector-unsafe'
* origin/topic/awelzel/spicy-rt-raise-event-vector-unsafe:
  spicy/runtime-support: Use Vector::unsafeBegin() for args iteration
  Bump spicy submodule
2023-12-06 11:41:40 +01:00
Arne Welzel
8ec69d084f spicy/runtime-support: Use Vector::unsafeBegin() for args iteration 2023-12-05 16:59:29 +01:00
Arne Welzel
5b1b1c6120 Bump spicy submodule 2023-12-05 16:59:29 +01:00
Arne Welzel
f39f1b0c68 Merge remote-tracking branch 'origin/topic/awelzel/random-perf-things'
* origin/topic/awelzel/random-perf-things:
  SegmentProfiler: Do not initialize initial_rusage
  EventMgr: Remove queue_flare, use GetNextTimeout() instead
  UpdateConnVal: Avoid FieldOffset() calls
2023-12-05 16:01:15 +01:00
Arne Welzel
efc69182b1 Merge remote-tracking branch 'origin/topic/awelzel/3403-signature-match-event'
* origin/topic/awelzel/3403-signature-match-event:
  RuleActionMIME: Switch to std::string
  signatures: Support custom event via [event_name] syntax
  zeek-setup: Exit when rule loading tickles reporter errors
  rule-scan: Copy yytext strings
2023-12-05 16:00:38 +01:00
Arne Welzel
e8241e16ee RuleActionMIME: Switch to std::string
And return const std::string& from GetMIME(). Probably not at all performance
relevant, but while I'm already here.
2023-12-05 15:28:24 +01:00
Arne Welzel
a7b077aa17 signatures: Support custom event via [event_name] syntax
This change allows to specify a per signature specific event, overriding
the default signature_match event. It further removes the message
parameter from such events if not provided in the signature.

This also tracks the message as StringValPtr directly to avoid
allocating the same StringVal for every DoAction() call.

Closes #3403
2023-12-05 15:28:21 +01:00
Arne Welzel
d11ac929af zeek-setup: Exit when rule loading tickles reporter errors
With custom events for signatures, Reporter::error() may be invoked
while loading them. Early exit in case that happens. We could continue
and either disable the signatures or fallback to the default
signature_match() event, but not sure that would be obviously better.
2023-12-05 15:26:40 +01:00
Arne Welzel
b795c0bfc1 rule-scan: Copy yytext strings
When trying to use TOK_IDENT and TOK_STRING in a single rule, that
resulted in "corrupt" strings.

https://www.gnu.org/software/bison/manual/html_node/Strings-are-Destroyed.html
2023-12-05 15:26:40 +01:00
Vern Paxson
c028901146 script optimization support for tracking information associated with BiFs/functions 2023-12-04 16:53:57 -08:00
Vern Paxson
ce09a11a0c fix for AST analysis of inlined functions 2023-12-04 16:53:32 -08:00
Vern Paxson
d0eb40ac92 improved AST optimizer's analysis of variable usage in inlined functions 2023-12-04 16:53:32 -08:00
Vern Paxson
087eb1ca4e new method for Stmt nodes to report whether they could execute a "return" 2023-12-04 16:53:32 -08:00
Vern Paxson
3d69b0551a bug fixes for indirect function calls when using ZAM 2023-12-04 16:50:27 -08:00
Vern Paxson
a927d61e59 minor fixes for script optimization, exporting of attr_name, script layout tweak 2023-12-04 16:48:19 -08:00
Christian Kreibich
0aef842f05 Merge branch 'topic/neverlord/broker-data'
* topic/neverlord/broker-data:
  Integrate review feedback
  Add facade types to avoid using raw Broker types
2023-12-04 12:32:35 -08:00
Arne Welzel
d70b3d6506 SegmentProfiler: Do not initialize initial_rusage
We use the SegmentProfiler in quite a few hot places and the memset of
the rusage structure (144bytes here) can show up significantly even if
the segment profiler itself isn't used.

Relates to #3485.
2023-12-04 20:03:31 +01:00
Arne Welzel
46acd9168e EventMgr: Remove queue_flare, use GetNextTimeout() instead
It can be visible overhead to call write() on the underlying pipe of the
EventMgr's flare whenever the first event is enqueued during an IO loop
iteration. Particularly in scenarios where there's about 1 event per packet
for long lived connections and script-side event processing is fast.

Given the event manager is drained anyhow at the end of the main loop, this
shouldn't be needed. In fact, the EventMgr.Process() method is basically
a stub. The one reason it is needed is when more events are enqueued during
a drain. That, however, can be dealt with by implementing GetNextTimeout()
to return 0.0 when there's more events queued. This way the main-loop's poll
timeout is 0.0 and it'll continue immediately.

This also allows to removes some extra code and drop the recently introduced
InitPostFork() addition: Without a pipe, there's no need to recreate it.
2023-12-04 20:03:31 +01:00
Arne Welzel
01e305edd8 UpdateConnVal: Avoid FieldOffset() calls
These can be significant if a lot of new connections and or events
are created for which an existing conn val needs updating and otherwise
things are very fast.
2023-12-04 20:03:27 +01:00
Arne Welzel
c3762ba9d3 Merge remote-tracking branch 'origin/topic/awelzel/log-delay-coverity-follow-up'
* origin/topic/awelzel/log-delay-coverity-follow-up:
  logging: Fix coverity std::move suggestions
  logging/Manager: Fix coverity null-deref
2023-12-04 18:57:55 +01:00
Arne Welzel
30314dd940 logging: Fix coverity std::move suggestions 2023-12-04 18:27:57 +01:00
Arne Welzel
52fba4aacf logging/Manager: Fix coverity null-deref
Prior code assumed non-null stream given the active_write_ctx matches,
but please coverity.
2023-12-04 18:27:57 +01:00
Dominik Charousset
a69928d977 Integrate review feedback 2023-12-04 15:23:56 +01:00
Dominik Charousset
647fdf7737 Add facade types to avoid using raw Broker types
By avoiding to use `broker::data` directly, we gain a degree of freedom
that allows us to swap out `broker::data` for something else (e.g.,
`broker::variant`) in the future. Furthermore, it also helps us to keep
Broker types "local" to the Broker manager and gives us a nicer
interface.

Also replaces uses of `broker::expected` with `std::optional`. While an
`expected `can carry additional information as to why a value is not
present, nothing in Zeek ever cared about that. Hence, using
`std::optional` removes an unnecessary dependency on a Broker detail
while also being more efficient (no extra heap allocation when no value
is present).
2023-12-04 15:23:28 +01:00
Tim Wojtulewicz
bc0f85caa8 Merge remote-tracking branch 'origin/topic/timw/logging-manager-warnings'
* origin/topic/timw/logging-manager-warnings:
  Fix some compiler warnings in logging::Manager
2023-12-01 11:55:59 -07:00
Tim Wojtulewicz
4fa06cef75 Fix some compiler warnings in logging::Manager 2023-12-01 11:49:26 -07:00
Arne Welzel
0eba31a80b Merge remote-tracking branch 'origin/topic/awelzel/log-delay-ubsan-fix'
* origin/topic/awelzel/log-delay-ubsan-fix:
  logging/Manager: Fix token_val->AsCount() in debug logging
2023-12-01 19:41:21 +01:00
Arne Welzel
3c99b7ae9c logging/Manager: Fix token_val->AsCount() in debug logging
Second UBSAN error triggered from log delay merge.
2023-12-01 16:01:45 +01:00
Arne Welzel
acf4ed9c6c logging/Manager: Fix AsTime() to AsInterval()
Found by UBSAN after merge of log delay branch.
2023-12-01 13:26:40 +01:00
Arne Welzel
28eef9e4b2 Merge remote-tracking branch 'origin/topic/awelzel/log-write-delay-3'
* origin/topic/awelzel/log-write-delay-3:
  logging: ref() to record_ref() renaming
  logging: Fix typos from review
  logging/Manager: Make LogDelayExpiredTimer an implementation detail
  logging/WriteToFilters: Use range-based for loop
  testing/btest: Log::delay() from JavaScript
  NEWS: Entry for delayed log writes
  Bump doc submodule to branch
  logging: Do not keep delay state persistent
  logging: delay documentation polishing
  logging: Better error messages for invalid Log::delay() calls
  logging/Manager: Implement DelayTokenType as an actual opaque
  logging: Implement get_delay_queue_size()
  logging: Introduce Log::delay() and Log::delay_finish()
  logging/Manager: zeek::detail'ify
  logging/Manager: Split Write()
  Timer: Add LOG_DELAY_EXPIRE timer type
  Ascii: Remove extra include
2023-12-01 12:05:02 +01:00
Arne Welzel
f61fa6c24f logging: ref() to record_ref() renaming 2023-11-30 12:32:12 +01:00
Arne Welzel
9956d96824 logging: Fix typos from review 2023-11-30 12:26:08 +01:00
Arne Welzel
ee65623600 logging/Manager: Make LogDelayExpiredTimer an implementation detail
The only reason this was a private component of Manager was to access
the Stream's function. Use a generic callback and a lambda to avoid
that exposure.
2023-11-30 12:25:49 +01:00
Arne Welzel
dfa8bac273 logging/WriteToFilters: Use range-based for loop 2023-11-30 11:37:10 +01:00
Arne Welzel
a59578e018 Merge branch 'topic/awelzel/3473-register-telemetry-opaque-vals'
* topic/awelzel/3473-register-telemetry-opaque-vals:
  OpaqueVal: Register TelemetryVals, too
2023-11-30 10:19:44 +01:00
Arne Welzel
0a3d98973f testing/btest: Log::delay() from JavaScript 2023-11-29 12:52:19 +01:00