Mirror/zeek
1
0
Fork 0
mirror of https://github.com/zeek/zeek.git synced 2025-10-02 06:38:20 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 1

logging: Introduce Log::delay() and Log::delay_finish()

Browse source 
This is a verbose, opinionated and fairly restrictive version of the log delay idea.
Main drivers are explicitly, foot-gun-avoidance and implementation simplicity.

Calling the new Log::delay() function is only allowed within the execution
of a Log::log_stream_policy() hook for the currently active log write.

Conceptually, the delay is placed between the execution of the global stream
policy hook and the individual filter policy hooks. A post delay callback
can be registered with every Log::delay() invocation. Post delay callbacks
can (1) modify a log record as they see fit, (2) veto the forwarding of the
log record to the log filters and (3) extend the delay duration by calling
Log::delay() again. The last point allows to delay a record by an indefinite
amount of time, rather than a fixed maximum amount. This should be rare and
is therefore explicit.

Log::delay() increases an internal reference count and returns an opaque
token value to be passed to Log::delay_finish() to release a delay reference.
Once all references are released, the record is forwarded to all filters
attached to a stream when the delay completes.

This functionality separates Log::log_stream_policy() and individual filter
policy hooks. One consequence is that a common use-case of filter policy hooks,
removing unproductive log records, may run after a record was delayed. Users
can lift their filtering logic to the stream level (or replicate the condition
before the delay decision). The main motivation here is that deciding on a
stream-level delay in per-filter hooks is too late. Attaching multiple filters
to a stream can additionally result in hard to understand behavior.

On the flip side, filter policy hooks are guaranteed to run after the delay
and can be used for further mangling or filtering of a delayed record.
This commit is contained in:
Arne Welzel 2023-11-12 16:03:23 +01:00
parent dc552e647f
commit f0e67022fd
105 changed files with 3505 additions and 86 deletions
Download patch file Download diff file Expand all files Collapse all files

1
testing/btest/Baseline/scripts.base.frameworks.logging.delay.queue-test-4/.stderr Normal file
View file

@ -0,0 +1 @@
### BTest baseline data generated by btest-diff. Do not edit. Use "btest -U/-u" to update. Requires BTest >= 0.63.

58
testing/btest/Baseline/scripts.base.frameworks.logging.delay.queue-test-4/.stdout Normal file
View file

@ -0,0 +1,58 @@
### BTest baseline data generated by btest-diff. Do not edit. Use "btest -U/-u" to update. Requires BTest >= 0.63.
1362692526.869344, new_packet, 1
1362692526.869344, log_stream_policy, LOG, [ts=1362692526.869344, post_ts=<uninitialized>, write_ts=<uninitialized>, msg=packet number 1]
1362692526.939084, new_packet, 2
1362692526.939084, log_stream_policy, LOG, [ts=1362692526.939084, post_ts=<uninitialized>, write_ts=<uninitialized>, msg=packet number 2]
1362692526.939378, new_packet, 3
1362692526.939378, log_stream_policy, LOG, [ts=1362692526.939378, post_ts=<uninitialized>, write_ts=<uninitialized>, msg=packet number 3]
1362692526.939527, new_packet, 4
1362692526.939527, log_stream_policy, LOG, [ts=1362692526.939527, post_ts=<uninitialized>, write_ts=<uninitialized>, msg=packet number 4]
1362692527.008509, new_packet, 5
1362692527.008509, log_stream_policy, LOG, [ts=1362692527.008509, post_ts=<uninitialized>, write_ts=<uninitialized>, msg=packet number 5]
1362692527.009512, new_packet, 6
1362692527.009512, log_stream_policy, LOG, [ts=1362692527.009512, post_ts=<uninitialized>, write_ts=<uninitialized>, msg=packet number 6]
1362692527.009512, post_delay_cb, [ts=1362692526.869344, post_ts=1362692527.009512, write_ts=<uninitialized>, msg=packet number 1], 140.0 msecs 167.951584 usecs
1362692527.009512, log_policy
1362692527.009721, new_packet, 7
1362692527.009721, log_stream_policy, LOG, [ts=1362692527.009721, post_ts=<uninitialized>, write_ts=<uninitialized>, msg=packet number 7]
1362692527.009721, post_delay_cb, [ts=1362692526.939084, post_ts=1362692527.009721, write_ts=<uninitialized>, msg=packet number 2], 70.0 msecs 636.987686 usecs
1362692527.009721, log_policy
1362692527.009765, new_packet, 8
1362692527.009765, log_stream_policy, LOG, [ts=1362692527.009765, post_ts=<uninitialized>, write_ts=<uninitialized>, msg=packet number 8]
1362692527.009765, post_delay_cb, [ts=1362692526.939378, post_ts=1362692527.009765, write_ts=<uninitialized>, msg=packet number 3], 70.0 msecs 386.886597 usecs
1362692527.009765, log_policy
1362692527.009775, new_packet, 9
1362692527.009775, log_stream_policy, LOG, [ts=1362692527.009775, post_ts=<uninitialized>, write_ts=<uninitialized>, msg=packet number 9]
1362692527.009775, post_delay_cb, [ts=1362692526.939527, post_ts=1362692527.009775, write_ts=<uninitialized>, msg=packet number 4], 70.0 msecs 247.888565 usecs
1362692527.009775, log_policy
1362692527.009855, new_packet, 10
1362692527.009855, log_stream_policy, LOG, [ts=1362692527.009855, post_ts=<uninitialized>, write_ts=<uninitialized>, msg=packet number 10]
1362692527.009855, post_delay_cb, [ts=1362692527.008509, post_ts=1362692527.009855, write_ts=<uninitialized>, msg=packet number 5], 1.0 msec 346.111298 usecs
1362692527.009855, log_policy
1362692527.009887, new_packet, 11
1362692527.009887, log_stream_policy, LOG, [ts=1362692527.009887, post_ts=<uninitialized>, write_ts=<uninitialized>, msg=packet number 11]
1362692527.009887, post_delay_cb, [ts=1362692527.009512, post_ts=1362692527.009887, write_ts=<uninitialized>, msg=packet number 6], 375.032425 usecs
1362692527.009887, log_policy
1362692527.011846, new_packet, 12
1362692527.011846, log_stream_policy, LOG, [ts=1362692527.011846, post_ts=<uninitialized>, write_ts=<uninitialized>, msg=packet number 12]
1362692527.011846, post_delay_cb, [ts=1362692527.009721, post_ts=1362692527.011846, write_ts=<uninitialized>, msg=packet number 7], 2.0 msecs 125.024796 usecs
1362692527.011846, log_policy
1362692527.080828, new_packet, 13
1362692527.080828, log_stream_policy, LOG, [ts=1362692527.080828, post_ts=<uninitialized>, write_ts=<uninitialized>, msg=packet number 13]
1362692527.080828, post_delay_cb, [ts=1362692527.009765, post_ts=1362692527.080828, write_ts=<uninitialized>, msg=packet number 8], 71.0 msecs 63.041687 usecs
1362692527.080828, log_policy
1362692527.080972, new_packet, 14
1362692527.080972, log_stream_policy, LOG, [ts=1362692527.080972, post_ts=<uninitialized>, write_ts=<uninitialized>, msg=packet number 14]
1362692527.080972, post_delay_cb, [ts=1362692527.009775, post_ts=1362692527.080972, write_ts=<uninitialized>, msg=packet number 9], 71.0 msecs 197.032928 usecs
1362692527.080972, log_policy
1362692527.080972, Pcap::file_done, <...>/get.trace
1362692527.080972, post_delay_cb, [ts=1362692527.009855, post_ts=1362692527.080972, write_ts=<uninitialized>, msg=packet number 10], 71.0 msecs 116.924286 usecs
1362692527.080972, log_policy
1362692527.080972, post_delay_cb, [ts=1362692527.009887, post_ts=1362692527.080972, write_ts=<uninitialized>, msg=packet number 11], 71.0 msecs 84.976196 usecs
1362692527.080972, log_policy
1362692527.080972, post_delay_cb, [ts=1362692527.011846, post_ts=1362692527.080972, write_ts=<uninitialized>, msg=packet number 12], 69.0 msecs 125.890732 usecs
1362692527.080972, log_policy
1362692527.080972, post_delay_cb, [ts=1362692527.080828, post_ts=1362692527.080972, write_ts=<uninitialized>, msg=packet number 13], 144.004822 usecs
1362692527.080972, log_policy
1362692527.080972, post_delay_cb, [ts=1362692527.080972, post_ts=1362692527.080972, write_ts=<uninitialized>, msg=packet number 14], 0 secs
1362692527.080972, log_policy

16
testing/btest/Baseline/scripts.base.frameworks.logging.delay.queue-test-4/test.cut Normal file
View file

@ -0,0 +1,16 @@
### BTest baseline data generated by btest-diff. Do not edit. Use "btest -U/-u" to update. Requires BTest >= 0.63.
ts|post_ts|write_ts|msg
1362692526.869344|1362692527.009512|1362692527.009512|packet number 1
1362692526.939084|1362692527.009721|1362692527.009721|packet number 2
1362692526.939378|1362692527.009765|1362692527.009765|packet number 3
1362692526.939527|1362692527.009775|1362692527.009775|packet number 4
1362692527.008509|1362692527.009855|1362692527.009855|packet number 5
1362692527.009512|1362692527.009887|1362692527.009887|packet number 6
1362692527.009721|1362692527.011846|1362692527.011846|packet number 7
1362692527.009765|1362692527.080828|1362692527.080828|packet number 8
1362692527.009775|1362692527.080972|1362692527.080972|packet number 9
1362692527.009855|1362692527.080972|1362692527.080972|packet number 10
1362692527.009887|1362692527.080972|1362692527.080972|packet number 11
1362692527.011846|1362692527.080972|1362692527.080972|packet number 12
1362692527.080828|1362692527.080972|1362692527.080972|packet number 13
1362692527.080972|1362692527.080972|1362692527.080972|packet number 14