zeek/scripts/base/frameworks
Christian Kreibich d5bbf05a32 Add peer buffer update tracking to the Broker manager's event_observer
This implements basic tracking of each peering's current fill level, the maximum
level over a recent time interval (via a new Broker::buffer_stats_reset_interval
tunable, defaulting to 1min), and the number of times a buffer overflows. For
the disconnect policy this is the number of depeerings, but for drop_newest and
drop_oldest it implies the number of messages lost.

This doesn't use "proper" telemetry metrics for a few reasons: this tracking is
Broker-specific, so we need to track each peering via endpoint_ids, while we
want the metrics to use Cluster node name labels, and the latter live in the
script layer. Using broker::endpoint_id directly as keys also means we rely on
their ability to hash in STL containers, which should be fast.

This does not track the buffer levels for Broker "clients" (as opposed to
"peers"), i.e. WebSockets, since we currently don't have a way to name these,
and we don't want to use ephemeral Broker IDs in their telemetry.

To make the stats accessible to the script layer the Broker manager (via a new
helper class that lives in the event_observer) maintains a TableVal mapping
Broker IDs to a new BrokerPeeringStats record. The table's members get updated
every time that table is requested. This minimizes new val instantiation and
allows the script layer to customize the BrokerPeeringStats record by redefing,
updating fields, etc. Since we can't use Zeek vals outside the main thread, this
requires some care so all table updates happen only in the Zeek-side table
updater, PeerBufferState::GetPeeringStatsTable().

(cherry picked from commit f5fbad23ff)
2025-04-29 15:08:05 -07:00
..
analyzer Add logging of disabled analyzers to analyzer.log 2024-07-09 18:22:43 +02:00
broker Add peer buffer update tracking to the Broker manager's event_observer 2025-04-29 15:08:05 -07:00
cluster Add backpressure disconnect notification to cluster.log and via telemetry 2025-04-08 15:09:44 -07:00
config Revert "Merge remote-tracking branch 'origin/topic/vern/at-if-analyze'" 2023-05-31 09:20:33 +02:00
control annotate base scripts with &is_used as needed 2022-05-26 17:39:17 -07:00
files Fix cid propagation into files.log 2024-04-29 14:13:19 +01:00
input More bro-to-zeek renaming in scripts and other files 2019-05-16 02:36:41 -05:00
intel Fix markup typo in the Intel::seen_policy hook docstring [skip ci] 2024-02-02 12:30:39 -08:00
logging logging: Do not keep delay state persistent 2023-11-29 11:53:11 +01:00
netcontrol Netcontrol: add rule_added_policy 2024-02-05 18:52:27 +00:00
notice http: Prevent request/response de-synchronization and unbounded state growth 2023-08-28 15:02:58 +02:00
openflow Revert "Merge remote-tracking branch 'origin/topic/vern/at-if-analyze'" 2023-05-31 09:20:33 +02:00
packet-filter Add PacketFilter::remove_exclude function 2024-04-17 21:25:35 +00:00
reporter Support for log filter policy hooks 2020-09-30 12:32:45 -07:00
signatures allow signature actions to be dynamically updated 2023-07-13 17:25:32 -07:00
software frameworks/software: Fix stale value used for stripping 2023-11-17 14:37:28 +01:00
spicy Merge remote-tracking branch 'origin/topic/robin/gh-3881-spicy-ports' 2024-08-30 13:26:16 -07:00
sumstats Remove script functions marked as unused (6.1 deprecations) 2023-06-14 10:07:22 -07:00
supervisor Management framework: add metrics port in management & Supervisor node records 2024-07-08 23:05:24 -07:00
telemetry Telemetry framework: move BIFs to the primary-bif stage 2025-04-08 14:50:45 -07:00
tunnels Add GTPv1 packet analyzer, disable old analyzer 2021-11-23 19:36:50 -07:00