Do not just track the analyzer instance in the stack, but also the
data span it is given. This allows to extract more information on-demand
during event processing.
TrackAnalyzer() is technically a public API, but no one should use it
outside of the Analyzer's Forward methods itself.
* origin/topic/bbannier/integer-conversion-error-handling:
Also trim trailing spaces in `to_count`/`to_int` inputs
Align error handling in `to_int` with existing behavior of `to_count`
Baseline handling of leading/trailing spaces in `to_count`/`to_int`.
Add error messages to `to_count`/`to_int` baselines
Previously we would already trim leading spaces in inputs to `to_count`
and `to_int`, effectively by just passing the behavior of the low-level
functions used in their implementations to the user. While this was
useful it was also inconsistent in that we did not allow trailing
spaces which we enable with this patch.
Previously `to_int` would silently ignore invalid inputs and simply
return `0` while `to_count` would return an error; this patch changes
`to_int` to behave like `to_count`.
This introduces a breaking change in that `to_int` now raises an error
for trailing spaces (but still accepts leading spaces) where it
previously would have silently accepted it. This is consistent with
the behavior of `to_count`, but one could also argue that both of
these should only accept properly trimmed input; I did not go that route
since that would introduce breaking changes for both these functions
instead of for just one of them.
Currently `to_count` reports an error for trailing spaces (but not for
leading ones) while `to_int` silently accepts them. This patch adds
baselines capture the current behavior.
CI is currently failing with:
{ "error": { "code": 404, "message": "The resource 'projects/freebsd-org-cloud-dev/global/images/family/freebsd-14-1' was not found" ...
* origin/topic/johanna/dpd-changes:
DPD: failed services logging alignment
DPD: update test baselines; change options for external tests.
DPD: change policy script for service violation logging; add NEWS
DPD changes - small script fixes and renames.
Update public and private test suite for DPD changes.
Allow to track service violations in conn.log.
Make conn.log service field ordered
DPD: change handling of pre-confirmation violations, remove max_violations
DPD: log analyzers that have confirmed
IRC analyzer - make protocol confirmation more robust.
There were some special cases in which the failed-service-logging policy
script might log a service being removed that was not removed due to an
analyzer violation. This change should fix these cases.
As services are sorted by default now, this disables the canonifier that
sorts the service field for the external baseline.
This also adds the tracking of disabled services in the service field
via DPD::track_removed_services_in_connection - to make such changes
more visible, and check that the feature works as desired.
This commit renames the `service_violation` column that can be added via
a policy script to `failed_service`. This expresses the intent of it
better - the column contains services that failed and were removed after
confirmation.
Furthermore, the script is fixed so it actually does this - before it
would sometimes add services to the list that were not actually removed.
In the course of this, the type of the column was changed from a vector
to an ordered set.
Due to the column rename, the policy script itself is also renamed.
Also adds a NEWS entry for the DPD changes.
This allows Zeek execution to continue gracefully in the presence of such
errors, particularly at zeek_init() time. Includes a tweak to expand the
bifs.directory_operations test to check continuation after errors.
Resolves#3595.
* origin/topic/awelzel/zeromq-fix-fmt-call-thanks-tsan:
cluster/zeromq: Call DoTerminate() in destructor
cluster/zeromq: Use lambda for thread trampoline
cluster/zeromq: Do not call util::fmt() from thread
Normal life-cycle is that Terminate() / DoTerminate() is called
by zeek-setup code. If that doesn't happen, shutdown and join
threads during destructor.
try { } catch (...) suggested by Benjamin.
...util::fmt() uses a static buffer, so this is problematic.
I've dabbled a bit replacing std::thread with using threading::BasicThread
which would offer Fmt(), but this makes things more complicated. Primarily
as BasicThread is registered with the thread manager and the shutdown
interactions become entangled. The thread might be terminated before the
backend, or vice-versa. Seems nicer for the thread to be owned by the backend.
* origin/topic/awelzel/4136-cluster-backend-pre-work:
cluster/zeromq: Fix Unsubscribe() bug caused by \x00 prefix
cluster: Add SubscribeCallback support
cluster/zeromq: Fix XSUB threading issues
cluster/zeromq: Use NodeId(), drop my_node_id
cluster/Backend: Pass node_id via Init()
cluster/Backend: Make backend event processing customizable
cluster/broker/Serializer: Fix adaptor to adapter
cluster/Backend: Do not use const std::string_view&
cluster/serializer/broker: Fix handler lookup
broker/Manager: Move name in PublishEvent()
btest/zeromq/test-bootstrap: Fix port parsing
EventHandler: Support operator!=
This allows callers of Subscribe() to pass in a callback that will be invoked
once the subscription is established or failed to establish. It is the
backend's responsibility to execute the callback on the main thread either
synchronously, or preferably asynchronously at a later point, by
scheduling a task on the IO main loop.
This turns on ZMQ_XPUB_VERBOSE for ZeroMQ so that notifications about
subscriptions are raised even if the subscriptions has previously been
observed.
It is not safe to use the same socket from different threads, but the
current code used the xsub socket directly from the main thread (to setup
subscriptions) and from the internal thread for polling and reading.
Leverage the PAIR socket already in use for forwarding publish operations
to the internal thread also for subscribe and unsubscribe.
The failure mode is/was a bit annoying. Essentially, closing of the
context would hang indefinitely in zmq_ctx_term().
This allows configurability at the code level to decide what to do with
a received remote events and events produced by a backend. For now, only
enqueue events into the process's script layer, but for the WebSocket
interface, the action would be to send out the event on a WebSocket
connection instead.
