This commit revamps the handling of analyzer violations that happen
before an analyzer confirms the protocol.
The current state is that an analyzer is disabled after 5 violations, if
it has not been confirmed. If it has been confirmed, it is disabled
after a single violation.
The reason for this is a historic mistake. In Zeek up to versions 1.5,
analyzers were unconditianally removed when they raised the first
protocol violation.
When this script was ported to the new layout for Zeek 2.0 in
b4b990cfb5, a logic error was introduced
that caused analyzers to no longer be disabled if they were not
confirmed.
This was the state for ~8 years, till the DPD::max_violations options
was added, which instates the current approach of disabling unconfirmed
analyzers after 5 violations. Sadly, there is not much discussion about
this change - from my hazy memory, I think this was discovered during
performance tests and the new behavior was added without checking into
the history of previous changes.
This commit reinstates the originally intended behavior of DPD. When an
analyzer that has not been confirmed raises a protocol violation, it is
immediately removed from the connection. This also makes a lot of sense
- this allows the analyzer to be in a "tasting" phase at the beginning
of the connection, and to error out quickly once it realizes that it was
attached to a connection not containing the desired protocol.
This change also removes the DPD::max_violations option, as it no longer
serves any purpose after this change. (In practice, the option remains
with an &deprecated warning, but it is no longer used for anything).
There are relatively minimal test-baseline changes due to this; they are
mostly triggered by the removal of the data structure and by less
analyzer errors being thrown, as unconfirmed analyzers are disabled
after the first error.
This switches the DPD logic to always log analyzers that raised a
protocol confirmation.
The logic is that, once a protocol has been confirmed - and thus there
probably is log output - it does not make sense to later remove it from
the log. It does make sense to somehow flag it as failed - but that
seems like a secondary step.
* origin/topic/johanna/gh-4061:
Update BiF-tracking, add is_event_handled
Address review comments and small updates for DNS warnings
Raise warnings when for DNS events that are not raised due to dns_skip_all_addl
There's two instances of WriterBackend::WriterInfo for a given
writer. One in Manager::WriterInfo that's accessible via
stream.writers and a copy within WriterFrontend.
Commit 78999d147d switched to use the
address of the frontend's info instance for HookLogWrite() invocations,
breaking users using the address for identification purposes.
By default, dns_skip_all_addl is set to false. This causes several
events to not be raised. This change emits warnings when a user defines
event handlers for events that will not be raised.
Furthermore, it adds notes about this behavior to the documentation. We
also introduce a new BIF, `is_event_handled`, which checks if an event
is handled.
Fixes GH-4061
* origin/topic/vern/ZAM-field-assign-in-op:
pre-commit: Bump spicy-format to 0.23
fix for ZAM optimization of assigning a record field to result of "in" operation
The fatal error is actually triggered at runtime, so it's a bit
dangerous for users, but not sure there's many use-cases to
compare vectors of patterns.
Closes#4084
* origin/topic/vern/ZAM-assert-streamlining:
btest/supervisor: Bump one timeout to 30 seconds
btest/cluster: Bump timeouts to 30 seconds
btest/validate-ZAM: Update baseline, run always
* origin/topic/vern/CPP-standalone-fixes:
BTest updates in support of compiling-scripts-to-C++
-O gen-standalone-C++ fixes for recent more aggressive AST profiling
* origin/topic/awelzel/fix-cluster-publish-any:
cluster/Backend: Handle unspecified table/set
cluster: Fix Cluster::publish() of Broker::Data
cluster: Be noisy when attempting to connect to an unknown node
Same as what we do in Broker. Use the expected type if publishing
a table() or set() parameter.
This fixes issues when switching sumstats to Cluster::publish()
The broker serializer leverages the existing data_to_val() function.
During unserialization, if the destination type is any, the logic
simply wraps the broker::data value into a Broker::Data record.
Therefore, events with any parameters are currently exposed to
the Broker::Data type.
There is a bigger issue in that re-publishing such Broker::Data
instances would encode them as a normal record. Explicitly prevent
this by serializing the contained data value directly instead, similar
to what Broker already did when publishing a record.
Mostly due to spending too much time wondering why nodes didn't connect
when there was a mismatch between "manager" and "manager-1" in the
cluster layout. Remove manager from test-all-policy-cluster test to
avoid connection attempts in this test.
ZAM's implementation deviates from Zeek's in consideration of remappings, which
means that running ZAM produced conn.log service names prefixed with "spicy_"
whereas regular Zeek doesn't.
...at the same time, add some `TEST-REQUIRES: have-zeromq` which
unfortunately means that developers will usually want libzmq
installed on their system.
This is a cluster backend implementation using a central XPUB/XSUB proxy
that by default runs on the manager node. Logging is implemented leveraging
PUSH/PULL sockets between logger and other nodes, rather than going
through XPUB/XSUB.
The test-all-policy-cluster baseline changed: Previously, Broker::peer()
would be called from setup-connections.zeek, causing the IO loop to be
alive. With the ZeroMQ backend, the IO loop is only alive when
Cluster::init() is called, but that doesn't happen anymore.
Also, run the ZAM-bif-tracking test in non-ZAM environments so
failures are caught immediately. There's nothing overly ZAM specific about
running this test.
I'm not sure I like the fact that any new contributor adding a BiF
will need to dig into this... it might be a bit intimidating.
