The main part of this commit are changes in tests. A lot of the tests
that previously relied on analyzer.log or dpd.log now use the new
analyzer-failed.log.
I verified all the changes and, as far as I can tell, everything
behaves as it should. This includes the external test baselines.
This change also enables logging of file and packet analyzer to
analyzer_failed.log and fixes some small behavior issues.
The analyzer_failed event is no longer raised when the removal of an
analyzer is vetoed.
If an analyzer is no longer active when an analyzer violation is raised,
currently the analyzer_failed event is raised. This can, e.g., happen
when an analyzer error happens at the very end of the connection. This
makes the behavior more similar to what happened in the past, and also
intuitively seems to make sense.
A bug introduced in the failed service logging was fixed.
As services are sorted by default now, this disables the canonifier that
sorts the service field for the external baseline.
This also adds the tracking of disabled services in the service field
via DPD::track_removed_services_in_connection - to make such changes
more visible, and check that the feature works as desired.
This commit renames the `service_violation` column that can be added via
a policy script to `failed_service`. This expresses the intent of it
better - the column contains services that failed and were removed after
confirmation.
Furthermore, the script is fixed so it actually does this - before it
would sometimes add services to the list that were not actually removed.
In the course of this, the type of the column was changed from a vector
to an ordered set.
Due to the column rename, the policy script itself is also renamed.
Also adds a NEWS entry for the DPD changes.
The changes are mostly quite minor. The main change reasons are:
* analyzers that were confirmed, and later removed now show up in the
conn.log.
* a couple of removed lines in analyzer.log, because non-confirmed
analyzers get removed more quickly.
* in some cases there are additional lines in analyzer.log. These are
cases in which an analyzer gets removed due to a violation and then
re-attached because of a later signature match, which replays the
violating content. In all examples that I have so far, this is caused
by both sides of a connection speaking a differing protocol. There
probably should be a better way to handle this - but it works.
* new column for failed analyzers in conn.log
The diffs produced by telemetry.log when introducing a weird or
removing/adding protocol specific logs is overwhelming and distracting
without providing value. Exclude telemetry.log similar to how we already
exclude stats.log.
Some more targeted telemetry.log tests exists in the normal testing/btest
suite and that appears more sensible.
For traces spanning a long time period, don't produce immense
amounts of logs. Also filter down the event handler invocation
entries to connection related events in telemetry.log.
By default this only logs all the violations, regardless of the
confirmation state (for which there's still dpd.log). It includes
packet, protocol and file analyzers.
This uses options, change handlers and event groups for toggling
the functionality at runtime.
Closes#2031
Adds base/frameworks/telemetry with wrappers around telemetry.bif
and updates telemetry/Manager to support collecting metrics from
script land.
Add policy/frameworks/telemetry/log for logging of metrics data
into a new telemetry.log and telemetry_histogram.log and add into
local.zeek by default.
