The main part of this commit are changes in tests. A lot of the tests
that previously relied on analyzer.log or dpd.log now use the new
analyzer-failed.log.
I verified all the changes and, as far as I can tell, everything
behaves as it should. This includes the external test baselines.
This change also enables logging of file and packet analyzer to
analyzer_failed.log and fixes some small behavior issues.
The analyzer_failed event is no longer raised when the removal of an
analyzer is vetoed.
If an analyzer is no longer active when an analyzer violation is raised,
currently the analyzer_failed event is raised. This can, e.g., happen
when an analyzer error happens at the very end of the connection. This
makes the behavior more similar to what happened in the past, and also
intuitively seems to make sense.
A bug introduced in the failed service logging was fixed.
As services are sorted by default now, this disables the canonifier that
sorts the service field for the external baseline.
This also adds the tracking of disabled services in the service field
via DPD::track_removed_services_in_connection - to make such changes
more visible, and check that the feature works as desired.
This commit renames the `service_violation` column that can be added via
a policy script to `failed_service`. This expresses the intent of it
better - the column contains services that failed and were removed after
confirmation.
Furthermore, the script is fixed so it actually does this - before it
would sometimes add services to the list that were not actually removed.
In the course of this, the type of the column was changed from a vector
to an ordered set.
Due to the column rename, the policy script itself is also renamed.
Also adds a NEWS entry for the DPD changes.
The changes are mostly quite minor. The main change reasons are:
* analyzers that were confirmed, and later removed now show up in the
conn.log.
* a couple of removed lines in analyzer.log, because non-confirmed
analyzers get removed more quickly.
* in some cases there are additional lines in analyzer.log. These are
cases in which an analyzer gets removed due to a violation and then
re-attached because of a later signature match, which replays the
violating content. In all examples that I have so far, this is caused
by both sides of a connection speaking a differing protocol. There
probably should be a better way to handle this - but it works.
* new column for failed analyzers in conn.log
The diffs produced by telemetry.log when introducing a weird or
removing/adding protocol specific logs is overwhelming and distracting
without providing value. Exclude telemetry.log similar to how we already
exclude stats.log.
Some more targeted telemetry.log tests exists in the normal testing/btest
suite and that appears more sensible.
For traces spanning a long time period, don't produce immense
amounts of logs. Also filter down the event handler invocation
entries to connection related events in telemetry.log.
By default this only logs all the violations, regardless of the
confirmation state (for which there's still dpd.log). It includes
packet, protocol and file analyzers.
This uses options, change handlers and event groups for toggling
the functionality at runtime.
Closes#2031
Adds base/frameworks/telemetry with wrappers around telemetry.bif
and updates telemetry/Manager to support collecting metrics from
script land.
Add policy/frameworks/telemetry/log for logging of metrics data
into a new telemetry.log and telemetry_histogram.log and add into
local.zeek by default.
This provides "make sync-repos" to check out all locally available testsuites at
the commits indicated in their commit files, and "make sync-commits" to update
the commit files to the HEADs of the local testsuite repos.
Also adds the commit -> repo sync for the Makefile init target so initialization
always lands on the right version, and removes the corresponding explicit
checkout from the CI repo setup.
All changes in this patch were performed automatically with `shfmt` with
configuration flags specified in `.pre-commit-config.yaml`.
In addition to fixing whitespace the roundtrip through shfmt's AST also
transforms command substitutions
`cmd`
# becomes
$(cmd)
and some redirects
>&2 echo "msg"
# becomes
echo >&2 "msg"
Delete the unused Baseline folder, remove a Bro-era name from gitignore,
change create-new-repo to work with a second argument that is actually
a URL, and tweak whitespace. Expand description in README and update the
explanation of OPENSSL_ENABLE_MD5_VERIFY in subdir-btest.cfg.
For trace files that require authentication to download, hide
part of the URL in output messages. This avoids leaking potentially
sensitive info when running tests using a continuous integration
service.
This also patches a few tests to contain certificates that were removed.
Furthermore, we include the old CA file with the external tests and load
it automatically. Those traces are kind of old now, more and more of the
CAs in them are no longer valid and it does not really make sense to
update them on each change...
* origin/topic/seth/json-formatter:
Updating a couple of tests.
Expanded support for modifying the timestamp format in the JSON formatter.
Ascii input reader now supports all config options per-input stream.
Added an option to the JSON formatter to use ISO 8601 for timestamps.
Refactored formatters and updated the the writers a bit.
Includes some minor bugfixes and cleanup at various places, including
in old code.
* origin/topic/script-reference: (50 commits)
A few updates for the FAQ.
Fixing some doc warnings.
Forgot to add protocol identifier support for TLS 1.2
Finished SSL & syslog autodocs.
Adding the draft SSL extension type next_protocol_negotiation.
Fix some documentation errors.
Tweaks.
A set of script-reference polishing.
fixed a couple typos in comments
Add summary documentation to bif files.
Add ssl and syslog script documentation
Add Conn and DNS protocol script documentation. (fixes#731)
Small updates to the default local.bro.
Documentation updates for HTTP & IRC scripts.
SSH&FTP Documentation updates.
Fixing a warning from the documentation generation.
This completes framework documentation package 4.
Minor notice documentation tweaks.
Fix some malformed Broxygen xref roles.
Minor doc tweaks to init-bare.bro.
...
Conflicts:
aux/broccoli
aux/broctl
src/bro.bif
src/strings.bif
Includes:
- Updated baselines for autodoc tests.
- Now excluding stats.bro from external texts, it's not stable.
- Skip diffing of debug.log always.
- Skip diffing of reporter.log if it only contains an error about
missing GeoIP support.
- Canonicalize X.509 Distinguished Name subjects since that can vary
depending on installed OpenSSL version.
- The absolute path canonifier was overzealously canonifying relevant
log fields, so it's no longer generally applied to diffing all
baselines. I don't think there's any logs that require local
filesystem path names that aren't already tested by a unit test,
but if any show up in the future, they can be canonified on a
case-by-case basis.
- Removed some logs from being diff'd in the diff-all script
because they're either already covered by a unit test
(load_scripts.log) or because of difficulty/maintainenance
tradeoff (prof.log).
Baselines for the external bro-testing repo still need updating.
entities.
Before, whether they did depended on libmagic. To do that,
smpt/entities.bro gets a new option `never_calc_md5`.
Also restructuring the tests a bit so that load a common
testing-setup.bro scripts that can set a global configuration.
The traces for external test-suites are no longer kept inside the
repositories themselves but downloaded separately via curl. This is
because git is pretty bad at dealing with large files.
See the README for more information.
- The Makefiles now run btest with "-f diag.log" so that diangnostics
output will always be written into that file.
- Makefiles now hardcode path to btest to avoid picking up the wrong version
if in PATH.
- The canonifier scripts now live in testing/scripts, and they are
used from both btest/ and external/.
- There's a new diff-remove-uids scripts that removed connection UIDs
for diffing. The external/* tests now use that by default.
- Timestamp removal now has its own script: diff-remove-timestamps.
diff-canonifier calls that.
- All Makefile have a "brief" target that runs btest with -b.
- The higher-level directories have Makefile to call the subdirs.
traces and more complex Bro configurations.
That's "Type 2" in the terminology of
http://www-new.bro-ids.org/devel/projects/testing.html
This leverages btest as well and will replace the set of scripts that
the old private test-suite is using. It can accomodate tests with both
public and private traces.
This is a checkpoint commmit, not much there yet in terms of actual
testing. Also still need to figure out a mechanmism to allow for
different baselines based on which branch we're testing.
