zeek/testing/btest
Johanna Amann c72c1cba6f DPD: change handling of pre-confirmation violations, remove max_violations
This commit revamps the handling of analyzer violations that happen
before an analyzer confirms the protocol.

The current state is that an analyzer is disabled after 5 violations, if
it has not been confirmed. If it has been confirmed, it is disabled
after a single violation.

The reason for this is a historic mistake. In Zeek up to versions 1.5,
analyzers were unconditianally removed when they raised the first
protocol violation.

When this script was ported to the new layout for Zeek 2.0 in
b4b990cfb5, a logic error was introduced
that caused analyzers to no longer be disabled if they were not
confirmed.

This was the state for ~8 years, till the DPD::max_violations options
was added, which instates the current approach of disabling unconfirmed
analyzers after 5 violations. Sadly, there is not much discussion about
this change - from my hazy memory, I think this was discovered during
performance tests and the new behavior was added without checking into
the history of previous changes.

This commit reinstates the originally intended behavior of DPD. When an
analyzer that has not been confirmed raises a protocol violation, it is
immediately removed from the connection. This also makes a lot of sense
- this allows the analyzer to be in a "tasting" phase at the beginning
of the connection, and to error out quickly once it realizes that it was
attached to a connection not containing the desired protocol.

This change also removes the DPD::max_violations option, as it no longer
serves any purpose after this change. (In practice, the option remains
with an &deprecated warning, but it is no longer used for anything).

There are relatively minimal test-baseline changes due to this; they are
mostly triggered by the removal of the data structure and by less
analyzer errors being thrown, as unconfirmed analyzers are disabled
after the first error.
2025-01-30 16:59:44 +00:00
..
af_packet
Baseline DPD: change handling of pre-confirmation violations, remove max_violations 2025-01-30 16:59:44 +00:00
Baseline.cpp BTest updates in support of compiling-scripts-to-C++ 2024-12-12 14:30:23 -08:00
Baseline.zam btest/validate-ZAM: Update baseline, run always 2024-12-13 18:25:28 +01:00
bifs DPD: change handling of pre-confirmation violations, remove max_violations 2025-01-30 16:59:44 +00:00
broker Return a warning and fail if creating a store at global scope 2024-12-17 11:32:43 -07:00
cluster btest/cluster: Bump timeouts to 30 seconds 2024-12-13 18:28:43 +01:00
core Merge remote-tracking branch 'J-Gras/topic/jgras/add-unknown-proto-count' 2024-12-10 16:16:12 -07:00
coverage cluster: Be noisy when attempting to connect to an unknown node 2024-12-12 13:01:04 +01:00
dns_mgr btest: Add integration test for DNS_Mgr 2024-11-08 11:29:31 +01:00
doc Add btest for Zeekygen docs extraction on identifiers defined by the Zeek core. 2024-05-03 12:22:42 -07:00
Files cluster/backend/zeromq: Add ZeroMQ based cluster backend 2024-12-10 20:33:02 +01:00
javascript testing/btest: Log::delay() from JavaScript 2023-11-29 12:52:19 +01:00
language Expr: Fix folding of pattern values to support == and != 2024-12-16 10:56:02 +01:00
misc
opt Merge remote-tracking branch 'origin/topic/johanna/gh-4061' 2025-01-14 14:39:56 +00:00
plugins logging: Fix HookLogInit() and HookLogWrite() info usage 2025-01-14 10:44:50 +01:00
scripts DPD: change handling of pre-confirmation violations, remove max_violations 2025-01-30 16:59:44 +00:00
signatures signatures: Add data_end_offset to signature_match() and custom events 2024-10-30 13:29:58 +01:00
spicy Require spicy for spicy.analyzer-id btest 2024-12-20 14:25:04 -07:00
supervisor btest/supervisor: Bump one timeout to 30 seconds 2024-12-13 18:35:39 +01:00
telemetry Change all instruments to only handle doubles 2024-05-31 13:36:37 -07:00
Traces QUIC/decrypt_crypto: Actually check if decryption was successful 2024-12-15 20:39:26 -07:00
.gitignore
btest.cfg removed a bunch of no-longer used BTest alternatives and their associated baselines 2024-12-12 19:16:34 -08:00
Makefile
random.seed
README

This a test suite of small "unit tests" that verify individual pieces of Zeek
functionality.  They all utilize BTest, a simple framework/driver for
writing unit tests.  More information about BTest can be found at
https://github.com/zeek/btest

The test suite's BTest configuration is handled through the
``btest.cfg`` file.  Of particular interest is the "TestDirs" settings,
which specifies which directories BTest will recursively search for
test files.

Significant Subdirectories
==========================

* Baseline/
	Validated baselines for comparison against the output of each
	test on future runs. If the new output differs from the Baseline
	output, then the test fails.

* Traces/
	Packet captures utilized by the various BTest tests.

* scripts/
	This hierarchy of tests emulates the hierarchy of the Zeek scripts/
	directory.

* coverage/
	This collection of tests relates to checking whether we're covering
	everything we want to in terms of tests, documentation, and which
	scripts get loaded in different Zeek configurations.  These tests are
	more prone to fail as new Zeek scripts are developed and added to the
	distribution -- checking the individual test's comments is the best
	place to check for more details on what exactly the test is checking
	and hints on how to fix it when it fails.

Running Tests
=============

Either use the ``make all`` or ``make brief`` ``Makefile`` targets, or
run ``btest`` directly with desired options/arguments.  Examples:

* btest <no arguments>
	If you simply execute btest in this directory with no arguments,
	then all directories listed as "TestDirs" in btest.cfg will be
	searched recursively for test files.


* btest <btest options> test_directory
	You can specify a directory on the command line to run just the
	tests contained in that directory. This is useful if you wish to
	run all of a given type of test, without running all the tests
	there are. For example, "btest scripts" will run all of the Zeek
	script unit tests.


* btest <btest options> test_directory/test_file
	You can specify a single test file to run just that test. This
	is useful when testing a single failing test or when developing
	a new test.

Adding Tests
=============

See either the `BTest documentation
<https://github.com/zeek/btest>`_ or the existing unit
tests for examples of what they actually look like.  The essential
components of a new test include:

* A test file in one of the subdirectories listed in the ``TestDirs``
  of the ``btest.cfg`` file.

* If the unit test requires a known-good baseline output against which
  future tests will be compared (via ``btest-diff``), then that baseline
  output will need to live in the ``Baseline`` directory.  Manually
  adding that is possible, but it's easier to just use the ``-u`` or
  ``-U`` options of ``btest`` to do it for you (using ``btest -d`` on a
  test for which no baseline exists will show you the output so it can
  be verified first before adding/updating the baseline output).

If you create a new top-level testing directory for collecting related
tests, then you'll need to add it to the list of ``TestDirs`` in
``btest.cfg``. Do this only if your test really doesn't fit logically in
any of the extant directories.