In the past, we used a default canonifier, which removes everything that
looks like a timestamp from log files. The goal of this is to prevent
logs from changing, e.g., due to local system times ending up in log
files.
This, however, also has the side-effect of removing information that is
parsed from protocols which probably should be part of our tests.
There is at least one test (1999 certificates) where the entire test
output was essentially removed by the canonifier.
GH-4521 was similarly masked by this.
This commit changes the default canonifier, so that only the first
timestamp in a line is removed. This should skip timestamps that are
likely to change while keeping timestamps that are parsed
from protocol information.
A pass has been made over the tests, with some additional adjustments
for cases which require the old canonifier.
There are some cases in which we probably could go further and not
remove timestamps at all - that, however, seems like a follow-up
project.
The way that SMB timestamps were calculated used a "long double" for
the conversion calculation.
OS-X and Linux seem to have different sizes for long doubles. On the
Linux distributions that I have available, a long double is 128bits, vs
64 bits in OS-X.
This leads to slight discrepancies in the timestamps. This commit
changes this calculation to use a double on both systems, which is 64
bits and eliminates this difference.
Not the not_before/not_after fields output GMT based times.
Also adds a new btest diff canonifier which only removes the first
timestamp in a line.
Fixes GH-4521
The Spicy SSL analyzer was using references pretty heavily, probably to
work around now fixed issues with `inout` parameters in older Spicy
versions. At least for units this seems not needed anymore, and was also
partially incorrect, e.g., the Spicy docs call out that when using
`inout` parameters, passed and expected types should match exactly so
passing a reference as an `inout` value seems incorrect. Additionally,
one use case for references in Spicy is to use their interior
mutability, i.e., a reference never needs to be passed `inout` since
their can always be mutated.
Internally units are stored as reference-counted values, and references
to units are not much cheaper since they also need to be refcounted.
With that there seems litle reason to use references in this analyzer at
all, and this patch drops their use completely; instead we either pass
values, or values declared `inout`.
We leave the use of references for sharing sinks in place.
When looking up the postprocessor function from shadow files, id::find_func()
would abort() if the function wasn't available instead of falling back
to the default postprocessor.
Fix by using id::find() and checking the type explicitly and also adding a
strict type check while at it.
This issue was tickled by loading the json-streaming-logs package,
Zeek creating shadow files containing its custom postprocessor function,
then restarting Zeek without the package loaded.
Closes#4562
* origin/topic/timw/util-types:
Remove some unused #includes from spicy code
Remove using util.h in various headers in favor of util-types.h
Move type definitions/aliases from util.h to a separate file
After a public discussion and also chatting with Vern directly, disable the
ZAM bif tracking test to avoid an update every time new functions are
added. Usually these aren't performance critical and the defaults
characterization is fine. If they are performance critical, then Vern
is currently best positioned to properly integrate an optimized version.
* topic/christian/btest-trace-cleanup:
Btests: don't use -C in Zeek invocations that don't actually need it
Remove executable file permission bits from a bunch of our pcaps
