better (than nothing) run-time errors for compiled scripts
fixes for dealing with recursive types in compiled scripts
fix for values in compiled scripts containing embedded comment markers
Since ccache in Cirrus currently does not preserve timestamps the
timestamp-based cache pruning implemented by ccache likely would just
remove random files. To work around this, this patch implements a manual
pruning step. This step runs after the build so that at least the files
used in the build should have their timestamps updated. We can then
force eviction of some of the unused files from the cache by cleaning
the cache with a size less than the maximum cache size.
This fixes a potential crash due to trigger_mgr getting shutdown earlier
than dns_mgr, and dns_mgr then trying to use it after it's been deleted.
This change forces the order of initialization/destruction in
iosource_mgr to cause dns_mgr to be deleted first.
The ja3 package uses next instead of return and triggers the new
errors with Zeek 5.2. That seems somewhat bad.
In case we want to do a 5.2.1 that makes this a warning, this
would be change to do so.
* origin/topic/awelzel/zeekctl-ci:
Bump zeekctl submodule
cirrus: Add zeekctl CI task
cirrus: Do not run tasks for builds on cron triggered jobs
cirrus: Rename BRANCH_WHITELIST
This change exposes the signature tyope inside the signed portion of an
X.509 certificate. In the past, we only exposed the signature type that
is contained inside the signature, which is outside the signed portion
of the X.509 certificate.
In theory, both signature fields should have the same value; it is,
however, possible to encode differing values in both fields. The new
field is not logged by default.
Run the zeekctl testing suite within a Debian 11 container. This runs on
pushes to master and release branches. Also, tagging PRs with fullci or
zeekctlci triggers it.
Closeszeek/zeekctl#49
This adds one metric per log stream and one metric per log writer (path based)
to track the number of writes on a stream level as well as on a writer level.
$ curl -sSf localhost:8181/metrics | grep Conn
zeek_log_writer_writes_total{endpoint="",filter-name="default",module="HTTP",path="http",stream="HTTP::LOG",writer="Log::WRITER_SQLITE"} 1 1677497572770
zeek_log_stream_writes_total{endpoint="",module="HTTP",stream="HTTP::LOG"} 1 1677497572770
The initial version of this change also included metrics around log
write vetoes, but given no log policies exist in the default configuration
and they are mostly interesting for a few streams/writers only, skip this
for now. These can always be added by the script writer, too.
The difference between the stream level writes and concrete writers can
be used to deduce the number of vetoes (or errors) as a starting point.
* origin/topic/awelzel/cap-ccache-sizes:
cirrus: Also add ccache limits to container builds
cirrus: Add an epoch to the ccache fingerprint, bump to 1000M
cirrus: Cap ccache max size to 500M and max files to 20000
When an analyzer is being removed from a connection's analyzer tree,
it's at first only marked as removed, but not yet actually deleted.
The problem was that until destruction happens, the analyzer methods
checking presence of an analyzer would continue to indicate that it's
there, even though semantically one wouldn't expect to find it anymore
after that removal operation. That in turn then prevented one from
adding a new analyzer of the same type during that time until
destruction (because we don't allow more than one of each kind at the
same time).
This fix changes the logic searching for analyzers to ignore any
existing ones scheduled for removal. I believe this shouldn't have any
further side effects.
Closes#2801.
Merging this will flush the caches once by changing the fingerprint.
Also increase the limit to 1GB to strike a balance between fast
down/uploads and suboptimal ccache pruning:
After #2802, depending on the task/platform, a mostly cached build on Cirrus CI
is taking 30-60seconds. Eye balling a few tasks, it is now taking longer to
download and unpack as well as check and re-upload the accumulated cache in
the beginning and end of a task.
For Debian 11, this was ~1:20 and 1:10 with a cache size of ~4.2GB. The
default size limit for ccache is 5GB, there's no limit to the number of files.
Running a fresh build on Debian 11, ccache -s indicates the actual
required cache size is ~100MB and the number of files in cache is ~2.7k.
Lower cache size to 500M and number of files to 20000, such that we don't
unnecessarily accumulate the cache and spend resources on downloading,
checking and re-uploading the cache.
root@cirrus-ci-task-4907974120964096# ccache -s | grep -E 'files|size'
files in cache 2736
cache size 96.7 MB
max cache size 5.0 GB
PRs have their own cache namespace, so they won't thrash the main's
branch cache. I think main and release share their branch, so we should
not be super aggressive. Though we're probably okay with cache misses on
release once in a while.
* origin/topic/timw/2574-windows-build-warnings:
Include stdint.h early in flex files to prevent redefinition of int type macros
Update bifcl submodule [nomail]
Update binpac submodule [nomail]
SMB: clarify a confusing line in one of the pac files
module_util: Change streq helper method to just return bool
NTLM analyzer: Don't bother checking boolean values for > 0
FTP analyzer: Remove unnecessary lines from constructor
IP_Hdr: take length value as uint64_t instead of int
EventTrace: take size_t as a constructor argument instead of int
Change argument to Tempvar constructor to size_t, avoids conversion warning in template construction
