This adds one metric per log stream and one metric per log writer (path based)
to track the number of writes on a stream level as well as on a writer level.
$ curl -sSf localhost:8181/metrics | grep Conn
zeek_log_writer_writes_total{endpoint="",filter-name="default",module="HTTP",path="http",stream="HTTP::LOG",writer="Log::WRITER_SQLITE"} 1 1677497572770
zeek_log_stream_writes_total{endpoint="",module="HTTP",stream="HTTP::LOG"} 1 1677497572770
The initial version of this change also included metrics around log
write vetoes, but given no log policies exist in the default configuration
and they are mostly interesting for a few streams/writers only, skip this
for now. These can always be added by the script writer, too.
The difference between the stream level writes and concrete writers can
be used to deduce the number of vetoes (or errors) as a starting point.
* origin/topic/awelzel/cap-ccache-sizes:
cirrus: Also add ccache limits to container builds
cirrus: Add an epoch to the ccache fingerprint, bump to 1000M
cirrus: Cap ccache max size to 500M and max files to 20000
When an analyzer is being removed from a connection's analyzer tree,
it's at first only marked as removed, but not yet actually deleted.
The problem was that until destruction happens, the analyzer methods
checking presence of an analyzer would continue to indicate that it's
there, even though semantically one wouldn't expect to find it anymore
after that removal operation. That in turn then prevented one from
adding a new analyzer of the same type during that time until
destruction (because we don't allow more than one of each kind at the
same time).
This fix changes the logic searching for analyzers to ignore any
existing ones scheduled for removal. I believe this shouldn't have any
further side effects.
Closes#2801.
Merging this will flush the caches once by changing the fingerprint.
Also increase the limit to 1GB to strike a balance between fast
down/uploads and suboptimal ccache pruning:
After #2802, depending on the task/platform, a mostly cached build on Cirrus CI
is taking 30-60seconds. Eye balling a few tasks, it is now taking longer to
download and unpack as well as check and re-upload the accumulated cache in
the beginning and end of a task.
For Debian 11, this was ~1:20 and 1:10 with a cache size of ~4.2GB. The
default size limit for ccache is 5GB, there's no limit to the number of files.
Running a fresh build on Debian 11, ccache -s indicates the actual
required cache size is ~100MB and the number of files in cache is ~2.7k.
Lower cache size to 500M and number of files to 20000, such that we don't
unnecessarily accumulate the cache and spend resources on downloading,
checking and re-uploading the cache.
root@cirrus-ci-task-4907974120964096# ccache -s | grep -E 'files|size'
files in cache 2736
cache size 96.7 MB
max cache size 5.0 GB
PRs have their own cache namespace, so they won't thrash the main's
branch cache. I think main and release share their branch, so we should
not be super aggressive. Though we're probably okay with cache misses on
release once in a while.
* origin/topic/timw/2574-windows-build-warnings:
Include stdint.h early in flex files to prevent redefinition of int type macros
Update bifcl submodule [nomail]
Update binpac submodule [nomail]
SMB: clarify a confusing line in one of the pac files
module_util: Change streq helper method to just return bool
NTLM analyzer: Don't bother checking boolean values for > 0
FTP analyzer: Remove unnecessary lines from constructor
IP_Hdr: take length value as uint64_t instead of int
EventTrace: take size_t as a constructor argument instead of int
Change argument to Tempvar constructor to size_t, avoids conversion warning in template construction
The py39-pip package is sometimes not available on FreeBSD, and using
ensurepip is the recommended way to install pip according to the pip
documentation. ensurepip should be available in every python
installation.
The OBS builds run in an environment where no git is installed and
release tarballs assembled manually.
Drop the hard-requirement on git from ci/collect-repo-info.json.
When bumping the VERSION file, everything that includes the auto-generated
zeek/zeek-config.h needs to rebuild and ccache usage is voided due the file
changing. Split the mutable version information into a new zeek-version.h
file to avoid this.
Further, do not include zeek-version.h within Plugin.h and avoid the reference
to ZEEK_VERSION_FUNCTION unless we're building an external plugin.
Closes#2776.
* origin/topic/awelzel/2572-event-handler-stats:
Bump external baselines
external/testing-setup: Less telemetry.log volume
EventHandler: Use telemetry framework for EventHandler.call_count
For traces spanning a long time period, don't produce immense
amounts of logs. Also filter down the event handler invocation
entries to connection related events in telemetry.log.
Put the IntCounter into a std::optional rather than initializing
it at EventHandler construction time as that will currently expose
a time series per event handler through the Prometheus API.
