* topic/christian/management-verify-nodestarts:
Management framework: bump external cluster testsuite
Management framework: bump zeek-client to pull in set-config rendering
Management framework: enable stdout/stderr reporting
Management framework: Supervisor extensions for stdout/stderr handling
Management framework: disambiguate redef field names in agent and controller
Management framework: move to ResultVec in agent's set_configuration response
Management framework: tune request timeout granularity and interval
Management framework: verify node starts when deploying a configuration
Management framework: a bit of debug-level logging for troubleshooting
This improves the framework's handling of Zeek node stdout and stderr by
extending the (script-layer) Supervisor functionality.
- The Supervisor _either_ directs Zeek nodes' stdout/stderr to files _or_ lets
you hook into it at the script level. We'd like both: files make sense to allow
inspection outside of the framework, and the framework would benefit from
tapping into the streams e.g. for error context. We now provide the file
redirection functionality in the Supervisor, in addition to the hook
mechanism. The hook mechanism also builds up rolling windows of up to
100 lines (configurable) into stdout/stderr.
- The new Mangement::Supervisor::API::notify_node_exit event notifies
subscribers (agents, really) that a particular node has exited (and is possibly
being restarted by the Supervisor). The event includes the name of the node,
plus its recent stdout/stderr context.
The Supervisor generates this event every time it receives a status update from
the stem, meaning a node got created or re-created. A corresponding
SupervisorControl::node_status event relays the same information for users
interacting with the Supervisor over Broker.
* topic/christian/management-cluster-dirs:
Management framework: bump zeek-client to pull in instance serialization fixes
Management framework: bump external cluster testsuite
Management framework: update agent-checkin test to reflect recent changes
Management framework: place each Zeek process in its own working dir
Management framework: set defaults for log rotation and persistent state
Management framework: add spool and state directory config settings
Management framework: establish stdout/stderr files also for cluster nodes
Management framework: default to having agents check in with the (local) controller
Management framework: move role variable from logging into framework-wide config
Management framework: distinguish supervisor/supervisee when loading agent/controller
Management framework: simplify agent and controller stdout/stderr files
Management framework: prefix the management logs with "management-"
Management framework: comment and layouting tweaks, no functional change
Management framework: rename env var that labels agents/controllers
Management framework: increase robustness of agent/controller naming
This adds management/persistence.zeek to establish common configuration for log
rotation and persistent variable state. Log-writing Zeek processes initially
write locally in their working directory, and rotate into subdirectory
"log-queue" of the spool. Since agent and controller have no logger,
persistence.zeek puts in place compatible configurations for them.
Storage folders for Broker-backed tables and clusterized stores default to
subdirectories of the new Zeek-level state folder.
When setting the ZEEK_MANAGEMENT_TESTING environment variable, persistent state
is kept in the local directory, and log rotation remains disabled.
This also tweaks @loads a bit in favor of simply loading frameworks/management,
which is easier to keep track of.
When passing an empty string as a directory, the function would produce
filenames starting with a slash even when the given file_name is not an absolute
path. Defaulting to the root directory is likely never intended and might
conveivably be dangerous. The middle "/" is now skipped also if dir is an empty
string.
* origin/topic/bbannier/spicy:
Add NEWS item for new requirements due to default-enabled Spicy support.
Include headers in binary dir before other headers.
Increase memory in CI.
Add tests for bundled Spicy infrastructure.
Include spicy in build.
Add `bare_mode` bif.
Change test so included plugins can load notice framework.
Do not log function arguments in test.
* origin/topic/timw/1857-input-log-messages:
Move some repetitive code into a separate method
Store location during Ascii input parsing for use in error messages
Include file information in input reader error messages
* origin/topic/vern/CPP-feature-complete:
NEWS items for script profiling, "footprint" BiFs, and Trigger method deprecations
tweaks per reviewing feedback
make deprecation advice a bit more helpful
format shell scripts per style guide
add deprecation for Trigger Cache/Lookup interfaces
helper scripts for -O C++ maintenance
"-O use-C++" now fails if no C++ bodies wound up being applicable
update environemnt variable names to match command-line arguments
final changes for -O C++ feature completeness, mainly "when" statement support
Change "when" statements that don't require closures to use new implementation. Provide hooks for script optimization access to "when" statements. Regularize treatment of naming and timeouts for Triggers.
switch cached Trigger values to be opaque-and-generic, rather than assuming CallExpr's
tracking of when statements/expressions occur in a "when" context
fix bug that sometimes associated the wrong location with global statements
fix incomplete AST traversal of "when" statements
add some missing "when" closures
regularize test suite names to avoid custom/outdated suffices
This was excercising hooks to validate that they were called. For that
it uses pretty verbose logging. Since version numbers are not stable and
hard to canonify this script already unloads the version module (there
is e.g., code in `base/misc/version` which splits the version strings
and works on single components which might appear like pretty ordinary
floating point numbers in the log).
This test however worked under the assumption that nothing in bare mode
loads `base/misc/version` which is hard to guarantee, especially
considering that plugins can be embedded and might load that module
themself.
With this patch we now make the logging slightly less verbose so that
function call arguments are not logged anymore.
* origin/topic/vern/script-profiling:
tidy up after generating profile
test suite updates for refined script coverage, use of new BiF to speed startup
fix for coverage reporting for functions that use "when" statements
new global_options() BiF to speed up startup, plus a micro-preen
hooks for new --profile-scripts option
classes for managing script profiles
address some holes in script coverage
fix for script coverage missing on-exit activity
memory management fixes for loggers
make curr_CPU_time() broadly available rather than just isolated to ZAM
* origin/topic/vern/type-names:
regularize deprecation warning
make DoDescribe protected for a tidier interface mark use of DESC_PORTABLE as deprecated
when printing a description of a type, use its name if available
* origin/topic/vern/footprint:
new environment variable to enable BTests to skip ASAN checks
skip new BiF test for ASAN CI runs
btest update to include recursive value that doesn't require a record
to avoid recursion, track all aggregates, not just records isolate the internal methods
simpler public calling interface for computing footprint
use stack-based set to prevent infinite recursion rather than a static one
change value_footprint() to val_footprint() to be more similar to val_size()
make including count of container elements non-optional
btest for mutually-recursive case
fix for tracking footprints of mutually-recursive records
added value_footprint() and global_container_footprints() BiFs
