zeek/doc/frameworks/configuration.rst

.. _framework-configuration:

=======================
Configuration Framework
=======================

Zeek includes a configuration framework that allows updating script options at
runtime. This functionality consists of an :zeek:see:`option` declaration in
the Zeek language, configuration files that enable changing the value of
options at runtime, option-change callbacks to process updates in your Zeek
scripts, a couple of script-level functions to manage config settings directly,
and a log file (:file:`config.log`) that contains information about every
option value change according to :zeek:see:`Config::Info`.

Introduction
============

The configuration framework provides an alternative to using Zeek script
constants to store various Zeek settings.

While traditional constants work well when a value is not expected to change at
runtime, they cannot be used for values that need to be modified occasionally.
While a :zeek:see:`redef` allows a re-definition of an already defined constant
in Zeek, these redefinitions can only be performed when Zeek first starts.
Afterwards, constants can no longer be modified.

However, it is clearly desirable to be able to change at runtime many of the
configuration options that Zeek offers. Restarting Zeek can be time-consuming
and causes it to lose all connection state and knowledge that it accumulated.
Zeek’s configuration framework solves this problem.

Declaring Options
=================

The :zeek:see:`option` keyword allows variables to be declared as configuration
options:

.. code-block:: zeek

  module Test;

  export {
      option my_networks: set[subnet] = {};
      option enable_feature = F;
      option hostname = "testsystem";
      option timeout_after = 1min;
      option my_ports: vector of port = {};
  }

Options combine aspects of global variables and constants. Like global
variables, options cannot be declared inside a function, hook, or event
handler. Like constants, options must be initialized when declared (the type
can often be inferred from the initializer but may need to be specified when
ambiguous). The value of an option can change at runtime, but options cannot be
assigned a new value using normal assignments.

The initial value of an option can be redefined with a :zeek:see:`redef`
declaration just like for global variables and constants. However, there is no
need to specify the :zeek:see:`&redef` attribute in the declaration of an
option. For example, given the above option declarations, here are possible
redefs that work anyway:

.. code-block:: zeek

  redef Test::enable_feature = T;
  redef Test::my_networks += { 10.1.0.0/16, 10.2.0.0/16 };

Changing Options
================

The configuration framework facilitates reading in new option values from
external files at runtime. Configuration files contain a mapping between option
names and their values. Each line contains one option assignment, formatted as
follows::

  [option name][tab/spaces][new value]

Lines starting with ``#`` are comments and ignored.

You register configuration files by adding them to
:zeek:see:`Config::config_files`, a set of filenames. Simply say something like
the following in :file:`local.zeek`:

.. code-block:: zeek

  redef Config::config_files += { "/path/to/config.dat" };

Zeek will then monitor the specified file continuously for changes. For
example, editing a line containing::

  Test::enable_feature T

to the config file while Zeek is running will cause it to automatically update
the option’s value in the scripting layer. The next time your code accesses the
option, it will see the new value.

.. note::

  The config framework is clusterized. In a cluster configuration, only the
  manager node watches the specified configuration files, and relays option
  updates across the cluster.

Config File Formatting
----------------------

The formatting of config option values in the config file is not the same as in
Zeek’s scripting language. Keep an eye on the :file:`reporter.log` for warnings
from the config reader in case of incorrectly formatted values, which it’ll
generally ignore when encountered. The following table summarizes supported
types and their value representations:

.. list-table::
  :header-rows: 1

  * - Data Type
    - Sample Config File Entry
    - Comments

  * - :zeek:see:`addr`
    - ``1.2.3.4``
    - Plain IPv4 or IPv6 address, as in Zeek. No ``/32`` or similar netmasks.

  * - :zeek:see:`bool`
    - ``T``
    - ``T`` or ``1`` for true, ``F`` or ``0`` for false

  * - :zeek:see:`count`
    - ``42``
    - Plain, nonnegative integer.

  * - :zeek:see:`double`
    - ``-42.5``
    - Plain double number.

  * - :zeek:see:`enum`
    - ``Enum::FOO_A``
    - Plain enum string.

  * - :zeek:see:`int`
    - ``-1``
    - Plain integer.

  * - :zeek:see:`interval`
    - ``3600.0``
    - Always in epoch seconds, with optional fraction of seconds. Never
      includes a time unit.

  * - :zeek:see:`pattern`
    - ``/(foo|bar)/``
    - The regex pattern, within forward-slash characters.

  * - :zeek:see:`port`
    - ``42/tcp``
    - Port number with protocol, as in Zeek. When the protocol part is missing,
      Zeek interprets it as ``/unknown``.

  * - :zeek:see:`set`
    - ``80/tcp,53/udp``
    - The set members, formatted as per their own type, separated by commas.
      For an empty set, use an empty string: just follow the option name with
      whitespace.

      Sets with multiple index types (e.g. ``set[addr,string]``) are currently
      not supported in config files.

  * - :zeek:see:`string`
    - ``Don’t bite, Zeek``
    - Plain string, no quotation marks. Given quotation marks become part of
      the string. Everything after the whitespace separator delineating the
      option name becomes the string. Saces and special characters are fine.
      Backslash characters (e.g. ``\n``) have no special meaning.

  * - :zeek:see:`subnet`
    - ``1.2.3.4/16``
    - Plain subnet, as in Zeek.

  * - :zeek:see:`time`
    - ``1608164505.5``
    - Always in epoch seconds, with optional fraction of seconds. Never
      includes a time unit.

  * - :zeek:see:`vector`
    - ``1,2,3,4``
    - The set members, formatted as per their own type, separated by commas.
      For an empty vector, use an empty string: just follow the option name
      with whitespace.

This leaves a few data types unsupported, notably tables and records. If you
require these, build up an instance of the corresponding type manually (perhaps
from a separate input framework file) and then call
:zeek:see:`Config::set_value` to update the option:

.. code-block:: zeek

  module Test;

  export {
      option host_port: table[addr] of port = {};
  }

  event zeek_init() {
      local t: table[addr] of port = { [10.0.0.2] = 123/tcp };
      Config::set_value("Test::host_port", t);
  }


Regardless of whether an option change is triggered by a config file or via
explicit :zeek:see:`Config::set_value` calls, Zeek always logs the change to
:file:`config.log`. A sample entry::

  #fields ts      id      old_value       new_value       location
  #types  time    string  string  string  string
  1608167352.498872      Test::a_count     42      3      config.txt

Mentioning options repeatedly in the config files leads to multiple update
events; the last entry “wins”. Mentioning options that do not correspond to
existing options in the script layer is safe, but triggers warnings in
:file:`reporter.log`::

  warning: config.txt/Input::READER_CONFIG: Option 'an_unknown' does not exist. Ignoring line.

Internally, the framework uses the Zeek input framework to learn about config
changes. If you inspect the configuration framework scripts, you will notice
that the scripts simply catch input framework events and call
:zeek:see:`Config::set_value` to set the relevant option to the new value. If
you want to change an option in your scripts at runtime, you can likewise call
:zeek:see:`Config::set_value` directly from a script (in a cluster
configuration, this only needs to happen on the manager, as the change will be
automatically sent to all other nodes in the cluster).

.. note::

  The input framework is usually very strict about the syntax of input files, but
  that is not the case for configuration files. These require no header lines,
  and both tabs and spaces are accepted as separators. A custom input reader,
  specifically for reading config files, facilitates this.

.. tip::

  The gory details of option-parsing reside in ``Ascii::ParseValue()`` in
  :file:`src/threading/formatters/Ascii.cc` and ``Value::ValueToVal`` in
  :file:`src/threading/SerialTypes.cc` in the Zeek core.

Change Handlers
===============

A change handler is a user-defined function that Zeek calls each time an option
value changes. This allows you to react programmatically to option changes. The
following example shows how to register a change handler for an option that has
a data type of :zeek:see:`addr` (for other data types, the return type and
second parameter data type must be adjusted accordingly):

.. code-block:: zeek

  module Test;

  export {
      option testaddr = 127.0.0.1;
  }

  # Note: the data type of 2nd parameter and return type must match
  function change_addr(id: string, new_value: addr): addr
      {
      print fmt("Value of %s changed from %s to %s", id, testaddr, new_value);
      return new_value;
      }

  event zeek_init()
      {
      Option::set_change_handler("Test::testaddr", change_addr);
      }

Immediately before Zeek changes the specified option value, it invokes any
registered change handlers. The value returned by the change handler is the
value Zeek assigns to the option.  This allows, for example, checking of values
to reject invalid input (the original value can be returned to override the
change).

.. note::

  :zeek:see:`Option::set_change_handler` expects the name of the option to
  invoke the change handler for, not the option itself. Also, that name
  includes the module name, even when registering from within the module.

It is possible to define multiple change handlers for a single option. In this
case, the change handlers are chained together: the value returned by the first
change handler is the “new value” seen by the next change handler, and so on.
The built-in function :zeek:see:`Option::set_change_handler` takes an optional
third argument that can specify a priority for the handlers.

A change handler function can optionally have a third argument of type string.
When a config file triggers a change, then the third argument is the pathname
of the config file. When the :zeek:see:`Config::set_value` function triggers a
change, then the third argument of the change handler is the value passed to
the optional third argument of the :zeek:see:`Config::set_value` function.

.. tip::

  Change handlers are also used internally by the configuration framework. If
  you look at the script-level source code of the config framework, you can see
  that change handlers log the option changes to :file:`config.log`.

When Change Handlers Trigger
----------------------------

Change handlers often implement logic that manages additional internal state.
For example, depending on a performance toggle option, you might initialize or
clean up a caching structure. In such scenarios you need to know exactly when
and whether a handler gets invoked. The following hold:

* When no config files get registered in :zeek:see:`Config::config_files`,
  change handlers do not run.
* When none of any registered config files exist on disk, change handlers do
  not run.

That is, change handlers are tied to config files, and don’t automatically run
with the option’s default values.

* When a config file exists on disk at Zeek startup, change handlers run with
  the file’s config values.
* When the config file contains the same value the option already defaults to,
  its change handlers are invoked anyway.
* :zeek:see:`zeek_init` handlers run before any change handlers — i.e., they
  run with the options’ default values.
* Since the config framework relies on the input framework, the input
  framework’s inherent asynchrony applies: you can’t assume when exactly an
  option change manifests in the code.

If your change handler needs to run consistently at startup and when options
change, you can call the handler manually from :zeek:see:`zeek_init` when you
register it. That way, initialization code always runs for the option’s default
value, and also for any new values.

.. code-block:: zeek

  module Test;

  export {
      option use_cache = T;
  }

  function use_cache_hdlr(id: string, new_value: bool): bool
      {
      if ( new_value ) {
          # Ensure caching structures are set up properly
      }

      return new_value;
      }

  event zeek_init()
      {
      use_cache_hdlr("Test::use_cache", use_cache);
      Option::set_change_handler("Test::use_cache", use_cache_hdlr);
      }