logging: Introduce Log::delay() and Log::delay_finish()

This is a verbose, opinionated and fairly restrictive version of the log delay idea.
Main drivers are explicitly, foot-gun-avoidance and implementation simplicity.

Calling the new Log::delay() function is only allowed within the execution
of a Log::log_stream_policy() hook for the currently active log write.

Conceptually, the delay is placed between the execution of the global stream
policy hook and the individual filter policy hooks. A post delay callback
can be registered with every Log::delay() invocation. Post delay callbacks
can (1) modify a log record as they see fit, (2) veto the forwarding of the
log record to the log filters and (3) extend the delay duration by calling
Log::delay() again. The last point allows to delay a record by an indefinite
amount of time, rather than a fixed maximum amount. This should be rare and
is therefore explicit.

Log::delay() increases an internal reference count and returns an opaque
token value to be passed to Log::delay_finish() to release a delay reference.
Once all references are released, the record is forwarded to all filters
attached to a stream when the delay completes.

This functionality separates Log::log_stream_policy() and individual filter
policy hooks. One consequence is that a common use-case of filter policy hooks,
removing unproductive log records, may run after a record was delayed. Users
can lift their filtering logic to the stream level (or replicate the condition
before the delay decision). The main motivation here is that deciding on a
stream-level delay in per-filter hooks is too late. Attaching multiple filters
to a stream can additionally result in hard to understand behavior.

On the flip side, filter policy hooks are guaranteed to run after the delay
and can be used for further mangling or filtering of a delayed record.

src/logging/Manager.h

@@ -30,6 +30,33 @@ class WriterFrontend;
 class RotationFinishedMessage;
 class RotationTimer;
 
+namespace detail {
+
+class DelayInfo;
+
+using WriteIdx = uint64_t;
+
+/**
+ * Information about a Log::write() call.
+ */
+struct WriteContext {
+    EnumValPtr id = nullptr;
+    RecordValPtr record = nullptr;
+    WriteIdx idx = 0; // Ever increasing counter.
+
+    bool operator<(const WriteContext& o) const {
+        assert(id == o.id);
+        return idx < o.idx;
+    }
+
+    bool operator==(const WriteContext& o) const {
+        assert(id == o.id);
+        return idx == o.idx;
+    }
+};
+
+} // namespace detail
+
 /**
  * Singleton class for managing log streams.
  */
@@ -152,6 +179,63 @@ public:
      */
     bool Write(EnumVal* id, RecordVal* columns);
 
+    /**
+     * Delay the currently active @ref Write operation.
+     *
+     * This method is only allowed to be called during the execution of the
+     * Log::log_stream_policy Zeek script hook. This restriction may be
+     * relaxed in the future.
+     *
+     * @param id  The enum value corresponding the log stream.
+     *
+     * @param record The log record to delay.
+     *
+     * @param post_delay_cb A callback function to invoke when the delay
+     * has completed or nullptr.
+     *
+     * @return An opaque token that can be passed to DelayFinish() to
+     * release a delayed Log::write() operation.
+     */
+    ValPtr Delay(const EnumValPtr& id, const RecordValPtr record, FuncPtr post_delay_cb);
+
+    /**
+     * Release reference for a delayed Log::write().
+     *
+     * @param id  The enum value corresponding the log stream.
+     *
+     * @param record The log record previously passed to Delay()
+     *
+     * @param token The token returned by the Delay() call.
+     *
+     * @return Returns true if the call was successful.
+     */
+    bool DelayFinish(const EnumValPtr& id, const RecordValPtr& record, const ValPtr& token);
+
+    /**
+     * Update the maximum delay interval of a given stream.
+     *
+     * Currently, it is only allowed to increase the maximum
+     * delay of a stream.
+     *
+     * @param id The enum value corresponding to the log stream.
+     *
+     * @param max_delay  The new maximum delay, in seconds.
+     *
+     * @return Returns true if the call was successful, else false.
+     */
+    bool SetMaxDelayInterval(const EnumValPtr& id, double max_delay);
+
+    /**
+     * Set the maximum delay queue size for the given stream.
+     *
+     * @param id The enum value corresponding to the log stream.
+     *
+     * @param max_queue_length The new maximum queue length.
+     *
+     * @return Returns true if the call was successful, else false.
+     */
+    bool SetMaxDelayQueueSize(const EnumValPtr& id, zeek_uint_t max_queue_length);
+
     /**
      * Create a new log writer frontend. This is exposed so that the
      * communication system can recreate remote log streams locally.
@@ -279,6 +363,7 @@ private:
     struct Filter;
     struct Stream;
     struct WriterInfo;
+    class LogDelayExpiredTimer;
 
     bool TraverseRecord(Stream* stream, Filter* filter, RecordType* rt, TableVal* include, TableVal* exclude,
                         const std::string& path, const std::list<int>& indices);
@@ -303,6 +388,8 @@ private:
 
     bool RemoveStream(unsigned int idx);
 
+    bool DelayCompleted(Manager::Stream* stream, detail::DelayInfo& delay_info);
+
     std::vector<Stream*> streams; // Indexed by stream enum.
     int rotations_pending;        // Number of rotations not yet finished.
     FuncPtr rotation_format_func;
@@ -310,6 +397,9 @@ private:
 
     telemetry::IntCounterFamily total_log_stream_writes_family;
     telemetry::IntCounterFamily total_log_writer_writes_family;
+
+    zeek_uint_t last_delay_token = 0;
+    std::vector<detail::WriteContext> active_writes;
 };
 
 } // namespace logging