Management framework: tune request timeout granularity and interval

When the controller relays requests to agents, we want agents to time out more
quickly than the corresponding controller requests. This allows agents to
respond with more meaningful errors, while the controller's timeout acts mostly
as a last resort to ensure a response to the client actually happens.

This dials down the table_expire_interval to 2 seconds in both agent and
controller, for more predictable timeout behavior. It also dials the agent-side
request expiration interval down to 5 seconds, compared to the agent's 10
seconds.

We may have to revisit this to allow custom expiration intervals per
request/response message type.

scripts/policy/frameworks/management/agent/main.zeek

@@ -52,6 +52,14 @@ redef record Management::Request::Request += {
 # Tag our logs correctly
 redef Management::role = Management::AGENT;
 
+# Conduct more frequent table expiration checks. This helps get more predictable
+# timing for request timeouts and only affects the controller, which is mostly idle.
+redef table_expire_interval = 2 sec;
+
+# Tweak the request timeout so it's relatively quick, and quick enough always to
+# time out strictly before the controller's request state (at 10 sec).
+redef Management::Request::timeout_interval = 5 sec;
+
 # Returns the effective agent topic for this agent.
 global agent_topic: function(): string;
 

scripts/policy/frameworks/management/controller/main.zeek

@@ -73,6 +73,10 @@ redef record Management::Request::Request += {
 # Tag our logs correctly
 redef Management::role = Management::CONTROLLER;
 
+# Conduct more frequent table expiration checks. This helps get more predictable
+# timing for request timeouts and only affects the agent, which is mostly idle.
+redef table_expire_interval = 2 sec;
+
 global check_instances_ready: function();
 global add_instance: function(inst: Management::Instance);
 global drop_instance: function(inst: Management::Instance);

scripts/policy/frameworks/management/request.zeek

@@ -32,11 +32,15 @@ export {
 		finished: bool &default=F;
 	};
 
-	## The timeout for request state. Such state (see the :zeek:see:`Management::Request`
+	## The timeout interval for request state. Such state (see the
-	## module) ties together request and response event pairs. The timeout causes
+	## :zeek:see:`Management::Request` module) ties together request and
-	## its cleanup in the absence of a timely response. It applies both to
+	## response event pairs. A timeout causes cleanup of request state if
-	## state kept for client requests, as well as state in the agents for
+	## regular request/response processing hasn't already done so. It
-	## requests to the supervisor.
+	## applies both to request state kept in the controller and the agent,
+	## though the two use different timeout values: agent-side requests time
+	## out more quickly. This allows agents to send more meaningful error
+	## messages, while the controller's timeouts serve as a last resort to
+	## ensure response to the client.
 	const timeout_interval = 10sec &redef;
 
 	## A token request that serves as a null/nonexistant request.