Commit graph

13262 commits

Author SHA1 Message Date
Christian Kreibich
30db1b3bfb First uses of request state timeouts
This now features support for the test_timeout_request/response events, as
supported by the client, and also adds a timeout event for set_configuration, in
case agents do not respond in time.

Includes corresponding zeek-client submodule bump.
2021-12-21 14:52:29 -08:00
Christian Kreibich
1e823f931e Add expiration mechanism to client request state.
This establishes a timeout controlled via ClusterController::request_timeout,
triggering a ClusterController::Request::request_expired event whenever a
timeout rolls around before request state has been finalized by a request's
normal processing.
2021-12-21 14:52:29 -08:00
Christian Kreibich
fc9679e510 Move get_instances_response event to using a Result record
Includes corresponding zeek-client bump.
2021-12-21 14:52:29 -08:00
Christian Kreibich
1461d56340 Track successful config deployment in cluster controller
This allows us to start returning deployed configurations to the client upon
request.
2021-12-21 14:52:29 -08:00
Christian Kreibich
bbe7d32529 Bump zeek-client 2021-12-21 14:52:29 -08:00
Christian Kreibich
09d9be3433 Add ClusterController::API::notify_agents_ready event
This changes the basic agent-management model to one in which the configurations
received from the client define not just the data cluster, but also set the set
of acceptable instances. Unless connectivity already exists, the controller will
establish peerings with new agents that listen, or wait for ones that connect to
the controller to check in.

Once all required agents are available, the controller triggers the new
notify_agents_ready event, an agent/controller-level "cluster-is-ready"
event. The controller also uses this event to submit a pending config update to
the now-ready instances.
2021-12-21 14:52:29 -08:00
Christian Kreibich
b57be021b7 Make all globals start with a "g_" prefix
This makes it easier to spot them in code, and is shorter than using explicit
namespacing.
2021-12-21 14:52:28 -08:00
Christian Kreibich
14a8c979c1 Add missing debug() log function to log module's API 2021-12-21 14:52:28 -08:00
Christian Kreibich
a56ee6b9a6 Add separate utility module for controller and agent
We can figure out later whether & where to re-settle helper functions that end
up in there.
2021-12-21 14:52:28 -08:00
Christian Kreibich
6fcd99b4a0 Bump zeek-client 2021-12-21 14:52:28 -08:00
Christian Kreibich
ddbd83fee4 Support for dropping instances no longer needed after config updates
This sends such expired instances empty configurations that will cause them to
shut down their remaining data cluster nodes.
2021-12-21 14:52:28 -08:00
Christian Kreibich
8eee5bb3d2 Additional infrastructure for printing types
Also added convenience for instantiating (dummy) configuration records.
2021-12-21 14:52:28 -08:00
Christian Kreibich
2395301168 Bump zeek-client 2021-12-21 14:52:28 -08:00
Christian Kreibich
5cb44c2f69 Support on-demand peering with agents when receiving new cluster configuration
Prior to this, static configuration needed to be in place to configure the
controller/agent layout. The configuration update can now include new instances
that the controller will connect to, assuming they're instances with a listening
agent.
2021-12-21 14:52:28 -08:00
Christian Kreibich
484f79f599 Expand requests support in the controller
Request records for configuration updates now store the full configuration. The
ClusterController::Request module now provies a to_string() function for
rendering requests to a string.
2021-12-21 14:52:28 -08:00
Christian Kreibich
aceb05099a Whitespace tweaks in cluster controller and agent scripts 2021-12-21 14:52:28 -08:00
Christian Kreibich
e9bdaebc70 Add Github action job for cluster tests
This job runs in sequence after the image build one, using its resulting image.
The actual tests live in the external zeek-testing-cluster testsuite, which
the new job clones and runs.

To specify a version of the testsuite to use, testing/external/ has a new
commit-hash.zeek-testing-cluster file that tracks the testsuite's relevant
commit ref
2021-12-21 14:52:28 -08:00
Christian Kreibich
2a5690fd00 Tweak Docker image configure invocation to include zeek-client
We don't yet install zeek-client by default, but need it in the image for
testing the cluster.
2021-12-21 14:52:28 -08:00
Christian Kreibich
eafa77b2b6 Add Docker image artifact storage
This enables subsequent jobs/workflows to access the generated image without use
of an image registry.
2021-12-21 14:52:28 -08:00
Robin Sommer
07045ec254
Fix host header normalization in intel framework.
The way we were splitting off ports from host names could fail for IPv6
addresses.

Closes #1844.
2021-12-21 21:54:47 +01:00
Robin Sommer
a7427e95bf
Switch to recording unmodified HTTP header.
We used to attempt to remove any port specification before recording
HTTP host headers in logs. Doing so would (1) remove potentially useful
information, (2) not match what the documentation seemed to suggest, and
(3) fail for IP6 addresses containing colons.

We now record the original HOST header as is.

Addresses #1844.
2021-12-21 21:54:47 +01:00
Robin Sommer
0507f6005c
Adding test for BitTorrent tracker.
Our test trace is extracted from https://www.cloudshark.org/captures/b9089aac6eee.

There actually seems to be a bug in the existing code: the URI passed to
bt_tracker_request() includes a partial HTTP version. This commits
includes the baseline as the current code produces it, we'll fix that in
a subsequent comment.
2021-12-21 17:48:26 +01:00
Dominik Charousset
da2a5ef455 Fix GCC builds and string output for Broker errors 2021-12-21 11:02:26 +01:00
Christian Kreibich
1e60264548 Update auxil/broker submodule [nomail] [skip ci] 2021-12-20 19:33:46 -08:00
Dominik Charousset
56f30b500a Update to latest Broker without public CAF dep 2021-12-20 08:16:21 +01:00
zeek-bot
e93fcd3c64 Update doc submodule [nomail] [skip ci] 2021-12-15 00:36:00 +00:00
Tim Wojtulewicz
2376fe414c Merge remote-tracking branch 'origin/topic/vern/usage-usage'
* origin/topic/vern/usage-usage:
  fixes for double-delete and reducing '?' operator with constant alternatives
  additional test suite updates for "-u" usage issues
  test suite updates for "xform" and "usage" alternatives, plus test name change
  removed unused script variable
  correct usage info for -u flag; -uu no longer supported
  fix typo in btest filename
2021-12-14 16:54:26 -07:00
Vern Paxson
06ffd97749 fixes for double-delete and reducing '?' operator with constant alternatives 2021-12-14 15:36:03 -08:00
Vern Paxson
77f6a658e6 additional test suite updates for "-u" usage issues 2021-12-14 15:35:24 -08:00
Tim Wojtulewicz
ba0a7c0fed NEWS: Fix some formatting issues 2021-12-14 14:01:27 -07:00
Tim Wojtulewicz
bfaf891a99 NEWS: add updates for new ZAM/script_opt features 2021-12-14 14:01:10 -07:00
Vern Paxson
dff664dc1f test suite updates for "xform" and "usage" alternatives, plus test name change 2021-12-14 12:50:17 -08:00
Vern Paxson
72a59bf828 removed unused script variable 2021-12-14 12:49:27 -08:00
Vern Paxson
5e71e757fe correct usage info for -u flag; -uu no longer supported 2021-12-14 11:37:51 -08:00
Vern Paxson
deb518588a fix typo in btest filename 2021-12-14 11:33:21 -08:00
Tim Wojtulewicz
4920b6affb Merge remote-tracking branch 'origin/topic/vern/cpp-incr'
* origin/topic/vern/cpp-incr:
  tweak to keep clang-format happy
  minor note regarding improving performance of C++-generated code
  factoring of generating C++ initializations, no semantic changes
  restored support for incremental compilation of scripts to C++
  fixes for -O gen-standalone-C++
  new ZEEK_FILE_ONLY and ZEEK_FUNC_ONLY environment variables for debugging script optimization - replaces ZEEK_ONLY
  fix for compiling record constructors to C++
  fixes for compiling vector operations to C++
  fixed for profiling missing some profile elements
  removing vestigial methods
  minor efficiency tweak for ZAM record construction
2021-12-14 11:35:11 -07:00
Tim Wojtulewicz
1b1cf8142d Merge remote-tracking branch 'origin/topic/timw/1860-double-to-int'
* origin/topic/timw/1860-double-to-int:
  GH-1860: Add double_to_int() bif
2021-12-14 09:23:59 -07:00
Tim Wojtulewicz
7de9a7d76f GH-1860: Add double_to_int() bif 2021-12-13 10:46:14 -07:00
Tim Wojtulewicz
f2b74d21f7 Merge remote-tracking branch 'origin/topic/timw/null-checks'
* origin/topic/timw/null-checks:
  Check for sets before attempting to check for same Yield types
  Add early bail-outs to same_type()
  Fix types for Analyzer::register_for_port(s) to be the same
2021-12-13 09:41:48 -07:00
Vern Paxson
a10fafe398 tweak to keep clang-format happy 2021-12-12 12:43:30 -08:00
Vern Paxson
3fc58bdd9f minor note regarding improving performance of C++-generated code 2021-12-12 12:42:16 -08:00
Vern Paxson
86288426fb factoring of generating C++ initializations, no semantic changes 2021-12-12 12:41:16 -08:00
Vern Paxson
2f7137999f restored support for incremental compilation of scripts to C++ 2021-12-12 12:36:45 -08:00
Vern Paxson
3b3cea799b fixes for -O gen-standalone-C++ 2021-12-12 12:36:08 -08:00
Vern Paxson
d5e7db1070 new ZEEK_FILE_ONLY and ZEEK_FUNC_ONLY environment variables for debugging script optimization - replaces ZEEK_ONLY 2021-12-12 12:34:50 -08:00
Vern Paxson
96ed944571 fix for compiling record constructors to C++ 2021-12-12 12:34:23 -08:00
Vern Paxson
52ed9351a9 fixes for compiling vector operations to C++ 2021-12-12 12:33:38 -08:00
Vern Paxson
4ea5785908 fixed for profiling missing some profile elements 2021-12-12 12:33:06 -08:00
Vern Paxson
ce7f886077 removing vestigial methods 2021-12-12 12:32:27 -08:00
Vern Paxson
bbae2cf36c minor efficiency tweak for ZAM record construction 2021-12-12 12:31:28 -08:00