Commit graph

2844 commits

Author SHA1 Message Date
Johanna Amann
d1e7134156 Merge remote-tracking branch 'origin/master' into topic/johanna/tls12-decryption 2022-01-05 10:27:55 +00:00
Christian Kreibich
5a72864ae8 Docs/comment pass over the cluster controller framework 2022-01-03 00:31:03 -08:00
Christian Kreibich
ac40d5c5b2 Remove periodic pinging of controller by agents
This changes the agent-controller communication to remove the need for ongoing
pinging of the controller by agents not actively "in service". Instead, agents
now use the notify_agent_hello event to the controller to report only their
identity. The controller puts them into service via an agent_welcome_request/
response pair, and takes them out of service via agent_standby_request/response.

This removes the on_change handler from the set of agents that is ready for
service, because not every change to this set is now a suitable time to
potentially send out the configuration. We now invoke this check explicitly in
the two situations where it's warranted: when a agent reports ready for service,
and when we've received a new configuration.
2021-12-21 16:44:04 -08:00
Christian Kreibich
8463f14a52 Move cluster controller/agent main.zeek scripts into their own modules
This has no practical relevance other than allowing the two to be loaded a the
same time, which some of our (cluster-unrelated) tests require. Absence of
namespacing would trigger symbol clashes at this point.
2021-12-21 14:52:29 -08:00
Christian Kreibich
30db1b3bfb First uses of request state timeouts
This now features support for the test_timeout_request/response events, as
supported by the client, and also adds a timeout event for set_configuration, in
case agents do not respond in time.

Includes corresponding zeek-client submodule bump.
2021-12-21 14:52:29 -08:00
Christian Kreibich
1e823f931e Add expiration mechanism to client request state.
This establishes a timeout controlled via ClusterController::request_timeout,
triggering a ClusterController::Request::request_expired event whenever a
timeout rolls around before request state has been finalized by a request's
normal processing.
2021-12-21 14:52:29 -08:00
Christian Kreibich
fc9679e510 Move get_instances_response event to using a Result record
Includes corresponding zeek-client bump.
2021-12-21 14:52:29 -08:00
Christian Kreibich
1461d56340 Track successful config deployment in cluster controller
This allows us to start returning deployed configurations to the client upon
request.
2021-12-21 14:52:29 -08:00
Christian Kreibich
09d9be3433 Add ClusterController::API::notify_agents_ready event
This changes the basic agent-management model to one in which the configurations
received from the client define not just the data cluster, but also set the set
of acceptable instances. Unless connectivity already exists, the controller will
establish peerings with new agents that listen, or wait for ones that connect to
the controller to check in.

Once all required agents are available, the controller triggers the new
notify_agents_ready event, an agent/controller-level "cluster-is-ready"
event. The controller also uses this event to submit a pending config update to
the now-ready instances.
2021-12-21 14:52:29 -08:00
Christian Kreibich
b57be021b7 Make all globals start with a "g_" prefix
This makes it easier to spot them in code, and is shorter than using explicit
namespacing.
2021-12-21 14:52:28 -08:00
Christian Kreibich
14a8c979c1 Add missing debug() log function to log module's API 2021-12-21 14:52:28 -08:00
Christian Kreibich
a56ee6b9a6 Add separate utility module for controller and agent
We can figure out later whether & where to re-settle helper functions that end
up in there.
2021-12-21 14:52:28 -08:00
Christian Kreibich
ddbd83fee4 Support for dropping instances no longer needed after config updates
This sends such expired instances empty configurations that will cause them to
shut down their remaining data cluster nodes.
2021-12-21 14:52:28 -08:00
Christian Kreibich
8eee5bb3d2 Additional infrastructure for printing types
Also added convenience for instantiating (dummy) configuration records.
2021-12-21 14:52:28 -08:00
Christian Kreibich
5cb44c2f69 Support on-demand peering with agents when receiving new cluster configuration
Prior to this, static configuration needed to be in place to configure the
controller/agent layout. The configuration update can now include new instances
that the controller will connect to, assuming they're instances with a listening
agent.
2021-12-21 14:52:28 -08:00
Christian Kreibich
484f79f599 Expand requests support in the controller
Request records for configuration updates now store the full configuration. The
ClusterController::Request module now provies a to_string() function for
rendering requests to a string.
2021-12-21 14:52:28 -08:00
Christian Kreibich
aceb05099a Whitespace tweaks in cluster controller and agent scripts 2021-12-21 14:52:28 -08:00
Vern Paxson
72a59bf828 removed unused script variable 2021-12-14 12:49:27 -08:00
Tim Wojtulewicz
8816547964 Fix types for Analyzer::register_for_port(s) to be the same 2021-12-10 17:48:19 +00:00
Tim Wojtulewicz
248325e301 Fix ethertype for ARP in Geneve forwarding rules 2021-12-09 14:58:08 -07:00
Christian Kreibich
1aaed1cc2e Add LogAscii::json_include_unset_fields flag to control unset field rendering
The flag controls whether JSON rendering includes unset &optional log fields
(F, the default), or includes them with a null value (T).
2021-12-08 17:29:07 -08:00
Tim Wojtulewicz
368dec8372 GH-1764: Update mappings for Geneve analyzer to IP4/IP6/ARP 2021-12-06 12:26:16 -07:00
Tim Wojtulewicz
e82a78616b Update NEWS and some minor fixes for docs/zeekygen 2021-11-23 19:39:36 -07:00
Tim Wojtulewicz
2044fbe53b Add GTPv1 packet analyzer, disable old analyzer 2021-11-23 19:36:50 -07:00
Tim Wojtulewicz
dc0ecf9811 Add Teredo packet analyzer, disable old analyzer 2021-11-23 19:36:50 -07:00
Tim Wojtulewicz
05574ecce1 Add VXLAN packet analyzer, disable old analyzer 2021-11-23 19:36:50 -07:00
Tim Wojtulewicz
cbb0bcd49c Add Geneve packet analyzer, disable old analyzer 2021-11-23 19:36:50 -07:00
Tim Wojtulewicz
7e40094f2c Add AYIYA packet analyzer, disable old analyzer 2021-11-23 19:36:50 -07:00
Tim Wojtulewicz
44e0760e96 Add PacketAnalyzer::register_for_port(s) functions
These allow packet analyzers to register ports as identifiers to forward from
parent analyzers, while also adding those ports to the now-global
Analyzer::ports table at the same time.
2021-11-23 19:36:50 -07:00
Tim Wojtulewicz
612212568a Add analyzer_confirmation and analyzer_violation events 2021-11-23 19:36:50 -07:00
Tim Wojtulewicz
a7d3cb48ef Add concept of "parent" tag namespaces
This allows us to create an EnumType that groups all of the analyzer
tag values into a single type, while still having the existing types
that split them up. We can then use this for certain events that benefit
from taking all of the tag types at once.
2021-11-23 19:36:49 -07:00
Johanna Amann
14f919895d Add documentation for GH-1829
This adds documentation that clarifies that the `ignore_checksums`
option now also allows IPv4 packets with a length of 0.
2021-11-16 13:51:29 +00:00
Tim Wojtulewicz
a6378531db Remove trailing whitespace from script files 2021-10-20 09:57:09 -07:00
Johanna Amann
fe4e06e8ca TLS decryption: remove payload from ssl_encrypted_data again.
There is no reason to make the payload available in the event - it is
still encrypted.
2021-10-19 17:36:48 +02:00
Johanna Amann
303e84ad86 Merge branch 'master' of https://github.com/FlyingWithJerome/zeek
Merge includes small changes, e.g. fixing the comsumption of remaining
raw data.

* 'master' of https://github.com/FlyingWithJerome/zeek:
  remove excussive fields in dns_svcb_rr
  address code reviews (formatting and type and intrusiveptr)
  newlines at the end of test outputs
  lazy commit
  use tabs in init-bare.zeek
  add svcb test case
  add a dns https test case
  remove test logs
  fix a few syntax errors
  initial commit for SVCB/HTTPS records
2021-10-19 15:03:08 +02:00
Johanna Amann
b8b6ac744e Merge remote-tracking branch 'origin/master' into topic/johanna/tls12-decryption 2021-10-13 10:49:29 +01:00
FlyingWithJerome
605d4024e4 remove excussive fields in dns_svcb_rr 2021-10-12 21:40:56 -04:00
FlyingWithJerome
c957e3e91e address code reviews (formatting and type and intrusiveptr) 2021-10-12 20:36:35 -04:00
FlyingWithJerome
b238cf3dca lazy commit 2021-10-12 17:43:32 -04:00
FlyingWithJerome
33c7fd5fba use tabs in init-bare.zeek 2021-10-12 17:43:32 -04:00
FlyingWithJerome
0849332eb9 fix a few syntax errors 2021-10-12 17:43:32 -04:00
FlyingWithJerome
8fce51bf83 initial commit for SVCB/HTTPS records 2021-10-12 17:43:32 -04:00
Tim Wojtulewicz
0f348ea042 GHI-1766: Remove address from Site::private_address_space that converts into 0.0.0.0/0 2021-09-27 14:24:14 -07:00
Tim Wojtulewicz
e5b163290d Merge remote-tracking branch 'origin/topic/vern/remove-uu'
* origin/topic/vern/remove-uu:
  fix up for linking w/ doc update
  documentation update
  script simplification that removes an unnecessary &is_assigned
  removing -uu functionality and associated script analysis now no longer needed
2021-09-24 10:31:56 -07:00
Robin Sommer
47c35190a4 Sanity-check the method passed into ActiveHTTP.
Reported by Pierre Gaulon.
2021-09-23 12:21:23 +02:00
Tim Wojtulewicz
0a0ed65306 Merge remote-tracking branch 'origin/topic/robin/gh-54-sanitize'
* origin/topic/robin/gh-54-sanitize:
  Sanitize log files names before they go into system().
2021-09-22 12:17:05 -07:00
Tim Wojtulewicz
a49dcc8954 Merge remote-tracking branch 'origin/topic/johanna/dpd-packet-limit'
* origin/topic/johanna/dpd-packet-limit:
  PIA - switch size to int64_t
  Introduce dpd_max_packets
2021-09-22 12:16:56 -07:00
Vern Paxson
385e49491b script simplification that removes an unnecessary &is_assigned 2021-09-22 11:18:52 -07:00
Robin Sommer
74680bf4e6 Merge remote-tracking branch 'origin/topic/justin/software-framework-parse-cache'
* origin/topic/justin/software-framework-parse-cache:
  Restore behavior of Software::register event
  Optimzie software framework version parsing
2021-09-21 18:00:46 +02:00
Robin Sommer
2fc12d5bed Merge branch 'topic/foxds/dce_itype_opnums' of ssh://github.com/fox-ds/zeek
* 'topic/foxds/dce_itype_opnums' of ssh://github.com/fox-ds/zeek:
  Add IType opnum mapping
2021-09-21 17:56:13 +02:00