Commit graph

3015 commits

Author SHA1 Message Date
Christian Kreibich
46a11ec37d Add Cluster::nodeid_to_node() helper function
This translates backend-specific node identifiers (like Broker IDs) to
cluster nodes and their names, if available.
2024-12-06 15:18:05 -08:00
Christian Kreibich
0010e65f6d Support re-peering with Broker peers that fall behind
This adds re-peering at the Broker level for peers that Broker decided to
unpeer. We keep this at the Broker level since this behavior is specific to
it (as opposed to other cluster backends).

Includes baseline updates for btests that pick up on the new script's @load.
2024-12-06 15:18:05 -08:00
Dominik Charousset
4c4eb4b8e2 Add Zeek-level configurability of Broker slow-peer disconnects 2024-12-06 15:18:05 -08:00
Christian Kreibich
e81856a4af No need to namespace Cluster:: functions in their own namespace 2024-12-06 15:18:05 -08:00
Tim Wojtulewicz
bbd7f56dcc Add signatures for Python bytecode for 3.8-3.14 2024-12-06 13:45:46 -07:00
Arne Welzel
e773fe0506 init-bare: Deprecate tcp_reassembler_ports
...and remove from NetVar, seems left-over.
2024-12-05 16:18:11 +01:00
Arne Welzel
b3080f2652 init-bare/zeek-setup: Groundwork for instantiating cluster backends 2024-12-04 12:40:35 +01:00
Johanna Amann
7b582bc345 Merge remote-tracking branch 'origin/topic/johanna/sqlite-pragmas'
* origin/topic/johanna/sqlite-pragmas:
  Options for SQLite log writer, eliminate duplicate definitions
  Test synchronous/journal mode options for SQLite log writer
  Added default options for synchronous and journal mode
  Support for synchronous and journal_mode
2024-11-27 08:32:08 +00:00
Johanna Amann
d592942ccb Test synchronous/journal mode options for SQLite log writer
Also adds some small tweaks and adds the new feature to NEWS.
2024-11-26 12:26:38 +00:00
Arne Welzel
fc12be1f17 cluster/setup-connections: Switch to Cluster::subscribe(), short-circuit broker
For the time being, this is easiest, otherwise we'd need to
conditionally load a broker-specific policy script based on
Cluster::backend being set.
2024-11-26 12:58:23 +01:00
Arne Welzel
ef04a199c8 cluster: Add Cluster scoped bifs
... and a broker based test using Cluster::publish() and
Cluster::subscribe().
2024-11-26 12:58:23 +01:00
Arne Welzel
c7c96610e3 init-bare/zeek-setup: Add Cluster::backend const &redef 2024-11-26 12:13:14 +01:00
Mymaqn
3ca56f7e0f Added default options for synchronous and journal mode
Added enum options SQLITE_SYNCHRONOUS_DEFAULT and SQLITE_JOURNAL_MODE_DEFAULT
and changed the default to be these instead.
2024-11-26 11:08:30 +00:00
Mymaqn
6e026ba313 Support for synchronous and journal_mode 2024-11-26 11:08:18 +00:00
Arne Welzel
97f05b2f8c Merge remote-tracking branch 'origin/topic/awelzel/pluggable-cluster-backends-part1'
* origin/topic/awelzel/pluggable-cluster-backends-part1:
  btest: Test Broker::make_event() together with Cluster::publish_hrw()
  btest: Add cluster dir, minimal test for enum value
  broker: Add shim plugin adding a backend component
  zeek-setup: Instantiate backend::manager
  cluster: Add to src/CMakeLists.txt
  cluster: Add Components and ComponentManager for new components
  cluster/Backend: Interface for cluster backends
  cluster/Serializer: Interface for event and log serializers
  logging: Introduce logging/Types.h
  SerialTypes/Field: Allow default construction and add move constructor
  DebugLogger: Add cluster debugging stream
  plugin: Add component enums for pluggable cluster backends
  broker: Pass frame to MakeEvent()
2024-11-22 12:53:23 +01:00
Arne Welzel
fb23a06f6f cluster/Backend: Interface for cluster backends 2024-11-22 10:43:50 +01:00
Arne Welzel
91f5945f92 sumstat/non-cluster: Move last epoch processing to zeek_done()
@Sheco reported that standalone epoch processing may exclude scheduled
events when the final sumstat epoch runs before. For example, this easily
happens when attempting to do sumstat observations within connection_state_remove().

Delay final epoch processing to zeek_done() instead.

This doesn't deal with the clustered version - this would need something
more elaborate and potentially a mechanism to delay the shutdown of
other cluster nodes until/after sumstat processing completed.
2024-11-18 15:58:01 +01:00
Arne Welzel
18bfdb8a2b Merge remote-tracking branch 'origin/topic/awelzel/deprecate-broker-auto-publish'
* origin/topic/awelzel/deprecate-broker-auto-publish:
  sumstats: Remove copy() for Broker::publish() calls
  broker/Publish: Use event time instead of network time
  broker/Eventhandler: Deprecate Broker::auto_publish() for v8.1
  btest: Remove Broker::auto_publish() usages
  frameworks/control: Remove Broker::auto_publish()
  catch-and-release: Remove Broker::auto_publish()
  ssl/validate-certs: Remove Broker::auto_publish()
  sumstats: Remove Broker::auto_publish()
  cluster_started: No Broker::auto_publish() use
  openflow: Remove Broker::auto_publish()
  dhcp: Remove Broker::auto_publish()
  frameworks/notice: Remove Broker::auto_publish()
  netcontrol: Replace Broker::auto_publish()
  intel: Switch to Cluster::publish()
  broker: Support publish() of unspecified set() / table()
  types: Fix table() resulting in table_type->IsSet() == true
2024-11-14 14:17:13 +01:00
Arne Welzel
aabc4a4114 sumstats: Remove copy() for Broker::publish() calls
Serialization happens immediately at Broker::publish() time, there
should be no caching issues.
2024-11-14 12:59:22 +01:00
Arne Welzel
6abb9d7eda broker/Eventhandler: Deprecate Broker::auto_publish() for v8.1
Relates to #3637
2024-11-14 12:59:22 +01:00
Arne Welzel
883ae3694c sumstats: Remove Broker::auto_publish() 2024-11-14 12:59:22 +01:00
Arne Welzel
b32153037a openflow: Remove Broker::auto_publish() 2024-11-14 12:59:22 +01:00
Arne Welzel
cb10852f99 dhcp: Remove Broker::auto_publish()
This isn't prettier, but neither worse IMO. A test would be good.
2024-11-14 12:59:22 +01:00
Arne Welzel
08f2198d3e frameworks/notice: Remove Broker::auto_publish() 2024-11-14 12:59:22 +01:00
Tim Wojtulewicz
e33aee8ca2 Move IP protocol names table out of policy script to init-bare 2024-11-13 14:08:30 -07:00
Tim Wojtulewicz
fd67206865 Minor review nits 2024-11-13 14:08:30 -07:00
Tim Wojtulewicz
5e5aceb6f7 Rename protocol_id field to ip_proto and similar renaming for name field 2024-11-13 12:02:00 -07:00
Tim Wojtulewicz
d0896e81d6 Increase size of proto fields to uint16_t, add common default value 2024-11-13 11:25:46 -07:00
Tim Wojtulewicz
35ec9733c0 Add conn.log entries for connections with unhandled IP protocols 2024-11-13 11:25:40 -07:00
Arne Welzel
219d621234 netcontrol: Replace Broker::auto_publish()
I'd think we could drop the cluster.zeek and non-cluster.zeek and
just unconditionally do the publish(), but keeping it for now.
2024-11-06 15:27:48 +01:00
Arne Welzel
93478a246e intel: Switch to Cluster::publish()
This isn't quite making things a lot nicer, but more explicit.
2024-11-06 15:27:48 +01:00
Arne Welzel
fd0181c2cb Merge branch 'modbus-fc-fixes' of https://github.com/zambo99/zeek
* 'modbus-fc-fixes' of https://github.com/zambo99/zeek:
  Fix issues with Modbus message logging
2024-10-30 14:38:53 +01:00
Emmanuele Zambon
792aacc8e5 Fix issues with Modbus message logging
This commit fixes three issues with Zeek's Modbus message logging:

1 - Some exception responses (e.g., READ_COILS_EXCEPTION) are logged
    twice: once without and once with the exception message.
2 - Some exception responses (e.g., PROGRAM_484_EXCEPTION) are not
    logged.
3 - Some known but reserved function codes (e.g., PROGRAM_UNITY) are
    logged as unk-xxx (e.g., unk-90), while it would be possible to
    log their known name.

To address these inconsistencies, the modbus parser has been updated
to parse all exception responses (i.e., all responses where the MSB
of the function code is set) using the already defined Exception
message.
Also, the Modbus main.zeek script has been updated to consistently
demand logging exception responses to the specialized
modbus_exception event, rather than logging some exception responses
in the modbus_message event and others in the modbus_exception event.
Finally, the main.zeek script has been updated to make sure that
for every known function code, the corresponding exception code was
also present, and the enumeration of known function codes in
consts.zeek has been expanded.

Closes #3984
2024-10-30 13:12:27 +01:00
Christian Kreibich
66173633f4 Merge branch 'topic/christian/telemetry-make-bifs-primary'
* topic/christian/telemetry-make-bifs-primary:
  Telemetry framework: move BIFs to the primary-bif stage
  Minor comment tweaks for init-frameworks-and-bifs.zeek
2024-10-24 07:09:16 -07:00
Arne Welzel
daa358c840 Merge remote-tracking branch 'origin/topic/awelzel/3947-telemetry-hook-scrape'
* origin/topic/awelzel/3947-telemetry-hook-scrape:
  btest/telemetry: Fix "Note compilable" typo
  misc/stats: Add zeek_net_timestamp_seconds
  telemetry/Manager: Remove variant include
  telemetry: Invoke Telemetry::sync() only at scrape/collection time
2024-10-22 19:04:51 +02:00
Arne Welzel
70872673a1 telemetry: Invoke Telemetry::sync() only at scrape/collection time
This stops invoking Telemetry::sync() via a scheduled event and instead
only invokes it on-demand. This makes metric collection network time
independent and lazier, too.

With Prometheus scrape requests being processed on Zeek's main thread
now, we can safely invoke the script layer Telemetry::sync() hook.

Closes #3947
2024-10-22 18:49:11 +02:00
Christian Kreibich
71f7e89974 Telemetry framework: move BIFs to the primary-bif stage
This moves the Telemetry framework's BIF-defined functionalit from the
secondary-BIFs stage to the primary one. That is, this functionality is now
available from the end of init-bare.zeek, not only after the end of
init-frameworks-and-bifs.zeek.

This allows us to use script-layer telemetry in our Zeek's own code that get
pulled in during init-frameworks-and-bifs.

This change splits up the BIF features into functions, constants, and types,
because that's the granularity most workable in Func.cc and NetVar. It also now
defines the Telemetry::MetricsType enum once, not redundantly in BIFs and script
layer.

Due to subtle load ordering issues between the telemetry and cluster frameworks
this pushes the redef stage of Telemetry::metrics_port and address into
base/frameworks/telemetry/options.zeek, which is loaded sufficiently late in
init-frameworks-and-bifs.zeek to sidestep those issues. (When not doing this,
the effect is that the redef in telemetry/main.zeek doesn't yet find the
cluster-provided values, and Zeek does not end up listening on these ports.)

The need to add basic Zeek headers in script_opt/ZAM/ZBody.cc as a side-effect
of this is curious, but looks harmless.

Also includes baseline updates for the usual btests and adds a few doc strings.
2024-10-18 09:56:29 -07:00
Christian Kreibich
acdd7a7934 Minor comment tweaks for init-frameworks-and-bifs.zeek 2024-10-18 09:56:29 -07:00
Benjamin Bannier
cfd66ec6f3 Fix invalid Sphinx directive in docstring
Use of `:zeek::see:..` instead of `:zeek:see:..` caused a Sphinx build
failure which prevented automatic regeneration of docs.
2024-10-15 12:47:39 +02:00
Martin van Hensbergen
0e926afbc1 Honor no plaintext password logging for PASS cmd 2024-09-27 16:45:06 +02:00
Arne Welzel
6bb7b9d726 scripts/base/cluster: Move active node management into node_down()
With the idea of an alternative cluster backend, we should
not maintain Cluster state within low-level Broker events.
2024-09-27 15:32:09 +02:00
Arne Welzel
0d925e935e logging: Dedicated log flush timer
Log flushing is currently triggered based on the threading heartbeat timer
of WriterBackends and the hard-coded WRITE_BUFFER_SIZE 1000.

This change introduces a separate timer that is managed by the logger
manager instead of piggy-backing on the heartbeat timer, as well as a
const &redef for the buffer size.

This allows to modify the log flush frequency and batch size independently
of the threading heartbeat interval. Later, this will allow to re-use the
buffering and flushing logic of writer frontends for non-Broker cluster
backends, too.

One change here is that even frontends that do not have a backend will
be flushed regularly. This is wanted for non-Broker backends and should be
very cheap. Possibly, Broker can piggy back on this timer down the road, too,
rather than using its own script-level timer (see Broker::log_flush()).
2024-09-27 15:30:35 +02:00
Arne Welzel
cf9fe91705 pop3: Prevent unbounded state growth
The cmds list may grow unbounded due to the POP3 analyzer being in
multiLine mode after seeing `AUTH` in a Redis connection, but never
a `.` terminator. This can easily be provoked by the Redis ping
command.

This adds two heuristics: 1) Forcefully process the oldest commands in
the cmds list and cap it at max_pending_commands. 2) Start raising
analyzer violations if the client has been using more than
max_unknown_client_commands commands (default 10).

Closes #3936
2024-09-18 19:05:39 +02:00
Arne Welzel
9d4025804d gtpv1: Replace connection_state_remove() with RemovalHook 2024-09-17 18:15:09 +02:00
Arne Welzel
34956f4ca4 teredo: Replace connection_state_remove() with RemovalHook
Remove overhead of unconditionally calling remove_teredo_connection()
for *every* connection by installing a connection removal hook for only
when state was allocated.
2024-09-17 18:13:44 +02:00
Arne Welzel
85ca59484b postgresql: Initial parser implementation
This adds a protocol parser for the PostgreSQL protocol and a new
postgresql.log similar to the existing mysql.log.

This should be considered preliminary and hopefully during 7.1 and 7.2
with feedback from the community, we can improve on the events and logs.
Even if most PostgreSQL communication is encrypted in the real-world, this
will minimally allow monitoring of the SSLRequest and hand off further
analysis to the SSL analyzer.

This originates from github.com/awelzel/spicy-postgresql, with lots of
polishing happening in the past two days.
2024-09-06 16:10:48 +02:00
Pierre Lalet
88368ae856 http: fix password capture when enabled
The current implementation would only log, if the password contains a
colon, the part before the first colon (e.g., the password
`password:password` would be logged as `password`).

A test has been added to confirm the expected behaviour.
2024-08-28 21:44:39 +02:00
Johanna Amann
d89c0d1ef7 Merge remote-tracking branch 'origin/topic/johanna/fix-ssl2-client-hello-version-parsing'
* origin/topic/johanna/fix-ssl2-client-hello-version-parsing:
  Fix parsing of version field in SSLv2 client hello
2024-08-27 13:31:12 +01:00
Robin Sommer
a2079bcda6
Merge remote-tracking branch 'origin/topic/robin/gh-3881-spicy-ports'
* origin/topic/robin/gh-3881-spicy-ports:
  Spicy: Register well-known ports through an event handler.
  Revert "Remove deprecated port/ports fields for spicy analyzers"
2024-08-23 08:10:02 +02:00
Johanna Amann
a6edbf8bcd Fix parsing of version field in SSLv2 client hello
It turns out that, for probably a long time, we have reported an
incorrect version when parsing an SSLv2 client hello. We always reported
this as SSLv2, no matter which version the client hello actually
contained.

This bug probably went unnoticed for a long time, as SSLv2 is
essentially unused nowadays, and as this field does not show up in the
default logs.

This was found due to a baseline difference when writing the Spicy SSL
analyzer.
2024-08-22 13:14:24 +01:00