Commit graph

2800 commits

Author SHA1 Message Date
Jan Grashoefer
0c06c604ab Add logging of disabled analyzers to analyzer.log 2024-07-09 18:22:43 +02:00
Christian Kreibich
fa6361af56 Management framework: propagate metrics port from agent
This propagates the metrics port from the node config passed through the
supervisor all the way into the script layer.
2024-07-08 23:05:24 -07:00
Christian Kreibich
563704a26e Management framework: add metrics port in management & Supervisor node records
This allows setting a metrics port for creation in new nodes.
2024-07-08 23:05:24 -07:00
Christian Kreibich
3ecacf4f50 Comment-only tweaks for telemetry-related settings.
These weren't quite accurate any more.
2024-07-08 23:05:24 -07:00
Christian Kreibich
737b1a2013 Remove the Supervisor's internal ClusterEndpoint struct.
This eliminates one place in which we currently need to mirror changes to the
script-land Cluster::Node record. Instead of keeping an exact in-core equivalent, the
Supervisor now treats the data structure as opaque, and stores the whole cluster
table as a JSON string.

We may replace the script-layer Supervisor::ClusterEndpoint in the future, using
Cluster::Node directly. But that's a more invasive change that will affect how
people invoke Supervisor::create() and similars.

Relying on JSON for serialization has the side-effect of removing the
Supervisor's earlier quirk of using 0/tcp, not 0/unknown, to indicate unused
ports in the Supervisor::ClusterEndpoint record.
2024-07-02 14:52:17 -07:00
Christian Kreibich
a98ec6b08b Provide a script-layer equivalent to Supervisor::__init_cluster().
If the script layer is able to access the current node's config via
Supervisor::node(), it can handle populating Cluster::nodes. That code
is much more straightforward than an equivalent in-core implementation
(especially with the upcoming change to the cluster table's implementation).
This introduces base/frameworks/cluster/supervisor.zeek and
Cluster::Supervisor::__init_cluster_nodes() for that purpose.

The @load of the Supervisor API in cluster/main.zeek isn't technically
necessary since we already load it explicitly even in init-bare.zeek,
but being explicit seems better.
2024-07-02 14:52:13 -07:00
Tim Wojtulewicz
d549e3d56a Add Telemetry::metrics_address option 2024-06-07 09:28:27 -07:00
Tim Wojtulewicz
99e64aa113 Restore label_names field in MetricOpts record 2024-06-04 14:14:58 -07:00
Tim Wojtulewicz
433c257886 Move telmetry label names out of opts records, into main metric records 2024-06-04 14:14:58 -07:00
Tim Wojtulewicz
93717ca8f8 Remove is_sum arguments from counters and gauges 2024-05-31 13:36:37 -07:00
Tim Wojtulewicz
46ff48c29a Change all instruments to only handle doubles 2024-05-31 13:36:37 -07:00
Tim Wojtulewicz
e3e806ca23 Remove all of the ZEEK_METRICS_ environment variables 2024-05-31 13:36:37 -07:00
Tim Wojtulewicz
9fb952a5f3 Regenerate docs [nomail] 2024-05-31 13:30:32 -07:00
Tim Wojtulewicz
53c3d2032a Remove the is_sum argument from BIF histogram creation methods 2024-05-31 13:30:31 -07:00
Tim Wojtulewicz
4361880e09 Remove Telemetry::metrics_export_prefixes option 2024-05-31 13:30:31 -07:00
Tim Wojtulewicz
e195d3d778 Fix some determinism issues with btests 2024-05-31 13:30:31 -07:00
Tim Wojtulewicz
84aa308527 Rework everything to access the prometheus-cpp objects more directly 2024-05-31 13:30:31 -07:00
Tim Wojtulewicz
128bf3fe9f Remove Broker metrics configuration values and methods 2024-05-31 13:30:31 -07:00
Tim Wojtulewicz
e93e4cc26d Add a services.json endpoint for Prometheus service discovery 2024-05-31 13:30:31 -07:00
Tim Wojtulewicz
4718e5cf00 Remove everything related to aggregation 2024-05-31 13:30:31 -07:00
Tim Wojtulewicz
643bb38419 Make all options const-redefs, remove all of the change handler code 2024-05-31 13:30:31 -07:00
Tim Wojtulewicz
17d09c657b Move base types from telemetry framework to init-bare 2024-05-31 13:30:31 -07:00
Tim Wojtulewicz
97a35011a7 Add necessary script-land changes 2024-05-31 13:30:31 -07:00
Johanna Amann
03b358f6d1 Merge branch 'files_pe_timestamp_sync' of https://github.com/mvhensbergen/zeek
* 'files_pe_timestamp_sync' of https://github.com/mvhensbergen/zeek:
  Don't hardcode values
  Add btest for timestamp check
  Copy timestamp from file object
2024-05-29 14:16:31 +01:00
Christian Kreibich
3d88918071 Fix Zeekygen warning for QUIC::unrecognized_version event
Docs updates have been warning about being unable to locate that identifier, and
it's because we've not defined it alongside the other QUIC events.
2024-05-24 14:30:42 -07:00
Johanna Amann
34225e83ba Update TLS consts, mainly new named curves.
Add test for X25519Kyber768Draft00 (post-quantum key agreement)
2024-05-23 14:50:36 +01:00
Vern Paxson
74bf453d6d Fix for suppressing SMB logging of previously-logged files 2024-05-18 14:13:52 -07:00
mvhensbergen
a4f73ee45f
Copy timestamp from file object
In some cases, e.g. running zeek on short pcaps as opposed to continuous packet streams, network_time() may not equal the time that was used when generating the file object.

This results in the pe.log entry having a different timestamp than its corresponding files.log entry which is strange as they refer to the exact same file.
2024-05-17 15:03:06 +02:00
Tim Wojtulewicz
29f5a49baf Merge remote-tracking branch 'origin/topic/christian/private_address_clarification'
* origin/topic/christian/private_address_clarification:
  Clarify membership in the Site::private_address_space prefix list. [skip ci]
2024-05-07 10:26:50 -07:00
Tim Wojtulewicz
6821a41c4e Move the options from policy/tuning/defaults to actual Zeek defaults, deprecate that package 2024-05-06 11:13:04 -07:00
Christian Kreibich
0554951cb6 Clarify membership in the Site::private_address_space prefix list. [skip ci]
IANA's IPv6 special-purpose address registry now has members that technically
meet the definition of not being globally reachable, but don't imply operating
locally. An example: https://datatracker.ietf.org/doc/draft-ietf-6man-sids/06/

This change just explains that distinction.
2024-05-04 11:21:26 -07:00
Johanna Amann
2217eab38a Fix cid propagation into files.log
Changes to the connection id were not propagated to files.log in all
cases.

Fixes GH-3700
2024-04-29 14:13:19 +01:00
Vern Paxson
263093af78 flag base script as being known to ZAM optimization 2024-04-25 09:15:13 -07:00
Vern Paxson
c11c2830b1 performance speed-up for SMB base scripts 2024-04-25 09:15:12 -07:00
Tim Wojtulewicz
d566961da5 Merge remote-tracking branch 'pbcullen/topic/pbcullen/remove_exclude'
* pbcullen/topic/pbcullen/remove_exclude:
  Update core.pcap.filter-warning baseline
  Add PacketFilter::remove_exclude function
2024-04-18 09:03:57 -07:00
Peter Cullen
81856cc391 Add PacketFilter::remove_exclude function
This adds a public function to the PacketFilter framework that
allows the caller to remove an exclude filter by filter id.
2024-04-17 21:25:35 +00:00
Peter Cullen
187f93cb96 Parse and store localversion string
Add localversion to the VersionDescription record and populate it
during version string parsing.
This change also modifies the version string syntax, removing the
deprecated dash (-) between beta|dev|rc and the commmit count; those
must now be separated by a period.
The test version strings were updated accordingly along with the
baseline.
2024-04-17 14:17:22 -07:00
Peter Cullen
e4942d45b9 Modify version parsing for localversion
The regex used for validating version strings was updated to allow
for an optional localversion component. The piece of the version string
also needs to be removed before parsing a possible commit number.
2024-04-17 14:17:05 -07:00
Arne Welzel
d728981b77 Merge remote-tracking branch 'origin/topic/awelzel/move-iso-9660-sig-to-policy'
* origin/topic/awelzel/move-iso-9660-sig-to-policy:
  signatures/iso-9660: Add \x01 suffix to CD001
  test-all-policy: Do not load iso-9660.zeek
  signatures: Move ISO 9660 signature to policy
2024-02-26 21:16:48 +01:00
Arne Welzel
77d57fccfd Merge remote-tracking branch 'origin/topic/awelzel/find-in-zeekpath'
* origin/topic/awelzel/find-in-zeekpath:
  NEWS: Mention can_load() and find_in_zeekpath()
  utils: Introduce packages.zeek with can_load() helper
  zeek.bif: Add find_in_zeekpath() helper
2024-02-26 21:14:59 +01:00
Arne Welzel
1a5ce65e3d signatures: Move ISO 9660 signature to policy
The previous "fix" caused significant performance degradation without
the signature ever having a chance to trigger. Moving it to policy
seems the best compromise, the alternative being outright removing it.
2024-02-26 13:35:23 +01:00
Johanna Amann
c81274b549 Update Mozilla CA and Google CT list and related tests.
Test updates are necessary due to removed CT logs.

Fixes GH-3620
2024-02-23 11:50:31 +00:00
Arne Welzel
d2409dd432 signatures: Fix ISO 9960 signature
This signature only really works when default_file_bof_buffer_size is bumped
to a sufficient value (40k).
2024-02-22 12:37:40 +01:00
Arne Welzel
036bcfe919 Merge remote-tracking branch 'origin/topic/awelzel/ftp-clear-fuid-after-logging'
* origin/topic/awelzel/ftp-clear-fuid-after-logging:
  testing: Update zeek-testing-private commit hash
  ftp: Reset fuid after logging
2024-02-22 12:23:21 +01:00
Arne Welzel
947294efab utils: Introduce packages.zeek with can_load() helper 2024-02-22 11:41:50 +01:00
Arne Welzel
31b548babc ftp: Reset fuid after logging
A user reported being confused about the fuid association of subsequent
FTP commands when a data transfer has completed. It seems reasonable to
unset fuid upon logging a FTP command which had a fuid.

The current behavior results in the PORT or PASV commands after a RETR or STOR
to have the fuid of the prior file transfer. Similarly, any CWD or DEL commands
following a file transfer will unnecessarily be logged with the fuid of the
prior file transfer.

This tickles the baselines for the private testing PCAP a lot, primarily
because there data connections in that pcap are never established properly.
E.g, the fuids FzDzid1Dxm9srVKHXf and FEfYX73q5C6GEQZXX9 have been re-used
for multiple commands.

This may look like we're losing information, but the fuids vanishing
in the normal btests belong to a LIST command that isn't logged by
default into ftp.log. If it was, the fuid would be attached to it.
2024-02-21 12:41:32 +01:00
Arne Welzel
c1a685a05d websocket: Add Spicy parser version, too.
The Spicy analyzer is added as a child analyzer when enabled and the
WebSocket.cc logic dispatches between the BinPac and Spicy version.

It substantially slower when tested against a somewhat artificial
2.4GB PCAP. The first flamegraph indicates that the unmask() function
stands out with 35% of all samples, and above it shared_ptr samples.
2024-02-06 17:29:55 +01:00
Johanna Amann
160ccda45f Merge remote-tracking branch 'origin/topic/johanna/netcontrol-updates'
* origin/topic/johanna/netcontrol-updates:
  Netcontrol: add rule_added_policy
  Netcontrol: more logging in catch-and-release
  Netcontrol: allow supplying explicit name to Debug plugin
2024-02-06 11:54:26 +00:00
Christian Kreibich
873d734c79 Do not default PoolSpec topics to the empty string.
Similar to `node_topic`, we already spell out a topic in the existing use and
there's no obviously meaningful default value.
2024-02-05 18:03:08 -08:00
Christian Kreibich
8437012346 Do not default to proxy nodes in Broker::PoolSpec
This requires pool creation to spell out a spec explicitly, which the only code
using these types already does. There's no reason for pools to automatically
refer to proxies.
2024-02-05 17:51:11 -08:00