Commit graph

2460 commits

Author SHA1 Message Date
Robin Sommer
691b099de1 Merge remote-tracking branch 'origin/topic/awelzel/2120-logdir-leftover'
* origin/topic/awelzel/2120-logdir-leftover:
  sqlite default-logdir test: Remove ls ./logs baseline
  logging/sqlite: Recognize Log::default_logdir and place files there if set
  logging: Introduce Log::default_logdir deprecate LogAscii::logdir and per writer logdir
  logging/ascii: Fix .shadow paths when using LogAscii::logdir
2022-07-07 08:06:13 +02:00
Arne Welzel
aaa47a709c logging: Introduce Log::default_logdir deprecate LogAscii::logdir and per writer logdir
Also modify FormatRotationPath to keep rotated logs within
Log::default_logdir unless the rotation function explicitly
set dir, e.g. by when the user redef'ed default_rotation_interval.
2022-07-06 18:54:29 +02:00
Johanna Amann
6e1e6fefe5 Merge remote-tracking branch 'origin/topic/johanna/2198'
* origin/topic/johanna/2198:
  SSL/GH-2211: Address review feedback, remove USE_FLIPPED
  SSL Analyzer: track connection direction by messages
2022-07-05 15:09:34 +01:00
Tim Wojtulewicz
fb16ce3711 Remove other general deprecations 2022-06-30 19:17:13 +00:00
Tim Wojtulewicz
70e63d4749 Remove deprecated MemoryAllocation() methods and related code 2022-06-30 18:56:52 +00:00
Johanna Amann
e14eddeb97 SSL Analyzer: track connection direction by messages
This PR changes the way in which the SSL analyzer tracks the direction
of connections. So far, the SSL analyzer assumed that the originator of
a connection would send the client hello (and other associated
client-side events), and that the responder would be the SSL servers.

In some circumstances this is not true, and the initiator of a
connection is the server, with the responder being the client. So far
this confused some of the internal statekeeping logic and could lead to
mis-parsing of extensions.

This reversal of roles can happen in DTLS, if a connection uses STUN -
and potentially in some StartTLS protocols.

This PR tracks the direction of a TLS connection using the hello
request, client hello and server hello handshake messages. Furthermore,
it changes the SSL events from providing is_orig to providing is_client,
where is_client is true for the client_side of a connection. Since the
argument positioning in the event has not changed, old scripts will
continue to work seamlessly - the new semantics are what everyone
writing SSL scripts will have expected in any case.

There is a new event that is raised when a connection is flipped. A
weird is raised if a flip happens repeatedly.

Addresses GH-2198.
2022-06-24 18:35:44 +01:00
Tim Wojtulewicz
a9fd4a60e0 GH-1991: Add option to limit the number of tunnel_changed events 2022-06-22 09:39:42 -07:00
Christian Kreibich
edef3736fb Additional &is_used tags in the Netcontrol and Sumstats frameworks
When running a cluster, these functions only get called in select node types and
could trigger no-caller warnings on stderr.
2022-06-02 22:57:07 -07:00
Tim Wojtulewicz
535a6013aa Merge remote-tracking branch 'zeek-as-org/as-org'
* zeek-as-org/as-org:
  Mark lookup_asn() BIF as deprecated in v6.1
  Define geo_autonomous_system record type
  Add lookup_autonomous_system() BIF that returns AS number and org
2022-06-02 16:59:29 -07:00
Christian Kreibich
1cebdd569d Merge branch 'topic/christian/gh-2134-fix-intel-test-races'
* topic/christian/gh-2134-fix-intel-test-races:
  Expand scripts.base.frameworks.intel.cluster-transparency test
  Fix races in scripts.base.frameworks.intel.cluster-transparency-with-proxy test
  Add Intel::send_store_on_node_up boolean to control min_data_store delivery
2022-06-02 12:20:06 -07:00
Robin Sommer
d99f041ac5
Add WebSocket support for exchanging events with external clients.
This exposes Broker's new WebSocket support in Zeek. To enable it,
call `Broker::listen_websocket()`. Zeek will then start listening on
port 9997 for incoming WebSocket connections.

See the Broker documentation for a description of the message format
expected over these WebSocket connections.
2022-06-02 10:31:52 +02:00
Phil Rzewski
c08fe7c237 Define geo_autonomous_system record type 2022-06-01 18:39:07 -07:00
Christian Kreibich
892a3a8452 Add Intel::send_store_on_node_up boolean to control min_data_store delivery
This adds a redefinable const to the internals of the Intel framework, to allow
suppression of the manager sending its current min_data_store when a worker
connects. This feature is desirable for nodes that check in "late" to bring them
up to speed, but during testing it introduces nondeterminism.
2022-06-01 17:45:19 -07:00
Christian Kreibich
93bed5a261 Merge branch 'topic/christian/node-status-notification'
* topic/christian/node-status-notification:
  Add Supervisor::node_status notification event
2022-05-31 12:53:18 -07:00
Christian Kreibich
14188fc7a7 Add Supervisor::node_status notification event
The Supervisor generates this event every time it receives a status update from
the stem, meaning a node got created or re-created. A corresponding
SupervisorControl::node_status event relays the same information for users
interacting with the Supervisor over Broker.
2022-05-30 21:36:35 -07:00
Vern Paxson
07cf5cb089 deprecation messages for unused base script functions 2022-05-27 14:36:30 -07:00
Vern Paxson
6dc711c39e annotate orphan base script components with &deprecated 2022-05-26 17:39:17 -07:00
Vern Paxson
9b8ac44169 annotate base scripts with &is_used as needed 2022-05-26 17:39:17 -07:00
Tim Wojtulewicz
f8bc23d3e1 Propagate BPF_Program error message to script land 2022-05-25 09:41:35 -07:00
Christian Kreibich
d4ecfa0a67 Merge branch 'topic/christian/installation-dirs-in-scriptland'
* topic/christian/installation-dirs-in-scriptland:
  Add scripts.base.misc.installation btest
  Add base/misc/installation.zeek, with Zeek installation directories
  Ensure presence of Zeek-related directories in toplevel CMakeLists.txt
2022-05-24 12:12:05 -07:00
Christian Kreibich
84a09debe3 Add base/misc/installation.zeek, with Zeek installation directories
This makes several of the installation's main directories available to the
script layer.
2022-05-23 14:16:59 -07:00
Christian Kreibich
9d59a48ae2 Expand build_path() function to handle empty dir arguments gracefully
When passing an empty string as a directory, the function would produce
filenames starting with a slash even when the given file_name is not an absolute
path. Defaulting to the root directory is likely never intended and might
conveivably be dangerous. The middle "/" is now skipped also if dir is an empty
string.
2022-05-19 09:45:52 -07:00
Tim Wojtulewicz
8b0263cb39 Merge remote-tracking branch 'origin/topic/vern/script-profiling'
* origin/topic/vern/script-profiling:
  tidy up after generating profile
  test suite updates for refined script coverage, use of new BiF to speed startup
  fix for coverage reporting for functions that use "when" statements
  new global_options() BiF to speed up startup, plus a micro-preen
  hooks for new --profile-scripts option
  classes for managing script profiles
  address some holes in script coverage
  fix for script coverage missing on-exit activity
  memory management fixes for loggers
  make curr_CPU_time() broadly available rather than just isolated to ZAM
2022-05-11 12:56:41 -07:00
Johanna Amann
3a7b127a71 Merge remote-tracking branch 'origin/topic/johanna/topk-cite'
* origin/topic/johanna/topk-cite:
  Add exact name of the Top-k algorithm.
2022-05-11 18:45:48 +01:00
Johanna Amann
25c33d2a29 Add exact name of the Top-k algorithm.
I needed to figure out which exact algorithm we use for our
probabilistic top-k measurements. It turns out that we do not mention
this in our source tree at all so far.
2022-05-11 13:21:26 +01:00
Vern Paxson
fab4905fc2 new global_options() BiF to speed up startup, plus a micro-preen 2022-05-03 11:13:15 -07:00
Johanna Amann
a26a915a63 Update X.509 and CT trust anchors 2022-04-28 13:58:29 +01:00
Tim Wojtulewicz
d0e13a1e79 Merge remote-tracking branch 'origin/topic/vern/table-attr-fixes'
* origin/topic/vern/table-attr-fixes:
  updates for btests - new cases to check, new baselines
  updates for btests - new cases to check, new baselines
  fix for ill-formed (complex) &default function
  type-checking for use of empty table constructors in expressions
  catch empty constructors used for type inference suppress repeated error messages
  factoring to make checking of &default attributes externally accessible
  bug fix for empty table constructors with &default attributes (plus a typo)
2022-04-05 18:05:15 -07:00
Christian Kreibich
669cfbe109 Correct origin documentation of the version field in the HTTP log. 2022-04-04 14:22:58 -07:00
Vern Paxson
24be03f7c5 fix for ill-formed (complex) &default function 2022-03-31 19:31:21 -07:00
Vern Paxson
f62c04c9eb tweaks to base scripts revealed by switch to expression-based initialization 2022-03-23 15:36:35 -07:00
Johanna Amann
f02079e3c8 Improve the formatting of the SSL::Info::ssl_history documentation 2022-03-23 10:03:45 +01:00
Vern Paxson
c77f543a17 fix base scripts to include mandatory fields in record constructors 2022-02-28 15:33:46 -08:00
Tim Wojtulewicz
08bf6f13e2 GH-1949: Remove unused timer_mgr_inactivity_timeout global 2022-02-22 15:16:36 -07:00
Christian Kreibich
6a46d1584c Add capture to a Sumstats when-statement
This resolves a deprecation warning that currently triggers when running
Sumstats in clusterized Zeek.
2022-02-04 12:54:34 -08:00
Johanna Amann
95f1565498 Match DPD TLS signature on one-sided connections.
This commit changes DPD matching for TLS connections. A one-sided match
is enough to enable DPD now.

This commit also removes DPD for SSLv2 connections. SSLv2 connections do
basically no longer happen in the wild. SSLv2 is also really finnicky to
identify correctly - there is very little data required to match it, and
basically all matches today will be false positives. If DPD for SSLv2 is
still desired, the optional signature in policy/protocols/ssl/dpd-v2.sig
can be loaded.

Fixes GH-1952
2022-02-01 16:51:21 +00:00
Tim Wojtulewicz
3d9d6e953b Merge remote-tracking branch 'origin/topic/vern/when-lambda'
* origin/topic/vern/when-lambda:
  explicitly provide the frame for evaluating a "when" timeout expression
  attempt to make "when" btest deterministic
  tests for new "when" semantics/errors
  update existing test suite usage of "when" statements to include captures
  update uses of "when" in base scripts to include captures
  captures for "when" statements update Triggers to IntrusivePtr's and simpler AST traversal introduce IDSet type, migrate associated "ID*" types to "const ID*"
  logic (other than in profiling) for assignments that yield separate values
  option for internal use to mark a function type as allowing non-expression returns
  removed some now-obsolete profiling functionality
  minor commenting clarifications
2022-01-14 14:41:42 -07:00
Vern Paxson
98cd3f2213 update uses of "when" in base scripts to include captures 2022-01-07 14:53:33 -08:00
Robin Sommer
a7427e95bf
Switch to recording unmodified HTTP header.
We used to attempt to remove any port specification before recording
HTTP host headers in logs. Doing so would (1) remove potentially useful
information, (2) not match what the documentation seemed to suggest, and
(3) fail for IP6 addresses containing colons.

We now record the original HOST header as is.

Addresses #1844.
2021-12-21 21:54:47 +01:00
Vern Paxson
72a59bf828 removed unused script variable 2021-12-14 12:49:27 -08:00
Tim Wojtulewicz
8816547964 Fix types for Analyzer::register_for_port(s) to be the same 2021-12-10 17:48:19 +00:00
Tim Wojtulewicz
248325e301 Fix ethertype for ARP in Geneve forwarding rules 2021-12-09 14:58:08 -07:00
Christian Kreibich
1aaed1cc2e Add LogAscii::json_include_unset_fields flag to control unset field rendering
The flag controls whether JSON rendering includes unset &optional log fields
(F, the default), or includes them with a null value (T).
2021-12-08 17:29:07 -08:00
Tim Wojtulewicz
368dec8372 GH-1764: Update mappings for Geneve analyzer to IP4/IP6/ARP 2021-12-06 12:26:16 -07:00
Tim Wojtulewicz
e82a78616b Update NEWS and some minor fixes for docs/zeekygen 2021-11-23 19:39:36 -07:00
Tim Wojtulewicz
2044fbe53b Add GTPv1 packet analyzer, disable old analyzer 2021-11-23 19:36:50 -07:00
Tim Wojtulewicz
dc0ecf9811 Add Teredo packet analyzer, disable old analyzer 2021-11-23 19:36:50 -07:00
Tim Wojtulewicz
05574ecce1 Add VXLAN packet analyzer, disable old analyzer 2021-11-23 19:36:50 -07:00
Tim Wojtulewicz
cbb0bcd49c Add Geneve packet analyzer, disable old analyzer 2021-11-23 19:36:50 -07:00
Tim Wojtulewicz
7e40094f2c Add AYIYA packet analyzer, disable old analyzer 2021-11-23 19:36:50 -07:00