Commit graph

4584 commits

Author SHA1 Message Date
Jan Grashoefer
c12640b048 Add timestamps to manually published broker events. 2023-05-11 12:53:52 +02:00
Jan Grashoefer
ae152f1777 Annotate scheduled events with intended timestamp. 2023-05-11 12:51:06 +02:00
Jan Grashoefer
fa84d2896a Add timestamp to events. 2023-05-11 12:51:06 +02:00
Arne Welzel
8c0e7cd8bc Merge remote-tracking branch 'origin/topic/neverlord/version-header'
* origin/topic/neverlord/version-header:
  testing/zeek-version-link: Assume nm is there
  Drop dependency for zeek_dynamic_plugin_base
  Fixup ifdef check in Plugin.h
  Update cmake_minimum_required() in test plugins
  testing: Add zeek-version-link tests
  Clean up ZEEK_CONFIG_SKIP_VERSION_H issues

cmake-format during merge
2023-05-10 23:33:11 +02:00
Arne Welzel
ca75834c83 testing/zeek-version-link: Assume nm is there 2023-05-10 17:37:32 +02:00
Arne Welzel
85934ec632 Update cmake_minimum_required() in test plugins 2023-05-10 15:53:13 +02:00
Arne Welzel
460dc16d32 testing: Add zeek-version-link tests 2023-05-10 15:53:10 +02:00
Johanna Amann
33ee2bffa9 DTLS 1.3: Update test baseline 2023-05-10 12:27:27 +00:00
Johanna Amann
527c0dc09f Merge remote-tracking branch 'origin/master' into topic/johanna/dtls13
* origin/master: (35 commits)
  Update doc submodule [nomail] [skip ci]
  Updating submodule(s) [nomail]
  zeek.bif: Add log2() and ceil()
  Use the same rules as cmake submodule to reformat Zeek
  Update cmake submodule after reformat
  Fixup Val.h/Val.cc: Actually move ValFromJSON into zeek::detail
  Implement from_json bif
  Revert "Skip version.h by default for Zeek sources"
  BTest baseline updates for -O gen-C++
  updates to C++ maintenance scripts to better handle uncompilable BTests
  added ZEEK_REPORT_UNCOMPILABLE environment variable for "-O report-uncompilable"
  Skip version.h by default for Zeek sources
  core.network_time.broker: Test reliability improvement
  cluster/supervisor: Multi-logger awareness
  Bump zeek-archiver submodule
  ci: Add public-ecr-vacuum.sh
  Update doc submodule [nomail] [skip ci]
  generate-docs: Only update submodule pointer during scheduled builds
  BTest baseline updates for ZAM
  NTP: Detect out-of-order packets
  ...
2023-05-10 13:02:08 +01:00
Johanna Amann
a8e84c6192 DTLS 1.3: finish implementation, add connection_id extension
This commit adds support for the connection_id extension, adds a trace
that uses DTLS 1.3 connection IDs, and adds parsing for the DTLS 1.3
unified header, in case connection IDs are not used.

In case connection IDs are used, parsing of the DTLS 1.3 unified header
is skipped. This is due to the fact, that the header then contains a
variable length element, with the length of the element not given in the
header. Instead, the length is given in the client/server hello message
of the opposite side of the connection (which we might have missed).

Furthermore, parsing is not of a high importance, since we are not
passing the connection ID, or any of the other parsed values of the
unified header into scriptland.
2023-05-10 11:17:24 +01:00
mAsk°
e08e4a5fee Fix memory leak caused by pattern compilation failure 2023-05-10 05:42:03 +00:00
Tim Wojtulewicz
04d23a8b5b Merge remote-tracking branch 'origin/topic/awelzel/3013-btest-core-network-time-broker'
* origin/topic/awelzel/3013-btest-core-network-time-broker:
  core.network_time.broker: Test reliability improvement
2023-05-09 13:38:52 -07:00
Arne Welzel
6cb391f9a2 zeek.bif: Add log2() and ceil()
Closes #2930, #2931.
2023-05-09 18:57:57 +02:00
Tim Wojtulewicz
84e3f414a7 Use the same rules as cmake submodule to reformat Zeek 2023-05-09 08:31:43 -07:00
Arne Welzel
9330a74fe1 Merge remote-tracking branch 'origin/topic/awelzel/zeek-archiver-multiple-loggers'
* origin/topic/awelzel/zeek-archiver-multiple-loggers:
  cluster/supervisor: Multi-logger awareness
  Bump zeek-archiver submodule
2023-05-09 15:20:53 +02:00
Arne Welzel
264284150b Merge remote-tracking branch 'amazing-pp/topic/fupeng/from_json_bif'
* amazing-pp/topic/fupeng/from_json_bif:
  Implement from_json bif

Minor updates during merge: Moved ValFromJSON into zeek::detail for the
time being, removed gotos, normalized some error messages to lower case,
minimal test extension and added a raw reader input framework test reading
"json lines" as a demo, adding notes about the implicit type
conversions.
2023-05-09 10:36:58 +02:00
Fupeng Zhao
584e68434d Implement from_json bif 2023-05-06 00:42:46 +00:00
Tim Wojtulewicz
49b01c3867 Merge remote-tracking branch 'origin/topic/vern/script-opt-maint.May23'
* origin/topic/vern/script-opt-maint.May23:
  BTest baseline updates for -O gen-C++
  updates to C++ maintenance scripts to better handle uncompilable BTests
  added ZEEK_REPORT_UNCOMPILABLE environment variable for "-O report-uncompilable"
  BTest baseline updates for ZAM
2023-05-05 14:08:28 -07:00
Vern Paxson
832f48382c BTest baseline updates for -O gen-C++ 2023-05-05 12:03:00 -07:00
Arne Welzel
979f2df57d core.network_time.broker: Test reliability improvement
I wasn't able to reproduce this locally, but after looking at
-B main-loop,tm for a bit it dawned that if the manager is sending
ticks too fast, the Broker IO source may consume two ticks in one go
before expiring timers and that would explain the observed baseline
differences.

Solve this by removing the reliance on realtime delays and switch to
a request-reply pattern instead.

Also fix indentation and bogus messages.

Fixes #3013
2023-05-05 14:08:29 +02:00
Arne Welzel
c813872915 cluster/supervisor: Multi-logger awareness
When multiple loggers are configured in a Supervisor controlled cluster
configuration, encode extra information into the rotated filename to
identify which logger produced the log.

This is similar to the approach taken for ZeekControl, re-using the
log_suffix terminology, but as there's only a single zeek-archiver
process and no postprocessors and no other side-channel for additional
information, we encode extra metadata into the filename. zeek-archiver
is extended to recognize the special metadata part of the filename.

This also solves the issue that multiple loggers in a supervisor setup
overwrite each others log files within a single log-queue directory.
2023-05-05 12:27:25 +02:00
Vern Paxson
148673a5c1 BTest baseline updates for ZAM 2023-05-04 10:52:38 -07:00
Arne Welzel
2c8b97c522 NTP: Detect out-of-order packets
The NTP mode provides us with the identity of the endpoints. For the
simple CLIENT / SERVER modes, flip the connection if we detect
orig/resp disagreeing with what the message says. This mainly
results in the history getting a ^ and the ntp.log / conn.log
showing the corrected endpoints.

Closes #2998.
2023-05-04 19:44:02 +02:00
Arne Welzel
12252743b1 Merge remote-tracking branch 'origin/topic/awelzel/smb2-state-handling'
* origin/topic/awelzel/smb2-state-handling:
  NEWS: Add entry about SMB::max_pending_messages and state discarding
  scripts/smb2-main: Reset script-level state upon smb2_discarded_messages_state()
  smb2: Limit per-connection read/ioctl/tree state
2023-05-04 09:40:19 +02:00
Tim Wojtulewicz
8eed46dfa1 Merge remote-tracking branch 'origin/topic/johanna/update-ssl-databases'
* origin/topic/johanna/update-ssl-databases:
  Update Mozilla CA and Google CT lists
2023-05-03 11:13:23 -07:00
Tim Wojtulewicz
7305f2ba9b Merge remote-tracking branch 'origin/topic/timw/2997-ding-dong-bro-init-is-dead'
* origin/topic/timw/2997-ding-dong-bro-init-is-dead:
  Remove parser error message for bro_init, et al
2023-05-03 09:31:26 -07:00
Johanna Amann
d6c4c510ea Add basic DTLSv1.3 support
DTLSv1.3 changes the DTLS record format, introducing a completely new
header - which is a first for DTLS.

We don't currently completely parse this header, as this requires a bit
more statekeeping. This will be added in a future revision. This also
also has little practical implications.
2023-05-03 16:17:31 +01:00
Johanna Amann
e7c9fa1f6e SSL: Fix logging of Hello Retry Requests
It turns out that we never logged hello retry requests correctly in the
ssl_history field.

Hello retry requests are (in their final version) signaled by a specific
random value in the server random.

This commit fixes this oversight, and hello retry requests are now
correctly logged as such.
2023-05-03 13:34:48 +01:00
Johanna Amann
21888a145a SSL: do not try to disable failed analyzer
Currently, if a TLS/DTLS analyzer fails with a protocol violation, we
will still try to remove the analyzer later, which results in the
following error message:

error: connection does not have analyzer specified to disable

Now, instead we don't try removing the analyzer anymore, after a
violation occurred.
2023-05-03 11:16:14 +01:00
Johanna Amann
40a4da94b6 Update Mozilla CA and Google CT lists 2023-05-03 10:46:41 +01:00
Arne Welzel
3ac877e20d scripts/smb2-main: Reset script-level state upon smb2_discarded_messages_state()
This is similar to what the external corelight/zeek-smb-clear-state script
does, but leverages the smb2_discarded_messages_state() event instead of
regularly checking on the state of SMB connections.

The pcap was created using the dperson/samba container image and mounting
a share with Linux's CIFS filesystem, then copying the content of a
directory with 100 files. The test uses a BPF filter to imitate mostly
"half-duplex" traffic.
2023-05-03 11:22:01 +02:00
Tim Wojtulewicz
e07f3f305d Remove parser error message for bro_init, et al 2023-05-02 12:06:41 -07:00
Arne Welzel
df0788252e Merge remote-tracking branch 'origin/topic/awelzel/2994-ip-mf-df-offset'
* origin/topic/awelzel/2994-ip-mf-df-offset:
  ip4_hdr: Add DF, MF, offset and sum fields
2023-05-02 20:45:43 +02:00
Arne Welzel
5570f806ca Merge remote-tracking branch 'origin/topic/awelzel/2991-unload-zeek-package'
* origin/topic/awelzel/2991-unload-zeek-package:
  unload: Fix unloading of packages
2023-05-02 20:28:07 +02:00
Tim Wojtulewicz
81b94423a0 Fix smith-waterman sorting to follow correct Compare semantics 2023-04-30 11:54:08 -07:00
Arne Welzel
df4f90b65c ip4_hdr: Add DF, MF, offset and sum fields
For low-level packet analysis use-cases, these fields are currently
not script-land accessible via raw_packet() or so. They are accessible
on the icmp_context record, but not on the actual ip4_hdr record, so
add them.
2023-04-29 17:09:43 +02:00
Arne Welzel
b677f5b37b unload: Fix unloading of packages
@ynadji found that unloading packages doesn't work due to @unload not
resolving the __load__.zeek file within a directory like @load does.

Fixes #2991
2023-04-28 11:47:55 +02:00
Arne Welzel
3c646b5bf6 Merge remote-tracking branch 'origin/topic/awelzel/record-fields-optional'
* origin/topic/awelzel/record-fields-optional:
  record_fields: Include information about optionality of fields
2023-04-28 10:00:27 +02:00
Arne Welzel
667cdd5c27 Merge remote-tracking branch 'jgras/topic/jgras/cluster-init-tests'
* jgras/topic/jgras/cluster-init-tests:
  Simplify btests using cluster_started event.
  Remove old cluster start tests.

Reverted removal of old cluster start tests during merge.
2023-04-28 09:37:25 +02:00
Arne Welzel
f4bb8fae33 record_fields: Include information about optionality of fields
This was reported as a wish for log schema generation, so add it...
2023-04-27 21:18:35 +02:00
Arne Welzel
004dce2cf2 Merge remote-tracking branch 'origin/topic/awelzel/zeekctl-multiple-loggers'
* origin/topic/awelzel/zeekctl-multiple-loggers:
  NEWS: Add entry for ZeekControl and multi-loggers
  Bump zeekctl to multi-logger version
  logging: Support rotation_postprocessor_command_env
2023-04-27 12:17:02 +02:00
Tim Wojtulewicz
b65697633f Merge remote-tracking branch 'origin/topic/vern/func-coverage'
* origin/topic/vern/func-coverage:
  extend ZEEK_PROFILER_FILE profiling to include summaries for functions/hooks/event handlers
2023-04-26 12:43:00 -07:00
Vern Paxson
5b98538c8b extend ZEEK_PROFILER_FILE profiling to include summaries for functions/hooks/event handlers 2023-04-26 12:23:52 -07:00
Jan Grashoefer
342d88fbd4 Simplify btests using cluster_started event. 2023-04-26 20:00:44 +02:00
Jan Grashoefer
4ddf7562b9 Remove old cluster start tests.
These tests should be redundant to the cluster_started tests found in
policy/frameworks/cluster.
2023-04-26 19:29:55 +02:00
Arne Welzel
8f262df192 Merge remote-tracking branch 'origin/topic/awelzel/bump-af-packet'
* origin/topic/awelzel/bump-af-packet:
  Update AF-Packet submodule
  btest: Add af_packet to TestDirs
  Update AF-Packet submodule
2023-04-26 15:38:17 +02:00
Tim Wojtulewicz
7aa7909c94 Add forwarding from VLAN analyzer into LLC, SNAP, and Novell 802.3 analyzers 2023-04-25 12:29:55 -07:00
Tim Wojtulewicz
c5b8603218 Remove non-standard way of forwarding out of the Ethernet analyzer 2023-04-25 12:29:55 -07:00
Tim Wojtulewicz
7e88a2b3fb Add basic LLC, SNAP, and Novell 802.3 packet analyzers 2023-04-25 12:29:54 -07:00
Tim Wojtulewicz
f62f8e5cc9 Remove workaround for tunnels from IEEE 802.11 analyzer 2023-04-25 09:28:20 -07:00