Commit graph

3128 commits

Author SHA1 Message Date
Arne Welzel
d4e31e7d2b RunState: Implement forward_network_time_if_applicable()
Add a central place where the decision when it's okay to update network time
to the current time (wallclock) is. It checks for pseudo_realtime and packet
source existence as well as packet source idleness.

A new const &redef allows to completely disable forwarding of network time.
2023-03-23 12:40:39 +01:00
Jan Grashoefer
1882307cf3 Add pcap_file option to supervised nodes.
This allows to start Supervised nodes with a pcap_file argument
rather than interface.

This is based on changes from @J-Gras.
2023-03-21 16:18:02 +01:00
Arne Welzel
46c432dc8b iosource: Make poll intervals configurable
This probably should not be changed by users, but it's useful for
testing and experimentation rather than needing to recompile.

Processing 100 packets without checking an FD based IO source can
actually mean that FD based sources are never checked during a read
of a very small pcap...
2023-03-21 09:15:33 +01:00
Arne Welzel
cf2da5160b dns: Remove AD and CD flags from log
There was a misunderstanding whether to include them by default in
the dns.log, so remove them again.

There had also been a discussion and quirk that AD of a request would
always be overwritten by reply in the dns.log unless the reply is
missing. For now, let users extend dns.log themselves for what best
fits their requirements, rather than adding these flags by default.

Add a btest to print AD and CD flags for smoke testing still.
2023-03-16 10:09:27 +01:00
Christian Kreibich
693d8e9251 Treat private address space as site-local by default
This makes Site::private_address_space work like a subset of Site::local_nets,
to match many user's intuition of how we should treat site locality out of the
box. As config options, changes/redefs to Site::private_address_space propagate
to Site::local_nets, while changes to the latter don't affect the former.

A new global bit `Site::private_address_space_is_local` controls the behavior.
It defaults to true, and redefing to false brings back the original behavior.
2023-03-15 17:01:00 -07:00
Christian Kreibich
19829765d4 Provide a mechanism to suppress logging of internal config framework activity 2023-03-15 17:01:00 -07:00
Johanna Amann
dcbc809189 Merge remote-tracking branch 'origin/topic/johanna/certificate-req'
* origin/topic/johanna/certificate-req:
  SSL/TLS CertificateRequest message: Address review feedback
  SSL/TLS: Parse CertificateRequest message
2023-03-15 10:03:25 +01:00
Johanna Amann
b8d658ac77 SSL/TLS CertificateRequest message: Address review feedback
Minor stylistic changes; see https://github.com/zeek/zeek/pull/2855 for
details
2023-03-15 09:19:26 +01:00
Arne Welzel
33090d7a27 Merge branch 'dnssec-flag-parse' of github.com:micrictor/zeek-codespace
* 'dnssec-flag-parse' of github.com:micrictor/zeek-codespace:
  Update external testing commit hash for DNS flag changes
  Parse DNSSEC AD and CD bits

Updated dump-events baseline which seemed unrelated.
2023-03-14 10:35:50 +01:00
Michael R. Torres
fe8390c646 Parse DNSSEC AD and CD bits
Parse authentic data (AD) and checking disabled (CD) bits according to
RFC 2535. Leaves the Z field as-is, in case users are already handling
this elsewhere and depend on the value being the integer for all 3 bits.

https://www.rfc-editor.org/rfc/rfc2535#section-6.1

Fixes #2672
2023-03-13 14:35:06 -07:00
Arne Welzel
2251c67e56 get_dns_stats: Expose total cache size and cached text entries
It wasn't possible from script land to determine the total size
of the cache table held by the DNS_Mgr. Add the total and also
also the TEXT entries count.
2023-03-10 09:22:45 +01:00
Johanna Amann
b56b856da9 SSL/TLS: Parse CertificateRequest message
This commit introduces parsing of the CertificateRequest message in the
TLS handshake. It introduces a new event ssl_certificate_request, as
well as a new function parse_distinguished_name, which can be used to
parse part of the ssl_certificate_request event parameters.

This commit also introduces a new policy script, which appends
information about the CAs a TLS server requests in the
CertificateRequest message, if it sends it.
2023-03-09 09:12:29 +01:00
Tim Wojtulewicz
f6d3ed2395 Default known stores to false to avoid Broker stores 2023-03-02 12:48:53 -07:00
Johanna Amann
989e9c29d2 X.509: expose the signature type inside the tbs certificate
This change exposes the signature tyope inside the signed portion of an
X.509 certificate. In the past, we only exposed the signature type that
is contained inside the signature, which is outside the signed portion
of the X.509 certificate.

In theory, both signature fields should have the same value; it is,
however, possible to encode differing values in both fields. The new
field is not logged by default.
2023-02-28 19:24:16 +00:00
Arne Welzel
f56785740c ftp: Limit user, password, arg and reply_msg column sizes in log
The user and password fields are replicated to each of the ftp.log
entries. Using a very large username (100s of KBs) allows to bloat
the log without actually sending much traffic. Further, limit the
arg and reply_msg columns to large, but not unbounded values.
2023-02-21 12:28:07 -07:00
Tim Wojtulewicz
8cf1e51623 Add max_size argument for find_all/find_all_ordered BIFs 2023-02-21 12:27:54 -07:00
Eldon Koyle
32afbae9db Use a default analyzer
Use a default analyzer instead of hardcoding a protocol number.
2023-02-16 19:39:27 -07:00
Eldon Koyle
56aa03031d Simplify PBB analyzer by using Ethernet analyzer
After the first 4 bytes, this traffic actually just looks like Ethernet.
Rather than try to re-implement the ethernet analyzer, just check the
length, skip 4 bytes, and pass it on.
2023-02-16 08:19:30 -07:00
Eldon Koyle
269cc15888 Cleanup and add customer MAC addresses
* Put c-dst/c-src in l2_dst/l2_src
 * use #define instead of const int and move to PBB.h
2023-02-10 17:42:25 -07:00
Eldon Koyle
28d540483e Add PBB (802.1ah) support 2023-02-10 15:30:01 -07:00
Arne Welzel
e4ab7b2d70 files/main: No empty file_ids
When an analyzer calls DataIn(), there's a costly callback construct
going through the event queue. If an analyzer does not have a
get_file_handle() handler installed, the produced file_id would
end up empty and ignored. Consequently, the get_file_handle() callback
was invoked for every new DataIn() invocations.

This is surprising and costly. Log a warning when this happens and
instead set a generically generated file handle value instead to
prevent the repeated get_file_handle() invocations.
2023-02-06 18:08:05 +01:00
Robin Sommer
bc252c63dc
Add BIF have_spicy_analyzers().
We previously used the Spicy plugin's `Spicy::available` to test for
Spicy support. However, having Spicy support does not necessarily mean that we
have built Zeek with its in-tree Spicy analyzers: the Spicy plugin
could have been pulled in from external. The new BIF now reliably
tells us whether the Spicy analyzers are available; its result
corresponds to what `zeek-config --have-spicy-analyzers` returns as
well.

We also move the two current checks over to use this BIF.

(Note: I refrained from renaming the CMake-side `USE_SPICY_ANALYERS`
to `HAVE_SPICY_ANALYZERS`. We should do this eventually for
consistency, but I didn't want to make more changes than necessary
right now.)
2023-02-03 13:47:26 +01:00
Tim Wojtulewicz
0fd335f7f0 Merge remote-tracking branch 'security/topic/timw/131-smb-fscontrol-overflow'
* security/topic/timw/131-smb-fscontrol-overflow:
  Restore/rename field in SMB2::Fscontrol record type
2023-02-01 10:48:16 -07:00
Robin Sommer
04a1ead978
Provide infrastructure to migrate legacy analyzers to Spicy.
As initial examples, this branch ports the Syslog and Finger analyzers
over. We leave the old analyzers in place for now and activate them
iff we compile without any Spicy.

Needs `zeek-spicy-infra` branches in `spicy/`, `spicy-plugin/`,
`CMake/`, and `zeek/zeek-testing-private`.

Note that the analyzer events remain associated with the Spicy plugin
for now: that's where they will show up with `-NN`, and also inside
the Zeekygen documentation.

We switch CMake over to linking the runtime library into the plugin,
vs. at the top-level through object libraries.
2023-02-01 11:33:48 +01:00
Arne Welzel
be44c642e1 Merge remote-tracking branch 'origin/topic/awelzel/move-disabling-analyzer-out-of-global'
* origin/topic/awelzel/move-disabling-analyzer-out-of-global:
  analyzer: Move disabling_analyzer() hook into Analyzer module
2023-01-31 14:48:56 +01:00
Arne Welzel
f35cf228dc broker/store: Extend SQLiteOptions around data safety and performance
Add configurability of synchronous and journal_mode for SQLite backed
Broker data stores. Setting these to synchronous=normal and journal_mode=wal
can significantly improve throughput at the cost of some durability in
the presence of power loss or OS crash. In the context of Zeek, this is
likely more than acceptable.

Additionally, add integrity_check and failure_mode options to support deleting
and re-opening a corrupted SQLite database at store creation.

Closes #2698
2023-01-30 10:25:37 +01:00
Tim Wojtulewicz
84ac362c67 Restore/rename field in SMB2::Fscontrol record type
b41a4bf06d removed a field from this record
because it had a duplicate name as another field. The field does need to
exist, but it needs the correct name.
2023-01-27 17:39:10 -07:00
Arne Welzel
8be8c22b3e smb1: Prevent accessing uninitialized referenced_tree
The added pcap was created from an OSS Fuzz test case and is borderline
valid SMB traffic, but it triggered a scripting error.

Closes #2726
2023-01-27 19:22:13 +01:00
Arne Welzel
672602dae7 MySQL: Fix endianness, introduce mysql_eof() event
We were parsing MySQL using bigendian even though the protocol is
specified as with "least significant byte first" [1]. This is most
problematic when parsing length encoded strings with 2 byte length
fields...

Further, I think, the EOF_Packet parsing was borked, either due to
testing the CLIENT_DEPRECATE_EOF with the wrong endianness, or due to
the workaround in Resultset processing raising mysql_ok(). Introduce a
new mysql_eof() that triggers for EOF_Packet's and remove the fake
mysql_ok() Resultset invocation to fix. Adapt the mysql script and tests
to account for the new event.

This is a quite backwards incompatible change on the event level, but
due to being quite buggy in general, doubt this matters to many.

I think there is more buried, but this fixes the violation of the simple
"SHOW ENGINE INNODB STATUS" and the existing tests continue to
succeed...

[1] https://dev.mysql.com/doc/dev/mysql-server/latest/page_protocol_basic_dt_integers.html
2023-01-27 10:59:23 +01:00
Tim Wojtulewicz
6cfb45d24f Merge remote-tracking branch 'jeff-bb/patch-2'
* jeff-bb/patch-2:
  Log raw keyboard value on best guess
  Avoid excessive fmt calls, return default behavior on unknown
  "Best Guess" unknown keyboard / language variants
2023-01-23 12:50:23 -07:00
jeff-bb
7085104c33
Log raw keyboard value on best guess 2023-01-23 09:12:48 -06:00
Arne Welzel
26b1558cd1 analyzer: Move disabling_analyzer() hook into Analyzer module
When disabling_analyzer() was introduced, it was added to the GLOBAL
module. The awkward side-effect is that implementing a hook handler
in another module requires to prefix it with GLOBAL. Alternatively, one
can re-open the GLOBAL module and implement the handler in that scope.

Both are not great, and prefixing with GLOBAL is ugly, so move the
identifier to the Analyzer module and ask users to prefix with Analyzer.
2023-01-23 12:22:05 +01:00
jeff-bb
04113b13d5
Avoid excessive fmt calls, return default behavior on unknown
Using "in" to query the language const. This also handles the case of not having a best guess and continue using the existing behavior.

Given
keyboard_layout = 1033 (0x0409), "keyboard-English - United States"
keyboard_layout = 66569 (0x00010409), "keyboard-English - United States (Best Guess)"
keyboard_layout = 12345 (0x3039), "keyboard-12345"
2023-01-20 08:29:55 -06:00
jeff-bb
3012e0417a
Remove Duplicate 4122 Croatian 2023-01-19 17:04:42 -06:00
jeff-bb
dd2cdb064b
"Best Guess" unknown keyboard / language variants
If the lookup table does not have an entry, it will just log as the raw decimal language/keyboard code. With this change, if we do not have an entry in the lookup table, we'll look at the low order / 4 least significant bits to see if we have a match. The high order / 4 most significant bits are flags/modifiers to the base language/keyboard code. We'll append that it is a "Best Guess"

(This is my first attempt at Zeek scripting, apologies upfront if I'm missing obvious language features. I feel like the const language lookup should return a success/fail return code that we would key off of, but unsure how to accomplish that so instead went for string matching on value in == value out).
2023-01-19 16:55:23 -06:00
jeff-bb
ee21b30a18
Revert Sort 2023-01-19 16:05:43 -06:00
jeff-bb
498aaef428
Update RDP Keyboard Languages
Using additional sources to add more languages / locales. Changed sort order to match other sections.
2023-01-19 15:44:57 -06:00
Christian Kreibich
12885c7475 Fix a docstring typo 2023-01-10 18:49:19 -08:00
Arne Welzel
2d852209b0 Merge remote-tracking branch 'origin/topic/awelzel/analyzer-log'
* origin/topic/awelzel/analyzer-log:
  btest/net-control: Use different expiration times for rules
  analyzer: Add analyzer.log for logging violations/confirmations
2023-01-10 10:22:58 +01:00
Arne Welzel
a004bdf5d9 Merge remote-tracking branch 'origin/topic/awelzel/2647-intel-seen-file-names'
* origin/topic/awelzel/2647-intel-seen-file-names:
  intel/seen/file-names: Use file_over_new_connection()
2023-01-10 10:12:04 +01:00
Arne Welzel
6d19c49efe intel/seen/file-names: Use file_over_new_connection()
The seen/file-names script relies on f$info$filename to be populated.
For HTTP and other network protocols, however, this field is only
populated during file_over_new_connection() that's running after
file_new().

Use the file_new() event only for files without connections and
file_over_new_connection() implies that f$conns is populated, anyway.

Special case SMB to avoid finding files twice, because there's a
custom implementation in seen/smb-filenames.zeek.

Fixes #2647
2023-01-10 10:10:28 +01:00
Arne Welzel
17d0ade26a analyzer: Add analyzer.log for logging violations/confirmations
By default this only logs all the violations, regardless of the
confirmation state (for which there's still dpd.log). It includes
packet, protocol and file analyzers.

This uses options, change handlers and event groups for toggling
the functionality at runtime.

Closes #2031
2023-01-09 18:11:49 +01:00
Arne Welzel
4e75d54d49 scripts/analyzer: Introduce Analyzer::requested_analyzers
In certain deployment scenarios, all analyzers are disabled by default.
However, conditionally/optionally loaded scripts may rely on analyzers
functioning and declare a request for them.

Add a global set set to the Analyzer module where external scripts can record
their requirement/request for a certain analyzer. Analyzers found in this
set are enabled at zeek_init() time.
2022-12-13 14:28:16 +01:00
Arne Welzel
a0aa00fa81 logging: Add event_groups to Stream
This commit adds an optional event_groups field to the Logging::Stream record
to associated event groups with logging streams.

This can be used to disable all event groups of a logging stream when it is
disabled. It does require making an explicit connection between the
logging stream and the involved groups, however.
2022-12-09 16:59:36 +01:00
Arne Welzel
85ce48eb1e analyzer/files: handle non-analyzer names in describe_file()
When a fa_file object is created through the use of Input::add_analysis(),
the fa_file's source is likely not valid representation of an analyzer's
tag and a Files::describe() should not error and instead return an empty
description.

Add a new Analyzer::is_tag() helper that can be used to pre-check `f$source`.
2022-12-06 11:17:30 +01:00
Arne Welzel
1e06c8bfda frameworks/notice: Handle fa_file with no or more than a single connection better
* When a file is transferred over multiple connection, have
  create_file_info() just pick the first one instead of none.

* Do not unconditionally assume cid and cuid as set on a
  Notice::FileInfo object.
2022-12-06 11:17:30 +01:00
Christian Kreibich
1c381b5531 Merge branch 'topic/christian/gh-846-tcp-duration-docs'
* topic/christian/gh-846-tcp-duration-docs:
  Expand Conn::Info$duration comment to clarify TCP end-of-connection handling
2022-11-30 09:42:18 -08:00
Christian Kreibich
b0f96fa22c Expand Conn::Info$duration comment to clarify TCP end-of-connection handling
From Vern in GH-846: This is a conscious decision in the TCP analysis to
consider a connection's "duration" to run up through the end of its
productive (= data can be delivered) lifetime, not extending beyond that. So
once it's closed, packets seen subsequently (until the state-holding for the
connection times out) get processed in terms of updating the associated
history, but not the duration. This can include (unnecessarily) retransmitted
data packets, like in one of the examples above. An advantage of this definition
of "duration" is it allows more accurate computation of connection data rates.
2022-11-30 09:39:57 -08:00
Johanna Amann
f4527ee4db Merge remote-tracking branch 'origin/topic/awelzel/2514-expire-all-timers-special-case'
* origin/topic/awelzel/2514-expire-all-timers-special-case:
  TimerMgr: Add back max_timer_expires=0 special case
  Add btest for expiration of all pending timers.
2022-11-30 13:45:09 +00:00
Johanna Amann
3253168a53 Merge remote-tracking branch 'origin/topic/awelzel/2583-mqtt-to-base'
* origin/topic/awelzel/2583-mqtt-to-base:
  mqtt: Move from policy/ into base/
2022-11-30 13:44:27 +00:00