Commit graph

4348 commits

Author SHA1 Message Date
Tim Wojtulewicz
7aa7909c94 Add forwarding from VLAN analyzer into LLC, SNAP, and Novell 802.3 analyzers 2023-04-25 12:29:55 -07:00
Tim Wojtulewicz
c5b8603218 Remove non-standard way of forwarding out of the Ethernet analyzer 2023-04-25 12:29:55 -07:00
Tim Wojtulewicz
7e88a2b3fb Add basic LLC, SNAP, and Novell 802.3 packet analyzers 2023-04-25 12:29:54 -07:00
Tim Wojtulewicz
f62f8e5cc9 Remove workaround for tunnels from IEEE 802.11 analyzer 2023-04-25 09:28:20 -07:00
Tim Wojtulewicz
2d05beac06 Add support for 802.11 A-MSDU aggregates 2023-04-25 09:28:20 -07:00
Tim Wojtulewicz
5b1c6216bd Fix IEEE 802.11 analyzer to properly forward tunneled packets
This mostly happens with Aruba, but could possibly happen with other tunnels too.
2023-04-25 09:28:20 -07:00
Tim Wojtulewicz
e4a1c30828 Fix IEEE 802.11 analyzer to skip packets with the Protected bit enabled 2023-04-25 09:28:20 -07:00
Christian Kreibich
1a1feec887 Add btest for "-C" flag vs the script-layer ignore_checksums global. 2023-04-24 21:19:05 -07:00
Arne Welzel
1b69b4d26f Merge branch 'topic/amazingpp/irc-fuid-missing' of github.com:AmazingPP/zeek
* 'topic/amazingpp/irc-fuid-missing' of github.com:AmazingPP/zeek:
  Add irc_dcc_send_ack event and fix missing fields

I've moved IRC_Data back into the zeek::analyzer::file namespace, but
we did move the declaration from protocol/file/File.h to protocol/irc/IRC.h.
But, if someone actually customized IRC_Data and didn't include protocol/irc/IRC.h
for other reasons, I'll be surprised (and also just suggest to update the include).
2023-04-24 18:22:50 +02:00
Arne Welzel
ffb73e4de9 Merge remote-tracking branch 'origin/topic/awelzel/add-community-id'
* origin/topic/awelzel/add-community-id:
  testing/external: Bump hashes for community_id addition
  NEWS: Add entry for Community ID
  policy: Import zeek-community-id scripts into protocols/conn frameworks/notice
  Add community_id_v1() based on corelight/zeek-community-id
2023-04-24 10:12:56 +02:00
Arne Welzel
0bbd7cab1b testing/external: Bump hashes for community_id addition 2023-04-24 09:43:22 +02:00
Christian Kreibich
b90351b7e6 policy: Import zeek-community-id scripts into protocols/conn frameworks/notice
Slightly adapted for indentation.
2023-04-24 09:43:19 +02:00
Fupeng Zhao
161ffb4192 Add irc_dcc_send_ack event and fix missing fields 2023-04-24 07:29:51 +00:00
Christian Kreibich
99de7b7526 Add community_id_v1() based on corelight/zeek-community-id
"Community ID" has become an established flow hash for connection correlation
across different monitoring and storage systems. Other NSMs have had native
and built-in support for Community ID since late 2018. And even though the
roots of "Community ID" are very close to Zeek, Zeek itself has never provided
out-of-the-box support and instead required users to install an external plugin.

While we try to make that installation as easy as possible, an external plugin
always sets the bar higher for an initial setup and can be intimidating.
It also requires a rebuild operation of the plugin during upgrades. Nothing
overly complicated, but somewhat unnecessary for such popular functionality.

This isn't a 1:1 import. The options are parameters and the "verbose"
functionality  has been removed. Further, instead of a `connection`
record, the new bif works with `conn_id`, allowing computation of the
hash with little effort on the command line:

    $ zeek -e 'print community_id_v1([$orig_h=1.2.3.4, $orig_p=1024/tcp, $resp_h=5.6.7.8, $resp_p=80/tcp])'
    1:RcCrCS5fwYUeIzgDDx64EN3+okU

Reference: https://github.com/corelight/zeek-community-id/
2023-04-21 20:44:09 +02:00
Jan Grashoefer
893d31e1aa Add cluster_started restart tests. 2023-04-21 19:04:52 +02:00
Jan Grashoefer
2dcca004d3 Add basic cluster_started tests. 2023-04-21 19:04:52 +02:00
Jan Grashoefer
3db8bb4a44 Generalize Cluster::worker_count. 2023-04-21 19:04:39 +02:00
Arne Welzel
10dc1c4bc2 zeek.bif: Remove cat_sep() fully var_arg changes
These were introduced to better catch type violations at runtime. With
bifcl doing these checks, revert to a better documented version.
2023-04-19 19:30:46 +02:00
Arne Welzel
f44279cc3e broker/messaging: Runtime type checks for pool
publish_hrw() and publish_rr() are excluded from type checking due to their
variadic nature. Passing a wrong type for the pool argument previously triggered
an abort, now the result is runtime errors. This isn't great, but it's
better than crashing Zeek.

Closes #2935
2023-04-19 19:30:46 +02:00
Arne Welzel
89c828ac14 Merge remote-tracking branch 'origin/topic/vern/record-optimizations.Apr23B'
* origin/topic/vern/record-optimizations.Apr23B:
  different fix for MSVC compiler issues
  more general approach for addressing MSVC compiler issues with IntrusivePtr
  restored RecordType::Create, now marked as deprecated tidying of namespaces and private class members simplification of flagging record field initializations that should be skipped address peculiar MSVC compilation complaint for IntrusivePtr's
  clarifications and tidying for record field initializations
  optimize record construction by deferring initializations of aggregates
  compile-scripts-to-C++ speedups by switching to raw record access
  logging speedup by switching to raw record access
  remove redundant record coercions

Removed the `#if 0` hunk during merging: Probably could have gone with a
doctest instead.
2023-04-19 11:59:56 +02:00
Arne Welzel
a9a37c865e log-caching-cluster: Wait for X509::known_log_certs to populate
The known_log_certs table is populated asynchronously via broker after a
Broker::peer_added. It may take a variable amount of time depending on where
we run this test and it has been observed flaky specifically for the
arm_debian11 task. Instead of racing, give worker-2 3 seconds for receiving
the expected table content before continuing.

Fixes #2885
2023-04-18 15:19:12 +02:00
Arne Welzel
f0b9c59adb Add experimental JavaScript support when libnode is available
zeek.on('zeek_init', () => {
        console.log('Hello, Zeek!');
    });

For interaction with external systems and HTTP APIs, JavaScript and the
Node.js ecosystem beat Zeek script. Make it more easily accessible by
including ZeekJS with Zeek directly.

When a recent enough libnode version is found on the build system, ZeekJS is
added as a builtin plugin. This behavior can be disabled via
``--disable-javascript``. Linux distributions providing such a package are
Ubuntu (22.10) and Debian (testing/bookworm) as libnode-dev.
Fedora provides it as nodejs-devel.

This plugin takes over loading of .js or .cjs files. When no such files
are provided to Zeek, Node and the V8 engine are not initialized and
should not get into the way.

This should be considered experimental.
2023-04-14 11:26:41 +02:00
Dominik Charousset
9aeed5284d Redesign subdir-libs and plugin scaffolding 2023-04-13 22:23:46 +02:00
Tim Wojtulewicz
fcf7af259c GH-2920: Don't warn on uninitialized container options 2023-04-13 09:05:17 -07:00
Arne Welzel
a5e7faf564 logging/Manager: Fix crash for rotation format function not returning
While working on a rotation format function, ran into Zeek crashing
when not returning a value from it, fix and recover the same way as
for scripting errors.
2023-04-13 09:23:51 +02:00
Tim Wojtulewicz
f812ce53cf Merge remote-tracking branch 'security/topic/timw/154-rdp-timeout'
* security/topic/timw/154-rdp-timeout:
  RDP: Instantiate SSL analyzer instead of PIA
  RDP: add some enforcement to required values based on MS-RDPBCGR docs
2023-04-11 15:23:52 -07:00
Tim Wojtulewicz
f701f1fc94 Merge remote-tracking branch 'security/topic/awelzel/152-smtp-validate-mail-transactions'
* security/topic/awelzel/152-smtp-validate-mail-transactions:
  smtp: Validate mail transaction and disable SMTP analyzer if excessive
  generic-analyzer-fuzzer: Detect disable_analyzer() from scripts
2023-04-11 15:16:25 -07:00
Tim Wojtulewicz
c670f3fdb2 Merge remote-tracking branch 'security/topic/awelzel/148-ftp-skip-get-pending-commands-multi-line-response'
* security/topic/awelzel/148-ftp-skip-get-pending-commands-multi-line-response:
  ftp/main: Special case for intermediate reply lines
  ftp/main: Skip get_pending_command() for intermediate reply lines
2023-04-11 14:50:55 -07:00
Vern Paxson
f866252e5e remove redundant record coercions 2023-04-10 11:42:48 -07:00
Tim Wojtulewicz
51ca741d92 Merge remote-tracking branch 'origin/topic/timw/continue-processing-math'
* origin/topic/timw/continue-processing-math:
  Fix a long-standing bug in the math around continue_processing
  Add is_processing_suspended BIF
2023-04-08 11:23:06 -07:00
Tim Wojtulewicz
67802e711a Report packet statistics via the telemetry framework 2023-04-06 13:41:09 -07:00
Tim Wojtulewicz
95b6069c6e Fix a long-standing bug in the math around continue_processing 2023-04-06 10:49:05 -07:00
Tim Wojtulewicz
ae3d6a4df0 Add optional packet filtered statistics for packet sources 2023-04-06 09:47:04 -07:00
Arne Welzel
d9611132e8 scan.l: Support @pragma, specifically push/pop ignore-deprecations
This allows to ignore-deprecations as follows:

    @pragma push ignore-deprecations
    print r$deprecated;
    @pragma pop

A bit of infrastructure is put in place for the push and pop, but
nothing overly generalized for future pragmas.
2023-04-05 10:24:30 +02:00
Arne Welzel
92f09f0db7 Merge remote-tracking branch 'origin/topic/vern/when-cleanup'
* origin/topic/vern/when-cleanup:
  test suite update for minor change in "when" error messages
  removed skeletal (non-functioning) "when" support from ZAM
  simplify WhenInfo and Trigger classes given removal of old capture semantics
  introduced notion of light-weight Frame clones
  changed function_ingredients struct to FunctionIngredients class with accessors

Renamed Frame::LightClone() to Frame::CloneForTrigger() during merge.
2023-04-04 10:02:01 +02:00
Arne Welzel
1858c6b0c2 Merge remote-tracking branch 'origin/topic/vern/CPP-Apr23-maint'
* origin/topic/vern/CPP-Apr23-maint:
  addressed static analysis concern about possible null pointer
  tweaks for "-O C++" of BTest's with conditional code

Backed out changes from at-if-lambda during merge and instead skip
test when running with ZEEK_USE_CPP.
2023-04-03 16:42:18 +02:00
Vern Paxson
910b50ef0d test suite update for minor change in "when" error messages 2023-04-02 11:38:30 -07:00
Vern Paxson
2f9272f2d5 tweaks for "-O C++" of BTest's with conditional code 2023-04-01 20:26:55 -07:00
Vern Paxson
da643add2e minor ZAM BTest baseline updates 2023-04-01 12:57:21 -07:00
Vern Paxson
9560e38784 fixed type mismatch for ssl_certificate_request event 2023-04-01 12:56:54 -07:00
Arne Welzel
5718046b96 Merge branch 'topic/jgras/tunnel-fwd-fix' of https://github.com/J-Gras/zeek
* 'topic/jgras/tunnel-fwd-fix' of https://github.com/J-Gras/zeek:
  Add test cases for Geneve.
  Fix forwarding of tunnelled packets.
2023-03-31 10:39:42 +02:00
Jan Grashoefer
65f0a09a83 Add test cases for Geneve. 2023-03-30 22:58:54 +02:00
Tim Wojtulewicz
d2eaced96a Add reporter framework to core.packet-analyzer-override btest 2023-03-30 12:28:39 -07:00
Jan Grashoefer
bfa466f2eb Reintroduce info when overriding packet analyzer mappings.
In #2464 the warning when overriding a packet analyzer mapping was
removed. While a warning seems indeed excessive, some info would still
be nice to have.
2023-03-30 16:19:39 +02:00
Arne Welzel
cd0d55e3d6 btest/broker: Add test using Python bindings and zeek -r
This came up in community.zeek.org as a use-case and there
were a few gaps and holes, so add a test showing minimally
the current behavior.
2023-03-27 21:13:03 +02:00
Arne Welzel
b8dc6ad120 smtp: Validate mail transaction and disable SMTP analyzer if excessive
An invalid mail transaction is determined as

* RCPT TO command without a preceding MAIL FROM
* a DATA command without a preceding RCPT TO

and logged as a weird.

The testing pcap for invalid mail transactions was produced with a Python
script against a local exim4 configured to accept more errors and unknown
commands than 3 by default:

    # exim4.conf.template
    smtp_max_synprot_errors = 100
    smtp_max_unknown_commands = 100

See also: https://www.rfc-editor.org/rfc/rfc5321#section-3.3
2023-03-27 18:41:47 +02:00
Tim Wojtulewicz
a9d3245e80 RDP: add some enforcement to required values based on MS-RDPBCGR docs 2023-03-24 10:33:21 -07:00
Arne Welzel
59b1e29e81 Broker: Remove network time initialization
Remove the special case of initializing network time if it hasn't
happened yet. The argument about broker.log containing 0.0 timestamps
is more a problem of the log, not something that would justify modifying
network time globally. For broker.log and possibly cluster.log, it might
be more reasonable to use current time, anyway.

I was a bit wary about tables backed by broker stores being populated
with network_time set to 0.0, but there seems to exist logic and assumptions
that this is okay: It should be the same as if one populates a table with
expirations set within zeek_init().

In fact, staring a bit more, *not setting* network time might be more correct
as workers that don't see packets would never set zeek_start_network_time
which is used within the expiration computation.
2023-03-24 15:13:29 +01:00
Arne Welzel
1b3e8a611e ftp/main: Skip get_pending_command() for intermediate reply lines
Intermediate lines of multiline replies usually do not contain valid status
codes (even if servers may opt to include them). Their content may be anything
and likely unrelated to the original command. There's little reason for us
trying to match them with a corresponding command.

OSS-Fuzz generated a large command reply with very many intermediate lines
which caused long processing times due to matching every line with all
currently pending commands.
This is a DoS vector against Zeek. The new ipv6-multiline-reply.trace and
ipv6-retr-samba.trace files have been extracted from the external ipv6.trace.
2023-03-23 13:50:36 +01:00
Arne Welzel
8734f23724 A set of tests around set_network_time() and timer expiration
This tests that timer expiration happens after a call to set_network_time()
upon the next time around the loop. This should be fairly stable, but
suspect major changes in the main loop or around timer expiration may
subtly change behavior things.
2023-03-23 12:41:56 +01:00