Commit graph

18601 commits

Author SHA1 Message Date
Arne Welzel
359f8d2ae6 quic: Squashed follow-ups: quic.log, tests, various fixes, performance 2023-10-11 14:10:22 +02:00
Joost
44d7c45723 quic: Initial implementation 2023-10-11 14:10:22 +02:00
Arne Welzel
ffc35d90ba Merge remote-tracking branch 'origin/topic/awelzel/ldap-follow-up'
* origin/topic/awelzel/ldap-follow-up:
  protocol/ldap: Add doc-id/doc-description to .evt file
  spicy/manager: Ensure Zeekygen knows identifier for registered types
2023-10-11 14:09:32 +02:00
Arne Welzel
6abe557502 protocol/ldap: Add doc-id/doc-description to .evt file 2023-10-11 13:31:25 +02:00
Arne Welzel
101c6696b9 spicy/manager: Ensure Zeekygen knows identifier for registered types
Without this, Zeekygen won't generate documentation about exported
enum types as it can not resolve the identifier. Also, only register a
type as item with the Spicy plugin if there's no _module_info currently
active.
2023-10-11 13:31:20 +02:00
Arne Welzel
72df1a0216 Merge remote-tracking branch 'origin/topic/bbannier/issue-3234'
* origin/topic/bbannier/issue-3234:
  Introduce dedicated `LDAP::Info`
  Remove redundant storing of protocol in LDAP logs
  Use LDAP `RemovalHook` instead of implementing `connection_state_remove`
  Tidy up LDAP code by using local references
  Pluralize container names in LDAP types
  Move LDAP script constants to their own file
  Name `LDAP::Message` and `LDAP::Search` `*Info`
  Make ports for LDAP analyzers fully configurable
  Require have-spicy for tests which log spicy-ldap information
  Fix LDAP analyzer setup for when Spicy analyzers are disabled
  Bump zeek-testing-private
  Integrate spicy-ldap test suite
  Move spicy-ldap into Zeek protocol analyzer tree
  Explicitly use all of spicy-ldap's modules
  Explicitly list `asn1.spicy` as spicy-ldap source
  Remove uses of `zeek` module in spicy-ldap
  Fix typos in spicy-ldap
  Remove project configuration files in spicy-ldap
  Integrate spicy-ldap into build
  Import zeek/spicy-ldap@57b5eff988
2023-10-10 20:07:03 +02:00
Benjamin Bannier
346d2c49a9 Introduce dedicated LDAP::Info 2023-10-10 18:49:25 +02:00
Benjamin Bannier
301d8722bf Remove redundant storing of protocol in LDAP logs 2023-10-10 18:49:25 +02:00
Benjamin Bannier
82b3a4048f Use LDAP RemovalHook instead of implementing connection_state_remove 2023-10-10 18:49:25 +02:00
Benjamin Bannier
1d4412a9e7 Tidy up LDAP code by using local references 2023-10-10 18:49:25 +02:00
Benjamin Bannier
3a60a60619 Pluralize container names in LDAP types 2023-10-10 18:49:25 +02:00
Benjamin Bannier
0c126f3c6b Move LDAP script constants to their own file 2023-10-10 18:28:13 +02:00
Benjamin Bannier
c43bc52e18 Name LDAP::Message and LDAP::Search *Info 2023-10-10 18:28:13 +02:00
Benjamin Bannier
9b02b93889 Make ports for LDAP analyzers fully configurable
This moves the ports the LDAP analyzers should be triggered on from the
EVT file to the Zeek module. This gives users full control over which
ports the analyzers are registered for while previously they could only
register them for additional ports (there is no Zeek script equivalent
of `Manager::UnregisterAnalyzerForPort`).

The analyzers could still be triggered via DPD, but this is intentional.
To fully disable analyzers users can use e.g.,

```zeek
event zeek_init()
    {
    Analyzer::disable_analyzer(Analyzer::ANALYZER_LDAP_TCP);
    }
```
2023-10-10 18:28:13 +02:00
Arne Welzel
7fac5837c3 iosource/pcap: Support configurable buffer size
On Linux with a default ext4 or tmpfs filesystem, the default buffer size for
reading a pcap is chosen as 4k (strace/gdb validated). When reading large pcaps
containing raw data transfers, the syscall overhead for read becomes visible
in profiles. Support configurability of the buffer size and default to 128kb.

When processing a ~830M PCAP (16 UDP connections, each transferring ~50MB) in
bare mode, this change improves runtime from 1.39 sec to 1.29 sec. Increasing
the buffer further didn't provide a noticeable boost.
2023-10-10 15:08:51 +02:00
Arne Welzel
c161b1c4b1 util/setvbuf: Respect buf argument 2023-10-10 09:29:21 +02:00
Benjamin Bannier
c0daacfbd1 Require have-spicy for tests which log spicy-ldap information 2023-10-10 09:21:57 +02:00
Benjamin Bannier
53d4052d68 Fix LDAP analyzer setup for when Spicy analyzers are disabled 2023-10-10 09:21:57 +02:00
Benjamin Bannier
20ae1129ea Bump zeek-testing-private 2023-10-10 09:21:57 +02:00
Benjamin Bannier
d7db52dff6 Integrate spicy-ldap test suite 2023-10-10 09:21:57 +02:00
Benjamin Bannier
f172febbcb Move spicy-ldap into Zeek protocol analyzer tree 2023-10-10 09:21:57 +02:00
Benjamin Bannier
e544540986 Explicitly use all of spicy-ldap's modules
This fixes building of this multi-module analyzer.
2023-10-10 09:21:57 +02:00
Benjamin Bannier
cfe24189f2 Explicitly list asn1.spicy as spicy-ldap source
This will lead to the file being installed so it can be consumed by
other parsers.
2023-10-10 09:21:57 +02:00
Benjamin Bannier
6e87f49ca8 Remove uses of zeek module in spicy-ldap 2023-10-10 09:21:57 +02:00
Benjamin Bannier
4251557464 Fix typos in spicy-ldap 2023-10-10 09:21:57 +02:00
Benjamin Bannier
ffbc45ed6e Remove project configuration files in spicy-ldap 2023-10-10 09:21:57 +02:00
Benjamin Bannier
dc0ef39fb0 Integrate spicy-ldap into build 2023-10-10 09:21:57 +02:00
Benjamin Bannier
080d7418d7 Import zeek/spicy-ldap@57b5eff988 2023-10-10 09:21:57 +02:00
Tim Wojtulewicz
255103b50a Merge branch 'topic/timw/arm-build-failure'
* topic/timw/arm-build-failure:
  CI: Disable spicy on arm_debian11 and opensuse_leap_15_4 builds
  CI: Remove sanitizers resource template
2023-10-09 12:53:30 -07:00
Tim Wojtulewicz
bbf7f34cdc CI: Disable spicy on arm_debian11 and opensuse_leap_15_4 builds
These two builds are struggling to complete due to OOM issues. Reducing the
number of CPUs didn't help, so this temporarily disables Spicy from
building on those hosts to reduce the memory usage.
2023-10-09 11:57:16 -07:00
Tim Wojtulewicz
4c6c89d863 CI: Remove sanitizers resource template
This template was added when the sanitizer builds needed more memory than all
of the other builds. We've since increased the amount of memory that the other
builds use beyond what's requested in this template, so it doesn't make sense
to keep it around.
2023-10-09 10:56:16 -07:00
Arne Welzel
d08b472871 Updating zeek-aux submodule [nomail] 2023-10-09 16:47:23 +02:00
Arne Welzel
51cbdc53f3 NEWS: Polish external plugins message [nomail] [skip ci] 2023-10-09 16:35:11 +02:00
Arne Welzel
48b27f30c9 Merge remote-tracking branch 'origin/topic/awelzel/plugin-hooks-bare-mode'
* origin/topic/awelzel/plugin-hooks-bare-mode:
  btest/plugins/hooks: Run in bare mode
2023-10-09 16:27:10 +02:00
Arne Welzel
2ed89ff84f Bump cmake submodule for 3.15 requirement 2023-10-09 16:23:12 +02:00
Arne Welzel
07ac6fa074 btest/plugins/hooks: Run in bare mode
Motivation is basically the same as in 88bb527026.
For plugin.hooks, one example is that adding a new option in the default script
changes the baseline due registration of change handlers. Also, the connection
record is printed in various places, resulting in churn when the default
scripts change.
2023-10-09 16:13:59 +02:00
Arne Welzel
4a8f868471 Merge remote-tracking branch 'origin/topic/awelzel/dump-events-in-bare-mode'
* origin/topic/awelzel/dump-events-in-bare-mode:
  btest/coverage: Add record-fields test
  btest/dump-events: Run in bare mode, load conn and smtp scripts only
2023-10-09 14:15:49 +02:00
Arne Welzel
998ab80385 btest/coverage: Add record-fields test
Justin pointed out that the misc/dump-events test shows added fields to
the connection record. Add a new test that prints the connection record
recursively in bare and default mode to cover that use-case
specifically.
2023-10-09 13:58:08 +02:00
Arne Welzel
88bb527026 btest/dump-events: Run in bare mode, load conn and smtp scripts only
The dump-events baseline changes are pure noise and have spurred confusion
for internal and external contributors. For example, adding new
analyzers have perturbed orderings of sets holding analyzer tags.

Running in non-bare mode, the baselines change almost whenever any of the
record types attached to connections change in the default scripts. This
causes continuous and seemingly little useful updates to the baselines.

This change switches the test to run in bare mode and explicitly loads
just base/protocols/conn and base/protocols/smtp. The primary intention
of the test should be testing the functionality of the misc/dump-events
script, not the raised events of all loaded default scripts (for that the
used PCAP is too narrow).

Protocol specific scripts that do want to leverage misc/dump-events for
baseline creation of their or their analyzer's events can add additional
specific tests with suitable PCAP files.
2023-10-09 12:20:10 +02:00
zeek-bot
c0e209f870 Update doc submodule [nomail] [skip ci] 2023-10-07 01:02:22 +00:00
Benjamin Bannier
8793cde485 Merge remote-tracking branch 'origin/topic/bbannier/bump-spicy' 2023-10-06 16:09:05 +02:00
Benjamin Bannier
8d55a10ab0 Bump auxil/spicy to latest development snapshot 2023-10-06 12:40:45 +02:00
Christian Kreibich
26897bf906 Merge branch 'topic/christian/normalize-file-paths'
* topic/christian/normalize-file-paths:
  Btest for normalized reporter path reporting
  Normalize file paths in find_file_in_path()
2023-10-04 22:46:54 -07:00
Arne Welzel
0630bdbbd5 Merge remote-tracking branch 'origin/topic/awelzel/3314-lambda-redefinition-segfault'
* origin/topic/awelzel/3314-lambda-redefinition-segfault:
  Var/Func: Render function parameters using comma, not semicolon
  Var: Fix null-pointer deref on redefinition of lambdas
2023-10-04 20:16:29 +02:00
Arne Welzel
8109bbc52f Var/Func: Render function parameters using comma, not semicolon
A bit larger follow-up to what Tim pointed out: Function prototype descriptions
previously used semicolons to separate parameters.

Switch to use commas when a RecordType is used as function parameter.
Use existing "func_args" naming for consistency.
2023-10-04 17:42:30 +02:00
Arne Welzel
8ede22f6ec Var: Fix null-pointer deref on redefinition of lambdas
Closes #3314
2023-10-04 15:18:24 +02:00
Christian Kreibich
e84a7ab7f1 Btest for normalized reporter path reporting 2023-10-03 22:16:00 -07:00
Christian Kreibich
6108b18a3b Normalize file paths in find_file_in_path()
This renders script file names more nicely, for example when printed by the
reporter. It also avoids redundant prefixing of "./" to local scripts. I'm not
adding unit tests since normalize_path() already has them. A btest follows in
the next commit.
2023-10-03 22:14:40 -07:00
zeek-bot
e22bf8ebb6 Update doc submodule [nomail] [skip ci] 2023-10-03 01:28:03 +00:00
Tim Wojtulewicz
ff1d07b548 Merge remote-tracking branch 'origin/topic/awelzel/3330-get-current-packet-docs'
* origin/topic/awelzel/3330-get-current-packet-docs:
  get_current_packet: Add note/warning to docs
2023-10-02 11:51:07 -07:00