Commit graph

15560 commits

Author SHA1 Message Date
Arne Welzel
3b859f3def Merge remote-tracking branch 'origin/topic/christian/gh-3263-table-constructor-index'
* origin/topic/christian/gh-3263-table-constructor-index:
  Avoid assertion/tag error on invalid table constructor index expression
2023-09-15 11:57:22 +02:00
Arne Welzel
30dc6af831 Merge remote-tracking branch 'origin/topic/bbannier/issue-3279'
* origin/topic/bbannier/issue-3279:
  Fix handling of module scope when checking exported Spicy types for collisions
2023-09-15 11:57:04 +02:00
Arne Welzel
bdb085af5f CMakeLists: Set Python3_EXECUTABLE hint for spicy 2023-09-15 11:39:21 +02:00
Arne Welzel
e424b383cd Update zeekjs submodule
Primarily for the following fixes:

	4dfc0dd Nodejs: Do not use UV_RUN_DEFAULT during InitPostScript()
	ade764e Nodejs: Unset kNoInitializeCppgc for Node 20.6 or higher
	a38df85 Fix type check of parameters array with zeek.event
	deddadd Fix type check of parameters array with zeek.invoke
2023-09-15 11:33:37 +02:00
Johanna Amann
e18edfa452 Add extract_limit_includes_missing option for file extraction
Setting this option to false does not count missing bytes in files towards the
extraction limits, and allows to extract data up to the desired limit,
even when partial files are written.

When missing bytes are encountered, files are now written as sparse
files.

Using this option requires the underlying storage and utilities to support
sparse files.
2023-09-14 12:11:42 -07:00
Johanna Amann
9928f7efb7 File extraction: use fseek
In the past, we allocated a buffer with zeroes and wrote that with
fwrite. Now, instead we just fseek to the correct offset.

This changes the way in which the file extract limit is counted a bit;
skipped bytes do no longer count against the file size limit.

(cherry picked from commit 5071592e9b7105090a1d9de19689c499070749d4)
2023-09-14 12:11:37 -07:00
Tim Wojtulewicz
5934e143aa Revert "Add extract_limit_includes_missing option for file extraction"
This reverts commit f4d0fdcd5c.
2023-09-14 12:10:40 -07:00
Christian Kreibich
87874a62d1 Avoid assertion/tag error on invalid table constructor index expression
If an index expression in a table constructor isn't a list, don't abort with an
assertion failure (on debug builds) or a bad tag check. Instead, mark the
constructor expression erroneous and return gracefully. The following...

  global a_table: table[subnet] of string = {
        1.2.3.4/24 = "unspecified",    # should have been [1.2.3.4/24] = "unspecified",
  };

...now yields:

  $ zeek ./test.zeek
  error in ././test.zeek, line 2: table constructor index is not a list (1.2.3.0/24 = unspecified)
  error in ././test.zeek, line 2: type clash in assignment (a_table = table(1.2.3.0/24 = unspecified))
2023-09-13 23:10:44 -07:00
Benjamin Bannier
4f0f22ec78 Fix handling of module scope when checking exported Spicy types for collisions
When checking exported Spicy types for collisions with existing Zeek
types we previously would also check whether they collide with names in
global scope, i.e., we didn't provide a `no_global` arg to
`detail::lookup_ID` which defaulted to false (since we also provided a
module name I'd argue that the behavior of that function is confusing
and probably error-prone -- like seen here).

This meant that e.g., a Spicy enum `foo::Direction` (automatically in
implicit Spicy module scope) would be detected to collide with the
existing Zeek `Direction` enum.

With this patch we use the `lookup_ID` API correctly and do not check
against potential collisions with globals anymore since it is not
needed.

Closes #3279.
2023-09-13 16:33:07 +02:00
zeek-bot
17347df036 Update doc submodule [nomail] [skip ci] 2023-09-13 00:46:38 +00:00
Tim Wojtulewicz
8c559915f7 Merge remote-tracking branch 'origin/topic/timw/fix-master-tests'
* origin/topic/timw/fix-master-tests:
  Update dump-events test baseline after latest master changes
2023-09-12 17:00:39 -07:00
Tim Wojtulewicz
dc23d7d3f3 Update dump-events test baseline after latest master changes 2023-09-12 15:34:38 -07:00
Johanna Amann
f4d0fdcd5c Add extract_limit_includes_missing option for file extraction
Setting this option to false does not count missing bytes in files towards the
extraction limits, and allows to extract data up to the desired limit,
even when partial files are written.

When missing bytes are encountered, files are now written as sparse
files.

Using this option requires the underlying storage and utilities to support
sparse files.

(cherry picked from commit afa6f3a0d3b8db1ec5b5e82d26225504c2891089)
2023-09-12 12:00:36 -07:00
Arne Welzel
b2c40a22cb ftp: Do not log non-pending commands
OSS Fuzz generated a CWD request and reply followed by very many EPRT
requests. This caused Zeek to re-log the CWD request and invoke `build_url_ftp()`
over and over again resulting in long processing times.

Avoid this scenario by not logging commands that aren't pending anymore.

(cherry picked from commit b05dd31667ff634ec7d017f09d122f05878fdf65)
2023-09-12 12:00:36 -07:00
Arne Welzel
f6e7ea43c3 http/smtp: Fix wrong character class usage
A call to `extract_filename_from_content_disposition()` is only
efficient if the string is guaranteed to contain the pattern that
is removed by `sub()`. Due to missing brackets around the `[:blank:]`
character class, an overly long string (756kb) ending in
"Type:dtanameaa=" matched the wrong pattern causing `sub()` to
exhibit quadratic runtime. Besides that, we may have potentially
extracted wrong information from a crafted header value.

(cherry picked from commit 6d385b1ca724a10444865e4ad38a58b31a2e2288)
2023-09-12 12:00:36 -07:00
Tim Wojtulewicz
9af7d830e7 VLAN: Fix length checking for non-Ethernet type 2 frames
(cherry picked from commit c579eb7b7cf3ca1d9008e74d7934155866c7e4f7)
2023-09-12 12:00:36 -07:00
Tim Wojtulewicz
f5a3da5945 Make sure that all sessions/connections are done before deleting plugins
(cherry picked from commit e29b499a211137cf3cf0c24a4ff82db1b806f132)
2023-09-12 12:00:36 -07:00
Arne Welzel
cca5a8ca5d Merge remote-tracking branch 'origin/topic/awelzel/deferred-default-non-const-v4'
* origin/topic/awelzel/deferred-default-non-const-v4:
  CreationInitsOptimizer: Use PreTypedef() instead of PreType()
  Fix deferred record initialization
  testing/btest: Un-deferred record initalization tests
2023-09-12 20:52:17 +02:00
Arne Welzel
6c76df4e10 CreationInitsOptimizer: Use PreTypedef() instead of PreType()
PreTypedef() does not require the PreID() trampoline and ensures
it's only called for IDs that are types. Also allows dropping
the const_cast<> due to id->GetType() returning a const TypePtr which
is different from a `const Type*`...
2023-09-12 20:23:46 +02:00
Arne Welzel
384e7e6b25 Fix deferred record initialization
Put RecordFieldInit instances into creation_inits during parsing and
determine their deferrability in an InitPostScript step. Any
RecordFieldInits can be deferred are moved into deferred_inits.

Closes #3260
2023-09-12 12:21:31 +02:00
Arne Welzel
7d6c8d7224 testing/btest: Un-deferred record initalization tests
These baselines should align with what Zeek 5.2 or before did.
2023-09-12 12:12:46 +02:00
zeek-bot
e7a747d3b2 Update doc submodule [nomail] [skip ci] 2023-09-12 00:31:02 +00:00
Tim Wojtulewicz
b783441428 Merge branch 'topic/timw/file-analyzer-dict-hash'
* topic/timw/file-analyzer-dict-hash:
  Move allocation of analyzer_hash to file_mgr
2023-09-11 10:16:05 -07:00
Tim Wojtulewicz
743d7e96f5 Move allocation of analyzer_hash to file_mgr 2023-09-11 10:15:38 -07:00
zeek-bot
607d72f7b9 Update doc submodule [nomail] [skip ci] 2023-09-09 00:38:25 +00:00
Arne Welzel
f4322f9584 Merge remote-tracking branch 'origin/topic/vern/script-opt-maint.Sep23'
* origin/topic/vern/script-opt-maint.Sep23:
  fix for ZAM statement-level profiling (broken by GH-3199)
  ZAM fixes for compatibility with GH-3249 changes
  -O gen-C++ fixes for compatibility with GH-3249 changes minor -O gen-C++ BTest updates
  minor BTest reordering to diminish differences with script optimization
2023-09-08 16:47:29 +02:00
Arne Welzel
02184625b0 Merge remote-tracking branch 'origin/topic/awelzel/3267-avoid-unnecessary-inits'
* origin/topic/awelzel/3267-avoid-unnecessary-inits:
  ID/Stmt: Introduce INIT_SKIP and use in ForStmt
  CompositeHash: Skip record initialization when recovering vals
2023-09-08 13:48:47 +02:00
Arne Welzel
cea7c0ab46 ID/Stmt: Introduce INIT_SKIP and use in ForStmt
Currently, loop vars are added to a function scope's inits and
initialized upon entering a function with default values. This
applies to vector, record and table types.

This is unnecessary for variables used in for loops as they are
guaranteed to be initialized while iterating.
2023-09-08 13:05:44 +02:00
Arne Welzel
aaa81cae5d CompositeHash: Skip record initialization when recovering vals
Initializing fields of recovered records caused running &default expression
of fields just so that they are re-assigned in the next step with the
recovered fields. The second test case still shows that the loop var
is initialized as well even though that's not needed.

Add tests for iterating over records with &default attributes for both,
tables and vectors.

Fixes #3267
2023-09-08 13:02:34 +02:00
Arne Welzel
057bc673a8 Merge remote-tracking branch 'origin/topic/bbannier/issue-3276'
* origin/topic/bbannier/issue-3276:
  Propagate failure reason in `spicy::decline_input`/`zeek::reject_protocol`.
2023-09-07 16:18:20 +02:00
Arne Welzel
2fd83acc49 Merge remote-tracking branch 'origin/topic/awelzel/3275-fix-with-python'
* origin/topic/awelzel/3275-fix-with-python:
  ci: update-alternative to have python3 be python3.9
  Add OpenSUSE Leap 15.5 to ci
  configure/CMakelists: Use Python_EXECUTABLE as hint for interpreter
2023-09-07 14:28:16 +02:00
Arne Welzel
6e6a2bee8a Merge remote-tracking branch 'origin/topic/awelzel/3278-spicy-fix-port-range-off-by-one'
* origin/topic/awelzel/3278-spicy-fix-port-range-off-by-one:
  spicy: Do not register port N+1 for port N in .evt file
2023-09-07 13:29:03 +02:00
Arne Welzel
d7b9d70fa8 ci: update-alternative to have python3 be python3.9
Tweak so that python3 is correct for the Python_FIND_UNVERSIONED_NAMES
change. python3 on 15.4 was pointing at Python3.6 still and for that
there are no dev headers installed.

A normal user could use --with-python instead, but in CI this is easier.
2023-09-07 13:03:03 +02:00
Arne Welzel
8cd2eceed1 spicy: Do not register port N+1 for port N in .evt file
Closes #3278
2023-09-07 12:58:34 +02:00
Johanna Amann
4ad2a6ab9a Add OpenSUSE Leap 15.5 to ci
This also adds 15.4 to the list of distributions that are skipped by
default - let me know if this is not a good idea. I assume that by now
most people will be running 15.5 (15.4 will be EOL in ~3 months).
2023-09-07 12:48:46 +02:00
Arne Welzel
09b7caf986 configure/CMakelists: Use Python_EXECUTABLE as hint for interpreter
The PYTHON_EXECUTABLE variable was only interpreted by our own FindPythonDev.
cmake's FindPython uses Python_EXECUTABLE instead.

Also, if --with-python is not used, prefer unversioned executables that
match the version requirement. Further, remove Python_ADDITIONAL_VERSIONS
that is interpreted by FindPythonInterp which we're not using anymore.

https://cmake.org/cmake/help/latest/module/FindPython.html#artifacts-specification
https://cmake.org/cmake/help/latest/module/FindPythonInterp.html

Closes #3275
2023-09-07 12:48:37 +02:00
Christian Kreibich
21562cfbb2 Updating auxil/zeek-client submodule [nomail] 2023-09-06 23:26:25 -07:00
zeek-bot
151a4999a3 Update doc submodule [nomail] [skip ci] 2023-09-07 00:22:26 +00:00
Benjamin Bannier
d7c3e0ac69 Propagate failure reason in spicy::decline_input/zeek::reject_protocol.
Closes #3276.
2023-09-06 10:32:11 +02:00
Arne Welzel
1441b83411 Merge remote-tracking branch 'origin/topic/awelzel/dns-mgr-fixes'
* origin/topic/awelzel/dns-mgr-fixes:
  DNS_Mgr: Use Process() for timeout expiration
  DNS_Mgr: Fix GetNextTimeout() returning absolute values
2023-09-05 19:57:10 +02:00
Arne Welzel
a57c45428f DNS_Mgr: Use Process() for timeout expiration
DNS_Mgr has a GetNextTimeout() implementation that may return 0.0. When
that is the case, its IO source is enqueued as ready with an fd of -1.
This in turn results in Process() being called instead of ProcessFd()
in RunState.cc.

Ensure timeouts behavior is properly handled by actually forwarding
timeout indications to c-ares via DNS_Mgr::Process(). This results
in pending DNS queries for which a timeout happened to actually
timeout (when there's no other connectivity).
2023-09-04 18:09:40 +02:00
Arne Welzel
62c06f7e3a DNS_Mgr: Fix GetNextTimeout() returning absolute values
Not sure, must have been some sort of left-over, but wasn't really
effective due to Process() not being implemented.
2023-09-04 17:50:15 +02:00
Arne Welzel
7e11501d3c Merge remote-tracking branch 'origin/topic/jazoff/gh-3268t '
* origin/topic/jazoff/gh-3268:
  Fix check for emailed notices

Changes: Added a test-case printing email_delay_tokens to compare email vs
non-email notice types. Previously, both notice types would have email
delay tokens at that point in the flow.
2023-09-04 14:01:23 +02:00
zeek-bot
7a04dad95c Update doc submodule [nomail] [skip ci] 2023-09-02 00:20:12 +00:00
Vern Paxson
a36c06e5cd fix for ZAM statement-level profiling (broken by GH-3199) 2023-09-01 12:20:46 -07:00
Vern Paxson
e78570ad89 ZAM fixes for compatibility with GH-3249 changes 2023-09-01 12:20:35 -07:00
Vern Paxson
80eed34985 -O gen-C++ fixes for compatibility with GH-3249 changes
minor -O gen-C++ BTest updates
2023-09-01 12:17:38 -07:00
Vern Paxson
85cd1d69a4 minor BTest reordering to diminish differences with script optimization 2023-09-01 12:17:06 -07:00
Justin Azoff
047232df37 Fix check for emailed notices
This bug was introduced in 9d0bd96
2023-09-01 13:23:45 -04:00
Arne Welzel
14a2c02f9d Merge remote-tracking branch 'origin/topic/awelzel/1705-http-pending-requests'
* origin/topic/awelzel/1705-http-pending-requests:
  http: Prevent request/response de-synchronization and unbounded state growth
2023-09-01 11:54:10 +02:00