Repeating the message for every new call to get_file_handle() is not
very useful. It's pretty much an analyzer configuration issue so logging
it once should be enough.
If DataIn() was called and a cur_entity_id (file_id) has been produced
previously, re-use it for calls to EndOfFile(). This avoids a costly
event_mgr.Drain() when we already have that information. It should be safer,
too, as `get_file_handle()` in script may generate a different ID and
thereby de-synchronizing.
I'm not sure if we somehow set this for oss-fuzz through the environment,
but didn't find anything obvious.
Running oss-fuzz reproducers locally can triggers lookups to malware.hash.cymru.com
and potentially other domains due to loading local.zeek.
When the parent of a support analyzer has been disabled, short-circuit
delivering stream or packet data to it.
The specific scenario this avoids is the Content-Line analyzer continuing
to feed data lines into an disabled SMTP analyzer in turn creating more
events.
This is primarily useful for our fuzzing setup where data chunks up to 1MB
are generated and fed into the analyzer pipeline. In the real-world, chunk
sizes are usually bounded to packet size. Certain TCP reassembly constellations
may trigger these scenarios, however.
Closes#168
OSS-Fuzz generated traffic containing a CWD command with a single very large
path argument (427kb) starting with ".___/` \x00\x00...", This is followed
by a large number of ftp replies with code 250. The directory logic in
ftp_reply() would match every incoming reply with the one pending CWD command,
triggering path buildup ending with something 120MB in size.
Protect from re-using a directory command by setting a flag in the
CmdArg record when it was consumed for the path traversal logic.
This doesn't prevent unbounded path build-up generally, but does prevent the
amplification of a single large command with very many small ftp_replies.
Re-using a pending path command seems like a bug as well.
* origin/topic/awelzel/install-spicyz-config-h:
Add license header to zeek-config*.h and zeek-version.h
Rename util-config.h to zeek-config-paths.h and install it
Ensure spicyz/config.h is installed
The util-config.h has never been installed previously. Skimming the history,
it was only meant for inclusion from util.cc, hence the name. Now that it's
included from some other headers, rename it to align with what it
contains and install it, too.
* origin/topic/robin/spicy-path-fixes:
Introduce environment variables to override more paths configured into `spicyz`.
Fix an ordering issue in Spicy support's CMake config.
* jgras/topic/jgras/cluster-active-node-count-fix:
Fix get_active_node_count for node types not present.
Changed over to explicit existence check instead to avoid the set()
creation upon missed lookups.
We're currently rendering parameter descriptions from .bif file into
the .rst as follows:
:cid: The connection identifier.
:aid: The analyzer ID.
Switch this to :param cid: instead so that we can have Sphinx deal with
this as param docfield and group all parameters into a single section.
Currently, having the bare :cid: style causes sphinx to treat it as an
unknown field type, capitalize it and render it.
This moves `auxil/spicy/spicy` to `auxil/spicy`. It relocates the
pieces previously inside that intermediary directory. The main change
is that now tweak the compilation flags for Spicy through target
options.
This is on top of `topic/robin/spicy-plugin`.
* origin/topic/robin/spicy-plugin:
Update doc submodule.
Remove unused file.
Move Spicy version information out of `zeek-config.h`.
Revert unnecessary change to plugin class.
Tweak format of `zkg` information in build info.
Remove left-over file.
Move `spicy/misc` scripts to policy and clarify purpose.
Do not load Spicy scripts if Spicy is not available.
Update comment.
Use ccache on CI when compiling Spicy analyzers.
Add `zkg_provides` to `--build-info`.
Fix tuple type conversion.
Fix generation of file IDs.
Integrate the Spicy plugin into Zeek proper.
Modernize plugin test.
This makes dependencies explicit that Zeek provides built-in for
`zkg`. It's in support of
https://github.com/zeek/package-manager/pull/157.
For now, `zkg_provides` contains the same data as `included_plugins`
plus an entry for `spicy-plugin`.
With an anonymous Spicy-side tuple type, we'd be missing an ID to
create a Zeek-side record, leading to undefined behavior. To still
support this case, we now make up an ID. In addition, we also could
end up not correctly tracking type ID during conversion; using a stack
now to handle recursion correctly.
We actually already have a test exercising the tuple code path, but it
took ASAN to spot the issue. Not sure how to test this beyond that.
Note: Need to backport this to the external plugin.
This reflects the `spicy-plugin` code as of `d8c296b81cc2a11`.
In addition to moving the code into Zeek's source tree, this comes
with a couple small functional changes:
- `spicyz` no longer tries to infer if it's running from the build
directory. Instead `ZEEK_SPICY_LIBRARY` can be set to a custom
location. `zeek-set-path.sh` does that now.
- ZEEK_CONFIG can be set to change what `spicyz -z` print out. This is
primarily for backwards compatibility.
Some further notes on specifics:
- We raise the minimum Spicy version to 1.8 (i.e., current `main`
branch).
- Renamed the `compiler/` subdirectory to `spicyz` to avoid
include-path conflicts with the Spicy headers.
- In `cmake/`, the corresponding PR brings a new/extended version of
`FindZeek`, which Spicy analyzer packages need. We also now install
some of the files that the Spicy plugin used to bring for testing,
so that existing packages keep working.
- For now, this all remains backwards compatible with the current
`zkg` analyzer templates so that they work with both external and
integrated Spicy support. Later, once we don't need to support any
external Spicy plugin versions anymore, we can clean up the
templates as well.
- All the plugin's tests have moved into the standard test suite. They
are skipped if configure with `--disable-spicy`.
This holds off on adapting the new code further to Zeek's coding
conventions, so that it remains easier to maintain it in parallel to
the (now legacy) external plugin. We'll make a pass over the
formatting for (presumable) Zeek 6.1.
* origin/topic/vern/event-tracing-update.May23B:
enhancements for event-tracing: - reporting of potentially sensitive constants - tracking of unsupported types enabling hand-editing to fix them - fixed generation of "unspecified" aggregates - fixed generation of IPv6 constants - fixed generation when running without a packet source
- reporting of potentially sensitive constants
- tracking of unsupported types enabling hand-editing to fix them
- fixed generation of "unspecified" aggregates
- fixed generation of IPv6 constants
- fixed generation when running without a packet source
For "individually different but deterministic" runs specifying Zeek's
seed as an environment variable eases setups as one can avoid creating
extra seed files for each of the individual processes.
It is an error to specify the new ZEEK_SEED_VALUES variable together
with the existing ZEEK_SEED_FILE and -G. ZEEK_SEED takes precedence over
deterministic mode (-D) like ZEEK_SEED_FILE does today already.
