This change adds two new hooks to the Intel framework that can be used
to intercept added and removed indicators and their type.
These hooks are fairly low-level. One immediate use-case is to count the
number of indicators loaded per Intel::Type and enable and disable the
corresponding event groups of the intel/seen scripts.
I attempted to gauge the overhead and while it's definitely there, loading
a file with ~500k DOMAIN entries takes somewhere around ~0.5 seconds hooks
when populated via the min_data_store store mechanism. While that
doesn't sound great, it actually takes the manager on my system 2.5
seconds to serialize and Cluster::publish() the min_data_store alone
and its doing that serially for every active worker. Mostly to say that
the bigger overhead in that area on the manager doing redundant work
per worker.
Co-authored-by: Mohan Dhawan <mohan@corelight.com>
* origin/topic/vern/ZAM-maint.May25:
fix for crash when interpreting transformed ASTs that include multi-field record assignments/additions
Remove unused ZAM compiler method
This only changes the script-layer API, but keeps the std::string host
in the C++ layer's ServerOptions. Mostly because the ixwebsocket library
takes host as std::string. Also, maybe at some point we'd want to
support something scheme-based like unix:///var/run/zeek.sock and placing
that in a string could not be totally wrong.
Add tests for IPV6, too.
* origin/topic/etyp/redis-analyzer:
spicy-redis: Add NEWS entry
spicy-redis: Separate error replies from success
spicy-redis: Cleanup scripts and tests
spciy-redis: Bring Redis analyzer into Zeek proper
spicy-redis: Abort parsing if server data comes first
spicy-redis: Add recursion depth to server data
spicy-redis: Make client data only accept bulk strings
spicy-redis: Add dpd signature and clean pcaps
spicy-redis: Add some commands and touch up parsing
spicy-redis: Add some script logic for logging
spicy-redis: Separate client/server
spicy-redis: Touchup logging and Spicy issues
spicy-redis: Add synchronization and pipeline support
spicy-redis: Begin Spicy Redis analyzer
- Recomputes checksums for pcaps to keep clean
- Removes some tests that had big pcaps or weren't necessary
- Cleans up scripting names and minor points
- Comments out Spicy code that causes a build failure now with a TODO to
uncomment it
Redis seems to only want client data first to request server data. The
DPD signature seems to pick up on some cases where server data comes
first, but is otherwise "valid" RESP. See if this helps lower FP rates.
* origin/topic/awelzel/4177-4178-custom-event-metadata-part-1:
Event: Move meta after args
Event: Use IntrusivePtr to manage obj refcount
btest/zam: Update for new EventMetadata bifs
broker and cluster: Switch to new Enqueue() API
Event/zeek.bif: Add EventMetadata current() and current_values() accessors
Event: Deprecate default network timestamp metadata
Event: Store timestamp in metadata vector
EventRegistry/zeek.bif/init-bare: Add event metadata infrastructure
EventMgr: Add CurrentEvent() accessor
This deprecates the Event constructor and the ``ts`` parameter of Enqueue()
Instead, versions are introduced that take a detail::MetadataVectorPtr which
can hold the network timestamp metadata and is meant to be allocated by the
caller instead of automatically during Enqueue() or within the Event
constructor.
This also introduces a BifConst ``EventMetadata::add_network_timestamp`` to
opt-in adding network timestamps to events globally. It's disabled by
default as there are not a lot of known use cases that need this.
