Currently this requires using this with a normal cluster - or sending
messages by yourself.
It, in principle, should also work with SQLITE - but that is a bit
nonsensical without being able to change the storage location.
The &backend attribute allows for a much more convenient way of
interacting with brokerstores. One does not need to create a broker
store anymore - instead all of this is done internally.
The current state of this partially works. This should work fine for
persistence - but clones are currently not yet correctly attached.
Changes during merge
- Changed the policy script to use an event handler that behaves
for like the base script: &priority=5, msg$opcode != early-out,
no record field existence checks
- Also extended dns_query_reply event with original_query param
- Removed ExtractName overload, and just use default param
* 'dns-original-query-case' of https://github.com/rvictory/zeek:
Fixed some places where tabs became spaces
Stricter checking if we have a dns field on the connection being processed
Modified the DNS protocol analyzer to add a new parameter to the dns_request event which includes the DNS query in its original case. Added a policy script that will add the original_case to the dns.log file as well. Created new btests to test both.
Previously, more data than could effectively be utilized by any remote
Zeek was published (e.g. full list of pending commands or other
transient state that may add up to non-trivial amount of bytes).
This introduces a new sampling state-map for expired connections to fix
segfaults that previously occured when passing in a `connection` record
to `Reporter::conn_weird()` for which the internal `Connection` object
had already been expired and deleted. This also introduces a new event
called `expired_conn_weird`, which is similar to `conn_weird`, except
the full `connection` record is no longer available, just the `conn_id`
and UID string.
* origin/topic/jsiwek/gh-893-intrusive-ptr-migration: (151 commits)
Integrate review feedback
Switch Broker Val converter visitor to return IntrusivePtr
Change BroFunc ctor to take const-ref IntrusivePtr<ID>
Add version of Frame::SetElement() taking IntrusivePtr<ID>
Change Scope/Func inits from id_list* to vector<IntrusivePtr<ID>>
Change Scope::GenerateTemporary() to return IntrusivePtr
Deprecate Scope::ReturnType(), replace with GetReturnType()
Deprecate Scope::ScopeID(), replace with GetID()
Switch parsing to use vector<IntrusivePtr<Attr>> from attr_list
Deprecate TableVal::FindAttr(), replace with GetAttr()
Deprecate TypeDecl::FindAttr(), replace with GetAttr()
Deprecate ID::FindAttr(), replace with GetAttr()
Deprecate Attributes::FindAttr(), replace with Find()
Deprecate Attributes::AddAttrs(Attributes*)
Add Attributes ctor that takes IntrusivePtrs
Change Attributes to store std:vector<IntrusivePtr<Attr>>
Change Attr::SetAttrExpr() to non-template
Deprecate Attr::AttrExpr(), replace with GetExpr()
Deprecate ID::Attrs(), replace with GetAttrs()
Remove weak_ref param from ID::SetVal()
...
- Updated the logic significantly: still filters out ICMP from being
considered an active service (like before) and adds a new
"Known::service_udp_requires_response" option (defaults to true) for
whether to require UDP server response before being considered an
active service.
* 'topic/dopheide/known-services' of https://github.com/dopheide-esnet/zeek:
Log services with unknown protocols
* origin/topic/timw/906-find-all-urls-regex:
Restore previous url scheme capture group
GH-906: Fix the regex in url.zeek to better match for find_all_urls
- Adjusted the formatting during merge
* 'set_to_regex-docs' of https://github.com/jlagermann/zeek:
added examples to set_to_regex comments Signed-ff-by: James Lagermann <james.lagermann@corelight.com>
This commit switches UID hashing from md5 to a highway hash. It also
moves the salt value out of the file plugin - and makes it
installation-specific instead - it is moved to the global namespace.
There now are digest hash functions to make "static"
installation-specific hashes that are stable over workers available to
everyone; hashes can be 64, 128 or 256 bits in size.
Due to the fact that we switch the file hashing algorithm, all file
hashes change.
The underlyigng algorithm that is used for hashing is highwayhash-128,
which is significantly faster than md5.
* origin/topic/seth/more-file-type-ident:
Adds a note in NEWS about the signature addition and restructuring
Fixes a small bug in one signature with a duplicate name.
Organized and added to the shipped file identification signatures.
- Added test case and adjusted whitespace in merge
* 'stats-logging-fix' of https://github.com/brittanydonowho/zeek:
Fixed stats.zeek to log all data before zeek terminates rather than return too soon
The "http_header" event now has an "original_name" parameter that allows
access to the original header name (the "name" parameter reamins the
same as before: it's the uppercased header name).
The "mime_header_rec" record type now also includes an "original_name"
field to similarly provide access to original header name in the
following events: "http_all_headers", "mime_one_header", and
"mime_all_headers".
* 'master' of https://github.com/mmguero-dev/zeek:
check for the existance of f?$conns in file_sniff event in policy/protocols/ssl/log-hostcerts-only.zeek
In using the corelight/bro-xor-exe-plugin (https://github.com/corelight/bro-xor-exe-plugin) I noticed this error when running the PCAP trace file in its tests directory:
1428602842.525435 expression error in /opt/zeek/share/zeek/policy/protocols/ssl/log-hostcerts-only.zeek, line 44: field value missing (X509::f$conns)
Examining log-hostcerts-only.zeek, I saw that although f$conns is being checked for length, it's not being checked to see if it exists first.
This commit changes "if ( |f$conns| != 1 )" to "if (( ! f?$conns ) || ( |f$conns| != 1 ))" so that the script returns if there is no f$conns field.
In my local testing, this seems to fix the error. My testing was being done with v3.0.5, but I think this patch can be applied to both the 3.0.x and 3.1.x branches.
Node-specific topic prefix subscriptions/publications now add a trailing
slash like "zeek/cluster/node/<name>/". Without the trailing slash,
messages attempting to target "proxy-10" may also be sent to "proxy-1"
since subscription matching is prefix-based.
For event/hook handlers that had a previous declaration, any &default
arguments are ineffective. Only &default uses in the initial
prototype's arguments have an effect (that includes if the handler
is actually the site at which the declaration occurs).
This controls whether ports given by "udp_content_delivery_ports_orig" and
"udp_content_delivery_ports_orig" are in terms of the UDP packet's
destination port or by the Connection's "responder" port (the former is
the unchanged default behavior).
- Squashed the original commit set
- Cleaned up formatting
- Fixed register_for_ports() for right RDPEUDP analyzer
* topic/ak/rdpeudp:
Add RDP over UDP analyzer
