* origin/topic/jsiwek/gh-1024-broker-store-handle-type-checks:
Improve Broker store API's handling of invalid arguments
Add builtin_exception() functions
GH-1024: fix crash on passing wrong types to Broker store API
This was a bit of a bigger merge since Zeek changed inbetween the time
of the PR and me actually merging it.
I put the new functions into the zeek::detail namespace -- since it
seems unlikely that those will be used by something external.
I also renamed them to fit better with the naming scheme of the new
error functions.
Fixes GH-1024
Previously, a single `icmp_conn` record was built per ICMP "connection"
and re-used for all events generated from it. This may have been a
historical attempt at performance optimization, but:
* By default, Zeek does not load any scripts that handle ICMP events.
* The one script Zeek ships with that does handle ICMP events,
"detect-traceroute", is already noted as being disabled due to
potential performance problems of doing that kind of analysis.
* Re-use of the original `icmp_conn` record tends to misreport
TTL and length values since they come from original packet instead
of the current one.
* Even if we chose to still re-use `icmp_conn` records and just fill
in a new TTL and length value each packet, a user script could have
stored a reference to the record and not be expecting those values
to be changed out from underneath them.
Now, a new `icmp_info` record is created/populated in all ICMP events
and should be used instead of `icmp_conn`. It also removes the
orig_h/resp_h fields as those are redundant with what's already
available in the connection record.
Local frame offsets were being assigned based on number of the alternate
prototype's parameters, which may end up having less total parameters
than the canonical prototype, causing the local value to incorrectly
overwrite an event/hook argument value.
The location information now points out the place of the deprecated
prototype instead of the location where the ID was initially declared
(which may not itself be a deprecated prototype).
Particularly, this is meant for using &deprecated on canonical
event/hook prototype parameters to encourage users to create handlers
to another, non-deprecated prototype. i.e. for canonical prototypes,
we may not always want to put &deprecated directly on the prototype
itself since that signals deprecation of the ID entirely.
These may be redefined to customize log rotation path prefixes,
including use of a directory. File extensions are still up to
individual log writers to add themselves during the actual rotation.
These new also allow for some simplication to the default
ASCII postprocessor function: it eliminates the need for it doing an
extra/awkward rename() operation that only changes the timestamp format.
This also teaches the supervisor framework to use these new options
to rotate ascii logs into a log-queue/ directory with a specific
file name format (intended for an external archiver process to
monitor separately).
This helps prevent a node from being killed/crashing in the middle
of writing a log, restarting, and eventually clobbering that log
file that never underwent the rotation/archival process.
The old `archive-log` and `post-terminate` scripts as used by
ZeekControl previously implemented this behavior, but the new logic is
entirely in the ASCII writer. It uses ".shadow" log files stored
alongside the real log to help detect such scenarios and rotate them
correctly upon the next startup of the Zeek process.
The stdout/stderr of child processes is now redirected over a pipe back
to the supervisor process so that it can prefix the output with
the name of the emitting node.
Duplicate script `@load` directives are now detected by comparing
against canonical paths formed by realpath(). This fixes the previous,
unexpected behavior of treating scripts that hardlink to same
inode as duplicates: such links will now be loaded as distinct scripts
since their canonical path differs.
Changes during merge
- Changed the policy script to use an event handler that behaves
for like the base script: &priority=5, msg$opcode != early-out,
no record field existence checks
- Also extended dns_query_reply event with original_query param
- Removed ExtractName overload, and just use default param
* 'dns-original-query-case' of https://github.com/rvictory/zeek:
Fixed some places where tabs became spaces
Stricter checking if we have a dns field on the connection being processed
Modified the DNS protocol analyzer to add a new parameter to the dns_request event which includes the DNS query in its original case. Added a policy script that will add the original_case to the dns.log file as well. Created new btests to test both.
* Some methods mistakenly returned a bool instead of QueryResult
when passed an invalid `opaque of Broker::Store` handle.
* Now generates a runtime exception for store_name() and is_closed()
calls that pass an invalid `opaque of Broker::Store` handle as any
returned value can't be reasonably used in any subsequent logic.
* Descriptions of any invalid arguments are now given in the error
message.
* Variables of `string` type can now be set to an empty string
* Trying to set a variable with non-`string` type to an empty value
now emits an error instead of silently doing nothing
* Providing an invalid identifier now emits an "unknown identifier"
error instead of silently doing nothing
This introduces a new sampling state-map for expired connections to fix
segfaults that previously occured when passing in a `connection` record
to `Reporter::conn_weird()` for which the internal `Connection` object
had already been expired and deleted. This also introduces a new event
called `expired_conn_weird`, which is similar to `conn_weird`, except
the full `connection` record is no longer available, just the `conn_id`
and UID string.
Merge adjustments:
- Preserved original `base_type_no_ref` argument type as ::TypeTag
- Removed superfluous #pragma guard around deprecated TableVal ctor
- Clarify NEWS regarding MetaHook{Pre,Post} deprecations
- Simplify some `::zeek::` qualifications to just `zeek::`
- Prefixed FORWARD_DECLARE_NAMESPACED macro with ZEEK_
* origin/topic/timw/266-namespaces:
Disable some deprecation diagnostics for GCC
Rename BroType to Type
Update NEWS
Review cleanup
Move Type types to zeek namespace
Move Flare/Pipe from the bro namespace to zeek::detail
Move Attr to the zeek::detail namespace
Move Trigger into the zeek::detail namespace
Move ID to the zeek::detail namespace
Move Anon.h into zeek::detail namespace
Mark all of the aliased classes in plugin/Plugin.h deprecated, and fix all of the plugins that were using them
Move all of the base plugin classes into the zeek::plugin namespace
Expr: move all classes into zeek::detail
Stmt: move Stmt classes into zeek::detail namespace
Add utility macro for creating namespaced aliases for classes
Switches from pcap_next() to pcap_next_ex() to better handle all error
conditions. This allows, for example, to have a non-zero exit code for
a Zeek process that fails to fully process all packets in a pcap file.
For `DHCP::ClientID$hwtype` fields equal to 0, the `hwaddr` field is
no longer misformatted as a MAC and instead just contains the raw bytes
seen in the DHCP Client ID Option.
This also updates all usages of the deprecated Val ctor to use
either IntervalVal, TimeVal, or DoubleVal ctors. The reason for
doing away with the old constructor is that using it with TYPE_INTERVAL
isn't strictly correct since there exists a more specific subclass,
IntervalVal, with overriden ValDescribe() method that ought to be used
to print such values in a more descriptive way.
The BIF was not returning an IntervalVal which has an overriden
ValDescribe() method that allows for prettier printing like "6.0 secs"
instead of just "6.0".
* origin/topic/jsiwek/gh-893-intrusive-ptr-migration: (151 commits)
Integrate review feedback
Switch Broker Val converter visitor to return IntrusivePtr
Change BroFunc ctor to take const-ref IntrusivePtr<ID>
Add version of Frame::SetElement() taking IntrusivePtr<ID>
Change Scope/Func inits from id_list* to vector<IntrusivePtr<ID>>
Change Scope::GenerateTemporary() to return IntrusivePtr
Deprecate Scope::ReturnType(), replace with GetReturnType()
Deprecate Scope::ScopeID(), replace with GetID()
Switch parsing to use vector<IntrusivePtr<Attr>> from attr_list
Deprecate TableVal::FindAttr(), replace with GetAttr()
Deprecate TypeDecl::FindAttr(), replace with GetAttr()
Deprecate ID::FindAttr(), replace with GetAttr()
Deprecate Attributes::FindAttr(), replace with Find()
Deprecate Attributes::AddAttrs(Attributes*)
Add Attributes ctor that takes IntrusivePtrs
Change Attributes to store std:vector<IntrusivePtr<Attr>>
Change Attr::SetAttrExpr() to non-template
Deprecate Attr::AttrExpr(), replace with GetExpr()
Deprecate ID::Attrs(), replace with GetAttrs()
Remove weak_ref param from ID::SetVal()
...
- Updated the logic significantly: still filters out ICMP from being
considered an active service (like before) and adds a new
"Known::service_udp_requires_response" option (defaults to true) for
whether to require UDP server response before being considered an
active service.
* 'topic/dopheide/known-services' of https://github.com/dopheide-esnet/zeek:
Log services with unknown protocols
This also changes the argument type of Func::operator() to zeek::Args*
to allow plugins to be able to alter function arguments in place as
was previously documented.
