External plugins depend on the API for `OpaqueVal`. This set of changes
brings back the previous signature for the `Serialize` and `Unserialize`
member functions. The new set of functions that operate on the recently
added `BrokerData` API were renamed accordingly and use a `Data` suffix to
distinguish between the old and new interface.
For the transition period, `OpaqueVal` now has two "sets" of
serialization functions: old and new (using the suffix). By default, the
new functions call the old API and then convert to the new types. Hence,
plugins that override the old set of member functions will continue to
work. New code should only override the new set of functions.
Since the macro `DECLARE_OPAQUE_VALUE` (a convenience macro for adding a
default set of member functions to a subtype of `OpaqueVal`) might be
used by 3rd parties, the macro has been "restored" to its previous
behavior, i.e., it will override the old set of member functions. The
new macro `DECLARE_OPAQUE_VALUE_V2` is similar but overrides the new set
of functions instead.
The class `BloomFilter` uses the same member function signatures as
`OpaqueVal` for serialization. Hence, the same old/new split was
implemented to keep the APIs consistent.
This largely copies over Spicy's `.clang-format` configuration file. The
one place where we deviate is header include order since Zeek depends on
headers being included in a certain order.
* 'intrusive_ptr' of https://github.com/MaxKellermann/zeek: (32 commits)
Scope: store IntrusivePtr in `local`
Scope: pass IntrusivePtr to AddInit()
DNS_Mgr: use class IntrusivePtr
Scope: use class IntrusivePtr
Attr: use class IntrusivePtr
Expr: check_and_promote_expr() returns IntrusivePtr
Frame: use class IntrusivePtr
Val: RecordVal::LookupWithDefault() returns IntrusivePtr
Type: RecordType::FieldDefault() returns IntrusivePtr
Val: TableVal::Delete() returns IntrusivePtr
Type: base_type() returns IntrusivePtr
Type: init_type() returns IntrusivePtr
Type: merge_types() returns IntrusivePtr
Type: use class IntrusivePtr in VectorType
Type: use class IntrusivePtr in EnumType
Type: use class IntrusivePtr in FileType
Type: use class IntrusivePtr in TypeDecl
Type: make TypeDecl `final` and the dtor non-`virtual`
Type: use class IntrusivePtr in TypeType
Type: use class IntrusivePtr in FuncType
...
I moved the replay function to a callback - which now means that the replayed
functions are called before file_state remove. Which makes this virtually
identical with the events raised by the core.
Currently this is mostly missing tests, leak-testing and performance-tests.
This approach mostly relies on script-level changes. In scriptland, we track
which certificates should be cached - and also perform the caching and the
replaying of events there.
This actually is probably nearly functional - the problem is the fact that now
the event ordering is wrong - and file_state_remove is called before the x509
events are thrown.
The fix probably is to move to a callback from the core - which can execute
things in the right order again. (Or just write the whole event-raising inside
the core - but that is both less flexible and a lot more cumbersome).
We need this to sender through Broker, and we also leverage it for
cloning opaques. The serialization methods now produce Broker data
instances directly, and no longer go through the binary formatter.
Summary of the new API for types derived from OpaqueVal:
- Add DECLARE_OPAQUE_VALUE(<class>) to the class declaration
- Add IMPLEMENT_OPAQUE_VALUE(<class>) to the class' implementation file
- Implement these two methods (which are declated by the 1st macro):
- broker::data DoSerialize() const
- bool DoUnserialize(const broker::data& data)
This machinery should work correctly from dynamic plugins as well.
OpaqueVal provides a default implementation of DoClone() as well that
goes through serialization. Derived classes can provide a more
efficient version if they want.
The declaration of the "OpaqueVal" class has moved into the header
file "OpaqueVal.h", along with the new serialization infrastructure.
This is breaking existing code that relies on the location, but
because the API is changing anyways that seems fine.
This adds an internal BiF
"Broker::__opaque_clone_through_serialization" that does what the name
says: deep-copying an opaque by serializing, then-deserializing. That
can be used to tests the new functionality from btests.
Not quite done yet. TODO:
- Not all tests pass yet:
[ 0%] language.named-set-ctors ... failed
[ 16%] language.copy-all-opaques ... failed
[ 33%] language.set-type-checking ... failed
[ 50%] language.table-init-container-ctors ... failed
[ 66%] coverage.sphinx-zeekygen-docs ... failed
[ 83%] scripts.base.frameworks.sumstats.basic-cluster ... failed
(Some of the serialization may still be buggy.)
- Clean up the code a bit more.
All types (besides EntropyVal) now support a native copy operation,
which uses primitives of the underlying datatypes to perform a quick
copy, without serialization.
EntropyVal is the one exception - since that type is rather complex
(many members) and will probably not be copied a lot, if at all, it
makes sense to just use the serialization function.
This will have to be slightly re-written in the near-term-future to use
the new serialization function for that opaque type.
This change also introduces a new x509_from_der bif, which allows to
parse a der into an opaque of x509.
This change removes the d2i_X509_ wrapper function; this was a remnant
when d2i_X509 took non-const arguments. We directly use d2i_X509 at
several places assuming const-ness, so there does not seem to ba a
reason to keep the wrapper.
This change also exposed a problem in the File cache - cases in which an
object was brought back into the cache, and writing occurred in the
file_open event were never correctly handeled as far as I can tell.
Note - this compiles, but you cannot run Bro anymore - it crashes
immediately with a 0-pointer access. The reason behind it is that the
required clone functionality does not work anymore.
This changes many weird names to move non-static content from the
weird name into the "addl" field to help ensure the total number of
weird names is reasonably bounded. Note the net_weird and flow_weird
events do not have an "addl" parameter, so information may no longer
be available in those cases -- to make it available again we'd need
to either (1) define new events that contain such a parameter, or
(2) change net_weird/flow_weird event signature (which is a breaking
change for user-code at the moment).
Also, the generic handling of binpac exceptions for analyzers which
to not otherwise catch and handle them has been changed from a Weird
to a ProtocolViolation.
Finally, a new "file_weird" event has been added for reporting
weirdness found during file analysis.
This commit marks (hopefully) ever one-parameter constructor as explicit.
It also uses override in (hopefully) all circumstances where a virtual
method is overridden.
There are a very few other minor changes - most of them were necessary
to get everything to compile (like one additional constructor). In one
case I changed an implicit operation to an explicit string conversion -
I think the automatically chosen conversion was much more convoluted.
This took longer than I want to admit but not as long as I feared :)
With this change, we also parse signed certificate timestamps from OCSP
replies. This introduces a common base class between the OCSP and X509
analyzer, which now share a bit of common code. The event for signed
certificate timestamps is raised by both and thus renamed do:
x509_ocsp_ext_signed_certificate_timestamp
This is a tiny bit evil because it uses parts of the SSL protocol
analyzer in the X.509 certificate parser. Which is the fault of the
protocol, which replicates the functionality.
These changes should be safe -- testing the failure cases proves a bit
difficult at the moment due to the fact that OpenSSL seems to fix the
values that are present in the original ASN.1 before passing them on to
us. It is thus not directly easily possible to trigger the error cases
from scriptland.
This also means that a lot of the new error cases we try to catch here
can probably never happen.
* origin/fastpath:
Fix missing "irc-dcc-data" service field from IRC DCC connections.
Change X509 extension value parsing to not abort on malloc failures.
* origin/topic/bernhard/file-analysis-x509:
Forgot the preamble for the new leak test
(hopefully) last change -> return real opaque vec instead of any_vec
Fix dump-events - it cannot be used with ssl anymore, because openssl does not give the same string results in all versions.
Finishing touches of the x509 file analyzer.
Revert change to only log certificates once per hour.
Change x509 log - now certificates are only logged once per hour.
Fix circular reference problem and a few other small things.
X509 file analyzer nearly done. Verification and most other policy scripts work fine now.
Add verify functionality, including the ability to get the validated chain. This means that it is now possible to get information about the root-certificates that were used to secure a connection.
Second try on the event interface.
Backport crash fix that made it into master with the x509_extension backport from here.
Make x509 certificates an opaque type
rip out x509 code from ssl analyzer. Note that since at the moment the file analyzer does not yet re-populate the info record that means quite a lot of information is simply not available.
parse out extension. One event for general extensions (just returns the openssl-parsed string-value), one event for basicconstraints (is a certificate a CA or not) and one event for subject-alternative-names (only DNS parts).
Very basic file-analyzer for x509 certificates. Mostly ripped from the ssl-analyzer and the topic/bernhard/x509 branch.
work fine now.
Todo:
* update all baselines
* fix the circular reference to the fa_file structure I introduced :)
Sadly this does not seem to be entirely straightforward.
addresses BIT-953, BIT-760
chain. This means that it is now possible to get information about the
root-certificates that were used to secure a connection.
Intermediate commit before changing the script interface again.
addresses BIT-953, BIT-760
