Commit graph

533 commits

Author SHA1 Message Date
Johanna Amann
2b2a40f49c Zeek Table<->Brokerstore: cleanup, documentation, small fixes
This commit adds script/c++ documentation and fixes a few loose ends.
It also adds tests for corner cases and massively improves error
messages.

This also actually introduces type-compatibility checking and introduces
a new attribute that lets a user override this if they really know what
they are doing. I am not quite sure if we should really let that stay in
- but it can be very convenient to have this functionality.

One test is continuing to fail - the expiry test is very flaky. This is,
I think, caused by delays of the broker store forwarding. I am unsure if
we can actually do anything about that.
2020-07-10 16:58:34 -07:00
Jon Siwek
6908d1b919 GH-1019: deprecate icmp_conn params for ICMP events
Previously, a single `icmp_conn` record was built per ICMP "connection"
and re-used for all events generated from it.  This may have been a
historical attempt at performance optimization, but:

  * By default, Zeek does not load any scripts that handle ICMP events.

  * The one script Zeek ships with that does handle ICMP events,
    "detect-traceroute", is already noted as being disabled due to
    potential performance problems of doing that kind of analysis.

  * Re-use of the original `icmp_conn` record tends to misreport
    TTL and length values since they come from original packet instead
    of the current one.

  * Even if we chose to still re-use `icmp_conn` records and just fill
    in a new TTL and length value each packet, a user script could have
    stored a reference to the record and not be expecting those values
    to be changed out from underneath them.

Now, a new `icmp_info` record is created/populated in all ICMP events
and should be used instead of `icmp_conn`.  It also removes the
orig_h/resp_h fields as those are redundant with what's already
available in the connection record.
2020-07-10 11:06:28 -07:00
Jon Siwek
7b15b82009 Merge remote-tracking branch 'origin/topic/timw/1040-str-split'
* origin/topic/timw/1040-str-split:
  GH-1040: Add zero-indexed version of str_split
2020-07-06 21:06:51 -07:00
Jon Siwek
76e67ff239 Merge remote-tracking branch 'origin/topic/timw/266-namespaces-part2'
* origin/topic/timw/266-namespaces-part2:
  Rename BroString files to ZeekString
  Update NEWS entry with note about class renames
  Rename BroObj to Obj
  Rename BroString to zeek::String
  Move Func up to zeek namespace, rename BroFunc to ScriptFunc
  Mark global val_mgr as deprecated and fix uses of it to use namespaced version
  Minor cleanup items from PR review
  Update binpac and bifcl submodules with review changes
  Move Location to zeek::detail and BroObj to zeek
  Move BroString to zeek namespace
  Move Dictionary/PDict, List/PList, and Queue/PQueue to zeek namespace
  Remove typedef that should have been removed in 3.1
  Move Func and associated classes into zeek::detail namespace
  Move Frame and Scope to zeek::detail namespace
  Move all Val classes to the zeek namespaces
  Use type aliases for IntrusivePtr definitions
  Move deprecation macro to zeek-config.h.in to avoid having to over-include util.h
  Move IntrusivePtr and utility methods to the zeek namespace
2020-07-02 19:25:13 -07:00
Tim Wojtulewicz
fbab72866c Update NEWS entry with note about class renames 2020-07-02 16:15:01 -07:00
Jon Siwek
4668378d91 Merge remote-tracking branch 'origin/topic/timw/1032-namespaced-enums'
Merge adjustments:
- Added back in deprecation tag for base_type_no_ref()
- Added back the deprecated plugin::hook_name() function

* origin/topic/timw/1032-namespaced-enums:
  Deprecate plugin::HookType and plugin::component::Type in a different way
  Deprecate init_class and IDScope in another way.
  Deprecate TypeTag and friends in a different way
  Deprecate attr_tag in a different way, rename to AttrTag
2020-06-30 16:55:36 -07:00
Tim Wojtulewicz
5b8aaf6497 Revert Attributes::Attrs back to return an attr_list and mark it deprecated 2020-06-30 10:39:03 -07:00
Tim Wojtulewicz
e1338cc379 GH-1034: Revert TypeList::Types() back to return a type_list* and mark it deprecated 2020-06-29 17:50:09 -07:00
Johanna Amann
af2110cfc9 Merge remote-tracking branch 'origin/topic/jsiwek/reduce-ftp-cluster-msg-sizes'
* origin/topic/jsiwek/reduce-ftp-cluster-msg-sizes:
  Minimize data published for expected FTP data channel analysis
2020-06-18 20:07:26 +00:00
Jon Siwek
d4f3cad7d1 Merge remote-tracking branch 'origin/topic/timw/266-namespaces'
Merge adjustments:

- Preserved original `base_type_no_ref` argument type as ::TypeTag
- Removed superfluous #pragma guard around deprecated TableVal ctor
- Clarify NEWS regarding MetaHook{Pre,Post} deprecations
- Simplify some `::zeek::` qualifications to just `zeek::`
- Prefixed FORWARD_DECLARE_NAMESPACED macro with ZEEK_

* origin/topic/timw/266-namespaces:
  Disable some deprecation diagnostics for GCC
  Rename BroType to Type
  Update NEWS
  Review cleanup
  Move Type types to zeek namespace
  Move Flare/Pipe from the bro namespace to zeek::detail
  Move Attr to the zeek::detail namespace
  Move Trigger into the zeek::detail namespace
  Move ID to the zeek::detail namespace
  Move Anon.h into zeek::detail namespace
  Mark all of the aliased classes in plugin/Plugin.h deprecated, and fix all of the plugins that were using them
  Move all of the base plugin classes into the zeek::plugin namespace
  Expr: move all classes into zeek::detail
  Stmt: move Stmt classes into zeek::detail namespace
  Add utility macro for creating namespaced aliases for classes
2020-06-11 23:20:51 -07:00
Tim Wojtulewicz
c04d9cae45 Update NEWS 2020-06-10 13:46:42 -07:00
Jon Siwek
5b4313b593 Deprecate Val(double, TypeTag) ctor, add TimeVal/DoubleVal subclasses
This also updates all usages of the deprecated Val ctor to use
either IntervalVal, TimeVal, or DoubleVal ctors.  The reason for
doing away with the old constructor is that using it with TYPE_INTERVAL
isn't strictly correct since there exists a more specific subclass,
IntervalVal, with overriden ValDescribe() method that ought to be used
to print such values in a more descriptive way.
2020-06-02 23:33:40 -07:00
Tim Wojtulewicz
503ef26a17 Merge remote-tracking branch 'origin/topic/jsiwek/gh-893-intrusive-ptr-migration'
* origin/topic/jsiwek/gh-893-intrusive-ptr-migration: (151 commits)
  Integrate review feedback
  Switch Broker Val converter visitor to return IntrusivePtr
  Change BroFunc ctor to take const-ref IntrusivePtr<ID>
  Add version of Frame::SetElement() taking IntrusivePtr<ID>
  Change Scope/Func inits from id_list* to vector<IntrusivePtr<ID>>
  Change Scope::GenerateTemporary() to return IntrusivePtr
  Deprecate Scope::ReturnType(), replace with GetReturnType()
  Deprecate Scope::ScopeID(), replace with GetID()
  Switch parsing to use vector<IntrusivePtr<Attr>> from attr_list
  Deprecate TableVal::FindAttr(), replace with GetAttr()
  Deprecate TypeDecl::FindAttr(), replace with GetAttr()
  Deprecate ID::FindAttr(), replace with GetAttr()
  Deprecate Attributes::FindAttr(), replace with Find()
  Deprecate Attributes::AddAttrs(Attributes*)
  Add Attributes ctor that takes IntrusivePtrs
  Change Attributes to store std:vector<IntrusivePtr<Attr>>
  Change Attr::SetAttrExpr() to non-template
  Deprecate Attr::AttrExpr(), replace with GetExpr()
  Deprecate ID::Attrs(), replace with GetAttrs()
  Remove weak_ref param from ID::SetVal()
  ...
2020-06-01 10:58:02 -07:00
Jon Siwek
9c133b9b10 Integrate review feedback
* Add deprecation for MIME_Entity::ContentType(), use GetContentType()

* Add deprecation for MIME_Entity::ContentSubType(), use GetContentSubType()

* Add deprecation for MIME_Message::BuildHeaderVal(), use ToHeaderVal()

* Add deprecation for MIME_Message::BuildHeaderTable(), use ToHeaderTable()

* Add deprecation for mime::new_string_val(), use mime::to_stringval()

* Add deprecation for ARP_Analyzer::ConstructAddrVal(), use ToAddrVal()

* Add deprecation for ARP_Analyzer::EthAddrToStr(), use ToEthAddrStr()

* Change the Func::Call() replacement to be named Func::Invoke()
2020-05-29 19:14:35 -07:00
Jon Siwek
138c9402c3 Update NEWS 2020-05-28 14:51:08 -07:00
Jon Siwek
102e58b80b Change Attributes to store std:vector<IntrusivePtr<Attr>>
This also changes the return type of Attributes::Attrs() from attr_list*
2020-05-26 13:05:24 -07:00
Jon Siwek
272db640aa Deprecate Plugin::HookCallFunction(), replace with HookFunctionCall()
This also changes the argument type of Func::operator() to zeek::Args*
to allow plugins to be able to alter function arguments in place as
was previously documented.
2020-05-22 21:01:38 -07:00
Jon Siwek
57a6069cd1 Deprecate file analyzer construction methods taking raw RecordVal*
Replaced with versions that instead take IntrusivePtr
2020-05-22 16:13:15 -07:00
Jon Siwek
a384bb8b81 Deprecate VectorVal::Lookup(), replace with At() 2020-05-21 17:03:46 -07:00
Jon Siwek
69533bcbc6 Switch VectorVal BroValUnion to store std::vector<IntrusivePtr<Val>>
This changes the return type of AsVector() from std::vector<Val*>*
2020-05-21 16:24:34 -07:00
Jon Siwek
40db09ccbf Deprecate VectorVal::Insert() taking raw Val*, use IntrusivePtr 2020-05-21 15:44:06 -07:00
Jon Siwek
de1e3d7d6d Deprecate VectorVal::Assign methods taking raw Val*
And adapt usages to pass in to alternate method taking IntrusivePtr
2020-05-21 15:31:04 -07:00
Jon Siwek
087a0f3636 Switch Func::Call(val_list*) back to returning Val*
And renamed the method returning IntrusivePtr to operator().
This corrects the deprecation process for Func::Call(val_list*).
2020-05-20 18:41:59 -07:00
Jon Siwek
85a0ddd62d Deprecate TableVal::Lookup(), replace with Find()/FindOrDefault() 2020-05-20 18:00:50 -07:00
Jon Siwek
7e89c8f0df Deprecate TableVal::Assign methods with Val*, add IntrusivePtr overloads 2020-05-20 12:46:51 -07:00
Jon Siwek
5bf2ed02d7 Deprecate RecordVal::Lookup(const char*, bool)
Replace with GetField(const char*) and GetFieldOrDefault(const char*).
2020-05-19 21:09:40 -07:00
Jon Siwek
2b4d80c849 Deprecate RecordVal::LookupWithDefault() replace with GetFieldOrDefault()
(The former was previously changed during this release cycle to return
Intrusive pointer, but this just changes it back to return Val* and
deprecates it).
2020-05-19 20:11:06 -07:00
Jon Siwek
f729247778 Deprecate RecordVal::Lookup(int), replace with GetField(int) 2020-05-19 18:19:58 -07:00
Jon Siwek
377779bb2a Change BroValUnion to use IntrusivePtr for record field storage
This also changes the AsRecord() and AsNonConstRecord() accessors
to return std::vector<IntrusivePtr<Val>>* instead of val_list*
2020-05-19 17:15:13 -07:00
Jon Siwek
f3d160d034 Deprecate RecordVal::Assign(int, Val*)
And adapt all usages to the existing overload taking IntrusivePtr.
2020-05-19 15:44:15 -07:00
Jon Siwek
83f1a911d7 Deprecate FuncType::ArgTypes(), replace with ParamList() 2020-05-15 19:15:24 -07:00
Jon Siwek
40153cc5cb Deprecate FuncType::Args(), replace with Params() 2020-05-15 19:04:31 -07:00
Jon Siwek
440b0623ac Deprecate RecordVal(RecordType*) ctor
Replaced with one that takes IntrusivePtr
2020-05-14 19:31:43 -07:00
Jon Siwek
4debad8caf Switch zeek:🆔:lookup to zeek:🆔:find
For parity with Scope since it now uses Find instead of Lookup
2020-05-14 18:00:18 -07:00
Jon Siwek
7843416e51 Use zeek::BifEvent:: for enqueue_ functions instead of BifEvent:: 2020-05-14 17:27:42 -07:00
Jon Siwek
0db5c920f2 Deprecate names in BifConst, replace with zeek::BifConst
Some Val* types are also replaced with IntrusivePtr at the new location
2020-05-14 17:26:00 -07:00
Jon Siwek
eedeb07550 Deprecate all BroType* in BifType:: namespace
Replaced with equivalently named IntrusivePtr in zeek::BifType::
2020-05-14 17:25:35 -07:00
Jon Siwek
8f95a2a0bb Deprecate Scope::Lookup(), replace with Scope::Find() 2020-05-14 17:24:21 -07:00
Jon Siwek
a5762c12cc Move various elements into ID.h and zeek::id namespace
* A handful of generic/useful/common global type pointers that used
  to be in NetVar.h

* Lookup functions that used to be Var.h
2020-05-14 17:24:20 -07:00
Jon Siwek
9210d443d3 Trim the list of "global type pointers" from NetVar.h further
Most of them are deprecated now, with usage sites now doing the lookup
themselves.
2020-05-14 17:23:20 -07:00
Jon Siwek
c0986f0739 Deprecate global type pointers in NetVar.h
There's analogous IntrusivePtrs in zeek::vars
2020-05-14 17:23:20 -07:00
Jon Siwek
1eb723fc9d Deprecate Val::Type(), replace with GetType() 2020-05-14 17:23:20 -07:00
Jon Siwek
447b052d11 Deprecate opt_internal_val() 2020-05-14 17:23:20 -07:00
Jon Siwek
26f6fe01c8 Deprecate internal_func(), replace with zeek::lookup_func() 2020-05-14 17:23:20 -07:00
Jon Siwek
a83941d64d Deprecate internal_val() and internal_const_val()
Replaced with zeek::lookup_val() and zeek::lookup_const()
2020-05-14 17:23:19 -07:00
Jon Siwek
ac06259eec Deprecate internal_type(), replace with zeek::lookup_type() 2020-05-14 17:22:26 -07:00
Jon Siwek
32b895f4ba Deprecate ID::ID_Val(), replace with ID::GetVal() 2020-05-14 17:18:00 -07:00
Jon Siwek
3f07c57523 Deprecate ID::Type(), replace with GetType() 2020-05-14 17:18:00 -07:00
Jon Siwek
6e647416d5 Deprecate ID::AsType(), add ID::IsType() and ID::GetType() 2020-05-14 17:18:00 -07:00
Jon Siwek
f26904e031 Deprecate BroType::YieldType(), replace with Yield() 2020-05-14 17:18:00 -07:00