Commit graph

11634 commits

Author SHA1 Message Date
Jon Siwek
8561c79363 Remove inline from some static KeyedHash members
Coverity Scan builds currently encounter catastrophic error, claiming
alignas requires use on both declaration and definition, so appears to
actually not understand "static inline" in combo with alignas.
2020-06-05 18:20:05 -07:00
Jon Siwek
2893eea045 Improve Func.h inclusion
Now forward declares some Broker types since Broker/CAF headers
generally slow things down and also Coverity Scan currently has a
catastrophic error on some CAF headers.

Also a few other changes to EventHandler/BifReturnVal to reduce number
of places that depend on Func.h.
2020-06-05 17:57:42 -07:00
Johanna Amann
65c12ba6e9 Zeek/Brokerstore updates: first working end-to-end test
This commit fixes a few more loose ends to actually make the
Zeek Table<->brokerstore syncing work. This mostly slightly changes the
TableVal assign/remove operators to prevent loops when a remote change
arrives.

The tests inserts a value into a table on the manager, and it pops out
in a table on a clone - which is the easiest case.

Timeouts are still not handled at all; the behavior when inserting into
a clone is untested.
2020-06-05 16:17:05 -07:00
Johanna Amann
62f208086c Update to recent broker changes.
Specifically the store name is now part of the messages.
2020-06-05 14:33:45 -07:00
Johanna Amann
9d9aefaec3 Merge remote-tracking branch 'origin/master' into topic/johanna/table-changes
* origin/master:
  Fix use-after-move of proc_status_file breaking -U flag
  Change Cirrus CI config to use org-level secured variable
  Deprecate Val(double, TypeTag) ctor, add TimeVal/DoubleVal subclasses
  GH-973: Fix the return type of topk_get_top()
2020-06-05 14:29:14 -07:00
Jon Siwek
e532335f97 Fix NVT analyzer memory leak from multiple telnet authn name options
Credit to OSS-Fuzz for discovery
https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=23069
2020-06-04 16:54:08 -07:00
Jon Siwek
23ad81400b Rename aux/ to auxil/
Since "aux" is not an allowed file/dir name on Windows.
2020-06-04 15:18:44 -07:00
Jon Siwek
ef6bd3ee39 Fix use-after-move of proc_status_file breaking -U flag 2020-06-04 15:13:10 -07:00
Jon Siwek
88654fa989 Change Cirrus CI config to use org-level secured variable 2020-06-04 10:54:06 -07:00
Johanna Amann
f080c8c294 Broker changes: adopt to recent IntrusivePtr API changes 2020-06-03 16:48:16 -07:00
Johanna Amann
d9de6c4522 Merge remote-tracking branch 'origin/master' into topic/johanna/table-changes
* origin/master: (172 commits)
  GH-985: Fix descriptions of double_to_interval() return values
  Decrease number of CPUs/memory for Cirrus CI tasks
  Change CI script to compile from build/Makefile
  Add missing include in util.cc
  Remove Analyzer.h from bro-bif.h
  Remove IPAddr.h from Reporter.h
  Remove the inclusion of Func.h from NetVar.h, which reduces the inclusion of Func.h overall.
  Update submodule(s)
  Update submodule(s)
  Integrate review feedback
  Update submodule(s)
  Fix crash on using some deprecated environment variables
  Update NEWS
  Update test baselines for new Broker connection status/error strings
  Switch Broker Val converter visitor to return IntrusivePtr
  Change BroFunc ctor to take const-ref IntrusivePtr<ID>
  Add version of Frame::SetElement() taking IntrusivePtr<ID>
  Change Scope/Func inits from id_list* to vector<IntrusivePtr<ID>>
  Change Scope::GenerateTemporary() to return IntrusivePtr
  Deprecate Scope::ReturnType(), replace with GetReturnType()
  ...
2020-06-03 16:09:42 -07:00
Tim Wojtulewicz
4a1b39a2be Move Flare/Pipe from the bro namespace to zeek::detail 2020-06-03 15:16:19 -07:00
Tim Wojtulewicz
60ed29c3b6 Move Attr to the zeek::detail namespace 2020-06-03 15:16:19 -07:00
Tim Wojtulewicz
54233ce596 Move Trigger into the zeek::detail namespace 2020-06-03 15:16:19 -07:00
Tim Wojtulewicz
0d623d003c Move ID to the zeek::detail namespace 2020-06-03 15:16:19 -07:00
Tim Wojtulewicz
9992ec5c11 Move Anon.h into zeek::detail namespace 2020-06-03 15:16:19 -07:00
Tim Wojtulewicz
7a5dae4354 Mark all of the aliased classes in plugin/Plugin.h deprecated, and fix all of the plugins that were using them 2020-06-03 15:16:18 -07:00
Tim Wojtulewicz
e77e8c4b7b Move all of the base plugin classes into the zeek::plugin namespace 2020-06-03 15:16:18 -07:00
Tim Wojtulewicz
cbdb8ee074 Expr: move all classes into zeek::detail 2020-06-03 15:16:18 -07:00
Tim Wojtulewicz
3fa4acc175 Stmt: move Stmt classes into zeek::detail namespace 2020-06-03 13:37:57 -07:00
Tim Wojtulewicz
f952acaddc Add utility macro for creating namespaced aliases for classes 2020-06-03 13:37:56 -07:00
Tim Wojtulewicz
b4aa5d9456 Merge remote-tracking branch 'origin/topic/jsiwek/gh-973-fix-topk-get-top-return-type'
* origin/topic/jsiwek/gh-973-fix-topk-get-top-return-type:
  GH-973: Fix the return type of topk_get_top()
2020-06-03 13:29:31 -07:00
Tim Wojtulewicz
640ad2d010 Merge remote-tracking branch 'origin/topic/jsiwek/double-val-subclasses'
* origin/topic/jsiwek/double-val-subclasses:
  Deprecate Val(double, TypeTag) ctor, add TimeVal/DoubleVal subclasses
2020-06-03 13:20:04 -07:00
Jon Siwek
5b4313b593 Deprecate Val(double, TypeTag) ctor, add TimeVal/DoubleVal subclasses
This also updates all usages of the deprecated Val ctor to use
either IntervalVal, TimeVal, or DoubleVal ctors.  The reason for
doing away with the old constructor is that using it with TYPE_INTERVAL
isn't strictly correct since there exists a more specific subclass,
IntervalVal, with overriden ValDescribe() method that ought to be used
to print such values in a more descriptive way.
2020-06-02 23:33:40 -07:00
Jon Siwek
49fc7924e0 GH-973: Fix the return type of topk_get_top()
e.g. when using `string` elements it should return `vector of string`,
not `vector of list of string` like it did before.
2020-06-02 19:42:12 -07:00
Jon Siwek
a431f6b45d Merge remote-tracking branch 'origin/topic/jsiwek/gh-985-double-to-interval-descs'
* origin/topic/jsiwek/gh-985-double-to-interval-descs:
  GH-985: Fix descriptions of double_to_interval() return values
2020-06-02 18:14:20 -07:00
Jon Siwek
e17487e799 GH-985: Fix descriptions of double_to_interval() return values
The BIF was not returning an IntervalVal which has an overriden
ValDescribe() method that allows for prettier printing like "6.0 secs"
instead of just "6.0".
2020-06-02 16:47:11 -07:00
Jon Siwek
7b0931058e Decrease number of CPUs/memory for Cirrus CI tasks
New concurrency limits are 16 Linux CPUs per user rather than 8 Linux
Tasks per user (at a max of 8 CPUs per Task, they essentially allowed 64
CPUs per user before the change in policy).
2020-06-02 12:40:26 -07:00
Jon Siwek
3b51d72aa1 Fix potential stack overflow in NVT analyzer
The NVT_Analyzer (e.g. as instantiated to support the FTP analyzer)
uses a recursive parsing function that may only advance one byte at a
time and can easily cause a stack overflow as a result.  This change
replaces the recursive calls with equivalent iterative logic.

Credit to OSS-Fuzz for discovery
https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=22898
https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=22972
2020-06-02 11:13:16 -07:00
Jon Siwek
e1f35c46f9 Change CI script to compile from build/Makefile
Helps some `make` versions properly use the `-j` option
2020-06-02 10:56:48 -07:00
Jon Siwek
1a60fb7c0d Add missing include in util.cc 2020-06-01 20:45:37 -07:00
Jon Siwek
34a1875e74 Merge remote-tracking branch 'origin/topic/timw/reduce-func-inclusion'
- Minor tweaks to some odd includes during merge

* origin/topic/timw/reduce-func-inclusion:
  Remove Analyzer.h from bro-bif.h
  Remove IPAddr.h from Reporter.h
  Remove the inclusion of Func.h from NetVar.h, which reduces the inclusion of Func.h overall.
2020-06-01 19:26:55 -07:00
Jon Siwek
0195880c48 Fix multiple content-transfer-encoding headers causing a memory leak
The MIME analyzer may also leak if it sees many multipart boundary
parameters.

Credit to OSS-Fuzz for discovery
https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=22871
2020-06-01 16:21:44 -07:00
Tim Wojtulewicz
b75d2a31cf Remove Analyzer.h from bro-bif.h 2020-06-01 15:00:40 -07:00
Tim Wojtulewicz
c558d771e4 Remove IPAddr.h from Reporter.h 2020-06-01 15:00:39 -07:00
Tim Wojtulewicz
ea3c679101 Remove the inclusion of Func.h from NetVar.h, which reduces the inclusion of Func.h overall. 2020-06-01 15:00:35 -07:00
Tim Wojtulewicz
11ecc7d7eb Add new CI task for running benchmarks on a remote host
- Adds centos-8 docker instance to the CI configuration
- Adds new benchmark.sh script for passing a build artifact to a remote host to run benchmarks
2020-06-01 14:46:50 -07:00
Jon Siwek
5bcdc46272 Update submodule(s)
[nomail]
2020-06-01 12:17:29 -07:00
Tim Wojtulewicz
fcb28c54d2 Update submodule(s)
[nomail]
2020-06-01 11:00:13 -07:00
Tim Wojtulewicz
503ef26a17 Merge remote-tracking branch 'origin/topic/jsiwek/gh-893-intrusive-ptr-migration'
* origin/topic/jsiwek/gh-893-intrusive-ptr-migration: (151 commits)
  Integrate review feedback
  Switch Broker Val converter visitor to return IntrusivePtr
  Change BroFunc ctor to take const-ref IntrusivePtr<ID>
  Add version of Frame::SetElement() taking IntrusivePtr<ID>
  Change Scope/Func inits from id_list* to vector<IntrusivePtr<ID>>
  Change Scope::GenerateTemporary() to return IntrusivePtr
  Deprecate Scope::ReturnType(), replace with GetReturnType()
  Deprecate Scope::ScopeID(), replace with GetID()
  Switch parsing to use vector<IntrusivePtr<Attr>> from attr_list
  Deprecate TableVal::FindAttr(), replace with GetAttr()
  Deprecate TypeDecl::FindAttr(), replace with GetAttr()
  Deprecate ID::FindAttr(), replace with GetAttr()
  Deprecate Attributes::FindAttr(), replace with Find()
  Deprecate Attributes::AddAttrs(Attributes*)
  Add Attributes ctor that takes IntrusivePtrs
  Change Attributes to store std:vector<IntrusivePtr<Attr>>
  Change Attr::SetAttrExpr() to non-template
  Deprecate Attr::AttrExpr(), replace with GetExpr()
  Deprecate ID::Attrs(), replace with GetAttrs()
  Remove weak_ref param from ID::SetVal()
  ...
2020-06-01 10:58:02 -07:00
Jon Siwek
9c133b9b10 Integrate review feedback
* Add deprecation for MIME_Entity::ContentType(), use GetContentType()

* Add deprecation for MIME_Entity::ContentSubType(), use GetContentSubType()

* Add deprecation for MIME_Message::BuildHeaderVal(), use ToHeaderVal()

* Add deprecation for MIME_Message::BuildHeaderTable(), use ToHeaderTable()

* Add deprecation for mime::new_string_val(), use mime::to_stringval()

* Add deprecation for ARP_Analyzer::ConstructAddrVal(), use ToAddrVal()

* Add deprecation for ARP_Analyzer::EthAddrToStr(), use ToEthAddrStr()

* Change the Func::Call() replacement to be named Func::Invoke()
2020-05-29 19:14:35 -07:00
Johanna Amann
ebb106c9b8 Store updates: downlevel and store table elements as keys.
In this commit we change our approach and just store table keys as
broker keys and table values as broker indexes. This means we only can
have a single table in a broker store.

This seems to work perfectly - and leads to vastly less complex code.

Code should work, but is actually pretty untested. Might break with
certain types. Does not yet handle item expiration.

Also - this has the tiny issue that all remote operations currently lead
to loops (the operation will immediately be sent to the broker-store
again), which is not quite optimal.
2020-05-29 17:56:11 -07:00
Jon Siwek
0fc59c357f Update submodule(s)
[nomail]
2020-05-29 17:34:05 -07:00
Jon Siwek
2f918ed9b2 Merge branch 'topic/dopheide/known-services' of https://github.com/dopheide-esnet/zeek
- Updated the logic significantly: still filters out ICMP from being
  considered an active service (like before) and adds a new
  "Known::service_udp_requires_response" option (defaults to true) for
  whether to require UDP server response before being considered an
  active service.

* 'topic/dopheide/known-services' of https://github.com/dopheide-esnet/zeek:
  Log services with unknown protocols
2020-05-29 17:19:47 -07:00
Johanna Amann
558e89b3ba Broker Store updates: get a bit more infrastructure in place.
This compiles, but besides giving debug messages (and partially
performing inserts/updates) it is not really helpful and definitely WIP.

This also shows that I might have to re-think the approach that we will
take here. So far, we actually insert tables as tables into
Brokerstores. This opens up the potential to just have several tables
synchronized via a single brokerstore.

However, it turns out, that the current store_event API sends the
complete table with each update. Which is problematic for obvious
reasons - and not really sustainable.
2020-05-29 15:28:42 -07:00
Jon Siwek
d2eb701b7e Fix potential leak of Analyzers added to tree during Analyzer::Done
Credit to OSS-Fuzz for discovery
https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=22630
2020-05-28 16:23:58 -07:00
Jon Siwek
1c08be1c0f Fix crash on using some deprecated environment variables
If the global Reporter hasn't been created before trying to use a
deprecated environment variable, emit the warning to stderr directly
instead of via Reporter.

Fixes GH-989
2020-05-28 15:24:25 -07:00
Jon Siwek
138c9402c3 Update NEWS 2020-05-28 14:51:08 -07:00
Jon Siwek
54e40d4544 Merge branch 'topic/ak/rdp-nego-flags' of https://github.com/anthonykasza/zeek
- Added check in rdp_connect_request before accessing member of the
  switch since it wasn't necessarily the case that got parsed (guard
  against nullptr deref).

* 'topic/ak/rdp-nego-flags' of https://github.com/anthonykasza/zeek:
  change: now using alternate event prototypes
  add: two new RDP scriptland events for negotiation flags
2020-05-28 14:33:12 -07:00
Jon Siwek
8c03e47b12 Update test baselines for new Broker connection status/error strings 2020-05-28 14:11:03 -07:00