Commit graph

6667 commits

Author SHA1 Message Date
Tim Wojtulewicz
4a1e17f4e0 Remove typedef that should have been removed in 3.1 2020-06-30 20:51:58 -07:00
Tim Wojtulewicz
d6f1ea16ac Move Func and associated classes into zeek::detail namespace 2020-06-30 20:51:58 -07:00
Tim Wojtulewicz
937a462e70 Move Frame and Scope to zeek::detail namespace 2020-06-30 20:51:53 -07:00
Tim Wojtulewicz
64332ca22c Move all Val classes to the zeek namespaces 2020-06-30 20:48:09 -07:00
Tim Wojtulewicz
ec9eff0bd5 Use type aliases for IntrusivePtr definitions 2020-06-30 20:37:30 -07:00
Tim Wojtulewicz
f6a251cdac Move deprecation macro to zeek-config.h.in to avoid having to over-include util.h 2020-06-30 20:19:14 -07:00
Tim Wojtulewicz
9364e6a5b7 Move IntrusivePtr and utility methods to the zeek namespace 2020-06-30 20:19:12 -07:00
Jon Siwek
4668378d91 Merge remote-tracking branch 'origin/topic/timw/1032-namespaced-enums'
Merge adjustments:
- Added back in deprecation tag for base_type_no_ref()
- Added back the deprecated plugin::hook_name() function

* origin/topic/timw/1032-namespaced-enums:
  Deprecate plugin::HookType and plugin::component::Type in a different way
  Deprecate init_class and IDScope in another way.
  Deprecate TypeTag and friends in a different way
  Deprecate attr_tag in a different way, rename to AttrTag
2020-06-30 16:55:36 -07:00
Johanna Amann
318a72c303 BrokerStore<->Zeek table - introdude &backend attribute
The &backend attribute allows for a much more convenient way of
interacting with brokerstores. One does not need to create a broker
store anymore - instead all of this is done internally.

The current state of this partially works. This should work fine for
persistence - but clones are currently not yet correctly attached.
2020-06-30 16:33:52 -07:00
Jon Siwek
35fa042fd6 Merge remote-tracking branch 'origin/topic/timw/1034-IndexTypes'
Minor tweaks during merge:
- Remove/default superfluous ~Attributes()
- Added in resize() calls to accompany reserve() of associated vectors

* origin/topic/timw/1034-IndexTypes:
  Revert Attributes::Attrs back to return an attr_list and mark it deprecated
  GH-1034: Revert TypeList::Types() back to return a type_list* and mark it deprecated
2020-06-30 15:56:49 -07:00
Tim Wojtulewicz
b1b1ec5171 Deprecate plugin::HookType and plugin::component::Type in a different way 2020-06-30 13:38:39 -07:00
Tim Wojtulewicz
7ee8e11a8f Deprecate init_class and IDScope in another way.
This also renamed init_class to InitClass for consistency.
2020-06-30 13:06:34 -07:00
Tim Wojtulewicz
361ad8f2f7 Deprecate TypeTag and friends in a different way 2020-06-30 12:48:07 -07:00
Tim Wojtulewicz
a280bfa0b1 Deprecate attr_tag in a different way, rename to AttrTag 2020-06-30 12:47:54 -07:00
Tim Wojtulewicz
5b8aaf6497 Revert Attributes::Attrs back to return an attr_list and mark it deprecated 2020-06-30 10:39:03 -07:00
Tim Wojtulewicz
e1338cc379 GH-1034: Revert TypeList::Types() back to return a type_list* and mark it deprecated 2020-06-29 17:50:09 -07:00
Jon Siwek
54d8954c80 GH-1036: change print.log to log network time instead of current 2020-06-26 19:55:09 -07:00
Johanna Amann
b9fe79c697 BrokerStore<->Zeek tables: load persistent tables on startup.
This currently only handles the most basic case, and is not thoroughly
tested.

When initializing a master store, we now check if there already is data
in it. If yes, we load it directly into the zeek table when the store is
created. We assume that this is happening at Zeek startup - and are
supremely evil and just load it synchronously. Which could block
execution for a bit for larger stores.

That being said - this might sidestep other issues that would arise when
doing this async (like scripts already inserting data).

Next step: check if this approach also works for clones.
2020-06-26 17:00:00 -07:00
Johanna Amann
38a3d67643 Merge remote-tracking branch 'origin/master' into topic/johanna/table-changes
* origin/master:
  Fix shadowed variable that breaks lookup_hostname()
  GH-1025: allow copying/cloning of `opaque of Broker::Store`
  Fix "possibly-truncated" compiler warning in BuildJSON snprintf()
  Update submodule(s)
  Fixed some places where tabs became spaces
  Convert to using permissions to check for access to cirrus variables in benchmark script
  Integrate review feedback: improve command-line option redef parsing
  Fix several issues with command-line option redefs
  Remove last_access_time from TableEntryVal.
  Minimize data published for expected FTP data channel analysis
  Stricter checking if we have a dns field on the connection being processed
  Modified the DNS protocol analyzer to add a new parameter to the dns_request event which includes the DNS query in its original case. Added a policy script that will add the original_case to the dns.log file as well. Created new btests to test both.
  Place build file in explicit location for benchmarking to work correctly
  cmake: Make musl support more distro agnostic
  Update highwayhash submodule to upstream.
  GH-998: Fix Reporter::conn_weird() to handle expired connections
2020-06-26 13:52:25 -07:00
Johanna Amann
eb1a408b6a Merge remote-tracking branch 'origin/topic/jsiwek/improve-command-line-option-redefs'
* origin/topic/jsiwek/improve-command-line-option-redefs:
  Integrate review feedback: improve command-line option redef parsing
  Fix several issues with command-line option redefs
2020-06-26 16:27:42 +00:00
Johanna Amann
3adaf2cc12 Merge remote-tracking branch 'origin/topic/jsiwek/fix-build-json-warning'
* origin/topic/jsiwek/fix-build-json-warning:
  Fix "possibly-truncated" compiler warning in BuildJSON snprintf()
2020-06-26 16:25:12 +00:00
Johanna Amann
7eb3156ad0 Merge remote-tracking branch 'origin/topic/jsiwek/gh-1025-fix-data-store-handle-cloning'
* origin/topic/jsiwek/gh-1025-fix-data-store-handle-cloning:
  GH-1025: allow copying/cloning of `opaque of Broker::Store`

Fixes GH-1025
2020-06-26 16:24:39 +00:00
Johanna Amann
77143dd58f Merge remote-tracking branch 'origin/topic/jsiwek/fix-lookup-hostname'
* origin/topic/jsiwek/fix-lookup-hostname:
  Fix shadowed variable that breaks lookup_hostname()
2020-06-26 16:24:08 +00:00
Jon Siwek
00a4865885 Merge branch 'dns-original-query-case' of https://github.com/rvictory/zeek
Changes during merge
- Changed the policy script to use an event handler that behaves
  for like the base script: &priority=5, msg$opcode != early-out,
  no record field existence checks
- Also extended dns_query_reply event with original_query param
- Removed ExtractName overload, and just use default param

* 'dns-original-query-case' of https://github.com/rvictory/zeek:
  Fixed some places where tabs became spaces
  Stricter checking if we have a dns field on the connection being processed
  Modified the DNS protocol analyzer to add a new parameter to the dns_request event which includes the DNS query in its original case. Added a policy script that will add the original_case to the dns.log file as well. Created new btests to test both.
2020-06-26 00:07:02 -07:00
Johanna Amann
b027b69f5d Brokerstore<->Tables: attribute conflicts
Makes some attributes conflict with each other. This also needed the
test to change.

The test is a bit flaky - but I can, for the heck of it, not figure out
why. I am punting that for the future after spending a few hours on it.
2020-06-25 19:28:35 -07:00
Jon Siwek
9473c32b5d Fix shadowed variable that breaks lookup_hostname()
A bug introduced in b62727a7fa
2020-06-25 18:56:00 -07:00
Jon Siwek
85d0825e2a GH-1025: allow copying/cloning of opaque of Broker::Store
Implemented simply as a reference count increment of the
data store handle.
2020-06-24 23:27:04 -07:00
Jon Siwek
a9f853efcd Improve Broker store API's handling of invalid arguments
* Some methods mistakenly returned a bool instead of QueryResult
  when passed an invalid `opaque of Broker::Store` handle.

* Now generates a runtime exception for store_name() and is_closed()
  calls that pass an invalid `opaque of Broker::Store` handle as any
  returned value can't be reasonably used in any subsequent logic.

* Descriptions of any invalid arguments are now given in the error
  message.
2020-06-24 22:56:14 -07:00
Jon Siwek
2b698c4e1b Add builtin_exception() functions
These work like builtin_error(), but also throw an InterpreterException
2020-06-24 22:54:46 -07:00
Jon Siwek
9f16fa6474 GH-1024: fix crash on passing wrong types to Broker store API 2020-06-24 20:21:12 -07:00
Jon Siwek
946ceba832 Fix "possibly-truncated" compiler warning in BuildJSON snprintf() 2020-06-24 17:59:28 -07:00
Johanna Amann
09119ae69d Zeek/Brokerstore updates: expiration
Expiration is done separately by the manager, the clones, and the
brokerstore. But - it should happen roughly at the same time.
2020-06-24 16:50:22 -07:00
Ryan Victory
7a91b49f5a Fixed some places where tabs became spaces 2020-06-22 11:17:56 -05:00
Jon Siwek
997c81955b Integrate review feedback: improve command-line option redef parsing
No functional change, just remove an unneeded copy, simplify loop logic,
other "modernization", and fix typo in comment.
2020-06-19 12:49:02 -07:00
Jon Siwek
8c90ef4459 Fix several issues with command-line option redefs
* Variables of `string` type can now be set to an empty string

* Trying to set a variable with non-`string` type to an empty value
  now emits an error instead of silently doing nothing

* Providing an invalid identifier now emits an "unknown identifier"
  error instead of silently doing nothing
2020-06-18 20:07:47 -07:00
Johanna Amann
64af3cdf05 Remove last_access_time from TableEntryVal.
Turns out - this was no longer used. And it takes up at least 8 bytes
of space for every single table/set entry.
2020-06-18 22:59:08 +00:00
Ryan Victory
63d99595fe Modified the DNS protocol analyzer to add a new parameter to the dns_request event which includes the DNS query in its original case. Added a policy script that will add the original_case to the dns.log file as well. Created new btests to test both. 2020-06-17 10:13:04 -05:00
Johanna Amann
c8a3a90339 Merge remote-tracking branch 'origin/master' into topic/johanna/table-changes
* origin/master: (33 commits)
  Fix location where CI places build.tgz
  Update submodule(s)
  Disable some deprecation diagnostics for GCC
  Compare pcap_next_ex() result to PCAP_ERROR/PCAP_ERROR_BREAK
  Optimize Connection::RemovalEvent() for bare-mode usage
  Rename BroType to Type
  Update NEWS
  Review cleanup
  Move Type types to zeek namespace
  Review cleanup
  Restrict Cirrus CI to only zeek repo's branches
  GH-977: Improve pcap error handling
  Remove not-useful code in iosource::Manager::OpenPktSrc
  GH-999: Stop formatting DHCP Client ID Hardware Type 0 as MAC
  Remove inline from some static KeyedHash members
  Improve Func.h inclusion
  Fix NVT analyzer memory leak from multiple telnet authn name options
  Rename aux/ to auxil/
  Move Flare/Pipe from the bro namespace to zeek::detail
  Move Attr to the zeek::detail namespace
  ...
2020-06-15 14:19:02 -07:00
Jon Siwek
51e738a1c0 GH-998: Fix Reporter::conn_weird() to handle expired connections
This introduces a new sampling state-map for expired connections to fix
segfaults that previously occured when passing in a `connection` record
to `Reporter::conn_weird()` for which the internal `Connection` object
had already been expired and deleted.  This also introduces a new event
called `expired_conn_weird`, which is similar to `conn_weird`, except
the full `connection` record is no longer available, just the `conn_id`
and UID string.
2020-06-15 12:57:47 -07:00
Jon Siwek
6cec268e43 Merge remote-tracking branch 'origin/topic/jsiwek/gh-977-improve-pcap-error-handling'
* origin/topic/jsiwek/gh-977-improve-pcap-error-handling:
  Compare pcap_next_ex() result to PCAP_ERROR/PCAP_ERROR_BREAK
  GH-977: Improve pcap error handling
  Remove not-useful code in iosource::Manager::OpenPktSrc
2020-06-11 23:22:19 -07:00
Jon Siwek
d4f3cad7d1 Merge remote-tracking branch 'origin/topic/timw/266-namespaces'
Merge adjustments:

- Preserved original `base_type_no_ref` argument type as ::TypeTag
- Removed superfluous #pragma guard around deprecated TableVal ctor
- Clarify NEWS regarding MetaHook{Pre,Post} deprecations
- Simplify some `::zeek::` qualifications to just `zeek::`
- Prefixed FORWARD_DECLARE_NAMESPACED macro with ZEEK_

* origin/topic/timw/266-namespaces:
  Disable some deprecation diagnostics for GCC
  Rename BroType to Type
  Update NEWS
  Review cleanup
  Move Type types to zeek namespace
  Move Flare/Pipe from the bro namespace to zeek::detail
  Move Attr to the zeek::detail namespace
  Move Trigger into the zeek::detail namespace
  Move ID to the zeek::detail namespace
  Move Anon.h into zeek::detail namespace
  Mark all of the aliased classes in plugin/Plugin.h deprecated, and fix all of the plugins that were using them
  Move all of the base plugin classes into the zeek::plugin namespace
  Expr: move all classes into zeek::detail
  Stmt: move Stmt classes into zeek::detail namespace
  Add utility macro for creating namespaced aliases for classes
2020-06-11 23:20:51 -07:00
Tim Wojtulewicz
149e3b3c32 Disable some deprecation diagnostics for GCC
Clang automatically disables deprecation warnings for types used within
already-deprecated contexts, such as if you use a deprecated type inside
of a method that's beeen marked as deprecated. GCC doesn't have this
feature so it spews a lot more warnings. These functions are now wrapped
in pragmas that disable the warnings for the usage.
2020-06-11 15:43:11 -07:00
Jon Siwek
65ae4d732a Compare pcap_next_ex() result to PCAP_ERROR/PCAP_ERROR_BREAK 2020-06-11 15:01:06 -07:00
Jon Siwek
0bc1d73c84 Optimize Connection::RemovalEvent() for bare-mode usage
It was creating RecordVals even if they wouldn't be used by any event
handler and that situation is common/expected for `zeek -b` mode.
Normally, there's at least the tunnel scripts with a `new_connection`
handler causing the connection RecordVals to be built.
2020-06-10 16:38:39 -07:00
Tim Wojtulewicz
137e416a03 Rename BroType to Type 2020-06-10 14:27:36 -07:00
Tim Wojtulewicz
bac318475a Review cleanup 2020-06-10 13:46:42 -07:00
Tim Wojtulewicz
ed13972924 Move Type types to zeek namespace 2020-06-09 17:20:45 -07:00
Jon Siwek
3255930738 Merge remote-tracking branch
* Prevent IP fragment reassembly on packets without minimal IP header
2020-06-09 09:46:46 -07:00
Jon Siwek
97e141bc5d Merge remote-tracking branch
* Fix potential leak of Analyzers added to tree during Analyzer::Done
2020-06-09 09:38:28 -07:00
Jon Siwek
aada21879f Merge remote-tracking branch
* Fix multiple content-transfer-encoding headers causing a memory leak
2020-06-09 09:37:32 -07:00