Commit graph

10798 commits

Author SHA1 Message Date
Jon Siwek
7e03233d55 Fix incorrect/missing Broker error status code numbers 2020-07-16 18:03:41 -07:00
Jon Siwek
d60f16c229 Fix race condition in ensure_dir()
If something else created the dir between the stat() and mkdir(),
it previously reported that as a failure.
2020-07-16 12:32:10 -07:00
Jon Siwek
dfc34563b5 Fix tokenize_string() to work with delimiters of length > 1 2020-07-16 11:51:40 -07:00
Tim Wojtulewicz
4d96b7f1cc Various deprecation fixes, reported by failed Spicy builds
- Add deprecated version of ID::SetType() that takes Type*
- Add deprecated versions of zeek::set_location in the global namespace
- Fix global namespace version of lookup_ID to return ID*
2020-07-15 15:24:19 -07:00
Tim Wojtulewicz
118605f4ac Add deprecated version of EnumType::GetVal() to returns EnumVal*, rename IntrusivePtr version to GetEnumVal 2020-07-15 14:56:05 -07:00
Jon Siwek
eb826bd337 GH-734: Improve handling of lambdas that escape enclosing frame 2020-07-14 22:38:43 -07:00
Jon Siwek
fd8db24daf Merge remote-tracking branch 'origin/topic/jsiwek/gh-1062-fix-tag-subtypes'
* origin/topic/jsiwek/gh-1062-fix-tag-subtypes:
  GH-1062: fix integer conversion errors related to Tag subtypes
2020-07-14 19:11:09 -07:00
Jon Siwek
7f2aae1721 GH-1062: fix integer conversion errors related to Tag subtypes 2020-07-14 15:55:28 -07:00
Jon Siwek
da105b3bf9 Update submodule(s)
[nomail]
2020-07-14 14:58:50 -07:00
Johanna Amann
1888d6acae BrokerStore <-> Zeek Tables: cleanup and bug workaround 2020-07-13 17:23:35 -07:00
Johanna Amann
7c37226eaa Merge remote-tracking branch 'origin/master' into topic/johanna/table-changes 2020-07-13 17:11:55 -07:00
Johanna Amann
da2c968ba0 Merge remote-tracking branch 'origin/topic/jsiwek/gh-1024-broker-store-handle-type-checks'
* origin/topic/jsiwek/gh-1024-broker-store-handle-type-checks:
  Improve Broker store API's handling of invalid arguments
  Add builtin_exception() functions
  GH-1024: fix crash on passing wrong types to Broker store API

This was a bit of a bigger merge since Zeek changed inbetween the time
of the PR and me actually merging it.

I put the new functions into the zeek::detail namespace -- since it
seems unlikely that those will be used by something external.

I also renamed them to fit better with the naming scheme of the new
error functions.

Fixes GH-1024
2020-07-14 00:05:09 +00:00
Tim Wojtulewicz
37dbca965e Update submodule(s)
[nomail]
2020-07-13 12:39:32 -07:00
Tim Wojtulewicz
75cb35e2d9 Merge remote-tracking branch 'origin/topic/jsiwek/gh-1019-deprecate-icmp-conn'
* origin/topic/jsiwek/gh-1019-deprecate-icmp-conn:
  GH-1019: deprecate icmp_conn params for ICMP events
2020-07-13 12:12:44 -07:00
Tim Wojtulewicz
64af3ec67a Merge remote-tracking branch 'origin/topic/jsiwek/deprecation-improvements'
* origin/topic/jsiwek/deprecation-improvements:
  Fix wrong frame offsets for locals of alternate event/hook prototypes
  Add deprecation expression to deprecated prototype/parameter messages
  Improve "use of deprecated prototype" warning message
  Emit deprecation warning for use of &deprecated function parameters
2020-07-13 12:11:12 -07:00
Jon Siwek
034be246ba Update submodule(s)
[nomail]
2020-07-13 09:20:30 -07:00
Johanna Amann
2b2a40f49c Zeek Table<->Brokerstore: cleanup, documentation, small fixes
This commit adds script/c++ documentation and fixes a few loose ends.
It also adds tests for corner cases and massively improves error
messages.

This also actually introduces type-compatibility checking and introduces
a new attribute that lets a user override this if they really know what
they are doing. I am not quite sure if we should really let that stay in
- but it can be very convenient to have this functionality.

One test is continuing to fail - the expiry test is very flaky. This is,
I think, caused by delays of the broker store forwarding. I am unsure if
we can actually do anything about that.
2020-07-10 16:58:34 -07:00
Jon Siwek
11df252f5f Update submodule(s)
[nomail]
2020-07-10 16:52:18 -07:00
Jon Siwek
ba0ba7c684 Add more error checks to shadow log parsing
i.e. Coverity warns about possible use of ftell() negative return value
2020-07-10 11:21:49 -07:00
Jon Siwek
6908d1b919 GH-1019: deprecate icmp_conn params for ICMP events
Previously, a single `icmp_conn` record was built per ICMP "connection"
and re-used for all events generated from it.  This may have been a
historical attempt at performance optimization, but:

  * By default, Zeek does not load any scripts that handle ICMP events.

  * The one script Zeek ships with that does handle ICMP events,
    "detect-traceroute", is already noted as being disabled due to
    potential performance problems of doing that kind of analysis.

  * Re-use of the original `icmp_conn` record tends to misreport
    TTL and length values since they come from original packet instead
    of the current one.

  * Even if we chose to still re-use `icmp_conn` records and just fill
    in a new TTL and length value each packet, a user script could have
    stored a reference to the record and not be expecting those values
    to be changed out from underneath them.

Now, a new `icmp_info` record is created/populated in all ICMP events
and should be used instead of `icmp_conn`.  It also removes the
orig_h/resp_h fields as those are redundant with what's already
available in the connection record.
2020-07-10 11:06:28 -07:00
Jon Siwek
20294d372c Fix wrong frame offsets for locals of alternate event/hook prototypes
Local frame offsets were being assigned based on number of the alternate
prototype's parameters, which may end up having less total parameters
than the canonical prototype, causing the local value to incorrectly
overwrite an event/hook argument value.
2020-07-10 10:59:12 -07:00
Jon Siwek
8597b998bb Add deprecation expression to deprecated prototype/parameter messages 2020-07-10 10:57:57 -07:00
Jon Siwek
ac1ec7668d Improve "use of deprecated prototype" warning message
The location information now points out the place of the deprecated
prototype instead of the location where the ID was initially declared
(which may not itself be a deprecated prototype).
2020-07-10 10:57:32 -07:00
Jon Siwek
26b3d406b4 Emit deprecation warning for use of &deprecated function parameters
Particularly, this is meant for using &deprecated on canonical
event/hook prototype parameters to encourage users to create handlers
to another, non-deprecated prototype.  i.e. for canonical prototypes,
we may not always want to put &deprecated directly on the prototype
itself since that signals deprecation of the ID entirely.
2020-07-10 10:57:05 -07:00
Johanna Amann
67917b83aa Merge remote-tracking branch 'origin/master' into topic/johanna/table-changes 2020-07-09 17:02:57 -07:00
Johanna Amann
41dd7df69a BrokerStore<->Zeek table: adopt to recent Zeek API changes 2020-07-09 14:31:59 -07:00
Jon Siwek
39f549ed68 Merge remote-tracking branch 'origin/topic/jsiwek/3.2-supervisor'
* origin/topic/jsiwek/3.2-supervisor:
  Integrate Supervisor code review suggestions
  Add Supervisor::{stdout,stderr}_hook
  Add Log::rotation_format_func and Log::default_rotation_dir options
  Allow Zeek function equality comparisons
  Implement leftover log rotation/archival for supervised nodes
  Move some supervisor structures to detail namespace
  Improve default supervisor stdout/stderr handling
  Factor Supervisor initialization data to separate struct from Stem
  Move ParentProcessCheckTimer to detail namespace
2020-07-09 14:13:26 -07:00
Johanna Amann
e1a45d33e0 Merge remote-tracking branch 'origin/master' into topic/johanna/table-changes
* origin/master: (47 commits)
  scan.l: Remove "constant" did_module_restore logic
  Fix FreeBSD CI script to install right SWIG package
  Update submodule(s)
  GH-928: use realpath() instead of inode to de-duplicate scripts
  Update submodule(s)
  GH-1040: Add zero-indexed version of str_split
  Fix WhileStmt to call Stmt(Tag) ctor
  GH-1041: Move compress_path to a bif that uses normalize_path
  Update submodule(s)
  Update submodule(s)
  Update submodule(s)
  Fix --enable-mobile-ipv6 build
  Fix namespace of GetCurrentLocation() to zeek::detail
  Add backtrace() and print_backtrace()
  Rename BroString files to ZeekString
  Update NEWS entry with note about class renames
  Rename BroObj to Obj
  Rename BroString to zeek::String
  Move Func up to zeek namespace, rename BroFunc to ScriptFunc
  Mark global val_mgr as deprecated and fix uses of it to use namespaced version
  ...
2020-07-09 14:07:03 -07:00
Jon Siwek
7669f560d1 Integrate Supervisor code review suggestions 2020-07-09 13:56:11 -07:00
Johanna Amann
3eac12b40d BrokerStore<->Zeek Tables Fix a few small test failures. 2020-07-09 19:43:45 +00:00
Jon Siwek
48362cc0c0 Merge remote-tracking branch 'origin/topic/jsiwek/remove-lexer-dead-code'
* origin/topic/jsiwek/remove-lexer-dead-code:
  scan.l: Remove "constant" did_module_restore logic
2020-07-08 17:47:07 -07:00
Arne Welzel
93e7f40b70 scan.l: Remove "constant" did_module_restore logic
Doesn't seem like did_module_restore has any effect on the code flow.
2020-07-08 16:20:57 -07:00
Jon Siwek
d15dd37b84 Merge remote-tracking branch 'origin/topic/jsiwek/fix-ci-freebsd-swig'
* origin/topic/jsiwek/fix-ci-freebsd-swig:
  Fix FreeBSD CI script to install right SWIG package
2020-07-08 12:25:31 -07:00
Jon Siwek
3c1f5914c6 Fix FreeBSD CI script to install right SWIG package 2020-07-08 11:07:56 -07:00
Robin Sommer
1fe822a745 Merge remote-tracking branch 'origin/topic/jsiwek/fix-while-stmt-ctor'
* origin/topic/jsiwek/fix-while-stmt-ctor:
  Fix WhileStmt to call Stmt(Tag) ctor
2020-07-08 15:41:50 +00:00
Robin Sommer
776250fd8c Merge remote-tracking branch 'origin/topic/jsiwek/gh-928-realpath-load-dedup'
* origin/topic/jsiwek/gh-928-realpath-load-dedup:
  GH-928: use realpath() instead of inode to de-duplicate scripts
2020-07-08 15:40:24 +00:00
Jon Siwek
7bcba16837 Update submodule(s)
[nomail]
2020-07-08 08:33:23 -07:00
Jon Siwek
10709c627b Add Supervisor::{stdout,stderr}_hook
These allow capturing/handling the stdout/stderr of child processes
via Zeek scripts.
2020-07-07 20:21:32 -07:00
Jon Siwek
a06ef66edc Add Log::rotation_format_func and Log::default_rotation_dir options
These may be redefined to customize log rotation path prefixes,
including use of a directory.  File extensions are still up to
individual log writers to add themselves during the actual rotation.

These new also allow for some simplication to the default
ASCII postprocessor function: it eliminates the need for it doing an
extra/awkward rename() operation that only changes the timestamp format.

This also teaches the supervisor framework to use these new options
to rotate ascii logs into a log-queue/ directory with a specific
file name format (intended for an external archiver process to
monitor separately).
2020-07-07 18:42:37 -07:00
Jon Siwek
6e67a40d24 Allow Zeek function equality comparisons
Equality between two functions acts like a pointer comparison.
2020-07-07 18:39:23 -07:00
Jon Siwek
11949ce37a Implement leftover log rotation/archival for supervised nodes
This helps prevent a node from being killed/crashing in the middle
of writing a log, restarting, and eventually clobbering that log
file that never underwent the rotation/archival process.

The old `archive-log` and `post-terminate` scripts as used by
ZeekControl previously implemented this behavior, but the new logic is
entirely in the ASCII writer.  It uses ".shadow" log files stored
alongside the real log to help detect such scenarios and rotate them
correctly upon the next startup of the Zeek process.
2020-07-07 18:39:23 -07:00
Jon Siwek
a46e24091a Move some supervisor structures to detail namespace
* StemHandle -> detail::SupervisorStemHandle
* SupervisedNode -> detail::SupervisedNode
* Supervisor::Node -> detail::SupervisorNode
2020-07-07 18:39:23 -07:00
Jon Siwek
9087621714 Improve default supervisor stdout/stderr handling
The stdout/stderr of child processes is now redirected over a pipe back
to the supervisor process so that it can prefix the output with
the name of the emitting node.
2020-07-07 18:39:23 -07:00
Jon Siwek
0acd5fea0c Factor Supervisor initialization data to separate struct from Stem 2020-07-07 18:38:38 -07:00
Jon Siwek
d00b3a8daa Move ParentProcessCheckTimer to detail namespace 2020-07-07 18:38:38 -07:00
Jon Siwek
320b14ff09 GH-928: use realpath() instead of inode to de-duplicate scripts
Duplicate script `@load` directives are now detected by comparing
against canonical paths formed by realpath().  This fixes the previous,
unexpected behavior of treating scripts that hardlink to same
inode as duplicates: such links will now be loaded as distinct scripts
since their canonical path differs.
2020-07-07 17:29:52 -07:00
Jon Siwek
7f347a3b1e Update submodule(s)
[nomail]
2020-07-07 12:47:30 -07:00
Jon Siwek
7b15b82009 Merge remote-tracking branch 'origin/topic/timw/1040-str-split'
* origin/topic/timw/1040-str-split:
  GH-1040: Add zero-indexed version of str_split
2020-07-06 21:06:51 -07:00
Jon Siwek
09483619ef Merge remote-tracking branch 'origin/topic/timw/1041-compress-path'
* origin/topic/timw/1041-compress-path:
  GH-1041: Move compress_path to a bif that uses normalize_path
2020-07-06 20:41:05 -07:00
ronwellman
b17ec6bf93 Adjust for zeek namespace. 2020-07-06 22:59:43 -04:00