Commit graph

11634 commits

Author SHA1 Message Date
Jon Siwek
a2b0abe283 Add environment variable to disable supervisor SIGKILL
For use in btests since SIGKILL can otherwise bypass the usual
btest-bg-wait timeout mechanism and make reason for test failures less
obvious.
2020-02-06 17:50:17 -08:00
Jon Siwek
5b74bbf99a Fix a btest's race condition by handling Pcap::file_done 2020-02-06 17:50:17 -08:00
Jon Siwek
7c124881cd Add Pcap::file_done event
It signals when a pcap file is done being processed.
2020-02-06 17:50:17 -08:00
Jon Siwek
4fbcca04e8 Improve btest timeouts
* Generally increase timeouts for tests that have recent transient
  failures

* Change any test that relied on `btest-bg-wait -k` since that's never
  going to play with with CI systems.  Instead, we always need to have
  a well-defined termination condition in the test itself (and most
  already did, so didn't really need the `-k` flag anyway).
2020-02-06 17:50:17 -08:00
Jon Siwek
63e68665d3 Use 2 btest retries for CI 2020-02-06 17:47:38 -08:00
Johanna Amann
391164c9bc Merge remote-tracking branch 'origin/topic/johanna/table-on-change'
* origin/topic/johanna/table-on-change:
  &on_change: Address feedback of Jon.
  &on_change: incooperate feedback of Jon.
  &on_change wrapup: documentation, tests, whitespacing
  Table expiry now raises &on_change handlers
  Prevent recursion of &on_change handlers.
  Make bro_broker::val_to_data take a const Val* instead of a Val
  &on_change: add insertion/change notifications.
  &on_change working for removals from tables.
  Change signature of &on_change handler
  Beginning implementation of &on_change for tables.
2020-02-06 15:01:03 -08:00
Johanna Amann
b69a059b1a &on_change: Address feedback of Jon.
Forgot to commit a file *sigh*
2020-02-06 14:42:03 -08:00
Johanna Amann
d6d2bbe878 &on_change: incooperate feedback of Jon. 2020-02-06 14:40:38 -08:00
Jon Siwek
8e812df6c5 Switch GitHub Action for CI emails to use zeek organization 2020-02-06 13:06:56 -08:00
Tim Wojtulewicz
23e82b8121 Restore accidentally reverted change to fix FreeBSD 11 build 2020-02-06 10:25:21 -07:00
Jon Siwek
3ee5f5fcaf Update CAF to current 0.17.4 release branch 2020-02-05 21:47:28 -08:00
Jon Siwek
7f76381f13 Remove presumed ownership of HeartbeatTimer
threading::Manager is currently never deleted, but if that ever changes,
deleting the stored HeartbeatTimer pointer would be a double-free
since TimerMgr owns it.
2020-02-05 17:56:02 -08:00
Jon Siwek
e62c0e0c9f Set threading::Manager terminating state
Coverity CID 1417429
2020-02-05 17:35:46 -08:00
Jon Siwek
4e87b6ee73 Remove unused HeartbeatTimer member: do_expire
Coverity CID 1417434
2020-02-05 16:43:22 -08:00
Jon Siwek
c5340c7c3e Merge remote-tracking branch 'origin/topic/timw/773-eventmgr-iosource'
Fixes GH-773

* origin/topic/timw/773-eventmgr-iosource:
  GH-773: Make EventMgr an IOSource so that events interrupt kevent waits
2020-02-05 14:57:26 -08:00
Johanna Amann
b1040f88c9 &on_change wrapup: documentation, tests, whitespacing
Adds documentation, fixes a whitespace issues, fixes compiler warning on
some Linux system, extends test.
2020-02-05 14:15:14 -08:00
Jon Siwek
43e54c7930 GH-780: Prevent log batches from indefinite buffering
Logs that got sent sparsely or burstily would get buffered for long
periods of time since the logic to flush them only does so on the next
log write.  In the worst case, a subsequent log write could never happen
and cause a log entry to be indefinitely buffered.

This fix introduces a recurring event/timer to simply flush all pending
logs at frequency of Broker::log_batch_interval.
2020-02-05 13:06:52 -08:00
Tim Wojtulewicz
4f23c0360a GH-773: Make EventMgr an IOSource so that events interrupt kevent waits 2020-02-05 15:43:37 -05:00
Johanna Amann
b6a244f784 Merge remote-tracking branch 'origin/master' into topic/johanna/table-on-change 2020-02-05 12:41:19 -08:00
Johanna Amann
18ab7838a9 Table expiry now raises &on_change handlers
Expiry is remprted as ELEMENT_EXPIRED (which was newly introduced), to
allow distinction from table deletion.
2020-02-05 12:37:52 -08:00
Johanna Amann
7f9f66fce9 Prevent recursion of &on_change handlers.
This change prevents &on_change handlers for a table from running if an
&on_change handler for the same table is already running.
2020-02-05 11:45:16 -08:00
Jon Siwek
0de6bba95e Merge remote-tracking branch 'origin/topic/johanna/692'
* origin/topic/johanna/692:
  Resolve race condition in test
  Change threading formatter errors to warnings.
2020-02-05 10:52:39 -08:00
Johanna Amann
50ebbc90ad Resolve race condition in test 2020-02-05 09:34:29 -08:00
Johanna Amann
deaab42187 Change threading formatter errors to warnings.
This change standardizes threading formatter error handling and moves
the remaining error calls to be warnings instead.

This is in line with already existing code - in most cases warnings were
raised, only a few cases raised errors. These cases do not differ
significantly from other cases in which warnings are raised.

This also fixes GH-692, in which misformatted lines prevent future file
parsing.

This commit also moves the FailWarn method that is used by both the
config and the ascii reader up to the ReaderBackend. Furthermore it
makes the Warning method of ReaderBackend respect the warning
suppression that is introduced by the FailWarn method.
2020-02-04 16:31:29 -08:00
Tim Wojtulewicz
5171f25e3a Update NEWS about plugin breakage from the include-file changes 2020-02-04 14:15:14 -07:00
Tim Wojtulewicz
4a7da2669c Merge remote-tracking branch 'max/include_cleanup'
* max/include_cleanup:
  UID, ..: un-inline methods to reduce header dependencies
  include cleanup
2020-02-04 14:06:05 -07:00
Jon Siwek
99b1dccb8d Updating submodule(s).
[nomail]
2020-02-04 12:08:13 -08:00
Max Kellermann
6a815b4b06 UID, ..: un-inline methods to reduce header dependencies
Only 1% build time speedup, but still, it declutters the headers a bit.

Before this patch:

 2565.17user 141.83system 2:25.46elapsed 1860%CPU (0avgtext+0avgdata 1489076maxresident)k
 72576inputs+9130920outputs (1667major+49400430minor)pagefaults 0swaps

After this patch:

 2537.19user 142.94system 2:26.90elapsed 1824%CPU (0avgtext+0avgdata 1434268maxresident)k
 16240inputs+8887152outputs (1931major+48728888minor)pagefaults 0swaps
2020-02-04 20:51:02 +01:00
Max Kellermann
0db61f3094 include cleanup
The Zeek code base has very inconsistent #includes.  Many sources
included a few headers, and those headers included other headers, and
in the end, nearly everything is included everywhere, so missing
#includes were never noticed.  Another side effect was a lot of header
bloat which slows down the build.

First step to fix it: in each source file, its own header should be
included first to verify that each header's includes are correct, and
none is missing.

After adding the missing #includes, I replaced lots of #includes
inside headers with class forward declarations.  In most headers,
object pointers are never referenced, so declaring the function
prototypes with forward-declared classes is just fine.

This patch speeds up the build by 19%, because each compilation unit
gets smaller.  Here are the "time" numbers for a fresh build (with a
warm page cache but without ccache):

Before this patch:

 3144.94user 161.63system 3:02.87elapsed 1808%CPU (0avgtext+0avgdata 2168608maxresident)k
 760inputs+12008400outputs (1511major+57747204minor)pagefaults 0swaps

After this patch:

 2565.17user 141.83system 2:25.46elapsed 1860%CPU (0avgtext+0avgdata 1489076maxresident)k
 72576inputs+9130920outputs (1667major+49400430minor)pagefaults 0swaps
2020-02-04 20:51:02 +01:00
Robin Sommer
532c66df51 Updating submodule.
[nomail]
2020-02-04 11:58:06 +00:00
Jon Siwek
6d2a789243 Updating submodule(s).
[nomail]
2020-02-03 18:01:17 -08:00
Tim Wojtulewicz
37e0130461 Merge remote-tracking branch 'origin/topic/jsiwek/consistent-string-view-args'
* origin/topic/jsiwek/consistent-string-view-args:
  Change various functions to by-value std::string_view args
2020-02-03 15:44:28 -07:00
Tim Wojtulewicz
5aa8120259 Merge remote-tracking branch 'origin/topic/jsiwek/pktsrc-options'
* origin/topic/jsiwek/pktsrc-options:
  Change packet source fields of Options to std::optional
2020-02-03 15:42:48 -07:00
Johanna Amann
2129a808f6 Updating CHANGES and VERSION after messing up last commit message. 2020-02-03 13:25:57 -08:00
Johanna Amann
c20551e48e Update submodule
[nomail]
2020-02-03 13:23:22 -08:00
Jon Siwek
2cda94e8ee Disable Travis leak test
Travis environment doesn't offer enough memory for running ASan very
well, but Cirrus has it covered anyway.
2020-02-03 13:21:32 -08:00
Jon Siwek
d33613c2a5 Merge branch 'master' of https://github.com/ffontaine/zeek
- Also removed the setting of BinPAC_ROOT_DIR in the configure
  script's --with-binpac= option as that breaks the cross-compilation
  use-case

* 'master' of https://github.com/ffontaine/zeek:
  CMakeLists.txt: fix cross-compilation with binpac
2020-02-03 13:05:50 -08:00
Jon Siwek
1e2dd4466c Change requested memory in Cirrus CI config
The ASan build does require quite a bit more, but also the default
release builds should do ok with slightly less.
2020-02-03 10:11:46 -08:00
Jon Siwek
1df5793479 Merge remote-tracking branch 'origin/topic/timw/ioloop-followup'
* origin/topic/timw/ioloop-followup:
  Increase timeout value for live interfaces without file descriptors
  Use ranged-for loops in a few places in iosource Manager
  Change order of includes in iosource Manager, fixes build on FreeBSD 11
2020-02-03 10:04:26 -08:00
Tim Wojtulewicz
edb87821da Increase timeout value for live interfaces without file descriptors 2020-02-03 07:34:27 -07:00
Tim Wojtulewicz
efaa756539 Use ranged-for loops in a few places in iosource Manager 2020-02-03 07:34:27 -07:00
Tim Wojtulewicz
d12d0332a7 Change order of includes in iosource Manager, fixes build on FreeBSD 11 2020-02-01 19:23:17 -07:00
Fabrice Fontaine
63344b32f5 CMakeLists.txt: fix cross-compilation with binpac
When cross-compiling, BINPAC_EXE_PATH will be set by the user to the
host binpac binary which is fine however aux/binpac won't be built which
will raise a build failure as target binpac (headers, library) won't be
installed or built

Signed-off-by: Fabrice Fontaine <fontaine.fabrice@gmail.com>
2020-02-01 14:38:17 +01:00
Jon Siwek
2f36113743 Change packet source fields of Options to std::optional 2020-01-31 15:41:23 -08:00
Jon Siwek
cd74d6f392 Change various functions to by-value std::string_view args 2020-01-31 15:08:48 -08:00
Jon Siwek
b0a5eb27b1 Updating submodule(s).
[nomail]
2020-01-31 14:19:21 -08:00
Jon Siwek
fa5b3bb91e Merge branch 'no_sscanf' of https://github.com/MaxKellermann/zeek
* 'no_sscanf' of https://github.com/MaxKellermann/zeek:
  util: optimize expand_escape() by avoiding sscanf()
2020-01-31 14:19:12 -08:00
Jon Siwek
d39bb42b14 Merge branch 'optimize_normalize_path' of https://github.com/MaxKellermann/zeek
- Minor changes in merge: extended unit test, prefer emplace_back(),
  remove unused "found" count in new function

* 'optimize_normalize_path' of https://github.com/MaxKellermann/zeek:
  util: add a tokenize_string() overload which returns string_views
  util: store std::string_view in "final_components" vector
  util: use "auto" in normalize_path()
  util: reserve space in normalize_path()
  util: skip "." completely in normalize_path()
  util: pass std::string_view to normalize_path()
  util: pass std::string_view to tokenize_string()
  util: don't modify the input string in tokenize_string()
2020-01-31 13:23:39 -08:00
Jon Siwek
719011cef0 Remove extra fmt() in a reporter->Error() call
Else compiler may warn about format string not being a string literal
2020-01-31 11:25:07 -08:00
Jon Siwek
0c68730b8b Merge branch 'fix-function-start-end-location' of https://github.com/awelzel/zeek
* 'fix-function-start-end-location' of https://github.com/awelzel/zeek:
  parse.y: Properly set location info for functions
2020-01-31 11:18:07 -08:00