Commit graph

9997 commits

Author SHA1 Message Date
Johanna Amann
49a21d73d0 Merge remote-tracking branch 'origin/topic/jsiwek/gh-780-indefinite-log-buffer'
* origin/topic/jsiwek/gh-780-indefinite-log-buffer:
  GH-780: Prevent log batches from indefinite buffering
2020-02-07 09:35:29 -08:00
Johanna Amann
78a5b02744 Update submodule
[nomail]
2020-02-07 09:26:43 -08:00
Johanna Amann
bf0ecbd577 Update CHANGES, make &on_change description nicer 2020-02-07 09:12:30 -08:00
Robin Sommer
a394d2e0d7 Update NEWS to point out Broker wireformat change. 2020-02-07 10:38:25 +00:00
Max Kellermann
298fd125ae plugin/Manager: migrate to std::string_view 2020-02-07 10:56:14 +01:00
Max Kellermann
95e646fca7 util: optimize the normal_path() common case
Speeds up Zeek startup by 2%.
2020-02-07 10:56:14 +01:00
Max Kellermann
98241bbc60 util: pass string_view to without_bropath_component() 2020-02-07 10:56:14 +01:00
Max Kellermann
0548e1255f module_util: make GLOBAL_MODULE_NAME constexpr
This allows the compiler to hard-code pointers to the string without
looking up a global variable at runtime.
2020-02-07 10:56:14 +01:00
Max Kellermann
f1908b6212 Scope: convert Scope::Lookup() and others to template
Allows passing rvalue references which eliminates unnecessary
std::string copies.

This speeds up Zeek startup by 1-2%.
2020-02-07 10:56:14 +01:00
Max Kellermann
f8e9cc0fc5 Scope: Vars() returns const reference
No caller wants to modify the container.
2020-02-07 10:56:14 +01:00
Max Kellermann
acdfd5706e Anon: remove unnecessary {map,vector}::clear() calls
The destructor will do this automatically.
2020-02-07 10:56:14 +01:00
Max Kellermann
05f692995d Dict: make the destructor non-virtual 2020-02-07 10:56:14 +01:00
Max Kellermann
b4966858ce Obj: make the Location constructors constexpr 2020-02-07 10:56:14 +01:00
Max Kellermann
ba445d36f9 Obj: remove unused fields Location::{timestamp,text} 2020-02-07 10:56:14 +01:00
Max Kellermann
91e0860cd4 Obj: remove Location::delete_data, nobody ever sets it
This allows removing the destructor completely, making the class
trivially-destructible.
2020-02-07 10:56:14 +01:00
Max Kellermann
2041921fcb Obj: make the Location destructor non-virtual
Nobody ever derives from this class.  This removes an indirect call
and allows inlining the destructor.

This speeds up Zeek startup by 1-2%.
2020-02-07 10:56:14 +01:00
Jon Siwek
a2b0abe283 Add environment variable to disable supervisor SIGKILL
For use in btests since SIGKILL can otherwise bypass the usual
btest-bg-wait timeout mechanism and make reason for test failures less
obvious.
2020-02-06 17:50:17 -08:00
Jon Siwek
5b74bbf99a Fix a btest's race condition by handling Pcap::file_done 2020-02-06 17:50:17 -08:00
Jon Siwek
7c124881cd Add Pcap::file_done event
It signals when a pcap file is done being processed.
2020-02-06 17:50:17 -08:00
Jon Siwek
4fbcca04e8 Improve btest timeouts
* Generally increase timeouts for tests that have recent transient
  failures

* Change any test that relied on `btest-bg-wait -k` since that's never
  going to play with with CI systems.  Instead, we always need to have
  a well-defined termination condition in the test itself (and most
  already did, so didn't really need the `-k` flag anyway).
2020-02-06 17:50:17 -08:00
Jon Siwek
63e68665d3 Use 2 btest retries for CI 2020-02-06 17:47:38 -08:00
Johanna Amann
391164c9bc Merge remote-tracking branch 'origin/topic/johanna/table-on-change'
* origin/topic/johanna/table-on-change:
  &on_change: Address feedback of Jon.
  &on_change: incooperate feedback of Jon.
  &on_change wrapup: documentation, tests, whitespacing
  Table expiry now raises &on_change handlers
  Prevent recursion of &on_change handlers.
  Make bro_broker::val_to_data take a const Val* instead of a Val
  &on_change: add insertion/change notifications.
  &on_change working for removals from tables.
  Change signature of &on_change handler
  Beginning implementation of &on_change for tables.
2020-02-06 15:01:03 -08:00
Johanna Amann
b69a059b1a &on_change: Address feedback of Jon.
Forgot to commit a file *sigh*
2020-02-06 14:42:03 -08:00
Johanna Amann
d6d2bbe878 &on_change: incooperate feedback of Jon. 2020-02-06 14:40:38 -08:00
Jon Siwek
8e812df6c5 Switch GitHub Action for CI emails to use zeek organization 2020-02-06 13:06:56 -08:00
Tim Wojtulewicz
23e82b8121 Restore accidentally reverted change to fix FreeBSD 11 build 2020-02-06 10:25:21 -07:00
Jon Siwek
3ee5f5fcaf Update CAF to current 0.17.4 release branch 2020-02-05 21:47:28 -08:00
Jon Siwek
7f76381f13 Remove presumed ownership of HeartbeatTimer
threading::Manager is currently never deleted, but if that ever changes,
deleting the stored HeartbeatTimer pointer would be a double-free
since TimerMgr owns it.
2020-02-05 17:56:02 -08:00
Jon Siwek
e62c0e0c9f Set threading::Manager terminating state
Coverity CID 1417429
2020-02-05 17:35:46 -08:00
Jon Siwek
4e87b6ee73 Remove unused HeartbeatTimer member: do_expire
Coverity CID 1417434
2020-02-05 16:43:22 -08:00
Jon Siwek
c5340c7c3e Merge remote-tracking branch 'origin/topic/timw/773-eventmgr-iosource'
Fixes GH-773

* origin/topic/timw/773-eventmgr-iosource:
  GH-773: Make EventMgr an IOSource so that events interrupt kevent waits
2020-02-05 14:57:26 -08:00
Johanna Amann
b1040f88c9 &on_change wrapup: documentation, tests, whitespacing
Adds documentation, fixes a whitespace issues, fixes compiler warning on
some Linux system, extends test.
2020-02-05 14:15:14 -08:00
Jon Siwek
43e54c7930 GH-780: Prevent log batches from indefinite buffering
Logs that got sent sparsely or burstily would get buffered for long
periods of time since the logic to flush them only does so on the next
log write.  In the worst case, a subsequent log write could never happen
and cause a log entry to be indefinitely buffered.

This fix introduces a recurring event/timer to simply flush all pending
logs at frequency of Broker::log_batch_interval.
2020-02-05 13:06:52 -08:00
Tim Wojtulewicz
4f23c0360a GH-773: Make EventMgr an IOSource so that events interrupt kevent waits 2020-02-05 15:43:37 -05:00
Johanna Amann
b6a244f784 Merge remote-tracking branch 'origin/master' into topic/johanna/table-on-change 2020-02-05 12:41:19 -08:00
Johanna Amann
18ab7838a9 Table expiry now raises &on_change handlers
Expiry is remprted as ELEMENT_EXPIRED (which was newly introduced), to
allow distinction from table deletion.
2020-02-05 12:37:52 -08:00
Johanna Amann
7f9f66fce9 Prevent recursion of &on_change handlers.
This change prevents &on_change handlers for a table from running if an
&on_change handler for the same table is already running.
2020-02-05 11:45:16 -08:00
Jon Siwek
0de6bba95e Merge remote-tracking branch 'origin/topic/johanna/692'
* origin/topic/johanna/692:
  Resolve race condition in test
  Change threading formatter errors to warnings.
2020-02-05 10:52:39 -08:00
Johanna Amann
50ebbc90ad Resolve race condition in test 2020-02-05 09:34:29 -08:00
Johanna Amann
deaab42187 Change threading formatter errors to warnings.
This change standardizes threading formatter error handling and moves
the remaining error calls to be warnings instead.

This is in line with already existing code - in most cases warnings were
raised, only a few cases raised errors. These cases do not differ
significantly from other cases in which warnings are raised.

This also fixes GH-692, in which misformatted lines prevent future file
parsing.

This commit also moves the FailWarn method that is used by both the
config and the ascii reader up to the ReaderBackend. Furthermore it
makes the Warning method of ReaderBackend respect the warning
suppression that is introduced by the FailWarn method.
2020-02-04 16:31:29 -08:00
Tim Wojtulewicz
5171f25e3a Update NEWS about plugin breakage from the include-file changes 2020-02-04 14:15:14 -07:00
Tim Wojtulewicz
4a7da2669c Merge remote-tracking branch 'max/include_cleanup'
* max/include_cleanup:
  UID, ..: un-inline methods to reduce header dependencies
  include cleanup
2020-02-04 14:06:05 -07:00
Jon Siwek
99b1dccb8d Updating submodule(s).
[nomail]
2020-02-04 12:08:13 -08:00
Max Kellermann
6a815b4b06 UID, ..: un-inline methods to reduce header dependencies
Only 1% build time speedup, but still, it declutters the headers a bit.

Before this patch:

 2565.17user 141.83system 2:25.46elapsed 1860%CPU (0avgtext+0avgdata 1489076maxresident)k
 72576inputs+9130920outputs (1667major+49400430minor)pagefaults 0swaps

After this patch:

 2537.19user 142.94system 2:26.90elapsed 1824%CPU (0avgtext+0avgdata 1434268maxresident)k
 16240inputs+8887152outputs (1931major+48728888minor)pagefaults 0swaps
2020-02-04 20:51:02 +01:00
Max Kellermann
0db61f3094 include cleanup
The Zeek code base has very inconsistent #includes.  Many sources
included a few headers, and those headers included other headers, and
in the end, nearly everything is included everywhere, so missing
#includes were never noticed.  Another side effect was a lot of header
bloat which slows down the build.

First step to fix it: in each source file, its own header should be
included first to verify that each header's includes are correct, and
none is missing.

After adding the missing #includes, I replaced lots of #includes
inside headers with class forward declarations.  In most headers,
object pointers are never referenced, so declaring the function
prototypes with forward-declared classes is just fine.

This patch speeds up the build by 19%, because each compilation unit
gets smaller.  Here are the "time" numbers for a fresh build (with a
warm page cache but without ccache):

Before this patch:

 3144.94user 161.63system 3:02.87elapsed 1808%CPU (0avgtext+0avgdata 2168608maxresident)k
 760inputs+12008400outputs (1511major+57747204minor)pagefaults 0swaps

After this patch:

 2565.17user 141.83system 2:25.46elapsed 1860%CPU (0avgtext+0avgdata 1489076maxresident)k
 72576inputs+9130920outputs (1667major+49400430minor)pagefaults 0swaps
2020-02-04 20:51:02 +01:00
Robin Sommer
532c66df51 Updating submodule.
[nomail]
2020-02-04 11:58:06 +00:00
Jon Siwek
6d2a789243 Updating submodule(s).
[nomail]
2020-02-03 18:01:17 -08:00
Tim Wojtulewicz
37e0130461 Merge remote-tracking branch 'origin/topic/jsiwek/consistent-string-view-args'
* origin/topic/jsiwek/consistent-string-view-args:
  Change various functions to by-value std::string_view args
2020-02-03 15:44:28 -07:00
Tim Wojtulewicz
5aa8120259 Merge remote-tracking branch 'origin/topic/jsiwek/pktsrc-options'
* origin/topic/jsiwek/pktsrc-options:
  Change packet source fields of Options to std::optional
2020-02-03 15:42:48 -07:00
Johanna Amann
2129a808f6 Updating CHANGES and VERSION after messing up last commit message. 2020-02-03 13:25:57 -08:00