Commit graph

9997 commits

Author SHA1 Message Date
Johanna Amann
c20551e48e Update submodule
[nomail]
2020-02-03 13:23:22 -08:00
Jon Siwek
2cda94e8ee Disable Travis leak test
Travis environment doesn't offer enough memory for running ASan very
well, but Cirrus has it covered anyway.
2020-02-03 13:21:32 -08:00
Jon Siwek
d33613c2a5 Merge branch 'master' of https://github.com/ffontaine/zeek
- Also removed the setting of BinPAC_ROOT_DIR in the configure
  script's --with-binpac= option as that breaks the cross-compilation
  use-case

* 'master' of https://github.com/ffontaine/zeek:
  CMakeLists.txt: fix cross-compilation with binpac
2020-02-03 13:05:50 -08:00
Jon Siwek
1e2dd4466c Change requested memory in Cirrus CI config
The ASan build does require quite a bit more, but also the default
release builds should do ok with slightly less.
2020-02-03 10:11:46 -08:00
Jon Siwek
1df5793479 Merge remote-tracking branch 'origin/topic/timw/ioloop-followup'
* origin/topic/timw/ioloop-followup:
  Increase timeout value for live interfaces without file descriptors
  Use ranged-for loops in a few places in iosource Manager
  Change order of includes in iosource Manager, fixes build on FreeBSD 11
2020-02-03 10:04:26 -08:00
Tim Wojtulewicz
edb87821da Increase timeout value for live interfaces without file descriptors 2020-02-03 07:34:27 -07:00
Tim Wojtulewicz
efaa756539 Use ranged-for loops in a few places in iosource Manager 2020-02-03 07:34:27 -07:00
Tim Wojtulewicz
d12d0332a7 Change order of includes in iosource Manager, fixes build on FreeBSD 11 2020-02-01 19:23:17 -07:00
Fabrice Fontaine
63344b32f5 CMakeLists.txt: fix cross-compilation with binpac
When cross-compiling, BINPAC_EXE_PATH will be set by the user to the
host binpac binary which is fine however aux/binpac won't be built which
will raise a build failure as target binpac (headers, library) won't be
installed or built

Signed-off-by: Fabrice Fontaine <fontaine.fabrice@gmail.com>
2020-02-01 14:38:17 +01:00
Jon Siwek
2f36113743 Change packet source fields of Options to std::optional 2020-01-31 15:41:23 -08:00
Jon Siwek
cd74d6f392 Change various functions to by-value std::string_view args 2020-01-31 15:08:48 -08:00
Jon Siwek
b0a5eb27b1 Updating submodule(s).
[nomail]
2020-01-31 14:19:21 -08:00
Jon Siwek
fa5b3bb91e Merge branch 'no_sscanf' of https://github.com/MaxKellermann/zeek
* 'no_sscanf' of https://github.com/MaxKellermann/zeek:
  util: optimize expand_escape() by avoiding sscanf()
2020-01-31 14:19:12 -08:00
Jon Siwek
d39bb42b14 Merge branch 'optimize_normalize_path' of https://github.com/MaxKellermann/zeek
- Minor changes in merge: extended unit test, prefer emplace_back(),
  remove unused "found" count in new function

* 'optimize_normalize_path' of https://github.com/MaxKellermann/zeek:
  util: add a tokenize_string() overload which returns string_views
  util: store std::string_view in "final_components" vector
  util: use "auto" in normalize_path()
  util: reserve space in normalize_path()
  util: skip "." completely in normalize_path()
  util: pass std::string_view to normalize_path()
  util: pass std::string_view to tokenize_string()
  util: don't modify the input string in tokenize_string()
2020-01-31 13:23:39 -08:00
Jon Siwek
719011cef0 Remove extra fmt() in a reporter->Error() call
Else compiler may warn about format string not being a string literal
2020-01-31 11:25:07 -08:00
Jon Siwek
0c68730b8b Merge branch 'fix-function-start-end-location' of https://github.com/awelzel/zeek
* 'fix-function-start-end-location' of https://github.com/awelzel/zeek:
  parse.y: Properly set location info for functions
2020-01-31 11:18:07 -08:00
Jon Siwek
e2e90ac477 Merge branch 'misc_cleanup' of https://github.com/MaxKellermann/zeek
- Minor code formatting change in merge

* 'misc_cleanup' of https://github.com/MaxKellermann/zeek:
  Desc: move realloc() call out of the loop
  SerializationFormat: move realloc() call out of the loop
  PacketDumper: remove unused types
2020-01-31 11:16:56 -08:00
Jon Siwek
077fdac332 Merge remote-tracking branch 'origin/topic/timw/264-alternative'
* origin/topic/timw/264-alternative:
  Add better error messaging when RegisterFd/UnregisterFd fail
  Update Supervisor code for the new IOSource API
  Show percentage of packets dropped in the final process output
  Update NEWS and docs submodule
  Remove concept of multiple timer managers
  Test changes caused by minor order-of-operation changes related to the new loop architecture
  Have terminate_processing() raise SIGTERM instead of calling the signal handler directly
  PktSrc iosource changes to match the new IOSource API
  Broker manager changes to match the new IOSource API and loop architecture
  DNS_Mgr changes to match the new IOSource API and loop architecture
  Threading changes for the new loop architecture
  Add Trigger manager for managing triggers created by things like 'when' statements
  Make TimerMgr an IOSource
  Modify IOSource Manager to implement new loop architecture
  Minor amount of code cleanup in Pcap IO source
  IOSource API changes for new loop architecture
  Type and variable usage cleanup in Net.h
  Remove #include of some iosource files from Net.h
  Only allow a single trace file (-r) or interface (-i) option on the command-line
  Remove CQ_TimerMgr in favor of PQ_TimerMgr
2020-01-31 11:04:27 -08:00
Johanna Amann
7166cb7373 Merge remote-tracking branch 'origin/master' into topic/johanna/table-on-change 2020-01-31 10:06:52 -08:00
Tim Wojtulewicz
fea0339aca Add better error messaging when RegisterFd/UnregisterFd fail 2020-01-31 10:13:10 -07:00
Tim Wojtulewicz
0cfb115c1b Update Supervisor code for the new IOSource API 2020-01-31 10:13:10 -07:00
Tim Wojtulewicz
6ddcc87c48 Show percentage of packets dropped in the final process output 2020-01-31 10:13:10 -07:00
Tim Wojtulewicz
fda8e3fd79 Update NEWS and docs submodule 2020-01-31 10:13:10 -07:00
Tim Wojtulewicz
be42608b51 Remove concept of multiple timer managers
- All timers are now handled by a single global timer manager, which simplifies how they handled by the IOSource manager.
- This change flows down a number of changes to other parts of the code. The timer manager tag field is removed, which means that matching connections to a timer manager is also removed. This removes the ability to tag a connection as internal or external, since that's how the connections where differentiated. This in turn removes the `current_conns_extern` field from the `ConnStats` record type in the script layer.
2020-01-31 10:13:09 -07:00
Tim Wojtulewicz
2dcc936787 Test changes caused by minor order-of-operation changes related to the new loop architecture 2020-01-31 10:13:09 -07:00
Tim Wojtulewicz
a2a2ff325f Have terminate_processing() raise SIGTERM instead of calling the signal handler directly 2020-01-31 10:13:09 -07:00
Tim Wojtulewicz
4751783d56 PktSrc iosource changes to match the new IOSource API 2020-01-31 10:13:09 -07:00
Tim Wojtulewicz
c5462eaa80 Broker manager changes to match the new IOSource API and loop architecture 2020-01-31 10:13:09 -07:00
Tim Wojtulewicz
c4d9566294 DNS_Mgr changes to match the new IOSource API and loop architecture 2020-01-31 10:13:09 -07:00
Tim Wojtulewicz
8b9160fb7e Threading changes for the new loop architecture
- threading::Manager is no longer an IOSource.
- threading::MsgThread is now an IOSource. This allows threads themselves to signal when they have data to process instead of continually checking each of the threads on every loop pass.
- Make the thread heartbeat timer an actual timer and let it fire as necessary instead of checking to see if it should fire
2020-01-31 10:13:09 -07:00
Tim Wojtulewicz
a159d075cf Add Trigger manager for managing triggers created by things like 'when' statements
- Adds new trigger namespace
- Adds trigger::Manager class as a new IOSource for keeping track of triggers and integrating them into the loop. Previously the loop relied on the event manager Drain() method to process all triggers on every loop, but now that the loop actively waits for events to occur, triggers would not fire when they needed to. Adding them as part of the loop ensures they're checked.
2020-01-31 10:13:09 -07:00
Tim Wojtulewicz
92bde61b78 Make TimerMgr an IOSource
- This allows the loop to check what the next timeout is and use that as the basis for the timeout of the poll
- This commit also removes the TimerMgr::Tag type, since it causes a name clash with other code in IOSource
2020-01-31 10:13:09 -07:00
Tim Wojtulewicz
4fa3e4b9b4 Modify IOSource Manager to implement new loop architecture
- Removes entire FindSoonest method that includes all of the calls to select() for checking for ready sources
- Removes FD_Set checking against IOSources
- Adds system for registering and unregistering file descriptors from IOSources. This allows individual sources to mark themselves as ready to be checked by the loop as they become available.
- Adds entirely new loop architecture based on checking the IOSources for when their next timeout is, and then waiting for either that timeout or when the next source is ready. This also implements the polling based on what the OS supports, instead of just calling select() on all platforms. Currently it supports kqueue, epoll, and plain poll.
- Adds system for pinging the loop to force it to wake up
2020-01-31 10:13:09 -07:00
Tim Wojtulewicz
062cadb124 Minor amount of code cleanup in Pcap IO source 2020-01-31 09:34:54 -07:00
Tim Wojtulewicz
a00d11e44d IOSource API changes for new loop architecture
- Removed GetFD and NextTimestamp methods
- Removed concept of idle sources
- Renamed Init to InitSource to avoid a name clash with the thread code
- Added GetNextTimeout method

This commit also removes the FD_Set file since it's no longer used
2020-01-31 09:34:54 -07:00
Tim Wojtulewicz
eee2abf729 Type and variable usage cleanup in Net.h 2020-01-31 09:34:54 -07:00
Tim Wojtulewicz
fa9a568e8f Remove #include of some iosource files from Net.h 2020-01-31 09:34:54 -07:00
Tim Wojtulewicz
f16f0360ff Only allow a single trace file (-r) or interface (-i) option on the command-line 2020-01-31 09:34:54 -07:00
Tim Wojtulewicz
2b2121be60 Remove CQ_TimerMgr in favor of PQ_TimerMgr 2020-01-31 09:34:54 -07:00
Max Kellermann
26da10ca05 util: add a tokenize_string() overload which returns string_views
Additionally, it uses a single "char" as delimiter, which is also
faster.

This patch speeds up Zeek startup by 10%.
2020-01-31 13:46:45 +01:00
Max Kellermann
763afe6f5f util: store std::string_view in "final_components" vector
Don't copy those path segments - instead, use std::string_view to
store references into the existing std::strings.  This saves a good
amount of allocation overhead.
2020-01-31 13:46:45 +01:00
Max Kellermann
37cbd98e34 util: use "auto" in normalize_path() 2020-01-31 13:46:45 +01:00
Max Kellermann
53c4e30024 util: reserve space in normalize_path()
Pessimistic reservations to ensure that it does not need to be
reallocated.
2020-01-31 13:46:45 +01:00
Max Kellermann
5c0c336c6b util: skip "." completely in normalize_path()
Don't copy "." segments to the final_components list only to remove it
afterwards.
2020-01-31 13:46:45 +01:00
Max Kellermann
0589f295fa util: pass std::string_view to normalize_path()
Reduce overhead in some callers.
2020-01-31 13:46:44 +01:00
Max Kellermann
f1566bda14 util: pass std::string_view to tokenize_string()
This saves some overhead because some callers pass a plain C string
here which needed to be copied to a temporary std::string.
2020-01-31 13:46:42 +01:00
Max Kellermann
e068ad8a53 util: don't modify the input string in tokenize_string()
This saves one full copy of the input string and avoids moving memory
around at O(n^2) in the erase() call in each loop iteration.
2020-01-31 13:42:30 +01:00
Arne Welzel
800c0c7132 parse.y: Properly set location info for functions
When defining a function, remember the location where the function header
was and restore it before calling `end_func()`. Inside `end_func()`, a
`BroFunc` object is created using the current global location information.

This came up while experimenting with zeek script profiling and wondering
why the locations set for `BroFunc` were "somewhere" in the middle of
functions instead of spanning them.
2020-01-31 10:47:20 +01:00
Max Kellermann
0b3317b1c2 util: optimize expand_escape() by avoiding sscanf()
sscanf() is notoriously slow, and the default scripts have lots of hex
escapes.  This patch reduces Zeek's startup time by 9%.

Before:

            245.04 msec task-clock:u              #    1.002 CPUs utilized
                 0      context-switches:u        #    0.000 K/sec
                 0      cpu-migrations:u          #    0.000 K/sec
            16,411      page-faults:u             #    0.067 M/sec
       629,238,575      cycles:u                  #    2.568 GHz
     1,237,236,556      instructions:u            #    1.97  insn per cycle
       262,223,957      branches:u                # 1070.142 M/sec
         3,351,083      branch-misses:u           #    1.28% of all branches

After:

            220.99 msec task-clock:u              #    1.002 CPUs utilized
                 0      context-switches:u        #    0.000 K/sec
                 0      cpu-migrations:u          #    0.000 K/sec
            16,419      page-faults:u             #    0.074 M/sec
       544,603,653      cycles:u                  #    2.464 GHz
     1,065,862,324      instructions:u            #    1.96  insn per cycle
       229,207,957      branches:u                # 1037.181 M/sec
         3,045,270      branch-misses:u           #    1.33% of all branches
2020-01-31 10:32:37 +01:00
Jon Siwek
70b45d1aba Merge remote-tracking branch 'origin/topic/robin/631-deprecation-v2'
During merge I split the test for bro_init/bro_done/bro_script_loaded
event errors into individual tests since the other testing of the zeek
versions of those events seemed fine to otherwise keep.

* origin/topic/robin/631-deprecation-v2:
  Update NEWS for naming changes.
  Small cleanup and updating submodules.
  Remove test for legacy plugin.
  Remove legancy symlinks in aux/.
  Add warnings when loading scripts ending in ".bro", or using legacy environment variables.
  Fix missing rename.
  No longer symlink local.zeek to local.bro.
  Update notice user agent.
  Remove old_comm_usage_is_ok.
  Remove bro-config.h.in and bro-path-dev.in.
  Change Bro wrapper script to now abort when old executable names are still used.
  Remove APIs that were explicitly deprecated to be removed in 3.1.
2020-01-30 19:19:56 -08:00