Commit graph

10798 commits

Author SHA1 Message Date
Justin Azoff
751ea5edd7 avoid scheduling redundant inactivity timers
When setting the inactivity timeout, don't schedule a new timer if the
new one would be redundant.
2020-05-01 09:36:50 -04:00
Jon Siwek
c94638a9aa Update submodule(s)
[nomail]
2020-04-30 18:30:51 -07:00
Jon Siwek
009c1488fc Update submodule(s)
[nomail]
2020-04-30 16:41:05 -07:00
Jon Siwek
1b190906c7 GH-938: fix IO loop iterations sometimes skipping offline pcap sources 2020-04-30 16:19:30 -07:00
Tim Wojtulewicz
499a3353b5 Change timer_list in BroList to be an unordered list.
This type is used by Conn and Analyzer to hold onto timers being added and removed. We
don't expect the elements in those lists to maintain an order as the list is being
modified.
2020-04-30 15:23:48 -07:00
Tim Wojtulewicz
28e5100842 Add ability for List to be ordered/unordered
This fixes a "bug" with List where remove_nth() can be an O(n) operation when it doesn't
need to be. remove_nth for lists that don't necessarily need to keep an order can be an
O(1) operation instead.
2020-04-30 15:23:48 -07:00
Tim Wojtulewicz
0558a7bfed Add unit testing for List 2020-04-30 15:23:48 -07:00
Tim Wojtulewicz
615f8cd443 Fix bug with List where replace() doesn't work with non-pointer types 2020-04-30 15:23:48 -07:00
Jon Siwek
787ebd369e Improve standalone fuzzer driver error messages 2020-04-30 13:43:57 -07:00
Johanna Amann
3bce313b12 Switch file UID hashing from md5 to highwayhash.
This commit switches UID hashing from md5 to a highway hash. It also
moves the salt value out of the file plugin - and makes it
installation-specific instead - it is moved to the global namespace.

There now are digest hash functions to make "static"
installation-specific hashes that are stable over workers available to
everyone; hashes can be 64, 128 or 256 bits in size.

Due to the fact that we switch the file hashing algorithm, all file
hashes change.

The underlyigng algorithm that is used for hashing is highwayhash-128,
which is significantly faster than md5.
2020-04-30 10:20:09 -07:00
Jon Siwek
fda9498276 Merge remote-tracking branch 'origin/topic/seth/more-file-type-ident'
* origin/topic/seth/more-file-type-ident:
  Adds a note in NEWS about the signature addition and restructuring
  Fixes a small bug in one signature with a duplicate name.
  Organized and added to the shipped file identification signatures.
2020-04-29 16:32:41 -07:00
Johanna Amann
bc546634d1 Switch most internal md5 calls to digest calls.
The places that used md5 basically already used it as a digest
algorithm. Switching to a digest just means that the internal values
used to not change between runs - which is actually wanted in these
cases.

This commit also removes our special cmake subdirectory. We don't expose
highwayhash in headers anymore - so we can just treat it as an internal
implementation choice that is not directly exposed to plugins.
2020-04-29 16:05:31 -07:00
Jon Siwek
ccdaf5f111 Merge branch 'stats-logging-fix' of https://github.com/brittanydonowho/zeek
- Added test case and adjusted whitespace in merge

* 'stats-logging-fix' of https://github.com/brittanydonowho/zeek:
  Fixed stats.zeek to log all data before zeek terminates rather than return too soon
2020-04-29 15:58:35 -07:00
Jon Siwek
a08b1ff56f Add warning message for unknown Broker statuses
There's now a couple placeholder/unimplemented status values in Broker
related to upcoming routing features that we don't want to handle
explicitly for compatibility reasons, but also don't want the compiler
warning about unhandled values in the switch.
2020-04-29 15:28:18 -07:00
Brittany Donowho
97c8912443 Fixed stats.zeek to log all data before zeek terminates rather than return too soon 2020-04-29 17:37:29 -04:00
Jon Siwek
fddb06c3cd Update submodule(s)
[nomail]
2020-04-29 14:25:53 -07:00
Seth Hall
43661a2b0b Adds a note in NEWS about the signature addition and restructuring 2020-04-29 11:34:00 -04:00
Seth Hall
dac96a6be3 Fixes a small bug in one signature with a duplicate name.
Also update a single failing test.
2020-04-29 11:22:42 -04:00
Seth Hall
15d43dfbcd Organized and added to the shipped file identification signatures.
- Added ISO 9660 disk image
 - Created new files for categorizing signatures better.
   - executable.sig - Executable (and bytecode) files.
   - java.sig - Java related files (class/jar, etc).
   - programming.sig - Mostly scripting language identification
2020-04-29 11:08:32 -04:00
Jan Grashoefer
bcf2357632 Improve network_time_init() test and add baseline. 2020-04-29 13:53:30 +02:00
Jan Grashoefer
97d1acf86f Add network_time_init() event. 2020-04-29 13:19:57 +02:00
Jan Grashoefer
c99f825e22 Add test for network_time behavior. 2020-04-29 13:18:49 +02:00
Jon Siwek
2107c3e0b7 Extend CI config to cover building with libmaxminddb support 2020-04-28 17:37:31 -07:00
Jon Siwek
d533212a6a Merge branch 'master' into topic/jsiwek/fuzzing 2020-04-28 17:25:32 -07:00
Jon Siwek
4a76229209 GH-941: Fix build when configured to use libmaxminddb 2020-04-28 17:12:51 -07:00
Jon Siwek
db5248ad85 Test fuzzers against seed corpus under CI ASan build 2020-04-28 16:17:39 -07:00
Jon Siwek
f1d21fa489 Update fuzzing README with OSS-Fuzz integration notes 2020-04-28 14:32:37 -07:00
Robin Sommer
f7fd24cf15 Updating submodule. 2020-04-28 10:07:31 +00:00
Jon Siwek
8ec807bd76 Link fuzzers against shared library to reduce executable sizes 2020-04-27 20:04:33 -07:00
Jon Siwek
2922bf71b6 Improve FuzzBuffer chunking
Now allocates a new buffer for each chunk to better detect over-reads
2020-04-27 16:33:05 -07:00
Jon Siwek
8e6539b55f Fix compiler warning in standalone fuzzer driver 2020-04-27 15:53:40 -07:00
Jon Siwek
0623539d80 Adjust minor fuzzing documentation 2020-04-27 15:53:32 -07:00
Jon Siwek
91eff92335 Exit immediately after running unit tests 2020-04-27 15:26:44 -07:00
Jon Siwek
034b4b5ea7 Merge remote-tracking branch 'origin/topic/timw/intrusiveptr-warnings'
* origin/topic/timw/intrusiveptr-warnings:
  Fix a few more IntrusivePtr deprecation warnings
2020-04-27 13:38:45 -07:00
Jon Siwek
b776f17cea Merge remote-tracking branch 'origin/topic/vern/any-typetype-when-fix'
- Minor whitespace adjutment in merge

* origin/topic/vern/any-typetype-when-fix:
  bug fixes for using "when" in functions that have a local of type "any"
2020-04-27 13:33:40 -07:00
Tim Wojtulewicz
07a1821bcc Fix a few more IntrusivePtr deprecation warnings 2020-04-27 12:59:19 -07:00
Johanna Amann
faa8a38578 Merge remote-tracking branch 'origin/topic/jsiwek/gh-854-preserve-header-name'
* origin/topic/jsiwek/gh-854-preserve-header-name:
  GH-854: provide access to original HTTP/MIME header names
2020-04-27 19:31:49 +00:00
Johanna Amann
0136648342 Merge remote-tracking branch 'origin/topic/jsiwek/empty-bloomfilter-lookup'
* origin/topic/jsiwek/empty-bloomfilter-lookup:
  Remove error message from empty bloomfilter lookups
2020-04-27 19:23:16 +00:00
Johanna Amann
bb2f328cff Merge remote-tracking branch 'origin/topic/vern/unused'
* origin/topic/vern/unused:
  unused variables found via use-def analysis (plus an indentation micro-nit)
2020-04-27 19:21:33 +00:00
Tim Wojtulewicz
13674e7c31 Merge remote-tracking branch 'origin/topic/jsiwek/bif-return-intrusive'
* origin/topic/jsiwek/bif-return-intrusive:
  Update various BIFs to return IntrusivePtr
2020-04-27 12:09:58 -07:00
Jon Siwek
9e56881c70 Update various BIFs to return IntrusivePtr 2020-04-27 11:50:35 -07:00
Vern Paxson
fe46ef06a0 unused variables found via use-def analysis (plus an indentation micro-nit) 2020-04-25 18:06:47 -07:00
Vern Paxson
b86d5b4424 bug fixes for using "when" in functions that have a local of type "any" 2020-04-25 16:56:24 -07:00
Johanna Amann
360c06a3f8 Start refactoring hashing.
This commit moves some of the hash datastructures and code from
util.cc into Hash.cc - where it seems more appropriate.

It also starts to make more Keyed hash functions available - still
using siphash as the default 64 bit keyed hash, but also making
128 and 256 bit highway hashes available.

There already are a few other functions that are defined but not
yet implemented - these will be "static" keyed hashes - which use
an installation specific key. These will be used to, e.g., get
rid of md5 hashing for the generation of file UIDs.
2020-04-24 18:27:09 -07:00
Jon Siwek
98845e89aa Add OSS-Fuzz Zeek script search path to fuzzers 2020-04-24 17:53:52 -07:00
Johanna Amann
d34532f847 Introduce calculate_digest convenience function
This function just calculates the chosen digest and returns the result
in either the passed buffer, or in a static buffer. Basically a superset
to the surprisingly popular internal_md5.
2020-04-24 17:00:10 -07:00
Tim Wojtulewicz
17f72d6be6 Update submodule
[nomail]
2020-04-24 16:22:33 -07:00
Tim Wojtulewicz
cb40dbd58b Merge remote-tracking branch 'origin/topic/jsiwek/intrusive-ptr-chipping'
* origin/topic/jsiwek/intrusive-ptr-chipping:
  Deprecate returning Val* from BIFs
  Deprecate binpac::string_to_val
  Deprecate binpac::bytestring_to_val, replace with binpac::to_stringval
  Update deprecated BifEvent::generate_* usages
  Deprecate Connection::Event and Analyzer::Event methods
  Deprecate BuildConnVal() methods and update usages to ConnVal()
  Update all BIFs to return IntrusivePtr instead of Val*
  Update deprecated ValManager::GetPort usages
  Update deprecated ValManager::GetEmptyString usages
  Update deprecated ValManager::GetCount usages
  Update deprecated ValManager::GetInt usages
  Update deprecated ValManager::GetBool usages
  Update deprecated ValManager GetTrue/GetFalse usages
  Deprecate all ValManager "Get" methods
  Change BIFs to return a wrapper object
2020-04-24 16:20:08 -07:00
Jon Siwek
a4244bc72b Assume libFuzzer when LIB_FUZZING_ENGINE file doesn't exist
i.e. environment variable may be set, but not point to existing file
2020-04-24 14:51:32 -07:00
Jon Siwek
2ef182076e Change handling of LIB_FUZZING_ENGINE
Should better support OSS-Fuzz, which may set it to either
be path to library to link against or linker flag like
"-fsanitize=fuzzer"
2020-04-24 14:43:14 -07:00