Commit graph

6667 commits

Author SHA1 Message Date
Jon Siwek
5622df4d1b Add new BIF: print_raw()
This is a convenience function to make it easier to print literal byte
sequences to stdout without additional escaping like what may be added
by the default `print` statement behavior.

For example, related to GH-596, `print` currently escapes even valid
UTF-8 byte sequences and makes it difficult to output valid JSON strings
containing such.
2019-10-02 15:21:24 -07:00
Jon Siwek
a5d71ed2d2 Merge remote-tracking branch 'origin/topic/timw/595-json-perf'
* origin/topic/timw/595-json-perf:
  Update COPYING.3rdparty
  Use json::emplace to avoid some extra calls to operator[]
  Use tessil/unordered-map instead of nlohmann/fifo-map to mitigate performance issues when logging JSON
2019-10-01 16:42:59 -07:00
Jon Siwek
583458665d Updating submodule(s).
[nomail]
2019-10-01 12:03:48 -07:00
Robin Sommer
e263589140 Merge remote-tracking branch 'origin/topic/jsiwek/fix-to-json-record-leak'
* origin/topic/jsiwek/fix-to-json-record-leak:
  Improve RecordVal JSON formatting
  Fix memory leak in RecordVal JSON formatting
2019-10-01 09:06:30 +00:00
Johanna Amann
1b02aebe4e Merge remote-tracking branch 'origin/topic/jsiwek/paraglob-headers'
* origin/topic/jsiwek/paraglob-headers:
  GH-602: fix paraglob header inclusion, installation, and search paths
2019-09-30 21:10:12 -07:00
Jon Siwek
a34ade4876 Improve RecordVal JSON formatting
No need to create a record introspection table each time when all
the required information can be obtained directly in the RecordVal
and RecordType objects.  Besides the additional overhead, using such
a table will re-order the fields arbitrarily instead of using the
order in which they're defined.
2019-09-30 19:04:55 -07:00
Jon Siwek
d258ebe5ca Fix memory leak in RecordVal JSON formatting 2019-09-30 19:02:33 -07:00
Jon Siwek
97519e4350 Merge remote-tracking branch 'origin/topic/timw/606-json-nulls'
* origin/topic/timw/606-json-nulls:
  GH-606: Output nulls into json data if a field isn't set
2019-09-30 17:40:27 -07:00
Jon Siwek
3d4fef012a Merge remote-tracking branch 'origin/topic/timw/598-multikey-set-json'
Adjustments:
- Changed a push_back(...) into emplace_back(std:move(...))
- Removed superfluous table Lookup() since we already have the value
  while iterating

* origin/topic/timw/598-multikey-set-json:
  GH-598: handle multi-key sets correctly when outputting json
2019-09-30 17:34:11 -07:00
Jon Siwek
4959d438fa Initial structure for supervisor-mode
The full process hierarchy isn't set up yet, but these changes
help prepare by doing two things:

- Add a -j option to enable supervisor-mode.  Currently, just a single
  "stem" process gets forked early on to be used as the basis for
  further forking into real cluster nodes.

- Separates the parsing of command-line options from their consumption.
  i.e. need to parse whether we're in -j supervisor-mode before
  modifying any global state since that would taint the "stem" process.
  The new intermediate structure containing the parsed options may
  also serve as a way to pass configuration info from "stem" to its
  descendent cluster node processes.
2019-09-27 19:17:58 -07:00
Tim Wojtulewicz
d4c394b72a GH-606: Output nulls into json data if a field isn't set 2019-09-27 14:12:48 -07:00
Tim Wojtulewicz
756f363185 GH-598: handle multi-key sets correctly when outputting json 2019-09-27 12:01:47 -07:00
Tim Wojtulewicz
d4d1009b5c Use json::emplace to avoid some extra calls to operator[] 2019-09-27 09:58:24 -07:00
Jon Siwek
7613ab0af0 GH-602: fix paraglob header inclusion, installation, and search paths 2019-09-26 17:39:25 -07:00
Jon Siwek
bc18ca44e6 Fix Xcode deprecation warning for std::ptr_fun
Replaced logic in strstrip() with a lambda to avoid deprecations:

- std::ptr_fun is deprecated in C++11, removed C++17
- std::not1 is deprecated in C++17. removed C++20
2019-09-26 09:45:44 -07:00
Tim Wojtulewicz
c8f2d52d91 Use tessil/unordered-map instead of nlohmann/fifo-map to mitigate performance issues when logging JSON 2019-09-25 16:21:38 -07:00
Jon Siwek
1253a61340 Simplify reassembly code using DataBlockList::LastBlock() 2019-09-24 10:26:09 -07:00
Jon Siwek
5ce68bd20a Fix DataBlockList::DataSize()
Parameters got filled opposite to what they were supposed to be and
also didn't consider cutoffs that land in the middle of a block.
2019-09-24 10:21:20 -07:00
Robin Sommer
e30035910e Add missing commit to previous merge.
(This was a leftover from a merge conflict.)
2019-09-24 10:32:40 +00:00
Robin Sommer
c23764483d Merge remote-tracking branch 'origin/topic/jsiwek/reassembly-improvements-map'
* origin/topic/jsiwek/reassembly-improvements-map:
  Rename a reassembly DataBlockList function
  Add comments to reassembly classes
  Use DataBlock value instead of pointer in reassembly map
  Remove linked list from reassembly data structures
  Use an std::map for reassembly DataBlock searches
  Refactor Reassembler/DataBlock bookkeeping
  Reorganize reassembly data structures
  Remove a superfluous reassembler DataBlock member
2019-09-24 09:16:51 +00:00
Jon Siwek
f287d9abe8 GH-589: improve printing of sub-microsecond intervals 2019-09-23 19:57:49 -07:00
Jon Siwek
916491e2b4 Merge remote-tracking branch 'origin/topic/neverlord/scaffolding'
* origin/topic/neverlord/scaffolding:
  Fix second overly specific include_directories
  Add --build-dir as alias for --builddir
  Fix overly specific include_directories usage
2019-09-23 13:09:04 -07:00
Dominik Charousset
b56028fba0 Fix second overly specific include_directories 2019-09-21 12:17:06 +02:00
Jon Siwek
bd6f89da05 Merge remote-tracking branch 'origin/topic/robin/cid-1402823'
- Fixed a reference counting issue

* origin/topic/robin/cid-1402823:
  Fix for CIDs 1402823 and 1394050.
2019-09-20 10:42:58 -07:00
Robin Sommer
c38e9b2ff2 Fix for CIDs 1402823 and 1394050.
An InterpreterException from clone framing could go uncaught.
2019-09-20 07:55:09 +00:00
Jon Siwek
aeef4bf030 Merge branch 'topic/jgras/dpd-late-match' of https://github.com/J-Gras/zeek
* 'topic/jgras/dpd-late-match' of https://github.com/J-Gras/zeek:
  Improve dpd_late_match event generation.
  Improve logging of speculative service.
  Update test-all-policy script.
  Add speculative service script.
  Allow to handle late DPD matches.
2019-09-17 11:17:41 -07:00
Robin Sommer
6f9d1ec72d Merge remote-tracking branch 'origin/topic/jsiwek/gh-566-fix-ssh-encrypted-packet'
* origin/topic/jsiwek/gh-566-fix-ssh-encrypted-packet:
  GH-566: fix cases where ssh_encrypted_packet event wasn't raised
2019-09-17 17:22:27 +00:00
Jon Siwek
1affbad4b7 Merge remote-tracking branch 'origin/topic/neverlord/header-guards'
* origin/topic/neverlord/header-guards:
  Switch from header guards to pragma once
2019-09-17 10:09:04 -07:00
Jon Siwek
b19ffba25f Merge branch 'patch-1' of https://github.com/dcode/zeek
* 'patch-1' of https://github.com/dcode/zeek:
  Add additional 3rd party headers to includes
2019-09-17 09:33:46 -07:00
Derek Ditch
9d35aa1fef
Add additional 3rd party headers to includes
Adds FIFO Map and JSON headers to include distribution

Fixes #580
2019-09-17 07:40:04 -05:00
Dominik Charousset
c1f3fe7829 Switch from header guards to pragma once 2019-09-17 14:10:30 +02:00
Jon Siwek
2967724355 Rename a reassembly DataBlockList function 2019-09-16 13:43:04 -07:00
Tim Wojtulewicz
48e7169bd8 Add move assignment operator to logging::Tag (Coverity 1357732) 2019-09-16 10:56:41 -07:00
Tim Wojtulewicz
1cf8a530cf Throw an error during input::Manager::CreateTableStream if val is null (Coverity 1402730) 2019-09-16 10:56:41 -07:00
Tim Wojtulewicz
68accaa791 Use ntohl instead of manually swapping bytes in extract_XDR_uint32 (Coverity 1375796 and others) 2019-09-16 10:56:41 -07:00
Tim Wojtulewicz
aacd2134fa Add check for null init variable during make_var. (Coverity 1403419) 2019-09-16 10:56:41 -07:00
Tim Wojtulewicz
53c82cc872 Remove dead check in Val::check_and_promote. (Coverity 1401810) 2019-09-16 10:56:41 -07:00
Tim Wojtulewicz
c009cd3289 Handle failure of fcntl in Pipe. (Coverity 1241934, 1241935) 2019-09-16 10:56:41 -07:00
Tim Wojtulewicz
dddba3432f Initialize missing field in File. (Coverity 1057851, 1057852, 1057853) 2019-09-16 10:56:41 -07:00
Tim Wojtulewicz
0140098adb Add null check for results of dynamic_cast in AssignExpr::TypeCheck. Fixes coverity findings 1403416 and 1403417 2019-09-16 10:56:41 -07:00
Tim Wojtulewicz
40d4004453 Add null check when getting key size for List types. Fixes coverity finding 1058242 2019-09-16 10:56:41 -07:00
Jon Siwek
0caa30076f Add comments to reassembly classes 2019-09-13 15:23:03 -07:00
Jon Siwek
69d1620374 Use DataBlock value instead of pointer in reassembly map 2019-09-13 14:17:41 -07:00
Jon Siwek
e1e779e90b Remove linked list from reassembly data structures
Everything, including iteration is now done via an std::map
2019-09-13 13:57:32 -07:00
Jon Siwek
9b13825e16 Use an std::map for reassembly DataBlock searches
It's not free and adds some overhead to the common case where it
won't help much, but improves worst case overlap-checking situations.
2019-09-12 18:01:25 -07:00
Jon Siwek
989ae91c94 Refactor Reassembler/DataBlock bookkeeping
At least saves having to store a Reassembler pointer for each DataBlock
2019-09-11 16:25:34 -07:00
Jon Siwek
b19c8fad7a Reorganize reassembly data structures
Started by factoring some details into a new DataBlockList class to at
least make it more clear where modifications occur.  More abstractions
likely to happen later as I experiment with alternate data structures
aimed at improving worse-case scenarios.
2019-09-11 16:25:34 -07:00
Jon Siwek
395c685da1 Remove a superfluous reassembler DataBlock member 2019-09-11 16:25:34 -07:00
Jon Siwek
c88568db4d Update embedded CAF to 0.17.1 (plus cherry-picked memory leak fix) 2019-09-04 13:42:55 -07:00
Jon Siwek
30da2f83d0 GH-566: fix cases where ssh_encrypted_packet event wasn't raised
When encrypted data was bundled within the same segment as the NewKeys
message, it wasn't not reported via a ssh_encrypted_package event as
it should have been.
2019-09-03 17:34:24 -07:00