Commit graph

2919 commits

Author SHA1 Message Date
Jon Siwek
4bf3c6ff22 Improve &default validation for global vars: exclude sets 2019-10-15 18:24:35 -07:00
Johanna Amann
5265613a9e Merge remote-tracking branch 'origin/topic/jsiwek/gh-618-tcp-option-values'
* origin/topic/jsiwek/gh-618-tcp-option-values:
  Add weirds for invalid TCP option lengths
  GH-618: add "tcp_options" event containing TCP option values
  Fix tcp_option event
2019-10-15 16:50:43 -07:00
Johanna Amann
f62a51b41b Merge remote-tracking branch 'origin/topic/jsiwek/fix-table-default-redef'
* origin/topic/jsiwek/fix-table-default-redef:
  Fix redef'ing a table with a new &default attribute
2019-10-14 22:06:35 -07:00
Johanna Amann
66effde97d Merge remote-tracking branch 'origin/topic/seth/624'
* origin/topic/seth/624:
  Support whitespace at end of line for config reader.

This merge fixes a failing test; it also sprinkles a few more spaces
into another test file.

The main change is that this now also works with configuration lines
that don't have a value.
2019-10-14 21:47:59 -07:00
Seth Hall
cd06cb24cc Support whitespace at end of line for config reader.
This commit addresses issue #624
2019-10-14 11:43:16 -04:00
Jon Siwek
42b6040952 Fix redef'ing a table with a new &default attribute 2019-10-11 18:57:03 -07:00
Johanna Amann
916205e254 Merge remote-tracking branch 'origin/topic/jsiwek/print-raw-bif'
* origin/topic/jsiwek/print-raw-bif:
  Add new BIF: print_raw()
2019-10-08 15:15:55 -07:00
Johanna Amann
e615d1f4a8 Merge remote-tracking branch 'origin/topic/jsiwek/gh-589-sub-microsecond-printing'
* origin/topic/jsiwek/gh-589-sub-microsecond-printing:
  GH-589: improve printing of sub-microsecond intervals
2019-10-07 10:51:40 -07:00
Jon Siwek
052feacbda GH-618: add "tcp_options" event containing TCP option values 2019-10-03 18:59:02 -07:00
Jon Siwek
222e3ad3ea Fix tcp_option event
It was not being raised in all the cases it should have been due to
an incorrect/unnecessary truncation check.
2019-10-03 17:55:07 -07:00
Jon Siwek
39b06e3cee GH-591: allow Config::set_value() to use empty/unspecified table/sets 2019-10-02 22:13:43 -07:00
Jon Siwek
fd66e7b9f6 GH-591: fix reading set[enum] values from input files 2019-10-02 22:09:16 -07:00
Jon Siwek
5622df4d1b Add new BIF: print_raw()
This is a convenience function to make it easier to print literal byte
sequences to stdout without additional escaping like what may be added
by the default `print` statement behavior.

For example, related to GH-596, `print` currently escapes even valid
UTF-8 byte sequences and makes it difficult to output valid JSON strings
containing such.
2019-10-02 15:21:24 -07:00
Jon Siwek
a5d71ed2d2 Merge remote-tracking branch 'origin/topic/timw/595-json-perf'
* origin/topic/timw/595-json-perf:
  Update COPYING.3rdparty
  Use json::emplace to avoid some extra calls to operator[]
  Use tessil/unordered-map instead of nlohmann/fifo-map to mitigate performance issues when logging JSON
2019-10-01 16:42:59 -07:00
Jon Siwek
a34ade4876 Improve RecordVal JSON formatting
No need to create a record introspection table each time when all
the required information can be obtained directly in the RecordVal
and RecordType objects.  Besides the additional overhead, using such
a table will re-order the fields arbitrarily instead of using the
order in which they're defined.
2019-09-30 19:04:55 -07:00
Jon Siwek
d258ebe5ca Fix memory leak in RecordVal JSON formatting 2019-09-30 19:02:33 -07:00
Jon Siwek
97519e4350 Merge remote-tracking branch 'origin/topic/timw/606-json-nulls'
* origin/topic/timw/606-json-nulls:
  GH-606: Output nulls into json data if a field isn't set
2019-09-30 17:40:27 -07:00
Tim Wojtulewicz
d4c394b72a GH-606: Output nulls into json data if a field isn't set 2019-09-27 14:12:48 -07:00
Tim Wojtulewicz
756f363185 GH-598: handle multi-key sets correctly when outputting json 2019-09-27 12:01:47 -07:00
Tim Wojtulewicz
d4d1009b5c Use json::emplace to avoid some extra calls to operator[] 2019-09-27 09:58:24 -07:00
Jon Siwek
f287d9abe8 GH-589: improve printing of sub-microsecond intervals 2019-09-23 19:57:49 -07:00
Jon Siwek
aeef4bf030 Merge branch 'topic/jgras/dpd-late-match' of https://github.com/J-Gras/zeek
* 'topic/jgras/dpd-late-match' of https://github.com/J-Gras/zeek:
  Improve dpd_late_match event generation.
  Improve logging of speculative service.
  Update test-all-policy script.
  Add speculative service script.
  Allow to handle late DPD matches.
2019-09-17 11:17:41 -07:00
Robin Sommer
6f9d1ec72d Merge remote-tracking branch 'origin/topic/jsiwek/gh-566-fix-ssh-encrypted-packet'
* origin/topic/jsiwek/gh-566-fix-ssh-encrypted-packet:
  GH-566: fix cases where ssh_encrypted_packet event wasn't raised
2019-09-17 17:22:27 +00:00
Dominik Charousset
c1f3fe7829 Switch from header guards to pragma once 2019-09-17 14:10:30 +02:00
Jon Siwek
9c8db5f6ca Stop loading scan detection in local.zeek by default 2019-09-16 10:51:50 -07:00
Jon Siwek
30da2f83d0 GH-566: fix cases where ssh_encrypted_packet event wasn't raised
When encrypted data was bundled within the same segment as the NewKeys
message, it wasn't not reported via a ssh_encrypted_package event as
it should have been.
2019-09-03 17:34:24 -07:00
Jan Grashoefer
81b2b21211 Improve logging of speculative service. 2019-08-30 15:16:37 +02:00
Jan Grashoefer
788b56a652 Add speculative service script.
The speculative service script handles dpd_late_match events to extend
conn.log with infos about potential protocol identifications.
2019-08-29 11:47:04 +02:00
Johanna Amann
bb98559c0d Merge remote-tracking branch 'origin/topic/jsiwek/gh-545-weird-addl'
* origin/topic/jsiwek/gh-545-weird-addl:
  GH-545: add "addl" parameter to flow_weird and net_weird events
2019-08-28 14:27:53 -07:00
Johanna Amann
ec57894a85 Merge remote-tracking branch 'origin/topic/jsiwek/gh-541-ntlm-fix'
* origin/topic/jsiwek/gh-541-ntlm-fix:
  GH-541: add test cases for NTLM AV Pair sequence handling
  GH-541: fix handling of NTLM AV Pair sequences
2019-08-28 11:33:49 -07:00
Jon Siwek
289a1e2e8e Merge branch '555-smb3-negotiate-context-fix' of https://github.com/mad/zeek
- Fixed the context list padding to only be used for dialect 0x0311.
  The new test case includes an example where parsing the optional
  padding would fail for another dialect.

* '555-smb3-negotiate-context-fix' of https://github.com/mad/zeek:
  Fix for smb3 negotiate context
2019-08-27 10:08:42 -07:00
Pavel Ershov
de4a83206d Fix for smb3 negotiate context 2019-08-27 12:21:03 +03:00
Jon Siwek
b954767488 GH-541: add test cases for NTLM AV Pair sequence handling 2019-08-26 10:28:46 -07:00
Tim Wojtulewicz
01207c0720 Fix unit tests for new ordering from NetSessions::Drain 2019-08-21 09:24:02 -04:00
Jon Siwek
2d7c926291 GH-545: add "addl" parameter to flow_weird and net_weird events 2019-08-20 22:45:22 -04:00
Robin Sommer
8ab0650c1e Merge remote-tracking branch 'origin/topic/jsiwek/gh-532-improve-disable-analyzer'
Includes fix for potential iterator invalidation during iteration.

* origin/topic/jsiwek/gh-532-improve-disable-analyzer:
  GH-532: improve disable_analyzer BIF
2019-08-16 18:46:35 +00:00
Jon Siwek
9ccf3549fd Merge remote-tracking branch 'origin/topic/timw/360-log-gzip'
* origin/topic/timw/360-log-gzip:
  Add new LogAscii::gzip_file_extension option.

Adjusted documentation during merge.
2019-08-13 13:49:45 -07:00
Jon Siwek
c91b12863e Add tests for {http,mime}_all_headers events
And remove unnecessary check for mime_all_headers in HTTP entities
(they ony raise the http_all_headers event, never mime_all_headers).
2019-08-13 11:28:00 -07:00
Jon Siwek
18e4976c6c Merge remote-tracking branch 'origin/topic/timw/travis-leaks'
* origin/topic/timw/travis-leaks:
  fixup! Add new distro to Travis CI configuration for running leak tests
  Add new distro to Travis CI configuration for running leak tests
2019-08-12 13:01:55 -07:00
Jon Siwek
a1c5eddb95 GH-532: improve disable_analyzer BIF
- Add an extra "prevent" parameter (default value of false), which
  helps prevent the same analyzer type from being attached in the
  future.  It's useful in situations where you want to disable early
  on, but a DPD signature may still trigger later and re-attach
  the same analyzer.  E.g. when not using this flag, but calling
  disable_analyzer() inside an http_request event, will remove the
  HTTP analyzer that was attached due to well-known-port, but a later
  DPD signature match from upon seeing the HTTP reply will end up
  attaching another HTTP analyzer.  More surprising is that upon
  re-attaching that analyzer, you'll get the same http_request as
  before since the DPD buffer will get replayed into the new analyzer.

- Fixes disable_analyzer() to work when called even earlier, like
  within the protocol_confirmation event.  At that time, the
  Analyzer tree may have not properly added the new analyzer into
  Analyzer::children yet, but rather the temporary waiting list,
  Analyzer::new_children.  Analyzer::RemoveChildAnalyzer previously
  did not inspect the later list.

- Fixes disable_analyzer() when called on an analyzer added to the
  tree via TCP_Analyzer::AddChildPacketAnalyzer.  TCP_Analyzer
  keeps track of such children in its own list,
  TCP_Analyzer::packet_children, which the previous
  Analyzer::RemoveChildAnalyzer implementation didn't inspect.
2019-08-09 20:03:26 -07:00
Tim Wojtulewicz
b9f6992c0e fixup! Add new distro to Travis CI configuration for running leak tests 2019-08-09 11:01:16 -07:00
Jon Siwek
bf9b983f00 Merge branch 'known_services_multiprotocols' of https://github.com/mauropalumbo75/zeek
* 'known_services_multiprotocols' of https://github.com/mauropalumbo75/zeek:
  improve logging with broker store
  drop services starting with -
  remove service from key for Cluster::publish_hrw
  remove check for empty services
  update tests
  order list of services in store key
  remove repeated services in logs if already seen
  add multiprotocol known_services when Known::use_service_store = T
  remove hyphen in front of some services (for example -HTTP, -SSL)   In some cases, there is an hyphen before the protocol name in the field   connection$service. This can cause problems in known_services and   is removed here. It originates probably in some analyzer where it   would be better removed in the future.
  add multiprotocol known_services when Known::use_service_store = F

Changes during merge:
  * whitespace
  * add unit test
2019-08-09 10:47:34 -07:00
Jon Siwek
a68c9f6b71 Merge branch 'empty_services' of https://github.com/mauropalumbo75/zeek
* 'empty_services' of https://github.com/mauropalumbo75/zeek:
  remove empty services and include udp active connections when logging in connection_state_remove
2019-08-09 09:59:50 -07:00
Tim Wojtulewicz
80e3c42526 Add new distro to Travis CI configuration for running leak tests 2019-08-08 16:36:17 -07:00
Jon Siwek
ba56cf62a7 Change version to 3.1.0-dev 2019-08-08 16:08:50 -07:00
Jon Siwek
d236f377e7 Add memory leak test of closure (un)serialization 2019-08-07 12:41:22 -07:00
Jon Siwek
35c42b4b09 GH-474: change MQTT::max_payload_size to be a runtime option 2019-08-05 18:11:54 -07:00
Johanna Amann
0f96a9dedf Disable MQTT by default
To enable MQTT, one has to load policy/scripts/mqtt. Like with smb in
2.5, the consts are loaded by default.
2019-08-05 17:04:39 -07:00
Tim Wojtulewicz
337da50da6 Add new LogAscii::gzip_file_extension option.
This can be used with the LogAscii::gzip_level option to set the file extension of log files when they are compressed at creation time.
2019-08-05 14:36:34 -07:00
Jon Siwek
6bc947a48e GH-474: add MQTT::max_payload_size option
This caps size of payload strings within mqtt_publish events and
mqtt_publish.log files.  A new "payload_len" field in the log file
shows the real payload size in cases where it may have been truncated.
2019-08-02 14:28:55 -07:00