Commit graph

2898 commits

Author SHA1 Message Date
Jon Siwek
aeef4bf030 Merge branch 'topic/jgras/dpd-late-match' of https://github.com/J-Gras/zeek
* 'topic/jgras/dpd-late-match' of https://github.com/J-Gras/zeek:
  Improve dpd_late_match event generation.
  Improve logging of speculative service.
  Update test-all-policy script.
  Add speculative service script.
  Allow to handle late DPD matches.
2019-09-17 11:17:41 -07:00
Robin Sommer
6f9d1ec72d Merge remote-tracking branch 'origin/topic/jsiwek/gh-566-fix-ssh-encrypted-packet'
* origin/topic/jsiwek/gh-566-fix-ssh-encrypted-packet:
  GH-566: fix cases where ssh_encrypted_packet event wasn't raised
2019-09-17 17:22:27 +00:00
Dominik Charousset
c1f3fe7829 Switch from header guards to pragma once 2019-09-17 14:10:30 +02:00
Jon Siwek
9c8db5f6ca Stop loading scan detection in local.zeek by default 2019-09-16 10:51:50 -07:00
Jon Siwek
30da2f83d0 GH-566: fix cases where ssh_encrypted_packet event wasn't raised
When encrypted data was bundled within the same segment as the NewKeys
message, it wasn't not reported via a ssh_encrypted_package event as
it should have been.
2019-09-03 17:34:24 -07:00
Jan Grashoefer
81b2b21211 Improve logging of speculative service. 2019-08-30 15:16:37 +02:00
Jan Grashoefer
788b56a652 Add speculative service script.
The speculative service script handles dpd_late_match events to extend
conn.log with infos about potential protocol identifications.
2019-08-29 11:47:04 +02:00
Johanna Amann
bb98559c0d Merge remote-tracking branch 'origin/topic/jsiwek/gh-545-weird-addl'
* origin/topic/jsiwek/gh-545-weird-addl:
  GH-545: add "addl" parameter to flow_weird and net_weird events
2019-08-28 14:27:53 -07:00
Johanna Amann
ec57894a85 Merge remote-tracking branch 'origin/topic/jsiwek/gh-541-ntlm-fix'
* origin/topic/jsiwek/gh-541-ntlm-fix:
  GH-541: add test cases for NTLM AV Pair sequence handling
  GH-541: fix handling of NTLM AV Pair sequences
2019-08-28 11:33:49 -07:00
Jon Siwek
289a1e2e8e Merge branch '555-smb3-negotiate-context-fix' of https://github.com/mad/zeek
- Fixed the context list padding to only be used for dialect 0x0311.
  The new test case includes an example where parsing the optional
  padding would fail for another dialect.

* '555-smb3-negotiate-context-fix' of https://github.com/mad/zeek:
  Fix for smb3 negotiate context
2019-08-27 10:08:42 -07:00
Pavel Ershov
de4a83206d Fix for smb3 negotiate context 2019-08-27 12:21:03 +03:00
Jon Siwek
b954767488 GH-541: add test cases for NTLM AV Pair sequence handling 2019-08-26 10:28:46 -07:00
Tim Wojtulewicz
01207c0720 Fix unit tests for new ordering from NetSessions::Drain 2019-08-21 09:24:02 -04:00
Jon Siwek
2d7c926291 GH-545: add "addl" parameter to flow_weird and net_weird events 2019-08-20 22:45:22 -04:00
Robin Sommer
8ab0650c1e Merge remote-tracking branch 'origin/topic/jsiwek/gh-532-improve-disable-analyzer'
Includes fix for potential iterator invalidation during iteration.

* origin/topic/jsiwek/gh-532-improve-disable-analyzer:
  GH-532: improve disable_analyzer BIF
2019-08-16 18:46:35 +00:00
Jon Siwek
9ccf3549fd Merge remote-tracking branch 'origin/topic/timw/360-log-gzip'
* origin/topic/timw/360-log-gzip:
  Add new LogAscii::gzip_file_extension option.

Adjusted documentation during merge.
2019-08-13 13:49:45 -07:00
Jon Siwek
c91b12863e Add tests for {http,mime}_all_headers events
And remove unnecessary check for mime_all_headers in HTTP entities
(they ony raise the http_all_headers event, never mime_all_headers).
2019-08-13 11:28:00 -07:00
Jon Siwek
18e4976c6c Merge remote-tracking branch 'origin/topic/timw/travis-leaks'
* origin/topic/timw/travis-leaks:
  fixup! Add new distro to Travis CI configuration for running leak tests
  Add new distro to Travis CI configuration for running leak tests
2019-08-12 13:01:55 -07:00
Jon Siwek
a1c5eddb95 GH-532: improve disable_analyzer BIF
- Add an extra "prevent" parameter (default value of false), which
  helps prevent the same analyzer type from being attached in the
  future.  It's useful in situations where you want to disable early
  on, but a DPD signature may still trigger later and re-attach
  the same analyzer.  E.g. when not using this flag, but calling
  disable_analyzer() inside an http_request event, will remove the
  HTTP analyzer that was attached due to well-known-port, but a later
  DPD signature match from upon seeing the HTTP reply will end up
  attaching another HTTP analyzer.  More surprising is that upon
  re-attaching that analyzer, you'll get the same http_request as
  before since the DPD buffer will get replayed into the new analyzer.

- Fixes disable_analyzer() to work when called even earlier, like
  within the protocol_confirmation event.  At that time, the
  Analyzer tree may have not properly added the new analyzer into
  Analyzer::children yet, but rather the temporary waiting list,
  Analyzer::new_children.  Analyzer::RemoveChildAnalyzer previously
  did not inspect the later list.

- Fixes disable_analyzer() when called on an analyzer added to the
  tree via TCP_Analyzer::AddChildPacketAnalyzer.  TCP_Analyzer
  keeps track of such children in its own list,
  TCP_Analyzer::packet_children, which the previous
  Analyzer::RemoveChildAnalyzer implementation didn't inspect.
2019-08-09 20:03:26 -07:00
Tim Wojtulewicz
b9f6992c0e fixup! Add new distro to Travis CI configuration for running leak tests 2019-08-09 11:01:16 -07:00
Jon Siwek
bf9b983f00 Merge branch 'known_services_multiprotocols' of https://github.com/mauropalumbo75/zeek
* 'known_services_multiprotocols' of https://github.com/mauropalumbo75/zeek:
  improve logging with broker store
  drop services starting with -
  remove service from key for Cluster::publish_hrw
  remove check for empty services
  update tests
  order list of services in store key
  remove repeated services in logs if already seen
  add multiprotocol known_services when Known::use_service_store = T
  remove hyphen in front of some services (for example -HTTP, -SSL)   In some cases, there is an hyphen before the protocol name in the field   connection$service. This can cause problems in known_services and   is removed here. It originates probably in some analyzer where it   would be better removed in the future.
  add multiprotocol known_services when Known::use_service_store = F

Changes during merge:
  * whitespace
  * add unit test
2019-08-09 10:47:34 -07:00
Jon Siwek
a68c9f6b71 Merge branch 'empty_services' of https://github.com/mauropalumbo75/zeek
* 'empty_services' of https://github.com/mauropalumbo75/zeek:
  remove empty services and include udp active connections when logging in connection_state_remove
2019-08-09 09:59:50 -07:00
Tim Wojtulewicz
80e3c42526 Add new distro to Travis CI configuration for running leak tests 2019-08-08 16:36:17 -07:00
Jon Siwek
ba56cf62a7 Change version to 3.1.0-dev 2019-08-08 16:08:50 -07:00
Jon Siwek
d236f377e7 Add memory leak test of closure (un)serialization 2019-08-07 12:41:22 -07:00
Jon Siwek
35c42b4b09 GH-474: change MQTT::max_payload_size to be a runtime option 2019-08-05 18:11:54 -07:00
Johanna Amann
0f96a9dedf Disable MQTT by default
To enable MQTT, one has to load policy/scripts/mqtt. Like with smb in
2.5, the consts are loaded by default.
2019-08-05 17:04:39 -07:00
Tim Wojtulewicz
337da50da6 Add new LogAscii::gzip_file_extension option.
This can be used with the LogAscii::gzip_level option to set the file extension of log files when they are compressed at creation time.
2019-08-05 14:36:34 -07:00
Jon Siwek
6bc947a48e GH-474: add MQTT::max_payload_size option
This caps size of payload strings within mqtt_publish events and
mqtt_publish.log files.  A new "payload_len" field in the log file
shows the real payload size in cases where it may have been truncated.
2019-08-02 14:28:55 -07:00
Jon Siwek
c43e809a69 GH-474: use topic vectors for MQTT (un)subscribe events/logs 2019-08-02 13:48:43 -07:00
Johanna Amann
6f25125443 Update CA store to NSS 3.45
This also required updating a test that required a root-certificate that
was removed from the Mozilla store - the test now directly includes that
specific root-cert.
2019-08-02 12:36:54 -07:00
Johanna Amann
6fa0f4ac49 Merge remote-tracking branch 'origin/topic/johanna/conn-duration-thresholds'
* origin/topic/johanna/conn-duration-thresholds:
  Add duration thresholding to the conn-size analyzer.
2019-08-01 14:20:49 -07:00
Johanna Amann
f9ee0079a5 Merge remote-tracking branch 'origin/topic/jsiwek/lambda-name-fixes'
* origin/topic/jsiwek/lambda-name-fixes:
  Guarantee unique internal name for each lambda function
  Use consistent hashing method for internal lambda function names
2019-08-01 13:37:45 -07:00
Johanna Amann
ca36728a4e Merge remote-tracking branch 'origin/topic/jsiwek/gh-514-improve-addr-conversion-errors'
* origin/topic/jsiwek/gh-514-improve-addr-conversion-errors:
  Improve error messages from to_addr and to_subnet BIFs
2019-08-01 13:15:43 -07:00
Jon Siwek
70359c703f Fix a test that used a hardcoded Broker port 2019-08-01 12:28:50 -07:00
Johanna Amann
9d489cde20 Add duration thresholding to the conn-size analyzer.
Now, in addition to setting thresholds for bytes and packet, one can set
a threshold for connection duration. Note that the threshold event is
only raised once the next packet in the connection is seen.

This also fixes a small pre-existing bug, in which a bunch of warnings
were raised if someone just used the lower-level functions without going
through the higher-level scripting API.
2019-08-01 11:57:40 -07:00
Jon Siwek
ec4df80c67 Improve error messages from to_addr and to_subnet BIFs
Related to GH-514
2019-08-01 10:49:03 -07:00
Jon Siwek
ee28e9e9f3 Merge remote-tracking branch 'origin/topic/seth/mqtt'
* origin/topic/seth/mqtt:
  Bug fixes and test baseline updates
  Fix an issue with bro_init -> zeek_init
  MQTT Analyzer heavily updated and ported from the analyzer originally by Supriya Kumar

Adjustments during merge:

* Minor whitespace cleanups
* Some bro to zeek renaming
* Fixed the parsing of unsubscribe messages to generate an event for each topic
2019-07-31 21:29:38 -07:00
Jon Siwek
09ea4ceb7e Rename a broxygen unit test to zeekygen 2019-07-31 14:25:22 -07:00
Jon Siwek
8575c9daed Use consistent hashing method for internal lambda function names
The results of std::hash<std::string> may vary depending on platform.
E.g. test suite failed on macOS due to Linux generating different lambda
function names.
2019-07-31 12:06:27 -07:00
Mauro Palumbo
b4ac0b54fe update tests 2019-07-31 15:48:30 +02:00
Jon Siwek
d1770853b3 Merge remote-tracking branch 'origin/topic/dev/non-ascii-logging'
* origin/topic/dev/non-ascii-logging:
  Removed Policy Script for UTF-8 Logs
  Commented out UTF-8 Script in Test All Policy
  Minor Style Tweak
  Use getNumBytesForUTF8 method to determine number of bytes
  Added Jon's test cases as unit tests
  Prioritizes escaping predefined Escape Sequences over Unescaping UTF-8 Sequences
  Added additional check to confirm anything unescaping is a multibyte UTF-8 sequence, addressing the test case Jon brought up
  Added optional script and redef bool to enable utf-8 in ASCII logs
  Initial Commit, removed std::isprint check to escape

Made minor code format and logic adjustments during merge.
2019-07-30 19:42:34 -07:00
Dev Bali
da5a0e800e Removed Policy Script for UTF-8 Logs 2019-07-30 11:19:06 -07:00
Jon Siwek
db9f81a890 Merge branch 'topic/jsbarber/rexmit-patch' of https://github.com/jsbarber/zeek
* 'topic/jsbarber/rexmit-patch' of https://github.com/jsbarber/zeek:
  Duplicate TCP segment should trigger tcp_multiple_retransmissions
2019-07-29 20:16:48 -07:00
Robin Sommer
13c373086d Merge remote-tracking branch 'origin/topic/zeke/closures' 2019-07-30 02:32:34 +00:00
Seth Hall
7bf856768a Bug fixes and test baseline updates
- ConnAckMsg return_code wasn't passed as a count on accident.
 - Some renaming items were missing due to the age of this code.
2019-07-29 15:12:11 -04:00
Seth Hall
a42b3a1d63 MQTT Analyzer heavily updated and ported from the analyzer originally by Supriya Kumar
This analyzer generates three logs to fully display what is happening over the MQTT connection.
  - mqtt_connect.log
  - mqtt_subscribe.log
  - mqtt_publish.log

At this time it only supports MQTT 3.1 and 3.1.1
2019-07-29 13:45:10 -04:00
Johanna Amann
95ce177682 Merge remote-tracking branch 'origin/topic/jsiwek/gh-488-new-versioning'
* origin/topic/jsiwek/gh-488-new-versioning:
  Add release branches to Travis CI whitelist
  GH-488: teach the Version module to parse new version scheme
2019-07-29 10:02:00 -07:00
Johanna Amann
02f19f7420 Merge remote-tracking branch 'origin/topic/jsiwek/gh-491-vector-of-enum-ctor'
* origin/topic/jsiwek/gh-491-vector-of-enum-ctor:
  Improve type inference for vector-of-enum constructor
2019-07-29 09:30:04 -07:00
Johanna Amann
486bf1e713 Merge remote-tracking branch 'origin/topic/timw/cleaner-utf8'
* origin/topic/timw/cleaner-utf8:
  GHI-486: Switch over to using LLVM utf8-checking code to better validate characters

I addressed a buffer over-read during the merge and added test-cases for
it.
2019-07-29 09:25:25 -07:00