Commit graph

2462 commits

Author SHA1 Message Date
Frerich Raabe
bb879fc2aa Fixed decompose_uri choking on URIs with empty port component
A call such as

  decompose_uri("git://git.kernel.org:/pub/scm/linux/");

would raise an error along the lines of

  error in /usr/local/zeek-3.0.0/share/zeek/base/utils/urls.zeek, line 122: bad conversion to count (to_count(parts[1]) and )

This was because an empty string got passsed to the to_count()
function.

Let's improve the behaviour and rather consider the portnum component
of the URI to be uninitialized.
2020-01-24 22:24:00 +01:00
Jon Siwek
7a748526c0 Use consistent zeek_init priority for Log::create_stream calls
Typically in base scripts, Log::create_stream() is called in zeek_init()
handler with &priority=5 such that it will have already been created
in the default zeek_init() &priority=0.
2020-01-22 13:58:20 -08:00
Seth Hall
d9ed76c90a
Fix warning when reading files from non-network sources
If files are being read from non-network sources, there was a warning in the SSL base scripts about missing the f$conns field.
2020-01-14 10:53:02 -05:00
Justin Azoff
80469a1fde fix NTLM field value access
The fields being checked for existence were not the same as the fields
being accessed.
2020-01-08 11:40:28 -05:00
Jon Siwek
5658a529f0 Update URL for Malware Hash Registry website 2019-12-16 08:22:46 -08:00
Tim Wojtulewicz
5c85b083b6 Merge remote-tracking branch 'origin/topic/jsiwek/gh-700-packet-sources-bif'
* origin/topic/jsiwek/gh-700-packet-sources-bif:
  GH-700: add packet_sources() BIF
2019-12-05 11:58:06 -07:00
Jon Siwek
30d0b21ecc Merge remote-tracking branch 'origin/topic/dev/print-to-log'
Adjustments during merge:

- kept the UNKNOWN Log::ID as placeholder value
- changed the coverage.find-bro-logs test to check for arbitrary $path
  field values instead of just string literals
- don't force EnumVal to unsigned integer since the relevant union member
  is the signed integer and added the relevant enum values/types to
  .bif files for easier access
- compare FILE* versus file name to check for stdout equality (don't
  think it matters much, just a bit more efficient)
- minor whitespace/style tweaks

* origin/topic/dev/print-to-log:
  Added a non boolean configuration and other changes as suggested by Jon
  Allow Print Statements to be redirected to a Log# This is a combination of 3 commits.
2019-12-02 13:47:09 -08:00
Jon Siwek
6e7b101df2 GH-700: add packet_sources() BIF
Provides access to properties of packet sources, like interface or pcap
file name.
2019-11-28 09:44:38 -08:00
Dev Bali
4be6871df2 Added a non boolean configuration and other changes as suggested by Jon 2019-11-26 21:53:21 -08:00
Robin Sommer
a00f139f1c Merge remote-tracking branch 'origin/topic/jsiwek/gh-646-conn-removal'
Clarified doc strings a bit.

* origin/topic/jsiwek/gh-646-conn-removal:
  GH-646: add new "successful_connection_remove" event
2019-11-18 12:08:12 +00:00
Jon Siwek
2395a73f26 Install test-all-policy.zeek script
The zeekygen script, which is already installed, refers to it, and
could also generally be useful for "test everything" type scenarios.
2019-11-14 17:00:29 -08:00
Dev Bali
c9016f1397 Allow Print Statements to be redirected to a Log# This is a combination of 3 commits. 2019-11-13 19:59:42 -08:00
Jon Siwek
31f60853c9 GH-646: add new "successful_connection_remove" event
And switch Zeek's base scripts over to using it in place of
"connection_state_remove".  The difference between the two is
that "connection_state_remove" is raised for all events while
"successful_connection_remove" excludes TCP connections that were never
established (just SYN packets).  There can be performance benefits
to this change for some use-cases.

There's also a new event called ``connection_successful`` and a new
``connection`` record field named "successful" to help indicate this new
property of connections.
2019-11-11 19:52:59 -08:00
Jon Siwek
872adda5b1 Merge branch 'topic/jsbarber/ftp-cluster-fix-patch' of https://github.com/jsbarber/zeek
Minor cleanup in merge: remove print statements and unnecessary @if
directive.

* 'topic/jsbarber/ftp-cluster-fix-patch' of https://github.com/jsbarber/zeek:
  Publish ftp_data_expected updates to other workers for synchronization
2019-11-04 17:31:59 -08:00
Jeff Barber
d698bddc7d Publish ftp_data_expected updates to other workers for synchronization 2019-10-30 15:50:22 -06:00
Robin Sommer
9d7c9f37d4 Merge branch 'Fix_http_build_url' of ssh://github.com/Olerdrive/zeek
I changed the format string of the port to %d and added a test.
2019-10-28 09:32:32 +00:00
Alexey Mokeev
6ba1d1dcd3 Make http::build_url work correctly 2019-10-27 11:17:53 +03:00
Jon Siwek
81ab0b0d05 Use explicit path name for NTP log stream
For consistency (we do this for all other logs) and just to avoid
the extra path function calls.
2019-10-25 10:38:58 -07:00
Jon Siwek
4c066ca914 Remove unnecessary &optional from a Config::set_value() parameter 2019-10-15 17:46:05 -07:00
Jon Siwek
052feacbda GH-618: add "tcp_options" event containing TCP option values 2019-10-03 18:59:02 -07:00
Jon Siwek
aeef4bf030 Merge branch 'topic/jgras/dpd-late-match' of https://github.com/J-Gras/zeek
* 'topic/jgras/dpd-late-match' of https://github.com/J-Gras/zeek:
  Improve dpd_late_match event generation.
  Improve logging of speculative service.
  Update test-all-policy script.
  Add speculative service script.
  Allow to handle late DPD matches.
2019-09-17 11:17:41 -07:00
Jon Siwek
9c8db5f6ca Stop loading scan detection in local.zeek by default 2019-09-16 10:51:50 -07:00
Jan Grashoefer
81b2b21211 Improve logging of speculative service. 2019-08-30 15:16:37 +02:00
Jan Grashoefer
a810365f0e Update test-all-policy script. 2019-08-30 11:30:33 +02:00
Jan Grashoefer
788b56a652 Add speculative service script.
The speculative service script handles dpd_late_match events to extend
conn.log with infos about potential protocol identifications.
2019-08-29 11:47:04 +02:00
Jan Grashoefer
5901b6d33c Allow to handle late DPD matches.
If dpd_match_only_beginning is disabled, matches of protocol signatures
can be handeld using protocol_late_match. To prevent further matching in
this case, dpd_late_match_stop may be activated.
2019-08-22 19:00:11 +02:00
Jon Siwek
2d7c926291 GH-545: add "addl" parameter to flow_weird and net_weird events 2019-08-20 22:45:22 -04:00
Jon Siwek
9ccf3549fd Merge remote-tracking branch 'origin/topic/timw/360-log-gzip'
* origin/topic/timw/360-log-gzip:
  Add new LogAscii::gzip_file_extension option.

Adjusted documentation during merge.
2019-08-13 13:49:45 -07:00
Jon Siwek
31d30bb47e GH-535: fix typo of "C_HESIOD" in DNS::classes
Fixes GH-535
2019-08-12 13:32:28 -07:00
Jon Siwek
bf9b983f00 Merge branch 'known_services_multiprotocols' of https://github.com/mauropalumbo75/zeek
* 'known_services_multiprotocols' of https://github.com/mauropalumbo75/zeek:
  improve logging with broker store
  drop services starting with -
  remove service from key for Cluster::publish_hrw
  remove check for empty services
  update tests
  order list of services in store key
  remove repeated services in logs if already seen
  add multiprotocol known_services when Known::use_service_store = T
  remove hyphen in front of some services (for example -HTTP, -SSL)   In some cases, there is an hyphen before the protocol name in the field   connection$service. This can cause problems in known_services and   is removed here. It originates probably in some analyzer where it   would be better removed in the future.
  add multiprotocol known_services when Known::use_service_store = F

Changes during merge:
  * whitespace
  * add unit test
2019-08-09 10:47:34 -07:00
Jon Siwek
a68c9f6b71 Merge branch 'empty_services' of https://github.com/mauropalumbo75/zeek
* 'empty_services' of https://github.com/mauropalumbo75/zeek:
  remove empty services and include udp active connections when logging in connection_state_remove
2019-08-09 09:59:50 -07:00
Jon Siwek
acd1be6e7b Merge branch 'export_intel_events' of https://github.com/mauropalumbo75/zeek
* 'export_intel_events' of https://github.com/mauropalumbo75/zeek:
  minor restyle and add comments
  add an empty read_error event to the intel framework (in the export block, so that users can implement further checks with it)
  move event Intel::read_entry to export block

Adjusted whitespace in merge.
2019-08-09 09:36:19 -07:00
Mauro Palumbo
f93e41fd3f minor restyle and add comments 2019-08-07 10:32:36 +02:00
Mauro Palumbo
8e418d3c7b add an empty read_error event to the intel framework (in the export block, so that users can implement further checks with it) 2019-08-06 12:20:47 +02:00
Mauro Palumbo
1011abd5e0 move event Intel::read_entry to export block 2019-08-06 11:17:34 +02:00
Johanna Amann
3080290a5e Merge remote-tracking branch 'origin/topic/jsiwek/gh-474-mqtt-option'
* origin/topic/jsiwek/gh-474-mqtt-option:
  GH-474: change MQTT::max_payload_size to be a runtime option

Relates to GH-474
2019-08-05 21:06:50 -07:00
Jon Siwek
1eb1771c25 Fix documentation warnings for MQTT identifiers
I.e. a type was not in the export section, but a field was added
to connection record via a redef that uses the "hidden" type.
That generally doesn't help to hide it that way since a user comes
to rely on it indirectly anyway, and it also causes problems with
the Zeekygen documentation not being able to find it.
2019-08-05 18:55:48 -07:00
Jon Siwek
35c42b4b09 GH-474: change MQTT::max_payload_size to be a runtime option 2019-08-05 18:11:54 -07:00
Johanna Amann
0f96a9dedf Disable MQTT by default
To enable MQTT, one has to load policy/scripts/mqtt. Like with smb in
2.5, the consts are loaded by default.
2019-08-05 17:04:39 -07:00
Tim Wojtulewicz
337da50da6 Add new LogAscii::gzip_file_extension option.
This can be used with the LogAscii::gzip_level option to set the file extension of log files when they are compressed at creation time.
2019-08-05 14:36:34 -07:00
Jon Siwek
6bc947a48e GH-474: add MQTT::max_payload_size option
This caps size of payload strings within mqtt_publish events and
mqtt_publish.log files.  A new "payload_len" field in the log file
shows the real payload size in cases where it may have been truncated.
2019-08-02 14:28:55 -07:00
Jon Siwek
c43e809a69 GH-474: use topic vectors for MQTT (un)subscribe events/logs 2019-08-02 13:48:43 -07:00
Johanna Amann
649d9f502b Update Certificate Transparency list 2019-08-02 12:43:43 -07:00
Johanna Amann
6f25125443 Update CA store to NSS 3.45
This also required updating a test that required a root-certificate that
was removed from the Mozilla store - the test now directly includes that
specific root-cert.
2019-08-02 12:36:54 -07:00
Jon Siwek
d2eed166bd GH-517: fix MQTT suback/unsuback accessing non-existent index 2019-08-02 11:57:46 -07:00
Johanna Amann
6fa0f4ac49 Merge remote-tracking branch 'origin/topic/johanna/conn-duration-thresholds'
* origin/topic/johanna/conn-duration-thresholds:
  Add duration thresholding to the conn-size analyzer.
2019-08-01 14:20:49 -07:00
Johanna Amann
9d489cde20 Add duration thresholding to the conn-size analyzer.
Now, in addition to setting thresholds for bytes and packet, one can set
a threshold for connection duration. Note that the threshold event is
only raised once the next packet in the connection is seen.

This also fixes a small pre-existing bug, in which a bunch of warnings
were raised if someone just used the lower-level functions without going
through the higher-level scripting API.
2019-08-01 11:57:40 -07:00
Jon Siwek
ee28e9e9f3 Merge remote-tracking branch 'origin/topic/seth/mqtt'
* origin/topic/seth/mqtt:
  Bug fixes and test baseline updates
  Fix an issue with bro_init -> zeek_init
  MQTT Analyzer heavily updated and ported from the analyzer originally by Supriya Kumar

Adjustments during merge:

* Minor whitespace cleanups
* Some bro to zeek renaming
* Fixed the parsing of unsubscribe messages to generate an event for each topic
2019-07-31 21:29:38 -07:00
Mauro Palumbo
e206347d1a improve logging with broker store 2019-07-31 17:40:02 +02:00
Seth Hall
7626344122 Tiny tweaks to try and address ticket #506 2019-07-31 11:17:53 -04:00