Commit graph

3725 commits

Author SHA1 Message Date
Jon Siwek
0ff99c3df8 Separate supervisor node config and status data structures 2020-01-13 20:09:05 -08:00
Jon Siwek
520c6e3ebf Merge branch 'master' into topic/jsiwek/supervisor 2020-01-13 10:27:34 -08:00
Jon Siwek
5191e14eff Add stdout/stderr redirection option to supervised node config 2020-01-10 19:48:31 -08:00
Jon Siwek
263a5f404a Add cpu affinity option to supervised node config 2020-01-10 18:25:42 -08:00
Jon Siwek
00cd04b0ae Extend Supervisor Node config with list of custom scripts 2020-01-08 14:52:46 -08:00
Jon Siwek
297317b232 Organize command-line options for Supervisor filtering/inheritance
Also have stem process execv() with original command-line arguments so
that they're re-parsed and inherited correctly by supervised-nodes in
the event the stem process needs to be re-created.
2020-01-08 14:05:19 -08:00
Justin Azoff
80469a1fde fix NTLM field value access
The fields being checked for existence were not the same as the fields
being accessed.
2020-01-08 11:40:28 -05:00
Jon Siwek
6046da9993 Merge branch 'master' into topic/jsiwek/supervisor 2020-01-07 16:57:58 -08:00
Jon Siwek
b114766205 Add Supervisor::is_supervised()
And use to to avoid redef'ing the log rotation postprocessor to
"archive-log" by default since it's unlikely PATH is configured to find
that script.
2020-01-07 10:27:07 -08:00
Jon Siwek
aaa702fb4d Add option to change supervised node's working directory 2020-01-06 18:39:14 -08:00
Jon Siwek
5658a529f0 Update URL for Malware Hash Registry website 2019-12-16 08:22:46 -08:00
Tim Wojtulewicz
5c85b083b6 Merge remote-tracking branch 'origin/topic/jsiwek/gh-700-packet-sources-bif'
* origin/topic/jsiwek/gh-700-packet-sources-bif:
  GH-700: add packet_sources() BIF
2019-12-05 11:58:06 -07:00
Jon Siwek
30d0b21ecc Merge remote-tracking branch 'origin/topic/dev/print-to-log'
Adjustments during merge:

- kept the UNKNOWN Log::ID as placeholder value
- changed the coverage.find-bro-logs test to check for arbitrary $path
  field values instead of just string literals
- don't force EnumVal to unsigned integer since the relevant union member
  is the signed integer and added the relevant enum values/types to
  .bif files for easier access
- compare FILE* versus file name to check for stdout equality (don't
  think it matters much, just a bit more efficient)
- minor whitespace/style tweaks

* origin/topic/dev/print-to-log:
  Added a non boolean configuration and other changes as suggested by Jon
  Allow Print Statements to be redirected to a Log# This is a combination of 3 commits.
2019-12-02 13:47:09 -08:00
Jon Siwek
6e7b101df2 GH-700: add packet_sources() BIF
Provides access to properties of packet sources, like interface or pcap
file name.
2019-11-28 09:44:38 -08:00
Dev Bali
4be6871df2 Added a non boolean configuration and other changes as suggested by Jon 2019-11-26 21:53:21 -08:00
Robin Sommer
a00f139f1c Merge remote-tracking branch 'origin/topic/jsiwek/gh-646-conn-removal'
Clarified doc strings a bit.

* origin/topic/jsiwek/gh-646-conn-removal:
  GH-646: add new "successful_connection_remove" event
2019-11-18 12:08:12 +00:00
Jon Siwek
2395a73f26 Install test-all-policy.zeek script
The zeekygen script, which is already installed, refers to it, and
could also generally be useful for "test everything" type scenarios.
2019-11-14 17:00:29 -08:00
Dev Bali
c9016f1397 Allow Print Statements to be redirected to a Log# This is a combination of 3 commits. 2019-11-13 19:59:42 -08:00
Jon Siwek
31f60853c9 GH-646: add new "successful_connection_remove" event
And switch Zeek's base scripts over to using it in place of
"connection_state_remove".  The difference between the two is
that "connection_state_remove" is raised for all events while
"successful_connection_remove" excludes TCP connections that were never
established (just SYN packets).  There can be performance benefits
to this change for some use-cases.

There's also a new event called ``connection_successful`` and a new
``connection`` record field named "successful" to help indicate this new
property of connections.
2019-11-11 19:52:59 -08:00
Jon Siwek
cc37e505e4 Merge remote-tracking branch 'origin/master' into topic/jsiwek/supervisor 2019-11-05 10:11:47 -08:00
Jon Siwek
872adda5b1 Merge branch 'topic/jsbarber/ftp-cluster-fix-patch' of https://github.com/jsbarber/zeek
Minor cleanup in merge: remove print statements and unnecessary @if
directive.

* 'topic/jsbarber/ftp-cluster-fix-patch' of https://github.com/jsbarber/zeek:
  Publish ftp_data_expected updates to other workers for synchronization
2019-11-04 17:31:59 -08:00
Jeff Barber
d698bddc7d Publish ftp_data_expected updates to other workers for synchronization 2019-10-30 15:50:22 -06:00
Robin Sommer
9d7c9f37d4 Merge branch 'Fix_http_build_url' of ssh://github.com/Olerdrive/zeek
I changed the format string of the port to %d and added a test.
2019-10-28 09:32:32 +00:00
Alexey Mokeev
6ba1d1dcd3 Make http::build_url work correctly 2019-10-27 11:17:53 +03:00
Jon Siwek
81ab0b0d05 Use explicit path name for NTP log stream
For consistency (we do this for all other logs) and just to avoid
the extra path function calls.
2019-10-25 10:38:58 -07:00
Jon Siwek
29f386e388 Implement minimal supervised cluster configuration
More aspects of the cluster configuration to get fleshed out later,
but a basic cluster like one would use for a live deployment
can now be instantiated and run under supervision.  The new
clusterized-pcap-processing supervisor mode is also not done yet.
2019-10-23 17:37:53 -07:00
Jon Siwek
25a8ba99fa Change supervisor event response topic names
They now include the request ID at the end of their topic, to possibly
help with subscription filtering.
2019-10-21 09:16:26 -07:00
Jon Siwek
7a6355f64f Change supervisor event API to use strings for request IDs 2019-10-21 09:12:50 -07:00
Jon Siwek
773b39e52e Finish implementing supervisor infrastructure
The process hierarchy and all supervisor control commands are now
working (e.g. status, create, destroy, restart), but nodes are
not currently spawned with the desired configuration parameters so
they don't yet operate as real cluster nodes (e.g. worker, logger,
manager, proxy).
2019-10-18 17:57:20 -07:00
Jon Siwek
7c08488dfc Add skeleton logic for handling supevisor control messages 2019-10-16 15:13:15 -07:00
Jon Siwek
4c066ca914 Remove unnecessary &optional from a Config::set_value() parameter 2019-10-15 17:46:05 -07:00
Jon Siwek
e46cf88435 Add Supervisor BIF/event API skeleton 2019-10-15 12:59:45 -07:00
Jon Siwek
052feacbda GH-618: add "tcp_options" event containing TCP option values 2019-10-03 18:59:02 -07:00
Jon Siwek
aeef4bf030 Merge branch 'topic/jgras/dpd-late-match' of https://github.com/J-Gras/zeek
* 'topic/jgras/dpd-late-match' of https://github.com/J-Gras/zeek:
  Improve dpd_late_match event generation.
  Improve logging of speculative service.
  Update test-all-policy script.
  Add speculative service script.
  Allow to handle late DPD matches.
2019-09-17 11:17:41 -07:00
Jon Siwek
9c8db5f6ca Stop loading scan detection in local.zeek by default 2019-09-16 10:51:50 -07:00
Jan Grashoefer
81b2b21211 Improve logging of speculative service. 2019-08-30 15:16:37 +02:00
Jan Grashoefer
a810365f0e Update test-all-policy script. 2019-08-30 11:30:33 +02:00
Jan Grashoefer
788b56a652 Add speculative service script.
The speculative service script handles dpd_late_match events to extend
conn.log with infos about potential protocol identifications.
2019-08-29 11:47:04 +02:00
Jan Grashoefer
5901b6d33c Allow to handle late DPD matches.
If dpd_match_only_beginning is disabled, matches of protocol signatures
can be handeld using protocol_late_match. To prevent further matching in
this case, dpd_late_match_stop may be activated.
2019-08-22 19:00:11 +02:00
Jon Siwek
2d7c926291 GH-545: add "addl" parameter to flow_weird and net_weird events 2019-08-20 22:45:22 -04:00
Jon Siwek
9ccf3549fd Merge remote-tracking branch 'origin/topic/timw/360-log-gzip'
* origin/topic/timw/360-log-gzip:
  Add new LogAscii::gzip_file_extension option.

Adjusted documentation during merge.
2019-08-13 13:49:45 -07:00
Jon Siwek
31d30bb47e GH-535: fix typo of "C_HESIOD" in DNS::classes
Fixes GH-535
2019-08-12 13:32:28 -07:00
Jon Siwek
bf9b983f00 Merge branch 'known_services_multiprotocols' of https://github.com/mauropalumbo75/zeek
* 'known_services_multiprotocols' of https://github.com/mauropalumbo75/zeek:
  improve logging with broker store
  drop services starting with -
  remove service from key for Cluster::publish_hrw
  remove check for empty services
  update tests
  order list of services in store key
  remove repeated services in logs if already seen
  add multiprotocol known_services when Known::use_service_store = T
  remove hyphen in front of some services (for example -HTTP, -SSL)   In some cases, there is an hyphen before the protocol name in the field   connection$service. This can cause problems in known_services and   is removed here. It originates probably in some analyzer where it   would be better removed in the future.
  add multiprotocol known_services when Known::use_service_store = F

Changes during merge:
  * whitespace
  * add unit test
2019-08-09 10:47:34 -07:00
Jon Siwek
a68c9f6b71 Merge branch 'empty_services' of https://github.com/mauropalumbo75/zeek
* 'empty_services' of https://github.com/mauropalumbo75/zeek:
  remove empty services and include udp active connections when logging in connection_state_remove
2019-08-09 09:59:50 -07:00
Jon Siwek
acd1be6e7b Merge branch 'export_intel_events' of https://github.com/mauropalumbo75/zeek
* 'export_intel_events' of https://github.com/mauropalumbo75/zeek:
  minor restyle and add comments
  add an empty read_error event to the intel framework (in the export block, so that users can implement further checks with it)
  move event Intel::read_entry to export block

Adjusted whitespace in merge.
2019-08-09 09:36:19 -07:00
Mauro Palumbo
f93e41fd3f minor restyle and add comments 2019-08-07 10:32:36 +02:00
Mauro Palumbo
8e418d3c7b add an empty read_error event to the intel framework (in the export block, so that users can implement further checks with it) 2019-08-06 12:20:47 +02:00
Mauro Palumbo
1011abd5e0 move event Intel::read_entry to export block 2019-08-06 11:17:34 +02:00
Johanna Amann
3080290a5e Merge remote-tracking branch 'origin/topic/jsiwek/gh-474-mqtt-option'
* origin/topic/jsiwek/gh-474-mqtt-option:
  GH-474: change MQTT::max_payload_size to be a runtime option

Relates to GH-474
2019-08-05 21:06:50 -07:00
Jon Siwek
1eb1771c25 Fix documentation warnings for MQTT identifiers
I.e. a type was not in the export section, but a field was added
to connection record via a redef that uses the "hidden" type.
That generally doesn't help to hide it that way since a user comes
to rely on it indirectly anyway, and it also causes problems with
the Zeekygen documentation not being able to find it.
2019-08-05 18:55:48 -07:00