Commit graph

1002 commits

Author SHA1 Message Date
Tim Wojtulewicz
ad19f1e1bb GHI-486: Switch over to using LLVM utf8-checking code to better validate characters 2019-07-24 10:58:00 -07:00
Dev Bali
6fcb23066d Added Jon's test cases as unit tests 2019-07-23 11:59:33 -07:00
Dev Bali
66557d3178 Added optional script and redef bool to enable utf-8 in ASCII logs 2019-07-23 11:59:33 -07:00
Jon Siwek
07e0dba3c6 Add DPD::max_violations option
This allows one to tune the number of protocol violations to tolerate
from any given analyzer type before just disabling a given instance
of it.

Also removes the "disabled_aids" field from the DPD::Info record
since it serves no purpose: in this case, calling disable_analyzer
multiple times for the same analyzer is a no-op.
2019-07-19 13:00:02 -07:00
Johanna Amann
1f329ad541 Merge remote-tracking branch 'origin/topic/timw/150-to-json'
* origin/topic/timw/150-to-json:
  Update submodules for JSON work
  Update unit tests for JSON logger to match new output
  Modify JSON log writer to use the external JSON library
  Update unit test output to match json.zeek being deprecated and slight format changes to JSON output
  Add proper JSON serialization via C++, deprecate json.zeek
  Add new method for escaping UTF8 strings for JSON output
  Move do_sub method from zeek.bif to StringVal class method
  Move record_fields method from zeek.bif to Val class method
  Add ToStdString method for StringVal
2019-07-11 11:17:32 -07:00
Jon Siwek
002109973d Improve stability of a unit test 2019-07-03 13:04:57 -07:00
Tim Wojtulewicz
d732fb4d4b Update unit tests for JSON logger to match new output 2019-07-02 13:14:52 -07:00
Tim Wojtulewicz
d27c846ec5 Update unit test output to match json.zeek being deprecated and slight format changes to JSON output 2019-07-02 13:14:52 -07:00
Johanna Amann
1ebd3adf20 Merge remote-tracking branch 'origin/topic/jsiwek/gh-243-wrap-up-deprecation-removal'
* origin/topic/jsiwek/gh-243-wrap-up-deprecation-removal:
  Improve deprecation warning messages
  Remove deprecated DNS events
  Remove BackDoor analyzer
  Remove InterConn analyzer
  Remove deprecated/unused irc_servers option
  Remove deprecated print_hook event
  Remove dead code: dump_used_event_handlers
  Remove unused software_version_found events
  Remove deprecated open_log_file and log_file_name functions
  Remove deprecated/unused "packet" type
  Un-deprecate anonymizer BIFs
  Un-deprecate file rotation functions
2019-07-01 01:14:29 -07:00
Seth Hall
9795782ecb
Merge pull request #324 from zeek/topic/jsiwek/gh-320
Improve RFB (VNC) protocol parsing
2019-06-28 17:27:16 -04:00
Seth Hall
e3b080c741
Fixed a small issue due to the name changes 2019-06-28 14:49:57 -04:00
Jon Siwek
bfd037989b Remove deprecated open_log_file and log_file_name functions 2019-06-27 17:43:20 -07:00
Seth Hall
eb690a18cc
Merge branch 'master' into topic/jsiwek/gh-320 2019-06-26 14:47:01 -04:00
Johanna Amann
3ec9fb0f7f Change notices to be processed on worker.
In the past they were processed on the manager - which requires big
records to be sent around.

This has a potential of incompatibilities if someone relied on global
state for notice processing.

GH-214
2019-06-25 13:51:27 -07:00
Jon Siwek
d72f5458f1 Merge branch 'rdp_client_cluster_data' of https://github.com/neslog/zeek
* 'rdp_client_cluster_data' of https://github.com/neslog/zeek:
  Fixing types.
  Correcting types.
  Removing misc data from Client Cluster data trying to assign values.
  Adding options field to RDP::ClientChannelDef Adding Client Cluster Data

Adjustments:

- Reformatting
- Added comments
- Changed the REDIRECTED_SESSIONID_FIELD_VALID field to a bool
2019-06-20 20:41:16 -07:00
jatkinosn
3a19af86c5 Fixing types.
Added handling for fields sub fields.

Added test script and output.
2019-06-20 10:47:05 -04:00
Johanna Amann
446b5cb90e Remove opaque of ocsp_resp.
Only used in one event, without any way to use the opaque for anything
else. At this point this just seems like a complication that has no
reason to be there.
2019-06-18 11:09:16 -07:00
Jon Siwek
e2dc0092f3 Merge branch 'ntp-rewrite' of https://github.com/mauropalumbo75/zeek
* 'ntp-rewrite' of https://github.com/mauropalumbo75/zeek: (25 commits)
  update tests baseline
  Apply requested changes: - file dpd.sig and TODO comments for signature protocol detection removed - missing doc field filled in events.bif - rename OpCode and ReqCode fields into op_code and req_code respectively - removed unnecessary child method in NTP.h/.cc - main.zeek and ntp-protocol.pac reformatted
  minor changes in the documentation
  fix some initializations
  fix wrong assignment of control key_id/crypto_checksum
  code clean up
  add extension fields parsing
  add extended mac field with 20 byte digest (+4 byte key id)
  update tests and add a new one for key_id and mac
  fix auth field (key_id and mac) in standard and control msg
  remove old NTP record in init-bare.zeek
  fix key_id and digest (WIP)
  fix wrong Assign with reference_id
  add tests for ntp protocol (finished)
  add tests for ntp protocol (WIP)
  fix problem with time vals
  add ntp records to init-bare.zeek
  update ntp analyzer to val_mgr
  extend and refact script-side of NTP analyzer
  extend and refactor several fields
  ...
2019-06-15 19:11:34 -07:00
Jon Siwek
852506e26a Merge remote-tracking branch 'origin/topic/vlad/dns_spf'
* origin/topic/vlad/dns_spf:
  DNS: Add support for SPF response records
2019-06-14 20:58:54 -07:00
Jon Siwek
56bb28a636 Merge remote-tracking branch 'origin/topic/jsiwek/gh-387-broker-topic-names'
* origin/topic/jsiwek/gh-387-broker-topic-names:
  GH-387: update Broker topic names to use "zeek/" prefix
2019-06-14 19:30:51 -07:00
Vlad Grigorescu
5f0023b3b0 DNS: Add support for SPF response records
SPF response records are identical to TXT records in structure, and
can be parsed and interpreted the same way. However, they have a
different RR type, so they would generate weird events and not be
parsed by Zeek before this change.

Even though they're the same as TXT records from a protocol stance, I
created a new event type (dns_SPF_reply), and call the records out as
SPF in the logs, instead of as TXT records, since the distinction
could be important for detection purposes.

SPF records have been obsoleted, but continue to be seen in the wild.
2019-06-14 10:18:37 -05:00
Jon Siwek
5331bf10ec GH-323: change builtin plugin namespaces to Zeek 2019-06-07 20:56:54 -07:00
Johanna Amann
6707328c55 Merge remote-tracking branch 'origin/master' into topic/johanna/tls13-details 2019-06-07 16:52:38 +10:00
Jon Siwek
7c5e74e577 Merge branch 'master' of https://github.com/anthonykasza/zeek
* 'master' of https://github.com/anthonykasza/zeek:
  add: rdp_native_encrytped_data event

I added a unit test.
2019-06-06 17:22:49 -07:00
Jon Siwek
c193582623 Merge branch 'rdp_client_security' of https://github.com/neslog/zeek
* 'rdp_client_security' of https://github.com/neslog/zeek:
  Adding comments specific to client security data in record definition.
  Cleaning up indentations and return true.
  Adding record to init-bare
  Adding client_security_data to the analyzer.

I added a unit test.
2019-06-06 16:53:48 -07:00
Jon Siwek
eef669f048 Improve sqlite logging unit tests
By using a consistent timestamp. That avoids rare chances of sqlite
output from rounding the current time into such a form that happens
to bypass the timestamp canonifier script (whenever it happened to
land on a whole or tenth second).
2019-06-06 12:06:55 -07:00
Mauro Palumbo
38ad648082 update tests and add a new one for key_id and mac 2019-06-06 16:45:09 +02:00
Jon Siwek
dfed213f31 Deprecate functions with "bro" in them.
* "bro_is_terminating" is now "zeek_is_terminating"

* "bro_version" is now "zeek_version"

The old function names still exist for now, but are deprecated.
2019-06-05 16:18:57 -07:00
Jon Siwek
b5050437fa GH-379: move catch-and-release and unified2 scripts to policy/
These are no longer loaded by default due to the performance impact they
cause simply by being loaded (they have event handlers for commonly
generated events) and they aren't generally useful enough to justify it.
2019-06-05 13:33:45 -07:00
Mauro Palumbo
48cda6a81d add tests for ntp protocol (finished) 2019-06-05 11:17:40 +02:00
Jon Siwek
59596e0bfa Merge branch 'master' of https://github.com/ZekeMedley/zeek
* 'master' of https://github.com/ZekeMedley/zeek:
  Use the right delete and improve the leak test. Increases the size of the table being loaded in the pattern leak test and uses the right delete method.
  Fix formatting.
  Fix memory leak and add test.
  Add pattern support to input framework.
2019-06-04 19:19:19 -07:00
Mauro Palumbo
50f265353b add tests for ntp protocol (WIP) 2019-06-04 17:59:18 +02:00
Johanna Amann
86ac468882 support the newer TLS 1.3 key_share extension.
This one adds a separate new case that has to be parsed differently - if
a hello-retry-request is sent, only the namedgroup is sent - without the
additional key material.

Support for the legacy extension is retained.
2019-06-03 14:40:33 +10:00
Johanna Amann
af59ed6bdb Merge remote-tracking branch 'origin/master' into topic/johanna/tls13-details 2019-06-03 12:19:12 +10:00
Jon Siwek
1ce0fcce49 GH-387: update Broker topic names to use "zeek/" prefix 2019-05-29 15:56:37 -07:00
ZekeMedley
cfeb6f0f0d Add pattern support to input framework. 2019-05-29 12:19:40 -07:00
Jon Siwek
748a4f2ea2 Merge branch '001-dhcp-options' of https://github.com/jrwren/zeek
* '001-dhcp-options' of https://github.com/jrwren/zeek:
  add some dhcp options
2019-05-24 18:08:34 -07:00
Jay Wren
1eda26d16f add some dhcp options 2019-05-24 16:52:12 -04:00
Jon Siwek
d886f40728 GH-378: check validity of missing 'val' field in Input::add_table
It's only allowed to be missing when the 'destination' field is a
"set" type, but not for a "table" type.

Fixes GH-378
2019-05-24 10:30:49 -07:00
Jon Siwek
580822a32c Merge remote-tracking branch 'origin/topic/dnthayer/gh-339'
* origin/topic/dnthayer/gh-339:
  Rename all BRO-prefixed environment variables
2019-05-23 20:23:14 -07:00
Johanna Amann
74bb7716f6 Finish implementation of copy method.
All types (besides EntropyVal) now support a native copy operation,
which uses primitives of the underlying datatypes to perform a quick
copy, without serialization.

EntropyVal is the one exception - since that type is rather complex
(many members) and will probably not be copied a lot, if at all, it
makes sense to just use the serialization function.

This will have to be slightly re-written in the near-term-future to use
the new serialization function for that opaque type.

This change also introduces a new x509_from_der bif, which allows to
parse a der into an opaque of x509.

This change removes the d2i_X509_ wrapper function; this was a remnant
when d2i_X509 took non-const arguments. We directly use d2i_X509 at
several places assuming const-ness, so there does not seem to ba a
reason to keep the wrapper.

This change also exposed a problem in the File cache - cases in which an
object was brought back into the cache, and writing occurred in the
file_open event were never correctly handeled as far as I can tell.
2019-05-22 14:29:37 -07:00
Daniel Thayer
1a74516db1 Rename all BRO-prefixed environment variables
For backward compatibility when reading values, we first check
the ZEEK-prefixed value, and if not set, then check the corresponding
BRO-prefixed value.
2019-05-22 00:12:31 -05:00
Jon Siwek
dd173f4961 Always emit scripting errors to stderr during zeek_init
Otherwise, setting Reporter::errors_to_stderr=F causes important
error messages to be lost (and this setting is the default for
ZeekCtl).  E.g. now that we terminate if there's errors during
zeek_init, GH-369 shows that the only error message given was
"fatal error: errors occurred while initializing", which is not
helpful in determining the actual issue.
2019-05-17 14:18:45 -07:00
Daniel Thayer
3f9e7138bd More bro-to-zeek renaming in the unit tests 2019-05-16 02:27:54 -05:00
Jon Siwek
a8c0cd7dee Fix potential race in openflow broker plugin
Broker::subscribe() after Broker::peer() may result in losing messages,
always best to do the reverse order.

Also possibly improved chance of unstable unit test output order.
2019-05-15 10:12:32 -07:00
Jon Siwek
6ad7099f7e Merge remote-tracking branch 'origin/topic/robin/gh-239'
* origin/topic/robin/gh-239:
  Undo a change to btest.cfg from a recent commit
  Updating submodule.
  Fix zeek-wrapper
  Update for renaming BroControl to ZeekControl.
  Updating submodule.
  GH-239: Rename bro to zeek, bro-config to zeek-config, and bro-path-dev to zeek-path-dev.
2019-05-14 13:27:40 -07:00
Jon Siwek
eda7610806 Fix sporadic openflow/broker test failure
Looked like a possible race condition in how the test was structured: an
endpoint sees its peer got lost and likewise exits immediately before
having a chance to process events the peer had sent just before exiting.
Fix is to reverse which endpoint initiates the termination sequence so
we can be sure we see the required events.
2019-05-03 11:22:56 -07:00
Johanna Amann
6d47077222 Merge remote-tracking branch 'origin/topic/jsiwek/gh-340'
* origin/topic/jsiwek/gh-340:
  GH-340: Improve IPv4/IPv6 regexes, extraction, and validity functions
2019-05-02 12:28:31 -07:00
Robin Sommer
789cb376fd GH-239: Rename bro to zeek, bro-config to zeek-config, and bro-path-dev to zeek-path-dev.
This also installs symlinks from "zeek" and "bro-config" to a wrapper
script that prints a deprecation warning.

The btests pass, but this is still WIP. broctl renaming is still
missing.

#239
2019-05-01 21:43:45 +00:00
Johanna Amann
e85a016521 Parse pre-shared-key extension.
No documentation yet...
2019-04-22 23:02:39 +02:00