Commit graph

1331 commits

Author SHA1 Message Date
Christian Kreibich
3f02c0a67c Source file path control for Input and Intel frameworks
This introduces the following redefinable string constants, empty by
default:

- InputAscii::path_prefix
- InputBinary::path_prefix
- Intel::path_prefix

When using ASCII or binary reades in the Input/Intel Framework with an
input stream source that does not have an absolute path, these
constants cause Zeek to prefix the resulting paths accordingly. For
example, in the following the location on disk from which Zeek loads
the input becomes "/path/to/input/whitelist.data":

redef InputAscii::path_prefix = "/path/to/input";

event bro_init()
        {
        Input::add_table([$source="whitelist.data", ...]);
	}

These path prefixes can be absolute or relative. When an input stream
source already uses an absolute path, this path is preserved and the
new variables have no effect (i.e., we do not affect configurations
already using absolute paths).

Since the Intel framework builds upon the Input framework, the first
two paths also affect Intel file locations. If this is undesirable,
the Intel::path_prefix variable allows specifying a separate path:
when its value is absolute, the resulting source seen by the Input
framework is absolute, therefore no further changes to the paths
happen.
2019-03-15 16:43:36 -07:00
Jon Siwek
be7110f6c0 Make Syslog analyzer accept messages that omit Priority
Essentially, it will now process/parse priority values if they are
there, or else just accept whatever remaining data/text is there as the
syslog message.  Reasoning is that there's syslog producers out there
that may have simply forgotten/neglected to send the priority value
and technically won't conform to what the standard says, though we can
infer the intent (some syslog consumers already may do similarly, but
I didn't verify).
2019-03-14 18:47:32 -07:00
Robin Sommer
56da70c148 Merge remote-tracking branch 'origin/topic/jsiwek/gh-289'
* origin/topic/jsiwek/gh-289:
  GH-289: Add options to limit entries in http.log file fields
2019-03-13 18:23:06 +00:00
Jon Siwek
92b6656bad GH-289: Add options to limit entries in http.log file fields
The "orig_fuids", "orig_filenames", "orig_mime_types" http.log fields as
well as their "resp" counterparts are now limited to having
"HTTP::max_files_orig" or "HTTP::max_files_resp" entries, which are 15
by default.  The limit can also be ignored case-by-case via the
"HTTP::max_files_policy" hook.

Fixes GH-289
2019-02-27 14:25:15 -06:00
Jon Siwek
53dc1176b7 GH-281: Improve parsing of Google Pixel user agent
Fixes GH-281
2019-02-25 14:07:53 -06:00
mauro
a346b01a85 clean up, test and pcap for transform_header added 2019-02-21 12:01:02 +01:00
Johanna Amann
cb47b37215 Be more liberal with whitespaces for ip-addresses, subnets, etc.
This change ignores leading/trailing whitespaces for a couple of
data-types (bool, port, subnet, addr) and just parses them as if the
whitespace was not present.
2019-02-19 15:06:06 -08:00
mauro
84afafc512 added test and pcap files for smb_files.log fix 2019-02-14 16:51:50 +01:00
mauro
c9cc1a55b9 added test and pcap files for smb 3.1.1 negotiate-response 2019-02-13 17:39:37 +01:00
Jon Siwek
fd63168171 Merge remote-tracking branch 'origin/topic/jsiwek/parallelize-comm-tests'
* origin/topic/jsiwek/parallelize-comm-tests:
  Parallelize communication tests using btest TEST-PORT
2018-11-29 17:06:30 -06:00
Jon Siwek
635e030be2 Merge remote-tracking branch 'origin/master' into dev/2.7
* origin/master:
  Fix SumStats "last" plugin in cluster mode
  Remove unnessary check
  Support appending to vector of any
  Fix coding conventions nits/typos
  Updating submodule(s).
  Revert "Switch frag_timeout from redef to option"
  Improve error handling in x509_ocsp_verify function
  Updating submodule(s).
  Updating submodule(s).
  Update baseline for plugins.hooks for redef -> option changes
  Switch frag_timeout from redef to option
  Switch GridFTP options from redef to option
  Updating submodule(s).
  Fix a unit test relying on a bash-ism
2018-11-08 12:43:22 -06:00
Jon Siwek
01e17b5ea0 Fix SumStats "last" plugin in cluster mode 2018-11-08 12:27:37 -06:00
Jon Siwek
907297ba59 Parallelize communication tests using btest TEST-PORT 2018-11-04 15:29:59 -06:00
Jon Siwek
ebd8132063 Merge remote-tracking branch 'origin/master' into dev/2.7
* origin/master: (23 commits)
  Improve Travis script to show multiple core dump stacks
  Improve a weird stats unit test
  Updating submodule(s).
  Fix Travis script typo
  Add more debug output to Travis script
  Add core file search and stack trace output for Travis builds
  Updating submodule(s).
  Update license year for 2018
  GH-199: change `bro --help` exit status from 1 to 0
  Updating submodule(s).
  Add a test with an encrypted MySQL connection
  Fix parsing of MySQL NUL Strings, where we now require it to have a NUL value at the end.
  Updating submodule(s).
  Add missing record field comment
  Add missing record field comments
  Fix minor documentation mistakes
  Updating submodule(s).
  Updating submodule(s).
  Update btest baselines for fix in 46f727a6fa
  Generate ssh_auth_attempted for the 'none' authentication method.
  ...
2018-11-02 14:45:32 -05:00
Vlad Grigorescu
b0638dbdcf Add a test with an encrypted MySQL connection 2018-10-29 15:58:06 -05:00
Jon Siwek
2586e5aa3e Improve scripts/base/utils/dir unit test 2018-10-19 11:16:38 -05:00
Jon Siwek
45526e6063 Merge remote-tracking branch 'origin/master' into dev/2.7
* origin/master:
  Fix typo in Sessions.h
  Update baselines for SSH capabilities fix
  Fix SSH analyzer bug where is_server in capabilities is wrong.
  Revert DNS query 255 from ANY to *
  Missing commas...
  Fix typo in dce-rpc consts
  Refactor to use consistent numeric type in dce_rpc. Add missing DNS query type codes
  Add DCE_RPC exchange_mapi operations to relevant consts.bro file
  GH-186: fix JSON formatting of timestamps before Unix epoch
  Fix test baseline for plugin skeleton update
  Fix crash when modifying a table from within its &expire_func
  Convert site::local_nets, etc. into options.
  GH-184: add `bro-config --build_type`, outputs CMake build type
  Updating submodule(s).
  Add return value checks for some RPC parsing functions
  Add 'fallthrough' comment to a switch/case block
  Improve broker.remote_id unit test
  Increase broker unit test timeout intervals
2018-10-16 16:50:27 -05:00
Jon Siwek
70233148be GH-186: fix JSON formatting of timestamps before Unix epoch 2018-10-12 21:34:28 +00:00
Jon Siwek
71ef5c8428 Merge branch 'fatemabw/bro' of https://github.com/fatemabw/bro into dev/2.7
* 'fatemabw/bro' of https://github.com/fatemabw/bro:
  DNSSEC support in Bro

I made several changes:

  - renamed event/record types
  - reformatted the info added to dns.log
  - removed the "addl" scripts that added extended dnssec info to dns.log
  - simplifications/improvements to the internal parsing logic
2018-09-21 16:40:41 -05:00
Jon Siwek
c2b18849f8 Fix IRC names command parsing 2018-09-12 19:47:57 -05:00
Jon Siwek
c89c09fda3 Fix invalid memory free when using Log::default_field_name_map 2018-09-10 19:06:35 -05:00
Jon Siwek
c73bb8fdc4 Disable broker message forwarding by default
Still finding it to not be foolproof enough to enable generally for all
nodes in a cluster.  Specific/advanced use-cases may still consider
enabling, possibly just for specific nodes.
2018-09-06 18:32:22 -05:00
Johanna Amann
bec98b98f3 Weird settings: make constants into options.
The new weird settings are now all updateable during runtime.
2018-09-05 13:12:23 -07:00
fatema
ff5c11975d DNSSEC support in Bro 2018-09-05 14:12:07 -04:00
Jon Siwek
7e6fc58ab4 Merge remote-tracking branch 'origin/topic/johanna/tls-more-data'
* origin/topic/johanna/tls-more-data:
  Update NEWS for ssl changes.
  SSL: test updates for record_layer version
  Final touches to SSL events with record layer version.
  Introduce ssl_plaintext_data event.
  Add record layer version to event ssl_encrypted_data.
  Add compression methods to ssl_client_hello event.
2018-08-30 09:48:25 -05:00
Robin Sommer
6de436f3f6 Merge remote-tracking branch 'origin/topic/jsiwek/bit-1967'
* origin/topic/jsiwek/bit-1967:
  Fix a routing loop in control framework
  Add Broker::forward() function
  Enable implicit Broker message forwarding by default
  Remove Cluster::broadcast_topic
  Remove Intel Broker topics, re-use existing Cluster topics
  Remove "relay" family of Broker functions
2018-08-30 00:08:12 +00:00
Jon Siwek
01300f8706 Fix a unit test 2018-08-29 17:23:48 -05:00
Jon Siwek
1dcead93bf Add Broker::forward() function
This enables explicit forwarding of events matching a given topic
prefix.  Even if a receiving node has an event handler, it will not
be raised if the event was sent along a topic that matches a previous
call to Broker::forward().
2018-08-28 19:42:22 -05:00
Johanna Amann
4fd6cbd138 Merge remote-tracking branches 'origin/topic/dnthayer/ticket1963' and 'origin/topic/jsiwek/improve-input-reread'
* origin/topic/dnthayer/ticket1963:
  Convert more redef-able constants to runtime options

* origin/topic/jsiwek/improve-input-reread:
  Improve input framework re-read logic
2018-08-28 14:36:28 -07:00
Johanna Amann
23eb8096fc SSL: test updates for record_layer version
Update the tests to also include the recently included record layer
fields.
2018-08-28 11:02:20 -07:00
Johanna Amann
27d47314f7 Merge remote-tracking branch 'origin/master' into topic/johanna/tls-more-data 2018-08-27 09:25:40 -07:00
Johanna Amann
82cefd23c4 Fix base/misc/version.bro version parsing
Turns out that base/misc/version.bro did not parse Bro versions
correctly in case the version is just 2.5-12 or similar. This commit
fixes this oversight and adds a few more small testcases.
2018-08-24 17:25:16 -07:00
Johanna Amann
4b40b6ebe4 Merge remote-tracking branch 'origin/master' into topic/johanna/tls-more-data 2018-08-24 15:36:34 -07:00
Jon Siwek
1eeecf5fcc Stabilize a cluster logging unit test 2018-08-24 14:58:43 -05:00
Jon Siwek
f41f392743 Improve input framework re-read logic
Changed from checking for "has newer modification time" to "has
different modification time or inode number".
2018-08-24 12:46:31 -05:00
Johanna Amann
b2a0418dc5 Final touches to SSL events with record layer version. 2018-08-23 14:18:38 -07:00
Jon Siwek
620cd671ba Fix tracking of DCE-RPC context identifier mappings
This adds previously-missing support for "Alter Context"
request/response PDUs (initial patch contributed by Mark Fernandez).

Also, context ID arguments were added to dce_rpc_bind, dce_rpc_request,
and dce_rpc_response in order to properly track what endpoint/operation
a given opnum maps to.
2018-08-23 15:11:38 -05:00
Johanna Amann
b1dbd757a6 Merge remote-tracking branch 'origin/master' into topic/johanna/tls-more-data 2018-08-17 11:52:00 -07:00
Jon Siwek
a04c76c035 Enable SMB by default by moving scripts from policy/ to base/ 2018-08-16 17:23:28 -05:00
Jon Siwek
da9f91fc19 Add env. variables to override Broker listen/connect retry intervals
And use them to default retries to 1sec for all unit tests.
2018-08-16 12:16:03 -05:00
Jon Siwek
2d47586473 Merge remote-tracking branch 'origin/topic/johanna/config-framework-fixes'
* origin/topic/johanna/config-framework-fixes:
  Fix test that fails now that options are automatically redefable.
  Make options redef-able by default.
  Ascii formatter: do not complain about port text.
  Make parsing of booleans a little bit more lenient.
2018-08-13 10:54:39 -05:00
Jon Siwek
9f12b56105 Misc. unit test improvements 2018-08-10 16:58:27 -05:00
Johanna Amann
26ea1999ec Ascii formatter: do not complain about port text.
The ascii formatter already was happy to read ports in the form
"42/tcp"; however it emitted a warning message for each line.

This patch fixes this and adds a bit more testing for the existing
behavior.
2018-08-10 11:29:35 -07:00
Johanna Amann
116079a9ad Make parsing of booleans a little bit more lenient.
This makes the input framework (and everything else that uses the Ascii
parser) accept 0 and 1 as valid values for booleans.
2018-08-10 10:03:22 -07:00
Jon Siwek
599af26496 Merge remote-tracking branch 'origin/topic/vern/vec-append'
* origin/topic/vern/vec-append:
  d'oh, still have a (deprecated) string_array rather than string_vector
  forgot to update test suite results for v += e
  reap the fruits of v += e
  test case for v += e
  documentation of v += e
  v += e implemented

Fixed a mistake in find_ip_addresses()
2018-08-03 11:38:40 -05:00
Johanna Amann
a251b32d85 Merge remote-tracking branch 'origin/topic/jsiwek/openssl-1.1'
* origin/topic/jsiwek/openssl-1.1:
  Update install instructions for OpenSSL 1.1 compat
  Remove requestorName parameter of ocsp_request event
  Adjust x509 unit tests to work around OpenSSL 1.0 vs. 1.1 differences
  Fixes for OpenSSL 1.1 support
2018-07-31 13:13:06 +02:00
Vern Paxson
88fd7510c6 reap the fruits of v += e 2018-07-26 12:51:36 -07:00
Robin Sommer
0f74e1eedd Add serialization group to clustered config framework tests. 2018-07-24 01:39:39 +00:00
Robin Sommer
f4cd567f84 Merge remote-tracking branch 'origin/topic/johanna/config-cluster'
BIT-1958 #merged

* origin/topic/johanna/config-cluster:
  Add vector to read_config_cluster test.
  Fix special-case-bug for vectors in UnaryExpr.
  Config: another cluster test-case, this time reading in a file.
  Add sending of values to nodes that dropped out.
  Continue work on config framework clusterization.
  Teach Option::set to unwrap Broker::Data values
  Start clusterizing configuration framework.
2018-07-24 01:36:59 +00:00
Johanna Amann
f3d4ba51af Add vector to read_config_cluster test.
It works now after fixing the vector-any-UnaryExpr Bro bug.
2018-07-20 13:41:44 -07:00