Commit graph

3495 commits

Author SHA1 Message Date
Jan Grashoefer
2d9b90cddc Added test for intel removal policy script. 2019-03-24 22:24:12 +01:00
Jan Grashoefer
0f86aaff4b Added test for intel item filtering. 2019-03-24 21:32:45 +01:00
Jan Grashoefer
3e31f3d554 Added hook to filter intelligence items. 2019-03-24 21:31:48 +01:00
Jon Siwek
1b76d92e97 Merge branch 'smb3-negotiate-response' of https://github.com/mauropalumbo75/zeek
* 'smb3-negotiate-response' of https://github.com/mauropalumbo75/zeek:
  added test and pcap files for smb 3.1.1 negotiate-response
  smb3.1.1 additions to negotiate-response command

I made several modifications:

  - Code format, style, naming changes

  - For completeness/correctness, I added parsing support for the remaining
    context type structures.

  - Moved the optional padding before the NegotiateContextList field to
    also require the 0x0311 dialect version (some failures in
    pre-existing unit tests pointed this out as an issue)
2019-03-21 14:13:21 -07:00
Jon Siwek
9eb5449ecb Merge branch 'smb3-transform-header' of https://github.com/mauropalumbo75/zeek
* 'smb3-transform-header' of https://github.com/mauropalumbo75/zeek:
  clean up, test and pcap for transform_header added
  added smb2-com-transform-header for smb3.x
2019-03-20 19:30:32 -07:00
Jon Siwek
2d8acab664 Merge branch 'smb2-fix' of https://github.com/mauropalumbo75/zeek
* 'smb2-fix' of https://github.com/mauropalumbo75/zeek:
  added test and pcap files for smb_files.log fix
  fixing some missing log lines in smb_files.log
2019-03-20 18:01:35 -07:00
Jon Siwek
a262ed8d9c Merge branch 'master' of https://github.com/rdenniston/zeek
* 'master' of https://github.com/rdenniston/zeek:
  Add linux netfilter NFLOG capture functionality initial commit

I made modifications:

  - Formatting / code style

  - More error handling and validity checks

  - The Type and Length value of TLVs is technically host order

  - Changed / fixed the Length value padding check: it's generally
    32-bit alignment, not just aligning any TLV less than 8 bytes.
2019-03-19 19:19:02 -07:00
Jon Siwek
f19db92508 Merge branch 'topic/christian/inputframework-paths' of https://github.com/ckreibich/zeek
* 'topic/christian/inputframework-paths' of https://github.com/ckreibich/zeek:
  Source file path control for Input and Intel frameworks
2019-03-18 11:48:22 -07:00
Jon Siwek
01d303b480 Migrate table-based for-loops to key-value iteration 2019-03-15 19:54:44 -07:00
Jon Siwek
41c7b229d3 Merge branch 'master' of https://github.com/ZekeMedley/zeek
* 'master' of https://github.com/ZekeMedley/zeek:
  Add key-value for loop

Fixes GH-154
2019-03-15 19:45:07 -07:00
Christian Kreibich
3f02c0a67c Source file path control for Input and Intel frameworks
This introduces the following redefinable string constants, empty by
default:

- InputAscii::path_prefix
- InputBinary::path_prefix
- Intel::path_prefix

When using ASCII or binary reades in the Input/Intel Framework with an
input stream source that does not have an absolute path, these
constants cause Zeek to prefix the resulting paths accordingly. For
example, in the following the location on disk from which Zeek loads
the input becomes "/path/to/input/whitelist.data":

redef InputAscii::path_prefix = "/path/to/input";

event bro_init()
        {
        Input::add_table([$source="whitelist.data", ...]);
	}

These path prefixes can be absolute or relative. When an input stream
source already uses an absolute path, this path is preserved and the
new variables have no effect (i.e., we do not affect configurations
already using absolute paths).

Since the Intel framework builds upon the Input framework, the first
two paths also affect Intel file locations. If this is undesirable,
the Intel::path_prefix variable allows specifying a separate path:
when its value is absolute, the resulting source seen by the Input
framework is absolute, therefore no further changes to the paths
happen.
2019-03-15 16:43:36 -07:00
Jon Siwek
be7110f6c0 Make Syslog analyzer accept messages that omit Priority
Essentially, it will now process/parse priority values if they are
there, or else just accept whatever remaining data/text is there as the
syslog message.  Reasoning is that there's syslog producers out there
that may have simply forgotten/neglected to send the priority value
and technically won't conform to what the standard says, though we can
infer the intent (some syslog consumers already may do similarly, but
I didn't verify).
2019-03-14 18:47:32 -07:00
Robin Sommer
72fa3f16ad Merge remote-tracking branch 'origin/topic/jsiwek/vxlan'
* origin/topic/jsiwek/vxlan:
  GH-250: Improve/cleanup VXLAN decapsulation support
  Initial VXLAN support, need -C flag to work when running bro
2019-03-14 17:59:42 +00:00
ZekeMedley
1f7924754e Add key-value for loop 2019-03-14 09:46:16 -07:00
Jon Siwek
1239a286e2 Merge branch 'patch-1' of https://github.com/neu5ron/bro
* 'patch-1' of https://github.com/neu5ron/bro:
  Update main.bro
  Update main.bro
2019-03-13 13:49:17 -07:00
Robin Sommer
56da70c148 Merge remote-tracking branch 'origin/topic/jsiwek/gh-289'
* origin/topic/jsiwek/gh-289:
  GH-289: Add options to limit entries in http.log file fields
2019-03-13 18:23:06 +00:00
Robin Sommer
4cf6cb783a Merge remote-tracking branch 'origin/topic/jsiwek/gh-281'
Closes #281.

* origin/topic/jsiwek/gh-281:
  GH-281: Improve parsing of Google Pixel user agent
2019-03-13 18:09:21 +00:00
Robin Sommer
e2172018ee Merge remote-tracking branch 'origin/topic/jsiwek/gh-286'
Added note about different behaviour to NEWS.

Closes #286.

* origin/topic/jsiwek/gh-286:
  GH-286: Check for record type mismatch in ternary operator
2019-03-13 18:01:36 +00:00
Jon Siwek
09ae539ea8 GH-250: Improve/cleanup VXLAN decapsulation support
* Better parsing/error-checking of VXLAN and encapsulated packet headers

* Add/implement the "vxlan_packet" event

* Add "Tunnel::vxlan_ports" option to tune the set of VXLAN ports to
  analyze/decapsulate

* Add "Tunnel::validate_vxlan_checksums" option to allow for tuning of how
  checksums associated with the outer UDP header of a possible VXLAN
  tunnel are handled

Fixes GH-250
2019-03-12 18:15:34 -07:00
Henrik Lund Kramshoej
f4088be8a6 Initial VXLAN support, need -C flag to work when running bro 2019-03-11 14:15:39 -07:00
Jon Siwek
011b6e10bf Merge remote-tracking branch 'origin/topic/jazoff/dns-perf'
* origin/topic/jazoff/dns-perf:
  update baselines
  Use more efficient method to grab the protocol.
  improve performance of dns policy
2019-03-08 16:23:30 -08:00
Jon Siwek
628a46d8fd GH-219: revert previous change to |x| operator for interval/time
The result of the |x| operator for interval and time types historically
returned a value of type double.  This was changed as part of
3256ac7c49 to return interval/time, but
this now reverts to returning a double again to avoid introducing a
change that may break user code.

Fixes GH-219
2019-03-06 15:36:31 -08:00
Jon Siwek
92b6656bad GH-289: Add options to limit entries in http.log file fields
The "orig_fuids", "orig_filenames", "orig_mime_types" http.log fields as
well as their "resp" counterparts are now limited to having
"HTTP::max_files_orig" or "HTTP::max_files_resp" entries, which are 15
by default.  The limit can also be ignored case-by-case via the
"HTTP::max_files_policy" hook.

Fixes GH-289
2019-02-27 14:25:15 -06:00
Jon Siwek
8b9d525097 Update external test suite commit hash 2019-02-26 15:09:32 -06:00
Jon Siwek
53dc1176b7 GH-281: Improve parsing of Google Pixel user agent
Fixes GH-281
2019-02-25 14:07:53 -06:00
Jon Siwek
74c225c7cb GH-286: Check for record type mismatch in ternary operator
Fixes GH-286
2019-02-25 12:55:03 -06:00
mauro
a346b01a85 clean up, test and pcap for transform_header added 2019-02-21 12:01:02 +01:00
Johanna Amann
cb47b37215 Be more liberal with whitespaces for ip-addresses, subnets, etc.
This change ignores leading/trailing whitespaces for a couple of
data-types (bool, port, subnet, addr) and just parses them as if the
whitespace was not present.
2019-02-19 15:06:06 -08:00
Justin Azoff
ea167846fa update baselines 2019-02-17 12:48:24 -05:00
Jon Siwek
aff3f4b3fd Skip autogenerated doc coverage test for Travis pull requests 2019-02-15 17:53:53 -06:00
Jon Siwek
e6e328efb4 Merge branch 'master' of https://github.com/ZekeMedley/zeek
* 'master' of https://github.com/ZekeMedley/zeek:
  lstrip test output cleanup
  implemented rstrip
  add rstrip tests
  cleanup of lstrip function
  added implementation of lstrip
  added tests for lstrip function
2019-02-15 17:51:29 -06:00
mauro
84afafc512 added test and pcap files for smb_files.log fix 2019-02-14 16:51:50 +01:00
Zeke Medley
ffbe366646 lstrip test output cleanup 2019-02-14 00:02:41 -08:00
Zeke Medley
8ac53a5fae add rstrip tests 2019-02-13 23:51:27 -08:00
Zeke Medley
f9191d439b added tests for lstrip function 2019-02-13 23:19:31 -08:00
mauro
c9cc1a55b9 added test and pcap files for smb 3.1.1 negotiate-response 2019-02-13 17:39:37 +01:00
Jon Siwek
0790c1c559 Merge remote-tracking branch 'origin/topic/jsiwek/gh-211'
* origin/topic/jsiwek/gh-211:
  GH-208: change invalid subnet expressions to a runtime error
  GH-211: improve consistency of how scripting errors are handled

Removed the 'allow_init_errors' option.
2019-02-07 10:33:52 -06:00
Jon Siwek
01a8418d79 GH-208: change invalid subnet expressions to a runtime error
Rather than abort.
2019-01-30 12:01:46 -06:00
Jon Siwek
67484a90fa GH-211: improve consistency of how scripting errors are handled
Scripting errors/mistakes now consistently generate a runtime error
which have the behavior of unwinding the call stack all the way out of
the current event handler.

Before, such errors were not treated consistently and either aborted
the process entirely or emitted a message while continuing to execute
subsequent statements without well-defined behavior (possibly causing
a cascade of errors).

The previous behavior also would only unwind out of the current
function (if within a function body), not out the current event
handler, which is especially problematic for functions that return
a value: the caller is essentially left a mess with no way to deal
with it.

This also changes the behavior of the startup/initialization process
to abort if there's errors during bro_init() rather than continue one
to the main run loop.  The `allow_init_errors` option may change this
new, default behavior.
2019-01-30 11:20:09 -06:00
Jon Siwek
49a30d61cf Fix memory leak due to enum type/val circular references
Introduced in 2982765128
2019-01-29 18:28:13 -06:00
Jon Siwek
3e2c2a6140 GH-210: improve call stack tracking
Adds tracking of arguments
2019-01-28 14:17:04 -06:00
Johanna Amann
7465bceb7e Merge remote-tracking branch 'origin/topic/jsiwek/gh-167'
* origin/topic/jsiwek/gh-167:
  GH-167: improve error message for unclosed function at EOF
2019-01-24 15:22:42 -08:00
Jon Siwek
7a3ecd76b4 Merge remote-tracking branch 'origin/topic/johanna/md5-fips'
* origin/topic/johanna/md5-fips:
  A few more updates to the digest functions.
  Tell OpenSSL that MD5 is not used for security in order to allow bro to work properly on a FIPS system

I changed a couple places that looked like memory management pitfalls:
moved some cleanup code into the dtors of HashVal derived classes
(seemed like it got stuck in ctors by accident) and also added a
cautionary cleanup in the MIME code.

Plus minor formatting changes.
2019-01-24 17:20:06 -06:00
Jon Siwek
9b43f1d5f4 GH-167: improve error message for unclosed function at EOF 2019-01-24 14:54:39 -06:00
Johanna Amann
ec2d98b382 Merge remote-tracking branch 'origin/master' into topic/johanna/md5-fips 2019-01-24 12:19:17 -08:00
Johanna Amann
86161c85c4 A few more updates to the digest functions.
This builds upon the previous commit to make Zeek compile on FIPS
systems.

This patch makes the changes a bit more aggressive. Instead of having a
number of different hash functions with different return values, we now
standardize on EVP_MD_CTX and just have one set of functions, to which
the hash algorithm that is desired is passed.

On the positive side, this enables us to support a wider range of hash
algorithm (and to easily add to them in the future).

I reimplemented the internal_md5 function - we don't support ebdic
systems in any case.

The md5/sha1 serialization functions are now also tested (I don't think
they were before).
2019-01-24 10:44:28 -08:00
Stu H
5ef70c4df1 Added ERSPAN III testing 2019-01-24 14:05:13 +00:00
Jon Siwek
3256ac7c49 GH-219: fix |x| operator int overflow / floating point type inconsistency
When 'x' is an integral arithmetic expression, it's now coerced to
yield a signed integer before taking the absolute value of it to
prevent the common issue of unsigned integer overflow/wraparound for
values below zero.

Using a time or interval value/expression for 'x' now also yields a
time or interval, respective, from the |x| operation instead of
a double.
2019-01-22 16:42:40 -06:00
Johanna Amann
7c892ef7d4 Merge remote-tracking branch 'origin/topic/jsiwek/gh-151'
* origin/topic/jsiwek/gh-151:
  GH-151: fix hash calculation for nested sets
2019-01-22 09:08:14 -08:00
Jon Siwek
d86fb9f87a GH-151: fix hash calculation for nested sets
Hash key construction of nested sets depended on the order in
which their elements are iterated, which varied even between sets
containing equivalent elements.  The iteration order is now sorted
by each element's hash value (or, on collision, by full key) such
that equivalent sets no longer hash differently.
2019-01-18 20:54:22 -06:00