Commit graph

6667 commits

Author SHA1 Message Date
Johanna Amann
95c72f3717 Update submodule
[nomail]
2018-08-17 11:25:58 -07:00
Jon Siwek
15dc5d1dda BIT-1850: add missing DCE/RPC PDU type enum values 2018-08-16 14:09:03 -05:00
Jon Siwek
da9f91fc19 Add env. variables to override Broker listen/connect retry intervals
And use them to default retries to 1sec for all unit tests.
2018-08-16 12:16:03 -05:00
Jon Siwek
05b10fe2e7 BIT-1544: allow NULs in file analysis handles 2018-08-15 18:03:02 -05:00
Jon Siwek
f336c8c710 Fix seg fault on trying to type-cast invalid/nil Broker::Data
This situation now throws a runtime expression exception instead of
crashing on null pointer access.
2018-08-15 11:02:52 -05:00
Jon Siwek
0e6913fba0 BIT-1798: fix PPTP GRE tunnel decapsulation 2018-08-14 16:48:04 -05:00
Jon Siwek
d66a589558 Add 'smtp_excessive_pending_cmds' weird 2018-08-13 16:31:11 -05:00
Jon Siwek
1f7e112879 Fix SMTP command string comparisons 2018-08-13 16:30:44 -05:00
Jon Siwek
2d47586473 Merge remote-tracking branch 'origin/topic/johanna/config-framework-fixes'
* origin/topic/johanna/config-framework-fixes:
  Fix test that fails now that options are automatically redefable.
  Make options redef-able by default.
  Ascii formatter: do not complain about port text.
  Make parsing of booleans a little bit more lenient.
2018-08-13 10:54:39 -05:00
Jon Siwek
67524f26d5 Immediately apply broker subscriptions made during bro_init()
Otherwise that's begging for unit test failures due to races
2018-08-10 17:18:21 -05:00
Jon Siwek
083947af41 Update default broker threading configuration
Now defaults to a max of 4 threads typically indepedent of core
count (previously could go up to a hard cap of 8).  Also now allow
controlling this setting via BRO_BROKER_MAX_THREADS environment
variable.
2018-08-10 17:08:26 -05:00
Johanna Amann
c34fbee0d1 Make options redef-able by default. 2018-08-10 11:53:36 -07:00
Johanna Amann
26ea1999ec Ascii formatter: do not complain about port text.
The ascii formatter already was happy to read ports in the form
"42/tcp"; however it emitted a warning message for each line.

This patch fixes this and adds a bit more testing for the existing
behavior.
2018-08-10 11:29:35 -07:00
Johanna Amann
116079a9ad Make parsing of booleans a little bit more lenient.
This makes the input framework (and everything else that uses the Ascii
parser) accept 0 and 1 as valid values for booleans.
2018-08-10 10:03:22 -07:00
Jon Siwek
df2e2672d9 Updating submodule(s).
[nomail]
2018-08-08 09:43:46 -05:00
Jon Siwek
7126616707 Fix an "uninitialized" compiler warning
Though it is actually initialized on all non-aborting code paths.
2018-08-06 17:10:18 -05:00
Jon Siwek
599af26496 Merge remote-tracking branch 'origin/topic/vern/vec-append'
* origin/topic/vern/vec-append:
  d'oh, still have a (deprecated) string_array rather than string_vector
  forgot to update test suite results for v += e
  reap the fruits of v += e
  test case for v += e
  documentation of v += e
  v += e implemented

Fixed a mistake in find_ip_addresses()
2018-08-03 11:38:40 -05:00
Jon Siwek
06c6e1188a Merge remote-tracking branch 'origin/topic/vern/set-ops2'
* origin/topic/vern/set-ops2:
  documentation, test suite update
  implemented set relationals
  bug fix for set intersection
  set intersection implemented
  mirroring previous topic/vern/set-ops to get branch up to date, since I'm a n00b

Fixed a couple memory leaks and added a leak test
2018-08-02 10:43:41 -05:00
Jon Siwek
5804c940f1 Merge remote-tracking branch 'origin/topic/vern/perf-history'
* origin/topic/vern/perf-history:
  only generate history threshold events for > 1 instance mention those events in NEWS
  a different sort of history update
  'W' for zero window implemented; logarithmic 'T'/'C'/'W' history repetitions

I reverted a change that made TCP window tracking unconditional (possibly
accepting out-of-order packets) until further verification of test suite
changes.
2018-08-01 16:37:10 -05:00
Jon Siwek
a4dbc5b310 Merge branch 'fix-ntlm-negotiate-flags' of https://github.com/jbencteux/bro
* 'fix-ntlm-negotiate-flags' of https://github.com/jbencteux/bro:
  fix NTLM NegotiateFlags field offsets
2018-08-01 12:16:19 -05:00
Jon Siwek
46bf6731e7 Merge branch 'topic/feature/upstream/cfp' of https://github.com/corelight/bro
* 'topic/feature/upstream/cfp' of https://github.com/corelight/bro:
  Add Cisco FabricPath support
2018-07-31 16:51:41 -05:00
Jon Siwek
a71d84c968 Merge branch 'topic/feature/upstream/geoip' of https://github.com/corelight/bro
* 'topic/feature/upstream/geoip' of https://github.com/corelight/bro:
  Replace GeoIP Legacy DB support with MaxMind DB support
2018-07-31 16:50:15 -05:00
Jon Siwek
6044983666 Improve handling of empty lines in several text protocol analyzers 2018-07-31 13:31:59 -05:00
Johanna Amann
a251b32d85 Merge remote-tracking branch 'origin/topic/jsiwek/openssl-1.1'
* origin/topic/jsiwek/openssl-1.1:
  Update install instructions for OpenSSL 1.1 compat
  Remove requestorName parameter of ocsp_request event
  Adjust x509 unit tests to work around OpenSSL 1.0 vs. 1.1 differences
  Fixes for OpenSSL 1.1 support
2018-07-31 13:13:06 +02:00
Jonathan Perkins
1ae7d3b349 Replace GeoIP Legacy DB support with MaxMind DB support
This updates the "lookup_location" and "lookup_asn" BIFs to use
libmaxminddb.  The motivation for this is that MaxMind is discontinuing
GeoLite Legacy databases: no updates after April 1, 2018, no downloads
after January 2, 2019.  It's also noted that all GeoIP Legacy databases
may be discontinued as they are superseded by GeoIP2.
2018-07-30 16:04:58 -05:00
Damani Wade
3710ff936f Add Cisco FabricPath support 2018-07-27 16:00:54 -05:00
Jon Siwek
35827eeb31 Add rate-limiting sampling mechanism for weird events
The generation of weird events, by default, are now rate-limited
according to these tunable options:

  - Weird::sampling_whitelist
  - Weird::sampling_threshold
  - Weird::sampling_rate
  - Weird::sampling_duration

The new get_reporter_stats() BIF also allows one to query the
total number of weirds generated (pre-sampling) which the new
policy/misc/weird-stats.bro script uses periodically to populate
a weird_stats.log.

There's also new reporter BIFs to allow generating weirds from the
script-layer such that they go through the same, internal
rate-limiting/sampling mechanisms:

  - Reporter::conn_weird
  - Reporter::flow_weird
  - Reporter::net_weird

Some of the code was adapted from previous work by Johanna Amann.
2018-07-26 19:57:36 -05:00
Vern Paxson
dfe0768fa1 v += e implemented 2018-07-26 12:18:31 -07:00
Jeffrey Bencteux
1bee7277e0 fix NTLM NegotiateFlags field offsets
Wrong offsets were used for the NegotiateFlags field of the
NEGOTIATE_MESSAGE, CHALLENGE_MESSAGE and AUTHENTICATE_MESSAGE. See
[MS-NLMP].pdf section 2.2.2.5 for a definition of that field.
2018-07-25 13:27:11 +02:00
Jon Siwek
3ed59249ba Exclude CMakeFiles from header installation path 2018-07-24 16:38:30 -05:00
Robin Sommer
8ac17d99a1 Merge remote-tracking branch 'origin/topic/jsiwek/bit-1950'
BIT-1950 #merged

* origin/topic/jsiwek/bit-1950:
  BIT-1950: support PPPoE over QinQ
2018-07-24 15:05:48 +00:00
Robin Sommer
566c50447f Merge remote-tracking branch 'origin/topic/jsiwek/source-tree-independence'
BIT-1955 #merged

* origin/topic/jsiwek/source-tree-independence:
  Support building plugins from Bro installation root
  Install binpac
  Move bifcl to a separate repo
2018-07-24 01:58:58 +00:00
Robin Sommer
f4cd567f84 Merge remote-tracking branch 'origin/topic/johanna/config-cluster'
BIT-1958 #merged

* origin/topic/johanna/config-cluster:
  Add vector to read_config_cluster test.
  Fix special-case-bug for vectors in UnaryExpr.
  Config: another cluster test-case, this time reading in a file.
  Add sending of values to nodes that dropped out.
  Continue work on config framework clusterization.
  Teach Option::set to unwrap Broker::Data values
  Start clusterizing configuration framework.
2018-07-24 01:36:59 +00:00
Robin Sommer
8233d82144 Fix some compiler warnings. 2018-07-23 18:07:15 +00:00
Johanna Amann
12add53131 Fix special-case-bug for vectors in UnaryExpr.
In some cases one can get the Type() of unaryexpr to be ANY. Vectors so
far did not deal gracefully with this and crashed because trying to
convert any to a vectortype.

This patch fixes this by just using the original vector-type in this
case.
2018-07-20 13:36:38 -07:00
Jon Siwek
385350a1f3 Make Broker congestion queue size tunable and increase default 2018-07-20 12:14:33 -05:00
Johanna Amann
da58f9d4a6 Merge remote-tracking branch 'origin/master' into topic/johanna/config-cluster 2018-07-18 09:56:01 -07:00
Jon Siwek
35b778eb4e Updating submodule(s).
[nomail]
2018-07-17 14:42:52 -05:00
Jon Siwek
9caad8a042 Port broker::data variant usages to use CAF API directly
Old code still all worked, but made use of Broker functions which
now just redirect to CAF ones.
2018-07-17 14:25:21 -05:00
Vern Paxson
f4728bd603 only generate history threshold events for > 1 instance
mention those events in NEWS
2018-07-17 10:25:45 -07:00
Jon Siwek
c09fe427a8 Improve Specific_RE_Matcher::CompileSet() error condition cleanup 2018-07-16 16:07:34 -05:00
Jon Siwek
463e540c9b Merge remote-tracking branch 'origin/topic/vern/case-insensitive-patterns'
* origin/topic/vern/case-insensitive-patterns:
  use PCRE syntax instead of the beautiful new (?i ...) syntax
  nitlet in NEWS entry
  test suite update for case-insensitive patterns
  document use of double quotes to escape case-insensitivity
  bug fix for recent memory leak patch
  documentation updates for case-insensitive patterns
  d'oh there's isalpha.  I looked earlier for isletter :-P
  fix for handling [:(lower|upper):] in case-insensitive patterns
  implemented /re/i for case-insensitive patterns
2018-07-16 16:04:38 -05:00
Jon Siwek
a45039de00 Merge remote-tracking branch 'origin/topic/vern/runtime-pattern-bifs'
* origin/topic/vern/runtime-pattern-bifs:
  de-restrict pattern-oriented BiFs to no longer require only running at init
2018-07-16 11:03:27 -05:00
Jon Siwek
2ffaa1cdb1 Support building plugins from Bro installation root
As opposed to plugins depending on a Bro source/build tree.  This
required installing various Bro headers, BinPAC and it's headers,
bifcl, and Bro's custom CMake modules.
2018-07-13 17:23:08 -05:00
Jon Siwek
e1b7820b01 Move bifcl to a separate repo 2018-07-12 17:51:23 -05:00
Vern Paxson
73349362a3 'W' for zero window implemented; logarithmic 'T'/'C'/'W' history repetitions 2018-07-09 13:05:10 -07:00
Vern Paxson
2a8ea87c9f implemented set relationals 2018-07-06 16:22:06 -07:00
Vern Paxson
e416d34f1f bug fix for set intersection 2018-07-06 13:46:06 -07:00
Jon Siwek
ad9abd4c9b BIT-1950: support PPPoE over QinQ 2018-07-06 08:04:02 -05:00
Vern Paxson
b9a5d9ccbe de-restrict pattern-oriented BiFs to no longer require only running at init 2018-07-05 10:13:20 -07:00