Commit graph

646 commits

Author SHA1 Message Date
Jon Siwek
186e67ec1d Allow logging filters to inherit default path from stream.
This allows the path for the default filter to be specified explicitly
when creating a stream and reduces the need to rely on the default path
function to magically supply the path.

The default path function is now only used if, when a filter is added to
a stream, it has neither a path nor a path function already.

Adapted the existing Log::create_stream calls to explicitly specify a
path value.

Addresses BIT-1324
2015-03-19 14:49:55 -05:00
Jon Siwek
9e53722b57 Rename comm/ directories to broker/ 2015-03-05 17:02:25 -06:00
Jon Siwek
fa08083a92 Rename broker-related namespaces.
c++ namespace "comm" -> bro_broker
script module "Comm" -> BrokerComm
script module "Store" -> BrokerStore
2015-03-05 16:20:51 -06:00
Robin Sommer
dfc88094ab Merge remote-tracking branch 'origin/topic/jsiwek/broker'
* origin/topic/jsiwek/broker: (34 commits)
  Update broker submodule.
  Update broker submodule.
  broker integration: add missing baselines for doc tests
  broker integration: add prof.log statistics
  broker integration: add high-level usage documentation
  broker integration: add API documentation (broxygen/doxygen)
  broker integration: fix memory leak, add leak tests
  Update broker submodule.
  Improve comm tests.
  Fix gcc compile warnings.
  broker integration: fix unit tests to work when broker is not enabled.
  Add --enable-c++11 configure flag.
  broker integration: add (un)publish/(un)advertise functions
  broker integration: add knobs to set auto publish/advertise behavior
  broker integration: move listen port for unit tests to a btest variable
  broker integration: add events for incoming connection status updates
  broker integration: adapt to change in expiration_time
  Update coverage unit test baselines.
  broker integration: add Comm::enable function
  broker integration: process debug/diagnostic reports from broker
  ...

Conflicts:
	cmake
	testing/btest/Baseline/plugins.hooks/output
2015-03-02 17:10:15 -08:00
Jon Siwek
b06d82cced broker integration: add API documentation (broxygen/doxygen)
Also changed asynchronous data store query code a bit; trying to make
memory management and handling of corner cases a bit clearer (former
maybe could still be better, but I need to lookup queries by memory
address to associate response cookies to them, and so wrapping pointers
kind of just gets in the way).
2015-02-17 10:50:57 -06:00
Jon Siwek
e95116ba85 Merge branch 'master' into topic/jsiwek/broker 2015-02-16 10:00:17 -06:00
Jon Siwek
ebc9407a2b broker integration: add knobs to set auto publish/advertise behavior 2015-02-09 16:26:31 -06:00
Robin Sommer
23b9705a7b Fixing analyzer tag types for some Files::* functions. 2015-02-08 18:23:22 -08:00
Jon Siwek
6b115c6999 Merge branch 'master' into topic/jsiwek/broker 2015-02-02 11:45:21 -06:00
Jon Siwek
9875f5d3eb broker integration: add distributed data store api
But haven't done the full gamut of testing on it yet.
2015-01-30 14:39:16 -06:00
Jon Siwek
d2ea87735a broker integration: add bifs to inspect/manipulate broker data
i.e. script-layer functions to convert between bro values and broker
values; mostly for use w/ Bro's data store interface (coming soon).
2015-01-29 10:42:48 -06:00
Jon Siwek
23f04835c6 Deprecate split* family of BIFs.
These functions are now deprecated in favor of alternative versions that
return a vector of strings rather than a table of strings.

Deprecated functions:

- split: use split_string instead.
- split1: use split_string1 instead.
- split_all: use split_string_all instead.
- split_n: use split_string_n instead.
- cat_string_array: see join_string_vec instead.
- cat_string_array_n: see join_string_vec instead.
- join_string_array: see join_string_vec instead.
- sort_string_array: use sort instead instead.
- find_ip_addresses: use extract_ip_addresses instead.

Changed functions:

- has_valid_octets: uses a string_vec parameter instead of string_array.

Addresses BIT-924, BIT-757.
2015-01-21 15:34:42 -06:00
Jon Siwek
7e563b7275 broker integration: add remote events 2015-01-15 15:45:08 -06:00
Jon Siwek
1e462481dc broker integration: add remote printing 2015-01-14 13:28:34 -06:00
Jon Siwek
0daa954ddb broker integration: add remote connection status events. 2015-01-14 10:40:11 -06:00
Daniel Thayer
b5e9433b04 Improve documentation of the Intelligence Framework
Added some missing information and rearranged a few sentences so the order
makes more sense.
2015-01-07 00:01:35 -06:00
Jon Siwek
cbbe7b52dc Review/fix/change file reassembly functionality.
- Re-arrange how some fa_file fields (e.g. source, connection info, mime
  type) get updated/set for consistency.

- Add more robust mechanisms for flushing the reassembly buffer.
  The goal being to report all gaps and deliveries to file analyzers
  regardless of the state of the reassembly buffer at the time it has to
  be flushed.
2014-12-16 14:05:15 -06:00
Seth Hall
e879aa78f5 Merge remote-tracking branch 'origin/topic/seth/mime-updates' into topic/seth/files-reassembly-and-mime-updates
Conflicts:
	scripts/base/init-bare.bro
	testing/btest/Baseline/scripts.policy.misc.dump-events/all-events-no-args.log
	testing/btest/Baseline/scripts.policy.misc.dump-events/all-events.log
2014-11-05 11:42:34 -05:00
Seth Hall
842dfd8b4a Merge remote-tracking branch 'origin/topic/seth/files-tracking' into topic/seth/files-reassembly-and-mime-updates
Conflicts:
	testing/btest/Baseline/scripts.base.frameworks.file-analysis.http.multipart/out
	testing/btest/Baseline/scripts.policy.misc.dump-events/all-events.log
2014-11-05 11:40:26 -05:00
Seth Hall
7ee34981aa Improve TAR file detection and other small changes.
- Remove all of the x-c detections.  Nearly all false
    positives.

  - Remove the back up TAR detections.  Not very helpful.

  - Remove one of the x-elc detections that was too loose
    and caused many false positives.
2014-11-05 11:31:48 -05:00
Seth Hall
d77243823f Updates for file mime type identification.
- Change to the default BOF buffer size to 3000 (was 1024).
 - Reorganized MS signatures into a separate file
 - Improved lots of the signatures and added new ones.
2014-10-08 02:12:10 -04:00
Seth Hall
80656d5294 Improves shockwave flash file signatures.
- This moves the signatures out of the libmagic imported signatures
   and into our own general.sig.

 - Expand the detection to LZMA compressed flash files.
2014-10-06 11:13:13 -04:00
Seth Hall
cafd35e746 Updates the files event api and brings file reassembly up to master. 2014-09-26 00:40:37 -04:00
Seth Hall
42b2d56279 Merge remote-tracking branch 'origin/master' into topic/seth/files-tracking
Conflicts:
	scripts/base/frameworks/files/main.bro
	src/file_analysis/File.cc
	testing/btest/Baseline/scripts.base.frameworks.file-analysis.actions.data_event/out
2014-09-23 13:05:39 -04:00
Seth Hall
8d283db63b Adds a "node" field to Intel::Seen struture and intel.log.
The intel framework can now indicate which node discovered a
hit on an intel item through the new "node" field in the
Intel::Seen data structure.  On clusters, this field will
contain the name of the node where the hit was seen.
2014-09-23 12:23:39 -04:00
Daniel Thayer
d226fef723 Fixed some "make doc" warnings caused by reST formatting 2014-09-16 12:44:51 -05:00
Robin Sommer
996d118d68 Fixing tests. 2014-08-13 21:33:03 -07:00
Robin Sommer
58f3a715f2 Merge branch 'topic/robin/reader-writer-plugins' of git.bro.org:bro into topic/robin/reader-writer-plugins
Conflicts:
	scripts/base/frameworks/logging/writers/dataseries.bro
2014-08-08 18:36:09 -07:00
Robin Sommer
8737eae906 Move DataSeries and ElasticSearch into plugins. 2014-08-08 18:32:21 -07:00
Robin Sommer
8031da4ee7 More polishing of some of the branche's changes. 2014-08-08 18:32:05 -07:00
Robin Sommer
ffd3d9d185 More polishing. 2014-07-31 15:08:45 -07:00
Robin Sommer
2b505b07c1 Merge remote-tracking branch 'origin/master' into topic/robin/reader-writer-plugins 2014-07-31 10:10:39 -07:00
Jon Siwek
69b1ba653d Minor adjustments to plugin code/docs.
Mostly whitespace/typos.
Moved some Plugin methods out from public access.
2014-07-30 16:48:23 -05:00
Robin Sommer
48b251abd1 Merge branch 'topic/robin/dynamic-plugins-2.3' into topic/robin/reader-writer-plugins 2014-07-22 17:27:16 -07:00
Robin Sommer
c9524757d2 Adding Files::register_for_mime_type() to associate a file analyzer
with a MIME type.

Whenever that MIME is detected, Bro will now automatically activate
the analyzer. The interface mimics how well-known ports are defined
for protocol analyzers.

This isn't actually used by any existing file analyzer (because we
don't have any yet that target a specific file format), but there's a
test making sure it works.
2014-07-21 16:31:22 +02:00
Robin Sommer
f4cbcb9b03 Converting log writers and input readers to plugins. 2014-07-20 19:17:58 +02:00
Robin Sommer
9616cd8e61 Further polishing and cleanup in preparation for merge. 2014-07-12 18:12:09 -07:00
Robin Sommer
aeb8e71e8c Merge remote-tracking branch 'origin/master' into topic/robin/dynamic-plugins-2.3
Conflicts:
	aux/bro-aux
	aux/broccoli
2014-07-10 20:11:52 -07:00
Robin Sommer
a7746afa0a Fixing DataSeries, which was using a now illegal value as default
compression level.
2014-07-10 14:50:15 -07:00
Robin Sommer
ba7af428a7 Merge remote-tracking branch 'origin/master' into topic/robin/dynamic-plugins-2.3 2014-06-13 09:27:02 -07:00
Daniel Thayer
5e23e57025 Fix minor formatting issues in script docs 2014-06-12 00:33:55 -05:00
Daniel Thayer
95c7128d71 Update some info in the docs 2014-06-07 12:31:32 -05:00
Seth Hall
8d9940c8c3 Merge remote-tracking branch 'origin/master' into topic/seth/files-tracking
Conflicts:
	src/Reassem.cc
	src/Reassem.h
	src/analyzer/protocol/tcp/TCP_Reassembler.cc
	testing/btest/Baseline/scripts.base.frameworks.file-analysis.bifs.set_timeout_interval/bro..stdout
	testing/btest/Baseline/scripts.base.frameworks.file-analysis.http.partial-content/b.out
	testing/btest/Baseline/scripts.base.frameworks.file-analysis.http.partial-content/c.out
	testing/btest/Baseline/scripts.base.frameworks.file-analysis.logging/files.log
2014-05-27 10:56:11 -04:00
Seth Hall
c536db0feb Merge remote-tracking branch 'origin/topic/bernhard/ticket1177'
* origin/topic/bernhard/ticket1177:
  define empty request_key method for sumstats in cluster mode.
2014-05-15 13:50:16 -04:00
Robin Sommer
bbd409d274 Merge remote-tracking branch 'origin/master' into topic/robin/dynamic-plugins-2.3
(Never good to name a branch after version anticipated to include it ...)
2014-05-14 16:23:04 -07:00
Robin Sommer
988ba2e897 Add Java version to software framework
BIT-1168 #merged
2014-04-24 17:09:40 -07:00
Bernhard Amann
4ae52d9e1c Support parsing of several TLS extensions.
At the moment, we have support for:
elliptic_curves: client supported elliptic curves
ec_point_formats: list of client supported EC point formats
application_layer_protocol_negotiation: list of supported application layer protocols (used for spdy/http2 negotiation)
server_name: server name sent by client. This was supported before, but... a bit brittle.
2014-04-23 14:34:06 -07:00
Bernhard Amann
b283883997 define empty request_key method for sumstats in cluster mode.
This prevents the worker nodes from crashing, when request_key is used
in cluster mode and called on the worker and the manager nodes (i.e. when
a non-cluster-aware script is used).

Addresses BIT-1177
2014-04-18 16:29:51 -07:00
Bernhard Amann
9438bc166b change the sumstats table that tracks recent intermediate requests
back to create interval.

In this instance a read expiry actually makes much less sense.
2014-04-03 13:52:26 -07:00
Seth Hall
df1647ee00 Add a uid field to the signatures.log (contributed by Anthony Verez)
- Addresses BIT-1172
2014-04-03 10:49:41 -04:00