Commit graph

1007 commits

Author SHA1 Message Date
Seth Hall
19f498b4a4 Even more file type ident clean up.
- Add detection for ColdFusion scripts.
 - Support detection of XML/HTML with prefixed comment blocks.
2015-03-14 00:25:13 -04:00
Seth Hall
ee3e885712 Lots of fixes for file type identification.
- Plain text now identified with BOMs for UTF8,16,32
   (even though 16 and 32 wouldn't get identified as plain text, oh-well)
 - X.509 certificates are now populating files.log with
   the mime type application/pkix-cert.
 - File signatures are split apart into file types
   to help group and organize signatures a bit better.
 - Normalized some FILE_ANALYSIS debug messages.
 - Improved Javascript detection.
 - Improved HTML detection.
 - Removed a bunch of bad signatures.
 - Merged a bunch of signatures that ultimately detected
   the same mime type.
 - Added detection for MS LNK files.
 - Added detection for cross-domain-policy XML files.
 - Added detection for SOAP envelopes.
2015-03-13 22:14:44 -04:00
Jon Siwek
9e53722b57 Rename comm/ directories to broker/ 2015-03-05 17:02:25 -06:00
Jon Siwek
fa08083a92 Rename broker-related namespaces.
c++ namespace "comm" -> bro_broker
script module "Comm" -> BrokerComm
script module "Store" -> BrokerStore
2015-03-05 16:20:51 -06:00
Robin Sommer
dfc88094ab Merge remote-tracking branch 'origin/topic/jsiwek/broker'
* origin/topic/jsiwek/broker: (34 commits)
  Update broker submodule.
  Update broker submodule.
  broker integration: add missing baselines for doc tests
  broker integration: add prof.log statistics
  broker integration: add high-level usage documentation
  broker integration: add API documentation (broxygen/doxygen)
  broker integration: fix memory leak, add leak tests
  Update broker submodule.
  Improve comm tests.
  Fix gcc compile warnings.
  broker integration: fix unit tests to work when broker is not enabled.
  Add --enable-c++11 configure flag.
  broker integration: add (un)publish/(un)advertise functions
  broker integration: add knobs to set auto publish/advertise behavior
  broker integration: move listen port for unit tests to a btest variable
  broker integration: add events for incoming connection status updates
  broker integration: adapt to change in expiration_time
  Update coverage unit test baselines.
  broker integration: add Comm::enable function
  broker integration: process debug/diagnostic reports from broker
  ...

Conflicts:
	cmake
	testing/btest/Baseline/plugins.hooks/output
2015-03-02 17:10:15 -08:00
Jon Siwek
b06d82cced broker integration: add API documentation (broxygen/doxygen)
Also changed asynchronous data store query code a bit; trying to make
memory management and handling of corner cases a bit clearer (former
maybe could still be better, but I need to lookup queries by memory
address to associate response cookies to them, and so wrapping pointers
kind of just gets in the way).
2015-02-17 10:50:57 -06:00
Jon Siwek
e95116ba85 Merge branch 'master' into topic/jsiwek/broker 2015-02-16 10:00:17 -06:00
Jon Siwek
ebc9407a2b broker integration: add knobs to set auto publish/advertise behavior 2015-02-09 16:26:31 -06:00
Robin Sommer
23b9705a7b Fixing analyzer tag types for some Files::* functions. 2015-02-08 18:23:22 -08:00
Jon Siwek
6b115c6999 Merge branch 'master' into topic/jsiwek/broker 2015-02-02 11:45:21 -06:00
Jon Siwek
9875f5d3eb broker integration: add distributed data store api
But haven't done the full gamut of testing on it yet.
2015-01-30 14:39:16 -06:00
Jon Siwek
d2ea87735a broker integration: add bifs to inspect/manipulate broker data
i.e. script-layer functions to convert between bro values and broker
values; mostly for use w/ Bro's data store interface (coming soon).
2015-01-29 10:42:48 -06:00
Jon Siwek
23f04835c6 Deprecate split* family of BIFs.
These functions are now deprecated in favor of alternative versions that
return a vector of strings rather than a table of strings.

Deprecated functions:

- split: use split_string instead.
- split1: use split_string1 instead.
- split_all: use split_string_all instead.
- split_n: use split_string_n instead.
- cat_string_array: see join_string_vec instead.
- cat_string_array_n: see join_string_vec instead.
- join_string_array: see join_string_vec instead.
- sort_string_array: use sort instead instead.
- find_ip_addresses: use extract_ip_addresses instead.

Changed functions:

- has_valid_octets: uses a string_vec parameter instead of string_array.

Addresses BIT-924, BIT-757.
2015-01-21 15:34:42 -06:00
Jon Siwek
7e563b7275 broker integration: add remote events 2015-01-15 15:45:08 -06:00
Jon Siwek
1e462481dc broker integration: add remote printing 2015-01-14 13:28:34 -06:00
Jon Siwek
0daa954ddb broker integration: add remote connection status events. 2015-01-14 10:40:11 -06:00
Daniel Thayer
b5e9433b04 Improve documentation of the Intelligence Framework
Added some missing information and rearranged a few sentences so the order
makes more sense.
2015-01-07 00:01:35 -06:00
Jon Siwek
cbbe7b52dc Review/fix/change file reassembly functionality.
- Re-arrange how some fa_file fields (e.g. source, connection info, mime
  type) get updated/set for consistency.

- Add more robust mechanisms for flushing the reassembly buffer.
  The goal being to report all gaps and deliveries to file analyzers
  regardless of the state of the reassembly buffer at the time it has to
  be flushed.
2014-12-16 14:05:15 -06:00
Christian Struck
df12384758 [ADD] base pacf framework and shunt script.
It seems that there is a bug where things
are loaded in the wrong way.
2014-12-01 10:16:38 -08:00
Christian Struck
d80cc9ea10 [ADD] reworked code to new design suggested by seth.
The openflow framework now supports multiple
controllers. The design now looks a bit object
oriented and a new() function creates a controller
record.

Moved the JSON script from the JSON namespace into a
openflow specific OpenflowJSON namespace
2014-11-13 16:49:50 -08:00
Christian Struck
fef8476690 [FIX] small codestyle changes 2014-11-10 16:49:20 -08:00
Christian Struck
8e2c269c2e [ADD] status function to get flows from switch, error handling.
Added a function to receive status from the switch through the
openflow controller. (not yet implemented anywhere)

The flow_mod and flow_stats function now have default values which
report that the function is not implemented when they're called but
no plugin has registered its functions for them.
2014-11-06 15:53:09 -08:00
Seth Hall
e879aa78f5 Merge remote-tracking branch 'origin/topic/seth/mime-updates' into topic/seth/files-reassembly-and-mime-updates
Conflicts:
	scripts/base/init-bare.bro
	testing/btest/Baseline/scripts.policy.misc.dump-events/all-events-no-args.log
	testing/btest/Baseline/scripts.policy.misc.dump-events/all-events.log
2014-11-05 11:42:34 -05:00
Seth Hall
842dfd8b4a Merge remote-tracking branch 'origin/topic/seth/files-tracking' into topic/seth/files-reassembly-and-mime-updates
Conflicts:
	testing/btest/Baseline/scripts.base.frameworks.file-analysis.http.multipart/out
	testing/btest/Baseline/scripts.policy.misc.dump-events/all-events.log
2014-11-05 11:40:26 -05:00
Seth Hall
7ee34981aa Improve TAR file detection and other small changes.
- Remove all of the x-c detections.  Nearly all false
    positives.

  - Remove the back up TAR detections.  Not very helpful.

  - Remove one of the x-elc detections that was too loose
    and caused many false positives.
2014-11-05 11:31:48 -05:00
Christian Struck
c705375537 [ADD] made code event based, changed code style
the openflow framework does now use events to signal
the success or failure of openflow commands, further
the reporter framework is used to log errors.

added bro unique cookie, so the framework can recognize
which flows it installed and which not.

documented all of the code.

the code style should now me more like the rest of the
bro code.
2014-10-30 18:12:55 -07:00
Christian Struck
bf6dc12be4 [ADD] the possibility to remove flows and refactored the flow_mod function to fit the new capabilities. Also started to comment more of the code 2014-10-23 18:20:40 -07:00
Christian Struck
6c2a8cdff4 Seth's *any type* to JSON converter, slightly changed 2014-10-20 17:13:01 -07:00
Christian Struck
676207e968 Small implementation of the RYU restAPI functionality to add flows. 2014-10-20 17:10:49 -07:00
Christian Struck
d426f36ebe Small openflow api, that provides functionality to add flows. 2014-10-20 17:09:44 -07:00
Seth Hall
d77243823f Updates for file mime type identification.
- Change to the default BOF buffer size to 3000 (was 1024).
 - Reorganized MS signatures into a separate file
 - Improved lots of the signatures and added new ones.
2014-10-08 02:12:10 -04:00
Seth Hall
80656d5294 Improves shockwave flash file signatures.
- This moves the signatures out of the libmagic imported signatures
   and into our own general.sig.

 - Expand the detection to LZMA compressed flash files.
2014-10-06 11:13:13 -04:00
Seth Hall
cafd35e746 Updates the files event api and brings file reassembly up to master. 2014-09-26 00:40:37 -04:00
Seth Hall
42b2d56279 Merge remote-tracking branch 'origin/master' into topic/seth/files-tracking
Conflicts:
	scripts/base/frameworks/files/main.bro
	src/file_analysis/File.cc
	testing/btest/Baseline/scripts.base.frameworks.file-analysis.actions.data_event/out
2014-09-23 13:05:39 -04:00
Seth Hall
8d283db63b Adds a "node" field to Intel::Seen struture and intel.log.
The intel framework can now indicate which node discovered a
hit on an intel item through the new "node" field in the
Intel::Seen data structure.  On clusters, this field will
contain the name of the node where the hit was seen.
2014-09-23 12:23:39 -04:00
Daniel Thayer
d226fef723 Fixed some "make doc" warnings caused by reST formatting 2014-09-16 12:44:51 -05:00
Robin Sommer
996d118d68 Fixing tests. 2014-08-13 21:33:03 -07:00
Robin Sommer
58f3a715f2 Merge branch 'topic/robin/reader-writer-plugins' of git.bro.org:bro into topic/robin/reader-writer-plugins
Conflicts:
	scripts/base/frameworks/logging/writers/dataseries.bro
2014-08-08 18:36:09 -07:00
Robin Sommer
8737eae906 Move DataSeries and ElasticSearch into plugins. 2014-08-08 18:32:21 -07:00
Robin Sommer
8031da4ee7 More polishing of some of the branche's changes. 2014-08-08 18:32:05 -07:00
Robin Sommer
ffd3d9d185 More polishing. 2014-07-31 15:08:45 -07:00
Robin Sommer
2b505b07c1 Merge remote-tracking branch 'origin/master' into topic/robin/reader-writer-plugins 2014-07-31 10:10:39 -07:00
Jon Siwek
69b1ba653d Minor adjustments to plugin code/docs.
Mostly whitespace/typos.
Moved some Plugin methods out from public access.
2014-07-30 16:48:23 -05:00
Robin Sommer
48b251abd1 Merge branch 'topic/robin/dynamic-plugins-2.3' into topic/robin/reader-writer-plugins 2014-07-22 17:27:16 -07:00
Robin Sommer
9f0bc0fdf1 Starting to implement the proposed PACF API. 2014-07-22 03:57:05 +02:00
Robin Sommer
c9524757d2 Adding Files::register_for_mime_type() to associate a file analyzer
with a MIME type.

Whenever that MIME is detected, Bro will now automatically activate
the analyzer. The interface mimics how well-known ports are defined
for protocol analyzers.

This isn't actually used by any existing file analyzer (because we
don't have any yet that target a specific file format), but there's a
test making sure it works.
2014-07-21 16:31:22 +02:00
Robin Sommer
f4cbcb9b03 Converting log writers and input readers to plugins. 2014-07-20 19:17:58 +02:00
Robin Sommer
9616cd8e61 Further polishing and cleanup in preparation for merge. 2014-07-12 18:12:09 -07:00
Robin Sommer
aeb8e71e8c Merge remote-tracking branch 'origin/master' into topic/robin/dynamic-plugins-2.3
Conflicts:
	aux/bro-aux
	aux/broccoli
2014-07-10 20:11:52 -07:00
Robin Sommer
a7746afa0a Fixing DataSeries, which was using a now illegal value as default
compression level.
2014-07-10 14:50:15 -07:00