Commit graph

2220 commits

Author SHA1 Message Date
Matthias Vallentin
7355a0089a Adhere to Bro coding style guidelines. 2012-12-21 17:17:58 -08:00
Matthias Vallentin
ff8184242a Enhance ssl.log with information from notary.
This commit brings enhances each log line with the data from the notary when
available. The added fields include:

  - notary.first_seen
  - notary.last_seen
  - notary.times_seen
  - notary.valid

The semantics of these fields map 1-to-1 to the corresponding fields in DNS TXT
lookups from the notary. The implementation of this feature required a bit
plumbing: when Bro finishes the analysis, the log record is copied into table
indexed by connection ID where it remains until either Bro terminates or the
answer of the notary arrives. The script accummulates requests for a given
digest into a "waitlist," to avoid multiple redundant lookups for high-profile
websites who receive a large chunk of traffic. When a DNS reply arrives
asynchronously, the when handler clears the waitlist and assigns the
information to all records in the buffered.

The script also adds Each log entry into a double-ended queue to make sure the
records arrive on disk in the same way Bro sees them. Each reply also triggers
a sweep through this deque which flushes the buffer up to the first outstanding
reply.

Here is an example from the public M57 trace from 2009:

  % bro-cut ts id.orig_h id.resp_h server_name notary.first_seen notary.last_seen notary.times_seen notary.valid < ssl.log
  1258562650.121682 192.168.1.104 208.97.132.223  mail.m57.biz  - - - -
  1258535660.267128 192.168.1.104 65.55.184.16  - - - - -
  1258561662.604948 192.168.1.105 66.235.128.158  - - - - -
  1258561885.571010 192.168.1.105 65.55.184.155 www.update.microsoft.com  - - - -
  1258563578.455331 192.168.1.103 208.97.132.223  - - - - -
  1258563716.527681 192.168.1.103 96.6.248.124  - - - - -
  1258563884.667153 192.168.1.103 66.235.139.152  - - - - -
  1258564818.755676 192.168.1.103 12.41.118.177 - - - - -
  1258564821.637874 192.168.1.103 12.41.118.177 - - - - -
  1258564821.637871 192.168.1.103 12.41.118.177 - - - - -
  1258564821.637876 192.168.1.103 12.41.118.177 - - - - -
  1258564821.638126 192.168.1.103 12.41.118.177 - - - - -
  1258562467.525034 192.168.1.104 208.97.132.223  mail.m57.biz  15392 15695 301 F
  1258563063.965975 192.168.1.104 63.245.209.105  aus2.mozilla.org  - - - -
  1258563064.091396 192.168.1.104 63.245.209.91 addons.mozilla.org  - - - -
  1258563329.202273 192.168.1.103 208.97.132.223  - 15392 15695 301 F
  1258563712.945933 192.168.1.103 65.55.16.121  - - - - -
  1258563714.044500 192.168.1.103 65.54.186.79  - - - - -
  1258563716.146680 192.168.1.103 96.6.248.124  - - - - -
  1258563737.432312 192.168.1.103 96.6.245.186  - - - - -
  1258563716.526933 192.168.1.103 96.6.245.186  - - - - -
  1258563716.527430 192.168.1.103 96.6.245.186  - - - - -
  1258563716.527179 192.168.1.103 96.6.245.186  - - - - -
  1258563716.527683 192.168.1.103 96.6.245.186  - - - - -
  1258563716.527432 192.168.1.103 96.6.245.186  - - - - -
  1258563751.178683 192.168.1.103 66.235.139.152  - - - - -
  1258563751.171938 192.168.1.103 65.54.234.75  - - - - -
  1258563751.182433 192.168.1.103 65.242.27.35  - - - - -
  1258563883.414188 192.168.1.103 65.55.16.121  - - - - -
  1258563884.702380 192.168.1.103 65.242.27.35  - - - - -
  1258563885.678766 192.168.1.103 65.54.186.79  - - - - -
  1258563886.124987 192.168.1.103 65.54.186.79  - - - - -
  1258564027.877525 192.168.1.103 65.54.234.75  - - - - -
  1258564688.206859 192.168.1.103 65.54.186.107 - - - - -
  1258567162.001225 192.168.1.105 208.97.132.223  mail.m57.biz  - - - -
  1258568040.512840 192.168.1.103 208.97.132.223  - - - - -
  1258564688.577376 192.168.1.103 207.46.120.170  - - - - -
  1258564723.029005 192.168.1.103 65.54.186.107 - - - - -
  1258564723.784032 192.168.1.103 65.55.194.249 - - - - -
  1258564748.521756 192.168.1.103 65.54.186.107 - - - - -
  1258564817.601152 192.168.1.103 12.41.118.177 - - - - -
  1258565684.353653 192.168.1.105 208.97.132.223  mail.m57.biz  15392 15695 301 F
  1258565710.188691 192.168.1.105 74.125.155.109  pop.gmail.com - - - -
  1258566061.103696 192.168.1.103 208.97.132.223  - 15392 15695 301 F
  1258566893.914987 192.168.1.102 208.97.132.223  - 15392 15695 301 F
2012-12-21 17:03:39 -08:00
Robin Sommer
da90976170 Merge remote-tracking branch 'origin/topic/matthias/opaque'
* origin/topic/matthias/opaque:
  Add new unit test for opaque serialization.
  Migrate entropy testing to opaque.
  C++ify RandTest.*
  Fix a hard-to-spot bug.
  Use more descriptive error message.
  Fix the fix :-/.
  Fix initialization of hash values.
  Be clearer about delegation.
  Implement serialization of opaque types.
  Update hash BiF documentation.
  Migrate free SHA* functions to SHA*Val::digest().
  Add missing type name that caused failing tests.
  Update base scripts and unit tests.
  Simplify hash function BiFs.
  Add support for opaque hash values.
  Adapt BiF & Bro parser to handle opaque types.
  More lexer/parser work.
  Implement equivalence relation for opaque types.
  Support basic serialization of opaque.
  Add opaque type to lexer, parser, and BroType.

Closes #925

Conflicts:
	aux/broccoli
2012-12-20 16:30:22 -08:00
Seth Hall
9c00ef3ccd Doing some code reorganization and small changes to hopefully fix a memory issue. 2012-12-18 12:22:28 -05:00
Seth Hall
f31de38c9b Bug fix. 2012-12-18 01:44:46 -05:00
Seth Hall
69b7ce12d2 API updates for metrics framework.
- Removed default logging.  Now a function is available for the new
  $period_finished filter field to get the same behavior for logging
  named Metrics::write_log.

- Added index rollups for getting multiple metrics result values
  as the same time.
2012-12-18 01:08:59 -05:00
Seth Hall
69030fdff3 Merge remote-tracking branch 'origin/master' into topic/seth/metrics-merge 2012-12-17 10:00:34 -05:00
Robin Sommer
ea6b62f586 Merge remote-tracking branch 'origin/topic/jsiwek/const'
* origin/topic/jsiwek/const:
  Make const variables actually constant.  Addresses #922.

Closes #922.
2012-12-14 14:56:37 -08:00
Robin Sommer
e4353fb96b Merge remote-tracking branch 'origin/fastpath'
* origin/fastpath:
  Fix memory leak in ascii input reader.
  Improvements for the "bad checksums" detector to make it detect bad TCP checksums.
  Improved file name extraction for SMTP when file name is included in Content-Type header.

Small tweak to "bad checksum" script to avoid potential division by
zeros.
2012-12-14 14:34:51 -08:00
Jon Siwek
290c2a0b4d Make const variables actually constant. Addresses #922.
Both local and global variables declared with "const" could be modified,
but now expressions that would modify them should generate an error
message at parse-time.
2012-12-13 15:05:29 -06:00
Bernhard Amann
12753f31ae Merge remote-tracking branch 'origin/master' into topic/bernhard/input-logging-commmon-functions 2012-12-13 11:22:58 -08:00
Seth Hall
3c27267223 Improvements for the "bad checksums" detector to make it detect bad TCP checksums. 2012-12-13 11:09:41 -05:00
Seth Hall
0cf98ac325 Improved file name extraction for SMTP when file name is included in Content-Type header. 2012-12-13 10:27:08 -05:00
Matthias Vallentin
816965f3c7 Merge remote-tracking branch 'origin/master' into topic/matthias/opaque 2012-12-11 16:32:01 -08:00
Matthias Vallentin
30bab14dbf Update base scripts and unit tests. 2012-12-11 16:26:17 -08:00
Robin Sommer
b867333c2e Merge remote-tracking branch 'origin/topic/jsiwek/gtp'
* origin/topic/jsiwek/gtp:
  Change binpac exceptions in AYIYA/GTP analyzers to do protocol_violation
  Add GTP tunnel analyzer memory leak unit test.
  Add GPRS Tunnelling Protocol (GTPv1) decapsulation.

Closes #690.
2012-12-10 14:48:18 -08:00
Seth Hall
dda36672ac Merge remote-tracking branch 'origin/master' into topic/seth/metrics-merge 2012-12-06 11:07:35 -05:00
Robin Sommer
57510464a1 Adapting the HTTP request line parsing to only accept methods
consisting of letters [A-Za-z].

I had some bogus HTTP sessions now with the test-suite that reported
data as HTTP because it started with "<!... ". Requiring letters seems
a reasonable constraint.
2012-12-05 16:56:54 -08:00
Robin Sommer
177c014cb7 Merge remote-tracking branch 'vlad/topic/vladg/http-verbs'
* vlad/topic/vladg/http-verbs:
  A test for HTTP methods, including some horribly illegal requests.
  Remove hardcoded HTTP verbs from the analyzer (#741)

I added a "bad_HTTP_request" weird for HTTP request lines that don't
have more than a single word.

Closes #741.
2012-12-05 15:27:42 -08:00
Seth Hall
3af4517e2a Adding an $end time for result values to measure the length of time a measurement took. 2012-12-04 11:04:01 -05:00
Seth Hall
d61d175a04 Merge remote-tracking branch 'origin/master' into topic/seth/metrics-merge 2012-12-04 00:17:43 -05:00
Seth Hall
3ca0333294 Fix to checking metrics thresholds at the end of the break interval ($every field). 2012-12-04 00:15:19 -05:00
Robin Sommer
63d43e6545 Renaming ASCII writer filter option 'only_single_header_row' to 'tsv'.
Also clarifying usage.

Closes #912.
2012-12-03 14:40:38 -08:00
Bernhard Amann
9c09dee294 and adapt to AsciiInputOutput - seems to work... 2012-12-03 14:14:40 -08:00
Bernhard Amann
0a59d0d4db Merge branch 'topic/bernhard/input-logging-commmon-functions' into topic/bernhard/sqlite 2012-12-03 13:46:58 -08:00
Bernhard Amann
9b2265877d and factor stuff out the input framework too. 2012-12-03 13:41:19 -08:00
Bernhard Amann
501328d61a factor out ascii input/output.
First step - factored out everything the logging classes
use ( so only output ).

Moved the script-level configuration to logging/main,
and made the individual writers just refer to it -
no idea if this is good design. It works. But I am happy
about opinions :)

Next step - add support for input...
2012-12-03 12:59:11 -08:00
Seth Hall
f956554c74 Slightly fix up file name extraction from Content-Disposition headers. 2012-12-03 11:57:00 -05:00
Vlad Grigorescu
e98343b562 Remove hardcoded HTTP verbs from the analyzer (#741) 2012-11-30 20:08:20 -05:00
Jon Siwek
cc8f20c104 Merge branch 'master' into topic/jsiwek/gtp 2012-11-29 16:11:27 -06:00
Seth Hall
f1b7ca62ee Actually fix the problem I just tried to fix a minute ago. 2012-11-28 15:58:29 -05:00
Seth Hall
92285a9711 Fix a race condition when multiple workers report intermediate indexes simultaneously. 2012-11-28 15:52:41 -05:00
Seth Hall
2add60b4b1 A function wasn't returning a value like it should be. 2012-11-28 15:22:45 -05:00
Seth Hall
956c23eb66 Merge remote-tracking branch 'origin/master' into topic/seth/metrics-merge 2012-11-28 14:57:42 -05:00
Bernhard Amann
2d7ffd8269 Merge remote-tracking branch 'origin/master' into topic/bernhard/sqlite 2012-11-26 20:46:27 -08:00
Seth Hall
6bdcdcecf9 Fixed a problem with metrics aggregation on clusters (thanks Jon!). 2012-11-26 16:17:35 -05:00
Seth Hall
c98301e51f Fixed a DNS attribute issue (reported by Matt Thompson). 2012-11-26 15:58:25 -05:00
Robin Sommer
a5e237f50c The ASCII writer now supports a filter config option
'only_single_header_row' that turns the output into CSV format.

In that mode all meta data is skipped except for a single header line
with the fields names. Example:

    local my_filter: Log::Filter = [$name = "my-filter", $writer = Log::WRITER_ASCII, $config = table(["only_single_header_row"] = "T")];

Contributed by Carsten Langer.
2012-11-23 19:38:53 -08:00
Seth Hall
3546d93f36 Merging master. 2012-11-21 12:18:03 -05:00
Seth Hall
08538211e1 Some test updates. 2012-11-20 02:08:49 -05:00
Seth Hall
20fdd36a44 Updated the SQL injection detection script to make it include samples in notice emails. 2012-11-20 01:02:23 -05:00
Seth Hall
95b12262e4 More cleanup and fixed to the metrics framework. 2012-11-19 23:43:15 -05:00
Seth Hall
5b81cfe7e2 Implemented a nearly generic Queue in scriptland. 2012-11-19 23:42:19 -05:00
Seth Hall
e99e090b85 Merge remote-tracking branch 'origin/master' into topic/seth/metrics-merge 2012-11-16 02:49:36 -05:00
Seth Hall
d9195076b1 Metrics framework checkpoint.
- New scan.bro merged in and reworked a bit.

 - Updated metrics API.  Now possible to calculate much more.
2012-11-16 02:37:52 -05:00
Jon Siwek
e0805498c6 Fix some warnings from sphinx when building docs. 2012-11-15 16:40:18 -06:00
Robin Sommer
a40b00d4ab Merge branch 'topic/robin/intel-framework-merge'
* topic/robin/intel-framework-merge: (22 commits)
  Fixing tests after intel-framework merge.
  Extracting URLs from message bodies over SMTP and sending them to Intel framework.
  Small comment updates in the Intel framework CIF support.
  Intelligence framework documentation first draft.
  Only the manager tries to read files with the input framework now.
  Initial support for Bro's Intel framework with the Collective Intelligence Framework.
  Initial API for Intel framework is complete.
  Fixed an issue with cluster data distribution.
  Updating some intel framework test baselines.
  Reworked cluster intelligence data distribution mechanism and fixed tests.
  Lots more intelligence checking in SMTP traffic.
  Added intelligence check for "Received" path checking and a bit of reshuffling.
  Added sources to the intel log.
  Fixing a problem with intel distribution on clusters.
  Updated intel framework test to include matching.
  Restructuring the scripts that feed data into the intel framework slightly.
  One test for cluster transparency of the intel framework.
  Fixed a cluster support bug.
  Intelligence framework checkpoint
  Major updates to fix the Intel framework API.
  ...

Closes #914.
2012-11-05 16:00:38 -08:00
Robin Sommer
4e12813445 Fixing tests after modbus merge. 2012-11-05 15:58:38 -08:00
Robin Sommer
86ce564107 Merge remote-tracking branch 'remotes/origin/topic/seth/modbus-merge'
* remotes/origin/topic/seth/modbus-merge:
  Small modbus documentation update and tiny refactoring.
  Final touches to modbus analyzer for now.
  Major revisions to Modbus analyzer support (not quite done yet).
  put some make-up on Modbus analyser
  Modbus analyser, added support: FC=20,21
  Modbus analyzer,added support: FC=1,2,15,24
  Modbus analyzer, current support: FC=3,4,5,6,7,16,22,23

Closes #915.
2012-11-05 15:26:57 -08:00
Robin Sommer
a6216969e6 Merge remote-tracking branch 'origin/fastpath'
* origin/fastpath:
  Script in base for detecting cases of checksum offloading.
2012-11-05 13:59:30 -08:00