Commit graph

6849 commits

Author SHA1 Message Date
Robin Sommer
1289790f13 Merge remote-tracking branch 'origin/topic/bernhard/file-analysis-x509'
* origin/topic/bernhard/file-analysis-x509:
  x509 version always has to be incremented. Baselines already committed.
  Fix x509 certificate Version (the +1 was missing...).
  Add policy script to suppress non host-certificate logging in x509.log
  Forgot the preamble for the new leak test
  (hopefully) last change -> return real opaque vec instead of any_vec
  Fix dump-events - it cannot be used with ssl anymore, because openssl does not give the same string results in all versions.
  Finishing touches of the x509 file analyzer.
  Revert change to only log certificates once per hour.
  Change x509 log - now certificates are only logged once per hour.
  Fix circular reference problem and a few other small things.
  X509 file analyzer nearly done. Verification and most other policy scripts work fine now.
  Add verify functionality, including the ability to get the validated chain. This means that it is now possible to get information about the root-certificates that were used to secure a connection.
  Second try on the event interface.
  Backport crash fix that made it into master with the x509_extension backport from here.
  Make x509 certificates an opaque type
  rip out x509 code from ssl analyzer. Note that since at the moment the file analyzer does not yet re-populate the info record that means quite a lot of information is simply not available.
  parse out extension. One event for general extensions (just returns the openssl-parsed string-value), one event for basicconstraints (is a certificate a CA or not) and one event for subject-alternative-names (only DNS parts).
  Very basic file-analyzer for x509 certificates. Mostly ripped from the ssl-analyzer and the topic/bernhard/x509 branch.

BIT-1150 #merged
2014-03-30 22:16:36 +02:00
Robin Sommer
27b661cc24 Merge remote-tracking branch 'origin/topic/jsiwek/improve-type-checks'
* origin/topic/jsiwek/improve-type-checks:
  Add unit tests covering vector/set/table ctors/inits.
  Fix parsing of "local" named table constructors.
  Improve type checking of records, addresses BIT-1159.

BIT-1159 #merged
2014-03-30 20:28:07 +02:00
Robin Sommer
1292f1957d Updating submodule(s).
[nomail]
2014-03-30 20:22:23 +02:00
Robin Sommer
1b2bf6e1b3 Merge remote-tracking branch 'origin/topic/dnthayer/ticket1160'
* origin/topic/dnthayer/ticket1160:
  More improvements to install/setup documentation
  Improve documentation of Bro clusters

BIT-1160 #merged
2014-03-30 20:21:21 +02:00
Robin Sommer
806851a21e Don't include locations into serialization when cloning values.
This should save some memory, see BIT-1161.
2014-03-30 20:20:06 +02:00
Robin Sommer
2f6146aa55 Merge remote-tracking branch 'origin/topic/jsiwek/faster-val-clone'
* origin/topic/jsiwek/faster-val-clone:
  Refactor SerializationFormat::EndWrite and ChunkedIO::Chunk mem mgmt.
  Improve SerializationFormat's write buffer growth strategy.

BIT-1161 #merged
2014-03-30 20:13:26 +02:00
Robin Sommer
a0d389bf4b Merge remote-tracking branch 'origin/topic/jsiwek/parse-only'
* origin/topic/jsiwek/parse-only:
  Add --parse-only option to exit after parsing scripts.

BIT-1169 #merged
2014-03-30 20:01:12 +02:00
Robin Sommer
c35b5f80c2 Merge remote-tracking branch 'origin/topic/bernhard/sumstats-read-expire'
* origin/topic/bernhard/sumstats-read-expire:
  Change all &create_expire attributes to &read_expire in the cluster part of the sumstats framework.

BIT-1170 #merged
2014-03-30 19:58:33 +02:00
Robin Sommer
6cac21a9bf Merge remote-tracking branch 'origin/fastpath'
* origin/fastpath:
  Update instructions on how to build Bro docs
2014-03-30 19:55:39 +02:00
Robin Sommer
53dd2bb62d Updating CHANGES and VERSION. 2014-03-30 19:54:45 +02:00
Bernhard Amann
254dd85bff Change all &create_expire attributes to &read_expire in the cluster
part of the sumstats framework.

This seems to fix a few rare problems.
2014-03-28 10:57:45 -07:00
Seth Hall
9d6c8769ea Quick fix to the ElasticSearch writer. 2014-03-28 08:37:37 -04:00
Jon Siwek
06b6dc8447 Add --parse-only option to exit after parsing scripts.
May be useful for syntax-checking tools.
2014-03-27 16:07:52 -05:00
Daniel Thayer
11d3685f88 Update instructions on how to build Bro docs
Also added a note in the main docs about building the Bro docs.
2014-03-25 13:55:20 -05:00
Jon Siwek
8dad5026fd File type detection changes and fix https.log {orig,resp}_fuids fields.
- Removed "binary" and "octet-stream" mime type detections. They don't
  provide any more information than an uninitialized mime_type field
  which implicitly means no magic signature matches and so the media
  type is unknown to Bro.

- Slight change to "text/plain" signature.  It's still not the most
  accurate, which is reflected in its -20 strength value.

- The logic for adding file ids to {orig,resp}_fuids fields of
  the http.log incorrectly depended on the state of
  {orig,resp}_mime_types fields, so sometimes not all file ids
  associated w/ the session were logged.
2014-03-25 12:44:11 -05:00
Jon Siwek
bebedcd873 Merge branch 'master' into topic/jsiwek/file-signatures
Conflicts:
	scripts/policy/frameworks/files/detect-MHR.bro
2014-03-24 14:35:37 -05:00
Jon Siwek
9a8226935d Add unit tests covering vector/set/table ctors/inits. 2014-03-21 09:22:03 -05:00
Jon Siwek
bf3c3887fd Fix parsing of "local" named table constructors. 2014-03-20 16:47:20 -05:00
Jon Siwek
b1fd161274 Improve type checking of records, addresses BIT-1159. 2014-03-20 13:54:26 -05:00
Bernhard Amann
ae165fb262 x509 version always has to be incremented. Baselines already committed. 2014-03-19 21:50:02 -07:00
Bernhard Amann
a185631e1e Fix x509 certificate Version (the +1 was missing...).
This was wrong all along in the ssl protocol parser and no one ever
noticed :)

(And I missed it in the new logfiles until now *cough*)
2014-03-19 21:46:19 -07:00
Bernhard Amann
b32c7c7a88 Add policy script to suppress non host-certificate logging in x509.log
Addresses BIT-1150
2014-03-19 21:32:01 -07:00
Daniel Thayer
80fe5874a5 More improvements to install/setup documentation
Addresses BIT-1160
2014-03-19 16:56:29 -05:00
Seth Hall
0f3ed1a553 Merge remote-tracking branch 'origin/topic/jsiwek/faster-mhr'
* origin/topic/jsiwek/faster-mhr:
  Improve performance of MHR script, addresses BIT-1139.
2014-03-19 17:22:23 -04:00
Seth Hall
63751665ce Merge remote-tracking branch 'origin/topic/seth/sumstats-standalone-fix'
* origin/topic/seth/sumstats-standalone-fix:
  Now the standalone sumstats works even if Bro is shutting down by blocking.
  Change the standalone sumstats mode to work incrementally.
2014-03-19 14:51:05 -04:00
Daniel Thayer
2e8d0945a4 Improve documentation of Bro clusters
Renamed the bro cluster doc to better indicate its purpose (it provides
a high-level overview rather than detailed configuration instructions).
Moved the location of the bro cluster doc in the index so that it makes
more sense (it is an introductory section, not a section about using bro).

Added links in the quick start guide and the bro cluster doc so that
readers can more easily locate more detailed information on configuring
a bro cluster.

Addresses BIT-1160
2014-03-18 16:29:27 -05:00
Jon Siwek
70131b5c84 Refactor SerializationFormat::EndWrite and ChunkedIO::Chunk mem mgmt.
SerializationFormat::EndWrite now transfers ownership of the buffer
to the caller instead of doing a memcpy.

ChunkedIO::Chunk is no longer a POD type, hopefully the ctor/dtor
make it easier to manage its associated memory.  It also now
tracks how to deallocate its buffer (i.e. delete vs. free).
2014-03-18 14:42:38 -05:00
Seth Hall
a07a40dc86 Now the standalone sumstats works even if Bro is shutting down by blocking. 2014-03-17 16:45:36 -04:00
Seth Hall
af36915186 Change the standalone sumstats mode to work incrementally. 2014-03-17 16:06:14 -04:00
Robin Sommer
bf6f21041c Updating submodule(s).
[nomail]
2014-03-17 09:18:01 -07:00
Jon Siwek
07a4a8d147 Merge branch 'master' into topic/jsiwek/faster-val-clone 2014-03-17 11:05:08 -05:00
Robin Sommer
13081c984c Merge remote-tracking branch 'origin/fastpath'
* origin/fastpath:
  update submodules
  Fix compile errror on freebsd - defines have to be moved up due to header dependencies.
2014-03-17 08:25:24 -07:00
Bernhard Amann
66ec267b2a update submodules 2014-03-17 00:11:27 -07:00
Bernhard Amann
636d25e526 Fix compile errror on freebsd - defines have to be moved up due to
header dependencies.
2014-03-16 22:22:44 -07:00
Robin Sommer
908b574c18 Updating submodule(s).
[nomail]
2014-03-16 07:47:06 -07:00
Jon Siwek
6595c7c75b Improve SerializationFormat's write buffer growth strategy. 2014-03-14 17:26:38 -05:00
Robin Sommer
8061a34b3e Merge remote-tracking branch 'origin/fastpath'
* origin/fastpath:
  Derive results of DNS lookups from from input when in BRO_DNS_FAKE mode.

BIT-1134 #closed
2014-03-14 10:24:26 -07:00
Jon Siwek
dbe5dfb3c3 Derive results of DNS lookups from from input when in BRO_DNS_FAKE mode.
Addresses BIT-1134.
2014-03-14 12:01:09 -05:00
Robin Sommer
e8339d5c63 Merge remote-tracking branch 'origin/topic/bernhard/file-analysis-x509'
* origin/topic/bernhard/file-analysis-x509:
  Forgot the preamble for the new leak test
  (hopefully) last change -> return real opaque vec instead of any_vec
  Fix dump-events - it cannot be used with ssl anymore, because openssl does not give the same string results in all versions.
  Finishing touches of the x509 file analyzer.
  Revert change to only log certificates once per hour.
  Change x509 log - now certificates are only logged once per hour.
  Fix circular reference problem and a few other small things.
  X509 file analyzer nearly done. Verification and most other policy scripts work fine now.
  Add verify functionality, including the ability to get the validated chain. This means that it is now possible to get information about the root-certificates that were used to secure a connection.
  Second try on the event interface.
  Backport crash fix that made it into master with the x509_extension backport from here.
  Make x509 certificates an opaque type
  rip out x509 code from ssl analyzer. Note that since at the moment the file analyzer does not yet re-populate the info record that means quite a lot of information is simply not available.
  parse out extension. One event for general extensions (just returns the openssl-parsed string-value), one event for basicconstraints (is a certificate a CA or not) and one event for subject-alternative-names (only DNS parts).
  Very basic file-analyzer for x509 certificates. Mostly ripped from the ssl-analyzer and the topic/bernhard/x509 branch.
2014-03-14 09:53:07 -07:00
Robin Sommer
8b241947d6 Fixing a few cases of undefined behaviour introduced by recent
formatter work.

Thanks, Coverity!
2014-03-14 08:45:36 -07:00
Robin Sommer
17f9d0a47d Fixing compiler error.
Hopefully ...
2014-03-13 18:02:41 -07:00
Robin Sommer
00755f1e40 Fixing (very unlikely) double delete in HTTP analyzer when decapsulating
CONNECTs.

BIT-1149 #closed
2014-03-13 16:51:21 -07:00
Robin Sommer
ff261ea626 Merge remote-tracking branch 'origin/topic/seth/json-formatter'
* origin/topic/seth/json-formatter:
  Updating a couple of tests.
  Expanded support for modifying the timestamp format in the JSON formatter.
  Ascii input reader now supports all config options per-input stream.
  Added an option to the JSON formatter to use ISO 8601 for timestamps.
  Refactored formatters and updated the the writers a bit.

Includes some minor bugfixes and cleanup at various places, including
in old code.
2014-03-13 16:50:56 -07:00
Bernhard Amann
285de1390a Forgot the preamble for the new leak test 2014-03-13 16:20:49 -07:00
Bernhard Amann
d42d9bbc3f (hopefully) last change -> return real opaque vec instead of any_vec 2014-03-13 16:10:59 -07:00
Bernhard Amann
3f52eeacda Fix dump-events - it cannot be used with ssl anymore, because openssl
does not give the same string results in all versions.

Add leak test for x509 verify and fix small leak (type).
2014-03-13 15:41:57 -07:00
Bernhard Amann
4da0718511 Finishing touches of the x509 file analyzer.
Mostly baseline updates and new tests.

addresses BIT-953, BIT-760, BIT-1150
2014-03-13 15:21:30 -07:00
Robin Sommer
4eb81de08e Refactoring code to reuse ApplyScheduledAnalyzers().
This is potentially changing the exact time when the
scheduled_analyzer_applied() event is executed, but that should be
fine afaict.
2014-03-13 15:00:19 -07:00
Robin Sommer
126bd298fe Merge remote-tracking branch 'origin/topic/jsiwek/dns_fake'
* origin/topic/jsiwek/dns_fake:
  Teach async DNS lookup builtin-functions about BRO_DNS_FAKE.
2014-03-13 14:27:44 -07:00
Robin Sommer
a90bf11e35 Merge remote-tracking branch 'origin/topic/jsiwek/flip-roles'
* origin/topic/jsiwek/flip-roles:
  Improve analysis of TCP SYN/SYN-ACK reversal situations.
  Fix bug in Connection::FlipRoles, addresses BIT-1148.

BIT-1148 #merged
2014-03-13 14:15:29 -07:00