Commit graph

704 commits

Author SHA1 Message Date
Seth Hall
93eca70e6b Merge remote-tracking branch 'origin/master' into topic/seth/metrics-merge 2013-04-01 14:16:46 -04:00
Seth Hall
53f9948b02 Measurement framework tests all pass now. 2013-04-01 14:16:37 -04:00
Seth Hall
6dc204b385 Checkpoint, don't try running this. It's broken all over the place. 2013-03-19 11:39:58 -04:00
Robin Sommer
d58a02aa01 Merge remote-tracking branch 'origin/topic/bernhard/base64'
* origin/topic/bernhard/base64:
  and re-enable caching of extracted certs
  and add bae64 bif tests.
  re-unify classes
  and modernize script.
  add base64-encode functionality and bif.

Closes #965.
2013-03-17 13:00:52 -07:00
Robin Sommer
788c0d547d Merge remote-tracking branch 'origin/topic/seth/software-version-updates2'
* origin/topic/seth/software-version-updates2:
  Correctly handle DNS lookups for software version ranges.
  Improvements to vulnerable software detection.
  Update software version parsing and comparison to account for a third numeric subversion.

Closes #938.
2013-03-17 12:38:25 -07:00
Seth Hall
5734f9ef71 Merge remote-tracking branch 'origin/master' into topic/seth/software-version-updates2 2013-03-14 09:21:13 -04:00
Seth Hall
8778761c07 Checkpoint 2013-03-13 22:55:03 -04:00
Bernhard Amann
457ce10e99 and re-enable caching of extracted certs
I kind of deleted the line by accident...
2013-03-13 00:34:15 -07:00
Seth Hall
09cbaa7ccc Merge remote-tracking branch 'origin/master' into topic/seth/metrics-merge
Conflicts:
	testing/btest/Baseline/coverage.default-load-baseline/canonified_loaded_scripts.log
	testing/btest/Baseline/scripts.base.frameworks.notice.cluster/manager-1.notice.log
	testing/btest/Baseline/scripts.base.frameworks.notice.suppression-cluster/manager-1.notice.log
	testing/btest/Baseline/scripts.base.protocols.ftp.gridftp/notice.log
2013-03-13 00:26:55 -04:00
Bernhard Amann
a1896fde90 Merge remote-tracking branch 'origin/master' into topic/bernhard/base64 2013-03-12 09:28:07 -07:00
Robin Sommer
f830ed3edf s/bro-ids.org/bro.org/g 2013-03-07 19:33:04 -08:00
Robin Sommer
8a6d68e00f Merge remote-tracking branch 'origin/topic/bernhard/remove-length'
Closes #955.

* origin/topic/bernhard/remove-length:
  forgot to remove the baselines for the now unnecessary bifs
  remove the byte_len and length bifs
2013-03-06 16:46:20 -08:00
Robin Sommer
1bd2f26df3 Merge remote-tracking branch 'origin/topic/seth/notice-framework-updates'
So much nicer!

Closes #954.

* origin/topic/seth/notice-framework-updates:
  Update notice framework documentation to represent the new reality.
  Complete removal of the old table based notice policy mechanism.
  Updates for the notices framework.
2013-03-06 16:45:30 -08:00
Bernhard Amann
986b346e3f remove the byte_len and length bifs 2013-03-06 13:45:42 -08:00
Bernhard Amann
cfada61672 and modernize script.
thanks Seth.
2013-03-06 13:30:13 -08:00
Seth Hall
00eeadf2f5 Complete removal of the old table based notice policy mechanism. 2013-03-06 09:44:25 -05:00
Bernhard Amann
5e8e12182a add base64-encode functionality and bif.
This allows replacing an ugly openssl-call from one of
the policy scripts. The openssl call is now replaced with
a still-but-less-ugly call to base64_encode.

I do not know if I split the Base64 classes in a "smart" way... :)
2013-03-05 16:05:07 -08:00
Seth Hall
e3856d7681 Removing a field that is no longer logged through the standard metrics log. 2013-02-27 11:25:01 -05:00
Seth Hall
9f8ba408ba Updates for the notices framework.
- Moved the Notice::notice event and Notice::policy table to both be hooks.

 - Renamed the old Notice::policy to Notice::policy_table and documented it as deprecated.
2013-02-11 14:36:14 -05:00
Jon Siwek
26bf99c5a3 Add parsing for GTPv1 extension headers and control messages.
Added a generic gtpv1_message event generated for any GTP message type.

Added specific events for the create/update/delete PDP context
request/response messages.

Addresses #934.
2013-02-07 14:59:02 -06:00
Seth Hall
f8be65ca56 Merge remote-tracking branch 'origin/master' into topic/seth/metrics-merge 2013-02-07 10:01:13 -05:00
Seth Hall
e974950c6d Correctly handle DNS lookups for software version ranges. 2013-02-06 13:59:17 -05:00
Seth Hall
ff3ec04f32 Improvements to vulnerable software detection.
- Add a DNS based updating method.  This needs to be tested still.

 - Vulnerable version ranges are used now instead
   of only single versions.  This can deal with
   software with multiple stable major versions.
2013-02-06 13:59:12 -05:00
Seth Hall
ad9e2a143c Update software version parsing and comparison to account for a third numeric subversion.
- $addl is also compared numerically now if the value is actually numeric.
2013-02-06 13:59:05 -05:00
Seth Hall
a2556642e6 Merge remote-tracking branch 'origin/topic/matthias/notary'
* origin/topic/matthias/notary:
  Small cosmetic changes.
  Give log buffer the correct name.
  Simplify delayed logging of SSL records.
  Implement delay-token style SSL logging.
  More style tweaks: replace spaces with tabs.
  Factor notary code into separte file.
  Adhere to Bro coding style guidelines.
  Enhance ssl.log with information from notary.

Closes #928
2013-02-05 02:06:33 -05:00
Robin Sommer
52f52f60a5 Merge branch 'topic/robin/exit-after-terminate'
* topic/robin/exit-after-terminate:
  Updating submodule(s).
  Fixing exit-after-terminate when used with bare mode.
  New option exit_only_after_terminate to prevent Bro from exiting.
2013-02-04 09:44:33 -08:00
Jon Siwek
69afc4a882 Add an error for record coercions that would orphan a field.
These cases should be avoidable by fixing scripts where they occur and
they can also help catch typos that would lead to unintentional runtime
behavior.

Adding this already revealed several scripts where a field in an inlined
record was never removed after a code refactor.
2013-01-24 09:56:19 -06:00
Robin Sommer
762c034ec2 Merge remote-tracking branch 'origin/topic/bernhard/input-logging-commmon-functions'
* origin/topic/bernhard/input-logging-commmon-functions:
  add the last of Robins suggestions (separate info-struct for constructors).
  port memory leak fix from master
  harmonize function naming
  move AsciiInputOutput over to threading
  and thinking about it, ascii-io doesn't need the separator
  change constructors
  and factor stuff out the input framework too.
  factor out ascii input/output.
  std::string accessors to escape_sequence functionality
  intermediate commit - it has been over a month since I touched this...

I cleaned up the AsciiInputOutput class somewhat, including renaming
it to AsciiFormatter, renaming some of its methods, and turning the
static methods into members for consistency.

Closes #929.
2013-01-23 16:51:54 -08:00
Robin Sommer
4f39470c1b New option exit_only_after_terminate to prevent Bro from exiting.
If set, the main loop won't terminate before somebody calls
terminate().

This should make input framework testing more reliable I'd hope.
2013-01-18 17:34:33 -08:00
Jon Siwek
fdd11428c1 Change reporter messages to more reliably print to stderr.
Moved this functionality to be internal instead of in the script-layer
event handlers.  The issue with the later is that bad things can happen
between the time a reporter event handler is dispatched and the time it
is executed, and if bro crashes in that time, the message may never be
seen/logged.

Addressed #930 (and revisits #836).
2013-01-18 12:46:00 -06:00
Matthias Vallentin
a635f96518 Small cosmetic changes. 2013-01-15 14:56:02 -08:00
Bernhard Amann
1b0bb5063a Merge remote-tracking branch 'origin/master' into topic/bernhard/input-logging-commmon-functions
Conflicts:
	src/input/readers/Ascii.cc
2013-01-11 09:48:53 -08:00
Seth Hall
720089c03f Add a feature to better select the value threshold will apply to in the metrics framework.
- The feature was primarily added to allow the value to be
   modified for cluster based intermediate threshold checks
   without requiring the user to write the metrics filter
   differently for cluster consideration.  It's also a nice
   way to calculate some related information to the metric
   without accidently applying thresholds to that value.

 - Fixed a few small bugs in ftp detect-bruteforcing script
   and adapted it to the new threshold value selection feature.
2013-01-05 22:37:19 -05:00
Seth Hall
bcd7fe114d Fixed an issue causing reporter messages from the metrics framework. 2013-01-05 22:27:17 -05:00
Seth Hall
9366411cf4 Fix the FTP bruteforce threshold to what it's really supposed to be. 2013-01-04 23:49:09 -05:00
Seth Hall
283f7840b3 Removing some debugging print statements I accidently left behind. 2013-01-04 23:38:10 -05:00
Seth Hall
9e1d5d87de New script to detect FTP bruteforcing.
- Created a new time utils library.
2013-01-04 23:34:57 -05:00
Seth Hall
ab7087f953 Hooks work now, enabling the hooks in the scan.bro script. 2013-01-04 21:01:49 -05:00
Seth Hall
89bc65f23c Merge remote-tracking branch 'origin/master' into topic/seth/metrics-merge 2013-01-04 20:49:37 -05:00
Seth Hall
c3a6916572 More cluster tuning for the metrics framework.
- Fixed several state maintenance issues for intermediate updates.

 - Added a new tuning variable Metrics::max_outstanding_global_views
   which limits the number of in-flight intermediate updates per
   metric filter.

 - Changed the default global view threshold percent to 20% (up from 10%)
2013-01-04 16:54:13 -05:00
Matthias Vallentin
32a0ead698 Give log buffer the correct name. 2012-12-24 23:06:56 -08:00
Matthias Vallentin
7ff15f4599 Simplify delayed logging of SSL records. 2012-12-24 22:57:49 -08:00
Matthias Vallentin
9e81342c92 Implement delay-token style SSL logging.
This commit moves the notary script into the policy directory, along with some
architectural changes: the main SSL script now has functionality to add and
remove tokens for a given record. When adding a token, the script delays the
logging until the token has been removed or until the record exceeds a maximum
delay time.

As before, the base SSL script stores all records sequentially and buffers even
non-delayed records for the sake of having an ordered log file. If this turns
out to be not so important, we can easily revert to a simpler logic.

(This is still WiP, some debuggin statements still linger.)
2012-12-22 20:30:17 -08:00
Seth Hall
50827d8df0 Implement an option to disable intermediate updates for the metrics framework.
- There are some large sites having trouble and I'm thinking it
   might be overload from intermediate updates.
2012-12-21 23:17:27 -05:00
Matthias Vallentin
8a569facd6 More style tweaks: replace spaces with tabs. 2012-12-21 18:04:19 -08:00
Matthias Vallentin
382262e286 Factor notary code into separte file.
There exists one complication: the new file notary.bro requires the definition
of the SSL::Info record, but as does main.bro. Because I did not really know
where to put the common code (it's not a constant, so ssl/const.bro does not
really fit), I put it into __load.bro__ so that it sticks out for now. If
anybody has an idea how to solve this elegantly, please let me know.
2012-12-21 17:56:31 -08:00
Matthias Vallentin
7355a0089a Adhere to Bro coding style guidelines. 2012-12-21 17:17:58 -08:00
Matthias Vallentin
ff8184242a Enhance ssl.log with information from notary.
This commit brings enhances each log line with the data from the notary when
available. The added fields include:

  - notary.first_seen
  - notary.last_seen
  - notary.times_seen
  - notary.valid

The semantics of these fields map 1-to-1 to the corresponding fields in DNS TXT
lookups from the notary. The implementation of this feature required a bit
plumbing: when Bro finishes the analysis, the log record is copied into table
indexed by connection ID where it remains until either Bro terminates or the
answer of the notary arrives. The script accummulates requests for a given
digest into a "waitlist," to avoid multiple redundant lookups for high-profile
websites who receive a large chunk of traffic. When a DNS reply arrives
asynchronously, the when handler clears the waitlist and assigns the
information to all records in the buffered.

The script also adds Each log entry into a double-ended queue to make sure the
records arrive on disk in the same way Bro sees them. Each reply also triggers
a sweep through this deque which flushes the buffer up to the first outstanding
reply.

Here is an example from the public M57 trace from 2009:

  % bro-cut ts id.orig_h id.resp_h server_name notary.first_seen notary.last_seen notary.times_seen notary.valid < ssl.log
  1258562650.121682 192.168.1.104 208.97.132.223  mail.m57.biz  - - - -
  1258535660.267128 192.168.1.104 65.55.184.16  - - - - -
  1258561662.604948 192.168.1.105 66.235.128.158  - - - - -
  1258561885.571010 192.168.1.105 65.55.184.155 www.update.microsoft.com  - - - -
  1258563578.455331 192.168.1.103 208.97.132.223  - - - - -
  1258563716.527681 192.168.1.103 96.6.248.124  - - - - -
  1258563884.667153 192.168.1.103 66.235.139.152  - - - - -
  1258564818.755676 192.168.1.103 12.41.118.177 - - - - -
  1258564821.637874 192.168.1.103 12.41.118.177 - - - - -
  1258564821.637871 192.168.1.103 12.41.118.177 - - - - -
  1258564821.637876 192.168.1.103 12.41.118.177 - - - - -
  1258564821.638126 192.168.1.103 12.41.118.177 - - - - -
  1258562467.525034 192.168.1.104 208.97.132.223  mail.m57.biz  15392 15695 301 F
  1258563063.965975 192.168.1.104 63.245.209.105  aus2.mozilla.org  - - - -
  1258563064.091396 192.168.1.104 63.245.209.91 addons.mozilla.org  - - - -
  1258563329.202273 192.168.1.103 208.97.132.223  - 15392 15695 301 F
  1258563712.945933 192.168.1.103 65.55.16.121  - - - - -
  1258563714.044500 192.168.1.103 65.54.186.79  - - - - -
  1258563716.146680 192.168.1.103 96.6.248.124  - - - - -
  1258563737.432312 192.168.1.103 96.6.245.186  - - - - -
  1258563716.526933 192.168.1.103 96.6.245.186  - - - - -
  1258563716.527430 192.168.1.103 96.6.245.186  - - - - -
  1258563716.527179 192.168.1.103 96.6.245.186  - - - - -
  1258563716.527683 192.168.1.103 96.6.245.186  - - - - -
  1258563716.527432 192.168.1.103 96.6.245.186  - - - - -
  1258563751.178683 192.168.1.103 66.235.139.152  - - - - -
  1258563751.171938 192.168.1.103 65.54.234.75  - - - - -
  1258563751.182433 192.168.1.103 65.242.27.35  - - - - -
  1258563883.414188 192.168.1.103 65.55.16.121  - - - - -
  1258563884.702380 192.168.1.103 65.242.27.35  - - - - -
  1258563885.678766 192.168.1.103 65.54.186.79  - - - - -
  1258563886.124987 192.168.1.103 65.54.186.79  - - - - -
  1258564027.877525 192.168.1.103 65.54.234.75  - - - - -
  1258564688.206859 192.168.1.103 65.54.186.107 - - - - -
  1258567162.001225 192.168.1.105 208.97.132.223  mail.m57.biz  - - - -
  1258568040.512840 192.168.1.103 208.97.132.223  - - - - -
  1258564688.577376 192.168.1.103 207.46.120.170  - - - - -
  1258564723.029005 192.168.1.103 65.54.186.107 - - - - -
  1258564723.784032 192.168.1.103 65.55.194.249 - - - - -
  1258564748.521756 192.168.1.103 65.54.186.107 - - - - -
  1258564817.601152 192.168.1.103 12.41.118.177 - - - - -
  1258565684.353653 192.168.1.105 208.97.132.223  mail.m57.biz  15392 15695 301 F
  1258565710.188691 192.168.1.105 74.125.155.109  pop.gmail.com - - - -
  1258566061.103696 192.168.1.103 208.97.132.223  - 15392 15695 301 F
  1258566893.914987 192.168.1.102 208.97.132.223  - 15392 15695 301 F
2012-12-21 17:03:39 -08:00
Robin Sommer
da90976170 Merge remote-tracking branch 'origin/topic/matthias/opaque'
* origin/topic/matthias/opaque:
  Add new unit test for opaque serialization.
  Migrate entropy testing to opaque.
  C++ify RandTest.*
  Fix a hard-to-spot bug.
  Use more descriptive error message.
  Fix the fix :-/.
  Fix initialization of hash values.
  Be clearer about delegation.
  Implement serialization of opaque types.
  Update hash BiF documentation.
  Migrate free SHA* functions to SHA*Val::digest().
  Add missing type name that caused failing tests.
  Update base scripts and unit tests.
  Simplify hash function BiFs.
  Add support for opaque hash values.
  Adapt BiF & Bro parser to handle opaque types.
  More lexer/parser work.
  Implement equivalence relation for opaque types.
  Support basic serialization of opaque.
  Add opaque type to lexer, parser, and BroType.

Closes #925

Conflicts:
	aux/broccoli
2012-12-20 16:30:22 -08:00
Jon Siwek
2f0c698ed5 Improve error for invalid use of types as values (addresses #923).
This scripting error can now generate an error message at parse-time
instead of run-time and also includes location information.
2012-12-18 14:31:39 -06:00