Commit graph

356 commits

Author SHA1 Message Date
Seth Hall
b477d2b02d Measurement framework is ready for testing.
- New, expanded API.
 - Calculations moved into plugins.
 - Scripts using measurement framework ported.
 - Updated the script-land queue implementation to make it more generic.
 -
2013-04-01 17:04:15 -04:00
Seth Hall
93eca70e6b Merge remote-tracking branch 'origin/master' into topic/seth/metrics-merge 2013-04-01 14:16:46 -04:00
Seth Hall
53f9948b02 Measurement framework tests all pass now. 2013-04-01 14:16:37 -04:00
Seth Hall
6dc204b385 Checkpoint, don't try running this. It's broken all over the place. 2013-03-19 11:39:58 -04:00
Seth Hall
5734f9ef71 Merge remote-tracking branch 'origin/master' into topic/seth/software-version-updates2 2013-03-14 09:21:13 -04:00
Seth Hall
8778761c07 Checkpoint 2013-03-13 22:55:03 -04:00
Seth Hall
09cbaa7ccc Merge remote-tracking branch 'origin/master' into topic/seth/metrics-merge
Conflicts:
	testing/btest/Baseline/coverage.default-load-baseline/canonified_loaded_scripts.log
	testing/btest/Baseline/scripts.base.frameworks.notice.cluster/manager-1.notice.log
	testing/btest/Baseline/scripts.base.frameworks.notice.suppression-cluster/manager-1.notice.log
	testing/btest/Baseline/scripts.base.protocols.ftp.gridftp/notice.log
2013-03-13 00:26:55 -04:00
Robin Sommer
f830ed3edf s/bro-ids.org/bro.org/g 2013-03-07 19:33:04 -08:00
Robin Sommer
8a6d68e00f Merge remote-tracking branch 'origin/topic/bernhard/remove-length'
Closes #955.

* origin/topic/bernhard/remove-length:
  forgot to remove the baselines for the now unnecessary bifs
  remove the byte_len and length bifs
2013-03-06 16:46:20 -08:00
Robin Sommer
1bd2f26df3 Merge remote-tracking branch 'origin/topic/seth/notice-framework-updates'
So much nicer!

Closes #954.

* origin/topic/seth/notice-framework-updates:
  Update notice framework documentation to represent the new reality.
  Complete removal of the old table based notice policy mechanism.
  Updates for the notices framework.
2013-03-06 16:45:30 -08:00
Bernhard Amann
986b346e3f remove the byte_len and length bifs 2013-03-06 13:45:42 -08:00
Seth Hall
00eeadf2f5 Complete removal of the old table based notice policy mechanism. 2013-03-06 09:44:25 -05:00
Seth Hall
e3856d7681 Removing a field that is no longer logged through the standard metrics log. 2013-02-27 11:25:01 -05:00
Seth Hall
9f8ba408ba Updates for the notices framework.
- Moved the Notice::notice event and Notice::policy table to both be hooks.

 - Renamed the old Notice::policy to Notice::policy_table and documented it as deprecated.
2013-02-11 14:36:14 -05:00
Jon Siwek
26bf99c5a3 Add parsing for GTPv1 extension headers and control messages.
Added a generic gtpv1_message event generated for any GTP message type.

Added specific events for the create/update/delete PDP context
request/response messages.

Addresses #934.
2013-02-07 14:59:02 -06:00
Seth Hall
f8be65ca56 Merge remote-tracking branch 'origin/master' into topic/seth/metrics-merge 2013-02-07 10:01:13 -05:00
Seth Hall
ad9e2a143c Update software version parsing and comparison to account for a third numeric subversion.
- $addl is also compared numerically now if the value is actually numeric.
2013-02-06 13:59:05 -05:00
Jon Siwek
69afc4a882 Add an error for record coercions that would orphan a field.
These cases should be avoidable by fixing scripts where they occur and
they can also help catch typos that would lead to unintentional runtime
behavior.

Adding this already revealed several scripts where a field in an inlined
record was never removed after a code refactor.
2013-01-24 09:56:19 -06:00
Robin Sommer
762c034ec2 Merge remote-tracking branch 'origin/topic/bernhard/input-logging-commmon-functions'
* origin/topic/bernhard/input-logging-commmon-functions:
  add the last of Robins suggestions (separate info-struct for constructors).
  port memory leak fix from master
  harmonize function naming
  move AsciiInputOutput over to threading
  and thinking about it, ascii-io doesn't need the separator
  change constructors
  and factor stuff out the input framework too.
  factor out ascii input/output.
  std::string accessors to escape_sequence functionality
  intermediate commit - it has been over a month since I touched this...

I cleaned up the AsciiInputOutput class somewhat, including renaming
it to AsciiFormatter, renaming some of its methods, and turning the
static methods into members for consistency.

Closes #929.
2013-01-23 16:51:54 -08:00
Jon Siwek
fdd11428c1 Change reporter messages to more reliably print to stderr.
Moved this functionality to be internal instead of in the script-layer
event handlers.  The issue with the later is that bad things can happen
between the time a reporter event handler is dispatched and the time it
is executed, and if bro crashes in that time, the message may never be
seen/logged.

Addressed #930 (and revisits #836).
2013-01-18 12:46:00 -06:00
Bernhard Amann
1b0bb5063a Merge remote-tracking branch 'origin/master' into topic/bernhard/input-logging-commmon-functions
Conflicts:
	src/input/readers/Ascii.cc
2013-01-11 09:48:53 -08:00
Seth Hall
720089c03f Add a feature to better select the value threshold will apply to in the metrics framework.
- The feature was primarily added to allow the value to be
   modified for cluster based intermediate threshold checks
   without requiring the user to write the metrics filter
   differently for cluster consideration.  It's also a nice
   way to calculate some related information to the metric
   without accidently applying thresholds to that value.

 - Fixed a few small bugs in ftp detect-bruteforcing script
   and adapted it to the new threshold value selection feature.
2013-01-05 22:37:19 -05:00
Seth Hall
bcd7fe114d Fixed an issue causing reporter messages from the metrics framework. 2013-01-05 22:27:17 -05:00
Seth Hall
c3a6916572 More cluster tuning for the metrics framework.
- Fixed several state maintenance issues for intermediate updates.

 - Added a new tuning variable Metrics::max_outstanding_global_views
   which limits the number of in-flight intermediate updates per
   metric filter.

 - Changed the default global view threshold percent to 20% (up from 10%)
2013-01-04 16:54:13 -05:00
Seth Hall
50827d8df0 Implement an option to disable intermediate updates for the metrics framework.
- There are some large sites having trouble and I'm thinking it
   might be overload from intermediate updates.
2012-12-21 23:17:27 -05:00
Seth Hall
9c00ef3ccd Doing some code reorganization and small changes to hopefully fix a memory issue. 2012-12-18 12:22:28 -05:00
Seth Hall
f31de38c9b Bug fix. 2012-12-18 01:44:46 -05:00
Seth Hall
69b7ce12d2 API updates for metrics framework.
- Removed default logging.  Now a function is available for the new
  $period_finished filter field to get the same behavior for logging
  named Metrics::write_log.

- Added index rollups for getting multiple metrics result values
  as the same time.
2012-12-18 01:08:59 -05:00
Seth Hall
69030fdff3 Merge remote-tracking branch 'origin/master' into topic/seth/metrics-merge 2012-12-17 10:00:34 -05:00
Jon Siwek
290c2a0b4d Make const variables actually constant. Addresses #922.
Both local and global variables declared with "const" could be modified,
but now expressions that would modify them should generate an error
message at parse-time.
2012-12-13 15:05:29 -06:00
Bernhard Amann
12753f31ae Merge remote-tracking branch 'origin/master' into topic/bernhard/input-logging-commmon-functions 2012-12-13 11:22:58 -08:00
Robin Sommer
b867333c2e Merge remote-tracking branch 'origin/topic/jsiwek/gtp'
* origin/topic/jsiwek/gtp:
  Change binpac exceptions in AYIYA/GTP analyzers to do protocol_violation
  Add GTP tunnel analyzer memory leak unit test.
  Add GPRS Tunnelling Protocol (GTPv1) decapsulation.

Closes #690.
2012-12-10 14:48:18 -08:00
Seth Hall
3af4517e2a Adding an $end time for result values to measure the length of time a measurement took. 2012-12-04 11:04:01 -05:00
Seth Hall
d61d175a04 Merge remote-tracking branch 'origin/master' into topic/seth/metrics-merge 2012-12-04 00:17:43 -05:00
Seth Hall
3ca0333294 Fix to checking metrics thresholds at the end of the break interval ($every field). 2012-12-04 00:15:19 -05:00
Robin Sommer
63d43e6545 Renaming ASCII writer filter option 'only_single_header_row' to 'tsv'.
Also clarifying usage.

Closes #912.
2012-12-03 14:40:38 -08:00
Bernhard Amann
9b2265877d and factor stuff out the input framework too. 2012-12-03 13:41:19 -08:00
Bernhard Amann
501328d61a factor out ascii input/output.
First step - factored out everything the logging classes
use ( so only output ).

Moved the script-level configuration to logging/main,
and made the individual writers just refer to it -
no idea if this is good design. It works. But I am happy
about opinions :)

Next step - add support for input...
2012-12-03 12:59:11 -08:00
Jon Siwek
cc8f20c104 Merge branch 'master' into topic/jsiwek/gtp 2012-11-29 16:11:27 -06:00
Seth Hall
f1b7ca62ee Actually fix the problem I just tried to fix a minute ago. 2012-11-28 15:58:29 -05:00
Seth Hall
92285a9711 Fix a race condition when multiple workers report intermediate indexes simultaneously. 2012-11-28 15:52:41 -05:00
Seth Hall
956c23eb66 Merge remote-tracking branch 'origin/master' into topic/seth/metrics-merge 2012-11-28 14:57:42 -05:00
Seth Hall
6bdcdcecf9 Fixed a problem with metrics aggregation on clusters (thanks Jon!). 2012-11-26 16:17:35 -05:00
Robin Sommer
a5e237f50c The ASCII writer now supports a filter config option
'only_single_header_row' that turns the output into CSV format.

In that mode all meta data is skipped except for a single header line
with the fields names. Example:

    local my_filter: Log::Filter = [$name = "my-filter", $writer = Log::WRITER_ASCII, $config = table(["only_single_header_row"] = "T")];

Contributed by Carsten Langer.
2012-11-23 19:38:53 -08:00
Seth Hall
08538211e1 Some test updates. 2012-11-20 02:08:49 -05:00
Seth Hall
20fdd36a44 Updated the SQL injection detection script to make it include samples in notice emails. 2012-11-20 01:02:23 -05:00
Seth Hall
95b12262e4 More cleanup and fixed to the metrics framework. 2012-11-19 23:43:15 -05:00
Seth Hall
e99e090b85 Merge remote-tracking branch 'origin/master' into topic/seth/metrics-merge 2012-11-16 02:49:36 -05:00
Seth Hall
d9195076b1 Metrics framework checkpoint.
- New scan.bro merged in and reworked a bit.

 - Updated metrics API.  Now possible to calculate much more.
2012-11-16 02:37:52 -05:00
Jon Siwek
e0805498c6 Fix some warnings from sphinx when building docs. 2012-11-15 16:40:18 -06:00