Commit graph

370 commits

Author SHA1 Message Date
Seth Hall
bec965b66f Large update for the SumStats framework.
- On-demand access to sumstats results through "return from"
   functions named SumStats::request and Sumstats::request_key.
   Both functions are tested in standalone and clustered modes.

 - $name field has returned to SumStats which simplifies cluster
   code and makes the on-demand access stuff possible.

 - Clustered results can only be collected for 1 minute from their
   time of creation now instead of time of last read.

 - Thresholds use doubles instead of counts everywhere now.

 - Calculation dependency resolution occurs at start up time now
   instead of doing it at observation time which provide a minor
   cpu performance improvement.  A new plugin registration mechanism
   was created to support this change.

 - AppStats now has a minimal doc string and is broken into hook-based
   plugins.

 - AppStats and traceroute detection added to local.bro
2013-05-21 15:52:59 -04:00
Robin Sommer
b9249ecf9d Layout tweaks for the sumstats code, and preliminary updates for NEWS.
The layout changes are mostly whitespace and some comment rewrapping.
No functional changes.
2013-04-28 15:35:21 -07:00
Seth Hall
48cbb31747 Added an automatic state limiter for threshold based SumStats. 2013-04-25 12:51:55 -04:00
Seth Hall
91362717da Renamed a plugin hook in sumstats framework. 2013-04-22 15:27:03 -04:00
Seth Hall
9574499382 Move loading variance back to where it should be alphabetically. 2013-04-22 14:15:37 -04:00
Seth Hall
1cac89e4f8 SumStats test checkpoint. 2013-04-16 00:54:41 -04:00
Seth Hall
437815454d SumStats tests pass. 2013-04-15 15:28:11 -04:00
Seth Hall
fbe967e16a Checkpoint for SumStats rename. 2013-04-15 15:12:28 -04:00
Seth Hall
8165d6077d Fix another occasional reporter error. 2013-04-12 11:20:45 -04:00
Seth Hall
e93fd69cf2 Small updates to hopefully correct reporter errors leading to lost memory. 2013-04-12 09:28:38 -04:00
Seth Hall
a615601269 Trying to fix a state maintenance issue. 2013-04-11 09:42:46 -04:00
Seth Hall
e8b60d1ba8 Updated FTP bruteforce detection and a few other small changes. 2013-04-02 00:55:25 -04:00
Seth Hall
423bf3b3bf Test updates and cleanup. 2013-04-02 00:30:14 -04:00
Seth Hall
0e3c84e863 Fixed the measurement "sample" plugin. 2013-04-02 00:19:06 -04:00
Seth Hall
b477d2b02d Measurement framework is ready for testing.
- New, expanded API.
 - Calculations moved into plugins.
 - Scripts using measurement framework ported.
 - Updated the script-land queue implementation to make it more generic.
 -
2013-04-01 17:04:15 -04:00
Seth Hall
93eca70e6b Merge remote-tracking branch 'origin/master' into topic/seth/metrics-merge 2013-04-01 14:16:46 -04:00
Seth Hall
53f9948b02 Measurement framework tests all pass now. 2013-04-01 14:16:37 -04:00
Seth Hall
6dc204b385 Checkpoint, don't try running this. It's broken all over the place. 2013-03-19 11:39:58 -04:00
Seth Hall
5734f9ef71 Merge remote-tracking branch 'origin/master' into topic/seth/software-version-updates2 2013-03-14 09:21:13 -04:00
Seth Hall
8778761c07 Checkpoint 2013-03-13 22:55:03 -04:00
Seth Hall
09cbaa7ccc Merge remote-tracking branch 'origin/master' into topic/seth/metrics-merge
Conflicts:
	testing/btest/Baseline/coverage.default-load-baseline/canonified_loaded_scripts.log
	testing/btest/Baseline/scripts.base.frameworks.notice.cluster/manager-1.notice.log
	testing/btest/Baseline/scripts.base.frameworks.notice.suppression-cluster/manager-1.notice.log
	testing/btest/Baseline/scripts.base.protocols.ftp.gridftp/notice.log
2013-03-13 00:26:55 -04:00
Robin Sommer
f830ed3edf s/bro-ids.org/bro.org/g 2013-03-07 19:33:04 -08:00
Robin Sommer
8a6d68e00f Merge remote-tracking branch 'origin/topic/bernhard/remove-length'
Closes #955.

* origin/topic/bernhard/remove-length:
  forgot to remove the baselines for the now unnecessary bifs
  remove the byte_len and length bifs
2013-03-06 16:46:20 -08:00
Robin Sommer
1bd2f26df3 Merge remote-tracking branch 'origin/topic/seth/notice-framework-updates'
So much nicer!

Closes #954.

* origin/topic/seth/notice-framework-updates:
  Update notice framework documentation to represent the new reality.
  Complete removal of the old table based notice policy mechanism.
  Updates for the notices framework.
2013-03-06 16:45:30 -08:00
Bernhard Amann
986b346e3f remove the byte_len and length bifs 2013-03-06 13:45:42 -08:00
Seth Hall
00eeadf2f5 Complete removal of the old table based notice policy mechanism. 2013-03-06 09:44:25 -05:00
Seth Hall
e3856d7681 Removing a field that is no longer logged through the standard metrics log. 2013-02-27 11:25:01 -05:00
Seth Hall
9f8ba408ba Updates for the notices framework.
- Moved the Notice::notice event and Notice::policy table to both be hooks.

 - Renamed the old Notice::policy to Notice::policy_table and documented it as deprecated.
2013-02-11 14:36:14 -05:00
Jon Siwek
26bf99c5a3 Add parsing for GTPv1 extension headers and control messages.
Added a generic gtpv1_message event generated for any GTP message type.

Added specific events for the create/update/delete PDP context
request/response messages.

Addresses #934.
2013-02-07 14:59:02 -06:00
Seth Hall
f8be65ca56 Merge remote-tracking branch 'origin/master' into topic/seth/metrics-merge 2013-02-07 10:01:13 -05:00
Seth Hall
ad9e2a143c Update software version parsing and comparison to account for a third numeric subversion.
- $addl is also compared numerically now if the value is actually numeric.
2013-02-06 13:59:05 -05:00
Jon Siwek
69afc4a882 Add an error for record coercions that would orphan a field.
These cases should be avoidable by fixing scripts where they occur and
they can also help catch typos that would lead to unintentional runtime
behavior.

Adding this already revealed several scripts where a field in an inlined
record was never removed after a code refactor.
2013-01-24 09:56:19 -06:00
Robin Sommer
762c034ec2 Merge remote-tracking branch 'origin/topic/bernhard/input-logging-commmon-functions'
* origin/topic/bernhard/input-logging-commmon-functions:
  add the last of Robins suggestions (separate info-struct for constructors).
  port memory leak fix from master
  harmonize function naming
  move AsciiInputOutput over to threading
  and thinking about it, ascii-io doesn't need the separator
  change constructors
  and factor stuff out the input framework too.
  factor out ascii input/output.
  std::string accessors to escape_sequence functionality
  intermediate commit - it has been over a month since I touched this...

I cleaned up the AsciiInputOutput class somewhat, including renaming
it to AsciiFormatter, renaming some of its methods, and turning the
static methods into members for consistency.

Closes #929.
2013-01-23 16:51:54 -08:00
Jon Siwek
fdd11428c1 Change reporter messages to more reliably print to stderr.
Moved this functionality to be internal instead of in the script-layer
event handlers.  The issue with the later is that bad things can happen
between the time a reporter event handler is dispatched and the time it
is executed, and if bro crashes in that time, the message may never be
seen/logged.

Addressed #930 (and revisits #836).
2013-01-18 12:46:00 -06:00
Bernhard Amann
1b0bb5063a Merge remote-tracking branch 'origin/master' into topic/bernhard/input-logging-commmon-functions
Conflicts:
	src/input/readers/Ascii.cc
2013-01-11 09:48:53 -08:00
Seth Hall
720089c03f Add a feature to better select the value threshold will apply to in the metrics framework.
- The feature was primarily added to allow the value to be
   modified for cluster based intermediate threshold checks
   without requiring the user to write the metrics filter
   differently for cluster consideration.  It's also a nice
   way to calculate some related information to the metric
   without accidently applying thresholds to that value.

 - Fixed a few small bugs in ftp detect-bruteforcing script
   and adapted it to the new threshold value selection feature.
2013-01-05 22:37:19 -05:00
Seth Hall
bcd7fe114d Fixed an issue causing reporter messages from the metrics framework. 2013-01-05 22:27:17 -05:00
Seth Hall
c3a6916572 More cluster tuning for the metrics framework.
- Fixed several state maintenance issues for intermediate updates.

 - Added a new tuning variable Metrics::max_outstanding_global_views
   which limits the number of in-flight intermediate updates per
   metric filter.

 - Changed the default global view threshold percent to 20% (up from 10%)
2013-01-04 16:54:13 -05:00
Seth Hall
50827d8df0 Implement an option to disable intermediate updates for the metrics framework.
- There are some large sites having trouble and I'm thinking it
   might be overload from intermediate updates.
2012-12-21 23:17:27 -05:00
Seth Hall
9c00ef3ccd Doing some code reorganization and small changes to hopefully fix a memory issue. 2012-12-18 12:22:28 -05:00
Seth Hall
f31de38c9b Bug fix. 2012-12-18 01:44:46 -05:00
Seth Hall
69b7ce12d2 API updates for metrics framework.
- Removed default logging.  Now a function is available for the new
  $period_finished filter field to get the same behavior for logging
  named Metrics::write_log.

- Added index rollups for getting multiple metrics result values
  as the same time.
2012-12-18 01:08:59 -05:00
Seth Hall
69030fdff3 Merge remote-tracking branch 'origin/master' into topic/seth/metrics-merge 2012-12-17 10:00:34 -05:00
Jon Siwek
290c2a0b4d Make const variables actually constant. Addresses #922.
Both local and global variables declared with "const" could be modified,
but now expressions that would modify them should generate an error
message at parse-time.
2012-12-13 15:05:29 -06:00
Bernhard Amann
12753f31ae Merge remote-tracking branch 'origin/master' into topic/bernhard/input-logging-commmon-functions 2012-12-13 11:22:58 -08:00
Robin Sommer
b867333c2e Merge remote-tracking branch 'origin/topic/jsiwek/gtp'
* origin/topic/jsiwek/gtp:
  Change binpac exceptions in AYIYA/GTP analyzers to do protocol_violation
  Add GTP tunnel analyzer memory leak unit test.
  Add GPRS Tunnelling Protocol (GTPv1) decapsulation.

Closes #690.
2012-12-10 14:48:18 -08:00
Seth Hall
3af4517e2a Adding an $end time for result values to measure the length of time a measurement took. 2012-12-04 11:04:01 -05:00
Seth Hall
d61d175a04 Merge remote-tracking branch 'origin/master' into topic/seth/metrics-merge 2012-12-04 00:17:43 -05:00
Seth Hall
3ca0333294 Fix to checking metrics thresholds at the end of the break interval ($every field). 2012-12-04 00:15:19 -05:00
Robin Sommer
63d43e6545 Renaming ASCII writer filter option 'only_single_header_row' to 'tsv'.
Also clarifying usage.

Closes #912.
2012-12-03 14:40:38 -08:00