Commit graph

96 commits

Author SHA1 Message Date
Robin Sommer
81dcda3eb4 Merge remote-tracking branch 'origin/topic/bernhard/topk'
* origin/topic/bernhard/topk:
  adapt to new folder structure
  fix opaqueval-related memleak
  synchronize pruned attribute
  potentially found wrong Ref.
  add sum function that can be used to get the number of total observed elements.
  in cluster settings, the resultvals can apparently been uninitialized in some special cases
  fix memory leaks
  fix warnings
  add topk cluster test
  make size of topk-list configureable when using sumstats
  implement merging for top-k.
  add serialization for topk
  make the get function const
  topk for sumstats
  well, a test that works..
  implement topk.
2013-08-01 10:27:18 -07:00
Seth Hall
7db531e162 Merge remote-tracking branch 'origin/master' into topic/seth/sumstats-updates
Conflicts:
	doc/scripts/DocSourcesList.cmake
	scripts/test-all-policy.bro
2013-07-30 11:48:03 -04:00
Seth Hall
0e23a8bc9e Beginning rework of SumStats API. 2013-07-30 11:46:51 -04:00
Bernhard Amann
9e0fd963e0 Merge remote-tracking branch 'origin/topic/robin/bloom-filter-merge' into topic/bernhard/hyperloglog
Conflicts:
	scripts/base/frameworks/sumstats/plugins/__load__.bro
	src/CMakeLists.txt
	src/NetVar.cc
	src/NetVar.h
	src/OpaqueVal.h
	src/SerialTypes.h
	src/bro.bif
2013-07-23 21:31:05 -07:00
Seth Hall
26f8bd7ad7 Fix a reporter message in sumstats. 2013-07-20 01:21:01 -04:00
Bernhard Amann
03b584c34a Merge remote-tracking branch 'origin/master' into topic/bernhard/topk 2013-07-09 14:56:05 -07:00
Jon Siwek
e45933562e Fix broken/missing documentation. 2013-05-23 16:53:42 -05:00
Seth Hall
4f4ef99a6b SumStats changes to how thresholding works to simplify and reduce memory use. 2013-05-23 10:12:17 -04:00
Seth Hall
6bd9ab3bd6 More adjustments to try and correct SumStats memory use. 2013-05-22 16:41:46 -04:00
Seth Hall
c4a1f30a87 Hopefully fixing a strange error. 2013-05-22 14:59:31 -04:00
Seth Hall
0a18b62d12 Merge remote-tracking branch 'origin/master' into topic/seth/sumstats-updates
Conflicts:
	scripts/base/frameworks/sumstats/cluster.bro
	scripts/base/frameworks/sumstats/plugins/average.bro
	scripts/base/frameworks/sumstats/plugins/max.bro
	scripts/base/frameworks/sumstats/plugins/min.bro
	scripts/base/frameworks/sumstats/plugins/sample.bro
	scripts/base/frameworks/sumstats/plugins/std-dev.bro
	scripts/base/frameworks/sumstats/plugins/sum.bro
	scripts/base/frameworks/sumstats/plugins/unique.bro
	scripts/base/frameworks/sumstats/plugins/variance.bro
	scripts/policy/protocols/http/detect-sqli.bro
	testing/btest/scripts/base/frameworks/sumstats/cluster-intermediate-update.bro
2013-05-21 22:33:16 -04:00
Seth Hall
bec965b66f Large update for the SumStats framework.
- On-demand access to sumstats results through "return from"
   functions named SumStats::request and Sumstats::request_key.
   Both functions are tested in standalone and clustered modes.

 - $name field has returned to SumStats which simplifies cluster
   code and makes the on-demand access stuff possible.

 - Clustered results can only be collected for 1 minute from their
   time of creation now instead of time of last read.

 - Thresholds use doubles instead of counts everywhere now.

 - Calculation dependency resolution occurs at start up time now
   instead of doing it at observation time which provide a minor
   cpu performance improvement.  A new plugin registration mechanism
   was created to support this change.

 - AppStats now has a minimal doc string and is broken into hook-based
   plugins.

 - AppStats and traceroute detection added to local.bro
2013-05-21 15:52:59 -04:00
Robin Sommer
f76446fb4e Merge remote-tracking branch 'origin/topic/bernhard/metrics-samples'
Closes #1003.

* origin/topic/bernhard/metrics-samples:
  finishing touches, make test more robust, rename function in last again
  change names of data structures after talking with seth
  make last plugin nicer and samplify sqli detector
  add tests for sampler
  reservoir sampler. untested.
2013-05-15 16:11:27 -07:00
Bernhard Amann
ab6d5b08a8 finishing touches, make test more robust, rename function in last again 2013-05-15 11:33:25 -07:00
Bernhard Amann
80962ad74b change names of data structures after talking with seth 2013-05-15 09:44:43 -07:00
Bernhard Amann
b0c4dcdfed make last plugin nicer and samplify sqli detector 2013-05-15 01:09:52 -07:00
Bernhard Amann
d939c2bdfc add tests for sampler 2013-05-13 22:11:17 -07:00
Bernhard Amann
fa58e26aa0 Merge remote-tracking branch 'origin/master' into topic/bernhard/metrics-samples 2013-05-13 21:20:25 -07:00
Bernhard Amann
56ab9285a4 Merge remote-tracking branch 'origin/master' into topic/bernhard/topk 2013-05-13 21:03:23 -07:00
Bernhard Amann
70f3f4343a prevent merge-hook of sumstats unique plugin from damaging source data. 2013-05-07 11:16:59 -07:00
Bernhard Amann
663082e2d5 reservoir sampler. untested. 2013-05-05 11:19:53 -07:00
Bernhard Amann
6acbbe0231 Merge remote-tracking branch 'origin/master' into topic/bernhard/topk 2013-05-03 23:04:22 -07:00
Bernhard Amann
3e74cdc6e0 Merge remote-tracking branch 'origin/master' into topic/bernhard/hyperloglog 2013-05-03 22:58:02 -07:00
Robin Sommer
8992dc6cff Merge remote-tracking branch 'origin/topic/bernhard/metrics-bug'
* origin/topic/bernhard/metrics-bug:
  add comment for seth to make us not forget about the copy statements
  fix the fix (thanks seth)
  duct-tape fix of values not propagating after intermediate check in cluster environments.
  Fixing coverage.bare-mode-errors test.
2013-05-02 12:47:36 -07:00
Bernhard Amann
2cfef36116 add comment for seth to make us not forget about the copy statements 2013-05-02 11:42:34 -07:00
Bernhard Amann
fe779575d5 fix the fix (thanks seth) 2013-05-02 11:38:40 -07:00
Bernhard Amann
d984243a77 duct-tape fix of values not propagating after intermediate check in cluster environments. 2013-05-02 11:34:33 -07:00
Robin Sommer
9d483b7e74 Fixing coverage.bare-mode-errors test. 2013-05-01 17:52:16 -07:00
Robin Sommer
9ea5a470e6 Fixing coverage.bare-mode-errors test. 2013-05-01 15:28:45 -07:00
Bernhard Amann
321dfadaab Merge remote-tracking branch 'origin/topic/robin/metrics-merge' into topic/bernhard/topk 2013-04-29 14:08:17 -07:00
Bernhard Amann
07ecd31bbd in cluster settings, the resultvals can apparently
been uninitialized in some special cases
2013-04-28 21:21:22 -07:00
Robin Sommer
b9249ecf9d Layout tweaks for the sumstats code, and preliminary updates for NEWS.
The layout changes are mostly whitespace and some comment rewrapping.
No functional changes.
2013-04-28 15:35:21 -07:00
Bernhard Amann
5608caf79a make error rate configureable 2013-04-25 14:20:13 -07:00
Bernhard Amann
9802e2332d Merge branch 'topic/bernhard/hyperloglog-with-measurement' into topic/bernhard/hyperloglog 2013-04-25 13:46:36 -07:00
Bernhard Amann
166fc4765a Merge remote-tracking branch 'origin/topic/seth/metrics-merge' into topic/bernhard/topk 2013-04-25 13:21:18 -07:00
Seth Hall
48cbb31747 Added an automatic state limiter for threshold based SumStats. 2013-04-25 12:51:55 -04:00
Bernhard Amann
c0890f2a0f make size of topk-list configureable when using sumstats 2013-04-24 15:01:06 -07:00
Bernhard Amann
2f48008c42 implement merging for top-k.
I am not (entirely) sure that this is mathematically correct, but
I am (more and more) getting the feeling that it... might be.

In any case - this was the last step and now it should work
in cluster settings.
2013-04-24 06:17:51 -07:00
Bernhard Amann
567fee6439 Merge remote-tracking branch 'origin/topic/seth/metrics-merge' into topic/bernhard/hyperloglog-with-measurement
Conflicts:
	scripts/base/frameworks/sumstats/plugins/__load__.bro
2013-04-23 15:27:17 -07:00
Bernhard Amann
de5769a88f topk for sumstats 2013-04-23 15:19:01 -07:00
Seth Hall
91362717da Renamed a plugin hook in sumstats framework. 2013-04-22 15:27:03 -04:00
Seth Hall
9574499382 Move loading variance back to where it should be alphabetically. 2013-04-22 14:15:37 -04:00
Bernhard Amann
dc18a6d6e3 Merge remote-tracking branch 'origin/topic/seth/metrics-merge' into topic/bernhard/hyperloglog-with-measurement
and fix up the hll scripts for it.

Conflicts:
	scripts/base/frameworks/sumstats/plugins/__load__.bro
	testing/btest/scripts/base/frameworks/measurement/basic.bro
2013-04-16 05:25:10 -07:00
Seth Hall
1cac89e4f8 SumStats test checkpoint. 2013-04-16 00:54:41 -04:00
Seth Hall
437815454d SumStats tests pass. 2013-04-15 15:28:11 -04:00
Seth Hall
fbe967e16a Checkpoint for SumStats rename. 2013-04-15 15:12:28 -04:00