Commit graph

440 commits

Author SHA1 Message Date
Robin Sommer
eb637f9f3e Merge remote-tracking branch 'origin/master' into topic/robin/plugins
Thanks to git this merge was less troublesome that I was afraid it
would be. Not all tests pass yet though (and file hashes have changed
unfortunately).

Conflicts:
	cmake
	doc/scripts/DocSourcesList.cmake
	scripts/base/init-bare.bro
	scripts/base/protocols/ftp/main.bro
	scripts/base/protocols/irc/dcc-send.bro
	scripts/test-all-policy.bro
	src/AnalyzerTags.h
	src/CMakeLists.txt
	src/analyzer/Analyzer.cc
	src/analyzer/protocol/file/File.cc
	src/analyzer/protocol/file/File.h
	src/analyzer/protocol/http/HTTP.cc
	src/analyzer/protocol/http/HTTP.h
	src/analyzer/protocol/mime/MIME.cc
	src/event.bif
	src/main.cc
	src/util-config.h.in
	testing/btest/Baseline/coverage.bare-load-baseline/canonified_loaded_scripts.log
	testing/btest/Baseline/coverage.default-load-baseline/canonified_loaded_scripts.log
	testing/btest/Baseline/istate.events-ssl/receiver.http.log
	testing/btest/Baseline/istate.events-ssl/sender.http.log
	testing/btest/Baseline/istate.events/receiver.http.log
	testing/btest/Baseline/istate.events/sender.http.log
2013-05-16 17:58:48 -07:00
Robin Sommer
639a6410c6 Merge remote-tracking branch 'origin/topic/bernhard/thread-cleanup'
* origin/topic/bernhard/thread-cleanup:
  and just to be really sure - always make threads go through OnWaitForStop
  hopefully finally fix last interesting race-condition
  it is apparently getting a bit late for changes at important code...
  spoke to soon (forgot to comment in line again).
  Change thread shutdown again to also work with input framework.
  Changing semantics of thread stop methods.
  Support for cleaning up threads that have terminated.
2013-05-15 17:16:41 -07:00
Robin Sommer
f76446fb4e Merge remote-tracking branch 'origin/topic/bernhard/metrics-samples'
Closes #1003.

* origin/topic/bernhard/metrics-samples:
  finishing touches, make test more robust, rename function in last again
  change names of data structures after talking with seth
  make last plugin nicer and samplify sqli detector
  add tests for sampler
  reservoir sampler. untested.
2013-05-15 16:11:27 -07:00
Robin Sommer
358528732c Merge branch 'topic/robin/sqlite-merge'
Closes #997.

* topic/robin/sqlite-merge: (25 commits)
  Fix to make sqlite test consistent, and updating coverage baselines
  Avoid a CMake warning about 3rdparty looking like a number.
  Fixing linker error.
  and there is no has-reader.
  make sqlite3 executable required and add test-cases for errors
  Renaming src/external -> src/3rdparty
  fix a few small rough edges (mostly comments that do no longer apply)
  fix bug in input-manager regarding enums that a writer reads without 0-terminating the string
  actually make sqlite work again (tests passed because the writer was not actually defined because of the define.)
  add sqlite distribution.
  fix warnings, update baselines, handle rotation
  add sqlite tests and fix small vector/set escaping bugs
  fix small bug with vectors and sets.
  make work with newer AsciiFormatter.
  start adding a different text for empty records for the sqlite writer.
  no, you will never guess from where I copied this file...
  make sqlite support more or less work for logging and input
  make sqlite-writer more stable.
  make it compile with new version of AsciiInputOutput
  and adapt to AsciiInputOutput - seems to work...
  ...

Conflicts:
	scripts/base/frameworks/input/__load__.bro
	src/CMakeLists.txt
	src/input.bif
	src/input/Manager.cc
	src/main.cc
	src/types.bif
	testing/btest/Baseline/coverage.bare-load-baseline/canonified_loaded_scripts.log
	testing/btest/Baseline/coverage.default-load-baseline/canonified_loaded_scripts.log
2013-05-15 16:03:19 -07:00
Bernhard Amann
f389cafc3b Merge remote-tracking branch 'origin/master' into topic/bernhard/thread-cleanup
Conflicts:
	src/main.cc
2013-05-15 16:00:49 -07:00
Robin Sommer
e050648621 Merge branch 'topic/robin/file-analysis-merge'
Closes #982.

* topic/robin/file-analysis-merge: (64 commits)
  A few more small tweaks.
  Various smalle tweaks in preparation for merging.
  FileAnalysis: load custom mime magic database just once.
  Improve a libmagic-related error message.
  FileAnalysis: add is_orig field to fa_file & Info.
  FileAnalysis: inlined doc fixes.
  FileAnalysis: optimizate connection set updating.
  FileAnalysis: optimize file handle construction.
  FileAnalysis: workarounds for older libmagics.
  FileAnalysis: add custom libmagic database.
  FileAnalysis: change terminology s/action/analyzer
  FileAnalysis: libmagic tweaks.
  FileAnalysis: add bif for setting timeout interval
  FileAnalysis: add more params to some events.
  FileAnalysis: insert explicit event queue flush points.
  FileAnalysis: remove some file events.
  FileAnalysis: finish switching hooks to events.
  FileAnalysis: checkpoint in middle of big reorganization.
  FileAnalysis: fix file type canonification for file_analysis.log
  Revert "FileAnalysis: optimize get_file_handle event queueing."
  ...

Conflicts:
	NEWS
2013-05-15 15:14:21 -07:00
Bernhard Amann
ab6d5b08a8 finishing touches, make test more robust, rename function in last again 2013-05-15 11:33:25 -07:00
Bernhard Amann
80962ad74b change names of data structures after talking with seth 2013-05-15 09:44:43 -07:00
Bernhard Amann
b0c4dcdfed make last plugin nicer and samplify sqli detector 2013-05-15 01:09:52 -07:00
Robin Sommer
de88645d05 Merge remote-tracking branch 'origin/topic/bernhard/sqlite'
* origin/topic/bernhard/sqlite:
  fix a few small rough edges (mostly comments that do no longer apply)
  fix bug in input-manager regarding enums that a writer reads without 0-terminating the string
  actually make sqlite work again (tests passed because the writer was not actually defined because of the define.)
  add sqlite distribution.
  fix warnings, update baselines, handle rotation
  add sqlite tests and fix small vector/set escaping bugs
  fix small bug with vectors and sets.
  make work with newer AsciiFormatter.
  start adding a different text for empty records for the sqlite writer.
  no, you will never guess from where I copied this file...
  make sqlite support more or less work for logging and input
  make sqlite-writer more stable.
  make it compile with new version of AsciiInputOutput
  and adapt to AsciiInputOutput - seems to work...
  make it compile
  add SQLite reader.
  ...adapt to new api...
  now the writer supports tables and vectors.
  basic sqlite writer seems to work.
2013-05-14 17:11:09 -07:00
Bernhard Amann
d939c2bdfc add tests for sampler 2013-05-13 22:11:17 -07:00
Bernhard Amann
fa58e26aa0 Merge remote-tracking branch 'origin/master' into topic/bernhard/metrics-samples 2013-05-13 21:20:25 -07:00
Bernhard Amann
bb1e2f57b9 Merge remote-tracking branch 'origin/master' into topic/bernhard/thread-cleanup 2013-05-13 21:19:09 -07:00
Bernhard Amann
6392acecd2 fix warnings, update baselines, handle rotation 2013-05-12 20:48:17 -07:00
Bernhard Amann
747ba68030 Merge remote branch 'origin/master' into topic/bernhard/sqlite 2013-05-12 20:47:55 -07:00
Bernhard Amann
70f3f4343a prevent merge-hook of sumstats unique plugin from damaging source data. 2013-05-07 11:16:59 -07:00
Jon Siwek
ec50cad9db Merge branch 'master' into topic/jsiwek/file-analysis
Conflicts:
	scripts/base/protocols/ftp/main.bro
	src/OpaqueVal.h
	testing/btest/Baseline/coverage.bare-load-baseline/canonified_loaded_scripts.log
	testing/btest/Baseline/coverage.default-load-baseline/canonified_loaded_scripts.log
2013-05-06 10:21:16 -05:00
Bernhard Amann
663082e2d5 reservoir sampler. untested. 2013-05-05 11:19:53 -07:00
Robin Sommer
8992dc6cff Merge remote-tracking branch 'origin/topic/bernhard/metrics-bug'
* origin/topic/bernhard/metrics-bug:
  add comment for seth to make us not forget about the copy statements
  fix the fix (thanks seth)
  duct-tape fix of values not propagating after intermediate check in cluster environments.
  Fixing coverage.bare-mode-errors test.
2013-05-02 12:47:36 -07:00
Bernhard Amann
2cfef36116 add comment for seth to make us not forget about the copy statements 2013-05-02 11:42:34 -07:00
Bernhard Amann
fe779575d5 fix the fix (thanks seth) 2013-05-02 11:38:40 -07:00
Bernhard Amann
d984243a77 duct-tape fix of values not propagating after intermediate check in cluster environments. 2013-05-02 11:34:33 -07:00
Robin Sommer
9d483b7e74 Fixing coverage.bare-mode-errors test. 2013-05-01 17:52:16 -07:00
Robin Sommer
9ea5a470e6 Fixing coverage.bare-mode-errors test. 2013-05-01 15:28:45 -07:00
Bernhard Amann
b968103c92 Merge remote-tracking branch 'origin/master' into topic/bernhard/sqlite 2013-04-28 22:06:34 -07:00
Robin Sommer
b9249ecf9d Layout tweaks for the sumstats code, and preliminary updates for NEWS.
The layout changes are mostly whitespace and some comment rewrapping.
No functional changes.
2013-04-28 15:35:21 -07:00
Seth Hall
48cbb31747 Added an automatic state limiter for threshold based SumStats. 2013-04-25 12:51:55 -04:00
Jon Siwek
f07760ba00 FileAnalysis: add is_orig field to fa_file & Info. 2013-04-23 10:50:43 -05:00
Seth Hall
91362717da Renamed a plugin hook in sumstats framework. 2013-04-22 15:27:03 -04:00
Seth Hall
9574499382 Move loading variance back to where it should be alphabetically. 2013-04-22 14:15:37 -04:00
Robin Sommer
aeddca6523 More API documentation. 2013-04-16 14:28:23 -07:00
Seth Hall
1cac89e4f8 SumStats test checkpoint. 2013-04-16 00:54:41 -04:00
Seth Hall
437815454d SumStats tests pass. 2013-04-15 15:28:11 -04:00
Seth Hall
fbe967e16a Checkpoint for SumStats rename. 2013-04-15 15:12:28 -04:00
Seth Hall
8165d6077d Fix another occasional reporter error. 2013-04-12 11:20:45 -04:00
Seth Hall
e93fd69cf2 Small updates to hopefully correct reporter errors leading to lost memory. 2013-04-12 09:28:38 -04:00
Jon Siwek
b8c98b8bf7 FileAnalysis: change terminology s/action/analyzer 2013-04-11 14:53:54 -05:00
Jon Siwek
e81f2ae7b0 FileAnalysis: libmagic tweaks.
Remove verbose file type detection and automatically strip out charset
from mime type.
2013-04-11 13:11:46 -05:00
Jon Siwek
2fba37e277 FileAnalysis: add bif for setting timeout interval 2013-04-11 12:08:46 -05:00
Seth Hall
a615601269 Trying to fix a state maintenance issue. 2013-04-11 09:42:46 -04:00
Jon Siwek
a2d9b47bcd FileAnalysis: finish switching hooks to events. 2013-04-10 11:13:43 -05:00
Jon Siwek
641154f8e8 FileAnalysis: checkpoint in middle of big reorganization.
- FileAnalysis::Info is now just a record used for logging, the fa_file
  record type is defined in init-bare.bro as the analogue to a
  connection record.

- Starting to transfer policy hook triggers and analyzer results to
  events.
2013-04-09 15:49:58 -05:00
Bernhard Amann
2cc1f82425 Merge remote-tracking branch 'origin/master' into topic/bernhard/thread-cleanup 2013-04-07 20:43:47 +02:00
Robin Sommer
bccaea6883 Adding options Analyzer::disable_all to disable all analyzers at
startup.

One can then selectively enable the ones one wants inside a bro_init()
handler.
2013-04-04 15:24:15 -07:00
Seth Hall
e8b60d1ba8 Updated FTP bruteforce detection and a few other small changes. 2013-04-02 00:55:25 -04:00
Seth Hall
423bf3b3bf Test updates and cleanup. 2013-04-02 00:30:14 -04:00
Seth Hall
0e3c84e863 Fixed the measurement "sample" plugin. 2013-04-02 00:19:06 -04:00
Seth Hall
b477d2b02d Measurement framework is ready for testing.
- New, expanded API.
 - Calculations moved into plugins.
 - Scripts using measurement framework ported.
 - Updated the script-land queue implementation to make it more generic.
 -
2013-04-01 17:04:15 -04:00
Robin Sommer
e0c4bd1a82 Lots of cleanup and API documentation for the analyzer/* classes.
I've used the opportunity to also cleanup DPD's expect_connection()
infrastructure, and renamed that bif to schedule_analyzer(), which
seems more appropiate. One can now also schedule more than one
analyzer per connection.

TODOs:
        - "make install" is probably broken.
        - Broxygen is probably broken for plugin-defined events.
        - event groups are broken (do we want to keep them?)
        - parallel btest is broken, but I'm not sure why ...
          (tests all pass individually, but lots of error when running
          in parallel; must be related to *.bif restructuring).
        - Document API for src/plugin/*
        - Document API for src/analyzer/Analyzer.h
        - Document API for scripts/base/frameworks/analyzer
2013-04-01 13:12:21 -07:00
Seth Hall
93eca70e6b Merge remote-tracking branch 'origin/master' into topic/seth/metrics-merge 2013-04-01 14:16:46 -04:00