Commit graph

282 commits

Author SHA1 Message Date
Seth Hall
af87126521 Updating test baselines. 2013-07-05 01:27:59 -04:00
Robin Sommer
a329c3e7c3 Merge remote-tracking branch 'origin/topic/jsiwek/plugin-docs'
Closes #1019.

* origin/topic/jsiwek/plugin-docs:
  Teach broxygen to generate protocol analyzer plugin reference.
  const adjustments
2013-07-03 16:32:00 -07:00
Robin Sommer
d8b05af7e5 Merge remote-tracking branch 'origin/topic/jsiwek/faf-cleanup'
Closes #1002.

* origin/topic/jsiwek/faf-cleanup:
  Move file analyzers to new plugin infrastructure.
  Add a general file analysis overview/how-to document.
  Improve file analysis doxygen comments.
  Improve tracking of HTTP file extraction (addresses #988).
  Fix HTTP multipart body file analysis.
  Remove logging of analyzers field of FileAnalysis::Info.
  Remove extraction counter in default file extraction scripts.
  Remove FileAnalysis::postpone_timeout.
  Make default get_file_handle handlers &priority=5.
  Add input interface to forward data for file analysis.
  File analysis framework interface simplifications.
2013-07-03 16:27:16 -07:00
Jon Siwek
7c7b6214a6 Move file analyzers to new plugin infrastructure. 2013-06-10 15:50:18 -05:00
Jon Siwek
f2574636b6 Merge branch 'master' into topic/jsiwek/faf-cleanup
Conflicts:
	scripts/base/protocols/ftp/file-analysis.bro
	scripts/base/protocols/http/file-analysis.bro
	scripts/base/protocols/irc/file-analysis.bro
	scripts/base/protocols/smtp/file-analysis.bro
	src/file_analysis/File.cc
	src/file_analysis/File.h
	src/file_analysis/Manager.cc
	src/file_analysis/Manager.h
	testing/btest/Baseline/scripts.base.frameworks.file-analysis.logging/file_analysis.log
	testing/btest/Baseline/scripts.base.protocols.ftp.ftp-extract/ftp-item-0.dat
	testing/btest/Baseline/scripts.base.protocols.ftp.ftp-extract/ftp-item-1.dat
	testing/btest/Baseline/scripts.base.protocols.ftp.ftp-extract/ftp-item-2.dat
	testing/btest/Baseline/scripts.base.protocols.ftp.ftp-extract/ftp-item-3.dat
	testing/btest/Baseline/scripts.base.protocols.ftp.ftp-extract/ftp-item-BTsa70Ua9x7-1.dat
	testing/btest/Baseline/scripts.base.protocols.ftp.ftp-extract/ftp-item-BTsa70Ua9x7.dat
	testing/btest/Baseline/scripts.base.protocols.ftp.ftp-extract/ftp-item-Rqjkzoroau4-0.dat
	testing/btest/Baseline/scripts.base.protocols.ftp.ftp-extract/ftp-item-Rqjkzoroau4.dat
	testing/btest/Baseline/scripts.base.protocols.ftp.ftp-extract/ftp-item-VLQvJybrm38-2.dat
	testing/btest/Baseline/scripts.base.protocols.ftp.ftp-extract/ftp-item-VLQvJybrm38.dat
	testing/btest/Baseline/scripts.base.protocols.ftp.ftp-extract/ftp-item-zrfwSs9K1yk-3.dat
	testing/btest/Baseline/scripts.base.protocols.ftp.ftp-extract/ftp-item-zrfwSs9K1yk.dat
	testing/btest/Baseline/scripts.base.protocols.ftp.ftp-extract/ftp.log
	testing/btest/Baseline/scripts.base.protocols.http.http-extract-files/http-item-BFymS6bFgT3-0.dat
	testing/btest/Baseline/scripts.base.protocols.http.http-extract-files/http-item-BFymS6bFgT3.dat
	testing/btest/Baseline/scripts.base.protocols.http.http-extract-files/http-item.dat
	testing/btest/Baseline/scripts.base.protocols.http.http-extract-files/http.log
	testing/btest/Baseline/scripts.base.protocols.irc.dcc-extract/irc-dcc-item-wqKMAamJVSb-0.dat
	testing/btest/Baseline/scripts.base.protocols.irc.dcc-extract/irc-dcc-item-wqKMAamJVSb.dat
	testing/btest/Baseline/scripts.base.protocols.irc.dcc-extract/irc-dcc-item.dat
	testing/btest/Baseline/scripts.base.protocols.irc.dcc-extract/irc.log
	testing/btest/Baseline/scripts.base.protocols.smtp.mime-extract/smtp-entity-0.dat
	testing/btest/Baseline/scripts.base.protocols.smtp.mime-extract/smtp-entity-1.dat
	testing/btest/Baseline/scripts.base.protocols.smtp.mime-extract/smtp-entity-Ltd7QO7jEv3-1.dat
	testing/btest/Baseline/scripts.base.protocols.smtp.mime-extract/smtp-entity-Ltd7QO7jEv3.dat
	testing/btest/Baseline/scripts.base.protocols.smtp.mime-extract/smtp-entity-cwR7l6Zctxb-0.dat
	testing/btest/Baseline/scripts.base.protocols.smtp.mime-extract/smtp-entity-cwR7l6Zctxb.dat
	testing/btest/Baseline/scripts.base.protocols.smtp.mime-extract/smtp_entities.log
	testing/btest/scripts/base/protocols/ftp/ftp-extract.bro
	testing/btest/scripts/base/protocols/http/http-extract-files.bro
	testing/btest/scripts/base/protocols/irc/dcc-extract.test
	testing/btest/scripts/base/protocols/smtp/mime-extract.test
2013-06-07 15:44:36 -05:00
Jon Siwek
e56a17102e Teach broxygen to generate protocol analyzer plugin reference. 2013-06-07 13:21:18 -05:00
Robin Sommer
da3eb2d3e2 Merge remote-tracking branch 'origin/topic/robin/plugins' 2013-06-04 20:58:16 -07:00
Jon Siwek
a5e1810aa8 Fix various documentation/typos; remove a few superfluous things. 2013-06-03 16:03:25 -05:00
Robin Sommer
c6ad731562 More smaller cleanup. 2013-06-02 18:21:45 -07:00
Robin Sommer
d3d14e10cf Merge remote-tracking branch 'origin/topic/jsiwek/983'
Closes #983.

* origin/topic/jsiwek/983:
  Add named constructor examples to docs.
  Allow named vector constructors. Addresses #983.
  Allow named table constructors.  Addresses #983.
  Improve set constructor argument coercion.
  Allow named set constructors. Addresses #983.
  Allow named record constructors. Addresses #983.
2013-06-02 17:41:33 -07:00
Robin Sommer
c049c758c3 Merge remote-tracking branch 'origin/master' into topic/robin/plugins
Conflicts:
	aux/bro-aux
	aux/broctl
	src/DPM.cc
2013-05-30 17:43:50 -07:00
Jon Siwek
4301002f1a Add named constructor examples to docs. 2013-05-30 11:41:22 -05:00
Jon Siwek
e45933562e Fix broken/missing documentation. 2013-05-23 16:53:42 -05:00
Jon Siwek
9c86a3ee0e Add a general file analysis overview/how-to document. 2013-05-23 14:29:13 -05:00
Robin Sommer
4ccd6d76fd Fixing tests.
Part of this involves making the file-analysis tests independent of
specific hash values. I've done that only partially though.
2013-05-17 18:14:35 -07:00
Robin Sommer
85e3eb4c57 Fixing Broxygen generation.
Needs to have BROMAGIC set to find the magic database.
2013-05-17 07:44:01 -07:00
Robin Sommer
eb637f9f3e Merge remote-tracking branch 'origin/master' into topic/robin/plugins
Thanks to git this merge was less troublesome that I was afraid it
would be. Not all tests pass yet though (and file hashes have changed
unfortunately).

Conflicts:
	cmake
	doc/scripts/DocSourcesList.cmake
	scripts/base/init-bare.bro
	scripts/base/protocols/ftp/main.bro
	scripts/base/protocols/irc/dcc-send.bro
	scripts/test-all-policy.bro
	src/AnalyzerTags.h
	src/CMakeLists.txt
	src/analyzer/Analyzer.cc
	src/analyzer/protocol/file/File.cc
	src/analyzer/protocol/file/File.h
	src/analyzer/protocol/http/HTTP.cc
	src/analyzer/protocol/http/HTTP.h
	src/analyzer/protocol/mime/MIME.cc
	src/event.bif
	src/main.cc
	src/util-config.h.in
	testing/btest/Baseline/coverage.bare-load-baseline/canonified_loaded_scripts.log
	testing/btest/Baseline/coverage.default-load-baseline/canonified_loaded_scripts.log
	testing/btest/Baseline/istate.events-ssl/receiver.http.log
	testing/btest/Baseline/istate.events-ssl/sender.http.log
	testing/btest/Baseline/istate.events/receiver.http.log
	testing/btest/Baseline/istate.events/sender.http.log
2013-05-16 17:58:48 -07:00
Robin Sommer
11fd12b18e Band-aid to get Broxygen's bif documentation back.
We'll need a different approach here eventually.
2013-05-16 16:57:49 -07:00
Robin Sommer
36e56c725b Updating test baselines. 2013-05-15 17:29:53 -07:00
Robin Sommer
358528732c Merge branch 'topic/robin/sqlite-merge'
Closes #997.

* topic/robin/sqlite-merge: (25 commits)
  Fix to make sqlite test consistent, and updating coverage baselines
  Avoid a CMake warning about 3rdparty looking like a number.
  Fixing linker error.
  and there is no has-reader.
  make sqlite3 executable required and add test-cases for errors
  Renaming src/external -> src/3rdparty
  fix a few small rough edges (mostly comments that do no longer apply)
  fix bug in input-manager regarding enums that a writer reads without 0-terminating the string
  actually make sqlite work again (tests passed because the writer was not actually defined because of the define.)
  add sqlite distribution.
  fix warnings, update baselines, handle rotation
  add sqlite tests and fix small vector/set escaping bugs
  fix small bug with vectors and sets.
  make work with newer AsciiFormatter.
  start adding a different text for empty records for the sqlite writer.
  no, you will never guess from where I copied this file...
  make sqlite support more or less work for logging and input
  make sqlite-writer more stable.
  make it compile with new version of AsciiInputOutput
  and adapt to AsciiInputOutput - seems to work...
  ...

Conflicts:
	scripts/base/frameworks/input/__load__.bro
	src/CMakeLists.txt
	src/input.bif
	src/input/Manager.cc
	src/main.cc
	src/types.bif
	testing/btest/Baseline/coverage.bare-load-baseline/canonified_loaded_scripts.log
	testing/btest/Baseline/coverage.default-load-baseline/canonified_loaded_scripts.log
2013-05-15 16:03:19 -07:00
Robin Sommer
4fe0e22128 Fix to make sqlite test consistent, and updating coverage baselines 2013-05-15 15:56:53 -07:00
Robin Sommer
e050648621 Merge branch 'topic/robin/file-analysis-merge'
Closes #982.

* topic/robin/file-analysis-merge: (64 commits)
  A few more small tweaks.
  Various smalle tweaks in preparation for merging.
  FileAnalysis: load custom mime magic database just once.
  Improve a libmagic-related error message.
  FileAnalysis: add is_orig field to fa_file & Info.
  FileAnalysis: inlined doc fixes.
  FileAnalysis: optimizate connection set updating.
  FileAnalysis: optimize file handle construction.
  FileAnalysis: workarounds for older libmagics.
  FileAnalysis: add custom libmagic database.
  FileAnalysis: change terminology s/action/analyzer
  FileAnalysis: libmagic tweaks.
  FileAnalysis: add bif for setting timeout interval
  FileAnalysis: add more params to some events.
  FileAnalysis: insert explicit event queue flush points.
  FileAnalysis: remove some file events.
  FileAnalysis: finish switching hooks to events.
  FileAnalysis: checkpoint in middle of big reorganization.
  FileAnalysis: fix file type canonification for file_analysis.log
  Revert "FileAnalysis: optimize get_file_handle event queueing."
  ...

Conflicts:
	NEWS
2013-05-15 15:14:21 -07:00
Jon Siwek
e2a1d4a233 Allow default function/hook/event parameters. Addresses #972.
And changed the endianness parameter of bytestring_to_count() BIF to
default to false (big endian), mostly just to prove that the BIF parser
doesn't choke on default parameters.
2013-05-07 14:32:22 -05:00
Jon Siwek
ec50cad9db Merge branch 'master' into topic/jsiwek/file-analysis
Conflicts:
	scripts/base/protocols/ftp/main.bro
	src/OpaqueVal.h
	testing/btest/Baseline/coverage.bare-load-baseline/canonified_loaded_scripts.log
	testing/btest/Baseline/coverage.default-load-baseline/canonified_loaded_scripts.log
2013-05-06 10:21:16 -05:00
Seth Hall
60605412ab Fix a few tests. 2013-04-22 14:14:50 -04:00
Robin Sommer
10dc8b9279 Updating tests. 2013-04-19 16:35:24 -07:00
Robin Sommer
dfc4cb0881 Moving all analyzers over to new structure.
This is a checkpoint, it works but there's more cleanup to do. TODOs in
src/analyzer/protocols/TODO.
2013-04-16 20:52:03 -07:00
Robin Sommer
52cd02173d Removing event groups. 2013-04-09 16:49:47 -07:00
Robin Sommer
86551cd429 Fixing test. 2013-04-05 12:38:21 -07:00
Seth Hall
423bf3b3bf Test updates and cleanup. 2013-04-02 00:30:14 -04:00
Robin Sommer
e0c4bd1a82 Lots of cleanup and API documentation for the analyzer/* classes.
I've used the opportunity to also cleanup DPD's expect_connection()
infrastructure, and renamed that bif to schedule_analyzer(), which
seems more appropiate. One can now also schedule more than one
analyzer per connection.

TODOs:
        - "make install" is probably broken.
        - Broxygen is probably broken for plugin-defined events.
        - event groups are broken (do we want to keep them?)
        - parallel btest is broken, but I'm not sure why ...
          (tests all pass individually, but lots of error when running
          in parallel; must be related to *.bif restructuring).
        - Document API for src/plugin/*
        - Document API for src/analyzer/Analyzer.h
        - Document API for scripts/base/frameworks/analyzer
2013-04-01 13:12:21 -07:00
Robin Sommer
2be985433c Test-suite passes.
All tests pass with one exception: some Broxygen tests are broken
because dpd_config doesn't exist anymore. Need to update the mechanism
for auto-documenting well-known ports.
2013-03-26 15:40:23 -07:00
Jon Siwek
b30211c178 FileAnalysis: fix unit tests. 2013-03-18 12:29:41 -05:00
Seth Hall
8778761c07 Checkpoint 2013-03-13 22:55:03 -04:00
Seth Hall
09cbaa7ccc Merge remote-tracking branch 'origin/master' into topic/seth/metrics-merge
Conflicts:
	testing/btest/Baseline/coverage.default-load-baseline/canonified_loaded_scripts.log
	testing/btest/Baseline/scripts.base.frameworks.notice.cluster/manager-1.notice.log
	testing/btest/Baseline/scripts.base.frameworks.notice.suppression-cluster/manager-1.notice.log
	testing/btest/Baseline/scripts.base.protocols.ftp.gridftp/notice.log
2013-03-13 00:26:55 -04:00
Jon Siwek
1f6cac9b6d Merge branch 'master' into topic/jsiwek/file-analysis 2013-03-11 13:20:45 -05:00
Robin Sommer
f830ed3edf s/bro-ids.org/bro.org/g 2013-03-07 19:33:04 -08:00
Jon Siwek
589952f4d9 Merge branch 'master' into topic/jsiwek/file-analysis
Conflicts:
	src/FileAnalyzer.cc
	testing/btest/Baseline/coverage.default-load-baseline/canonified_loaded_scripts.log
2013-03-07 11:06:00 -06:00
Robin Sommer
1bd2f26df3 Merge remote-tracking branch 'origin/topic/seth/notice-framework-updates'
So much nicer!

Closes #954.

* origin/topic/seth/notice-framework-updates:
  Update notice framework documentation to represent the new reality.
  Complete removal of the old table based notice policy mechanism.
  Updates for the notices framework.
2013-03-06 16:45:30 -08:00
Seth Hall
e56a33b6c5 Update notice framework documentation to represent the new reality. 2013-03-06 10:04:50 -05:00
Seth Hall
9f8ba408ba Updates for the notices framework.
- Moved the Notice::notice event and Notice::policy table to both be hooks.

 - Renamed the old Notice::policy to Notice::policy_table and documented it as deprecated.
2013-02-11 14:36:14 -05:00
Jon Siwek
b9d204005d Merge branch 'master' into topic/jsiwek/file-analysis 2013-02-08 09:53:27 -06:00
Seth Hall
f8be65ca56 Merge remote-tracking branch 'origin/master' into topic/seth/metrics-merge 2013-02-07 10:01:13 -05:00
Seth Hall
a2556642e6 Merge remote-tracking branch 'origin/topic/matthias/notary'
* origin/topic/matthias/notary:
  Small cosmetic changes.
  Give log buffer the correct name.
  Simplify delayed logging of SSL records.
  Implement delay-token style SSL logging.
  More style tweaks: replace spaces with tabs.
  Factor notary code into separte file.
  Adhere to Bro coding style guidelines.
  Enhance ssl.log with information from notary.

Closes #928
2013-02-05 02:06:33 -05:00
Jon Siwek
69bd46b3af Fix coverage unit tests due to added file analysis scripts. 2013-01-16 12:08:09 -06:00
Seth Hall
69030fdff3 Merge remote-tracking branch 'origin/master' into topic/seth/metrics-merge 2012-12-17 10:00:34 -05:00
Daniel Thayer
cb7fd7c87c Remove unused attributes
Removed attributes &postprocessor and &match from documentation
and source code.  Removed undocumented attribute &attr from
source code.  Removed internal attribute (&tracked) from documentation.
2012-12-10 15:43:13 -06:00
Seth Hall
d61d175a04 Merge remote-tracking branch 'origin/master' into topic/seth/metrics-merge 2012-12-04 00:17:43 -05:00
Jon Siwek
4e85fe0454 Change hook calls to only be allowed when preceded by "hook" keyword. 2012-11-30 15:39:00 -06:00
Jon Siwek
3b3b05fbbb Clarification in hook documentation. 2012-11-29 10:07:38 -06:00