* origin/topic/seth/faf-updates: (27 commits)
Undoing the FTP tests I updated earlier.
Update the last two btest FAF tests.
File analysis fixes and test updates.
Fix a bug with getting analyzer tags.
A few test updates.
Some tests work now (at least they all don't fail anymore!)
Forgot a file.
Added protocol description functions that provide a super compressed log representation.
Fix a bug where orig file information in http wasn't working right.
Added mime types to http.log
Clean up queued but unused file_over_new_connections event args.
Add jar files to the default MHR lookups.
Adding CAB files for MHR checking.
Improve malware hash registry script.
Fix a small issue with finding smtp entities.
Added support for files to the notice framework.
Make the custom libmagic database a git submodule.
Add an is_orig parameter to file_over_new_connection event.
Make magic for emitting application/msword mime type less strict.
Disable more libmagic builtin checks that override the magic database.
...
Conflicts:
doc/scripts/DocSourcesList.cmake
scripts/base/init-bare.bro
scripts/test-all-policy.bro
testing/btest/Baseline/coverage.bare-load-baseline/canonified_loaded_scripts.log
testing/btest/Baseline/coverage.default-load-baseline/canonified_loaded_scripts.log
- Several places were just using old variable names or not loading
scripts correctly after they'd been renamed/moved.
- Revert/adjust a change in how HTTP file handles are generated that
broke partial content responses.
- Turn some libmagic builtin checks back on; seems some are actually
useful (e.g. text detection seems to be a builtin). The rule going
forward probably will be only to turn off a builtin if we confirm it
causes issues.
- Removed some tests that are redundant or not necessary anymore because
the generic file analysis tests cover them.
- A couple FTP tests still fail that I think need an actual solution via
script changes.
- Missing GeoIP databases now generate warnings/errors that go through
the reporter framework instead of hitting GeoIP's internal use of
stderr
- lookup_location now just queries for country code if the city database
was not loaded, which gets rid of invalid database type errors.
- lookup_location now leaves missing fields uninitialized in the
returned geo_location record value. Updated existing scripts to
check for initialized fields in geo_location records before use.
- Fixed support for GeoIP's IPv6 API and databases
- While updating, I did some further work on the branch.
- New function in the base/utils/files for extracting filenames
from content-dispositions.
- New script for entity excerpt extraction if you aren't interested
in full extraction. The data goes a log field too.
- Some renaming and reorganization of types.
- Updated tests to work with new code.
* origin/topic/jsiwek/smtp-refactor:
Make the doc.coverage test happy.
SMTP script refactor. (addresses #509)
Conflicts:
doc/scripts/DocSourcesList.cmake
policy/protocols/smtp/__load__.bro
policy/protocols/smtp/base/__load__.bro
- policy/ renamed to scripts/
- By default BROPATH now contains:
- scripts/
- scripts/policy
- scripts/site
- *Nearly* all tests pass.
- All of scripts/base/ is loaded by main.cc
- Can be disabled by setting $BRO_NO_BASE_SCRIPTS
- Scripts in scripts/base/ don't use relative path loading to ease use of BRO_NO_BASE_SCRIPTS (to copy and paste that script).
- The scripts in scripts/base/protocols/ only (or soon will only) do logging and state building.
- The scripts in scripts/base/frameworks/ add functionality without causing any additional overhead.
- All "detection" activity happens through scripts in scripts/policy/.
- Communications framework modified temporarily to need an environment variable to actually enable (ENABLE_COMMUNICATION=1)
- This is so the communications framework can be loaded as part
of the base without causing trouble when it's not needed.
- This will be removed once a resolution to ticket #540 is reached.
