- Added the known-devices log, and the script to log DHCP hostname per MAC address,
- Added DPD sig,
- Removed some scripts following a discussion with Seth.
* origin/topic/seth/faf-updates: (27 commits)
Undoing the FTP tests I updated earlier.
Update the last two btest FAF tests.
File analysis fixes and test updates.
Fix a bug with getting analyzer tags.
A few test updates.
Some tests work now (at least they all don't fail anymore!)
Forgot a file.
Added protocol description functions that provide a super compressed log representation.
Fix a bug where orig file information in http wasn't working right.
Added mime types to http.log
Clean up queued but unused file_over_new_connections event args.
Add jar files to the default MHR lookups.
Adding CAB files for MHR checking.
Improve malware hash registry script.
Fix a small issue with finding smtp entities.
Added support for files to the notice framework.
Make the custom libmagic database a git submodule.
Add an is_orig parameter to file_over_new_connection event.
Make magic for emitting application/msword mime type less strict.
Disable more libmagic builtin checks that override the magic database.
...
Conflicts:
doc/scripts/DocSourcesList.cmake
scripts/base/init-bare.bro
scripts/test-all-policy.bro
testing/btest/Baseline/coverage.bare-load-baseline/canonified_loaded_scripts.log
testing/btest/Baseline/coverage.default-load-baseline/canonified_loaded_scripts.log
- Fix examples/references in the file analysis how-to/usage doc.
- Add Broxygen-generated docs for file analyzer plugins.
- Break FTP::Info type declaration out in to its own file to get
rid of some circular dependencies (between s/b/p/ftp/main and
s/b/p/ftp/utils).
- policy/frameworks/intel/seen is the new location for the scripts
that push data into the intel framework for checking.
- The new policy/frameworks/intel/do_notice script adds an example
mechanism for data driven notices.
- Several places were just using old variable names or not loading
scripts correctly after they'd been renamed/moved.
- Revert/adjust a change in how HTTP file handles are generated that
broke partial content responses.
- Turn some libmagic builtin checks back on; seems some are actually
useful (e.g. text detection seems to be a builtin). The rule going
forward probably will be only to turn off a builtin if we confirm it
causes issues.
- Removed some tests that are redundant or not necessary anymore because
the generic file analysis tests cover them.
- A couple FTP tests still fail that I think need an actual solution via
script changes.
- Intel importing format has changed (refer to docs).
- All string matching is now case insensitive.
- SMTP intel script has been updated to extract email
addresses correctly.
- Small fix sneaking into the smtp base script to actually
extract individual email addresses in the To: field
correctly.
- Include a link to a virustotal search in the notice sub message field.
- Give all information returned from Team Cymru in the notice message.
- Add more file types to match on to the default set.
Closes#1030.
* origin/topic/seth/packet-filter-updates:
Missed a test fix.
Updating test baselines.
Updates for the PacketFilter framework to simplify it.
Last test update for PacketFilter framework.
Several final fixes for PacketFilter framework.
Packet filter framework checkpoint.
Checkpoint on the packet filter framework.
Initial rework of packet filter framework.
- On-demand access to sumstats results through "return from"
functions named SumStats::request and Sumstats::request_key.
Both functions are tested in standalone and clustered modes.
- $name field has returned to SumStats which simplifies cluster
code and makes the on-demand access stuff possible.
- Clustered results can only be collected for 1 minute from their
time of creation now instead of time of last read.
- Thresholds use doubles instead of counts everywhere now.
- Calculation dependency resolution occurs at start up time now
instead of doing it at observation time which provide a minor
cpu performance improvement. A new plugin registration mechanism
was created to support this change.
- AppStats now has a minimal doc string and is broken into hook-based
plugins.
- AppStats and traceroute detection added to local.bro
Thanks to git this merge was less troublesome that I was afraid it
would be. Not all tests pass yet though (and file hashes have changed
unfortunately).
Conflicts:
cmake
doc/scripts/DocSourcesList.cmake
scripts/base/init-bare.bro
scripts/base/protocols/ftp/main.bro
scripts/base/protocols/irc/dcc-send.bro
scripts/test-all-policy.bro
src/AnalyzerTags.h
src/CMakeLists.txt
src/analyzer/Analyzer.cc
src/analyzer/protocol/file/File.cc
src/analyzer/protocol/file/File.h
src/analyzer/protocol/http/HTTP.cc
src/analyzer/protocol/http/HTTP.h
src/analyzer/protocol/mime/MIME.cc
src/event.bif
src/main.cc
src/util-config.h.in
testing/btest/Baseline/coverage.bare-load-baseline/canonified_loaded_scripts.log
testing/btest/Baseline/coverage.default-load-baseline/canonified_loaded_scripts.log
testing/btest/Baseline/istate.events-ssl/receiver.http.log
testing/btest/Baseline/istate.events-ssl/sender.http.log
testing/btest/Baseline/istate.events/receiver.http.log
testing/btest/Baseline/istate.events/sender.http.log
Closes#1003.
* origin/topic/bernhard/metrics-samples:
finishing touches, make test more robust, rename function in last again
change names of data structures after talking with seth
make last plugin nicer and samplify sqli detector
add tests for sampler
reservoir sampler. untested.
* origin/topic/seth/metrics-merge: (70 commits)
Added protocol to the traceroute detection script.
Added an automatic state limiter for threshold based SumStats.
Removed some dead code in scan.bro
Renamed a plugin hook in sumstats framework.
Move loading variance back to where it should be alphabetically.
Fix a bug with path building in FTP. Came up when changing the path utils.
Fix a few tests.
SumStats test checkpoint.
SumStats tests pass.
Checkpoint for SumStats rename.
Fix another occasional reporter error.
Small updates to hopefully correct reporter errors leading to lost memory.
Trying to fix a state maintenance issue.
Updating DocSourcesList
Updated FTP bruteforce detection and a few other small changes.
Test updates and cleanup.
Fixed the measurement "sample" plugin.
Fix path compression to include removing "/./".
Removed the example metrics scripts. Better real world examples exist now.
Measurement framework is ready for testing.
...
- FileAnalysis::Info is now just a record used for logging, the fa_file
record type is defined in init-bare.bro as the analogue to a
connection record.
- Starting to transfer policy hook triggers and analyzer results to
events.
