- New script extracted from weird.bro to implement the
connection related "weird" data into an optionally
loaded script.
- Adjusted the default notice tuning to stop ignoring
the connection related weirds since they aren't loaded
by default anymore.
- Certificate validation volume has been greatly cut down by
caching results.
- Cert hashing is now done in one place instead of being repeated
everywhere a cert hash was needed.
- Some small cleanups for notice suppression that should greatly reduce
duplicate notice volume about invalid certificates.
- With the software-browser-plugins script you can watch for Omniture
advertising servers to grab the list of installed plugins.
- I reorganized the plugin detection a bit too to abstract it better.
- Removed the WEB_ prefix from all of the Software::Type HTTP enums.
They were essentially redundant due to the full name already being
HTTP::SERVER (for example).
- I was wildly misunderstanding the semantics of the
connection_state_remove event. That's fixed now in
my brain and in the script.
- If a service isn't detected, logging is delayed by
2 minutes to try and allow for another connection
to happen that actually does the protocol correctly
and detectably.
- scan.bro and hot.conn.bro will be returning soon.
- The rest are going to return as updated protocol analysis
scripts and new/updated frameworks later.
Updated README and collected coverage-related tests in a common dir.
There are still coverage failures resulting from either the following
scripts not being @load'd in the default bro mode:
base/frameworks/time-machine/notice.bro
base/protocols/http/partial-content.bro
base/protocols/rpc/main.bro
Or the following result in errors when @load'd:
policy/protocols/conn/scan.bro
policy/hot.conn.bro
If these are all scripts-in-progress, can we move them all to live
outside the main scripts/ directory until they're ready?
- protocols/ssl/expiring-certs uses time based information from
certificates to determine if they will expire soon, have already
expired, or haven't yet become valid.
- protocols/ssl/extract-certs-pem is a script for taking certs off
the line and converting them to PEM certificates with the openssl
command line tool then dumping them to a file.
- Removed an notice definition from the base SSL scripts.
- Moved a logging stream ID into the export section for known-services
and bumped priority for creating the stream.
- Adding configuration knobs for the SQL injection attack detection
script and renaming the HTTP::SQL_Injection_Attack notice to
HTTP::SQL_Injection_Attack_Against
- Bumped priority when creating Known::CERTS_LOG.
- Log path's are generated in the scripting land
now. The default Log stream ID to path string
mapping works like this:
- Notice::LOG -> "notice"
- Notice::POLICY_LOG -> "notice_policy"
- TestModule::LOG -> "test_module"
- Logging streams updated across all of the shipped
scripts to be more user friendly. Instead of
the logging stream ID HTTP::HTTP, we now have
HTTP::LOG, etc.
- The priorities on some bro_init handlers have
been adjusted to make the process of applying
filters or disabling streams easier for users.
- Missing GeoIP databases now generate warnings/errors that go through
the reporter framework instead of hitting GeoIP's internal use of
stderr
- lookup_location now just queries for country code if the city database
was not loaded, which gets rid of invalid database type errors.
- lookup_location now leaves missing fields uninitialized in the
returned geo_location record value. Updated existing scripts to
check for initialized fields in geo_location records before use.
- Fixed support for GeoIP's IPv6 API and databases
* origin/topic/gregor/script-polishing:
Tune when c$conn is set.
Set c$conn (for logging) in new_connection() event.
(Semiautomatically) convert the comments in bare-init.bro into autodoc ones.
Add ConnSize_Analyzer's fields to conn.log
SSH base scripts: make sure ConnSizeAnalyzer variables are available before using them.
Make reference to the other script a link
- Since each host in a cluster has it's own view of the metrics
the only time the manager would get a chance for a global view
is the break_interval. This update improves that time. If a
worker crosses 10% of the full threshold, it will send it's
value to the manager which can then ask the rest of the cluster
for a global view. The manager then adds all of the values for
each workers metric indexes together and will do the notice
if it crosses the threshold so that it isn't dependent on
waiting for the break interval to hit. This functionality
works completely independently of the break_interval too. Logging
will happen as normal.
- Small update for SSH bruteforcer detection to match additions in
the metrics framework API.
- The hope is that this update is mostly invisible from anyone's
perspective. The only affect it should have on users is to better
the detection of metric values crossing thresholds on cluster
deployments.
- fixing some Metrics::add_data() call signatures
- slight refactors to cluster framework @if and adding a NONE NodeType for
so local_node_type() will return that instead of just emitting an error
when cluster mode isn't enabled
- `make restdoc` target now uses bro's bare-mode
- bro scripts generated from bifs now really only live in the build/src/base/
directory and changed the DocSourcesList.cmake to dynamically figure out
what bifs exist by looking in src/ instead of build/src/
- add some missing @load dependencies
- While updating, I did some further work on the branch.
- New function in the base/utils/files for extracting filenames
from content-dispositions.
- New script for entity excerpt extraction if you aren't interested
in full extraction. The data goes a log field too.
- Some renaming and reorganization of types.
- Updated tests to work with new code.
* origin/topic/jsiwek/smtp-refactor:
Make the doc.coverage test happy.
SMTP script refactor. (addresses #509)
Conflicts:
doc/scripts/DocSourcesList.cmake
policy/protocols/smtp/__load__.bro
policy/protocols/smtp/base/__load__.bro
- Metrics API is much more similar to the Logging framework's API now.
- Filters define all output and metrics collection now.
- Initial attempt at thresholding and generating notices.
- Fixing the parts of the `make restdoc` and `make doc` process that were
broken by the last Bro script re-organization
- Generated documentation for Bro scripts derived from BiFs now use the
original BiF source file as the "original source file" link
- Renaming of the internal POLICYDEST definition and other misc places that
refer to "policy" scripts; that terminology doesn't make total sense now
- Added a documentation blacklist reminder test that will fail if there's
scripts that are blacklisted from being documentated because they're still
in progress
- Some minor Bro script changes to fix small @load dependency errors
Addresses #543
- policy/ renamed to scripts/
- By default BROPATH now contains:
- scripts/
- scripts/policy
- scripts/site
- *Nearly* all tests pass.
- All of scripts/base/ is loaded by main.cc
- Can be disabled by setting $BRO_NO_BASE_SCRIPTS
- Scripts in scripts/base/ don't use relative path loading to ease use of BRO_NO_BASE_SCRIPTS (to copy and paste that script).
- The scripts in scripts/base/protocols/ only (or soon will only) do logging and state building.
- The scripts in scripts/base/frameworks/ add functionality without causing any additional overhead.
- All "detection" activity happens through scripts in scripts/policy/.
- Communications framework modified temporarily to need an environment variable to actually enable (ENABLE_COMMUNICATION=1)
- This is so the communications framework can be loaded as part
of the base without causing trouble when it's not needed.
- This will be removed once a resolution to ticket #540 is reached.
