* origin/topic/jsiwek/path-func-record-demote:
Fix filter path_func to allow record argument as a subset of stream's columns.
Conflicts:
src/LogMgr.cc
Closes#600.
* origin/topic/jsiwek/filter-rotation:
Remove Log::rotation_control (addresses #572).
Add an optional Log::RotationControl to Log::Filter records.
Closes#572.
entities.
Before, whether they did depended on libmagic. To do that,
smpt/entities.bro gets a new option `never_calc_md5`.
Also restructuring the tests a bit so that load a common
testing-setup.bro scripts that can set a global configuration.
This required adding the ability for RecordVal::CoerceTo functions to
optionally allow orphaning fields. The default is to not allow it, but
now before writing to a log, the value of the stream's columns is coerced
down, if necessary, before passing it on to the filter's path_func.
Addresses #600.
This is because libmagic can return different results depending on
installation.
Note, this still doesn't fix all problems: smtp-entities sets md5
depending on the mime-type returned by libmagic ...
Log rotation is now controlled directly through Filter records.
Also addressed a TODO in the default_path_func regarding the
LogMgr::AddFilter function generating internal filter path
suggestions/fallbacks. Now, if the user doesn't explicitly set a filter
path, the filter's path will be the result of the first call to
default_path_func (happens during the first write to the log). And in
that case the path suggestion argument to the path_func is an empty
string.
The traces for external test-suites are no longer kept inside the
repositories themselves but downloaded separately via curl. This is
because git is pretty bad at dealing with large files.
See the README for more information.
* origin/topic/gilbert/rand-pool:
Updating tests.
Updated uid pools to use integer values instead of strings.
Updating tests.
Test no longer relevant. Need a way to generate and test collisions.
A few minor tweaks to make code less braindead. Fixed-length piece of pool name now only used to hash when determinism is not required; otherwise, whole pool name is used. Note that collisions between pool name hashes will lead to sensitivity to initialization order within the UID generator.
Testing long (>32 character) pool names.
Simple test to verify various pools are not affecting each other.
Some working code. Adds UID pools classified by string. Just compiles and runs; need to go back through and make sure this code is actually doing what I want it to do.
Note, I've removed the collision detection. Seems unlikely to occur
and even if, it's not really that bad.
- The "private interface" section is still tracked, but no longer rendered.
We can judge the usefulness of it later and add back if there's demand.
- Documentation of event handlers was being treated the same as event
declarations. Now handlers are tracked separately, but not currently
rendered in the generated doc output since usefulness is questionable.
- Reorganize top-level 'doc' Makefile target so submodules can easily
add their own doc-generating routines to it. e.g. the Bro project
makes a placeholder 'doc' target, then adds 'restdoc', 'sphinxdoc';
later Broccoli can add it's own target as a dependency for generating
API docs.
- Fixed generated docs for BIFs not being organized under a base/
subdirectory like the original source files.
- Fixed documentation style for function parameters not applying to
functions declared as record fields.
- Misc. script documentation tweaks to address warnings given by Sphinx.
- Log path's are generated in the scripting land
now. The default Log stream ID to path string
mapping works like this:
- Notice::LOG -> "notice"
- Notice::POLICY_LOG -> "notice_policy"
- TestModule::LOG -> "test_module"
- Logging streams updated across all of the shipped
scripts to be more user friendly. Instead of
the logging stream ID HTTP::HTTP, we now have
HTTP::LOG, etc.
- The priorities on some bro_init handlers have
been adjusted to make the process of applying
filters or disabling streams easier for users.
This allows for the RotationControl to be automatically added to the
Log::rotation_control table for the filter's (writer, path) when it is
added to a stream via Log::add_filter. Log::remove_filter now also
removes any RotationControl's associated with the filter from the
Log::rotation_control table.
This can be used from a Bro script to cause Bro to exit with non-zero
status and a message to stderr. Uses of the exit() BIF in scripts
for terminating Bro because of an error were replaced with this one,
which addresses #564.
* origin/topic/gregor/script-polishing:
Tune when c$conn is set.
Set c$conn (for logging) in new_connection() event.
(Semiautomatically) convert the comments in bare-init.bro into autodoc ones.
Add ConnSize_Analyzer's fields to conn.log
SSH base scripts: make sure ConnSizeAnalyzer variables are available before using them.
Make reference to the other script a link
When a given log was rotated, a new log was opened immediately. If that
log was never written to again, those empty logs perpetually rotate, too.
This change makes it so a log won't be created for a given rotation interval
if there was nothing written to it in that interval.
- Since each host in a cluster has it's own view of the metrics
the only time the manager would get a chance for a global view
is the break_interval. This update improves that time. If a
worker crosses 10% of the full threshold, it will send it's
value to the manager which can then ask the rest of the cluster
for a global view. The manager then adds all of the values for
each workers metric indexes together and will do the notice
if it crosses the threshold so that it isn't dependent on
waiting for the break interval to hit. This functionality
works completely independently of the break_interval too. Logging
will happen as normal.
- Small update for SSH bruteforcer detection to match additions in
the metrics framework API.
- The hope is that this update is mostly invisible from anyone's
perspective. The only affect it should have on users is to better
the detection of metric values crossing thresholds on cluster
deployments.
