This allows the path for the default filter to be specified explicitly
when creating a stream and reduces the need to rely on the default path
function to magically supply the path.
The default path function is now only used if, when a filter is added to
a stream, it has neither a path nor a path function already.
Adapted the existing Log::create_stream calls to explicitly specify a
path value.
Addresses BIT-1324
These functions are now deprecated in favor of alternative versions that
return a vector of strings rather than a table of strings.
Deprecated functions:
- split: use split_string instead.
- split1: use split_string1 instead.
- split_all: use split_string_all instead.
- split_n: use split_string_n instead.
- cat_string_array: see join_string_vec instead.
- cat_string_array_n: see join_string_vec instead.
- join_string_array: see join_string_vec instead.
- sort_string_array: use sort instead instead.
- find_ip_addresses: use extract_ip_addresses instead.
Changed functions:
- has_valid_octets: uses a string_vec parameter instead of string_array.
Addresses BIT-924, BIT-757.
This cleans up most of the warnings from sphinx (broken :doc: links,
broxygen role misuses, etc.). The remaining ones should be harmless,
but not quick to silence.
I found that the README for each component was a copy from the actual
repo, so I turned those in to symlinks so they don't get out of date.
Thanks to git this merge was less troublesome that I was afraid it
would be. Not all tests pass yet though (and file hashes have changed
unfortunately).
Conflicts:
cmake
doc/scripts/DocSourcesList.cmake
scripts/base/init-bare.bro
scripts/base/protocols/ftp/main.bro
scripts/base/protocols/irc/dcc-send.bro
scripts/test-all-policy.bro
src/AnalyzerTags.h
src/CMakeLists.txt
src/analyzer/Analyzer.cc
src/analyzer/protocol/file/File.cc
src/analyzer/protocol/file/File.h
src/analyzer/protocol/http/HTTP.cc
src/analyzer/protocol/http/HTTP.h
src/analyzer/protocol/mime/MIME.cc
src/event.bif
src/main.cc
src/util-config.h.in
testing/btest/Baseline/coverage.bare-load-baseline/canonified_loaded_scripts.log
testing/btest/Baseline/coverage.default-load-baseline/canonified_loaded_scripts.log
testing/btest/Baseline/istate.events-ssl/receiver.http.log
testing/btest/Baseline/istate.events-ssl/sender.http.log
testing/btest/Baseline/istate.events/receiver.http.log
testing/btest/Baseline/istate.events/sender.http.log
* origin/topic/bernhard/thread-cleanup:
and just to be really sure - always make threads go through OnWaitForStop
hopefully finally fix last interesting race-condition
it is apparently getting a bit late for changes at important code...
spoke to soon (forgot to comment in line again).
Change thread shutdown again to also work with input framework.
Changing semantics of thread stop methods.
Support for cleaning up threads that have terminated.
There's now a new directory "src/protocols/", and the plan is for each
protocol analyzer to eventually have its own subdirectory in there
that contains everything it defines (C++/pac/bif). The infrastructure
to make that happen is in place, and two analyzers have been
converted to the new model, HTTP and SSL; there's no further
HTTP/SSL-specific code anywhere else in the core anymore (I believe :-)
Further changes:
- -N lists available plugins, -NN lists more details on what these
plugins provide (analyzers, bif elements). (The latter does not
work for analyzers that haven't been converted yet).
- *.bif.bro files now go into scripts/base/bif/; and
scripts/base/bif/plugins/ for bif files provided by plugins.
- I've factored out the bifcl/binpac CMake magic from
src/CMakeLists.txt to cmake/{BifCl,Binpac}
- There's a new cmake/BroPlugin that contains magic to allow
plugins to have a simple CMakeLists.txt. The hope is that
eventually the same CMakeLists.txt can be used for compiling a
plugin either statically or dynamically.
- bifcl has a new option -c that changes the code it generates so
that it can be used with a plugin.
TODOs:
- "make install" is probably broken.
- Broxygen is probably broken for plugin-defined events.
- event groups are broken (do we want to keep them?)
Once a BasicThread leaves its run() method, a thread is now marked for
cleaning up, and the ThreadMgr will soon join it to release the OS
resources.
Also, adding a function Log::remove_stream() that remove a logging
stream, stopping all writer threads that are associated with it.
Note, however, that removing a *filter* from a stream still doesn't
clean up any threads. The problem is that because of the output paths
potentially being created dynamically it's unclear if the writer
thread will still be needed in the future. We could add clean writers
up with timeouts, but that doesn't sound great either. So for now, the
only way to sure clean up logging threads is to remove the entire
stream.
Also note that cleanup doesn't work with input threads yet, which
don't seem to terminate (at least in the case I tried).
First step - factored out everything the logging classes
use ( so only output ).
Moved the script-level configuration to logging/main,
and made the individual writers just refer to it -
no idea if this is good design. It works. But I am happy
about opinions :)
Next step - add support for input...
Also reenabling the logs-to-elasticsearch.bro script in
test-all-policy.bro, that seems to work now.
* origin/fastpath:
Reworked how the logs-to-elasticsearch scripts works to stop abusing the logging framework.
This change makes it so when differing logging filters on the same
stream attempt to write to the same writer/path combination, the path
of the filter doing the later write will be automatically adjusted so
that it does not conflict with the other. The path is adjusted by
appending "-N", where N is the smallest integer greater or equal to 2
required to resolve the path name conflict.
Addresses #842.
* origin/topic/bernhard/reader-info:
fix small bug - now configuration actually is passed.
add mode to readerinfo - no need to have it separately everywhere anymore.
introduce reader-info struct analogous to writer-info.
Introduce support for a table of key/value pairs with further configuration options, with the same userinterface as in the logging interface.
make writer-info work when debugging is enabled
Conflicts:
testing/btest/Baseline/scripts.base.frameworks.input.event/out
testing/btest/Baseline/scripts.base.frameworks.input.executeraw/out
testing/btest/Baseline/scripts.base.frameworks.input.raw/out
testing/btest/Baseline/scripts.base.frameworks.input.rereadraw/out
testing/btest/Baseline/scripts.base.frameworks.input.tableevent/out
Closes#841.
* robin/topic/writer-info:
Extending the log writer DoInit() API.
Reworking log writer API to make it easier to pass additional information to a writer's initialization method.
Conflicts:
src/logging/WriterBackend.cc
src/logging/WriterBackend.h
src/logging/WriterFrontend.cc
We now pass in a Info struct that contains:
- the path name (as before)
- the rotation interval
- the log_rotate_base_time in seconds
- a table of key/value pairs with further configuration options.
To fill the table, log filters have a new field "config: table[string]
of strings". This gives a way to pass arbitrary values from
script-land to writers. Interpretation is left up to the writer.
Also splits calc_next_rotate() into two functions, one of which is
thread-safe and can be used with the log_rotate_base_time value from
DoInit().
Includes also updates to the None writer:
- It gets its own script writers/none.bro.
- New bool option LogNone::debug to enable debug output. It then
prints out all the values passed to DoInit(). That's used by a
btest test to ensure the new DoInit() values are right.
- Fixed a bug that prevented Bro from terminating..
(scripts.base.frameworks.logging.rotate-custom currently fails.
Haven't yet investigated why.)
* origin/topic/robin/dataseries:
Moving trace for rotation test into traces directory.
Fixing a rotation race condition at termination.
Portability fixes.
Extending DS docs with some examples.
Updating doc.
Fixing pack_scale and time-as-int.
Adding format specifier to DS spec to print out double as %.6f.
DataSeries updates and fixes.
DataSeries tuning.
Tweaking DataSeries support.
Extending log post-processor call to include the name of the writer.
Removing an unnecessary const cast.
DataSeries TODO list with open issues/questions.
Starting DataSeries HowTo.
Additional test output canonification for ds2txt's timestamps.
In threads, an internal error now immediately aborts.
DataSeries cleanup.
Working on DataSeries support.
Merging in DataSeries support from topic/gilbert/logging.
Fixing threads' DoFinish() method.
* origin/fastpath:
Fix PktSrc setting next_timestamp even when no packet available
Fix lack of NUL-termination in to_upper/to_lower BIF's return val.
Fixing unit tests and some minor bugs.
Fix broctl cluster log rotation (addresses #619)
* master:
* origin/topic/jsiwek/filter-rotation:
Remove Log::rotation_control (addresses #572).
Add an optional Log::RotationControl to Log::Filter records.
Closes#572.
Log rotation is now controlled directly through Filter records.
Also addressed a TODO in the default_path_func regarding the
LogMgr::AddFilter function generating internal filter path
suggestions/fallbacks. Now, if the user doesn't explicitly set a filter
path, the filter's path will be the result of the first call to
default_path_func (happens during the first write to the log). And in
that case the path suggestion argument to the path_func is an empty
string.
- Reorganize top-level 'doc' Makefile target so submodules can easily
add their own doc-generating routines to it. e.g. the Bro project
makes a placeholder 'doc' target, then adds 'restdoc', 'sphinxdoc';
later Broccoli can add it's own target as a dependency for generating
API docs.
- Fixed generated docs for BIFs not being organized under a base/
subdirectory like the original source files.
- Fixed documentation style for function parameters not applying to
functions declared as record fields.
- Misc. script documentation tweaks to address warnings given by Sphinx.
- Log path's are generated in the scripting land
now. The default Log stream ID to path string
mapping works like this:
- Notice::LOG -> "notice"
- Notice::POLICY_LOG -> "notice_policy"
- TestModule::LOG -> "test_module"
- Logging streams updated across all of the shipped
scripts to be more user friendly. Instead of
the logging stream ID HTTP::HTTP, we now have
HTTP::LOG, etc.
- The priorities on some bro_init handlers have
been adjusted to make the process of applying
filters or disabling streams easier for users.
This allows for the RotationControl to be automatically added to the
Log::rotation_control table for the filter's (writer, path) when it is
added to a stream via Log::add_filter. Log::remove_filter now also
removes any RotationControl's associated with the filter from the
Log::rotation_control table.
* topic/robin/rotation-pp:
Adding a default_path_func that makes the default naming scheme script-level controlled.
Reworking logging's postprocessor logic.
Conflicts:
scripts/base/frameworks/logging/main.bro
testing/btest/policy/frameworks/logging/rotate-custom.bro
- policy/ renamed to scripts/
- By default BROPATH now contains:
- scripts/
- scripts/policy
- scripts/site
- *Nearly* all tests pass.
- All of scripts/base/ is loaded by main.cc
- Can be disabled by setting $BRO_NO_BASE_SCRIPTS
- Scripts in scripts/base/ don't use relative path loading to ease use of BRO_NO_BASE_SCRIPTS (to copy and paste that script).
- The scripts in scripts/base/protocols/ only (or soon will only) do logging and state building.
- The scripts in scripts/base/frameworks/ add functionality without causing any additional overhead.
- All "detection" activity happens through scripts in scripts/policy/.
- Communications framework modified temporarily to need an environment variable to actually enable (ENABLE_COMMUNICATION=1)
- This is so the communications framework can be loaded as part
of the base without causing trouble when it's not needed.
- This will be removed once a resolution to ticket #540 is reached.
2011-08-05 23:09:53 -04:00
Renamed from policy/frameworks/logging/base.bro (Browse further)
