- Beginning rework of metrics interface.
- Updates to URI based SQLI detection to match metrics framework.
- Addition to SQLI regex to catch use of XOR.
The main change is that the postprocessor commands are no longer run
by the log writers themselves. Instead, the writers send back a
message to the log mgr once they have rotated. The manager then calls
a script level function to do somethign with the rotated file. By
default, it will be renamed to somethingn nice and then a
postprocessor shell command will be run on it if defined.
Pieces going into this:
- Terminology change: "postprocessor" now refers to a script
*function*. In addition, there are "postprocessor commands", which
are shell commands that may be triggered by the function to run on
a rotated file.
- The RotationInfo record now comes with all the information that
was previously provided internally to the C++ function running the
post-processor command.
- Changing the default time format to %Y-%m-%d-%H-%M-%S
- rotation_path_func is gone
- The default postprocessor function is defined individually by
each LogWriter in frameworks/logging/plugin/*
- The interface to postprocessor shell commands remains the same.
Needs a bit more testing ...
- message header state tracking is now done by handling mime_one_header
instead of parsing the data in the smtp_data event
- changed the logging point to be when an smtp_reply is seen in response
to the end of a DATA section
- the smtp package now uses it's own mime script and logging stream for
logging entities, extraction, etc.
- fixes for mime file extraction: now logs the extracted file name, and
the count of extracted files needed to be maintained in the State record
* origin/fastpath:
Normalize Notice::Type identifiers per convention. (closes#484)
Another fix to the default-loaded-scripts test.
Add new piped_exec BiF.
Revert "Fixes for email_notice_to() function."
Fixes for email_notice_to() function.
sed on some platforms like OS X (maybe FreeBSD in general) won't recognize
semi-colon delimited commands as multiple commands, instead use the -e
option multiple times to build the command list.
Newline characters need escaping so that an echo command can interpret
them into a newline in the output piped to sendmail, else sendmail can't
parse the headers correctly.
I made the echo command a configurable option of the notice framework
in case `echo -e` is overshadowed by some shell-specific implementation
that doesn't support that option for interpreting char sequences.
* origin/fastpath:
Updating baseline for default loaded scripts... again.
Update core.conn-uid test baseline.
Rename/change policy.misc.loaded-scripts, again baselines default loaded scripts
Changes to unit tests that rely on libmagic.
Change policy.misc.loaded-scripts tests.
core.load-pkg test now insensitive to default-loaded scripts.
LogWriterAscii now prints time values w/ constant 6 digit precision.
- The CMake targets for generating reST docs from policy scripts are now
automatically generated via the genDocSourcesList.sh script
- Fixed a lot of parsing errors in policy scripts that I saw along the way
The currently loading script's path is prepended (vs. appended) to
BROPATH to search for the @load'd file to prevent being overshadowed by
scripts/directories/packages in the normal BROPATH with the same name.
This extra search path should also only be prepended to BROPATH in the
case when the @load'd file we're looking for is actually relative
(i.e. the name starts with "./" or "../").
- Updates to cluster framework to finish the separation between broctl and bro
- Extension technique for extending notice emails with extra content.
- Deleting the connection record from notices after calling apply_policy.
It may have been causing some load and memory issues from copying
lots of data to other cluster members. This is a test to see if we are
right about the memory trouble.
- Abstracted some of the notice actions into separate scripts.
-
- Lots of small cleanup and fixes.
If a test doesn't rely on libmagic, mime type related columns of baselined
logs are filtered out.
If a test does rely on libmagic, it needs to use the TEST-REQUIRES btest
macro to check that the bro build supports it, and then mime type related
columns of logs can be normalized via a logging filter to reduce sensitivity
to varying version of libmagic.
When calling an Analyzer's method to remove a child analyzer, we now
postpone the actual removal to later, as otherwise the call to Done()
might trigger further analyzer activity that can interfere with code
running after that that triggered the removal.
This should fix the SSL assertion crashes that we have seen.
This change is a bit tricky internally, but the trace-based tests
produce the same output as before so things should be fine ...
If a test doesn't rely on libmagic, mime type related columns of baselined
logs are filtered out.
If a test does rely on libmagic, it needs to use the TEST-REQUIRES btest
macro to check that the bro build supports it, and then mime type related
columns of logs can be normalized via a logging filter to reduce sensitivity
to varying version of libmagic.
* origin/topic/script-load-changes:
Fix reST file name associated w/ stdin when in doc mode (closes#497)
Update @prefixes test.
Rewrite a test using btest's TEST-START-FILE directive
Fix @unload'd files from generating bro_script_loaded event.
Renaming a test better.
Reimplementation of the @prefixes statement.
Fix accidental overwrite of BROPATH copy.
Make @load statements recognize relative paths.
* origin/topic/jsiwek/irc-orig:
Shorten what's displayed in the IRC's log mime_type column for DCC transfers
Add IRC unit tests.
Small tweak to IRC event handlder priorities
Fix IRC analyzer supplying wrong type to irc_dcc_message event.
Changes to IRC analyzer and events (addresses #469).
