* origin/topic/jsiwek/http-1xx-replies:
Change logging of HTTP 1xx responses to occur in their own columns.
Fix handling of HTTP 1xx response codes (addresses #411).
- I was wildly misunderstanding the semantics of the
connection_state_remove event. That's fixed now in
my brain and in the script.
- If a service isn't detected, logging is delayed by
2 minutes to try and allow for another connection
to happen that actually does the protocol correctly
and detectably.
- A null value no longer fits since if there is no body
a value of zero makes sense. Previously, a null value would
makes sense because the Content-Length header may not have
been sent which would leave the field null.
- I removed the byte loss estimate since everything indicated that
it could frequently be inaccurate.
- It has it's own logging stream now. Notices are only used
when too much loss has been detected (CaptureLoss::Too_Much_Loss)
- The gap_report event isn't used anymore. I'm scheduling events
to get the same effect by using the get_gap_summary BiF to
collect the gap summary on demand.
- scan.bro and hot.conn.bro will be returning soon.
- The rest are going to return as updated protocol analysis
scripts and new/updated frameworks later.
Updated README and collected coverage-related tests in a common dir.
There are still coverage failures resulting from either the following
scripts not being @load'd in the default bro mode:
base/frameworks/time-machine/notice.bro
base/protocols/http/partial-content.bro
base/protocols/rpc/main.bro
Or the following result in errors when @load'd:
policy/protocols/conn/scan.bro
policy/hot.conn.bro
If these are all scripts-in-progress, can we move them all to live
outside the main scripts/ directory until they're ready?
Changed the parser to not treat 1xx response codes as a final answer
to an unanswered request -- a later response is still expected.
The scripting layer will also not finish a request-reply pair when
seeing 1xx's, instead it logs both the 1xx and final response messages
with associated information of the current request as they're seen.
- protocols/ssl/expiring-certs uses time based information from
certificates to determine if they will expire soon, have already
expired, or haven't yet become valid.
- protocols/ssl/extract-certs-pem is a script for taking certs off
the line and converting them to PEM certificates with the openssl
command line tool then dumping them to a file.
- Fixed the notice_policy log.
- Predicate functions are now logged.
- Predicate functions are now optional. If not given, it's assumed that
the result should always apply.
* origin/topic/seth/notice-suppression:
Updated a notice related baseline and added a necessary @load line.
Notice suppression clean up and notice/cluster integrtion fixes.
Updates for notice suppression to use the &create_expire attribute
Small, mostly cosmetic updates and fixing a test.
Fix crash on exit (addresses #607).
Duplicate notice suppression.
Closes#623.
* origin/fastpath:
Fix PktSrc setting next_timestamp even when no packet available
Fix lack of NUL-termination in to_upper/to_lower BIF's return val.
Fixing unit tests and some minor bugs.
Fix broctl cluster log rotation (addresses #619)
* master:
- Worker raised notices are printed a single time by the manager now.
- Cluster/notices integration cleaned up.
- New tests for cluster/notice integration.
- Notice suppression fixes and tests.
* origin/topic/jsiwek/filter-rotation:
Remove Log::rotation_control (addresses #572).
Add an optional Log::RotationControl to Log::Filter records.
Closes#572.
entities.
Before, whether they did depended on libmagic. To do that,
smpt/entities.bro gets a new option `never_calc_md5`.
Also restructuring the tests a bit so that load a common
testing-setup.bro scripts that can set a global configuration.
- The value of the content-length headers has now been removed
but it could be added back locally at an installation by a user.
- Added fields to indicate if some parsing interruption happened
during the body transfer.
- Closes#581
- If a protocol violation happens, Bro now logs what it has seen
up until the protocol violation and deletes the c$ssl record
so that a long lived connection with a protocol violation does
continue to hold the memory.
- Removed an notice definition from the base SSL scripts.
- Moved a logging stream ID into the export section for known-services
and bumped priority for creating the stream.
- Adding configuration knobs for the SQL injection attack detection
script and renaming the HTTP::SQL_Injection_Attack notice to
HTTP::SQL_Injection_Attack_Against
- Bumped priority when creating Known::CERTS_LOG.
- The test needed to append the second notice the event queue so that
the notice would be suppressed correctly.
- Setting $suppress_for manually in apply_policy since it doesn't pick
up the &default when directly creating the record inside the NOTICE call.
- Duplicate notices are discovered with the new Notice::Info
field $identifier. It's a string that is left up to the
notice implementor to define which would indicate a
fundamentally duplicate notice. The field is optional and
if it's not included it's not possible for notice
suppression to take place.
- Duplicate notices are suppressed by default for the interval
defined by the Notice::default_suppression_interval variable
(1 hour by default).
- A new notice action was defined ACTION_NO_SUPPRESS to prevent
suppression for a specific notice instance. A convenience set
named not_suppressed_types was also created to not suppress
entire notice types.
- A new field was added to the PolicyItem type to modify the length
of time a notice should be suppressed if the predicate matches.
The field is named $suppress_for. This name makes the code more
readable like this: $suppress_for = 1day
- New events were created to give visibility into the notice
framework's suppression activity.
- event Notice::begin_suppression(n: Notice::Info)
- event Notice::suppressed(n: Notice::Info)
- event Notice::end_suppression(n: Notice::Info)
- The suppression.bro script doesn't have a baseline because
it is causing a segfault in Bro. This one test is the
reason that this is being integrated into a branch instead
of master.
Log rotation is now controlled directly through Filter records.
Also addressed a TODO in the default_path_func regarding the
LogMgr::AddFilter function generating internal filter path
suggestions/fallbacks. Now, if the user doesn't explicitly set a filter
path, the filter's path will be the result of the first call to
default_path_func (happens during the first write to the log). And in
that case the path suggestion argument to the path_func is an empty
string.
- This is an attempt at fixing the memory issues brought about by
the introduction of the new SSL analyzer. My initial testing
shows a hefty memory saving.
- Reorganize top-level 'doc' Makefile target so submodules can easily
add their own doc-generating routines to it. e.g. the Bro project
makes a placeholder 'doc' target, then adds 'restdoc', 'sphinxdoc';
later Broccoli can add it's own target as a dependency for generating
API docs.
- Fixed generated docs for BIFs not being organized under a base/
subdirectory like the original source files.
- Fixed documentation style for function parameters not applying to
functions declared as record fields.
- Misc. script documentation tweaks to address warnings given by Sphinx.
* origin/topic/gilbert/ascii-header:
Updated tests; removed net type from type conversion code.
Updated header format (see #558)
Header modification to LogWriterAscii to make it easier for scripts to understand bro log files.
Notes:
- I've refactored the code a bit, also adapting the style a bit.
Also edited the header format slightly.
- I'm skipping the testing/btest/profiles directory, which seems
unrelated.
- I'm also skipping the baseline updates as they weren't
up-to-date anymore. Will update them in a subsequent commit.
