Once a BasicThread leaves its run() method, a thread is now marked for
cleaning up, and the ThreadMgr will soon join it to release the OS
resources.
Also, adding a function Log::remove_stream() that remove a logging
stream, stopping all writer threads that are associated with it.
Note, however, that removing a *filter* from a stream still doesn't
clean up any threads. The problem is that because of the output paths
potentially being created dynamically it's unclear if the writer
thread will still be needed in the future. We could add clean writers
up with timeouts, but that doesn't sound great either. So for now, the
only way to sure clean up logging threads is to remove the entire
stream.
Also note that cleanup doesn't work with input threads yet, which
don't seem to terminate (at least in the case I tried).
The framework now cycles through callbacks based on a table indexed
by analyzer tags, or the special case of service strings if a given
analyzer is overloaded for multiple protocols (FTP/IRC data). This
lets each protocol script bundle implement the callback locally and
reduces the FAF's external dependencies.
For files that go over a single connection, add connection start time
to handle, so the file id will always differ even if the same connection
parameters are later used to transfer a file (same one or different).
So much nicer!
Closes#954.
* origin/topic/seth/notice-framework-updates:
Update notice framework documentation to represent the new reality.
Complete removal of the old table based notice policy mechanism.
Updates for the notices framework.
The add_action, remove_action, and stop BIFs now go through a queue to
ensure that modifications are made at well-defined times and don't end
up invalidating loop iterators.
The Info record now uses a "table[ActionArgs] of ActionResults", which
allows for simultaneous actions of a given type as long as other args
(fields in the ActionArgs record) are different.
Added the file extraction action and did other misc. cleanup. Most of
the minimal core features/support for file analysis should be working at
this point, just have to start fleshing things out.
- Moved the Notice::notice event and Notice::policy table to both be hooks.
- Renamed the old Notice::policy to Notice::policy_table and documented it as deprecated.
Added a generic gtpv1_message event generated for any GTP message type.
Added specific events for the create/update/delete PDP context
request/response messages.
Addresses #934.
These cases should be avoidable by fixing scripts where they occur and
they can also help catch typos that would lead to unintentional runtime
behavior.
Adding this already revealed several scripts where a field in an inlined
record was never removed after a code refactor.
* origin/topic/bernhard/input-logging-commmon-functions:
add the last of Robins suggestions (separate info-struct for constructors).
port memory leak fix from master
harmonize function naming
move AsciiInputOutput over to threading
and thinking about it, ascii-io doesn't need the separator
change constructors
and factor stuff out the input framework too.
factor out ascii input/output.
std::string accessors to escape_sequence functionality
intermediate commit - it has been over a month since I touched this...
I cleaned up the AsciiInputOutput class somewhat, including renaming
it to AsciiFormatter, renaming some of its methods, and turning the
static methods into members for consistency.
Closes#929.
Moved this functionality to be internal instead of in the script-layer
event handlers. The issue with the later is that bad things can happen
between the time a reporter event handler is dispatched and the time it
is executed, and if bro crashes in that time, the message may never be
seen/logged.
Addressed #930 (and revisits #836).
Sadly there also seems to be another deadlock issue which I am currently
not really able to figure out - on shutdown sometimes (too often) the main
thread + all sqlite threads wait for semaphores or mutexes.
- The feature was primarily added to allow the value to be
modified for cluster based intermediate threshold checks
without requiring the user to write the metrics filter
differently for cluster consideration. It's also a nice
way to calculate some related information to the metric
without accidently applying thresholds to that value.
- Fixed a few small bugs in ftp detect-bruteforcing script
and adapted it to the new threshold value selection feature.
- Fixed several state maintenance issues for intermediate updates.
- Added a new tuning variable Metrics::max_outstanding_global_views
which limits the number of in-flight intermediate updates per
metric filter.
- Changed the default global view threshold percent to 20% (up from 10%)
- Removed default logging. Now a function is available for the new
$period_finished filter field to get the same behavior for logging
named Metrics::write_log.
- Added index rollups for getting multiple metrics result values
as the same time.
Both local and global variables declared with "const" could be modified,
but now expressions that would modify them should generate an error
message at parse-time.
