This avoids a problem identified by amanbansal2709 in pull
request #1288. I fixed it in a different way than that pull request
by making sure the ts field is always set so that this isssue doesn't
return in the future.
This adds a "policy" hook into the logging framework's streams and
filters to replace the existing log filter predicates. The hook
signature is as follows:
hook(rec: any, id: Log::ID, filter: Log::Filter);
The logging manager invokes hooks on each log record. Hooks can veto
log records via a break, and modify them if necessary. Log filters
inherit the stream-level hook, but can override or remove the hook as
needed.
The distribution's existing log streams now come with pre-defined
hooks that users can add handlers to. Their name is standardized as
"log_policy" by convention, with additional suffixes when a module
provides multiple streams. The following adds a handler to the Conn
module's default log policy hook:
hook Conn::log_policy(rec: Conn::Info, id: Log::ID, filter: Log::Filter)
{
if ( some_veto_reason(rec) )
break;
}
By default, this handler will get invoked for any log filter
associated with the Conn::LOG stream.
The existing predicates are deprecated for removal in 4.1 but continue
to work.
* origin/topic/jsiwek/empty-lines:
Add 'smtp_excessive_pending_cmds' weird
Fix SMTP command string comparisons
Improve handling of empty lines in several text protocol analyzers
Add rate-limiting sampling mechanism for weird events
Teach timestamp canonifier about timestamps before ~2001
SMB error handling improved. The analyzer isn't destroyed when a problem
is encoutered anymore. The flowbuffer in the parser is now flushed and
the analyzer is set to resync against an SMB command. This was needed
because there is some state about open files that is kept within the
parser itself which was being destroyed and that was causing analysis
after content gaps or parse errors to be faulty. The new mechanism
doesn't detroy the parser so parsing after gaps is improved.
DCE_RPC handling in SMB is improved in the edge case where a drive
mapping isn't seen. There is a new const named SMB::pipe_filenames
which is used as a heuristic for identifying "files" opened on named
pipe shares. If the share mapping type isn't known and a filename
in this set is found, the share type will change to "PIPE" by
generating an event named "smb_pipe_connect_heuristic". Reads and
writes to that file will be sent to the DCE_RPC analyzer instead of
to the files framework.
The concept of "unknown" share types has been removed due to the new
heuristic detection of share types.
Some general clean up of how the SMB cmd log is written and when.
- Looser coupling between these analyzers.
- New ntlm.log (still pretty early)
- Improved string handling for NTLM (convert UTF16 to UTF8)
- SMB2 analyzer now supports GSSAPI.
- Improved abstraction of DCE_RPC operations (still not finished)
- Lots of whitespace cleanup.
There were some cases where the log would be missing a field
or data wouldn't get sent to file analysis. At least some of
this is fixed now and I get confused a bit less when I look
at the logs now.
Also, I made the default handling "FILE" so that things like
FILE_UNKNOWN wouldn't show up in the logs so regularly. It's
technically correct that way, but it doesn't look good and it's
correct as FILE often enough that it make sense to make it the
default I think.
- Actually get the path into the smb_files.log now.
- When a share root is having the "create" message used on it,
instead of giving a null file name, now give a special
indicator of "<share_root>".
- Update test baselines.
This would come up when a string is UTF-16 containing characters
outside of straight ASCII. The file analysis framework uses
CheckString to create file IDs which can't cope with the NULL bytes.
- Add beginning of infrastructure for pipe support in SMB2.
- Improve identification of non-file tree mappings.
- Stop passing pipe data to the file analysis framework.
- Reduce log volume in smb_files.log by watching for repeated
files being seen so that you don't end up with nearly
the exact same log line over and over and over.
- Lots of little whitespace and indentation changes.
