- I added this to the base scripts because it's very minimal overhead
and we get questions about this all the time. Now that reporter
messages are printed to the console, people will see the message
clearly if they have bad checksums.
In addition to checking for a finished SSL handshake over an FTP
connection, it now also requires that the SSL handshake occurs after
the FTP client requested AUTH GSSAPI, more specifically identifying the
characteristics of GridFTP control channels.
Addresses #891.
* origin/topic/jsiwek/gridftp:
Add memory leak unit test for GridFTP.
Enable GridFTP detection by default. Track/log SSL client certs.
Add analyzer for GSI mechanism of GSSAPI FTP AUTH method.
Add an example of a GridFTP data channel detection script.
more cases.
It will now not only fire after table-reads have been completed,
but also after the last event of a whole-file-read (or whole-db-read, etc.).
The interface also has been extended a bit to allow readers to
directly fire the event should they so choose. This allows the
event to be fired in direct table-setting/event-sending modes,
which was previously not possible.
In the *service* field of connection records, GridFTP control channels
are labeled as "gridftp" and data channels as "gridftp-data".
Added *client_subject* and *client_issuer_subject* as &log'd fields to
SSL::Info record. Also added *client_cert* and *client_cert_chain*
fields to track client cert chain.
GSI authentication involves an encoded TLS/SSL handshake over the FTP
control session. Decoding the exchanged tokens and passing them to an
SSL analyzer instance allows use of all the familiar script-layer events
in inspecting the handshake (e.g. client/server certificats are
available). For FTP sessions that attempt GSI authentication, the
service field of the connection record will have both "ftp" and "ssl".
One additional change is an FTP server's acceptance of an AUTH request
no longer causes analysis of the connection to cease (because further
analysis likely wasn't possible). This decision can be made more
dynamically at the script-layer (plus there's now the fact that further
analysis can be done at least on the GSSAPI AUTH method).
This could, for example, result in duplicate emails being sent (one from
manager and one from worker) if Notice::emailed_types is redef'd in
local.bro (or any script that gets loaded on all cluster nodes).
The problem was that Notice::policy is used to populate the internal
Notice::ordered_policy vector in a priority 10 bro_init handler (in
scripts/base/frameworks/notice/main.bro) and then that is what is used
when applying policy to notices. In order for
scripts/base/frameworks/notice/cluster.bro to prevent Notice::policy
from being used on non-manager nodes, it needs to clear it in a
bro_init hander of higher priority than 10.
This option indicates that the Teredo analyzer should wait until
it sees both sides of a connection using a valid Teredo encapsulation
before issuing a protocol_confirmation. Previous behavior confirmed
on the first instance of a valid encapsulation, which could result
in more false positives (and e.g. bogus entries in known-services.log).
Addresses #890.
It relies on the heuristics of GridFTP data channels commonly default to
SSL mutual authentication with a NULL bulk cipher and that they usually
transfer large datasets (default threshold of script is 1 GB). The
script also defaults to skip_further_processing() after detection to try
to save cycles analyzing the large, benign connection.
Also added a script in base/protocols/conn/polling that generalizes the
process of polling a connection for interesting features. The GridFTP
data channel detection script depends on it to monitor bytes
transferred.
Since the millisecond resolution cannot be harnessed universally and is not
supported by older version of libcurl, we will allow only specifications at the
granularity of seconds.
This commit also fixes a typing issue that causes that prevented the
ElasticSearch timeout to work in the first place: curl_easy_setopt requires a
long but was given a uint64_t.
Authentication Headers with a Payload Len field set to zero would cause
a crash due to invalid memory allocation because the previous code
assumed Payload Len would always be great enough to contain all
mandatory fields of the header. This changes it so the length of
the header is explicitly checked before attempting to extract fields
located past the minimum length (8 bytes) of an Authentication Header.
Crashes due to this are only possible when handling script-layer events
ipv6_ext_headers, new_packet, esp_packet, or teredo_*. Or also when
implementing one of the discarder_check_* family of functions.
Otherwise, Bro correctly parses past such a header.
* origin/topic/bernhard/input-allow_invalid_types:
to be sure - add a small assertion
add an option to the input framework that allows the user to chose to not die upon encountering files/functions.
That's the last feature for 2.1!
* origin/topic/seth/reporter-to-stderr:
A couple of tests for printing reporter messages to STDERR.
Small improvements for printing reporter messages to STDERR.
Reporter warnings and error now print to stderr by default.
Closes#836.
Also reenabling the logs-to-elasticsearch.bro script in
test-all-policy.bro, that seems to work now.
* origin/fastpath:
Reworked how the logs-to-elasticsearch scripts works to stop abusing the logging framework.
This change makes it so when differing logging filters on the same
stream attempt to write to the same writer/path combination, the path
of the filter doing the later write will be automatically adjusted so
that it does not conflict with the other. The path is adjusted by
appending "-N", where N is the smallest integer greater or equal to 2
required to resolve the path name conflict.
Addresses #842.
I've only tested that it compiles, not whether it still works. The
fact that we don't have any tests for this makes me uneasy ...
* remotes/origin/topic/seth/elasticsearch: (35 commits)
Some documentation updates for elasticsearch plugin.
Temporarily removing the ES timeout because it works with signals and is incompatible with Bro threads.
Changed ES index names to localtime and added a meta index.
New script for easily duplicating logs to ElasticSearch.
Some better elasticsearch reliability.
Fixed small elasticsearch problem in configure output.
Re-adding the needed call to FinishedRotation in the ES writer plugin.
Tiny updates.
Bringing elasticsearch branch up to date with master.
Adding a define to make the stdint C macros available.
Adding an extra header.
Fixed a bug with messed up time value passing to elasticsearch.
Small updates and a little standardization for config.h.in naming.
Bug fixes.
Bug fix and feature.
Forgot to call the parent method for DoHeartBeat.
Changed the escaping method.
Flush logs to ES daemon as Bro is shutting down.
Reduce the batch size to 1000 and add a maximum time interval for batches.
Reworked bulk operation string construction to use ODesc and added json escaping.
...
Turns out the finish methods weren't called correctly, caused by a
mess up with method names which all sounded too similar and the wrong
one ended up being called. I've reworked this by changing the
thread/writer/reader interfaces, which actually also simplifies them
by getting rid of the requirement for writer backends to call their
parent methods (i.e., less opportunity for errors).
This commit also includes the following (because I noticed the problem
above when working on some of these):
- The ASCII log writer now includes "#start <timestamp>" and
"#end <timestamp> lines in the each file. The latter supersedes
Bernhard's "EOF" patch.
This required a number of tests updates. The standard canonifier
removes the timestamps, but some tests compare files directly,
which doesn't work if they aren't printing out the same
timestamps (like the comm tests).
- The above required yet another change to the writer API to
network_time to methods.
- Renamed ASCII logger "header" options to "meta".
- Fixes#763 "Escape # when first character in log file line".
All btests pass for me on Linux FC15. Will try MacOS next.
no reply package was seen.
In the fixed version rcode is only set when a reply packet was seen.
Updates for the baseline have been commited separately in the
topic/bernhard/dns-fix branch.
