That means that, for example, connections that are terminated with an alert during the
handshake never appear in the ssl.log.
This patch changes this behavior - now all ssl connections that fire any event are logged.
The protocol confirmation of the ssl analyzer is moved to the client_hello instead to
the server hello. Furthermore, an additional field is added to ssl.log, which indicates
if a connection has been established or not (which probably indicates a handshake problem).
* topic/robin/http-connect:
HTTP fix for output handlers.
Expanding the HTTP methods used in the signature to detect HTTP traffic.
Updating submodule(s).
Fixing removal of support analyzers, plus some tweaking and cleanup of CONNECT code.
HTTP CONNECT proxy support.
BIT-1132 #merged
CONNECT code.
Removal of support analyzers was broken. The code now actually doesn't
delete them immediately anymore but instead just flags them as
disabled. They'll be destroyed with the parent analyzer later.
Also includes a new leak tests exercising the CONNECT code.
Lines starting # with '#' will be ignored, and an empty message aborts
the commit. # On branch topic/robin/http-connect # Changes to be
committed: # modified: scripts/base/protocols/http/main.bro #
modified: scripts/base/protocols/ssl/consts.bro # modified:
src/analyzer/Analyzer.cc # modified: src/analyzer/Analyzer.h #
modified: src/analyzer/protocol/http/HTTP.cc # new file:
testing/btest/core/leaks/http-connect.bro # modified:
testing/btest/scripts/base/protocols/http/http-connect.bro # #
Untracked files: # .tags # changes.txt # conn.log # debug.log # diff #
mpls-in-vlan.patch # newfile.pcap # packet_filter.log # reporter.log #
src/PktSrc.cc.orig # weird.log #
This changes the internal type that is used to signal that a vector
is unspecified from any to void.
I tried to verify that the behavior of Bro is still the same. After
a lot of playing around, I think everything still should worl as before.
However, it might be good for someone to take a look at this.
addresses BIT-1144
The amount of timers involved in DNS::PendingMessage tables'
expiration attributes have a significant performance hit. Instead the
script now relies solely on maximum thresholds for pending message
quantities to limit amount of accumulated state. There's a new option,
"DNS::max_pending_query_ids", to limit the number outstanding messages
across all DNS query IDs ("DNS::max_pending_msgs" still limits number
of outstanding messages for a *given* query ID).
* origin/topic/jsiwek/dns-improvements:
Rewrite DNS state tracking which matches queries and replies.
Change dns.log to include only standard DNS queries.
Improve DNS analysis.
* origin/topic/jsiwek/tcp-improvements:
Add script to detect filtered TCP traces, addresses BIT-1119.
BIT-1119 #merged
If we could only disable an event handler dynamically ...
If reading a trace file w/ only TCP control packets, a warning is
emitted to suggest the 'detect_filtered_traces' option if the user
doesn't desire Bro to report missing TCP segments for such a trace file.
* origin/topic/jsiwek/tcp-improvements:
Fix file_over_new_connection event to trigger when entire file is missed.
Improve TCP connection size reporting for half-open connections.
Improve gap reporting in TCP connections that never see data.
Improve TCP FIN retransmission handling.
BIT-1119
The scope of dns.log is now only standard queries (OPCODE == 0). Other
kinds of queries (e.g. inverse query) were not handled correctly and
could interfere with the state tracking of the default DNS scripts.
- Fix parsing of empty question sections (when QDCOUNT == 0). In this
case, the DNS parser would extract two 2-byte fields for use in either
"dns_query_reply" or "dns_rejected" events (dependent on value of
RCODE) as qclass and qtype parameters. This is not correct, because
such fields don't actually exist in the DNS message format when
QDCOUNT is 0. As a result, these events are no longer raised when
there's an empty question section. Scripts that depends on checking
for an empty question section can do that in the "dns_message" event.
- Add a new "dns_unknown_reply" event, for when Bro does not know how
to fully parse a particular resource record type. This helps fix a
problem in the default DNS scripts where the logic to complete
request-reply pair matching doesn't work because it's waiting on more
RR events to complete the reply. i.e. it expects ANCOUNT number of
dns_*_reply events and will wait until it gets that many before
completing a request-reply pair and logging it to dns.log. This could
cause bogus replies to match a previous request if they happen to
share a DNS transaction ID.
The event now really returns the extension. If openssl supports printing
it, it is converted into the openssl ascii output.
The output does not always look pretty because it can contain newlines.
New event syntax:
event x509_extension(c: connection, is_orig: bool, cert:X509, extension: X509_extension_info)
Example output for extension:
[name=X509v3 Extended Key Usage,
short_name=extendedKeyUsage,
oid=2.5.29.37,
critical=F,
value=TLS Web Server Authentication, TLS Web Client Authentication]
[name=X509v3 Certificate Policies,
short_name=certificatePolicies,
oid=2.5.29.32,
critical=F,
value=Policy: 1.3.6.1.4.1.6449.1.2.1.3.4^J CPS: https://secure.comodo.com/CPS^J]
The previous behavior was to accomodate SYN/FIN/RST-filtered traces by
not reporting missing data (via the content_gap event) for such
connections. The new behavior always reports gaps for connections that
are established and terminate normally, but sequence numbers indicate
that all data packets of the connection were missed. The behavior can
be reverted by redef'ing "detect_filtered_trace".
In the case multiple FIN packets are seen from a TCP endpoint (e.g.
when one is retransmitted), only the first counted towards a byte in the
sequence space. This could cause a subsequent FIN packet to induce an
incorrect wrap around in the sequence numbers (e.g. the retransmitted
FIN packet now is one sequence number behind the the first) and
misleadingly large connection sizes. The change is to always treat a
FIN packet as counting one byte in to the sequence space.
When an anonymoys function was defined inside an initialization
context, that context transfered over to the function body and could
lead to spurious error messages.
This includes enhanced GRE headers. GRE tunnels are treated just like
IP-in-IP tunnels by parsing past the GRE header in between the delivery
and payload IP packets.
lookup_hostname("localhost") occassionally timed out (after allowed 10
secs) when running test suite on some systems. Not sure why, but
changed to use the Exec module for when block conditions instead as the
scope of the test doesn't depend on a particular type of condition, it
just needs something that will work reliably/quickly.
* origin/topic/seth/ie11-software-parsing:
Updated software framework to support parsing IE11 user-agent strings.
Fix the irc_reply event for several server message types.
BIT-1104 #merged
- Internals: move type alias table to private static BroType member.
- Sphinx extension: now uses absolute path to bro binary.
- reST ouput formatting: remove "param" from function desriptions
and change package overview docs so script link+summaries render
consistently.
- First:
Due to architectural constraints, it is very hard for the
input framework to handle optional records. For an optional record,
either the whole record has to be missing, or all non-optional elements
of the record have to be defined. This information is not available
to input readers after the records have been unrolled into the threading
types.
Behavior so far was to treat optional records like they are non-optional,
without warning. The patch changes this behavior to emit an error on stream-
creation (during type-checking) and refusing to open the file. I think this
is a better idea - the behavior so far was undocumented and unintuitive.
- Second:
For table and event streams, reader backend creation was done very early,
before actually checking if all arguments are valid. Initialization is moved
after the checks now - this makes a number of delete statements unnecessary.
Also - I suspect threads of failed input reader instances were not deleted
until shutdown
- Third:
Add a couple more consistency checks, e.g. checking if the destination value
of a table has the same type as we need. We did not check everything in all
instances, instead we just assigned the things without caring (which works,
but is not really desirable).
This change also exposed a few bugs in other testcases where table definitions
were wrong (did not respect $want_record)
- Fourth:
Improve error messages and write testcases for all error messages (I think).
If the input framework was used to read event streams and
those streams contained records with more than one field, not all
elements of the threading Values were cleaned up.
The reason for this is, that the SendEventStreamEvent function
returned the number of record elements in the outmost record
instead of the number of unrolled elements in the whole vector.
This number is later used to determine how many objects to delete.
Also - add a whole bunch of leak checks for the input framework
that would have caught that and should cover quite a number of
use-cases.
The dump-events.bro generates output that would show in the Broxygen
all-script tests. Suppressing that manually in the test for lack of a
better idea.
Omission of the low index defaults to 0:
s = "12345"; s[:3] == "123"
Omission of the high index defaults to length of the string:
s = "12345"; s[3:] == "45"
