* origin/topic/bernhard/file-analysis-x509:
Forgot the preamble for the new leak test
(hopefully) last change -> return real opaque vec instead of any_vec
Fix dump-events - it cannot be used with ssl anymore, because openssl does not give the same string results in all versions.
Finishing touches of the x509 file analyzer.
Revert change to only log certificates once per hour.
Change x509 log - now certificates are only logged once per hour.
Fix circular reference problem and a few other small things.
X509 file analyzer nearly done. Verification and most other policy scripts work fine now.
Add verify functionality, including the ability to get the validated chain. This means that it is now possible to get information about the root-certificates that were used to secure a connection.
Second try on the event interface.
Backport crash fix that made it into master with the x509_extension backport from here.
Make x509 certificates an opaque type
rip out x509 code from ssl analyzer. Note that since at the moment the file analyzer does not yet re-populate the info record that means quite a lot of information is simply not available.
parse out extension. One event for general extensions (just returns the openssl-parsed string-value), one event for basicconstraints (is a certificate a CA or not) and one event for subject-alternative-names (only DNS parts).
Very basic file-analyzer for x509 certificates. Mostly ripped from the ssl-analyzer and the topic/bernhard/x509 branch.
- It's not *exactly* ISO 8601 which doesn't seem to support
subseconds, but subseconds are very important to us and
most things that support ISO8601 seem to also support subseconds
in the way I'm implemented it.
- Formatters have been abstracted similarly to readers and writers now.
- The Ascii writer has a new option for writing out logs as JSON.
- The Ascii writer now has all options availble as per-filter
options as well as global.
That means that, for example, connections that are terminated with an alert during the
handshake never appear in the ssl.log.
This patch changes this behavior - now all ssl connections that fire any event are logged.
The protocol confirmation of the ssl analyzer is moved to the client_hello instead to
the server hello. Furthermore, an additional field is added to ssl.log, which indicates
if a connection has been established or not (which probably indicates a handshake problem).
* topic/robin/http-connect:
HTTP fix for output handlers.
Expanding the HTTP methods used in the signature to detect HTTP traffic.
Updating submodule(s).
Fixing removal of support analyzers, plus some tweaking and cleanup of CONNECT code.
HTTP CONNECT proxy support.
BIT-1132 #merged
CONNECT code.
Removal of support analyzers was broken. The code now actually doesn't
delete them immediately anymore but instead just flags them as
disabled. They'll be destroyed with the parent analyzer later.
Also includes a new leak tests exercising the CONNECT code.
Lines starting # with '#' will be ignored, and an empty message aborts
the commit. # On branch topic/robin/http-connect # Changes to be
committed: # modified: scripts/base/protocols/http/main.bro #
modified: scripts/base/protocols/ssl/consts.bro # modified:
src/analyzer/Analyzer.cc # modified: src/analyzer/Analyzer.h #
modified: src/analyzer/protocol/http/HTTP.cc # new file:
testing/btest/core/leaks/http-connect.bro # modified:
testing/btest/scripts/base/protocols/http/http-connect.bro # #
Untracked files: # .tags # changes.txt # conn.log # debug.log # diff #
mpls-in-vlan.patch # newfile.pcap # packet_filter.log # reporter.log #
src/PktSrc.cc.orig # weird.log #
This supports parsing of SNMPv1 (RFC 1157), SNMPv2 (RFC 1901/3416), and
SNMPv2 (RFC 3412). An event is raised for each SNMP PDU type, though
there's not currently any event handlers for them and not a default
snmp.log either. However, simple presence of SNMP is currently visible
now in conn.log service field and known_services.log.
* origin/topic/jsiwek/dns-improvements:
Rewrite DNS state tracking which matches queries and replies.
Change dns.log to include only standard DNS queries.
Improve DNS analysis.
* origin/topic/jsiwek/tcp-improvements:
Add script to detect filtered TCP traces, addresses BIT-1119.
BIT-1119 #merged
If we could only disable an event handler dynamically ...
If reading a trace file w/ only TCP control packets, a warning is
emitted to suggest the 'detect_filtered_traces' option if the user
doesn't desire Bro to report missing TCP segments for such a trace file.
The scope of dns.log is now only standard queries (OPCODE == 0). Other
kinds of queries (e.g. inverse query) were not handled correctly and
could interfere with the state tracking of the default DNS scripts.
The event now really returns the extension. If openssl supports printing
it, it is converted into the openssl ascii output.
The output does not always look pretty because it can contain newlines.
New event syntax:
event x509_extension(c: connection, is_orig: bool, cert:X509, extension: X509_extension_info)
Example output for extension:
[name=X509v3 Extended Key Usage,
short_name=extendedKeyUsage,
oid=2.5.29.37,
critical=F,
value=TLS Web Server Authentication, TLS Web Client Authentication]
[name=X509v3 Certificate Policies,
short_name=certificatePolicies,
oid=2.5.29.32,
critical=F,
value=Policy: 1.3.6.1.4.1.6449.1.2.1.3.4^J CPS: https://secure.comodo.com/CPS^J]
* origin/topic/seth/ie11-software-parsing:
Updated software framework to support parsing IE11 user-agent strings.
Fix the irc_reply event for several server message types.
BIT-1104 #merged
- First:
Due to architectural constraints, it is very hard for the
input framework to handle optional records. For an optional record,
either the whole record has to be missing, or all non-optional elements
of the record have to be defined. This information is not available
to input readers after the records have been unrolled into the threading
types.
Behavior so far was to treat optional records like they are non-optional,
without warning. The patch changes this behavior to emit an error on stream-
creation (during type-checking) and refusing to open the file. I think this
is a better idea - the behavior so far was undocumented and unintuitive.
- Second:
For table and event streams, reader backend creation was done very early,
before actually checking if all arguments are valid. Initialization is moved
after the checks now - this makes a number of delete statements unnecessary.
Also - I suspect threads of failed input reader instances were not deleted
until shutdown
- Third:
Add a couple more consistency checks, e.g. checking if the destination value
of a table has the same type as we need. We did not check everything in all
instances, instead we just assigned the things without caring (which works,
but is not really desirable).
This change also exposed a few bugs in other testcases where table definitions
were wrong (did not respect $want_record)
- Fourth:
Improve error messages and write testcases for all error messages (I think).
Changes:
- Changing semantics of the new_event() meta event: it's raised
only for events that have a handler defined. There are too many
checks in Bro that prevent events wo/ handler from being even
prepared to raise to do that differently.
- Adding test case.
* topic/robin/event-dumper:
New script misc/dump-events.bro, along with core support, that dumps events Bro is raising in an easily readable form.
Prettyfing Describe() for record types.
Sorry for this - I noticed that I named this option quite unfortunately
while writing the documentation.
The patch also removes the dbname configuration option from the sqlite
input reader - it was not used there at all anymore (and I did not notice
that).
* origin/fastpath:
Add check for curl command to active-http.test
I've moved the check from TEST-EXEC to TEST-REQUIRES. I think it's
fine to just skip the test if curl is not available.
Added a check if the curl command is available when running the
active-http.test so that it fails more quickly and with a clear error
message if it's not available.
* origin/topic/bernhard/hyperloglog: (32 commits)
add clustered leak test for hll. No issues.
make gcc happy
(hopefully) fix refcounting problem in hll/bloom-filter opaque vals. Thanks Robin.
re-use same hash class for all add operations
get hll ready for merging
and forgot a file...
adapt to new structure
fix opaqueval-related memleak.
make it compile on case-sensitive file systems and fix warnings
make error rate configureable
add persistence test not using predetermined random seeds.
update cluster test to also use hll
persistence really works.
well, with this commit synchronizing the data structure should work.. ...if we had consistent hashing.
and also serialize the other things we need
ok, this bug was hard to find.
serialization compiles.
change plugin after feedback of seth
Forgot a file. Again. Like always. Basically.
do away with old file.
...
- Generally increased the time allowed before they timeout.
- For tests w/ a clear termination condition (most of them), made
timeouts result in a test failure.
- Seemed to be a race in some cases between tests generating output and
the input reader stream getting removed/closed, so moved stream removal
closer to termination time, when all output should be available.
- Primarily working around an issue that occurs when threads
concurrently create pipes and fork a child process. See comment in
code...
- Other minor cleanup of the code: making sure the child process calls
_exit() versus exit(), limits itself to few select system calls before
the exec(), and closes more unused file descriptors.
BIT-1054 #merged
* origin/topic/seth/unified2-analyzer:
Fixes in case a packet isn't seen that matches an event.
Finished work on unified2 analyzer.
Fixed some tests.
Working unified2 analyzer.
Unified2 file analyzer updated to new plugin style.
Adding the unified2 analyzer.
Conflicts:
testing/btest/Baseline/coverage.bare-load-baseline/canonified_loaded_scripts.log
testing/btest/Baseline/coverage.default-load-baseline/canonified_loaded_scripts.log
