* origin/topic/jsiwek/tcp-improvements:
Fix file_over_new_connection event to trigger when entire file is missed.
Improve TCP connection size reporting for half-open connections.
Improve gap reporting in TCP connections that never see data.
Improve TCP FIN retransmission handling.
BIT-1119
The event now really returns the extension. If openssl supports printing
it, it is converted into the openssl ascii output.
The output does not always look pretty because it can contain newlines.
New event syntax:
event x509_extension(c: connection, is_orig: bool, cert:X509, extension: X509_extension_info)
Example output for extension:
[name=X509v3 Extended Key Usage,
short_name=extendedKeyUsage,
oid=2.5.29.37,
critical=F,
value=TLS Web Server Authentication, TLS Web Client Authentication]
[name=X509v3 Certificate Policies,
short_name=certificatePolicies,
oid=2.5.29.32,
critical=F,
value=Policy: 1.3.6.1.4.1.6449.1.2.1.3.4^J CPS: https://secure.comodo.com/CPS^J]
The previous behavior was to accomodate SYN/FIN/RST-filtered traces by
not reporting missing data (via the content_gap event) for such
connections. The new behavior always reports gaps for connections that
are established and terminate normally, but sequence numbers indicate
that all data packets of the connection were missed. The behavior can
be reverted by redef'ing "detect_filtered_trace".
In the case multiple FIN packets are seen from a TCP endpoint (e.g.
when one is retransmitted), only the first counted towards a byte in the
sequence space. This could cause a subsequent FIN packet to induce an
incorrect wrap around in the sequence numbers (e.g. the retransmitted
FIN packet now is one sequence number behind the the first) and
misleadingly large connection sizes. The change is to always treat a
FIN packet as counting one byte in to the sequence space.
When an anonymoys function was defined inside an initialization
context, that context transfered over to the function body and could
lead to spurious error messages.
This includes enhanced GRE headers. GRE tunnels are treated just like
IP-in-IP tunnels by parsing past the GRE header in between the delivery
and payload IP packets.
lookup_hostname("localhost") occassionally timed out (after allowed 10
secs) when running test suite on some systems. Not sure why, but
changed to use the Exec module for when block conditions instead as the
scope of the test doesn't depend on a particular type of condition, it
just needs something that will work reliably/quickly.
* origin/topic/seth/ie11-software-parsing:
Updated software framework to support parsing IE11 user-agent strings.
Fix the irc_reply event for several server message types.
BIT-1104 #merged
- Internals: move type alias table to private static BroType member.
- Sphinx extension: now uses absolute path to bro binary.
- reST ouput formatting: remove "param" from function desriptions
and change package overview docs so script link+summaries render
consistently.
- First:
Due to architectural constraints, it is very hard for the
input framework to handle optional records. For an optional record,
either the whole record has to be missing, or all non-optional elements
of the record have to be defined. This information is not available
to input readers after the records have been unrolled into the threading
types.
Behavior so far was to treat optional records like they are non-optional,
without warning. The patch changes this behavior to emit an error on stream-
creation (during type-checking) and refusing to open the file. I think this
is a better idea - the behavior so far was undocumented and unintuitive.
- Second:
For table and event streams, reader backend creation was done very early,
before actually checking if all arguments are valid. Initialization is moved
after the checks now - this makes a number of delete statements unnecessary.
Also - I suspect threads of failed input reader instances were not deleted
until shutdown
- Third:
Add a couple more consistency checks, e.g. checking if the destination value
of a table has the same type as we need. We did not check everything in all
instances, instead we just assigned the things without caring (which works,
but is not really desirable).
This change also exposed a few bugs in other testcases where table definitions
were wrong (did not respect $want_record)
- Fourth:
Improve error messages and write testcases for all error messages (I think).
If the input framework was used to read event streams and
those streams contained records with more than one field, not all
elements of the threading Values were cleaned up.
The reason for this is, that the SendEventStreamEvent function
returned the number of record elements in the outmost record
instead of the number of unrolled elements in the whole vector.
This number is later used to determine how many objects to delete.
Also - add a whole bunch of leak checks for the input framework
that would have caught that and should cover quite a number of
use-cases.
The dump-events.bro generates output that would show in the Broxygen
all-script tests. Suppressing that manually in the test for lack of a
better idea.
Omission of the low index defaults to 0:
s = "12345"; s[:3] == "123"
Omission of the high index defaults to length of the string:
s = "12345"; s[3:] == "45"
Changes:
- Changing semantics of the new_event() meta event: it's raised
only for events that have a handler defined. There are too many
checks in Bro that prevent events wo/ handler from being even
prepared to raise to do that differently.
- Adding test case.
* topic/robin/event-dumper:
New script misc/dump-events.bro, along with core support, that dumps events Bro is raising in an easily readable form.
Prettyfing Describe() for record types.
BIT-1098
* origin/topic/jsiwek/broxygen:
Fix Broxygen-related compile errors.
Add a Broxygen coverage test.
Internal Broxygen organization/documentation/polish.
Add unit tests for Broxygen config file targets.
Change Broxygen config file format.
Broxygen doc-related test updates. Fix two regressions.
A couple documentation fixes.
Integrate new Broxygen functionality into Sphinx.
Implement majority of Broxygen features delegated to Bro.
Broxygen can now read a config file specifying particular targets.
Remove unneeded Broxygen comments in scan.bro.
Replace safe_basename/safe_dirname w/ SafeBasename/SafeDirname.
Add BIF interface for retrieving comments/docs.
Quick optimization to Broxygen doc gathering.
Flesh out Broxygen doc-gathering skeleton.
Refactor search_for_file() util function.
Initial skeleton of new Broxygen infrastructure.
May fix a sporadic failure, in which case it was just not getting enough
time to run or lookup_hostname() taking longer than ~3 seconds. Else,
the new output should give more hints on what's going wrong. In any
case, termination conditions for the test are now explicit.
Slice ranges were not correctly determined for negative indices and also
off by one in general (included one more element at the end of the
substring than what actually matched the index range).
It's now equivalent to Python slice notation. Accessing a string at
a single index is also the same as Python except that an out-of-range
index returns an empty string instead of throwing an expection.
Broccoli clients can only recv events w/ vectors for now.
Also changed ordering of Bro type tag enum -- the addition of opaque
types changed the value of the vector type, making broccoli.h's
definition out of sync. Probably could have just changed broccoli's
definition, but seems more correct to go back to using the same value
for vectors as they were before opaques. It's also better in case
there's some other location I'm not aware of where the values are
replicated.
Add a "broxygen" domain Sphinx extension w/ directives to allow
on-the-fly documentation to be generated w/ Bro and included in files.
This means all autogenerated reST docs are now done by Bro. The odd
CMake/Python glue scipts which used to generate some portions are now
gone. Bro and the Sphinx extension handle checking for outdated docs
themselves.
Parallel builds of `make doc` target should now work (mostly because
I don't think there's any tasks that can be done in parallel anymore).
Overall, this seems to simplify things and make the Broxygen-generated
portions of the documentation visible/traceable from the main Sphinx
source tree. The one odd thing still is that per-script documentation
is rsync'd in to a shadow copy of the Sphinx source tree within the
build dir. This is less elegant than using the new broxygen extension
to make per-script docs, but rsync is faster and simpler. Simpler as in
less code because it seems like, in the best case, I'd need to write a
custom Sphinx Builder to be able to get that to even work.
