Due to the change in f1cef9d2a9, it was possible for the TCP reassembler
to deliver the same data twice because Undelivered did not take in to
account that the reassembly stream could now advance past the end of the
gap.
Addresses BIT-1259.
* origin/topic/jsiwek/jj-bugs:
Fix incorrect data delivery skips after gap in HTTP Content-Range.
Fix file analysis placement of data after gap in HTTP Content-Range.
Fix issue w/ TCP reassembler not delivering some segments.
Raise http_entity_data in line with data arrival.
Implement file ID caching for MIME_Mail.
BIT-1240: Fix MIME entity file data/gap ordering.
BIT-1240 #closed
BIT-1246 #closed
BIT-1247 #closed
BIT-1248 #closed
For example, if we have a connection between TCP "A" and TCP "B" and "A"
sends segments "1" and "2", but we don't see the first and then the next
acknowledgement from "B" is for everything up to, and including, "2",
the gap would be reported to include both segments instead of just the
first and then delivering the second. Put generally: any segments that
weren't yet delivered because they're waiting for an earlier gap to be
filled would be dropped when an ACK comes in that includes the gap as
well as those pending segments. (If a distinct ACK was seen for just
the gap, that situation would have worked).
Addresses BIT-1246.
The main change is that reassembly code (e.g. for TCP) now uses
int64/uint64 (signedness is situational) data types in place of int
types in order to support delivering data to analyzers that pass 2GB
thresholds. There's also changes in logic that accompany the change in
data types, e.g. to fix TCP sequence space arithmetic inconsistencies.
Another significant change is in the Analyzer API: the *Packet and
*Undelivered methods now use a uint64 in place of an int for the
relative sequence space offset parameter.
- Since it's just the handshake packets out of order, they're no
longer treated as partial connections, which some protocol analyzers
immediately refuse to look at.
- The TCP_Reassembler "is_orig" state failed to change, which led to
protocol analyzers sometimes using the wrong value for that.
- Add a unit test which exercises the Connection::FlipRoles() code
path (i.e. the SYN/SYN-ACK reversal situation).
Addresses BIT-1148.
The previous behavior was to accomodate SYN/FIN/RST-filtered traces by
not reporting missing data (via the content_gap event) for such
connections. The new behavior always reports gaps for connections that
are established and terminate normally, but sequence numbers indicate
that all data packets of the connection were missed. The behavior can
be reverted by redef'ing "detect_filtered_trace".
Replaced some with InternalWarning or InternalAnalyzerError, the later
being a new method which signals the analyzer to not process further
input. Some usages I just removed if they didn't make sense or clearly
couldn't happen. Also did some minor refactors of related code while
reviewing/exploring ways to get rid of InternalError usages.
Also, for TCP content file write failures there's a new event:
"contents_file_write_failure".
