- FileAnalysis::Info is now just a record used for logging, the fa_file
record type is defined in init-bare.bro as the analogue to a
connection record.
- Starting to transfer policy hook triggers and analyzer results to
events.
This reverts commit fc267d010d.
There were some diffs caused by this in external test suites I'm
unsure about, I'm going to go over optimizations more closely in
a different branch.
When a file handle is needed and the last event in the queue is also
a get_file_handle event with the same arguments, instead of queueing
a new event, just remember to cache/re-use the resulting handle from
the previous event. This depends on get_file_handle handlers not
changing global state that is also used to derive the file handle
string.
- Add a timeout flag to file_analysis.log so it's easy to tell what
has had at least one timeout trigger happen.
- Fix ftp-data service tag not being set for reused connections.
- Fix HTTP::Incorrect_File_Type because mime types returned by FAF have
the charset still in them, but the HTTP::mime_types_extensions table
does not and it requires an exact string match. (still ugly)
- Add TRIGGER_NEW_CONN to track files going over multiple connections.
- Add an initial file/mime type guess for non-linear file transfers.
- Fix a case where file/mime type detection would never be attempted
if the start of the file was a content gap.
- Improve mime type tracking of HTTP byte-range/partial-content,
even if the requests are pipelined or over multiple connections.
- I changed the modbus.events test because having the baseline output
be 80+ MB is nuts and it was sensitive to connection record redefs.
Versus from synchronous function calls, which doesn't work well because
the function call can see a script-layer state that doesn't reflect
the state as it will be in terms of the event/network stream.
Other misc:
- Remove HTTP::MD5 notice.
- Add "last_active" field to FileAnalysis::Info record.
- Replace "conn_uids", "conn_ids" fields in FileAnalysis::Info record
with just a "conns" fields containing full connection records.
- The http-methods unit test is failing now, but I think it will be
fixed once I change the file handle callback mechanism to use events
instead.
A retry happens on every new input and also periodically based on a
timer. If a file handle is returned at those times, the input is
forwarded for analysis, else it keeps retrying until a timeout
threshold.
The framework now cycles through callbacks based on a table indexed
by analyzer tags, or the special case of service strings if a given
analyzer is overloaded for multiple protocols (FTP/IRC data). This
lets each protocol script bundle implement the callback locally and
reduces the FAF's external dependencies.
These cases should be avoidable by fixing scripts where they occur and
they can also help catch typos that would lead to unintentional runtime
behavior.
Adding this already revealed several scripts where a field in an inlined
record was never removed after a code refactor.
* origin/topic/matthias/opaque:
Add new unit test for opaque serialization.
Migrate entropy testing to opaque.
C++ify RandTest.*
Fix a hard-to-spot bug.
Use more descriptive error message.
Fix the fix :-/.
Fix initialization of hash values.
Be clearer about delegation.
Implement serialization of opaque types.
Update hash BiF documentation.
Migrate free SHA* functions to SHA*Val::digest().
Add missing type name that caused failing tests.
Update base scripts and unit tests.
Simplify hash function BiFs.
Add support for opaque hash values.
Adapt BiF & Bro parser to handle opaque types.
More lexer/parser work.
Implement equivalence relation for opaque types.
Support basic serialization of opaque.
Add opaque type to lexer, parser, and BroType.
Closes#925
Conflicts:
aux/broccoli
consisting of letters [A-Za-z].
I had some bogus HTTP sessions now with the test-suite that reported
data as HTTP because it started with "<!... ". Requiring letters seems
a reasonable constraint.
* vlad/topic/vladg/http-verbs:
A test for HTTP methods, including some horribly illegal requests.
Remove hardcoded HTTP verbs from the analyzer (#741)
I added a "bad_HTTP_request" weird for HTTP request lines that don't
have more than a single word.
Closes#741.
Instead, the `addr_to_uri` script-level function can be used to
explicitly add brackets to an address if it's IPv6 and will be
included in a URI or when a ":<port>" needs to be appended to it.
* origin/topic/script-reference: (50 commits)
A few updates for the FAQ.
Fixing some doc warnings.
Forgot to add protocol identifier support for TLS 1.2
Finished SSL & syslog autodocs.
Adding the draft SSL extension type next_protocol_negotiation.
Fix some documentation errors.
Tweaks.
A set of script-reference polishing.
fixed a couple typos in comments
Add summary documentation to bif files.
Add ssl and syslog script documentation
Add Conn and DNS protocol script documentation. (fixes#731)
Small updates to the default local.bro.
Documentation updates for HTTP & IRC scripts.
SSH&FTP Documentation updates.
Fixing a warning from the documentation generation.
This completes framework documentation package 4.
Minor notice documentation tweaks.
Fix some malformed Broxygen xref roles.
Minor doc tweaks to init-bare.bro.
...
Conflicts:
aux/broccoli
aux/broctl
src/bro.bif
src/strings.bif
Includes:
- Updated baselines for autodoc tests.
- Now excluding stats.bro from external texts, it's not stable.
* origin/fastpath:
Binary packaging script tweaks.
More default "weird" tuning for the "SYN_with_data" notice.
Tiny bugfix for http file extraction along with test.
* origin/topic/jsiwek/http-1xx-replies:
Change logging of HTTP 1xx responses to occur in their own columns.
Fix handling of HTTP 1xx response codes (addresses #411).
- A null value no longer fits since if there is no body
a value of zero makes sense. Previously, a null value would
makes sense because the Content-Length header may not have
been sent which would leave the field null.
- scan.bro and hot.conn.bro will be returning soon.
- The rest are going to return as updated protocol analysis
scripts and new/updated frameworks later.
Changed the parser to not treat 1xx response codes as a final answer
to an unanswered request -- a later response is still expected.
The scripting layer will also not finish a request-reply pair when
seeing 1xx's, instead it logs both the 1xx and final response messages
with associated information of the current request as they're seen.
- The value of the content-length headers has now been removed
but it could be added back locally at an installation by a user.
- Added fields to indicate if some parsing interruption happened
during the body transfer.
- Closes#581
- Log path's are generated in the scripting land
now. The default Log stream ID to path string
mapping works like this:
- Notice::LOG -> "notice"
- Notice::POLICY_LOG -> "notice_policy"
- TestModule::LOG -> "test_module"
- Logging streams updated across all of the shipped
scripts to be more user friendly. Instead of
the logging stream ID HTTP::HTTP, we now have
HTTP::LOG, etc.
- The priorities on some bro_init handlers have
been adjusted to make the process of applying
filters or disabling streams easier for users.
