Commit graph

3113 commits

Author SHA1 Message Date
Jon Siwek
c1b3c9593b Add "udp_content_ports" option
Any port in that set found as either source or destination port
of a UDP packet will cause the "udp_contents" event to be raised.
2020-04-07 13:02:29 -07:00
Tim Wojtulewicz
08fbdb1418 Merge remote-tracking branch 'origin/topic/jsiwek/fix-patterns-as-table-index'
* origin/topic/jsiwek/fix-patterns-as-table-index:
  Fix using patterns stored as table/set indices
2020-04-07 12:05:44 -07:00
Jon Siwek
da56cd44d9 Fix using patterns stored as table/set indices
Pattern values used as table/set indices are stored in serialized form
using just the pattern text, but re-creating the value from that didn't
fully initialize/compile the regex matcher after (re-)creating it from
that pattern text.
2020-04-06 17:12:33 -07:00
Jon Siwek
2da84020cf Add new "udp_content_delivery_ports_use_resp" option
This controls whether ports given by "udp_content_delivery_ports_orig" and
"udp_content_delivery_ports_orig" are in terms of the UDP packet's
destination port or by the Connection's "responder" port (the former is
the unchanged default behavior).
2020-04-06 14:51:34 -07:00
Jon Siwek
bc5844b24c Fix connection duration thresholding
Connection durations were being compared "less than" instead of "greater
than" the desired duration threshold as provided to
set_current_conn_bytes_threshold() or
ConnThreshold::set_duration_threshold()
2020-04-06 13:32:32 -07:00
Anthony Kasza
19bb5210f4 change: -C flag to btest 2020-04-06 10:17:37 -06:00
Anthony Kasza
980e6299d3 add: ignore checksum added to btest script 2020-04-06 10:17:06 -06:00
Anthony Kasza
d10c9ab667 change: update btest baselines for rdpeudp-handshake-success 2020-04-03 16:11:18 -06:00
Anthony Kasza
243095fd9b add: rdpeudp2-handshake-success btest baseline 2020-04-03 16:09:53 -06:00
Anthony Kasza
d38599ecb4 add: btest for rdpeudp2 2020-04-03 11:48:26 -06:00
Anthony Kasza
9b1b8ae292 add: test trace for when RDPEUDP negotiates RDPEUDP2 2020-04-03 09:59:39 -06:00
Jon Siwek
e66148a13a Merge branch 'topic/ak/rdpeudp'
- Squashed the original commit set
- Cleaned up formatting
- Fixed register_for_ports() for right RDPEUDP analyzer

* topic/ak/rdpeudp:
  Add RDP over UDP analyzer
2020-04-02 18:31:40 -07:00
Anthony Kasza
60644bc85f Add RDP over UDP analyzer 2020-04-02 17:53:47 -07:00
Jon Siwek
8c0e8ecd28 Allow alternate event/hook prototype declarations
The alternates must be some subset of the canonical prototype (the one
that's first declared) and allows users to define handlers for any
such prototype.  Example:

    # Prototype declarations
    global my_event: event(s: string, c: count);
    global my_event: event(c: count);
    global my_event: event();

    # Handler definitions
    event my_event(s: string, c: count) { print s, c; }
    event my_event(c: count) { print c; }
    event my_event() { }

This allows handlers to consume a subset of the arguments or even
re-order them.  This makes it easier to either extend an existing
event/hook's arguments and/or deprecate usages of certain prototypes.
2020-04-01 17:25:28 -07:00
Johanna Amann
3ce1c9ffd6 Merge remote-tracking branch 'origin/topic/jsiwek/misc-lambda-fixes'
* origin/topic/jsiwek/misc-lambda-fixes:
  GH-725: fix logic for finding a lambda's usage of outer IDs
  Change record field anonymous functions to use lambda expressions
2020-03-31 15:46:54 -07:00
Johanna Amann
3c470ffe13 Merge remote-tracking branch 'origin/topic/jsiwek/gh-887-fix-gre-erspan-dot1q'
* origin/topic/jsiwek/gh-887-fix-gre-erspan-dot1q:
  GH-887: improve GRE/ERSPAN parsing of non-IPv4/IPv6 inner payload
  Fix ERSPAN III optional sub-header parsing
2020-03-30 14:30:00 -07:00
Johanna Amann
2e6eb7795c Merge remote-tracking branch 'origin/topic/jsiwek/tcp-hdr-reserved-bits'
* origin/topic/jsiwek/tcp-hdr-reserved-bits:
  Add a "reserved" field to tcp_hdr record
2020-03-30 13:23:51 -07:00
Jon Siwek
b7dee712d5 GH-887: improve GRE/ERSPAN parsing of non-IPv4/IPv6 inner payload
This changes the decapsulation logic for GRE/ERSPAN payloads to re-use
existing Layer 2 parsing logic that already handles things like 802.1Q
tags correctly before going on to process the inner IPv4/IPv6 payload.
2020-03-27 15:22:00 -07:00
Tim Wojtulewicz
85c6c2b9ee Merge remote-tracking branch 'origin/topic/jsiwek/runtime-exception-leak-cleanup'
* origin/topic/jsiwek/runtime-exception-leak-cleanup:
  Func::DescribeDebug: move a NumFields() call out of loop
  Use const-ref parameter for zeek::val_list_to_args()
  Fix missing IntrusivePtr.h include and ambiguous ODesc::Add call
  Remove TimerMgr arg from event queuing/scheduling methods
  Deprecate Analyzer::ConnectionEvent()
  Deprecate file_analysis::File::FileEvent methods using val_list args
  Deprecate Connection::ConnectionEvent methods
  Deprecate EventMgr::QueueEventFast() and update usages to Enqueue()
  Deprecate EventMgr::QueueEvent() and update usages to Enqueue()
  Deprecate Func::Call(val_list*, ...)
  Use vector<IntrusivePtr<Val>> for Func::Call and Event queuing args
  Fix memory leak in Zeek when-statement bodies with runtime errors
  Change TableVal::RecoverIndex() to return IntrusivePtr
  Use IntrusivePtr in TableVal::CallExpireFunc
  Fix memory leak when runtime error occurs in a Zeek for-loop
  Enable leak checks for btests that produce runtime exceptions
2020-03-27 11:49:45 -07:00
Jon Siwek
1ca11f11c7 GH-725: fix logic for finding a lambda's usage of outer IDs 2020-03-26 17:05:59 -07:00
Jon Siwek
f032885085 Change record field anonymous functions to use lambda expressions
There was an alternate syntax to assign anonymous functions to record
fields that was never migrated to use the new lambda expression
machinery (and so didn't allow referencing variables in outer scope):

    type myrec: record {
        foo: function(a: string);
    };

    local o = "o";
    local mr = myrec($foo(a: string) = { print a + o; });
2020-03-26 15:48:18 -07:00
Jon Siwek
13391806ba Add a "reserved" field to tcp_hdr record
This record is available by handling either raw_packet or new_packet
events.
2020-03-26 11:48:53 -07:00
Johanna Amann
8a4ccabc3e Merge remote-tracking branch 'origin/topic/jsiwek/fix-greedy-flowbuffer-bounds-check'
* origin/topic/jsiwek/fix-greedy-flowbuffer-bounds-check:
  Add test case for binpac flowbuffer frame length parsing bug
2020-03-24 13:49:58 -07:00
Jon Siwek
98d94ec785 Enable leak checks for btests that produce runtime exceptions
These were previously reporting leaks due to various allocations not
getting cleaned up during the stack unwind, but at the current state of
the transition toward IntrusivePtr usage, theses tests no longer leak.
2020-03-23 21:51:12 -07:00
Jon Siwek
427150b27a Merge remote-tracking branch 'origin/topic/jsiwek/gh-857-rebuild-tables-on-record-redef'
* origin/topic/jsiwek/gh-857-rebuild-tables-on-record-redef:
  Initialize RecordVal default fields when redef'd
  GH-857: fix redefining record types used to index tables
  Change RecordVals to get resized at time of RecordType redef
2020-03-20 11:31:11 -07:00
Tim Wojtulewicz
535c70b087 Merge remote-tracking branch 'origin/topic/jsiwek/gh-865-smb-negotiate-context-list'
* origin/topic/jsiwek/gh-865-smb-negotiate-context-list:
  GH-865: fix parsing of SMB NegotiateContextList
2020-03-20 10:05:34 -07:00
Jon Siwek
7e57f0788c Add test case for binpac flowbuffer frame length parsing bug 2020-03-19 22:09:23 -07:00
Jon Siwek
b62727a7fa Merge branch 'intrusive_ptr' of https://github.com/MaxKellermann/zeek
* 'intrusive_ptr' of https://github.com/MaxKellermann/zeek: (32 commits)
  Scope: store IntrusivePtr in `local`
  Scope: pass IntrusivePtr to AddInit()
  DNS_Mgr: use class IntrusivePtr
  Scope: use class IntrusivePtr
  Attr: use class IntrusivePtr
  Expr: check_and_promote_expr() returns IntrusivePtr
  Frame: use class IntrusivePtr
  Val: RecordVal::LookupWithDefault() returns IntrusivePtr
  Type: RecordType::FieldDefault() returns IntrusivePtr
  Val: TableVal::Delete() returns IntrusivePtr
  Type: base_type() returns IntrusivePtr
  Type: init_type() returns IntrusivePtr
  Type: merge_types() returns IntrusivePtr
  Type: use class IntrusivePtr in VectorType
  Type: use class IntrusivePtr in EnumType
  Type: use class IntrusivePtr in FileType
  Type: use class IntrusivePtr in TypeDecl
  Type: make TypeDecl `final` and the dtor non-`virtual`
  Type: use class IntrusivePtr in TypeType
  Type: use class IntrusivePtr in FuncType
  ...
2020-03-17 22:51:46 -07:00
Jon Siwek
9c70bcecbc GH-865: fix parsing of SMB NegotiateContextList
* The compression capability was incorrectly set to 0x0004 instead of 0x0003

* The padding was 4-byte instead of 8-byte aligned and also the spec.
  does not strictly require the padding for the last item in the list.

* Add a default case to handle parsing of unknown context types.
2020-03-16 19:00:01 -07:00
Jon Siwek
7a397d2746 Initialize RecordVal default fields when redef'd
If a RecordVal had been created, but later its RecordType redef'd to
contain fields with &default, those fields were incorrectly left
uninitialized.
2020-03-12 18:46:37 -07:00
Jon Siwek
71b82595ba GH-857: fix redefining record types used to index tables
This change tracks all TableVals created at parse-time whose index
depends on a given RecordType.  Should that RecordType be redef'd, those
TableVals are immediately rebuilt such that they are valid to
subsequently use in either parse-time initializations or eventually in
any arbitrary run-time expression.
2020-03-12 18:25:48 -07:00
Johanna Amann
da5fca7163 Merge branch 'topic/johanna/gh-744-cache-the-heck-out-of-these-certs'
* topic/johanna/gh-744-cache-the-heck-out-of-these-certs:
  X509 caching API change: callback gets entry directly
  Address feedback
  Small default updates & external baselines.
  X509 caching: small API changes, tests & test updates
  Certificate caching - now working in principle.
  First (sadly not completely working) approach at caching X509 certificates
2020-03-12 13:44:07 -07:00
Johanna Amann
e228061f1d X509 caching API change: callback gets entry directly
Suggestion from Jon: now the callback is passed the table-entry
directly. This allows us to skip one lookup.
2020-03-12 11:31:13 -07:00
Johanna Amann
4b09947f41 X509 caching: small API changes, tests & test updates
Changed some configuration defaults to potentially more same values.

The callback function is now a hook to allow costomization of the events
that are raised.

Tests now exist. Test baselines are updated.
2020-03-11 13:27:56 -07:00
Jon Siwek
26af1f55af Merge remote-tracking branch 'origin/topic/vladg/gh-843'
Added a comment to clarify comparison of EOF packet length to 13.

* origin/topic/vladg/gh-843:
  Update baselines
  Made additional MySQL fixes.
  Add support to MySQL for deprecation of EOF packets.
  Whitespace cleanup & fixes
  Fix EOF detection in the MySQL protocol analyzer.
2020-03-11 11:25:00 -07:00
Vlad Grigorescu
a362189856 Update baselines 2020-03-06 22:49:26 -06:00
Jon Siwek
363fc49518 Fix a btest to order zeek_init() events via &priority 2020-03-05 14:19:34 -08:00
Tim Wojtulewicz
822567b3f9 Merge remote-tracking branch 'origin/topic/jsiwek/32bit-compat'
* origin/topic/jsiwek/32bit-compat:
  Improve formatting of doubles that are close to integers
  Improve HTTP version number comparisons
  Add a 32-bit task to Cirrus CI config
  Replace va_list fmt() overload with vfmt()
  Format tables indexed by patterns consistently across 32-bit/64-bit
  Format interval values consistently across 32-bit/64-bit platforms
2020-02-24 18:49:33 -07:00
Jon Siwek
a785212e80 Improve formatting of doubles that are close to integers
Now checks for approximate floating point equality so that more doubles
get properly disambiguated from integers
2020-02-15 00:54:44 -08:00
Jon Siwek
11d8b8c040 Fix code format of various reporter btests 2020-02-14 22:03:11 -08:00
Jon Siwek
996b3d523a Update to start of 3.2.0 development 2020-02-08 16:08:01 -08:00
Jon Siwek
e50a8848ae Improve an input framework test
A race condition could cause unstable output: if the thread reading the
file is fast, often you see both "pred" functions execute and then both
"line" events execute with both entries already in the table, but if the
thread reading the file is slow, you see pred, event, pred, event, with
only one entry available in the first event.
2020-02-07 20:03:10 -08:00
Jon Siwek
a3b1d202a5 Improve stability of a &expire_func btest
The order in which &expire_func's get called isn't well-defined, so
separate the output from either to ensure diffs against the Baseline are
always consistent.
2020-02-07 15:11:06 -08:00
Johanna Amann
56dda98512 Merge remote-tracking branch 'origin/topic/jsiwek/stabilize-ci'
* origin/topic/jsiwek/stabilize-ci:
  Add environment variable to disable supervisor SIGKILL
  Fix a btest's race condition by handling Pcap::file_done
  Add Pcap::file_done event
  Improve btest timeouts
2020-02-07 09:52:57 -08:00
Johanna Amann
49a21d73d0 Merge remote-tracking branch 'origin/topic/jsiwek/gh-780-indefinite-log-buffer'
* origin/topic/jsiwek/gh-780-indefinite-log-buffer:
  GH-780: Prevent log batches from indefinite buffering
2020-02-07 09:35:29 -08:00
Jon Siwek
a2b0abe283 Add environment variable to disable supervisor SIGKILL
For use in btests since SIGKILL can otherwise bypass the usual
btest-bg-wait timeout mechanism and make reason for test failures less
obvious.
2020-02-06 17:50:17 -08:00
Jon Siwek
5b74bbf99a Fix a btest's race condition by handling Pcap::file_done 2020-02-06 17:50:17 -08:00
Jon Siwek
7c124881cd Add Pcap::file_done event
It signals when a pcap file is done being processed.
2020-02-06 17:50:17 -08:00
Jon Siwek
4fbcca04e8 Improve btest timeouts
* Generally increase timeouts for tests that have recent transient
  failures

* Change any test that relied on `btest-bg-wait -k` since that's never
  going to play with with CI systems.  Instead, we always need to have
  a well-defined termination condition in the test itself (and most
  already did, so didn't really need the `-k` flag anyway).
2020-02-06 17:50:17 -08:00
Johanna Amann
391164c9bc Merge remote-tracking branch 'origin/topic/johanna/table-on-change'
* origin/topic/johanna/table-on-change:
  &on_change: Address feedback of Jon.
  &on_change: incooperate feedback of Jon.
  &on_change wrapup: documentation, tests, whitespacing
  Table expiry now raises &on_change handlers
  Prevent recursion of &on_change handlers.
  Make bro_broker::val_to_data take a const Val* instead of a Val
  &on_change: add insertion/change notifications.
  &on_change working for removals from tables.
  Change signature of &on_change handler
  Beginning implementation of &on_change for tables.
2020-02-06 15:01:03 -08:00