Commit graph

3047 commits

Author SHA1 Message Date
Jon Siwek
c1b3c9593b Add "udp_content_ports" option
Any port in that set found as either source or destination port
of a UDP packet will cause the "udp_contents" event to be raised.
2020-04-07 13:02:29 -07:00
Jon Siwek
2da84020cf Add new "udp_content_delivery_ports_use_resp" option
This controls whether ports given by "udp_content_delivery_ports_orig" and
"udp_content_delivery_ports_orig" are in terms of the UDP packet's
destination port or by the Connection's "responder" port (the former is
the unchanged default behavior).
2020-04-06 14:51:34 -07:00
Jon Siwek
e66148a13a Merge branch 'topic/ak/rdpeudp'
- Squashed the original commit set
- Cleaned up formatting
- Fixed register_for_ports() for right RDPEUDP analyzer

* topic/ak/rdpeudp:
  Add RDP over UDP analyzer
2020-04-02 18:31:40 -07:00
Anthony Kasza
60644bc85f Add RDP over UDP analyzer 2020-04-02 17:53:47 -07:00
Johanna Amann
3ce1c9ffd6 Merge remote-tracking branch 'origin/topic/jsiwek/misc-lambda-fixes'
* origin/topic/jsiwek/misc-lambda-fixes:
  GH-725: fix logic for finding a lambda's usage of outer IDs
  Change record field anonymous functions to use lambda expressions
2020-03-31 15:46:54 -07:00
Johanna Amann
3c470ffe13 Merge remote-tracking branch 'origin/topic/jsiwek/gh-887-fix-gre-erspan-dot1q'
* origin/topic/jsiwek/gh-887-fix-gre-erspan-dot1q:
  GH-887: improve GRE/ERSPAN parsing of non-IPv4/IPv6 inner payload
  Fix ERSPAN III optional sub-header parsing
2020-03-30 14:30:00 -07:00
Johanna Amann
2e6eb7795c Merge remote-tracking branch 'origin/topic/jsiwek/tcp-hdr-reserved-bits'
* origin/topic/jsiwek/tcp-hdr-reserved-bits:
  Add a "reserved" field to tcp_hdr record
2020-03-30 13:23:51 -07:00
Jon Siwek
b7dee712d5 GH-887: improve GRE/ERSPAN parsing of non-IPv4/IPv6 inner payload
This changes the decapsulation logic for GRE/ERSPAN payloads to re-use
existing Layer 2 parsing logic that already handles things like 802.1Q
tags correctly before going on to process the inner IPv4/IPv6 payload.
2020-03-27 15:22:00 -07:00
Tim Wojtulewicz
85c6c2b9ee Merge remote-tracking branch 'origin/topic/jsiwek/runtime-exception-leak-cleanup'
* origin/topic/jsiwek/runtime-exception-leak-cleanup:
  Func::DescribeDebug: move a NumFields() call out of loop
  Use const-ref parameter for zeek::val_list_to_args()
  Fix missing IntrusivePtr.h include and ambiguous ODesc::Add call
  Remove TimerMgr arg from event queuing/scheduling methods
  Deprecate Analyzer::ConnectionEvent()
  Deprecate file_analysis::File::FileEvent methods using val_list args
  Deprecate Connection::ConnectionEvent methods
  Deprecate EventMgr::QueueEventFast() and update usages to Enqueue()
  Deprecate EventMgr::QueueEvent() and update usages to Enqueue()
  Deprecate Func::Call(val_list*, ...)
  Use vector<IntrusivePtr<Val>> for Func::Call and Event queuing args
  Fix memory leak in Zeek when-statement bodies with runtime errors
  Change TableVal::RecoverIndex() to return IntrusivePtr
  Use IntrusivePtr in TableVal::CallExpireFunc
  Fix memory leak when runtime error occurs in a Zeek for-loop
  Enable leak checks for btests that produce runtime exceptions
2020-03-27 11:49:45 -07:00
Jon Siwek
1ca11f11c7 GH-725: fix logic for finding a lambda's usage of outer IDs 2020-03-26 17:05:59 -07:00
Jon Siwek
f032885085 Change record field anonymous functions to use lambda expressions
There was an alternate syntax to assign anonymous functions to record
fields that was never migrated to use the new lambda expression
machinery (and so didn't allow referencing variables in outer scope):

    type myrec: record {
        foo: function(a: string);
    };

    local o = "o";
    local mr = myrec($foo(a: string) = { print a + o; });
2020-03-26 15:48:18 -07:00
Jon Siwek
13391806ba Add a "reserved" field to tcp_hdr record
This record is available by handling either raw_packet or new_packet
events.
2020-03-26 11:48:53 -07:00
Johanna Amann
8a4ccabc3e Merge remote-tracking branch 'origin/topic/jsiwek/fix-greedy-flowbuffer-bounds-check'
* origin/topic/jsiwek/fix-greedy-flowbuffer-bounds-check:
  Add test case for binpac flowbuffer frame length parsing bug
2020-03-24 13:49:58 -07:00
Jon Siwek
98d94ec785 Enable leak checks for btests that produce runtime exceptions
These were previously reporting leaks due to various allocations not
getting cleaned up during the stack unwind, but at the current state of
the transition toward IntrusivePtr usage, theses tests no longer leak.
2020-03-23 21:51:12 -07:00
Jon Siwek
427150b27a Merge remote-tracking branch 'origin/topic/jsiwek/gh-857-rebuild-tables-on-record-redef'
* origin/topic/jsiwek/gh-857-rebuild-tables-on-record-redef:
  Initialize RecordVal default fields when redef'd
  GH-857: fix redefining record types used to index tables
  Change RecordVals to get resized at time of RecordType redef
2020-03-20 11:31:11 -07:00
Tim Wojtulewicz
535c70b087 Merge remote-tracking branch 'origin/topic/jsiwek/gh-865-smb-negotiate-context-list'
* origin/topic/jsiwek/gh-865-smb-negotiate-context-list:
  GH-865: fix parsing of SMB NegotiateContextList
2020-03-20 10:05:34 -07:00
Jon Siwek
7e57f0788c Add test case for binpac flowbuffer frame length parsing bug 2020-03-19 22:09:23 -07:00
Jon Siwek
b62727a7fa Merge branch 'intrusive_ptr' of https://github.com/MaxKellermann/zeek
* 'intrusive_ptr' of https://github.com/MaxKellermann/zeek: (32 commits)
  Scope: store IntrusivePtr in `local`
  Scope: pass IntrusivePtr to AddInit()
  DNS_Mgr: use class IntrusivePtr
  Scope: use class IntrusivePtr
  Attr: use class IntrusivePtr
  Expr: check_and_promote_expr() returns IntrusivePtr
  Frame: use class IntrusivePtr
  Val: RecordVal::LookupWithDefault() returns IntrusivePtr
  Type: RecordType::FieldDefault() returns IntrusivePtr
  Val: TableVal::Delete() returns IntrusivePtr
  Type: base_type() returns IntrusivePtr
  Type: init_type() returns IntrusivePtr
  Type: merge_types() returns IntrusivePtr
  Type: use class IntrusivePtr in VectorType
  Type: use class IntrusivePtr in EnumType
  Type: use class IntrusivePtr in FileType
  Type: use class IntrusivePtr in TypeDecl
  Type: make TypeDecl `final` and the dtor non-`virtual`
  Type: use class IntrusivePtr in TypeType
  Type: use class IntrusivePtr in FuncType
  ...
2020-03-17 22:51:46 -07:00
Jon Siwek
9c70bcecbc GH-865: fix parsing of SMB NegotiateContextList
* The compression capability was incorrectly set to 0x0004 instead of 0x0003

* The padding was 4-byte instead of 8-byte aligned and also the spec.
  does not strictly require the padding for the last item in the list.

* Add a default case to handle parsing of unknown context types.
2020-03-16 19:00:01 -07:00
Jon Siwek
7a397d2746 Initialize RecordVal default fields when redef'd
If a RecordVal had been created, but later its RecordType redef'd to
contain fields with &default, those fields were incorrectly left
uninitialized.
2020-03-12 18:46:37 -07:00
Jon Siwek
71b82595ba GH-857: fix redefining record types used to index tables
This change tracks all TableVals created at parse-time whose index
depends on a given RecordType.  Should that RecordType be redef'd, those
TableVals are immediately rebuilt such that they are valid to
subsequently use in either parse-time initializations or eventually in
any arbitrary run-time expression.
2020-03-12 18:25:48 -07:00
Johanna Amann
da5fca7163 Merge branch 'topic/johanna/gh-744-cache-the-heck-out-of-these-certs'
* topic/johanna/gh-744-cache-the-heck-out-of-these-certs:
  X509 caching API change: callback gets entry directly
  Address feedback
  Small default updates & external baselines.
  X509 caching: small API changes, tests & test updates
  Certificate caching - now working in principle.
  First (sadly not completely working) approach at caching X509 certificates
2020-03-12 13:44:07 -07:00
Jon Siwek
860f851a66 Update pointer to external test repo 2020-03-12 12:04:24 -07:00
Johanna Amann
e228061f1d X509 caching API change: callback gets entry directly
Suggestion from Jon: now the callback is passed the table-entry
directly. This allows us to skip one lookup.
2020-03-12 11:31:13 -07:00
Johanna Amann
0a7b358985 Small default updates & external baselines.
Update default caching time to be slightly more than a minute - to catch
cases that happen once a minute.

Also update external baselines.
2020-03-12 10:52:39 -07:00
Johanna Amann
4b09947f41 X509 caching: small API changes, tests & test updates
Changed some configuration defaults to potentially more same values.

The callback function is now a hook to allow costomization of the events
that are raised.

Tests now exist. Test baselines are updated.
2020-03-11 13:27:56 -07:00
Jon Siwek
26af1f55af Merge remote-tracking branch 'origin/topic/vladg/gh-843'
Added a comment to clarify comparison of EOF packet length to 13.

* origin/topic/vladg/gh-843:
  Update baselines
  Made additional MySQL fixes.
  Add support to MySQL for deprecation of EOF packets.
  Whitespace cleanup & fixes
  Fix EOF detection in the MySQL protocol analyzer.
2020-03-11 11:25:00 -07:00
Vlad Grigorescu
a362189856 Update baselines 2020-03-06 22:49:26 -06:00
Jon Siwek
363fc49518 Fix a btest to order zeek_init() events via &priority 2020-03-05 14:19:34 -08:00
Jon Siwek
a15f471a87 Update zeek-testing commit 2020-02-24 18:13:17 -08:00
Tim Wojtulewicz
822567b3f9 Merge remote-tracking branch 'origin/topic/jsiwek/32bit-compat'
* origin/topic/jsiwek/32bit-compat:
  Improve formatting of doubles that are close to integers
  Improve HTTP version number comparisons
  Add a 32-bit task to Cirrus CI config
  Replace va_list fmt() overload with vfmt()
  Format tables indexed by patterns consistently across 32-bit/64-bit
  Format interval values consistently across 32-bit/64-bit platforms
2020-02-24 18:49:33 -07:00
Jon Siwek
a785212e80 Improve formatting of doubles that are close to integers
Now checks for approximate floating point equality so that more doubles
get properly disambiguated from integers
2020-02-15 00:54:44 -08:00
Jon Siwek
11d8b8c040 Fix code format of various reporter btests 2020-02-14 22:03:11 -08:00
Jon Siwek
996b3d523a Update to start of 3.2.0 development 2020-02-08 16:08:01 -08:00
Jon Siwek
e50a8848ae Improve an input framework test
A race condition could cause unstable output: if the thread reading the
file is fast, often you see both "pred" functions execute and then both
"line" events execute with both entries already in the table, but if the
thread reading the file is slow, you see pred, event, pred, event, with
only one entry available in the first event.
2020-02-07 20:03:10 -08:00
Jon Siwek
a3b1d202a5 Improve stability of a &expire_func btest
The order in which &expire_func's get called isn't well-defined, so
separate the output from either to ensure diffs against the Baseline are
always consistent.
2020-02-07 15:11:06 -08:00
Johanna Amann
56dda98512 Merge remote-tracking branch 'origin/topic/jsiwek/stabilize-ci'
* origin/topic/jsiwek/stabilize-ci:
  Add environment variable to disable supervisor SIGKILL
  Fix a btest's race condition by handling Pcap::file_done
  Add Pcap::file_done event
  Improve btest timeouts
2020-02-07 09:52:57 -08:00
Johanna Amann
49a21d73d0 Merge remote-tracking branch 'origin/topic/jsiwek/gh-780-indefinite-log-buffer'
* origin/topic/jsiwek/gh-780-indefinite-log-buffer:
  GH-780: Prevent log batches from indefinite buffering
2020-02-07 09:35:29 -08:00
Jon Siwek
a2b0abe283 Add environment variable to disable supervisor SIGKILL
For use in btests since SIGKILL can otherwise bypass the usual
btest-bg-wait timeout mechanism and make reason for test failures less
obvious.
2020-02-06 17:50:17 -08:00
Jon Siwek
5b74bbf99a Fix a btest's race condition by handling Pcap::file_done 2020-02-06 17:50:17 -08:00
Jon Siwek
7c124881cd Add Pcap::file_done event
It signals when a pcap file is done being processed.
2020-02-06 17:50:17 -08:00
Jon Siwek
4fbcca04e8 Improve btest timeouts
* Generally increase timeouts for tests that have recent transient
  failures

* Change any test that relied on `btest-bg-wait -k` since that's never
  going to play with with CI systems.  Instead, we always need to have
  a well-defined termination condition in the test itself (and most
  already did, so didn't really need the `-k` flag anyway).
2020-02-06 17:50:17 -08:00
Johanna Amann
391164c9bc Merge remote-tracking branch 'origin/topic/johanna/table-on-change'
* origin/topic/johanna/table-on-change:
  &on_change: Address feedback of Jon.
  &on_change: incooperate feedback of Jon.
  &on_change wrapup: documentation, tests, whitespacing
  Table expiry now raises &on_change handlers
  Prevent recursion of &on_change handlers.
  Make bro_broker::val_to_data take a const Val* instead of a Val
  &on_change: add insertion/change notifications.
  &on_change working for removals from tables.
  Change signature of &on_change handler
  Beginning implementation of &on_change for tables.
2020-02-06 15:01:03 -08:00
Jon Siwek
c5340c7c3e Merge remote-tracking branch 'origin/topic/timw/773-eventmgr-iosource'
Fixes GH-773

* origin/topic/timw/773-eventmgr-iosource:
  GH-773: Make EventMgr an IOSource so that events interrupt kevent waits
2020-02-05 14:57:26 -08:00
Johanna Amann
b1040f88c9 &on_change wrapup: documentation, tests, whitespacing
Adds documentation, fixes a whitespace issues, fixes compiler warning on
some Linux system, extends test.
2020-02-05 14:15:14 -08:00
Jon Siwek
43e54c7930 GH-780: Prevent log batches from indefinite buffering
Logs that got sent sparsely or burstily would get buffered for long
periods of time since the logic to flush them only does so on the next
log write.  In the worst case, a subsequent log write could never happen
and cause a log entry to be indefinitely buffered.

This fix introduces a recurring event/timer to simply flush all pending
logs at frequency of Broker::log_batch_interval.
2020-02-05 13:06:52 -08:00
Tim Wojtulewicz
4f23c0360a GH-773: Make EventMgr an IOSource so that events interrupt kevent waits 2020-02-05 15:43:37 -05:00
Johanna Amann
b6a244f784 Merge remote-tracking branch 'origin/master' into topic/johanna/table-on-change 2020-02-05 12:41:19 -08:00
Johanna Amann
18ab7838a9 Table expiry now raises &on_change handlers
Expiry is remprted as ELEMENT_EXPIRED (which was newly introduced), to
allow distinction from table deletion.
2020-02-05 12:37:52 -08:00
Johanna Amann
7f9f66fce9 Prevent recursion of &on_change handlers.
This change prevents &on_change handlers for a table from running if an
&on_change handler for the same table is already running.
2020-02-05 11:45:16 -08:00