Commit graph

2173 commits

Author SHA1 Message Date
Tim Wojtulewicz
503ef26a17 Merge remote-tracking branch 'origin/topic/jsiwek/gh-893-intrusive-ptr-migration'
* origin/topic/jsiwek/gh-893-intrusive-ptr-migration: (151 commits)
  Integrate review feedback
  Switch Broker Val converter visitor to return IntrusivePtr
  Change BroFunc ctor to take const-ref IntrusivePtr<ID>
  Add version of Frame::SetElement() taking IntrusivePtr<ID>
  Change Scope/Func inits from id_list* to vector<IntrusivePtr<ID>>
  Change Scope::GenerateTemporary() to return IntrusivePtr
  Deprecate Scope::ReturnType(), replace with GetReturnType()
  Deprecate Scope::ScopeID(), replace with GetID()
  Switch parsing to use vector<IntrusivePtr<Attr>> from attr_list
  Deprecate TableVal::FindAttr(), replace with GetAttr()
  Deprecate TypeDecl::FindAttr(), replace with GetAttr()
  Deprecate ID::FindAttr(), replace with GetAttr()
  Deprecate Attributes::FindAttr(), replace with Find()
  Deprecate Attributes::AddAttrs(Attributes*)
  Add Attributes ctor that takes IntrusivePtrs
  Change Attributes to store std:vector<IntrusivePtr<Attr>>
  Change Attr::SetAttrExpr() to non-template
  Deprecate Attr::AttrExpr(), replace with GetExpr()
  Deprecate ID::Attrs(), replace with GetAttrs()
  Remove weak_ref param from ID::SetVal()
  ...
2020-06-01 10:58:02 -07:00
Johanna Amann
433e1154da Merge branch 'add_bzar_dce_rpc_consts' of https://github.com/ct-square/zeek
* 'add_bzar_dce_rpc_consts' of https://github.com/ct-square/zeek:
  Remove dupplicate DCE-RPC endpoint
  Add DCE-RPC constants from BZAR project

Closes GH-953
2020-05-26 22:04:33 +00:00
Jon Siwek
78e3267c44 Deprecate internal_handler(), replace with EventRegistry::Register()
Added a couple explicit event declarations that were missing: "net_done"
and "dns_mapping_name_changed".
2020-05-14 17:25:02 -07:00
Johanna Amann
2aeb3d8e39 Merge remote-tracking branch 'origin/topic/timw/906-find-all-urls-regex'
* origin/topic/timw/906-find-all-urls-regex:
  Restore previous url scheme capture group
  GH-906: Fix the regex in url.zeek to better match for find_all_urls
2020-05-13 15:05:54 -07:00
Johanna Amann
a259e8bbda Merge remote-tracking branch 'origin/master' into topic/johanna/hash-unification 2020-05-12 00:29:02 +00:00
Jon Siwek
b5531ecbd3 Merge branch 'set_to_regex-docs' of https://github.com/jlagermann/zeek
- Adjusted the formatting during merge

* 'set_to_regex-docs' of https://github.com/jlagermann/zeek:
  added examples to set_to_regex comments Signed-ff-by: James Lagermann <james.lagermann@corelight.com>
2020-05-08 11:48:44 -07:00
James Lagermann
2c04a56236
added examples to set_to_regex comments
Signed-ff-by: James Lagermann <james.lagermann@corelight.com>
2020-05-08 12:31:56 -05:00
Johanna Amann
04ed125941 Merge remote-tracking branch 'origin/master' into topic/johanna/hash-unification 2020-05-06 23:18:33 +00:00
Jon Siwek
b749dda520 Fix SSL scripting error leading to access of unitialized field
Reported by Justin Azoff
2020-05-06 09:52:31 -07:00
Johanna Amann
7d28a6ee9a Remove outdated comment on set_to_regex.
We can add patterns at runtime since 2.6.
2020-05-05 14:23:33 -07:00
Jon Siwek
156686b237 Correct spelling of DCE/RPC operation string NetrLogonSameLogonWithFlags
Fixes GH-952
2020-05-04 18:03:14 -07:00
V
45a5b1b0cf Remove dupplicate DCE-RPC endpoint 2020-05-04 18:02:04 +02:00
V
7cf8c7a6d2 Add DCE-RPC constants from BZAR project 2020-05-04 17:15:27 +02:00
Johanna Amann
3bce313b12 Switch file UID hashing from md5 to highwayhash.
This commit switches UID hashing from md5 to a highway hash. It also
moves the salt value out of the file plugin - and makes it
installation-specific instead - it is moved to the global namespace.

There now are digest hash functions to make "static"
installation-specific hashes that are stable over workers available to
everyone; hashes can be 64, 128 or 256 bits in size.

Due to the fact that we switch the file hashing algorithm, all file
hashes change.

The underlyigng algorithm that is used for hashing is highwayhash-128,
which is significantly faster than md5.
2020-04-30 10:20:09 -07:00
Seth Hall
dac96a6be3 Fixes a small bug in one signature with a duplicate name.
Also update a single failing test.
2020-04-29 11:22:42 -04:00
Seth Hall
15d43dfbcd Organized and added to the shipped file identification signatures.
- Added ISO 9660 disk image
 - Created new files for categorizing signatures better.
   - executable.sig - Executable (and bytecode) files.
   - java.sig - Java related files (class/jar, etc).
   - programming.sig - Mostly scripting language identification
2020-04-29 11:08:32 -04:00
Johanna Amann
faa8a38578 Merge remote-tracking branch 'origin/topic/jsiwek/gh-854-preserve-header-name'
* origin/topic/jsiwek/gh-854-preserve-header-name:
  GH-854: provide access to original HTTP/MIME header names
2020-04-27 19:31:49 +00:00
Vern Paxson
fe46ef06a0 unused variables found via use-def analysis (plus an indentation micro-nit) 2020-04-25 18:06:47 -07:00
Jon Siwek
5032993b94 GH-854: provide access to original HTTP/MIME header names
The "http_header" event now has an "original_name" parameter that allows
access to the original header name (the "name" parameter reamins the
same as before: it's the uppercased header name).

The "mime_header_rec" record type now also includes an "original_name"
field to similarly provide access to original header name in the
following events: "http_all_headers", "mime_one_header", and
"mime_all_headers".
2020-04-20 16:56:41 -07:00
Jon Siwek
8843f69002 Remove ineffective &default in netcontrol cluster event handler args 2020-04-16 15:40:27 -07:00
Jon Siwek
c8e070b8ee Add default function for Kerberos constant-lookup-tables 2020-04-16 12:34:41 -07:00
Tim Wojtulewicz
612c59e099 Restore previous url scheme capture group 2020-04-14 16:33:19 -07:00
Tim Wojtulewicz
ba1c03188f Merge remote-tracking branch 'origin/topic/jsiwek/alternate-hook-event-prototypes'
* origin/topic/jsiwek/alternate-hook-event-prototypes:
  Add warning for ineffective &default arguments in handlers
  Fix frame size allocation of alternate event/hook handlers
  Emit error for alternate event/hook prototype args with attributes
  Improve alternate event/hook prototype matching
  Allow alternate event/hook prototype declarations
2020-04-13 15:00:25 -07:00
Tim Wojtulewicz
0d31d39de9 GH-906: Fix the regex in url.zeek to better match for find_all_urls 2020-04-13 13:17:57 -07:00
Jon Siwek
ce9183a2ed Fix Broker topics used to uniquely identify cluster nodes
Node-specific topic prefix subscriptions/publications now add a trailing
slash like "zeek/cluster/node/<name>/".  Without the trailing slash,
messages attempting to target "proxy-10" may also be sent to "proxy-1"
since subscription matching is prefix-based.
2020-04-10 14:36:00 -07:00
Jon Siwek
640dbea57c Add warning for ineffective &default arguments in handlers
For event/hook handlers that had a previous declaration, any &default
arguments are ineffective.  Only &default uses in the initial
prototype's arguments have an effect (that includes if the handler
is actually the site at which the declaration occurs).
2020-04-09 22:51:01 -07:00
Jon Siwek
c1b3c9593b Add "udp_content_ports" option
Any port in that set found as either source or destination port
of a UDP packet will cause the "udp_contents" event to be raised.
2020-04-07 13:02:29 -07:00
Jon Siwek
2da84020cf Add new "udp_content_delivery_ports_use_resp" option
This controls whether ports given by "udp_content_delivery_ports_orig" and
"udp_content_delivery_ports_orig" are in terms of the UDP packet's
destination port or by the Connection's "responder" port (the former is
the unchanged default behavior).
2020-04-06 14:51:34 -07:00
Jon Siwek
e66148a13a Merge branch 'topic/ak/rdpeudp'
- Squashed the original commit set
- Cleaned up formatting
- Fixed register_for_ports() for right RDPEUDP analyzer

* topic/ak/rdpeudp:
  Add RDP over UDP analyzer
2020-04-02 18:31:40 -07:00
Anthony Kasza
60644bc85f Add RDP over UDP analyzer 2020-04-02 17:53:47 -07:00
Robin Sommer
f62d7dd091 Merge branch 'topic/antonio.nappa/fix_lowercase_intel_bug' of ssh://github.com/jeppojeps/zeek 2020-03-31 07:08:14 +00:00
Antonio Nappa
cc309a5c99 fixed a newline issue with baselines 2020-03-27 11:27:28 -04:00
Antonio Nappa
dd6d379b40 fix-lowercase-intel-bug 2020-03-26 15:55:01 -04:00
Jon Siwek
13391806ba Add a "reserved" field to tcp_hdr record
This record is available by handling either raw_packet or new_packet
events.
2020-03-26 11:48:53 -07:00
M.Shirk
9cd914b5cf Duplicate smb2_negotiate_response events defined. 2020-03-16 11:47:39 -04:00
Johanna Amann
da5fca7163 Merge branch 'topic/johanna/gh-744-cache-the-heck-out-of-these-certs'
* topic/johanna/gh-744-cache-the-heck-out-of-these-certs:
  X509 caching API change: callback gets entry directly
  Address feedback
  Small default updates & external baselines.
  X509 caching: small API changes, tests & test updates
  Certificate caching - now working in principle.
  First (sadly not completely working) approach at caching X509 certificates
2020-03-12 13:44:07 -07:00
Johanna Amann
e228061f1d X509 caching API change: callback gets entry directly
Suggestion from Jon: now the callback is passed the table-entry
directly. This allows us to skip one lookup.
2020-03-12 11:31:13 -07:00
Johanna Amann
3ed9379b9e Address feedback
Smaller fixes. I split out the API change of the fallback function into
a separate commit.
2020-03-12 11:21:39 -07:00
Johanna Amann
0a7b358985 Small default updates & external baselines.
Update default caching time to be slightly more than a minute - to catch
cases that happen once a minute.

Also update external baselines.
2020-03-12 10:52:39 -07:00
Jon Siwek
b811f7cee3 Fix whitespace in connection record comments 2020-03-11 16:40:20 -07:00
Johanna Amann
4b09947f41 X509 caching: small API changes, tests & test updates
Changed some configuration defaults to potentially more same values.

The callback function is now a hook to allow costomization of the events
that are raised.

Tests now exist. Test baselines are updated.
2020-03-11 13:27:56 -07:00
Johanna Amann
65e99bafed Certificate caching - now working in principle.
I moved the replay function to a callback - which now means that the replayed
functions are called before file_state remove. Which makes this virtually
identical with the events raised by the core.

Currently this is mostly missing tests, leak-testing and performance-tests.
2020-03-09 14:40:10 -07:00
Johanna Amann
0829164a3e First (sadly not completely working) approach at caching X509 certificates
This approach mostly relies on script-level changes. In scriptland, we track
which certificates should be cached - and also perform the caching and the
replaying of events there.

This actually is probably nearly functional - the problem is the fact that now
the event ordering is wrong - and file_state_remove is called before the x509
events are thrown.

The fix probably is to move to a callback from the core - which can execute
things in the right order again. (Or just write the whole event-raising inside
the core - but that is both less flexible and a lot more cumbersome).
2020-03-06 15:21:45 -08:00
Jon Siwek
43e54c7930 GH-780: Prevent log batches from indefinite buffering
Logs that got sent sparsely or burstily would get buffered for long
periods of time since the logic to flush them only does so on the next
log write.  In the worst case, a subsequent log write could never happen
and cause a log entry to be indefinitely buffered.

This fix introduces a recurring event/timer to simply flush all pending
logs at frequency of Broker::log_batch_interval.
2020-02-05 13:06:52 -08:00
Johanna Amann
c20551e48e Update submodule
[nomail]
2020-02-03 13:23:22 -08:00
Tim Wojtulewicz
be42608b51 Remove concept of multiple timer managers
- All timers are now handled by a single global timer manager, which simplifies how they handled by the IOSource manager.
- This change flows down a number of changes to other parts of the code. The timer manager tag field is removed, which means that matching connections to a timer manager is also removed. This removes the ability to tag a connection as internal or external, since that's how the connections where differentiated. This in turn removes the `current_conns_extern` field from the `ConnStats` record type in the script layer.
2020-01-31 10:13:09 -07:00
Tim Wojtulewicz
f16f0360ff Only allow a single trace file (-r) or interface (-i) option on the command-line 2020-01-31 09:34:54 -07:00
Jon Siwek
70b45d1aba Merge remote-tracking branch 'origin/topic/robin/631-deprecation-v2'
During merge I split the test for bro_init/bro_done/bro_script_loaded
event errors into individual tests since the other testing of the zeek
versions of those events seemed fine to otherwise keep.

* origin/topic/robin/631-deprecation-v2:
  Update NEWS for naming changes.
  Small cleanup and updating submodules.
  Remove test for legacy plugin.
  Remove legancy symlinks in aux/.
  Add warnings when loading scripts ending in ".bro", or using legacy environment variables.
  Fix missing rename.
  No longer symlink local.zeek to local.bro.
  Update notice user agent.
  Remove old_comm_usage_is_ok.
  Remove bro-config.h.in and bro-path-dev.in.
  Change Bro wrapper script to now abort when old executable names are still used.
  Remove APIs that were explicitly deprecated to be removed in 3.1.
2020-01-30 19:19:56 -08:00
Robin Sommer
6bcd583836 Merge remote-tracking branch 'origin/topic/jsiwek/supervisor'
* origin/topic/jsiwek/supervisor: (44 commits)
  Add note that Supervisor script APIs are unstable until 4.0
  Move command-line arg parsing functions to Options.{h,cc}
  Add btests for supervisor stem/leaf process revival
  Move supervisor control events into SupervisorControl namespace
  Fix supervisor "destroy" call on nodes not currently alive
  Move supervisor source files into supervisor/
  Address supervisor code re-factoring feedback from Robin
  Convert supervisor internals to rapidjson
  Add Supervisor documentation
  Add supervisor btests
  Improve logging of supervised node errors
  Fix supervised node inheritence of command-line script paths
  Improve normalize_path() util function
  Use a timer to check for death of supervised node's parent
  Improve supervisor checks for parent process termination
  Improve handling of premature supervisor stem exit
  Improve supervisor signal handler safety
  Remove unused supervisor config options
  Cleanup minor Supervisor TODOs
  Improve supervisor debug logging
  ...
2020-01-29 13:11:04 +00:00
Robin Sommer
ad8e659943 Update notice user agent. 2020-01-29 12:08:10 +00:00