Commit graph

2808 commits

Author SHA1 Message Date
Tim Wojtulewicz
a7d3cb48ef Add concept of "parent" tag namespaces
This allows us to create an EnumType that groups all of the analyzer
tag values into a single type, while still having the existing types
that split them up. We can then use this for certain events that benefit
from taking all of the tag types at once.
2021-11-23 19:36:49 -07:00
Johanna Amann
14f919895d Add documentation for GH-1829
This adds documentation that clarifies that the `ignore_checksums`
option now also allows IPv4 packets with a length of 0.
2021-11-16 13:51:29 +00:00
Tim Wojtulewicz
a6378531db Remove trailing whitespace from script files 2021-10-20 09:57:09 -07:00
Johanna Amann
303e84ad86 Merge branch 'master' of https://github.com/FlyingWithJerome/zeek
Merge includes small changes, e.g. fixing the comsumption of remaining
raw data.

* 'master' of https://github.com/FlyingWithJerome/zeek:
  remove excussive fields in dns_svcb_rr
  address code reviews (formatting and type and intrusiveptr)
  newlines at the end of test outputs
  lazy commit
  use tabs in init-bare.zeek
  add svcb test case
  add a dns https test case
  remove test logs
  fix a few syntax errors
  initial commit for SVCB/HTTPS records
2021-10-19 15:03:08 +02:00
FlyingWithJerome
605d4024e4 remove excussive fields in dns_svcb_rr 2021-10-12 21:40:56 -04:00
FlyingWithJerome
c957e3e91e address code reviews (formatting and type and intrusiveptr) 2021-10-12 20:36:35 -04:00
FlyingWithJerome
b238cf3dca lazy commit 2021-10-12 17:43:32 -04:00
FlyingWithJerome
33c7fd5fba use tabs in init-bare.zeek 2021-10-12 17:43:32 -04:00
FlyingWithJerome
0849332eb9 fix a few syntax errors 2021-10-12 17:43:32 -04:00
FlyingWithJerome
8fce51bf83 initial commit for SVCB/HTTPS records 2021-10-12 17:43:32 -04:00
Tim Wojtulewicz
0f348ea042 GHI-1766: Remove address from Site::private_address_space that converts into 0.0.0.0/0 2021-09-27 14:24:14 -07:00
Tim Wojtulewicz
e5b163290d Merge remote-tracking branch 'origin/topic/vern/remove-uu'
* origin/topic/vern/remove-uu:
  fix up for linking w/ doc update
  documentation update
  script simplification that removes an unnecessary &is_assigned
  removing -uu functionality and associated script analysis now no longer needed
2021-09-24 10:31:56 -07:00
Robin Sommer
47c35190a4 Sanity-check the method passed into ActiveHTTP.
Reported by Pierre Gaulon.
2021-09-23 12:21:23 +02:00
Tim Wojtulewicz
0a0ed65306 Merge remote-tracking branch 'origin/topic/robin/gh-54-sanitize'
* origin/topic/robin/gh-54-sanitize:
  Sanitize log files names before they go into system().
2021-09-22 12:17:05 -07:00
Tim Wojtulewicz
a49dcc8954 Merge remote-tracking branch 'origin/topic/johanna/dpd-packet-limit'
* origin/topic/johanna/dpd-packet-limit:
  PIA - switch size to int64_t
  Introduce dpd_max_packets
2021-09-22 12:16:56 -07:00
Vern Paxson
385e49491b script simplification that removes an unnecessary &is_assigned 2021-09-22 11:18:52 -07:00
Robin Sommer
74680bf4e6 Merge remote-tracking branch 'origin/topic/justin/software-framework-parse-cache'
* origin/topic/justin/software-framework-parse-cache:
  Restore behavior of Software::register event
  Optimzie software framework version parsing
2021-09-21 18:00:46 +02:00
Robin Sommer
2fc12d5bed Merge branch 'topic/foxds/dce_itype_opnums' of ssh://github.com/fox-ds/zeek
* 'topic/foxds/dce_itype_opnums' of ssh://github.com/fox-ds/zeek:
  Add IType opnum mapping
2021-09-21 17:56:13 +02:00
Justin Azoff
ef5fb790ef Restore behavior of Software::register event
Use an intermediary event to ensure that software versions are parsed
before calling Software::register.
2021-09-20 14:38:47 -04:00
Robin Sommer
31d3fb0f6c Merge https://github.com/gpotter2/zeek.
Changes during merge:
  - Add dedicated test (w/ trace "client_timestamp_enabled.pcapng" from Cloudshark)
  - Change types from signed to unsigned.
  - Add cast for bit-shifting operand.
  - clang-format run
2021-09-20 11:41:29 +02:00
FOX-DS
4a19acbef2 Add IType opnum mapping 2021-09-20 03:56:20 -04:00
gpotter2
d4db9bf6d0 Add TSval and TSecr to TCPSyn 2021-09-17 11:35:16 +02:00
Robin Sommer
6c128a21e1 Merge branch 'topic/foxds/dcerpc_ms-oaut' of ssh://github.com/fox-ds/zeek
* 'topic/foxds/dcerpc_ms-oaut' of ssh://github.com/fox-ds/zeek:
  Added four new opnum mappings for MS-OAUT IDispatch methods
2021-09-16 11:15:36 +02:00
Tim Wojtulewicz
0dca1a70a7 Reformat docs in addrs.zeek to fix doc generation 2021-09-14 19:26:28 -07:00
FOX-DS
d3ca226e96 Added four new opnum mappings for MS-OAUT IDispatch methods 2021-09-14 15:31:08 +02:00
Seth Hall
a4ceb98bf8 Switch the TSV Zeek logs to be UTF8 by default.
There is a paired zeek-testing branch for some updates there.
2021-09-07 09:16:53 -07:00
Justin Azoff
3bf8c8ceb6 Optimzie software framework version parsing
Add a small cache in front of the parse method.  This cache should
reduce most of the calls to parse, and ultimately save memory because
redundant versions of the parsed strings will not be created in memory.

Move the parsing itself to the proxies where the caching can be more
efficient.
2021-09-05 13:18:51 -04:00
Tim Wojtulewicz
569552b320 GH-1589: Avoid extracting IP-like strings from SMTP headers 2021-09-03 17:35:10 +00:00
Christian Kreibich
90b7c6961e Simplify the supervisor's listen() on default address/port 2021-08-18 12:35:49 -07:00
Johanna Amann
8192ad581d Do not lookup ignore_checksums_nets for every packet
This could lead to a noticeable (single-percent) performance
improvement.

Most of the functionality for this is in the packet analyzers that now
cache ignore_chesksums_nets.

Based on a patch by Arne Welzel (Corelight).
2021-08-06 10:32:53 +01:00
Tim Wojtulewicz
3038ec9578 Merge remote-tracking branch 'origin/topic/vlad/bogon'
* origin/topic/vlad/bogon:
  Zeekygen doesn't support comments on set members
  Add 6to4 addresses, if the matching IPv4 address is private
  Update Site::private_address_space to latest IANA networks
2021-07-13 12:25:32 -07:00
Christian Kreibich
281eef398f Merge branch 'topic/vlad/doc_tweaks'
* topic/vlad/doc_tweaks:
  docs: escape * in documentation
  Update the mail_dest documentation with the new email_dest field
2021-07-09 12:02:32 -07:00
Christian Kreibich
8db985ea78 Merge branch 'topic/christian/cluster-controller'
* topic/christian/cluster-controller:
  Add a cluster controller testcase for agent-controller checkin
  Add zeek-client via new submodule
  Update baselines affected by cluster controller changes
  Introduce cluster controller and cluster agent scripting
  Establish a separate init script when using the supervisor
  Add optional bare-mode boolean flag to Supervisor's node configuration
  Add support for making the supervisor listen for requests
  Add support for setting environment variables via supervisor
2021-07-08 16:51:11 -07:00
Christian Kreibich
6eddaed62d Fix reference in the logging framework docs re. Log::StreamPolicyHook 2021-07-08 15:39:46 -07:00
Vlad Grigorescu
10159a3fdf Update the mail_dest documentation with the new email_dest field 2021-07-08 16:01:21 -05:00
Christian Kreibich
a3623bfb2d Establish a separate init script when using the supervisor
The supervisor does not require the full weight of scripts that
init-default.zeek brings with it. The new file, init-supervisor.zeek, contains
only what's required by the supervisor in addition to the other always-loaded
init files.
2021-07-08 13:12:53 -07:00
Christian Kreibich
7bee79b400 Add optional bare-mode boolean flag to Supervisor's node configuration
When omitted, the node inherits the Supervisor's bare-mode
status. When true/false, the new Zeek node will enable/disable bare
mode, respectively. It continues to load any scripts passed at the
command line and in the additional scripts list already provided in
the node configuration.

Includes testcase.
2021-07-08 13:12:53 -07:00
Christian Kreibich
efaa9ec3be Add support for making the supervisor listen for requests
The supervisor now starts listening on the configured Broker default address and
port when the new boolean SupervisorControl::enable_listen is T. Listening
remains disabled by default. Listening allows nodes to communicate with the
supervisor via the events laid out in control.zeek, to conduct further node
management.
2021-07-08 13:12:53 -07:00
Christian Kreibich
36051dc9a1 Add support for setting environment variables via supervisor
The NodeConfig record now has a table for specifying environment variable names
and values, which the supervisor sets in the created node.

This also repositions the cpu_affinity member to keep the order the same in
the corresponding script-layer and in-core types.

Includes testcase.
2021-07-08 13:12:53 -07:00
Vlad Grigorescu
5f5ea7c031 Zeekygen doesn't support comments on set members 2021-07-08 06:15:21 -05:00
Vlad Grigorescu
d930f6aa7d Add 6to4 addresses, if the matching IPv4 address is private 2021-07-08 06:13:51 -05:00
Vlad Grigorescu
632182d1a9 Update Site::private_address_space to latest IANA networks 2021-07-07 14:41:31 -05:00
Johanna Amann
132b9ef2cc Add a few DNS query types. 2021-07-06 10:40:22 +01:00
Johanna Amann
7ec50bf434 Merge remote-tracking branch 'origin/topic/johanna/gh-859'
* origin/topic/johanna/gh-859:
  Add X509/SSL changes to NEWS
  X509: add check if function succeeds
  GH-1634: Address feedback
  Small indentation fixes in ssl-log-ext.zeek
  Fix memory leak in x509_check_cert_hostname bif
  Small bugfix and updates for external test hashes (SSL/X509)
  Baseline updates for recent SSL changes.
  Add ability to check if hostname is valid for a specific cert
  Add ssl_history field to ssl.log
  Add policy script suppressing certificate events
  Add new ssl-log-ext policy script
  Deprecate extract-certs-pem.zeek and add log-certs-base64.zeek
  Implement X509 certificate log caching
  Deprecate ICSI SSL notary script.
  Change SSL and X.509 logging format
  Enable OCSP logging by default.
  Split the code that handles X509 event hashing into its own file

Closes GH-859
2021-07-05 10:12:46 +01:00
Christian Kreibich
795a7ea98e Add a global log policy hook to the logging framework
This addresses the need for a central hook on any log write, which
wasn't previously doable without a lot of effort. The log manager
invokes the new Log::log_stream_policy hook prior to any filter-specific
hooks. Like filter-level hooks, it may veto a log write. Even when
it does, filter-level hooks still get invoked, but cannot "un-veto".

Includes test cases.
2021-07-02 12:42:45 -07:00
Johanna Amann
8972676e92 GH-1634: Address feedback
This commit addreses feedback for GH-1643, changing typos and renaming
one of the fields in x509.log.
2021-07-02 15:12:58 +01:00
Johanna Amann
a193b08ecf Update Mozilla CA list and CT anchors.
This commit switches to only allowing the CT logs that are currently
accepted by Google Chrome - which makes much more sense for us since
this is (potentially) used for validation.

Additional CT logs can be added in user-scripts.
2021-06-30 15:14:22 +01:00
Johanna Amann
509b501e1b Merge remote-tracking branch 'origin/master' into topic/johanna/gh-859 2021-06-30 10:36:54 +01:00
Robin Sommer
369e42a6e4 Fix SMB tests on Apple M1.
Due to different double precision on M1, file IDs for SMB could end up
changing on M1 because the access time of a file goes into their
computation. The real solution for this would be changing Zeek's
internal "time" representation to uint64; that's planned, but requires
major surgery. For now, this PR changes the SMB code to also pass SMB's
original time representation (which is a uint64) into script-land, and
then use that for computing the file ID.

Closes #1406
2021-06-29 20:17:02 +02:00
Johanna Amann
93d7778f97 Small bugfix and updates for external test hashes (SSL/X509) 2021-06-29 15:25:08 +01:00