Commit graph

312 commits

Author SHA1 Message Date
Dominik Charousset
647fdf7737 Add facade types to avoid using raw Broker types
By avoiding to use `broker::data` directly, we gain a degree of freedom
that allows us to swap out `broker::data` for something else (e.g.,
`broker::variant`) in the future. Furthermore, it also helps us to keep
Broker types "local" to the Broker manager and gives us a nicer
interface.

Also replaces uses of `broker::expected` with `std::optional`. While an
`expected `can carry additional information as to why a value is not
present, nothing in Zeek ever cared about that. Hence, using
`std::optional` removes an unnecessary dependency on a Broker detail
while also being more efficient (no extra heap allocation when no value
is present).
2023-12-04 15:23:28 +01:00
Arne Welzel
02d00a1984 OCSP: Open-code unknown revoke reason strings
OpenSSL 3.2.0 knows about more reasons. Add some backwards compatibility.

Reference: 1c8a7f5091
2023-11-27 20:53:00 +01:00
Dominik Charousset
c500370563 Avoid OpenSSL header dependencies 2023-11-03 15:54:46 +01:00
Benjamin Bannier
f5a76c1aed Reformat Zeek in Spicy style
This largely copies over Spicy's `.clang-format` configuration file. The
one place where we deviate is header include order since Zeek depends on
headers being included in a certain order.
2023-10-30 09:40:55 +01:00
Arne Welzel
ab157777e3 X509Common/SCT parsing: Fix memory leak in error case
Failed to free the copy of the OCSP extension's value the inner
octet string couldn't be parsed.
2023-10-23 16:57:15 +02:00
Johanna Amann
e18edfa452 Add extract_limit_includes_missing option for file extraction
Setting this option to false does not count missing bytes in files towards the
extraction limits, and allows to extract data up to the desired limit,
even when partial files are written.

When missing bytes are encountered, files are now written as sparse
files.

Using this option requires the underlying storage and utilities to support
sparse files.
2023-09-14 12:11:42 -07:00
Johanna Amann
9928f7efb7 File extraction: use fseek
In the past, we allocated a buffer with zeroes and wrote that with
fwrite. Now, instead we just fseek to the correct offset.

This changes the way in which the file extract limit is counted a bit;
skipped bytes do no longer count against the file size limit.

(cherry picked from commit 5071592e9b7105090a1d9de19689c499070749d4)
2023-09-14 12:11:37 -07:00
Tim Wojtulewicz
5934e143aa Revert "Add extract_limit_includes_missing option for file extraction"
This reverts commit f4d0fdcd5c.
2023-09-14 12:10:40 -07:00
Johanna Amann
f4d0fdcd5c Add extract_limit_includes_missing option for file extraction
Setting this option to false does not count missing bytes in files towards the
extraction limits, and allows to extract data up to the desired limit,
even when partial files are written.

When missing bytes are encountered, files are now written as sparse
files.

Using this option requires the underlying storage and utilities to support
sparse files.

(cherry picked from commit afa6f3a0d3b8db1ec5b5e82d26225504c2891089)
2023-09-12 12:00:36 -07:00
Arne Welzel
7a043e5e8f all: Fix typos identified by typos pre-commit hook 2023-06-13 17:57:32 +02:00
Tim Wojtulewicz
84e3f414a7 Use the same rules as cmake submodule to reformat Zeek 2023-05-09 08:31:43 -07:00
Tim Wojtulewicz
29b0380815 Use workaround for setvbuf on Windows in DebugLogger/Extract file analyzer 2023-05-02 10:28:50 -07:00
Dominik Charousset
9aeed5284d Redesign subdir-libs and plugin scaffolding 2023-04-13 22:23:46 +02:00
Johanna Amann
989e9c29d2 X.509: expose the signature type inside the tbs certificate
This change exposes the signature tyope inside the signed portion of an
X.509 certificate. In the past, we only exposed the signature type that
is contained inside the signature, which is outside the signed portion
of the X.509 certificate.

In theory, both signature fields should have the same value; it is,
however, possible to encode differing values in both fields. The new
field is not logged by default.
2023-02-28 19:24:16 +00:00
Tim Wojtulewicz
1766aa2808 GH-2730: Use binary mode when opening files in Extract file analyzer 2023-02-23 08:22:57 -07:00
Tim Wojtulewicz
a8fc63e182 Merge remote-tracking branch 'microsoft/master'
* microsoft/master: (71 commits)
  Clang formatting
  Mask ports before inserting them into the map
  Fix compiler warning from applied patch
  Remove statistics plugin in favor of stats bif
  Add EventHandler version of stats plugin
  Mark a few EventHandler methods const
  Changed implementation from std::map to std::unordered_map of Val.cc
  Removed const, Windows build is now working
  Added fixes suggested in PR
  Update src/packet_analysis/protocol/ip/IP.cc
  Apply suggestions from code review
  Clang format again but now with v13.0.1
  Rewrote usages of define(_MSC_VER) to ifdef _MSC_VER
  Clang format it all
  Fixed initial CR comments
  Add NEWS entry about Windows port
  Add a couple of extra unistd.h includes to fix a build failure
  Use std::chrono instead of gettimeofday
  Update libkqueue submodule [nomail]
  Don't call tokenize_string if the input string is empty
  ...
2022-11-11 15:23:21 -07:00
Josh Soref
cd201aa24e Spelling src
These are non-functional changes.

* accounting
* activation
* actual
* added
* addresult
* aggregable
* aligned
* alternatively
* ambiguous
* analysis
* analyzer
* anticlimactic
* apparently
* application
* appropriate
* arithmetic
* assignment
* assigns
* associated
* authentication
* authoritative
* barrier
* boundary
* broccoli
* buffering
* caching
* called
* canonicalized
* capturing
* certificates
* ciphersuite
* columns
* communication
* comparison
* comparisons
* compilation
* component
* concatenating
* concatenation
* connection
* convenience
* correctly
* corresponding
* could
* counting
* data
* declared
* decryption
* defining
* dependent
* deprecated
* detached
* dictionary
* directional
* directly
* directory
* discarding
* disconnecting
* distinguishes
* documentation
* elsewhere
* emitted
* empty
* endianness
* endpoint
* enumerator
* essentially
* evaluated
* everything
* exactly
* execute
* explicit
* expressions
* facilitates
* fiddling
* filesystem
* flag
* flagged
* for
* fragments
* guarantee
* guaranteed
* happen
* happening
* hemisphere
* identifier
* identifies
* identify
* implementation
* implemented
* implementing
* including
* inconsistency
* indeterminate
* indices
* individual
* information
* initial
* initialization
* initialize
* initialized
* initializes
* instantiate
* instantiated
* instantiates
* interface
* internal
* interpreted
* interpreter
* into
* it
* iterators
* length
* likely
* log
* longer
* mainly
* mark
* maximum
* message
* minimum
* module
* must
* name
* namespace
* necessary
* nonexistent
* not
* notifications
* notifier
* number
* objects
* occurred
* operations
* original
* otherwise
* output
* overridden
* override
* overriding
* overwriting
* ownership
* parameters
* particular
* payload
* persistent
* potential
* precision
* preexisting
* preservation
* preserved
* primarily
* probably
* procedure
* proceed
* process
* processed
* processes
* processing
* propagate
* propagated
* prototype
* provides
* publishing
* purposes
* queue
* reached
* reason
* reassem
* reassemble
* reassembler
* recommend
* record
* reduction
* reference
* regularly
* representation
* request
* reserved
* retrieve
* returning
* separate
* should
* shouldn't
* significant
* signing
* simplified
* simultaneously
* single
* somebody
* sources
* specific
* specification
* specified
* specifies
* specify
* statement
* subdirectories
* succeeded
* successful
* successfully
* supplied
* synchronization
* tag
* temporarily
* terminating
* that
* the
* transmitted
* true
* truncated
* try
* understand
* unescaped
* unforwarding
* unknown
* unknowndata
* unspecified
* update
* usually
* which
* wildcard

Signed-off-by: Josh Soref <2119212+jsoref@users.noreply.github.com>
2022-11-09 12:08:15 -05:00
Elad Solomon
3a80b79497 Compile Zeek with MSVC
Allow Zeek to be embedded in another project
2022-11-09 18:15:30 +02:00
Arne Welzel
d5cd023dff file_analysis: Implement AnalyzerViolation() for file_analysis/Analyzer
Add a test parsing a malformed PE file showing that analyzer_violation_info
is raised with the fa_file object set.

It could be interesting to pass through an optional connection if one
exists, but access is provided through f$conns, too.
2022-09-27 17:49:58 +02:00
Arne Welzel
8d19fa23ef Remove unified2 file analyzer 2022-08-19 14:05:00 +02:00
Tim Wojtulewicz
d73f9a78dd Remove unnecessary uses of util::fmt() 2022-05-16 13:49:04 -07:00
Tim Wojtulewicz
b850d1dc51 Store hash analyzer kind StringVals for later reuse 2022-05-10 15:15:41 -07:00
Robin Sommer
5f1f005142
Merge remote-tracking branch 'origin/topic/neverlord/hide-caf'
* origin/topic/neverlord/hide-caf:
  Fix GCC builds and string output for Broker errors
  Update to latest Broker without public CAF dep
2022-02-07 16:25:22 +01:00
Vern Paxson
d758585e42 updated Bro->Zeek in comments in the source tree 2022-01-24 14:26:20 -08:00
Dominik Charousset
56f30b500a Update to latest Broker without public CAF dep 2021-12-20 08:16:21 +01:00
Tim Wojtulewicz
8b544d648d Remove uses of deprecated Tag types 2021-11-23 19:36:49 -07:00
Tim Wojtulewicz
331161138a Unify all of the Tag types into one type
- Remove tag types for each component type (analyzer, etc)
- Add deprecated versions of the old types
- Remove unnecessary tag element from templates for TaggedComponent and ComponentManager
- Enable TaggedComponent to pass an EnumType when initializing Tag objects
- Update some tests that are affected by the tag enum values changing order
2021-11-23 19:36:49 -07:00
Johanna Amann
ff0bc86021 OpenSSL 3: normalize self-signed-cert error message
The message changed between OpenSSL 1.1 and 3.0 - we just use the old
one to not mess up the external test baselines.
2021-11-17 17:21:21 +00:00
Johanna Amann
253d214126 OpenSSL 3: fix warnings and tests
This commit fixes the compile-time warnings that OpenSSL 3.0 raises for
our source-code. For the cases where this was necessary we now have two
implementations - one for OpenSSL 1.1 and earlier, and one for OpenSSL
3.0.

This also makes our testsuite pass with OpenSSL 3.0

Relates to GH-1379
2021-11-17 13:33:49 +00:00
Tim Wojtulewicz
9af6b2f48d clang-format: Set penalty for breaking after assignment operator 2021-09-27 10:49:48 -07:00
Tim Wojtulewicz
4423574d26 clang-format: Set IndentCaseBlocks to false 2021-09-27 10:49:48 -07:00
Tim Wojtulewicz
b2f171ec69 Reformat the world 2021-09-16 15:35:39 -07:00
Vlad Grigorescu
8d2a6f48e8 docs: escape * in documentation 2021-07-08 18:26:42 -05:00
Johanna Amann
43ef7d43fb X509: add check if function succeeds
Addresses feedback of GH-1634
2021-07-05 09:40:04 +01:00
Johanna Amann
4ad5d2031b Fix memory leak in x509_check_cert_hostname bif 2021-06-30 10:16:38 +01:00
Johanna Amann
833168090a Add ability to check if hostname is valid for a specific cert
This commit adds two new bifs, x509_check_hostname and
x509_check_cert_hostname. These bifs can be used to check if a given
hostname which can, e.g., be sent in a SNI is valid for a specific
certificate.

This PR furthermore modifies the ssl logs again, and adds information
about this to the log-file. Furthermore we now by default remove the
server certificate information from ssl.log - I doubt that this is often
looked at, it is not present in TLS 1.3, we do still have the SNI, and
if you need it you have the information in x509.log.

This also fixes a small potential problem in X509.cc assuming there
might be SAN-entries that contain null-bytes.

Baseline update will follow in another commit.
2021-06-29 15:00:48 +01:00
Tim Wojtulewicz
f45df63cd0 Merge remote-tracking branch 'origin/topic/vern/zval'
* origin/topic/vern/zval: (42 commits)
  whitespace tweaks
  resolved some TODO comments
  remove unnecessary casts, and change necessary ones to use static_cast<>
  explain cmp_func default
  change functions for ZVal type management to static members
  fix some unsigned/signed integer warnings
  address lint concern about uninitialized variable
  Remove use of obsolete forward-declaration macros
  fix #include's that lack zeek/ prefixes
  explicitly populate holes created in vectors
  fixes for now-incorrect assumption that GetField always returns an existing ValPtr
  memory management for assignment to vector elements
  memory management for assignment to record fields
  destructor cleanup from ZAM_vector/ZAM_record
  fix #include's that lack zeek/ prefixes
  overlooked another way in which vector holes can be created
  initialize vector holes to the correct corresponding type
  explicitly populate holes created in vectors
  fix other instances of GetField().get() assuming long-lived ValPtr's
  fix for now-incorrect assumption that GetField always returns an existing ValPtr
  ...
2021-03-23 20:44:19 -07:00
Vern Paxson
d5b523cd2c whitespace tweaks 2021-03-22 21:23:15 -07:00
Vern Paxson
245108e86e remove unnecessary casts, and change necessary ones to use static_cast<> 2021-03-18 13:24:25 -07:00
Jon Siwek
1c58a2d86b GH-1432: Use buffered IO for file extraction
This can improve performance significantly: ~3.5x faster when tested on
a large file passing data to the file analysis framework in small chunks
of 20 bytes.
2021-03-09 18:26:48 -08:00
Vern Paxson
62bab66114 migration to using new differentiated methods for setting record fields 2021-02-25 16:59:26 -08:00
Vern Paxson
0118b6ee38 migrate to differentiated vector "At" accessors to support future efficiency 2021-02-25 13:28:23 -08:00
Vern Paxson
6121e409d3 convert RecordVal's to use vector<ZVal> as internal representation 2021-02-24 17:02:47 -08:00
Tim Wojtulewicz
4ad08172d0 Remove obsolete ZEEK_FORWARD_DECLARE_NAMESPACED macros 2021-02-24 14:35:44 -07:00
Jon Siwek
69d73f7c83 Fix major_subsys_version field in pe_optional_header event
It was incorrectly set the same as the `minor_subsys_version` field
of the `PE::OptionalHeader` record.
2021-02-11 21:14:34 -08:00
Jon Siwek
c44cbe1feb Prefix #includes of .bif.h files with zeek/
This enables locating the headers within the install-tree using the
dirs provided by `zeek-config --include_dir`.

To enable locating these headers within the build-tree, this change also
creates a 'build/src/include/zeek -> ..' symlink.
2021-02-02 19:15:05 -08:00
Tim Wojtulewicz
0618be792f Remove all of the random single-file deprecations
These are the changes that don't require a ton of changes to other files outside
of the original removal.
2021-01-27 10:52:40 -07:00
Tim Wojtulewicz
5589484f26 Fix includes of bif.h and _pac.h files to use full paths inside build directory 2020-11-12 12:15:26 -07:00
Tim Wojtulewicz
96d9115360 GH-1079: Use full paths starting with zeek/ when including files 2020-11-12 12:15:26 -07:00
Jon Siwek
ff0aa6b050 Merge remote-tracking branch 'origin/topic/timw/rename-binpac_bro'
* origin/topic/timw/rename-binpac_bro:
  Rename bro-bif.h to zeek-bif.h
  Fix uses of bro.pac to use zeek.pac
  Rename the bro-related binpac files to zeek
2020-08-26 11:21:17 -07:00