Commit graph

537 commits

Author SHA1 Message Date
Christian Kreibich
f97a33e14d Fix type clash fatal error with log filters that use $ext_func and $include/$exclude
The logging manager's Manager::TraverseRecord(), called when adding a
log filter to a stream, skipped any fields intoduced by a filter's
$ext_func when such fields weren't mentioned in a $include restriction
or mentioned in an $exclude restriction. This was inconsistent with
Manager::RecordToFilterVals, used when actually writing log entries,
which does include those values.

The result was that the record indices descent in Manager::RecordToFilterVals
expects to find only record values, when in fact only the record
provided by ext_func is present. This leads to type mismatches and
hard Zeek exits like this one:

1300475173.475401 fatal error in zeek/share/zeek//base/init-bare.zeek, line 4810: Val::CONVERTER (string/record) (zeek)

The fix makes ext_func's field additions decisive, meaning the
filter's include/exclude lists don't apply to it. If a user really
wants to override this, they can reset the filter's ext_func back to
our no-op default.

The included btest produces the above error when the fix is not present.
2020-11-10 18:13:16 -08:00
Christian Kreibich
c662237499 Update testing/btest/scripts/base/frameworks/logging/hooks.zeek
Co-authored-by: Robin Sommer <robin@corelight.com>
2020-09-30 13:10:35 -07:00
Christian Kreibich
3d2c3284c7 Btests for log filter policy hooks 2020-09-30 13:10:35 -07:00
Jon Siwek
01d83c3204 Avoid race condition in control/id_value btest 2020-09-22 14:11:17 -07:00
Jon Siwek
1046e45dba Improve notice/suppression-cluster btest to avoid race potential 2020-09-14 12:28:25 -07:00
Jon Siwek
dc9e175a61 Potentially improve intel/remove-item-cluster btest 2020-08-27 18:01:33 -07:00
Jon Siwek
99d9a3a48c Fix closing timestamp of rotated log files in supervised-cluster mode 2020-08-25 17:06:10 -07:00
Jon Siwek
8d3c07b62a Improve an Intel framework btest
Relying on non-atomic `cp` operation may not cooperate well with the
Input framework file re-reading logic.
2020-08-25 14:36:52 -07:00
Jon Siwek
038fb1c08e Improve stability of config/cluster_resend btest 2020-08-20 16:26:46 -07:00
Johanna Amann
7f267d3e87 Merge remote-tracking branch 'origin/topic/jazoff/gh-1105'
* origin/topic/jazoff/gh-1105:
  Accept bracketed ipv6 addresses
2020-08-12 20:04:37 +00:00
Johanna Amann
123e45ca4e Merge remote-tracking branch 'origin/topic/seth/config-reader-musl-support'
* origin/topic/seth/config-reader-musl-support:
  Fix config reader regular expression for MUSL.
2020-08-12 19:50:43 +00:00
Jon Siwek
ccf8e271a1 Adjust more btest timings 2020-08-11 11:26:22 -07:00
Jon Siwek
7967a5b0aa General btest cleanup
- Use `-b` most everywhere, it will save time.

- Start some intel tests upon the input file being fully read instead of
  at an arbitrary time.

- Improve termination condition for some sumstats/cluster tests.

- Filter uninteresting output from some supervisor tests.

- Test for `notice_policy.log` is no longer needed.
2020-08-11 11:26:22 -07:00
Jon Siwek
3ad306355f Improve Intel expire-item btest to be less time-sensitive 2020-08-11 11:26:21 -07:00
Jon Siwek
af2262ddff Improve btests with unstable table/set output ordering
Particularly, the final output order of a table/set is sensitive to
order of input/insertions and some tests were converting
std::unordered_{set,map} to Zeek table/set and iteration over those
standard containers may not always loop through elements in the same
order across all platforms.
2020-08-11 11:26:21 -07:00
Jon Siwek
48a243069c Adjust a few btests that were unstable due to time-sensitivity 2020-08-11 11:26:21 -07:00
Jon Siwek
b96b1593d8 Sort output of a few SumStats cluster tests
The order of $epoch_result() function calls among all keys within a
single epoch isn't consequential.
2020-08-11 11:26:21 -07:00
Jon Siwek
94aee910d5 Update btests/baselines for OpenDict compat
Haven't checked different build configurations yet, but all except
a few SumStats tests are stable for me now.  The external tests
are also completely failing, but haven't looked at those yet.
2020-08-11 11:26:21 -07:00
Tim Wojtulewicz
7496cf14c7 Extend the timeouts on a few intel tests, update baselines 2020-08-09 21:13:12 -07:00
Justin Azoff
211789bbb3 Accept bracketed ipv6 addresses 2020-08-05 17:18:36 -04:00
Seth Hall
7f4914c219 Fix config reader regular expression for MUSL.
It was not dealing with multiple spaces between the key and the value
with MUSL correctly. This change ensures that if a value exists, that it
begins and ends with a non-blank character.
2020-08-04 12:35:20 -04:00
Jon Siwek
c765fd20fd GH-1083: Fix Input Framework 'change' events for 'set' destinations 2020-07-23 23:29:09 -07:00
Jon Siwek
7669f560d1 Integrate Supervisor code review suggestions 2020-07-09 13:56:11 -07:00
Jon Siwek
a06ef66edc Add Log::rotation_format_func and Log::default_rotation_dir options
These may be redefined to customize log rotation path prefixes,
including use of a directory.  File extensions are still up to
individual log writers to add themselves during the actual rotation.

These new also allow for some simplication to the default
ASCII postprocessor function: it eliminates the need for it doing an
extra/awkward rename() operation that only changes the timestamp format.

This also teaches the supervisor framework to use these new options
to rotate ascii logs into a log-queue/ directory with a specific
file name format (intended for an external archiver process to
monitor separately).
2020-07-07 18:42:37 -07:00
Johanna Amann
3bce313b12 Switch file UID hashing from md5 to highwayhash.
This commit switches UID hashing from md5 to a highway hash. It also
moves the salt value out of the file plugin - and makes it
installation-specific instead - it is moved to the global namespace.

There now are digest hash functions to make "static"
installation-specific hashes that are stable over workers available to
everyone; hashes can be 64, 128 or 256 bits in size.

Due to the fact that we switch the file hashing algorithm, all file
hashes change.

The underlyigng algorithm that is used for hashing is highwayhash-128,
which is significantly faster than md5.
2020-04-30 10:20:09 -07:00
Jon Siwek
ce9183a2ed Fix Broker topics used to uniquely identify cluster nodes
Node-specific topic prefix subscriptions/publications now add a trailing
slash like "zeek/cluster/node/<name>/".  Without the trailing slash,
messages attempting to target "proxy-10" may also be sent to "proxy-1"
since subscription matching is prefix-based.
2020-04-10 14:36:00 -07:00
Jon Siwek
98d94ec785 Enable leak checks for btests that produce runtime exceptions
These were previously reporting leaks due to various allocations not
getting cleaned up during the stack unwind, but at the current state of
the transition toward IntrusivePtr usage, theses tests no longer leak.
2020-03-23 21:51:12 -07:00
Jon Siwek
e50a8848ae Improve an input framework test
A race condition could cause unstable output: if the thread reading the
file is fast, often you see both "pred" functions execute and then both
"line" events execute with both entries already in the table, but if the
thread reading the file is slow, you see pred, event, pred, event, with
only one entry available in the first event.
2020-02-07 20:03:10 -08:00
Jon Siwek
5b74bbf99a Fix a btest's race condition by handling Pcap::file_done 2020-02-06 17:50:17 -08:00
Jon Siwek
4fbcca04e8 Improve btest timeouts
* Generally increase timeouts for tests that have recent transient
  failures

* Change any test that relied on `btest-bg-wait -k` since that's never
  going to play with with CI systems.  Instead, we always need to have
  a well-defined termination condition in the test itself (and most
  already did, so didn't really need the `-k` flag anyway).
2020-02-06 17:50:17 -08:00
Johanna Amann
50ebbc90ad Resolve race condition in test 2020-02-05 09:34:29 -08:00
Johanna Amann
deaab42187 Change threading formatter errors to warnings.
This change standardizes threading formatter error handling and moves
the remaining error calls to be warnings instead.

This is in line with already existing code - in most cases warnings were
raised, only a few cases raised errors. These cases do not differ
significantly from other cases in which warnings are raised.

This also fixes GH-692, in which misformatted lines prevent future file
parsing.

This commit also moves the FailWarn method that is used by both the
config and the ascii reader up to the ReaderBackend. Furthermore it
makes the Warning method of ReaderBackend respect the warning
suppression that is introduced by the FailWarn method.
2020-02-04 16:31:29 -08:00
Tim Wojtulewicz
2dcc936787 Test changes caused by minor order-of-operation changes related to the new loop architecture 2020-01-31 10:13:09 -07:00
Jon Siwek
6218f99b96 Update timeouts for a btest 2020-01-09 19:05:23 -08:00
Jon Siwek
1f3ec434a8 Extend ASAN_OPTIONS in btests instead of overwrite 2020-01-08 21:04:18 -08:00
Jon Siwek
66ca20b737 Increase timeout for a few btests
Saw these start being sensitive running an "ASan -O0" build locally
2020-01-08 10:31:16 -08:00
Jon Siwek
0fe2a14d98 Disable LeakSanitizer for btests that have known leaks
E.g. ones that throw interpreter exceptions,  as those are currently
known to potentially cause leaks.  Fixing the underlying leaks involves
the larger task of more IntrusivePtr usage.

Reference cycles may also cause leaks.
2020-01-02 12:05:49 -08:00
Jon Siwek
cd9fec7bdb Add Cirrus CI config 2019-12-17 15:28:25 -08:00
Jon Siwek
30d0b21ecc Merge remote-tracking branch 'origin/topic/dev/print-to-log'
Adjustments during merge:

- kept the UNKNOWN Log::ID as placeholder value
- changed the coverage.find-bro-logs test to check for arbitrary $path
  field values instead of just string literals
- don't force EnumVal to unsigned integer since the relevant union member
  is the signed integer and added the relevant enum values/types to
  .bif files for easier access
- compare FILE* versus file name to check for stdout equality (don't
  think it matters much, just a bit more efficient)
- minor whitespace/style tweaks

* origin/topic/dev/print-to-log:
  Added a non boolean configuration and other changes as suggested by Jon
  Allow Print Statements to be redirected to a Log# This is a combination of 3 commits.
2019-12-02 13:47:09 -08:00
Dev Bali
4be6871df2 Added a non boolean configuration and other changes as suggested by Jon 2019-11-26 21:53:21 -08:00
Dev Bali
c9016f1397 Allow Print Statements to be redirected to a Log# This is a combination of 3 commits. 2019-11-13 19:59:42 -08:00
Jon Siwek
0c71715cf4 Adjust btests for OpenBSD portability 2019-11-08 17:32:44 -08:00
Johanna Amann
66effde97d Merge remote-tracking branch 'origin/topic/seth/624'
* origin/topic/seth/624:
  Support whitespace at end of line for config reader.

This merge fixes a failing test; it also sprinkles a few more spaces
into another test file.

The main change is that this now also works with configuration lines
that don't have a value.
2019-10-14 21:47:59 -07:00
Seth Hall
cd06cb24cc Support whitespace at end of line for config reader.
This commit addresses issue #624
2019-10-14 11:43:16 -04:00
Jon Siwek
39b06e3cee GH-591: allow Config::set_value() to use empty/unspecified table/sets 2019-10-02 22:13:43 -07:00
Jon Siwek
fd66e7b9f6 GH-591: fix reading set[enum] values from input files 2019-10-02 22:09:16 -07:00
Tim Wojtulewicz
d4c394b72a GH-606: Output nulls into json data if a field isn't set 2019-09-27 14:12:48 -07:00
Jon Siwek
2d7c926291 GH-545: add "addl" parameter to flow_weird and net_weird events 2019-08-20 22:45:22 -04:00
Tim Wojtulewicz
337da50da6 Add new LogAscii::gzip_file_extension option.
This can be used with the LogAscii::gzip_level option to set the file extension of log files when they are compressed at creation time.
2019-08-05 14:36:34 -07:00
Jon Siwek
d1770853b3 Merge remote-tracking branch 'origin/topic/dev/non-ascii-logging'
* origin/topic/dev/non-ascii-logging:
  Removed Policy Script for UTF-8 Logs
  Commented out UTF-8 Script in Test All Policy
  Minor Style Tweak
  Use getNumBytesForUTF8 method to determine number of bytes
  Added Jon's test cases as unit tests
  Prioritizes escaping predefined Escape Sequences over Unescaping UTF-8 Sequences
  Added additional check to confirm anything unescaping is a multibyte UTF-8 sequence, addressing the test case Jon brought up
  Added optional script and redef bool to enable utf-8 in ASCII logs
  Initial Commit, removed std::isprint check to escape

Made minor code format and logic adjustments during merge.
2019-07-30 19:42:34 -07:00