Commit graph

4151 commits

Author SHA1 Message Date
Vern Paxson
ab7cccf9b6 maintenance updates for -O ZAM 2022-12-04 17:58:33 -08:00
Vern Paxson
66c5bdbeac update test suite to avoid GH-2385 problems, including incorrect typing 2022-12-04 17:56:30 -08:00
Arne Welzel
203a309612 parse.y: Allow trailing commas for table, set, vector and record construction
Python, Ruby, Javascript, Go, ..., allow use of trailing commas and is even
recommended in some style-guides as it keeps diffs smaller. The black formatter
for Python even goes as far to take a trailing comma as an indication to
format a list one-item on a line.

It has been a bit unusual to not be able to put trailing commas in Zeek
scripts, so this change allows for it.

It explicitly prevents trailing commas in list expressions on the left
hand side. Concretely, this disallows trailing commas in the key list
expression during table initializations.

It probably allows for commas in more places that I haven't fully grasped.
Maybe we should tighten those down again if we find them surprising.
2022-12-01 12:05:29 +01:00
Johanna Amann
d38e227a04 Merge remote-tracking branch 'origin/topic/awelzel/2583-mqtt-cluster-testing-follow-up'
* origin/topic/awelzel/2583-mqtt-cluster-testing-follow-up:
  testing/external: Bump cluster testsuite
2022-11-30 15:41:50 +00:00
Arne Welzel
6a98386820 testing/external: Bump cluster testsuite
...also add/propose aliases for sync-commits / sync-repos, because I could not
remember for the past 5 times or so in which direction they would sync and
would find myself reading the README, Makefile, or comments in the scripts.
2022-11-30 15:10:14 +01:00
Johanna Amann
f4527ee4db Merge remote-tracking branch 'origin/topic/awelzel/2514-expire-all-timers-special-case'
* origin/topic/awelzel/2514-expire-all-timers-special-case:
  TimerMgr: Add back max_timer_expires=0 special case
  Add btest for expiration of all pending timers.
2022-11-30 13:45:09 +00:00
Johanna Amann
a7695bc784 Merge remote-tracking branch 'origin/topic/awelzel/2613-report-missing-intel-enum'
* origin/topic/awelzel/2613-report-missing-intel-enum:
  input: Add location info for invalid enums
2022-11-30 13:44:50 +00:00
Johanna Amann
3253168a53 Merge remote-tracking branch 'origin/topic/awelzel/2583-mqtt-to-base'
* origin/topic/awelzel/2583-mqtt-to-base:
  mqtt: Move from policy/ into base/
2022-11-30 13:44:27 +00:00
Johanna Amann
cb365d0ec5 Merge remote-tracking branch 'origin/topic/awelzel/2486-count-underflow'
* origin/topic/awelzel/2486-count-underflow:
  Expr: Warn on count underflow for c -= 1 and c = c - 1
  Reporter: Add ExprRuntimeWarning()
2022-11-30 13:43:37 +00:00
Arne Welzel
eb3bea4e4a mqtt: Move from policy/ into base/
Register dpd signatures and the analyzer when running in default mode.

Closes #2583
2022-11-30 10:14:20 +01:00
Arne Welzel
e48618e244 Expr: Warn on count underflow for c -= 1 and c = c - 1
I've skipped treating overflows as warnings, as ++ wrapping around at 0
doesn't currently trigger a runtime error and might be expected to be
quiet and silently wrap.

Closes #2486
2022-11-30 10:08:52 +01:00
Arne Welzel
a07b0c333f Reporter: Add ExprRuntimeWarning()
...and update Expr.cc invalidation messages to use it. This aligns the
warning format to the one used by runtime errors.
2022-11-29 15:28:58 +01:00
Arne Welzel
e5d9a715ce input: Add location info for invalid enums
Fixes #2613
2022-11-29 12:36:11 +01:00
Christian Kreibich
17039c1298 Merge remote-tracking branch 'origin/topic/vern/vec-construct-compat'
* origin/topic/vern/vec-construct-compat:
  provide deprecated version of merge_type_list()
  keep merge_types() externally available; address reviewing suggestion
  change vector constructors to require direct type equivalence for non-arithmetics
2022-11-28 17:43:19 -08:00
Jan Grashoefer
8cdc3e4374 Add btest for expiration of all pending timers. 2022-11-27 15:02:09 +01:00
Christian Kreibich
0a760c50b7 Merge remote-tracking branch 'origin/topic/vern/add-to-remove-from-coercions'
* origin/topic/vern/add-to-remove-from-coercions:
  Type coercion fix: transform +=/-= operators with arithmetic targets to explict assignments
2022-11-22 21:16:03 -08:00
Christian Kreibich
7c5f3c94b7 Merge remote-tracking branch 'origin/topic/awelzel/zeekygen-declaring-script'
* origin/topic/awelzel/zeekygen-declaring-script:
  zeekygen.bif: Drop using namespace zeekygen
  zeekygen: Add BIFs to access declaring scripts
2022-11-22 15:49:38 -08:00
Vern Paxson
baf2a91a85 Type coercion fix: transform +=/-= operators with arithmetic targets to explict assignments 2022-11-22 13:01:10 -08:00
Tim Wojtulewicz
9e8833e2d5 Merge remote-tracking branch 'security/topic/awelzel/121-ftp-timeout-again'
* security/topic/awelzel/121-ftp-timeout-again:
  ftp: Introduce FTP::max_command_length
2022-11-22 12:27:37 -07:00
Tim Wojtulewicz
eb3fb68fcc Merge remote-tracking branch 'security/topic/awelzel/119-http-timeout-tspacerr'
* security/topic/awelzel/119-http-timeout-tspacerr:
  http: Heuristic around rejecting malformed HTTP/0.9 traffic
2022-11-22 12:27:10 -07:00
Arne Welzel
ef166649bf zeekygen: Add BIFs to access declaring scripts
Two new BIFs get_identifier_declaring_script() and get_record_field_declaring_script()
mirroring existing functions for comments. This allows to query the declaring
script information from Zeek scripts and further determine if a redef operations
were involved for record fields or enum names by comparing the declaring script
with the one of the involved type. See the tests.

Yet another one of @stevesmoot's requests.
2022-11-22 13:23:01 +01:00
Vern Paxson
47152e38c4 change vector constructors to require direct type equivalence for non-arithmetics 2022-11-21 15:32:46 -08:00
Arne Welzel
3f5cb75a2a ftp: Introduce FTP::max_command_length
oss-fuzz produced FTP traffic with a ~550KB long FTP command. Cap FTP command
length at 100 bytes, log a weird if a command is larger than that and move
on to the next. Likely it's not actual FTP traffic, but raising an
analyzer violation would allow clients an easy way to disable the analyzer
by sending an overly long command.

The added test PCAP was generated using a fake Python socket server/client.
2022-11-21 09:36:29 +01:00
Arne Welzel
540fe7aff7 http: Heuristic around rejecting malformed HTTP/0.9 traffic
oss-fuzz generated "HTTP traffic" containing 250k+ sequences of "T<space>\r\r"
which Zeek then logged as individual HTTP requests. Add a heuristic to bail
on such request lines. It's a bit specific to the test case, but should work.

There are more issues around handling HTTP/0.9, e.g. triggering
"not a http reply line" when HTTP/0.9 never had such a thing, but
I don't think that's worth fixing up.

Fixes #119
2022-11-18 18:19:58 +01:00
Tim Wojtulewicz
1ed15fb807 Merge remote-tracking branch 'origin/topic/neverlord/web-socket-events-warning'
* origin/topic/neverlord/web-socket-events-warning:
  Fix warning in broker.web-socket-events btest
2022-11-18 09:41:18 -07:00
Tim Wojtulewicz
976caf61ba Merge remote-tracking branch 'jsoref/spelling-repetition'
* jsoref/spelling-repetition:
  spelling: repetitions
2022-11-18 09:40:43 -07:00
Josh Soref
186fe346ad spelling: repetitions
Signed-off-by: Josh Soref <2119212+jsoref@users.noreply.github.com>
2022-11-18 10:26:33 -05:00
Robin Sommer
d2585e21be
Merge remote-tracking branch 'origin/topic/robin/gh-2426-flipping'
* origin/topic/robin/gh-2426-flipping:
  Fixing productive connections with missing SYN still considered partial after flipping direction.
  Add some missing bits when flipping endpoints.
2022-11-18 11:50:28 +01:00
Dominik Charousset
1b1af51bdc Fix warning in broker.web-socket-events btest 2022-11-17 12:13:30 +01:00
Josh Soref
74af1ebe16 Spelling testing
* alphabet
* another
* associated
* avoiding
* base
* because
* constructors
* defining
* deterministic
* directly
* endlessly
* entity
* function
* indefinitely
* initial
* interpreter
* into
* modifying
* negotiate
* nonexistent
* observations
* occasional
* omission
* orphaned
* overridden
* passing
* primitive
* produces
* reassembly
* repository
* restore
* shouldn't
* something
* statement
* the
* therefore
* transferred
* uninitialized
* unsuccessful

Signed-off-by: Josh Soref <2119212+jsoref@users.noreply.github.com>
2022-11-16 20:05:03 -05:00
Robin Sommer
6fbebc5e94
Fixing productive connections with missing SYN still considered partial after flipping direction.
In https://github.com/zeek/zeek/pull/2191, we added endpoint flipping
for cases where a connection starts with a SYN/ACK followed by ACK or
data. The goal was to treat the connection as productive and go ahead
and parse it. But the TCP analyzer could continue to consider it
partial after flipping, meaning that app layers would bail out. #2426
shows such a case: HTTP gets correctly activated after flipping
through content inspection, but it won't process anything because
`IsPartial()` returns true. As the is-partial state reflects
whether we saw the first packets each in direction, this patch now
overrides that state for the originally missing SYN after flipping.

We actually had the same problem at a couple of other locations already
as well. One of that only happened to work because of the originally
inconsistent state flipping that was fixed in the previous commit. The
corresponding unit test now broke after that change. This commit
updates that logic as well to override the state.

This fix is a bit of a hack, but the best solution I could think of
without introducing larger changes.

Closes #2426.
2022-11-16 09:56:51 +01:00
Tim Wojtulewicz
bfd5b06943 Minor renaming changes to event handler stats bif, plus a test 2022-11-14 09:13:31 -07:00
Tim Wojtulewicz
2739275b88 Merge remote-tracking branch 'jsoref/spelling-src'
* jsoref/spelling-src:
  Spelling src
2022-11-11 12:49:15 -07:00
Tim Wojtulewicz
951250b753 Merge remote-tracking branch 'origin/topic/vern/script-opt-Nov22-maint'
* origin/topic/vern/script-opt-Nov22-maint:
  Script optimization maintenance and updates:   maintenance fixes for variadic run-time checks, '_' placeholder identifier   "-O allow-cond" permits compiling scripts to C++ when influenced by @if conditionals   more robust standalone compile-to-C++ properties   fix for nested "when" statements   test suite updates
2022-11-09 18:24:03 -07:00
Josh Soref
cd201aa24e Spelling src
These are non-functional changes.

* accounting
* activation
* actual
* added
* addresult
* aggregable
* aligned
* alternatively
* ambiguous
* analysis
* analyzer
* anticlimactic
* apparently
* application
* appropriate
* arithmetic
* assignment
* assigns
* associated
* authentication
* authoritative
* barrier
* boundary
* broccoli
* buffering
* caching
* called
* canonicalized
* capturing
* certificates
* ciphersuite
* columns
* communication
* comparison
* comparisons
* compilation
* component
* concatenating
* concatenation
* connection
* convenience
* correctly
* corresponding
* could
* counting
* data
* declared
* decryption
* defining
* dependent
* deprecated
* detached
* dictionary
* directional
* directly
* directory
* discarding
* disconnecting
* distinguishes
* documentation
* elsewhere
* emitted
* empty
* endianness
* endpoint
* enumerator
* essentially
* evaluated
* everything
* exactly
* execute
* explicit
* expressions
* facilitates
* fiddling
* filesystem
* flag
* flagged
* for
* fragments
* guarantee
* guaranteed
* happen
* happening
* hemisphere
* identifier
* identifies
* identify
* implementation
* implemented
* implementing
* including
* inconsistency
* indeterminate
* indices
* individual
* information
* initial
* initialization
* initialize
* initialized
* initializes
* instantiate
* instantiated
* instantiates
* interface
* internal
* interpreted
* interpreter
* into
* it
* iterators
* length
* likely
* log
* longer
* mainly
* mark
* maximum
* message
* minimum
* module
* must
* name
* namespace
* necessary
* nonexistent
* not
* notifications
* notifier
* number
* objects
* occurred
* operations
* original
* otherwise
* output
* overridden
* override
* overriding
* overwriting
* ownership
* parameters
* particular
* payload
* persistent
* potential
* precision
* preexisting
* preservation
* preserved
* primarily
* probably
* procedure
* proceed
* process
* processed
* processes
* processing
* propagate
* propagated
* prototype
* provides
* publishing
* purposes
* queue
* reached
* reason
* reassem
* reassemble
* reassembler
* recommend
* record
* reduction
* reference
* regularly
* representation
* request
* reserved
* retrieve
* returning
* separate
* should
* shouldn't
* significant
* signing
* simplified
* simultaneously
* single
* somebody
* sources
* specific
* specification
* specified
* specifies
* specify
* statement
* subdirectories
* succeeded
* successful
* successfully
* supplied
* synchronization
* tag
* temporarily
* terminating
* that
* the
* transmitted
* true
* truncated
* try
* understand
* unescaped
* unforwarding
* unknown
* unknowndata
* unspecified
* update
* usually
* which
* wildcard

Signed-off-by: Josh Soref <2119212+jsoref@users.noreply.github.com>
2022-11-09 12:08:15 -05:00
Tim Wojtulewicz
b9a4afe9bf Update zeek-testing-private commit hash 2022-11-08 16:53:58 -07:00
Arne Welzel
8b04868de3 {http,smtp}/entities: Align header regexes with extract_filename_from_content_disposition() 2022-11-08 16:45:25 -07:00
Arne Welzel
fa956efa79 file_analysis/Analyzer: Limit maximum number of violations
Just the same as for protocol analyzers, prevent from violation event overload.
2022-11-08 16:44:39 -07:00
Arne Welzel
e688bfcf73 test: Add btest verifying max_analyzer_violations functionality
The pcap has been generated roughly based on the example found on wikipedia
with some added garbled response after the STAT command from the client.
2022-11-08 16:44:34 -07:00
Arne Welzel
c132d140ae ftp: Limit pending commands to FTP::max_pending_commands (default 20) 2022-11-08 16:44:17 -07:00
Arne Welzel
d2689379bc Update baselines after AnalyzePacket changes
Mainly, for bad headers we do not (can not) put addresses into
the log anymore. Some were rather obviously bogus previously
already.
2022-11-08 16:44:15 -07:00
Vern Paxson
ee0a6f6835 Script optimization maintenance and updates:
maintenance fixes for variadic run-time checks, '_' placeholder identifier
  "-O allow-cond" permits compiling scripts to C++ when influenced by @if conditionals
  more robust standalone compile-to-C++ properties
  fix for nested "when" statements
  test suite updates
2022-11-08 11:56:32 -08:00
Robin Sommer
2a36441b8e
Merge remote-tracking branch 'origin/topic/awelzel/2483-builtin-plugin-includes'
* origin/topic/awelzel/2483-builtin-plugin-includes:
  zeek-setup: Load builtin-plugins/__preload__.zeek before initializing bifs
  cmake: Introduce bro_PLUGIN_LINK_LIBS and add to link command.
  cmake: Do not use include_directories() for included plugins
  cmake: Bump submodule
2022-11-07 11:25:03 +01:00
Tim Wojtulewicz
68450eac47 Merge remote-tracking branch 'origin/topic/timw/update-dns-types'
* origin/topic/timw/update-dns-types:
  Update external test hashes
  Update DNS type strings to match correct mappings
2022-11-03 08:57:19 -07:00
Tim Wojtulewicz
d65c75e2ad Merge remote-tracking branch 'jsoref/spelling-scripts'
* jsoref/spelling-scripts:
  Spelling fixes: scripts
2022-11-02 16:56:27 -07:00
Tim Wojtulewicz
66e818e69b Update external test hashes 2022-11-02 16:38:24 -07:00
Josh Soref
21e0d777b3 Spelling fixes: scripts
* accessing
* across
* adding
* additional
* addresses
* afterwards
* analyzer
* ancillary
* answer
* associated
* attempts
* because
* belonging
* buffer
* cleanup
* committed
* connects
* database
* destination
* destroy
* distinguished
* encoded
* entries
* entry
* hopefully
* image
* include
* incorrect
* information
* initial
* initiate
* interval
* into
* java
* negotiation
* nodes
* nonexistent
* ntlm
* occasional
* omitted
* otherwise
* ourselves
* paragraphs
* particular
* perform
* received
* receiver
* referring
* release
* repetitions
* request
* responded
* retrieval
* running
* search
* separate
* separator
* should
* synchronization
* target
* that
* the
* threshold
* timeout
* transaction
* transferred
* transmission
* triggered
* vetoes
* virtual

Signed-off-by: Josh Soref <2119212+jsoref@users.noreply.github.com>
2022-11-02 17:36:39 -04:00
Arne Welzel
d34167b2c4 zeek-setup: Load builtin-plugins/__preload__.zeek before initializing bifs
Prevent errors as follows with the bro-http2 plugin.

    error in /mitrecnd_HTTP2.events.bif.zeek, line 95: identifier not defined: http2_stream_stat
    error in /mitrecnd_HTTP2.events.bif.zeek, line 363: identifier not defined: http2_settings
    internal error in /mitrecnd_HTTP2.events.bif.zeek, line 460: Failed to find type named: http2_settings_unrecognized_table
2022-11-02 12:21:44 +01:00
Robin Sommer
f8eb2d9241
Merge remote-tracking branch 'origin/topic/awelzel/2440-break-next-usage-validation'
* origin/topic/awelzel/2440-break-next-usage-validation:
  parse.y: Traverse AST post parsing to detect break/next usage issues
2022-11-02 10:59:29 +01:00
Tim Wojtulewicz
bc0284aefa Merge remote-tracking branch 'origin/topic/awelzel/2446-runtime-error-for-variadic-functions'
* origin/topic/awelzel/2446-runtime-error-for-variadic-functions:
  Func: Do not crash on va_args confusion for script funcs
2022-11-01 07:47:50 -07:00