Commit graph

9176 commits

Author SHA1 Message Date
Jon Siwek
ba929ce2c4 Add/use unspecified IPAddr constants
The analyzer-scheduling code was otherwise frequently converting the
unspecified v4/v6 addresses from strings.
2019-08-23 11:15:02 -04:00
Jon Siwek
a9f14bf503 GH-541: fix handling of NTLM AV Pair sequences
Empty AV Pair sequences or AV Pair sequences that lack a terminator
could cause accesses past the end of the parsed vector.
2019-08-23 07:22:18 -04:00
Jon Siwek
48873570b5 Updating submodule(s).
[nomail]
2019-08-23 06:37:27 -04:00
Jon Siwek
e7a2ee6edc Merge remote-tracking branch 'origin/topic/timw/mapping'
- Minor whitespace and comment adjustments

* origin/topic/timw/mapping:
  Fix unit tests for new ordering from NetSessions::Drain
  Change FragReassembler to use a tuple as a key and use std::map for fragments in Sessions
  Rework Session/Connection tracking to use a std::map instead of PDict
2019-08-23 06:28:44 -04:00
Jon Siwek
24a72a76ef Merge remote-tracking branch 'origin/topic/seth/github-linguist-fixes'
- Changed .bif files to be detected as C++

* origin/topic/seth/github-linguist-fixes:
  Adjusting how files are ignored for linguist and adding bif files to ignore.
  Stop incorrect language identification on github.
2019-08-23 05:45:34 -04:00
Jon Siwek
104633737d Merge remote-tracking branch 'origin/topic/seth/zeek-fix-thread-names'
* origin/topic/seth/zeek-fix-thread-names:
  Zeekify thread naming.
2019-08-23 05:40:09 -04:00
Seth Hall
162f4012b5 Adjusting how files are ignored for linguist and adding bif files to ignore. 2019-08-22 22:17:04 -04:00
Seth Hall
1a07df196e Zeekify thread naming.
I copied the same style that caf uses ("zk" with single dot and no space).
This gives some consistency with caf and avoids us wasting more
space beyond "bro: ".  OSs only give 16 characters for thread names
so anything we can gain here is nice.
2019-08-22 22:10:27 -04:00
Seth Hall
ac7e64bcad Stop incorrect language identification on github.
If you look at the languages that github is implemented in according
to our page on github, it has several incorrect things. PAC files and
SIG files being the big misidentifications. In my opinion it would look
better to mark these as no language so that javascript and ML stop showing
up as languages that Zeek is implemented in.

This change should make fix that on github according to:
	https://github.com/github/linguist#overrides
2019-08-22 22:00:38 -04:00
Jan Grashoefer
5901b6d33c Allow to handle late DPD matches.
If dpd_match_only_beginning is disabled, matches of protocol signatures
can be handeld using protocol_late_match. To prevent further matching in
this case, dpd_late_match_stop may be activated.
2019-08-22 19:00:11 +02:00
Tim Wojtulewicz
01207c0720 Fix unit tests for new ordering from NetSessions::Drain 2019-08-21 09:24:02 -04:00
Tim Wojtulewicz
a4b8aa1f30 Change FragReassembler to use a tuple as a key and use std::map for fragments in Sessions 2019-08-21 09:24:02 -04:00
Tim Wojtulewicz
57f29f3e7c Rework Session/Connection tracking to use a std::map instead of PDict 2019-08-21 09:24:02 -04:00
Jon Siwek
2d7c926291 GH-545: add "addl" parameter to flow_weird and net_weird events 2019-08-20 22:45:22 -04:00
Robin Sommer
8ab0650c1e Merge remote-tracking branch 'origin/topic/jsiwek/gh-532-improve-disable-analyzer'
Includes fix for potential iterator invalidation during iteration.

* origin/topic/jsiwek/gh-532-improve-disable-analyzer:
  GH-532: improve disable_analyzer BIF
2019-08-16 18:46:35 +00:00
Robin Sommer
f592ffe672 Updating submodule. 2019-08-16 18:01:59 +00:00
Jon Siwek
6255ab6584 Fix misc. Coverity warnings 2019-08-14 16:19:56 -07:00
Jon Siwek
47235b57a6 Merge remote-tracking branch 'origin/topic/timw/deprecate-int-types'
* origin/topic/timw/deprecate-int-types:
  Deprecate the internal int/uint types in favor of the cstdint types they were based on

Merge adjustments:
  * A bpf type mistakenly got replaced (inside an unlikely #ifdef)
  * Did a few substitutions that got missed (likely due to
    pre-processing out of DEBUG macros)
2019-08-14 15:49:24 -07:00
Jon Siwek
03c50fe520 Change file_analysis::Manager::ignored to use std::set 2019-08-14 13:48:39 -07:00
Jon Siwek
814d0ea969 Updating submodule(s).
[nomail]
2019-08-14 13:26:39 -07:00
Jon Siwek
396c1cb066 Update documentation for string_to_pattern BIF 2019-08-14 13:22:02 -07:00
Jon Siwek
6a3e53547c Updating submodule(s).
[nomail]
2019-08-14 10:18:34 -07:00
Jon Siwek
94c380297e Merge remote-tracking branch 'origin/topic/timw/easy-pdict-replacements'
* origin/topic/timw/easy-pdict-replacements:
  Cleanups related to PDict -> std::map replacements
  Remove other simple uses of PDict
  Protocols: Remove uses of PDict
  g_dbgfilemaps: Remove uses of PDict
  Scope: remove uses of PDict
  DFA: remove uses of PDict
  EventRegistry: remove uses of PDict
2019-08-13 20:16:02 -07:00
Jon Siwek
87f85ecca1 Cleanups related to PDict -> std::map replacements 2019-08-13 19:57:42 -07:00
Tim Wojtulewicz
e6558d1f19 Remove other simple uses of PDict 2019-08-13 19:57:42 -07:00
Tim Wojtulewicz
8beb710a5e Protocols: Remove uses of PDict 2019-08-13 19:57:42 -07:00
Tim Wojtulewicz
d0e8af9343 g_dbgfilemaps: Remove uses of PDict 2019-08-13 19:57:42 -07:00
Tim Wojtulewicz
a3deb0446c Scope: remove uses of PDict 2019-08-13 19:57:42 -07:00
Robin Sommer
00987c4411 Merge remote-tracking branch 'origin/topic/timw/clang-tidy'
* origin/topic/timw/clang-tidy:
  Change over to whitelisting clang-tidy options instead of blacklisting
  Use FindClangTidy in the cmake submodule to make things easier
2019-08-13 23:45:16 +00:00
Robin Sommer
4ee9276e3a Updating submodule(s).
[nomail]
2019-08-13 23:41:55 +00:00
Robin Sommer
ecd6139d09 Merge remote-tracking branch 'origin/topic/jsiwek/replace-pcap-lookupdev'
* origin/topic/jsiwek/replace-pcap-lookupdev:
  Replace use of deprecated pcap_lookupdev()
2019-08-13 23:02:43 +00:00
Robin Sommer
8167320bb7 Merge remote-tracking branch 'origin/topic/jsiwek/fatal-error-exit-behavior'
* origin/topic/jsiwek/fatal-error-exit-behavior:
  Use _exit() in Reporter::FatalError
2019-08-13 22:40:06 +00:00
Robin Sommer
59c306905c Merge remote-tracking branch 'origin/topic/jsiwek/gh-533-libdir'
* origin/topic/jsiwek/gh-533-libdir:
  GH-533: use consistent "lib" install dir
2019-08-13 22:39:13 +00:00
Jon Siwek
9ccf3549fd Merge remote-tracking branch 'origin/topic/timw/360-log-gzip'
* origin/topic/timw/360-log-gzip:
  Add new LogAscii::gzip_file_extension option.

Adjusted documentation during merge.
2019-08-13 13:49:45 -07:00
Jon Siwek
6e48bdeb38 Merge remote-tracking branch 'origin/topic/jazoff/memory-reduction-contentline'
* origin/topic/jazoff/memory-reduction-contentline:
  Remove redundant buffering in contentline

Adjusted NVT analyzer during merge for cases where it may have been
assuming the buffer was already resized before DoDeliver().
2019-08-13 12:04:15 -07:00
Jon Siwek
c91b12863e Add tests for {http,mime}_all_headers events
And remove unnecessary check for mime_all_headers in HTTP entities
(they ony raise the http_all_headers event, never mime_all_headers).
2019-08-13 11:28:00 -07:00
Jon Siwek
bc06768a67 Updating submodule(s).
[nomail]
2019-08-13 11:06:29 -07:00
Jon Siwek
7fc664b545 Merge remote-tracking branch 'origin/topic/jazoff/memory-reduction-mime'
* origin/topic/jazoff/memory-reduction-mime:
  Avoid buffering all http headers

Adjusted during merge:
  * whitespace
  * logic for HTTP needs to consider whether either mime_all_headers
    or http_all_headers events are being used
2019-08-12 19:34:42 -07:00
Jon Siwek
55fb48d04d GH-533: use consistent "lib" install dir
Previously, some sub-projects, like binpac, use GNUInstallDirs.cmake
to choose the default name of the library install directory while
others hard-code "lib" as the default.  The former may pick "lib64"
on some platforms, so for now, when installing such sub-projects as
part of Zeek, it's overridden to consistently be "lib".
2019-08-12 18:19:27 -07:00
Jon Siwek
41882263db Use _exit() in Reporter::FatalError
Using exit() here may generally not work well since:

* That will result in calling global destructors

* We have global state that we potentially modify at run-time and
  are in the middle of modiying at the time the FatalError occurs.
  E.g. out-of-memory is one situation where it's likely we could
  call the dtor of an object in which operation on it's internal
  state is no longer consistent/safe.
2019-08-12 14:13:46 -07:00
Tim Wojtulewicz
2e6e1aa5a8 Change over to whitelisting clang-tidy options instead of blacklisting 2019-08-12 13:59:17 -07:00
Tim Wojtulewicz
b42e726a8e Use FindClangTidy in the cmake submodule to make things easier 2019-08-12 13:59:17 -07:00
Tim Wojtulewicz
54752ef9a1 Deprecate the internal int/uint types in favor of the cstdint types they were based on 2019-08-12 13:50:07 -07:00
Jon Siwek
31d30bb47e GH-535: fix typo of "C_HESIOD" in DNS::classes
Fixes GH-535
2019-08-12 13:32:28 -07:00
Jon Siwek
18e4976c6c Merge remote-tracking branch 'origin/topic/timw/travis-leaks'
* origin/topic/timw/travis-leaks:
  fixup! Add new distro to Travis CI configuration for running leak tests
  Add new distro to Travis CI configuration for running leak tests
2019-08-12 13:01:55 -07:00
Jon Siwek
a1c5eddb95 GH-532: improve disable_analyzer BIF
- Add an extra "prevent" parameter (default value of false), which
  helps prevent the same analyzer type from being attached in the
  future.  It's useful in situations where you want to disable early
  on, but a DPD signature may still trigger later and re-attach
  the same analyzer.  E.g. when not using this flag, but calling
  disable_analyzer() inside an http_request event, will remove the
  HTTP analyzer that was attached due to well-known-port, but a later
  DPD signature match from upon seeing the HTTP reply will end up
  attaching another HTTP analyzer.  More surprising is that upon
  re-attaching that analyzer, you'll get the same http_request as
  before since the DPD buffer will get replayed into the new analyzer.

- Fixes disable_analyzer() to work when called even earlier, like
  within the protocol_confirmation event.  At that time, the
  Analyzer tree may have not properly added the new analyzer into
  Analyzer::children yet, but rather the temporary waiting list,
  Analyzer::new_children.  Analyzer::RemoveChildAnalyzer previously
  did not inspect the later list.

- Fixes disable_analyzer() when called on an analyzer added to the
  tree via TCP_Analyzer::AddChildPacketAnalyzer.  TCP_Analyzer
  keeps track of such children in its own list,
  TCP_Analyzer::packet_children, which the previous
  Analyzer::RemoveChildAnalyzer implementation didn't inspect.
2019-08-09 20:03:26 -07:00
Tim Wojtulewicz
b9f6992c0e fixup! Add new distro to Travis CI configuration for running leak tests 2019-08-09 11:01:16 -07:00
Jon Siwek
bf9b983f00 Merge branch 'known_services_multiprotocols' of https://github.com/mauropalumbo75/zeek
* 'known_services_multiprotocols' of https://github.com/mauropalumbo75/zeek:
  improve logging with broker store
  drop services starting with -
  remove service from key for Cluster::publish_hrw
  remove check for empty services
  update tests
  order list of services in store key
  remove repeated services in logs if already seen
  add multiprotocol known_services when Known::use_service_store = T
  remove hyphen in front of some services (for example -HTTP, -SSL)   In some cases, there is an hyphen before the protocol name in the field   connection$service. This can cause problems in known_services and   is removed here. It originates probably in some analyzer where it   would be better removed in the future.
  add multiprotocol known_services when Known::use_service_store = F

Changes during merge:
  * whitespace
  * add unit test
2019-08-09 10:47:34 -07:00
Jon Siwek
a68c9f6b71 Merge branch 'empty_services' of https://github.com/mauropalumbo75/zeek
* 'empty_services' of https://github.com/mauropalumbo75/zeek:
  remove empty services and include udp active connections when logging in connection_state_remove
2019-08-09 09:59:50 -07:00
Jon Siwek
acd1be6e7b Merge branch 'export_intel_events' of https://github.com/mauropalumbo75/zeek
* 'export_intel_events' of https://github.com/mauropalumbo75/zeek:
  minor restyle and add comments
  add an empty read_error event to the intel framework (in the export block, so that users can implement further checks with it)
  move event Intel::read_entry to export block

Adjusted whitespace in merge.
2019-08-09 09:36:19 -07:00