And switch Zeek's base scripts over to using it in place of
"connection_state_remove". The difference between the two is
that "connection_state_remove" is raised for all events while
"successful_connection_remove" excludes TCP connections that were never
established (just SYN packets). There can be performance benefits
to this change for some use-cases.
There's also a new event called ``connection_successful`` and a new
``connection`` record field named "successful" to help indicate this new
property of connections.
* origin/topic/seth/624:
Support whitespace at end of line for config reader.
This merge fixes a failing test; it also sprinkles a few more spaces
into another test file.
The main change is that this now also works with configuration lines
that don't have a value.
* origin/topic/timw/deprecate-int-types:
Deprecate the internal int/uint types in favor of the cstdint types they were based on
Merge adjustments:
* A bpf type mistakenly got replaced (inside an unlikely #ifdef)
* Did a few substitutions that got missed (likely due to
pre-processing out of DEBUG macros)
* 'known_services_multiprotocols' of https://github.com/mauropalumbo75/zeek:
improve logging with broker store
drop services starting with -
remove service from key for Cluster::publish_hrw
remove check for empty services
update tests
order list of services in store key
remove repeated services in logs if already seen
add multiprotocol known_services when Known::use_service_store = T
remove hyphen in front of some services (for example -HTTP, -SSL) In some cases, there is an hyphen before the protocol name in the field connection$service. This can cause problems in known_services and is removed here. It originates probably in some analyzer where it would be better removed in the future.
add multiprotocol known_services when Known::use_service_store = F
Changes during merge:
* whitespace
* add unit test
* origin/topic/seth/mqtt:
Bug fixes and test baseline updates
Fix an issue with bro_init -> zeek_init
MQTT Analyzer heavily updated and ported from the analyzer originally by Supriya Kumar
Adjustments during merge:
* Minor whitespace cleanups
* Some bro to zeek renaming
* Fixed the parsing of unsubscribe messages to generate an event for each topic
* origin/topic/dev/non-ascii-logging:
Removed Policy Script for UTF-8 Logs
Commented out UTF-8 Script in Test All Policy
Minor Style Tweak
Use getNumBytesForUTF8 method to determine number of bytes
Added Jon's test cases as unit tests
Prioritizes escaping predefined Escape Sequences over Unescaping UTF-8 Sequences
Added additional check to confirm anything unescaping is a multibyte UTF-8 sequence, addressing the test case Jon brought up
Added optional script and redef bool to enable utf-8 in ASCII logs
Initial Commit, removed std::isprint check to escape
Made minor code format and logic adjustments during merge.
* origin/topic/zeke/expire-func:
Ignore abs-path in test.
Report argument # type check failed on.
Update test baseline.
Improve func arg type checking.
&expire_func(table, arg1, arg2, ...) + type checking.
* topic/timw/algorithms:
Remove List::append deprecation
Mark List::append/insert deprecated in favor of push_back/push_front for consistency with Queue
Mark List::sort as deprecated, remove List::sortedinsert
Change container iterators to just use pointers directly into the container elements
* origin/topic/jsiwek/gh-475-no-default-tcmalloc:
GH-475: Require --enable-perftools to link in tcmalloc
I added an error message is --enable-perftools is given, but perftools
is not found - it does not seem great to not abort in these cases.
Before, Linux systems would automatically use tcmalloc if found.
Remove --disable-perftools since there's no longer any case where
it's used by default.
* topic/jsiwek/template-containers-merge:
Fix a potential usage of List::remove_nth(-1)
Change List::remote(const T&) to return a bool
Fix debug build due to old int_list usage within assert
Convert uses of loop_over_list to ranged-for loops
Remove loop_over_queue (as an example for later removing loop_over_list)
Change int_list in CCL.h to be a vector, fix uses of int_list to match
Remove List<> usage from strings.bif
Replace uses of the old Queue/PQueue generation code with new template versions
Convert BaseQueue/Queue/PQueue into templates, including iterator support
Replace uses of the old Dict generation code with new template versions
Convert PDict into template
Replace uses of the old List generation code with new template versions
Convert BaseList/List/PList into templates, including iterator support
* Generally squashed fixups from topic/timw/template-containers
* Add missing include file in List.h: <cassert>
* origin/topic/timw/150-to-json:
Update submodules for JSON work
Update unit tests for JSON logger to match new output
Modify JSON log writer to use the external JSON library
Update unit test output to match json.zeek being deprecated and slight format changes to JSON output
Add proper JSON serialization via C++, deprecate json.zeek
Add new method for escaping UTF8 strings for JSON output
Move do_sub method from zeek.bif to StringVal class method
Move record_fields method from zeek.bif to Val class method
Add ToStdString method for StringVal
- rotate_file
- rotate_file_by_name
- calc_next_rotate
These still have use-cases even though no longer used for our logging
functionality. E.g. rotate_file_by_name may be used to rotate
pcap dump files.
Also the log_rotate_base_time option was marked deprecated, but still
used in the new logging framework.
In the past they were processed on the manager - which requires big
records to be sent around.
This has a potential of incompatibilities if someone relied on global
state for notice processing.
GH-214
* 'paraglob' of https://github.com/ZekeMedley/zeek:
Add leak test to paraglob.
Catch paraglob serialization errors in DoClone.
Update paraglob serialization.
Stop execution on paraglob error.
Update paraglob submodule
Change C++11 detection in paraglob.
Make paraglob serializable and copyable.
Initial paraglob integration.
I made a bunch of small changes:
* paraglob now deals better with \0 characters
* I rolled back the changes to Binary Serialization format,
* there were some small formatting issue
* the error output was slightly unsafe
* build_unique is now in util.h.
and perhaps a few more small things.
* 'rdp_client_cluster_data' of https://github.com/neslog/zeek:
Fixing types.
Correcting types.
Removing misc data from Client Cluster data trying to assign values.
Adding options field to RDP::ClientChannelDef Adding Client Cluster Data
Adjustments:
- Reformatting
- Added comments
- Changed the REDIRECTED_SESSIONID_FIELD_VALID field to a bool
* origin/topic/johanna/remove-serializer:
Fix memory leak introduced by removing opaque of ocsp_resp.
Change return value of OpaqueVal::DoSerialize.
Add missing ShallowClone implementation for SetType
Remove opaque of ocsp_resp.
Remove remnants of event serializer.
Fix cardinalitycounter deserialization.
Smaller compile fixes for the new opaque serialization.
Reimplement serialization infrastructure for OpaqueVals.
Couple of compile fixes.
Remove const from ShallowClone.
Remove test-case for removed functionality
Implement a Shallow Clone operation for types.
Remove value serialization.
Various changes I made:
- Fix memory leak in type-checker for opaque vals wrapped in broker::data
- Noticed the two "copy-all" leak tests weren't actually checking for
memory leaks because the heap checker isn't active until after zeek_init()
is evaluated.
- Change OpaqueVal::DoClone to use the clone caching mechanism
- Improve copy elision for broker::expected return types in the various
OpaqueVal serialize methods
- Not all compilers end up properly treating the return of
local/automatic variable as an rvalue that can be moved, and ends up
copying it instead.
- Particularly, until GCC 8, this pattern ends up copying instead of
moving, and we still support platforms whose default compiler
pre-dates that version.
- Generally seems it's something that wasn't addressed until C++14.
See http://www.open-std.org/jtc1/sc22/wg21/docs/cwg_defects.html#1579
- Change OpaqueVal::SerializeType to return broker::expected
- Change probabilistic DoSerialize methods to return broker::expected
* origin/topic/timw/393-vector-slicing:
Fix memory leak in vector slice assignment
Misc. tweaks to vector slicing implementation
Add ability to grow/shrink a vector using slicing, also adds Insert/Remove methods for VectorVal
Allow assignment for vectors using slices
Check for integral slice indexes, add extra test for [:]
Return an empty vector if the indices for slicing don't make sense
GH-393: Add slice notation for vectors
* Minor style/format changes
* Fix a signed/unsigned comparison compiler warning
* Use a non-fatal error for non-integral slice indices so we can
report any further scripting errors instead of stopping the parse
right there
Only used in one event, without any way to use the opaque for anything
else. At this point this just seems like a complication that has no
reason to be there.
* 'ntp-rewrite' of https://github.com/mauropalumbo75/zeek: (25 commits)
update tests baseline
Apply requested changes: - file dpd.sig and TODO comments for signature protocol detection removed - missing doc field filled in events.bif - rename OpCode and ReqCode fields into op_code and req_code respectively - removed unnecessary child method in NTP.h/.cc - main.zeek and ntp-protocol.pac reformatted
minor changes in the documentation
fix some initializations
fix wrong assignment of control key_id/crypto_checksum
code clean up
add extension fields parsing
add extended mac field with 20 byte digest (+4 byte key id)
update tests and add a new one for key_id and mac
fix auth field (key_id and mac) in standard and control msg
remove old NTP record in init-bare.zeek
fix key_id and digest (WIP)
fix wrong Assign with reference_id
add tests for ntp protocol (finished)
add tests for ntp protocol (WIP)
fix problem with time vals
add ntp records to init-bare.zeek
update ntp analyzer to val_mgr
extend and refact script-side of NTP analyzer
extend and refactor several fields
...
