This largely copies over Spicy's `.clang-format` configuration file. The
one place where we deviate is header include order since Zeek depends on
headers being included in a certain order.
* origin/topic/timw/266-namespaces-part4: (34 commits)
Add missing namespace to usage of get_exe_path in fuzzer
Rename methods in RunState to remove 'net' from their names
Move zeek::net namespace to zeek::run_state namespace.
Move ScannedFile class and associated globals into ScannedFile.h and out of Net.h and scan.l
Rename types in ZeekList.h to be consistent with the style guide
Move NetVar from zeek to zeek::detail namespace
Remove PRI_PTR_COMPAT macros
Fix indentation of namespaced aliases
Move zeek-setup code into namespaces
Move ZeekList types to zeek namespace
Move __RegisterBif from zeek::detail::plugin to zeek::plugin::detail
Remove unimplemented zeek_magic_path/bro_magic_path method
Move all plugin classes into zeek::plugin::detail namespaces
Rename BroList.h to ZeekList.h
Move a few smaller files to zeek namespaces
Tag the end of some namespaces for consistency
Move a few of the zeek::util methods and variables to zeek::util::detail
Move zeekygen code to zeek::zeekygen::detail namespace
Mark zeek::util::pad_size as constexpr, which provides a small performance improvement
Move everything in util.h to zeek::util namespace.
...
The Zeek code base has very inconsistent #includes. Many sources
included a few headers, and those headers included other headers, and
in the end, nearly everything is included everywhere, so missing
#includes were never noticed. Another side effect was a lot of header
bloat which slows down the build.
First step to fix it: in each source file, its own header should be
included first to verify that each header's includes are correct, and
none is missing.
After adding the missing #includes, I replaced lots of #includes
inside headers with class forward declarations. In most headers,
object pointers are never referenced, so declaring the function
prototypes with forward-declared classes is just fine.
This patch speeds up the build by 19%, because each compilation unit
gets smaller. Here are the "time" numbers for a fresh build (with a
warm page cache but without ccache):
Before this patch:
3144.94user 161.63system 3:02.87elapsed 1808%CPU (0avgtext+0avgdata 2168608maxresident)k
760inputs+12008400outputs (1511major+57747204minor)pagefaults 0swaps
After this patch:
2565.17user 141.83system 2:25.46elapsed 1860%CPU (0avgtext+0avgdata 1489076maxresident)k
72576inputs+9130920outputs (1667major+49400430minor)pagefaults 0swaps
* origin/topic/jsiwek/reassembly-improvements-map:
Rename a reassembly DataBlockList function
Add comments to reassembly classes
Use DataBlock value instead of pointer in reassembly map
Remove linked list from reassembly data structures
Use an std::map for reassembly DataBlock searches
Refactor Reassembler/DataBlock bookkeeping
Reorganize reassembly data structures
Remove a superfluous reassembler DataBlock member
Started by factoring some details into a new DataBlockList class to at
least make it more clear where modifications occur. More abstractions
likely to happen later as I experiment with alternate data structures
aimed at improving worse-case scenarios.
Note - this compiles, but you cannot run Bro anymore - it crashes
immediately with a 0-pointer access. The reason behind it is that the
required clone functionality does not work anymore.
This commit marks (hopefully) ever one-parameter constructor as explicit.
It also uses override in (hopefully) all circumstances where a virtual
method is overridden.
There are a very few other minor changes - most of them were necessary
to get everything to compile (like one additional constructor). In one
case I changed an implicit operation to an explicit string conversion -
I think the automatically chosen conversion was much more convoluted.
This took longer than I want to admit but not as long as I feared :)
Includes small readability tweaks, see BIT-1854.
Closes BIT-1854.
* origin/topic/jsiwek/bit-1854-reassembler-improvements:
BIT-1854: improve reassembly overlap checking
BIT-1854: fix the 'tcp_excessive_data_without_further_acks' option
This previously checked against the amount of out-of-sequence data
being buffered by the reassembler. It now checks against the total
size of all blocks being buffered in the reassembler, which, by nature
of still being buffered there, means it's not been acked yet.
(Cleaned up some code a little bit.)
* origin/topic/seth/stats-improvement:
Fixing tests for stats improvements
Rename the reporting interval variable for stats.
Removing more broken functionality due to changed stats apis.
Removing some references to resource_usage()
Removing Broker stats, it was broken and incomplete.
Fixing default stats collection interval to every 5 minutes.
Add DNS stats to the stats.log
Small stats script tweaks and beginning broker stats.
Continued stats cleanup and extension.
More stats collection extensions.
More stats improvements
Slight change to Mach API for collecting memory usage.
Fixing some small mistakes.
Updating the cmake submodule for the stats updates.
Fix memory usage collection on Mac OS X.
Cleaned up stats collection.
BIT-1581 #merged
- Removed the gap_report event. It wasn't used anymore
and functionally no more capable that scheduling events
and using the get_gap_summary bif.
- Added functionality to Dictionaries to count cumulative
numbers of inserts performed. This is further used to
measure the total number of connections of various types.
Previously only the number of active connections was
available.
- The Reassembler base class now tracks active reassembly
size for all subclasses (File/TCP/Frag & unknown).
- Improvements to the stats.log. Mostly, more information.
TCP_Reassembler can now keep a history of old TCP segments using the
`tcp_max_old_segments` option. A value of zero will disable it.
An overlapping segment with different data can indicate a possible
TCP injection attack. The rexmit_inconsistency event will fire if this
is the case.
The main change is that reassembly code (e.g. for TCP) now uses
int64/uint64 (signedness is situational) data types in place of int
types in order to support delivering data to analyzers that pass 2GB
thresholds. There's also changes in logic that accompany the change in
data types, e.g. to fix TCP sequence space arithmetic inconsistencies.
Another significant change is in the Analyzer API: the *Packet and
*Undelivered methods now use a uint64 in place of an int for the
relative sequence space offset parameter.
- flow_weird event with name argument value of "routing0_hdr" is raised
for packets containing an IPv6 routing type 0 header because this
type of header is now deprecated according to RFC 5095.
- packets with a routing type 0 header and non-zero segments left
now use the last address in that header in order to associate
with a connection/flow and for calculating TCP/UDP checksums.
- added a set of IPv4/IPv6 TCP/UDP checksum unit tests
Internally, all BROv6 preprocessor switches were removed and
addr/subnet representations wrapped in the new IPAddr/IPPrefix classes.
Some script-layer changes of note:
- dns_AAAA_reply event signature changed: the string representation
of an IPv6 addr is easily derived from the addr value, it doesn't
need to be another parameter. This event also now generated directly
by the DNS analyzer instead of being "faked" into a dns_A_reply event.
- removed addr_to_count BIF. It used to return the host-order
count representation of IPv4 addresses only. To make it more
generic, we might later add a BIF to return a vector of counts
in order to support IPv6.
- changed the result of enclosing addr variables in vertical pipes
(e.g. |my_addr|) to return the bit-width of the address type which
is 128 for IPv6 and 32 for IPv4. It used to function the same
way as addr_to_count mentioned above.
- remove bro_has_ipv6 BIF
The TCP Reassembler does not deliver any data to analyzers after the
first 2GB due to signed integer overflow (Actually it will deliver again
between 4--6GB, etc.) This happens silently, i.e., without content_gap
events or Undelivered calls.
See Comments in TCP_Reassembler.cc for more details.
As a hotfix that seems to work I disabled the seq_to_skip features. It
wasn't used by any analyzer or policy script (Note, that seq_to_skip is
different from skip_deliveries).
See also ticket #348
