Merge adjustments:
- Preserved original `base_type_no_ref` argument type as ::TypeTag
- Removed superfluous #pragma guard around deprecated TableVal ctor
- Clarify NEWS regarding MetaHook{Pre,Post} deprecations
- Simplify some `::zeek::` qualifications to just `zeek::`
- Prefixed FORWARD_DECLARE_NAMESPACED macro with ZEEK_
* origin/topic/timw/266-namespaces:
Disable some deprecation diagnostics for GCC
Rename BroType to Type
Update NEWS
Review cleanup
Move Type types to zeek namespace
Move Flare/Pipe from the bro namespace to zeek::detail
Move Attr to the zeek::detail namespace
Move Trigger into the zeek::detail namespace
Move ID to the zeek::detail namespace
Move Anon.h into zeek::detail namespace
Mark all of the aliased classes in plugin/Plugin.h deprecated, and fix all of the plugins that were using them
Move all of the base plugin classes into the zeek::plugin namespace
Expr: move all classes into zeek::detail
Stmt: move Stmt classes into zeek::detail namespace
Add utility macro for creating namespaced aliases for classes
Now forward declares some Broker types since Broker/CAF headers
generally slow things down and also Coverity Scan currently has a
catastrophic error on some CAF headers.
Also a few other changes to EventHandler/BifReturnVal to reduce number
of places that depend on Func.h.
* Add deprecation for MIME_Entity::ContentType(), use GetContentType()
* Add deprecation for MIME_Entity::ContentSubType(), use GetContentSubType()
* Add deprecation for MIME_Message::BuildHeaderVal(), use ToHeaderVal()
* Add deprecation for MIME_Message::BuildHeaderTable(), use ToHeaderTable()
* Add deprecation for mime::new_string_val(), use mime::to_stringval()
* Add deprecation for ARP_Analyzer::ConstructAddrVal(), use ToAddrVal()
* Add deprecation for ARP_Analyzer::EthAddrToStr(), use ToEthAddrStr()
* Change the Func::Call() replacement to be named Func::Invoke()
This also changes the argument type of Func::operator() to zeek::Args*
to allow plugins to be able to alter function arguments in place as
was previously documented.
Names of functions also changed slightly, like bro_fmt -> fmt_bif.
Should generally be unusual/unexpected to see somone calling these
directly from C++ in their plugin, but since technically possible in
previous versions, I also removed the "private" restriction on accessing
the BifReturnVal member.
* origin/topic/timw/776-using-statements:
Remove 'using namespace std' from SerialTypes.h
Remove other using statements from headers
GH-776: Remove using statements added by PR 770
Includes small fixes in files that changed since the merge request was
made.
Also includes a few small indentation fixes.
* origin/topic/timw/nullptr:
The remaining nulls
plugin/probabilistic/zeekygen: Replace nulls with nullptr
file_analysis: Replace nulls with nullptr
analyzer: Replace nulls with nullptr
iosource/threading/input/logging: Replace nulls with nullptr
During merge, changed a bit of how Frame OffsetMap
assignments/contruction were handled to keep parity with old version.
* origin/topic/timw/structure-packing:
Lazy-initalize some of the fields in Frame to reduce the size of all Frames when they're not used
Set InternalHashTag to a uint16_t so CompositeHash doesn't have a gap in it.
Mark constants in List constexpr so they don't actually take up space in created objects
Reorder some class variables to fill in gaps in structure packing
The big hitters:
Dict: Fills in four 4-byte holes in the structure. This shrinks Dictionary from 136 bytes to 114 bytes.
Desc: Fills in a 6-byte hole in the structure. This shrinks ODesc from 152 bytes to 144 bytes.
Frame: Moves and combines 4 bool variables from a few places into one single 4-byte block. This resolves all of the holes at once. This shrinks Frame from 216 bytes to 192 bytes and removes one cache line.
Func: Moves one int32_t variable to fill in a 4-byte hole. This shrinks Func from 112 bytes to 104 bytes.
ID: Moves two bool variables to fill in a 3-byte hole. This leaves behind a 1-byte hole, but removes a 6-byte pad from the end of the structure. This shrinks ID from 144 bytes to 136 bytes.
Other changes:
RuleHdrTest: Fills in one 4-byte hole in the structure. This shrinks RuleHdrTest from 248 bytes to 240 bytes.
RuleEndpointState: Moves one bool variable down in the structure to reduce a 7-byte hole. This unfortunately causes a 3-byte hole later in the structure but there’s no easy way to filll it in. This does shrink RuleEndpointState from 128 bytes to 120 bytes though.
ScannedFile: Moves two bool values to reduce a 4-byte hole by 2 bytes. This shrinks ScannedFile from 64 bytes to 56 bytes.
Brofiler: Moves one char value to reduce a 4-byte hole by 1 byte. This shrinks Brofiler from 96 bytes to 88 bytes and removes one cache line.
DbgBreakpoint: Moves some values around to fill in a 4-byte hole and reduce a second. A 2-byte hole still exists, but the structure shrinks from 632 bytes to 624 bytes. It’s possible on this one that one of the int32_t values could be an int16_t and remove the last 2-byte gap.
ParseLocationRec: Moves one int to fill in a 4-byte hole. This shrinks ParseLocationRec from 32 bytes to 24 bytes.
DebugCmdInfo: Moves one bool variable to shift a few others up. This results in a 6-byte pad at the end of the structure but removes a 7-byte hole in the middle. This shrinks DebugCmdInfo from 56 bytes to 48 bytes.
FragReassembler: Moves one variable down to fill in a 4-byte hole. This shrinks FragReassembler from 272 bytes to 264 bytes.
nb_dns_result: Moves ones uint32_t variable to fill in a 4-byte hole, also removing a 4-byte pad from the end of the structure. This shrinks nb_dns_result from 32 bytes to 24 bytes.
nb_dns_entry: Moves one short value to fill in a 2-byte hole, also removing a 6-byte hole. This shrinks nb_dns_entry from 1064 bytes to 1056 bytes.
* 'intrusive_ptr' of https://github.com/MaxKellermann/zeek: (32 commits)
Scope: store IntrusivePtr in `local`
Scope: pass IntrusivePtr to AddInit()
DNS_Mgr: use class IntrusivePtr
Scope: use class IntrusivePtr
Attr: use class IntrusivePtr
Expr: check_and_promote_expr() returns IntrusivePtr
Frame: use class IntrusivePtr
Val: RecordVal::LookupWithDefault() returns IntrusivePtr
Type: RecordType::FieldDefault() returns IntrusivePtr
Val: TableVal::Delete() returns IntrusivePtr
Type: base_type() returns IntrusivePtr
Type: init_type() returns IntrusivePtr
Type: merge_types() returns IntrusivePtr
Type: use class IntrusivePtr in VectorType
Type: use class IntrusivePtr in EnumType
Type: use class IntrusivePtr in FileType
Type: use class IntrusivePtr in TypeDecl
Type: make TypeDecl `final` and the dtor non-`virtual`
Type: use class IntrusivePtr in TypeType
Type: use class IntrusivePtr in FuncType
...
The Zeek code base has very inconsistent #includes. Many sources
included a few headers, and those headers included other headers, and
in the end, nearly everything is included everywhere, so missing
#includes were never noticed. Another side effect was a lot of header
bloat which slows down the build.
First step to fix it: in each source file, its own header should be
included first to verify that each header's includes are correct, and
none is missing.
After adding the missing #includes, I replaced lots of #includes
inside headers with class forward declarations. In most headers,
object pointers are never referenced, so declaring the function
prototypes with forward-declared classes is just fine.
This patch speeds up the build by 19%, because each compilation unit
gets smaller. Here are the "time" numbers for a fresh build (with a
warm page cache but without ccache):
Before this patch:
3144.94user 161.63system 3:02.87elapsed 1808%CPU (0avgtext+0avgdata 2168608maxresident)k
760inputs+12008400outputs (1511major+57747204minor)pagefaults 0swaps
After this patch:
2565.17user 141.83system 2:25.46elapsed 1860%CPU (0avgtext+0avgdata 1489076maxresident)k
72576inputs+9130920outputs (1667major+49400430minor)pagefaults 0swaps
For example, circular references between a lambda function the frame
it's stored within and/or its closure could cause memory leaks.
This also fixes other various reference-count ownership issues that
could lead to memory errors.
There may still be some potential/undiscovered issues because the "outer
ID" finding logic doesn't look quite right as the AST traversal descends
within nested lambdas and considers their locals as "outer", but
possibly the other logic for locating values in closures or cloning
closures just works around that behavior.
anonymous-functions, their closures, can now be sent over broker.
In order to send an anonymous function the receiver must have parsed
a definition of the functon, but it need not to have been evaluated.
See testing/btest/language/closure-sending.zeek for an example of how
this can be done.
This also sends their closures as well as the closures of regular
functions.
This allows anonymous functions in Zeek to capture their closures.
they do so by creating a copy of their enclosing frame and joining
that with their own frame.
There is no way to specify what specific items to capture from the
closure like C++, nor is there a nonlocal keyword like Python.
Attemptying to declare a local variable that has already been caught
by the closure will error nicely. At the worst this is an inconvenience
for people who are using lambdas which use the same variable names
as their closures.
As a result of functions copying their enclosing frames there is no
way for a function with a closure to reach back up and modify the
state of the frame that it was created in. This lets functions that
generate functions work as expected. The function can reach back and
modify its copy of the frame that it is captured in though.
Implementation wise this is done by creating two new subclasses in
Zeek. The first is a LambdaExpression which can be thought of as a
function generator. It gathers all of the ingredients for a function
at parse time, and then when evaluated creats a new version of that
function with the frame it is being evaluated in as a closure. The
second subclass is a ClosureFrame. This acts for most intents and
purposes like a regular Frame, but it routes lookups of values to its
closure as needed.
