There was an alternate syntax to assign anonymous functions to record
fields that was never migrated to use the new lambda expression
machinery (and so didn't allow referencing variables in outer scope):
type myrec: record {
foo: function(a: string);
};
local o = "o";
local mr = myrec($foo(a: string) = { print a + o; });
* 'intrusive_ptr' of https://github.com/MaxKellermann/zeek: (32 commits)
Scope: store IntrusivePtr in `local`
Scope: pass IntrusivePtr to AddInit()
DNS_Mgr: use class IntrusivePtr
Scope: use class IntrusivePtr
Attr: use class IntrusivePtr
Expr: check_and_promote_expr() returns IntrusivePtr
Frame: use class IntrusivePtr
Val: RecordVal::LookupWithDefault() returns IntrusivePtr
Type: RecordType::FieldDefault() returns IntrusivePtr
Val: TableVal::Delete() returns IntrusivePtr
Type: base_type() returns IntrusivePtr
Type: init_type() returns IntrusivePtr
Type: merge_types() returns IntrusivePtr
Type: use class IntrusivePtr in VectorType
Type: use class IntrusivePtr in EnumType
Type: use class IntrusivePtr in FileType
Type: use class IntrusivePtr in TypeDecl
Type: make TypeDecl `final` and the dtor non-`virtual`
Type: use class IntrusivePtr in TypeType
Type: use class IntrusivePtr in FuncType
...
Only one instance of base_type() getting a NewRef instead of AdoptRef
fixed in merge. All other changes are superficial formatting and
factoring.
* 'leaks' of https://github.com/MaxKellermann/zeek: (22 commits)
Stmt: use class IntrusivePtr
Stmt: remove unused default constructors and `friend` declarations
Val: remove unimplemented prototype recover_val()
Val: cast_value_to_type() returns IntrusivePtr
Val: use IntrusivePtr in check_and_promote()
Val: use nullptr instead of 0
zeekygen: use class IntrusivePtr
ID: use class IntrusivePtr
Expr: use class IntrusivePtr
Var: copy Location to stack, to fix use-after-free crash bug
Scope: lookup_ID() and install_ID() return IntrusivePtr<ID>
Scope: delete duplicate locals
EventRegistry: automatically delete EventHandlers
main: destroy event_registry after iosource_mgr
zeekygen/IdentifierInfo: delete duplicate fields
main: free the global scope in terminate_bro()
Scope: pop_scope() returns IntrusivePtr<>
Scope: unref all inits in destructor
Var: pass IntrusivePtr to add_global(), add_local() etc.
plugin/ComponentManager: hold a reference to the EnumType
...
The AssignExpr constructor may free the "init" pointer via
AssignExpr::TypeCheck(), resulting in a crash due to use-after-free.
To fix the crash bug, this patch copies the `Location` to the stack
instead of using a potentially-dangling pointer.
Make sure unused scopes are freed to fix memory leaks.
The comment inside pop_scope() is now obsolete and I deleted it,
because this commit implements the real solution.
Note that this requires us to add a reference to the
push_existing_scope() call in dbg_eval_expr(), because it never owned
the reference.
Those functions don't have a well-defined reference passing API, and
we had lots of memory leaks here. By using IntrusivePtr, reference
ownership is well-defined.
The following source code:
function foo(foo: int) {}
function foo() {}
... first produces this error:
error in crash.zeek, line 1 and crash.zeek, line 2: incompatible types (function(foo:int;) : void and function() : void)
... and then crashes:
Thread 1 "zeek" received signal SIGSEGV, Segmentation fault.
0x000055555617d970 in RecordType::FieldDecl (this=0x555557cbdd80, field=0) at ../src/Type.cc:735
735 return (*types)[field];
(gdb) bt
#0 0x000055555617d970 in RecordType::FieldDecl (this=0x555557cbdd80, field=0) at ../src/Type.cc:735
#1 0x000055555619c0e2 in transfer_arg_defaults (args=0x555557cbf270, recv=0x555557cbdd80) at ../src/Var.cc:315
#2 0x000055555619c40c in begin_func (id=0x555557cbf070, module_name=0x5555579dd4a0 "GLOBAL", flavor=FUNC_FLAVOR_FUNCTION, is_redef=0, t=0x555557cbde00,
attrs=0x0) at ../src/Var.cc:371
#3 0x0000555555f5723b in yyparse () at parse.y:1174
#4 0x0000555556038bf6 in main (argc=5, argv=0x7fffffffe658) at ../src/main.cc:646
This is because `begin_func()` checks if the old and new functions
have the same type via same_type(), but continues anyway, and then
transfer_arg_defaults() crashes because both `Args()` have different
lengths.
- Various minor code formatting/styling during the merge
* 'leaks' of https://github.com/MaxKellermann/zeek:
parse.y: fix memory leak in FieldAssignExpr call
parse.y: fix use-after-free bug in open-ended index_slice
Type: fix use-after-free bug in init_type()
Expr: fix memory leak in RecordCoerceExpr::Fold()
Expr: fix memory leak in RecordCoerceExpr::InitVal()
zeekygen/IdentifierInfo: fix memory leak in operator=()
Func: fix memory leaks in get_func_priority()
parse.y: fix several memory leaks after lookup_ID()
Func: fix memory leaks in check_built_in_call()
Var: fix memory leaks in add_global() and add_local()
Var: add missing references to `init` in add{,_and_assign}_local()
parse.y: hold reference on init_expr for zeekygen::Manager::Redef()
Expr: fix two memory leaks in AssignExpr::InitVal()
parse.y: fix memory leak after "&derepcated" without string
RuleMatcher: delete PatternSet instances in destructor (memleak)
option.bif: fix crash bug by referencing `Func`, not `Val`
The Zeek code base has very inconsistent #includes. Many sources
included a few headers, and those headers included other headers, and
in the end, nearly everything is included everywhere, so missing
#includes were never noticed. Another side effect was a lot of header
bloat which slows down the build.
First step to fix it: in each source file, its own header should be
included first to verify that each header's includes are correct, and
none is missing.
After adding the missing #includes, I replaced lots of #includes
inside headers with class forward declarations. In most headers,
object pointers are never referenced, so declaring the function
prototypes with forward-declared classes is just fine.
This patch speeds up the build by 19%, because each compilation unit
gets smaller. Here are the "time" numbers for a fresh build (with a
warm page cache but without ccache):
Before this patch:
3144.94user 161.63system 3:02.87elapsed 1808%CPU (0avgtext+0avgdata 2168608maxresident)k
760inputs+12008400outputs (1511major+57747204minor)pagefaults 0swaps
After this patch:
2565.17user 141.83system 2:25.46elapsed 1860%CPU (0avgtext+0avgdata 1489076maxresident)k
72576inputs+9130920outputs (1667major+49400430minor)pagefaults 0swaps
For example, circular references between a lambda function the frame
it's stored within and/or its closure could cause memory leaks.
This also fixes other various reference-count ownership issues that
could lead to memory errors.
There may still be some potential/undiscovered issues because the "outer
ID" finding logic doesn't look quite right as the AST traversal descends
within nested lambdas and considers their locals as "outer", but
possibly the other logic for locating values in closures or cloning
closures just works around that behavior.
* topic/jsiwek/template-containers-merge:
Fix a potential usage of List::remove_nth(-1)
Change List::remote(const T&) to return a bool
Fix debug build due to old int_list usage within assert
Convert uses of loop_over_list to ranged-for loops
Remove loop_over_queue (as an example for later removing loop_over_list)
Change int_list in CCL.h to be a vector, fix uses of int_list to match
Remove List<> usage from strings.bif
Replace uses of the old Queue/PQueue generation code with new template versions
Convert BaseQueue/Queue/PQueue into templates, including iterator support
Replace uses of the old Dict generation code with new template versions
Convert PDict into template
Replace uses of the old List generation code with new template versions
Convert BaseList/List/PList into templates, including iterator support
* Generally squashed fixups from topic/timw/template-containers
* Add missing include file in List.h: <cassert>
This allows anonymous functions in Zeek to capture their closures.
they do so by creating a copy of their enclosing frame and joining
that with their own frame.
There is no way to specify what specific items to capture from the
closure like C++, nor is there a nonlocal keyword like Python.
Attemptying to declare a local variable that has already been caught
by the closure will error nicely. At the worst this is an inconvenience
for people who are using lambdas which use the same variable names
as their closures.
As a result of functions copying their enclosing frames there is no
way for a function with a closure to reach back up and modify the
state of the frame that it was created in. This lets functions that
generate functions work as expected. The function can reach back and
modify its copy of the frame that it is captured in though.
Implementation wise this is done by creating two new subclasses in
Zeek. The first is a LambdaExpression which can be thought of as a
function generator. It gathers all of the ingredients for a function
at parse time, and then when evaluated creats a new version of that
function with the frame it is being evaluated in as a closure. The
second subclass is a ClosureFrame. This acts for most intents and
purposes like a regular Frame, but it routes lookups of values to its
closure as needed.
This is needed to track name changes for the documentation.
With this things, which do not need val-cloning, generally seem to work
again. There are a whole bunch of test failures at the moment.
* origin/topic/robin/gh-239:
Undo a change to btest.cfg from a recent commit
Updating submodule.
Fix zeek-wrapper
Update for renaming BroControl to ZeekControl.
Updating submodule.
GH-239: Rename bro to zeek, bro-config to zeek-config, and bro-path-dev to zeek-path-dev.
Note - this compiles, but you cannot run Bro anymore - it crashes
immediately with a 0-pointer access. The reason behind it is that the
required clone functionality does not work anymore.
This also installs symlinks from "zeek" and "bro-config" to a wrapper
script that prints a deprecation warning.
The btests pass, but this is still WIP. broctl renaming is still
missing.
#239
Majority of PLists are now created as automatic/stack objects,
rather than on heap and initialized either with the known-capacity
reserved upfront or directly from an initializer_list (so there's no
wasted slack in the memory that gets allocated for lists containing
a fixed/known number of elements).
Added versions of the ConnectionEvent/QueueEvent methods that take
a val_list by value.
Added a move ctor/assign-operator to Plists to allow passing them
around without having to copy the underlying array of pointers.
