Mirror/zeek
1
0
Fork 0
mirror of https://github.com/zeek/zeek.git synced 2025-10-02 06:38:20 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 1

Merge remote-tracking branch 'origin/master' into topic/johanna/spicy-tls

Browse source 
* origin/master: (60 commits)
  Update gen-zam submodule [nomail] [skip ci]
  Update doc submodule [nomail] [skip ci]
  Remove unused wrapper packet analyzer
  Add DNS TKEY event
  ScriptOpt: Ensure global statements have non-null scope
  simpler and more robust identification of function parameters for AST profiling
  fixes to limit AST traversal in the face of recursive types
  address some script optimization compiler warnings under Linux
  fix for -O C++ construction of variable names that use multiple module namespaces
  fix for script optimization of "opaque" values that are run-time constants
  fix for script optimization of nested switch statements
  script optimization fix for complex "in" expressions in conditionals
  updates to typos allow-list reflecting ZAM regularization changes
  BTest updates for ZAM regularization changes
  convert new ZAM operations to use typed operands
  complete migration of ZAM to use only public ZVal methods
  "-O validate-ZAM" option to validate generated ZAM instructions
  internal option to suppress control-flow optimization
  exposing some functionality for greater flexibility in structuring run-time execution
  rework ZAM compilation of type switches to leverage value switches
  ...
This commit is contained in:
Johanna Amann 2024-08-20 13:01:31 +01:00
commit 71d2e8d961
141 changed files with 5519 additions and 4348 deletions
Download patch file Download diff file Expand all files Collapse all files

9
.typos.toml
View file

@ -9,6 +9,7 @@ extend-ignore-re = [
"Remove in v6.1.*SupressWeird",
"max_repititions:.*Remove in v6.1",
"mis-aliasing of",
"mis-indexing",
# On purpose
"\"THE NETBIOS NAM\"",
# NFS stuff.
@ -20,6 +21,12 @@ extend-ignore-re = [
"ot->Tag\\(\\) == TYPE_.*",
"auto.* ot =",
"ot = OP_.*",
"ot\\[",
"ot.size",
"ot.empty",
"ot_i",
"ot.c_str",
"have_ot",
"if \\( ot == OP_.*",
"ot->Yield\\(\\)->InternalType\\(\\)",
"switch \\( ot \\)",
@ -53,7 +60,7 @@ ND_REDIRECT = "ND_REDIRECT"
NED_ACK = "NED_ACK"
NFS3ERR_ACCES = "NFS3ERR_ACCES"
NO_SEH = "NO_SEH"
OP_SWITCHS_VVV = "OP_SWITCHS_VVV"
OP_SWITCHS_Vii = "OP_SWITCHS_Vii"
O_WRONLY = "O_WRONLY"
RPC_NT_CALL_FAILED_DNE = "RPC_NT_CALL_FAILED_DNE"
RpcAddPrintProvidor = "RpcAddPrintProvidor"

174
CHANGES
View file

@ -1,3 +1,177 @@
7.1.0-dev.209 | 2024-08-20 10:10:46 +0200
* broker: Deprecate MakeEvent(ValPList*) (Arne Welzel, Corelight)
The variadic broker messaging BIFs currently convert @ARGS@ into a
ValPList before passing it on to MakeEvent(). This appears historic
plumbing. Implement the same functionality using Span<const ValPtr>
and do the extra copying in the now deprecated MakeEvent().
Further, make passing a frame optional as not all callers may
have one available.
* Span: Remove deduction guideline for Iter, Iter, include cleanup (Arne Welzel, Corelight)
We don't have a constructor for that, so that's confusing, also
<array> isn't used.
* ScriptOpt: Ensure global statements have non-null scope (Arne Welzel, Corelight)
The ProfileFunc() logic assumed that GetScope() returned a non-nullptr.
This holds except for the synthetic global statements function.
Fix the latter and add an assert, also add a name to the type so it's
easier to recognize in a debugger what's going on, otherwise the name
is "".
This was found by UBSAN due to it seeing the ->OrderedVars() call on a
nullptr. Elsewhere, num_params == 0 shielded from that access and so
didn't lead to crashes.
* Update gen-zam submodule [nomail] [skip ci] (Tim Wojtulewicz, Corelight)
7.1.0-dev.202 | 2024-08-16 09:06:41 -0700
* Remove unused wrapper packet analyzer (Jan Grashoefer, Corelight)
This is a leftover from the migration to the packet analysis framework.
The analyzer wrapped the original packet analysis code for comparison.
7.1.0-dev.200 | 2024-08-16 16:50:47 +0200
* Add DNS TKEY event (Evan Typanski, Corelight)
7.1.0-dev.198 | 2024-08-16 12:10:00 +0200
* simpler and more robust identification of function parameters for AST profiling (Vern Paxson, Corelight)
* fixes to limit AST traversal in the face of recursive types (Vern Paxson, Corelight)
* address some script optimization compiler warnings under Linux (Vern Paxson, Corelight)
* fix for -O C++ construction of variable names that use multiple module namespaces (Vern Paxson, Corelight)
* fix for script optimization of "opaque" values that are run-time constants (Vern Paxson, Corelight)
* fix for script optimization of nested switch statements (Vern Paxson, Corelight)
* script optimization fix for complex "in" expressions in conditionals (Vern Paxson, Corelight)
* updates to typos allow-list reflecting ZAM regularization changes (Vern Paxson, Corelight)
* BTest updates for ZAM regularization changes (Vern Paxson, Corelight)
* convert new ZAM operations to use typed operands (Vern Paxson, Corelight)
* complete migration of ZAM to use only public ZVal methods (Vern Paxson, Corelight)
* "-O validate-ZAM" option to validate generated ZAM instructions (Vern Paxson, Corelight)
* internal option to suppress control-flow optimization (Vern Paxson, Corelight)
* exposing some functionality for greater flexibility in structuring run-time execution (Vern Paxson, Corelight)
* rework ZAM compilation of type switches to leverage value switches (Vern Paxson, Corelight)
* add tracking of control flow information (Vern Paxson, Corelight)
* factoring of ZAM operation specifications into separate files (Vern Paxson, Corelight)
* updates to ZAM operations / gen-zam regularization, other than the operations themselves (Vern Paxson, Corelight)
* type-checking fix for vector-of-string operations (Vern Paxson, Corelight)
* ZVal constructor for booleans (Vern Paxson, Corelight)
* fix for nit in base/protocols/krb/main.zeek (Vern Paxson, Corelight)
* mark functions skipped by ZAM compilation as such (Vern Paxson, Corelight)
* fix for avoiding inadvertent interpreter errors in CallExpr::IsPure() (Vern Paxson, Corelight)
* support for traversing ZAM code similar to AST traversal (Vern Paxson, Corelight)
* run-time warnings for scripts compiled to C++ (Vern Paxson, Corelight)
* allow C++ script compiler access to type internals (Vern Paxson, Corelight)
* fixes for script optimization of coerce-to-any expressions (Vern Paxson, Corelight)
* fix to correctly track whether a capture needs deep-copying (Vern Paxson, Corelight)
* fix for -O report-C++ (Vern Paxson, Corelight)
* support for more in-depth AST profiling (Vern Paxson, Corelight)
* allow profiling without updating of hash values (Vern Paxson, Corelight)
* ListVal method to clear the list to allow reusing w/o new construction (Vern Paxson, Corelight)
* accessor for smart-pointer version of FileVal's value (Vern Paxson, Corelight)
7.1.0-dev.164 | 2024-08-15 10:30:37 +0200
* Func: Add SetCapturesVec() (Arne Welzel, Corelight)
Add an API to directly set captures_vec for use by C++ compilation. The
current code keys off or asserts on ZAM stmts, making it difficult to
leverage captures_vec in other contexts.
* marked some recently added BTests as not suitable for -O gen-C++ testing (Vern Paxson, Corelight)
* robustness improvements for -O gen-C++ generation of lambdas / "when"s (Vern Paxson, Corelight)
* speedups for compilation of initializers in -O gen-C++ generated code (Vern Paxson, Corelight)
* fixes for -O gen-C++ generation of floating point constants (Vern Paxson, Corelight)
* -O gen-C++ fix for dealing with use of more than one module qualifier (Vern Paxson, Corelight)
* header tweaks to provide gen-C++ script optimization with more flexibility (Vern Paxson, Corelight)
* fix for script optimization of constants of type "opaque" (Vern Paxson, Corelight)
* fix for script optimization of "in" operations (Vern Paxson, Corelight)
* some minor tidying of -O gen-C++ sources (Vern Paxson, Corelight)
7.1.0-dev.152 | 2024-08-14 20:08:14 +0200
* mysql: Implement and test COM_CHANGE_USER (Arne Welzel, Corelight)
This reworks the parser such that COM_CHANGE_USER switches the
connection back into the CONNECTION_PHASE so that we can remove the
EXPECT_AUTH_SWITCH special case in the COMMAND_PHASE. Adds two pcaps
produced with Python that actually do COM_CHANGE_USER as it seems
not possible from the MySQL CLI.
* mysql: Remove auth plugin state (Arne Welzel, Corelight)
Now that we use CONNECTION_PHASE and interpret AuthSwitchRequest
correctly, we can remove reliance on the plugin discrepancy to
determine expectations.
* GH-3880: mysql: Handle server connection phase separately from command phase (Arne Welzel, Corelight)
This avoids interpreting an AuthSwitchRequest (0xfe) during the command
phase as EOF_Packet.
Thanks @AmazingPP.
Closes #3880
7.1.0-dev.147 | 2024-08-13 12:37:17 +0200
* Deprecate old unknown_protocol event signature (Jan Grashoefer, Corelight)
* Use raw pointer for packet analyzer history (Jan Grashoefer, Corelight)
* Update external baselines for analyzer history (Jan Grashoefer, Corelight)
* Update btest baselines for analyzer history (Jan Grashoefer, Corelight)
* Add packet analyzer history (Jan Grashoefer, Corelight)
7.1.0-dev.141 | 2024-08-12 11:07:32 +0200
* spicyz: Add back message about removed support for port / ports in evt (Arne Welzel, Corelight)

13
NEWS
View file

@ -25,6 +25,16 @@ New Functionality
for connections where client and server negotiate to TLS through the extended
request/response mechanism.
* The ``unknown_protocols()`` event now includes the name of all packet
analyzer used for processing the packet when the event is raised. The
``unknown_protocol.log`` file was extended to include this information.
* The MySQL analyzer now generates a ``mysql_user_change()`` event when
the user changes mid-session via the ``COM_USER_CHANGE`` command.
* The DNS analyzer was extended to support TKEY RRs (RFC 2390). A corresponding
``dns_TKEY`` event was added.
Changed Functionality
---------------------
@ -36,6 +46,9 @@ Changed Functionality
mechanisms, like caching_sha2_password, as well as recognizing MySQL query
attributes.
* The ``mysql.log`` for user change commands will contain *just* the username
instead of the remaining parts of the command, including auth plugin data.
Removed Functionality
---------------------

2
VERSION
View file

@ -1 +1 @@
7.1.0-dev.141
7.1.0-dev.209

2
auxil/gen-zam

@ -1 +1 @@
Subproject commit 610cf8527dad7033b971595a1d556c2c95294f2b
Subproject commit 3c17f9797ebab5157d7b08265540cd7f4de15d1f

2
doc

@ -1 +1 @@
Subproject commit f450f803d3e69cb2fd474a919b7a6c6885f1f433
Subproject commit 2f14aee921e8f7df417423079f9341e1e72c5d7e

15
scripts/base/init-bare.zeek
View file

@ -4083,6 +4083,21 @@ type dns_edns_cookie: record {
server_cookie: string &default=""; ##< Cookie from the server (0 bytes if missing, or 8 to 32 bytes).
};
## A DNS TKEY record.
##
## .. zeek:see:: dns_TKEY
type dns_tkey: record {
query: string; ##< Query.
qtype: count; ##< Query type.
alg_name: string; ##< Algorithm name.
inception: time; ##< Requested or provided start of validity interval for keying material.
expiration: time; ##< Requested or provided end of validity interval for keying material.
mode: count; ##< Key agreement or purpose of the message.
rr_error: count; ##< Error code.
key_data: string; ##< Key exchange data field.
is_query: count; ##< The RR is a query/Response.
};
## An additional DNS TSIG record.
##
## .. zeek:see:: dns_TSIG_addl

3
scripts/base/protocols/krb/main.zeek
View file

@ -190,8 +190,7 @@ event krb_as_response(c: connection, msg: KDC_Response) &priority=-5
event krb_ap_request(c: connection, ticket: KRB::Ticket, opts: KRB::AP_Options) &priority=5
{
if ( set_session(c) )
return;
set_session(c);
}
event krb_tgs_request(c: connection, msg: KDC_Request) &priority=5

5
scripts/base/protocols/mysql/main.zeek
View file

@ -84,6 +84,11 @@ event mysql_command_request(c: connection, command: count, arg: string) &priorit
Conn::register_removal_hook(c, finalize_mysql);
}
event mysql_change_user(c: connection, username: string) &priority=5
{
c$mysql$arg = username;
}
event mysql_command_request(c: connection, command: count, arg: string) &priority=-5
{
if ( c?$mysql && c$mysql?$cmd && c$mysql$cmd == "quit" )

9
scripts/policy/misc/unknown-protocols.zeek
View file

@ -25,16 +25,23 @@ export {
## A certain number of bytes at the start of the unknown protocol's
## header.
first_bytes: string &log;
## The chain of packet analyzers that processed the packet up to this
## point. This includes the history of encapsulating packets in case
## of tunneling.
analyzer_history: vector of string &log;
};
}
event unknown_protocol(analyzer_name: string, protocol: count, first_bytes: string)
event unknown_protocol(analyzer_name: string, protocol: count, first_bytes: string,
analyzer_history: string_vec)
{
local info : Info;
info$ts = network_time();
info$analyzer = analyzer_name;
info$protocol_id = fmt("0x%x", protocol);
info$first_bytes = bytestring_to_hexstr(first_bytes);
info$analyzer_history = analyzer_history;
Log::write(LOG, info);
}

19
src/CMakeLists.txt
View file

@ -162,7 +162,23 @@ list(APPEND BINPAC_OUTPUTS "${BINPAC_OUTPUT_CC}")
include(Gen-ZAM)
set(GEN_ZAM_SRC_DIR ${CMAKE_CURRENT_SOURCE_DIR}/script_opt/ZAM/OPs)
set(GEN_ZAM_SRC ${GEN_ZAM_SRC_DIR}/ZAM.op)
set(ZAM_OP_SRCS
${GEN_ZAM_SRC_DIR}/aggr-assignments.op
${GEN_ZAM_SRC_DIR}/binary-exprs.op
${GEN_ZAM_SRC_DIR}/calls.op
${GEN_ZAM_SRC_DIR}/coercions.op
${GEN_ZAM_SRC_DIR}/constructors.op
${GEN_ZAM_SRC_DIR}/indexing.op
${GEN_ZAM_SRC_DIR}/internal.op
${GEN_ZAM_SRC_DIR}/iterations.op
${GEN_ZAM_SRC_DIR}/macros.op
${GEN_ZAM_SRC_DIR}/non-uniform.op
${GEN_ZAM_SRC_DIR}/rel-exprs.op
${GEN_ZAM_SRC_DIR}/script-idioms.op
${GEN_ZAM_SRC_DIR}/stmts.op
${GEN_ZAM_SRC_DIR}/unary-exprs.op
${GEN_ZAM_SRC_DIR}/ZBI.op)
set(GEN_ZAM_SRC ${ZAM_OP_SRCS})
gen_zam_target(${GEN_ZAM_SRC_DIR})
@ -430,6 +446,7 @@ set(MAIN_SRCS
script_opt/ZAM/Profile.cc
script_opt/ZAM/Stmt.cc
script_opt/ZAM/Support.cc
script_opt/ZAM/Validate.cc
script_opt/ZAM/Vars.cc
script_opt/ZAM/ZBody.cc
script_opt/ZAM/ZInst.cc

14
src/Expr.cc
View file

@ -1985,8 +1985,10 @@ EqExpr::EqExpr(ExprTag arg_tag, ExprPtr arg_op1, ExprPtr arg_op2)
}
}
else if ( bt1 == TYPE_PATTERN && bt2 == TYPE_STRING )
;
else if ( (bt1 == TYPE_PATTERN && bt2 == TYPE_STRING) || (bt1 == TYPE_STRING && bt2 == TYPE_PATTERN) ) {
if ( op1->GetType()->Tag() == TYPE_VECTOR )
ExprError("cannot compare string vectors with pattern vectors");
}
else
ExprError("type clash in comparison");
@ -4063,11 +4065,15 @@ bool CallExpr::IsPure() const {
if ( IsError() )
return true;
if ( ! func->IsPure() )
if ( func->Tag() != EXPR_NAME )
// Indirect call, can't resolve up front.
return false;
auto func_val = func->Eval(nullptr);
auto func_id = func->AsNameExpr()->Id();
if ( ! func_id->IsGlobal() )
return false;
auto func_val = func_id->GetVal();
if ( ! func_val )
return false;

14
src/Expr.h
View file

@ -474,7 +474,16 @@ public:
// Optimization-related:
ExprPtr Duplicate() override;
ValPtr FoldVal() const override { return val; }
ValPtr FoldVal() const override {
if ( type->Tag() == TYPE_OPAQUE )
// Aggressive constant propagation can lead to the appearance of
// opaque "constants". Don't consider these as foldable because
// they're problematic to generate independently.
return nullptr;
return val;
}
protected:
void ExprDescribe(ODesc* d) const override;
@ -1642,6 +1651,9 @@ class CoerceToAnyExpr : public UnaryExpr {
public:
CoerceToAnyExpr(ExprPtr op);
bool IsReduced(Reducer* c) const override;
ExprPtr Reduce(Reducer* c, StmtPtr& red_stmt) override;
protected:
ValPtr Fold(Val* v) const override;

15
src/Func.h
View file

@ -211,6 +211,15 @@ public:
return *captures_vec;
}
/**
* Set the set of ZVal's used for captures.
*
* Used for script optimization purposes.
*
* @param cv The value used for captures_vec.
*/
void SetCapturesVec(std::unique_ptr<std::vector<ZVal>> cv) { captures_vec = std::move(cv); }
// Same definition as in Frame.h.
using OffsetMap = std::unordered_map<std::string, int>;
@ -291,6 +300,9 @@ protected:
*/
virtual void SetCaptures(Frame* f);
// Captures when using ZVal block instead of a Frame.
std::unique_ptr<std::vector<ZVal>> captures_vec;
private:
size_t frame_size = 0;
@ -304,9 +316,6 @@ private:
OffsetMap* captures_offset_mapping = nullptr;
// Captures when using ZVal block instead of a Frame.
std::unique_ptr<std::vector<ZVal>> captures_vec;
// The most recently added/updated body ...
StmtPtr current_body;

11
src/ID.cc
View file

@ -100,20 +100,15 @@ ID::ID(const char* arg_name, IDScope arg_scope, bool arg_is_export) {
name = util::copy_string(arg_name);
scope = arg_scope;
is_export = arg_is_export;
is_option = false;
is_blank = name && extract_var_name(name) == "_";
is_const = false;
is_enum_const = false;
is_type = false;
offset = 0;
if ( is_blank )
if ( name && extract_var_name(name) == "_" ) {
is_blank = true;
SetType(base_type(TYPE_ANY));
}
opt_info = new IDOptInfo(this);
infer_return_type = false;
SetLocationInfo(&start_location, &end_location);
}

10
src/ID.h
View file

@ -81,7 +81,6 @@ public:
}
bool IsType() const { return is_type; }
void MakeType() { is_type = true; }
void SetVal(ValPtr v);
@ -160,9 +159,14 @@ protected:
const char* name;
IDScope scope;
bool is_export;
bool infer_return_type;
TypePtr type;
bool is_const, is_enum_const, is_type, is_option, is_blank;
bool is_capture = false;
bool is_const = false;
bool is_enum_const = false;
bool is_type = false;
bool is_option = false;
bool is_blank = false;
bool infer_return_type = false;
int offset;
ValPtr val;
AttributesPtr attrs;

9
src/Options.cc
View file

@ -194,6 +194,7 @@ static void print_analysis_help() {
fprintf(stderr, " optimize-AST optimize the (transformed) AST; implies xform\n");
fprintf(stderr, " profile-ZAM generate to zprof.out a ZAM execution profile; implies -O ZAM\n");
fprintf(stderr, " report-recursive report on recursive functions and exit\n");
fprintf(stderr, " validate-ZAM perform internal assessment of synthesized ZAM instructions and exit\n");
fprintf(stderr, " xform transform scripts to \"reduced\" form\n");
fprintf(stderr, "\n--optimize options when generating C++:\n");
@ -220,14 +221,14 @@ static void set_analysis_option(const char* opt, Options& opts) {
exit(0);
}
if ( util::streq(opt, "dump-uds") )
if ( util::streq(opt, "allow-cond") )
a_o.allow_cond = true;
else if ( util::streq(opt, "dump-uds") )
a_o.activate = a_o.dump_uds = true;
else if ( util::streq(opt, "dump-xform") )
a_o.activate = a_o.dump_xform = true;
else if ( util::streq(opt, "dump-ZAM") )
a_o.activate = a_o.dump_ZAM = true;
else if ( util::streq(opt, "allow-cond") )
a_o.allow_cond = true;
else if ( util::streq(opt, "gen-C++") )
a_o.gen_CPP = true;
else if ( util::streq(opt, "gen-standalone-C++") )
@ -254,6 +255,8 @@ static void set_analysis_option(const char* opt, Options& opts) {
a_o.report_uncompilable = true;
else if ( util::streq(opt, "use-C++") )
a_o.use_CPP = true;
else if ( util::streq(opt, "validate-ZAM") )
a_o.validate_ZAM = true;
else if ( util::streq(opt, "xform") )
a_o.activate = true;

8
src/Reporter.cc
View file

@ -203,6 +203,14 @@ void Reporter::CPPRuntimeError(const char* fmt, ...) {
throw InterpreterException();
}
void Reporter::CPPRuntimeWarning(const char* fmt, ...) {
va_list ap;
va_start(ap, fmt);
FILE* out = EmitToStderr(warnings_to_stderr) ? stderr : nullptr;
DoLog("runtime warning in compiled code", reporter_error, out, nullptr, nullptr, true, true, "", fmt, ap);
va_end(ap);
}
void Reporter::InternalError(const char* fmt, ...) {
va_list ap;
va_start(ap, fmt);

3
src/Reporter.h
View file

@ -115,6 +115,9 @@ public:
// function will not return but raise an InterpreterException.
[[noreturn]] void CPPRuntimeError(const char* fmt, ...) __attribute__((format(printf, 2, 3)));
// Similar, but for warnings. This function does return.
void CPPRuntimeWarning(const char* fmt, ...) __attribute__((format(printf, 2, 3)));
// Report a traffic weirdness, i.e., an unexpected protocol situation
// that may lead to incorrectly processing a connection.
void Weird(const char* name, const char* addl = "",

10
src/ScriptValidation.cc
View file

@ -58,6 +58,15 @@ public:
return TC_CONTINUE;
}
TraversalCode PreType(const Type* t) override {
if ( types_seen.count(t) > 0 )
return TC_ABORTSTMT;
types_seen.insert(t);
return TC_CONTINUE;
}
void SetHookDepth(int hd) { hook_depth = hd; }
bool IsValid() const { return valid_script; }
@ -83,6 +92,7 @@ private:
}
std::unordered_map<StmtTag, int> stmt_depths;
std::unordered_set<const Type*> types_seen;
int hook_depth = 0;
bool report; // whether to report problems via "reporter"
bool valid_script = true;

4
src/Span.h
View file

@ -2,7 +2,6 @@
#pragma once
#include <array>
#include <cstddef>
#include <iterator>
#include <type_traits>
@ -128,9 +127,6 @@ private:
template<class T>
Span(T*, size_t) -> Span<T>;
template<class Iter>
Span(Iter, Iter) -> Span<typename std::iterator_traits<Iter>::value_type>;
template<class T, size_t N>
Span(T (&)[N]) -> Span<T>;

3
src/Stmt.cc
View file

@ -1843,7 +1843,8 @@ void WhenInfo::Build(StmtPtr ws) {
auto else_branch = timeout_s ? timeout_s : empty;
auto do_bodies = make_intrusive<IfStmt>(two_test, s, else_branch);
auto dummy_return = make_intrusive<ReturnStmt>(true_const);
auto any_true_const = make_intrusive<CoerceToAnyExpr>(true_const);
auto dummy_return = make_intrusive<ReturnStmt>(any_true_const);
auto shebang = make_intrusive<StmtList>(do_test, do_bodies, dummy_return);

3
src/Type.cc
View file

@ -601,6 +601,9 @@ SetType::~SetType() = default;
FuncType::Capture::Capture(detail::IDPtr _id, bool _deep_copy) : id(std::move(_id)), deep_copy(_deep_copy) {
is_managed = id ? ZVal::IsManagedType(id->GetType()) : false;
if ( ! is_managed )
// For non-managed types, deep copying isn't applicable.
deep_copy = false;
}
FuncType::FuncType(RecordTypePtr arg_args, TypePtr arg_yield, FunctionFlavor arg_flavor)

5
src/Type.h
View file

@ -35,6 +35,7 @@ class CompositeHash;
class Expr;
class ListExpr;
class ZAMCompiler;
class CPPRuntime;
using ExprPtr = IntrusivePtr<Expr>;
using ListExprPtr = IntrusivePtr<ListExpr>;
@ -341,6 +342,9 @@ public:
void Append(TypePtr t);
void AppendEvenIfNotPure(TypePtr t);
// Resets the list to be empty.
void Clear() { types.clear(); }
detail::TraversalCode Traverse(detail::TraversalCallback* cb) const override;
protected:
@ -752,6 +756,7 @@ private:
class CreationInitsOptimizer;
friend zeek::RecordVal;
friend zeek::detail::ZAMCompiler;
friend zeek::detail::CPPRuntime;
const auto& DeferredInits() const { return deferred_inits; }
const auto& CreationInits() const { return creation_inits; }

2
src/Val.cc
View file

@ -1277,6 +1277,8 @@ FileVal::FileVal(FilePtr f) : Val(make_intrusive<FileType>(base_type(TYPE_STRING
assert(file_val->GetType()->Tag() == TYPE_STRING);
}
FilePtr FileVal::AsFilePtr() const { return file_val; }
ValPtr FileVal::SizeVal() const { return make_intrusive<DoubleVal>(file_val->Size()); }
void FileVal::ValDescribe(ODesc* d) const { file_val->Describe(d); }

10
src/Val.h
View file

@ -609,6 +609,8 @@ class FileVal final : public Val {
public:
explicit FileVal(FilePtr f);
FilePtr AsFilePtr() const;
ValPtr SizeVal() const override;
File* Get() const { return file_val.get(); }
@ -682,6 +684,14 @@ public:
*/
void Append(ValPtr v);
/**
* Empties the list.
*/
void Clear() {
vals.clear();
type->AsTypeList()->Clear();
}
// Returns a Set representation of the list (which must be homogeneous).
TableValPtr ToSetVal() const;

2
src/ZVal.h
View file

@ -62,6 +62,7 @@ union ZVal {
ZVal(const TypePtr& t);
// Construct directly.
ZVal(bool v) { int_val = v; }
ZVal(zeek_int_t v) { int_val = v; }
ZVal(zeek_uint_t v) { uint_val = v; }
ZVal(double v) { double_val = v; }
@ -160,7 +161,6 @@ union ZVal {
private:
friend class RecordVal;
friend class VectorVal;
friend class zeek::detail::ZBody;
// Used for bool, int, enum.
zeek_int_t int_val;

61
src/analyzer/protocol/dns/DNS.cc
View file

@ -256,6 +256,8 @@ bool DNS_Interpreter::ParseAnswer(detail::DNS_MsgInfo* msg, const u_char*& data,
case detail::TYPE_EDNS: status = ParseRR_EDNS(msg, data, len, rdlength, msg_start); break;
case detail::TYPE_TKEY: status = ParseRR_TKEY(msg, data, len, rdlength, msg_start); break;
case detail::TYPE_TSIG: status = ParseRR_TSIG(msg, data, len, rdlength, msg_start); break;
case detail::TYPE_RRSIG: status = ParseRR_RRSIG(msg, data, len, rdlength, msg_start); break;
@ -824,6 +826,48 @@ bool DNS_Interpreter::ParseRR_TSIG(detail::DNS_MsgInfo* msg, const u_char*& data
return true;
}
bool DNS_Interpreter::ParseRR_TKEY(detail::DNS_MsgInfo* msg, const u_char*& data, int& len, int rdlength,
const u_char* msg_start) {
if ( ! dns_TKEY || msg->skip_event ) {
data += rdlength;
len -= rdlength;
return true;
}
if ( len < 16 )
return false;
const u_char* data_start = data;
u_char alg_name[513];
int alg_name_len = sizeof(alg_name) - 1;
u_char* alg_name_end = ExtractName(data, len, alg_name, alg_name_len, msg_start);
if ( ! alg_name_end )
return false;
uint32_t inception = ExtractLong(data, len);
uint32_t expiration = ExtractLong(data, len);
uint16_t mode = ExtractShort(data, len);
uint16_t error = ExtractShort(data, len);
String* key_data;
ExtractOctets(data, len, dns_TKEY ? &key_data : nullptr);
ExtractOctets(data, len, nullptr); // Other data
if ( dns_TKEY ) {
detail::TKEY_DATA tkey;
tkey.alg_name = new String(alg_name, int(alg_name_end - alg_name), true);
tkey.inception = inception;
tkey.expiration = expiration;
tkey.mode = mode;
tkey.error = error;
tkey.key = key_data;
analyzer->EnqueueConnEvent(dns_TKEY, analyzer->ConnVal(), msg->BuildHdrVal(), msg->BuildTKEY_Val(&tkey));
}
return true;
}
bool DNS_Interpreter::ParseRR_RRSIG(detail::DNS_MsgInfo* msg, const u_char*& data, int& len, int rdlength,
const u_char* msg_start) {
if ( ! dns_RRSIG || msg->skip_event ) {
@ -1660,6 +1704,23 @@ RecordValPtr DNS_MsgInfo::BuildEDNS_COOKIE_Val(struct EDNS_COOKIE* opt) {
return r;
}
RecordValPtr DNS_MsgInfo::BuildTKEY_Val(struct TKEY_DATA* tkey) {
static auto dns_tkey = id::find_type<RecordType>("dns_tkey");
auto r = make_intrusive<RecordVal>(dns_tkey);
r->Assign(0, query_name);
r->Assign(1, answer_type);
r->Assign(2, tkey->alg_name);
r->AssignTime(3, static_cast<double>(tkey->inception));
r->AssignTime(4, static_cast<double>(tkey->expiration));
r->Assign(5, static_cast<uint16_t>(tkey->mode));
r->Assign(6, static_cast<uint16_t>(tkey->error));
r->Assign(7, tkey->key);
r->Assign(8, is_query);
return r;
}
RecordValPtr DNS_MsgInfo::BuildTSIG_Val(struct TSIG_DATA* tsig) {
static auto dns_tsig_additional = id::find_type<RecordType>("dns_tsig_additional");
auto r = make_intrusive<RecordVal>(dns_tsig_additional);

11
src/analyzer/protocol/dns/DNS.h
View file

@ -191,6 +191,15 @@ struct EDNS_COOKIE {
zeek::String* server_cookie; ///< cookie value sent by the server (0 or 8-32 bytes)
};
struct TKEY_DATA {
String* alg_name;
unsigned long inception;
unsigned long expiration;
unsigned short mode;
unsigned short error;
String* key;
};
struct TSIG_DATA {
String* alg_name;
unsigned long time_s;
@ -278,6 +287,7 @@ public:
RecordValPtr BuildEDNS_ECS_Val(struct EDNS_ECS*);
RecordValPtr BuildEDNS_TCP_KA_Val(struct EDNS_TCP_KEEPALIVE*);
RecordValPtr BuildEDNS_COOKIE_Val(struct EDNS_COOKIE*);
RecordValPtr BuildTKEY_Val(struct TKEY_DATA*);
RecordValPtr BuildTSIG_Val(struct TSIG_DATA*);
RecordValPtr BuildRRSIG_Val(struct RRSIG_DATA*);
RecordValPtr BuildDNSKEY_Val(struct DNSKEY_DATA*);
@ -361,6 +371,7 @@ protected:
bool ParseRR_TXT(detail::DNS_MsgInfo* msg, const u_char*& data, int& len, int rdlength, const u_char* msg_start);
bool ParseRR_SPF(detail::DNS_MsgInfo* msg, const u_char*& data, int& len, int rdlength, const u_char* msg_start);
bool ParseRR_CAA(detail::DNS_MsgInfo* msg, const u_char*& data, int& len, int rdlength, const u_char* msg_start);
bool ParseRR_TKEY(detail::DNS_MsgInfo* msg, const u_char*& data, int& len, int rdlength, const u_char* msg_start);
bool ParseRR_TSIG(detail::DNS_MsgInfo* msg, const u_char*& data, int& len, int rdlength, const u_char* msg_start);
bool ParseRR_RRSIG(detail::DNS_MsgInfo* msg, const u_char*& data, int& len, int rdlength, const u_char* msg_start);
bool ParseRR_DNSKEY(detail::DNS_MsgInfo* msg, const u_char*& data, int& len, int rdlength, const u_char* msg_start);

17
src/analyzer/protocol/dns/events.bif
View file

@ -578,6 +578,23 @@ event dns_EDNS_tcp_keepalive%(c: connection, msg: dns_msg, opt: dns_edns_tcp_kee
## dns_skip_all_addl dns_skip_all_auth dns_skip_auth
event dns_EDNS_cookie%(c: connection, msg: dns_msg, opt: dns_edns_cookie%);
## Generated for DNS replies of type *TKEY*. For replies with multiple answers,
## an individual event of the corresponding type is raised for each.
##
## See `Wikipedia <http://en.wikipedia.org/wiki/Domain_Name_System>`__ for more
## information about the DNS protocol. See `RFC2930 <https://tools.ietf.org/html/rfc2930>`__
## for more information about TKEY. Zeek analyzes both UDP and TCP DNS sessions.
##
## c: The connection, which may be UDP or TCP depending on the type of the
## transport-layer session being analyzed.
##
## msg: The parsed DNS message header.
##
## ans: The parsed TKEY reply.
##
## .. zeek:see:: dns_TSIG_addl
event dns_TKEY%(c: connection, msg: dns_msg, ans: dns_tkey%);
## Generated for DNS replies of type *TSIG*. For replies with multiple answers,
## an individual event of the corresponding type is raised for each.
##

12
src/analyzer/protocol/mysql/events.bif
View file

@ -12,6 +12,18 @@
## .. zeek:see:: mysql_error mysql_ok mysql_server_version mysql_handshake
event mysql_command_request%(c: connection, command: count, arg: string%);
## Generated for a change user command from a MySQL client.
##
## See the MySQL `documentation <http://dev.mysql.com/doc/internals/en/client-server-protocol.html>`__
## for more information about the MySQL protocol.
##
## c: The connection.
##
## username: The username supplied by the client
##
## .. zeek:see:: mysql_error mysql_ok mysql_server_version mysql_handshake
event mysql_change_user%(c: connection, username: string%);
## Generated for an unsuccessful MySQL response.
##
## See the MySQL `documentation <http://dev.mysql.com/doc/internals/en/client-server-protocol.html>`__

38
src/analyzer/protocol/mysql/mysql-analyzer.pac
View file

@ -87,10 +87,44 @@ refine flow MySQL_Flow += {
function proc_mysql_command_request_packet(msg: Command_Request_Packet): bool
%{
if ( mysql_command_request )
{
auto arg = to_stringval(${msg.arg});
// CHANGE_USER will have parsed away the arg,
// restore it for backwards compat.
if ( ${msg.command} == COM_CHANGE_USER )
arg = to_stringval(${msg.change_user.sourcedata});
zeek::BifEvent::enqueue_mysql_command_request(connection()->zeek_analyzer(),
connection()->zeek_analyzer()->Conn(),
${msg.command},
to_stringval(${msg.arg}));
std::move(arg));
}
return true;
%}
function proc_mysql_change_user_packet(msg: Change_User_Packet): bool
%{
if ( mysql_change_user )
zeek::BifEvent::enqueue_mysql_change_user(connection()->zeek_analyzer(),
connection()->zeek_analyzer()->Conn(),
zeek::make_intrusive<zeek::StringVal>(c_str(${msg.username})));
if ( mysql_auth_plugin )
{
auto data = to_stringval(${msg.auth_plugin_data});
auto auth_plugin = zeek::val_mgr->EmptyString();
if ( ${msg.have_more_data} )
auth_plugin = zeek::make_intrusive<zeek::StringVal>(c_str(${msg.auth_plugin_name}));
zeek::BifEvent::enqueue_mysql_auth_plugin(connection()->zeek_analyzer(),
connection()->zeek_analyzer()->Conn(),
true /*is_orig*/,
std::move(auth_plugin),
std::move(data));
}
return true;
%}
@ -183,6 +217,8 @@ refine typeattr Handshake_Response_Packet += &let {
refine typeattr Command_Request_Packet += &let {
proc = $context.flow.proc_mysql_command_request_packet(this);
# Enqueue mysql_change_user() *after* mysql_command_request().
proc_change_user = $context.flow.proc_mysql_change_user_packet(change_user) &if(is_change_user);
};
refine typeattr ERR_Packet += &let {

103
src/analyzer/protocol/mysql/mysql-protocol.pac
View file

@ -154,7 +154,6 @@ enum Expected {
EXPECT_EOF_THEN_RESULTSET,
EXPECT_RESULTSET,
EXPECT_REST_OF_PACKET,
EXPECT_AUTH_SWITCH,
};
enum EOFType {
@ -296,8 +295,8 @@ type EmptyOrNUL_String = RE/([^\0]*\0)?/;
type MySQL_PDU(is_orig: bool) = record {
hdr : Header;
msg : case is_orig of {
false -> server_msg: Server_Message(hdr.seq_id, hdr.len);
true -> client_msg: Client_Message(state);
false -> server_msg: Server_Message(hdr.seq_id, hdr.len, state);
true -> client_msg: Client_Message(hdr.len, state);
} &requires(state);
} &let {
state: int = $context.connection.get_state();
@ -310,17 +309,17 @@ type Header = record {
len : uint32 = to_int()(le_len) + 4;
} &length=4;
type Server_Message(seq_id: uint8, pkt_len: uint32) = case is_initial of {
true -> initial_handshake: Initial_Handshake_Packet;
false -> command_response : Command_Response(pkt_len);
} &let {
type Server_Message(seq_id: uint8, pkt_len: uint32, state: int) = case state of {
CONNECTION_PHASE -> connection_phase: Server_Connection_Phase(is_initial);
COMMAND_PHASE -> command_response: Command_Response(pkt_len);
} &requires(is_initial) &let {
is_initial : bool = (seq_id == 0) && ($context.connection.get_previous_seq_id() != 255);
update_seq_id : bool = $context.connection.set_previous_seq_id(seq_id);
};
type Client_Message(state: int) = case state of {
CONNECTION_PHASE -> connection_phase: Connection_Phase_Packets;
COMMAND_PHASE -> command_phase : Command_Request_Packet;
type Server_Connection_Phase(is_initial: bool) = case is_initial of {
true -> initial_handshake: Initial_Handshake_Packet;
false -> subsequent_handshake: Server_Connection_Phase_Packets;
};
# Handshake Request
@ -360,8 +359,6 @@ type Handshake_v10 = record {
#
# https://dev.mysql.com/doc/dev/mysql-server/latest/page_protocol_connection_phase_packets_protocol_handshake_v10.html
auth_plugin_data_part_2_len = auth_plugin_data_len > 21 ? auth_plugin_data_len - 8 : 13;
update_auth_plugin: bool = $context.connection.set_auth_plugin(auth_plugin)
&if( ( capability_flags_2 << 16 ) & CLIENT_PLUGIN_AUTH );
server_query_attrs: bool = $context.connection.set_server_query_attrs(( capability_flags_2 << 16 ) & CLIENT_QUERY_ATTRIBUTES);
};
@ -371,6 +368,19 @@ type Handshake_v9 = record {
scramble : NUL_String;
};
# While in the CONNECTION_PHASE, handle the following packets. Note that
# this is subtly different from Command_Response_Status which interprets
# 0xfe as EOF packet and also has does not support AuthMoreData.
type Server_Connection_Phase_Packets = record {
pkt_type: uint8;
packet: case pkt_type of {
0x00 -> data_ok: OK_Packet;
0x01 -> auth_more_data: AuthMoreData(false);
0xfe -> auth_switch_request: AuthSwitchRequest;
0xff -> data_err: ERR_Packet;
};
};
# Handshake Response
type Handshake_Response_Packet = case $context.connection.get_version() of {
@ -414,9 +424,6 @@ type Handshake_Plain_v10(cap_flags: uint32) = record {
0x0 -> none_4 : empty;
};
} &let {
update_auth_plugin: bool = $context.connection.set_auth_plugin(auth_plugin)
&if( cap_flags & CLIENT_PLUGIN_AUTH );
# Switch client state into expecting more auth data. If the server responds
# with an OK_Packet before, will switch into COMMAND_PHASE.
update_conn_expectation: bool = $context.connection.set_next_conn_expected(EXPECT_AUTH_DATA)
@ -435,6 +442,7 @@ type Handshake_Response_Packet_v10 = record {
} &let {
deprecate_eof: bool = $context.connection.set_deprecate_eof(cap_flags & CLIENT_DEPRECATE_EOF);
client_query_attrs: bool = $context.connection.set_client_query_attrs(cap_flags & CLIENT_QUERY_ATTRIBUTES);
proc_cap_flags: bool = $context.connection.set_client_capabilities(cap_flags);
};
type Handshake_Response_Packet_v9 = record {
@ -451,6 +459,11 @@ type Handshake_Response_Packet_v9 = record {
# Connection Phase
type Client_Message(pkt_len: uint32, state: int) = case state of {
CONNECTION_PHASE -> connection_phase: Connection_Phase_Packets;
COMMAND_PHASE -> command_phase : Command_Request_Packet(pkt_len);
};
type Connection_Phase_Packets = case $context.connection.get_conn_expectation() of {
EXPECT_HANDSHAKE -> handshake_resp: Handshake_Response_Packet;
EXPECT_AUTH_DATA -> auth_data: AuthMoreData(true);
@ -501,17 +514,49 @@ type Query_Attributes = record {
# Command Request
type Command_Request_Packet = record {
type Command_Request_Packet(pkt_len: uint32) = record {
command: uint8;
attrs : case ( command == COM_QUERY && $context.connection.get_client_query_attrs() && $context.connection.get_server_query_attrs() ) of {
true -> query_attrs: Query_Attributes;
false -> none: empty;
};
have_change_user: case is_change_user of {
true -> change_user: Change_User_Packet(pkt_len);
false -> none_change_user: empty;
};
arg : bytestring &restofdata;
} &let {
is_change_user = command == COM_CHANGE_USER;
update_expectation: bool = $context.connection.set_next_expected_from_command(command);
};
# Command from the client to switch the user mid-session.
#
# https://dev.mysql.com/doc/dev/mysql-server/latest/page_protocol_com_change_user.html
type Change_User_Packet(pkt_len: uint32) = record {
username : NUL_String;
auth_plugin_data_len: uint8;
auth_plugin_data: bytestring &length=auth_plugin_data_len;
database: NUL_String;
charset: uint16;
auth_plugin_name_case: case have_auth_plugin_name of {
true -> auth_plugin_name: NUL_String;
false -> no_more_data1: empty;
};
conn_attrs_case: case have_conn_attrs of {
true -> conn_attrs: Handshake_Connection_Attributes;
false -> no_conn_attrs: empty;
};
} &let {
have_more_data = offsetof(auth_plugin_name_case) < pkt_len;
have_auth_plugin_name = have_more_data && ($context.connection.get_client_capabilities() & CLIENT_PLUGIN_AUTH) == CLIENT_PLUGIN_AUTH;
have_conn_attrs = have_more_data && ($context.connection.get_client_capabilities() & CLIENT_CONNECT_ATTRS) == CLIENT_CONNECT_ATTRS;
} &exportsourcedata;
# Command Response
type Command_Response(pkt_len: uint32) = case $context.connection.get_expectation() of {
@ -521,7 +566,6 @@ type Command_Response(pkt_len: uint32) = case $context.connection.get_expectatio
EXPECT_RESULTSET -> resultset : Resultset(pkt_len);
EXPECT_REST_OF_PACKET -> rest : bytestring &restofdata;
EXPECT_STATUS -> status : Command_Response_Status;
EXPECT_AUTH_SWITCH -> auth_switch : AuthSwitchRequest;
EXPECT_EOF_THEN_RESULTSET -> eof : EOFIfLegacyThenResultset(pkt_len);
default -> unknown : empty;
};
@ -530,10 +574,6 @@ type Command_Response_Status = record {
pkt_type: uint8;
response: case pkt_type of {
0x00 -> data_ok: OK_Packet;
# When still in the CONNECTION_PHASE, the server can reply
# with AuthMoreData which is 0x01 stuffed opaque payload.
# https://dev.mysql.com/doc/dev/mysql-server/latest/page_protocol_connection_phase_packets_protocol_auth_more_data.html
0x01 -> auth_more_data: AuthMoreData(false);
0xfe -> data_eof: EOF_Packet(EOF_END);
0xff -> data_err: ERR_Packet;
default -> unknown: empty;
@ -634,11 +674,9 @@ type AuthMoreData(is_orig: bool) = record {
};
type AuthSwitchRequest = record {
status: uint8 &enforce(status==254);
name : NUL_String;
data : bytestring &restofdata;
} &let {
update_auth_plugin : bool = $context.connection.set_auth_plugin(name);
update_conn_expectation: bool = $context.connection.set_next_conn_expected(EXPECT_AUTH_DATA);
# After an AuthSwitchRequest, server replies with OK_Packet, ERR_Packet or AuthMoreData.
update_expectation: bool = $context.connection.set_next_expected(EXPECT_STATUS);
@ -696,6 +734,7 @@ refine connection MySQL_Conn += {
bool deprecate_eof_;
bool server_query_attrs_;
bool client_query_attrs_;
uint32 client_capabilities_;
std::string auth_plugin_;
int query_attr_idx_;
%}
@ -785,19 +824,17 @@ refine connection MySQL_Conn += {
return true;
%}
function set_auth_plugin(a: bytestring): bool
function set_client_capabilities(c: uint32): bool
%{
// binpac::std_str() includes trailing \0 from parsing.
auto new_auth_plugin = std::string(binpac::c_str(a));
if ( ! auth_plugin_.empty() && new_auth_plugin != auth_plugin_ )
{
expected_ = EXPECT_AUTH_SWITCH;
}
auth_plugin_ = std::move(new_auth_plugin);
client_capabilities_ = c;
return true;
%}
function get_client_capabilities(): uint32
%{
return client_capabilities_;
%}
function get_expectation(): Expected
%{
return expected_;
@ -875,7 +912,7 @@ refine connection MySQL_Conn += {
expected_ = EXPECT_STATUS;
break;
case COM_CHANGE_USER:
expected_ = EXPECT_AUTH_SWITCH;
update_state(CONNECTION_PHASE);
break;
case COM_BINLOG_DUMP:
expected_ = NO_EXPECTATION;

47
src/broker/Manager.cc
View file

@ -853,16 +853,29 @@ bool Manager::AutoUnpublishEvent(const string& topic, Val* event) {
}
RecordVal* Manager::MakeEvent(ValPList* args, zeek::detail::Frame* frame) {
auto rval = new RecordVal(BifType::Record::Broker::Event);
// Deprecated MakeEvent() version using ValPList - requires extra copy.
zeek::Args cargs;
cargs.reserve(args->size());
for ( auto* a : *args )
cargs.push_back({zeek::NewRef{}, a});
return MakeEvent(ArgsSpan{cargs}, frame)->Ref()->AsRecordVal();
}
zeek::RecordValPtr Manager::MakeEvent(ArgsSpan args, zeek::detail::Frame* frame) {
scoped_reporter_location srl{frame};
return MakeEvent(args);
}
zeek::RecordValPtr Manager::MakeEvent(ArgsSpan args) {
auto rval = zeek::make_intrusive<RecordVal>(BifType::Record::Broker::Event);
auto arg_vec = make_intrusive<VectorVal>(vector_of_data_type);
rval->Assign(1, arg_vec);
Func* func = nullptr;
scoped_reporter_location srl{frame};
const Func* func = nullptr;
for ( auto i = 0; i < args->length(); ++i ) {
auto arg_val = (*args)[i];
if ( i == 0 ) {
for ( size_t index = 0; index < args.size(); index++ ) {
const auto& arg_val = args[index];
if ( index == 0 ) {
// Event val must come first.
if ( arg_val->GetType()->Tag() != TYPE_FUNC ) {
@ -877,10 +890,10 @@ RecordVal* Manager::MakeEvent(ValPList* args, zeek::detail::Frame* frame) {
return rval;
}
auto num_args = func->GetType()->Params()->NumFields();
auto num_args = static_cast<size_t>(func->GetType()->Params()->NumFields());
if ( num_args != args->length() - 1 ) {
Error("bad # of arguments: got %d, expect %d", args->length(), num_args + 1);
if ( num_args != args.size() - 1 ) {
Error("bad # of arguments: got %zu, expect %zu", args.size() - 1, num_args + 1);
return rval;
}
@ -888,12 +901,12 @@ RecordVal* Manager::MakeEvent(ValPList* args, zeek::detail::Frame* frame) {
continue;
}
const auto& got_type = (*args)[i]->GetType();
const auto& expected_type = func->GetType()->ParamList()->GetTypes()[i - 1];
const auto& got_type = arg_val->GetType();
const auto& expected_type = func->GetType()->ParamList()->GetTypes()[index - 1];
if ( ! same_type(got_type, expected_type) ) {
rval->Remove(0);
Error("event parameter #%d type mismatch, got %s, expect %s", i, type_name(got_type->Tag()),
Error("event parameter #%zu type mismatch, got %s, expect %s", index, type_name(got_type->Tag()),
type_name(expected_type->Tag()));
return rval;
}
@ -901,17 +914,17 @@ RecordVal* Manager::MakeEvent(ValPList* args, zeek::detail::Frame* frame) {
RecordValPtr data_val;
if ( same_type(got_type, detail::DataVal::ScriptDataType()) )
data_val = {NewRef{}, (*args)[i]->AsRecordVal()};
data_val = {NewRef{}, arg_val->AsRecordVal()};
else
data_val = BrokerData::ToRecordVal((*args)[i]);
data_val = BrokerData::ToRecordVal(arg_val);
if ( ! data_val->HasField(0) ) {
rval->Remove(0);
Error("failed to convert param #%d of type %s to broker data", i, type_name(got_type->Tag()));
Error("failed to convert param #%zu of type %s to broker data", index, type_name(got_type->Tag()));
return rval;
}
arg_vec->Assign(i - 1, std::move(data_val));
arg_vec->Assign(index - 1, std::move(data_val));
}
return rval;

23
src/broker/Manager.h
View file

@ -14,6 +14,7 @@
#include <unordered_map>
#include "zeek/IntrusivePtr.h"
#include "zeek/Span.h"
#include "zeek/broker/Data.h"
#include "zeek/iosource/IOSource.h"
#include "zeek/logging/WriterBackend.h"
@ -262,7 +263,27 @@ public:
* @return an `Event` record value. If an invalid event or arguments
* were supplied the optional "name" field will not be set.
*/
RecordVal* MakeEvent(ValPList* args, zeek::detail::Frame* frame);
[[deprecated("Remove in v8.1: Use the ArgsSpan version instead")]] RecordVal* MakeEvent(ValPList* args,
zeek::detail::Frame* frame);
using ArgsSpan = Span<const ValPtr>;
/**
* Create an `Event` record value from an event and its arguments.
* @param args A span pointing at the event arguments.
* @return an `Event` record value. If an invalid event or arguments
* were supplied the optional "name" field will not be set.
*/
zeek::RecordValPtr MakeEvent(ArgsSpan args);
/**
* Create an `Event` record value from an event and its arguments.
* @param args A span pointing at the event arguments.
* @param frame the calling frame, used to report location info upon error
* @return an `Event` record value. If an invalid event or arguments
* were supplied the optional "name" field will not be set.
*/
zeek::RecordValPtr MakeEvent(ArgsSpan args, zeek::detail::Frame* frame);
/**
* Register interest in peer event messages that use a certain topic prefix.

49
src/broker/messaging.bif
View file

@ -5,9 +5,16 @@
#include <set>
#include <string>
#include "zeek/Span.h"
#include "zeek/broker/Manager.h"
#include "zeek/logging/Manager.h"
namespace {
using ArgsSpan = zeek::Span<const zeek::ValPtr>;
}
static bool is_string_set(const zeek::Type* type)
{
if ( ! type->IsSet() )
@ -46,7 +53,7 @@ std::set<std::string> val_to_topic_set(zeek::Val* val)
return rval;
}
static bool publish_event_args(zeek::ValPList& args, const zeek::String* topic,
static bool publish_event_args(ArgsSpan args, const zeek::String* topic,
zeek::detail::Frame* frame)
{
zeek::Broker::Manager::ScriptScopeGuard ssg;
@ -57,9 +64,8 @@ static bool publish_event_args(zeek::ValPList& args, const zeek::String* topic,
args[0]->AsRecordVal());
else
{
auto ev = zeek::broker_mgr->MakeEvent(&args, frame);
rval = zeek::broker_mgr->PublishEvent(topic->CheckString(), ev);
Unref(ev);
auto ev = zeek::broker_mgr->MakeEvent(args, frame);
rval = zeek::broker_mgr->PublishEvent(topic->CheckString(), ev->AsRecordVal());
}
return rval;
@ -92,13 +98,9 @@ type Broker::Event: record;
function Broker::make_event%(...%): Broker::Event
%{
zeek::Broker::Manager::ScriptScopeGuard ssg;
const auto& bif_args = @ARGS@;
ValPList args(bif_args->size());
for ( auto i = 0u; i < bif_args->size(); ++i )
args.push_back((*bif_args)[i].get());
return RecordValPtr{zeek::AdoptRef{}, zeek::broker_mgr->MakeEvent(&args, frame)};
auto ev = zeek::broker_mgr->MakeEvent(ArgsSpan{*@ARGS@});
return zeek::cast_intrusive<RecordVal>(ev);
%}
## Publishes an event at a given topic.
@ -112,13 +114,8 @@ function Broker::make_event%(...%): Broker::Event
## Returns: true if the message is sent.
function Broker::publish%(topic: string, ...%): bool
%{
const auto& bif_args = @ARGS@;
ValPList args(bif_args->size() - 1);
for ( auto i = 1u; i < bif_args->size(); ++i )
args.push_back((*bif_args)[i].get());
auto rval = publish_event_args(args, topic->AsString(), frame);
auto rval = publish_event_args(ArgsSpan{*@ARGS@}.subspan(1),
topic->AsString(), frame);
return zeek::val_mgr->Bool(rval);
%}
@ -208,13 +205,8 @@ function Cluster::publish_rr%(pool: Pool, key: string, ...%): bool
if ( ! topic->AsString()->Len() )
return zeek::val_mgr->False();
const auto& bif_args = @ARGS@;
ValPList args(bif_args->size() - 2);
for ( auto i = 2u; i < bif_args->size(); ++i )
args.push_back((*bif_args)[i].get());
auto rval = publish_event_args(args, topic->AsString(), frame);
auto rval = publish_event_args(ArgsSpan{*@ARGS@}.subspan(2),
topic->AsString(), frame);
return zeek::val_mgr->Bool(rval);
%}
@ -251,12 +243,7 @@ function Cluster::publish_hrw%(pool: Pool, key: any, ...%): bool
if ( ! topic->AsString()->Len() )
return zeek::val_mgr->False();
const auto& bif_args = @ARGS@;
ValPList args(bif_args->size() - 2);
for ( auto i = 2u; i < bif_args->size(); ++i )
args.push_back((*bif_args)[i].get());
auto rval = publish_event_args(args, topic->AsString(), frame);
auto rval = publish_event_args(ArgsSpan{*@ARGS@}.subspan(2),
topic->AsString(), frame);
return zeek::val_mgr->Bool(rval);
%}

7
src/event.bif
View file

@ -885,8 +885,13 @@ event Pcap::file_done%(path: string%);
##
## first_bytes: A certain number of bytes at the start of the unknown protocol's header.
##
## analyzer_history: The chain of packet analyzers that processed the packet up to this
## point. This includes the history of encapsulating packets in case
## of tunneling.
##
## .. zeek:see:: UnknownProtocol::first_bytes_count
event unknown_protocol%(analyzer_name: string, protocol: count, first_bytes: string%);
event unknown_protocol%(analyzer_name: string, protocol: count, first_bytes: string, analyzer_history: string_vec%);
event unknown_protocol%(analyzer_name: string, protocol: count, first_bytes: string%) &deprecated="Remove in v8.1. Use the version that includes analyzer_history.";
## An event for handling packets that reached the end of processing without
## being marked as processed. Note that this event may lead to unpredictable

2
src/packet_analysis/Analyzer.cc
View file

@ -114,6 +114,7 @@ bool Analyzer::ForwardPacket(size_t len, const uint8_t* data, Packet* packet, ui
DBG_LOG(DBG_PACKET_ANALYSIS, "Analysis in %s succeeded, next layer identifier is %#x.", GetAnalyzerName(),
identifier);
packet_mgr->TrackAnalyzer(inner_analyzer.get());
return inner_analyzer->AnalyzePacket(len, data, packet);
}
@ -129,6 +130,7 @@ bool Analyzer::ForwardPacket(size_t len, const uint8_t* data, Packet* packet) co
return false;
}
packet_mgr->TrackAnalyzer(inner_analyzer.get());
return inner_analyzer->AnalyzePacket(len, data, packet);
}

14
src/packet_analysis/Manager.cc
View file

@ -113,6 +113,7 @@ void Manager::ProcessPacket(Packet* packet) {
}
// Start packet analysis
analyzer_stack.clear();
root_analyzer->ForwardPacket(packet->cap_len, packet->data, packet, packet->link_type);
if ( ! packet->processed ) {
@ -227,13 +228,24 @@ bool Manager::PermitUnknownProtocol(const std::string& analyzer, uint32_t protoc
return false;
}
zeek::VectorValPtr Manager::BuildAnalyzerHistory() const {
auto history = zeek::make_intrusive<zeek::VectorVal>(zeek::id::string_vec);
for ( unsigned int i = 0; i < analyzer_stack.size(); i++ ) {
auto analyzer_name = analyzer_stack[i]->GetAnalyzerName();
history->Assign(i, make_intrusive<StringVal>(analyzer_name));
}
return history;
}
void Manager::ReportUnknownProtocol(const std::string& analyzer, uint32_t protocol, const uint8_t* data, size_t len) {
if ( unknown_protocol ) {
if ( PermitUnknownProtocol(analyzer, protocol) ) {
int bytes_len = std::min(unknown_first_bytes_count, static_cast<uint64_t>(len));
event_mgr.Enqueue(unknown_protocol, make_intrusive<StringVal>(analyzer), val_mgr->Count(protocol),
make_intrusive<StringVal>(bytes_len, (const char*)data));
make_intrusive<StringVal>(bytes_len, (const char*)data), BuildAnalyzerHistory());
}
}
}

19
src/packet_analysis/Manager.h
View file

@ -176,6 +176,15 @@ public:
*/
uint64_t GetUnprocessedCount() const { return total_not_processed; }
/**
* Tracks the given analyzer for the current packet's analyzer history.
* The packet analyzer history is implemented in form of a stack, which is reset on a
* call to ProcessPacket() but maintained throughout calls to ProcessInnerPacket().
*
* @param analyzer The analyzer to track.
*/
void TrackAnalyzer(Analyzer* analyzer) { analyzer_stack.push_back(analyzer); }
private:
/**
* Instantiates a new analyzer instance.
@ -197,6 +206,14 @@ private:
*/
AnalyzerPtr InstantiateAnalyzer(const std::string& name);
/**
* Generates a string vector that represents the analyzer history of the
* current packet based on the analyzers' tags.
*
* @return A vector of strings representing the packet analyzer history.
*/
VectorValPtr BuildAnalyzerHistory() const;
bool PermitUnknownProtocol(const std::string& analyzer, uint32_t protocol);
std::map<std::string, AnalyzerPtr> analyzers;
@ -216,6 +233,8 @@ private:
uint64_t total_not_processed = 0;
iosource::PktDumper* unprocessed_dumper = nullptr;
std::vector<Analyzer*> analyzer_stack;
};
} // namespace packet_analysis

1
src/packet_analysis/protocol/wrapper/CMakeLists.txt
View file

@ -1 +0,0 @@
zeek_add_plugin(Zeek Wrapper SOURCES Wrapper.cc Plugin.cc)

25
src/packet_analysis/protocol/wrapper/Plugin.cc
View file

@ -1,25 +0,0 @@
// See the file "COPYING" in the main distribution directory for copyright.
#include "zeek/plugin/Plugin.h"
#include "zeek/packet_analysis/Component.h"
#include "zeek/packet_analysis/protocol/wrapper/Wrapper.h"
namespace zeek::plugin::Zeek_Wrapper {
class Plugin final : public zeek::plugin::Plugin {
public:
zeek::plugin::Configuration Configure() override {
AddComponent(
new zeek::packet_analysis::Component("Wrapper",
zeek::packet_analysis::Wrapper::WrapperAnalyzer::Instantiate));
zeek::plugin::Configuration config;
config.name = "Zeek::Wrapper";
config.description = "A wrapper for the original zeek code.";
return config;
}
} plugin;
} // namespace zeek::plugin::Zeek_Wrapper

135
src/packet_analysis/protocol/wrapper/Wrapper.cc
View file

@ -1,135 +0,0 @@
// See the file "COPYING" in the main distribution directory for copyright.
#include "zeek/packet_analysis/protocol/wrapper/Wrapper.h"
using namespace zeek::packet_analysis::Wrapper;
WrapperAnalyzer::WrapperAnalyzer() : zeek::packet_analysis::Analyzer("Wrapper") {}
bool WrapperAnalyzer::Analyze(Packet* packet, const uint8_t*& data) {
// Unfortunately some packets on the link might have MPLS labels
// while others don't. That means we need to ask the link-layer if
// labels are in place.
bool have_mpls = false;
auto end_of_data = packet->GetEndOfData();
// Skip past Cisco FabricPath to encapsulated ethernet frame.
if ( data[12] == 0x89 && data[13] == 0x03 ) {
auto constexpr cfplen = 16;
if ( data + cfplen + 14 >= end_of_data ) {
Weird("truncated_link_header_cfp", packet);
return false;
}
data += cfplen;
}
// Extract protocol identifier
uint32_t protocol = (data[12] << 8u) + data[13];
packet->eth_type = protocol;
packet->l2_dst = data;
packet->l2_src = data + 6;
data += 14;
bool saw_vlan = false;
while ( protocol == 0x8100 || protocol == 0x9100 || protocol == 0x8864 ) {
switch ( protocol ) {
// VLAN carried over the ethernet frame.
// 802.1q / 802.1ad
case 0x8100:
case 0x9100: {
if ( data + 4 >= end_of_data ) {
Weird("truncated_link_header", packet);
return false;
}
auto& vlan_ref = saw_vlan ? packet->inner_vlan : packet->vlan;
vlan_ref = ((data[0] << 8u) + data[1]) & 0xfff;
protocol = ((data[2] << 8u) + data[3]);
data += 4; // Skip the vlan header
saw_vlan = true;
packet->eth_type = protocol;
} break;
// PPPoE carried over the ethernet frame.
case 0x8864: {
if ( data + 8 >= end_of_data ) {
Weird("truncated_link_header", packet);
return false;
}
protocol = (data[6] << 8u) + data[7];
data += 8; // Skip the PPPoE session and PPP header
if ( protocol == 0x0021 )
packet->l3_proto = L3_IPV4;
else if ( protocol == 0x0057 )
packet->l3_proto = L3_IPV6;
else {
// Neither IPv4 nor IPv6.
Weird("non_ip_packet_in_pppoe_encapsulation", packet);
return false;
}
} break;
}
}
// Check for MPLS in VLAN.
if ( protocol == 0x8847 )
have_mpls = true;
// Normal path to determine Layer 3 protocol.
if ( ! have_mpls && packet->l3_proto == L3_UNKNOWN ) {
if ( protocol == 0x800 )
packet->l3_proto = L3_IPV4;
else if ( protocol == 0x86dd )
packet->l3_proto = L3_IPV6;
else if ( protocol == 0x0806 || protocol == 0x8035 )
packet->l3_proto = L3_ARP;
else {
// Neither IPv4 nor IPv6.
Weird("non_ip_packet_in_ethernet", packet);
return false;
}
}
if ( have_mpls ) {
// Skip the MPLS label stack.
bool end_of_stack = false;
while ( ! end_of_stack ) {
if ( data + 4 >= end_of_data ) {
Weird("truncated_link_header", packet);
return false;
}
end_of_stack = *(data + 2u) & 0x01;
data += 4;
}
// We assume that what remains is IP
if ( data + sizeof(struct ip) >= end_of_data ) {
Weird("no_ip_in_mpls_payload", packet);
return false;
}
const struct ip* ip = (const struct ip*)data;
if ( ip->ip_v == 4 )
packet->l3_proto = L3_IPV4;
else if ( ip->ip_v == 6 )
packet->l3_proto = L3_IPV6;
else {
// Neither IPv4 nor IPv6.
Weird("no_ip_in_mpls_payload", packet);
return false;
}
}
return AnalyzeInnerPacket(packet, data, protocol);
}

20
src/packet_analysis/protocol/wrapper/Wrapper.h
View file

@ -1,20 +0,0 @@
// See the file "COPYING" in the main distribution directory for copyright.
#pragma once
#include "zeek/packet_analysis/Analyzer.h"
#include "zeek/packet_analysis/Component.h"
namespace zeek::packet_analysis::Wrapper {
class WrapperAnalyzer : public Analyzer {
public:
WrapperAnalyzer();
~WrapperAnalyzer() override = default;
bool Analyze(Packet* packet, const uint8_t*& data) override;
static zeek::packet_analysis::AnalyzerPtr Instantiate() { return std::make_shared<WrapperAnalyzer>(); }
};
} // namespace zeek::packet_analysis::Wrapper

14
src/script_opt/CPP/Compile.h
View file

@ -143,6 +143,7 @@ public:
// cohort associated with a given type.
int TypeOffset(const TypePtr& t) { return GI_Offset(RegisterType(t)); }
int TypeCohort(const TypePtr& t) { return GI_Cohort(RegisterType(t)); }
int TypeFinalCohort(const TypePtr& t) { return GI_FinalCohort(RegisterType(t)); }
// Tracks a Zeek ValPtr used as a constant value. These occur
// in two contexts: directly as constant expressions, and indirectly
@ -384,6 +385,10 @@ private:
std::string LocalName(const ID* l) const;
std::string LocalName(const IDPtr& l) const { return LocalName(l.get()); }
// The same, but for a capture.
std::string CaptureName(const ID* l) const;
std::string CaptureName(const IDPtr& l) const { return CaptureName(l.get()); }
// Returns a canonicalized name, with various non-alphanumeric
// characters stripped or transformed, and guaranteed not to
// conflict with C++ keywords.
@ -584,8 +589,11 @@ private:
// Maps function names to events relevant to them.
std::unordered_map<std::string, std::vector<std::string>> body_events;
// Full type of the function we're currently compiling.
FuncTypePtr func_type;
// Return type of the function we're currently compiling.
TypePtr ret_type = nullptr;
TypePtr ret_type;
// Internal name of the function we're currently compiling.
std::string body_name;
@ -696,6 +704,8 @@ private:
void GenValueSwitchStmt(const Expr* e, const case_list* cases);
void GenWhenStmt(const WhenStmt* w);
void GenWhenStmt(const WhenInfo* wi, const std::string& when_lambda, const Location* loc,
std::vector<std::string> local_aggrs);
void GenForStmt(const ForStmt* f);
void GenForOverTable(const ExprPtr& tbl, const IDPtr& value_var, const IDPList* loop_vars);
void GenForOverVector(const ExprPtr& tbl, const IDPtr& value_var, const IDPList* loop_vars);
@ -770,6 +780,7 @@ private:
std::string GenSizeExpr(const Expr* e, GenType gt);
std::string GenScheduleExpr(const Expr* e);
std::string GenLambdaExpr(const Expr* e);
std::string GenLambdaExpr(const Expr* e, std::string capture_args);
std::string GenIsExpr(const Expr* e, GenType gt);
std::string GenArithCoerceExpr(const Expr* e, GenType gt);
@ -963,6 +974,7 @@ private:
// associated with an initialization.
int GI_Offset(const std::shared_ptr<CPP_InitInfo>& gi) const { return gi ? gi->Offset() : -1; }
int GI_Cohort(const std::shared_ptr<CPP_InitInfo>& gi) const { return gi ? gi->InitCohort() : 0; }
int GI_FinalCohort(const std::shared_ptr<CPP_InitInfo>& gi) const { return gi ? gi->FinalInitCohort() : 0; }
// Generate code to initialize the mappings for record field
// offsets for field accesses into regions of records that

21
src/script_opt/CPP/DeclFunc.cc
View file

@ -31,7 +31,7 @@ void CPPCompile::DeclareLambda(const LambdaExpr* l, const ProfileFunc* pf) {
auto& ids = l->OuterIDs();
for ( auto id : ids )
lambda_names[id] = LocalName(id);
lambda_names[id] = CaptureName(id);
CreateFunction(l_id->GetType<FuncType>(), pf, lname, body, 0, l, FUNC_FLAVOR_FUNCTION);
}
@ -40,7 +40,12 @@ void CPPCompile::CreateFunction(const FuncTypePtr& ft, const ProfileFunc* pf, co
int priority, const LambdaExpr* l, FunctionFlavor flavor) {
const auto& yt = ft->Yield();
in_hook = flavor == FUNC_FLAVOR_HOOK;
const IDPList* lambda_ids = l ? &l->OuterIDs() : nullptr;
IDPList effective_lambda_ids;
if ( l )
effective_lambda_ids = l->OuterIDs();
const IDPList* lambda_ids = l ? &effective_lambda_ids : nullptr;
string args = BindArgs(ft, lambda_ids);
@ -328,18 +333,16 @@ void CPPCompile::GatherParamNames(vector<string>& p_names, const FuncTypePtr& ft
if ( param_id ) {
if ( t->Tag() == TYPE_ANY && param_id->GetType()->Tag() != TYPE_ANY )
// We'll need to translate the parameter
// from its current representation to
// type "any".
// We'll need to translate the parameter from its current
// representation to type "any".
p_names.emplace_back(string("any_param__CPP_") + Fmt(i));
else
p_names.emplace_back(LocalName(param_id));
}
else
// Parameters that are unused don't wind up in the
// ProfileFunc. Rather than dig their name out of
// the function's declaration, we explicitly name
// them to reflect that they're unused.
// Parameters that are unused don't wind up in the ProfileFunc.
// Rather than dig their name out of the function's declaration,
// we explicitly name them to reflect that they're unused.
p_names.emplace_back(string("unused_param__CPP_") + Fmt(i));
}

7
src/script_opt/CPP/Driver.cc
View file

@ -37,11 +37,12 @@ void CPPCompile::Compile(bool report_uncompilable) {
// previously compiled instances of those if present.
for ( auto& func : funcs ) {
const auto& f = func.Func();
auto& body = func.Body();
auto& ofiles = analysis_options.only_files;
auto allow_cond = analysis_options.allow_cond;
string fn = func.Body()->GetLocationInfo()->filename;
string fn = body->GetLocationInfo()->filename;
if ( ! allow_cond && ! func.ShouldSkip() && ! ofiles.empty() && files_with_conditionals.count(fn) > 0 ) {
if ( report_uncompilable )
@ -184,8 +185,8 @@ void CPPCompile::GenProlog() {
Emit("namespace CPP_%s { // %s\n", Fmt(total_hash), string(working_dir));
// The following might-or-might-not wind up being populated/used.
Emit("std::vector<int> field_mapping;");
Emit("std::vector<int> enum_mapping;");
Emit("std::vector<zeek_int_t> field_mapping;");
Emit("std::vector<zeek_int_t> enum_mapping;");
NL();
const_info[TYPE_BOOL] = CreateConstInitInfo("Bool", "ValPtr", "bool");

18
src/script_opt/CPP/Exprs.cc
View file

@ -639,13 +639,15 @@ string CPPCompile::GenScheduleExpr(const Expr* e) {
}
string CPPCompile::GenLambdaExpr(const Expr* e) {
auto l = static_cast<const LambdaExpr*>(e);
auto& body = l->Ingredients()->Body();
return GenLambdaExpr(e, GenLambdaClone(l, false));
}
string CPPCompile::GenLambdaExpr(const Expr* e, string capture_args) {
auto l = static_cast<const LambdaExpr*>(e);
auto name = Canonicalize(l->Name().c_str()) + "_lb_cl";
auto cl_args = string("\"") + name + "\"";
if ( l->OuterIDs().size() > 0 )
cl_args = cl_args + GenLambdaClone(l, false);
auto cl_args = string("\"") + name + "\"" + std::move(capture_args);
auto body = string("make_intrusive<") + name + ">(" + cl_args + ")";
auto func = string("make_intrusive<CPPLambdaFunc>(\"") + l->Name() + "\", cast_intrusive<FuncType>(" +
GenTypeName(l->GetType()) + "), " + body + ")";
@ -1175,7 +1177,7 @@ string CPPCompile::GenLambdaClone(const LambdaExpr* l, bool all_deep) {
for ( const auto& id : ids ) {
const auto& id_t = id->GetType();
auto arg = LocalName(id);
auto arg = CaptureName(id);
if ( captures && ! IsNativeType(id_t) ) {
for ( const auto& c : *captures )
@ -1183,7 +1185,7 @@ string CPPCompile::GenLambdaClone(const LambdaExpr* l, bool all_deep) {
arg = string("cast_intrusive<") + TypeName(id_t) + ">(" + arg + "->Clone())";
}
cl_args = cl_args + ", " + arg;
cl_args += ", " + arg;
}
return cl_args;
@ -1248,7 +1250,7 @@ string CPPCompile::GenEnum(const TypePtr& t, const ValPtr& ev) {
if ( ! et->HasRedefs() )
// Can use direct access.
return std::to_string(v);
return "zeek_int_t(" + std::to_string(v) + ")";
// Need to dynamically map the access.
int mapping_slot;

14
src/script_opt/CPP/Func.h
View file

@ -8,9 +8,7 @@
#include "zeek/Func.h"
#include "zeek/script_opt/ProfileFunc.h"
namespace zeek {
namespace detail {
namespace zeek::detail {
// A subclass of Func used for lambdas that the compiler creates for
// complex initializations (expressions used in type attributes).
@ -42,11 +40,6 @@ public:
const std::string& Name() { return name; }
// Sets/returns a hash associated with this statement. A value
// of 0 means "not set".
p_hash_type GetHash() const { return hash; }
void SetHash(p_hash_type h) { hash = h; }
// The following only get defined by lambda bodies.
virtual void SetLambdaCaptures(Frame* f) {}
virtual std::vector<ValPtr> SerializeLambdaCaptures() const { return std::vector<ValPtr>{}; }
@ -64,7 +57,6 @@ protected:
TraversalCode Traverse(TraversalCallback* cb) const override { return TC_CONTINUE; }
std::string name;
p_hash_type hash = 0ULL;
// A pseudo AST "call" node, used to support error localization.
CallExprPtr ce;
@ -117,6 +109,4 @@ extern std::unordered_map<p_hash_type, void (*)()> standalone_callbacks;
// Callbacks to finalize initialization of standalone compiled scripts.
extern std::vector<void (*)()> standalone_finalizations;
} // namespace detail
} // namespace zeek
} // namespace zeek::detail

25
src/script_opt/CPP/GenFunc.cc
View file

@ -38,12 +38,18 @@ void CPPCompile::GenInvokeBody(const string& call, const TypePtr& t) {
void CPPCompile::DefineBody(const FuncTypePtr& ft, const ProfileFunc* pf, const string& fname, const StmtPtr& body,
const IDPList* lambda_ids, FunctionFlavor flavor) {
IDPList l_ids;
if ( lambda_ids )
l_ids = *lambda_ids;
locals.clear();
params.clear();
body_name = fname;
func_type = ft;
ret_type = ft->Yield();
in_hook = flavor == FUNC_FLAVOR_HOOK;
auto ret_type_str = in_hook ? "bool" : FullTypeName(ret_type);
@ -52,7 +58,7 @@ void CPPCompile::DefineBody(const FuncTypePtr& ft, const ProfileFunc* pf, const
NL();
Emit("%s %s(%s)", ret_type_str, fname, ParamDecl(ft, lambda_ids, pf));
Emit("%s %s(%s)", ret_type_str, fname, ParamDecl(ft, &l_ids, pf));
StartBlock();
@ -64,7 +70,7 @@ void CPPCompile::DefineBody(const FuncTypePtr& ft, const ProfileFunc* pf, const
InitializeEvents(pf);
// Create the local variables.
DeclareLocals(pf, lambda_ids);
DeclareLocals(pf, &l_ids);
GenStmt(body);
@ -135,11 +141,12 @@ void CPPCompile::InitializeEvents(const ProfileFunc* pf) {
}
void CPPCompile::DeclareLocals(const ProfileFunc* pf, const IDPList* lambda_ids) {
// It's handy to have a set of the lambda captures rather than a list.
IDSet lambda_set;
// We track captures by their names rather than their ID*'s because the
// latter can be inconsistent when inlining.
set<string> capture_names;
if ( lambda_ids )
for ( auto li : *lambda_ids )
lambda_set.insert(li);
capture_names.insert(CaptureName(li));
const auto& ls = pf->Locals();
@ -149,11 +156,11 @@ void CPPCompile::DeclareLocals(const ProfileFunc* pf, const IDPList* lambda_ids)
for ( const auto& l : ls ) {
auto ln = LocalName(l);
auto cn = CaptureName(l);
if ( lambda_set.count(l) > 0 )
// No need to declare these, they're passed in as
// parameters.
ln = lambda_names[l];
if ( capture_names.count(cn) > 0 )
// No need to declare these, they're passed in as parameters.
ln = cn;
else if ( params.count(l) == 0 ) { // Not a parameter, so must be a local.
Emit("%s %s;", FullTypeName(l->GetType()), ln);

2
src/script_opt/CPP/Inits.cc
View file

@ -166,7 +166,7 @@ void CPPCompile::InitializeConsts() {
StartBlock();
for ( const auto& c : consts )
Emit("CPP_ValElem(%s, %s),", TypeTagName(c.first), Fmt(c.second));
Emit("{%s, %s},", TypeTagName(c.first), Fmt(c.second));
EndBlock(true);
}

79
src/script_opt/CPP/InitsInfo.cc
View file

@ -7,6 +7,7 @@
#include "zeek/ZeekString.h"
#include "zeek/script_opt/CPP/Attrs.h"
#include "zeek/script_opt/CPP/Compile.h"
#include "zeek/script_opt/CPP/RuntimeInits.h"
using namespace std;
@ -38,6 +39,13 @@ void CPP_InitsInfo::GenerateInitializers(CPPCompile* c) {
c->Emit("%s %s = %s(%s, %s,", gt, InitializersName(), gt, base_name, Fmt(offset_set));
c->IndentUp();
GenerateCohorts(c);
c->IndentDown();
c->Emit(");");
}
void CPP_InitsInfo::GenerateCohorts(CPPCompile* c) {
c->Emit("{");
int n = 0;
@ -47,7 +55,7 @@ void CPP_InitsInfo::GenerateInitializers(CPPCompile* c) {
if ( ++n > 1 )
c->Emit("");
if ( cohort.size() == 1 && ! IsCompound() )
if ( cohort.size() == 1 && ! UsesCompoundVectors() )
BuildCohort(c, cohort);
else {
c->Emit("{");
@ -57,8 +65,6 @@ void CPP_InitsInfo::GenerateInitializers(CPPCompile* c) {
}
c->Emit("}");
c->IndentDown();
c->Emit(");");
}
void CPP_InitsInfo::BuildOffsetSet(CPPCompile* c) {
@ -80,25 +86,25 @@ void CPP_InitsInfo::BuildOffsetSet(CPPCompile* c) {
offset_set = c->IndMgr().AddIndices(offsets_vec);
}
void CPP_InitsInfo::BuildCohort(CPPCompile* c, std::vector<std::shared_ptr<CPP_InitInfo>>& cohort) {
int n = 0;
for ( auto& co : cohort ) {
vector<string> ivs;
auto o = co->InitObj();
if ( o ) {
static std::string describe_initializer(const Obj* o) {
auto od = obj_desc(o);
// Escape any embedded comment characters.
od = regex_replace(od, std::regex("/\\*"), "<<SLASH-STAR>>");
od = regex_replace(od, std::regex("\\*/"), "<<STAR-SLASH>>");
c->Emit("/* #%s: Initializing %s: */", Fmt(co->Offset()), od);
return od;
}
void CPP_InitsInfo::BuildCohort(CPPCompile* c, std::vector<std::shared_ptr<CPP_InitInfo>>& cohort) {
for ( auto& co : cohort ) {
vector<string> ivs;
auto o = co->InitObj();
if ( o )
c->Emit("/* #%s: Initializing %s: */", Fmt(co->Offset()), describe_initializer(o));
co->InitializerVals(ivs);
BuildCohortElement(c, co->InitializerType(), ivs);
++n;
}
}
@ -117,12 +123,50 @@ void CPP_InitsInfo::BuildCohortElement(CPPCompile* c, string init_type, vector<s
c->Emit("std::make_shared<%s>(%s),", init_type, full_init);
}
void CPP_CompoundInitsInfo::GenerateInitializers(CPPCompile* c) {
c->Emit("");
c->Emit("static int %s_init[] = {", tag);
int n = 0;
c->IndentUp();
for ( auto& cohort : instances ) {
if ( ++n > 1 )
c->Emit("");
// Figure out the size of the cohort.
for ( auto& co : cohort ) {
auto o = co->InitObj();
if ( o )
c->Emit("/* #%s: Initializing %s: */", Fmt(co->Offset()), describe_initializer(o));
vector<string> ivs;
co->InitializerVals(ivs);
c->Emit(Fmt(int(ivs.size())) + ",");
BuildCohortElement(c, co->InitializerType(), ivs);
}
static const auto end_of_vv = Fmt(END_OF_VEC_VEC) + ",";
c->Emit(end_of_vv);
}
static const auto end_of_vvv = Fmt(END_OF_VEC_VEC_VEC) + ",";
c->Emit(end_of_vvv);
c->IndentDown();
c->Emit("};");
CPP_InitsInfo::GenerateInitializers(c);
}
void CPP_CompoundInitsInfo::GenerateCohorts(CPPCompile* c) { c->Emit("%s_init", tag); }
void CPP_CompoundInitsInfo::BuildCohortElement(CPPCompile* c, string init_type, vector<string>& ivs) {
string init_line;
for ( auto& iv : ivs )
init_line += iv + ",";
c->Emit("{ %s},", init_line);
c->Emit("%s", init_line);
}
void CPP_BasicConstInitsInfo::BuildCohortElement(CPPCompile* c, string init_type, vector<string>& ivs) {
@ -174,7 +218,7 @@ PatternConstInfo::PatternConstInfo(CPPCompile* c, ValPtr v) : CPP_InitInfo(v) {
CompoundItemInfo::CompoundItemInfo(CPPCompile* _c, ValPtr v) : CPP_InitInfo(v), c(_c) {
auto& t = v->GetType();
type = c->TypeOffset(t);
init_cohort = c->TypeCohort(t) + 1;
init_cohort = c->TypeFinalCohort(t) + 1;
}
ListConstInfo::ListConstInfo(CPPCompile* _c, ValPtr v) : CompoundItemInfo(_c) {
@ -400,6 +444,10 @@ void TypeTypeInfo::AddInitializerVals(std::vector<std::string>& ivs) const {
}
VectorTypeInfo::VectorTypeInfo(CPPCompile* _c, TypePtr _t) : AbstractTypeInfo(_c, std::move(_t)) {
auto vt = t->AsVectorType();
if ( vt->IsUnspecifiedVector() )
yield = base_type(TYPE_VOID);
else
yield = t->Yield();
auto gi = c->RegisterType(yield);
if ( gi )
@ -552,7 +600,8 @@ void IndicesManager::Generate(CPPCompile* c) {
c->Emit(line);
}
c->Emit("-1");
static const auto end_of_vv = Fmt(END_OF_VEC_VEC);
c->Emit(end_of_vv);
c->EndBlock(true);
}

44
src/script_opt/CPP/InitsInfo.h
View file

@ -18,7 +18,7 @@
// standalone globals (for example, one for each BiF that a compiled script
// may call).
//
// For each of these types of initialization, our general approach is to a
// For each of these types of initialization, our general approach is to have a
// class that manages a single instance of that type, and an an object that
// manages all of those instances collectively. The latter object will, for
// example, attend to determining the offset into the run-time vector associated
@ -48,8 +48,15 @@
// safely use cohort(X) = cohort(Y).) We then execute run-time initialization
// in waves, one cohort at a time.
//
// Many forms of initialization are specified in terms of indices into globals
// that hold items of various types. Thus, the most common initialization
// information is a vector of integers/indices. These data structures can
// be recursive, too, namely we sometimes associate an index with a vector
// of integers/indices and then we can track multiple such vectors using
// another vector of integers/indices.
//
// Because C++ compilers can struggle when trying to optimize large quantities
// of code - clang in particular could take many CPU *hours* back when our
// of code - clang in particular could take many CPU *hours* back when the
// compiler just generated C++ code snippets for each initialization - rather
// than producing code that directly executes each given initialization, we
// instead employ a table-driven approach. The C++ initializers for the
@ -58,12 +65,14 @@
// cohort at a time) to obtain the information needed to initialize any given
// item.
//
// Many forms of initialization are specified in terms of indices into globals
// that hold items of various types. Thus, the most common initialization
// information is a vector of integers/indices. These data structures can
// be recursive, too, namely we sometimes associate an index with a vector
// of integers/indices and then we can track multiple such vectors using
// another vector of integers/indices.
// Even this has headaches for very large initializations: both clang and g++
// are *much* slower to initialize large vectors of simple template types
// (such as std::pair) than non-template types (such as a struct with two
// fields, which is all std::pair is, at the end of the day). A similar problem
// holds for initializing vectors-of-vectors-of-vectors, so we reduce these
// cases to simpler forms (structs for the first example, a single vector
// with information embedded within it for how to expand its values into
// a vector-of-vector-of-vector fr the second).
#include "zeek/File.h"
#include "zeek/Val.h"
@ -124,10 +133,10 @@ public:
// Sets the associated C++ type.
virtual void SetCPPType(std::string ct) { CPP_type = std::move(ct); }
// Whether this initializer is in terms of compound objects. Used
// Whether this initializer is in terms of compound vectors. Used
// for avoiding compiler warnings about singleton initializations in
// braces.
virtual bool IsCompound() const { return false; }
virtual bool UsesCompoundVectors() const { return false; }
// Returns the type associated with the table used for initialization
// (i.e., this is the type of the global returned by InitializersName()).
@ -137,9 +146,11 @@ public:
void AddInstance(std::shared_ptr<CPP_InitInfo> g);
// Emit code to populate the table used to initialize this collection.
void GenerateInitializers(CPPCompile* c);
virtual void GenerateInitializers(CPPCompile* c);
protected:
virtual void GenerateCohorts(CPPCompile* c);
// Computes offset_set - see below.
void BuildOffsetSet(CPPCompile* c);
@ -205,7 +216,7 @@ public:
BuildInitType();
}
bool IsCompound() const override { return true; }
bool UsesCompoundVectors() const override { return true; }
private:
void BuildInitType() { inits_type = std::string("CPP_CustomInits<") + CPPType() + ">"; }
@ -227,7 +238,7 @@ public:
inits_type = std::string("CPP_BasicConsts<") + CPP_type + ", " + c_type + ", " + tag + "Val>";
}
bool IsCompound() const override { return false; }
bool UsesCompoundVectors() const override { return false; }
void BuildCohortElement(CPPCompile* c, std::string init_type, std::vector<std::string>& ivs) override;
};
@ -245,7 +256,12 @@ public:
inits_type = std::string("CPP_IndexedInits<") + CPPType() + ">";
}
bool IsCompound() const override { return true; }
// This isn't true (anymore) because we separately build up the compound
// vectors needed for the initialization.
bool UsesCompoundVectors() const override { return false; }
void GenerateInitializers(CPPCompile* c) override;
void GenerateCohorts(CPPCompile* c) override;
void BuildCohortElement(CPPCompile* c, std::string init_type, std::vector<std::string>& ivs) override;
};

20
src/script_opt/CPP/RuntimeInits.cc
View file

@ -465,12 +465,12 @@ void CPP_GlobalInit::Generate(InitsManager* im, std::vector<void*>& /* inits_vec
global->SetAttrs(im->Attributes(attrs));
}
void generate_indices_set(int* inits, std::vector<std::vector<int>>& indices_set) {
size_t generate_indices_set(int* inits, std::vector<std::vector<int>>& indices_set) {
// First figure out how many groups of indices there are, so we
// can pre-allocate the outer vector.
auto i_ptr = inits;
int num_inits = 0;
while ( *i_ptr >= 0 ) {
while ( *i_ptr != END_OF_VEC_VEC && *i_ptr != END_OF_VEC_VEC_VEC ) {
++num_inits;
int n = *i_ptr;
i_ptr += n + 1; // skip over vector elements
@ -479,7 +479,7 @@ void generate_indices_set(int* inits, std::vector<std::vector<int>>& indices_set
indices_set.reserve(num_inits);
i_ptr = inits;
while ( *i_ptr >= 0 ) {
while ( *i_ptr != END_OF_VEC_VEC ) {
int n = *i_ptr;
++i_ptr;
std::vector<int> indices;
@ -490,6 +490,20 @@ void generate_indices_set(int* inits, std::vector<std::vector<int>>& indices_set
indices_set.emplace_back(std::move(indices));
}
return i_ptr - inits + 1;
}
std::vector<std::vector<std::vector<int>>> generate_indices_set(int* inits) {
std::vector<std::vector<std::vector<int>>> indices_set;
while ( *inits != END_OF_VEC_VEC_VEC ) {
std::vector<std::vector<int>> cohort_inits;
inits += generate_indices_set(inits, cohort_inits);
indices_set.push_back(std::move(cohort_inits));
}
return indices_set;
}
} // namespace zeek::detail

57
src/script_opt/CPP/RuntimeInits.h
View file

@ -19,6 +19,28 @@ using FuncValPtr = IntrusivePtr<FuncVal>;
class InitsManager;
// Helper function that takes a (large) array of int's and from them
// constructs the corresponding vector-of-vector-of-indices. Each
// vector-of-indices is represented first by an int specifying its
// size, and then that many int's for its values. We recognize the
// end of the array upon encountering a "size" entry of END_OF_VEC_VEC.
//
// Returns how many elements were processed out of "inits", including its
// terminator.
extern size_t generate_indices_set(int* inits, std::vector<std::vector<int>>& indices_set);
// The same but for one more level of vector construction. The source array
// has sub-arrays terminated with END_OF_VEC_VEC per the above, and the whole
// shebang is terminated with END_OF_VEC_VEC_VEC.
//
// Returns the vector construction.
extern std::vector<std::vector<std::vector<int>>> generate_indices_set(int* inits);
// These need to be distinct from any values that can appear, which means
// they should be negative, and not -1, which is used as a "N/A" value.
#define END_OF_VEC_VEC -100
#define END_OF_VEC_VEC_VEC -200
// An abstract helper class used to access elements of an initialization vector.
// We need the abstraction because InitsManager below needs to be able to refer
// to any of a range of templated classes.
@ -29,7 +51,12 @@ public:
};
// Convenient way to refer to an offset associated with a particular Zeek type.
using CPP_ValElem = std::pair<TypeTag, int>;
// A "struct" rather than a std::pair because C++ compilers are terribly slow
// at initializing large numbers of the latter.
struct CPP_ValElem {
TypeTag tag;
int offset;
};
// This class groups together all of the vectors needed for run-time
// initialization. We gather them together into a single object so as
@ -57,7 +84,7 @@ public:
// index.
ValPtr ConstVals(int offset) const {
auto& cv = const_vals[offset];
return Consts(cv.first, cv.second);
return Consts(cv.tag, cv.offset);
}
// Retrieves the Zeek constant value for a particular Zeek type.
@ -157,9 +184,6 @@ protected:
// Pre-initialize all elements requiring it.
virtual void DoPreInits(InitsManager* im, const std::vector<int>& offsets_vec) {}
// Generate a single element.
virtual void GenerateElement(InitsManager* im, T2& init, int offset) {}
// The initialization vector in its entirety.
std::vector<T1>& inits_vec;
@ -221,16 +245,16 @@ using ValElemVecVec = std::vector<ValElemVec>;
template<class T>
class CPP_IndexedInits : public CPP_AbstractInits<T, ValElemVecVec> {
public:
CPP_IndexedInits(std::vector<T>& _inits_vec, int _offsets_set, std::vector<ValElemVecVec> _inits)
: CPP_AbstractInits<T, ValElemVecVec>(_inits_vec, _offsets_set, std::move(_inits)) {}
CPP_IndexedInits(std::vector<T>& _inits_vec, int _offsets_set, int* raw_inits)
: CPP_AbstractInits<T, ValElemVecVec>(_inits_vec, _offsets_set, generate_indices_set(raw_inits)) {}
protected:
void InitializeCohortWithOffsets(InitsManager* im, int cohort, const std::vector<int>& cohort_offsets) override;
// Note, in the following we pass in the inits_vec, even though
// the method will have direct access to it, because we want to
// use overloading to dispatch to custom generation for different
// types of values.
// Note, in the following we pass in the inits_vec ("ivec"), even though
// the method will have direct access to it, because we want to use
// overloading to dispatch to custom generation for different types of
// values.
void Generate(InitsManager* im, std::vector<EnumValPtr>& ivec, int offset, ValElemVec& init_vals);
void Generate(InitsManager* im, std::vector<StringValPtr>& ivec, int offset, ValElemVec& init_vals);
void Generate(InitsManager* im, std::vector<PatternValPtr>& ivec, int offset, ValElemVec& init_vals);
@ -254,8 +278,8 @@ protected:
// on subclasses of TypePtr.
class CPP_TypeInits : public CPP_IndexedInits<TypePtr> {
public:
CPP_TypeInits(std::vector<TypePtr>& _inits_vec, int _offsets_set, std::vector<std::vector<ValElemVec>> _inits)
: CPP_IndexedInits<TypePtr>(_inits_vec, _offsets_set, _inits) {}
CPP_TypeInits(std::vector<TypePtr>& _inits_vec, int _offsets_set, int* raw_inits)
: CPP_IndexedInits<TypePtr>(_inits_vec, _offsets_set, raw_inits) {}
protected:
void DoPreInits(InitsManager* im, const std::vector<int>& offsets_vec) override;
@ -504,11 +528,4 @@ struct CPP_RegisterBody {
std::vector<std::string> events;
};
// Helper function that takes a (large) array of int's and from them
// constructs the corresponding vector-of-vector-of-indices. Each
// vector-of-indices is represented first by an int specifying its
// size, and then that many int's for its values. We recognize the
// end of the array upon encountering a "size" entry of -1.
extern void generate_indices_set(int* inits, std::vector<std::vector<int>>& indices_set);
} // namespace zeek::detail

10
src/script_opt/CPP/RuntimeOps.cc
View file

@ -91,7 +91,7 @@ ValPtr when_index_slice__CPP(VectorVal* vec, const ListVal* lv) {
return v;
}
ValPtr when_invoke__CPP(Func* f, std::vector<ValPtr> args, Frame* frame, void* caller_addr) {
ValPtr when_invoke__CPP(Func* f, ValVec args, Frame* frame, void* caller_addr) {
auto trigger = frame->GetTrigger();
if ( trigger ) {
@ -194,11 +194,7 @@ void remove_element__CPP(TableValPtr aggr, ListValPtr indices) {
check_iterators__CPP(iterators_invalidated);
}
// A helper function that takes a parallel vectors of attribute tags
// and values and returns a collective AttributesPtr corresponding to
// those instantiated attributes. For attributes that don't have
// associated expressions, the corresponding value should be nil.
static AttributesPtr build_attrs__CPP(vector<int> attr_tags, vector<ValPtr> attr_vals) {
AttributesPtr build_attrs__CPP(IntVec attr_tags, vector<ValPtr> attr_vals) {
vector<AttrPtr> attrs;
int nattrs = attr_tags.size();
for ( auto i = 0; i < nattrs; ++i ) {
@ -243,7 +239,7 @@ TableValPtr table_constructor__CPP(vector<ValPtr> indices, vector<ValPtr> vals,
return aggr;
}
void assign_attrs__CPP(IDPtr id, std::vector<int> attr_tags, std::vector<ValPtr> attr_vals) {
void assign_attrs__CPP(IDPtr id, IntVec attr_tags, ValVec attr_vals) {
id->SetAttrs(build_attrs__CPP(std::move(attr_tags), std::move(attr_vals)));
}

55
src/script_opt/CPP/RuntimeOps.h
View file

@ -10,13 +10,24 @@
namespace zeek {
using IntVec = std::vector<int>;
using ValVec = std::vector<ValPtr>;
using SubNetValPtr = IntrusivePtr<zeek::SubNetVal>;
namespace detail {
class CPPRuntime {
public:
static auto RawOptField(const RecordValPtr& rv, int field) { return rv->RawOptField(field); }
static auto& RawField(const RecordValPtr& rv, int field) { return rv->RawField(field); }
static auto& RawField(RecordVal* rv, int field) { return rv->RawField(field); }
static auto& RawOptField(const RecordValPtr& rv, int field) { return rv->RawOptField(field); }
static auto& RawOptField(RecordVal* rv, int field) { return rv->RawOptField(field); }
static const auto& GetCreationInits(const RecordType* rt) { return rt->CreationInits(); }
static RecordVal* BuildRecordVal(RecordTypePtr t, std::vector<std::optional<ZVal>> init_vals) {
return new RecordVal(std::move(t), std::move(init_vals));
}
};
// Returns the concatenation of the given strings.
@ -27,21 +38,21 @@ extern bool str_in__CPP(const String* s1, const String* s2);
// Converts a vector of individual ValPtr's into a single ListValPtr
// suitable for indexing an aggregate.
extern ListValPtr index_val__CPP(std::vector<ValPtr> indices);
extern ListValPtr index_val__CPP(ValVec indices);
// Returns the value corresponding to indexing the given table/vector/string
// with the given set of indices. These are functions rather than something
// generated directly so that they can package up the error handling for
// the case where there's no such index. "patstr" refers to indexing a
// table[pattern] of X with a string value.
extern ValPtr index_table__CPP(const TableValPtr& t, std::vector<ValPtr> indices);
extern ValPtr index_patstr_table__CPP(const TableValPtr& t, std::vector<ValPtr> indices);
extern ValPtr index_table__CPP(const TableValPtr& t, ValVec indices);
extern ValPtr index_patstr_table__CPP(const TableValPtr& t, ValVec indices);
extern ValPtr index_vec__CPP(const VectorValPtr& vec, int index);
extern ValPtr index_string__CPP(const StringValPtr& svp, std::vector<ValPtr> indices);
extern ValPtr index_string__CPP(const StringValPtr& svp, ValVec indices);
// The same, but for indexing happening inside a "when" clause.
extern ValPtr when_index_table__CPP(const TableValPtr& t, std::vector<ValPtr> indices);
extern ValPtr when_index_patstr__CPP(const TableValPtr& t, std::vector<ValPtr> indices);
extern ValPtr when_index_table__CPP(const TableValPtr& t, ValVec indices);
extern ValPtr when_index_patstr__CPP(const TableValPtr& t, ValVec indices);
extern ValPtr when_index_vec__CPP(const VectorValPtr& vec, int index);
// For vector slices, we use the existing index_slice(), but we need a
@ -50,7 +61,7 @@ extern ValPtr when_index_slice__CPP(VectorVal* vec, const ListVal* lv);
// Calls out to the given script or BiF function, which does not return
// a value.
inline ValPtr invoke_void__CPP(Func* f, std::vector<ValPtr> args, Frame* frame) { return f->Invoke(&args, frame); }
inline ValPtr invoke_void__CPP(Func* f, ValVec args, Frame* frame) { return f->Invoke(&args, frame); }
// Used for error propagation by failed calls.
class CPPInterpreterException : public InterpreterException {};
@ -58,7 +69,7 @@ class CPPInterpreterException : public InterpreterException {};
// Calls out to the given script or BiF function. A separate function because
// of the need to (1) construct the "args" vector using {} initializers,
// but (2) needing to have the address of that vector.
inline ValPtr invoke__CPP(Func* f, std::vector<ValPtr> args, Frame* frame) {
inline ValPtr invoke__CPP(Func* f, ValVec args, Frame* frame) {
auto v = f->Invoke(&args, frame);
if ( ! v )
throw CPPInterpreterException();
@ -71,7 +82,7 @@ inline ValPtr invoke__CPP(Func* f, std::vector<ValPtr> args, Frame* frame) {
// last argument is the address of the calling function; we just need
// it to be distinct to the call, so we can associate a Trigger cache
// with it.
extern ValPtr when_invoke__CPP(Func* f, std::vector<ValPtr> args, Frame* frame, void* caller_addr);
extern ValPtr when_invoke__CPP(Func* f, ValVec args, Frame* frame, void* caller_addr);
// Thrown when a call inside a "when" delays.
class CPPDelayedCallException : public InterpreterException {};
@ -201,29 +212,35 @@ inline VectorValPtr vector_coerce__CPP(const ValPtr& v, const TypePtr& t) {
return make_intrusive<VectorVal>(cast_intrusive<VectorType>(t));
}
// Takes parallel vectors of attribute tags and values and returns a
// collective AttributesPtr corresponding to those instantiated attributes.
// For attributes that don't have associated expressions, the corresponding
// value should be nil.
extern AttributesPtr build_attrs__CPP(IntVec attr_tags, std::vector<ValPtr> attr_vals);
// Constructs a set of the given type, containing the given elements, and
// with the associated attributes.
extern TableValPtr set_constructor__CPP(std::vector<ValPtr> elements, TableTypePtr t, std::vector<int> attr_tags,
std::vector<ValPtr> attr_vals);
extern TableValPtr set_constructor__CPP(ValVec elements, TableTypePtr t, IntVec attr_tags, ValVec attr_vals);
// Constructs a table of the given type, containing the given elements
// (specified as parallel index/value vectors), and with the associated
// attributes.
extern TableValPtr table_constructor__CPP(std::vector<ValPtr> indices, std::vector<ValPtr> vals, TableTypePtr t,
std::vector<int> attr_tags, std::vector<ValPtr> attr_vals);
extern TableValPtr table_constructor__CPP(ValVec indices, ValVec vals, TableTypePtr t, IntVec attr_tags,
ValVec attr_vals);
// Assigns a set of attributes to an identifier.
extern void assign_attrs__CPP(IDPtr id, std::vector<int> attr_tags, std::vector<ValPtr> attr_vals);
extern void assign_attrs__CPP(IDPtr id, IntVec attr_tags, ValVec attr_vals);
// Constructs a record of the given type, whose (ordered) fields are
// assigned to the corresponding elements of the given vector of values.
extern RecordValPtr record_constructor__CPP(std::vector<ValPtr> vals, RecordTypePtr t);
extern RecordValPtr record_constructor__CPP(ValVec vals, RecordTypePtr t);
// Same, but with a map when using a named constructor.
extern RecordValPtr record_constructor_map__CPP(std::vector<ValPtr> vals, std::vector<int> map, RecordTypePtr t);
extern RecordValPtr record_constructor_map__CPP(ValVec vals, IntVec map, RecordTypePtr t);
// Constructs a vector of the given type, populated with the given values.
extern VectorValPtr vector_constructor__CPP(std::vector<ValPtr> vals, VectorTypePtr t);
extern VectorValPtr vector_constructor__CPP(ValVec vals, VectorTypePtr t);
// For patterns, executes p1 += p2.
inline PatternValPtr re_append__CPP(const PatternValPtr& p1, const PatternValPtr& p2) {
@ -234,7 +251,7 @@ inline PatternValPtr re_append__CPP(const PatternValPtr& p1, const PatternValPtr
// Schedules an event to occur at the given absolute time, parameterized
// with the given set of values. A separate function to facilitate avoiding
// the scheduling if Zeek is terminating.
extern ValPtr schedule__CPP(double dt, EventHandlerPtr event, std::vector<ValPtr> args);
extern ValPtr schedule__CPP(double dt, EventHandlerPtr event, ValVec args);
// Simple helper functions for supporting absolute value.
inline zeek_uint_t iabs__CPP(zeek_int_t v) { return v < 0 ? -v : v; }

2
src/script_opt/CPP/RuntimeVec.cc
View file

@ -109,7 +109,7 @@ VEC_OP1(comp, ~, )
}
// Analogous to VEC_OP1, instantiates a function for a given binary operation,
// with customimzable kernels for "int" and "double" operations.
// with customizable kernels for "int" and "double" operations.
// This version is for operations whose result type is the same as the
// operand type.
#define VEC_OP2(name, op, int_kernel, double_kernel, zero_check, is_bool) \

26
src/script_opt/CPP/Stmts.cc
View file

@ -305,15 +305,22 @@ void CPPCompile::GenValueSwitchStmt(const Expr* e, const case_list* cases) {
void CPPCompile::GenWhenStmt(const WhenStmt* w) {
auto wi = w->Info();
auto wl = wi->Lambda();
if ( ! wl )
reporter->FatalError("cannot compile deprecated \"when\" statement");
vector<string> local_aggrs;
for ( auto& l : wi->WhenExprLocals() )
if ( IsAggr(l->GetType()) )
local_aggrs.push_back(IDNameStr(l.get()));
auto when_lambda = GenExpr(wi->Lambda(), GEN_NATIVE);
GenWhenStmt(wi.get(), when_lambda, w->GetLocationInfo(), std::move(local_aggrs));
}
void CPPCompile::GenWhenStmt(const WhenInfo* wi, const std::string& when_lambda, const Location* loc,
vector<string> local_aggrs) {
auto is_return = wi->IsReturn() ? "true" : "false";
auto timeout = wi->TimeoutExpr();
auto timeout_val = timeout ? GenExpr(timeout, GEN_NATIVE) : "-1.0";
auto loc = w->GetLocationInfo();
Emit("{ // begin a new scope for internal variables");
@ -331,17 +338,18 @@ void CPPCompile::GenWhenStmt(const WhenStmt* w) {
NL();
Emit("std::vector<ValPtr> CPP__local_aggrs;");
for ( auto& l : wi->WhenExprLocals() )
if ( IsAggr(l->GetType()) )
Emit("CPP__local_aggrs.emplace_back(%s);", IDNameStr(l.get()));
for ( auto& la : local_aggrs )
Emit("CPP__local_aggrs.emplace_back(%s);", la);
Emit("CPP__wi->Instantiate(%s);", GenExpr(wi->Lambda(), GEN_NATIVE));
Emit("CPP__wi->Instantiate(%s);", when_lambda);
// We need a new frame for the trigger to unambiguously associate
// with, in case we're called multiple times with our existing frame.
Emit("auto new_frame = make_intrusive<Frame>(0, nullptr, nullptr);");
Emit("auto curr_t = f__CPP->GetTrigger();");
Emit("auto curr_assoc = f__CPP->GetTriggerAssoc();");
if ( ! ret_type || ret_type->Tag() == TYPE_VOID )
Emit("// Note, the following works even if curr_t is nil.");
Emit("new_frame->SetTrigger({NewRef{}, curr_t});");
Emit("new_frame->SetTriggerAssoc(curr_assoc);");
@ -352,7 +360,7 @@ void CPPCompile::GenWhenStmt(const WhenStmt* w) {
if ( ret_type && ret_type->Tag() != TYPE_VOID ) {
// Note, ret_type can be active but we *still* don't have
// a return type, due to the faked-up "any" return type
// a return value, due to the faked-up "any" return type
// associated with "when" lambdas, so check for that case.
Emit("if ( curr_t )");
StartBlock();

10
src/script_opt/CPP/Util.cc
View file

@ -17,6 +17,16 @@ string Fmt(double d) {
if ( d == 0.0 && signbit(d) )
return "-0.0";
if ( isinf(d) ) {
string infty = "std::numeric_limits<double>::infinity()";
if ( d < 0.0 )
infty = "-" + infty;
return infty;
}
if ( isnan(d) )
return "std::numeric_limits<double>::quiet_NaN()";
// Unfortunately, to_string(double) is hardwired to use %f with
// default of 6 digits precision.
char buf[8192];

28
src/script_opt/CPP/Vars.cc
View file

@ -111,12 +111,32 @@ string CPPCompile::LocalName(const ID* l) const {
auto n = l->Name();
auto without_module = strstr(n, "::");
if ( without_module )
return Canonicalize(without_module + 2);
else
while ( without_module ) {
n = without_module + 2;
without_module = strstr(n, "::");
}
return Canonicalize(n);
}
string CPPCompile::CaptureName(const ID* l) const {
// We want to strip both the module and any inlining appendage.
auto n = l->Name();
auto without_module = strstr(n, "::");
while ( without_module ) {
n = without_module + 2;
without_module = strstr(n, "::");
}
auto appendage = strchr(n, '.');
if ( appendage )
return string(n, appendage - n) + "_";
return string(n) + "_";
}
string CPPCompile::Canonicalize(const char* name) const {
string cname;
@ -127,7 +147,7 @@ string CPPCompile::Canonicalize(const char* name) const {
if ( c == '<' || c == '>' )
continue;
if ( c == ':' || c == '-' )
if ( c == ':' || c == '-' || c == '.' )
c = '_';
cname += c;

10
src/script_opt/CSE.h
View file

@ -24,6 +24,13 @@ public:
TraversalCode PreExpr(const Expr*) override;
TraversalCode PostExpr(const Expr*) override;
TraversalCode PreType(const Type* t) override {
if ( types_seen.count(t) > 0 )
return TC_ABORTSTMT;
types_seen.insert(t);
return TC_CONTINUE;
}
// Returns the ultimate verdict re safety.
bool IsValid() const {
if ( ! is_valid )
@ -105,6 +112,9 @@ protected:
//
// A count to allow for nesting.
int in_aggr_mod_expr = 0;
// Used to limit traversal of recursive types.
std::unordered_set<const Type*> types_seen;
};
// Used for debugging, to communicate which expression wasn't

22
src/script_opt/Expr.cc
View file

@ -115,6 +115,9 @@ bool Expr::IsReducedConditional(Reducer* c) const {
return NonReduced(this);
if ( op1->Tag() == EXPR_LIST ) {
if ( ! op1->IsReduced(c) )
return NonReduced(this);
auto l1 = op1->AsListExpr();
auto& l1_e = l1->Exprs();
@ -472,6 +475,7 @@ ExprPtr UnaryExpr::Reduce(Reducer* c, StmtPtr& red_stmt) {
auto op_val = op->FoldVal();
if ( op_val ) {
auto fold = Fold(op_val.get());
if ( fold->GetType()->Tag() != TYPE_OPAQUE )
return TransformMe(make_intrusive<ConstExpr>(fold), c, red_stmt);
}
@ -520,6 +524,7 @@ ExprPtr BinaryExpr::Reduce(Reducer* c, StmtPtr& red_stmt) {
auto op2_fold_val = op2->FoldVal();
if ( op1_fold_val && op2_fold_val ) {
auto fold = Fold(op1_fold_val.get(), op2_fold_val.get());
if ( fold->GetType()->Tag() != TYPE_OPAQUE )
return TransformMe(make_intrusive<ConstExpr>(fold), c, red_stmt);
}
@ -3109,6 +3114,23 @@ CoerceToAnyExpr::CoerceToAnyExpr(ExprPtr arg_op) : UnaryExpr(EXPR_TO_ANY_COERCE,
type = base_type(TYPE_ANY);
}
bool CoerceToAnyExpr::IsReduced(Reducer* c) const { return HasReducedOps(c); }
ExprPtr CoerceToAnyExpr::Reduce(Reducer* c, StmtPtr& red_stmt) {
if ( c->Optimizing() )
op = c->UpdateExpr(op);
red_stmt = nullptr;
if ( ! op->IsSingleton(c) )
op = op->ReduceToSingleton(c, red_stmt);
if ( c->Optimizing() )
return ThisPtr();
else
return AssignToTemporary(c, red_stmt);
}
ValPtr CoerceToAnyExpr::Fold(Val* v) const { return {NewRef{}, v}; }
ExprPtr CoerceToAnyExpr::Duplicate() { return SetSucc(new CoerceToAnyExpr(op->Duplicate())); }

78
src/script_opt/ProfileFunc.cc
View file

@ -24,11 +24,12 @@ p_hash_type p_hash(const Obj* o) {
ProfileFunc::ProfileFunc(const Func* func, const StmtPtr& body, bool _abs_rec_fields) {
profiled_func = func;
profiled_scope = profiled_func->GetScope();
profiled_body = body.get();
abs_rec_fields = _abs_rec_fields;
auto ft = func->GetType()->AsFuncType();
auto& fcaps = ft->GetCaptures();
profiled_func_t = cast_intrusive<FuncType>(func->GetType());
auto& fcaps = profiled_func_t->GetCaptures();
if ( fcaps ) {
int offset = 0;
@ -40,7 +41,7 @@ ProfileFunc::ProfileFunc(const Func* func, const StmtPtr& body, bool _abs_rec_fi
}
}
Profile(ft, body);
Profile(profiled_func_t.get(), body);
}
ProfileFunc::ProfileFunc(const Stmt* s, bool _abs_rec_fields) {
@ -56,6 +57,9 @@ ProfileFunc::ProfileFunc(const Expr* e, bool _abs_rec_fields) {
if ( e->Tag() == EXPR_LAMBDA ) {
auto func = e->AsLambdaExpr();
ASSERT(func->GetType()->Tag() == TYPE_FUNC);
profiled_scope = func->GetScope();
profiled_func_t = cast_intrusive<FuncType>(func->GetType());
int offset = 0;
@ -75,6 +79,13 @@ ProfileFunc::ProfileFunc(const Expr* e, bool _abs_rec_fields) {
void ProfileFunc::Profile(const FuncType* ft, const StmtPtr& body) {
num_params = ft->Params()->NumFields();
assert(profiled_scope != nullptr);
auto& ov = profiled_scope->OrderedVars();
for ( int i = 0; i < num_params; ++i )
params.insert(ov[i].get());
TrackType(ft);
body->Traverse(this);
}
@ -181,28 +192,10 @@ TraversalCode ProfileFunc::PreExpr(const Expr* e) {
TrackType(id->GetType());
if ( id->IsGlobal() ) {
globals.insert(id);
all_globals.insert(id);
const auto& t = id->GetType();
if ( t->Tag() == TYPE_FUNC )
if ( t->AsFuncType()->Flavor() == FUNC_FLAVOR_EVENT )
events.insert(id->Name());
PreID(id);
break;
}
// This is a tad ugly. Unfortunately due to the weird way
// that Zeek function *declarations* work, there's no reliable
// way to get the list of parameters for a function *definition*,
// since they can have different names than what's present in the
// declaration. So we identify them directly, by knowing that
// they come at the beginning of the frame ... and being careful
// to avoid misconfusing a lambda capture with a low frame offset
// as a parameter.
if ( captures.count(id) == 0 && id->Offset() < num_params )
params.insert(id);
locals.insert(id);
break;
@ -426,11 +419,6 @@ TraversalCode ProfileFunc::PreExpr(const Expr* e) {
for ( const auto& i : l->OuterIDs() ) {
locals.insert(i);
TrackID(i);
// See above re EXPR_NAME regarding the following
// logic.
if ( captures.count(i) == 0 && i->Offset() < num_params )
params.insert(i);
}
// In general, we don't want to recurse into the body.
@ -487,10 +475,29 @@ TraversalCode ProfileFunc::PreExpr(const Expr* e) {
TraversalCode ProfileFunc::PreID(const ID* id) {
TrackID(id);
if ( id->IsGlobal() ) {
globals.insert(id);
all_globals.insert(id);
const auto& t = id->GetType();
TrackType(t);
if ( t->Tag() == TYPE_FUNC )
if ( t->AsFuncType()->Flavor() == FUNC_FLAVOR_EVENT )
events.insert(id->Name());
}
// There's no need for any further analysis of this ID.
return TC_ABORTSTMT;
}
TraversalCode ProfileFunc::PreType(const Type* t) {
TrackType(t);
// There's no need for any further analysis of this type.
return TC_ABORTSTMT;
}
void ProfileFunc::TrackType(const Type* t) {
if ( ! t )
return;
@ -514,6 +521,11 @@ void ProfileFunc::TrackID(const ID* id) {
// Already tracked.
return;
if ( id->IsGlobal() ) {
globals.insert(id);
all_globals.insert(id);
}
ordered_ids.push_back(id);
}
@ -546,7 +558,9 @@ void ProfileFunc::CheckRecordConstructor(TypePtr t) {
}
}
ProfileFuncs::ProfileFuncs(std::vector<FuncInfo>& funcs, is_compilable_pred pred, bool _full_record_hashes) {
ProfileFuncs::ProfileFuncs(std::vector<FuncInfo>& funcs, is_compilable_pred pred, bool _compute_func_hashes,
bool _full_record_hashes) {
compute_func_hashes = _compute_func_hashes;
full_record_hashes = _full_record_hashes;
for ( auto& f : funcs ) {
@ -558,6 +572,11 @@ ProfileFuncs::ProfileFuncs(std::vector<FuncInfo>& funcs, is_compilable_pred pred
// Track the profile even if we're not compiling the function, since
// the AST optimizer will still need it to reason about function-call
// side effects.
// Propagate previous hash if requested.
if ( ! compute_func_hashes && f.Profile() )
pf->SetHashVal(f.Profile()->HashVal());
f.SetProfile(std::move(pf));
func_profs[f.Func()] = f.ProfilePtr();
}
@ -805,6 +824,7 @@ void ProfileFuncs::ComputeTypeHashes(const std::vector<const Type*>& types) {
}
void ProfileFuncs::ComputeBodyHashes(std::vector<FuncInfo>& funcs) {
if ( compute_func_hashes )
for ( auto& f : funcs )
if ( ! f.ShouldSkip() )
ComputeProfileHash(f.ProfilePtr());
@ -813,6 +833,8 @@ void ProfileFuncs::ComputeBodyHashes(std::vector<FuncInfo>& funcs) {
auto pf = ExprProf(l);
func_profs[l->PrimaryFunc().get()] = pf;
lambda_primaries[l->Name()] = l->PrimaryFunc().get();
if ( compute_func_hashes )
ComputeProfileHash(pf);
}
}

23
src/script_opt/ProfileFunc.h
View file

@ -83,6 +83,7 @@ public:
// Returns the function, body, or expression profiled. Each can be
// null depending on the constructor used.
const Func* ProfiledFunc() const { return profiled_func; }
const ScopePtr& ProfiledScope() const { return profiled_scope; }
const Stmt* ProfiledBody() const { return profiled_body; }
const Expr* ProfiledExpr() const { return profiled_expr; }
@ -139,6 +140,7 @@ protected:
TraversalCode PreStmt(const Stmt*) override;
TraversalCode PreExpr(const Expr*) override;
TraversalCode PreID(const ID*) override;
TraversalCode PreType(const Type*) override;
// Take note of the presence of a given type.
void TrackType(const Type* t);
@ -157,6 +159,8 @@ protected:
// The function, body, or expression profiled. Can be null
// depending on which constructor was used.
const Func* profiled_func = nullptr;
ScopePtr profiled_scope; // null when not in a full function context
FuncTypePtr profiled_func_t; // null when not in a full function context
const Stmt* profiled_body = nullptr;
const Expr* profiled_expr = nullptr;
@ -347,13 +351,15 @@ using is_compilable_pred = bool (*)(const ProfileFunc*, const char** reason);
// Collectively profile an entire collection of functions.
class ProfileFuncs {
public:
// Updates entries in "funcs" to include profiles. If pred is
// non-nil, then it is called for each profile to see whether it's
// compilable, and, if not, the FuncInfo is marked as ShouldSkip().
// "full_record_hashes" controls whether the hashes for extended
// records covers their final, full form, or should only their
// original fields.
ProfileFuncs(std::vector<FuncInfo>& funcs, is_compilable_pred pred, bool full_record_hashes);
// Updates entries in "funcs" to include profiles. If pred is non-nil,
// then it is called for each profile to see whether it's compilable,
// and, if not, the FuncInfo is marked as ShouldSkip().
// "compute_func_hashes" governs whether we compute hashes for the
// FuncInfo entries, or keep their existing ones. "full_record_hashes"
// controls whether the hashes for extended records covers their final,
// full form, or should only their original fields.
ProfileFuncs(std::vector<FuncInfo>& funcs, is_compilable_pred pred, bool compute_func_hashes,
bool full_record_hashes);
// The following accessors provide a global profile across all of
// the (non-skipped) functions in "funcs". See the comments for
@ -604,6 +610,9 @@ protected:
// These can arise for example due to lambdas or record attributes.
std::vector<const Expr*> pending_exprs;
// Whether to compute new hashes for the FuncInfo entries.
bool compute_func_hashes;
// Whether the hashes for extended records should cover their final,
// full form, or only their original fields.
bool full_record_hashes;

49
src/script_opt/ScriptOpt.cc
View file

@ -66,12 +66,14 @@ void analyze_global_stmts(Stmt* stmts) {
auto id = install_ID("<global-stmts>", GLOBAL_MODULE_NAME, true, false);
auto empty_args_t = make_intrusive<RecordType>(nullptr);
auto func_t = make_intrusive<FuncType>(empty_args_t, nullptr, FUNC_FLAVOR_FUNCTION);
func_t->SetName("<global-stmts>");
id->SetType(func_t);
auto sc = current_scope();
std::vector<IDPtr> empty_inits;
global_stmts = make_intrusive<ScriptFunc>(id);
global_stmts->AddBody(stmts->ThisPtr(), empty_inits, sc->Length());
global_stmts->SetScope(sc);
global_stmts_ind = funcs.size();
funcs.emplace_back(global_stmts, sc, stmts->ThisPtr(), 0);
@ -271,6 +273,7 @@ static void init_options() {
check_env_opt("ZEEK_REPORT_UNCOMPILABLE", analysis_options.report_uncompilable);
check_env_opt("ZEEK_ZAM_CODE", analysis_options.gen_ZAM_code);
check_env_opt("ZEEK_NO_ZAM_OPT", analysis_options.no_ZAM_opt);
check_env_opt("ZEEK_NO_ZAM_CONTROL_FLOW_OPT", analysis_options.no_ZAM_control_flow_opt);
check_env_opt("ZEEK_DUMP_ZAM", analysis_options.dump_ZAM);
check_env_opt("ZEEK_PROFILE", analysis_options.profile_ZAM);
@ -391,7 +394,7 @@ static void use_CPP() {
int num_used = 0;
auto pfs = std::make_unique<ProfileFuncs>(funcs, is_CPP_compilable, false);
auto pfs = std::make_unique<ProfileFuncs>(funcs, is_CPP_compilable, true, false);
for ( auto& f : funcs ) {
auto hash = f.Profile()->HashVal();
@ -401,7 +404,6 @@ static void use_CPP() {
++num_used;
auto b = s->second.body;
b->SetHash(hash);
// We may have already updated the body if
// we're using code compiled for standalone.
@ -430,18 +432,16 @@ static void use_CPP() {
reporter->FatalError("no C++ functions found to use");
}
static void generate_CPP() {
static void generate_CPP(std::shared_ptr<ProfileFuncs> pfs) {
const auto gen_name = CPP_dir + "CPP-gen.cc";
const bool standalone = analysis_options.gen_standalone_CPP;
const bool report = analysis_options.report_uncompilable;
auto pfs = std::make_shared<ProfileFuncs>(funcs, is_CPP_compilable, false);
CPPCompile cpp(funcs, pfs, gen_name, standalone, report);
}
static void analyze_scripts_for_ZAM() {
static void analyze_scripts_for_ZAM(std::shared_ptr<ProfileFuncs> pfs) {
if ( analysis_options.usage_issues > 0 && analysis_options.optimize_AST ) {
fprintf(stderr,
"warning: \"-O optimize-AST\" option is incompatible with -u option, "
@ -449,8 +449,6 @@ static void analyze_scripts_for_ZAM() {
analysis_options.optimize_AST = false;
}
auto pfs = std::make_shared<ProfileFuncs>(funcs, nullptr, true);
if ( analysis_options.profile_ZAM ) {
#ifdef ENABLE_ZAM_PROFILE
AST_blocks = std::make_unique<ASTBlockAnalyzer>(funcs);
@ -506,12 +504,12 @@ static void analyze_scripts_for_ZAM() {
if ( ! analysis_options.compile_all && ! is_lambda && inl && inl->WasFullyInlined(func.get()) &&
func_used_indirectly.count(func.get()) == 0 ) {
// No need to compile as it won't be called directly.
// We'd like to zero out the body to recover the
// memory, but a *few* such functions do get called,
// such as by the event engine reaching up, or
// BiFs looking for them, so we can't safely zero
// them.
// No need to compile as it won't be called directly. We'd
// like to zero out the body to recover the memory, but a *few*
// such functions do get called, such as by the event engine
// reaching up, or BiFs looking for them, so we can't safely
// zero them.
f.SetSkip(true);
continue;
}
@ -532,6 +530,9 @@ static void analyze_scripts_for_ZAM() {
}
void clear_script_analysis() {
if ( analysis_options.gen_CPP )
return;
IDOptInfo::ClearGlobalInitExprs();
// We need to explicitly clear out the optimization information
@ -556,6 +557,11 @@ void clear_script_analysis() {
void analyze_scripts(bool no_unused_warnings) {
init_options();
if ( analysis_options.validate_ZAM ) {
validate_ZAM_insts();
return;
}
// Any standalone compiled scripts have already been instantiated
// at this point, but may require post-loading-of-scripts finalization.
for ( auto cb : standalone_finalizations )
@ -599,6 +605,7 @@ void analyze_scripts(bool no_unused_warnings) {
}
if ( analysis_options.report_CPP ) {
auto pfs = std::make_unique<ProfileFuncs>(funcs, is_CPP_compilable, true, false);
report_CPP();
exit(0);
}
@ -606,17 +613,23 @@ void analyze_scripts(bool no_unused_warnings) {
if ( analysis_options.use_CPP )
use_CPP();
std::shared_ptr<ProfileFuncs> pfs;
// Note, in the following it's not clear whether the final argument
// for absolute/relative record fields matters any more ...
if ( generating_CPP )
pfs = std::make_shared<ProfileFuncs>(funcs, is_CPP_compilable, true, false);
else
pfs = std::make_shared<ProfileFuncs>(funcs, nullptr, true, true);
if ( generating_CPP ) {
if ( analysis_options.gen_ZAM )
reporter->FatalError("-O ZAM and -O gen-C++ conflict");
generate_CPP();
generate_CPP(pfs);
exit(0);
}
// At this point we're done with C++ considerations, so instead
// are compiling to ZAM.
analyze_scripts_for_ZAM();
analyze_scripts_for_ZAM(pfs);
if ( reporter->Errors() > 0 )
reporter->FatalError("Optimized script execution aborted due to errors");

12
src/script_opt/ScriptOpt.h
View file

@ -61,6 +61,10 @@ struct AnalyOpt {
// recursive, and exit. Only germane if running the inliner.
bool report_recursive = false;
// If true, assess the instructions generated from ZAM templates
// for validity, and exit.
bool validate_ZAM = false;
// If true, generate ZAM code for applicable function bodies,
// activating all optimizations.
bool gen_ZAM = false;
@ -72,6 +76,9 @@ struct AnalyOpt {
// Deactivate the low-level ZAM optimizer.
bool no_ZAM_opt = false;
// Deactivate ZAM optimization of control flow.
bool no_ZAM_control_flow_opt = false;
// Produce a profile of ZAM execution.
bool profile_ZAM = false;
@ -241,6 +248,11 @@ extern bool should_analyze(const ScriptFuncPtr& f, const StmtPtr& body);
// suppressed by the flag) and optimization.
extern void analyze_scripts(bool no_unused_warnings);
// Conduct internal validation of ZAM instructions. Upon success, generates
// a terse report to stdout. Exits with an internal error if a problem is
// encountered.
extern void validate_ZAM_insts();
// Called when all script processing is complete and we can discard
// unused ASTs and associated state.
extern void clear_script_analysis();

144
src/script_opt/ZAM/AM-Opt.cc
View file

@ -144,6 +144,9 @@ bool ZAMCompiler::RemoveDeadCode() {
if ( ! i0->live )
continue;
if ( analysis_options.no_ZAM_control_flow_opt )
continue;
auto i1 = NextLiveInst(i0);
// Look for degenerate branches.
@ -181,6 +184,9 @@ bool ZAMCompiler::RemoveDeadCode() {
}
bool ZAMCompiler::CollapseGoTos() {
if ( analysis_options.no_ZAM_control_flow_opt )
return false;
bool did_change = false;
for ( auto& i0 : insts1 ) {
@ -303,7 +309,7 @@ bool ZAMCompiler::PruneUnused() {
if ( assignmentless_op.count(inst->op) == 0 )
reporter->InternalError("inconsistency in re-flavoring instruction with side effects");
inst->op_type = assignmentless_op_type[inst->op];
inst->op_type = assignmentless_op_class[inst->op];
inst->op = assignmentless_op[inst->op];
inst->v1 = inst->v2;
@ -336,8 +342,8 @@ void ZAMCompiler::ComputeFrameLifetimes() {
// Some special-casing.
switch ( inst->op ) {
case OP_NEXT_TABLE_ITER_VV:
case OP_NEXT_TABLE_ITER_VAL_VAR_VVV: {
case OP_NEXT_TABLE_ITER_fb:
case OP_NEXT_TABLE_ITER_VAL_VAR_Vfb: {
// These assign to an arbitrary long list of variables.
auto& iter_vars = inst->aux->loop_vars;
auto depth = inst->loop_depth;
@ -361,21 +367,21 @@ void ZAMCompiler::ComputeFrameLifetimes() {
}
// No need to check the additional "var" associated
// with OP_NEXT_TABLE_ITER_VAL_VAR_VVV as that's
// with OP_NEXT_TABLE_ITER_VAL_VAR_Vfb as that's
// a slot-1 assignment. However, similar to other
// loop variables, mark this as a usage.
if ( inst->op == OP_NEXT_TABLE_ITER_VAL_VAR_VVV )
if ( inst->op == OP_NEXT_TABLE_ITER_VAL_VAR_Vfb )
ExtendLifetime(inst->v1, EndOfLoop(inst, depth));
} break;
case OP_NEXT_TABLE_ITER_NO_VARS_VV: break;
case OP_NEXT_TABLE_ITER_NO_VARS_fb: break;
case OP_NEXT_TABLE_ITER_VAL_VAR_NO_VARS_VVV: {
case OP_NEXT_TABLE_ITER_VAL_VAR_NO_VARS_Vfb: {
auto depth = inst->loop_depth;
ExtendLifetime(inst->v1, EndOfLoop(inst, depth));
} break;
case OP_NEXT_VECTOR_ITER_VAL_VAR_VVVV: {
case OP_NEXT_VECTOR_ITER_VAL_VAR_VVsb: {
CheckSlotAssignment(inst->v2, inst);
auto depth = inst->loop_depth;
@ -383,13 +389,13 @@ void ZAMCompiler::ComputeFrameLifetimes() {
ExtendLifetime(inst->v2, EndOfLoop(inst, depth));
} break;
case OP_NEXT_VECTOR_BLANK_ITER_VAL_VAR_VVV: {
case OP_NEXT_VECTOR_BLANK_ITER_VAL_VAR_Vsb: {
auto depth = inst->loop_depth;
ExtendLifetime(inst->v1, EndOfLoop(inst, depth));
} break;
case OP_NEXT_VECTOR_ITER_VVV:
case OP_NEXT_STRING_ITER_VVV:
case OP_NEXT_VECTOR_ITER_Vsb:
case OP_NEXT_STRING_ITER_Vsb:
// Sometimes loops are written that don't actually
// use the iteration variable. However, we still
// need to mark the variable as having usage
@ -401,12 +407,12 @@ void ZAMCompiler::ComputeFrameLifetimes() {
ExtendLifetime(inst->v1, EndOfLoop(inst, inst->loop_depth));
break;
case OP_NEXT_VECTOR_BLANK_ITER_VV:
case OP_NEXT_STRING_BLANK_ITER_VV: break;
case OP_NEXT_VECTOR_BLANK_ITER_sb:
case OP_NEXT_STRING_BLANK_ITER_sb: break;
case OP_INIT_TABLE_LOOP_VV:
case OP_INIT_VECTOR_LOOP_VV:
case OP_INIT_STRING_LOOP_VV: {
case OP_INIT_TABLE_LOOP_Vf:
case OP_INIT_VECTOR_LOOP_Vs:
case OP_INIT_STRING_LOOP_Vs: {
// For all of these, the scope of the aggregate being
// looped over is the entire loop, even if it doesn't
// directly appear in it, and not just the initializer.
@ -423,14 +429,30 @@ void ZAMCompiler::ComputeFrameLifetimes() {
continue;
}
case OP_STORE_GLOBAL_V: {
case OP_STORE_GLOBAL_g: {
// Use of the global goes to here.
auto slot = frame_layout1[globalsI[inst->v1].id.get()];
ExtendLifetime(slot, EndOfLoop(inst, 1));
break;
}
case OP_LAMBDA_VV: {
case OP_DETERMINE_TYPE_MATCH_VV: {
auto aux = inst->aux;
int n = aux->n;
for ( int i = 0; i < n; ++i ) {
auto slot_i = aux->elems[i].Slot();
if ( slot_i >= 0 ) {
CheckSlotAssignment(slot_i, inst);
// The variable gets used in the switch that
// immediately follows this instruction, hence
// "i + 1" in the following.
ExtendLifetime(slot_i, insts1[i + 1]);
}
}
break;
}
case OP_LAMBDA_Vi: {
auto aux = inst->aux;
int n = aux->n;
for ( int i = 0; i < n; ++i ) {
@ -486,8 +508,7 @@ void ZAMCompiler::ReMapFrame() {
auto vars = inst_beginnings[inst];
for ( auto v : vars ) {
// Don't remap variables whose values aren't actually
// used.
// Don't remap variables whose values aren't actually used.
int slot = frame_layout1[v];
if ( denizen_ending.count(slot) > 0 )
ReMapVar(v, slot, i);
@ -549,9 +570,15 @@ void ZAMCompiler::ReMapFrame() {
// Handle special cases.
switch ( inst->op ) {
case OP_NEXT_TABLE_ITER_VV:
case OP_NEXT_TABLE_ITER_VAL_VAR_VVV: {
// Rewrite iteration variables.
case OP_INIT_TABLE_LOOP_Vf:
case OP_NEXT_TABLE_ITER_fb:
case OP_NEXT_TABLE_ITER_VAL_VAR_Vfb: {
// Rewrite iteration variables. Strictly speaking we only
// need to do this for the INIT, not the NEXT, since the
// latter currently doesn't access the variables directly but
// instead uses pointers set up by the INIT. We do both types
// here, though, to keep things consistent and to help avoid
// surprises if the implementation changes in the future.
auto& iter_vars = inst->aux->loop_vars;
for ( auto& v : iter_vars ) {
if ( v < 0 )
@ -941,17 +968,76 @@ void ZAMCompiler::KillInst(zeek_uint_t i) {
}
}
if ( num_labels == 0 )
// No labels to propagate.
return;
ZInstI* succ = nullptr;
if ( num_labels > 0 ) {
for ( auto j = i + 1; j < insts1.size(); ++j ) {
auto succ = insts1[j];
if ( succ->live ) {
succ->num_labels += num_labels;
if ( insts1[j]->live ) {
succ = insts1[j];
break;
}
}
if ( succ )
succ->num_labels += num_labels;
}
// Look into propagating control flow info.
if ( inst->aux && ! inst->aux->cft.empty() ) {
auto& cft = inst->aux->cft;
if ( cft.count(CFT_ELSE) > 0 ) {
// Push forward unless this was the end of the block.
if ( cft.count(CFT_BLOCK_END) == 0 ) {
ASSERT(succ);
AddCFT(succ, CFT_ELSE);
}
else
// But if it *was* the end of the block, remove that block.
--cft[CFT_BLOCK_END];
}
if ( cft.count(CFT_BREAK) > 0 ) {
// ### Factor this with the following
// Propagate breaks backwards.
int j = i;
while ( --j >= 0 )
if ( insts1[j]->live )
break;
ASSERT(j >= 0);
// Make sure the CFT entry is created.
AddCFT(insts1[j], CFT_BREAK);
auto be_cnt = cft[CFT_BREAK];
--be_cnt; // we already did one above
insts1[j]->aux->cft[CFT_BREAK] += be_cnt;
}
if ( cft.count(CFT_BLOCK_END) > 0 ) {
// Propagate block-ends backwards.
int j = i;
while ( --j >= 0 )
if ( insts1[j]->live )
break;
ASSERT(j >= 0);
// Make sure the CFT entry is created.
AddCFT(insts1[j], CFT_BLOCK_END);
auto be_cnt = cft[CFT_BLOCK_END];
--be_cnt; // we already did one above
insts1[j]->aux->cft[CFT_BLOCK_END] += be_cnt;
}
// If's can be killed because their bodies become empty,
// break's because they just lead to their following instruction,
// and next's if they become dead code.
// However, loop's and next's should not be killed.
ASSERT(cft.count(CFT_LOOP) == 0);
ASSERT(cft.count(CFT_LOOP_COND) == 0);
}
}
void ZAMCompiler::KillInsts(zeek_uint_t i) {

11
src/script_opt/ZAM/Branches.cc
View file

@ -10,9 +10,12 @@ namespace zeek::detail {
void ZAMCompiler::PushGoTos(GoToSets& gotos) { gotos.emplace_back(); }
void ZAMCompiler::ResolveGoTos(GoToSets& gotos, const InstLabel l) {
for ( auto& gi : gotos.back() )
void ZAMCompiler::ResolveGoTos(GoToSets& gotos, const InstLabel l, ControlFlowType cft) {
for ( auto& gi : gotos.back() ) {
SetGoTo(gi, l);
if ( cft != CFT_NONE )
AddCFT(insts1[gi.stmt_num], cft);
}
gotos.pop_back();
}
@ -25,13 +28,13 @@ ZAMStmt ZAMCompiler::GenGoTo(GoToSet& v) {
}
ZAMStmt ZAMCompiler::GoToStub() {
ZInstI z(OP_GOTO_V, 0);
ZInstI z(OP_GOTO_b, 0);
z.op_type = OP_V_I1;
return AddInst(z);
}
ZAMStmt ZAMCompiler::GoTo(const InstLabel l) {
ZInstI inst(OP_GOTO_V, 0);
ZInstI inst(OP_GOTO_b, 0);
inst.target = l;
inst.target_slot = 1;
inst.op_type = OP_V_I1;

87
src/script_opt/ZAM/BuiltIn.cc
View file

@ -27,8 +27,10 @@ SimpleZBI::SimpleZBI(std::string name, ZOp _const_op, ZOp _op, bool _ret_val_mat
bool SimpleZBI::Build(ZAMCompiler* zam, const NameExpr* n, const ExprPList& args) const {
ZInstI z;
if ( nargs == 0 ) {
if ( n )
if ( n ) {
z = ZInstI(op, zam->Frame1Slot(n, OP1_WRITE));
z.is_managed = ZVal::IsManagedType(n->GetType());
}
else
z = ZInstI(op);
}
@ -59,12 +61,9 @@ bool SimpleZBI::Build(ZAMCompiler* zam, const NameExpr* n, const ExprPList& args
z.c = ZVal(args[0]->AsConstExpr()->ValuePtr(), t);
}
z.t = t;
z.SetType(t);
}
if ( n )
z.is_managed = ZVal::IsManagedType(n->GetType());
zam->AddInst(z);
return true;
@ -104,7 +103,7 @@ bool CondZBI::BuildCond(ZAMCompiler* zam, const ExprPList& args, int& branch_v)
auto a0_slot = zam->FrameSlot(a0->AsNameExpr());
z = ZInstI(cond_op, a0_slot, 0);
z.op_type = OP_VV_I2;
z.t = a0->GetType();
z.SetType(a0->GetType());
branch_v = 2;
}
@ -129,7 +128,7 @@ bool OptAssignZBI::Build(ZAMCompiler* zam, const NameExpr* n, const ExprPList& a
ASSERT(nargs == 1);
auto a0 = zam->FrameSlot(args[0]->AsNameExpr());
z = ZInstI(op2, a0);
z.t = args[0]->GetType();
z.SetType(args[0]->GetType());
}
zam->AddInst(z);
@ -145,7 +144,7 @@ bool CatZBI::Build(ZAMCompiler* zam, const NameExpr* n, const ExprPList& args) c
if ( args.empty() ) {
// Weird, but easy enough to support.
z = ZInstI(OP_CAT1_VC, nslot);
z.t = n->GetType();
z.SetType(n->GetType());
z.c = ZVal(val_mgr->EmptyString());
}
@ -168,18 +167,18 @@ bool CatZBI::Build(ZAMCompiler* zam, const NameExpr* n, const ExprPList& args) c
else if ( a0->GetType()->Tag() != TYPE_STRING ) {
if ( a0->Tag() == EXPR_NAME ) {
z = zam->GenInst(OP_CAT1FULL_VV, n, a0->AsNameExpr());
z.t = a0->GetType();
z.SetType(a0->GetType());
}
else {
z = ZInstI(OP_CAT1_VC, nslot);
z.t = n->GetType();
z.SetType(n->GetType());
z.c = ZVal(ZAM_val_cat(a0->AsConstExpr()->ValuePtr()));
}
}
else if ( a0->Tag() == EXPR_CONST ) {
z = zam->GenInst(OP_CAT1_VC, n, a0->AsConstExpr());
z.t = n->GetType();
z.SetType(n->GetType());
}
else
@ -388,12 +387,12 @@ bool MultiZBI::Build(ZAMCompiler* zam, const NameExpr* n, const ExprPList& args)
z.is_managed = ZVal::IsManagedType(n->GetType());
if ( ! consts.empty() ) {
z.t = consts[0]->GetType();
z.c = ZVal(consts[0], z.t);
z.SetType(consts[0]->GetType());
z.c = ZVal(consts[0], z.GetType());
}
if ( type_arg >= 0 && ! z.t )
z.t = args[type_arg]->GetType();
if ( type_arg >= 0 && ! z.GetType() )
z.SetType(args[type_arg]->GetType());
zam->AddInst(z);
@ -434,14 +433,14 @@ SimpleZBI sta_ZBI{"subnet_to_addr", OP_SUBNET_TO_ADDR_VV, 1};
SimpleZBI ttd_ZBI{"time_to_double", OP_TIME_TO_DOUBLE_VV, 1};
SimpleZBI tl_ZBI{"to_lower", OP_TO_LOWER_VV, 1};
CondZBI ce_ZBI{"connection_exists", OP_CONN_EXISTS_VV, OP_CONN_EXISTS_COND_VV, 1};
CondZBI iip_ZBI{"is_icmp_port", OP_IS_ICMP_PORT_VV, OP_IS_ICMP_PORT_COND_VV, 1};
CondZBI itp_ZBI{"is_tcp_port", OP_IS_TCP_PORT_VV, OP_IS_TCP_PORT_COND_VV, 1};
CondZBI iup_ZBI{"is_udp_port", OP_IS_UDP_PORT_VV, OP_IS_UDP_PORT_COND_VV, 1};
CondZBI iv4_ZBI{"is_v4_addr", OP_IS_V4_ADDR_VV, OP_IS_V4_ADDR_COND_VV, 1};
CondZBI iv6_ZBI{"is_v6_addr", OP_IS_V6_ADDR_VV, OP_IS_V6_ADDR_COND_VV, 1};
CondZBI rlt_ZBI{"reading_live_traffic", OP_READING_LIVE_TRAFFIC_V, OP_READING_LIVE_TRAFFIC_COND_V, 0};
CondZBI rt_ZBI{"reading_traces", OP_READING_TRACES_V, OP_READING_TRACES_COND_V, 0};
CondZBI ce_ZBI{"connection_exists", OP_CONN_EXISTS_VV, OP_CONN_EXISTS_COND_Vb, 1};
CondZBI iip_ZBI{"is_icmp_port", OP_IS_ICMP_PORT_VV, OP_IS_ICMP_PORT_COND_Vb, 1};
CondZBI itp_ZBI{"is_tcp_port", OP_IS_TCP_PORT_VV, OP_IS_TCP_PORT_COND_Vb, 1};
CondZBI iup_ZBI{"is_udp_port", OP_IS_UDP_PORT_VV, OP_IS_UDP_PORT_COND_Vb, 1};
CondZBI iv4_ZBI{"is_v4_addr", OP_IS_V4_ADDR_VV, OP_IS_V4_ADDR_COND_Vb, 1};
CondZBI iv6_ZBI{"is_v6_addr", OP_IS_V6_ADDR_VV, OP_IS_V6_ADDR_COND_Vb, 1};
CondZBI rlt_ZBI{"reading_live_traffic", OP_READING_LIVE_TRAFFIC_V, OP_READING_LIVE_TRAFFIC_COND_b, 0};
CondZBI rt_ZBI{"reading_traces", OP_READING_TRACES_V, OP_READING_TRACES_COND_b, 0};
// These have a different form to avoid invoking copy constructors.
auto cat_ZBI = CatZBI();
@ -468,48 +467,48 @@ OptAssignZBI rtc_ZBI{ "PacketAnalyzer::TEREDO::remove_teredo_connection",
MultiZBI faa_ZBI{ "Files::__add_analyzer",
{{{VVV}, {OP_FILES_ADD_ANALYZER_VVV, OP_VVV}},
{{VCV}, {OP_FILES_ADD_ANALYZER_ViV, OP_VVC}}},
{{VCV}, {OP_FILES_ADD_ANALYZER_VCV, OP_VVC}}},
{{{VVV}, {OP_FILES_ADD_ANALYZER_VVVV, OP_VVVV}},
{{VCV}, {OP_FILES_ADD_ANALYZER_VViV, OP_VVVC}}},
{{VCV}, {OP_FILES_ADD_ANALYZER_VVCV, OP_VVVC}}},
1
};
MultiZBI fra_ZBI{ "Files::__remove_analyzer",
{{{VVV}, {OP_FILES_REMOVE_ANALYZER_VVV, OP_VVV}},
{{VCV}, {OP_FILES_REMOVE_ANALYZER_ViV, OP_VVC}}},
{{VCV}, {OP_FILES_REMOVE_ANALYZER_VCV, OP_VVC}}},
{{{VVV}, {OP_FILES_REMOVE_ANALYZER_VVVV, OP_VVVV}},
{{VCV}, {OP_FILES_REMOVE_ANALYZER_VViV, OP_VVVC}}},
{{VCV}, {OP_FILES_REMOVE_ANALYZER_VVCV, OP_VVVC}}},
1
};
MultiZBI fsrb_ZBI{ "Files::__set_reassembly_buffer",
{{{VV}, {OP_FILES_SET_REASSEMBLY_BUFFER_VV, OP_VV}},
{{VC}, {OP_FILES_SET_REASSEMBLY_BUFFER_VC, OP_VV_I2}}},
{{VC}, {OP_FILES_SET_REASSEMBLY_BUFFER_Vi, OP_VV_I2}}},
{{{VV}, {OP_FILES_SET_REASSEMBLY_BUFFER_VVV, OP_VVV}},
{{VC}, {OP_FILES_SET_REASSEMBLY_BUFFER_VVC, OP_VVV_I3}}}
{{VC}, {OP_FILES_SET_REASSEMBLY_BUFFER_VVi, OP_VVV_I3}}}
};
MultiZBI lw_ZBI{ "Log::__write",
{{{VV}, {OP_LOG_WRITE_VV, OP_VV}},
{{CV}, {OP_LOG_WRITEC_V, OP_V}}},
{{CV}, {OP_LOG_WRITE_CV, OP_V}}},
{{{VV}, {OP_LOG_WRITE_VVV, OP_VVV}},
{{CV}, {OP_LOG_WRITEC_VV, OP_VV}}}
{{CV}, {OP_LOG_WRITEC_VCV, OP_VV}}}
};
MultiZBI gccbt_ZBI{ "get_current_conn_bytes_threshold", true,
{{{VV}, {OP_GET_BYTES_THRESH_VVV, OP_VVV}},
{{VC}, {OP_GET_BYTES_THRESH_VVi, OP_VVC}}}
{{VC}, {OP_GET_BYTES_THRESH_VVC, OP_VVC}}}
};
MultiZBI sccbt_ZBI{ "set_current_conn_bytes_threshold",
{{{VVV}, {OP_SET_BYTES_THRESH_VVV, OP_VVV}},
{{VVC}, {OP_SET_BYTES_THRESH_VVi, OP_VVC}},
{{VCV}, {OP_SET_BYTES_THRESH_ViV, OP_VVC}},
{{VCC}, {OP_SET_BYTES_THRESH_Vii, OP_VVC_I2}}},
{{VVC}, {OP_SET_BYTES_THRESH_VVC, OP_VVC}},
{{VCV}, {OP_SET_BYTES_THRESH_VCV, OP_VVC}},
{{VCC}, {OP_SET_BYTES_THRESH_VCi, OP_VVC_I2}}},
{{{VVV}, {OP_SET_BYTES_THRESH_VVVV, OP_VVVV}},
{{VVC}, {OP_SET_BYTES_THRESH_VVVi, OP_VVVC}},
{{VCV}, {OP_SET_BYTES_THRESH_VViV, OP_VVVC}},
{{VCC}, {OP_SET_BYTES_THRESH_VVii, OP_VVVC_I3}}}
{{VVC}, {OP_SET_BYTES_THRESH_VVVC, OP_VVVC}},
{{VCV}, {OP_SET_BYTES_THRESH_VVCV, OP_VVVC}},
{{VCC}, {OP_SET_BYTES_THRESH_VVCi, OP_VVVC_I3}}}
};
MultiZBI sw_ZBI{ "starts_with", true,
@ -532,12 +531,12 @@ MultiZBI strstr_ZBI{ "strstr", true,
MultiZBI sb_ZBI{ "sub_bytes", true,
{{{VVV}, {OP_SUB_BYTES_VVVV, OP_VVVV}},
{{VVC}, {OP_SUB_BYTES_VVVi, OP_VVVC}},
{{VCV}, {OP_SUB_BYTES_VViV, OP_VVVC}},
{{VCC}, {OP_SUB_BYTES_VVii, OP_VVVC_I3}},
{{CVV}, {OP_SUB_BYTES_VVVC, OP_VVVC}},
{{CVC}, {OP_SUB_BYTES_VViC, OP_VVVC_I3}},
{{CCV}, {OP_SUB_BYTES_ViVC, OP_VVVC_I3}}}
{{VVC}, {OP_SUB_BYTES_VVVC, OP_VVVC}},
{{VCV}, {OP_SUB_BYTES_VVCV, OP_VVVC}},
{{VCC}, {OP_SUB_BYTES_VVCi, OP_VVVC_I3}},
{{CVV}, {OP_SUB_BYTES_VCVV, OP_VVVC}},
{{CVC}, {OP_SUB_BYTES_VCVi, OP_VVVC_I3}},
{{CCV}, {OP_SUB_BYTES2_VCVi, OP_VVVC_I3}}}
};
// clang-format on

7
src/script_opt/ZAM/BuiltInSupport.cc
View file

@ -48,8 +48,7 @@ FixedCatArg::FixedCatArg(TypePtr _t) : t(std::move(_t)) {
}
}
void FixedCatArg::RenderInto(ZVal* zframe, int slot, char*& res) {
auto& z = zframe[slot];
void FixedCatArg::RenderInto(const ZVal& z, char*& res) {
int n;
const char* text;
std::string str;
@ -140,8 +139,8 @@ void FixedCatArg::RenderInto(ZVal* zframe, int slot, char*& res) {
}
}
size_t PatternCatArg::ComputeMaxSize(ZVal* zframe, int slot) {
text = zframe[slot].AsPattern()->AsPattern()->PatternText();
size_t PatternCatArg::ComputeMaxSize(const ZVal& zv) {
text = zv.AsPattern()->AsPattern()->PatternText();
n = strlen(text);
return n;
}

24
src/script_opt/ZAM/BuiltInSupport.h
View file

@ -13,9 +13,9 @@ public:
virtual ~CatArg() {}
size_t MaxSize(ZVal* zframe, int slot) { return max_size ? *max_size : ComputeMaxSize(zframe, slot); }
size_t MaxSize(const ZVal& zv) { return max_size ? *max_size : ComputeMaxSize(zv); }
virtual void RenderInto(ZVal* zframe, int slot, char*& res) {
virtual void RenderInto(const ZVal& zv, char*& res) {
auto n = *max_size;
memcpy(res, s->data(), n);
res += n;
@ -25,7 +25,7 @@ protected:
CatArg() {}
CatArg(size_t _max_size) : max_size(_max_size) {}
virtual size_t ComputeMaxSize(ZVal* zframe, int slot) { return 0; }
virtual size_t ComputeMaxSize(const ZVal& zv) { return 0; }
// Present if max size is known a priori.
std::optional<size_t> max_size;
@ -38,7 +38,7 @@ class FixedCatArg : public CatArg {
public:
FixedCatArg(TypePtr t);
void RenderInto(ZVal* zframe, int slot, char*& res) override;
void RenderInto(const ZVal& zv, char*& res) override;
protected:
TypePtr t;
@ -49,22 +49,22 @@ class StringCatArg : public CatArg {
public:
StringCatArg() : CatArg() {}
void RenderInto(ZVal* zframe, int slot, char*& res) override {
auto s = zframe[slot].AsString();
void RenderInto(const ZVal& zv, char*& res) override {
auto s = zv.AsString();
auto n = s->Len();
memcpy(res, s->Bytes(), n);
res += n;
}
protected:
size_t ComputeMaxSize(ZVal* zframe, int slot) override { return zframe[slot].AsString()->Len(); }
size_t ComputeMaxSize(const ZVal& zv) override { return zv.AsString()->Len(); }
};
class PatternCatArg : public CatArg {
public:
PatternCatArg() : CatArg() {}
void RenderInto(ZVal* zframe, int slot, char*& res) override {
void RenderInto(const ZVal& zv, char*& res) override {
*(res++) = '/';
strcpy(res, text);
res += n;
@ -72,7 +72,7 @@ public:
}
protected:
size_t ComputeMaxSize(ZVal* zframe, int slot) override;
size_t ComputeMaxSize(const ZVal& zv) override;
const char* text = nullptr;
size_t n = 0;
@ -82,7 +82,7 @@ class DescCatArg : public CatArg {
public:
DescCatArg(TypePtr _t) : CatArg(), t(std::move(_t)) { d.SetStyle(RAW_STYLE); }
void RenderInto(ZVal* zframe, int slot, char*& res) override {
void RenderInto(const ZVal& zv, char*& res) override {
auto n = d.Len();
memcpy(res, d.Bytes(), n);
res += n;
@ -90,8 +90,8 @@ public:
}
protected:
size_t ComputeMaxSize(ZVal* zframe, int slot) override {
zframe[slot].ToVal(t)->Describe(&d);
size_t ComputeMaxSize(const ZVal& zv) override {
zv.ToVal(t)->Describe(&d);
return d.Len();
}

16
src/script_opt/ZAM/Compile.h
View file

@ -159,16 +159,17 @@ private:
const ZAMStmt ValueSwitch(const SwitchStmt* sw, const NameExpr* v, const ConstExpr* c);
const ZAMStmt TypeSwitch(const SwitchStmt* sw, const NameExpr* v, const ConstExpr* c);
const ZAMStmt GenSwitch(const SwitchStmt* sw, int slot, InternalTypeTag it);
void PushNexts() { PushGoTos(nexts); }
void PushBreaks() { PushGoTos(breaks); }
void PushFallThroughs() { PushGoTos(fallthroughs); }
void PushCatchReturns() { PushGoTos(catches); }
void ResolveNexts(const InstLabel l) { ResolveGoTos(nexts, l); }
void ResolveBreaks(const InstLabel l) { ResolveGoTos(breaks, l); }
void ResolveNexts(const InstLabel l) { ResolveGoTos(nexts, l, CFT_NEXT); }
void ResolveBreaks(const InstLabel l) { ResolveGoTos(breaks, l, CFT_BREAK); }
void ResolveFallThroughs(const InstLabel l) { ResolveGoTos(fallthroughs, l); }
void ResolveCatchReturns(const InstLabel l) { ResolveGoTos(catches, l); }
void ResolveCatchReturns(const InstLabel l) { ResolveGoTos(catches, l, CFT_INLINED_RETURN); }
const ZAMStmt LoopOverTable(const ForStmt* f, const NameExpr* val);
const ZAMStmt LoopOverVector(const ForStmt* f, const NameExpr* val);
@ -229,8 +230,8 @@ private:
const ZAMStmt CompileIndex(const NameExpr* n1, int n2_slot, const TypePtr& n2_type, const ListExpr* l,
bool in_when);
const ZAMStmt BuildLambda(const NameExpr* n, LambdaExpr* le);
const ZAMStmt BuildLambda(int n_slot, LambdaExpr* le);
const ZAMStmt BuildLambda(const NameExpr* n, ExprPtr le);
const ZAMStmt BuildLambda(int n_slot, ExprPtr le);
// Second argument is which instruction slot holds the branch target.
const ZAMStmt GenCond(const Expr* e, int& branch_v);
@ -277,7 +278,7 @@ private:
using GoToSets = std::vector<GoToSet>;
void PushGoTos(GoToSets& gotos);
void ResolveGoTos(GoToSets& gotos, const InstLabel l);
void ResolveGoTos(GoToSets& gotos, const InstLabel l, ControlFlowType cft = CFT_NONE);
ZAMStmt GenGoTo(GoToSet& v);
ZAMStmt GoToStub();
@ -322,6 +323,9 @@ private:
const ZAMStmt ErrorStmt();
const ZAMStmt LastInst();
// Adds control flow information to an instruction.
void AddCFT(ZInstI* inst, ControlFlowType cft);
// Returns a handle to state associated with building
// up a list of values.
std::unique_ptr<OpaqueVals> BuildVals(const ListExprPtr&);

3
src/script_opt/ZAM/Driver.cc
View file

@ -203,6 +203,7 @@ StmtPtr ZAMCompiler::CompileBody() {
auto zb = make_intrusive<ZBody>(fname, this);
zb->SetInsts(insts2);
zb->SetLocationInfo(body->GetLocationInfo());
// Could erase insts1 here to recover memory, but it's handy
// for debugging.
@ -218,7 +219,9 @@ void ZAMCompiler::ResolveHookBreaks() {
// Rewrite the breaks.
for ( auto& b : breaks[0] ) {
auto& i = insts1[b.stmt_num];
auto aux = i->aux;
*i = ZInstI(OP_HOOK_BREAK_X);
i->aux = aux;
}
}

103
src/script_opt/ZAM/Expr.cc
View file

@ -246,7 +246,7 @@ const ZAMStmt ZAMCompiler::CompileAssignExpr(const AssignExpr* e) {
}
if ( rhs->Tag() == EXPR_LAMBDA )
return BuildLambda(lhs, rhs->AsLambdaExpr());
return BuildLambda(lhs, op2);
if ( rhs->Tag() == EXPR_COND && r1->GetType()->Tag() == TYPE_VECTOR )
return Bool_Vec_CondVVVV(lhs, r1->AsNameExpr(), r2->AsNameExpr(), r3->AsNameExpr());
@ -466,20 +466,23 @@ const ZAMStmt ZAMCompiler::CompileFieldLHSAssignExpr(const FieldLHSAssignExpr* e
auto field = e->Field();
if ( rhs->Tag() == EXPR_NAME )
return Field_LHS_AssignFV(e, rhs->AsNameExpr());
return Field_LHS_AssignFVi(e, rhs->AsNameExpr(), field);
if ( rhs->Tag() == EXPR_CONST )
return Field_LHS_AssignFC(e, rhs->AsConstExpr());
return Field_LHS_AssignFCi(e, rhs->AsConstExpr(), field);
auto r1 = rhs->GetOp1();
auto r2 = rhs->GetOp2();
if ( rhs->Tag() == EXPR_FIELD ) {
auto rhs_f = rhs->AsFieldExpr();
if ( r1->Tag() == EXPR_NAME )
return Field_LHS_AssignFVi(e, r1->AsNameExpr(), rhs_f->Field());
return Field_LHS_AssignFCi(e, r1->AsConstExpr(), rhs_f->Field());
// Note, the LHS field comes after the RHS field rather than before,
// to maintain layout symmetry close to that for non-field RHS's.
if ( r1->Tag() == EXPR_NAME )
return Field_LHS_AssignFVii(e, r1->AsNameExpr(), rhs_f->Field(), field);
return Field_LHS_AssignFCii(e, r1->AsConstExpr(), rhs_f->Field(), field);
}
if ( r1 && r1->IsConst() )
@ -564,7 +567,7 @@ const ZAMStmt ZAMCompiler::CompileEvent(EventHandler* h, const ListExpr* l) {
else {
auto n0 = exprs[0]->AsNameExpr();
z.v1 = FrameSlot(n0);
z.t = n0->GetType();
z.SetType(n0->GetType());
if ( n == 1 ) {
z.op = OP_EVENT1_V;
@ -574,7 +577,7 @@ const ZAMStmt ZAMCompiler::CompileEvent(EventHandler* h, const ListExpr* l) {
else {
auto n1 = exprs[1]->AsNameExpr();
z.v2 = FrameSlot(n1);
z.t2 = n1->GetType();
z.SetType2(n1->GetType());
if ( n == 2 ) {
z.op = OP_EVENT2_VV;
@ -630,7 +633,7 @@ const ZAMStmt ZAMCompiler::CompileInExpr(const NameExpr* n1, const NameExpr* n2,
if ( op3_t->AsTableType()->IsPatternIndex() && op2_t->Tag() == TYPE_STRING )
a = n2 ? OP_STR_IN_PAT_TBL_VVV : OP_STR_IN_PAT_TBL_VCV;
else
a = n2 ? OP_VAL_IS_IN_TABLE_VVV : OP_CONST_IS_IN_TABLE_VCV;
a = n2 ? OP_VAL_IS_IN_TABLE_VVV : OP_CONST_IS_IN_TABLE_VVC;
}
else if ( op2->GetType()->Tag() == TYPE_PATTERN )
a = n2 ? (n3 ? OP_P_IN_S_VVV : OP_P_IN_S_VVC) : OP_P_IN_S_VCV;
@ -692,11 +695,11 @@ const ZAMStmt ZAMCompiler::CompileInExpr(const NameExpr* n1, const ListExpr* l,
else {
auto l_e0_c = l_e[0]->AsConstExpr();
ZOp op = is_vec ? OP_CONST_IS_IN_VECTOR_VCV : OP_CONST_IS_IN_TABLE_VCV;
ZOp op = is_vec ? OP_CONST_IS_IN_VECTOR_VCV : OP_CONST_IS_IN_TABLE_VVC;
z = GenInst(op, n1, l_e0_c, n2);
}
z.t = l_e[0]->GetType();
z.SetType(l_e[0]->GetType());
return AddInst(z);
}
@ -717,21 +720,21 @@ const ZAMStmt ZAMCompiler::CompileInExpr(const NameExpr* n1, const ListExpr* l,
if ( l_e0_n && l_e1_n ) {
z = GenInst(OP_VAL2_IS_IN_TABLE_VVVV, n1, l_e0_n, l_e1_n, n2);
z.t2 = l_e0_n->GetType();
z.SetType2(l_e0_n->GetType());
}
else if ( l_e0_n ) {
ASSERT(l_e1_c);
z = GenInst(OP_VAL2_IS_IN_TABLE_VVVC, n1, l_e0_n, n2, l_e1_c);
z.t2 = l_e0_n->GetType();
z.SetType2(l_e0_n->GetType());
}
else if ( l_e1_n ) {
ASSERT(l_e0_c);
z = GenInst(OP_VAL2_IS_IN_TABLE_VVCV, n1, l_e1_n, n2, l_e0_c);
z.t2 = l_e1_n->GetType();
z.SetType2(l_e1_n->GetType());
}
else {
@ -743,7 +746,7 @@ const ZAMStmt ZAMCompiler::CompileInExpr(const NameExpr* n1, const ListExpr* l,
auto slot = TempForConst(l_e0_c);
z = ZInstI(OP_VAL2_IS_IN_TABLE_VVVC, FrameSlot(n1), slot, FrameSlot(n2), l_e1_c);
z.op_type = OP_VVVC;
z.t2 = l_e0_c->GetType();
z.SetType2(l_e0_c->GetType());
}
return AddInst(z);
@ -817,7 +820,7 @@ const ZAMStmt ZAMCompiler::CompileIndex(const NameExpr* n1, int n2_slot, const T
z = ZInstI(zop, Frame1Slot(n1, zop), n2_slot, n3_slot);
}
else {
auto zop = OP_INDEX_STRINGC_VVV;
auto zop = OP_INDEX_STRINGC_VVi;
z = ZInstI(zop, Frame1Slot(n1, zop), n2_slot, c);
z.op_type = OP_VVV_I3;
}
@ -846,11 +849,11 @@ const ZAMStmt ZAMCompiler::CompileIndex(const NameExpr* n1, int n2_slot, const T
ZOp zop;
if ( in_when )
zop = OP_WHEN_INDEX_VECC_VVV;
zop = OP_WHEN_INDEX_VECC_VVi;
else if ( is_any )
zop = OP_INDEX_ANY_VECC_VVV;
zop = OP_INDEX_ANY_VECC_VVi;
else
zop = OP_INDEX_VECC_VVV;
zop = OP_INDEX_VECC_VVi;
z = ZInstI(zop, Frame1Slot(n1, zop), n2_slot, c);
z.op_type = OP_VVV_I3;
@ -932,18 +935,18 @@ const ZAMStmt ZAMCompiler::CompileIndex(const NameExpr* n1, int n2_slot, const T
return AddInst(z);
}
const ZAMStmt ZAMCompiler::BuildLambda(const NameExpr* n, LambdaExpr* le) {
return BuildLambda(Frame1Slot(n, OP1_WRITE), le);
const ZAMStmt ZAMCompiler::BuildLambda(const NameExpr* n, ExprPtr e) {
return BuildLambda(Frame1Slot(n, OP1_WRITE), std::move(e));
}
const ZAMStmt ZAMCompiler::BuildLambda(int n_slot, LambdaExpr* le) {
auto& captures = le->GetCaptures();
const ZAMStmt ZAMCompiler::BuildLambda(int n_slot, ExprPtr e) {
auto lambda = cast_intrusive<LambdaExpr>(e);
auto& captures = lambda->GetCaptures();
int ncaptures = captures ? captures->size() : 0;
auto aux = new ZInstAux(ncaptures);
aux->primary_func = le->PrimaryFunc();
aux->lambda_name = le->Name();
aux->id_val = le->Ingredients()->GetID();
aux->lambda = cast_intrusive<LambdaExpr>(std::move(e));
aux->id_val = lambda->Ingredients()->GetID();
for ( int i = 0; i < ncaptures; ++i ) {
auto& id_i = (*captures)[i].Id();
@ -954,7 +957,7 @@ const ZAMStmt ZAMCompiler::BuildLambda(int n_slot, LambdaExpr* le) {
aux->Add(i, FrameSlot(id_i), id_i->GetType());
}
auto z = ZInstI(OP_LAMBDA_VV, n_slot, le->PrimaryFunc()->FrameSize());
auto z = ZInstI(OP_LAMBDA_Vi, n_slot, lambda->PrimaryFunc()->FrameSize());
z.op_type = OP_VV_I2;
z.aux = aux;
@ -1031,7 +1034,7 @@ const ZAMStmt ZAMCompiler::AssignVecElems(const Expr* e) {
inst = Vector_Elem_AssignVVC(lhs, n2, c3);
}
TopMainInst()->t = t3;
TopMainInst()->SetType(t3);
return inst;
}
@ -1048,7 +1051,7 @@ const ZAMStmt ZAMCompiler::AssignVecElems(const Expr* e) {
else
inst = Vector_Elem_AssignVVi(lhs, n3, index);
TopMainInst()->t = t3;
TopMainInst()->SetType(t3);
return inst;
}
@ -1068,7 +1071,7 @@ const ZAMStmt ZAMCompiler::AssignTableElem(const Expr* e) {
z = GenInst(OP_TABLE_ELEM_ASSIGN_VC, op1, op3->AsConstExpr());
z.aux = InternalBuildVals(op2);
z.t = op3->GetType();
z.SetType(op3->GetType());
if ( pfs->HasSideEffects(SideEffectsOp::WRITE, op1->GetType()) )
z.aux->can_change_non_locals = true;
@ -1159,7 +1162,7 @@ const ZAMStmt ZAMCompiler::DoCall(const CallExpr* c, const NameExpr* n) {
}
}
z.t = arg0->GetType();
z.SetType(arg0->GetType());
}
else {
@ -1184,10 +1187,12 @@ const ZAMStmt ZAMCompiler::DoCall(const CallExpr* c, const NameExpr* n) {
default:
if ( in_when ) {
if ( indirect )
op = OP_WHENINDCALLN_VV;
else
if ( ! indirect )
op = OP_WHENCALLN_V;
else if ( func_id->IsGlobal() )
op = OP_WHEN_ID_INDCALLN_V;
else
op = OP_WHENINDCALLN_VV;
}
else if ( indirect ) {
@ -1279,9 +1284,9 @@ const ZAMStmt ZAMCompiler::ConstructTable(const NameExpr* n, const Expr* e) {
auto tt = cast_intrusive<TableType>(n->GetType());
auto width = tt->GetIndices()->GetTypes().size();
auto z = GenInst(OP_CONSTRUCT_TABLE_VV, n, width);
auto z = GenInst(OP_CONSTRUCT_TABLE_Vi, n, width);
z.aux = InternalBuildVals(con, width + 1);
z.t = tt;
z.SetType(tt);
ASSERT(e->Tag() == EXPR_TABLE_CONSTRUCTOR);
z.aux->attrs = static_cast<const TableConstructorExpr*>(e)->GetAttrs();
@ -1291,7 +1296,7 @@ const ZAMStmt ZAMCompiler::ConstructTable(const NameExpr* n, const Expr* e) {
if ( ! def_attr || def_attr->GetExpr()->Tag() != EXPR_LAMBDA )
return zstmt;
auto def_lambda = def_attr->GetExpr()->AsLambdaExpr();
auto def_lambda = cast_intrusive<LambdaExpr>(def_attr->GetExpr());
auto dl_t = def_lambda->GetType()->AsFuncType();
auto& captures = dl_t->GetCaptures();
@ -1308,7 +1313,7 @@ const ZAMStmt ZAMCompiler::ConstructTable(const NameExpr* n, const Expr* e) {
z = GenInst(OP_SET_TABLE_DEFAULT_LAMBDA_VV, n, slot);
z.op_type = OP_VV;
z.t = def_lambda->GetType();
z.SetType(def_lambda->GetType());
return AddInst(z);
}
@ -1318,9 +1323,9 @@ const ZAMStmt ZAMCompiler::ConstructSet(const NameExpr* n, const Expr* e) {
auto tt = n->GetType()->AsTableType();
auto width = tt->GetIndices()->GetTypes().size();
auto z = GenInst(OP_CONSTRUCT_SET_VV, n, width);
auto z = GenInst(OP_CONSTRUCT_SET_Vi, n, width);
z.aux = InternalBuildVals(con, width);
z.t = e->GetType();
z.SetType(e->GetType());
ASSERT(e->Tag() == EXPR_SET_CONSTRUCTOR);
z.aux->attrs = static_cast<const SetConstructorExpr*>(e)->GetAttrs();
@ -1391,13 +1396,13 @@ const ZAMStmt ZAMCompiler::ConstructRecord(const NameExpr* n, const Expr* e, boo
if ( fi->empty() ) {
if ( network_time_index >= 0 )
op = OP_CONSTRUCT_KNOWN_RECORD_WITH_NT_VV;
op = OP_CONSTRUCT_KNOWN_RECORD_WITH_NT_Vi;
else
op = OP_CONSTRUCT_KNOWN_RECORD_V;
}
else {
if ( network_time_index >= 0 )
op = OP_CONSTRUCT_KNOWN_RECORD_WITH_INITS_AND_NT_VV;
op = OP_CONSTRUCT_KNOWN_RECORD_WITH_INITS_AND_NT_Vi;
else
op = OP_CONSTRUCT_KNOWN_RECORD_WITH_INITS_V;
aux->field_inits = std::move(fi);
@ -1411,12 +1416,12 @@ const ZAMStmt ZAMCompiler::ConstructRecord(const NameExpr* n, const Expr* e, boo
if ( is_from_rec ) {
// Map non-from-rec operand to the from-rec equivalent.
switch ( op ) {
case OP_CONSTRUCT_KNOWN_RECORD_WITH_NT_VV: op = OP_CONSTRUCT_KNOWN_RECORD_WITH_NT_FROM_VVV; break;
case OP_CONSTRUCT_KNOWN_RECORD_WITH_NT_Vi: op = OP_CONSTRUCT_KNOWN_RECORD_WITH_NT_FROM_VVi; break;
case OP_CONSTRUCT_KNOWN_RECORD_V: op = OP_CONSTRUCT_KNOWN_RECORD_FROM_VV; break;
case OP_CONSTRUCT_KNOWN_RECORD_WITH_INITS_AND_NT_VV:
op = OP_CONSTRUCT_KNOWN_RECORD_WITH_INITS_AND_NT_FROM_VVV;
case OP_CONSTRUCT_KNOWN_RECORD_WITH_INITS_AND_NT_Vi:
op = OP_CONSTRUCT_KNOWN_RECORD_WITH_INITS_AND_NT_FROM_VVi;
break;
case OP_CONSTRUCT_KNOWN_RECORD_WITH_INITS_V:
@ -1448,7 +1453,7 @@ const ZAMStmt ZAMCompiler::ConstructRecord(const NameExpr* n, const Expr* e, boo
z = network_time_index >= 0 ? GenInst(op, n, network_time_index) : GenInst(op, n);
z.aux = aux;
z.t = rec_e->GetType();
z.SetType(rec_e->GetType());
auto inst = AddInst(z);
@ -1488,7 +1493,7 @@ const ZAMStmt ZAMCompiler::ConstructRecord(const NameExpr* n, const Expr* e, boo
// Need to add a separate instruction for concretizing the fields.
z = GenInst(OP_CONCRETIZE_VECTOR_FIELDS_V, n);
z.t = rec_e->GetType();
z.SetType(rec_e->GetType());
int nf = static_cast<int>(vector_fields.size());
z.aux = new ZInstAux(nf);
z.aux->elems_has_slots = false; // we're storing field offsets, not slots
@ -1503,7 +1508,7 @@ const ZAMStmt ZAMCompiler::ConstructVector(const NameExpr* n, const Expr* e) {
auto z = GenInst(OP_CONSTRUCT_VECTOR_V, n);
z.aux = InternalBuildVals(con);
z.t = e->GetType();
z.SetType(e->GetType());
return AddInst(z);
}
@ -1626,8 +1631,8 @@ const ZAMStmt ZAMCompiler::Is(const NameExpr* n, const Expr* e) {
int op_slot = FrameSlot(op);
ZInstI z(OP_IS_VV, Frame1Slot(n, OP_IS_VV), op_slot);
z.t2 = op->GetType();
z.SetType(is->TestType());
z.SetType2(op->GetType());
return AddInst(z);
}

74
src/script_opt/ZAM/IterInfo.h
View file

@ -15,21 +15,54 @@ namespace zeek::detail {
class TableIterInfo {
public:
// No constructor needed, as all of our member variables are
// instead instantiated via BeginLoop(). This allows us to
// reuse TableIterInfo objects to lower the overhead associated
// with executing ZBody::Exec for non-recursive functions.
// Empty constructor for a simple version that initializes all the
// member variables via BeginLoop(). Helpful for supporting recursive
// functions that include table iterations.
TableIterInfo() {}
// Version that populates the fixed fields up front, with the
// dynamic ones being done with SetLoopVars().
TableIterInfo(const std::vector<TypePtr>* _loop_var_types, const std::vector<bool>* _lvt_is_managed,
TypePtr _value_var_type) {
SetIterInfo(_loop_var_types, _lvt_is_managed, std::move(_value_var_type));
}
// Sets the fixed fields.
void SetIterInfo(const std::vector<TypePtr>* _loop_var_types, const std::vector<bool>* _lvt_is_managed,
TypePtr _value_var_type) {
loop_var_types = _loop_var_types;
lvt_is_managed = _lvt_is_managed;
value_var_type = std::move(_value_var_type);
}
// We do, however, want to make sure that when we go out of scope,
// if we have any pending iterators we clear them.
~TableIterInfo() { Clear(); }
// Start looping over the elements of the given table. "_aux"
// Start looping over the elements of the given table. "aux"
// provides information about the index variables, their types,
// and the type of the value variable (if any).
void BeginLoop(TableValPtr _tv, ZInstAux* _aux) {
tv = _tv;
aux = _aux;
void BeginLoop(TableValPtr _tv, ZVal* frame, ZInstAux* aux) {
tv = std::move(_tv);
for ( auto lv : aux->loop_vars )
if ( lv < 0 )
loop_vars.push_back(nullptr);
else
loop_vars.push_back(&frame[lv]);
SetIterInfo(&aux->types, &aux->is_managed, aux->value_var_type);
PrimeIter();
}
void BeginLoop(TableValPtr _tv, std::vector<ZVal*> _loop_vars) {
tv = std::move(_tv);
loop_vars = std::move(_loop_vars);
PrimeIter();
}
void PrimeIter() {
auto tvd = tv->AsTable();
tbl_iter = tvd->begin();
tbl_end = tvd->end();
@ -43,18 +76,17 @@ public:
// Performs the next iteration (assuming IsDoneIterating() returned
// false), assigning to the index variables.
void NextIter(ZVal* frame) {
void NextIter() {
auto ind_lv = tv->RecreateIndex(*(*tbl_iter)->GetHashKey());
for ( int i = 0; i < ind_lv->Length(); ++i ) {
ValPtr ind_lv_p = ind_lv->Idx(i);
auto lv = aux->loop_vars[i];
if ( lv < 0 )
auto lv = loop_vars[i];
if ( ! lv )
continue;
auto& var = frame[lv];
if ( aux->is_managed[i] )
ZVal::DeleteManagedType(var);
auto& t = aux->types[i];
var = ZVal(ind_lv_p, t);
ValPtr ind_lv_p = ind_lv->Idx(i);
if ( (*lvt_is_managed)[i] )
ZVal::DeleteManagedType(*lv);
*lv = ZVal(ind_lv_p, (*loop_var_types)[i]);
}
IterFinished();
@ -63,7 +95,7 @@ public:
// For the current iteration, returns the corresponding value.
ZVal IterValue() {
auto tev = (*tbl_iter)->value;
return ZVal(tev->GetVal(), aux->value_var_type);
return ZVal(tev->GetVal(), value_var_type);
}
// Called upon finishing the iteration.
@ -78,8 +110,10 @@ public:
private:
TableValPtr tv = nullptr;
// Associated auxiliary information.
ZInstAux* aux = nullptr;
std::vector<ZVal*> loop_vars;
const std::vector<TypePtr>* loop_var_types;
const std::vector<bool>* lvt_is_managed;
TypePtr value_var_type;
std::optional<DictIterator<TableEntryVal>> tbl_iter;
std::optional<DictIterator<TableEntryVal>> tbl_end;

28
src/script_opt/ZAM/Low-Level.cc
View file

@ -20,9 +20,25 @@ bool ZAMCompiler::NullStmtOK() const {
const ZAMStmt ZAMCompiler::EmptyStmt() { return ZAMStmt(insts1.size() - 1); }
const ZAMStmt ZAMCompiler::ErrorStmt() { return ZAMStmt(0); }
const ZAMStmt ZAMCompiler::LastInst() { return ZAMStmt(insts1.size() - 1); }
const ZAMStmt ZAMCompiler::ErrorStmt() { return ZAMStmt(0); }
void ZAMCompiler::AddCFT(ZInstI* inst, ControlFlowType cft) {
if ( cft == CFT_NONE )
return;
if ( ! inst->aux )
inst->aux = new ZInstAux(0);
auto cft_entry = inst->aux->cft.find(cft);
if ( cft_entry == inst->aux->cft.end() )
inst->aux->cft[cft] = 1;
else {
ASSERT(cft == CFT_BLOCK_END || cft == CFT_BREAK);
++cft_entry->second;
}
}
std::unique_ptr<OpaqueVals> ZAMCompiler::BuildVals(const ListExprPtr& l) {
return std::make_unique<OpaqueVals>(InternalBuildVals(l.get()));
@ -127,9 +143,9 @@ const ZAMStmt ZAMCompiler::AddInst(const ZInstI& inst, bool suppress_non_local)
auto gs = pending_global_store;
pending_global_store = -1;
auto store_inst = ZInstI(OP_STORE_GLOBAL_V, gs);
auto store_inst = ZInstI(OP_STORE_GLOBAL_g, gs);
store_inst.op_type = OP_V_I1;
store_inst.t = globalsI[gs].id->GetType();
store_inst.SetType(globalsI[gs].id->GetType());
return AddInst(store_inst);
}
@ -138,15 +154,15 @@ const ZAMStmt ZAMCompiler::AddInst(const ZInstI& inst, bool suppress_non_local)
auto cs = pending_capture_store;
pending_capture_store = -1;
auto& cv = *func->GetType()->AsFuncType()->GetCaptures();
auto& cv = *func->GetType()->GetCaptures();
auto& c_id = cv[cs].Id();
ZOp op;
if ( ZVal::IsManagedType(c_id->GetType()) )
op = OP_STORE_MANAGED_CAPTURE_VV;
op = OP_STORE_MANAGED_CAPTURE_Vi;
else
op = OP_STORE_CAPTURE_VV;
op = OP_STORE_CAPTURE_Vi;
auto store_inst = ZInstI(op, RawSlot(c_id.get()), cs);
store_inst.op_type = OP_VV_I2;

286
src/script_opt/ZAM/OPs/README.txt Normal file
View file

@ -0,0 +1,286 @@
# See the file "COPYING" in the main distribution directory for copyright.
# This directory contains templates used to generate virtual functions, opcodes,
# and evaluation code for compiled code. Each template describes a ZAM
# "operation", which generally corresponds to a set of concrete ZAM
# "instructions". (See ZInst.h for the layout of ZAM instructions.) Often
# a single ZAM operation gives rise to a family of instructions that differ
# in either the nature of the instruction's operands (typically, whether
# they are variables residing on the ZAM execution frame, or constants)
# and/or the Zeek type of the operands (e.g., "count" or "double" or "addr").
#
# The Gen-ZAM utility processes this file to generate numerous C++ inclusion
# files that are then compiled into Zeek. These files span the range of (1)
# hooks that enable run-time generation of ZAM code to execute ASTs (which
# have first been transformed to "reduced" form), (2) specifications of the
# properties of the different instructions, (3) code to evaluate (execute)
# each instruction, and (4) macros (C++ #define's) to aid in writing that
# code. See Gen-ZAM.h for a list of the different inclusion files.
#
# Operation templates are declarative, other than the imperative C++ snippets
# they include for instruction evaluation/execution. You specify a template
# using lines of text for which, for the most part, the first word on the
# line designates an "attribute" associated with the template, and the
# remainder of the line provides specifiers/arguments for that attribute.
# A blank line (or end of file) ends the template. By convention, for
# templates that include C++ evaluation snippets, those are specified as the
# last attribute. Comments begin with '#' at the start of the line (no
# leading whitespace allowed), and can be intermingled with a template's
# attributes.
#
# Each ZAM instruction includes up to 4 integer values and one constant
# (specified as a ZVal). Often, the integer values are interpreted as offsets
# ("slots") into the ZAM execution "frame", though sometimes they have other
# meanings, such as the offset of a particular field in a record, or an index
# into the ZAM code for a branch instruction. Most instructions compute
# some sort of result (expressed as a ZVal) that is stored into the frame
# slot specified by the instruction's first integer value. We refer to this
# target as the "assignment slot", and to the other 3 integer values as
# "operands". Thus, for example, an instruction with two operands used the
# first 3 integer values, the first as the assignment slot and the other two
# for computing the result to put in that slot.
#
# Instruction templates have one or more "type"s associated with them (as
# discussed below) specifying the types of operands (variables corresponding
# to slots, or constants) associated with the instruction. In the evaluation
# code for an instruction, these are referred to with $-parameters, such as
# $1 for the first operand. The special parameter $$ refers to the *assignment
# target* of the instruction, if applicable. These parameters always come
# first when specifying an instruction's type. For example, a type of "VVC"
# specifies an instruction with two variables and one constant associated
# with it. If the instruction assigns a value, then in the evaluation these
# will be specified as $$, $1 and $2, respectively. If it does not (usually
# reflected by the template having the "op1-read" attribute) then they
# are specified as $1, $2 and $3, respectively. See "eval" below.
#
# The first attribute of each template states the type of operation specified
# in the template, along with the name of the operation. The possible types
# are:
#
# op an operation that generally corresponds to a single ZAM
# instruction, and is fully specified
#
# expr-op an operation corresponding to an AST expression node
# (some sort of Expr object). Gen-ZAM generates code for
# automatically converting Expr objects to ZAM instructions.
# The name of the operation must match that used in the AST
# tag, so for example for "expr-op Foo" there must be a
# corresponding "EXPR_FOO" tag.
#
# unary-expr-op an expr-op for a unary Expr object
# binary-expr-op an expr-op for a binary Expr object
# rel-expr-op an expr-op for a (binary) Expr object that
# represents a relational operation
#
# assign-op directly assigning either a ZVal or a record field
# to either a frame slot or a record field
#
# unary-op an operation with one operand that requires special
# treatment that doesn't fit with how unary-expr-op's
# are expressed
#
# direct-unary-op an operation with one operand that corresponds to
# a specific ZAMCompiler method for generating its
# instruction
#
# internal-op similar to "op", but for ZAM instructions only used
# internally, and thus not having any AST counterpart
# internal-assignment-op the same, for operations that assign ZVals
# produced by loading interpreter variables
# or calling functions
#
# After specifying the type of operation, you list additional attributes to
# fill out the template, ending by convention with the C++ evaluation snippet
# (if appropriate). The most significant (and complex) of these are:
#
# class specifies how to interpret the operation in terms of ZAM
# instruction slots (and constant). The specification is
# in terms of single-letter mnemonics for the different
# possible classes:
#
# F special value designating a record field being
# assigned to
# H event handler
# L list of values
# O opaque value (here, "opaque" refers to ZAM
# internals, not OpaqueVal)
# R record
# V variable (frame slot)
# X used to indicate an empty specifier
# b branch target
# f iteration information associated with table "for" loop
# g access to a global
# i integer constant, often a record field offset
# s iteration information associated with stepping
# through a vector or string
#
# The full specification consists of concatenating mnemonics
# with the order left-to-right corresponding to each of the
# instruction's 4 integer values (stopping with the last one
# used). If the operation includes a constant, then it is
# listed at the point reflecting where the constant is used as
# an operand. For example, a class of "VVCV" means that the
# first integer is used as a frame variable (i.e., the usual
# "assignment slot"), the second integer (first "operand") is
# also a frame variable, the second operand is the instruction's
# constant, and the third operand is the instruction's third
# integer value, with the fourth integer value not being used.
#
# classes specifies a number of "class" values to instantiate over.
# Cannot be combined with "class", nor used for expressions.
#
# op-type for some form of expr-op, specifies to which Zeek scripting
# types the expression applies:
#
# A addr
# D double
# F file
# I int
# N subnet
# P pattern
# R record
# S string
# T table
# U count
# V vector
#
# along with two special types: 'X' indicates that Gen-ZAM
# should not iterate over any possible values, and '*'
# indicates that Gen-ZAM should additionally iterate over
# all of possible values not explicitly listed (used in
# conjunction with eval-type - see below)
#
# op-types similar to op-type, but lists a type for each operand
# (including assignment target), so for example "A N A"
# would correspond to a 3-operand instruction for which
# the first operand (or assignment target) is an "addr",
# the second a "subnet", and the third another "addr".
#
# Note that these types collectively apply to each instance of
# an operation, whereas listing multiple "op-type" types
# iterates through those one-at-a-time in turn (and generally
# the point is that the each type applies to *all* operands,
# rather than a per-operand list). Given that, the two are
# incompatible.
#
# For operands corresponding to 'i' or any of the internal types,
# such as 'b', 'f', 'g', and 's', the corresponding type to
# list is 'I', used for integer access.
#
# eval specifies a block of C++ code used to evaluation the
# execution of the instruction. The block begins with the
# remainder of the "eval" line and continues until either a
# blank line or a line that starts with non-whitespace.
#
# Blocks can include special '$' parameters that Gen-ZAM
# automatically expands. "$1" refers to an operation's first
# operand, "$2" to its second, etc. "$$" refers to the
# operation's assignment target.
#
# For simple expr-op's you can express the block as simply
# the C++ expression to compute. For example, for multiplication
# (named "Times"), the "eval" block is simply "$1 * $2",
# rather than "$$ = $1 * $2"; Gen-ZAM knows to expand it
# accordingly.
#
# Finally, to help with avoiding duplicate code, you can
# define macros that expand to code snippets you want to use
# in multiple places. You specify these using a "macro"
# keyword followed by the name of the macro and an evaluation
# block. Macros behave identically to C++ #define's, except
# you don't use "\" to continue them across line breaks, but
# instead just indent the lines you want included, ending
# (as with "eval" blocks) with an empty line or a line that
# starts with non-whitespace.
#
# We list the remaining types of attributes alphabetically. Note that some
# only apply to certain types of operations.
#
# assign-val for an assignment operation, the name of the
# C++ variable that holds the value to assign
#
# custom-method a ZAMCompiler method that Gen-ZAM should use for
# this operation, rather than generating one
#
# eval-mixed an expression "eval" block that applies to two
# different op-type's
#
# eval-type evaluation code associated with one specific op-type
#
# explicit-result-type the operation's evaluation yields a ZVal
# rather than a low-level C++ type
#
# field-op the operation is a direct assignment to a record field
#
# includes-field-op the operation should include a version
# that assigns to a record field as well as a
# version for assigning to a frame variable
#
# indirect-call the operation represents an indirect call (through
# a global variable, rather than directly). Only
# meaningful if num-call-args is also specified.
#
# indirect-local-call same, but via a local variable rather than
# global
#
# method-post C++ code to add to the end of the method that
# dynamically generates ZAM code
#
# no-const do not generate a version of the unary-expr-op
# where the operand is a constant
#
# no-eval this operation does not have an "eval" block
# (because it will be translated instead into internal
# operations)
#
# num-call-args indicates that the operation is a function call,
# and specifies how many arguments the call takes.
# A specification of 'n' means "build a ZAM instruction
# for calling with an arbitrary number of arguments".
#
# op1-internal states that the operation's treatment of the
# instruction's first integer value is for internal
# purposes; the value does not correspond to a frame
# variable
#
# op1-read the operation treats the instruction's first integer
# value as a frame variable, but only reads the value.
# (The default is that the frame variable is written
# to but not read.)
#
# op1-read-write the operation treats the instruction's first integer
# value as a frame variable, and both reads and
# writes the value.
#
# precheck a test conducted before evaluating an expression,
# which is skipped if the test is true. Must be used
# in conjunction with precheck-action.
#
# precheck-action code to execute if a precheck is true, instead
# of evaluating the expression. Must be used in
# conjunction with precheck.
#
# set-type the instruction's primary type comes from either the
# assignment target ("$$"), the first operand ("$1"),
# or the second operand ("$2")
#
# set-type2 the same as set-type but for the instruction's
# secondary type
#
# side-effects the operation has side-effects, so even if its
# assignment target winds up being "dead" (the value is
# no longer used), the operation should still occur.
# Optionally, this attribute can include two arguments
# specifying the ZAM opcode to use if the assignment
# is dead, and the internal "type" of that opcode.
#
# For example, "side-effects OP_CALL1_V OP_V" means
# "this operation has side-effects; if eliminating
# its assignment, change the ZAM op-code to OP_CALL1_V,
# which has an internal type of OP_V".
#
# vector generate a version of the operation that takes
# vectors as operands
#
# Finally, a note concernning comments: due to internal use of C++ #define
# macros, comments in C++ code should use /* ... */ rather than // delimiters.

3368
src/script_opt/ZAM/OPs/ZAM.op
View file

File diff suppressed because it is too large Load diff

627
src/script_opt/ZAM/OPs/ZBI.op Normal file
View file

@ -0,0 +1,627 @@
# Operations corresponding to ZAM BuiltIn Functions.
internal-op Remove-Teredo
op1-read
class V
op-types R
eval auto teredo = zeek::packet_mgr->GetAnalyzer("Teredo");
if ( teredo )
{
zeek::detail::ConnKey conn_key($1);
static_cast<zeek::packet_analysis::teredo::TeredoAnalyzer*>(teredo.get())->RemoveConnection(conn_key);
}
internal-op Remove-Teredo
side-effects OP_REMOVE_TEREDO_V OP_V
class VV
op-types I R
eval auto teredo = zeek::packet_mgr->GetAnalyzer("Teredo");
if ( teredo )
{
zeek::detail::ConnKey conn_key($1);
static_cast<zeek::packet_analysis::teredo::TeredoAnalyzer*>(teredo.get())->RemoveConnection(conn_key);
}
$$ = 1;
internal-op Remove-GTPv1
op1-read
class V
op-types R
eval auto gtpv1 = zeek::packet_mgr->GetAnalyzer("GTPv1");
if ( gtpv1 )
{
zeek::detail::ConnKey conn_key($1);
static_cast<zeek::packet_analysis::gtpv1::GTPv1_Analyzer*>(gtpv1.get())->RemoveConnection(conn_key);
}
internal-op Remove-GTPv1
side-effects OP_REMOVE_GTPV1_V OP_V
class VV
op-types I R
eval auto gtpv1 = zeek::packet_mgr->GetAnalyzer("GTPv1");
if ( gtpv1 )
{
zeek::detail::ConnKey conn_key($1);
static_cast<zeek::packet_analysis::gtpv1::GTPv1_Analyzer*>(gtpv1.get())->RemoveConnection(conn_key);
}
$$ = 1;
internal-op Set-File-Handle
op1-read
class V
op-types S
eval auto handle = $1;
auto bytes = reinterpret_cast<const char*>(handle->Bytes());
auto h = std::string(bytes, handle->Len());
zeek::file_mgr->SetHandle(h);
internal-op Subnet-To-Addr
class VV
op-types X N
eval auto addr_v = make_intrusive<AddrVal>($1->Prefix());
ZVal::DeleteManagedType($$);
$$ = ZVal(std::move(addr_v));
macro EvalSubBytes(lhs, arg1, arg2, arg3)
{
auto sv = ZAM_sub_bytes(arg1, arg2, arg3);
Unref(lhs);
lhs = sv;
}
internal-op Sub-Bytes
classes VVVV VVVC VVCV VCVV VVCi VCVi
op-types S S U I
eval EvalSubBytes($$, $1, $2, $3)
# Use a distinct name because due to the convention when constructing
# instructions, frame slots are always positioned earlier than non-frame
# slots, i.e. we can't construct "VCiV", which is why the arguments are
# in a different order than above.
internal-op Sub-Bytes2
class VCVi
op-types S S I U
eval EvalSubBytes($$, $1, $3, $2)
internal-op Time-To-Double
class VV
op-types D D
eval $$ = $1;
internal-op To-Lower
class VV
op-types S S
eval auto sv = ZAM_to_lower($1);
Unref($$);
$$ = sv;
# A ZAM version of Log::__write. In calls to it, the first argument
# is generally a constant (enum) *if we inlined*, but otherwise a
# parameter, so we support both VVV ad VVC.
#
# It's actually the case that the return value is pretty much always
# ignored ... plus optimization can elide it away. See the second
# pair of built-ins for versions that discard the return value.
#
# Could speed things up further by modifying the Write method to just
# take the raw enum value, as it appears that that's all that's ever
# actually used.
macro LogWritePre(id_val, columns_val)
auto id = id_val;
auto columns = columns_val;
macro LogWriteResPost(lhs)
bool result = log_mgr->Write(id->AsEnumVal(), columns->AsRecordVal());
lhs = result;
internal-op Log-Write
side-effects OP_LOG_WRITE_VV OP_VV
class VVV
op-types I X R
eval LogWritePre(LogEnum($1), $2)
LogWriteResPost($$)
### Check that invoked correctly
internal-op Log-WriteC
side-effects OP_LOG_WRITE_CV OP_VC
class VCV
op-types I X R
eval LogWritePre(LogEnum($1), $2)
LogWriteResPost($$)
# Versions that discard the return value.
internal-op Log-Write
side-effects
op1-read
classes VV CV
op-types X R
eval LogWritePre(LogEnum($1), $2)
(void) log_mgr->Write(id->AsEnumVal(), columns->AsRecordVal());
internal-op Broker-Flush-Logs
side-effects OP_BROKER_FLUSH_LOGS_X OP_X
class V
op-types U
eval $$ = broker_mgr->FlushLogBuffers();
internal-op Broker-Flush-Logs
side-effects
class X
eval (void) broker_mgr->FlushLogBuffers();
internal-op Get-Port-Transport-Proto
class VV
op-types U U
eval auto mask = $1 & PORT_SPACE_MASK;
auto v = 0; /* TRANSPORT_UNKNOWN */
if ( mask == TCP_PORT_MASK )
v = 1;
else if ( mask == UDP_PORT_MASK )
v = 2;
else if ( mask == ICMP_PORT_MASK )
v = 3;
$$ = v;
predicate-op Conn-Exists
class V
op-types R
eval session_mgr->FindConnection($1) != nullptr
internal-op Lookup-Conn
class VV
op-types X R
eval auto cid = $1;
Connection* conn = session_mgr->FindConnection(cid);
ValPtr res;
if ( conn )
res = conn->GetVal();
else
{
ERROR2("connection ID not a known connection", cid);
res = build_dummy_conn_record();
}
AssignTarget($$, ZVal(res, res->GetType()));
predicate-op Is-ICMP-Port
class V
op-types U
eval ($1 & PORT_SPACE_MASK) == ICMP_PORT_MASK
predicate-op Is-TCP-Port
class V
op-types U
eval ($1 & PORT_SPACE_MASK) == TCP_PORT_MASK
predicate-op Is-UDP-Port
class V
op-types U
eval ($1 & PORT_SPACE_MASK) == UDP_PORT_MASK
predicate-op Is-V4-Addr
class V
op-types A
eval $1->AsAddr().GetFamily() == IPv4
predicate-op Is-V6-Addr
class V
op-types A
eval $1->AsAddr().GetFamily() == IPv6
internal-op Network-Time
class V
op-types D
eval $$ = run_state::network_time;
internal-op Current-Time
class V
op-types D
eval $$ = util::current_time();
predicate-op Reading-Live-Traffic
class X
eval run_state::reading_live
predicate-op Reading-Traces
class X
eval run_state::reading_traces
internal-op Sort
op1-read
class V
op-types V
eval if ( $1->Size() > 1 )
$1->Sort();
internal-op Sort
class VV
op-types V V
eval auto vv = $1;
if ( vv->Size() > 1 )
vv->Sort();
zeek::Ref(vv);
Unref($$);
$$ = vv;
internal-op Sort-With-Cmp
op1-read
class VV
op-types V F
eval if ( $1->Size() > 1 )
$1->Sort($2);
internal-op Sort-With-Cmp
class VVV
op-types V V F
eval auto vv = $1;
if ( vv->Size() > 1 )
vv->Sort($2);
zeek::Ref(vv);
Unref($$);
$$ = vv;
internal-op Starts-With
classes VVV VCV VVC
op-types I S S
eval auto str = $1;
auto sub = $2;
auto str_n = str->Len();
auto sub_n = sub->Len();
if ( str_n < sub_n )
$$ = 0;
else
{
auto str_b = str->Bytes();
auto sub_b = sub->Bytes();
int i;
for ( i = 0; i < sub_n; ++i )
if ( str_b[i] != sub_b[i] )
break;
$$ = i == sub_n;
}
internal-op StrCmp
classes VVV VCV VVC
op-types I S S
eval auto s1 = $1;
auto s2 = $2;
$$ = Bstr_cmp(s1->AsString(), s2->AsString());
internal-op StrStr
classes VVV VCV VVC
op-types I S S
eval auto big = $1;
auto little = $2;
$$ = 1 + big->AsString()->FindSubstring(little->AsString());
macro Cat1Op(lhs, val)
auto& v1 = lhs;
ZVal::DeleteManagedType(v1);
v1 = val;
internal-op Cat1
classes VV VC
eval Cat1Op($$, $1)
zeek::Ref(v1.AsString());
internal-op Cat1Full
classes VV VC
eval auto formatted_val = ZVal(ZAM_val_cat($1.ToVal(Z_TYPE)));
Cat1Op($$, formatted_val)
internal-op CatN
class V
eval CatNPre()
int n = aux->n;
size_t max_size = 0;
for ( int i = 0; i < n; ++i )
max_size += ca[i]->MaxSize(aux->elems[i].ToDirectZVal(frame));
auto res = new char[max_size + /* slop */ n + 1];
auto res_p = res;
for ( int i = 0; i < n; ++i )
ca[i]->RenderInto(aux->elems[i].ToDirectZVal(frame), res_p);
*res_p = '\0';
auto s = new String(true, reinterpret_cast<byte_vec>(res), res_p - res);
Cat1Op($$, ZVal(new StringVal(s)))
macro CatNPre()
auto aux = Z_AUX;
auto& ca = aux->cat_args;
macro CatNMid()
auto res = new char[max_size + /* slop */ 10];
auto res_p = res;
macro CatNPost(lhs)
*res_p = '\0';
auto s = new String(true, reinterpret_cast<byte_vec>(res), res_p - res);
Cat1Op(lhs, ZVal(new StringVal(s)))
internal-op Cat2
class V
eval CatNPre()
size_t max_size = ca[0]->MaxSize(aux->elems[0].ToDirectZVal(frame));
max_size += ca[1]->MaxSize(aux->elems[1].ToDirectZVal(frame));
CatNMid()
ca[0]->RenderInto(aux->elems[0].ToDirectZVal(frame), res_p);
ca[1]->RenderInto(aux->elems[1].ToDirectZVal(frame), res_p);
CatNPost($$)
internal-op Cat3
class V
eval CatNPre()
size_t max_size = ca[0]->MaxSize(aux->elems[0].ToDirectZVal(frame));
max_size += ca[1]->MaxSize(aux->elems[1].ToDirectZVal(frame));
max_size += ca[2]->MaxSize(aux->elems[2].ToDirectZVal(frame));
CatNMid()
ca[0]->RenderInto(aux->elems[0].ToDirectZVal(frame), res_p);
ca[1]->RenderInto(aux->elems[1].ToDirectZVal(frame), res_p);
ca[2]->RenderInto(aux->elems[2].ToDirectZVal(frame), res_p);
CatNPost($$)
internal-op Cat4
class V
eval CatNPre()
size_t max_size = ca[0]->MaxSize(aux->elems[0].ToDirectZVal(frame));
max_size += ca[1]->MaxSize(aux->elems[1].ToDirectZVal(frame));
max_size += ca[2]->MaxSize(aux->elems[2].ToDirectZVal(frame));
max_size += ca[3]->MaxSize(aux->elems[3].ToDirectZVal(frame));
CatNMid()
ca[0]->RenderInto(aux->elems[0].ToDirectZVal(frame), res_p);
ca[1]->RenderInto(aux->elems[1].ToDirectZVal(frame), res_p);
ca[2]->RenderInto(aux->elems[2].ToDirectZVal(frame), res_p);
ca[3]->RenderInto(aux->elems[3].ToDirectZVal(frame), res_p);
CatNPost($$)
internal-op Cat5
class V
eval CatNPre()
size_t max_size = ca[0]->MaxSize(aux->elems[0].ToDirectZVal(frame));
max_size += ca[1]->MaxSize(aux->elems[1].ToDirectZVal(frame));
max_size += ca[2]->MaxSize(aux->elems[2].ToDirectZVal(frame));
max_size += ca[3]->MaxSize(aux->elems[3].ToDirectZVal(frame));
max_size += ca[4]->MaxSize(aux->elems[4].ToDirectZVal(frame));
CatNMid()
ca[0]->RenderInto(aux->elems[0].ToDirectZVal(frame), res_p);
ca[1]->RenderInto(aux->elems[1].ToDirectZVal(frame), res_p);
ca[2]->RenderInto(aux->elems[2].ToDirectZVal(frame), res_p);
ca[3]->RenderInto(aux->elems[3].ToDirectZVal(frame), res_p);
ca[4]->RenderInto(aux->elems[4].ToDirectZVal(frame), res_p);
CatNPost($$)
internal-op Cat6
class V
eval CatNPre()
size_t max_size = ca[0]->MaxSize(aux->elems[0].ToDirectZVal(frame));
max_size += ca[1]->MaxSize(aux->elems[1].ToDirectZVal(frame));
max_size += ca[2]->MaxSize(aux->elems[2].ToDirectZVal(frame));
max_size += ca[3]->MaxSize(aux->elems[3].ToDirectZVal(frame));
max_size += ca[4]->MaxSize(aux->elems[4].ToDirectZVal(frame));
max_size += ca[5]->MaxSize(aux->elems[5].ToDirectZVal(frame));
CatNMid()
ca[0]->RenderInto(aux->elems[0].ToDirectZVal(frame), res_p);
ca[1]->RenderInto(aux->elems[1].ToDirectZVal(frame), res_p);
ca[2]->RenderInto(aux->elems[2].ToDirectZVal(frame), res_p);
ca[3]->RenderInto(aux->elems[3].ToDirectZVal(frame), res_p);
ca[4]->RenderInto(aux->elems[4].ToDirectZVal(frame), res_p);
ca[5]->RenderInto(aux->elems[5].ToDirectZVal(frame), res_p);
CatNPost($$)
internal-op Cat7
class V
eval CatNPre()
size_t max_size = ca[0]->MaxSize(aux->elems[0].ToDirectZVal(frame));
max_size += ca[1]->MaxSize(aux->elems[1].ToDirectZVal(frame));
max_size += ca[2]->MaxSize(aux->elems[2].ToDirectZVal(frame));
max_size += ca[3]->MaxSize(aux->elems[3].ToDirectZVal(frame));
max_size += ca[4]->MaxSize(aux->elems[4].ToDirectZVal(frame));
max_size += ca[5]->MaxSize(aux->elems[5].ToDirectZVal(frame));
max_size += ca[6]->MaxSize(aux->elems[6].ToDirectZVal(frame));
CatNMid()
ca[0]->RenderInto(aux->elems[0].ToDirectZVal(frame), res_p);
ca[1]->RenderInto(aux->elems[1].ToDirectZVal(frame), res_p);
ca[2]->RenderInto(aux->elems[2].ToDirectZVal(frame), res_p);
ca[3]->RenderInto(aux->elems[3].ToDirectZVal(frame), res_p);
ca[4]->RenderInto(aux->elems[4].ToDirectZVal(frame), res_p);
ca[5]->RenderInto(aux->elems[5].ToDirectZVal(frame), res_p);
ca[6]->RenderInto(aux->elems[6].ToDirectZVal(frame), res_p);
CatNPost($$)
internal-op Cat8
class V
eval CatNPre()
size_t max_size = ca[0]->MaxSize(aux->elems[0].ToDirectZVal(frame));
max_size += ca[1]->MaxSize(aux->elems[1].ToDirectZVal(frame));
max_size += ca[2]->MaxSize(aux->elems[2].ToDirectZVal(frame));
max_size += ca[3]->MaxSize(aux->elems[3].ToDirectZVal(frame));
max_size += ca[4]->MaxSize(aux->elems[4].ToDirectZVal(frame));
max_size += ca[5]->MaxSize(aux->elems[5].ToDirectZVal(frame));
max_size += ca[6]->MaxSize(aux->elems[6].ToDirectZVal(frame));
max_size += ca[7]->MaxSize(aux->elems[7].ToDirectZVal(frame));
CatNMid()
ca[0]->RenderInto(aux->elems[0].ToDirectZVal(frame), res_p);
ca[1]->RenderInto(aux->elems[1].ToDirectZVal(frame), res_p);
ca[2]->RenderInto(aux->elems[2].ToDirectZVal(frame), res_p);
ca[3]->RenderInto(aux->elems[3].ToDirectZVal(frame), res_p);
ca[4]->RenderInto(aux->elems[4].ToDirectZVal(frame), res_p);
ca[5]->RenderInto(aux->elems[5].ToDirectZVal(frame), res_p);
ca[6]->RenderInto(aux->elems[6].ToDirectZVal(frame), res_p);
ca[7]->RenderInto(aux->elems[7].ToDirectZVal(frame), res_p);
CatNPost($$)
internal-op Analyzer-Name
classes VV VC
op-types S X
eval auto atype = $1.ToVal(Z_TYPE);
auto val = atype->AsEnumVal();
Unref($$);
plugin::Component* component = zeek::analyzer_mgr->Lookup(val);
if ( ! component )
component = zeek::packet_mgr->Lookup(val);
if ( ! component )
component = zeek::file_mgr->Lookup(val);
if ( component )
$$ = new StringVal(component->CanonicalName());
else
$$ = new StringVal("<error>");
macro FilesAddOrRemoveAnalyzer(file_id_val, tag, args_val, METHOD)
auto file_id = file_id_val;
using zeek::BifType::Record::Files::AnalyzerArgs;
auto rv = args_val->CoerceTo(AnalyzerArgs);
bool result = zeek::file_mgr->METHOD(
file_id->CheckString(),
zeek::file_mgr->GetComponentTag(tag.ToVal(Z_TYPE).get()),
std::move(rv));
macro FilesAddAnalyzer(file_id_val, tag, args_val)
FilesAddOrRemoveAnalyzer(file_id_val, tag, args_val, AddAnalyzer)
internal-op Files-Add-Analyzer
op1-read
classes VVV VCV
op-types S X R
eval FilesAddAnalyzer($1, $2, $3)
internal-op Files-Add-Analyzer
class VVVV
side-effects OP_FILES_ADD_ANALYZER_VVV OP_VVV
op-types I S X R
eval FilesAddAnalyzer($1, $2, $3)
$$ = result;
internal-op Files-Add-Analyzer
class VVCV
op-types I S X R
side-effects OP_FILES_ADD_ANALYZER_VCV OP_VVC
eval FilesAddAnalyzer($1, $2, $3)
$$ = result;
macro FilesRemoveAnalyzer(file_id_val, tag, args_slot)
FilesAddOrRemoveAnalyzer(file_id_val, tag, args_slot, RemoveAnalyzer)
internal-op Files-Remove-Analyzer
op1-read
classes VVV VCV
op-types S X R
eval FilesRemoveAnalyzer($1, $2, $3)
internal-op Files-Remove-Analyzer
class VVVV
op-types I S X R
side-effects OP_FILES_REMOVE_ANALYZER_VVV OP_VVV
eval FilesRemoveAnalyzer($1, $2, $3)
$$ = result;
internal-op Files-Remove-Analyzer
class VVCV
op-types I S X R
side-effects OP_FILES_REMOVE_ANALYZER_VCV OP_VVC
eval FilesRemoveAnalyzer($1, $2, $3)
$$ = result;
internal-op Analyzer-Enabled
classes VV VC
op-types I X
eval auto atype = $1.ToVal(Z_TYPE);
auto c = zeek::file_mgr->Lookup(atype->AsEnumVal());
$$ = c && c->Enabled();
internal-op File-Analyzer-Name
classes VV VC
eval auto atype = $1.ToVal(Z_TYPE);
Unref($$.AsString());
$$ = ZVal(file_mgr->GetComponentNameVal({NewRef{}, atype->AsEnumVal()}));
internal-op Is-Protocol-Analyzer
classes VV VC
op-types I X
eval auto atype = $1.ToVal(Z_TYPE);
$$ = analyzer_mgr->Lookup(atype->AsEnumVal()) != nullptr;
internal-op Clear-Table
op1-read
class V
op-types T
eval $1->RemoveAll();
internal-op Files-Enable-Reassembly
op1-read
class V
op-types S
eval auto f = $1->CheckString();
file_mgr->EnableReassembly(f);
internal-op Files-Set-Reassembly-Buffer
op1-read
classes VV Vi
op-types S U
eval auto f = $1->CheckString();
file_mgr->SetReassemblyBuffer(f, $2);
internal-op Files-Set-Reassembly-Buffer
class VVV
op-types I S U
side-effects OP_FILES_SET_REASSEMBLY_BUFFER_VV OP_VV
eval auto f = $1->CheckString();
$$ = file_mgr->SetReassemblyBuffer(f, $2);
internal-op Files-Set-Reassembly-Buffer
class VVi
op-types I S U
side-effects OP_FILES_SET_REASSEMBLY_BUFFER_Vi OP_VV_I2
eval auto f = $1->CheckString();
$$ = file_mgr->SetReassemblyBuffer(f, $2);
internal-op Get-Bytes-Thresh
classes VVV VVC
op-types U R I
eval auto a = analyzer::conn_size::GetConnsizeAnalyzer($1);
auto res = 0U;
if ( a )
res = static_cast<analyzer::conn_size::ConnSize_Analyzer*>(a)->GetByteAndPacketThreshold(true, $2);
$$ = res;
macro SetBytesThresh(cid, threshold, is_orig)
bool res = false;
auto a = analyzer::conn_size::GetConnsizeAnalyzer(cid);
if ( a )
{
static_cast<analyzer::conn_size::ConnSize_Analyzer*>(a)->SetByteAndPacketThreshold(threshold, true, is_orig);
res = true;
}
internal-op Set-Bytes-Thresh
op1-read
classes VVV VVC VCV VCi
op-types R U I
eval SetBytesThresh($1, $2, $3)
internal-op Set-Bytes-Thresh
class VVVV
op-types I R U I
side-effects OP_SET_BYTES_THRESH_VVV OP_VVV
eval SetBytesThresh($1, $2, $3)
$$ = res;
internal-op Set-Bytes-Thresh
class VVVC
op-types I R U I
side-effects OP_SET_BYTES_THRESH_VVC OP_VVC
eval SetBytesThresh($1, $2, $3)
$$ = res;
internal-op Set-Bytes-Thresh
class VVCV
op-types I R U I
side-effects OP_SET_BYTES_THRESH_VCV OP_VVC
eval SetBytesThresh($1, $2, $3)
$$ = res;
internal-op Set-Bytes-Thresh
class VVCi
op-types I R U I
side-effects OP_SET_BYTES_THRESH_VCi OP_VVC_I2
eval SetBytesThresh($1, $2, $3)
$$ = res;

89
src/script_opt/ZAM/OPs/aggr-assignments.op Normal file
View file

@ -0,0 +1,89 @@
# Operations corresponding to assigning to elements of aggregates.
macro VectorElemAssignPre(vec, index)
auto ind = index.AsCount();
auto vv = vec.AsVector();
macro EvalVectorElemAssign(vec, index, val_setup, assign_op)
VectorElemAssignPre(vec, index)
val_setup
if ( ! assign_op )
ERROR("value used but not set");
op Vector-Elem-Assign
op1-read
set-type $1
class VVV
eval EvalVectorElemAssign($1, $2,, copy_vec_elem(vv, ind, $3, Z_TYPE))
op Any-Vector-Elem-Assign
op1-read
set-type $1
classes VVV VVC
eval EvalVectorElemAssign($1, $2,, vv->Assign(ind, $3.ToVal(Z_TYPE)))
op Vector-Elem-Assign-Any
op1-read
class VVV
op-types X X a
eval EvalVectorElemAssign($1, $2, auto any_v = $3;, vv->Assign(ind, {NewRef{}, any_v}))
op Vector-Elem-Assign
op1-read
set-type $2
class VVC
eval VectorElemAssignPre($1, $2)
(void) copy_vec_elem(vv, ind, $3, Z_TYPE);
# These versions are used when the constant is the index, not the new value.
op Vector-Elem-Assign
op1-read
set-type $1
class VVi
op-types V X U
eval auto vv = $1;
if ( ! copy_vec_elem(vv, $3, $2, Z_TYPE) )
ERROR("value used but not set");
op Any-Vector-Elem-Assign
op1-read
set-type $1
class VVi
op-types V X I
eval auto vv = $1;
if ( ! vv->Assign($3, $2.ToVal(Z_TYPE)) )
ERROR("value used but not set");
op Vector-Elem-Assign-Any
op1-read
class VVi
op-types V a I
eval auto vv = $1;
auto any_v = $2;
vv->Assign($3, {NewRef{}, any_v});
internal-op Vector-Slice-Assign
op1-read
class VV
op-types V V
eval ValPtr vec = {NewRef{}, $1};
auto indices = Z_AUX->ToListVal(frame);
ValPtr vals = {NewRef{}, $2};
bool iterators_invalidated;
auto error = assign_to_index(std::move(vec), std::move(indices), std::move(vals), iterators_invalidated);
if ( error )
ERROR(error);
if ( iterators_invalidated )
WARN("possible loop/iterator invalidation");
internal-op Table-Elem-Assign
op1-read
classes VV VC
op-types T X
eval auto indices = Z_AUX->ToListVal(frame);
auto val = $2.ToVal(Z_TYPE);
bool iterators_invalidated = false;
$1->Assign(std::move(indices), std::move(val), true, &iterators_invalidated);
if ( iterators_invalidated )
WARN("possible loop/iterator invalidation");

104
src/script_opt/ZAM/OPs/binary-exprs.op Normal file
View file

@ -0,0 +1,104 @@
# Operations corresponding to binary expressions.
binary-expr-op Add
op-type I U D S
vector
eval $1 + $2
eval-type S vector<const String*> strings;
strings.push_back($1->AsString());
strings.push_back($2->AsString());
auto res = new StringVal(concatenate(strings));
$$ = res;
binary-expr-op Sub
op-type I U D T
vector
eval $1 - $2
#
eval-type T auto v = $1->Clone();
auto s = v.release()->AsTableVal();
$2->RemoveFrom(s);
$$ = s;
binary-expr-op Times
op-type I U D
vector
eval $1 * $2
binary-expr-op Divide
op-type I U D
vector
#
precheck $2 == 0
precheck-action ERROR("division by zero");
eval $1 / $2
binary-expr-op Mask
# Signal that this expression only has mixed-type evaluation.
op-type X
explicit-result-type
eval-mixed A I auto mask = static_cast<uint32_t>($2);
auto a = $1->AsAddr();
if ( a.GetFamily() == IPv4 && mask > 32 )
ERROR(util::fmt("bad IPv4 subnet prefix length: %" PRIu32, mask));
if ( a.GetFamily() == IPv6 && mask > 128 )
ERROR(util::fmt("bad IPv6 subnet prefix length: %" PRIu32, mask));
auto v = make_intrusive<SubNetVal>(a, mask);
Unref($$.AsSubNet());
$$.AsSubNetRef() = v.release();
binary-expr-op Mod
op-type I U
vector
precheck $2 == 0
precheck-action ERROR("modulo by zero");
eval $1 % $2
binary-expr-op And-And
op-type I
vector
eval zeek_int_t($1 && $2)
binary-expr-op Or-Or
op-type I
vector
eval zeek_int_t($1 || $2)
binary-expr-op And
op-type U P T
vector
eval $1 & $2
#
eval-type P $$ = new PatternVal(RE_Matcher_conjunction($1->AsPattern(), $2->AsPattern()));
#
eval-type T $$ = $1->Intersection(*$2).release();
binary-expr-op Or
op-type U P T
vector
eval $1 | $2
#
eval-type P $$ = new PatternVal(RE_Matcher_disjunction($1->AsPattern(), $2->AsPattern()));
#
eval-type T auto v = $1->Clone();
auto s = v.release()->AsTableVal();
(void) $2->AddTo(s, false, false);
$$ = s;
binary-expr-op Xor
op-type U
vector
eval $1 ^ $2
binary-expr-op Lshift
op-type I U
vector
eval-type I if ( $1 < 0 )
ERROR("left shifting a negative number is undefined");
$$ = $1 << $2;
eval $1 << $2
binary-expr-op Rshift
op-type I U
vector
eval $1 >> $2

180
src/script_opt/ZAM/OPs/calls.op Normal file
View file

@ -0,0 +1,180 @@
# Operations corresponding to function calls.
# A call with no arguments and no return value.
internal-op Call0
op1-read
class X
side-effects
num-call-args 0
# A call with no arguments and a return value.
internal-assignment-op Call0
class V
side-effects OP_CALL0_X OP_X
assign-val v
num-call-args 0
# Calls with 1 argument and no return value.
internal-op Call1
op1-read
classes V C
side-effects
num-call-args 1
# Same but with a return value.
internal-assignment-op Call1
class VV
side-effects OP_CALL1_V OP_V
assign-val v
num-call-args 1
internal-assignment-op Call1
class VC
side-effects OP_CALL1_C OP_C
assign-val v
num-call-args 1
# Calls with 2-5 arguments and no return value.
internal-op Call2
class X
side-effects
num-call-args 2
# Same with a return value.
internal-assignment-op Call2
class V
side-effects OP_CALL2_X OP_X
assign-val v
num-call-args 2
internal-op Call3
class X
side-effects
num-call-args 3
# Same with a return value.
internal-assignment-op Call3
class V
side-effects OP_CALL3_X OP_X
assign-val v
num-call-args 3
internal-op Call4
class X
side-effects
num-call-args 4
# Same with a return value.
internal-assignment-op Call4
class V
side-effects OP_CALL4_X OP_X
assign-val v
num-call-args 4
internal-op Call5
class X
side-effects
num-call-args 5
# Same with a return value.
internal-assignment-op Call5
class V
side-effects OP_CALL5_X OP_X
assign-val v
num-call-args 5
# ... and with an arbitrary number of arguments.
internal-op CallN
class X
side-effects
num-call-args n
# Same with a return value.
internal-assignment-op CallN
class V
side-effects OP_CALLN_X OP_X
assign-val v
num-call-args n
# Same, but for indirect calls via a global variable.
internal-op IndCallN
class X
side-effects
indirect-call
num-call-args n
# Same with a return value.
internal-assignment-op IndCallN
class V
side-effects OP_INDCALLN_X OP_X
assign-val v
indirect-call
num-call-args n
# And versions with a local variable rather than a global.
internal-op Local-IndCallN
op1-read
class V
side-effects
indirect-local-call
num-call-args n
internal-assignment-op Local-IndCallN
class VV
side-effects OP_LOCAL_INDCALLN_V OP_V
assign-val v
indirect-local-call
num-call-args n
# A call made in a "when" context. These always have assignment targets.
# To keep things simple, we just use one generic flavor (for N arguments,
# doing a less-streamlined-but-simpler Val-based assignment).
macro WhenCall(lhs, func)
if ( ! func )
throw ZAMDelayedCallException();
auto trigger = Z_FRAME->GetTrigger();
Val* v = trigger ? trigger->Lookup(Z_AUX->call_expr.get()) : nullptr;
ValPtr vp;
if ( v )
vp = {NewRef{}, v};
else
{
auto aux = Z_AUX;
auto current_assoc = Z_FRAME->GetTriggerAssoc();
auto n = aux->n;
std::vector<ValPtr> args;
for ( auto i = 0; i < n; ++i )
args.push_back(aux->ToVal(frame, i));
Z_FRAME->SetCall(Z_AUX->call_expr.get());
/* It's possible that this function will call another that
* itself returns null because *it* is the actual blocker.
* That will set ZAM_error, which we need to ignore.
*/
auto hold_ZAM_error = ZAM_error;
vp = func->Invoke(&args, Z_FRAME);
ZAM_error = hold_ZAM_error;
Z_FRAME->SetTriggerAssoc(current_assoc);
if ( ! vp )
throw ZAMDelayedCallException();
}
if ( Z_IS_MANAGED )
ZVal::DeleteManagedType(lhs);
lhs = ZVal(vp, Z_TYPE);
internal-op WhenCallN
class V
side-effects
eval WhenCall($$, Z_AUX_FUNC)
internal-op WhenIndCallN
class VV
op-types X F
side-effects
eval WhenCall($$, $1)
# Form for when we need to look up the function value at run-time.
internal-op When-ID-IndCallN
class V
side-effects
eval WhenCall($$, Z_AUX_ID->GetVal()->AsFunc())

151
src/script_opt/ZAM/OPs/coercions.op Normal file
View file

@ -0,0 +1,151 @@
# Operations corresponding to type coercions.
direct-unary-op Arith-Coerce ArithCoerce
internal-op Coerce-UI
class VV
op-types U I
eval auto v = $1;
if ( v < 0 )
ERROR("underflow converting int to count");
else
$$ = zeek_uint_t(v);
internal-op Coerce-UD
class VV
op-types U D
eval auto v = $1;
if ( v < 0.0 )
ERROR("underflow converting double to count");
else if ( v > static_cast<double>(UINT64_MAX) )
ERROR("overflow converting double to count");
else
$$ = zeek_uint_t(v);
internal-op Coerce-IU
class VV
op-types I U
eval auto v = $1;
if ( v > INT64_MAX )
ERROR("overflow converting count to int");
else
$$ = zeek_int_t(v);
internal-op Coerce-ID
class VV
op-types I D
eval auto v = $1;
if ( v < static_cast<double>(INT64_MIN) )
ERROR("underflow converting double to int");
else if ( v > static_cast<double>(INT64_MAX) )
ERROR("overflow converting double to int");
else
$$ = zeek_int_t(v);
internal-op Coerce-DI
class VV
op-types D I
eval $$ = double($1);
internal-op Coerce-DU
class VV
op-types D U
eval $$ = double($1);
macro EvalCoerceVec(lhs, rhs, coercer)
auto old_v1 = lhs.AsVector();
lhs.AsVectorRef() = coercer(rhs.AsVector(), Z_LOC);
Unref(old_v1); /* delayed to allow for same value on both sides */
internal-op Coerce-UI-Vec
class VV
eval EvalCoerceVec($$, $1, vec_coerce_UI)
internal-op Coerce-UD-Vec
class VV
eval EvalCoerceVec($$, $1, vec_coerce_UD)
internal-op Coerce-IU-Vec
class VV
eval EvalCoerceVec($$, $1, vec_coerce_IU)
internal-op Coerce-ID-Vec
class VV
eval EvalCoerceVec($$, $1, vec_coerce_ID)
internal-op Coerce-DI-Vec
class VV
eval EvalCoerceVec($$, $1, vec_coerce_DI)
internal-op Coerce-DU-Vec
class VV
eval EvalCoerceVec($$, $1, vec_coerce_DU)
direct-unary-op Record-Coerce RecordCoerce
internal-op Record-Coerce
class VV
op-types R R
eval auto rt = cast_intrusive<RecordType>(Z_TYPE);
auto v = $1;
auto to_r = coerce_to_record(std::move(rt), v, Z_AUX_MAP);
Unref($$);
$$ = to_r.release();
direct-unary-op Table-Coerce TableCoerce
internal-op Table-Coerce
class VV
op-types T T
eval auto tv = $1;
if ( tv->Size() > 0 )
ERROR("coercion of non-empty table/set");
else
{
auto tt = cast_intrusive<TableType>(Z_TYPE);
AttributesPtr attrs = tv->GetAttrs();
auto t = make_intrusive<TableVal>(tt, attrs);
Unref($$);
$$ = t.release();
}
direct-unary-op Vector-Coerce VectorCoerce
internal-op Vector-Coerce
class VV
op-types V V
eval if ( $1->Size() > 0 )
ERROR("coercion of non-empty vector");
else
{
auto vv = new VectorVal(cast_intrusive<VectorType>(Z_TYPE));
Unref($$);
$$ = vv;
}
unary-expr-op To-Any-Coerce
op-type X
set-type $1
eval AssignTarget($$, ZVal($1.ToVal(Z_TYPE), ZAM::any_base_type))
unary-expr-op From-Any-Coerce
no-const
op-type X
set-type $$
eval auto v = $1.AsAny();
AssignTarget($$, ZVal({NewRef{}, v}, Z_TYPE))
unary-expr-op From-Any-Vec-Coerce
no-const
op-type X
set-type $$
eval auto vv = $1.AsVector();
if ( ! vv->Concretize(Z_TYPE->Yield()) )
ERROR("incompatible vector-of-any");
else
{
zeek::Ref(vv);
AssignTarget($$, ZVal(vv))
}

263
src/script_opt/ZAM/OPs/constructors.op Normal file
View file

@ -0,0 +1,263 @@
# Operations corresponding to aggregated constructors.
# Table construction requires atypical evaluation of list elements
# using information from their expression specifics.
direct-unary-op Table-Constructor ConstructTable
macro ConstructTableOrSetPre(width)
auto tt = cast_intrusive<TableType>(Z_TYPE);
auto new_t = new TableVal(tt, Z_AUX_ATTRS);
auto aux = Z_AUX;
auto n = aux->n;
auto ind_width = width;
macro ConstructTableOrSetPost(lhs)
auto& t = lhs.AsTableRef();
Unref(t);
t = new_t;
internal-op Construct-Table
class Vi
eval ConstructTableOrSetPre($1)
for ( auto i = 0; i < n; ++i )
{
auto indices = aux->ToIndices(frame, i, ind_width);
auto v = aux->ToVal(frame, i + ind_width);
new_t->Assign(indices, v);
i += ind_width;
}
ConstructTableOrSetPost($$)
# When tables are constructed, if their &default is a lambda with captures
# then we need to explicitly set up the default.
internal-op Set-Table-Default-Lambda
op1-read
class VV
op-types T X
eval auto tbl = $1;
auto lambda = $2.ToVal(Z_TYPE);
tbl->InitDefaultVal(std::move(lambda));
direct-unary-op Set-Constructor ConstructSet
internal-op Construct-Set
class Vi
eval ConstructTableOrSetPre($1)
for ( auto i = 0; i < n; i += ind_width )
{
auto indices = aux->ToIndices(frame, i, ind_width);
new_t->Assign(indices, nullptr);
}
ConstructTableOrSetPost($$)
direct-unary-op Record-Constructor ConstructRecord
direct-unary-op Rec-Construct-With-Rec ConstructRecordFromRecord
macro ConstructRecordPost(lhs)
Unref(lhs);
lhs = new RecordVal(cast_intrusive<RecordType>(Z_TYPE), std::move(init_vals));
op Construct-Direct-Record
class V
op-types R
eval auto init_vals = Z_AUX->ToZValVec(frame);
ConstructRecordPost($$)
op Construct-Known-Record
class V
op-types R
eval auto init_vals = Z_AUX->ToZValVecWithMap(frame);
ConstructRecordPost($$)
macro AssignFromRec(rhs)
/* The following is defined below, for use by Rec-Assign-Fields */
SetUpRecFieldOps(lhs_map)
auto is_managed = Z_AUX->is_managed;
for ( size_t i = 0U; i < n; ++i )
{
auto rhs_i = rhs->RawField(rhs_map[i]);
if ( is_managed[i] )
zeek::Ref(rhs_i.ManagedVal());
init_vals[lhs_map[i]] = rhs_i;
}
op Construct-Known-Record-From
class VV
op-types R R
eval auto init_vals = Z_AUX->ToZValVecWithMap(frame);
AssignFromRec($1)
ConstructRecordPost($$)
macro DoNetworkTimeInit(slot)
init_vals[slot] = ZVal(run_state::network_time);
op Construct-Known-Record-With-NT
class Vi
op-types R I
eval auto init_vals = Z_AUX->ToZValVecWithMap(frame);
DoNetworkTimeInit($1)
ConstructRecordPost($$)
op Construct-Known-Record-With-NT-From
class VVi
op-types R R I
eval auto init_vals = Z_AUX->ToZValVecWithMap(frame);
DoNetworkTimeInit($2)
AssignFromRec($1)
ConstructRecordPost($$)
macro GenInits()
auto init_vals = Z_AUX->ToZValVecWithMap(frame);
for ( auto& fi : *Z_AUX->field_inits )
init_vals[fi.first] = fi.second->Generate();
op Construct-Known-Record-With-Inits
class V
op-types R
eval GenInits()
ConstructRecordPost($$)
op Construct-Known-Record-With-Inits-From
class VV
op-types R R
eval GenInits()
AssignFromRec($1)
ConstructRecordPost($$)
op Construct-Known-Record-With-Inits-And-NT
class Vi
op-types R I
eval GenInits()
DoNetworkTimeInit($1)
ConstructRecordPost($$)
op Construct-Known-Record-With-Inits-And-NT-From
class VVi
op-types R R I
eval GenInits()
DoNetworkTimeInit($2)
AssignFromRec($1)
ConstructRecordPost($$)
macro SetUpRecFieldOps(which_lhs_map)
auto& lhs_map = Z_AUX->which_lhs_map;
auto& rhs_map = Z_AUX->rhs_map;
auto n = rhs_map.size();
op Rec-Assign-Fields
op1-read
class VV
op-types R R
eval SetUpRecFieldOps(map)
for ( size_t i = 0U; i < n; ++i )
$1->RawOptField(lhs_map[i]) = $2->RawField(rhs_map[i]);
macro DoManagedRecAssign(lhs, rhs)
auto is_managed = Z_AUX->is_managed;
for ( size_t i = 0U; i < n; ++i )
if ( is_managed[i] )
{
auto& lhs_i = lhs->RawOptField(lhs_map[i]);
auto rhs_i = rhs->RawField(rhs_map[i]);
zeek::Ref(rhs_i.ManagedVal());
if ( lhs_i )
ZVal::DeleteManagedType(*lhs_i);
lhs_i = rhs_i;
}
else
lhs->RawOptField(lhs_map[i]) = rhs->RawField(rhs_map[i]);
op Rec-Assign-Fields-Managed
op1-read
class VV
op-types R R
eval SetUpRecFieldOps(map)
DoManagedRecAssign($1, $2)
op Rec-Assign-Fields-All-Managed
op1-read
class VV
op-types R R
eval SetUpRecFieldOps(map)
for ( size_t i = 0U; i < n; ++i )
{
auto& lhs_i = $1->RawOptField(lhs_map[i]);
auto rhs_i = $2->RawField(rhs_map[i]);
zeek::Ref(rhs_i.ManagedVal());
if ( lhs_i )
ZVal::DeleteManagedType(*lhs_i);
lhs_i = rhs_i;
}
op Rec-Add-Int-Fields
op1-read
class VV
op-types R R
eval SetUpRecFieldOps(map)
for ( size_t i = 0U; i < n; ++i )
$1->RawField(lhs_map[i]).AsIntRef() += $2->RawField(rhs_map[i]).AsInt();
op Rec-Add-Double-Fields
op1-read
class VV
op-types R R
eval SetUpRecFieldOps(map)
for ( size_t i = 0U; i < n; ++i )
$1->RawField(lhs_map[i]).AsDoubleRef() += $2->RawField(rhs_map[i]).AsDouble();
op Rec-Add-Fields
op1-read
class VV
op-types R R
eval SetUpRecFieldOps(map)
auto& types = Z_AUX->types;
for ( size_t i = 0U; i < n; ++i )
{
auto& lhs_i = $1->RawField(lhs_map[i]);
auto rhs_i = $2->RawField(rhs_map[i]);
auto tag = types[i]->Tag();
if ( tag == TYPE_INT )
lhs_i.AsIntRef() += rhs_i.AsInt();
else if ( tag == TYPE_COUNT )
lhs_i.AsCountRef() += rhs_i.AsCount();
else
lhs_i.AsDoubleRef() += rhs_i.AsDouble();
}
# Special instruction for concretizing vectors that are fields in a
# newly-constructed record. "aux" holds which fields in the record to
# inspect.
op Concretize-Vector-Fields
op1-read
class V
op-types R
eval auto rt = cast_intrusive<RecordType>(Z_TYPE);
auto r = $1;
auto aux = Z_AUX;
auto n = aux->n;
for ( auto i = 0; i < n; ++i )
{
auto ind = aux->elems[i].IntVal();
auto v_i = r->GetField(ind);
ASSERT(v_i);
if ( v_i->GetType<VectorType>()->IsUnspecifiedVector() )
{
const auto& t_i = rt->GetFieldType(ind);
v_i->AsVectorVal()->Concretize(t_i->Yield());
}
}
direct-unary-op Vector-Constructor ConstructVector
internal-op Construct-Vector
class V
op-types V
eval auto new_vv = new VectorVal(cast_intrusive<VectorType>(Z_TYPE));
auto aux = Z_AUX;
auto n = aux->n;
for ( auto i = 0; i < n; ++i )
new_vv->Assign(i, aux->ToVal(frame, i));
auto& vv = $$;
Unref(vv);
vv = new_vv;

212
src/script_opt/ZAM/OPs/indexing.op Normal file
View file

@ -0,0 +1,212 @@
# Operations corresponding to indexing of tables, vectors, strings,
# and "any" values.
op IndexVecBoolSelect
classes VVV VCV
op-types V V V
set-type $$
eval if ( $1->Size() != $2->Size() )
ERROR("size mismatch, boolean index and vector");
else
{
auto vt = cast_intrusive<VectorType>(Z_TYPE);
auto v2 = $1;
auto v3 = $2;
auto v = vector_bool_select(std::move(vt), v2, v3);
Unref($$);
$$ = v.release();
}
op IndexVecIntSelect
classes VVV VCV
op-types V V V
set-type $$
eval auto vt = cast_intrusive<VectorType>(Z_TYPE);
auto v2 = $1;
auto v3 = $2;
auto v = vector_int_select(std::move(vt), v2, v3);
Unref($$);
$$ = v.release();
op Index
class VVL
custom-method return CompileIndex(n1, n2, l, false);
no-eval
op Index
class VCL
custom-method return CompileIndex(n, c, l, false);
no-eval
op WhenIndex
class VVL
custom-method return CompileIndex(n1, n2, l, true);
no-eval
op WhenIndex
class VCL
custom-method return CompileIndex(n, c, l, true);
no-eval
internal-op Index-Vec
class VVV
op-types X V I
eval EvalIndexVec($$, $1, $2)
macro EvalIndexVec(lhs, rhs_vec, index)
auto& vv = rhs_vec->RawVec();
zeek_int_t ind = index;
if ( ind < 0 )
ind += vv.size();
if ( ind < 0 || ind >= int(vv.size()) )
ERROR("no such index");
AssignTarget(lhs, CopyVal(*(vv[ind])))
internal-op Index-VecC
class VVi
op-types X V I
eval EvalIndexVec($$, $1, $2)
internal-op Index-Any-Vec
class VVV
op-types X V I
eval EvalIndexAnyVec($$, $1, $2)
macro EvalIndexAnyVec(lhs, vec, index)
auto vv = vec;
zeek_int_t ind = index;
if ( ind < 0 )
ind += vv->Size();
if ( ind < 0 || ind >= int(vv->Size()) )
ERROR("no such index");
AssignTarget(lhs, ZVal(vv->ValAt(ind).release()))
internal-op Index-Any-VecC
class VVi
op-types X V I
eval EvalIndexAnyVec($$, $1, $2)
macro WhenIndexResCheck(vec)
if ( vec && IndexExprWhen::evaluating > 0 )
IndexExprWhen::results.push_back({NewRef{}, vec});
internal-op When-Index-Vec
class VVV
op-types X V I
eval EvalIndexAnyVec($$, $1, $2)
WhenIndexResCheck($$.AsVector())
internal-op When-Index-VecC
class VVi
op-types X V I
eval EvalIndexAnyVec($$, $1, $2)
WhenIndexResCheck($$.AsVector())
macro EvalVecSlice(lhs, vv)
auto vec = vv;
auto v = index_slice(vec, indices.get());
Unref(lhs);
lhs = v.release();
internal-op Index-Vec-Slice
class VV
op-types V V
eval auto indices = Z_AUX->ToListVal(frame);
EvalVecSlice($$, $1)
internal-op When-Index-Vec-Slice
class VV
op-types V V
eval auto indices = Z_AUX->ToListVal(frame);
EvalVecSlice($$, $1)
WhenIndexResCheck($$)
internal-op Table-Index
class VV
eval auto indices = Z_AUX->ToListVal(frame);
EvalTableIndex($1, indices)
if ( v )
AssignTarget($$, BuildVal(v, Z_TYPE))
internal-op Table-PatStr-Index
classes VVV VVC
op-types X T S
eval auto vec = ZVal($1->LookupPattern({NewRef{}, $2}));
ZVal::DeleteManagedType($$);
$$ = vec;
internal-op When-Table-Index
class VV
eval auto indices = Z_AUX->ToListVal(frame);
EvalTableIndex($1, indices)
if ( v )
{
if ( IndexExprWhen::evaluating > 0 )
IndexExprWhen::results.emplace_back(v);
AssignTarget($$, BuildVal(v, Z_TYPE))
}
macro EvalTableIndex(tbl, index)
auto v = tbl.AsTable()->FindOrDefault(index);
if ( ! v )
ERROR("no such index");
internal-op When-PatStr-Index
class VV
op-types X T
eval auto indices = Z_AUX->ToListVal(frame);
auto arg0 = indices->Idx(0);
auto v = $1->LookupPattern({NewRef{}, arg0->AsStringVal()});
if ( IndexExprWhen::evaluating > 0 )
IndexExprWhen::results.emplace_back(v);
AssignTarget($$, BuildVal(v, Z_TYPE))
internal-assignment-op Table-Index1
classes VVV VVC
assign-val v
eval EvalTableIndex($1, $2.ToVal(Z_TYPE))
# No AssignTarget needed, as this is an assignment-op
# This version is for a variable v3.
internal-op Index-String
class VVV
op-types S S I
eval EvalIndexString($$, $1, $2)
macro EvalIndexString(lhs, s, index)
auto str = s->AsString();
auto len = str->Len();
auto idx = index;
if ( idx < 0 )
idx += len;
auto v = str->GetSubstring(idx, 1);
Unref(lhs);
lhs = new StringVal(v ? v : new String(""));
# This version is for a constant v3.
internal-op Index-StringC
class VVi
op-types S S I
eval EvalIndexString($$, $1, $2)
internal-op Index-String-Slice
class VV
op-types S S
eval auto str = $1->AsString();
auto indices = Z_AUX->ToListVal(frame);
auto slice = index_string(str, indices.get());
Unref($$);
$$ = new StringVal(slice->ToStdString());
op AnyIndex
class VVi
op-types X a I
set-type $$
eval auto lv = $1->AsListVal();
if ( $2 < 0 || $2 >= lv->Length() )
reporter->InternalError("bad \"any\" element index");
ValPtr elem = lv->Idx($2);
if ( CheckAnyType(elem->GetType(), Z_TYPE, Z_LOC) )
AssignTarget($$, BuildVal(elem, Z_TYPE))
else
ZAM_error = true;

124
src/script_opt/ZAM/OPs/internal.op Normal file
View file

@ -0,0 +1,124 @@
# Internal operations not directly driven off of AST elements.
# These two are only needed for type-based switch statements. Could think
# about replacing them using CoerceFromAnyExpr.
op Assign-Any
classes VV VC
set-type $1
op-types a X
eval auto v = $1.ToVal(Z_TYPE);
$$ = v.release();
# Lazy way to assign without having to track the specific type of
# a constant.
internal-op Assign-Const
class VC
eval AssignTarget($$, BuildVal($1.ToVal(Z_TYPE), Z_TYPE))
internal-assignment-op Load-Val
class Vi
assign-val v
eval auto& v = Z_FRAME->GetElement($1);
internal-assignment-op Load-Global
# We don't use GlobalVal() for the assignment because we want to leverage
# the bookkeeping that assign-val gives us in terms of memory management.
class Vg
assign-val v
eval auto& v = GlobalID($1)->GetVal();
if ( ! v )
ERROR2("value used but not set", Z_AUX_ID.get());
# We need a special form here for loading global types, as they don't
# fit the usual template.
internal-op Load-Global-Type
class Vg
op-types t I
eval auto& v = $$;
Unref(v);
auto& t = GlobalID($1)->GetType();
v = new TypeVal(t, true);
internal-op Load-Capture
class Vi
eval $$ = Z_FRAME->GetFunction()->GetCapturesVec()[$1];
internal-op Load-Managed-Capture
class Vi
eval auto& lhs = $$;
auto& rhs = Z_FRAME->GetFunction()->GetCapturesVec()[$1];
zeek::Ref(rhs.ManagedVal());
ZVal::DeleteManagedType(lhs);
lhs = rhs;
internal-op Store-Global
op1-internal
class g
eval GlobalID($1)->SetVal(GlobalVal($1).ToVal(Z_TYPE));
# Both of these have the LHS as v2 not v1, to keep with existing
# conventions of OP_VV_I2 op type (as opposed to OP_VV_I1_V2, which doesn't
# currently exist, and would be a pain to add).
internal-op Store-Capture
op1-read
class Vi
eval Z_FRAME->GetFunction()->GetCapturesVec()[$2] = $1;
internal-op Store-Managed-Capture
op1-read
class Vi
eval auto& lhs = Z_FRAME->GetFunction()->GetCapturesVec()[$2];
auto& rhs = $1;
zeek::Ref(rhs.ManagedVal());
ZVal::DeleteManagedType(lhs);
lhs = rhs;
internal-op Copy-To
class VC
set-type $1
eval AssignTarget($$, CopyVal($1))
internal-op GoTo
class b
eval $1
internal-op Hook-Break
class X
eval flow = FLOW_BREAK;
pc = end_pc;
DO_ZAM_PROFILE
continue;
# Slot 2 gives frame size.
internal-op Lambda
class Vi
op-types F I
eval auto& primary_func = Z_AUX_PRIMARY_FUNC;
auto& body = primary_func->GetBodies()[0].stmts;
ASSERT(body->Tag() == STMT_ZAM);
auto lamb = make_intrusive<ScriptFunc>(Z_AUX_ID);
lamb->AddBody(body, $1);
lamb->SetName(Z_AUX_LAMBDA_NAME.c_str());
auto& aux = Z_AUX;
if ( aux->n > 0 )
{
auto captures = std::make_unique<std::vector<ZVal>>();
for ( auto i = 0; i < aux->n; ++i )
{
auto slot = aux->elems[i].Slot();
if ( slot >= 0 )
{
auto& cp = frame[slot];
if ( aux->elems[i].IsManaged() )
zeek::Ref(cp.ManagedVal());
captures->push_back(cp);
}
else
/* Used for when-locals. */
captures->push_back(ZVal());
}
lamb->CreateCaptures(std::move(captures));
}
Unref($$);
$$ = lamb.release();

124
src/script_opt/ZAM/OPs/iterations.op Normal file
View file

@ -0,0 +1,124 @@
# Operations corresponding to iterations.
internal-op Init-Table-Loop
op1-read
class Vf
op-types T I
eval $2.BeginLoop({NewRef{}, $1}, frame, Z_AUX);
internal-op Next-Table-Iter
op1-read
class fb
eval NextTableIterPre($1, $2)
$1.NextIter();
macro NextTableIterPre(iter, BRANCH)
if ( iter.IsDoneIterating() )
BRANCH
internal-op Next-Table-Iter-No-Vars
op1-read
class fb
eval NextTableIterPre($1, $2)
$1.IterFinished();
internal-op Next-Table-Iter-Val-Var
# v1 = slot of the "ValueVar"
class Vfb
eval NextTableIterPre($1, $2)
AssignTarget($$, $1.IterValue());
$1.NextIter();
internal-op Next-Table-Iter-Val-Var-No-Vars
# v1 = slot of the "ValueVar"
class Vfb
eval NextTableIterPre($1, $2)
AssignTarget($$, $1.IterValue());
$1.IterFinished();
internal-op Init-Vector-Loop
op1-read
class Vs
op-types V I
eval auto& vv = $1->RawVec();
$2.InitLoop(&vv);
macro NextVectorIterCore(info, BRANCH)
if ( info.IsDoneIterating() )
BRANCH
const auto& vv = *info.vv;
if ( ! vv[info.iter] )
{ /* Account for vector hole. Re-execute for next position. */
info.IterFinished();
REDO
}
internal-op Next-Vector-Iter
# v1 = iteration variable
class Vsb
op-types U I I
eval NextVectorIterCore($1, $2)
$$ = $1.iter;
$1.IterFinished();
internal-op Next-Vector-Blank-Iter
op1-internal
class sb
eval NextVectorIterCore($1, $2)
$1.IterFinished();
internal-op Next-Vector-Iter-Val-Var
# v1 = iteration variable
# v2 = value variable
op1-read-write
class VVsb
op-types U X I I
eval NextVectorIterCore($2, $3)
$$ = $2.iter;
if ( Z_IS_MANAGED )
$1 = BuildVal(vv[$2.iter]->ToVal(Z_TYPE), Z_TYPE);
else
$1 = *vv[$2.iter];
$2.IterFinished();
internal-op Next-Vector-Blank-Iter-Val-Var
# v1 = value variable
class Vsb
eval NextVectorIterCore($1, $2)
if ( Z_IS_MANAGED )
$$ = BuildVal(vv[$1.iter]->ToVal(Z_TYPE), Z_TYPE);
else
$$ = *vv[$1.iter];
$1.IterFinished();
internal-op Init-String-Loop
op1-read
classes Vs Cs
op-types S I
eval $2.InitLoop($1->AsString());
internal-op Next-String-Iter
# v1 = iteration variable
class Vsb
op-types S I I
eval if ( $1.IsDoneIterating() )
$2
auto bytes = (const char*) $1.s->Bytes() + $1.iter;
auto sv = new StringVal(1, bytes);
Unref($$);
$$ = sv;
$1.IterFinished();
internal-op Next-String-Blank-Iter
op1-internal
class sb
eval if ( $1.IsDoneIterating() )
$2
$1.IterFinished();
internal-op End-Table-Loop
op1-internal
class f
eval $1.Clear();

74
src/script_opt/ZAM/OPs/macros.op Normal file
View file

@ -0,0 +1,74 @@
# General-purpose macros. Those that are specific to a group of instructions
# are defined with those templates rather than appearing here.
# Macros for information associated with the current instruction.
# The Val frame used to pass in arguments.
macro Z_FRAME f
# The main type.
macro Z_TYPE z.GetType()
# Whether it's managed.
macro Z_IS_MANAGED *(z.is_managed)
# Secondary type.
macro Z_TYPE2 z.GetType2()
# Auxiliary information.
macro Z_AUX z.aux
macro Z_AUX_ID z.aux->id_val
macro Z_AUX_FUNC z.aux->func
macro Z_AUX_MAP z.aux->map
macro Z_AUX_ATTRS z.aux->attrs
macro Z_AUX_WHEN_INFO z.aux->wi
macro Z_AUX_EVENT_HANDLER z.aux->event_handler
macro Z_AUX_PRIMARY_FUNC z.aux->lambda->PrimaryFunc()
macro Z_AUX_LAMBDA_NAME z.aux->lambda->Name()
# Location in the original script.
macro Z_LOC z.loc
macro SET_RET_TYPE(type) ret_type = type;
macro INDEX_LIST zam_index_val_list
macro ERROR(msg) ZAM_run_time_error(Z_LOC, msg)
macro ERROR2(msg, obj) ZAM_run_time_error(Z_LOC, msg, obj)
macro WARN(msg) ZAM_run_time_warning(Z_LOC, msg)
# The following abstracts the process of creating a frame-assignable value.
macro BuildVal(v, t) ZVal(v, t)
# Returns a memory-managed-if-necessary copy of an existing value.
macro CopyVal(v) (Z_IS_MANAGED ? BuildVal((v).ToVal(Z_TYPE), Z_TYPE) : (v))
# Managed assignments to the given target.
macro AssignTarget(target, v) {
if ( Z_IS_MANAGED )
{
/* It's important to hold a reference to v here prior
to the deletion in case target points to v. */
auto v2 = v;
ZVal::DeleteManagedType(target);
target = v2;
}
else
target = v;
}
macro Branch(target) { DO_ZAM_PROFILE; pc = target; continue; }
macro REDO { --pc; /* so we then increment to here again */ break; }
macro GlobalID(g) globals[g].id
macro GlobalVal(g) frame[globals[g].slot]
macro StepIter(slot) step_iters[slot]
macro TableIter(slot) (*tiv_ptr)[slot]
macro DirectField(r, f) r->RawField(f)
macro DirectOptField(r, f) r->RawOptField(f)
macro LogEnum(v) v.ToVal(ZAM::log_ID_enum_type)

267
src/script_opt/ZAM/OPs/non-uniform.op Normal file
View file

@ -0,0 +1,267 @@
# Operations corresponding to non-uniform expressions.
assign-op Field
class R
field-op
assign-val v
eval auto r = $1.AsRecord();
auto& rv = DirectOptField(r, $2);
ZVal v;
if ( ! rv )
{
auto def = r->GetType<RecordType>()->FieldDefault($2);
if ( def )
v = ZVal(def, Z_TYPE);
else
ERROR(util::fmt("field value missing: $%s", r->GetType()->AsRecordType()->FieldName($2)));
}
else
v = *rv;
expr-op Has-Field
class VRi
includes-field-op
no-eval
internal-op Has-Field
class VRi
op-types I R I
eval $$ = $1->HasField($2);
internal-op Has-Field
class VRii
op-types R R I I
eval DirectOptField($$, $2) = ZVal(zeek_int_t($1->HasField($3)));
# The following generates an assignment version of Has-Field that we
# don't use (because we need the one above that uses "includes-field-op")
# but lets us compress the two conditionals.
predicate-op Has-Field
class Vi
op-types R I
eval $1->HasField($2)
predicate-op Table-Has-Elements
class V
op-types T
eval $1->Size() > 0
predicate-op Vector-Has-Elements
class V
op-types V
eval $1->Size() > 0
expr-op In
class VVV
custom-method return CompileInExpr(n1, n2, n3);
no-eval
expr-op In
class VCV
custom-method return CompileInExpr(n1, c, n2);
no-eval
expr-op In
class VVC
custom-method return CompileInExpr(n1, n2, c);
no-eval
internal-op P-In-S
classes VVV VCV VVC
op-types I P S
eval $$ = $1->MatchAnywhere($2->AsString()) != 0;
internal-op Str-In-Pat-Tbl
classes VVV VCV
op-types I S T
eval $$ = $2->MatchPattern({NewRef{}, $1});
internal-op S-In-S
classes VVV VCV VVC
op-types I S S
eval auto sc = reinterpret_cast<const unsigned char*>($1->CheckString());
auto cmp = util::strstr_n($2->Len(), $2->Bytes(), $1->Len(), sc);
$$ = cmp != -1;
internal-op A-In-S
classes VVV VCV VVC
op-types I A N
eval $$ = $2->Contains($1->AsAddr());
# Handled differently because of the unusual middle argument.
op L-In-T
class VLV
custom-method return CompileInExpr(n1, l, n2);
no-eval
op L-In-T
class VLC
custom-method return CompileInExpr(n, l, c);
no-eval
op L-In-Vec
class VLV
custom-method return CompileInExpr(n1, l, n2);
no-eval
op L-In-Vec
class VLC
custom-method return CompileInExpr(n, l, c);
no-eval
predicate-op Val-Is-In-Table
class VV
op-types X T
eval $2->Find($1.ToVal(Z_TYPE)) != nullptr
# Variants for indexing two values, one of which might be a constant.
# We set the instructions's *second* type to be that of the first variable
# index. We get the type of the second variable (if any) by digging it
# out of the table's type. For a constant in either position, we use
# the main instruction type, as always.
macro EvalVal2InTableCore(op1, op2)
INDEX_LIST->Clear();
INDEX_LIST->Append(op1);
INDEX_LIST->Append(op2);
macro EvalVal2InTableAssignCore(lhs, tbl)
lhs.AsIntRef() = tbl.AsTable()->Find(INDEX_LIST) != nullptr;
macro EvalVal2InTablePre(op1, op2, tbl)
auto& tt_ind = tbl.AsTable()->GetType()->AsTableType()->GetIndexTypes();
EvalVal2InTableCore(op1.ToVal(Z_TYPE2), op2.ToVal(tt_ind[1]))
internal-op Val2-Is-In-Table
class VVVV
eval EvalVal2InTablePre($1,$2,$3)
EvalVal2InTableAssignCore($$, $3)
internal-op Val2-Is-In-Table-Cond
op1-read
class VVVb
eval EvalVal2InTablePre($1,$2,$3)
EvalVal2InTableCond($3, INDEX_LIST, $4, !)
macro EvalVal2InTableCond(tbl, op, BRANCH, negate)
if ( negate tbl.AsTable()->Find(op) )
BRANCH
internal-op Val2-Is-Not-In-Table-Cond
op1-read
class VVVb
eval EvalVal2InTablePre($1,$2,$3)
EvalVal2InTableCond($3, INDEX_LIST, $4,)
internal-op Val2-Is-In-Table
class VVVC
eval EvalVal2InTableCore($1.ToVal(Z_TYPE2), $3.ToVal(Z_TYPE))
EvalVal2InTableAssignCore($$, $2)
internal-op Val2-Is-In-Table
class VVCV
eval EvalVal2InTableCore($2.ToVal(Z_TYPE), $1.ToVal(Z_TYPE2))
EvalVal2InTableAssignCore($$, $3)
internal-op Val2-Is-In-Table-Cond
op1-read
class VVbC
eval EvalVal2InTableCore($1.ToVal(Z_TYPE2), $4.ToVal(Z_TYPE))
EvalVal2InTableCond($2, INDEX_LIST, $3, !)
internal-op Val2-Is-In-Table-Cond
op1-read
class VVCb
eval EvalVal2InTableCore($3.ToVal(Z_TYPE), $1.ToVal(Z_TYPE2))
EvalVal2InTableCond($2, INDEX_LIST, $4, !)
internal-op Val2-Is-Not-In-Table-Cond
op1-read
class VVbC
eval EvalVal2InTableCore($1.ToVal(Z_TYPE2), $4.ToVal(Z_TYPE))
EvalVal2InTableCond($2, INDEX_LIST, $3, )
internal-op Val2-Is-Not-In-Table-Cond
op1-read
class VVCb
eval EvalVal2InTableCore($3.ToVal(Z_TYPE), $1.ToVal(Z_TYPE2))
EvalVal2InTableCond($2, INDEX_LIST, $4, )
predicate-op Const-Is-In-Table
class VC
op-types T X
eval $1->Find($2.ToVal(Z_TYPE)) != nullptr
internal-op List-Is-In-Table
classes VV VC
op-types I T
eval auto indices = Z_AUX->ToListVal(frame);
$$ = $1->Find(std::move(indices)) != nullptr;
internal-op Val-Is-In-Vector
class VVV
op-types I I V
eval auto vec = $2;
auto ind = $1;
$$ = vec->Has(ind);
internal-op Const-Is-In-Vector
class VCV
op-types I I V
eval auto vec = $2;
auto ind = $1;
$$ = vec->Has(ind);
expr-op Cond
class VVVV
op-types X I X X
set-type $2
eval AssignTarget($$, $1 ? CopyVal($2) : CopyVal($3))
expr-op Cond
class VVVC
op-types X I X X
set-type $2
eval AssignTarget($$, $1 ? CopyVal($2) : CopyVal($3))
expr-op Cond
class VVCV
op-types X I X X
set-type $2
eval AssignTarget($$, $1 ? CopyVal($2) : CopyVal($3))
op Bool-Vec-Cond
class VVVV
op-types V V V V
set-type $2
eval auto& vsel = $1->RawVec();
auto& v1 = $2->RawVec();
auto& v2 = $3->RawVec();
auto n = v1.size();
auto res = new vector<std::optional<ZVal>>(n);
for ( auto i = 0U; i < n; ++i )
if ( vsel[i] )
(*res)[i] = vsel[i]->AsInt() ? v1[i] : v2[i];
auto& full_res = $$;
Unref(full_res);
full_res = new VectorVal(cast_intrusive<VectorType>(Z_TYPE), res);
# Our instruction format doesn't accommodate two constants, so for
# the singular case of a V ? C1 : C2 conditional, we split it into
# two operations, V ? C1 and !V ? C2.
op CondC1
class VVC
op-types X I X
set-type $$
eval if ( $1 )
AssignTarget($$, CopyVal($2))
op CondC2
class VVC
op-types X I X
set-type $$
eval if ( ! $1 )
AssignTarget($$, CopyVal($2))

55
src/script_opt/ZAM/OPs/rel-exprs.op Normal file
View file

@ -0,0 +1,55 @@
# Operations corresponding to relational expressions.
rel-expr-op LT
op-type I U D S T A
vector
eval $1 < $2
eval-type S Bstr_cmp($1->AsString(), $2->AsString()) < 0
eval-type T $1->IsSubsetOf(*$2) && $1->Size() < $2->Size()
eval-type A $1->AsAddr() < $2->AsAddr()
rel-expr-op LE
op-type I U D S T A
vector
eval $1 <= $2
eval-type S Bstr_cmp($1->AsString(), $2->AsString()) <= 0
eval-type T $1->IsSubsetOf(*$2)
eval-type A $1->AsAddr() < $2->AsAddr() || $1->AsAddr() == $2->AsAddr()
rel-expr-op EQ
op-type I U D S T A N F
vector
eval $1 == $2
eval-type S Bstr_cmp($1->AsString(), $2->AsString()) == 0
eval-type T $1->EqualTo(*$2)
eval-type A $1->AsAddr() == $2->AsAddr()
eval-type N $1->AsSubNet() == $2->AsSubNet()
eval-type F util::streq($1->Name(), $2->Name())
eval-mixed P S $1->MatchExactly($2->AsString())
rel-expr-op NE
op-type I U D S T A N F
vector
eval $1 != $2
eval-type S Bstr_cmp($1->AsString(), $2->AsString()) != 0
eval-type T ! $1->EqualTo(*$2)
eval-type A $1->AsAddr() != $2->AsAddr()
eval-type N $1->AsSubNet() != $2->AsSubNet()
eval-type F ! util::streq($1->Name(), $2->Name())
eval-mixed P S ! $1->MatchExactly($2->AsString())
# Note, canonicalization means that GE and GT shouldn't occur
# for Sets (type T).
rel-expr-op GE
op-type I U D S A
vector
eval $1 >= $2
eval-type S Bstr_cmp($1->AsString(), $2->AsString()) >= 0
eval-type A ! ($1->AsAddr() < $2->AsAddr())
rel-expr-op GT
op-type I U D S A
vector
eval $1 > $2
eval-type S Bstr_cmp($1->AsString(), $2->AsString()) > 0
eval-type A ! ($1->AsAddr() < $2->AsAddr()) && $1->AsAddr() != $2->AsAddr()

57
src/script_opt/ZAM/OPs/script-idioms.op Normal file
View file

@ -0,0 +1,57 @@
# Operations corresponding to scripting idioms / known script functions.
internal-op MinU
classes VVV VVC
op-types U U U
eval $$ = std::min($1, $2);
internal-op MinI
classes VVV VVC
op-types I I I
eval $$ = std::min($1, $2);
internal-op MinD
classes VVV VVC
op-types D D D
eval $$ = std::min($1, $2);
internal-op MaxU
classes VVV VVC
op-types U U U
eval $$ = std::max($1, $2);
internal-op MaxI
classes VVV VVC
op-types I I I
eval $$ = std::max($1, $2);
internal-op MaxD
classes VVV VVC
op-types D D D
eval $$ = std::max($1, $2);
internal-op Func-Id-String
class VV
op-types S R
eval auto id_rec = $1;
auto orig_h = DirectField(id_rec, 0).AsAddr()->AsAddr().AsString();
auto resp_h = DirectField(id_rec, 2).AsAddr()->AsAddr().AsString();
auto orig_p = static_cast<uint32_t>(DirectField(id_rec, 1).AsCount()) & ~PORT_SPACE_MASK;
auto resp_p = static_cast<uint32_t>(DirectField(id_rec, 3).AsCount()) & ~PORT_SPACE_MASK;
/* Maximum address size is for IPv6 with no compression. Each
* 8 16-bit hex elements plus 7 colons between them plus the two []'s
* = 8*4 + 7 + 2 = 41 characters.
*
* Maximum port size is 5.
*
* Two of these = 2*41 + 2*5 = 92.
* Other delimiters: two ':', one ' < ' for 5 more.
*
* TOTAL: 97 characters.
*
* We use considerably more for safety.
*/
char buf[128];
snprintf(buf, sizeof buf, "%s:%u > %s:%u", orig_h.c_str(), orig_p, resp_h.c_str(), resp_p);
Unref($$);
$$ = new StringVal(buf);

339
src/script_opt/ZAM/OPs/stmts.op Normal file
View file

@ -0,0 +1,339 @@
# Operations corresponding to statements, other than iterations.
macro EvalScheduleArgs(time, is_delta, build_args)
if ( ! run_state::terminating )
{
double dt = time;
if ( is_delta )
dt += run_state::network_time;
auto handler = EventHandlerPtr(Z_AUX_EVENT_HANDLER);
ValVec args;
build_args
auto timer = new ScheduleTimer(handler, std::move(args), dt);
timer_mgr->Add(timer);
}
macro EvalSchedule(time, is_delta)
EvalScheduleArgs(time, is_delta, Z_AUX->FillValVec(args, frame);)
op Schedule
class ViHL
op-types D I X X
op1-read
custom-method return CompileSchedule(n, nullptr, i, h, l);
eval EvalSchedule($1, $2)
op Schedule
class CiHL
op-types D I X X
op1-read
custom-method return CompileSchedule(nullptr, c, i, h, l);
eval EvalSchedule($1, $2)
internal-op Schedule0
classes ViH CiH
op-types D I X
op1-read
eval EvalScheduleArgs($1, $2,)
macro QueueEvent(eh, args)
if ( *eh )
event_mgr.Enqueue(eh, std::move(args));
op Event
class HL
op1-read
custom-method return CompileEvent(h, l);
eval ValVec args;
Z_AUX->FillValVec(args, frame);
QueueEvent(Z_AUX_EVENT_HANDLER, args);
internal-op Event0
class X
eval ValVec args(0);
QueueEvent(Z_AUX_EVENT_HANDLER, args);
internal-op Event1
class V
op1-read
eval ValVec args(1);
args[0] = $1.ToVal(Z_TYPE);
QueueEvent(Z_AUX_EVENT_HANDLER, args);
internal-op Event2
class VV
op1-read
eval ValVec args(2);
args[0] = $1.ToVal(Z_TYPE);
args[1] = $2.ToVal(Z_TYPE2);
QueueEvent(Z_AUX_EVENT_HANDLER, args);
internal-op Event3
class VVV
op1-read
eval ValVec args(3);
auto& aux = Z_AUX;
args[0] = $1.ToVal(Z_TYPE);
args[1] = $2.ToVal(Z_TYPE2);
args[2] = $3.ToVal(aux->elems[2].GetType());
QueueEvent(Z_AUX_EVENT_HANDLER, args);
internal-op Event4
class VVVV
op1-read
eval ValVec args(4);
auto& aux = Z_AUX;
args[0] = $1.ToVal(Z_TYPE);
args[1] = $2.ToVal(Z_TYPE2);
args[2] = $3.ToVal(aux->elems[2].GetType());
args[3] = $4.ToVal(aux->elems[3].GetType());
QueueEvent(Z_AUX_EVENT_HANDLER, args);
op Return
class X
eval EvalReturn(nullptr,)
macro EvalReturn(val, type)
ret_u = val;
type
DO_ZAM_PROFILE
pc = end_pc;
continue;
op Return
op1-read
classes V C
set-type $$
eval EvalReturn(&$$, SET_RET_TYPE(Z_TYPE))
op When-Return
class X
eval static auto any_val = ZVal();
EvalReturn(&any_val,);
# Branch on the value of v1 using switch table v2, with default branch to v3
macro EvalSwitchBody(index, branch, cases, postscript)
{
auto t = cases[index];
if ( t.find(v) == t.end() )
pc = branch;
else
pc = t[v];
postscript
DO_ZAM_PROFILE
continue;
}
internal-op SwitchI
op1-read
class Vii
op-types I I I
eval auto v = $1;
EvalSwitchBody($2, $3, int_cases,)
internal-op SwitchU
op1-read
class Vii
op-types U I I
eval auto v = $1;
EvalSwitchBody($2, $3, uint_cases,)
internal-op SwitchD
op1-read
class Vii
op-types D I I
eval auto v = $1;
EvalSwitchBody($2, $3, double_cases,)
internal-op SwitchS
op1-read
class Vii
op-types S I I
eval auto vs = $1->AsString()->Render();
std::string v(vs);
EvalSwitchBody($2, $3, str_cases,delete[] vs;)
internal-op SwitchA
op1-read
class Vii
op-types A I I
eval auto v = $1->AsAddr().AsString();
EvalSwitchBody($2, $3, str_cases,)
internal-op SwitchN
op1-read
class Vii
op-types N I I
eval auto v = $1->AsSubNet().AsString();
EvalSwitchBody($2, $3, str_cases,)
internal-op Determine-Type-Match
class VV
op-types I a
eval auto& aux = Z_AUX;
int match = -1;
for ( int i = 0; i < aux->n; ++i )
{
auto& el = aux->elems[i];
auto& et = el.GetType();
if ( can_cast_value_to_type($1, et.get()) )
{
match = i;
if ( el.Slot() >= 0 )
{
auto& tv = frame[el.Slot()];
if ( el.IsManaged() )
Unref(tv.ManagedVal());
tv = ZVal(cast_value_to_type($1, et.get()), et);
}
break;
}
}
$$ = match;
op CheckAnyLen
op1-read
class Vi
op-types L U
eval auto v = $1;
if ( v->Vals().size() != $2 )
ERROR("mismatch in list lengths");
op Print
class O
eval do_print_stmt(Z_AUX->ToValVec(frame));
method-post z.aux = v->aux;
op Print1
op1-read
classes V C
set-type $$
eval std::vector<ValPtr> vals;
vals.push_back($$.ToVal(Z_TYPE));
do_print_stmt(vals);
internal-op If-Else
op1-read
class Vb
op-types I I
eval if ( ! $1 ) $2
internal-op If
op1-read
class Vb
op-types I I
eval if ( ! $1 ) $2
internal-op If-Not
op1-read
class Vb
op-types I I
eval if ( $1 ) $2
op AddStmt
op1-read
class VO
eval auto indices = Z_AUX->ToListVal(frame);
EvalAddStmt($1, indices)
method-post z.aux = v->aux;
macro EvalAddStmt(lhs, ind)
auto index = ind;
bool iterators_invalidated = false;
lhs.AsTable()->Assign(std::move(index), nullptr, true, &iterators_invalidated);
if ( iterators_invalidated )
WARN("possible loop/iterator invalidation");
op AddStmt1
op1-read
set-type $1
classes VV VC
eval EvalAddStmt($1, $2.ToVal(Z_TYPE))
op ClearTable
op1-read
class V
op-types T
eval $1->RemoveAll();
op ClearVector
op1-read
class V
op-types V
eval $1->Resize(0);
op DelTable
op1-read
class VO
op-types T X
eval auto indices = Z_AUX->ToListVal(frame);
bool iterators_invalidated = false;
$1->Remove(*indices, true, &iterators_invalidated);
if ( iterators_invalidated )
WARN("possible loop/iterator invalidation");
method-post z.aux = v->aux;
op DelField
op1-read
class Vi
op-types R I
eval $1->Remove($2);
internal-op Init-Record
class V
op-types R
eval auto r = new RecordVal(cast_intrusive<RecordType>(Z_TYPE));
Unref($$);
$$ = r;
internal-op Init-Vector
class V
op-types V
eval auto vt = cast_intrusive<VectorType>(Z_TYPE);
auto vec = new VectorVal(std::move(vt));
Unref($$);
$$ = vec;
internal-op Init-Table
class V
op-types T
eval auto tt = cast_intrusive<TableType>(Z_TYPE);
auto t = new TableVal(tt, Z_AUX_ATTRS);
Unref($$);
$$ = t;
op When
class V
op1-read
op-types F
eval BuildWhen($1, -1.0)
op When-Timeout
classes VV VC
op1-read
op-types F D
eval BuildWhen($1, $2)
macro BuildWhen(zf, timeout)
auto& aux = Z_AUX;
auto wi = Z_AUX_WHEN_INFO;
FuncPtr func{NewRef{}, zf};
auto lambda = make_intrusive<FuncVal>(func);
wi->Instantiate(std::move(lambda));
std::vector<ValPtr> local_aggrs;
for ( int i = 0; i < aux->n; ++i )
{
auto v = aux->ToVal(frame, i);
if ( v )
local_aggrs.push_back(v);
}
(void)make_intrusive<trigger::Trigger>(wi, wi->WhenExprGlobals(), local_aggrs, timeout, Z_FRAME, Z_LOC->Loc());

181
src/script_opt/ZAM/OPs/unary-exprs.op Normal file
View file

@ -0,0 +1,181 @@
# Operations corresponding to unary expressions.
# Direct assignment of an existing value.
assign-op Assign
class V
# The same, but where the assignment target (LHS) is a record field.
assign-op Field-LHS-Assign
op1-read
class F
unary-expr-op Clone
no-const
op-type X
set-type $$
set-type2 $1
eval auto v = $1.ToVal(Z_TYPE2)->Clone();
AssignTarget($$, BuildVal(v, Z_TYPE))
unary-expr-op Size
no-const
op-type I U D A N S T V *
explicit-result-type
set-type $$
set-type2 $1
eval-type I $$ = ZVal(zeek_int_t($1 < 0 ? -$1 : $1));
eval-type U $$ = ZVal($1);
eval-type D $$ = ZVal($1 < 0 ? -$1 : $1);
eval-type A $$ = ZVal(zeek_uint_t($1->AsAddr().GetFamily() == IPv4 ? 32 : 128));
eval-type N $$ = ZVal(pow(2.0, double(128 - $1->AsSubNet().LengthIPv6())));
eval-type S $$ = ZVal(zeek_uint_t($1->Len()));
eval-type T $$ = ZVal(zeek_uint_t($1->Size()));
eval-type V $$ = ZVal(zeek_uint_t($1->Size()));
eval auto v = $1.ToVal(Z_TYPE2)->SizeVal();
$$ = BuildVal(v, Z_TYPE);
unary-expr-op Not
op-type I
eval ! $1
unary-expr-op Complement
op-type U
eval ~ $1
unary-expr-op Positive
op-type I U D
vector
eval $1
unary-expr-op Negate
op-type I U D
vector
eval -$1
op IncrI
op1-read-write
class V
op-types I
eval ++$$;
op IncrU
op1-read-write
class V
op-types U
eval ++$$;
op DecrI
op1-read-write
class V
op-types I
eval --$$;
op DecrU
op1-read-write
class V
op-types U
eval auto& u = $$;
if ( u == 0 )
WARN("count underflow");
--u;
unary-op AppendTo
# Note, even though it feels like appending both reads and modifies
# its first operand, for our purposes it just reads it (to get the
# aggregate), and then modifies its *content* but not the operand's
# value itself.
op1-read
set-type $1
eval auto vv = $1.AsVector();
if ( vv->Size() == 0 )
/* Use the slightly more expensive Assign(), since it
* knows how to deal with empty vectors that do not yet
* have concrete types.
*/
vv->Assign(0, $2.ToVal(Z_TYPE));
else
{
vv->RawVec().push_back(CopyVal($2));
vv->Modified();
}
# For vectors-of-any, we always go through the Assign() interface because
# it's needed for tracking the potentially differing types.
unary-op AppendToAnyVec
op1-read
set-type $1
eval auto vv = $1.AsVector();
vv->Assign(vv->Size(), $2.ToVal(Z_TYPE));
internal-op AddPatternToField
classes VVi VCi
op1-read
op-types R P I
eval auto r = $$;
auto fpat = r->GetField($2)->AsPatternVal();
if ( fpat )
{
$1->AddTo(fpat, false);
r->Modified();
}
else
ERROR(util::fmt("field value missing: $%s", r->GetType()->AsRecordType()->FieldName($2)));
unary-op ExtendPattern
op1-read
eval $1.AsPattern()->AddTo($$.AsPattern(), false);
unary-op AddVecToVec
op1-read
eval if ( ! $1.AsVector()->AddTo($$.AsVector(), false) )
ERROR("incompatible vector element assignment");
unary-op AddTableToTable
op1-read
eval auto t = $$.AsTable();
auto v = $1.AsTable();
if ( v->Size() > 0 )
{
v->AddTo(t, false);
t->Modified();
}
unary-op RemoveTableFromTable
op1-read
eval auto t = $$.AsTable();
auto v = $1.AsTable();
if ( v->Size() > 0 )
{
v->RemoveFrom(t);
t->Modified();
}
unary-expr-op Cast
op-type X
set-type $$
set-type2 $1
eval EvalCast($$, $1.ToVal(Z_TYPE2))
macro EvalCast(lhs, rhs)
std::string error;
auto res = cast_value(rhs, Z_TYPE, error);
if ( res )
AssignTarget(lhs, BuildVal(res, Z_TYPE))
else
ERROR(error.c_str());
# Cast an "any" type to the given type. Only needed for type-based switch
# statements.
internal-op Cast-Any
class VV
op-types X a
eval ValPtr rhs = {NewRef{}, $1};
EvalCast($$, rhs)
direct-unary-op Is Is
internal-op Is
class VV
op-types I X
eval auto rhs = $1.ToVal(Z_TYPE2).get();
$$ = can_cast_value_to_type(rhs, Z_TYPE.get());

1
src/script_opt/ZAM/README.md
View file

@ -100,6 +100,7 @@ issues:
|`profile-ZAM` | Generate to "zprof.out" a ZAM execution profile. (Requires configuring with `--enable-ZAM-profiling` or `--enable-debug`.)|
|`report-recursive` | Report on recursive functions and exit.|
|`report-uncompilable` | Report on uncompilable functions and exit. For ZAM, all functions should be compilable.|
|`validate-ZAM` | Perform internal validation of ZAM instructions and exit.|
|`xform` | Transform scripts to "reduced" form.|
<a name="ZAM-profiling"></a>

378
src/script_opt/ZAM/Stmt.cc
View file

@ -132,7 +132,7 @@ const ZAMStmt ZAMCompiler::IfElse(const Expr* e, const Stmt* s1, const Stmt* s2)
if ( e->Tag() == EXPR_NAME ) {
auto n = e->AsNameExpr();
ZOp op = (s1 && s2) ? OP_IF_ELSE_VV : (s1 ? OP_IF_VV : OP_IF_NOT_VV);
ZOp op = (s1 && s2) ? OP_IF_ELSE_Vb : (s1 ? OP_IF_Vb : OP_IF_NOT_Vb);
ZInstI cond(op, FrameSlot(n), 0);
cond_stmt = AddInst(cond);
@ -141,13 +141,21 @@ const ZAMStmt ZAMCompiler::IfElse(const Expr* e, const Stmt* s1, const Stmt* s2)
else
cond_stmt = GenCond(e, branch_v);
AddCFT(insts1.back(), CFT_IF);
if ( s1 ) {
auto s1_end = CompileStmt(s1);
AddCFT(insts1.back(), CFT_BLOCK_END);
if ( s2 ) {
auto branch_after_s1 = GoToStub();
auto else_start = insts1.size();
auto s2_end = CompileStmt(s2);
SetV(cond_stmt, GoToTargetBeyond(branch_after_s1), branch_v);
SetGoTo(branch_after_s1, GoToTargetBeyond(s2_end));
AddCFT(insts1[else_start], CFT_ELSE);
AddCFT(insts1.back(), CFT_BLOCK_END);
return s2_end;
}
@ -160,66 +168,67 @@ const ZAMStmt ZAMCompiler::IfElse(const Expr* e, const Stmt* s1, const Stmt* s2)
// Only the else clause is non-empty.
auto s2_end = CompileStmt(s2);
AddCFT(insts1.back(), CFT_BLOCK_END);
// For complex conditionals, we need to invert their sense since
// we're switching to "if ( ! cond ) s2".
auto z = insts1[cond_stmt.stmt_num];
switch ( z->op ) {
case OP_IF_ELSE_VV:
case OP_IF_VV:
case OP_IF_NOT_VV:
case OP_IF_ELSE_Vb:
case OP_IF_Vb:
case OP_IF_NOT_Vb:
// These are generated correctly above, no need
// to fix up.
break;
case OP_HAS_FIELD_COND_VVV: z->op = OP_NOT_HAS_FIELD_COND_VVV; break;
case OP_NOT_HAS_FIELD_COND_VVV: z->op = OP_HAS_FIELD_COND_VVV; break;
case OP_HAS_FIELD_COND_Vib: z->op = OP_NOT_HAS_FIELD_COND_Vib; break;
case OP_NOT_HAS_FIELD_COND_Vib: z->op = OP_HAS_FIELD_COND_Vib; break;
case OP_CONN_EXISTS_COND_VV: z->op = OP_NOT_CONN_EXISTS_COND_VV; break;
case OP_NOT_CONN_EXISTS_COND_VV: z->op = OP_CONN_EXISTS_COND_VV; break;
case OP_CONN_EXISTS_COND_Vb: z->op = OP_NOT_CONN_EXISTS_COND_Vb; break;
case OP_NOT_CONN_EXISTS_COND_Vb: z->op = OP_CONN_EXISTS_COND_Vb; break;
case OP_IS_ICMP_PORT_COND_VV: z->op = OP_NOT_IS_ICMP_PORT_COND_VV; break;
case OP_NOT_IS_ICMP_PORT_COND_VV: z->op = OP_IS_ICMP_PORT_COND_VV; break;
case OP_IS_ICMP_PORT_COND_Vb: z->op = OP_NOT_IS_ICMP_PORT_COND_Vb; break;
case OP_NOT_IS_ICMP_PORT_COND_Vb: z->op = OP_IS_ICMP_PORT_COND_Vb; break;
case OP_IS_TCP_PORT_COND_VV: z->op = OP_NOT_IS_TCP_PORT_COND_VV; break;
case OP_NOT_IS_TCP_PORT_COND_VV: z->op = OP_IS_TCP_PORT_COND_VV; break;
case OP_IS_TCP_PORT_COND_Vb: z->op = OP_NOT_IS_TCP_PORT_COND_Vb; break;
case OP_NOT_IS_TCP_PORT_COND_Vb: z->op = OP_IS_TCP_PORT_COND_Vb; break;
case OP_IS_UDP_PORT_COND_VV: z->op = OP_NOT_IS_UDP_PORT_COND_VV; break;
case OP_NOT_IS_UDP_PORT_COND_VV: z->op = OP_IS_UDP_PORT_COND_VV; break;
case OP_IS_UDP_PORT_COND_Vb: z->op = OP_NOT_IS_UDP_PORT_COND_Vb; break;
case OP_NOT_IS_UDP_PORT_COND_Vb: z->op = OP_IS_UDP_PORT_COND_Vb; break;
case OP_IS_V4_ADDR_COND_VV: z->op = OP_NOT_IS_V4_ADDR_COND_VV; break;
case OP_NOT_IS_V4_ADDR_COND_VV: z->op = OP_IS_V4_ADDR_COND_VV; break;
case OP_IS_V4_ADDR_COND_Vb: z->op = OP_NOT_IS_V4_ADDR_COND_Vb; break;
case OP_NOT_IS_V4_ADDR_COND_Vb: z->op = OP_IS_V4_ADDR_COND_Vb; break;
case OP_IS_V6_ADDR_COND_VV: z->op = OP_NOT_IS_V6_ADDR_COND_VV; break;
case OP_NOT_IS_V6_ADDR_COND_VV: z->op = OP_IS_V6_ADDR_COND_VV; break;
case OP_IS_V6_ADDR_COND_Vb: z->op = OP_NOT_IS_V6_ADDR_COND_Vb; break;
case OP_NOT_IS_V6_ADDR_COND_Vb: z->op = OP_IS_V6_ADDR_COND_Vb; break;
case OP_READING_LIVE_TRAFFIC_COND_V: z->op = OP_NOT_READING_LIVE_TRAFFIC_COND_V; break;
case OP_NOT_READING_LIVE_TRAFFIC_COND_V: z->op = OP_READING_LIVE_TRAFFIC_COND_V; break;
case OP_READING_LIVE_TRAFFIC_COND_b: z->op = OP_NOT_READING_LIVE_TRAFFIC_COND_b; break;
case OP_NOT_READING_LIVE_TRAFFIC_COND_b: z->op = OP_READING_LIVE_TRAFFIC_COND_b; break;
case OP_READING_TRACES_COND_V: z->op = OP_NOT_READING_TRACES_COND_V; break;
case OP_NOT_READING_TRACES_COND_V: z->op = OP_READING_TRACES_COND_V; break;
case OP_READING_TRACES_COND_b: z->op = OP_NOT_READING_TRACES_COND_b; break;
case OP_NOT_READING_TRACES_COND_b: z->op = OP_READING_TRACES_COND_b; break;
case OP_TABLE_HAS_ELEMENTS_COND_VV: z->op = OP_NOT_TABLE_HAS_ELEMENTS_COND_VV; break;
case OP_NOT_TABLE_HAS_ELEMENTS_COND_VV: z->op = OP_TABLE_HAS_ELEMENTS_COND_VV; break;
case OP_TABLE_HAS_ELEMENTS_COND_Vb: z->op = OP_NOT_TABLE_HAS_ELEMENTS_COND_Vb; break;
case OP_NOT_TABLE_HAS_ELEMENTS_COND_Vb: z->op = OP_TABLE_HAS_ELEMENTS_COND_Vb; break;
case OP_VECTOR_HAS_ELEMENTS_COND_VV: z->op = OP_NOT_VECTOR_HAS_ELEMENTS_COND_VV; break;
case OP_NOT_VECTOR_HAS_ELEMENTS_COND_VV: z->op = OP_VECTOR_HAS_ELEMENTS_COND_VV; break;
case OP_VECTOR_HAS_ELEMENTS_COND_Vb: z->op = OP_NOT_VECTOR_HAS_ELEMENTS_COND_Vb; break;
case OP_NOT_VECTOR_HAS_ELEMENTS_COND_Vb: z->op = OP_VECTOR_HAS_ELEMENTS_COND_Vb; break;
case OP_VAL_IS_IN_TABLE_COND_VVV: z->op = OP_VAL_IS_NOT_IN_TABLE_COND_VVV; break;
case OP_VAL_IS_NOT_IN_TABLE_COND_VVV: z->op = OP_VAL_IS_IN_TABLE_COND_VVV; break;
case OP_VAL_IS_IN_TABLE_COND_VVb: z->op = OP_NOT_VAL_IS_IN_TABLE_COND_VVb; break;
case OP_NOT_VAL_IS_IN_TABLE_COND_VVb: z->op = OP_VAL_IS_IN_TABLE_COND_VVb; break;
case OP_CONST_IS_IN_TABLE_COND_VVC: z->op = OP_CONST_IS_NOT_IN_TABLE_COND_VVC; break;
case OP_CONST_IS_NOT_IN_TABLE_COND_VVC: z->op = OP_CONST_IS_IN_TABLE_COND_VVC; break;
case OP_CONST_IS_IN_TABLE_COND_VCb: z->op = OP_NOT_CONST_IS_IN_TABLE_COND_VCb; break;
case OP_NOT_CONST_IS_IN_TABLE_COND_VCb: z->op = OP_CONST_IS_IN_TABLE_COND_VCb; break;
case OP_VAL2_IS_IN_TABLE_COND_VVVV: z->op = OP_VAL2_IS_NOT_IN_TABLE_COND_VVVV; break;
case OP_VAL2_IS_NOT_IN_TABLE_COND_VVVV: z->op = OP_VAL2_IS_IN_TABLE_COND_VVVV; break;
case OP_VAL2_IS_IN_TABLE_COND_VVVb: z->op = OP_VAL2_IS_NOT_IN_TABLE_COND_VVVb; break;
case OP_VAL2_IS_NOT_IN_TABLE_COND_VVVb: z->op = OP_VAL2_IS_IN_TABLE_COND_VVVb; break;
case OP_VAL2_IS_IN_TABLE_COND_VVVC: z->op = OP_VAL2_IS_NOT_IN_TABLE_COND_VVVC; break;
case OP_VAL2_IS_NOT_IN_TABLE_COND_VVVC: z->op = OP_VAL2_IS_IN_TABLE_COND_VVVC; break;
case OP_VAL2_IS_IN_TABLE_COND_VVbC: z->op = OP_VAL2_IS_NOT_IN_TABLE_COND_VVbC; break;
case OP_VAL2_IS_NOT_IN_TABLE_COND_VVbC: z->op = OP_VAL2_IS_IN_TABLE_COND_VVbC; break;
case OP_VAL2_IS_IN_TABLE_COND_VVCV: z->op = OP_VAL2_IS_NOT_IN_TABLE_COND_VVCV; break;
case OP_VAL2_IS_NOT_IN_TABLE_COND_VVCV: z->op = OP_VAL2_IS_IN_TABLE_COND_VVCV; break;
case OP_VAL2_IS_IN_TABLE_COND_VVCb: z->op = OP_VAL2_IS_NOT_IN_TABLE_COND_VVCb; break;
case OP_VAL2_IS_NOT_IN_TABLE_COND_VVCb: z->op = OP_VAL2_IS_IN_TABLE_COND_VVCb; break;
default: reporter->InternalError("inconsistency in ZAMCompiler::IfElse");
}
@ -234,7 +243,7 @@ const ZAMStmt ZAMCompiler::GenCond(const Expr* e, int& branch_v) {
if ( e->Tag() == EXPR_HAS_FIELD ) {
auto hf = e->AsHasFieldExpr();
auto z = GenInst(OP_HAS_FIELD_COND_VVV, op1->AsNameExpr(), hf->Field());
auto z = GenInst(OP_HAS_FIELD_COND_Vib, op1->AsNameExpr(), hf->Field());
z.op_type = OP_VVV_I2_I3;
branch_v = 3;
return AddInst(z);
@ -251,15 +260,15 @@ const ZAMStmt ZAMCompiler::GenCond(const Expr* e, int& branch_v) {
}
if ( op1->Tag() == EXPR_NAME ) {
auto z = GenInst(OP_VAL_IS_IN_TABLE_COND_VVV, op1->AsNameExpr(), op2, 0);
z.t = op1->GetType();
auto z = GenInst(OP_VAL_IS_IN_TABLE_COND_VVb, op1->AsNameExpr(), op2, 0);
z.SetType(op1->GetType());
branch_v = 3;
return AddInst(z);
}
if ( op1->Tag() == EXPR_CONST ) {
auto z = GenInst(OP_CONST_IS_IN_TABLE_COND_VVC, op2, op1->AsConstExpr(), 0);
z.t = op1->GetType();
auto z = GenInst(OP_CONST_IS_IN_TABLE_COND_VCb, op2, op1->AsConstExpr(), 0);
z.SetType(op1->GetType());
branch_v = 2;
return AddInst(z);
}
@ -286,30 +295,30 @@ const ZAMStmt ZAMCompiler::GenCond(const Expr* e, int& branch_v) {
ZInstI z;
if ( name0 && name1 ) {
z = GenInst(OP_VAL2_IS_IN_TABLE_COND_VVVV, n0, n1, op2, 0);
z = GenInst(OP_VAL2_IS_IN_TABLE_COND_VVVb, n0, n1, op2, 0);
branch_v = 4;
z.t2 = n0->GetType();
z.SetType2(n0->GetType());
}
else if ( name0 ) {
z = GenInst(OP_VAL2_IS_IN_TABLE_COND_VVVC, n0, op2, c1, 0);
z = GenInst(OP_VAL2_IS_IN_TABLE_COND_VVbC, n0, op2, c1, 0);
branch_v = 3;
z.t2 = n0->GetType();
z.SetType2(n0->GetType());
}
else if ( name1 ) {
z = GenInst(OP_VAL2_IS_IN_TABLE_COND_VVCV, n1, op2, c0, 0);
z = GenInst(OP_VAL2_IS_IN_TABLE_COND_VVCb, n1, op2, c0, 0);
branch_v = 3;
z.t2 = n1->GetType();
z.SetType2(n1->GetType());
}
else { // Both are constants, assign first to temporary.
auto slot = TempForConst(c0);
z = ZInstI(OP_VAL2_IS_IN_TABLE_COND_VVVC, slot, FrameSlot(op2), 0, c1);
z = ZInstI(OP_VAL2_IS_IN_TABLE_COND_VVbC, slot, FrameSlot(op2), 0, c1);
z.op_type = OP_VVVC_I3;
branch_v = 3;
z.t2 = c0->GetType();
z.SetType2(c0->GetType());
}
return AddInst(z);
@ -328,9 +337,9 @@ const ZAMStmt ZAMCompiler::GenCond(const Expr* e, int& branch_v) {
ZOp op;
if ( aggr->GetType()->Tag() == TYPE_TABLE )
op = OP_TABLE_HAS_ELEMENTS_COND_VV;
op = OP_TABLE_HAS_ELEMENTS_COND_Vb;
else
op = OP_VECTOR_HAS_ELEMENTS_COND_VV;
op = OP_VECTOR_HAS_ELEMENTS_COND_Vb;
branch_v = 2;
return AddInst(GenInst(op, aggr, +0));
@ -409,45 +418,32 @@ const ZAMStmt ZAMCompiler::ValueSwitch(const SwitchStmt* sw, const NameExpr* v,
// Figure out which jump table we're using.
auto t = v ? v->GetType() : c->GetType();
int tbl = 0;
return GenSwitch(sw, slot, t->InternalType());
}
const ZAMStmt ZAMCompiler::GenSwitch(const SwitchStmt* sw, int slot, InternalTypeTag it) {
ZOp op;
switch ( t->InternalType() ) {
case TYPE_INTERNAL_INT:
op = OP_SWITCHI_VVV;
tbl = int_casesI.size();
break;
switch ( it ) {
case TYPE_INTERNAL_INT: op = OP_SWITCHI_Vii; break;
case TYPE_INTERNAL_UNSIGNED:
op = OP_SWITCHU_VVV;
tbl = uint_casesI.size();
break;
case TYPE_INTERNAL_UNSIGNED: op = OP_SWITCHU_Vii; break;
case TYPE_INTERNAL_DOUBLE:
op = OP_SWITCHD_VVV;
tbl = double_casesI.size();
break;
case TYPE_INTERNAL_DOUBLE: op = OP_SWITCHD_Vii; break;
case TYPE_INTERNAL_STRING:
op = OP_SWITCHS_VVV;
tbl = str_casesI.size();
break;
case TYPE_INTERNAL_STRING: op = OP_SWITCHS_Vii; break;
case TYPE_INTERNAL_ADDR:
op = OP_SWITCHA_VVV;
tbl = str_casesI.size();
break;
case TYPE_INTERNAL_ADDR: op = OP_SWITCHA_Vii; break;
case TYPE_INTERNAL_SUBNET:
op = OP_SWITCHN_VVV;
tbl = str_casesI.size();
break;
case TYPE_INTERNAL_SUBNET: op = OP_SWITCHN_Vii; break;
default: reporter->InternalError("bad switch type");
}
// Add the "head", i.e., the execution of the jump table.
auto sw_head_op = ZInstI(op, slot, tbl, 0);
// Add the "head", i.e., the execution of the jump table. At this point,
// we leave the table (v2) and default (v3) TBD.
auto sw_head_op = ZInstI(op, slot, 0, 0);
sw_head_op.op_type = OP_VVV_I2_I3;
auto sw_head = AddInst(sw_head_op);
@ -456,6 +452,7 @@ const ZAMStmt ZAMCompiler::ValueSwitch(const SwitchStmt* sw, const NameExpr* v,
// Generate each of the cases.
auto cases = sw->Cases();
std::vector<InstLabel> case_start;
int case_index = 0;
PushFallThroughs();
for ( auto sw_case : *cases ) {
@ -471,8 +468,11 @@ const ZAMStmt ZAMCompiler::ValueSwitch(const SwitchStmt* sw, const NameExpr* v,
ResolveBreaks(sw_end);
int def_ind = sw->DefaultCaseIndex();
if ( def_ind >= 0 )
SetV3(sw_head, case_start[def_ind]);
if ( def_ind >= 0 ) {
auto def = case_start[def_ind];
SetV3(sw_head, def);
AddCFT(def, CFT_DEFAULT);
}
else
SetV3(sw_head, sw_end);
@ -520,103 +520,93 @@ const ZAMStmt ZAMCompiler::ValueSwitch(const SwitchStmt* sw, const NameExpr* v,
}
}
// For type switches, we map them to consecutive numbers, and then use
// a integer-valued switch on those.
int tm_ctr = 0;
for ( auto [_, index] : *sw->TypeMap() ) {
auto case_body_start = case_start[index];
new_int_cases[tm_ctr++] = case_body_start;
}
// Now add the jump table to the set we're keeping for the
// corresponding type.
switch ( t->InternalType() ) {
case TYPE_INTERNAL_INT: int_casesI.push_back(new_int_cases); break;
size_t tbl;
case TYPE_INTERNAL_UNSIGNED: uint_casesI.push_back(new_uint_cases); break;
switch ( it ) {
case TYPE_INTERNAL_INT:
tbl = int_casesI.size();
int_casesI.push_back(new_int_cases);
break;
case TYPE_INTERNAL_DOUBLE: double_casesI.push_back(new_double_cases); break;
case TYPE_INTERNAL_UNSIGNED:
tbl = uint_casesI.size();
uint_casesI.push_back(new_uint_cases);
break;
case TYPE_INTERNAL_DOUBLE:
tbl = double_casesI.size();
double_casesI.push_back(new_double_cases);
break;
case TYPE_INTERNAL_STRING:
case TYPE_INTERNAL_ADDR:
case TYPE_INTERNAL_SUBNET: str_casesI.push_back(new_str_cases); break;
case TYPE_INTERNAL_SUBNET:
tbl = str_casesI.size();
str_casesI.push_back(new_str_cases);
break;
default: reporter->InternalError("bad switch type");
}
insts1[sw_head.stmt_num]->v2 = int(tbl);
AddCFT(insts1[body_end.stmt_num], CFT_BLOCK_END);
return body_end;
}
const ZAMStmt ZAMCompiler::TypeSwitch(const SwitchStmt* sw, const NameExpr* v, const ConstExpr* c) {
auto cases = sw->Cases();
auto type_map = sw->TypeMap();
auto body_end = EmptyStmt();
auto tmp = NewSlot(true); // true since we know "any" is managed
int slot = v ? FrameSlot(v) : 0;
if ( v && v->GetType()->Tag() != TYPE_ANY ) {
if ( v ) {
if ( v->GetType()->Tag() != TYPE_ANY ) {
auto z = ZInstI(OP_ASSIGN_ANY_VV, tmp, slot);
body_end = AddInst(z);
AddInst(z);
slot = tmp;
}
}
if ( c ) {
else {
ASSERT(c);
auto z = ZInstI(OP_ASSIGN_ANY_VC, tmp, c);
body_end = AddInst(z);
AddInst(z);
slot = tmp;
}
int def_ind = sw->DefaultCaseIndex();
ZAMStmt def_succ(0); // successor to default, if any
bool saw_def_succ = false; // whether def_succ is meaningful
int ntypes = type_map->size();
auto aux = new ZInstAux(ntypes);
PushFallThroughs();
for ( auto& i : *type_map ) {
auto id = i.first;
auto type = id->GetType();
for ( size_t i = 0; i < type_map->size(); ++i ) {
auto& tm = (*type_map)[i];
auto id_i = tm.first;
auto id_case = tm.second;
ZInstI z;
z = ZInstI(OP_BRANCH_IF_NOT_TYPE_VV, slot, 0);
z.SetType(type);
auto case_test = AddInst(z);
// Type cases that don't use "as" create a placeholder
// ID with a null name.
if ( id->Name() ) {
int id_slot = Frame1Slot(id, OP_CAST_ANY_VV);
z = ZInstI(OP_CAST_ANY_VV, id_slot, slot);
z.SetType(type);
body_end = AddInst(z);
}
else
body_end = case_test;
ResolveFallThroughs(GoToTargetBeyond(body_end));
body_end = CompileStmt((*cases)[i.second]->Body());
SetV2(case_test, GoToTargetBeyond(body_end));
if ( def_ind >= 0 && i.second == def_ind + 1 ) {
def_succ = case_test;
saw_def_succ = true;
auto slot = id_i->Name() ? FrameSlot(id_i) : -1;
aux->Add(i, slot, id_i->GetType());
}
PushFallThroughs();
}
auto match_tmp = NewSlot(false);
auto z = ZInstI(OP_DETERMINE_TYPE_MATCH_VV, match_tmp, slot);
z.op_type = OP_VV;
z.aux = aux;
AddInst(z);
ResolveFallThroughs(GoToTargetBeyond(body_end));
if ( def_ind >= 0 ) {
PushFallThroughs();
body_end = CompileStmt((*sw->Cases())[def_ind]->Body());
// Now resolve any fallthrough's in the default.
if ( saw_def_succ )
ResolveFallThroughs(GoToTargetBeyond(def_succ));
else
ResolveFallThroughs(GoToTargetBeyond(body_end));
}
ResolveBreaks(GoToTargetBeyond(body_end));
return body_end;
return GenSwitch(sw, match_tmp, TYPE_INTERNAL_INT);
}
const ZAMStmt ZAMCompiler::CompileWhile(const WhileStmt* ws) {
@ -645,18 +635,23 @@ const ZAMStmt ZAMCompiler::While(const Stmt* cond_stmt, const Expr* cond, const
if ( cond->Tag() == EXPR_NAME ) {
auto n = cond->AsNameExpr();
cond_IF = AddInst(ZInstI(OP_IF_VV, FrameSlot(n), 0));
cond_IF = AddInst(ZInstI(OP_IF_Vb, FrameSlot(n), 0));
branch_v = 2;
}
else
cond_IF = GenCond(cond, branch_v);
AddCFT(insts1[head.stmt_num], CFT_LOOP);
AddCFT(insts1[cond_IF.stmt_num], CFT_LOOP_COND);
PushNexts();
PushBreaks();
if ( body && body->Tag() != STMT_NULL )
(void)CompileStmt(body);
AddCFT(insts1.back(), CFT_BLOCK_END);
auto tail = GoTo(GoToTarget(head));
auto beyond_tail = GoToTargetBeyond(tail);
@ -676,17 +671,21 @@ const ZAMStmt ZAMCompiler::CompileFor(const ForStmt* f) {
PushNexts();
PushBreaks();
ZAMStmt z;
if ( et == TYPE_TABLE )
return LoopOverTable(f, val);
z = LoopOverTable(f, val);
else if ( et == TYPE_VECTOR )
return LoopOverVector(f, val);
z = LoopOverVector(f, val);
else if ( et == TYPE_STRING )
return LoopOverString(f, e);
z = LoopOverString(f, e);
else
reporter->InternalError("bad \"for\" loop-over value when compiling");
return z;
}
const ZAMStmt ZAMCompiler::LoopOverTable(const ForStmt* f, const NameExpr* val) {
@ -723,29 +722,39 @@ const ZAMStmt ZAMCompiler::LoopOverTable(const ForStmt* f, const NameExpr* val)
auto iter_slot = table_iters.size();
table_iters.emplace_back();
auto z = ZInstI(OP_INIT_TABLE_LOOP_VV, FrameSlot(val), iter_slot);
z.op_type = OP_VV_I2;
z.SetType(value_var ? value_var->GetType() : nullptr);
z.aux = aux;
auto zi = ZInstI(OP_INIT_TABLE_LOOP_Vf, FrameSlot(val), iter_slot);
zi.op_type = OP_VV_I2;
if ( value_var )
zi.SetType(value_var->GetType());
zi.aux = aux;
auto init_end = AddInst(z);
(void)AddInst(zi);
ZInstI zn;
auto iter_head = StartingBlock();
if ( value_var ) {
ZOp op = no_loop_vars ? OP_NEXT_TABLE_ITER_VAL_VAR_NO_VARS_VVV : OP_NEXT_TABLE_ITER_VAL_VAR_VVV;
z = ZInstI(op, FrameSlot(value_var), iter_slot, 0);
z.CheckIfManaged(value_var->GetType());
z.op_type = OP_VVV_I2_I3;
ZOp op = no_loop_vars ? OP_NEXT_TABLE_ITER_VAL_VAR_NO_VARS_Vfb : OP_NEXT_TABLE_ITER_VAL_VAR_Vfb;
zn = ZInstI(op, FrameSlot(value_var), iter_slot, 0);
zn.CheckIfManaged(value_var->GetType());
zn.op_type = OP_VVV_I2_I3;
}
else {
ZOp op = no_loop_vars ? OP_NEXT_TABLE_ITER_NO_VARS_VV : OP_NEXT_TABLE_ITER_VV;
z = ZInstI(op, iter_slot, 0);
z.op_type = OP_VV_I1_I2;
ZOp op = no_loop_vars ? OP_NEXT_TABLE_ITER_NO_VARS_fb : OP_NEXT_TABLE_ITER_fb;
zn = ZInstI(op, iter_slot, 0);
zn.op_type = OP_VV_I1_I2;
}
z.aux = aux; // so ZOpt.cc can get to it
// Need a separate instance of aux so the CFT info doesn't get shared with
// the loop init. We populate it with the loop_vars (only) because the
// optimizer needs access to those for (1) tracking their lifetime, and
// (2) remapping them (not strictly needed, see the comment in ReMapFrame()).
zn.aux = new ZInstAux(0);
zn.aux->loop_vars = aux->loop_vars;
AddCFT(&zn, CFT_LOOP);
AddCFT(&zn, CFT_LOOP_COND);
return FinishLoop(iter_head, z, body, iter_slot, true);
return FinishLoop(iter_head, zn, body, iter_slot, true);
}
const ZAMStmt ZAMCompiler::LoopOverVector(const ForStmt* f, const NameExpr* val) {
@ -755,7 +764,7 @@ const ZAMStmt ZAMCompiler::LoopOverVector(const ForStmt* f, const NameExpr* val)
int iter_slot = num_step_iters++;
auto z = ZInstI(OP_INIT_VECTOR_LOOP_VV, FrameSlot(val), iter_slot);
auto z = ZInstI(OP_INIT_VECTOR_LOOP_Vs, FrameSlot(val), iter_slot);
z.op_type = OP_VV_I2;
auto init_end = AddInst(z);
@ -765,29 +774,31 @@ const ZAMStmt ZAMCompiler::LoopOverVector(const ForStmt* f, const NameExpr* val)
if ( value_var ) {
if ( slot >= 0 ) {
z = ZInstI(OP_NEXT_VECTOR_ITER_VAL_VAR_VVVV, slot, FrameSlot(value_var), iter_slot, 0);
z = ZInstI(OP_NEXT_VECTOR_ITER_VAL_VAR_VVsb, slot, FrameSlot(value_var), iter_slot, 0);
z.op_type = OP_VVVV_I3_I4;
}
else {
z = ZInstI(OP_NEXT_VECTOR_BLANK_ITER_VAL_VAR_VVV, FrameSlot(value_var), iter_slot, 0);
z = ZInstI(OP_NEXT_VECTOR_BLANK_ITER_VAL_VAR_Vsb, FrameSlot(value_var), iter_slot, 0);
z.op_type = OP_VVV_I2_I3;
}
z.t = value_var->GetType();
z.is_managed = ZVal::IsManagedType(z.t);
z.SetType(value_var->GetType());
}
else {
if ( slot >= 0 ) {
z = ZInstI(OP_NEXT_VECTOR_ITER_VVV, slot, iter_slot, 0);
z = ZInstI(OP_NEXT_VECTOR_ITER_Vsb, slot, iter_slot, 0);
z.op_type = OP_VVV_I2_I3;
}
else {
z = ZInstI(OP_NEXT_VECTOR_BLANK_ITER_VV, iter_slot, 0);
z = ZInstI(OP_NEXT_VECTOR_BLANK_ITER_sb, iter_slot, 0);
z.op_type = OP_VV_I1_I2;
}
}
AddCFT(&z, CFT_LOOP);
AddCFT(&z, CFT_LOOP_COND);
return FinishLoop(iter_head, z, f->LoopBody(), iter_slot, false);
}
@ -802,12 +813,12 @@ const ZAMStmt ZAMCompiler::LoopOverString(const ForStmt* f, const Expr* e) {
ZInstI z;
if ( n ) {
z = ZInstI(OP_INIT_STRING_LOOP_VV, FrameSlot(n), iter_slot);
z = ZInstI(OP_INIT_STRING_LOOP_Vs, FrameSlot(n), iter_slot);
z.op_type = OP_VV_I2;
}
else {
ASSERT(c);
z = ZInstI(OP_INIT_STRING_LOOP_VC, iter_slot, c);
z = ZInstI(OP_INIT_STRING_LOOP_Cs, iter_slot, c);
z.op_type = OP_VC_I1;
}
@ -815,15 +826,18 @@ const ZAMStmt ZAMCompiler::LoopOverString(const ForStmt* f, const Expr* e) {
auto iter_head = StartingBlock();
if ( loop_var->IsBlank() ) {
z = ZInstI(OP_NEXT_STRING_BLANK_ITER_VV, iter_slot, 0);
z = ZInstI(OP_NEXT_STRING_BLANK_ITER_sb, iter_slot, 0);
z.op_type = OP_VV_I1_I2;
}
else {
z = ZInstI(OP_NEXT_STRING_ITER_VVV, FrameSlot(loop_var), iter_slot, 0);
z = ZInstI(OP_NEXT_STRING_ITER_Vsb, FrameSlot(loop_var), iter_slot, 0);
z.op_type = OP_VVV_I2_I3;
z.is_managed = true;
}
AddCFT(&z, CFT_LOOP);
AddCFT(&z, CFT_LOOP_COND);
return FinishLoop(iter_head, z, f->LoopBody(), iter_slot, false);
}
@ -832,7 +846,11 @@ const ZAMStmt ZAMCompiler::Loop(const Stmt* body) {
PushBreaks();
auto head = StartingBlock();
(void)CompileStmt(body);
auto b = CompileStmt(body);
AddCFT(insts1[head.stmt_num], CFT_LOOP);
AddCFT(insts1[b.stmt_num], CFT_BLOCK_END);
auto tail = GoTo(GoToTarget(head));
ResolveNexts(GoToTarget(head));
@ -845,11 +863,12 @@ const ZAMStmt ZAMCompiler::FinishLoop(const ZAMStmt iter_head, ZInstI& iter_stmt
bool is_table) {
auto loop_iter = AddInst(iter_stmt);
auto body_end = CompileStmt(body);
AddCFT(insts1[body_end.stmt_num], CFT_BLOCK_END);
// We only need cleanup for looping over tables, but for now we
// need some sort of placeholder instruction (until the optimizer
// can elide it) to resolve loop exits.
ZOp op = is_table ? OP_END_TABLE_LOOP_V : OP_NOP;
ZOp op = is_table ? OP_END_TABLE_LOOP_f : OP_NOP;
auto loop_end = GoTo(GoToTarget(iter_head));
auto z = ZInstI(op, iter_slot);
@ -875,6 +894,12 @@ const ZAMStmt ZAMCompiler::CompileReturn(const ReturnStmt* r) {
if ( retvars.empty() ) { // a "true" return
if ( e ) {
if ( pf->ProfiledFunc()->Flavor() == FUNC_FLAVOR_HOOK ) {
ASSERT(e->GetType()->Tag() == TYPE_BOOL);
auto true_c = make_intrusive<ConstExpr>(val_mgr->True());
return ReturnC(true_c.get());
}
if ( e->Tag() == EXPR_NAME )
return ReturnV(e->AsNameExpr());
else
@ -970,7 +995,7 @@ const ZAMStmt ZAMCompiler::CompileWhen(const WhenStmt* ws) {
auto timeout = wi->TimeoutExpr();
auto lambda = NewSlot(true);
(void)BuildLambda(lambda, wi->Lambda().get());
(void)BuildLambda(lambda, wi->Lambda());
std::vector<IDPtr> local_aggr_slots;
for ( auto& l : wi->WhenExprLocals() )
@ -1006,8 +1031,7 @@ const ZAMStmt ZAMCompiler::CompileWhen(const WhenStmt* ws) {
if ( ws->IsReturn() ) {
(void)AddInst(z);
z = ZInstI(OP_RETURN_C);
z.c = ZVal();
z = ZInstI(OP_WHEN_RETURN_X);
}
return AddInst(z);

34
src/script_opt/ZAM/Support.cc
View file

@ -16,7 +16,7 @@ namespace ZAM {
std::string curr_func;
std::shared_ptr<ZAMLocInfo> curr_loc;
TypePtr log_ID_enum_type;
TypePtr any_base_type;
TypePtr any_base_type = base_type(TYPE_ANY);
} // namespace ZAM
bool ZAM_error = false;
@ -35,6 +35,32 @@ bool is_ZAM_compilable(const ProfileFunc* pf, const char** reason) {
bool IsAny(const Type* t) { return t->Tag() == TYPE_ANY; }
bool CheckAnyType(const TypePtr& any_type, const TypePtr& expected_type, const std::shared_ptr<ZAMLocInfo>& loc) {
if ( IsAny(expected_type) )
return true;
if ( ! same_type(any_type, expected_type, false, false) ) {
auto at = any_type->Tag();
auto et = expected_type->Tag();
if ( at == TYPE_RECORD && et == TYPE_RECORD ) {
auto at_r = any_type->AsRecordType();
auto et_r = expected_type->AsRecordType();
if ( record_promotion_compatible(et_r, at_r) )
return true;
}
char buf[8192];
snprintf(buf, sizeof buf, "run-time type clash (%s/%s)", type_name(at), type_name(et));
reporter->RuntimeError(loc->Loc(), "%s", buf);
return false;
}
return true;
}
StringVal* ZAM_to_lower(const StringVal* sv) {
auto bs = sv->AsString();
const u_char* s = bs->Bytes();
@ -82,7 +108,10 @@ void ZAM_run_time_error(const char* msg) {
}
void ZAM_run_time_error(std::shared_ptr<ZAMLocInfo> loc, const char* msg) {
if ( loc )
reporter->RuntimeError(loc->Loc(), "%s", msg);
else
fprintf(stderr, "<no location in optimized code>: %s\n", msg);
ZAM_error = true;
}
@ -92,7 +121,10 @@ void ZAM_run_time_error(const char* msg, const Obj* o) {
}
void ZAM_run_time_error(std::shared_ptr<ZAMLocInfo> loc, const char* msg, const Obj* o) {
if ( loc )
reporter->RuntimeError(loc->Loc(), "%s (%s)", msg, obj_desc(o).c_str());
else
ZAM_run_time_error(msg, o);
ZAM_error = true;
}

4
src/script_opt/ZAM/Support.h
View file

@ -42,6 +42,10 @@ extern bool IsAny(const Type* t);
inline bool IsAny(const TypePtr& t) { return IsAny(t.get()); }
inline bool IsAny(const Expr* e) { return IsAny(e->GetType()); }
// Run-time checking for "any" type being consistent with
// expected typed. Returns true if the type match is okay.
extern bool CheckAnyType(const TypePtr& any_type, const TypePtr& expected_type, const std::shared_ptr<ZAMLocInfo>& loc);
extern void ZAM_run_time_error(const char* msg);
extern void ZAM_run_time_error(std::shared_ptr<ZAMLocInfo> loc, const char* msg);
extern void ZAM_run_time_error(std::shared_ptr<ZAMLocInfo> loc, const char* msg, const Obj* o);

110
src/script_opt/ZAM/Validate.cc Normal file
View file

@ -0,0 +1,110 @@
// See the file "COPYING" in the main distribution directory for copyright.
#include <string>
#include "zeek/script_opt/ZAM/ZBody.h"
#include "zeek/script_opt/ZAM/ZOp.h"
using std::string;
namespace zeek::detail {
std::unordered_map<ZOp, ZAMInstDesc> zam_inst_desc = {
#include "ZAM-Desc.h"
};
// While the following has commonalities that could be factored out,
// for now we keep this form because it provides flexibility for
// accommodating other forms of accessors.
static std::map<char, string> type_pats = {
{'A', "Addr"}, {'a', "Any"}, {'D', "Double"}, {'F', "Func"}, {'I', "Int"}, {'L', "List"}, {'N', "SubNet"},
{'P', "Pattern"}, {'R', "Record"}, {'S', "String"}, {'T', "Table"}, {'t', "Type"}, {'U', "Count"}, {'V', "Vector"},
};
static int num_valid = 0;
static int num_tested = 0;
static int num_skipped = 0;
void analyze_ZAM_inst(const char* op_name, const ZAMInstDesc& zid) {
auto& oc = zid.op_class;
auto& ot = zid.op_types;
auto& eval = zid.op_eval;
bool have_ot = ! ot.empty();
if ( have_ot && oc.size() != ot.size() )
reporter->InternalError("%s: instruction class/types mismatch (%s/%s)", op_name, oc.c_str(), ot.c_str());
int nslot = 0;
for ( size_t i = 0; i < oc.size(); ++i ) {
auto oc_i = oc[i];
string op;
switch ( oc_i ) {
case 'V':
case 'R': op = "frame\\[z\\.v" + std::to_string(++nslot) + "\\]"; break;
case 'b':
case 'f':
case 'g':
case 's':
case 'i': op = "z\\.v" + std::to_string(++nslot); break;
case 'C': op = "z\\.c"; break;
default:
if ( have_ot && ot[i] != 'X' )
reporter->InternalError("instruction types mismatch: %s (%c)", op_name, oc_i);
}
auto match_pat = op;
if ( have_ot ) {
auto ot_i = ot[i];
bool bare_int = std::string("bfgis").find(oc_i) != std::string::npos;
if ( ot_i == 'X' || bare_int ) {
if ( ot_i == 'X' && bare_int )
reporter->InternalError("empty instruction type for '%c' class element: %s", oc_i, op_name);
if ( ! std::regex_search(eval, std::regex(op)) )
reporter->InternalError("%s: operand %s not found", op_name, op.c_str());
++num_skipped;
continue;
}
auto tp = type_pats.find(ot_i);
if ( tp == type_pats.end() )
reporter->InternalError("%s: instruction type %c not found", op_name, ot_i);
match_pat += ".As" + tp->second + "(Ref)?\\(\\)";
++num_tested;
}
if ( ! std::regex_search(eval, std::regex(match_pat)) )
reporter->InternalError("%s: did not find /%s/ in %s", op_name, match_pat.c_str(), eval.c_str());
}
++num_valid;
}
void validate_ZAM_insts() {
// The following primes a data structure we access.
(void)AssignmentFlavor(OP_NOP, TYPE_VOID, false);
for ( int i = 0; i < int(OP_NOP); ++i ) {
auto zop = ZOp(i);
if ( zam_inst_desc.find(zop) == zam_inst_desc.end() && assignment_flavor.find(zop) == assignment_flavor.end() )
reporter->InternalError("op %s missing from description", ZOP_name(zop));
}
for ( auto& zid : zam_inst_desc )
analyze_ZAM_inst(ZOP_name(zid.first), zid.second);
printf("%d valid, %d tested, %d skipped\n", num_valid, num_tested, num_skipped);
}
} // namespace zeek::detail

11
src/script_opt/ZAM/Vars.cc
View file

@ -40,7 +40,7 @@ void ZAMCompiler::LoadParam(const ID* id) {
ZOp op;
op = AssignmentFlavor(OP_LOAD_VAL_VV, id->GetType()->Tag());
op = AssignmentFlavor(OP_LOAD_VAL_Vi, id->GetType()->Tag());
int slot = AddToFrame(id);
@ -57,9 +57,9 @@ const ZAMStmt ZAMCompiler::LoadGlobal(const ID* id) {
if ( id->IsType() )
// Need a special load for these, as they don't fit
// with the usual template.
op = OP_LOAD_GLOBAL_TYPE_VV;
op = OP_LOAD_GLOBAL_TYPE_Vg;
else
op = AssignmentFlavor(OP_LOAD_GLOBAL_VV, id->GetType()->Tag());
op = AssignmentFlavor(OP_LOAD_GLOBAL_Vg, id->GetType()->Tag());
auto slot = RawSlot(id);
@ -78,13 +78,14 @@ const ZAMStmt ZAMCompiler::LoadCapture(const ID* id) {
ZOp op;
if ( ZVal::IsManagedType(id->GetType()) )
op = OP_LOAD_MANAGED_CAPTURE_VV;
op = OP_LOAD_MANAGED_CAPTURE_Vi;
else
op = OP_LOAD_CAPTURE_VV;
op = OP_LOAD_CAPTURE_Vi;
auto slot = RawSlot(id);
ZInstI z(op, slot, CaptureOffset(id));
z.SetType(id->GetType());
z.op_type = OP_VV_I2;
return AddInst(z, true);

Some files were not shown because too many files have changed in this diff Show more