Mirror/zeek
1
0
Fork 0
mirror of https://github.com/zeek/zeek.git synced 2025-10-02 14:48:21 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 2

Merge remote-tracking branch 'origin/topic/christian/extensible-conntuples'

Browse source 
* origin/topic/christian/extensible-conntuples:
  btest/plugins: Add test for custom ConnKey factory
  NEWS updates for pluggable connection tuples.
  Add a VLAN-aware flow tuple implementation.
  Deprecate ConnTuple and related APIs.
  Deprecate the old Connection constructor and detail::ConnKey class.
  Switch to virtualized use of new zeek::ConnKey class tree
  Provide a connkey factory for Zeek's default five-tuples.
  Add IP-specific ConnKey implementation.
  Establish plugin infrastructure for ConnKey factories.
  Add new ConnKey abstraction.
This commit is contained in:
Arne Welzel 2025-06-25 14:17:35 +02:00
commit cd934c460b
72 changed files with 1417 additions and 130 deletions
Download patch file Download diff file Expand all files Collapse all files

57
CHANGES
View file

@ -1,3 +1,60 @@
8.0.0-dev.528 | 2025-06-25 14:17:35 +0200
* btest/plugins: Add test for custom ConnKey factory (Arne Welzel, Corelight)
This just counts DoInits() and adds that information to the conn_id
record, but without including it into the hash. Mostly for smoke
testing.
* NEWS updates for pluggable connection tuples. (Christian Kreibich, Corelight)
* Add a VLAN-aware flow tuple implementation. (Christian Kreibich, Corelight)
This is a first "real" implementation of a custom tuple, adding additional
fields over the standard five-tuple.
Includes test cases.
* Deprecate ConnTuple and related APIs. (Christian Kreibich, Corelight)
Given IP-aware ConnKeys, ConnTuples aren't really required any more. ConnTuple
had two benefits:
- It preserved the original src/dst orientation from the packet headers it was
based on, which IPBasedConnKey now tracks and provides accessor methods for.
- In IPBasedAnalyzer::AnalyzePacket() its instance survived past the std:move()
of the key into NewConn(), which we sidestep by keeping the original src address
and port around until we need after the connection is obtained.
* Deprecate the old Connection constructor and detail::ConnKey class. (Christian Kreibich, Corelight)
The new key-based Connection constructor replaces the former, and the new
ConnKey class tree replaces the latter.
* Switch to virtualized use of new zeek::ConnKey class tree (Christian Kreibich, Corelight)
This touches quite a few places, but each just swaps out existing
APIs and/or zeek::detail::ConnKey instances.
* Provide a connkey factory for Zeek's default five-tuples. (Christian Kreibich, Corelight)
Since the base factory is pure virtual this is now the first full
implementation, but still a bit of a special case because it implements Zeek's
default behavior and doesn't add "custom" content to the tuple.
* Add IP-specific ConnKey implementation. (Christian Kreibich, Corelight)
The InitTuple() implementation here is a placeholder for a fuller one following
later, when we do away with the need for ConnTuple.
* Establish plugin infrastructure for ConnKey factories. (Christian Kreibich, Corelight)
ConnKey factories are intermediaries that encapsulate the details of how to
instantiate ConnKeys, which codify the hash input for connection lookups.
* Add new ConnKey abstraction. (Christian Kreibich, Corelight)
8.0.0-dev.517 | 2025-06-25 09:33:46 +0200
* telemetry: Rename endpoint label to node label (Arne Welzel, Corelight)

28
NEWS
View file

@ -73,6 +73,29 @@ Breaking Changes
New Functionality
-----------------
- Zeek now supports pluggable and customizable connection tracking. The default
behavior remains unchanged and uses a connection's five tuple based on the
IP/port pairs and proto field. Zeek 8 ships with one additional implementation,
to factor VLAN tags into the connection tracking. To switch to VLAN-aware
connection tracking:
@load frameworks/conn_key/vlan_fivetuple
This results in two additional fields in the conn_id record, showing any VLAN
tags involved in the flow. (Accordingly, every log using conn_id reflects the
change as well as these fields have the ``&log`` attribute.)
This feature does not automatically provide a notion of endpoint that
corresponds with the effective flow tuple. For example, applications tracking
endpoints by IP address do not somehow become VLAN-aware when enabling
VLAN-aware tracking.
Users may add their own plugins (for example via a zkg package) to provide
alternative implementations. This involves implementing a factory for
connection "keys" that factor in additional flow information. See the VLAN
implementation in the ``src/packet_analysis/protocol/ip/conn_key/vlan_fivetuple``
directory for an example.
- Generic event metadata support. A new ``EventMetadata`` module was added allowing
to register generic event metadata types and accessing the current event's metadata
using the functions ``current()`` and ``current_all()`` of this module.
@ -234,6 +257,11 @@ Deprecated Functionality
and will lead to compile time warnings. Use ``EventMgr::Enqueue(detail::MetadataVectorPtr meta, ...)``
for populating ``meta`` accordingly.
- For plugin authors: in the core, the constructor for Connection instances has
been deprecated in favor of a new one to support pluggable connection
tuples. The ConnTuple struct, used by this deprecated Connection constructor,
is now deprecated as well.
Zeek 7.2.0
==========

2
VERSION
View file

@ -1 +1 @@
8.0.0-dev.517
8.0.0-dev.528

13
scripts/base/init-bare.zeek
View file

@ -629,6 +629,19 @@ export {
const add_missing_remote_network_timestamp: bool = F &redef;
}
module ConnKey;
export {
## The connection key factory to use for Zeek's internal connection
## tracking. This is a ``ConnKey::Tag`` plugin component enum value,
## and the default is Zeek's traditional 5-tuple-tracking based on
## IP/port endpoint pairs, plus transport protocol. Plugins can provide
## their own implementation. You'll usually not adjust this value in
## isolation, but with a corresponding redef of the :zeek:type:`conn_id`
## record to represent additional connection tuple members.
const factory = ConnKey::CONNKEY_FIVETUPLE &redef;
}
module FTP;
export {

14
scripts/policy/frameworks/conn_key/vlan_fivetuple.zeek Normal file
View file

@ -0,0 +1,14 @@
##! This script adapts Zeek's connection key to include 802.1Q VLAN and
##! Q-in-Q tags, when available. Zeek normally ignores VLAN tags for connection
##! lookups; this change makes it factor them in and also makes those VLAN tags
##! part of the :zeek:see:`conn_id` record.
redef record conn_id += {
## The outer VLAN for this connection, if applicable.
vlan: int &log &optional;
## The inner VLAN for this connection, if applicable.
inner_vlan: int &log &optional;
};
redef ConnKey::factory = ConnKey::CONNKEY_VLAN_FIVETUPLE;

1
scripts/test-all-policy.zeek
View file

@ -114,6 +114,7 @@
@load protocols/conn/mac-logging.zeek
@load protocols/conn/vlan-logging.zeek
@load protocols/conn/weirds.zeek
#@load frameworks/conn_key/vlan_fivetuple.zeek
#@load protocols/conn/speculative-service.zeek
@load protocols/dhcp/msg-orig.zeek
@load protocols/dhcp/software.zeek

1
scripts/zeekygen/__load__.zeek
View file

@ -2,6 +2,7 @@
# Scripts which are commented out in test-all-policy.zeek.
@load frameworks/analyzer/deprecated-dpd-log.zeek
@load frameworks/conn_key/vlan_fivetuple.zeek
# Remove in v8.1: replaced by frameworks/analyzer/detect-protocols.zeek
@pragma push ignore-deprecations

2
src/CMakeLists.txt
View file

@ -197,6 +197,7 @@ gen_zam_target(${GEN_ZAM_SRC_DIR})
option(USE_SQLITE "Should Zeek use SQLite?" ON)
add_subdirectory(analyzer)
add_subdirectory(conn_key)
add_subdirectory(cluster)
add_subdirectory(packet_analysis)
add_subdirectory(broker)
@ -328,6 +329,7 @@ set(MAIN_SRCS
CCL.cc
CompHash.cc
Conn.cc
ConnKey.h
DFA.cc
DbgBreakpoint.cc
DbgHelp.cc

71
src/Conn.cc
View file

@ -23,8 +23,29 @@ namespace zeek {
uint64_t Connection::total_connections = 0;
uint64_t Connection::current_connections = 0;
Connection::Connection(zeek::IPBasedConnKeyPtr k, double t, uint32_t flow, const Packet* pkt)
: Session(t, connection_timeout, connection_status_update, detail::connection_status_update_interval),
key(std::move(k)) {
orig_addr = key->SrcAddr();
resp_addr = key->DstAddr();
orig_port = key->SrcPort();
resp_port = key->DstPort();
switch ( key->Proto() ) {
case IPPROTO_TCP: proto = TRANSPORT_TCP; break;
case IPPROTO_UDP: proto = TRANSPORT_UDP; break;
case IPPROTO_ICMP:
case IPPROTO_ICMPV6: proto = TRANSPORT_ICMP; break;
default: proto = TRANSPORT_UNKNOWN; break;
}
Init(flow, pkt);
}
#pragma GCC diagnostic push
#pragma GCC diagnostic ignored "-Wdeprecated-declarations"
Connection::Connection(const detail::ConnKey& k, double t, const ConnTuple* id, uint32_t flow, const Packet* pkt)
: Session(t, connection_timeout, connection_status_update, detail::connection_status_update_interval), key(k) {
: Session(t, connection_timeout, connection_status_update, detail::connection_status_update_interval) {
orig_addr = id->src_addr;
resp_addr = id->dst_addr;
orig_port = id->src_port;
@ -38,6 +59,29 @@ Connection::Connection(const detail::ConnKey& k, double t, const ConnTuple* id,
default: proto = TRANSPORT_UNKNOWN; break;
}
key = std::make_unique<zeek::IPConnKey>();
key->InitTuple(id->src_addr, id->src_port, id->dst_addr, id->dst_port, id->proto, id->is_one_way);
key->Init(*pkt);
Init(flow, pkt);
}
#pragma GCC diagnostic pop
Connection::~Connection() {
if ( ! finished )
reporter->InternalError("Done() not called before destruction of Connection");
CancelTimers();
if ( conn_val )
conn_val->SetOrigin(nullptr);
delete adapter;
--current_connections;
}
void Connection::Init(uint32_t flow, const Packet* pkt) {
orig_flow_label = flow;
resp_flow_label = 0;
saw_first_orig_packet = 1;
@ -71,20 +115,6 @@ Connection::Connection(const detail::ConnKey& k, double t, const ConnTuple* id,
encapsulation = pkt->encap;
}
Connection::~Connection() {
if ( ! finished )
reporter->InternalError("Done() not called before destruction of Connection");
CancelTimers();
if ( conn_val )
conn_val->SetOrigin(nullptr);
delete adapter;
--current_connections;
}
void Connection::CheckEncapsulation(const std::shared_ptr<EncapsulationStack>& arg_encap) {
if ( encapsulation && arg_encap ) {
if ( *encapsulation != *arg_encap ) {
@ -157,6 +187,13 @@ void Connection::NextPacket(double t, bool is_orig, const IP_Hdr* ip, int len, i
run_state::current_pkt = nullptr;
}
const ConnKey& Connection::Key() const { return *key; }
session::detail::Key Connection::SessionKey(bool copy) const { return key->SessionKey(); }
uint8_t Connection::KeyProto() const { return key->PackedTuple().proto; }
bool Connection::IsReuse(double t, const u_char* pkt) { return adapter && adapter->IsReuse(t, pkt); }
namespace {
@ -186,6 +223,7 @@ const RecordValPtr& Connection::GetVal() {
TransportProto prot_type = ConnTransport();
// XXX this could technically move into IPBasedConnKey.
auto id_val = make_intrusive<RecordVal>(id::conn_id);
id_val->Assign(0, make_intrusive<AddrVal>(orig_addr));
id_val->Assign(1, val_mgr->Port(ntohs(orig_port), prot_type));
@ -193,6 +231,9 @@ const RecordValPtr& Connection::GetVal() {
id_val->Assign(3, val_mgr->Port(ntohs(resp_port), prot_type));
id_val->Assign(4, KeyProto());
// Allow customized ConnKeys to augment the conn_id:
key->PopulateConnIdVal(*id_val);
auto orig_endp = make_intrusive<RecordVal>(id::endpoint);
orig_endp->Assign(0, 0);
orig_endp->Assign(1, 0);

44
src/Conn.h
View file

@ -5,6 +5,7 @@
#include <sys/types.h>
#include <string>
#include "zeek/ConnKey.h"
#include "zeek/IPAddr.h"
#include "zeek/IntrusivePtr.h"
#include "zeek/Rule.h"
@ -27,6 +28,9 @@ class RecordVal;
using ValPtr = IntrusivePtr<Val>;
using RecordValPtr = IntrusivePtr<RecordVal>;
class IPBasedConnKey;
using IPBasedConnKeyPtr = std::unique_ptr<IPBasedConnKey>;
namespace detail {
class Specific_RE_Matcher;
@ -49,13 +53,19 @@ enum ConnEventToFlag : uint8_t {
NUM_EVENTS_TO_FLAG,
};
// Deprecated without replacement: remove in v8.1.
// XXX using [[deprecated]] for the whole struct leads to hard errors on FreeBSD/MacOS.
#pragma GCC diagnostic push
#pragma GCC diagnostic ignored "-Wdeprecated-declarations"
struct ConnTuple {
IPAddr src_addr;
IPAddr dst_addr;
uint32_t src_port = 0;
uint32_t dst_port = 0;
uint16_t proto = UNKNOWN_IP_PROTO;
bool is_one_way = false; // if true, don't canonicalize order
#pragma GCC diagnostic pop
[[deprecated("Remove in v8.1: Switch to new conn_key framework")]] IPAddr src_addr;
[[deprecated("Remove in v8.1: Switch to new conn_key framework")]] IPAddr dst_addr;
[[deprecated("Remove in v8.1: Switch to new conn_key framework")]] uint32_t src_port = 0;
[[deprecated("Remove in v8.1: Switch to new conn_key framework")]] uint32_t dst_port = 0;
[[deprecated("Remove in v8.1: Switch to new conn_key framework")]] uint16_t proto = UNKNOWN_IP_PROTO;
[[deprecated("Remove in v8.1: Switch to new conn_key framework")]] bool is_one_way =
false; // if true, don't canonicalize order
};
static inline int addr_port_canon_lt(const IPAddr& addr1, uint32_t p1, const IPAddr& addr2, uint32_t p2) {
@ -64,7 +74,11 @@ static inline int addr_port_canon_lt(const IPAddr& addr1, uint32_t p1, const IPA
class Connection final : public session::Session {
public:
Connection(zeek::IPBasedConnKeyPtr k, double t, uint32_t flow, const Packet* pkt);
[[deprecated("Remove in v8.1. Switch to ConnKey factories and the new zeek::ConnKey tree.")]]
Connection(const detail::ConnKey& k, double t, const ConnTuple* id, uint32_t flow, const Packet* pkt);
~Connection() override;
/**
@ -101,10 +115,12 @@ public:
// Keys are only considered valid for a connection when a
// connection is in the session map. If it is removed, the key
// should be marked invalid.
const detail::ConnKey& Key() const { return key; }
session::detail::Key SessionKey(bool copy) const override {
return session::detail::Key{&key, sizeof(key), session::detail::Key::CONNECTION_KEY_TYPE, copy};
}
//
// These touch the key, which we forward-declared above. Therefore this
// hides the implementation, which has the full class definition.
const ConnKey& Key() const;
session::detail::Key SessionKey(bool copy) const override;
uint8_t KeyProto() const;
const IPAddr& OrigAddr() const { return orig_addr; }
const IPAddr& RespAddr() const { return resp_addr; }
@ -130,8 +146,6 @@ public:
return "unknown";
}
uint8_t KeyProto() const { return key.transport; }
// Returns true if the packet reflects a reuse of this
// connection (i.e., not a continuation but the beginning of
// a new connection).
@ -196,6 +210,10 @@ public:
bool IsFinished() { return finished; }
private:
// Common initialization for the constructors. This can move back into the
// (sole) constructor when we remove the deprecated one in 8.1.
void Init(uint32_t flow, const Packet* pkt);
friend class session::detail::Timer;
IPAddr orig_addr;
@ -211,7 +229,7 @@ private:
std::shared_ptr<EncapsulationStack> encapsulation; // tunnels
uint8_t tunnel_changes = 0;
detail::ConnKey key;
IPBasedConnKeyPtr key;
unsigned int weird : 1;
unsigned int finished : 1;

89
src/ConnKey.h Normal file
View file

@ -0,0 +1,89 @@
// See the file "COPYING" in the main distribution directory for copyright.
#pragma once
#include <memory>
#include "zeek/IntrusivePtr.h"
#include "zeek/session/Key.h"
namespace zeek {
class Packet;
class RecordVal;
using RecordValPtr = zeek::IntrusivePtr<RecordVal>;
/**
* Abstract ConnKey, for any type of connection.
*/
class ConnKey {
public:
virtual ~ConnKey() = default;
/**
* Initialization of this key with the current packet.
*
* @param pkt The packet that's currently being processed.
*/
void Init(const Packet& pkt) { DoInit(pkt); }
/**
* When Zeek renders a connection into a script-layer record, it calls this
* method to populate custom conn_id fields unique to this ConnKey, such as
* VLAN fields. This only needs to populate in fields in addition to Zeek's
* five-tuple (i.e., complete the record, not populate all of it).
*
* The default implementation does nothing.
*
* @param conn_id The conn_id record to populate.
*/
void PopulateConnIdVal(RecordVal& conn_id) { DoPopulateConnIdVal(conn_id); };
/**
* Return a non-owning session::detail::Key instance for connection lookups.
*
* Callers that need more than just a view of the key should copy the data.
* Callers are not supposed to hold on to the returned Key for longer than
* the ConnKey instance exists.
*
* @return A zeek::session::detail::Key
*/
zeek::session::detail::Key SessionKey() const { return DoSessionKey(); }
protected:
/**
* Hook method for ConnKey::Init.
*
* Note that a given ConnKey instance may be re-used for different
* packets if it wasn't consumed to create a new connection. Therefore,
* implementers of this method are required to always set all fields
* that will affect the SessionKey result within DoInit anew.
*
* This a bit of an optimization done in the packet path that's shining
* through here. Rather than introducing a dedicated Reset method,
* implementers are asked to reset the key at initialization time
* which they most likely would do anyhow.
*
* @param pkt The packet that's currently being processed.
*/
virtual void DoInit(const Packet& pkt) {};
/**
* Hook method for ConnKey::PopulateConnIdVal.
*
* The default implementation does nothing.
*/
virtual void DoPopulateConnIdVal(RecordVal& conn_id) {}
/**
* Hook method for implementing ConnKey::SessionKey.
*
* @return A zeek::session::detail::Key
*/
virtual session::detail::Key DoSessionKey() const = 0;
};
using ConnKeyPtr = std::unique_ptr<ConnKey>;
} // namespace zeek

3
src/IPAddr.cc
View file

@ -23,6 +23,8 @@ ConnKey::ConnKey(const IPAddr& src, const IPAddr& dst, uint16_t src_port, uint16
Init(src, dst, src_port, dst_port, proto, one_way);
}
#pragma GCC diagnostic push
#pragma GCC diagnostic ignored "-Wdeprecated-declarations"
ConnKey::ConnKey(const ConnTuple& id) {
Init(id.src_addr, id.dst_addr, id.src_port, id.dst_port, id.proto, id.is_one_way);
}
@ -45,6 +47,7 @@ ConnKey& ConnKey::operator=(const ConnKey& rhs) {
return *this;
}
#pragma GCC diagnostic pop
ConnKey::ConnKey(Val* v) {
const auto& vt = v->GetType();

11
src/IPAddr.h
View file

@ -25,6 +25,7 @@ constexpr uint16_t INVALID_CONN_KEY_IP_PROTO = 65534;
class HashKey;
// Deprecated: Remove the whole class in v8.1. Switch usage to the conntuple factories and the new zeek::ConnKey tree.
class ConnKey {
public:
in6_addr ip1;
@ -33,10 +34,12 @@ public:
uint16_t port2 = 0;
uint16_t transport = INVALID_CONN_KEY_IP_PROTO;
ConnKey(const IPAddr& src, const IPAddr& dst, uint16_t src_port, uint16_t dst_port, uint16_t proto, bool one_way);
ConnKey(const ConnTuple& conn);
ConnKey(const ConnKey& rhs) { *this = rhs; }
ConnKey(Val* v);
[[deprecated("Remove in v8.1: Switch to new conn_key framework")]] ConnKey(const IPAddr& src, const IPAddr& dst,
uint16_t src_port, uint16_t dst_port,
uint16_t proto, bool one_way);
[[deprecated("Remove in v8.1: Switch to new conn_key framework")]] ConnKey(const ConnTuple& conn);
[[deprecated("Remove in v8.1: Switch to new conn_key framework")]] ConnKey(const ConnKey& rhs) { *this = rhs; }
[[deprecated("Remove in v8.1: Switch to new conn_key framework")]] ConnKey(Val* v);
// FIXME: This is getting reworked as part of the connection tuple changes. Suppress
// the clang-tidy warning for the time being.

9
src/analyzer/Analyzer.cc
View file

@ -8,6 +8,7 @@
#include "zeek/Conn.h"
#include "zeek/Event.h"
#include "zeek/analyzer/Manager.h"
#include "zeek/packet_analysis/protocol/ip/conn_key/IPBasedConnKey.h"
#include "zeek/packet_analysis/protocol/tcp/TCPSessionAdapter.h"
#include "zeek/3rdparty/doctest.h"
@ -806,8 +807,8 @@ TEST_SUITE("Analyzer management") {
REQUIRE(zeek::analyzer_mgr);
zeek::Packet p;
zeek::ConnTuple t;
auto conn = std::make_unique<zeek::Connection>(zeek::detail::ConnKey(t), 0, &t, 0, &p);
zeek::IPBasedConnKeyPtr kp = std::make_unique<zeek::IPConnKey>();
auto conn = std::make_unique<zeek::Connection>(std::move(kp), 0, 0, &p);
auto* tcp = new zeek::packet_analysis::TCP::TCPSessionAdapter(conn.get());
conn->SetSessionAdapter(tcp, nullptr);
@ -838,8 +839,8 @@ TEST_SUITE("Analyzer management") {
REQUIRE(zeek::analyzer_mgr);
zeek::Packet p;
zeek::ConnTuple t;
auto conn = std::make_unique<zeek::Connection>(zeek::detail::ConnKey(t), 0, &t, 0, &p);
zeek::IPBasedConnKeyPtr kp = std::make_unique<zeek::IPConnKey>();
auto conn = std::make_unique<zeek::Connection>(std::move(kp), 0, 0, &p);
auto ssh = zeek::analyzer_mgr->InstantiateAnalyzer("SSH", conn.get());
REQUIRE(ssh);

5
src/analyzer/protocol/smtp/BDAT.cc
View file

@ -5,6 +5,7 @@
#include "zeek/Conn.h"
#include "zeek/DebugLogger.h"
#include "zeek/analyzer/protocol/mime/MIME.h"
#include "zeek/packet_analysis/protocol/ip/conn_key/IPBasedConnKey.h"
#include "zeek/util.h"
#include "zeek/3rdparty/doctest.h"
@ -327,8 +328,8 @@ private:
TEST_CASE("line forward testing") {
zeek::Packet p;
zeek::ConnTuple t;
auto conn = std::make_unique<zeek::Connection>(zeek::detail::ConnKey(t), 0, &t, 0, &p);
zeek::IPBasedConnKeyPtr kp = std::make_unique<zeek::IPConnKey>();
auto conn = std::make_unique<zeek::Connection>(std::move(kp), 0, 0, &p);
auto smtp_analyzer =
std::unique_ptr<zeek::analyzer::Analyzer>(zeek::analyzer_mgr->InstantiateAnalyzer("SMTP", conn.get()));
auto mail = std::make_unique<Test_MIME_Message>(smtp_analyzer.get());

1
src/conn_key/CMakeLists.txt Normal file
View file

@ -0,0 +1 @@
zeek_add_subdir_library(connkey SOURCES Factory.h Component.cc Manager.cc)

27
src/conn_key/Component.cc Normal file
View file

@ -0,0 +1,27 @@
// See the file "COPYING" in the main distribution directory for copyright.
#include "zeek/conn_key/Component.h"
#include "zeek/Desc.h"
#include "zeek/conn_key/Manager.h"
using namespace zeek::conn_key;
Component::Component(const std::string& name, factory_callback arg_factory, Tag::subtype_t arg_subtype)
: plugin::Component(plugin::component::CONNKEY, name, arg_subtype, conn_key_mgr->GetTagType()),
factory(std::move(arg_factory)) {}
void Component::Initialize() {
InitializeTag();
conn_key_mgr->RegisterComponent(this, "CONNKEY_");
}
void Component::DoDescribe(ODesc* d) const {
if ( factory ) {
d->Add("CONNKEY_");
d->Add(CanonicalName());
d->Add(", ");
}
d->Add(Enabled() ? "enabled" : "disabled");
}

45
src/conn_key/Component.h Normal file
View file

@ -0,0 +1,45 @@
// See the file "COPYING" in the main distribution directory for copyright.
#pragma once
#include <functional>
#include <memory>
#include "zeek/Tag.h"
#include "zeek/plugin/Component.h"
namespace zeek::conn_key {
class Factory;
using FactoryPtr = std::unique_ptr<Factory>;
class Component : public plugin::Component {
public:
using factory_callback = std::function<FactoryPtr()>;
Component(const std::string& name, factory_callback factory, zeek::Tag::subtype_t subtype = 0);
~Component() override = default;
/**
* Initialization function. This function has to be called before any
* plugin component functionality is used; it is used to add the
* plugin component to the list of components and to initialize tags
*/
void Initialize() override;
/**
* Returns the analyzer's factory function.
*/
factory_callback Factory() const { return factory; }
protected:
/**
* Overridden from plugin::Component.
*/
void DoDescribe(ODesc* d) const override;
private:
factory_callback factory; // The tuple factory's factory callback.
};
} // namespace zeek::conn_key

62
src/conn_key/Factory.h Normal file
View file

@ -0,0 +1,62 @@
// See the file "COPYING" in the main distribution directory for copyright.
#pragma once
#include "zeek/ConnKey.h"
#include "zeek/util-types.h"
namespace zeek {
class Packet;
class RecordVal;
using RecordValPtr = IntrusivePtr<RecordVal>;
namespace conn_key {
class Factory;
using FactoryPtr = std::unique_ptr<Factory>;
/**
* ConnKey factories instantiate derivatives of ConnKeys, to provide pluggable flow hashing.
*/
class Factory {
public:
virtual ~Factory() = default;
/**
* Instantiates a clean ConnKey derivative and returns it.
*
* @return A unique pointer to the ConnKey instance.
*/
zeek::ConnKeyPtr NewConnKey() const { return DoNewConnKey(); }
/**
* Instantiates a filled-in ConnKey derivative from a script-layer
* record, usually a conn_id instance. Implementations are free to
* implement this liberally, i.e. the input does not _have_ to be a
* conn_id.
*
* @param v The script-layer value providing key input.
* @return A unique pointer to the ConnKey instance, or an error message.
*/
zeek::expected<zeek::ConnKeyPtr, std::string> ConnKeyFromVal(const zeek::Val& v) const {
return DoConnKeyFromVal(v);
}
protected:
/**
* Hook for Factory::NewConnKey.
*
* @return A unique pointer to the ConnKey instance.
*/
virtual zeek::ConnKeyPtr DoNewConnKey() const = 0;
/**
* Hook for Factory::ConnKeyFromVal.
*
* @return A unique pointer to the ConnKey instance, or an error message.
*/
virtual zeek::expected<zeek::ConnKeyPtr, std::string> DoConnKeyFromVal(const zeek::Val& v) const = 0;
};
} // namespace conn_key
} // namespace zeek

39
src/conn_key/Manager.cc Normal file
View file

@ -0,0 +1,39 @@
// See the file "COPYING" in the main distribution directory for copyright.
#include "zeek/conn_key/Manager.h"
#include "zeek/conn_key/Component.h"
using namespace zeek::conn_key;
Manager::Manager() : plugin::ComponentManager<zeek::conn_key::Component>("ConnKey", "Tag") {}
void Manager::InitPostScript() {
const auto& factory_val = id::find_val<zeek::EnumVal>("ConnKey::factory");
factory = InstantiateFactory(factory_val);
}
FactoryPtr Manager::InstantiateFactory(const zeek::EnumValPtr& tag) {
Component* c = Lookup(tag);
if ( ! c ) {
reporter->FatalError(
"request to instantiate unknown connection tuple factory %s, please review ConnTuple::factory value",
tag->GetType()->AsEnumType()->Lookup(tag->Get()));
}
if ( ! c->Factory() ) {
reporter->FatalError("factory %s cannot be instantiated dynamically", GetComponentName(tag).c_str());
}
FactoryPtr factory = c->Factory()();
if ( ! factory ) {
reporter->FatalError("factory instantiation failed");
}
// Could add validation of actual tag vs obtained one here, as we do e.g. in
// the packet_analysis Manager.
return factory;
}

57
src/conn_key/Manager.h Normal file
View file

@ -0,0 +1,57 @@
// See the file "COPYING" in the main distribution directory for copyright.
#pragma once
#include "zeek/Tag.h"
#include "zeek/conn_key/Component.h"
#include "zeek/conn_key/Factory.h"
#include "zeek/plugin/Component.h"
#include "zeek/plugin/ComponentManager.h"
namespace zeek {
namespace conn_key {
/**
* This component manager is for registration of pluggable ConnKey factories
* that provide a zeek::plugin::component::CONNKEY component.
*/
class Manager : public plugin::ComponentManager<conn_key::Component> {
public:
/**
* Constructor.
*/
Manager();
/**
* Destructor.
*/
~Manager() = default;
/**
* Hook called during Zeek's startup sequence at InitPostScript() time.
*/
void InitPostScript();
/**
* Return the instantiated Factory selected by the @c ConnKey::factory script-level variable.
*
* @return A reference to the selected see Factory.
*/
Factory& GetFactory() { return *factory; }
private:
/**
* @return A pointer to a Factory given @arg tag.
*/
FactoryPtr InstantiateFactory(const EnumValPtr& tag);
FactoryPtr factory;
};
} // namespace conn_key
extern zeek::conn_key::Manager* conn_key_mgr;
} // namespace zeek

10
src/packet_analysis/protocol/gtpv1/GTPv1.cc
View file

@ -19,11 +19,15 @@ bool GTPv1_Analyzer::AnalyzePacket(size_t len, const uint8_t* data, Packet* pack
}
auto conn = static_cast<Connection*>(packet->session);
zeek::detail::ConnKey conn_key = conn->Key();
const auto& key = conn->Key();
auto sk = key.SessionKey();
auto cm_it = conn_map.find(conn_key);
auto cm_it = conn_map.find(sk);
if ( cm_it == conn_map.end() ) {
cm_it = conn_map.insert(cm_it, {conn_key, std::make_unique<binpac::GTPv1::GTPv1_Conn>(this)});
sk.CopyData(); // Copy key data to store in map.
auto [it, inserted] = conn_map.emplace(std::move(sk), std::make_unique<binpac::GTPv1::GTPv1_Conn>(this));
assert(inserted);
cm_it = it;
// Let script land know about the state we created, so it will
// register a conn removal hook for cleanup.

6
src/packet_analysis/protocol/gtpv1/GTPv1.h
View file

@ -3,6 +3,7 @@
#pragma once
#include "zeek/packet_analysis/Analyzer.h"
#include "zeek/session/Key.h"
#include "packet_analysis/protocol/gtpv1/gtpv1_pac.h"
@ -27,11 +28,10 @@ public:
gtp_hdr_val = std::move(val);
}
void RemoveConnection(const zeek::detail::ConnKey& conn_key) { conn_map.erase(conn_key); }
void RemoveConnection(const zeek::session::detail::Key& conn_key) { conn_map.erase(conn_key); }
protected:
using ConnMap = std::map<zeek::detail::ConnKey, std::unique_ptr<binpac::GTPv1::GTPv1_Conn>>;
ConnMap conn_map;
std::map<zeek::session::detail::Key, std::unique_ptr<binpac::GTPv1::GTPv1_Conn>> conn_map;
int inner_packet_offset = -1;
uint8_t next_header = 0;

9
src/packet_analysis/protocol/gtpv1/functions.bif
View file

@ -2,6 +2,7 @@ module PacketAnalyzer::GTPV1;
%%{
#include "zeek/Conn.h"
#include "zeek/conn_key/Manager.h"
#include "zeek/session/Manager.h"
#include "zeek/packet_analysis/Manager.h"
#include "zeek/packet_analysis/protocol/gtpv1/GTPv1.h"
@ -12,8 +13,12 @@ function remove_gtpv1_connection%(cid: conn_id%) : bool
zeek::packet_analysis::AnalyzerPtr gtpv1 = zeek::packet_mgr->GetAnalyzer("GTPv1");
if ( gtpv1 )
{
zeek::detail::ConnKey conn_key(cid);
static_cast<zeek::packet_analysis::gtpv1::GTPv1_Analyzer*>(gtpv1.get())->RemoveConnection(conn_key);
auto r = zeek::conn_key_mgr->GetFactory().ConnKeyFromVal(*cid);
if ( ! r.has_value() )
return zeek::val_mgr->False();
auto sk = r.value()->SessionKey();
static_cast<zeek::packet_analysis::gtpv1::GTPv1_Analyzer*>(gtpv1.get())->RemoveConnection(sk);
}
return zeek::val_mgr->True();

17
src/packet_analysis/protocol/icmp/ICMP.cc
View file

@ -28,24 +28,25 @@ SessionAdapter* ICMPAnalyzer::MakeSessionAdapter(Connection* conn) {
return root;
}
bool ICMPAnalyzer::BuildConnTuple(size_t len, const uint8_t* data, Packet* packet, ConnTuple& tuple) {
bool ICMPAnalyzer::InitConnKey(size_t len, const uint8_t* data, Packet* packet, IPBasedConnKey& key) {
if ( ! CheckHeaderTrunc(ICMP_MINLEN, len, packet) )
return false;
tuple.src_addr = packet->ip_hdr->SrcAddr();
tuple.dst_addr = packet->ip_hdr->DstAddr();
tuple.proto = packet->proto;
const struct icmp* icmpp = (const struct icmp*)data;
tuple.src_port = htons(icmpp->icmp_type);
uint32_t icmp_counter_type = 0;
bool is_one_way = false;
if ( packet->proto == IPPROTO_ICMP )
tuple.dst_port = htons(ICMP4_counterpart(icmpp->icmp_type, icmpp->icmp_code, tuple.is_one_way));
icmp_counter_type = ICMP4_counterpart(icmpp->icmp_type, icmpp->icmp_code, is_one_way);
else if ( packet->proto == IPPROTO_ICMPV6 )
tuple.dst_port = htons(ICMP6_counterpart(icmpp->icmp_type, icmpp->icmp_code, tuple.is_one_way));
icmp_counter_type = ICMP6_counterpart(icmpp->icmp_type, icmpp->icmp_code, is_one_way);
else
reporter->InternalError("Reached ICMP packet analyzer with unknown packet protocol %x", packet->proto);
key.InitTuple(packet->ip_hdr->SrcAddr(), htons(icmpp->icmp_type), packet->ip_hdr->DstAddr(),
htons(icmp_counter_type), packet->proto, is_one_way);
return true;
}

5
src/packet_analysis/protocol/icmp/ICMP.h
View file

@ -29,10 +29,7 @@ public:
packet_analysis::IP::SessionAdapter* MakeSessionAdapter(Connection* conn) override;
protected:
/**
* Parse the header from the packet into a ConnTuple object.
*/
bool BuildConnTuple(size_t len, const uint8_t* data, Packet* packet, ConnTuple& tuple) override;
bool InitConnKey(size_t len, const uint8_t* data, Packet* packet, IPBasedConnKey& key) override;
void DeliverPacket(Connection* c, double t, bool is_orig, int remaining, Packet* pkt) override;

2
src/packet_analysis/protocol/ip/CMakeLists.txt
View file

@ -1,3 +1,5 @@
zeek_add_plugin(
PacketAnalyzer IP
SOURCES IP.cc IPBasedAnalyzer.cc SessionAdapter.cc Plugin.cc)
add_subdirectory(conn_key)

48
src/packet_analysis/protocol/ip/IPBasedAnalyzer.cc
View file

@ -7,6 +7,8 @@
#include "zeek/Val.h"
#include "zeek/analyzer/Manager.h"
#include "zeek/analyzer/protocol/pia/PIA.h"
#include "zeek/conn_key/Manager.h"
#include "zeek/packet_analysis/protocol/ip/conn_key/IPBasedConnKey.h"
#include "zeek/plugin/Manager.h"
#include "zeek/session/Manager.h"
@ -22,17 +24,39 @@ IPBasedAnalyzer::~IPBasedAnalyzer() {
}
bool IPBasedAnalyzer::AnalyzePacket(size_t len, const uint8_t* data, Packet* pkt) {
static IPBasedConnKeyPtr key; // Note, this is static for reuse:
if ( ! key ) {
ConnKeyPtr ck = conn_key_mgr->GetFactory().NewConnKey();
// The IPBasedAnalyzer requires a factory that produces IPBasedConnKey instances.
// We could check with dynamic_cast, but that's probably slow, so assume plugin
// providers know what they're doing here and anyhow, we don't really have analyzers
// that instantiate non-IP connections today and definitely not here!
key = IPBasedConnKeyPtr(static_cast<IPBasedConnKey*>(ck.release()));
}
// Deprecated: remove ConnTuple use in 8.1 and only use InitConnKey().
#pragma GCC diagnostic push
#pragma GCC diagnostic ignored "-Wdeprecated-declarations"
ConnTuple tuple;
if ( ! BuildConnTuple(len, data, pkt, tuple) )
if ( BuildConnTuple(len, data, pkt, tuple) ) {
key->InitTuple(tuple.src_addr, tuple.src_port, tuple.dst_addr, tuple.dst_port, pkt->proto);
#pragma GCC diagnostic pop
}
else if ( ! InitConnKey(len, data, pkt, *key) ) {
return false;
}
key->Init(*pkt);
const std::shared_ptr<IP_Hdr>& ip_hdr = pkt->ip_hdr;
zeek::detail::ConnKey key(tuple);
auto src_addr = key->SrcAddr();
auto src_port = key->SrcPort();
Connection* conn = session_mgr->FindConnection(key);
Connection* conn = session_mgr->FindConnection(*key);
if ( ! conn ) {
conn = NewConn(&tuple, key, pkt);
conn = NewConn(std::move(key), pkt);
if ( conn )
session_mgr->Insert(conn, false);
}
@ -41,7 +65,7 @@ bool IPBasedAnalyzer::AnalyzePacket(size_t len, const uint8_t* data, Packet* pkt
conn->Event(connection_reused, nullptr);
session_mgr->Remove(conn);
conn = NewConn(&tuple, key, pkt);
conn = NewConn(std::move(key), pkt);
if ( conn )
session_mgr->Insert(conn, false);
}
@ -57,7 +81,7 @@ bool IPBasedAnalyzer::AnalyzePacket(size_t len, const uint8_t* data, Packet* pkt
// get logged, which means we can mark this packet as having been processed.
pkt->processed = true;
bool is_orig = (tuple.src_addr == conn->OrigAddr()) && (tuple.src_port == conn->OrigPort());
bool is_orig = (src_addr == conn->OrigAddr()) && (src_port == conn->OrigPort());
pkt->is_orig = is_orig;
conn->CheckFlowLabel(is_orig, ip_hdr->FlowLabel());
@ -140,18 +164,18 @@ bool IPBasedAnalyzer::IsLikelyServerPort(uint32_t port) const {
return port_cache.find(port) != port_cache.end();
}
zeek::Connection* IPBasedAnalyzer::NewConn(const ConnTuple* id, const zeek::detail::ConnKey& key, const Packet* pkt) {
int src_h = ntohs(id->src_port);
int dst_h = ntohs(id->dst_port);
zeek::Connection* IPBasedAnalyzer::NewConn(IPBasedConnKeyPtr key, const Packet* pkt) {
auto src_p = ntohs(key->SrcPort());
auto dst_p = ntohs(key->DstPort());
bool flip = false;
if ( ! WantConnection(src_h, dst_h, pkt->ip_hdr->Payload(), flip) )
if ( ! WantConnection(src_p, dst_p, pkt->ip_hdr->Payload(), flip) )
return nullptr;
Connection* conn = new Connection(key, run_state::processing_start_time, id, pkt->ip_hdr->FlowLabel(), pkt);
Connection* conn = new Connection(std::move(key), run_state::processing_start_time, pkt->ip_hdr->FlowLabel(), pkt);
conn->SetTransport(transport);
if ( flip && ! id->dst_addr.IsBroadcast() )
if ( flip && ! conn->RespAddr().IsBroadcast() )
conn->FlipRoles();
BuildSessionAnalyzerTree(conn);

30
src/packet_analysis/protocol/ip/IPBasedAnalyzer.h
View file

@ -7,6 +7,7 @@
#include "zeek/Tag.h"
#include "zeek/packet_analysis/Analyzer.h"
#include "zeek/packet_analysis/protocol/ip/conn_key/IPBasedConnKey.h"
namespace zeek::analyzer::pia {
class PIA;
@ -97,10 +98,29 @@ protected:
*/
IPBasedAnalyzer(const char* name, TransportProto proto, uint32_t mask, bool report_unknown_protocols);
/**
* Initialize the given ConnKey from the packet header & data.
*
* @param len Remaining length of data.
* @param data Remaining packet data.
* @param packet The packet being processed.
* @param key The ConnKey instance to initialize.
*
* @return True if initialization succeeded, false otherwise (e.g. because
* there wasn't enough data available).
*/
virtual bool InitConnKey(size_t len, const uint8_t* data, Packet* packet, IPBasedConnKey& key) {
// Given deprecation of BuildConnTuple below, make this pure virtual in 8.1.
return false;
}
/**
* Parse the header from the packet into a ConnTuple object.
*/
virtual bool BuildConnTuple(size_t len, const uint8_t* data, Packet* packet, ConnTuple& tuple) = 0;
[[deprecated("Remove in v8.1. Switch to InitConnKey() and key-only initialization.")]]
virtual bool BuildConnTuple(size_t len, const uint8_t* data, Packet* packet, ConnTuple& tuple) {
return false;
}
/**
* Continues process of packet after the connection has been inserted into the
@ -179,12 +199,10 @@ private:
/**
* Creates a new Connection object from data gleaned from the current packet.
*
* @param id A connection ID generated from the packet data. This should have been
* passed in from a child analyzer.
* @param key A connection ID key generated from the ID.
* @param pkt The packet associated with the new connection.
* @param key A ConnKey with common 5-tuple information.
* @param pkt The packet associated with the new connection, for additional connection info.
*/
zeek::Connection* NewConn(const ConnTuple* id, const zeek::detail::ConnKey& key, const Packet* pkt);
zeek::Connection* NewConn(IPBasedConnKeyPtr key, const Packet* pkt);
void BuildSessionAnalyzerTree(Connection* conn);

4
src/packet_analysis/protocol/ip/conn_key/CMakeLists.txt Normal file
View file

@ -0,0 +1,4 @@
zeek_add_subdir_library(connkey-ip SOURCES IPBasedConnKey.cc)
add_subdirectory(fivetuple)
add_subdirectory(vlan_fivetuple)

30
src/packet_analysis/protocol/ip/conn_key/IPBasedConnKey.cc Normal file
View file

@ -0,0 +1,30 @@
// See the file "COPYING" in the main distribution directory for copyright.
#include "zeek/packet_analysis/protocol/ip/conn_key/IPBasedConnKey.h"
#include "zeek/Conn.h"
using namespace zeek;
using namespace zeek::packet_analysis::IP;
void IPBasedConnKey::InitTuple(const IPAddr& src_addr, uint32_t src_port, const IPAddr& dst_addr, uint32_t dst_port,
uint16_t proto, bool is_one_way) {
auto& tuple = PackedTuple();
if ( is_one_way || addr_port_canon_lt(src_addr, src_port, dst_addr, dst_port) ) {
src_addr.CopyIPv6(&tuple.ip1);
dst_addr.CopyIPv6(&tuple.ip2);
tuple.port1 = src_port;
tuple.port2 = dst_port;
flipped = false;
}
else {
dst_addr.CopyIPv6(&tuple.ip1);
src_addr.CopyIPv6(&tuple.ip2);
tuple.port1 = dst_port;
tuple.port2 = src_port;
flipped = true;
}
tuple.proto = proto;
}

130
src/packet_analysis/protocol/ip/conn_key/IPBasedConnKey.h Normal file
View file

@ -0,0 +1,130 @@
// See the file "COPYING" in the main distribution directory for copyright.
#pragma once
#include <netinet/in.h>
#include "zeek/Conn.h"
#include "zeek/ConnKey.h"
#include "zeek/IPAddr.h"
namespace zeek {
namespace detail {
/**
* Struct for embedding into an IPBasedConnKey.
*/
struct PackedConnTuple {
in6_addr ip1;
in6_addr ip2;
uint16_t port1 = 0;
uint16_t port2 = 0;
uint16_t proto = 0;
} __attribute__((packed, aligned));
} // namespace detail
/**
* Abstract key class for IP-based connections.
*
* ConnKey instances for IP always hold a ConnTuple instance which is provided
* by the IPBasedAnalyzer. The InitConnTuple() method stores a normalized version
* in the tuple, losing the information about orig and responder.
*/
class IPBasedConnKey : public zeek::ConnKey {
public:
/**
* Initializes the key to the given 5-tuple. This canonicalizes the
* packed tuple storage, including potential endpoint flips for
* consistent connection lookups regardless of directionality.
*/
void InitTuple(const IPAddr& src_addr, uint32_t src_port, const IPAddr& dst_addr, uint32_t dst_port, uint16_t proto,
bool is_one_way = false);
/**
* The source address the key got initialized with.
*/
IPAddr SrcAddr() const { return flipped ? IPAddr(PackedTuple().ip2) : IPAddr(PackedTuple().ip1); }
/**
* The destination address the key got initialized with.
*/
IPAddr DstAddr() const { return flipped ? IPAddr(PackedTuple().ip1) : IPAddr(PackedTuple().ip2); }
/**
* The source port the key got initialized with.
*/
uint16_t SrcPort() const { return flipped ? PackedTuple().port2 : PackedTuple().port1; }
/**
* The destination port the key got initialized with.
*/
uint16_t DstPort() const { return flipped ? PackedTuple().port1 : PackedTuple().port2; }
/**
* The IP protocol the key got initialized with.
*/
uint16_t Proto() const { return PackedTuple().proto; }
/**
* Return a modifiable reference to the embedded PackedConnTuple.
*
* This is virtual to give subclasses control over where
* to place the tuple within the key.
*
* @return A modifiable reference to the embedded PackedConnTuple.
*/
virtual detail::PackedConnTuple& PackedTuple() = 0;
/**
* Return a non-modifiable reference to the embedded PackedConnTuple.
*
* This is virtual to give subclasses control over where
* to place the tuple within the key.
*
* @return A non-modifiable reference to the embedded PackedConnTuple.
*/
virtual const detail::PackedConnTuple& PackedTuple() const = 0;
protected:
bool flipped = false;
};
using IPBasedConnKeyPtr = std::unique_ptr<IPBasedConnKey>;
/**
* The usual 5-tuple ConnKey, fully instantiable.
*/
class IPConnKey : public IPBasedConnKey {
public:
/**
* Constructor.
*
* Fill any holes in the key struct as we use the full tuple as a key.
*/
IPConnKey() { memset(static_cast<void*>(&key), 0, sizeof(key)); }
/**
* @copydoc
*/
detail::PackedConnTuple& PackedTuple() override { return key.tuple; }
/**
* @copydoc
*/
const detail::PackedConnTuple& PackedTuple() const override { return key.tuple; }
protected:
/**
* @copydoc
*/
zeek::session::detail::Key DoSessionKey() const override {
return {reinterpret_cast<const void*>(&key), sizeof(key),
// XXX: Not sure we need CONNECTION_KEY_TYPE?
session::detail::Key::CONNECTION_KEY_TYPE};
}
private:
struct {
struct detail::PackedConnTuple tuple;
} key;
};
} // namespace zeek

3
src/packet_analysis/protocol/ip/conn_key/fivetuple/CMakeLists.txt Normal file
View file

@ -0,0 +1,3 @@
zeek_add_plugin(
Zeek ConnKey_Fivetuple
SOURCES Factory.cc Plugin.cc)

75
src/packet_analysis/protocol/ip/conn_key/fivetuple/Factory.cc Normal file
View file

@ -0,0 +1,75 @@
// See the file "COPYING" in the main distribution directory for copyright.
#include "zeek/packet_analysis/protocol/ip/conn_key/fivetuple/Factory.h"
#include "zeek/IP.h"
#include "zeek/Val.h"
#include "zeek/packet_analysis/protocol/ip/conn_key/IPBasedConnKey.h"
#include "zeek/util-types.h"
namespace zeek::conn_key::fivetuple {
zeek::ConnKeyPtr Factory::DoNewConnKey() const { return std::make_unique<zeek::IPConnKey>(); }
zeek::expected<zeek::ConnKeyPtr, std::string> Factory::DoConnKeyFromVal(const zeek::Val& v) const {
static auto unexpected_conn_id = zeek::unexpected<std::string>{"invalid connection ID record encountered"};
auto ck = NewConnKey();
auto* ick = static_cast<zeek::IPBasedConnKey*>(ck.get());
auto& pt = ick->PackedTuple();
const auto& vt = v.GetType();
if ( ! IsRecord(vt->Tag()) )
return unexpected_conn_id;
auto* vr = vt->AsRecordType();
auto vl = v.AsRecordVal();
// Indices into conn_id's record field value list:
int orig_h = 0, orig_p = 1, resp_h = 2, resp_p = 3, proto = 4;
if ( vr != id::conn_id ) {
// While it's not a conn_id, it may have equivalent fields.
orig_h = vr->FieldOffset("orig_h");
resp_h = vr->FieldOffset("resp_h");
orig_p = vr->FieldOffset("orig_p");
resp_p = vr->FieldOffset("resp_p");
proto = vr->FieldOffset("proto");
// clang-format off
if ( orig_h < 0 || vr->GetFieldType(orig_h)->Tag() != TYPE_ADDR ||
resp_h < 0 || vr->GetFieldType(resp_h)->Tag() != TYPE_ADDR ||
orig_p < 0 || vr->GetFieldType(orig_p)->Tag() != TYPE_PORT ||
resp_p < 0 || vr->GetFieldType(resp_p)->Tag() != TYPE_PORT ||
proto < 0 || vr->GetFieldType(proto)->Tag() != TYPE_COUNT ) {
return unexpected_conn_id;
}
// clang-format on
}
if ( ! vl->HasField(orig_h) || ! vl->HasField(resp_h) || ! vl->HasField(orig_p) || ! vl->HasField(resp_p) ||
! vl->HasField(proto) ) {
return unexpected_conn_id;
}
const IPAddr& orig_addr = vl->GetFieldAs<AddrVal>(orig_h);
const IPAddr& resp_addr = vl->GetFieldAs<AddrVal>(resp_h);
const auto& orig_portv = vl->GetFieldAs<PortVal>(orig_p);
const auto& resp_portv = vl->GetFieldAs<PortVal>(resp_p);
const auto& protov = vl->GetField<CountVal>(proto);
auto proto16_t = static_cast<uint16_t>(protov->AsCount());
if ( proto16_t == UNKNOWN_IP_PROTO )
return zeek::unexpected<std::string>(
"invalid connection ID record encountered: the proto field has the \"unknown\" 65535 value. "
"Did you forget to set it?");
ick->InitTuple(orig_addr, htons(orig_portv->Port()), resp_addr, htons(resp_portv->Port()), proto16_t);
// Asserting here on the absence of errors can fail btests.
return ck;
}
} // namespace zeek::conn_key::fivetuple

33
src/packet_analysis/protocol/ip/conn_key/fivetuple/Factory.h Normal file
View file

@ -0,0 +1,33 @@
// See the file "COPYING" in the main distribution directory for copyright.
#pragma once
#include "zeek/ConnKey.h"
#include "zeek/conn_key/Factory.h"
namespace zeek::conn_key::fivetuple {
class Factory : public zeek::conn_key::Factory {
public:
static zeek::conn_key::FactoryPtr Instantiate() { return std::make_unique<Factory>(); }
protected:
/**
* Instantiates a clean ConnKey derivative and returns it.
*
* @return A unique pointer to the ConnKey instance.
*/
zeek::ConnKeyPtr DoNewConnKey() const override;
/**
* Instantiates a filled-in ConnKey derivative from a script-layer
* value, usually a conn_id instance. Implementations are free to
* implement this liberally, i.e. the input does not _have_ to be a
* conn_id instance.
*
* @param v The script-layer value providing key input.
* @return A unique pointer to the ConnKey instance, or an error message.
*/
zeek::expected<zeek::ConnKeyPtr, std::string> DoConnKeyFromVal(const zeek::Val& v) const override;
};
} // namespace zeek::conn_key::fivetuple

24
src/packet_analysis/protocol/ip/conn_key/fivetuple/Plugin.cc Normal file
View file

@ -0,0 +1,24 @@
// See the file "COPYING" in the main distribution directory for copyright.
#include "zeek/plugin/Plugin.h"
#include "zeek/conn_key/Component.h"
#include "zeek/packet_analysis/protocol/ip/conn_key/fivetuple/Factory.h"
namespace zeek::plugin::Zeek_Conntuple_Fivetuple {
class Plugin : public zeek::plugin::Plugin {
public:
zeek::plugin::Configuration Configure() override {
AddComponent(new conn_key::Component("Fivetuple", zeek::conn_key::fivetuple::Factory::Instantiate));
zeek::plugin::Configuration config;
config.name = "Zeek::ConnKey_Fivetuple";
config.description = "ConnKey factory for Zeek's default IP/port/proto five-tuples";
return config;
}
};
Plugin plugin;
} // namespace zeek::plugin::Zeek_Conntuple_Fivetuple

3
src/packet_analysis/protocol/ip/conn_key/vlan_fivetuple/CMakeLists.txt Normal file
View file

@ -0,0 +1,3 @@
zeek_add_plugin(
Zeek Conntuple_VLAN
SOURCES Factory.cc Plugin.cc)

129
src/packet_analysis/protocol/ip/conn_key/vlan_fivetuple/Factory.cc Normal file
View file

@ -0,0 +1,129 @@
// See the file "COPYING" in the main distribution directory for copyright.
#include "zeek/packet_analysis/protocol/ip/conn_key/vlan_fivetuple/Factory.h"
#include <memory>
#include "zeek/ID.h"
#include "zeek/Val.h"
#include "zeek/iosource/Packet.h"
#include "zeek/packet_analysis/protocol/ip/conn_key/IPBasedConnKey.h"
#include "zeek/packet_analysis/protocol/ip/conn_key/fivetuple/Factory.h"
#include "zeek/util-types.h"
namespace zeek::conn_key::vlan_fivetuple {
class IPVlanConnKey : public zeek::IPBasedConnKey {
public:
/**
* Constructor.
*
* Fill any holes in the key struct as we use the full tuple as a key.
*/
IPVlanConnKey() { memset(static_cast<void*>(&key), 0, sizeof(key)); }
/**
* @copydoc
*/
detail::PackedConnTuple& PackedTuple() override { return key.tuple; }
/**
* @copydoc
*/
const detail::PackedConnTuple& PackedTuple() const override { return key.tuple; }
protected:
zeek::session::detail::Key DoSessionKey() const override {
return {reinterpret_cast<const void*>(&key), sizeof(key), session::detail::Key::CONNECTION_KEY_TYPE};
}
void DoPopulateConnIdVal(RecordVal& conn_id) override {
if ( conn_id.NumFields() <= 5 )
return;
// Nothing to do if we have no VLAN tags at all.
if ( key.vlan == 0 && key.inner_vlan == 0 )
return;
auto [vlan_offset, inner_vlan_offset] = GetConnIdFieldOffsets();
if ( key.vlan && vlan_offset >= 0 )
conn_id.Assign(vlan_offset, static_cast<int>(key.vlan));
if ( key.inner_vlan && inner_vlan_offset >= 0 )
conn_id.Assign(inner_vlan_offset, static_cast<int>(key.inner_vlan));
};
std::pair<int, int> GetConnIdFieldOffsets() {
static int vlan_offset = -2, inner_vlan_offset = -2;
if ( vlan_offset == -2 && inner_vlan_offset == -2 ) {
vlan_offset = id::conn_id->FieldOffset("vlan");
if ( vlan_offset < 0 || id::conn_id->GetFieldType(vlan_offset)->Tag() != TYPE_INT )
vlan_offset = -1;
inner_vlan_offset = id::conn_id->FieldOffset("inner_vlan");
if ( inner_vlan_offset < 0 || id::conn_id->GetFieldType(inner_vlan_offset)->Tag() != TYPE_INT )
inner_vlan_offset = -1;
}
return {vlan_offset, inner_vlan_offset};
}
protected:
void DoInit(const Packet& pkt) override {
key.vlan = pkt.vlan;
key.inner_vlan = pkt.inner_vlan;
}
private:
friend class Factory;
// Key bytes:
struct {
struct detail::PackedConnTuple tuple;
// Add 802.1Q vlan tags to connection tuples. The tag representation
// here is as in the Packet class (where it's oddly 32-bit), since
// that's where we learn the tag values from. 0 indicates absence.
uint32_t vlan;
uint32_t inner_vlan;
} __attribute__((packed, aligned)) key;
};
zeek::ConnKeyPtr Factory::DoNewConnKey() const { return std::make_unique<IPVlanConnKey>(); }
zeek::expected<zeek::ConnKeyPtr, std::string> Factory::DoConnKeyFromVal(const zeek::Val& v) const {
auto ck = zeek::conn_key::fivetuple::Factory::DoConnKeyFromVal(v);
if ( ! ck.has_value() )
return ck;
auto* k = static_cast<IPVlanConnKey*>(ck.value().get());
auto rt = v.GetType()->AsRecordType();
auto vl = v.AsRecordVal();
int vlan_offset, inner_vlan_offset;
if ( rt == id::conn_id ) {
std::tie(vlan_offset, inner_vlan_offset) = k->GetConnIdFieldOffsets();
}
else {
// We don't know what we've been passed.
vlan_offset = rt->FieldOffset("vlan");
inner_vlan_offset = rt->FieldOffset("inner_vlan");
}
if ( vlan_offset < 0 || inner_vlan_offset < 0 )
return zeek::unexpected<std::string>{"missing vlan or inner_vlan field"};
if ( rt->GetFieldType(vlan_offset)->Tag() != TYPE_INT || rt->GetFieldType(inner_vlan_offset)->Tag() != TYPE_INT )
return zeek::unexpected<std::string>{"vlan or inner_vlan field not of type int"};
if ( vl->HasField(vlan_offset) )
k->key.vlan = vl->GetFieldAs<zeek::IntVal>(vlan_offset);
if ( vl->HasField(inner_vlan_offset) )
k->key.inner_vlan = vl->GetFieldAs<zeek::IntVal>(inner_vlan_offset);
return ck;
}
} // namespace zeek::conn_key::vlan_fivetuple

33
src/packet_analysis/protocol/ip/conn_key/vlan_fivetuple/Factory.h Normal file
View file

@ -0,0 +1,33 @@
// See the file "COPYING" in the main distribution directory for copyright.
#pragma once
#include "zeek/ConnKey.h"
#include "zeek/packet_analysis/protocol/ip/conn_key/fivetuple/Factory.h"
namespace zeek::conn_key::vlan_fivetuple {
class Factory : public zeek::conn_key::fivetuple::Factory {
public:
static zeek::conn_key::FactoryPtr Instantiate() { return std::make_unique<Factory>(); }
private:
/**
* Instantiates a clean ConnKey derivative and returns it.
*
* @return A unique pointer to the ConnKey instance.
*/
zeek::ConnKeyPtr DoNewConnKey() const override;
/**
* Instantiates a filled-in ConnKey derivative from a script-layer
* record, usually a conn_id instance. Implementations are free to
* implement this liberally, i.e. the input does not _have_ to be a
* conn_id.
*
* @param v The script-layer value providing key input.
* @return A unique pointer to the ConnKey instance, or an error message.
*/
zeek::expected<zeek::ConnKeyPtr, std::string> DoConnKeyFromVal(const zeek::Val& v) const override;
};
} // namespace zeek::conn_key::vlan_fivetuple

24
src/packet_analysis/protocol/ip/conn_key/vlan_fivetuple/Plugin.cc Normal file
View file

@ -0,0 +1,24 @@
// See the file "COPYING" in the main distribution directory for copyright.
#include "zeek/plugin/Plugin.h"
#include "zeek/conn_key/Component.h"
#include "zeek/packet_analysis/protocol/ip/conn_key/vlan_fivetuple/Factory.h"
namespace zeek::plugin::Zeek_ConnKey_VLAN {
class Plugin : public zeek::plugin::Plugin {
public:
zeek::plugin::Configuration Configure() override {
AddComponent(new conn_key::Component("VLAN_FIVETUPLE", zeek::conn_key::vlan_fivetuple::Factory::Instantiate));
zeek::plugin::Configuration config;
config.name = "Zeek::ConnKey_Vlan_Fivetuple";
config.description = "ConnKey factory for 802.1Q VLAN/Q-in-Q + IP/port/proto five-tuples";
return config;
}
};
Plugin plugin;
} // namespace zeek::plugin::Zeek_ConnKey_VLAN

14
src/packet_analysis/protocol/tcp/TCP.cc
View file

@ -26,21 +26,13 @@ SessionAdapter* TCPAnalyzer::MakeSessionAdapter(Connection* conn) {
zeek::analyzer::pia::PIA* TCPAnalyzer::MakePIA(Connection* conn) { return new analyzer::pia::PIA_TCP(conn); }
bool TCPAnalyzer::BuildConnTuple(size_t len, const uint8_t* data, Packet* packet, ConnTuple& tuple) {
bool TCPAnalyzer::InitConnKey(size_t len, const uint8_t* data, Packet* packet, IPBasedConnKey& key) {
uint32_t min_hdr_len = sizeof(struct tcphdr);
if ( ! CheckHeaderTrunc(min_hdr_len, len, packet) )
return false;
tuple.src_addr = packet->ip_hdr->SrcAddr();
tuple.dst_addr = packet->ip_hdr->DstAddr();
data = packet->ip_hdr->Payload();
const struct tcphdr* tp = (const struct tcphdr*)data;
tuple.src_port = tp->th_sport;
tuple.dst_port = tp->th_dport;
tuple.is_one_way = false;
tuple.proto = packet->proto;
const struct tcphdr* tp = (const struct tcphdr*)packet->ip_hdr->Payload();
key.InitTuple(packet->ip_hdr->SrcAddr(), tp->th_sport, packet->ip_hdr->DstAddr(), tp->th_dport, packet->proto);
return true;
}

5
src/packet_analysis/protocol/tcp/TCP.h
View file

@ -35,10 +35,7 @@ public:
}
protected:
/**
* Parse the header from the packet into a ConnTuple object.
*/
bool BuildConnTuple(size_t len, const uint8_t* data, Packet* packet, ConnTuple& tuple) override;
bool InitConnKey(size_t len, const uint8_t* data, Packet* packet, IPBasedConnKey& key) override;
void DeliverPacket(Connection* c, double t, bool is_orig, int remaining, Packet* pkt) override;

10
src/packet_analysis/protocol/teredo/Teredo.cc
View file

@ -185,15 +185,19 @@ bool TeredoAnalyzer::AnalyzePacket(size_t len, const uint8_t* data, Packet* pack
return false;
}
zeek::detail::ConnKey conn_key = conn->Key();
OrigRespMap::iterator or_it = orig_resp_map.find(conn_key);
const auto& k = conn->Key();
auto sk = k.SessionKey();
OrigRespMap::iterator or_it = orig_resp_map.find(sk);
// The first time a teredo packet is parsed successfully, insert
// state into orig_resp_map so we can confirm when both sides
// see valid Teredo packets. Further, raise an event so that script
// layer can install a connection removal hooks to cleanup later.
if ( or_it == orig_resp_map.end() ) {
or_it = orig_resp_map.insert(or_it, {conn_key, {}});
sk.CopyData(); // Copy key data to store in map.
auto [it, inserted] = orig_resp_map.emplace(std::move(sk), OrigResp{});
assert(inserted);
or_it = it;
packet->session->EnqueueEvent(new_teredo_state, nullptr, packet->session->GetVal());
}

5
src/packet_analysis/protocol/teredo/Teredo.h
View file

@ -8,6 +8,7 @@
#include "zeek/RE.h"
#include "zeek/Reporter.h"
#include "zeek/packet_analysis/Analyzer.h"
#include "zeek/session/Key.h"
namespace zeek::packet_analysis::teredo {
@ -44,7 +45,7 @@ public:
bool DetectProtocol(size_t len, const uint8_t* data, Packet* packet) override;
void RemoveConnection(const zeek::detail::ConnKey& conn_key) { orig_resp_map.erase(conn_key); }
void RemoveConnection(const zeek::session::detail::Key& conn_key) { orig_resp_map.erase(conn_key); }
protected:
struct OrigResp {
@ -52,7 +53,7 @@ protected:
bool valid_resp = false;
bool confirmed = false;
};
using OrigRespMap = std::map<zeek::detail::ConnKey, OrigResp>;
using OrigRespMap = std::map<zeek::session::detail::Key, OrigResp>;
OrigRespMap orig_resp_map;
std::unique_ptr<zeek::detail::Specific_RE_Matcher> pattern_re;

9
src/packet_analysis/protocol/teredo/functions.bif
View file

@ -2,6 +2,7 @@ module PacketAnalyzer::TEREDO;
%%{
#include "zeek/Conn.h"
#include "zeek/conn_key/Manager.h"
#include "zeek/session/Manager.h"
#include "zeek/packet_analysis/Manager.h"
#include "zeek/packet_analysis/protocol/teredo/Teredo.h"
@ -12,8 +13,12 @@ function remove_teredo_connection%(cid: conn_id%) : bool
zeek::packet_analysis::AnalyzerPtr teredo = zeek::packet_mgr->GetAnalyzer("Teredo");
if ( teredo )
{
zeek::detail::ConnKey conn_key(cid);
static_cast<zeek::packet_analysis::teredo::TeredoAnalyzer*>(teredo.get())->RemoveConnection(conn_key);
auto r = zeek::conn_key_mgr->GetFactory().ConnKeyFromVal(*cid);
if ( ! r.has_value() )
return zeek::val_mgr->False();
auto sk = r.value()->SessionKey();
static_cast<zeek::packet_analysis::teredo::TeredoAnalyzer*>(teredo.get())->RemoveConnection(sk);
}
return zeek::val_mgr->True();

10
src/packet_analysis/protocol/udp/UDP.cc
View file

@ -53,19 +53,13 @@ bool UDPAnalyzer::WantConnection(uint16_t src_port, uint16_t dst_port, const u_c
return true;
}
bool UDPAnalyzer::BuildConnTuple(size_t len, const uint8_t* data, Packet* packet, ConnTuple& tuple) {
bool UDPAnalyzer::InitConnKey(size_t len, const uint8_t* data, Packet* packet, IPBasedConnKey& key) {
uint32_t min_hdr_len = sizeof(struct udphdr);
if ( ! CheckHeaderTrunc(min_hdr_len, len, packet) )
return false;
tuple.src_addr = packet->ip_hdr->SrcAddr();
tuple.dst_addr = packet->ip_hdr->DstAddr();
const struct udphdr* up = (const struct udphdr*)packet->ip_hdr->Payload();
tuple.src_port = up->uh_sport;
tuple.dst_port = up->uh_dport;
tuple.is_one_way = false;
tuple.proto = packet->proto;
key.InitTuple(packet->ip_hdr->SrcAddr(), up->uh_sport, packet->ip_hdr->DstAddr(), up->uh_dport, packet->proto);
return true;
}

5
src/packet_analysis/protocol/udp/UDP.h
View file

@ -24,10 +24,7 @@ public:
void Initialize() override;
protected:
/**
* Parse the header from the packet into a ConnTuple object.
*/
bool BuildConnTuple(size_t len, const uint8_t* data, Packet* packet, ConnTuple& tuple) override;
bool InitConnKey(size_t len, const uint8_t* data, Packet* packet, IPBasedConnKey& key) override;
void DeliverPacket(Connection* c, double t, bool is_orig, int remaining, Packet* pkt) override;

6
src/packet_analysis/protocol/unknown_ip_transport/UnknownIPTransport.cc
View file

@ -33,10 +33,8 @@ SessionAdapter* UnknownIPTransportAnalyzer::MakeSessionAdapter(Connection* conn)
return root;
}
bool UnknownIPTransportAnalyzer::BuildConnTuple(size_t len, const uint8_t* data, Packet* packet, ConnTuple& tuple) {
tuple.src_addr = packet->ip_hdr->SrcAddr();
tuple.dst_addr = packet->ip_hdr->DstAddr();
tuple.proto = packet->proto;
bool UnknownIPTransportAnalyzer::InitConnKey(size_t len, const uint8_t* data, Packet* packet, IPBasedConnKey& key) {
key.InitTuple(packet->ip_hdr->SrcAddr(), 0, packet->ip_hdr->DstAddr(), 0, packet->proto);
return true;
}

5
src/packet_analysis/protocol/unknown_ip_transport/UnknownIPTransport.h
View file

@ -23,10 +23,7 @@ public:
packet_analysis::IP::SessionAdapter* MakeSessionAdapter(Connection* conn) override;
protected:
/**
* Parse the header from the packet into a ConnTuple object.
*/
bool BuildConnTuple(size_t len, const uint8_t* data, Packet* packet, ConnTuple& tuple) override;
bool InitConnKey(size_t len, const uint8_t* data, Packet* packet, IPBasedConnKey& key) override;
void DeliverPacket(Connection* c, double t, bool is_orig, int remaining, Packet* pkt) override;
};

2
src/plugin/Component.cc
View file

@ -52,6 +52,8 @@ void Component::Describe(ODesc* d) const {
case component::STORAGE_SERIALIZER: d->Add("Storage Serializer"); break;
case component::CONNKEY: d->Add("ConnKey Factory"); break;
default:
reporter->InternalWarning("unknown component type in plugin::Component::Describe");
d->Add("<unknown component type>");

1
src/plugin/Component.h
View file

@ -38,6 +38,7 @@ enum Type : uint8_t {
LOG_SERIALIZER, /// A serializer for log batches, used by cluster backends.
STORAGE_BACKEND, /// A backend for the storage framework.
STORAGE_SERIALIZER, /// A serializer for the storage framework.
CONNKEY, /// A factory for connection keys.
};
} // namespace component

21
src/session/Manager.cc
View file

@ -17,6 +17,7 @@
#include "zeek/Stats.h"
#include "zeek/Timer.h"
#include "zeek/TunnelEncapsulation.h"
#include "zeek/conn_key/Manager.h"
#include "zeek/packet_analysis/Manager.h"
#include "zeek/session/Session.h"
#include "zeek/telemetry/Manager.h"
@ -88,23 +89,23 @@ Manager::~Manager() {
}
Connection* Manager::FindConnection(Val* v) {
zeek::detail::ConnKey conn_key(v);
// XXX: This could in the future dispatch to different factories for
// different kinds of Vals. ``v`` will usually be a conn_id instance, which
// is IP-specific. If ``v`` is something else, maybe we'd like to use a
// different builder.
auto r = conn_key_mgr->GetFactory().ConnKeyFromVal(*v);
if ( ! conn_key.Valid() ) {
if ( ! r.has_value() ) {
// Produce a loud error for invalid script-layer conn_id records.
const char* extra = "";
if ( conn_key.transport == UNKNOWN_IP_PROTO )
extra = ": the proto field has the \"unknown\" 65535 value. Did you forget to set it?";
zeek::emit_builtin_error(zeek::util::fmt("invalid connection ID record encountered%s", extra));
zeek::emit_builtin_error(r.error().c_str());
return nullptr;
}
return FindConnection(conn_key);
return FindConnection(*r.value());
}
Connection* Manager::FindConnection(const zeek::detail::ConnKey& conn_key) {
detail::Key key(&conn_key, sizeof(conn_key), detail::Key::CONNECTION_KEY_TYPE, false);
Connection* Manager::FindConnection(const zeek::ConnKey& conn_key) {
auto key = conn_key.SessionKey();
auto it = session_map.find(key);
if ( it != session_map.end() )

3
src/session/Manager.h
View file

@ -5,6 +5,7 @@
#include <sys/types.h> // for u_char
#include <unordered_map>
#include "zeek/ConnKey.h"
#include "zeek/Frag.h"
#include "zeek/session/Session.h"
@ -70,7 +71,7 @@ public:
* @param conn_key The key for the connection to search for.
* @return The connection, or nullptr if one doesn't exist.
*/
Connection* FindConnection(const zeek::detail::ConnKey& conn_key);
Connection* FindConnection(const zeek::ConnKey& conn_key);
void Remove(Session* s);
void Insert(Session* c, bool remove_existing = true);

5
src/zeek-setup.cc
View file

@ -54,6 +54,7 @@
#include "zeek/broker/Manager.h"
#include "zeek/cluster/Backend.h"
#include "zeek/cluster/Manager.h"
#include "zeek/conn_key/Manager.h"
#include "zeek/file_analysis/Manager.h"
#include "zeek/input.h"
#include "zeek/input/Manager.h"
@ -161,6 +162,7 @@ void do_ssl_deinit() {
zeek::ValManager* zeek::val_mgr = nullptr;
zeek::packet_analysis::Manager* zeek::packet_mgr = nullptr;
zeek::conn_key::Manager* zeek::conn_key_mgr = nullptr;
zeek::analyzer::Manager* zeek::analyzer_mgr = nullptr;
zeek::plugin::Manager* zeek::plugin_mgr = nullptr;
@ -408,6 +410,7 @@ static void terminate_zeek() {
delete zeekygen_mgr;
delete packet_mgr;
delete conn_key_mgr;
delete analyzer_mgr;
delete file_mgr;
delete cluster::manager;
@ -690,6 +693,7 @@ SetupResult setup(int argc, char** argv, Options* zopts) {
iosource_mgr = new iosource::Manager();
event_registry = new EventRegistry();
packet_mgr = new packet_analysis::Manager();
conn_key_mgr = new conn_key::Manager();
analyzer_mgr = new analyzer::Manager();
log_mgr = new logging::Manager();
input_mgr = new input::Manager();
@ -838,6 +842,7 @@ SetupResult setup(int argc, char** argv, Options* zopts) {
RecordType::InitPostScript();
conn_key_mgr->InitPostScript();
telemetry_mgr->InitPostScript();
thread_mgr->InitPostScript();
iosource_mgr->InitPostScript();

4
src/zeekygen/ScriptInfo.cc
View file

@ -309,6 +309,10 @@ void ScriptInfo::DoInitPostScript() {
const auto& id = zeek::detail::global_scope()->Find("Input::Reader");
types.push_back(new IdentifierInfo(id, this));
}
else if ( name == "base/init-bare.zeek" ) {
const auto& id = zeek::detail::global_scope()->Find("ConnKey::Tag");
types.push_back(new IdentifierInfo(id, this));
}
else if ( name == "base/frameworks/logging/main.zeek" ) {
const auto& id = zeek::detail::global_scope()->Find("Log::Writer");
types.push_back(new IdentifierInfo(id, this));

7
testing/btest/Baseline/plugins.connkey/conn.log.cut Normal file
View file

@ -0,0 +1,7 @@
### BTest baseline data generated by btest-diff. Do not edit. Use "btest -U/-u" to update. Requires BTest >= 0.63.
ts uid id.orig_h id.orig_p id.resp_h id.resp_p id.inits proto service orig_pkts resp_pkts
XXXXXXXXXX.XXXXXX ClEkJM2Vm5giqnMf4h 141.142.220.235 37604 199.233.217.249 56666 1 tcp ftp-data 4 4
XXXXXXXXXX.XXXXXX C4J4Th3PJpwUYZZ6gc 141.142.220.235 59378 199.233.217.249 56667 22 tcp ftp-data 4 4
XXXXXXXXXX.XXXXXX CtPZjS20MLrsMUOJi2 199.233.217.249 61920 141.142.220.235 33582 40 tcp ftp-data 5 3
XXXXXXXXXX.XXXXXX CUM0KZ3MLUfNB0cl11 199.233.217.249 61918 141.142.220.235 37835 60 tcp ftp-data 5 3
XXXXXXXXXX.XXXXXX CHhAvVGS1DHFjwGM9 141.142.220.235 50003 199.233.217.249 21 0 tcp ftp 38 25

15
testing/btest/Baseline/plugins.connkey/output Normal file
View file

@ -0,0 +1,15 @@
### BTest baseline data generated by btest-diff. Do not edit. Use "btest -U/-u" to update. Requires BTest >= 0.63.
Demo::Foo - A Foo ConnKey factory (dynamic, version 1.0.0)
[ConnKey Factory] Foo (CONNKEY_FOO, enabled)
===
DoNewConnKey (0 key all_inits)
DoNewConnKey (1 key all_inits)
DoConnKeyFromVal for [orig_h=141.142.220.235, orig_p=50003/tcp, resp_h=199.233.217.249, resp_p=21/tcp, proto=6, inits=0]
DoNewConnKey (2 key all_inits)
DoConnKeyFromVal for [orig_h=141.142.220.235, orig_p=50003/tcp, resp_h=199.233.217.249, resp_p=21/tcp, proto=6, inits=0]
DoNewConnKey (6 key all_inits)
DoNewConnKey (22 key all_inits)
DoNewConnKey (40 key all_inits)
DoNewConnKey (60 key all_inits)
DoNewConnKey (78 key all_inits)

5
testing/btest/Baseline/scripts.policy.frameworks.conn_key.vlan_fivetuple-2/conn.log.cut Normal file
View file

@ -0,0 +1,5 @@
### BTest baseline data generated by btest-diff. Do not edit. Use "btest -U/-u" to update. Requires BTest >= 0.63.
ts uid id.orig_h id.orig_p id.resp_h id.resp_p id.vlan id.inner_vlan orig_pkts resp_pkts service
XXXXXXXXXX.XXXXXX CHhAvVGS1DHFjwGM9 141.142.228.5 59856 192.150.187.43 80 - - 7 7 http
XXXXXXXXXX.XXXXXX C4J4Th3PJpwUYZZ6gc 141.142.228.5 59856 192.150.187.43 80 10 20 7 7 http
XXXXXXXXXX.XXXXXX ClEkJM2Vm5giqnMf4h 141.142.228.5 59856 192.150.187.43 80 42 - 7 7 http

5
testing/btest/Baseline/scripts.policy.frameworks.conn_key.vlan_fivetuple-3/conn.log.cut Normal file
View file

@ -0,0 +1,5 @@
### BTest baseline data generated by btest-diff. Do not edit. Use "btest -U/-u" to update. Requires BTest >= 0.63.
ts uid id.orig_h id.orig_p id.resp_h id.resp_p orig_pkts resp_pkts service
XXXXXXXXXX.XXXXXX CHhAvVGS1DHFjwGM9 141.142.228.5 59856 192.150.187.43 80 7 7 http
XXXXXXXXXX.XXXXXX C4J4Th3PJpwUYZZ6gc 141.142.228.5 59856 192.150.187.43 80 7 7 http
XXXXXXXXXX.XXXXXX ClEkJM2Vm5giqnMf4h 141.142.228.5 59856 192.150.187.43 80 7 7 http

5
testing/btest/Baseline/scripts.policy.frameworks.conn_key.vlan_fivetuple-4/conn.log.cut Normal file
View file

@ -0,0 +1,5 @@
### BTest baseline data generated by btest-diff. Do not edit. Use "btest -U/-u" to update. Requires BTest >= 0.63.
ts uid id.orig_h id.orig_p id.resp_h id.resp_p id.vlan id.inner_vlan orig_pkts resp_pkts service
XXXXXXXXXX.XXXXXX CHhAvVGS1DHFjwGM9 141.142.228.5 59856 192.150.187.43 80 - - 7 7 http
XXXXXXXXXX.XXXXXX C4J4Th3PJpwUYZZ6gc 141.142.228.5 59856 192.150.187.43 80 10 20 7 7 http
XXXXXXXXXX.XXXXXX ClEkJM2Vm5giqnMf4h 141.142.228.5 59856 192.150.187.43 80 42 - 7 7 http

5
testing/btest/Baseline/scripts.policy.frameworks.conn_key.vlan_fivetuple-5/conn.log.cut Normal file
View file

@ -0,0 +1,5 @@
### BTest baseline data generated by btest-diff. Do not edit. Use "btest -U/-u" to update. Requires BTest >= 0.63.
ts uid id.orig_h id.orig_p id.resp_h id.resp_p id.vlan id.inner_vlan orig_pkts resp_pkts service
XXXXXXXXXX.XXXXXX CHhAvVGS1DHFjwGM9 141.142.228.5 59856 192.150.187.43 80 - - 7 7 http
XXXXXXXXXX.XXXXXX C4J4Th3PJpwUYZZ6gc 141.142.228.5 59856 192.150.187.43 80 10 20 7 7 http
XXXXXXXXXX.XXXXXX ClEkJM2Vm5giqnMf4h 141.142.228.5 59856 192.150.187.43 80 42 - 7 7 http

5
testing/btest/Baseline/scripts.policy.frameworks.conn_key.vlan_fivetuple-6/conn.log.cut Normal file
View file

@ -0,0 +1,5 @@
### BTest baseline data generated by btest-diff. Do not edit. Use "btest -U/-u" to update. Requires BTest >= 0.63.
ts uid id.orig_h id.orig_p id.resp_h id.resp_p id.vlan id.inner_vlan orig_pkts resp_pkts service
XXXXXXXXXX.XXXXXX CHhAvVGS1DHFjwGM9 141.142.228.5 59856 192.150.187.43 80 - - 7 7 http
XXXXXXXXXX.XXXXXX C4J4Th3PJpwUYZZ6gc 141.142.228.5 59856 192.150.187.43 80 - - 7 7 http
XXXXXXXXXX.XXXXXX ClEkJM2Vm5giqnMf4h 141.142.228.5 59856 192.150.187.43 80 - - 7 7 http

3
testing/btest/Baseline/scripts.policy.frameworks.conn_key.vlan_fivetuple/conn.log.cut Normal file
View file

@ -0,0 +1,3 @@
### BTest baseline data generated by btest-diff. Do not edit. Use "btest -U/-u" to update. Requires BTest >= 0.63.
ts uid id.orig_h id.orig_p id.resp_h id.resp_p orig_pkts resp_pkts service
XXXXXXXXXX.XXXXXX CHhAvVGS1DHFjwGM9 141.142.228.5 59856 192.150.187.43 80 21 21 http

BIN
testing/btest/Traces/vlan-collisions.pcap Normal file
View file

Binary file not shown.

0
testing/btest/plugins/connkey-plugin/.btest-ignore Normal file
View file

15
testing/btest/plugins/connkey-plugin/CMakeLists.txt Normal file
View file

@ -0,0 +1,15 @@
project(Zeek-Plugin-Demo-Foo)
cmake_minimum_required(VERSION 3.15)
if (NOT ZEEK_DIST)
message(FATAL_ERROR "ZEEK_DIST not set")
endif ()
set(CMAKE_MODULE_PATH ${ZEEK_DIST}/cmake)
include(ZeekPlugin)
zeek_add_plugin(
Demo Foo
SOURCES src/Plugin.cc src/Foo.cc)

45
testing/btest/plugins/connkey-plugin/src/Foo.cc Normal file
View file

@ -0,0 +1,45 @@
#include "Foo.h"
#include <cstdio>
#include <memory>
#include "zeek/Desc.h"
#include "zeek/Val.h"
#include "zeek/iosource/Packet.h"
#include "zeek/packet_analysis/protocol/ip/conn_key/IPBasedConnKey.h"
#include "zeek/session/Key.h"
using namespace btest::plugin::Demo_Foo;
namespace {
// Just track how often DoInit() was called for baselining.
int all_inits = 0;
class MyConnKey : public zeek::IPConnKey {
public:
MyConnKey(int inits) : zeek::IPConnKey(), inits(inits) {}
void DoInit(const zeek::Packet& pkt) override { ++all_inits; }
void DoPopulateConnIdVal(zeek::RecordVal& rv) override {
static int offset = rv.GetType<zeek::RecordType>()->FieldOffset("inits");
rv.Assign(offset, zeek::make_intrusive<zeek::IntVal>(inits));
}
private:
int inits;
};
} // namespace
zeek::ConnKeyPtr FooFactory::DoNewConnKey() const {
std::printf("DoNewConnKey (%d key all_inits)\n", all_inits);
return std::make_unique<MyConnKey>(all_inits);
}
zeek::expected<zeek::ConnKeyPtr, std::string> FooFactory::DoConnKeyFromVal(const zeek::Val& v) const {
std::printf("DoConnKeyFromVal for %s\n", zeek::obj_desc_short(&v).c_str());
return zeek::conn_key::fivetuple::Factory::DoConnKeyFromVal(v);
}
zeek::conn_key::FactoryPtr FooFactory::Instantiate() { return std::make_unique<FooFactory>(); }

25
testing/btest/plugins/connkey-plugin/src/Foo.h Normal file
View file

@ -0,0 +1,25 @@
#pragma once
#include "zeek/IntrusivePtr.h"
#include "zeek/conn_key/Factory.h"
#include "zeek/packet_analysis/protocol/ip/conn_key/fivetuple/Factory.h"
namespace zeek {
class Val;
using ValPtr = zeek::IntrusivePtr<Val>;
} // namespace zeek
namespace btest::plugin::Demo_Foo {
class FooFactory : public zeek::conn_key::fivetuple::Factory {
public:
static zeek::conn_key::FactoryPtr Instantiate();
protected:
zeek::ConnKeyPtr DoNewConnKey() const override;
zeek::expected<zeek::ConnKeyPtr, std::string> DoConnKeyFromVal(const zeek::Val& v) const override;
private:
};
} // namespace btest::plugin::Demo_Foo

24
testing/btest/plugins/connkey-plugin/src/Plugin.cc Normal file
View file

@ -0,0 +1,24 @@
#include "Plugin.h"
#include "zeek/conn_key/Component.h"
#include "Foo.h"
namespace btest::plugin::Demo_Foo {
Plugin plugin;
}
using namespace btest::plugin::Demo_Foo;
zeek::plugin::Configuration Plugin::Configure() {
AddComponent(new zeek::conn_key::Component("Foo", btest::plugin::Demo_Foo::FooFactory::Instantiate));
zeek::plugin::Configuration config;
config.name = "Demo::Foo";
config.description = "A Foo ConnKey factory";
config.version.major = 1;
config.version.minor = 0;
config.version.patch = 0;
return config;
}

15
testing/btest/plugins/connkey-plugin/src/Plugin.h Normal file
View file

@ -0,0 +1,15 @@
#pragma once
#include "zeek/plugin/Plugin.h"
namespace btest::plugin::Demo_Foo {
class Plugin : public zeek::plugin::Plugin {
protected:
zeek::plugin::Configuration Configure() override;
};
extern Plugin plugin;
} // namespace btest::plugin::Demo_Foo

16
testing/btest/plugins/connkey.zeek Normal file
View file

@ -0,0 +1,16 @@
# @TEST-EXEC: ${DIST}/auxil/zeek-aux/plugin-support/init-plugin -u . Demo Foo
# @TEST-EXEC: cp -r %DIR/connkey-plugin/* .
# @TEST-EXEC: ./configure --zeek-dist=${DIST} && make
# @TEST-EXEC: ZEEK_PLUGIN_PATH=`pwd` zeek -NN Demo::Foo >>output
# @TEST-EXEC: echo === >>output
# @TEST-EXEC: ZEEK_PLUGIN_PATH=`pwd` zeek -r $TRACES/ftp/ipv4.trace %INPUT >>output
# @TEST-EXEC: zeek-cut -m ts uid id.orig_h id.orig_p id.resp_h id.resp_p id.inits proto service orig_pkts resp_pkts < conn.log > conn.log.cut
# @TEST-EXEC: btest-diff conn.log.cut
# @TEST-EXEC: btest-diff output
redef ConnKey::factory = ConnKey::CONNKEY_FOO;
redef record conn_id += {
inits: int &log &default=-1; # Number of inits happened until the key was created. Not part of the hash, just metadata.
};

56
testing/btest/scripts/policy/frameworks/conn_key/vlan_fivetuple.zeek Normal file
View file

@ -0,0 +1,56 @@
# @TEST-DOC: Verify VLAN-aware flow tuples on colliding traffic.
#
# The test pcap has 3 overlapping healthy TCP connections, each with different VLAN tagging: none, one VLAN tag, two VLAN tags.
# To create: tcprewrite --enet-vlan=add --enet-vlan-tag 20 --enet-vlan-cfi=1 --enet-vlan-pri=2 -i in.pcap -o out.pcap
#
# @TEST-EXEC: zeek -r $TRACES/vlan-collisions.pcap %INPUT
# @TEST-EXEC: zeek-cut -m ts uid id.orig_h id.orig_p id.resp_h id.resp_p id.vlan id.inner_vlan orig_pkts resp_pkts service <conn.log >conn.log.cut
# @TEST-EXEC: btest-diff conn.log.cut
# Default operation: Zeek isn't VLAN-aware, a single conn.log entry results.
# @TEST-START-NEXT
# Switch to VLAN-aware flow tuples: multiple conn.log entries with full
# information.
@load frameworks/conn_key/vlan_fivetuple
# @TEST-START-NEXT
# Leave out the conn_id redef: Zeek still distinguishes flows so multiple
# conn.log entries result, but conn.log doesn't show the VLAN fields.
redef ConnKey::factory = ConnKey::CONNKEY_VLAN_FIVETUPLE;
# @TEST-START-NEXT
# Add an extra field before the VLAN ones, to throw off any fixed-offset code.
redef record conn_id += {
foo: int &default=1;
};
@load frameworks/conn_key/vlan_fivetuple
# @TEST-START-NEXT
# Add the right fields, but in the wrong order. (zeek-cut obscures the difference.)
redef record conn_id += {
inner_vlan: int &log &optional;
vlan: int &log &optional;
};
redef ConnKey::factory = ConnKey::CONNKEY_VLAN_FIVETUPLE;
# @TEST-START-NEXT
# Add the right fields, but with the wrong types.
redef record conn_id += {
vlan: string &log &optional;
inner_vlan: string &log &optional;
};
redef ConnKey::factory = ConnKey::CONNKEY_VLAN_FIVETUPLE;