When a clone attaches to a master, it just gets the diffs sent as
events. Which is neat because it means that we pretty much don't need
any extra code to handle this.
This currently only handles the most basic case, and is not thoroughly
tested.
When initializing a master store, we now check if there already is data
in it. If yes, we load it directly into the zeek table when the store is
created. We assume that this is happening at Zeek startup - and are
supremely evil and just load it synchronously. Which could block
execution for a bit for larger stores.
That being said - this might sidestep other issues that would arise when
doing this async (like scripts already inserting data).
Next step: check if this approach also works for clones.
* origin/master:
Fix shadowed variable that breaks lookup_hostname()
GH-1025: allow copying/cloning of `opaque of Broker::Store`
Fix "possibly-truncated" compiler warning in BuildJSON snprintf()
Update submodule(s)
Fixed some places where tabs became spaces
Convert to using permissions to check for access to cirrus variables in benchmark script
Integrate review feedback: improve command-line option redef parsing
Fix several issues with command-line option redefs
Remove last_access_time from TableEntryVal.
Minimize data published for expected FTP data channel analysis
Stricter checking if we have a dns field on the connection being processed
Modified the DNS protocol analyzer to add a new parameter to the dns_request event which includes the DNS query in its original case. Added a policy script that will add the original_case to the dns.log file as well. Created new btests to test both.
Place build file in explicit location for benchmarking to work correctly
cmake: Make musl support more distro agnostic
Update highwayhash submodule to upstream.
GH-998: Fix Reporter::conn_weird() to handle expired connections
Changes during merge
- Changed the policy script to use an event handler that behaves
for like the base script: &priority=5, msg$opcode != early-out,
no record field existence checks
- Also extended dns_query_reply event with original_query param
- Removed ExtractName overload, and just use default param
* 'dns-original-query-case' of https://github.com/rvictory/zeek:
Fixed some places where tabs became spaces
Stricter checking if we have a dns field on the connection being processed
Modified the DNS protocol analyzer to add a new parameter to the dns_request event which includes the DNS query in its original case. Added a policy script that will add the original_case to the dns.log file as well. Created new btests to test both.
Makes some attributes conflict with each other. This also needed the
test to change.
The test is a bit flaky - but I can, for the heck of it, not figure out
why. I am punting that for the future after spending a few hours on it.
* Some methods mistakenly returned a bool instead of QueryResult
when passed an invalid `opaque of Broker::Store` handle.
* Now generates a runtime exception for store_name() and is_closed()
calls that pass an invalid `opaque of Broker::Store` handle as any
returned value can't be reasonably used in any subsequent logic.
* Descriptions of any invalid arguments are now given in the error
message.
* Variables of `string` type can now be set to an empty string
* Trying to set a variable with non-`string` type to an empty value
now emits an error instead of silently doing nothing
* Providing an invalid identifier now emits an "unknown identifier"
error instead of silently doing nothing
* origin/master: (33 commits)
Fix location where CI places build.tgz
Update submodule(s)
Disable some deprecation diagnostics for GCC
Compare pcap_next_ex() result to PCAP_ERROR/PCAP_ERROR_BREAK
Optimize Connection::RemovalEvent() for bare-mode usage
Rename BroType to Type
Update NEWS
Review cleanup
Move Type types to zeek namespace
Review cleanup
Restrict Cirrus CI to only zeek repo's branches
GH-977: Improve pcap error handling
Remove not-useful code in iosource::Manager::OpenPktSrc
GH-999: Stop formatting DHCP Client ID Hardware Type 0 as MAC
Remove inline from some static KeyedHash members
Improve Func.h inclusion
Fix NVT analyzer memory leak from multiple telnet authn name options
Rename aux/ to auxil/
Move Flare/Pipe from the bro namespace to zeek::detail
Move Attr to the zeek::detail namespace
...
This introduces a new sampling state-map for expired connections to fix
segfaults that previously occured when passing in a `connection` record
to `Reporter::conn_weird()` for which the internal `Connection` object
had already been expired and deleted. This also introduces a new event
called `expired_conn_weird`, which is similar to `conn_weird`, except
the full `connection` record is no longer available, just the `conn_id`
and UID string.
Merge adjustments:
- Preserved original `base_type_no_ref` argument type as ::TypeTag
- Removed superfluous #pragma guard around deprecated TableVal ctor
- Clarify NEWS regarding MetaHook{Pre,Post} deprecations
- Simplify some `::zeek::` qualifications to just `zeek::`
- Prefixed FORWARD_DECLARE_NAMESPACED macro with ZEEK_
* origin/topic/timw/266-namespaces:
Disable some deprecation diagnostics for GCC
Rename BroType to Type
Update NEWS
Review cleanup
Move Type types to zeek namespace
Move Flare/Pipe from the bro namespace to zeek::detail
Move Attr to the zeek::detail namespace
Move Trigger into the zeek::detail namespace
Move ID to the zeek::detail namespace
Move Anon.h into zeek::detail namespace
Mark all of the aliased classes in plugin/Plugin.h deprecated, and fix all of the plugins that were using them
Move all of the base plugin classes into the zeek::plugin namespace
Expr: move all classes into zeek::detail
Stmt: move Stmt classes into zeek::detail namespace
Add utility macro for creating namespaced aliases for classes
Switches from pcap_next() to pcap_next_ex() to better handle all error
conditions. This allows, for example, to have a non-zero exit code for
a Zeek process that fails to fully process all packets in a pcap file.
For `DHCP::ClientID$hwtype` fields equal to 0, the `hwaddr` field is
no longer misformatted as a MAC and instead just contains the raw bytes
seen in the DHCP Client ID Option.
This commit fixes a few more loose ends to actually make the
Zeek Table<->brokerstore syncing work. This mostly slightly changes the
TableVal assign/remove operators to prevent loops when a remote change
arrives.
The tests inserts a value into a table on the manager, and it pops out
in a table on a clone - which is the easiest case.
Timeouts are still not handled at all; the behavior when inserting into
a clone is untested.
* origin/master:
Fix use-after-move of proc_status_file breaking -U flag
Change Cirrus CI config to use org-level secured variable
Deprecate Val(double, TypeTag) ctor, add TimeVal/DoubleVal subclasses
GH-973: Fix the return type of topk_get_top()
* origin/master: (172 commits)
GH-985: Fix descriptions of double_to_interval() return values
Decrease number of CPUs/memory for Cirrus CI tasks
Change CI script to compile from build/Makefile
Add missing include in util.cc
Remove Analyzer.h from bro-bif.h
Remove IPAddr.h from Reporter.h
Remove the inclusion of Func.h from NetVar.h, which reduces the inclusion of Func.h overall.
Update submodule(s)
Update submodule(s)
Integrate review feedback
Update submodule(s)
Fix crash on using some deprecated environment variables
Update NEWS
Update test baselines for new Broker connection status/error strings
Switch Broker Val converter visitor to return IntrusivePtr
Change BroFunc ctor to take const-ref IntrusivePtr<ID>
Add version of Frame::SetElement() taking IntrusivePtr<ID>
Change Scope/Func inits from id_list* to vector<IntrusivePtr<ID>>
Change Scope::GenerateTemporary() to return IntrusivePtr
Deprecate Scope::ReturnType(), replace with GetReturnType()
...
This also updates all usages of the deprecated Val ctor to use
either IntervalVal, TimeVal, or DoubleVal ctors. The reason for
doing away with the old constructor is that using it with TYPE_INTERVAL
isn't strictly correct since there exists a more specific subclass,
IntervalVal, with overriden ValDescribe() method that ought to be used
to print such values in a more descriptive way.
The BIF was not returning an IntervalVal which has an overriden
ValDescribe() method that allows for prettier printing like "6.0 secs"
instead of just "6.0".
* origin/topic/jsiwek/gh-893-intrusive-ptr-migration: (151 commits)
Integrate review feedback
Switch Broker Val converter visitor to return IntrusivePtr
Change BroFunc ctor to take const-ref IntrusivePtr<ID>
Add version of Frame::SetElement() taking IntrusivePtr<ID>
Change Scope/Func inits from id_list* to vector<IntrusivePtr<ID>>
Change Scope::GenerateTemporary() to return IntrusivePtr
Deprecate Scope::ReturnType(), replace with GetReturnType()
Deprecate Scope::ScopeID(), replace with GetID()
Switch parsing to use vector<IntrusivePtr<Attr>> from attr_list
Deprecate TableVal::FindAttr(), replace with GetAttr()
Deprecate TypeDecl::FindAttr(), replace with GetAttr()
Deprecate ID::FindAttr(), replace with GetAttr()
Deprecate Attributes::FindAttr(), replace with Find()
Deprecate Attributes::AddAttrs(Attributes*)
Add Attributes ctor that takes IntrusivePtrs
Change Attributes to store std:vector<IntrusivePtr<Attr>>
Change Attr::SetAttrExpr() to non-template
Deprecate Attr::AttrExpr(), replace with GetExpr()
Deprecate ID::Attrs(), replace with GetAttrs()
Remove weak_ref param from ID::SetVal()
...
In this commit we change our approach and just store table keys as
broker keys and table values as broker indexes. This means we only can
have a single table in a broker store.
This seems to work perfectly - and leads to vastly less complex code.
Code should work, but is actually pretty untested. Might break with
certain types. Does not yet handle item expiration.
Also - this has the tiny issue that all remote operations currently lead
to loops (the operation will immediately be sent to the broker-store
again), which is not quite optimal.
- Updated the logic significantly: still filters out ICMP from being
considered an active service (like before) and adds a new
"Known::service_udp_requires_response" option (defaults to true) for
whether to require UDP server response before being considered an
active service.
* 'topic/dopheide/known-services' of https://github.com/dopheide-esnet/zeek:
Log services with unknown protocols
This compiles, but besides giving debug messages (and partially
performing inserts/updates) it is not really helpful and definitely WIP.
This also shows that I might have to re-think the approach that we will
take here. So far, we actually insert tables as tables into
Brokerstores. This opens up the potential to just have several tables
synchronized via a single brokerstore.
However, it turns out, that the current store_event API sends the
complete table with each update. Which is problematic for obvious
reasons - and not really sustainable.
It turns out that directly passing a Broker::Store is not really a
bright idea. Because - if we do that - we have to later try to intercept
when the master/clone is generated to figure out what the actual name of
the backing store is.
Turns out that it is much easier to just use the name directly - and
check if a store with that name exists when we want to insert something.
I might want to reconsider this in the future in any case. At the moment
this approach just stores one table into an entire store. In theory, one
store should be able to handle several tables, but... that's more
complex. So let's start with this for now.
This also changes the argument type of Func::operator() to zeek::Args*
to allow plugins to be able to alter function arguments in place as
was previously documented.
This commit refactors the SendEvent call and moves it from the Input
ReaderBackend to to MsgThread. This allows all other types of threads
to access this functionality.
This necessitated a few more changes. Most importantly, one of the
ValueToVal methods was moved over to SerialTypes. Whereit arguably
belongs - there was nothing that was input-framework specific in
that method - and the functionality could come in useful in a number
of cases.
* origin/topic/timw/906-find-all-urls-regex:
Restore previous url scheme capture group
GH-906: Fix the regex in url.zeek to better match for find_all_urls
