* origin/topic/vern/ZAM-prep: (45 commits)
whoops overlooked the need to canonicalize filenames
another set of tweaks per review comments
addressed a number of code review comments
baseline updates for merge
support "any" coercions for "-O gen-C++"
better descriptions for named record constructors
test suite baseline updates for "-a opt" optimize-AST alternative
test suite baseline updates for "-a xform" alternative / AST transformation
error propagation fix for AST reduction
updates to "-a inline" test suite alternative baseline
updates for the main test suite baseline
updates to test suite tests for compatibility with upcoming ZAM functionality
"-O compile-all" option to specify compilation of inlined functions
compile inlined functions if they're also used indirectly
provide ZAM-generated code with low-level access to record fields
fix for cloning records with fields of type "any"
direct access for ZAM to VectorVal internal vector
ZVal constructors, accessors & methods in support of ZAM
switch ZVal representation of types from Type objects to TypeVal's
revised error-reporting interface for ZVal's, to accommodate ZAM inner loop
...
Broker::create_master() and Broker::create_clone() now return
a valid value even when there's a failure to open the backend database
(e.g. SQLite filesystem error). In that case, the returned value can
still be passed into other data store operations, but they'll fail
immediately with an error. Broker::is_closed() can now also be used to
determine whether the data store creation calls failed.
This change adds support for complex indexes for sets and tables. With
this change, sets with indexes like:
set[string, count, count]
will function. Before this change, Zeek raised an error message in these
cases.
Addresses GH-1033
The explicit sorting function definition was a temporary stability
workaround with a better fix now being available via
https://github.com/zeek/btest/pull/25
- Use `-b` most everywhere, it will save time.
- Start some intel tests upon the input file being fully read instead of
at an arbitrary time.
- Improve termination condition for some sumstats/cluster tests.
- Filter uninteresting output from some supervisor tests.
- Test for `notice_policy.log` is no longer needed.
Particularly, the final output order of a table/set is sensitive to
order of input/insertions and some tests were converting
std::unordered_{set,map} to Zeek table/set and iteration over those
standard containers may not always loop through elements in the same
order across all platforms.
Haven't checked different build configurations yet, but all except
a few SumStats tests are stable for me now. The external tests
are also completely failing, but haven't looked at those yet.
This addresses PR feedback. The main component in this commit is to
disable &on_change notifications when &backend loads a table from sqlite
on startup.
* origin/topic/jsiwek/gh-1024-broker-store-handle-type-checks:
Improve Broker store API's handling of invalid arguments
Add builtin_exception() functions
GH-1024: fix crash on passing wrong types to Broker store API
This was a bit of a bigger merge since Zeek changed inbetween the time
of the PR and me actually merging it.
I put the new functions into the zeek::detail namespace -- since it
seems unlikely that those will be used by something external.
I also renamed them to fit better with the naming scheme of the new
error functions.
Fixes GH-1024
This commit adds script/c++ documentation and fixes a few loose ends.
It also adds tests for corner cases and massively improves error
messages.
This also actually introduces type-compatibility checking and introduces
a new attribute that lets a user override this if they really know what
they are doing. I am not quite sure if we should really let that stay in
- but it can be very convenient to have this functionality.
One test is continuing to fail - the expiry test is very flaky. This is,
I think, caused by delays of the broker store forwarding. I am unsure if
we can actually do anything about that.
With this, the basic functionality of &backend seems to be working.
It is not yet integrated with zeekctl, one has to manually specify the
storage location for the sqlite files somewhere when using sqlite.
Usage for memory stores:
global table_to_share: table[string] of count &backend=Broker::MEMORY;
Usage for sqlite stores:
redef Broker::auto_store_db_directory = "[path]";
global table_to_share: table[string] of count &backend=Broker::SQLITE;
In both cases, the cluster should automatically sync to changes done by
any node. When using sqlite, data should also be saved to disk and
re-loaded on startup.
Currently this requires using this with a normal cluster - or sending
messages by yourself.
It, in principle, should also work with SQLITE - but that is a bit
nonsensical without being able to change the storage location.
The &backend attribute allows for a much more convenient way of
interacting with brokerstores. One does not need to create a broker
store anymore - instead all of this is done internally.
The current state of this partially works. This should work fine for
persistence - but clones are currently not yet correctly attached.
When a clone attaches to a master, it just gets the diffs sent as
events. Which is neat because it means that we pretty much don't need
any extra code to handle this.
This currently only handles the most basic case, and is not thoroughly
tested.
When initializing a master store, we now check if there already is data
in it. If yes, we load it directly into the zeek table when the store is
created. We assume that this is happening at Zeek startup - and are
supremely evil and just load it synchronously. Which could block
execution for a bit for larger stores.
That being said - this might sidestep other issues that would arise when
doing this async (like scripts already inserting data).
Next step: check if this approach also works for clones.
* origin/master:
Fix shadowed variable that breaks lookup_hostname()
GH-1025: allow copying/cloning of `opaque of Broker::Store`
Fix "possibly-truncated" compiler warning in BuildJSON snprintf()
Update submodule(s)
Fixed some places where tabs became spaces
Convert to using permissions to check for access to cirrus variables in benchmark script
Integrate review feedback: improve command-line option redef parsing
Fix several issues with command-line option redefs
Remove last_access_time from TableEntryVal.
Minimize data published for expected FTP data channel analysis
Stricter checking if we have a dns field on the connection being processed
Modified the DNS protocol analyzer to add a new parameter to the dns_request event which includes the DNS query in its original case. Added a policy script that will add the original_case to the dns.log file as well. Created new btests to test both.
Place build file in explicit location for benchmarking to work correctly
cmake: Make musl support more distro agnostic
Update highwayhash submodule to upstream.
GH-998: Fix Reporter::conn_weird() to handle expired connections
Makes some attributes conflict with each other. This also needed the
test to change.
The test is a bit flaky - but I can, for the heck of it, not figure out
why. I am punting that for the future after spending a few hours on it.
* Some methods mistakenly returned a bool instead of QueryResult
when passed an invalid `opaque of Broker::Store` handle.
* Now generates a runtime exception for store_name() and is_closed()
calls that pass an invalid `opaque of Broker::Store` handle as any
returned value can't be reasonably used in any subsequent logic.
* Descriptions of any invalid arguments are now given in the error
message.
This commit fixes a few more loose ends to actually make the
Zeek Table<->brokerstore syncing work. This mostly slightly changes the
TableVal assign/remove operators to prevent loops when a remote change
arrives.
The tests inserts a value into a table on the manager, and it pops out
in a table on a clone - which is the easiest case.
Timeouts are still not handled at all; the behavior when inserting into
a clone is untested.
This also installs symlinks from "zeek" and "bro-config" to a wrapper
script that prints a deprecation warning.
The btests pass, but this is still WIP. broctl renaming is still
missing.
#239
This builds upon the previous commit to make Zeek compile on FIPS
systems.
This patch makes the changes a bit more aggressive. Instead of having a
number of different hash functions with different return values, we now
standardize on EVP_MD_CTX and just have one set of functions, to which
the hash algorithm that is desired is passed.
On the positive side, this enables us to support a wider range of hash
algorithm (and to easily add to them in the future).
I reimplemented the internal_md5 function - we don't support ebdic
systems in any case.
The md5/sha1 serialization functions are now also tested (I don't think
they were before).
