I needed to figure out which exact algorithm we use for our
probabilistic top-k measurements. It turns out that we do not mention
this in our source tree at all so far.
This change allows users to specify an epoch length of 0, which means
that the user manually has to finish the epochs. A new next_epoch
function is introduced to allow users to manually end epochs.
Addresses GH-348
In non-clustered mode, epoch_finished was not called when there was no
data during the epoch.
This behavior does not fit the documentation, and also is different in
cluster-mode, where epoch_finished is, indeed, called after every epoch.
This small change fixes this behavior.
* "bro_is_terminating" is now "zeek_is_terminating"
* "bro_version" is now "zeek_version"
The old function names still exist for now, but are deprecated.
* All "Broxygen" usages have been replaced in
code, documentation, filenames, etc.
* Sphinx roles/directives like ":bro:see" are now ":zeek:see"
* The "--broxygen" command-line option is now "--zeexygen"
The loop over Reducers in SumStats::observe performs a key normalization
and inadvertently modifies the key used for subsequent iterations.
Reported by Jim Mellander.
* Add proper namespace scoping to a 'SumStats::process_epoch_result'
scheduled event.
* Fix iterator invalidation within 'SumStats::process_epoch_result'
* Give 'SumStats::process_epoch_result' a copy of the result table so
that the SumStats framework can clear the original and move on to the
next epoch immediately.
* The previous baseline of the basic sumstats unit test did look wrong
to me and probably was actually indicative of the iterator invalidation
problem.
Thanks to Jim Mellander for reporting the issues.
Currently outstanding_global_views values are only decremented during
the end of epoch cleanup, but not when handle_end_of_result_collection
is called for the specific uid that actually triggered the result
collection (which is specifically NOT a cleanup event).
This changes outstanding_global_views values to be a set of outstanding
uids, instead of a count. This allows handle_end_of_result_collection
to remove any uids from the set as it sees them.
Previously, recent_global_view_keys was only tracked on workers causing
a popular key to be sent up and handled by the manager once for each
worker.
This records the key inside recent_global_view_keys on the manager after
the first update, making the rest of the updates no-ops.
Additionally, since the counter value was never used, it has been
changed from a table to a set.
This prevents the worker nodes from crashing, when request_key is used
in cluster mode and called on the worker and the manager nodes (i.e. when
a non-cluster-aware script is used).
Addresses BIT-1177
Some documentation of commented-out items was being output to HTML,
which was causing that documentation to be merged with the
documentation of the next item. Fixed by changing "##" to "#" so
that the doc comments are not included in the HTML.
The text from these README files appears on the "Bro Script Packages"
page after building the documentation. The text for these was mostly just
copied from the existing docs.
* origin/topic/bernhard/ticket1072:
and const 2 more functions
update hll documentation, make a few functions private and create a new copy constructor.
fix case where hll_error_margin could be undefined (thanks John)
BIT-1072 #merged
This cleans up most of the warnings from sphinx (broken :doc: links,
broxygen role misuses, etc.). The remaining ones should be harmless,
but not quick to silence.
I found that the README for each component was a copy from the actual
repo, so I turned those in to symlinks so they don't get out of date.
* topic/robin/hyperloglog-merge: (35 commits)
Making the confidence configurable.
Renaming HyperLogLog->CardinalityCounter.
Fixing bug introduced during merging.
add clustered leak test for hll. No issues.
make gcc happy
(hopefully) fix refcounting problem in hll/bloom-filter opaque vals. Thanks Robin.
re-use same hash class for all add operations
get hll ready for merging
and forgot a file...
adapt to new structure
fix opaqueval-related memleak.
make it compile on case-sensitive file systems and fix warnings
make error rate configureable
add persistence test not using predetermined random seeds.
update cluster test to also use hll
persistence really works.
well, with this commit synchronizing the data structure should work.. ...if we had consistent hashing.
and also serialize the other things we need
ok, this bug was hard to find.
serialization compiles.
...
* origin/topic/bernhard/hyperloglog: (32 commits)
add clustered leak test for hll. No issues.
make gcc happy
(hopefully) fix refcounting problem in hll/bloom-filter opaque vals. Thanks Robin.
re-use same hash class for all add operations
get hll ready for merging
and forgot a file...
adapt to new structure
fix opaqueval-related memleak.
make it compile on case-sensitive file systems and fix warnings
make error rate configureable
add persistence test not using predetermined random seeds.
update cluster test to also use hll
persistence really works.
well, with this commit synchronizing the data structure should work.. ...if we had consistent hashing.
and also serialize the other things we need
ok, this bug was hard to find.
serialization compiles.
change plugin after feedback of seth
Forgot a file. Again. Like always. Basically.
do away with old file.
...
