Mirror/zeek
1
0
Fork 0
mirror of https://github.com/zeek/zeek.git synced 2025-10-06 16:48:19 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 2

Merge branch 'master' into topic/jsiwek/gh-320

Browse source
This commit is contained in:
Seth Hall 2019-06-26 14:47:01 -04:00 • committed by GitHub
commit eb690a18cc
No known key found for this signature in database
GPG key ID: 4AEE18F83AFDEB23
3266 changed files with 87729 additions and 103138 deletions
Download patch file Download diff file Expand all files Collapse all files

4
.gitignore vendored
View file

@ -1,3 +1,7 @@
build
tmp
*.gcov
# Configuration and build directories for CLion
.idea
cmake-build-debug

12
.gitmodules vendored
View file

@ -4,12 +4,9 @@
[submodule "aux/binpac"]
path = aux/binpac
url = https://github.com/zeek/binpac
[submodule "aux/broccoli"]
path = aux/broccoli
url = https://github.com/zeek/broccoli
[submodule "aux/broctl"]
path = aux/broctl
url = https://github.com/zeek/broctl
[submodule "aux/zeekctl"]
path = aux/zeekctl
url = https://github.com/zeek/zeekctl
[submodule "aux/btest"]
path = aux/btest
url = https://github.com/zeek/btest
@ -31,3 +28,6 @@
[submodule "doc"]
path = doc
url = https://github.com/zeek/zeek-docs
[submodule "aux/paraglob"]
path = aux/paraglob
url = https://github.com/zeek/paraglob

6
.travis.yml
View file

@ -16,10 +16,10 @@ branches:
notifications:
email:
recipients:
- bro-commits-internal@bro.org
- zeek-commits-internal@zeek.org
# Build Bro and run tests in the following Linux distros (specifying "travis"
# builds bro in Travis without using docker).
# Build Zeek and run tests in the following Linux distros (specifying "travis"
# builds Zeek in Travis without using docker).
env:
- distro: centos_7
- distro: debian_9

12
.update-changes.cfg
View file

@ -7,15 +7,7 @@ function new_version_hook
# test suite repos to check out on a CI system.
version=$1
if [ -d testing/external/zeek-testing ]; then
echo "Updating testing/external/commit-hash.zeek-testing"
( cd testing/external/zeek-testing && git fetch origin && git rev-parse origin/master ) > testing/external/commit-hash.zeek-testing
git add testing/external/commit-hash.zeek-testing
fi
./testing/scripts/update-external-repo-pointer.sh testing/external/zeek-testing testing/external/commit-hash.zeek-testing
if [ -d testing/external/zeek-testing-private ]; then
echo "Updating testing/external/commit-hash.zeek-testing-private"
( cd testing/external/zeek-testing-private && git fetch origin && git rev-parse origin/master ) > testing/external/commit-hash.zeek-testing-private
git add testing/external/commit-hash.zeek-testing-private
fi
./testing/scripts/update-external-repo-pointer.sh testing/external/zeek-testing-private testing/external/commit-hash.zeek-testing-private
}

707
CHANGES
View file

@ -1,4 +1,711 @@
2.6-526 | 2019-06-25 12:45:31 -0700
* Make a paraglob unit test parallelizable (Jon Siwek, Corelight)
2.6-523 | 2019-06-25 10:38:24 -0700
* GH-427: improve default ID values shown by Zeekygen
The default value of an ID is now truly the one used to initialize it,
unaltered by any subsequent redefs.
Redefs are now shown separately, along with the expression that
modifies the ID's value. (Jon Siwek, Corelight)
* Unbreak build on Linux (Johanna Amann, Corelight)
2.6-519 | 2019-06-24 15:25:08 -0700
* GH-435: fix null pointer deref in RPC analyzer. (Jon Siwek, Corelight)
2.6-517 | 2019-06-24 15:20:39 -0700
* Add paraglob, a fairly quick data structure for matching a string against a large list of patterns.
(Zeke Medley, Corelight)
* GH-171: support warning messages alongside deprecated attributes (Tim Wojtulewicz, Corelight)
2.6-503 | 2019-06-21 11:17:58 -0700
* GH-417: Remove old, unmaintained p0f support. (Johanna Amann, Corelight)
2.6-500 | 2019-06-20 20:54:15 -0700
* Add new RDP event: rdp_client_cluster_data (Jeff Atkinson)
* Added "options" field to RDP::ClientChannelDef (Jeff Atkinson)
2.6-494 | 2019-06-20 20:24:38 -0700
* Renaming src/StateAccess.{h,cc} to src/Notifier.{h,cc}.
The old names did not reflect the content of the files anymore. (Robin Sommer, Corelight)
* Remove MutableVal, StateAccess classes, enum Opcode. (Robin Sommer, Corelight)
* Redo API for notifiers.
There's now an notifier::Modifiable interface class that class
supposed to signal modifications are to be derived from. This takes
the place of the former MutableValue class and also unifies how Val
and IDs signal modifications. (Robin Sommer, Corelight)
* Redo NotfifierRegistry to no longer rely on StateAccess.
We simplify the API to a simple Modified() operation. (Robin Sommer, Corelight)
* Add new test for when-statement watching global variables. (Robin Sommer, Corelight)
2.6-482 | 2019-06-20 19:57:20 -0700
* Make configure complain if submodules are not checked out. (Johanna Amann, Corelight)
* Improve C++ header includes to improve build time (Jon Siwek, Corelight)
2.6-479 | 2019-06-20 18:31:58 -0700
* Fix TableVal::DoClone to use CloneState cache (Jon Siwek, Corelight)
2.6-478 | 2019-06-20 14:19:11 -0700
* Remove old Broccoli SSL options (Jon Siwek, Corelight)
- ssl_ca_certificate
- ssl_private_key
- ssl_passphrase
2.6-477 | 2019-06-20 14:00:22 -0700
* Remove unused SerialInfo.h and SerialTypes.h headers (Jon Siwek, Corelight)
2.6-476 | 2019-06-20 13:23:22 -0700
* Remove opaque of ocsp_resp. (Johanna Amann, Corelight)
Only used in one event, without any way to use the opaque for anything
else. At this point this just seems like a complication that has no
reason to be there.
* Remove remnants of event serializer. (Johanna Amann, Corelight)
* Reimplement serialization infrastructure for OpaqueVals.
(Robin Sommer, Corelight & Johanna Amann, Corelight)
We need this to sender through Broker, and we also leverage it for
cloning opaques. The serialization methods now produce Broker data
instances directly, and no longer go through the binary formatter.
Summary of the new API for types derived from OpaqueVal:
- Add DECLARE_OPAQUE_VALUE(<class>) to the class declaration
- Add IMPLEMENT_OPAQUE_VALUE(<class>) to the class' implementation file
- Implement these two methods (which are declated by the 1st macro):
- broker::data DoSerialize() const
- bool DoUnserialize(const broker::data& data)
This machinery should work correctly from dynamic plugins as well.
OpaqueVal provides a default implementation of DoClone() as well that
goes through serialization. Derived classes can provide a more
efficient version if they want.
The declaration of the "OpaqueVal" class has moved into the header
file "OpaqueVal.h", along with the new serialization infrastructure.
This is breaking existing code that relies on the location, but
because the API is changing anyways that seems fine.
* Implement a Shallow Clone operation for types. (Johanna Amann, Corelight)
This is needed to track name changes for the documentation.
* Remove old serialization infrastrucutre. (Johanna Amann, Corelight)
2.6-454 | 2019-06-19 09:39:06 -0700
* GH-393: Add slice notation for vectors (Tim Wojtulewicz, Corelight & Jon Siwek, Corelight)
Example Syntax:
local v = vector(1, 2, 3, 4, 5);
v[2:4] = vector(6, 7, 8); # v is now [1, 2, 6, 7, 8, 5]
print v[:4]; # prints [1, 2, 6, 7]
2.6-446 | 2019-06-17 20:26:49 -0700
* Rename bro to zeek in error messages (Daniel Thayer)
2.6-444 | 2019-06-15 19:09:03 -0700
* Add/rewrite NTP support (Vlad Grigorescu and Mauro Palumbo)
2.6-416 | 2019-06-14 20:57:57 -0700
* DNS: Add support for SPF response records (Vlad Grigorescu)
2.6-413 | 2019-06-14 19:51:28 -0700
* GH-406: rename bro.bif to zeek.bif (Jon Siwek, Corelight)
2.6-412 | 2019-06-14 19:26:21 -0700
* GH-387: update Broker topic names to use "zeek/" prefix (Jon Siwek, Corelight)
* GH-323: change builtin plugin namespaces to Zeek (Jon Siwek, Corelight)
2.6-408 | 2019-06-13 11:19:50 -0700
* Fix potential null-dereference in current_time() (Tim Wojtulewicz, Corelight)
* Add --sanitizers configure script to enable Clang sanitizers (Tim Wojtulewicz, Corelight)
2.6-404 | 2019-06-12 15:10:19 -0700
* Rename directories from bro to zeek (Daniel Thayer)
The new default installation prefix is /usr/local/zeek
2.6-400 | 2019-06-07 20:06:33 -0700
* Adapt bro_plugin CMake macros to use zeek_plugin (Jon Siwek, Corelight)
2.6-399 | 2019-06-07 14:02:18 -0700
* Update SSL documentation. (Johanna Amann)
* Support the newer TLS 1.3 key_share extension. (Johanna Amann)
* Include all data of the server-hello random (Johanna Amann)
Before we cut the first 4 bytes, which makes it impossible to recognize
several newer packets (like the hello retry).
* Parse TLS 1.3 pre-shared-key extension. (Johanna Amann)
Adds new events:
- ssl_extension_pre_shared_key_client_hello
- ssl_extension_pre_shared_key_server_hello
2.6-391 | 2019-06-07 17:29:28 +1000
* GH-209: replace "remote_ip" field of radius.log with "tunnel_client".
Also changes type from addr to string. (Jon Siwek, Corelight)
2.6-389 | 2019-06-06 20:02:19 -0700
* Update plugin unit tests to use --zeek-dist (Jon Siwek, Corelight)
2.6-388 | 2019-06-06 19:48:55 -0700
* Change default value of peer_description "zeek" (Jon Siwek, Corelight)
2.6-387 | 2019-06-06 18:51:09 -0700
* Rename Bro to Zeek in Zeekygen-generated documentation (Jon Siwek, Corelight)
2.6-386 | 2019-06-06 17:17:55 -0700
* Add new RDP event: rdp_native_encrytped_data (Anthony Kasza, Corelight)
2.6-384 | 2019-06-06 16:49:14 -0700
* Add new RDP event: rdp_client_security_data (Jeff Atkinson)
2.6-379 | 2019-06-06 11:56:58 -0700
* Improve sqlite logging unit tests (Jon Siwek, Corelight)
2.6-378 | 2019-06-05 16:23:04 -0700
* Rename BRO_DEPRECATED macro to ZEEK_DEPRECATED (Jon Siwek, Corelight)
2.6-377 | 2019-06-05 16:15:58 -0700
* Deprecate functions with "bro" in them. (Jon Siwek, Corelight)
* "bro_is_terminating" is now "zeek_is_terminating"
* "bro_version" is now "zeek_version"
The old functions still exist for now, but are deprecated.
2.6-376 | 2019-06-05 13:29:57 -0700
* GH-379: move catch-and-release and unified2 scripts to policy/ (Jon Siwek, Corelight)
These are no longer loaded by default due to the performance impact they
cause simply by being loaded (they have event handlers for commonly
generated events) and they aren't generally useful enough to justify it.
2.6-375 | 2019-06-04 19:28:06 -0700
* Simplify threading::Value destructor (Jon Siwek, Corelight)
* Add pattern support to input framework. (Zeke Medley, Corelight)
2.6-369 | 2019-06-04 17:53:10 -0700
* GH-155: Improve coercion of expression lists to vector types (Tim Wojtulewicz, Corelight)
* GH-159: Allow coercion of numeric record field values to other types (Tim Wojtulewicz, Corelight)
* Allow passing a location to BroObj::Warning and BroObj::Error. (Tim Wojtulewicz, Corelight)
This allows callers (such as check_and_promote) to pass an expression
location to be logged if the location doesn't exist in the value being
promoted.
* Add CLion directories to gitignore (Tim Wojtulewicz, Corelight)
* Move #define outside of max_type for clarity (Tim Wojtulewicz, Corelight)
2.6-361 | 2019-06-04 10:30:21 -0700
* GH-293: Protect copy() against reference cycles. (Robin Sommer, Corelight)
Reference cycles shouldn't occur but there's nothing really preventing
people from creating them, so may just as well be safe and deal with
them when cloning values.
2.6-359 | 2019-05-31 13:37:17 -0700
* Remove old documentation reference to rotate_interval (Jon Siwek, Corelight)
2.6-357 | 2019-05-30 10:57:54 -0700
* Tweak to ASCII reader warning suppression (Christian Kreibich, Corelight)
Warnings in the ASCII reader so far remained suppressed even
when an input file changed. It's helpful to learn about problems
in the data when putting in place new data files, so this change
maintains the existing warning suppression while processing a file,
but re-enables warnings after updates to a file.
2.6-354 | 2019-05-29 09:46:19 -0700
* Add weird: "RDP_channels_requested_exceeds_max" (Vlad Grigorescu)
2.6-352 | 2019-05-28 17:57:36 -0700
* Reduce data copying in Broker message processing (Jon Siwek, Corelight)
* Improve Broker I/O loop integration: less mutex locking (Jon Siwek, Corelight)
Checking a subscriber for available messages required locking a mutex,
but we should never actually need to do that in the main-loop to check
for Broker readiness since we can rely on file descriptor polling.
* Improve processing of broker data store responses (Jon Siwek, Corelight)
Now retrieves and processes all N available responses at once instead
of one-by-one-until-empty.
2.6-345 | 2019-05-28 11:32:16 -0700
* RDP: Add parsing and logging of channels requested by the client. (Vlad Grigorescu)
Can determine capabilities requested by the client, as well as attacks such
as CVE-2019-0708.
2.6-342 | 2019-05-28 10:48:37 -0700
* GH-168: Improve type-checking for table/set list assignment. (Zeke Medley and Jon Siwek, Corelight)
2.6-340 | 2019-05-24 18:02:43 -0700
* Add support for parsing additional DHCP options (Jay Wren)
The following optional fields were added to the DHCP::Options record:
- time_offset (Option 2)
- time_servers (Option 4)
- name_servers (Option 5)
- ntp_servers (Option 42)
2.6-338 | 2019-05-24 17:06:08 -0700
* Add input file name to additional ASCII reader warning messages (Christian Kreibich, Corelight)
2.6-336 | 2019-05-24 10:23:20 -0700
* GH-378: check validity of missing 'val' field in Input::add_table (Jon Siwek, Corelight)
2.6-335 | 2019-05-24 08:58:59 -0700
* Fix memory leak when no protocol_violation event handler exists (Jon Siwek, Corelight)
2.6-334 | 2019-05-23 20:40:03 -0700
* Add an internal getenv wrapper function: zeekenv (Jon Siwek, Corelight)
It maps newer environment variable names starting with ZEEK to the
legacy names starting with BRO.
* Rename all BRO-prefixed environment variables (Daniel Thayer)
For backward compatibility when reading values, we first check
the ZEEK-prefixed value, and if not set, then check the corresponding
BRO-prefixed value.
2.6-331 | 2019-05-23 18:03:42 -0700
* Update broker unit test output. (Jon Siwek, Corelight)
Due to string representation of Broker vectors changing (they now
use parentheses instead of square brackets).
2.6-330 | 2019-05-23 13:04:26 -0700
* GH-173: Support ranges of values for value_list elements in the signature parser
(Tim Wojtulewicz, Corelight)
* GH-173: Modify the signature parser so ID components can't start with numbers
(Tim Wojtulewicz, Corelight)
2.6-327 | 2019-05-23 11:56:11 -0700
* Remove redundant RecordVal::record_type member (Jon Siwek, Corelight)
2.6-326 | 2019-05-23 10:49:38 -0700
* Fix parse-time RecordVal tracking containing duplicates (Jon Siwek, Corelight)
2.6-325 | 2019-05-22 23:56:23 -0700
* Add leak-checks for new copy operations (Johanna Amann, Corelight)
* Finish implementation of new copy method. (Johanna Amann, Corelight)
All types (besides EntropyVal) now support a native copy operation,
which uses primitives of the underlying datatypes to perform a quick
copy, without serialization.
EntropyVal is the one exception - since that type is rather complex
(many members) and will probably not be copied a lot, if at all, it
makes sense to just use the serialization function.
This will have to be slightly re-written in the near-term-future to use
the new serialization function for that opaque type.
This change also introduces a new x509_from_der bif, which allows to
parse a der into an opaque of x509.
This change removes the d2i_X509_ wrapper function; this was a remnant
when d2i_X509 took non-const arguments. We directly use d2i_X509 at
several places assuming const-ness, so there does not seem to ba a
reason to keep the wrapper.
This change also exposed a problem in the File cache - cases in which an
object was brought back into the cache, and writing occurred in the
file_open event were never correctly handeled as far as I can tell.
* Reimplement copy(). (Robin Sommer, Corelight)
The old implementation used the serialization framework, which is
going away. This is a new standalone implementation that should also
be quite a bit faster.
2.6-318 | 2019-05-21 09:17:53 -0700
* Remove state_dir and state_write_delay options (Jon Siwek, Corelight)
* Remove a reference to &synchronized from docs (Jon Siwek, Corelight)
2.6-316 | 2019-05-20 20:56:46 -0700
* Additional Bro to Zeek renaming (Daniel Thayer)
* Added a new unit test for legacy Bro Plugins (Daniel Thayer)
* Added a symlink bro-path-dev.in for use by legacy Bro packages (Daniel Thayer)
2.6-314 | 2019-05-20 16:20:33 -0700
* Remove deprecated attributes. (Johanna Amann, Corelight)
To be more exact: &encrypt, &mergeable, &rotate_interval, &rotate_size
Also removes no longer used redef-able constants:
log_rotate_interval, log_max_size, log_encryption_key
2.6-311 | 2019-05-20 09:07:58 -0700
* Add missing &optional attr to KRB record fields; also add existence
checks to scripts (Jon Siwek, Corelight).
2.6-308 | 2019-05-17 14:13:46 -0700
* Always emit scripting errors to stderr during zeek_init (Jon Siwek, Corelight)
2.6-307 | 2019-05-16 13:37:24 -0700
* More bro-to-zeek renaming in scripts and other files (Daniel Thayer)
* More bro-to-zeek renaming in the unit tests (Daniel Thayer)
2.6-303 | 2019-05-15 15:03:11 -0700
* Changes needed due to bro-to-zeek renaming in broker (Daniel Thayer)
2.6-301 | 2019-05-15 10:05:53 -0700
* Fix potential race in openflow broker plugin (Jon Siwek, Corelight)
2.6-300 | 2019-05-15 09:00:57 -0700
* Fixes to DNS lookup, including ref-counting bugs, preventing starvation
of the DNS_Mgr in the I/O loop, dead code removal, and a fix that
prevents the timeout of already resolved DNS lookups (Jon Siwek, Corelight)
2.6-292 | 2019-05-14 19:01:05 -0700
* Fix maybe-uninitialized compiler warning (Jon Siwek, Corelight)
2.6-290 | 2019-05-14 18:35:25 -0700
* Update btest.cfg path to use zeek-aux (Jon Siwek, Corelight)
2.6-288 | 2019-05-14 17:47:55 -0700
* Update CMake to use aux/zeekctl and aux/zeek-aux submodules (Jon Siwek, Corelight)
2.6-287 | 2019-05-14 17:40:40 -0700
* Rename broctl submodule to zeekctl (Jon Siwek, Corelight)
2.6-286 | 2019-05-14 13:19:12 -0700
* Undo an unintentional change to btest.cfg from a recent commit (Daniel Thayer)
* Fix zeek-wrapper and improve error messages (Daniel Thayer)
The script was not passing command-line arguments to the new program.
* Update for renaming BroControl to ZeekControl. (Robin Sommer, Corelight)
* GH-239: Rename bro to zeek, bro-config to zeek-config, and bro-path-dev to zeek-path-dev.
(Robin Sommer, Corelight)
This also installs symlinks from "zeek" and "bro-config" to a wrapper
script that prints a deprecation warning.
2.6-279 | 2019-05-13 20:02:59 -0700
* GH-365: improve un-indexable type error message (Jon Siwek, Corelight)
2.6-277 | 2019-05-08 12:42:18 -0700
* Allow tuning Broker log batching via scripts (Jon Siwek, Corelight)
Via redefining "Broker::log_batch_size" or "Broker::log_batch_interval"
2.6-276 | 2019-05-08 09:03:27 -0700
* Force the Broker IOSource to idle periodically, preventing packet
IOSource starvation. (Jon Siwek, Corelight).
2.6-274 | 2019-05-08 08:58:25 -0700
* GH-353: Add `/<re>/i` case-insensitive signature syntax (Jon Siwek, Corelight)
2.6-272 | 2019-05-06 18:43:13 -0700
* Remove support for using && and || with patterns. (Johanna Amann, Corelight)
This was never documented and previously deprecated.
* Remove RemoteSerializer and related code/types. (Johanna Amann, Corelight)
Also removes broccoli from the source tree.
* Remove PersistenceSerializer. (Johanna Amann, Corelight)
* Remove &synchronized and &persistent attributes. (Johanna Amann, Corelight)
2.6-264 | 2019-05-03 11:16:38 -0700
* Fix sporadic openflow/broker test failure (Jon Siwek, Corelight)
2.6-263 | 2019-05-02 22:49:40 -0700
* Install local.zeek as symlink to pre-existing local.bro (Jon Siwek, Corelight)
This a convenience for those that are upgrading. If we didn't do
this, then deployments can silently break until the user intervenes
since BroControl now prefers to load the initially-vanilla local.zeek
instead of the formerly-customized local.bro.
2.6-262 | 2019-05-02 21:39:01 -0700
* Rename Zeexygen to Zeekygen (Jon Siwek, Corelight)
2.6-261 | 2019-05-02 20:49:23 -0700
* Remove previously deprecated policy/protocols/smb/__load__ (Jon Siwek, Corelight)
2.6-260 | 2019-05-02 19:16:48 -0700
* GH-243: Remove deprecated functions/events from 2.6 and earlier (Johanna Amann, Corelight)
2.6-258 | 2019-05-02 12:26:54 -0700
* GH-340: Improve IPv4/IPv6 regexes, extraction, and validity functions.
is_valid_ip() is not a BIF, the IP regular expressions are improved and
extract_ip_addresses should give better results due to this.
(Jon Siwek, Corelight)
2.6-255 | 2019-05-01 08:38:49 -0700
* Add methods to queue events without handler existence check
Added ConnectionEventFast() and QueueEventFast() methods to avoid
redundant event handler existence checks.
It's common practice for caller to already check for event handler
existence before doing all the work of constructing the arguments, so
it's desirable to not have to check for existence again.
E.g. going through ConnectionEvent() means 3 existence checks:
one you do yourself before calling it, one in ConnectionEvent(), and then
another in QueueEvent().
The existence check itself can be more than a few operations sometimes
as it needs to check a few flags that determine if it's enabled, has
a local body, or has any remote receivers in the old comm. system or
has been flagged as something to publish in the new comm. system. (Jon Siwek, Corelight)
* Cleanup/improve PList usage and Event API
Majority of PLists are now created as automatic/stack objects,
rather than on heap and initialized either with the known-capacity
reserved upfront or directly from an initializer_list (so there's no
wasted slack in the memory that gets allocated for lists containing
a fixed/known number of elements).
Added versions of the ConnectionEvent/QueueEvent methods that take
a val_list by value.
Added a move ctor/assign-operator to Plists to allow passing them
around without having to copy the underlying array of pointers. (Jon Siwek, Corelight)
2.6-250 | 2019-04-29 18:09:29 -0700
* Remove 'dns_resolver' option, replace w/ ZEEK_DNS_RESOLVER env. var. (Jon Siwek, Corelight)
2.6-249 | 2019-04-26 19:26:44 -0700
* Fix parsing of hybrid IPv6-IPv4 addr literals with no zero compression (Jon Siwek, Corelight)
2.6-246 | 2019-04-25 10:22:11 -0700
* Add Zeexygen cross-reference links for some events (Jon Siwek, Corelight)
2.6-245 | 2019-04-23 18:42:02 -0700
* Expose TCP analyzer utility functions to derived classes (Vern Paxson, Corelight)
2.6-243 | 2019-04-22 19:42:52 -0700
* GH-234: rename Broxygen to Zeexygen along with roles/directives (Jon Siwek, Corelight)
* All "Broxygen" usages have been replaced in
code, documentation, filenames, etc.
* Sphinx roles/directives like ":bro:see" are now ":zeek:see"
* The "--broxygen" command-line option is now "--zeexygen"
2.6-242 | 2019-04-22 22:43:09 +0200
* update SSL consts from TLS 1.3 (Johanna Amann)
2.6-241 | 2019-04-22 12:38:06 -0700
* Add 'g' character to conn.log history field to flag content gaps (Vern Paxson, Corelight)
There's also a small change to TCP state machine that distrusts ACKs
appearing at the end of connections (in FIN or RST) such that they won't
count towards revealing a true content gap.
2.6-237 | 2019-04-19 12:00:37 -0700
* GH-236: Add zeek_script_loaded event, deprecate bro_script_loaded (Jon Siwek, Corelight)
Existing handlers for bro_script_loaded automatically alias to the new
zeek_script_loaded event, but emit a deprecation warning.
2.6-236 | 2019-04-19 11:16:35 -0700
* Add zeek_init/zeek_done events and deprecate bro_init/bro_done (Seth Hall, Corelight)
Any existing handlers for bro_init and bro_done will automatically alias
to the new zeek_init and zeek_done events such that code will not break,
but will emit a deprecation warning.
2.6-232 | 2019-04-18 09:34:13 +0200
* Prevent topk_merge from crashing when second argument is empty set (Jeff Barber)
2.6-230 | 2019-04-17 16:44:16 -0700
* Fix unit test failures on case-insensitive file systems (Jon Siwek, Corelight)
2.6-227 | 2019-04-16 17:44:31 -0700
* GH-237: add `@load foo.bro` -> foo.zeek fallback (Jon Siwek, Corelight)
When failing to locate a script with explicit .bro suffix, check for
whether one with a .zeek suffix exists and use it instead.
2.6-225 | 2019-04-16 16:07:49 -0700
* Use .zeek file suffix in unit tests (Jon Siwek, Corelight)
2.6-223 | 2019-04-16 11:56:00 -0700
* Update tests and baselines due to renaming all scripts (Daniel Thayer)
* Rename all scripts to have ".zeek" file extension (Daniel Thayer)
* Add test cases to verify new file extension is recognized (Daniel Thayer)
* Fix the core/load-duplicates.bro test (Daniel Thayer)
* Update script search logic for new .zeek file extension (Daniel Thayer)
When searching for script files, look for both the new and old file
extensions. If a file with ".zeek" can't be found, then search for
a file with ".bro" as a fallback.
* Remove unnecessary ".bro" from @load directives (Daniel Thayer)
2.6-212 | 2019-04-12 10:12:31 -0700
* smb2_write_response event added (Mauro Palumbo)
2.6-210 | 2019-04-10 09:54:27 -0700
* Add options to tune BinPAC flowbuffer policy (Jon Siwek, Corelight)
2.6-208 | 2019-04-10 11:36:17 +0000
* Improve PE file analysis (Jon Siwek, Corelight)
* Set PE analyzer CMake dependencies correctly (Jon Siwek, Corelight)
2.6-205 | 2019-04-05 17:06:26 -0700
* Add script to update external test repo commit pointers (Jon Siwek, Corelight)
2.6-203 | 2019-04-04 16:35:52 -0700
* Update DTLS error handling (Johanna Amann, Corelight)
- Adds tuning options: SSL::dtls_max_version_errors and
SSL::dtls_max_reported_version_errors
2.6-200 | 2019-04-03 09:44:53 -0700
* Fix reporter net_weird API usage for unknown_mobility_type

139
CMakeLists.txt
View file

@ -1,7 +1,7 @@
project(Bro C CXX)
project(Zeek C CXX)
# When changing the minimum version here, also adapt
# aux/bro-aux/plugin-support/skeleton/CMakeLists.txt
# aux/zeek-aux/plugin-support/skeleton/CMakeLists.txt
cmake_minimum_required(VERSION 2.8.12 FATAL_ERROR)
include(cmake/CommonCMakeConfig.cmake)
@ -21,34 +21,34 @@ if ( ENABLE_CCACHE )
set(CMAKE_CXX_COMPILER_LAUNCHER ${CCACHE_PROGRAM})
endif ()
set(BRO_ROOT_DIR ${CMAKE_INSTALL_PREFIX})
if (NOT BRO_SCRIPT_INSTALL_PATH)
# set the default Bro script installation path (user did not specify one)
set(BRO_SCRIPT_INSTALL_PATH ${BRO_ROOT_DIR}/share/bro)
set(ZEEK_ROOT_DIR ${CMAKE_INSTALL_PREFIX})
if (NOT ZEEK_SCRIPT_INSTALL_PATH)
# set the default Zeek script installation path (user did not specify one)
set(ZEEK_SCRIPT_INSTALL_PATH ${ZEEK_ROOT_DIR}/share/zeek)
endif ()
if (NOT BRO_MAN_INSTALL_PATH)
# set the default Bro man page installation path (user did not specify one)
set(BRO_MAN_INSTALL_PATH ${BRO_ROOT_DIR}/share/man)
if (NOT ZEEK_MAN_INSTALL_PATH)
# set the default Zeek man page installation path (user did not specify one)
set(ZEEK_MAN_INSTALL_PATH ${ZEEK_ROOT_DIR}/share/man)
endif ()
# sanitize the Bro script install directory into an absolute path
# sanitize the Zeek script install directory into an absolute path
# (CMake is confused by ~ as a representation of home directory)
get_filename_component(BRO_SCRIPT_INSTALL_PATH ${BRO_SCRIPT_INSTALL_PATH}
get_filename_component(ZEEK_SCRIPT_INSTALL_PATH ${ZEEK_SCRIPT_INSTALL_PATH}
ABSOLUTE)
set(BRO_PLUGIN_INSTALL_PATH ${BRO_ROOT_DIR}/lib/bro/plugins CACHE STRING "Installation path for plugins" FORCE)
set(BRO_PLUGIN_INSTALL_PATH ${ZEEK_ROOT_DIR}/lib/zeek/plugins CACHE STRING "Installation path for plugins" FORCE)
configure_file(bro-path-dev.in ${CMAKE_CURRENT_BINARY_DIR}/bro-path-dev)
configure_file(zeek-path-dev.in ${CMAKE_CURRENT_BINARY_DIR}/zeek-path-dev)
file(WRITE ${CMAKE_CURRENT_BINARY_DIR}/bro-path-dev.sh
"export BROPATH=`${CMAKE_CURRENT_BINARY_DIR}/bro-path-dev`\n"
"export BRO_PLUGIN_PATH=\"${CMAKE_CURRENT_BINARY_DIR}/src\":${BRO_PLUGIN_PATH}\n"
file(WRITE ${CMAKE_CURRENT_BINARY_DIR}/zeek-path-dev.sh
"export ZEEKPATH=`${CMAKE_CURRENT_BINARY_DIR}/zeek-path-dev`\n"
"export ZEEK_PLUGIN_PATH=\"${CMAKE_CURRENT_BINARY_DIR}/src\":${ZEEK_PLUGIN_PATH}\n"
"export PATH=\"${CMAKE_CURRENT_BINARY_DIR}/src\":$PATH\n")
file(WRITE ${CMAKE_CURRENT_BINARY_DIR}/bro-path-dev.csh
"setenv BROPATH `${CMAKE_CURRENT_BINARY_DIR}/bro-path-dev`\n"
"setenv BRO_PLUGIN_PATH \"${CMAKE_CURRENT_BINARY_DIR}/src\":${BRO_PLUGIN_PATH}\n"
file(WRITE ${CMAKE_CURRENT_BINARY_DIR}/zeek-path-dev.csh
"setenv ZEEKPATH `${CMAKE_CURRENT_BINARY_DIR}/zeek-path-dev`\n"
"setenv ZEEK_PLUGIN_PATH \"${CMAKE_CURRENT_BINARY_DIR}/src\":${ZEEK_PLUGIN_PATH}\n"
"setenv PATH \"${CMAKE_CURRENT_BINARY_DIR}/src\":$PATH\n")
file(STRINGS "${CMAKE_CURRENT_SOURCE_DIR}/VERSION" VERSION LIMIT_COUNT 1)
@ -72,6 +72,40 @@ if(${ENABLE_DEBUG})
set(VERSION_C_IDENT "${VERSION_C_IDENT}_debug")
endif()
if ( NOT BINARY_PACKAGING_MODE )
macro(_make_install_dir_symlink _target _link)
install(CODE "
if ( \"\$ENV{DESTDIR}\" STREQUAL \"\" )
if ( EXISTS \"${_target}\" AND NOT EXISTS \"${_link}\" )
message(STATUS \"WARNING: installed ${_link} as symlink to ${_target}\")
execute_process(COMMAND \"${CMAKE_COMMAND}\" -E create_symlink
\"${_target}\" \"${_link}\")
endif ()
endif ()
")
endmacro()
if ( "${CMAKE_INSTALL_PREFIX}" STREQUAL "/usr/local/zeek" )
# If we're installing into the default prefix, check if the
# old default prefix already exists and symlink to it.
# This is done to help keep custom user configuration/installation
# if they're upgrading from a version before Zeek 3.0.
_make_install_dir_symlink("/usr/local/bro" "/usr/local/zeek")
endif ()
# Check whether we need to symlink directories used by versions
# before Zeek 3.0.
_make_install_dir_symlink("${CMAKE_INSTALL_PREFIX}/include/bro" "${CMAKE_INSTALL_PREFIX}/include/zeek")
_make_install_dir_symlink("${CMAKE_INSTALL_PREFIX}/share/bro" "${CMAKE_INSTALL_PREFIX}/share/zeek")
_make_install_dir_symlink("${CMAKE_INSTALL_PREFIX}/lib/bro" "${CMAKE_INSTALL_PREFIX}/lib/zeek")
endif ()
if ( SANITIZERS )
set(CMAKE_C_FLAGS "${CMAKE_C_FLAGS} -fsanitize=${SANITIZERS} -fno-omit-frame-pointer")
set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -fsanitize=${SANITIZERS} -fno-omit-frame-pointer")
set(CMAKE_LD_FLAGS "${CMAKE_LD_FLAGS} -fsanitize=${SANITIZERS} -fno-omit-frame-pointer")
endif()
########################################################################
## Dependency Configuration
@ -211,7 +245,7 @@ if ( ${CMAKE_SYSTEM_NAME} MATCHES Linux AND EXISTS /etc/os-release )
endif ()
endif ()
set(brodeps
set(zeekdeps
${BinPAC_LIBRARY}
${PCAP_LIBRARY}
${OPENSSL_LIBRARIES}
@ -241,49 +275,56 @@ include(GetArchitecture)
include(RequireCXX11)
if ( (OPENSSL_VERSION VERSION_EQUAL "1.1.0") OR (OPENSSL_VERSION VERSION_GREATER "1.1.0") )
set(BRO_HAVE_OPENSSL_1_1 true CACHE INTERNAL "" FORCE)
set(ZEEK_HAVE_OPENSSL_1_1 true CACHE INTERNAL "" FORCE)
endif()
# Tell the plugin code that we're building as part of the main tree.
set(BRO_PLUGIN_INTERNAL_BUILD true CACHE INTERNAL "" FORCE)
set(ZEEK_PLUGIN_INTERNAL_BUILD true CACHE INTERNAL "" FORCE)
set(DEFAULT_BROPATH .:${BRO_SCRIPT_INSTALL_PATH}:${BRO_SCRIPT_INSTALL_PATH}/policy:${BRO_SCRIPT_INSTALL_PATH}/site)
set(DEFAULT_ZEEKPATH .:${ZEEK_SCRIPT_INSTALL_PATH}:${ZEEK_SCRIPT_INSTALL_PATH}/policy:${ZEEK_SCRIPT_INSTALL_PATH}/site)
if ( NOT BINARY_PACKAGING_MODE )
set(BRO_DIST ${CMAKE_SOURCE_DIR})
set(ZEEK_DIST ${CMAKE_SOURCE_DIR})
endif ()
string(TOLOWER ${CMAKE_BUILD_TYPE} CMAKE_BUILD_TYPE_LOWER)
configure_file(${CMAKE_CURRENT_SOURCE_DIR}/bro-config.h.in
${CMAKE_CURRENT_BINARY_DIR}/bro-config.h)
configure_file(${CMAKE_CURRENT_SOURCE_DIR}/zeek-config.h.in
${CMAKE_CURRENT_BINARY_DIR}/zeek-config.h)
include_directories(${CMAKE_CURRENT_BINARY_DIR})
install(FILES ${CMAKE_CURRENT_BINARY_DIR}/bro-config.h DESTINATION include/bro)
install(FILES ${CMAKE_CURRENT_BINARY_DIR}/zeek-config.h DESTINATION include/zeek)
if ( CAF_ROOT_DIR )
set(BRO_CONFIG_CAF_ROOT_DIR ${CAF_ROOT_DIR})
set(ZEEK_CONFIG_CAF_ROOT_DIR ${CAF_ROOT_DIR})
else ()
set(BRO_CONFIG_CAF_ROOT_DIR ${BRO_ROOT_DIR})
set(ZEEK_CONFIG_CAF_ROOT_DIR ${ZEEK_ROOT_DIR})
endif ()
if ( BinPAC_ROOT_DIR )
set(BRO_CONFIG_BINPAC_ROOT_DIR ${BinPAC_ROOT_DIR})
set(ZEEK_CONFIG_BINPAC_ROOT_DIR ${BinPAC_ROOT_DIR})
else ()
set(BRO_CONFIG_BINPAC_ROOT_DIR ${BRO_ROOT_DIR})
set(ZEEK_CONFIG_BINPAC_ROOT_DIR ${ZEEK_ROOT_DIR})
endif ()
if ( BROKER_ROOT_DIR )
set(BRO_CONFIG_BROKER_ROOT_DIR ${BROKER_ROOT_DIR})
set(ZEEK_CONFIG_BROKER_ROOT_DIR ${BROKER_ROOT_DIR})
else ()
set(BRO_CONFIG_BROKER_ROOT_DIR ${BRO_ROOT_DIR})
set(ZEEK_CONFIG_BROKER_ROOT_DIR ${ZEEK_ROOT_DIR})
endif ()
configure_file(${CMAKE_CURRENT_SOURCE_DIR}/bro-config.in
${CMAKE_CURRENT_BINARY_DIR}/bro-config @ONLY)
install(PROGRAMS ${CMAKE_CURRENT_BINARY_DIR}/bro-config DESTINATION bin)
configure_file(${CMAKE_CURRENT_SOURCE_DIR}/zeek-config.in
${CMAKE_CURRENT_BINARY_DIR}/zeek-config @ONLY)
install(PROGRAMS ${CMAKE_CURRENT_BINARY_DIR}/zeek-config DESTINATION bin)
install(DIRECTORY ${CMAKE_CURRENT_SOURCE_DIR}/cmake DESTINATION share/bro
install(DIRECTORY ${CMAKE_CURRENT_SOURCE_DIR}/cmake DESTINATION share/zeek
USE_SOURCE_PERMISSIONS)
# Install wrapper script for Bro-to-Zeek renaming.
include(InstallShellScript)
include(InstallSymlink)
InstallShellScript("bin" "zeek-wrapper.in" "zeek-wrapper")
InstallSymlink("${CMAKE_INSTALL_PREFIX}/bin/zeek-wrapper" "${CMAKE_INSTALL_PREFIX}/bin/bro-config")
InstallSymlink("${CMAKE_INSTALL_PREFIX}/include/zeek/zeek-config.h" "${CMAKE_INSTALL_PREFIX}/include/zeek/bro-config.h")
########################################################################
## Recurse on sub-directories
@ -291,7 +332,7 @@ if ( BROKER_ROOT_DIR )
find_package(Broker REQUIRED)
find_package(CAF COMPONENTS core io openssl REQUIRED)
set(brodeps ${brodeps} ${BROKER_LIBRARY} ${CAF_LIBRARIES})
set(zeekdeps ${zeekdeps} ${BROKER_LIBRARY} ${CAF_LIBRARIES})
include_directories(BEFORE ${BROKER_INCLUDE_DIR})
else ()
set(ENABLE_STATIC_ONLY_SAVED ${ENABLE_STATIC_ONLY})
@ -304,9 +345,9 @@ else ()
set(ENABLE_STATIC_ONLY ${ENABLE_STATIC_ONLY_SAVED})
if ( BUILD_STATIC_BROKER )
set(brodeps ${brodeps} broker_static)
set(zeekdeps ${zeekdeps} broker_static)
else()
set(brodeps ${brodeps} broker)
set(zeekdeps ${zeekdeps} broker)
endif()
include_directories(BEFORE ${CMAKE_CURRENT_SOURCE_DIR}/aux/broker
${CMAKE_CURRENT_BINARY_DIR}/aux/broker)
@ -318,20 +359,23 @@ include_directories(BEFORE ${CAF_INCLUDE_DIR_CORE})
include_directories(BEFORE ${CAF_INCLUDE_DIR_IO})
include_directories(BEFORE ${CAF_INCLUDE_DIR_OPENSSL})
add_subdirectory(aux/paraglob)
set(zeekdeps ${zeekdeps} paraglob)
include_directories(BEFORE ${CMAKE_CURRENT_SOURCE_DIR}/aux/paraglob)
add_subdirectory(src)
add_subdirectory(scripts)
add_subdirectory(man)
include(CheckOptionalBuildSources)
CheckOptionalBuildSources(aux/broctl Broctl INSTALL_BROCTL)
CheckOptionalBuildSources(aux/bro-aux Bro-Aux INSTALL_AUX_TOOLS)
CheckOptionalBuildSources(aux/broccoli Broccoli INSTALL_BROCCOLI)
CheckOptionalBuildSources(aux/zeekctl ZeekControl INSTALL_ZEEKCTL)
CheckOptionalBuildSources(aux/zeek-aux Zeek-Aux INSTALL_AUX_TOOLS)
########################################################################
## Packaging Setup
if (INSTALL_BROCTL)
if (INSTALL_ZEEKCTL)
# CPack RPM Generator may not automatically detect this
set(CPACK_RPM_PACKAGE_REQUIRES "python >= 2.6.0")
endif ()
@ -352,12 +396,12 @@ if (CMAKE_BUILD_TYPE)
endif ()
message(
"\n====================| Bro Build Summary |====================="
"\n====================| Zeek Build Summary |===================="
"\n"
"\nBuild type: ${CMAKE_BUILD_TYPE}"
"\nBuild dir: ${CMAKE_BINARY_DIR}"
"\nInstall prefix: ${CMAKE_INSTALL_PREFIX}"
"\nBro Script Path: ${BRO_SCRIPT_INSTALL_PATH}"
"\nZeek Script Path: ${ZEEK_SCRIPT_INSTALL_PATH}"
"\nDebug mode: ${ENABLE_DEBUG}"
"\n"
"\nCC: ${CMAKE_C_COMPILER}"
@ -366,8 +410,7 @@ message(
"\nCXXFLAGS: ${CMAKE_CXX_FLAGS} ${CMAKE_CXX_FLAGS_${BuildType}}"
"\nCPP: ${CMAKE_CXX_COMPILER}"
"\n"
"\nBroccoli: ${INSTALL_BROCCOLI}"
"\nBroctl: ${INSTALL_BROCTL}"
"\nZeekControl: ${INSTALL_ZEEKCTL}"
"\nAux. Tools: ${INSTALL_AUX_TOOLS}"
"\n"
"\nlibmaxminddb: ${USE_GEOIP}"

4
Makefile
View file

@ -55,9 +55,9 @@ test:
-@( cd testing && make )
test-aux:
-test -d aux/broctl && ( cd aux/broctl && make test-all )
-test -d aux/zeekctl && ( cd aux/zeekctl && make test-all )
-test -d aux/btest && ( cd aux/btest && make test )
-test -d aux/bro-aux && ( cd aux/bro-aux && make test )
-test -d aux/zeek-aux && ( cd aux/zeek-aux && make test )
-test -d aux/plugins && ( cd aux/plugins && make test-all )
test-all: test test-aux

342
NEWS
View file

@ -1,10 +1,10 @@
This document summarizes the most important changes in the current Bro
This document summarizes the most important changes in the current Zeek
release. For an exhaustive list of changes, see the ``CHANGES`` file
(note that submodules, such as Broker, come with their own ``CHANGES``.)
Bro 2.7
=======
Zeek 3.0.0
==========
New Functionality
-----------------
@ -18,7 +18,10 @@ New Functionality
- dns_NSEC
- dns_NSEC3
- Bro's Plugin framework now allows a patch version. If a patch version is not
- Added support for parsing and logging DNS SPF resource records.
A new ``dns_SPF_reply`` event is also available.
- Zeek's Plugin framework now allows a patch version. If a patch version is not
provided, it will default to 0. To specify this, modify the plugin
Configuration class in your ``src/Plugin.cc`` and set
``config.version.patch``. Note that the default plugin skeleton
@ -68,9 +71,103 @@ New Functionality
- Added a new event for weirdness found via file analysis: ``file_weird``.
- The conn.log "history" field supports a new character 'G' or 'g'
(capital for originator, lowercase responder) to indicate a content
gap in the TCP stream. These are recorded logarithmically.
- The ``ZEEK_DNS_RESOLVER`` environment variable now controls
the DNS resolver to use by setting it to an IPv4 or IPv6 address. If
not set, then the first IPv4 address from /etc/resolv.conf gets used.
- The ``/<re>/i`` convenience syntax for case-insensitive patterns is now
also allowed when specifying patterns used in signature files.
- New RDP functionality.
- New events:
- rdp_client_network_data
- rdp_client_security_data
- rdp_client_cluster_data
- rdp_native_encrypted_data
- Add a new "client_channels" field to rdp.log based on data parsed from
the Client Network Data (TS_UD_CS_NET) packet. The channel list is also
available in the new ``rdp_client_network_data`` event.
- Add parsing support for TLS 1.3 pre-shared key extension. This info
is available in the events: ``ssl_extension_pre_shared_key_client_hello``
and ``ssl_extension_pre_shared_key_server_hello``.
- Added/re-wrote support for NTP.
- Parsing support for modes 1-7, with parsed structures available in
the ``ntp_message`` event.
- An ntp.log is produced by default, containing data extracted from
NTP messages with modes 1-5.
- Add support for vector slicing operations. For example::
local v = vector(1, 2, 3, 4, 5);
v[2:4] = vector(6, 7, 8); # v is now [1, 2, 6, 7, 8, 5]
print v[:4]; # prints [1, 2, 6, 7]
- Add support for paraglob, a fairly quick data structure for matching a string
against a large list of patterns. For example::
local v1 = vector("*", "d?g", "*og", "d?", "d[!wl]g");
local p1 = paraglob_init(v1);
print paraglob_match(p1, "dog");
Changed Functionality
---------------------
- The following executable names have changed (the old names will
continue to work, but emit a deprecation warning):
- ``bro`` is now ``zeek``
- ``bro-config`` is now ``zeek-config``
- ``broctl`` is now ``zeekctl``
- ``bro-cut`` is now ``zeek-cut``
- BroControl has been completely renamed to ZeekControl. Many installation
directories and files with "broctl" in their name have been changed
to use "zeekctl" instead. It's expected this has been done in a way
that's backwards compatible with previous Bro installations. E.g.
if you made customizations to the ``broctl.cfg`` file of a previous
installation, installing the newer Zeek version over it will retain that
file and even symlink the new ``zeekctl.cfg`` to it.
- The default install prefix is now ``/usr/local/zeek`` instead of
``/usr/local/bro``. If you have an existing installation that used
the previous default and are still using the new default when upgrading,
we'll crate ``/usr/local/zeek`` as a symlink to ``/usr/local/bro``.
Certain subdirectories will also get similar treatment: ``share/bro``,
``include/bro``, and ``lib/bro``.
- ``$prefix/share/bro/site/local.bro`` has been renamed to
``local.zeek``. If you have a ``local.bro`` file from a previous
installation, possibly with customizations made to it, the new
version of Zeek will install a ``local.zeek`` file that is a symlink
to the pre-existing ``local.bro``. In that case, you may want to
just copy ``local.bro`` into the new ``local.zeek`` location to
avoid confusion, but things are otherwise meant to work properly
without intervention.
- All scripts ending in ``.bro`` that ship with the Zeek source tree have
been renamed to ``.zeek``.
- The search logic for the ``@load`` script directive now prefers files
ending in ``.zeek``, but will fallback to loading a ``.bro`` file if
it exists. E.g. ``@load foo`` will first check for a ``foo.zeek``
file to load and then otherwise ``foo.bro``. Note that
``@load foo.bro`` (with the explicit ``.bro`` file suffix) prefers
in the opposite order: it first checks for ``foo.bro`` and then
falls back to a ``foo.zeek``, if it exists.
- The for-loop index variable for vectors has been changed from
'int' to 'count' type. It's unlikely this would alter/break any
script behavior unless they were explicitly inspecting the variable's
@ -149,16 +246,249 @@ Changed Functionality
- "unknown_gre_version_%d" -> unknown_gre_version
- "unknown_gre_protocol_%u16" -> unknown_gre_protocol
- The "missed_bytes" field of conn.log can be calculated slightly differently
in some cases: ACKs that reveal a content gap, but also come at
the end of a connection (in a FIN or RST) are considered unreliable
and aren't counted as true gaps.
- The Broxygen component, which is used to generate our Doxygen-like
scripting API documentation has been renamed to Zeekygen. This likely has
no breaking or visible changes for most users, except in the case one
used it to generate their own documentation via the ``--broxygen`` flag,
which is now named ``--zeekygen``. Besides that, the various documentation
in scripts has also been updated to replace Sphinx cross-referencing roles
and directives like ":bro:see:" with ":zeek:zee:".
- The catch-and-release and unified2 scripts are no longer loaded by
default. Because there was a performance impact simply from loading
them and it's unlikely a majority of user make use of their features,
they've been moved from the scripts/base/ directory into
scripts/policy/ and must be manually loaded to use their
functionality. The "drop" action for the notice framework is likewise
moved since it was implemented via catch-and-release. As a result,
the default notice.log no longer contains a "dropped" field.
If you previously used the catch-and-release functionality add this:
@load policy/frameworks/netcontrol/catch-and-release
If you previously used Notice::ACTION_DROP add:
@load policy/frameworks/notice/actions/drop
If you previously used the Unified2 file analysis support add:
@load policy/files/unified2
- The default value of ``peer_description`` has changed from "bro"
to "zeek". This won't effect most users, except for the fact that
this value may appear in several log files, so any external plugins
that have written unit tests that compare baselines of such log
files may need to be updated.
- The "remote_ip" field of "addr" type was removed from radius.log and
replaced with a field named "tunnel_client" of "string" type. The
reason for this is that the Tunnel-Client-Endpoint RADIUS attribute
this data is derived from may also be a FQDN, not just an IP address.
- The ``ssl_server_hello`` event's ``server_random`` parameter has been
changed to always include the full 32-byte field from the
ServerHello. Previously a 4-byte timestamp and 28-byte random data
were parsed separately as some TLS protocol versions specified a
separate timestamp field as part of the full 32-byte random sequence.
- The namespace used by all the builtin plugins that ship with Zeek have
changed to use "Zeek::" instead of "Bro::".
- Any Broker topic names used in scripts shipped with Zeek that
previously were prefixed with "bro/" are now prefixed with "zeek/"
instead.
In the case where external applications were using a "bro/" topic
to send data into a Bro process, a Zeek process still subscribes
to those topics in addition to the equivalently named "zeek/" topic.
In the case where external applications were using a "bro/" topic
to subscribe to remote messages or query data stores, there's no
backwards compatibility and external applications must be changed
to use the new "zeek/" topic. The thought is this change will have
low impact since most data published under "bro/" topic names is
intended for use only as a detail of implementing cluster-enabled
versions of various scripts.
A list of the most relevant/common topic names that could potentially
be used in external applications to consume/query remote data that
one may need to change:
- store names
- bro/known/services
- bro/known/hosts
- bro/known/certs
- cluster nodes
- bro/cluster/<node type>
- bro/cluster/node/<name>
- bro/cluster/nodeid/<id>
- logging
- bro/logs/<stream>
- The ``resp_ref`` argument was removed from the ``ocsp_response_bytes``
event. ``resp_ref`` was not used by anything in the codebase and could not be
passed to any other functions for further processing. The remainder of the
``ocsp_response_bytes`` is unchanged.
Removed Functionality
---------------------
- A number of functions that were deprecated in version 2.6 or below and completely
removed from this release. Most of the functions were used for the old communication
code.
- ``find_ip_addresses``
- ``cat_string_array``
- ``cat_string_array_n``
- ``complete_handshake``
- ``connect``
- ``decode_base64_custom``
- ``disconnect``
- ``enable_communication``
- ``encode_base64_custom``
- ``get_event_peer``
- ``get_local_event_peer``
- ``join_string_array``
- ``listen``
- ``merge_pattern``
- ``request_remote_events``
- ``request_remote_logs``
- ``request_remote_sync``
- ``resume_state_updates``
- ``send_capture_filter``
- ``send_current_packet``
- ``send_id``
- ``send_ping``
- ``set_accept_state``
- ``set_compression_level``
- ``sort_string_array``
- ``split1``
- ``split_all``
- ``split``
- ``suspend_state_updates``
- ``terminate_communication``
- ``split``
- ``send_state``
- ``checkpoint_state``
- ``rescan_state``
- The following events were deprecated in version 2.6 or below and are completely
removed from this release:
- ``ssl_server_curve``
- ``dhcp_ack``
- ``dhcp_decline``
- ``dhcp_discover``
- ``dhcp_inform``
- ``dhcp_nak``
- ``dhcp_offer``
- ``dhcp_release``
- ``dhcp_request``
- ``remote_state_access_performed``
- ``remote_state_inconsistency``
- ``remote_connection_established``
- ``remote_connection_closed``
- ``remote_connection_handshake_done``
- ``remote_event_registered``
- ``remote_connection_error``
- ``remote_capture_filter``
- ``remote_log_peer``
- ``remote_log``
- ``finished_send_state``
- ``remote_pong``
- The following types/records were deprecated in version 2.6 or below and are
removed from this release:
- ``peer_id``
- ``event_peer``
- The following configuration options were deprecated in version 2.6 or below and are
removed from this release:
- ``max_remote_events_processed``
- ``forward_remote_events``
- ``forward_remote_state_changes``
- ``enable_syslog``
- ``remote_trace_sync_interval``
- ``remote_trace_sync_peers``
- ``remote_check_sync_consistency``
- ``log_rotate_interval``
- ``log_max_size``
- ``log_encryption_key``
- ``state_dir``
- ``state_write_delay``
- ``ssl_ca_certificate``
- ``ssl_private_key``
- ``ssl_passphrase``
- The following constants were used as part of deprecated functionality in version 2.6
or below and are removed from this release:
- ``PEER_ID_NONE``
- ``REMOTE_LOG_INFO``
- ``REMOTE_SRC_CHILD``
- ``REMOTE_SRC_PARENT``
- ``REMOTE_SRC_SCRIPT``
- The deprecated script ``policy/protocols/smb/__load__.bro`` was removed.
Instead of ``@load policy/protocols/smb`` use ``@load base/protocols/smb``.
- Broccoli, which had been deprecated in version 2.6 and was no longer built by default
was removed from the source tree.
- Support for the &persistent, &synchronized, &mergeable, &encrypt, &rotate_interval,
and &rotate_size attributes, which were deprecated in Bro 2.6, was removed. The ``-g``
command-line option (dump-config) which relied on this functionality was also removed.
- Functionality for writing state updates for variables with the
&synchronized attribute was removed. This entails the
``-x`` command-line option (print-state) as well as the
``capture_state_updates`` function.
- Removed the BroControl ``update`` command, which was deprecated in Bro 2.6.
- Functionality for writing/reading binary event streams was
removed. This functionality relied on the old communication code
anc was basically untested. The ``-R`` command-line option (replay)
as well as the ``capture_events`` function were removed.
- Removed p0f (passive OS fingerprinting) support. The version of
p0f shipped with zeek was ancient, probably did not give
any reliable support anymore and did not offer a clear
upgrade path. The ``OS_version_found`` event as well as the
``generate_OS_version_event`` configuration option were removed.
Deprecated Functionality
------------------------
- The ``str_shell_escape` function is now deprecated, use ``safe_shell_quote``
- The ``str_shell_escape`` function is now deprecated, use ``safe_shell_quote``
instead. The later will automatically return a value that is enclosed
in double-quotes.
- The ``bro_init``, ``bro_done``, and ``bro_script_loaded`` events are now
deprecated, use ``zeek_init``, ``zeek_done``, and
``zeek_script_loaded`` instead. Any existing event handlers for
the deprecated versions will automatically alias to the new events
such that existing code will not break, but will emit a deprecation
warning.
- The ``bro_is_terminating`` and ``bro_version`` function are deprecated and
replaced by functions named ``zeek_is_terminating`` and ``zeek_version``.
- The ``rotate_file``, ``rotate_file_by_name`` and ``calc_next_rotate`` functions
were marked as deprecated. These functions were used with the old pre-2.0 logging
framework and are no longer used. They also were marked as deprecated in their
documentation, however the functions themselves did not carry the deprecation marker.
Bro 2.6
=======
@ -530,7 +860,7 @@ New Functionality
Each has the same form, e.g.::
event tcp_multiple_retransmissions(c: connection, is_orig: bool,
threshold: count);
threshold: count);
- Added support for set union, intersection, difference, and comparison
operations. The corresponding operators for the first three are

2
VERSION
View file

@ -1 +1 @@
2.6-200
2.6-526

2
aux/bifcl

@ -1 +1 @@
Subproject commit 44622332fb1361383799be33e365704caacce199
Subproject commit 699ffb13c986aca599b70735b368a515c2149982

2
aux/binpac

@ -1 +1 @@
Subproject commit bb2476465e304a00c368bd73d40cc6f734be5311
Subproject commit baabe22a2b8a68fac448e862e1c2acc46f89c5fc

1
aux/broccoli

@ -1 +0,0 @@
Subproject commit 41841d8f64bdb062860309f7b36513212e81befa

1
aux/broctl

@ -1 +0,0 @@
Subproject commit afc0260abf663f4b44d535d66d378fde7b0d5206

1
aux/broctl Symbolic link
View file

@ -0,0 +1 @@
zeekctl

2
aux/broker

@ -1 +1 @@
Subproject commit 7dab576984dee1f58fe5ceb81f36b63128d58860
Subproject commit 3f827567edca20eb0fe9ad071519f305699296ea

2
aux/btest

@ -1 +1 @@
Subproject commit 6ece47ba6438e7a6db5c7b85a68b3c16f0911871
Subproject commit bcda130bfae106b3d071c76cd9a3f0bde66e66da

2
aux/netcontrol-connectors

@ -1 +1 @@
Subproject commit 6501fef1fffc0b49dda59b3716b03034edcfeee6
Subproject commit 8a6f3f7c506ac483265afc77d3c1b0861db79601

1
aux/paraglob Submodule

@ -0,0 +1 @@
Subproject commit 1a8d674d2ccbef06a6e4e6f1a9c8747a2eadf026

2
aux/zeek-aux

@ -1 +1 @@
Subproject commit 96c787cb396a5aad2d3ea3b2087f2a1fcd6b7216
Subproject commit e0689c1c9565ba7ffcab011e9f22f6a17a67e40a

1
aux/zeekctl Submodule

@ -0,0 +1 @@
Subproject commit b6642b80f1436751884e0dd12fa16930fabc66ca

239
bro-config.h.in
View file

@ -1,239 +0,0 @@
/* Old libpcap versions (< 0.6.1) need defining pcap_freecode and
pcap_compile_nopcap */
#cmakedefine DONT_HAVE_LIBPCAP_PCAP_FREECODE
/* should explicitly declare socket() and friends */
#cmakedefine DO_SOCK_DECL
/* Define if you have the <getopt.h> header file. */
#cmakedefine HAVE_GETOPT_H
/* Define if you have the `getopt_long' function. */
#cmakedefine HAVE_GETOPT_LONG
/* We are on a Linux system */
#cmakedefine HAVE_LINUX
/* We are on a Mac OS X (Darwin) system */
#cmakedefine HAVE_DARWIN
/* Define if you have the `mallinfo' function. */
#cmakedefine HAVE_MALLINFO
/* Define if you have the <memory.h> header file. */
#cmakedefine HAVE_MEMORY_H
/* Define if you have the <netinet/ether.h> header file */
#cmakedefine HAVE_NETINET_ETHER_H
/* Define if you have the <netinet/if_ether.h> header file. */
#cmakedefine HAVE_NETINET_IF_ETHER_H
/* Define if you have the <netinet/ip6.h> header file. */
#cmakedefine HAVE_NETINET_IP6_H
/* Define if you have the <net/ethernet.h> header file. */
#cmakedefine HAVE_NET_ETHERNET_H
/* Define if you have the <net/ethertypes.h> header file. */
#cmakedefine HAVE_NET_ETHERTYPES_H
/* have os-proto.h */
#cmakedefine HAVE_OS_PROTO_H
/* Define if you have the <pcap-int.h> header file. */
#cmakedefine HAVE_PCAP_INT_H
/* line editing & history powers */
#cmakedefine HAVE_READLINE
/* Define if you have the `sigaction' function. */
#cmakedefine HAVE_SIGACTION
/* Define if you have the `sigset' function. */
#cmakedefine HAVE_SIGSET
/* Define if you have the `strcasestr' function. */
#cmakedefine HAVE_STRCASESTR
/* Define if you have the `strerror' function. */
#cmakedefine HAVE_STRERROR
/* Define if you have the `strsep' function. */
#cmakedefine HAVE_STRSEP
/* Define if you have the <sys/ethernet.h> header file. */
#cmakedefine HAVE_SYS_ETHERNET_H
/* Some libpcap versions use an extra parameter (error) in pcap_compile_nopcap
*/
#cmakedefine LIBPCAP_PCAP_COMPILE_NOPCAP_HAS_ERROR_PARAMETER
/* Include krb5.h */
#cmakedefine NEED_KRB5_H
/* Compatibility for Darwin */
#cmakedefine NEED_NAMESER_COMPAT_H
/* d2i_x509 uses const char** */
#cmakedefine OPENSSL_D2I_X509_USES_CONST_CHAR
/* Define as the return type of signal handlers (`int' or `void'). */
#define RETSIGTYPE @RETSIGTYPE@
/* signal function return value */
#define RETSIGVAL @RETSIGVAL@
/* have sin_len field in sockaddr_in */
#cmakedefine SIN_LEN
/* The size of `long int', as computed by sizeof. */
#define SIZEOF_LONG_INT @SIZEOF_LONG_INT@
/* The size of `long long', as computed by sizeof. */
#define SIZEOF_LONG_LONG @SIZEOF_LONG_LONG@
/* The size of `void *', as computed by sizeof. */
#define SIZEOF_VOID_P @SIZEOF_VOID_P@
/* should we declare syslog() and openlog() */
#cmakedefine SYSLOG_INT
/* Define if you have <sys/time.h> */
#cmakedefine HAVE_SYS_TIME_H
/* Define if you can safely include both <sys/time.h> and <time.h>. */
#cmakedefine TIME_WITH_SYS_TIME
/* GeoIP geographic lookup functionality */
#cmakedefine USE_GEOIP
/* Define if KRB5 is available */
#cmakedefine USE_KRB5
/* Use Google's perftools */
#cmakedefine USE_PERFTOOLS_DEBUG
/* Analyze Mobile IPv6 traffic */
#cmakedefine ENABLE_MOBILE_IPV6
/* Use libCurl. */
#cmakedefine USE_CURL
/* Use the DataSeries writer. */
#cmakedefine USE_DATASERIES
/* Use the ElasticSearch writer. */
#cmakedefine USE_ELASTICSEARCH
/* Version number of package */
#define VERSION "@VERSION@"
/* whether words are stored with the most significant byte first */
#cmakedefine WORDS_BIGENDIAN
/* whether htonll/ntohll is defined in <arpa/inet.h> */
#cmakedefine HAVE_BYTEORDER_64
/* ultrix can't hack const */
#cmakedefine NEED_ULTRIX_CONST_HACK
#ifdef NEED_ULTRIX_CONST_HACK
#define const
#endif
/* Define int32_t */
#cmakedefine int32_t @int32_t@
/* use sigset() instead of signal() */
#ifdef HAVE_SIGSET
#define signal sigset
#endif
/* define to int if socklen_t not available */
#cmakedefine socklen_t @socklen_t@
/* Define u_int16_t */
#cmakedefine u_int16_t @u_int16_t@
/* Define u_int32_t */
#cmakedefine u_int32_t @u_int32_t@
/* Define u_int8_t */
#cmakedefine u_int8_t @u_int8_t@
/* OpenBSD's bpf.h may not declare this data link type, but it's supposed to be
used consistently for the same purpose on all platforms. */
#cmakedefine HAVE_DLT_PPP_SERIAL
#ifndef HAVE_DLT_PPP_SERIAL
#define DLT_PPP_SERIAL @DLT_PPP_SERIAL@
#endif
/* IPv6 Next Header values defined by RFC 3542 */
#cmakedefine HAVE_IPPROTO_HOPOPTS
#ifndef HAVE_IPPROTO_HOPOPTS
#define IPPROTO_HOPOPTS 0
#endif
#cmakedefine HAVE_IPPROTO_IPV6
#ifndef HAVE_IPPROTO_IPV6
#define IPPROTO_IPV6 41
#endif
#cmakedefine HAVE_IPPROTO_IPV4
#ifndef HAVE_IPPROTO_IPV4
#define IPPROTO_IPV4 4
#endif
#cmakedefine HAVE_IPPROTO_ROUTING
#ifndef HAVE_IPPROTO_ROUTING
#define IPPROTO_ROUTING 43
#endif
#cmakedefine HAVE_IPPROTO_FRAGMENT
#ifndef HAVE_IPPROTO_FRAGMENT
#define IPPROTO_FRAGMENT 44
#endif
#cmakedefine HAVE_IPPROTO_ESP
#ifndef HAVE_IPPROTO_ESP
#define IPPROTO_ESP 50
#endif
#cmakedefine HAVE_IPPROTO_AH
#ifndef HAVE_IPPROTO_AH
#define IPPROTO_AH 51
#endif
#cmakedefine HAVE_IPPROTO_ICMPV6
#ifndef HAVE_IPPROTO_ICMPV6
#define IPPROTO_ICMPV6 58
#endif
#cmakedefine HAVE_IPPROTO_NONE
#ifndef HAVE_IPPROTO_NONE
#define IPPROTO_NONE 59
#endif
#cmakedefine HAVE_IPPROTO_DSTOPTS
#ifndef HAVE_IPPROTO_DSTOPTS
#define IPPROTO_DSTOPTS 60
#endif
/* IPv6 options structure defined by RFC 3542 */
#cmakedefine HAVE_IP6_OPT
/* Common IPv6 extension structure */
#cmakedefine HAVE_IP6_EXT
/* String with host architecture (e.g., "linux-x86_64") */
#define HOST_ARCHITECTURE "@HOST_ARCHITECTURE@"
/* String with extension of dynamic libraries (e.g., ".so") */
#define DYNAMIC_PLUGIN_SUFFIX "@CMAKE_SHARED_MODULE_SUFFIX@"
/* True if we're building outside of the main Bro source code tree. */
#ifndef BRO_PLUGIN_INTERNAL_BUILD
#define BRO_PLUGIN_INTERNAL_BUILD @BRO_PLUGIN_INTERNAL_BUILD@
#endif
/* A C function that has the Bro version encoded into its name. */
#define BRO_VERSION_FUNCTION bro_version_@VERSION_C_IDENT@
#ifdef __cplusplus
extern "C" {
#endif
extern const char* BRO_VERSION_FUNCTION();
#ifdef __cplusplus
}
#endif

87
bro-config.in
View file

@ -1,87 +0,0 @@
#!/bin/sh
version=@VERSION@
build_type=@CMAKE_BUILD_TYPE_LOWER@
prefix=@CMAKE_INSTALL_PREFIX@
script_dir=@BRO_SCRIPT_INSTALL_PATH@
site_dir=@BRO_SCRIPT_INSTALL_PATH@/site
plugin_dir=@BRO_PLUGIN_INSTALL_PATH@
config_dir=@BRO_ETC_INSTALL_DIR@
python_dir=@PY_MOD_INSTALL_DIR@
cmake_dir=@CMAKE_INSTALL_PREFIX@/share/bro/cmake
include_dir=@CMAKE_INSTALL_PREFIX@/include/bro
bropath=@DEFAULT_BROPATH@
bro_dist=@BRO_DIST@
binpac_root=@BRO_CONFIG_BINPAC_ROOT_DIR@
caf_root=@BRO_CONFIG_CAF_ROOT_DIR@
broker_root=@BRO_CONFIG_BROKER_ROOT_DIR@
usage="\
Usage: bro-config [--version] [--build_type] [--prefix] [--script_dir] [--site_dir] [--plugin_dir] [--config_dir] [--python_dir] [--include_dir] [--cmake_dir] [--bropath] [--bro_dist] [--binpac_root] [--caf_root] [--broker_root]"
if [ $# -eq 0 ] ; then
echo "${usage}" 1>&2
exit 1
fi
while [ $# -ne 0 ]; do
case "$1" in
-*=*) optarg=`echo "$1" | sed 's/[-_a-zA-Z0-9]*=//'` ;;
*) optarg= ;;
esac
case $1 in
--version)
echo $version
;;
--prefix)
echo $prefix
;;
--build_type)
echo $build_type
;;
--script_dir)
echo $script_dir
;;
--site_dir)
echo $site_dir
;;
--plugin_dir)
echo $plugin_dir
;;
--config_dir)
echo $config_dir
;;
--python_dir)
echo $python_dir
;;
--cmake_dir)
echo $cmake_dir
;;
--include_dir)
echo $include_dir
;;
--bropath)
echo $bropath
;;
--bro_dist)
echo $bro_dist
;;
--binpac_root)
echo $binpac_root
;;
--caf_root)
echo $caf_root
;;
--broker_root)
echo $broker_root
;;
*)
echo "${usage}" 1>&2
exit 1
;;
esac
shift
done
exit 0

13
bro-path-dev.in
View file

@ -1,13 +0,0 @@
#!/bin/sh
# After configured by CMake, this file prints the absolute path to Bro scripts
# that come with the source distributions of Bro as well as scripts that are
# generated by the BIF compiler at compile time.
#
# The intended use of this script is to make it easier to run Bro from
# the build directory, avoiding the need to install it. This could be
# done like:
#
# BROPATH=`./bro-path-dev` ./src/bro
#
echo .:${CMAKE_SOURCE_DIR}/scripts:${CMAKE_SOURCE_DIR}/scripts/policy:${CMAKE_SOURCE_DIR}/scripts/site:${CMAKE_BINARY_DIR}/scripts

1
bro-path-dev.in Symbolic link
View file

@ -0,0 +1 @@
zeek-path-dev.in

2
cmake

@ -1 +1 @@
Subproject commit 0c1ee634a8f915e738da72c797a17aad9cb618dd
Subproject commit 58e4eebe3aebd0cf608e51046805a9ab1ffa6c1b

79
configure vendored
View file

@ -31,15 +31,15 @@ Usage: $0 [OPTION]... [VAR=VALUE]...
(useful for cross-compiling)
Installation Directories:
--prefix=PREFIX installation directory [/usr/local/bro]
--scriptdir=PATH root installation directory for Bro scripts
[PREFIX/share/bro]
--localstatedir=PATH when using BroControl, path to store log files
--prefix=PREFIX installation directory [/usr/local/zeek]
--scriptdir=PATH root installation directory for Zeek scripts
[PREFIX/share/zeek]
--localstatedir=PATH when using ZeekControl, path to store log files
and run-time data (within log/ and spool/ subdirs)
[PREFIX]
--spooldir=PATH when using BroControl, path to store run-time data
--spooldir=PATH when using ZeekControl, path to store run-time data
[PREFIX/spool]
--logdir=PATH when using BroControl, path to store log file
--logdir=PATH when using ZeekControl, path to store log file
[PREFIX/logs]
--conf-files-dir=PATH config files installation directory [PREFIX/etc]
@ -51,14 +51,14 @@ Usage: $0 [OPTION]... [VAR=VALUE]...
(automatically on when perftools is present on Linux)
--enable-perftools-debug use Google's perftools for debugging
--enable-jemalloc link against jemalloc
--enable-broccoli build or install the Broccoli library (deprecated)
--enable-static-broker build broker statically (ignored if --with-broker is specified)
--enable-static-broker build Broker statically (ignored if --with-broker is specified)
--enable-static-binpac build binpac statically (ignored if --with-binpac is specified)
--disable-broctl don't install Broctl
--disable-zeekctl don't install ZeekControl
--disable-auxtools don't build or install auxiliary tools
--disable-perftools don't try to build with Google Perftools
--disable-python don't try to build python bindings for broker
--disable-python don't try to build python bindings for Broker
--disable-broker-tests don't try to build Broker unit tests
--sanitizers=SANITIZERS comma-separated list of Clang sanitizers to enable
Required Packages in Non-Standard Locations:
--with-openssl=PATH path to OpenSSL install root
@ -66,13 +66,13 @@ Usage: $0 [OPTION]... [VAR=VALUE]...
--with-pcap=PATH path to libpcap install root
--with-binpac=PATH path to BinPAC executable
(useful for cross-compiling)
--with-bifcl=PATH path to Bro BIF compiler executable
--with-bifcl=PATH path to Zeek BIF compiler executable
(useful for cross-compiling)
--with-flex=PATH path to flex executable
--with-bison=PATH path to bison executable
--with-python=PATH path to Python executable
--with-broker=PATH path to Broker install root
(Bro uses an embedded version by default)
(Zeek uses an embedded version by default)
--with-caf=PATH path to C++ Actor Framework install root
(a Broker dependency that is embedded by default)
@ -106,6 +106,18 @@ Usage: $0 [OPTION]... [VAR=VALUE]...
sourcedir="$( cd "$( dirname "$0" )" && pwd )"
if [ ! -e "$sourcedir/cmake/COPYING" ] && [ -d "$sourcedir/.git" ]; then
echo "\
You seem to be missing the content of the cmake directory.
This typically means that you performed a non-recursive git clone of
Zeek. To check out the required subdirectories, please execute:
( cd $sourcedir && git submodule update --recursive --init )
" >&2;
exit 1;
fi
# Function to append a CMake cache entry definition to the
# CMakeCacheEntries variable.
# $1 is the cache entry variable name
@ -128,24 +140,24 @@ remove_cache_entry () {
# set defaults
builddir=build
prefix=/usr/local/bro
prefix=/usr/local/zeek
CMakeCacheEntries=""
append_cache_entry CMAKE_INSTALL_PREFIX PATH $prefix
append_cache_entry BRO_ROOT_DIR PATH $prefix
append_cache_entry PY_MOD_INSTALL_DIR PATH $prefix/lib/broctl
append_cache_entry BRO_SCRIPT_INSTALL_PATH STRING $prefix/share/bro
append_cache_entry BRO_ETC_INSTALL_DIR PATH $prefix/etc
append_cache_entry ZEEK_ROOT_DIR PATH $prefix
append_cache_entry PY_MOD_INSTALL_DIR PATH $prefix/lib/zeekctl
append_cache_entry ZEEK_SCRIPT_INSTALL_PATH STRING $prefix/share/zeek
append_cache_entry ZEEK_ETC_INSTALL_DIR PATH $prefix/etc
append_cache_entry ENABLE_DEBUG BOOL false
append_cache_entry ENABLE_PERFTOOLS BOOL false
append_cache_entry ENABLE_PERFTOOLS_DEBUG BOOL false
append_cache_entry ENABLE_JEMALLOC BOOL false
append_cache_entry BUILD_SHARED_LIBS BOOL true
append_cache_entry INSTALL_BROCCOLI BOOL false
append_cache_entry INSTALL_AUX_TOOLS BOOL true
append_cache_entry INSTALL_BROCTL BOOL true
append_cache_entry INSTALL_ZEEKCTL BOOL true
append_cache_entry CPACK_SOURCE_IGNORE_FILES STRING
append_cache_entry ENABLE_MOBILE_IPV6 BOOL false
append_cache_entry DISABLE_PERFTOOLS BOOL false
append_cache_entry SANITIZERS STRING ""
# parse arguments
while [ $# -ne 0 ]; do
@ -181,25 +193,25 @@ while [ $# -ne 0 ]; do
--prefix=*)
prefix=$optarg
append_cache_entry CMAKE_INSTALL_PREFIX PATH $optarg
append_cache_entry BRO_ROOT_DIR PATH $optarg
append_cache_entry PY_MOD_INSTALL_DIR PATH $optarg/lib/broctl
append_cache_entry ZEEK_ROOT_DIR PATH $optarg
append_cache_entry PY_MOD_INSTALL_DIR PATH $optarg/lib/zeekctl
;;
--scriptdir=*)
append_cache_entry BRO_SCRIPT_INSTALL_PATH STRING $optarg
append_cache_entry ZEEK_SCRIPT_INSTALL_PATH STRING $optarg
user_set_scriptdir="true"
;;
--conf-files-dir=*)
append_cache_entry BRO_ETC_INSTALL_DIR PATH $optarg
append_cache_entry ZEEK_ETC_INSTALL_DIR PATH $optarg
user_set_conffilesdir="true"
;;
--localstatedir=*)
append_cache_entry BRO_LOCAL_STATE_DIR PATH $optarg
append_cache_entry ZEEK_LOCAL_STATE_DIR PATH $optarg
;;
--spooldir=*)
append_cache_entry BRO_SPOOL_DIR PATH $optarg
append_cache_entry ZEEK_SPOOL_DIR PATH $optarg
;;
--logdir=*)
append_cache_entry BRO_LOG_DIR PATH $optarg
append_cache_entry ZEEK_LOG_DIR PATH $optarg
;;
--enable-coverage)
append_cache_entry ENABLE_COVERAGE BOOL true
@ -218,21 +230,20 @@ while [ $# -ne 0 ]; do
append_cache_entry ENABLE_PERFTOOLS BOOL true
append_cache_entry ENABLE_PERFTOOLS_DEBUG BOOL true
;;
--sanitizers=*)
append_cache_entry SANITIZERS STRING $optarg
;;
--enable-jemalloc)
append_cache_entry ENABLE_JEMALLOC BOOL true
;;
--enable-broccoli)
append_cache_entry DISABLE_RUBY_BINDINGS BOOL true
append_cache_entry INSTALL_BROCCOLI BOOL yes
;;
--enable-static-broker)
append_cache_entry BUILD_STATIC_BROKER BOOL true
;;
--enable-static-binpac)
append_cache_entry BUILD_STATIC_BINPAC BOOL true
;;
--disable-broctl)
append_cache_entry INSTALL_BROCTL BOOL false
--disable-zeekctl)
append_cache_entry INSTALL_ZEEKCTL BOOL false
;;
--disable-auxtools)
append_cache_entry INSTALL_AUX_TOOLS BOOL false
@ -327,11 +338,11 @@ while [ $# -ne 0 ]; do
done
if [ "$user_set_scriptdir" != "true" ]; then
append_cache_entry BRO_SCRIPT_INSTALL_PATH STRING $prefix/share/bro
append_cache_entry ZEEK_SCRIPT_INSTALL_PATH STRING $prefix/share/zeek
fi
if [ "$user_set_conffilesdir" != "true" ]; then
append_cache_entry BRO_ETC_INSTALL_DIR PATH $prefix/etc
append_cache_entry ZEEK_ETC_INSTALL_DIR PATH $prefix/etc
fi
if [ -d $builddir ]; then

2
doc

@ -1 +1 @@
Subproject commit 5aa921f0f6ce2931e446a11f2a10cffb7f0dbc09
Subproject commit a840cea13e7c21951079422d6ec8971fb6812b06

2
man/CMakeLists.txt
View file

@ -1,5 +1,5 @@
install(DIRECTORY . DESTINATION ${BRO_MAN_INSTALL_PATH}/man8 FILES_MATCHING
install(DIRECTORY . DESTINATION ${ZEEK_MAN_INSTALL_PATH}/man8 FILES_MATCHING
PATTERN "*.8"
)

156
man/bro.8
View file

@ -1,156 +0,0 @@
.TH BRO "8" "November 2014" "bro" "System Administration Utilities"
.SH NAME
bro \- passive network traffic analyzer
.SH SYNOPSIS
.B bro
\/\fP [\fIoptions\fR] [\fIfile\fR ...]
.SH DESCRIPTION
Bro is primarily a security monitor that inspects all traffic on a link in
depth for signs of suspicious activity. More generally, however, Bro
supports a wide range of traffic analysis tasks even outside of the
security domain, including performance measurements and helping with
trouble-shooting.
Bro comes with built-in functionality for a range of analysis and detection
tasks, including detecting malware by interfacing to external registries,
reporting vulnerable versions of software seen on the network, identifying
popular web applications, detecting SSH brute-forcing, validating SSL
certificate chains, among others.
.SH OPTIONS
.TP
.B <file>
policy file, or read stdin
.TP
\fB\-a\fR,\ \-\-parse\-only
exit immediately after parsing scripts
.TP
\fB\-b\fR,\ \-\-bare\-mode
don't load scripts from the base/ directory
.TP
\fB\-d\fR,\ \-\-debug\-policy
activate policy file debugging
.TP
\fB\-e\fR,\ \-\-exec <bro code>
augment loaded policies by given code
.TP
\fB\-f\fR,\ \-\-filter <filter>
tcpdump filter
.TP
\fB\-g\fR,\ \-\-dump\-config
dump current config into .state dir
.TP
\fB\-h\fR,\ \-\-help|\-?
command line help
.TP
\fB\-i\fR,\ \-\-iface <interface>
read from given interface
.TP
\fB\-p\fR,\ \-\-prefix <prefix>
add given prefix to policy file resolution
.TP
\fB\-r\fR,\ \-\-readfile <readfile>
read from given tcpdump file
.TP
\fB\-s\fR,\ \-\-rulefile <rulefile>
read rules from given file
.TP
\fB\-t\fR,\ \-\-tracefile <tracefile>
activate execution tracing
.TP
\fB\-w\fR,\ \-\-writefile <writefile>
write to given tcpdump file
.TP
\fB\-v\fR,\ \-\-version
print version and exit
.TP
\fB\-x\fR,\ \-\-print\-state <file.bst>
print contents of state file
.TP
\fB\-C\fR,\ \-\-no\-checksums
ignore checksums
.TP
\fB\-F\fR,\ \-\-force\-dns
force DNS
.TP
\fB\-I\fR,\ \-\-print\-id <ID name>
print out given ID
.TP
\fB\-N\fR,\ \-\-print\-plugins
print available plugins and exit (\fB\-NN\fR for verbose)
.TP
\fB\-P\fR,\ \-\-prime\-dns
prime DNS
.TP
\fB\-Q\fR,\ \-\-time
print execution time summary to stderr
.TP
\fB\-R\fR,\ \-\-replay <events.bst>
replay events
.TP
\fB\-S\fR,\ \-\-debug\-rules
enable rule debugging
.TP
\fB\-T\fR,\ \-\-re\-level <level>
set 'RE_level' for rules
.TP
\fB\-U\fR,\ \-\-status\-file <file>
Record process status in file
.TP
\fB\-W\fR,\ \-\-watchdog
activate watchdog timer
.TP
\fB\-X\fR,\ \-\-broxygen <cfgfile>
generate documentation based on config file
.TP
\fB\-\-pseudo\-realtime[=\fR<speedup>]
enable pseudo\-realtime for performance evaluation (default 1)
.TP
\fB\-\-load\-seeds\fR <file>
load seeds from given file
.TP
\fB\-\-save\-seeds\fR <file>
save seeds to given file
.TP
The following option is available only when Bro is built with the \-\-enable\-debug configure option:
.TP
\fB\-B\fR,\ \-\-debug <dbgstreams>
Enable debugging output for selected streams ('-B help' for help)
.TP
The following options are available only when Bro is built with gperftools support (use the \-\-enable\-perftools and \-\-enable\-perftools\-debug configure options):
.TP
\fB\-m\fR,\ \-\-mem-leaks
show leaks
.TP
\fB\-M\fR,\ \-\-mem-profile
record heap
.SH ENVIRONMENT
.TP
.B BROPATH
file search path
.TP
.B BRO_PLUGIN_PATH
plugin search path
.TP
.B BRO_PLUGIN_ACTIVATE
plugins to always activate
.TP
.B BRO_PREFIXES
prefix list
.TP
.B BRO_DNS_FAKE
disable DNS lookups
.TP
.B BRO_SEED_FILE
file to load seeds from
.TP
.B BRO_LOG_SUFFIX
ASCII log file extension
.TP
.B BRO_PROFILER_FILE
Output file for script execution statistics
.TP
.B BRO_DISABLE_BROXYGEN
Disable Broxygen documentation support
.SH AUTHOR
.B bro
was written by The Bro Project <info@bro.org>.

153
man/zeek.8 Normal file
View file

@ -0,0 +1,153 @@
.TH ZEEK "8" "November 2014" "zeek" "System Administration Utilities"
.SH NAME
zeek \- passive network traffic analyzer
.SH SYNOPSIS
.B zeek
\/\fP [\fIoptions\fR] [\fIfile\fR ...]
.SH DESCRIPTION
Zeek is primarily a security monitor that inspects all traffic on a link in
depth for signs of suspicious activity. More generally, however, Zeek
supports a wide range of traffic analysis tasks even outside of the
security domain, including performance measurements and helping with
trouble-shooting.
Zeek comes with built-in functionality for a range of analysis and detection
tasks, including detecting malware by interfacing to external registries,
reporting vulnerable versions of software seen on the network, identifying
popular web applications, detecting SSH brute-forcing, validating SSL
certificate chains, among others.
.SH OPTIONS
.TP
.B <file>
policy file, or read stdin
.TP
\fB\-a\fR,\ \-\-parse\-only
exit immediately after parsing scripts
.TP
\fB\-b\fR,\ \-\-bare\-mode
don't load scripts from the base/ directory
.TP
\fB\-d\fR,\ \-\-debug\-policy
activate policy file debugging
.TP
\fB\-e\fR,\ \-\-exec <zeek code>
augment loaded policies by given code
.TP
\fB\-f\fR,\ \-\-filter <filter>
tcpdump filter
.TP
\fB\-h\fR,\ \-\-help|\-?
command line help
.TP
\fB\-i\fR,\ \-\-iface <interface>
read from given interface
.TP
\fB\-p\fR,\ \-\-prefix <prefix>
add given prefix to policy file resolution
.TP
\fB\-r\fR,\ \-\-readfile <readfile>
read from given tcpdump file
.TP
\fB\-s\fR,\ \-\-rulefile <rulefile>
read rules from given file
.TP
\fB\-t\fR,\ \-\-tracefile <tracefile>
activate execution tracing
.TP
\fB\-w\fR,\ \-\-writefile <writefile>
write to given tcpdump file
.TP
\fB\-v\fR,\ \-\-version
print version and exit
.TP
\fB\-x\fR,\ \-\-print\-state <file.bst>
print contents of state file
.TP
\fB\-C\fR,\ \-\-no\-checksums
ignore checksums
.TP
\fB\-F\fR,\ \-\-force\-dns
force DNS
.TP
\fB\-I\fR,\ \-\-print\-id <ID name>
print out given ID
.TP
\fB\-N\fR,\ \-\-print\-plugins
print available plugins and exit (\fB\-NN\fR for verbose)
.TP
\fB\-P\fR,\ \-\-prime\-dns
prime DNS
.TP
\fB\-Q\fR,\ \-\-time
print execution time summary to stderr
.TP
\fB\-R\fR,\ \-\-replay <events.bst>
replay events
.TP
\fB\-S\fR,\ \-\-debug\-rules
enable rule debugging
.TP
\fB\-T\fR,\ \-\-re\-level <level>
set 'RE_level' for rules
.TP
\fB\-U\fR,\ \-\-status\-file <file>
Record process status in file
.TP
\fB\-W\fR,\ \-\-watchdog
activate watchdog timer
.TP
\fB\-X\fR,\ \-\-zeekygen <cfgfile>
generate documentation based on config file
.TP
\fB\-\-pseudo\-realtime[=\fR<speedup>]
enable pseudo\-realtime for performance evaluation (default 1)
.TP
\fB\-\-load\-seeds\fR <file>
load seeds from given file
.TP
\fB\-\-save\-seeds\fR <file>
save seeds to given file
.TP
The following option is available only when Zeek is built with the \-\-enable\-debug configure option:
.TP
\fB\-B\fR,\ \-\-debug <dbgstreams>
Enable debugging output for selected streams ('-B help' for help)
.TP
The following options are available only when Zeek is built with gperftools support (use the \-\-enable\-perftools and \-\-enable\-perftools\-debug configure options):
.TP
\fB\-m\fR,\ \-\-mem-leaks
show leaks
.TP
\fB\-M\fR,\ \-\-mem-profile
record heap
.SH ENVIRONMENT
.TP
.B ZEEKPATH
file search path
.TP
.B ZEEK_PLUGIN_PATH
plugin search path
.TP
.B ZEEK_PLUGIN_ACTIVATE
plugins to always activate
.TP
.B ZEEK_PREFIXES
prefix list
.TP
.B ZEEK_DNS_FAKE
disable DNS lookups
.TP
.B ZEEK_SEED_FILE
file to load seeds from
.TP
.B ZEEK_LOG_SUFFIX
ASCII log file extension
.TP
.B ZEEK_PROFILER_FILE
Output file for script execution statistics
.TP
.B ZEEK_DISABLE_ZEEKYGEN
Disable Zeekygen (Broxygen) documentation support
.SH AUTHOR
.B zeek
was written by The Zeek Project <info@zeek.org>.

35
scripts/CMakeLists.txt
View file

@ -1,16 +1,35 @@
include(InstallPackageConfigFile)
install(DIRECTORY ./ DESTINATION ${BRO_SCRIPT_INSTALL_PATH} FILES_MATCHING
install(DIRECTORY ./ DESTINATION ${ZEEK_SCRIPT_INSTALL_PATH} FILES_MATCHING
PATTERN "site/local*" EXCLUDE
PATTERN "test-all-policy.bro" EXCLUDE
PATTERN "*.bro"
PATTERN "test-all-policy.zeek" EXCLUDE
PATTERN "*.zeek"
PATTERN "*.sig"
PATTERN "*.fp"
)
# Install all local* scripts as config files since they are meant to be
# user modify-able.
if ( NOT BINARY_PACKAGING_MODE )
# If the user has a local.bro file from a previous installation, prefer to
# symlink local.zeek to it to avoid breaking their custom configuration --
# because ZeekControl will now prefer to load local.zeek rather than local.bro
# and we're about to install a default version of local.zeek.
set(_local_bro_dst ${ZEEK_SCRIPT_INSTALL_PATH}/site/local.bro)
set(_local_zeek_dst ${ZEEK_SCRIPT_INSTALL_PATH}/site/local.zeek)
install(CODE "
if ( \"\$ENV{DESTDIR}\" STREQUAL \"\" )
if ( EXISTS \"${_local_bro_dst}\" AND NOT EXISTS \"${_local_zeek_dst}\" )
message(STATUS \"WARNING: installed ${_local_zeek_dst} as symlink to ${_local_bro_dst}\")
execute_process(COMMAND \"${CMAKE_COMMAND}\" -E create_symlink
\"${_local_bro_dst}\" \"${_local_zeek_dst}\")
endif ()
endif ()
")
endif ()
# Install local script as a config file since it's meant to be modified directly.
InstallPackageConfigFile(
${CMAKE_CURRENT_SOURCE_DIR}/site/local.bro
${BRO_SCRIPT_INSTALL_PATH}/site
local.bro)
${CMAKE_CURRENT_SOURCE_DIR}/site/local.zeek
${ZEEK_SCRIPT_INSTALL_PATH}/site
local.zeek)

0
scripts/base/files/extract/__load__.bro → scripts/base/files/extract/__load__.zeek
View file

81
scripts/base/files/extract/main.bro
View file

@ -1,81 +0,0 @@
@load base/frameworks/files
@load base/utils/paths
module FileExtract;
export {
## The prefix where files are extracted to.
const prefix = "./extract_files/" &redef;
## The default max size for extracted files (they won't exceed this
## number of bytes). A value of zero means unlimited.
option default_limit = 0;
redef record Files::Info += {
## Local filename of extracted file.
extracted: string &optional &log;
## Set to true if the file being extracted was cut off
## so the whole file was not logged.
extracted_cutoff: bool &optional &log;
## The number of bytes extracted to disk.
extracted_size: count &optional &log;
};
redef record Files::AnalyzerArgs += {
## The local filename to which to write an extracted file.
## This field is used in the core by the extraction plugin
## to know where to write the file to. If not specified, then
## a filename in the format "extract-<source>-<id>" is
## automatically assigned (using the *source* and *id*
## fields of :bro:see:`fa_file`).
extract_filename: string &optional;
## The maximum allowed file size in bytes of *extract_filename*.
## Once reached, a :bro:see:`file_extraction_limit` event is
## raised and the analyzer will be removed unless
## :bro:see:`FileExtract::set_limit` is called to increase the
## limit. A value of zero means "no limit".
extract_limit: count &default=default_limit;
};
## Sets the maximum allowed extracted file size.
##
## f: A file that's being extracted.
##
## args: Arguments that identify a file extraction analyzer.
##
## n: Allowed number of bytes to be extracted.
##
## Returns: false if a file extraction analyzer wasn't active for
## the file, else true.
global set_limit: function(f: fa_file, args: Files::AnalyzerArgs, n: count): bool;
}
function set_limit(f: fa_file, args: Files::AnalyzerArgs, n: count): bool
{
return __set_limit(f$id, args, n);
}
function on_add(f: fa_file, args: Files::AnalyzerArgs)
{
if ( ! args?$extract_filename )
args$extract_filename = cat("extract-", f$last_active, "-", f$source,
"-", f$id);
f$info$extracted = args$extract_filename;
args$extract_filename = build_path_compressed(prefix, args$extract_filename);
f$info$extracted_cutoff = F;
mkdir(prefix);
}
event file_extraction_limit(f: fa_file, args: Files::AnalyzerArgs, limit: count, len: count) &priority=10
{
f$info$extracted_cutoff = T;
f$info$extracted_size = limit;
}
event bro_init() &priority=10
{
Files::register_analyzer_add_callback(Files::ANALYZER_EXTRACT, on_add);
}

81
scripts/base/files/extract/main.zeek Normal file
View file

@ -0,0 +1,81 @@
@load base/frameworks/files
@load base/utils/paths
module FileExtract;
export {
## The prefix where files are extracted to.
const prefix = "./extract_files/" &redef;
## The default max size for extracted files (they won't exceed this
## number of bytes). A value of zero means unlimited.
option default_limit = 0;
redef record Files::Info += {
## Local filename of extracted file.
extracted: string &optional &log;
## Set to true if the file being extracted was cut off
## so the whole file was not logged.
extracted_cutoff: bool &optional &log;
## The number of bytes extracted to disk.
extracted_size: count &optional &log;
};
redef record Files::AnalyzerArgs += {
## The local filename to which to write an extracted file.
## This field is used in the core by the extraction plugin
## to know where to write the file to. If not specified, then
## a filename in the format "extract-<source>-<id>" is
## automatically assigned (using the *source* and *id*
## fields of :zeek:see:`fa_file`).
extract_filename: string &optional;
## The maximum allowed file size in bytes of *extract_filename*.
## Once reached, a :zeek:see:`file_extraction_limit` event is
## raised and the analyzer will be removed unless
## :zeek:see:`FileExtract::set_limit` is called to increase the
## limit. A value of zero means "no limit".
extract_limit: count &default=default_limit;
};
## Sets the maximum allowed extracted file size.
##
## f: A file that's being extracted.
##
## args: Arguments that identify a file extraction analyzer.
##
## n: Allowed number of bytes to be extracted.
##
## Returns: false if a file extraction analyzer wasn't active for
## the file, else true.
global set_limit: function(f: fa_file, args: Files::AnalyzerArgs, n: count): bool;
}
function set_limit(f: fa_file, args: Files::AnalyzerArgs, n: count): bool
{
return __set_limit(f$id, args, n);
}
function on_add(f: fa_file, args: Files::AnalyzerArgs)
{
if ( ! args?$extract_filename )
args$extract_filename = cat("extract-", f$last_active, "-", f$source,
"-", f$id);
f$info$extracted = args$extract_filename;
args$extract_filename = build_path_compressed(prefix, args$extract_filename);
f$info$extracted_cutoff = F;
mkdir(prefix);
}
event file_extraction_limit(f: fa_file, args: Files::AnalyzerArgs, limit: count, len: count) &priority=10
{
f$info$extracted_cutoff = T;
f$info$extracted_size = limit;
}
event zeek_init() &priority=10
{
Files::register_analyzer_add_callback(Files::ANALYZER_EXTRACT, on_add);
}

0
scripts/base/files/hash/__load__.bro → scripts/base/files/hash/__load__.zeek
View file

0
scripts/base/files/hash/main.bro → scripts/base/files/hash/main.zeek
View file

0
scripts/base/files/pe/__load__.bro → scripts/base/files/pe/__load__.zeek
View file

0
scripts/base/files/pe/consts.bro → scripts/base/files/pe/consts.zeek
View file

137
scripts/base/files/pe/main.bro
View file

@ -1,137 +0,0 @@
module PE;
@load ./consts.bro
export {
redef enum Log::ID += { LOG };
type Info: record {
## Current timestamp.
ts: time &log;
## File id of this portable executable file.
id: string &log;
## The target machine that the file was compiled for.
machine: string &log &optional;
## The time that the file was created at.
compile_ts: time &log &optional;
## The required operating system.
os: string &log &optional;
## The subsystem that is required to run this file.
subsystem: string &log &optional;
## Is the file an executable, or just an object file?
is_exe: bool &log &default=T;
## Is the file a 64-bit executable?
is_64bit: bool &log &default=T;
## Does the file support Address Space Layout Randomization?
uses_aslr: bool &log &default=F;
## Does the file support Data Execution Prevention?
uses_dep: bool &log &default=F;
## Does the file enforce code integrity checks?
uses_code_integrity: bool &log &default=F;
## Does the file use structured exception handing?
uses_seh: bool &log &default=T;
## Does the file have an import table?
has_import_table: bool &log &optional;
## Does the file have an export table?
has_export_table: bool &log &optional;
## Does the file have an attribute certificate table?
has_cert_table: bool &log &optional;
## Does the file have a debug table?
has_debug_data: bool &log &optional;
## The names of the sections, in order.
section_names: vector of string &log &optional;
};
## Event for accessing logged records.
global log_pe: event(rec: Info);
## A hook that gets called when we first see a PE file.
global set_file: hook(f: fa_file);
}
redef record fa_file += {
pe: Info &optional;
};
const pe_mime_types = { "application/x-dosexec" };
event bro_init() &priority=5
{
Files::register_for_mime_types(Files::ANALYZER_PE, pe_mime_types);
Log::create_stream(LOG, [$columns=Info, $ev=log_pe, $path="pe"]);
}
hook set_file(f: fa_file) &priority=5
{
if ( ! f?$pe )
f$pe = [$ts=network_time(), $id=f$id];
}
event pe_dos_header(f: fa_file, h: PE::DOSHeader) &priority=5
{
hook set_file(f);
}
event pe_file_header(f: fa_file, h: PE::FileHeader) &priority=5
{
hook set_file(f);
f$pe$machine = machine_types[h$machine];
f$pe$compile_ts = h$ts;
f$pe$is_exe = ( h$optional_header_size > 0 );
for ( c in h$characteristics )
{
if ( file_characteristics[c] == "32BIT_MACHINE" )
f$pe$is_64bit = F;
}
}
event pe_optional_header(f: fa_file, h: PE::OptionalHeader) &priority=5
{
hook set_file(f);
# Only EXEs have optional headers
if ( ! f$pe$is_exe )
return;
f$pe$os = os_versions[h$os_version_major, h$os_version_minor];
f$pe$subsystem = windows_subsystems[h$subsystem];
for ( c in h$dll_characteristics )
{
if ( dll_characteristics[c] == "DYNAMIC_BASE" )
f$pe$uses_aslr = T;
if ( dll_characteristics[c] == "FORCE_INTEGRITY" )
f$pe$uses_code_integrity = T;
if ( dll_characteristics[c] == "NX_COMPAT" )
f$pe$uses_dep = T;
if ( dll_characteristics[c] == "NO_SEH" )
f$pe$uses_seh = F;
}
f$pe$has_export_table = (|h$table_sizes| > 0 && h$table_sizes[0] > 0);
f$pe$has_import_table = (|h$table_sizes| > 1 && h$table_sizes[1] > 0);
f$pe$has_cert_table = (|h$table_sizes| > 4 && h$table_sizes[4] > 0);
f$pe$has_debug_data = (|h$table_sizes| > 6 && h$table_sizes[6] > 0);
}
event pe_section_header(f: fa_file, h: PE::SectionHeader) &priority=5
{
hook set_file(f);
# Only EXEs have section headers
if ( ! f$pe$is_exe )
return;
if ( ! f$pe?$section_names )
f$pe$section_names = vector();
f$pe$section_names += h$name;
}
event file_state_remove(f: fa_file) &priority=-5
{
if ( f?$pe && f$pe?$machine )
Log::write(LOG, f$pe);
}

137
scripts/base/files/pe/main.zeek Normal file
View file

@ -0,0 +1,137 @@
module PE;
@load ./consts
export {
redef enum Log::ID += { LOG };
type Info: record {
## Current timestamp.
ts: time &log;
## File id of this portable executable file.
id: string &log;
## The target machine that the file was compiled for.
machine: string &log &optional;
## The time that the file was created at.
compile_ts: time &log &optional;
## The required operating system.
os: string &log &optional;
## The subsystem that is required to run this file.
subsystem: string &log &optional;
## Is the file an executable, or just an object file?
is_exe: bool &log &default=T;
## Is the file a 64-bit executable?
is_64bit: bool &log &default=T;
## Does the file support Address Space Layout Randomization?
uses_aslr: bool &log &default=F;
## Does the file support Data Execution Prevention?
uses_dep: bool &log &default=F;
## Does the file enforce code integrity checks?
uses_code_integrity: bool &log &default=F;
## Does the file use structured exception handing?
uses_seh: bool &log &default=T;
## Does the file have an import table?
has_import_table: bool &log &optional;
## Does the file have an export table?
has_export_table: bool &log &optional;
## Does the file have an attribute certificate table?
has_cert_table: bool &log &optional;
## Does the file have a debug table?
has_debug_data: bool &log &optional;
## The names of the sections, in order.
section_names: vector of string &log &optional;
};
## Event for accessing logged records.
global log_pe: event(rec: Info);
## A hook that gets called when we first see a PE file.
global set_file: hook(f: fa_file);
}
redef record fa_file += {
pe: Info &optional;
};
const pe_mime_types = { "application/x-dosexec" };
event zeek_init() &priority=5
{
Files::register_for_mime_types(Files::ANALYZER_PE, pe_mime_types);
Log::create_stream(LOG, [$columns=Info, $ev=log_pe, $path="pe"]);
}
hook set_file(f: fa_file) &priority=5
{
if ( ! f?$pe )
f$pe = [$ts=network_time(), $id=f$id];
}
event pe_dos_header(f: fa_file, h: PE::DOSHeader) &priority=5
{
hook set_file(f);
}
event pe_file_header(f: fa_file, h: PE::FileHeader) &priority=5
{
hook set_file(f);
f$pe$machine = machine_types[h$machine];
f$pe$compile_ts = h$ts;
f$pe$is_exe = ( h$optional_header_size > 0 );
for ( c in h$characteristics )
{
if ( file_characteristics[c] == "32BIT_MACHINE" )
f$pe$is_64bit = F;
}
}
event pe_optional_header(f: fa_file, h: PE::OptionalHeader) &priority=5
{
hook set_file(f);
# Only EXEs have optional headers
if ( ! f$pe$is_exe )
return;
f$pe$os = os_versions[h$os_version_major, h$os_version_minor];
f$pe$subsystem = windows_subsystems[h$subsystem];
for ( c in h$dll_characteristics )
{
if ( dll_characteristics[c] == "DYNAMIC_BASE" )
f$pe$uses_aslr = T;
if ( dll_characteristics[c] == "FORCE_INTEGRITY" )
f$pe$uses_code_integrity = T;
if ( dll_characteristics[c] == "NX_COMPAT" )
f$pe$uses_dep = T;
if ( dll_characteristics[c] == "NO_SEH" )
f$pe$uses_seh = F;
}
f$pe$has_export_table = (|h$table_sizes| > 0 && h$table_sizes[0] > 0);
f$pe$has_import_table = (|h$table_sizes| > 1 && h$table_sizes[1] > 0);
f$pe$has_cert_table = (|h$table_sizes| > 4 && h$table_sizes[4] > 0);
f$pe$has_debug_data = (|h$table_sizes| > 6 && h$table_sizes[6] > 0);
}
event pe_section_header(f: fa_file, h: PE::SectionHeader) &priority=5
{
hook set_file(f);
# Only EXEs have section headers
if ( ! f$pe$is_exe )
return;
if ( ! f$pe?$section_names )
f$pe$section_names = vector();
f$pe$section_names += h$name;
}
event file_state_remove(f: fa_file) &priority=-5
{
if ( f?$pe && f$pe?$machine )
Log::write(LOG, f$pe);
}

297
scripts/base/files/unified2/main.bro
View file

@ -1,297 +0,0 @@
@load base/utils/dir
@load base/utils/paths
module Unified2;
export {
redef enum Log::ID += { LOG };
## File to watch for Unified2 files.
const watch_file = "" &redef;
## Directory to watch for Unified2 records.
const watch_dir = "" &redef;
## The sid-msg.map file you would like to use for your alerts.
const sid_msg = "" &redef;
## The gen-msg.map file you would like to use for your alerts.
const gen_msg = "" &redef;
## The classification.config file you would like to use for your alerts.
const classification_config = "" &redef;
## Reconstructed "alert" which combines related events
## and packets.
global alert: event(f: fa_file, ev: Unified2::IDSEvent, pkt: Unified2::Packet);
type PacketID: record {
src_ip: addr;
src_p: port;
dst_ip: addr;
dst_p: port;
} &log;
type Info: record {
## Timestamp attached to the alert.
ts: time &log;
## Addresses and ports for the connection.
id: PacketID &log;
## Sensor that originated this event.
sensor_id: count &log;
## Sig id for this generator.
signature_id: count &log;
## A string representation of the *signature_id* field if a sid_msg.map file was loaded.
signature: string &log &optional;
## Which generator generated the alert?
generator_id: count &log;
## A string representation of the *generator_id* field if a gen_msg.map file was loaded.
generator: string &log &optional;
## Sig revision for this id.
signature_revision: count &log;
## Event classification.
classification_id: count &log;
## Descriptive classification string.
classification: string &log &optional;
## Event priority.
priority_id: count &log;
## Event ID.
event_id: count &log;
## Some of the packet data.
packet: string &log &optional;
} &log;
## The event for accessing logged records.
global log_unified2: event(rec: Info);
}
# Mappings for extended information from alerts.
global classification_map: table[count] of string;
global sid_map: table[count] of string;
global gen_map: table[count] of string;
global num_classification_map_reads = 0;
global num_sid_map_reads = 0;
global num_gen_map_reads = 0;
global watching = F;
# For reading in config files.
type OneLine: record {
line: string;
};
function mappings_initialized(): bool
{
return num_classification_map_reads > 0 &&
num_sid_map_reads > 0 &&
num_gen_map_reads > 0;
}
function start_watching()
{
if ( watching )
return;
watching = T;
if ( watch_dir != "" )
{
Dir::monitor(watch_dir, function(fname: string)
{
Input::add_analysis([$source=fname,
$reader=Input::READER_BINARY,
$mode=Input::STREAM,
$name=fname]);
}, 10secs);
}
if ( watch_file != "" )
{
Input::add_analysis([$source=watch_file,
$reader=Input::READER_BINARY,
$mode=Input::STREAM,
$name=watch_file]);
}
}
function create_info(ev: IDSEvent): Info
{
local info = Info($ts=ev$ts,
$id=PacketID($src_ip=ev$src_ip, $src_p=ev$src_p,
$dst_ip=ev$dst_ip, $dst_p=ev$dst_p),
$sensor_id=ev$sensor_id,
$signature_id=ev$signature_id,
$generator_id=ev$generator_id,
$signature_revision=ev$signature_revision,
$classification_id=ev$classification_id,
$priority_id=ev$priority_id,
$event_id=ev$event_id);
if ( ev$signature_id in sid_map )
info$signature=sid_map[ev$signature_id];
if ( ev$generator_id in gen_map )
info$generator=gen_map[ev$generator_id];
if ( ev$classification_id in classification_map )
info$classification=classification_map[ev$classification_id];
return info;
}
redef record fa_file += {
## Recently received IDS events. This is primarily used
## for tying together Unified2 events and packets.
u2_events: table[count] of Unified2::IDSEvent
&optional &create_expire=5sec
&expire_func=function(t: table[count] of Unified2::IDSEvent, event_id: count): interval
{
Log::write(LOG, create_info(t[event_id]));
return 0secs;
};
};
event Unified2::read_sid_msg_line(desc: Input::EventDescription, tpe: Input::Event, line: string)
{
local parts = split_string_n(line, / \|\| /, F, 100);
if ( |parts| >= 2 && /^[0-9]+$/ in parts[0] )
sid_map[to_count(parts[0])] = parts[1];
}
event Unified2::read_gen_msg_line(desc: Input::EventDescription, tpe: Input::Event, line: string)
{
local parts = split_string_n(line, / \|\| /, F, 3);
if ( |parts| >= 2 && /^[0-9]+$/ in parts[0] )
gen_map[to_count(parts[0])] = parts[2];
}
event Unified2::read_classification_line(desc: Input::EventDescription, tpe: Input::Event, line: string)
{
local parts = split_string_n(line, /: /, F, 2);
if ( |parts| == 2 )
{
local parts2 = split_string_n(parts[1], /,/, F, 4);
if ( |parts2| > 1 )
classification_map[|classification_map|+1] = parts2[0];
}
}
event Input::end_of_data(name: string, source: string)
{
if ( name == classification_config )
++num_classification_map_reads;
else if ( name == sid_msg )
++num_sid_map_reads;
else if ( name == gen_msg )
++num_gen_map_reads;
else
return;
if ( watching )
return;
if ( mappings_initialized() )
start_watching();
}
event bro_init() &priority=5
{
Log::create_stream(Unified2::LOG, [$columns=Info, $ev=log_unified2, $path="unified2"]);
if ( sid_msg == "" )
{
num_sid_map_reads = 1;
}
else
{
Input::add_event([$source=sid_msg,
$reader=Input::READER_RAW,
$mode=Input::REREAD,
$name=sid_msg,
$fields=Unified2::OneLine,
$want_record=F,
$ev=Unified2::read_sid_msg_line]);
}
if ( gen_msg == "" )
{
num_gen_map_reads = 1;
}
else
{
Input::add_event([$source=gen_msg,
$name=gen_msg,
$reader=Input::READER_RAW,
$mode=Input::REREAD,
$fields=Unified2::OneLine,
$want_record=F,
$ev=Unified2::read_gen_msg_line]);
}
if ( classification_config == "" )
{
num_classification_map_reads = 1;
}
else
{
Input::add_event([$source=classification_config,
$name=classification_config,
$reader=Input::READER_RAW,
$mode=Input::REREAD,
$fields=Unified2::OneLine,
$want_record=F,
$ev=Unified2::read_classification_line]);
}
if ( mappings_initialized() )
start_watching();
}
event file_new(f: fa_file)
{
local file_dir = "";
local parts = split_string_all(f$source, /\/[^\/]*$/);
if ( |parts| == 3 )
file_dir = parts[0];
if ( (watch_file != "" && f$source == watch_file) ||
(watch_dir != "" && compress_path(watch_dir) == file_dir) )
{
Files::add_analyzer(f, Files::ANALYZER_UNIFIED2);
f$u2_events = table();
}
}
event unified2_event(f: fa_file, ev: Unified2::IDSEvent)
{
f$u2_events[ev$event_id] = ev;
}
event unified2_packet(f: fa_file, pkt: Unified2::Packet)
{
if ( f?$u2_events && pkt$event_id in f$u2_events)
{
local ev = f$u2_events[pkt$event_id];
event Unified2::alert(f, ev, pkt);
delete f$u2_events[pkt$event_id];
}
}
event Unified2::alert(f: fa_file, ev: IDSEvent, pkt: Packet)
{
local info = create_info(ev);
info$packet=pkt$data;
Log::write(LOG, info);
}
event file_state_remove(f: fa_file)
{
if ( f?$u2_events )
{
# In case any events never had matching packets, flush
# the extras to the log.
for ( i, ev in f$u2_events )
{
Log::write(LOG, create_info(ev));
}
}
}

0
scripts/base/files/unified2/__load__.bro → scripts/base/files/x509/__load__.zeek
View file

90
scripts/base/files/x509/main.bro
View file

@ -1,90 +0,0 @@
@load base/frameworks/files
@load base/files/hash
module X509;
export {
redef enum Log::ID += { LOG };
## The record type which contains the fields of the X.509 log.
type Info: record {
## Current timestamp.
ts: time &log;
## File id of this certificate.
id: string &log;
## Basic information about the certificate.
certificate: X509::Certificate &log;
## The opaque wrapping the certificate. Mainly used
## for the verify operations.
handle: opaque of x509;
## All extensions that were encountered in the certificate.
extensions: vector of X509::Extension &default=vector();
## Subject alternative name extension of the certificate.
san: X509::SubjectAlternativeName &optional &log;
## Basic constraints extension of the certificate.
basic_constraints: X509::BasicConstraints &optional &log;
};
## Event for accessing logged records.
global log_x509: event(rec: Info);
}
event bro_init() &priority=5
{
Log::create_stream(X509::LOG, [$columns=Info, $ev=log_x509, $path="x509"]);
# We use MIME types internally to distinguish between user and CA certificates.
# The first certificate in a connection always gets tagged as user-cert, all
# following certificates get tagged as CA certificates. Certificates gotten via
# other means (e.g. identified from HTTP traffic when they are transfered in plain
# text) get tagged as application/pkix-cert.
Files::register_for_mime_type(Files::ANALYZER_X509, "application/x-x509-user-cert");
Files::register_for_mime_type(Files::ANALYZER_X509, "application/x-x509-ca-cert");
Files::register_for_mime_type(Files::ANALYZER_X509, "application/pkix-cert");
# Always calculate hashes. They are not necessary for base scripts
# but very useful for identification, and required for policy scripts
Files::register_for_mime_type(Files::ANALYZER_MD5, "application/x-x509-user-cert");
Files::register_for_mime_type(Files::ANALYZER_MD5, "application/x-x509-ca-cert");
Files::register_for_mime_type(Files::ANALYZER_MD5, "application/pkix-cert");
Files::register_for_mime_type(Files::ANALYZER_SHA1, "application/x-x509-user-cert");
Files::register_for_mime_type(Files::ANALYZER_SHA1, "application/x-x509-ca-cert");
Files::register_for_mime_type(Files::ANALYZER_SHA1, "application/pkix-cert");
}
redef record Files::Info += {
## Information about X509 certificates. This is used to keep
## certificate information until all events have been received.
x509: X509::Info &optional;
};
event x509_certificate(f: fa_file, cert_ref: opaque of x509, cert: X509::Certificate) &priority=5
{
f$info$x509 = [$ts=f$info$ts, $id=f$id, $certificate=cert, $handle=cert_ref];
}
event x509_extension(f: fa_file, ext: X509::Extension) &priority=5
{
if ( f$info?$x509 )
f$info$x509$extensions += ext;
}
event x509_ext_basic_constraints(f: fa_file, ext: X509::BasicConstraints) &priority=5
{
if ( f$info?$x509 )
f$info$x509$basic_constraints = ext;
}
event x509_ext_subject_alternative_name(f: fa_file, ext: X509::SubjectAlternativeName) &priority=5
{
if ( f$info?$x509 )
f$info$x509$san = ext;
}
event file_state_remove(f: fa_file) &priority=5
{
if ( ! f$info?$x509 )
return;
Log::write(LOG, f$info$x509);
}

90
scripts/base/files/x509/main.zeek Normal file
View file

@ -0,0 +1,90 @@
@load base/frameworks/files
@load base/files/hash
module X509;
export {
redef enum Log::ID += { LOG };
## The record type which contains the fields of the X.509 log.
type Info: record {
## Current timestamp.
ts: time &log;
## File id of this certificate.
id: string &log;
## Basic information about the certificate.
certificate: X509::Certificate &log;
## The opaque wrapping the certificate. Mainly used
## for the verify operations.
handle: opaque of x509;
## All extensions that were encountered in the certificate.
extensions: vector of X509::Extension &default=vector();
## Subject alternative name extension of the certificate.
san: X509::SubjectAlternativeName &optional &log;
## Basic constraints extension of the certificate.
basic_constraints: X509::BasicConstraints &optional &log;
};
## Event for accessing logged records.
global log_x509: event(rec: Info);
}
event zeek_init() &priority=5
{
Log::create_stream(X509::LOG, [$columns=Info, $ev=log_x509, $path="x509"]);
# We use MIME types internally to distinguish between user and CA certificates.
# The first certificate in a connection always gets tagged as user-cert, all
# following certificates get tagged as CA certificates. Certificates gotten via
# other means (e.g. identified from HTTP traffic when they are transfered in plain
# text) get tagged as application/pkix-cert.
Files::register_for_mime_type(Files::ANALYZER_X509, "application/x-x509-user-cert");
Files::register_for_mime_type(Files::ANALYZER_X509, "application/x-x509-ca-cert");
Files::register_for_mime_type(Files::ANALYZER_X509, "application/pkix-cert");
# Always calculate hashes. They are not necessary for base scripts
# but very useful for identification, and required for policy scripts
Files::register_for_mime_type(Files::ANALYZER_MD5, "application/x-x509-user-cert");
Files::register_for_mime_type(Files::ANALYZER_MD5, "application/x-x509-ca-cert");
Files::register_for_mime_type(Files::ANALYZER_MD5, "application/pkix-cert");
Files::register_for_mime_type(Files::ANALYZER_SHA1, "application/x-x509-user-cert");
Files::register_for_mime_type(Files::ANALYZER_SHA1, "application/x-x509-ca-cert");
Files::register_for_mime_type(Files::ANALYZER_SHA1, "application/pkix-cert");
}
redef record Files::Info += {
## Information about X509 certificates. This is used to keep
## certificate information until all events have been received.
x509: X509::Info &optional;
};
event x509_certificate(f: fa_file, cert_ref: opaque of x509, cert: X509::Certificate) &priority=5
{
f$info$x509 = [$ts=f$info$ts, $id=f$id, $certificate=cert, $handle=cert_ref];
}
event x509_extension(f: fa_file, ext: X509::Extension) &priority=5
{
if ( f$info?$x509 )
f$info$x509$extensions += ext;
}
event x509_ext_basic_constraints(f: fa_file, ext: X509::BasicConstraints) &priority=5
{
if ( f$info?$x509 )
f$info$x509$basic_constraints = ext;
}
event x509_ext_subject_alternative_name(f: fa_file, ext: X509::SubjectAlternativeName) &priority=5
{
if ( f$info?$x509 )
f$info$x509$san = ext;
}
event file_state_remove(f: fa_file) &priority=5
{
if ( ! f$info?$x509 )
return;
Log::write(LOG, f$info$x509);
}

2
scripts/base/frameworks/analyzer/README
View file

@ -1,3 +1,3 @@
The analyzer framework allows to dynamically enable or disable Bro's
The analyzer framework allows to dynamically enable or disable Zeek's
protocol analyzers, as well as to manage the well-known ports which
automatically activate a particular analyzer for new connections.

0
scripts/base/files/x509/__load__.bro → scripts/base/frameworks/analyzer/__load__.zeek
View file

229
scripts/base/frameworks/analyzer/main.bro
View file

@ -1,229 +0,0 @@
##! Framework for managing Bro's protocol analyzers.
##!
##! The analyzer framework allows to dynamically enable or disable analyzers, as
##! well as to manage the well-known ports which automatically activate a
##! particular analyzer for new connections.
##!
##! Protocol analyzers are identified by unique tags of type
##! :bro:type:`Analyzer::Tag`, such as :bro:enum:`Analyzer::ANALYZER_HTTP`.
##! These tags are defined internally by
##! the analyzers themselves, and documented in their analyzer-specific
##! description along with the events that they generate.
@load base/frameworks/packet-filter/utils
module Analyzer;
export {
## If true, all available analyzers are initially disabled at startup.
## One can then selectively enable them with
## :bro:id:`Analyzer::enable_analyzer`.
global disable_all = F &redef;
## Enables an analyzer. Once enabled, the analyzer may be used for analysis
## of future connections as decided by Bro's dynamic protocol detection.
##
## tag: The tag of the analyzer to enable.
##
## Returns: True if the analyzer was successfully enabled.
global enable_analyzer: function(tag: Analyzer::Tag) : bool;
## Disables an analyzer. Once disabled, the analyzer will not be used
## further for analysis of future connections.
##
## tag: The tag of the analyzer to disable.
##
## Returns: True if the analyzer was successfully disabled.
global disable_analyzer: function(tag: Analyzer::Tag) : bool;
## Registers a set of well-known ports for an analyzer. If a future
## connection on one of these ports is seen, the analyzer will be
## automatically assigned to parsing it. The function *adds* to all ports
## already registered, it doesn't replace them.
##
## tag: The tag of the analyzer.
##
## ports: The set of well-known ports to associate with the analyzer.
##
## Returns: True if the ports were successfully registered.
global register_for_ports: function(tag: Analyzer::Tag, ports: set[port]) : bool;
## Registers an individual well-known port for an analyzer. If a future
## connection on this port is seen, the analyzer will be automatically
## assigned to parsing it. The function *adds* to all ports already
## registered, it doesn't replace them.
##
## tag: The tag of the analyzer.
##
## p: The well-known port to associate with the analyzer.
##
## Returns: True if the port was successfully registered.
global register_for_port: function(tag: Analyzer::Tag, p: port) : bool;
## Returns a set of all well-known ports currently registered for a
## specific analyzer.
##
## tag: The tag of the analyzer.
##
## Returns: The set of ports.
global registered_ports: function(tag: Analyzer::Tag) : set[port];
## Returns a table of all ports-to-analyzer mappings currently registered.
##
## Returns: A table mapping each analyzer to the set of ports
## registered for it.
global all_registered_ports: function() : table[Analyzer::Tag] of set[port];
## Translates an analyzer type to a string with the analyzer's name.
##
## tag: The analyzer tag.
##
## Returns: The analyzer name corresponding to the tag.
global name: function(tag: Analyzer::Tag) : string;
## Translates an analyzer's name to a tag enum value.
##
## name: The analyzer name.
##
## Returns: The analyzer tag corresponding to the name.
global get_tag: function(name: string): Analyzer::Tag;
## Schedules an analyzer for a future connection originating from a
## given IP address and port.
##
## orig: The IP address originating a connection in the future.
## 0.0.0.0 can be used as a wildcard to match any originator address.
##
## resp: The IP address responding to a connection from *orig*.
##
## resp_p: The destination port at *resp*.
##
## analyzer: The analyzer ID.
##
## tout: A timeout interval after which the scheduling request will be
## discarded if the connection has not yet been seen.
##
## Returns: True if successful.
global schedule_analyzer: function(orig: addr, resp: addr, resp_p: port,
analyzer: Analyzer::Tag, tout: interval) : bool;
## Automatically creates a BPF filter for the specified protocol based
## on the data supplied for the protocol through the
## :bro:see:`Analyzer::register_for_ports` function.
##
## tag: The analyzer tag.
##
## Returns: BPF filter string.
global analyzer_to_bpf: function(tag: Analyzer::Tag): string;
## Create a BPF filter which matches all of the ports defined
## by the various protocol analysis scripts as "registered ports"
## for the protocol.
global get_bpf: function(): string;
## A set of analyzers to disable by default at startup. The default set
## contains legacy analyzers that are no longer supported.
global disabled_analyzers: set[Analyzer::Tag] = {
ANALYZER_INTERCONN,
ANALYZER_STEPPINGSTONE,
ANALYZER_BACKDOOR,
ANALYZER_TCPSTATS,
} &redef;
}
@load base/bif/analyzer.bif
global ports: table[Analyzer::Tag] of set[port];
event bro_init() &priority=5
{
if ( disable_all )
__disable_all_analyzers();
for ( a in disabled_analyzers )
disable_analyzer(a);
}
function enable_analyzer(tag: Analyzer::Tag) : bool
{
return __enable_analyzer(tag);
}
function disable_analyzer(tag: Analyzer::Tag) : bool
{
return __disable_analyzer(tag);
}
function register_for_ports(tag: Analyzer::Tag, ports: set[port]) : bool
{
local rc = T;
for ( p in ports )
{
if ( ! register_for_port(tag, p) )
rc = F;
}
return rc;
}
function register_for_port(tag: Analyzer::Tag, p: port) : bool
{
if ( ! __register_for_port(tag, p) )
return F;
if ( tag !in ports )
ports[tag] = set();
add ports[tag][p];
return T;
}
function registered_ports(tag: Analyzer::Tag) : set[port]
{
return tag in ports ? ports[tag] : set();
}
function all_registered_ports(): table[Analyzer::Tag] of set[port]
{
return ports;
}
function name(atype: Analyzer::Tag) : string
{
return __name(atype);
}
function get_tag(name: string): Analyzer::Tag
{
return __tag(name);
}
function schedule_analyzer(orig: addr, resp: addr, resp_p: port,
analyzer: Analyzer::Tag, tout: interval) : bool
{
return __schedule_analyzer(orig, resp, resp_p, analyzer, tout);
}
function analyzer_to_bpf(tag: Analyzer::Tag): string
{
# Return an empty string if an undefined analyzer was given.
if ( tag !in ports )
return "";
local output = "";
for ( p in ports[tag] )
output = PacketFilter::combine_filters(output, "or", PacketFilter::port_to_bpf(p));
return output;
}
function get_bpf(): string
{
local output = "";
for ( tag in ports )
{
output = PacketFilter::combine_filters(output, "or", analyzer_to_bpf(tag));
}
return output;
}

229
scripts/base/frameworks/analyzer/main.zeek Normal file
View file

@ -0,0 +1,229 @@
##! Framework for managing Zeek's protocol analyzers.
##!
##! The analyzer framework allows to dynamically enable or disable analyzers, as
##! well as to manage the well-known ports which automatically activate a
##! particular analyzer for new connections.
##!
##! Protocol analyzers are identified by unique tags of type
##! :zeek:type:`Analyzer::Tag`, such as :zeek:enum:`Analyzer::ANALYZER_HTTP`.
##! These tags are defined internally by
##! the analyzers themselves, and documented in their analyzer-specific
##! description along with the events that they generate.
@load base/frameworks/packet-filter/utils
module Analyzer;
export {
## If true, all available analyzers are initially disabled at startup.
## One can then selectively enable them with
## :zeek:id:`Analyzer::enable_analyzer`.
global disable_all = F &redef;
## Enables an analyzer. Once enabled, the analyzer may be used for analysis
## of future connections as decided by Zeek's dynamic protocol detection.
##
## tag: The tag of the analyzer to enable.
##
## Returns: True if the analyzer was successfully enabled.
global enable_analyzer: function(tag: Analyzer::Tag) : bool;
## Disables an analyzer. Once disabled, the analyzer will not be used
## further for analysis of future connections.
##
## tag: The tag of the analyzer to disable.
##
## Returns: True if the analyzer was successfully disabled.
global disable_analyzer: function(tag: Analyzer::Tag) : bool;
## Registers a set of well-known ports for an analyzer. If a future
## connection on one of these ports is seen, the analyzer will be
## automatically assigned to parsing it. The function *adds* to all ports
## already registered, it doesn't replace them.
##
## tag: The tag of the analyzer.
##
## ports: The set of well-known ports to associate with the analyzer.
##
## Returns: True if the ports were successfully registered.
global register_for_ports: function(tag: Analyzer::Tag, ports: set[port]) : bool;
## Registers an individual well-known port for an analyzer. If a future
## connection on this port is seen, the analyzer will be automatically
## assigned to parsing it. The function *adds* to all ports already
## registered, it doesn't replace them.
##
## tag: The tag of the analyzer.
##
## p: The well-known port to associate with the analyzer.
##
## Returns: True if the port was successfully registered.
global register_for_port: function(tag: Analyzer::Tag, p: port) : bool;
## Returns a set of all well-known ports currently registered for a
## specific analyzer.
##
## tag: The tag of the analyzer.
##
## Returns: The set of ports.
global registered_ports: function(tag: Analyzer::Tag) : set[port];
## Returns a table of all ports-to-analyzer mappings currently registered.
##
## Returns: A table mapping each analyzer to the set of ports
## registered for it.
global all_registered_ports: function() : table[Analyzer::Tag] of set[port];
## Translates an analyzer type to a string with the analyzer's name.
##
## tag: The analyzer tag.
##
## Returns: The analyzer name corresponding to the tag.
global name: function(tag: Analyzer::Tag) : string;
## Translates an analyzer's name to a tag enum value.
##
## name: The analyzer name.
##
## Returns: The analyzer tag corresponding to the name.
global get_tag: function(name: string): Analyzer::Tag;
## Schedules an analyzer for a future connection originating from a
## given IP address and port.
##
## orig: The IP address originating a connection in the future.
## 0.0.0.0 can be used as a wildcard to match any originator address.
##
## resp: The IP address responding to a connection from *orig*.
##
## resp_p: The destination port at *resp*.
##
## analyzer: The analyzer ID.
##
## tout: A timeout interval after which the scheduling request will be
## discarded if the connection has not yet been seen.
##
## Returns: True if successful.
global schedule_analyzer: function(orig: addr, resp: addr, resp_p: port,
analyzer: Analyzer::Tag, tout: interval) : bool;
## Automatically creates a BPF filter for the specified protocol based
## on the data supplied for the protocol through the
## :zeek:see:`Analyzer::register_for_ports` function.
##
## tag: The analyzer tag.
##
## Returns: BPF filter string.
global analyzer_to_bpf: function(tag: Analyzer::Tag): string;
## Create a BPF filter which matches all of the ports defined
## by the various protocol analysis scripts as "registered ports"
## for the protocol.
global get_bpf: function(): string;
## A set of analyzers to disable by default at startup. The default set
## contains legacy analyzers that are no longer supported.
global disabled_analyzers: set[Analyzer::Tag] = {
ANALYZER_INTERCONN,
ANALYZER_STEPPINGSTONE,
ANALYZER_BACKDOOR,
ANALYZER_TCPSTATS,
} &redef;
}
@load base/bif/analyzer.bif
global ports: table[Analyzer::Tag] of set[port];
event zeek_init() &priority=5
{
if ( disable_all )
__disable_all_analyzers();
for ( a in disabled_analyzers )
disable_analyzer(a);
}
function enable_analyzer(tag: Analyzer::Tag) : bool
{
return __enable_analyzer(tag);
}
function disable_analyzer(tag: Analyzer::Tag) : bool
{
return __disable_analyzer(tag);
}
function register_for_ports(tag: Analyzer::Tag, ports: set[port]) : bool
{
local rc = T;
for ( p in ports )
{
if ( ! register_for_port(tag, p) )
rc = F;
}
return rc;
}
function register_for_port(tag: Analyzer::Tag, p: port) : bool
{
if ( ! __register_for_port(tag, p) )
return F;
if ( tag !in ports )
ports[tag] = set();
add ports[tag][p];
return T;
}
function registered_ports(tag: Analyzer::Tag) : set[port]
{
return tag in ports ? ports[tag] : set();
}
function all_registered_ports(): table[Analyzer::Tag] of set[port]
{
return ports;
}
function name(atype: Analyzer::Tag) : string
{
return __name(atype);
}
function get_tag(name: string): Analyzer::Tag
{
return __tag(name);
}
function schedule_analyzer(orig: addr, resp: addr, resp_p: port,
analyzer: Analyzer::Tag, tout: interval) : bool
{
return __schedule_analyzer(orig, resp, resp_p, analyzer, tout);
}
function analyzer_to_bpf(tag: Analyzer::Tag): string
{
# Return an empty string if an undefined analyzer was given.
if ( tag !in ports )
return "";
local output = "";
for ( p in ports[tag] )
output = PacketFilter::combine_filters(output, "or", PacketFilter::port_to_bpf(p));
return output;
}
function get_bpf(): string
{
local output = "";
for ( tag in ports )
{
output = PacketFilter::combine_filters(output, "or", analyzer_to_bpf(tag));
}
return output;
}

2
scripts/base/frameworks/broker/README
View file

@ -1,2 +1,2 @@
The Broker communication framework facilitates connecting to remote Bro
The Broker communication framework facilitates connecting to remote Zeek
instances to share state and transfer events.

0
scripts/base/frameworks/broker/__load__.bro → scripts/base/frameworks/broker/__load__.zeek
View file

80
scripts/base/frameworks/broker/log.bro
View file

@ -1,80 +0,0 @@
@load ./main
module Broker;
export {
## The Broker logging stream identifier.
redef enum Log::ID += { LOG };
## The type of a Broker activity being logged.
type Type: enum {
## An informational status update.
STATUS,
## An error situation.
ERROR
};
## A record type containing the column fields of the Broker log.
type Info: record {
## The network time at which a Broker event occurred.
ts: time &log;
## The type of the Broker event.
ty: Type &log;
## The event being logged.
ev: string &log;
## The peer (if any) with which a Broker event is
## concerned.
peer: NetworkInfo &log &optional;
## An optional message describing the Broker event in more detail
message: string &log &optional;
};
}
event bro_init() &priority=5
{
Log::create_stream(Broker::LOG, [$columns=Info, $path="broker"]);
}
function log_status(ev: string, endpoint: EndpointInfo, msg: string)
{
local r: Info;
r = [$ts = network_time(),
$ev = ev,
$ty = STATUS,
$message = msg];
if ( endpoint?$network )
r$peer = endpoint$network;
Log::write(Broker::LOG, r);
}
event Broker::peer_added(endpoint: EndpointInfo, msg: string)
{
log_status("peer-added", endpoint, msg);
}
event Broker::peer_removed(endpoint: EndpointInfo, msg: string)
{
log_status("peer-removed", endpoint, msg);
}
event Broker::peer_lost(endpoint: EndpointInfo, msg: string)
{
log_status("connection-terminated", endpoint, msg);
}
event Broker::error(code: ErrorCode, msg: string)
{
local ev = cat(code);
ev = subst_string(ev, "Broker::", "");
ev = subst_string(ev, "_", "-");
ev = to_lower(ev);
Log::write(Broker::LOG, [$ts = network_time(),
$ev = ev,
$ty = ERROR,
$message = msg]);
}

80
scripts/base/frameworks/broker/log.zeek Normal file
View file

@ -0,0 +1,80 @@
@load ./main
module Broker;
export {
## The Broker logging stream identifier.
redef enum Log::ID += { LOG };
## The type of a Broker activity being logged.
type Type: enum {
## An informational status update.
STATUS,
## An error situation.
ERROR
};
## A record type containing the column fields of the Broker log.
type Info: record {
## The network time at which a Broker event occurred.
ts: time &log;
## The type of the Broker event.
ty: Type &log;
## The event being logged.
ev: string &log;
## The peer (if any) with which a Broker event is
## concerned.
peer: NetworkInfo &log &optional;
## An optional message describing the Broker event in more detail
message: string &log &optional;
};
}
event zeek_init() &priority=5
{
Log::create_stream(Broker::LOG, [$columns=Info, $path="broker"]);
}
function log_status(ev: string, endpoint: EndpointInfo, msg: string)
{
local r: Info;
r = [$ts = network_time(),
$ev = ev,
$ty = STATUS,
$message = msg];
if ( endpoint?$network )
r$peer = endpoint$network;
Log::write(Broker::LOG, r);
}
event Broker::peer_added(endpoint: EndpointInfo, msg: string)
{
log_status("peer-added", endpoint, msg);
}
event Broker::peer_removed(endpoint: EndpointInfo, msg: string)
{
log_status("peer-removed", endpoint, msg);
}
event Broker::peer_lost(endpoint: EndpointInfo, msg: string)
{
log_status("connection-terminated", endpoint, msg);
}
event Broker::error(code: ErrorCode, msg: string)
{
local ev = cat(code);
ev = subst_string(ev, "Broker::", "");
ev = subst_string(ev, "_", "-");
ev = to_lower(ev);
Log::write(Broker::LOG, [$ts = network_time(),
$ev = ev,
$ty = ERROR,
$message = msg]);
}

439
scripts/base/frameworks/broker/main.bro
View file

@ -1,439 +0,0 @@
##! The Broker-based communication API and its various options.
module Broker;
export {
## Default port for Broker communication. Where not specified
## otherwise, this is the port to connect to and listen on.
const default_port = 9999/tcp &redef;
## Default interval to retry listening on a port if it's currently in
## use already. Use of the BRO_DEFAULT_LISTEN_RETRY environment variable
## (set as a number of seconds) will override this option and also
## any values given to :bro:see:`Broker::listen`.
const default_listen_retry = 30sec &redef;
## Default address on which to listen.
##
## .. bro:see:: Broker::listen
const default_listen_address = getenv("BRO_DEFAULT_LISTEN_ADDRESS") &redef;
## Default interval to retry connecting to a peer if it cannot be made to
## work initially, or if it ever becomes disconnected. Use of the
## BRO_DEFAULT_CONNECT_RETRY environment variable (set as number of
## seconds) will override this option and also any values given to
## :bro:see:`Broker::peer`.
const default_connect_retry = 30sec &redef;
## If true, do not use SSL for network connections. By default, SSL will
## even be used if no certificates / CAs have been configured. In that case
## (which is the default) the communication will be encrypted, but not
## authenticated.
const disable_ssl = F &redef;
## Path to a file containing concatenated trusted certificates
## in PEM format. If set, Bro will require valid certificates for
## all peers.
const ssl_cafile = "" &redef;
## Path to an OpenSSL-style directory of trusted certificates.
## If set, Bro will require valid certificates for
## all peers.
const ssl_capath = "" &redef;
## Path to a file containing a X.509 certificate for this
## node in PEM format. If set, Bro will require valid certificates for
## all peers.
const ssl_certificate = "" &redef;
## Passphrase to decrypt the private key specified by
## :bro:see:`Broker::ssl_keyfile`. If set, Bro will require valid
## certificates for all peers.
const ssl_passphrase = "" &redef;
## Path to the file containing the private key for this node's
## certificate. If set, Bro will require valid certificates for
## all peers.
const ssl_keyfile = "" &redef;
## The number of buffered messages at the Broker/CAF layer after which
## a subscriber considers themselves congested (i.e. tune the congestion
## control mechanisms).
const congestion_queue_size = 200 &redef;
## Max number of threads to use for Broker/CAF functionality. The
## BRO_BROKER_MAX_THREADS environment variable overrides this setting.
const max_threads = 1 &redef;
## Interval of time for under-utilized Broker/CAF threads to sleep
## when in "moderate" mode.
const moderate_sleep = 16 msec &redef;
## Interval of time for under-utilized Broker/CAF threads to sleep
## when in "relaxed" mode.
const relaxed_sleep = 64 msec &redef;
## Number of work-stealing polling attempts for Broker/CAF threads
## in "aggressive" mode.
const aggressive_polls = 5 &redef;
## Number of work-stealing polling attempts for Broker/CAF threads
## in "moderate" mode.
const moderate_polls = 5 &redef;
## Frequency of work-stealing polling attempts for Broker/CAF threads
## in "aggressive" mode.
const aggressive_interval = 4 &redef;
## Frequency of work-stealing polling attempts for Broker/CAF threads
## in "moderate" mode.
const moderate_interval = 2 &redef;
## Frequency of work-stealing polling attempts for Broker/CAF threads
## in "relaxed" mode.
const relaxed_interval = 1 &redef;
## Forward all received messages to subscribing peers.
const forward_messages = F &redef;
## Whether calling :bro:see:`Broker::peer` will register the Broker
## system as an I/O source that will block the process from shutting
## down. For example, set this to false when you are reading pcaps,
## but also want to initaiate a Broker peering and still shutdown after
## done reading the pcap.
option peer_counts_as_iosource = T;
## The default topic prefix where logs will be published. The log's stream
## id is appended when writing to a particular stream.
const default_log_topic_prefix = "bro/logs/" &redef;
## The default implementation for :bro:see:`Broker::log_topic`.
function default_log_topic(id: Log::ID, path: string): string
{
return default_log_topic_prefix + cat(id);
}
## A function that will be called for each log entry to determine what
## broker topic string will be used for sending it to peers. The
## default implementation will return a value based on
## :bro:see:`Broker::default_log_topic_prefix`.
##
## id: the ID associated with the log stream entry that will be sent.
##
## path: the path to which the log stream entry will be output.
##
## Returns: a string representing the broker topic to which the log
## will be sent.
const log_topic: function(id: Log::ID, path: string): string = default_log_topic &redef;
type ErrorCode: enum {
## The unspecified default error code.
UNSPECIFIED = 1,
## Version incompatibility.
PEER_INCOMPATIBLE = 2,
## Referenced peer does not exist.
PEER_INVALID = 3,
## Remote peer not listening.
PEER_UNAVAILABLE = 4,
## A peering request timed out.
PEER_TIMEOUT = 5,
## Master with given name already exists.
MASTER_EXISTS = 6,
## Master with given name does not exist.
NO_SUCH_MASTER = 7,
## The given data store key does not exist.
NO_SUCH_KEY = 8,
## The store operation timed out.
REQUEST_TIMEOUT = 9,
## The operation expected a different type than provided.
TYPE_CLASH = 10,
## The data value cannot be used to carry out the desired operation.
INVALID_DATA = 11,
## The storage backend failed to execute the operation.
BACKEND_FAILURE = 12,
## The storage backend failed to execute the operation.
STALE_DATA = 13,
## Catch-all for a CAF-level problem.
CAF_ERROR = 100
};
## The possible states of a peer endpoint.
type PeerStatus: enum {
## The peering process is initiated.
INITIALIZING,
## Connection establishment in process.
CONNECTING,
## Connection established, peering pending.
CONNECTED,
## Successfully peered.
PEERED,
## Connection to remote peer lost.
DISCONNECTED,
## Reconnecting to peer after a lost connection.
RECONNECTING,
};
type NetworkInfo: record {
## The IP address or hostname where the endpoint listens.
address: string &log;
## The port where the endpoint is bound to.
bound_port: port &log;
};
type EndpointInfo: record {
## A unique identifier of the node.
id: string;
## Network-level information.
network: NetworkInfo &optional;
};
type PeerInfo: record {
peer: EndpointInfo;
status: PeerStatus;
};
type PeerInfos: vector of PeerInfo;
## Opaque communication data.
type Data: record {
data: opaque of Broker::Data &optional;
};
## Opaque communication data sequence.
type DataVector: vector of Broker::Data;
## Opaque event communication data.
type Event: record {
## The name of the event. Not set if invalid event or arguments.
name: string &optional;
## The arguments to the event.
args: DataVector;
};
## Opaque communication data used as a convenient way to wrap key-value
## pairs that comprise table entries.
type TableItem : record {
key: Broker::Data;
val: Broker::Data;
};
## Listen for remote connections.
##
## a: an address string on which to accept connections, e.g.
## "127.0.0.1". An empty string refers to INADDR_ANY.
##
## p: the TCP port to listen on. The value 0 means that the OS should choose
## the next available free port.
##
## retry: If non-zero, retries listening in regular intervals if the port cannot be
## acquired immediately. 0 disables retries. If the
## BRO_DEFAULT_LISTEN_RETRY environment variable is set (as number
## of seconds), it overrides any value given here.
##
## Returns: the bound port or 0/? on failure.
##
## .. bro:see:: Broker::status
global listen: function(a: string &default = default_listen_address,
p: port &default = default_port,
retry: interval &default = default_listen_retry): port;
## Initiate a remote connection.
##
## a: an address to connect to, e.g. "localhost" or "127.0.0.1".
##
## p: the TCP port on which the remote side is listening.
##
## retry: an interval at which to retry establishing the
## connection with the remote peer if it cannot be made initially, or
## if it ever becomes disconnected. If the
## BRO_DEFAULT_CONNECT_RETRY environment variable is set (as number
## of seconds), it overrides any value given here.
##
## Returns: true if it's possible to try connecting with the peer and
## it's a new peer. The actual connection may not be established
## until a later point in time.
##
## .. bro:see:: Broker::status
global peer: function(a: string, p: port &default=default_port,
retry: interval &default=default_connect_retry): bool;
## Remove a remote connection.
##
## Note that this does not terminate the connection to the peer, it
## just means that we won't exchange any further information with it
## unless peering resumes later.
##
## a: the address used in previous successful call to :bro:see:`Broker::peer`.
##
## p: the port used in previous successful call to :bro:see:`Broker::peer`.
##
## Returns: true if the arguments match a previously successful call to
## :bro:see:`Broker::peer`.
##
## TODO: We do not have a function yet to terminate a connection.
global unpeer: function(a: string, p: port): bool;
## Get a list of all peer connections.
##
## Returns: a list of all peer connections.
global peers: function(): vector of PeerInfo;
## Get a unique identifier for the local broker endpoint.
##
## Returns: a unique identifier for the local broker endpoint.
global node_id: function(): string;
## Sends all pending log messages to remote peers. This normally
## doesn't need to be used except for test cases that are time-sensitive.
global flush_logs: function(): count;
## Publishes the value of an identifier to a given topic. The subscribers
## will update their local value for that identifier on receipt.
##
## topic: a topic associated with the message.
##
## id: the identifier to publish.
##
## Returns: true if the message is sent.
global publish_id: function(topic: string, id: string): bool;
## Register interest in all peer event messages that use a certain topic
## prefix. Note that subscriptions may not be altered immediately after
## calling (except during :bro:see:`bro_init`).
##
## topic_prefix: a prefix to match against remote message topics.
## e.g. an empty prefix matches everything and "a" matches
## "alice" and "amy" but not "bob".
##
## Returns: true if it's a new event subscription and it is now registered.
global subscribe: function(topic_prefix: string): bool;
## Unregister interest in all peer event messages that use a topic prefix.
## Note that subscriptions may not be altered immediately after calling
## (except during :bro:see:`bro_init`).
##
## topic_prefix: a prefix previously supplied to a successful call to
## :bro:see:`Broker::subscribe` or :bro:see:`Broker::forward`.
##
## Returns: true if interest in the topic prefix is no longer advertised.
global unsubscribe: function(topic_prefix: string): bool;
## Register a topic prefix subscription for events that should only be
## forwarded to any subscribing peers and not raise any event handlers
## on the receiving/forwarding node. i.e. it's the same as
## :bro:see:`Broker::subscribe` except matching events are not raised
## on the receiver, just forwarded. Use :bro:see:`Broker::unsubscribe`
## with the same argument to undo this operation.
##
## topic_prefix: a prefix to match against remote message topics.
## e.g. an empty prefix matches everything and "a" matches
## "alice" and "amy" but not "bob".
##
## Returns: true if a new event forwarding/subscription is now registered.
global forward: function(topic_prefix: string): bool;
## Automatically send an event to any interested peers whenever it is
## locally dispatched. (For example, using "event my_event(...);" in a
## script.)
##
## topic: a topic string associated with the event message.
## Peers advertise interest by registering a subscription to some
## prefix of this topic name.
##
## ev: a Bro event value.
##
## Returns: true if automatic event sending is now enabled.
global auto_publish: function(topic: string, ev: any): bool;
## Stop automatically sending an event to peers upon local dispatch.
##
## topic: a topic originally given to :bro:see:`Broker::auto_publish`.
##
## ev: an event originally given to :bro:see:`Broker::auto_publish`.
##
## Returns: true if automatic events will not occur for the topic/event
## pair.
global auto_unpublish: function(topic: string, ev: any): bool;
}
@load base/bif/comm.bif
@load base/bif/messaging.bif
module Broker;
event retry_listen(a: string, p: port, retry: interval)
{
listen(a, p, retry);
}
function listen(a: string, p: port, retry: interval): port
{
local bound = __listen(a, p);
if ( bound == 0/tcp )
{
local e = getenv("BRO_DEFAULT_LISTEN_RETRY");
if ( e != "" )
retry = double_to_interval(to_double(e));
if ( retry != 0secs )
schedule retry { retry_listen(a, p, retry) };
}
return bound;
}
function peer(a: string, p: port, retry: interval): bool
{
return __peer(a, p, retry);
}
function unpeer(a: string, p: port): bool
{
return __unpeer(a, p);
}
function peers(): vector of PeerInfo
{
return __peers();
}
function node_id(): string
{
return __node_id();
}
function flush_logs(): count
{
return __flush_logs();
}
function publish_id(topic: string, id: string): bool
{
return __publish_id(topic, id);
}
function subscribe(topic_prefix: string): bool
{
return __subscribe(topic_prefix);
}
function forward(topic_prefix: string): bool
{
return __forward(topic_prefix);
}
function unsubscribe(topic_prefix: string): bool
{
return __unsubscribe(topic_prefix);
}
function auto_publish(topic: string, ev: any): bool
{
return __auto_publish(topic, ev);
}
function auto_unpublish(topic: string, ev: any): bool
{
return __auto_unpublish(topic, ev);
}

447
scripts/base/frameworks/broker/main.zeek Normal file
View file

@ -0,0 +1,447 @@
##! The Broker-based communication API and its various options.
module Broker;
export {
## Default port for Broker communication. Where not specified
## otherwise, this is the port to connect to and listen on.
const default_port = 9999/tcp &redef;
## Default interval to retry listening on a port if it's currently in
## use already. Use of the ZEEK_DEFAULT_LISTEN_RETRY environment variable
## (set as a number of seconds) will override this option and also
## any values given to :zeek:see:`Broker::listen`.
const default_listen_retry = 30sec &redef;
## Default address on which to listen.
##
## .. zeek:see:: Broker::listen
const default_listen_address = getenv("ZEEK_DEFAULT_LISTEN_ADDRESS") &redef;
## Default interval to retry connecting to a peer if it cannot be made to
## work initially, or if it ever becomes disconnected. Use of the
## ZEEK_DEFAULT_CONNECT_RETRY environment variable (set as number of
## seconds) will override this option and also any values given to
## :zeek:see:`Broker::peer`.
const default_connect_retry = 30sec &redef;
## If true, do not use SSL for network connections. By default, SSL will
## even be used if no certificates / CAs have been configured. In that case
## (which is the default) the communication will be encrypted, but not
## authenticated.
const disable_ssl = F &redef;
## Path to a file containing concatenated trusted certificates
## in PEM format. If set, Zeek will require valid certificates for
## all peers.
const ssl_cafile = "" &redef;
## Path to an OpenSSL-style directory of trusted certificates.
## If set, Zeek will require valid certificates for
## all peers.
const ssl_capath = "" &redef;
## Path to a file containing a X.509 certificate for this
## node in PEM format. If set, Zeek will require valid certificates for
## all peers.
const ssl_certificate = "" &redef;
## Passphrase to decrypt the private key specified by
## :zeek:see:`Broker::ssl_keyfile`. If set, Zeek will require valid
## certificates for all peers.
const ssl_passphrase = "" &redef;
## Path to the file containing the private key for this node's
## certificate. If set, Zeek will require valid certificates for
## all peers.
const ssl_keyfile = "" &redef;
## The number of buffered messages at the Broker/CAF layer after which
## a subscriber considers themselves congested (i.e. tune the congestion
## control mechanisms).
const congestion_queue_size = 200 &redef;
## The max number of log entries per log stream to batch together when
## sending log messages to a remote logger.
const log_batch_size = 400 &redef;
## Max time to buffer log messages before sending the current set out as a
## batch.
const log_batch_interval = 1sec &redef;
## Max number of threads to use for Broker/CAF functionality. The
## ZEEK_BROKER_MAX_THREADS environment variable overrides this setting.
const max_threads = 1 &redef;
## Interval of time for under-utilized Broker/CAF threads to sleep
## when in "moderate" mode.
const moderate_sleep = 16 msec &redef;
## Interval of time for under-utilized Broker/CAF threads to sleep
## when in "relaxed" mode.
const relaxed_sleep = 64 msec &redef;
## Number of work-stealing polling attempts for Broker/CAF threads
## in "aggressive" mode.
const aggressive_polls = 5 &redef;
## Number of work-stealing polling attempts for Broker/CAF threads
## in "moderate" mode.
const moderate_polls = 5 &redef;
## Frequency of work-stealing polling attempts for Broker/CAF threads
## in "aggressive" mode.
const aggressive_interval = 4 &redef;
## Frequency of work-stealing polling attempts for Broker/CAF threads
## in "moderate" mode.
const moderate_interval = 2 &redef;
## Frequency of work-stealing polling attempts for Broker/CAF threads
## in "relaxed" mode.
const relaxed_interval = 1 &redef;
## Forward all received messages to subscribing peers.
const forward_messages = F &redef;
## Whether calling :zeek:see:`Broker::peer` will register the Broker
## system as an I/O source that will block the process from shutting
## down. For example, set this to false when you are reading pcaps,
## but also want to initaiate a Broker peering and still shutdown after
## done reading the pcap.
option peer_counts_as_iosource = T;
## The default topic prefix where logs will be published. The log's stream
## id is appended when writing to a particular stream.
const default_log_topic_prefix = "zeek/logs/" &redef;
## The default implementation for :zeek:see:`Broker::log_topic`.
function default_log_topic(id: Log::ID, path: string): string
{
return default_log_topic_prefix + cat(id);
}
## A function that will be called for each log entry to determine what
## broker topic string will be used for sending it to peers. The
## default implementation will return a value based on
## :zeek:see:`Broker::default_log_topic_prefix`.
##
## id: the ID associated with the log stream entry that will be sent.
##
## path: the path to which the log stream entry will be output.
##
## Returns: a string representing the broker topic to which the log
## will be sent.
const log_topic: function(id: Log::ID, path: string): string = default_log_topic &redef;
type ErrorCode: enum {
## The unspecified default error code.
UNSPECIFIED = 1,
## Version incompatibility.
PEER_INCOMPATIBLE = 2,
## Referenced peer does not exist.
PEER_INVALID = 3,
## Remote peer not listening.
PEER_UNAVAILABLE = 4,
## A peering request timed out.
PEER_TIMEOUT = 5,
## Master with given name already exists.
MASTER_EXISTS = 6,
## Master with given name does not exist.
NO_SUCH_MASTER = 7,
## The given data store key does not exist.
NO_SUCH_KEY = 8,
## The store operation timed out.
REQUEST_TIMEOUT = 9,
## The operation expected a different type than provided.
TYPE_CLASH = 10,
## The data value cannot be used to carry out the desired operation.
INVALID_DATA = 11,
## The storage backend failed to execute the operation.
BACKEND_FAILURE = 12,
## The storage backend failed to execute the operation.
STALE_DATA = 13,
## Catch-all for a CAF-level problem.
CAF_ERROR = 100
};
## The possible states of a peer endpoint.
type PeerStatus: enum {
## The peering process is initiated.
INITIALIZING,
## Connection establishment in process.
CONNECTING,
## Connection established, peering pending.
CONNECTED,
## Successfully peered.
PEERED,
## Connection to remote peer lost.
DISCONNECTED,
## Reconnecting to peer after a lost connection.
RECONNECTING,
};
type NetworkInfo: record {
## The IP address or hostname where the endpoint listens.
address: string &log;
## The port where the endpoint is bound to.
bound_port: port &log;
};
type EndpointInfo: record {
## A unique identifier of the node.
id: string;
## Network-level information.
network: NetworkInfo &optional;
};
type PeerInfo: record {
peer: EndpointInfo;
status: PeerStatus;
};
type PeerInfos: vector of PeerInfo;
## Opaque communication data.
type Data: record {
data: opaque of Broker::Data &optional;
};
## Opaque communication data sequence.
type DataVector: vector of Broker::Data;
## Opaque event communication data.
type Event: record {
## The name of the event. Not set if invalid event or arguments.
name: string &optional;
## The arguments to the event.
args: DataVector;
};
## Opaque communication data used as a convenient way to wrap key-value
## pairs that comprise table entries.
type TableItem : record {
key: Broker::Data;
val: Broker::Data;
};
## Listen for remote connections.
##
## a: an address string on which to accept connections, e.g.
## "127.0.0.1". An empty string refers to INADDR_ANY.
##
## p: the TCP port to listen on. The value 0 means that the OS should choose
## the next available free port.
##
## retry: If non-zero, retries listening in regular intervals if the port cannot be
## acquired immediately. 0 disables retries. If the
## ZEEK_DEFAULT_LISTEN_RETRY environment variable is set (as number
## of seconds), it overrides any value given here.
##
## Returns: the bound port or 0/? on failure.
##
## .. zeek:see:: Broker::status
global listen: function(a: string &default = default_listen_address,
p: port &default = default_port,
retry: interval &default = default_listen_retry): port;
## Initiate a remote connection.
##
## a: an address to connect to, e.g. "localhost" or "127.0.0.1".
##
## p: the TCP port on which the remote side is listening.
##
## retry: an interval at which to retry establishing the
## connection with the remote peer if it cannot be made initially, or
## if it ever becomes disconnected. If the
## ZEEK_DEFAULT_CONNECT_RETRY environment variable is set (as number
## of seconds), it overrides any value given here.
##
## Returns: true if it's possible to try connecting with the peer and
## it's a new peer. The actual connection may not be established
## until a later point in time.
##
## .. zeek:see:: Broker::status
global peer: function(a: string, p: port &default=default_port,
retry: interval &default=default_connect_retry): bool;
## Remove a remote connection.
##
## Note that this does not terminate the connection to the peer, it
## just means that we won't exchange any further information with it
## unless peering resumes later.
##
## a: the address used in previous successful call to :zeek:see:`Broker::peer`.
##
## p: the port used in previous successful call to :zeek:see:`Broker::peer`.
##
## Returns: true if the arguments match a previously successful call to
## :zeek:see:`Broker::peer`.
##
## TODO: We do not have a function yet to terminate a connection.
global unpeer: function(a: string, p: port): bool;
## Get a list of all peer connections.
##
## Returns: a list of all peer connections.
global peers: function(): vector of PeerInfo;
## Get a unique identifier for the local broker endpoint.
##
## Returns: a unique identifier for the local broker endpoint.
global node_id: function(): string;
## Sends all pending log messages to remote peers. This normally
## doesn't need to be used except for test cases that are time-sensitive.
global flush_logs: function(): count;
## Publishes the value of an identifier to a given topic. The subscribers
## will update their local value for that identifier on receipt.
##
## topic: a topic associated with the message.
##
## id: the identifier to publish.
##
## Returns: true if the message is sent.
global publish_id: function(topic: string, id: string): bool;
## Register interest in all peer event messages that use a certain topic
## prefix. Note that subscriptions may not be altered immediately after
## calling (except during :zeek:see:`zeek_init`).
##
## topic_prefix: a prefix to match against remote message topics.
## e.g. an empty prefix matches everything and "a" matches
## "alice" and "amy" but not "bob".
##
## Returns: true if it's a new event subscription and it is now registered.
global subscribe: function(topic_prefix: string): bool;
## Unregister interest in all peer event messages that use a topic prefix.
## Note that subscriptions may not be altered immediately after calling
## (except during :zeek:see:`zeek_init`).
##
## topic_prefix: a prefix previously supplied to a successful call to
## :zeek:see:`Broker::subscribe` or :zeek:see:`Broker::forward`.
##
## Returns: true if interest in the topic prefix is no longer advertised.
global unsubscribe: function(topic_prefix: string): bool;
## Register a topic prefix subscription for events that should only be
## forwarded to any subscribing peers and not raise any event handlers
## on the receiving/forwarding node. i.e. it's the same as
## :zeek:see:`Broker::subscribe` except matching events are not raised
## on the receiver, just forwarded. Use :zeek:see:`Broker::unsubscribe`
## with the same argument to undo this operation.
##
## topic_prefix: a prefix to match against remote message topics.
## e.g. an empty prefix matches everything and "a" matches
## "alice" and "amy" but not "bob".
##
## Returns: true if a new event forwarding/subscription is now registered.
global forward: function(topic_prefix: string): bool;
## Automatically send an event to any interested peers whenever it is
## locally dispatched. (For example, using "event my_event(...);" in a
## script.)
##
## topic: a topic string associated with the event message.
## Peers advertise interest by registering a subscription to some
## prefix of this topic name.
##
## ev: a Zeek event value.
##
## Returns: true if automatic event sending is now enabled.
global auto_publish: function(topic: string, ev: any): bool;
## Stop automatically sending an event to peers upon local dispatch.
##
## topic: a topic originally given to :zeek:see:`Broker::auto_publish`.
##
## ev: an event originally given to :zeek:see:`Broker::auto_publish`.
##
## Returns: true if automatic events will not occur for the topic/event
## pair.
global auto_unpublish: function(topic: string, ev: any): bool;
}
@load base/bif/comm.bif
@load base/bif/messaging.bif
module Broker;
event retry_listen(a: string, p: port, retry: interval)
{
listen(a, p, retry);
}
function listen(a: string, p: port, retry: interval): port
{
local bound = __listen(a, p);
if ( bound == 0/tcp )
{
local e = getenv("ZEEK_DEFAULT_LISTEN_RETRY");
if ( e != "" )
retry = double_to_interval(to_double(e));
if ( retry != 0secs )
schedule retry { retry_listen(a, p, retry) };
}
return bound;
}
function peer(a: string, p: port, retry: interval): bool
{
return __peer(a, p, retry);
}
function unpeer(a: string, p: port): bool
{
return __unpeer(a, p);
}
function peers(): vector of PeerInfo
{
return __peers();
}
function node_id(): string
{
return __node_id();
}
function flush_logs(): count
{
return __flush_logs();
}
function publish_id(topic: string, id: string): bool
{
return __publish_id(topic, id);
}
function subscribe(topic_prefix: string): bool
{
return __subscribe(topic_prefix);
}
function forward(topic_prefix: string): bool
{
return __forward(topic_prefix);
}
function unsubscribe(topic_prefix: string): bool
{
return __unsubscribe(topic_prefix);
}
function auto_publish(topic: string, ev: any): bool
{
return __auto_publish(topic, ev);
}
function auto_unpublish(topic: string, ev: any): bool
{
return __auto_unpublish(topic, ev);
}

1043
scripts/base/frameworks/broker/store.bro
View file

File diff suppressed because it is too large Load diff

1043
scripts/base/frameworks/broker/store.zeek Normal file
View file

File diff suppressed because it is too large Load diff

2
scripts/base/frameworks/cluster/README
View file

@ -1,2 +1,2 @@
The cluster framework provides for establishing and controlling a cluster
of Bro instances.
of Zeek instances.

48
scripts/base/frameworks/cluster/__load__.bro
View file

@ -1,48 +0,0 @@
# Load the core cluster support.
@load ./main
@load ./pools
@if ( Cluster::is_enabled() )
# Give the node being started up it's peer name.
redef peer_description = Cluster::node;
@if ( Cluster::enable_round_robin_logging )
redef Broker::log_topic = Cluster::rr_log_topic;
@endif
# Add a cluster prefix.
@prefixes += cluster
# If this script isn't found anywhere, the cluster bombs out.
# Loading the cluster framework requires that a script by this name exists
# somewhere in the BROPATH. The only thing in the file should be the
# cluster definition in the :bro:id:`Cluster::nodes` variable.
@load cluster-layout
@if ( Cluster::node in Cluster::nodes )
@load ./setup-connections
@if ( Cluster::local_node_type() == Cluster::MANAGER )
@load ./nodes/manager
# If no logger is defined, then the manager receives logs.
@if ( Cluster::manager_is_logger )
@load ./nodes/logger
@endif
@endif
@if ( Cluster::local_node_type() == Cluster::LOGGER )
@load ./nodes/logger
@endif
@if ( Cluster::local_node_type() == Cluster::PROXY )
@load ./nodes/proxy
@endif
@if ( Cluster::local_node_type() == Cluster::WORKER )
@load ./nodes/worker
@endif
@endif
@endif

48
scripts/base/frameworks/cluster/__load__.zeek Normal file
View file

@ -0,0 +1,48 @@
# Load the core cluster support.
@load ./main
@load ./pools
@if ( Cluster::is_enabled() )
# Give the node being started up it's peer name.
redef peer_description = Cluster::node;
@if ( Cluster::enable_round_robin_logging )
redef Broker::log_topic = Cluster::rr_log_topic;
@endif
# Add a cluster prefix.
@prefixes += cluster
# If this script isn't found anywhere, the cluster bombs out.
# Loading the cluster framework requires that a script by this name exists
# somewhere in the ZEEKPATH. The only thing in the file should be the
# cluster definition in the :zeek:id:`Cluster::nodes` variable.
@load cluster-layout
@if ( Cluster::node in Cluster::nodes )
@load ./setup-connections
@if ( Cluster::local_node_type() == Cluster::MANAGER )
@load ./nodes/manager
# If no logger is defined, then the manager receives logs.
@if ( Cluster::manager_is_logger )
@load ./nodes/logger
@endif
@endif
@if ( Cluster::local_node_type() == Cluster::LOGGER )
@load ./nodes/logger
@endif
@if ( Cluster::local_node_type() == Cluster::PROXY )
@load ./nodes/proxy
@endif
@if ( Cluster::local_node_type() == Cluster::WORKER )
@load ./nodes/worker
@endif
@endif
@endif

459
scripts/base/frameworks/cluster/main.bro
View file

@ -1,459 +0,0 @@
##! A framework for establishing and controlling a cluster of Bro instances.
##! In order to use the cluster framework, a script named
##! ``cluster-layout.bro`` must exist somewhere in Bro's script search path
##! which has a cluster definition of the :bro:id:`Cluster::nodes` variable.
##! The ``CLUSTER_NODE`` environment variable or :bro:id:`Cluster::node`
##! must also be sent and the cluster framework loaded as a package like
##! ``@load base/frameworks/cluster``.
@load base/frameworks/control
@load base/frameworks/broker
module Cluster;
export {
## Whether to distribute log messages among available logging nodes.
const enable_round_robin_logging = T &redef;
## The topic name used for exchanging messages that are relevant to
## logger nodes in a cluster. Used with broker-enabled cluster communication.
const logger_topic = "bro/cluster/logger" &redef;
## The topic name used for exchanging messages that are relevant to
## manager nodes in a cluster. Used with broker-enabled cluster communication.
const manager_topic = "bro/cluster/manager" &redef;
## The topic name used for exchanging messages that are relevant to
## proxy nodes in a cluster. Used with broker-enabled cluster communication.
const proxy_topic = "bro/cluster/proxy" &redef;
## The topic name used for exchanging messages that are relevant to
## worker nodes in a cluster. Used with broker-enabled cluster communication.
const worker_topic = "bro/cluster/worker" &redef;
## The topic name used for exchanging messages that are relevant to
## time machine nodes in a cluster. Used with broker-enabled cluster communication.
const time_machine_topic = "bro/cluster/time_machine" &redef;
## The topic prefix used for exchanging messages that are relevant to
## a named node in a cluster. Used with broker-enabled cluster communication.
const node_topic_prefix = "bro/cluster/node/" &redef;
## The topic prefix used for exchanging messages that are relevant to
## a unique node in a cluster. Used with broker-enabled cluster communication.
const nodeid_topic_prefix = "bro/cluster/nodeid/" &redef;
## Name of the node on which master data stores will be created if no other
## has already been specified by the user in :bro:see:`Cluster::stores`.
## An empty value means "use whatever name corresponds to the manager
## node".
const default_master_node = "" &redef;
## The type of data store backend that will be used for all data stores if
## no other has already been specified by the user in :bro:see:`Cluster::stores`.
const default_backend = Broker::MEMORY &redef;
## The type of persistent data store backend that will be used for all data
## stores if no other has already been specified by the user in
## :bro:see:`Cluster::stores`. This will be used when script authors call
## :bro:see:`Cluster::create_store` with the *persistent* argument set true.
const default_persistent_backend = Broker::SQLITE &redef;
## Setting a default dir will, for persistent backends that have not
## been given an explicit file path via :bro:see:`Cluster::stores`,
## automatically create a path within this dir that is based on the name of
## the data store.
const default_store_dir = "" &redef;
## Information regarding a cluster-enabled data store.
type StoreInfo: record {
## The name of the data store.
name: string &optional;
## The store handle.
store: opaque of Broker::Store &optional;
## The name of the cluster node on which the master version of the data
## store resides.
master_node: string &default=default_master_node;
## Whether the data store is the master version or a clone.
master: bool &default=F;
## The type of backend used for storing data.
backend: Broker::BackendType &default=default_backend;
## Parameters used for configuring the backend.
options: Broker::BackendOptions &default=Broker::BackendOptions();
## A resync/reconnect interval to pass through to
## :bro:see:`Broker::create_clone`.
clone_resync_interval: interval &default=Broker::default_clone_resync_interval;
## A staleness duration to pass through to
## :bro:see:`Broker::create_clone`.
clone_stale_interval: interval &default=Broker::default_clone_stale_interval;
## A mutation buffer interval to pass through to
## :bro:see:`Broker::create_clone`.
clone_mutation_buffer_interval: interval &default=Broker::default_clone_mutation_buffer_interval;
};
## A table of cluster-enabled data stores that have been created, indexed
## by their name. This table will be populated automatically by
## :bro:see:`Cluster::create_store`, but if you need to customize
## the options related to a particular data store, you may redef this
## table. Calls to :bro:see:`Cluster::create_store` will first check
## the table for an entry of the same name and, if found, will use the
## predefined options there when setting up the store.
global stores: table[string] of StoreInfo &default=StoreInfo() &redef;
## Sets up a cluster-enabled data store. They will also still properly
## function for uses that are not operating a cluster.
##
## name: the name of the data store to create.
##
## persistent: whether the data store must be persistent.
##
## Returns: the store's information. For master stores, the store will be
## ready to use immediately. For clones, the store field will not
## be set until the node containing the master store has connected.
global create_store: function(name: string, persistent: bool &default=F): StoreInfo;
## The cluster logging stream identifier.
redef enum Log::ID += { LOG };
## The record type which contains the column fields of the cluster log.
type Info: record {
## The time at which a cluster message was generated.
ts: time;
## The name of the node that is creating the log record.
node: string;
## A message indicating information about the cluster's operation.
message: string;
} &log;
## Types of nodes that are allowed to participate in the cluster
## configuration.
type NodeType: enum {
## A dummy node type indicating the local node is not operating
## within a cluster.
NONE,
## A node type which is allowed to view/manipulate the configuration
## of other nodes in the cluster.
CONTROL,
## A node type responsible for log management.
LOGGER,
## A node type responsible for policy management.
MANAGER,
## A node type for relaying worker node communication and synchronizing
## worker node state.
PROXY,
## The node type doing all the actual traffic analysis.
WORKER,
## A node acting as a traffic recorder using the
## `Time Machine <https://www.zeek.org/community/time-machine.html>`_
## software.
TIME_MACHINE,
};
## Record type to indicate a node in a cluster.
type Node: record {
## Identifies the type of cluster node in this node's configuration.
node_type: NodeType;
## The IP address of the cluster node.
ip: addr;
## If the *ip* field is a non-global IPv6 address, this field
## can specify a particular :rfc:`4007` ``zone_id``.
zone_id: string &default="";
## The port that this node will listen on for peer connections.
p: port;
## Identifier for the interface a worker is sniffing.
interface: string &optional;
## Name of the manager node this node uses. For workers and proxies.
manager: string &optional;
## Name of a time machine node with which this node connects.
time_machine: string &optional;
## A unique identifier assigned to the node by the broker framework.
## This field is only set while a node is connected.
id: string &optional;
};
## This function can be called at any time to determine if the cluster
## framework is being enabled for this run.
##
## Returns: True if :bro:id:`Cluster::node` has been set.
global is_enabled: function(): bool;
## This function can be called at any time to determine what type of
## cluster node the current Bro instance is going to be acting as.
## If :bro:id:`Cluster::is_enabled` returns false, then
## :bro:enum:`Cluster::NONE` is returned.
##
## Returns: The :bro:type:`Cluster::NodeType` the calling node acts as.
global local_node_type: function(): NodeType;
## This gives the value for the number of workers currently connected to,
## and it's maintained internally by the cluster framework. It's
## primarily intended for use by managers to find out how many workers
## should be responding to requests.
global worker_count: count = 0;
## The cluster layout definition. This should be placed into a filter
## named cluster-layout.bro somewhere in the BROPATH. It will be
## automatically loaded if the CLUSTER_NODE environment variable is set.
## Note that BroControl handles all of this automatically.
## The table is typically indexed by node names/labels (e.g. "manager"
## or "worker-1").
const nodes: table[string] of Node = {} &redef;
## Indicates whether or not the manager will act as the logger and receive
## logs. This value should be set in the cluster-layout.bro script (the
## value should be true only if no logger is specified in Cluster::nodes).
## Note that BroControl handles this automatically.
const manager_is_logger = T &redef;
## This is usually supplied on the command line for each instance
## of the cluster that is started up.
const node = getenv("CLUSTER_NODE") &redef;
## Interval for retrying failed connections between cluster nodes.
## If set, the BRO_DEFAULT_CONNECT_RETRY (given in number of seconds)
## overrides this option.
const retry_interval = 1min &redef;
## When using broker-enabled cluster framework, nodes broadcast this event
## to exchange their user-defined name along with a string that uniquely
## identifies it for the duration of its lifetime. This string may change
## if the node dies and has to reconnect later.
global hello: event(name: string, id: string);
## When using broker-enabled cluster framework, this event will be emitted
## locally whenever a cluster node connects or reconnects.
global node_up: event(name: string, id: string);
## When using broker-enabled cluster framework, this event will be emitted
## locally whenever a connected cluster node becomes disconnected.
global node_down: event(name: string, id: string);
## Write a message to the cluster logging stream.
global log: function(msg: string);
## Retrieve the topic associated with a specific node in the cluster.
##
## name: the name of the cluster node (e.g. "manager").
##
## Returns: a topic string that may used to send a message exclusively to
## a given cluster node.
global node_topic: function(name: string): string;
## Retrieve the topic associated with a specific node in the cluster.
##
## id: the id of the cluster node (from :bro:see:`Broker::EndpointInfo`
## or :bro:see:`Broker::node_id`.
##
## Returns: a topic string that may used to send a message exclusively to
## a given cluster node.
global nodeid_topic: function(id: string): string;
}
global active_worker_ids: set[string] = set();
type NamedNode: record {
name: string;
node: Node;
};
function nodes_with_type(node_type: NodeType): vector of NamedNode
{
local rval: vector of NamedNode = vector();
local names: vector of string = vector();
for ( name in Cluster::nodes )
names += name;
names = sort(names, strcmp);
for ( i in names )
{
name = names[i];
local n = Cluster::nodes[name];
if ( n$node_type != node_type )
next;
rval += NamedNode($name=name, $node=n);
}
return rval;
}
function is_enabled(): bool
{
return (node != "");
}
function local_node_type(): NodeType
{
return is_enabled() ? nodes[node]$node_type : NONE;
}
function node_topic(name: string): string
{
return node_topic_prefix + name;
}
function nodeid_topic(id: string): string
{
return node_topic_prefix + id;
}
event Cluster::hello(name: string, id: string) &priority=10
{
if ( name !in nodes )
{
Reporter::error(fmt("Got Cluster::hello msg from unexpected node: %s", name));
return;
}
local n = nodes[name];
if ( n?$id )
{
if ( n$id != id )
Reporter::error(fmt("Got Cluster::hello msg from duplicate node:%s",
name));
}
else
event Cluster::node_up(name, id);
n$id = id;
Cluster::log(fmt("got hello from %s (%s)", name, id));
if ( n$node_type == WORKER )
{
add active_worker_ids[id];
worker_count = |active_worker_ids|;
}
}
event Broker::peer_added(endpoint: Broker::EndpointInfo, msg: string) &priority=10
{
if ( ! Cluster::is_enabled() )
return;
local e = Broker::make_event(Cluster::hello, node, Broker::node_id());
Broker::publish(nodeid_topic(endpoint$id), e);
}
event Broker::peer_lost(endpoint: Broker::EndpointInfo, msg: string) &priority=10
{
for ( node_name, n in nodes )
{
if ( n?$id && n$id == endpoint$id )
{
Cluster::log(fmt("node down: %s", node_name));
delete n$id;
if ( n$node_type == WORKER )
{
delete active_worker_ids[endpoint$id];
worker_count = |active_worker_ids|;
}
event Cluster::node_down(node_name, endpoint$id);
break;
}
}
}
event bro_init() &priority=5
{
# If a node is given, but it's an unknown name we need to fail.
if ( node != "" && node !in nodes )
{
Reporter::error(fmt("'%s' is not a valid node in the Cluster::nodes configuration", node));
terminate();
}
Log::create_stream(Cluster::LOG, [$columns=Info, $path="cluster"]);
}
function create_store(name: string, persistent: bool &default=F): Cluster::StoreInfo
{
local info = stores[name];
info$name = name;
if ( Cluster::default_store_dir != "" )
{
local default_options = Broker::BackendOptions();
local path = Cluster::default_store_dir + "/" + name;
if ( info$options$sqlite$path == default_options$sqlite$path )
info$options$sqlite$path = path + ".sqlite";
if ( info$options$rocksdb$path == default_options$rocksdb$path )
info$options$rocksdb$path = path + ".rocksdb";
}
if ( persistent )
{
switch ( info$backend ) {
case Broker::MEMORY:
info$backend = Cluster::default_persistent_backend;
break;
case Broker::SQLITE:
fallthrough;
case Broker::ROCKSDB:
# no-op: user already asked for a specific persistent backend.
break;
default:
Reporter::error(fmt("unhandled data store type: %s", info$backend));
break;
}
}
if ( ! Cluster::is_enabled() )
{
if ( info?$store )
{
Reporter::warning(fmt("duplicate cluster store creation for %s", name));
return info;
}
info$store = Broker::create_master(name, info$backend, info$options);
info$master = T;
stores[name] = info;
return info;
}
if ( info$master_node == "" )
{
local mgr_nodes = nodes_with_type(Cluster::MANAGER);
if ( |mgr_nodes| == 0 )
Reporter::fatal(fmt("empty master node name for cluster store " +
"'%s', but there's no manager node to default",
name));
info$master_node = mgr_nodes[0]$name;
}
else if ( info$master_node !in Cluster::nodes )
Reporter::fatal(fmt("master node '%s' for cluster store '%s' does not exist",
info$master_node, name));
if ( Cluster::node == info$master_node )
{
info$store = Broker::create_master(name, info$backend, info$options);
info$master = T;
stores[name] = info;
Cluster::log(fmt("created master store: %s", name));
return info;
}
info$master = F;
stores[name] = info;
info$store = Broker::create_clone(info$name,
info$clone_resync_interval,
info$clone_stale_interval,
info$clone_mutation_buffer_interval);
Cluster::log(fmt("created clone store: %s", info$name));
return info;
}
function log(msg: string)
{
Log::write(Cluster::LOG, [$ts = network_time(), $node = node, $message = msg]);
}

459
scripts/base/frameworks/cluster/main.zeek Normal file
View file

@ -0,0 +1,459 @@
##! A framework for establishing and controlling a cluster of Zeek instances.
##! In order to use the cluster framework, a script named
##! ``cluster-layout.zeek`` must exist somewhere in Zeek's script search path
##! which has a cluster definition of the :zeek:id:`Cluster::nodes` variable.
##! The ``CLUSTER_NODE`` environment variable or :zeek:id:`Cluster::node`
##! must also be sent and the cluster framework loaded as a package like
##! ``@load base/frameworks/cluster``.
@load base/frameworks/control
@load base/frameworks/broker
module Cluster;
export {
## Whether to distribute log messages among available logging nodes.
const enable_round_robin_logging = T &redef;
## The topic name used for exchanging messages that are relevant to
## logger nodes in a cluster. Used with broker-enabled cluster communication.
const logger_topic = "zeek/cluster/logger" &redef;
## The topic name used for exchanging messages that are relevant to
## manager nodes in a cluster. Used with broker-enabled cluster communication.
const manager_topic = "zeek/cluster/manager" &redef;
## The topic name used for exchanging messages that are relevant to
## proxy nodes in a cluster. Used with broker-enabled cluster communication.
const proxy_topic = "zeek/cluster/proxy" &redef;
## The topic name used for exchanging messages that are relevant to
## worker nodes in a cluster. Used with broker-enabled cluster communication.
const worker_topic = "zeek/cluster/worker" &redef;
## The topic name used for exchanging messages that are relevant to
## time machine nodes in a cluster. Used with broker-enabled cluster communication.
const time_machine_topic = "zeek/cluster/time_machine" &redef;
## The topic prefix used for exchanging messages that are relevant to
## a named node in a cluster. Used with broker-enabled cluster communication.
const node_topic_prefix = "zeek/cluster/node/" &redef;
## The topic prefix used for exchanging messages that are relevant to
## a unique node in a cluster. Used with broker-enabled cluster communication.
const nodeid_topic_prefix = "zeek/cluster/nodeid/" &redef;
## Name of the node on which master data stores will be created if no other
## has already been specified by the user in :zeek:see:`Cluster::stores`.
## An empty value means "use whatever name corresponds to the manager
## node".
const default_master_node = "" &redef;
## The type of data store backend that will be used for all data stores if
## no other has already been specified by the user in :zeek:see:`Cluster::stores`.
const default_backend = Broker::MEMORY &redef;
## The type of persistent data store backend that will be used for all data
## stores if no other has already been specified by the user in
## :zeek:see:`Cluster::stores`. This will be used when script authors call
## :zeek:see:`Cluster::create_store` with the *persistent* argument set true.
const default_persistent_backend = Broker::SQLITE &redef;
## Setting a default dir will, for persistent backends that have not
## been given an explicit file path via :zeek:see:`Cluster::stores`,
## automatically create a path within this dir that is based on the name of
## the data store.
const default_store_dir = "" &redef;
## Information regarding a cluster-enabled data store.
type StoreInfo: record {
## The name of the data store.
name: string &optional;
## The store handle.
store: opaque of Broker::Store &optional;
## The name of the cluster node on which the master version of the data
## store resides.
master_node: string &default=default_master_node;
## Whether the data store is the master version or a clone.
master: bool &default=F;
## The type of backend used for storing data.
backend: Broker::BackendType &default=default_backend;
## Parameters used for configuring the backend.
options: Broker::BackendOptions &default=Broker::BackendOptions();
## A resync/reconnect interval to pass through to
## :zeek:see:`Broker::create_clone`.
clone_resync_interval: interval &default=Broker::default_clone_resync_interval;
## A staleness duration to pass through to
## :zeek:see:`Broker::create_clone`.
clone_stale_interval: interval &default=Broker::default_clone_stale_interval;
## A mutation buffer interval to pass through to
## :zeek:see:`Broker::create_clone`.
clone_mutation_buffer_interval: interval &default=Broker::default_clone_mutation_buffer_interval;
};
## A table of cluster-enabled data stores that have been created, indexed
## by their name. This table will be populated automatically by
## :zeek:see:`Cluster::create_store`, but if you need to customize
## the options related to a particular data store, you may redef this
## table. Calls to :zeek:see:`Cluster::create_store` will first check
## the table for an entry of the same name and, if found, will use the
## predefined options there when setting up the store.
global stores: table[string] of StoreInfo &default=StoreInfo() &redef;
## Sets up a cluster-enabled data store. They will also still properly
## function for uses that are not operating a cluster.
##
## name: the name of the data store to create.
##
## persistent: whether the data store must be persistent.
##
## Returns: the store's information. For master stores, the store will be
## ready to use immediately. For clones, the store field will not
## be set until the node containing the master store has connected.
global create_store: function(name: string, persistent: bool &default=F): StoreInfo;
## The cluster logging stream identifier.
redef enum Log::ID += { LOG };
## The record type which contains the column fields of the cluster log.
type Info: record {
## The time at which a cluster message was generated.
ts: time;
## The name of the node that is creating the log record.
node: string;
## A message indicating information about the cluster's operation.
message: string;
} &log;
## Types of nodes that are allowed to participate in the cluster
## configuration.
type NodeType: enum {
## A dummy node type indicating the local node is not operating
## within a cluster.
NONE,
## A node type which is allowed to view/manipulate the configuration
## of other nodes in the cluster.
CONTROL,
## A node type responsible for log management.
LOGGER,
## A node type responsible for policy management.
MANAGER,
## A node type for relaying worker node communication and synchronizing
## worker node state.
PROXY,
## The node type doing all the actual traffic analysis.
WORKER,
## A node acting as a traffic recorder using the
## `Time Machine <https://www.zeek.org/community/time-machine.html>`_
## software.
TIME_MACHINE,
};
## Record type to indicate a node in a cluster.
type Node: record {
## Identifies the type of cluster node in this node's configuration.
node_type: NodeType;
## The IP address of the cluster node.
ip: addr;
## If the *ip* field is a non-global IPv6 address, this field
## can specify a particular :rfc:`4007` ``zone_id``.
zone_id: string &default="";
## The port that this node will listen on for peer connections.
p: port;
## Identifier for the interface a worker is sniffing.
interface: string &optional;
## Name of the manager node this node uses. For workers and proxies.
manager: string &optional;
## Name of a time machine node with which this node connects.
time_machine: string &optional;
## A unique identifier assigned to the node by the broker framework.
## This field is only set while a node is connected.
id: string &optional;
};
## This function can be called at any time to determine if the cluster
## framework is being enabled for this run.
##
## Returns: True if :zeek:id:`Cluster::node` has been set.
global is_enabled: function(): bool;
## This function can be called at any time to determine what type of
## cluster node the current Zeek instance is going to be acting as.
## If :zeek:id:`Cluster::is_enabled` returns false, then
## :zeek:enum:`Cluster::NONE` is returned.
##
## Returns: The :zeek:type:`Cluster::NodeType` the calling node acts as.
global local_node_type: function(): NodeType;
## This gives the value for the number of workers currently connected to,
## and it's maintained internally by the cluster framework. It's
## primarily intended for use by managers to find out how many workers
## should be responding to requests.
global worker_count: count = 0;
## The cluster layout definition. This should be placed into a filter
## named cluster-layout.zeek somewhere in the ZEEKPATH. It will be
## automatically loaded if the CLUSTER_NODE environment variable is set.
## Note that ZeekControl handles all of this automatically.
## The table is typically indexed by node names/labels (e.g. "manager"
## or "worker-1").
const nodes: table[string] of Node = {} &redef;
## Indicates whether or not the manager will act as the logger and receive
## logs. This value should be set in the cluster-layout.zeek script (the
## value should be true only if no logger is specified in Cluster::nodes).
## Note that ZeekControl handles this automatically.
const manager_is_logger = T &redef;
## This is usually supplied on the command line for each instance
## of the cluster that is started up.
const node = getenv("CLUSTER_NODE") &redef;
## Interval for retrying failed connections between cluster nodes.
## If set, the ZEEK_DEFAULT_CONNECT_RETRY (given in number of seconds)
## environment variable overrides this option.
const retry_interval = 1min &redef;
## When using broker-enabled cluster framework, nodes broadcast this event
## to exchange their user-defined name along with a string that uniquely
## identifies it for the duration of its lifetime. This string may change
## if the node dies and has to reconnect later.
global hello: event(name: string, id: string);
## When using broker-enabled cluster framework, this event will be emitted
## locally whenever a cluster node connects or reconnects.
global node_up: event(name: string, id: string);
## When using broker-enabled cluster framework, this event will be emitted
## locally whenever a connected cluster node becomes disconnected.
global node_down: event(name: string, id: string);
## Write a message to the cluster logging stream.
global log: function(msg: string);
## Retrieve the topic associated with a specific node in the cluster.
##
## name: the name of the cluster node (e.g. "manager").
##
## Returns: a topic string that may used to send a message exclusively to
## a given cluster node.
global node_topic: function(name: string): string;
## Retrieve the topic associated with a specific node in the cluster.
##
## id: the id of the cluster node (from :zeek:see:`Broker::EndpointInfo`
## or :zeek:see:`Broker::node_id`.
##
## Returns: a topic string that may used to send a message exclusively to
## a given cluster node.
global nodeid_topic: function(id: string): string;
}
global active_worker_ids: set[string] = set();
type NamedNode: record {
name: string;
node: Node;
};
function nodes_with_type(node_type: NodeType): vector of NamedNode
{
local rval: vector of NamedNode = vector();
local names: vector of string = vector();
for ( name in Cluster::nodes )
names += name;
names = sort(names, strcmp);
for ( i in names )
{
name = names[i];
local n = Cluster::nodes[name];
if ( n$node_type != node_type )
next;
rval += NamedNode($name=name, $node=n);
}
return rval;
}
function is_enabled(): bool
{
return (node != "");
}
function local_node_type(): NodeType
{
return is_enabled() ? nodes[node]$node_type : NONE;
}
function node_topic(name: string): string
{
return node_topic_prefix + name;
}
function nodeid_topic(id: string): string
{
return node_topic_prefix + id;
}
event Cluster::hello(name: string, id: string) &priority=10
{
if ( name !in nodes )
{
Reporter::error(fmt("Got Cluster::hello msg from unexpected node: %s", name));
return;
}
local n = nodes[name];
if ( n?$id )
{
if ( n$id != id )
Reporter::error(fmt("Got Cluster::hello msg from duplicate node:%s",
name));
}
else
event Cluster::node_up(name, id);
n$id = id;
Cluster::log(fmt("got hello from %s (%s)", name, id));
if ( n$node_type == WORKER )
{
add active_worker_ids[id];
worker_count = |active_worker_ids|;
}
}
event Broker::peer_added(endpoint: Broker::EndpointInfo, msg: string) &priority=10
{
if ( ! Cluster::is_enabled() )
return;
local e = Broker::make_event(Cluster::hello, node, Broker::node_id());
Broker::publish(nodeid_topic(endpoint$id), e);
}
event Broker::peer_lost(endpoint: Broker::EndpointInfo, msg: string) &priority=10
{
for ( node_name, n in nodes )
{
if ( n?$id && n$id == endpoint$id )
{
Cluster::log(fmt("node down: %s", node_name));
delete n$id;
if ( n$node_type == WORKER )
{
delete active_worker_ids[endpoint$id];
worker_count = |active_worker_ids|;
}
event Cluster::node_down(node_name, endpoint$id);
break;
}
}
}
event zeek_init() &priority=5
{
# If a node is given, but it's an unknown name we need to fail.
if ( node != "" && node !in nodes )
{
Reporter::error(fmt("'%s' is not a valid node in the Cluster::nodes configuration", node));
terminate();
}
Log::create_stream(Cluster::LOG, [$columns=Info, $path="cluster"]);
}
function create_store(name: string, persistent: bool &default=F): Cluster::StoreInfo
{
local info = stores[name];
info$name = name;
if ( Cluster::default_store_dir != "" )
{
local default_options = Broker::BackendOptions();
local path = Cluster::default_store_dir + "/" + name;
if ( info$options$sqlite$path == default_options$sqlite$path )
info$options$sqlite$path = path + ".sqlite";
if ( info$options$rocksdb$path == default_options$rocksdb$path )
info$options$rocksdb$path = path + ".rocksdb";
}
if ( persistent )
{
switch ( info$backend ) {
case Broker::MEMORY:
info$backend = Cluster::default_persistent_backend;
break;
case Broker::SQLITE:
fallthrough;
case Broker::ROCKSDB:
# no-op: user already asked for a specific persistent backend.
break;
default:
Reporter::error(fmt("unhandled data store type: %s", info$backend));
break;
}
}
if ( ! Cluster::is_enabled() )
{
if ( info?$store )
{
Reporter::warning(fmt("duplicate cluster store creation for %s", name));
return info;
}
info$store = Broker::create_master(name, info$backend, info$options);
info$master = T;
stores[name] = info;
return info;
}
if ( info$master_node == "" )
{
local mgr_nodes = nodes_with_type(Cluster::MANAGER);
if ( |mgr_nodes| == 0 )
Reporter::fatal(fmt("empty master node name for cluster store " +
"'%s', but there's no manager node to default",
name));
info$master_node = mgr_nodes[0]$name;
}
else if ( info$master_node !in Cluster::nodes )
Reporter::fatal(fmt("master node '%s' for cluster store '%s' does not exist",
info$master_node, name));
if ( Cluster::node == info$master_node )
{
info$store = Broker::create_master(name, info$backend, info$options);
info$master = T;
stores[name] = info;
Cluster::log(fmt("created master store: %s", name));
return info;
}
info$master = F;
stores[name] = info;
info$store = Broker::create_clone(info$name,
info$clone_resync_interval,
info$clone_stale_interval,
info$clone_mutation_buffer_interval);
Cluster::log(fmt("created clone store: %s", info$name));
return info;
}
function log(msg: string)
{
Log::write(Cluster::LOG, [$ts = network_time(), $node = node, $message = msg]);
}

29
scripts/base/frameworks/cluster/nodes/logger.bro
View file

@ -1,29 +0,0 @@
##! This is the core Bro script to support the notion of a cluster logger.
##!
##! The logger is passive (other Bro instances connect to us), and once
##! connected the logger receives logs from other Bro instances.
##! This script will be automatically loaded if necessary based on the
##! type of node being started.
##! This is where the cluster logger sets it's specific settings for other
##! frameworks and in the core.
@prefixes += cluster-logger
## Turn on local logging.
redef Log::enable_local_logging = T;
## Turn off remote logging since this is the logger and should only log here.
redef Log::enable_remote_logging = F;
## Log rotation interval.
redef Log::default_rotation_interval = 1 hrs;
## Alarm summary mail interval.
redef Log::default_mail_alarms_interval = 24 hrs;
## Use the cluster's archive logging script.
redef Log::default_rotation_postprocessor_cmd = "archive-log";
## We're processing essentially *only* remote events.
redef max_remote_events_processed = 10000;

26
scripts/base/frameworks/cluster/nodes/logger.zeek Normal file
View file

@ -0,0 +1,26 @@
##! This is the core Zeek script to support the notion of a cluster logger.
##!
##! The logger is passive (other Zeek instances connect to us), and once
##! connected the logger receives logs from other Zeek instances.
##! This script will be automatically loaded if necessary based on the
##! type of node being started.
##! This is where the cluster logger sets it's specific settings for other
##! frameworks and in the core.
@prefixes += cluster-logger
## Turn on local logging.
redef Log::enable_local_logging = T;
## Turn off remote logging since this is the logger and should only log here.
redef Log::enable_remote_logging = F;
## Log rotation interval.
redef Log::default_rotation_interval = 1 hrs;
## Alarm summary mail interval.
redef Log::default_mail_alarms_interval = 24 hrs;
## Use the cluster's archive logging script.
redef Log::default_rotation_postprocessor_cmd = "archive-log";

26
scripts/base/frameworks/cluster/nodes/manager.bro
View file

@ -1,26 +0,0 @@
##! This is the core Bro script to support the notion of a cluster manager.
##!
##! The manager is passive (the workers connect to us), and once connected
##! the manager registers for the events on the workers that are needed
##! to get the desired data from the workers. This script will be
##! automatically loaded if necessary based on the type of node being started.
##! This is where the cluster manager sets it's specific settings for other
##! frameworks and in the core.
@prefixes += cluster-manager
## Don't do any local logging since the logger handles writing logs.
redef Log::enable_local_logging = F;
## Turn on remote logging since the logger handles writing logs.
redef Log::enable_remote_logging = T;
## Log rotation interval.
redef Log::default_rotation_interval = 24 hrs;
## Use the cluster's delete-log script.
redef Log::default_rotation_postprocessor_cmd = "delete-log";
## We're processing essentially *only* remote events.
redef max_remote_events_processed = 10000;

23
scripts/base/frameworks/cluster/nodes/manager.zeek Normal file
View file

@ -0,0 +1,23 @@
##! This is the core Zeek script to support the notion of a cluster manager.
##!
##! The manager is passive (the workers connect to us), and once connected
##! the manager registers for the events on the workers that are needed
##! to get the desired data from the workers. This script will be
##! automatically loaded if necessary based on the type of node being started.
##! This is where the cluster manager sets it's specific settings for other
##! frameworks and in the core.
@prefixes += cluster-manager
## Don't do any local logging since the logger handles writing logs.
redef Log::enable_local_logging = F;
## Turn on remote logging since the logger handles writing logs.
redef Log::enable_remote_logging = T;
## Log rotation interval.
redef Log::default_rotation_interval = 24 hrs;
## Use the cluster's delete-log script.
redef Log::default_rotation_postprocessor_cmd = "delete-log";

22
scripts/base/frameworks/cluster/nodes/proxy.bro
View file

@ -1,22 +0,0 @@
##! Redefines the options common to all proxy nodes within a Bro cluster.
##! In particular, proxies are not meant to produce logs locally and they
##! do not forward events anywhere, they mainly synchronize state between
##! worker nodes.
@prefixes += cluster-proxy
## The proxy only syncs state; does not forward events.
redef forward_remote_events = F;
redef forward_remote_state_changes = T;
## Don't do any local logging.
redef Log::enable_local_logging = F;
## Make sure that remote logging is enabled.
redef Log::enable_remote_logging = T;
redef Log::default_rotation_interval = 24hrs;
## Use the cluster's delete-log script.
redef Log::default_rotation_postprocessor_cmd = "delete-log";

18
scripts/base/frameworks/cluster/nodes/proxy.zeek Normal file
View file

@ -0,0 +1,18 @@
##! Redefines the options common to all proxy nodes within a Zeek cluster.
##! In particular, proxies are not meant to produce logs locally and they
##! do not forward events anywhere, they mainly synchronize state between
##! worker nodes.
@prefixes += cluster-proxy
## Don't do any local logging.
redef Log::enable_local_logging = F;
## Make sure that remote logging is enabled.
redef Log::enable_remote_logging = T;
redef Log::default_rotation_interval = 24hrs;
## Use the cluster's delete-log script.
redef Log::default_rotation_postprocessor_cmd = "delete-log";

24
scripts/base/frameworks/cluster/nodes/worker.bro
View file

@ -1,24 +0,0 @@
##! Redefines some options common to all worker nodes within a Bro cluster.
##! In particular, worker nodes do not produce logs locally, instead they
##! send them off to a logger node for processing.
@prefixes += cluster-worker
## Don't do any local logging.
redef Log::enable_local_logging = F;
## Make sure that remote logging is enabled.
redef Log::enable_remote_logging = T;
redef Log::default_rotation_interval = 24hrs;
## Use the cluster's delete-log script.
redef Log::default_rotation_postprocessor_cmd = "delete-log";
@load misc/trim-trace-file
## Record all packets into trace file.
##
## Note that this only indicates that *if* we are recording packets, we want all
## of them (rather than just those the core deems sufficiently important).
## Setting this does not turn recording on. Use '-w <trace>' for that.
redef record_all_packets = T;

24
scripts/base/frameworks/cluster/nodes/worker.zeek Normal file
View file

@ -0,0 +1,24 @@
##! Redefines some options common to all worker nodes within a Zeek cluster.
##! In particular, worker nodes do not produce logs locally, instead they
##! send them off to a logger node for processing.
@prefixes += cluster-worker
## Don't do any local logging.
redef Log::enable_local_logging = F;
## Make sure that remote logging is enabled.
redef Log::enable_remote_logging = T;
redef Log::default_rotation_interval = 24hrs;
## Use the cluster's delete-log script.
redef Log::default_rotation_postprocessor_cmd = "delete-log";
@load misc/trim-trace-file
## Record all packets into trace file.
##
## Note that this only indicates that *if* we are recording packets, we want all
## of them (rather than just those the core deems sufficiently important).
## Setting this does not turn recording on. Use '-w <trace>' for that.
redef record_all_packets = T;

458
scripts/base/frameworks/cluster/pools.bro
View file

@ -1,458 +0,0 @@
##! Defines an interface for managing pools of cluster nodes. Pools are
##! a useful way to distribute work or data among nodes within a cluster.
@load ./main
@load base/utils/hash_hrw
module Cluster;
export {
## Store state of a cluster within the context of a work pool.
type PoolNode: record {
## The node name (e.g. "manager").
name: string;
## An alias of *name* used to prevent hashing collisions when creating
## *site_id*.
alias: string;
## A 32-bit unique identifier for the pool node, derived from name/alias.
site_id: count;
## Whether the node is currently alive and can receive work.
alive: bool &default=F;
};
## A pool specification.
type PoolSpec: record {
## A topic string that can be used to reach all nodes within a pool.
topic: string &default = "";
## The type of nodes that are contained within the pool.
node_type: Cluster::NodeType &default = Cluster::PROXY;
## The maximum number of nodes that may belong to the pool.
## If not set, then all available nodes will be added to the pool,
## else the cluster framework will automatically limit the pool
## membership according to the threshhold.
max_nodes: count &optional;
## Whether the pool requires exclusive access to nodes. If true,
## then *max_nodes* nodes will not be assigned to any other pool.
## When using this flag, *max_nodes* must also be set.
exclusive: bool &default = F;
};
type PoolNodeTable: table[string] of PoolNode;
type RoundRobinTable: table[string] of int;
## A pool used for distributing data/work among a set of cluster nodes.
type Pool: record {
## The specification of the pool that was used when registering it.
spec: PoolSpec &default = PoolSpec();
## Nodes in the pool, indexed by their name (e.g. "manager").
nodes: PoolNodeTable &default = PoolNodeTable();
## A list of nodes in the pool in a deterministic order.
node_list: vector of PoolNode &default = vector();
## The Rendezvous hashing structure.
hrw_pool: HashHRW::Pool &default = HashHRW::Pool();
## Round-Robin table indexed by arbitrary key and storing the next
## index of *node_list* that will be eligible to receive work (if it's
## alive at the time of next request).
rr_key_seq: RoundRobinTable &default = RoundRobinTable();
## Number of pool nodes that are currently alive.
alive_count: count &default = 0;
};
## The specification for :bro:see:`Cluster::proxy_pool`.
global proxy_pool_spec: PoolSpec =
PoolSpec($topic = "bro/cluster/pool/proxy",
$node_type = Cluster::PROXY) &redef;
## The specification for :bro:see:`Cluster::worker_pool`.
global worker_pool_spec: PoolSpec =
PoolSpec($topic = "bro/cluster/pool/worker",
$node_type = Cluster::WORKER) &redef;
## The specification for :bro:see:`Cluster::logger_pool`.
global logger_pool_spec: PoolSpec =
PoolSpec($topic = "bro/cluster/pool/logger",
$node_type = Cluster::LOGGER) &redef;
## A pool containing all the proxy nodes of a cluster.
## The pool's node membership/availability is automatically
## maintained by the cluster framework.
global proxy_pool: Pool;
## A pool containing all the worker nodes of a cluster.
## The pool's node membership/availability is automatically
## maintained by the cluster framework.
global worker_pool: Pool;
## A pool containing all the logger nodes of a cluster.
## The pool's node membership/availability is automatically
## maintained by the cluster framework.
global logger_pool: Pool;
## Registers and initializes a pool.
global register_pool: function(spec: PoolSpec): Pool;
## Retrieve the topic associated with the node mapped via Rendezvous hash
## of an arbitrary key.
##
## pool: the pool of nodes to consider.
##
## key: data used for input to the hashing function that will uniformly
## distribute keys among available nodes.
##
## Returns: a topic string associated with a cluster node that is alive
## or an empty string if nothing is alive.
global hrw_topic: function(pool: Pool, key: any): string;
## Retrieve the topic associated with the node in a round-robin fashion.
##
## pool: the pool of nodes to consider.
##
## key: an arbitrary string to identify the purpose for which you're
## requesting the topic. e.g. consider using a name-spaced key
## like "Intel::cluster_rr_key" if you need to guarantee that
## a group of messages get distributed in a well-defined pattern
## without other messages being interleaved within the round-robin.
## Usually sharing the default key is fine for load-balancing
## purposes.
##
## Returns: a topic string associated with a cluster node that is alive,
## or an empty string if nothing is alive.
global rr_topic: function(pool: Pool, key: string &default=""): string;
## Distributes log message topics among logger nodes via round-robin.
## This will be automatically assigned to :bro:see:`Broker::log_topic`
## if :bro:see:`Cluster::enable_round_robin_logging` is enabled.
## If no logger nodes are active, then this will return the value
## of :bro:see:`Broker::default_log_topic`.
global rr_log_topic: function(id: Log::ID, path: string): string;
}
## Initialize a node as a member of a pool.
##
## pool: the pool to which the node will belong.
##
## name: the name of the node (e.g. "manager").
##
## Returns: F if a node of the same name already exists in the pool, else T.
global init_pool_node: function(pool: Pool, name: string): bool;
## Mark a pool node as alive/online/available. :bro:see:`Cluster::hrw_topic`
## will distribute keys to nodes marked as alive.
##
## pool: the pool to which the node belongs.
##
## name: the name of the node to mark.
##
## Returns: F if the node does not exist in the pool, else T.
global mark_pool_node_alive: function(pool: Pool, name: string): bool;
## Mark a pool node as dead/offline/unavailable. :bro:see:`Cluster::hrw_topic`
## will not distribute keys to nodes marked as dead.
##
## pool: the pool to which the node belongs.
##
## name: the name of the node to mark.
##
## Returns: F if the node does not exist in the pool, else T.
global mark_pool_node_dead: function(pool: Pool, name: string): bool;
global registered_pools: vector of Pool = vector();
function register_pool(spec: PoolSpec): Pool
{
local rval = Pool($spec = spec);
registered_pools += rval;
return rval;
}
function hrw_topic(pool: Pool, key: any): string
{
if ( |pool$hrw_pool$sites| == 0 )
return "";
local site = HashHRW::get_site(pool$hrw_pool, key);
local pn: PoolNode = site$user_data;
return node_topic_prefix + pn$name;
}
function rr_topic(pool: Pool, key: string): string
{
if ( key !in pool$rr_key_seq )
pool$rr_key_seq[key] = 0;
local next_idx = pool$rr_key_seq[key];
local start = next_idx;
local rval = "";
if ( next_idx >= |pool$node_list| )
return rval;
while ( T )
{
local pn = pool$node_list[next_idx];
++next_idx;
if ( next_idx == |pool$node_list| )
next_idx = 0;
if ( pn$alive )
{
rval = node_topic_prefix + pn$name;
break;
}
if ( next_idx == start )
# no nodes alive
break;
}
pool$rr_key_seq[key] = next_idx;
return rval;
}
function rr_log_topic(id: Log::ID, path: string): string
{
local rval = rr_topic(logger_pool, "Cluster::rr_log_topic");
if ( rval != "" )
return rval;
rval = Broker::default_log_topic(id, path);
return rval;
}
event Cluster::node_up(name: string, id: string) &priority=10
{
for ( i in registered_pools )
{
local pool = registered_pools[i];
if ( name in pool$nodes )
mark_pool_node_alive(pool, name);
}
}
event Cluster::node_down(name: string, id: string) &priority=10
{
for ( i in registered_pools )
{
local pool = registered_pools[i];
if ( name in pool$nodes )
mark_pool_node_dead(pool, name);
}
}
function site_id_in_pool(pool: Pool, site_id: count): bool
{
for ( i, pn in pool$nodes )
{
if ( pn$site_id == site_id )
return T;
}
return F;
}
function init_pool_node(pool: Pool, name: string): bool
{
if ( name in pool$nodes )
return F;
local loop = T;
local c = 0;
while ( loop )
{
# site id collisions are unlikely, but using aliases handles it...
# alternatively could terminate and ask user to pick a new node name
# if it ends up colliding.
local alias = name + fmt(".%s", c);
local site_id = fnv1a32(alias);
if ( site_id_in_pool(pool, site_id) )
++c;
else
{
local pn = PoolNode($name=name, $alias=alias, $site_id=site_id,
$alive=Cluster::node == name);
pool$nodes[name] = pn;
pool$node_list += pn;
if ( pn$alive )
++pool$alive_count;
loop = F;
}
}
return T;
}
function mark_pool_node_alive(pool: Pool, name: string): bool
{
if ( name !in pool$nodes )
return F;
local pn = pool$nodes[name];
if ( ! pn$alive )
{
pn$alive = T;
++pool$alive_count;
}
HashHRW::add_site(pool$hrw_pool, HashHRW::Site($id=pn$site_id, $user_data=pn));
return T;
}
function mark_pool_node_dead(pool: Pool, name: string): bool
{
if ( name !in pool$nodes )
return F;
local pn = pool$nodes[name];
if ( pn$alive )
{
pn$alive = F;
--pool$alive_count;
}
HashHRW::rem_site(pool$hrw_pool, HashHRW::Site($id=pn$site_id, $user_data=pn));
return T;
}
event bro_init()
{
worker_pool = register_pool(worker_pool_spec);
proxy_pool = register_pool(proxy_pool_spec);
logger_pool = register_pool(logger_pool_spec);
}
type PoolEligibilityTracking: record {
eligible_nodes: vector of NamedNode &default = vector();
next_idx: count &default = 0;
excluded: count &default = 0;
};
global pool_eligibility: table[Cluster::NodeType] of PoolEligibilityTracking = table();
function pool_sorter(a: Pool, b: Pool): int
{
return strcmp(a$spec$topic, b$spec$topic);
}
# Needs to execute before the bro_init in setup-connections
event bro_init() &priority=-5
{
if ( ! Cluster::is_enabled() )
return;
# Sorting now ensures the node distribution process is stable even if
# there's a change in the order of time-of-registration between Bro runs.
sort(registered_pools, pool_sorter);
pool_eligibility[Cluster::WORKER] =
PoolEligibilityTracking($eligible_nodes = nodes_with_type(Cluster::WORKER));
pool_eligibility[Cluster::PROXY] =
PoolEligibilityTracking($eligible_nodes = nodes_with_type(Cluster::PROXY));
pool_eligibility[Cluster::LOGGER] =
PoolEligibilityTracking($eligible_nodes = nodes_with_type(Cluster::LOGGER));
if ( manager_is_logger )
{
local mgr = nodes_with_type(Cluster::MANAGER);
if ( |mgr| > 0 )
{
local eln = pool_eligibility[Cluster::LOGGER]$eligible_nodes;
eln += mgr[0];
}
}
local pool: Pool;
local pet: PoolEligibilityTracking;
local en: vector of NamedNode;
for ( i in registered_pools )
{
pool = registered_pools[i];
if ( pool$spec$node_type !in pool_eligibility )
Reporter::fatal(fmt("invalid pool node type: %s", pool$spec$node_type));
if ( ! pool$spec$exclusive )
next;
if ( ! pool$spec?$max_nodes )
Reporter::fatal("Cluster::PoolSpec 'max_nodes' field must be set when using the 'exclusive' flag");
pet = pool_eligibility[pool$spec$node_type];
pet$excluded += pool$spec$max_nodes;
}
for ( nt, pet in pool_eligibility )
{
if ( pet$excluded > |pet$eligible_nodes| )
Reporter::fatal(fmt("not enough %s nodes to satisfy pool exclusivity requirements: need %d nodes", nt, pet$excluded));
}
for ( i in registered_pools )
{
pool = registered_pools[i];
if ( ! pool$spec$exclusive )
next;
pet = pool_eligibility[pool$spec$node_type];
local e = 0;
while ( e < pool$spec$max_nodes )
{
init_pool_node(pool, pet$eligible_nodes[e]$name);
++e;
}
local nen: vector of NamedNode = vector();
for ( j in pet$eligible_nodes )
{
if ( j < e )
next;
nen += pet$eligible_nodes[j];
}
pet$eligible_nodes = nen;
}
for ( i in registered_pools )
{
pool = registered_pools[i];
if ( pool$spec$exclusive )
next;
pet = pool_eligibility[pool$spec$node_type];
local nodes_to_init = |pet$eligible_nodes|;
if ( pool$spec?$max_nodes &&
pool$spec$max_nodes < |pet$eligible_nodes| )
nodes_to_init = pool$spec$max_nodes;
local nodes_inited = 0;
while ( nodes_inited < nodes_to_init )
{
init_pool_node(pool, pet$eligible_nodes[pet$next_idx]$name);
++nodes_inited;
++pet$next_idx;
if ( pet$next_idx == |pet$eligible_nodes| )
pet$next_idx = 0;
}
}
}

458
scripts/base/frameworks/cluster/pools.zeek Normal file
View file

@ -0,0 +1,458 @@
##! Defines an interface for managing pools of cluster nodes. Pools are
##! a useful way to distribute work or data among nodes within a cluster.
@load ./main
@load base/utils/hash_hrw
module Cluster;
export {
## Store state of a cluster within the context of a work pool.
type PoolNode: record {
## The node name (e.g. "manager").
name: string;
## An alias of *name* used to prevent hashing collisions when creating
## *site_id*.
alias: string;
## A 32-bit unique identifier for the pool node, derived from name/alias.
site_id: count;
## Whether the node is currently alive and can receive work.
alive: bool &default=F;
};
## A pool specification.
type PoolSpec: record {
## A topic string that can be used to reach all nodes within a pool.
topic: string &default = "";
## The type of nodes that are contained within the pool.
node_type: Cluster::NodeType &default = Cluster::PROXY;
## The maximum number of nodes that may belong to the pool.
## If not set, then all available nodes will be added to the pool,
## else the cluster framework will automatically limit the pool
## membership according to the threshhold.
max_nodes: count &optional;
## Whether the pool requires exclusive access to nodes. If true,
## then *max_nodes* nodes will not be assigned to any other pool.
## When using this flag, *max_nodes* must also be set.
exclusive: bool &default = F;
};
type PoolNodeTable: table[string] of PoolNode;
type RoundRobinTable: table[string] of int;
## A pool used for distributing data/work among a set of cluster nodes.
type Pool: record {
## The specification of the pool that was used when registering it.
spec: PoolSpec &default = PoolSpec();
## Nodes in the pool, indexed by their name (e.g. "manager").
nodes: PoolNodeTable &default = PoolNodeTable();
## A list of nodes in the pool in a deterministic order.
node_list: vector of PoolNode &default = vector();
## The Rendezvous hashing structure.
hrw_pool: HashHRW::Pool &default = HashHRW::Pool();
## Round-Robin table indexed by arbitrary key and storing the next
## index of *node_list* that will be eligible to receive work (if it's
## alive at the time of next request).
rr_key_seq: RoundRobinTable &default = RoundRobinTable();
## Number of pool nodes that are currently alive.
alive_count: count &default = 0;
};
## The specification for :zeek:see:`Cluster::proxy_pool`.
global proxy_pool_spec: PoolSpec =
PoolSpec($topic = "zeek/cluster/pool/proxy",
$node_type = Cluster::PROXY) &redef;
## The specification for :zeek:see:`Cluster::worker_pool`.
global worker_pool_spec: PoolSpec =
PoolSpec($topic = "zeek/cluster/pool/worker",
$node_type = Cluster::WORKER) &redef;
## The specification for :zeek:see:`Cluster::logger_pool`.
global logger_pool_spec: PoolSpec =
PoolSpec($topic = "zeek/cluster/pool/logger",
$node_type = Cluster::LOGGER) &redef;
## A pool containing all the proxy nodes of a cluster.
## The pool's node membership/availability is automatically
## maintained by the cluster framework.
global proxy_pool: Pool;
## A pool containing all the worker nodes of a cluster.
## The pool's node membership/availability is automatically
## maintained by the cluster framework.
global worker_pool: Pool;
## A pool containing all the logger nodes of a cluster.
## The pool's node membership/availability is automatically
## maintained by the cluster framework.
global logger_pool: Pool;
## Registers and initializes a pool.
global register_pool: function(spec: PoolSpec): Pool;
## Retrieve the topic associated with the node mapped via Rendezvous hash
## of an arbitrary key.
##
## pool: the pool of nodes to consider.
##
## key: data used for input to the hashing function that will uniformly
## distribute keys among available nodes.
##
## Returns: a topic string associated with a cluster node that is alive
## or an empty string if nothing is alive.
global hrw_topic: function(pool: Pool, key: any): string;
## Retrieve the topic associated with the node in a round-robin fashion.
##
## pool: the pool of nodes to consider.
##
## key: an arbitrary string to identify the purpose for which you're
## requesting the topic. e.g. consider using a name-spaced key
## like "Intel::cluster_rr_key" if you need to guarantee that
## a group of messages get distributed in a well-defined pattern
## without other messages being interleaved within the round-robin.
## Usually sharing the default key is fine for load-balancing
## purposes.
##
## Returns: a topic string associated with a cluster node that is alive,
## or an empty string if nothing is alive.
global rr_topic: function(pool: Pool, key: string &default=""): string;
## Distributes log message topics among logger nodes via round-robin.
## This will be automatically assigned to :zeek:see:`Broker::log_topic`
## if :zeek:see:`Cluster::enable_round_robin_logging` is enabled.
## If no logger nodes are active, then this will return the value
## of :zeek:see:`Broker::default_log_topic`.
global rr_log_topic: function(id: Log::ID, path: string): string;
}
## Initialize a node as a member of a pool.
##
## pool: the pool to which the node will belong.
##
## name: the name of the node (e.g. "manager").
##
## Returns: F if a node of the same name already exists in the pool, else T.
global init_pool_node: function(pool: Pool, name: string): bool;
## Mark a pool node as alive/online/available. :zeek:see:`Cluster::hrw_topic`
## will distribute keys to nodes marked as alive.
##
## pool: the pool to which the node belongs.
##
## name: the name of the node to mark.
##
## Returns: F if the node does not exist in the pool, else T.
global mark_pool_node_alive: function(pool: Pool, name: string): bool;
## Mark a pool node as dead/offline/unavailable. :zeek:see:`Cluster::hrw_topic`
## will not distribute keys to nodes marked as dead.
##
## pool: the pool to which the node belongs.
##
## name: the name of the node to mark.
##
## Returns: F if the node does not exist in the pool, else T.
global mark_pool_node_dead: function(pool: Pool, name: string): bool;
global registered_pools: vector of Pool = vector();
function register_pool(spec: PoolSpec): Pool
{
local rval = Pool($spec = spec);
registered_pools += rval;
return rval;
}
function hrw_topic(pool: Pool, key: any): string
{
if ( |pool$hrw_pool$sites| == 0 )
return "";
local site = HashHRW::get_site(pool$hrw_pool, key);
local pn: PoolNode = site$user_data;
return node_topic_prefix + pn$name;
}
function rr_topic(pool: Pool, key: string): string
{
if ( key !in pool$rr_key_seq )
pool$rr_key_seq[key] = 0;
local next_idx = pool$rr_key_seq[key];
local start = next_idx;
local rval = "";
if ( next_idx >= |pool$node_list| )
return rval;
while ( T )
{
local pn = pool$node_list[next_idx];
++next_idx;
if ( next_idx == |pool$node_list| )
next_idx = 0;
if ( pn$alive )
{
rval = node_topic_prefix + pn$name;
break;
}
if ( next_idx == start )
# no nodes alive
break;
}
pool$rr_key_seq[key] = next_idx;
return rval;
}
function rr_log_topic(id: Log::ID, path: string): string
{
local rval = rr_topic(logger_pool, "Cluster::rr_log_topic");
if ( rval != "" )
return rval;
rval = Broker::default_log_topic(id, path);
return rval;
}
event Cluster::node_up(name: string, id: string) &priority=10
{
for ( i in registered_pools )
{
local pool = registered_pools[i];
if ( name in pool$nodes )
mark_pool_node_alive(pool, name);
}
}
event Cluster::node_down(name: string, id: string) &priority=10
{
for ( i in registered_pools )
{
local pool = registered_pools[i];
if ( name in pool$nodes )
mark_pool_node_dead(pool, name);
}
}
function site_id_in_pool(pool: Pool, site_id: count): bool
{
for ( i, pn in pool$nodes )
{
if ( pn$site_id == site_id )
return T;
}
return F;
}
function init_pool_node(pool: Pool, name: string): bool
{
if ( name in pool$nodes )
return F;
local loop = T;
local c = 0;
while ( loop )
{
# site id collisions are unlikely, but using aliases handles it...
# alternatively could terminate and ask user to pick a new node name
# if it ends up colliding.
local alias = name + fmt(".%s", c);
local site_id = fnv1a32(alias);
if ( site_id_in_pool(pool, site_id) )
++c;
else
{
local pn = PoolNode($name=name, $alias=alias, $site_id=site_id,
$alive=Cluster::node == name);
pool$nodes[name] = pn;
pool$node_list += pn;
if ( pn$alive )
++pool$alive_count;
loop = F;
}
}
return T;
}
function mark_pool_node_alive(pool: Pool, name: string): bool
{
if ( name !in pool$nodes )
return F;
local pn = pool$nodes[name];
if ( ! pn$alive )
{
pn$alive = T;
++pool$alive_count;
}
HashHRW::add_site(pool$hrw_pool, HashHRW::Site($id=pn$site_id, $user_data=pn));
return T;
}
function mark_pool_node_dead(pool: Pool, name: string): bool
{
if ( name !in pool$nodes )
return F;
local pn = pool$nodes[name];
if ( pn$alive )
{
pn$alive = F;
--pool$alive_count;
}
HashHRW::rem_site(pool$hrw_pool, HashHRW::Site($id=pn$site_id, $user_data=pn));
return T;
}
event zeek_init()
{
worker_pool = register_pool(worker_pool_spec);
proxy_pool = register_pool(proxy_pool_spec);
logger_pool = register_pool(logger_pool_spec);
}
type PoolEligibilityTracking: record {
eligible_nodes: vector of NamedNode &default = vector();
next_idx: count &default = 0;
excluded: count &default = 0;
};
global pool_eligibility: table[Cluster::NodeType] of PoolEligibilityTracking = table();
function pool_sorter(a: Pool, b: Pool): int
{
return strcmp(a$spec$topic, b$spec$topic);
}
# Needs to execute before the zeek_init in setup-connections
event zeek_init() &priority=-5
{
if ( ! Cluster::is_enabled() )
return;
# Sorting now ensures the node distribution process is stable even if
# there's a change in the order of time-of-registration between Zeek runs.
sort(registered_pools, pool_sorter);
pool_eligibility[Cluster::WORKER] =
PoolEligibilityTracking($eligible_nodes = nodes_with_type(Cluster::WORKER));
pool_eligibility[Cluster::PROXY] =
PoolEligibilityTracking($eligible_nodes = nodes_with_type(Cluster::PROXY));
pool_eligibility[Cluster::LOGGER] =
PoolEligibilityTracking($eligible_nodes = nodes_with_type(Cluster::LOGGER));
if ( manager_is_logger )
{
local mgr = nodes_with_type(Cluster::MANAGER);
if ( |mgr| > 0 )
{
local eln = pool_eligibility[Cluster::LOGGER]$eligible_nodes;
eln += mgr[0];
}
}
local pool: Pool;
local pet: PoolEligibilityTracking;
local en: vector of NamedNode;
for ( i in registered_pools )
{
pool = registered_pools[i];
if ( pool$spec$node_type !in pool_eligibility )
Reporter::fatal(fmt("invalid pool node type: %s", pool$spec$node_type));
if ( ! pool$spec$exclusive )
next;
if ( ! pool$spec?$max_nodes )
Reporter::fatal("Cluster::PoolSpec 'max_nodes' field must be set when using the 'exclusive' flag");
pet = pool_eligibility[pool$spec$node_type];
pet$excluded += pool$spec$max_nodes;
}
for ( nt, pet in pool_eligibility )
{
if ( pet$excluded > |pet$eligible_nodes| )
Reporter::fatal(fmt("not enough %s nodes to satisfy pool exclusivity requirements: need %d nodes", nt, pet$excluded));
}
for ( i in registered_pools )
{
pool = registered_pools[i];
if ( ! pool$spec$exclusive )
next;
pet = pool_eligibility[pool$spec$node_type];
local e = 0;
while ( e < pool$spec$max_nodes )
{
init_pool_node(pool, pet$eligible_nodes[e]$name);
++e;
}
local nen: vector of NamedNode = vector();
for ( j in pet$eligible_nodes )
{
if ( j < e )
next;
nen += pet$eligible_nodes[j];
}
pet$eligible_nodes = nen;
}
for ( i in registered_pools )
{
pool = registered_pools[i];
if ( pool$spec$exclusive )
next;
pet = pool_eligibility[pool$spec$node_type];
local nodes_to_init = |pet$eligible_nodes|;
if ( pool$spec?$max_nodes &&
pool$spec$max_nodes < |pet$eligible_nodes| )
nodes_to_init = pool$spec$max_nodes;
local nodes_inited = 0;
while ( nodes_inited < nodes_to_init )
{
init_pool_node(pool, pet$eligible_nodes[pet$next_idx]$name);
++nodes_inited;
++pet$next_idx;
if ( pet$next_idx == |pet$eligible_nodes| )
pet$next_idx = 0;
}
}
}

126
scripts/base/frameworks/cluster/setup-connections.bro
View file

@ -1,126 +0,0 @@
##! This script establishes communication among all nodes in a cluster
##! as defined by :bro:id:`Cluster::nodes`.
@load ./main
@load ./pools
@load base/frameworks/broker
module Cluster;
function connect_peer(node_type: NodeType, node_name: string)
{
local nn = nodes_with_type(node_type);
for ( i in nn )
{
local n = nn[i];
if ( n$name != node_name )
next;
local status = Broker::peer(cat(n$node$ip), n$node$p,
Cluster::retry_interval);
Cluster::log(fmt("initiate peering with %s:%s, retry=%s, status=%s",
n$node$ip, n$node$p, Cluster::retry_interval,
status));
}
}
function connect_peers_with_type(node_type: NodeType)
{
local rval: vector of NamedNode = vector();
local nn = nodes_with_type(node_type);
for ( i in nn )
{
local n = nn[i];
local status = Broker::peer(cat(n$node$ip), n$node$p,
Cluster::retry_interval);
Cluster::log(fmt("initiate peering with %s:%s, retry=%s, status=%s",
n$node$ip, n$node$p, Cluster::retry_interval,
status));
}
}
event bro_init() &priority=-10
{
if ( getenv("BROCTL_CHECK_CONFIG") != "" )
return;
local self = nodes[node];
for ( i in registered_pools )
{
local pool = registered_pools[i];
if ( node in pool$nodes )
Broker::subscribe(pool$spec$topic);
}
switch ( self$node_type ) {
case NONE:
return;
case CONTROL:
break;
case LOGGER:
Broker::subscribe(Cluster::logger_topic);
Broker::subscribe(Broker::default_log_topic_prefix);
break;
case MANAGER:
Broker::subscribe(Cluster::manager_topic);
if ( Cluster::manager_is_logger )
Broker::subscribe(Broker::default_log_topic_prefix);
break;
case PROXY:
Broker::subscribe(Cluster::proxy_topic);
break;
case WORKER:
Broker::subscribe(Cluster::worker_topic);
break;
case TIME_MACHINE:
Broker::subscribe(Cluster::time_machine_topic);
break;
default:
Reporter::error(fmt("Unhandled cluster node type: %s", self$node_type));
return;
}
Broker::subscribe(nodeid_topic(Broker::node_id()));
Broker::subscribe(node_topic(node));
Broker::listen(Broker::default_listen_address,
self$p,
Broker::default_listen_retry);
Cluster::log(fmt("listening on %s:%s", Broker::default_listen_address, self$p));
switch ( self$node_type ) {
case MANAGER:
connect_peers_with_type(LOGGER);
if ( self?$time_machine )
connect_peer(TIME_MACHINE, self$time_machine);
break;
case PROXY:
connect_peers_with_type(LOGGER);
if ( self?$manager )
connect_peer(MANAGER, self$manager);
break;
case WORKER:
connect_peers_with_type(LOGGER);
connect_peers_with_type(PROXY);
if ( self?$manager )
connect_peer(MANAGER, self$manager);
if ( self?$time_machine )
connect_peer(TIME_MACHINE, self$time_machine);
break;
}
}

126
scripts/base/frameworks/cluster/setup-connections.zeek Normal file
View file

@ -0,0 +1,126 @@
##! This script establishes communication among all nodes in a cluster
##! as defined by :zeek:id:`Cluster::nodes`.
@load ./main
@load ./pools
@load base/frameworks/broker
module Cluster;
function connect_peer(node_type: NodeType, node_name: string)
{
local nn = nodes_with_type(node_type);
for ( i in nn )
{
local n = nn[i];
if ( n$name != node_name )
next;
local status = Broker::peer(cat(n$node$ip), n$node$p,
Cluster::retry_interval);
Cluster::log(fmt("initiate peering with %s:%s, retry=%s, status=%s",
n$node$ip, n$node$p, Cluster::retry_interval,
status));
}
}
function connect_peers_with_type(node_type: NodeType)
{
local rval: vector of NamedNode = vector();
local nn = nodes_with_type(node_type);
for ( i in nn )
{
local n = nn[i];
local status = Broker::peer(cat(n$node$ip), n$node$p,
Cluster::retry_interval);
Cluster::log(fmt("initiate peering with %s:%s, retry=%s, status=%s",
n$node$ip, n$node$p, Cluster::retry_interval,
status));
}
}
event zeek_init() &priority=-10
{
if ( getenv("ZEEKCTL_CHECK_CONFIG") != "" )
return;
local self = nodes[node];
for ( i in registered_pools )
{
local pool = registered_pools[i];
if ( node in pool$nodes )
Broker::subscribe(pool$spec$topic);
}
switch ( self$node_type ) {
case NONE:
return;
case CONTROL:
break;
case LOGGER:
Broker::subscribe(Cluster::logger_topic);
Broker::subscribe(Broker::default_log_topic_prefix);
break;
case MANAGER:
Broker::subscribe(Cluster::manager_topic);
if ( Cluster::manager_is_logger )
Broker::subscribe(Broker::default_log_topic_prefix);
break;
case PROXY:
Broker::subscribe(Cluster::proxy_topic);
break;
case WORKER:
Broker::subscribe(Cluster::worker_topic);
break;
case TIME_MACHINE:
Broker::subscribe(Cluster::time_machine_topic);
break;
default:
Reporter::error(fmt("Unhandled cluster node type: %s", self$node_type));
return;
}
Broker::subscribe(nodeid_topic(Broker::node_id()));
Broker::subscribe(node_topic(node));
Broker::listen(Broker::default_listen_address,
self$p,
Broker::default_listen_retry);
Cluster::log(fmt("listening on %s:%s", Broker::default_listen_address, self$p));
switch ( self$node_type ) {
case MANAGER:
connect_peers_with_type(LOGGER);
if ( self?$time_machine )
connect_peer(TIME_MACHINE, self$time_machine);
break;
case PROXY:
connect_peers_with_type(LOGGER);
if ( self?$manager )
connect_peer(MANAGER, self$manager);
break;
case WORKER:
connect_peers_with_type(LOGGER);
connect_peers_with_type(PROXY);
if ( self?$manager )
connect_peer(MANAGER, self$manager);
if ( self?$time_machine )
connect_peer(TIME_MACHINE, self$time_machine);
break;
}
}

2
scripts/base/frameworks/config/README
View file

@ -1,2 +1,2 @@
The configuration framework provides a way to change the Bro configuration
The configuration framework provides a way to change the Zeek configuration
in "option" values at run-time.

0
scripts/base/frameworks/config/__load__.bro → scripts/base/frameworks/config/__load__.zeek
View file

77
scripts/base/frameworks/config/input.bro
View file

@ -1,77 +0,0 @@
##! File input for the configuration framework using the input framework.
@load ./main
@load base/frameworks/cluster
module Config;
export {
## Configuration files that will be read off disk. Files are reread
## every time they are updated so updates should be atomic with "mv"
## instead of writing the file in place.
##
## If the same configuration option is defined in several files with
## different values, behavior is unspecified.
const config_files: set[string] = {} &redef;
## Read specified configuration file and apply values; updates to file
## are not tracked.
global read_config: function(filename: string);
}
global current_config: table[string] of string = table();
type ConfigItem: record {
option_nv: string;
};
type EventFields: record {
option_name: string;
option_val: string;
};
event config_line(description: Input::EventDescription, tpe: Input::Event, p: EventFields)
{
}
event bro_init() &priority=5
{
if ( Cluster::is_enabled() && Cluster::local_node_type() != Cluster::MANAGER )
return;
for ( fi in config_files )
Input::add_table([$reader=Input::READER_CONFIG,
$mode=Input::REREAD,
$source=fi,
$name=cat("config-", fi),
$idx=ConfigItem,
$val=ConfigItem,
$want_record=F,
$destination=current_config]);
}
event InputConfig::new_value(name: string, source: string, id: string, value: any)
{
if ( sub_bytes(name, 1, 15) != "config-oneshot-" && source !in config_files )
return;
Config::set_value(id, value, source);
}
function read_config(filename: string)
{
# Only read the configuration on the manager. The other nodes are being fed
# from the manager.
if ( Cluster::is_enabled() && Cluster::local_node_type() != Cluster::MANAGER )
return;
local iname = cat("config-oneshot-", filename);
Input::add_event([$reader=Input::READER_CONFIG,
$mode=Input::MANUAL,
$source=filename,
$name=iname,
$fields=EventFields,
$ev=config_line]);
Input::remove(iname);
}

77
scripts/base/frameworks/config/input.zeek Normal file
View file

@ -0,0 +1,77 @@
##! File input for the configuration framework using the input framework.
@load ./main
@load base/frameworks/cluster
module Config;
export {
## Configuration files that will be read off disk. Files are reread
## every time they are updated so updates should be atomic with "mv"
## instead of writing the file in place.
##
## If the same configuration option is defined in several files with
## different values, behavior is unspecified.
const config_files: set[string] = {} &redef;
## Read specified configuration file and apply values; updates to file
## are not tracked.
global read_config: function(filename: string);
}
global current_config: table[string] of string = table();
type ConfigItem: record {
option_nv: string;
};
type EventFields: record {
option_name: string;
option_val: string;
};
event config_line(description: Input::EventDescription, tpe: Input::Event, p: EventFields)
{
}
event zeek_init() &priority=5
{
if ( Cluster::is_enabled() && Cluster::local_node_type() != Cluster::MANAGER )
return;
for ( fi in config_files )
Input::add_table([$reader=Input::READER_CONFIG,
$mode=Input::REREAD,
$source=fi,
$name=cat("config-", fi),
$idx=ConfigItem,
$val=ConfigItem,
$want_record=F,
$destination=current_config]);
}
event InputConfig::new_value(name: string, source: string, id: string, value: any)
{
if ( sub_bytes(name, 1, 15) != "config-oneshot-" && source !in config_files )
return;
Config::set_value(id, value, source);
}
function read_config(filename: string)
{
# Only read the configuration on the manager. The other nodes are being fed
# from the manager.
if ( Cluster::is_enabled() && Cluster::local_node_type() != Cluster::MANAGER )
return;
local iname = cat("config-oneshot-", filename);
Input::add_event([$reader=Input::READER_CONFIG,
$mode=Input::MANUAL,
$source=filename,
$name=iname,
$fields=EventFields,
$ev=config_line]);
Input::remove(iname);
}

170
scripts/base/frameworks/config/main.bro
View file

@ -1,170 +0,0 @@
##! The configuration framework provides a way to change Bro options
##! (as specified by the "option" keyword) at runtime. It also logs runtime
##! changes to options to config.log.
@load base/frameworks/cluster
module Config;
export {
## The config logging stream identifier.
redef enum Log::ID += { LOG };
## Represents the data in config.log.
type Info: record {
## Timestamp at which the configuration change occured.
ts: time &log;
## ID of the value that was changed.
id: string &log;
## Value before the change.
old_value: string &log;
## Value after the change.
new_value: string &log;
## Optional location that triggered the change.
location: string &optional &log;
};
## Event that can be handled to access the :bro:type:`Config::Info`
## record as it is sent on to the logging framework.
global log_config: event(rec: Info);
## This function is the config framework layer around the lower-level
## :bro:see:`Option::set` call. Config::set_value will set the configuration
## value for all nodes in the cluster, no matter where it was called. Note
## that :bro:see:`Option::set` does not distribute configuration changes
## to other nodes.
##
## ID: The ID of the option to update.
##
## val: The new value of the option.
##
## location: Optional parameter detailing where this change originated from.
##
## Returns: true on success, false when an error occurs.
global set_value: function(ID: string, val: any, location: string &default = "" &optional): bool;
}
@if ( Cluster::is_enabled() )
type OptionCacheValue: record {
val: any;
location: string;
};
global option_cache: table[string] of OptionCacheValue;
global Config::cluster_set_option: event(ID: string, val: any, location: string);
function broadcast_option(ID: string, val: any, location: string)
{
# There's not currently a common topic to broadcast to as then enabling
# implicit Broker forwarding would cause a routing loop.
Broker::publish(Cluster::worker_topic, Config::cluster_set_option,
ID, val, location);
Broker::publish(Cluster::proxy_topic, Config::cluster_set_option,
ID, val, location);
Broker::publish(Cluster::logger_topic, Config::cluster_set_option,
ID, val, location);
}
event Config::cluster_set_option(ID: string, val: any, location: string)
{
@if ( Cluster::local_node_type() == Cluster::MANAGER )
option_cache[ID] = OptionCacheValue($val=val, $location=location);
broadcast_option(ID, val, location);
@endif
Option::set(ID, val, location);
}
function set_value(ID: string, val: any, location: string &default = "" &optional): bool
{
# Always copy the value to break references -- if caller mutates their
# value afterwards, we still guarantee the option has not changed. If
# one wants it to change, they need to explicitly call Option::set_value
# or Option::set with the intended value at the time of the call.
val = copy(val);
# First try setting it locally - abort if not possible.
if ( ! Option::set(ID, val, location) )
return F;
@if ( Cluster::local_node_type() == Cluster::MANAGER )
option_cache[ID] = OptionCacheValue($val=val, $location=location);
broadcast_option(ID, val, location);
@else
Broker::publish(Cluster::manager_topic, Config::cluster_set_option,
ID, val, location);
@endif
return T;
}
@else # Standalone implementation
function set_value(ID: string, val: any, location: string &default = "" &optional): bool
{
return Option::set(ID, val, location);
}
@endif # Cluster::is_enabled
@if ( Cluster::is_enabled() && Cluster::local_node_type() == Cluster::MANAGER )
# Handling of new worker nodes.
event Cluster::node_up(name: string, id: string) &priority=-10
{
# When a node connects, send it all current Option values.
if ( name in Cluster::nodes )
for ( ID in option_cache )
Broker::publish(Cluster::node_topic(name), Config::cluster_set_option, ID, option_cache[ID]$val, option_cache[ID]$location);
}
@endif
function format_value(value: any) : string
{
local tn = type_name(value);
local part: string_vec = vector();
if ( /^set/ in tn )
{
local it: set[bool] = value;
for ( sv in it )
part += cat(sv);
return join_string_vec(part, ",");
}
else if ( /^vector/ in tn )
{
local vit: vector of any = value;
for ( i in vit )
part += cat(vit[i]);
return join_string_vec(part, ",");
}
else if ( tn == "string" )
return value;
return cat(value);
}
function config_option_changed(ID: string, new_value: any, location: string): any
{
local log = Info($ts=network_time(), $id=ID, $old_value=format_value(lookup_ID(ID)), $new_value=format_value(new_value));
if ( location != "" )
log$location = location;
Log::write(LOG, log);
return new_value;
}
event bro_init() &priority=10
{
Log::create_stream(LOG, [$columns=Info, $ev=log_config, $path="config"]);
# Limit logging to the manager - everyone else just feeds off it.
@if ( !Cluster::is_enabled() || Cluster::local_node_type() == Cluster::MANAGER )
# Iterate over all existing options and add ourselves as change handlers
# with a low priority so that we can log the changes.
local gids = global_ids();
for ( i, gid in gids )
{
if ( ! gid$option_value )
next;
Option::set_change_handler(i, config_option_changed, -100);
}
@endif
}

170
scripts/base/frameworks/config/main.zeek Normal file
View file

@ -0,0 +1,170 @@
##! The configuration framework provides a way to change Zeek options
##! (as specified by the "option" keyword) at runtime. It also logs runtime
##! changes to options to config.log.
@load base/frameworks/cluster
module Config;
export {
## The config logging stream identifier.
redef enum Log::ID += { LOG };
## Represents the data in config.log.
type Info: record {
## Timestamp at which the configuration change occured.
ts: time &log;
## ID of the value that was changed.
id: string &log;
## Value before the change.
old_value: string &log;
## Value after the change.
new_value: string &log;
## Optional location that triggered the change.
location: string &optional &log;
};
## Event that can be handled to access the :zeek:type:`Config::Info`
## record as it is sent on to the logging framework.
global log_config: event(rec: Info);
## This function is the config framework layer around the lower-level
## :zeek:see:`Option::set` call. Config::set_value will set the configuration
## value for all nodes in the cluster, no matter where it was called. Note
## that :zeek:see:`Option::set` does not distribute configuration changes
## to other nodes.
##
## ID: The ID of the option to update.
##
## val: The new value of the option.
##
## location: Optional parameter detailing where this change originated from.
##
## Returns: true on success, false when an error occurs.
global set_value: function(ID: string, val: any, location: string &default = "" &optional): bool;
}
@if ( Cluster::is_enabled() )
type OptionCacheValue: record {
val: any;
location: string;
};
global option_cache: table[string] of OptionCacheValue;
global Config::cluster_set_option: event(ID: string, val: any, location: string);
function broadcast_option(ID: string, val: any, location: string)
{
# There's not currently a common topic to broadcast to as then enabling
# implicit Broker forwarding would cause a routing loop.
Broker::publish(Cluster::worker_topic, Config::cluster_set_option,
ID, val, location);
Broker::publish(Cluster::proxy_topic, Config::cluster_set_option,
ID, val, location);
Broker::publish(Cluster::logger_topic, Config::cluster_set_option,
ID, val, location);
}
event Config::cluster_set_option(ID: string, val: any, location: string)
{
@if ( Cluster::local_node_type() == Cluster::MANAGER )
option_cache[ID] = OptionCacheValue($val=val, $location=location);
broadcast_option(ID, val, location);
@endif
Option::set(ID, val, location);
}
function set_value(ID: string, val: any, location: string &default = "" &optional): bool
{
# Always copy the value to break references -- if caller mutates their
# value afterwards, we still guarantee the option has not changed. If
# one wants it to change, they need to explicitly call Option::set_value
# or Option::set with the intended value at the time of the call.
val = copy(val);
# First try setting it locally - abort if not possible.
if ( ! Option::set(ID, val, location) )
return F;
@if ( Cluster::local_node_type() == Cluster::MANAGER )
option_cache[ID] = OptionCacheValue($val=val, $location=location);
broadcast_option(ID, val, location);
@else
Broker::publish(Cluster::manager_topic, Config::cluster_set_option,
ID, val, location);
@endif
return T;
}
@else # Standalone implementation
function set_value(ID: string, val: any, location: string &default = "" &optional): bool
{
return Option::set(ID, val, location);
}
@endif # Cluster::is_enabled
@if ( Cluster::is_enabled() && Cluster::local_node_type() == Cluster::MANAGER )
# Handling of new worker nodes.
event Cluster::node_up(name: string, id: string) &priority=-10
{
# When a node connects, send it all current Option values.
if ( name in Cluster::nodes )
for ( ID in option_cache )
Broker::publish(Cluster::node_topic(name), Config::cluster_set_option, ID, option_cache[ID]$val, option_cache[ID]$location);
}
@endif
function format_value(value: any) : string
{
local tn = type_name(value);
local part: string_vec = vector();
if ( /^set/ in tn )
{
local it: set[bool] = value;
for ( sv in it )
part += cat(sv);
return join_string_vec(part, ",");
}
else if ( /^vector/ in tn )
{
local vit: vector of any = value;
for ( i in vit )
part += cat(vit[i]);
return join_string_vec(part, ",");
}
else if ( tn == "string" )
return value;
return cat(value);
}
function config_option_changed(ID: string, new_value: any, location: string): any
{
local log = Info($ts=network_time(), $id=ID, $old_value=format_value(lookup_ID(ID)), $new_value=format_value(new_value));
if ( location != "" )
log$location = location;
Log::write(LOG, log);
return new_value;
}
event zeek_init() &priority=10
{
Log::create_stream(LOG, [$columns=Info, $ev=log_config, $path="config"]);
# Limit logging to the manager - everyone else just feeds off it.
@if ( !Cluster::is_enabled() || Cluster::local_node_type() == Cluster::MANAGER )
# Iterate over all existing options and add ourselves as change handlers
# with a low priority so that we can log the changes.
local gids = global_ids();
for ( i, gid in gids )
{
if ( ! gid$option_value )
next;
Option::set_change_handler(i, config_option_changed, -100);
}
@endif
}

44
scripts/base/frameworks/config/weird.bro
View file

@ -1,44 +0,0 @@
##! This script sets up the config framework change handlers for weirds.
@load ./main
module Config;
function weird_option_change_sampling_whitelist(ID: string, new_value: string_set, location: string) : string_set
{
if ( ID == "Weird::sampling_whitelist" )
{
Reporter::set_weird_sampling_whitelist(new_value);
}
return new_value;
}
function weird_option_change_count(ID: string, new_value: count, location: string) : count
{
if ( ID == "Weird::sampling_threshold" )
{
Reporter::set_weird_sampling_threshold(new_value);
}
else if ( ID == "Weird::sampling_rate" )
{
Reporter::set_weird_sampling_rate(new_value);
}
return new_value;
}
function weird_option_change_interval(ID: string, new_value: interval, location: string) : interval
{
if ( ID == "Weird::sampling_duration" )
{
Reporter::set_weird_sampling_duration(new_value);
}
return new_value;
}
event bro_init() &priority=5
{
Option::set_change_handler("Weird::sampling_whitelist", weird_option_change_sampling_whitelist, 5);
Option::set_change_handler("Weird::sampling_threshold", weird_option_change_count, 5);
Option::set_change_handler("Weird::sampling_rate", weird_option_change_count, 5);
Option::set_change_handler("Weird::sampling_duration", weird_option_change_interval, 5);
}

44
scripts/base/frameworks/config/weird.zeek Normal file
View file

@ -0,0 +1,44 @@
##! This script sets up the config framework change handlers for weirds.
@load ./main
module Config;
function weird_option_change_sampling_whitelist(ID: string, new_value: string_set, location: string) : string_set
{
if ( ID == "Weird::sampling_whitelist" )
{
Reporter::set_weird_sampling_whitelist(new_value);
}
return new_value;
}
function weird_option_change_count(ID: string, new_value: count, location: string) : count
{
if ( ID == "Weird::sampling_threshold" )
{
Reporter::set_weird_sampling_threshold(new_value);
}
else if ( ID == "Weird::sampling_rate" )
{
Reporter::set_weird_sampling_rate(new_value);
}
return new_value;
}
function weird_option_change_interval(ID: string, new_value: interval, location: string) : interval
{
if ( ID == "Weird::sampling_duration" )
{
Reporter::set_weird_sampling_duration(new_value);
}
return new_value;
}
event zeek_init() &priority=5
{
Option::set_change_handler("Weird::sampling_whitelist", weird_option_change_sampling_whitelist, 5);
Option::set_change_handler("Weird::sampling_threshold", weird_option_change_count, 5);
Option::set_change_handler("Weird::sampling_rate", weird_option_change_count, 5);
Option::set_change_handler("Weird::sampling_duration", weird_option_change_interval, 5);
}

2
scripts/base/frameworks/control/README
View file

@ -1,3 +1,3 @@
The control framework provides the foundation for providing "commands"
that can be taken remotely at runtime to modify a running Bro instance
that can be taken remotely at runtime to modify a running Zeek instance
or collect information from the running instance.

0
scripts/base/frameworks/control/__load__.bro → scripts/base/frameworks/control/__load__.zeek
View file

80
scripts/base/frameworks/control/main.bro
View file

@ -1,80 +0,0 @@
##! The control framework provides the foundation for providing "commands"
##! that can be taken remotely at runtime to modify a running Bro instance
##! or collect information from the running instance.
module Control;
export {
## The topic prefix used for exchanging control messages via Broker.
const topic_prefix = "bro/control";
## Whether the controllee should call :bro:see:`Broker::listen`.
## In a cluster, this isn't needed since the setup process calls it.
const controllee_listen = T &redef;
## The address of the host that will be controlled.
const host = 0.0.0.0 &redef;
## The port of the host that will be controlled.
const host_port = 0/tcp &redef;
## If :bro:id:`Control::host` is a non-global IPv6 address and
## requires a specific :rfc:`4007` ``zone_id``, it can be set here.
const zone_id = "" &redef;
## The command that is being done. It's typically set on the
## command line.
const cmd = "" &redef;
## This can be used by commands that take an argument.
const arg = "" &redef;
## The commands that can currently be given on the command line for
## remote control.
const commands: set[string] = {
"id_value",
"peer_status",
"net_stats",
"configuration_update",
"shutdown",
} &redef;
## Variable IDs that are to be ignored by the update process.
const ignore_ids: set[string] = { };
## Event for requesting the value of an ID (a variable).
global id_value_request: event(id: string);
## Event for returning the value of an ID after an
## :bro:id:`Control::id_value_request` event.
global id_value_response: event(id: string, val: string);
## Requests the current communication status.
global peer_status_request: event();
## Returns the current communication status.
global peer_status_response: event(s: string);
## Requests the current net_stats.
global net_stats_request: event();
## Returns the current net_stats.
global net_stats_response: event(s: string);
## Inform the remote Bro instance that it's configuration may have been
## updated.
global configuration_update_request: event();
## This event is a wrapper and alias for the
## :bro:id:`Control::configuration_update_request` event.
## This event is also a primary hooking point for the control framework.
global configuration_update: event();
## Message in response to a configuration update request.
global configuration_update_response: event();
## Requests that the Bro instance begins shutting down.
global shutdown_request: event();
## Message in response to a shutdown request.
global shutdown_response: event();
}
event terminate_event()
{
terminate();
}

80
scripts/base/frameworks/control/main.zeek Normal file
View file

@ -0,0 +1,80 @@
##! The control framework provides the foundation for providing "commands"
##! that can be taken remotely at runtime to modify a running Zeek instance
##! or collect information from the running instance.
module Control;
export {
## The topic prefix used for exchanging control messages via Broker.
const topic_prefix = "zeek/control";
## Whether the controllee should call :zeek:see:`Broker::listen`.
## In a cluster, this isn't needed since the setup process calls it.
const controllee_listen = T &redef;
## The address of the host that will be controlled.
const host = 0.0.0.0 &redef;
## The port of the host that will be controlled.
const host_port = 0/tcp &redef;
## If :zeek:id:`Control::host` is a non-global IPv6 address and
## requires a specific :rfc:`4007` ``zone_id``, it can be set here.
const zone_id = "" &redef;
## The command that is being done. It's typically set on the
## command line.
const cmd = "" &redef;
## This can be used by commands that take an argument.
const arg = "" &redef;
## The commands that can currently be given on the command line for
## remote control.
const commands: set[string] = {
"id_value",
"peer_status",
"net_stats",
"configuration_update",
"shutdown",
} &redef;
## Variable IDs that are to be ignored by the update process.
const ignore_ids: set[string] = { };
## Event for requesting the value of an ID (a variable).
global id_value_request: event(id: string);
## Event for returning the value of an ID after an
## :zeek:id:`Control::id_value_request` event.
global id_value_response: event(id: string, val: string);
## Requests the current communication status.
global peer_status_request: event();
## Returns the current communication status.
global peer_status_response: event(s: string);
## Requests the current net_stats.
global net_stats_request: event();
## Returns the current net_stats.
global net_stats_response: event(s: string);
## Inform the remote Zeek instance that it's configuration may have been
## updated.
global configuration_update_request: event();
## This event is a wrapper and alias for the
## :zeek:id:`Control::configuration_update_request` event.
## This event is also a primary hooking point for the control framework.
global configuration_update: event();
## Message in response to a configuration update request.
global configuration_update_response: event();
## Requests that the Zeek instance begins shutting down.
global shutdown_request: event();
## Message in response to a shutdown request.
global shutdown_response: event();
}
event terminate_event()
{
terminate();
}

0
scripts/base/frameworks/analyzer/__load__.bro → scripts/base/frameworks/dpd/__load__.zeek
View file

104
scripts/base/frameworks/dpd/main.bro
View file

@ -1,104 +0,0 @@
##! Activates port-independent protocol detection and selectively disables
##! analyzers if protocol violations occur.
module DPD;
export {
## Add the DPD logging stream identifier.
redef enum Log::ID += { LOG };
## The record type defining the columns to log in the DPD logging stream.
type Info: record {
## Timestamp for when protocol analysis failed.
ts: time &log;
## Connection unique ID.
uid: string &log;
## Connection ID containing the 4-tuple which identifies endpoints.
id: conn_id &log;
## Transport protocol for the violation.
proto: transport_proto &log;
## The analyzer that generated the violation.
analyzer: string &log;
## The textual reason for the analysis failure.
failure_reason: string &log;
## Disabled analyzer IDs. This is only for internal tracking
## so as to not attempt to disable analyzers multiple times.
disabled_aids: set[count];
};
## Analyzers which you don't want to throw
option ignore_violations: set[Analyzer::Tag] = set();
## Ignore violations which go this many bytes into the connection.
## Set to 0 to never ignore protocol violations.
option ignore_violations_after = 10 * 1024;
}
redef record connection += {
dpd: Info &optional;
};
event bro_init() &priority=5
{
Log::create_stream(DPD::LOG, [$columns=Info, $path="dpd"]);
}
event protocol_confirmation(c: connection, atype: Analyzer::Tag, aid: count) &priority=10
{
local analyzer = Analyzer::name(atype);
if ( fmt("-%s",analyzer) in c$service )
delete c$service[fmt("-%s", analyzer)];
add c$service[analyzer];
}
event protocol_violation(c: connection, atype: Analyzer::Tag, aid: count,
reason: string) &priority=10
{
local analyzer = Analyzer::name(atype);
# If the service hasn't been confirmed yet, don't generate a log message
# for the protocol violation.
if ( analyzer !in c$service )
return;
delete c$service[analyzer];
add c$service[fmt("-%s", analyzer)];
local info: Info;
info$ts=network_time();
info$uid=c$uid;
info$id=c$id;
info$proto=get_conn_transport_proto(c$id);
info$analyzer=analyzer;
info$failure_reason=reason;
c$dpd = info;
}
event protocol_violation(c: connection, atype: Analyzer::Tag, aid: count, reason: string) &priority=5
{
if ( !c?$dpd || aid in c$dpd$disabled_aids )
return;
local size = c$orig$size + c$resp$size;
if ( ignore_violations_after > 0 && size > ignore_violations_after )
return;
if ( atype in ignore_violations )
return;
# Disable the analyzer that raised the last core-generated event.
disable_analyzer(c$id, aid, F);
add c$dpd$disabled_aids[aid];
}
event protocol_violation(c: connection, atype: Analyzer::Tag, aid: count,
reason: string) &priority=-5
{
if ( c?$dpd )
{
Log::write(DPD::LOG, c$dpd);
delete c$dpd;
}
}

104
scripts/base/frameworks/dpd/main.zeek Normal file
View file

@ -0,0 +1,104 @@
##! Activates port-independent protocol detection and selectively disables
##! analyzers if protocol violations occur.
module DPD;
export {
## Add the DPD logging stream identifier.
redef enum Log::ID += { LOG };
## The record type defining the columns to log in the DPD logging stream.
type Info: record {
## Timestamp for when protocol analysis failed.
ts: time &log;
## Connection unique ID.
uid: string &log;
## Connection ID containing the 4-tuple which identifies endpoints.
id: conn_id &log;
## Transport protocol for the violation.
proto: transport_proto &log;
## The analyzer that generated the violation.
analyzer: string &log;
## The textual reason for the analysis failure.
failure_reason: string &log;
## Disabled analyzer IDs. This is only for internal tracking
## so as to not attempt to disable analyzers multiple times.
disabled_aids: set[count];
};
## Analyzers which you don't want to throw
option ignore_violations: set[Analyzer::Tag] = set();
## Ignore violations which go this many bytes into the connection.
## Set to 0 to never ignore protocol violations.
option ignore_violations_after = 10 * 1024;
}
redef record connection += {
dpd: Info &optional;
};
event zeek_init() &priority=5
{
Log::create_stream(DPD::LOG, [$columns=Info, $path="dpd"]);
}
event protocol_confirmation(c: connection, atype: Analyzer::Tag, aid: count) &priority=10
{
local analyzer = Analyzer::name(atype);
if ( fmt("-%s",analyzer) in c$service )
delete c$service[fmt("-%s", analyzer)];
add c$service[analyzer];
}
event protocol_violation(c: connection, atype: Analyzer::Tag, aid: count,
reason: string) &priority=10
{
local analyzer = Analyzer::name(atype);
# If the service hasn't been confirmed yet, don't generate a log message
# for the protocol violation.
if ( analyzer !in c$service )
return;
delete c$service[analyzer];
add c$service[fmt("-%s", analyzer)];
local info: Info;
info$ts=network_time();
info$uid=c$uid;
info$id=c$id;
info$proto=get_conn_transport_proto(c$id);
info$analyzer=analyzer;
info$failure_reason=reason;
c$dpd = info;
}
event protocol_violation(c: connection, atype: Analyzer::Tag, aid: count, reason: string) &priority=5
{
if ( !c?$dpd || aid in c$dpd$disabled_aids )
return;
local size = c$orig$size + c$resp$size;
if ( ignore_violations_after > 0 && size > ignore_violations_after )
return;
if ( atype in ignore_violations )
return;
# Disable the analyzer that raised the last core-generated event.
disable_analyzer(c$id, aid, F);
add c$dpd$disabled_aids[aid];
}
event protocol_violation(c: connection, atype: Analyzer::Tag, aid: count,
reason: string) &priority=-5
{
if ( c?$dpd )
{
Log::write(DPD::LOG, c$dpd);
delete c$dpd;
}
}

2
scripts/base/frameworks/files/__load__.bro
View file

@ -1,2 +0,0 @@
@load ./main.bro
@load ./magic

2
scripts/base/frameworks/files/__load__.zeek Normal file
View file

@ -0,0 +1,2 @@
@load ./main
@load ./magic

0
scripts/base/frameworks/files/magic/__load__.bro → scripts/base/frameworks/files/magic/__load__.zeek
View file

545
scripts/base/frameworks/files/main.bro
View file

@ -1,545 +0,0 @@
##! An interface for driving the analysis of files, possibly independent of
##! any network protocol over which they're transported.
@load base/bif/file_analysis.bif
@load base/frameworks/analyzer
@load base/frameworks/logging
@load base/utils/site
module Files;
export {
redef enum Log::ID += {
## Logging stream for file analysis.
LOG
};
## A structure which parameterizes a type of file analysis.
type AnalyzerArgs: record {
## An event which will be generated for all new file contents,
## chunk-wise. Used when *tag* (in the
## :bro:see:`Files::add_analyzer` function) is
## :bro:see:`Files::ANALYZER_DATA_EVENT`.
chunk_event: event(f: fa_file, data: string, off: count) &optional;
## An event which will be generated for all new file contents,
## stream-wise. Used when *tag* is
## :bro:see:`Files::ANALYZER_DATA_EVENT`.
stream_event: event(f: fa_file, data: string) &optional;
} &redef;
## Contains all metadata related to the analysis of a given file.
## For the most part, fields here are derived from ones of the same name
## in :bro:see:`fa_file`.
type Info: record {
## The time when the file was first seen.
ts: time &log;
## An identifier associated with a single file.
fuid: string &log;
## If this file was transferred over a network
## connection this should show the host or hosts that
## the data sourced from.
tx_hosts: set[addr] &default=addr_set() &log;
## If this file was transferred over a network
## connection this should show the host or hosts that
## the data traveled to.
rx_hosts: set[addr] &default=addr_set() &log;
## Connection UIDs over which the file was transferred.
conn_uids: set[string] &default=string_set() &log;
## An identification of the source of the file data. E.g. it
## may be a network protocol over which it was transferred, or a
## local file path which was read, or some other input source.
source: string &log &optional;
## A value to represent the depth of this file in relation
## to its source. In SMTP, it is the depth of the MIME
## attachment on the message. In HTTP, it is the depth of the
## request within the TCP connection.
depth: count &default=0 &log;
## A set of analysis types done during the file analysis.
analyzers: set[string] &default=string_set() &log;
## A mime type provided by the strongest file magic signature
## match against the *bof_buffer* field of :bro:see:`fa_file`,
## or in the cases where no buffering of the beginning of file
## occurs, an initial guess of the mime type based on the first
## data seen.
mime_type: string &log &optional;
## A filename for the file if one is available from the source
## for the file. These will frequently come from
## "Content-Disposition" headers in network protocols.
filename: string &log &optional;
## The duration the file was analyzed for.
duration: interval &log &default=0secs;
## If the source of this file is a network connection, this field
## indicates if the data originated from the local network or not as
## determined by the configured :bro:see:`Site::local_nets`.
local_orig: bool &log &optional;
## If the source of this file is a network connection, this field
## indicates if the file is being sent by the originator of the
## connection or the responder.
is_orig: bool &log &optional;
## Number of bytes provided to the file analysis engine for the file.
seen_bytes: count &log &default=0;
## Total number of bytes that are supposed to comprise the full file.
total_bytes: count &log &optional;
## The number of bytes in the file stream that were completely missed
## during the process of analysis e.g. due to dropped packets.
missing_bytes: count &log &default=0;
## The number of bytes in the file stream that were not delivered to
## stream file analyzers. This could be overlapping bytes or
## bytes that couldn't be reassembled.
overflow_bytes: count &log &default=0;
## Whether the file analysis timed out at least once for the file.
timedout: bool &log &default=F;
## Identifier associated with a container file from which this one was
## extracted as part of the file analysis.
parent_fuid: string &log &optional;
} &redef;
## A table that can be used to disable file analysis completely for
## any files transferred over given network protocol analyzers.
const disable: table[Files::Tag] of bool = table() &redef;
## The salt concatenated to unique file handle strings generated by
## :bro:see:`get_file_handle` before hashing them in to a file id
## (the *id* field of :bro:see:`fa_file`).
## Provided to help mitigate the possibility of manipulating parts of
## network connections that factor in to the file handle in order to
## generate two handles that would hash to the same file id.
const salt = "I recommend changing this." &redef;
## Decide if you want to automatically attached analyzers to
## files based on the detected mime type of the file.
const analyze_by_mime_type_automatically = T &redef;
## The default setting for file reassembly.
option enable_reassembler = T;
## The default per-file reassembly buffer size.
const reassembly_buffer_size = 524288 &redef;
## Lookup to see if a particular file id exists and is still valid.
##
## fuid: the file id.
##
## Returns: T if the file uid is known.
global file_exists: function(fuid: string): bool;
## Lookup an :bro:see:`fa_file` record with the file id.
##
## fuid: the file id.
##
## Returns: the associated :bro:see:`fa_file` record.
global lookup_file: function(fuid: string): fa_file;
## Allows the file reassembler to be used if it's necessary because the
## file is transferred out of order.
##
## f: the file.
global enable_reassembly: function(f: fa_file);
## Disables the file reassembler on this file. If the file is not
## transferred out of order this will have no effect.
##
## f: the file.
global disable_reassembly: function(f: fa_file);
## Set the maximum size the reassembly buffer is allowed to grow
## for the given file.
##
## f: the file.
##
## max: Maximum allowed size of the reassembly buffer.
global set_reassembly_buffer_size: function(f: fa_file, max: count);
## Sets the *timeout_interval* field of :bro:see:`fa_file`, which is
## used to determine the length of inactivity that is allowed for a file
## before internal state related to it is cleaned up. When used within
## a :bro:see:`file_timeout` handler, the analysis will delay timing out
## again for the period specified by *t*.
##
## f: the file.
##
## t: the amount of time the file can remain inactive before discarding.
##
## Returns: true if the timeout interval was set, or false if analysis
## for the file isn't currently active.
global set_timeout_interval: function(f: fa_file, t: interval): bool;
## Adds an analyzer to the analysis of a given file.
##
## f: the file.
##
## tag: the analyzer type.
##
## args: any parameters the analyzer takes.
##
## Returns: true if the analyzer will be added, or false if analysis
## for the file isn't currently active or the *args*
## were invalid for the analyzer type.
global add_analyzer: function(f: fa_file,
tag: Files::Tag,
args: AnalyzerArgs &default=AnalyzerArgs()): bool;
## Removes an analyzer from the analysis of a given file.
##
## f: the file.
##
## tag: the analyzer type.
##
## args: the analyzer (type and args) to remove.
##
## Returns: true if the analyzer will be removed, or false if analysis
## for the file isn't currently active.
global remove_analyzer: function(f: fa_file,
tag: Files::Tag,
args: AnalyzerArgs &default=AnalyzerArgs()): bool;
## Stops/ignores any further analysis of a given file.
##
## f: the file.
##
## Returns: true if analysis for the given file will be ignored for the
## rest of its contents, or false if analysis for the file
## isn't currently active.
global stop: function(f: fa_file): bool;
## Translates a file analyzer enum value to a string with the
## analyzer's name.
##
## tag: The analyzer tag.
##
## Returns: The analyzer name corresponding to the tag.
global analyzer_name: function(tag: Files::Tag): string;
## Provides a text description regarding metadata of the file.
## For example, with HTTP it would return a URL.
##
## f: The file to be described.
##
## Returns: a text description regarding metadata of the file.
global describe: function(f: fa_file): string;
type ProtoRegistration: record {
## A callback to generate a file handle on demand when
## one is needed by the core.
get_file_handle: function(c: connection, is_orig: bool): string;
## A callback to "describe" a file. In the case of an HTTP
## transfer the most obvious description would be the URL.
## It's like an extremely compressed version of the normal log.
describe: function(f: fa_file): string
&default=function(f: fa_file): string { return ""; };
};
## Register callbacks for protocols that work with the Files framework.
## The callbacks must uniquely identify a file and each protocol can
## only have a single callback registered for it.
##
## tag: Tag for the protocol analyzer having a callback being registered.
##
## reg: A :bro:see:`Files::ProtoRegistration` record.
##
## Returns: true if the protocol being registered was not previously registered.
global register_protocol: function(tag: Analyzer::Tag, reg: ProtoRegistration): bool;
## Register a callback for file analyzers to use if they need to do some
## manipulation when they are being added to a file before the core code
## takes over. This is unlikely to be interesting for users and should
## only be called by file analyzer authors but is *not required*.
##
## tag: Tag for the file analyzer.
##
## callback: Function to execute when the given file analyzer is being added.
global register_analyzer_add_callback: function(tag: Files::Tag, callback: function(f: fa_file, args: AnalyzerArgs));
## Registers a set of MIME types for an analyzer. If a future connection on one of
## these types is seen, the analyzer will be automatically assigned to parsing it.
## The function *adds* to all MIME types already registered, it doesn't replace
## them.
##
## tag: The tag of the analyzer.
##
## mts: The set of MIME types, each in the form "foo/bar" (case-insensitive).
##
## Returns: True if the MIME types were successfully registered.
global register_for_mime_types: function(tag: Files::Tag, mts: set[string]) : bool;
## Registers a MIME type for an analyzer. If a future file with this type is seen,
## the analyzer will be automatically assigned to parsing it. The function *adds*
## to all MIME types already registered, it doesn't replace them.
##
## tag: The tag of the analyzer.
##
## mt: The MIME type in the form "foo/bar" (case-insensitive).
##
## Returns: True if the MIME type was successfully registered.
global register_for_mime_type: function(tag: Files::Tag, mt: string) : bool;
## Returns a set of all MIME types currently registered for a specific analyzer.
##
## tag: The tag of the analyzer.
##
## Returns: The set of MIME types.
global registered_mime_types: function(tag: Files::Tag) : set[string];
## Returns a table of all MIME-type-to-analyzer mappings currently registered.
##
## Returns: A table mapping each analyzer to the set of MIME types
## registered for it.
global all_registered_mime_types: function() : table[Files::Tag] of set[string];
## Event that can be handled to access the Info record as it is sent on
## to the logging framework.
global log_files: event(rec: Info);
}
redef record fa_file += {
info: Info &optional;
};
# Store the callbacks for protocol analyzers that have files.
global registered_protocols: table[Analyzer::Tag] of ProtoRegistration = table();
# Store the MIME type to analyzer mappings.
global mime_types: table[Files::Tag] of set[string];
global mime_type_to_analyzers: table[string] of set[Files::Tag];
global analyzer_add_callbacks: table[Files::Tag] of function(f: fa_file, args: AnalyzerArgs) = table();
event bro_init() &priority=5
{
Log::create_stream(Files::LOG, [$columns=Info, $ev=log_files, $path="files"]);
}
function set_info(f: fa_file)
{
if ( ! f?$info )
{
local tmp: Info = Info($ts=f$last_active,
$fuid=f$id);
f$info = tmp;
}
if ( f?$parent_id )
f$info$parent_fuid = f$parent_id;
if ( f?$source )
f$info$source = f$source;
f$info$duration = f$last_active - f$info$ts;
f$info$seen_bytes = f$seen_bytes;
if ( f?$total_bytes )
f$info$total_bytes = f$total_bytes;
f$info$missing_bytes = f$missing_bytes;
f$info$overflow_bytes = f$overflow_bytes;
if ( f?$is_orig )
f$info$is_orig = f$is_orig;
}
function file_exists(fuid: string): bool
{
return __file_exists(fuid);
}
function lookup_file(fuid: string): fa_file
{
return __lookup_file(fuid);
}
function set_timeout_interval(f: fa_file, t: interval): bool
{
return __set_timeout_interval(f$id, t);
}
function enable_reassembly(f: fa_file)
{
__enable_reassembly(f$id);
}
function disable_reassembly(f: fa_file)
{
__disable_reassembly(f$id);
}
function set_reassembly_buffer_size(f: fa_file, max: count)
{
__set_reassembly_buffer(f$id, max);
}
function add_analyzer(f: fa_file, tag: Files::Tag, args: AnalyzerArgs): bool
{
add f$info$analyzers[Files::analyzer_name(tag)];
if ( tag in analyzer_add_callbacks )
analyzer_add_callbacks[tag](f, args);
if ( ! __add_analyzer(f$id, tag, args) )
{
Reporter::warning(fmt("Analyzer %s not added successfully to file %s.", tag, f$id));
return F;
}
return T;
}
function register_analyzer_add_callback(tag: Files::Tag, callback: function(f: fa_file, args: AnalyzerArgs))
{
analyzer_add_callbacks[tag] = callback;
}
function remove_analyzer(f: fa_file, tag: Files::Tag, args: AnalyzerArgs): bool
{
return __remove_analyzer(f$id, tag, args);
}
function stop(f: fa_file): bool
{
return __stop(f$id);
}
function analyzer_name(tag: Files::Tag): string
{
return __analyzer_name(tag);
}
function register_protocol(tag: Analyzer::Tag, reg: ProtoRegistration): bool
{
local result = (tag !in registered_protocols);
registered_protocols[tag] = reg;
return result;
}
function register_for_mime_types(tag: Files::Tag, mime_types: set[string]) : bool
{
local rc = T;
for ( mt in mime_types )
{
if ( ! register_for_mime_type(tag, mt) )
rc = F;
}
return rc;
}
function register_for_mime_type(tag: Files::Tag, mt: string) : bool
{
if ( tag !in mime_types )
{
mime_types[tag] = set();
}
add mime_types[tag][mt];
if ( mt !in mime_type_to_analyzers )
{
mime_type_to_analyzers[mt] = set();
}
add mime_type_to_analyzers[mt][tag];
return T;
}
function registered_mime_types(tag: Files::Tag) : set[string]
{
return tag in mime_types ? mime_types[tag] : set();
}
function all_registered_mime_types(): table[Files::Tag] of set[string]
{
return mime_types;
}
function describe(f: fa_file): string
{
local tag = Analyzer::get_tag(f$source);
if ( tag !in registered_protocols )
return "";
local handler = registered_protocols[tag];
return handler$describe(f);
}
event get_file_handle(tag: Files::Tag, c: connection, is_orig: bool) &priority=5
{
if ( tag !in registered_protocols )
return;
local handler = registered_protocols[tag];
set_file_handle(handler$get_file_handle(c, is_orig));
}
event file_new(f: fa_file) &priority=10
{
set_info(f);
if ( enable_reassembler )
{
Files::enable_reassembly(f);
Files::set_reassembly_buffer_size(f, reassembly_buffer_size);
}
}
event file_over_new_connection(f: fa_file, c: connection, is_orig: bool) &priority=10
{
set_info(f);
add f$info$conn_uids[c$uid];
local cid = c$id;
add f$info$tx_hosts[f$is_orig ? cid$orig_h : cid$resp_h];
if( |Site::local_nets| > 0 )
f$info$local_orig=Site::is_local_addr(f$is_orig ? cid$orig_h : cid$resp_h);
add f$info$rx_hosts[f$is_orig ? cid$resp_h : cid$orig_h];
}
event file_sniff(f: fa_file, meta: fa_metadata) &priority=10
{
set_info(f);
if ( ! meta?$mime_type )
return;
f$info$mime_type = meta$mime_type;
if ( analyze_by_mime_type_automatically &&
meta$mime_type in mime_type_to_analyzers )
{
local analyzers = mime_type_to_analyzers[meta$mime_type];
for ( a in analyzers )
{
add f$info$analyzers[Files::analyzer_name(a)];
Files::add_analyzer(f, a);
}
}
}
event file_timeout(f: fa_file) &priority=10
{
set_info(f);
f$info$timedout = T;
}
event file_state_remove(f: fa_file) &priority=10
{
set_info(f);
}
event file_state_remove(f: fa_file) &priority=-10
{
Log::write(Files::LOG, f$info);
}

545
scripts/base/frameworks/files/main.zeek Normal file
View file

@ -0,0 +1,545 @@
##! An interface for driving the analysis of files, possibly independent of
##! any network protocol over which they're transported.
@load base/bif/file_analysis.bif
@load base/frameworks/analyzer
@load base/frameworks/logging
@load base/utils/site
module Files;
export {
redef enum Log::ID += {
## Logging stream for file analysis.
LOG
};
## A structure which parameterizes a type of file analysis.
type AnalyzerArgs: record {
## An event which will be generated for all new file contents,
## chunk-wise. Used when *tag* (in the
## :zeek:see:`Files::add_analyzer` function) is
## :zeek:see:`Files::ANALYZER_DATA_EVENT`.
chunk_event: event(f: fa_file, data: string, off: count) &optional;
## An event which will be generated for all new file contents,
## stream-wise. Used when *tag* is
## :zeek:see:`Files::ANALYZER_DATA_EVENT`.
stream_event: event(f: fa_file, data: string) &optional;
} &redef;
## Contains all metadata related to the analysis of a given file.
## For the most part, fields here are derived from ones of the same name
## in :zeek:see:`fa_file`.
type Info: record {
## The time when the file was first seen.
ts: time &log;
## An identifier associated with a single file.
fuid: string &log;
## If this file was transferred over a network
## connection this should show the host or hosts that
## the data sourced from.
tx_hosts: set[addr] &default=addr_set() &log;
## If this file was transferred over a network
## connection this should show the host or hosts that
## the data traveled to.
rx_hosts: set[addr] &default=addr_set() &log;
## Connection UIDs over which the file was transferred.
conn_uids: set[string] &default=string_set() &log;
## An identification of the source of the file data. E.g. it
## may be a network protocol over which it was transferred, or a
## local file path which was read, or some other input source.
source: string &log &optional;
## A value to represent the depth of this file in relation
## to its source. In SMTP, it is the depth of the MIME
## attachment on the message. In HTTP, it is the depth of the
## request within the TCP connection.
depth: count &default=0 &log;
## A set of analysis types done during the file analysis.
analyzers: set[string] &default=string_set() &log;
## A mime type provided by the strongest file magic signature
## match against the *bof_buffer* field of :zeek:see:`fa_file`,
## or in the cases where no buffering of the beginning of file
## occurs, an initial guess of the mime type based on the first
## data seen.
mime_type: string &log &optional;
## A filename for the file if one is available from the source
## for the file. These will frequently come from
## "Content-Disposition" headers in network protocols.
filename: string &log &optional;
## The duration the file was analyzed for.
duration: interval &log &default=0secs;
## If the source of this file is a network connection, this field
## indicates if the data originated from the local network or not as
## determined by the configured :zeek:see:`Site::local_nets`.
local_orig: bool &log &optional;
## If the source of this file is a network connection, this field
## indicates if the file is being sent by the originator of the
## connection or the responder.
is_orig: bool &log &optional;
## Number of bytes provided to the file analysis engine for the file.
seen_bytes: count &log &default=0;
## Total number of bytes that are supposed to comprise the full file.
total_bytes: count &log &optional;
## The number of bytes in the file stream that were completely missed
## during the process of analysis e.g. due to dropped packets.
missing_bytes: count &log &default=0;
## The number of bytes in the file stream that were not delivered to
## stream file analyzers. This could be overlapping bytes or
## bytes that couldn't be reassembled.
overflow_bytes: count &log &default=0;
## Whether the file analysis timed out at least once for the file.
timedout: bool &log &default=F;
## Identifier associated with a container file from which this one was
## extracted as part of the file analysis.
parent_fuid: string &log &optional;
} &redef;
## A table that can be used to disable file analysis completely for
## any files transferred over given network protocol analyzers.
const disable: table[Files::Tag] of bool = table() &redef;
## The salt concatenated to unique file handle strings generated by
## :zeek:see:`get_file_handle` before hashing them in to a file id
## (the *id* field of :zeek:see:`fa_file`).
## Provided to help mitigate the possibility of manipulating parts of
## network connections that factor in to the file handle in order to
## generate two handles that would hash to the same file id.
const salt = "I recommend changing this." &redef;
## Decide if you want to automatically attached analyzers to
## files based on the detected mime type of the file.
const analyze_by_mime_type_automatically = T &redef;
## The default setting for file reassembly.
option enable_reassembler = T;
## The default per-file reassembly buffer size.
const reassembly_buffer_size = 524288 &redef;
## Lookup to see if a particular file id exists and is still valid.
##
## fuid: the file id.
##
## Returns: T if the file uid is known.
global file_exists: function(fuid: string): bool;
## Lookup an :zeek:see:`fa_file` record with the file id.
##
## fuid: the file id.
##
## Returns: the associated :zeek:see:`fa_file` record.
global lookup_file: function(fuid: string): fa_file;
## Allows the file reassembler to be used if it's necessary because the
## file is transferred out of order.
##
## f: the file.
global enable_reassembly: function(f: fa_file);
## Disables the file reassembler on this file. If the file is not
## transferred out of order this will have no effect.
##
## f: the file.
global disable_reassembly: function(f: fa_file);
## Set the maximum size the reassembly buffer is allowed to grow
## for the given file.
##
## f: the file.
##
## max: Maximum allowed size of the reassembly buffer.
global set_reassembly_buffer_size: function(f: fa_file, max: count);
## Sets the *timeout_interval* field of :zeek:see:`fa_file`, which is
## used to determine the length of inactivity that is allowed for a file
## before internal state related to it is cleaned up. When used within
## a :zeek:see:`file_timeout` handler, the analysis will delay timing out
## again for the period specified by *t*.
##
## f: the file.
##
## t: the amount of time the file can remain inactive before discarding.
##
## Returns: true if the timeout interval was set, or false if analysis
## for the file isn't currently active.
global set_timeout_interval: function(f: fa_file, t: interval): bool;
## Adds an analyzer to the analysis of a given file.
##
## f: the file.
##
## tag: the analyzer type.
##
## args: any parameters the analyzer takes.
##
## Returns: true if the analyzer will be added, or false if analysis
## for the file isn't currently active or the *args*
## were invalid for the analyzer type.
global add_analyzer: function(f: fa_file,
tag: Files::Tag,
args: AnalyzerArgs &default=AnalyzerArgs()): bool;
## Removes an analyzer from the analysis of a given file.
##
## f: the file.
##
## tag: the analyzer type.
##
## args: the analyzer (type and args) to remove.
##
## Returns: true if the analyzer will be removed, or false if analysis
## for the file isn't currently active.
global remove_analyzer: function(f: fa_file,
tag: Files::Tag,
args: AnalyzerArgs &default=AnalyzerArgs()): bool;
## Stops/ignores any further analysis of a given file.
##
## f: the file.
##
## Returns: true if analysis for the given file will be ignored for the
## rest of its contents, or false if analysis for the file
## isn't currently active.
global stop: function(f: fa_file): bool;
## Translates a file analyzer enum value to a string with the
## analyzer's name.
##
## tag: The analyzer tag.
##
## Returns: The analyzer name corresponding to the tag.
global analyzer_name: function(tag: Files::Tag): string;
## Provides a text description regarding metadata of the file.
## For example, with HTTP it would return a URL.
##
## f: The file to be described.
##
## Returns: a text description regarding metadata of the file.
global describe: function(f: fa_file): string;
type ProtoRegistration: record {
## A callback to generate a file handle on demand when
## one is needed by the core.
get_file_handle: function(c: connection, is_orig: bool): string;
## A callback to "describe" a file. In the case of an HTTP
## transfer the most obvious description would be the URL.
## It's like an extremely compressed version of the normal log.
describe: function(f: fa_file): string
&default=function(f: fa_file): string { return ""; };
};
## Register callbacks for protocols that work with the Files framework.
## The callbacks must uniquely identify a file and each protocol can
## only have a single callback registered for it.
##
## tag: Tag for the protocol analyzer having a callback being registered.
##
## reg: A :zeek:see:`Files::ProtoRegistration` record.
##
## Returns: true if the protocol being registered was not previously registered.
global register_protocol: function(tag: Analyzer::Tag, reg: ProtoRegistration): bool;
## Register a callback for file analyzers to use if they need to do some
## manipulation when they are being added to a file before the core code
## takes over. This is unlikely to be interesting for users and should
## only be called by file analyzer authors but is *not required*.
##
## tag: Tag for the file analyzer.
##
## callback: Function to execute when the given file analyzer is being added.
global register_analyzer_add_callback: function(tag: Files::Tag, callback: function(f: fa_file, args: AnalyzerArgs));
## Registers a set of MIME types for an analyzer. If a future connection on one of
## these types is seen, the analyzer will be automatically assigned to parsing it.
## The function *adds* to all MIME types already registered, it doesn't replace
## them.
##
## tag: The tag of the analyzer.
##
## mts: The set of MIME types, each in the form "foo/bar" (case-insensitive).
##
## Returns: True if the MIME types were successfully registered.
global register_for_mime_types: function(tag: Files::Tag, mts: set[string]) : bool;
## Registers a MIME type for an analyzer. If a future file with this type is seen,
## the analyzer will be automatically assigned to parsing it. The function *adds*
## to all MIME types already registered, it doesn't replace them.
##
## tag: The tag of the analyzer.
##
## mt: The MIME type in the form "foo/bar" (case-insensitive).
##
## Returns: True if the MIME type was successfully registered.
global register_for_mime_type: function(tag: Files::Tag, mt: string) : bool;
## Returns a set of all MIME types currently registered for a specific analyzer.
##
## tag: The tag of the analyzer.
##
## Returns: The set of MIME types.
global registered_mime_types: function(tag: Files::Tag) : set[string];
## Returns a table of all MIME-type-to-analyzer mappings currently registered.
##
## Returns: A table mapping each analyzer to the set of MIME types
## registered for it.
global all_registered_mime_types: function() : table[Files::Tag] of set[string];
## Event that can be handled to access the Info record as it is sent on
## to the logging framework.
global log_files: event(rec: Info);
}
redef record fa_file += {
info: Info &optional;
};
# Store the callbacks for protocol analyzers that have files.
global registered_protocols: table[Analyzer::Tag] of ProtoRegistration = table();
# Store the MIME type to analyzer mappings.
global mime_types: table[Files::Tag] of set[string];
global mime_type_to_analyzers: table[string] of set[Files::Tag];
global analyzer_add_callbacks: table[Files::Tag] of function(f: fa_file, args: AnalyzerArgs) = table();
event zeek_init() &priority=5
{
Log::create_stream(Files::LOG, [$columns=Info, $ev=log_files, $path="files"]);
}
function set_info(f: fa_file)
{
if ( ! f?$info )
{
local tmp: Info = Info($ts=f$last_active,
$fuid=f$id);
f$info = tmp;
}
if ( f?$parent_id )
f$info$parent_fuid = f$parent_id;
if ( f?$source )
f$info$source = f$source;
f$info$duration = f$last_active - f$info$ts;
f$info$seen_bytes = f$seen_bytes;
if ( f?$total_bytes )
f$info$total_bytes = f$total_bytes;
f$info$missing_bytes = f$missing_bytes;
f$info$overflow_bytes = f$overflow_bytes;
if ( f?$is_orig )
f$info$is_orig = f$is_orig;
}
function file_exists(fuid: string): bool
{
return __file_exists(fuid);
}
function lookup_file(fuid: string): fa_file
{
return __lookup_file(fuid);
}
function set_timeout_interval(f: fa_file, t: interval): bool
{
return __set_timeout_interval(f$id, t);
}
function enable_reassembly(f: fa_file)
{
__enable_reassembly(f$id);
}
function disable_reassembly(f: fa_file)
{
__disable_reassembly(f$id);
}
function set_reassembly_buffer_size(f: fa_file, max: count)
{
__set_reassembly_buffer(f$id, max);
}
function add_analyzer(f: fa_file, tag: Files::Tag, args: AnalyzerArgs): bool
{
add f$info$analyzers[Files::analyzer_name(tag)];
if ( tag in analyzer_add_callbacks )
analyzer_add_callbacks[tag](f, args);
if ( ! __add_analyzer(f$id, tag, args) )
{
Reporter::warning(fmt("Analyzer %s not added successfully to file %s.", tag, f$id));
return F;
}
return T;
}
function register_analyzer_add_callback(tag: Files::Tag, callback: function(f: fa_file, args: AnalyzerArgs))
{
analyzer_add_callbacks[tag] = callback;
}
function remove_analyzer(f: fa_file, tag: Files::Tag, args: AnalyzerArgs): bool
{
return __remove_analyzer(f$id, tag, args);
}
function stop(f: fa_file): bool
{
return __stop(f$id);
}
function analyzer_name(tag: Files::Tag): string
{
return __analyzer_name(tag);
}
function register_protocol(tag: Analyzer::Tag, reg: ProtoRegistration): bool
{
local result = (tag !in registered_protocols);
registered_protocols[tag] = reg;
return result;
}
function register_for_mime_types(tag: Files::Tag, mime_types: set[string]) : bool
{
local rc = T;
for ( mt in mime_types )
{
if ( ! register_for_mime_type(tag, mt) )
rc = F;
}
return rc;
}
function register_for_mime_type(tag: Files::Tag, mt: string) : bool
{
if ( tag !in mime_types )
{
mime_types[tag] = set();
}
add mime_types[tag][mt];
if ( mt !in mime_type_to_analyzers )
{
mime_type_to_analyzers[mt] = set();
}
add mime_type_to_analyzers[mt][tag];
return T;
}
function registered_mime_types(tag: Files::Tag) : set[string]
{
return tag in mime_types ? mime_types[tag] : set();
}
function all_registered_mime_types(): table[Files::Tag] of set[string]
{
return mime_types;
}
function describe(f: fa_file): string
{
local tag = Analyzer::get_tag(f$source);
if ( tag !in registered_protocols )
return "";
local handler = registered_protocols[tag];
return handler$describe(f);
}
event get_file_handle(tag: Files::Tag, c: connection, is_orig: bool) &priority=5
{
if ( tag !in registered_protocols )
return;
local handler = registered_protocols[tag];
set_file_handle(handler$get_file_handle(c, is_orig));
}
event file_new(f: fa_file) &priority=10
{
set_info(f);
if ( enable_reassembler )
{
Files::enable_reassembly(f);
Files::set_reassembly_buffer_size(f, reassembly_buffer_size);
}
}
event file_over_new_connection(f: fa_file, c: connection, is_orig: bool) &priority=10
{
set_info(f);
add f$info$conn_uids[c$uid];
local cid = c$id;
add f$info$tx_hosts[f$is_orig ? cid$orig_h : cid$resp_h];
if( |Site::local_nets| > 0 )
f$info$local_orig=Site::is_local_addr(f$is_orig ? cid$orig_h : cid$resp_h);
add f$info$rx_hosts[f$is_orig ? cid$resp_h : cid$orig_h];
}
event file_sniff(f: fa_file, meta: fa_metadata) &priority=10
{
set_info(f);
if ( ! meta?$mime_type )
return;
f$info$mime_type = meta$mime_type;
if ( analyze_by_mime_type_automatically &&
meta$mime_type in mime_type_to_analyzers )
{
local analyzers = mime_type_to_analyzers[meta$mime_type];
for ( a in analyzers )
{
add f$info$analyzers[Files::analyzer_name(a)];
Files::add_analyzer(f, a);
}
}
}
event file_timeout(f: fa_file) &priority=10
{
set_info(f);
f$info$timedout = T;
}
event file_state_remove(f: fa_file) &priority=10
{
set_info(f);
}
event file_state_remove(f: fa_file) &priority=-10
{
Log::write(Files::LOG, f$info);
}

2
scripts/base/frameworks/input/README
View file

@ -1,2 +1,2 @@
The input framework provides a way to read previously stored data either as
an event stream or into a Bro table.
an event stream or into a Zeek table.

0
scripts/base/frameworks/input/__load__.bro → scripts/base/frameworks/input/__load__.zeek
View file

281
scripts/base/frameworks/input/main.bro
View file

@ -1,281 +0,0 @@
##! The input framework provides a way to read previously stored data either
##! as an event stream or into a Bro table.
module Input;
export {
## Type that describes what kind of change occurred.
type Event: enum {
## New data has been imported.
EVENT_NEW = 0,
## Existing data has been changed.
EVENT_CHANGED = 1,
## Previously existing data has been removed.
EVENT_REMOVED = 2,
};
## Type that defines the input stream read mode.
type Mode: enum {
## Do not automatically reread the file after it has been read.
MANUAL = 0,
## Reread the entire file each time a change is found.
REREAD = 1,
## Read data from end of file each time new data is appended.
STREAM = 2
};
## The default input reader used. Defaults to `READER_ASCII`.
option default_reader = READER_ASCII;
## The default reader mode used. Defaults to `MANUAL`.
option default_mode = MANUAL;
## Separator between fields.
## Please note that the separator has to be exactly one character long.
## Individual readers can use a different value.
const separator = "\t" &redef;
## Separator between set elements.
## Please note that the separator has to be exactly one character long.
## Individual readers can use a different value.
const set_separator = "," &redef;
## String to use for empty fields.
## Individual readers can use a different value.
const empty_field = "(empty)" &redef;
## String to use for an unset &optional field.
## Individual readers can use a different value.
const unset_field = "-" &redef;
## Flag that controls if the input framework accepts records
## that contain types that are not supported (at the moment
## file and function). If true, the input framework will
## warn in these cases, but continue. If false, it will
## abort. Defaults to false (abort).
const accept_unsupported_types = F &redef;
## A table input stream type used to send data to a Bro table.
type TableDescription: record {
# Common definitions for tables and events
## String that allows the reader to find the source of the data.
## For `READER_ASCII`, this is the filename.
source: string;
## Reader to use for this stream.
reader: Reader &default=default_reader;
## Read mode to use for this stream.
mode: Mode &default=default_mode;
## Name of the input stream. This is used by some functions to
## manipulate the stream.
name: string;
# Special definitions for tables
## Table which will receive the data read by the input framework.
destination: any;
## Record that defines the values used as the index of the table.
idx: any;
## Record that defines the values used as the elements of the table.
## If this is undefined, then *destination* must be a set.
val: any &optional;
## Defines if the value of the table is a record (default), or a single
## value. When this is set to false, then *val* can only contain one
## element.
want_record: bool &default=T;
## The event that is raised each time a value is added to, changed in,
## or removed from the table. The event will receive an
## Input::TableDescription as the first argument, an Input::Event
## enum as the second argument, the *idx* record as the third argument
## and the value (record) as the fourth argument.
ev: any &optional;
## Predicate function that can decide if an insertion, update or removal
## should really be executed. Parameters have same meaning as for the
## event.
## If true is returned, the update is performed. If false is returned,
## it is skipped.
pred: function(typ: Input::Event, left: any, right: any): bool &optional;
## Error event that is raised when an information, warning or error
## is raised by the input stream. If the level is error, the stream will automatically
## be closed.
## The event receives the Input::TableDescription as the first argument, the
## message as the second argument and the Reporter::Level as the third argument.
##
## The event is raised like if it had been declared as follows:
## error_ev: function(desc: TableDescription, message: string, level: Reporter::Level) &optional;
## The actual declaration uses the ``any`` type because of deficiencies of the Bro type system.
error_ev: any &optional;
## A key/value table that will be passed to the reader.
## Interpretation of the values is left to the reader, but
## usually they will be used for configuration purposes.
config: table[string] of string &default=table();
};
## An event input stream type used to send input data to a Bro event.
type EventDescription: record {
# Common definitions for tables and events
## String that allows the reader to find the source.
## For `READER_ASCII`, this is the filename.
source: string;
## Reader to use for this stream.
reader: Reader &default=default_reader;
## Read mode to use for this stream.
mode: Mode &default=default_mode;
## Descriptive name. Used to remove a stream at a later time.
name: string;
# Special definitions for events
## Record type describing the fields to be retrieved from the input
## source.
fields: any;
## If this is false, the event receives each value in *fields* as a
## separate argument.
## If this is set to true (default), the event receives all fields in
## a single record value.
want_record: bool &default=T;
## The event that is raised each time a new line is received from the
## reader. The event will receive an Input::EventDescription record
## as the first argument, an Input::Event enum as the second
## argument, and the fields (as specified in *fields*) as the following
## arguments (this will either be a single record value containing
## all fields, or each field value as a separate argument).
ev: any;
## Error event that is raised when an information, warning or error
## is raised by the input stream. If the level is error, the stream will automatically
## be closed.
## The event receives the Input::EventDescription as the first argument, the
## message as the second argument and the Reporter::Level as the third argument.
##
## The event is raised like it had been declared as follows:
## error_ev: function(desc: EventDescription, message: string, level: Reporter::Level) &optional;
## The actual declaration uses the ``any`` type because of deficiencies of the Bro type system.
error_ev: any &optional;
## A key/value table that will be passed to the reader.
## Interpretation of the values is left to the reader, but
## usually they will be used for configuration purposes.
config: table[string] of string &default=table();
};
## A file analysis input stream type used to forward input data to the
## file analysis framework.
type AnalysisDescription: record {
## String that allows the reader to find the source.
## For `READER_ASCII`, this is the filename.
source: string;
## Reader to use for this stream. Compatible readers must be
## able to accept a filter of a single string type (i.e.
## they read a byte stream).
reader: Reader &default=Input::READER_BINARY;
## Read mode to use for this stream.
mode: Mode &default=default_mode;
## Descriptive name that uniquely identifies the input source.
## Can be used to remove a stream at a later time.
## This will also be used for the unique *source* field of
## :bro:see:`fa_file`. Most of the time, the best choice for this
## field will be the same value as the *source* field.
name: string;
## A key/value table that will be passed to the reader.
## Interpretation of the values is left to the reader, but
## usually they will be used for configuration purposes.
config: table[string] of string &default=table();
};
## Create a new table input stream from a given source.
##
## description: `TableDescription` record describing the source.
##
## Returns: true on success.
global add_table: function(description: Input::TableDescription) : bool;
## Create a new event input stream from a given source.
##
## description: `EventDescription` record describing the source.
##
## Returns: true on success.
global add_event: function(description: Input::EventDescription) : bool;
## Create a new file analysis input stream from a given source. Data read
## from the source is automatically forwarded to the file analysis
## framework.
##
## description: A record describing the source.
##
## Returns: true on success.
global add_analysis: function(description: Input::AnalysisDescription) : bool;
## Remove an input stream.
##
## id: string value identifying the stream to be removed.
##
## Returns: true on success and false if the named stream was not found.
global remove: function(id: string) : bool;
## Forces the current input to be checked for changes.
##
## id: string value identifying the stream.
##
## Returns: true on success and false if the named stream was not found.
global force_update: function(id: string) : bool;
## Event that is called when the end of a data source has been reached,
## including after an update.
##
## name: Name of the input stream.
##
## source: String that identifies the data source (such as the filename).
global end_of_data: event(name: string, source: string);
}
@load base/bif/input.bif
module Input;
function add_table(description: Input::TableDescription) : bool
{
return __create_table_stream(description);
}
function add_event(description: Input::EventDescription) : bool
{
return __create_event_stream(description);
}
function add_analysis(description: Input::AnalysisDescription) : bool
{
return __create_analysis_stream(description);
}
function remove(id: string) : bool
{
return __remove_stream(id);
}
function force_update(id: string) : bool
{
return __force_update(id);
}

281
scripts/base/frameworks/input/main.zeek Normal file
View file

@ -0,0 +1,281 @@
##! The input framework provides a way to read previously stored data either
##! as an event stream or into a Zeek table.
module Input;
export {
## Type that describes what kind of change occurred.
type Event: enum {
## New data has been imported.
EVENT_NEW = 0,
## Existing data has been changed.
EVENT_CHANGED = 1,
## Previously existing data has been removed.
EVENT_REMOVED = 2,
};
## Type that defines the input stream read mode.
type Mode: enum {
## Do not automatically reread the file after it has been read.
MANUAL = 0,
## Reread the entire file each time a change is found.
REREAD = 1,
## Read data from end of file each time new data is appended.
STREAM = 2
};
## The default input reader used. Defaults to `READER_ASCII`.
option default_reader = READER_ASCII;
## The default reader mode used. Defaults to `MANUAL`.
option default_mode = MANUAL;
## Separator between fields.
## Please note that the separator has to be exactly one character long.
## Individual readers can use a different value.
const separator = "\t" &redef;
## Separator between set elements.
## Please note that the separator has to be exactly one character long.
## Individual readers can use a different value.
const set_separator = "," &redef;
## String to use for empty fields.
## Individual readers can use a different value.
const empty_field = "(empty)" &redef;
## String to use for an unset &optional field.
## Individual readers can use a different value.
const unset_field = "-" &redef;
## Flag that controls if the input framework accepts records
## that contain types that are not supported (at the moment
## file and function). If true, the input framework will
## warn in these cases, but continue. If false, it will
## abort. Defaults to false (abort).
const accept_unsupported_types = F &redef;
## A table input stream type used to send data to a Zeek table.
type TableDescription: record {
# Common definitions for tables and events
## String that allows the reader to find the source of the data.
## For `READER_ASCII`, this is the filename.
source: string;
## Reader to use for this stream.
reader: Reader &default=default_reader;
## Read mode to use for this stream.
mode: Mode &default=default_mode;
## Name of the input stream. This is used by some functions to
## manipulate the stream.
name: string;
# Special definitions for tables
## Table which will receive the data read by the input framework.
destination: any;
## Record that defines the values used as the index of the table.
idx: any;
## Record that defines the values used as the elements of the table.
## If this is undefined, then *destination* must be a set.
val: any &optional;
## Defines if the value of the table is a record (default), or a single
## value. When this is set to false, then *val* can only contain one
## element.
want_record: bool &default=T;
## The event that is raised each time a value is added to, changed in,
## or removed from the table. The event will receive an
## Input::TableDescription as the first argument, an Input::Event
## enum as the second argument, the *idx* record as the third argument
## and the value (record) as the fourth argument.
ev: any &optional;
## Predicate function that can decide if an insertion, update or removal
## should really be executed. Parameters have same meaning as for the
## event.
## If true is returned, the update is performed. If false is returned,
## it is skipped.
pred: function(typ: Input::Event, left: any, right: any): bool &optional;
## Error event that is raised when an information, warning or error
## is raised by the input stream. If the level is error, the stream will automatically
## be closed.
## The event receives the Input::TableDescription as the first argument, the
## message as the second argument and the Reporter::Level as the third argument.
##
## The event is raised like if it had been declared as follows:
## error_ev: function(desc: TableDescription, message: string, level: Reporter::Level) &optional;
## The actual declaration uses the ``any`` type because of deficiencies of the Zeek type system.
error_ev: any &optional;
## A key/value table that will be passed to the reader.
## Interpretation of the values is left to the reader, but
## usually they will be used for configuration purposes.
config: table[string] of string &default=table();
};
## An event input stream type used to send input data to a Zeek event.
type EventDescription: record {
# Common definitions for tables and events
## String that allows the reader to find the source.
## For `READER_ASCII`, this is the filename.
source: string;
## Reader to use for this stream.
reader: Reader &default=default_reader;
## Read mode to use for this stream.
mode: Mode &default=default_mode;
## Descriptive name. Used to remove a stream at a later time.
name: string;
# Special definitions for events
## Record type describing the fields to be retrieved from the input
## source.
fields: any;
## If this is false, the event receives each value in *fields* as a
## separate argument.
## If this is set to true (default), the event receives all fields in
## a single record value.
want_record: bool &default=T;
## The event that is raised each time a new line is received from the
## reader. The event will receive an Input::EventDescription record
## as the first argument, an Input::Event enum as the second
## argument, and the fields (as specified in *fields*) as the following
## arguments (this will either be a single record value containing
## all fields, or each field value as a separate argument).
ev: any;
## Error event that is raised when an information, warning or error
## is raised by the input stream. If the level is error, the stream will automatically
## be closed.
## The event receives the Input::EventDescription as the first argument, the
## message as the second argument and the Reporter::Level as the third argument.
##
## The event is raised like it had been declared as follows:
## error_ev: function(desc: EventDescription, message: string, level: Reporter::Level) &optional;
## The actual declaration uses the ``any`` type because of deficiencies of the Zeek type system.
error_ev: any &optional;
## A key/value table that will be passed to the reader.
## Interpretation of the values is left to the reader, but
## usually they will be used for configuration purposes.
config: table[string] of string &default=table();
};
## A file analysis input stream type used to forward input data to the
## file analysis framework.
type AnalysisDescription: record {
## String that allows the reader to find the source.
## For `READER_ASCII`, this is the filename.
source: string;
## Reader to use for this stream. Compatible readers must be
## able to accept a filter of a single string type (i.e.
## they read a byte stream).
reader: Reader &default=Input::READER_BINARY;
## Read mode to use for this stream.
mode: Mode &default=default_mode;
## Descriptive name that uniquely identifies the input source.
## Can be used to remove a stream at a later time.
## This will also be used for the unique *source* field of
## :zeek:see:`fa_file`. Most of the time, the best choice for this
## field will be the same value as the *source* field.
name: string;
## A key/value table that will be passed to the reader.
## Interpretation of the values is left to the reader, but
## usually they will be used for configuration purposes.
config: table[string] of string &default=table();
};
## Create a new table input stream from a given source.
##
## description: `TableDescription` record describing the source.
##
## Returns: true on success.
global add_table: function(description: Input::TableDescription) : bool;
## Create a new event input stream from a given source.
##
## description: `EventDescription` record describing the source.
##
## Returns: true on success.
global add_event: function(description: Input::EventDescription) : bool;
## Create a new file analysis input stream from a given source. Data read
## from the source is automatically forwarded to the file analysis
## framework.
##
## description: A record describing the source.
##
## Returns: true on success.
global add_analysis: function(description: Input::AnalysisDescription) : bool;
## Remove an input stream.
##
## id: string value identifying the stream to be removed.
##
## Returns: true on success and false if the named stream was not found.
global remove: function(id: string) : bool;
## Forces the current input to be checked for changes.
##
## id: string value identifying the stream.
##
## Returns: true on success and false if the named stream was not found.
global force_update: function(id: string) : bool;
## Event that is called when the end of a data source has been reached,
## including after an update.
##
## name: Name of the input stream.
##
## source: String that identifies the data source (such as the filename).
global end_of_data: event(name: string, source: string);
}
@load base/bif/input.bif
module Input;
function add_table(description: Input::TableDescription) : bool
{
return __create_table_stream(description);
}
function add_event(description: Input::EventDescription) : bool
{
return __create_event_stream(description);
}
function add_analysis(description: Input::AnalysisDescription) : bool
{
return __create_analysis_stream(description);
}
function remove(id: string) : bool
{
return __remove_stream(id);
}
function force_update(id: string) : bool
{
return __force_update(id);
}

56
scripts/base/frameworks/input/readers/ascii.bro
View file

@ -1,56 +0,0 @@
##! Interface for the ascii input reader.
##!
##! The defaults are set to match Bro's ASCII output.
module InputAscii;
export {
## Separator between fields.
## Please note that the separator has to be exactly one character long.
const separator = Input::separator &redef;
## Separator between set and vector elements.
## Please note that the separator has to be exactly one character long.
const set_separator = Input::set_separator &redef;
## String to use for empty fields.
const empty_field = Input::empty_field &redef;
## String to use for an unset &optional field.
const unset_field = Input::unset_field &redef;
## Fail on invalid lines. If set to false, the ascii
## input reader will jump over invalid lines, reporting
## warnings in reporter.log. If set to true, errors in
## input lines will be handled as fatal errors for the
## reader thread; reading will abort immediately and
## an error will be logged to reporter.log.
## Individual readers can use a different value using
## the $config table.
## fail_on_invalid_lines = T was the default behavior
## until Bro 2.6.
const fail_on_invalid_lines = F &redef;
## Fail on file read problems. If set to true, the ascii
## input reader will fail when encountering any problems
## while reading a file different from invalid lines.
## Examples of such problems are permission problems, or
## missing files.
## When set to false, these problems will be ignored. This
## has an especially big effect for the REREAD mode, which will
## seamlessly recover from read errors when a file is
## only temporarily inaccessible. For MANUAL or STREAM files,
## errors will most likely still be fatal since no automatic
## re-reading of the file is attempted.
## Individual readers can use a different value using
## the $config table.
## fail_on_file_problem = T was the default behavior
## until Bro 2.6.
const fail_on_file_problem = F &redef;
## On input streams with a pathless or relative-path source filename,
## prefix the following path. This prefix can, but need not be, absolute.
## The default is to leave any filenames unchanged. This prefix has no
## effect if the source already is an absolute path.
const path_prefix = "" &redef;
}

56
scripts/base/frameworks/input/readers/ascii.zeek Normal file
View file

@ -0,0 +1,56 @@
##! Interface for the ascii input reader.
##!
##! The defaults are set to match Zeek's ASCII output.
module InputAscii;
export {
## Separator between fields.
## Please note that the separator has to be exactly one character long.
const separator = Input::separator &redef;
## Separator between set and vector elements.
## Please note that the separator has to be exactly one character long.
const set_separator = Input::set_separator &redef;
## String to use for empty fields.
const empty_field = Input::empty_field &redef;
## String to use for an unset &optional field.
const unset_field = Input::unset_field &redef;
## Fail on invalid lines. If set to false, the ascii
## input reader will jump over invalid lines, reporting
## warnings in reporter.log. If set to true, errors in
## input lines will be handled as fatal errors for the
## reader thread; reading will abort immediately and
## an error will be logged to reporter.log.
## Individual readers can use a different value using
## the $config table.
## fail_on_invalid_lines = T was the default behavior
## until Bro 2.6.
const fail_on_invalid_lines = F &redef;
## Fail on file read problems. If set to true, the ascii
## input reader will fail when encountering any problems
## while reading a file different from invalid lines.
## Examples of such problems are permission problems, or
## missing files.
## When set to false, these problems will be ignored. This
## has an especially big effect for the REREAD mode, which will
## seamlessly recover from read errors when a file is
## only temporarily inaccessible. For MANUAL or STREAM files,
## errors will most likely still be fatal since no automatic
## re-reading of the file is attempted.
## Individual readers can use a different value using
## the $config table.
## fail_on_file_problem = T was the default behavior
## until Bro 2.6.
const fail_on_file_problem = F &redef;
## On input streams with a pathless or relative-path source filename,
## prefix the following path. This prefix can, but need not be, absolute.
## The default is to leave any filenames unchanged. This prefix has no
## effect if the source already is an absolute path.
const path_prefix = "" &redef;
}

0
scripts/base/frameworks/input/readers/benchmark.bro → scripts/base/frameworks/input/readers/benchmark.zeek
View file

Some files were not shown because too many files have changed in this diff Show more