Commit graph

121 commits

Author SHA1 Message Date
Jon Siwek
7f0fb49612 Add an internal getenv wrapper function: zeekenv
It maps newer environment variable names starting with ZEEK to the
legacy names starting with BRO.
2019-05-23 20:42:42 -07:00
Daniel Thayer
1a74516db1 Rename all BRO-prefixed environment variables
For backward compatibility when reading values, we first check
the ZEEK-prefixed value, and if not set, then check the corresponding
BRO-prefixed value.
2019-05-22 00:12:31 -05:00
Daniel Thayer
be182aac83 More bro-to-zeek renaming in scripts and other files 2019-05-16 02:36:41 -05:00
Jon Siwek
aebcb1415d GH-234: rename Broxygen to Zeexygen along with roles/directives
* All "Broxygen" usages have been replaced in
  code, documentation, filenames, etc.

* Sphinx roles/directives like ":bro:see" are now ":zeek:see"

* The "--broxygen" command-line option is now "--zeexygen"
2019-04-22 19:45:50 -07:00
Jon Siwek
a994be9eeb Merge remote-tracking branch 'origin/topic/seth/zeek_init'
* origin/topic/seth/zeek_init:
  Some more testing fixes.
  Update docs and tests for bro_(init|done) -> zeek_(init|done)
  Implement the zeek_init handler.
2019-04-19 11:24:29 -07:00
Seth Hall
8cefb9be42 Implement the zeek_init handler.
Implements the change and a test.
2019-04-14 08:37:35 -04:00
Daniel Thayer
18bd74454b Rename all scripts to have ".zeek" file extension 2019-04-11 21:12:40 -05:00
Jon Siwek
4bbdd63054 BIT-1779: use BRO_LOG_SUFFIX env var in ascii log rotation function 2018-09-05 15:09:57 -05:00
Jon Siwek
5ed5587b5c Merge remote-tracking branch 'origin/topic/feature/logging-filter-list'
* origin/topic/feature/logging-filter-list:
  Logging: implement get_filter_names and small fixes.

Removed some superfluous existence checks before deleting table indices.

BIT-1890 #merged
2018-01-17 16:17:43 -06:00
Johanna Amann
a4a9bf4199 Logging: implement get_filter_names and small fixes.
get_filter_names(id: ID) : set[string] returns the names of the current
list of filters for a specified log stream.

Furthermore this commit makes a number of logging functions more robust
by checking existence of values before trying to modify them. This
commit also really implements (and tests) the enable_stream function.
2018-01-17 09:47:20 -08:00
Daniel Thayer
823fba1713 Fix ascii writer to not discard a ".gz" file extension
When Bro writes a compressed log, it uses a file extension of ".gz".
However, upon log rotation the ascii writer script function
"default_rotation_postprocessor_func" was discarding the ".gz"
file extension.  Fixed so that the correct file extension is
preserved after rotation.
2017-08-25 15:39:12 -05:00
Jon Siwek
4d06ee85ce Merge branch 'topic/corelight/ascii-gzip' of https://github.com/corelight/bro
* 'topic/corelight/ascii-gzip' of https://github.com/corelight/bro:
  Add gzip log writing to the ascii writer.
2017-05-22 00:10:09 -05:00
Johanna Amann
c868a19a28 Add gzip log writing to the ascii writer.
This feature can be enabled globally for all logs by setting
LogAscii::gzip_level to a value greater than 0.

This feature can be enabled on a per-log basis by setting gzip-level in
$confic to a value greater than 0.
2017-04-24 13:15:32 -07:00
Daniel Thayer
475682ba7f Fix minor typos in documentation
Some of these fixes are for broken links in the auto-generated docs.
2016-11-14 09:50:19 -06:00
Daniel Thayer
5745213326 Fix minor typos in documentation of various scripts 2016-11-11 14:08:17 -06:00
Daniel Thayer
462eaefc43 Fix some warnings seen while building the docs
Removed references in the docs to a few identifiers that aren't
defined in any Bro scripts.
2016-11-09 14:16:42 -06:00
Johanna Amann
6e769db23b Log extensions: series of small fixes and new tests.
The extensions now work with optional types, as well with complex types
(like subrecords). Not returning a record in the ext_func no longer
crashes bro.

The default_ext_func was switched to return void in
cases where no extension revord is defined (was bool).

I also got rid of the offsets in the indices - with the rest of the
implementation, that was not really necessary and made the code more
complex.
2016-08-10 14:14:03 -07:00
Seth Hall
79e49e1ab5 Change the function for log extension to take a path only and update tests.
We were having stack overflow issues creep in with the self-referential
field for the log extension function.
2016-08-10 13:53:27 -04:00
Seth Hall
5f6565d62c Final changes to log framework ext code.
The "metadata" functionality has been renamed to "ext" to
represent that the logs are being extended.  The function that
returns the record which is used to extend the log now receives
a log filter as it's single argument.

The field name "unrolling" is now renamed to "scope" so the variables
names now look like this: "Log::default_scope_sep"
2016-08-10 12:43:32 -04:00
Seth Hall
3e3f6f13cc Add logging framework metadata mechanism.
Example script coming soon.
2016-05-24 09:28:07 -04:00
Seth Hall
b28801ce95 Add unrolling separator & field name map to logging framework.
- When a log record is being "unrolled" (sub-records flattened
   out into a single record), it's now possible to choose the
   character/string to separate the outer name from the inner
   name.  This can be used to work around the problems
   with ElasticSearch 2.0 not supporting dots "." in field names.
   This value can be provided per-filter as well as a global
   default value.
 - Log fields can be renamed by providing a table per-filter
   (or a global default) to rename fields for any log writer.
   The name translation is performed after unrolling so the
   value in the field name table must match whatever is being
   used to separate field names.

   For example if the unrolling separator was set to "*":
	redef Log::default_unrolling_sep = "*";

   The field name map would need to reflect it:
	redef Log::default_field_name_map = {
		["id*orig_h"] = "src",
		["id*orig_p"] = "src_port",
		["id*resp_h"] = "dst",
		["id*resp_p"] = "dst_port",
	};
2016-05-16 12:28:45 -04:00
Daniel Thayer
24701f2678 Fix a "make doc" warning
Also fixed some indentation.
2015-05-29 14:38:50 -05:00
Daniel Thayer
9cde2be727 Merge remote-tracking branch 'origin/master' into topic/dnthayer/doc-improvements-2.4 2015-05-25 11:59:34 -05:00
Johanna Amann
f607d6aa30 allow to specify a remote host port to use for the sftp log rotator. 2015-04-03 08:21:22 -07:00
Daniel Thayer
8841d0ae77 Minor improvements to logging framework documentation 2015-03-19 16:01:28 -05:00
Jon Siwek
186e67ec1d Allow logging filters to inherit default path from stream.
This allows the path for the default filter to be specified explicitly
when creating a stream and reduces the need to rely on the default path
function to magically supply the path.

The default path function is now only used if, when a filter is added to
a stream, it has neither a path nor a path function already.

Adapted the existing Log::create_stream calls to explicitly specify a
path value.

Addresses BIT-1324
2015-03-19 14:49:55 -05:00
Jon Siwek
23f04835c6 Deprecate split* family of BIFs.
These functions are now deprecated in favor of alternative versions that
return a vector of strings rather than a table of strings.

Deprecated functions:

- split: use split_string instead.
- split1: use split_string1 instead.
- split_all: use split_string_all instead.
- split_n: use split_string_n instead.
- cat_string_array: see join_string_vec instead.
- cat_string_array_n: see join_string_vec instead.
- join_string_array: see join_string_vec instead.
- sort_string_array: use sort instead instead.
- find_ip_addresses: use extract_ip_addresses instead.

Changed functions:

- has_valid_octets: uses a string_vec parameter instead of string_array.

Addresses BIT-924, BIT-757.
2015-01-21 15:34:42 -06:00
Robin Sommer
996d118d68 Fixing tests. 2014-08-13 21:33:03 -07:00
Robin Sommer
58f3a715f2 Merge branch 'topic/robin/reader-writer-plugins' of git.bro.org:bro into topic/robin/reader-writer-plugins
Conflicts:
	scripts/base/frameworks/logging/writers/dataseries.bro
2014-08-08 18:36:09 -07:00
Robin Sommer
8737eae906 Move DataSeries and ElasticSearch into plugins. 2014-08-08 18:32:21 -07:00
Robin Sommer
8031da4ee7 More polishing of some of the branche's changes. 2014-08-08 18:32:05 -07:00
Robin Sommer
ffd3d9d185 More polishing. 2014-07-31 15:08:45 -07:00
Robin Sommer
f4cbcb9b03 Converting log writers and input readers to plugins. 2014-07-20 19:17:58 +02:00
Robin Sommer
a7746afa0a Fixing DataSeries, which was using a now illegal value as default
compression level.
2014-07-10 14:50:15 -07:00
Daniel Thayer
5e23e57025 Fix minor formatting issues in script docs 2014-06-12 00:33:55 -05:00
Robin Sommer
ff261ea626 Merge remote-tracking branch 'origin/topic/seth/json-formatter'
* origin/topic/seth/json-formatter:
  Updating a couple of tests.
  Expanded support for modifying the timestamp format in the JSON formatter.
  Ascii input reader now supports all config options per-input stream.
  Added an option to the JSON formatter to use ISO 8601 for timestamps.
  Refactored formatters and updated the the writers a bit.

Includes some minor bugfixes and cleanup at various places, including
in old code.
2014-03-13 16:50:56 -07:00
Seth Hall
c591e4f57f Expanded support for modifying the timestamp format in the JSON formatter. 2014-03-12 10:01:59 -04:00
Seth Hall
c9aaf9e753 Added an option to the JSON formatter to use ISO 8601 for timestamps.
- It's not *exactly* ISO 8601 which doesn't seem to support
   subseconds, but subseconds are very important to us and
   most things that support ISO8601 seem to also support subseconds
   in the way I'm implemented it.
2014-03-10 14:22:35 -04:00
Seth Hall
a56c343715 Refactored formatters and updated the the writers a bit.
- Formatters have been abstracted similarly to readers and writers now.
 - The Ascii writer has a new option for writing out logs as JSON.
 - The Ascii writer now has all options availble as per-filter
   options as well as global.
2014-03-10 10:42:59 -04:00
Robin Sommer
2818bb3dbc Merge remote-tracking branch 'origin/fastpath'
* origin/fastpath:
  make the documentation of the SQLite reader/writer a bit nicer.
  Wrong example file was included - reported by Michael Auger @LM4K
2013-10-25 11:12:51 -07:00
Bernhard Amann
a09d8e94e0 make the documentation of the SQLite reader/writer a bit nicer. 2013-10-25 10:21:33 -07:00
Daniel Thayer
72a4a90416 Add more script package README files
The text from these README files appears on the "Bro Script Packages"
page after building the documentation.
2013-10-23 16:36:14 -05:00
Daniel Thayer
4b8171f74a Document which Bro script vars are set by BroControl 2013-10-22 16:40:29 -05:00
Daniel Thayer
60b2c5f1fe Add README files for most Bro frameworks
The text from these README files appears on the "Bro Script Packages"
page after building the documentation.  The text for these was mostly just
copied from the existing docs.
2013-10-11 00:19:37 -05:00
Daniel Thayer
ee921a419f Fix typos and formatting in the logging framework docs 2013-10-10 16:10:00 -05:00
Jon Siwek
db470a637a Documentation fixes.
This cleans up most of the warnings from sphinx (broken :doc: links,
broxygen role misuses, etc.).  The remaining ones should be harmless,
but not quick to silence.

I found that the README for each component was a copy from the actual
repo, so I turned those in to symlinks so they don't get out of date.
2013-09-03 15:59:40 -05:00
Robin Sommer
c049c758c3 Merge remote-tracking branch 'origin/master' into topic/robin/plugins
Conflicts:
	aux/bro-aux
	aux/broctl
	src/DPM.cc
2013-05-30 17:43:50 -07:00
Jon Siwek
e45933562e Fix broken/missing documentation. 2013-05-23 16:53:42 -05:00
Robin Sommer
eb637f9f3e Merge remote-tracking branch 'origin/master' into topic/robin/plugins
Thanks to git this merge was less troublesome that I was afraid it
would be. Not all tests pass yet though (and file hashes have changed
unfortunately).

Conflicts:
	cmake
	doc/scripts/DocSourcesList.cmake
	scripts/base/init-bare.bro
	scripts/base/protocols/ftp/main.bro
	scripts/base/protocols/irc/dcc-send.bro
	scripts/test-all-policy.bro
	src/AnalyzerTags.h
	src/CMakeLists.txt
	src/analyzer/Analyzer.cc
	src/analyzer/protocol/file/File.cc
	src/analyzer/protocol/file/File.h
	src/analyzer/protocol/http/HTTP.cc
	src/analyzer/protocol/http/HTTP.h
	src/analyzer/protocol/mime/MIME.cc
	src/event.bif
	src/main.cc
	src/util-config.h.in
	testing/btest/Baseline/coverage.bare-load-baseline/canonified_loaded_scripts.log
	testing/btest/Baseline/coverage.default-load-baseline/canonified_loaded_scripts.log
	testing/btest/Baseline/istate.events-ssl/receiver.http.log
	testing/btest/Baseline/istate.events-ssl/sender.http.log
	testing/btest/Baseline/istate.events/receiver.http.log
	testing/btest/Baseline/istate.events/sender.http.log
2013-05-16 17:58:48 -07:00
Robin Sommer
639a6410c6 Merge remote-tracking branch 'origin/topic/bernhard/thread-cleanup'
* origin/topic/bernhard/thread-cleanup:
  and just to be really sure - always make threads go through OnWaitForStop
  hopefully finally fix last interesting race-condition
  it is apparently getting a bit late for changes at important code...
  spoke to soon (forgot to comment in line again).
  Change thread shutdown again to also work with input framework.
  Changing semantics of thread stop methods.
  Support for cleaning up threads that have terminated.
2013-05-15 17:16:41 -07:00