Commit graph

56 commits

Author SHA1 Message Date
Daniel Thayer
24701f2678 Fix a "make doc" warning
Also fixed some indentation.
2015-05-29 14:38:50 -05:00
Daniel Thayer
8841d0ae77 Minor improvements to logging framework documentation 2015-03-19 16:01:28 -05:00
Robin Sommer
996d118d68 Fixing tests. 2014-08-13 21:33:03 -07:00
Robin Sommer
58f3a715f2 Merge branch 'topic/robin/reader-writer-plugins' of git.bro.org:bro into topic/robin/reader-writer-plugins
Conflicts:
	scripts/base/frameworks/logging/writers/dataseries.bro
2014-08-08 18:36:09 -07:00
Robin Sommer
8737eae906 Move DataSeries and ElasticSearch into plugins. 2014-08-08 18:32:21 -07:00
Robin Sommer
8031da4ee7 More polishing of some of the branche's changes. 2014-08-08 18:32:05 -07:00
Robin Sommer
ffd3d9d185 More polishing. 2014-07-31 15:08:45 -07:00
Robin Sommer
f4cbcb9b03 Converting log writers and input readers to plugins. 2014-07-20 19:17:58 +02:00
Robin Sommer
a7746afa0a Fixing DataSeries, which was using a now illegal value as default
compression level.
2014-07-10 14:50:15 -07:00
Daniel Thayer
5e23e57025 Fix minor formatting issues in script docs 2014-06-12 00:33:55 -05:00
Robin Sommer
ff261ea626 Merge remote-tracking branch 'origin/topic/seth/json-formatter'
* origin/topic/seth/json-formatter:
  Updating a couple of tests.
  Expanded support for modifying the timestamp format in the JSON formatter.
  Ascii input reader now supports all config options per-input stream.
  Added an option to the JSON formatter to use ISO 8601 for timestamps.
  Refactored formatters and updated the the writers a bit.

Includes some minor bugfixes and cleanup at various places, including
in old code.
2014-03-13 16:50:56 -07:00
Seth Hall
c591e4f57f Expanded support for modifying the timestamp format in the JSON formatter. 2014-03-12 10:01:59 -04:00
Seth Hall
c9aaf9e753 Added an option to the JSON formatter to use ISO 8601 for timestamps.
- It's not *exactly* ISO 8601 which doesn't seem to support
   subseconds, but subseconds are very important to us and
   most things that support ISO8601 seem to also support subseconds
   in the way I'm implemented it.
2014-03-10 14:22:35 -04:00
Seth Hall
a56c343715 Refactored formatters and updated the the writers a bit.
- Formatters have been abstracted similarly to readers and writers now.
 - The Ascii writer has a new option for writing out logs as JSON.
 - The Ascii writer now has all options availble as per-filter
   options as well as global.
2014-03-10 10:42:59 -04:00
Robin Sommer
2818bb3dbc Merge remote-tracking branch 'origin/fastpath'
* origin/fastpath:
  make the documentation of the SQLite reader/writer a bit nicer.
  Wrong example file was included - reported by Michael Auger @LM4K
2013-10-25 11:12:51 -07:00
Bernhard Amann
a09d8e94e0 make the documentation of the SQLite reader/writer a bit nicer. 2013-10-25 10:21:33 -07:00
Daniel Thayer
ee921a419f Fix typos and formatting in the logging framework docs 2013-10-10 16:10:00 -05:00
Robin Sommer
de88645d05 Merge remote-tracking branch 'origin/topic/bernhard/sqlite'
* origin/topic/bernhard/sqlite:
  fix a few small rough edges (mostly comments that do no longer apply)
  fix bug in input-manager regarding enums that a writer reads without 0-terminating the string
  actually make sqlite work again (tests passed because the writer was not actually defined because of the define.)
  add sqlite distribution.
  fix warnings, update baselines, handle rotation
  add sqlite tests and fix small vector/set escaping bugs
  fix small bug with vectors and sets.
  make work with newer AsciiFormatter.
  start adding a different text for empty records for the sqlite writer.
  no, you will never guess from where I copied this file...
  make sqlite support more or less work for logging and input
  make sqlite-writer more stable.
  make it compile with new version of AsciiInputOutput
  and adapt to AsciiInputOutput - seems to work...
  make it compile
  add SQLite reader.
  ...adapt to new api...
  now the writer supports tables and vectors.
  basic sqlite writer seems to work.
2013-05-14 17:11:09 -07:00
Bernhard Amann
0fcc3db9a0 start adding a different text for empty records for the sqlite writer.
Sadly there also seems to be another deadlock issue which I am currently
not really able to figure out - on shutdown sometimes (too often) the main
thread + all sqlite threads wait for semaphores or mutexes.
2013-01-16 18:13:39 -08:00
Bernhard Amann
5704496f26 Merge branch 'topic/bernhard/input-logging-commmon-functions' into topic/bernhard/sqlite 2013-01-13 19:24:44 -08:00
Bernhard Amann
12753f31ae Merge remote-tracking branch 'origin/master' into topic/bernhard/input-logging-commmon-functions 2012-12-13 11:22:58 -08:00
Robin Sommer
63d43e6545 Renaming ASCII writer filter option 'only_single_header_row' to 'tsv'.
Also clarifying usage.

Closes #912.
2012-12-03 14:40:38 -08:00
Bernhard Amann
9c09dee294 and adapt to AsciiInputOutput - seems to work... 2012-12-03 14:14:40 -08:00
Bernhard Amann
0a59d0d4db Merge branch 'topic/bernhard/input-logging-commmon-functions' into topic/bernhard/sqlite 2012-12-03 13:46:58 -08:00
Bernhard Amann
501328d61a factor out ascii input/output.
First step - factored out everything the logging classes
use ( so only output ).

Moved the script-level configuration to logging/main,
and made the individual writers just refer to it -
no idea if this is good design. It works. But I am happy
about opinions :)

Next step - add support for input...
2012-12-03 12:59:11 -08:00
Bernhard Amann
2d7ffd8269 Merge remote-tracking branch 'origin/master' into topic/bernhard/sqlite 2012-11-26 20:46:27 -08:00
Robin Sommer
a5e237f50c The ASCII writer now supports a filter config option
'only_single_header_row' that turns the output into CSV format.

In that mode all meta data is skipped except for a single header line
with the fields names. Example:

    local my_filter: Log::Filter = [$name = "my-filter", $writer = Log::WRITER_ASCII, $config = table(["only_single_header_row"] = "T")];

Contributed by Carsten Langer.
2012-11-23 19:38:53 -08:00
Bernhard Amann
87ef8fe649 Merge remote-tracking branch 'origin/master' into topic/bernhard/sqlite 2012-10-08 10:31:22 -07:00
Robin Sommer
6ae8ac310a Merge remote-tracking branch 'origin/topic/matthias/libcurl-fix'
* origin/topic/matthias/libcurl-fix:
  Use second granularity for ElasticSearch timeouts.
  Fix compile issues with older versions of libcurl.
2012-10-02 12:07:26 -07:00
Matthias Vallentin
1ce76da90f Use second granularity for ElasticSearch timeouts.
Since the millisecond resolution cannot be harnessed universally and is not
supported by older version of libcurl, we will allow only specifications at the
granularity of seconds.

This commit also fixes a typing issue that causes that prevented the
ElasticSearch timeout to work in the first place: curl_easy_setopt requires a
long but was given a uint64_t.
2012-09-27 16:34:16 -07:00
Daniel Thayer
c3aba199f6 Fix build warnings 2012-07-25 17:40:21 -05:00
Bernhard Amann
da157c8ded Merge remote-tracking branch 'origin/master' into topic/bernhard/sqlite
Conflicts:
	scripts/base/frameworks/logging/__load__.bro
	src/CMakeLists.txt
	src/logging.bif
	src/types.bif
2012-07-25 15:04:23 -07:00
Robin Sommer
eef8b7d1c4 Merge remote-tracking branch 'remotes/origin/topic/seth/elasticsearch' into topic/robin/master-test
I've only tested that it compiles, not whether it still works. The
fact that we don't have any tests for this makes me uneasy ...

* remotes/origin/topic/seth/elasticsearch: (35 commits)
  Some documentation updates for elasticsearch plugin.
  Temporarily removing the ES timeout because it works with signals and is incompatible with Bro threads.
  Changed ES index names to localtime and added a meta index.
  New script for easily duplicating logs to ElasticSearch.
  Some better elasticsearch reliability.
  Fixed small elasticsearch problem in configure output.
  Re-adding the needed call to FinishedRotation in the ES writer plugin.
  Tiny updates.
  Bringing elasticsearch branch up to date with master.
  Adding a define to make the stdint C macros available.
  Adding an extra header.
  Fixed a bug with messed up time value passing to elasticsearch.
  Small updates and a little standardization for config.h.in naming.
  Bug fixes.
  Bug fix and feature.
  Forgot to call the parent method for DoHeartBeat.
  Changed the escaping method.
  Flush logs to ES daemon as Bro is shutting down.
  Reduce the batch size to 1000 and add a maximum time interval for batches.
  Reworked bulk operation string construction to use ODesc and added json escaping.
  ...
2012-07-20 07:43:05 -07:00
Seth Hall
db3d89d290 Some documentation updates for elasticsearch plugin. 2012-07-20 08:51:39 -04:00
Robin Sommer
f73eb3b086 Reworking thread termination logic.
Turns out the finish methods weren't called correctly, caused by a
mess up with method names which all sounded too similar and the wrong
one ended up being called. I've reworked this by changing the
thread/writer/reader interfaces, which actually also simplifies them
by getting rid of the requirement for writer backends to call their
parent methods (i.e., less opportunity for errors).

This commit also includes the following (because I noticed the problem
above when working on some of these):

     - The ASCII log writer now includes "#start <timestamp>" and
      "#end <timestamp> lines in the each file. The latter supersedes
      Bernhard's "EOF" patch.

      This required a number of tests updates. The standard canonifier
      removes the timestamps, but some tests compare files directly,
      which doesn't work if they aren't printing out the same
      timestamps (like the comm tests).

     - The above required yet another change to the writer API to
       network_time to methods.

     - Renamed ASCII logger "header" options to "meta".

     - Fixes #763 "Escape # when first character in log file line".

All btests pass for me on Linux FC15. Will try MacOS next.
2012-07-19 21:21:53 -07:00
Seth Hall
1fa182c169 Some better elasticsearch reliability.
- Added a configurable option for timing out ES HTTP requests.

 - Stop sending reporter messages after one message for one failure.
2012-07-18 00:00:31 -04:00
Seth Hall
84e91b8b8d Bringing elasticsearch branch up to date with master. 2012-07-09 16:38:05 -04:00
Seth Hall
601d1cf37e Merge remote-tracking branch 'origin/master' into topic/seth/elasticsearch
Conflicts:
	aux/binpac
	aux/bro-aux
	aux/broccoli
	aux/broctl
	scripts/base/frameworks/logging/__load__.bro
	src/logging.bif
2012-07-06 12:01:16 -04:00
Robin Sommer
90763bb2f2 Merge branch 'robin/topic/writer-info'
* robin/topic/writer-info:
  Extending the log writer DoInit() API.
  Reworking log writer API to make it easier to pass additional information to a writer's initialization method.

Conflicts:
	src/logging/WriterBackend.cc
	src/logging/WriterBackend.h
	src/logging/WriterFrontend.cc
2012-07-02 15:20:37 -07:00
Bernhard Amann
b8ad4567fb Merge branch 'topic/bernhard/reader-info' into topic/bernhard/sqlite
Now uses optional dbname configuration option

Conflicts:
	scripts/base/frameworks/logging/__load__.bro
	src/logging.bif
2012-07-02 10:13:54 -07:00
Robin Sommer
19eea409c3 Extending the log writer DoInit() API.
We now pass in a Info struct that contains:

    - the path name (as before)
    - the rotation interval
    - the log_rotate_base_time in seconds
    - a table of key/value pairs with further configuration options.

To fill the table, log filters have a new field "config: table[string]
of strings". This gives a way to pass arbitrary values from
script-land to writers. Interpretation is left up to the writer.

Also splits calc_next_rotate() into two functions, one of which is
thread-safe and can be used with the log_rotate_base_time value from
DoInit().

Includes also updates to the None writer:

    - It gets its own script writers/none.bro.

    - New bool option LogNone::debug to enable debug output. It then
      prints out all the values passed to DoInit(). That's used by a
      btest test to ensure the new DoInit() values are right.

    - Fixed a bug that prevented Bro from terminating..

(scripts.base.frameworks.logging.rotate-custom currently fails.
Haven't yet investigated why.)
2012-06-21 17:42:33 -07:00
Seth Hall
cd8169dda3 Bug fix and feature.
- Fixed bug with how data is sent to elasticsearch.

- Added a feature to only allow data of a certain
  size to be buffered before sending to the
  elasticsearch server.  Configured with the
  LogElasticSearch::max_byte_size variable.
2012-06-16 22:22:40 -04:00
Seth Hall
a4df914ab7 Reduce the batch size to 1000 and add a maximum time interval for batches. 2012-06-15 20:53:09 -04:00
Bernhard Amann
c664c40ac2 now the writer supports tables and vectors.
still not tested, but using Log::default_writer=Log::WRITER_SQLITE seems to generate all
the right log-databases, etc.
2012-06-14 15:54:22 -07:00
Vlad Grigorescu
a3b330dbc6 Make default index name 'bro'. 2012-06-05 17:24:13 -04:00
Vlad Grigorescu
7bee0b0d8e Added sending messages to ElasticSearch over HTTP. 2012-06-03 22:07:20 -04:00
Robin Sommer
da34266a52 Switching default DS compression to gzip. 2012-05-25 08:36:59 -07:00
Robin Sommer
7cc863c5fc Fix for when not producing local output; that hung.
* origin/topic/robin/dataseries:
  Moving trace for rotation test into traces directory.
  Fixing a rotation race condition at termination.
  Portability fixes.
  Extending DS docs with some examples.
  Updating doc.
  Fixing pack_scale and time-as-int.
  Adding format specifier to DS spec to print out double as %.6f.
  DataSeries updates and fixes.
  DataSeries tuning.
  Tweaking DataSeries support.
  Extending log post-processor call to include the name of the writer.
  Removing an unnecessary const cast.
  DataSeries TODO list with open issues/questions.
  Starting DataSeries HowTo.
  Additional test output canonification for ds2txt's timestamps.
  In threads, an internal error now immediately aborts.
  DataSeries cleanup.
  Working on DataSeries support.
  Merging in DataSeries support from topic/gilbert/logging.
  Fixing  threads' DoFinish() method.
2012-05-17 12:38:47 -07:00
Robin Sommer
7131feefbc Working on DataSeries support.
- The option to use integers insteads of double was ignored.

   - Renaming script-level options to remove the ds_ prefix.

   - Log rotation didn't work.

   - A set of simple unit tests.
2012-04-09 17:46:56 -07:00
Robin Sommer
952b6b293a Merging in DataSeries support from topic/gilbert/logging.
I copied the code over manually, no merging, because (1) it needed to
be adapted to the new threading API, and (2) there's more stuff in the
branch that I haven't ported yet.

The DS output generally seems to work, but it has seen no further
testing yet.

Not unit tests yet either.
2012-04-03 22:14:56 -07:00