Commit graph

11634 commits

Author SHA1 Message Date
Robin Sommer
f7a6407ab1 Reworking thread termination logic.
Turns out the finish methods weren't called correctly, caused by a
mess up with method names which all sounded too similar and the wrong
one ended up being called. I've reworked this by changing the
thread/writer/reader interfaces, which actually also simplifies them
by getting rid of the requirement for writer backends to call their
parent methods (i.e., less opportunity for errors).

This commit also includes the following (because I noticed the problem
above when working on some of these):

     - The ASCII log writer now includes "#start <timestamp>" and
      "#end <timestamp> lines in the each file. The latter supersedes
      Bernhard's "EOF" patch.

      This required a number of tests updates. The standard canonifier
      removes the timestamps, but some tests compare files directly,
      which doesn't work if they aren't printing out the same
      timestamps (like the comm tests).

     - The above required yet another change to the writer API to
       network_time to methods.

     - Renamed ASCII logger "header" options to "meta".

     - Fixes #763 "Escape # when first character in log file line".

All btests pass for me on Linux FC15. Will try MacOS next.
2012-07-19 21:22:22 -07:00
Robin Sommer
c8789cff94 If a thread doesn't terminate, we log that but not longer proceed
(because it could hang later still).

Also logging to stderr as well to make sure one sees it.

Also adding code to the ASCII writer to catch termination
inconsistencies.
2012-07-19 21:21:53 -07:00
Robin Sommer
1ca0d970fc Removing the thread kill functionality.
Wasn't really used and has potential for trouble.
2012-07-19 21:21:53 -07:00
Robin Sommer
f73eb3b086 Reworking thread termination logic.
Turns out the finish methods weren't called correctly, caused by a
mess up with method names which all sounded too similar and the wrong
one ended up being called. I've reworked this by changing the
thread/writer/reader interfaces, which actually also simplifies them
by getting rid of the requirement for writer backends to call their
parent methods (i.e., less opportunity for errors).

This commit also includes the following (because I noticed the problem
above when working on some of these):

     - The ASCII log writer now includes "#start <timestamp>" and
      "#end <timestamp> lines in the each file. The latter supersedes
      Bernhard's "EOF" patch.

      This required a number of tests updates. The standard canonifier
      removes the timestamps, but some tests compare files directly,
      which doesn't work if they aren't printing out the same
      timestamps (like the comm tests).

     - The above required yet another change to the writer API to
       network_time to methods.

     - Renamed ASCII logger "header" options to "meta".

     - Fixes #763 "Escape # when first character in log file line".

All btests pass for me on Linux FC15. Will try MacOS next.
2012-07-19 21:21:53 -07:00
Seth Hall
0c4c5ff335 Temporarily removing the ES timeout because it works with signals and is incompatible with Bro threads. 2012-07-19 12:14:13 -04:00
Seth Hall
1826827359 Changed ES index names to localtime and added a meta index. 2012-07-18 15:42:23 -04:00
Seth Hall
6335dbb5e1 Fixing calc_next_rotate to use UTC based time functions. 2012-07-18 11:32:14 -04:00
Seth Hall
43507b1bb9 New script for easily duplicating logs to ElasticSearch. 2012-07-18 11:28:41 -04:00
Robin Sommer
50f5f8131d Revert "Fixed a couple of init-time mem leaks."
This reverts commit 750e1ddf69.
2012-07-18 07:29:01 -07:00
Seth Hall
1fa182c169 Some better elasticsearch reliability.
- Added a configurable option for timing out ES HTTP requests.

 - Stop sending reporter messages after one message for one failure.
2012-07-18 00:00:31 -04:00
Bernhard Amann
81edec8b2e Fix bug, where in dns.log rcode always was set to 0/NOERROR when
no reply package was seen.

In the fixed version rcode is only set when a reply packet was seen.

Updates for the baseline have been commited separately in the
topic/bernhard/dns-fix branch.
2012-07-17 14:16:15 -07:00
Seth Hall
485e473561 Merge remote-tracking branch 'origin/master' into topic/seth/elasticsearch 2012-07-17 13:57:59 -04:00
Seth Hall
5d04d58385 Fixed small elasticsearch problem in configure output. 2012-07-17 13:57:23 -04:00
Seth Hall
750e1ddf69 Fixed a couple of init-time mem leaks. 2012-07-16 15:51:47 -04:00
Seth Hall
ce05600a71 Mozilla's current certificate bundle. 2012-07-13 22:24:34 -04:00
Robin Sommer
86f6f36f70 Merge remote-tracking branch 'vlad/info_record_fixes'
* vlad/info_record_fixes:
  Fix some Info:Record field documentation.

Closes #851. Many thanks!
2012-07-13 16:20:49 -07:00
Robin Sommer
731292a7a9 Merge remote-tracking branch 'origin/fastpath'
* origin/fastpath:
  Fix overrides of TCP_ApplicationAnalyzer::EndpointEOF.
  Fix segfault when incrementing whole vector values.
  Remove baselines for some leak-detecting unit tests.
  Unblock SIGFPE, SIGILL, SIGSEGV and SIGBUS for threads.
2012-07-13 16:00:26 -07:00
Jon Siwek
0ef91538db Fix overrides of TCP_ApplicationAnalyzer::EndpointEOF.
In many cases, classes derived from TCP_ApplicationAnalyzer were
*overloading* instead of overriding EndpointEOF causing the parent
class version to become hidden in the child and also for the child's
version to never to called polymorphically from
TCP_Analyzer::EndpointEOF.  Clang gave a warning in each case.
2012-07-13 16:25:58 -05:00
Jon Siwek
353393f9bd Fix segfault when incrementing whole vector values.
Also removed RefExpr::Eval(Val*) method since it was never called
(Clang emitted warning about this hiding overloaded virtual function
UnaryExpr::Eval(Frame*)) and doesn't appear to be necessary even if it
was called to avoid the default vector handling of UnaryExpr::Eval
(as the comment suggests as the intention).
2012-07-13 14:32:50 -05:00
Jon Siwek
8279de25c9 Remove baselines for some leak-detecting unit tests.
They were drifting from the non-leak-detecting unit tests and the point
of these was just leak detecting anyway, don't need the redundancy.
2012-07-13 14:25:31 -05:00
Jon Siwek
491c78e788 Merge branch 'master' into fastpath 2012-07-13 14:23:40 -05:00
Vlad Grigorescu
f43576cff3 Fix some Info:Record field documentation. 2012-07-13 14:04:24 -04:00
Bernhard Amann
e1bd960926 Unblock SIGFPE, SIGILL, SIGSEGV and SIGBUS for threads.
According to POSIX, behavior is unspecified if a specific thread receives one of those signals (because of e.g. executing an invalid instruction) if the signal is blocked.

This resulted in segfaults in threads not propagating to the main thread.

Adresses #848
2012-07-13 02:20:41 -07:00
Daniel Thayer
56b4209597 Merge remote-tracking branch 'origin/master' into topic/dnthayer/alarms-mail 2012-07-12 12:20:16 -05:00
Seth Hall
5607e86ad3 Reporter warnings and error now print to stderr by default.
- Changed the geoip warnings to Info.
2012-07-12 12:55:34 -04:00
Robin Sommer
5d13e4f949 Merge remote-tracking branch 'origin/topic/seth/socks-fixes'
* origin/topic/seth/socks-fixes:
  Some small fixes to further reduce SOCKS false positive logs.

Closes #847.
2012-07-12 08:16:28 -07:00
Robin Sommer
5eda72142f Merge remote-tracking branch 'origin/fastpath'
* origin/fastpath:
  make pthread_mutex_unlock include the reason for why the unlock fails.
2012-07-12 08:13:02 -07:00
Bernhard Amann
8ff8c66655 make pthread_mutex_unlock include the reason for why the unlock fails. 2012-07-11 20:10:49 -07:00
Seth Hall
a44612788e Some small fixes to further reduce SOCKS false positive logs. 2012-07-11 16:53:46 -04:00
Robin Sommer
f92e47c220 Merge remote-tracking branch 'origin/fastpath'
* origin/fastpath:
  Fixing memory leak.
2012-07-11 08:32:02 -07:00
Seth Hall
b31ef8cde5 Fixing memory leak. 2012-07-11 10:58:57 -04:00
Seth Hall
6e5382da54 Re-adding the needed call to FinishedRotation in the ES writer plugin. 2012-07-10 23:49:31 -04:00
Seth Hall
728888ad45 Merge remote-tracking branch 'origin/master' into topic/seth/elasticsearch 2012-07-10 20:32:42 -04:00
Robin Sommer
e3f6a467a4 Merge remote-tracking branch 'origin/fastpath'
* origin/fastpath:
  Add sorting canonifier to rotate-custom unit test. (addresses #846)
  Fix compiler warnings
  Fix segfault when there's an error/timeout resolving DNS requests.
2012-07-10 16:26:34 -07:00
Jon Siwek
c4b6499d85 Add sorting canonifier to rotate-custom unit test. (addresses #846)
The output on stderr for this test is the results of many backgrounded
"echo" commands, one for each rotation, so the order in which they
occur may be subject to OS process scheduling and can't be relied upon
2012-07-10 16:27:03 -05:00
Daniel Thayer
7f4b0b52f8 Fix compiler warnings
These changes eliminate 405 of 571 warnings seen on OS X 10.7.4 with clang.
2012-07-10 15:39:05 -05:00
Jon Siwek
c0bbd78ee1 Fix segfault when there's an error/timeout resolving DNS requests.
Addresses #846.
2012-07-10 11:15:48 -05:00
Seth Hall
9b70ee8799 Tiny updates. 2012-07-09 16:50:42 -04:00
Seth Hall
84e91b8b8d Bringing elasticsearch branch up to date with master. 2012-07-09 16:38:05 -04:00
Seth Hall
601d1cf37e Merge remote-tracking branch 'origin/master' into topic/seth/elasticsearch
Conflicts:
	aux/binpac
	aux/bro-aux
	aux/broccoli
	aux/broctl
	scripts/base/frameworks/logging/__load__.bro
	src/logging.bif
2012-07-06 12:01:16 -04:00
Robin Sommer
77261ec0db Merge remote-tracking branch 'origin/fastpath'
* origin/fastpath:
  Remove a non-portable test case
2012-07-06 07:36:57 -07:00
Daniel Thayer
1b8673f4b2 Remove a non-portable test case 2012-07-05 17:58:44 -05:00
Robin Sommer
818c76243f Updating submodule(s).
[nomail]
2012-07-05 13:00:41 -07:00
Robin Sommer
658d1d0804 Updating submodule(s).
[nomail]
2012-07-05 12:58:19 -07:00
Robin Sommer
11bc88e41a Updating submodule(s).
[nomail]
2012-07-05 12:33:57 -07:00
Robin Sommer
533d01590a Merge remote-tracking branch 'origin/fastpath'
* origin/fastpath:
  Fix minor typos in input framework doc
  Fix minor typos in dataseries documentation
2012-07-05 12:30:08 -07:00
Daniel Thayer
cee78f8f5d Fix minor typos in input framework doc
Also simplified the opening paragraph, and reformatted input text to fit
on 80-column display for better readability.
2012-07-05 12:59:19 -05:00
Daniel Thayer
8dc1e41876 Fix minor typos in dataseries documentation 2012-07-03 18:20:52 -05:00
Robin Sommer
acb6c0a0a5 Merge branch 'robin/topic/writer-info'
* robin/topic/writer-info:
  Bugfix. Using a custom rotate function was broken.
2012-07-02 16:59:56 -07:00
Robin Sommer
d26a96bd47 Bugfix. Using a custom rotate function was broken. 2012-07-02 16:57:16 -07:00