Commit graph

11634 commits

Author SHA1 Message Date
Robin Sommer
4b86730ef2 Reintroducing the logging::Manager's Terminate() method.
It doesn't do anything else than simply forwarding to FlushBuffers().

This is just for consistency in terminate_bro() where components get
their Terminate() called so that the main code doesn't need to know
anything more specific about what particular action to take at
shutdown.
2013-05-15 17:19:52 -07:00
Robin Sommer
639a6410c6 Merge remote-tracking branch 'origin/topic/bernhard/thread-cleanup'
* origin/topic/bernhard/thread-cleanup:
  and just to be really sure - always make threads go through OnWaitForStop
  hopefully finally fix last interesting race-condition
  it is apparently getting a bit late for changes at important code...
  spoke to soon (forgot to comment in line again).
  Change thread shutdown again to also work with input framework.
  Changing semantics of thread stop methods.
  Support for cleaning up threads that have terminated.
2013-05-15 17:16:41 -07:00
Robin Sommer
f76446fb4e Merge remote-tracking branch 'origin/topic/bernhard/metrics-samples'
Closes #1003.

* origin/topic/bernhard/metrics-samples:
  finishing touches, make test more robust, rename function in last again
  change names of data structures after talking with seth
  make last plugin nicer and samplify sqli detector
  add tests for sampler
  reservoir sampler. untested.
2013-05-15 16:11:27 -07:00
Robin Sommer
358528732c Merge branch 'topic/robin/sqlite-merge'
Closes #997.

* topic/robin/sqlite-merge: (25 commits)
  Fix to make sqlite test consistent, and updating coverage baselines
  Avoid a CMake warning about 3rdparty looking like a number.
  Fixing linker error.
  and there is no has-reader.
  make sqlite3 executable required and add test-cases for errors
  Renaming src/external -> src/3rdparty
  fix a few small rough edges (mostly comments that do no longer apply)
  fix bug in input-manager regarding enums that a writer reads without 0-terminating the string
  actually make sqlite work again (tests passed because the writer was not actually defined because of the define.)
  add sqlite distribution.
  fix warnings, update baselines, handle rotation
  add sqlite tests and fix small vector/set escaping bugs
  fix small bug with vectors and sets.
  make work with newer AsciiFormatter.
  start adding a different text for empty records for the sqlite writer.
  no, you will never guess from where I copied this file...
  make sqlite support more or less work for logging and input
  make sqlite-writer more stable.
  make it compile with new version of AsciiInputOutput
  and adapt to AsciiInputOutput - seems to work...
  ...

Conflicts:
	scripts/base/frameworks/input/__load__.bro
	src/CMakeLists.txt
	src/input.bif
	src/input/Manager.cc
	src/main.cc
	src/types.bif
	testing/btest/Baseline/coverage.bare-load-baseline/canonified_loaded_scripts.log
	testing/btest/Baseline/coverage.default-load-baseline/canonified_loaded_scripts.log
2013-05-15 16:03:19 -07:00
Bernhard Amann
f389cafc3b Merge remote-tracking branch 'origin/master' into topic/bernhard/thread-cleanup
Conflicts:
	src/main.cc
2013-05-15 16:00:49 -07:00
Robin Sommer
4fe0e22128 Fix to make sqlite test consistent, and updating coverage baselines 2013-05-15 15:56:53 -07:00
Bernhard Amann
7bf456c11a and just to be really sure - always make threads go through OnWaitForStop 2013-05-15 15:55:38 -07:00
Bernhard Amann
37566e73d8 hopefully finally fix last interesting race-condition 2013-05-15 15:36:34 -07:00
Robin Sommer
4a134872d5 Avoid a CMake warning about 3rdparty looking like a number. 2013-05-15 15:35:38 -07:00
Robin Sommer
d7df8cec24 Fixing linker error.
On Linux we need -ldl to find dlclose(), which SQLite needs. CMake has
macro for that fortunately.
2013-05-15 15:30:37 -07:00
Robin Sommer
f4fc9dd947 Merge remote-tracking branch 'origin/topic/bernhard/sqlite' into topic/robin/sqlite-merge
* origin/topic/bernhard/sqlite:
  and there is no has-reader.
  make sqlite3 executable required and add test-cases for errors
2013-05-15 15:24:02 -07:00
Robin Sommer
e050648621 Merge branch 'topic/robin/file-analysis-merge'
Closes #982.

* topic/robin/file-analysis-merge: (64 commits)
  A few more small tweaks.
  Various smalle tweaks in preparation for merging.
  FileAnalysis: load custom mime magic database just once.
  Improve a libmagic-related error message.
  FileAnalysis: add is_orig field to fa_file & Info.
  FileAnalysis: inlined doc fixes.
  FileAnalysis: optimizate connection set updating.
  FileAnalysis: optimize file handle construction.
  FileAnalysis: workarounds for older libmagics.
  FileAnalysis: add custom libmagic database.
  FileAnalysis: change terminology s/action/analyzer
  FileAnalysis: libmagic tweaks.
  FileAnalysis: add bif for setting timeout interval
  FileAnalysis: add more params to some events.
  FileAnalysis: insert explicit event queue flush points.
  FileAnalysis: remove some file events.
  FileAnalysis: finish switching hooks to events.
  FileAnalysis: checkpoint in middle of big reorganization.
  FileAnalysis: fix file type canonification for file_analysis.log
  Revert "FileAnalysis: optimize get_file_handle event queueing."
  ...

Conflicts:
	NEWS
2013-05-15 15:14:21 -07:00
Bernhard Amann
ab6d5b08a8 finishing touches, make test more robust, rename function in last again 2013-05-15 11:33:25 -07:00
Bernhard Amann
80962ad74b change names of data structures after talking with seth 2013-05-15 09:44:43 -07:00
Bernhard Amann
b0c4dcdfed make last plugin nicer and samplify sqli detector 2013-05-15 01:09:52 -07:00
Bernhard Amann
808639fc5f it is apparently getting a bit late for changes at important code... 2013-05-15 00:56:27 -07:00
Bernhard Amann
b947394990 spoke to soon (forgot to comment in line again).
Now it should work. However - this commit changes a basic assumption
of the threading queue. This basic assumption is, that nothing can
be read out of the out-queue of a dead thread. I think that reading
out of the queue of a dead thread makes perfect sense (when the thread
shuts down, pushes the rest of its work on the queue and says bye,
and wants the main thread to pick it up afterwards) - however, I
guess one can be of a differing opinion here.

In any case, it makes stuff a bit easier to understand - in my opinion.
It took me a while to find out why the messages disappear in thin
air and never arrive in the main thread ;)
2013-05-15 00:49:57 -07:00
Bernhard Amann
39f1b9e01f Change thread shutdown again to also work with input framework.
Seems to work, tests pass, but not really verified.

Major change 1:
finished flag in MsgThread was replaced by 2 flags:
child_finished and main_finished.

child_finished is set by child_thread and means that the processing
loop is stopped immediately (no longer needed, no new input messages
will be processed, if loop continues running there is an ugly delay
on shutdown). (This took me a while to realize...)

main_finished is set by a message that is sent back by the child
to the main thread when Finished() is called (and child_finished
is set). when main_finished is set, processing of output messages
stops. But all messages that the child thread pushed in the queue
before calling Finish() are still processed.

Change 2:
Logging terminate call was replaced by a smaller call that just
flushes out the cache held by the main thread. This call
has to be done before thread shutdown is called - otherwhise
the threads will be shut down before all messages are pushed
on them. (This also took me a while to realize...).

Change 3:
Input framework actually calls it stop methods correctly (everything
was prepared, function call was missing)
2013-05-14 23:45:55 -07:00
Bernhard Amann
2b009939f5 and there is no has-reader. 2013-05-14 22:12:23 -07:00
Bernhard Amann
2ad67643c0 make sqlite3 executable required and add test-cases for errors
logging: table does not contain all required columns (when extending
data structures)

input: table does not contain all required columns (when extending
data structure), wrong sql statement
2013-05-14 22:09:46 -07:00
Robin Sommer
abb350b535 Renaming src/external -> src/3rdparty
We should eventually move more 3rdparty code in here.
2013-05-14 17:14:08 -07:00
Robin Sommer
de88645d05 Merge remote-tracking branch 'origin/topic/bernhard/sqlite'
* origin/topic/bernhard/sqlite:
  fix a few small rough edges (mostly comments that do no longer apply)
  fix bug in input-manager regarding enums that a writer reads without 0-terminating the string
  actually make sqlite work again (tests passed because the writer was not actually defined because of the define.)
  add sqlite distribution.
  fix warnings, update baselines, handle rotation
  add sqlite tests and fix small vector/set escaping bugs
  fix small bug with vectors and sets.
  make work with newer AsciiFormatter.
  start adding a different text for empty records for the sqlite writer.
  no, you will never guess from where I copied this file...
  make sqlite support more or less work for logging and input
  make sqlite-writer more stable.
  make it compile with new version of AsciiInputOutput
  and adapt to AsciiInputOutput - seems to work...
  make it compile
  add SQLite reader.
  ...adapt to new api...
  now the writer supports tables and vectors.
  basic sqlite writer seems to work.
2013-05-14 17:11:09 -07:00
Robin Sommer
8220d70e88 A few more small tweaks. 2013-05-14 16:19:41 -07:00
Bernhard Amann
d939c2bdfc add tests for sampler 2013-05-13 22:11:17 -07:00
Bernhard Amann
fa58e26aa0 Merge remote-tracking branch 'origin/master' into topic/bernhard/metrics-samples 2013-05-13 21:20:25 -07:00
Bernhard Amann
bb1e2f57b9 Merge remote-tracking branch 'origin/master' into topic/bernhard/thread-cleanup 2013-05-13 21:19:09 -07:00
Bernhard Amann
56ab9285a4 Merge remote-tracking branch 'origin/master' into topic/bernhard/topk 2013-05-13 21:03:23 -07:00
Bernhard Amann
6036872faa fix a few small rough edges (mostly comments that do no longer apply) 2013-05-13 20:40:00 -07:00
Bernhard Amann
8f7619971f fix bug in input-manager regarding enums that a writer reads without
0-terminating the string
2013-05-13 20:26:41 -07:00
Bernhard Amann
6c99df508c actually make sqlite work again (tests passed because the writer
was not actually defined because of the define.)
2013-05-13 19:27:11 -07:00
Bernhard Amann
2b8c2f2316 add sqlite distribution.
Note that tests currently still require an external sqlite3
for the command-line client.

However, for that the version does not matter - so I guess while
not completely nice it could be ok...
2013-05-13 18:41:04 -07:00
Robin Sommer
1459216b20 Merge branch 'topic/robin/file-analysis-merge' of git.bro-ids.org:bro into topic/robin/file-analysis-merge
Conflicts:
	testing/btest/Baseline/coverage.bare-load-baseline/canonified_loaded_scripts.log
	testing/btest/Baseline/coverage.default-load-baseline/canonified_loaded_scripts.log
2013-05-13 17:01:55 -07:00
Robin Sommer
2aa2641844 Updating submodule(s).
[nomail]
2013-05-13 16:51:34 -07:00
Robin Sommer
7610aa31b6 Various smalle tweaks in preparation for merging. 2013-05-13 16:47:00 -07:00
Bernhard Amann
6392acecd2 fix warnings, update baselines, handle rotation 2013-05-12 20:48:17 -07:00
Bernhard Amann
747ba68030 Merge remote branch 'origin/master' into topic/bernhard/sqlite 2013-05-12 20:47:55 -07:00
Robin Sommer
e89e8d7306 Merge remote-tracking branch 'origin/topic/jsiwek/972'
Closes #972.

* origin/topic/jsiwek/972:
  Allow default function/hook/event parameters.  Addresses #972.
2013-05-10 19:30:24 -07:00
Robin Sommer
b46d261557 Merge remote-tracking branch 'origin/fastpath'
* origin/fastpath:
  prevent merge-hook of sumstats unique plugin from damaging source data.
2013-05-10 19:22:57 -07:00
Jon Siwek
e2a1d4a233 Allow default function/hook/event parameters. Addresses #972.
And changed the endianness parameter of bytestring_to_count() BIF to
default to false (big endian), mostly just to prove that the BIF parser
doesn't choke on default parameters.
2013-05-07 14:32:22 -05:00
Bernhard Amann
70f3f4343a prevent merge-hook of sumstats unique plugin from damaging source data. 2013-05-07 11:16:59 -07:00
Scott Runnels
4aa51e7608 Basic cross-referencing UIDs between files, btests, and baselines.
Also includes appropriate btest-rst-cmd directives with titles.
2013-05-07 13:33:38 -04:00
Scott Runnels
5734d7fe25 A truly embarassing number of spelling corrections. 2013-05-07 06:01:04 -04:00
Scott Runnels
a93a6535e7 Using redirection with bro-cut. Include initial btests for this document. 2013-05-07 05:56:45 -04:00
Jon Siwek
ec50cad9db Merge branch 'master' into topic/jsiwek/file-analysis
Conflicts:
	scripts/base/protocols/ftp/main.bro
	src/OpaqueVal.h
	testing/btest/Baseline/coverage.bare-load-baseline/canonified_loaded_scripts.log
	testing/btest/Baseline/coverage.default-load-baseline/canonified_loaded_scripts.log
2013-05-06 10:21:16 -05:00
Scott Runnels
4a7de4a616 Corrected a mis-typed RST include. 2013-05-05 15:24:30 -04:00
Bernhard Amann
663082e2d5 reservoir sampler. untested. 2013-05-05 11:19:53 -07:00
Scott Runnels
8ba9250cf4 Initial start for "Using Bro". 2013-05-05 00:01:12 -04:00
Bernhard Amann
cf6e768ad6 fix opaqueval-related memleak 2013-05-03 23:08:26 -07:00
Bernhard Amann
6acbbe0231 Merge remote-tracking branch 'origin/master' into topic/bernhard/topk 2013-05-03 23:04:22 -07:00
Bernhard Amann
f6e99fce11 fix opaqueval-related memleak. 2013-05-03 23:03:31 -07:00