Commit graph

6667 commits

Author SHA1 Message Date
Robin Sommer
4a235f7bbb Merge remote-tracking branch 'origin/fastpath'
* origin/fastpath:
  Fix uninitialized DPM member.
2013-05-17 13:38:01 -07:00
Jon Siwek
945aa8a550 Fix uninitialized DPM member.
Was seeing crashes due to this primarily on Ubuntu 12.04 when
generating reST docs.
2013-05-17 14:13:47 -05:00
Robin Sommer
7b50f97d39 Removing Broxygen's dpd_config magic.
The table doesn't exist anymore. The functionality has been replaced
with a function call but I can't really see how to get it out of
there. We could get it from the analyzer manager, however then we
can't tie it back to a script anymore. What we could do eventually is
add this to the information about the analyzer plugin.
2013-05-16 21:09:44 -07:00
Robin Sommer
eb637f9f3e Merge remote-tracking branch 'origin/master' into topic/robin/plugins
Thanks to git this merge was less troublesome that I was afraid it
would be. Not all tests pass yet though (and file hashes have changed
unfortunately).

Conflicts:
	cmake
	doc/scripts/DocSourcesList.cmake
	scripts/base/init-bare.bro
	scripts/base/protocols/ftp/main.bro
	scripts/base/protocols/irc/dcc-send.bro
	scripts/test-all-policy.bro
	src/AnalyzerTags.h
	src/CMakeLists.txt
	src/analyzer/Analyzer.cc
	src/analyzer/protocol/file/File.cc
	src/analyzer/protocol/file/File.h
	src/analyzer/protocol/http/HTTP.cc
	src/analyzer/protocol/http/HTTP.h
	src/analyzer/protocol/mime/MIME.cc
	src/event.bif
	src/main.cc
	src/util-config.h.in
	testing/btest/Baseline/coverage.bare-load-baseline/canonified_loaded_scripts.log
	testing/btest/Baseline/coverage.default-load-baseline/canonified_loaded_scripts.log
	testing/btest/Baseline/istate.events-ssl/receiver.http.log
	testing/btest/Baseline/istate.events-ssl/sender.http.log
	testing/btest/Baseline/istate.events/receiver.http.log
	testing/btest/Baseline/istate.events/sender.http.log
2013-05-16 17:58:48 -07:00
Robin Sommer
11fd12b18e Band-aid to get Broxygen's bif documentation back.
We'll need a different approach here eventually.
2013-05-16 16:57:49 -07:00
Robin Sommer
8752870967 Fixing cmake dependencies and "make {install,clean}"
Also includes a bunch of cleanup.
2013-05-16 15:23:23 -07:00
Seth Hall
a65966c2d1 Make the dos code available in script land. 2013-05-15 23:34:01 -04:00
Seth Hall
7ff8c1ebdd Add the PE analyzer back in as a registered file analyzer. 2013-05-15 23:33:37 -04:00
Robin Sommer
5ff7621328 Fix for 'fchmod undeclared here' on FreeBSD.
This is from http://www.sqlite.org/cgi/src/tktview/de87b8dc7b367965c13d16becfd6996bbcd4be80

Doesn't seem applied yet, and may not be the best solution anyways.
2013-05-15 19:13:51 -07:00
Robin Sommer
bdc43fc8dd CMake policy fix to avoid errors with older version.
Looks like we could avoid this by raising the minimum cmake version to 2.6.4.
2013-05-15 19:06:38 -07:00
Seth Hall
98f6be4d7c Merge remote-tracking branch 'origin/master' into topic/seth/file-analysis-exe-analyzer
Conflicts:
	src/types.bif
2013-05-15 21:35:28 -04:00
Seth Hall
d1dd4cb688 PE analyzer checkpoint 2013-05-15 21:33:14 -04:00
Robin Sommer
b9b691e31b CMake cosmetics. 2013-05-15 18:25:50 -07:00
Robin Sommer
4b86730ef2 Reintroducing the logging::Manager's Terminate() method.
It doesn't do anything else than simply forwarding to FlushBuffers().

This is just for consistency in terminate_bro() where components get
their Terminate() called so that the main code doesn't need to know
anything more specific about what particular action to take at
shutdown.
2013-05-15 17:19:52 -07:00
Robin Sommer
639a6410c6 Merge remote-tracking branch 'origin/topic/bernhard/thread-cleanup'
* origin/topic/bernhard/thread-cleanup:
  and just to be really sure - always make threads go through OnWaitForStop
  hopefully finally fix last interesting race-condition
  it is apparently getting a bit late for changes at important code...
  spoke to soon (forgot to comment in line again).
  Change thread shutdown again to also work with input framework.
  Changing semantics of thread stop methods.
  Support for cleaning up threads that have terminated.
2013-05-15 17:16:41 -07:00
Robin Sommer
358528732c Merge branch 'topic/robin/sqlite-merge'
Closes #997.

* topic/robin/sqlite-merge: (25 commits)
  Fix to make sqlite test consistent, and updating coverage baselines
  Avoid a CMake warning about 3rdparty looking like a number.
  Fixing linker error.
  and there is no has-reader.
  make sqlite3 executable required and add test-cases for errors
  Renaming src/external -> src/3rdparty
  fix a few small rough edges (mostly comments that do no longer apply)
  fix bug in input-manager regarding enums that a writer reads without 0-terminating the string
  actually make sqlite work again (tests passed because the writer was not actually defined because of the define.)
  add sqlite distribution.
  fix warnings, update baselines, handle rotation
  add sqlite tests and fix small vector/set escaping bugs
  fix small bug with vectors and sets.
  make work with newer AsciiFormatter.
  start adding a different text for empty records for the sqlite writer.
  no, you will never guess from where I copied this file...
  make sqlite support more or less work for logging and input
  make sqlite-writer more stable.
  make it compile with new version of AsciiInputOutput
  and adapt to AsciiInputOutput - seems to work...
  ...

Conflicts:
	scripts/base/frameworks/input/__load__.bro
	src/CMakeLists.txt
	src/input.bif
	src/input/Manager.cc
	src/main.cc
	src/types.bif
	testing/btest/Baseline/coverage.bare-load-baseline/canonified_loaded_scripts.log
	testing/btest/Baseline/coverage.default-load-baseline/canonified_loaded_scripts.log
2013-05-15 16:03:19 -07:00
Bernhard Amann
f389cafc3b Merge remote-tracking branch 'origin/master' into topic/bernhard/thread-cleanup
Conflicts:
	src/main.cc
2013-05-15 16:00:49 -07:00
Bernhard Amann
7bf456c11a and just to be really sure - always make threads go through OnWaitForStop 2013-05-15 15:55:38 -07:00
Bernhard Amann
37566e73d8 hopefully finally fix last interesting race-condition 2013-05-15 15:36:34 -07:00
Robin Sommer
4a134872d5 Avoid a CMake warning about 3rdparty looking like a number. 2013-05-15 15:35:38 -07:00
Robin Sommer
d7df8cec24 Fixing linker error.
On Linux we need -ldl to find dlclose(), which SQLite needs. CMake has
macro for that fortunately.
2013-05-15 15:30:37 -07:00
Robin Sommer
e050648621 Merge branch 'topic/robin/file-analysis-merge'
Closes #982.

* topic/robin/file-analysis-merge: (64 commits)
  A few more small tweaks.
  Various smalle tweaks in preparation for merging.
  FileAnalysis: load custom mime magic database just once.
  Improve a libmagic-related error message.
  FileAnalysis: add is_orig field to fa_file & Info.
  FileAnalysis: inlined doc fixes.
  FileAnalysis: optimizate connection set updating.
  FileAnalysis: optimize file handle construction.
  FileAnalysis: workarounds for older libmagics.
  FileAnalysis: add custom libmagic database.
  FileAnalysis: change terminology s/action/analyzer
  FileAnalysis: libmagic tweaks.
  FileAnalysis: add bif for setting timeout interval
  FileAnalysis: add more params to some events.
  FileAnalysis: insert explicit event queue flush points.
  FileAnalysis: remove some file events.
  FileAnalysis: finish switching hooks to events.
  FileAnalysis: checkpoint in middle of big reorganization.
  FileAnalysis: fix file type canonification for file_analysis.log
  Revert "FileAnalysis: optimize get_file_handle event queueing."
  ...

Conflicts:
	NEWS
2013-05-15 15:14:21 -07:00
Bernhard Amann
808639fc5f it is apparently getting a bit late for changes at important code... 2013-05-15 00:56:27 -07:00
Bernhard Amann
b947394990 spoke to soon (forgot to comment in line again).
Now it should work. However - this commit changes a basic assumption
of the threading queue. This basic assumption is, that nothing can
be read out of the out-queue of a dead thread. I think that reading
out of the queue of a dead thread makes perfect sense (when the thread
shuts down, pushes the rest of its work on the queue and says bye,
and wants the main thread to pick it up afterwards) - however, I
guess one can be of a differing opinion here.

In any case, it makes stuff a bit easier to understand - in my opinion.
It took me a while to find out why the messages disappear in thin
air and never arrive in the main thread ;)
2013-05-15 00:49:57 -07:00
Bernhard Amann
39f1b9e01f Change thread shutdown again to also work with input framework.
Seems to work, tests pass, but not really verified.

Major change 1:
finished flag in MsgThread was replaced by 2 flags:
child_finished and main_finished.

child_finished is set by child_thread and means that the processing
loop is stopped immediately (no longer needed, no new input messages
will be processed, if loop continues running there is an ugly delay
on shutdown). (This took me a while to realize...)

main_finished is set by a message that is sent back by the child
to the main thread when Finished() is called (and child_finished
is set). when main_finished is set, processing of output messages
stops. But all messages that the child thread pushed in the queue
before calling Finish() are still processed.

Change 2:
Logging terminate call was replaced by a smaller call that just
flushes out the cache held by the main thread. This call
has to be done before thread shutdown is called - otherwhise
the threads will be shut down before all messages are pushed
on them. (This also took me a while to realize...).

Change 3:
Input framework actually calls it stop methods correctly (everything
was prepared, function call was missing)
2013-05-14 23:45:55 -07:00
Robin Sommer
abb350b535 Renaming src/external -> src/3rdparty
We should eventually move more 3rdparty code in here.
2013-05-14 17:14:08 -07:00
Robin Sommer
de88645d05 Merge remote-tracking branch 'origin/topic/bernhard/sqlite'
* origin/topic/bernhard/sqlite:
  fix a few small rough edges (mostly comments that do no longer apply)
  fix bug in input-manager regarding enums that a writer reads without 0-terminating the string
  actually make sqlite work again (tests passed because the writer was not actually defined because of the define.)
  add sqlite distribution.
  fix warnings, update baselines, handle rotation
  add sqlite tests and fix small vector/set escaping bugs
  fix small bug with vectors and sets.
  make work with newer AsciiFormatter.
  start adding a different text for empty records for the sqlite writer.
  no, you will never guess from where I copied this file...
  make sqlite support more or less work for logging and input
  make sqlite-writer more stable.
  make it compile with new version of AsciiInputOutput
  and adapt to AsciiInputOutput - seems to work...
  make it compile
  add SQLite reader.
  ...adapt to new api...
  now the writer supports tables and vectors.
  basic sqlite writer seems to work.
2013-05-14 17:11:09 -07:00
Robin Sommer
8220d70e88 A few more small tweaks. 2013-05-14 16:19:41 -07:00
Bernhard Amann
bb1e2f57b9 Merge remote-tracking branch 'origin/master' into topic/bernhard/thread-cleanup 2013-05-13 21:19:09 -07:00
Bernhard Amann
56ab9285a4 Merge remote-tracking branch 'origin/master' into topic/bernhard/topk 2013-05-13 21:03:23 -07:00
Bernhard Amann
6036872faa fix a few small rough edges (mostly comments that do no longer apply) 2013-05-13 20:40:00 -07:00
Bernhard Amann
8f7619971f fix bug in input-manager regarding enums that a writer reads without
0-terminating the string
2013-05-13 20:26:41 -07:00
Bernhard Amann
6c99df508c actually make sqlite work again (tests passed because the writer
was not actually defined because of the define.)
2013-05-13 19:27:11 -07:00
Bernhard Amann
2b8c2f2316 add sqlite distribution.
Note that tests currently still require an external sqlite3
for the command-line client.

However, for that the version does not matter - so I guess while
not completely nice it could be ok...
2013-05-13 18:41:04 -07:00
Robin Sommer
7610aa31b6 Various smalle tweaks in preparation for merging. 2013-05-13 16:47:00 -07:00
Bernhard Amann
6392acecd2 fix warnings, update baselines, handle rotation 2013-05-12 20:48:17 -07:00
Bernhard Amann
747ba68030 Merge remote branch 'origin/master' into topic/bernhard/sqlite 2013-05-12 20:47:55 -07:00
Robin Sommer
e89e8d7306 Merge remote-tracking branch 'origin/topic/jsiwek/972'
Closes #972.

* origin/topic/jsiwek/972:
  Allow default function/hook/event parameters.  Addresses #972.
2013-05-10 19:30:24 -07:00
Jon Siwek
e2a1d4a233 Allow default function/hook/event parameters. Addresses #972.
And changed the endianness parameter of bytestring_to_count() BIF to
default to false (big endian), mostly just to prove that the BIF parser
doesn't choke on default parameters.
2013-05-07 14:32:22 -05:00
Jon Siwek
ec50cad9db Merge branch 'master' into topic/jsiwek/file-analysis
Conflicts:
	scripts/base/protocols/ftp/main.bro
	src/OpaqueVal.h
	testing/btest/Baseline/coverage.bare-load-baseline/canonified_loaded_scripts.log
	testing/btest/Baseline/coverage.default-load-baseline/canonified_loaded_scripts.log
2013-05-06 10:21:16 -05:00
Bernhard Amann
cf6e768ad6 fix opaqueval-related memleak 2013-05-03 23:08:26 -07:00
Bernhard Amann
6acbbe0231 Merge remote-tracking branch 'origin/master' into topic/bernhard/topk 2013-05-03 23:04:22 -07:00
Bernhard Amann
f6e99fce11 fix opaqueval-related memleak. 2013-05-03 23:03:31 -07:00
Bernhard Amann
3e74cdc6e0 Merge remote-tracking branch 'origin/master' into topic/bernhard/hyperloglog 2013-05-03 22:58:02 -07:00
Robin Sommer
69c7363147 Merge remote-tracking branch 'origin/topic/jsiwek/981'
Closes #981.

* origin/topic/jsiwek/981:
  Table lookups return copy of non-const &default vals (addresses #981).
2013-05-03 17:44:37 -07:00
Robin Sommer
75cbce8ea4 Merge remote-tracking branch 'origin/fastpath'
Closes #986

* origin/fastpath:
  Do not allocate one OpaqueType per OpaqueVal.
  Fix memory-leak in OpaqueVal.
2013-05-03 15:55:22 -07:00
Matthias Vallentin
9ac00f8c79 Do not allocate one OpaqueType per OpaqueVal.
Instead, we now allocate type information globally in NetVar.cc.

Addresses #986.
2013-05-03 15:48:06 -07:00
Matthias Vallentin
e78c20c0f8 Fix memory-leak in OpaqueVal.
Addresses #986.
2013-05-03 14:26:02 -07:00
Jon Siwek
6a7a242db9 Table lookups return copy of non-const &default vals (addresses #981).
This prevents unintentional modifications to the &default value itself.
2013-05-03 11:22:15 -05:00
Bernhard Amann
075bfc5b3d synchronize pruned attribute 2013-05-02 12:09:35 -07:00