Commit graph

2220 commits

Author SHA1 Message Date
Jon Siwek
5c119561ad UID optimizations addressing BIT-1016.
Max UID bit-length is now 128, but can be increased w/ trivial source
code change of BRO_UID_LEN.
2013-08-28 15:35:18 -05:00
Seth Hall
0592b528ef Merge branch 'master' of ssh://git.bro-ids.org/bro 2013-08-28 00:58:03 -04:00
Seth Hall
2b78922af8 Added a $unique_max field to Reducers for the SumStats::UNIQUE calculation
- Using the new option in scan.bro and the FTP bruteforce detection.
2013-08-28 00:57:44 -04:00
Jon Siwek
d31b59a5d0 Merge branch 'master' into topic/jsiwek/uid 2013-08-26 15:51:18 -05:00
Jon Siwek
22bf3e1196 Increase UIDs to 96 bits w/ C/F prefix - BIT-1016
- The bit-length is adjustable via redef'ing bits_per_uid.

- Prefix 'C' is used for connection UIDS (including IP tunnels) and
  'F' for files.
2013-08-26 15:36:31 -05:00
Bernhard Amann
74f96d22ef Merge remote branch 'origin/master' into topic/bernhard/hyperloglog
Conflicts:
	src/3rdparty
2013-08-26 12:53:13 -07:00
Seth Hall
ea89e30a9c One more tiny SumStats change to clean up cluster state. 2013-08-26 15:29:27 -04:00
Seth Hall
abd7cdff43 Merge branch 'master' of ssh://git.bro-ids.org/bro 2013-08-26 13:57:47 -04:00
Seth Hall
72d2cd363e Removed the when statement from general use in the sumstats cluster code. 2013-08-26 13:57:42 -04:00
Jon Siwek
17d0ecd388 File extraction tweaks.
- Default extraction limit of 100MB now provided via a tuning script
  loaded in local.bro so that command-line Bro is unlimited by default.

- Extraction directory is now created on request of file extraction
  rather than unconditionally in bro_init().
2013-08-23 11:57:07 -05:00
Jon Siwek
89ae4ffd05 Add options to limit extracted file sizes w/ 100MB default. 2013-08-22 16:37:58 -05:00
Seth Hall
b3a5c5f412 Tiny fix to correct a reporter message. 2013-08-21 19:11:34 -04:00
Seth Hall
2091bbbf6f Fixing a problem with resetting sumstats on managers in the wrong location. 2013-08-21 14:47:05 -04:00
Seth Hall
22b1e78ec1 Changing how threshold reseting happens (actually do it on managers!) 2013-08-21 14:08:14 -04:00
Seth Hall
1430c28ba3 Merge remote-tracking branch 'origin/master' into topic/seth/sumstats-updates 2013-08-21 12:45:50 -04:00
Vlad Grigorescu
73a5106bd9 Fix some reporter.log errors by adding a few extra key membership checks. 2013-08-21 07:29:22 -04:00
Robin Sommer
a646fde884 Merge remote-tracking branch 'origin/topic/seth/unified2-analyzer' into topic/robin/unified2-analyzer-merge
* origin/topic/seth/unified2-analyzer:
  Fixed a problem where the Unified2 analyzer was attached to every file.
  Fixing intel framework tests.
  Updating submodule(s).
  Add file name support to intel framework.
  Add file support to intel framework and slightly restructure intel http handling.

Conflicts:
	CHANGES
	VERSION
	scripts/base/files/unified2/main.bro
	testing/btest/Baseline/coverage.bare-load-baseline/canonified_loaded_scripts.log
	testing/btest/Baseline/coverage.default-load-baseline/canonified_loaded_scripts.log
2013-08-14 10:27:26 -07:00
Seth Hall
f8f465e259 Fixed a problem where the Unified2 analyzer was attached to every file. 2013-08-14 01:01:03 -04:00
Seth Hall
efca3c0840 Merge remote-tracking branch 'origin/master' into topic/seth/unified2-analyzer
Conflicts:
	testing/btest/Baseline/coverage.bare-load-baseline/canonified_loaded_scripts.log
	testing/btest/Baseline/coverage.default-load-baseline/canonified_loaded_scripts.log
2013-08-14 00:54:41 -04:00
Seth Hall
a98c78c0d1 Fixing intel framework tests. 2013-08-13 23:49:39 -04:00
Robin Sommer
83eae53f54 Merge remote-tracking branch 'origin/topic/seth/unified2-analyzer'
BIT-1054 #merged

* origin/topic/seth/unified2-analyzer:
  Fixes in case a packet isn't seen that matches an event.
  Finished work on unified2 analyzer.
  Fixed some tests.
  Working unified2 analyzer.
  Unified2 file analyzer updated to new plugin style.
  Adding the unified2 analyzer.

Conflicts:
	testing/btest/Baseline/coverage.bare-load-baseline/canonified_loaded_scripts.log
	testing/btest/Baseline/coverage.default-load-baseline/canonified_loaded_scripts.log
2013-08-13 18:37:52 -07:00
Seth Hall
ed14bdc77e Add file name support to intel framework. 2013-08-13 13:21:31 -04:00
Seth Hall
0bde911bd4 Add file support to intel framework and slightly restructure intel http handling. 2013-08-13 13:21:08 -04:00
Seth Hall
e0de1a2d00 Fixes in case a packet isn't seen that matches an event. 2013-08-13 08:55:11 -04:00
Seth Hall
f7c6dd7f7e Finished work on unified2 analyzer. 2013-08-13 03:21:43 -04:00
Seth Hall
95161a920c Fixed some tests. 2013-08-12 15:31:31 -04:00
Seth Hall
091c8f3ebc Working unified2 analyzer.
- No output by default yet.  Most of the activity is centered
   around generating the Unified2::alert event which ties together
   an IDSEvent and a packet.
2013-08-12 14:57:12 -04:00
Robin Sommer
45f1b89f60 Merge branch 'topic/robin/dnp3-merge-v3'
Includes a bit more docs/comments cleanup. We should eventually
document the events further but it should suffice for now.

* topic/robin/dnp3-merge-v3:
  Tiny bit of cleanup and adapting the new test.
  added a test case for dnp3 packets with only link layer
  added condition to check DNP3 packet without app layer data
  Fixing well-known port.
  Pluginizing the DNP3 analyzer, plus a basic script logging requests and replies.
2013-08-12 11:39:03 -07:00
Bernhard Amann
2a684cd486 Merge remote-tracking branch 'origin/master' into topic/bernhard/hyperloglog 2013-08-12 09:48:03 -07:00
Bernhard Amann
d83edf8068 Merge remote-tracking branch 'origin/master' into topic/bernhard/hyperloglog
Conflicts:
	src/NetVar.cc
	src/NetVar.h
	src/SerialTypes.h
	src/probabilistic/CMakeLists.txt
	testing/btest/scripts/base/frameworks/sumstats/basic-cluster.bro
	testing/btest/scripts/base/frameworks/sumstats/basic.bro
2013-08-12 09:47:53 -07:00
Robin Sommer
36c2433075 Fixing well-known port.
This fixes the remaining test.
2013-08-11 15:59:32 -07:00
Seth Hall
04de4ce24b Unified2 file analyzer updated to new plugin style. 2013-08-10 22:26:32 -04:00
Robin Sommer
0e7f51f78c Merge branch 'master' into topic/robin/dnp3-merge-v3
Conflicts:
	scripts/base/init-default.bro
2013-08-09 17:11:51 -07:00
Seth Hall
e52b174594 Fix the SSL infinite loop I just created. 2013-08-05 17:29:39 -04:00
Seth Hall
595e2f3c8a Change to SSL log delay to cause the log to write even if delay times out. 2013-08-05 16:45:05 -04:00
Robin Sommer
e7aefcdf36 Merge branch 'master' into topic/robin/dhcp-merge 2013-08-03 19:14:01 -07:00
Robin Sommer
308db797c3 Merge remote-tracking branch 'origin/topic/vladg/dhcp'
* origin/topic/vladg/dhcp:
  DHCP: Adding unit tests.
  DHCP:
  Rework the DHCP analyzer to make it compatible again.
2013-08-03 19:14:00 -07:00
Seth Hall
1eadeaec3c Fix a major memory issue in the SumStats framework.
- There are still problems, but this should prevent a deadlock issue
   and help with memory use.
2013-08-03 01:57:51 -04:00
Seth Hall
56de65461e Merge remote-tracking branch 'origin/topic/seth/sumstats-updates'
* origin/topic/seth/sumstats-updates:
  Still fixing bugs in sumstats updated api cluster support.
  Hopefully fix the SumStats cluster support.
  Fix the SumStats top-k plugin and test.
  Updates for SumStats API to deal with high memory stats.
  Beginning rework of SumStats API.
  Tiny fix to account for missing str field  (not sure how this happens yet)
  Add server samples to SSH bruteforce detection.
  Fix a reporter message in sumstats.
  SumStats changes to how thresholding works to simplify and reduce memory use.
  More adjustments to try and correct SumStats memory use.
  Hopefully fixing a strange error.
  Large update for the SumStats framework.
2013-08-02 22:37:39 -04:00
Jon Siwek
1531980f3d Exec module and raw input reader fixes.
- Do stream mode for commands done by exec module, it seems important
  in some cases (e.g. ensure requested stdin is fully written).

- For cases where the raw input reader knows the child process has been
  reaped, set the childpid member to a sentinel value to indicate such
  so we don't later think we should kill it or wait on it anymore.

- More error checking on dup2/close calls.  Set sentinel values when
  closing ends of pipes to prevent double closing a fd.

- Signal flag not set when raw input reader's child exits as a result
  of a signal.  Left out a test for this -- might be portability issues
  (e.g. Ubuntu seems to do things different regarding the exit code and
  also is printing "Killed" to stderr where other platforms don't).
2013-08-02 17:27:10 -05:00
Seth Hall
6b58ef12d7 Still fixing bugs in sumstats updated api cluster support. 2013-08-02 16:53:56 -04:00
Seth Hall
135094428e Hopefully fix the SumStats cluster support. 2013-08-02 16:30:34 -04:00
Seth Hall
7b8073556e Fix the SumStats top-k plugin and test. 2013-08-02 16:30:20 -04:00
Seth Hall
d6edbd27b1 Merge remote-tracking branch 'origin/master' into topic/seth/sumstats-updates 2013-08-02 13:17:48 -04:00
Seth Hall
4f8100774c Updates for SumStats API to deal with high memory stats.
- The code is a mess and will need to be cleaned up, but the
    tests do pass.
2013-08-02 12:44:33 -04:00
Robin Sommer
306d4fa6f9 Pluginizing the DNP3 analyzer, plus a basic script logging requests
and replies.

Almost ready, but now 1 test fails again ...

[Note I broke git history by copying over the files into a new branch.]
2013-08-01 17:36:11 -07:00
Robin Sommer
04ccb12183 Merge branch 'topic/robin/topk-merge'
BIT-1048 #merged

I'm reverting the serializer version update for now as that breaks
Broccoli. Let's do that later for 2.2.

* topic/robin/topk-merge:
  update documentation, rename get* to Get* and make hasher persistent
  adapt to new folder structure
  fix opaqueval-related memleak
  synchronize pruned attribute
  potentially found wrong Ref.
  add sum function that can be used to get the number of total observed elements.
  in cluster settings, the resultvals can apparently been uninitialized in some special cases
  fix memory leaks
  fix warnings
  add topk cluster test
  make size of topk-list configureable when using sumstats
  implement merging for top-k.
  add serialization for topk
  make the get function const
  topk for sumstats
  well, a test that works..
  implement topk.
2013-08-01 14:39:16 -07:00
Robin Sommer
6b6b409c8c Merge remote-tracking branch 'origin/topic/jsiwek/faf-updates'
Nice solution with the ComponentManager/TaggedComponent!

BIT-1049 #Merged Merged into master.

* origin/topic/jsiwek/faf-updates:
  Fix some build errors.
  Minor fix to file/protocol analyzer plugin reference doc.
  Internal refactoring of how plugin components are tagged/managed.
  Factor out the need for a tag field in Files::AnalyzerArgs record.
  Add a distinct tag class for file analyzers.
  Fix various documentation, mostly related to file analysis.
2013-08-01 11:14:11 -07:00
Robin Sommer
948441e176 Test expected false positive, but it isn't one any more.
Matthias, please check if this is correct.
2013-08-01 10:52:15 -07:00
Robin Sommer
32a403cdaf Merge branch 'topic/robin/bloom-filter-merge'
* topic/robin/bloom-filter-merge:
  Using a real hash function for hashing a BitVector's internal state.
  Support UHF hashing for >= UHASH_KEY_SIZE bytes.
  Changing the Bloom filter hashing so that it's independent of CompositeHash.
  Add new BiF for low-level Bloom filter initialization.
  Introduce global_hash_seed script variable.

Conflicts:
	testing/btest/Baseline/bifs.bloomfilter/output
2013-08-01 10:52:08 -07:00