Commit graph

4693 commits

Author SHA1 Message Date
Robin Sommer
ac66db286a Merge remote-tracking branch 'origin/topic/documentation'
I'm merging in the documentation branch to avoid that it keeps getting
out of sync. We still need to work on this a bit more, I'll summarize
in a mail.

* origin/topic/documentation: (68 commits)
  Going over initial parts of the scripting overview.
  Adding lines in scripting/index.txt. No other change.
  Updating submodule.
  Pass over the Using Bro section.
  Applying doc updates from BIT-1063. (Anthony Verez)
  Breaking lines in using/*.
  More doc reorg, and a light pass over the first 3 sections.
  Starting to put a new structure in place.
  Updating submodule(s).
  Updating submodule(s).
  Adding some temporary testing/demonstration to front page.
  Switching btest to topic/robin/parts branch.
  Basic cross-referencing UIDs between files, btests, and baselines.
  A truly embarassing number of spelling corrections.
  Using redirection with bro-cut.  Include initial btests for this document.
  Corrected a mis-typed RST include.
  Initial start for "Using Bro".
  Spelling corrections.
  Include Notice Policy shortcuts in the Scripting User Manual.
  Notice::policy hooks and tests.
  ...

Conflicts:
	src/3rdparty
2013-08-31 16:07:44 -07:00
Robin Sommer
bf9eb7c44f Updating submodule(s).
[nomail]
2013-08-31 11:22:40 -07:00
Robin Sommer
de5bb65ff7 Removing the "uint8*" methods from SerializationFormat.
They conflict with the "char" version, so that other classes would now
pick the wrong one. Added a bit of casting to HLL to use the "char"
versions instead.
2013-08-31 11:17:49 -07:00
Robin Sommer
a8ac7d8a5b Updating tests for HLL merge. 2013-08-31 11:17:49 -07:00
Robin Sommer
41cab5e0dd Updating tests. 2013-08-31 10:47:19 -07:00
Robin Sommer
6f9d28cc18 Merge branch 'topic/robin/hyperloglog-merge'
* topic/robin/hyperloglog-merge: (35 commits)
  Making the confidence configurable.
  Renaming HyperLogLog->CardinalityCounter.
  Fixing bug introduced during merging.
  add clustered leak test for hll. No issues.
  make gcc happy
  (hopefully) fix refcounting problem in hll/bloom-filter opaque vals. Thanks Robin.
  re-use same hash class for all add operations
  get hll ready for merging
  and forgot a file...
  adapt to new structure
  fix opaqueval-related memleak.
  make it compile on case-sensitive file systems and fix warnings
  make error rate configureable
  add persistence test not using predetermined random seeds.
  update cluster test to also use hll
  persistence really works.
  well, with this commit synchronizing the data structure should work.. ...if we had consistent hashing.
  and also serialize the other things we need
  ok, this bug was hard to find.
  serialization compiles.
  ...
2013-08-31 10:42:42 -07:00
Robin Sommer
295987c8d0 Making the confidence configurable. 2013-08-31 10:34:50 -07:00
Robin Sommer
fb3ceae6d5 Renaming HyperLogLog->CardinalityCounter.
For consistency with the class' name.
2013-08-31 10:22:27 -07:00
Robin Sommer
ef04ce809b Fixing bug introduced during merging. 2013-08-31 10:17:13 -07:00
Robin Sommer
b74bf10565 Going over initial parts of the scripting overview.
I made a light pass over the text. Switched the includes over to the
new btest-include and adapted the other TEXT-EXECs a bit.

Also includes more tweaking all over the Sphinx setup.
2013-08-30 16:23:17 -07:00
Robin Sommer
4dcf8fc0db Merge remote-tracking branch 'origin/topic/bernhard/hyperloglog'
* origin/topic/bernhard/hyperloglog: (32 commits)
  add clustered leak test for hll. No issues.
  make gcc happy
  (hopefully) fix refcounting problem in hll/bloom-filter opaque vals. Thanks Robin.
  re-use same hash class for all add operations
  get hll ready for merging
  and forgot a file...
  adapt to new structure
  fix opaqueval-related memleak.
  make it compile on case-sensitive file systems and fix warnings
  make error rate configureable
  add persistence test not using predetermined random seeds.
  update cluster test to also use hll
  persistence really works.
  well, with this commit synchronizing the data structure should work.. ...if we had consistent hashing.
  and also serialize the other things we need
  ok, this bug was hard to find.
  serialization compiles.
  change plugin after feedback of seth
  Forgot a file. Again. Like always. Basically.
  do away with old file.
  ...
2013-08-30 11:30:05 -07:00
Bernhard Amann
2dd0d057e6 Merge remote-tracking branch 'origin/master' into topic/bernhard/hyperloglog
Conflicts:
	src/NetVar.cc
	src/NetVar.h
2013-08-30 08:43:47 -07:00
Robin Sommer
23144e44a7 Merge remote-tracking branch 'origin/fastpath'
* origin/fastpath:
  Fix global opaque val segfault, addresses BIT-1071
  Fix malloc/delete mismatch.
  Fix invalid pointer dereference in AsciiFormatter.
2013-08-30 08:28:18 -07:00
Jon Siwek
dc2e3d6e04 Fix global opaque val segfault, addresses BIT-1071
The opaque types need to be created before scripts are parsed.
2013-08-29 17:17:40 -05:00
Jon Siwek
742a047a40 Fix malloc/delete mismatch. 2013-08-29 16:22:59 -05:00
Robin Sommer
c8a2fbeadd Merge remote-tracking branch 'origin/fastpath'
Changed two deletes to Unrefs.

* origin/fastpath:
  Fix input framework memory leaks.
  Fix mem leak in socks analyzer for bad addr types.
  Fix bloom filter memory leaks.
2013-08-29 14:20:59 -07:00
Bernhard Amann
4cd20c6375 add clustered leak test for hll. No issues. 2013-08-29 14:01:22 -07:00
Jon Siwek
c4e8908c8e Fix invalid pointer dereference in AsciiFormatter.
Using a temporary object with strtol() makes the end pointer that it
sets invalid after the call.
2013-08-29 15:50:46 -05:00
Jon Siwek
b6e2505202 Fix input framework memory leaks.
Couldn't figure out a test case: gperftools didn't report them (maybe
due to threading?), but valgrind did.
2013-08-29 14:13:16 -05:00
Jon Siwek
e54ea6d7e9 Fix mem leak in socks analyzer for bad addr types. 2013-08-29 14:09:56 -05:00
Jon Siwek
fb8b78840b Fix bloom filter memory leaks. 2013-08-29 11:24:24 -05:00
Robin Sommer
13842f2fd5 Updating submodule(s).
[nomail]
2013-08-28 21:12:15 -07:00
Robin Sommer
39a848d76b Updating submodule(s).
[nomail]
2013-08-28 18:57:23 -07:00
Robin Sommer
6373d817a5 Merge remote-tracking branch 'origin/topic/jsiwek/uid'
Thanks!

* origin/topic/jsiwek/uid:
  Add bits_per_uid unit test, addressing BIT0-1016.
  UID optimizations addressing BIT-1016.

BIT-1016 #merged
2013-08-28 18:57:00 -07:00
Bernhard Amann
dc9fd36497 Merge remote branch 'origin/master' into topic/bernhard/hyperloglog 2013-08-28 17:48:59 -07:00
Robin Sommer
f8c9d5e3d3 Adding lines in scripting/index.txt. No other change. 2013-08-28 14:36:59 -07:00
Jon Siwek
7a2972d0e4 Add bits_per_uid unit test, addressing BIT0-1016. 2013-08-28 16:12:11 -05:00
Jon Siwek
5c119561ad UID optimizations addressing BIT-1016.
Max UID bit-length is now 128, but can be increased w/ trivial source
code change of BRO_UID_LEN.
2013-08-28 15:35:18 -05:00
Seth Hall
0592b528ef Merge branch 'master' of ssh://git.bro-ids.org/bro 2013-08-28 00:58:03 -04:00
Seth Hall
2b78922af8 Added a $unique_max field to Reducers for the SumStats::UNIQUE calculation
- Using the new option in scan.bro and the FTP bruteforce detection.
2013-08-28 00:57:44 -04:00
Robin Sommer
ea9cfc749d Updating submodule(s).
[nomail]
2013-08-27 14:12:20 -07:00
Robin Sommer
0cdc868446 Updating submodule(s).
[nomail]
2013-08-27 14:10:38 -07:00
Robin Sommer
f46f692185 Merge remote-tracking branch 'origin/topic/bernhard/hexstr'
BIT-1069 #merged

* origin/topic/bernhard/hexstr:
  add hexstr_to_bytestring bif that does exactly the opposite of bytestring_to_hexstr.
2013-08-27 13:30:25 -07:00
Robin Sommer
f6b689db81 Merge remote-tracking branch 'origin/topic/jsiwek/uid'
* origin/topic/jsiwek/uid:
  Fix UID compiler warning/error & missed baselines.
  Increase UIDs to 96 bits w/ C/F prefix - BIT-1016
2013-08-27 12:36:12 -07:00
Bernhard Amann
8a5a2b5b39 add hexstr_to_bytestring bif that does exactly the opposite of
bytestring_to_hexstr.
2013-08-27 12:20:03 -07:00
Jon Siwek
50dbbab52c Fix UID compiler warning/error & missed baselines. 2013-08-26 16:11:20 -05:00
Jon Siwek
d31b59a5d0 Merge branch 'master' into topic/jsiwek/uid 2013-08-26 15:51:18 -05:00
Jon Siwek
22bf3e1196 Increase UIDs to 96 bits w/ C/F prefix - BIT-1016
- The bit-length is adjustable via redef'ing bits_per_uid.

- Prefix 'C' is used for connection UIDS (including IP tunnels) and
  'F' for files.
2013-08-26 15:36:31 -05:00
Bernhard Amann
74f96d22ef Merge remote branch 'origin/master' into topic/bernhard/hyperloglog
Conflicts:
	src/3rdparty
2013-08-26 12:53:13 -07:00
Seth Hall
ea89e30a9c One more tiny SumStats change to clean up cluster state. 2013-08-26 15:29:27 -04:00
Seth Hall
abd7cdff43 Merge branch 'master' of ssh://git.bro-ids.org/bro 2013-08-26 13:57:47 -04:00
Seth Hall
72d2cd363e Removed the when statement from general use in the sumstats cluster code. 2013-08-26 13:57:42 -04:00
Robin Sommer
df84083227 Merge remote-tracking branch 'origin/fastpath'
* origin/fastpath:
  Remove code relict pointed out by Bernhard.
  Fix wrong documentation for mkdir BIF.
  File extraction tweaks.
2013-08-24 10:29:07 -07:00
Robin Sommer
391bfdc317 Merge remote-tracking branch 'origin/topic/hui/dnp3-merge-v2'
* origin/topic/hui/dnp3-merge-v2:
  fixed number of object bug in dnp3-protocol pac; update two base test trases
2013-08-23 16:34:26 -07:00
Hui Lin
3e3ca1bb74 fixed number of object bug in dnp3-protocol pac; update two base test trases 2013-08-23 18:10:30 -05:00
Jon Siwek
6dbbce8e05 Remove code relict pointed out by Bernhard.
The condition should never be true, it's leftover from my
hacking/debugging code.
2013-08-23 12:03:13 -05:00
Jon Siwek
288ef20a4e Fix wrong documentation for mkdir BIF. 2013-08-23 11:57:37 -05:00
Jon Siwek
17d0ecd388 File extraction tweaks.
- Default extraction limit of 100MB now provided via a tuning script
  loaded in local.bro so that command-line Bro is unlimited by default.

- Extraction directory is now created on request of file extraction
  rather than unconditionally in bro_init().
2013-08-23 11:57:07 -05:00
Robin Sommer
927f534833 Merge remote-tracking branch 'origin/topic/jsiwek/extract-limit'
* origin/topic/jsiwek/extract-limit:
  Use macros to create file analyzer plugin classes.
  Add options to limit extracted file sizes w/ 100MB default.
2013-08-22 16:53:06 -07:00
Robin Sommer
1cf535dbbb Updating submodule. 2013-08-22 16:33:32 -07:00