Commit graph

13262 commits

Author SHA1 Message Date
Robin Sommer
40201a180e Fixing for unserializion error.
Because BloomFilter is a base class, with other classes derived from
it, it needs special treatment.
2013-07-09 21:00:53 -07:00
Seth Hall
8322bbfd62 Small test fixes. 2013-07-09 23:28:09 -04:00
Seth Hall
60da0f4764 Added a missing curly brace in smtp/dpd.sig 2013-07-09 22:57:36 -04:00
Seth Hall
4dda9cd3ba Fix a bug where the same analyzer tag was reused for two different analyzers. 2013-07-09 22:45:21 -04:00
Seth Hall
39444b5af7 Moved DPD signatures into script specific directories.
- This caused us to lose signatures for POP3 and Bittorrent.  These will
   need discovered in the repository again when we add scripts
   for those analyzers.
2013-07-09 22:44:55 -04:00
Bernhard Amann
03b584c34a Merge remote-tracking branch 'origin/master' into topic/bernhard/topk 2013-07-09 14:56:05 -07:00
Jon Siwek
73155c321b Add an is_orig parameter to file_over_new_connection event. 2013-07-09 15:58:28 -05:00
Jon Siwek
efe878f3de Make magic for emitting application/msword mime type less strict. 2013-07-09 15:56:47 -05:00
Jon Siwek
da4a0bed03 Disable more libmagic builtin checks that override the magic database. 2013-07-09 15:55:33 -05:00
Jon Siwek
6a5b825058 Delay file_over_new_connection events until after file_new occurs. 2013-07-09 14:25:41 -05:00
Seth Hall
5dbc354898 extract_filename_from_content_disposition is still hacky but more closely aligns with RFC5987 2013-07-09 14:05:36 -04:00
Seth Hall
ecfac31de0 Fixed SMTP URL extraction for the Intel framework with Files updates. 2013-07-09 11:51:23 -04:00
Seth Hall
cdf6b7864e More file analysis updates.
- Recorrected the module name to Files.

  - Added Files::analyzer_name to get a more readable name for a
    file analyzer.

  - Improved and just overall better handled multipart mime
    transfers in HTTP and SMTP.  HTTP now has orig_fuids and resp_fuids
    log fields since multiple "files" can be transferred with
    multipart mime in a single request/response pair.  SMTP has
    an fuids field which has file unique IDs for all parts
    transferred. FTP and IRC have a log field named fuid added
    because only a single file can be transferred per irc and ftp
    log line.
2013-07-09 11:50:54 -04:00
Robin Sommer
841604bebe Updating submodule(s).
[nomail]
2013-07-08 20:46:52 -07:00
Robin Sommer
7fe7684d4a Updating submodule(s).
[nomail]
2013-07-08 13:28:07 -07:00
Robin Sommer
2ea1f483db Bringing back test for enable_auto_protocol_capture_filters (formerly
all_packets).
2013-07-08 13:06:03 -07:00
Robin Sommer
b62927e9de Merge remote-tracking branch 'origin/topic/seth/packet-filter-updates'
Closes #1030.

* origin/topic/seth/packet-filter-updates:
  Missed a test fix.
  Updating test baselines.
  Updates for the PacketFilter framework to simplify it.
  Last test update for PacketFilter framework.
  Several final fixes for PacketFilter framework.
  Packet filter framework checkpoint.
  Checkpoint on the packet filter framework.
  Initial rework of packet filter framework.
2013-07-07 21:09:28 -07:00
Seth Hall
58d133e764 Merge remote-tracking branch 'origin/master' into topic/seth/faf-updates
Conflicts:
	scripts/base/frameworks/files/main.bro
	scripts/base/init-bare.bro
	scripts/base/protocols/ftp/file-analysis.bro
	scripts/base/protocols/http/file-analysis.bro
	scripts/base/protocols/irc/file-analysis.bro
	scripts/base/protocols/smtp/file-analysis.bro
	src/const.bif
	src/event.bif
	src/file_analysis/Analyzer.h
	src/file_analysis/file_analysis.bif
2013-07-05 02:13:27 -04:00
Seth Hall
2b48396d23 Check file_over_new_connetion to fire for each connection (including the first). 2013-07-05 02:00:35 -04:00
Seth Hall
df2841458d Large overhaul in name and appearance for file analysis. 2013-07-05 02:00:14 -04:00
Seth Hall
1e5906af08 Missed a test fix. 2013-07-05 01:52:37 -04:00
Seth Hall
af87126521 Updating test baselines. 2013-07-05 01:27:59 -04:00
Seth Hall
4149724f59 Updates for the PacketFilter framework to simplify it. 2013-07-05 01:12:22 -04:00
Seth Hall
5f8ee93ef0 Merge remote-tracking branch 'origin/master' into topic/seth/analyzer-framework
Conflicts:
	scripts/base/init-default.bro
	scripts/base/protocols/dns/main.bro
	scripts/base/protocols/ftp/main.bro
	scripts/base/protocols/http/main.bro
	scripts/base/protocols/irc/main.bro
	scripts/base/protocols/smtp/main.bro
	scripts/base/protocols/ssh/main.bro
	scripts/base/protocols/ssl/main.bro
	scripts/base/protocols/syslog/main.bro
	src/main.cc
	testing/btest/Baseline/coverage.default-load-baseline/canonified_loaded_scripts.log
2013-07-04 23:07:52 -04:00
Seth Hall
fefef47f30 Merge branch 'topic/seth/ssh-login-monitoring-fix'
* topic/seth/ssh-login-monitoring-fix:
  Add a call to lookup_connection in SSH scripts to update connval.
2013-07-04 22:47:56 -04:00
Seth Hall
ca6d2bb6bc Add a call to lookup_connection in SSH scripts to update connval. 2013-07-04 22:32:07 -04:00
Robin Sommer
23b58d62d2 Updating submodule(s).
[nomail]
2013-07-03 17:24:11 -07:00
Robin Sommer
fa8777cbd2 Merge remote-tracking branch 'origin/topic/seth/ssl-remove-log-queue'
Closes #1027.

* origin/topic/seth/ssl-remove-log-queue:
  Remove the log queueing mechanism that was included with the SSL log delay mechanism.
2013-07-03 17:01:20 -07:00
Robin Sommer
96fe05633a Merge remote-tracking branch 'origin/topic/bernhard/input-update'
Closes #1021.

* origin/topic/bernhard/input-update:
  this event handler fails the unused-event-handlers test because it is a bit of a special case.
  ...and fix the event ordering issue. Dispatch != QueueEvent
  add Terminate to input framework to prevent potential shutdown race-conditions.
  fix warning.
  fix stderr test. ls behaves differently on errors on linux...
  small fixes.
  linux does not have strnstr
  and close only fds that are currently open (the logging framework really did not like that :) )
  A bunch of more changes for the raw reader
  make reading from stdout and stderr simultaneously work.
  allow sending data to stdin of child process
  Streaming reads from external commands work without blocking anything.
  replace popen with fork and exec.
  change raw reader to use basic c io instead of fdstream encapsulation class.
2013-07-03 16:52:28 -07:00
Robin Sommer
ba4f03bc98 Merge remote-tracking branch 'origin/topic/seth/tls-1.2-fix'
Closes #1020.

* origin/topic/seth/tls-1.2-fix:
  Single character fix to correct support for TLS 1.2 (my bad).
2013-07-03 16:34:21 -07:00
Robin Sommer
a329c3e7c3 Merge remote-tracking branch 'origin/topic/jsiwek/plugin-docs'
Closes #1019.

* origin/topic/jsiwek/plugin-docs:
  Teach broxygen to generate protocol analyzer plugin reference.
  const adjustments
2013-07-03 16:32:00 -07:00
Robin Sommer
ed45a6ea60 Merge remote-tracking branch 'origin/topic/jsiwek/1013'
Closes #1013.

* origin/topic/jsiwek/1013:
  Fix redef of table index from clearing table.  Addresses #1013.
2013-07-03 16:28:33 -07:00
Robin Sommer
d8b05af7e5 Merge remote-tracking branch 'origin/topic/jsiwek/faf-cleanup'
Closes #1002.

* origin/topic/jsiwek/faf-cleanup:
  Move file analyzers to new plugin infrastructure.
  Add a general file analysis overview/how-to document.
  Improve file analysis doxygen comments.
  Improve tracking of HTTP file extraction (addresses #988).
  Fix HTTP multipart body file analysis.
  Remove logging of analyzers field of FileAnalysis::Info.
  Remove extraction counter in default file extraction scripts.
  Remove FileAnalysis::postpone_timeout.
  Make default get_file_handle handlers &priority=5.
  Add input interface to forward data for file analysis.
  File analysis framework interface simplifications.
2013-07-03 16:27:16 -07:00
Bernhard Amann
fef3180942 bump sqlite to 3.7.17. 2013-07-02 18:54:46 -07:00
Seth Hall
030564a710 Single character fix to correct support for TLS 1.2 (my bad).
- Thanks for help from Rafal Lesniak in nailing down the location
   of the bug and supplying test traffic.

 - Test traffic with a TLS 1.2 connection.

 - Addresses ticket #1020
2013-07-02 14:49:36 -04:00
Seth Hall
7c50efde80 Remove the log queueing mechanism that was included with the SSL log delay mechanism.
- One obvious downside is that queued logs at termination may not
    get logged because the trigger for the when statement never matches.
2013-06-28 11:40:02 -04:00
Matthias Vallentin
5f70452a9a Small fixes and style tweaks. 2013-06-18 10:40:00 -07:00
Matthias Vallentin
273629de36 Only serialize Bloom filter type if available. 2013-06-18 10:23:07 -07:00
Matthias Vallentin
e6e5f4926f Create hash policies through factory. 2013-06-17 16:28:16 -07:00
Matthias Vallentin
85668e7054 Remove lingering debug code. 2013-06-17 16:16:44 -07:00
Matthias Vallentin
532fbfb4d2 Factor implementation and change interface.
When constructing a Bloom filter, one now has to pass a HashPolicy instance to
it. This separates more clearly the concerns of hashing and Bloom filter
management.

This commit also changes the interface to initialize Bloom filters: there exist
now two initialization functions, one for each type:

  (1) bloomfilter_basic_init(fp: double,
                             capacity: count,
                             name: string &default=""): opaque of bloomfilter

  (2) bloomfilter_counting_init(k: count,
                                cells: count,
                                max: count,
                                name: string &default=""): opaque of bloomfilter

The BiFs for adding elements and performing lookups remain the same. This
essentially gives us "BiF polymorphism" at script land, where the
initialization BiF constructs the most derived type while subsequent BiFs
adhere to the same interface.

The reason why we split up the constructor in this case is that we have not yet
derived the math that computes the optimal number of hash functions for
counting Bloom filters---users have to explicitly parameterize them for now.
2013-06-17 16:14:11 -07:00
Matthias Vallentin
9f74064289 Expose Bro's linear congruence PRNG as utility function.
It was previously not possible to crank the wheel on the PRNG in a
deterministic way without affecting the globally unique seed. The new extra
utility function bro_prng takes a state in the form of a long int and returns
the new PRNG state, now allowing arbitrary code parts to use the random number
functionality.

This commit also fixes a problem in the H3 constructor, which requires use
of multiple seeds. The single seed passed in now serves as seed to crank out as
many value needed using bro_prng.
2013-06-17 14:02:14 -07:00
Matthias Vallentin
79a6a26f9f H3 does not check for zero length input. 2013-06-14 10:20:33 -07:00
Matthias Vallentin
1576239f67 Support seeding for hashers. 2013-06-14 10:19:39 -07:00
Matthias Vallentin
d2d8aff814 Add utility function to access first random seed. 2013-06-14 09:22:48 -07:00
Matthias Vallentin
a6d7b7856e Update H3 documentation (and minor style nits.) 2013-06-13 23:21:54 -07:00
Matthias Vallentin
529d120376 Make H3 seed configurable. 2013-06-13 23:07:31 -07:00
Matthias Vallentin
1f90b539a8 Make H3 class adhere to Bro coding style. 2013-06-13 23:06:01 -07:00
Jon Siwek
ae5a75bad9 Fix redef of table index from clearing table. Addresses #1013.
`redef foo["x"] = 1` now acts like `redef foo += { ["x"] = 1 }`
instead of `redef foo = { ["x"] = 1 }`.
2013-06-12 15:18:58 -05:00
Matthias Vallentin
14a701a237 Implement value merging.
The actual BloomFilter merging still lacks, this is just the first step in the
right direction from the user interface side.
2013-06-10 22:46:24 -07:00