Mirror/zeek
1
0
Fork 0
mirror of https://github.com/zeek/zeek.git synced 2025-10-03 23:28:20 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 3
4247 commits 384 branches 195 tags 258 MiB
Commit graph

876 commits

Author SHA1 Message Date
Seth Hall
0bfdcc1fbc Added protocol description functions that provide a super compressed log representation. 2013-07-16 12:01:50 -04:00
Seth Hall
4dd4c5344e Fix a bug where orig file information in http wasn't working right. 2013-07-12 16:12:26 -04:00
Seth Hall
b14f5a853e Added mime types to http.log 2013-07-12 16:06:40 -04:00
Robin Sommer
06287966a1 Bringing the DPD POP3 signature back. 
This also avoids the need for updating the external test suite.
 2013-07-10 14:19:00 -07:00
Seth Hall
22b4f8dd90 Fix a small issue with finding smtp entities. 2013-07-10 16:51:22 -04:00
Seth Hall
788a31edcd Added support for files to the notice framework. 2013-07-10 16:29:07 -04:00
Seth Hall
2e0912b543 Merge remote-tracking branch 'origin/topic/seth/bittorrent-fix-and-dpd-sig-breakout' into topic/seth/faf-updates 
Conflicts:
	magic
	scripts/base/protocols/http/__load__.bro
	scripts/base/protocols/irc/__load__.bro
	scripts/base/protocols/smtp/__load__.bro
 2013-07-10 16:28:38 -04:00
Seth Hall
60da0f4764 Added a missing curly brace in smtp/dpd.sig 2013-07-09 22:57:36 -04:00
Seth Hall
39444b5af7 Moved DPD signatures into script specific directories. 
- This caused us to lose signatures for POP3 and Bittorrent.  These will
   need discovered in the repository again when we add scripts
   for those analyzers.
 2013-07-09 22:44:55 -04:00
Bernhard Amann
03b584c34a Merge remote-tracking branch 'origin/master' into topic/bernhard/topk 2013-07-09 14:56:05 -07:00
Jon Siwek
73155c321b Add an is_orig parameter to file_over_new_connection event. 2013-07-09 15:58:28 -05:00
Seth Hall
5dbc354898 extract_filename_from_content_disposition is still hacky but more closely aligns with RFC5987 2013-07-09 14:05:36 -04:00
Seth Hall
cdf6b7864e More file analysis updates. 
- Recorrected the module name to Files.

  - Added Files::analyzer_name to get a more readable name for a
    file analyzer.

  - Improved and just overall better handled multipart mime
    transfers in HTTP and SMTP.  HTTP now has orig_fuids and resp_fuids
    log fields since multiple "files" can be transferred with
    multipart mime in a single request/response pair.  SMTP has
    an fuids field which has file unique IDs for all parts
    transferred. FTP and IRC have a log field named fuid added
    because only a single file can be transferred per irc and ftp
    log line.
 2013-07-09 11:50:54 -04:00
Robin Sommer
b62927e9de Merge remote-tracking branch 'origin/topic/seth/packet-filter-updates' 
Closes #1030.

* origin/topic/seth/packet-filter-updates:
  Missed a test fix.
  Updating test baselines.
  Updates for the PacketFilter framework to simplify it.
  Last test update for PacketFilter framework.
  Several final fixes for PacketFilter framework.
  Packet filter framework checkpoint.
  Checkpoint on the packet filter framework.
  Initial rework of packet filter framework.
 2013-07-07 21:09:28 -07:00
Seth Hall
58d133e764 Merge remote-tracking branch 'origin/master' into topic/seth/faf-updates 
Conflicts:
	scripts/base/frameworks/files/main.bro
	scripts/base/init-bare.bro
	scripts/base/protocols/ftp/file-analysis.bro
	scripts/base/protocols/http/file-analysis.bro
	scripts/base/protocols/irc/file-analysis.bro
	scripts/base/protocols/smtp/file-analysis.bro
	src/const.bif
	src/event.bif
	src/file_analysis/Analyzer.h
	src/file_analysis/file_analysis.bif
 2013-07-05 02:13:27 -04:00
Seth Hall
df2841458d Large overhaul in name and appearance for file analysis. 2013-07-05 02:00:14 -04:00
Seth Hall
4149724f59 Updates for the PacketFilter framework to simplify it. 2013-07-05 01:12:22 -04:00
Seth Hall
5f8ee93ef0 Merge remote-tracking branch 'origin/master' into topic/seth/analyzer-framework 
Conflicts:
	scripts/base/init-default.bro
	scripts/base/protocols/dns/main.bro
	scripts/base/protocols/ftp/main.bro
	scripts/base/protocols/http/main.bro
	scripts/base/protocols/irc/main.bro
	scripts/base/protocols/smtp/main.bro
	scripts/base/protocols/ssh/main.bro
	scripts/base/protocols/ssl/main.bro
	scripts/base/protocols/syslog/main.bro
	src/main.cc
	testing/btest/Baseline/coverage.default-load-baseline/canonified_loaded_scripts.log
 2013-07-04 23:07:52 -04:00
Seth Hall
ca6d2bb6bc Add a call to lookup_connection in SSH scripts to update connval. 2013-07-04 22:32:07 -04:00
Robin Sommer
fa8777cbd2 Merge remote-tracking branch 'origin/topic/seth/ssl-remove-log-queue' 
Closes #1027.

* origin/topic/seth/ssl-remove-log-queue:
  Remove the log queueing mechanism that was included with the SSL log delay mechanism.
 2013-07-03 17:01:20 -07:00
Robin Sommer
96fe05633a Merge remote-tracking branch 'origin/topic/bernhard/input-update' 
Closes #1021.

* origin/topic/bernhard/input-update:
  this event handler fails the unused-event-handlers test because it is a bit of a special case.
  ...and fix the event ordering issue. Dispatch != QueueEvent
  add Terminate to input framework to prevent potential shutdown race-conditions.
  fix warning.
  fix stderr test. ls behaves differently on errors on linux...
  small fixes.
  linux does not have strnstr
  and close only fds that are currently open (the logging framework really did not like that :) )
  A bunch of more changes for the raw reader
  make reading from stdout and stderr simultaneously work.
  allow sending data to stdin of child process
  Streaming reads from external commands work without blocking anything.
  replace popen with fork and exec.
  change raw reader to use basic c io instead of fdstream encapsulation class.
 2013-07-03 16:52:28 -07:00
Robin Sommer
a329c3e7c3 Merge remote-tracking branch 'origin/topic/jsiwek/plugin-docs' 
Closes #1019.

* origin/topic/jsiwek/plugin-docs:
  Teach broxygen to generate protocol analyzer plugin reference.
  const adjustments
 2013-07-03 16:32:00 -07:00
Robin Sommer
d8b05af7e5 Merge remote-tracking branch 'origin/topic/jsiwek/faf-cleanup' 
Closes #1002.

* origin/topic/jsiwek/faf-cleanup:
  Move file analyzers to new plugin infrastructure.
  Add a general file analysis overview/how-to document.
  Improve file analysis doxygen comments.
  Improve tracking of HTTP file extraction (addresses #988).
  Fix HTTP multipart body file analysis.
  Remove logging of analyzers field of FileAnalysis::Info.
  Remove extraction counter in default file extraction scripts.
  Remove FileAnalysis::postpone_timeout.
  Make default get_file_handle handlers &priority=5.
  Add input interface to forward data for file analysis.
  File analysis framework interface simplifications.
 2013-07-03 16:27:16 -07:00
Seth Hall
7c50efde80 Remove the log queueing mechanism that was included with the SSL log delay mechanism. 
- One obvious downside is that queued logs at termination may not
    get logged because the trigger for the when statement never matches.
 2013-06-28 11:40:02 -04:00
Jon Siwek
7c7b6214a6 Move file analyzers to new plugin infrastructure. 2013-06-10 15:50:18 -05:00
Bernhard Amann
b39bffd9aa Merge remote-tracking branch 'origin/master' into topic/bernhard/input-update 2013-06-08 05:43:21 -07:00
Jon Siwek
f2574636b6 Merge branch 'master' into topic/jsiwek/faf-cleanup 
Conflicts:
	scripts/base/protocols/ftp/file-analysis.bro
	scripts/base/protocols/http/file-analysis.bro
	scripts/base/protocols/irc/file-analysis.bro
	scripts/base/protocols/smtp/file-analysis.bro
	src/file_analysis/File.cc
	src/file_analysis/File.h
	src/file_analysis/Manager.cc
	src/file_analysis/Manager.h
	testing/btest/Baseline/scripts.base.frameworks.file-analysis.logging/file_analysis.log
	testing/btest/Baseline/scripts.base.protocols.ftp.ftp-extract/ftp-item-0.dat
	testing/btest/Baseline/scripts.base.protocols.ftp.ftp-extract/ftp-item-1.dat
	testing/btest/Baseline/scripts.base.protocols.ftp.ftp-extract/ftp-item-2.dat
	testing/btest/Baseline/scripts.base.protocols.ftp.ftp-extract/ftp-item-3.dat
	testing/btest/Baseline/scripts.base.protocols.ftp.ftp-extract/ftp-item-BTsa70Ua9x7-1.dat
	testing/btest/Baseline/scripts.base.protocols.ftp.ftp-extract/ftp-item-BTsa70Ua9x7.dat
	testing/btest/Baseline/scripts.base.protocols.ftp.ftp-extract/ftp-item-Rqjkzoroau4-0.dat
	testing/btest/Baseline/scripts.base.protocols.ftp.ftp-extract/ftp-item-Rqjkzoroau4.dat
	testing/btest/Baseline/scripts.base.protocols.ftp.ftp-extract/ftp-item-VLQvJybrm38-2.dat
	testing/btest/Baseline/scripts.base.protocols.ftp.ftp-extract/ftp-item-VLQvJybrm38.dat
	testing/btest/Baseline/scripts.base.protocols.ftp.ftp-extract/ftp-item-zrfwSs9K1yk-3.dat
	testing/btest/Baseline/scripts.base.protocols.ftp.ftp-extract/ftp-item-zrfwSs9K1yk.dat
	testing/btest/Baseline/scripts.base.protocols.ftp.ftp-extract/ftp.log
	testing/btest/Baseline/scripts.base.protocols.http.http-extract-files/http-item-BFymS6bFgT3-0.dat
	testing/btest/Baseline/scripts.base.protocols.http.http-extract-files/http-item-BFymS6bFgT3.dat
	testing/btest/Baseline/scripts.base.protocols.http.http-extract-files/http-item.dat
	testing/btest/Baseline/scripts.base.protocols.http.http-extract-files/http.log
	testing/btest/Baseline/scripts.base.protocols.irc.dcc-extract/irc-dcc-item-wqKMAamJVSb-0.dat
	testing/btest/Baseline/scripts.base.protocols.irc.dcc-extract/irc-dcc-item-wqKMAamJVSb.dat
	testing/btest/Baseline/scripts.base.protocols.irc.dcc-extract/irc-dcc-item.dat
	testing/btest/Baseline/scripts.base.protocols.irc.dcc-extract/irc.log
	testing/btest/Baseline/scripts.base.protocols.smtp.mime-extract/smtp-entity-0.dat
	testing/btest/Baseline/scripts.base.protocols.smtp.mime-extract/smtp-entity-1.dat
	testing/btest/Baseline/scripts.base.protocols.smtp.mime-extract/smtp-entity-Ltd7QO7jEv3-1.dat
	testing/btest/Baseline/scripts.base.protocols.smtp.mime-extract/smtp-entity-Ltd7QO7jEv3.dat
	testing/btest/Baseline/scripts.base.protocols.smtp.mime-extract/smtp-entity-cwR7l6Zctxb-0.dat
	testing/btest/Baseline/scripts.base.protocols.smtp.mime-extract/smtp-entity-cwR7l6Zctxb.dat
	testing/btest/Baseline/scripts.base.protocols.smtp.mime-extract/smtp_entities.log
	testing/btest/scripts/base/protocols/ftp/ftp-extract.bro
	testing/btest/scripts/base/protocols/http/http-extract-files.bro
	testing/btest/scripts/base/protocols/irc/dcc-extract.test
	testing/btest/scripts/base/protocols/smtp/mime-extract.test
 2013-06-07 15:44:36 -05:00
Jon Siwek
e56a17102e Teach broxygen to generate protocol analyzer plugin reference. 2013-06-07 13:21:18 -05:00
Robin Sommer
433c85540c Merge remote-tracking branch 'origin/topic/jsiwek/plugins-cleanup' into topic/robin/plugins 
Adding one todo back in as that's something we indeed still need to do.

* origin/topic/jsiwek/plugins-cleanup:
  Fix various documentation/typos; remove a few superfluous things.
 2013-06-03 20:16:19 -07:00
Jon Siwek
a5e1810aa8 Fix various documentation/typos; remove a few superfluous things. 2013-06-03 16:03:25 -05:00
Seth Hall
caf61f619b Merge remote-tracking branch 'origin/topic/jsiwek/faf-cleanup' into topic/seth/faf-updates 2013-06-03 10:51:55 -04:00
Seth Hall
190f98f8a9 Beginning some rework. 2013-06-03 10:51:53 -04:00
Robin Sommer
c6ad731562 More smaller cleanup. 2013-06-02 18:21:45 -07:00
Robin Sommer
c049c758c3 Merge remote-tracking branch 'origin/master' into topic/robin/plugins 
Conflicts:
	aux/bro-aux
	aux/broctl
	src/DPM.cc
 2013-05-30 17:43:50 -07:00
Bernhard Amann
3719524a6a Merge remote branch 'origin/master' into topic/bernhard/input-update 2013-05-27 20:32:50 -07:00
Jon Siwek
e45933562e Fix broken/missing documentation. 2013-05-23 16:53:42 -05:00
Jon Siwek
9c86a3ee0e Add a general file analysis overview/how-to document. 2013-05-23 14:29:13 -05:00
Seth Hall
4f4ef99a6b SumStats changes to how thresholding works to simplify and reduce memory use. 2013-05-23 10:12:17 -04:00
Seth Hall
6bd9ab3bd6 More adjustments to try and correct SumStats memory use. 2013-05-22 16:41:46 -04:00
Seth Hall
c4a1f30a87 Hopefully fixing a strange error. 2013-05-22 14:59:31 -04:00
Seth Hall
0a18b62d12 Merge remote-tracking branch 'origin/master' into topic/seth/sumstats-updates 
Conflicts:
	scripts/base/frameworks/sumstats/cluster.bro
	scripts/base/frameworks/sumstats/plugins/average.bro
	scripts/base/frameworks/sumstats/plugins/max.bro
	scripts/base/frameworks/sumstats/plugins/min.bro
	scripts/base/frameworks/sumstats/plugins/sample.bro
	scripts/base/frameworks/sumstats/plugins/std-dev.bro
	scripts/base/frameworks/sumstats/plugins/sum.bro
	scripts/base/frameworks/sumstats/plugins/unique.bro
	scripts/base/frameworks/sumstats/plugins/variance.bro
	scripts/policy/protocols/http/detect-sqli.bro
	testing/btest/scripts/base/frameworks/sumstats/cluster-intermediate-update.bro
 2013-05-21 22:33:16 -04:00
Jon Siwek
705a84d688 Improve tracking of HTTP file extraction (addresses #988). 
http.log now has files taken from request and response bodies in
different fields for each, and can now track multiple files per body.
That is, the "extraction_file" field is now "extracted_request_files"
and "extracted_response_files".
 2013-05-21 16:42:35 -05:00
Jon Siwek
3cbef60f57 Fix HTTP multipart body file analysis. 
Each part now gets assigned a different file handle/id.
 2013-05-21 15:35:22 -05:00
Seth Hall
bec965b66f Large update for the SumStats framework. 
- On-demand access to sumstats results through "return from"
   functions named SumStats::request and Sumstats::request_key.
   Both functions are tested in standalone and clustered modes.

 - $name field has returned to SumStats which simplifies cluster
   code and makes the on-demand access stuff possible.

 - Clustered results can only be collected for 1 minute from their
   time of creation now instead of time of last read.

 - Thresholds use doubles instead of counts everywhere now.

 - Calculation dependency resolution occurs at start up time now
   instead of doing it at observation time which provide a minor
   cpu performance improvement.  A new plugin registration mechanism
   was created to support this change.

 - AppStats now has a minimal doc string and is broken into hook-based
   plugins.

 - AppStats and traceroute detection added to local.bro
 2013-05-21 15:52:59 -04:00
Jon Siwek
38ac03d558 Remove logging of analyzers field of FileAnalysis::Info. 
It was mostly redundant when logged, but still can be useful to
inspect at runtime.  In the future, a better field for logging
will be available which will be similar to the "service" field
for connection records (there's not any file-format-specific
analyzers that would currently make use of such a thing).
 2013-05-21 12:01:40 -05:00
Jon Siwek
28f51a9a22 Remove extraction counter in default file extraction scripts. 2013-05-21 11:12:00 -05:00
Jon Siwek
16f924c2c0 Remove FileAnalysis::postpone_timeout. 
FileAnalysis::set_timeout_interval can now perform same function.
 2013-05-21 10:50:07 -05:00
Jon Siwek
bc5cd3acc8 Make default get_file_handle handlers &priority=5. 
So they're easier to override (just provide a new handler without
specifying a priority).
 2013-05-21 10:34:19 -05:00
Jon Siwek
0ef074594d Add input interface to forward data for file analysis. 
The new Input::add_analysis function is used to automatically forward
input data on to the file analysis framework.
 2013-05-21 10:29:22 -05:00
Jon Siwek
90fa331279 File analysis framework interface simplifications. 
- Remove script-layer data input interface (will be managed directly
  by input framework later).

- Only track files internally by file id hash.  Chance of collision
  too small to justify also tracking unique file string.
 2013-05-20 12:02:48 -05:00