Commit graph

27 commits

Author SHA1 Message Date
Seth Hall
a2556642e6 Merge remote-tracking branch 'origin/topic/matthias/notary'
* origin/topic/matthias/notary:
  Small cosmetic changes.
  Give log buffer the correct name.
  Simplify delayed logging of SSL records.
  Implement delay-token style SSL logging.
  More style tweaks: replace spaces with tabs.
  Factor notary code into separte file.
  Adhere to Bro coding style guidelines.
  Enhance ssl.log with information from notary.

Closes #928
2013-02-05 02:06:33 -05:00
Matthias Vallentin
32a0ead698 Give log buffer the correct name. 2012-12-24 23:06:56 -08:00
Matthias Vallentin
7ff15f4599 Simplify delayed logging of SSL records. 2012-12-24 22:57:49 -08:00
Matthias Vallentin
9e81342c92 Implement delay-token style SSL logging.
This commit moves the notary script into the policy directory, along with some
architectural changes: the main SSL script now has functionality to add and
remove tokens for a given record. When adding a token, the script delays the
logging until the token has been removed or until the record exceeds a maximum
delay time.

As before, the base SSL script stores all records sequentially and buffers even
non-delayed records for the sake of having an ordered log file. If this turns
out to be not so important, we can easily revert to a simpler logic.

(This is still WiP, some debuggin statements still linger.)
2012-12-22 20:30:17 -08:00
Matthias Vallentin
382262e286 Factor notary code into separte file.
There exists one complication: the new file notary.bro requires the definition
of the SSL::Info record, but as does main.bro. Because I did not really know
where to put the common code (it's not a constant, so ssl/const.bro does not
really fit), I put it into __load.bro__ so that it sticks out for now. If
anybody has an idea how to solve this elegantly, please let me know.
2012-12-21 17:56:31 -08:00
Matthias Vallentin
7355a0089a Adhere to Bro coding style guidelines. 2012-12-21 17:17:58 -08:00
Matthias Vallentin
ff8184242a Enhance ssl.log with information from notary.
This commit brings enhances each log line with the data from the notary when
available. The added fields include:

  - notary.first_seen
  - notary.last_seen
  - notary.times_seen
  - notary.valid

The semantics of these fields map 1-to-1 to the corresponding fields in DNS TXT
lookups from the notary. The implementation of this feature required a bit
plumbing: when Bro finishes the analysis, the log record is copied into table
indexed by connection ID where it remains until either Bro terminates or the
answer of the notary arrives. The script accummulates requests for a given
digest into a "waitlist," to avoid multiple redundant lookups for high-profile
websites who receive a large chunk of traffic. When a DNS reply arrives
asynchronously, the when handler clears the waitlist and assigns the
information to all records in the buffered.

The script also adds Each log entry into a double-ended queue to make sure the
records arrive on disk in the same way Bro sees them. Each reply also triggers
a sweep through this deque which flushes the buffer up to the first outstanding
reply.

Here is an example from the public M57 trace from 2009:

  % bro-cut ts id.orig_h id.resp_h server_name notary.first_seen notary.last_seen notary.times_seen notary.valid < ssl.log
  1258562650.121682 192.168.1.104 208.97.132.223  mail.m57.biz  - - - -
  1258535660.267128 192.168.1.104 65.55.184.16  - - - - -
  1258561662.604948 192.168.1.105 66.235.128.158  - - - - -
  1258561885.571010 192.168.1.105 65.55.184.155 www.update.microsoft.com  - - - -
  1258563578.455331 192.168.1.103 208.97.132.223  - - - - -
  1258563716.527681 192.168.1.103 96.6.248.124  - - - - -
  1258563884.667153 192.168.1.103 66.235.139.152  - - - - -
  1258564818.755676 192.168.1.103 12.41.118.177 - - - - -
  1258564821.637874 192.168.1.103 12.41.118.177 - - - - -
  1258564821.637871 192.168.1.103 12.41.118.177 - - - - -
  1258564821.637876 192.168.1.103 12.41.118.177 - - - - -
  1258564821.638126 192.168.1.103 12.41.118.177 - - - - -
  1258562467.525034 192.168.1.104 208.97.132.223  mail.m57.biz  15392 15695 301 F
  1258563063.965975 192.168.1.104 63.245.209.105  aus2.mozilla.org  - - - -
  1258563064.091396 192.168.1.104 63.245.209.91 addons.mozilla.org  - - - -
  1258563329.202273 192.168.1.103 208.97.132.223  - 15392 15695 301 F
  1258563712.945933 192.168.1.103 65.55.16.121  - - - - -
  1258563714.044500 192.168.1.103 65.54.186.79  - - - - -
  1258563716.146680 192.168.1.103 96.6.248.124  - - - - -
  1258563737.432312 192.168.1.103 96.6.245.186  - - - - -
  1258563716.526933 192.168.1.103 96.6.245.186  - - - - -
  1258563716.527430 192.168.1.103 96.6.245.186  - - - - -
  1258563716.527179 192.168.1.103 96.6.245.186  - - - - -
  1258563716.527683 192.168.1.103 96.6.245.186  - - - - -
  1258563716.527432 192.168.1.103 96.6.245.186  - - - - -
  1258563751.178683 192.168.1.103 66.235.139.152  - - - - -
  1258563751.171938 192.168.1.103 65.54.234.75  - - - - -
  1258563751.182433 192.168.1.103 65.242.27.35  - - - - -
  1258563883.414188 192.168.1.103 65.55.16.121  - - - - -
  1258563884.702380 192.168.1.103 65.242.27.35  - - - - -
  1258563885.678766 192.168.1.103 65.54.186.79  - - - - -
  1258563886.124987 192.168.1.103 65.54.186.79  - - - - -
  1258564027.877525 192.168.1.103 65.54.234.75  - - - - -
  1258564688.206859 192.168.1.103 65.54.186.107 - - - - -
  1258567162.001225 192.168.1.105 208.97.132.223  mail.m57.biz  - - - -
  1258568040.512840 192.168.1.103 208.97.132.223  - - - - -
  1258564688.577376 192.168.1.103 207.46.120.170  - - - - -
  1258564723.029005 192.168.1.103 65.54.186.107 - - - - -
  1258564723.784032 192.168.1.103 65.55.194.249 - - - - -
  1258564748.521756 192.168.1.103 65.54.186.107 - - - - -
  1258564817.601152 192.168.1.103 12.41.118.177 - - - - -
  1258565684.353653 192.168.1.105 208.97.132.223  mail.m57.biz  15392 15695 301 F
  1258565710.188691 192.168.1.105 74.125.155.109  pop.gmail.com - - - -
  1258566061.103696 192.168.1.103 208.97.132.223  - 15392 15695 301 F
  1258566893.914987 192.168.1.102 208.97.132.223  - 15392 15695 301 F
2012-12-21 17:03:39 -08:00
Robin Sommer
5e12a53ae5 Merge remote-tracking branch 'origin/topic/jsiwek/gridftp'
* origin/topic/jsiwek/gridftp:
  Add memory leak unit test for GridFTP.
  Enable GridFTP detection by default.  Track/log SSL client certs.
  Add analyzer for GSI mechanism of GSSAPI FTP AUTH method.
  Add an example of a GridFTP data channel detection script.
2012-10-12 10:43:16 -07:00
Jon Siwek
e34f6d9e3b Enable GridFTP detection by default. Track/log SSL client certs.
In the *service* field of connection records, GridFTP control channels
are labeled as "gridftp" and data channels as "gridftp-data".

Added *client_subject* and *client_issuer_subject* as &log'd fields to
SSL::Info record.  Also added *client_cert* and *client_cert_chain*
fields to track client cert chain.
2012-10-08 11:38:29 -05:00
Vlad Grigorescu
f43576cff3 Fix some Info:Record field documentation. 2012-07-13 14:04:24 -04:00
Seth Hall
0a6104fe66 More bugfixs, cleanup, and test for SSL analyzer
- SSL related files and classes renamed to remove the "binpac" term.

- A small fix for DPD scripts to make the DPD log more helpful if
  there are multiple continued failures.  Also, fixed the SSL
  analyzer to make it stop doing repeated violation messages for
  some handshake failures.

- Added a $issuer_subject to the SSL log.

- Created a basic test for SSL.
2012-05-03 10:52:24 -04:00
Seth Hall
911d7d8436 Finished SSL & syslog autodocs. 2012-01-10 00:56:12 -05:00
Daniel Thayer
acf5537acf Add ssl and syslog script documentation 2012-01-09 15:26:34 -06:00
Robin Sommer
f3c2811e14 Merge remote branch 'origin/topic/seth/ssl-updates-for-2.0'
* origin/topic/seth/ssl-updates-for-2.0:
  Added is_orig fields to the SSL events and adapted script.

Closes #692.
2011-12-18 15:15:57 -08:00
Seth Hall
ec721dffec Added is_orig fields to the SSL events and adapted script.
- Added a field named $last_alert to the SSL log.  This doesn't even
  indicate the direction the alert was sent, but we need to start somewhere.

- The x509_certificate function has an is_orig field now instead of
  is_server and it's position in the argument list has moved.

- A bit of reorganization and cleanup in the core analyzer.
2011-12-09 16:56:12 -05:00
Seth Hall
6d67f7830d Added to the likely_server_ports set for protocols with analyzers.
- Updated some tests since Bro is getting the direction
  correct now.

- Updated BPF filter test since I added a few ports to IRC
  as well.
2011-10-07 13:44:28 -04:00
Seth Hall
8710d3749f New SSL policy scripts.
- protocols/ssl/expiring-certs uses time based information from
  certificates to determine if they will expire soon, have already
  expired, or haven't yet become valid.

- protocols/ssl/extract-certs-pem is a script for taking certs off
  the line and converting them to PEM certificates with the openssl
  command line tool then dumping them to a file.
2011-09-25 02:42:36 -04:00
Seth Hall
f53d5fe0b9 Added session ID to the SSL logging. 2011-09-20 13:59:22 -04:00
Seth Hall
123a3bd4e3 Small rework with ssl base script to reduce memory usage.
- We are now removing the SSL analyzer after logging the session
  infomrtion.  This seems to help a lot with overly high memroy
  consumption.
2011-09-16 23:47:04 -04:00
Seth Hall
ee1884ca93 Another change to possibly fix the SSL memory consumption problem.
- If a protocol violation happens, Bro now logs what it has seen
  up until the protocol violation and deletes the c$ssl record
  so that a long lived connection with a protocol violation does
  continue to hold the memory.
2011-09-13 09:09:55 -04:00
Seth Hall
c87704cc25 Small usability and correctness updates.
- Removed an notice definition from the base SSL scripts.

- Moved a logging stream ID into the export section for known-services
  and bumped priority for creating the stream.

- Adding configuration knobs for the SQL injection attack detection
  script and renaming the HTTP::SQL_Injection_Attack notice to
  HTTP::SQL_Injection_Attack_Against

- Bumped priority when creating Known::CERTS_LOG.
2011-09-12 16:14:28 -04:00
Seth Hall
4931aa815f Delete SSL certificates from memory after ssl_established event.
- This is an attempt at fixing the memory issues brought about by
  the introduction of the new SSL analyzer.  My initial testing
  shows a hefty memory saving.
2011-09-08 01:52:25 -04:00
Seth Hall
e07e4ca117 Cleaned up some small SSL mistakes. 2011-09-07 15:11:01 -04:00
Seth Hall
11c437faa3 Logging framework update and mass Log::ID renaming.
- Log path's are generated in the scripting land
  now.  The default Log stream ID to path string
  mapping works like this:
    - Notice::LOG -> "notice"
    - Notice::POLICY_LOG -> "notice_policy"
    - TestModule::LOG -> "test_module"

- Logging streams updated across all of the shipped
  scripts to be more user friendly.  Instead of
  the logging stream ID HTTP::HTTP, we now have
  HTTP::LOG, etc.

- The priorities on some bro_init handlers have
  been adjusted to make the process of applying
  filters or disabling streams easier for users.
2011-09-03 01:10:17 -04:00
Seth Hall
fc5f22cb5d Merge remote-tracking branch 'origin/topic/jsiwek/reorg-followup' 2011-08-25 16:44:31 -04:00
Jon Siwek
47500ceef4 Add a test that checks each individual script can be loaded in bare-mode.
Fixed most @load dependency issues in the process.  The test is still
failing in a "known" way due to hot.conn.bro and scan.bro.

Adressess #545
2011-08-10 15:38:21 -05:00
Seth Hall
597a4d6704 Hopefully the last major script reorganization.
- policy/ renamed to scripts/

- By default BROPATH now contains:
	- scripts/
	- scripts/policy
	- scripts/site

- *Nearly* all tests pass.

- All of scripts/base/ is loaded by main.cc
	- Can be disabled by setting $BRO_NO_BASE_SCRIPTS
	- Scripts in scripts/base/ don't use relative path loading to ease use of BRO_NO_BASE_SCRIPTS (to copy and paste that script).

- The scripts in scripts/base/protocols/ only (or soon will only) do logging and state building.

- The scripts in scripts/base/frameworks/ add functionality without causing any additional overhead.

- All "detection" activity happens through scripts in scripts/policy/.

- Communications framework modified temporarily to need an environment variable to actually enable (ENABLE_COMMUNICATION=1)
	- This is so the communications framework can be loaded as part
	  of the base without causing trouble when it's not needed.
	- This will be removed once a resolution to ticket #540 is reached.
2011-08-05 23:09:53 -04:00
Renamed from policy/protocols/ssl/base.bro (Browse further)