Commit graph

11634 commits

Author SHA1 Message Date
Scott Runnels
f9359ad40d Moving to new rootedliteralinclude directive. 2013-01-09 14:23:09 -05:00
Scott Runnels
546be58ded Merge remote-tracking branch 'origin/master' into topic/documentation 2013-01-09 13:32:00 -05:00
Jon Siwek
01090cf09f Teach sphinx a new ".. rootedliteralinclude::" directive.
It's like ".. literalinclude::" except the argument is an absolute path
which may contain environment variables to be be expanded when
generating documents.
2013-01-09 11:16:43 -06:00
Scott Runnels
c453c228cb Iteration over vectors. 2013-01-08 15:31:56 -05:00
Scott Runnels
7d583b6b78 Vector declarations and explanations, including btests 2013-01-08 14:47:39 -05:00
Robin Sommer
564e27abb6 Merge remote-tracking branch 'origin/topic/jsiwek/string-indexing'
* origin/topic/jsiwek/string-indexing:
  Change substring index notation to use a colon (addresses #422).

Tweaked slightly to make it more generic, we may index other types
with slices eventually too.

Closes #422.
2013-01-08 10:20:12 -08:00
Jon Siwek
8b46bbb1c0 Change substring index notation to use a colon (addresses #422).
String slice notation is written as `s[1:2]` instead of `s[1, 2]`
because the later is ambiguous with composite index types.
2013-01-07 13:29:05 -06:00
Robin Sommer
e638f04301 Updating submodule(s).
[nomail]
2013-01-07 09:47:09 -08:00
Robin Sommer
4488dce493 Merge remote-tracking branch 'origin/fastpath'
* origin/fastpath:
  Fix memory leak in OpaqueType::DoUnserialize.
2013-01-07 09:43:56 -08:00
Jon Siwek
8000c40fee Fix memory leak in OpaqueType::DoUnserialize. 2013-01-07 11:41:36 -06:00
Seth Hall
720089c03f Add a feature to better select the value threshold will apply to in the metrics framework.
- The feature was primarily added to allow the value to be
   modified for cluster based intermediate threshold checks
   without requiring the user to write the metrics filter
   differently for cluster consideration.  It's also a nice
   way to calculate some related information to the metric
   without accidently applying thresholds to that value.

 - Fixed a few small bugs in ftp detect-bruteforcing script
   and adapted it to the new threshold value selection feature.
2013-01-05 22:37:19 -05:00
Seth Hall
bcd7fe114d Fixed an issue causing reporter messages from the metrics framework. 2013-01-05 22:27:17 -05:00
Scott Runnels
5260e10290 A short discussion of variables versus values in the section on Local Variables. 2013-01-05 17:59:47 -05:00
Seth Hall
9366411cf4 Fix the FTP bruteforce threshold to what it's really supposed to be. 2013-01-04 23:49:09 -05:00
Seth Hall
283f7840b3 Removing some debugging print statements I accidently left behind. 2013-01-04 23:38:10 -05:00
Seth Hall
9e1d5d87de New script to detect FTP bruteforcing.
- Created a new time utils library.
2013-01-04 23:34:57 -05:00
Seth Hall
ab7087f953 Hooks work now, enabling the hooks in the scan.bro script. 2013-01-04 21:01:49 -05:00
Seth Hall
89bc65f23c Merge remote-tracking branch 'origin/master' into topic/seth/metrics-merge 2013-01-04 20:49:37 -05:00
Seth Hall
c3a6916572 More cluster tuning for the metrics framework.
- Fixed several state maintenance issues for intermediate updates.

 - Added a new tuning variable Metrics::max_outstanding_global_views
   which limits the number of in-flight intermediate updates per
   metric filter.

 - Changed the default global view threshold percent to 20% (up from 10%)
2013-01-04 16:54:13 -05:00
Scott Runnels
b8c31458ff Local variables. Baselines and data_type_local.bro btest-able script. 2013-01-03 17:03:45 -05:00
Matthias Vallentin
32a0ead698 Give log buffer the correct name. 2012-12-24 23:06:56 -08:00
Matthias Vallentin
7ff15f4599 Simplify delayed logging of SSL records. 2012-12-24 22:57:49 -08:00
Matthias Vallentin
9e81342c92 Implement delay-token style SSL logging.
This commit moves the notary script into the policy directory, along with some
architectural changes: the main SSL script now has functionality to add and
remove tokens for a given record. When adding a token, the script delays the
logging until the token has been removed or until the record exceeds a maximum
delay time.

As before, the base SSL script stores all records sequentially and buffers even
non-delayed records for the sake of having an ordered log file. If this turns
out to be not so important, we can easily revert to a simpler logic.

(This is still WiP, some debuggin statements still linger.)
2012-12-22 20:30:17 -08:00
Seth Hall
50827d8df0 Implement an option to disable intermediate updates for the metrics framework.
- There are some large sites having trouble and I'm thinking it
   might be overload from intermediate updates.
2012-12-21 23:17:27 -05:00
Matthias Vallentin
8a569facd6 More style tweaks: replace spaces with tabs. 2012-12-21 18:04:19 -08:00
Matthias Vallentin
382262e286 Factor notary code into separte file.
There exists one complication: the new file notary.bro requires the definition
of the SSL::Info record, but as does main.bro. Because I did not really know
where to put the common code (it's not a constant, so ssl/const.bro does not
really fit), I put it into __load.bro__ so that it sticks out for now. If
anybody has an idea how to solve this elegantly, please let me know.
2012-12-21 17:56:31 -08:00
Matthias Vallentin
7355a0089a Adhere to Bro coding style guidelines. 2012-12-21 17:17:58 -08:00
Matthias Vallentin
ff8184242a Enhance ssl.log with information from notary.
This commit brings enhances each log line with the data from the notary when
available. The added fields include:

  - notary.first_seen
  - notary.last_seen
  - notary.times_seen
  - notary.valid

The semantics of these fields map 1-to-1 to the corresponding fields in DNS TXT
lookups from the notary. The implementation of this feature required a bit
plumbing: when Bro finishes the analysis, the log record is copied into table
indexed by connection ID where it remains until either Bro terminates or the
answer of the notary arrives. The script accummulates requests for a given
digest into a "waitlist," to avoid multiple redundant lookups for high-profile
websites who receive a large chunk of traffic. When a DNS reply arrives
asynchronously, the when handler clears the waitlist and assigns the
information to all records in the buffered.

The script also adds Each log entry into a double-ended queue to make sure the
records arrive on disk in the same way Bro sees them. Each reply also triggers
a sweep through this deque which flushes the buffer up to the first outstanding
reply.

Here is an example from the public M57 trace from 2009:

  % bro-cut ts id.orig_h id.resp_h server_name notary.first_seen notary.last_seen notary.times_seen notary.valid < ssl.log
  1258562650.121682 192.168.1.104 208.97.132.223  mail.m57.biz  - - - -
  1258535660.267128 192.168.1.104 65.55.184.16  - - - - -
  1258561662.604948 192.168.1.105 66.235.128.158  - - - - -
  1258561885.571010 192.168.1.105 65.55.184.155 www.update.microsoft.com  - - - -
  1258563578.455331 192.168.1.103 208.97.132.223  - - - - -
  1258563716.527681 192.168.1.103 96.6.248.124  - - - - -
  1258563884.667153 192.168.1.103 66.235.139.152  - - - - -
  1258564818.755676 192.168.1.103 12.41.118.177 - - - - -
  1258564821.637874 192.168.1.103 12.41.118.177 - - - - -
  1258564821.637871 192.168.1.103 12.41.118.177 - - - - -
  1258564821.637876 192.168.1.103 12.41.118.177 - - - - -
  1258564821.638126 192.168.1.103 12.41.118.177 - - - - -
  1258562467.525034 192.168.1.104 208.97.132.223  mail.m57.biz  15392 15695 301 F
  1258563063.965975 192.168.1.104 63.245.209.105  aus2.mozilla.org  - - - -
  1258563064.091396 192.168.1.104 63.245.209.91 addons.mozilla.org  - - - -
  1258563329.202273 192.168.1.103 208.97.132.223  - 15392 15695 301 F
  1258563712.945933 192.168.1.103 65.55.16.121  - - - - -
  1258563714.044500 192.168.1.103 65.54.186.79  - - - - -
  1258563716.146680 192.168.1.103 96.6.248.124  - - - - -
  1258563737.432312 192.168.1.103 96.6.245.186  - - - - -
  1258563716.526933 192.168.1.103 96.6.245.186  - - - - -
  1258563716.527430 192.168.1.103 96.6.245.186  - - - - -
  1258563716.527179 192.168.1.103 96.6.245.186  - - - - -
  1258563716.527683 192.168.1.103 96.6.245.186  - - - - -
  1258563716.527432 192.168.1.103 96.6.245.186  - - - - -
  1258563751.178683 192.168.1.103 66.235.139.152  - - - - -
  1258563751.171938 192.168.1.103 65.54.234.75  - - - - -
  1258563751.182433 192.168.1.103 65.242.27.35  - - - - -
  1258563883.414188 192.168.1.103 65.55.16.121  - - - - -
  1258563884.702380 192.168.1.103 65.242.27.35  - - - - -
  1258563885.678766 192.168.1.103 65.54.186.79  - - - - -
  1258563886.124987 192.168.1.103 65.54.186.79  - - - - -
  1258564027.877525 192.168.1.103 65.54.234.75  - - - - -
  1258564688.206859 192.168.1.103 65.54.186.107 - - - - -
  1258567162.001225 192.168.1.105 208.97.132.223  mail.m57.biz  - - - -
  1258568040.512840 192.168.1.103 208.97.132.223  - - - - -
  1258564688.577376 192.168.1.103 207.46.120.170  - - - - -
  1258564723.029005 192.168.1.103 65.54.186.107 - - - - -
  1258564723.784032 192.168.1.103 65.55.194.249 - - - - -
  1258564748.521756 192.168.1.103 65.54.186.107 - - - - -
  1258564817.601152 192.168.1.103 12.41.118.177 - - - - -
  1258565684.353653 192.168.1.105 208.97.132.223  mail.m57.biz  15392 15695 301 F
  1258565710.188691 192.168.1.105 74.125.155.109  pop.gmail.com - - - -
  1258566061.103696 192.168.1.103 208.97.132.223  - 15392 15695 301 F
  1258566893.914987 192.168.1.102 208.97.132.223  - 15392 15695 301 F
2012-12-21 17:03:39 -08:00
Robin Sommer
5526f09b60 Updating submodule(s).
[nomail]
2012-12-20 17:54:44 -08:00
Robin Sommer
5e3c286490 Merge remote-tracking branch 'origin/topic/jsiwek/string-indexing'
* origin/topic/jsiwek/string-indexing:
  Add array-style index accessor for strings.  Addresses #422.

Closes #422.
2012-12-20 17:43:26 -08:00
Robin Sommer
da90976170 Merge remote-tracking branch 'origin/topic/matthias/opaque'
* origin/topic/matthias/opaque:
  Add new unit test for opaque serialization.
  Migrate entropy testing to opaque.
  C++ify RandTest.*
  Fix a hard-to-spot bug.
  Use more descriptive error message.
  Fix the fix :-/.
  Fix initialization of hash values.
  Be clearer about delegation.
  Implement serialization of opaque types.
  Update hash BiF documentation.
  Migrate free SHA* functions to SHA*Val::digest().
  Add missing type name that caused failing tests.
  Update base scripts and unit tests.
  Simplify hash function BiFs.
  Add support for opaque hash values.
  Adapt BiF & Bro parser to handle opaque types.
  More lexer/parser work.
  Implement equivalence relation for opaque types.
  Support basic serialization of opaque.
  Add opaque type to lexer, parser, and BroType.

Closes #925

Conflicts:
	aux/broccoli
2012-12-20 16:30:22 -08:00
Robin Sommer
037085605c Merge remote-tracking branch 'origin/topic/jsiwek/ticket-923'
* origin/topic/jsiwek/ticket-923:
  Improve error for invalid use of types as values (addresses #923).

Close 923.
2012-12-20 15:21:27 -08:00
Robin Sommer
9301589219 Merge remote-tracking branch 'origin/topic/jsiwek/switch-statement'
* origin/topic/jsiwek/switch-statement:
  Finish implementation of script-layer switch statement.  Addresses #754.

Closes #754.
2012-12-20 15:13:30 -08:00
Jon Siwek
55c515d50a Add array-style index accessor for strings. Addresses #422.
The index expression can take up to two indices for the start and end
index of the substring to return (e.g. "mystring[1,3]").  Negative
indices are allowed, with -1 representing the last character in the
string.  The indexing is not cyclic -- if the starting index is >= the
length of the string an empty string is returned, and if the ending
index is >= the length of the string then it's interpreted as the last
index of the string.  Assigning to substrings accessed like this isn't
allowed.
2012-12-20 17:13:06 -06:00
Robin Sommer
ca8a8b09fa Merge remote-tracking branch 'origin/topic/jsiwek/ticket-730'
* origin/topic/jsiwek/ticket-730:
  Fix a case where c$resp$size is misrepresented.  Addresses #730.

Closes #730.
2012-12-20 14:53:43 -08:00
Daniel Thayer
1c0905914c Document undocumented attributes 2012-12-20 15:25:29 -06:00
Robin Sommer
40007d075c Merge remote-tracking branch 'origin/fastpath'
* origin/fastpath:
  Fix return value of hook calls that have no handlers.
  Fix to_port() BIF for port strings with a port number of zero.
2012-12-20 12:12:44 -08:00
Daniel Thayer
75f32093ab Merge remote-tracking branch 'origin/master' into topic/dnthayer/lang-docs 2012-12-20 14:06:36 -06:00
Jon Siwek
939a64b73a Finish implementation of script-layer switch statement. Addresses #754.
They behave like C-style switches except case labels can be comprised
of multiple literal constants delimited by commas.  Only atomic types
are allowed for now.  Case label bodies that don't execute a "return"
or "break" statement will fall through to subsequent cases.  A default
case label is allowed.
2012-12-20 12:49:50 -06:00
Scott Runnels
cce5565d63 Merge remote-tracking branch 'origin/master' into topic/documentation 2012-12-19 23:20:52 -05:00
Scott Runnels
7dc4fd34d3 Merge remote-tracking branch 'origin/master' into topic/documentation 2012-12-19 22:54:23 -05:00
Jon Siwek
98663fd534 Fix return value of hook calls that have no handlers.
For this case, the return value is always true.
2012-12-18 15:31:50 -06:00
Jon Siwek
4a09c12882 Fix to_port() BIF for port strings with a port number of zero. 2012-12-18 15:08:18 -06:00
Jon Siwek
2f0c698ed5 Improve error for invalid use of types as values (addresses #923).
This scripting error can now generate an error message at parse-time
instead of run-time and also includes location information.
2012-12-18 14:31:39 -06:00
Seth Hall
9c00ef3ccd Doing some code reorganization and small changes to hopefully fix a memory issue. 2012-12-18 12:22:28 -05:00
Seth Hall
6e9e3a5e88 Small fixes.
- Slight wording change in scan notices.

 - AppMetrics no longer writes to metrics.log.
2012-12-18 10:31:38 -05:00
Seth Hall
ed36f37643 A few more small fixes. 2012-12-18 02:23:56 -05:00
Seth Hall
f31de38c9b Bug fix. 2012-12-18 01:44:46 -05:00
Seth Hall
7edef1f2c4 Disable the hook execution in the scan.bro script. It's not working like I expected. 2012-12-18 01:31:52 -05:00
Seth Hall
69b7ce12d2 API updates for metrics framework.
- Removed default logging.  Now a function is available for the new
  $period_finished filter field to get the same behavior for logging
  named Metrics::write_log.

- Added index rollups for getting multiple metrics result values
  as the same time.
2012-12-18 01:08:59 -05:00