Commit graph

3048 commits

Author SHA1 Message Date
Robin Sommer
63d43e6545 Renaming ASCII writer filter option 'only_single_header_row' to 'tsv'.
Also clarifying usage.

Closes #912.
2012-12-03 14:40:38 -08:00
Bernhard Amann
9c09dee294 and adapt to AsciiInputOutput - seems to work... 2012-12-03 14:14:40 -08:00
Bernhard Amann
0a59d0d4db Merge branch 'topic/bernhard/input-logging-commmon-functions' into topic/bernhard/sqlite 2012-12-03 13:46:58 -08:00
Bernhard Amann
9b2265877d and factor stuff out the input framework too. 2012-12-03 13:41:19 -08:00
Bernhard Amann
501328d61a factor out ascii input/output.
First step - factored out everything the logging classes
use ( so only output ).

Moved the script-level configuration to logging/main,
and made the individual writers just refer to it -
no idea if this is good design. It works. But I am happy
about opinions :)

Next step - add support for input...
2012-12-03 12:59:11 -08:00
Seth Hall
f956554c74 Slightly fix up file name extraction from Content-Disposition headers. 2012-12-03 11:57:00 -05:00
Vlad Grigorescu
e98343b562 Remove hardcoded HTTP verbs from the analyzer (#741) 2012-11-30 20:08:20 -05:00
Jon Siwek
cc8f20c104 Merge branch 'master' into topic/jsiwek/gtp 2012-11-29 16:11:27 -06:00
Seth Hall
f1b7ca62ee Actually fix the problem I just tried to fix a minute ago. 2012-11-28 15:58:29 -05:00
Seth Hall
92285a9711 Fix a race condition when multiple workers report intermediate indexes simultaneously. 2012-11-28 15:52:41 -05:00
Seth Hall
2add60b4b1 A function wasn't returning a value like it should be. 2012-11-28 15:22:45 -05:00
Seth Hall
956c23eb66 Merge remote-tracking branch 'origin/master' into topic/seth/metrics-merge 2012-11-28 14:57:42 -05:00
Bernhard Amann
2d7ffd8269 Merge remote-tracking branch 'origin/master' into topic/bernhard/sqlite 2012-11-26 20:46:27 -08:00
Seth Hall
6bdcdcecf9 Fixed a problem with metrics aggregation on clusters (thanks Jon!). 2012-11-26 16:17:35 -05:00
Seth Hall
c98301e51f Fixed a DNS attribute issue (reported by Matt Thompson). 2012-11-26 15:58:25 -05:00
Robin Sommer
a5e237f50c The ASCII writer now supports a filter config option
'only_single_header_row' that turns the output into CSV format.

In that mode all meta data is skipped except for a single header line
with the fields names. Example:

    local my_filter: Log::Filter = [$name = "my-filter", $writer = Log::WRITER_ASCII, $config = table(["only_single_header_row"] = "T")];

Contributed by Carsten Langer.
2012-11-23 19:38:53 -08:00
Seth Hall
3546d93f36 Merging master. 2012-11-21 12:18:03 -05:00
Seth Hall
08538211e1 Some test updates. 2012-11-20 02:08:49 -05:00
Seth Hall
20fdd36a44 Updated the SQL injection detection script to make it include samples in notice emails. 2012-11-20 01:02:23 -05:00
Seth Hall
95b12262e4 More cleanup and fixed to the metrics framework. 2012-11-19 23:43:15 -05:00
Seth Hall
5b81cfe7e2 Implemented a nearly generic Queue in scriptland. 2012-11-19 23:42:19 -05:00
Seth Hall
e99e090b85 Merge remote-tracking branch 'origin/master' into topic/seth/metrics-merge 2012-11-16 02:49:36 -05:00
Seth Hall
d9195076b1 Metrics framework checkpoint.
- New scan.bro merged in and reworked a bit.

 - Updated metrics API.  Now possible to calculate much more.
2012-11-16 02:37:52 -05:00
Jon Siwek
e0805498c6 Fix some warnings from sphinx when building docs. 2012-11-15 16:40:18 -06:00
Robin Sommer
a40b00d4ab Merge branch 'topic/robin/intel-framework-merge'
* topic/robin/intel-framework-merge: (22 commits)
  Fixing tests after intel-framework merge.
  Extracting URLs from message bodies over SMTP and sending them to Intel framework.
  Small comment updates in the Intel framework CIF support.
  Intelligence framework documentation first draft.
  Only the manager tries to read files with the input framework now.
  Initial support for Bro's Intel framework with the Collective Intelligence Framework.
  Initial API for Intel framework is complete.
  Fixed an issue with cluster data distribution.
  Updating some intel framework test baselines.
  Reworked cluster intelligence data distribution mechanism and fixed tests.
  Lots more intelligence checking in SMTP traffic.
  Added intelligence check for "Received" path checking and a bit of reshuffling.
  Added sources to the intel log.
  Fixing a problem with intel distribution on clusters.
  Updated intel framework test to include matching.
  Restructuring the scripts that feed data into the intel framework slightly.
  One test for cluster transparency of the intel framework.
  Fixed a cluster support bug.
  Intelligence framework checkpoint
  Major updates to fix the Intel framework API.
  ...

Closes #914.
2012-11-05 16:00:38 -08:00
Robin Sommer
4e12813445 Fixing tests after modbus merge. 2012-11-05 15:58:38 -08:00
Robin Sommer
86ce564107 Merge remote-tracking branch 'remotes/origin/topic/seth/modbus-merge'
* remotes/origin/topic/seth/modbus-merge:
  Small modbus documentation update and tiny refactoring.
  Final touches to modbus analyzer for now.
  Major revisions to Modbus analyzer support (not quite done yet).
  put some make-up on Modbus analyser
  Modbus analyser, added support: FC=20,21
  Modbus analyzer,added support: FC=1,2,15,24
  Modbus analyzer, current support: FC=3,4,5,6,7,16,22,23

Closes #915.
2012-11-05 15:26:57 -08:00
Robin Sommer
a6216969e6 Merge remote-tracking branch 'origin/fastpath'
* origin/fastpath:
  Script in base for detecting cases of checksum offloading.
2012-11-05 13:59:30 -08:00
Robin Sommer
fb7ba82bab Merge remote-tracking branch 'origin/topic/seth/intel-framework'
* origin/topic/seth/intel-framework: (21 commits)
  Extracting URLs from message bodies over SMTP and sending them to Intel framework.
  Small comment updates in the Intel framework CIF support.
  Intelligence framework documentation first draft.
  Only the manager tries to read files with the input framework now.
  Initial support for Bro's Intel framework with the Collective Intelligence Framework.
  Initial API for Intel framework is complete.
  Fixed an issue with cluster data distribution.
  Updating some intel framework test baselines.
  Reworked cluster intelligence data distribution mechanism and fixed tests.
  Lots more intelligence checking in SMTP traffic.
  Added intelligence check for "Received" path checking and a bit of reshuffling.
  Added sources to the intel log.
  Fixing a problem with intel distribution on clusters.
  Updated intel framework test to include matching.
  Restructuring the scripts that feed data into the intel framework slightly.
  One test for cluster transparency of the intel framework.
  Fixed a cluster support bug.
  Intelligence framework checkpoint
  Major updates to fix the Intel framework API.
  Checkpoint commit.  This is all a huge mess right now. :)
  ...

Closes #914.
2012-11-01 08:21:52 -07:00
Seth Hall
e020e034ae Script in base for detecting cases of checksum offloading.
- I added this to the base scripts because it's very minimal overhead
  and we get questions about this all the time.  Now that reporter
  messages are printed to the console, people will see the message
  clearly if they have bad checksums.
2012-11-01 00:16:28 -04:00
Seth Hall
c32b179ac5 Small modbus documentation update and tiny refactoring. 2012-10-31 23:57:38 -04:00
Seth Hall
a2f336cc72 Final touches to modbus analyzer for now.
- There are still some broken events in the modbus analyzer because
  I don't have traffic to test with (coil and record related events primarily).

- There are a few example scripts in policy/protocols/modbus
2012-10-31 23:34:43 -04:00
Seth Hall
172a6345b8 Extracting URLs from message bodies over SMTP and sending them to Intel framework.
- New utils package for URLs.

- Two functions in the URLs utils.  find_all_urls and
  find_all_urls_without_scheme.
2012-10-31 08:46:40 -04:00
Daniel Thayer
c46029d828 Rename the alarm summary mail variable 2012-10-30 16:31:18 -05:00
Daniel Thayer
0f97f0b6e4 Merge remote-tracking branch 'origin/master' into topic/dnthayer/alarms-mail 2012-10-30 11:32:58 -05:00
Robin Sommer
ae38aad2bb Adding missing &redef for some TCP options.
Reported by Carsten Langer.

Closes #905.
Closes #906.
Closes #907.
2012-10-24 15:41:27 -07:00
Seth Hall
1200d04f81 Merge branch 'master' into topic/seth/metrics-merge 2012-10-22 10:06:02 -04:00
Jon Siwek
9edbf3e53c Add GPRS Tunnelling Protocol (GTPv1) decapsulation.
This currently supports automatic decapsulation of GTP-U packets on
UDP port 2152.

The GTPv1 headers for such tunnels can be inspected by handling the
"gtpv1_g_pdu_packet" event, which has a parameter of type "gtpv1_hdr".

Analyzer and test cases are derived from submissions by Carsten Langer.

Addresses #690.
2012-10-19 14:02:35 -05:00
Sheharbano Khattak
d5cf730b51 Added str to pred function 2012-10-18 13:45:31 +05:00
Jon Siwek
18f8427579 Change how "gridftp" gets added to service field of connection records.
In addition to checking for a finished SSL handshake over an FTP
connection, it now also requires that the SSL handshake occurs after
the FTP client requested AUTH GSSAPI, more specifically identifying the
characteristics of GridFTP control channels.

Addresses #891.
2012-10-17 12:09:12 -05:00
Sheharbano Khattak
6244bf4308 Started the branch testing2 2012-10-17 04:13:13 +05:00
Sheharbano Khattak
d510702078 Added the branch /testing 2012-10-16 05:54:38 +05:00
Robin Sommer
5e12a53ae5 Merge remote-tracking branch 'origin/topic/jsiwek/gridftp'
* origin/topic/jsiwek/gridftp:
  Add memory leak unit test for GridFTP.
  Enable GridFTP detection by default.  Track/log SSL client certs.
  Add analyzer for GSI mechanism of GSSAPI FTP AUTH method.
  Add an example of a GridFTP data channel detection script.
2012-10-12 10:43:16 -07:00
Robin Sommer
503412e472 Merge remote-tracking branch 'origin/topic/bernhard/input-end-of-data'
* origin/topic/bernhard/input-end-of-data:
  rename the update_finished event to end_of_data and make it fire in more cases.

Closes #894.
2012-10-12 09:48:58 -07:00
Robin Sommer
b1b1991cc7 Merge remote-tracking branch 'origin/topic/seth/dns-zero-responses-fix'
* origin/topic/seth/dns-zero-responses-fix:
  Fix for DNS log problem when a DNS response is seen with 0 RRs.

Closes #892.
2012-10-12 09:39:18 -07:00
Bernhard Amann
a6d87fcab7 rename the update_finished event to end_of_data and make it fire in
more cases.

It will now not only fire after table-reads have been completed,
but also after the last event of a whole-file-read (or whole-db-read, etc.).

The interface also has been extended a bit to allow readers to
directly fire the event should they so choose. This allows the
event to be fired in direct table-setting/event-sending modes,
which was previously not possible.
2012-10-10 11:51:20 -07:00
Seth Hall
9ea52fe9cd Only the manager tries to read files with the input framework now. 2012-10-10 13:00:48 -04:00
Seth Hall
5aa12984ee Merge remote-tracking branch 'origin/master' into topic/seth/intel-framework 2012-10-10 11:22:34 -04:00
Seth Hall
21473b0557 Initial API for Intel framework is complete.
- More inline docs added.

- Removing some debugging code.

- New test for the intel framework data distribution mechanism.
2012-10-10 11:15:34 -04:00
Seth Hall
6538f70e2c Fixed an issue with cluster data distribution. 2012-10-10 11:13:16 -04:00