files.log: Unroll and introduce uid and id fields

This is a script-only change that unrolls File::Info records into
multiple files.log entries if the same file was seen over different
connections by single worker. Consequently, the File::Info record
gets the commonly used uid and id fields added. These fields are
optional for File::Info - a file may be analyzed without relation
to a network connection (e.g by using Input::add_analysis()).

The existing tx_hosts, rx_hosts and conn_uids fields of Files::Info
are not meaningful after this change and removed by default. Therefore,
files.log will have them removed, too.

The tx_hosts, rx_hosts and conn_uids fields can be revived by using the
policy script frameworks/files/deprecated-txhosts-rxhosts-connuids.zeek
included in the distribution. However, with v6.1 this script will be
removed.
This commit is contained in:
Arne Welzel 2022-08-16 17:22:20 +02:00
parent f631551ffb
commit d2314d2666
39 changed files with 412 additions and 113 deletions

View file

@ -0,0 +1,13 @@
### BTest baseline data generated by btest-diff. Do not edit. Use "btest -U/-u" to update. Requires BTest >= 0.63.
#separator \x09
#set_separator ,
#empty_field (empty)
#unset_field -
#path conn
#open XXXX-XX-XX-XX-XX-XX
#fields ts uid id.orig_h id.orig_p id.resp_h id.resp_p proto service duration orig_bytes resp_bytes conn_state local_orig local_resp missed_bytes history orig_pkts orig_ip_bytes resp_pkts resp_ip_bytes tunnel_parents
#types time string addr port addr port enum string interval count count string bool bool count string count count count count set[string]
XXXXXXXXXX.XXXXXX CHhAvVGS1DHFjwGM9 192.168.0.107 58716 88.198.248.254 80 tcp - 0.125216 117 10290 SF - - 0 ShADadFf 9 593 7 10662 -
XXXXXXXXXX.XXXXXX ClEkJM2Vm5giqnMf4h 192.168.0.107 58718 88.198.248.254 80 tcp - 0.173517 111 10284 SF - - 0 ShADadtFf 11 703 10 10812 -
XXXXXXXXXX.XXXXXX C4J4Th3PJpwUYZZ6gc 192.168.0.107 58720 88.198.248.254 80 tcp - 0.124639 117 10290 SF - - 0 ShADadFf 11 697 9 10766 -
#close XXXX-XX-XX-XX-XX-XX

View file

@ -0,0 +1,13 @@
### BTest baseline data generated by btest-diff. Do not edit. Use "btest -U/-u" to update. Requires BTest >= 0.63.
#separator \x09
#set_separator ,
#empty_field (empty)
#unset_field -
#path files
#open XXXX-XX-XX-XX-XX-XX
#fields ts fuid uid id.orig_h id.orig_p id.resp_h id.resp_p source depth analyzers mime_type filename duration local_orig is_orig seen_bytes total_bytes missing_bytes overflow_bytes timedout parent_fuid
#types time string string addr port addr port string count set[string] string string interval bool bool count count count count bool string
XXXXXXXXXX.XXXXXX FaGjhv1ozACeoEnwg5 C4J4Th3PJpwUYZZ6gc 192.168.0.107 58720 88.198.248.254 80 HTTP 0 (empty) - - 0.076646 - F 30003 104857600 179998 0 T -
XXXXXXXXXX.XXXXXX FaGjhv1ozACeoEnwg5 CHhAvVGS1DHFjwGM9 192.168.0.107 58716 88.198.248.254 80 HTTP 0 (empty) - - 0.076646 - F 30003 104857600 179998 0 T -
XXXXXXXXXX.XXXXXX FaGjhv1ozACeoEnwg5 ClEkJM2Vm5giqnMf4h 192.168.0.107 58718 88.198.248.254 80 HTTP 0 (empty) - - 0.076646 - F 30003 104857600 179998 0 T -
#close XXXX-XX-XX-XX-XX-XX

View file

@ -0,0 +1,13 @@
### BTest baseline data generated by btest-diff. Do not edit. Use "btest -U/-u" to update. Requires BTest >= 0.63.
#separator \x09
#set_separator ,
#empty_field (empty)
#unset_field -
#path http
#open XXXX-XX-XX-XX-XX-XX
#fields ts uid id.orig_h id.orig_p id.resp_h id.resp_p trans_depth method host uri referrer version user_agent origin request_body_len response_body_len status_code status_msg info_code info_msg tags username password proxied orig_fuids orig_filenames orig_mime_types resp_fuids resp_filenames resp_mime_types
#types time string addr port addr port count string string string string string string string count count count string count string set[enum] string string set[string] vector[string] vector[string] vector[string] vector[string] vector[string] vector[string]
XXXXXXXXXX.XXXXXX CHhAvVGS1DHFjwGM9 192.168.0.107 58716 88.198.248.254 80 1 GET speed.hetzner.de /100MB.bin - 1.1 curl/7.74.0 - 0 10001 206 Partial Content - - (empty) - - - - - - FaGjhv1ozACeoEnwg5 - -
XXXXXXXXXX.XXXXXX C4J4Th3PJpwUYZZ6gc 192.168.0.107 58720 88.198.248.254 80 1 GET speed.hetzner.de /100MB.bin - 1.1 curl/7.74.0 - 0 10001 206 Partial Content - - (empty) - - - - - - FaGjhv1ozACeoEnwg5 - -
XXXXXXXXXX.XXXXXX ClEkJM2Vm5giqnMf4h 192.168.0.107 58718 88.198.248.254 80 1 GET speed.hetzner.de /100MB.bin - 1.1 curl/7.74.0 - 0 10001 206 Partial Content - - (empty) - - - - - - FaGjhv1ozACeoEnwg5 - -
#close XXXX-XX-XX-XX-XX-XX