Implement X509 certificate log caching

By default, each certificate is now output only once per hour. This also
should work in cluster mode, where we use the net broker-table-syncing
feature to distribute the information about already seen certificates
across the entire cluster.

Log caching is also pretty configureable and can be changed using a
range of confiuration options and hooks.

Note that this is currently completely separate from X509 events
caching, which prevents duplicate parsing of X509 certificates.
This commit is contained in:
Johanna Amann 2021-06-28 15:41:57 +01:00
parent 311e113ff6
commit dde1e2e77e
17 changed files with 400 additions and 152 deletions

View file

@ -0,0 +1,16 @@
### BTest baseline data generated by btest-diff. Do not edit. Use "btest -U/-u" to update. Requires BTest >= 0.63.
#separator \x09
#set_separator ,
#empty_field (empty)
#unset_field -
#path x509
#open XXXX-XX-XX-XX-XX-XX
#fields ts fp certificate.version certificate.serial certificate.subject certificate.issuer certificate.not_valid_before certificate.not_valid_after certificate.key_alg certificate.sig_alg certificate.key_type certificate.key_length certificate.exponent certificate.curve san.dns san.uri san.email san.ip basic_constraints.ca basic_constraints.path_len host_cert client_cert
#types time string count string string string time time string string string count string string vector[string] vector[string] vector[string] vector[addr] bool count bool bool
XXXXXXXXXX.XXXXXX a97a22b691caf62c1623d14abf8a31ac915f14d87f77d8a37c47eb4785b484ad 3 1E58FDC12DE4C703 CN=*.google.com,O=Google Inc,L=Mountain View,ST=California,C=US CN=Google Internet Authority G2,O=Google Inc,C=US XXXXXXXXXX.XXXXXX XXXXXXXXXX.XXXXXX rsaEncryption sha1WithRSAEncryption rsa 2048 65537 - *.google.com,*.android.com,*.appengine.google.com,*.cloud.google.com,*.google-analytics.com,*.google.ca,*.google.cl,*.google.co.in,*.google.co.jp,*.google.co.uk,*.google.com.ar,*.google.com.au,*.google.com.br,*.google.com.co,*.google.com.mx,*.google.com.tr,*.google.com.vn,*.google.de,*.google.es,*.google.fr,*.google.hu,*.google.it,*.google.nl,*.google.pl,*.google.pt,*.googleapis.cn,*.googlecommerce.com,*.googlevideo.com,*.gstatic.com,*.gvt1.com,*.urchin.com,*.url.google.com,*.youtube-nocookie.com,*.youtube.com,*.youtubeeducation.com,*.ytimg.com,android.com,g.co,goo.gl,google-analytics.com,google.com,googlecommerce.com,urchin.com,youtu.be,youtube.com,youtubeeducation.com - - - F - T F
XXXXXXXXXX.XXXXXX a047a37fa2d2e118a4f5095fe074d6cfe0e352425a7632bf8659c03919a6c81d 3 023A69 CN=Google Internet Authority G2,O=Google Inc,C=US CN=GeoTrust Global CA,O=GeoTrust Inc.,C=US XXXXXXXXXX.XXXXXX XXXXXXXXXX.XXXXXX rsaEncryption sha1WithRSAEncryption rsa 2048 65537 - - - - - T 0 F F
XXXXXXXXXX.XXXXXX 3c35cc963eb004451323d3275d05b353235053490d9cd83729a2faf5e7ca1cc0 3 12BBE6 CN=GeoTrust Global CA,O=GeoTrust Inc.,C=US OU=Equifax Secure Certificate Authority,O=Equifax,C=US XXXXXXXXXX.XXXXXX XXXXXXXXXX.XXXXXX rsaEncryption sha1WithRSAEncryption rsa 2048 65537 - - - - - T - F F
XXXXXXXXXX.XXXXXX a97a22b691caf62c1623d14abf8a31ac915f14d87f77d8a37c47eb4785b484ad 3 1E58FDC12DE4C703 CN=*.google.com,O=Google Inc,L=Mountain View,ST=California,C=US CN=Google Internet Authority G2,O=Google Inc,C=US XXXXXXXXXX.XXXXXX XXXXXXXXXX.XXXXXX rsaEncryption sha1WithRSAEncryption rsa 2048 65537 - *.google.com,*.android.com,*.appengine.google.com,*.cloud.google.com,*.google-analytics.com,*.google.ca,*.google.cl,*.google.co.in,*.google.co.jp,*.google.co.uk,*.google.com.ar,*.google.com.au,*.google.com.br,*.google.com.co,*.google.com.mx,*.google.com.tr,*.google.com.vn,*.google.de,*.google.es,*.google.fr,*.google.hu,*.google.it,*.google.nl,*.google.pl,*.google.pt,*.googleapis.cn,*.googlecommerce.com,*.googlevideo.com,*.gstatic.com,*.gvt1.com,*.urchin.com,*.url.google.com,*.youtube-nocookie.com,*.youtube.com,*.youtubeeducation.com,*.ytimg.com,android.com,g.co,goo.gl,google-analytics.com,google.com,googlecommerce.com,urchin.com,youtu.be,youtube.com,youtubeeducation.com - - - F - T F
XXXXXXXXXX.XXXXXX a047a37fa2d2e118a4f5095fe074d6cfe0e352425a7632bf8659c03919a6c81d 3 023A69 CN=Google Internet Authority G2,O=Google Inc,C=US CN=GeoTrust Global CA,O=GeoTrust Inc.,C=US XXXXXXXXXX.XXXXXX XXXXXXXXXX.XXXXXX rsaEncryption sha1WithRSAEncryption rsa 2048 65537 - - - - - T 0 F F
XXXXXXXXXX.XXXXXX 3c35cc963eb004451323d3275d05b353235053490d9cd83729a2faf5e7ca1cc0 3 12BBE6 CN=GeoTrust Global CA,O=GeoTrust Inc.,C=US OU=Equifax Secure Certificate Authority,O=Equifax,C=US XXXXXXXXXX.XXXXXX XXXXXXXXXX.XXXXXX rsaEncryption sha1WithRSAEncryption rsa 2048 65537 - - - - - T - F F
#close XXXX-XX-XX-XX-XX-XX