From c8103dd9636d57672c54cc149d7f078d0ddaff06 Mon Sep 17 00:00:00 2001 From: xb-anssi <149695709+xb-anssi@users.noreply.github.com> Date: Thu, 2 Nov 2023 17:09:56 +0100 Subject: [PATCH 1/2] Test how the signature framework matches HTTP body This adds a signatures/http-body-match btest to verify how the signature framework matches HTTP body in requests and responses. It currently fails because the 'http-request-body' and 'http-reply-body' clauses never match anything when there is a '$' in their regular expressions. The other pattern clauses such as the 'payload' clause do not suffer from that restriction and it is not documented as a limitation of HTTP body pattern clauses either, so it is probably a bug. The "http-body-match" btest shows that without a fix any signatures which ends with a '$' in a http-request-body or http-reply-body rule will never raise a signature_match() event, and that signatures which do not end with a '$' cannot distinguish an HTTP body prefixed by the matching pattern (ex: ABCD) from an HTTP body consisting entirely of the matching pattern (ex: AB). Test cases by source port: - 13579: - GET without body, plain res body (CD, only) - 13578: - GET without body, plain res body (CDEF, prefix) - 24680: - POST plain req body (AB, only), plain res body (CD, only) - 24681: - POST plain req body (ABCD, prefix), plain res body (CDEF, prefix) - 24682: - POST gzipped req body (AB, only), gzipped res body (CD, only) - POST plain req body (CD, only), plain res body (EF, only) - 33210: - POST multipart plain req body (AB;CD;EF, prefix) - plain res body (CD, only) - 33211: - POST multipart plain req body (ABCD;EF, prefix) - plain res body (CDEF, prefix) - 34527: - POST chunked gzipped req body (AB, only) - chunked gzipped res body (CD, only) - 34528: - POST chunked gzipped req body (ABCD, prefix) - chunked gzipped res body (CDEF, prefix) The tests with source ports 24680, 24682 and 34527 should match the signature http_request_body_AB_only and the signature http_request_body_AB_prefix, but they only match the latter. The tests with source ports 13579, 24680, 24682, 33210 and 34527 should match the signature http_response_body_CD_only and the signature http_response_body_CD_prefix, but they only match the latter. The tests with source ports 24680, 24681, 33210 and 33211 show how the http_request_body_AB_then_CD signature with two http-request-body conditions match either on one or multiple requests (documented behaviour). The test cases with other source ports show where the http_request_body_AB_only and http_response_body_CD_only signatures should not match because their bodies include more than the searched patterns. --- .../Baseline/signatures.http-body-match/out | 27 +++++++++++ .../btest/Traces/http/http-body-match.pcap | Bin 0 -> 21623 bytes testing/btest/signatures/http-body-match.zeek | 43 ++++++++++++++++++ 3 files changed, 70 insertions(+) create mode 100644 testing/btest/Baseline/signatures.http-body-match/out create mode 100644 testing/btest/Traces/http/http-body-match.pcap create mode 100644 testing/btest/signatures/http-body-match.zeek diff --git a/testing/btest/Baseline/signatures.http-body-match/out b/testing/btest/Baseline/signatures.http-body-match/out new file mode 100644 index 0000000000..6a8982e7ce --- /dev/null +++ b/testing/btest/Baseline/signatures.http-body-match/out @@ -0,0 +1,27 @@ +### BTest baseline data generated by btest-diff. Do not edit. Use "btest -U/-u" to update. Requires BTest >= 0.63. +HTTP body match for 192.0.2.42:13578 -> 192.88.99.42:80 with signature 'http_response_body_CD_prefix', data: 'D' +HTTP body match for 192.0.2.42:13579 -> 192.88.99.42:80 with signature 'http_response_body_CD_only', data: '' +HTTP body match for 192.0.2.42:13579 -> 192.88.99.42:80 with signature 'http_response_body_CD_prefix', data: 'D' +HTTP body match for 192.0.2.42:24680 -> 192.88.99.42:80 with signature 'http_request_body_AB_only', data: '' +HTTP body match for 192.0.2.42:24680 -> 192.88.99.42:80 with signature 'http_request_body_AB_prefix', data: 'B' +HTTP body match for 192.0.2.42:24680 -> 192.88.99.42:80 with signature 'http_response_body_CD_only', data: '' +HTTP body match for 192.0.2.42:24680 -> 192.88.99.42:80 with signature 'http_response_body_CD_prefix', data: 'D' +HTTP body match for 192.0.2.42:24681 -> 192.88.99.42:80 with signature 'http_request_body_AB_prefix', data: 'B' +HTTP body match for 192.0.2.42:24681 -> 192.88.99.42:80 with signature 'http_response_body_CD_prefix', data: 'D' +HTTP body match for 192.0.2.42:24682 -> 192.88.99.42:80 with signature 'http_request_body_AB_only', data: '' +HTTP body match for 192.0.2.42:24682 -> 192.88.99.42:80 with signature 'http_request_body_AB_prefix', data: 'AB' +HTTP body match for 192.0.2.42:24682 -> 192.88.99.42:80 with signature 'http_request_body_AB_then_CD', data: 'CD' +HTTP body match for 192.0.2.42:24682 -> 192.88.99.42:80 with signature 'http_response_body_CD_only', data: '' +HTTP body match for 192.0.2.42:24682 -> 192.88.99.42:80 with signature 'http_response_body_CD_prefix', data: 'CD' +HTTP body match for 192.0.2.42:33210 -> 192.88.99.42:80 with signature 'http_request_body_AB_prefix', data: 'AB' +HTTP body match for 192.0.2.42:33210 -> 192.88.99.42:80 with signature 'http_request_body_AB_then_CD', data: 'CD' +HTTP body match for 192.0.2.42:33210 -> 192.88.99.42:80 with signature 'http_response_body_CD_only', data: '' +HTTP body match for 192.0.2.42:33210 -> 192.88.99.42:80 with signature 'http_response_body_CD_prefix', data: 'D' +HTTP body match for 192.0.2.42:33211 -> 192.88.99.42:80 with signature 'http_request_body_AB_prefix', data: 'ABCD' +HTTP body match for 192.0.2.42:33211 -> 192.88.99.42:80 with signature 'http_response_body_CD_prefix', data: 'D' +HTTP body match for 192.0.2.42:34527 -> 192.88.99.42:80 with signature 'http_request_body_AB_only', data: '' +HTTP body match for 192.0.2.42:34527 -> 192.88.99.42:80 with signature 'http_request_body_AB_prefix', data: 'AB' +HTTP body match for 192.0.2.42:34527 -> 192.88.99.42:80 with signature 'http_response_body_CD_only', data: '' +HTTP body match for 192.0.2.42:34527 -> 192.88.99.42:80 with signature 'http_response_body_CD_prefix', data: 'CD' +HTTP body match for 192.0.2.42:34528 -> 192.88.99.42:80 with signature 'http_request_body_AB_prefix', data: 'ABCD' +HTTP body match for 192.0.2.42:34528 -> 192.88.99.42:80 with signature 'http_response_body_CD_prefix', data: 'CDEF' diff --git a/testing/btest/Traces/http/http-body-match.pcap b/testing/btest/Traces/http/http-body-match.pcap new file mode 100644 index 0000000000000000000000000000000000000000..bc90359f1fde49f77eefc85cc8fe50493424bee8 GIT binary patch literal 21623 zcmdU%3s_ZE+Q-j<1Io=ylZrVNhcppI?kX3#i%ex6Cp9#bgLqIb0s@+v_Y^8M2h1*3 zDt_rq%|$J{d8te#Ck3@}ta)Y}8^0lyOj9|CaLn)hueI0S8}Tg5zQ$%h&wid6)Zh8< z_kGv@eb?HDj_!H0+QncpXqU-gr2q4o%k#fVIB7IIK>r)J{_1ELW-!E7>`5>rxYs`T zk-;VKqx9@RYo5W<=W%Q~v{R2P$e&o8RKrSyAWb0IeO5K{lR&<6vqm;od9166 zPndP9u-S5xH$fc$W&j$G9*7L|9aW@83pUN|Odf1pct^{(9z%IN_5Le=>iQR;v1ozNSkC=7k5+>( zGBr?E)&F@cLgz$yW%*i zeisn;dnrV!cr#T#h@+&rHnBctQTv$E)eETV-9hX^^8%^|m2R$Vc=qPR{EpDr&`D@i zdhR5u_6!ieQ;DwKQ0X9!BaMTov~lY%R^b6@Z-MAtEQm_4HB!F2ScsVH$tR=9r;LVp zqk-MmQ;eY%`#?E3T2R_lwXd=1i3Cx2Q1MFW+V>4$3OXok{dz zvx@1*^h6;cLD}?4xrH<6r#_xYN-l-dx$UG6MK~RQ(-t6Wc6<9=PWGoXI}#q@;iOQe zBBX-oMG3;KzgU{D=6is6CPhsUAJ%x`P=dbbAWG4-I|}D^B`9|?_TaWuLqlO1N}Z!Sg8ecPV<^zL^%sI|3`*Mu zqF0^>b~&G4`I@3&7tnlly8`^9SHS3!tisY9T9w+;$euc*FyA)Wc^YIZVFQKIxjil4 zk5KL(uO)}IqAtDTF1nLF*^}J4J?$XMNy-Fn{l%*PXbiI_-#0}<+O%;>$kMwMds-8I z6m5L?u%C%L-OyKfQGX;Y>Ltw94`*v}V}tl98pL;_sGF}S4sxY#3JotuQPlq?Vh9y9 zZvDlI+TVT|nV1+VV%TQPpcwx4ec>Ppb1-K7EDL39TpbPB;uONn>WGgYO+&qtVfn!my zW+D!!*y7e-%tE5-Uj(smwuoVyT>!;!@CHt#UBExF3%Fec`?Kqjy(z5Do!P{^e&kFM z);8M=3Tx>Zr8MKsQ?bqXS%&r2ojEk(rUPJKX`zP?LRp&sr4>3mrt#f)AfO0^?xSp= zsU+P{nke)@DbYUz#IPM2ksWoouf6Ox9>mUcn#}B{vSB4{{-%O>yvWs%OltzxV$PwvrAUZzO5phPxw=KR#hsO?1q;1O^*tUqRA9hH7Xt??v z(kS{U=i)1B@KLoEH6H9JKG~l>?l_5Wil>8Yb}+MUP7iz>!jVVBd88V*{$k>b?9Cuf z{9i%j^UY}sJ4T@h$MhEwPWUIyH`PHwHyRgGa!ZSgN^|*UP&Z=~W=*pV^)C+f7aPDb z65~AGb;}^8{23ueY3&!-=?0wgB_YQ81bP@t_Z(@&G8%wv9-t*hoj;Z#-EqI*NOP5^ z8_gfA%Ab@HTiAig_;)p;+GH<+IE%)5rO9rfCi?_(ut}N&(>7?d1c=AxPUbeA9hlH# zdyNy1ZLTFJTlph%@V4~v{3k$6*mmvl)J41+kJl+ddA!PoMq0$Rc)ZVv*h;m9TYs?< z`(_DyyhE>RMBPwM>%;aMC(;(=Ack^feT4Xh(fSzqppRz?omxcurIzFl@Cyz1OSMhH zpG5hEg-3^nMu&&^4IeSu$MX?e$xK^GwBHTSORgVl$vPjzkwqeoEa(U%cbcteM%mEF z0e;i6W(|$vgRr^_(f+D7Ah+M2^8xLYG`KNr{q`IsGOg^$c0Y4u=XwgR{W2|5wtm+{ zL?IjqZr|y$;Kt5Jx_W%63oZ_>`M5&Cjp!`WwfQ8*71exxy5Q0ft3~YpHMoN*6wb}@ zToyu+^R^mXev)~G!m$Iv-FJ$Ig2J$aX4R7jZYmMs>bUh6clPkXAeukXf~#wh6x?Go zl;9>U&P9t5`JX}DpP&qC0&= zKb)W@p5;dr{>>(dm5N`oID8vOPsC1>%jC#b(fNvo`Nrv`rl#I9FlBI}leSpKvslE3V{DB4owAtBOwg4)ssCelnu=OSCI zK#qS$ClYV6uv5DPk&jPPL{ zOh&UyWGV+qRc3q*W^;c{rP@4KK;_WkLZxS3O?+Z9Liq?0Q%N3f{l!!cF|n$-{*Axe zYSHSRrdoyp*2MdVQmboBOiE_&g0iJL{n@@Z?0ub1yko=Q`4V*clqxzaza9rNbQTiP z`6RN0={#)Ji0D~6*Rz8pTKU^NIC0dKLm@UYYxd~#q#|QJK`?t-M1pkQUri$IP1`t< zaIXRjk;jN=p(x_kUo4nO6^vM$D2V*^gE9@$+)&-35ZBF_i)rxQWgR)hwY1EIeevTT z(jdexfMVYlbU{FG*}=BL6okcUShu9YR9O(dk_I6?9!SUM8j>CHAEzV!5M(NDjS|;u z-8$PyHiB?TN<23S#GGv+2+9csTz@CRy6+xNtcg2yueowdDu|a`OolNO3f%gOg>`uS z8W4+e1+k44Q&?BFaw1vrZ(+sscB5@%`6SSxRb~L;L+~Ny6IvixYYC;@*J1?1i&D;} z$fU(}_#jrYS51b8NH1>v#bn2&y#u26qpEDZA(QOWT?~fMkhWS&W*QA#?NVYpH~ZzE zg1tXorxv}WADu`466ev0pL4U{w9s8*QTg7WG;|aS6|)Z#X8&HGlB#%9vYFhOFa5WP z>D)-I+rezi(3GieAB;G*dMk0PtR;&wu@c?2$lb7j0*G6Gu{ch*w}3eDgixs%=?Y<_ z15}Gj{4E)2o{#4(&EsL}p6CF{asGfAXCbGVee(*E3>MXcKg9#a(m& zCj6sg)`jFakC}faViu-52UYhl#}G4b(OoOQ-bMZ3c`UB3Qeq5yA=7iDctD+b*jnpirlcnL%AU? zgaLVhRBL(yp5AdxwWEh{6l~&yf1h?HRcquC6$YUsk85ejcYT~m}5DoHEnL;@DG$)5@#iT)`T_`dVoy{X;Q4e{|5wZN_W4XNPpDB#Wi)N-db znw6*qPi1MElb#I6M?i2|h0H(F-UZP+Q&es_Z#2JTlRpW*Qg1Mj$c=tI{%)eD<{~o* zDd{f5e;f+Dur897{y!YstC;#7i4Ak-COOYiY_mLS*eB zG_3!?TKefuL&i}kBwLnNgyQ-y>7M>nblT?Ul;FlEc~hsYS}cCc~TJBYB~+#CL%f?JX$7!IQ}gyk~ld%@)+CnQt5}y6_E@)4F6=q z+B;H_b{*;<-zBPv(A565%CxFrVh9MLSHtzr6+>uFzG~U z-IGKPQ1;Q(lfOXs{Bi5^Mqlm{hn3nlP4H$;J9<>#&Hr=gSS6M`N~MvsnVTld5;gb6_p_U>q)(*ky1A9O=h{VNE1(F{RUwp=Sk58QW&6RANJ zqfyZ|hPRbrgmv!T;nBdFk88P9d*5?d6dr$23NGo%7l4M)j$Jz%Vn`uw{l%uXptScu z^v+R*^4`}^^uDXr^hFXOt~a41J?|h;`^W0UiJmu*^}Kz&=RL4l_K4>ugOc;65Tt8R zA0QWdtjYq2JAH>}P$g9!5?5Zr-086H{ETf6Hi+%PW-&fqIXnU5E_bIz#B`pB7*;Vo zA68X*vM=@H-1{}CYyt*I0#9XKQ?z6R8p>QiXpC;vLV);&{#mjX7II zHMUuWU1=5eR-QWU^a!yE1EfTQ0mLyqw9K$s_5M70lz7$_TBbOm-0 z7c-IbiT%bX`V0`?SnXlxMQs?j{$j1zCp-p<&90haoqdzy^bjQmNl!Xp-~YK*Ec-U- zPxS6J8{>s~~sU_149q$2Y>19|=LW#*&t^gt*tJH&&#O{d25B1CrnrA%Z? zHi+Zj)E=8{S2DzQB}8cx=BoZAGV66F@_s@(k7V6SAj>O6c9bu;V7oE_B1=~bk&~ta z3z4Z(VhlSuY4_`rn5;G~q|S5cJSW=L3i8+bE=7W z=F&X=%v4fq$f;h!GdKQ^w`4zK{|>}VcR^G-lb7gBA8r%G8k||D6oALP4@b& z@|%(5U3=NaZU!Npmy?Ml$TC3<1bUvSck4#5T#I zCQuqAh?9)JobuWh_44R(48LgV;Mt6H`*qc0Gr7%AATL+m^>qptOE zM|cqIS|W}l)5Wd7SjSBa9}1$mRwFW7yj$2}@*7I}l9ud)Ytic(;o|(*pk6AtejV^d zkBeODb{m1HV0UG!N9eU=i2ES;>CLibw)_x?`%`s-VTiuM5RblQz&oWuzy74wM32iw zzD`Jq5AFccd#Wl@^T9sM2mj1{@U8)!B=fZO15g@ur*v#m?I9-Bt^ZIWS$QanCe`VD zo(?3Ya|6il0%n7|P^NoF(G%fon#!3q@$S)P;v#&ovvpOk37 zhTyi__uW1J_=esf4kn}E=WfYs%@kk|{eoqcUr`HU?Qo%ypXa0)*|0{2ZBOGwa-`4U zO#JO!2r->Mclt0e9Y?ehP@T?|RJIQ3&zZ!_0~ac-L>x|Q9=HBtP8Ss(2cr3?Mr2MG zD4cG$==e!X`@xRX!}%hq(E9bj7R?k7uX8wjt;p_H$%XDazthD`Xb4>u-AVH1VEJJX z_m9vCg~Rm`4u{`6QN{g{P!eZjht1woRB@WK1u|#b%bji2pJlno?+QT;Zi1q7p}X*i zP4S{)+$X_>{z8vrk#($d#rw{szX^hOc2x9YnS{h(N8+xxQICD4Vxl4cJ8qHNc zfn-8!lYtyaNIde5AG?W(%gOm+(O?-+gupmB6Ue`iI-1>`&Je!^h zkq%!?q;8q`8e3nv#zoRHIfP~M1D!Io?QcxT&f(N|jq>mGQ){NTmOuCxCJ>2<6ba|C ztza9a+B6Yc#e`hDRtC>yvE9rgVOuvZ7SRAP_Hh}pz8bM@_o@+_xnLZaxG5GyD|-r7 zHi%!&krBh=K{S7%Ws6x*Fj>%QB05aFl#)tXx)M|n?NVa>2Bxu0JhkFJt4Slu-fr5% ziA0-98drkoPE{tcj9nml=cq)zWzbV65D}Ek6*9{5BcSXbrF&9%SRi>=umzNlIps4c zC5??#*Z+6UE(Q~+KeI1|66Y{V1~Y#fw1jQZw6FnXw-``%Kd(f+^4vTcP*w~}#q*6M zrt@kuF=MqnCZeQk%X<)a_xm`rCcY2-w)44pEbdojME@M7a_e;}_b`=I8}7aID*KED z@$woMnMRgL%lI2;WM?RDWGTGS_n9v=W-O8Y)?LOWXl(Y^X@n*A5tex2_lm|UpNsIY z6o@P#r1L;F8G+2~rix_$ai%k{#QPD*h)Y6b$^o*(iy-XeyTxO{@$p<5Z2>HheEL0%h4!vbT#s_L33}258iGD^`+wDcN7sQ#;&#t!S(~ z_b*VPaSahur~}~EU#t-fN*@lQL${AZBM6>LJ409MR;}Nz|NRawEd zTJ7dKSn7P_CQ>O$+q(H~;6i0ag)EZkX-s9ERzOVUe5SIiu{H8ey~_D+AXWy*z8zn2 z7{uCCksF=uFJSd^ffGsNK%wzue;KjK1;ord1yR?e`!Zq;CsOS+QAJ*Y#%`V5WHs)} zh`+cYk<2h+nIH}e5yT)V(a^qrun_xFjo*X%iSXk@b5*BDYvPB6K;r{lWEv}&Vb=~5 n8r#GX8kI9QiesLLqg6_5uaV_?0p;368e1cN#WfP`DyaVlAOmL+ literal 0 HcmV?d00001 diff --git a/testing/btest/signatures/http-body-match.zeek b/testing/btest/signatures/http-body-match.zeek new file mode 100644 index 0000000000..e7b944e65f --- /dev/null +++ b/testing/btest/signatures/http-body-match.zeek @@ -0,0 +1,43 @@ +# @TEST-EXEC: zeek -b -r $TRACES/http/http-body-match.pcap %INPUT | sort >out +# @TEST-EXEC: btest-diff out + +@load-sigs test.sig +@load base/protocols/http + +@TEST-START-FILE test.sig +signature http_request_body_AB_prefix { + http-request-body /^AB/ + event "HTTP request body starting with AB" +} + +signature http_request_body_AB_only { + http-request-body /^AB$/ + event "HTTP request body containing AB only" +} + +signature http_request_body_AB_then_CD { + http-request-body /AB/ + http-request-body /CD/ + event "HTTP request body containing AB and CD, but maybe not be on same request (documented behaviour)" +} + +signature http_response_body_CD_prefix { + http-reply-body /^CD/ + event "HTTP response body starting with CD" +} + +signature http_response_body_CD_only { + http-reply-body /^CD$/ + event "HTTP response body containing CD only" +} +@TEST-END-FILE + +event signature_match(state: signature_state, msg: string, data: string) +{ + print(fmt("HTTP body match for %s:%d -> %s:%d with signature '%s', data: '%s'", + state$conn$id$orig_h, state$conn$id$orig_p, + state$conn$id$resp_h, state$conn$id$resp_p, + state$sig_id, + data + )); +} From 9e61bfd0108f5e47c6cd124ca35fdf47a228d400 Mon Sep 17 00:00:00 2001 From: xb-anssi <149695709+xb-anssi@users.noreply.github.com> Date: Fri, 3 Nov 2023 13:21:16 +0100 Subject: [PATCH 2/2] Let signature framework match HTTP body end The HTTP analyzer never tells the signature framework when the body of a request or a response ends, so any signature regex ending in a '$' used in an 'http-request-body' or in an 'http-reply-body' condition will never match. This made it impossible to write a signature which could distinguish an HTTP body consisting only of something from an HTTP body prefixed by that same something. - Fix: The fix notifies the signature framework on EndOfData() that there will be no further data to match for this body by giving it an empty buffer of length 0 with the eol parameter set to true and all others set to false. This lets it reach the '$' state in its DFA, and doesn't affect other documented HTTP match behaviours. - Limitation: Since the signature framework doesn't appear to keep previously consumed data on hand, any match of an http-*-body condition whose patterns ends with a '$' will lead to an empty data parameter being passed to the signature_match() event because the body data is no longer available when EndOfData() happens. Due to segmentation there is anyway no guarantee the data parameter would have held the entire match even without the '$', since the data parameter only receives the last chunk of data which completed the match condition, as can be seen on prefix matches in the btest cases where the matching data spans multiple segments (the event gives 'B' and not 'AB'), so this is only an extreme case of partial data being given to that event. --- src/analyzer/protocol/http/HTTP.cc | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/src/analyzer/protocol/http/HTTP.cc b/src/analyzer/protocol/http/HTTP.cc index 863c65ad02..4a0fd83201 100644 --- a/src/analyzer/protocol/http/HTTP.cc +++ b/src/analyzer/protocol/http/HTTP.cc @@ -69,6 +69,12 @@ void HTTP_Entity::EndOfData() { encoding = IDENTITY; } + zeek::detail::Rule::PatternType rule = + http_message->IsOrig() ? zeek::detail::Rule::HTTP_REQUEST_BODY : zeek::detail::Rule::HTTP_REPLY_BODY; + + http_message->MyHTTP_Analyzer()->Conn()->Match(rule, reinterpret_cast(""), 0, + http_message->IsOrig(), false, true, false); + if ( body_length ) http_message->MyHTTP_Analyzer()->ForwardEndOfData(http_message->IsOrig());