DDoS protection
Using Netfilter/iptables
Jesper Dangaard Brouer
Senior Kernel Engineer, Red Hat Network-Services-Team Dev on!"c# $e% &'()
1/36
Email* %rouer+red,at"com - netoptimi#er+%rouer"com - ,awk+kernel"org DDoS protection using Netfilter/iptables
.,o am /
Name* Jesper Dangaard Brouer
0inu1 Kernel Developer at Red Hat Edu*
omputer Science !or 2ni"
open,agen
$ocus on Network, Dist" s3s and 4S S3sadm, Kernel Developer, Em%edded
0inu1 user since (556, pro!essional since (557
4penSource pro8ects, aut,or o!
9DS0-optimi#er, :9N /:Ta%les**li%iptc, /:T;-9nal3#er 0inu1 kernel, iproute&, ipta%les, li%pcap and .ires,ark
:atc,es accepted into
4rgani#er o! Net!ilter .orks,op &'(<
DDoS protection using Netfilter/iptables
2/36
.,at will 3ou learn=
0inu1 Kernel is vulnera%le to simple S>N attacks End-,ost mitigation?s alread3 implemented in kernel
s,ow it is not enoug, solution is stalled """ ,ow to work-around t,is
Kernel* serious @listen@ socket scala%ilit3 pro%lem
$irewall-%ased solution* s3npro13 Aipta%les-net!ilterB How !ast is state!ul !irewalling
.,ere is our pain points 0earn Net!ilter tricks* %oost per!ormance a !actor ('
DDoS protection using Netfilter/iptables
3/36
$irst* Basic N/
tuning ('(
9ll tests in presentation Basic tuning
$irst kill CirD%alanceE N/ ,ardware Dueue, are :2 aligned
Disa%le Et,ernet !low-control
/ntel i1g%e ,w-driver issue
single %locked ,w Dueue %locks ot,ers
$i1 in kernel v<"F"' commit <e%e7!de%' Ai1g%e* Set DropGEN %it w,en multiple R1 Dueues are present w-o !low controlB
4/36
DDoS protection using Netfilter/iptables
$ocus* $looding DoS attack
Denial o! Service ADoSB attacks $ocus* T : !looding attacks
9ttacking t,e <-Wa3 HandS,ake A<.HSB End-,ost resource attack
S>N !lood S>N-9 K !loods 9 K !loods A<rd packet in <.HSB
9ttacker o!ten spoo!s src /:
T : S>N $looding 9ttacks and ommon Iitigations
Descri%ed in R$ )57H*
5/36
DDoS protection using Netfilter/iptables
0inu1 current end-,ost mitigations
Jargon R$ )57H AT : S>N $looding 9ttacks and ommon IitigationsB
0inu1 uses ,3%rid solution
S>N Ccac,eE Iini reDuest socket Iinimi#e state, dela3 !ull state alloc S>N C%acklogE o! outstanding reDuest sockets
9%ove limit, use S>N CcookiesE
6/36
DDoS protection using Netfilter/iptables
Details* S>N @cac,e@ savings
Small initial T B ATransmission ontrol BlockB struct reDuestGsock Asi#e F6 %3tesB
mini sock to represent a connection reDuest S09B %e,ind ,ave si#eo!Astruct tcpGreDuestGsockB
Structs em%edded in eac,-ot,er
But alloc si#e is ((& %3tes
F6 %3tes JJ struct reDuestGsock 7' %3tes JJ struct inetGreDuestGsock ((& %3tes JJ struct tcpGreDuestGsock
$ull T B Astruct inetGsockB is 7<& %3tes
Anote, si#es will increase-c,ange in more recent kernelsB
7/36
DDoS protection using Netfilter/iptables
Details* /ncreasing S>N %acklog
Not recommended to increase !or DoS
4nl3 increase, i! legitimate tra!!ic cause log*
CT :* :ossi%le S>N !looding """E
/ncreasing S>N %acklog is not o%vious
9d8ust all t,ese*
-proc-s3s-net-ipv)-tcpGma1Gs3nG%acklog -proc-s3s-net-core-soma1conn S3scall listenAint sock!d, int backlogBK
8/36
DDoS protection using Netfilter/iptables
S>N cookies
Simpli!ied description
S>N packet
don?t create an3 local state Encode state in SELM Aand T : optionsB ontains SELMN( Aand T : timestampB Recover state ;alidate A<.HSB 9 K packet state
S>N-9 K packet
9 K packet
SH9 ,as, is computed wit, local secret
9/36
DDoS protection using Netfilter/iptables
Details* S>N-cookies
S>N cookies SH9 calculation is e1pensive SNI: counters ASince kernel v<"(B
TCPReqQFullDoCookies * num%er o! times a S>N 44K/E was replied to client TCPReqQFullDrop * num%er o! times a S>N reDuest was dropped %ecause s3ncookies were not ena%led" -proc-s3s-net-ipv)-tcpGs3ncookies J &
9lwa3s on option
10/36
DDoS protection using Netfilter/iptables
So, w,at is t,e pro%lem=
Oood End-Host counter-measurements :ro%lem* 0/STEN state scala%ilit3 pro%lem
;ulnera%le !or all !loods
S>N, S>N-9 K and 9 K !loods
Num%ers* Peon :2 PFFF' ('O i1g%e
N4 0/STEN socket*
&"5')"(&7 pkts-sec -- S>N attack &F&"'<& pkts-sec -- S>N attack <<6"FH6 pkts-sec -- S>NN9 K attack <<("'H& pkts-sec -- 9 K attack
DDoS protection using Netfilter/iptables
0/STEN socket*
11/36
:ro%lem* S>N-cookie vs 0/STEN lock
Iain pro%lem*
S>N cookies live under 0/STEN lock
,ttp*--t,read"gmane"org-gmane"linu1"network-&<&&<7
/ proposed S>N %rownies !i1 AIa3 &'(&B
Oot re8ected, %ecause not general solution
e"g" don?t ,andle S>N-9 K and <.HS
Need to C!orward-portE patc,es
ABug ('FH<6) - R$E* :arallel S>N cookies ,andlingB
N$.S&'(< got clearance as a !irst step solution
12/36
DDoS protection using Netfilter/iptables
$irewall and :ro13 solutions
Net ork!"ase# ountermeasures
.esle3 I" Edd3, descri%es S>N-pro13
/n isco* T,e /nternet :rotocol Journal - ;olume 5, Num%er ), &''6, link* ,ttp*--goo"gl-9 (99Q 9vail in kernel <"(< and RHE0H
Net!ilter* ipta%les target S$NPR%&$
B3 :atrick IcHard3, Iartin Top,olm and Ie
9lso works on local,ost Oeneral solution Solves S>N and 9 K !loods /ndirect trick also solves S>NN9 K
DDoS protection using Netfilter/iptables
13/36
S>N pro13 concept
14/36
DDoS protection using Netfilter/iptables
onntrack per!ormanceA(B
S>N:R4P> needs conntrack
.ill t,at %e a per!ormance issue=
&"56)"'5( pkts-sec -- N4 0/STEN sock N no ipta%les rules &))"(&5 pkts-sec -- 0/STEN sock N no ipta%les rules )<F"F&' pkts-sec -- N4 0/STEN sock ' conntrack (H&"55& pkts-sec -- 0/STEN sock ' conntrack %ut / ,ave some tricks !or 3ou K-B
Base per!ormance*
0oading conntrack* AS>N !lood, causing new conntrackB
0ooks %ad"""
15/36
DDoS protection using Netfilter/iptables
onntrack per!ormanceA&B
onntrack Alock-lessB lookups are really fast
:ro%lem is insert and delete conntracks 2se to protect against S>NN9 K and 9 K attacks 9llow 9 K pkts to create new connection Disa%le via cmd*
sysctl -w net/netfilter/nf_conntrack_tcp_loose=0
De!ault net!ilter is in T : ClooseE mode
Take advantage o! state C/N;90/DE
Drop invalid pkts before reac,ing 0/STEN socket
iptables -m state --state INVALID -j DR !
16/36
DDoS protection using Netfilter/iptables
onntrack per!A<B 9 K-attacks
(C) attacks, conntrack per!ormance De!ault ClooseJ(E and pass /N;90/D pkts
(H5"'&H pkts-sec &<F"5') pkts-sec Alisten lock scalingB F"F<<"'F6 pkts-sec
0ooseJ' and and pass /N;90/D pkts
0ooseJ' and and DR4: /N;90/D pkts
17/36
DDoS protection using Netfilter/iptables
onntrack per!A)B S>N-9 K attack
S$N!(C) attacks, conntrack per!ormance
S>N-9 Ks don?t auto create connections T,us, c,anging ClooseE setting is not important &<'"<)7 pkts-sec F"<7&"&6F pkts-sec F")'7"<'H pkts-sec
De!ault pass /N;90/D pkts Aand ClooseJ(EB
De!ault DR4: /N;90/D pkts Aand ClooseJ(EB
De!ault DR4: /N;90/D pkts Aand ClooseJ'EB
18/36
DDoS protection using Netfilter/iptables
S3npro13 per!ormance
%nl* conntrack S$N attack proble+ left
Due to conntrack insert lock scaling &))"(&5 pkts-sec -- 0/STEN sock N no ipta%les rules (H&"55& pkts-sec -- 0/STEN sock ' conntrack
Base per!ormance*
0oading conntrack* AS>N !lood, causing new conntrackB
Using S$NPR%&$
,-./0-.,1 pkts-sec -- 0/STEN sock N s*npro2* N conntrack
19/36
DDoS protection using Netfilter/iptables
ipta%les* s3npro13 setupA(B
2sing S>N:R4P> target is complicated
S>N:R4P> works on untracked conntracks
/n CrawE ta%le, CnotrackE S>N packets*
iptables -t raw -I PREROUTING -i $DEV -p tcp -m tcp --syn \ --dport $PORT -j CT --notrack
20/36
DDoS protection using Netfilter/iptables
ipta%les* s3npro13 setupA&B
Iore strict conntrack ,andling
Need to get unknown 9 Ks A!rom <.HSB to %e marked as /N;90/D state
Aelse a conntrack is 8ust createdB
Done %3 s3sctl setting*
sbin s!sctl -" net net#ilter n#$conntrack$tcp$loose%&
21/36
DDoS protection using Netfilter/iptables
ipta%les* s3npro13 setupA<B
atc,ing state*
2NTR9 KED JJ S>N packets /N;90/D JJ 9 K !rom <.HS
2sing S>N:R4P> target*
iptables -' INPUT -i $DEV -p tcp -m tcp --dport $PORT ( -m state --state INV')ID*UNTR'C+ED ( -j SYNPROXY --sack-perm --timestamp --"scale , --mss -./&
22/36
DDoS protection using Netfilter/iptables
ipta%les* s3npro13 setupA)B
Trick to catc, S>N-9 K !loods
Drop rest o! state /N;90/D, contains S>N-9 K
iptables -' INPUT -i $DEV -p tcp -m tcp --dport $PORT ( -m state --state INV')ID -j DROP
Ena%le T : timestamping
Because S>N cookies uses T : options !ield
sbin s!sctl -" net ip0. tcp$timestamps%-
23/36
DDoS protection using Netfilter/iptables
ipta%les* s3npro13 setupAFB
onntrack entries tuning
Ia1 possi%le entries & Iill
&77 %3tes R & Iill J FH6"' IB
net net#ilter n#$conntrack$ma1%2&&&&&&
/I:4RT9NT* 9lso ad8ust ,as, %ucket si#e
-proc-s3s-net-net!ilter-n!GconntrackG%uckets writea%le via -s3s-module-n!Gconntrack-parameters-,as,si#e
Has, 7 %3tes R &Iill J (6 IB ec3o 2&&&&&& 4 s!s mod5le n#$conntrack parameters 3as3si6e
24/36
DDoS protection using Netfilter/iptables
:er!ormance S>N:R4P>
Script ipta%lesGs3npro13"s, avail ,ere*
,ttps*--git,u%"com-netoptimi#er-network-testing-%lo%-master-ipta%les-ip ta%lesGs3npro13"s,
2sing S>N:R4P> under attack t3pes*
&"765"7&) pkts-sec S S>N-!lood )"5)7")7' pkts-sec S 9 K-!lood F"6F<"(&' pkts-sec S S>NN9 K-!lood
25/36
DDoS protection using Netfilter/iptables
S>N:R4P> parameters
T,e parameters given to S>N:R4P> target
Iust matc, t,e %ackend-server T : options Ianual setup A,elper tool n!s3npro13B 4nl3 one setting per rule Not use!ul !or DH : %ased network 9uto detect server T : options Simpl3 allow !irst S>N t,roug,
Future plan
atc, S>N-9 K and decode options
ARHBQ ('F56H5 - R$E* S3npro13* auto detect T : optionsB
26/36
DDoS protection using Netfilter/iptables
Real-li!eA(B* Handle 5'' Kpps
27/36
DDoS protection using Netfilter/iptables
Real-li!eA&B* SH9 sum e1pensive
S>N cookie SH9 sum is e1pensive
Bug ('FH<F& - R$E* /mprove S>N cookies calculations
28/36
DDoS protection using Netfilter/iptables
Real-li!eA<B* 4ut tra!!ic normal
29/36
DDoS protection using Netfilter/iptables
/ssue* $ull connection scala%ilit3
Still e1ists* Scala%ilit3 issue wit, !ull conn
Iade it signi!icantl3 more e1pensive !or attackers
At,e3 need real ,ostsB
$uture work* !i1 scala%ilit3 !or
entral lock* 0/STEN socket lock entral lock* Net!ilter new conntracks A.ork-in-progressB
30/36
DDoS protection using Netfilter/iptables
$i1ing central conntrack lock
onntrack issue
/nsert - delete conntracks takes central lock .orking on removing t,is central lock
ABased on patc, !rom Eric Duma#etB
ARHBQ (')<'(& - @net!ilter* conntrack* remove t,e central spinlock@B
:reliminar3 results, S>N-!lood
No 0/STEN socket to leave out t,at issue
)<F"F&' pkts-sec S conntrack wit, central lock ("6&6"H76 pkts-sec S conntrack wit, parallel lock
31/36
DDoS protection using Netfilter/iptables
Hack* Iulti listen sockets
Hack to work-around 0/STEN socket lock
Simpl3 0/STEN on several ports 2se ipta%les to rewrite-DN9T to t,ese ports
32/36
DDoS protection using Netfilter/iptables
Hack* $ull conn ,as,limit trickA(B
:ro%lem* $ull connections still ,ave scala%ilit3 :artition /nternet in -&) su%nets
A(&7R&F6R&F6 - &'5H(F& J ) ma1 ,as, listB
0imit S>N packets e"g" &'' S>N pps per src su%net Iem usage* !airl3 ,ig,
$i1ed* ,ta%le-si#e &'5H(F& R 7 %3tes J (6"H IB ;aria%le* entr3 si#e (') %3tes R F''''' J F& IB
33/36
DDoS protection using Netfilter/iptables
Hack* $ull conn ,as,limit trickA&B
2sing ,as,limit as work-around
9ttacker needs man3 real ,osts, to reac, !ull conn scala%ilit3 limit
iptables -t ra" -' PREROUTING -i $DEV ( -p tcp -m tcp --dport 7& --s!n ( -m 3as3limit ( --3as3limit-abo0e 2&& sec --3as3limit-b5rst -&&& ( --3as3limit-mode srcip --3as3limit-name s!n ( --3as3limit-3table-si6e 2&8,-92 ( --3as3limit-srcmask 2. -j DROP
34/36
DDoS protection using Netfilter/iptables
9lternative usage o! @socket@ module
9void using conntrack
2se 1tGsocket module
$or local socket matc,ing an !ilter out <.HS-9 Ks Aand ot,er com%osB
:arameter --nowildcard :ro%lem can still %e invalid-!lood 9 Ks Iitigate %3 limiting e"g",as,limit
Didn?t scale as well as e1pected
,ttps*--git,u%"com-netoptimi#er-network-testing-%lo%-master-ipta%les-ipta%lesGloc alGsocketG,ack"s,
35/36
DDoS protection using Netfilter/iptables
T,e End
T,anks to Iartin Top,olm and 4ne"com
$or providing real-li!e attack data
,ttp*--people"net!ilter"org-,awk-presentations-devcon!&'()-
Download slides ,ere*
$eed%ack-rating o! talk on*
,ttp*--devcon!"c#-!-<H Luestions=
/! unlikel3Atime !or DuestionsB
36/36
DDoS protection using Netfilter/iptables
E1tra Slides
37/36
DDoS protection using Netfilter/iptables
Disa%le ,elper auto loading
De!ault is to auto load conntrack ,elpers
/t is a securit3 riskT
:oking ,oles in 3our !irewallT
Disa%le via cmd*
ec"o 0 # /proc/sys/net/netfilter/nf_conntrack_"elper
ontrolled con!ig e1ample*
iptables -t raw -p tcp -p $%$% -j &' --"elper ftp
Read guide ,ere*
,ttps*--,ome"regit"org-net!ilter-en-secure-use-o!-,elpers-
38/36
DDoS protection using Netfilter/iptables
