Teen 10 oF | you'll QD : MS 2, cee Ce uae tcpdump! A SMALL ° “ard HaNdBO0k FOR ANYON WHOWRITES (OR RUNSII) PROGRAMS ON Linux COMPUTERS BY: JULIA EVANSwhat's this? Hil This is me: Juua Evans blog: jyns. cad twitter:@bOrk and in this zine I want to tell you about how I qot better at debugging These are 5 ways I've changed how L think about debugging. j Remember the bug is happening fora logical reason. THs never magic. Really. Even when it makes no sense. 3 J {be confident L can fix it {ERD this Ewell Ive fixed before: (Be alot of hacd bugs before G Talk to my coworkers gEA know my debugging toolkit before: Now: ZI want to know S$ THING bet L don't Know how to Fiad out G most im 5) C5 : LT learned to like et before: fhe) about to leam SN facial expression: determination Some thing T can't teach you in 20 pages to y debugging (though Til try anywayt) LT can show you some of my debugging toolkit though | These are the tools L reach for when I have A question about a program I want to Know the answer to. By the end of this, L hope +o have given you a few new tools to use!Section L: I/O and x System calls & Hello, dear reader In this zine, there are 3% sections of tools that I love. For each tool, I'll tell yeu why it's useful and give an example. Each one is either i) Some of the most basic questions you might have when you log into a mis behaving machine are: - is this machine writing to ar reading $rom disk? The network? - are the programs reading Files? Which Files? So, we're starting with Finding out which resources are being used and what our programs are doing. Let's golNX — = tat | o dstat |= T love dstat because it's Super simple, Every second, ut prints out how much network and disk your computer used that second. @ Once Lhod an intermittent! slow database server. I opened up dstat and stared at the out put while monitoring database speed. I$ dstat_| send |recv during this period, sk everything is normal ° 300 MB DATARASE 43 MB GETS SLow é 3 back to norma Could 340MB coming in over the network Mean... a 30OMB database query}! ZyEsve This wos an AWESOME CLUE that helped us isolate the problem query,Vistrace} 17 (chove a SO ey sticker on™ P os) Ls my favourite program. Tt peints every system call your program used. Tt's & cool way to get an overall picture of what your PrOgram Ts doin ,and LY using ct to answer questions like “which Files are being apened? . ~ C 1g strace e python my_program. Py | A oe ceo open("/home/bork/ .config_file") = 3 gue read(3, “the contents of the file”) ~ hundeeds of lines. F (connects, "172.217.0.163") & \sendto(S, “hi!!") WARNING strace can make your program run $0x slower. Don't run it on your Production database TL can't do justice to strace here, but I have Q whole other zine aboutit at jvns.ca/zinesopensnoop ¥ eBPFX (kind of) When you run ‘Spensnoop p {PID | T will print out every File being opened by a program. You might think 20/7 Strace can do this tool Sust use strace -e open -P $PIO and you would be right. But Strace can make Uour program run 10x slower. Opensnoop won't slow you down, = how to get ite Requires : Ubuntu 16.64 + or a ~4.4+ Kernel version = how it works = Opensnoop is & script thot uses a new kernel Installation instructions at: Feels ae SeePr 3 : github. com / iovisor /bec. eBPFis Fas There's also an opensnoap on OSX& gspl That one is powered by DTrace. Shere are. lots of eBPE- powered Foals¥ Check out Shak GitHub cepo to \earn more!Wh OE fase, section 1:8 networking 8 “447 ps ot Tve devoted a lot of space inthis zine to networking tools, and I want to explain why. A \ot of the programs L work with communicate over HTTP. response request mm GET 7) Pan far {name traf” Ieots /42." program colour: “blue” Every programming language uses the same network protocols So the network iS a nice language -inde pendent place ta Answer questions like: * Was the request wrong , or was tt the response? aS my service even running ? *my program is slow. Whose Fault is that ? Let 's 96 vHTTP requests are fundamental really simple — they're gust text! To see that, let's make one by handY First, make a File: request. tat! _ i 1 \ { Host: ask.metofilter.com ' { User-Agent + zine 1 ( (2 new \ines! important |!) ‘ ioe —_—- sr Then: nc stands for netcat ! § cat request. txt nc metafilter.com 80 e back with a bunch etcat to send am \ ‘ You should get a respons: of HT™L | You can also use 1 huge Files over a local network quickly: step: (on target machine) step 2: (on the source) 4 eee ' $ hostname -L \ , . 1 192.168.2.132 «= 1! cat bigfile | \ ne 192.168. 2.132 91934} V ne -£ 9931 7 bigfile i} 's this \istens on the pot! this sends the data ) a“ Spo-z-& netstat feceive a request, a program (aka" server’) needs to be’ listening’ on the port. Finding out which Programs are Listening on which ports is really, easy . Its gest Every network request gets sent to oO port (like $0) ona computer. To xe < & ‘ tuna, please Y "ye whe og —— alse known os _ (Sudo netstat -tunapl Here's what you'll sees ~ proto —local_address PID / program name tcp 0.0.0.0 8353 2993 / python port ? So! L & netstat because it tells me which processes are cunning on which ports. On 0S X , use | lsof -i-P instead.narep ismy favourite starter network spy tool | Try it right now! Run: sudo ngrep -d any metafilter | \ Then go to http:// metafilter. com in youe browser. You should see matching network packets in ngrep’s output T We are SPIES Y Recently at work T’d made a chang @ To a client so that it sent {*some-id “>... 3 with all its requests. I wanted +o make sure iF was working, so TL ran: —_-— —_--T —_ “ ; sudo ngrep some-id Z - ~~ -—- > - ~~ — TI found out that everything was ok vbGe tcpdump u cpdump is the most difficult networking tool we'll discuss here and it took mea while to y ct. T use it to save network traffic to analyze later | “port 8977" is 7T7T actually ating sudo tcpdump poct 38997 \ Program inthe 1 -W service. | | “Berkeley Packet Lae ca Filter” (BPF) rT language. BPF oT *) isthe] | Filfers get compited a Pcap File’ C packet capture") ise and thes ron really, Standard for saving network troSFfic. Everything understands prap & fast { Some situations where Tl use tcpdump: *T'm Sending arequest to amachine and T want to know whether it's even getting there, ( _tepdump port & 1 will peint every packet on port 80) * Thave some slow network connections and T want +o know whether to blame the client or Server. (we'll also need wireshark!) * I just want to print aut packets +o see them (tcpdump -A)wireshack Wireshark isan GUL +too\ For network analysis. Here's an exercise ta Qearn itY Run this: : sudo tcpdump port 80 -w hitp.peap ' While that 5 running, open metafilter. com in your browser. Then press CtrltC to stop tepdump. Now we havea pcap File to analyze! joocc ene eae eee Explore the Wireshark interface | Questions You can try te answer: © What HTTP headers did your browser send to metafilter.cam? Chint: search | Frame contains “GET"}) @® How long did the longest request take? Chint: click Statistics > Conversations ) ® How many packets were exchanged pt with metafilter.com's servers? |e nang (hint: search Vip. dat. 2 etafilter.comsection 3: CPU * Your programs spend a lot of time on the CPU! Billions oF cycles. What are they, DOING @! This section is atcut using spect 3 to answer that question. pecf is a Linux -onlu tool that is extremely useful and not as well -Known as it should be. Cin general, mu aim in this zine is to showcase tools that I think don't get enough love ¢) Some things I didnt have space For in this section but wanted to mentian ANYWAY * ryalgrind «the Sava ecosystem's fantastic tools (stack, Visual VM, Youck:4) which your language is probably jealous of * Ftrace (For linux kernel tracing) * Ltt (ditto) * eBPF9 pert y perfis not simple or elegant. Tt is a weird multitool that does a few different, very useful things. First, it's a Sam Ving /_ = \profiler/ = Try Funning : rot Tt st -- a a '§ sudo pect record python : “(press GNC after a f ew Second s) You can look at the results with: ‘$ sudo pect ceport 4s it spent S% of itS time in the Py Dict - GetItem function. Cool! We learned a Tiny thing about the CPythen interpreter. [orks everywhere 7] if you use pecf to profile a Python program, itll Mine SQ pect can be installed an pretty much any Linux Show you the C functions machine. The exact (symbols) from the CPython Features it has will interpreter, not the depend on Your Python fyactions . kernel version.pect is for everyone One dav, I had a secver that was Using 100% of its CPU.Within about 60 seconds, L knew it was doing reqular expression matching in Ruby | How ? ‘pect top’ is like top, but for functions instead of programs . ror ttTtaTT 1% sudo pect top process PID % Function perf top doesn't always help. + Ruby's internal rea, exp i Bot it's easy to try, and sometimes "7, Matchi function » T learn something ~ +++ especially Sava and node devs ' Remember when I said perf only know S C functions? It's not quite true. node.)S and the SVM (java, scala, clojure...) have bath taught peck about their functions. = [node] > | Sava| 2 | Use the Look up ‘pecf-map-agent -- pecf-basic-prof on GitHub ang follow Commard line. option the directionsSe RAR + Flame faphs are an awesome Wau, to visualize CPU performance, popularized by Brendan Grega's Flameg raph -pl tool. v 2 Qith.com /bendangegg / Flame graph 0 9 Here's whot they lock like: [ore mo% | [teeth 28% Panda 20% | alligator 80% Main 100% thousands) of stack traces sampled from A program. The one above means 0% of the stack traces started with © 4948 10% with banda 4 eat You can construct them fram ‘perf “recarding 5 (see Brendan Gegg's Flame graph github foc how) but lots of other unrelated tools can produce them too. 9 them. Main 4 alligatorSpy on your CPU tip: Google “Latency Numbers every Programmer Should lenow " Your CPU has a small cache on it (the LL cache \ that it can access in ~O.S nanosecond s | faster than RAM \ Tf you're trying, to da an apecation iA microseconds, CPU cache usage matters | Spec; Pow to | | pef stat Is | G use it lp eee eon fre statis Ire This runs ‘Js’ and prints a cepoct at the end. how do I know if mi program is Using ‘Pose caches @ POSS -e request ae p how it | Your CPU can track all kinds of counters Aworks | About what it's doing. ipecf stat! asks i+ +6 count things (like U 1 i d report the results. cache misses) L Hardware is cool Tye never used perf stat in eaenest but I think i4's awesome you can get SO much infe From your CPU.T hope you leaned Some thing new. Thanks for reading yg Thanks to my pactner kamal for help reviewing and to the amazin Monica Dinculescu C@not waldorf) for the cover art. To learn more, see: x my, blog - JYNS.CQ. x my other zines: Jvns.ca/2ines x brendangregg . com But really You just need ta experiment. Try, these tools everywhere. See where they help you track down bugs and where they don't. oop Strace reals, obthat didn't helped with tell me much, that problem ¥ oh well f Tt takes practice, but 1 find these tools both fun and a useful job skill. Z hope you will too flike this ? there are more Zines at: http ://jvns.ca/zines CC-BY-NC-SA Svlia Evans , wizard debugging industries