lsof COVER STORY
Looking for intruders with lsof
QUICK CHECK
Track down and expose intruders with the versatile admin tool lsof.
BY CASPAR CLEMENS MIERAU
H
as your server been cracked? obtaining information that would other- has gotten comfortable. If you are seri-
Are your processes running wise require a whole collection of admin ous about using lsof for intrusion detec-
wild? If you suspect an intru- utilities. tion, leave out the make install step after
sion, you’ll need accurate information As the adage goes, “everything is a compiling and manually move the bi-
on what’s happening with the system. file” in Unix. Almost all activities on a nary to a write-protected medium such
Open file handles are a useful source Unix-like system bear some relation to as a CD ROM. Of course, if a sophisti-
for this information. lsof [1] scans the an open file. Unix-style systems use reg- cated attacker has directly modified the
depths of the filesystem for these files ular files, special block files, executables, kernel (through a kernel rootkit, for ex-
and then returns comprehensive and libraries, directo-
detailed output. ries, internal data
To be fully prepared for an attack, streams (Unix Do-
you’ll need an Intrusion Detection Sys- main Sockets),
tem (IDS) like Snort, Tripwire or Aide to and network con-
check the filesystem and data streams nections. lsof is
for suspicious patterns. However, if you able to centrally
don’t have the time or resources for a collect and syn-
full-blown intrusion response, Linux has thesize all this
a number of standard command line information into
programs capable of discovering tell-tale meaningful clues
traces on a system. The usual suspects about the nature
for server diagnosis are ps, netstat, top, of an attack.
fuser, and other friendly helpers. Like any utility,
lsof is a single tool that provides a lsof is subject to
summary of similar system information. manipulation Figure 1: The promising glsof tool gives GUI fans easy access to filter
You can use lsof as a single source for once the attacker settings.
W W W. L I N U X - M A G A Z I N E . C O M ISSUE 77 APRIL 2007 29
COVER STORY lsof
ample) the output of lsof will be unreli- You can manipulate the output for pro-
able even if the tool itself is untouched. cessing with other tools using lsof -F.
However, as you’ll learn in this article, Special formatting helps the downstream
many attackers try tricks that aren’t tools parse the individual fields (see the
especially sophisticated and are easily manpage section Output for other pro-
exposed with a tool like lsof. grams for details).
lsof is no substitute for a full-featured
IDS, but if you are too late for that or if Flood of Information
you aren’t interested in implementing Calling lsof without setting parameters
or managing a more comprehensive returns too much information to provide
system, you can still use lsof to look a useful overview – the flood of informa-
for footprints. tion would scare off many users. How- Figure 2: The Jlsof filter dialog tells you the
ever, command line parameters can help correct lsof parameters. Although this might
Investigations lsof concentrate on the data you need. If confuse GUI-only users, it does make it eas-
Table 1 lists a number of examples for you combine multiple parameters, lsof ier for users to move to the command line.
investigating a system. If you enable assumes a logical OR operation by de-
lsof’s security option, only root will re- fault; however, you can specify -a for an
ceive detailed output for these com- AND operation (last line in Table 1). However, this method is as drastic as it
mands. In secure mode, lsof will only Identifying processes that are prevent- is effective.
show users the details that directly affect ing users from unmounting a storage
them, however, even in insecure mode, medium is a typical task for lsof. Calling Expectations
lsof gives users without root privileges lsof with the -t directoryname option The commands listed in Table 1 are fine
fewer details, as you need root privileges returns a list of numeric process IDs for discovering important facts about a
to access the details in /proc. accessing the CD-ROM: system before or after an attack. To de-
lsof uses a tabular format to output fend yourself against invaders, you need
the information filtered as specified by $ umount /dev/cdrom to be familiar with the normal status,
the parameter list, including the follow- umount: /cdrom: device is busy and to be aware of where suspicious
ing columns by default: $ kill -9 `lsof -t /dev/cdrom` entries are likely to appear.
• Process name: COMMAND $ umount /dev/cdrom The following example uses a tradi-
• Process ID: PID $ eject tional LAMP system (Linux, Apache,
• Name of system user account under
which the process is running: USER Finding and Building lsof
• File descriptor: FD Lsof supports a number of Unix deriva- obtained from an insecure source), con-
• File type: TYPE tives, and it is probably part of your basic figure the source code, and compile it.
• Device: DEVICE Linux system, or at least it should reside During the configuration phase, you are
• Size: SIZE in the standard repository. To install on prompted to make a few decisions. The
• Connection: NODE Debian, for example, you just need to HASSECURITY and HASNOSOCKSECU-
• Full name: NAME issue a apt-get install lsof command. RITY options are important. If you would
The lean package has no dependencies, only like the root user to be able to use
apart from the mandatory Libc 6. lsof to list open files and sockets for all
Listing 1: Compiling lsof users, you need to answer [y] and [n].
This said, there are two reasons for
01 wget ftp://lsof.itap.purdue. The inconsistent terminology does tend
avoiding the prebuilt binary: system
edu/pub/tools/unix/lsof/lsof. compatibility and security. As lsof’s de- to be confusing.
tar.bz2 veloper Vic Abel points out in the FAQ On completing the build, ./lsof -v tells
02 tar xjf lsof.tar.bz2 [2], you can only guarantee a full feature you the options it was compiled with.
set and optimum stability, if you build The Only root can list all files message
03 cd lsof_4.77
the current lsof version on the target ma- means that normal users will be unable
04 wget ftp://lsof.itap.purdue. chine, since lsof digs deep into the sys- to misuse the program to list system-
edu/pub/Victor_A_Abell.gpg tem architecture and kernel. It is always critical information. (Restricting access
05 gpg --import Victor_A_Abell. better to obtain tools you will be using to lsof is a rather cosmetic security solu-
gpg for preventive or forensic analysis from tion, since much of the information avail-
a safe source and not to mix them with able through lsof can also be obtained
06 gpg --verify lsof_4.77_src. standard system tools to avoid the dan- with tools such as ps and netstat, al-
tar.sig lsof_4.77_src.tar ger of manipulation by rootkits. though the process may not be quite as
07 tar xf lsof_4.77_src.tar The lsof sources are easily compiled, convenient. The prebuilt versions in vari-
so you might as well build a version that ous distributions handle security differ-
08 cd lsof_4.77_src
matches your system. The commands ently. Debian grants non-administrative
09 ./Configure linux users unrestricted use of lsof, whereas
in Listing 1 grab the sources of the net-
10 make -s work; use GnuPG to check the signature Red Hat Enterprise applies restrictions.
11 ./lsof -v (note that the key in our example was
30 ISSUE 77 APRIL 2007 W W W. L I N U X - M A G A Z I N E . C O M
lsof COVER STORY
be able to open investigates the suspicious processes for
log files. In con- details of network connections, libraries
trast to this, data that have been loaded, open files, and
communications many other things.
are handled by As malevolent hackers tend to use
unprivileged pro- their own FTP, IRC, telnet, or SSH serv-
cesses. Thus, a ers, initial analysis should include
web server offers searching for open ports. The lsof -a -i -u
a guessable con- www-data | grep LISTEN command lists
figuration of users, all the IP sockets (-i), which sockets the
executable files, Apache user has opened, (-u www-data),
and open ports. and which are listening for connections
For example, (this explains grep LISTEN).
Figure 3: The Java-based JLsof tool converts lsof output to a simple www-data runs Everything apart from 80 (HTTP) and
table. /usr/sbin/apache2 443 (HTTPS) is suspicious. Although a
on Debian. call to netstat will give you similar
MySQL, PHP). The administrator notices results, lsof can help you perform more
an enormous increase in network load What’s Running Where? detailed analysis without needing to
that doesn’t reflect the number of page In this context, we need a call to lsof -a switch to another tool.
hits. The administrator suspects that an -d txt -u www-data to list processes that
attacker has injected a trojan that copies execute the file /usr/sbin/apache2 as the The Real World
files over the wire, launches distributed www-data user account. The -a option Apache and PHP exploits are fairly com-
network attacks or sends spam mail. In gives us a logical AND, -d txt lists exe- mon. Listings 2a and 2b show two ex-
a LAMP environment, the PHP system cuted files only, and -u www-data re- cerpts from the lsof logs on compro-
interface is one of the major targets, as stricts the output to just one user. Under mised servers, and they are all I need to
PHP suffers from a couple of design normal circumstances, this will give you diagnose an attack. The output results
weaknesses [6], but poorly crafted just the Apache processes. from analyzing processes belonging to
scripts can just as easily give an attacker If an attacker manages to manipulate the www-data account. See Listing 3 for
a foothold. PHP or your PHP scripts and execute another abridged example.
If you are familiar with typical PHP system commands and programs on the In the first example, the attacker
attack patterns, you will probably al- server, these commands and processes exploits an obsolete version of W-Agora
ready have guessed what kind of infor- will typically run under the same ac- (online forum software) and a directory
mation you need to look for with lsof. count as Apache – that is, unless the without write protection (Listing 2a,
The Apache web server runs under its attacker has escalated his privileges and Line 2: /home/user/public_html/
own user account by default, www-data gained root access by exploiting other w-agora/).
(Debian), apache, httpd, or if the worst security holes.
comes to the worst, nobody (this ac- Finding processes that belong to the Listing 2a: Bash
count is normally reserved for NFS). Apache user and that also access other Camouflage
Typically, additional processes will run binaries or open unexpected ports,
01 COMMAND PID USER FD
as root to support privileged ports and to should set off the alarms. lsof -p PID
TYPE DEVICE SIZE NODE
NAME
Table 1: lsof Examples
02 bash 30334 www-data cwd
Command Explanation
DIR 3,8 4096 1571340 /
lsof Without any parameters, the command gives you an overview.
home/user/public_html/
lsof /bin/bash Lists all processes that use bash.
lsof -p PID Lists the open files for the process with the specified process ID. w-agora/.m
lsof +D /tmp Lists all open files in /tmp and its subdirectories without symbolic links. 03 bash 30334 www-data txt
lsof -u Benutzer Lists all open files for the specified user. REG 3,8 496231 1571405 /
lsof -u ^root Lists all open files, except for those opened by root. home/user/public_html/
lsof -d txt Displays a process list, similar to ps aux, by listing entries with the file w-agora/.m/bash
descriptor entry txt, instead of the normal number (txt refers to program
code and data, that is, for executed files). 04 bash 30334 www-data 0w
lsof +L1 Displays all deleted files that are still open, and thus still occupy disk REG 3,8 125 1571408 /
space, but are not part of any directory (files with less than one link). home/user/public_html/
lsof -i Network-related files. w-agora/.m/LinkEvents
lsof -i -P -n All network-related files without the port number as a service identifier,
05 bash 30334 www-data 2u
and without resolving hostnames (for faster response).
IPv4 4709341 TCP
lsof -i6 Shows IPv6-related files.
server.com:40001->undernet.
lsof -i | grep ‘\->‘ All active connections.
lsof -a -i -u www-data All open network files for the www-data account (AND relation -a). xs4all.nl:ircd ESTABLISHED)
W W W. L I N U X - M A G A Z I N E . C O M ISSUE 77 APRIL 2007 31
COVER STORY lsof
The attacker has created a new direc- count has a num-
tory .m to use as a working directory ber of open ports,
(Line 2, Column FD: Current Working including on the
Directory). The attacker has uploaded C Psybnc IRC proxy.
files to the directory and then compiled The unique pro-
and executed the files using a harmless- cess name of
sounding account name of bash. psybnc (Lines 5
However, as you can see in Line 5, the through 8) is a
programs are not as harmless as the real give-away,
name might suggest; this bash has an but at least there
open connection to an IRC server. Plus, is an attempt to
bash has written data to the LinkEvents hide the processes
file, which is obvious from the file de- behind a familiar
scriptor 0w (that is, bash has opened name – as the
stdout for writing). name server bind
in Line 9.
Cheeky but Dumb In fact, this is a
Our cyber criminal is really cheeky, but patched SSH
the attacker’s methods reveal more self- server that grants Figure 4: Sloth provides a native lsof interface for Mac OS X.
confidence than technical ability – espe- system access to
cially considering the fact that he has www-data without requiring a password. lsof, it makes sense to monitor a list of
not bothered to cover his tracks. Hiding There also is a server process with the open ports, adding process names, user-
the directory by starting the directory suspicious name of a (see Lines 2 names, and interfaces.
with a dot and using bash as the account through 4). The command shown in Line 1 of List-
name for the processes, are both begin- ing 3 handles the first part of this task in
ner’s tricks. Automatation
In the second example (Listing 2b), You may have the need for a script that Listing 2b: Injecting an IRC
the attacker has found a similar security compares a known system status with Proxy
hole and installed several applications. the current status and responds in a pre-
01 COMMAND PID USER FD
Again, the attacker has not taken the defined way in case of deviations – that
TYPE DEVICE SIZE NODE NAME
trouble to cover up; the www-data ac- is, an anomaly detection system. With
02 a 10555 www-data 266u
IPv4 2808 TCP *:
GUI Tools
https (LISTEN)
GUI fans may find it hard to locate a time when it was called. As this ap-
graphical front-end for lsof. The Libg- proach is not event-based, glsof can 03 a 10555 www-data 267u
nome-based glsof tool [3] is fairly new, easily overlook short-term access. IPv4 2809 TCP *:www
and its developers are still extremely The fairly ancient Java front-end, JLsof (LISTEN)
busy, although they have not made it [4], which has not been updated since 04 a 10555 www-data 543u
past the alpha stage thus far. The release 2003, has less in the way of functionality IPv4 757852768 TCP
cycles are fairly short, so you might like than glsof, but it also has fewer depen-
to download the latest version from the *:9713 (LISTEN)
dencies. To install JLsof, you need to
Subversion repository. This actually download and unpack the archive. You 05 psybnc 10615 www-data 266u
turned out to be the only way to get may need to modify the path to the Java IPv4 2808 TCP *:
things working in our lab. The glsof interpreter and to lsof in the jlsof start https (LISTEN)
homepage has the usual howtos, and script. JLsof has a far more spartan look
the developers will answer your email if 06 psybnc 10615 www-data 267u
than glsof, but it does show you how
you get stuck. IPv4 2809 TCP *:www
your filter settings resolve to lsof com-
Glsof gives users the ability to set filters mand line parameters (Figure 2), which (LISTEN)
by pointing and clicking (Figure 1) and to is a good thing if you are trying to under- 07 psybnc 10615 www-data 543u
store the settings so that they can run stand the filter rules. Although you can’t IPv4 757871322 TCP *:
the same queries later. Filters support actually store filters, JLsof will export the ircd (LISTEN)
fairly complex rulesets, which you can output (Figure 3) to an XML document.
08 psybnc 10615 www-data 549u
view and analyze in the query debug- If you use a Mac, there is no need to do
ging window. Thus, glsof considerably IPv4 762054917 TCP
without a native lsof GUI. Sloth [5],
shortens the learning curve for lsof new- server.com:35614->oslo1.
which was written in Objective C, scores
comers. The ability to set up file moni- with a nicely organized interface (Figure no.eu.undernet.org:ircd
tors in glsof, to watch freely definable re- 4) that offers predefined filters catego- (ESTABLISHED)
sources, and to notify administrators in rized by resource type and the ability to 09 bind 22004 www-data 543u
case of access is also useful. Under the terminate processes by clicking with the IPv4 696149859 TCP
hood, all glsof does is repeatedly call mouse (kill).
lsof, which lists access for the point in *:1982 (LISTEN)
32 ISSUE 77 APRIL 2007 W W W. L I N U X - M A G A Z I N E . C O M
lsof COVER STORY
launching the program,
Listing 3: Open TCP Ports is characteristic for the
Read initial
01 $ lsof -i TCP -n -P | awk '/ web server.
port assignments
LISTEN/ {print $1"/"$3"/"$8}'
| sort -u Do-It-Yourself
02 apache/root/*:443
IDS
The miniature lsof-based
03 apache/root/*:80
IDS in Listing 3 works Read current Current port
04 apache/www-data/*:443 as depicted in Figure 5. port assignments assignment be-
05 apache/www-data/*:80 When launched, the comes new standard
06 mysqld/mysql/127.0.0.1:3306 script remembers (List-
ing 4, Lines 4 through 8)
07 sshd/root/*:22
the current port configu-
ration. Every 10 seconds,
No port Email
an elegant way. It tells lsof to output net- it calls lsof to fetch the
assignment to
work-related files (-i) without writing list of open ports and
No changed? Yes administrator
out the port numbers as service names compares the list with
(-P) and without resolving IP addresses the last known status
to hostnames (-n). Awk checks the out- (Line 12). If a change Figure 5: The script shown in Listing 4 alerts the administra-
put for listening ports (LISTEN status) occurs, the script mails tor whenever a change occurs in the port assignments. To do
and formats the output as: username/ the before/after status so, it compares the original status with the current status
processname/IP:Port, where an IP ad- (Lines 14 through 0) and every 10 seconds.
dress of * stands for a server that listens uses the new status for
to all interfaces. further comparisons (Line 22). certain circumstances, processes like this
The final sort organizes the output in To test your do-it-yourself anomaly de- can cause false alerts, although it's fairly
alphabetical order, and -u ensures that tection system, you might like to tempo- easy to avoid by modifying the logic be-
each combination of user, process, and rarily open a port. Netcat offers an easy hind the query. Adding | grep -v tempo-
service occurs only once. way to do so. Give a command like nc rary service in Line 5 should do the trick.
The output shown in Line 2 of Listing -l -p 12345 to launch Netcat in LISTEN
3 was taken from a Debian Sarge server mode (-l) and keep port 12345 open. Conclusion
with Apache 1.3, MySQL, and an SSH Within 10 seconds, the shell script in the A simple shell script can’t hope to re-
daemon. In our example, MySQL only infinite loop should have noticed the sta- place a full-fledged IDS, but if you’re
binds to the local interface (Line 6), tus change and responded accordingly. looking for a no-frills detection tool or
while Apache and SSH are accessible via Be aware that some processes change an extra line of defense, lsof could be
any interface. lsof’s view of the port assignments. For a part of the solution. Useful additions
The grouping of the Apache processes example, some email servers fork addi- might be cryptography-based configura-
in root and www-data, which results tional processes, depending on the sta- tion management in the style of Aide,
from dropping root privileges after tus of the incoming connection. Under checks on executed files, evaluation of
the UDP configuration, and many other
Listing 4: Port Monitoring things. Repeated calls to lsof can also
01 #!/bin/bash 13 echo "Port assignments open up new fields of application, as evi-
changed! Notify administrator denced by the file monitor options in
02 MAILTO="root"
by email" Glsof. Whether you script with lsof or
03 HOSTNAME=`hostname`
use it as a fast, universal administration
14 mail -s "Attention:
04 getports() { tool, lsof is an easy, if limited, tool for
$HOSTNAME LISTEN status
05 lsof -i -n -P | awk '/LISTEN/ spotting intruders. ■
changed" $MAILTO <<EOF
{print $1"/"$3"/"$8}' | sort
15 Status prior to change:
-u INFO
16 $OLD
06 } [1] lsof homepage on Freshmeat:
17 http://freshmeat.net/projects/lsof/
07
18 Status after change: [2] lsof FAQ: ftp://lsof.itap.purdue.edu/
08 OLD="$(getports)" pub/tools/unix/lsof/FAQ
19 $NEW
09 echo -e "Start with following [3] Glsof: http://glsof.sourceforge.net
port assignments:\n$OLD" 20 EOF [4] JLsof: http://www.geocities.co.jp/Sili-
21 fi conValley/1596/jlsof/readme.html
10 while sleep 10 ; do
22 OLD="$NEW" [5] Sloth:
11 NEW="$(getports)" http://www.sveinbjorn.org/sloth/
12 if test "$OLD" != "$NEW" ; 23 done
[6] Hardened PHP:
then http://www.hardened-php.net
W W W. L I N U X - M A G A Z I N E . C O M ISSUE 77 APRIL 2007 33
