Simple Filter Commands

A filter is a program that reads standard input, performs an operation upon it and writes
the results to standard output.
For this reason, it can be used to process information in powerful ways such as
restructuring output to generate useful reports, modifying text in files and many other
system administration tasks.

With that said, below are some of the useful file or text filters in Linux.

awk Command
Awk is a remarkable pattern scanning and processing language, it can be used to build
useful filters in Linux.

sed Command
sed is a powerful stream editor for filtering and transforming text. We’ve already written
a two useful articles on sed, that you can go through it here:
The sed man page has added control options and instructions:
$ man sed

grep, egrep, fgrep, rgrep commands

These filters output lines matching a given pattern. They read lines from a file or standard
input, and print all matching lines by default to standard output.

Note: The main program is grep, the variations are simply the same as using specific grep
options as below (and they are still being used for backward compatibility):
$ egrep = grep -E
$ fgrep = grep -F
$ rgrep = grep -r

Below are some basic grep commands:

tecmint@TecMint ~ $ grep "aaronkilik" /etc/passwd
aaronkilik:x:1001:1001::/home/aaronkilik:

tecmint@TecMint ~ $ cat /etc/passwd | grep "aronkilik"

aaronkilik:x:1001:1001::/home/aaronkilik:

head Command
head is used to display the first parts of a file, it outputs the first 10 lines by default. You
can use the -n num flag to specify the number of lines to be displayed:
tecmint@TecMint ~ $ head /var/log/auth.log
Jan 2 10:45:01 TecMint CRON[3383]: pam_unix(cron:session): session opened for user root by (uid=0)
Jan 2 10:45:01 TecMint CRON[3383]: pam_unix(cron:session): session closed for user root
Jan 2 10:51:34 TecMint sudo: tecmint : TTY=unknown ; PWD=/home/tecmint ; USER=root ;
COMMAND=/usr/lib/linuxmint/mintUpdate/checkAPT.py
Jan 2 10:51:34 TecMint sudo: pam_unix(sudo:session): session opened for user root by (uid=0)
Jan 2 10:51:39 TecMint sudo: pam_unix(sudo:session): session closed for user root
Jan 2 10:55:01 TecMint CRON[4099]: pam_unix(cron:session): session opened for user root by (uid=0)
Jan 2 10:55:01 TecMint CRON[4099]: pam_unix(cron:session): session closed for user root
Jan 2 11:05:01 TecMint CRON[4138]: pam_unix(cron:session): session opened for user root by (uid=0)
Jan 2 11:05:01 TecMint CRON[4138]: pam_unix(cron:session): session closed for user root
Jan 2 11:09:01 TecMint CRON[4146]: pam_unix(cron:session): session opened for user root by (uid=0)

tecmint@TecMint ~ $ head -n 5 /var/log/auth.log

Jan 2 10:45:01 TecMint CRON[3383]: pam_unix(cron:session): session opened for user root by (uid=0)
Jan 2 10:45:01 TecMint CRON[3383]: pam_unix(cron:session): session closed for user root
Jan 2 10:51:34 TecMint sudo: tecmint : TTY=unknown ; PWD=/home/tecmint ; USER=root ;
COMMAND=/usr/lib/linuxmint/mintUpdate/checkAPT.py
Jan 2 10:51:34 TecMint sudo: pam_unix(sudo:session): session opened for user root by (uid=0)
Jan 2 10:51:39 TecMint sudo: pam_unix(sudo:session): session closed for user root

tail Command
tail outputs the last parts (10 lines by default) of a file. Use the -n num switch to specify
the number of lines to be displayed.
The command below will output the last 5 lines of the specified file:
tecmint@TecMint ~ $ tail -n 5 /var/log/auth.log
Jan 6 13:01:27 TecMint sshd[1269]: Server listening on 0.0.0.0 port 22.
Jan 6 13:01:27 TecMint sshd[1269]: Server listening on :: port 22.
Jan 6 13:01:27 TecMint sshd[1269]: Received SIGHUP; restarting.
Jan 6 13:01:27 TecMint sshd[1269]: Server listening on 0.0.0.0 port 22.
Jan 6 13:01:27 TecMint sshd[1269]: Server listening on :: port 22.

The following command will enable you monitor changes in the specified file:
tecmint@TecMint ~ $ tail -f /var/log/auth.log
Jan 6 12:58:01 TecMint sshd[1269]: Server listening on :: port 22.
Jan 6 12:58:11 TecMint sshd[1269]: Received SIGHUP; restarting.
Jan 6 12:58:12 TecMint sshd[1269]: Server listening on 0.0.0.0 port 22.
Jan 6 12:58:12 TecMint sshd[1269]: Server listening on :: port 22.
Jan 6 13:01:27 TecMint sshd[1269]: Received SIGHUP; restarting.
Jan 6 13:01:27 TecMint sshd[1269]: Server listening on 0.0.0.0 port 22.
Jan 6 13:01:27 TecMint sshd[1269]: Server listening on :: port 22.
Jan 6 13:01:27 TecMint sshd[1269]: Received SIGHUP; restarting.
Jan 6 13:01:27 TecMint sshd[1269]: Server listening on 0.0.0.0 port 22.
Jan 6 13:01:27 TecMint sshd[1269]: Server listening on :: port 22.

Read through the tail man page for a complete list of usage options and instructions:
$ man tail
sort Command
sort is used to sort lines of a text file or from standard input.
Below is the content of a file named domains.list:
tecmint@TecMint ~ $ cat domains.list
tecmint.com
tecmint.com
news.tecmint.com
news.tecmint.com
linuxsay.com
linuxsay.com
windowsmint.com
windowsmint.com

You can run a simple sort command to sort the file content like so:
tecmint@TecMint ~ $ sort domains.list
linuxsay.com
linuxsay.com
news.tecmint.com
news.tecmint.com
tecmint.com
tecmint.com
windowsmint.com
windowsmint.com

uniq Command
uniq command is used to report or omit repeated lines, it filters lines from standard input
and writes the outcome to standard output.
After running sort on an input stream, you can remove repeated lines with uniq as in the
example below.

To indicate the number of occurrences of a line, use the -c option and ignore differences
in case while comparing by including the -i option:
tecmint@TecMint ~ $ cat domains.list
tecmint.com
tecmint.com
news.tecmint.com
news.tecmint.com
linuxsay.com
linuxsay.com
windowsmint.com
tecmint@TecMint ~ $ sort domains.list | uniq -c
2 linuxsay.com
2 news.tecmint.com
2 tecmint.com
1 windowsmint.com
Read through the uniq man page for further usage info and flags:
$ man uniq

fmt Command
fmt simple optimal text formatter, it reformats paragraphs in specified file and prints
results to the standard output.
The following is the content extracted from the file domain-list.txt:
1.tecmint.com 2.news.tecmint.com 3.linuxsay.com 4.windowsmint.com
To reformat the above content to a standard list, run the following command with -
w switch is used to define the maximum line width:
tecmint@TecMint ~ $ cat domain-list.txt
1.tecmint.com 2.news.tecmint.com 3.linuxsay.com 4.windowsmint.com

tecmint@TecMint ~ $ fmt -w 1 domain-list.txt

1.tecmint.com
2.news.tecmint.com
3.linuxsay.com
4.windowsmint.com

pr Command
pr command converts text files or standard input for printing. For instance
on Debian systems, you can list all installed packages as follows:
$ dpkg -l
To organize the list in pages and columns ready for printing, issue the following
command.
tecmint@TecMint ~ $ dpkg -l | pr --columns 3 -l 20

2017-01-06 13:19 Page 1

Desired=Unknown/Install ii adduser ii apg

| Status=Not/Inst/Conf- ii adwaita-icon-theme ii app-install-data
|/ Err?=(none)/Reinst-r ii adwaita-icon-theme- ii apparmor
||/ Name ii alsa-base ii apt
+++-=================== ii alsa-utils ii apt-clone
ii accountsservice ii anacron ii apt-transport-https
ii acl ii apache2 ii apt-utils
ii acpi-support ii apache2-bin ii apt-xapian-index
ii acpid ii apache2-data ii aptdaemon
ii add-apt-key ii apache2-utils ii aptdaemon-data

2017-01-06 13:19 Page 2

ii aptitude ii avahi-daemon ii bind9-host

ii aptitude-common ii avahi-utils ii binfmt-support
ii apturl ii aview ii binutils
ii apturl-common ii banshee ii bison
ii archdetect-deb ii baobab ii blt
ii aspell ii base-files ii blueberry
ii aspell-en ii base-passwd ii bluetooth
ii at-spi2-core ii bash ii bluez
ii attr ii bash-completion ii bluez-cups
ii avahi-autoipd ii bc ii bluez-obexd

.....
The flags used here are:
--column defines number of columns created in the output.
-l specifies page length (default is 66 lines).

tr Command
This tool translates or deletes characters from standard input and writes results to
standard output.
The syntax for using tr is as follows:
$ tr options set1 set2
Take a look at the examples below, in the first command, set1( [:upper:] ) represents the
case of input characters (all upper case).
Then set2([:lower:]) represents the case in which the resultant characters will be. It’s same
thing in the second example and the escape sequence \n means print output on a new
line:
tecmint@TecMint ~ $ echo "WWW.TECMINT.COM" | tr [:upper:] [:lower:]
www.tecmint.com
tecmint@TecMint ~ $ echo "news.tecmint.com" | tr [:lower:] [:upper:]
NEWS.TECMINT.COM

more Command
more command is a useful file perusal filter created basically for certificate viewing. It
shows file content in a page like format, where users can press [Enter] to view more
information.

You can use it to view large files like so:

tecmint@TecMint ~ $ dmesg | more
[ 0.000000] Initializing cgroup subsys cpuset
[ 0.000000] Initializing cgroup subsys cpu
[ 0.000000] Initializing cgroup subsys cpuacct
[ 0.000000] Linux version 4.4.0-21-generic (buildd@lgw01-21) (gcc version 5.3.1 20160413 (Ubuntu 5.3.1-14ubuntu2)
) #37-Ubuntu SMP Mon Apr 18 18:33:37 UTC 2016 (Ubuntu 4.4.0-21.37-generic
4.4.6)
[ 0.000000] Command line: BOOT_IMAGE=/boot/vmlinuz-4.4.0-21-generic root=UUID=bb29dda3-bdaa-4b39-86cf-
4a6dc9634a1b ro quiet splash vt.handoff=7
[ 0.000000] KERNEL supported cpus:
[ 0.000000] Intel GenuineIntel
[ 0.000000] AMD AuthenticAMD
[ 0.000000] Centaur CentaurHauls
[ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256
[ 0.000000] x86/fpu: Supporting XSAVE feature 0x01: 'x87 floating point registers'
[ 0.000000] x86/fpu: Supporting XSAVE feature 0x02: 'SSE registers'
[ 0.000000] x86/fpu: Supporting XSAVE feature 0x04: 'AVX registers'
[ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format.
[ 0.000000] x86/fpu: Using 'eager' FPU context switches.
[ 0.000000] e820: BIOS-provided physical RAM map:
[ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009d3ff] usable
[ 0.000000] BIOS-e820: [mem 0x000000000009d400-0x000000000009ffff] reserved
[ 0.000000] BIOS-e820: [mem 0x00000000000e0000-0x00000000000fffff] reserved
[ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000a56affff] usable
[ 0.000000] BIOS-e820: [mem 0x00000000a56b0000-0x00000000a5eaffff] reserved
[ 0.000000] BIOS-e820: [mem 0x00000000a5eb0000-0x00000000aaabefff] usable
--More--

less Command
less is the opposite of more command above but it offers extra features and it’s a little
faster with large files.

Use it in the same way as more:

tecmint@TecMint ~ $ dmesg | less
[ 0.000000] Initializing cgroup subsys cpuset
[ 0.000000] Initializing cgroup subsys cpu
[ 0.000000] Initializing cgroup subsys cpuacct
[ 0.000000] Linux version 4.4.0-21-generic (buildd@lgw01-21) (gcc version 5.3.1 20160413 (Ubuntu 5.3.1-14ubuntu2)
) #37-Ubuntu SMP Mon Apr 18 18:33:37 UTC 2016 (Ubuntu 4.4.0-21.37-generic
 4.4.6)
[ 0.000000] Command line: BOOT_IMAGE=/boot/vmlinuz-4.4.0-21-generic root=UUID=bb29dda3-bdaa-4b39-86cf-
4a6dc9634a1b ro quiet splash vt.handoff=7
[ 0.000000] KERNEL supported cpus:
[ 0.000000] Intel GenuineIntel
[ 0.000000] AMD AuthenticAMD
[ 0.000000] Centaur CentaurHauls
[ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256
[ 0.000000] x86/fpu: Supporting XSAVE feature 0x01: 'x87 floating point registers'
[ 0.000000] x86/fpu: Supporting XSAVE feature 0x02: 'SSE registers'
[ 0.000000] x86/fpu: Supporting XSAVE feature 0x04: 'AVX registers'
[ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format.
[ 0.000000] x86/fpu: Using 'eager' FPU context switches.
[ 0.000000] e820: BIOS-provided physical RAM map:
[ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009d3ff] usable
[ 0.000000] BIOS-e820: [mem 0x000000000009d400-0x000000000009ffff] reserved
[ 0.000000] BIOS-e820: [mem 0x00000000000e0000-0x00000000000fffff] reserved
[ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000a56affff] usable
[ 0.000000] BIOS-e820: [mem 0x00000000a56b0000-0x00000000a5eaffff] reserved
[ 0.000000] BIOS-e820: [mem 0x00000000a5eb0000-0x00000000aaabefff] usable
:
Understanding Various Servers
Servers are important in the industry and the field of computers. Servers are an important
component in networks. Linux, like any system, supports networking and can be used as
a server. There are numerous server types. A variety of server types exist due to the many
kinds of needs clients and networks may require. Linux supports all of them.

Some of the most popular distros to use as a Linux server include Ubuntu, RedHat,
CentOS, OpenSuse, Mandriva, Xandros, Debian, and others. No matter which distro is
chosen, there are many server types that an admin may choose based on the network's
needs. Also, a single Linux machine can be multiple server types at once.

NOTE: CentOS is RedHat without the paid support and any proprietary software.

Application Server - A server that has the sole purpose of running some type of software.

Cache Server – A cache server stores previously visited webpages. A network in a business
may query the cache server for a webpage. If the webpage is not cached on the server,
then the server will get the webpage and send it to the client. Cache servers can help
increase perceived Internet speed since previously accessed pages are cached on the local
network. Squid (http://www.squid-cache.org/) is an example of a cache server.

Database Server - A database server provides databases services to clients. The server
has some type of database or multiple databases (SQL, JSON, etc.). Some databases server
software includes Ingres, PostgreSQL, MySQL, Informix, and many others. Some database
server software is cross-platform and others can only be installed on certain systems.
Database server may be used to track and store information such as accounting, banking,
products/merchandise data, sales, etc.

DHCP Server - DHCP servers assign IP addresses to clients on the network. This saves
time since admins and users will not need to manually configure their IP addresses.
DNS Server - DNS servers allow Domain Names to be resolved to IP addresses. Examples
of DNS server software includes BIND and djbdns. DNS servers can be public (like the
ones used for the Internet) or private (like within a company network).

Fax Server - Offers fax services to clients. Fax servers are useful on networks with many
fax requests. HylaFAX (http://www.hylafax.org) is a specific example of software used by
Linux to become a fax server. HylaFax is capable of using Fax over IP (FoIP). To install
HylaFAX on Debian-based distros, type apt-get install hylafax-server. If it is not found,
then a repo/PPA may need to be added.
File Server - Provides files. Many servers match this description such as FTP, HTTP, Samba,
and other servers. Some FTP server include ProFTPD, Pure-FTPd, and vsftpd.

Game Server - Some games (like Minetest) can be hosted on a server. Such servers
provide a way for many client (players) to join the same game.
Intrusion Detection Systems (IDSs) - These servers monitor networks for malicious activity
and log such events.

POP3 and IMAP (incoming mail) - Servers that receive mail (mainly to be sent to the
recipient client/user) may use the POP3 or IMAP protocol. Some examples of outgoing
mail servers include qpopper, UW IMAP, Courier-IMAP, and others.

Print Server - A print server provides print services to clients. Such print servers may be
real/physical such as a hardware printer or virtual like a PDF-printer. Samba servers are an
example of a printer server that serves Linux and Windows clients.

LDAP Server - Provides LDAP services. LDAP servers offer information directories. This
allows a user to login to a network once and then have access to all of the resources.

Monitoring Server - Some servers monitor a network for certain activity. For instance, a
Multi Router Traffic Grapher (MRTG) server keeps statistics concerning network traffic.
SNMP (Simple Network Management Protocol) servers watch networks for important
events that may require an admin's attention.

NTP Server - NTP servers synchronize the time/clocks on the networks. This is important
because some network services or processing may be dependent on precise time. Also,
when viewing logs for issues, it may be important to know specifically where the error
occurred first. If some of the computers have their time off by a few seconds, this can
make it difficult to track the issue. For instance, if a client and a server crash a few seconds
apart from each other, but the time is not synchronized, then how will the admin know
which crash caused the other system to crash? On very large networks that span across a
country or the world will need to be synchronized, especially if the data is to be synced
with the newest data. "ntpd" is a popular NTP server daemon for Linux and other Unixoid
systems. An alternative to the NTP protocol is SNTP (Simple Network Time Protocol) which
is a more lightweight and simplified NTP.

Proxy Server – Proxy servers get data and send requests on behalf of the client. Proxies
an be used to protect privacy, control content, log client activity, data caching, etc. Cache
servers are a special form of proxy server. Some proxy server software includes Polipo,
Varnish, Nginx, HAProxy, and others.
Router - Routers are specialized servers. They route and transfer data packets and traffic.
Routers may be an embedded device, laptop, or a full-sized server. Routers may use a
variety of protocols such as Routing Information Protocol (RIP), Virtual Router
Redundancy Protocol (VRRP), Cache Array Routing Protocol (CARP), and others.

SMTP (Outgoing mail) - Servers that send/deliver emails to clients typically use SMTP.
Examples of SMTP servers include exim, postfix, qmail, sendmail, and others.

Sound Server - These servers provide multimedia broadcasting and streaming. Many
"Internet Radio Stations" are sound servers. Icecast and Jamendo are examples of sound
servers. Also, "SoundCloud.com" is a sound server with a web server interface.

SSH - Secure Shell provides a security tunnel which may be used to transfer files, X11,
remote login (like VNC), etc. An example of an SSH server daemon is OpenSSH.

Tripwire - Tripwire is a special network service that monitors files on a network and then
reports malicious or invalid changes/differences. This helps to detect malicious code or
intrusions.

VNC - VNC servers allow a client to access the software and data of a remote computer
(the server). Clients using one operating system can use the software of a different
operating system without virtualization. An admin could setup dummy clients that are
meant to access a different operating system depending on the needed task. Assume a
business has a piece of software that only works on MS-Windows and other software on
a Linux system. The dummy clients can login to the needed operating system depending
on the needed software.

Virtualization Server - This is a server that specializes in virtualization of multiple

operating systems. VMware Server is a popular example. An alternative is Xen. Xen is a
hypervisor that provides virtualization.
Citrix XenServer (http://www.citrix.com/products/xenserver/overview.html),
XenServer (http://www.xenserver.org/), and
Amazon EC2 Cloud (http://aws.amazon.com/ec2/) are two examples of Xen servers.

Web Server - Web servers provide websites. A popular web server is Apache, which is
open source and free.