sed & awk
sed
Create a file [Link] with following content:
# text for sed starts from here
feb mar april may #ramadan starts in april :enable
attending tahoe class every morning Feb Mar April May :enable
sat mon tues wed :enable
doing linux everyday #everynight sat mon :enable
scripting in bash is fun :enable
cat file | sed ‘’
cat file | sed ‘p’
cat file | sed -n ‘p’
sed ‘s/name1/name2’ file
cat -n file | sed ‘s/name1/name2/g’
cat -n file | sed ‘s/name1/name2/gi’ (i flag will work case insensitively)
cat -n file | sed -r ‘3,5 s/name1/name2/g’
sed ‘s/^/name2/g’ file
sed ‘s/$/name2/g’ file
echo “[Link] | sed ‘s/http:\/\//www./g’
To not apply search and replace on any specific line:
cat -n file | sed ‘4! s/name1/name2/g’
To give space after every line:
cat file | sed G
Sed switches:
sed -i option will modify the original file
sed -[Link] will create a backup file of the original file before applying
modification
sed -r for matching extended regular expression
sed -e for multiple command
To print any specific line:
cat -n file | sed -n ‘3p’
cat -n file | sed -n ‘3,5p’
cat -n file | sed -n ‘/sat/p’ (Print line where the pattern matches)
To delete line:
cat -n file | sed ‘3d’
cat -n file | sed ‘3,5d’
cat -n file | sed -r ‘/sat/d’ (Delete line where the pattern matches)
cat -n file | sed ‘/^$/d’ (Delete all the blank line)
cat -n file | sed -e ‘/ *$//’ (Delete all the spaces at the end of the line)
cat -n file | sed -e ‘s/ #.*//g’ (Delete # and anything after that, all the
comments)
To perform multiple action:
cat -n file | sed -e ‘3d’ -e ‘G’
To search and replace using condition:
cat -n file | sed ‘/sat/ s/name1/name2/g’ (wherever sat is found, on
that line only search and replace will work)
To change any specific word to capital letter:
cat file | sed ‘s/firefox/\U&/g
awk
Syntax:
awk options ‘selection_criteria {actions}’ input_file
Selection_criteria: filtering condition and execute once for a given input file
Actions: print statement, assignment statement, arithmetic manipulation,
increment/decrement, awk built in functions/variables, loops or if statement.
Variables:
$0 -entire line
$1,$2,$3….$n -1st column, 2nd column, 3rd column…..nth column
NR -line number in the given input file
NF -number of columns in each line of the given input file
Create a file emp_ns.txt with the following contents:
100|Thomas|Manager|Sales|5000|21/01/98
200|Jason|Developer|Technology|5500|22/02/89
300|Sanjay|Sysadmin|Technology|7000|23/03/78
400|Nisha|Manager|Marketing|9500|24/04/87
500|Randy|DBA|Technology|6000|25/05/67
600|Chowdhary|DBA|Marketing|7000|26/06/76
700|Choudhary|Manager|Technology|8000|27/07/93
awk -F"|" '{print $0}' emp_ns.txt
Pattern matching in selection criteria:
awk -F"|" '/DBA/ {print $1,$2,$3,$4}' emp_ns.txt
awk -F"|" '/DBA/ {print $0}' emp_ns.txt
awk -F"|" '/[Cc]ho[wu]dhary/ {print $1,$2,$3,$4}' emp_ns.txt
awk -F"|" '/[Cc]ho[wu]dhary/ {print NR,$1,$2,$3,$4}' emp_ns.txt
Select lines from 3 to 6:
awk -F"|" 'NR==3,NR==6 {print NR,$1,$2,$3,$4}' emp_ns.txt
awk -F"|" 'NR==3,NR==6 {print NR,$1,$2,$3,$4 > "[Link]"}' emp_ns.txt
Comparison operators (if there are no spaces):
awk -F"|" '$3 == "DBA" || $3 == "Manager" {print $1,$2,$3,$5}' emp_ns.txt (add
DBA in the last column to show the difference)
(If there are spaces):
awk -F"|" '$3 ~ "DBA" || $3 ~ "Manager" {print $1,$2,$3,$5}' spa_emp_ns.txt
Regular expression in selection criteria:
awk -F"|" ' $2 == /[Cc]ho[wu]dhary/ {print $0}' emp_ns.txt (won’t work due to
reg expression)
awk -F"|" ' $2 ~ /[Cc]ho[wu]dhary/ {print $0}' emp_ns.txt (use ~ for reg
expression)
Number comparison:
awk -F"|" '$5 > 600 && $4 ~ "Technology" {print $0,$5*.25,$5*.50}' emp_ns.txt
Command inside another file:
Copy $5 > 600 && $4 ~ "Technology" inside [Link] then call it using -f
switch after awk
awk -F"|" -f [Link] emp_ns.txt
Variables:
To count how many employees are in tech department and their salary is
greater than 5000?
awk -F"|" '$4 == "Technology" && $5 > 5000 {count++;print
count,$2,$3,$4,$5}' emp_ns.txt
BEGIN and END in awk:
awk options ‘BEGIN {actions}
Processing statements
END {actions}’ input_file
BEGIN: Initialize the value of the variables. Print the heading. Execute only
once.
Processing statements: will execute for all the records present in the input
file.
END: Print the output variable after completing the manipulation in
processing.
Total number of lines in a file:
awk -F"|" 'BEGIN { count=0; } { count++; } END { print "Total number of line: "
count; }' emp_ns.txt
awk -F"|" 'BEGIN { print "TAHOE PNR"; } { count++; } END { print "Total number
of line: " count; }' emp_ns.txt
Calculate bonus for all the employees and add it along with input file:
awk -F"|" 'BEGIN { bonus_pct=.25; } { bonus=$5*bonus_pct;print $0"|"bonus }'
emp_ns.txt
Calculate total no of employees in Technology dept.:
awk 'BEGIN { FS="|";count=0; } $4 == "Technology" { count++; } END { print
"Total no of employees:" count }' emp_ns.txt
Print average salary of the employees in technology department:
awk -F"|" '$4 == "Technology" { count++; tot+=$5; print count,$2,$3,$4,$5; } END
{print "The average salary is:", tot/count}' emp_ns.txt
