Data Structure
Networking
RDBMS
Operating System
Java
MS Excel
iOS
HTML
CSS
Android
Python
C Programming
C++
C#
MongoDB
MySQL
Javascript
PHP
- Selected Reading
- UPSC IAS Exams Notes
- Developer's Best Practices
- Questions and Answers
- Effective Resume Writing
- HR Interview Questions
- Computer Glossary
- Who is Who
Remove Duplicate Lines from a Sorted File in Linux
To remove duplicate lines from a sorted file and make it unique, we use the uniq command in the Linux system. The uniq command work as a kind of filter program that reports out the duplicate lines in a file. It filters adjacent matching lines from the input and gives a unique output. This command is also available in the Windows and IBM i operating system.
Syntax
The general syntax of the uniq command is as follows
uniq [OPTION]... [INPUT [OUTPUT]]
Brief description of options available in the fmt command.
| Sr.No. | Option & Description |
|---|---|
| 1 |
-c, --count Display how many times line was repeated. |
| 2 |
-d—repeated Display only repeated lines, one for each group. |
| 3 |
-D Display all duplicate lines. |
| 4 |
-f, --skip-fields=N Avoid comparing the first N fields. |
| 5 |
-i, --ignore-case While comparing ignore differences in case. |
| 6 |
-s, --skip-chars=N Avoid comparing the first N characters. |
| 7 |
-u, --unique Prints only unique lines |
| 8 |
-w, --check-chars=N Line delimiter is NULL, not newline |
| 9 |
-v, --verbose Compare no more than N characters in lines. |
| 10 |
--help Display help and exit. |
| 11 |
--version Output the version information and exit. |
To print after removal repeated lines in a file, we use the uniq command in the Linux system as shown below.
$ cat >text.txt Print only unique lines. The earth is round. The earth is round. Welcome to the tutorialpoint... Welcome to the tutorialspint... $ uniq text.txt Print only unique lines. The earth is round. Welcome to the tutorialpoint...
To print number of duplicate lines of a file, we use the -c or --count option with the unique command as shown below.
$ uniq –c text.txt 2 The earth is round. 2 Welcome to the tutorialspoint... 1 Print only unique lines.
To prints only unique lines of a file, we use the -u or –unique option with the uniq command as shown below.
$ uniq –u text.txt Print only unique lines.
To check more information about the uniq command, we use the --help option with the uniq command in the Linux operating system as shown below.
$ uniq --help
To check version information of the uniq command, we use the --version option with the uniq command in the Linux operating system as shown below.
$ uniq --version
7K+ Views
