uniq

Show or ignore duplicate rows.

Summary

uniq [OPTION]... [INPUT [OUTPUT]]

The main purpose

• Write adjacent duplicate lines from the input file (or standard input) to the output file (or standard output).
• When there is no option, adjacent duplicate rows will be merged into one.

Options

-c, --count Increase the number of repetitions at the beginning of each line.
-d, --repeated All adjacent repeated lines are printed only once.
-D All adjacent duplicate lines will be printed.
--all-repeated[=METHOD] Like -D, but allows each group to be separated by a blank line. The METHOD value range is {none (default), prepend, separate}.
-f, --skip-fields=N Skip comparison of the first N columns.
--group[=METHOD] displays all lines, allowing groups to be separated by blank lines. METHOD value range: {separate (default), prepend, append, both}.
-i, --ignore-case Ignore differences in case.
-s, --skip-chars=N Skip comparison of the first N characters.
-u, --unique Print only non-contiguous duplicate lines.
-z, --zero-terminated Set the line terminator to NUL (empty) instead of newline.
-w, --check-chars=N Compare only the first N characters of each line.
--help Display help information and exit.
--version Display version information and exit.

Parameters

INPUT (optional): input file, standard input if not provided.

OUTPUT (optional): Output file, standard output if not provided.

return value

Returning 0 indicates success, returning a non-zero value indicates failure.

example

Note: The results of command 2 and command 3 are the same. Command 1 only deduplicates adjacent lines.

uniq file.txt
sort file.txt | uniq
sort -u file.txt

Only a single row is displayed, the difference is whether sorting is performed:

uniq -u file.txt
sort file.txt | uniq -u

Count the number of times each line appears in the file:

sort file.txt | uniq -c

Find duplicate lines in a file:

sort file.txt | uniq -d

Notice

1. uniq only detects whether adjacent lines are duplicated, sort -u sorts the input file first and then processes duplicate lines.

2. This command is a command in the GNU coreutils package. For related help information, please see man -s 1 uniq, info coreutils 'uniq invocation'.