Sorting and Deduplication with sort & uniq
The `sort` command organizes lines alphabetically or numerically, and `uniq` filters or counts consecutive repeated lines. Combining `sort | uniq -c | sort -nr` is the quintessential Unix pipeline for frequency analysis.
"`sort` is like lining up a crowd by height or name; `uniq -c` is having an official count how many people in each group share the exact same height."
Deep Dive: How It Works
sort Flags: `-n` (numeric sort), `-r` (reverse order), `-k <pos>` (sort by specific column), `-t <char>` (custom field separator), `-u` (deduplicate during sort).
uniq Requirement: `uniq` only detects adjacent matching lines; input must always be sorted first.
uniq Flags: `-c` (prefix lines with repetition count), `-d` (print only duplicate lines), `-u` (print only unique lines).
Syntax Blueprint
sort file.txt # Alphabetical sort sort -n numbers.txt # Numeric sort (1, 2, 10) sort -k2 -nr sales.tsv # Sort by 2nd column numerically descending sort file.txt | uniq -c | sort -nr # Top frequency distribution
Always sort before piping to uniq for accurate frequency analysis.
Core Rules to Remember
Common Beginner Traps & How to Fix Them
Running `uniq` on an unsorted file and expecting global deduplication.Why it happens: `uniq` only checks whether the immediate next line matches the current line.
How to fix: Always pipe through `sort` first: `sort file.txt | uniq`.
Live Interactive Example
Hit Run Code to see it liveYour Turn: Micro Challenge
No pressure! Edit the starter code below and test your solution with instant feedback.
Sort Numbers in Descending Order
- Create `/tmp/nums.txt` containing 42, 5, 100, 12 on separate lines.
- Use `sort -nr /tmp/nums.txt` to output numbers largest first.
- Delete the temporary file.
Finished reading and practicing?
Mark this lesson as completed to update your course progress.