Linux/Unix: Pipes & Filters – |, tee, xargs, sort, uniq Explained

Introduction – Why Learn Pipes & Filters in Linux/Unix?

In Linux/Unix, pipes and filters allow you to build powerful command chains to transform and analyze text streams. Using tools like |, tee, xargs, sort, and uniq, you can filter, redirect, and process data efficiently in real-time, all from the terminal.

In this guide, you’ll learn:

• How to chain commands using pipes
• Use tee to split output to screen and file
• Pass arguments dynamically with xargs
• Filter data with sort and uniq
• Real-world usage for scripting and automation

| – Pipe Operator

Syntax:

command1 | command2

Description:

Takes the stdout of the first command and passes it as stdin to the next.

Example:

ls -l | grep ".sh"

Lists only .sh files from the current directory.

Pipes are the foundation of UNIX-style command chaining.

tee – Split Output to File and Terminal

Syntax:

command | tee filename

Description:

• Displays command output on screen and saves it to a file.
• Useful for logging and monitoring simultaneously.

Example:

df -h | tee disk_report.txt

Output is shown and written to disk_report.txt.

Append using:

command | tee -a logfile.txt

xargs – Build Commands from Input

Syntax:

command | xargs command2

Description:

Reads items from stdin and executes the specified command with them as arguments.

Examples:

echo "file1.txt file2.txt" | xargs rm

Deletes both files.

Combine with find:

find . -name "*.log" | xargs grep "error"

Searches “error” inside all .log files.

sort – Sort Input Text

Syntax:

sort [options] filename

Description:

Sorts lines in a file or stream alphabetically or numerically.

Examples:

sort names.txt
sort -r names.txt          # Reverse sort
sort -n numbers.txt        # Numeric sort

Use with pipe:

cat file.txt | sort | uniq

uniq – Filter Repeated Lines

Syntax:

uniq [options] filename

Description:

Removes consecutive duplicate lines. Often used with sort.

Examples:

uniq list.txt
sort list.txt | uniq       # Remove all duplicates
uniq -c sorted.txt         # Show counts of duplicates

uniq only works on adjacent lines—use sort first to group duplicates.

Combining Pipes and Filters – Real-World Example

Example: Find most frequent IPs from a log file

cat access.log | awk '{print $1}' | sort | uniq -c | sort -nr | head

Breakdown:

• awk '{print $1}' → extract IPs
• sort → group identical IPs
• uniq -c → count duplicates
• sort -nr → sort by frequency
• head → show top 10

Summary – Recap & Next Steps

Pipes and filters let you build mini data processing pipelines in your terminal. Whether parsing logs, removing duplicates, or passing file lists dynamically, tools like |, tee, xargs, sort, and uniq make Linux command-line work fast, scriptable, and powerful.

Key Takeaways:

• | connects output of one command to the input of another.
• tee displays and writes output to a file.
• xargs transforms input into arguments for other commands.
• sort and uniq clean and deduplicate data.
• Combine them for real-time stream processing.

FAQs

What’s the difference between | and xargs?
| connects output between commands; xargs builds command-line arguments from input.

How do I log and view output at the same time?
Use:

command | tee output.log

Why does uniq not remove all duplicates?
It only removes consecutive duplicates. Sort the input first:

sort file.txt | uniq

Can I delete files listed in a text file using xargs?
Yes:

cat files.txt | xargs rm

How do I append tee output to an existing file?
Use:

command | tee -a logfile.txt