🧪 Linux/Unix: Pipes & Filters – |, tee, xargs, sort, uniq Explained

🧲 Introduction – Why Learn Pipes & Filters in Linux/Unix?

In Linux/Unix, pipes and filters allow you to build powerful command chains to transform and analyze text streams. Using tools like |, tee, xargs, sort, and uniq, you can filter, redirect, and process data efficiently in real-time, all from the terminal.

🎯 In this guide, you’ll learn:

• How to chain commands using pipes
• Use tee to split output to screen and file
• Pass arguments dynamically with xargs
• Filter data with sort and uniq
• Real-world usage for scripting and automation

---

🔗 | – Pipe Operator

✅ Syntax:

command1 | command2

📌 Description:

Takes the stdout of the first command and passes it as stdin to the next.

🧪 Example:

ls -l | grep ".sh"

✅ Lists only .sh files from the current directory.

🧠 Pipes are the foundation of UNIX-style command chaining.

---

📝 tee – Split Output to File and Terminal

✅ Syntax:

command | tee filename

📌 Description:

• Displays command output on screen and saves it to a file.
• Useful for logging and monitoring simultaneously.

🧪 Example:

df -h | tee disk_report.txt

✅ Output is shown and written to disk_report.txt.

🧠 Append using:

command | tee -a logfile.txt

---

🔁 xargs – Build Commands from Input

✅ Syntax:

command | xargs command2

📌 Description:

Reads items from stdin and executes the specified command with them as arguments.

🧪 Examples:

echo "file1.txt file2.txt" | xargs rm

✅ Deletes both files.

Combine with find:

find . -name "*.log" | xargs grep "error"

✅ Searches “error” inside all .log files.

---

🔢 sort – Sort Input Text

✅ Syntax:

sort [options] filename

📌 Description:

Sorts lines in a file or stream alphabetically or numerically.

🧪 Examples:

sort names.txt
sort -r names.txt          # Reverse sort
sort -n numbers.txt        # Numeric sort

✅ Use with pipe:

cat file.txt | sort | uniq

---

🔂 uniq – Filter Repeated Lines

✅ Syntax:

uniq [options] filename

📌 Description:

Removes consecutive duplicate lines. Often used with sort.

🧪 Examples:

uniq list.txt
sort list.txt | uniq       # Remove all duplicates
uniq -c sorted.txt         # Show counts of duplicates

✅ uniq only works on adjacent lines—use sort first to group duplicates.

---

⚙️ Combining Pipes and Filters – Real-World Example

🔍 Example: Find most frequent IPs from a log file

cat access.log | awk '{print $1}' | sort | uniq -c | sort -nr | head

📌 Breakdown:

• awk '{print $1}' → extract IPs
• sort → group identical IPs
• uniq -c → count duplicates
• sort -nr → sort by frequency
• head → show top 10

---

📌 Summary – Recap & Next Steps

Pipes and filters let you build mini data processing pipelines in your terminal. Whether parsing logs, removing duplicates, or passing file lists dynamically, tools like |, tee, xargs, sort, and uniq make Linux command-line work fast, scriptable, and powerful.

🔍 Key Takeaways:

• | connects output of one command to the input of another.
• tee displays and writes output to a file.
• xargs transforms input into arguments for other commands.
• sort and uniq clean and deduplicate data.
• Combine them for real-time stream processing.

---

❓ FAQs

❓ What’s the difference between | and xargs?
✅ | connects output between commands; xargs builds command-line arguments from input.

❓ How do I log and view output at the same time?
✅ Use:

command | tee output.log

❓ Why does uniq not remove all duplicates?
✅ It only removes consecutive duplicates. Sort the input first:

sort file.txt | uniq

❓ Can I delete files listed in a text file using xargs?
✅ Yes:

cat files.txt | xargs rm

❓ How do I append tee output to an existing file?
✅ Use:

command | tee -a logfile.txt

---

Share Now :