🧾 Linux/Unix: Text Processing – awk, sed Explained with Output & Real-World Examples

🧲 Introduction – Why Learn Text Processing with awk and sed?

Linux/Unix provides incredibly powerful tools for processing text directly from the terminal. Whether you’re analyzing logs, formatting reports, or editing config files, awk and sed let you filter, transform, and automate with precision.

🎯 In this guide, you’ll learn:

• ✅ How to use awk for column-based operations and calculations
• ✅ How sed can perform inline substitutions and text manipulations
• ✅ Real-world examples with command output
• ✅ Best practices and practical use cases for log parsing and data cleaning

---

📊 awk – Pattern Scanning and Field Processing Tool

✅ Syntax:

awk 'pattern { action }' filename

📁 Sample File: students.txt

Alice 85
Bob 92
Charlie 78

---

🔹 Example 1: Print First Column (Names)

awk '{print $1}' students.txt

📤 Output:

Alice
Bob
Charlie

🔹 Example 2: Print Name and Score

awk '{print "Name:", $1, "- Score:", $2}' students.txt

📤 Output:

Name: Alice - Score: 85
Name: Bob - Score: 92
Name: Charlie - Score: 78

🔹 Example 3: Print Students with Score > 80

awk '$2 > 80 {print $1, $2}' students.txt

📤 Output:

Alice 85
Bob 92

🔹 Example 4: Calculate Total Score

awk '{sum += $2} END {print "Total:", sum}' students.txt

📤 Output:

Total: 255

🔹 Example 5: Using Field Separator (CSV Example)

awk -F ',' '{print $2}' data.csv

📤 Output: (Assuming data.csv has id,name,city)

name
John
Sarah

---

🛠️ sed – Stream Editor for Inline Text Manipulation

✅ Syntax:

sed [options] 'command' filename

📁 Sample File: logs.txt

Error: Disk full
Warning: CPU high
Error: Memory low

---

🔹 Example 1: Replace First “Error” with “ALERT”

sed 's/Error/ALERT/' logs.txt

📤 Output:

ALERT: Disk full
Warning: CPU high
ALERT: Memory low

🔹 Example 2: Replace All Occurrences Globally

sed 's/Error/ALERT/g' logs.txt

📤 Same Output as above (all “Error” replaced with “ALERT”)

---

🔹 Example 3: Delete Blank Lines

📁 File: sample.txt

Hello

World

sed '/^$/d' sample.txt

📤 Output:

Hello
World

---

🔹 Example 4: Delete Line Number 2

sed '2d' sample.txt

📤 Output:

Hello

---

🔹 Example 5: Edit File In-Place (Replace http with https)

sed -i 's/http/https/g' urls.txt

✅ The file urls.txt is now updated with all http links changed to https.

---

🔄 Combined Use Case: awk + sed Workflow

🔧 Task: Extract usernames from a file and convert to uppercase

📁 File: users.txt

alice
bob
charlie

awk '{print $1}' users.txt | sed 's/.*/\U&/'

📤 Output:

ALICE
BOB
CHARLIE

✅ awk extracts names, sed transforms to uppercase.

---

⚖️ awk vs sed – When to Use?

Feature	awk	sed
Works with fields?	✅ Yes	❌ No
Works with patterns?	✅ Yes	✅ Yes
Supports math?	✅ Yes (sum, avg, count)	❌ No
In-place editing?	❌ No (use redirection)	✅ Yes (-i flag)
Best for	Reports, CSVs, filters	Inline substitution, deletions

---

📌 Summary – Recap & Next Steps

Mastering awk and sed gives you a significant advantage in processing structured or unstructured data from logs, config files, reports, and scripts.

🔍 Key Takeaways:

• 🟩 Use awk for column operations, calculations, and report formatting
• 🟨 Use sed for editing streams, substitution, and line manipulation
• 🛠️ Combine both for advanced automation and log parsing workflows

---

❓ FAQs

❓ How do I change column separators in awk?
✅ Use -F to define a new delimiter:

awk -F ':' '{print $1}' /etc/passwd

❓ Can sed make permanent changes to a file?
✅ Yes, use -i:

sed -i 's/foo/bar/g' file.txt

❓ How do I sum a column using awk?
✅ Use:

awk '{total += $2} END {print total}' numbers.txt

❓ Can awk and sed be used in shell scripts?
✅ Absolutely—they are widely used in bash scripts for parsing, editing, and output formatting.

❓ What’s the difference between awk {print} and sed s/?
✅ awk processes fields/columns, sed replaces or deletes text via pattern matching.

---

Share Now :