Linux/Unix: Text Processing – awk, sed Explained with Output & Real-World Examples

Introduction – Why Learn Text Processing with awk and sed?

Linux/Unix provides incredibly powerful tools for processing text directly from the terminal. Whether you’re analyzing logs, formatting reports, or editing config files, awk and sed let you filter, transform, and automate with precision.

In this guide, you’ll learn:

• How to use awk for column-based operations and calculations
• How sed can perform inline substitutions and text manipulations
• Real-world examples with command output
• Best practices and practical use cases for log parsing and data cleaning

awk – Pattern Scanning and Field Processing Tool

Syntax:

awk 'pattern { action }' filename

Sample File: students.txt

Alice 85
Bob 92
Charlie 78

Example 1: Print First Column (Names)

awk '{print $1}' students.txt

Output:

Alice
Bob
Charlie

Example 2: Print Name and Score

awk '{print "Name:", $1, "- Score:", $2}' students.txt

Output:

Name: Alice - Score: 85
Name: Bob - Score: 92
Name: Charlie - Score: 78

Example 3: Print Students with Score > 80

awk '$2 > 80 {print $1, $2}' students.txt

Output:

Alice 85
Bob 92

Example 4: Calculate Total Score

awk '{sum += $2} END {print "Total:", sum}' students.txt

Output:

Total: 255

Example 5: Using Field Separator (CSV Example)

awk -F ',' '{print $2}' data.csv

Output: (Assuming data.csv has id,name,city)

name
John
Sarah

sed – Stream Editor for Inline Text Manipulation

Syntax:

sed [options] 'command' filename

Sample File: logs.txt

Error: Disk full
Warning: CPU high
Error: Memory low

Example 1: Replace First “Error” with “ALERT”

sed 's/Error/ALERT/' logs.txt

Output:

ALERT: Disk full
Warning: CPU high
ALERT: Memory low

Example 2: Replace All Occurrences Globally

sed 's/Error/ALERT/g' logs.txt

Same Output as above (all “Error” replaced with “ALERT”)

Example 3: Delete Blank Lines

File: sample.txt

Hello

World

sed '/^$/d' sample.txt

Output:

Hello
World

Example 4: Delete Line Number 2

sed '2d' sample.txt

Output:

Hello

Example 5: Edit File In-Place (Replace http with https)

sed -i 's/http/https/g' urls.txt

The file urls.txt is now updated with all http links changed to https.

Combined Use Case: awk + sed Workflow

Task: Extract usernames from a file and convert to uppercase

File: users.txt

alice
bob
charlie

awk '{print $1}' users.txt | sed 's/.*/\U&/'

Output:

ALICE
BOB
CHARLIE

awk extracts names, sed transforms to uppercase.

awk vs sed – When to Use?

Summary – Recap & Next Steps

Mastering awk and sed gives you a significant advantage in processing structured or unstructured data from logs, config files, reports, and scripts.

Key Takeaways:

• 🟩 Use awk for column operations, calculations, and report formatting
• 🟨 Use sed for editing streams, substitution, and line manipulation
• Combine both for advanced automation and log parsing workflows

FAQs

How do I change column separators in awk?
Use -F to define a new delimiter:

awk -F ':' '{print $1}' /etc/passwd

Can sed make permanent changes to a file?
Yes, use -i:

sed -i 's/foo/bar/g' file.txt

How do I sum a column using awk?
Use:

awk '{total += $2} END {print total}' numbers.txt

Can awk and sed be used in shell scripts?
Absolutely—they are widely used in bash scripts for parsing, editing, and output formatting.

What’s the difference between awk {print} and sed s/?
awk processes fields/columns, sed replaces or deletes text via pattern matching.