Text Processing: grep, sed, awk, and Pipelines

Text Processing: grep, sed, awk, and Pipelines

Why Text Processing Matters

Web servers, applications, and system services write their activity to plain-text log and configuration files. Being able to extract, transform, and summarize that text at the command line is one of the most practical skills a Linux-based developer or administrator can have. Instead of opening a 50,000-line log file in a text editor, you write a short pipeline that answers a specific question in seconds.

The Linux Pipeline Model

Linux follows the Unix philosophy: small, single-purpose tools that communicate through standard input (stdin) and standard output (stdout). The pipe character `|` connects tools so the output of one becomes the input of the next. This lets you build powerful one-liners without writing a full script.

```

command1 | command2 | command3

```

Data flows left to right through the pipeline. Nothing is written to disk unless you redirect the final output with `>` (overwrite) or `>>` (append).

grep — Search and Filter

`grep` scans lines of text and prints only those that match a pattern.

• `grep "404" access.log` — prints every line containing the string 404
• `grep -i "error" app.log` — case-insensitive match
• `grep -v "GET" access.log` — prints lines that do not match (invert)
• `grep -c "POST" access.log` — counts matching lines instead of printing them
• `grep -E "error|warn" app.log` — extended regular expressions with alternation

Regular expressions (regex) are patterns that describe sets of strings. Common metacharacters: `.` (any character), `*` (zero or more of the preceding), `^` (start of line), `$` (end of line), `[0-9]` (character class).

sed — Stream Editor

`sed` reads lines one at a time, applies editing commands, and writes the result to stdout. It is most commonly used for search-and-replace.

• `sed 's/old/new/' file` — replaces the first occurrence of "old" with "new" on each line
• `sed 's/old/new/g' file` — replaces all occurrences on each line (global flag)
• `sed -i 's/old/new/g' file` — edits the file in place (modifies the actual file)
• `sed '/pattern/d' file` — deletes lines matching the pattern

Safety note: The `-i` flag rewrites the original file permanently. Always keep a backup copy of any production configuration or log file before running an in-place substitution. A common practice is `sed -i.bak 's/old/new/g' file`, which saves the original as `file.bak` before making changes.

awk — Pattern-Action Processing

`awk` reads input line by line, splits each line into fields, and lets you act on specific fields. By default, fields are separated by whitespace and are referenced as `$1`, `$2`, `$3`, and so on. `$0` refers to the entire line.

• `awk '{print $1}' access.log` — prints the first field of every line
• `awk '$9 == "404" {print $1}' access.log` — prints field 1 only on lines where field 9 equals 404
• `awk -F: '{print $1}' /etc/passwd` — uses `:` as the field delimiter
• `awk '{sum += $1} END {print sum}'` — accumulates a running total and prints it at the end

Apache's combined log format places the client IP address in field 1 and the HTTP status code in field 9, which makes `awk` a natural fit for log analysis.

sort and uniq — Counting and Ranking

• `sort file` — sorts lines alphabetically
• `sort -n file` — sorts numerically
• `sort -r file` — reverses the order
• `sort -rn file` — numeric, descending (highest first)
• `uniq file` — collapses consecutive duplicate lines into one (input must be sorted first)
• `uniq -c file` — prefixes each line with its count

Combining `sort | uniq -c | sort -rn` produces a ranked frequency table — exactly what you need to find the IP addresses generating the most 404 errors.

Saving Output

Redirect the final output of any pipeline to a file:

```

grep "404" access.log | awk '{print $1}' | sort | uniq -c | sort -rn > top_404_ips.txt

```

Use `>>` to append to an existing file rather than overwriting it.

Reading man Pages

Every tool covered here has a manual page. Run `man grep`, `man sed`, or `man awk` to read it. Press `q` to quit. The SYNOPSIS section shows the syntax; the OPTIONS section explains each flag. Consulting the man page before guessing at a flag is a professional habit.

Good Practices in the Lab

• Work on copies of log files when possible; never modify an original file without a backup.
• Test a `sed` command without `-i` first to confirm the output looks correct before writing changes to disk.
• Save your working commands in your lab log so you can reproduce results and share them with your peer reviewer.
• Pipelines can be interrupted with `Ctrl+C` if a command hangs or produces unexpected output.