awk Cheat Sheet: Every Pattern and Function You Need

awk is a text-processing language built for pattern-matching, field extraction, and report generation on structured text. It processes input line-by-line, splits each line into fields, and runs your program against each line that matches a pattern.

Quick reference

How awk works

An awk program is a sequence of pattern { action } rules:

awk 'pattern1 { action1 }
     pattern2 { action2 }' inputfile

For each input line:

1. awk splits the line into fields $1, $2, … $NF using FS (default: whitespace)
2. Each rule is tested in order; if the pattern matches, the action runs
3. If no pattern is given, the action runs on every line
4. If no action is given, the default action is print

Built-in variables

# Print number of fields per line
awk '{print NF}' data.txt

# Print second-to-last field
awk '{print $(NF-1)}' data.txt

# Use environment variable
awk 'BEGIN{print ENVIRON["HOME"]}'

# FNR resets per file; NR does not
awk '{print FILENAME, FNR, NR, $0}' a.txt b.txt

BEGIN and END blocks

BEGIN runs before any input is read. END runs after all input is processed.

# Print header and footer
awk 'BEGIN{print "Name\tScore"} {print $1"\t"$2} END{print "---"}' scores.txt

# Sum a column and compute average
awk '{sum+=$2; n++} END{print "avg:", sum/n}' data.txt

# Initialize variables
awk 'BEGIN{FS=","; OFS="\t"} {print $1,$3}' data.csv

Pattern types

# Regex pattern — match any line containing "error"
awk '/error/' log.txt

# Negated regex
awk '!/error/' log.txt

# Relational expression — field comparison
awk '$3 > 50' data.txt
awk '$1 == "Alice"' data.txt
awk '$2 ~ /^[0-9]+$/' data.txt   # field matches regex
awk '$2 !~ /foo/' data.txt        # field does NOT match regex

# Range pattern — from START line to END line (inclusive)
awk '/BEGIN/,/END/' file.txt

# Compound conditions
awk '$1=="error" && $3>100' log.txt
awk '/warn/ || /error/' log.txt

# Line number conditions
awk 'NR==1' file.txt           # first line only
awk 'NR>=5 && NR<=10' file.txt # lines 5–10
awk 'NR%2==0' file.txt         # even lines only

Arithmetic

awk supports full arithmetic: +, -, *, /, %, ^ (power).

# Running total
awk '{total += $2} END{print total}' sales.txt

# Compute percentage
awk '{pct = $2/$3 * 100; printf "%.1f%%\n", pct}' data.txt

# Increment counter
awk '/404/{errors++} END{print errors, "errors"}' access.log

# Built-in math functions
awk 'BEGIN{print sin(0), cos(0), sqrt(2), int(3.9), log(1), exp(1)}'
# → 0 1 1.41421 3 0 2.71828

String functions

# Substring (1-indexed, inclusive)
echo "hello world" | awk '{print substr($0, 7)}'      # → world
echo "hello world" | awk '{print substr($0, 1, 5)}'   # → hello

# Replace first occurrence
echo "foo foo foo" | awk '{sub(/foo/, "bar"); print}'  # → bar foo foo

# Replace all occurrences
echo "foo foo foo" | awk '{gsub(/foo/, "bar"); print}' # → bar bar bar

# Replace in specific field
awk '{gsub(/,/, ";", $2); print}' data.txt

# Split field into array
awk '{n=split($1, arr, ":"); for(i=1;i<=n;i++) print arr[i]}' data.txt

# match — get position and length
echo "2026-07-14" | awk '{match($0, /[0-9]{4}/); print substr($0, RSTART, RLENGTH)}'
# → 2026

# sprintf for formatting
awk '{printf "%05d  %-20s  %8.2f\n", NR, $1, $2}' data.txt

Arrays (associative)

awk arrays are key-value maps. Keys are always strings, but numbers work via automatic coercion.

# Count occurrences
awk '{count[$1]++} END{for(k in count) print k, count[k]}' file.txt

# Group sum by key
awk -F, '{sum[$1]+=$2} END{for(k in sum) print k, sum[k]}' data.csv

# Check if key exists
awk '{if ($1 in seen) print "dup:", $1; else seen[$1]=1}' file.txt

# Delete a key
awk '{delete arr[$1]}' file.txt

# Multi-dimensional (simulated)
awk '{matrix[$1][$2]++} END{for(r in matrix) for(c in matrix[r]) print r, c, matrix[r][c]}' file.txt
# gawk syntax — mawk uses matrix[r,c] (SUBSEP-separated key)

# Sorted output (gawk only)
awk '{count[$1]++} END{PROCINFO["sorted_in"]="@val_num_desc"; for(k in count) print k, count[k]}' file.txt

printf formatting

# Syntax: printf "format", arg1, arg2, ...
awk '{printf "%-15s %5d %8.2f\n", $1, $2, $3}' data.txt

# Format specifiers
# %d   integer          %05d  zero-padded
# %f   float            %.2f  2 decimal places
# %s   string           %-10s left-aligned in 10 chars
# %e   scientific       %g    shorter of %e/%f
# %x   hex              %o    octal
# \t   tab              \n    newline

Multi-file and getline

# Process two files differently
awk 'FNR==NR{a[$1]=$2; next} $1 in a{print $1, a[$1], $2}' prices.txt items.txt
# FNR==NR is true only for the first file — classic awk two-file join

# getline — read next line explicitly
awk '/HEADER/{getline; print "after header:", $0}' file.txt

# getline from a command
awk 'BEGIN{while(("ls" | getline line) > 0) print "file:", line}'

# getline from a file
awk '{while((getline line < "lookup.txt") > 0) print line}' main.txt

Output and redirection

# Append to file
awk '{print > "output.txt"}' data.txt        # overwrite each run
awk '{print >> "output.txt"}' data.txt       # append

# Redirect to different files based on field
awk '{print > ($3 > 100 ? "high.txt" : "low.txt")}' data.txt

# Pipe to a shell command
awk '{print $2 | "sort -n"}' data.txt

# Close a pipe between iterations (to re-open fresh)
awk '{print | "sort"; close("sort")}' data.txt

# Multiple output files
awk -F, '{print > $1".txt"}' data.csv   # one file per value of $1

Practical one-liners

# Print lines between two timestamps
awk '$1 >= "10:00" && $1 <= "11:00"' access.log

# Sum file sizes from ls -l
ls -l | awk 'NR>1{sum+=$5} END{print sum, "bytes"}'

# Find lines longer than 80 chars
awk 'length > 80' file.txt

# Print only duplicate lines
awk 'seen[$0]++ == 1' file.txt

# Remove blank lines
awk 'NF' file.txt

# Extract IP addresses (first field) and count unique ones
awk '{print $1}' access.log | sort | uniq -c | sort -rn | head -20

# CSV: print columns 1 and 3, skip header
awk -F, 'NR>1{print $1","$3}' data.csv

# Convert CSV to TSV
awk 'BEGIN{FS=","; OFS="\t"} {$1=$1; print}' data.csv

# Print every Nth line
awk 'NR%5==0' file.txt   # every 5th line

# Word frequency count
awk '{for(i=1;i<=NF;i++) freq[$i]++} END{for(w in freq) print freq[w], w}' text.txt | sort -rn

# Extract value from key=value log
awk '{for(i=1;i<=NF;i++) if($i~/^user=/) {split($i,a,"="); print a[2]}}' app.log

# Running average of column
awk '{sum+=$2; print $1, sum/NR}' data.txt

# Print lines where field 3 is not a number
awk '$3 !~ /^[0-9]+(\.[0-9]+)?$/' data.txt

# Transpose a matrix (rows → columns)
awk '{for(i=1;i<=NF;i++) row[i]=row[i] (NR==1?"":"\t") $i} END{for(i=1;i<=NF;i++) print row[i]}' matrix.txt

awk vs gawk vs nawk vs mawk

Use gawk when you need: regex intervals, sorted iteration, networking, or @include. Use mawk in pipelines where speed matters. On macOS, /usr/bin/awk is a BSD one-true-awk variant — install gawk via Homebrew for full features.

7 common mistakes

6 FAQ

Q: How do I process a CSV that has quoted fields with commas inside? awk -F, breaks on commas inside quotes. Use a proper CSV parser: Python's csv module, csvkit, or miller (mlr). For simple CSVs without quoted commas, -F, works fine.

Q: What's the difference between sub and gsub? sub replaces the first match; gsub replaces all matches. Both return the number of replacements made and modify $0 (or the third argument) in place.

Q: How do I pass a shell variable into awk? Use -v: awk -v threshold="$LIMIT" '$2 > threshold' file. Never interpolate shell variables directly inside single-quoted awk programs — it breaks the quoting. Alternatively wrap in double quotes and escape $ for awk fields: awk "\$2 > $LIMIT" — but this is fragile.

Q: How do I join two files on a common key? Classic idiom: awk 'FNR==NR{a[$1]=$2; next} $1 in a{print $1, a[$1], $2}' lookup.txt data.txt. FNR==NR is true only for the first file — it loads the lookup table. The second block matches rows in the second file.

Q: How do I handle Windows line endings (\r\n) in awk? Add sub(/\r$/, "") at the start of your action, or set RS="\r\n". On GNU awk you can also use --posix with RS="\r?\n".

Q: Is awk still worth learning in 2026? Yes. awk is on every Unix/Linux/macOS system, has no dependencies, and handles structured text (logs, CSVs, tabular data) faster than Python for quick one-liners. For complex logic or JSON, reach for Python or jq. For line-oriented text processing in shell scripts, awk is often the right tool.