awk in ten minutes

general / published / guide #19 / ≈ 4 min read

awk is a tiny programming language designed for one job: chewing through text, line by line, and doing something with the columns. It's been on every Unix-like system since the 1970s. You don't need to "learn awk" the way you'd learn Python — most of what you'll ever do with it is one-liners.

This guide is a tour of the patterns that show up over and over. Memorise the shapes, not the syntax — when you need to do something specific, you'll know which one to reach for.

The default: awk prints lines

Run it with no script and awk just echoes its input:

echo -e "one
two
three" | awk '{print}'
# one
# two
# three

The single-quoted bit is the awk program. {print} means "for every line, print it." That's the smallest meaningful awk you can write.

Pick a column ($1, $2, $3)

Awk splits every line into fields, separated by whitespace by default. $1 is the first field, $2 the second, and so on. $0 is the whole line. $NF is the last field (NF = number of fields).

# show just the file size from ls -l
ls -l | awk '{print $5}'

# show name and size only
ls -l | awk '{print $9, $5}'

# show only the last column
ls -l | awk '{print $NF}'

Use a different separator (-F)

Sometimes the columns aren't whitespace-separated:

# /etc/passwd uses colons. print just the usernames:
awk -F: '{print $1}' /etc/passwd

# a comma-separated file:
awk -F, '{print $3}' data.csv

# any single character works as -F.
# for regex separators use -F'[, 	]' and so on.

Patterns: only act on matching lines

An awk program is pattern { action }. If you give a pattern but no action, the default action is print:

# print only lines containing "error"
awk '/error/' /var/log/syslog

# print only lines starting with #
awk '/^#/' config.conf

# combine: print just the second column of error lines
awk '/error/ {print $2}' /var/log/syslog

BEGIN and END — run before / after everything

BEGIN runs before the first line, END runs after the last. Brilliant for headers, totals, and summaries.

# count the lines (the awk way to do "wc -l")
awk 'END {print NR}' file.txt

# sum the file sizes from ls -l
ls -l | awk '{total += $5} END {print total}'

# print a header, then the data
awk 'BEGIN {print "USER   SHELL"} {print $1, $7}' FS=: /etc/passwd

NR is the "number of records seen so far." After the last line, it's the line count.

Conditions inside the action

Awk has a normal-looking if/else:

# print only lines where 3rd column is greater than 100
awk '$3 > 100 {print $0}' data.txt

# print "big" or "small" next to each number
awk '{if ($1 > 100) print $1, "big"; else print $1, "small"}' numbers.txt

You can use ==, !=, <, >, &&, || — exactly what you'd expect.

Built-in variables worth knowing

  • $0 — the whole line
  • $1, $2, ... — individual fields
  • $NF — the last field
  • NR — line number (Number of Records so far)
  • NF — how many fields on the current line
  • FS — input field separator (same thing as -F)
  • OFS — output field separator (defaults to a space)
  • FILENAME — the file currently being read

A handful of real one-liners

# count unique IPs in an apache access log
awk '{print $1}' access.log | sort | uniq -c | sort -rn | head

# sum the 3rd column of a CSV (skip header)
awk -F, 'NR > 1 {total += $3} END {print total}' sales.csv

# print line numbers next to every line
awk '{print NR": "$0}' file.txt

# print every other line
awk 'NR % 2 == 1' file.txt

# remove duplicates without sorting
awk '!seen[$0]++' file.txt

# print the longest line
awk 'length > max {max = length; line = $0} END {print line}' file.txt

That last one is delightful — it tracks the longest line as it sees them and prints the winner at the end, all in one short line of awk.

When awk becomes a real script

If a one-liner is getting unreadable, put it in a file:

# savings.awk
BEGIN { FS = ","; OFS = " | "; print "Month", "Total" }
NR > 1 { totals[$1] += $3 }
END {
    for (month in totals) print month, totals[month]
}

Run it with:

awk -f savings.awk sales.csv

For anything beyond a couple of lines, this is much nicer than wrestling with quoting on the command line.

The shortest summary

  • awk '{print $1}' — print the first column.
  • awk -F: — change the separator.
  • awk '/pattern/ {action}' — act only on matching lines.
  • awk 'END {...}' — run at the very end (great for totals).
  • Composable: chain awk into pipes alongside sort, uniq, head.

With just those, you can do an astonishing amount of work without ever opening a text editor.

← back to index