Filtering text streams

The filter commands: small programs that read a text stream, change it, and write it out again — cut, sort, uniq, tr, sed and the rest — plus the flags on each that the exam actually asks about.

Lesson 2 of 8 in objective 103. GNU and Unix commands, part of LPIC-1 Exam 101-500.

A pipeline, and what the stream looks like after each filter. In order: cat access.log (every line of the file), then cut -d" " -f1 (the first field of each: an address), then sort (without this, uniq below sees nothing adjacent), then uniq -c (collapse ADJACENT repeats, and count them), then sort -rn | head (the busiest addresses first). cat access.log every line of the file cut -d" " -f1 the first field of each: an address sort without this, uniq below sees nothing adjacent uniq -c collapse ADJACENT repeats, and count them sort -rn | head the busiest addresses first
A pipeline, and what the stream looks like after each filter.

Slicing and joining

cut takes columns out of each line: cut -d: -f1 /etc/passwd takes the first colon-separated field, and cut -c1-10 takes characters by position. It cannot handle runs of spaces as one delimiter, which is why awk is so often reached for instead. paste does the reverse, joining files side by side; join merges two sorted files on a shared field, like a database join.

head and tail take the first or last lines — ten by default, -n to change it. tail -f follows a file as it grows and is the single most-used command in this list on a live system. nl numbers lines, wc counts them (-l lines, -w words, -c bytes), and split cuts a large file into pieces.

Logs are rotated and compressed, and none of these filters can read a .gz on its own, so each compressor ships a cat-style front end that decompresses to standard output and writes nothing to disk: zcat for gzip, bzcat for bzip2, xzcat for xz. zcat access.log.gz | grep 404 | wc -l is the whole idiom, and the point of it is what does NOT happen — the compressed file is left alone and no uncompressed copy appears beside it, which matters on the nearly full partition that made the log worth compressing. gunzip -c and gzip -dc are the same thing spelled with options, and zgrep, zless and zdiff wrap the common filters the same way.

Sorting, deduplicating, translating

sort orders lines: -n sorts numerically rather than as text (so 10 comes after 9), -r reverses, -k selects a field, -u drops duplicates as it goes. uniq collapses ADJACENT duplicate lines only — which is why it is nearly always preceded by sort — and uniq -c prefixes each with a count, uniq -d shows only the repeats.

tr translates or deletes characters, reading standard input only: tr a-z A-Z upcases, tr -d '\r' strips carriage returns, tr -s squeezes runs into one. od dumps a file as octal, hex or characters, which is how you see what is actually in a file that looks empty.

Why uniq is nearly always preceded by sort, and what sort -u does in one move. A column of 4 states: Unsorted lines (Repeats sitting anywhere in the file, not next to each other); Sorted lines (Every repeat now sits beside its twin); One line per value (The duplicates are gone, and nothing says how many there were); Each value with its count (uniq -c prefixes every line with the number of times it appeared). You get from Unsorted lines to Sorted lines by sort; from Sorted lines to One line per value by uniq, now that every duplicate is adjacent; from Sorted lines to Each value with its count by uniq -c, which counts as it collapses; from Unsorted lines to One line per value by sort -u, which drops the duplicates as it sorts. One arrow is drawn crossed through, because that move does not exist: Unsorted lines to Each value with its count — never: uniq -c alone counts only ADJACENT repeats, so a scattered value is counted several times. Unsorted lines Repeats sitting anywhere in the file, not next to each other sort Sorted lines Every repeat now sits beside its twin uniq, now that every duplicate is adjacent One line per value The duplicates are gone, and nothing says how many there were Each value with its count uniq -c prefixes every line with the number of times it appeared sort -u, which drops the duplicates as it sorts never: uniq -c alone counts only ADJACENT repeats, so a scattered value is counted several times uniq -c, which counts as it collapses
Why uniq is nearly always preceded by sort, and what sort -u does in one move.

sed, the stream editor

sed applies an editing script to every line as it passes. The substitution form is the one to know cold: sed 's/old/new/' replaces the first match on each line, sed 's/old/new/g' replaces every match, and a number in that position replaces the nth. Addresses restrict where a command applies — sed '2d' deletes line 2, sed '/error/d' deletes matching lines, sed -n '5,10p' prints a range and nothing else, because -n suppresses the automatic printing.

sed -i edits the file in place instead of writing to standard output. It is the flag that turns a safe experiment into a change, and on the exam it is the answer whenever a question says the file itself must be modified.

A sed substitution, part by part: which lines it touches, and how many matches. sed '/error/s/old/new/g' — part 1, /error/: address: only matching lines. 2 picks a line, 5,10 a range; part 2, s: substitute. d here deletes those lines; p with -n prints only them; part 3, old: the pattern to find on each line; part 4, new: what replaces it; part 5, g: every match on the LINE. Without it only the first; a number picks the nth. sed ' 1 /error/ 2 s / 3 old / 4 new / 5 g ' 1 address: only matching lines. 2 picks a line, 5,10 a range 2 substitute. d here deletes those lines; p with -n prints only them 3 the pattern to find on each line 4 what replaces it 5 every match on the LINE. Without it only the first; a number picks the nth
A sed substitution, part by part: which lines it touches, and how many matches.

Worth carrying in

cut -d: -f1
First colon-delimited field of each line.
sort -n -k2
Numeric sort on the second field. -u also drops duplicates.
uniq -c
Collapse adjacent duplicates and count them. Sort first.
tr a-z A-Z
Translate characters. -d deletes, -s squeezes repeats.
wc -l
Count lines. -w words, -c bytes.
tail -f
Follow a growing file.
sed 's/a/b/g'
Substitute every occurrence on each line. Without g, only the first.
sed -n '5,10p'
Print a line range only; -n suppresses the default output.
sed -i
Edit the file in place.
od -c
Dump a file as characters, revealing what you cannot see.
zcat file.gz
Decompress to standard output, leaving the file compressed. bzcat and xzcat for the other two.

What the exam does with this

Objective
103. GNU and Unix commands
Share of the exam
43.33% (the whole objective)
Questions in this lesson
10
Signed for by a person
0

Partly checked. None of the 10 questions here has been read against the cited source by a person. 10 questions have been checked against their cited clause by an automated pass — which is not the same thing, and is not a signature.

Only questions a person has signed for are used in mock exams here. That is the whole difference between the two kinds of checking above.

How these questions are written — where each question comes from, what the verification ledger records, and what happens when one is found wrong.

Drill this lesson

A lesson is one sitting: the trainer draws a short run from these questions alone and spaces the ones you get wrong.

Practise Filtering text streams

Questions in this lesson

Practise Filtering text streams

The rest of objective 103