Streams, pipes and redirection

Three descriptors, and the numbers matter

Every process starts with standard input (descriptor 0), standard output (1) and standard error (2). > redirects standard output to a file, truncating it; >> appends; < takes standard input from a file. Errors are not redirected by > at all, which is why a command can appear to write nothing to its log file and still print a message on the terminal — that message went to descriptor 2.

2> redirects standard error, and 2>&1 makes descriptor 2 point at wherever descriptor 1 currently points. Order is everything: command > file 2>&1 sends both to the file, while command 2>&1 > file sends errors to the terminal, because at the moment 2 was pointed at 1, descriptor 1 was still the terminal. &> is the bash shorthand for both. /dev/null is the sink that discards anything written to it. Nothing forces the two streams together either: backup.sh > /var/log/backup.out 2> /var/log/backup.err gives each its own file, which is how a nightly job keeps a result log worth reading and an error log worth alerting on.

The other direction is asked less often than it deserves. sort < names.txt has the SHELL open the file and hand sort an already-open descriptor 0, so the command reads a stream and never learns the file's name; 0< is the same operator with the descriptor spelled out, exactly as 1> is the long spelling of >. The difference shows in anything that labels its results: wc -l /var/log/syslog prints the count and the file name after it, wc -l < /var/log/syslog prints the bare number, which is the form a script wants inside $( ). set -o noclobber, also spelled set -C, makes the shell refuse a plain > onto a file that already exists — the guard against typing > where >> was meant — and >| overrides it for one redirection without switching the option off. Because it is a property of the shell doing the redirecting rather than of the file, it never stops a program that opens the file itself, tee included.

Pipes, tee and xargs

A pipe connects one command's standard output to the next command's standard input, and the whole pipeline runs concurrently rather than one command finishing before the next starts. Standard error is not piped, which again is why errors appear on the terminal in the middle of a paged output.

tee writes its input to a file AND passes it on, so a pipeline can be logged without breaking it; tee -a appends. That is what it is for: dmesg | tee boot.log puts the kernel ring buffer on the screen and in a file in one pass, where dmesg > boot.log would give you the file and nothing to read. xargs solves the opposite problem: some commands, notably rm and grep, take file names as ARGUMENTS rather than on standard input, so find . -name "*.log" | xargs rm turns lines into arguments. Pairing find -print0 with xargs -0 is the version that survives file names containing spaces.

xargs is one of two ways across that gap and command substitution is the other. $(command) runs the command, captures its standard output and drops the text into the command line before the outer command is run at all, so kill $(pgrep httpd) reaches kill as kill 812 813 814. Use it when the list is short and known; use xargs when it might be long enough to overflow the argument list, or when -0 is needed for names with spaces in them. What does not work is pgrep httpd | kill, and the reason is the same one xargs exists for: kill reads its arguments and never looks at standard input.

The other thing worth knowing about tee is why it turns up with sudo. A redirection is performed by the SHELL parsing the line, before anything is run, so sudo echo text > /etc/sysctl.d/99-fw.conf opens the file as the unprivileged shell and fails there — sudo is never consulted, because the part that needed privilege was not the part sudo was applied to. echo text | sudo tee /etc/sysctl.d/99-fw.conf moves the opening into a privileged process instead, and sudo tee -a is the appending version. sudo sh -c "echo text > /etc/sysctl.d/99-fw.conf" works for exactly the same reason: this time the shell doing the redirecting is itself the privileged one.

A here-document feeds literal text to a command's standard input: command <<EOF, then lines, then EOF on a line of its own. It is how a script supplies multi-line input without a temporary file. A here-STRING, command <<<"text", is the one-line version.

$ dmesg | tee boot.log | grep -i error[   3.2] usb 2-1: device descriptor read/64, error -71# the whole buffer is in boot.log; only errors reached the screen$ make 2>&1 | tee build.log# both streams logged: 2>&1 comes before the pipe$ make > build.log 2>&1# both to the file$ make 2>&1 > build.log# errors to the terminal — 1 was still the terminal then$ backup.sh > /var/log/backup.out 2> /var/log/backup.err# two streams, two files: they need not end up together