System logging

Facilities and severities

A syslog message carries a FACILITY, saying what kind of thing produced it — auth and authpriv, cron, daemon, kern, mail, user — and a SEVERITY. The severities, most to least serious, are emerg, alert, crit, err, warning, notice, info, debug. A configuration line names a selector like mail.err and a destination, and it matches that severity AND EVERYTHING MORE SEVERE. mail.=err matches only that level, and mail.* matches all of them.

Eight of the facilities are reserved for whatever a site wants, and they are named individually rather than as a range: local0, local1, local2, local3, local4, local5, local6, local7. Nothing in the system claims them, which is the point — an in-house application logs to one of the eight and gets its own file without competing with daemon or user for a selector. A question that asks which facility to give a bespoke service is asking for one of these, and it will spell it out in full.

rsyslog is the usual daemon and reads /etc/rsyslog.conf plus /etc/rsyslog.d/; syslog-ng is the alternative with a rather different configuration syntax. Destinations are usually files under /var/log — /var/log/messages or /var/log/syslog for the general stream, /var/log/auth.log or /var/log/secure for authentication — and can also be a remote host, which is how central logging is built. logger sends a message into syslog from a script or the command line, which is also how you test that a rule works.

The journal, and log rotation

systemd-journald collects everything — kernel messages, service output on standard output and standard error, and syslog traffic — into a structured binary journal. journalctl reads it: -u NAME for one unit, -b for this boot, -f to follow, -p err to filter by priority, --since and --until for a time window. It is binary by design, which is why it cannot be read with less and why it can be queried by field. By default it may be volatile, kept in /run and lost on reboot; making it persistent means creating /var/log/journal.

Getting a script INTO the journal has its own command, and it is not logger. systemd-cat -t maintenance /usr/local/sbin/maintenance.sh runs the script with both its standard output and its standard error redirected into the journal, so progress and problems alike arrive tagged and come back out with journalctl -t maintenance. Given no command it forwards its own standard input instead, which is the piped form. -t sets the identifier and -p the default priority. logger is the neighbouring tool and writes ONE message through the syslog interface — it takes -t and -p too, which is what makes the pair easy to confuse — so logger is for a line and systemd-cat is for a whole program's two streams.

Text logs grow forever unless something stops them, and that something is logrotate: /etc/logrotate.conf plus per-package files in /etc/logrotate.d/ say how often to rotate a file, how many old copies to keep, whether to compress them, and what command to run afterwards so the daemon reopens its file. The journal handles its own size limits in journald.conf instead.

The default rotation is a RENAME followed by the creation of a fresh file, and it only works because the writing process is then told to reopen its log — which is what the postrotate script calling systemctl reload or kill -HUP is for. A daemon that holds its file open and offers no way of being signalled keeps writing into the renamed file, and the new one stays empty forever. copytruncate is the directive for that case: logrotate copies the file and then truncates the original in place, so the descriptor the daemon is holding stays valid. It carries a small race — anything written between the copy and the truncate is lost — which is why it is the exception rather than the default.