Booting the system

The chain, stage by stage

Firmware (BIOS or UEFI) tests the hardware and hands control to a boot loader. On a BIOS machine that loader lives in the first sector of the disk, the MBR, which has 512 bytes to work with — enough for a stub that loads the rest. On a UEFI machine the firmware reads a real file from the EFI system partition instead, which is why UEFI boot loaders are easier to inspect and to break by deleting a file. UEFI also keeps its own menu of those loaders in NVRAM rather than on the disk — one BootXXXX variable per entry, plus a BootOrder saying which is tried first — and efibootmgr is what reads and rewrites that menu from a running system, so efibootmgr -v lists the entries and efibootmgr -o reorders them without going near the firmware setup screen.

The boot loader loads two things: the kernel and an initramfs. The initramfs is a small compressed root filesystem held in memory, and it exists to solve a chicken-and-egg problem — the drivers needed to read the real root filesystem (a RAID controller, an LVM volume, an encrypted disk) may themselves live on that filesystem. The initramfs carries just enough to mount the real root, then hands over to it.

The kernel then starts exactly one userspace process, PID 1, and everything else on the machine is a descendant of it. On most current distributions that is systemd; older ones use SysVinit, and a few use Upstart. Booting with init=/bin/bash as a kernel parameter replaces PID 1 with a shell, which is the classic way back into a system whose root password is lost.

Rebuilding the initramfs when the storage under root changes

An initramfs is built for one kernel on one machine and carries only the modules that machine needed at the time. Move the root filesystem onto an LVM logical volume, onto a RAID array, or behind LUKS, and the image that worked yesterday no longer contains the code to reach it: the kernel starts normally, the early userspace finds nothing it can mount as root, and the boot stops in an emergency shell complaining that there is no live root filesystem. Correcting root= does not help, because the kernel is not lost — it is unequipped. The image has to be regenerated.

The command for that differs by family, and both spellings are asked. Red Hat family systems use dracut: dracut -f overwrites the image for the running kernel in place, dracut -f --kver 6.1.0-18 names another one, and lsinitrd lists what ended up inside so you can check the device-mapper modules are actually there. Debian family systems use update-initramfs, where -u updates the image for the running kernel and -u -k all does every installed kernel; mkinitramfs is the lower-level tool it drives, and mkinitrd is the older name still met on SUSE. Reaching for the other family's tool is the wrong answer even when the reasoning behind it was right.

# Red Hat family: rebuild for the running kernel$ dracut -f$ lsinitrd | grep -c dm-mod1# Debian family: the same job under another name$ update-initramfs -u -k allupdate-initramfs: Generating /boot/initrd.img-6.1.0-18

Changing the kernel command line for one boot

A machine that hangs during boot has to be argued with before anything is running to argue with, and the place to do it is the boot loader menu. In GRUB, e opens the selected entry for editing; you append to the line beginning linux and press Ctrl-X or F10 to boot with the change. Nothing is written to disk, so the next boot is the old one again — which is what you want while you are still guessing.

Two parameters carry most of the recovery work. systemd.unit= names the target to boot into, so systemd.unit=rescue.target reaches maintenance without touching the default, and systemd.unit=emergency.target goes further down when even rescue will not come up. The older spellings single, 1 and init=/bin/bash do a similar job on a SysVinit machine, and systemd still honours them. systemd.mask= switches off one hanging unit for a boot without disabling it permanently.

Everything the running kernel was actually given is readable afterwards in /proc/cmdline, which is the fastest way to confirm whether the parameter you typed at the menu made it through.

# at the menu: press e, then edit the line starting linuxlinux /vmlinuz-6.1.0-18 root=UUID=1c3f... ro quiet# append the target, then Ctrl-X to bootlinux /vmlinuz-6.1.0-18 root=UUID=1c3f... systemd.unit=rescue.target$ cat /proc/cmdlineBOOT_IMAGE=/vmlinuz-6.1.0-18 root=UUID=1c3f... rosystemd.unit=rescue.target

Reading what the boot said

The kernel writes to a fixed-size ring buffer as it starts, and dmesg prints it. On a systemd machine journalctl -k prints the same thing from the journal and, unlike dmesg, journalctl -b -1 can show the boot before this one — which is what you want when the question is why the machine rebooted.

Userspace logging lands in /var/log. On systemd systems most of it is in the binary journal, read with journalctl; on others, and often on both, plain text arrives in /var/log/messages or /var/log/syslog.

A boot that succeeded but took two minutes is a different question, and systemd-analyze is what answers it. On its own it prints how long the firmware, the loader, the kernel and userspace each took. systemd-analyze blame prints one line per unit ranked by its own initialisation time, longest first, which is the list a question describing a slow boot is asking for. systemd-analyze critical-chain prints something that looks similar and is not: the chain of units that actually held the target up, which is usually shorter than the blame list because most of the slow units were starting in parallel and delayed nothing.