Keyboard shortcuts

Press ← or → to navigate between chapters

Press S or / to search in the book

Press ? to show this help

Press Esc to hide this help

🏠 Back to Blog

Linux Kernel Boot Process

High Level Process:

  1. The machine’s BIOS or EUFI loads and runs a boot loader
  2. The boot loader finds the kernel image on disk, loads it into memory, and starts it.
  3. The kernel takes over and initializes the devices and drivers for each. This happens in the following order:
    1. CPU inspection
    2. memory inspection
    3. device bus discovery
    4. device discovery
    5. Auxiliary kernel subsystem setup (networking, etc.)
  4. The kernel mounts the root filesystem
  5. The kernel starts a program called init (systemd) with a PID of 1. This point is the user-space startup.
  6. init sets the rest of the system processes in motion
  7. At some point, init starts a process allowing you to login, usually at the end or near the end of the boot sequence.

The best way to view the boot process diagnostic logs is with journalctl. You can use journalctl -k to view messages from the current boot. You can use the -b option to view messages from previous boots. You can also check for a log file such as /var/log/kern.log or run the dmesg command to view the messages in the kernel ring buffer.

Kernel parameters

When the linux kernel starts, it receives a list of text parameters containg a few additional system details. The parameters specify many different types of behavior, such as the amount of diagnostic output the kernel should produce and device driver-specific options.

  • You can view the parameters passed to the kernel by looking at the /proc/cmdline file: 
    root@nginx-vm-00:~# cat /proc/cmdline
BOOT_IMAGE=/boot/vmlinuz-5.15.0-1029-azure root=PARTUUID=c51187ab-04cc-499f-8947-0211dc8d74e7 ro console=tty1 console=ttyS0 earlyprintk=ttyS0 panic=-1
    Upon encountering a parameter that the kernel does not understand, the kernel passes that parameter to the init system. For example, if you were to pass the -s parameter to the kernel, the kernel would pass that parameter to systemd to boot the system into single-user mode. Read the bootparam(7) man page for more info on kernel boot parameters.

Boot Loaders

  • At the start of the boot process, a boot loader starts the kernel. It loads the kernel into memory from somewhere on disk, and then starts the kernel with a set of kernel parameters as described above. This process sounds simple, right? Well, it gets a bit more complicated. Some questions need to be answered: “where is the kernel?” and “what boot parameters do we use?”. It seems like these answers should be easy to find. But remember, the kernel is not yet running, and it’s the kernel’s job to traverse a file system to locate files. We have a ‘chicken and egg’ problem.
  • A boot loader does need a driver to access a disk. On PCs, the boot loader uses the BIOS of UEFI to access disks. Disk hardware typically includes firmware that allows the BIOS or UEFI to acecss attached storage hardware via Logical Block Addressing (LBA). LBA is a universal, simple way to access data from any disk.
  • To determine if your system uses BIOS or UEFI, you can run efibootmgr. If you get a list of boot devices, your system is using UEFI. If you get an error stating UEFI parameters are not supported, your system is using BIOS. Alternatively, if /sys/firmware/efi exists, your system is using UEFI.
  • Boot loaders typically allow users to switch between different kernels and operating systems.
  • Common Boot Loaders
    • GRUB = Used on most linux systems. Supports BIOS and UEFI
    • LILO = One of the first boot loaders available for linux.
    • SYSLINUX
    • systemd-boot
    • coreboot
  • Accessing the boot loader may be different on each system. Linux distrobutions tend to heavily modify the boot loader, causing some confusion. On a PC, you can typically hold down shift or esc to access the boot loader shortly after powering on the system.
  • To generate a grub configuration file:
    • grub2-mkconfig -o /boot/grub2/grub.cfg for BIOS systems
    • grub2-mkconfig -o /boot/efi/EFI/grub.cfg for EFI systems
    • grub2-install can be used to install grub on a disk
    • The boot loader is typically stored on the first few sectors of a disk
    • The grub2.cfg file is typically stored at /boot/grub2/grub.cfg for BIOS systems
  • /etc/default/grub is used by the grub2-mkconfig utility to determine what settings to use when it generates the grub2.cfg file. After you modify this file, you need to run grub2-mkconfig to actually regenerate the grub2 config 
    GRUB_TIMEOUT=1
GRUB_TIMEOUT_STYLE=countdown
GRUB_DISTRIBUTOR="$(sed 's, release .*$,,g' /etc/system-release)"
GRUB_DEFAULT=saved
GRUB_DISABLE_SUBMENU=true
GRUB_TERMINAL="serial console"
GRUB_CMDLINE_LINUX="console=tty1 console=ttyS0,115200n8 earlyprintk=ttyS0,115200 rootdelay=300 scsi_mod.use_blk_mq=y crashkernel=auto"
GRUB_DISABLE_RECOVERY="true"
GRUB_ENABLE_BLSCFG=true
GRUB_SERIAL_COMMAND="serial --speed=115200 --unit=0 --word=8 --parity=no --stop=1"
  • Install, Configure, and Troubleshoot BootLoaders
    • To regenerate grub2 config:
      • Boot into recovery media, then:
        • chroot /mnt/sysroot
        • To regenerate grub config for BIOS system
          • grub2-mkconfig -o /boot/grub2/grub.cfg
        • To regenerate grub config for EFI system:
          • grub2-mkconfig -o /boot/efi/EFI/centos/grub.cfg
    • To reinstall the boot loader:
      • BIOS systems:
        • Use lsblk to look at block devices. Try to identify the boot device
        • Use grub2-install /dev/sda to install grub to the boot device
      • EFI Systems:
      • Use dnf reinstall grub2-efi grub2-efi-modules shim to reinstall grub to the boot device

Grub

  • The grub configuration directory is usually /boot/grub or /boot/grub2 (grub2 on redhat distros)
  • The main configuration file for grub is grub.cfg. Do not modify this file directly, instead use grub-mkconfig. The files in /etc/grub.d are shell scripts that make up the grub.cfg. When you call grub-mkconfig, it references these scripts in /etc/grub.d to create the grub.cfg. To modify the grub configuration, simply add another script to this directory. Then call grub-mkconfig, overwriting the /boot/grub/grub.cfg file: grub-mkconfig -o /boot/grub/grub.cfg
  • To (re)install grub, you can use grub-install
    • Example for installing grub on a mounted storage device: grub-install --boot-directory=/mnt/boot /dev/sdc

User space init

Process overview:

  1. init system starts (typically systemd)
  2. Essential low-level services start (think udevd and syslogd)
  3. Network services start
  4. Mid and high-level services start (cron, printing, etc.)
  5. Login prompts, GUIs, and high-level apps, such as web servers start

Shutting down the system

  • You can use sudo systemctl reboot --force to force a system to reboot
  • only the superuser can reboot
  • shutdown takes a time parameter for scheduling a shutdown.
    • ex: shutdown 03:00 = shutdown at 3AM
    • shutdown +15 shutdown in 15 minutes
    • If you specify a time in the future, the shutdown command creates a file called /etc/nologin and no one but the superuser is able to login to the system.
    • When the shutdown time arrives, shutdown tells the init system to begin the shutdown process. On a system using systemd, this means activating the shutdown units.
  • If you halt the system, it shuts the machine down immediately. To do this, run: shutdown -h now or halt
    • On most versions of Linux, a halt cuts power to the system. This can be unforgiving, as it does not give disk buffers time to sync (potentially causing corrupt data).
  • The shutdown process:
    1. init asks every process to shut down cleanly
    2. If a process doesn’t respond after a while, init kills it, first trying a TERM signal
    3. If the TERM signal doesn’t work, init uses the KILL signal
    4. The system locks system files to prevent modification
    5. The system unmounts all filesystems other than root
    6. The system remounts root as read-only
    7. The final step is to call the kernel to reboot or stop with the reboot(2) system call

InitRam FS

  • We need the initramfs because the kernel does not talk directly to the PC BIOS or EFI to get data from disks. So in order to mount its root filesystem, it needs driver support for the underlying storage. There are so many storage controllers that having a driver for each one in the kernel is not feasible. Therefore, these drivers are shipped as loadable modules. These modules exist on disk, so we have a chicken and egg scenario. How can the kernel load these drivers from disk if it cannot read the disk because it doesn’t currently have these drivers loaded?
  • The workaround is to gather these drives along with a few other utilities into a cpio archive. The boot loader loads this archive into memory before running the kernel. Upon start, the kernel reads the contents of the archive into a temp file system in RAM known as the initramfs, mounts it at /, and performs the user-mode handoff to the init on the initramfs. Then, the utilities included in the initramfs allow the kernel to load the necessary driver modules for the real root filesystem. Finally, the utilities mount the real root filesystem and start the init system.