A short note on setting up a Linux kernel debugging environment
This is a relatively short post on how to set up a Linux environment for kernel debugging using Buildroot.
This note overlaps a lot with this blog post by Nick Desaulniers, and another post from Guanzhou Hu. Relative to both of them I’ve added a little more information around Buildroot configuration, but they are nonetheless good resources.
Setup
To start off, you’ll want to clone the Buildroot sources:
git clone https://git.buildroot.net/buildroot
You’ll need to make an initial .config
file for Buildroot. For the rest of
this note, I’m assuming that we’ll want to build our kernel as x86-64 and run it
under QEMU, so we’ll use the following command:
make qemu_x86_64_defconfig
Now start the Buildroot TUI with
make menuconfig
If you’ve built the Linux kernel from source before, the Buildroot UI should feel familiar. Instead of being presented with kernel build parameters to tweak, you’ll have various options for how you want to configure the operating system, such as
- what version of the Linux kernel you want to use;
- what packages should be installed by default;
- whether login with the root user is permitted (and the root user’s default password);
and so on.
Once you’ve finished selecting the options that you want for Buildroot, you
should save your config file (by default, to .config
) and exit. For the
purposes of this note, I used the following configuration options:
# Some packages require C++ support in order to be installed
# -> Toolchain -> Enable C++ support
BR2_TOOLCHAIN_BUILDROOT_CXX=y
# I like having OpenSSH installed by default as another means of getting access
# to the environment, but it isn't strictly necessary.
# -> Target packages -> Networking applications -> openssh
BR2_PACKAGE_OPENSSH=y
BR2_PACKAGE_OPENSSH_CLIENT=y
BR2_PACKAGE_OPENSSH_SERVER=y
BR2_PACKAGE_OPENSSH_KEY_UTILS=y
BR2_PACKAGE_OPENSSH_SANDBOX=y
# Add a post-build script for some minor SSH configuration
# -> System configuration -> Custom scripts to run before creating filesystem images
BR2_ROOTFS_POST_BUILD_SCRIPT="board/qemu/x86_64/post-build.sh ./scripts/config.sh"
# Use Ext4 for the filesystem image
# -> Filesystem images -> ext2/3/4 root filesystem
BR2_TARGET_ROOTFS_EXT2_4=y
BR2_TARGET_ROOTFS_EXT2_GEN=4
The ./scripts/config.sh
shell script just contains the following:
#!/bin/sh
PUBKEY="..." # Replace with your public key
OUTDIR=output/target
addline() {
grep -qxF "$1" "$2" || echo "$1" >> "$2"
}
mkdir -p ${OUTDIR}/root/.ssh
addline "PermitRootLogin yes" ${OUTDIR}/etc/ssh/sshd_config
addline "${PUBKEY}" ${OUTDIR}/root/.ssh/authorized_keys
chmod 600 ${OUTDIR}/root/.ssh/authorized_keys
Once you’ve set up your configuration, you can start building the Linux environment with
make -j$(nproc)
Using a custom kernel
If you just want to build the kernel with custom configuration parameters, you can run
make linux-menuconfig
to create a kernel build config file. Then you can add the following to your Buildroot config:
BR2_LINUX_KERNEL_USE_CUSTOM_CONFIG=y
BR2_LINUX_KERNEL_CUSTOM_CONFIG_FILE=./path/to/config/file
If you’re experimenting with some changes to the kernel sources, however, you’ll need to do one of the following:
- Configure some custom kernel
patches
via
BR2_GLOBAL_PATCH_DIR
orBR2_LINUX_KERNEL_PATCH
. - Point to a tarball with the modified kernel source using
BR2_LINUX_KERNEL_CUSTOM_TARBALL_LOCATION="file:///path/to/sources.tar.gz"
Booting the kernel with QEMU
If everything has gone smoothly up to this point, your newly-built Linux system
should now be in the output/images
directory (relative to the root of the
Buildroot repository). You can now run your environment under QEMU with (for
instance)
qemu-system-x86_64 \
-kernel ./output/images/bzImage \
-drive file=./output/images/rootfs.ext4,if=virtio,format=raw \
-nographic \
-enable-kvm \
-cpu host \
-m 512M \
-append "root=/dev/vda console=tty1 console=ttyS0"
You should see your system booting and reach the Buildroot login prompt (by
default the only user you can login as is root
, without any password):
Welcome to Buildroot
buildroot login: root
#
Press Ctrl-a x
to exit.
If you installed OpenSSH, you should also add
-net nic,model=virtio -net user,hostfwd=tcp::2222-:22
to your command. This will forward port 22 from the virtual machine to port 2222 on your host, making the VM accessible over SSH.
Debugging the kernel with GDB
Now that we have our VM running, we want to start using GDB to debug it. First,
we’ll need to build the kernel to support debugging using a custom build config.
You should run make linux-menuconfig
and set the following options:
Kernel hacking -> Kernel debugging
Kernel hacking -> Compile-time checks and compiler options -> Debug information
Kernel hacking -> Compile-time checks and compiler options -> Provide GDB scripts for kernel debugging
Alternatively, put the following in your kernel build config:
CONFIG_DEBUG_KERNEL=y
CONFIG_DEBUG_INFO_DWARF_TOOLCHAIN_DEFAULT=y
CONFIG_GDB_SCRIPTS=y
Run make
again to build your environment. When we run QEMU now, we’ll add the
following to our command:
-gdb tcp::1234
will open up a gdbserver on TCP port 1234.- The
-S
flag will tell the kernel to wait before booting. This will allow us to attach GDB to the kernel before the boot process starts. - We’ll add
nokaslr
to the kernel command line parameters to disable address space layout randomization in the kernel.
Your full command should now look something like this:
qemu-system-x86_64 \
-kernel ./output/images/bzImage \
-drive file=./output/images/rootfs.ext4,if=virtio,format=raw \
-nographic \
-append "nokaslr root=/dev/vda console=tty1 console=ttyS0" \
-enable-kvm \
-cpu host \
-m 512M \
-gdb tcp::1234 \
-S
Before starting GDB, you’ll want to ensure that you’ll be able to load the
vmlinux-gdb.py
script by adding it to your auto-load safe-path:
echo "add-auto-load-safe-path $(realpath output/images/linux/vmlinux-gdb.py)" \
>> ~/.gdbinit
Then run
gdb -x ./kerneldebug.gdb ./output/images/linux/vmlinux
where kerneldebug.gdb
contains the following:
# Attach to QEMU
target remote :1234
# Add any breakpoints you already know you want to set here, e.g.
#
# hbreak function_1
# hbreak function_2
# ...
hbreak start_kernel
If all went well, you should be able to run
(gdb) c
in GDB, and then see Linux boot up in the terminal where you ran QEMU.
Additional references
As mentioned earlier, Nick Desaulnier’s post and Guanzhou Hu’s post were both helpful references for me.
The Buildroot user manual is the most complete source of information on how to use Buildroot.
The kernel sources have some documentation on how to debug the kernel with GDB.