David Schlachter

bhyve UEFI shell, Linux, and grub

You've installed Linux (e.g. Debian or Ubuntu) into an UEFI bhyve guest on FreeBSD, only to be greeted with a message about PXE boot after restarting from the installer:

BdsDxe: failed to load Boot0001 "UEFI BHYVE SATA DISK BHYVE-48FF-992B-D5E0" from PciRoot(0x0)/Pci(0x4,0x0)/Sata(0x0,0xFFFF,0x0): Not Found
>>Start PXE over IPv4.

This article will describe what happened, and show you how to fix this and similar issues. I'll assume that you've started the virtual machine using the bhyve command and are familiar with its various options (see man 8 bhyve).

In the message above, the UEFI firmware could not find an EFI bootloader, so it is attempting to perform a network boot. To skip this and simplify debugging, restart the bhyve virtual machine without a network adapter. You'll be greeted by the UEFI shell:

bhyve UEFI shell: UEFI Interactive Shell EDK II mapping table

Our goal is to find the EFI bootloader file that the firmware could not locate.

Helpfully, the EFI shell lists all of the storage devices and partitions that it could find. There will usually be at least two partitions on the Linux virtual machine's hard drive: the first, a small partition holding EFI firmware, the second, the Linux system. In this case, the EFI partition is FS0 (listed as "HD1(1,GPT,…" i.e. the first GPT partition of the first drive) the main Linux partition is BLK2 (listed as "HD1(2,GPT,…" i.e. the second GPT partition of the first drive). BLK0 is the drive itself (not considering the partition table) and BLK3 here is the swap partition.

We can navigate into a filesystem by typing its name, followed by a colon. From there, familiar commands like ls, mkdir, mv, and rm will be available. Let's see what was installed to the EFI partition:

Shell>FS0:
FS0:\>ls
Directory of: FS0:\
03/02/2021  16:54 <DIR>    4,096  EFI
          0 File(s)      0 bytes
          1 Dir(s)
FS0:\>cd EFI
FS0:\EFI\>ls
Directory of: FS0:\EFI\
03/02/2021  16:54 <DIR>    4,096  .
03/02/2021  16:54 <DIR>        0  ..
03/02/2021  16:54 <DIR>    4,096  debian
          0 File(s)      0 bytes
          1 Dir(s)

Aha! The UEFI firmware expects to find the bootloader in EFI/boot, not in EFI/debian. Typically, bootloader paths would be set as UEFI variables, but bhyve is unable to store these between reboots.

The easiest way to start the Linux guest from here is to execute the EFI bootloader inside the debian folder:

FS:\EFI\>cd debian
FS:\EFI\debian\>grubx64.efi

Boot should now continue normally.

How can you avoid having to manually run the bootloader from the EFI shell? Here are three options: