Nvidia vGPU Driver fails to load with error Code 43 on XenServer 7 Hosts with more than 512GB-Ram

When you try to start a VM with an attached vGPU and the XenServer itself has more than 512GB-Ram it might happen that the driver does not start. You just get an error message with Code 43:

(Sorry for poor quality.)

In some situations it’s even not possible to boot the whole VM:

An emulator required to run this VM failed to start

If you google for the problem you find the following Knowledge-Base-Article from Nvidia.

Unfortunately the information’s in this article only partly worked for us. The first command should automatically add the required parameters to the grub.cfg

Command line: (/opt/xensource/libexec/xen-cmdline –set-dom0 iommu=dom0-passthrough)

Alternatively there is a second method described how to fix it manually:

Or by editing the bootloader (/etc/grub.conf) grub.conf to contain:iommu=Dom0-passthrough

To check the grub.cfg you have to open a console session and browse to one of these folders (depending on the servers boot configuration (BIOS or UEFI Boot)):

BIOS:
/boot/grub

UEFI:
/boot/efi/EFI/xenserver

Open grub.cfg with vi:
vi grub.cfg

Originally the file looks like this:

Now run the following command:

/opt/xensource/libexec/xen-cmdline –set-dom0 iommu=dom0-passthrough

After that it looks like this:

The iommu parameter is added to module2.

Although it looks like the parameter was added correctly, the error still occurred. I discussed it with Ronald Graß (Citrix Engineer). He thought that the parameter should be added to line multiboot2 (the “xen-line”) and not to module2. Thus I moved it up – the problem was fixed. Later I talked with a XenServer Engineer about it and he confirmed that the multiboot2 line is the correct one. After changing the grub.cfg looked like this:

Later I had a second system with the same problem. So I opened the grub.cfg and added iommu=Dom0-passthrough (the second method from the article). This time it did not work . We did many tests and even reinstalled the whole server – the error still occurred. When I compared the working one and the other one – there was only a really tiny difference between the grub.cfg files:
Working:

iommu=dom0-passthrough

Not working:

iommu=Dom0-passthrough

As you can see – the only difference was that the parameter in the not working one was with a capital D. So I replaced it with a lowercased d and rebooted the system. Booom – problem fixed. So the parameter is case-sensitive – write everything lower case.