Workaround for crashing XenServer 7 with Xeon v4 CPUs, Nvidia M60 / M10 (and Dell R730)

You might have read my last article about a crashing XenServer 7 when it’s used with an Intel Xeon v4 and a NVIDIA M60 in a Dell R730. The same problem happens when you use a M10 from NVIDIA. Furthermore I have heard that the same problem happens with Fujitsu Servers.

Our workaround until now was to replace the Xeon v4 CPUs with v3 ones (with less performance). Luckily Citrix Engineering (and Support) found the issue for the problem. It’s related to the Intel PML (Page Modification Logging) function. You can find some detailed information on this side from Intel.

Thus we disabled this function in XenServer and replaced the v3 CPUs with v4 ones. After that we started stress testing the system. No error occurred. Thus we placed users on the system. The system was still running. Until now we had no more crashes while the function is disabled.

To disable the function you need to modify the grub.cfg – those of you who have more than 512GB Ram in their hosts should already now that process.

Open a console session and either switch to /boot/grub (Bios) or /boot/efi/EFI/xenserver (UEFI) – depending on the boot setting of your server.

BIOS:
/boot/grub

UEFI:
/boot/efi/EFI/xenserver

Open grub.cfg with vi:
vi grub.cfg

Now you need to add the following command to the multiboot2 /boot/xen.gz line:

ept=no-pml

After that the file should look like this (don’t remove iommu… if you have more than 512GB Ram):

Save and close the file with :wq and reboot the host. That’s it. Now the function is disabled and the problem is gone. A big thanks to Antony Peter (Citrix Support) and Anshul Makkar (Citrix Engineering) for taking so much time to look at our environment and debugging the problem with me.