No vGPU is available with NVIDIA Tesla M60 on Citrix Xenserver

During the last days I had to invest an interesting problem. On a XenServer there were no vGPUs available on a Tesla M60 from NVIDIA. Only GPU Pass-through was available.

Ok – time to check if the necessary supplemental pack is installed:

As you can see this was installed – so maybe the Tesla is running in Compute-Mode instead of Graphics-Mode. To check this, connect to the console of the XenServer and login. Now run the following command:

lspci -n | grep 10de

This will show you all available cards with their current configuration (compute or gpu):

If the marked value is 0300 everything is fine and the card is in graphics mode. If it’s 0302 it’s in compute mode and you have to change it to graphics mode.

So it looks everything correct – but vGPUs are still not available. Possible it’s just a problem with the NVIDIA vGPU pack – let’s reinstall it with the XenCenter.
Tools => Install Update
Choose “Select update or supplement pack from disk” and browse to the vGPU Iso from the Nvidia Driver Download Package.

Continue with the installation – but what’s that:

The installation fails (without a detailed information why).

So time for the manual way– copy the vGPU rpm File to the XenServer with WinSCP.

And start the installation with the following command (after browsing to the correct folder):

rpm -Uv NVIDIA-vGPU-xenserver-7.0-361.45.09.x86_64.rpm

But guess what happened again? The installation failed with the following messages:

Preparing packages…
file /usr/share/doc/NVIDIA_VGX/LICENSE from install of NVIDIA-vGPU-xenserver-7.0-361.45.09.x86_64 conflicts with file from package NVIDIA-vGPU-kepler-xenserver-7.0-361.45.09.x86_64
file /usr/share/hwdata/pci.ids.d/nvidia.ids from install of NVIDIA-vGPU-xenserver-7.0-361.45.09.x86_64 conflicts with file from package NVIDIA-vGPU-kepler-xenserver-7.0-361.45.09.x86_64
file /usr/share/nvidia/monitoring.conf from install of NVIDIA-vGPU-xenserver-7.0-361.45.09.x86_64 conflicts with file from package NVIDIA-vGPU-kepler-xenserver-7.0-361.45.09.x86_64
file /usr/share/nvidia/vgpu/vgpuConfig.xml from install of NVIDIA-vGPU-xenserver-7.0-361.45.09.x86_64 conflicts with file from package NVIDIA-vGPU-kepler-xenserver-7.0-361.45.09.x86_64

Luckily this time the error message is much more helpful – it tells that there is a conflict with the NVIDIA-vGPU-kepler-xenserver-7.0-361.45.09.x86_64 file.

So why can’t I upgrade the vGPU Package? The most important part is the word KEPLER in the conflicting file. A lot of people don’t know (and it’s not well documented until now) that NVIDA deploys two different versions of the vGPU Package. One is valid for GRID K1 and K2 – the other one for the Tesla cards. So at the end you might have two packages:
K1 + K2:
NVIDIA-vGPU-kepler-xenserver-VERSION-NUMBER
Tesla:
NVIDIA-vGPU-xenserver-VERSION-NUMBER

If you now think – ok just remove the K1/K2 package – sorry that’s (still) not supported in XenServer (XenServer 7.0 – Point 7.4):

At present, uninstallation of supplemental packs is not supported. However, to ensure that it is simple to provide such a feature in the future, all RPMs that are included in supplemental packs must be able to be uninstalled cleanly (using rpm -e). This includes reverting any configuration such as firewall rules or log rotation configuration.

The only (supported) solution is to completely reinstall the XenServer. After that install the Tesla vGPU Package and you should be fine and see all the available vGPU Types:

Hope that helps some of you to know that there is a difference between the K1/K2 and Tesla package.

There are only a few questions which are still open for me at the moment:
Why is it necessary to switch from one vGPU Type (K1/K2 to Tesla) to reinstall the XenServer?
What happens if a customer would like to use a K2 and Tesla M60 in one physical Server?