Skip to content

Configuring a Nvidia Tesla M60 under Citrix XenServer (and assign a vGPU to a VM)

18. July 2016

In this blog post I would like to show you how to configure a Nvidia Tesla M60 under a XenServer and deploy a VM with a vGPU assigned. There are a lot of good documentations from Nvidia for the different steps – but I didn’t find one complete for the whole process after putting the Tesla into the physical Server and installing XenServer.

In this post I will cover the following areas:
Installing the XenServer Nvidia vGPU Supplemental pack
Changing the Tesla M60 from Compute to Graphics Mode
Installing and Configuring the Nvidia Licensing Service
Configure a VM with a vGPU and assign a license

Installing the XenServer Nvidia vGPU Supplemental Pack

After the installation of the XenServer (I am not going to cover that here – there are enough other resources describing this) the necessary Nvidia supplemental pack is not available. You can see the installed packs in the XenCenter under the Server General Properties => Updates.
nvidia_tesla_m60_xen_01

To download the necessary package you need an Nvidia Portal Account. Interestingly at the moment there is no Register-Button on the Login-Page. You can register on this page. The problem only is that you require either a valid Key or a Demo-Key. It’s not possible to get one here. So you need to contact an Nvidia Sales Guy to get a Demo key. After registering and logging in you have the area Product Download available. Here you can download the necessary extensions for XenServer, ESX and so on. Download the package corresponding to your XenServer version (currently 6.2, 6.5 and 7).
nvidia_tesla_m60_xen_02
The download is a zip file containing different documentations, packs and applications. First of all you only need the following file:
NVIDIA-vGPU-xenserver-VERSION-NUMBER.x86_64.iso

To install the supplemental pack open the XenCenter and navigate to Tools => Install Update…
nvidia_tesla_m60_xen_03

Confirm the Starting guide with Next.
nvidia_tesla_m60_xen_04

Now you have to select the ISO file lower area Select update or supplemental pack from disk.
Important:
There are different supplemental packs for the different Nvidia vGPU cards available. One is for K1/K2 – the other for Tesla cards. The name of the package is different:
K1/K2: Contains the word Keplar
Tesla: without Keplar (see screenshot).
At the moment it is not supported to run cards of both types in the same server (thanks @Rachel Berry for the information).
Furthermore it is currently not possible to remove an installed pack – so if you have a Tesla card and accidentally installed the Keplar pack you have to reinstall your XenServer.
nvidia_tesla_m60_xen_05

After choosing the ISO select on which XenServers you would like to install the pack.
nvidia_tesla_m60_xen_06

The pack will be uploaded to the XenServer.
nvidia_tesla_m60_xen_07

After installing the updates some tasks have to be done (e.g. rebooting) – you can either let that happen automatically (from the XenCenter) or do that manual (like you know from every other update).
nvidia_tesla_m60_xen_08

The installation starts, VMs (if there are any) are migrated off the XenServer and it is rebooted (if you selected automatically before Winking smile).
nvidia_tesla_m60_xen_09

After the reboot the Maintenance mode is quite and some checks are done.
nvidia_tesla_m60_xen_10

If you now check the Updates section again you should see the NVIDIA vGPU package.
nvidia_tesla_m60_xen_11

Moreover you can run the following command on the console to check if the pack is correctly installed:

lsmod | grep nvidia

If everything is correct the output should look like this.
nvidia_tesla_m60_xen_13

When you now open the GPU area of the XenServer it might be possible that still no vGPU Profiles are shown on the right side – also the Tesla Card is correctly detected. The reason for this is that the Tesla card is often shipped in Compute mode and not in Graphics mode.
nvidia_tesla_m60_xen_14

Changing the Tesla M60 from Compute to Graphics Mode

To check in which mode the Tesla card is running open the XenCenter Console. Now enter the following command:

lspci -n | grep 10de

The red marked area shows the mode:
0302: Compute
0300: Graphics
nvidia_tesla_m60_xen_15

To switch the mode, you need an Utility from Nvidia. You can either Download this from the Product Download page (after logging in) or under the Nvidia support download for the Tesla cards.
nvidia_tesla_m60_xen_16

The downloaded Zip-Files contains the following files. There are different options how to change the GPU mode (e.g. install Linux) – to make it easy we will integrate the utility in the XenServer. Thus we only need the file gpumodeswitch.
Important: Not the file with the extension exe – the file without an extension (the first in the screenshot).
nvidia_tesla_m60_xen_17

To copy the file to your XenServer you need to start WinSCP. Select SFTP, enter the XenServer IP-Address as the Host name. Username is root with the same password you use in the XenCenter.
nvidia_tesla_m60_xen_18

Confirm the Warning.
nvidia_tesla_m60_xen_19

Now copy the gpumodeswitch file to a temporary folder (e.g. tmp) on the XenServer.
nvidia_tesla_m60_xen_20

Switch to the command line from the XenServer and navigate to the tmp folder. To show the current GPU mode enter the following command:

./gpumodeswitch –listgpumodes

But what’s that – the permission is denied.
nvidia_tesla_m60_xen_21

To allow the execution of the file you have to modify the file permission. Enter the following command:

chmod 700 gpumodeswitch

Now you should be able to execute the gpumodeswtich utility. After the execution you get another warning that first the NVIDIA kernel driver has to be unloaded.
nvidia_tesla_m60_xen_22

Before you can remove the driver first stop the xcp-rrdd-gpumon service:

service xcp-rrdd-gpumon stop

Now remove the unload the NVIDIA driver.

rmmod nvidia

nvidia_tesla_m60_xen_23

You can now change the graphics mode with this command:

./gpumodeswitch –gpumode graphics

The mode change needs to be confirmed with Y.
nvidia_tesla_m60_xen_24

The Tesla cards now receive the graphics mode firmware.
nvidia_tesla_m60_xen_25

After the update finished you can again check if the graphics mode is now active.

lspci –n | grep 10de

In the red marked area you now see a 300 for all Tesla cards – the Graphics mode is now active.
nvidia_tesla_m60_xen_26

If you switch to the GPU area from the XenServer the different vGPUs are shown.
nvidia_tesla_m60_xen_27

Installing and Configuring the Nvidia Licensing Service

The next step is to install and configure the Nvidia Licensing service. The necessary downloads are also available in the Nvidia portal under Product Download.

nvidia_tesla_m60_xen_28

The package contains a setup.exe and two documents. Start the installation with the setup.exe
nvidia_tesla_m60_xen_29

If you haven’t read the requirements you might have missed that the service requires a Java 32Bit installation.
nvidia_tesla_m60_xen_30

So first install Java….
nvidia_tesla_m60_xen_31

And then start the setup again.
nvidia_tesla_m60_xen_32

Accept the license agreement for the Nvidia software.
nvidia_tesla_m60_xen_33

And also accept the license for Apache – an Apache Tomcat is used for the license service.
nvidia_tesla_m60_xen_34

You can now change the installation path (if you like).
nvidia_tesla_m60_xen_35

The default setting is to configure the Windows Firewall to only allow connections to the license service itself – not to the management website. If you want to access the website not only from the license server itself (and the Windows Firewall is active) you need to activate the Management interface as well.
nvidia_tesla_m60_xen_36

A summary shows the installation path and the required disk space.
nvidia_tesla_m60_xen_37

The license service (including Apache Tomcat) is now installed.
nvidia_tesla_m60_xen_38

Finish the installation with Done.
nvidia_tesla_m60_xen_39

The first check is now if the tomcat is running. Open a browser on the server and open http://localhost:8080 – you should now see the Tomcat welcome page if everything is running correctly.
nvidia_tesla_m60_xen_40

To open the Nvidia license server enter http://localhost/8080/licserver
The first things shown are the Licensed Clients – of course there is no client shown until now because we didn’t configure a client or install a license.
nvidia_tesla_m60_xen_41

The licenses are managed under License Management. As you can see you need to upload a license file.
nvidia_tesla_m60_xen_42

To get the license file browse again to the Nvidia portal and switch do License History. Here you can see all of your licenses. Select the licenses you would like to assign.
nvidia_tesla_m60_xen_43

For assigning the licenses you need the MAC address of the license server.
nvidia_tesla_m60_xen_44

To get the MAC open a Command prompt or a Windows PowerShell on the licensing server. Enter the following command:

ipconfig /all

Search for the Ethernet-Connection you plan to use and note the Physical Address.
nvidia_tesla_m60_xen_45

Enter the Physical Address in the MAC address field (without any signs like “-“ or “:”). Add a free choose able Alias and Site Name.
nvidia_tesla_m60_xen_46

The server is now created – but no licenses are assigned. Choose Map Add-Ons to add a license.
nvidia_tesla_m60_xen_47

Press Search to view your current licenses. Enter the Qty to Add of the licenses you want to assign and press Map Add-Ons.
nvidia_tesla_m60_xen_48

Now the licenses are mapped to the server. Select Download License File to download the license BIN file.
nvidia_tesla_m60_xen_49

Put the downloaded BIN file on a folder on the license server.
nvidia_tesla_m60_xen_50

Now open again the license management console and go to License Management.
nvidia_tesla_m60_xen_51

Select the downloaded BIN file and press Upload.
nvidia_tesla_m60_xen_52

A sucessful upload is confirmed with Successfully applied license file to license server.
nvidia_tesla_m60_xen_53

Go to License Feature Usage to show the currently available licenses.
nvidia_tesla_m60_xen_54

Configure a VM with a vGPU and assign a license

The last configuration step is to assign a vGPU and install the Nvidia Drivers inside the VM.

Open the properties of a VM (I am not going to cover how to create a VM and install Windows on the VM) and switch to GPU.
Now you have to select the vGPU-Profile.
Depending on the profile you need a different license. The profiles are split into the following licenses:
A: Grid Virtual Applications
B: Grid Virtual PC
C: Grid Virtual Workstation

You can find a good comparison of the types in this blog post from Richard Hoffman.
nvidia_tesla_m60_xen_55

When you now start the VM and check the Device Manager you will still only see a Default VGA Graphics Card. To change that the necessary drivers need to be installed.
nvidia_tesla_m60_xen_56

Copy the ones for your OS to the VM.
nvidia_tesla_m60_xen_57

Start the setup and confirm the Windows UAC Message with yes.
nvidia_tesla_m60_xen_58

Select a folder to extract the driver files.
nvidia_tesla_m60_xen_59

Now the actual setup starts. Agree to the licenses,
nvidia_tesla_m60_xen_60

Either choose Express or Custom installation.
nvidia_tesla_m60_xen_61

There is no difference between the installation except that in a Custom installation you can see which components are installed and it’s possible to Perform a clean installation if you had problems with the currently installed driver.
nvidia_tesla_m60_xen_62

The installation starts.
nvidia_tesla_m60_xen_63

After the installation is finished a reboot is necessary.
nvidia_tesla_m60_xen_64

If you now open the Device Manager again and check the Graphics Card you will see a NVIDIA GRID card – here you can also see which vGPU-Type is assigned.
nvidia_tesla_m60_xen_65

The last step is to assign a license to the VM. Open the Control Panel and search for Nvidia.
nvidia_tesla_m60_xen_66

Open the Nvidia Control Panel.
nvidia_tesla_m60_xen_67

Now select License management. Here you can enter License Server FQDN and the configured Port (default: 7070).
nvidia_tesla_m60_xen_68

After entering the information’s press Apply to assign a license.
nvidia_tesla_m60_xen_69

If everything is working correctly the license is now applied.
nvidia_tesla_m60_xen_70

If you now open the License Server Management Interface again you will find the client under Licensed Clients.
nvidia_tesla_m60_xen_71

For each client you can also open get detailed information’s.
nvidia_tesla_m60_xen_72

That’s it. Now you can continue with everything else (e.g. Citrix XenDesktop HDX 3D Pro VDA). Hope the article helped some of you Smile

6 Comments
  1. Tobias K permalink

    Jan: Excellent compilation of information and many thanks for taking the time to do such a thorough job!

  2. Alessandro Marques permalink

    Your blog has helped us solve a deployment problem! Thank you!

  3. Remi S permalink

    Wow thank you for this great write-up! I’m starting a deployment next week for a customer, first time in 3D with XenDesktop on XenServer, I feel a lot more confident now 🙂

Trackbacks & Pingbacks

  1. EUC Weekly Digest – July 23, 2016 – Carl Stalhood
  2. Crysis на максималках, или зачем серверу видеокарта — Восстановленные б/у серверы HP, IBM, Dell до 7 раз дешевле новыхВосстановленные б/у серверы

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: