Skip to content

Enable XenServer Live Migration for VMs with a NVIDIA vGPU

When you install NVIDIA Grid 6.0 on a XenServer you need to manually activate the possibility to Live-Migrate a VM which is using a (v)GPU. Otherwise, you see the following message when you try to migrate a VM:

The VGPU is not compatible with any PGPU in the destination

nvidia_enable_live_migration_01

The steps to enable the migration are described in the Grid vGPU User Guide. To enable the live migration you need to connect to the Console of the XenServer (eg using Putty). Login with the root user. The next step is to edit (create) the nvidia.conf in the folder etc/modprobe.d. Therefore, enter the following command:

vi /etc/modprobe.d/nvidia.conf

nvidia_enable_live_migration_02

Here you need to add the following line:

options nvidia NVreg_RegistryDwords="RMEnableVgpuMigration=1"

nvidia_enable_live_migration_03

Save the change with :wq and restart the Host. Repeat this for all Hosts in the Pool. After restarting you can now migrate a running VM with a (v)GPU on all Hosts that have the changed setting. If you haven’t configured the setting or didn’t reboot the Host only the other Hosts are available as a Target Server.

nvidia_enable_live_migration_04

New book: Citrix XenApp and XenDesktop 7.15 LTSR (German)

xa_xd_7_15_ltsr

My book about Citrix XenApp and XenDesktop 7.15 LTSR (German) is finally available in Store. It was a lot of work (and learnings) but also fun to test out some not so often used features. You can find more details here.

Update a single Citrix XenServer in a Pool

The Citrix XenCenter offers an easy way to update your XenServer (Pools). Furthermore, most updates can nowadays be installed without a reboot. Only a few require a reboot or a XE-Toolstack restart. This process is fully automated. This means that the first host is updated and (if necessary) rebooted. Then the second one follows and so on. To reboot a host the VMs are migrated to another host. This is fine until you use local storage for your VMs. (I don’t want to discuss here if this makes sense or not!) When the VMs are using local storage, they cannot be automatically migrated. When they are deployed using MCS they even can not be migrated manually (the base disk would be still on the initial host). In the past, this was not a problem. You could just put the VMs into maintenance mode and update one host.

Therefore, you started the Install Update Wizard, selected the updates you would like to install and had the possibility to select the host you would like to update:

xenserver_update_single_host_from_pool_01

When the update was finished, you disabled the Maintenance mode on the corresponding VMs and enabled it on the VMs on the next host. After some time the VMs from the second host were not in use any longer and you could update the second host. Since XenCenter 7.2, this is not possible any longer. After selecting an Update, you can only select to update the whole Pool:

xenserver_update_single_host_from_pool_02

Luckily, there is a small “workaround” to use the XenCenter to download the updates and copy them to the XenServer. (I really like the Update Overview in XenCenter – no hassle to look online which updates are available.) To use this workaround you start the Install Update Wizard and select the Update you would like to install.

xenserver_update_single_host_from_pool_03

In the next step, select the pool containing the server(s) on which you would like to install the update. Now the update is downloaded and transferred to the pool. It is important that you now do not press Next! When you close this dialog (Cancel), the update will be deleted from the Pool.

xenserver_update_single_host_from_pool_04

Instead, you need to connect to the console (e.g. using Putty) of a XenServer that is a member of the Pool. Now we need to figure out the UUID of the update. Therefore, you need to enter the following command:

xe update-list name-label=HOTFIXNAME

You can find the Name of the Update in the XenCenter Download Window. For example, Hotfix 11 for XenServer 7.1 CU1 has the Name XS71ECU1011. Copy the UUID and note if something is required after the hotfix is installed (after-apply-guidance). Either this can be a Toolstack restart (restartXAPI) or a host reboot (restartHost).

xenserver_update_single_host_from_pool_05

The next step is to install the Update on the required hosts. This can be achieved with the following command:

xe update-apply host=HOSTNAME uuid=PATCH-UUID

Replace HOSTNAME with the name of the XenServer you would like to update and PATCH-UUID with the copied UUID from the Patch. Repeat the same for all Hosts you would like to update. When the patch was applied, no further message is displayed.

xenserver_update_single_host_from_pool_06

That means you have to remember the after-apply-guidance which was shown with the Update UUID. The good thing is – if you have forgotten that you can check in XenServer if another step is necessary. Just open the General area from the Server. Under Updates you can see if an installed Update requires a Toolstack or Host restart. If a Toolstack or Host restart is necessary, do them to finish installing the update.

xenserver_update_single_host_from_pool_07

That is it – now you know how to install an Update to a single server on a XenServer Pool member. There are just two other things I would like to add.

The first is that you can install multiple updates at the same time. Therefore, you start with the same steps. You select an Update in the XenCenter and continue until it was transferred to the Pool. Now you go back with “Previous” to the Select Update area. Select the next Update and continue like before until the Update was transferred to the XenServer Pool. Repeat this for all updates you would like to install. Remember tha you not close the Update dialog – otherwise the update files will be removed from the Pool and you cannot install them any longer. Now note down the UUID from the updates and install all of them. It is not necessary to reboot after each update (which requires a Reboot of the Host / Restart of the Toolstack). Just install all and if an update (or multiple) require a reboot, reboot once at the end.

The next thing I would like to add is that you can also keep the files on the XenServer. Therefore, you need to kill the XenCenter trough the Task Manager when the Updates have been transferred to the Pool.

Reduced Performance with Citrix Linux Receiver 13.7 / 13.8 (compared to 13.5)

Today I would like to show you a current issue when upgrading your Linux Receiver 13.5 to 13.7 or 13.8. This is especially important for you when you are using Thin Clients. With a new Firmware a ThinClient Vendor often also updates the used Receiver Version. For example with one of the last Firmware’s Igel switched the default Receiver to Version 13.7. Unfortunately, it seems that there was a bigger change in the Linux Receiver, which leads to a performance reduction – especially when playing a video. This is a problem because it means that all HDX 3D Pro users are effected (HDX 3D Pro = H264 Stream). To show you the differences between both Receiver versions I created the following Video. Both screens are using the same Hardware / Firmware. Only the Linux Receiver Version was changed.

We currently have a ticket open at Citrix to fix the problem – but there is no solution until now available.

NVIDIA Grid Cards – OEM “List” Price Comparison

In a discussion with a smart guy from NVIDIA he showed me Thinkmate.com as a nice source for Supermicro Servers (and Grid Cards). The visible prices for the Grid cards were kind of “interesting” – so I thought a comparison of the somewhere public visible list prices for Grid cards from the different OEMs might be interesting. Prices might change in the future. In addition you need for some cards a separate Cable Kit. All Prices from 13.02.2018. (Don’t forget – that are list prices!)

OEM M10 M60 P4 P40
Dell $5,050.56 $9,821.84 $4,020.79 $12,305.72
Supermicro $2,299.00 $4,399.00 $1,899.00 $5,699.00
HP $3,999.00 $8,999.00 $3,699.00 $11,599.00
Cisco $8,750.00 $16,873.00 $6,999.00 $21,000.00
Lenovo ?? $9,649.00 ?? ??

If you know a source for the missing Lenovo List Prices – please let me know so I can add them.

Sources:
Dell:
M10: http://www.dell.com/en-us/shop/accessories/apd/490-bdig
M60: http://www.dell.com/en-us/work/shop/accessories/apd/490-bcwc
P4: http://www.dell.com/en-us/work/shop/accessories/apd/489-bbcp
P40: http://www.dell.com/en-us/work/shop/accessories/apd/489-bbco

Supermicro:
https://www.thinkmate.com/system/superserver-2029gp-tr

HP:
https://h22174.www2.hpe.com/SimplifiedConfig/Welcome
=> ProLiant DL Servers => ProLiant DL300 Servers => HPE ProLiant DL380 Gen10 Server => Configure (just choose one) => Graphics Options

Cisco:
http://itprice.com/cisco-gpl/GPU

Lenovo:
http://itprice.com/lenovo-price-list/nvidia.html

NVIDIA Grid Version Overview

When you have a look at Grid-Versioning you might getting confused by the Official Version Number, the Hypervisor Driver-Number and the VM-Driver Number. Especially when you have multiple systems and cannot upgrade all systems at the same time or would like to test new versions, it’s quite helpful to know which Grid Version is which Hypervisor- and VM-Driver version. Thus I created a simple table to show the corresponding versions:

GRID Version Hypervisor-Driver VM-Treiber (Windows) VM-Treiber (Linux)
4.0 367.43 369.17
4.1 367.64 369.61
4.2 367.92 369.95
4.5 367.123 370.17
4.6 367.124 370.21
5.0 384.73 385.41 384.73
5.1 384.99 385.90 384.73
5.2 384.111 386.09 384.111

Data comparison of NVIDIA GRID Tesla P4 and P40

For my book about NVIDIA GRID I created a Data comparison table of the two graphics cards P4 and P40. From my point of view the P4 is a real underestimated card. After my presentation at NVIDIA GTC Europe this year NVIDIA added the P4 to their comparison slides – thanks for listening. During DCUG TecCon I showed my Data comparison table and got some real positive feedback. So I thought it’s worth to publish the comparison in a blog post:

P40 (3X)

P4 (6X)

GPU

3X Pascal

6X Pascal

CUDA Cores (per Card)

3840

2560

CUDA Cores (Total)

11520

15360

Frame Buffer (Total)

72GB

48GB

H.264 1080p30 Streams (Total)

75

150

Max vGPU (Total)

72

48

Max Power (per Card)

250W

75W

Max Power (Total)

750W

450W

Price per Card (in $)*

11149,99

3649,99

Price for all Cards (in $)*

33449,97

21899,94

*NVIDIA doesn’t publish list prices – so I picked list prices from one server vendor.

Current server generations allow either the usage of six P4 or three P40. As you can see the price of the P4 is much lower. On the other side the available CUDE Cores on the P40 are higher. Thus one user could get a higher peak performance on a P40 card. On the other hand you have more CUDE cores available on all P4 cards compared to all P40 cards. With the P4 cards you are limited to 48 vGPU Instances – the P40 allows up to 72.

There is another fact most people don’t think about. Citrix uses H264 in it’s HDX 3D Pro Protocol. When a user now connects to a VM one stream is created for every monitor he is using. In many offices nearly every user already has two monitors – resulting in two streams. If you now connect 72 users to a P40 and every user has two monitors – it would require 144 streams. However there are only 75 available. This can lead to a reduced performance for the user. In contrast the P4 has 150 streams available – for 48 vGPU Instances.

In addition the maximum power of all six P4 is 300W lower than three P40 resulting in lower cooling requirements and power needs for the Data Center.

I hope you like the comparison – if you think something is wrong or missing please contact me.

%d bloggers like this: