In this blog post I would like to show you how to configure a Nvidia Tesla M60 under a XenServer and deploy a VM with a vGPU assigned. There are a lot of good documentations from Nvidia for the different steps – but I didn’t find one complete for the whole process after putting the Tesla into the physical Server and installing XenServer.

In this post I will cover the following areas:
Installing the XenServer Nvidia vGPU Supplemental pack
Changing the Tesla M60 from Compute to Graphics Mode
Installing and Configuring the Nvidia Licensing Service
Configure a VM with a vGPU and assign a license

Installing the XenServer Nvidia vGPU Supplemental Pack

After the installation of the XenServer (I am not going to cover that here – there are enough other resources describing this) the necessary Nvidia supplemental pack is not available. You can see the installed packs in the XenCenter under the Server General Properties => Updates.

To download the necessary package you need an Nvidia Portal Account. Interestingly at the moment there is no Register-Button on the Login-Page. You can register on this page. The problem only is that you require either a valid Key or a Demo-Key. It’s not possible to get one here. So you need to contact an Nvidia Sales Guy to get a Demo key. After registering and logging in you have the area Product Download available. Here you can download the necessary extensions for XenServer, ESX and so on. Download the package corresponding to your XenServer version (currently 6.2, 6.5 and 7).

The download is a zip file containing different documentations, packs and applications. First of all you only need the following file:
NVIDIA-vGPU-xenserver-VERSION-NUMBER.x86_64.iso

To install the supplemental pack open the XenCenter and navigate to Tools => Install Update…

Confirm the Starting guide with Next.

Now you have to select the ISO file lower area Select update or supplemental pack from disk.
Important:
There are different supplemental packs for the different Nvidia vGPU cards available. One is for K1/K2 – the other for Tesla cards. The name of the package is different:
K1/K2: Contains the word Keplar
Tesla: without Keplar (see screenshot).
At the moment it is not supported to run cards of both types in the same server (thanks @Rachel Berry for the information).
Furthermore it is currently not possible to remove an installed pack – so if you have a Tesla card and accidentally installed the Keplar pack you have to reinstall your XenServer.

After choosing the ISO select on which XenServers you would like to install the pack.

The pack will be uploaded to the XenServer.

After installing the updates some tasks have to be done (e.g. rebooting) – you can either let that happen automatically (from the XenCenter) or do that manual (like you know from every other update).

The installation starts, VMs (if there are any) are migrated off the XenServer and it is rebooted (if you selected automatically before ).

After the reboot the Maintenance mode is quite and some checks are done.

If you now check the Updates section again you should see the NVIDIA vGPU package.

Moreover you can run the following command on the console to check if the pack is correctly installed:

lsmod | grep nvidia

If everything is correct the output should look like this.

When you now open the GPU area of the XenServer it might be possible that still no vGPU Profiles are shown on the right side – also the Tesla Card is correctly detected. The reason for this is that the Tesla card is often shipped in Compute mode and not in Graphics mode.

Changing the Tesla M60 from Compute to Graphics Mode

To check in which mode the Tesla card is running open the XenCenter Console. Now enter the following command:

lspci -n | grep 10de

The red marked area shows the mode:
0302: Compute
0300: Graphics

To switch the mode, you need an Utility from Nvidia. You can either Download this from the Product Download page (after logging in) or under the Nvidia support download for the Tesla cards.

The downloaded Zip-Files contains the following files. There are different options how to change the GPU mode (e.g. install Linux) – to make it easy we will integrate the utility in the XenServer. Thus we only need the file gpumodeswitch.
Important: Not the file with the extension exe – the file without an extension (the first in the screenshot).

To copy the file to your XenServer you need to start WinSCP. Select SFTP, enter the XenServer IP-Address as the Host name. Username is root with the same password you use in the XenCenter.

Confirm the Warning.

Now copy the gpumodeswitch file to a temporary folder (e.g. tmp) on the XenServer.

Switch to the command line from the XenServer and navigate to the tmp folder. To show the current GPU mode enter the following command:

./gpumodeswitch –listgpumodes

But what’s that – the permission is denied.

To allow the execution of the file you have to modify the file permission. Enter the following command:

chmod 700 gpumodeswitch

Now you should be able to execute the gpumodeswtich utility. After the execution you get another warning that first the NVIDIA kernel driver has to be unloaded.

Before you can remove the driver first stop the xcp-rrdd-gpumon service:

service xcp-rrdd-gpumon stop

Now remove the unload the NVIDIA driver.

rmmod nvidia

You can now change the graphics mode with this command:

./gpumodeswitch –gpumode graphics

The mode change needs to be confirmed with Y.

The Tesla cards now receive the graphics mode firmware.

After the update finished you can again check if the graphics mode is now active.

lspci –n | grep 10de

In the red marked area you now see a 300 for all Tesla cards – the Graphics mode is now active.

If you switch to the GPU area from the XenServer the different vGPUs are shown.

Installing and Configuring the Nvidia Licensing Service

The next step is to install and configure the Nvidia Licensing service. The necessary downloads are also available in the Nvidia portal under Product Download.

The package contains a setup.exe and two documents. Start the installation with the setup.exe

If you haven’t read the requirements you might have missed that the service requires a Java 32Bit installation.

So first install Java….

And then start the setup again.

Accept the license agreement for the Nvidia software.

And also accept the license for Apache – an Apache Tomcat is used for the license service.

You can now change the installation path (if you like).

The default setting is to configure the Windows Firewall to only allow connections to the license service itself – not to the management website. If you want to access the website not only from the license server itself (and the Windows Firewall is active) you need to activate the Management interface as well.

A summary shows the installation path and the required disk space.

The license service (including Apache Tomcat) is now installed.

Finish the installation with Done.

The first check is now if the tomcat is running. Open a browser on the server and open http://localhost:8080 – you should now see the Tomcat welcome page if everything is running correctly.

To open the Nvidia license server enter http://localhost/8080/licserver
The first things shown are the Licensed Clients – of course there is no client shown until now because we didn’t configure a client or install a license.

The licenses are managed under License Management. As you can see you need to upload a license file.

To get the license file browse again to the Nvidia portal and switch do License History. Here you can see all of your licenses. Select the licenses you would like to assign.

For assigning the licenses you need the MAC address of the license server.

To get the MAC open a Command prompt or a Windows PowerShell on the licensing server. Enter the following command:

ipconfig /all

Search for the Ethernet-Connection you plan to use and note the Physical Address.

Enter the Physical Address in the MAC address field (without any signs like “-“ or “:”). Add a free choose able Alias and Site Name.

The server is now created – but no licenses are assigned. Choose Map Add-Ons to add a license.

Press Search to view your current licenses. Enter the Qty to Add of the licenses you want to assign and press Map Add-Ons.

Now the licenses are mapped to the server. Select Download License File to download the license BIN file.

Put the downloaded BIN file on a folder on the license server.

Now open again the license management console and go to License Management.

Select the downloaded BIN file and press Upload.

A sucessful upload is confirmed with Successfully applied license file to license server.

Go to License Feature Usage to show the currently available licenses.

Configure a VM with a vGPU and assign a license

The last configuration step is to assign a vGPU and install the Nvidia Drivers inside the VM.

Open the properties of a VM (I am not going to cover how to create a VM and install Windows on the VM) and switch to GPU.
Now you have to select the vGPU-Profile.
Depending on the profile you need a different license. The profiles are split into the following licenses:
A: Grid Virtual Applications
B: Grid Virtual PC
C: Grid Virtual Workstation
You can find a good comparison of the types in this blog post from Richard Hoffman.

When you now start the VM and check the Device Manager you will still only see a Default VGA Graphics Card. To change that the necessary drivers need to be installed.

Copy the ones for your OS to the VM.

Start the setup and confirm the Windows UAC Message with yes.

Select a folder to extract the driver files.

Now the actual setup starts. Agree to the licenses,

Either choose Express or Custom installation.

There is no difference between the installation except that in a Custom installation you can see which components are installed and it’s possible to Perform a clean installation if you had problems with the currently installed driver.

The installation starts.

After the installation is finished a reboot is necessary.

If you now open the Device Manager again and check the Graphics Card you will see a NVIDIA GRID card – here you can also see which vGPU-Type is assigned.

The last step is to assign a license to the VM. Open the Control Panel and search for Nvidia.

Open the Nvidia Control Panel.

Now select License management. Here you can enter License Server FQDN and the configured Port (default: 7070).

After entering the information’s press Apply to assign a license.

If everything is working correctly the license is now applied.

If you now open the License Server Management Interface again you will find the client under Licensed Clients.

For each client you can also open get detailed information’s.

That’s it. Now you can continue with everything else (e.g. Citrix XenDesktop HDX 3D Pro VDA). Hope the article helped some of you