CloudStack 4.22 – GPU passthrough VM creation fails due to invalid PCI slot mapping

[amitcsc1989]

problem

We are running CloudStack 4.22 with KVM hypervisors and NVIDIA GPUs (vendor 10de, device 233b). GPU discovery works correctly — devices are listed in the gpu_device table and show up as Free/Allocated in the UI. When attempting to create a VM with GPU passthrough, CloudStack fails to start the VM with the following error:"LibvirtException: XML error: Invalid PCI address slot='0xd6', must be <= 0x1F"

versions

Cloudstack 4.22

The steps to reproduce the bug

...

What to do about it?

No response

[amitcsc1989]

i have checked GPU discovery is posting additional PCI slot domain 0000 in discovery. when i 'am trying to add one more vm then it is not allowing beacause already 2 vm's are running same domain id 0000. it should have free starting with 0001 domain . Kindly look into this .

[vishesh92]

@amitcsc1989 I didn't understand this. How many cards are on the host & how many are allocated? Can you share some screenshot?

i have checked GPU discovery is posting additional PCI slot domain 0000 in discovery.

Also, can you confirm if you have xmlstarlet installed on the host?

[amitcsc1989]

Hi , Please find the attached screenshot .

when it is attaching 0001:d6:00.0 or 0001:d8:00.0 for new vm then it is throghing error "LibvirtException: XML error: Invalid PCI address slot='0xd6', must be <= 0x1F". xmlstarlet is already installed .

[vishesh92]

@amitcsc1989 Please share the output of the gpudiscovery.sh from the host.

[amitcsc1989]

Hi , Please find the details ../usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/gpudiscovery.sh

Output

```json { "gpus": [ { "pci_address":"ae:00.0", "vendor_id":"10de", "device_id":"233b", "vendor":"NVIDIA Corporation", "device":"Device", "driver":"vfio-pci", "pci_class":"3D controller [0302]", "iommu_group":"35", "pci_root":"0000:ab:02.0", "numa_node":0, "sriov_totalvfs":32, "sriov_numvfs":0, "max_instances":null, "video_ram":null, "max_heads":null, "max_resolution_x":null, "max_resolution_y":null,

  "full_passthrough": {
    "enabled":1,
    "libvirt_address": {
      "domain":"0x0000",
      "bus":"0xae",
      "slot":"0x00",
      "function":"0x0"
    },
    "used_by_vm":"i-2-385-VM"
  },

  "vgpu_instances":[],
  "vf_instances":[]
}

,
{
"pci_address":"b0:00.0",
"vendor_id":"10de",
"device_id":"233b",
"vendor":"NVIDIA Corporation",
"device":"Device",
"driver":"vfio-pci",
"pci_class":"3D controller [0302]",
"iommu_group":"36",
"pci_root":"0000:ab:02.0",
"numa_node":0,
"sriov_totalvfs":32,
"sriov_numvfs":0,
"max_instances":null,
"video_ram":null,
"max_heads":null,
"max_resolution_x":null,
"max_resolution_y":null,

  "full_passthrough": {
    "enabled":1,
    "libvirt_address": {
      "domain":"0x0000",
      "bus":"0xb0",
      "slot":"0x00",
      "function":"0x0"
    },
    "used_by_vm":"i-2-361-VM"
  },

  "vgpu_instances":[],
  "vf_instances":[]
}

,
{
"pci_address":"0001:d6:00.0",
"vendor_id":"10de",
"device_id":"233b",
"vendor":"NVIDIA Corporation",
"device":"Device",
"driver":"unknown",
"pci_class":"3D controller [0302]",
"iommu_group":"null",
"pci_root":"0000:0001:d6:00.0",
"numa_node":-1,
"sriov_totalvfs":0,
"sriov_numvfs":0,
"max_instances":null,
"video_ram":null,
"max_heads":null,
"max_resolution_x":null,
"max_resolution_y":null,

  "full_passthrough": {
    "enabled":1,
    "libvirt_address": {
      "domain":"0x0000",
      "bus":"0x00",
      "slot":"0x1:",
      "function":"0x6"
    },
    "used_by_vm":null
  },

  "vgpu_instances":[],
  "vf_instances":[]
}

,
{
"pci_address":"0001:d8:00.0",
"vendor_id":"10de",
"device_id":"233b",
"vendor":"NVIDIA Corporation",
"device":"Device",
"driver":"unknown",
"pci_class":"3D controller [0302]",
"iommu_group":"null",
"pci_root":"0000:0001:d8:00.0",
"numa_node":-1,
"sriov_totalvfs":0,
"sriov_numvfs":0,
"max_instances":null,
"video_ram":null,
"max_heads":null,
"max_resolution_x":null,
"max_resolution_y":null,

  "full_passthrough": {
    "enabled":1,
    "libvirt_address": {
      "domain":"0x0000",
      "bus":"0x00",
      "slot":"0x1:",
      "function":"0x8"
    },
    "used_by_vm":null
  },

  "vgpu_instances":[],
  "vf_instances":[]
}

]}

</p>
</details> 

[vishesh92]

@amitcsc1989 I checked the code and it seems to be a bug. We seem to have used 0000 domain as the default. Let me try to create a fix.

[amitcsc1989]

Thanks Vishesh , I'm waiting for the fix .

[vishesh92]

@amitcsc1989 I won't be able to reproduce this on my current hardware. Will you be able to help me with testing this?

[amitcsc1989]

sure . please let me know how i can help or any additional details do you want from my side .

[vishesh92]

Thanks @amitcsc1989 . I will share the jar and the bash file for testing once I have the fix ready.

[amitcsc1989]

ok . i hope that will not imact running VM's . Please share i will run that and will share the output . kindly let me know the process as well to run that jar and bash script .