Xenserver mount not working

[miesiu]

when trying to mount the disk I get the following message, is there any solution?

MountVolume.MountDevice failed for volume "pvc-19c4c40c-6c7e-450a-a99a-277cc11479fd" : rpc error: code = Internal desc = Cannot find device path for volume e1ff8220-1e76-455f-bfef-ef6b5ba2e014: Failed to find device for the volumeID: "e1ff8220-1e76-455f-bfef-ef6b5ba2e014" within the alloted time

Warning FailedMount 49s (x8 over 16m) kubelet Unable to attach or mount volumes: unmounted volumes=[example-volume], unattached volumes=[example-volume kube-api-access-g8mbx]: timed out waiting for the condition

[olivierlemasle]

Hi @miesiu, thank you for the report.

Currently, this CSI driver works only with KVM and has not yet been tested with Xenserver.

With KVM, I've found no direct link between CloudStack volume deviceId and the device id in the VM.

That's why this CSI driver identifies the disk device by its device ID (as found in /dev/disk/by-id), which contains the "serial" of the volume, which is itself generated from the CloudStack volume ID. This is specific to KVM - cf https://github.com/apalia/cloudstack-csi-driver/blob/master/pkg/mount/mount.go#L129-L130

However, CloudStack seems to define a direct link (in here):

deviceId
0	/dev/xvda
1	/dev/xvdb
2	/dev/xvdc
4	/dev/xvde
5	/dev/xvdf
...	...

I'll try to get access to a CloudStack + XenServer environment, but could you please tell me if the "link" in the table above really works?

In the following scenario:

1. I create a VM, with no data volume (only a root volume)
2. I create a volume A and attach it to the VM, with deviceId = 4
3. THEN I create a volume B and attach it to the VM, with no deviceId specified

Will I get:

• /dev/xvda is the root volume?
• /dev/xvde is volume A?
• /dev/xvdb is volume B?

[miesiu]

Hey @olivierlemasle

Thank you for your response. I can confirm that the above scenario will work this way, remember deviceID = 3 is reserved for cd-rom. Let me know if I can help you somehow.

[olivierlemasle]

Thank you @miesiu. Then I'll try to use the deviceId to search the device with the mapping above. When it's ready, will you be ok to test if it works well in a CloudStack+Xenserver environment?

I see that you started experimenting yourself. Did you get any success? I'll by happy to review a pull request if you want to contribute it.

[miesiu]

@olivierlemasle, unfortunately without success, I have no experience in this field. I will run tests just let me know.

[joschi36]

Hi @miesiu @olivierlemasle we are just working on VMware support for the CSI and CS+VMware also doesn't provide /dev/disk/by-id.
We noticed deviceId matches with our SCSI device number. Also, we are looking at chaininfo but this isn't always filled, at least with CS 4.14.1.0. We now try matching on both. My friend will publish a PR in a few days.
If you (@miesiu) can verify deviceId matches with /dev/disk/by-id we maybe can also support Xen with the same PR.

[olivierlemasle]

Hi @joschi36,

I'm also experimenting with VMware support, and I did not yet find how to match the devices with the CloudStack volumes.

For example (with CloudStack 4.14 and VMware):

• I create a VM with only a ROOT volume (20 GB)
• I attach a 20 GB volume, with deviceId = 2
• Then I attach a 10 GB volume, with deviceId = 1

I specified the deviceId to test the case where the deviceIds are not increasing. As the CSI driver may attach multiple volumes to the same VM at "the same time", it is not possible to rely on this order...

I have no chaininfo for my 2 data volumes.

Then, when the volumes are recognized, lsblk gives:

NAME                   MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
sda                      8:0    0   20G  0 disk 
├─sda1                   8:1    0    1G  0 part /boot
└─sda2                   8:2    0   19G  0 part 
  └─fedora_fedora-root 253:0    0   15G  0 lvm  /
sdb                      8:16   0   20G  0 disk 
sdc                      8:32   0   10G  0 disk 
sr0                     11:0    1 1024M  0 rom 

It shows that the device names (sdb, sdc) follow the order of the volume attachments, not the device ids as known by CloudStack.

And unlike with KVM, I do not find anything in /dev/disk/by-id (or any other /dev/disk/by-*) to link a device to its CloudStack volume ID, or to its CloudStack deviceId.

ls -d /sys/block/sd*/device/scsi_device/* | awk -F '[/]' '{print $4,"- SCSI",$7}'
sda - SCSI 0:0:1:0
sdb - SCSI 2:0:0:0
sdc - SCSI 2:0:1:0

Also, the SCSI device numbers follow the order of the volume attachments, which do not match the device ids known by CloudStack.

Do you have any idea?

[olivierlemasle]

cc @schlapzz

[schlapzz]

@olivierlemasle
I encountered following behaviour with CloudStack and VMWare

• CloudStack increases the device ID for every volume (or it takes the lowest unused device ID)
• SCSI Device ID 3 is never assigned to a CloudStack volume (see here) This is because ID 3 is usually the CD-ROM device (of course doesn't make much sense in a cloud environmnet)
• On VMWare Hosts SCSI DeviceID 7 is assigned to the Virtual SCSI Adapter and therefore can not used by a volume (see here) CloudStack however assigns this ID to the Volumes, which leads the they can not be found

So for VMWare we're trying to match the deviceID by matching the path /dev/disk/by-path/pci-0000:00:10.0-scsi-0:0:<deviceID>:0

There is maybe also a "bug" with the Unpublish Functions. According to the CSI Specification we need to undo everything we do in the Controller/Node Publish functions. This includes IMO also the cleaning of the SCSI Bus (counterpart to scan and add SCSI devices) For example:

echo 1 > /sys/calss/scsi_device/0:10:0<deviceID>/delete

Otherwise the controller is complaining about an already in use volume when you try to format the mounted volume.

[olivierlemasle]

@schlapzz Thank you for your answer.

However, this works only because both CloudStack and VMware increment the deviceIds (as I noted above, this does not work when a deviceId is specified in the attachVolume request, which means that the CloudStack deviceId is not used).
I feel it could create issues if multiple volumes were attached to the VM "simultaneously" (if possible).
I'll make tests to verify if it is reliable; perhaps my concern is only theoretical...

Also, thank you for your comment on the Unpublish functions; you're right.

[schlapzz]

@olivierlemasle
I did a Pull Request to add support for VMWare. See #9