Create Instance from Backup Failed

[leo79901]

problem

While I was trying to create an instance from a backup, it failed.

I have tried both NFS and Linstor storage as target pools when restoring from an NFS backup, and encountered this error.

2025-12-10 09:17:21,196 ERROR [c.c.a.ApiAsyncJobDispatcher] (API-Job-Executor-53:[ctx-29561268, job-857]) (logid:0ad3bb95) Unexpected exception while executing org.apache.cloudstack.api.command.admin.vm.CreateVMFromBackupCmdByAdmin com.cloud.utils.exception.CloudRuntimeException: Failed to create Instance [i-2-65-VM] from backup [{"externalId":"i-2-64-VM\/2025.12.10.08.53.42","name":"nfs-0-nfs-0","uuid":"3acd3982-663f-4d2f-874c-364c5bb22a4a"}] due to: Unable to restore contents from the backup volume [0c3785e2-9473-4c5f-89d7-7c9e449e8bc7]..

I suspect that during the restoration process, the file name being looked for may not match the actual file name stored in the backup, which could be causing the failure to locate the backup file.

Here is a summary of some information. I will try to upload the relevant logs for this part.

<style> </style>

No.	Storage Type	Backup Stroage	Volume ID	Volume Name in backup	Backup info in database
1	Linstor	NFS	d9283f2c-26e9-4adf-9816-a54537b6e81a	root.drbd1039.qcow2	[{"uuid":"d9283f2c-26e9-4adf-9816-a54537b6e81a","type":"ROOT","size":107374182400,"path":"d9283f2c-26e9-4adf-9816-a54537b6e81a","deviceId":0,"diskOfferingId":"5081ea5d-87ec-4903-a30d-7fab9c378ce8"}]
2	NFS	NFS	0c3785e2-9473-4c5f-89d7-7c9e449e8bc7	root.0c3785e2-9473-4c5f-89d7-7c9e449e8bc7.qcow2	[{"uuid":"0c3785e2-9473-4c5f-89d7-7c9e449e8bc7","type":"ROOT","size":107374182400,"path":"0c3785e2-9473-4c5f-89d7-7c9e449e8bc7","deviceId":0,"diskOfferingId":"880ba876-defd-4bf3-b28b-1a973c9dff53"}]

versions

4.22.0

The steps to reproduce the bug

1. Provision virtual machines using LINSTOR and NFS as the primary storage types respectively.
2. Back up the VMs to the NFS-based backup repository.
3. Attempt to restore the backup to a new virtual machine → Restore fails.
4. Shut down the original virtual machine, then restore the same backup to it → Restore succeeds.

What to do about it?

log.txt

[leo79901]

I tried modifying the file name after backup, and when I performed the restore, it prompted that the file could not be found. It seems the issue is not related to the file name as I previously guessed.

[sureshanaparti]

@leo79901 can you share the agent log from kvm host while restoring from the backup (for NFS and Linstor storages)? I think, for Linstor the restore volume path might be incorrect, it shouldn't be /mnt..., correct?

{
  "org.apache.cloudstack.backup.RestoreBackupCommand": {
    "vmName": "i-2-65-VM",
    "backupPath": "i-2-64-VM/2025.12.10.08.53.42",
    "backupRepoType": "nfs",
    "backupRepoAddress": "    10.89.52.101:/nfs_with_vdo/acs-dev",
    "backupVolumesUUIDs": [
      "0c3785e2-9473-4c5f-89d7-7c9e449e8bc7"
    ],
    "restoreVolumePools": [
      {
        "uuid": "732a881b-916f-4601-bb16-c5da58831047",
        "name": "Linstor",
        "id": "1",
        "poolType": "Linstor",
        "host": "http://10.89.52.111:3370",
        "path": "cloudstack",
        "port": "3370",
        "url": "Linstor://http://10.89.52.111:3370/cloudstack/?ROLE=Primary&STOREUUID=732a881b-916f-4601-bb16-c5da58831047",
        "isManaged": "false"
      }
    ],
    "restoreVolumePaths": [
      "/mnt/732a881b-916f-4601-bb16-c5da58831047/d16d7121-1dc7-4588-9c3d-d141581c485a"
    ],
    "vmExists": "true",
    "vmState": "Restoring",
    "mountTimeout": "30",
    "wait": "0",
    "bypassHostMainten    ance": "false"
  }
}

[leo79901]

An incorrect file access path was found in the logs.

The log file:
agent.log

[abh1sar]

I think the issue is not with the linstor volume path, but the backup file path itself.

@leo79901 could you please perform a "Create instance from backup" command and a "Restore backup to the same vm" command back to back using the same backup and share the management server log.

[leo79901]

Done. Logs of 3 management servers was here:

management-server.node1.log
management-server.node2.log
management-server.node3.log

[leo79901]

The error message on the web is : Failed to create Instance [i-2-74-VM] from backup [{"externalId":"i-2-64-VM/2025.12.11.09.42.11","name":"nfs-0-2025-12-11T01:42:11+0000","uuid":"b712b318-75e3-4e3a-b309-e8353dded853"}] due to: Unable to restore contents from the backup volume [0c3785e2-9473-4c5f-89d7-7c9e449e8bc7]..

[abh1sar]

Thanks @leo79901
In the above logs, the source vm is on an NFS pool and the destination vm is created on the Linstor Pool.

Earlier you mentioned that you tried both NFS and Linstor as target pools, but it failed.

1. Can you please try to create the new instance from backup on the NFS pool and see if it fails. And also share the logs from that.
2. Does backup creation and restore to the same VM work if the VM is on Linstor?

[leo79901]

I'm not entirely sure about the content you need me to work on, so I conducted a full test. Here are the test steps, results, and logs:

Backed up the Linstor instance, which uses Linstor storage. The backup was successful, with the backup name: linstor-2025-12-12T07:21:22+0000.

Created a new instance using this backup without specifying a storage solution. The creation failed, with the following web information:
Failed to create Instance [i-2-93-VM] from backup [{"externalId":"i-2-91-VM/2025.12.12.15.21.22","name":"linstor-2025-12-12T07:21:22+0000","uuid":"c8da7d8a-7b32-470d-bf30-b9ac695f9549"}] due to: Backup file for the volume [d22e4725-c83b-4fe8-a462-99dc5a40305b] does not exist..

Created a new instance using this backup, specifying NFS storage. The creation failed, with the following web information:
Failed to create Instance [i-2-94-VM] from backup [{"externalId":"i-2-91-VM/2025.12.12.15.21.22","name":"linstor-2025-12-12T07:21:22+0000","uuid":"c8da7d8a-7b32-470d-bf30-b9ac695f9549"}] due to: Backup file for the volume [d22e4725-c83b-4fe8-a462-99dc5a40305b] does not exist..

Created a new instance using this backup, specifying Linstor storage. The creation failed, with the following web information:
Failed to create Instance [i-2-95-VM] from backup [{"externalId":"i-2-91-VM/2025.12.12.15.21.22","name":"linstor-2025-12-12T07:21:22+0000","uuid":"c8da7d8a-7b32-470d-bf30-b9ac695f9549"}] due to: Backup file for the volume [d22e4725-c83b-4fe8-a462-99dc5a40305b] does not exist..

Shut down the virtual machine and performed a self-restore of the virtual machine. The restore failed, with the following log:
(linstor-2025-12-12T07:21:22+0000) Error restoring VM from backup [{"externalId":"i-2-91-VM/2025.12.12.15.21.22","name":"linstor-2025-12-12T07:21:22+0000","uuid":"c8da7d8a-7b32-470d-bf30-b9ac695f9549","vmId":91}].

Backed up the NFS instance, which uses NFS storage. The backup was successful, with the backup name: nfs-0.

Created a new instance using this backup without specifying a storage solution. The creation failed, with the following web information:
Failed to create Instance [i-2-96-VM] from backup [{"externalId":"i-2-92-VM/2025.12.12.15.29.17","name":"nfs-0","uuid":"b20c9561-438a-442e-90e8-62161f04de82"}] due to: Unable to restore contents from the backup volume [26acbb35-5b85-4610-b2e9-6f6f8e84d205]..

Created a new instance using this backup, specifying NFS storage. The creation was successful.

Shut down the virtual machine and performed a self-restore of the virtual machine. The restore was successful.

management-server-node1.log
management-server-node2.log
management-server-node3.log

[abh1sar]

@leo79901 thanks for the detailed info.

There are 3 notable cases:

1. Source VM on Linstor and Destination VM on NFS -> Failed with Backup File doesn't exist.
2. Source VM on NFS and Destination VM on Linstor -> Backup file exists but copying the backup file to the destination volume path failed.
3. Source VM on NFS and Destination VM on NFS -> This was successful

NAS Bnr plugin currently only supports VMs running on NFS, Ceph, Shared mount point or the local storage. The plugin is treating the Linstor pool like an NFS pool and is creating the backup and restore path incorrectly for it.

Could you execute these two commands please?
These will tell us how the volume path in Linstor differs from what NAS BnR is using and what we need to do so that the NAS BnR plugin supports Linstor as well.

1. On the kvm host: virsh -c qemu:///system domblklist i-2-91-VM --details
2. On the NAS backup repository 10.89.52.101 : ls /nfs_with_vdo/acs-dev/i-2-91-VM/2025.12.12.15.21.22

[leo79901]

The result:

[root@yf-acs-hci-dev-3 ~]# virsh -c qemu:///system domblklist i-2-91-VM --details
 Type    Device   Target   Source
------------------------------------------
 block   disk     vda      /dev/drbd1030
 file    cdrom    hdc     

[root@yf-acs-nfs ~]# ls /nfs_with_vdo/acs-dev/i-2-91-VM/2025.12.12.15.21.22
domain-config.xml  domblklist.xml  domiflist.xml  dominfo.xml  root.drbd1030.qcow2

[abh1sar]

Linstor names volumes based on the mirror it is currently using. So for example /dev/drbd1030.
The NAS plugin backs up the volume to the path <backup_dir>/root.drbd1030.qcow2.

Then during restore, it doesn't know what was the backup file path (it is not constructed using the volume uuid). Also, the actual volume could now be pointing to a different mirror e.g, /dev/drbd1029.

@rp- Is my understanding correct? Could you share some ideas about how to support Backups of Linstor Instances using the NAS BnR plugin?
We basically want two things

1. Create the backup path and volume path (to restore) in the code
2. Rsync should work from the backup path to the volume path.

[rp-]

Linstor names volumes based on the mirror it is currently using. So for example /dev/drbd1030. The NAS plugin backs up the volume to the path <backup_dir>/root.drbd1030.qcow2.

Then during restore, it doesn't know what was the backup file path (it is not constructed using the volume uuid). Also, the actual volume could now be pointing to a different mirror e.g, /dev/drbd1029.

@rp- Is my understanding correct? Could you share some ideas about how to support Backups of Linstor Instances using the NAS BnR plugin? We basically want two things

1. Create the backup path and volume path (to restore) in the code
2. Rsync should work from the backup path to the volume path.

/dev/drbd1030 is what we write as device path in the libvirt xml, that is the path the DRBD device is exposing for usage (there would be also mapped paths by resource name in /dev/drbd/by-res/, those minor numbers are stable as long the linstor resources weren't completely deleted and recreated.

I'm not sure how the restore works with the NAS backup plugin, can you only restore/revert the same VM? or is a completely new VM instance created and the backup applied? (but than you would need to restore the same paths as the backup had?)