Kubernetes cluster downscale fails and remains in SCALE state with IllegalArgumentException: fromIndex(x) > toIndex(y) on CloudStack 4.22

[akoskuczi-bw]

problem

While downscaling a Kubernetes cluster, the async job fails with an IllegalArgumentException during execution of ScaleKubernetesClusterCmd.
After the failure, the Kubernetes cluster remains stuck in SCALE state and does not recover automatically.

This issue is reproducible when reducing the number of worker nodes.

cmk listKubernetesClusters id=cff9d134-82a9-4383-a953-db8098536d68 filter=size
{
"count": 1,
"kubernetescluster": [
{
"size": 3
}
]
}

cloud@acsm1:~$ cmk scaleKubernetesCluster id=cff9d134-82a9-4383-a953-db8098536d68 size=2
{
"account": "admin",
"accountid": "67aa9f23-d520-11f0-b69a-525400aa0210",
"cmd": "org.apache.cloudstack.api.command.user.kubernetes.cluster.ScaleKubernetesClusterCmd",
"completed": "2025-12-10T16:25:57+0000",
"created": "2025-12-10T16:25:57+0000",
"domainid": "e89c5713-d51f-11f0-b69a-525400aa0210",
"domainpath": "ROOT",
"jobid": "560b7207-4c03-4d86-b6f5-f4822ad6c314",
"jobprocstatus": 0,
"jobresult": {
"errorcode": 530,
"errortext": "fromIndex(4) > toIndex(3)"
},
"jobresultcode": 530,
"jobresulttype": "object",
"jobstatus": 2,
"userid": "67ab52c3-d520-11f0-b69a-525400aa0210"
}
🙈 Error: async API failed for job 560b7207-4c03-4d86-b6f5-f4822ad6c314

versions

Apache CloudStack 4.22
Management Server running on Ubuntu 24.04
Kubernetes Server Version: v1.33.1

The steps to reproduce the bug

...

What to do about it?

No response

[Pearl1594]

@akoskuczi-bw I wasn't able to reproduce the same on 4.22. Is this consistently reproducible? Can you check the database is all the VMs (control and worker nodes) info are present in the kubernetes_cluster_vm_map table - as in, when you deployed the cluster with 3 worker nodes this table should have 4 records for the corresponding cluster (assuming control node count is 1) .
The error you seem to be occurring may happen if some records are missing in the aforementioned table and that ideally shouldn't be the case.

[kiranchavala]

@akoskuczi-bw Cross checked on

Apache CloudStack 4.22
Management Server running on Ubuntu 24.04
Kubernetes Server Version: v1.33.1

Not able to reproduce the issue , the cluster downscaled successfully

mysql> select * from kubernetes_cluster_vm_map;
+----+------------+-------+--------------+-----------+---------------+----------------+-------------------------+
| id | cluster_id | vm_id | control_node | etcd_node | external_node | manual_upgrade | kubernetes_node_version |
+----+------------+-------+--------------+-----------+---------------+----------------+-------------------------+
|  4 |          2 |    24 |            1 |         0 |             0 |              0 | 1.33.1                  |
|  5 |          2 |    25 |            0 |         0 |             0 |              0 | 1.33.1                  |
|  6 |          2 |    26 |            0 |         0 |             0 |              0 | 1.33.1                  |
|  8 |          2 |    28 |            0 |         0 |             0 |              0 | 1.33.1                  |
+----+------------+-------+--------------+-----------+---------------+----------------+-------------------------+
4 rows in set (0.00 sec)

After downscaling 

mysql> select * from kubernetes_cluster_vm_map;
+----+------------+-------+--------------+-----------+---------------+----------------+-------------------------+
| id | cluster_id | vm_id | control_node | etcd_node | external_node | manual_upgrade | kubernetes_node_version |
+----+------------+-------+--------------+-----------+---------------+----------------+-------------------------+
|  4 |          2 |    24 |            1 |         0 |             0 |              0 | 1.33.1                  |
|  5 |          2 |    25 |            0 |         0 |             0 |              0 | 1.33.1                  |
|  6 |          2 |    26 |            0 |         0 |             0 |              0 | 1.33.1                  |
+----+------------+-------+--------------+-----------+---------------+----------------+-------------------------+
3 rows in set (0.00 sec)

Could you let us know if autoscaling was enabled for the cluster ?

[akoskuczi-bw]

This cluster does not have autoscaling enabled. I can’t access this environment right now. I’ll get back to you once I can check the MySQL.

[akoskuczi-bw]

Reproduced in a new cluster:
HA setup with 3 control plane nodes and 2 workers.
Scaled workers up to 3, then attempted to scale back down to 2.
The exception was raised during the scale-down operation.

{
"count": 1,
"kubernetescluster": [
{
"account": "admin",
"associatednetworkname": "VPC1-NET-K8S-1",
"autoscalingenabled": false,
"clustertype": "CloudManaged",
"controlnodes": 3,
"controlofferingid": "163b275b-9ff2-4dbd-b10c-c71403a15c52",
"controlofferingname": "k8s-small",
"cpunumber": "36",
"created": "2025-12-17T01:41:32+0000",
"csienabled": true,
"description": "k8s-1",
"domain": "ROOT",
"domainid": "e89c5713-d51f-11f0-b69a-525400aa0210",
"domainpath": "ROOT",
"endpoint": "https://192.168.205.19:6443/",
"etcdips": {},
"etcdnodes": 0,
"hasannotations": false,
"id": "60c485e3-4763-400e-9af1-40c8ded126d8",
"keypair": "Akoss-MacBook-Air",
"kubernetesversionid": "033cd216-a754-409b-b579-3aeafd841b25",
"kubernetesversionname": "v1.33.1-calico-x86_64",
"masternodes": 3,
"memory": "61440",
"name": "k8s-1",
"networkid": "3007e805-e186-487b-85f2-6af091a1352c",
"serviceofferingid": "163b275b-9ff2-4dbd-b10c-c71403a15c52",
"serviceofferingname": "k8s-small",
"size": 3,
"state": "Scaling",
"templateid": "e8b53ef3-d51f-11f0-b69a-525400aa0210",
"templatename": "SystemVM Template (KVM)",
"virtualmachines": [
{
"account": "admin",
"affinitygroup": [],
"arch": "x86_64",
"created": "2025-12-17T01:41:43+0000",
"deleteprotection": false,
"details": {
"SSH.KeyPairNames": "Akoss-MacBook-Air",
"SSH.PublicKey": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC8243fYoN0KVpO7qfhJAi+EXH/0k37N3Vspmfkcd3oY8Nv07pq8++2+gtHF0HF8wf/um5FLK82iYhnD2FaICdYpvslMLNZiMH75szFnvt+KwNINvMoNrxzTXKMNpoGRFiljC63uwODFHrv+xLZZo36NPopDs91FONteaTtUYoyYsKogRQ4BH+A1yNIJF6lT3+e9PiazHRlJ4XqXWXPnE7x4xnrlVsoHpajt1xrUpgpwKq3Qf/uADQjV2Sy/qJmOPwaHrCYjKgUl4fuYgqFuW0R3OLKu4+MGniLUkJ9nO3qgps6QukZ05u8w9D7ay1xML6sN2F2lklvQzFRO4Hrxu/TUCmZgDfU8dWEB1AhTFWkrJp0sxR5ghh4qcBiUmCi6QcsGbtk56PEiWk4ZcvG3IxqXWKz2GGbDn5+dNIZGY6FolT4pdfFUftzdKDDtwS/RVKYeD2PCCKRowNsExrTjFOhPlEQTD5Lu0ly/TkZKXTXYJQbh3OA7TNAAg+S/bRCq78=",
"controlNodeLoginUser": "cloud",
"rootdisksize": "20"
},
"displayname": "k8s-1-control-19b29f86601",
"displayvm": true,
"domain": "ROOT",
"domainid": "e89c5713-d51f-11f0-b69a-525400aa0210",
"guestosid": "94fd284d-27fc-4a50-990f-d2a10cc162ab",
"haenable": false,
"hostcontrolstate": "Enabled",
"hostid": "e1f21bff-9033-4823-aec2-329a14a06453",
"hostname": "acsh2.hu.nttlab.com",
"hypervisor": "KVM",
"id": "b7feed25-254a-49e2-905a-468691f5fa2a",
"instancename": "i-2-49-VM",
"isdynamicallyscalable": false,
"isetcdnode": false,
"isexternalnode": false,
"keypairs": "Akoss-MacBook-Air",
"kubernetesnodeversion": "1.33.1",
"lastupdated": "2025-12-17T01:42:02+0000",
"name": "k8s-1-control-19b29f86601",
"nic": [
{
"broadcasturi": "vxlan://37314",
"deviceid": "0",
"extradhcpoption": [],
"gateway": "10.0.2.1",
"id": "dd2e38fe-4ca6-4c60-bab9-5681579c3d05",
"ipaddress": "10.0.2.36",
"isdefault": true,
"isolationuri": "vxlan://37314",
"macaddress": "02:01:00:d3:00:16",
"netmask": "255.255.255.0",
"networkid": "3007e805-e186-487b-85f2-6af091a1352c",
"networkname": "VPC1-NET-K8S-1",
"secondaryip": [],
"traffictype": "Guest",
"type": "Isolated",
"vpcid": "ac204c2f-6a63-414e-82bb-2c45b87306cd",
"vpcname": "VPC1"
}
],
"osdisplayname": "Debian GNU/Linux 12 (64-bit)",
"ostypeid": "94fd284d-27fc-4a50-990f-d2a10cc162ab",
"pooltype": "NetworkFilesystem",
"receivedbytes": 0,
"securitygroup": [],
"sentbytes": 0,
"state": "Running",
"tags": [],
"userid": "67ab52c3-d520-11f0-b69a-525400aa0210",
"username": "admin",
"vmtype": "cksnode",
"zoneid": "3e69da92-3496-4e85-a996-526087ba81ee",
"zonename": "Zone-A"
},
{
"account": "admin",
"affinitygroup": [],
"arch": "x86_64",
"created": "2025-12-17T01:42:02+0000",
"deleteprotection": false,
"details": {
"SSH.KeyPairNames": "Akoss-MacBook-Air",
"SSH.PublicKey": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC8243fYoN0KVpO7qfhJAi+EXH/0k37N3Vspmfkcd3oY8Nv07pq8++2+gtHF0HF8wf/um5FLK82iYhnD2FaICdYpvslMLNZiMH75szFnvt+KwNINvMoNrxzTXKMNpoGRFiljC63uwODFHrv+xLZZo36NPopDs91FONteaTtUYoyYsKogRQ4BH+A1yNIJF6lT3+e9PiazHRlJ4XqXWXPnE7x4xnrlVsoHpajt1xrUpgpwKq3Qf/uADQjV2Sy/qJmOPwaHrCYjKgUl4fuYgqFuW0R3OLKu4+MGniLUkJ9nO3qgps6QukZ05u8w9D7ay1xML6sN2F2lklvQzFRO4Hrxu/TUCmZgDfU8dWEB1AhTFWkrJp0sxR5ghh4qcBiUmCi6QcsGbtk56PEiWk4ZcvG3IxqXWKz2GGbDn5+dNIZGY6FolT4pdfFUftzdKDDtwS/RVKYeD2PCCKRowNsExrTjFOhPlEQTD5Lu0ly/TkZKXTXYJQbh3OA7TNAAg+S/bRCq78=",
"controlNodeLoginUser": "cloud",
"rootdisksize": "20"
},
"displayname": "k8s-1-control-19b29f8af29",
"displayvm": true,
"domain": "ROOT",
"domainid": "e89c5713-d51f-11f0-b69a-525400aa0210",
"guestosid": "94fd284d-27fc-4a50-990f-d2a10cc162ab",
"haenable": false,
"hostcontrolstate": "Enabled",
"hostid": "772f4811-59dd-4530-9fe5-fb6aa19d92e8",
"hostname": "acsh1.hu.nttlab.com",
"hypervisor": "KVM",
"id": "3f417ea5-f57c-4f9e-a1b1-7ac020d3bbc3",
"instancename": "i-2-50-VM",
"isdynamicallyscalable": false,
"isetcdnode": false,
"isexternalnode": false,
"keypairs": "Akoss-MacBook-Air",
"kubernetesnodeversion": "1.33.1",
"lastupdated": "2025-12-17T01:42:19+0000",
"name": "k8s-1-control-19b29f8af29",
"nic": [
{
"broadcasturi": "vxlan://37314",
"deviceid": "0",
"extradhcpoption": [],
"gateway": "10.0.2.1",
"id": "d210d83f-8c0b-46ff-8c2e-0ce3fc2a2b0f",
"ipaddress": "10.0.2.254",
"isdefault": true,
"isolationuri": "vxlan://37314",
"macaddress": "02:01:00:d3:00:17",
"netmask": "255.255.255.0",
"networkid": "3007e805-e186-487b-85f2-6af091a1352c",
"networkname": "VPC1-NET-K8S-1",
"secondaryip": [],
"traffictype": "Guest",
"type": "Isolated",
"vpcid": "ac204c2f-6a63-414e-82bb-2c45b87306cd",
"vpcname": "VPC1"
}
],
"osdisplayname": "Debian GNU/Linux 12 (64-bit)",
"ostypeid": "94fd284d-27fc-4a50-990f-d2a10cc162ab",
"pooltype": "NetworkFilesystem",
"receivedbytes": 0,
"securitygroup": [],
"sentbytes": 0,
"state": "Running",
"tags": [],
"userid": "67ab52c3-d520-11f0-b69a-525400aa0210",
"username": "admin",
"vmtype": "cksnode",
"zoneid": "3e69da92-3496-4e85-a996-526087ba81ee",
"zonename": "Zone-A"
},
{
"account": "admin",
"affinitygroup": [],
"arch": "x86_64",
"created": "2025-12-17T01:42:19+0000",
"deleteprotection": false,
"details": {
"SSH.KeyPairNames": "Akoss-MacBook-Air",
"SSH.PublicKey": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC8243fYoN0KVpO7qfhJAi+EXH/0k37N3Vspmfkcd3oY8Nv07pq8++2+gtHF0HF8wf/um5FLK82iYhnD2FaICdYpvslMLNZiMH75szFnvt+KwNINvMoNrxzTXKMNpoGRFiljC63uwODFHrv+xLZZo36NPopDs91FONteaTtUYoyYsKogRQ4BH+A1yNIJF6lT3+e9PiazHRlJ4XqXWXPnE7x4xnrlVsoHpajt1xrUpgpwKq3Qf/uADQjV2Sy/qJmOPwaHrCYjKgUl4fuYgqFuW0R3OLKu4+MGniLUkJ9nO3qgps6QukZ05u8w9D7ay1xML6sN2F2lklvQzFRO4Hrxu/TUCmZgDfU8dWEB1AhTFWkrJp0sxR5ghh4qcBiUmCi6QcsGbtk56PEiWk4ZcvG3IxqXWKz2GGbDn5+dNIZGY6FolT4pdfFUftzdKDDtwS/RVKYeD2PCCKRowNsExrTjFOhPlEQTD5Lu0ly/TkZKXTXYJQbh3OA7TNAAg+S/bRCq78=",
"controlNodeLoginUser": "cloud",
"rootdisksize": "20"
},
"displayname": "k8s-1-control-19b29f8f27a",
"displayvm": true,
"domain": "ROOT",
"domainid": "e89c5713-d51f-11f0-b69a-525400aa0210",
"guestosid": "94fd284d-27fc-4a50-990f-d2a10cc162ab",
"haenable": false,
"hostcontrolstate": "Enabled",
"hostid": "e1f21bff-9033-4823-aec2-329a14a06453",
"hostname": "acsh2.hu.nttlab.com",
"hypervisor": "KVM",
"id": "683a6849-fe57-4299-8ec6-87ccc4f58a9c",
"instancename": "i-2-51-VM",
"isdynamicallyscalable": false,
"isetcdnode": false,
"isexternalnode": false,
"keypairs": "Akoss-MacBook-Air",
"kubernetesnodeversion": "1.33.1",
"lastupdated": "2025-12-17T01:42:38+0000",
"name": "k8s-1-control-19b29f8f27a",
"nic": [
{
"broadcasturi": "vxlan://37314",
"deviceid": "0",
"extradhcpoption": [],
"gateway": "10.0.2.1",
"id": "547e94b4-d7a4-45d6-8a00-37723f631d51",
"ipaddress": "10.0.2.108",
"isdefault": true,
"isolationuri": "vxlan://37314",
"macaddress": "02:01:00:d3:00:18",
"netmask": "255.255.255.0",
"networkid": "3007e805-e186-487b-85f2-6af091a1352c",
"networkname": "VPC1-NET-K8S-1",
"secondaryip": [],
"traffictype": "Guest",
"type": "Isolated",
"vpcid": "ac204c2f-6a63-414e-82bb-2c45b87306cd",
"vpcname": "VPC1"
}
],
"osdisplayname": "Debian GNU/Linux 12 (64-bit)",
"ostypeid": "94fd284d-27fc-4a50-990f-d2a10cc162ab",
"pooltype": "NetworkFilesystem",
"receivedbytes": 0,
"securitygroup": [],
"sentbytes": 0,
"state": "Running",
"tags": [],
"userid": "67ab52c3-d520-11f0-b69a-525400aa0210",
"username": "admin",
"vmtype": "cksnode",
"zoneid": "3e69da92-3496-4e85-a996-526087ba81ee",
"zonename": "Zone-A"
},
{
"account": "admin",
"affinitygroup": [],
"arch": "x86_64",
"created": "2025-12-17T01:42:39+0000",
"deleteprotection": false,
"details": {
"SSH.KeyPairNames": "Akoss-MacBook-Air",
"SSH.PublicKey": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC8243fYoN0KVpO7qfhJAi+EXH/0k37N3Vspmfkcd3oY8Nv07pq8++2+gtHF0HF8wf/um5FLK82iYhnD2FaICdYpvslMLNZiMH75szFnvt+KwNINvMoNrxzTXKMNpoGRFiljC63uwODFHrv+xLZZo36NPopDs91FONteaTtUYoyYsKogRQ4BH+A1yNIJF6lT3+e9PiazHRlJ4XqXWXPnE7x4xnrlVsoHpajt1xrUpgpwKq3Qf/uADQjV2Sy/qJmOPwaHrCYjKgUl4fuYgqFuW0R3OLKu4+MGniLUkJ9nO3qgps6QukZ05u8w9D7ay1xML6sN2F2lklvQzFRO4Hrxu/TUCmZgDfU8dWEB1AhTFWkrJp0sxR5ghh4qcBiUmCi6QcsGbtk56PEiWk4ZcvG3IxqXWKz2GGbDn5+dNIZGY6FolT4pdfFUftzdKDDtwS/RVKYeD2PCCKRowNsExrTjFOhPlEQTD5Lu0ly/TkZKXTXYJQbh3OA7TNAAg+S/bRCq78=",
"controlNodeLoginUser": "cloud",
"rootdisksize": "20"
},
"displayname": "k8s-1-node-19b29f93e7e",
"displayvm": true,
"domain": "ROOT",
"domainid": "e89c5713-d51f-11f0-b69a-525400aa0210",
"guestosid": "94fd284d-27fc-4a50-990f-d2a10cc162ab",
"haenable": false,
"hostcontrolstate": "Enabled",
"hostid": "772f4811-59dd-4530-9fe5-fb6aa19d92e8",
"hostname": "acsh1.hu.nttlab.com",
"hypervisor": "KVM",
"id": "3cadc08b-3bc3-4626-b4de-e3b4fd327866",
"instancename": "i-2-52-VM",
"isdynamicallyscalable": false,
"isetcdnode": false,
"isexternalnode": false,
"keypairs": "Akoss-MacBook-Air",
"kubernetesnodeversion": "1.33.1",
"lastupdated": "2025-12-17T01:42:58+0000",
"name": "k8s-1-node-19b29f93e7e",
"nic": [
{
"broadcasturi": "vxlan://37314",
"deviceid": "0",
"extradhcpoption": [],
"gateway": "10.0.2.1",
"id": "b70be17e-7bbc-490d-a69c-07527eb798ff",
"ipaddress": "10.0.2.186",
"isdefault": true,
"isolationuri": "vxlan://37314",
"macaddress": "02:01:00:d3:00:19",
"netmask": "255.255.255.0",
"networkid": "3007e805-e186-487b-85f2-6af091a1352c",
"networkname": "VPC1-NET-K8S-1",
"secondaryip": [],
"traffictype": "Guest",
"type": "Isolated",
"vpcid": "ac204c2f-6a63-414e-82bb-2c45b87306cd",
"vpcname": "VPC1"
}
],
"osdisplayname": "Debian GNU/Linux 12 (64-bit)",
"ostypeid": "94fd284d-27fc-4a50-990f-d2a10cc162ab",
"pooltype": "NetworkFilesystem",
"receivedbytes": 0,
"securitygroup": [],
"sentbytes": 0,
"state": "Running",
"tags": [],
"userid": "67ab52c3-d520-11f0-b69a-525400aa0210",
"username": "admin",
"vmtype": "cksnode",
"zoneid": "3e69da92-3496-4e85-a996-526087ba81ee",
"zonename": "Zone-A"
},
{
"account": "admin",
"affinitygroup": [],
"arch": "x86_64",
"created": "2025-12-17T01:42:58+0000",
"deleteprotection": false,
"details": {
"SSH.KeyPairNames": "Akoss-MacBook-Air",
"SSH.PublicKey": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC8243fYoN0KVpO7qfhJAi+EXH/0k37N3Vspmfkcd3oY8Nv07pq8++2+gtHF0HF8wf/um5FLK82iYhnD2FaICdYpvslMLNZiMH75szFnvt+KwNINvMoNrxzTXKMNpoGRFiljC63uwODFHrv+xLZZo36NPopDs91FONteaTtUYoyYsKogRQ4BH+A1yNIJF6lT3+e9PiazHRlJ4XqXWXPnE7x4xnrlVsoHpajt1xrUpgpwKq3Qf/uADQjV2Sy/qJmOPwaHrCYjKgUl4fuYgqFuW0R3OLKu4+MGniLUkJ9nO3qgps6QukZ05u8w9D7ay1xML6sN2F2lklvQzFRO4Hrxu/TUCmZgDfU8dWEB1AhTFWkrJp0sxR5ghh4qcBiUmCi6QcsGbtk56PEiWk4ZcvG3IxqXWKz2GGbDn5+dNIZGY6FolT4pdfFUftzdKDDtwS/RVKYeD2PCCKRowNsExrTjFOhPlEQTD5Lu0ly/TkZKXTXYJQbh3OA7TNAAg+S/bRCq78=",
"controlNodeLoginUser": "cloud",
"rootdisksize": "20"
},
"displayname": "k8s-1-node-19b29f98aa5",
"displayvm": true,
"domain": "ROOT",
"domainid": "e89c5713-d51f-11f0-b69a-525400aa0210",
"guestosid": "94fd284d-27fc-4a50-990f-d2a10cc162ab",
"haenable": false,
"hostcontrolstate": "Enabled",
"hostid": "909a292d-d1e0-4a0c-8d53-a0865ee8c6fa",
"hostname": "acsh4.hu.nttlab.com",
"hypervisor": "KVM",
"id": "c56312a3-c953-4198-ab08-8bae8df36248",
"instancename": "i-2-53-VM",
"isdynamicallyscalable": false,
"isetcdnode": false,
"isexternalnode": false,
"keypairs": "Akoss-MacBook-Air",
"kubernetesnodeversion": "1.33.1",
"lastupdated": "2025-12-17T01:43:15+0000",
"name": "k8s-1-node-19b29f98aa5",
"nic": [
{
"broadcasturi": "vxlan://37314",
"deviceid": "0",
"extradhcpoption": [],
"gateway": "10.0.2.1",
"id": "d23c35f8-aa37-4d9c-90dd-162dadf7b17d",
"ipaddress": "10.0.2.130",
"isdefault": true,
"isolationuri": "vxlan://37314",
"macaddress": "02:01:00:d3:00:1a",
"netmask": "255.255.255.0",
"networkid": "3007e805-e186-487b-85f2-6af091a1352c",
"networkname": "VPC1-NET-K8S-1",
"secondaryip": [],
"traffictype": "Guest",
"type": "Isolated",
"vpcid": "ac204c2f-6a63-414e-82bb-2c45b87306cd",
"vpcname": "VPC1"
}
],
"osdisplayname": "Debian GNU/Linux 12 (64-bit)",
"ostypeid": "94fd284d-27fc-4a50-990f-d2a10cc162ab",
"pooltype": "NetworkFilesystem",
"receivedbytes": 0,
"securitygroup": [],
"sentbytes": 0,
"state": "Running",
"tags": [],
"userid": "67ab52c3-d520-11f0-b69a-525400aa0210",
"username": "admin",
"vmtype": "cksnode",
"zoneid": "3e69da92-3496-4e85-a996-526087ba81ee",
"zonename": "Zone-A"
},
{
"account": "admin",
"affinitygroup": [],
"arch": "x86_64",
"created": "2025-12-17T01:52:47+0000",
"deleteprotection": false,
"details": {
"SSH.KeyPairNames": "Akoss-MacBook-Air",
"SSH.PublicKey": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC8243fYoN0KVpO7qfhJAi+EXH/0k37N3Vspmfkcd3oY8Nv07pq8++2+gtHF0HF8wf/um5FLK82iYhnD2FaICdYpvslMLNZiMH75szFnvt+KwNINvMoNrxzTXKMNpoGRFiljC63uwODFHrv+xLZZo36NPopDs91FONteaTtUYoyYsKogRQ4BH+A1yNIJF6lT3+e9PiazHRlJ4XqXWXPnE7x4xnrlVsoHpajt1xrUpgpwKq3Qf/uADQjV2Sy/qJmOPwaHrCYjKgUl4fuYgqFuW0R3OLKu4+MGniLUkJ9nO3qgps6QukZ05u8w9D7ay1xML6sN2F2lklvQzFRO4Hrxu/TUCmZgDfU8dWEB1AhTFWkrJp0sxR5ghh4qcBiUmCi6QcsGbtk56PEiWk4ZcvG3IxqXWKz2GGbDn5+dNIZGY6FolT4pdfFUftzdKDDtwS/RVKYeD2PCCKRowNsExrTjFOhPlEQTD5Lu0ly/TkZKXTXYJQbh3OA7TNAAg+S/bRCq78=",
"controlNodeLoginUser": "cloud",
"rootdisksize": "20"
},
"displayname": "k8s-1-node-19b2a0288a6",
"displayvm": true,
"domain": "ROOT",
"domainid": "e89c5713-d51f-11f0-b69a-525400aa0210",
"guestosid": "94fd284d-27fc-4a50-990f-d2a10cc162ab",
"haenable": false,
"hostcontrolstate": "Enabled",
"hostid": "909a292d-d1e0-4a0c-8d53-a0865ee8c6fa",
"hostname": "acsh4.hu.nttlab.com",
"hypervisor": "KVM",
"id": "7ea2afff-300f-47b2-a077-490c38d86bfa",
"instancename": "i-2-54-VM",
"isdynamicallyscalable": false,
"isetcdnode": false,
"isexternalnode": false,
"keypairs": "Akoss-MacBook-Air",
"kubernetesnodeversion": "1.33.1",
"lastupdated": "2025-12-17T01:53:08+0000",
"name": "k8s-1-node-19b2a0288a6",
"nic": [
{
"broadcasturi": "vxlan://37314",
"deviceid": "0",
"extradhcpoption": [],
"gateway": "10.0.2.1",
"id": "a5e6071c-3286-410c-b466-0e24da6e9d68",
"ipaddress": "10.0.2.110",
"isdefault": true,
"isolationuri": "vxlan://37314",
"macaddress": "02:01:00:d3:00:1b",
"netmask": "255.255.255.0",
"networkid": "3007e805-e186-487b-85f2-6af091a1352c",
"networkname": "VPC1-NET-K8S-1",
"secondaryip": [],
"traffictype": "Guest",
"type": "Isolated",
"vpcid": "ac204c2f-6a63-414e-82bb-2c45b87306cd",
"vpcname": "VPC1"
}
],
"osdisplayname": "Debian GNU/Linux 12 (64-bit)",
"ostypeid": "94fd284d-27fc-4a50-990f-d2a10cc162ab",
"pooltype": "NetworkFilesystem",
"receivedbytes": 0,
"securitygroup": [],
"sentbytes": 0,
"state": "Running",
"tags": [],
"userid": "67ab52c3-d520-11f0-b69a-525400aa0210",
"username": "admin",
"vmtype": "cksnode",
"zoneid": "3e69da92-3496-4e85-a996-526087ba81ee",
"zonename": "Zone-A"
}
],
"workerofferingid": "8ae8a869-13cb-4af1-afc7-03124dc573aa",
"workerofferingname": "k8s-large",
"zoneid": "3e69da92-3496-4e85-a996-526087ba81ee",
"zonename": "Zone-A"
}
]
}

select * from kubernetes_cluster_vm_map;
+----+------------+-------+--------------+-----------+---------------+----------------+-------------------------+
| id | cluster_id | vm_id | control_node | etcd_node | external_node | manual_upgrade | kubernetes_node_version |
+----+------------+-------+--------------+-----------+---------------+----------------+-------------------------+
| 19 | 4 | 49 | 1 | 0 | 0 | 0 | 1.33.1 |
| 20 | 4 | 50 | 1 | 0 | 0 | 0 | 1.33.1 |
| 21 | 4 | 51 | 1 | 0 | 0 | 0 | 1.33.1 |
| 22 | 4 | 52 | 0 | 0 | 0 | 0 | 1.33.1 |
| 23 | 4 | 53 | 0 | 0 | 0 | 0 | 1.33.1 |
| 24 | 4 | 54 | 0 | 0 | 0 | 0 | 1.33.1 |
+----+------------+-------+--------------+-----------+---------------+----------------+-------------------------+
6 rows in set (0.01 sec)

The error is: fromIndex(4) > toIndex(3)

Additional note: the default pod network subnet did not fit my requirements, so I modified Calico’s default-ipv4-ippool and restarted all pods. I’m not sure if this change is related to the scaling issue.

[kiranchavala]

@akoskuczi-bw thanks

i am to reporduce the issue now , issue occurs when the cluster is HA enabled and the cluster is struck in scaling state

cc @Pearl1594 @weizhouapache

(localcloud) 🐱 > scale kubernetescluster id=aebb9cd2-60e6-4e4a-b121-567db53805c4 size=2
{
  "account": "admin",
  "accountid": "69c635a2-d5e9-11f0-a8f5-bc241173f3f6",
  "cmd": "org.apache.cloudstack.api.command.user.kubernetes.cluster.ScaleKubernetesClusterCmd",
  "completed": "2025-12-17T09:14:11+0000",
  "created": "2025-12-17T09:14:11+0000",
  "domainid": "b58f4d57-d5e8-11f0-a8f5-bc241173f3f6",
  "domainpath": "ROOT",
  "jobid": "cfbe5f80-7483-4f14-9f26-3046e63f8a95",
  "jobprocstatus": 0,
  "jobresult": {
    "errorcode": 530,
    "errortext": "fromIndex(4) > toIndex(3)"
  },
  "jobresultcode": 530,
  "jobresulttype": "object",
  "jobstatus": 2,
  "userid": "69c67410-d5e9-11f0-a8f5-bc241173f3f6"
}
🙈 Error: async API failed for job cfbe5f80-7483-4f14-9f26-3046e63f8a95

2025-12-17 09:14:11,461 INFO  [c.c.k.c.a.KubernetesClusterScaleWorker] (API-Job-Executor-22:[ctx-2f12808a, job-613, ctx-c7e3dbb2]) (logid:cfbe5f80) Scaling Kubernetes cluster KubernetesCluster {"id":10,"name":"ha-cks","uuid":"aebb9cd2-60e6-4e4a-b121-567db53805c4"}
2025-12-17 09:14:11,473 ERROR [c.c.a.ApiAsyncJobDispatcher] (API-Job-Executor-22:[ctx-2f12808a, job-613]) (logid:cfbe5f80) Unexpected exception while executing org.apache.cloudstack.api.command.user.kubernetes.cluster.ScaleKubernetesClusterCmd java.lang.IllegalArgumentException: fromIndex(4) > toIndex(3)
        at java.base/java.util.AbstractList.subListRangeCheck(AbstractList.java:509)
        at java.base/java.util.ArrayList.subList(ArrayList.java:1108)
        at com.cloud.kubernetes.cluster.actionworkers.KubernetesClusterScaleWorker.scaleDownKubernetesClusterSize(KubernetesClusterScaleWorker.java:448)
        at com.cloud.kubernetes.cluster.actionworkers.KubernetesClusterScaleWorker.scaleKubernetesClusterSize(KubernetesClusterScaleWorker.java:507)
        at com.cloud.kubernetes.cluster.actionworkers.KubernetesClusterScaleWorker.scaleCluster(KubernetesClusterScaleWorker.java:584)
        at com.cloud.kubernetes.cluster.KubernetesClusterManagerImpl.scaleKubernetesCluster(KubernetesClusterManagerImpl.java:2142)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:569)
        at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
        at org.apache.cloudstack.network.contrail.management.EventUtils$EventInterceptor.invoke(EventUtils.java:109)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:175)
        at com.cloud.event.ActionEventInterceptor.invoke(ActionEventInterceptor.java:52)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:175)
        at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:97)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
        at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:215)
        at jdk.proxy3/jdk.proxy3.$Proxy537.scaleKubernetesCluster(Unknown Source)
        at org.apache.cloudstack.api.command.user.kubernetes.cluster.ScaleKubernetesClusterCmd.execute(ScaleKubernetesClusterCmd.java:174)
        at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:173)
        at com.cloud.api.ApiAsyncJobDispatcher.runJob(ApiAsyncJobDispatcher.java:110)
        at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:698)
        at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
        at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
        at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
        at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
        at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
        at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:646)
        at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
        at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
        at java.base/java.lang.Thread.run(Thread.java:840)

2025-12-17 09:14:11,473 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-22:[ctx-2f12808a, job-613]) (logid:cfbe5f80) Complete async job-613, jobStatus: FAILED, resultCode: 530, result: org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":[],"errorcode":"530","errortext":"fromIndex(4) > toIndex(3)"}