Unable to Autoscale K8s in CS 4.21.0.0

[amitmvyas]

Hello,

I have deployed K8s cluster in my CS 4.21.0.0 environment and I am trying to Autoscale the same to add a worker node and I am seeing below message in management server logs.

2025-11-05 16:07:17,543 INFO  [c.c.k.c.a.KubernetesClusterScaleWorker] (API-Job-Executor-34:[ctx-228e86cd, job-1437, ctx-43619b99]) (logid:56268dc3) Scaling Kubernetes cluster KubernetesCluster {"id":71,"name":"CL8S","uuid":"da626491-7457-4106-a165-de0d41d2417a"}
2025-11-05 16:07:17,553 WARN  [c.c.k.c.a.KubernetesClusterScaleWorker] (API-Job-Executor-34:[ctx-228e86cd, job-1437, ctx-43619b99]) (logid:56268dc3) Scaling Kubernetes cluster : CL8S failed, scaling Kubernetes cluster with running VMs on hypervisor KVM is not supported!
2025-11-05 16:07:17,553 INFO  [c.c.k.c.a.KubernetesClusterScaleWorker] (API-Job-Executor-34:[ctx-228e86cd, job-1437, ctx-43619b99]) (logid:56268dc3) Revoking launch permission for systemVM template
2025-11-05 16:07:17,648 INFO  [o.a.c.f.j.i.AsyncJobMonitor] (API-Job-Executor-34:[ctx-228e86cd, job-1437]) (logid:56268dc3) Remove job-1437 from job monitoring

I am having a hypevisor as KVM which is running on Ubuntu 24.04.3.

Can anyone help figure out why I am facing this problem and whether KVM is really not supported for CKS Autoscale Feature ? if not, then which hypevisor is supported for Autoscale ?

Regards,
Amit Vyas

[Jayd603]

I can't imagine why scaling k8s (by adding nodes right?) would not work on KVM. That sounds ridiculous. Now, resource scaling of existing nodes, maybe. I am actually bummed because I upgraded 4.19 to 4.21 not realizing it was a new type of release (non-LTS) , and it had broken kubernetes management but hey, so far everything else works great so it just needs some k8s fixes.

[kiranchavala]

@amitmvyas

Did you enable autoscaling on a kubernetes cluster which is in running state

Is the autoscaler pod running in the kubernetes cluster , can you provide the logs from the autoscaler pod

k get pods -A
NAMESPACE              NAME                                               READY   STATUS    RESTARTS   AGE
kube-system            calico-kube-controllers-7bfdc5b57c-nnzm4           1/1     Running   0          2m36s
kube-system            calico-node-7qxkb                                  1/1     Running   0          2m32s
kube-system            calico-node-lh5tk                                  1/1     Running   0          2m36s
kube-system            cloud-controller-manager-857cd68cf5-p5tvc          1/1     Running   0          110s
kube-system            cluster-autoscaler-6d469c6c6-x6f9m                 1/1     Running   0          40s

Example

k logs -f cluster-autoscaler-6d469c6c6-x6f9m -n kube-system

[amitmvyas]

kubectl --insecure-skip-tls-verify=true --kubeconfig kube.conf get pods -A                                               
NAMESPACE              NAME                                               READY   STATUS    RESTARTS      AGE
kube-system            calico-kube-controllers-7bfdc5b57c-kj99k           1/1     Running   1 (19h ago)   23h
kube-system            calico-node-d8dsj                                  1/1     Running   1 (19h ago)   23h
kube-system            calico-node-hqlxb                                  1/1     Running   1 (19h ago)   23h
kube-system            calico-node-lpnrl                                  1/1     Running   1 (19h ago)   23h
kube-system            cloud-controller-manager-857cd68cf5-8xbfr          1/1     Running   2 (19h ago)   23h
kube-system            cluster-autoscaler-744b9d65b-zzw2p                 1/1     Running   2 (19h ago)   19h
kube-system            coredns-674b8bbfcf-9wgdv                           1/1     Running   1 (19h ago)   23h
kube-system            coredns-674b8bbfcf-s5v82                           1/1     Running   1 (19h ago)   23h
kube-system            etcd-cl8s-control-19a5261efb6                      1/1     Running   1 (19h ago)   23h
kube-system            kube-apiserver-cl8s-control-19a5261efb6            1/1     Running   1 (19h ago)   23h
kube-system            kube-controller-manager-cl8s-control-19a5261efb6   1/1     Running   1 (19h ago)   23h
kube-system            kube-proxy-g98w9                                   1/1     Running   1 (19h ago)   23h
kube-system            kube-proxy-kg5t4                                   1/1     Running   1 (19h ago)   23h
kube-system            kube-proxy-prtkk                                   1/1     Running   1 (19h ago)   23h
kube-system            kube-scheduler-cl8s-control-19a5261efb6            1/1     Running   1 (19h ago)   23h
kubernetes-dashboard   dashboard-metrics-scraper-5bd45c9dd6-fm7wn         1/1     Running   1 (19h ago)   23h
kubernetes-dashboard   kubernetes-dashboard-687457b9bd-pcdt7              1/1     Running   1 (19h ago)   23h
scale-test             big-request-697b5d9654-hscsm                       1/1     Running   1 (19h ago)   20h
scale-test             big-request-697b5d9654-p224c                       0/1     Pending   0             20h
scale-test             big-request-697b5d9654-tw7fq                       1/1     Running   1 (19h ago)   20h

Tried with CMK as well

root@innmi1vcsmgt-d001:~# cmk scale kubernetescluster id=da626491-7457-4106-a165-de0d41d2417a size=4
{
  "account": "admin",
  "accountid": "9b2313ee-b4ae-11f0-914b-bc2411c796cd",
  "cmd": "org.apache.cloudstack.api.command.user.kubernetes.cluster.ScaleKubernetesClusterCmd",
  "completed": "2025-11-06T11:38:18+0530",
  "created": "2025-11-06T11:38:18+0530",
  "domainid": "abbdc016-b4ad-11f0-914b-bc2411c796cd",
  "domainpath": "ROOT",
  "jobid": "62e801e2-cc16-4014-9b50-d78b67e875e7",
  "jobprocstatus": 0,
  "jobresult": {
    "errorcode": 530,
    "errortext": "Scaling Kubernetes cluster : CL8S failed, scaling Kubernetes cluster with running VMs on hypervisor KVM is not supported!"
  },
  "jobresultcode": 530,
  "jobresulttype": "object",
  "jobstatus": 2,
  "userid": "9b23987d-b4ae-11f0-914b-bc2411c796cd"
}
🙈 Error: async API failed for job 62e801e2-cc16-4014-9b50-d78b67e875e7

This is the logs for cluster-autoscaler.

I1106 02:53:19.477668       1 client.go:169] NewAPIRequest API request URL:http://142.79.253.232:8080/client/api?apiKey=***&command=scaleKubernetesCluster&id=da626491-7457-4106-a165-de0d41d2417a&response=json&size=3&signature=***
I1106 02:53:19.477890       1 event_sink_logging_wrapper.go:48] Event(v1.ObjectReference{Kind:"ConfigMap", Namespace:"kube-system", Name:"cluster-autoscaler-status", UID:"ad1efd21-9a52-4226-9c1d-15396820bba5", APIVersion:"v1", ResourceVersion:"205055", FieldPath:""}): type: 'Normal' reason: 'ScaledUpGroup' Scale-up: setting group da626491-7457-4106-a165-de0d41d2417a size to 3 instead of 2 (max: 4)
I1106 02:53:19.556058       1 client.go:175] NewAPIRequest response status code:200
I1106 02:53:19.863209       1 reflector.go:255] Listing and watching *v1beta1.PodDisruptionBudget from /home/djumani/lab/autoscaler/cluster-autoscaler/utils/kubernetes/listers.go:309
W1106 02:53:19.865022       1 reflector.go:324] /home/djumani/lab/autoscaler/cluster-autoscaler/utils/kubernetes/listers.go:309: failed to list *v1beta1.PodDisruptionBudget: the server could not find the requested resource
E1106 02:53:19.865056       1 reflector.go:138] /home/djumani/lab/autoscaler/cluster-autoscaler/utils/kubernetes/listers.go:309: Failed to watch *v1beta1.PodDisruptionBudget: failed to list *v1beta1.PodDisruptionBudget: the server could not find the requested resource
I1106 02:53:29.557052       1 client.go:169] NewAPIRequest API request URL:http://142.79.253.232:8080/client/api?apiKey=***&command=queryAsyncJobResult&jobid=acee7bf9-c308-4929-87e5-42f74652d5eb&response=json&signature=***
I1106 02:53:29.597851       1 client.go:175] NewAPIRequest response status code:200
E1106 02:53:29.598077       1 client.go:118] API failed for job acee7bf9-c308-4929-87e5-42f74652d5eb : map[account:admin accountid:9b2313ee-b4ae-11f0-914b-bc2411c796cd cmd:org.apache.cloudstack.api.command.user.kubernetes.cluster.ScaleKubernetesClusterCmd completed:2025-11-06T08:23:19+0530 created:2025-11-06T08:23:19+0530 domainid:abbdc016-b4ad-11f0-914b-bc2411c796cd domainpath:ROOT jobid:acee7bf9-c308-4929-87e5-42f74652d5eb jobprocstatus:0 jobresult:map[errorcode:530 errortext:Scaling Kubernetes cluster : CL8S failed, scaling Kubernetes cluster with running VMs on hypervisor KVM is not supported!] jobresultcode:530 jobresulttype:object jobstatus:2 userid:80acdbf0-fe5c-4ed1-a5ea-4d8e9fe7307a]
W1106 02:53:29.598146       1 clusterstate.go:277] Disabling scale-up for node group da626491-7457-4106-a165-de0d41d2417a until 2025-11-06 03:23:19.477324222 +0000 UTC m=+64348.574537733; errorClass=Other; errorCode=cloudProviderError
I1106 02:53:29.598178       1 metrics.go:380] Function scaleUp took 10.120850343s to complete
I1106 02:53:29.598196       1 eventing_scale_up_processor.go:47] Skipping event processing for unschedulable pods since there is a ScaleUp attempt this loop
E1106 02:53:29.598212       1 static_autoscaler.go:459] Failed to scale up: failed to increase node group size: Unable to scale cluster : API failed for job acee7bf9-c308-4929-87e5-42f74652d5eb : map[account:admin accountid:9b2313ee-b4ae-11f0-914b-bc2411c796cd cmd:org.apache.cloudstack.api.command.user.kubernetes.cluster.ScaleKubernetesClusterCmd completed:2025-11-06T08:23:19+0530 created:2025-11-06T08:23:19+0530 domainid:abbdc016-b4ad-11f0-914b-bc2411c796cd domainpath:ROOT jobid:acee7bf9-c308-4929-87e5-42f74652d5eb jobprocstatus:0 jobresult:map[errorcode:530 errortext:Scaling Kubernetes cluster : CL8S failed, scaling Kubernetes cluster with running VMs on hypervisor KVM is not supported!] jobresultcode:530 jobresulttype:object jobstatus:2 userid:80acdbf0-fe5c-4ed1-a5ea-4d8e9fe7307a]
I1106 02:53:29.598232       1 event_sink_logging_wrapper.go:48] Event(v1.ObjectReference{Kind:"ConfigMap", Namespace:"kube-system", Name:"cluster-autoscaler-status", UID:"ad1efd21-9a52-4226-9c1d-15396820bba5", APIVersion:"v1", ResourceVersion:"205055", FieldPath:""}): type: 'Warning' reason: 'FailedToScaleUpGroup' Scale-up failed for group da626491-7457-4106-a165-de0d41d2417a: Unable to scale cluster : API failed for job acee7bf9-c308-4929-87e5-42f74652d5eb : map[account:admin accountid:9b2313ee-b4ae-11f0-914b-bc2411c796cd cmd:org.apache.cloudstack.api.command.user.kubernetes.cluster.ScaleKubernetesClusterCmd completed:2025-11-06T08:23:19+0530 created:2025-11-06T08:23:19+0530 domainid:abbdc016-b4ad-11f0-914b-bc2411c796cd domainpath:ROOT jobid:acee7bf9-c308-4929-87e5-42f74652d5eb jobprocstatus:0 jobresult:map[errorcode:530 errortext:Scaling Kubernetes cluster : CL8S failed, scaling Kubernetes cluster with running VMs on hypervisor KVM is not supported!] jobresultcode:530 jobresulttype:object jobstatus:2 userid:80acdbf0-fe5c-4ed1-a5ea-4d8e9fe7307a]
I1106 02:53:29.613991       1 metrics.go:380] Function main took 10.194459637s to complete
I1106 02:53:30.318148       1 reflector.go:536] k8s.io/client-go/informers/factory.go:134: Watch close - *v1.StatefulSet total 8 items received
I1106 02:53:36.016083       1 reflector.go:255] Listing and watching *v1beta1.CSIStorageCapacity from k8s.io/client-go/informers/factory.go:134
W1106 02:53:36.018008       1 reflector.go:324] k8s.io/client-go/informers/factory.go:134: failed to list *v1beta1.CSIStorageCapacity: the server could not find the requested resource
E1106 02:53:36.018094       1 reflector.go:138] k8s.io/client-go/informers/factory.go:134: Failed to watch *v1beta1.CSIStorageCapacity: failed to list *v1beta1.CSIStorageCapacity: the server could not find the requested resource
I1106 02:53:36.329876       1 reflector.go:536] k8s.io/client-go/informers/factory.go:134: Watch close - *v1.PodDisruptionBudget total 11 items received
I1106 02:53:39.614541       1 static_autoscaler.go:235] Starting main loop
I1106 02:53:39.614764       1 client.go:169] NewAPIRequest API request URL:http://142.79.253.232:8080/client/api?apiKey=***&command=listKubernetesClusters&id=da626491-7457-4106-a165-de0d41d2417a&response=json&signature=***
I1106 02:53:39.679225       1 client.go:175] NewAPIRequest response status code:200

I have attached entire logs as well.

cluster-autoscaler-744b9d65b-zzw2p.log

[weizhouapache]

the whole block was introduced in the original PR #3680

cloudstack/plugins/integrations/kubernetes-service/src/main/java/com/cloud/kubernetes/cluster/actionworkers/KubernetesClusterScaleWorker.java

Lines 316 to 324 in dbda673

	for (KubernetesClusterVmMapVO vmMapVO : vmList) {
	VMInstanceVO vmInstance = vmInstanceDao.findById(vmMapVO.getVmId());
	if (vmInstance != null && vmInstance.getState().equals(VirtualMachine.State.Running) &&
	vmInstance.getHypervisorType() != Hypervisor.HypervisorType.XenServer &&
	vmInstance.getHypervisorType() != Hypervisor.HypervisorType.VMware &&
	vmInstance.getHypervisorType() != Hypervisor.HypervisorType.Simulator) {
	logTransitStateToFailedIfNeededAndThrow(Level.WARN, String.format("Scaling Kubernetes cluster : %s failed, scaling Kubernetes cluster with running VMs on hypervisor %s is not supported!", kubernetesCluster.getName(), vmInstance.getHypervisorType()));
	}
	}

cc @shwstppr
do you remember why KVM is not supported ?

[shwstppr]

@weizhouapache that check was for scaling the cluster with a new offering. It shouldn't affect scaling for node count unless something changed recently. Scaling running VM may not be working on KVM at the time of #3680

[weizhouapache]

@weizhouapache that check was for scaling the cluster with a new offering. It shouldn't affect scaling for node count unless something changed recently. Scaling running VM may not be working on KVM at the time of #3680

thanks @shwstppr , makes sense.
so the issue could be related to #9102. cc @nvazquez @Pearl1594

[chandrakantrai]

I have implemented a fix in KubernetesClusterScaleWorker.java (lines 358-389) to address issues with cluster scaling. Updated code is as follows.

private void scaleKubernetesClusterOffering(KubernetesClusterNodeType nodeType, ServiceOffering serviceOffering,
boolean updateNodeOffering, boolean updateClusterOffering)
throws CloudRuntimeException {
// Do not validate for clusters in Created state (no VMs exist yet to upgrade).
// Validation will be done later only if we are actually upgrading existing VMs.
List<KubernetesCluster.State> scalingStates =
List.of(KubernetesCluster.State.Scaling, KubernetesCluster.State.ScalingStoppedCluster);
if (!scalingStates.contains(kubernetesCluster.getState())) {
stateTransitTo(kubernetesCluster.getId(), KubernetesCluster.Event.ScaleUpRequested);
}
if (KubernetesCluster.State.Created.equals(originalState)) {
kubernetesCluster = updateKubernetesClusterEntryForNodeType(null, nodeType, serviceOffering,
updateNodeOffering, updateClusterOffering);
return;
}
final long size = getNodeCountForType(nodeType, kubernetesCluster);
List vmList =
kubernetesClusterVmMapDao.listByClusterIdAndVmType(kubernetesCluster.getId(), nodeType);
final long tobeScaledVMCount = Math.min(vmList.size(), size);
    // Only validate hypervisor compatibility when upgrading existing VMs.
    // Horizontal scaling (adding new nodes) on KVM must not be blocked here.
    if (tobeScaledVMCount > 0) {
        validateKubernetesClusterScaleOfferingParameters();
    }

    for (long i = 0; i < tobeScaledVMCount; i++) {
        KubernetesClusterVmMapVO vmMapVO = vmList.get((int) i);
        UserVmVO userVM = userVmDao.findById(vmMapVO.getVmId());
        boolean result = false;
        try {
            result = userVmManager.upgradeVirtualMachine(userVM.getId(), serviceOffering.getId(),
                    new HashMap<String, String>());
        } catch (RuntimeException | ResourceUnavailableException | ManagementServerException
                 | VirtualMachineMigrationException e) {
            logTransitStateAndThrow(Level.ERROR,
                    String.format("Scaling Kubernetes cluster : %s failed, unable to scale cluster VM : %s due to %s",
                            kubernetesCluster.getName(), userVM.getDisplayName(), e.getMessage()),
                    kubernetesCluster.getId(), KubernetesCluster.Event.OperationFailed, e);
        }
        if (!result) {
            logTransitStateAndThrow(Level.WARN,
                    String.format("Scaling Kubernetes cluster : %s failed, unable to scale cluster VM : %s",
                            kubernetesCluster.getName(), userVM.getDisplayName()),
                    kubernetesCluster.getId(), KubernetesCluster.Event.OperationFailed);
        }
        if (System.currentTimeMillis() > scaleTimeoutTime) {
            logTransitStateAndThrow(Level.WARN,
                    String.format("Scaling Kubernetes cluster : %s failed, scaling action timed out",
                            kubernetesCluster.getName()),
                    kubernetesCluster.getId(), KubernetesCluster.Event.OperationFailed);
        }
    }
    kubernetesCluster = updateKubernetesClusterEntryForNodeType(null, nodeType, serviceOffering,
            updateNodeOffering, updateClusterOffering);
}

Status:

Confirmed: Scale-up and scale-down are working correctly on KVM hypervisors.

Pending: Testing on other hypervisors is still required.

Testing Instructions
If you would like to test this fix, I have attached a patched cloudstack-4.22.0.0.jar file. You can apply it to your CloudStack Management Server using the following steps:

1. Stop the CloudStack management service.
2. Backup your existing JAR file: cp /usr/share/cloudstack-management/lib/cloudstack-4.22.0.0.jar /usr/share/cloudstack-management/lib/cloudstack-4.22.0.0.jar.bak
3. Replace the existing JAR file with the provided patched version.
4. Start the CloudStack management service.

Link for Patched jar file

https://github.com/chandrakantrai/cloudstack4.22.0-Patch/blob/main/cloudstack-4.22.0.0.jar

@weizhouapache, thank you for pointing toward the relevant code section.

[sureshanaparti]

I have implemented a fix in KubernetesClusterScaleWorker.java (lines 358-389) to address issues with cluster scaling. Updated code is as follows.

Hi @chandrakantrai can you raise PR (against 4.22) with these changes? cc @weizhouapache