run kvms in k3s

Author: Magicloud

_posts/2025-11-16-run-kvms-in-k3s.markdown

@@ -0,0 +1,91 @@
+---
+layout: post
+title:  "Run KVMs in K3S"
+date:   2025-11-16 22:06:53 +0800
+categories: KVM K3S
+---
+I have a few KVMs running in my dev host, Home-Assistant, OpenWRT, etc. And I manage them via Virt-Manager. One thing bothers me is that, the whole thing is so not IaC. When I need a new VM, I have to manually edit a few things in GUI. Editing those "full" XML is not an option.
+
+From what container is, it is possible to run Qemu VMs in K3S, with minor performance loss. But a few questions still remain, for example, network, and proper tools configuring and managing VMs.
+
+Then I met [Kubevirt](https://kubevirt.io). The whole thing is quite big, involving a few operators, one managing another. But to me, two items matter. One, Kubevirt, the major tool for the task. Two, containerized data importer (CDI) to bring existing VM data into Kubevirt. And [Multus-CNI](https://github.com/k8snetworkplumbingwg/multus-cni) for my network requirement.
+
+## Install and Demo
+
+Installation was done simply applying [manifest](https://github.com/kubevirt/kubevirt/releases/download/v1.6.3/kubevirt-operator.yaml), which installs operator that brings up a few other pods, in `kubevirt` namespace.
+
+After everything is ready, apply [demo](https://kubevirt.io/labs/manifests/vm.yaml). It brings up a **virt-launcher** prefixed pod, which would setup the network / storage and eventually run QEMU.
+
+The status of the pod and VM can be checked via `kubectl describe pod` and `kubectl describe vm`, `kubectl describe vmi`.
+
+In the demo, storage part is quite clear and self-explained, followed general idea as how it is used in `Pod`.
+
+The network part, is in the same way. `spec.template.spec.networks` specifies what devices in pod are to be used in VM. And `spec.template.spec.domain.devices.interfaces` states how to use those devices.
+
+In `networks`, each item can be specified as `pod` or `multus`, which means use the default NIC appears in pod, or Multus NIC.
+
+In `interfaces`, each item can be specified as `masquerade` or `bridge`, which means using the corresponding item in `networks` as NAT gateway, or bridge device. In bridge mode, the interface in Pod would be DOWN, which means the Pod does not have network (even to cluster). Masquerde is cluster network only.
+
+After checking the status K3S-wise, time to install Kubevirt tool by `kubectl krew install virt`. Then, for example, `kubectl virt console $VMI_NAME` to checkout the console. Only console output in **pure** text is supported. The `VGA` mode used by modern general Linux distros does not work. In that case, VNC could be the solution.
+
+Since my dev host is not local, I do not need the tool to start VNC client for me. I use `kubectl virt vnc --address=0.0.0.0 --proxy-only $VMI_NAME` to start a VNC server. It shows the random port in the first line of output. Then I start VNC client to connect to it.
+
+## VM storage importing
+
+The installation includes two parts, [operator](https://github.com/kubevirt/containerized-data-importer/releases/download/v1.63.1/cdi-operator.yaml) and [CRD](https://github.com/kubevirt/containerized-data-importer/releases/download/v1.63.1/cdi-cr.yaml). Same as Kubevirt, operator brings up some other pods, in `cdi` namespace.
+
+Then a manual task is required. Exposing the CDI upload proxy. Default installation assigned a self-signed cert to its HTTP interface. So I have to use `IngressRouteTCP` and `DNSEndpoint` for Traefik to work.
+
+Kubevirt lab shows a demo to use CRD directly download disk image from Internet. It does not work now since many things were changed. I used the "upload" way.
+
+There are two types of VM disk, image file and physical disk. But I do not have physical disk usage at hand. So I just talk about image file here.
+
+First, in Virt-Manager, find the "disk" of the VM that I want to import, locate its disk image file. Then `kubectl virt image-upload dv $DATAVOLUME_NAME --insecure --size=128Gi --access-mode=ReadWriteOnce --force-bind --uploadproxy-url=https://cdi-uploadproxy_URL --image-path=disk_image.qcow2`.
+
+`--insecure` is because although the self-signed cert was generated by its installation, the tool working with `kubectl` does not take it.... `--force-bind` is for my CSI. Its default configuration does not bind the PV when it is created. The image format of RAW or QCOW2 are both supported. And it also supports GZ or XZ compression. But I am not quite found of it. If the data was corrupted, there was no error anywhere. The process just hang.
+
+After the uploading, `kubectl describe dv $DATAVOLUME_NAME` could be used to check the status of the data volume. Be noted that the corresponding PVC/PV is not directly specified in the manifest spec, but in its status.
+
+Using the uploaded data in VM resource is pretty much like how using it in Pod. `spec.template.spec.volumes.persistentVolumeClaim.claimName` is the place to write PVC name. Yes, not the `DataVolume` just worked with.
+
+## Multus-CNI
+
+This is the major part for me. My goal this time is just running a few individual VMs in K3S, which access (and be accessed) the LAN directly. And no in-cluster traffic is needed.
+
+With default CNI, there cannot be another NIC attached to the Pod. And in K3S, the CNI is flannel, unlike EKS, by default, the Pods share network with Nodes. Thus the Pods cannot access the bridging device on host.
+
+This is where Multus comes in. It is a layer between Pod and actual CNIs to provide the ability to have more NICs in a Pod.
+
+K3S has something special for Multus, hence the installation should follow [k3s doc](https://docs.k3s.io/networking/multus-ipams) rather than Multus doc. After the installation, a few manual modifications are required.
+
+First, check K3S agent endpoint. By default, it is on port 6444. Multus talks to Kubelet which in K3S is `k3s agent` (`k3s server` is also an agent), which takes `lb-server-port` to specify the port.
+
+Then checkout the kubeconfig for Multus, path specified during the installing process. The server address `clusters.cluster.server` generated is the in-cluster one. It does not work since Multus works in Node network. Change it to `server: https://localhost:6444`.
+
+After those, Multus is supposed to be able to contact with K3S. But we still cannot tell as it is not happening right now.
+
+For my usage, the next step is creating bridging network attachment definition with DHCP IPAM. Please refer to [cni doc](https://www.cni.dev/plugins/current/main/bridge/) for what the configuration looks like. Kubevirt doc is out-of-dated.
+
+```yaml
+apiVersion: "k8s.cni.cncf.io/v1"
+kind: NetworkAttachmentDefinition
+metadata:
+  name: vm-bridge
+spec:
+  config: '{
+      "cniVersion": "1.0.0",
+      "name": "bridge",
+      "type": "bridge",
+      "bridge": "kvm0",
+      "ipam": {
+        "type": "dhcp",
+        "daemonSocketPath": "/run/cni/dhcp.sock"
+      }
+    }'
+```
+
+Then use it as usual, specify `multus` in `spec.template.spec.networks`. Be noted that, in Kubevirt doc, it specifies `spec.template.spec.networks.multus.default` as `true` when using only Multus. This is totally wrong. This conf does not mean to set Multus as default NIC or anything. It tells Multus what is the default network to use. In detail, it adds `v1.multus-cni.io/default-network` annotation to virt-launcher pod with the network name set in `spec.template.spec.networks.multus.netwowrkName`. And default network requires the network be in the same K8S namespace as Multus pods. Otherwise the virt-launcher won't start.
+
+When using Multus-only (assuming one interface), the VM gets one interface. But the virt-launcher pod still gets the cluster interface.
+
+With DHCP setup (in CNI configuration), `kubectl get vmi` shows an address maintained by multus-dhcp pod, which mostly likely not the same address shown within the VM (got by its own DHCP client or static address). And the Multus one does not work.