VR healthchecks fail on VPC with ConfigDrive (rather than VirtualRouter)

[bhouse-nexthop]

problem

When viewing the healthchecks on a VPC that is using ConfigDrive for address assignment rather than the VirtualRouter, it shows these healthcheck failures:

All other health checks are successful.

ConfigDrive has to be used due to RHEL 10 / AlmaLinux 10 / RockyLinux 10 not properly supporting cloud-init via dhcp with CloudStack (think this is due to the use of systemd's built-in dhcp client in these recent releases).

versions

4.22.0

The steps to reproduce the bug

1. Create VPC using ConfigDrive instead of Virtual Router for address assignment
2. Create Network Tier
3. Create VM instance in Network Tier
4. Observe healthcheck failures in the Virtual Router

What to do about it?

Ignore it?

[boring-cyborg]

Thanks for opening your first issue here! Be sure to follow the issue template!

[weizhouapache]

@bhouse-nexthop
did you say that the Dhcp/Dns provider ( UserData provider too) is ConfigDrive, not VpcVirtualRouter ?
have you tested Isolated network too ?

related to #9329

[bhouse-nexthop]

@bhouse-nexthop did you say that the Dhcp/Dns provider ( UserData provider too) is ConfigDrive, not VpcVirtualRouter ? have you tested Isolated network too ?

related to #9329

Here's the network offering config:

I haven't tested a basic isolated network to see what that looks like. But I do have 3 VPCs using that offering and all exhibit the same behavior.

[weizhouapache]

@bhouse-nexthop did you say that the Dhcp/Dns provider ( UserData provider too) is ConfigDrive, not VpcVirtualRouter ? have you tested Isolated network too ?
related to #9329

Here's the network offering config:

I haven't tested a basic isolated network to see what that looks like. But I do have 3 VPCs using that offering and all exhibit the same behavior.

@bhouse-nexthop
can you run the following commands in a VPC VR ?

/root/health_checks/dhcp_check.py
/root/health_checks/dns_check.py

cp /root/health_checks_data.json /root/health_checks_data.json.bkp
sed -i "s/dhcp=true,dns=true/dhcp=false,dns=false/g" /root/health_checks_data.json

/root/health_checks/dhcp_check.py
/root/health_checks/dns_check.py

[bhouse-nexthop]

Are there any flags i need to pass? I'm not getting any output from the scripts running it manually like that, so I didn't modify the json file like you said since I didn't see an error...

root@hv102:~# ssh -p 3922 -i /var/lib/cloudstack/management/.ssh/id_rsa 169.254.239.169
Linux r-664-VM 6.1.0-40-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.1.153-1 (2025-09-20) x86_64

The programs included with the Debian GNU/Linux system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.

Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
permitted by applicable law.

The programs included with the Debian GNU/Linux system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.

Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
permitted by applicable law.
root@r-664-VM:~# /root/health_checks/dhcp_check.py
root@r-664-VM:~# echo $?
0
root@r-664-VM:~# /root/health_checks/dns_check.py
root@r-664-VM:~# echo $?
0

[bhouse-nexthop]

Guess I should have read the script, you have to pass advanced as an argument for it to do anything at all:

root@r-664-VM:~# /root/health_checks/dns_check.py advanced
Missing entries for VMs in /etc/hosts -
10.252.4.11 ntp1, 10.252.4.12 ntp2, 10.252.5.11 secureproxy1, 10.252.5.12 secureproxy2, 10.252.5.22 cacheproxy2, 10.252.5.21 cacheproxy1, 10.252.6.11 ldapproxy1, 10.252.6.12 ldapproxy2, 10.252.0.10 bastion, 10.252.7.12 nfs2, 10.252.7.11 nfs1, 10.252.1.12 dns2, 10.252.1.11 dns1, 10.252.3.11 ipa1, 10.252.3.12 ipa2, 10.252.3.13 ipa3, 10.252.240.11 vr1, 10.252.240.12 vr2, 10.252.2.11 mirror1, 10.252.2.11 mirror1, 10.252.12.11 sccache1, 10.252.12.12 sccache2, 10.252.12.13 sccache3, 10.252.13.146 garm, 10.252.14.11 dhcp1, 10.252.14.12 dhcp2
root@r-664-VM:~# /root/health_checks/dhcp_check.py advanced
Missing elements in dhcphosts.txt -
02:01:00:dc:00:16 10.252.4.11 ntp1, 02:01:00:dc:00:18 10.252.4.12 ntp2, 02:01:00:dd:00:17 10.252.5.11 secureproxy1, 02:01:00:dd:00:18 10.252.5.12 secureproxy2, 02:01:00:dd:00:19 10.252.5.22 cacheproxy2, 02:01:00:dd:00:1b 10.252.5.21 cacheproxy1, 02:01:00:de:00:16 10.252.6.11 ldapproxy1, 02:01:00:de:00:17 10.252.6.12 ldapproxy2, 02:01:00:df:00:2e 10.252.0.10 bastion, 02:01:00:e0:00:1f 10.252.7.12 nfs2, 02:01:00:e0:00:20 10.252.7.11 nfs1, 02:01:00:e2:00:04 10.252.1.12 dns2, 02:01:00:e2:00:16 10.252.1.11 dns1, 02:01:00:e3:00:1b 10.252.3.11 ipa1, 02:01:00:e3:00:1c 10.252.3.12 ipa2, 02:01:00:e3:00:1d 10.252.3.13 ipa3, 02:01:00:e8:00:01 10.252.240.11 vr1, 02:01:00:e8:00:02 10.252.240.12 vr2, 02:01:00:e1:00:2f 10.252.2.11 mirror1, 02:01:00:e1:00:2f 10.252.2.11 mirror1, 02:01:00:f4:00:03 10.252.12.11 sccache1, 02:01:00:f4:00:04 10.252.12.12 sccache2, 02:01:00:f4:00:05 10.252.12.13 sccache3, 02:01:01:1e:00:0c 10.252.13.146 garm, 02:01:01:1f:00:01 10.252.14.11 dhcp1, 02:01:01:1f:00:04 10.252.14.12 dhcp2

root@r-664-VM:~# cp /root/health_checks_data.json /root/health_checks_data.json.bkp
sed -i "s/dhcp=true,dns=true/dhcp=false,dns=false/g" /root/health_checks_data.json
root@r-664-VM:~# /root/health_checks/dns_check.py advanced
All 0 VMs are present in /etc/hosts
root@r-664-VM:~# /root/health_checks/dhcp_check.py advanced
All 0 VMs are present in dhcphosts.txt

[weizhouapache]

Guess I should have read the script, you have to pass advanced as an argument for it to do anything at all:

oh yeah, you found it right. @bhouse-nexthop
advanced is missing in my previous reply.

I think I know what's wrong. It is a bug, I will create a PR for it