Skip to content

tests: fix failed ostree.sync on c10s/rhcos10.1 #4328

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
HuijingHei wants to merge 1 commit into
base: main
Choose a base branch
from
Open

tests: fix failed ostree.sync on c10s/rhcos10.1 #4328

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
49 changes: 31 additions & 18 deletions mantle/kola/tests/ostree/sync.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -150,21 +150,21 @@ func ostreeSyncTest(c cluster.TestCluster) {
version: 1.5.0
storage:
directories:
- path: /var/tmp/data1
- path: /var/mnt/data1
mode: 0777
- path: /var/tmp/data2
- path: /var/mnt/data2
mode: 0777
- path: /var/tmp/data3
- path: /var/mnt/data3
mode: 0777
- path: /var/tmp/data4
- path: /var/mnt/data4
mode: 0777
files:
- path: /etc/systemd/system.conf
overwrite: true
contents:
inline: |
[Manager]
DefaultTimeoutStopSec=10s
DefaultTimeoutStopSec=30s
- path: /usr/local/bin/nfs-random-write.sh
mode: 0755
overwrite: true
Expand All @@ -173,10 +173,26 @@ storage:
#!/bin/bash
i=$1
while true; do
sudo dd if=/dev/urandom of=/var/tmp/data$i/test bs=4096 count=2048 conv=notrunc oflag=append &> /dev/null
sleep 0.1
sudo rm -f /var/tmp/data$i/test
done`)
dd if=/dev/urandom of=/var/mnt/data$i/test bs=4096 count=2048 conv=notrunc oflag=append &> /dev/null
sleep 0.5
rm -f /var/mnt/data$i/test
done
systemd:
units:
- name: nfs-random-write@.service
enabled: false
contents: |
[Unit]
Description=Run NFS Random Write Test %I
Before=umount.target
After=remote-fs.target
[Service]
ExecStart=/usr/local/bin/nfs-random-write.sh %I
#ExecStopPost=/bin/umount -lf /var/mnt/data%I || true
Copy link
Member

@marmijo marmijo Dec 12, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
#ExecStopPost=/bin/umount -lf /var/mnt/data%I || true
ExecStopPost=/bin/umount -lf /var/mnt/data%I || true

I think we should uncomment the ExecStopPost here so the service can clean up its own NFS mount which avoids the later var-mnt-data*.mount/var.mount shutdown unmount failures.

Copy link
Member Author

@HuijingHei HuijingHei Dec 15, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we should uncomment the ExecStopPost here so the service can clean up its own NFS mount which avoids the later var-mnt-data*.mount/var.mount shutdown unmount failures.

Thank you @marmijo for your review and comments, I am trying to test to see how long the vm will start. Hopefully will find out how long we need for the client to boot, but agree this is a workaround.

Copy from comment
the client-10 hang about 806s ([23910.934397] ~ [23104.029251]) and finally success to reboot. While the client-9 hang 313s ([18009.805121] ~ [17696.012479]) which is much shorter.

Type=simple
KillMode=control-group
[Install]
WantedBy=multi-user.target`)
opts := platform.MachineOptions{
MinMemory: 2048,
}
Expand All @@ -198,7 +214,7 @@ storage:
// entry point /var/nfs with fsid=0 will be root for clients
// refer to https://access.redhat.com/solutions/107793
_ = c.MustSSHf(nfs_client, `for i in $(seq 4); do
sudo mount -t nfs4 %s:/share$i /var/tmp/data$i
sudo mount -t nfs4 %s:/share$i /var/mnt/data$i
done`, nfs_server.MachineAddress)

mounts := c.MustSSH(nfs_client, "sudo df -Th | grep nfs | wc -l")
Expand All @@ -217,23 +233,20 @@ storage:

func doSyncTest(c cluster.TestCluster, client platform.Machine, m platform.Machine) {
// Do simple touch to make sure nfs server works
c.RunCmdSync(client, "sudo touch /var/tmp/data3/test")
c.RunCmdSync(client, "sudo touch /var/mnt/data3/test")
// Continue writing while doing test
// gets run using systemd unit
for i := 1; i <= 4; i++ {
cmd := fmt.Sprintf("sudo systemd-run --unit=nfs%d --no-block sh -c '/usr/local/bin/nfs-random-write.sh %d'", i, i)
_, err := c.SSH(client, cmd)
if err != nil {
c.Fatalf("failed to run nfs-random-write: %v", err)
}
_, err := c.SSH(client, "sudo systemctl start --no-block nfs-random-write@{1..4}.service")
if err != nil {
c.Fatalf("failed to run nfs-random-write@.service: %v", err)
}

// block NFS traffic on nfs server
c.RunCmdSync(m, "sudo /usr/local/bin/block-nfs.sh")
// Create a stage deploy using kargs while writing
c.RunCmdSync(client, "sudo rpm-ostree kargs --append=test=1")

err := client.Reboot()
err = client.Reboot()
if err != nil {
c.Fatalf("Couldn't reboot machine: %v", err)
}
Expand Down
Loading