How I Recovered a Broken SolusVM 2 KVM Virtual Machine Using Chroot, Repaired the Linux System, and Safely Migrated Critical Data

Recently, I worked on a challenging recovery involving a SolusVM 2 KVM virtual machine that would no longer boot.

What initially appeared to be a kernel or filesystem problem turned out to involve multiple layers of system corruption:

• Broken and inconsistent dpkg packages
• Missing and zero-byte system binaries
• Failed initramfs generation
• Broken initramfs-tools hooks
• Missing /bin utilities
• Incorrect filesystem mount configuration
• A damaged kernel installation process
• KVM permission problems on the virtualization node
• The need to safely recover and migrate critical data before continuing troubleshooting

This article explains the recovery process, the commands used, what each command does, and how critical website and database data can be safely moved to another server from recovery mode.

Note: All IP addresses, usernames, domains, database names, VM UUIDs, logical volumes, and paths in this article are examples. Replace them with values appropriate for your environment.

1. Understanding the Environment

The affected virtual machine was hosted on a SolusVM 2 KVM node using an LVM-backed virtual disk.

The first step was identifying the VM storage layout:

lsblk

The VM disk appeared similar to:

vg0-demo_vm
├─vg0-demo_vmp1     149G
├─vg0-demo_vmp14      4M
├─vg0-demo_vmp15    106M
└─vg0-demo_vmp16    913M

In this example:

• p1 = root filesystem
• p15 = EFI System Partition
• p16 = /boot partition

Correctly identifying these partitions is extremely important. Mounting or modifying the wrong logical volume on a production virtualization node can cause serious data loss.

2. Mounting the Broken VM Filesystem

I created a temporary recovery directory:

mkdir -p /mnt/demo-vm

Then mounted the VM root filesystem:

mount /dev/mapper/vg0-demo_vmp1 /mnt/demo-vm

Next, I mounted the separate boot partition:

mkdir -p /mnt/demo-vm/boot

mount /dev/mapper/vg0-demo_vmp16 \
/mnt/demo-vm/boot

Then the EFI partition:

mkdir -p /mnt/demo-vm/boot/efi

mount /dev/mapper/vg0-demo_vmp15 \
/mnt/demo-vm/boot/efi

I verified everything before proceeding:

df -h

and:

mount | grep /mnt/demo-vm

At this stage, the VM filesystem was accessible from the virtualization host.

3. Preparing the Filesystem for Chroot

Mounting the filesystem alone is not enough.

For tools such as apt, dpkg, update-initramfs, and update-grub to function correctly, the chroot environment needs access to several virtual filesystems.

I mounted them using:

mount --bind /dev /mnt/demo-vm/dev
mount --bind /dev/pts /mnt/demo-vm/dev/pts
mount -t proc /proc /mnt/demo-vm/proc
mount -t sysfs /sys /mnt/demo-vm/sys
mount --bind /run /mnt/demo-vm/run

DNS resolution may also be required when reinstalling packages:

cp -L /etc/resolv.conf \
/mnt/demo-vm/etc/resolv.conf

I then entered the damaged system:

chroot /mnt/demo-vm /bin/bash

Inside the chroot, I restored a standard command path:

export PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin

I confirmed that the environment was working:

echo "CHROOT WORKS"
id
df -h
mount | grep -E '/boot|/boot/efi'

At this point, I was operating inside the broken VM installation rather than the host system.

4. Diagnosing the Package Corruption

The initial package repair command was:

dpkg --configure -a

This reported that some packages were in an inconsistent state and that kernel configuration was failing because initramfs could not be generated.

To inspect package integrity, I used:

dpkg -V

For easier review:

dpkg -V 2>&1 | tee /root/dpkg-verify.txt

This revealed several important problems:

• Missing executables
• Modified package files
• Zero-byte system binaries
• Missing systemd services
• Broken initramfs dependencies

I also searched specifically for zero-byte system files:

find /bin /sbin /usr/bin /usr/sbin /usr/lib/systemd \
-xdev -type f -size 0 -print

This was one of the most useful diagnostic commands during the recovery.

Several critical binaries had become empty files.

Examples included services such as:

• rsyslogd
• multipathd
• agetty
• systemd-timesyncd

A file can still exist at the expected path while containing zero bytes, so checking only with ls is not sufficient.

I verified suspicious binaries with:

ls -lh /path/to/binary

and:

file /path/to/binary

A healthy executable should normally be identified as an ELF binary.

5. Repairing the Broken Package State

One of the first problems involved the shell packages.

I reinstalled the affected packages:

apt-get install --reinstall -y bash

and:

apt-get install --reinstall -y dash

After that, I continued checking the package manager:

dpkg --configure -a

The shell problem was resolved, but kernel configuration continued failing during initramfs generation.

6. Diagnosing the Initramfs Failure

The kernel package installation failed with errors similar to:

update-initramfs: Generating /boot/initrd.img-6.8.0-XXX-generic

E: /usr/share/initramfs-tools/hooks/kbd failed with return 1.

Instead of repeatedly reinstalling the kernel, I investigated the failing hook.

The hook was inspected with:

cat /usr/share/initramfs-tools/hooks/kbd

It expected the following utilities:

• /bin/setfont
• /bin/kbd_mode
• /bin/loadkeys

I checked whether they existed:

ls -lh \
/bin/setfont \
/bin/kbd_mode \
/bin/loadkeys

The package itself showed that the correct executables were installed under /usr/bin:

dpkg -L kbd | grep -E '/(setfont|kbd_mode|loadkeys)$'

The output showed paths similar to:

/usr/bin/setfont
/usr/bin/kbd_mode
/usr/bin/loadkeys

This explained why the hook was failing.

7. Restoring the Missing KBD Utilities

I restored the missing binaries individually:

cp -a /usr/bin/setfont /bin/setfont
cp -a /usr/bin/kbd_mode /bin/kbd_mode
cp -a /usr/bin/loadkeys /bin/loadkeys

Then verified them:

ls -lh \
/bin/setfont \
/bin/kbd_mode \
/bin/loadkeys

and:

file \
/bin/setfont \
/bin/kbd_mode \
/bin/loadkeys

All three utilities were now recognized as valid ELF executables.

A useful lesson here was that copying the binaries was not enough until every executable referenced by the hook had been checked.

8. Removing Backup Hooks from the Active Initramfs Directory

During troubleshooting, I created a backup copy of an initramfs hook.

The problem was that initramfs-tools executes files located in its hooks directory.

That meant a backup such as:

kbd.backup

could also be executed and fail.

I checked for duplicate hook files:

find /usr/share/initramfs-tools/hooks/ \
-maxdepth 1 \
-type f \
-name 'kbd*' \
-ls

Any troubleshooting backup should be moved outside the active hooks directory.

For example:

mkdir -p /root/initramfs-hook-backups
mv /usr/share/initramfs-tools/hooks/kbd.backup \
/root/initramfs-hook-backups/

This was a small but important detail.

9. Rebuilding the Initramfs

Once the missing dependencies were repaired, I removed the failed image:

rm -f /boot/initrd.img-6.8.0-XXX-generic

Then rebuilt it:

update-initramfs -c -k 6.8.0-XXX-generic

This time the command completed successfully.

I verified the generated image:

ls -lh /boot/initrd.img-6.8.0-XXX-generic

and:

file /boot/initrd.img-6.8.0-XXX-generic

I also verified the kernel:

ls -lh \
/boot/vmlinuz-6.8.0-XXX-generic \
/boot/initrd.img-6.8.0-XXX-generic

10. Repairing Other Zero-Byte System Binaries

The integrity scan had identified additional damaged executables.

Instead of assuming that the entire operating system was unusable, I mapped each damaged file back to its package and reinstalled the appropriate package.

Examples included:

apt-get install --reinstall -y util-linux
apt-get install --reinstall -y rsyslog
apt-get install --reinstall -y multipath-tools
apt-get install --reinstall -y systemd-timesyncd
apt-get install --reinstall -y logrotate

One package configuration failed because /bin/more was missing, although /usr/bin/more existed.

I verified it:

ls -lh /usr/bin/more
file /usr/bin/more

Then restored it:

cp -a /usr/bin/more /bin/more

Afterward, I configured the affected package:

dpkg --configure util-linux

The configuration completed successfully.

11. Completing the Package Repair

Once the critical binaries and initramfs problems were resolved, I ran:

dpkg --configure -a

Then checked for unfinished package operations:

dpkg --audit

A clean dpkg --audit result indicated that no packages remained partially installed or unconfigured.

12. Verifying the System Again

I repeated the zero-byte scan:

find /bin /sbin /usr/bin /usr/sbin /usr/lib/systemd \
-xdev -type f -size 0 -print

I also checked the repaired services:

ls -lh \
/usr/sbin/agetty \
/usr/sbin/rsyslogd \
/usr/sbin/multipathd \
/usr/lib/systemd/systemd-timesyncd

Then:

file \
/usr/sbin/agetty \
/usr/sbin/rsyslogd \
/usr/sbin/multipathd \
/usr/lib/systemd/systemd-timesyncd

The files were now valid ELF executables instead of empty files.

13. Verifying /etc/fstab

Because the VM had separate root, boot, and EFI partitions, I verified the filesystem configuration:

cat /etc/fstab

A generic example would be:

LABEL=cloudimg-rootfs / ext4 discard,commit=30,errors=remount-ro 0 1

LABEL=BOOT /boot ext4 defaults 0 2

LABEL=UEFI /boot/efi vfat umask=0077 0 1

Incorrect fstab entries can prevent an otherwise repaired VM from booting correctly.

14. Updating GRUB

After repairing the kernel and initramfs, I rebuilt the GRUB configuration:

update-grub

Then verified again:

dpkg --configure -a
dpkg --audit

At this stage, the guest operating system recovery work was largely complete.

Safely Backing Up Data Before Further Recovery Work

Even after repairing the VM, I wanted a safe copy of critical data before attempting additional boot troubleshooting.

This is an important part of server recovery:

Repair the system, but protect the data independently.

15. Backing Up Website Files from Recovery Mode

Because the VM filesystem was mounted and accessible through chroot, application data could be copied directly.

For example, a website might be located at:

/home/demo_user/web/demo.example.com/public_html/

To transfer it to another server using a custom SSH port:

rsync -aHAXvP \
-e "ssh -p 22022" \
/home/demo_user/web/demo.example.com/public_html/ \
root@192.0.2.20:/home/demo_user/web/demo.example.com/public_html/

The options used here are:

• -a Archive mode
• -H Preserve hard links
• -A Preserve ACLs
• -X Preserve extended attributes
• -v Verbose output
• -P Show progress and preserve partially transferred files

One of the major advantages of rsync is that the command can be run again.

If most files have already been transferred, rsync will only send new or changed data.

16. Recovering a Database When MySQL Is Not Running Normally

The VM was being repaired through chroot, which meant the normal service manager was not necessarily available.

Attempting a database dump could result in:

Can't connect to local MySQL server through socket

I first identified the installed database server:

dpkg -l | grep -Ei 'mariadb-server|mysql-server'

Then located the actual server binary:

find /usr /opt \
-type f \
\( -name 'mariadbd' -o -name 'mysqld' \) \
-ls 2>/dev/null

In this example, MySQL was installed and the server binary was:

/usr/sbin/mysqld

I checked the database directory:

ls -lah /var/lib/mysql

Before manually starting MySQL, the runtime directory must exist:

mkdir -p /run/mysqld
chown mysql:mysql /run/mysqld

I then started MySQL locally without network access:

/usr/sbin/mysqld \
--user=mysql \
--datadir=/var/lib/mysql \
--socket=/run/mysqld/mysqld.sock \
--pid-file=/run/mysqld/mysqld.pid \
--skip-networking \
> /root/mysql-recovery.log 2>&1 &

Using:

--skip-networking

was useful during recovery because the temporary database process did not need to accept remote network connections.

I verified the process:

ps aux | grep '[m]ysqld'

Checked the socket:

ls -l /run/mysqld/mysqld.sock

And reviewed the recovery log:

tail -100 /root/mysql-recovery.log

17. Creating a Safe Database Dump

Once MySQL was running, I created a backup directory:

mkdir -p /root/db-backup

Then dumped the example database:

mysqldump \
--single-transaction \
--quick \
--routines \
--triggers \
--events \
demo_database \
> /root/db-backup/demo_database.sql

The options are important:

• --single-transaction Creates a consistent snapshot for transactional tables.
• --quick Reads rows incrementally instead of buffering large tables in memory.
• --routines Includes stored procedures and functions.
• --triggers Includes database triggers.
• --events Includes scheduled database events.

I verified the dump before transferring it:

ls -lh /root/db-backup/demo_database.sql

18. Compressing the Database Backup

To reduce transfer size:

gzip -9 /root/db-backup/demo_database.sql

The result:

/root/db-backup/demo_database.sql.gz

To extract it later:

gunzip demo_database.sql.gz

Or, if I wanted to preserve the compressed copy:

gzip -dk demo_database.sql.gz

19. Transferring the Database Backup Safely

The compressed backup was transferred using rsync over a custom SSH port:

rsync -avP \
-e "ssh -p 22022" \
/root/db-backup/demo_database.sql.gz \
root@192.0.2.20:/root/db-backup/

After transfer, the file should be verified on the destination server:

ls -lh /root/db-backup/demo_database.sql.gz

For stronger integrity verification, checksums can be compared.

On the source:

sha256sum /root/db-backup/demo_database.sql.gz

On the destination:

sha256sum /root/db-backup/demo_database.sql.gz

Both hashes should match.

Leaving the Chroot Safely

After completing repairs and backups, I exited the chroot:

exit

The filesystems then needed to be unmounted in reverse order.

For example:

umount /mnt/demo-vm/dev/pts
umount /mnt/demo-vm/dev
umount /mnt/demo-vm/proc
umount /mnt/demo-vm/run

The /sys mount required additional attention because nested mounts such as efivarfs could keep the root filesystem busy.

I checked active mounts:

mount | grep /mnt/demo-vm

Then recursively unmounted /sys:

umount -R /mnt/demo-vm/sys

After that:

umount /mnt/demo-vm/boot/efi
umount /mnt/demo-vm/boot
umount /mnt/demo-vm

If the filesystem remained busy, I checked:

fuser -vm /mnt/demo-vm

and:

findmnt -R /mnt/demo-vm

This helped identify nested mounts that had not yet been detached.

A Separate KVM Permission Problem on the SolusVM 2 Node

After the guest operating system repairs, another issue appeared when starting the VM:

Could not access KVM kernel module: Permission denied

qemu-kvm: -accel kvm: failed to initialize kvm: Permission denied

I verified the KVM modules:

lsmod | grep kvm

Then checked /dev/kvm:

stat /dev/kvm

The important discovery was that /dev/kvm had an unexpected group owner.

I checked the QEMU user:

id qemu

The QEMU process belonged to the kvm group, but /dev/kvm was assigned to a different group.

That meant the virtualization process could not access hardware acceleration.

This was a host-level permission issue, separate from the corruption inside the guest VM.

That distinction is important:

A VM can have both guest operating system problems and virtualization host problems at the same time.

Key Lessons from This Recovery

This incident reinforced several important principles for Linux and virtualization recovery:

• Do not assume a VM that fails to boot has only one problem.
• Always identify the VM storage layout before mounting anything.
• Use chroot to repair the installed operating system from the virtualization host.
• Run dpkg -V to identify missing or modified package files.
• Search explicitly for zero-byte system binaries.
• Use file to confirm whether critical executables are valid.
• Investigate the exact failing initramfs hook instead of repeatedly reinstalling the kernel.
• Never leave backup scripts inside active initramfs-tools/hooks directories.
• Verify /boot, /boot/efi, the kernel, initramfs, fstab, and GRUB before attempting another boot.
• Protect application data before continuing risky recovery operations.
• rsync is extremely useful for resumable server-to-server recovery transfers.
• A database server can be started manually inside a recovery environment when systemd is unavailable.
• Using --skip-networking reduces unnecessary exposure during temporary database recovery.
• Always verify transferred backups with checksums.
• Diagnose guest-level and host-level problems separately.

Final Thoughts

This was not a simple “reinstall the kernel and reboot” situation.

The recovery involved working through multiple layers:

SolusVM 2 → KVM → LVM storage → filesystem mounts → chroot → dpkg → system binaries → initramfs → kernel → GRUB → application data → MySQL recovery → rsync migration → KVM host permissions.

The most valuable part of the process was following the evidence from one failure to the next.

When dpkg failed, I investigated the package state.

When initramfs failed, I investigated the exact hook.

When the hook failed, I checked every binary it depended on.

When the VM still could not start, I separated guest operating system problems from KVM host permissions.

And before continuing with further recovery attempts, I made sure critical website and database data had been safely copied to another server.

Complex infrastructure incidents are rarely solved by a single command. They are solved through careful diagnostics, controlled changes, verification after every step, and always keeping data safety as the highest priority.