Rebuilding a server and using rescue mode

What is rescue mode and when do you need it?

When your server goes offline and you can't even log in via SSH, the first solution that should come to mind is rescue mode. This is a temporary, emergency environment that allows you to access the server's file system without needing to boot the main operating system. In this mode, the server boots with a minimal kernel and environment (usually Linux-based), and you can perform repair, reinstallation, or data recovery operations.

The most common reasons you might need rescue mode include:

• Forgetting the root password and losing SSH access
• Corrupted configuration files like /etc/fstab that prevent booting
• Malware attacks or changes to file system permissions
• Need for a full operating system reinstall without losing important data
• Failed kernel updates that prevent the system from booting

In this article, you'll learn step by step how to use rescue mode to reinstall your server and recover your data. This guide is written for Debian/Ubuntu and CentOS/Rocky Linux distributions, but the general principles are the same for any Linux operating system.

Entering rescue mode; where to start?

Entering rescue mode is usually done through the hosting provider's or data center's management panel. At ServerNet, this option is available in your server management section, but the general steps are similar in most panels:

1. Log into the management panel and select your server.
2. Find the "Rescue Mode" or "Boot in Rescue" option.
3. Choose a temporary operating system (usually a lightweight Linux distribution).
4. Set a temporary password for accessing the rescue environment.
5. Reboot the server. Instead of a normal boot, the system will enter rescue mode.

After a few minutes, the connection information (IP address, SSH port, and password) will be displayed in the panel. Usually, the SSH port in rescue mode differs from the normal one (for example, port 22 changes to 2222).

Connecting to the rescue environment

To connect, run the following command from your local terminal:

ssh root@YOUR_SERVER_IP -p 2222

If you're using an SSH key, you may need to specify the key path:

ssh -i ~/.ssh/id_rsa root@YOUR_SERVER_IP -p 2222

After logging in, you're in a minimal environment where your server's main file system is usually not automatically mounted. To access your data, you need to identify and mount the partitions.

Identifying and mounting partitions

The first step is to find the disk partitions. Use the lsblk or fdisk -l command:

lsblk

The output usually looks something like this:

NAME   MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
sda      8:0    0   40G  0 disk
├─sda1   8:1    0  512M  0 part
├─sda2   8:2    0   39G  0 part
└─sda3   8:3    0    2G  0 part [SWAP]

In this example, sda2 is the main system (root) partition. Now mount it:

mkdir -p /mnt/rescue
mount /dev/sda2 /mnt/rescue

If your system uses LVM (which is common on many modern servers), you'll need to take additional steps:

vgscan
vgchange -ay
ls /dev/mapper/

Then mount the logical partition:

mount /dev/mapper/ubuntu--vg-root /mnt/rescue

Common error: mount failed

If you encounter a wrong fs type error while mounting, you've probably identified the partition's file system incorrectly. Check the file system type with the blkid command:

blkid /dev/sda2

The output will be something like TYPE="ext4". Then mount with the explicit option:

mount -t ext4 /dev/sda2 /mnt/rescue

Reinstalling the server while preserving data

One of the most important uses of rescue mode is reinstalling the operating system without losing critical data. There are two scenarios: full reinstallation (which erases data) and repair reinstallation (which only reinstalls system files). In rescue mode, you usually select the full reinstallation option from the panel, but before that, you need to back up important data.

Backing up important data

After mounting the main partition, you can transfer important files to an external destination. For example, to back up the /var/www directory (website content):

tar -czf /mnt/rescue/backup/www-backup.tar.gz -C /mnt/rescue var/www

Or if you want to back up the entire MySQL database, first copy the database files:

cp -a /mnt/rescue/var/lib/mysql /mnt/rescue/backup/mysql-backup

Then transfer the backup file to your local system with scp:

scp -P 2222 root@YOUR_SERVER_IP:/mnt/rescue/backup/www-backup.tar.gz .

Reinstalling the operating system

After ensuring the backup is complete, select the "Reinstall" or "OS Reinstall" option from the management panel. Choose the new operating system and wait for the process to finish. Once complete, the server will boot with a clean operating system, and you can restore your backed-up data.

Repairing the system without a full reinstall

In many cases, a full reinstall isn't necessary, and you can fix the problem from within rescue mode. This method is faster and leaves your data untouched.

Fixing the root password issue

If you've forgotten the root password, after mounting the main partition, chroot into the file system:

mount --bind /dev /mnt/rescue/dev
mount --bind /proc /mnt/rescue/proc
mount --bind /sys /mnt/rescue/sys
chroot /mnt/rescue /bin/bash

Now you're in the chroot environment and can change the password:

passwd root

After changing the password, exit chroot and reboot the server:

exit
umount -R /mnt/rescue
reboot

Repairing a corrupted fstab file

If the system won't boot due to an error in /etc/fstab, edit the file:

nano /mnt/rescue/etc/fstab

Check the lines related to mounted partitions. A common mistake is using the wrong identifier (UUID). To find the correct UUID:

blkid

Then replace the correct UUID in the fstab file. After saving the changes, reboot the server.

Reinstalling GRUB

If the system encounters a GRUB error after a kernel update or partition changes, you need to reinstall the bootloader. In the chroot environment:

grub-install /dev/sda
update-grub

For UEFI systems, the path is different:

grub-install --target=x86_64-efi --efi-directory=/boot/efi
update-grub

Troubleshooting common issues in rescue mode

In this section, we'll cover several common problems users encounter when working with rescue mode.

Issue: Network access in rescue mode

Sometimes in rescue mode, the network isn't automatically enabled. If you need to download packages or transfer files, enable the network manually:

ip link set eth0 up
dhclient eth0

Or if you have a static IP:

ip addr add YOUR_IP/24 dev eth0
ip route add default via YOUR_GATEWAY

Issue: Insufficient space for backup

If the main partition is nearly full and you don't have enough space for a backup, you can use compression or transfer files directly with rsync to a remote system:

rsync -avz -e "ssh -p 2222" /mnt/rescue/var/www/ user@backup-server:/backup/www/

Issue: Corrupted file system

If you encounter an Input/output error while mounting, check and repair the file system:

fsck -y /dev/sda2

This command may take a few minutes. After it finishes, try mounting again.

Recovering data after reinstallation

After reinstalling the operating system, you need to restore your backed-up data to the server. First, transfer the backup to the server:

scp www-backup.tar.gz root@YOUR_SERVER_IP:/tmp/

Then, on the server, extract the file:

tar -xzf /tmp/www-backup.tar.gz -C /var/www/

For the database, restore the copied files to their original location and restart the MySQL service:

systemctl restart mysql

Summary and final tips

Rescue mode is a vital tool for any server administrator. By mastering these techniques, you can stay calm in critical situations and save your data. A few final tips:

• Always maintain regular backups of important data outside the server. Rescue mode is a last resort, not a replacement for backups.
• Before any major change (kernel update, partition changes), take a snapshot from the management panel.
• Read the rescue mode documentation in the ServerNet panel; there may be special features like automatic partition mounting in the rescue environment.
• Practice! Deliberately break a test server and try to fix it using rescue mode. This experience will be invaluable in real-world situations.

With this guide, you're ready to handle any crisis on your server. If you have questions, the comments section is available for you.