Surviving fsck.ext3: Unable to resolve LABEL problem

Published September 26th, 2009

Due to a forced hard reboot, thanks to the Data Center’s faulty PDU, one of our servers got the blues. After getting Nagios alert, we logged onto it via remote KVM and noticed that it is hanging at the maintenance shell prompt.

After logging in as root to the maintenance shell, we noticed that the system was complaining about mismatched label for the root partition.

When we configured the system we setup the root partitoin, /dev/sda1, using label / using the disk setup program that starts  up as part of the centos setup process. However, somehow the /etc/fstab now shows the root partition as LABEL=/1 instead of LABEL=/ and similarly we had our /var and /usr partition labels got messed up from being /usr and /var to /usr1 and /var1.

Since in maintenance mode, we could not edit /etc/fstab and single user mode was not an option since it also gets stuck at the maintenance prompt, here is how we survived this problem:

• Since we cannot go to single-user mode as disk mounting fails because of the bad labels, we decided to temporarily switch the labels to match the /etc/fstab using the following commands:

  # e2label /dev/sda1 /1
  # e2label /dev/sda2 /usr1
  # e2label /dev/sda4 /var1

• Next, we rebooted the system and when the grub prompt comes up, we edited the grub kernel line to use LABEL=/1 instead of LABEL=/ (as /etc/grub.conf was not corrupt)
• After editing the kernel line we booted into single user mode and then edited both /etc/fstab and restored the labels back to original by setting the as follows:

  # e2label /dev/sda1 /
  # e2label /dev/sda2 /usr
  # e2label /dev/sda4 /var

• One more reboot and we were able to login from the console as usual

Category: Disks | Tags: