Tech Notes

Tech Note #20090002

On systems that are built by most standard OS installs, the LABEL= parameter is used in several places to refer symbolically to disk partitions. Unfortunately, this can lead to all sorts of problems, including failure to upgrade the system properly during package updates and/or failure to boot the system, including when the system is upgrade or new drives are added.

Problem Description:

First, let us start by saying that the "LABEL=" parameter in the boot loader config file and file system table is about the dumbest idea anyone has come up with in a long, long time. All its ever done for us is provide no end of grief. The fact that it is used by default by every OS install these days is just sad. If anybody really wants this stupid feature, they should have to specifically turn it on, not have it turned on by default so it can continually screw people.

How does it screw people? Well, we've seen systems boot from one drive but run software from another. We've seen disks disappear after a system upgrade. We've seen the system try to boot from the wrong drive. And more. Personally, we've been burned by this feature at least 20 times.

So, what is happening? Basically, the OS labels disk partitions with either uniquely-generated labels or well-known labels. Then, one uses those labels for the boot loader and/or in the file system table to mount the labeled partitions. Sounds like an OK idea. However, if the disk label changes (e.g. when one copies partitions from one drive to another) or, better yet, if a second partition appears (e.g. when one adds another disk drive with the same well-known label), it spells trouble.

Here's how the problem manifests itself. You do a system upgrade and grubby complains that it can't find a suitable template:


grubby fatal error: unable to find a suitable template

This happens either because the "LABEL=/boot" option (which grubby uses) ends up pointing to somewhere else and grubby can't find the grub.conf file or there is a bug (which we've seen reported in RHEL but which may now have been fixed) in the "LABEL=" lookup function. This problem with grubby is, incidentally, flagged only as a warning, during system upgrade, but should probably be marked as a fatal error, since your system usually won't boot after you get it.

Moving right along, the problem next manifests itself at boot time when the kernel can't find stuff that should oughtta be there (like the root file system). You may see the problem as:


mount: Could not find filesystem '/dev/root'
setuproot: moving /dev failed: No such file or directory
setuproot: error mounting /proc: No such file or directory
setuproot: error mounting /sys: No such file or directory
switchroot: mount failed: No such file or directory
Kernel panic - not syncing: Attempted to kill init!

Further problems can appear if the kernel actually boots but the "LABEL=" parameter is used in /etc/fstab. Then, you get the wrong partition mounted and can have no end of fun.

Problem Resolution:

Since we've gotten burned by this feature so many times, we try to make it a policy to always go and remove the "LABEL=" parameter everywhere it sneaks in. For example, in /boot/grub/grub.conf. And, in /etc/fstab. You simply take out "LABEL=ScreamingOffIntoTheWeeds" and put in "/dev/sda3". Since we always wire up the drives to the same physical ports (e.g. IDE0/Master or SATA0), we never have a problem. If one copies a drive, adds drives, etc., etc., things keep working.

Unfortunately, the OS guys are really good at sneaking the "LABEL=" parameter back in. So, the chances are we're gonna get burnt again and again. However, recovery from this problem is quite easy, using a copy of KNOPPIX. When the problem happens, simply boot the KNOPPIX CD and mount the broken system partitions (e.g. /dev/sda1 for "/boot" and /dev/sda3 for "/", if you configured your system that way, or maybe just /dev/sda1 for "/" -- you get the idea), something like this:

su
mount /dev/sda1

KNOPPIX pre configures mount points in the /media tree for all of the partitions it finds connected to the system so you'll find what you're looking for under, for example, /media/sda1. If this is your boot partition, you can do:

cd /media/sda1/grub
kwrite grub.conf &

And commence editing grub.conf. Same thing for /etc/fstab:

su
mount /dev/sda3
cd /media/sda3/etc
kwrite fstab &

Incidentally, we're not saying that all that stuff that people told you about the device driver being missing from initrd and/or the kernel, etc., etc. is wrong. Only that, if you upgrade from a working distro to one that breaks or you add a second disk drive and your system no longer boots, you ought to take a look at the use of the "LABEL=" parameter in your grub.conf and/or /etc/fstab. It couldn't hurt.