Running out of disk space even when 20% is free!

Published June 8th, 2009

Today, we had a panic call from a customer who was hosting a small app on a non-dedicated server environment with us. The app uses a ton of images and was hosted on a box with terabytes of disk space. They called with a message saying that the system is out of disk space.

After investigating, we found out that we had plenty of disk space available and yet the error that they reported was showing up on Web app and even when we tried to create new files in the root partition.

Our favorite editor, vi, was showing:

E212: Can't open file for writing

At first, we thought we are in big trouble as the root partition has probably gone into read-only mode due to some disk hardware failure. But, it turned out that even though df -h reports 20% free disk space, we actually ran out of all the inodes in the root partition. To find what percent of inodes were available, we ran df -i and saw a 100% utilization for the root partition.

There is no way to add new inodes on a running filesystem without recreating the filesystem itself. So we had to figure out where did the inode problem come from on the root partition. After a bit more investigation, we found that a simple Web app was creating a ton — over half a million — files for the last few years as a way of keeping backup configuration.

Needless to say that after deleting the backup configuration files, we had our inode count back to a very low number.

Cheers.

Category: Disks | Tags: