Stop Lock Files from Locking Your Cron Jobs Upon Unscheduled Reboot

Published September 27th, 2009

Nobody likes unscheduled reboots but it happens to all of us. Recently, we had all our servers rebooted because our data center’s ATS — a device that switches to UPS power — failed. About 90% of the servers returned to life as usual but a few had cron jobs that got stuck due to previous lock files that were not removed as the cron jobs never completed. In cases like this we get email from cron jobs that they are aborting due to lock files not letting them run again. So its a bit of pain to go out and clean up these files In addition to the custom cron jobs that we created, we saw Dovecot, Apache, etc. having the same problem but of course they did not send any email. In that case we found out the hard way. So we came up with a nifty solution that will stop lock files from being a pain from now on. Here is how.

Instead of keeping the lock files on physical disks, we decided to migrate all lock files to /dev/shm/locks which is basically a RAM disk. So when the system gets rebooted due to scheduled or unscheduled event, we have no lock files in our way. So no lock file can lock up your cron jobs in the future in such an event.

However, be warned that if you rely on a lock file to be there to indicate a job being incomplete and perhaps one that needs manual help to restore, you might not want to use /dev/shm. In our case, our lock files are simply there to tell us something is running or what the current process ID (Apache, dovecot pid files) etc. So if a server gets rebooted and the cron’s lock is also removed, our crons do the work from where they left of in the first place with minimal duplication since we use a chunking technique for any cron that works with database or email lists.

Hope unscheduled reboots are far and wide for everyone.

Cheers!

Category: Disks | Tags: