When pop-before-smtp crashes, don’t pull your hair out just yet…

Published April 23rd, 2009

Today, one of our customer mail server decided to panic; we can understand — it has been a busy week for the server — lots of emails going in and out — it just decided to throw a kennel panic; normally, we would panic when a kernel does it before us. However, this one had been panicking so much that only thing we can think of is retiring this old one-eyed (1U) beast as soon possible. However, the customer is not ready to fork out big bucks to change it any time soon in this wonderful economy.

So we tried to give it the fingers but the remote kvm was not reachable; more panic on our end. However, we finally got the remote RPC to filp the power and got the server back up and running.  Sounds all very typical and simple, right?

Well, mail services (SMTP, POP3, IMAP) all came up fast and only problem we noticed is when users tried to send mail out, they were getting relay access denied message.

Since we use pop-before-smtp for authenticating the IP addresses that connects to the POP3 server (dovecot) as valid SMTP client, we figured restarting the pop-before-smtp, would just do us good on this wonderful Thursday afternoon.

Too bad, we were wrong. The pop-before-smtp script, which is written in Perl, uses BerkeleyDB which is not very good at handling a “CRASH”. It corrupted the existing /etc/postfix/pop-before-smtp.db file and even after deleting it and recreating by flipping the pop-before-smtp service, it turns out the file shows no IP whatsoever when we ran pop-before-smtp --list.

So that is when we starting pulling hair out and decided that we need to do things in baby steps. We did the following to make it all work:

• Stopped the pop-before-smtp service
• Manually removed the /etc/postfix/pop-before-smtp.db file
• Manually created an empty file called /etc/postfix/pop-before-smtp (without the .db ext)
• Manually ran postmap /etc/postfix/pop-before-smtp to create an empty Berkeley DB that postfix would love to read
• Manually ran the file /etc/postfix/pop-before-smtp.db to see if the file is really BerkeleyDB at all
• Manually deleted the /etc/postfix/pop-before-smtp file
• Manually restarted the pop-before-smtp service

Bang!, everything worked. So the lesson here is that, when your server crashes and you have pop-before-smtp running, the BerkeleyDB file might be corrupt and you might want to totally clear the server’s memory by creating empty file, empty BerkeleyDB file and then restarting the service. Just removing the file and restarting pop-before-smtp does not work because we think the BerkeleyDB backend dumps the old data or file bits that corrupts the file again.

So be aware. Better yet, dump pop-before-smtp totally and go with a different solution, which we will discuss in an upcoming post. So stay tuned.

Category: annoyance | Tags: