Published February 23rd, 2010 by admin
Recently, we needed a simple cron script to record load average, apache server count, existence of lighthttpd and mysql status in a log file every 10 minutes. So following shell script was born:
#!/bin/sh
httpd_cnt=`ps auxww | grep '/home/apache/bin/httpd' | wc -l`
lighthttpd_cnt=`ps auxww | grep 'lighthttpd' | wc -l`
mysql_status=`/usr/bin/mysqladmin -u root -pYOUR_DB_PWD status`
load=`/bin/cat /proc/loadavg`
now=`date "+%m-%d-%Y %H:%M:%S"`
today=`date "+%m-%d-%Y_%H00"`
logfile=/var/log/monitor.$today.log
lock=/dev/shm/monitor.$today.lock
if [ -e $lock ]; then
# Already running
exit
fi;
touch $lock;
if [ ! -e $logfile ]; then
echo "TIME, LOAD AVRG, APACHE COUNT, LIGHTHTTPD COUNT, MYSQL STATUS" > $logfile;
fi
echo $now, $load, $httpd_cnt, $lighthttpd_cnt, $mysql_status >> $logfile;
rm -f $lock;
Once this script was created and save into an executable (r-x) text file, it was scheduled in /etc/crontab to run every 10 minutes as follows:
*/10 * * * * root /root/monitor/monitor.sh
Here is a sample log file created by this script in /var/log:
Category: Performance, shell | Tags: | Be the First to Comment »
Published September 27th, 2009 by admin
Nobody likes unscheduled reboots but it happens to all of us. Recently, we had all our servers rebooted because our data center’s ATS — a device that switches to UPS power — failed. About 90% of the servers returned to life as usual but a few had cron jobs that got stuck due to previous lock files that were not removed as the cron jobs never completed. In cases like this we get email from cron jobs that they are aborting due to lock files not letting them run again. So its a bit of pain to go out and clean up these files In addition to the custom cron jobs that we created, we saw Dovecot, Apache, etc. having the same problem but of course they did not send any email. In that case we found out the hard way. So we came up with a nifty solution that will stop lock files from being a pain from now on. Here is how.
Instead of keeping the lock files on physical disks, we decided to migrate all lock files to /dev/shm/locks which is basically a RAM disk. So when the system gets rebooted due to scheduled or unscheduled event, we have no lock files in our way. So no lock file can lock up your cron jobs in the future in such an event.
However, be warned that if you rely on a lock file to be there to indicate a job being incomplete and perhaps one that needs manual help to restore, you might not want to use /dev/shm. In our case, our lock files are simply there to tell us something is running or what the current process ID (Apache, dovecot pid files) etc. So if a server gets rebooted and the cron’s lock is also removed, our crons do the work from where they left of in the first place with minimal duplication since we use a chunking technique for any cron that works with database or email lists.
Hope unscheduled reboots are far and wide for everyone.
Cheers!
Category: Disks | Tags: | Be the First to Comment »
Published September 27th, 2009 by admin
Recently our data center had a power issue and rebooted all our servers. In this process, one of the server had an issue with the swap partition (/dev/sda3). So we decided to recreate it as follows:
# mkfs -t ext3 /dev/sda3
# mkswp /dev/sda3 1024
Category: Disks | Tags: | Be the First to Comment »
Published September 26th, 2009 by admin
Due to a forced hard reboot, thanks to the Data Center’s faulty PDU, one of our servers got the blues. After getting Nagios alert, we logged onto it via remote KVM and noticed that it is hanging at the maintenance shell prompt.
After logging in as root to the maintenance shell, we noticed that the system was complaining about mismatched label for the root partition.
When we configured the system we setup the root partitoin, /dev/sda1, using label / using the disk setup program that starts up as part of the centos setup process. However, somehow the /etc/fstab now shows the root partition as LABEL=/1 instead of LABEL=/ and similarly we had our /var and /usr partition labels got messed up from being /usr and /var to /usr1 and /var1.
Since in maintenance mode, we could not edit /etc/fstab and single user mode was not an option since it also gets stuck at the maintenance prompt, here is how we survived this problem:
Category: Disks | Tags: | Be the First to Comment »
Published September 26th, 2009 by kabir
Problem Statement
We wanted to create SFTP-only user accounts that cannot SSH into the server to run commands. There is no built-in approach to this problem that we can find so we created a simple shell script to solve it. Here we will discuss how it works.
Step 1: Create a shell script to run as the user’s shell
Create a shell script called /sbin/sftp-only as follows:
#!/bin/sh
if [ “$*” != “-c /usr/lib/openssh/sftp-server” ]; then
echo “Sorry, ssh access not allowed.”;
exit;
fi;
exec /usr/lib/openssh/sftp-server
Step 2: Edit user accounts to use this shell script as user’s shell
Modify user accounts using usermod to set the shell to /sbin/sftp-only so that when user tries to SSH to the server, the shell script will display the “Sorry, ssh access not allowed.” message. And when the user tries to connect to the server via a SFTP client, the shell script will get executed and it will start the SFTP server for the user.
Category: SSH, Security | Tags: | 2 Comments »
Published June 26th, 2009 by admin
We are not a big fan of spamassassin, even though a lot of people swear by it. We use an anti-spam appliance hardware to deal with spam and let a third-party — MailFoundry– constantly update the appliance’s rules and tricks to fight spam. However, recently we had a hardware failure and had to return the appliance to get a replacement back. This process took a very long time.
Once the MailFoundry appliance was out of service, we started getting tons of spam and finally had to create a temporary measure to reduce spam without too elaborate system reconfiguration. Here is how we applied a few configuration changes to Postfix and got rid of majority of the spam.
smtpd_helo_required = yes
smtpd_helo_restrictions = permit_sasl_authenticated, \
permit_mynetworks, reject_invalid_hostname, \
check_helo_access hash:/etc/postfix/helo_access, permit
disable_vrfy_command = yes
strict_rfc821_envelopes = yes
smtpd_client_restrictions = permit_sasl_authenticated,permit_mynetworks, \
reject_rhsbl_client mydomain.tld, \
reject_rbl_client bl.spamcop.net, \
reject_rbl_client zen.spamhaus.org, \
reject_rbl_client safe.dnsbl.sorbs.net, \
reject_rbl_client cbl.abuseat.org, \
reject_rbl_client list.dsbl.org
smtpd_sender_restrictions = reject_non_fqdn_sender, reject_unknown_sender_domain
mime_header_checks = regexp:/etc/postfix/mime_header_checks
These settings are pretty straight forward. They basically “discourages” bad behavior on the other end of the SMTP transaction. Since most bad guys use improper headers, or have been already known as bad guys in real-time black lists, these configuration protects us from most spammers.
So you might be wondering if these settings are so good, why get the MailFoundry back? Well, the above-mentioend Postfix configuration is great but they do slow down the SMTP transactions. So we prefer to outsourcee the spam processing on the dedicated appliance. Until we get our MailFoundry appliance back in order, we have to fight spam with the above settings and we are not too unhappy about it.
Category: Email, Tuning | Tags: | 1 Comment »
Published June 8th, 2009 by admin
Today, we had a panic call from a customer who was hosting a small app on a non-dedicated server environment with us. The app uses a ton of images and was hosted on a box with terabytes of disk space. They called with a message saying that the system is out of disk space.
After investigating, we found out that we had plenty of disk space available and yet the error that they reported was showing up on Web app and even when we tried to create new files in the root partition.
Our favorite editor, vi, was showing:
E212: Can't open file for writing
At first, we thought we are in big trouble as the root partition has probably gone into read-only mode due to some disk hardware failure. But, it turned out that even though df -h reports 20% free disk space, we actually ran out of all the inodes in the root partition. To find what percent of inodes were available, we ran df -i and saw a 100% utilization for the root partition.
There is no way to add new inodes on a running filesystem without recreating the filesystem itself. So we had to figure out where did the inode problem come from on the root partition. After a bit more investigation, we found that a simple Web app was creating a ton — over half a million — files for the last few years as a way of keeping backup configuration.
Needless to say that after deleting the backup configuration files, we had our inode count back to a very low number.
Cheers.
Category: Disks | Tags: | Be the First to Comment »
Published May 24th, 2009 by admin
This post will be published when I am just about to get on a flight to Singapore on my way back home in Sacramento, CA from Dhaka. My three-week Dhaka trip ends today. For most of these three weeks, I spent working in our dev center in various “upgrades” but they are mostly business-ish and not really worth mentioning in this blog. However, I do wish to share one such upgrade with you — users in our Dhaka office always complain about bandwidth and this complain is amplified in the AM as everyone comes in and starts downloading email messages with attachments. Even with a 1.5Mbps (T-1) connection, the load just cannot keep up. So finally, I gave in and decided to localize email using a simple relay. Here is how:
Our primary corporate mail server is located behind a mail filter appliance from Mail Foundry in our Sacramento office. The users from Sacramento, New York, and Dhaka all connect to the mail server via POP3/IMAP and Web mail to access their daily dose of emails. However, the Dhaka staff always have a hard time as the mail download is horrible when large attachments are included by customers or spammers that get thru.
So the idea behind the mail relay was to accept mail using user@evoknow.com but send them over to user@<dhaka specific host> running SMTP mail server. The local Dhaka users then simply point their mail clients to the local mail server which uses IMAP/POP3 to service them. Since most mails are sent from the US, they are already downloaded in the local mail server, which is extremely fast for the users on the same LAN.
This trick involves the following on the US side:
- Edit /etc/postfix/virtual to add user@evoknow to user@dhaka.specific.host mappings
- Recreate the virtual alias database using postmap hash:/etc/postfix/virtual
- That’s all for this end.
On the Dhaka end, here is what was needed:
- Setup a CentOS 5.3 box as a mail server with Postfix for SMTP, DoveCot for POP3/IMAP
- Made sure that the IP address has a reverse DNS setup — i.e. PTR record from the ISP in Dhaka
- Since I compiled dovecot from source, I had to setup the /usr/local/etc/dovecot.conf with mail_location directive that looks as follows:
mail_location = mbox:~/mail:INBOX=/var/mail/%u
- Restart Dovecot.
To test the setup, I sent mail to user@evoknow.com which got relayed to user@<dhaka specific host> and when the user came in next morning, she picked up the email from the local mail server at high speed!
I guess, mail is now flying high as I am. :)
Cheers.
Category: Email | Tags: | Be the First to Comment »
Published May 24th, 2009 by kabir
I often forget all the parameters to mount a DVD on a running system, so here is my blog-reminder:
# mount -t iso9660 -o loop /path/to/dvd.iso /mount/point
Example:
mount -t iso9660 -o loop CentOS-5.3-i386-bin-DVD.iso /mnt/dvd
This will mount the CentOS 5.3 DVD iso on /mnt/dvd directory.
Category: Uncategorized | Tags: | Be the First to Comment »
Published May 20th, 2009 by kabir
We use BackupPC and have a love-hate relationship with it. We love that it works most of the time, and hate that it uses CGI Web interface and also outputs cryptic error messages when backup fails. But never the less, we have not found a better deal so far. Hence we continue to use it.
Here is our installation quick-guide:
Category: Uncategorized | Tags: | 1 Comment »
