Working around shell command-line wildcard limitations

Published January 5th, 2009

Problem Statement

The most commonly used shell in CentOS is /bin/bash and like other shells such as /bin/sh, /bin/tcsh, it has command line limitations. One such limitation is the wildcard expansion size. For example, say you have a directory with thousands of files or subdirectories and want to run the rm command to remove only files with .php extension using rm -f *.php, you will run into the wildcard limitation of the shell. Here we will show you a simple work around.

Trying to delete a lot of files in a directory with thousands of files using rm -f *

Say you are trying to delete all PHP files using rm -f *.php in a directory with thousands of PHP cache files generated by the Smarty template engine, in such a case you will see an error message from shell such as the following:

/bin/rm: Argument list too long

What happened is that shell tried to expand the wildcard *.php and placed the file list in rm’s command-line argument list. Since each command-line argument list is really an array and there is a limit to how many arguments can be stored in a process’s memory space, it fails.

Using a workaround for wildcard expansion problem for commands like rm

The easiest solution is to run the find command and execute rm one by one for each matching file. For example, to delete all PHP scripts in a directory, you can run:

$ find . -type f -name "*.php" -exec rm -f {} \;

This tells the find command to find all files matching the given wildcard (*.php) and execute the rm -f command on each of the matching file. Thus, this command can avoid the wildcard expansion issue as the rm -f command sees only one argument at a time.

Category: shell | Tags: