Shell Programming and Scripting

I met a problem on HPUX with 64G RAM and 20 CPU.

There are 5 million files with file name from file0000001.dat to file9999999.dat, in the same directory, and with some other files with random names.

I was trying to remove all the files from file0000001.dat to file9999999.dat at the same time.

If use 'rm file???????.dat', trying to remove them at once, then there was an error output.

If instead use 'rm file1??????.dat' , trying to remove 10% of them at a time, then the shell did not response in several hours and had to kill the process.

Does any expert of file system can help? Or is it possible to do that by any chance?

Thanks a lot!

What version of HP-UX?
What type of filesystem on what physical disc arrangement?
Is the filesystem mirrored?
Is NFS or anything slow involved?
What is the approximate total size of the files to be deleted?
What is the size of the directory file and how many inodes?

Code:

ls -lad /directory_name
df -i /directory_name

What was the "error output" mentioned above?

Are there any subdirectories under the directory containing these files?
i.e. Does this "find" command find all the files without unwanted hits and without finding files we don't want and without pointless searching?

Code:

find /directory_tree -type f -name file\?\?\?\?\?\?\?\.dat -print

This might help:
https://sites.google.com/site/tfsidc...umber-of-files

I suspect you're running into a shell limitation.

Let it run. Deleting that many files will take a very long time. It probably did delete some of them. Do "ls file1??????.dat | wc -l" to count how many are left.

give you some outputs, when deleting, so you can monitor the progress.

Code:

ls file???????.dat |while read file
do
  echo "Deleting file $file"
  rm $file
done

That is adding half a million fork() calls to a procedure that is already painfully long. I must advise against that. Once an hour or so he can count the remaining files using the command I gave. (Put the delete in the baxkground or use a second window.)

That'd probably die with 'too many arguments', too, just like the OP did. The answer to 'too many arguments' is not to cram the too many arguments into ls instead, the answer is to not use that many arguments because you cannot cram an unlimited number in there. On some OSes the limit is surprisingly small.