I have written a script in awk and the name of the script is "action.txt",
I try to run this with the command "awk -f action.txt abc/*",
where "abc" is a directory and I am running "action.txt" against all the files 
in the "abc" directory,
when I run this command, I get this error
"bash: /bin/awk: Argument list too long"..
There are approximately 3500 files in "abc" and the names of the files are
pretty big too.
My question is, how do I get this script to run against so many files?
PS: the script runs fine if only a few number of files are present in "abc"

Hi harshag-ga,

The shell has a limit to the size of command-line that it can process
at one time. You can overcome this limit by using the xargs command,
which will run your command once for each file.

First you need to build a list of files that you are processing, for example:

   find abc -type f

Next, tell 'xargs' to use the 'awk' command with that list:

   find abc -type f | xargs /usr/bin/awk -f action.txt

The manual page for 'xargs' stresses that if you file names might
contain spaces or newlines, you must use a different version of this
command:

   find abc -type f -print0 | xargs -0 /usr/bin/awk -f action.txt

On the line above, both occurrences of "0" are zeros. Note also that
you must use the full pathname of your 'awk' command. If you don't
know this already, you can find it by typing:

   which awk

I'm assuming that you are using awk as a filter. If you are doing
anything more complex with awk, in a way that would make this solution
inapplicable, please let me know and I'll try to find an alternative
solution.

Similarly, if anything doesn't work for you, please request clarification.

Regards,
eiffel-ga


Google Search Strategy:

arguments call "for each" command pipe grep
://www.google.com/search?q=arguments+call+%22for+each%22+command+pipe+grep


Additional Links:

Linux Man Page for FIND
http://linux.ctyme.com/man/man0763.htm

Linux Man Page for XARGS
http://linux.ctyme.com/man/man3870.htm

---

Request for Answer Clarification by harshag-ga on 13 Jul 2006 07:07 PDT

Thanks and sorry for the late reply eiffel-ga.
your answer worked for me but i was facing another problem for which i
have posted another question.

Thanks again,

-- harshag-ga

---

Clarification of Answer by eiffel-ga on 13 Jul 2006 09:41 PDT

Thanks, harshag-ga, for your comments and kind tip.

Unfortunately, I don't have an answer for your other question,
although I have posted a few suggestions over there.

Regards,
eiffel-ga

Very quick and helpful

I'll comment on this by saying it's not the shell but the kernel that
has the limit on argument size (in the execve() call).  E2BIG is the
name of this error as you'll see in a header file such as errno.h .