Google Answers Logo
View Question
 
Q: "Argument list too long" error when running an AWK script ( Answered 5 out of 5 stars,   1 Comment )
Question  
Subject: "Argument list too long" error when running an AWK script
Category: Computers > Operating Systems
Asked by: harshag-ga
List Price: $5.00
Posted: 10 Jul 2006 07:44 PDT
Expires: 09 Aug 2006 07:44 PDT
Question ID: 744908
I have written a script in awk and the name of the script is "action.txt",
I try to run this with the command "awk -f action.txt abc/*",
where "abc" is a directory and I am running "action.txt" against all the files 
in the "abc" directory,
when I run this command, I get this error
"bash: /bin/awk: Argument list too long"..
There are approximately 3500 files in "abc" and the names of the files are
pretty big too.
My question is, how do I get this script to run against so many files?
PS: the script runs fine if only a few number of files are present in "abc"
Answer  
Subject: Re: "Argument list too long" error when running an AWK script
Answered By: eiffel-ga on 10 Jul 2006 08:59 PDT
Rated:5 out of 5 stars
 
Hi harshag-ga,

The shell has a limit to the size of command-line that it can process
at one time. You can overcome this limit by using the xargs command,
which will run your command once for each file.

First you need to build a list of files that you are processing, for example:

   find abc -type f

Next, tell 'xargs' to use the 'awk' command with that list:

   find abc -type f | xargs /usr/bin/awk -f action.txt

The manual page for 'xargs' stresses that if you file names might
contain spaces or newlines, you must use a different version of this
command:

   find abc -type f -print0 | xargs -0 /usr/bin/awk -f action.txt

On the line above, both occurrences of "0" are zeros. Note also that
you must use the full pathname of your 'awk' command. If you don't
know this already, you can find it by typing:

   which awk

I'm assuming that you are using awk as a filter. If you are doing
anything more complex with awk, in a way that would make this solution
inapplicable, please let me know and I'll try to find an alternative
solution.

Similarly, if anything doesn't work for you, please request clarification.

Regards,
eiffel-ga


Google Search Strategy:

arguments call "for each" command pipe grep
://www.google.com/search?q=arguments+call+%22for+each%22+command+pipe+grep


Additional Links:

Linux Man Page for FIND
http://linux.ctyme.com/man/man0763.htm

Linux Man Page for XARGS
http://linux.ctyme.com/man/man3870.htm

Request for Answer Clarification by harshag-ga on 13 Jul 2006 07:07 PDT
Thanks and sorry for the late reply eiffel-ga.
your answer worked for me but i was facing another problem for which i
have posted another question.

Thanks again,

-- harshag-ga

Clarification of Answer by eiffel-ga on 13 Jul 2006 09:41 PDT
Thanks, harshag-ga, for your comments and kind tip.

Unfortunately, I don't have an answer for your other question,
although I have posted a few suggestions over there.

Regards,
eiffel-ga
harshag-ga rated this answer:5 out of 5 stars and gave an additional tip of: $2.00
Very quick and helpful

Comments  
Subject: Re: "Argument list too long" error when running an AWK script
From: bozo99-ga on 10 Jul 2006 11:30 PDT
 
I'll comment on this by saying it's not the shell but the kernel that
has the limit on argument size (in the execve() call).  E2BIG is the
name of this error as you'll see in a header file such as errno.h .

Important Disclaimer: Answers and comments provided on Google Answers are general information, and are not intended to substitute for informed professional medical, psychiatric, psychological, tax, legal, investment, accounting, or other professional advice. Google does not endorse, and expressly disclaims liability for any product, manufacturer, distributor, service or service provider mentioned or any opinion expressed in answers or comments. Please read carefully the Google Answers Terms of Service.

If you feel that you have found inappropriate content, please let us know by emailing us at answers-support@google.com with the question ID listed above. Thank you.
Search Google Answers for
Google Answers  


Google Home - Answers FAQ - Terms of Service - Privacy Policy