mikas blog

A problem everyone stumbles across (at least) once:

% tar zcf stats_2005.tar.gz stats_2005*
zsh: argument list too long: tar

The reason? Too many files for the ARG_MAX limit. ARG_MAX defines the maximum length of arguments to the exec function (more precise: bytes of args plus environment for exec), defined in /usr/include/linux/limits.h on your Linux system:

% ls -la | grep -c stats_2005
9037
% getconf ARG_MAX
131072
% cpp << EOF | tail -1
#include <limits.h>
ARG_MAX
EOF
131072
%

Ok. But how to work around the issue? Possible solution for GNU:

% find . -name 'stats_2005*' > filelist
% tar zcf stats_2005.tar.gz --files-from filelist

Without the temporary file (filelist in our example):

% find . -name 'stats_2005*' -print | tar zcf stats_2005.tar.gz --files-from -

zsh provides zargs (but AFAIK it doesn’t work with gzip in the same command line due to the way zargs works, so create the archive in one step and compress it in another step later):

% autoload zargs
% zargs -- stats_2005* -- tar --ignore-failed-read -rpf stats_2005.tar

The approach works with other tools than tar as well of course, usage example for afio:

find . -name 'stats_2005*' -print | afio -o -Z stats_2005.afio

Depending on the type of problem (rm, cp, mv,…) you can choose differented approaches as well, some further examples:

% for file in stats_2005* ; do rm -f -- "$file" ; done
% perl -e 'unlink <stats_2005*>'
% zsh -c 'zmodload zsh/files && rm -f -- stats_2005*'
% zsh -c 'autoload -U zargs; zargs -r ./stats_2005* -- rm -f'
% ksh93 -c 'command -x rm -f -- stats_2005*'
% find . -maxdepth 1 -name 'stats_2005*' -print0 | xargs -r0 rm -f
% find . -name 'stats_2005*' -print0 | xargs -0 mv -i --target-dir=/destination
% zmodload zsh/files ; mv -- stats_2005* /destination

This entry was posted by mika on Monday, February 26th, 2007 at 10:38 and is filed under Computer, Debian, English. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.