set x ${dir?}/[*] ${dir?}/*

is a usefull construct to prepare an iteration over all files in a
directory ${dir}, taking into account the cases where ${dir} is empty
or contains a file with name *. I suspect that one can also use --
instead of the x, but I am not sure.

The following steps for the iteration are:

shift &&
case "${1?} ${2?}" in
${dir?}/\[\*\]\ ${dir?}/\*) return 0 ;;
*) : ;;
esac &&
shift &&
while test "$#" -ge 1; do
{
# ... do something with $1 ...
shift
} || return "$?"
done

This does not use any variables, apart from ${dir} and the positional
parameters. This is usefull in order to put this construct into a
general-purpose shell function, for shells that do not support local
variables.

However, I am not yet convinced 100% that this is secure in case the
contents of ${dir} is under the control of an attacker (and this shell
construct runs with some higher privileges). The x in the call to set
should prevent any surprises triggered by funny filenames, I guess.
Maybe someone likes to comment on this? Have I missed something?

Regards, Lasse

Lasse Kliemann <stu33404@mail.uni-kiel.de> writes:

> set x ${dir?}/[*] ${dir?}/*
>
> is a usefull construct to prepare an iteration over all files in a
> directory ${dir}, taking into account the cases where ${dir} is empty
> or contains a file with name *. I suspect that one can also use --
> instead of the x, but I am not sure.
>
> The following steps for the iteration are:
>
> shift &&
> case "${1?} ${2?}" in
> ${dir?}/\[\*\]\ ${dir?}/\*) return 0 ;;
> *) : ;;
> esac &&
> shift &&
> while test "$#" -ge 1; do
> {
> # ... do something with $1 ...
> shift
> } || return "$?"
> done

???

#v+
for FILE in "$dir/"*; do
[ -e "$FILE" ] || continue
# ... do whatever you want with $FILE ...
done
#v-

or:

#v+
set -- "$dir/"*
while [ $# -gt 0 ]; do
[ -e "$1" ] || continue
# ... do whatever you want with $1 ...
done
#v-

--
Best regards, _ _
.o. | Liege of Serenly Enlightened Majesty of o' \,=./ `o
..o | Computer Science, Michal "mina86" Nazarewicz (o o)
ooo +--<mina86*tlen.pl>---<jid:mina86*chrome.pl>--ooO--(_)--Ooo--

Michal Nazarewicz <mina86@tlen.pl> wrote:
> Lasse Kliemann <stu33404@mail.uni-kiel.de> writes:
>
>> set x ${dir?}/[*] ${dir?}/*

${dir} should be quoted, I just notice.

[...]
>> shift &&
>> case "${1?} ${2?}" in
>> ${dir?}/\[\*\]\ ${dir?}/\*) return 0 ;;

Maybe this ${dir} should also be quoted; not sure right now.

>> *) : ;;
>> esac &&
>> shift &&
>> while test "$#" -ge 1; do
>> {
>> # ... do something with $1 ...
>> shift
>> } || return "$?"
>> done


> #v+
> set -- "$dir/"*
> while [ $# -gt 0 ]; do
> [ -e "$1" ] || continue
> # ... do whatever you want with $1 ...

shift

> done
> #v-


I see two drawbacks:

1. ``test -e'' is not available on some systems, e.g., Solaris /bin/sh.

2. With a lot of files, the ``test -e'' in each iteration might slow
things down.

Otherwise it should be the same.


Best regards, Lasse

2006-12-3, 20:30(+01), Lasse Kliemann:
> set x ${dir?}/[*] ${dir?}/*

set x "$dir"/[*] "$dir"/*

otherwise $dir is split and subject to filename generation.

> is a usefull construct to prepare an iteration over all files in a
> directory ${dir}, taking into account the cases where ${dir} is empty
> or contains a file with name *. I suspect that one can also use --
> instead of the x, but I am not sure.

Yes you could, problems with set -- may only arise if the list
after can be empty, which is not the case here.

>
> The following steps for the iteration are:
>
> shift &&
> case "${1?} ${2?}" in
> ${dir?}/\[\*\]\ ${dir?}/\*) return 0 ;;

"$dir/[*] $dir/*")

otherwise $dir is taken as a pattern.

> *) : ;;
> esac &&
> shift &&
> while test "$#" -ge 1; do
> {
> # ... do something with $1 ...
> shift
> } || return "$?"
> done

for i do
something with "$i"
done

>
> This does not use any variables, apart from ${dir} and the positional
> parameters. This is usefull in order to put this construct into a
> general-purpose shell function, for shells that do not support local
> variables.
>
> However, I am not yet convinced 100% that this is secure in case the
> contents of ${dir} is under the control of an attacker (and this shell
> construct runs with some higher privileges). The x in the call to set
> should prevent any surprises triggered by funny filenames, I guess.
> Maybe someone likes to comment on this? Have I missed something?
[...]

If that directory is potentially under control of an attacker,
you may want to first cd to that directory:

cd -P -- "$dir"

Then check that the current directory is indeed the one you
think it is (in case $dir was a symlink)

if [ "$PWD" = ... ]

Then it depends what you want to do with that list. There's no
guarantee that the list you built with set x... will reflect the
content of the directory at the time you process the list, you
may want to change the permission so that only root can modify
its content before doing anything.

What are you trying to achieve?

--
Stéphane

2006-12-03, 21:36(+01), Michal Nazarewicz:
> Lasse Kliemann <stu33404@mail.uni-kiel.de> writes:
>
>> set x ${dir?}/[*] ${dir?}/*
>>
>> is a usefull construct to prepare an iteration over all files in a
>> directory ${dir}, taking into account the cases where ${dir} is empty
>> or contains a file with name *. I suspect that one can also use --
>> instead of the x, but I am not sure.
>>
>> The following steps for the iteration are:
>>
>> shift &&
>> case "${1?} ${2?}" in
>> ${dir?}/\[\*\]\ ${dir?}/\*) return 0 ;;
>> *) : ;;
>> esac &&
>> shift &&
>> while test "$#" -ge 1; do
>> {
>> # ... do something with $1 ...
>> shift
>> } || return "$?"
>> done
>
> ???
>
> #v+
> for FILE in "$dir/"*; do
> [ -e "$FILE" ] || continue

You may want do use

[ -e "$FILE" ] || [ -L "$FILE" ] || continue

instead.

Note that OPs solution has the advantage to work even if you
don't have execute permission to the directory (but then, you
will not be able to do much with the files in that dir).

> # ... do whatever you want with $FILE ...
> done
> #v-
>
> or:
>
> #v+
> set -- "$dir/"*
> while [ $# -gt 0 ]; do
> [ -e "$1" ] || continue
> # ... do whatever you want with $1 ...
shift
> done
> #v-
>


--
Stéphane

2006-12-3, 21:47(+01), Lasse Kliemann:
[...]
> I see two drawbacks:
>
> 1. ``test -e'' is not available on some systems, e.g., Solaris /bin/sh.

You've got:

ls -d -- "$file" > /dev/null 2>&1

for an equivalent of

[ -e "$file" ] || [ -L "$file" ]

Solaris's sh is in /usr/xpg4/bin. The one in /bin is there only
for backward compatibility, one shouldn't use it.

> 2. With a lot of files, the ``test -e'' in each iteration might slow
> things down.
[...]

If the purpose is to run some commands for each file in the
directory, it may reveal simpler to use find:

cd -P -- "$dir" &&
find . ! -name . -prune -exec cmd {} \;


--
Stéphane

Stephane CHAZELAS <this.address@is.invalid> wrote:
> 2006-12-3, 21:47(+01), Lasse Kliemann:
> [...]
>> I see two drawbacks:
>>
>> 1. ``test -e'' is not available on some systems, e.g., Solaris /bin/sh.
>
> You've got:
>
> ls -d -- "$file" > /dev/null 2>&1
>
> for an equivalent of
>
> [ -e "$file" ] || [ -L "$file" ]
>
> Solaris's sh is in /usr/xpg4/bin. The one in /bin is there only
> for backward compatibility, one shouldn't use it.

Didn't know that. What would be the most elegant way to autodetect
whether /usr/xpg4/bin/sh is available and use it instead of /bin/sh?
Best I came up is the following, which uses a variable:

#!/bin/sh
case "$usr_xpg4_bin_sh" in
y) :;;
*)
if test -x /usr/xpg4/bin/sh; then
usr_xpg4_bin_sh=y &&
export usr_xpg4_bin_sh &&
exec /usr/xpg4/bin/sh "$0" ${1+"$@"}
else :; fi
;;
esac

>> 2. With a lot of files, the ``test -e'' in each iteration might slow
>> things down.
> [...]
>
> If the purpose is to run some commands for each file in the
> directory, it may reveal simpler to use find:
>
> cd -P -- "$dir" &&
> find . ! -name . -prune -exec cmd {} \;

I would like to use find, especially because I actually have to iterate
over sub-directories as well. I do this with a recursive shell function
now. That's why I insist on not using any variables except the
positional parameters.

However, find becomes unhandy when the command (cmd above) is a shell
function, which it is in many of my applications. I think there is a
way to do it whith sh -c "...", but I am unsure whether this is more
elegant.


Regards, Lasse

Stephane CHAZELAS <this.address@is.invalid> wrote:
> 2006-12-3, 20:30(+01), Lasse Kliemann:
>> set x ${dir?}/[*] ${dir?}/*
>
> set x "$dir"/[*] "$dir"/*

[...]
>> However, I am not yet convinced 100% that this is secure in case the
>> contents of ${dir} is under the control of an attacker (and this shell
>> construct runs with some higher privileges). The x in the call to set
>> should prevent any surprises triggered by funny filenames, I guess.
>> Maybe someone likes to comment on this? Have I missed something?
> [...]
>
> If that directory is potentially under control of an attacker,
> you may want to first cd to that directory:
>
> cd -P -- "$dir"

OMG, I just realize something. This ``--'' is there because "$dir"
might start with a dash and then be interpreted as a switch? I never
paid attention to this. It shouldn't matter for absolute pathnames. And
for relative ones, one could probably prepend ``./''. This would
relieve me of the duty to check for every command whether it
understands the doube-dash or not. As I just saw, e.g., test does *not*
like the double-dash. It looks to me that this is for a reason, because
test seems to expect exactly one switch. But to be on the secure side
in all cases, maybe I will start using ``./'' for all relative
pathnames.

> Then check that the current directory is indeed the one you
> think it is (in case $dir was a symlink)
>
> if [ "$PWD" = ... ]

Good point. Is ``test -d'' guaranteed to rule out symlinks? Then one
could use this (in cases where no race conditions can occur).

> Then it depends what you want to do with that list. There's no
> guarantee that the list you built with set x... will reflect the
> content of the directory at the time you process the list, you
> may want to change the permission so that only root can modify
> its content before doing anything.
>
> What are you trying to achieve?

The files in the directory and the directory itself will not be
writable by anyone else than the account under which the iteration
takes place. They are first written by a (probably malicious) process,
then all processes belonging to this account are killed (and it is
checked whether they are *really* dead using pgrep; as far as I saw in
the source of pkill, such a test ist *not* done by default) and then
they are chowned to that account under which the iteration will take
place.

regards,
Lasse

Lasse Kliemann <stu33404@mail.uni-kiel.de> writes:

> Michal Nazarewicz <mina86@tlen.pl> wrote:
>> #v+
>> set -- "$dir/"*
>> while [ $# -gt 0 ]; do
>> [ -e "$1" ] || continue
>> # ... do whatever you want with $1 ...
>
> shift

Ah! true...

>> done
>> #v-
>
> I see two drawbacks:
>
> 1. ``test -e'' is not available on some systems, e.g., Solaris /bin/sh.
>
> 2. With a lot of files, the ``test -e'' in each iteration might slow
> things down.

Since test -e is for checking '*' only (if there were no files) one
may do:

#v+
case "$1" in (*"*") [ -e "$1" ] || continue; esac
#v-

or something.

Anyways, OP said (in parallel reply) that he want to go into the
directories as well, so he'd have to test for files anyway, eg.:

#v-
foo () {
set -- "$1"/*
while [ $# -gt 1 ]; do
if [ -d "$1" ]; then
foo "$1"
elif [ -f "$1" ]; then
# ... do something funny with a file ...
fi
shift
done
}
#v+

Depending on the task the loop may vary.


--
Best regards, _ _
.o. | Liege of Serenly Enlightened Majesty of o' \,=./ `o
..o | Computer Science, Michal "mina86" Nazarewicz (o o)
ooo +--<mina86*tlen.pl>---<jid:mina86*chrome.pl>--ooO--(_)--Ooo--

2006-12-4, 13:45(+01), Lasse Kliemann:
> Stephane CHAZELAS <this.address@is.invalid> wrote:
>> 2006-12-3, 21:47(+01), Lasse Kliemann:
>> [...]
>>> I see two drawbacks:
>>>
>>> 1. ``test -e'' is not available on some systems, e.g., Solaris /bin/sh.
>>
>> You've got:
>>
>> ls -d -- "$file" > /dev/null 2>&1
>>
>> for an equivalent of
>>
>> [ -e "$file" ] || [ -L "$file" ]
>>
>> Solaris's sh is in /usr/xpg4/bin. The one in /bin is there only
>> for backward compatibility, one shouldn't use it.
>
> Didn't know that. What would be the most elegant way to autodetect
> whether /usr/xpg4/bin/sh is available and use it instead of /bin/sh?

According to POSIX, you should not use a #! line whose behavior
is unspecified. Instead, a POSIX conformant system, when you're
in a POSIX environment should call a conformant shell to
interpret your script. The problem with Solaris is that you're
not in a POSIX environment by default.

Generally, as I write scripts that need only to be portable to
Solaris and other systems that have a Unix sh as /bin/sh
(instead of a Bourne shell), I do:

#! /bin/sh -
: ^ false || exec /usr/xpg4/bin/sh - "$0" ${1+"$@"}

": ^ false" is a test for the Bourne shell (where "^" is an
alias for "|") and Solaris is the only system I've been given to
use where /bin/sh is a Bourne shell (I know there's Tru64 and
very old systems, but I've never used those).

> Best I came up is the following, which uses a variable:
>
> #!/bin/sh

#!/bin/sh -

> case "$usr_xpg4_bin_sh" in
> y) :;;
> *)
> if test -x /usr/xpg4/bin/sh; then
> usr_xpg4_bin_sh=y &&
> export usr_xpg4_bin_sh &&
> exec /usr/xpg4/bin/sh "$0" ${1+"$@"}
> else :; fi
> ;;
> esac

That would work I think. You may want to unset $usr_xpg4_bin_sh
afterwards in case your script calls itself or another script
that uses the same trick.

--
Stéphane

2006-12-4, 14:00(+01), Lasse Kliemann:
> Stephane CHAZELAS <this.address@is.invalid> wrote:
>> 2006-12-3, 20:30(+01), Lasse Kliemann:
>>> set x ${dir?}/[*] ${dir?}/*
>>
>> set x "$dir"/[*] "$dir"/*
>
> [...]
>>> However, I am not yet convinced 100% that this is secure in case the
>>> contents of ${dir} is under the control of an attacker (and this shell
>>> construct runs with some higher privileges). The x in the call to set
>>> should prevent any surprises triggered by funny filenames, I guess.
>>> Maybe someone likes to comment on this? Have I missed something?
>> [...]
>>
>> If that directory is potentially under control of an attacker,
>> you may want to first cd to that directory:
>>
>> cd -P -- "$dir"
>
> OMG, I just realize something. This ``--'' is there because "$dir"
> might start with a dash and then be interpreted as a switch? I never
> paid attention to this. It shouldn't matter for absolute pathnames. And
> for relative ones, one could probably prepend ``./''. This would
> relieve me of the duty to check for every command whether it
> understands the doube-dash or not. As I just saw, e.g., test does *not*
> like the double-dash. It looks to me that this is for a reason, because
> test seems to expect exactly one switch. But to be on the secure side
> in all cases, maybe I will start using ``./'' for all relative
> pathnames.

Most standard Unix utilities should recognise that. Only echo
and the utilities that don't follow the cmd -opts args such as
find, "[" or test may not.

>> Then check that the current directory is indeed the one you
>> think it is (in case $dir was a symlink)
>>
>> if [ "$PWD" = ... ]
>
> Good point. Is ``test -d'' guaranteed to rule out symlinks? Then one
> could use this (in cases where no race conditions can occur).

It's guaranteed not to. test -d symlink tests whether the file
pointed to by symlink exists and is a directory.

test -d "$dir" && test ! -h "$dir"

for directory and not symlink.

But dir could be /tmp/foo/passwd and "foo" be a symlink to /etc.

>> Then it depends what you want to do with that list. There's no
>> guarantee that the list you built with set x... will reflect the
>> content of the directory at the time you process the list, you
>> may want to change the permission so that only root can modify
>> its content before doing anything.
>>
>> What are you trying to achieve?
>
> The files in the directory and the directory itself will not be
> writable by anyone else than the account under which the iteration
> takes place. They are first written by a (probably malicious) process,
> then all processes belonging to this account are killed (and it is
> checked whether they are *really* dead using pgrep; as far as I saw in
> the source of pkill, such a test ist *not* done by default) and then
> they are chowned to that account under which the iteration will take
> place.
[...]

What prevents your user from starting new processes in between
those operations (via cron/at/.forward/telnet/ssh...)

--
Stéphane

Stephane CHAZELAS wrote:

> and Solaris is the only system I've been given to
> use where /bin/sh is a Bourne shell (I know there's Tru64 and
> very old systems, but I've never used those).

PS: other current systems are: UnixWare/OpenUnix + OpenServer.

It always boils down to the same result:
If you don't know the systems in advance, you might need traditional
portability - the best example: autoconf scripts need to be prepared
for this. Inbetween the two extremes, all shades are possible.

So the abovementioned "Solaris-Fix" only can be yet another special case.
--

Stephane CHAZELAS <this.address@is.invalid> wrote, on Mon, 04 Dec 2006:

>> OMG, I just realize something. This ``--'' is there because "$dir"
>> might start with a dash and then be interpreted as a switch? I never
>> paid attention to this. It shouldn't matter for absolute pathnames. And
>> for relative ones, one could probably prepend ``./''. This would
>> relieve me of the duty to check for every command whether it
>> understands the doube-dash or not. As I just saw, e.g., test does *not*
>> like the double-dash. It looks to me that this is for a reason, because
>> test seems to expect exactly one switch. But to be on the secure side
>> in all cases, maybe I will start using ``./'' for all relative
>> pathnames.
>
> Most standard Unix utilities should recognise that. Only echo
> and the utilities that don't follow the cmd -opts args such as
> find, "[" or test may not.

These days find has -H and -L options, so it does accept "--".
However, it isn't really much use because a filename beginning with
a dash is then interpreted as a (probably bad) primary instead of a
pathname.

E.g. on Solaris 10:

$ find -foo -print
find: illegal option -- f
find: [-H | -L] path-list predicate-list
$ find -- -foo -print
find: [-H | -L] path-list predicate-list

So to be safe you need to prefix "./" to relative pathnames passed
to find.

In general, prefixing "./" to relative pathnames is better than using
the "--" method because it also protects against the filename "-",
which some utilities take to mean stdin/stdout instead of a file
called "-". However, it is more work (because you have to check
whether the pathname begins with "/"), which is why the "--" method
is seen much more often.

--
Geoff Clare <netnews@gclare.org.uk>

> 2006-12-4, 13:45(+01), Lasse Kliemann:
>> #!/bin/sh

Stephane CHAZELAS <this.address@is.invalid> writes:
> #!/bin/sh -

What's the point? I mean what does the minus sign give you?

--
Best regards, _ _
.o. | Liege of Serenly Enlightened Majesty of o' \,=./ `o
..o | Computer Science, Michal "mina86" Nazarewicz (o o)
ooo +--<mina86*tlen.pl>---<jid:mina86*chrome.pl>--ooO--(_)--Ooo--

2006-12-06, 23:07(+01), Michal Nazarewicz:
>> 2006-12-4, 13:45(+01), Lasse Kliemann:
>>> #!/bin/sh
>
> Stephane CHAZELAS <this.address@is.invalid> writes:
>> #!/bin/sh -
>
> What's the point? I mean what does the minus sign give you?

When you don't know what will be the next argument of a command
such as in

cmd "$var"

or above as the next argument will be the path of the script on
which you (the writer of the script) have no control one, you
should mark the end of options (with "-" or "--" depending on
the cmd) so that that argument is not taken as an option ifever
it starts with a "-" (or "+" in some cases).

cmd -- "$var"

#! /bin/sh -

--
Stéphane