Hi -

I know you can use globs in shell scripts, but how do I control when the
globbing is done?

For instance

xpc:~:$ ls foo*
foo1 foo2 foo3
xpc:~:$ a=foo*
xpc:~:$ echo $a
foo1 foo2 foo3
xpc:~:$ echo "$a"
foo*

How do I get variable a to contain the expanded list of files, rather than
the glob itself?

One workaround I've found is to use ls to do it, but this seems very ugly
a=`ls foo*`

Presumably I could use a subshell to expand the variable, but I can't get
the syntax right.

Apologies if this is an FAQ, but I have ages looking through the Advanced
Bash Scripting Guide.

Thanks

Jeremy

2008-05-02, 10:30(+01), Jeremy Sanders:
> Hi -
>
> I know you can use globs in shell scripts, but how do I control when the
> globbing is done?
>
> For instance
>
> xpc:~:$ ls foo*
> foo1 foo2 foo3
> xpc:~:$ a=foo*
> xpc:~:$ echo $a
> foo1 foo2 foo3
> xpc:~:$ echo "$a"
> foo*
>
> How do I get variable a to contain the expanded list of files, rather than
> the glob itself?

Variables by default are of scalar type, they can contain only
one value.

For more than one value, you need arrays. But then note that you
cease to be portable.

a=(foo*)

The list of elements is accessed with "${a[@]}" (the double
quotes are important).

POSIXly, you can do

set -- foo*

Note that there's a misdesign in most Bourne-like shells in that
if "foo*" doesn't match any file, it expands to "foo*" literally
instead of no argument at all or an error.

With zsh, you get an error if there's no match, and if you want
the globbing pattern to expand to nothing if it doesn't match
anything, you simply need to add the (N) globbing qualifier:

foo_file=(foo*(N))

print "there are $#foo_file files: "
(($#foo_file)) && print -rl $foo_file

($foo_file instead of "${foo_file[@]}" works only in zsh, as for
zsh arrays are a different type than scalar, though in bash and
ksh, the $foo scalar is actually a short for ${foo[0]})

> One workaround I've found is to use ls to do it, but this seems very ugly
> a=`ls foo*`
[...]

This assignes the concatenation of the filenames with the
newline characters in between. To access the elements of the
list, you have to split that string back into arguments. That's
possible but that means that it won't work if some of the
filenames contain newline characters.

To split it:

a=`ls -d foo*` # or a=`printf '%s\n' foo*` if you know there are
# foo files
IFS='
' # set the field separator to the newline character
set -f # disable filename generation for the expansion of $a

set -- $a

The elements of $a are stored in $1, $2...

--
Stéphane

On 2 Mai, 11:30, Jeremy Sanders <jer...@jeremysanders.net> wrote:
> Hi -
>
> I know you can use globs in shell scripts, but how do I control when the
> globbing is done?
>
> For instance
>
> xpc:~:$ ls foo*
> foo1 foo2 foo3

The shell expands it before ls is called.

> xpc:~:$ a=foo*

Not yet expanded in case of the assignment.

> xpc:~:$ echo $a

First the variable $a (which still contains 'foo*') is expanded,
then the filename expansion (the globbing) is done.

> foo1 foo2 foo3
> xpc:~:$ echo "$a"

No filename expansion is done in (single or) double quotes.

> foo*
>
> How do I get variable a to contain the expanded list of files, rather than
> the glob itself?

Mind that you may get inconsistencies in your further processing
if the filenames may contain spaces, for example.

> One workaround I've found is to use ls to do it, but this seems very ugly
> a=`ls foo*`

You can use the echo builtin if that seems less ugly to you

a=$(echo foo*)

Shells (don't know whether that is true for bash, though) can
avoid a subshell if builtins are called inside $(...).

> Presumably I could use a subshell to expand the variable, but I can't get
> the syntax right.

You've already used a valid subshell expansion above. What
problem did you see?

Janis

> Apologies if this is an FAQ, but I have ages looking through the Advanced
> Bash Scripting Guide.
>
> Thanks
>
> Jeremy

On 2 Mai, 13:39, Janis <janis_papanag...@hotmail.com> wrote:
> On 2 Mai, 11:30, Jeremy Sanders <jer...@jeremysanders.net> wrote:
> [snip]
>
> > How do I get variable a to contain the expanded list of files, rather than
> > the glob itself?
>
> Mind that you may get inconsistencies in your further processing
> if the filenames may contain spaces, for example.
>
> > One workaround I've found is to use ls to do it, but this seems very ugly
> > a=`ls foo*`
>
> You can use the echo builtin if that seems less ugly to you
>
> a=$(echo foo*)

I guess I should have mentioned that the results differ from
the ls variant.

To have the expanded variables accessible individually you'd
best use an array, for example in the standard way

set - foo*

Janis

2008-05-2, 04:39(-07), Janis:
[...]
>> xpc:~:$ echo $a
>
> First the variable $a (which still contains 'foo*') is expanded,
> then the filename expansion (the globbing) is done.

I would rather say, that the "filename expansion" is part of the
"variable expansion" when the variable is not quoted. Other
transformations apply before that such as word splitting (note
that all that doesn't apply to zsh unless in sh/ksh emulation)

var="foo_bar*"
IFS=_
echo $var
doesn't list the "foo_bar..." files, but "foo" and the "bar..."
files.

>> foo1 foo2 foo3
>> xpc:~:$ echo "$a"
>
> No filename expansion is done in (single or) double quotes.

I would rather say that within double quotes, variable expansion
doesn't include filename generation.

[...]
>> One workaround I've found is to use ls to do it, but this seems very ugly
>> a=`ls foo*`
>
> You can use the echo builtin if that seems less ugly to you
>
> a=$(echo foo*)

That will be less reliable as "echo" may transform that, and
"foo*" will remain if there's no foo... file.

> Shells (don't know whether that is true for bash, though) can
> avoid a subshell if builtins are called inside $(...).
[...]

Only recent versions of ksh93 avoid subshells there.

--
Stéphane

On 2 Mai, 14:56, Stephane CHAZELAS <this.addr...@is.invalid> wrote:
> 2008-05-2, 04:39(-07), Janis:
> [...]
>
> >> xpc:~:$ echo $a
>
> > First the variable $a (which still contains 'foo*') is expanded,
> > then the filename expansion (the globbing) is done.
>
> I would rather say, that the "filename expansion" is part of the
> "variable expansion" when the variable is not quoted.

Well, I see it as separated steps in the parsing sequence (as
depicted e.g. in Rosenblatt/Robbins p.239).

> Other
> transformations apply before that such as word splitting (note
> that all that doesn't apply to zsh unless in sh/ksh emulation)
>
> var="foo_bar*"
> IFS=_
> echo $var
> doesn't list the "foo_bar..." files, but "foo" and the "bar..."
> files.
>
> >> foo1 foo2 foo3
> >> xpc:~:$ echo "$a"
>
> > No filename expansion is done in (single or) double quotes.
>
> I would rather say that within double quotes, variable expansion
> doesn't include filename generation.

Same as above. Within double quotes variable expansion (and
several other expansions) are done while file gobbing isn't.
I don't see globbing as part of the variable expansion, in my
view it's a separate concept and independent from variable
expansion.

[...]

> > Shells (don't know whether that is true for bash, though) can
> > avoid a subshell if builtins are called inside $(...).
>
> [...]
>
> Only recent versions of ksh93 avoid subshells there.

Yes, I had ksh in mind, but was reluctant to generalize/restrict
that statement to all current versions of other shells; often ksh
development took a leading role in inventions of new features and
I wouldn't be the least surprised if current versions of other
shells already adopted that optimization. But, frankly, I don't
know about this feature in current versions of other shells and
take your word for it.

Janis

2008-05-2, 06:30(-07), Janis:
> On 2 Mai, 14:56, Stephane CHAZELAS <this.addr...@is.invalid> wrote:
>> 2008-05-2, 04:39(-07), Janis:
>> [...]
>>
>> >> xpc:~:$ echo $a
>>
>> > First the variable $a (which still contains 'foo*') is expanded,
>> > then the filename expansion (the globbing) is done.
>>
>> I would rather say, that the "filename expansion" is part of the
>> "variable expansion" when the variable is not quoted.
>
> Well, I see it as separated steps in the parsing sequence (as
> depicted e.g. in Rosenblatt/Robbins p.239).

I don't know that "Rosenblatt/Robbins", but that would be over
simplification.

if it was as you said, that is if

a='* *'
echo $a

was the same as

echo * *

once the value of $a is expanded

Then

IFS=x
echo * *

would output the list of filenames with a space in them.


Now, it's true that

a='* b*'
echo a$a

expands to the a... and b... files

It's true it's source of confusion and in corner cases, shells
don't agree as in:

$ a='\*' export a
$ bash -c 'printf "%s\n" $a'
\*
$ ash -c 'printf "%s\n" $a'
*
$ ksh93 -c 'printf "%s\n" $a'
\a
\b
$ pdksh -c 'printf "%s\n" $a'
\a
\b
$ ARGV0=sh zsh -c 'printf "%s\n" $a'
\*

[...]
> Same as above. Within double quotes variable expansion (and
> several other expansions) are done while file gobbing isn't.
> I don't see globbing as part of the variable expansion, in my
> view it's a separate concept and independent from variable
> expansion.

I see your view, but you'll have to agree that globbing when
variables are involved is different from when not. Especially,
the way it works can't be described as:
- variables are expanded
- globbing is performed on the expanded version of the command
line.

There are a number of "processes" that are intertwinned
(tokenisation, word splitting, removal of quotes, filename
generation) which make the analysis of the whole process not
straightforward and behaviors differ from shell to shell.

>> Only recent versions of ksh93 avoid subshells there.
>
> Yes, I had ksh in mind, but was reluctant to generalize/restrict
> that statement to all current versions of other shells; often ksh
> development took a leading role in inventions of new features and
> I wouldn't be the least surprised if current versions of other
> shells already adopted that optimization. But, frankly, I don't
> know about this feature in current versions of other shells and
> take your word for it.
[...]

Looks like the trend in other actively developped Bourne like
shells (bash and zsh) is to add a -o VARNAME option to the
builtins where it may be useful to return the output in a
variable.

That optimisation of $(...) of ksh sounds a bit like
over-engineering to me and is bound to cause confusions. But I
guess that's a necessary step for ksh to be able to claim being
a programming language like perl or python (the next one being
to have all the Unix toolchest builtin).

--
Stéphane

Thanks for all your ful answers.

On Friday 2 May 2008 17:40, Stephane CHAZELAS wrote:

> if it was as you said, that is if
>
> a='* *'
> echo $a
>
> was the same as
>
> echo * *
>
> once the value of $a is expanded
>
> Then
>
> IFS=x
> echo * *
>
> would output the list of filenames with a space in them.

This is because the shell does a preliminary pass over the command line to
determine the input tokens, and that pass does not use IFS (only the shell
grammar). Instead, values obtained from variable expansion are subject to
word splitting using IFS.

When you do

IFS=x
echo * *

the shell gets the three tokens 'echo', '*', and '*', and proceeds from
there to do its expansions. Word splitting will not split anything, and
filename generation will expand each '*' independly.

Instead, when you do

IFS=x
a='* *'
echo $a

The shell gets two tokens, 'echo' and '$a', and goes from there. So, $a is
expanded to '* *', which is subject to word splitting; word splitting says
that '* *' is a single word according to IFS, and filename generation is
performed on that single word.

Is my understanding correct?

Thanks

--
All the commands are tested with bash and GNU tools, so they may use
nonstandard features. I try to mention when something is nonstandard (if
I'm aware of that), but I may miss something. Corrections are welcome.

Stephane CHAZELAS wrote:
> 2008-05-2, 06:30(-07), Janis:
>
>>On 2 Mai, 14:56, Stephane CHAZELAS <this.addr...@is.invalid> wrote:
>>
>>>2008-05-2, 04:39(-07), Janis:
>>>[...]
>>>
>>>
>>>>>xpc:~:$ echo $a
>>>
>>>>First the variable $a (which still contains 'foo*') is expanded,
>>>>then the filename expansion (the globbing) is done.
>>>
>>>I would rather say, that the "filename expansion" is part of the
>>>"variable expansion" when the variable is not quoted.
>>
>>Well, I see it as separated steps in the parsing sequence (as
>>depicted e.g. in Rosenblatt/Robbins p.239).
>
> I don't know that "Rosenblatt/Robbins", but that would be over
> simplification.

The book has a neat figure that depicts quite well the parsing
process that is defined in "The new Kornshell" (Bolksy/Korn) in
chapter 11 just in text form.

>
[...]
>>Same as above. Within double quotes variable expansion (and
>>several other expansions) are done while file gobbing isn't.
>>I don't see globbing as part of the variable expansion, in my
>>view it's a separate concept and independent from variable
>>expansion.
>
> I see your view, but you'll have to agree that globbing when
> variables are involved is different from when not. Especially,
> the way it works can't be described as:
> - variables are expanded
> - globbing is performed on the expanded version of the command
> line.
>
> There are a number of "processes" that are intertwinned
> (tokenisation, word splitting, removal of quotes, filename
> generation) which make the analysis of the whole process not
> straightforward and behaviors differ from shell to shell.

Yes, the process is a bit more complicated. The parsing sequence
shown in the above mentioned book - there are other sources that
illustrate the complex process as well, I am sure - shows all the
steps you mention.[*]
[*] And it still notes that the process description does not
describe all that in perfect detail, because things get a bit
more complex e.g. if compound commands are involved.

Janis