Hello

I am trying to find a simple way to loop through a string or array, and
remove duplicate elements or characters , respectfully..

I cannot change the order of the list, such as I cannot use sort... I
have been looking all around and I am stumped.....


I put together this code, inspired by 'bubblesort' but it isnt
removing the duplicate characters:

while [ "$arrayindex" -le "$arraylength" ]
do
indexchar=`print ${newvar[$arrayindex]}`
echo "indexchar: "$indexchar
echo "arrayindex: "$arrayindex
counter2=1


while [ "$arrayindex2" -le "$arraylength" ]
do
comparechar=`print ${newvar[$counter2]}`
echo "comparechar: "$comparechar

if [ "$indexchar" = "$comparechar" ]
then
newvar2[$arrayindex]="$indexchar"
else
newvar2[$counter2]="$comparechar"
newvar2[$arrayindex]="$indexchar"

fi


counter2=`expr "$counter2" + 1 `
arrayindex2=`expr "$arrayindex2" + 1 `
done



arrayindex=`expr "$arrayindex" + 1 `
arrayindex2=1

On 2006-11-04, thesoulcrusher wrote:
> Hello
>
> I am trying to find a simple way to loop through a string or array, and
> remove duplicate elements or characters , respectfully..
>
> I cannot change the order of the list, such as I cannot use sort... I
> have been looking all around and I am stumped.....

If the elements in the array do not contain newlines, this will
eliminate duplicates:

printf "%s\n" "${array[@]}" | awk 'x[$0]++ != 0'


--
Chris F.A. Johnson, author <http://cfaj.freeshell.org/shell>
Shell Scripting Recipes: A Problem-Solution Approach (2005, Apress)
===== My code in this post, if any, assumes the POSIX locale
===== and is released under the GNU General Public Licence

Chris F.A. Johnson wrote:
> On 2006-11-04, thesoulcrusher wrote:
>
>>Hello
>>
>>I am trying to find a simple way to loop through a string or array, and
>>remove duplicate elements or characters , respectfully..
>>
>>I cannot change the order of the list, such as I cannot use sort... I
>>have been looking all around and I am stumped.....
>
>
> If the elements in the array do not contain newlines, this will
> eliminate duplicates:
>
> printf "%s\n" "${array[@]}" | awk 'x[$0]++ != 0'

Shouldn't that be ... | awk 'x[$0]++ == 0'
or ... | awk '!x[$0]++'

Janis

On 2006-11-04, Janis Papanagnou wrote:
> Chris F.A. Johnson wrote:
>> On 2006-11-04, thesoulcrusher wrote:
>>
>>>Hello
>>>
>>>I am trying to find a simple way to loop through a string or array, and
>>>remove duplicate elements or characters , respectfully..
>>>
>>>I cannot change the order of the list, such as I cannot use sort... I
>>>have been looking all around and I am stumped.....
>>
>>
>> If the elements in the array do not contain newlines, this will
>> eliminate duplicates:
>>
>> printf "%s\n" "${array[@]}" | awk 'x[$0]++ != 0'
>
> Shouldn't that be ... | awk 'x[$0]++ == 0'
> or ... | awk '!x[$0]++'

Quite right.

--
Chris F.A. Johnson, author <http://cfaj.freeshell.org/shell>
Shell Scripting Recipes: A Problem-Solution Approach (2005, Apress)
===== My code in this post, if any, assumes the POSIX locale
===== and is released under the GNU General Public Licence

2006-11-3, 17:42(-08), thesoulcrusher:
> Hello
>
> I am trying to find a simple way to loop through a string or array, and
> remove duplicate elements or characters , respectfully..
[...]

Note that arrays are not a standard shell feature.

With zsh,

a=(1 2 1 2 3 4 5 3)
typeset -U a

forces "a" elements to be unique.

Alternatively, you can use the (u) expansion flag:

b=(${(u)a})

or

b=("${(u)a[@]}") or b=("${(@u)a}")
if you want to preserve the empty elements.

For dupplicate characters within a string, you can first split
the individual characters into an array then unique the array,
then join the elements again.

$ s="foo bar"
$ print -r -- ${(j:${(us:s}}
fo bar

With standard shell, you'd do:

for arrays (standard shells only have the positional parameters
as arrays):

set foo bar foo 1 2 3 2 3 1
start=true
for i do
if "$start"; then
set --
start=false
fi
for j do
[ "$i" = "$j" ] && continue 2
done
set -- "$@" "$i"
done

And with strings:

s="foo bar"
awk '
BEGIN {
s = ARGV[1]
n = length(s)
for (i = 1; i <= n; i++) {
c = substr(s, i, 1)
if (! (c in seen_already)) {
u = u c
seen_already[c] = ""
}
}
print u
exit
}' "$s"

--
Stéphane

Thank you all for posting,

Stephane

Yours removes any duplicate characters, what i need is each duplicate
item, i.e.
String1=" apple apple orange pear"
i need it to output:

"apple orange pear"

Im not sure how to do this, uniq will not work because I need to
preserve the order of the list...

2006-11-8, 19:06(-08), thesoulcrusher:
> Thank you all for posting,
>
> Stephane
>
> Yours removes any duplicate characters, what i need is each duplicate
> item, i.e.
> String1=" apple apple orange pear"
> i need it to output:
>
> "apple orange pear"
>
> Im not sure how to do this, uniq will not work because I need to
> preserve the order of the list...

zsh:

print -r -- ${(u)=String1}

portably:

set -f
IFS=" "
set -- $String1
start=true
for i do
if "$start"; then
set --
start=false
fi
for j do
[ "$i" = "$j" ] && continue 2
done
set -- "$@" "$i"
done
newString1="$*"

Or:

set -f
IFS=" "
newString1=
for i in $String1; do
case " $newString1 " in
*" $i "*) ;;
" ")
newString1=$i;;
*)
newString1="$newString1 $i";;
esac
done

--
Stéphane

thesoulcrusher wrote:
> Hello
>
> I am trying to find a simple way to loop through a string or array, and
> remove duplicate elements or characters , respectfully [respectively]..
>
> I cannot change the order of the list, such as I cannot use sort... I
> have been looking all around and I am stumped.....

ruby -e 'puts "foobar".split("").uniq.join'
fobar


ruby -e 'puts %w(ha ha hee ho).uniq'
ha
hee
ho

thesoulcrusher wrote:

> Thank you all for posting,
>
> Stephane
>
> Yours removes any duplicate characters, what i need is each duplicate
> item, i.e.
> String1=" apple apple orange pear"
> i need it to output:
>
> "apple orange pear"
>
> Im not sure how to do this, uniq will not work because I need to
> preserve the order of the list...
>

$ echo "apple apple orange pear" | awk '{for(i=1;i<=NF;i++) if(!($i in
w)){s=(s?s" ":"")$i; w[$i]}}END{print s}'
apple orange pear

Regards,

Ed.

Ed Morton wrote:

> thesoulcrusher wrote:
>
>> Thank you all for posting,
>>
>> Stephane
>>
>> Yours removes any duplicate characters, what i need is each duplicate
>> item, i.e.
>> String1=" apple apple orange pear"
>> i need it to output:
>>
>> "apple orange pear"
>>
>> Im not sure how to do this, uniq will not work because I need to
>> preserve the order of the list...
>>
>
> $ echo "apple apple orange pear" |
> awk '{for(i=1;i<=NF;i++) if(!($i in w)){s=(s?s" ":"")$i;w[$i]}}END{print s}'
> apple orange pear

FYI, the above can be made a bit briefer if your input can be one word
per line:

$ printf "%s\n" apple apple orange pear |
awk '!($0 in w){s=(s?s" ":"")$0;w[$0]}END{print s}'
apple orange pear

or if your awk supports character classes as record separators:

$ echo "apple apple orange pear" |
awk -v RS="[[:space:]]" '!($0 in w){s=(s?s" ":"")$0;w[$0]}END{print s}'
apple orange pear

or if you can always provide a blank character at the end of your input:

$ echo "apple apple orange pear " |
awk -v RS=" " '!($0 in w){s=(s?s" ":"")$0;w[$0]}END{printf "%s",s}'
apple pear orange

Regards,

Ed.



Regards,

Ed.