Hi,
I just got this script to write from my boss, but I thought I should
discuss my problem here too.

Ive got this situation where:
1) There are 2 server's ( A & B) located at quite a distance from each
other..

2) Only B can access A using the shell. A cannot access B.

3) There is a perl script on B which runs every hour. It downloads
certain files from another remote server using ftp. When these files
have been downloaded comepletely they are 'scp'd to A using a shell
script.

Now I have been told to:
1) Make a shell script in B, to compare, whether the downloaded files
in B have been completely 'scp'd to A.
2) When the comparing has been done, giving a result that the files are
same on both the servers, the script then should delete the downloaded
files from B

Frankly speaking, I am confused as to how to go about comparing the
files...


Thanks
Danish

Danish wrote:
> Hi,
> I just got this script to write from my boss, but I thought I should
> discuss my problem here too.
>
> Ive got this situation where:
> 1) There are 2 server's ( A & B) located at quite a distance from each
> other..
>
> 2) Only B can access A using the shell. A cannot access B.
>
> 3) There is a perl script on B which runs every hour. It downloads
> certain files from another remote server using ftp. When these files
> have been downloaded comepletely they are 'scp'd to A using a shell
> script.
>
> Now I have been told to:
> 1) Make a shell script in B, to compare, whether the downloaded files
> in B have been completely 'scp'd to A.
> 2) When the comparing has been done, giving a result that the files are
> same on both the servers, the script then should delete the downloaded
> files from B
>
> Frankly speaking, I am confused as to how to go about comparing the
> files...
>
>
> Thanks
> Danish

Forgot to add
> 2) Only B can access A using the shell. A cannot access B. The access is possible using ssh

Danish wrote:
> Hi,
> I just got this script to write from my boss, but I thought I should
> discuss my problem here too.
>
> Ive got this situation where:
> 1) There are 2 server's ( A & B) located at quite a distance from each
> other..
>
> 2) Only B can access A using the shell. A cannot access B.
>
> 3) There is a perl script on B which runs every hour. It downloads
> certain files from another remote server using ftp. When these files
> have been downloaded comepletely they are 'scp'd to A using a shell
> script.
>
> Now I have been told to:
> 1) Make a shell script in B, to compare, whether the downloaded files
> in B have been completely 'scp'd to A.
> 2) When the comparing has been done, giving a result that the files are
> same on both the servers, the script then should delete the downloaded
> files from B
>
> Frankly speaking, I am confused as to how to go about comparing the
> files...
>

You may want to check "rsync", which pretty much does most of the stuff
you want. try google or man rsync.

Good luck,
Xicheng

Yes , rsyncis one option. Also if the public keys are shared between A
and B , then
you can do this thru a script.

Some thing like . From node B> ssh A `ls filename`

It will return a value if it matches , the same way you can compare the
bytes

Thx
RC
Danish wrote:
> Hi,
> I just got this script to write from my boss, but I thought I should
> discuss my problem here too.
>
> Ive got this situation where:
> 1) There are 2 server's ( A & B) located at quite a distance from each
> other..
>
> 2) Only B can access A using the shell. A cannot access B.
>
> 3) There is a perl script on B which runs every hour. It downloads
> certain files from another remote server using ftp. When these files
> have been downloaded comepletely they are 'scp'd to A using a shell
> script.
>
> Now I have been told to:
> 1) Make a shell script in B, to compare, whether the downloaded files
> in B have been completely 'scp'd to A.
> 2) When the comparing has been done, giving a result that the files are
> same on both the servers, the script then should delete the downloaded
> files from B
>
> Frankly speaking, I am confused as to how to go about comparing the
> files...
>
>
> Thanks
> Danish

Danish wrote:
> Danish wrote:
>> Hi,
>> I just got this script to write from my boss, but I thought I should
>> discuss my problem here too.
>>
>> Ive got this situation where:
>> 1) There are 2 server's ( A & B) located at quite a distance from each
>> other..
>>
>> 2) Only B can access A using the shell. A cannot access B.
>>
>> 3) There is a perl script on B which runs every hour. It downloads
>> certain files from another remote server using ftp. When these files
>> have been downloaded comepletely they are 'scp'd to A using a shell
>> script.
>>
>> Now I have been told to:
>> 1) Make a shell script in B, to compare, whether the downloaded files
>> in B have been completely 'scp'd to A.
>> 2) When the comparing has been done, giving a result that the files are
>> same on both the servers, the script then should delete the downloaded
>> files from B
>>
>> Frankly speaking, I am confused as to how to go about comparing the
>> files...
>>

presumably you have a list of files on B that supposedly were
already uploaded to A. Use that list to generate "checksums"
of each of the files on each host. Compare the checksums and
if they match, the files on the two systems are identical.

You really should also do this on the original source host,
before it gets to B to ensure the file has been received on
B accurately before uploading to A.

Programs that do checksums include sha1sum, md5sum, cksum.

The files on A can be determined with ssh from B:

ssh A sha1sum file1 file2 file3 ...

Jon LaBadie wrote:
> Danish wrote:
> > Danish wrote:
> >> Hi,
> >> I just got this script to write from my boss, but I thought I should
> >> discuss my problem here too.
> >>
> >> Ive got this situation where:
> >> 1) There are 2 server's ( A & B) located at quite a distance from each
> >> other..
> >>
> >> 2) Only B can access A using the shell. A cannot access B.
> >>
> >> 3) There is a perl script on B which runs every hour. It downloads
> >> certain files from another remote server using ftp. When these files
> >> have been downloaded comepletely they are 'scp'd to A using a shell
> >> script.
> >>
> >> Now I have been told to:
> >> 1) Make a shell script in B, to compare, whether the downloaded files
> >> in B have been completely 'scp'd to A.
> >> 2) When the comparing has been done, giving a result that the files are
> >> same on both the servers, the script then should delete the downloaded
> >> files from B
> >>
> >> Frankly speaking, I am confused as to how to go about comparing the
> >> files...
> >>
>
> presumably you have a list of files on B that supposedly were
> already uploaded to A. Use that list to generate "checksums"
> of each of the files on each host. Compare the checksums and
> if they match, the files on the two systems are identical.
>
> You really should also do this on the original source host,
> before it gets to B to ensure the file has been received on
> B accurately before uploading to A.
>
> Programs that do checksums include sha1sum, md5sum, cksum.
>
> The files on A can be determined with ssh from B:
>
> ssh A sha1sum file1 file2 file3 ...

I spoke to my boss regarding the checksum, but he does not want it that
way...So ill have to think of some other way

Danish

Danish wrote:
> Jon LaBadie wrote:
> > Danish wrote:
> > > Danish wrote:
> > >> Hi,
> > >> I just got this script to write from my boss, but I thought I should
> > >> discuss my problem here too.
> > >>
> > >> Ive got this situation where:
> > >> 1) There are 2 server's ( A & B) located at quite a distance from each
> > >> other..
> > >>
> > >> 2) Only B can access A using the shell. A cannot access B.
> > >>
> > >> 3) There is a perl script on B which runs every hour. It downloads
> > >> certain files from another remote server using ftp. When these files
> > >> have been downloaded comepletely they are 'scp'd to A using a shell
> > >> script.
> > >>
> > >> Now I have been told to:
> > >> 1) Make a shell script in B, to compare, whether the downloaded files
> > >> in B have been completely 'scp'd to A.
> > >> 2) When the comparing has been done, giving a result that the files are
> > >> same on both the servers, the script then should delete the downloaded
> > >> files from B
> > >>
> > >> Frankly speaking, I am confused as to how to go about comparing the
> > >> files...
> > >>
> >
> > presumably you have a list of files on B that supposedly were
> > already uploaded to A. Use that list to generate "checksums"
> > of each of the files on each host. Compare the checksums and
> > if they match, the files on the two systems are identical.
> >
> > You really should also do this on the original source host,
> > before it gets to B to ensure the file has been received on
> > B accurately before uploading to A.
> >
> > Programs that do checksums include sha1sum, md5sum, cksum.
> >
> > The files on A can be determined with ssh from B:
> >
> > ssh A sha1sum file1 file2 file3 ...
>
> I spoke to my boss regarding the checksum, but he does not want it that
> way...So ill have to think of some other way

If you just want to get the status instead of updating files, with the
full-featured 'rsync':
1) no need to transfer files, with the option -n or --dry-run
2) use ssh/key-pairs with option like -e "ssh -i /path/to/.ssh/id_dsa
user@host"
3) skip files based on checksum, not mtime/size with the option -c,
--checksum (MD4 with my current rsync version).
........
you may also want the option -av to return more info and then parse it.

Xicheng

Xicheng Jia wrote:
> Danish wrote:
> > Jon LaBadie wrote:
> > > Danish wrote:
> > > > Danish wrote:
> > > >> Hi,
> > > >> I just got this script to write from my boss, but I thought I should
> > > >> discuss my problem here too.
> > > >>
> > > >> Ive got this situation where:
> > > >> 1) There are 2 server's ( A & B) located at quite a distance from each
> > > >> other..
> > > >>
> > > >> 2) Only B can access A using the shell. A cannot access B.
> > > >>
> > > >> 3) There is a perl script on B which runs every hour. It downloads
> > > >> certain files from another remote server using ftp. When these files
> > > >> have been downloaded comepletely they are 'scp'd to A using a shell
> > > >> script.
> > > >>
> > > >> Now I have been told to:
> > > >> 1) Make a shell script in B, to compare, whether the downloaded files
> > > >> in B have been completely 'scp'd to A.
> > > >> 2) When the comparing has been done, giving a result that the files are
> > > >> same on both the servers, the script then should delete the downloaded
> > > >> files from B
> > > >>
> > > >> Frankly speaking, I am confused as to how to go about comparing the
> > > >> files...
> > > >>
> > >
> > > presumably you have a list of files on B that supposedly were
> > > already uploaded to A. Use that list to generate "checksums"
> > > of each of the files on each host. Compare the checksums and
> > > if they match, the files on the two systems are identical.
> > >
> > > You really should also do this on the original source host,
> > > before it gets to B to ensure the file has been received on
> > > B accurately before uploading to A.
> > >
> > > Programs that do checksums include sha1sum, md5sum, cksum.
> > >
> > > The files on A can be determined with ssh from B:
> > >
> > > ssh A sha1sum file1 file2 file3 ...
> >
> > I spoke to my boss regarding the checksum, but he does not want it that
> > way...So ill have to think of some other way
>
> If you just want to get the status instead of updating files, with the
> full-featured 'rsync':
> 1) no need to transfer files, with the option -n or --dry-run
> 2) use ssh/key-pairs with option like -e "ssh -i /path/to/.ssh/id_dsa
> user@host"
> 3) skip files based on checksum, not mtime/size with the option -c,
> --checksum (MD4 with my current rsync version).
> .......
> you may also want the option -av to return more info and then parse it.
>
> Xicheng

Thanks for the rysnc, but as I had said, He does not want to use rsync,
so I cant. But I made a shell script with array's wherein:
1) Im doing `ls -l` on the dir and cutting the file size.
2) Feeding the output of `ls -l` into an array
3) Doing ssh remote_host "ls -l /path/to/files" and then feeding it
into an array

Im trying to compare the values of the two arrays, and if the the size
of each file is equal Ill put in the `rm`command to delete the files
from the local host..

Ive been able to feed the output into the array, but am not able to
figure out how to compare the value of the arrays....

Would like to have your advise on it....

#!/bin/sh



array=(`ls -l /home/danish/scp | tr -s " " | cut -d " " -f5`)

len=${#array[*]}


i=0

while [ $i -lt $len ]; do
echo "${array[$i]}"
let i++
done

echo "ssh to 192.168.10.98 starts here"

array1=`ssh 192.168.10.98 " vdir /home/danish/scp/* | tr -s \"
\"|cut -d \" \" -f5 "`

length=${#array1[*]}

j=0

while [ $j -lt $length ]; do
echo "${array1[$j]}"
let j++
done


Thanks
Danish

Danish wrote:
> Xicheng Jia wrote:
> > Danish wrote:
> > > Jon LaBadie wrote:
> > > > Danish wrote:
> > > > > Danish wrote:
> > > > >> Hi,
> > > > >> I just got this script to write from my boss, but I thought I should
> > > > >> discuss my problem here too.
> > > > >>
> > > > >> Ive got this situation where:
> > > > >> 1) There are 2 server's ( A & B) located at quite a distance from each
> > > > >> other..
> > > > >>
> > > > >> 2) Only B can access A using the shell. A cannot access B.
> > > > >>
> > > > >> 3) There is a perl script on B which runs every hour. It downloads
> > > > >> certain files from another remote server using ftp. When these files
> > > > >> have been downloaded comepletely they are 'scp'd to A using a shell
> > > > >> script.
> > > > >>
> > > > >> Now I have been told to:
> > > > >> 1) Make a shell script in B, to compare, whether the downloaded files
> > > > >> in B have been completely 'scp'd to A.
> > > > >> 2) When the comparing has been done, giving a result that the files are
> > > > >> same on both the servers, the script then should delete the downloaded
> > > > >> files from B
> > > > >>
> > > > >> Frankly speaking, I am confused as to how to go about comparing the
> > > > >> files...
> > > > >>
> > > >
> > > > presumably you have a list of files on B that supposedly were
> > > > already uploaded to A. Use that list to generate "checksums"
> > > > of each of the files on each host. Compare the checksums and
> > > > if they match, the files on the two systems are identical.
> > > >
> > > > You really should also do this on the original source host,
> > > > before it gets to B to ensure the file has been received on
> > > > B accurately before uploading to A.
> > > >
> > > > Programs that do checksums include sha1sum, md5sum, cksum.
> > > >
> > > > The files on A can be determined with ssh from B:
> > > >
> > > > ssh A sha1sum file1 file2 file3 ...
> > >
> > > I spoke to my boss regarding the checksum, but he does not want it that
> > > way...So ill have to think of some other way
> >
> > If you just want to get the status instead of updating files, with the
> > full-featured 'rsync':
> > 1) no need to transfer files, with the option -n or --dry-run
> > 2) use ssh/key-pairs with option like -e "ssh -i /path/to/.ssh/id_dsa
> > user@host"
> > 3) skip files based on checksum, not mtime/size with the option -c,
> > --checksum (MD4 with my current rsync version).
> > .......
> > you may also want the option -av to return more info and then parse it.
> >
> > Xicheng
>
> Thanks for the rysnc, but as I had said, He does not want to use rsync,
> so I cant. But I made a shell script with array's wherein:
> 1) Im doing `ls -l` on the dir and cutting the file size.
> 2) Feeding the output of `ls -l` into an array
> 3) Doing ssh remote_host "ls -l /path/to/files" and then feeding it
> into an array
>
> Im trying to compare the values of the two arrays, and if the the size
> of each file is equal Ill put in the `rm`command to delete the files
> from the local host..
>
> Ive been able to feed the output into the array, but am not able to
> figure out how to compare the value of the arrays....
>
> Would like to have your advise on it....
>
> #!/bin/sh
> array=(`ls -l /home/danish/scp | tr -s " " | cut -d " " -f5`)
> len=${#array[*]}
>
>
> i=0
>
> while [ $i -lt $len ]; do
> echo "${array[$i]}"
> let i++
> done
>
> echo "ssh to 192.168.10.98 starts here"
>
> array1=`ssh 192.168.10.98 " vdir /home/danish/scp/* | tr -s \"
> \"|cut -d \" \" -f5 "`
>
> length=${#array1[*]}
>
> j=0
>
> while [ $j -lt $length ]; do
> echo "${array1[$j]}"
> let j++
> done
>

If you do want to use scripts, then I'd suggest you use hash to handle
this problem, use the "filename filesize" as the hash key, and any
non-zero value as the corresponding value. shell arrays are associative
array(hash), so you can lookup the key-value pairs by using (key in
hash) instead of looping through the whole array elements. Below is an
awk script which might be ful for you to get started:

bash ~$ cat test.awk
#-----test.awk
BEGIN {
dir = "/path/to/local/dir/"
cmd = ("ls -l "dir)
while ((cmd | getline) > 0) {
key = $8$5
if (key != "") hash[key] = 1
}
close(cmd);
}
{
key = $8$5
}
key in hash {
print "rm -rf "dir $8 | "sh"
}

then on the command line, run:

ssh 192.168.10.98 "cd /home/danish/scp/ && ls -l" | awk -f test.awk

I assumed filename and filesize occupy 8th and 5th column of `ls -l`
output for both machines. and there is no whitespaces within
filenames(otherwise you may find some other ways to extract file/size
information). i.e. if you can use GNU find, then look at the option
-printf "%f%s" which might be better to extract filename-filesize info.

Good luck,
Xicheng

Danish wrote:
>
> Thanks for the rysnc, but as I had said, He does not want to use rsync,
> so I cant. But I made a shell script with array's wherein:
> 1) Im doing `ls -l` on the dir and cutting the file size.
> 2) Feeding the output of `ls -l` into an array
> 3) Doing ssh remote_host "ls -l /path/to/files" and then feeding it
> into an array
>
> Im trying to compare the values of the two arrays, and if the the size
> of each file is equal Ill put in the `rm`command to delete the files
> from the local host..
>
> Ive been able to feed the output into the array, but am not able to
> figure out how to compare the value of the arrays....

There is only one type of file for which identical ls -l results
guarantees identity of the two files -- the empty file.