Hi all,

I want to download the one year of the data and all these data are kept
on data server in following dir structure

/2005/001
/2005/002
/2005/003
..
..
..
/2005/365

Means all files for one day is kept in seperate dirs.

Now, I want to write a shell script, which will perform automatic ftp
to the server with anonymous username and passwd (email) and download
all the files from each of the dir (each day) one by one
to one single local dir

I am trying this and have following script

#!/bin/bash

username="anonymous"
password="pawanpgupta@gmail.com"
localDir="/home/abc/downloads"
remoteDir="pub"
hostname="ftp.mit.edu" # for instance
mode="binary"

ftp -n -i -v "$hostname" <<EndOfSession
user "$username" "$password"
"$mode"
lcd "$localDir"
cd "remoteDir"
mget <file specification>
bye
EndOfSession
# End of script

This is working fine for one given dir name (one day) but when I tried
to run this inside loop for each dir (each day), it does not work? I do
not know why? if some body can me in this that will be great.

Also, If some body have some other script to perform similar work that
will be fine too

Pawan

On Fri, 08 Dec 2006 14:00:46 -0800, pawanpgupta wrote:

> Hi all,
>
> I want to download the one year of the data and all these data are kept
> on data server in following dir structure
>
> /2005/001
> /2005/002
> /2005/003
> .
> .
> .
> /2005/365
>
> Means all files for one day is kept in seperate dirs.
>
> Now, I want to write a shell script, which will perform automatic ftp
> to the server with anonymous username and passwd (email) and download
> all the files from each of the dir (each day) one by one
> to one single local dir
>
> I am trying this and have following script
>
> #!/bin/bash
>
> username="anonymous"
> password="pawanpgupta@gmail.com"
> localDir="/home/abc/downloads"
> remoteDir="pub"
> hostname="ftp.mit.edu" # for instance
> mode="binary"
>
> ftp -n -i -v "$hostname" <<EndOfSession
> user "$username" "$password"
> "$mode"
> lcd "$localDir"
> cd "remoteDir"
> mget <file specification>
> bye
> EndOfSession
> # End of script
>
> This is working fine for one given dir name (one day) but when I tried
> to run this inside loop for each dir (each day), it does not work? I do
> not know why? if some body can me in this that will be great.
>
> Also, If some body have some other script to perform similar work that
> will be fine too

carbon@tux:~$ for x in $(seq -w 001 365); do echo /2005/$x; done

/2005/001
/2005/002
/2005/003
..
..
..
/2005/365

On 8 Dec 2006 14:00:46 -0800, pawanpgupta@gmail.com wrote:

> Hi all,
>
> I want to download the one year of the data and all these data are kept
> on data server in following dir structure
>
> /2005/001
> /2005/002
> /2005/003
> .
> .
> .
> /2005/365
>
> Means all files for one day is kept in seperate dirs.
>
> Now, I want to write a shell script, which will perform automatic ftp
> to the server with anonymous username and passwd (email) and download
> all the files from each of the dir (each day) one by one
> to one single local dir
>
> I am trying this and have following script
>
> #!/bin/bash
>
> username="anonymous"
> password="pawanpgupta@gmail.com"
> localDir="/home/abc/downloads"
> remoteDir="pub"
> hostname="ftp.mit.edu" # for instance
> mode="binary"
>
> ftp -n -i -v "$hostname" <<EndOfSession
> user "$username" "$password"
> "$mode"
> lcd "$localDir"
> cd "remoteDir"
> mget <file specification>
> bye
> EndOfSession
> # End of script
>
> This is working fine for one given dir name (one day) but when I tried
> to run this inside loop for each dir (each day), it does not work? I do
> not know why? if some body can me in this that will be great.
>
> Also, If some body have some other script to perform similar work that
> will be fine too
>
> Pawan

I know this doesn't with the script, but have you looked at wget? If
it is installed it will do all of this for you in one or two lines.

cd /home/abc/downloads
wget -m ftp://ftp.mit.edu/$remoteDir

Where remotedir is /x/y/2005 not /x/y/2005/1
That way wget will handle the recursion etc.
The whole directory structure will be mirrored locally.
Does that ?
Adam