Hi All,

I am facing a wierd problem. I have about a 100 shell scripts all coded
in a similar fashion. All of them read from some file(s) and process
the data.

But there is this one script which reads from a small file which has
only 2 lines of data. What happens (and happens only for this script)
is that only the first line is read, and the last line is ignored
completely. I do not get any errors.

What we do to workaround this problem is to just open the file (vi)..
add a new line to the end and just remove that line. Then when we run
the same script it works. But we do not want to keep doing that.

Here is my rather simple script:
************************************************** ***
cat ./data/BACSTEL_APPLICATION_DATA.txt | while IFS= read line
do
echo $line
done
************************************************** ***

And here is the data file:
************************************************** ***
$ more data/BACSTEL_APPLICATION_DATA.txt
BACSTEL~account type~Business User~~0~
BACSTEL~account type~Admin User~~0~
************************************************** ***

When I run the script. This is the output:
************************************************** ***
$ ./populate_bacstel.ksh
BACSTEL~account type~Business User~~0~
************************************************** ***

All other scripts are coded in a very similar fashion... they all run
fine, except this. Can it be that this is too small a file .....?? Coz
that is the only diff.. the next shortest data file is 4 lines long....
but that is soo wierd!

Thanks,
Anoop

On 18 Aug 2006 07:49:37 -0700, Anoop
<anoopkumarv@gmail.com> wrote:
> Hi All,
>
> I am facing a wierd problem. I have about a 100 shell scripts all coded
> in a similar fashion. All of them read from some file(s) and process
> the data.
>
> But there is this one script which reads from a small file which has
> only 2 lines of data. What happens (and happens only for this script)
> is that only the first line is read, and the last line is ignored
> completely. I do not get any errors.
>
> What we do to workaround this problem is to just open the file (vi)..
> add a new line to the end and just remove that line. Then when we run
> the same script it works. But we do not want to keep doing that.
>
> Here is my rather simple script:
> ************************************************** ***
> cat ./data/BACSTEL_APPLICATION_DATA.txt | while IFS= read line
> do
> echo $line
> done
> ************************************************** ***

Perhaps the file does not end with a newline. A workaround would be
{ cat ./data/BACSTEL_APPLICATION_DATA.txt; echo; } | while IFS= read line

or fix the program that creates the file.


--
Try to be the best of whatever you are, even if what you are is no good.

On Fri, 18 Aug 2006 11:21:25 -0400, Bill Marcum wrote:
[...]
>> cat ./data/BACSTEL_APPLICATION_DATA.txt | while IFS= read line
>> do
>> echo $line
>> done
>> ************************************************** ***
>
> Perhaps the file does not end with a newline. A workaround would be
> { cat ./data/BACSTEL_APPLICATION_DATA.txt; echo; } | while IFS= read line
[...]

I don't think any shell implementation would discard such a
line. I don't know of any that does.

POSIX doesn't seem very clear on that matter. It doesn't seem to
cover the case where eof is reached while reading a line. Reading
http://www.opengroup.org/onlinepubs/...ties/read.html,
one could tell that not a single shell is conformant.

Apparently, if $IFS doesn't contain the newline character, then
in

read var

$var should contain the entire line read, that is including the
trailing newline character. All the shells I know strip the line
terminator, which makes sense.

--
Stephane

Bill Marcum wrote:
> On 18 Aug 2006 07:49:37 -0700, Anoop
> <anoopkumarv@gmail.com> wrote:
> > Hi All,
> >
> > I am facing a wierd problem. I have about a 100 shell scripts all coded
> > in a similar fashion. All of them read from some file(s) and process
> > the data.
> >
> > But there is this one script which reads from a small file which has
> > only 2 lines of data. What happens (and happens only for this script)
> > is that only the first line is read, and the last line is ignored
> > completely. I do not get any errors.
> >
> > What we do to workaround this problem is to just open the file (vi)..
> > add a new line to the end and just remove that line. Then when we run
> > the same script it works. But we do not want to keep doing that.
> >
> > Here is my rather simple script:
> > ************************************************** ***
> > cat ./data/BACSTEL_APPLICATION_DATA.txt | while IFS= read line
> > do
> > echo $line
> > done
> > ************************************************** ***
>
> Perhaps the file does not end with a newline. A workaround would be
> { cat ./data/BACSTEL_APPLICATION_DATA.txt; echo; } | while IFS= read line
>
> or fix the program that creates the file.
>
>
> --
> Try to be the best of whatever you are, even if what you are is no good.

The files are created manually. And I assure you they were created in
exactly the same way as the others... The only thing (which is also
common for every file) is that we get it into windows - into CVS and
then we ftp the files onto the SunOS boxes. there we run dos2unix
command on every file we ftp-ed.

Let me try your workaround - i will have to fix this one shell script
with that. If that works - i will stick to it.

Thanks,
Anoop

"Anoop" <anoopkumarv@gmail.com> wrote in message
news:1155912577.327410.285380@75g2000cwc.googlegro ups.com...
> Hi All,
>
> I am facing a wierd problem. I have about a 100 shell scripts all coded
> in a similar fashion. All of them read from some file(s) and process
> the data.
>
> But there is this one script which reads from a small file which has
> only 2 lines of data. What happens (and happens only for this script)
> is that only the first line is read, and the last line is ignored
> completely. I do not get any errors.

As Bill Marcum suggested, the problem is in the file, not the
script. I can reproduce your problem by creating a file whose
last line does not have a newline character at the end (read
is looking for this character, cat doesn't care). A newline
tells read it has a full line in the buffer and it's time to
copy it to your variable, but it's hitting the end of the file
before that happens, so...

-Wm

William wrote:
> "Anoop" <anoopkumarv@gmail.com> wrote in message
> news:1155912577.327410.285380@75g2000cwc.googlegro ups.com...
> > Hi All,
> >
> > I am facing a wierd problem. I have about a 100 shell scripts all coded
> > in a similar fashion. All of them read from some file(s) and process
> > the data.
> >
> > But there is this one script which reads from a small file which has
> > only 2 lines of data. What happens (and happens only for this script)
> > is that only the first line is read, and the last line is ignored
> > completely. I do not get any errors.
>
> As Bill Marcum suggested, the problem is in the file, not the
> script. I can reproduce your problem by creating a file whose
> last line does not have a newline character at the end (read
> is looking for this character, cat doesn't care). A newline
> tells read it has a full line in the buffer and it's time to
> copy it to your variable, but it's hitting the end of the file
> before that happens, so...
>
> -Wm

I agree - but these files were handcoded and come from the client side
- we cannot control what comes from them..

But the workaround posted by Bill Marcum works fine. I think I am going
ahead with modifying just that one script...

Thanks alot to all,
Anoop

"Anoop" <anoopkumarv@gmail.com> wrote in message
news:1155919273.163202.268030@p79g2000cwp.googlegr oups.com...
>
>
> I agree - but these files were handcoded and come from the client side
> - we cannot control what comes from them..
>
> But the workaround posted by Bill Marcum works fine. I think I am going
> ahead with modifying just that one script...
>
> Thanks alot to all,
> Anoop

Watch out for other problems with these files - some things will
handle them fine, but many versions of common tools will drop the
last line. Running the files through sed, for example, may chop
off the last line. (We experienced that with some configuration
files - some of the developers decided to make them "neater" by
removing the last newline. That broke our build and was very
hard to track down.)

-Wm

Stephane Chazelas <stephane_chazelas@yahoo.fr> writes:

> I don't think any shell implementation would discard such a
> line. I don't know of any that does.

I do know of cases where other utilities will not see the last line if
it's not terminated by a '\n'. I think I had problems with sed and/or
grep a decade ago (was probably an old Solaris system).

I remember it because I had to write a work-around for it.


--
Sending unsolicited commercial e-mail to this account incurs a fee of
$500 per message, and acknowledges the legality of this contract.