Paul Scott wrote:
> I have a very large text file that gets dumped into a directoory every
> now and then. It is typically around 750MB long, at least, and my
> question is:
>
> What is the best method to parse this thing and insert the data into a
> postgres db?
>
> I have tried using file(), fget*() and some others, all with limited
> success. It goes through OK (I am sending it to a background process on
> the server and using a callback function to email me when done) but it
> is really knocking the machine hard, besides taking a real long time to
> finish.
>
> Is there a better way of approaching this? Any would be greatly
> appreciated.

First, which is your approach? I suspect that you are doing this with a
cron job through php-cli.

Now, to avoid using to many resources, try with fopen() and fgets().
Also work with persistent connections, so you don't have that overhead.

The problem with file() is that it will load all the file to memory, and
you don't want 700+MB in memory.

--
21:50:04 up 2 days, 9:07, 0 users, load average: 0.92, 0.37, 0.18
---------------------------------------------------------
Lic. Martín Marqués | SELECT 'mmarques' ||
Centro de Telemática | '@' || 'unl.edu.ar';
Universidad Nacional | DBA, Programador,
del Litoral | Administrador
---------------------------------------------------------