By previous replies, it seems that the following method somehow solves the
problem up to 1000 * 1000 2D data, but when I try 10k * 10k, the
segmentation fault problem appears again.

Richard Tobin told me there is a system limit that can be changed. But I
don't know which file is to be changed.

I have modified again and again and hope to find out a solution that can
handle 100k * 100k data.

float** array_to_matrix(float* m, int rows, int cols) {
int i,j;

float** r;

r = (float**)calloc(rows,sizeof(float*));

for(i=0;i<rows;i++)
{
r[i] = (float*)calloc(cols,sizeof(float));

for(j=0;j<cols;j++)
r[i][j] = m[i*cols+j];
}
return r;

}

a wrote:
> By previous replies, it seems that the following method somehow solves the
> problem up to 1000 * 1000 2D data, but when I try 10k * 10k, the
> segmentation fault problem appears again.
>
System memory is finite, if you attempt to allocate more than there is
available, you will fail.

> Richard Tobin told me there is a system limit that can be changed. But I
> don't know which file is to be changed.
>
> I have modified again and again and hope to find out a solution that can
> handle 100k * 100k data.
>
Which is 10GB*sizeof(float), do you have that much (virtual) memory to
play with?

It sound like you have more of an algorithm problem than a C one.

> float** array_to_matrix(float* m, int rows, int cols) {
> int i,j;
>
> float** r;
>
> r = (float**)calloc(rows,sizeof(float*));
>
Drop the cast.

--
Ian Collins.

Ian Collins wrote:
> a wrote:
>> By previous replies, it seems that the following method somehow solves the
>> problem up to 1000 * 1000 2D data, but when I try 10k * 10k, the
>> segmentation fault problem appears again.
>>
> System memory is finite, if you attempt to allocate more than there is
> available, you will fail.
>
>> Richard Tobin told me there is a system limit that can be changed. But I
>> don't know which file is to be changed.
>>
>> I have modified again and again and hope to find out a solution that can
>> handle 100k * 100k data.
>>
> Which is 10GB*sizeof(float), do you have that much (virtual) memory to
> play with?
>
> It sound like you have more of an algorithm problem than a C one.
>
>> float** array_to_matrix(float* m, int rows, int cols) {
>> int i,j;
>>
>> float** r;
>>
>> r = (float**)calloc(rows,sizeof(float*));
>>
> Drop the cast.
>
And always check the return of [mc]alloc isn't null.

--
Ian Collins.

In article <fia513$rvc$1@justice.itsc.cuhk.edu.hk>, a <a@a.com> wrote:

>Richard Tobin told me there is a system limit that can be changed. But I
>don't know which file is to be changed.

As I said, ask your system administrator. Or read the manual.
We can't tell you, because you haven't even told us what system
you're using.

>I have modified again and again and hope to find out a solution that can
>handle 100k * 100k data.

100k * 100k is 10g, and if floats are 4 bytes that's 40 gigabytes.
You really will need a supercomputer for that. Perhaps you should
reconsider your algorithm, or wait several years.

-- Richard
--
"Consideration shall be given to the need for as many as 32 characters
in some alphabets" - X3.4, 1963.

>
> 100k * 100k is 10g, and if floats are 4 bytes that's 40 gigabytes.
> You really will need a supercomputer for that. Perhaps you should
> reconsider your algorithm, or wait several years.
>
<OT> Workstation boards with support for 64GB of RAM are available from
several vendors! All it takes is a spare $10K for the RAM... </OT>

--
Ian Collins.

Richard Tobin schrieb:

> 100k * 100k is 10g, and if floats are 4 bytes that's 40 gigabytes.
> You really will need a supercomputer for that. Perhaps you should
> reconsider your algorithm, or wait several years.

It might work if enough virtual memory is available - serious thrashing
implied. Suitable for *some* problems, however, if access pattern to
this matrix are not arbitrary (which they aren't for most algorithms).

Usually data in such huge matrices is sparse anyways - so, I fully
second your statement the OP should reconsider his algorithm. If it
fails in the early stage of memory allocation he/she probably hasn't
thourhgt about it at all.

Greetings,
Johannes

--
"Viele der Theorien der Mathematiker sind falsch und klar
Gotteslästerlich. Ich vermute, dass diese falschen Theorien genau
deshalb so geliebt werden." -- Prophet und Visionär Hans Joss aka
HJP in de.sci.mathematik <4740ad67$0$3811$5402220f@news.sunrise.ch>

>> 100k * 100k is 10g, and if floats are 4 bytes that's 40 gigabytes.
>> You really will need a supercomputer for that. Perhaps you should
>> reconsider your algorithm, or wait several years.
>
>It might work if enough virtual memory is available - serious thrashing
>implied.

On a 32-bit machine (say, Pentium with PAE36) you could have 64G
of physical memory, and a lot more physical swap/page space, but
with a 32-bit address space for an individual process, you're limited
to 4G (and sometimes much less). So you need a machine with 64-bit
addressing and an OS that supports it for individual processes.
Simply adding lots of memory and swap/page space isn't enough.

>Suitable for *some* problems, however, if access pattern to
>this matrix are not arbitrary (which they aren't for most algorithms).
>
>Usually data in such huge matrices is sparse anyways - so, I fully
>second your statement the OP should reconsider his algorithm. If it
>fails in the early stage of memory allocation he/she probably hasn't
>thourhgt about it at all.

Thanks Richard. It's red hat enterprise, or Fedora related. The biggest
problem I'm having is that I don't know the keyword because compilation
memory program memory alike doesn't give me good results in google search.


"Richard Tobin" <richard@cogsci.ed.ac.uk> wrote in message
news:fiaav3$973$2@pc-news.cogsci.ed.ac.uk...
> In article <fia513$rvc$1@justice.itsc.cuhk.edu.hk>, a <a@a.com> wrote:
>
>>Richard Tobin told me there is a system limit that can be changed. But I
>>don't know which file is to be changed.
>
> As I said, ask your system administrator. Or read the manual.
> We can't tell you, because you haven't even told us what system
> you're using.
>
> -- Richard
> --
> "Consideration shall be given to the need for as many as 32 characters
> in some alphabets" - X3.4, 1963.