CJ wrote:

> We were discussing implementing malloc(), in particular the following
> situation.
>
> Suppose the user requests 1Mb of memory. Unfortunately, we only have
> 512Kb available. In this situation, most mallocs() would return null.
> The huge majority of programmers won't bother to check malloc() failure
> for such a small allocation, so the program will crash with a SIGSEGV as
> soon as the NULL pointer is dereferenced.
>
> So why not just return a pointer to the 512Kb that's available? It's
> quite possible that the user will never actually write into the upper
> half of the memory he's allocated, in which case the program will have
> continued successfully where before it would have crashed.
>
> The worst thing that can happen is that the programmer _does_ write to
> the end of the mallocated block. In this case, either there's a SIGSEGV
> again (no worse off than before), or if the 512Kb is in the middle of
> the heap malloc() is drawing from then the writes might well succeed,
> and the program can continue albeit with some possible minor data
> corruption.

On a related note, the Linux kernel may be configured so as to
overcommit memory. (It is even the default.)

http://lxr.linux.no/source/Documenta...mit-accounting

On 26 Nov 2007 at 22:34, Eric Sosman wrote:
> Not only does this implementation avoid the processing
> overhead of maintaining potentially large data structures
> describing the state of memory pools, but it also reduces
> the "memory footprint" of every program that uses it, thus
> lowering page fault rates, swap I/O rates, and out-of-memory
> problems.

All very amusing, but I think you and some of the other posters have
misunderstood the context of the discussion.

Of course an implementation of malloc should return a pointer to as much
memory as requested whenever that's possible! The discussion was about
the rare failure case. Typically, programs assume malloc returns a
non-null pointer, so if it returns null on failure then the program will
crash and burn.

The idea is that instead of a guaranteed crash, isn't it better to make
a last-ditch effort to save the program's bacon by returning a pointer
to what memory there _is_ available? If the program's going to crash
anyway, it's got to be worth a shot.


=======================================
There once was an old man from Esser,
Who's knowledge grew lesser and lesser.
It at last grew so small,
He knew nothing at all,
And now he's a College Professor.

CJ wrote:
> On 26 Nov 2007 at 22:34, Eric Sosman wrote:
>> Not only does this implementation avoid the processing
>> overhead of maintaining potentially large data structures
>> describing the state of memory pools, but it also reduces
>> the "memory footprint" of every program that uses it, thus
>> lowering page fault rates, swap I/O rates, and out-of-memory
>> problems.
>
> All very amusing, but I think you and some of the other posters have
> misunderstood the context of the discussion.
>
> Of course an implementation of malloc should return a pointer to as much
> memory as requested whenever that's possible! The discussion was about
> the rare failure case. Typically, programs assume malloc returns a
> non-null pointer, so if it returns null on failure then the program will
> crash and burn.
>
> The idea is that instead of a guaranteed crash, isn't it better to make
> a last-ditch effort to save the program's bacon by returning a pointer
> to what memory there _is_ available? If the program's going to crash
> anyway, it's got to be worth a shot.
>
>
> =======================================
> There once was an old man from Esser,
> Who's knowledge grew lesser and lesser.
> It at last grew so small,
> He knew nothing at all,
> And now he's a College Professor.

Well, I proposed the function
mallocTry
that could give you the best of both worlds. Why do
you ignore that suggestion?


--
jacob navia
jacob at jacob point remcomp point fr
logiciels/informatique
http://www.cs.virginia.edu/~lcc-win32

"CJ" <nospam@nospam.invalid> wrote in message
news:slrnfkomlu.fsa.nospam@nospam.invalid...
> Of course an implementation of malloc should return a pointer to as much
> memory as requested whenever that's possible! The discussion was about
> the rare failure case. Typically, programs assume malloc returns a
> non-null pointer, so if it returns null on failure then the program will
> crash and burn.

A program only crashes on malloc() returning NULL if the programmer is
incompetent.

> The idea is that instead of a guaranteed crash, isn't it better to make
> a last-ditch effort to save the program's bacon by returning a pointer
> to what memory there _is_ available? If the program's going to crash
> anyway, it's got to be worth a shot.

The change you propose would mean that malloc() _might_ save some
incompetent programmers but _definitely_ make it impossible for competent
programmers to write correct programs because there's no longer a way to
detect if they got the amount of memory they requested. That goes against
both the fundamental philosophy of C and common sense.

S

--
Stephen Sprunk "God does not play dice." --Albert Einstein
CCIE #3723 "God is an inveterate gambler, and He throws the
K5SSS dice at every possible opportunity." --Stephen Hawking

In article <slrnfkomlu.fsa.nospam@nospam.invalid>,
CJ <nospam@nospam.invalid> wrote:
>On 26 Nov 2007 at 22:34, Eric Sosman wrote:
>> Not only does this implementation avoid the processing
>> overhead of maintaining potentially large data structures
>> describing the state of memory pools, but it also reduces
>> the "memory footprint" of every program that uses it, thus
>> lowering page fault rates, swap I/O rates, and out-of-memory
>> problems.
>
>All very amusing, but I think you and some of the other posters have
>misunderstood the context of the discussion.

[...]

>The idea is that instead of a guaranteed crash,
the programmer should handle a failure to allocate resources properly,
and the implementation should make it possible to do so.

I think you're the one who's missing the point.


dave

CJ wrote:

> On 26 Nov 2007 at 22:34, Eric Sosman wrote:
>> Not only does this implementation avoid the processing
>> overhead of maintaining potentially large data structures
>> describing the state of memory pools, but it also reduces
>> the "memory footprint" of every program that uses it, thus
>> lowering page fault rates, swap I/O rates, and out-of-memory
>> problems.
>
> All very amusing, but I think you and some of the other posters have
> misunderstood the context of the discussion.
>
> Of course an implementation of malloc should return a pointer to as
> much memory as requested whenever that's possible! The discussion was
> about the rare failure case. Typically, programs assume malloc returns
> a non-null pointer, so if it returns null on failure then the program
> will crash and burn.

A properly written non-trivial program is not going to "crash and burn"
when malloc() returns NULL. It will shutdown gracefully, or even notify
the user to free up some memory and perhaps try again.

> The idea is that instead of a guaranteed crash,

No. There is no guaranteed crash. Consider:

#include <stdlib.h>

int main(void) {
char *p = malloc(SIZE_MAX);
if (p) { free(p); return 0; }
else return EXIT_FAILURE;
}

Where is the "crash"?

> isn't it better to
> make a last-ditch effort to save the program's bacon by returning a
> pointer to what memory there _is_ available? If the program's going to
> crash anyway, it's got to be worth a shot.

If malloc() returns NULL for failure, the program can at least exit
gracefully or even try and recover some memory and keep going, but if
it is going to indicate success but return insufficient space, then a
memory overwrite is almost certainly going to lead to either hard to
debug data corruption or a core dump from a segmentation violation.
This is (for any serious program) worse than a clean, controlled exit.

Besides your function (for whatever it's worth) can easily be written on
top of malloc() for anyone mad enough to want it. No need to include
another gets() into the Standard.

CJ wrote On 11/27/07 12:58,:
> On 26 Nov 2007 at 22:34, Eric Sosman wrote:
>
>> Not only does this implementation avoid the processing
>>overhead of maintaining potentially large data structures
>>describing the state of memory pools, but it also reduces
>>the "memory footprint" of every program that uses it, thus
>>lowering page fault rates, swap I/O rates, and out-of-memory
>>problems.
>
>
> All very amusing, but I think you and some of the other posters have
> misunderstood the context of the discussion.

Well, yes: I understood it as a joke. But if
you're actually serious ...

> Of course an implementation of malloc should return a pointer to as much
> memory as requested whenever that's possible! The discussion was about
> the rare failure case. Typically, programs assume malloc returns a
> non-null pointer, so if it returns null on failure then the program will
> crash and burn.

One could take issue with your assertion about
"typically," and even about "rare." But that's a
side-issue.

> The idea is that instead of a guaranteed crash, isn't it better to make
> a last-ditch effort to save the program's bacon by returning a pointer
> to what memory there _is_ available?

No. For starters, this makes what you consider an
"atypical" program as bad as a "typical" one, because
it becomes impossible to detect a malloc() failure: if
malloc() returns a non-NULL pointer, how can the caller
tell whether it was or wasn't able to supply the memory?

> If the program's going to crash
> anyway, it's got to be worth a shot.

Your unstated assumption is that a crash is worse
than all other outcomes, and I dispute that assumption.
A program that ceases to operate also ceases to produce
wrong answers. It probably also draws attention to the
fact that something has gone wrong, instead of plunging
silently ahead doing who-knows-what.

In my house I have smoke alarms that use nine-volt
batteries to supply their modest electrical needs. The
service life of the battery in this application is quite
long, well over a year, so the battery's "failure rate"
is quite low: When the alarm "requests more electricity"
from the battery, it nearly always succeeds. Yet on the
extremely rare occasion where the battery cannot deliver
enough, the alarm starts beeping at intervals to alert me
to the fact. Then I change the battery, and all is well.

If the battery were designed to work the way you want
malloc() to operate, this wouldn't happen. The battery
would (somehow) pretend to be able to supply the requested
voltage even when it could not, and the alarm would never
discover that the battery was dead. Instead, the alarm
would sit silently on my wall, giving the illusion of
performing its function without actually doing so, pretending
to protect me and my family while leaving us at risk. You
may not admire me much, but you know nothing of my family
and I request that you not condemn them to the flames.

--
Eric.Sosman@sun.com

CJ wrote, On 27/11/07 17:58:
> On 26 Nov 2007 at 22:34, Eric Sosman wrote:
>> Not only does this implementation avoid the processing
>> overhead of maintaining potentially large data structures
>> describing the state of memory pools, but it also reduces
>> the "memory footprint" of every program that uses it, thus
>> lowering page fault rates, swap I/O rates, and out-of-memory
>> problems.
>
> All very amusing, but I think you and some of the other posters have
> misunderstood the context of the discussion.

No, I believe that Eric and most of the other respondents understood it
perfectly, and that is why they all considered it a terrible idea.

> Of course an implementation of malloc should return a pointer to as much
> memory as requested whenever that's possible! The discussion was about
> the rare failure case.

Rare to you maybe, but I actually use machines fairly hard and it is not
uncommon for them to run out of memory.

> Typically, programs assume malloc returns a
> non-null pointer, so if it returns null on failure then the program will
> crash and burn.

Not if it is written properly, and a lot of programs *are* written properly.

> The idea is that instead of a guaranteed crash, isn't it better to make
> a last-ditch effort to save the program's bacon by returning a pointer
> to what memory there _is_ available? If the program's going to crash
> anyway, it's got to be worth a shot.

Oh dear, you have just caused my copy of VMWare to crash possibly
causing corruption of the guest systems file system and forcing me to
restart a long and complex set of tests involving a number of VMware
sessions. You have also prevented one of the applications I develop from
tidying up cleanly behind itself and notifying the user, possibly
causing corruption of a companies corporate accounts.

No, it is FAR better to do what the standard requires and actually give
the application a shot at doing something sensible when it runs out of
memory, such as warning the user and giving them a chance to free some
memory up, or shutting down tidily so that data is not corrupted etc.
--
Flash Gordon

In <slrnfkomlu.fsa.nospam@nospam.invalid> CJ <nospam@nospam.invalid> writes:

> Typically, programs assume malloc returns a non-null pointer, so if it
< returns null on failure then the program will crash and burn.

Huh? If malloc returns null, the program detects it handles it as best
it can. It certainly doesn't *use* the null pointer gotten from malloc.

Where's this "guaranteed crash?"

--
John Gordon A is for Amy, who fell down the stairs
gordon@panix.com B is for Basil, assaulted by bears
-- Edward Gorey, "The Gashlycrumb Tinies"

jacob navia wrote:
>
> Well, I proposed the function
> mallocTry
> that could give you the best of both worlds. Why do
> you ignore that suggestion?

He's begun by assuming a caller who's too [insert
derogatory adjective] even to compare malloc's value
to NULL. Would this [ida] person have the smarts to
make effective use of mallocTry?

--
Eric Sosman
esosman@ieee-dot-org.invalid

Al Balmer wrote:
> On Mon, 26 Nov 2007 22:40:52 +0100 (CET), CJ <nospam@nospam.invalid>
> wrote:
>
>> The worst thing that can happen is that the programmer _does_ write to
>> the end of the mallocated block. In this case, either there's a SIGSEGV
>> again (no worse off than before), or if the 512Kb is in the middle of
>> the heap malloc() is drawing from then the writes might well succeed,
>> and the program can continue albeit with some possible minor data
>> corruption.
>>
>> Do any implementations of malloc() use a strategy like this?
>>
> This is all a joke, isn't it?
>

On that note:

http://groups.google.com/group/comp....4?dmode=source

http://groups.google.com/group/comp....6?dmode=source

Same guy?

--
SM
rot13 for email

CJ wrote:
>
> We were discussing implementing malloc(), in particular the following
> situation.
>
> Suppose the user requests 1Mb of memory. Unfortunately, we only have
> 512Kb available. In this situation, most mallocs() would return null.

AFAIK, _all_ conforming mallocs return NULL. (With the possible
exception of things like Linux's overcommit scheme. However, it is my
understanding that it is the kernel that lies to malloc, and malloc
does, in fact, believe that the memory is available.)

> The huge majority of programmers won't bother to check malloc() failure

I have to take exception with that statement.

> for such a small allocation, so the program will crash with a SIGSEGV as
> soon as the NULL pointer is dereferenced.

Someone who doesn't check for malloc() failure, especially on such
a large size, deserves what he gets.

Consider, too, the ease in debugging the NULL pointer dereference.

> So why not just return a pointer to the 512Kb that's available? It's

Eww...

> quite possible that the user will never actually write into the upper
> half of the memory he's allocated, in which case the program will have
> continued successfully where before it would have crashed.

I would consider that strategy "asking for trouble".

> The worst thing that can happen is that the programmer _does_ write to
> the end of the mallocated block. In this case, either there's a SIGSEGV
> again (no worse off than before), or if the 512Kb is in the middle of
> the heap malloc() is drawing from then the writes might well succeed,
> and the program can continue albeit with some possible minor data
> corruption.

So continuing "with some possible minor data corruption" is a
viable programming strategy? You want to make buffer overruns a
design strategy?

Consider the difference in difficulty in "malloc returned NULL" to
"I malloced a million bytes, but something else is mysteriously
writing into my buffer, and writing into my buffer causes some
other code to mysteriously crash".

How many Windows updates were to fix "malicious code could allow an
attacker to take over your computer" bugs which were caused by such
buffer overruns?

> Do any implementations of malloc() use a strategy like this?

I sincerely hope not. I don't want my implementation lying to me.
("Lazy mallocs" are bad enough. We don't need "outright lying
mallocs".)

--
+-------------------------+--------------------+-----------------------+
| Kenneth J. Brody | www.hvcomputer.com | #include |
| kenbrody/at\spamcop.net | www.fptech.com | <std_disclaimer.h> |
+-------------------------+--------------------+-----------------------+
Don't e-mail me at: <mailto:ThisIsASpamTrap@gmail.com>

CJ wrote:


> Of course an implementation of malloc should return a pointer to as
> much memory as requested whenever that's possible! The discussion was
> about the rare failure case. Typically, programs assume malloc
> returns a non-null pointer, so if it returns null on failure then the
> program will crash and burn.

You're either an idiot or a troll. You haven't understood, or have
chosen to ignore all the follow-ups.

As such, you're a waste of my time (the greatest usenet sin).

*plonk*





Brian

On Nov 27, 9:58 am, CJ <nos...@nospam.invalid> wrote:
> On 26 Nov 2007 at 22:34, Eric Sosman wrote:
>
> > Not only does this implementation avoid the processing
> > overhead of maintaining potentially large data structures
> > describing the state of memory pools, but it also reduces
> > the "memory footprint" of every program that uses it, thus
> > lowering page fault rates, swap I/O rates, and out-of-memory
> > problems.
>
> All very amusing, but I think you and some of the other posters have
> misunderstood the context of the discussion.
>
> Of course an implementation of malloc should return a pointer to as much
> memory as requested whenever that's possible! The discussion was about
> the rare failure case. Typically, programs assume malloc returns a
> non-null pointer, so if it returns null on failure then the program will
> crash and burn.
>
> The idea is that instead of a guaranteed crash, isn't it better to make
> a last-ditch effort to save the program's bacon by returning a pointer
> to what memory there _is_ available? If the program's going to crash
> anyway, it's got to be worth a shot.

Surely, surely. This is a troll.

If I asked for 4 MB and the implementation returned 2 MB, I can't
really imagine anything worse than that. Not only is it undefined
behavior, but the behavior is very unlikely even to be repeatable. Why
(on earth) would the program 'crash anyway'? Anyone who does not
check the return of malloc() in a production application is an
incompetent dimwit of the highest order. So there is no way that the
application is going to crash. But of course, anyone who ever did
real-life programming in C knows all of this already.

If malloc() fails, and I could succeed with less memory (e.g. if I am
creating a hash table with a small fill factor, I can increase it),
then I am going to try to allocate less memory. To simply return a
smaller block... you're a riot.

Quite an effective troll, sir. I congratulate you.

CJ wrote:
> On 26 Nov 2007 at 22:34, Eric Sosman wrote:
> > Not only does this implementation avoid the processing
> > overhead of maintaining potentially large data structures
> > describing the state of memory pools, but it also reduces
> > the "memory footprint" of every program that uses it, thus
> > lowering page fault rates, swap I/O rates, and out-of-memory
> > problems.
>
> All very amusing, but I think you and some of the other posters have
> misunderstood the context of the discussion.

No, we understood the context; we just have a well-justified lack of
respect for the attitudes that could create such a context

> Of course an implementation of malloc should return a pointer to as much
> memory as requested whenever that's possible! The discussion was about
> the rare failure case. Typically, programs assume malloc returns a
> non-null pointer, so if it returns null on failure then the program will
> crash and burn.
>
> The idea is that instead of a guaranteed crash, isn't it better to make
> a last-ditch effort to save the program's bacon by returning a pointer
> to what memory there _is_ available? If the program's going to crash
> anyway, it's got to be worth a shot.

No, it is not. The sooner and more frequently such incompetently
written code fails, the sooner the programmer will realize why it was
a bad idea to ignore null pointers returned by malloc(). If the
programmer never realizes this, then the sooner and more frequently
the programs fail, the sooner the programmer will be fired and
replaced with someone more competent. Everyone who deserves to be will
be better off as a result. Even the fired programmer will be better
off in the long run, because getting fired will give the ex-programmer
an opportunity to change to a career which doesn't require as much
attention to detail as programming, which should make the ex-
programmer happier in the long run.

Competent programmers check for failed memory allocation, and take
appropriate action. If allocation fails, their programs will either
fail gracefully (NOT crash-and-burn), or they will free up memory
elsewhere and retry the allocation. Your proposed change would cause
competently written programs that rely upon the failure indication to
fail without warning, in order to provide very questionable protection
for incompetently written code.

CJ wrote:
> We were discussing implementing malloc(), in particular the following
> situation.
>
> Suppose the user requests 1Mb of memory. Unfortunately, we only have
> 512Kb available. In this situation, most mallocs() would return null.
> The huge majority of programmers won't bother to check malloc() failure
> for such a small allocation, so the program will crash with a SIGSEGV as
> soon as the NULL pointer is dereferenced.
>
> So why not just return a pointer to the 512Kb that's available?

Yike. Take this to the extreme. Imagine you only have one byte
available. Why not just return a pointer to that? How much use is /that/?

>It's
> quite possible that the user will never actually write into the upper
> half of the memory he's allocated,

Its also quite possible the programmer knew how much memory he needed,
and will be justifiably annoyed when his programme starts randomly failing.

In article <slrnfkomlu.fsa.nospam@nospam.invalid> CJ <nospam@nospam.invalid> writes:
....
> The idea is that instead of a guaranteed crash, isn't it better to make
> a last-ditch effort to save the program's bacon by returning a pointer
> to what memory there _is_ available? If the program's going to crash
> anyway, it's got to be worth a shot.

In that case there is no way a program can actually *check* whether there
was sufficient memory, which a well written program should do. The result
is probably a fault, but all kind of other nasty things can happen, like
completely wrong output.
--
dik t. winter, cwi, kruislaan 413, 1098 sj amsterdam, nederland, +31205924131
home: bovenover 215, 1025 jn amsterdam, nederland; http://www.cwi.nl/~dik/

CJ wrote:
> On 26 Nov 2007 at 22:34, Eric Sosman wrote:
>> Not only does this implementation avoid the processing
>> overhead of maintaining potentially large data structures
>> describing the state of memory pools, but it also reduces
>> the "memory footprint" of every program that uses it, thus
>> lowering page fault rates, swap I/O rates, and out-of-memory
>> problems.
>
> All very amusing, but I think you and some of the other posters have
> misunderstood the context of the discussion.
>
> Of course an implementation of malloc should return a pointer to as much
> memory as requested whenever that's possible! The discussion was about
> the rare failure case. Typically, programs assume malloc returns a
> non-null pointer, so if it returns null on failure then the program will
> crash and burn.

Why do you think such incompetently written buggy programs are "typical"?

> The idea is that instead of a guaranteed crash, isn't it better to make
> a last-ditch effort to save the program's bacon by returning a pointer
> to what memory there _is_ available? If the program's going to crash
> anyway, it's got to be worth a shot.

No, blatantly and obviously not. It is better for the programmer's
incompetence to cause the program to stop execution immediately rather
than allow it to go on with random behaviour doing who knows how much
damage to what. How does, for example, corrupting a few gigabytes of
valuable archive data "save the program's bacon"?

> The worst thing that can happen is that the programmer _does_ write to
> the end of the mallocated block. In this case, either there's a SIGSEGV
> again (no worse off than before), or if the 512Kb is in the middle of
> the heap malloc() is drawing from then the writes might well succeed,
> and the program can continue albeit with some possible minor data
> corruption.
>
> Do any implementations of malloc() use a strategy like this?

omg

this is dumbest thing I have read on the internet

"J. J. Farrell" <jjf@bcs.org.uk> wrote in message
>
> No, blatantly and obviously not. It is better for the programmer's
> incompetence to cause the program to stop execution immediately rather
> than allow it to go on with random behaviour doing who knows how much
> damage to what. How does, for example, corrupting a few gigabytes of
> valuable archive data "save the program's bacon"?
>
It does depend what the program is. For instance with a videogame there is
no point exiting with an error message. You might as well ignore the error,
and there's a chance the player won't notice. If he does notice, it won't
spoil his enjoyment any more than an exit.
Unless of course you manage to corrupt his saved character which he's spent
two years of solid gameplay bringing up to a high level. Then people get
annoyed.

--
Free games and programming goodies.
http://www.personal.leeds.ac.uk/~bgy1mm

"Malcolm McLean" <regniztar@btinternet.com> writes:
> "J. J. Farrell" <jjf@bcs.org.uk> wrote in message
>> No, blatantly and obviously not. It is better for the programmer's
>> incompetence to cause the program to stop execution immediately
>> rather than allow it to go on with random behaviour doing who knows
>> how much damage to what. How does, for example, corrupting a few
>> gigabytes of valuable archive data "save the program's bacon"?
>>
> It does depend what the program is. For instance with a videogame
> there is no point exiting with an error message. You might as well
> ignore the error, and there's a chance the player won't notice. If he
> does notice, it won't spoil his enjoyment any more than an exit.
> Unless of course you manage to corrupt his saved character which he's
> spent two years of solid gameplay bringing up to a high level. Then
> people get annoyed.

Even in that context, the proposed malloc implementation where a
request for 1024 kbytes can allocate just 512 kbytes and not report an
error doesn't seem particularly useful.

And if it is useful, you can always implement it on top of malloc *and
call it something else*.

--
Keith Thompson (The_Other_Keith) <kst-u@mib.org>
Looking for software development work in the San Diego area.
"We must do something. This is something. Therefore, we must do this."
-- Antony Jay and Jonathan Lynn, "Yes Minister"

On Nov 28, 10:40 pm, "Malcolm McLean" <regniz...@btinternet.com>
wrote:

> It does depend what the program is. For instance with a videogame there is
> no point exiting with an error message. You might as well ignore the error,
> and there's a chance the player won't notice. If he does notice, it won't
> spoil his enjoyment any more than an exit.

I once played "Who wants to be a millionaire" on a Playstation. The
program crashed at the £8,000 question. It would have been more
enjoyable if the program had crashed immediately.