Hi, all!

I'm having memory leak problem with my program, but can't find where's
the leak.
I have a program written in pure-Ruby. It implemented a Chinese word
segment
algorithm. I call `segment(text)' to get the result. But each time I
call `segment',
the memory usage of my program (Ruby) increased several handred K-
bytes -- the
number of bytes of increment is roughly equal during each call.

I can see from outside (using `ps' or `top') that the memory is
leaking. But I can't
find where the leaking goes. I tried to use ObjectSpace.each_object to
find out
what leaks. I followed this article

http://scottstuff.net/blog/articles/...ing-with-rails

and compare the objects after each call to `segment'. What surprise me
is that
there's no object leaked between each call. The number of objects
after each
call is roughly equal.

I also tried bleak_house:

http://blog.evanweaver.com/files/doc...es/README.html

I don't precisely how to use it. I just dump a snapshot after each
call to `segment'
and use `bleak' to analyze the result. However, it also shows no leak.
But if
there's no leak, where goes the memory?

Here's the result of using the `pmap' tool to inspect the running Ruby
process:

933: /usr/bin/ruby1.8 /usr/bin/rake PROFILE=false test:many
08048000 4K r-x-- /usr/bin/ruby1.8
08049000 4K rw--- /usr/bin/ruby1.8
0804a000 33412K rw--- [ anon ]
b74a8000 7016K rw--- [ anon ]
b7b82000 16K r-x-- /usr/lib/ruby/1.8/i486-linux/strscan.so
b7b86000 4K rw--- /usr/lib/ruby/1.8/i486-linux/strscan.so
b7b87000 16K r-x-- /var/lib/gems/1.8/gems/ruby-prof-0.6.0/lib/
ruby_prof.so
b7b8b000 4K rw--- /var/lib/gems/1.8/gems/ruby-prof-0.6.0/lib/
ruby_prof.so
b7b8c000 1192K rw--- [ anon ]
b7cb6000 1308K r-x-- /lib/i686/cmov/libc-2.7.so
b7dfd000 4K r---- /lib/i686/cmov/libc-2.7.so
b7dfe000 8K rw--- /lib/i686/cmov/libc-2.7.so
b7e00000 12K rw--- [ anon ]
b7e03000 140K r-x-- /lib/i686/cmov/libm-2.7.so
b7e26000 8K rw--- /lib/i686/cmov/libm-2.7.so
b7e28000 36K r-x-- /lib/i686/cmov/libcrypt-2.7.so
b7e31000 8K rw--- /lib/i686/cmov/libcrypt-2.7.so
b7e33000 156K rw--- [ anon ]
b7e5a000 8K r-x-- /lib/i686/cmov/libdl-2.7.so
b7e5c000 8K rw--- /lib/i686/cmov/libdl-2.7.so
b7e5e000 4K rw--- [ anon ]
b7e5f000 80K r-x-- /lib/i686/cmov/libpthread-2.7.so
b7e73000 8K rw--- /lib/i686/cmov/libpthread-2.7.so
b7e75000 8K rw--- [ anon ]
b7e77000 756K r-x-- /usr/lib/libruby1.8.so.1.8.6
b7f34000 8K rw--- /usr/lib/libruby1.8.so.1.8.6
b7f36000 64K rw--- [ anon ]
b7f48000 8K rw--- [ anon ]
b7f4a000 8K r-x-- /usr/lib/ruby/1.8/i486-linux/etc.so
b7f4c000 4K rw--- /usr/lib/ruby/1.8/i486-linux/etc.so
b7f4d000 12K r-x-- /usr/lib/ruby/1.8/i486-linux/thread.so
b7f50000 4K rw--- /usr/lib/ruby/1.8/i486-linux/thread.so
b7f51000 8K rw--- [ anon ]
b7f53000 104K r-x-- /lib/ld-2.7.so
b7f6d000 8K rw--- /lib/ld-2.7.so
bfdea000 116K rw--- [ stack ]
ffffe000 4K r-x-- [ anon ]
total 44568K

The section "0804a000 33412K rw--- [ anon ]" increased after each
call. I've no idea now. Do you have any suggestion for me?

On Mon, Feb 25, 2008 at 4:29 PM, pluskid <pluskid@gmail.com> wrote:
> Hi, all!
>
> I'm having memory leak problem with my program, but can't find where's
> the leak.
> I have a program written in pure-Ruby. It implemented a Chinese word
> segment
> algorithm. I call `segment(text)' to get the result. But each time I
> call `segment',
> the memory usage of my program (Ruby) increased several handred K-
> bytes -- the
> number of bytes of increment is roughly equal during each call.
>
> I can see from outside (using `ps' or `top') that the memory is
> leaking. But I can't
> find where the leaking goes. I tried to use ObjectSpace.each_object to
> find out
> what leaks. I followed this article
>
> http://scottstuff.net/blog/articles/...ing-with-rails
>
> and compare the objects after each call to `segment'. What surprise me
> is that
> there's no object leaked between each call. The number of objects
> after each
> call is roughly equal.
>
> I also tried bleak_house:
>
> http://blog.evanweaver.com/files/doc...es/README.html
>
> I don't precisely how to use it. I just dump a snapshot after each
> call to `segment'
> and use `bleak' to analyze the result. However, it also shows no leak.
> But if
> there's no leak, where goes the memory?
>
> Here's the result of using the `pmap' tool to inspect the running Ruby
> process:
>
> 933: /usr/bin/ruby1.8 /usr/bin/rake PROFILE=false test:many
> 08048000 4K r-x-- /usr/bin/ruby1.8
> 08049000 4K rw--- /usr/bin/ruby1.8
> 0804a000 33412K rw--- [ anon ]
> b74a8000 7016K rw--- [ anon ]
> b7b82000 16K r-x-- /usr/lib/ruby/1.8/i486-linux/strscan.so
> b7b86000 4K rw--- /usr/lib/ruby/1.8/i486-linux/strscan.so
> b7b87000 16K r-x-- /var/lib/gems/1.8/gems/ruby-prof-0.6.0/lib/
> ruby_prof.so
> b7b8b000 4K rw--- /var/lib/gems/1.8/gems/ruby-prof-0.6.0/lib/
> ruby_prof.so
> b7b8c000 1192K rw--- [ anon ]
> b7cb6000 1308K r-x-- /lib/i686/cmov/libc-2.7.so
> b7dfd000 4K r---- /lib/i686/cmov/libc-2.7.so
> b7dfe000 8K rw--- /lib/i686/cmov/libc-2.7.so
> b7e00000 12K rw--- [ anon ]
> b7e03000 140K r-x-- /lib/i686/cmov/libm-2.7.so
> b7e26000 8K rw--- /lib/i686/cmov/libm-2.7.so
> b7e28000 36K r-x-- /lib/i686/cmov/libcrypt-2.7.so
> b7e31000 8K rw--- /lib/i686/cmov/libcrypt-2.7.so
> b7e33000 156K rw--- [ anon ]
> b7e5a000 8K r-x-- /lib/i686/cmov/libdl-2.7.so
> b7e5c000 8K rw--- /lib/i686/cmov/libdl-2.7.so
> b7e5e000 4K rw--- [ anon ]
> b7e5f000 80K r-x-- /lib/i686/cmov/libpthread-2.7.so
> b7e73000 8K rw--- /lib/i686/cmov/libpthread-2.7.so
> b7e75000 8K rw--- [ anon ]
> b7e77000 756K r-x-- /usr/lib/libruby1.8.so.1.8.6
> b7f34000 8K rw--- /usr/lib/libruby1.8.so.1.8.6
> b7f36000 64K rw--- [ anon ]
> b7f48000 8K rw--- [ anon ]
> b7f4a000 8K r-x-- /usr/lib/ruby/1.8/i486-linux/etc.so
> b7f4c000 4K rw--- /usr/lib/ruby/1.8/i486-linux/etc.so
> b7f4d000 12K r-x-- /usr/lib/ruby/1.8/i486-linux/thread.so
> b7f50000 4K rw--- /usr/lib/ruby/1.8/i486-linux/thread.so
> b7f51000 8K rw--- [ anon ]
> b7f53000 104K r-x-- /lib/ld-2.7.so
> b7f6d000 8K rw--- /lib/ld-2.7.so
> bfdea000 116K rw--- [ stack ]
> ffffe000 4K r-x-- [ anon ]
> total 44568K
>
> The section "0804a000 33412K rw--- [ anon ]" increased after each
> call. I've no idea now. Do you have any suggestion for me?

This is what I have read here or somewhere else, and I an not
knowledgeable enough to confirm it, anyway:
Ruby allocs space from the OS in large chunks (initially 8 MB, and
each other chunk is bigger by a predefined coefficient).
Any chunk can be returned to the OS only if it is empty - so if your
memory is fragmented, ruby will have to alloc new chunks
in order to satisfy ruby script being run.

If this is true, it means: the order and size of your allocations is important.
Especially order of the allocations that you later release relative to
those that you don't.

If that is the case, that might be quite random, istead
of seeing the increment of every call roughly equal.

2008/2/25, Jano Svitok <jan.svitok@gmail.com>:
> On Mon, Feb 25, 2008 at 4:29 PM, pluskid <pluskid@gmail.com> wrote:
> > Hi, all!
> >
> > I'm having memory leak problem with my program, but can't find where's
> > the leak.
> > I have a program written in pure-Ruby. It implemented a Chinese word
> > segment
> > algorithm. I call `segment(text)' to get the result. But each time I
> > call `segment',
> > the memory usage of my program (Ruby) increased several handred K-
> > bytes -- the
> > number of bytes of increment is roughly equal during each call.
> >
> > I can see from outside (using `ps' or `top') that the memory is
> > leaking. But I can't
> > find where the leaking goes. I tried to use ObjectSpace.each_object to
> > find out
> > what leaks. I followed this article
> >
> > http://scottstuff.net/blog/articles/...ing-with-rails
> >
> > and compare the objects after each call to `segment'. What surprise me
> > is that
> > there's no object leaked between each call. The number of objects
> > after each
> > call is roughly equal.
> >
> > I also tried bleak_house:
> >
> > http://blog.evanweaver.com/files/doc...es/README.html
> >
> > I don't precisely how to use it. I just dump a snapshot after each
> > call to `segment'
> > and use `bleak' to analyze the result. However, it also shows no leak.
> > But if
> > there's no leak, where goes the memory?
> >
> > Here's the result of using the `pmap' tool to inspect the running Ruby
> > process:
> >
> > 933: /usr/bin/ruby1.8 /usr/bin/rake PROFILE=false test:many
> > 08048000 4K r-x-- /usr/bin/ruby1.8
> > 08049000 4K rw--- /usr/bin/ruby1.8
> > 0804a000 33412K rw--- [ anon ]
> > b74a8000 7016K rw--- [ anon ]
> > b7b82000 16K r-x-- /usr/lib/ruby/1.8/i486-linux/strscan.so
> > b7b86000 4K rw--- /usr/lib/ruby/1.8/i486-linux/strscan.so
> > b7b87000 16K r-x-- /var/lib/gems/1.8/gems/ruby-prof-0.6.0/lib/
> > ruby_prof.so
> > b7b8b000 4K rw--- /var/lib/gems/1.8/gems/ruby-prof-0.6.0/lib/
> > ruby_prof.so
> > b7b8c000 1192K rw--- [ anon ]
> > b7cb6000 1308K r-x-- /lib/i686/cmov/libc-2.7.so
> > b7dfd000 4K r---- /lib/i686/cmov/libc-2.7.so
> > b7dfe000 8K rw--- /lib/i686/cmov/libc-2.7.so
> > b7e00000 12K rw--- [ anon ]
> > b7e03000 140K r-x-- /lib/i686/cmov/libm-2.7.so
> > b7e26000 8K rw--- /lib/i686/cmov/libm-2.7.so
> > b7e28000 36K r-x-- /lib/i686/cmov/libcrypt-2.7.so
> > b7e31000 8K rw--- /lib/i686/cmov/libcrypt-2.7.so
> > b7e33000 156K rw--- [ anon ]
> > b7e5a000 8K r-x-- /lib/i686/cmov/libdl-2.7.so
> > b7e5c000 8K rw--- /lib/i686/cmov/libdl-2.7.so
> > b7e5e000 4K rw--- [ anon ]
> > b7e5f000 80K r-x-- /lib/i686/cmov/libpthread-2.7.so
> > b7e73000 8K rw--- /lib/i686/cmov/libpthread-2.7.so
> > b7e75000 8K rw--- [ anon ]
> > b7e77000 756K r-x-- /usr/lib/libruby1.8.so.1.8.6
> > b7f34000 8K rw--- /usr/lib/libruby1.8.so.1.8.6
> > b7f36000 64K rw--- [ anon ]
> > b7f48000 8K rw--- [ anon ]
> > b7f4a000 8K r-x-- /usr/lib/ruby/1.8/i486-linux/etc.so
> > b7f4c000 4K rw--- /usr/lib/ruby/1.8/i486-linux/etc.so
> > b7f4d000 12K r-x-- /usr/lib/ruby/1.8/i486-linux/thread.so
> > b7f50000 4K rw--- /usr/lib/ruby/1.8/i486-linux/thread.so
> > b7f51000 8K rw--- [ anon ]
> > b7f53000 104K r-x-- /lib/ld-2.7.so
> > b7f6d000 8K rw--- /lib/ld-2.7.so
> > bfdea000 116K rw--- [ stack ]
> > ffffe000 4K r-x-- [ anon ]
> > total 44568K
> >
> > The section "0804a000 33412K rw--- [ anon ]" increased after each
> > call. I've no idea now. Do you have any suggestion for me?
>
>
> This is what I have read here or somewhere else, and I an not
> knowledgeable enough to confirm it, anyway:
> Ruby allocs space from the OS in large chunks (initially 8 MB, and
> each other chunk is bigger by a predefined coefficient).
> Any chunk can be returned to the OS only if it is empty - so if your
> memory is fragmented, ruby will have to alloc new chunks
> in order to satisfy ruby script being run.
>
> If this is true, it means: the order and size of your allocations is important.
> Especially order of the allocations that you later release relative to
> those that you don't.
>
>