Hi list.

I've posted this problem to a few other lists over the past few days
(rdiff-backup, my local LUG), but I haven't had a reply yet, so I
thought I'd try here.

Short version: rdiff-backup is using 2 GB of memory (1 GB RAM, 1 GB
swap) on one of my backup servers. I'm using the latest Etch versions,
so there shouldn't be a memory leak in librsync.

Here is my (somewhat long) mail to my LUG:

http://lists.clug.org.za/pipermail/c...ay/040532.html

Does anyone on this list have suggestions?

Thanks in advance,

David.


--
To UNSUBSCRIBE, email to debian-user-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org

I read you CLUG post. It seems like you should be able to do everything that
you want using rdiff-backup and not using your temp work directory with rsync
(which looks to be messing things up).

Also, if you are using rdiff-backup on backup1, why do you need to preserve
file history on backup2? Shouldn't the copy of backup1 on backup2 also
contain the rdiff-backup-data directory? If this is the case then you can
just use rsync to move the backup from backup1 to backup2.

MM


--
To UNSUBSCRIBE, email to debian-user-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org

Hi there and thanks for your reply.

On Wed, May 7, 2008 at 6:39 PM, Matthew Dale Moore
<Matthew.D.Moore@colorado.edu> wrote:
> I read you CLUG post. It seems like you should be able to do everything that
> you want using rdiff-backup and not using your temp work directory with rsync
> (which looks to be messing things up).

Last time I checked, rdiff-backup only works over a network if you
have rdiff-backup on the other side. This means that for Windows boxes
we would need to install Cygwin etc. If there was a simple Windows
installer for rdiff-backup (similar to DeltaCopy for rsync) it would
be another story.

Also, I don't trust rdiff-backup as much as I do rsync. It seems a bit
too complicated/fragile by comparison. Rsync is very robust, simple,
and works every time. The only reason I use rdiff-backup is because of
it's reverse delta support. I would prefer to replace rdiff-backup if
possible, rather than rsync.

And finally, we already have rsync on most of the workstations (after
a long period of phasing it in, to enable faster backups than with SMB
shares). There would need to be a strong reason to change from rsync
(on the machines being backed up) to rdiff-backup.

>
> Also, if you are using rdiff-backup on backup1, why do you need to preserve
> file history on backup2? Shouldn't the copy of backup1 on backup2 also
> contain the rdiff-backup-data directory? If this is the case then you can
> just use rsync to move the backup from backup1 to backup2.
>

This is for a few reasons:

1) I'm using the same backup script on both servers (with different
config). It would be extra work to disable the rdiff-backup part.
2) If backup1 looses data, and backup2's backup runs, I don't want to
lose the data from backup1 at that time
3) I also want to keep history for the entire backup1 (not just the
backups). This is so I can restore the entire backup1 server as it
was X days ago if there are problems.

David


--
To UNSUBSCRIBE, email to debian-user-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org

On Wednesday 07 May 2008 11:58:20 am David wrote:
> Also, I don't trust rdiff-backup as much as I do rsync. It seems a bit
> too complicated/fragile by comparison. Rsync is very robust, simple,
> and works every time. The only reason I use rdiff-backup is because of
> it's reverse delta support. I would prefer to replace rdiff-backup if
> possible, rather than rsync.

If you are looking for a replacement, I don't know of any that do rdiffs
besides rdiff-backup. I think that a good incremental backup would be your
best option.

It looks like all the stuff with making the hardlinks and temp directory are
to avoid a potential conflict between the existing "rdiff-backup-data"
directory on backup1 and the other "rdiff-backup-data" directory that gets
written to on backup2. If backup1 and backup2 both have rdiff-backup
installed then you can do something like

rdiff-backup backup1::/backup/files /backup/path/on/backup2 --exclude
**rdiff-backup-data**

on backup2. This avoids making hardlinks and a temp directory and also avoids
your problem of having the two "rdiff-backup-data" directories conflicting.

MM


--
To UNSUBSCRIBE, email to debian-user-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org

> If you are looking for a replacement, I don't know of any that do rdiffs
> besides rdiff-backup. I think that a good incremental backup would be your
> best option.

All incrementals (that I know of) waste space when there are large
files where only a small part of the file changes. This is a problem
for me - many of our users have 1-2 GB Outlook files. I don't want to
use up an extra 1GB+ of storage for those users every time I make a
backup.

>
> It looks like all the stuff with making the hardlinks and temp directory are
> to avoid a potential conflict between the existing "rdiff-backup-data"
> directory on backup1 and the other "rdiff-backup-data" directory that gets
> written to on backup2. If backup1 and backup2 both have rdiff-backup
> installed then you can do something like
>

The hardlinks and temp directories have nothing (specific) to do with
the interaction between backup1 and backup2. Their only purpose is to
allow me to combine rsync and rdiff-backup.

ie:

1) rsync from remote location to a local path (without breaking
rdiff-backup store)

2) tell rdiff-backup to pull from this local path into it's store, so
I get rdiff-backup history.

The reason for using hard links is to conserve space - if 'files' is
100GB, I don't want the 'temp' directory to also be 100GB. Also, it's
much faster to make hardlinks than to physically copy all the bytes
over from 'files' to 'temp'

I exclude rdiff-backup metadata when I sync from 'files' to 'new' for
a few reasons:

1) The source (where I am rsyncing from) won't (shouldn't) have it, so
it will get erased anyway when I run the rsync.

2) In the event that the source does have a rdiff-backup-data
directory for some reason, I need to exclude it anyway, because it
will cause problems with rdiff-backup.

I use the same method on backup2 (to backup backup1) because I have
the same needs as on backup1 (to backup all the other servers,
workstations, etc):

- I want history of the source
- I want to conserve space

The main difference is that backup2 contains the sum of all files on
backup1, plus their rdiff-backup metadata (in sub-dirs, not under the
root of backup1 where it would cause problems for backup2). In theory
there shouldn't be any major problem with this approach.

rdiff-backup's docs say that a large number of files shouldn't cause
excessive memory usage, but that hardlinks can - if you're hard
linking 1000's of files together.

This fact and our discussion gives me an idea: I think that
rdiff-backup *might* have a bug, where if the source and dest files
are all hardlinked to each other, it will use extra memory.

> rdiff-backup backup1::/backup/files /backup/path/on/backup2 --exclude
> **rdiff-backup-data**
>
> on backup2. This avoids making hardlinks and a temp directory and also avoids
> your problem of having the two "rdiff-backup-data" directories conflicting.
>

This is worth testing. If my non-standard usage (all those hardlinks
between source and dest files) is causing excessive memory usage on
backup2 (due to a huge number of files), this should fix the problem.
backup1 will also have this problem, but to a lesser degree (it has 90
odd backups, instead of 1 huge single backup).

I'll look into it.

(It will be a bit of a hack, because currently my backup logic is
split into 2 separate steps for each backup (regardless of type -
files, db, etc). First get the data from the source, then
compress/make history/etc. I'll have to add a new 'source' type for
'rdiff-backup', where there is no applicable 'compress' logic)

Thanks for your feedback :-)

David.


--
To UNSUBSCRIBE, email to debian-user-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org