Is this possible for pbzip2 to run on a tar which outputs to standard out?

I am trying to speed up a backup process which takes hours, i have
about 100g of uncompressed data which will be tar'd up and is
currently running thru gzip. This will be running on a 16 cpu box
which would greatly speedup this issue.

from:

tar cf - . | gzip > /tmp/file.tar.gz

to:

tar cf - . | pbzip2 > /tmp/file.tar.bz2


This doesnt seem to work, is it because there is no way to split the
stdio to multiple processors on the fly?

Thanks for your , I'm not subscribed to the list so if you could
cc: me in addition to the list that would be great.


Thanks,
Stefhen


--
To UNSUBSCRIBE, email to debian-user-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org

On Thu, Sep 27, 2007 at 02:57:40PM -0400, Stefhen Hovland wrote:
> Is this possible for pbzip2 to run on a tar which outputs to standard out?
>
> I am trying to speed up a backup process which takes hours, i have
> about 100g of uncompressed data which will be tar'd up and is
> currently running thru gzip. This will be running on a 16 cpu box
> which would greatly speedup this issue.
>
> from:
>
> tar cf - . | gzip > /tmp/file.tar.gz
>
> to:
>
> tar cf - . | pbzip2 > /tmp/file.tar.bz2
>
>
> This doesnt seem to work, is it because there is no way to split the
> stdio to multiple processors on the fly?

per

http://compression.ca/pbzip2/

ToDo

- Add support for input from stdin & pipes

google man!

A

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.6 (GNU/Linux)

iD8DBQFG/AW/aIeIEqwil4YRAk0WAJ4lN73r36HQYVzdsKZ2iTWyKYNIowCg4h bU
CXuksZN276h8UGrltcq5zlU=
=wwKe
-----END PGP SIGNATURE-----

Ahh crap, I looked thru that page, but didnt notice that. My fault

On 9/27/07, Andrew Sackville-West <andrew@farwestbilliards.com> wrote:
> On Thu, Sep 27, 2007 at 02:57:40PM -0400, Stefhen Hovland wrote:
> > Is this possible for pbzip2 to run on a tar which outputs to standard out?
> >
> > I am trying to speed up a backup process which takes hours, i have
> > about 100g of uncompressed data which will be tar'd up and is
> > currently running thru gzip. This will be running on a 16 cpu box
> > which would greatly speedup this issue.
> >
> > from:
> >
> > tar cf - . | gzip > /tmp/file.tar.gz
> >
> > to:
> >
> > tar cf - . | pbzip2 > /tmp/file.tar.bz2
> >
> >
> > This doesnt seem to work, is it because there is no way to split the
> > stdio to multiple processors on the fly?
>
> per
>
> http://compression.ca/pbzip2/
>
> ToDo
>
> - Add support for input from stdin & pipes
>
> google man!
>
> A
>
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v1.4.6 (GNU/Linux)
>
> iD8DBQFG/AW/aIeIEqwil4YRAk0WAJ4lN73r36HQYVzdsKZ2iTWyKYNIowCg4h bU
> CXuksZN276h8UGrltcq5zlU=
> =wwKe
> -----END PGP SIGNATURE-----
>
>


--
To UNSUBSCRIBE, email to debian-user-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org

On Fri, Sep 28, 2007 at 10:08:14AM -0400, Stefhen Hovland wrote:
>
> On 9/27/07, Andrew Sackville-West <andrew@farwestbilliards.com> wrote:
> > On Thu, Sep 27, 2007 at 02:57:40PM -0400, Stefhen Hovland wrote:
> > > Is this possible for pbzip2 to run on a tar which outputs to standard out?
> > >
> > > I am trying to speed up a backup process which takes hours, i have
> > > about 100g of uncompressed data which will be tar'd up and is
> > > currently running thru gzip. This will be running on a 16 cpu box
> > > which would greatly speedup this issue.
> > >
> > > from:
> > >
> > > tar cf - . | gzip > /tmp/file.tar.gz
> > >
> > > to:
> > >
> > > tar cf - . | pbzip2 > /tmp/file.tar.bz2
> > >
> > >
> > > This doesnt seem to work, is it because there is no way to split the
> > > stdio to multiple processors on the fly?
> >
> > per
> >
> > http://compression.ca/pbzip2/
> >
> > ToDo
> >
> > - Add support for input from stdin & pipes
> >
> > google man!
> >

> Ahh crap, I looked thru that page, but didnt notice that. My fault



my thought on this... I don't know how tar links in to bzip2 (which is
a compression option for tar (-j?). But maybe its possible to
substitute pbzip2 through something as simple as a symlink. Its a
stretch I know, but worth a shot. Also, out of curiousity, why not

tar -czf /tmp/file.tgz .

instead of the pipe? (unless you're actually piping through ssh or
something.)

anyway, since the idea above (symlink) will almopst certainly not
work, you'll probably have to sacrifice the storage space to tar it
all up into a file and then pbzip2 it separately. or hack the code,
but I'm willing to bet that parallelizing something coming from stdin
is non-trivial (I'm no coder...) and that's why its ToDo...

A

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.6 (GNU/Linux)

iD4DBQFG/UOyaIeIEqwil4YRAhr9AJ97zH3LFCXmrz8E45V0pfFxbM6kowC YtwMC
PKg9bOspmvfR5ZUgb+Oxag==
=eLd7
-----END PGP SIGNATURE-----

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 09/28/07 13:10, Andrew Sackville-West wrote:
[snip]
>
> my thought on this... I don't know how tar links in to bzip2 (which is
> a compression option for tar (-j?). But maybe its possible to
> substitute pbzip2 through something as simple as a symlink. Its a
> stretch I know, but worth a shot. Also, out of curiousity, why not


$ apt-cache show tar
[snip]
Pre-Depends: libc6 (>= 2.6-1)
Suggests: bzip2, ncompress <<<<<<<<<<<

$ apt-rdepends bzip2
Reading package lists... Done
Building dependency tree
Reading state information... Done
bzip2
Depends: libbz2-1.0 (= 1.0.3-7) <<<<<<<<<<<<<<<
libbz2-1.0
Depends: libc6 (>= 2.5-5)
libc6
Depends: libgcc1
libgcc1
Depends: gcc-4.2-base (= 4.2.1-5)
Depends: libc6 (>= 2.6-1)
gcc-4.2-base

- --
Ron Johnson, Jr.
Jefferson LA USA

Give a man a fish, and he eats for a day.
Hit him with a fish, and he goes away for good!

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.6 (GNU/Linux)

iD8DBQFG/VjLS9HxQb37XmcRApLtAKDSsvpbsfivmWPPa5lPz9oRBC8WWwC gvg4S
4vgsp9UqGKKsvviWfrbY+XY=
=lmR5
-----END PGP SIGNATURE-----


--
To UNSUBSCRIBE, email to debian-user-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org

On Fri, Sep 28, 2007 at 02:40:59PM -0500, Ron Johnson wrote:
> On 09/28/07 13:10, Andrew Sackville-West wrote:
> [snip]
> >
> > my thought on this... I don't know how tar links in to bzip2 (which is
> > a compression option for tar (-j?). But maybe its possible to
> > substitute pbzip2 through something as simple as a symlink. Its a
> > stretch I know, but worth a shot. Also, out of curiousity, why not
>
>
> $ apt-cache show tar
> [snip]
> Pre-Depends: libc6 (>= 2.6-1)
> Suggests: bzip2, ncompress <<<<<<<<<<<
>
> $ apt-rdepends bzip2
> Reading package lists... Done
> Building dependency tree
> Reading state information... Done
> bzip2
> Depends: libbz2-1.0 (= 1.0.3-7) <<<<<<<<<<<<<<<

well I said its a stretch... and I see that pbzip2 depends on the same
library, so it must just be a frontend that carves up the file into
chunks and sends it out to the different processors (I know naught of
these sorts of mechanisms). IOW, some serious hackery would be
involved to get it to do what he wants...

A

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.6 (GNU/Linux)

iD8DBQFG/VuYaIeIEqwil4YRAvYRAKDJ6qGAAIM37RFz3hWNHdkueGtcMAC dENiT
YwIqb8cFjnFxTjod3W6oQYM=
=OC5h
-----END PGP SIGNATURE-----

On Thu, Sep 27, 2007 at 02:57:40PM -0400, Stefhen Hovland wrote:
> Is this possible for pbzip2 to run on a tar which outputs to standard out?
>
> I am trying to speed up a backup process which takes hours, i have
> about 100g of uncompressed data which will be tar'd up and is
> currently running thru gzip. This will be running on a 16 cpu box
> which would greatly speedup this issue.
>
> from:
>
> tar cf - . | gzip > /tmp/file.tar.gz
>
> to:
>
> tar cf - . | pbzip2 > /tmp/file.tar.bz2
>
>
> This doesnt seem to work, is it because there is no way to split the
> stdio to multiple processors on the fly?

I wonder if it would to create a named-pipe (fifo). Would that
trick the compressor (whichever) to treat it as a file (that doesn't
have an EOF until the true end) and run on a different processor?

Never having had more than one core, I can't test this.

Doug.


--
To UNSUBSCRIBE, email to debian-user-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org