Preparing to buy a new server...

What is debian's opinion about hardware/software raid?

I have the feeling that software raid 1 is more reliable since if
anything goes wrong with the hardware, I could just take one or two
disks out of the server and put them into a workstation to get up and
running or copy data of the disks for the time until the hardware gets
fixed.

On the other hand, performance will probably be better with a dedicated
hardware raid controller.

What are your experiences?

Thanks,
Johannes

NB: we are talking about ibm server sxeries 226, raid controller sata
raid 5 vs software raid 1 using mdadm.
I would either use 3x250 GB + raid 5 or 4x250 GB + raid 1


--
To UNSUBSCRIBE, email to debian-user-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org

On 30.11.06 15:44, Johannes Wiedersich wrote:
> Preparing to buy a new server...
>
> What is debian's opinion about hardware/software raid?
>
> I have the feeling that software raid 1 is more reliable since if
> anything goes wrong with the hardware, I could just take one or two
> disks out of the server and put them into a workstation to get up and
> running or copy data of the disks for the time until the hardware gets
> fixed.

it highly depends on the hardware raid. The true hardware RAID with
hotspare, hotswap etc. support and with proper drivers (so you won't have to
boot into its bios to reconfigure it) will be much more reliable.

> On the other hand, performance will probably be better with a dedicated
> hardware raid controller.

it highly depends on the hardware raid. The true hardware RAID with XOR
counting and mirroring support will be much faster.

--
Matus UHLAR - fantomas, uhlar@fantomas.sk ; http://www.fantomas.sk/
Warning: I wish NOT to receive e-mail advertising to this address.
Varovanie: na tuto adresu chcem NEDOSTAVAT akukolvek reklamnu postu.
Save the whales. Collect the whole set.


--
To UNSUBSCRIBE, email to debian-user-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org

On Thursday, 30.11.2006 at 16:56 +0100, Matus UHLAR - fantomas wrote:

> On 30.11.06 15:44, Johannes Wiedersich wrote:
> > Preparing to buy a new server...
> >
> > What is debian's opinion about hardware/software raid?
> >
> > I have the feeling that software raid 1 is more reliable since if
> > anything goes wrong with the hardware, I could just take one or two
> > disks out of the server and put them into a workstation to get up and
> > running or copy data of the disks for the time until the hardware gets
> > fixed.
>
> it highly depends on the hardware raid. The true hardware RAID with
> hotspare, hotswap etc. support and with proper drivers (so you won't haveto
> boot into its bios to reconfigure it) will be much more reliable.
>
> > On the other hand, performance will probably be better with a dedicated
> > hardware raid controller.
>
> it highly depends on the hardware raid. The true hardware RAID with XOR
> counting and mirroring support will be much faster.

Although it's worth pointing out that software RAID-*1* (one of the
options under consideration) has almost no CPU overhead, and is often a
good low-cost option.

Part of your decision must rest on what exactly the machine will be
doing. Different RAID setups are best suited for different usage
patterns, e.g. RAID-5 is often a good general-purpose storage server
option, RAID-10 is usually recommended for database servers, etc.

Dave.
--
Please don't CC me on list messages!
...
Dave Ewart - davee@sungate.co.uk - jabber: davee@jabber.org
All email from me is now digitally signed, key from http://www.sungate.co.uk/
Fingerprint: AEC5 9360 0A35 7F66 66E9 82E4 9E10 6769 CD28 DA92


-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.2.2 (GNU/Linux)

iD8DBQFFbwBhnhBnac0o2pIRAtgrAJ44yNj+eeq0VGD8kEv8RZ HqncfGBgCdHt+3
Jp0XmhQ2khvb+Hr/DC8uNcU=
=NFlF
-----END PGP SIGNATURE-----

Dave Ewart wrote:
> Although it's worth pointing out that software RAID-*1* (one of the
> options under consideration) has almost no CPU overhead, and is often a
> good low-cost option.
>
> Part of your decision must rest on what exactly the machine will be
> doing. Different RAID setups are best suited for different usage
> patterns, e.g. RAID-5 is often a good general-purpose storage server
> option, RAID-10 is usually recommended for database servers, etc.

It's a 'general purpose' data server for a group of ca. 20 people,
serving linux-homes and as a samba domain controller where all the data go.

At the moment we have two servers for different kind of data, but the
idea is to have just one server with all data. The hardware has
integrated Raid-1, Raid 5 costs extra.

My concern is that we are going to have only one server. So if there was
a hardware problem, with software raid I could just temporarily move the
disks to an ordinary workstation and serve the data from there. With
hardware raid, I suppose I would have to wait for replacement parts.

Johannes


--
To UNSUBSCRIBE, email to debian-user-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org

On Thu, 30 Nov 2006 15:44:38 +0100, Johannes Wiedersich wrote
> Preparing to buy a new server...
>
> What is debian's opinion about hardware/software raid?
>
> I have the feeling that software raid 1 is more reliable since if
> anything goes wrong with the hardware, I could just take one or two
> disks out of the server and put them into a workstation to get up and
> running or copy data of the disks for the time until the hardware
> gets fixed.
>
> On the other hand, performance will probably be better with a dedicated
> hardware raid controller.
>
> What are your experiences?

I'm a huge fan of software raid.
I think reliably, they are probably the same.

I prefer SW raid for its lower cost, and its ability to warn you of a failing
disk via email. HW raid will only yell at you if somethings wrong, (maybe expensive HW raid
has email features, but I'm not aware of any)

I find the mdadm tool very good at customizing your arrays too. Lots of flexability.

For performance, I'm not so sure any more that HW raid is faster. Even if the server is under load.
>From some basic bonnie++ tests on an LSI Raid controller compared to SW raid on a 2 x Xeon
system running Sarge, the SW raid 5 was faster. Sorry I don't have any actual stats.
and others may have stats that show different.

If possible, you could try and benchmark the system before you decide, but I don't know if you can
"borrow" a raid controller to perform your tests. Maybe your vendor will let you return it if you don't
like the results.

Another problem however, is finding a MB with lots of disk controllers. You can easily get Raid
controllers with 8 or more sata ports, but server boards usually have 2 or 4. Although recently, you
can get intel boards with 6 on board Sata controllers. Depends on your setup I guess.

Further, I'm more in favor of Raid 10 systems, for systems that do lots of different tasks.
Raid 5 begins to really slow down when your system is under stress.
There's tons of comparisons of which raid to use on google. You get to decide which one is best for
you.

Cheers,
Mike





--
To UNSUBSCRIBE, email to debian-user-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org

On Thu, 2006-11-30 at 17:20 +0100, Johannes Wiedersich wrote:
> My concern is that we are going to have only one server. So if there was
> a hardware problem, with software raid I could just temporarily move the
> disks to an ordinary workstation and serve the data from there. With
> hardware raid, I suppose I would have to wait for replacement parts.

If you do the following:

1. Take a snapshot of the system after installing and configuring
everything.

2. Make good backups of /home and (for Samba /etc/samba)

then you can get back up and running en very little time.

I take periodic snapshots of servers when I get a chance to restart them
(like after a kernel update scheduled for after hours. Takes me a
couple of minutes to make a tarball of the / partition and regular
bakcups take care of /home and databases. Thanks to the kind of
unstable power around here that can blow a UPS and everything on it in
one go, I've had to recover whole servers a few times. Above recipy
works every time.

As for software v.s. hardware, for a simple raid-1 I find software good
enough but only if cost is really a huge factor. If I do Raid-5 I
insist on a raid card. Adaptec 2410SA PCI SATA (and the SCSI
equivalent) has served me very well.

Hans


--
To UNSUBSCRIBE, email to debian-user-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org

> > On 30.11.06 15:44, Johannes Wiedersich wrote:
> > > What is debian's opinion about hardware/software raid?

> On Thursday, 30.11.2006 at 16:56 +0100, Matus UHLAR - fantomas wrote:
> > it highly depends on the hardware raid. The true hardware RAID with
> > hotspare, hotswap etc. support and with proper drivers (so you won't have to
> > boot into its bios to reconfigure it) will be much more reliable.
> >
> > > On the other hand, performance will probably be better with a dedicated
> > > hardware raid controller.
> >
> > it highly depends on the hardware raid. The true hardware RAID with XOR
> > counting and mirroring support will be much faster.

On 30.11.06 16:01, Dave Ewart wrote:
> Although it's worth pointing out that software RAID-*1* (one of the
> options under consideration) has almost no CPU overhead, and is often a
> good low-cost option.

I did not want to mention this, because:

- software RAID-1 has bus overhead (the same data have to be transferred
multiple times to multiple drives, so writing may be twice as slow)

- software RAID-1 has high CPU and bus overhead in case of resyncing after
disk outage (hw raid does that itself)

--
Matus UHLAR - fantomas, uhlar@fantomas.sk ; http://www.fantomas.sk/
Warning: I wish NOT to receive e-mail advertising to this address.
Varovanie: na tuto adresu chcem NEDOSTAVAT akukolvek reklamnu postu.
A day without sunshine is like, night.


--
To UNSUBSCRIBE, email to debian-user-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org

> On Thu, Nov 30, 2006 at 10:53:26PM +0100, Matus UHLAR - fantomas wrote:
> >
> > I did not want to mention this, because:
> >
> > - software RAID-1 has bus overhead (the same data have to be transferred
> > multiple times to multiple drives, so writing may be twice as slow)

On 30.11.06 17:00, Roberto C. Sanchez wrote:
> This is only true if you use IDE and also put the mirrored pair on the
> same channel (which would be incredibly foolish). Under every other
> conceivable scenario, the system is smart enough to do both writes
> simultaneously since they will be going difference places.

but the data will still be fetched two times from memory to a controller(s)

> > - software RAID-1 has high CPU and bus overhead in case of resyncing after
> > disk outage (hw raid does that itself)
>
> Huh? I have built lots of RAID one systems and seen many of them
> resync. They do have high I/O overhead (naturally, since it is a
> resync). However, I have never seen one produce high CPU overload.

I should write "higher CPU overload"

> I suppose if you forgot to enable DMA on your drives, or somehow managed
> to disable it (it is enabled by default), then you might see high CPU load
> during resync. But then, you would see high CPU load for anything that
> does any significant amount of I/O.

--
Matus UHLAR - fantomas, uhlar@fantomas.sk ; http://www.fantomas.sk/
Warning: I wish NOT to receive e-mail advertising to this address.
Varovanie: na tuto adresu chcem NEDOSTAVAT akukolvek reklamnu postu.
I just got lost in thought. It was unfamiliar territory.


--
To UNSUBSCRIBE, email to debian-user-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org

> Dave Ewart wrote:
> > Although it's worth pointing out that software RAID-*1* (one of the
> > options under consideration) has almost no CPU overhead, and is often a
> > good low-cost option.
> >
> > Part of your decision must rest on what exactly the machine will be
> > doing. Different RAID setups are best suited for different usage
> > patterns, e.g. RAID-5 is often a good general-purpose storage server
> > option, RAID-10 is usually recommended for database servers, etc.

On 30.11.06 17:20, Johannes Wiedersich wrote:
> It's a 'general purpose' data server for a group of ca. 20 people,
> serving linux-homes and as a samba domain controller where all the data go.
>
> At the moment we have two servers for different kind of data, but the
> idea is to have just one server with all data. The hardware has
> integrated Raid-1, Raid 5 costs extra.
>
> My concern is that we are going to have only one server. So if there was
> a hardware problem, with software raid I could just temporarily move the
> disks to an ordinary workstation and serve the data from there. With
> hardware raid, I suppose I would have to wait for replacement parts.

In such case, you should search for your RAID documentation, if it's real HW
RAID at all.

--
Matus UHLAR - fantomas, uhlar@fantomas.sk ; http://www.fantomas.sk/
Warning: I wish NOT to receive e-mail advertising to this address.
Varovanie: na tuto adresu chcem NEDOSTAVAT akukolvek reklamnu postu.
On the other hand, you have different fingers.


--
To UNSUBSCRIBE, email to debian-user-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org

On Thu, Nov 30, 2006 at 10:53:26PM +0100, Matus UHLAR - fantomas wrote:
>
> I did not want to mention this, because:
>
> - software RAID-1 has bus overhead (the same data have to be transferred
> multiple times to multiple drives, so writing may be twice as slow)
>
This is only true if you use IDE and also put the mirrored pair on the
same channel (which would be incredibly foolish). Under every other
conceivable scenario, the system is smart enough to do both writes
simultaneously since they will be going difference places.

> - software RAID-1 has high CPU and bus overhead in case of resyncing after
> disk outage (hw raid does that itself)
>

Huh? I have built lots of RAID one systems and seen many of them
resync. They do have high I/O overhead (naturally, since it is a
resync). However, I have never seen one produce high CPU overload. I
suppose if you forgot to enable DMA on your drives, or somehow managed
to disable it (it is enabled by default), then you might see high CPU
load during resync. But then, you would see high CPU load for anything
that does any significant amount of I/O.

Regards,

-Roberto
--
Roberto C. Sanchez
http://people.connexer.com/~roberto
http://www.connexer.com

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.1 (GNU/Linux)

iD8DBQFFb1Rl5SXWIKfIlGQRAvTPAKDA1A7Ep8b6j9CfR7HA6e JSxVyWJQCg0ve0
if3L8xZr5bXfHQEQK10Nbw0=
=E/er
-----END PGP SIGNATURE-----

On Thursday, 30.11.2006 at 22:53 +0100, Matus UHLAR - fantomas wrote:

> > > On 30.11.06 15:44, Johannes Wiedersich wrote:
> > > > What is debian's opinion about hardware/software raid?
>
> > On Thursday, 30.11.2006 at 16:56 +0100, Matus UHLAR - fantomas wrote:
> > > it highly depends on the hardware raid. The true hardware RAID with
> > > hotspare, hotswap etc. support and with proper drivers (so you won't have to
> > > boot into its bios to reconfigure it) will be much more reliable.
> > >
> > > > On the other hand, performance will probably be better with a dedicated
> > > > hardware raid controller.
> > >
> > > it highly depends on the hardware raid. The true hardware RAID with XOR
> > > counting and mirroring support will be much faster.
>
> On 30.11.06 16:01, Dave Ewart wrote:
> > Although it's worth pointing out that software RAID-*1* (one of the
> > options under consideration) has almost no CPU overhead, and is often a
> > good low-cost option.
>
> I did not want to mention this, because:
>
> - software RAID-1 has bus overhead (the same data have to be transferred
> multiple times to multiple drives, so writing may be twice as slow)

That's why I also said "Part of your decision must rest on what exactly
the machine will be doing" since one may or may not care about the extra
disk I/O in some environments. Equally, it could be completely
inappropriate in other environments.

As always, "it depends" :-)

Dave.

--
Please don't CC me on list messages!
...
Dave Ewart - davee@sungate.co.uk - jabber: davee@jabber.org
All email from me is now digitally signed, key from http://www.sungate.co.uk/
Fingerprint: AEC5 9360 0A35 7F66 66E9 82E4 9E10 6769 CD28 DA92


-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.2.2 (GNU/Linux)

iD8DBQFFcCXLnhBnac0o2pIRAqT2AKC68v/AnVwZV5tX06Wdc/20RlY6ZQCg8fqk
SR75xzt1r9+eKHRarND2AQI=
=uZmS
-----END PGP SIGNATURE-----

On Fri, Dec 01, 2006 at 09:19:08AM +0100, Matus UHLAR - fantomas wrote:
>
> On 30.11.06 17:00, Roberto C. Sanchez wrote:
> > This is only true if you use IDE and also put the mirrored pair on the
> > same channel (which would be incredibly foolish). Under every other
> > conceivable scenario, the system is smart enough to do both writes
> > simultaneously since they will be going difference places.
>
> but the data will still be fetched two times from memory to a controller(s)

I don't understand why this matters. The miracle of DMA makes it so
that this happens without CPU intervention. The fact that you can put
the drives on separate channels makes it so that the operations can
happen in parallel. In practice, RAID-1 performance is the same as
non-RAID for writing and up to twice as fast for reading.

>
> > > - software RAID-1 has high CPU and bus overhead in case of resyncing after
> > > disk outage (hw raid does that itself)
> >
> > Huh? I have built lots of RAID one systems and seen many of them
> > resync. They do have high I/O overhead (naturally, since it is a
> > resync). However, I have never seen one produce high CPU overload.
>
> I should write "higher CPU overload"
>
I should have written "high CPU load." Basically, the CPU only gets
loaded if you have DMA disabled.

> > I suppose if you forgot to enable DMA on your drives, or somehow managed
> > to disable it (it is enabled by default), then you might see high CPU load
> > during resync. But then, you would see high CPU load for anything that
> > does any significant amount of I/O.
>

Regards,

-Roberto

--
Roberto C. Sanchez
http://people.connexer.com/~roberto
http://www.connexer.com


--
To UNSUBSCRIBE, email to debian-user-REQUEST@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org