Performance of virtualized file server is horrible. I've tried everything I
can think of. Anybody out there have any ideas?

Here's my configuration.

Server cluster, three identical HP Proliant DL380 G4 servers:
2 3.6GHZ 1M processors
12GB RAM
2 72GB SCSI drives for host OS
Dual integrated gigabit NICs
Intel Dual-Port PRO/1000MT Server Adapter
HP NC1020 Gigabit Server Adapter
Alacritech SE2002XT Dual-Port iSCSI Accelerator
Windows Server 2003 Enterprise x64 Edition SP2
Virtual Server 2005 R2 Enterprise SP1

iSCSI SAN
2 Netgear GS516 gigabit switches
Promise m300i RAID Storage Array
2 WD 7200 500GB SATA
10 WD 10K 150GB SATA
Promise m310i RAID Storage Array
2 WD 10K 150GB SATA

I've already tried turning off tcp offloading. I isolated the file server
by moving all of the other virtual machines to other cluster nodes, but that
didn't have any noticable impact on performance. Disk read performance in
the VM is not great, 33MB/sec. Read performance over the network from a
high-end workstation (with a gigabit NIC) with nobody else on the network
drops to 11MB/sec. Performance between virtual machines on the same host is
roughly the same--even using the MS loopback adapter. Disk read performance
on the host is great--even from the iSCSI SAN: I transferred 3GB off a SAN
disk onto the host disk in about 5 seconds.

I have hyperthreading turned on. Does anybody out there think that might be
a problem?

Hiya,

First of all regarding hyperthreading, read this:
http://support.microsoft.com/default...867586?lc=2057

Secondly read this:
http://www.microsoft.com/communities...4-0d0406fd60ac

And third note that the guest systems can only emulate a 10/100 network
card. (which is usually good enough, as the host will run 1000, and that way
one guest system wont eat up the entire network. - it's a balance of good and
bad.
I cant see from your setup where you have your virtual files located either?

I hope it s you getting a conclusion on your problem, else ask again

On Sun, 17 Feb 2008 23:16:00 -0800, Jesper Arnecke
<JesperArnecke@discussions.microsoft.com> wrote:


>And third note that the guest systems can only emulate a 10/100 network
>card.

Which isn't a limit in itself. The only limit is the actual host's
connection, and the CPU overhead. A VM can get up to about 500Mb
throughput if the host is robust enough.


--
Cheers,
Steve Jain, Virtual Machine MVP
http://vpc.essjae.com/
I do not work for Microsoft.

"Steve Jain" wrote:
> Which isn't a limit in itself. The only limit is the actual host's
> connection, and the CPU overhead. A VM can get up to about 500Mb
> throughput if the host is robust enough.

Hey Steve,

I am sure when you say it, it holds truth. But to my head it doesnt really
makes sense that a specific 10/100 NIC can deliver more, if located in a vm.
As far as I understand, what happends is that it will emulate a specific
piece of hardware, leaving the boundries of a vm within that piece of
hardware...
I would very much like to read more on this overclocking of a NIC, as my
knowledge ends about the 100

The number that appears in the vm is just that- a number. Under Hyper-V
it is 10Gb (probably to avoid discussions like this) !

The rate at which the data is actually transferred depends on the NIC in
the host (and its driver) and the network to which it is attached, not to
anything in the vm.

"Jesper Arnecke" <JesperArnecke@discussions.microsoft.com> wrote in message
news:107D81E7-8CC7-4733-BEB0-FC8D00751C9C@microsoft.com...
> "Steve Jain" wrote:
>> Which isn't a limit in itself. The only limit is the actual host's
>> connection, and the CPU overhead. A VM can get up to about 500Mb
>> throughput if the host is robust enough.
>
> Hey Steve,
>
> I am sure when you say it, it holds truth. But to my head it doesnt really
> makes sense that a specific 10/100 NIC can deliver more, if located in a
> vm.
> As far as I understand, what happends is that it will emulate a specific
> piece of hardware, leaving the boundries of a vm within that piece of
> hardware...
> I would very much like to read more on this overclocking of a NIC, as my
> knowledge ends about the 100

"Jesper Arnecke" wrote:

> Hiya,
>
> First of all regarding hyperthreading, read this:
> http://support.microsoft.com/default...867586?lc=2057
>
> Secondly read this:
> http://www.microsoft.com/communities...4-0d0406fd60ac
>
> And third note that the guest systems can only emulate a 10/100 network
> card. (which is usually good enough, as the host will run 1000, and that way
> one guest system wont eat up the entire network. - it's a balance of good and
> bad.
> I cant see from your setup where you have your virtual files located either?
>
> I hope it s you getting a conclusion on your problem, else ask again

The virtual hard disks are fixed size SCSI, located on the iSCSI SAN, and
have to be in order for the VMs to be clustered. But the system isn't disk
bound, it's network bound. When I run a process directly on the VM, I get
about 33MB per second throughput. When I run the same process across the
network I get about 11MB per second throughput--despite there being gigabit
NICs and no other network traffic except between the high-end workstation and
the VM. That just doesn't make any sense. I've tried dedicating a 1GB
NIC--and it doesn't matter whether I use one of the embedded HP NICS or one
of the Intel NICS, or even the Microsoft Loopback adapter. The performance
is still roughly 1/3 of what in my opinion it should be. Something is screwy.

I tried disabling hyperthreading, but it didn't have any effect. The system
isn't by any means CPU bound.

"Bill Grant" wrote:
> The number that appears in the vm is just that- a number. Under Hyper-V
> it is 10Gb (probably to avoid discussions like this) !
>
> The rate at which the data is actually transferred depends on the NIC in
> the host (and its driver) and the network to which it is attached, not to
> anything in the vm.

Bill,
What your saying is that the virtual guest will run directly on the hardware
provided by the host. As this is almost the case with 2008 Hyper-V, it is not
the case for VS 2005.
Please, can you provide any MS documentation, white papers, test scenariors
etc. that can confirm what your saying?
We are facing similar issues the ones described earlier in this thread, so
anything to narrow it down would be high appreciated..

> The virtual hard disks are fixed size SCSI, located on the iSCSI SAN, and
> have to be in order for the VMs to be clustered. But the system isn't disk
> bound, it's network bound. When I run a process directly on the VM, I get
> about 33MB per second throughput. When I run the same process across the
> network I get about 11MB per second throughput--despite there being gigabit
> NICs and no other network traffic except between the high-end workstation and
> the VM. That just doesn't make any sense. I've tried dedicating a 1GB
> NIC--and it doesn't matter whether I use one of the embedded HP NICS or one
> of the Intel NICS, or even the Microsoft Loopback adapter. The performance
> is still roughly 1/3 of what in my opinion it should be. Something is screwy.
>
> I tried disabling hyperthreading, but it didn't have any effect. The system
> isn't by any means CPU bound.

If my assumption is correct, and the virtual guest only emulates a 10/100
NIC, then your max throughput will be around 12MB/sec.
So having a throughput of 11MB/sec is expected. there is a difference
between 100Mbit and 100Mbyte.
100Mbit per sec / 8 = speed in Mbyte per sec.- Approximately.

If possible, try and test your process from the host machine, to see if it
will perform accordingly. 1000Mbit per second.

Being about 89% on this part, I would still recommend that you check up on
it

"Jesper Arnecke" <JesperArnecke@discussions.microsoft.com> wrote in message
news:7F3EF2A1-28BB-4841-B788-66BA74C2DA46@microsoft.com...
> "Bill Grant" wrote:
>> The number that appears in the vm is just that- a number. Under
>> Hyper-V
>> it is 10Gb (probably to avoid discussions like this) !
>>
>> The rate at which the data is actually transferred depends on the NIC
>> in
>> the host (and its driver) and the network to which it is attached, not to
>> anything in the vm.
>
> Bill,
> What your saying is that the virtual guest will run directly on the
> hardware
> provided by the host. As this is almost the case with 2008 Hyper-V, it is
> not
> the case for VS 2005.
> Please, can you provide any MS documentation, white papers, test
> scenariors
> etc. that can confirm what your saying?
> We are facing similar issues the ones described earlier in this thread, so
> anything to narrow it down would be high appreciated..

What do you believe happens with VS2005? The data coming over a network
can only be handled by a physical device connected to the network. That is
your NIC. Virtual Machine Network Services works with the NIC driver to
split off traffic for the virtual machine (and delivers packets to the NIC
driver to send in the other direction).
All of this is controlled by the physical NIC, not by anything inside the
vm.

See this article in Ben's blog.

http://blogs.msdn.com/virtual_pc_guy...01/404816.aspx

"Bill Grant" wrote:
> What do you believe happens with VS2005? The data coming over a network
> can only be handled by a physical device connected to the network. That is
> your NIC. Virtual Machine Network Services works with the NIC driver to
> split off traffic for the virtual machine (and delivers packets to the NIC
> driver to send in the other direction).
> All of this is controlled by the physical NIC, not by anything inside the
> vm.
> See this article in Ben's blog.
> http://blogs.msdn.com/virtual_pc_guy...01/404816.aspx

Hey Bill,

First of all, thank you for your patience and assistance in inlighting me
Secondly, what he describes is exactly what I am saying, at least in my head.
Server A -> Physical NIC -> 'Virtual Machine Network Services' -> Emulated
NIC -> Server B
Considering the above, they emulated NIC is running 10/100, as per 2005
design, so no matter how fast your Physical NIC, you will not get speeds of
more than 100Mbit.
Unless it's because this emulation is not "true", and thus allowing the VM
NS to actually be the one controlling the speed(which is dependant on the
physical NIC), however that sounds a bit far fetched.

If I am still off in the understanding, I'll quit it and buy you a beer if I
ever bump into you ; )

"Jesper Arnecke" <JesperArnecke@discussions.microsoft.com> wrote in message
news:67581C89-CBE9-4DEB-832C-D59DC9C9FE15@microsoft.com...
>
> "Bill Grant" wrote:
>> What do you believe happens with VS2005? The data coming over a network
>> can only be handled by a physical device connected to the network. That
>> is
>> your NIC. Virtual Machine Network Services works with the NIC driver to
>> split off traffic for the virtual machine (and delivers packets to the
>> NIC
>> driver to send in the other direction).
>> All of this is controlled by the physical NIC, not by anything inside the
>> vm.
>> See this article in Ben's blog.
>> http://blogs.msdn.com/virtual_pc_guy...01/404816.aspx
>
> Hey Bill,
>
> First of all, thank you for your patience and assistance in inlighting me
>
> Secondly, what he describes is exactly what I am saying, at least in my
> head.
> Server A -> Physical NIC -> 'Virtual Machine Network Services' -> Emulated
> NIC -> Server B
> Considering the above, they emulated NIC is running 10/100, as per 2005
> design, so no matter how fast your Physical NIC, you will not get speeds
> of
> more than 100Mbit.
> Unless it's because this emulation is not "true", and thus allowing the VM
> NS to actually be the one controlling the speed(which is dependant on the
> physical NIC), however that sounds a bit far fetched.
>
> If I am still off in the understanding, I'll quit it and buy you a beer if
> I
> ever bump into you ; )


No, there is no speed limit set inside the vm. If the physical NIC is fast
enough it can run at four to six times faster than a 10/100 device, as I
think Steve Jain said much earlier in this thread.

> The virtual hard disks are fixed size SCSI, located on the iSCSI SAN, and
> have to be in order for the VMs to be clustered. But the system isn't
> disk
> bound, it's network bound. When I run a process directly on the VM, I get
> about 33MB per second throughput. When I run the same process across the
> network I get about 11MB per second throughput--despite there being
> gigabit
> NICs and no other network traffic except between the high-end workstation
> and
> the VM. That just doesn't make any sense. I've tried dedicating a 1GB
> NIC--and it doesn't matter whether I use one of the embedded HP NICS or
> one
> of the Intel NICS, or even the Microsoft Loopback adapter. The
> performance
> is still roughly 1/3 of what in my opinion it should be. Something is
> screwy.
>
> I tried disabling hyperthreading, but it didn't have any effect. The
> system
> isn't by any means CPU bound.

You say you're not CPU-bound, but have you measured processor activity
during heavy I/O? If you're using the Microsoft iSCSI initiator, you might
be surprised at how much of a hit you take. This white paper
[http://www.sanstor.info/5iSCSI%20sof...tors%20vs.pdf] implies
that a software initiatior "may consume up to 500MHz of CPU for 1 Gigabit
Ethernet line rate" - which is a heavy hit on a virtual machine that can
only utilize host one processor.

Almost all of my production servers are VS2003R2 guests, using the MS iSCSI
initiator to connect to EMC arrays, and I've had less-than-stellar
throughput for machines with heavy I/O.

Plus the emulated Intel/DEC NIC in the VM can't do TCP offloading or take
advantage of jumbo frames. I really wish Microsoft had offered an emulated
Gigabit NIC in R2 as an optional addition so we could have used jumbos -
even though it might not have been "faster" in terms of wire speed (and by
the way, I'm very skeptical of the claim that the emulated NIC can run 'four
to six times faster' than 100Mbps in real-world scenarios. I'd buy it when a
guest is talking to another guest on the same host, but I've never seen that
kind of performance when the guest is accessing a physical LAN.)

I will be very happy when I can move my production systems to Veridian.


Chuck