[aklug] Re: Reasonable average server load?

Arthur Corliss wrote:
> On Wed, 23 Jun 2010, adam bultman wrote:
>
>
>> Especially if the Virtualization Host you're on is making silly
>> decisions (like loading your working RAM into it's swap - I've seen it
>> happen) , or if your server's filesystem is on an high-latency platform
>> (example: saturated NFS mount or busy filesystem.)
>>
>> Your load will be high, your server will be extremely slow, but there
>> won't be 'anything' (even with a vmstat, and iostat, a sar, or a 'top')
>> that leads you to believe there's a problem. It'll just be pokey.
>>
>
> That's not true. vmstat and iostat would indicate high io-wait states, as
> would SAR. A full SAR report would even give you your NFS statistics,
> should that be the issue. If you're working from swap you'll see that in
> vmstat as well.
>
>
In my case, the ESX host decided to put the guest's RAM into swap, so
the guest had no idea it's active memory was swapped out. I sent the
results of just about everything on the guest (sar, vmstat, iostat, top,
ps -efww) to engineers at a software vendor, and they could find no more
than I could. I even made a cron job that ran any time the load went
about 2 to send me an email with the results. Never found anything;
sure did get some awesome eyestrain though.

(And I suppose my NFS comment should have said, "saturated NFS mount or
busy filesystem on the host". SAR on the guest won't tell you about
NFS statistics if the NFS mount is on the host. You'll know that it is
spending a lot of time waiting on I/O, but not particularly the "why". )

Only logging into the ESX server and checking out a particular graph for
the VM clued me in to the fact that ESX had decided to swap out the RAM
of the only VM it was running (and, I might add, needed only half the
RAM that the ESX server had). I have a munin plugin for that sucker
now, so I'll know pretty quickly if it happens again.

That was a real bear of a problem to find, but an easy one to fix.

-- 
Adam
---------
To unsubscribe, send email to <aklug-request@aklug.org>
with 'unsubscribe' in the message body.