how to troubleshoot a config that works fine for a while then suddenly slowwwer then just...stop?

From: Thomison, Lee <ThomisonL@ci.anchorage.ak.us>
Date: Tue Jun 20 2006 - 15:21:33 AKDT

I'm using IBM Netvista blackboxes. No choice, company standard,
blahblah.
Very often, several times a week I'll be working merrily along in
firefox or openoffice or konsole or whatever, then the audio will
stutter and stop. Typing slows and stops, and the clock down in the slit
is frozen. If I leave it for a few minutes, the clock may advance a few
seconds (or redisplay the right time), a couple of letters may appear
where I type, the audio may stutter a bit, but finally, response is just
nonexistent. The mouse moves, but will not select windows or highlight.

Three finger salute will start the halt process, but never get there

I can ping a stuck box from other boxes, and the pingtimes seem
reasonable. I can often (but not always) ssh into the stuck box and type
halt, but the box will start the shutdown process and never finish.
After about 5-10 minutes I just powercycle it. One evening I did leave
it on through the night to see if it would finally halt. It didn't.

This happens on the three netvistas I've got gentoo installed on (the
rest are XP, and I've not seen this on those boxes), so I don't think
it's a hardware issue.

It's almost as if the system clock just slows dowwwnnnnnn....stutters a
bit, and stops.

Except the mouse will move on the screen.

They've done this for at least a year, and I think longer, and I've just
lived with it, but the time is coming that these machines will have to
stay up. It's been this way for several kernel revs, and at least
through a couple of gentoo revs.

I've got ACPI disabled.

I've tried a couple of different window managers (fluxbox, icewm, even
kde3)

If I was a betting man, I'd say I've got something misconfigured
somewhere.

My question is how do I begin to troubleshoot this. Nothing shows up in
/var/log/messages, and there are no kernel barfs that I've found.

Thanks!

Particulars:

~ # emerge --info
Portage 2.1 (default-linux/x86/2005.1, gcc-hardened, glibc-2.3.6-r3,
2.6.16-gentoo-r9 i686)
=================================================================
System uname: 2.6.16-gentoo-r9 i686 Intel(R) Pentium(R) 4 CPU 2.53GHz
Gentoo Base System version 1.12.1
ccache version 2.3 [disabled]
dev-lang/python: 2.3.5-r2, 2.4.2
dev-python/pycrypto: 2.0.1-r5
dev-util/ccache: 2.3
dev-util/confcache: [Not Present]
sys-apps/sandbox: 1.2.17
sys-devel/autoconf: 2.13, 2.59-r7
sys-devel/automake: 1.4_p6, 1.5, 1.6.3, 1.7.9-r1, 1.8.5-r3, 1.9.6-r1
sys-devel/binutils: 2.16.1-r2
sys-devel/gcc-config: 1.3.13-r2
sys-devel/libtool: 1.5.22
virtual/os-headers: 2.6.11-r2
ACCEPT_KEYWORDS="x86"
AUTOCLEAN="yes"
CBUILD="i686-pc-linux-gnu"
CFLAGS="-march=pentium4 -mtune=pentium4 -pipe -O3 -msse2 -fweb
-frename-registers -fforce-addr -momit-leaf-frame-pointer
-fomit-frame-pointer -ftracer -mno-tls-direct-seg-refs "
CHOST="i686-pc-linux-gnu"
CONFIG_PROTECT="/etc /usr/kde/3.3/env /usr/kde/3.3/share/config
/usr/kde/3.3/shutdown /usr/kde/3.4/env /usr/kde/3.4/share/config
/usr/kde/3.4/shutdown /usr/kde/3.5/env /usr/kde/3.5/share/config
/usr/kde/3.5/shutdown /usr/lib/X11/xkb /usr/lib/mozilla/defaults/pref
/usr/share/config /usr/share/texmf/dvipdfm/config/
/usr/share/texmf/dvips/config/ /usr/share/texmf/tex/generic/config/
/usr/share/texmf/tex/platex/config/ /usr/share/texmf/xdvi/"
CONFIG_PROTECT_MASK="/etc/env.d /etc/gconf /etc/revdep-rebuild
/etc/terminfo"
CXXFLAGS="-march=pentium4 -mtune=pentium4 -pipe -O3 -msse2 -fweb
-frename-registers -fforce-addr -momit-leaf-frame-pointer
-fomit-frame-pointer -ftracer -mno-tls-direct-seg-refs
-fvisibility-inlines-hidden"
DISTDIR="/usr/portage/distfiles"
FEATURES="autoconfig distlocks metadata-transfer sandbox sfperms strict
userpriv usersandbox"
GENTOO_MIRRORS="http://gentoo.mirrors.easynews.com/linux/gentoo/
http://ftp.ucsb.edu/pub/mirrors/linux/gentoo/
http://mirror.datapipe.net/gentoo"
<http://mirror.datapipe.net/gentoo%22>
LDFLAGS="-W1"
MAKEOPTS="-j3"
PKGDIR="/usr/portage/packages"
PORTAGE_RSYNC_OPTS="--recursive --links --safe-links --perms --times
--compress --force --whole-file --delete --delete-after --stats
--timeout=180 --exclude='/distfiles' --exclude='/local'
--exclude='/packages'"
PORTAGE_TMPDIR="/var/tmp"
PORTDIR="/usr/portage"
PORTDIR_OVERLAY="/usr/local/portage /usr/local/gentoo-de"
SYNC="rsync://rsync.gentoo.org/gentoo-portage"
USE="x86 X alsa apache2 apm avi bash-completion berkdb bitmap-fonts
bzlib cdparanoia cli crypt cups dri dvd dvdr eds emboss fbcon foomaticdb
gdbm gif gpm gstreamer gt gtk2 imap imlib iproute2 ipv6 isdnlog jpeg
libg++ libwww lm_sensors mad maildir mikmod mime mmx motif mp3 mpeg
mysql ncurses nls nptl nptlonly nvidia ogg oggvorbis opengl pcre pda
pdflib perl pic pie pnc png posix ppds pppd prelude python qt quicktime
readline real reflection sasl sdl session smartcard smime socks5 spell
spl sse sse2 ssl tcpd truetype truetype-fonts type1-fonts unicode
userlocales v4l v4l2 vhosts vorbis win32codecs wma xml2 xorg xpm xv zlib
elibc_glibc kernel_linux userland_GNU video_cards_nvidia"
Unset: CTARGET, EMERGE_DEFAULT_OPTS, INSTALL_MASK, LANG, LC_ALL,
LINGUAS, PORTAGE_RSYNC_EXTRA_OPTS

----------------------------------------------------------
~ # uname -an
Linux gentoo 2.6.16-gentoo-r9 #1 SMP PREEMPT Mon Jun 5 13:03:57 AKDT
2006 i686 Intel(R) Pentium(R) 4 CPU 2.53GHz GNU/Linux
~ # lspci
00:00.0 Host bridge: Intel Corporation 82845G/GL[Brookdale-G]/GE/PE DRAM
Controller/Host-Hub Interface (rev 01)
00:01.0 PCI bridge: Intel Corporation 82845G/GL[Brookdale-G]/GE/PE
Host-to-AGP Bridge (rev 01)
00:1d.0 USB Controller: Intel Corporation 82801DB/DBL/DBM
(ICH4/ICH4-L/ICH4-M) USB UHCI Controller #1 (rev 01)
00:1d.1 USB Controller: Intel Corporation 82801DB/DBL/DBM
(ICH4/ICH4-L/ICH4-M) USB UHCI Controller #2 (rev 01)
00:1d.2 USB Controller: Intel Corporation 82801DB/DBL/DBM
(ICH4/ICH4-L/ICH4-M) USB UHCI Controller #3 (rev 01)
00:1d.7 USB Controller: Intel Corporation 82801DB/DBM (ICH4/ICH4-M) USB2
EHCI Controller (rev 01)
00:1e.0 PCI bridge: Intel Corporation 82801 PCI Bridge (rev 81)
00:1f.0 ISA bridge: Intel Corporation 82801DB/DBL (ICH4/ICH4-L) LPC
Interface Bridge (rev 01)
00:1f.1 IDE interface: Intel Corporation 82801DB (ICH4) IDE Controller
(rev 01)
00:1f.3 SMBus: Intel Corporation 82801DB/DBL/DBM (ICH4/ICH4-L/ICH4-M)
SMBus Controller (rev 01)
00:1f.5 Multimedia audio controller: Intel Corporation 82801DB/DBL/DBM
(ICH4/ICH4-L/ICH4-M) AC'97 Audio Controller (rev 01)
01:00.0 VGA compatible controller: nVidia Corporation NV43 [GeForce
6600/GeForce 6600 GT] (rev a2)
02:08.0 Ethernet controller: Intel Corporation 82801DB PRO/100 VE (LOM)
Ethernet Controller (rev 81)
02:09.0 Serial controller: NetMos Technology PCI 9845 Multi-I/O
Controller (rev 01)
02:0c.0 Ethernet controller: Linksys NC100 Network Everywhere Fast
Ethernet 10/100 (rev 11)
~ #

~ # df -h
Filesystem Size Used Avail Use% Mounted on
/dev/hda2 4.6G 412M 4.0G 10% /
udev 443M 248K 443M 1% /dev
/dev/hda5 4.7G 3.0G 1.7G 64% /usr
/dev/hda6 4.7G 1.7G 3.1G 35% /usr/src
/dev/hda7 4.6G 2.7G 1.7G 62% /usr/portage
/dev/hda8 4.6G 1.5G 3.0G 33% /tmp
/dev/hda9 9.4G 2.7G 6.7G 29% /var
/dev/hda10 9.2G 808M 8.0G 10% /var/tmp
/dev/hda11 47G 43G 3.8G 93% /home
/dev/hda12 26G 13G 13G 51% /pub
none 443M 0 443M 0% /dev/shm
~ # ps -ef
UID PID PPID C STIME TTY TIME CMD
root 1 0 0 16:48 ? 00:00:00 init [3]
root 2 1 0 16:48 ? 00:00:00 [migration/0]
root 3 1 0 16:48 ? 00:00:00 [ksoftirqd/0]
root 4 1 0 16:48 ? 00:00:00 [events/0]
root 5 1 0 16:48 ? 00:00:00 [khelper]
root 6 1 0 16:48 ? 00:00:00 [kthread]
root 8 6 0 16:48 ? 00:00:00 [kblockd/0]
root 56 6 0 16:48 ? 00:00:00 [pdflush]
root 57 6 0 16:48 ? 00:00:00 [pdflush]
root 58 1 0 16:48 ? 00:00:00 [kswapd0]
root 59 6 0 16:48 ? 00:00:00 [aio/0]
root 641 6 0 16:48 ? 00:00:00 [kseriod]
root 691 1 0 16:48 ? 00:00:00 [kjournald]
root 890 1 0 16:48 ? 00:00:00 /sbin/udevd --daemon
root 2102 6 0 16:48 ? 00:00:00 [kpsmoused]
root 2359 6 0 16:48 ? 00:00:00 [reiserfs/0]
root 2437 6 0 16:48 ? 00:00:00 [khubd]
root 5342 1 0 16:48 ? 00:00:00 /usr/sbin/syslog-ng
root 7298 1 0 16:48 ? 00:00:00 /usr/sbin/ifplugd --iface=eth1
root 7426 1 0 16:48 ? 00:00:00 /usr/sbin/ifplugd --iface=eth0
root 8241 1 0 16:48 ? 00:00:00 /sbin/dhcpcd -h gentoo -o eth1
mysql 8802 1 0 16:49 ? 00:00:00 /usr/sbin/mysqld
--defaults-file=/etc/mysql/my.cnf --basedir=/usr --datadi
root 8885 1 0 16:49 ? 00:00:00 /usr/sbin/sshd
root 8962 1 0 16:49 ? 00:00:00 /usr/sbin/apache2 -D DEFAULT_VHOST -D
SSL_DEFAULT_VHOST -DSSL -D PHP5 -d /
apache 9026 8962 0 16:49 ? 00:00:00 /usr/sbin/apache2 -D DEFAULT_VHOST
-D SSL_DEFAULT_VHOST -DSSL -D PHP5 -d /
apache 9027 8962 0 16:49 ? 00:00:00 /usr/sbin/apache2 -D DEFAULT_VHOST
-D SSL_DEFAULT_VHOST -DSSL -D PHP5 -d /
apache 9028 8962 0 16:49 ? 00:00:00 /usr/sbin/apache2 -D DEFAULT_VHOST
-D SSL_DEFAULT_VHOST -DSSL -D PHP5 -d /
apache 9029 8962 0 16:49 ? 00:00:00 /usr/sbin/apache2 -D DEFAULT_VHOST
-D SSL_DEFAULT_VHOST -DSSL -D PHP5 -d /
apache 9030 8962 0 16:49 ? 00:00:00 /usr/sbin/apache2 -D DEFAULT_VHOST
-D SSL_DEFAULT_VHOST -DSSL -D PHP5 -d /
apache 9031 8962 0 16:49 ? 00:00:00 /usr/sbin/apache2 -D DEFAULT_VHOST
-D SSL_DEFAULT_VHOST -DSSL -D PHP5 -d /
root 9088 1 0 16:49 ? 00:00:00 /opt/vmware/workstation/bin/vmnet-bridge
-d /var/run/vmnet-bridge-0.pid /d
root 9098 1 0 16:49 ? 00:00:00 /opt/vmware/workstation/bin/vmnet-bridge
-d /var/run/vmnet-bridge-2.pid /d
root 9171 1 0 16:49 ? 00:00:00 /usr/sbin/cupsd
at 9311 1 0 16:49 ? 00:00:00 /usr/sbin/atd
101 9675 1 0 16:49 ? 00:00:00 /usr/bin/dbus-daemon --system
root 9743 1 0 16:49 ? 00:00:00 /usr/sbin/gpm -m /dev/input/mice -t ps2
102 9812 1 0 16:49 ? 00:00:00 /usr/sbin/hald --use-syslog
102 9827 9812 0 16:49 ? 00:00:00 hald-addon-storage
102 9829 9812 0 16:49 ? 00:00:00 hald-addon-storage
ntp 10141 1 0 16:49 ? 00:00:00 /usr/sbin/ntpd -s
root 10142 1 0 16:49 ? 00:00:00 /usr/sbin/ntpd -s
privoxy 10581 1 0 16:49 ? 00:00:00 /usr/sbin/privoxy --pidfile
/var/run/privoxy.pid --user privoxy privoxy /e
tor 12135 1 0 16:49 ? 00:00:01 /usr/bin/tor --runasdaemon 1
root 12204 1 0 16:49 ? 00:00:00 /usr/sbin/cron
root 12228 12204 0 16:49 ? 00:00:00 /usr/sbin/cron
root 12229 12228 0 16:49 ? 00:00:00 /bin/sh -c /root/syslog_mysql.sh >>
/var/log/mysql.log 2>&1
root 12230 12229 0 16:49 ? 00:00:00 /bin/bash /root/syslog_mysql.sh
root 12238 12230 0 16:49 ? 00:00:00 /bin/bash /root/syslog_mysql.sh
root 12278 1 0 16:49 ? 00:00:00 /usr/bin/perl
/usr/libexec/webmin/miniserv.pl /etc/webmin/miniserv.conf
root 12348 1 0 16:49 ? 00:00:00 /usr/sbin/xinetd -pidfile
/var/run/xinetd.pid -stayalive -reuse
1010 12424 1 0 16:49 tty1 00:00:00 -bash
root 12425 1 0 16:49 tty2 00:00:00 /sbin/agetty 38400 tty2 linux
root 12426 1 0 16:49 tty3 00:00:00 /sbin/agetty 38400 tty3 linux
root 12427 1 0 16:49 tty4 00:00:00 /sbin/agetty 38400 tty4 linux
root 12428 1 0 16:49 tty5 00:00:00 /sbin/agetty 38400 tty5 linux
root 12429 1 0 16:49 tty6 00:00:00 /sbin/agetty 38400 tty6 linux
1010 12465 1 0 16:50 ? 00:00:00 /usr/bin/ssh-agent
1010 12468 12424 0 16:50 tty1 00:00:00 /bin/sh /usr/bin/startx
1010 12480 12468 0 16:50 tty1 00:00:00 xinit /etc/X11/xinit/xinitrc --
-nolisten tcp -br -deferglyphs 16
root 12481 12480 1 16:50 ? 00:00:20 X :0 -nolisten tcp -br -deferglyphs
16
1010 12512 12480 0 16:50 tty1 00:00:00 /bin/sh /etc/X11/Sessions/icewm
1010 12563 12512 0 16:50 tty1 00:00:00 /usr/bin/icewm-session
1010 12564 12563 0 16:50 ? 00:00:00 icewmbg
1010 12565 12563 0 16:50 ? 00:00:00 icewm
1010 12566 12563 0 16:50 ? 00:00:00 icewmtray
1010 12568 1 0 16:50 ? 00:00:01 gkrellm2
1010 12774 12565 0 17:03 ? 00:00:00 /bin/bash
/usr/libexec/mozilla-launcher
1010 12783 12774 6 17:03 ? 00:00:15 /usr/lib/mozilla-firefox/firefox-bin

1010 12790 1 0 17:03 ? 00:00:00 /usr/libexec/gconfd-2 16
1010 12802 12565 0 17:03 ? 00:00:01 konsole
1010 12806 1 0 17:03 ? 00:00:00 kdeinit Running...
1010 12810 1 0 17:03 ? 00:00:00 dcopserver [kdeinit] --nosid --suicide
1010 12812 12806 0 17:03 ? 00:00:00 klauncher [kdeinit]
1010 12814 1 0 17:03 ? 00:00:00 kded [kdeinit]
1010 12817 12802 0 17:03 pts/0 00:00:00 /bin/bash
root 12829 12817 0 17:04 pts/0 00:00:00 -su
1010 12939 12802 0 17:06 pts/1 00:00:00 /bin/bash
1010 12945 12939 2 17:07 pts/1 00:00:00 kedit
root 12953 12829 0 17:07 pts/0 00:00:00 ps -ef

--------------------------------------------------------------------
~ # lsmod
Module Size Used by
xt_tcpudp 3968 35
xt_state 2688 12
iptable_mangle 3072 1
ipt_TOS 2816 0
ipt_SAME 2944 0
ipt_REJECT 4864 4
ipt_REDIRECT 2688 0
ipt_NETMAP 2688 0
ipt_MASQUERADE 3456 0
ipt_LOG 6272 4
ipt_esp 2560 0
ipt_ECN 3456 0
ipt_DSCP 2816 0
ipt_ah 2560 0
iptable_nat 7300 0
ip_nat 13868 5
ipt_SAME,ipt_REDIRECT,ipt_NETMAP,ipt_MASQUERADE,iptable_nat
iptable_filter 3200 1
ip_tables 11508 3 iptable_mangle,iptable_nat,iptable_filter
x_tables 10244 15
xt_tcpudp,xt_state,ipt_TOS,ipt_SAME,ipt_REJECT,ipt_REDIRECT,ipt_NETMAP,i
pt_MASQUERADE,ipt_LOG,ipt_esp,ipt_ECN,ipt_DSCP,ipt_ah,iptable_nat,ip_tab
les
smsc47m1 8068 0
eeprom 6160 0
lm90 11428 0
hwmon 3076 2 smsc47m1,lm90
i2c_isa 4352 1 smsc47m1
"bc_serpent" 26148 0
"bc_rc6" 12068 0
"bc_cast" 20516 0
"bc_rijn" 42148 0
"bc_idea" 9124 0
"bc_3des" 16740 0
"bc_bf128" 11812 0
"bc_bf448" 11940 0
"bc_twofish" 19392 0
"bc_gost" 7460 0
"bc_des" 16100 0
"bc_blowfish" 11812 0
bc 16588 12
"bc_serpent","bc_rc6","bc_cast","bc_rijn","bc_idea","bc_3des","bc_bf128"
,"bc_bf448","bc_twofish","bc_gost","bc_des","bc_blowfish"
vmnet 26788 6
parport_pc 21444 0
parport 20032 1 parport_pc
vmmon 170956 0
tulip 43296 0
8250_pci 19072 0
eepro100 25488 0
nvidia 4545492 12
i2c_i801 8204 0
i2c_core 16768 6 smsc47m1,eeprom,lm90,i2c_isa,nvidia,i2c_i801
ehci_hcd 25224 0
usbhid 26628 0
uhci_hcd 27536 0
snd_seq 44368 0
snd_seq_device 7052 1 snd_seq
snd_intel8x0 27292 0
snd_ac97_codec 80416 1 snd_intel8x0
snd_ac97_bus 2816 1 snd_ac97_codec
snd_pcm 68740 2 snd_intel8x0,snd_ac97_codec
snd_timer 18948 2 snd_seq,snd_pcm
snd 38004 6
snd_seq,snd_seq_device,snd_intel8x0,snd_ac97_codec,snd_pcm,snd_timer
soundcore 8032 1 snd
snd_page_alloc 8200 2 snd_intel8x0,snd_pcm
usbcore 101764 4 ehci_hcd,usbhid,uhci_hcd
fuse 33032 0
8250 20444 1 8250_pci
serial_core 16384 1 8250
loop 13064 0
udf 71044 0
floppy 52100 0
ide_cd 33796 0
cdrom 32288 1 ide_cd
psmouse 33288 0

---------
To unsubscribe, send email to <aklug-request@aklug.org>
with 'unsubscribe' in the message body.
Received on Tue Jun 20 15:21:54 2006

This archive was generated by hypermail 2.1.8 : Tue Jun 20 2006 - 15:21:54 AKDT