Scheduling problem

I've got this problem with linux (2.4.26 i686). On a system that is usually lightly loaded:

top - 20:15:36 up 7:55, 7 users, load average: 2.84, 1.56, 1.49 Tasks: 162 total, 4 running, 158 sleeping, 0 stopped, 0 zombie Cpu(s): 0.3% user, 5.1% system, 1.0% nice, 93.6% idle Mem: 1031584k total, 1016112k used, 15472k free, 84104k buffers Swap: 1004020k total, 120k used, 1003900k free, 610552k cached

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ Command 5612 pjb 16 0 944 944 700 R 4.2 0.1 0:01.13 top 19 root 11 0 0 0 0 S 0.6 0.0 4:40.53 kjournald 20362 pjb 9 0 57504 13m 9092 S 0.3 1.3 1:01.75 aviplay 1 root 8 0 244 244 208 S 0.0 0.0 0:08.39 init 2 root 9 0 0 0 0 S 0.0 0.0 0:00.73 keventd 3 root 9 0 0 0 0 S 0.0 0.0 0:00.82 kapmd 4 root 19 19 0 0 0 S 0.0 0.0 0:04.72 ksoftirqdCPU0 5 root 9 0 0 0 0 S 0.0 0.0 0:00.33 kswapd 6 root 9 0 0 0 0 S 0.0 0.0 0:00.00 bdflush 7 root 9 0 0 0 0 S 0.0 0.0 5:04.36 kupdated 14 root 9 0 0 0 0 S 0.0 0.0 0:00.01 khubd 15 root 9 0 0 0 0 S 0.0 0.0 0:00.00 i2oevtd 16 root 9 0 0 0 0 S 0.0 0.0 0:00.00 i2oblock 17 root -1 -20 0 0 0 S 0.0 0.0 0:00.00 mdrecoveryd 119 root 9 0 0 0 0 S 0.0 0.0 0:37.84 kjournald 120 root 9 0 0 0 0 S 0.0 0.0 0:00.00 kjournald 121 root 9 0 0 0 0 S 0.0 0.0 0:00.00 kjournald 122 root 9 0 0 0 0 S 0.0 0.0 0:13.88 kjournald 123 root 9 0 0 0 0 S 0.0 0.0 0:00.00 kjournald 363 pjb 9 0 508 508 436 S 0.0 0.0 0:00.04 ptidoigt 604 root 9 0 604 604 504 S 0.0 0.1 0:00.78 syslogd 607 root 9 0 592 592 400 S 0.0 0.1 0:00.01 klogd 995 root 9 0 636 636 548 S 0.0 0.1 0:00.00 resmgrd 1008 irc 9 0 3332 3328 800 S 0.0 0.3 0:01.26 ircd 1010 irc 9 0 632 632 424 S 0.0 0.1 0:00.97 iauth 1024 bin 9 0 404 404 336 S 0.0 0.0 0:00.00 portmap 1059 root 9 0 660 660 584 S 0.0 0.1 0:00.00 rpc.statd 1115 root 9 0 1240 1240 984 S 0.0 0.1 0:02.58 master 1132 postfix 9 0 1296 1296 1028 S 0.0 0.1 0:00.46 qmgr 1192 root 9 0 10148 9.9m 9.8m S 0.0 1.0 0:06.31 httpd 1193 wwwrun 9 0 10336 10m 9.9m S 0.0 1.0 0:00.09 httpd 1209 at 9 0 548 548 480 S 0.0 0.1 0:00.00 atd 1227 root 8 0 652 652 560 S 0.0 0.1 0:00.61 cron

it happens quite periodically (between two times a minutes up to a few times per five minutes), that the computer "hangs" for a second or a second and a half, then resumes. When it resumes, I can see in top that a lot of processes are ready to run, and it spent a lot of time in the kernel.

top - 20:13:59 up 7:54, 7 users, load average: 0.83, 1.11, 1.36 Tasks: 165 total, 15 running, 149 sleeping, 1 stopped, 0 zombie Cpu(s): 4.7% user, 88.3% system, 0.0% nice, 7.0% idle Mem: 1031584k total, 1015568k used, 16016k free, 84040k buffers Swap: 1004020k total, 120k used, 1003900k free, 610104k cached

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ Command 1658 pjb 16 0 1152 1152 924 S 10.3 0.1 2:58.34 wmnet 5148 pjb 14 0 952 952 704 R 9.6 0.1 0:13.83 top 119 root 20 0 0 0 0 R 8.4 0.0 0:37.73 kjournald 20376 pjb 17 0 57480 13m 9092 R 8.4 1.3 1:12.28 aviplay 1656 pjb 14 0 34996 34m 4220 R 8.1 3.4 2:49.90 emacs 10704 pjb 14 0 64376 62m 29m S 7.5 6.2 0:36.41 mozilla-bin 1657 pjb 14 0 1296 1296 1076 R 7.2 0.1 0:40.61 wmcalclock 7862 pjb 14 0 27068 26m 4348 R 7.2 2.6 2:40.33 emacs 10671 pjb 14 0 64376 62m 29m R 7.2 6.2 16:34.37 mozilla-bin 19000 root 14 0 13812 13m 3980 R 7.2 1.3 0:34.60 emacs 20375 pjb 14 0 57480 13m 9092 R 7.2 1.3 2:55.62 aviplay 122 root 19 0 0 0 0 R 5.9 0.0 0:13.65 kjournald 1586 root 17 0 113m 56m 4556 R 5.0 5.6 5:22.48 X 20362 pjb 14 0 57480 13m 9092 R 5.0 1.3 1:01.28 aviplay 1632 pjb 10 0 2416 2416 1952 S 4.4 0.2 0:09.24 medusa-idled 20374 pjb 14 0 57480 13m 9092 R 3.7 1.3 0:18.94 aviplay 1614 pjb 12 0 3284 3284 2240 S 3.1 0.3 0:27.85 wmaker 1008 irc 12 0 3332 3328 800 S 1.6 0.3 0:01.26 ircd 1 root 11 0 244 244 208 S 1.2 0.0 0:08.33 init 604 root 11 0 604 604 504 S 1.2 0.1 0:00.78 syslogd 1192 root 12 0 10148 9.9m 9.8m S 1.2 1.0 0:06.30 httpd 1340 root 11 0 564 564 488 S 1.2 0.1 0:02.95 routed 1241 root 12 0 3088 3088 1308 S 0.9 0.3 0:12.28 cupsd 4 root 20 19 0 0 0 R 0.6 0.0 0:04.68 plus 1 ksoftirqdCPU0 19 root 10 0 0 0 0 S 0.6 0.0 4:39.00 kjournald 1266 nobody 11 0 872 872 720 S 0.6 0.1 0:00.55 gdomap 1303 root 11 0 716 716 600 S 0.6 0.1 0:00.97 nscd 10673 pjb 9 0 64376 62m 29m S 0.6 6.2 0:02.27 mozilla-bin 1663 pjb 9 0 12356 12m 2140 S 0.3 1.2 0:01.81 xterm 2 root 9 0 0 0 0 S 0.0 0.0 0:00.73 keventd 3 root 9 0 0 0 0 S 0.0 0.0 0:00.82 kapmd 5 root 9 0 0 0 0 S 0.0 0.0 0:00.33 kswapd 6 root 9 0 0 0 0 S 0.0 0.0 0:00.00 bdflush 7 root 9 0 0 0 0 S 0.0 0.0 4:55.00 kupdated 14 root 9 0 0 0 0 S 0.0 0.0 0:00.01 khubd

During these freeze events, the X mouse doesn't move and the sound output is suspended too. I've got a user-mode-linux instance running, but when I stop it, while it alleviates somewhat, the problem still happens. There's nothing reported invar-log-messages. And tail -fproc-kmsg doesn't output anything.

What can I do to track or eliminate this problem?

processor : 0 vendorid : AuthenticAMD cpu family : 6 model : 4 model name : AMD Athlon(tm) Processor stepping : 2 cpu MHz : 1200.085 cache size : 256 KB fdivbug : no hltbug : no f00fbug : no comabug : no fpu : yes fpuexception : yes cpuid level : 1 wp : yes flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 mmx fxsr syscall mmxext 3dnowext 3dnow bogomips : 2392.06

root=-dev-hda3 hdd=scsi pci=noacpi mem=1048496K CPU0 0: 2911905 XT-PIC timer 1: 24110 XT-PIC keyboard 2: 0 XT-PIC cascade 5: 250570 XT-PIC ide3, usb-uhci, usb-uhci 8: 0 XT-PIC rtc 10: 435785 XT-PIC eth1, cmpci 11: 90977 XT-PIC eth0 12: 202623 XT-PIC PS-2 Mouse 14: 1513997 XT-PIC ide0 15: 148 XT-PIC ide1 NMI: 0 LOC: 2911915 ERR: 1380 MIS: 0 slabinfo - version: 1.1 kmemcache 67 72 108 2 2 1 urbpriv 0 0 64 0 0 1 ipfibhash 23 112 32 1 1 1 cliparpcache 0 0 128 0 0 1 ipconntrack 7 24 320 2 2 1 ipmrtcache 0 0 128 0 0 1 tcptwbucket 0 30 128 0 1 1 tcpbindbucket 17 112 32 1 1 1 tcpopenrequest 0 59 64 0 1 1 inetpeercache 4 59 64 1 1 1 ipdstcache 10 60 192 2 3 1 arpcache 5 30 128 1 1 1 blkdevrequests 8192 8220 128 274 274 1 nfswritedata 0 0 384 0 0 1 nfsreaddata 0 0 384 0 0 1 nfspage 0 0 128 0 0 1 devfsdevent 0 0 20 0 0 1 journalhead 27 2464 48 2 32 1 revoketable 6 250 12 1 1 1 revokerecord 0 0 32 0 0 1 dnotifycache 0 0 20 0 0 1 filelockcache 7 42 92 1 1 1 fasynccache 3 202 16 1 1 1 uidcache 8 112 32 1 1 1 skbuffheadcache 204 240 192 12 12 1 sock 276 315 832 33 35 2 sigqueue 68 plus 1 87 132 3 3 1 kiobuf 0 0 64 0 0 1 cdevcache 503 531 64 9 9 1 bdevcache 7 59 64 1 1 1 mntcache 20 59 64 1 1 1 inodecache 27451 27489 512 3925 3927 1 dentrycache 61058 67470 128 2249 2249 1 dquot 0 0 128 0 0 1 filp 1722 1740 128 58 58 1 namescache 0 2 4096 0 2 1 bufferhead 171685 172830 128 5756 5761 1 mmstruct 104 140 192 6 7 1 vmareastruct 50029 62430 128 1853 2081 1 fscache 125 177 64 3 3 1 filescache 71 81 448 9 9 1 signalact 136 156 1344 49 52 1 size-131072(DMA) 0 0 131072 0 0 32 size-131072 0 0 131072 0 0 32 size-65536(DMA) 0 0 65536 0 0 16 size-65536 0 0 65536 0 0 16 size-32768(DMA) 0 0 32768 0 0 8 size-32768 1 1 32768 1 1 8 size-16384(DMA) 0 0 16384 0 0 4 size-16384 4 4 16384 4 4 4 size-8192(DMA) 0 0 8192 0 0 2 size-8192 29 29 8192 29 29 2 size-4096(DMA) 0 0 4096 0 0 1 size-4096 58 71 4096 58 71 1 size-2048(DMA) 0 0 2048 0 0 1 size-2048 107 110 2048 54 55 1 size-1024(DMA) 0 0 1024 0 0 1 size-1024 82 88 1024 22 22 1 size-512(DMA) 0 0 512 0 0 1 size-512 130 160 512 17 20 1 size-256(DMA) 0 0 256 0 0 1 size-256 60 75 256 5 5 1 size-128(DMA) 2 30 128 1 1 1 size-128 1756 1830 128 60 61 1 size-64(DMA) 0 0 64 0 0 1 size-64 607 708 64 12 12 1 size-32(DMA) 40 59 64 1 1 1 size-32 7064 10148 64 172 172 1

-- Until real software engineering is developed, the next best practice is to develop with a dynamic system that has extreme late binding in all aspects. The first system to really do this in an important way is Lisp. -- Alan Kay