df, mv, date, rmdir segfaults, other commands acting funny

Hello all,

A few nights ago my server lost power during a heavy thunderstorm and it's never been the same since. I should also mention that one of my non-critical parbreastions on my HD was full.

The first problem is that it wouldn't boot up properly. It starts with ppp not being turned on due to a fork failure.

Trying to investigate my startup problem, my best theory is that some fundamental module must have been corrupted. Here are my symptoms:

1) Running ps -aux reveals several zombied processes: ls, rm, touch, mkdir 2) Bash is working, however some of my environment variables (the colours specifically) don't get set. 3) Several commands that have the -h option error saying that -h is an invalid option, even though it obviously is. 4) date, df, mv, rmdir all segfault, 100% of the time. 5) This leads to several other programs not working properly, including locate, updatedb and apt.

Other than that - everything else seems to be working alright. PHP, Apache, postgres all start up and run fine.

I'm an experienced programmer, but only a moderate Linux user - if I could get some advice on how to trace back my problems to a single function or process request I can nip this issue in the bud.

Thanks in advance!

Aienthiwan

PS: I've included the output from my dmesg log here:

(Debian)) #1 Sun Nov 30 16:49:14 EST 2003 BIOS-provided physical RAM map: BIOS-e820: 0000000000000000 - 00000000000a0000 (usable) BIOS-e820: 00000000000f0000 - 0000000000100000 (reserved) BIOS-e820: 0000000000100000 - 000000001dff0000 (usable) BIOS-e820: 000000001dff0000 - 000000001dff3000 (ACPI NVS) BIOS-e820: 000000001dff3000 - 000000001e000000 (ACPI data) BIOS-e820: 00000000fec00000 - 00000000fec01000 (reserved) BIOS-e820: 00000000fee00000 - 00000000fee01000 (reserved) BIOS-e820: 00000000ffff0000 - 0000000100000000 (reserved) 479MB LOWMEM available. ACPI: have wakeup address 0xc0001000 On node 0 totalpages: 122864 zone(0): 4096 pages. zone(1): 118768 pages. zone(2): 0 pages. 0x1dff3000 0x1dff3040 0x1dff7180 0x00000000 Kernel command line: auto BOOTIMAGE=Linux root=301 Initializing CPU#0 Detected 1826.003 MHz processor. Console: colour VGA+ 80x25 Calibrating delay loop... 3643.80 BogoMIPS Memory: 480712k-491456k available (1044k kernel code, 10360k reserved, 445k data, 76k init, 0k highmem) Dentry cache hash table entries: 65536 (order: 7, 524288 bytes) Inode cache hash table entries: 32768 (order: 6, 262144 bytes) Mount cache hash table entries: 512 (order: 0, 4096 bytes) Buffer cache hash table entries: 32768 (order: 5, 131072 bytes) Page-cache hash table entries: 131072 (order: 7, 524288 bytes) CPU: L1 I Cache: 64K (64 bytes-line), D cache 64K (64 bytes-line) CPU: L2 Cache: 512K (64 bytes-line) CPU: After generic, caps: 0383fbff c1c3fbff 00000000 00000000 CPU: Common caps: 0383fbff c1c3fbff 00000000 00000000 CPU: AMD Athlon(tm) XP 2500+ stepping 00 Enabling fast FPU save and restore... done. Enabling unmasked SIMD FPU exception support... done. Checking 'hlt' instruction... OK. Checking for popad bug... OK. POSIX conformance testing by UNIFIX ACPI: Subsystem revision 20031002 ACPI: Interpreter disabled. PCI: PCI BIOS revision 2.10 entry at 0xfb770, last bus=1 PCI: Using configuration type 1 PCI: Probing PCI hardware PCI: ACPI tables contain no PCI IRQ routing entries PCI: Probing PCI hardware (bus 00) PCI: Using IRQ router VIA 1106-3177 at 00:11.0 Linux NET4.0 for Linux 2.4 Based upon Swansea University Computer Society NET3.039 Initializing RT netlink socket Starting kswapd VFS: Disk quotas vdquot6.5.1 devfs: bootoptions: 0x0 Detected PS-2 Mouse Port. pty: 256 Unix98 ptys configured Serial driver version 5.05c (2001-07-08) with HUB-6 MANYPORTS MULTIPORT SHAREIRQ SERIALPCI enabled ttyS00 at 0x03f8 (irq = 4) is a 16550A RAMDISK driver initialized: 16 RAM disks of 4096K size 1024 blocksize Initializing Cryptographic API NET4: Linux TCP-IP 1.0 for NET4.0 IP: routing cache hash table of 4096 buckets, 32Kbytes TCP: Hash tables configured (established 32768 bind 65536) Linux IP multicast router 0.06 plus PIM-SM RAMDISK: cramfs filesystem found at block 0 RAMDISK: Loading 3224 blocks 1 disk into ram disk... ^H-^H-^H-^H^H-^H-^H-^H^H-^H-^H-^H^H-^H-^H-^H^H-^H-^H-^H^H-^H-^H-^H^H-^H-^H^H^H-^H-^H-^H^H-^H-^H-^H^H-^H-^H-^H^H-^H-^H-^H^H-^H-^H-^H^H-^H-^H-^H^H-^H-^H-^H^H-^H-^H-^H^H-^H-^H-^H^H-^H-^H-^H^H-^H-^H-^H^H^H-^H-^H^H-^H-^H-^H^H-^H-^H-^H^H-^H-^H-^H^H-^H-^H-^H^H-^H-^H-^H^H-^H-^H-^H^H-^H-^H-^H^H-^H-^H-^H^H-^H-^H-^H^H-^H-^H-^H^H-^H-^ H-^H^H-^H-^H-^H^H-^H-^H-^H^H-^H-^H-^H^H-^H-^H-^H^H-^H-^H-^H^H-^H-^H-^H^H-^H-^H-^H^H-^H-^H-^H^H-^H-^H-^H^H-^H-^H-^H^H-^H-^H-^H ^H-^H-^H-^H^H-^H-^H-^H^H-^H-^H-^H^H-^H-^H-^H^H-^H-^H-^H^H-^H-^H-^H^H-^H-^H-^H^H-^H-^H-^H^H-^H-^H-^H^H-^Hdone. Freeing initrd memory: 3224k freed VFS: Mounted root (cramfs filesystem). Freeing unused kernel memory: 76k freed NET4: Unix domain sockets 1.0-SMP for Linux NET4.0. Uniform Multi-Platform E-IDE driver Revision: 7.00beta4-2.4 ide: buttuming 33MHz system bus speed for PIO modes; override with idebus=xx ide: late registration of driver. VPIDE: IDE controller at PCI slot 00:11.1 VPIDE: chipset revision 6 VPIDE: not 100% native mode: will probe irqs later ide: buttuming 33MHz system bus speed for PIO modes; override with idebus=xx VPIDE: VIA vt8235 (rev 00) IDE UDMA133 controller on pci00:11.1 ide0: BM-DMA at 0xe000-0xe007, BIOS settings: hda:DMA, hdb:DMA ide1: BM-DMA at 0xe008-0xe00f, BIOS settings: hdc:pio, hdd:pio hda: WDC WD800BB-75DKA0, ATA DISK drive hdb: WDC WD205BA, ATA DISK drive blk: queue de826540, I-O limit 4095Mb (mask 0xffffffff) blk: queue de82667c, I-O limit 4095Mb (mask 0xffffffff) ide0 at 0x1f0-0x1f7,0x3f6 on irq 14 hda: attached ide-disk driver. hda: 156250000 sectors (80000 MB) w-2048KiB Cache, CHS=9726-255-63, UDMA(100) hdb: attached ide-disk driver. hdb: 40088160 sectors (20525 MB) w-2048KiB Cache, CHS=39770-16-63, UDMA(66) Parbreastion check:dev-ide-host0-bus0-target1-lun0: p1 p2 Journalled Block Device driver loaded hda: attached ide-disk driver. hda: 156250000 sectors (80000 MB) w-2048KiB Cache, CHS=9726-255-63, UDMA(100) hdb: attached ide-disk driver. hdb: 40088160 sectors (20525 MB) w-2048KiB Cache, CHS=39770-16-63, UDMA(66)dev-ide-host0-bus0-target1-lun0: p1 p2 EXT3-fs: INFO: recovery required on readonly filesystem. EXT3-fs: write access will be enabled during recovery. kjournald starting. Commit interval 5 seconds EXT3-fs: recovery complete. EXT3-fs: mounted filesystem with ordered data mode. Adding Swap: 1156672k swap-space (priority -1) EXT3 FS 2.4-0.9.19, 19 August 2002 on ide0(3,1), internal journal touch uses obsolete (PFINET,SOCKPACKET) requestmoduleppp0: fork failed, errno 1 requestmoduleppp0: fork failed, errno 1 requestmoduleppp0: fork failed, errno 1 kjournald starting. Commit interval 5 seconds EXT3 FS 2.4-0.9.19, 19 August 2002 on ide0(3,7), internal journal EXT3-fs: mounted filesystem with ordered data mode. kjournald starting. Commit interval 5 seconds EXT3 FS 2.4-0.9.19, 19 August 2002 on ide0(3,6), internal journal EXT3-fs: mounted filesystem with ordered data mode. kjournald starting. Commit interval 5 seconds EXT3 FS 2.4-0.9.19, 19 August 2002 on ide0(3,8), internal journal EXT3-fs: mounted filesystem with ordered data mode. kjournald starting. Commit interval 5 seconds EXT3 FS 2.4-0.9.19, 19 August 2002 on ide0(3,5), internal journal EXT3-fs: mounted filesystem with ordered data mode. EXT2-fs warning (device ide0(3,65)): ext2readsuper: mounting ext3 filesystem as ext2

requestmoduleppp0: fork failed, errno 1 parport0: PC-style at 0x378 (0x778) PCSPP,TRISTATE,EPP parport0: irq 7 detected lp0: using parport0 (polling). requestmoduleppp0: fork failed, errno 1 requestmoduleppp0: fork failed, errno 1 requestmoduleppp0: fork failed, errno 1 requestmoduleppp0: fork failed, errno 1 requestmoduleppp0: fork failed, errno 1 requestmoduleppp0: fork failed, errno 1 requestmoduleppp0: fork failed, errno 1 requestmoduleppp0: fork failed, errno 1 Real Time Clock Driver v1.10e