I know this sounds vague, but I need a hand with something - I have a server that's running FC2 - that's primarily a PHP/mySQL machine. This is a P4 2.8HT with 2 gigs of ram, the disks are running in DMA mode 5.
Static pages are served pretty fast, so I'm sure that Apache is fine. However, some PHP scripts are taking a bit longer than usual to respond - to add to this - I've noticed that the 'slowdown' was accompanied by a huge increase in compilation time. Compiling PHP used to take 5 minutes, now it takes forever.
I'm not sure where to begin, so I'm turning to the experts. load average from uptime shows: 0.53 0.46 0.35
procinfo shows:
Linux 2.6.9-1.6_FC2smp (bhcompile@bugs) (gcc 3.3.3 20040412 ) #1 2CPU [inferis]
Memory: Total Used Free Shared Buffers
Mem: 2065432 1666788 398644 0 183988
Swap: 1048312 0 1048312
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
8055 root 17 0 2320 896 1624 R 4.5 0.0 0:00.67 top
6128 root 16 0 7820 2128 4252 S 0.3 0.1 0:06.57 sshd
1 root 16 0 2724 460 1320 S 0.0 0.0 0:19.12 init
2 root RT 0 0 0 0 S 0.0 0.0 0:01.50 migration/0
3 root 34 19 0 0 0 S 0.0 0.0 0:00.06 ksoftirqd/0
4 root RT 0 0 0 0 S 0.0 0.0 0:02.21 migration/1
5 root 34 19 0 0 0 S 0.0 0.0 0:00.05 ksoftirqd/1
6 root 5 -10 0 0 0 S 0.0 0.0 0:00.00 events/0
7 root 5 -10 0 0 0 S 0.0 0.0 0:00.00 events/1
8 root 6 -10 0 0 0 S 0.0 0.0 0:00.00 khelper
9 root 14 -10 0 0 0 S 0.0 0.0 0:00.00 kacpid
20 root 5 -10 0 0 0 S 0.0 0.0 0:00.00 kblockd/0
21 root 5 -10 0 0 0 S 0.0 0.0 0:00.00 kblockd/1
22 root 25 0 0 0 0 S 0.0 0.0 0:00.00 khubd
31 root 15 0 0 0 0 S 0.0 0.0 0:40.40 pdflush
34 root 7 -10 0 0 0 S 0.0 0.0 0:00.00 aio/0
35 root 7 -10 0 0 0 S 0.0 0.0 0:00.00 aio/1
33 root 16 0 0 0 0 S 0.0 0.0 0:13.20 kswapd0
Any help appreciated!
Cheers.
Alex
happybunny
03-16-2006, 10:11 PM
i was banging my head against the wall on a slow server not long ago....looked every where for the answer....memory, cpu, ps, top, etcetcetc....what i didn't check was how much space was left on my filesystems.
In my case, / was full. Oops.
I'd check there, too.
Lexx
03-16-2006, 11:33 PM
Hi Mike - I'd thought of that thinking back to my windows machine back in the days. It reports:
Hehe, it's just a webserver, 28 gigs is a lot of 300 meg plans ;) Still, I'm bashing my head on this one, I'm about to fold and purchase Zend Platform to debug the server
voidinit
03-17-2006, 01:26 AM
Hrm.. Not into swap. Plenty of memory.. Did you check dmesg to see if you have any disk or network errors?
Another cause of slowdowns can be DNS sometimes. Make sure the box can resolve it's own hostname forward and reverse, and has access to a reliable and timely nameserver so it can determine if remote machines are resovlable or not quickly. Most threads of execution will block at one point or another until a name is resolved or reversely resolved.
Those are two most common problems of server slowdown I've experienced. The less commons reasons I usually run into are bad ram, old buggy NIC drivers.
Lexx
03-17-2006, 02:05 AM
I must admit I'm not terribly apt at scanning dmesg
Linux version 2.6.9-1.6_FC2smp (bhcompile@bugs.build.redhat.com) (gcc version 3.3.3 20040412 (Red Hat Linux 3.3.3-7)) #1 SMP Thu Nov 18 22:20:16 EST 2004
BIOS-provided physical RAM map:
BIOS-e820: 0000000000000000 - 000000000009fc00 (usable)
BIOS-e820: 000000000009fc00 - 00000000000a0000 (reserved)
BIOS-e820: 00000000000e8000 - 0000000000100000 (reserved)
BIOS-e820: 0000000000100000 - 000000007f730000 (usable)
BIOS-e820: 000000007f730000 - 000000007f740000 (ACPI data)
BIOS-e820: 000000007f740000 - 000000007f7f0000 (ACPI NVS)
BIOS-e820: 000000007f7f0000 - 000000007f800000 (reserved)
BIOS-e820: 00000000ffb80000 - 0000000100000000 (reserved)
0MB HIGHMEM available.
2039MB LOWMEM available.
found SMP MP-table at 000ff780
On node 0 totalpages: 522032
DMA zone: 4096 pages, LIFO batch:1
Normal zone: 517936 pages, LIFO batch:16
HighMem zone: 0 pages, LIFO batch:1
DMI 2.3 present.
Using APIC driver default
ACPI: RSDP (v000 ACPIAM ) @ 0x000fadc0
ACPI: RSDT (v001 A M I OEMRSDT 0x06000411 MSFT 0x00000097) @ 0x7f730000
ACPI: FADT (v002 A M I OEMFACP 0x06000411 MSFT 0x00000097) @ 0x7f730200
ACPI: MADT (v001 A M I OEMAPIC 0x06000411 MSFT 0x00000097) @ 0x7f730390
ACPI: DSDT (v001 APPM1 APPM1202 0x00000202 INTL 0x02002026) @ 0x00000000
ACPI: PM-Timer IO Port: 0x808
ACPI: Local APIC address 0xfee00000
ACPI: LAPIC (acpi_id[0x01] lapic_id[0x00] enabled)
Processor #0 15:3 APIC version 20
ACPI: LAPIC (acpi_id[0x02] lapic_id[0x01] enabled)
Processor #1 15:3 APIC version 20
ACPI: IOAPIC (id[0x02] address[0xfec00000] gsi_base[0])
IOAPIC[0]: apic_id 2, version 32, address 0xfec00000, GSI 0-23
ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl)
ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level)
ACPI: IRQ0 used by override.
ACPI: IRQ2 used by override.
ACPI: IRQ9 used by override.
Enabling APIC mode: Flat. Using 1 I/O APICs
Using ACPI (MADT) for SMP configuration information
Built 1 zonelists
Kernel command line: ro root=LABEL=/
mapped 4G/4G trampoline to fffec000.
Initializing CPU#0
CPU 0 irqstacks, hard=023da000 soft=023ba000
PID hash table entries: 4096 (order: 12, 65536 bytes)
Detected 2800.085 MHz processor.
Using tsc for high-res timesource
Console: colour VGA+ 80x25
Dentry cache hash table entries: 262144 (order: 8, 1048576 bytes)
Inode-cache hash table entries: 131072 (order: 7, 524288 bytes)
Memory: 2064744k/2088128k available (1796k kernel code, 22692k reserved, 756k data, 188k init, 0k highmem)
Calibrating delay loop... 5521.40 BogoMIPS (lpj=2760704)
Security Scaffold v1.0.0 initialized
SELinux: Initializing.
SELinux: Starting in permissive mode
There is already a security framework initialized, register_security failed.
selinux_register_security: Registering secondary module capability
Capability LSM initialized as secondary
Mount-cache hash table entries: 512 (order: 0, 4096 bytes)
CPU: After generic identify, caps: bfebfbff 00000000 00000000 00000000
CPU: After vendor identify, caps: bfebfbff 00000000 00000000 00000000
monitor/mwait feature present.
using mwait in idle threads.
CPU: Trace cache: 12K uops, L1 D cache: 16K
CPU: L2 cache: 1024K
CPU: Physical Processor ID: 0
CPU: After all inits, caps: bfebf3ff 00000000 00000000 00000080
Intel machine check architecture supported.
Intel machine check reporting enabled on CPU#0.
CPU0: Intel P4/Xeon Extended MCE MSRs (12) available
CPU0: Thermal monitoring enabled
Enabling fast FPU save and restore... done.
Enabling unmasked SIMD FPU exception support... done.
Checking 'hlt' instruction... OK.
CPU0: Intel(R) Pentium(R) 4 CPU 2.80GHz stepping 03
per-CPU timeslice cutoff: 2925.31 usecs.
task migration cache decay timeout: 3 msecs.
Booting processor 1/1 eip 3000
CPU 1 irqstacks, hard=023db000 soft=023bb000
Initializing CPU#1
Calibrating delay loop... 5586.94 BogoMIPS (lpj=2793472)
CPU: After generic identify, caps: bfebfbff 00000000 00000000 00000000
CPU: After vendor identify, caps: bfebfbff 00000000 00000000 00000000
monitor/mwait feature present.
CPU: Trace cache: 12K uops, L1 D cache: 16K
CPU: L2 cache: 1024K
CPU: Physical Processor ID: 0
CPU: After all inits, caps: bfebf3ff 00000000 00000000 00000080
Intel machine check architecture supported.
Intel machine check reporting enabled on CPU#1.
CPU1: Intel P4/Xeon Extended MCE MSRs (12) available
CPU1: Thermal monitoring enabled
CPU1: Intel(R) Pentium(R) 4 CPU 2.80GHz stepping 03
Total of 2 processors activated (11108.35 BogoMIPS).
ENABLING IO-APIC IRQs
..TIMER: vector=0x31 pin1=2 pin2=-1
checking TSC synchronization across 2 CPUs: passed.
Brought up 2 CPUs
zapping low mappings.
checking if image is initramfs...it isn't (no cpio magic); looks like an initrd
Freeing initrd memory: 193k freed
NET: Registered protocol family 16
PCI: PCI BIOS revision 2.10 entry at 0xf0031, last bus=1
PCI: Using configuration type 1
mtrr: v2.0 (20020519)
ACPI: Subsystem revision 20040816
ACPI: Interpreter enabled
ACPI: Using IOAPIC for interrupt routing
ACPI: PCI Root Bridge [PCI0] (00:00)
PCI: Probing PCI hardware (bus 00)
PCI: Ignoring BAR0-3 of IDE controller 0000:00:1f.1
PCI: Transparent bridge - 0000:00:1e.0
ACPI: PCI Interrupt Routing Table [\_SB_.PCI0._PRT]
ACPI: PCI Interrupt Routing Table [\_SB_.PCI0.P0P4._PRT]
ACPI: PCI Interrupt Link [LNKA] (IRQs 3 4 5 6 7 *10 11 12 14 15)
ACPI: PCI Interrupt Link [LNKB] (IRQs 3 4 5 6 7 10 *11 12 14 15)
ACPI: PCI Interrupt Link [LNKC] (IRQs 3 4 *5 6 7 10 11 12 14 15)
ACPI: PCI Interrupt Link [LNKD] (IRQs 3 4 5 6 7 10 11 12 14 15) *0, disabled.
ACPI: PCI Interrupt Link [LNKE] (IRQs 3 4 5 6 7 10 11 12 14 15) *0, disabled.
ACPI: PCI Interrupt Link [LNKF] (IRQs 3 4 5 6 7 10 11 12 14 15) *0, disabled.
ACPI: PCI Interrupt Link [LNKG] (IRQs 3 4 5 6 7 10 11 12 14 15) *0, disabled.
ACPI: PCI Interrupt Link [LNKH] (IRQs *3 4 5 6 7 10 11 12 14 15)
Linux Plug and Play Support v0.97 (c) Adam Belay
usbcore: registered new driver usbfs
usbcore: registered new driver hub
PCI: Using ACPI for IRQ routing
ACPI: PCI interrupt 0000:00:02.0[A] -> GSI 16 (level, low) -> IRQ 169
ACPI: PCI interrupt 0000:00:1f.1[A] -> GSI 18 (level, low) -> IRQ 177
ACPI: PCI interrupt 0000:00:1f.3[B] -> GSI 17 (level, low) -> IRQ 185
ACPI: PCI interrupt 0000:00:1f.5[B] -> GSI 17 (level, low) -> IRQ 185
ACPI: PCI interrupt 0000:01:0d.0[A] -> GSI 23 (level, low) -> IRQ 193
apm: BIOS not found.
audit: initializing netlink socket (disabled)
audit(1142289036.882:0): initialized
Total HugeTLB memory allocated, 0
VFS: Disk quotas dquot_6.5.1
Dquot-cache hash table entries: 1024 (order 0, 4096 bytes)
SELinux: Registering netfilter hooks
Initializing Cryptographic API
ksign: Installing public key data
Loading keyring
- Added public key 5501D0A55CA2BC19
- User ID: Red Hat, Inc. (Kernel Module GPG key)
pci_hotplug: PCI Hot Plug PCI Core version: 0.5
vesafb: probe of vesafb0 failed with error -6
ACPI: Processor [CPU1] (supports C1)
ACPI: Processor [CPU2] (supports C1)
isapnp: Scanning for PnP cards...
isapnp: No Plug & Play device found
Real Time Clock Driver v1.12
Linux agpgart interface v0.100 (c) Dave Jones
agpgart: Detected an Intel 865 Chipset.
agpgart: Maximum main memory to use for agp memory: 1911M
agpgart: Detected 8060K stolen memory.
agpgart: AGP aperture is 128M @ 0xf0000000
serio: i8042 AUX port at 0x60,0x64 irq 12
serio: i8042 KBD port at 0x60,0x64 irq 1
Serial: 8250/16550 driver $Revision: 1.90 $ 8 ports, IRQ sharing enabled
ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A
RAMDISK driver initialized: 16 RAM disks of 16384K size 1024 blocksize
divert: not allocating divert_blk for non-ethernet device lo
Uniform Multi-Platform E-IDE driver Revision: 7.00alpha2
ide: Assuming 33MHz system bus speed for PIO modes; override with idebus=xx
ICH5: IDE controller at PCI slot 0000:00:1f.1
PCI: Enabling device 0000:00:1f.1 (0005 -> 0007)
ACPI: PCI interrupt 0000:00:1f.1[A] -> GSI 18 (level, low) -> IRQ 177
ICH5: chipset revision 2
ICH5: not 100% native mode: will probe irqs later
ide0: BM-DMA at 0xfc00-0xfc07, BIOS settings: hda:DMA, hdb:pio
ide1: BM-DMA at 0xfc08-0xfc0f, BIOS settings: hdc:DMA, hdd:pio
Probing IDE interface ide0...
hda: WDC WD400BB-00FJA0, ATA DISK drive
Using cfq io scheduler
ide0 at 0x1f0-0x1f7,0x3f6 on irq 14
Probing IDE interface ide1...
hdc: ASUS CD-S520/A4, ATAPI CD/DVD-ROM drive
ide1 at 0x170-0x177,0x376 on irq 15
Probing IDE interface ide2...
ide2: Wait for ready failed before probe !
Probing IDE interface ide3...
ide3: Wait for ready failed before probe !
Probing IDE interface ide4...
ide4: Wait for ready failed before probe !
Probing IDE interface ide5...
ide5: Wait for ready failed before probe !
hda: max request size: 128KiB
hda: 78165360 sectors (40020 MB) w/2048KiB Cache, CHS=65535/16/63, UDMA(100)
hda: cache flushes supported
hda: hda1 hda2 hda3
hdc: ATAPI 52X CD-ROM drive, 128kB Cache, UDMA(33)
Uniform CD-ROM driver Revision: 3.20
ide-floppy driver 0.99.newide
usbcore: registered new driver hiddev
usbcore: registered new driver usbhid
drivers/usb/input/hid-core.c: v2.0:USB HID core driver
mice: PS/2 mouse device common for all mice
md: md driver 0.90.0 MAX_MD_DEVS=256, MD_SB_DISKS=27
NET: Registered protocol family 2
IP: routing cache hash table of 8192 buckets, 128Kbytes
TCP: Hash tables configured (established 262144 bind 43690)
Initializing IPsec netlink socket
NET: Registered protocol family 1
NET: Registered protocol family 17
ACPI: (supports S0 S1 S3 S4 S5)
ACPI wakeup devices:
P0P4 RLAN MC97 USB1 USB2 USB3 USB4 EUSB ILAN
md: Autodetecting RAID arrays.
md: autorun ...
md: ... autorun DONE.
RAMDISK: Compressed image found at block 0
VFS: Mounted root (ext2 filesystem).
kjournald starting. Commit interval 5 seconds
EXT3-fs: mounted filesystem with ordered data mode.
Freeing unused kernel memory: 188k freed
SELinux: Disabled at runtime.
SELinux: Unregistering netfilter hooks
ACPI: Power Button (FF) [PWRF]
EXT3 FS on hda3, internal journal
device-mapper: 4.1.0-ioctl (2003-12-10) initialised: dm@uk.sistina.com
cdrom: open failed.
Adding 1048312k swap on /dev/hda2. Priority:-1 extents:1
kjournald starting. Commit interval 5 seconds
EXT3 FS on hda1, internal journal
EXT3-fs: mounted filesystem with ordered data mode.
What's the best way to check for resolution in both directions?
Thanks for the help!
voidinit
03-17-2006, 02:30 AM
Usually in dmesg errors begin with something obnoxious like "ERROR: BlahBlahBlah". Then that error is repeated all the time, especially if it's a disk or filesystem error. It seems like all your hardware and filesystems are checking out ok. As for resolution, you can use the dig utility or nslookup utility to check for forward or reverse resolution.
The Linux Kid
03-17-2006, 02:47 AM
Check mailq - I had a server that was doing some funny things and it turned out that someone had managed to make it an open relay (probably something I did wrong)
Syngin
03-17-2006, 07:10 PM
I agree with the mail server issue mentioned especially with Qmail. I had a heck of a time troubleshooting a slow system from top until I ran ps -aux and saw all of the qmail-remote processes. Singularly they weren't much but collectively they were contributing a lot.
Does it still do the same thing after a reboot? (last ditch effort of a sysadmin hehe)
Post your php.ini file too.
Lexx
03-23-2006, 02:46 PM
Turns out it was spamd chewing the hell out of the drive. Once turned off, things are flying again, thanks for the help.
justlinux.com
Copyright Internet.com Inc. All Rights Reserved.