Threads and processes on Linux 3271

sneck!

Profiling shows ... what?

I would look more closely at synchronization, implicit or explicit. If you're doing a lot of network I-O, it's possible that the overhead is going into the network stack (I don't know if the Linux network stack is multi-threaded), so you're possibly winding up waiting for that a fair amount.

The first 90% of the time, the problem you are seeing happens when there is heavy mutex-other sync primitive contention. The reason it runs better on a uniprocessor is that due to the nature of the time slices, threads wind up blocking in places (like I-Os) where they don't have things locked. When the box has multiple processors, suddenly the real contention rates go up.

The next 9% of the time, the problem is cache line thrashing.

The last 1% is various oddball effects, such as you're theorizing with interrupt affinity. It's not impossible that it's the problem, but it's also not very likely.

It's been a while since I looked at Linux profiling tools, so I don't know how well gprof will show where the time is going, but that should be your first line of attack. There were once companies (AMC, at least) that did hardware profiling tools, which made debugging your sort of problem somewhat easier, but I don't think they're around any more. -- Steve Watt KD6GGD PP-ASEL-IA ICBM: 121W 56' 57.8" 37N 20' 14.9" Free time? There's no such thing. It just comes in varying prices...