Threads and processes on Linux 3270

Loic,

Thanks for the post - very helpful - it clears a few things up for me!

I'm going to be cheeky and ask you a question that I posted below about a mulbreasthreaded bit of code I'm writing. Just incase you didn't see it :) I've got a 2.4 kernel with linuxthreads, and an app with about 300 threads doing network I-O in 20 millisecond bursts. On a single CPU box, this is fine. On an SMP box, performance drops through the floor. There are no data pages shared between threads. vmstat, etc. show the processors 60% idle.

My theory is that each thread is being repeatedly scheduled on a different CPU, and so a lot of time is being spent loading the memory accessed by the thread into the CPU cache, and then (once it's dirtied) invalidating the cache entries on the last processor to host it. Am I in the right ballpark? Even playing the right sport?

I hacked a box so it would run the thread affinity stuff added to the 2.5.8 kernel, but i'm not sure of my success. Binding each thread to a particular CPU made no difference. Binding them all to a single CPU (id 0) made a *huge* difference, but it still wasn't as good as uniprocessor.

I only got thread affinity - I didn't get interrupt affinity. My next theory is interrupts. The 2.4 kernel round-robins responsibility for handling interrupts around the cpus - each receives them for a short period. Each I-O operation generates an interrupt. If it's not delivered to the correct processor (75% chance), time is wasted transmitting the information between the cpu receiving the interrupt and the cpu running the thread. Ok - what sport am I playing now?

Phew. That's a lot of gumph. Upshot: - are my theories anywhere correct? And if not, what could be causing the performance drop on SMP? (I realise this is impossible to answer! But there's no data sharing, all objects and per-thread memory accesses lie on per-thread memory pages, there's no swapping going on, all data is resident in main memory, processors are barely sweating, aaarrggh!) - are there any reliable interrupt-thread affinity patches for 2.4 (being lazy here - I will try to find out myself) - is this function available in 2.6 and-or NTPL? (ditto)

Thanks for all the useful info in the post.

Doug