I have used cpuset to shield some cpus for exclusive use by some realtime threads.

Displaying the cpuset config with the test app RealtimeTest1 running and its tasks moved into the cpusets:

$ cset set --list -r
cset: 
         Name         CPUs-X    MEMs-X Tasks Subs Path
 ------------   ---------- - ------- - ----- ---- ----------
         root         0-23 y     0-1 y   279    2 /
       system 0,2,4,6,8,10 n       0 n   202    0 /system
       shield 1,3,5,7,9,11 n       1 n     0    2 /shield
RealtimeTest1      1,3,5,7 n       1 n     0    4 /shield/RealtimeTest1
      thread1            3 n       1 n     1    0 /shield/RealtimeTest1/thread1
      thread2            5 n       1 n     1    0 /shield/RealtimeTest1/thread2
         main            1 n       1 n     1    0 /shield/RealtimeTest1/main

I can interrogate the cpuset filesystem to show that my tasks are supposedly pinned to the cpus I requested:

/cpusets/shield/RealtimeTest1 $ for i in `find -name tasks`; do echo $i; cat $i; echo "------------"; done

./thread1/tasks
17651 
------------
./main/tasks
17649 
------------
./thread2/tasks
17654
------------

Further, if I use sched_getaffinity, it reports what cpuset does - that thread1 is on cpu 3 and thread2 is on cpu 5.

However, if I run top -p 17649 -H with f,j to bring up the last used cpu, it shows that thread 1 is running on thread 2's cpu, and main thread is running on a cpu in the system cpuset

(Note that thread 17654 is running FIFO, hence thread 17651 is blocked)

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  P COMMAND                                                                                                                                                                                             
17654 root      -2   0 54080  35m 7064 R  100  0.4   5:00.77 3 RealtimeTest                                                                                                                                                                                         
17649 root      20   0 54080  35m 7064 S    0  0.4   0:00.05 2 RealtimeTest                                                                                                                                                                                         
17651 root      20   0 54080  35m 7064 R    0  0.4   0:00.00 3 RealtimeTest    

Also, looking at /proc/17649/task to find the last_cpu each of its tasks ran on:

/proc/17649/task $  for i in `ls -1`; do cat $i/stat | awk '{print $1 " is on " $(NF - 5)}'; done
17649 is on 2
17651 is on 3
17654 is on 3

So cpuset and sched_getaffinity reports one thing, but reality is another

I would say that cpuset is not working?

My machine configuration is:

$ cat /etc/SuSE-release
SUSE Linux Enterprise Server 11 (x86_64)
VERSION = 11
PATCHLEVEL = 1
$ uname -a
Linux foobar 2.6.32.12-0.7-default #1 SMP 2010-05-20 11:14:20 +0200 x86_64 x86_64 x86_64 GNU/Linux