I'm using async io on linux with rtkaio library. In my tests everything works perfectly, but, in my real application i see that aio_write which is supposed to return very fast, is very slow. It can take more than 100 milis to write a 128KB to a O_DIRECT padded file. Both my test and the application use same I/O size, i check on the same file system (GFS).

I added counting and i see that there are about 50% of async io operations that are short (shorter then 2 milis) and 50% that are long (longer than 2 milis).

I also checked that the test and the application both use the same rtkaio library.

I'm pretty lost, anyone any ideas where should i look?

Another my related question: http://stackoverflow.com/questions/1799537/proc-sys-fs-aio-nr-is-never-higher-than-1024-aio-on-linux