We have this apache process which becomes io-bound time to time. Using atop, we can see it is a write operation.

Using lsof -p <PID> we can see a list of files open by the httpd process. First we thought "log" files must be the problem. So we turned them off just to test. However write operations still continues.

We will continue testing a few other things. For instance we use php session variables a lot. Maybe php session files are getting all the writing.

But is there a way to quickly identify files which get written to by the httpd process? This way we can focus our efforts on those files.

UPDATE:

We used the strace command as suggested. Here are two lines from the output.

write(23, "\27\0\0\0\3SET CHARACTER SET utf8", 27) = 27
write(23, "\17\0\0\0\3SET NAMES utf8", 19) = 19

We do not have a mysql process on this server. So is strace also showing what is being written to an ethernet port?

UPDATE2: During high io load, the process which consumes most of the write resources gives the following output to strace -e trace=write -p <PID>:

--- SIGCHLD (Child exited) @ 0 (0) ---
write(9, "!", 1) = 1
write(19, "OPTIONS * HTTP/1.0\r\nUser-Agent: Apache (internal dummy connection)\r\n\r\n", 70) = 70

However I cannot figure out where these are being written to.