How can I record what process or kernel activity is using the disk in GNU/Linux?

Question

On a particular Debian server, iostat (and similar) report an unexpectedly high volume (in bytes) of disk writes going on. I am having trouble working out which process is doing these writes.

Two interesting points:

1. Tried turning off system services one at a time to no avail. Disk activity remains fairly constant and unexpectedly high.

2. Despite the writing, do not seem to be consuming more overall space on the disk.

Both of those make me think that the writing may be something that the kernel is doing, but I'm not swapping, so it's not clear to me what Linux might try to write.

Could try out atop:

http://www.atcomputing.nl/Tools/atop/

but would like to avoid patching my kernel.

Any ideas on how to track this down?

Answer 1

iotop is good (great, actually).

If you have a kernel from before 2.6.20, you can't use most of these tools.

Instead, you can try the following (which should work for almost any 2.6 kernel IIRC):

    
sudo -s
dmesg -c
/etc/init.d/klogd stop
echo 1 > /proc/sys/vm/block_dump
rm /tmp/disklog
watch "dmesg -c >> /tmp/disklog"
   CTRL-C when you're done collecting data
echo 0 > /proc/sys/vm/block_dump
/etc/init.d/klogd start
exit (quit root shell)

cat /tmp/disklog | awk -F"[() \t]" '/(READ|WRITE|dirtied)/ {activity[$1]++} END {for (x in activity) print x, activity[x]}'| sort -nr -k2

The dmesg -c lines clear your kernel log . The logger is then shut off, manually (using watch) dumped to a disk (the memory buffer is small, which is why we need to do this). Let it run for about five minutes or so, and then CTRL-c the watch process. After shutting off the logging and restarting klogd, analyze the results using the little bit of awk at the end.