Ask Doctor Shell ...
This site uses stylesheets and XHTML1.1. You might want to switch to a modern browser like Firefox, Mozilla, or maybe IE6.

Something keeps filling up the root filesystem ...

Now I am trying to run a find command that will tell me what files have been accessed in the last hour. I have tried find / -atime 1 -mount -print, but it only returns two cron files. I know more files have been accessed in root in the last day. Why can't I find them?

Solution:

That's a nice problem. You could try -mtime or -ctime instead of -atime, but if someone is filling up your disk, chances are good that you will never see a file.

Why?

When a new file is created, it gets an entry in the inode table on disk and an entry in the directory that associates inode number and file name. If the file is opened by a process, the kernel increments a count in its internal file table. If this count is greater zero, the file is kept open by at least one process and cannot be physically removed on disk. What you may remove is the directory entry, so that you can no longer find the file via it's directory. But even if the directory entry has been removed, the file is still there and can still be accessed by all processes which have it open already.

Try this, to observe this behavior:

# get current state
df -k /tmp
du -sk /tmp

# fill up disk in the background ...
find / -print >/tmp/dummy 2>&1 & # fill up disk in the background

# ... and remove the directory entry:
rm /tmp/dummy

# wait some time ...
sleep 5

# ... and observe, what's happening:
df -k /tmp
du -sk /tmp

# now, that you've seen this, kill the background process ...
kill 0

# ... and assure, that the file disappeared:
df -k /tmp
du -sk /tmp

If you put these commands in a script file, e.g. "filldisk", you have to source it with the dot command. If not, kill 0 will kill the script as well and you won't see the output of the last 2 commands:

. filldisk

From the output of these commands you will see that the available disk space shrinks, while the du command always reports the same disk usage (if there are no interfering processes). This is, because the "dummy" file is open, so that the find can write to it and also acquire new disk blocks. But the file has no directory entry, so du doesn't see it and cannot count its blocks.

That's why you can't see a new file. And how can you find out, what's happening?

In such a situation, a file has been opened and removed afterwards. Removing a file changes the content of a directory. You could therefore look for modified directories:

find / -type d -mtime 1 -xdev -ls

This will give you more than you need, but you will get an indication, where something is happening in your directory tree.

Look at your processes - are there any suspects?

You could also look at your suspect directories by reading them like ordinary files: e.g. with a plain strings directoryname or: od -c . or cat -q . Are there any suspicious names?

Enjoy!

Valid XHTML 1.1! Valid CSS!

@(#) $Id: 0001.shtml,v 1.7 2005/06/24 04:49:44 kdo Exp $