Interesting AIX tip
Ever wonder were all the space went in a file system? Does du and df show different results on the same file system? Chances are some one deleted a file that was open in a particular file system. For example the /tmp file system on one of our machines has been filling up and then some one was deleting files from it. But the space is never recovered. In the AIX errpt you may see something like this:
IDENTIFIER: 369D049B
Date/Time: Wed Sep 27 16:58:08 2006
Sequence Number: 780
Machine Id: 002AA9AF4C00
Node Id: aixbox
Class: O
Type: INFO
Resource Name: SYSPFS
Description
UNABLE TO ALLOCATE SPACE IN FILE SYSTEM
Probable Causes
FILE SYSTEM FULL
Recommended Actions
USE FUSER UTILITY TO LOCATE UNLINKED FILES STILL REFERENCED
INCREASE THE SIZE OF THE ASSOCIATED FILE SYSTEM
REMOVE UNNECESSARY DATA FROM FILE SYSTEM
Detail Data
MAJOR/MINOR DEVICE NUMBER
000A 0007
FILE SYSTEM DEVICE AND MOUNT POINT
/dev/hd3, /tmp
The hint to finding the files is given in the first line of the recommended actions. Use the fuser command. The actual command is:
What this will show you is something similar to this:
/tmp:
inode=34 size=675155 fd=0 200858
inode=43 size=114531 fd=0 286764
inode=66 size=59021846 fd=0 335986
inode=77 size=2322588 fd=0 389232
inode=46 size=601938 fd=0 413872
inode=61 size=28498 fd=0 430332
inode=44 size=1280774965 fd=0 434292
inode=40 size=2884063 fd=0 442590
inode=51 size=2395908 fd=0 467132
inode=73 size=8224333 fd=0 479402
inode=42 size=140607 fd=0 524474
inode=64 size=163405 fd=0 553054
inode=49 size=350562 fd=0 618644
inode=63 size=2375730 fd=0 663568
inode=74 size=3372392 fd=0 696356
inode=58 size=65535 fd=0 819204
inode=57 size=424777 fd=0 1106024
inode=62 size=2030397 fd=0 1147064
inode=76 size=57187 fd=0 1163494
inode=31 size=1376255 fd=0 1171540
inode=56 size=53834 fd=0 1216530
inode=52 size=361520961 fd=0 1278152
inode=81 size=15972886 fd=0 1294462
inode=70 size=13390097 fd=0 1323224
inode=60 size=7559 fd=0 1380400
inode=50 size=40132 fd=0 1429514
inode=65 size=720895 fd=0 1450220
inode=79 size=12582477 fd=0 1507350
inode=47 size=169682 fd=0 1593582
inode=48 size=259432 fd=0 1605642
inode=78 size=1488191 fd=0 1671280
The column we are really interested in is the last one. That is the column that contains the Process ID of the process that has a particular file open in that file system. So for example process id 1671280 has inode 78 in the /tmp file system open, and that file is using ~1.4MB of space. So if you “kill 1671280″ you would recover the 1.4MB of space. (But make sure you kill it in the proper way, i.e. find out what the process really is and shut it down nicely.)
A pity that it didn’t work by me
I have the same problem but the /tmp doesn’t show anything.
/root$ fuser -dV /tmp
/tmp: