[Solved] Disk space eaten by UNKNOWN ( Culprit Process )

When suddenly all the available disk space on / has disappeared. If i make room in the disk (by deleting ~50GB of stuff, for example), after a few minutes I am back to 0 available disk space (according to df).
Clearly, some process is eating up disk space at a rapid rate, but I can't figure out what it is.
One thing is certain, though: whatever it is, it must be creating many small files, because there are no files bigger than 10GB on the disk, and all the ones bigger than 1GB are much older than today.

And the question is ""How can I find what's eating up disk space?"
[root@valuecard mnt]# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/vg_valuecard-lv_root 50G 47G 488M 99% /
tmpfs 2.0G 0 2.0G 0% /dev/shm /dev/mapper/ddf1_4c5349202020202080862682000000004711471100000a28p1 485M 63M 397M 14% /boot
/dev/mapper/vg_valuecard-lv_home 175G 24G 143G 14% /home

Root partitiion is mesh up... , then I run du to find what is eating my disk space

 [root@valuecard /]# du -h --max-depth=1
. 210M
 lib 14M
 ./sbin 0
 ./misc 188K
 ./dev 4.0K
 ./srv
du: cannot access `./proc/25765/task/25765/fd/4': No such file or directory
du: cannot access `./proc/25765/task/25765/fdinfo/4': No such file or directory
du: cannot access `./proc/25765/fd/4': No such file or directory
du: cannot access `./proc/25765/fdinfo/4': No such file or directory 0
./proc 4.0K
 ./home 144K
 ./root 53M
 ./boot 7.3M
 ./bin 6.4M
 ./share 0
 ./sys 4.2G
 ./var 16K
 ./lost+found 20K
 ./tmp 0
 ./selinux 4.0K
 ./cgroup 1.7G
 ./usr 87M
 ./mnt 0
 ./net 29M
 ./etc 8.0K
./opt 30G .

If the calculation betwen free space and du command not match, nah that mean You are dealing with deleted files, that is why du does not register used space, but df does.
Deleted files only disappear after the owner process is stopped; they remain in use while that does not happen. and it's called "Culprit process".

  So to find the culprit process, I recommend you doing this command:   lsof -nP | grep '(deleted)'

[root@valuecard mnt]# lsof -nP | grep '(deleted)'
php-fpm 2175 root txt REG 253,3 4017744 2108436 /usr/sbin/php-fpm (deleted)
php-fpm 2175 root 4u REG 253,3 0 1966095 /tmp/.ZendSem.ceje6o (deleted)
php-fpm 2176 data txt REG 253,3 4017744 2108436 /usr/sbin/php-fpm (deleted)
php-fpm 2176 data 4u REG 253,3 0 1966095 /tmp/.ZendSem.ceje6o (deleted)
php-fpm 2177 data txt REG 253,3 4017744 2108436 /usr/sbin/php-fpm (deleted)
php-fpm 2177 data 4u REG 253,3 0 1966095 /tmp/.ZendSem.ceje6o (deleted)
php-fpm 2178 data txt REG 253,3 4017744 2108436 /usr/sbin/php-fpm (deleted)
php-fpm 2178 data 4u REG 253,3 0 1966095 /tmp/.ZendSem.ceje6o (deleted)
php-fpm 2179 data txt REG 253,3 4017744 2108436 /usr/sbin/php-fpm (deleted)
php-fpm 2179 data 4u REG 253,3 0 1966095 /tmp/.ZendSem.ceje6o (deleted)
php-fpm 2180 data txt REG 253,3 4017744 2108436 /usr/sbin/php-fpm (deleted)
php-fpm 2180 data 4u REG 253,3 0 1966095 /tmp/.ZendSem.ceje6o (deleted)
certmonge 2239 root txt REG 253,3 245668 2108803 /usr/sbin/certmonger (deleted)
mysqld 8264 mysql 1w REG 253,3 40960 2490572 /var/log/mysqld.log-20170930 (deleted)
mysqld 8264 mysql 2w REG 253,3 40960 2490572 /var/log/mysqld.log-20170931 (deleted)
mysqld 8264 mysql 5u REG 253,3 0 1966083 /tmp/ibH9SS9X (deleted)
mysqld 8264 mysql 6u REG 253,3 0 1966084 /tmp/ibBhluEV (deleted)
mysqld 8264 mysql 7u REG 253,3 0 1966085 /tmp/ibPrt68S (deleted)
mysqld 8264 mysql 8u REG 253,3 118784 1966086 /tmp/ib1ncybO (deleted)
mysqld 8264 mysql 11w REG 253,3 42880016330 2490615 /var/log/mysql-slow.log-20170928 (deleted)
mysqld 8264 mysql 12u REG 253,3 0 1966087 /tmp/ibdtjnTL (deleted)
httpd 11609 root 72u REG 253,3 0 1966091 /tmp/.ZendSem.KD4Via (deleted)
httpd 18913 data 72u REG 253,3 0 1966091 /tmp/.ZendSem.KD4Via (deleted)

as you can see, mysql-slow.log look like with big size, and all you can do is kill that process

 --kill the proses--
 [root@valuecard mnt]# kill -9 8264

 --result--
 [root@valuecard mnt]# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/vg_valuecard-lv_root 50G 6.4G 41G 14% /
tmpfs 2.0G 0 2.0G 0% /dev/shm /dev/mapper/ddf1_4c5349202020202080862682000000004711471100000a28p1 485M 63M 397M 14% /boot
/dev/mapper/vg_valuecard-lv_home 175G 24G 143G 14% /home

and everyone is happy... ^_^

Previous
Next Post »

comment please ... ConversionConversion EmoticonEmoticon

Thanks for your comment