程序创建过多临时文件，导致系统inode不足问题排查

Author Avatar
Whark 8月 25, 2019

尝试运行 selenium 程序，发现如下报错

easyprocess.EasyProcessCheckInstalledError: cmd=['Xvfb', '-help']
OSError=None
Program install error!
You can install it in terminal:
sudo apt-get install xvfb

尝试安装 xvfb

sudo apt-get install xvfb

发现提示磁盘空间不足，No space left on device

dpkg: unrecoverable fatal error, aborting:
unable to create '/var/lib/dpkg/updates/tmp.i': No space left on device

查看磁盘剩余空间，发现空间还很充裕

root@aliyun:~/python# df -h

Filesystem      Size  Used Avail Use% Mounted on
udev            3.9G     0  3.9G   0% /dev
tmpfs           798M  3.1M  795M   1% /run
/dev/vda1        40G   14G   24G  37% /
tmpfs           3.9G     0  3.9G   0% /dev/shm
tmpfs           5.0M     0  5.0M   0% /run/lock
tmpfs           3.9G     0  3.9G   0% /sys/fs/cgroup
/dev/vdb1       196G   90G   96G  49% /opt
/dev/vdc1        98G   11G   83G  12% /data
tmpfs           798M     0  798M   0% /run/user/0

再次尝试安装 xvfb，安装成功，尝试运行 saver，错误依旧

测试运行最基础的 selenium 功能

from selenium import webdriver
from selenium.webdriver.chrome.options import Options
from pyvirtualdisplay import Display

display = Display(visible=0, size=(1920, 1000))
display.start()
options = Options()
options.add_argument("start-maximized")
driver = webdriver.Chrome()
driver.get('https://www.google.com')
print("Page Title is : %s" %driver.title)
driver.quit()
display.stop()

发现如下报错
unable to create ‘/tmp/.com.google.Chrome.DFid4m’: No space left on device

到 /tmp 下查看，发现超级多 .com.google.Chrome 开头的文件，猜想可能是因为很多小文件，把系统的 inode 用完了

查看磁盘inode使用情况，发现系统inode的确已经用完了

root@aliyun:~/python# df -i

Filesystem       Inodes   IUsed    IFree IUse% Mounted on
udev            1013942     420  1013522    1% /dev
tmpfs           1021032    1105  1019927    1% /run
/dev/vda1       2621440 2621438        2  100% /
tmpfs           1021032       1  1021031    1% /dev/shm
tmpfs           1021032       2  1021030    1% /run/lock
tmpfs           1021032      18  1021014    1% /sys/fs/cgroup
/dev/vdb1      13107200  690724 12416476    6% /opt
/dev/vdc1       6553600  120731  6432869    2% /data
tmpfs           1021032     275  1020757    1% /run/user/0

尝试使用 *rm -rf /tmp/.com.google.Chrome.** 删除，发现如下报错

-bash: /bin/rm: Argument list too long

尝试多种方法想快速删除，无果（/tmp 文件夹下有如 tmux-0 的文件，tmux 的 session 保存在这个文件夹，如果删掉，tmux可能会出问题，所以选择只删除 chrome 的临时文件，保留其它文件）

想到了一种笨办法：

rm -rf /tmp/.com.google.Chrome.1*
rm -rf /tmp/.com.google.Chrome.2*
...
rm -rf /tmp/.com.google.Chrome.z*

于是就有了下面的这个shell

#!/bin/bash

chars=("A" "B" "C" "D" "E" "F" "G" "H" "I" "J" "K" "L" "M" "N" "O" "P" "Q" "R" "S" "T" "U" "V" "W" "X" "Y" "Z" "a" "b" "c" "d" "e" "f" "g" "h" "i" "j" "k" "l" "m" "n" "o" "p" "q" "r" "s" "t" "u" "v" "w" "x" "y" "z" "1" "2" "3" "4" "5" "6" "7" "8" "9" "0")
for value in ${chars[@]}
do
{
    echo "/tmp/.com.google.Chrome."$value"*"
    rm -rf /tmp/.com.google.Chrome."$value"*
}&
done

再次查看 inode，恢复正常，启动 saver 正常运行

root@aliyun:/tmp# df -i

Filesystem       Inodes   IUsed    IFree IUse% Mounted on 
udev            1013942     420  1013522    1% /dev 
tmpfs           1021032    1105  1019927    1% /run 
/dev/vda1       2621440 1634826   986614   63% / 
tmpfs           1021032       1  1021031    1% /dev/shm 
tmpfs           1021032       2  1021030    1% /run/lock 
tmpfs           1021032      18  1021014    1% /sys/fs/cgroup 
/dev/vdb1      13107200  691129 12416071    6% /opt 
/dev/vdc1       6553600  120747  6432853    2% /data 
tmpfs           1021032     275  1020757    1% /run/user/0
expand_less