docker服务执行启动或其他命令时夯住问题解决-工具盒子

日常巡检时发现项目系统docker服务运行异常，无法使用docker ps等命令，正常启动docker服务夯住。通过开启debug模式，发现进程卡在activateDeviceIfNeeded处。

[root@localhost ~]# dockerd -D
DEBU[0000] docker group found. gid: 985     
DEBU[0000] Listener created for HTTP on unix (/var/run/docker.sock)
INFO[0000] libcontainerd: new containerd process, pid: 2053
DEBU[0000] containerd: read past events                  count=0
DEBU[0000] containerd: supervisor running                cpus=24 memory=63868 runtime=docker-runc runtimeArgs=[] stateDir=/var/run/docker/libcontainerd/containerd
DEBU[0000] containerd: grpc api on /var/run/docker/libcontainerd/docker-containerd.sock
DEBU[0000] libcontainerd: containerd health check returned error: rpc error: code = 14 desc = grpc: the connection is unavailable
DEBU[0001] Using default logging driver json-file
DEBU[0001] Golang's threads limit set to 919440
DEBU[0001] devicemapper: driver version is 4.27.0
DEBU[0001] devmapper: Generated prefix: docker-8:4-6979970053
DEBU[0001] devmapper: Checking for existence of the pool docker-8:4-6979970053-pool
DEBU[0001] devmapper: poolDataMajMin=7:0 poolMetaMajMin=7:1
DEBU[0001] devmapper: Major:Minor for device: /dev/loop0 is:7:0
DEBU[0001] devmapper: Major:Minor for device: /dev/loop1 is:7:1
DEBU[0001] devmapper: loadDeviceFilesOnStart()
DEBU[0001] devmapper: Loading data for file /var/lib/docker/devicemapper/metadata/03e256fbea9ba18116a7d7791db5a3127ca4d32a76c7ffbfd875c5d6249bcb7d
DEBU[0001] devmapper: Loading data for file /var/lib/docker/devicemapper/metadata/0f7dd65f2c27c2e34b121f5f620c7c321e309b049147a339ef79cee96e19f871
DEBU[0001] devmapper: Loading data for file /var/lib/docker/devicemapper/metadata/108e382272adf66ed9ad5163f43ea4b4953aca3d5d421d000d269debd481363c
DEBU[0001] devmapper: Loading data for file /var/lib/docker/devicemapper/metadata/15a4928b9197b5ee222d470e24ca71018509d995f06defc2d11cab58e3d84cb6
DEBU[0001] devmapper: Loading data for file /var/lib/docker/devicemapper/metadata/1c8d670e7d3b00a0fbc6ebe94ecea0216ea3240d8a96b521ad85a3512a776f41
DEBU[0001] devmapper: Loading data for file /var/lib/docker/devicemapper/metadata/2d233975dd78918b70be78cae4e0af0cb3e3e83e26ecfbdea91148147f7a9951
DEBU[0001] devmapper: Loading data for file /var/lib/docker/devicemapper/metadata/555353a25547464a91079e9bff6e798c4c34c9f5d157bcff8480498c739a76e4
DEBU[0001] devmapper: Loading data for file /var/lib/docker/devicemapper/metadata/5e656bebed763c165c36cf19866468f2c87821bc6ad11f55d6695cc456a2f130
DEBU[0001] devmapper: Loading data for file /var/lib/docker/devicemapper/metadata/60aa660dbbc8004ab2818f772041466d20e87ccafe1682a9373b9a5c04806f64
DEBU[0001] devmapper: Loading data for file /var/lib/docker/devicemapper/metadata/6726c7f7b16cbbaec2dd23d7ffcad1775a971fbf965906a7e8766202f6bcd0d5
DEBU[0001] devmapper: Loading data for file /var/lib/docker/devicemapper/metadata/6dde1dadc8a9d9c302d3cd7d8d82fa994736cb324ec914b68f18b8032941b5f3
DEBU[0001] devmapper: Loading data for file /var/lib/docker/devicemapper/metadata/72d1668e470316d4e3b6d5aba3f54fa135888dface99737f789cc42f8e407a67
DEBU[0001] devmapper: Loading data for file /var/lib/docker/devicemapper/metadata/84deeba59ae069d76681fda01e88157c191acacadb58aa126f4bb093a2d42998
DEBU[0001] devmapper: Loading data for file /var/lib/docker/devicemapper/metadata/9d1a83040e86c3ae8571a3a22f1d32ab7736a31aaad92ff46e83857ce3cbeb5d
DEBU[0001] devmapper: Loading data for file /var/lib/docker/devicemapper/metadata/a0540b592034b3b43cf2a8a06a68b42d081a2510411607ef01e8b24ecde00f5b
DEBU[0001] devmapper: Loading data for file /var/lib/docker/devicemapper/metadata/b0653bebc497ff4970e2985a0702c6fe27f4be9fb737766f1d84c3969d4c3259
DEBU[0001] devmapper: Loading data for file /var/lib/docker/devicemapper/metadata/b4dd4ea7f12c3081d816de124736a48e3996c5b39540b84c3f6069f56b3f7d53
DEBU[0001] devmapper: Loading data for file /var/lib/docker/devicemapper/metadata/b60fa4079466cef8c2dcba7f64ce59378e882dddb0bdfa2da224b7f6639981f4
DEBU[0001] devmapper: Loading data for file /var/lib/docker/devicemapper/metadata/b6925abe106ed9aca9351397cfc58bace782aa525d14ffa507d26b28c4efa7ba
DEBU[0001] devmapper: Loading data for file /var/lib/docker/devicemapper/metadata/b6925abe106ed9aca9351397cfc58bace782aa525d14ffa507d26b28c4efa7ba-init
DEBU[0001] devmapper: Loading data for file /var/lib/docker/devicemapper/metadata/base
DEBU[0001] devmapper: Loading data for file /var/lib/docker/devicemapper/metadata/c17763303cb58bcfe72b4267e75de044f8ad6f8c24a3dd329257318769e55d8e
DEBU[0001] devmapper: Loading data for file /var/lib/docker/devicemapper/metadata/c55e6b6b3a82dbd8dc40e782eabeabccd0845dc56b4f713218782637784a7a51
DEBU[0001] devmapper: Loading data for file /var/lib/docker/devicemapper/metadata/c8a693a2e9bf6634cdf0156cc990b91483ff0bca80a1f51c6a3f4915de336a3c
DEBU[0001] devmapper: Loading data for file /var/lib/docker/devicemapper/metadata/cb2214a20c23bd48fd0cbe3e34d97765d7a2f0400d2e8b1e7064d119ed09ea21
DEBU[0001] devmapper: Loading data for file /var/lib/docker/devicemapper/metadata/cd636def3514193903359afc4278164ab09ff38198451792b03e8f2a49565bfd
DEBU[0001] devmapper: Loading data for file /var/lib/docker/devicemapper/metadata/d12d69bcd1fe12911b7bc5aabf08f343b9873927f61c958754ebf2326f14bbc9
DEBU[0001] devmapper: Loading data for file /var/lib/docker/devicemapper/metadata/db22d9b4e3982d0d81a8ef33112a96d2dcb26b634fd824f93a620b5794b46589
DEBU[0001] devmapper: Skipping file /var/lib/docker/devicemapper/metadata/deviceset-metadata
DEBU[0001] devmapper: Loading data for file /var/lib/docker/devicemapper/metadata/e37ad1575371a43be1fd09eb5a84ac5552524dbe9d59d36b7a5131aabe5425bb
DEBU[0001] devmapper: Loading data for file /var/lib/docker/devicemapper/metadata/e49b4dfde600e6d43b9cb55018c48ac906a4ef22ed343685a64a559cf85ea310
DEBU[0001] devmapper: Loading data for file /var/lib/docker/devicemapper/metadata/ec927e08226ad1e1ce529864de989bef905eaf75d845ab05d047bc39437f2da6
DEBU[0001] devmapper: Loading data for file /var/lib/docker/devicemapper/metadata/f6bdce24ba334352f05283898d99961ce434fe8b1f6f8a17ce6b78a9a86f9faa
DEBU[0001] devmapper: Loading data for file /var/lib/docker/devicemapper/metadata/fdc0207c9d0ea4d93728573f06cace914332949eb98d53a8820c368676caccf3
DEBU[0001] devmapper: Loading data for file /var/lib/docker/devicemapper/metadata/fdc0207c9d0ea4d93728573f06cace914332949eb98d53a8820c368676caccf3-init
DEBU[0001] devmapper: Loading data for file /var/lib/docker/devicemapper/metadata/fdda33f56e7eb1ea0bf7d132610b286628de0e914d3f3d5307255f6f09bafae8
DEBU[0001] devmapper: Skipping file /var/lib/docker/devicemapper/metadata/transaction-metadata
DEBU[0001] devmapper: loadDeviceFilesOnStart() END
DEBU[0001] devmapper: constructDeviceIDMap()
DEBU[0001] devmapper: Added deviceId=10 to DeviceIdMap
DEBU[0001] devmapper: Added deviceId=36 to DeviceIdMap
DEBU[0001] devmapper: Added deviceId=16 to DeviceIdMap
DEBU[0001] devmapper: Added deviceId=37 to DeviceIdMap
DEBU[0001] devmapper: Added deviceId=86 to DeviceIdMap
DEBU[0001] devmapper: Added deviceId=35 to DeviceIdMap
DEBU[0001] devmapper: Added deviceId=341 to DeviceIdMap
DEBU[0001] devmapper: Added deviceId=9 to DeviceIdMap
DEBU[0001] devmapper: Added deviceId=54 to DeviceIdMap
DEBU[0001] devmapper: Added deviceId=6 to DeviceIdMap
DEBU[0001] devmapper: Added deviceId=41 to DeviceIdMap
DEBU[0001] devmapper: Added deviceId=13 to DeviceIdMap
DEBU[0001] devmapper: Added deviceId=14 to DeviceIdMap
DEBU[0001] devmapper: Added deviceId=340 to DeviceIdMap
DEBU[0001] devmapper: Added deviceId=19 to DeviceIdMap
DEBU[0001] devmapper: Added deviceId=43 to DeviceIdMap
DEBU[0001] devmapper: Added deviceId=57 to DeviceIdMap
DEBU[0001] devmapper: Added deviceId=3 to DeviceIdMap
DEBU[0001] devmapper: Added deviceId=17 to DeviceIdMap
DEBU[0001] devmapper: Added deviceId=40 to DeviceIdMap
DEBU[0001] devmapper: Added deviceId=34 to DeviceIdMap
DEBU[0001] devmapper: Added deviceId=1 to DeviceIdMap
DEBU[0001] devmapper: Added deviceId=12 to DeviceIdMap
DEBU[0001] devmapper: Added deviceId=7 to DeviceIdMap
DEBU[0001] devmapper: Added deviceId=56 to DeviceIdMap
DEBU[0001] devmapper: Added deviceId=42 to DeviceIdMap
DEBU[0001] devmapper: Added deviceId=55 to DeviceIdMap
DEBU[0001] devmapper: Added deviceId=4 to DeviceIdMap
DEBU[0001] devmapper: Added deviceId=8 to DeviceIdMap
DEBU[0001] devmapper: Added deviceId=2 to DeviceIdMap
DEBU[0001] devmapper: Added deviceId=87 to DeviceIdMap
DEBU[0001] devmapper: Added deviceId=11 to DeviceIdMap
DEBU[0001] devmapper: Added deviceId=5 to DeviceIdMap
DEBU[0001] devmapper: Added deviceId=15 to DeviceIdMap
DEBU[0001] devmapper: Added deviceId=18 to DeviceIdMap
DEBU[0001] devmapper: constructDeviceIDMap() END
WARN[0001] devmapper: Usage of loopback devices is strongly discouraged for production use. Please use `--storage-opt dm.thinpooldev` or use `man docker` to refer to dm.thinpooldev section.
DEBU[0001] devmapper: activateDeviceIfNeeded()

问题排查思路

docker服务在低版本内核中默认使用devicemapper存储驱动程序，因能力有限，无法排查到具体因为什么原因引起的。故需要更改storage-driver为overlay。

升级内核

内核升级成功后需要重启系统。

[root@localhost ~]# yum -y update kernel
[root@localhost ~]# reboot

修改daemon

编辑/etc/docker/daemon.json文件并添加或修改storage-driver内容。

{
    "log-driver": "json-file",
    "log-opts":
    {
        "max-size": "100m"
    },
    "storage-driver": "overlay",
    "registry-mirrors":
    [
        "http://hub-mirror.c.163.com"
    ]
}

修改完成后重启docker即可。