I think we've solved this, as I haven't seen any docker related issues for a while now... I'll close this ticket for now, and if something new occurs lets open a new one with fresh details.

This might be a duplicate of something we already fixed. please confirm and close this ticket if it is.

ping?

At this point it seems like docker-storage-setup is still involved in the issue as the system's rootfs was mounted into /var/lib/docker/devicemapper which may be why everything got stuck. Will work with the other ticket and hopefully closing that one will also solve docker issues as well.

, The restart getting stuck is a symptom, not the problem, we need to find and fix the problem. Not doing the restart will just, in the best scenario, sweep the issue under the rug and in the more likely scenario, will make it manifest in far more subtle and harder to find problems.

WRT docker-storage-setup, I was initially thinking that there is a dependency between it and the docker service in systemd, and that this is what causing the docker to fail to restart. But since I've found that it is failing in the same way on other nodes, where restarting the docker deamon does not fail, I've dropped that line of inquiry.

You can see further details of my investigation of docker-storage-setup in the dedicated ticket: