Docker storage issues on FC24 nodes
Description
Activity
Former user October 25, 2017 at 9:21 AM
Fedora 24 is no longer used and I haven't seen the issue in a while since we improved slave cleanups. Closing for now.
Former user August 10, 2017 at 1:01 PM
One thing I've noticed today while investigating devicemapper a similar issue in our staging slave, is that when running "docker info" the pool blocksize was 0b. After 's advice to remove /var/lib/docker, it seems that the daemon re-configured it's loop device and now the pool blocksize is 65.54kb.
I wonder, what could set the pool blocksize to 0?
Barak Korren June 11, 2017 at 7:17 AM
Barak Korren why do we even run docker-storage-setup? This is needed only when storage needs to be reconfigured, otherwise docker will use the default loopback option.
We're not explicitly running in, it gets run automatically by systemd.
Former user June 6, 2017 at 2:23 PM
why do we even run docker-storage-setup? This is needed only when storage needs to be reconfigured, otherwise docker will use the default loopback option.
Barak Korren May 31, 2017 at 7:43 AMEdited
It looks like perhaps this issue is not the cause for the startup failure of the docker deamon, because the same situation exists on other FC24 slaves, and the deamon starts up well there.
Opening individual separate ticket about that:
Barak Korren May 31, 2017 at 7:29 AM
Barak Korren May 31, 2017 at 7:18 AM
Looks like the DM loopback devices exist on both EL7 and FC24:
This looks more and more like a recent regression with "docker-storage-setup" that has more sever repercussions on FC24 because of a more strict systemd setup.
Barak Korren May 31, 2017 at 7:08 AM
More information:
The original issue was found on "vm0064.workers-phx.ovirt.org" which is an FC24 VM.
This is what you get when you try to run "docker-storage-setup" manually on it:
We seem to be seeing a very similar issue on EL7. the following is from "vm0024.workers-phx.ovirt.org".
This is the output when running the command manually:
This is, strangely enough, not causing processes to get stuck on EL7 though:
In fact, things seem to be working fine despite the failure:
I suspect we're seeing a recent regression in docker, and previously the setup script knew how to handle nodes without LVM. I'm not finding evidence of an upgrade in yum/dnf logs though.
Barak Korren May 31, 2017 at 6:33 AM
Need to also look into why we didn't see this so far. Did no jobs run on FC24 slaves so far? Are slaves without LVM very rare?
can you provide some insights here?
Details
Assignee
Former userFormer user(Deactivated)Reporter
Barak KorrenBarak Korren(Deactivated)Components
Priority
High
Details
Details
Assignee
Former userReporter
Barak KorrenComponents
Priority
High
docekr startup seems to be failing on FC24 nodes and making other dokcer commands get stuck forever.
The core reason behind the failure seems to be that the Docker storage setup failed:
This seem to be does to the fact that the system's root filesystem is on a plain disk rather then LVM. This is although the docker default DM storage driver is supposed to be able to use a plaing file on the filesystem. Need to check if this is an issue specific to this slave of something that has to do with all FC24 slaves.