Updated binaries are incompatible with btrfs

[worldofgeese]

Thanks for an excellent script!

Unfortunately, the updated binaries appear to be incompatible with BTRFS; given docker stack deploy stack -c stack.yaml all containers fail to launch with error: Failed to create btrfs snapshot: inappropriate ioctl for device.

[markdumay]

That's unfortunate. My NAS still uses EXT4, so I'm not able to reproduce your error. However, after some research, possibly changing the storage driver might help? I used Docker's guide as a reference. Following step 6 of the guide, add "storage-driver": "btrfs" to /var/packages/Docker/etc/dockerd.json. Make a backup of the file first to revert back to the previous version in case of any errors. The full configuration could look something like this:

{
    "storage-driver": "btrfs",
    "data-root" : "/var/packages/Docker/target/docker",
    "log-driver" : "json-file",
    "registry-mirrors" : [],
    "group": "administrators"
}

You will have to restart the Docker service to pick up the changes. Revert back to the old version of dockerd.json if starting of the service fails.

synoservicectl --stop pkgctl-Docker
synoservicectl --start pkgctl-Docker

[worldofgeese]

Apologies it's been busy with work. Hope to get to this this weekend if not earlier.

…

On October 28, 2020 6:38:28 AM GMT+01:00, Mark Dumay ***@***.***> wrote: That's unfortunate. My NAS still uses EXT4, so I'm not able to reproduce your error. However, after some research, possibly changing the storage driver might help? I used Docker's [guide](https://docs.docker.com/storage/storagedriver/btrfs-driver/) as reference. Following step 6 of the guide, add `"storage-driver": "btrfs"` to `/var/packages/Docker/etc/dockerd.json`. Make a backup of the file first to revert back to the previous version in case of any errors. The full configuration could look something like this: ``` { "storage-driver": "btrfs", "data-root" : "/var/packages/Docker/target/docker", "log-driver" : "json-file", "registry-mirrors" : [], "group": "administrators" } ``` You will have to restart the Docker service to pick up the changes. Revert back to the old version of `dockerd.json` if starting of the service fails. ``` synoservicectl --stop pkgctl-Docker synoservicectl --start pkgctl-Docker ```

[Zenedith]

Hi,
Not working for me :(

[markdumay]

Sorry to hear that @Zenedith. Would you happen to have a backup of /var/packages/Docker/etc/dockerd.json that you could share? Perhaps Synology's original configuration (prior to running the update with synology-docker) gives some additional clues to what's happening.

[mietzen]

Hi, I haven't tried this script yet, but wanted to.
This is the content of cat /var/packages/Docker/etc/dockerd.json

{
   "data-root" : "/var/packages/Docker/target/docker",
   "log-driver" : "db",
   "registry-mirrors" : []
}

Info:

DS418play
DSM 6.2.3-25426 Update 2
Docker version 18.09.8, build 2c0a67b
SHR, btrfs

Edit:
That might be a hint:
https://together.jolla.com/question/92841/bug-btrfs-error-unable-get-label-inappropriate-ioctl-for-device/

The error is normal and happens because the version of btrfs-progs (that provides the "btrfs" command line tool) expects a more updated version of the btrfs kernel module (that in the Jolla kernel is pretty old).

Seems reasonable that a newer docker version is compiled against a newer version of the btrfs kernel module which is not present on the NAS.

[markdumay]

Hi @ngerke, the dockerd.json looks like a standard Synology configuration to me. You might be right about the requirement to have a newer btrfs kernel module present on the NAS. If that's the case, the only thing that comes to mind is to test the script with an older target version for Docker.

[Zenedith]

Hi, mine looks like this:

cat /var/packages/Docker/etc/dockerd.json
{
   "data-root" : "/var/packages/Docker/target/docker",
   "log-driver" : "db",
   "registry-mirrors" : [],
   "storage-driver" : "btrfs"
}

[markdumay]

hi @Zenedith, the syno_docker_update.sh script uses below configuration by default. Synology's db log driver is not supported and is replaced by json-file instead.

{
    "data-root" : "/var/packages/Docker/target/docker",
    "log-driver" : "json-file",
    "registry-mirrors" : [],
    "group": "administrators"
}

What happens if you manually update the dockerd.json file yourself? E.g.

{
    "data-root" : "/var/packages/Docker/target/docker",
    "log-driver" : "json-file",
    "registry-mirrors" : [],
    "group": "administrators",
    "storage-driver" : "btrfs"
}

What comes to mind is that syno_docker_update.sh is a bit too aggressive in replacing the entire dockerd.json file. Instead, it would be better if it modifies the log-driver configuration only. I'll create a new issue to address this. However, I'm still not sure if this will fix the Btrfs issue.

[photocyte]

Hi there,
Thanks for the great script! I ran into this same issue on a DS1819+ with a brtfs filesystem, where after updating Docker through the script, running docker pull would produce the Failed to create btrfs snapshot: inappropriate ioctl for device. error.

I found that editing /var/packages/Docker/etc/dockerd.json to use the vfs storage device instead of brtfs worked. The Docker documentation suggests that the vfs storage driver, while not performant nor recommended for production use, is robust/stable, so I feel OK using it. In a brief test the overlay2 storage driver didn't work for me, hence why I used vfs

In my case I also deleted all my existing images before re-pulling everything, since the storage driver has something to do with how the images are stored:

docker system prune
docker system prune -a

Final dockerd.json was as follows:

{
    "data-root" : "/var/packages/Docker/target/docker",
    "log-driver" : "json-file",
    "registry-mirrors" : [],
    "group": "administrators",
    "storage-driver" : "vfs"
}

After all that, the updated non-DSM Docker seems to be working well for me after re-pulling and re-configuring my images. That being said, the GUI for Docker in DSM has a few bugs: the days since reboot for each container is some very large value, and the logs are not viewable (presumably from json-file vs db).

[markdumay]

Thanks for sharing @photocyte! This might be very helpful for others too. On a side note, the vfs storage driver is also how someone got Docker running on Apple silicon M1 apparently.

I'm considering to add a new flag to the script, supporting an alternative storage driver such as vfs. Some thoughts:

1. Add detection of the filesystem to detect_current_versions(), such as:

root_dir=$(docker info -f '{{ .DockerRootDir }}')
docker_fs=$(df "${root_dir}" --output=fstype | awk 'NR==2')

2. Terminate the script with an informative error in validate_current_version() when docker_fs is not equal to EXT?
3. Add a new flag --storage-driver to support alternative storage drivers such as vfs, also passing the check in step 2

Addressing issue #40 first would be a prerequisite.

That being said, the GUI for Docker in DSM has a few bugs: the days since reboot for each container is some very large value, and the logs are not viewable (presumably from json-file vs db).

Yes, unfortunately these are known issues. I haven't found a way to fix the incorrect time yet. Running docker ps from the command line works as expected though. I suspect you're right regarding the log driver too. Suggestions are welcome! ;-)

[1activegeek]

Thank you guys for this continued discussion and the solution using vfs driver. I was running into this very issue myself recently as I want the benefit of btrfs, but would like to operate on a newer Docker version as well. I wanted to report that I was successful with this method as well. I did have to simply stop and start the docker service through the Syno interface and was off to the races.

One thing I wanted to point out as a potential area to check/test out - custom networks. I ran into an issue and banged my head troubleshooting and hitting walls last night for about 2.5 hrs because my container would not get any internet connectivity. I finally decided after some sleep, to try reverting to the previous docker version - and voila the container could then get internet connectivity. Subsequently updated to the newer docker version, and the container still worked with the network that was created on the original docker version.

Bottom line - it seems there is an issue in Synology setup somewhere if you try to use the newer docker network commands to create a custom network. Might be something to check out - or maybe someone knows something more about the correlations and configurations of syno with docker.

[markdumay]

Thanks for sharing your findings @1activegeek. Regarding the network issues, are they related to user-defined bridge networks (#35)? I suspect the iptables configuration needs some additional tweaks on Synology for these kind of networks. In another repository I scripted some steps (see the function execute_create_macvlan() specifically), and added a static route among other things to get a macvlan network up and running.

Your approach to create the networks/containers by using the default (older) Synology Docker version might be a helpful workaround for now. I'm curious though, does it survive reboots?

[1activegeek]

Wow, yup you are correct. I must have been dense in my reading of that on the Known Issues. Sorry for that. Yes it would appear that is the issue I'm facing. I've just tested the reboot and it does look as though you are correct, it has lost it's connectivity. Will bring my finding to that issue to discuss.

So net/net - ignore my previous comment - and thanks for the workaround! 😄