Watchtower potentially broke installation

[Darkknight33]

Can't be 100% certain of it, but my install, which was working fine and hosting about 65GB of data just 404'ed on me. It was sitting unused for the last few weeks while I was working on another project, and as far as I can tell, the only thing that happened was watchtower upgraded a few of the containers. It otherwise hasn't been used since the beginning of August.

The updates, which I could see from the watchtower log somehow broke postgres throwing endless errors about Fatal: role postgres does not exist, I fixed that by creating the postgres role in the nextcloud-db container, but it did not fix the 404. No other logs indicated any issues that I could find, so I reverted to an earlier backup, but that is now inconsistent with the external S3 storage, and the files are unrecoverable. This is something of a nightmare. I'm going to have to rebuild the whole thing from scratch.

Not sure it's a good idea to have automatic updating of containers even to minor versions. It doesn't seem stable.

In a thread I was looking up about the Fatal: role error, one of the posted solutions was to change POSTGRES_DB: "default" in database.yml to POSTGRES_DB: "{{ nextcloud_db_name }}" then reinstall.

Seems like that may be worth taking a look at if 'default' was not the intended db name.

[Darkknight33]

I wanted to raise the issue that I think it's a bad idea to use the 'latest' tag in the playbook by default. Watchtower will pull down new, potentially incompatible versions of the database or nextcloud with no migration path back to previous working versions. This is what happened in my case. In theory, watchtower is a great idea, but in practice it creates a lot of problems, particularly when paired with the 'latest' tag. I believe the image variables should be marked with known compatible versions.

I recently reran this playbook (successfully) with the following image tags:
docker_postgres_image: 14-bullseye docker_mysql_image: latest docker_nextcloud_image: 22-fpm docker_nginx_image: 1.21 docker_adminer_image: latest docker_traefik_image: 2.5 docker_watchtower_image: amd64-1.3.0 docker_redis_image: 6.2-alpine docker_collabora_image: latest docker_onlyoffice_image: 6.3.1.32 docker_turnserver_image: 4 docker_dockerproxy_image: 0.1 docker_elasticsearch_image: 7.6.2

I chose the debian based database image because when I had a problem with the DB I needed to solve, working inside the container when it was based on alpine was a huge PITA. Same for nextcloud image. The space savings is minor in exchange for access to a better and more familiar toolset. I did not test Collabora or Adminer, so those were left as is. I think the user should be directed to this file in the documentation to choose their own tags to work from.

Further I used the labels to disable watchtower on the more critical components of Nextcloud, Postgres, Redis, and Elasticsearch, as breaking changes to any of those can easily render a system unrecoverable. I would recommend making these opted-out by default and let the user opt-in to them if they want to take the risk. I'm happy to submit a PR with these changes if you agree with the proposal.

[ReinerNippes]

I changed the default values of some images in the group_vars/main.yml. Commit 33095a8
I agree that these images should be pined to major versions.

One may change this before running the playbook. Infos are added to the README.md Commit 32d05ea

I still stick to the alpine images since they should also offer better security.

I also enable watchtower update for each container individually. Commit f237ffb
But I put it to default: true. If you pin your image version to a major/fixed version you will still participate from security fixes in the base images.