WIP: Support to build Hubs CE multi-arch containers (amd64, arm64) to support use with Raspberry Pi and ARM based clones

[ajlennon]

What?

Adds support to build multi-arch kubernetes pod/containers. The existing available Hubs containers are amd64 only and so Hubs CE can't be operated on ARM based single board computers such as the Raspberry Pi.

Why?

It feels like it would be really useful to the community for RPi enthusiasts to have an easy route to running up their own local Hubs CE kubernetes cluster on cheap hardware. Currently this is challenging as it requires docker container support for arm64 platforms.

Examples

I've made changes to the hubs-cloud/community-edition/services/dockerbuildall.sh to support building for amd64 (as currently), to add arm64 and also multi-arch which is where there are containers on Docker Hub with a manifest for platforms and are automatically currently pulled for the local system architecture by Docker/Kubernetes. For me a maintained mult-arch build system via CI would be the easiest way to engage with the Hubs community.

How to test

1. Go to hubs-cloud/community-edition/services
2. Read the modified readme.md and check it covers things correctly
3. Build up with default settings for amd64 and install Hubs CE on an Intel box to test
4. Build up for arm64 and install on a Pi to test
5. Build up for multiarch and install on a Pi to test

*also for each of 3, 4, 5 we want to check that npm apply gen-ssl automatically generates and installs webserver certificates as I have been having problems with this

NOTE This is up and running on my 8GB Pi5 here although I have only gotten as far as checking the interface runs

Documentation of functionality

Changes are in hubs-cloud/community-edition/services

Limitations

I am not entirely sure I have properly dealt with arm64 building with docker build versus cross-platform and multi-arch build with docker buildx build. There appear to be different ways to install docker to achieve this and it needs testing to make sure things are working correctly. OR perhaps we could leave the existing script as it is and create a new multi-arch script.

Alternatives considered

I tried running qemu support on target boards (e.g. so existing amd64 images could run on arm64) even knowing this would be slower. This was very very flakey and unworkable which lead me into rebuilding the containers for arm64

Open questions

1. Can people respond on the documentation and what needs improving?
2. Can people try some testing to see if everything is working as needed?
3. The SSL cert generation doesn't seem to be working and I had to generate certs manually and use kubectl create tls secret cert-$domain --cert $cert --key $key -n hcce for the domain and 3 sub-domains.

Additional details or related context

1. Next steps are to look at a script to build the latest hubs/reticulum/spoke/dialog containers as the existing copies in services are very out of date I am told

[DougReeder]

This is exciting!

However, due to this changing a number of configurations we haven't really touched, and our lack of automated tests, we'll need a solid test plan.

[ajlennon]

Glad you like it :) Yes I totally understand - and changes to the various containers to support arm are fairly invasive so it has the potential to cause all sorts of trouble if not tested properly!

[Edit: I am a newbie to Hubs so don't really know enough about what needs to be tested but I've been reaching out to people in the hope that interested parties would like to help out doing testing on Pi's and on existing platforms to ensure I've not broken things!]

[ajlennon]

So I am thinking.... Obviously I'm keen to get this accepted and also not to cause mayhem and confusion with all sorts of changes... What I think I will do is what Imaginer suggested and add a little test script to replace all the hubs-cloud/community-edition/services/* repos locally where there are newer ones available in the hubs-foundation github organisation. And get those building as multi-arch images. And raise PR's in each repo for relevant changes there referencing them as support for this PR. Then if we could work through those individually to get each new repo working as wanted we could come back to this PR to finish it up?

Does that sound like a sensible plan of action?

[ajlennon]

So I have been beavering away to try to get self-hosted Raspberry Pi 5 supported and I now have something that I think is working... I have a Pi5/8GB RAM/32GB SD box here which is under test https://pi.xrhub.org

I've moved the services in hubs-cloud over to submodules and to temporarily reference some forks in my dynamicdevices org. where I have had to make changes to the individual repositories

https://github.com/DynamicDevices/hubs-cloud/tree/master/community-edition/services

Unmodified hubs-foundation services but using the current source-of-truth repo

• nearspark
• speelycaptor

Modified forks I am hoping to upstream

• dialog [NEEDS MERGE]

Pin the worker image to the same explicit node version as the build image as the LTS seems to have changed and this caused build errors

Use netcat-openbsd (as elsewhere) as netcat is no longer available from the apt repo

• hubs [MERGED]

Needed to install phantomjs support as this was breaking the arm64 build

• photomnemonic [NEEDS MERGE]

Needed to install an architecture specific version of chromium to build for arm64

• spoke [NEEDS MERGE]

Needed to skip puppeteer download for arm64 build

Modified service within hubs-cloud without submodule

• coturn [NEEDS MERGE]

The coturn repo is quite different from the simple Dockerfile / entry point in the hubs-cloud service so I've made changes here instead of using a submodule

I no longer generate a self-cert in the Dockerfile as this was causing the TLS library to fail
Instead we use the stream.$domain cert within the kubernetes configuration
I set the coturn server name correct in the turnserver configuration to use this certificate
I have disabled authentication as I can't seem to get auth against the pgsql pods working [NEEDS UNDERSTANDING]

...

Could we have a look at raising and testing individual small PRs for each of the forks I've made to get those in place in order to be able to then be in a position to update hubs-cloud to support deployment to arm64? I think dialog, hubs, photomnemonic, and spoke should be easy to upstream. coturn needs a little more thought.