Support for Singularity

[YSLIU627]

Hi,
Thanks for your work! Is it possible for you to support singularity as docker requires sudo access. Such an access is not allowed for computing cluster. Or is there any alternative option? Thanks quite it a lot.

[YSLIU627]

Hi SumanthRH,
Thanks for your heads up! It could be really helpful if Rootless Podman can be supported to replace docker.

[SumanthRH]

Hi @YSLIU627 , thanks for your interest.

As mentioned in NovaSky-AI/SkyRL#4, our current remote server implemention doesn't support backends like E2B or Singularity.

However, if root access is the issue, then Podman should do the trick. Support for this is easier because you can start the podman service (without root access) and still be able to use aiodocker client to spawn and manage containers.

Installation of Podman can still require root access though. However, for running podman, you just need to start the Podman service:

 systemctl --user start podman.socket

After this, you can just configure DOCKER_HOST environment variable to point to podman (following official docs):

export DOCKER_HOST=unix://$XDG_RUNTIME_DIR/podman/podman.sock

and you should be good! We first read this env var during initialization of our docker client

Let me know if this is possible in your environment!

[YSLIU627]

Hi @SumanthRH,
Thanks for your information! Unfortunately, the AWS HPC cluster does not grant any sudo access even for the installation, so it is impossible to modify /etc to setup a rootless podman or docker (I can only install podman-remote but it seems that it is not enough to run systemctl --user start podman.socket). My administrator tells me that the only solution is to use singularity and I am wondering if it takes a lot of efforts to transfer to singularity? It will be helpful if you provide some high level guidance or insights! Thanks.

[YSLIU627]

Alternatively, could I use Kubernetes as the sandbox environment (as shown in your guidance) and, in doing so, eliminate the need for Docker for my local machine (used for training and inference)?

[SumanthRH]

@YSLIU627 I don't have much experience with Singularity, but AFAIK it doesn't implement a docker interface and thus we would need to change the remote server implementation (currently it's based on aiodocker)

Kubernetes is always an option - this will be a standalone remote server running on AWS EKS and would not require docker on your local machine. Let me know how it goes if you end up trying this!