Singularity image fail to start

[mfasco]

1. Bug description

Our cluster does not support Docker, so we would like to use the Singularity image.
Unfortunately, it fail to start with the following error message:
s6-overlay-preinit: fatal: unable to chown /var/run/s6: Read-only file system

Expected behaviour

The image should run.

2. Reproducible example

Code

$ singularity pull docker://bschilder/echolocator

INFO:    Converting OCI blobs to SIF format
INFO:    Starting build...
Getting image source signatures
Copying blob b50f1017556e done  
Copying blob ac2e64a9e36a done  
Copying blob e1b2b3a6a723 done  
Copying blob 3e1633226ea3 done  
Copying blob b3cacfb902ee done  
Copying blob e96e057aae67 done  
Copying blob ee6ca148ef07 done  
Copying blob 1b92e41706af done  
Copying blob 4c84da83eb41 done  
Copying blob ad8ee9a4fa31 done  
Copying blob 64c8c817d8b1 done  
Copying blob ba3f6b7e0f33 done  
Copying blob ab4919bad7d1 done  
Copying blob 1bd3d7f33b43 done  
Copying blob 8aa30d70216c done  
Copying config bb2d532c8c done  
Writing manifest to image destination
Storing signatures
2023/01/09 12:33:47  info unpack layer: sha256:e96e057aae67380a4ddb16c337c5c3669d97fdff69ec537f02aa2cc30d814281
2023/01/09 12:33:48  info unpack layer: sha256:3e1633226ea37a1c4c5f94733ed7d77c26f5d5714300ea8363dd096916184c75
2023/01/09 12:33:48  info unpack layer: sha256:e1b2b3a6a7232a3aae8f678c48c3188e7fb87a8ba5942ee82cb3f233659d6d86
2023/01/09 12:33:58  info unpack layer: sha256:ac2e64a9e36af4cdbc00c3639016e2382063af7738af64333b3a85cf77d02048
2023/01/09 12:33:59  info unpack layer: sha256:b3cacfb902eef0b6c03796b9e6e68711244dadcb14191371e4c2debc83d22240
2023/01/09 12:34:00  info unpack layer: sha256:b50f1017556e64d2d42f9c965fbce3e4067463e14029034358041449ed2f9899
2023/01/09 12:34:13  info unpack layer: sha256:ee6ca148ef071114d6c4999a2b7021b6a7e2a25b177192fe2dc05f18ed6fccc7
2023/01/09 12:34:13  info unpack layer: sha256:1b92e41706af18902da960e0eb198337c600a94b87d0660786588db3d9009b5b
2023/01/09 12:34:13  info unpack layer: sha256:4c84da83eb41120cd68c4fb81638fafe032723f8798d1e12d3d522ca9a48fb57
2023/01/09 12:34:13  info unpack layer: sha256:ad8ee9a4fa311d989b0701ef5553d9b0a63dd789a2100045cead08a45527549e
2023/01/09 12:34:56  info unpack layer: sha256:64c8c817d8b13243e15fa9b4af37f481f81c3d0f55679ccb493a18b96b67ddeb
2023/01/09 12:34:56  info unpack layer: sha256:ba3f6b7e0f33bbd418530ec08c450e84a8fc71b155110951bd747a1c9c50429c
2023/01/09 12:34:56  info unpack layer: sha256:ab4919bad7d196e713faf86f24f78f9ebf7170633ba8c19f367297a81f01cada
2023/01/09 12:34:59  info unpack layer: sha256:1bd3d7f33b437c11b03aeaa7d7edbdc0cc28b90f630c2e69d1598decc2e73a60
2023/01/09 12:34:59  info unpack layer: sha256:8aa30d70216cc13500ca458121b3c59aa08a2315629a78ce20c088869652823d
2023/01/09 12:35:44  warn rootless{usr/local/lib/R/site-library/Rhtslib/usrlib/libhts.so.2} ignoring (usually) harmless EPERM on setxattr "user.rootlesscontainers"
INFO:    Creating SIF file...

$ ./echolocator_latest.sif
s6-overlay-preinit: fatal: unable to chown /var/run/s6: Read-only file system

Environment

• OS: Debian GNU/Linux 10 (buster)
• Singularity version: 3.10.4

Console output

see above

Data

not applicable

3. Session info

not applicable

[bschilder]

hi @mfasco , this sounds like a permissions issue with how your HPC has set up singularity. i think the best course would be to reach out to your HPC support team directly.

[mfasco]

hi @bschilder, thanks for the quick reply. I actually replicated the same issue on my desktop, which has Ubuntu 18.04 and the latest version of Singularity, 3.10.4.

Thanks.

[mfasco]

sorry to bother you again @bschilder - I would like to help debug the problem - is there a way to access the Dockerfile and all it's needed to build the image?

[bschilder]

hi @bschilder, thanks for the quick reply. I actually replicated the same issue on my desktop, which has Ubuntu 18.04 and the latest version of Singularity, 3.10.4.

Thanks for this info @mfasco, this helps narrow down the possibilities.

Sorry haven't had a chance to look into this further just yet. WIll try to get to this soon.

The Docker image that is currently on Dockerhub is one that I created manually (without a Dockerfile) by pulling the Bioc docker image:
https://www.bioconductor.org/help/docker/

I'm working on automating this with my rworkflows GitHub Action, which uses a generic Dockerfile. You can access a copy of this Dockerfile with rworkflows::use_dockerfile() from the rworkflows R package, which can also be installed from here.

I've just launched a new GitHub Actions run for echolocatoR, so if all goes well, hopefully the new image will be be up soon on Dockerhub. I'll keep you posted!

[bschilder]

Mac

I've just tried regenerating the singularity container on my local Mac:

> singularity pull docker://bschilder/echolocator 

INFO:    Converting OCI blobs to SIF format
INFO:    Starting build...
Getting image source signatures
Copying blob sha256:e96e057aae67380a4ddb16c337c5c3669d97fdff69ec537f02aa2cc30d814281
 29.02 MiB / 29.02 MiB [====================================================] 1s
Copying blob sha256:3e1633226ea37a1c4c5f94733ed7d77c26f5d5714300ea8363dd096916184c75
 1.68 KiB / 1.68 KiB [======================================================] 0s
Copying blob sha256:e1b2b3a6a7232a3aae8f678c48c3188e7fb87a8ba5942ee82cb3f233659d6d86
 259.41 MiB / 259.41 MiB [=================================================] 13s
Copying blob sha256:ac2e64a9e36af4cdbc00c3639016e2382063af7738af64333b3a85cf77d02048
 19.29 KiB / 19.29 KiB [====================================================] 0s
Copying blob sha256:b3cacfb902eef0b6c03796b9e6e68711244dadcb14191371e4c2debc83d22240
 26.29 MiB / 26.29 MiB [====================================================] 0s
Copying blob sha256:b50f1017556e64d2d42f9c965fbce3e4067463e14029034358041449ed2f9899
 235.53 MiB / 235.53 MiB [=================================================] 10s
Copying blob sha256:ee6ca148ef071114d6c4999a2b7021b6a7e2a25b177192fe2dc05f18ed6fccc7
 26.60 KiB / 26.60 KiB [====================================================] 0s
Copying blob sha256:1b92e41706af18902da960e0eb198337c600a94b87d0660786588db3d9009b5b
 52.83 KiB / 52.83 KiB [====================================================] 0s
Copying blob sha256:4c84da83eb41120cd68c4fb81638fafe032723f8798d1e12d3d522ca9a48fb57
 1.18 KiB / 1.18 KiB [======================================================] 0s
Copying blob sha256:ad8ee9a4fa311d989b0701ef5553d9b0a63dd789a2100045cead08a45527549e
 832.87 MiB / 832.87 MiB [=================================================] 45s
Copying blob sha256:64c8c817d8b13243e15fa9b4af37f481f81c3d0f55679ccb493a18b96b67ddeb
 272 B / 272 B [============================================================] 0s
Copying blob sha256:ba3f6b7e0f33bbd418530ec08c450e84a8fc71b155110951bd747a1c9c50429c
 255 B / 255 B [============================================================] 0s
Copying blob sha256:ab4919bad7d196e713faf86f24f78f9ebf7170633ba8c19f367297a81f01cada
 73.90 MiB / 73.90 MiB [====================================================] 3s
Copying blob sha256:1bd3d7f33b437c11b03aeaa7d7edbdc0cc28b90f630c2e69d1598decc2e73a60
 1.44 KiB / 1.44 KiB [======================================================] 0s
Copying blob sha256:8aa30d70216cc13500ca458121b3c59aa08a2315629a78ce20c088869652823d
 950.35 MiB / 950.35 MiB [=================================================] 28s
Copying config sha256:bb2d532c8cbc41b332b9bbe82ac6f9d5accb2bf2e202e7cf4ad5fb8f9a7829b3
 10.24 KiB / 10.24 KiB [====================================================] 0s
Writing manifest to image destination
Storing signatures
INFO:    Creating SIF file...
INFO:    Build complete: echolocator_latest.sif

When i try the command I get a different error, but this has to do with Singularity not being supported on Mac:

> ./echolocator_latest.sif  
vmx_init: processor does not support desired secondary processor-based controls

The issue has something to do with how my VM is setup, which I'll also need to follow up on.

Linux

So I tried generating a new container on my HPC, which is Linux/CentOS-based:

> singularity pull docker://bschilder/echolocator 
INFO:    Detected Singularity user configuration directory
INFO:    Detected public Singularity pgp keyring, migrating...
INFO:    Detected private Singularity pgp keyring, migrating...
INFO:    Converting OCI blobs to SIF format
INFO:    Starting build...
Getting image source signatures
Copying blob b50f1017556e done  
Copying blob e96e057aae67 done  
Copying blob e1b2b3a6a723 done  
Copying blob b3cacfb902ee done  
Copying blob ac2e64a9e36a done  
Copying blob 3e1633226ea3 done  
Copying blob ee6ca148ef07 done  
Copying blob 1b92e41706af done  
Copying blob 4c84da83eb41 done  
Copying blob ad8ee9a4fa31 done  
Copying blob 64c8c817d8b1 done  
Copying blob ba3f6b7e0f33 done  
Copying blob ab4919bad7d1 done  
Copying blob 1bd3d7f33b43 done  
Copying blob 8aa30d70216c done  
Copying config bb2d532c8c done  
Writing manifest to image destination
Storing signatures
2023/01/20 14:45:11  info unpack layer: sha256:e96e057aae67380a4ddb16c337c5c3669d97fdff69ec537f02aa2cc30d814281
2023/01/20 14:45:49  info unpack layer: sha256:3e1633226ea37a1c4c5f94733ed7d77c26f5d5714300ea8363dd096916184c75
2023/01/20 14:45:49  info unpack layer: sha256:e1b2b3a6a7232a3aae8f678c48c3188e7fb87a8ba5942ee82cb3f233659d6d86
2023/01/20 14:47:45  info unpack layer: sha256:ac2e64a9e36af4cdbc00c3639016e2382063af7738af64333b3a85cf77d02048
2023/01/20 14:47:46  info unpack layer: sha256:b3cacfb902eef0b6c03796b9e6e68711244dadcb14191371e4c2debc83d22240
2023/01/20 14:47:49  info unpack layer: sha256:b50f1017556e64d2d42f9c965fbce3e4067463e14029034358041449ed2f9899
2023/01/20 14:49:56  info unpack layer: sha256:ee6ca148ef071114d6c4999a2b7021b6a7e2a25b177192fe2dc05f18ed6fccc7
2023/01/20 14:49:57  info unpack layer: sha256:1b92e41706af18902da960e0eb198337c600a94b87d0660786588db3d9009b5b
2023/01/20 14:49:57  info unpack layer: sha256:4c84da83eb41120cd68c4fb81638fafe032723f8798d1e12d3d522ca9a48fb57
2023/01/20 14:49:57  info unpack layer: sha256:ad8ee9a4fa311d989b0701ef5553d9b0a63dd789a2100045cead08a45527549e
2023/01/20 14:58:19  info unpack layer: sha256:64c8c817d8b13243e15fa9b4af37f481f81c3d0f55679ccb493a18b96b67ddeb
2023/01/20 14:58:19  info unpack layer: sha256:ba3f6b7e0f33bbd418530ec08c450e84a8fc71b155110951bd747a1c9c50429c
2023/01/20 14:58:19  info unpack layer: sha256:ab4919bad7d196e713faf86f24f78f9ebf7170633ba8c19f367297a81f01cada
2023/01/20 14:59:21  info unpack layer: sha256:1bd3d7f33b437c11b03aeaa7d7edbdc0cc28b90f630c2e69d1598decc2e73a60
2023/01/20 14:59:21  info unpack layer: sha256:8aa30d70216cc13500ca458121b3c59aa08a2315629a78ce20c088869652823d
2023/01/20 15:03:30  warn rootless{usr/local/lib/R/site-library/Rhtslib/usrlib/libhts.so.2} ignoring (usually) harmless EPERM on setxattr "user.rootlesscontainers"
INFO:    Creating SIF file...

Then ran this command to access the terminal within the container:

./echolocator_latest.sif exec bash

After an initial warning (seems unrelated to your issue), I'm able to access the terminal within the container and open R:

couldn't read file "/apps/modules/4.7.1/libexec/modulecmd.tcl": no such file or directory
Apptainer> R
WARNING: ignoring environment value of R_HOME

R version 3.6.1 (2019-07-05) -- "Action of the Toes"
Copyright (C) 2019 The R Foundation for Statistical Computing
Platform: x86_64-conda_cos6-linux-gnu (64-bit)

R is free software and comes with ABSOLUTELY NO WARRANTY.
You are welcome to redistribute it under certain conditions.
Type 'license()' or 'licence()' for distribution details.

  Natural language support but running in an English locale

R is a collaborative project with many contributors.
Type 'contributors()' for more information and
'citation()' on how to cite R or R packages in publications.

Type 'demo()' for some demos, 'help()' for on-line help, or
'help.start()' for an HTML browser interface to help.
Type 'q()' to quit R.

That said, it doesn't seem to have echolocatoR within it! Or at least it can't find the R lib to call it:

library(echolocatoR)
# Error in library(echolocatoR) : there is no package called ‘echolocatoR’

This may be resolved by using the Dockerfile-bases approach that rworkflows uses. Last run failed due to the Dockerfile file not getting passed the PAT_GITHUB token (and some rtracklayer dependency issues) but I'll try to fix this and rerun:
https://github.com/RajLabMSSM/echolocatoR/actions/runs/3968409612/jobs/6801540238