Building container image failes due to missing files

[serverhorror]

There are at least 2 files missing.

I just did a clean checkout of the repository and wanted to build the image (yes, I'm on Windows -- not all of us have a choice here).

PS C:\Users\marcherm\src\single-cell-tutorial> git status
On branch master
Your branch is up to date with 'origin/master'.

nothing to commit, working tree clean
PS C:\Users\marcherm\src\single-cell-tutorial> git rev-parse --short HEAD
8168bcb

It fails with the following message:

> docker build -t single-cell-tutorial .
[+] Building 9.8s (40/41)
 => [internal] load .dockerignore                                                                                                                                                                                     0.1s
 => => transferring context: 2B                                                                                                                                                                                       0.0s
 => [internal] load build definition from Dockerfile                                                                                                                                                                  0.3s
 => => transferring dockerfile: 3.23kB                                                                                                                                                                                0.0s
 => [internal] load metadata for docker.io/library/debian:10                                                                                                                                                          9.2s
 => [internal] load build context                                                                                                                                                                                     0.1s
 => => transferring context: 2B                                                                                                                                                                                       0.0s
 => CANCELED [1/37] FROM docker.io/library/debian:10@sha256:e2cc6fb403be437ef8af68bdc3a89fd58e80b4e390c58f14c77c466002391193                                                                                          0.1s
 => => resolve docker.io/library/debian:10@sha256:e2cc6fb403be437ef8af68bdc3a89fd58e80b4e390c58f14c77c466002391193                                                                                                    0.0s
 => CACHED [2/37] RUN apt-get update && apt-get install -y --no-install-recommends build-essential apt-utils ca-certificates zlib1g-dev gfortran locales libxml2-dev libcurl4-openssl-dev libssl-dev libzmq3-dev lib  0.0s
 => CACHED [3/37] RUN apt-get update && apt-get install -y --no-install-recommends wget curl htop less nano vim emacs git                                                                                             0.0s
 => CACHED [4/37] RUN echo "en_US.UTF-8 UTF-8" >> /etc/locale.gen     && locale-gen en_US.utf8     && /usr/sbin/update-locale LANG=en_US.UTF-8                                                                        0.0s
 => CACHED [5/37] WORKDIR /opt/R                                                                                                                                                                                      0.0s
 => CACHED [6/37] RUN wget https://cran.rstudio.com/src/base/R-3/R-3.6.3.tar.gz                                                                                                                                       0.0s
 => CACHED [7/37] RUN tar xvfz R-3.6.3.tar.gz && rm R-3.6.3.tar.gz                                                                                                                                                    0.0s
 => CACHED [8/37] WORKDIR /opt/R/R-3.6.3                                                                                                                                                                              0.0s
 => CACHED [9/37] RUN ./configure --enable-R-shlib --with-cairo --with-libpng --prefix=/opt/R/                                                                                                                        0.0s
 => CACHED [10/37] RUN make && make install                                                                                                                                                                           0.0s
 => CACHED [11/37] WORKDIR /opt/R                                                                                                                                                                                     0.0s
 => CACHED [12/37] RUN rm -rf /opt/R/R-3.6.3                                                                                                                                                                          0.0s
 => CACHED [13/37] RUN Rscript -e "update.packages(ask=FALSE, repos='https://ftp.gwdg.de/pub/misc/cran/')"                                                                                                            0.0s
 => CACHED [14/37] RUN Rscript -e "install.packages(c('devtools', 'gam', 'RColorBrewer', 'BiocManager', 'IRkernel'), repos='https://ftp.gwdg.de/pub/misc/cran/')"                                                     0.0s
 => CACHED [15/37] RUN Rscript -e "devtools::install_github(c('patzaw/neo2R', 'patzaw/BED'))"                                                                                                                         0.0s
 => CACHED [16/37] RUN Rscript -e "BiocManager::install(c('scran','MAST','monocle','ComplexHeatmap','limma','slingshot','clusterExperiment','DropletUtils'), version = '3.10')"                                       0.0s
 => CACHED [17/37] RUN Rscript -e 'writeLines(capture.output(sessionInfo()), "../package_versions_r.txt")' --default-packages=scran,RColorBrewer,slingshot,monocle,gam,clusterExperiment,ggplot2,plyr,MAST,DropletUt  0.0s
 => CACHED [18/37] WORKDIR /opt/python                                                                                                                                                                                0.0s
 => CACHED [19/37] RUN wget https://www.python.org/ftp/python/3.7.7/Python-3.7.7.tgz                                                                                                                                  0.0s
 => CACHED [20/37] RUN tar zxfv Python-3.7.7.tgz && rm Python-3.7.7.tgz                                                                                                                                               0.0s
 => CACHED [21/37] WORKDIR /opt/python/Python-3.7.7                                                                                                                                                                   0.0s
 => CACHED [22/37] RUN ./configure --enable-optimizations --with-lto --prefix=/opt/python/                                                                                                                            0.0s
 => CACHED [23/37] RUN make && make install                                                                                                                                                                           0.0s
 => CACHED [24/37] WORKDIR /opt/python                                                                                                                                                                                0.0s
 => CACHED [25/37] RUN rm -rf /opt/python/Python-3.7.7                                                                                                                                                                0.0s
 => CACHED [26/37] RUN ln -s /opt/python/bin/python3 /opt/python/bin/python                                                                                                                                           0.0s
 => CACHED [27/37] RUN ln -s /opt/python/bin/pip3 /opt/python/bin/pip                                                                                                                                                 0.0s
 => CACHED [28/37] RUN pip install --no-cache-dir -U pip wheel setuptools cmake                                                                                                                                       0.0s
 => CACHED [29/37] RUN pip install --no-cache-dir -U scanpy==1.4.6 python-igraph==0.8.0 louvain==0.6.1 jupyterlab=2.1.0 cellxgene==0.15.0 rpy2==3.2.7 anndata2ri==1.0.2 leidenalg==0.7.0 fa2==0.3.5 MulticoreTSNE==0  0.0s
 => CACHED [30/37] RUN pip install --no-cache-dir git+https://github.com/le-ander/epiScanpy.git                                                                                                                       0.0s
 => CACHED [31/37] RUN jupyter contrib nbextension install --system                                                                                                                                                   0.0s
 => CACHED [32/37] RUN jupyter nbextension enable --py widgetsnbextension                                                                                                                                             0.0s
 => CACHED [33/37] RUN pip freeze > ../package_versions_py.txt                                                                                                                                                        0.0s
 => CACHED [34/37] RUN Rscript -e "IRkernel::installspec(user = FALSE)"                                                                                                                                               0.0s
 => ERROR [35/37] COPY .bashrc_docker /root/.bashrc                                                                                                                                                                   0.0s
 => ERROR [36/37] COPY .profile_docker /root/.profile                                                                                                                                                                 0.0s
------
 > [35/37] COPY .bashrc_docker /root/.bashrc:
------
------
 > [36/37] COPY .profile_docker /root/.profile:
------
failed to solve with frontend dockerfile.v0: failed to build LLB: failed to compute cache key: "/.profile_docker" not found: not found

[LuckyMD]

Hi @serverhorror,

I don't think you need these .bashrc and .profile_docker files, so you could take the lines out of the Dockerfile. We put the Dockerfile in the repo in the first place so people could build upon this themselves. It might be easiest to just use the image uploaded the docker hub here.

Otherwise we could always make a docker/ folder and add those files as well @le-ander. What do you think?

[le-ander]

Hm, good point. Thanks for raising this.

The content of the two files is actually quite basic, however it might help to just add them to this repo as well for completeness.

For now, you can just create them yourself:
.bashrc_docker

# Manually added .bashrc
# Source global definitions
if [ -f /etc/bashrc ]; then
    . /etc/bashrc
fi

export USER=root
export LOGNAME=root
export HOME=/root
export PATH=/root/.local/bin:/root/bin:/opt/python/bin:/opt/R/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
export LD_LIBRARY_PATH=/opt/R/lib/R/lib
export SHELL=/bin/bash
export TERM=xterm-256color
export LANG=en_US.UTF-8

alias ll='ls -lha --color=auto'
alias jn='jupyter-notebook --no-browser --ip=0.0.0.0 --allow-root /root/host_home'
alias jl='jupyter-lab --no-browser --ip=0.0.0.0 --allow-root /root/host_home'
alias cx='cellxgene launch --host 0.0.0.0 --port 8888'

.profile_docker

# ~/.profile: executed by the command interpreter for login shells.
# This file is not read by bash(1), if ~/.bash_profile or ~/.bash_login exists.

# if running bash, source ~/.bashrc
if [ -n '$BASH_VERSION' ]; then
    # include .bashrc if it exists
    if [ -f '/root/.bashrc' ]; then
        . '/root/.bashrc'
    fi
fi

[LuckyMD]

@le-ander do you want to make a new docker folder in the repo in a PR so you're credited for code contribution, or shall I just create these files and do that quickly?

[serverhorror]

Hi,

I'm just reporting this because it seems pretty unreliable to "pull code from a repo that is broken in the first place". I'm not in the bio* domain at all. Just needed to use it a while ago as a component that would "just work". So I can't comment about the quality of the results.

I'm on the software development end and it seems strange to me having to pull in "untrusted" code that is in a random docker repository on the internet or even something where I (the client) doesn't control availability of image.

[LuckyMD]

I'm not quite sure I understand what you mean. I don't know if you're aware, but the code you are pulling is a workflow in a Jupyter notebook that goes with a tutorial for scRNA-seq analysis. The workflow is extensively documented so that people trying to update their pipelines or learn how to perform the analysis can adapt this to their own needs. None of this is an out-of-the-box pipeline that "just works" on any data and is immediately ready to use.

The provision of the docker image is a work-around for people who don't want to install the environment manually (instructions in the README). And the code for this image is primarily there for adaptation purposes. There are plenty of ways in which you can get around the issue you are facing here (including just quickly creating the files that were missing yourself, or removing the relevant lines in the Dockerfile).

We are working on updating the workflow now and are adding the missing files. However, none of these changes will make the notebooks here any more of a production level tool that "just works".

[serverhorror]

@LuckyMD I am aware what I'm working with. Let me try and rephrase: Usually when dealing with software projects one of the expectations is that the code is able to build succsessfully from source. There are quite a lot of environments where container images are not a workaround but the norm. Even more so, a requirement. So when a Dockerfile is present then a lot of people -- including me -- expect the source to build.

As I said. I come from the engineering side. All I'm trying to do is report the findings that I see that make it irreproducible. At least they make it hard to reproduce.

[LuckyMD]

Yes, thanks for reporting this. We are working to not only make this docker image build from source in #54 but also to update the jupyter notebook so it runs with the newest versions of several dependencies.

[serverhorror]

Cleaning up old issues on my profile