Python: Docker image build – install required packages via requirements.txt vs editable install. | behai-nguyen software development learnings and documentation

Install via requirements.txt means using this image build step command “RUN pip3 install -r requirements.txt”. Editable install means using the “RUN pip3 install -e .” command. I’ve experienced that install via requirements.txt resulted in images that do not run, whereas using editable install resulted in images that do work as expected. I’m presenting my findings in this post.


Python: Docker image build – install required packages via requirements.txt vs editable install.

The first Docker tutorial I took was Learn to build and deploy your distributed applications easily to the cloud with Docker, it's an excellent tutorial and I did successfully complete all of it. Then I did this Build your Python image -- only this first part.

What's common between them are -- in my observations:

The Python projects have only a single Python file. I did scan through some other tutorials, the projects also have a single Python file.
In the Dockerfile file, they both use RUN pip3 install -r requirements.txt to install required packages.

I would like to try building an image for a project which has more than a single module: most real projects would have more than one module. The code for Synology DS218: preparing Python 3.9 Beta compelete devepment environment. is fairly simple, and would be a good first try.

This is the repository https://github.com/behai-nguyen/app-demo for the code. For the above post, the tag is v1.0.0. It can be cloned with:

git clone -b v1.0.0 https://github.com/behai-nguyen/app-demo.git

Please note, all Docker builds discussed in this post've been out carried on Windows 10 Pro, using docker CLI version 20.10.12, build e91ed57.

To recap, the project layout for app-demo at tag v1.0.0 is:

D:\app_demo\
|
|-- .env
|-- app.py
|-- setup.py
|
|-- src\
|   |
|   |-- app_demo\
|       |   
|       |-- __init__.py
|       |-- config.py
|
|-- venv\

I did create virtualenv venv for this project in Windows 10.

Building using “RUN pip3 install -r requirements.txt” command

Please note, this image build step command renders setup.py not in use.

Generate the requirements.txt file with:

D:\app_demo>venv\Scripts\pip.exe freeze > requirements.txt

Then manually removed everything except for those packages specified in the section install_requires in the setup.py file.

File D:\app_demo\requirements.txt

Flask==2.1.2
python-dotenv==0.20.0

File D:\app_demo\Dockerfile

# syntax=docker/dockerfile:1

FROM python:3.10.5-slim-buster

WORKDIR /app_demo

COPY requirements.txt requirements.txt
RUN pip3 install -r requirements.txt

COPY . .

CMD [ "python3", "-m" , "flask", "run", "--host=0.0.0.0" ]

File D:\app_demo\.dockerignore

__pycache__
*.pyc
*.pyo
*.pyd
.Python
venv
pip-log.txt
pip-delete-this-directory.txt
.tox
.coverage
.coverage.*
.cache
nosetests.xml
coverage.xml
*.cover
*.log
.git
.mypy_cache
.pytest_cache
.hypothesis
FlaskApp.wsgi
Working

The root level of the project layout is now:

D:\app_demo\
|
|-- .env
|-- app.py
|-- setup.py
|-- requirements.txt 
|-- Dockerfile
|-- .dockerignore
|
...

The command to build:

D:\app_demo>docker build --tag app-demo .

The build runs successfully. To run the newly built image:

D:\app_demo>docker run --publish 8000:5000 --rm app-demo

It does not work, as can be seen in the screen capture below:

The error is ModuleNotFoundError: No module named 'app_demo'.

These changes for this not-working-built can be cloned using:

git clone -b v1.0.1 https://github.com/behai-nguyen/app-demo.git

✿✿✿

Google search suggests that others also have experienced similar error. The suggestion is to use absolute import -- for example, please see Module not found error with Python in Docker.

I did try out absolute import, since the project is small, the changes are minuscule, and the resultant image does run. However, that does not seem right... I should not have to do that...

The changes to use absolute import can be cloned using:

git clone -b v1.0.2 https://github.com/behai-nguyen/app-demo.git

Basically, relevant import statements in:

File D:\app_demo\app.py
File D:\app_demo\src\app_demo\__init__.py

were prefixed with src. to become respectively:

from src.app_demo import create_app
from src.app_demo.config import get_config

Build and run, respectively, with:

D:\app_demo>docker build --tag app-demo .
D:\app_demo>docker run --publish 8000:5000 --rm app-demo

It runs successfully. http://localhost:8000 displays the expect output of Hello, World!

Building using “RUN pip3 install -e .” command

Please note for this image build step:

❶ Both:

File D:\app_demo\app.py
File D:\app_demo\src\app_demo\__init__.py

have their import statements reversed back to relative import:

from app_demo import create_app
from app_demo.config import get_config

That is, prefix src. added in the last build was removed.

❷ This image build step command renders requirements.txt obsolete.

✿✿✿

I was doing another Docker image for another Python project, I thought I would just try editable install RUN pip3 install -e . instead. I did think that I would have to change the source codes to absolute import anyway, so why not just use setup.py that's already in place. The build just went through with no problem. I ran it to get the first import failure... It does not fail!

It works! I still don't know why it works!

So I go back to this project, and hence this post. Changes are:

File D:\app_demo\Dockerfile

# syntax=docker/dockerfile:1

FROM python:3.10.5-slim-buster

WORKDIR /app_demo

COPY . .

RUN /usr/local/bin/python -m pip install --upgrade pip && \
    pip3 install -e .

CMD [ "python3", "-m" , "flask", "run", "--host=0.0.0.0" ]

requirements.txt is obsolete; it's added to D:\app_demo\.dockerignore.

File D:\app_demo\.dockerignore

... 
requirements.txt

Clone the new changes with the below command, please discard requirements.txt:

git clone -b v1.0.3 https://github.com/behai-nguyen/app-demo.git

Build and run, respectively, with:

D:\app_demo>docker build --tag app-demo .
D:\app_demo>docker run --publish 8000:5000 --rm app-demo

It runs successfully:

http://localhost:8000 displays the expect output of Hello, World!

✿✿✿

I don't know why RUN pip3 install -e . works... Please tell me if you know, I would appreciate that very much.

Since I've made my decision to use setup.py, this Docker image build step just works out great. I'm happy with it. Thank you for reading and I hope you find this post useful somehow.