CPython Dev w/ Docker

Post by Saul Shanabrook

TLDR

To start developing on CPython (requires working Docker install):

sudo pip install hg-git

echo '[extensions]  
hgext.bookmarks =  
hggit =  
eol =  
' > ~/.hgrc

hg clone -U https://hg.python.org/cpython  
cd cpython


# this is needed on windows hosts so that they use linux
# line endings. however it means some windows programs
# won't be able to read this files correctly
# (Thanks so much to Tal Einat for finding this problem
# and solution!)
echo '  
[eol]
native = LF  
' >> .hg/hgrc  
hg update default

echo 'FROM ubuntu

RUN apt-get update  
RUN apt-get build-dep -y python3.4

ADD . /code/  
WORKDIR /code/  
RUN ./configure --with-pydebug  
RUN make -j2

ENTRYPOINT ["/code/python"]  
CMD ["-m", "test"]  
' >  Dockerfile

cp .gitignore .dockerignore  
echo '.git' >> .dockerignore

# rerun this whenever you change C code
docker build -t cpython .

# to test
docker run -i --rm cpython

# to run tests with folder shared, need this if you
# change files after building
docker run -i -v $PWD/Lib/test:/code/Lib/test --rm cpython

# to run different tests (add the -v to share folders)
docker run -i --rm cpython -m test -h  
docker run -i --rm cpython -m test -j0  
docker run -i --rm cpython -m test -v test_abc  
docker run -i --rm cpython -m unittest -v test.test_abc.TestABC  
# or any others from https://docs.python.org/devguide/runtests.html

Background

I am currently at trying to work on CPython for the first time, at the PyCon 2015 sprints.

I am developing on a mac, and looked at the Python Developer's Guide Quick Start to setup my workflow.

First I had to enable hg-git.

$ sudo pip install hg-git
$ echo '[extensions]
hgext.bookmarks =  
hggit =' > .hgrc  

But the more complicated problems started once I tried to run the tests with

./python -m test -j3

and got that I couldn't run some, because modules were missing:

Python build finished successfully!  
The necessary bits to build these optional modules were not found:  
_gdbm                 ossaudiodev           spwd  
To find the necessary bits, look in setup.py in detect_modules() for the module's name.


Failed to build these modules:  
_lzma  

If I wasn't used to running everything in Docker, I would try to isntall those modules on my Mac with homebrew, but I have becomes spoiled and am used to not playing that game anymore

Running in Docker

So instead, I figured I could just build a Docker image with Ubuntu and install the dependencies.

Using the build dependencies they reccomend, I created a Dockerfile

FROM ubuntu

RUN apt-get update  
RUN apt-get build-dep -y python3.4

ADD . /code/  
WORKDIR /code/  
RUN ./configure --with-pydebug  
RUN make -j2

ENTRYPOINT ["/code/python"]  
CMD ["-m", "test", "-j3"]  

It worked great! To run the tests all I needed to do was:

docker build -t cpython .  
docker run -i --rm cpython  

You need the -i so that CONTROL-C will stop the process when you want.

You need to rebuild whenever you change C code. If you are just changing Python modules, you can just mount whatever file/directory you changed and re-run. For example, if I just added a new test I can run this to mount my local ./Lib/test folder:

docker run -it -v $PWD/Lib/test:/code/Lib/test --rm cpython  

You can also override the default test command as well, by adding the arguments after the image name of cpython:

docker run -i -v $PWD/Lib/test:/code/Lib/test --rm cpython -m test -v test_abc  

You will notice I didn't include the ./python command first, because that is the entrypoint so any argument appended are appended to that.

.dockerignore

Docker has to copy the whole current directory when building the image, so if you can tell Docker that it can ignore some subdirectories it makes it a lot faster and smaller on disk.

So I just copied the .gitignore file to .dockerignore, plus addded .git. More could probably be added, and it probably doesn't need all that, but it's a start.

For reference my .dockerignore looks like:

.git
*.cover
*.o
*.orig
*.pyc
*.pyd
*.pyo
*.rej
*.swp
*~
.gdb_history
Doc/build/  
Doc/tools/docutils/  
Doc/tools/jinja/  
Doc/tools/jinja2/  
Doc/tools/pygments/  
Doc/tools/sphinx/  
Lib/lib2to3/*.pickle  
Lib/test/data/*  
Lib/_sysconfigdata.py  
Lib/plat-mac/errors.rsrc.df.rsrc  
Makefile  
Makefile.pre  
Misc/python.pc  
Misc/python-config.sh  
Modules/Setup  
Modules/Setup.config  
Modules/Setup.local  
Modules/config.c  
Modules/ld_so_aix  
Modules/_freeze_importlib  
Modules/_testembed  
PCbuild/*.bsc  
PCbuild/*.dll  
PCbuild/*.exe  
PCbuild/*.exp  
PCbuild/*.lib  
PCbuild/*.ncb  
PCbuild/*.o  
PCbuild/*.pdb  
PCbuild/Win32-temp-*  
PCbuild/amd64/  
.purify
Parser/pgen  
__pycache__  
autom4te.cache  
build/  
buildno  
config.cache  
config.log  
config.status  
config.status.lineno  
core  
db_home  
config.log  
config.status  
libpython*.a  
libpython*.so*  
platform  
pybuilddir.txt  
pyconfig.h  
python$  
python-config  
python-config.py  
python.exe  
python-gdb.py  
python.exe-gdb.py  
reflog.txt  
.svn/
tags  
TAGS  
.coverage
coverage/  
externals/  
htmlcov/  

C Code Folders

If the make part only needs some folder, it would make sense to only ADD those before that command in the Dockerfile, so that if other files are changed Docker will know it doesn't need to remake and can keep caching that. However, I am not sure which folders/files those are.

Running on Windows

This will all work fine on Windows, as long as you install Docker properly.

Also, you need to make sure that the files have linux newlines, and not windows ones, or else Docker will fail with an error like this:

Step 5 : RUN ./configure --with-pydebug  
 ---> Running in 88cc35498d41
/bin/sh: 1: ./configure: not found

This is rather cryptic, but most likely comes from the wrong line endings.

To solve, use Mercurial's eol extension, to make all line endings use the linux style. This will make some windows programs read the files wrong, but is the only easy solution.

echo 'eol =' >> ~/.hgrc

cd cpython  
echo '  
[eol]
native = LF  
' >> .hg/hgrc  
hg update default  

(Thank you Tal Einat for finding this solution!)