To start developing on CPython (requires working Docker install):
sudo pip install hg-git echo '[extensions] hgext.bookmarks = hggit = eol = ' > ~/.hgrc hg clone -U https://hg.python.org/cpython cd cpython # this is needed on windows hosts so that they use linux # line endings. however it means some windows programs # won't be able to read this files correctly # (Thanks so much to Tal Einat for finding this problem # and solution!) echo ' [eol] native = LF ' >> .hg/hgrc hg update default echo 'FROM ubuntu RUN apt-get update RUN apt-get build-dep -y python3.4 ADD . /code/ WORKDIR /code/ RUN ./configure --with-pydebug RUN make -j2 ENTRYPOINT ["/code/python"] CMD ["-m", "test"] ' > Dockerfile cp .gitignore .dockerignore echo '.git' >> .dockerignore # rerun this whenever you change C code docker build -t cpython . # to test docker run -i --rm cpython # to run tests with folder shared, need this if you # change files after building docker run -i -v $PWD/Lib/test:/code/Lib/test --rm cpython # to run different tests (add the -v to share folders) docker run -i --rm cpython -m test -h docker run -i --rm cpython -m test -j0 docker run -i --rm cpython -m test -v test_abc docker run -i --rm cpython -m unittest -v test.test_abc.TestABC # or any others from https://docs.python.org/devguide/runtests.html
I am currently at trying to work on CPython for the first time, at the PyCon 2015 sprints.
I am developing on a mac, and looked at the Python Developer's Guide Quick Start to setup my workflow.
First I had to enable hg-git.
$ sudo pip install hg-git $ echo '[extensions] hgext.bookmarks = hggit =' > .hgrc
But the more complicated problems started once I tried to run the tests with
./python -m test -j3
and got that I couldn't run some, because modules were missing:
Python build finished successfully! The necessary bits to build these optional modules were not found: _gdbm ossaudiodev spwd To find the necessary bits, look in setup.py in detect_modules() for the module's name. Failed to build these modules: _lzma
If I wasn't used to running everything in Docker, I would try to isntall those modules on my Mac with homebrew, but I have becomes spoiled and am used to not playing that game anymore
Running in Docker
So instead, I figured I could just build a Docker image with Ubuntu and install the dependencies.
Using the build dependencies they reccomend, I created a
FROM ubuntu RUN apt-get update RUN apt-get build-dep -y python3.4 ADD . /code/ WORKDIR /code/ RUN ./configure --with-pydebug RUN make -j2 ENTRYPOINT ["/code/python"] CMD ["-m", "test", "-j3"]
It worked great! To run the tests all I needed to do was:
docker build -t cpython . docker run -i --rm cpython
You need the
-i so that CONTROL-C will stop the process when you want.
You need to rebuild whenever you change C code. If you are just changing Python modules, you can just mount whatever file/directory you changed and re-run. For example, if I just added a new test I can run this to mount my local
docker run -it -v $PWD/Lib/test:/code/Lib/test --rm cpython
You can also override the default test command as well, by adding the arguments after the image name of
docker run -i -v $PWD/Lib/test:/code/Lib/test --rm cpython -m test -v test_abc
You will notice I didn't include the
./python command first, because that is the
entrypoint so any argument appended are appended to that.
Docker has to copy the whole current directory when building the image, so if you can tell Docker that it can ignore some subdirectories it makes it a lot faster and smaller on disk.
So I just copied the
.gitignore file to
.dockerignore, plus addded
.git. More could probably be added, and it probably doesn't need all that, but it's a start.
For reference my
.dockerignore looks like:
.git *.cover *.o *.orig *.pyc *.pyd *.pyo *.rej *.swp *~ .gdb_history Doc/build/ Doc/tools/docutils/ Doc/tools/jinja/ Doc/tools/jinja2/ Doc/tools/pygments/ Doc/tools/sphinx/ Lib/lib2to3/*.pickle Lib/test/data/* Lib/_sysconfigdata.py Lib/plat-mac/errors.rsrc.df.rsrc Makefile Makefile.pre Misc/python.pc Misc/python-config.sh Modules/Setup Modules/Setup.config Modules/Setup.local Modules/config.c Modules/ld_so_aix Modules/_freeze_importlib Modules/_testembed PCbuild/*.bsc PCbuild/*.dll PCbuild/*.exe PCbuild/*.exp PCbuild/*.lib PCbuild/*.ncb PCbuild/*.o PCbuild/*.pdb PCbuild/Win32-temp-* PCbuild/amd64/ .purify Parser/pgen __pycache__ autom4te.cache build/ buildno config.cache config.log config.status config.status.lineno core db_home config.log config.status libpython*.a libpython*.so* platform pybuilddir.txt pyconfig.h python$ python-config python-config.py python.exe python-gdb.py python.exe-gdb.py reflog.txt .svn/ tags TAGS .coverage coverage/ externals/ htmlcov/
C Code Folders
make part only needs some folder, it would make sense to only
ADD those before that command in the
Dockerfile, so that if other files are changed Docker will know it doesn't need to remake and can keep caching that. However, I am not sure which folders/files those are.
Running on Windows
This will all work fine on Windows, as long as you install Docker properly.
Also, you need to make sure that the files have linux newlines, and not windows ones, or else Docker will fail with an error like this:
Step 5 : RUN ./configure --with-pydebug ---> Running in 88cc35498d41 /bin/sh: 1: ./configure: not found
This is rather cryptic, but most likely comes from the wrong line endings.
To solve, use Mercurial's eol extension, to make all line endings use the linux style. This will make some windows programs read the files wrong, but is the only easy solution.
echo 'eol =' >> ~/.hgrc cd cpython echo ' [eol] native = LF ' >> .hg/hgrc hg update default
(Thank you Tal Einat for finding this solution!)