Dev:2.5/Source/Python/API/Py3.1 Migration

Python 3.x migration proposal

A while ago we agreed (general consensus)

• bundle python with blender on all OS's
• include all python modules with blender so scripts work without worrying about system python. (with the exception of test, distutils, and tk) ~4mb last I checked on win32.
• to use a single version of python not mix (no need to support both 2.x and 3.x)

Refer to the mail thread...
http://lists.blender.org/pipermail/bf-committers/2008-December/022295.html

Trying to support 2.x and 3.x wastes too much time with the C/API and also becomes problematic for script compatibility. Since python 3.x now has a stable release I think now is a good time to make the move.

Status

• Blender 2.5 builds against python 3.x
• User interface scripts in blender 2.5 currently run 3.x
• Python 3.x is included in the "lib" folder (used by windows and mac builds)
• On Windows all of Python 3.x modules/libraries are bundled with Blender. If built with scons, a complete package including Python 3.x is found in BF_INSTALLDIR. With other build systems the Python zip from lib/windows/release or lib/win64/release needs to be extracted to the correct location (.blender/scripts/python/lib)

Including the Python Library

This means distributing blender without the dependency on libpython3.x.so, or python3x.dll on windows. (Only our windows releases currently do this)

For all other OS's (unix based) we can either statically link libpython3.x.a to blender when building or include libpython3.x.so with blender.

I don't mind either way; however if we include the library separately we'll need to start blender with a shell script to set the LD_LIBRARY_PATH (firefox, maya and many other apps that include their own separate libs on linux, using a shell script to start).

We could do static linking to start with, its easy to change later on, can be decided per OS.

Including Python Modules

Note that on Windows since Blender 2.5 Alpha 0 below directions, including the next section, are no longer necessary. Everything is ready in BF_INSTALLDIR

Python modules are a mixture of .py files and compiled modules (.so files on linux)

This means copying /usr/lib/python3.x into some directory blender can be bundled with eg. ~/.blender/python (from C:\Python3x\Lib on windows)

Note that blender won't start unless these modules are found (error below), so it's important to get this path right.

Fatal Python error: Py_Initialize: can't initialize sys standard streams
ImportError: No module named encodings.utf_8

Python then needs to know to use this path for the modules rather than /usr/lib/python3.x There are a number of ways to do this....

1. Define our own Py_GetPath() that overrides pythons. See http://docs.python.org/c-api/init.html
2. Set the PYTHONPATH and PYTHONHOME (from C) before initializing python.
3. Start with a shell script that sets the PYTHONPATH/PYTHONHOME (all except win32).

What about zipping the modules

Python supports reading modules from a zip file rather than a directory. While the option is always open, I'm against this for a number of reasons.

1. Py3.x by default builds without zlib modules in libpython3.x.so; this means loading a zip as a module fails unless you build python with and edited Modules/Setup
   *Note* I only tested this on linux, perhaps windows or mac are different since python26.zip works on windows for blender 2.4x. because zlib.pyd is outside the zip

1. a small zip is fast to extract but the bundle will be around 4mb (compressed), a larger file could become slow - since this will be read on initializing blender/python.
2. We might want to include more modules (numpy, pygame, PIL for eg), at some point an even larger zip file could slow down blender's startup time.

The assumption about speed would need to be verified but needing to edit Modules/Setup before building python is reason enough for me not to do this.

Removing Modules

Directory's in the lib dir that can be removed.

• test
• distutils
• lib2to3
• idlelib
• tkinter
• config (not a module but contains files we don't need)

Test Modules...

• ctypes/test
• email/test
• importlib/test
• json/test
• sqlite3/test

Dynamic libs

• lib-dynload/_tkinter.so

Also...

• all *.pyc and *.pyo files
  rm find -name "*.pyo"
  find -name "*.pyc"`

Strip dynamic libs to save space

 strip -s lib-dynload/*.so

Proposal

Even though there are many options I think these should be the least hassle for devs and users.

• statically link python on all OS's (except windows which can include the DLL)
• distribute python modules in a directory with blender (~/.blender/python)
• include Python3.x in the lib/ directory.
  Linux devs have the option to use the lib folder or install python 3.x on their system.

Advantages with this method are...

• developers who have python3.x currently won't notice any difference
  (since system's python is still used).
• our own python library will be used, which is predictable for users.
• static linking, no need for shell scripts to start the blender binary on unix.

Tasks

• Use ~/.blender/python for modules. Done - rev 21334, falls back to system python if not found
• add pre-built python to the lib/ dir for MacOS/Windows/Linux (done)
• make static linking python default on MacOS and Linux. (not urgent)
• building release should copy modules into .blender/python (done)

--Ideasman42 01:26, 3 July 2009 (UTC)(Sign)

See also

Dev:Doc/Building_Blender/Linux/Troubleshooting#Python

Python Upgrade Policy

This section has been added after discussion on the bf-committers mailing list, see: [1]

When upgrading from versions of python 3.2 -> 3.3 for eg. This is problematic for Linux since we dont provide pre-compiled libs.

From discussion after 3.1 -> 3.2 upgrade here are steps we should take before dropping support for the previous python version.

• All platform maintainers first agree to drop support.
• Mail the list beforehand and include a note in meeting minutes that the python version will be no longer supported.
• Ensure build instructions are up to data for Ubuntu/Fedora to install packages (most likely from testing repo's).