Dev:2.4/Source/Build systems/Scons/Refactoring - Blender 2.42

by Nathan Letwory

assisted by lukep, Jonathan Jacobs, Johnny Matthews

Current status

Progress: The work sofar has been committed to CVS and development will continue from there on, so no more patches from here - just get the current CVS Note win32/mingw must be compiled WITH_BF_OPENEXR=0 , because there is no proper library in cvs yet - if you do have your own, use WITH_BF_OPENEXR=1 by all means :) Note Through sheer stupidity I accidently deleted the directory with mingw-compiled and -compatible libraries to link agains. These were for commit to the lib/windows module, but I have to search and rebuild these libs again. Sorry for the inconvenience.

Attention: (platform)-config.py -files needed for different platforms. See linux2-config.py, darwin-config.py, win32-vc-config.py and win32-mingw-config.py in config/ for template

blender-scons user docs

blender-scons-dev developer docs

Don't forget

• done
   added intern/elbeem/SConscript needs USE_MSVC6FIXES to compile ok for .net < 2003
• done
   added add the BF_OPENGL_LIBPATH note to my Don't Forget section on the SConsRefactoring page

Roadmap

• to do
   test test test
• in progress 50%
   documentation
• done
   port to orange Orange was merged with HEAD and is now HEAD
• in progress 75%
   compile flags
  • to do
     cygwin
  • done
     darwin (osx)
  • to do
     solaris
  • to do
     freebsd
  • done
     linux2 (dynamic)
  • done
     win32 (mingw)
  • done
     win32 (msvc)
• in progress 75%
   release build
  • to do
     gettext .mo creation
• in progress 25%
   cross-compile support
• in progress 75%
   profile build
• done
   clean up hard-coded tests
• done
   debug build
  • done
     directory specific debug flags
• done
   globbing of source directories
• done
   silent Builder for cleaner output like nan makefiles
• done
   create wrapper functions for blender library building
• done
   make central libraryname repository crossplatform , ie. mingw/linux does libsomename.a, whereas windows wants somename.lib
• done
   BF_QUICK=lib1,lib2, build only lib1 and lib2, then relink (libs in a comma-seperated list)
• done
   parse arguments and targets
• done
   parse configuration file
• done
   build dir
• done
   add libraries
• done
   platform specific code

Feature Requests

• It would be great if this could be developed with similar ideas to the makefile refactoring I've been working on. (I'm willing to change things in it as well to conform if needed.) I'm thinking things like variable names should be similar, build targets, etc... Here is the current implementation of it, I need to figureout the gameengine/plugin rules so that the seprate targets build correctly. After that I will submit it to the patch tracker. Anyway here is the link: http://www.cs.umn.edu/~mein/blender/makefiles.tgz

• scons vs config.opts -- Being able to recognize that the SConstruct file is newer than than config.opts and config.opts needs to be regenerated would be nice. Everyone (meaning me) always forgets to delete config.opts after changing SConstruct.

• A Builder for generating the BPy API docs using epydoc.

Arguments and Options

I'll use arguments as per Kent Meins makefile rewrite. See the above link for info. Additionally the following arguments shall be implemented: Almost done: rename arguments below to be similar to Meins used arguments: BF_ARGUMENT see current arguments/options

I decided to do a similar setup for platform-specific settings like the makefile: scons/(platform)-config.py. To override, just create a user-config.py in the rootdir. If you want event to override for ie. a single run a value from user-config, just specifiy it as an argument on the commandline.

• in progress 25%
   BF_CROSS=1 if 1 do a cross compile (on linux for windows)
• on hold
   update=1 if 1 get sources from CVS before compiling
• done
   BF_BUILDDIR=/build/dir/path use the specified path as root_build_dir
• done
   BF_TOOLSET=gcc use the specified compiler (toolset, see SCons documentation for available toolsets)
• done
   BF_INSTALLDIR=/build/dir/path use the specified path to install to
• to do
   program-name=orange build the executable as orange (instead of default banana)

Targets

The following targets shall be implemented:

• done
   blender: builds dynamic blender (this is the default, so just 'scons').
• to do
   blenderstatic: builds a static version of blender.
  • this is for static linking against OpenGL libraries
• in progress 50%
   blenderplayer: builds the stand alone game engine - linking problems.
• to do
   webplugin: builds the webplugin.
• done
   blendernogame: builds dyamic blender without game engine support.
• to do
   blenderstaticnogame: builds static blender without game engine support.
• in progress 75%
   release: builds release for whatever packages you have built.
• in progress 25%
   everything: builds blender, blenderstatic,blenderplayer, webplugin and release
• done
   clean does what you would expect.
• done
   install install blender and all necessary files to BF_INSTALLDIR
• done
   install-bin only install the executables to BF_INSTALLDIR
• (installer: create installer (win32))

Cleaner Output

done

 It would be nice to have similar progress printing as the nan makefiles do.

$ scons
building default
== source/blender/avi
== source/blender/blenapi/
== source/blender/blenkernel/
etc.

See http://www.letwory.net/albums/v/Rt/ for some screenshots of the new output.

Comparing old and new

Test figures from my setup (Dual AMD MP 2200+, 1GB RAM). Although it is a dual-processor system, only one processor was used (no -j4 switch), so just vanilla 'scons' with the --debug option as below.

	old	new
--debug=count	BuilderBase: 135 CommandAction: 2333 CommandGeneratorAction: 107 CompositeBuilder: 72 Executor: 939 FS: 1 FunctionAction: 14 LazyCmdGenerator: 35 ListAction: 155 MultiStepBuilder: 61 Node: 30912 Node.FS.Base: 30908 Node.FS.Dir: 1685 Node.FS.File: 29221 Node.FS.RootDir: 1 SConsEnvironment: 12	BlenderEnvironment: 1 BuilderBase: 35 CommandAction: 919 CommandGeneratorAction: 43 CompositeBuilder: 12 Executor: 977 FS: 1 FunctionAction: 12 LazyCmdGenerator: 31 ListAction: 53 MultiStepBuilder: 13 Node: 27496 Node.FS.Base: 27494 Node.FS.Dir: 1592 Node.FS.File: 25900 Node.FS.RootDir: 1 SConsEnvironment: 1
--debug=time	scons: done building targets. Total build time: 804.516563 seconds Total SConscript file execution time: 2.534984 seconds Total SCons execution time: 59.832944 seconds Total command execution time: 742.148635 seconds	scons: done building targets. Total build time: 526.923281 seconds Total SConscript file execution time: 2.012409 seconds Total SCons execution time: 43.008393 seconds Total command execution time: 481.902479 seconds

Old news

Jonathan Jacobs has sent me (NL) a very much cleaned implementation of scons for Blender. I'm currently working on getting it to work on win32 with msvc in a bare minimum setup to use as a further working platform.

Scons/Mingw/Win32 tutorial

NOTE Due to a very nice fix in June, 2007, the short-path workarounds are no longer necessary. You can safely skip these steps now and use the default build and install directories.

For the purposes of this exanmple Im assuming your blender source dir is "c:\blender" and your build dir is "c:\b"

Required Software

• TortoiseSVN from http://www.tortoisesvn.org/
• Python 2.4.x or greater from http://www.python.org
• Scons from http://www.scons.org (Tested with 0.96.91 - pre-release)
• Mingw from http://www.mingw.org (Tested with MinGW-5.0.2.exe - [1]) *Note, People have reported problems with mingw releases before 5.0.2, it is recommended you try whis release first
  Run MinGW-5.0.2.exe. It will ask you to select a mirror. Choose one close to you.
  Then select which Mingw package to install from Previous/Current/Candidate (Select Candidate) and packages. install packages "MinGW base tools" and "g++ compiler"
• or you can use Dev-C++ from http://www.bloodshed.net/devcpp.html as it is easier to setup and comes with mingW's g++,gcc compilers (but it lacks many of the optional programs that come with mingW) and also provides a nice IDE to code your little blender projects

Modify your path

To access commands directly from the command line you need to add some new directories to your path.

Go to Start Menu >> Control Panel >> System >> Advanced (Tab) >> Envoronmant Variables (button) >> System Variables scroll area double click on the PATH item

At the end of the existing path text add the following

;C:\Python25;C:\Python25\Scripts;C:\Program Files\TortoiseSVN\bin;c:\mingw\bin 
 
if you are using dev-c++ add the following 
;C:\Dev-Cpp\mingw32\bin;C:\Dev-Cpp\mingw32\lib;C:\Dev-Cpp\bin

Make sure to close all your command line windows before going further, the new path will not take effect on any cmd.exe's that are alredy open. Start Menu >> Run >> cmd.exe (Type into text feild)

Make sure you have commands available in your path. you should be able to type in the commands "python", "cvs", "g++" and "scons", 'note that running these commands by them selves will not do anything'

Getting the sources

The commands below make the folder c:\blender and populate it with the folders "blender" for the blender source code and "lib" for the windows spesific libraries

 mkdir c:\blender
 cd c:\blender
 svn checkout https://svn.blender.org/svnroot/bf-blender/trunk/blender blender

 mkdir c:\blender\lib
 cd c:\blender\lib
 svn checkout https://svn.blender.org/svnroot/bf-blender/trunk/lib/windows/ windows

Once you have these directories downloaded, you can keep them up to date with commands from the commandline (run cmd.exe as before).

 cd c:\blender\blender
 svn update
 cd c:\blender\lib\windows
 svn update

Configuring the Build

To set options for the build you will need to create the file c:\blender\blender\user-config.py When building for the first time it can save you a headache if you turn many of blenders features off. Here is the user-config.py I used to build blender with mingw for the first time.

 WITH_BF_OPENAL = 'false'
 WITH_BF_GAMEENGINE='false'
 WITH_BF_BULLET = 'false'
 WITH_BF_INTERNATIONAL = 'false'
 WITH_BF_OPENEXR = 'false'
 BF_PYTHON_VERSION = '2.5' #(whatever version you have installed)

Feel free to enable eny of these.

Compiler Flags

If you want you can specify custom compiler flags (options that are passed to the compiler - gcc, which can make fairly big performance improvements if used correctly)

Compiler flags are also set in the c:\blender\blender\user-config.py file.

It's important to use the defaults as a base and add more options from there, otherwise the .exe you make will have problems (in my case the screen started up and most of the display was black)

The default flags at the time of writing are

 CCFLAGS = ['-pipe', '-funsigned-char', '-fno-strict-aliasing' ]
 CXXFLAGS = ['-pipe', '-mwindows', '-funsigned-char', '-fno-strict-aliasing' ]

Check c:\blender\blender\config\win32-mingw-config.py for CFLAGS and CCFLAGS variables before you copy these flags as your default. you may want to add the optimize flag -O2 for a faster build or -Os for a smaller yet still optimized blender.exe

An examle of my flags

 CCFLAGS =   ['-pipe', '-funsigned-char', '-fno-strict-aliasing' ]
 CXXFLAGS = [ '-pipe', '-mwindows', '-funsigned-char', '-fno-strict-aliasing' ]
 
 # Add extra flags
 CCFLAGS.extend(   [ '-march=i586', '-Os', '-fomit-frame-pointer', '-mmmx' ] )
 CXXFLAGS.extend( [ '-march=i586', '-Os', '-fomit-frame-pointer', '-mmmx' ] )
 
 WITH_BF_OPENAL = 'false'
 WITH_BF_GAMEENGINE='false'
 WITH_BF_BULLET = 'false'
 WITH_BF_INTERNATIONAL = 'false'
 WITH_BF_OPENEXR = 'false'
 BF_PYTHON_VERSION = '2.5' #(whatever version you have installed)

Compiling Blender

Build blender with the command (Run cmd.exe) and type in...

 cd c:\blender\blender
 scons BF_TOOLSET=mingw

scons may take a while to compile so be patient (5 to 25min depending on the speed of your computer). Generally after that you can update the source with the CVS and running the compile again will only compile whats needed, taking a minute or 2.

When the build is done you will have a "blender.exe" in "C:\blender\install\win32-mingw"

Automating the build process

Typing in scons BF_TOOLSET=mingw all the time is a pain. So its a good idea to make a build script.

Open a text editor and copy the following into it.

 cd c:\blender\lib
 svn update
 cd c:\blender\blender
 svn update
 
 scons BF_TOOLSET=mingw
 strip C:\blender\install\win32-mingw\blender.exe
 pause

save this file as c:\blender\build.bat, you can double click on this or run from the command line.

SVN update commands are not needed but nice to keep up to date with daily SVN additions. strip will cut about 1 meg off you blender.exe, just removes unneeded data from the exe.

Cleaning the build

If you want to clean the build, you may have upgraded your compiler of changed flags,
you will first need to make a clean build.
Sometimes updates from SVN require you to make a clean build. So open up a text editor and save the following as c:\blender\build_clean.bat

 cd c:\blender\blender
 scons clean BF_TOOLSET=mingw

Hope all goes well.

Debugging

If you can compile Blender and are interested in development, then youll probably want to debug your code. There are 2 steps to this

1. Compile a debug build
2. Install a debugger

Compiling a debug build

is done by adding "-g" to the compiler flags. Its also a good to turn off optimizing flags like "-O2" when debugging. The resulting exe from a debug build may be 30 to 50 meg.

Installing a Debugger

I tried a few debuggers and insight seemed to be the best. Get Insight.exe here [2]

just extract it whereever you like.

Open insight\bin\insight.exe and from insight open your debug Blender.exe

Then run blender.exe by pressing the running man icon on the tool bar.

It will automaticaly break at main(), just press the "C" key to launch blender.

If you crash blender, go to View >> Trace and you can see the backtrace and what went wrong.