Org:Institute/Open projects/Peach/Renderfarm
Some notes on the sun grid renderfarm
- Systems run Solaris 10 64-bit, AMD-Processors, 4gig ram. [[1]]
- Internet access is available (inside a solaris zone[[2]], only 1 processor can be accessed)
See Building Blender on solaris http://wiki.blender.org/index.php/BlenderDev/BuildingBlender/Solaris
目次
Interactive Session
When running an interactive session on suns network.com you need to have a script that connects to your system. This script is the script that runs as a part of a job.
Our internet can disconnect so I added a infinate loop that reconnects every 2min. this way you can only stop the job be canceling through the web interface.
interactive_session.sh - XXXXXXX replace with the ip/domain name.
#!/bin/bash
#
if [ -f $HOME/.profile ]; then
. $HOME/.profile
fi
while true; do
if [[ -f ~/session.lock ]]; then
echo "Session running fine"
# session must re-create so we know its running
# while true; do touch ~/session.lock; echo "keepalive"; sleep 20; done
rm ~/session.lock
else
echo "There is no xterm session running, restarting"
qsh -display blender3d.dyndns.org:1
fi
sleep 120
done
Packaging Blender for the sun grid
The directory structure is as follows
./bundle/* - archive to upload to suns grid, will be zipped into an 'Application' resource. ./blender/* - blenders source code. (svn checkout) ./lib/solaris-2.10-x86_64/* - Solaris libs, these are used for building but OpenGL libs from here are also needed to run blender on the renderfarm.
The Files for the package are as follows
Blender binary (statically linked)
./bundle/bin/blender
Python so blender can load modules like math
./bundle/pythonhome/
Scripts (contense listed below)
./bundle/blenderBatch.sh
./bundle/blenderBatchItem.sh
./bundle/blenderTestRun.sh
Scripts to run the renderfarm
./bundle/commandBatch.sh - Master script that runs every line in a text file, and retries if the command fails.
#!/bin/bash
#
if [ -f $HOME/.profile ]; then
. $HOME/.profile
fi
ifile=$1 # Use first argument
[ ! -f $ifile ] && exit
while read line_iter
do
echo "Running" $line_iter
EXITSTATUS="1"
while [ ! "$EXITSTATUS" -eq 0 ]
do
# This line should contain the qsub command!
$line_iter
# Deal with failed attempts to add a job.
EXITSTATUS="$?"
if [ ! "$EXITSTATUS" -eq 0 ]
then
echo "Failed to add job, sleeping 10sec and try again..."
sleep 10
fi
done
done < $ifile
echo "done"
./bundle/blenderBatchItem.sh - Called from blenderBatch.sh, renders each line in "blender.in".
#!/bin/bash
#$ -N Blender
#$ -cwd
export NAN_BLENDERHOME=$PWD
export OUT=${NAN_BLENDERHOME}/render
export LD_LIBRARY_PATH=/usr/local/lib:/usr/X11/lib:${NAN_BLENDERHOME}/lib
export PYTHONHOME=${NAN_BLENDERHOME}/pythonhome
echo $LD_LIBRARY_PATH
echo $NAN_BLENDERHOME
mkdir $OUT
echo $NAN_BLENDERHOME/bin/blender -b $@
$NAN_BLENDERHOME/bin/blender -b $@
'''./bundle/blenderTestRun.sh''' - Only for running blender locally and testing.
#!/bin/bash
export NAN_BLENDERHOME=$PWD
export NAN_BLENDEROUT=${NAN_BLENDERHOME}/render
export LD_LIBRARY_PATH=/usr/local/lib:/usr/X11/lib:${NAN_BLENDERHOME}/lib
export PYTHONHOME=${NAN_BLENDERHOME}/pythonhome
echo $LD_LIBRARY_PATH
echo $NAN_BLENDERHOME
$NAN_BLENDERHOME/bin/blender
Updating The Package
./update.sh - update and repackage the application to run on sun grid nodes.
(note, required for the renderfarm, just makes updating blender_app.tar.bz2 easier)
#!/bin/bash
PACKAGE="blender_app"
rm $PACKAGE.tar.bz2
svn update ./lib/solaris-2.10-x86_64
svn update ./blender
cd blender # blenders svn repo
gmake NAN_NO_KETSJI=true NAN_NO_OPENAL=true WITH_BF_BLENDERPLAYER=false
cd ..
cp ./blender/obj/solaris-2.10-x86_64/bin/blender ./bundle/bin/blender
strip ./bundle/bin/blender
# Update python from solaris lib
rm -rf ./bundle/pythonhome
svn export ./lib/solaris-2.10-x86_64/python ./bundle/pythonhome
# Copy libGL
rm -rf ./bundle/lib
mkdir ./bundle/lib
cp /usr/lib/X11/mesa/amd64/libGL.so.1 ./bundle/lib/
cp /usr/lib/X11/mesa/amd64/libGLU.so.1 ./bundle/lib/
cp ./lib/solaris-2.10-x86_64/python/lib/libpython2.5.so.1.0 ./bundle/lib/
# Remove some unneeded python files for a smaller package
rm -rf ./bundle/pythonhome/include/
rm -rf ./bundle/pythonhome/bin
rm -rf ./bundle/pythonhome/man
rm -rf ./bundle/pythonhome/lib/python2.5/bsddb
rm -rf ./bundle/pythonhome/lib/python2.5/compiler
rm -rf ./bundle/pythonhome/lib/python2.5/config
rm -rf ./bundle/pythonhome/lib/python2.5/ctypes
rm -rf ./bundle/pythonhome/lib/python2.5/curses
rm -rf ./bundle/pythonhome/lib/python2.5/distutils
rm -rf ./bundle/pythonhome/lib/python2.5/email
rm -rf ./bundle/pythonhome/lib/python2.5/encodings
rm -rf ./bundle/pythonhome/lib/python2.5/hotshot
rm -rf ./bundle/pythonhome/lib/python2.5/idlelib
rm -rf ./bundle/pythonhome/lib/python2.5/lib-tk
rm -rf ./bundle/pythonhome/lib/python2.5/logging
rm -rf ./bundle/pythonhome/lib/python2.5/plat-sunos5
rm -rf ./bundle/pythonhome/lib/python2.5/site-packages
rm -rf ./bundle/pythonhome/lib/python2.5/sqlite3
rm -rf ./bundle/pythonhome/lib/python2.5/test
rm -rf ./bundle/pythonhome/lib/python2.5/wsgiref
rm -rf ./bundle/pythonhome/lib/python2.5/xml
# Create with no subdirectory
cd ./bundle/
tar -cvvf ../$PACKAGE.tar *
cd ..
bzip2 -9 $PACKAGE.tar
Example .job file
This file is really really simple, it needs to be uploaded with any job you submit.
01_intro-04.job - Render's 6 frames at once!
qsub -N 01_intro-04_0001 -cwd blenderBatchItem.sh ./production/scenes/01_intro/04.blend -P ./production/scripts/render_settings.py -o 01_intro-04_ -f 1
qsub -N 01_intro-04_0002 -cwd blenderBatchItem.sh ./production/scenes/01_intro/04.blend -P ./production/scripts/render_settings.py -o 01_intro-04_ -f 2
qsub -N 01_intro-04_0003 -cwd blenderBatchItem.sh ./production/scenes/01_intro/04.blend -P ./production/scripts/render_settings.py -o 01_intro-04_ -f 3
qsub -N 01_intro-04_0004 -cwd blenderBatchItem.sh ./production/scenes/01_intro/04.blend -P ./production/scripts/render_settings.py -o 01_intro-04_ -f 4
qsub -N 01_intro-04_0005 -cwd blenderBatchItem.sh ./production/scenes/01_intro/04.blend -P ./production/scripts/render_settings.py -o 01_intro-04_ -f 5
qsub -N 01_intro-04_0006 -cwd blenderBatchItem.sh ./production/scenes/01_intro/04.blend -P ./production/scripts/render_settings.py -o 01_intro-04_ -f 6
qsub -N 01_intro-04_0007 -cwd blenderBatchItem.sh ./production/scenes/01_intro/04.blend -P ./production/scripts/render_settings.py -o 01_intro-04_ -f 7
Run "blender --help" for details on other arguments you can use.
Render Presets
render_settings.py - Use when you want to render many files with some of the same settings.
# run with...
# blender -b myblendfile.blend -P render_settings.py -a
from Blender import Scene, Object
import sys
def main():
# Was a camera name given as a python arg???
# eg...
# -- SomeCameraName
camera = None
argv = sys.argv
if argv:
try:
camera = Object.Get(argv[argv.index('--')+1])
except:
print "ERROR, Camera as argument could not be set '%s'" % argv[0]
# Settings for all scenes
for sce in Scene.Get():
rend = sce.render
rend.freeImages = True
rend.saveBuffers = True
rend.compositeFree = True
rend.oversampling = True
# rend.OSALevel = 8 # We dont want to set this now.
rend.threads = 1
rend.motionBlur = False
rend.fieldRendering = False
if camera:
sce.camera = camera
# Settings for render scene
sce = Scene.GetCurrent()
rend = sce.render
rend.extensions = 1
rend.renderwinSize = 100
#rend.enableSky()
#rend.enablePremultiply()
#rend.enableKey()
#rend.enableBWColor()
rend.enableRGBColor()
#rend.enableRGBAColor()
rend.compositor = True
rend.sequencer = False
rend.sizeX = 1920
rend.sizeY = 1080
rend.xParts = 6
rend.yParts = 6
rend.fps = 24
rend.zbuf = False
rend.halfFloat = True
rend.preview = False
rend.touch = False
rend.noOverwrite =False
rend.imageType = Scene.Render.OPENEXR
if __name__ == '__main__':
main()
Packaging Libs
These commands make a package with a binary and all its dependencies
MYBIN=/opt/csw/bin/svn
MYDIR=./bundle
mkdir $MYDIR
export IFS=$'\n'
MYLIBS=$(for f in `ldd -d $MYBIN`; do echo $f | cut -f3; done)
export IFS=$'\n\t '
for f in $MYLIBS; do cp -L $f $MYDIR/ ; done
cp $MYBIN $MYDIR/
tar -cvfh svnlibs.tar $MYDIR/
EDIT, setting the library path did not seem to work, used a single solaris svn binary from...
http://osdir.com/ml/version-control.subversion.rapidsvn.devel/2006-10/msg00020.html
SSH On Sun's Grid
Port 22 is blocked from the sun grid, so a default ssh configuration wont be able to call out from an interactive session. This is easy to work around by using port 80 (usually used for http).
This will only work if your server is not currently using http of course.
Server
On a linux system edit /etc/ssh/sshd_config Change "Port 22" to "Port 80"
then restart the sshd server
sudo /etc/init.d/ssh restart
Client
When using ssh to do access the command prompt just add -p 80, for eg..
ssh user@biserver -p 80
When using svn you need to set the SVN_SSH environment variable
export SVN_SSH="ssh -p 80"
Now you can do an rsync copy through ssh with port 80
For rsync
rsync -ave ssh ~/test.txt myname@url.org:/home/myname/
Becomes...
rsync -ave 'ssh -p 80' ~/test.txt myname@url.org:/home/myname/