Org:Institute/Pipeline

Pipeline Tools

Introduction

This document reports the planning and progress of the development of pipeline tools for Open Movie projects. Such tools are meant to be free and available to anyone who wants produce computer animation or games. The primary focus of the tools is animated feature film production.

Check out the work in progress here.

The Story Tool

This is the over arching goal of the project.

• Asset Management Manage Files & Content
• Project Management Manage People & Tasks
• Automated Tasks Manage Geneated Content

With these three components is theoretically possible to manage all the aspects of a very large production pipeline. It is our intention to create a set of modular tools to cover these different areas.

Design Goals

This is a tool, not a framework, for anyone making animation (mainly) with Blender. As a minimum viable product, the Pipeline tools are designed to:

• Handle multiple projects
• Foresee use of other tools (as well as Blender), in the work-flow
• Support multi-site/distributed work-flow
• Support some of the functionality in the Blender-Cloud
• Design to allow for swappable modular components, even if we end up sticking with single technologies

Asset Managment

• Assets <-> Users
• Revisions
• Variations

Architecture Overview

Server

• File asset
• Tools (blend file packer, evaluate sequencer, visualize deps, automated tasks ...)
• Public API (web service), communicates with client.

Client

• Interface GUI/CLI (Blender/Web-UI also)
• Tools (manage files on the client, cache.)
• Local files (models, images)

Implementation Details

This document describes the layout for Blender pipeline.

Overview

• Use SVN for internal storage.
• SVN repository is for internal storage (but keep usable as *last resort*)
• Support extracting single .blend file, and committing it back (without a full checkout), useful for remote artists.

SVN Commit Abstraction

Motivation: Artists need to be able to work on jobs without downloading entire repository.

Workflow:

• Select an asset to checkout (Likely via a web-ui/blender-ui).
• Download the asset and its dependencies (web/cli/blender-ui).
• Modify data locally (images, 3d... text... etc).
• Submit task back with commit log (blender-ui/cli/web?). (Server handles commit).

Technical details:

• Server handles SVN integration which is hidden from the client.
• The job submission and editing workflow is handled by client/server,
• Server creates binary blobs for the bundles, client handles download and create a usable directory for editing.
• Path remapping of .blend files must be handled(in both directions, likely using blendfile.py).
• Use cache on client to avoid re-downloading the same resources.

Asset Managment

This part will be expanded soon.

Automated Tasks

There are 2 types of tasks to be automated.

• User submitted tasks.
• Tasks generated by events such as commit hooks, finished rendering... etc.

Automated tasks are broken into 3 steps.

• Creation (API/CLI/GUI)
• Scheduling/queueing (Managed by the server)
• Execution/job management (Controlled via the server, though API's & UI's)

User Stories

Here's a list of tasks we would expect the system to support

• Generating Renderfarm Preview
• Low resolution textures for animation
• High Resolution Simulation (hair, smoke)
• Final Render a Scene
• OpenGL Preview Every Shot
• Bundle a Blend file into a ZIP
• Synchronizing Data (SVN/Database... repositories... backups)
• Consistency checks (automated tasks to validate the state of the project)
• Blend file hygiene/lint (unused datablocks, images not used anywhere)
• Building Blender

Implementation

We plan to develop a very simple system leveraging existing technologies.

Dashboard

UI (web based), allows manual creation of tasks.

Server

Backend connected to database and scheduler, managing & assigning jobs to workers.

Worker

Simple client, exposes control of the machine via an API.

Milestones

1. Minimum Viable Product

Get existing project migrated to asset manager as soon as possible.

• No Client-side-SVN.
• Checkout/Commit assets.
• Basic UI.
• ZIP file access (as an example, users wouldn't have to notice)
• Testing (basic testing framework)

2. Performance

• Parallel downloads
• Cache (avoid re-downloading files)

3. Interface

• BAM project definition.
• Blender Addon (UI for performing basic actions)
• Maybe extend GUI client.
• Web UI
• Download Zipfiles

4. Scaling/Project

• Consistency checks (correct library linking)

5. Basic Automated Tasks

• Generated Previews (basic automated tasks hooked up to asset manager)
• Baking (physics)
• Web UI (create/view/manage jobs)

6. Story Tool

• Establish connection between sequencer and 'shot' assets *(abstract concept)*
• Integrate into Addon
• Connect automated tasks on edit updates, (sync sequence with short assets)
• Expose the edit outside of Blender *(web based UI, reviews, comment, feedback, tasks...)*

7. Project Management

• Web UI
• Connect basic communication tools to assets, commits.

Repository

Do you want to have a look at our work and give it a try? You can check it out on developer.blender.org. At the moment there are no instructions on how to set it up as we are working on it as a simple prototype. Better documentation is on our roadmap.