Dev:2.8/Source/Depsgraph/Planning-October2017

Dependency graph planning

Introduction

This document covers both on-going targets of dependency graph which needs to be addressed in Blender 2.8 branch and also all the targets needed to get fully operational Copy-on-Write, animation and modifiers support.

Categorized list of tasks

Here goes list of all tasks to be tackled, categorized into groups. Since there is inter-dependency between tasks, the actual roadmap will have those in a different order.

Technical targets

• Switch to new ID management from Bastien. It will:
  • Reduce amount of ID management logic in dependency graph itself
  • Make it more future-proof for all possible “nested” ID data blocks, such as NodeTree data block inside of Lamp.
  • Reduce number of temporary allocations.
  • Solve the issue with user counter being increased on every ID update.
• Remove ID tagging for update with a flag of 0.
  • This is unclear which ID components to tag for update and currently there is quite some magic which tries to deduct what to tag and what not. Having separate tag flags will make code easier to follow and more robust in the long-term.
• Remove legacy object->update flags
  • Those are getting confusing with the new tagging system and often are conflicting with each other.
• Re-scheduling of out-of-date data blocks on dependency graph rebuild is to be brought back.
  • Keep in mind, that original dataset is always out-of-date, so need to check whether evaluated counterpart is fully updated or not.
• Reshuffle collections base flags evaluation, make it so object is gathering its base flags from collections.
  • Avoids possible threading conflicts, and also removes relations which causes ALL objects to be updated when changing collections.
  • Need to first solve dependency graph ownership task.
• Go over all operators, make sure they are aware of copy-on-write concept and does proper tags after data modification.
• Make uniform buffers to work with copy-on-write.

Data ownership

• Make dependency graph being owned by (workspace, scene, layer, render engine) combination.
  • This way we can have properly working overrides from collections (currently, all collections are in the dependency graph, which breaks some logic).
• Long term task: allow sharing evaluated data across dependency graphs
  • This is so we can avoid evaluation of same object twice in different dependency graphs if they are evaluating within matching contexts.

Visibility flags

• Topics which needs a design / decision:
  • Local view
• Port DEG_graph_on_visible_update() to a new layers/collections concept. Avoid extra tagging as much as possible.

Copy on write

• Make all evaluation callbacks to receive CoW data block pointer.
  • Easy task, just go over the code and do modifications.
• Remove any possible access to G.main or other global states, use EvaluationContext instead (this is about data block evaluation functions).
• Topics which needs a design:
  • When CoW component gets tagged for update (aka, when we copy data from original data block to a copied one: are we doing this on any ID tag?)
  • What properties CoW component should preserve (for example, shall it preserve evaluated mesh? Shall it preserve object matrix?).
  • Shall we introduce per-component CoW operations?
• Implement all design decisions from above :)
• Move update tags from original data block to a copied one.
  • We need to tag all CoW data blocks corresponding to the changed original one, so we can handle their update independently.
• Overrides are implemented on top of CoW data block.

Modifiers support

• Topics which needs a design / decision:
  • Introduce Object->evaluated_mesh

    This became rather a complicated logic to deal with evaluated object’s mesh pointing to a mesh with all modifiers applied. It will be more clear to separate CoW object->mesh (which is a CoW data block of original mesh) and have object->evaluated_mesh which has all modifiers applied. Makes it more clear what is what.

  • Need a design about Subsurf mesh drawing (we don’t necessarily want to convert subdivided mesh to a dense mesh).
• Port modifiers from DerivedMesh to Mesh.

Query API

• Hide “private” parts of dependency graph iter structure.
• Remove any mention of base from iteration API or internal logic.
  • Dependency graph does not keep track of bases, those are lost after construction of dependency graph is over.
• Get rid of flags / settings flush from iterator.

Data masks

• Bring back a system which checks which CustomData layers are to be evaluated by modifier stacks.
  • Need to solve old issue with Cycles viewport being slow to initialize when switched from Solid mode due to requirement of extra CustomData layers.
• Reconsider evaluation flags stored in ID node.
  • Those are used in handful of places: OpenSubdiv and curve path evaluation.
    • OpenSubdiv will need to be moved away from modifier stack, so those evaluation flags might be simply removed.
    • For Path we can either set flag in copied version of the datablock or we can make it so dependency graph builder puts path evaluation operation for a target curve.

Support of non-mesh objects

• So far all design is being done around Mesh objects. This is to be defined how we want to deal with Curves, NURBS surfaces, Fonts and Metaballs.
  • Hopefully supporting those will give a better idea of how to support multires, with all the optimal drawing.

Various

• Lots of little TODOs which needs to be addressed, happens all over the code.
  • Addressing those will be spread over the whole duration of the project.
• Bug fixes, both for the studio and bug tracker.
  • Those are tricky to plan, some of issues might take a few minutes others might take a day.
• OpenSubdiv needs to be moved out of modifier stack.

Rough Planning

GOAL: Make Blender 2.8 usable for Hero and upcoming open movie.

NOTE: Here are only bigger tasks are listed, just to define overall priorities of development. Non-listed tasks are not THAT visible for artists and can happen on background in parallel.

• Make copy-on-write a command line argument rather than a compile option (2 hours)
• Switch to new ID management API (day? With all testing)
• Make dependency “owned” by (workspace, scene, layer) triplet (few days)
• [COULD HAPPEN IN PARALLEL WITH HELP OF OTHERS] go over operators, make sure they are usable within CoW idea. All those little things like 3D cursor position not being updated etc.. (1 week)
• Make all evaluation callbacks to receive CoW pointer (which is simple, 1 day)
• Port modifiers needed for facemaps (1 day -- make sure armature and shape keys are working)
• [COULD HAPPEN IN PARALLEL WITH HELP OF OTHERS] Port rest of modifiers. (3-4 weeks = 35 constructive modifiers including simulation,4 hours for, plus 1-2 days for all deformation modifiers)
• Support curves/nurbs/metas (1-2 weeks?)
• Support of multires / sculpting (week / two?)
• Full support of overrides (any rna datapath - viewport settings are supported already) (day work in depsgraph, but need UI)
• [HAPPENING IN PARALLEL WITH EVERYTHING]: solving smaller tasks / todos / bug fixes.
• Approach to a longer term goals:
  • Proper OpenSubdiv integration (as a draw / render option outside of modifier stack. With adaptive subdivisions, proper normals and everything) (4-6 weeks, based on experience of previous integration)
  • Sharing data across dependency graphs (weeks..)

Attempt to organize those in weeks/month

October

• Week 3
  • done
      Make copy-on-write a command line argument rather than a compile option
  • done
      Switch to new ID management API
  • done
      Make dependency “owned” by (workspace, scene, layer) triplet
• Week 4
  • done
      Finish ownership - 1d
  • in progress  %
      Port operators to CoW so others can help - 2d
  • done
      Blender Conference
• Week 5
  • done
      [post BConf / Cycles meetings ... ]
  • done
      Make all evaluation callbacks to receive CoW pointer - 1d
  • to do
      Port modifiers needed for facemaps - 1d
  • in progress  %
      More operators to CoW

November

• Week 1
  • to do
      Finish porting operators to CoW - 1w
• Week 2
  • in progress  %
      Modifiers - 1/4w
• Week 3
  • to do
      Modifiers - 2/4w
• Week 4
  • to do
      Modifiers - 3/4w

December

• Week 1
  • to do
      Modifiers - 4/4w
• Week 2
  • to do
      Support curves/nurbs/metas - 1w
• Week 3
  • to do
      Finish support for curves/nurbs/metas
  • to do
      Polishing code / bug fixes.
• Week 4
  • to do
      [week between Christmas and New Year’s Eve]

January

• Week 1
  • to do
      Support of multires / sculpting - 1/2w
• Week 2
  • to do
      Finish support of multires / sculpting - 2/2w
• Week 3
  • to do
      Buffer week, to finish up pending tasks, technical debts, ...
• Week 4
  • to do
      Full support of overrides - 1w
• Week 5
  • to do
      New round of planning, focus on wrapping depsgraph, OpenSubdiv (4-6 w) and assist wrapping up other areas of 2.8