Dev:2.5/Source/Architecture/DataAPI

Data API

This document is about the design of the Data API, how to use it for in the code is explained in the RNA documentation, and there is also a higher level document on how to define properties in Blender.

The Data API will provide access to data structures and their members. There are many applications of this system, and the design proposal attempts to take all of these into account. A list of possible applications:

• Automatically generated user interfaces with lists of data properties.
• Access to Blender data by plugins, for example in a render API.
• Automatic python scripting access to blender data, without manually written code that might be duplicated elsewhere.
• A uniform interface for native and custom defined properties.
• Making all properties animatable and proxyable.
• Generating documentation.

It will be a layer over the existing DNA structs, abstracting how the data is actually stored, and providing information about structs and their properties. It does not affect how the data is currently laid out in the DNA structs, and RNA structs and properties need not correspond exactly to any DNA struct or member, rather it is a higher level interpretation of this data.

-- Brecht 22:39, 31 October 2008 (UTC)

Implementation

The current implementation is available in the 2.5 Branch, under source/blender/makesrna. Very similar to DNA it runs as a preprocess, generating the struct RNA in advance.

There have been two earlier initial implementations of such a system based on parsing extra information in DNA files. One of those is in the animsys-aligorith branch written by Joshua: branch, sdna2kt.py.

The other is available as a patch against trunk revision r14532 and written by Campbell.

This document attempts to give a clearer overview of different data types, required information about properties and integration.

Data Definition

All Blender data is presented as structs, each being a collection of properties. These properties are all of one of these types:

• boolean
• string
• integer
• float
• enum
• pointer to a struct
• collection of pointers to structs

Additionally, booleans, integers and floats can optionally be an array with fixed length. This definition should ideally be sufficient to encompass all data. Next to the type a subtype is also defined, examples of which are file path, color or matrix.

Next to getting and setting the values of properties, information about each struct and property can be queried. That includes the name, description, type, number minimum/maximum value, pointer type, etcetera.

The RNA definition does not necessarily correspond to actual DNA structs and members as used in the Blender source code. It is an abstraction of the struct defined in the DNA header files, in order to hide implementation details. For example a boolean may actually be a bitflag of an integer, but the user of the API does not need to know this.

Many of the Blender data structures map straightforward into this format, however some are less obvious:

• C chars and shorts representing numbers map to ints, they simply have different min/max.
• The vertices of a mesh are represented as a collection.
• Nested structs may be represented as pointers to structs that are not editable.
• Tristate values as used for texture mapping are represented as enum with 3 choices (disabled, enabled, enabled inverted).
• Colors are represented as float arrays with length 3 or 4 and subtype color, and matrices are represented as float arrays with length 9 (3x3) or 16 (4x4) and subtype matrix.

Notes

• There are no 64 bit ints and doubles defined as types. They are currently not used in DNA where it matters, so they were left out.
• There is no unsigned integer type, although a C int can't hold the same data as an unsigned int. An unsigned int type can be added if there is a case where we need these higher numbers, but I can't think of a practical case so I'm leaving it out, and defining unsigned as a subtype of int.
• There is no support for inheritance currently. For ID structs, modifiers and constraints a simple form of this, inheriting properties may be useful for example, though it's unclear what implications this would have elsewhere.

Scope

The scope of this data API also needs to be defined. Clearly all data that is saved to file and editable in the user interface should be available through the API. However, what to do with data that is generated at runtime, like object matrices, image buffer pixels or derived meshes is less clear. This data must clearly be available to python or a render API for example, and preferably in a uniform way, however it is unclear which role the data API should play, if any.

At the very least, the data API should be compatible with API's for runtime data, with consistent function names, etc. But I think right now the focus should be on wrapping basically all data that is editable in the user interface and data that should be keyable.

Data Interface

All data will be accessible in such a way that user interfaces or documentation can be generated by going over all structs and properties, without having to know anything about the specific data structures. For example it will be possible to get a list of properties in a struct, and for pointer type properties it will be possible to query the struct type. Additionally, for native Blender data an API a get/set function per property may be generated, for use in plugin APIs, or even internal Blender code.

Properties of the collection type will be accessible through iterators, that make it possible to go over all collection members sequentially. Additionally, collection members may also be looked up with a key. By default that will be an integer to look up the n-th member in the collection, but for others this may be a string with the ID name for example.

Using the data API it will be possible to define a complete path to every struct and property, preferably in a way that is very similar to the experimental bpy python API in order to automatically generated parts of the python wrapping code there automatically.

Notes

• Iterators are generally ugly to define in C, but I cannot think of another abstraction that would not do memory allocation all the time. Additionally this should fit well with python, and most iterators would simply be using arrays or ListBase, so code can be shared here.

Data Mapping

As the data exposed by the API does not necessarily correspond directly to the way the data is stored, it is always accessed through a get/set interface, and an iterator for collections. However, most data corresponds to DNA data in a very similar way, only for few properties this mapping would to be written manually. Therefore most get/set functions would be generated automatically based on information available from parsing the DNA structs. Examples of such mappings would be string to fixed length char array, boolean to bitflag in a short, or collection to ListBase.

Validation, Context and Notifiers

When a property is being set, it must be verified to be in the limits of the allowed range, or if the property can be written at all in the current context. The set() function will clamp the values as needed. For each property type, a number of limits and constraints can be specified that will automatically be taken into account when generating code. Manually written set() functions may verify additional constraints.

A notify() function will also be available to call the necessary notifiers after changing the data. A modify() function will also exist, calling set() and notify(). The set() and modify() functions do almost the same thing then, the right function to call depends on the purpose. In an operator the modify() function would usually be called, while in dependency graph (IPO's, ..) evaluation, set() would be called.

With set() functions being called in the wrong context or outside of the range, errors need to be reported for python/plugins. This should be preferably be done into the context somehow, with some general error reporting mechanism, rather than using return values.

Notes

• Which kind of context and editable checks should be done as part of the RNA Api? For example editmesh vs non-editmesh, id.lib, proxified properties, etc, how that works together with the RNA system is not very clear yet.

Python and Plugins

Part of the python API may be automatically generated based on this data API, augmented with more functions and data types. A potential render plugin API would also be partially generated based on the data API, providing automatic access to a large part of the data they need. The data API only covers getting and setting properties, not creating new structs. Clearly this is functionality required by the python API as well as the user interface, though it seems difficult to make this uniform. The user interface will likely do such data manipulation with operators, while python will have it's own functions. It would be ideal if as much as possible of this code was common and accessible through a nice API, which may also be exposed to plugins in the future, but this seems like it is out of the scope of the data API.

Generating user defined properties would be done through the existing ID property system, and these ID properties would be available through a uniform interface together with native properties. There is a discrepancy between the defined ID property types and the types defined here, which needs to be resolved. ID properties do not have booleans and lack an ID pointer implementation too. ID properties have a double type which is not defined here. Arrays are also defined differently. However generating ID properties from an RNA definition should not give much trouble.

More complicated is the problem of how to associate RNA with user defined properties. One use case where you want plugins to define their own RNA would be a render plugin that augments materials with it's own properties. To interpret the properties in a list view user interface the RNA would need to be available, so perhaps the plugin should provide it when Blender starts up? Another use case would be an animator that makes user defined properties in a rig. In this case it seems the RNA would need to be stored in the .blend file?

Operators

Each operator will have it's own properties. It seems reasonable to use ID properties for this purpose, automatically generated from an RNA definition. An alternative would be to have operators define a DNA struct. ID properties may be the simplest choice.

Internal Code Usage

The RNA API as defined here is basically a nicer layer over the actual data structures used by Blender. The question then poses itself if this API should also be used internally. A mix of both direct access and these API functions would be confusing, but can also avoid duplicating code. I think that internally generally the rule would be to just use direct access to the DNA structs rather than the RNA API.

Naming Conventions

We need consistent names to be generated for all structs and properties, what should they look like? Proposal:

• Capitalized Words for UI names.
• Regular sentences for tooltips.
• CamelCase starting with capital letter for structs cnames.
• lower_case_with_underscores for property cnames.

IPO's

Nearly all properties should be animateable. IPO curves can correspond to a property + array index. It may be interesting for drivers to driver a quaternion x/y/z/w at once for example as well.

Properties should be set as animateable by default, with the exception of strings, pointers and collections. Making a property not animateable would require a flag to be set.

Each struct would have a collection of IPO's for it's properties stored in a list somewhere in the struct, with each member:

• An identifer for each property.
• A pointer to an IPO curve, driver, ..
• ..

Evaluation IPO's would then call set() on the properties that have an IPO attached to them. A question that arises is how this property identifier should look like. Currently this is based on fixed defines, and this could still work for native properties. But for user defined properties this does not work anymore. One way would be to just store the IPO curve pointer etc. along with each ID property.

For actions however the property identifier should be decoupled completely from the struct itself, as it can be reused by multiple datablocks. A solution then would be to make the property identifiers strings, which would work for both native and user defined properties. There is a bit of a performance hit then, compared to an array lookup with an integer for example, but it may be possible to optimize this to the point it doesn't matter anymore.

Issues

• How do IPO's and action refer to properties, i.e. what is the property identifier?
• For editing properties which have IPO's assigned, we need a way to transform them for example without flushing IPO's during dependency graph evaluation, before the key is inserted. So we will need to store temporary per property DONT_FLUSH flags to be respected by IPO evaluation.

Duplis and Proxies

I though of handling this unified with IPO's somehow since both are based on overriding certain properties. So duplis can be seen doing a shallow copy, overriding some properties, and proxies can be seen as overriding certain properties from a linked datablock.

Making any option proxyable or overrideable in a dupli may be out of the scope of 2.5 though, as I think it requires some deeper dependency graph related changes, as well as careful treatment for multithreading.

User Interface

Some information associated with properties is only designed to be used for generating user interfaces or documentation. That includes soft min/max, number sliding step, user inteface name, and a tooltip description.

A data view may then be generated that automatically gives an overview of all properties. A more user friendly interface may need to made manually, to give good button layouts, group, hiding things based on context, etc. However information like soft min/max can help reduce the amount of code here by using them automatically, unless there is an override to use something else.

User interface names and tooltips need to be translated to different languages, so it may be useful to store them separate from property definition. Additionaly doing this not only for other languages but also english, and making it possible to edit tooltips, it could become possible for users to help documenting Blender this way.

Notes

• It appears this data API could enable drag and drop with verification of data types etc quite well, is something missing for that use case? It seems that as long as you can query the type of a pointer property, that should enable dragging various datablocks into it.