Constant (including Bitfield Access) Through the BPython API

Purpose

Recent discussion on refactoring the API decided to explore using PyTypes to represent constants in the BPython API. The primary advantage of doing this is to avoid "mixing and matching" the values of constants, incorrectly applying them to BPy object attributes other than the intended ones.

Proposed Solution

A new constant PyType will be implemented which encapsulates information about specific constant types. As opposed to creating a class of constants and then subclasses of speficic constants, I propose a simpler solution which stores in each constant object information about the specific Blender constants it represents.

Defining Constant Catagories

Most constants currently provided by the BPython API represent a collection of possible values which a particular attribute can take. For example, the BPy Object type has a constant dictionary DrawTypes which contains the constants BOUNDBOX, SHADED, SOLID and WIRE. These represent the legitimate values for the Object.dataType attribute. The solution proposed here groups these values into a single constant category. Also, since attributes representing Blender data are generally strongly typed, all the constants in a constant category are of the same type.

A data structure is created in C which represents the information about a particular constant category. The information we need is

the name of the constant category (useful mainly for the PyType's __repr__ method, but also for easily creating the underlying constant category itself)
the type of the constants in the category
the name and value of each constant

A data structure is create in C which represents the information about a particular constant category. The information we need is

the name of the constant category (useful mainly for the PyType's __repr__ method, but also for easily creating the underlying constant category itself)
the type of the constants in the category
the name and value of each constant

This information is stored in three C data types

/* the actual value types for a constant */
typedef union {
    int i;                     /* integer constants */
    float f;                   /* floating constants */
    long b;                    /* bitfield constants */
} constValue;

/* the name and value for a constant */
typedef struct {
    char *repr;                /* constant's name */
    constValue value;          /* constant's value */
} constIdents;

/* the constant category */
typedef struct {
    unsigned char utype;       /* type indicator for all constants in this category */
    char *name;                /* constant class name */
    unsigned char tot_members; /* total number of constants */
    constIdents *members;      /* array of constant names/values */
} constDefinition;

So, for the Object.DrawTypes constant class, we would have this declaration:

#define CONSTANT_TYPE_INT 0

static constIdents drawTypesIdents[] = {
    {"BOUNDBOX",    {(int)OB_BOUNDBOX}},
    {"WIRE",        {(int)OB_WIRE}},
    {"SOLID",       {(int)OB_SOLID}},
    {"SHADED",      {(int)OB_SHADED}},
};

static constDefinition drawTypes = {
    CONSTANT_TYPE_INT, "DrawTypes", sizeof(drawTypesIdents)/sizeof(constIdents), drawTypesIdents
};

BPy_const type

With the constant category defined, we can define the BPy constant object:

typedef struct {
    PyObject_HEAD
    constDefinition *defn;     /* specific constant information */
    unsigned char utype;       /* type indicator (basically the same as defn->utype) */
    constValue value;          /* specific value of this constant */
} BPy_const;

The defn is used for a variety of purposes:

an attribute setter can use it to verify that the correct type of constant has been passed as an argument
the type's __repr__ method can use it to print the constant category name
for bitfields (future implementation), the __repr__ method can show all bits which are set

Adding Constants to Current Types/Classes

The current implementation uses three steps to create a new constant dictionary:

create a new constant dict with PyConstant_New()
populate the dict using PyConstant_Insert()
add the dict to the module using PyModule_AddObject()

Since the proposed implementation does not create a Python dictionary, it is much simpler to create a constant category object:

create a new constant category object with PyObject_New(), and identify the instance as a constant category for the specific constant type (the difference between the constant category "superset" and individual constants is discussed in the "Differences" section below)
add the dict to the module using PyModule_AddObject()

A simple helper procedure is provided in the new code (Const.c) to do this:

void PyConst_AddObjectToModule( PyObject *module, constDefinition *defn )
{
    BPy_const *constant = new_const( defn, &ConstCategory_Type );

    /* perform type initialization if it hasn't been done already */
    if( Const_Type.tp_dealloc == NULL )
        PyType_Ready( &Const_Type );

    if( constant != NULL ) {
        constant->utype = defn->utype;
        PyModule_AddObject( module, defn->name, (PyObject *)constant );
    }
}

new_const() is a local helper function which created the actual BPy_Const object.

Differences Between the Current Constant module and This Implementation

Constant values

The current BPy Constant type (constant.c) is essentially a dictionary of name/value pairs. Accessing a "member" of the constant category looks up the member's name in the dictionary and returns its value. Once this value is returned, there is no way to map back to the original constant (i.e., no way to tell where the value originated). So, for example, if {{{1}}}, then the statements below are identical:

ob.drawType = Object.DrawTypes.BOUNDBOX
ob.drawType = 1

The new BPy_const type is simialr to a wrapper around a constDefinition structure. Accessing a "member" of this constant category searches the structure for a matching name and returns a new Bpy_const object which contains the value for the member. This new object still contains the "link" to the corresponding constDefinition structure, which allows us to identify the specific type of the constant.

Python data types

Object instances from the current BPy Constant type are dictionary-like containers; the contents can be either Python ints, floats, strings, etc. The result is that accessing a "member" of the constant category results in an object instance of a basic Python type (i.e., an integer)

Objects instances from the new BPy_const type store an internal flags self-utype which identifies the data type of their constants. Additionally a flag is used to indicate whether the instance is a constant category (the "superset") or an individual constant. For example, Object.DataTypes would be an constant category instance. As such, when a reference is made to its attributes (such as Object.DataTypes.BOUNDBOX) the getAttr method for the type is allowed to search self->defn to see if a constant named "BOUNDBOX" exists. If it does, then a new Bpy_const instance is created which is identical to self except that (a) the self->value.i field is initialized to 1 and (b) the self->utype field has its "superset" flag cleared. The result is a "basic constant". A subsequent attempt to reference any attributes within this "basic constant" will result in an ATTRIBUTE_ERROR exception.

Note: this implementation probably can be "split" into two types: BPy_constCategory (the "superset") and BPy_const (the individual constants).

Use of Constants in Attribute Code

Most BPy types provide a macro to check an object's type. For the proposed constant module, this macro is extended to also check the constant category type:

#define BPy_ConstType_Check(v,d) ( BPy_Const_Check(v) && (v)->defn==d )

Within an type's attribute setter, the type of the constant (int, float, etc) is known so the correct field withing the constValue union can be accessed without needing to check self->utype.

Another benefit of this extra complexity is that, since a user cannot change the value of a constant, we rarely need to perform input checking, eliminating few extra comparisons. Additionally, we do not need to call functions such as PyInt_AsLong() since the data is already stored in the BPy_const object.

Performance

The Object_setDrawType setter for the BPy_Object type was modified to also use this new method in addition to the current constant/integer input types:

static int Object_setDrawType( BPy_Object * self, PyObject * value )
{
    BPy_const *c= (BPy_const *)value;
    static char warning = 1;

    /* since we mess with object, we need to flag for DAG recalc */
    self->object->recalc |= OB_RECALC_OB;

    if( BPy_ConstType_Check(c,&drawTypes) ) {
        self->object->dt = c->value.i;
        return 0;
    }
    if( warning ) {
        printf( "use of integers to set Object.drawType deprecated!\n
                 \tuse constants from Object.DrawTypes instead\n ");
        --warning;
    }

    return EXPP_setIValueRange( value, &self->object->dt,
            OB_BOUNDBOX, OB_TEXTURE, 'b' );
}
</<source lang="python">> 

A simple script was used to measure the elapsed 

<source lang="python">
import bpy
from Blender import *
from datetime import datetime

o = bpy.objects['Cube']

start=datetime.now()
cnst = Object.DrawTypes.BOUNDBOX
for x in xrange(5000000):
#    o.drawType = 1                           # test 1
#    o.drawType = cnst                        # test 2
#    o.drawType = Object.DrawTypes.BOUNDBOX   # test 3
#    o.drawType = o.drawType                  # test 4
#    o.drawType                               # test 5
finish = datetime.now()

print 'time = ',finish-start

The first test simply passes the integer constant of BOUNDBOX to the setter; the second passes a stored copy of the constand, and the third perform a look-up of the constant on each iteration. The fourth test assigns from a copy of itself (which calls the attribute getter), and the fifth simply gets the attribute.

current implementation
- test 1: 18.2 seconds
- test 2: 17.2 seconds
- test 3: 38.5 seconds
- test 4: 25.2 seconds
- test 5: 7.9 seconds
proposed implementation
- test 1: 18.1 seconds (no significant difference)
- test 2: 10.0 seconds (%70 faster)
- test 3: 24.7 seconds (%50 faster)
- test 4: 20.8 seconds (%20 faster)
- test 5: 12.8 seconds (%40 slower)

(The overhead for the loop itself was measured to be 5 seconds, and is removed from the measurements above.)

The first test shows no significant performance difference. The remaining tests show significant performance differences. In test 2, the overhead of extracting the integer type from the parameter and checking it are missing, and in test 3 the additional overhead to find the constant's value in the dictionary are also removed. Test 4 is similar in test 2 (we set the attribute from a variable), except the variable is created on each iteration. Test 5 is surprisingly faster in the current implementation; this involves a call to PyInt_FromLong() to create the attribute as opposed to a call to PyObject_NEW() in the new Bpy_const code.

Test 3 is may be somewhat misleading since the performance of a dictionary look-up versus a sequential search of an array will vary based on the array size and search technique. The proposed implementation can be modified to be more efficient, as necessary.

Use of Strings in Place of Constants

Some users expressed the desire (and presented arguments) for strings in place of constants types. The proposed implementation makes this quite easy (for attribute setters). If the argument to the setter is a PyString, the C string can be extracted and the constant category members searched for a matching name. If a match is found, the value corresponding to that name is returns; otherwise an AttributeError exception is thrown. Execution timing tests for these methods were comparable to the times for test 2 above.

Bitfield discussion

The general opinion is that manipulating Blender bitfields from Python is evil. One camp says that the internal implementation should be entirely hidden from BPython, and bits within bitfields should be exposed individually as boolean attributes with True/False values. Another camp says that accessing individual bits is slower and not as easy to write, so some form of bitfield access should still be available.

The major complaints about bitfields are:

the syntax is confusing to users unfamiliar with ORing and ANDing integers
checking for valid input is difficult
determining which flag is in which bitfield is tedious
the existing __repr__ output for bitfields is an integer, which is not meaningful

Proposed Constant Extensions to Handle Bitfields

My proposal, based on prior IRC discussions with a number of developers and users, is to add a constant type for bitfields which contains some features of Python sets. This constant type would implement the numeric protocols used in boolean algebra, possibly with some operator overloading: for example, AND would be implemented by the & and * operators. The subtraction operator (-) would be used to clear one or more bits, which it typically done now by ANDing with the one's complement of some bit or bits (which would be costly speed-wise to implement, since we technically would have to compute the complement of the actual bitfield).

The main difference between bitfield constants and integer constants is the support of numeric protocols. The implication of this is that a user can create new instances of BPy_const objects which are not specified explicitly in the category's constIdents list. For example, Objects.DrawModes is a constant dictionary which contains bitfield values for ob->dtx. The code below would create a new bitfield constant instance which is the combination of two individual flags:

modes = Object.DrawModes
flags = modes.AXIS + modes.WIRE
print flags

results in:

[Const (AXIS,WIRE), "DrawModes"]

Bitfield constant categories would also include two "default" constants: NULL and ALL. These would facilitate creation of bitfields with all bits cleared and all bits set .

modes = Object.DrawModes
flags = modes.ALL
print flags
print flags - modes.WIRE