Dev:Ref/Requests/Internal Refactor/Data System

Data System

Overview

This design of the data system has the goal of being flexible. The data blocks that the application modifies are composite elements built up from the different data types and defined by its SDNA (struct DNA).

Data Types

Data types are the base element of a data block. Built in data types include:

• octet (unsigned or signed)
• short (unsigned or signed)
• int (unsigned or signed)
• float
• double
• string
• vector[2,3,4][i,f,d]
• matrix[2,3,4][i,f,d]
• quaternion[f,d]
• function

Data types when defined to be in a block also include a cardinality. The cardinality defines how many of such elements are in the block. A cardinality of "+" means a variable length array with an initial size of 0. A cardinality of 3 means there are 3 elements in the array (cannot be grown later). A string is the exception to this rule, even though it normally has a cardinality of 1, in that it is technically a variable length array of characters that can grow as needed.

Data types can be expanded by the user through plug-in libraries. See the section on data type builders in the builder pattern below for an example of what needs to be implemented for user defined data types.

Data types are not saved in files, nor can they be directly accessed. To access a data type, you must use the data block. This is because all data in the system is saved (even internal system data) in data blocks. This is also because data types may be represented in memory differently than they would on disk. A std::string represents the data block in memory, but on disk it would be an unsigned int (for number of characters) and a variable length list of characters. Similar for arrays, but with std::vector.

The function data type is a script (internally a string) that is executed each time the data is accessed to generate new data (procedural or paged). This function may access other fields of this data block, but not outside the data block. This means for a paged_terrain_data_block that has a function field that produces the terrain in the camera view, it needs to a field for the camera that is initialized (or set by the user) to a valid camera of the application.

Data Blocks

Data blocks are built on data types. All data that an application may access is through a data block. These are saved and loaded from files. Data blocks are defined by their SDNA, which is developed through the GUI Data block builder, which is described below.

Get Function

The data block provides the interface to the contained data through a templated get function. The get function returns the address within the data block of the data requested. For example if the string name of the object is requested:

std::string& s=thisDataBlock->get("name");

will return the address of the string. This function uses an offset map to map the names (or hashed value of the name for quicker comparison) to the address of the data in the data buffer. This address is determined by the offset from the beginning of the data block of the particular field based on the SDNA. This offset map is generated as part of the data block definition.

The get function should be inlined where possible to assist in speeding up the data retrieval.

Data Manager

The data manager is the main entry point to the data system and is responsible for several functions:

• Registered data types and data builders
• Block definitions (SDNA, load/save/create/init/offset maps)
• a factory to create new blocks
• stores all create blocks in a database
• register blocks with the script environment
• Save and load data to persistent storage
• making sure dynamic content is up to date (via a separate thread)

Generating Data Blocks

To generate a data block the built in GUI editor can be used, however it is possible to write a data block definition with a text editor (in the case that the GUI editor is not written yet ;) ) This data description is then used by the data system to develop an SDNA, load script, save script, create script, init code, and offset map. This done through use of the data type builders.

Builder Pattern

The builder contains information specific to the data type that when combined with the other data types results in a complete data block. Each data type builder knows the size of its data type in bytes, contains any creation/initialization code snipets, save script snippets, load script snippet that are then compiled together in the block definition.

GUI Editor

The GUI editor allows for data types to be combined in a block, given a name and a cardinality. The GUI editor then can feed the data description in to the data system to build and load the data block or out to a text file.

Data Block Definition

A data block definition is generated based on the description (from a text file or from the GUI). This data block definition contains an SDNA, load/save script, create/initialize script, offset_map, version information and then is registered with the data system to be created (in a factory pattern). As new data blocks are added to the data system, if they have the same name, the version is incremented before they replace the new data block so that the SDNA system will be able to save/load the block properly with other versions.

To Do

Look at dynamic data blocks (procedural or paged data) and how that alerts the system that it needs updating.

Hcube's suggestion

It would be great if we make a datastream interface for raw data (file,stream,mem,etc) i/o like ogre3d does.
http://www.ogre3d.org/docs/api/html/classOgre_1_1DataStream.html
This will help to save data to mem, netstream or file.

Interfaces

 using namespace blenderKernel;
 using namespace std;
 
 /*
 	Every blender module's entity is represented as a DataBlock in a blend file.
 	Ex:
 	Texture has these attributes:
 		+ filename:string
 		+ width:int
 		+ etc
 
 	Material
 		+ specularColor: vec4f
  		+ textures: array of string
 
 	Mesh has these attributes:
 		+ vertexArray: array of vec3f
 		+ indexArray: array of vec3i
 		+ vertexColorArray: array of vec3f
 		+ etc
 
 	Every blend file must have a default datablock,
 	named 'Blend Information' with these attributes
 	+ blender version number what is produced the file
 	+ file format version
 */
 
 class DataBlock
 {
 public:
 	// can be useful, but i'm not sure if this is really necessary
 	// can be removed later
 	// maybe it can be the object name, but i'm not sure in it.
 	string getName();
 	
 	// can be anything, user can fill it
 	// blender should define some standard types: mesh, material, etc
 	string getType();
 
 	int getNumAttributes(void);
 	DataAttributeInterface *getAttributeByIndex( int index );
 	DataAttributeInterface *getAttribute( string name );
 	
 	void addAttribute( DataAttributeInterface *attribute );
 	void delAttribute( DataAttributeInterface *attribute );
 	
 protected:
 	// attributes
 	vector<DataAttributeInterface*>	mAttributes;
 };
 
 // interface
 class DataAttributeInterface
 {
 public:
 	virtual string& getName(void) = 0;
 
 	// allows more customizable usage
 	// useful ex: importing a mesh from another blend file
 	virtual bool isLoaded(void) = 0;
 	virtual void load(void) = 0;
 	virtual void unload(void) = 0;
 	
 	virtual <class data> data* getDataPtr(void) = 0;
 	
 	virtual AttributeType& getType() = 0;
 	
 	// returns the array size
 	virtual int	getSize(void) = 0; 
 	virtual void resize( int size ) = 0;
 };
 
 // attribute implementations
 
 // local data attribute
 class DataAttribute
 {
 public:
 };
 
 
 // network remote attribute
 class NetDataAttribute
 {
 public:
 };
 
 // the user can add custom DataAttributes,
 // with that custom engine data can be wrapped
 // ex. direct edit ogre3d engines mesh vertex buffer
 
 // procedural attribute, binary plugin or script generated
 class ProceduralDataAttribute
 {
 public:
 };
 
 // global, most important top class
 class DataManager
 {
 public:
 	void openFile( string filename, /*mode: read/write*/ );
 	void closeFile(void);
 	
 	void saveAllDataBlocks(void);
 	void saveDataBlock( DataBlock *dataBlock );
 	
 	// if we dont want all datas in mem, we can load only the necessary datablocks
 	void loadAllDataBlocks(void);
 	// useful ex: importing a mesh from another blend file
 	void loadDataBlock( DataBlock *dataBlock );
 	
 	int getNumDataBlocks(void);
 	DataBlock* getDataBlockByIndex( int index );
 	DataBlock* getDataBlock( string &name );
 };

Blend file format

Should be a tar/zip/gzip/7zip archive file. (only one will be implemented)
The archive (the blend file) is a packed directory.
The datablocks should save in a dataBlocks.bin file what is included the archive.
All packed data should also added to the archive, under a packedFiles subdirectory.
Using compression could be optional. (zip can pack without compression.)