Dev:Source/Development/Projects/Blender File Format/XML

BlenderXML

Introduction

This project intends to explore adding a human readable xml file format for blender much like maya has an ascii and binary format for the maya files. Blender could benefit from an ascii format for blender files that is human readable and enables blender scenes to be imported into other application easily.

The Goal

There are really two possible goals for having an XML format for blender.

• One is to have a human readable text file format that allows third party apps or people to edit the file outside of blender, or to convert to another format.
• The other is to export geometry and material information to a format which allows easier import into other apps. I have expanded my goals a bit. After a working export of the the blender file format has been finished, I plan to expand this project to offer collaborative tools for blender. Things like Collaborative modelling, Chat, File upload, Version tracking, and distributed rendering* are possible goals.
• Some of the other side benefits to an xml format are:
  • using xml includes to link data from other files to your file. This allows efficient creation of "libraries" for use in blender. models, materials, actions can all be saved individually and reused easier.
  • use available xml_RPC implementations to serve render data to distributed internet rendering farms. A group of people working over the internet on a project could conceivably set up an impromptu renderfarm using xml rpc to handle the communication.
  • set up a web server to handle blender data with version tracking and collaboration. It could handle large scenes easily with people checking out parts of the scene to work on it and even allowing people to sign up for the distributed rendering to render. Could make large complex scenes and animations possible for the Indie and hobbyist artist. This could all be done with just php or perl on the backend. I am considering expanding this to more than just an examination of using xml as a format for saving files. Perhaps by adding http and xml protocols I can work out a useable method for collaboration. Another project with similar goals is the DbBlender project.

Roadmap

• Decide on a format to use.
  • simplifies analysis of the structure of a blender file? [ y/n ]
  • makes import into other apps easier?
  • supports all data in binary format?
• Write the proposal for the specs of the file format.
• Code the parsing functions for the format. The ability to both read and write the format to file.
• design specs for http/xml communication.
• add code to communicate vie http with http backends.
  • also code a web backend to connect to.

Coding Proposal

• Writing the file:
  • The code will branch off from write_file_handle on line 1494 in writefile.c The code will follow a similar path as the binary file write with the difference that we will be creating tree structures to mimic the document tree structure of an xml file( see XMLStructDefinitions|).|I|am|writing|my|own|simplified|xml|parsing|library* for use in blender.
  • After the tree structure has been created we will start the process of creating a string to output into an xml file. The file will not contain line feeds or carriage returns to simplify. This will make the file difficult to read in a standard text editor but xml parsing and editing tools will have no problem viewing it.
  • After the file is written the tree structure will be deleted and the memory reclaimed.
• Reading the file: Reading the file will branch off from *todo find this function
  • Reading will involve parsing the file into a tree structure. Once the tree structure has been created we will use it to start creating the scene * todo I may decide to do this simultaneouse with reading the file if it doesn't create any problems
  • After the scene has been created the tree structure will be deleted and the memory reclaimed.
• Additional notes:
  • Reading and writing the file should be sufficiently divorced from the tree creation and parsing so that network protocols can be used in place of standard file writes and reads.

Links

• XMLSourceCode - 17 Feb 2005
• XMLStructDefinitions - 11 Feb 2005

General Resources

• X3D specs
• Expat SAX XML parser library
• http-tiny a small compact http library in C

FileFormat Reference on the blender file format

• Blender Data Architecture Documentation

-- DeveloperWikiJeremyWall - 11 Feb 2005

Comments

• I dont get it. Why are u reinventing the wheel and not using the Expat libary as noted in the Resources? Also take a look at:
  • [1]
  • [2]

Expat or libxml will probably be the best bet for speed and simplicity. Besides that, these libraries follow the standards as described on [3],|your|[4] for example does not even understand namespaces. The focus should be more on a DTD/Schema definition together with a SAX parser implementation IMHO. -- DeveloperWikiManolitoSnell - 15 Feb 2005

• I did consider using one of those libraries. Xerces was out because I don't like C++(please no flames. its just a personal preference) I'm familiar with expat but its not really useful for anything more than parsing the text stream. And I didn't want to have to learn a new library API (laziness) So I decided to go with a more integrated solution. As for DTD's/Schema's Blender's fileformat is in a rapid state of flux. Maintaining a DTD would get difficult fast. For now I just want to get a proof of concept xml support into blender. This was the quickest way to do that. Expat can easily be used later on without breaking the code when we need it's advanced features. I welcome discussion on this though so let me know what you think. -- JW

• Quote "One is to have a human readable text file format that allows third party apps or people to edit the file outside of blender, or to convert to another format." and quote "The file will not contain line feeds or carriage returns to simplify. This will make the file difficult to read in a standard text editor but xml parsing and editing tools will have no problem viewing it." These two seem a little contradictory - it's not very easily 'human readable' if it's all on one line. Is there such an advantage to doing it that way? -- DeveloperWikiMattEbb - 16 Feb 2005

• Whitespace takes up filespace. For potentially large scenes this could be a significant portion of the file size. By eliminating any excess whitespace we can reduce file size but still maintain readability in most XML readers. Firefox, for instance, will display the file in a readable format. if this proves to be unnecessary then it will be trivial to add linefeeds and indentation. -- JW

• Drop the tags, they take more space then whitespace. I would rather see a line-based ascii format with strict whitespace rules. One vertex per line, one face per line etc. So it can be meaningful to maintain blendfiles in bzr (bazaar), git or anything similar. Then we can start to talk about collaboration.

• Saving vertex coordinates in readable format could result in data loss for small models; You could add binary serialized data organized inside the XML, or even zlibbed text. Look at the inside of a photoshop file, for instance. it's XML with compressed binary data. It would be ridiculous to write every pixel into the file, even if it were in some sort of ascii format. Too much space. Just like in the case of our vertices. I also think that indentation and line breaks could be toggled by the user in some export option, but they could default to an ugly print, since XML is more used for program exchange than it is for human reading.