利用者:Gsrb3d/Accessing Files

The purpose of this page is to document the on going changes to accessing files methods from inside Blender, in a multi OS, multi user capable way. The most visible (side) effect would be a reorganization of where files go (user default scene, user scripts, shared plugins, etc) for users and some changes in code for programmers. And it will not only cover config files, but all files can (and should) be handled by it.

At first this can sound like adding complexity, but it is the other way around, it makes all files equal and simplifies code by centralizing it. No more "new file incomplete and previous version lost" because the quota was exceeded, no matter if it was a config file or a .blend, eg. It is the parallel of malloc() vs MEM_mallocN().

Another new thing is the use of Enviroment Variables internally to provide easy storage of some information, plus easy integration with other tools to cover the needs of advanced users, for example in renderfarms. Normal users will not notice anything different than before.

Classes of Files

Some basic types to handle:

Shared files

All files that are installed and that users will use, but must not written by them. Or in other words, the default data that is not the executable file. No coder should care if global install or "unpack and run", the point is they are defaults, shared by everyone for reading.

User files

User configuration files, overriding Shared files. The structure of directories mimics the Shared files.

Temporary files

Files that are needed for some time, but can be removed later. Typical place will be in local harddisk, not using disk quota, but not allowing others to overwrite. For example files used to communicate with external render engines.

Normal files

Other type of files not covered by above types, but which would benefit from the common file access API, specially for such features as backups or error checking.

Config File Layout

Shared and User files share internal layout by default, they differ in the leading path. Those base paths are determined in a per OS basis, via special calls GHOST_getSystemDir() and GHOST_getUserDir(). The reason to use GHOST is not arbitrary, it allows MacOSX to use Cocoa calls without problems, for example.

The getSystemDir will support "unpack and run" (where the binary is) as well as proper install style path. It can do checks to detect which is the case or be hardcoded at compile time. It can always be overriden with BLENDER_SHARE env var. The getUserDir will figure where the user config is to be stored (checking LDAP, env var HOME or whichever method applies in the OS).

Possible values for system dir:

 /some/where/blender-2.56-linux-glibc236-py31-i386/ # "unpack and run"
 /usr/share/blender/
 /Library/Application Support/Blender/
 # MSWindows example needed

Possible values for user dir:

 /home/USERNAME/.blender/
 /Users/USERNAME/Library/Application Support/Blender/
 c:\Documents and Settings\USERNAME\Application Data\Blender Fundation\Blender\Configuration\

The examples above are what the GHOST calls can return, not what they have to return. If the OS uses translated paths (c:\Dokumente und Einstellungen\...) or administrator made a different layout (alpha classification /home/a/aUSERNAME, /home/b/bUSERNAME ...) the values will vary. What matters is that the calls return the place where to start.

If in any OS it is desired, the user dir can reside in the same place than system dir or a child of it, for example for "unpack and run". In Unix it is not the norm, and in any OS user dir is only needed to write changed or new files at run time, so unifiying the places are only strongly recommended for border cases like "unpack and run from thumbdrive". Other cases should be fine using the real user directory, as otherwise it makes impossible to share that "unpack and run" install with other users (and uninstalling will lose the config, which maybe is not desired).

Versioning of Config

Hanging from the base path, we find (three digit) subdirectories to allow parallel installation of different Blender versions. The value is also stored in BLENDER_VERSION env var (set by program, not overrideable). Examples 255/ or 260/, so a complete path would be /home/USERNAME/.blender/255/.

Common Config Paths

Config files are grouped into 4 classes, BASE, DATAFILES, PYTHON and PLUGINS. This four classes are controlled by enviroment variables BLENDER_USER_* and BLENDER_SYSTEM_* (BLENDER_USER_BASE etc). The USER is a single path, and the SYSTEM is a list of paths where the system will look, in order, if there is no user version. Any reference to a plugin will only need to mention the relative path to the family it belongs too, and the system will take care of locating the right file.

The default values are loaded from environment file. This file is a special BASE file as it is needed to be loaded before all the variables are set, so we have to make the location of this file partially hardcoded, only affected by prelaunch enviroment variables. Example of defaults in Unix (except SYSTEM_PLUGINS that includes a local change to demostrate multiple paths):

 # This is a comment line and the next are valid Unix paths.
 # Notice the last envvar has two paths, separated by :
 #  and using a non Blender envvar
 BLENDER_USER_BASE=${HOME}/.blender/${BLENDER_VERSION}
 BLENDER_SYSTEM_BASE=${BLENDER_SHARE}/${BLENDER_VERSION}
 BLENDER_USER_DATAFILES=${HOME}/.blender/${BLENDER_VERSION}/datafiles
 BLENDER_SYSTEM_DATAFILES=${BLENDER_SHARE}/${BLENDER_VERSION}/datafiles
 BLENDER_USER_PY=${HOME}/.blender/${BLENDER_VERSION}/py
 BLENDER_SYSTEM_PY=${BLENDER_SHARE}/${BLENDER_VERSION}/py
 BLENDER_USER_PLUGINS=${HOME}/.blender/${BLENDER_VERSION}/plugins
 BLENDER_SYSTEM_PLUGINS=${BLENDER_SHARE}/${BLENDER_VERSION}/plugins:${MOVIE_PROJECT}/plugins

So continuing with the example, if trying to load a sequencer effect in Unix named stripes.so, the file to ask for is a PLUGINS type with path sequencer/stripes.so. Let's assume MOVIE_PROJECT has value /mnt/movie/ and Blender 2.55 with full install (thus SHARE as /usr/share/blender) would had loaded ~/.blender/250/environment at start up and then will try the following files in order, as instructed per it: ~/.blender/250/plugins/sequencer/stripes.so, /usr/share/blender/255/plugins/sequencer/stripes.so and lastly /mnt/movie/plugins/sequencer/stripes.so.

PY and PLUGINS are self descriptive, DATAFILES covers things like startup.blend, etc and BASE more general ones like enviroment or file-history files.

Points of View

Developers' PoV

The API matches typical C calls, just like Blender uses own memory allocation wrappers, we move to use file operation wrappers. They add some extra parameters to specify details, and otherwise just change slightly like making FILE* become BFILE*. In case of need, there are calls to fall back to plain C style (extract fd or FILE* from a BFILE*, eg). The two principal parameters to look for are int bflags and BEnvVarFam envvars, see BLI_bfile.h for the current implementation.

bflags is the basic param, a bitfield that decides what special handling to perform, like look up temporary file if called with BFILE_TEMP or compress with BFILE_GZIP, or give only the shared version of a config file with BFILE_CONFIG|BFILE_SYSONLY. This determines not only what special operations to perform in the data (like compression) but if any operation is needed for the provided path (like look up in user config then shared config).

In the case of config files, then envvars param should specify which config subtype. Otherwise just set to none.

Example, assume we are working with /somewhere/foo.blend and we want to write 2 bytes to //bar file, before it would be something like (more or less copied from real code, comments added about what happens and could be improved):

 strcpy(path, "//bar");
 BLI_path_abs(path, G.sce);
 FILE* file = fopen(path, "wb"); /* File data is already removed, no way back */
 /* Crash here means no new or old data avaliable */
 if (file)
   fwrite(&buf, 1, 2, file); /* Maybe write failed, but ignored */
 /* Crash here could mean loss, data not fsynced */
 fclose(file);

and it becomes (comments explain what the lib gives us automatically):

 BFILE* file = BLI_bfile_fopen("//bar", "wb", BFILE_NORMAL, BENV_NONE);
 /* Next operations are performed over safe path, old data still avaliable */
 if (file)
   BLI_bfile_fwrite(&buf, 1, 2, file); /* Error ignored, but not as catastrophic as before */
 BLI_bfile_close(file); /* If there were errors, the data is not commited, old is kept. And it is really synced to disk to be sure. */

Basically, the new API can handle Blender paths like // or "whereever the user config files for script are", but also provides extra services like backup before overwriting current data.

It can allow standarizing in / as directory separator and then make the API handle the OS details in one single location.

Users' PoV

New users should not need to touch anything, but enjoy proper separation of own vs shared files, and above all, enjoy that the file paths follow the expected style in the machine they are using.

Advanced users will be able to adjust paths via EnvironmentVariables, via the configuration file in their own config files location. They will also be able to run external process that know which Blender launched them and where to look up for things, or even adjust Blender configuration via variables. In conclusion, integrate better with automated systems like renderfarms or allows quick configuration like testings a new set of scripts.

Administrators' PoV

Each user has own data, central location for shared data, no writing to user config until needed... all should be covered. I can think one case in which Unix ones will have a small issue, plugins inside /usr/share/blender is not a good thing (different processor types), but as the only single item that is forced is the existance of environment text file, they can put something like BLENDER_SYSTEM_PLUGINS=/usr/lib/blender/ there and solve the problem.