Dev:Source/Architecture/Path Rewrite

Goals for the Rewrite

• make file handling code more robust and easier to understand
• provide a nice OS abstraction layer for file handling code
• clean up source file structure by putting filename handling code into its own header and source files.
• .blend files that contain (relative) filenames to referenced files should work reliably on all platforms. If you saved your file on Windows and your friend uses linux he should be able to use your file using the same relative directory layout.
• API functions should be thread-safe.

Scope of the Rewrite

Analysis of Blender file code shows that file handling code can be separated into three areas:

OS file function abstraction

• BLI_exists
• BLI_copy_fileops
• BLI_rename
• BLI_gzip
• BLI_delete
• BLI_move
• BLI_touch
• BLI_gethome
• BLI_where_am_i
• BLI_filesize
• BLI_diskfree
• BLI_getwdN
• BLI_exist

These functions looked ok, I just propose to move declarations into own header file (BLI_fileops) In a second step these could be made to use the new proposed filename/path struct

Directory operations

• int BLI_compare(struct direntry *entry1, struct direntry *entry2)
• void BLI_builddir(char *dirname, char *relname);
• void BLI_hide_dot_files(int set);
• unsigned int BLI_getdir(char *dirname, struct direntry **filelist);
• void sort_filelist(SpaceFile *sfile);
• test_flags_file(); /* tests all flags for the filelist */
• void library_to_filelist(SpaceFile *sfile)
• void main_to_filelist(SpaceFile *sfile);
• void read_dir(SpaceFile *sfile);

used by filesel and imasel space Suggestion: replace with a general struct FileList and a better API.

Filename operations

• char *BLI_last_slash(char *string);
• checkdir();
• void BLI_make_file_string(const char *relabase, char *string, const char dir, const char *file);#* void BLI_make_exist(char *dir);
• void BLI_split_dirfile(char *string, char *dir, char *file);
• int BLI_testextensie(char *str, char *ext);
• int BLI_convertstringcode(char *path, char *basepath, int framenum);
• void BLI_makestringcode(const char *relfile, char *file);
• void BLI_char_switch(char *string, char from, char to);
• void BLI_clean(char *path);
• void parent(SpaceFile *sfile);
• BLI_make_exist(sfile->dir);

see proposal for rewrite

Rewrite Proposal

Solutions to reach the goals

• robustness:
  • Consisten use of return values for error checking
  • Write unit tests (test functions) to ensure changes in the future don't break expected functionality
  • Hide internal data from unwanted access by other parts of Blender
• understandability and maitainability:
  • choosing appropriate names for functions and parameters.
  • provide a complete, yet minimal set of functions to manipulate path names
  • write functions that do only one job, but do that job completely
• portability
  • Encapsulate system functions and provide OS abstraction layer
  • Store filenames in a .blend file in the most portable format possible.
• Thread-safety:
  • avoid globals and static variables in functions
  • avoid non thread-safe functions from the C runtime library (strtok)

Use Cases for filenames and path names

Examples

• directory of current .blend file
• filename of current .blend file
• default path for scripts, render output, yafray export, textures,...
• filename of used images in the .blend file
• filename for linked library objects
• filename for loading/saving the .blend file
• filenames for plugins for texture or sequencer

Main Use Cases

1. store filename of current .blend file
2. store filename of referenced files (images, scripts etc.)
3. store directory for output (yafray export, tmp, ...)
4. store directory for input (default texture, scripts, plugin dir ...)
5. retrieve system dependent paths (home, root etc.)
6. filename for loading and saving files (also export and import)

Sources for filenames

1. User input
2. system functions (getcwd)
3. other linked .blend files

Naming conventions:

dir is the directory part of the path
path is the complete path, which means either the absolute path or the complete relative path.
filename is the filename plus extension, but without a directory or folder

Coding conventions

• all filename handling functions are located in the files BLI_path.h and BLI_path.c
• all filename handling functions start with BLI_path_
• no code outside the BLI_path_* functions shall handle directly with the internal members of the path struct
• The path that the filename API functions act upon is passed as a first parameter to those functions.
• The use of functions that manipulate filename strings directly should be eliminated from the rest of Blender code.
• Helper functions, such as searching for the last separator or splitting a string at a separator etc. should be declared static in the BLI_path.c file.

Pathname API (preliminary)

struct Path {
       char volume[MAX_VOLUME_LEN]; /* For platforms using a volume name like Windows (C:\xxx) or AmigaOS (Data:xxx) */
       char dir[MAX_PATH_LEN];
       char name[MAX_FILE_LEN];
       char is_relative;
};

/* allocate new path and init path */
Path* BLI_path_new();

/* free path struct */
void BLI_path_free(Path **path);

/* get the directory part of the path in system preferred way.
    Path - Path from which dir is retrieved
    dirname - pointer to valid mem of size len
    len - length of allocated buffer for dir
          if filename is longer than len the function fails with return 0.
   Returned value: success = 1 , failure == 0 */

int BLI_path_get_dir(const Path* path, char* dirname, size_t len);

/* get the directory part of the path in system preferred way.
   A relative dir is expanded with respect to the relbase path.
   If relbase path is not absolute, the current absolute working directory
   will be prepended before.

   Returned value: see BLI_path_get_dir.

   For platforms supporting a volume name:
   The first volume name found will be used as root to get a valid path.
   Volume name is searched in this order: path, relbase, current working 
   directory */
int BLI_path_get_absolute_dir(const Path* path, const Path* relbase, char* dirname, size_t len);

/* get the filename part. */
int BLI_path_get_filename(const Path* path, char* filename, size_t len);

/* get the filename extension part */
int BLI_path_get_extension(const Path* path, char* extname, size_t len);

/* get the volume part. */
int BLI_path_get_volume(const Path* path, char* volumename, size_t len);

/* get the complete path, can still be relative, for displaying,
   always in system preferred way. */
int BLI_path_get_fullpath(const Path* path, char* pathname, size_t len);

/* get non-relative path for accessing the file or the directory,
   always in system preferred way.
   Works as BLI_path_get_absolute_dir() but with the filename too. */
int BLI_path_get_absolute_path(const Path *path, const Path* relbase, char* pathname, size_t len);

/* makes the path relative to the path relbase.
   Returned value is path or NULL in case of errors */
int BLI_path_make_relative(Path* path, const Path* relbase);

/* parse a path from a string - this is where the pathstr is converted to <br>
   POSIX and stored in the path.
   Returned value is path or NULL in case of errors */
int BLI_path_parse(const char* pathstr, Path* path);

/* return system dependent file separator */
char BLI_get_filesep();

/* fill path with a blender internal form of the root name of current .blend file :
   '/' on POSIX platforms
   'X:\' for Win where X is the drive
   'Name:' for AmigaOS or compatibles where Name is the volume name
   Returned value: 1 success, 0 failure */
int BLI_get_root(Path* path);

/* Does the system have long filenames ? */
int BLI_filenames_long();

Directory API (to be done)

struct FileList {
    struct Path* path;
    struct direntry* entries;
    unsigned int numentries;
}

example functions:
int BLI_flist_readdir(FileList* flist, const Path* path);
unsigned int BLI_flist_numentries(FileList* flist);

The rest needs to be defined yet.

• Remove the global files variable.
• fileselect.c has to be adapted to use current structure
• remove the FSmenu functions from fileselect into own source files fsmenu.c

and BIF_fsmenu.h (maybe move to blenlib too)

Open Points

• Transition from the current SDNA structs where the path is stored as a char array
• Keeping backward and possibly forward compatibility ?

---

Suggestions and Reviews:

--Review from Yomgui

- See Replies below I've corrected the API to be more independend (like a volume name w/o a size limit).

- To be compliant with rules, I think that the Path structure should not be exported in a public header. Keep it in an internal header or in BLI_path.c.
Then, in the public BLI_path.h, define Path like this :

typedef void *Path;

With this, use Path and not Path * in the API.

- Accepted It's better to for each function a result, to handle possible errors.

- I don't see the purpose of BLI_filenames_long(), because users can do some assumptions on that, or the purpose of our API is to free the user of any assumptions on path model.

- Accepted I've added a goal : thread-safe, very important!

- An other solution for our problem is to use POSIX paths internally (i.e. in PATH structure), but using native paths (i.e. system prefered paths), because all IO functions uses native paths.
And there are many of such functions in Blender code (for IO and for ouput like Python or yafray). Converting from POSIX to native before each functions is not a necessary overhead.
My idea is to convert path from native to POSIX only when you save in a cross-platform file like a .blend file. Then convert from POSIX to native when you load the file. These operations don't occure so often. The overhead is reduced at the strictly minimum. I prefer this solution ;-)

For this solution we need to add some functions to parse each directory that build the whole path, so we don't have to handle any separators things.

---

Quick Reply:

The struct Path will be written to the .blend file. For this there are some requirements, like byte alignment and it can't have dynamically allocated members (char* for volume).

Since .blend file contains essentially a memory dump of the structure the internal representation is what is written to the file. This is why I favor having the internal representation in POSIX, which is almost what the code is currently doing too. In most places where filenames are used you will find some calling of BLI_makestringcode or BLI_convertstringcode which messes with the conversion to native format as well as retrieving an absolute path.

I haven't decided on the final place where the struct path will be declared - a forward declaration in the header file would be nice, but then it must also be made known to the SDNA stuff if used within other structs (like filesel space)

--Elubie 00:55, 24 February 2006 (CET)

Eeek! Also saw that you changed the API to return char* ! Are you going to return the internal member of the struct!! If not, you'd have to allocate a new char *, but then the user of the function would also be responsible to call free without having allocated the memory himself. I changed that back to allowing a return value indicating success/failure.

---

Quick Reply:

- No at all, the char * pointer is the same than the input. When you try to call a "get" function you pass as argument a string where path will be written, isn't? By experience, it would be great to return this pointer. By this way you can re-use directly the string and checking for error is permited (NULL value). There are no problems of allocation something, I don't return any private pointers.

- I've a problem with static arrays in Path: to keep the cross-platform compatibility you need have the same values on each platform, isn't? So, what's happening when you save a filename with more than 8+1 characters on a old Windows platform? How to choose the good values?

- About SDNA: a good solution is to see SDNA module as a server and our path module as a client of SDNA. We register on SDNA, then when SDNA saves data, we're called by a callback (for example) to pack ourself and return a packed chunck ready to be saved by SDNA. So, it's not really a problem to hide ours structures, many solutions exists ;-).

- Why BLI_path_free() use a double pointer (**)?

- New brainstorm: the problem of byte alignement for Path in SDNA is a false problem ;-) SDNA can pad automaticaly strings. What is the needed alignement?

--Yomgui 10:41, 24 February 2006 (GMT+1)

---

Answers:

- char* as return value: ok, I think I still like a return value int better yet, because I think other coders might be confused a little bit too whether a new pointer is returned.

- static arrays: As far as I've understood the SDNA struct, it is one of the requirements that the size of the struct that is written to a blend file is known at compile time. I possibly want to use our struct as part of other SDNA structs, so we can't use dynamically allocated char*

- Because I plan to write the new Path struct to the .blend file, I also prefer that the path names there are POSIX conforming, so we always know that the filenames we read from the .blend file are 'nice'.

- System dependency is handled then when getting a filename string out of the struct Path or when converting a user input string or a string returned from a system function into a Path struct.

--Elubie 18:34, 9 March 2006 (CET)