Dev:Source/Extensions/Assets/Repository Client Framework

Blender Repositories Client Framework

Project page and source code

The projects.blender.org page is : https://projects.blender.org/projects/repoclientfmw/ .

to get the source code do this :

svn checkout --username yourDevelopperName https://svn.blender.org/svnroot/repoclientfmw

the developperName is your name (you may have to sign up)on the projects.blender.org portal. committing, updating etc... work as usual for svn.

for anonymous svn access do :

svn checkout https://svn.blender.org/svnroot/repoclientfmw

About

The Blender Repositories Client Framework project will hopefully let anyone browse repositories from inside Blender3d, as well as download and upload content from/to those repositories.

A repository is a collection of .blend files, residing on

1. the local PC,
2. a storage device (flash drive) connected to the PC,
3. a PC on a local area network,
4. a PC continuously connected to the internet
5. a PC sporadically connected to the internet.

As re-usable content (mesh, material, armatures, games, actions) is developed and saved in a repository, the user will direct the client to register with the repository (access) and catalog what is in there. The user mirrors a local copy of some or all of the content by either a) cataloging meta-data about it for future use, or b) downloading it to their local PC. They then can use Blender's Linking and Appending function to bring that re-useable component into their working file.

This project's goal will develop a framework for client engines to access several online repositories rather than a single client for one sort of repository. All content will be assumed to be Open content distributed under the CC license. Therefore, anything mirrored must have attribution information.

It is first developed for Blender shall be written modularly enough and with a clean library to be reusable for other community software projects.

This page is here to help the contributors study and develop the project in a centralized fashion. You can check out the related BlenderArtists thread here.

Project structure

From the server to the client :

• online repositories(*)
• client wrappers/plugins to interface the servers the right way
• the framework's plug interface
• the framework's core (with a SQLLite DB to store things) + the client Blender scene & filesystem
• the framework's GUI

(*)(not dealt with in this project, we're just developping a client framework).

I have to integrate the design ideas written here to this wiki, and we'll adapt things in this wiki. This would also be nice to have a diagram here(jonathan)

Use Cases/Requirements

Please see this document for the logical functional requirements for this system.

Current ToDo list

This is a todo list for work that can be done right now, which for now is only research on specifications... We're not coding anything yet ourselves.

The research effort below can be done by several people simultaneously and progressively, so do feel free to edit this page and write interesting things you've found, inline with the questions/points.

Also, it would be nice that contributors put link to adresses of websites they've used to work during their research.

Things that do not need coding

• UI design research

- Could you show how other apps have done their UI for their update/submit interface. There are some unannotated screenshots I've taken here though I agree it's difficult to view them side by side(jonathan).

- Can you sketch some UI that would be cool ?

- Do you know pieces of code to help generating previews, viewing contents of a file in Blender UI ?

When this research part is done, and every one in the team likes the design, coding the UI could be started. The UI functionnality lays over the client core, so ask for the core API.

• existing repositories locations, provided contents and ways of access

- What are the existing online repositories which provide blender-compatible content ?

Could you give the following details for each (if you don't know, you could ask their maintainers) ?

- what kind of contents they provide ?

- how could we access their contents details & files in an easy/automated way ? if the site exposes an API to be fetched in another manner than just HTML browsing, that is : SOAP, REST or if there's a clean directory scheme that could written down...

Blender Model Repositories

Kator Legaz 3d models repository

http://www.katorlegaz.com/3d_models/index.php

• no known API
• folder scheme:

h_ttp://www.katorlegaz.com/3d_models/ + arcade_controls(the category name) + / + 0220 (id) + /index.php

content associated details (example page here)

• title
• 3d format used (only Blender it seems)
• preview image
• license

everything seems to be under the CC license. The link to the file (.zip) has the text "I agree to the licensing conditions, and want to download the model.". It would be nice to have the user to confirm this in the Blender GUI.

Blender Material Repositories

http://www.blender-materials.org/

existing Blender py client with Gui at : http://www.blender-materials.org/index.php?p=22

this script uses the php GET interface to access the website contents

Blender Python Scripts repositories

Blender Python Consortium (BPyC) repository

• project page: http://www.hhofministries.org/bpyc/
• repository page:http://www.hhofministries.org/bpyc/scripts.php (html)
• API:

• PHP calls API : Read List then Get by Id two commands can be passed to http://www.hhofministries.org/bpyc/interface.php? :

• 1. action=getlist with no more params, => returns a one-"scriptlist"-tag xml file containing "script" elements (see "script" elem sample below).
• 2. action=getscript with param &script=4 (the script id)

• a dynamic html page lists links to external scripts (file or packages). This is not accessible from the php calls API.

• provided file formats: .py only for BlenderPython (no GameEngine) for now apparently. *packages quick spec:

• Any downloaded content is packed in a [versionInteger].tar.gz archive containing: a .bpyc_xml folder containing a [versionInteger].xml and on the same root level as the folder the .py file.
• details provided in xml: (showing a sample here)

<scriptlist>
<script scriptid="4">
<scriptname>Script Installer</scriptname>
<version id="9" date="1164528000" number="2">
<versionname>0.3</versionname>
<file>system_install.py</file>
<blendversion>242</blendversion>
</version>
</script>
</scriptlist>

• clients:

• BpyMan v0.4

• authors: reD_Fox and SamAdam.
• depends: Blender v2.42+. Full python install. Uses full python install: urllib, os (and Blender.sys), xml.dom.minidom, tarfile.
• Download. No upload.
• Notes: server name is hardcoded in 2 place but can be easily made dynamic. Has an install function which is much less elaborate than what the install script provides. Directories .bpyc_download(1) and .bpyc_xml(2) are created into Blender's "scriptsdir" folder by BPyMan(1&2) and Script Installer scripts(2).

• useful: Script Installer(/Desinstaller) or on the blender wiki, by reD_Fox. Install scripts nicely with version control from any .py, .tar.gz or other archive format into Blender and proposes to choose the menu to them to. Not an online fetcher. BlenderArtists tool design talk(last post Nov 2006).

Non Blender 3d repositories

Google 3d Warehouse

The Google 3d Warehouse home page.

There's no API to fetch the site so a wrapper would take a bit harder to write.

No need to register to download data.

Available formats : collada (.zip) sketch up 4 and 5 (.skp) (sometimes all of them are provided, sometimes only one of them).

Features:

• always a preview image, title and author name(+link to his/her warehouse page)
• tagging support
• links to related items (more by author, things "you might like", collections containing this item)
• description.
• models can be aggregated in one or more collections (and one can see what collection a model belongs to)
• rating support.

Vector graphics repositories

Open Clip Art repository

• online API: works without login. get list by RSS, get direct link, preview&final are same
• provided file formats: svg,png,odg(?)
• provided file attributes: (todo) (??=>) filename,preview, author...

Clipartbrowser

Is obsolete and doesn't work anymore with the current online API

• project page:google code home page
• standalone: python & gtk & external xml and sql libs
• last changes date back to Dec 2006.

Inkscape

• project page:
• related blog:
• API:

works without login get list by RSS, get direct link, preview&final are same

• 1.query h_ttp://openclipart.org/media/feed/rss/[theSearchTerm] (can be fetched in a normal browser) which returns an RSS feed (empty if nothing found for search term). The rss fields contains "item"s, each item has direct links for preview (png) and download (png,svg...)
• 2.download from direct link of the full-size png/svg to show a preview in the dialog
• 3.the same file that was used for preview is not downloaded again and imported directly into inkscape.

• description: small dialog to search showing matching graphic files in a repo
• prog. language: ?
• functionnality: possible repo adress config (in the global prefs), search only.
• supported formats:SVG ? (todo)
• how to access it: builtin, go in File > Open Clip Art Library
• version:from 0.46+
• author:

Scribus OCAL helper

a 2006 SoC

• version: don't know if it's ever been integrated.

• author: greg.steffensen at gmail.com (wrote to him for more info (jonathan))
• project page: @berlios. not maintained anymore
• related blog: the person's SoC blog. No more postings.

• defining a standard protocol/folder structure(?) for communicating with repositories

Motivation: one current problem for Blender repositories is that there's no standard way to speak in an automated way. So wrappers have to be written for every single repository to be able to browse, fetch and upload.

If we specify some standard way to communicate/browse through the repositories, then all the new repositories could be designed to support that standard and our generic client will work right away with those. Older repositories can spare having people to write wrappers for the client framework to access them if they start conforming that standard as well.

Could you make specifications for either or both of the following ?

- define a server exposed call API + a corresponding protocol describing how the client&server should talk to each other.

- or have a description file at the server root explaining in a coded way how contents are arranged on the server.

• kinds of (blender)-compatible contents, attributes/meta-data to describe them

- We want to list the attributes that could be matched with any blender-compatible content:

1. What are the various blender-compatible components available? (ie materials, textures... Python scripts... ?)

2. What's relevant for users to know about them ? (ex : title, description, size, preview pic (one or more ?), version compatibility, author... ?)

Important: the contents specification generic design must be made so that other apps can use this system and define their own types if possible without changing any code (ie, use xml specs ?)

Inkscape

• project page:
• related blog:
• API:

works over loginless, unencrypted connection. get list by RSS, get direct link, preview&final are same

• 1.query h_ttp://openclipart.org/media/feed/rss/[theSearchTerm] (can be fetched in a normal browser) which returns an RSS feed (empty if nothing found for search term). The rss fields contains "item"s, each item has direct links for preview (png) and download (png,svg...)
• 2.download from direct link of the full-size png/svg to show a preview in the dialog
• 3.the same file that was used for preview is not downloaded again and imported directly into inkscape.

• description: small dialog to search showing matching graphic files in a repo
• prog. language: ?
• functionnality: possible repo adress config (in the global prefs), search only.
• supported formats:SVG ? (todo)
• how to access it: builtin, go in File > Open Clip Art Library
• version:from 0.46+
• author:

Scribus OCAL helper

a 2006 SoC

• version: don't know if it's ever been integrated.

• author: greg.steffensen at gmail.com (wrote to him for more info (jonathan))
• project page: @berlios. not maintained anymore
• related blog: the person's SoC blog. No more postings.

• defining a standard protocol/folder structure(?) for communicating with repositories

Motivation: one current problem for Blender repositories is that there's no standard way to speak in an automated way. So wrappers have to be written for every single repository to be able to browse, fetch and upload.

If we specify some standard way to communicate/browse through the repositories, then all the new repositories could be designed to support that standard and our generic client will work right away with those. Older repositories can spare having people to write wrappers for the client framework to access them if they start conforming that standard as well.

Could you make specifications for either or both of the following ?

- define a server exposed call API + a corresponding protocol describing how the client&server should talk to each other.

- or have a description file at the server root explaining in a coded way how contents are arranged on the server.

• kinds of (blender)-compatible contents, attributes/meta-data to describe them

- We want to list the attributes that could be matched with any blender-compatible content:

1. What are the various blender-compatible components available? (ie materials, textures... Python scripts... ?)

2. What's relevant for users to know about them ? (ex : title, description, size, preview pic (one or more ?), version compatibility, author... ?)

Important: the contents specification generic design must be made so that other apps can use this system and define their own types if possible without changing any code (ie, use xml specs ?)

Roadmap

By tickets ? By version number ? By one step = several tickets to do ?...

Notes

See the Blender.sys module which takes functions from python's standard os module : http://www.zoo-logique.org/3D.Blender/scripts_python/API/ .

todo : study this : http://lists.ibiblio.org/pipermail/cc-devel/2006-July/000164.html ccHost api ?

about annotating objects with custom properties : there's a detailed feature request here : http://en.wikibooks.org/wiki/Blender_3D:_Blending_Into_Python/Custom_datablock_properties Since 2.42 you can access to libdata ID properties of object from python: Here's the python API way to do it (see the properties attribute (of type IDGroup) http://www.blender.org/documentation/246PythonDoc/Object.Object-class.html )

from the bf-committers list : From cbarton at metavr.com Thu Jan 4 03:57:28 2007 From: cbarton at metavr.com (Campbell Barton) Date: Thu Jan 4 03:20:46 2007 Subject: [Bf-committers] BPython API changes since 2.42 Message-ID: <459C6D18.4030005@metavr.com>

• Blender lib data - Added IDproperties to most libdata - a way you can

assign arbitrary properties data to any Blender libdata.

how project petunia will use DBBlender so that many people can download/update a scene, animate some character and reupload it. Here's a schematic of their workflow : http://www.projectpetunia.org/docs/project_workflow basically they'll have an online portal + online api & a gui in blender to interact with the website & local dbblender part. they will need versionning, which is not the first aim of our own project I'd say.

http://highend3d.com/ has models&other contents for various software (see the Downloads category). Don't know if it has an API. To be looked further into.

http://www.blendernation.com/category/repositories/ for a list of other repositories (to study) there should a catalog for repositories on the Blender wiki I thing... !

Contributors

• Developpers:

• Roger Wickes (Roger) aka PapaSmurf,
• Jonathan-david Schröder (myselfhimself).

• Plug-in writers:

no wrappers/plug-ins can be written for now. The plug-in interface hasn't been coded yet.

• Testers:

nothing to test yet.

Want to participate ? Come tell us by writing a message in this thread.