利用者:Hcube/SummerOfCode2007

Blender Audio System Cleanup and Upgade

Status

This project is still running (2008.08.21). Unfortunatly it missed the 2.47 blender version, but hopefully will be included in the next release. There are only few things remained for merging with trunk.

• game engine support
• sound normalization after resampling
• packed file support via libsndfile (currently it is supported only via ffmpeg)

Download from SVN

svn co https://svn.blender.org/svnroot/bf-blender/branches/sound-branch

Proposal

User:Hcube/SummerOfCode2007/Proposal

Changes since the proposal

• drop away SoundSystem code
• make a completely new lib with clean design: tinySND
• the main motivation is getting blender's audio to work and make the new audio system well extensible for future.

Problems with current blender audio system

Mixture use of SDL and openal. SDL is used for sequencing, mixing, openal is used for game engine sound. These two system cant be used parallel. There is also an inner dependency due to hdaudio is using ffmpeg. From programmer's view the audio code is confusion too because the source is not centralized: SoundSystem, editsound.c, seqaudio.c implements different parts of audio features. There is audio feature implementation code in gui related part of source: editsound.c

Cleanup goals

It would be nice to unify every audio code into one library. It will give a more clean design and it can be more easily maintained too. We have to include further feature extensibility as a requirement, because without that after a long time we will get confusions in source code again. (New features are always coming.)

Cleanup requirements

• no feature loss
• clean blender side source code
• extensible sound system source code (Not an openal wrapper)

Cleanup analysis and design

After i've spent a lot of time to learn current SoundSystem, i've found that is specialized for openal or fmod like 3D APIs. Actually it is a wrapper for them. At blender side, the situation is more chaotic. There are several sound related source files and the solutions were too complex. Ex: the SoundSytem was not used, instead SDL was used in several times. FFmpeg was used directly. The ideal solution is putting every sound related implementation out from blender core code. ex to SoundSystem, but it has no 2d sound implementations. And it has no real feature implementation code, only it wraps openal and fmod.

SoundSystem problems:

• no file load/save api
• whole api is concentrate on 3d sound
• lacks of real feature implementations(only wraps external libs)

Due to these i've begun designed a new SoundSystem lib. It has taken me one week. I've not designed it alone, i discussed the requirements with Bob and schlaile. The original concept was that SDL will be the only dependency.

Bob's requirements

I would like to see the following use cases thought through and posted:

• basic wave reader/writer (Explicit task)
• basic aiff reader/writer (Explicit task)
• additional formats loaded via ffmpeg (Implied task)
• loading a sound (of different types)
• playing a sound
• playing multiple sounds
• sync with animation (including scrubs) (Explicit task)
• API spec (Explicit task)

schlaile's requirements

Schlaile is currently working on a generic plugin system.

• sdl independent, make backends pluggable
• jack backend. It allows cooperation with professional audio editors which supports jack like ardour.

hcube's requirements

I want something more than play/stop. I want features that are useful for gaming or making movie.

• keep extensible.
• Uni-Verse:UVAS module (later) Can be used for baking surround sounds (5.1 or 7.1, etc)
• LADSPA module (later)

The audio code consist of two parts:

• Blender side code. That handles sequencing, and handles bSound and bSample structure, but does not implements any sound playing feature, only calls the (new)SoundSystem lib.
• (new)SoundSystem lib. This implements every feature what we need. (load,save,play,stop,mixdown,conversions,etc)

After a week of designing i've got the final design and i found that is very generic. The new SoundSystem's name is tinySND. Old SoundSystem and new SoundSystem (tinySND) are not blender specific. That is not a problem because ex: we can add features to them without changing blender's file structure.

tinySND

This the new sound library it is named because it is blender independent and i'd like to release it as an independent sound library. It is written in C++. It is very modular and simple and customizable. It will have nice features:

• backends: SDL, portaudio, jack
• file handler backends: internal(wav, aiff), sndfile, ffmpeg
• components:
  • ladspa module
  • sample rate converter module
  • mixer module
  • UVAS module for 3d acoustics

Design

tinySND is based on 3 interface classes:

• SND_Device: abstract interface for audio backends.
• SND_DataProvider: abstract interface for audio file handling backends.
• SND_SoundInterface: abstract interface for all audio components

tinySND SND_Device Backends

• SDL [done]
• portaudio [done]
• jack [done]

tinySND SND_DataProvider Backends

• sndfile [done]
• vorbis(ogg) [done]
• mad(mp3) [in progress]
• ffmpeg [in progress]
• internal WAV [planned]
• internal AIFF [planned]

tinySND Components

• Mixer
• SampleRateConverter
• Sample - Loopable sound sample.
• Ladspa module (Future plan)- Useful for realtime game audio effects.
• 3D Simulation module (Future plan) - Useful for surround sound baking. (ex 5.1)

Blender side

feature dependency flow: BSE_seqaudio.h => BAU_soundsystem.h => tinySND

BAU_soundsystem.h

typedef void audio_callbackfunction( int framesNum, void *userData1, void *userData2 );

// general functions
void    audio_initialize(void);
int     audio_isinitialized(void);
void    audio_finalize(void);
int     audio_getmixrate(void);
void	audio_setcallback(audio_callbackfunction *callbackFunction, void *userData1, void *userData2);

// file functions, used for mixdown
// playing sound during writing into a file is disabled
int     audio_file_openforwrite(char *name, int sampleRate);
void    audio_file_writeframes(int framesNum);
void    audio_file_close(void);

// mixbuffer functions, allows to reach mixed and multiplexed sound data.
// now it supports signed 16 bit pcm data and 2 channel
int     audio_mixbuffer_open(int sampleRate);
void    audio_mixbuffer_fill(void *buffer, int framesNum);
void    audio_mixbuffer_close(void);

// bSample functions
struct bSample* audio_sample_new(char *name);
struct bSample* audio_sample_find(char *name);
int             audio_sample_load(struct bSample* sample);
void            audio_sample_unload(struct bSample* sample);

// pcm data parameters:  4KHz and 8 bit, it is just for visual purpuses
char*           audio_sample_getvisualpcmdata(struct bSample *sample, int channelIdx);

// bSound functions
// hint: a bSound->sample cant be NULL, always have to be a valid pointer
struct bSound*  audio_sound_new(char *name);
void            audio_sound_delete(struct bSound* sound);
struct bSound*  audio_sound_clone(struct bSound* originalsound);
struct bSound*  audio_sound_findbyid(char *id_name);

int             audio_sound_isplaying(struct bSound *sound);
void            audio_sound_play(struct bSound *sound);
void            audio_sound_stop(struct bSound *sound);

int             audio_sound_getframeposition(struct bSound *sound);
void            audio_sound_setframeposition(struct bSound *sound, int framePos);

// sound attributes
float           audio_sound_getgain(struct bSound *sound);
void            audio_sound_setgain(struct bSound *sound, float gain);

float           audio_sound_getpan(struct bSound *sound);
void            audio_sound_setpan(struct bSound *sound, float pan);

int             audio_sound_ismuted(struct bSound *sound);
void            audio_sound_setmute(struct bSound *sound, int mute);

// global sound functions
void            audio_sound_stopall(void);
void            audio_sound_loadall(void);

BSE_seqaudio.h

void seqaudio_initialize(void);
void seqaudio_finalize(void);

void seqaudio_play(void);
void seqaudio_stop(void);

int  seqaudio_getframeposition(void);
void seqaudio_scrub(void);

int  seqaudio_mixdown(char *fileName, int sampleRate);

blender side todo

• support packed sound files
• support scrub [done]
• support sound seq pan,gain,mute [done]
• support seqaudio get pos and animation playback sync with sound [done]
• support mixdown sound feature [done]
• support sound wave drawing [done]
• support ffmpeg writing [done]

Compile guide

Now i'm working on linux, ubuntu feisty fawn and debian etch 4.0 and i use scons for building. There is an initial build scripts are made to support the new soundsystem. For building my branch you need these libs:

• sdl: http://www.libsdl.org/
• portaudio: http://www.portaudio.com (v19 version, and dont forget to install c++ binding too)
• libsamplerate: http://www.mega-nerd.com/SRC/
• libsndfile: http://www.mega-nerd.com/libsndfile/
• libvorbis, libvorbisfile: http://xiph.org/downloads/
• libmad: http://www.underbit.com/products/mad/

Dependencies

device backend:

• SDL(optional)
• portaudio(optional)
• jack(optional)

loader backend:

• internal(always, will be multi platform)
• ffmpeg(optional)
• libsndfile(optional)
• vorbis(optional)
• mad(optional)

• SDL - multi platform: Linux, Windows, Windows CE, BeOS, MacOS, Mac OS X, FreeBSD, NetBSD, OpenBSD, BSD/OS, Solaris, IRIX, and QNX
• portaudio - multi platform: Windows, Macintosh (8,9,X), Unix (OSS), SGI, and BeOS
• libsamplerate - platform independent (Ansi C)
• libsndfile - multi platform: Unix, Win32, MacOS

Binary size

Now openal(346kb) is not part of the sound system, it is removed

• tinySND: 170kb (without debug info)
• libsamplerate: 114kb

License

All soundsystem code in blender and also tinySND's blender interface is released under GPLv2.

• BAU_soundsystem.h
• soundsystem.c
• BIF_seqaudio.h
• seqaudio.c
• files in tinySND/blender/ directory
  • SND_C-api.h
  • SND_C-api.cpp

The tinySND library is released under LGPL.

After SOC todo list

Documentation

• blender sound system overview
• tinySND overview (blender independent)

tinySND

• DataConverter (any type -> float converter) [not started]
• FXLADSPA [not started]
• FXLV2PLUG [not started]
• design CD API [not started]
• ffmpegReader [done]
• ffmpegWriter [done]
• jack DataProvider
• wavReader [in progress]
• wavWriter [not started]
• aiffReader [not started]
• aiffWriter [not started]
• finish all portaudio, and SDL device features
• SND_Device's getChannelName implementation, supporting 5.1, etc
• make it possible to directly connect SND_DataProvider and SND_DataConsumer
• implement endian conversion in SND_DataMultiplexer
• optimize data conversion in SND_DataMultiplexer

tinySND's blender interface

• add visual data cache system
• implement samplerate chain building if one is not enought for downsample
• reader/writer registration API [done]
• add API for device configuration [in progress]

Blender side

• support sound seq ipos
• support packed sound files [done]
• add sound system config UI
• support blender game engine [in progress]

Future plans

• design SND_FXLADSPA class (useful for adding innovative and interesting sound effects into games)
• design how to integrate UVAS
• add SSE optimalization
• design UI for FXLADSPA and FXUVAS in blender (Sound Nodes and 3d sound materials and sound baking)