Programming audio spatialization techniques (handling multiple outputs)

[tzortz]

Hello all,

I've hardly started writing a code using audio spatialization techniques (mainly wave field synthesis and binaural rendering), that in a nutshell will have to be able to:

1. Control individually each audio output channel (they will reach a number of 128 or even 256 outputs); by control I mean simple weighting (gain) and delay, and I may have to apply dsp filters on all channels.

2. have a gui, which from within will control the output channels, according to the spatialization technique.

3. the code would better be cross platform, to be easier to handle future changes.

I have to say that I'm quite new in C++ (I'm a bit experienced in creative coding, and creating trivial programs, I've been through a lot of tutorials for the language itself, but I'm note experienced in more complex projects), so sorry if this post may have inaccuracies. Being said that, I'll try to form some questions:

1. For audio I/O I found PortAudio library. I also found fmod, rtaudio, openal, but it seem that port audio will fit better as it integrates with the WASAPI and CoreAudio. There is also Audio Processing Framework ( https://dev.qu.tu-berlin.de/embedded/apf/index.html ) that is intended to be used in applications like this but I'm not sure how to use it. Is it sane to use any of these libraries for this goal or is there a better way? Another important thing is I don't even know how I should do this. I was told to better use circular/ring bufferring on each channel, which I know as a concept, but I don't even know where to begin from. It would be really helpful if you'd point me some examples or hints, as a starting point.

2. For the gui, in order to has a more native feel in each platform, I was thinking, writing each one in its own OS. Is this a good idea, or should I better use an API like Qt or wxWidgets? And also, should I begin with coding the gui, the way the program will handle audio (numbered as "1" here), or should it be done in parallel (for example should I first write the C++ code and then link it with the objective-c and cocoa in XCode)?

This may sound trivial, but after much time of searching I got "lost", as I don't know where I should focus more or less.

Thanks for any of your help!

[Torin]

To be honest, this doesn't belong in the Beginners forum.

[tzortz]

Thanks for the notice. Moved to "General C++ Programming".

[KRAkatau]

I cannot really help you with the audio part, but for the GUI I would definitely recommend using Qt (well, I prefer it over wx myself). It will give the native look and feel on each platform so you don't have to worry about this. Also take a look at QtMultimedia module - if it contains the functionality you need for audio processing - then to me it sounds best to use Qt for both GUI and audio.

For coding, I would start with coding the core algorithms with some tests. This will allow you to i) well, test your code and ii) design a good library API, that will then will be easy to hook up to GUI.

One last thing about audio - you may actually try to target any audio library as a backend. This, however, will require one more abstraction layer and thus will require some more work.

Hope this helps at least a bit :)

[htirwin]

Control individually each audio output channel (they will reach a number of 128 or even 256 outputs); by control I mean simple weighting (gain) and delay, and I may have to apply dsp filters on all channels.

That's a lot of outputs. Do you want to apply dsp filters in real time?

I've got an openAL circular buffer going in one of my projects, for 2 channels, but openAL is not really designed for this.

To get reasonable latency with that many channels, you will probably will want to use high performance drivers. ASIO/ ASIO for all, on windows, core audio on mac, and JACK on linux.

Accomplishing that many simultaneous outputs will also require a ton of memory.

I wish I could help with accomplishing that. I once tried to get port audio working with ASIO, when I was less experienced and failed. I plan to try again sometime relatively soon.

You could use openAL, and their built in EAX effects. The only EAX effect compatible with a non creative sound card is the reverb though. Although openAL soft might let you use the others.


My openAL circular buffer is very full of stuff specific to my program, and is definitely not the model solution. But I'll post it anyways. Maybe you'll be able to get something out of it.

[htirwin]

void OpenAL_Callback_Func(gpointer user_data){

    //note lowest successful buffer size has been 1400 samples
    State *SMdata = (State *)user_data;
    static int bfz;
    static unsigned long int NapTime = 250;
    static int feedback = 100;

    short buffers [2][44100];
    float buffers_L [22050];
    float buffers_R [22050];
    
    ALint processed;
    int BufferI;
    int BufferN;
    int pt;
    int li=0;
    int ri=0;	

    while (1){
        if(SMdata->OAL_Data.Play == true){
            bfz = SMdata->OAL_Data.BufferSize;
            if (SMdata->OAL_Data.call == true){
                SMdata->OAL_Data.ip = 0;
                SMdata->OAL_Data.pos = 0;
                SNDFILE *SoundWavFloat = sf_open(SMdata->OAL_Data.PlayNowPath.c_str(), SFM_READ, &SMdata->OAL_Data.SoundInfoFloat);		
                if(SoundWavFloat){   
                    delete [] SMdata->OAL_Data.Data_Float;
                    delete [] SMdata->OAL_Data.Data_Float_L;
                    delete [] SMdata->OAL_Data.Data_Float_R;

                    SMdata->OAL_Data.Data_Float = new float[SMdata->OAL_Data.SoundInfoFloat.channels * SMdata->OAL_Data.SoundInfoFloat.frames];
                    SMdata->OAL_Data.Data_Float_L = new float[SMdata->OAL_Data.SoundInfoFloat.frames];
                    SMdata->OAL_Data.Data_Float_R = new float[SMdata->OAL_Data.SoundInfoFloat.frames];

                    sf_readf_float(SoundWavFloat, SMdata->OAL_Data.Data_Float, SMdata->OAL_Data.SoundInfoFloat.frames);
        
                    ri=0;
                    li=0;
                    for (int i=0 ; i < (SMdata->OAL_Data.SoundInfoFloat.channels * SMdata->OAL_Data.SoundInfoFloat.frames); ++i) {
                        if (!(i & 1)) {
                            SMdata->OAL_Data.Data_Float_L[li] = SMdata->OAL_Data.Data_Float[i];
                            ++li;
                        }
                        if (i & 1) {
                            SMdata->OAL_Data.Data_Float_R[ri] = SMdata->OAL_Data.Data_Float[i];
                            ++ri;
                        }
                    }			

                    SMdata->OAL_Data.call = false;
                    for (int i=0; i < 2; ++i)
                        for (int ii=0; ii < bfz*2; ++ii)
                            buffers[i][ii] = (short)(SMdata->OAL_Data.Data_Float[i*bfz*2+ii]*32768);

                    alSourceUnqueueBuffers (SMdata->OAL_Data.OAl_Source, 2, SMdata->OAL_Data.OAl_Buffer);
                    for (int i=0; i < 2; ++i){
                        alBufferData(SMdata->OAL_Data.OAl_Buffer[i], AL_FORMAT_STEREO16, buffers[i], bfz*2*sizeof(short), SMdata->OAL_Data.SoundInfoFloat.samplerate);
                    }
                    alSourceQueueBuffers (SMdata->OAL_Data.OAl_Source, 2, SMdata->OAL_Data.OAl_Buffer);
                    alSourcePlay (SMdata->OAL_Data.OAl_Source);
                    SMdata->OAL_Data.ip=2;
                }
                else {
                    alSourceStop(SMdata->OAL_Data.OAl_Source);
                    SMdata->OAL_Data.Play = false;
                }
            }
            if (SMdata->OAL_Data.pos < SMdata->OAL_Data.SoundInfoFloat.frames - bfz){
                alGetSourcei(SMdata->OAL_Data.OAl_Source, AL_BUFFERS_PROCESSED, &processed);
            if(processed > 0){
                ++SMdata->OAL_Data.ip;
                    if(SMdata->OAL_Data.ip & 1){
                        BufferI = 0;
                    }
                    else {
                        BufferI = 1;
                    }
                    if (SMdata->OAL_Data.Position_Changed == true){
                        if((SMdata->OAL_Data.ip & 1) && (!(SMdata->OAL_Data.PosTemp & 1)))
                            --SMdata->OAL_Data.PosTemp;
                        else if((!(SMdata->OAL_Data.ip & 1)) && (SMdata->OAL_Data.PosTemp & 1))
                            --SMdata->OAL_Data.PosTemp;

                        SMdata->OAL_Data.ip = SMdata->OAL_Data.PosTemp;
                        SMdata->OAL_Data.Position_Changed = false;
                    }

                    alSourceUnqueueBuffers (SMdata->OAL_Data.OAl_Source, 1, SMdata->OAL_Data.OAl_Buffer);
            
                    if(SMdata->OAL_Data.ip > 0 && bfz*(SMdata->OAL_Data.ip-1)+bfz <= SMdata->OAL_Data.SoundInfoFloat.frames){
                        for (int i=0; i < bfz; ++i){		    
                            buffers_L[i] = SMdata->OAL_Data.Data_Float_L[bfz*(SMdata->OAL_Data.ip-1)+i];  //process here
                            buffers_R[i] = SMdata->OAL_Data.Data_Float_R[bfz*(SMdata->OAL_Data.ip-1)+i];  //process here						
                        }
                    }                                                                                  
                    else if(bfz*(SMdata->OAL_Data.ip)+bfz <= SMdata->OAL_Data.SoundInfoFloat.frames){     
                        for (int i=0; i < bfz; ++i){
                            buffers_L[i] = SMdata->OAL_Data.Data_Float_L[bfz*(SMdata->OAL_Data.ip)+i];  
                            buffers_R[i] = SMdata->OAL_Data.Data_Float_R[bfz*(SMdata->OAL_Data.ip)+i];  
                        }
                    }
                    li=0;
                    ri=0;
                    for(int i=0; i < bfz*2; ++i){
                        if(!(i & 1)){
                            buffers[BufferI][i] = (short)(buffers_L[li]*32768);
                            ++li;
                        }
                        if(i & 1){
                            buffers[BufferI][i] = (short)(buffers_R[ri]*32768);
                            ++ri;
                        }
                    }

                    alBufferData(SMdata->OAL_Data.OAl_Buffer[BufferI], AL_FORMAT_STEREO16, buffers[BufferI], bfz*2*sizeof(short), SMdata->OAL_Data.SoundInfoFloat.samplerate);
                    alSourceQueueBuffers (SMdata->OAL_Data.OAl_Source, 1, SMdata->OAL_Data.OAl_Buffer);
                }
            }
            else if (SMdata->OAL_Data.Loop == true){
                SMdata->OAL_Data.call = true;
            }
            if (SMdata->OAL_Data.BufferSize_Changed == true){
                NapTime = (unsigned long int)((((float)bfz/(float)SMdata->OAL_Data.SoundInfoFloat.samplerate)*1000.0f)-20.0f);
                SMdata->OAL_Data.BufferSize_Changed = false;
            }
        }
    Sleep(NapTime);
    }
}

[htirwin]

The basic Idea, is that I read the files into an array using libsndfile.

Then I de-interleave the data, into individual arrays per channel (left, and right).
If your sound files are in mono you can skip this.

Blah Blah ...

Then the main thing is here,

alGetSourcei(SMdata->OAL_Data.OAl_Source, AL_BUFFERS_PROCESSED, &processed);
if(processed > 0){    
    /*then you've play through the first buffer in the queue.
    that means your currently playing the second buffer
    so you unqeue the buffer done playing*/
    alSourceUnqueueBuffers (SMdata->OAL_Data.OAl_Source, 1, SMdata->OAL_Data.OAl_Buffer);
    /*the second buffer has now become the first and only queued buffer
    fill an array up with the next samples for each channel; you can process here
    reinterleave the left and right channels into one array
    turn it into an openAL buffer*/
    alBufferData(SMdata->OAL_Data.OAl_Buffer[BufferI], AL_FORMAT_STEREO16, buffers[BufferI], bfz*2*sizeof(short), SMdata->OAL_Data.SoundInfoFloat.samplerate);
   // then queue it, putting it now into the second position, as the next to be played.
   alSourceQueueBuffers (SMdata->OAL_Data.OAl_Source, 1, SMdata->OAL_Data.OAl_Buffer)

// and you need to have a worker thread, perpetually doing this, 
//checking if the first buffer has been played, if it has, unqueue it and fill it with new data, and then queue it.  
}

This is probably a bad example, but it might help you get an idea.
It would be way different with port audio.

For EQ filters, you can google, DSPfilters.

[tzortz]

@KRAkatau, yes, Qt seems really nice and a reasonable way to go, as it will save a lot of time at the end of the day. I think I'll use it, and check also QtMultimedia. I didn't know I could target any audio library somehow; I'll try to figure this out. Do you have anywhere to point to, by any chance? Thanks also for the coding tips!

@iseeplusplus, thanks for the circular buffering example! Really glad! I'll try to figure how it works out and I'm sure it will help for my progress. Yes, I'm afraid I'll have to use dsp on realtime; it will be just an IIR filter though, applied the same on all channels. What I need individually, is time delay on the signals (with I think I can do it through the circle buffer), and gain. Found also the DSPfilters library; seems handy.

You think it's better to use ASIO rather than WASAPI or JACK on windows? I was thinking using the apf libraries I mentioned ( https://dev.qu.tu-berlin.de/embedded/apf/index.html ), which include multichannel processing, filter classes, and wrappers for JACK, which in its turn uses the main drivers used by the OS (ASIO, CoreAudio etc), isn't that right (I think this is what KRAkatau was thinking about, no?)? Or head another way figuring out how to use PortAudio.

Any other thoughts on it are more than welcome!

Thanks again!

[htirwin]

You think it's better to use ASIO rather than WASAPI or JACK on windows? I was thinking using the apf libraries I mentioned ( https://dev.qu.tu-berlin.de/embedded/apf/index.html ), which include multichannel processing, filter classes, and wrappers for JACK, which in its turn uses the main drivers used by the OS (ASIO, CoreAudio etc), isn't that right (I think this is what KRAkatau was thinking about, no?)? Or head another way figuring out how to use PortAudio.

It looks like JACK can actually use ASIO on windows.

Although I noticed this.

Currently Jack for Windows only supports 32-bit applications, but a new version is currently being tested that supports both 32-bit and 64-bit audio applications. If you are working with 64-bit applications then contact the Jack Developers list for more information.

http://jackaudio.org/jack_on_windows

With 256 outputs, you might need a lot of memory, and a 32 bit application I think can only use 4 gig's? Something to consider.

I would recommend downloading a multitrack audio program like, http://www.reaper.fm/ , open a sound file add, a delay effect, and a filter, duplicate it into 256 channels, press play, and watch your CPU and memory usage. You could also go into preferences and change the audio device for comparison.

Keep in mind reaper is very efficient.

[KRAkatau]

About targeting any audio library - it is quite a general comment. By that I mean that you could add your own abstraction layer (such as abstract interfaces for audio device, audio channel, etc.) and then write implementations for each particular platform/library. Then you can instantiate the concrete classes using, for example, a factory. However, I think this is quite a lot of work which will only pay off if you really need this flexibility in the long run.

I am pretty sure that QtMultimedia is actually this kind of abstraction layer and it uses different ways to access audio devices on different platform. If you take a look inside the source code you will find implementations for windows, mac, alsa and symbian. However, looking at the answer by iseeplusplus, it seems to me that Qt might not have enough functionality for your task (but please check :)). Still, Qt usually has really good APIs and abstractions so you may borrow some of the ideas from it if you decide to target a variety of platforms/libraries.