Android Speex echo cancellation problems

Question

I have a basic audiorecord-audiotrack, udp packets voice chat between two android devices. It works, but I have a bad echo. I'm trying to remove the echo using Speex ported to android by JNI. The speex I imported works, but the echo cancellation doesn't. Native C code is:

#include <jni.h>
#include <speex/speex_echo.h> 

#define FRAME_SIZE 256
#define FILTER_LENGTH 800

SpeexEchoState *echoState;

// Initialization of echo cancelation
void Java_telefonie_voip_VoIPActivity_InitEcho(JNIEnv * env, jobject jobj)
{
    echoState = speex_echo_state_init(FRAME_SIZE, FILTER_LENGTH);
}

//  Queue the frame to soundcard for playing (receiving)
void Java_telefonie_voip_VoIPActivity_Playback(JNIEnv * env, jobject jobj, jshortArray     inputFrame)
{
    speex_echo_playback(echoState, inputFrame);
}

jshortArray Java_telefonie_voip_VoIPActivity_Capture(JNIEnv * env, jobject jobj,     jshortArray inputFrame)
{
    jshortArray outputFrame;

    speex_echo_capture(echoState, inputFrame, *&outputFrame);

    return outputFrame;
}

// Destroing echo cancelation
void Java_telefonie_voip_VoIPActivity_DestroyEcho(JNIEnv * env, jobject jobj)
{
speex_echo_state_destroy(echoState); 
}

And some of important java code:

native void InitEcho();
native void DestroyEcho();
native void Playback(short[] inputData);  // listener/receiving
native short[] Capture(short[] inputData);  // recorder/sender
static {
    System.loadLibrary("speex");
}

public void Initialize ()
{
    minBufferSize = 4096;
    try
    {
        InitEcho();
    }
    catch (Exception e){Log.e(TAG, "jni " + e.getMessage());}
}

public void startRecording ()
{
    isStreaming = true;
streamingThread = new Thread(new Runnable(){
        @Override
        public void run ()
        {
            try
            {
                audioRecorder = new AudioRecord(
                        MediaRecorder.AudioSource.MIC,
                        sampleRate,
                        channelConfig,
                        audioFormat,
                        minBufferSize*10
                );

                buffer = new short[minBufferSize];
                audioRecorder.startRecording();

                while(isStreaming)
                {
                    sendPackets++;
                    minBufferSize = audioRecorder.read(buffer, 0, buffer.length);     
                    //  Echo cancelling
                    buffer = Capture(buffer);

                    //  Send
                    dataPacket = new DatagramPacket(ShortToByteArray(buffer), buffer.length*2, receiverAddressIA, serverPort1);
                    dataSocket.send(dataPacket);
                }
            }
            catch(Exception e)
            {
                Log.e(TAG, "2" + e.getMessage());
            }
        }
    });
    streamingThread.start();
}

public void startListen ()
{
    isListening = true;
receivingThread = new Thread(new Runnable(){
        @Override
        public void run ()
        {
            try
            {
                audioTrack = new AudioTrack(
                        AudioManager.STREAM_VOICE_CALL,
                        sampleRate,
                        channelConfig,
                        audioFormat,
                        minBufferSize*10,
                        AudioTrack.MODE_STREAM
                );

                audioTrack.play();

                buffer2 = new short[minBufferSize];

                while (isListening)
                {
                    receivedPackets++;
                    dataPacket2 = new DatagramPacket(ShortToByteArray(buffer2), buffer2.length*2);
                    dataSocket2.receive(dataPacket2);

                    //  Cancel echo
                    Playback(buffer2);

                    //  Queue to soundcard
                    audioTrack.write(buffer2, 0, minBufferSize);
                }
            }
            catch(Exception e)
            {
                Log.e(TAG, "3" + e.getMessage());
            }
        }
    });
    receivingThread.start();
}

public short[] ByteToShortArray(byte[] byteArray)
{
    short[] shortArray = new short[byteArray.length/2];
ByteBuffer.wrap(byteArray).order(ByteOrder.BIG_ENDIAN).asShortBuffer().get(shortArray);
    return shortArray;
}

The problem is that when I start the recorder/streaming thread, it shows me that it sends one package and then the app crashes with no message. Do you have any sugestions or advices? Please help me because I need to do this project asap and I've worked hard and documented myself but still it doesn't want to work well. Thank you!

Edit: I've just discovered that

//  Echo cancelling
buffer = Capture(buffer);

is triggering the crash.

Answer 1

UPDATE: Regarding crash - This is due to mis-use of JNI. You can't treat jshortarray as a C pointer. You need to call the relevant JNI functions to convert it to a C pointer (GetShortArrayElements and so on).

Regarding echo canceler - you currently use the internal buffer, which is about 2 frames, and that might not be enough for you. Android's context-switching is not like on a PC, and you might get 4 consecutive playback frames -> 4 consecutive recorded frames -> and so on. So you lose frames that way.

Also, Android audio streaming has higher latency than a PC, so the 2-frame buffer isn't enough. You need to implement your own buffering and play with the amount of frames by which you delay, so the echo canceler sees the echo after the playback inside the boundaries of its filter length.

Other than that, the best way to analyze and debug the echo canceler is: 1. Actually understand how it works on a high level and what are the requirements - not a rocket science, you can find everything on the official Speex docs. 2. Use the provided echo_diagnostic tool, and examine the streams in a tool such as Audacity to see where it went wrong.

The Speex AEC works pretty well when used correctly, and I've already done 3 different projects using it, so it's just a matter of correcting your implementation.