Audio extension

The audio filters you created are easily integrated into apps to supply your voice effects and noise cancellation.

Understand the tech

An audio filter accesses voice data when it is captured from the user's local device, modifies it, then plays the updated data to local and remote video channels.

A typical transmission pipeline consists of a chain of procedures, including capture, pre-processing, encoding, transmitting, decoding, post-processing, and play. In order to modify the voice or video data in the transmission pipeline, audio extensions are inserted into either the pre-processing or post-processing procedure.

Prerequisites

In order to follow this procedure you must have:

Android Studio 4.1 or higher.
Android SDK API Level 24 or higher.
A mobile device that runs Android 4.1 or higher.

A project to develop in.

Project setup

In order to integrate an extension into your project:

To integrate into your project:

Unzip Video SDK to a local directory.
Copy the header files in rtc/sdk/low_level_api/include under the directory of your project file.

You are now ready to develop your extension.

Create an audio extension

To create an extension in your Agora project, use the following APIs:

IAudioFilter: Implement receiving, processing, and delivering audio data.
IExtensionProvider: encapsulate your IAudioFilter implementation into an extension.

Develop an audio filter

Use the IAudioFilter interface to implement an audio filter. You can find the interface in the NGIAgoraMediaNode.h file. You need to implement this interface first, and you must implement at least the following methods from this interface:

adaptAudioFrame
setEnabled
isEnabled
setProperty
getProperty
getName
getPreferredSampleRate(Optional)
getPreferredChannelNumbers(Optional)

The following code sample shows how to use these methods together to implement an audio filter extension:

// After receiving the audio frames to be processed, call adaptAudioFrame to process the audio frames.
bool ExtensionAudioFilter::adaptAudioFrame(const media::base::AudioPcmFrame &inAudioPcmFrame,
                                           media::base::AudioPcmFrame &adaptedPcmFrame) {
    return audioProcess_->processFrame(inAudioPcmFrame, adaptedPcmFrame) == 0;
}
// Call setProperty to set the property of the audio filter.
int ExtensionAudioFilter::setProperty(const char* key, const void* buf, int buf_size) {
    std::string str_volume = "100";
    if (std::string(key) == "volume") {
        str_volume = std::string(static_cast<const char*>(buf), buf_size);
    }
    int int volume_ = atoi(str_volume.c_str());
    audioProcessor_->setVolume(int_volume_);
    return ERR_OK;
}
// Call getProperty to get the property of the audio filter.
int ExtensionAudioFilter::getProperty(const char* key, void* buf, int buf_size) const override {return ERR_OK; }
// Call setEnabled to enable the audio filter.
void ExtensionAudioFilter::setEnabled(bool enable) override { enabled_ = enable; }
// Call isEnabled to check whether the audio filter is enabled.
bool ExtensionAudioFilter::isEnabled() const override {return enabled_; }
// Set the vendor name in the return value of getName.
const char* getName() const override { return filterName_.c_str(); }
// (Optional) Specify the audio sample rate in the return value of getPreferredSampleRate
const char* ExtensionAudioFilter::getPreferredSampleRate() override { return 48000; }
// (Optional) Specify the number of audio channels in the return value of getPreferredChannelNumbers
int ExtensionAudioFilter::getPreferredChannelNumbers() override { return 2; };

Encapsulate the filter into an extension

To encapsulate the audio filter into an extension, you need to implement the IExtensionProvider interface. You can find the interface in the NGIAgoraExtensionProvider.h file. The following methods from this interface must be implemented:

enumerateExtensions
createAudioFilter
setExtensionControl

The following code sample shows how to use these methods to encapsulate the audio filter:

void ExtensionProvider::enumerateExtensions(ExtensionMetaInfo* extension_list,
                                            int& extension_count) {
    extension_count = 2;
    //Declare a Video Filter, and IExtensionProvider::createVideoFilter will be called
    ExtensionMetaInfo i;
    i.type = EXTENSION_TYPE::VIDEO_PRE_PROCESSING_FILTER;
    i.extension_name = agora::extension::VIDEO_FILTER_NAME;
    extension_list[0] = i;
    //Declare an Audio Filter, and IExtensionProvider::createAudioFilter will be called
    ExtensionMetaInfo j;
    j.type = EXTENSION_TYPE::AUDIO_FILTER;
    j.extension_name = agora::extension::AUDIO_FILTER_NAME;
    extension_list[1] = j;
}
agora_refptr<agora::rtc::IAudioFilter> ExtensionProvider::createAudioFilter(const char* name) {
    PRINTF_INFO("ExtensionProvider::createAudioFilter %s", name);
    auto audioFilter = new agora::RefCountedObject<agora::extension::ExtensionAudioFilter>(name, audioProcessor_);
    return audioFilter;
}
void ExtensionAudioProvider::setExtensionControl(rtc::IExtensionControl* control){
    audioProcessor_->setExtensionControl(control);
}

Package the extension

After encapsulating the filter into an extension, you need to register and package it into a .aar or .so file, and submit it together with a file that contains the extension name, vendor name and filter name to Agora.

Register the extension

Register the extension with the macro REGISTER_AGORA_EXTENSION_PROVIDER, which is in the AgoraExtensionProviderEntry.h file. Use this macro at the entrance of the extension implementation. When the SDK loads the extension, this macro automatically registers it to the SDK. For example:

_1REGISTER_AGORA_EXTENSION_PROVIDER(ByteDance, agora::extension::ExtensionProvider);

Link the libagora-rtc-sdk-jni.so file

In CMakeLists.txt, specify the path to save the libagora-rtc-sdk-jni.so file in the downloaded SDK package according to the following table:

File	Path
64-bit `libagora-rtc-sdk-jni.so`	`AgoraWithByteDanceAndroid/agora-bytedance/src/main/agoraLibs/arm64-v8a`
32-bit `libagora-rtc-sdk-jni.so`	`AgoraWithByteDanceAndroid/agora-bytedance/src/main/agoraLibs/arm64-v7a`

Provide extension information

Create a .java or .md file to provide the following information:
- EXTENSION_NAME: The name of the target link library used in CMakeLists.txt. For example, for a .so file named libagora-bytedance.so, the EXTENSION_NAME should be agora-bytedance.
- EXTENSION_VENDOR_NAME: The name of the extension provider, which is used for registering in the agora-bytedance.cpp file.
- EXTENSION_FILTER_NAME: The name of the filter, which is defined in ExtensionProvider.h.

Test your implementation

To ensure that you have integrated the extension in your app:

Once you have developed your extension and API endpoints, the next step is to test whether they work properly.

Functional and performance tests

Test the functionality and performance of your extension and submit a test report to Agora. This report must contain:
- The following proof of functionality:
  - The extension is enabled and loaded in the SDK normally.
  - All key-value pairs in the setExtensionProperty or setExtensionPropertyWithVendor method work properly.
  - All event callbacks of your extension work properly through IMediaExtensionObserver.
- The following performance data:
  - The average time the extension needs to process an audio or video frame.
  - The maximum amount of memory required by the extension.
  - The maximum amount of CPU/GPU consumption required by the extension.

Extension listing test The Extensions Marketplace is where developers discover your extension. In the Marketplace, each extension has a product listing that provides detailed information such as feature overview and implementation guides. Before making your extension listing publicly accessible, the best practice is to see how everything looks and try every function in a test environment.
Write the integration document for your extension

The easier it is for other developers to integrate your extension, the more it will be used. Follow the guidelines and create the best Integration guide for your extension
Apply for testing

To apply for access to the test environment, contact Agora and provide the following:
- Your extension package
- Extension listing assets, including:
- Your company name
- Your public email address
- The Provisioning API endpoints
- The Usage and Billing API endpoints
- Your draft business model or pricing plan
- Your support page URL
- Your official website URL
- Your implementation guides URL
Test your extension listing

Once your application is approved, Agora publishes your extension in the test environment and sends you an e-mail.

To test if everything works properly with your extension in the Marketplace, do the following:
- Activate and deactivate your extension in an Agora project, and see whether the Provisioning APIs work properly.
- Follow your implementation guides to implement your extension in an Agora project, and see whether you need to update your documentation.
- By the end of the month, check the billing information and see whether the Usage and Billing APIs work properly.

Now you are ready to submit your extension for final review by Agora. You can now Publish Your Extension.

Reference

This section contains content that completes the information on this page, or points you to documentation that explains other aspects to this product.

Sample project

Agora provides an Android sample project agora-simple-filter for developing audio and video filter extensions.

API reference

The classes used to create and encapsulate filters are:

IAudioFilter: Implement receiving, processing, and delivering audio data.
IExtensionProvider: encapsulate your IAudioFilter implementation into an extension.

IAudioFilter

Implement receiving, processing, and delivering audio data.

Methods include:

adaptAudioFrame
setEnabled
isEnabled
setProperty
getProperty
getName
getPreferredSampleRate(Optional)
getPreferredChannelNumbers(Optional)

adaptAudioFrame

Adapts the audio frame. This is the core method of the IAudioFilter interface. By calling this method, the SDK processes audio frames from inAudioFrame and returns the adapted frames with adaptedFrame. This method supports audio data in the PCM format only.

virtual bool adaptAudioFrame(const media::base::AudioPcmFrame& inAudioFrame,
                             media::base::AudioPcmFrame& adaptedFrame) = 0;

Parameter	Description
`inAudioFrame`	An input parameter. The pointer to the audio frames to be processed.
`adaptedFrame`	An output parameter. The pointer to the processed audio frames.

setEnabled

Enables or disables the audio filter.

virtual void setEnabled(bool enable) {}

Parameter	Description
`enable`	Whether to enable the audio filter： true: Enable the audio filter. false: (Default) Disable the audio filter.

isEnabled

Checks whether the audio filter is enabled.

virtual bool isEnabled() { return true; }

Returns Whether the audio filter is enabled:

true: The audio filter is enabled.
false: (Default) The audio filter is disabled.

setProperty

Sets the property of the audio filter. When an app client calls setExtensionProperty, the SDK triggers this callback. In the callback, you need to return the property of the audio filter.

size_t setProperty(const char* key, const void* buf, size_t buf_size)

Parameter	Description
`key`	The key of the property.
`buf`	The buffer of the property in the JSON format. You can use the open source `nlohmann/json` library for the serialization and deserialization between the C++ struct and the JSON string.
`buf_size`	The size of the buffer.

getProperty

Gets the property of the audio filter. When the app client calls getExtensionProperty, the SDK calls this method to get the property of the audio filter.

size_t getProperty(const char* key, char* property, size_t buf_size)

Parameter	Description
`key`	The key of the property.
`property`	The pointer to the property.
`buf_size`	The size of the buffer.

getName

Retrieves the vendor name. You need to set the VENDOR_NAME in the return value of this method.

virtual const char * getName() const = 0;

getPreferredSampleRate

Retrieves the preferred sample rate of the audio filter.

This method is optional. If you specify a sample rate in the return value of this method, the SDK resamples the audio data accordingly before sending it to the audio filter.

virtual int getPreferredSampleRate()  { return 0; };

getPreferredChannelNumbers

Retrieves the preferred number of channels of the audio filter.

This method is optional. If you specify a number in the return value of this method, the SDK resamples the audio data accordingly before sending it to the audio filter.

virtual int getPreferredChannelNumbers()  { return 0; };

IExtensionProvider

Encapsulate your IAudioFilter implementation into an extension.

Methods include:

enumerateExtensions
createAudioFilter
setExtensionControl

enumerateExtensions

Enumerates your extensions that can be encapsulated. The SDK triggers this callback when loading the extension. In the callback, you need to return information about all of your extensions that can be encapsulated.

virtual void enumerateExtensions(ExtensionMetaInfo* extension_list,
                                 int& extension_count) {
    (void) extension_list;
    extension_count = 0;
}

Parameter	Description
`extension_list`	Extension information, including extension type and name. For details, see the definition of ExtensionMetaInfo.
`extension_count`	The total number of the extensions that can be encapsulated.

The definition of ExtensionExtensionMetaInfo is as follows:

// EXTENSION_TYPE represents where the extension is located in the media transmission pipeline
enum EXTENSION_TYPE {
    // Audio processing filter
    AUDIO_FILTER,
    // Video preprocessing filter
    VIDEO_PRE_PROCESSING_FILTER,
    // Video postprocessing filter
    VIDEO_POST_PROCESSING_FILTER,
    // Reserved for future use
    AUDIO_SINK,
    // Reserved for future use
    VIDEO_SINK,
    // Reserved for future use
    UNKNOWN,
};
// Extension information, including extension type and name
struct ExtensionMetaInfo {
    EXTENSION_TYPE type;
    const char* extension_name;
};

If you specify AUDIO_FILTER as EXTENSION_TYPE, after the customer creates the IExtensionProvider object when initializing RtcEngine, the SDK calls the createAudioFilter method, and you need to return the IAudioFilter instance in this method.

createAudioFilter

Creates an audio filter. You need to pass the IAudioFilter instance in this method.

virtual agora_refptr<IAudioFilter> createAudioFilter()

After creating an audio filter object, the extension processes the input audio frames with methods in IAudioFilter.

setExtensionControl

Sets the extension control.

virtual void setExtensionControl(IExtensionControl* control)

After calling this method, you need to maintain the IExtensionControl object returned in this method. The IExtensionControl object manages the interaction between the extension and the SDK by triggering callbacks and sending logs. For example, if you have called fireEvent in IExtensionControl:

void ByteDanceProcessor::dataCallback(const char* data){
    if (control_ != nullptr) {
        control_->fireEvent(id_, "beauty", data);
    }
}

And if the app registers the IMediaExtensionObserver class when initializing RtcEngine, the SDK triggers the following callback on the app client:

@Override
public void onEvent(String vendor, String key, String value) {
...
}