Audio extension

The audio filters you created are easily integrated into apps to supply your voice effects and noise cancellation.

An audio filter accesses voice data when it is captured from the user's local device, modifies it, then plays the updated data to local and remote video channels.

A typical transmission pipeline consists of a chain of procedures, including capture, pre-processing, encoding, transmitting, decoding, post-processing, and play. In order to modify the voice or video data in the transmission pipeline, audio extensions are inserted into either the pre-processing or post-processing procedure.

In order to follow this procedure you must have:

• Android Studio 4.1 or higher.
• Android SDK API Level 24 or higher.
• A mobile device that runs Android 4.1 or higher.

• A project to develop in.

In order to integrate an extension into your project:

To integrate into your project:

1. Unzip Video SDK to a local directory.
2. Copy the header files in rtc/sdk/low_level_api/include under the directory of your project file.

You are now ready to develop your extension.

To create an extension in your Agora project, use the following APIs:

• IAudioFilter: Implement receiving, processing, and delivering audio data.
• IExtensionProvider: encapsulate your IAudioFilter implementation into an extension.

Use the IAudioFilter interface to implement an audio filter. You can find the interface in the NGIAgoraMediaNode.h file. You need to implement this interface first, and you must implement at least the following methods from this interface:

• adaptAudioFrame
• setEnabled
• isEnabled
• setProperty
• getProperty
• getName
• getPreferredSampleRate(Optional)
• getPreferredChannelNumbers(Optional)

The following code sample shows how to use these methods together to implement an audio filter extension:

// After receiving the audio frames to be processed, call adaptAudioFrame to process the audio frames.

bool ExtensionAudioFilter::adaptAudioFrame(const media::base::AudioPcmFrame &inAudioPcmFrame,

media::base::AudioPcmFrame &adaptedPcmFrame) {

return audioProcess_->processFrame(inAudioPcmFrame, adaptedPcmFrame) == 0;

}

// Call setProperty to set the property of the audio filter.

int ExtensionAudioFilter::setProperty(const char* key, const void* buf, int buf_size) {

std::string str_volume = "100";

if (std::string(key) == "volume") {

str_volume = std::string(static_cast<const char*>(buf), buf_size);

}

int int volume_ = atoi(str_volume.c_str());

audioProcessor_->setVolume(int_volume_);

return ERR_OK;

}

// Call getProperty to get the property of the audio filter.

int ExtensionAudioFilter::getProperty(const char* key, void* buf, int buf_size) const override {return ERR_OK; }

// Call setEnabled to enable the audio filter.

void ExtensionAudioFilter::setEnabled(bool enable) override { enabled_ = enable; }

// Call isEnabled to check whether the audio filter is enabled.

bool ExtensionAudioFilter::isEnabled() const override {return enabled_; }

// Set the vendor name in the return value of getName.

const char* getName() const override { return filterName_.c_str(); }

// (Optional) Specify the audio sample rate in the return value of getPreferredSampleRate

const char* ExtensionAudioFilter::getPreferredSampleRate() override { return 48000; }

// (Optional) Specify the number of audio channels in the return value of getPreferredChannelNumbers

int ExtensionAudioFilter::getPreferredChannelNumbers() override { return 2; };

To encapsulate the audio filter into an extension, you need to implement the IExtensionProvider interface. You can find the interface in the NGIAgoraExtensionProvider.h file. The following methods from this interface must be implemented:

• enumerateExtensions
• createAudioFilter
• setExtensionControl

The following code sample shows how to use these methods to encapsulate the audio filter:

void ExtensionProvider::enumerateExtensions(ExtensionMetaInfo* extension_list,

int& extension_count) {

extension_count = 2;

//Declare a Video Filter, and IExtensionProvider::createVideoFilter will be called

ExtensionMetaInfo i;

i.type = EXTENSION_TYPE::VIDEO_PRE_PROCESSING_FILTER;

i.extension_name = agora::extension::VIDEO_FILTER_NAME;

extension_list[0] = i;

//Declare an Audio Filter, and IExtensionProvider::createAudioFilter will be called

ExtensionMetaInfo j;

j.type = EXTENSION_TYPE::AUDIO_FILTER;

j.extension_name = agora::extension::AUDIO_FILTER_NAME;

extension_list[1] = j;

}

agora_refptr<agora::rtc::IAudioFilter> ExtensionProvider::createAudioFilter(const char* name) {

PRINTF_INFO("ExtensionProvider::createAudioFilter %s", name);

auto audioFilter = new agora::RefCountedObject<agora::extension::ExtensionAudioFilter>(name, audioProcessor_);

return audioFilter;

}

void ExtensionAudioProvider::setExtensionControl(rtc::IExtensionControl* control){

audioProcessor_->setExtensionControl(control);

}

After encapsulating the filter into an extension, you need to register and package it into a .aar or .so file, and submit it together with a file that contains the extension name, vendor name and filter name to Agora.

1. Register the extension

   Register the extension with the macro REGISTER_AGORA_EXTENSION_PROVIDER, which is in the AgoraExtensionProviderEntry.h file. Use this macro at the entrance of the extension implementation. When the SDK loads the extension, this macro automatically registers it to the SDK. For example:

   REGISTER_AGORA_EXTENSION_PROVIDER(ByteDance, agora::extension::ExtensionProvider);
   

2. Link the libagora-rtc-sdk-jni.so file

   In CMakeLists.txt, specify the path to save the libagora-rtc-sdk-jni.so file in the downloaded SDK package according to the following table:

   File	Path
   64-bit libagora-rtc-sdk-jni.so	AgoraWithByteDanceAndroid/agora-bytedance/src/main/agoraLibs/arm64-v8a
   32-bit libagora-rtc-sdk-jni.so	AgoraWithByteDanceAndroid/agora-bytedance/src/main/agoraLibs/arm64-v7a

3. Provide extension information

   Create a .java or .md file to provide the following information:

   • EXTENSION_NAME: The name of the target link library used in CMakeLists.txt. For example, for a .so file named libagora-bytedance.so, the EXTENSION_NAME should be agora-bytedance.
   • EXTENSION_VENDOR_NAME: The name of the extension provider, which is used for registering in the agora-bytedance.cpp file.
   • EXTENSION_FILTER_NAME: The name of the filter, which is defined in ExtensionProvider.h.

To ensure that you have integrated the extension in your app:

Once you have developed your extension and API endpoints, the next step is to test whether they work properly.

• Functional and performance tests

  Test the functionality and performance of your extension and submit a test report to Agora. This report must contain:

  • The following proof of functionality:
    • The extension is enabled and loaded in the SDK normally.
    • All key-value pairs in the setExtensionProperty or setExtensionPropertyWithVendor method work properly.
    • All event callbacks of your extension work properly through IMediaExtensionObserver.
  • The following performance data:
    • The average time the extension needs to process an audio or video frame.
    • The maximum amount of memory required by the extension.
    • The maximum amount of CPU/GPU consumption required by the extension.

• Extension listing test The Extensions Marketplace is where developers discover your extension. In the Marketplace, each extension has a product listing that provides detailed information such as feature overview and implementation guides. Before making your extension listing publicly accessible, the best practice is to see how everything looks and try every function in a test environment.

• Write the integration document for your extension

  The easier it is for other developers to integrate your extension, the more it will be used. Follow the guidelines and create the best Integration guide for your extension

• Apply for testing

  To apply for access to the test environment, contact Agora and provide the following:

  • Your extension package
  • Extension listing assets, including:
  • Your company name
  • Your public email address
  • The Provisioning API endpoints
  • The Usage and Billing API endpoints
  • Your draft business model or pricing plan
  • Your support page URL
  • Your official website URL
  • Your implementation guides URL

• Test your extension listing

  Once your application is approved, Agora publishes your extension in the test environment and sends you an e-mail.

  To test if everything works properly with your extension in the Marketplace, do the following:

  • Activate and deactivate your extension in an Agora project, and see whether the Provisioning APIs work properly.
  • Follow your implementation guides to implement your extension in an Agora project, and see whether you need to update your documentation.
  • By the end of the month, check the billing information and see whether the Usage and Billing APIs work properly.

Now you are ready to submit your extension for final review by Agora. You can now Publish Your Extension.

This section contains content that completes the information on this page, or points you to documentation that explains other aspects to this product.

Agora provides an Android sample project agora-simple-filter for developing audio and video filter extensions.

The classes used to create and encapsulate filters are:

• IAudioFilter: Implement receiving, processing, and delivering audio data.
• IExtensionProvider: encapsulate your IAudioFilter implementation into an extension.

Implement receiving, processing, and delivering audio data.

Methods include:

• adaptAudioFrame
• setEnabled
• isEnabled
• setProperty
• getProperty
• getName
• getPreferredSampleRate(Optional)
• getPreferredChannelNumbers(Optional)

Adapts the audio frame. This is the core method of the IAudioFilter interface. By calling this method, the SDK processes audio frames from inAudioFrame and returns the adapted frames with adaptedFrame. This method supports audio data in the PCM format only.

virtual bool adaptAudioFrame(const media::base::AudioPcmFrame& inAudioFrame,

media::base::AudioPcmFrame& adaptedFrame) = 0;

Parameter	Description
inAudioFrame	An input parameter. The pointer to the audio frames to be processed.
adaptedFrame	An output parameter. The pointer to the processed audio frames.

Enables or disables the audio filter.

virtual void setEnabled(bool enable) {}

Parameter	Description
enable	Whether to enable the audio filter： true: Enable the audio filter. false: (Default) Disable the audio filter.

Checks whether the audio filter is enabled.

virtual bool isEnabled() { return true; }

Returns Whether the audio filter is enabled:

• true: The audio filter is enabled.
• false: (Default) The audio filter is disabled.

Sets the property of the audio filter. When an app client calls setExtensionProperty, the SDK triggers this callback. In the callback, you need to return the property of the audio filter.

size_t setProperty(const char* key, const void* buf, size_t buf_size)

Parameter	Description
key	The key of the property.
buf	The buffer of the property in the JSON format. You can use the open source nlohmann/json library for the serialization and deserialization between the C++ struct and the JSON string.
buf_size	The size of the buffer.

Gets the property of the audio filter. When the app client calls getExtensionProperty, the SDK calls this method to get the property of the audio filter.

size_t getProperty(const char* key, char* property, size_t buf_size)

Parameter	Description
key	The key of the property.
property	The pointer to the property.
buf_size	The size of the buffer.

Retrieves the vendor name. You need to set the VENDOR_NAME in the return value of this method.

virtual const char * getName() const = 0;

Retrieves the preferred sample rate of the audio filter.

This method is optional. If you specify a sample rate in the return value of this method, the SDK resamples the audio data accordingly before sending it to the audio filter.

virtual int getPreferredSampleRate() { return 0; };

Retrieves the preferred number of channels of the audio filter.

This method is optional. If you specify a number in the return value of this method, the SDK resamples the audio data accordingly before sending it to the audio filter.

virtual int getPreferredChannelNumbers() { return 0; };

Encapsulate your IAudioFilter implementation into an extension.

Methods include:

• enumerateExtensions
• createAudioFilter
• setExtensionControl

Enumerates your extensions that can be encapsulated. The SDK triggers this callback when loading the extension. In the callback, you need to return information about all of your extensions that can be encapsulated.

virtual void enumerateExtensions(ExtensionMetaInfo* extension_list,

int& extension_count) {

(void) extension_list;

extension_count = 0;

}

Parameter	Description
extension_list	Extension information, including extension type and name. For details, see the definition of ExtensionMetaInfo.
extension_count	The total number of the extensions that can be encapsulated.

The definition of ExtensionExtensionMetaInfo is as follows:

// EXTENSION_TYPE represents where the extension is located in the media transmission pipeline

enum EXTENSION_TYPE {

// Audio processing filter

AUDIO_FILTER,

// Video preprocessing filter

VIDEO_PRE_PROCESSING_FILTER,

// Video postprocessing filter

VIDEO_POST_PROCESSING_FILTER,

// Reserved for future use

AUDIO_SINK,

// Reserved for future use

VIDEO_SINK,

// Reserved for future use

UNKNOWN,

};

// Extension information, including extension type and name

struct ExtensionMetaInfo {

EXTENSION_TYPE type;

const char* extension_name;

};

If you specify AUDIO_FILTER as EXTENSION_TYPE, after the customer creates the IExtensionProvider object when initializing RtcEngine, the SDK calls the createAudioFilter method, and you need to return the IAudioFilter instance in this method.

Creates an audio filter. You need to pass the IAudioFilter instance in this method.

virtual agora_refptr<IAudioFilter> createAudioFilter()

After creating an audio filter object, the extension processes the input audio frames with methods in IAudioFilter.

Sets the extension control.

virtual void setExtensionControl(IExtensionControl* control)

After calling this method, you need to maintain the IExtensionControl object returned in this method. The IExtensionControl object manages the interaction between the extension and the SDK by triggering callbacks and sending logs. For example, if you have called fireEvent in IExtensionControl:

void ByteDanceProcessor::dataCallback(const char* data){

if (control_ != nullptr) {

control_->fireEvent(id_, "beauty", data);

}

}

And if the app registers the IMediaExtensionObserver class when initializing RtcEngine, the SDK triggers the following callback on the app client:

@Override

public void onEvent(String vendor, String key, String value) {

...

}