Skip to main content

You are looking at Video Calling v3.x Docs. The newest version is  Video Calling 4.x

Android
iOS
macOS
Windows C++
Windows C#
Unity
Flutter
React Native
Electron
Cocos Creator
Cocos2d-x

Custom Video Source and Renderer

Introduction

Generally, Agora SDKs use default video modules for capturing and rendering in real-time communications.

However, these default modules might not meet your development requirements, such as in the following scenarios:

  • Your app has its own video module.
  • You want to use a non-camera source, such as recorded screen data.
  • You need to process the captured video with a pre-processing library for functions such as image enhancement.
  • You need flexible device resource allocation to avoid conflicts with other services.

Agora provides a solution to enable a custom video source and/or renderer in the above scenarios. This article describes how to do so using the Agora Native SDK.

Before proceeding, ensure that you have implemented the basic real-time communication functions in your project. For details, see Start a Video Call or Start Live Interactive Video Streaming.

Sample project

Agora provides the following open-source sample projects on GitHub:

You can view the source code on Github or download the project to try it out.

Custom video source

The Agora Native SDK provides the following two modes for customizing the video source:

  • Push mode: In this mode, call the setExternalVideoSource method to specify the custom video source. After implementing video capture using the custom video source, call the pushVideoFrame method to send the captured video frames to the SDK.
  • Video frames captured in Push mode cannot be rendered by the SDK. If you capture video frames in Push mode and need to enable local preview, you must use a custom video renderer.
  • Switching in the channel from custom video capture by Push to SDK capture is not supported. To switch the video source directly, you must use the custom video capture by MediaIO. See How can I switch from custom video capture to SDK capture.
  • MediaIO mode: In this mode, call the setVideoSource method to specify the custom video source. Then call the consumeByteBufferFrame, consumeByteArrayFrame, or consumeTextureFrame method to retrieve the captured video frames and send them to the SDK.

Push mode

Refer to the following steps to customize the video source in your project:

  1. Before calling joinChannel, call setExternalVideoSource to specify the custom video source.
  2. Implement video capture and processing yourself using methods from outside the SDK. According to your app scenario, you can call AgoraVideoFrame before sending the captured video frames to the SDK. For example, you can set rotation as 180 to rotate the video frames by 180 degrees clockwise.
  3. Call pushExternalVideoFrame to send the video frames to the SDK for later use.

API call sequence

Refer to the following diagram to implement the custom video source.

If you are not sure whether your custom video source supports Texture encoding, call isTextureEncodeSupported to find out. Then use the returned result to set the useTexture parameter in the setExternalVideoSource method.

img

Video data transfer

The following diagram shows how the video data is transferred when you customize the video source in Push mode:

1607670382235

  • You need to implement the capture module yourself using methods from outside the SDK.
  • Captured video frames are sent to the SDK via the pushExternalVideoFrame method.

Code samples

The following code samples use the camera as the custom video source.

  1. Before joining a channel, call setExternalVideoSource to specify the custom video source.

_13
// Creates TextureView
_13
TextureView textureView = new TextureView(getContext());
_13
// Adds SurfaceTextureListener, which triggers the onSurfaceTextureAvailable callback if SurfaceTexture in TextureView is available
_13
textureView.setSurfaceTextureListener(this);
_13
// Adds TextureView to local video layout
_13
fl_local.addView(textureView, new FrameLayout.LayoutParams(ViewGroup.LayoutParams.MATCH_PARENT,
_13
ViewGroup.LayoutParams.MATCH_PARENT));
_13
_13
_13
// Specifies the custom video source
_13
engine.setExternalVideoSource(true, true, true);
_13
// Joins the channel
_13
int res = engine.joinChannel(accessToken, channelId, "Extra Optional Data", 0);

  1. Configure the video capture module, and implement your custom video source. The code sample uses the camera as the custom video source.

_39
// Triggers this callback if SurfaceTexture in TextureView is available
_39
public void onSurfaceTextureAvailable(SurfaceTexture surface, int width, int height) {
_39
Log.i(TAG, "onSurfaceTextureAvailable");
_39
mTextureDestroyed = false;
_39
mSurfaceWidth = width;
_39
mSurfaceHeight = height;
_39
mEglCore = new EglCore();
_39
mDummySurface = mEglCore.createOffscreenSurface(1, 1);
_39
mEglCore.makeCurrent(mDummySurface);
_39
mPreviewTexture = GlUtil.createTextureObject(GLES11Ext.GL_TEXTURE_EXTERNAL_OES);
_39
// Creates a SurfaceTexture object for camera preview
_39
mPreviewSurfaceTexture = new SurfaceTexture(mPreviewTexture);
_39
// Creates OnFrameAvailableListener using the Android API setOnFrameAvailableListener, which triggers the onFrameAvailable callback if there are new video frames for SurfaceTexture
_39
mPreviewSurfaceTexture.setOnFrameAvailableListener(this);
_39
mDrawSurface = mEglCore.createWindowSurface(surface);
_39
mProgram = new ProgramTextureOES();
_39
if (mCamera != null || mPreviewing) {
_39
Log.e(TAG, "Camera preview has been started");
_39
return;
_39
}
_39
try {
_39
// Enables the camera (the code sample uses Android's Camera class)
_39
mCamera = Camera.open(mFacing);
_39
// Sets the most suitable resolution for your app scenario
_39
Camera.Parameters parameters = mCamera.getParameters();
_39
parameters.setPreviewSize(DEFAULT_CAPTURE_WIDTH, DEFAULT_CAPTURE_HEIGHT);
_39
mCamera.setParameters(parameters);
_39
// Sets mPreviewSurfaceTexture as the SurfaceTexture object for camera preview
_39
mCamera.setPreviewTexture(mPreviewSurfaceTexture);
_39
// Enables the portrait mode for camera preview. To ensure that camera preview stays in the portrait mode, rotate the preview image by 90 degrees clockwise
_39
mCamera.setDisplayOrientation(90);
_39
// The camera starts capturing video frames and rendering them to the specified SurfaceView
_39
mCamera.startPreview();
_39
mPreviewing = true;
_39
}
_39
catch (IOException e) {
_39
e.printStackTrace();
_39
}
_39
}

  1. The onFrameAvailable callback (Android's method, see Android's help document) is triggered when new video frames appear in TextureView. The callback implements the following operations:
  • Renders the captured video frames using the custom renderer for later use in local view.
  • Calls pushExternalVideoFrame to send the captured video frames to the SDK.

_61
// The onFrameAvailable callback gets new video frames captured by SurfaceTexture
_61
// Renders the video frames using EGL for later use in local view
_61
// Calls pushExternalVideoFrame to send the video frames to the SDK
_61
public void onFrameAvailable(SurfaceTexture surfaceTexture) {
_61
if (mTextureDestroyed) {
_61
return;
_61
}
_61
_61
if (!mEglCore.isCurrent(mDrawSurface)) {
_61
mEglCore.makeCurrent(mDrawSurface);
_61
}
_61
// Calls updateTexImage() to update data to the OpenGL ES texture object
_61
// Calls getTransformMatrix() to transform the texture's matrix
_61
try {
_61
mPreviewSurfaceTexture.updateTexImage();
_61
mPreviewSurfaceTexture.getTransformMatrix(mTransform);
_61
}
_61
catch (Exception e) {
_61
e.printStackTrace();
_61
}
_61
// Configures MVP matrix
_61
if (!mMVPMatrixInit) {
_61
// This code sample defines activity as the portrait mode. Since the captured video frames are rotated by 90 degrees, you need to switch the width and height data when calculating the frame ratio.
_61
float frameRatio = DEFAULT_CAPTURE_HEIGHT / (float) DEFAULT_CAPTURE_WIDTH;
_61
float surfaceRatio = mSurfaceWidth / (float) mSurfaceHeight;
_61
Matrix.setIdentityM(mMVPMatrix, 0);
_61
_61
if (frameRatio >= surfaceRatio) {
_61
float w = DEFAULT_CAPTURE_WIDTH * surfaceRatio;
_61
float scaleW = DEFAULT_CAPTURE_HEIGHT / w;
_61
Matrix.scaleM(mMVPMatrix, 0, scaleW, 1, 1);
_61
} else {
_61
float h = DEFAULT_CAPTURE_HEIGHT / surfaceRatio;
_61
float scaleH = DEFAULT_CAPTURE_WIDTH / h;
_61
Matrix.scaleM(mMVPMatrix, 0, 1, scaleH, 1);
_61
}
_61
mMVPMatrixInit = true;
_61
}
_61
// Sets the size of the video view
_61
GLES20.glViewport(0, 0, mSurfaceWidth, mSurfaceHeight);
_61
// Draws video frames
_61
mProgram.drawFrame(mPreviewTexture, mTransform, mMVPMatrix);
_61
// Sends the buffer of EGL image to EGL Surface for local playback and preview. mDrawSurface is an object of the EGLSurface class.
_61
mEglCore.swapBuffers(mDrawSurface);
_61
_61
// If the user has joined the channel, configures the external video frames and sends them to the SDK.
_61
if (joined) {
_61
// Configures external video frames
_61
AgoraVideoFrame frame = new AgoraVideoFrame();
_61
frame.textureID = mPreviewTexture;
_61
frame.format = AgoraVideoFrame.FORMAT_TEXTURE_OES;
_61
frame.transform = mTransform;
_61
frame.stride = DEFAULT_CAPTURE_HEIGHT;
_61
frame.height = DEFAULT_CAPTURE_WIDTH;
_61
frame.eglContext14 = mEglCore.getEGLContext();
_61
frame.timeStamp = System.currentTimeMillis();
_61
// Sends external video frames to the SDK
_61
boolean a = engine.pushExternalVideoFrame(frame);
_61
Log.e(TAG, "pushExternalVideoFrame:" + a);
_61
}
_61
}

API reference

MediaIO mode

In MediaIO mode, the Agora SDK provides the IVideoSource interface and the IVideoFrameConsumer class to configure the format of captured video frames and control the process of video capturing.

Refer to the following steps to customize the video source in your project:

  1. Implement the IVideoSource interface, which configures the format of captured video frames and controls the process of video capturing through a set of callbacks:

    • After receiving the getBufferType callback, specify the format of the captured video frames in the return value.
    • After receiving the onInitialize callback, save the IVideoFrameConsumer object, which sends and receives video frames captured by a custom source.
    • After receiving the onStart callback, start sending the captured video frames to the SDK by calling the consumeByteBufferFrame, consumeByteArrayFrame, or consumeTextureFrame method in the IVideoFrameConsumer object. Before sending the video frames, you can modify the video frame parameters in IVideoFrameConsumer, such as rotation, according to your app scenario.
    • After receiving the onStop callback, stop the IVideoFrameConsumer object from sending video frames to the SDK.
    • After receiving the onDispose callback, release the IVideoFrameConsumer object.
  2. Inherit the IVideoSource class implemented in step 1, and construct an object for the custom video source.

  3. Call the setVideoSource method to assign the custom video source object to RtcEngine.

  4. According to your app scenario, call the startPreview or joinChannel method to preview or publish the captured video frames.

API call sequence

Refer to the following diagram to implement the custom video source:

1607670515750

Video data transfer

The following diagram shows how the video data is transferred when you customize the video source in MediaIO mode:

1607670413195

  • You need to implement the capture module yourself using methods from outside the SDK.
  • Captured video frames are sent to the SDK via the consumeByteBufferFrame, consumeByteArrayFrame, or consumeTextureFrame method.

Code samples

The following code samples use a local video file as the custom video source.

  1. Implement the IVideoSource interface and the IVideoFrameConsumer class, and rewrite the callbacks in the IVideoSource interface.

_47
// Implements the IVideoSource interface
_47
public class ExternalVideoInputManager implements IVideoSource {
_47
_47
...
_47
_47
// Gets the IVideoFrameConsumer object from this callback when initializing the video source
_47
@Override
_47
public boolean onInitialize(IVideoFrameConsumer consumer) {
_47
mConsumer = consumer;
_47
return true;
_47
}
_47
_47
@Override
_47
public boolean onStart() {
_47
return true;
_47
}
_47
_47
@Override
_47
public void onStop() {
_47
_47
}
_47
_47
// Sets the IVideoFrameConsumer object as null when media engine releases IVideoFrameConsumer
_47
@Override
_47
public void onDispose() {
_47
Log.e(TAG, "SwitchExternalVideo-onDispose");
_47
mConsumer = null;
_47
}
_47
_47
@Override
_47
public int getBufferType() {
_47
return TEXTURE.intValue();
_47
}
_47
_47
@Override
_47
public int getCaptureType() {
_47
return CAMERA;
_47
}
_47
_47
@Override
_47
public int getContentHint() {
_47
return MediaIO.ContentHint.NONE.intValue();
_47
}
_47
_47
...
_47
_47
}


_2
// Implements the IVideoFrameConsumer class
_2
private volatile IVideoFrameConsumer mConsumer;

  1. Specify the custom video source before joining a channel.

_2
// Specifies the custom video source
_2
ENGINE.setVideoSource(ExternalVideoInputManager.this);

  1. Configure external video input.

_15
// Creates an intent for local video input, sets video frame parameters, and sets external video input
_15
// Calls setExternalVideoInput to create a new LocalVideoInput object which retrieves the location of the local video file
_15
// The setExternalVideoInput method also sets a Surface Texture monitor for TextureView
_15
// Adds TextureView to subviews in relative layout for local preview
_15
Intent intent = new Intent();
_15
setVideoConfig(ExternalVideoInputManager.TYPE_LOCAL_VIDEO, LOCAL_VIDEO_WIDTH, LOCAL_VIDEO_HEIGHT);
_15
intent.putExtra(ExternalVideoInputManager.FLAG_VIDEO_PATH, mLocalVideoPath);
_15
if (mService.setExternalVideoInput(ExternalVideoInputManager.TYPE_LOCAL_VIDEO, intent)) {
_15
// Deletes all subviews in relative layout
_15
fl_local.removeAllViews();
_15
// Adds TextureView as a subview
_15
fl_local.addView(TEXTUREVIEW,
_15
RelativeLayout.LayoutParams.MATCH_PARENT,
_15
RelativeLayout.LayoutParams.MATCH_PARENT);
_15
}

Refer to the following code sample to implement the setExternalVideoInput method:


_25
// Implements the setExternalVideoInput method
_25
boolean setExternalVideoInput(int type, Intent intent) {
_25
_25
if (mCurInputType == type && mCurVideoInput != null
_25
&& mCurVideoInput.isRunning()) {
_25
return false;
_25
}
_25
// Creates a new LocalVideoInput object which retrieves the location of the local video file
_25
IExternalVideoInput input;
_25
switch (type) {
_25
case TYPE_LOCAL_VIDEO:
_25
input = new LocalVideoInput(intent.getStringExtra(FLAG_VIDEO_PATH));
_25
// If TextureView is not null, sets a Surface Texture monitor for this TextureView
_25
if (TEXTUREVIEW != null) {
_25
TEXTUREVIEW.setSurfaceTextureListener((LocalVideoInput) input);
_25
}
_25
break;
_25
_25
...
_25
}
_25
// Sets the created LocalVideoInput object as the video source
_25
setExternalVideoInput(input);
_25
mCurInputType = type;
_25
return true;
_25
}

  1. Implement the local video thread, and decode the local video file. The decoded video frames are rendered to Surface.

_5
// Decodes the local video file and renders it to Surface
_5
LocalVideoThread(String filePath, Surface surface) {
_5
initMedia(filePath);
_5
mSurface = surface;
_5
}

  1. After the local user joins the channel, the capture module consumes the video frames through the consumeTextureFrame method in ExternalVideoInputThread and sends the frames to the SDK.

_36
public void run() {
_36
_36
...
_36
// Calls updateTexImage() to update data to the OpenGL ES texture object
_36
// Calls getTransformMatrix() to transform the texture's matrix
_36
try {
_36
mSurfaceTexture.updateTexImage();
_36
mSurfaceTexture.getTransformMatrix(mTransform);
_36
}
_36
catch (Exception e) {
_36
e.printStackTrace();
_36
}
_36
// Gets the captured video frames through the onFrameAvailable callback. This callback is rewritten in the LocalVideoInput class based on Android's corresponding API.
_36
// The onFrameAvailable callback creates EGL Surface through the SurfaceTexture for local preview and uses its context as the current context. You can use this callback to render video frames locally, get Texture ID and transform information, and send the video frames to the SDK.
_36
if (mCurVideoInput != null) {
_36
mCurVideoInput.onFrameAvailable(mThreadContext, mTextureId, mTransform);
_36
}
_36
// Links EGLSurface
_36
mEglCore.makeCurrent(mEglSurface);
_36
// Sets the EGL video view
_36
GLES20.glViewport(0, 0, mVideoWidth, mVideoHeight);
_36
_36
if (mConsumer != null) {
_36
Log.e(TAG, "Width and Height ->width:" + mVideoWidth + ",height:" + mVideoHeight);
_36
// Calls consumeTextureFrame to consume the video frames and send them to the SDK
_36
mConsumer.consumeTextureFrame(mTextureId,
_36
TEXTURE_OES.intValue(),
_36
mVideoWidth, mVideoHeight, 0,
_36
System.currentTimeMillis(), mTransform);
_36
}
_36
_36
// Waiting for the next frame
_36
waitForNextFrame();
_36
...
_36
_36
}

API reference

See also

If your app has its own video capture module and needs to integrate the Agora SDK for real-time communication purposes, you can use the Agora Component to enable and disable video frame input through the callbacks in Media Engine. For details, see Customize the Video Source with the Agora Component.

Custom video renderer

The Agora SDK provides the IVideoSink interface to customize the video renderer in your project.

Refer to the following steps to implement the video renderer:

  1. Implement the IVideoSink interface, which configures the format of captured video frames and controls the process of video rendering through a set of callbacks:

    • After receiving the getBufferType and getPixelFormat callbacks, specify the format of the rendered video frames in the return value.
    • After receiving the onInitialize, onStart, onStop, onDispose, and getEglContextHandle callbacks, perform the corresponding operations.
    • Implement the IVideoFrameConsumer class for the rendered video frames' format to retrieve the video frames.
  2. Inherit the IVideoSource class implemented in step 1, and create a video capture module for the custom renderer.

  3. Call the setLocalVideoRenderer or setRemoteVideoRenderer method to render the video of the local user or remote user.

  4. According to your app scenario, call the startPreview or joinChannel method to preview or publish the rendered video frames.

API call sequence

Refer to the following diagram to implement the custom video renderer in MediaIO mode:

img

Video data transfer

The following diagram shows how the video data is transferred when you customize the video renderer in MediaIO mode:

1607670404631

  • You need to implement the rendering module yourself using methods from outside the SDK.
  • Captured video frames are sent to the capture module via the consumeByteBufferFrame, consumeByteArrayFrame, or consumeTextureFrame method.

Code samples

The code samples provide two options for implementing the custom video renderer in your project.

Option 1: Use components provided by Agora

The Agora SDK provides classes and code samples that are designed to help you easily integrate and create a custom video renderer. You can use these components directly, or you can create a custom renderer based on these components. See Customize the Video Sink with the Agora Component.

After the local user joins the channel, import and implement the AgoraSurfaceView class, then set the remote video renderer. The AgoraSurfaceView class inherits the SurfaceView class and implements the IVideoSink class. AgoraSurfaceView also embeds a BaseVideoRenderer object that serves as the rendering module, which means you do not need to implement the IVideoSink class and customize the rendering module yourself. The BaseVideoRenderer object uses OpenGL as the renderer and creates EGLContext, and it shares the Handle of EGLContext with Media Engine. For more information about how to implement the AgoraSurfaceView class, see the demo project.


_26
@Override
_26
public void onUserJoined(int uid, int elapsed) {
_26
super.onUserJoined(uid, elapsed);
_26
Log.i(TAG, "onUserJoined->" + uid);
_26
showLongToast(String.format("user %d joined!", uid));
_26
Context context = getContext();
_26
if (context == null) {
_26
return;
_26
}
_26
handler.post(() ->
_26
{
_26
// Implements the AgoraSurfaceView class
_26
AgoraSurfaceView surfaceView = new AgoraSurfaceView(getContext());
_26
surfaceView.init(null);
_26
surfaceView.setZOrderMediaOverlay(true);
_26
// Calls the setBufferType and setPixelFormat methods in the embedded BaseVideoRenderer object to set the type and format of the video frames
_26
surfaceView.setBufferType(MediaIO.BufferType.BYTE_BUFFER);
_26
surfaceView.setPixelFormat(MediaIO.PixelFormat.I420);
_26
if (fl_remote.getChildCount() > 0) {
_26
fl_remote.removeAllViews();
_26
}
_26
fl_remote.addView(surfaceView);
_26
// Sets the remote video renderer
_26
engine.setRemoteVideoRenderer(uid, surfaceView);
_26
});
_26
}

Option 2: Use the IVideoSink interface

You can implement the IVideoSink class and inherit it to construct a rendering module for the custom renderer.


_67
// Creates an instance to implement the IVideoSink interface
_67
IVideoSink sink = new IVideoSink() {
_67
@Override
_67
// Initializes the renderer either in this method or beforehand. Sets the return value as true to indicate that the renderer has initialized
_67
public boolean onInitialize () {
_67
return true;
_67
}
_67
_67
@Override
_67
// Starts the renderer
_67
public boolean onStart() {
_67
return true;
_67
}
_67
_67
@Override
_67
// Stops the renderer
_67
public void onStop() {
_67
_67
}
_67
_67
@Override
_67
// Releases the renderer
_67
public void onDispose() {
_67
_67
}
_67
_67
@Override
_67
public long getEGLContextHandle() {
_67
// Constructs your Egl context
_67
// If the return value is 0, no Egl context has been created in the renderer
_67
return 0;
_67
}
_67
_67
// Returns the Buffer type the renderer requires
_67
// If you want to switch to a different VideoSink type, you must create another instance
_67
// There are three Buffer types: BYTE_BUFFER(1), BYTE_ARRAY(2), and TEXTURE(3)
_67
@Override
_67
public int getBufferType() {
_67
return BufferType.BYTE_ARRAY;
_67
}
_67
_67
// Returns the Pixel format the renderer requires
_67
@Override
_67
public int getPixelFormat() {
_67
return PixelFormat.NV21;
_67
}
_67
_67
// SDK calls this method to send the captured video frames to the renderer
_67
// Use the corresponding callback for the format of the captured video frames
_67
@Override
_67
public void consumeByteArrayFrame(byte[] data, int format, int width, int height, int rotation, long timestamp) {
_67
_67
// The renderer is working
_67
}
_67
public void consumeByteBufferFrame(ByteBuffer buffer, int format, int width, int height, int rotation, long timestamp) {
_67
_67
_67
// The renderer is working
_67
}
_67
public void consumeTextureFrame(int textureId, int format, int width, int height, int rotation, long timestamp, float[] matrix) {
_67
_67
// The renderer is working
_67
}
_67
_67
}
_67
_67
rtcEngine.setLocalVideoRenderer(sink);

API reference

Considerations

  • Performing the following operations requires you to use methods from outside the Agora SDK:

    • Manage the capture and processing of video frames when using a custom video source.
    • Manage the processing and display of video frames when using a custom video renderer.
  • When using a custom video renderer, if the consumeByteArrayFrame, consumeByteBufferFrame, or consumeTextureFrame callback reports that rotation is not 0, the rendered video frames are rotated by a certain degree. This may be caused by the capture settings of the SDK or your custom video source. You need to modify rotation according to your application scenario.

  • If the format of the custom captured video is Texture and the remote user sees anomalies (such as flickering and distortion) in the local custom captured video, Agora recommends that you make a copy of the video data before sending the custom video data back to the SDK, and then send both the original video data and the copied video data back to the SDK. This eliminates the anomalies during the internal data encoding.

See also

If you want to customize the audio source or renderer in your project, see Custom Audio Source and Renderer.

Video Calling