Skip to main content
Android
iOS
macOS
Web
Windows
Electron
Flutter
React Native
React JS
Unity
Unreal Engine
Unreal (Blueprint)

Raw video processing

In certain scenarios, it may be necessary to process raw video captured through the camera and microphone to achieve desired functionality or enhance the user experience. Video SDK provides the capability to pre-process and post-process the captured video data, allowing for the implementation of custom playback effects.

Understand the tech

Video SDK enables you to pre-process the captured video frames before sending the data to the encoder or perform post-processing on the received video frames after sending the data to the decoder.

The following figure shows the video data processing flow in the SDK video module.

  • Position (2) corresponds to the onCaptureVideoFrame callback.
  • Position (3) corresponds to the onPreEncodeVideoFrame callback.
  • Position (4) corresponds to theonRenderVideoFrame callback.

Prerequisites

Ensure that you have implemented the SDK quickstart in your project.

Implement raw video processing

To implement raw video data functionality in your project, refer to the following steps:

  1. Before joining the channel, create an IVideoFrameObserver object and register the video observer by calling the registerVideoFrameObserver method.


    _1
    int ret = engine.registerVideoFrameObserver(iVideoFrameObserver);

  2. Implement the onCaptureVideoFrame and onRenderVideoFrame callbacks. After obtaining the video data, process it according to your specific scenario.


    _72
    private final IVideoFrameObserver iVideoFrameObserver = new IVideoFrameObserver() {
    _72
    @Override
    _72
    public boolean onCaptureVideoFrame(VideoFrame videoFrame) {
    _72
    Log.i(TAG, "OnEncodedVideoImageReceived" + Thread.currentThread().getName());
    _72
    if (isSnapshot) {
    _72
    isSnapshot = false;
    _72
    _72
    // Get the image bitmap
    _72
    VideoFrame.Buffer buffer = videoFrame.getBuffer();
    _72
    VideoFrame.I420Buffer i420Buffer = buffer.toI420();
    _72
    int width = i420Buffer.getWidth();
    _72
    int height = i420Buffer.getHeight();
    _72
    _72
    ByteBuffer bufferY = i420Buffer.getDataY();
    _72
    ByteBuffer bufferU = i420Buffer.getDataU();
    _72
    ByteBuffer bufferV = i420Buffer.getDataV();
    _72
    _72
    byte[] i420 = YUVUtils.toWrappedI420(bufferY, bufferU, bufferV, width, height);
    _72
    _72
    Bitmap bitmap = YUVUtils.NV21ToBitmap(getContext(),
    _72
    YUVUtils.I420ToNV21(i420, width, height),
    _72
    width,
    _72
    height);
    _72
    _72
    Matrix matrix = new Matrix();
    _72
    matrix.setRotate(270);
    _72
    // Rotate around the center
    _72
    Bitmap newBitmap = Bitmap.createBitmap(bitmap, 0, 0, width, height, matrix, false);
    _72
    // Save to file
    _72
    saveBitmap2Gallery(newBitmap);
    _72
    _72
    bitmap.recycle();
    _72
    i420Buffer.release();
    _72
    }
    _72
    return false;
    _72
    }
    _72
    _72
    @Override
    _72
    public boolean onScreenCaptureVideoFrame(VideoFrame videoFrame) {
    _72
    return false;
    _72
    }
    _72
    _72
    @Override
    _72
    public boolean onMediaPlayerVideoFrame(VideoFrame videoFrame, int i) {
    _72
    return false;
    _72
    }
    _72
    _72
    @Override
    _72
    public boolean onRenderVideoFrame(String s, int i, VideoFrame videoFrame) {
    _72
    return false;
    _72
    }
    _72
    _72
    @Override
    _72
    public int getVideoFrameProcessMode() {
    _72
    return 0;
    _72
    }
    _72
    _72
    @Override
    _72
    public int getVideoFormatPreference() {
    _72
    return 1;
    _72
    }
    _72
    _72
    @Override
    _72
    public int getRotationApplied() {
    _72
    return 0;
    _72
    }
    _72
    _72
    @Override
    _72
    public boolean getMirrorApplied() {
    _72
    return false;
    _72
    }
    _72
    };

    Caution

    When modifying parameters in a VideoFrame, ensure that the updated parameters match the actual video frame in the buffer. Mismatches may cause issues like unexpected rotation, distortion, or other visual problems in the local preview and the remote video.

Reference

This section contains content that completes the information on this page, or points you to documentation that explains other aspects to this product.

Video Calling