Virtual Background

Virtual Background enables users to blur their background, or replace it with a solid color or an image. This feature is applicable to scenarios such as online conferences, online classes, and live streaming. It helps protect personal privacy and reduces audience distraction.

Try out the online demo for Virtual background.

Virtual Background offers the following options:

Feature	Example
Blurred background and image background
Video/Animated background
Portrait-in-picture	Allows the presenter to use slides as the virtual background while superimposing their video. The effect is similar to a weather news cast on television, preventing interruptions during a layout toggle.

Want to test Virtual Background? Try the online demo.

A typical transmission pipeline in the Agora Web SDK consists of a chain of procedures, including capture, preprocessing, encoding, transmitting, decoding, post-processing, and playback. In the preprocessing stage, extensions can modify the audio and video data in the pipeline to implement features such as virtual background and noise cancellation.

Ensure that you have implemented the SDK quickstart in your project.

This section shows you how to add a virtual background to the local video.

Copy the following code into your script file:

Complete sample code for virtual background

// Create Clientvar client = AgoraRTC.createClient({mode: "rtc", codec: "vp8"});// Create VirtualBackgroundExtension instanceconst extension = new VirtualBackgroundExtension();// Register pluginAgoraRTC.registerExtensions([extension]);let processor = null;var localTracks = {  videoTrack: null,  audioTrack: null};// Initializeasync function getProcessorInstance() {  if (!processor && localTracks.videoTrack) {    // Create VirtualBackgroundProcessor instance    processor = extension.createProcessor();    try {    // Initialize plugin    await processor.init();    } catch(e) {      console.log("Fail to load WASM resource!");      return null;    }    // Inject plugin into SDK's video processing pipeline    localTracks.videoTrack.pipe(processor).pipe(localTracks.videoTrack.processorDestination);  }  return processor;}// Join channelasync function join() {  // Add event listeners  client.on("user-published", handleUserPublished);  client.on("user-unpublished", handleUserUnpublished);  [options.uid, localTracks.audioTrack, localTracks.videoTrack] = await Promise.all([    // Join channel    client.join(options.appid, options.channel, options.token || null),    // Create local microphone and camera tracks    localTracks.audioTrack || AgoraRTC.createMicrophoneAudioTrack(),    localTracks.videoTrack || AgoraRTC.createCameraVideoTrack({encoderConfig: '720p_4'})  ]);  // Play local tracks  localTracks.videoTrack.play("local-player");}// Set solid color backgroundasync function setBackgroundColor() {  if (localTracks.videoTrack) {    document.getElementById("loading").style.display = "block";    let processor = await getProcessorInstance();    try {      processor.setOptions({type: 'color', color: '#00ff00'});      await processor.enable();    } finally {      document.getElementById("loading").style.display = "none";    }    virtualBackgroundEnabled = true;  }}// Blur user's actual backgroundasync function setBackgroundBlurring() {  if (localTracks.videoTrack) {    document.getElementById("loading").style.display = "block";    let processor = await getProcessorInstance();    try {      processor.setOptions({type: 'blur', blurDegree: 2});      await processor.enable();    } finally {      document.getElementById("loading").style.display = "none";    }    virtualBackgroundEnabled = true;  }}// Set image backgroundasync function setBackgroundImage() {    const imgElement = document.createElement('img');    imgElement.onload = async() => {      document.getElementById("loading").style.display = "block";      let processor = await getProcessorInstance();      try {        processor.setOptions({type: 'img', source: imgElement});        await processor.enable();      } finally {        document.getElementById("loading").style.display = "none";      }      virtualBackgroundEnabled = true;    }    imgElement.src = '/images/background.png';}

Run the following comment in the root directory of your project:

npm install agora-extension-virtual-background

You can add the extension in your app in the following ways:

1. Add the following code to the script file:

   import VirtualBackgroundExtension from "agora-extension-virtual-background";
   

2. Add the following code to the html file:

   <script src="./agora-extension-virtual-background.js"></script>
   

Register the extension instance with the Video SDK. To do this, add the following code to the script file:

// Create Client

var client = AgoraRTC.createClient({mode: "rtc", codec: "vp8"});

// Create VirtualBackgroundExtension instance

const extension = new VirtualBackgroundExtension();

// Check compatibility

if (!extension.checkCompatibility()) {

// The current browser does not support the virtual background plugin, you can stop executing the subsequent logic

console.error("Does not support Virtual Background!");

}

// Register plugin

AgoraRTC.registerExtensions([extension]);

1. Call extension.createProcessor to create a VirtualBackgroundProcessor instance.

   processor = extension.createProcessor();
   

2. Call processor.init to initialize the plugin. When resource loading or plug-in initialization fails, this method will throw an exception, and the application can catch the exception and handle it accordingly:

   await processor.init();
   

3. After creating the local camera video track, call videoTrack.pipe and specify the videoTrack.processorDestination attribute to inject the plug-in into the SDK's media processing pipeline.

   Info

   For computing performance reasons, Agora recommends using the virtual background plug-in on a single video track. If you need two video tracks for preview, create multiple instances of VirtualBackgroundProcessor.

   localTracks.videoTrack.pipe(processor).pipe(localTracks.videoTrack.processorDestination);
   

Call processor.setOptions to set the virtual background type and corresponding parameters. The following examples show how you use different backgrounds:

1. Set a solid color background: Set the type parameter to color, then set the color parameter to a color value:

   processor.setOptions({type: 'color', color: '#00ff00'});
   

2. Set the picture background: Set the type parameter to img, and then set the source parameter to HTMLImageElement:

   processor.setOptions({type: 'img', source: HTMLImageElement});
   

3. Blur the user's actual background: Set the type parameter to blur, and then set the blurDegree parameter to low (1), medium (2), or high (3):

   processor.setOptions({type: 'blur', blurDegree: 2});
   

4. Set a dynamic background: Set the type parameter to video, and then set the source parameter to a video object:

   processor.setOptions({type: 'video', source: HTMLVideoElement});
   

To enable a virtual background, call processor.enable:

await processor.enable();

If enable is not called before calling setOptions, the SDK will enable the background blur effect by default, with a blur level of 1. After turning on the virtual background, if you need to switch the virtual background effect, just adjust setOptions.

To disable the virtual background, call disable and unpipe:

await processor.disable();

To remove the processor from the local video track, call videoTrack.unpipe:

localTracks.videoTrack.unpipe();

info

If you need to enable the virtual background again, you can reuse the existing VirtualBackgroundProcessor instance and do not need to recreate it. If multiple instances of VirtualBackgroundProcessor are created, it is recommended to call the processor.release method to release the plug-in resources that are no longer needed.

This section contains content that completes the information in this page, or points you to documentation that explains other aspects to this product.

• For a working example, check out the Virtual background web demo and the associated source code.

• This extension works best when there is only one user in the video captured by the camera.
• The browser support for the virtual background extension is as follows:
  • To get a better virtual background experience, Agora recommends using this feature on the latest version of Desktop Chrome.
  • Agora does not recommend enabling the virtual background feature on Firefox and Safari browsers. Backgrounding the web app on Firefox may cause the video to freeze, while the performance on Safari could be poor due to the browser's own performance issues.
  • Agora does not recommend enabling the virtual background feature on mobile browsers.
• The virtual background feature has high performance requirements. Make sure your computer meets the following requirements:
  • CPU: Intel Core i5 4 cores or higher
  • 8G RAM or more
  • 64-bit operating system
• If multiple extensions are enabled concurrently and other programs are occupying high system resources at the same time, your app may experience audio and video freezes.
• When using this extension, Agora recommends selecting Performance mode or Balanced mode for your laptops. The computing requirements for this extension may not be met if the laptop is in a battery-saving mode.
• This extension supports using a video as a dynamic virtual background since v1.0.0-beta-3. Videos must be in a format supported by <video> HTML elements. Agora also recommends meeting the following requirements:
  • The video adopts a resolution close to that of a portrait. Agora recommends not using a video with too high a bit rate.
  • Video content is suitable for looping to achieve a more natural dynamic virtual background effect.
  • Properly reduce the video frame rate, such as 15 fps or below. Video above 30 fps is not recommended by Agora.

createProcessor(): IVirtualBackgroundProcessor;

Creates a VirtualBackgroundProcessor object.

init(wasmDir: string): Promise<void>;

Initializes the extension.

Parameters:

• wasmDir: The URL where the virtual background WASM module is located (without the WASM filename). Starting from v1.2.0, this parameter is optional.

If the initialization of the extension fails due to the failure to access the Wasm file, this method throws an exception. Agora recommends that you disable the virtual background feature catching the exception.

If the Wasm file is deployed on third-party services such as CDN and OSS across domains, you need to enable cross-domain access. For example, when deploying Nginx servers across domains, to enable cross-domain access, add the following configurations:

add_header 'Access-Control-Allow-Origin' "$http_origin";

add_header 'Access-Control-Allow-Credentials' "true";

setOptions(options: VirtualBackgroundEffectOptions): void;

Chooses the virtual background type, and sets parameters.

Parameters:

• options: The virtual background options. See VirtualBackgroundEffectOptions.

enable(): void | Promise<void>;

Enables the virtual background feature.

If you do not call setOptions before calling this method, the default behavior of the SDK is to blur users' actual background with the blurring degree set as 1.

disable(): void | Promise<void>;

Disables the virtual background feature.

release(): Promise<void>;

Releases all resources used by the extension, including created web workers .

If IVirtualBackgroundProcessor is repeatedly created without releasing the resources occupied by the extension, it may cause memory exhaustion.

onoverload?: () => void;

When the system performance cannot meet the processing requirements, the SDK triggers onoverload.

Agora recommends calling disable in this event to disable virtual background and adding a UI prompt.

Virtual background types and settings. Used in the setOptions method.

export type VirtualBackgroundEffectOptions = {

type: string,

color?: string;

source?: HTMLImageElement;

blurDegree?: Number;

};

Properties:

• type: String. Choose the virtual background type:

  • "color": Sets a solid color as the background.
  • "img": Sets an image as the background.
  • "blur": Blurs the user's original background.
  • "video": Sets a video as the dynamic background.
  • "none": Removes the background, that is, creates the effect of a portrait cutout.

• color: String. When you set type as "color", set this parameter to specify the color. The value must be a valid CSS color such as "white", "#00ff00", or "RGB(255, 0, 0)".

• source: The HTMLImageElement object. When you set type as "img", you can set a custom background image through this parameter.

info

• If the error "texture bound to texture unit 2 is not renderable. It might be non-power-of-2 or have incompatible texture filtering (maybe)?" occurs, check whether the product of the picture's width and height is a multiple of 2.
• Due to the restriction of browsers' security policy, if your background image resources are deployed across domains, you need to enable your servers' cross-domain permission and set the crossOrigin property of the HTMLImageElement object as anonymous.
• Since it takes time to load the image in the HTMLImageElement object, Agora recommends calling setOptions in the onload callback of the HTMLImageElement object, otherwise, the background momentarily turns black.

• blurDegree: Number. When you set type as "blur", set this parameter to choose the blurring degree:

  • 1: Low
  • 2: Medium
  • 3: High

• fit: String. Set this parameter to choose how the virtual background is filled:

  • "contain": Fill in proportion to ensure that the background can be displayed completely, and fill the insufficient part with black.
  • "cover": Fill in proportion and ensure that the background can fill the area, and the excess part will be cropped.
  • "fill": Stretch to fill the area.