Share the Screen

Screen sharing enables the host of an interactive live streaming broadcast or video call to display what is on their screen to other users in the channel. This technology has many obvious advantages for communicating information, particularly in the following scenarios:

• During video conferencing, the speaker can share a local image, web page, or full presentation with other participants.
• For online instruction, the teacher can share slides or notes with students.

Screen sharing on iOS is implemented by using the native iOS ReplayKit framework in an Extension to record the screen and by treating the screen sharing stream as a user joining the channel. As Apple does not support screen capture in the main app process, you need to create a separate Extension for the screen sharing stream.

This section describes how to implement screen sharing using the iOS SDK v3.7.0 and later.

Due to system limitations, screen sharing is only available for iOS 12.0 or later.

This feature requires a high-performance device. Agora recommends that you use this feature on iPhone X or later models.

Before you begin, ensure that you understand how to start a video call or start interactive video streaming. For details, see Start a Video Call or Start Interactive Video Streaming.

1. Ensure that AgoraReplayKitExtension.xcframework is integrated into the project. This dynamic library wraps the following functionality:

   • Record the screen with Apple ReplayKit.

   • Use the custom video source feature of the SDK to capture the screen recording data and send it to other users in the channel.

2. Create a Broadcast Upload Extension to enable the screen sharing process:

   1. In Xcode, click File > New > Target..., select Broadcast Upload Extension in the pop-up window, and click Next.

      

   2. In the pop-up window, fill in Product Name and other fields, uncheck Include UI Extension, and click Finish. Xcode automatically creates the Extension folder, which contains the SampleHandler.h file.

   3. Modify the SampleHandler.h file to modify the code logic that implements screen sharing:

      • If you only need to use the functionality in AgoraReplayKitExtension.xcframework provided by Agora, modify it by checking Target for the Extension you created, and setting the Value of the NSExtension > NSExtensionPrincipalClass key in the Info settings from SampleHandler to AgoraReplayKitHandler.

        

      • If you need to add some business logic in addition to using the functionality in AgoraReplayKitExtension.xcframework provided by Agora, modify it by replacing the code in the SampleHandler.h file with the following code:

        // Objective-C
        

        #import "SampleHandler.h"
        

        #import "AgoraReplayKitExt.h"
        

        #import <sys/time.h>
        

        
        

        @interface SampleHandler ()<AgoraReplayKitExtDelegate>
        

        
        

        @end
        

        
        

        @implementation SampleHandler
        

        
        

        - (void)broadcastStartedWithSetupInfo:(NSDictionary<NSString *,NSObject *> *)setupInfo {
        

        // User has requested to start the broadcast. Setup info from the UI extension can be supplied but is optional.
        

        [[AgoraReplayKitExt shareInstance] start:self];
        

        
        

        }
        

        
        

        - (void)broadcastPaused {
        

        // User has requested to pause the broadcast. Samples are no longer delivered.
        

        NSLog(@"broadcastPaused");
        

        [[AgoraReplayKitExt shareInstance] pause];
        

        }
        

        
        

        - (void)broadcastResumed {
        

        // User has requested to resume the broadcast. Samples delivery resumes.
        

        NSLog(@"broadcastResumed");
        

        [[AgoraReplayKitExt shareInstance] resume];
        

        
        

        }
        

        
        

        - (void)broadcastFinished {
        

        // User has requested to finish the broadcast.
        

        NSLog(@"broadcastFinished");
        

        [[AgoraReplayKitExt shareInstance] stop];
        

        
        

        }
        

        
        

        - (void)processSampleBuffer:(CMSampleBufferRef)sampleBuffer withType:(RPSampleBufferType)sampleBufferType {
        

        [[AgoraReplayKitExt shareInstance] pushSampleBuffer:sampleBuffer withType:sampleBufferType];
        

        }
        

        
        

        #pragma mark - AgoraReplayKitExtDelegate
        

        
        

        - (void)broadcastFinished:(AgoraReplayKitExt *_Nonnull)broadcast reason:(AgoraReplayKitExtReason)reason {
        

        switch (reason) {
        

        case AgoraReplayKitExtReasonInitiativeStop:
        

        {
        

        // NSDictionary *userInfo = @{NSLocalizedDescriptionKey : @"Host app stop srceen capture"};
        

        // NSError *error = [NSError errorWithDomain:NSCocoaErrorDomain code:0 userInfo:userInfo];
        

        // [self finishBroadcastWithError:error];
        

        NSLog(@"AgoraReplayKitExtReasonInitiativeStop");
        

        }
        

        break;
        

        case AgoraReplayKitExtReasonConnectFail:
        

        {
        

        // NSDictionary *userInfo = @{NSLocalizedDescriptionKey : @"Connect host app fail need startScreenCapture in host app"};
        

        // NSError *error = [NSError errorWithDomain:NSCocoaErrorDomain code:0 userInfo:userInfo];
        

        // [self finishBroadcastWithError:error];
        

        NSLog(@"AgoraReplayKitExReasonConnectFail");
        

        }
        

        break;
        

        
        

        case AgoraReplayKitExtReasonDisconnect:
        

        {
        

        // NSDictionary *userInfo = @{NSLocalizedDescriptionKey : @"disconnect with host app"};
        

        // NSError *error = [NSError errorWithDomain:NSCocoaErrorDomain code:0 userInfo:userInfo];
        

        // [self finishBroadcastWithError:error];
        

        NSLog(@"AgoraReplayKitExReasonDisconnect");
        

        }
        

        break;
        

        default:
        

        break;
        

        }
        

        }
        

        
        

        @end
        

3. Call startScreenCapture, and prompt the user's manual action to make the app start screen sharing. You can choose either of the following two methods:

   • Method 1: Prompt the user to tap and hold the Screen Sharing button in the control center of the iOS system, and select the Extension you created to start screen sharing.

   • Method 2: Use RPSystemBroadcastPickerView, which is added by Apple in iOS 12.0, to pop up the "Start screen sharing" button in the app interface. It prompts the user to start screen sharing by tapping this button.

   RPSystemBroadcastPickerView is limited in use and might fail in later versions of iOS systems. Therefore, use the method 2 at your discretion.

Agora provides an open-source sample project on GitHub, where you can refer to the following files:

• /APIExample/Examples/Advanced/ScreenShare/ScreenShare.swift
• /Agora-ScreenShare-Extension/SampleHandler.swift

Some restrictions and cautions exist for using the screen sharing feature, and charges can apply. Agora recommends that you read the following API references before calling the API:

• startScreenCapture
• updateScreenCapture
• stopScreenCapture

This section describes how to implement screen sharing using the iOS SDK earlier than v3.7.0.

The implementation method introduced in this section applies to iOS 12.0 or later.

Before you begin, ensure that you understand how to start a video call or start interactive video streaming. For details, see Start a Video Call or Start Interactive Video Streaming.

Create a Broadcast Upload Extension with the following steps:

1. Open the project file with Xcode, and select Editor > Add Target... in the menu bar.

2. In the pop-up window, choose Broadcast Upload Extension on the iOS page, and click Next.

3. In the Product Name field, enter a name for the Extension, for example, Agora-ScreenShare, and click Finish. You should now see a folder for the Extension in your project; this folder holds the code for implementing screen sharing.

   

4. Open the Podfile in the project, and add dependencies to the Extension.

   target 'Agora-ScreenShare-Extension' do
   

   
   

   use_frameworks!
   

   
   

   pod 'AgoraRtcEngine_iOS', '~> 3.1.1'
   

   
   

   # Use the following code to achieve media stream encryption
   

   # pod 'AgoraRtcEngine_iOS_Crypto', '~> 3.1.1'
   

   
   

   end
   

5. Run the pod install command in the project root directory to install the dependencies.

Add RPSystemBroadcastPickerView to the view of the app as a button to start screen recording.

// Swift

func prepareSystemBroadcaster() {

if #available(iOS 12.0, *) {

let frame = CGRect(x: 0, y:0, width: 60, height: 60)

let systemBroadcastPicker = RPSystemBroadcastPickerView(frame: frame)

systemBroadcastPicker.autoresizingMask = [.flexibleTopMargin, .flexibleRightMargin]

if let url = Bundle.main.url(forResource: "Agora-ScreenShare-Extension", withExtension: "appex", subdirectory: "PlugIns") {

if let bundle = Bundle(url: url) {

systemBroadcastPicker.preferredExtension = bundle.bundleIdentifier

}

}

broadcasterPickerContainer.addSubview(systemBroadcastPicker)

} else {

self.showAlert(message: "Minimum support iOS version is 12.0")

}

}

Use the custom video source function of the Agora SDK to send the system-recorded screen data to remote users for screen sharing.

Create an instance of AgoraRtcEngineKit for the screen sharing stream, and set up the custom video source function.

// Swift

private static let sharedAgoraEngine: AgoraRtcEngineKit = {

let kit = AgoraRtcEngineKit.sharedEngine(withAppId: KeyCenter.AppId, delegate: nil)

kit.enableVideo()

// Use custom video sources

kit.setExternalVideoSource(true, useTexture: true, pushMode: true)

// Do not receive video and audio streams from all remote users

kit.muteAllRemoteVideoStreams(true)

kit.muteAllRemoteAudioStreams(true)

return kit

}()

Use Apple's native ReplayKit framework for screen recording.

1. In the Info.plist file in the Extension for screen sharing, set the RPBroadcastProcessMode as RPBroadcastProcessModeSampleBuffer.

2. Start screen recording in the broadcastStarted callback.

   The channel name in the code example is set as ScreenShare. If you want the user to enter the channel name, you must use App Group to pass the parameter value of the main process to the Extension.
   // Swift
   

   override func broadcastStarted(withSetupInfo setupInfo: [String : NSObject]?) {
   

   AgoraUploader.startBroadcast(to: "ScreenShare")
   

   
   

   DispatchQueue.main.async {
   

   Timer.scheduledTimer(withTimeInterval: 0.1, repeats: true) {[weak self] (timer:Timer) in
   

   guard let weakSelf = self else {return}
   

   let elapse = Int64(Date().timeIntervalSince1970 * 1000) - weakSelf.lastSendTs
   

   print("elapse: \(elapse)")
   

   // If the inter-frame interval of the video is too long, resend the previous frame.
   

   if(elapse > 300) {
   

   if let buffer = weakSelf.bufferCopy {
   

   weakSelf.processSampleBuffer(buffer, with: .video)
   

   }
   

   }
   

   }
   

   }
   

   }
   

3. Send the system-collected data to the Agora SDK.

   // Swift
   

   override func processSampleBuffer(_ sampleBuffer: CMSampleBuffer, with sampleBufferType: RPSampleBufferType) {
   

   DispatchQueue.main.async {[weak self] in
   

   switch sampleBufferType {
   

   case .video:
   

   if let weakSelf = self {
   

   weakSelf.bufferCopy = sampleBuffer
   

   weakSelf.lastSendTs = Int64(Date().timeIntervalSince1970 * 1000)
   

   }
   

   // Send video buffer
   

   AgoraUploader.sendVideoBuffer(sampleBuffer)
   

   @unknown default:
   

   break
   

   }
   

   }
   

   }
   

   Use pushExternalVideoFrame to enable sendVideoBuffer.

   // Swift
   

   static func sendVideoBuffer(_ sampleBuffer: CMSampleBuffer) {
   

   guard let videoFrame = CMSampleBufferGetImageBuffer(sampleBuffer)
   

   else {
   

   return
   

   }
   

   
   

   var rotation : Int32 = 0
   

   if let orientationAttachment = CMGetAttachment(sampleBuffer, key: RPVideoSampleOrientationKey as CFString, attachmentModeOut: nil) as? NSNumber {
   

   if let orientation = CGImagePropertyOrientation(rawValue: orientationAttachment.uint32Value) {
   

   switch orientation {
   

   case .up, .upMirrored: rotation = 0
   

   case .down, .downMirrored: rotation = 180
   

   case .left, .leftMirrored: rotation = 90
   

   case .right, .rightMirrored: rotation = 270
   

   default: break
   

   }
   

   }
   

   }
   

   
   

   let time = CMTime(seconds: CACurrentMediaTime(), preferredTimescale: 1000)
   

   
   

   let frame = AgoraVideoFrame()
   

   frame.format = 12
   

   frame.time = time
   

   frame.textureBuf = videoFrame
   

   frame.rotation = rotation
   

   sharedAgoraEngine.pushExternalVideoFrame(frame)
   

   }
   

4. Call joinChannelByToken in the AgoraRtcEngineKit instance corresponding to the screen sharing stream to join the channel, and then start screen sharing.

   // Swift
   

   static func startBroadcast(to channel: String) {
   

   print("joining \(channel)")
   

   sharedAgoraEngine.joinChannel(byToken: KeyCenter.Token, channelId: channel, info: nil, uid: SCREEN_SHARE_UID, joinSuccess: nil)
   

   }
   

Implementing screen sharing using the iOS SDK earlier than v3.7.0 is complicated. To reduce the difficulty of using the sample project, this section describes the files or folders used in the sample project.

File/Folder	Description
/iOS/Agora-ScreenShare-Extension	The Extension for the screen sharing process. The main code files are as follows: SampleHandler.swift：Records the screen with Apple ReplayKit. AgoraUploader.swift：Uses the custom source function of the SDK to capture the screen recording data and send it to other users in the channel.
/iOS/APIExample/Examples/Advanced/ScreenShare/ScreenShare.swift	The main code of the screen sharing app, which allows the local user to join a channel and enable screen sharing.
/iOS/Agora-ScreenShare-Extension/AgoraAudioTube.mm	Shares both the screen and audio of the app at the same time by using the custom audio source function.

• If you use Cocoapods, add the following lines to the Podfile file to add dependencies for your screen sharing Extension. Remember to replace ScreenSharing with the target name of your screen sharing Extension.

  target 'ScreenSharing' do
  

  pod 'AgoraRtcEngine_iOS', '3.7.0'
  

  end
  

• The memory limit of a Broadcast Upload Extension is 50 MB. Ensure that your memory usage does not exceed this limit.

• In the screen sharing process, you need to call muteAllRemoteVideoStreams and muteAllRemoteAudioStreams to cancel receiving streams from remote users and avoid double subscriptions.

• For the main process, the screen sharing stream is equivalent to a remote user in the channel. To avoid unnecessary costs, you can exclude the screen sharing stream when setting up the remote user view.

  // Swift
  

  // Determine if the uid is used for a screen sharing stream
  

  func isScreenShareUid(uid: UInt) -> Bool {
  

  return uid >= SCREEN_SHARE_UID_MIN && uid <= SCREEN_SHARE_UID_MAX
  

  }
  

  
  

  // The callback of adding remote users to the channel
  

  func rtcEngine(_ engine: AgoraRtcEngineKit, didJoinedOfUid uid: UInt, elapsed: Int) {
  

  LogUtils.log(message: "remote user join: \(uid) \(elapsed)ms", level: .info)
  

  
  

  // In the case of a uid for screen sharing, ignore
  

  if(isScreenShareUid(uid: uid)) {
  

  LogUtils.log(message: "Ignore screen share uid", level: .info)
  

  return
  

  }
  

  
  

  // Set up the remote user view
  

  ......
  

  }