BlueJeans is being sunset. Please refer to more details

Custom Video Source

The Custom Video Source feature allows developers to integrate a variety of video frames, including pre-recorded clips, 3D renders, or a blend of both, as a video input, offering endless possibilities for video customization. For example, the Custom Video Source feature provides the capability to alter the plain/vanilla (default/raw) video stream from the device's front or rear camera to include any effects on top of the video stream and send it back to the BlueJeans platform. The effects may include visual enhancements, such as placing stickers on top of someone's face, introducing AR objects into the video frame, or displaying images or GIFs of products during a live video call.

Using the BlueJeans Custom Video Source feature, a retail store can enhance its virtual customer experience by showcasing its latest iPhone as augmented reality (AR) objects in a live video call. Instead of the default camera feed, a sales representative can replace it with a customized video feed that allows for the use of virtual objects. This feature enables participants to engage in an immersive and interactive meeting experience, where they can manipulate virtual objects and interact with the sales representative to know about the products.

Pre-requisites

The supported aspect ratios are 4:3 and 16:9

Steps for Building Custom Video Source

Here are the steps for creating a custom video source using the iOS Client SDK:

Step 1

Set your video source to ".custom" for initiating the custom video source flow.

 func setVideoSource(videoSource: .custom, completion: nil)
 var videoSource: BJNObservable<.custom> { get }

When you call setVideoSource with ".custom" that means you can pass custom frames such as; AR object, pre-recorded clips, 3D renders, custom camera feeds instead of the normal camera feed. By using the completion handler or observing the videoSource change to .custom a custom capture session can be started safely.

Note
By default, the video source is set as ".camera", which means you will see a video feed from your default front or back camera.

Step 2

The pushCustomVideoFrame() API is a function that allows you to send your custom video feed one frame at a time. In order to use the pushCustomVideoFrame() API, you must first set the video source to Custom by calling the setVideoSource API. The pushCustomVideoFrame() API takes the CustomVideoServiceFrameInfo object as an argument.

public class CustomVideoServiceFrameInfo {
    public let frameData: CustomVideoServiceFrameData
    public let frameOrientation: CustomVideoServiceFrameOrientation
    public let frameFormat: CustomVideoServiceFrameFormat
    
    public init(frameData: CustomVideoServiceFrameData,
                frameOrientation: CustomVideoServiceFrameOrientation,
                frameFormat: CustomVideoServiceFrameFormat) {
        self.frameData = frameData
        self.frameOrientation = frameOrientation
        self.frameFormat = frameFormat
            }
    }

CustomVideoServiceFrameData is an object that contains the frame converted to bytes as well as some other frame-related metadata such as frame width and height.

public class CustomVideoServiceFrameData {
    public let width: Int
    public let height: Int
    public let aspectRatio: CustomVideoServiceFrameAspectRatio
    public let dataSize: Int
    public let mirrorSelfView: Bool
    public let bytes: UnsafeMutablePointer<UInt8>
    
    public init(width: Int,
                height: Int,
                aspectRatio: CustomVideoServiceFrameAspectRatio = ._16by9,
                dataSize: Int,
                mirrorSelfView: Bool = false,
                bytes: UnsafeMutablePointer<UInt8>) {
        self.width = width
        self.height = height
        self.aspectRatio = aspectRatio
        self.dataSize = dataSize
        self.mirrorSelfView = mirrorSelfView
        self.bytes = bytes
    }
}

Irrespective of the frame format chosen, a CustomVideoServiceFrameData object should always contain the bytes field and should also specify the width and height of the frame in width and height fields of the CustomVideoServiceFrameData object.

To display the custom video frame in your desired orientation, you must specify its orientation. The custom video frame can be displayed in any of the following orientations.
```
enum CustomVideoServiceFrameOrientation {
case .portrait:
case .landscapeLeft:
case .portraitUpsideDown:
case .landscapeRight:
case .unknown:
}
```
- frameOrientation = .portrait and .unknown rotates the custom video frame by 0 degrees i.e no rotation applied.
- frameOrientation = .landscapeLeft rotates the custom video frame by 90 degrees.
- frameOrientation = .portraitUpsideDown rotates the custom video frame by 180 degrees.
- frameOrientation = .landscapeRight rotates the custom video frame by 270 degrees.
The following is a list of the frame formats that are supported for custom video
- NV12
- BGRA

Step 3

You are now ready to send your custom video frame-by-frame using the following API with 'CustomVideoServiceFrameInfo' as its argument. As this API can only take one frame at a time, it must be called every time a custom video frame is available for transmission.

@discardableResult func pushCustomVideoFrame(frame: CustomVideoServiceFrameInfo) -> CustomVideoServicePushVideoFrameResult
    func setVideoSource(videoSource: .custom, completion: nil)

The results of the pushCustomVideoFrame() API are below:

Response Codes	Description
Success	This means that the frame has been sent to all remote participants successfully.
NotUsingCustomVideoSource	You will get this error when you call `pushCustomVideoFrame()` directly without setting setVideoSource to "`CustomVideoServiceVideoSource`".
VideoMuted	You are sending custom video frames while your video is on mute.

Congratulation! You are now ready to display your custom video in the meeting.

Use Cases of Custom Video Source

Retail Store

The custom video source feature enables retailers to offer an immersive and interactive shopping experience for their customers, which can increase sales and differentiate them from competitors. An example of this is selling Macbooks through a virtual store where customers can see and experience the product in 3D.

Remote Learning

You can use the Custom Video Source feature to enhance the learning experience by bringing in visual aids such as diagrams, charts, and animations that can be manipulated by all participants/students. This feature can help students better understand complex concepts, leading to more engagement and participation in the virtual class.

For example, a teacher is using a 3D representation of the sun and earth to show students how the earth revolves around the sun, and how its own rotations are related. By using this feature, teachers can create a more immersive and engaging learning experience for their students in a virtual class. Other potential use cases for the feature in remote learning include virtual field trips and interactive history lessons.

AR object

Custom video source enables you to utilize AR objects and visual improvements in the live video stream. For instance, you can enhance the video's appeal by placing glasses or applying various AR filters near your face, as depicted in the image below. Additionally, you have the option to insert AR objects, like virtual products or brand logos, into the video frame to advertise your sponsorships or merchandise.