This document is about: FUSION 2

SWITCH TO

FUSION 1 FUSION 2

Screen sharing

Fusion Industries prototyping Addons

This module demonstrates how to use the Photon Video SDK to share the user screen, captured through the uWindowCapture third party module.

Concepts

Photon VideoSDK

The Photon Video SDK is a special version of the Photon Voice SDK, including support for video streaming.

More details are available in its dedicated README, available on the SDK download page Photon video SDK.

The capture of video streams can be made with recorders implementing IVideoRecorderPusher, and the playback can be handled by players implementing IVideoPlayer.

uWindowCapture

In this module, the screen content is captured by uWindowCapture.

uWindowCapture offers specific windows capture, or full desktop capture. In this sample, for simplification, only full desktop capture is supported.

The uWindowCaptureRecorder class implementing the IVideoRecorderPusher interface allows to collect frames for the PhotonVideo SDK.

To enable it, U_WINDOW_CAPTURE_RECORDER_ENABLE has to be added to the define symbols.

Module logic

Setup

To be able to emit, the scene must contain:

a ScreenSharingEmitter component: it starts and stops the capture and screen sharing emission.
a UwcManager component: it allows the screen to be capture by uWindowCapture
a uWindowCaptureRecorder component: it collects the uWindowCapture texture and pass it to the PhotonVideo SDK
some component calling ConnectScreenSharing() and DisconnectScreenSharing() on the ScreenSharingEmitter component
a FusionVoiceClient component on the NetworkRunner gameobject, that starts the voice session (see Voice - Fusion Integration)

Please note :

screensharing in FullHD resolution (1920x1080) works correctly on Quest2 and Quest3 devices.
some higher resolutions (3440x1440 for example) work well on Quest2 but are no longer supported on Quest3 due to an operating system update.

Start emitting

To be able to send video, a Photon Voice connection has to been initialized. The module suppose this is already being done classicaly, and looks in the Update() to know when the session is started.

Upon calling ConnectScreenSharing(), ScreenSharingEmitter will first wait for the voice session initialization to be finished.

Then, it will also wait for the uWindowCapture initialization to be finished. This is done through the uWindowCaptureRecorder OnReady callback.

When everything is ready, a transmission channel (a "voice") is created by calling VoiceClient.CreateLocalVoiceVideo on the FusionVoiceClient: from now on, the recorder will stream the desktop capture.

If an IEmitterListener is provided, it will be warn of the emission start with the OnStartEmitting callback.

Note: in multiscreen situations, it is possible to chosse the desktop shared by calling SelectDesktop with a screen id, starting at 0.

Stop emitting

Calling DisconnectScreenSharing() disconnect the video streaming voice.

If an IEmitterListener is provided, it will be warn of the emission end with the OnStopEmitting callback.

Setup

To be able to receive screen sharing, the scene must contain:

a ScreenSharingReceiver component: it detects new video connections, then creates the video player and the associated texture.
a ScreenSharingScreen component: it describes the receiving renderer, and ensure to pass the proper data required by the Oculus Quest shader.

Optionally, it is possible to set up a LOD screen handler, to enable mip mapping (usefull for low pixel density screens, like on VR headsets).

ScreensharingReceiver

The receiver watch for new voice connections, with the VoiceClient.OnRemoteVoiceInfoAction callback, whose code is a video codec.

Upon such a connection, it creates a video player with Platform.CreateVideoPlayerUnityTexture.

Then, when this video player is ready (OnVideoPlayerReady), it creates a material containing the video player texture, and pass it to the ScreenSharingScreen with EnablePlayback: the screen will then change its renderer material to this new one.

Oculus Quest shader

A specific settings requires a custom shader for the Oculus Quest:

if the URP pipeline is used
on Android (that is the case for the Oculus quest)
if single-passed (or multi-view) rendering is used (which is common in VR)

For this configuration, the default Photon Video shader won't work.

This module provides a specific QuestVideoTextureExt3D shader to handle this, and the ScreenReceiver use it when needed.

To work, this shader needs to receive during each update some additional info about the renderer: it is handled in the ScreenSharingScreen Update().

Make sure to add this shader in the Always Included Shaders list (Unity Project Settings/Graphics)

Mip-mapping

The texture created on the fly by the video SDK cannot provide mip-mapping. On some platofrms with low density of pixels, like VR headsets, it might lead to shivering effects.

To handle that, the ScreenSharingScreenLODHandler drives a camera, that capture at a slow rate a picture of the video texture, to project it on another renderer, which does support mip mapping.

Install notes

When installing uWindowsCapture in a project, its asmdef has to be added to the Photon/PhotonVoice/PhotonVoiceApi/PhotonVoice.API.asmdef, so that uWindowCaptureRecorder manages to access it.

Dependencies

The current version has been tested with:

Fusion SDK 2.0
Photon Video SDK 2.53
uWindowCapture 1.1.0

Demo

A demo scene can be found in the Assets\Photon\FusionAddons\ScreenSharing\Demo\Scenes\ folder :

Two clients must be used to test the scene :

first client is the screensharing emitter : the Emitter game object must be enabled while the Receiver game object must be disabled,
second client is the screensharing receiver : the Emitter game object must be disabled while the Receiver game object must be enabled,

Because the StartSharingOnVoiceConnectionAvailable is set to true by default, the emitter starts to share the desktop when the scene is launched.