Texture Drawing

This addon provides a solution to synchronize a texture modification.

Overview

The logic of this add-on is to share drawing points details (position, color, .. see DrawinPoint) between users.

A sample pen is included, drawing point when detecting drawable blocking surfaces (see the BlockingContact addon).

Features:

• supports concurrent edition: several pens can draw on the same surface at the same time
• drawing interpolation: the drawing will be displayed on remote aligned with current interpolation state, allowing them to be in sync with pen objects whose position is interpolated with a NetworkTransform
• late joiners support: the full drawing will be shared to people connecting to the room after they have been drawn, using Fusion's streaming API
• shader based drawing: limit the performance impact of texture edits. To ensure that performances scale to the number of simultaneous drawers, a drawing budget system has also been added
• limited cost on memory preallocated for Fusion (no large network array per drawing): each drawing just synchornize its total number of points. The synchronization is done through the "pens", holding the network array of the latest point, and late joiner support is done through the streaming API.

State authority

The logic is split between pens, and drawing surfaces, both having network behaviours, and potentially different state authority between a pen and its target drawing:

• the pen state authority is the grabber of the pen
• the drawing surface state authority might not be the drawing user, as several users might be drawing at the same time on the surface.

Drawing steps

First, initialy, when an user connect to a scene, if it is a late joiners (was not here from the starts):

1. the TextureDrawing detects that it is lacking some drawing information, and request them to other users (the state authority of the drawing in priority)
2. stores locally the existing points received (see Late joiners), and draws them during the next Render() calls (to avoid performance issue, a maximum "budget" of allowed drawing per frame is set to prevet late joiners to freeze while recover plenty of large drawings).

When a new drawing part is added by a pen, first:
3. the pen detect that a drawing point that should be added to a drawing surface (see TexturePen), and prepares the drawing point info (coordinates, ...).
4. the latest points to draw for a pen are stored in a networked ring buffer on the pen (see TextureDrawer), so that all users can be aware of its will to add points on the drawing
5. the pen shares the point info directly with the local version of the texture
6. thus, during the next Render(), the texture will be immediate updated

Then, the drawing pen, on all remote clients:
7. detects that it has new points for a drawing,
8. store these new drawing points in the drawing local cache. See TextureDrawing
9. during the next Render, draws them on the actual texture (see TextureSurface), based on the pen current interpolation state (see Interpolation)

Texture drawing

The addon provides two ways to edit a texture:

• the default solution uses a specific drawing shader, made with Shader Graph, and so only compatible with URP and HDRP render pipelines. This solution is more suitable for high resolution textures.
• the alternative solution edits the texture using the ProtoTurtle.BitmapDrawing third party solution, which provides a bitmap drawing API.

Shader-based drawing

The LinePainter shader can draw a line on a texture. It contains a subgraph LineSDK determining the distance from a point to the drawn line.

The drawing logic is to:

• use a render texture in the drawing surface material
• every time a line needs to be drawn, use Graphics.Blit to feed the shader with the current render texture and line details, and store the resulting results

Class details

TexturePen

The TexturePen is located on the pen (with a BlockableTip component) and tries to detect a contact with a BlockingSurface component, having also a TextureDrawing component.

When a contact is detected, AddPointWithThrottle is called on the TextureDrawer, to ask it to plan to add a point.

During the next FixedUpdateNetwork(), the TextureDrawer will call AddDrawingPoint for all of these planned points. It will:

• store the points in the networked ring buffer (a network array), so that the pen on all users clients will receive those points
• transmit the point immediatly to local TextureDrawing, so that then can be drawn on the texture during the next Render call.

On remote clients, OnNewEntries then triggers to warn than new points were added to the ring buffer, and they then store them in their local target TextureDrawing.
The position index of the point for the pen (aka the number of point previously drawn, no matter the target drawing) is added to the data, so that the drawing could then interpolate.

TextureDrawer

This is subclass of the RingBufferSyncBehaviour class from DataSyncHelpers addon.
It is used to record the drawing points that must be edited on the texture when a contact is detected between the pen and the drawing surface.

After a TexturePen calls the drawer's AddPointWithThrottle, during the next FixedUpdateNetwork(), the TextureDrawer will call AddDrawingPoint for all of these planned points. It will:

• store the points in the networked ring buffer (a network array), so that the pen on all users clients will receive those points
• transmit the point immediatly to local TextureDrawing, so that then can be drawn on the texture during the next Render call.

On remote clients, OnNewEntries then triggers to warn than new points were added to the ring buffer, and they then store them in their local target TextureDrawing.
The position index of the point for the pen (aka the number of point previously drawn, no matter the target drawing) is added to the data, so that the drawing could then interpolate (see Interpolation).

Note:

• filling the drawer ring buffer too quickly might lead to data loss (in the current settings, it should not occur in normal network conditions). To prevent this, either throttle the transmission of points with TexturePen.maxPointInsertionPerSeconds, or increase the RingBufferSyncBehaviour.BUFFER_SIZE

TextureDrawing

When adding a new drawing point, if a line was not yet finished for the requesting TextureDrawer, the TextureDrawing creates a line between the previous point and the new one.

Several drawers can have drawings in parallel, as a TextureDrawing keeps a cache of the latest point drawn per drawer.
The TextureDrawing finally calls the Draw() method on the referenced TextureSurface, to add a point or draw a line.

Note:

Interpolation and respecting the drawing budget might delay the drawin to the next frames

Interpolation

Usually, a TexturePen position in space is interpolated (usually thanks to a NetworkTransform interpolation implementaiton) on remote users. It means that we see its position slightly in the "past".

If on remote users drawing, we immediatly draw all the available points, then the lines would seem to appear a bit in advance of the actual pen moves.

So, in the same way the position of the pen is interpolated on remote users, the displayed drawing points index should be interpolated should too.

To do so, every point stored in a TextureDrawing stores the position index of the point for the origin pen (aka the number of point previously drawn, no matter the target drawing).
This way, it is possible during Render() calls to ask the pen what would be its interpolated position index: if we have a point with a lower index for this pen, then we can draw it, otherwise we should wait a bit.

Drawing budget

When a late joiners connects, or when we ask locally to redraw a drawing (if we changed its background color for instance), a lot of drawing calls cna be triggered, leading to a lot of shader calls through Graphics.Blit in the TextureSurface, which might have an impact on performances.

To prevent this, a maximum a drawing calls is allowed per frame, shared among all TextureDrawing. It can be changed in its code, by editing the globalFrameLineDrawingBudget value.

The budget logic:

• shares the budget between TextureDrawing components requiring to be drawn
• for a given drawing, shares the budget between its drawers, if several pens just added points simultaneously on it

Late joiners

For late joiners, TextureDrawing uses the Fusion 2 streaming API, to send the complete list of points added from the start.

TextureDrawing subclasses StreamingSyncBehaviour and changes its logic.

While a StreamingSyncBehaviour is built to send data in real time, and then share them to late joiners:

• here the DrawingPoints data are just stored in a local cache (the real time transmission has already been handled by the TextureDrawer network var)
• for late joiners, the reception data logic is changed, to mix between the data already received previously through the network var of the TextureDrawers (usually pretty quickly), and the full data cache that takes a few frame to be received.

Merging those data requires a reconciliation, as the last point in the full cache might also have been received first through the TextureDrawer earlier (the same point might be received both through the streaming API providing the full cache backup, and through the TextureDrawer networked var provide real time data), and as those points could be associated to unfinished lines.
A simple way to solve this issue is to go through all the points stored in the full cache, and add a line end point for all drawing lines that were not finished upon transmission. The TextureDrawer data being more recent, they will in any case finish any pending lines again.

Note:

• another way to solve this would be to use the full byte count, to detect duplicate points precisely.
• this approach might lead to different drawing on each client if people draw on the same pixel at the same time. If it is an issue, another approach would be to use a RingBufferLosslessSyncBehaviour subclass instead of a StreamingSyncBehaviour: drawer not having authority on the drawing would draw temporary lines, and upon reception of the TextureDrawing "confirmation", the actual lines would be drawn.

Clear a drawing

To clear a drawing of all drawing points and lines, several steps must be taken:

• clear the underlying data storage (so that data are not sent anymore to late joiners)
• clear the local drawing point cache, so that these are not redrawn if a full redrawn is required on the associated TextureSurface (for instance when changing the background color)
• resetting the TextureSurface texture, but refilling with the background color
• make sure that all the pens in the scene don't make late joiner redraw the last bits of the drawing when they join because they were still containing a bit of the drawing in their buffer

This last point, erasing the drawing in the pen too, is important, as a bit counterintuitive.

The concurrent drawing mechanism works as in fact, each pens contains a bit of the drawing, that the TextureDrawing "detects" to actualling apply them on the drawing.
Due to that, if the pen has not been used on another drawing, its buffer will still retain a small part of the drawing, that late joiners could not detect as unrelevant anymore.

To purge the pen, we can't remove points in the underlying ring buffer. So we edit all the points associated with this texture drawing in the TextureDrawer.ForgetTextureDrawingPoints method. This method uses the RingBuffer.EditEntriesStillInbuffer method, that triggers a callback for each entry in the ring buffer, and provide a window of opportunity to replace it. In this window, if the textureDrawingId is the one of the cleared drawing, we set textureDrawingId to NetworkBehaviourId.None. this value is then ignored when a TextureDrawer determines to which TextureDrawing send its drawing points.

TextureSurface

TextureSurface references the Renderer component and contains the utility methods for textures editing: initialize the texture, change the texture color, draw a point, draw a line.
So, this class is not linked to the network part.

It implements the IRenderTextureProvider interface (from the DataSyncHelpers addon). The onRedrawRequired event is raised when the texture has been edited externally, and TextureDrawing subscribes to it to redraw all the point if this happens.

DrawingPoint

The DrawingPoint and DrawerDrawingPoint classes define the drawing points of the surface (position, color, pressure and a reference Id), the DrawingPoint containing in addition the position index of the point for the origin pen (used for interpolation).
They implement the RingBuffer.IRingBufferEntry interface of the DataSyncHelpers addon (required for the DrawerDrawingPoint, to be stored in the pen ring buffer).

Dependencies

• DataSyncHelpers addon
• BlockingContact addon

Demo

A demo scene can be found in the Assets\Photon\FusionAddons\TextureDrawing\Demo\Scenes\ folder.

Download

This addon latest version is included into the Industries addon project

It is also included into the free XR addon project

Supported topologies

• shared mode

Third party

• ProtoTurtle.BitmapDrawing, MIT license, https://github.com/ProtoTurtle/UnityBitmapDrawing

Changelog

• Version 2.1.2:
  • Add option to clear a post-it content
• Version 2.1.1:
  • Add method to clear TextureDrawing
  • ResetTexture() now use the last used color
• Version 2.1.0: Refactoring to add interpolation
• Version 2.0.0: First release

Back to top