Data Sync Helpers

Data Sync Helpers Principle

The Data Sync Helpers addon provides some helpers classes that simplify synchronizing data for use cases with special needs.

Typically, the target use cases are, for instance:

• Sending very regularly small chunk of data, whose full list is also required for late joiners: drawing points with a 3D pen and allowing late joiners to have the full drawing too
• Sending very regularly small chunk of data with no need to keep the full list but that can be sent several time before the next data reception: for instance, it can be used for a wand drawing a temporary line in air, or for a gun, to be sure that all shots are fired
• Synchronizing a large one-shot data: sending a photo shot took in an application to other users

RingBufferLossLessSyncBehaviour class

Overview

As seen in Projectile essential - Projectile data buffer, a ring buffer is a good approach to store a succession of small data, even if several ones could be sent before the first one is received: the ring buffer logic will keep several of them, without losing data.

However, for information that needs to be accessible for late joiners too (drawing points, etc. ), a ring buffer alone will progressively lose the data that passed through it.

The RingBufferLossLessSyncBehaviour<TEntry> class solves this problem:

• In addition to a classical ring buffer, the underlying storage keeps track of the full count of data that were shared,
• When a late joiner arrives, they can thus know they are missing data,
• If that occurs, this class will send a message through a RPC to other users (the class will handle finding which user can send the data, even if the original writer has since left room),
• Other users having the missing data will share it through the Runner.SendReliableDataToPlayer method.

Under the hood, this class relies on a NetworkArray<byte> to store both the ring buffer data (start index, next write the need), the total amount of data count, and the actual data:

The missing data can occurs either when the ring buffer is filled too fast, or for late joiners, and that's when messages requesting the missing data ranges are sent:

RingBufferLosslessSyncBehaviour<TEntry> is a generic NetworkBehaviour subclass that needs to be subclassed to specify TEntry.
TEntry should be a struct representing the data to share, implementing the RingBuffer.IRingBufferEntry interface to define how it converts to a byte array (AsByteArray) and how to load its content from a byte array (FillFromBytes).
To help implement those methods, the SerializationTools contains static methods to help.

Usage

Add data

As the state authority, use AddEntry(TEntry) to add an object in the synched data.

Overriding callbacks methods

Some callbacks can be overridden to react upon live data reception, or complete recovery of missed data, for instance.

C#

    // newEntries will contain the new TEntry structs detected in the last data update.The received data at start might not be long enough to form an entry (for late joiners first ring reception, ...), those additional bytes are stored in newPaddingStartBytes
    public virtual void OnNewEntries(byte[] newPaddingStartBytes, TEntry[] newEntries) { }

    // OnDataLoss will warn of data loss: no need to handle the loss, the request are sent automatically
    public virtual void OnDataloss(RingBuffer.LossRange lossRange) { }

    // Called when one loss is restored
    protected virtual void OnLossRestored(LossRequest request, byte[] receivedData) { }

    // When all loss in the total data remains. SplitCompleteData() can be called to retrieve all the TEntry 
    protected virtual void OnNoLossRemaining() { }

    // When a loss is permanent (no one having the data is still in the room 
    public virtual void OnNoAnswerForALossRequest(LossRequest request) { }

Implementing IRingBufferEntry

The SerializationTools class simplifies implementing the IRingBufferEntry expected methods.

To represent a struct as a byte array (AsByteArray), simply pass all its basic supported types (float, Vector3, byte, Quaternion, int) to SerializationTools.AsByteArray, and it will return a byte array.
For instance, for the following struct:

C#

    [System.Serializable]
    public struct DrawPoint : RingBuffer.IRingBufferEntry
    {
        public Vector3 localPosition;
        public float drawPressure; 
        /* ... */
    }

This implementation AsByteArray will be optimal:

C#

    public byte[] AsByteArray => SerializationTools.AsByteArray(localPosition, drawPressure);

To load a byte array into the struct, the SerializationTools.Unserialize method handles the unserialization to basic supported types (float, Vector3, byte, Quaternion, int), while keeping track of the position in the array.

So for the same sample struct, this implementation of FillFromBytes will load the data serialized with the AsByteArray previously defined:

C#

    public void FillFromBytes(byte[] entryBytes)
    {
        int unserializePosition = 0;
        SerializationTools.Unserialize(entryBytes, ref unserializePosition, out localPosition);
        SerializationTools.Unserialize(entryBytes, ref unserializePosition, out drawPressure);
    }

RingBufferSyncBehaviour class

RingBufferSyncBehaviour is the parent class of RingBufferLossLessSyncBehaviour, which handles the ring buffer aspect, without the message to recover data not stored in the underlying ring buffer.

It can be used as RingBufferLossLessSyncBehaviour, without anything related to loss recovery (the losses are still detected, but not recoverable here).

This class can be used for classical ring buffer needs (gun projectiles, etc.).

The ring buffer index tracking itself is done in a struct named Ringbuffer, which updates start index and next write index, and can add data/update the actual data storage if provided.

StreamingSyncBehaviour class

Overview

The StreamSynchedBehaviour allows sending arbitrary chunks of data to other players.
This helper uses the Fusion SendReliableDataToPlayer API.
The first part of the 4-part key used to identify a message with this API is used to store the Object id, so that several objects in the room could send and receive data, without fear of sending them to the wrong object.

The StreamSynchedBehaviour main specific point is to handle the late joiners case: how to send data to users that missed the initial transmission, especially if, in shared mode, the state authority has left the room.

This is done by an RPC message sent by late joiners, requesting the reception of the full cache, from either the state authority or another player if they left the room.

Usage

Sending data

Sending data is done by calling Send(byte[] data) with the data to send.
Data are received as stored as chunks, and the subclasses need to merge them if needed.

Overriding callbacks methods

Some callbacks can be overridden to react upon live data reception, or recovery of missed data, for instance.

C#

    // Provides the 0-1 download progress of a currently received chunk of data
    protected virtual void OnDataProgress(float progress) { }
    // Called on complete reception of a chunk of data
    protected virtual void OnNewBytes(byte[] newData) { }
    // Called to know the client is a late joiner, waiting for data
    protected virtual void OnLateJoinersWaitingForData() { }
    // Called if missing data for a late joiner are unavailable anywhere
    protected virtual void OnMissingData() { }

SerializationTools and ByteArrayTools

To facilitate the buffer content manipulation, and also to simplify the implementation of RingBuffer.IRingBufferEntryexpected methods, the SerializationTools and ByteArrayTools static class contain helpers methods.

For instance, let's consider a LineDrawingPoint struct implementing RingBuffer.IRingBufferEntry that contains those fields:

C#

public struct LineDrawingPoint : RingBuffer.IRingBufferEntry
{
    public float drawPressure;
    public Vector3 localPosition; 

    /* ... */
}

Some helpers allow to quickly implement the RingBuffer.IRingBufferEntry methods:

C#

public byte[] AsByteArray => SerializationTools.AsByteArray(localPosition, drawPressure);

public void FillFromBytes(byte[] entryBytes)
{
    int unserializePosition = 0;
    SerializationTools.Unserialize(entryBytes, ref unserializePosition, out localPosition);
    SerializationTools.Unserialize(entryBytes, ref unserializePosition, out drawPressure);
}

ByteArraySize attribute

Some advanced methods might need to know the serialization size of a RingBuffer.IRingBufferEntry.

To do that, the helper ByteArrayTools.ByteArrayRepresentationLength is used. However, with no additional information, this helper will trigger an allocation.

To avoid that, it is possible to add a [ByteArraySize(SERIALIZATION_SIZE)] attribute before structs implementing the RingBuffer.IRingBufferEntry interface.

To determine the value to provide to the ByteArraySize attribute, note that if you call ByteArrayTools.ByteArrayRepresentationLength on a struct without this attribute, the call would still work (and cause an allocation), but in debug mode, a warning log will appear in the console to suggest you to add this attribute to the struct, and it will also write the value to provide to it:

To optimize alloc, add [ByteArraySize(29)] before DrawerDrawingPoint definition

Demo

The helpers have some sample classes showing how to subclass them:

• TestStreamSynchedBehaviour for StreamingSyncBehaviour
• TestLosslessRingBufferSync for RingBufferLossLessSyncBehaviour
• TestRingBufferSync for RingBufferSyncBehaviour

The DemoStreaminghelpers scene contains the TestStreamSynchedBehaviour implementation of StreamingSyncBehaviour.
The scene contains 2 of those components to show that the data is sent to the proper StreamSynchedBehaviour object.
By pressing the SendRandomData button, a chunk of data is sent to connected users, and will appear in the All data attributes.
It will also be available for late joiners.

The DemoRingBuffer contains both a TestLosslessRingBufferSync and a TestRingBufferSync to show both lossless and regular ring buffers.
On the ring objects, by pressing either AddSingleEntry (which will add to the data the entry described in the inspector Single entry to add field), AddRandomEntry or AddSeveralRandomEntries, entries will be added to the entries field in the inspector.
They will appear on both connected clients, and on late joiner clients.
As the log level includes Data loss recovery, it will be possible to see messages about data loss recovery in the console.
Late joiners will systematically trigger such messages, and AddSeveralRandomEntries calls fill the ring buffer too fast on purpose, so such messages will appear.
In the end, all users will have the same entries.

Download

This addon latest version is included into the Industries addon project

It is also included into the free XR addon project

Supported topologies

• shared mode

Please note that StreamingSyncBehaviour can be used in host topology too.

Changelog

• Version 2.0.8:
  • Add ByteArraySize attribute for IByteArraySerializable struct, to provide a way to explicitly give the serialization size of a IByteArraySerializable
  • Give hint on how to fill the ByteArraySize size value, if not provided
  • Add RingBuffer.EditEntriesStillInbuffer to have a callback allowing to edit the latest entries still stored in the buffer
• Version 2.0.7:
  • Add in StreamSynchedBehaviour a way to clear data on all users
• Version 2.0.6:
  • Add RingBufferLosslessSyncBehaviour.TryGetInterpolationTotalEntryCount, to find the entry count interpolated between each interpolation state entry count (from and to)
• Version 2.0.5:
  • Add RingBuffer.PositionInfo to simplify finding the current position in a from/to buffer
• Version 2.0.4:
  • Add sample drawing code
• Version 2.0.3:
  • Allow demo CameraPicture to forward to a specific RenderTexture
  • Add solutions (StreamingAPIConfiguration and overridable TargetId methods) if a NetworkObject contains 2 components using streaming API at the same time
• Version 2.0.2: Add support for NetworkBehaviourId serialization in tools
• Version 2.0.1:
• Add Vector2 and uint support in serialization tools
• add flexibility to ring buffer entry size determination
• Version 2.0.0: First release

Back to top