Debugging Desyncs

A desync is short for desynchronization.
A desync occurs when two players end up with different states after simulating given frames' inputs.

Things to check for:

• Do not use float, double, FP.FromFloat_UNSAFE, FP.FromString_UNSAFE during system simulation.
• Do not use state that is outside Frame, unless that state does not outlive one Frame.

If these things are not used in your code and you still end up with desyncs, you either:

• Have a bug in your own code.
• Stumbled upon a bug in quantum code.

To find out which case is it, you can calculate checksums for each part of your simulation.
The idea is that if step N was in sync and step N+1 did desync (Frame checksums were different in different players, meaning Frame had different data), the problem lies between steps N and N+1.

To debug this, use this code:

quantum.state: ChecksumStep.qtn

C#

  // This gives a name to each checksum step. Feel free to replace it with your own values.
  enum ChecksumStep {
    None,
    ChecksumCheckSystem,
    PrePhysicsSystem,
    PhysicsSystemPre,
    CharacterInitSystem,
    MovementSystem,
    LootBoxSystemUpdate,
    LootBoxSystemCallbackPre,
    LootBoxSystemCallbackPost,
    ChecksumSystem
  }

quantum.state: checksums.qtn

  import ChecksumStep;
  
  struct PartialChecksum {
    ChecksumStep step;
    UInt64 checksum;
  }
  
  global {
    // increase 100 if you have more than 100 potential checksum steps.
    array<PartialChecksum>[100] stepChecksums;
  }

quantum.systems: ChecksumSystem.cs

C#

  using System.Collections.Generic;
  using System.Linq;
  
  namespace Quantum.Game {
    public class ChecksumSystem : SystemBase {
      public static readonly List<PartialChecksum> checksums = new List<PartialChecksum>();
  
      public override void Update(Frame f) => f.recordChecksum(ChecksumStep.ChecksumSystem);
  
      public static string status => string.Join("\n", checksums.Select(e => e.ToString()).ToArray());
    }
  
    public static unsafe class ChecksumSystemExts {
      public static void recordChecksum(this Frame f, ChecksumStep step) {
        var cs = new PartialChecksum { step = step, checksum = f.CalculateChecksum() };
        if (ChecksumSystem.checksums.Count >= f.Global->stepChecksumsSize) {
          Log.Error("Out of checksums!");
        }
        else {
          *f.Global->stepChecksums(ChecksumSystem.checksums.Count) = cs;
        }
  
        ChecksumSystem.checksums.Add(cs);
      }
    } 
  }

quantum.systems: ChecksumCheckSystem.cs

C#

  namespace Quantum.Game {
    public unsafe class ChecksumCheckSystem : SystemBase {
      public override void Update(Frame f) {
        for (var idx = 0; idx < f.Global->stepChecksumsSize; idx++) {
          *f.Global->stepChecksums(idx) = default;
        }
        ChecksumSystem.checksums.Clear();
        f.recordChecksum(ChecksumStep.ChecksumCheckSystem);
      }
    }
  }

quantum.systems: SystemSetup.cs

C#

  namespace Quantum {
    public static class SystemSetup {
      public static SystemBase[] CreateSystems(RuntimeConfig gameConfig, SimulationConfig simulationConfig) => 
        new SystemBase[] {
          new Game.ChecksumCheckSystem(),
  
          // other systems go here
          
          new Game.ChecksumSystem()
        };
    }
  }

unity: any class extending QuantumCallbacks

You need to dump the frame state, so you can compare them between clients. Example code from our game is given here, but you'll need to write your own, as this depends on our libraries.

C#

    string dumpFullFrame(Frame f) =>
      $"Frame[number: {f.Number} is predicted: {f.IsPredicted}, is verified: {f.IsVerified}, dump: {f.DumpFrame()}]";
  
    // random id for a client to discern between two clients running on same machine
    readonly uint randomId = Rng.now.nextUInt(out _);
    uint desyncNo;
    public override void OnChecksumError(DeterministicTickChecksumError error, Frame[] frames) {
      var checksumsS =
        error.Checksums
          .Select((cs, idx) => $"idx: {idx}, player: {cs.Player}, checksum: {cs.Checksum}")
          .mkString("\n");
  
      var msg = 
        $"Desync at tick {error.Tick}. Checksums:\n{checksumsS}\n\n" +
        $"Local checksums:\n{ChecksumSystem.status}\n\n";
      if (frames.find(f => f.Number == error.Tick).valueOut(out var badFrame)) {
        msg += $"Frame {error.Tick}: {dumpFullFrame(badFrame)}";
      }
      else {
        for (var idx = 0; idx < frames.Length; idx++) {
          var frame = frames[idx];
          msg += $"Frame {idx}: {dumpFullFrame(frame)}\n";
        }
      }
      if (Environment.GetEnvironmentVariable("LOCALAPPDATA").opt().valueOut(out var home)) {
        var path = PathStr.a(home) / $"desync_id{randomId}_no{desyncNo}_frame{error.Tick}.txt";
        File.WriteAllText(path, msg);
        Log.d.error($"Wrote desync log to {path}");
      }
      else {
        Log.d.error(msg);
      }
      desyncNo++;
    }

Regular C# version:

C#

    public void Start()
    {
        
        if ( randomId == 0U )
        {
            randomId = (uint) UnityEngine.Random.Range( int.MinValue, int.MaxValue );
        }        
    }
    
    string DumpFullFrame( Frame f ) => $"Frame[number: {f.Number} is predicted: {f.IsPredicted}, is verified: {f.IsVerified}, dump: {f.DumpFrame()}]";

    // random id for a client to discern between two clients running on same machine
    static uint randomId;
    uint desyncNo;

    public override void OnChecksumError( DeterministicTickChecksumError error, Frame[] frames )
    {
        var checksumsS =
    error.Checksums
      .Select((cs, idx) => $"idx: {idx}, player: {cs.Client}, checksum: {cs.Checksum}")
      .mkString("\n");

        var msg =
            $"Desync at tick {error.Tick}. Checksums:\n{checksumsS}\n\n" +
            $"Local checksums:\n{Quantum.Game.ChecksumSystem.status}\n\n";

        bool foundBadFrame = false;
        Frame badFrame = null;

        for ( int idx = 0; idx < frames.Length; idx++ )
        {
            if ( frames[idx].Number == error.Tick )
                badFrame = frames[idx];
        }

        if ( badFrame != null )
        {
            msg += $"Frame {error.Tick}: {DumpFullFrame( badFrame )}";
        }
        else {
            for (var idx = 0; idx<frames.Length; idx++) {
                var frame = frames[idx];
                msg += $"Frame {idx}: {DumpFullFrame( frame)}\n";
            }
        }

        var envPath = Environment.GetEnvironmentVariable("LOCALAPPDATA");
        if (envPath == null )
        {
            envPath = ".";
        }

        if ( envPath != null ) 
        {
            var path = Path.Combine( Path.GetDirectoryName(envPath) , $"desync_id{randomId}_no{desyncNo}_frame{error.Tick}.txt");
            File.WriteAllText(path, msg);
            Log.Error($"Wrote desync log to {path}");
        }
        else {
            Log.Error(msg);
        }
            desyncNo++;
    }

Usage

Record a step checksum like this: frame.recordChecksum(ChecksumStep.ChecksumCheckSystem);.
You can do this as often as you want, given that you do not run out of checksum steps.
If you do, increase number of them in quantum definition.

When a desync occurs, take log files from two clients for the same frame, compare them with a tool like kdiff3, and look at which step desynced.

For example:

We can see that after PhysicsSystemPre step, checksums desynced, which indicates either a problem with one of our physics signals or a problem with quantum physics simulation itself.