This document is about: FUSION 2
SWITCH TO

VR Shared - Local rig grabbing

Level 4
This page describes an alternative implementation of a grabbing system for the VR Shared technical sample. To understand how the rig is setup, or any other details regarding this sample, please refer to the main VR Shared page first.

Differences with the basic VR Shared sample grabbing logic

This grabbing system, while very similar to the one described in the VR Shared technical sample, has some differences that can be useful in some situations:

  • the collider detecting grabbable objects is localized on the hardware rig instead of the network rig
  • this way, it can be used even when the network rig is not yet spawned (like full offline situations)
  • besides, it can be used in situation where on purpose we place the network rig at a position differing form the hardware rig (if the network rig position is smoothed during teleportation, ...)

Downloads

This sample is included in the VR Shared page download. The scene demonstrating the local rig grabbing is located in the Scenes/AlternativeHardwareBasedGrabbingDemo folder.

Overview

The grabbing logic here is separated in 2 parts:

  • the local part, non networked, that detected the actual grabbing/ungrabbing when the hardware hand has triggered a grab/ungrab action over a grabbable object (Grabber and Grabbable classes)
  • the networked part, that ensure that all players are aware of the grabbing status, and which manages the actual position change to follow the grabbing hand (NetworkGrabber and NetworkGrabbable classes).
fusion vr shared local rig grabbing logic

Note: the code contains a few lines to allow the local part to manage the following itself when used offline, for instance in use cases where the same components are used for an offline lobby. But this document will focus on the actual networked usages.

Details

Grabbing

The HardwareHand class, located on each hand, updates the isGrabbing bool at each update : the bool is true when the user presses the grip button. Please note that, the updateGrabWithAction bool is used to support the deskop rig, a version of the rig that can be driven with the mouse and keyboard (this bool must be set to False for desktop mode, True for VR mode)

C#

 protected virtual void Update()
 {
    // update hand pose
    // (...)

    // update hand interaction
    if(updateGrabWithAction) isGrabbing = grabAction.action.ReadValue<float>() > grabThreshold;
}

To detect collisions with grabbable objects, a simple box collider is located on each hardware hand, used by a Grabber component placed on this hand: when a collision occurs, the method OnTriggerStay(Collider other) is called.

First, it checks if an object is already grabbed. For simplification, multiple grabbing is not allowed in this sample.

C#

// Exit if an object is already grabbed
if (grabbedObject != null)
{
    // It is already the grabbed object or another, but we don't allow shared grabbing here
    return;
}

Then it checks that :

  • the collided object can be grabbed (it has a Grabbable component)
  • the user presses the grip button

If these conditions are met, the grabbed object is asked to follow the hand ((1) in the diagram above) thanks to the Grabbable Grab method

C#

Grabbable grabbable;

if (lastCheckedCollider == other)
{
    grabbable = lastCheckColliderGrabbable;
}
else
{
    grabbable = other.GetComponentInParent<Grabbable>();
}
// To limit the number of GetComponent calls, we cache the latest checked collider grabbable result
lastCheckedCollider = other;
lastCheckColliderGrabbable = grabbable;
if (grabbable != null)
{
    if(wasHovered || grabbable.allowedClosedHandGrabing)
    {
        Grab(grabbable);
    }
}

Grabbing synchronization

The Grabbable Grab() method stores the grabbing position offset. It also tells the NetworkGrabbable associated component that a grab by the local user occured ((2) in the diagram above), through the networkGrabbable.LocalGrab() call

C#

public virtual void Grab(Grabber newGrabber, Transform grabPointTransform = null)
{
    if (onWillGrab != null) onWillGrab.Invoke(newGrabber);

    // Find grabbable position/rotation in grabber referential
    localPositionOffset = newGrabber.transform.InverseTransformPoint(transform.position);
    localRotationOffset = Quaternion.Inverse(newGrabber.transform.rotation) * transform.rotation;
    currentGrabber = newGrabber;

    if (networkGrabbable)
    {
        networkGrabbable.LocalGrab();
    }
    else
    {
        // We handle the following if we are not online (online, the DidGrab will be called by the NetworkGrabbable DidGrab, itself called on all clients by HandleGrabberChange when the grabber networked var has changed)
        LockObjectPhysics();
    }
}

On the NetworkGrabbable, the LocalGrab() call :

  • first request the state authority on the object: this way, it will be possible to store [Networked] attributes values
  • once the state authority is received, it stores in those [Networked] attributes the details describing the grabbing (the grabber and the offset)

C#

public async virtual void LocalGrab()
{
    // Ask and wait to receive the stateAuthority to move the object
    isTakingAuthority = true;
    await Object.WaitForStateAuthority();
    isTakingAuthority = false;

    // We waited to have the state authority before setting Networked vars
    LocalPositionOffset = grabbable.localPositionOffset;
    LocalRotationOffset = grabbable.localRotationOffset;

    if(grabbable.currentGrabber==null)
    {
        // The grabbable has already been ungrabbed
        return;
    }

    // Update the CurrentGrabber in order to start following position in the FixedUpdateNetwork
    CurrentGrabber = grabbable.currentGrabber.networkGrabber;
}

Note: WaitForStateAuthority is an helper extension method

C#

public static async Task<bool> WaitForStateAuthority(this NetworkObject o, float maxWaitTime = 8)
{
    float waitStartTime = Time.time;
    o.RequestStateAuthority();
    while (!o.HasStateAuthority && (Time.time - waitStartTime) < maxWaitTime)
    {
        await System.Threading.Tasks.Task.Delay(1);
    }
    return o.HasStateAuthority;
}

ChangeDetector is used to detect a modification to the [Networked] variable CurrentGrabber on all clients ((3) in the diagram above):

Because the FixedUpdateNetwork() is not called on proxies (remote player without state authority on the network object), two ChangeDetector are required to detect modification on networked variables. The grabbing logic in handled in the FixedUpdateNetwork() for the state authority while the callbacks are called by all users during the Render().

The helper method TryDetectGrabberChange() is used in both cases.

C#


    ChangeDetector funChangeDetector;
    ChangeDetector renderChangeDetector;

    public override void Spawned()
    {
    [...]
        funChangeDetector = GetChangeDetector(NetworkBehaviour.ChangeDetector.Source.SimulationState);
        renderChangeDetector = GetChangeDetector(NetworkBehaviour.ChangeDetector.Source.SnapshotFrom);
    }

    public override void FixedUpdateNetwork()
    {
        // Check if the grabber changed
        if (TryDetectGrabberChange(funChangeDetector, out var previousGrabber, out var currentGrabber))
        {
            if (previousGrabber)
            {
                // Object ungrabbed
                UnlockObjectPhysics();
            }
            if (currentGrabber)
            {
                // Object grabbed
                LockObjectPhysics();
            }
        }
    [...]
    }

    public override void Render()
    {
        // Check if the grabber changed, to trigger callbacks only (actual grabbing logic in handled in FUN for the state authority)
        // Those callbacks can't be called in FUN, as FUN is not called on proxies, while render is called for everybody
        if (TryDetectGrabberChange(renderChangeDetector, out var previousGrabber, out var currentGrabber))
        {
            if (previousGrabber)
            {
                if (onDidUngrab != null) onDidUngrab.Invoke();
            }
            if (currentGrabber)
            {
                if (onDidGrab != null) onDidGrab.Invoke(currentGrabber);
            }
        }
    [...]
    }

    bool TryDetectGrabberChange(ChangeDetector changeDetector, out NetworkHandColliderGrabber previousGrabber, out NetworkHandColliderGrabber currentGrabber)
    {
        previousGrabber = null;
        currentGrabber = null;
        foreach (var changedNetworkedVarName in changeDetector.DetectChanges(this, out var previous, out var current))
        {
            if (changedNetworkedVarName == nameof(CurrentGrabber))
            {
                var grabberReader = GetBehaviourReader<NetworkHandColliderGrabber>(changedNetworkedVarName);
                previousGrabber = grabberReader.Read(previous);
                currentGrabber = grabberReader.Read(current);
                return true;
            }
        }
        return false;
    }

This way, the state authority can properly set the isKinematic value (the object physics is disabled when grabbed), and apply release velocity.

C#

public virtual void LockObjectPhysics()
{
    // While grabbed, we disable physics forces on the object, to force a position based tracking
    if (rb) rb.isKinematic = true;
}

public virtual void UnlockObjectPhysics()
{
    // We restore the default isKinematic state if needed
    if (rb) rb.isKinematic = expectedIsKinematic;

    // We apply release velocity if needed
    if (rb && rb.isKinematic == false && applyVelocityOnRelease)
    {
        rb.velocity = Velocity;
        rb.angularVelocity = AngularVelocity;
    }

    ResetVelocityTracking();
}

Follow

As the object physics is disabled while grabbed, following the current grabber is simply teleporting to its actual position in the NetworkGrabbable. This is done on the FixedUpdateNetwork() for the state authority and then the NetworkTransform ensures that all players receive the position updates.

C#

public override void FixedUpdateNetwork()
{
    [...]

    // We only update the object position if we have the state authority
    if (!Object.HasStateAuthority) return;

    if (!IsGrabbed) return;
    // Follow grabber, adding position/rotation offsets
    grabbable.Follow(followedTransform: CurrentGrabber.transform, LocalPositionOffset, LocalRotationOffset);
}

The Follow() method is located in the Grabbable class.

C#

public virtual void Follow(Transform followingtransform, Transform followedTransform, Vector3 localPositionOffsetToFollowed, Quaternion localRotationOffsetTofollowed)
{
    followingtransform.position = followedTransform.TransformPoint(localPositionOffsetToFollowed);
    followingtransform.rotation = followedTransform.rotation * localRotationOffsetTofollowed;
}

Render

Regarding the extrapolation ((6) in the diagram above), made during the Render() (the NetworkGrabbable class has a OrderAfter directive to override the NetworkTransform interpolation if needed ), 2 cases have to be handled here :

  • extrapolation while the object authority is requested, for the grabbing client: during a few frames, CurrentGrabber is not yet set has the authority request is still pending ([Networked] var can only be set while having the state authority). So IsGrabbed won't yet return true, and the actual NetworkGrabber to follow has to be found through the local Grabbable and Grabber components.
  • extrapolation while the object is grabbed, for all client: the object expected position is known, it should be on the hand position.

C#

public override void Render()
{
    [...]

    if (isTakingAuthority && extrapolateWhileTakingAuthority)
    {
        // If we are currently taking the authority on the object due to a grab, the network info are still not set
        //  but we will extrapolate anyway (if the option extrapolateWhileTakingAuthority is true) to avoid having the grabbed object staying still until we receive the authority
        ExtrapolateWhileTakingAuthority();
        return;
    }

    // No need to extrapolate if the object is not grabbed
    if (!IsGrabbed) return;

    // Extrapolation: Make visual representation follow grabber, adding position/rotation offsets
    // We extrapolate for all users: we know that the grabbed object should follow accuratly the grabber, even if the network position might be a bit out of sync
    grabbable.Follow(followedTransform: CurrentGrabber.hand.transform, LocalPositionOffset, LocalRotationOffset);
}

protected virtual void ExtrapolateWhileTakingAuthority()
{
    // No need to extrapolate if the object is not really grabbed
    if (grabbable.currentGrabber == null) return;
    NetworkGrabber networkGrabber = grabbable.currentGrabber.networkGrabber;

    // Extrapolation: Make visual representation follow grabber, adding position/rotation offsets
    // We use grabberWhileTakingAuthority instead of CurrentGrabber as we are currently waiting for the authority transfer: the network vars are not already set, so we use the temporary versions
    grabbable.Follow(followedTransform: networkGrabber.hand.transform, grabbable.localPositionOffset, grabbable.localRotationOffset);
}
Back to top