mixed-reality
MRTK
Spatial awareness system
In MRTK, look at the Spatial awareness getting started guide for information on setting up various spatial mesh observers.
For information on on-device observers, look at the Configuring mesh observers for device guide.
For information on scene understanding observers, look at the Scene understanding observer guide.
XR SDK
ARMeshManager
Unity’s ARFoundation provides an ARMeshManager component for built-in visualization of spatial meshes. See Unity’s documentation for more information on usage.
XRMeshSubsystem
If you’d rather work with Unity’s XRMeshSubsystem directly, see Unity’s documentation for more information on usage.
Windows XR Plugin
Windows XR Plugin provides some additional extension methods for configuring the XRMeshSubsystem, such as setting a bounding sphere or accessing the underlying platform mesh representation. All of these other extensions can be found in Unity’s documentation.
Legacy WSA
Getting started with Unity’s built-in spatial mapping components
Unity offers two components for easily adding spatial mapping to your app, Spatial Mapping Renderer and Spatial Mapping Collider.
Spatial Mapping Renderer
The Spatial Mapping Renderer allows for visualization of the spatial mapping mesh.
Spatial Mapping Collider
The Spatial Mapping Collider allows for holographic content (or character) interaction, such as physics, with the spatial mapping mesh.
Using the built-in spatial mapping components
You may add both components to your app if you’d like to both visualize and interact with physical surfaces.
To use these two components in your Unity app:
- Select a GameObject at the center of the area in which you’d like to detect spatial surface meshes.
- In the Inspector window, Add Component > XR > Spatial Mapping Collider or Spatial Mapping Renderer.
You can find more details on how to use these components at the Unity documentation site.
Going beyond the built-in spatial mapping components
These components make it drag-and-drop easy to get started with Spatial Mapping. When you want to go further, there are two main paths to explore:
- To do your own lower-level mesh processing, see the section below about the low-level Spatial Mapping script API.
- To do higher-level mesh analysis, see the section below about the SpatialUnderstanding library in MixedRealityToolkit.
Using the low-level Unity spatial mapping API
If you need more control than the Spatial Mapping Renderer and Spatial Mapping Collider components offer, use the low-level Spatial Mapping APIs.
Namespace: UnityEngine.XR.WSA
Types: SurfaceObserver, SurfaceChange, SurfaceData, SurfaceId
We’ve outlined the suggested flow for an application that uses the spatial mapping APIs in the sections below.
Set up the SurfaceObserver(s)
Instantiate one SurfaceObserver object for each application-defined region of space that you need spatial mapping data for.
SurfaceObserver surfaceObserver;
private void Start()
{
surfaceObserver = new SurfaceObserver();
}
Specify the region of space that each SurfaceObserver object provide datas for by calling SetVolumeAsSphere, SetVolumeAsAxisAlignedBox, SetVolumeAsOrientedBox, or SetVolumeAsFrustum. You can redefine the region of space in the future by calling one of these methods again.
private void Start()
{
surfaceObserver.SetVolumeAsAxisAlignedBox(Vector3.zero, new Vector3(3, 3, 3));
}
When you call SurfaceObserver.Update(), you must provide a handler for each spatial surface in the SurfaceObserver’s region of space that the spatial mapping system has new information for. The handler receives, for one spatial surface:
private void OnSurfaceChanged(SurfaceId surfaceId, SurfaceChange changeType, Bounds bounds, System.DateTime updateTime)
{
// see Handling Surface Changes
}
Handling surface changes
There are several main cases to handle: added and updated, which can use the same code path, and removed.
- In the added and updated cases, we add or get the GameObject representing this mesh from the dictionary. We create a SurfaceData struct with the necessary components, then call RequestMeshDataAsync to populate the GameObject with the mesh data, and then position it in the scene.
- In the removed case, we remove the GameObject representing this mesh from the dictionary and destroy it.
System.Collections.Generic.Dictionary<SurfaceId, GameObject> spatialMeshObjects =
new System.Collections.Generic.Dictionary<SurfaceId, GameObject>();
private void OnSurfaceChanged(SurfaceId surfaceId, SurfaceChange changeType, Bounds bounds, System.DateTime updateTime)
{
switch (changeType)
{
case SurfaceChange.Added:
case SurfaceChange.Updated:
if (!spatialMeshObjects.ContainsKey(surfaceId))
{
spatialMeshObjects[surfaceId] = new GameObject("spatial-mapping-" + surfaceId);
spatialMeshObjects[surfaceId].transform.parent = this.transform;
spatialMeshObjects[surfaceId].AddComponent<MeshRenderer>();
}
GameObject target = spatialMeshObjects[surfaceId];
SurfaceData sd = new SurfaceData(
// the surface id returned from the system
surfaceId,
// the mesh filter that is populated with the spatial mapping data for this mesh
target.GetComponent<MeshFilter>() ?? target.AddComponent<MeshFilter>(),
// the world anchor used to position the spatial mapping mesh in the world
target.GetComponent<WorldAnchor>() ?? target.AddComponent<WorldAnchor>(),
// the mesh collider that is populated with collider data for this mesh, if true is passed to bakeMeshes below
target.GetComponent<MeshCollider>() ?? target.AddComponent<MeshCollider>(),
// triangles per cubic meter requested for this mesh
1000,
// bakeMeshes - if true, the mesh collider is populated, if false, the mesh collider is empty.
true
);
SurfaceObserver.RequestMeshAsync(sd, OnDataReady);
break;
case SurfaceChange.Removed:
var obj = spatialMeshObjects[surfaceId];
spatialMeshObjects.Remove(surfaceId);
if (obj != null)
{
GameObject.Destroy(obj);
}
break;
default:
break;
}
}
Handling data ready
The OnDataReady handler receives a SurfaceData object. The WorldAnchor, MeshFilter, and (optionally) MeshCollider objects it contains reflect the latest state of the associated spatial surface. Optionally, analyze and/or process the mesh data by accessing the Mesh member of the MeshFilter object. Render the spatial surface with the latest mesh and (optionally) use it for physics collisions and raycasts. It’s important to confirm that the contents of the SurfaceData aren’t null.
Start processing on updates
SurfaceObserver.Update() should be called on a delay, not every frame.
void Start ()
{
StartCoroutine(UpdateLoop());
}
IEnumerator UpdateLoop()
{
var wait = new WaitForSeconds(2.5f);
while (true)
{
surfaceObserver.Update(OnSurfaceChanged);
yield return wait;
}
}