mixed-reality

Porting HoloLens (1st Gen) apps to HoloLens 2

This guide is tailored to help developers with an existing Unity application for HoloLens (1st gen) port their application to the HoloLens 2 device. There are four key steps to porting a HoloLens (1st gen) Unity application to HoloLens 2.

The sections below detail information for each stage:

Step 1	Step 2	Step 3	Step 4

Download latest tools	Update Unity Project	Compile for ARM	Migrate to MRTK v2

Prerequisites

We highly recommend that you use source control to save a snapshot your application’s original state before starting the porting process. We also recommend that you save checkpoint states at various times during the process. You may find it helpful to have another instance of the original application open in Unity so you can compare side-by-side during the porting process.

[!NOTE] Before porting, ensure that you have the latest tools installed for Windows Mixed Reality development. For most existing HoloLens developers, this involves updating to the latest version of Visual Studio 2019 and installing the appropriate Windows SDK. The content that follows dives further into different Unity versions and the Mixed Reality Toolkit (MRTK) Version 2.

For more information, see Install the tools.

Migrate your project to the latest version of Unity

If you’re using MRTK v2, we recommend that you update to MRTK 2.7 before you upgrade your project to Unity 2020.3 LTS. MRTK 2.7 supports Unity 2018, 2019 and 2020, allowing you to ensure your project is ready for Unity 2020 even before you upgrade Unity. Assess any plugin dependencies that currently exist in your project and determine whether these DLLs can be built for ARM64. For projects with a hard ARM dependent plugin, you may need to continue building your app for ARM.

Update scene/project settings in Unity

After updating to Unity 2020.3 LTS, we recommend that you update particular settings in Unity for optimal results on the device. These settings are outlined in detail under recommended settings for Unity.

To reiterate, the .NET scripting back-end is being deprecated in Unity 2018 and removed as of Unity 2019. You should strongly consider switching your project to IL2CPP.

[!NOTE] The IL2CPP scripting backend can cause longer build times from Unity to Visual Studio. Developers should set up their developer machine for optimizing IL2CPP build times. You might also benefit from setting up a cache server, especially for Unity projects with a large amount of assets (excluding script files) or constantly changing scenes and assets. When opening a project, Unity stores qualifying assets into an internal cache format on the developer machine. Items must be re-imported and re-processed when modified. This process can be done once, saved in a cache server, and then shared with other developers to save time, as opposed to every developer processing the re-import of new changes locally.

After addressing any breaking changes that result from moving to the updated Unity version, build and test your current applications on HoloLens (1st gen). This is a good time to create and save a commit into source control.

Compile dependencies/plugins for ARM processor

HoloLens (1st gen) executes applications on an x86 processor; the HoloLens 2 uses an ARM processor. Existing HoloLens applications need to be ported over to support ARM. As noted earlier, Unity 2018 LTS supports compiling ARM32 apps while Unity 2019 and later supports compiling ARM32 and ARM64 apps. Developing for ARM64 applications is preferred because there’s a material difference in performance. However, this requires all plugin dependencies to also be built for ARM64.

Review all DLL dependencies in your application. We recommend removing dependencies that are no longer needed for your project. For remaining plugins that are required, ingest the respective ARM32 or ARM64 binaries into your Unity project.

After ingesting the relevant DLLs, build a Visual Studio solution from Unity and compile an AppX for ARM in Visual Studio to to verify that your application can be built for ARM processors. We recommend that you save the application as a commit in your source control solution.

[!IMPORTANT] Applications using MRTK v1 can be run on HoloLens 2 after changing the build target to ARM, assuming that all other requirements are met. This includes making sure you have ARM versions of all your plugins. However, your app won’t have access to HoloLens 2-specific functions like articulated hand and eye tracking. MRTK v1 and MRTK v2 have different namespaces that allow both versions to be in the same project, which is useful for transitioning from one to the other.

Update to MRTK version 2

MRTK Version 2 is the new toolkit on top of Unity that supports both HoloLens (1st gen) and HoloLens 2. It’s also where all the new HoloLens 2 capabilities have been added, such as hand interactions and eye tracking.

Check out the following resources for more information on using MRTK version 2:

Prepare for the migration

Before ingesting the new *.unitypackage files for MRTK v2, we recommend that you take an inventory of (1) any custom-built code that integrates with MRTK v1, and (2) any custom-built code for input interactions or UX components. The most common and prevalent conflict for a mixed reality developer ingesting MRTK v2 involves input and interactions. We recommend that you review MRTK v2 input model.

Finally, the new MRTK v2 has transitioned from a model of scripts and in-scene manager objects to a configuration and services provider architecture. This results in a cleaner scene hierarchy and architecture model, but requires a learning curve for understanding the new configuration profiles. Read the Mixed Reality Toolkit Configuration Guide to start becoming familiar with the important settings and profiles that you must adjust to the needs of your application.

Migrating the project

After importing MRTK v2, your Unity project most likely has many compiler-related errors. These are common because of the new namespace structure and component names. Continue to resolve these errors by modifying your scripts to the new namespaces and components.

For information on the specific API differences between HTK/MRTK and MRTK v2, see the porting guide on the MRTK Version 2 wiki.

Best practices

Give priorty to the MRTK standard shader.
Work on one breaking change type at a time (example: IFocusable to IMixedRealityFocusHandler).
Test after every change and use source control.
Use default MRTK UX (buttons, slates, and so on), when possible.
Refrain from modifying MRTK files directly; create wrappers around MRTK components.
- This action eases future MRTK ingestion and updates.
Review and explore sample scenes provided in the MRTK, especially HandInteractionExamples.scene.
Rebuild canvas-based UI with quads, colliders, and TextMeshPro text.
Enable Depth Buffer Sharing or set focus point; for better performance, use a 16-bit depth buffer. Ensure that when you render color, you also render depth. Unity generally doesn’t write depth for transparent and text gameobjects.
Select Single Pass Instanced Rendering.
Use the HoloLens 2 configuration profile for MRTK

Testing your application

In MRTK Version 2, you can simulate hand interactions directly in Unity and develop with the new APIs for hand interactions and eye tracking. The HoloLens 2 device is required to create a satisfying user experience. We recommend that you study the documentation and tools for greater understanding. MRTK v2 supports development on HoloLens (1st gen) and traditional input models such as “select via air-tap” can be tested on HoloLens (1st gen).

Updating your interaction model for HoloLens 2

[!CAUTION] If your project is using any of the XR.WSA APIs, note that these are being phased out in favor of Unity’s new XR input APIs in future Unity releases. You can find more information about the XR input APIs here.

Once your application is ported and prepped for HoloLens 2, you’re ready to consider updating your interaction model and hologram design placements. In HoloLens (1st gen), your application likely has a gaze and commit interaction model with holograms relatively far away to fit into the field of view.

To update your application design to be best suited for HoloLens 2:

MRTK components: Per the pre-work, if you added MRTK v2, there are various components/scripts to leverage that have been designed and optimized for HoloLens 2.
Interaction model: Consider updating your interaction model. For most scenarios, we recommend that you switch from gaze and commit to hands. Some of your holograms may be out of arm’s reach, and switching to hands results in far interaction pointing rays and grab gestures.
Hologram placement: After switching to a hands interaction model, consider moving some holograms closer so users can directly interact with them by using near-interaction grab gestures with their hands. The types of holograms to move closer to directly grab or interact with are:
- small target menus
- controls
- buttons
- smaller holograms that, when grabbed and inspected, fit within the HoloLens 2 field of view.

Applications and scenarios vary; we’ll continue to refine and post design guidance based on feedback and continued learnings.

Additional tips about moving applications from x86 to ARM

Straightforward Unity applications are simple because you can build an ARM application bundle or deploy directly to the device for the bundle to run. Some Unity native plugins can present certain development challenges. Because of this, you must upgrade all Unity native plugins to Visual Studio 2019 and then rebuild for ARM.
One application used the Unity AudioKinetic Wwise plugin. The Unity version in use didn’t have a UWP ARM plugin, and there was a considerable effort to rework sound capabilities into the application in question to run on ARM. Ensure that all required plugins for your development plans are installed and available in Unity.
In some cases, a UWP/ARM plugin might not exist for application-required plugins, which blocks the ability to port the application and run it on HoloLens 2. Contact your plugin provider to resolve the issue and provide support for ARM.
The minfloat (and variants such as min16float, minint, and so on) in shaders might behave differently on HoloLens 2 than on HoloLens (1st gen). Specifically, these guarantee that at least the specified number of bits will be used. On Intel/Nvidia GPUs, minfloats are largely treated as 32 bits. On ARM, the number of bits specified is actually adhered to. In practice, these numbers might have less precision or range on HoloLens 2 than they did on HoloLens (1st gen).
The _asm instructions don’t appear to work on ARM, meaning any code using _asm instructions must be rewritten.
ARM doesn’t support the SIMD instruction set because various headers, such as xmmintrin.h, emmintrin.h, tmmintrin.h, and immintrin.h, aren’t available on ARM.
The shader compiler on ARM runs during the first draw call after the shader has been loaded or something the shader relies on has changed, not at shader load time. The impact on framerate can be noticeable, depending on how many shaders need to be compiled, with implications for how shaders should be handled, packaged, and updated differently on HoloLens 2 vs HoloLens (1st gen).