mixed-reality

Locatable camera overview

HoloLens includes a world-facing camera mounted on the front of the device that enables apps to see what the user sees. Developers have access to and control of the camera, just as they would for color cameras on smartphones, portables, or desktops. The same universal Windows media capture and Windows Media Foundation APIs that work on mobile and desktop work on HoloLens. Unity has wrapped these windows APIs to abstract camera usage features on HoloLens. Feature tasks include taking regular photos and videos (with or without holograms) and locating the camera’s position in and perspective on the scene.

Device camera information

HoloLens (first-generation)

HoloLens 2

[!NOTE] Customers can leverage mixed reality capture to take videos or photos of your app that include holograms and employ video stabilization.

If you want the content of your user’s capture to look as good as possible, there are some things you should consider. You can also enable (and customize) mixed reality capture from directly within your app. Learn more at mixed reality capture for developers.

Locating the Device Camera in the World

When HoloLens takes photos and videos, the captured frames include the location of the camera in the world and the lens model of the camera. This information allows applications to reason about the position of the camera in the real world for augmented imaging scenarios. Developers can creatively roll their own scenarios using their favorite image processing or custom computer vision libraries.

“Camera” elsewhere in HoloLens documentation may refer to the “virtual game camera” (the frustum the app renders to). Unless described otherwise, “camera” on this page refers to the real-world RGB color camera.

Distortion Error

On HoloLens 2, the video and still image streams are undistorted in the system’s image-processing pipeline before the frames are made available to the application. The preview stream contains the original distorted frames. Because only the CameraIntrinsics are made available, applications must assume that image frames represent a perfect pinhole camera.

On HoloLens (first-generation), the undistortion function in the image processor may still leave an error of up to 10 pixels when using the CameraIntrinsics in the frame metadata. In many use cases, this error won’t matter. However, if, for example, you’re aligning holograms to real-world posters or markers and you notice a < 10-px offset (roughly 11 mm for holograms positioned 2 meters away), this distortion error could be the cause.

Locatable Camera Usage Scenarios

Show a photo or video in the world where it was captured

The Device Camera frames come with a “Camera To World” transform that can be used to show exactly where the device was when it captured the image. For example, you could position a small holographic icon at this location (CameraToWorld.MultiplyPoint(Vector3.zero)) and even draw a little arrow in the direction that the camera was facing (CameraToWorld.MultiplyVector(Vector3.forward)).

Tag / Pattern / Poster / Object Tracking

Many mixed reality applications use a recognizable image or visual pattern to create a trackable point in space. An application can render objects relative to that point or create a known location. A typical use for HoloLens is finding a real-world object that’s tagged with fiducials. This might occur, for example, on tablets that have been set up to communicate with HoloLens via Wi-Fi.

You’ll need a few things to recognize a visual pattern and place an object in the application’s world space:

  1. An image pattern recognition toolkit, such as QR code, AR tags, face finder, circle trackers, OCR, and so on.
  2. Collect image frames at runtime and pass them to the recognition layer.
  3. Unproject their image locations back into world positions or likely world rays.
  4. Position your virtual models over these world locations.

Some important image-processing links:

Keeping an interactive application frame-rate is critical, especially when dealing with long-running image recognition algorithms. For this reason, we commonly use the following pattern:

  1. Main Thread: manages the camera object.
  2. Main Thread: requests new frames (async).
  3. Main Thread: pass new frames to tracking thread.
  4. Tracking Thread: processes image to collect key points.
  5. Main Thread: moves virtual model to match found key points.
  6. Main Thread: repeat from step 2.

Some image marker systems only provide a single-pixel location, which equates to a ray of possible locations. (Others provide the full transform, in which case this section isn’t needed.) To get to a single 3D location, we can calculate multiple rays and find the final result by their approximate intersection. To get this result, you’ll need to:

  1. Create a loop that collects multiple camera images.
  2. Find the associated feature points and their world rays.

Given two or more tracked tag locations, you can position a modeled scene to fit the user’s current scenario. If you can’t assume gravity, then you’ll need three tag locations. In many cases, we use a color scheme where white spheres represent real-time tracked tag locations, and blue spheres represent modeled tag locations. This allows the user to visually gauge the alignment quality. We assume the following setup in all our applications:

Track or identify tagged stationary or moving real-world objects/faces using LEDs or other recognizer libraries

Examples:

See also

This site is open source. Improve this page.