The “WebCam” capability must be declared for an app to use the camera.
Only a single operation can occur with the camera at a time. You can check which mode the camera is currently in with UnityEngine.XR.WSA.WebCam.Mode in Unity 2018 and earlier or UnityEngine.Windows.WebCam.Mode in Unity 2019 and later. Available modes are photo, video, or none.
Namespace (before Unity 2019): UnityEngine.XR.WSA.WebCam
Namespace (Unity 2019 and later): UnityEngine.Windows.WebCam
Type: PhotoCapture
The PhotoCapture type allows you to take still photographs with the Photo Video Camera. The general pattern for using PhotoCapture to take a photo is as follows:
For all three uses, start with the same first three steps above
Start by creating a PhotoCapture object
private void Start()
{
PhotoCapture.CreateAsync(false, OnPhotoCaptureCreated);
}
Next, store your object, set your parameters, and start Photo Mode
private PhotoCapture photoCaptureObject = null;
void OnPhotoCaptureCreated(PhotoCapture captureObject)
{
photoCaptureObject = captureObject;
Resolution cameraResolution = PhotoCapture.SupportedResolutions.OrderByDescending((res) => res.width * res.height).First();
CameraParameters c = new CameraParameters();
c.hologramOpacity = 0.0f;
c.cameraResolutionWidth = cameraResolution.width;
c.cameraResolutionHeight = cameraResolution.height;
c.pixelFormat = CapturePixelFormat.BGRA32;
captureObject.StartPhotoModeAsync(c, false, OnPhotoModeStarted);
}
In the end, you’ll also use the same clean-up code presented here
void OnStoppedPhotoMode(PhotoCapture.PhotoCaptureResult result)
{
photoCaptureObject.Dispose();
photoCaptureObject = null;
}
After these steps, you can choose which type of photo to capture.
The simplest operation is to capture a photo directly to a file. The photo can be saved as a JPG or a PNG.
If you successfully started photo mode, take a photo and store it on disk
private void OnPhotoModeStarted(PhotoCapture.PhotoCaptureResult result)
{
if (result.success)
{
string filename = string.Format(@"CapturedImage{0}_n.jpg", Time.time);
string filePath = System.IO.Path.Combine(Application.persistentDataPath, filename);
photoCaptureObject.TakePhotoAsync(filePath, PhotoCaptureFileOutputFormat.JPG, OnCapturedPhotoToDisk);
}
else
{
Debug.LogError("Unable to start photo mode!");
}
}
After capturing the photo to disk, exit photo mode and then clean up your objects
void OnCapturedPhotoToDisk(PhotoCapture.PhotoCaptureResult result)
{
if (result.success)
{
Debug.Log("Saved Photo to disk!");
photoCaptureObject.StopPhotoModeAsync(OnStoppedPhotoMode);
}
else
{
Debug.Log("Failed to save Photo to disk");
}
}
When capturing data to a Texture2D, the process is similar to capturing to disk.
Follow the setup process above.
In OnPhotoModeStarted, capture a frame to memory.
private void OnPhotoModeStarted(PhotoCapture.PhotoCaptureResult result)
{
if (result.success)
{
photoCaptureObject.TakePhotoAsync(OnCapturedPhotoToMemory);
}
else
{
Debug.LogError("Unable to start photo mode!");
}
}
You’ll then apply your result to a texture and use the common clean-up code above.
void OnCapturedPhotoToMemory(PhotoCapture.PhotoCaptureResult result, PhotoCaptureFrame photoCaptureFrame)
{
if (result.success)
{
// Create our Texture2D for use and set the correct resolution
Resolution cameraResolution = PhotoCapture.SupportedResolutions.OrderByDescending((res) => res.width * res.height).First();
Texture2D targetTexture = new Texture2D(cameraResolution.width, cameraResolution.height);
// Copy the raw image data into our target texture
photoCaptureFrame.UploadImageDataToTexture(targetTexture);
// Do as we wish with the texture such as apply it to a material, etc.
}
// Clean up
photoCaptureObject.StopPhotoModeAsync(OnStoppedPhotoMode);
}
To place this texture in the scene and display it using the locatable camera matrices, add the following code to OnCapturedPhotoToMemory in the result.success check:
if (photoCaptureFrame.hasLocationData)
{
photoCaptureFrame.TryGetCameraToWorldMatrix(out Matrix4x4 cameraToWorldMatrix);
Vector3 position = cameraToWorldMatrix.GetColumn(3) - cameraToWorldMatrix.GetColumn(2);
Quaternion rotation = Quaternion.LookRotation(-cameraToWorldMatrix.GetColumn(2), cameraToWorldMatrix.GetColumn(1));
photoCaptureFrame.TryGetProjectionMatrix(Camera.main.nearClipPlane, Camera.main.farClipPlane, out Matrix4x4 projectionMatrix);
}
Unity has provided sample code for applying the projection matrix to a specific shader on their forums.
To interact with the raw bytes of an in memory frame, follow the same setup steps as above and OnPhotoModeStarted as in capturing a photo to a Texture2D. The difference is in OnCapturedPhotoToMemory where you can get the raw bytes and interact with them.
In this example, you’ll create a List to be further processed or applied to a texture via SetPixels()
void OnCapturedPhotoToMemory(PhotoCapture.PhotoCaptureResult result, PhotoCaptureFrame photoCaptureFrame)
{
if (result.success)
{
List<byte> imageBufferList = new List<byte>();
// Copy the raw IMFMediaBuffer data into our empty byte list.
photoCaptureFrame.CopyRawImageDataIntoBuffer(imageBufferList);
// In this example, we captured the image using the BGRA32 format.
// So our stride will be 4 since we have a byte for each rgba channel.
// The raw image data will also be flipped so we access our pixel data
// in the reverse order.
int stride = 4;
float denominator = 1.0f / 255.0f;
List<Color> colorArray = new List<Color>();
for (int i = imageBufferList.Count - 1; i >= 0; i -= stride)
{
float a = (int)(imageBufferList[i - 0]) * denominator;
float r = (int)(imageBufferList[i - 1]) * denominator;
float g = (int)(imageBufferList[i - 2]) * denominator;
float b = (int)(imageBufferList[i - 3]) * denominator;
colorArray.Add(new Color(r, g, b, a));
}
// Now we could do something with the array such as texture.SetPixels() or run image processing on the list
}
photoCaptureObject.StopPhotoModeAsync(OnStoppedPhotoMode);
}
Namespace (before Unity 2019): UnityEngine.XR.WSA.WebCam
Namespace (Unity 2019 and later): UnityEngine.Windows.WebCam
Type: VideoCapture
VideoCapture functions similarly to PhotoCapture. The only two differences are that you must specify a Frames Per Second (FPS) value and you can only save directly to disk as a .mp4 file. The steps to use VideoCapture are as follows:
Start by creating our VideoCapture object VideoCapture m_VideoCapture = null;
void Start ()
{
VideoCapture.CreateAsync(false, OnVideoCaptureCreated);
}
Next, set up the parameters you’ll want for the recording and start.
void OnVideoCaptureCreated(VideoCapture videoCapture)
{
if (videoCapture != null)
{
m_VideoCapture = videoCapture;
Resolution cameraResolution = VideoCapture.SupportedResolutions.OrderByDescending((res) => res.width * res.height).First();
float cameraFramerate = VideoCapture.GetSupportedFrameRatesForResolution(cameraResolution).OrderByDescending((fps) => fps).First();
CameraParameters cameraParameters = new CameraParameters();
cameraParameters.hologramOpacity = 0.0f;
cameraParameters.frameRate = cameraFramerate;
cameraParameters.cameraResolutionWidth = cameraResolution.width;
cameraParameters.cameraResolutionHeight = cameraResolution.height;
cameraParameters.pixelFormat = CapturePixelFormat.BGRA32;
m_VideoCapture.StartVideoModeAsync(cameraParameters,
VideoCapture.AudioState.None,
OnStartedVideoCaptureMode);
}
else
{
Debug.LogError("Failed to create VideoCapture Instance!");
}
}
Once started, begin the recording
void OnStartedVideoCaptureMode(VideoCapture.VideoCaptureResult result)
{
if (result.success)
{
string filename = string.Format("MyVideo_{0}.mp4", Time.time);
string filepath = System.IO.Path.Combine(Application.persistentDataPath, filename);
m_VideoCapture.StartRecordingAsync(filepath, OnStartedRecordingVideo);
}
}
After recording has started, you could update your UI or behaviors to enable stopping. Here you just log.
void OnStartedRecordingVideo(VideoCapture.VideoCaptureResult result)
{
Debug.Log("Started Recording Video!");
// We will stop the video from recording via other input such as a timer or a tap, etc.
}
At a later point, you’ll want to stop the recording using a timer or user input, for instance.
// The user has indicated to stop recording
void StopRecordingVideo()
{
m_VideoCapture.StopRecordingAsync(OnStoppedRecordingVideo);
}
Once the recording has stopped, stop video mode and clean up your resources.
void OnStoppedRecordingVideo(VideoCapture.VideoCaptureResult result)
{
Debug.Log("Stopped Recording Video!");
m_VideoCapture.StopVideoModeAsync(OnStoppedVideoCaptureMode);
}
void OnStoppedVideoCaptureMode(VideoCapture.VideoCaptureResult result)
{
m_VideoCapture.Dispose();
m_VideoCapture = null;
}
If you’re following the Unity development checkpoint journey we’ve laid out, you’re in the midst of exploring the Mixed Reality platform capabilities and APIs. From here, you can continue to the next topic:
[!div class=”nextstepaction”] Focus point
Or jump directly to deploying your app on a device or emulator:
[!div class=”nextstepaction”] Deploy to HoloLens or Windows Mixed Reality immersive headsets
You can always go back to the Unity development checkpoints at any time.