Quick Start
Please note that the code provided in this page is purely for learning purposes and is far from perfect. Remember to null-check all responses!
Breaking Changes Notice
If you've just updated the package, it is recommended to check the changelogs for information on breaking changes.
Setup
Dependencies
UXR.QuestCamera uses an Android native AAR plugin, written in Kotlin, to access the Android Camera2 API. The plugin thus requires the External Dependency Manager for Unity (EDM4U) package to add the android.hardware.camera2 package. If your project uses Firebase or Google SDKs, it is likely already installed. If not, you can see the installation steps here: https://github.com/googlesamples/unity-jar-resolver?tab=readme-ov-file#getting-started
Unity Compatibility
The package uses some Awaitable
methods for switching between the JNI and Unity threads, for frame processing. Since
Awaitable
was only added in Unity 6, you will have to install com.utilities.async by Stephen Hodgson on older versions
of Unity. You can see the installation steps here: https://github.com/RageAgainstThePixel/com.utilities.async
Installation
To install UXR.QuestCamera:
- Go to
Project Settings
in your Unity project - Under
Package Manager
, inScoped Registries
, create a new registry with the following settings:- Name:
OpenUPM
- URL:
https://package.openupm.com
- Scope(s):
com.uralstech
- Name:
- Click
Apply
- Open the package manager and go to
My Registries
->OpenUPM
- Install UXR.QuestCamera >=
2.2.0
AndroidManifest.xml
Add the following to your project's AndroidManifest.xml
file:
<uses-feature android:name="android.hardware.camera2.any" android:required="true"/>
<uses-permission android:name="horizonos.permission.HEADSET_CAMERA" android:required="true"/>
The HEADSET_CAMERA
permission is required by Horizon OS for apps to be able to access the headset cameras.
You will have to request it at runtime, like so:
if (!Permission.HasUserAuthorizedPermission(UCameraManager.HeadsetCameraPermission))
Permission.RequestUserPermission(UCameraManager.HeadsetCameraPermission);
Usage
Device Support
Since the Passthrough Camera API is restricted to the Quest 3 family and Horizon OS version >= 74, you have to check
if it is supported by the current device before doing anything. Just check the static property CameraSupport.IsSupported
before doing anything.
Choosing the Camera
UCameraManager
is the script that will allow you to open and close camera devices. It is a persistent singleton, so add
it to the first scene that is loaded in your app, so that it can be referenced from all other scripts at all times.
You can add the QuestCameraManager
prefab, which contains an instance of UCameraManager
, along with the default YUV to
RGBA conversion compute shader, by right clicking on the Hierarchy and clicking on Quest Camera Manager
, under Quest Camera
.
To start getting images, you first have to open the camera device. You can have multiple cameras open at the same time, as long as the system allows it.
You can get an array of all available cameras in UCameraManager.Cameras
, or you can request the camera nearest to the left or
right eye of the user by calling UCameraManager.GetCamera(CameraInfo.CameraEye)
. The information for each camera, like supported
resolutions, position and rotation relative to the headset, focal length, and more are stored in CameraInfo
objects.
Once you've chosen a camera, you have to select a resolution for the images. You can get them from CameraInfo.SupportedResolutions
.
For example, you can get the highest supported resolution with the following code:
Resolution highestResolution = default;
foreach (Resolution resolution in cameraInfo.SupportedResolutions)
{
if (resolution.width * resolution.height > highestResolution.width * highestResolution.height)
highestResolution = resolution;
}
Now you're ready to open the camera and start a capture session!
Opening the Camera
You can open the camera by calling UCameraManager.OpenCamera(cameraId)
. You can pass the previous CameraInfo
object into this function, as CameraInfo
will implicitly return its camera's ID as a string.
The function returns a CameraDevice
object, which is a wrapper for the native Camera2 CameraDevice
class.
You should then wait for the camera to open. To do so, yield CameraDevice.WaitForInitialization()
, or on
Unity 6 and above, await CameraDevice.WaitForInitializationAsync()
.
After this, you can check the state of the camera by accessing its CurrentState
property. If it is
NativeWrapperState.Opened
, the camera is ready for use. Otherwise, it means the camera could not be opened successfully.
For error details, check logcat or add listeners to CameraDevice.OnDeviceDisconnected
and CameraDevice.OnDeviceErred
.
If the camera could not be opened successfully, release its native resources by calling CameraDevice.Destroy
.
Creating a Capture Session
You can create two kinds of capture session: continuous and on-demand. A continuous capture session will send each frame recorded by the camera to Unity, and convert it to RGBA. If you don't need the continuous stream of frames, you can save on resources by using an on-demand capture session. On-demand capture sessions will only send camera frames to Unity when requested to do so. Other than that, both function the exact same way.
To create a new continuous capture session, call CameraDevice.CreateContinuousCaptureSession(resolution)
with the previously
chosen resolution. To create an on-demand capture session, call CameraDevice.CreateOnDemandCaptureSession(resolution)
instead.
They return CaptureSessionObject<ContinuousCaptureSession>
and CaptureSessionObject<OnDemandCaptureSession>
respectively,
which contain the session object (ContinuousCaptureSession
or OnDemandCaptureSession
), a YUV to RGBA texture converter,
the native-to-unity frame forwarder, and the GameObject
that they are components of.
Yield CaptureSessionObject.CaptureSession.WaitForInitialization()
or await CaptureSessionObject.CaptureSession.WaitForInitializationAsync()
and check CaptureSessionObject.CaptureSession.CurrentState
, just like with CameraDevice
. If the capture session could not be started
successfully, release the native resources and ComputeBuffer
s held by it by calling CaptureSessionObject.Destroy()
.
Once started successfully, you will receive the frames from the camera in an ARGB32 format RenderTexture
as
CaptureSessionObject.TextureConverter.FrameRenderTexture
. For OnDemandCaptureSession
s, the RenderTexture
will remain
black until you call CaptureSessionObject.CaptureSession.RequestCapture()
, which can be called any number of times.
The first few frames after the session has been started will be black. I've found that waiting around 0.1 seconds after the session has started and then requesting the capture or accessing the image works.
See the documentation for RenderTexture
on how to get its pixel data to the CPU: https://docs.unity3d.com/6000.0/Documentation/ScriptReference/RenderTexture.html
Capture Templates
Camera2 allows you to set capture templates for capture requests. CameraDevice.CreateContinuousCaptureSession()
and OnDemandCaptureSession.RequestCapture()
also allow you to do so. By default, continuous captures use TEMPLATE_PREVIEW,
which is suitable for camera preview windows, and on-demand captures use TEMPLATE_STILL_CAPTURE, which is suitable for still image capture. You can change them by specifying one of the templates
defined in the CaptureTemplate
enum.
Releasing Resources
It is highly recommended to release or destroy all CameraDevice
s and CaptureSessionObject
s immediately after you have finished
using them, as not doing so may result in the app not closing quickly. They will automatically be released and destroyed by Unity,
like when a new scene is loaded, but do not rely on it!
Example Script
using System.Collections;
using UnityEngine;
using UnityEngine.Android;
using UnityEngine.UI;
using Uralstech.UXR.QuestCamera;
public class CameraTest : MonoBehaviour
{
[SerializeField] private RawImage _rawImage;
private IEnumerator Start()
{
// Check if the current device is supported.
if (!CameraSupport.IsSupported)
{
Debug.LogError("Device does not support the Passthrough Camera API!");
yield break;
}
// Check for permission.
if (!Permission.HasUserAuthorizedPermission(UCameraManager.HeadsetCameraPermission))
{
// If the has not yet given the permission, request it and exit out of this function.
Permission.RequestUserPermission(UCameraManager.HeadsetCameraPermission);
yield break;
}
// Get a camera device.
CameraInfo currentCamera = UCameraManager.Instance.GetCamera(CameraInfo.CameraEye.Left);
// Get the supported resolutions of the camera and choose the highest resolution.
Resolution highestResolution = default;
foreach (Resolution resolution in currentCamera.SupportedResolutions)
{
if (resolution.width * resolution.height > highestResolution.width * highestResolution.height)
highestResolution = resolution;
}
// Open the camera.
CameraDevice camera = UCameraManager.Instance.OpenCamera(currentCamera);
yield return camera.WaitForInitialization();
// Check if it opened successfully
if (camera.CurrentState != NativeWrapperState.Opened)
{
Debug.LogError("Could not open camera!");
// Very important, this frees up any resources held by the camera.
camera.Destroy();
yield break;
}
// Create a capture session with the camera, at the chosen resolution.
CaptureSessionObject<ContinuousCaptureSession> sessionObject = camera.CreateContinuousCaptureSession(highestResolution);
yield return sessionObject.CaptureSession.WaitForInitialization();
// Check if it opened successfully
if (sessionObject.CaptureSession.CurrentState != NativeWrapperState.Opened)
{
Debug.LogError("Could not open camera session!");
// Both of these are important for releasing the camera and session resources.
sessionObject.Destroy();
camera.Destroy();
yield break;
}
// Set the image texture.
_rawImage.texture = sessionObject.TextureConverter.FrameRenderTexture;
}
}