Quest’s long-awaited "Passthrough Camera API" is now open for developers to try out, even though it hasn’t been integrated into store app builds yet.
Unveiled during Meta Connect 2024 last September, this new feature has been made available experimentally for Quest 3 and Quest 3S. This means developers have the green light to experiment and share APKs on platforms like SideQuest. However, integration into Meta Horizon Store apps remains off-limits for now. Meta has a history of rolling out new APIs this way, typically following up with full store app integration within a few months.
A handful of developers have already had early access to play around with this new functionality, and Meta plans to host discussions with Niantic, Creature, and Resolution at GDC next week to delve deeper into its potential.
Understanding Passthrough Camera Access
Typically, Quest 3 headsets use their cameras to give users a view of the real world, but only the system software had direct access to these cameras until now. While third-party developers could incorporate the passthrough as a background, they didn’t have direct access—relying instead on higher-level data processed by the system. This data included hand and body skeletal info, a 3D environmental mesh, and basic object tracking. The lack of raw camera access meant developers couldn’t deploy their own computer vision models, limiting enhancement possibilities.
Contrastingly, Apple’s visionOS 2 provides enterprise companies with extensive access to Vision Pro’s passthrough cameras for private, internal use. However, this requires a special license and is strictly for business settings.
For an app to utilize the "Passthrough Camera API," users must grant permission, similar to a microphone access request. Once approved, the app gains access to the forward-facing color cameras and relevant metadata, such as lens properties and headset positioning, which can then aid in running custom computer vision models.
Potential app applications include recognizing and tracking QR codes, identifying game boards on tables for virtual character integration, detecting physical objects for enterprise experience guides, or using visual AI capabilities from cloud LLMs. Limitations arise from the capabilities of the XR2 Gen 2 chipset or cloud-based image models developers are willing to fund.
Developed by Meta software engineer Roberto Coviello, the QuestCameraKit already offers exciting samples.
Apps receive the passthrough camera stream in up to 1280×960 resolution per camera at 30 frames per second, with a latency of 40-60 milliseconds. This means it’s not fit for tracking fast-moving items or noting tiny details like small text.
From a technical perspective, there isn’t a distinct Meta Quest Camera Passthrough API or an OpenXR extension. Developers must request a Horizon OS-specific Headset Cameras permission. Beyond this, Quest’s passthrough functionality taps into Android’s existing Camera2 API, merging headset pose data from OpenXR, applicable for those using customized engines or Unreal or Godot source code. This same setup will apply to Google’s upcoming Android XR platform, earmarked for Samsung’s standalone headset, albeit with a modified permission request.
Unity developers can effortlessly tap into the cameras through Unity’s WebCamTexture API, which has already been used for camera and webcam access across other devices like phones and PCs. One limitation is that the Unity WebCamTexture API supports one camera at a time.
Developers keen to dive in can find documentation for the Quest passthrough camera access via Unity and Native Android links.
Meta has also shared five official Unity samples on GitHub: CameraViewer, CameraToWorld, BrightnessEstimation, MultiObjectDetection, ShaderSample. In a different vein, Roberto Coviello presented QuestCameraKit on GitHub, featuring five more samples: Color Picker, Object Detection with Unity Sentis, QR Code Tracking with ZXing, Frosted Glass Shader, and an OpenAI vision model.
