Passthrough camera access is quite the buzz in the XR community these days. With Meta, Apple, and Pico having laid down their plans, everyone is curious about Google’s approach with Android XR. Having had the chance to sit down and chat with Google, I can share that their solution will mirror what we see on smartphones today. Stick around to find out the full scoop!
The Camera Access Challenge
Let me explain for those who might be scratching their heads right now. Recently, standalone VR headsets, which basically double as MR headsets, have been providing users with an RGB passthrough vision of the real world via front cameras. This nifty feature allows for some exciting mixed reality applications like Cubism, Starship Home, and more.
The operating system usually showcases this passthrough using frames captured by these cameras. However, as developers, we also want access to these frames. With them, we could analyze the user’s environment using AI and computer vision algorithms, potentially enhancing their reality like never before. I’ve previously argued how crucial camera access is for unlocking genuine mixed reality. I’ve even managed to prototype an AI-powered MR app that aids in interior design using a workaround on Quest. This feat wouldn’t have been feasible without some form of camera access.
While all this sounds promising, there’s a major downside: privacy issues. A developer with ill intent could potentially access and misuse the data, capturing everything from ID documents to bank card details and even faces for mischief. It’s a balancing act—protecting user privacy while tapping into the full potential of mixed reality.
How XR Companies Are Handling It
Initially, companies allowed unrestricted camera access without batting an eye. Those who’ve been following me from the start might recall my experiments with camera textures on the Vive Focus with NTW back in 2019. We delved into diminished reality, Aruco marker tracking, and sound reactivity, among others.
But as mixed reality gained traction, companies grew cautious, barring camera frame access, citing privacy concerns. Meta, Pico, HTC, and even Apple have put their foot down, not allowing developers to access camera frames.
This restrictive approach became the norm until a collective of XR developers, including myself, began advocating for the necessity of camera access. We argued for a transparent, user-consented framework where we could use object recognition or run vision algorithms by simply asking for permission. If smartphones, ubiquitous as they are, can grant camera access via simple permission requests, why not XR devices?
This advocacy bore fruit, with Meta promising to introduce a "Passthrough API" this year. But what about Google’s Android XR?
Android XR: Taking a Leaf from Smartphones
Android, the kingpin of mobile operating systems, naturally governs most phones worldwide. Developers crafting Android apps can request camera access directly from users. Once granted, you specify the camera ID you want—usually "0" for the rear camera—and then the frames are yours to handle.
Google is leaning towards this same compatibility for Android XR. Although shrouded in whispers, I managed to secure confirmation from an official Google representative. Here’s what she had to say about camera access for Android XR:
"Like on Android, developers can work with camera frames, given user permission, for XR. For detailed instructions on other permissions, check our developer article. The primary world-facing camera stream is accessed by requesting camera_id=0, akin to the ‘rear camera’ on Android. Camera_id=1 fetches the selfie-camera stream, akin to the ‘front camera.’ These streams are accessed using standard Android Camera APIs, like Camera2 and CameraX.
The world-facing camera feed needs explicit user permission similar to smartphones. When apps request the selfie-camera, they receive a user avatar image stream. This video stream, generated by specific apps on the headset, is based on tracking data like head and hand movements gathered from OpenXR APIs."
In essence, Android developers can utilize their familiar classes, like CameraX, for managing camera streams on Android XR-based devices. Whether you’re capturing frames or running machine learning analyses, the possibilities are akin to those on smartphones—a promising development!
However, while the selfie camera is your face-avatar, the rear camera acts more like what Apple does with its Vision Pro—a smart move. This keeps Android XR behavior aligned with smartphones: the rear camera peers into the real world while the selfie camera conjures a reconstructed user face.
A Few Cautionary Notes
It’s important to remember that Android XR is still in its preview phase, with no devices officially released as of yet. This means some details might shift by the time it officially rolls out, though I doubt drastic changes are imminent.
Expanding Horizons with Camera Access
As Google and Meta make room for camera access, it’s likely that others will follow suit. The year 2025 seems poised to usher in a new era of mixed reality exploration. I’m excited to see what groundbreaking innovations the developer community will come up with!
(Header image credit: Samsung)
Disclaimer: This blog supports itself through advertisements and affiliate links. Clicking on an affiliate link means I earn a tiny commission at no extra cost to you. For the full (boring) disclosure, see here.
Share this article with other innovators!
