The Best and Worst of the camera API on Quest

Link to Ball tracking game included ⚽

Discover the true potential and hidden challenges with Meta’s camera API through an insightful interview with XR developer Christoph Springer. The Passthrough Camera API provides access to the forward-facing cameras on the Quest 3 and Quest 3S for the purpose of supporting Computer Vision and Machine Learning and gives us a glimpse of the opportunities unlocked by AR glasses powered by AI. From practical limitations to exciting and unexplored applications, Christoph shares his learnings from creating some of the most compelling and thoughtful experiences available today, including Ballee, a ball-tracking game you can try for free right now.

Before getting into the interview, I wanted to quickly introduce you to today’s sponsor Gracia AI. Gracia AI is the only app that allows you to experience Gaussian Splatting volumetric videos on a standalone headset, either in VR or MR.

It is a truly impressive experience and I recommend you to try it out right now on your Quest or PC-powered headset.

Try out Gracia AI right here

---

Interview with Christoph Springer

What exactly is the camera API, and was it worth the wait?

Christoph: It really depends on what you’re trying to do. I think Meta already did a great job with the existing tools that use the camera, so the camera API is more for experimental stuff. It’s great because you don’t have to rely on specific API calls; you can just use the raw camera feed. However, I wouldn’t say it’s a game-changer for every developer or every game. It’s best suited for niche, innovative uses, and that’s what excites me most.

Can you explain in simple terms what the camera API currently can and can’t do?

Christoph: If you use Meta’s standard SDK samples, you’re limited to webcam textures with delays of about 20 to 80 milliseconds, suitable for basic tracking. For more advanced uses, you’d need to access the Android Camera2 API directly, which isn’t well-documented by Meta yet. This approach is technically demanding but necessary for precision. Less latency always improves performance, but right now it’s a bit complicated to implement.

What examples of creative uses work well with this delay in the camera feed?

Christoph: Color detection works nicely, like changing virtual objects to match real-world colors. I made a cute chameleon demo that shows this. Delay isn’t a big deal for such uses or for matching virtual objects’ lighting and noise levels to the real-world camera feed. Making virtual objects look genuinely integrated is achievable even with current latency, and that’s an exciting potential.

Can the camera API realistically integrate with AI?

Christoph: You can certainly use the camera feed and connect it to APIs like OpenAI for image classification or analysis. It works technically, but honestly, I haven’t found a really compelling use case yet. While AI vision can help people with visual impairments significantly, for general consumer applications or gaming, practical, game-changing examples aren’t abundant yet.

Do you see different opportunities for the Quest versus AR glasses like Ray-Ban?

Christoph: Absolutely. The Ray-Ban style glasses have huge potential for outdoor use cases, like augmented true crime experiences. The Quest, though, is mostly indoor. Its camera API suits applications that understand your immediate environment, like differentiating your kitchen from your bedroom, but doesn’t have the broader potential of outdoor devices.

Can you describe how moving object tracking works with the camera API?

Christoph: For simple objects like balls, basic blob tracking works great. It uses the latest camera frame, so the slight delay isn’t problematic. For complex scenarios, like large spaces or precision-required setups, exact synchronization of camera frames with the virtual environment becomes crucial. Currently, achieving sub-frame accuracy demands extra technical measures not yet built into Meta’s SDK.

What are the current technical limitations of the camera API?

Christoph: A major limitation is performance. The Quest uses a mobile chip, making heavy tracking tasks difficult. Tracking complex objects or markers continually can easily eat up 20-30% of your available performance, restricting detailed game interactions. So, developers need to find a balance between tracking quality and efficiency, which can be tricky.

Do you have specific advice for developers experimenting with the camera API?

Christoph: Be very clear on what you want to achieve. The camera API is not suitable for every app or game. Most games won’t significantly benefit from it. Focus on niche experimental projects or tasks that truly require environmental understanding. Test and optimise performance constantly; otherwise, you’ll find yourself with a project that runs poorly.

What implications for privacy does the camera API bring?

Christoph: Privacy issues here aren’t much different from a smartphone camera. Meta provides notifications if an app accesses your camera feed. Still, users should be responsible, especially in public. If you’re pointing your device or headset around, you’re responsible for respecting others’ privacy. Clear user awareness and communication from developers are key here.

What exciting future uses do you envision for the camera API beyond gaming?

Christoph: DIY and instructional content can greatly benefit from this technology. Imagine AI analyzing and assisting with bicycle repairs or cooking instructions directly in your environment. Real-world guidance and assistance, provided through mixed reality, offer practical and compelling opportunities that gaming doesn’t always tap into.

Do not forget to check out the full interview on YouTube 👇

That’s it for today

See you next week