Mixed Reality Capture: Why it Matters and How to Do It
In this episode, we sit down with Dr. Doom, the founder of Livđ the best mixed reality capture tools, to discuss the importance and challenges of capturing and sharing XR gameplay, Livâs partnership with Meta, and how they are making MR capture a breeze thanks to their Creator Kit. For the product Spotlight, we will look at their latest release which is revolutionizing the way VR gameplay is captured.
Before we dive into the interview I wanted to give a quick shout-out to the XR Awards. If you havenât secured your ticket yet you should. The XR Awardsđ will take place in Rotterdam on the 5th of December and bring together a night of red-carpet highlights, the celebration of excellence, and unique access to the worldâs most influential names throughout the industry.
Use the code XRAIspotlight15OFF for a 15% discount đ
Interview with Dr. Doom
Why is it taking so long to develop an easy-to-use solution for recording third-person footage of XR gameplay?
Dr. Doom: It really depends on how you define third-person footage. Thereâs a difference between using a physical external camera and virtual cameras that are inside the app. When you use an external camera, it adds complexityâlike needing a PC to handle the extra processing power. Plus, our solution involves both the Liv app and SDK, which adds some setup. The good news is weâre working on a leaner solution, where you can pull out a virtual camera just like using your phone to take a picture in the real world.
Can Quest apps take advantage of Liv's mixed reality capture, or do they need a PC?
Dr. Doom: For mixed reality captureâthe composite of a real person into the virtual sceneâyou still need a PC. If a developer wants to film high-quality trailers or marketing materials for their app, theyâll need the power of a PC to capture 4K resolution or better. So while the gameplay runs on Quest, the high-end capture still requires external hardware for now.
Is mixed reality capture evolving toward more âaugmented realityâ?
Dr. Doom: Absolutely. As the industry moves towards AR, recording these experiences becomes crucial. Right now, mixed reality is a bit more challenging because youâre not using a green screen. To segment the human from the background, we rely on post-processing. Itâs time-consuming, which is why most developers use this method for high-quality trailers. In the future, as AR headsets become more widespread, capturing those experiences will become more streamlined and necessary for storytelling.
Will we ever see an easy-to-use, no-PC-required option for creators?
Dr. Doom: Weâre getting closer. Right now, for high-quality mixed reality, a PC is required. However, weâre launching Quest Creator Kit (QCK) later this year, which will allow users to film gameplay directly on the Quest without a PC. Itâs a game-changer for virtual camerasâselfie, third-person, and drone views will be available natively on the headset. This makes content creation more accessible, especially for players who arenât equipped with high-end setups.
How easy is it for developers to integrate Livâs Quest Creator Kit into their apps?
Dr. Doom: Itâs pretty straightforward! For a competent developer, it should take less than 30 minutes to integrate. You drag and drop the QCK into the project, assign the player avatar, and choose what you donât want to render in the outputâlike certain UI elements. Itâs as simple as that. If thereâs a custom camera system already in place, there might be some back and forth, but weâre always available to help developers sort that out.
Whatâs the actual value for developers in adding these recording features to their apps?
Dr. Doom: There are two main benefits. First, it gives developers high-quality footage for trailers and marketing, which can boost sales. Mixed reality trailers tend to convert better because they show the gameplay experience in a more relatable way. Second, it empowers content creators in their community to make and share their own videos. If you give creators easy tools to capture gameplay, theyâll create content that brings more eyes to your game, boosting long-tail engagement.
What are some creative ways content creators are using Liv beyond just gameplay footage?
Dr. Doom: Some creators use Liv to insert their avatars into VR games as a way to represent themselves. Weâve seen VTubers overlay their avatars even outside of VR, like when playing regular games. Competitive gamers also use it to record smooth, high-FOV footage of their highlights. Then there are creators with massive followings, like Lizi and Nacy, who use green screen setups to film high-quality mixed reality content for their TikTok and YouTube channels.
How are non-gaming industries using Liv?
Dr. Doom: About 30% of our users arenât in gaming at allâthey come from industries like education, architecture, police training, and even space agencies. These users are building simulations in VR and need a way to showcase their work to clients or stakeholders. Liv helps them contextualize what theyâre building by creating videos that can be shared with people who might not understand VR or have a headset. Itâs a powerful way to sell or explain a project in both internal and external environments.
How do you envision immersive gameplay being consumed in the future?
Dr. Doom: The player and viewer experiences are going to converge. Today, you play a VR game fully immersed, but when you share that experience, itâs reduced to a 2D video. In the future, I see immersive video becoming the normâwhere the viewer gets a sense of presence just like the player. Thatâs what weâre working on with Project Spatial, which is aiming to capture and share content volumetrically. Itâs early days, but by 2025, we hope to have a public beta that starts bringing that vision to life.
Will volumetric capture allow players to re-experience someone elseâs gameplay inside their own world?
Dr. Doom: Exactly! Thatâs part of the dream. Imagine watching your friendâs gameplay through a volumetric "window" in your own game, or even being able to jump directly into their experience. Weâve experimented with this idea, called Holographic Calls, where you can peek into someoneâs gameplay in real time. Long term, we want to bring players and their fans closer by using the unique sense of presence VR offers. Itâs still early, but weâre making strides toward that future.
What role do you see AI playing in improving the capture and gameplay sharing process?
Dr. Doom: AI is going to help in many ways. Weâre already using AI models for background segmentation to speed up the process of cutting out humans from mixed reality footage, which previously required manual rotoscoping. AI will also help with better motion capture and even lower-body puppeteeringâthere are papers showing it can be done with just a few hours of motion data. Long term, I think AI will improve video encoding and compression, making immersive video more accessible on mobile and standalone devices.
What does your recent partnership with Meta mean for developers and creators?
Dr. Doom: The partnership is a big deal because it unlocks Metaâs Presence Platform for Liv. That means developers using features like hand tracking, scene understanding, and body tracking can now use Liv to capture their content. It required a lot of engineering on our end to integrate with the Oculus runtime, but it means we can accurately capture and render those features in trailers or marketing content. This is a big step toward providing comprehensive capture solutions across all Meta apps and experiences.
How did you navigate the relationship with Meta, given they were developing their own tools?
Dr. Doom: It took us seven years to get to this point. For most of that time, Meta was developing their own solutions. My advice is to stay transparent and keep building. We always showed Meta what we were doing and offered to collaborate, but we respected that they were working on their own tools. Eventually, we knew that if we kept doing great work, theyâd come to us. We were focused solely on this, while Meta had many priorities. Our persistence and expertise in this niche paid off when the timing was right.
Check out the full interview right here đ
Product Spotlight: LIV Creator Kit
The LIV Creator Kit (LCK) is a flexible, modular, and lightweight SDK that allows users to record themselves directly within your Unity VR game or app. It includes a plug-and-play camera prefab or a full API to create custom camera prefabs.
I have tried it and itâs awesome! I wish every app had something like this to help the team and users share quick gameplays from other perspectives than the shaky and nauseating first-person view.
Features include:
Less than one hour to set up
Plug-and-play prefab providing core LCK features with an intuitive, sleek interface
Flexible and modular scripting API for building custom integrations
Fully customizable recording settings (resolution, fps, bitrate)
No licensing fees for encoding/decoding â weâve got it covered!
Compatible with the Meta Horizon App for seamless video syncing
Thatâs it for today, and donât forget to subscribe to the newsletter if you find this interesting
See you next week