Experience 4D Gaussian Splats in VR: from Capture to Distribution
In this issue, we sit down with Georgii Vysotskii, the CEO of Gracia AI🔗: the app that lets you experience 4D Gaussian splatting in fully immersive VR. Georgiii breaks down the differences between 3D and 4D Gaussian splatting, the potential of real-time volumetric capture, and how these innovations are set to transform the way we experience digital content. To further quench your thirst for 3D volumetric content the Product Spotlight section is dedicated to an app that lets you experience some VERY HIGH quality volumetric captures directly on your Quest.
The newsletter is also shared with our 10k audience on LinkedIn and if you would like to become a sponsor just answer this email or contact me on LinkedIn 👉 Gabriele Romagnoli.
Interview with Georgii Vysotskii
What is Gaussian splatting, and why should people care about it?
Georgii Vysotskii: Gaussian splatting is a new way to represent 3D scenes. It’s different from techniques like NeRF and traditional meshes. Think of a 3D object made up of tiny points, and each point has color. Unlike older methods, Gaussian splatting makes it easier and faster to train models and render them in real-time. This makes it much more feasible to deliver to consumers.
Can you explain the difference between 3D and 4D Gaussian splatting?
Georgii Vysotskii: Sure! We divide Gaussian splatting into two types: 3D and 4D. 3D Gaussian splatting deals with static scenes, where objects don’t move—like a chair or a table. You capture it from different angles using a single camera. 4D Gaussian splatting, on the other hand, handles volumetric videos, capturing dynamic scenes where there’s movement, like a person walking or talking. Our focus at Gracia AI is mostly on 4D Gaussian splatting because it’s ideal for volumetric video content.
How do you capture a 4D Gaussian splatting?
Georgii Vysotskii: Capturing dynamic scenes requires a complex setup, especially if you want full 360-degree viewpoint freedom. You need about 40 to 50 cameras positioned around the subject to capture the scene from every angle. If you're okay with less freedom—like a 180-degree view—you can use fewer cameras. We’ve even experimented with capturing dynamic scenes using GoPros, and while it's not as precise as a studio setup, it still delivers good results.
How does Gaussian splatting differ from traditional volumetric capture?
Georgii Vysotskii: Gaussian splatting and traditional volumetric capture differ in how they reconstruct 3D space. Traditional volumetric methods often use point clouds or meshes, which can be more complex and slower to process. Gaussian splatting uses a simpler, more efficient method where the scene is represented by "splats" or blobs that define the geometry and texture. This allows for faster rendering and smoother playback, which is crucial if you want to experience the capture in VR.
How does Gracia AI handle the distribution of Gaussian splatting content?
Georgii Vysotskii: Distribution is one of the biggest challenges we’re working on. Gaussian splatting files can be quite large, especially for dynamic scenes. We’ve developed compression methods that reduce file size by up to six times without losing quality. We also offer cloud streaming solutions, allowing the heavy processing to be done remotely, which makes it easier to deliver high-quality content to users on VR headsets like the Quest 3 or Apple Vision Pro.
What kind of tools do you provide for developers and creators who want to use Gaussian splatting?
Georgii Vysotskii: We’ve integrated our Gaussian splatting technology into Unity and Unreal Engine, so creators can use it in their existing workflows. We also provide SDKs that help developers distribute their content efficiently, whether they’re rendering it on-device or streaming it from the cloud. While these tools are primarily tailored for professional creators right now, we’re always working on making them more accessible.
How does Gaussian splatting perform on different VR headsets?
Georgii Vysotskii: Gaussian splatting is quite demanding, especially in VR where you need higher resolution and frame rates. For example, in VR, you need to render at 72 frames per second for each eye, which effectively doubles the workload. Right now, our technology works best on high-performance headsets like the Quest 3 and the Apple Vision Pro, especially when they are connected to a PC for extra processing power. But we’re continuously optimizing the rendering process to make it more efficient.
What are some of the key use cases for 4D Gaussian splatting?
Georgii Vysotskii: We’re focused on a few key industries where 4D Gaussian splatting can make a real impact. These include sports, like capturing a UFC fight where you can watch the fighters move around in real-time from any angle; fashion, where you can experience a runway show from the stage; music concerts, where you can stand next to your favorite artist; and cooking, where you can learn complex techniques by observing chefs in 3D.
When do you think we will be able to stream Gaussian splatting content in real-time?
Georgii Vysotskii: We’re getting closer to real-time distribution, especially for use cases like sports and concerts, where timing is crucial. Technologically, we’re almost ready, but the logistical challenges—like data transfer and ensuring the processing speed—are still being refined. I’d say we’re about one to two years away from having a fully real-time solution for large-scale events.
What’s currently available for users who want to explore Gaussian splatting?
Georgii Vysotskii: Right now, we have Gracia AI available on Steam, where users can upload their own Gaussian splats or explore splats created by others. We’ve also launched an app on the Meta Store, so VR users can view 2D splats in immersive environments. It’s free for creators to upload their splats, and we’re building a community of users who are excited to explore this new medium. We’re also continuing to improve the platform and add new features based on user feedback.
What’s next for Gracia AI in 2024 and beyond?
Georgii Vysotskii: Our two key priorities are improving distribution and launching specific use-case projects. We want to make Gaussian splatting available on any VR headset and ensure that the content can be delivered in real-time or near real-time. At the same time, we’re working on industry-specific projects, like cooking tutorials or sports events, to showcase the power of volumetric media. We’re excited to push this technology forward and make it accessible to more users and creators.
Check out the full interview right here 👇
Product Spotlight: Voluverse
There are several apps that allows users to experience live performances in MR or VR directly on standalone devices like the Quest. The best captures I have seen are in Voluverse. The level of details is very impressive and there are very little artefacts even on parts that are complicated to renders like hairs. It is not surprising the team from Volucap was the one behind the special effect of “The Matrix: Resurrection”. The only problem with the app is that the UI and UX is a bit clunky and you would need to have some patience to position the scan in the right spot and navigate the various scans that are available for download.
That’s it for today, and don’t forget to subscribe to the newsletter if you find this interesting
See you next week