Meta has opened the display on its Ray-Ban glasses to developers. Oscar Falmer, wearable developer advocate at Meta with nearly a decade in XR across Apple and Snap, explains what changed, why it matters, and who should build for it. He breaks down the two development paths, why the EMG neural wristband is underrated, how the ecosystem differs from everything that came before, and what kinds of experiences are emerging from early builders.

Before getting into the interview, I wanted to quickly introduce you to today’s sponsor Gracia AI. Gracia AI is the only app that allows you to experience Gaussian Splatting volumetric videos on a standalone headset, either in VR or MR.

It is a truly impressive experience and I recommend you to try it out right now on your Quest or PC-powered headset.

Try Gracia AI

Interview with Oscar Falmer

What can developers currently build for Meta AI Glasses?

Oscar: Until recently, developers could only build apps for the non-display Ray-Ban which already represents millions of users globally. Not long ago, Meta opened up the display. There are now two development paths. The first is the Device Access Toolkit, which works as a mobile app extension for iOS and Android. Think of it like a smartwatch app: your existing app gets extended to the glasses. The second is web apps for display. This one is especially notable because it is built for AI-first development. The skills files have documentation that AI can easily understand, because the assumption is that most people will generate their apps with an LLM first. It is genuinely vibe-coding friendly out of the box.

What is the difference between the Device Access Toolkit and web apps, and when does each one make sense?

Oscar: The Device Access Toolkit makes sense when you have an existing app with a user base and want to extend it into daily life, especially when you want to leverage the camera and microphone for real-time context and LLM processing. If your app is doing heavy AI or LLM work on the mobile side, the phone handles the processing power. Web apps are better for lightweight, standalone experiences that load quickly over 4G or 5G from the phone to the glasses. They are display-only, so they only work on the display version of the glasses. Web apps also cannot handle the kind of intensive AI processing that benefits from offloading to a phone chip. The distinction is roughly: heavy, context-aware, app-connected experiences go through the toolkit; simple, standalone, quickly-loaded experiences go through web apps.

If you want to start developing for the Meta Ray Ban Display, Meta released a free Chrome extension that simulates the 600×600 display directly in the browser. Very handy if you don’t have the glasses yet.

This is the type of updates I share every week with the Pro Newsletter Subscribers. Click below to see a preview and start saving ton of time to stay up to date on the latest relevant news and update at the intersection of 3D, XR and AI.

Can you tell us more about the “display”?

Oscar: These are monocular glasses with a single 2D screen on the right eye. There is no stereoscopic effect and no depth perception in the VR or AR sense. The experience of adapting depends heavily on which eye is your dominant one. If your right eye is dominant, most people do not even notice there is only one screen. For people whose left eye is dominant, there is more initial confusion as the brain reconciles two different images. However, from hundreds of demos, most people adapt within a day or two.

How should AR and VR developers approach building for a monocular 2D display?

Oscar: The instinct to bring your most graphically intensive VR experience over directly is going to lead to disappointment. The more useful frame is to ask which AR use cases can be approximated without stereoscopic depth. Object recognition and labeling, navigation overlays, contextual information triggered by camera and location: most of these work, even without world anchoring. My recommendation is not to stop at what is missing and instead prototype quickly. With the web app path and an LLM, you can have a working prototype in under two hours without any new development tools.

Who should actually be building for Meta AI Glasses right now?

Oscar: The developers finding it most natural are AI-first builders, people who want to give their LLMs eyes and ears. Enterprise use cases in hospitality, warehousing, and field service are already emerging, as are consumer utility apps. For developers coming from VR, the adjustment is significant because this is not about intensive 3D graphics or spatial UI. For mobile app developers, it is closer to a smartwatch extension with the added power of a camera, microphone, and real-time AI processing. This is the first XR device I have worked on since 2017, where genuinely non-technical people are creating working apps.

Can you put 3D content on the display, and what are the practical limits?

Oscar: Yes. 3D works, and someone has already made Doom run on the glasses. Someone else created a Tron-style game using the IMU sensor, where you move your head to navigate a laser grid. That said, because this is a monocular 2D screen, the VR or AR depth experience is absent. For developers coming from immersive XR, the right framing is to start with what the device is good at: lightweight, useful, on-the-go experiences.

What is the EMG neural wristband, and what can developers access through it?

Oscar: The wristband uses electromyography sensors to read electrical signals from your muscles and detect hand gestures without any computer vision. It does not need cameras looking at your hands, which is good for battery life and allows use in dark environments or with your hands in a pocket or hoodie. To my knowledge, this is the only consumer-produced EMG device at this scale globally. Through the SDK, developers get D-pad inputs: swipe left, right, up, and down, plus a tap gesture. This covers most use cases. Neural handwriting was also released to US users: you write letters with your fingertip on any surface, your leg or a table, and the glasses interpret them as keyboard input. Developers receive that as standard keyboard input.

Do you see the glasses as a phone companion or replacement?

Oscar: The glasses are built around a tight relationship with the phone because the phone handles heavy processing, network connectivity, and battery demands that are impractical at glasses scale. It is like a smartwatch: you can run with just the watch and collect health data, but the phone is where you get the full picture. The difference here is that the glasses add camera, microphone, IMU, and a display at a weight and form factor no phone can match. Some use cases do not require phone connectivity at all: non-display apps and native local apps can run independently. Web apps need internet. I feel that right now the relationship is companion rather than replacement.

Are there enterprise use cases emerging, and what challenges come with deploying a consumer device in professional environments?

Oscar: Enterprise use cases are appearing without Meta pushing them. Warehouse operations, hospitality, field service coaching, and IoT device control are already live. The current gap is Mobile Device Management, which does not yet exist for these glasses. Meta is in developer preview and is learning in real time what enterprise customers require. The enterprise tooling is not yet there but the demand signal is strong enough that it is being taken seriously. We expect it to evolve as the use cases become clearer.

How is the early developer community forming around Meta AI Glasses?

Oscar: An unofficial subreddit called MetaRayBan Display existed before the SDK launch, driven by power users who loved their glasses and wanted more from them. As soon as the web app SDK launched, that community began building immediately. We launched an official Meta called MetaGlasses for Dev as the central space for all developers. Within ten days of launch, developers had built a Philips Hue IoT light controller, a robot controller for a Unitree Go2 robot dog, and numerous utility and game prototypes. Because the apps are web-based, iteration is fast: someone shares a working URL, another person prompts an LLM to adapt the idea to their own use case, and something new is running the same day. The pace is genuinely different from any prior XR platform launch.

What is available in Europe and how can people outside the US get started?

Oscar: The glasses are currently only available in the US. A release in Canada, Italy, France, and the UK was announced for earlier this year but was pushed back because US demand exceeded supply expectations. There is no workaround through Meta for getting them outside official channels. Sadly, I cannot help individual people acquire glasses outside the US, despite frequent requests. Practical options include waiting for the European launch, picking them up during a US trip, or finding secondary market sellers who imported them at a higher price. For anyone who does have access, getting started means enabling developer mode, visiting the Meta World Developer Center online for full instructions, and joining the MetaGlasses for Dev subreddit.

That’s it for today

See you next week

This week’s lineup is packed.

We’ve got Meta’s own Chrome extension for testing Ray-Ban Display apps, a drone flight planner built specifically for Gaussian splatting, an RC Pilot Simulator for Meta Quest, and so much more!

Let’s dive into it!

Ray-Ban Display Web App Simulator:

Meta released a free Chrome extension that simulates the 600×600 display of Meta Ray-Ban Display Glasses directly in the browser. Very handy if you don’t have the glasses yet.

Read more

Mindy Li is International Business Development Director and Product Manager at XGRIDS , the company behind PortalCam, a SLAM-based spatial camera designed for high-fidelity 3D Gaussian Splatting capture. In this conversation, she explains how PortalCam’s approach differs from traditional photogrammetry, why the data it produces is accurate enough for engineering work, and how a single scan ends up serving real estate agents, VFX crews, and robotics researchers in completely different ways.

Before getting into the interview, I wanted to quickly introduce you to today’s sponsor Gracia AI. Gracia AI is the only app that allows you to experience Gaussian Splatting volumetric videos on a standalone headset, either in VR or MR.

It is a truly impressive experience and I recommend you to try it out right now on your Quest or PC-powered headset.

Try Gracia AI

Read more

This week, I personally curated a packed issue for you: from SplatEdit adding in-browser Segment Anything to your Gaussian Splats, to the first playable multiplayer world model, stunning new apps for VisionPro, a free autoremesher tool and more.

In case you missed it last week I also published a video comparing 10 pipelines to reconstruct a 3DGS from 360 drone footage.

Let’s now get into the best of the week

• SplatEdit, a powerful 3DGS editing tool, has added a segmentation selection feature powered by Segment Anything, running locally in the browser. Just upload a splat, click on the “mug” icon and select entire objects in the scene. (I tested it using a splat of my studio right here 👇)

To access the rest, become a Pro member and you will get the Full Handbook on Gaussian Splatting and a weekly recap of the best I found and tested at the intersection of 3D XR and AI.

Read more

This week, I have published an in-depth comparison of 10 tools and workflows to turn 360 drone footage into Gaussian Splatting.

The video is 30 minutes long and I wanted to share a few key takeaways here so you can decide if you want to check out the full walkthrough.

You can also get access to the raw data by subscribing to the XR AI Spotlight newsletter.

This includes:

• The .insv from the Antigravity A1

• The converted mp4

• The extracted equirectangular frames

• The converted Cubemaps

The comparison includes Free, Cloud and Local tools and for each, I looked at the number of splats of the final reconstruction, the file size and the price.

Read more

Every week, I curate the latest news, resources tools and product updates hard to find anywhere else, and share them with all Pro Subscribers of the XR AI Spotlight Newsletter.

This week’s highlight is an essay on the privacy implications of mass-market smart glasses adoption written by Alex Levin.

This paper argues for a radically different approach to computer vision, one that discards raw visual data the moment meaning has been extracted. Learn why this architectural shift could redefine how developers build trusted AI experiences on wearable devices.

One of the resources I shared last week is this 1h 20 min tutorial by Lazar Bogdan on animating a 3DGS fly in Houdini.

I even made a post on LinkedIn that completely blew up, gathering over 100K views and 1.5 K engagements. This is just a glimpse of the hard to find updates and news I share every week.

This week, I was also at XRCC in Berlin (the biggest XR Hackathon in Europe) and I got the chance to meet many folks who shared kind words about the newsletter, but what Frederick Franken said really stuck with me.

This is the type of feedback that really keeps me going and validates all the hard work of curating and compiling manually each of the updates I share every week.

You can check out the rest of the news by becoming a Pro Member.

This week features a deep dive into the evolution of Meta Avatars, several exciting Vision Pro apps, new AI-powered creative workflows, advances in Gaussian Splatting, and tools that make building XR experiences faster than ever.

Let’s dive into it!

Read more

Mariem Ferhat is co-founder and COO of Solaya, a deep tech company making high-fidelity 3D Gaussian Splatting accessible through a smartphone. In this conversation, she explains how Solaya’s guided capture workflow produces retail-ready 3D models in under two minutes, and how one scan can replace the five separate product shoots a retailer typically needs per item. Mariem closes with a candid take on the European deep tech funding gap and what it actually takes to build a startup from scratch.

But before getting into the interview, I wanted to let you know that this week there is an incredible lineup of workshops organised by SensAI Hackademy.

Over the next 10 days, the team will cover 𝟭𝟬+ 𝘄𝗼𝗿𝗸𝗳𝗹𝗼𝘄𝘀 𝗮𝗰𝗿𝗼𝘀𝘀 𝟳 𝗽𝗹𝗮𝘁𝗳𝗼𝗿𝗺𝘀 𝗮𝗻𝗱 𝘁𝗼𝗼𝗹𝘀, from 3D reconstruction & 3DGS to world models, from MCP to agentic spatial interfaces

Register Here

Read more

Ryan Moore is a pediatric cardiologist and Chief Emerging Technology Officer at Cincinnati Children’s Hospital, one of the top-ranked children’s hospitals in the US. In this conversation, he explains how his in-house team of Unity and Unreal developers builds patient-specific anatomic digital twins that surgeons use to plan complex heart surgeries in virtual reality. Ryan breaks down how the hospital demonstrates value to leadership, why personalising care is the north star guiding every technology decision, and how they converted early proof-of-concept projects into a 2.7 million-funded VR platform.

Before diving into the summer, make sure to check out Next App DevCon and add it to your calendar. It is simply put the world’s LARGEST conference for mobile app developers and has an XR track where I will be giving a talk in Berlin on the 8th of October.

You can use the code GR-XR-25OFF to get a 25% Discount on the full access ticket.

Read more

Evan Spiegel has announced SPECS at AWE and I want to take this opportunity to share a couple of in-depth resources to get you up to speed on the Smart Glasses space.

• Let’s start with the MOST COMPREHENSIVE list of AR glasses on the market. The list is updated and curated by Oscar Falmer and you can find it right here.

• Oscar was also the latest guest on the XR AI Spotlight podcast. Check out the full conversation on YouTube where we dive into the opportunities behind the Meta RayBan display, who is building apps and what are the main usecases.

• Of course SPECS are all over the news but if you want to go beyond the hype, celebrity pictures or disappointment you should certainly check out the article from Antony Vitillo, who was at AWE.

• Last week at AWE, I also won the award as best creator…. But I was not there to receive it. I wrote a post on LinkedIn to explain how that happened.

It is always a pleasure to connect with founders and teams building a product and help them create tangible awareness. Just reach out to me if you feel like working together and be featured in any of the platforms that I am building:

• LinkedIn

• X

• Newsletter

• YouTube

There is a lot more I cover this week, like a full tutorial on how to animate 3DGS, a new app on VisionPro that let’s you experience animated splats, and many more updates to get you up to speed on everything new in 3D XR and AI without wasting time. Become a pro member and:

- Read the rest of the news and receive weekly updates

- Get the complete handbook on 3DGS

- Access the full database of interviews nicely sorted based on your topic of interest (e.g. VR and MR, 3DGS, Smart Glasses etc)

Read more

Today, we dive into how OpenBrush, the free community-led spin-off from Google Tilt Brush, has transformed into something almost unrecognisable… in a good way. Andy Baker, lead Open Brush developer, explains the recent shift toward plugins, interoperability, web publishing, and live connections to tools like Blender and Godot. Sutu breaks down how he uses OpenBrush to prototype virtual concerts, shape spatial ideas in real time, and build assets that later move into production. The interview also explores how artists are sharing work across VR, video, web, and AR, why browser access matters for adoption, and what both guests think about VR’s future as a creative medium.

Before getting into the interview, I wanted to quickly introduce you to today’s sponsor Gracia AI. Gracia AI is the only app that allows you to experience Gaussian Splatting volumetric videos on a standalone headset, either in VR or MR.

It is a truly impressive experience and I recommend you to try it out right now on yo…

Read more

This is your weekly digest to make sure you haven’t missed anything around the latest in 3D, XR and AI.

With WWDC behind us, we have several interesting Vision Pro updates, a video of the Pico Swan leaked, developers building crazy 3D apps using Fable 5 from Claude, and much more cool stuff.

Let’s dive into it!

• A reliable video leaked of the new Pico “Swan” was posted on X by @Lunayan. Guess what it looks like?

Read more

James C. Kane works at the intersection of WebXR, AI, and game design, building polished narrative experiences for the browser. In this conversation, he breaks down what kinds of games actually make sense on the web today, how he chooses between different web engines, and what it takes to make immersive experiences feel premium from the first second. He also shares the workflow behind Shadow of the Tsar, from generating and stitching Gaussian splat environments to building interactions, lighting, and sound. Along the way, he explains where AI is genuinely useful in production, where it still falls short, and how it can unlock new gameplay mechanics rather than just speed up development.

Before getting into the interview, I wanted to quickly introduce you to today’s sponsor Gracia AI. Gracia AI is the only app that allows you to experience Animated Gaussian Splatting on a standalone headset, either in VR or MR.

Their 4DGS technology has been used in the latest Muse video, which you should absolutely check out whether you are a Muse fan or not 😉.

Interview with James C. Kane

What type of games can you currently create on the web today?

James C. Kane: I think there is such a huge range. The web does not apply a special limitation over what kind of game you might want to build, so whatever your passion is, you can really follow that. Web tech is really catching up to native platforms in many ways. My own work has focused on narrative experiences and what you can now do in WebXR and VR headsets. One example is The Nanauts, a mixed reality game where three robots enter your home and interact through computer vision, voice, and intent analysis. That feels like a new type of gameplay that only recently became possible.

Try Nanauts in a Headset

How do you find your way toward building something polished and worth making, instead of another disposable web game?

James C. Kane: You have to be obsessed with quality in every aspect, from the marketing and positioning to the intro scene, establishing shot, music, sound effects, visual effects, every part of it. You are competing for attention with every other kind of entertainment in the world, from AAA games to endless short videos. Your game has to be good from the first moment and attention grabbing in every way you can think to make it. Then what you make has to come from your own vision and what you find compelling, because that is how you build a real niche audience over time.

How should developers choose the right engine for a web project?

James C. Kane: There are many options, so it can be a little head spinning. I have used three.js, A-Frame, Wonderland, React Three Fiber, PlayCanvas, and SparkJS. They all have different strengths. For example, I made Escape Artist in Wonderland Engine, and one of its biggest strengths is how it handles textures. You can pile many 4K textures into it, and it creates a virtual texture atlas in the browser, streaming only what you need in frame. Managing texture memory in many other web frameworks is a huge pain, so if I were making a texture-heavy app, I would probably go Wonderland.

What stood out to you about Immersive Web SDK and mixed reality development?

James C. Kane: I came into that hackathon organized by Meta cold and had three weeks to figure out what I could make, but I was impressed by how easy Immersive Web SDK made mixed reality features, especially camera passthrough. That is a relatively new feature, and in this framework you can get it more or less out of the box. With that, I could send frames from the mixed reality view to an AI server for analysis and know there is a banana here, a stool here, a lamp here. Once you have the mesh and a semantic understanding of the scene, it unlocks a lot. I am still only scratching the surface of what that means.

Can you tell us about Shadow of the Tzar and its opening scene?

James C. Kane: I think of that opening as the establishing shot. I want to immediately give a sense of quality and the cinematic nature of the title. If you think about great cinema, you are told a lot about the story from the very first shot. Here you get a winding arc through narrow city streets, fog drifting over the scene, black and white imagery, so you are gaining a lot of information very quickly. I also want to make it clear what the user should do, so the interface stays simple. Then once you hit start, the camera drives you into the beginning and the experience starts moving right away.

Try the 3DGS based game on the Web

Why is constant momentum so important in your games?

James C. Kane: That is completely by design and intentional. It is a criticism I have of a lot of web content: you get there, it looks cool, you look around, but that interest fades very quickly. I want something to happen immediately, and then something right after that. There should be moment to moment excitement and interest. In Shadow of the Tsar, even though you only move a short distance down the alley, you run into a cat, then a radio that changes the music and gives you story context, then choices about where to go. It keeps pulling you forward every few seconds.

How did you generate and stitch together the Gaussian splat environments in Shadow of the Tzar?

James C. Kane: The process starts from images, because Marble is an image to 3D Gaussian splat generator. I began with reference photography from public domain classic cinema and film noir. From there I went into ComfyUI and prompted variations, or masked out things I did not want. That gave me stronger input images for Marble. You upload one image and within three or four minutes you get back a fairly high quality scene. For stitching, I experimented with both Blender tools and SuperSplat. I also worked with my friend Noah Ilberry, who helped polish and clean up the stitched scenes so I could focus on the rest of the game.

What is still difficult about editing Gaussian splats today?

James C. Kane: The editing experience is probably still the worst part of working with Gaussians. It is very fiddly. It is difficult to create straight lines when you remove anything. There is not really an ability to paint things unless you are copying and pasting. Even just selecting what you want to delete without grabbing everything behind it is a challenge. I would still say SuperSplat is the best tool I have used for this, but even there it can be a painful part of the process. The quality of generation is moving fast, but the editing tools are still fairly rough.

What did SparkJS unlock for you creatively?

James C. Kane: When I start a project, I look at what the engine or library can actually do and how I can lean into those capabilities while avoiding the pitfalls. With SparkJS, one big advantage was the cutting edge Gaussian splat rendering and the ability to mix GLTF and 3DGS pipelines right next to each other. I was also looking through the Spark examples library, which already had a lot of shaders and effects. That gave me a vocabulary to work from. I could see an effect, ask how it might fit this world or story, and then build around it in a cohesive way instead of starting from nothing.

How do you approach sound design, music, and voice overs?

James C. Kane: Sound is super important for immersion. Your ears are a huge part of how you perceive the world, and that carries into games. Lately I have set a lot of my work between 1900 and 1930, which gives me access to incredible public domain music at good quality. In In Tyrol I used Mahler symphonies from archival recordings, and in Shadow of the Tsar I played with Duke Ellington against Rachmaninoff to reinforce the themes and characters. For sound effects, freesound.org is useful, but I also love recording my own foley. For voices, I record myself into Ableton and use ElevenLabs voice changer to shape different characters.

Where has AI genuinely helped your workflow, and where does it still fall short?

James C. Kane: AI has been a game changer for me, especially on the code side. I can attempt features now that I would not have tried a year ago because I would have assumed they would take too long and blow up my production schedule. Sometimes there is enough well-documented open source code around a feature that AI helps me get it working very quickly. That saves time, and I can spend that time on things like physical props, costumes, acting, performance, and all the extra layers that make a project feel alive. Where it still falls flat for me is music and sound.

What kinds of gameplay become possible when AI is part of the game itself, not just the production process?

James C. Kane: That is exactly what I am trying to emphasize in my work now. In Shadow of the Tsar, there is a drawing sequence where the player has to recreate runes they found in the environment. I trained a custom neural net whose only job is to turn on during that moment and identify whether the player has drawn a star, a lightning bolt, or a circle. The model is tiny, but it powers a real mechanic in the game. On the XR side, semantic understanding of the room opens even more possibilities. Once a system knows there is a laptop here and a lamp there, entirely new interactions start to become possible.

What do you think is the biggest challenge when publishing web games?

James C. Kane: Discovery is probably the biggest challenge of web gaming. Getting people to know your game exists and take the risk with their time to try it is difficult, and I would not claim I have fully solved it. A lot of it starts in pre-production. Before I even lock a concept, I am trying to line up partners and sponsors because I have to support myself while doing this work. Platform sponsorships have been one of the best routes for me. I also think a track record matters a lot. If your portfolio shows strong case studies and consistent quality, people can make a judgment about what your next project is likely to deliver.

What are your thoughts on monetization, from ads to paid access?

James C. Kane: Ad driven monetization is an option, but it is not really what I want to do. A landing page full of ads can instantly create an impression of lower quality, and that first impression matters a lot to me. If I ever do ads, I would rather do them in a diegetic way, like a billboard inside a mini golf world. As for pay to play, my feeling right now is that reach matters more than direct monetization. Some of my WebXR games have been played by millions because Meta distributed them on the new tab page, and at this stage I care more that people try the work and come back than extracting a dollar from each user.

That’s it for today

See you next week

Today I am coming with 10+ juicy updates and resources, plus a request for help/opportunity

I am working on a video on my YouTube channel to compare 7 workflows to reconstruct 3DGS from 360 drone footage.

I would like to include Lichtfeld in the analysis but I don’t know where to start. I can share the original 360 footage and if anybody could help out, I would be happy to mention you during the video and thank you for creating the most comprehensive and in-depth comparison for 3DGS reconstruction from 360 drone footage….I am sure people are going to love it.

If you are interested, just reply to this email sharing a scene you reconstructed using LichFeld (ideally from 360 footage).

And now let’s get into the top 10 updates around 3D, XR and AI curated by me so you don’t need to waste a ton of time understanding what’s relevant.

• Fei-Fei Li Founder of World Labs shares a taxonomy article to help us understand what World Models are, clearing up a lot of confusion.

• I interviewed the founder of Splatica and we geeked out about using 360 footage to reconstruct 3DGS. I think you’ll love the conversation.

Do you want to check out the rest of the list and get a 60+ pages handbook on 3DGS?

Become a Pro Member

Read more

Philipp Sieben from PixelArtistry and Stefan Vaskevich from Stefan 3D AI, have spent two years testing nearly every 3D generative AI tool available while building a following of over 100,000 artists in the process. In this conversation, they break down the current leaders in paid and open source 3D generation, explain why ComfyUI is both powerful and difficult to set up, and share how different creative profiles from game developers to 3D printing artists should approach 3DGenAI tools. They also weigh in on segmentation, texturing solutions that go beyond the main generators, and why neither of them feels guilty about teaching people to use AI tools.

Before getting into the interview, I wanted to quickly introduce you to today’s sponsor Gracia AI. Gracia AI is the only app that allows you to experience Gaussian Splatting volumetric videos on a standalone headset, either in VR or MR.

It is a truly impressive experience and I recommend you to try it out right now on your Quest or PC-powered headset.

Try Gracia AI

Interview with Philipp and Stefan

What is the best 3D GenAI tool right now, both paid and open source?

Philipp: This is something that can change weekly because we see new updates all the time. On the paid side, from our latest tests, Tripo has been really promising. We saw great improvements in shape details, and texturing-wise it is quite strong. The smart mesh topology mode they introduced is something we had not seen at that level of detail before. On the open source side, Stefan and I are on the same line: Trellis II is probably the best option out there right now. It stands out as the main 3D AI generator you would choose above anything else when it comes to open source.

What makes Trellis stand out even compared to paid tools?

Stefan: When Trellis released, it was already clear it was a revolution for open source. But it actually has one highlight that even paid 3D AI tools do not have. It is the first to support transparency, meaning it creates geometry it understands as transparent and applies the correct PBR material. You can create things like fish bowls where the bowl is generated as a transparent sphere. That is huge. On the low poly side, Hunyuan with Polygen 1.5 still holds up well, especially with quads. But for high poly, Trellis is definitely the best right now. Of course, if something like Rodin 2.5 drops tomorrow, we will need to revisit that.

Have you missed the latest news and product updates in 3D XR and AI? Consider becoming a Pro member to receive the best of the week curated directly by me.

If you had to choose only one tool for your own work, which would it be and why?

Philipp: It depends on your use case, but for the best quality right now, I would choose Tripo if I have the budget. They spent a huge amount on training their models and delivered great quality. But when you are in a studio pipeline where privacy matters, you have signed NDAs, or you need to work offline, that is really where Trellis shines. You can iterate on hundreds of assets locally, you are not bound by the restrictions a paid tool might impose, and you can customize it as much as you want in ComfyUI by combining it with other methods. For me personally, Tripo is the best choice right now, but the use case truly determines it.

What would you tell someone just discovering 3D AI who has no budget?

Stefan: Do not start with paid tools. Hunyuan is absolutely free and gives you a perfect sense of what 3D AI can do. It might be around 20 percent worse than Tripo in some details like texture, but overall the quality is very good, especially to try out for free first. When you are ready and understand you are getting something real from it, then go to Tripo. It will save you more time and give you better results on the first try. With Trellis and the right approach, you can also reach similar results if you split the model into parts. But for any commercial project, I would prefer Tripo.

You built the Top 3D AI ranking platform. How reliable is it for tracking how models evolve over time?

Stefan: It absolutely reflects the current state. What we described with Tripo and Hunyuan leading right now is exactly what the platform shows. The reason I created it around six months ago was that it was very hard to keep pace. At that point there were ten providers all on roughly the same level, and some were good at specific subjects like faces. Things changed this year. Tripo went high and Hunyuan 3.1 as well. Now they are taking a clear lead. There is also a tool called Wyvor that people like for texturing. But right now we have sort of hit a ceiling with these two main providers, and the interval before the next meaningful update is probably around three months.

Check 3DGenAI tools Ranking

Beyond the main generation tools, what is actually working well for texturing?

Stefan: There is a tool called Motif, developed by one person, that does texturing in a way no foundation model quite matches. He uses an advanced approach with a unique method for the output, and the result feels like an AI version of Substance Painter. I tested it myself and gave it to my team. Philipp and I also used it in collaboration videos because it is genuinely the answer to making textures work. Every time you generate in Tripo or Hunyuan, there will be imperfections. For experienced artists that is manageable, but for newcomers it is a difficult gap. Motif closes that gap. It has a free tier on the web, and for a fully local option, Stable Projector is the closest alternative.

What tools can actually do 3D segmentation today, and how useful is it in real production work?

Stefan: There are not many that do it well. I would list four: Hunyuan, which I consider the best for segmentation. Then Tripo, though their tool still needs to improve. Hitem3D, which recently rebranded as Hi3D, also has one. And Rodin. The difference is in how much control you get. With Rodin, you press a button and it shows a split and you agree or disagree. With Hunyuan, you get a brush and paint the areas you want to extract, which gives much better control. But honestly, I have never used segmentation in any near-production case. For something like a car, I would rather define the parts cleanly, generate each one separately, and combine them in Blender.

Can ComfyUI be used to combine the best parts of different 3D AI models into a single workflow?

Philipp: Yes, you can connect Hunyuan 3D and Tripo through API connections inside ComfyUI, and then combine those with Trellis for mesh texturing and so on. There are a lot of ways to build workflows that use each tool’s strengths. What ComfyUI is really about, beyond just running a model locally, is building up these pipelines and combining the best of each approach. You can design a workflow that takes the geometry strength of one model, the texturing of another, and the resolution upscaling of a third. It takes setup, but the level of control you get is something no cloud solution gives you.

For a complete beginner, what is the simplest way to get started and generate a first 3D model with ComfyUI?

Philipp: The best starting point is checking out my video on setting up the ComfyUI Easy Install package combined with Trellis II. We also built a landing page with the full Trellis II installation guide step by step. From there you can set up ComfyUI, apply Trellis II, upload your first image, and start playing. I also always recommend joining the Tencent Hunyuan 3D Discord. There are great people there, including Visual Rune who developed the ComfyUI Trellis II integration. All of it is open source, and they put their own free time into building it for the community. That community spirit is really what keeps pushing both open source and paid tools to keep improving.

Where do you see the paid versus open source competition heading long term?

Stefan: I think at some point the problem of generating a 3D model will be solved. So the question is what comes next. The price will drop regardless. Running ComfyUI locally is not technically free because you still need a good GPU. For people without money, saving one dollar is easier than saving three thousand for a nice Nvidia card. I think it comes down to whoever shows the shortest path to a specific use case, like dropping a model directly into Unreal Engine. Whether that ends up being Tripo or someone else, that is where the market goes. Right now we are in a polishing phase, similar to what we see with LLMs, and the big step forward will probably come in jumps.

How should different creative profiles, from 3D printing to VFX to game development, approach choosing a tool?

Philipp: For 3D printing, the priority is shape detail, not texturing. Hi-Tem 3D is a great option because they focus on exporting a solid mesh ready for printing. For VFX, PBR lighting response and photorealism matter, so texturing quality is critical. For game development, Tripo with SmartMesh topology is a strong match. I also get a lot of questions about exporting Trellis II without textures, because some users do not care at all while others say it is their biggest blocker. Even within game dev, some need low poly characters and others do not. The right tool depends entirely on what you are building and where it needs to end up.

Don’t you feel responsible for enabling the kind of AI-generated content that floods the internet?

Stefan: Not at all. What does AI slop actually mean? AI helps you shape your imagination. If the output is slop, that reflects the input. My journey started with fear, honestly. I am a ten-year engineer in 3D, and when this emerged I thought all my skills were useless. But over a year I realized we only ever talk about one side: the artists already in the industry. There is another side that rarely gets discussed. It is how many people now have access to 3D, game development, and 3D printing who never had it before. People who dreamed 20 years ago of making a game, who now have kids, and thought that window had closed. They can now create a character in a day and run it in Unreal Engine. That is what makes this meaningful.

That’s it for today

See you next week

Let’s dive into it!

• Autodesk VRED is now available on the Vision Pro App Store, featuring KIA Motors mixed reality design reviews. Check the full release article here

Read more

Marin, the Maritime Research Institute in the Netherlands, has spent decades building high-fidelity maritime simulations, and virtual reality is opening new possibilities. In this interview, Giorgio Ballestin explains how cave systems and VR fit together, where each one performs best, and why the transition is slower than many people expect. He breaks down how Marin combines hydrodynamic models, Unreal Engine, motion platforms, and tracked physical objects to test operations before vessels are built. The conversation also explores interaction design, visual fidelity, security constraints, mixed reality experiments, and how teams decide which XR capabilities are worth developing long-term.

Before getting into the interview, I wanted to quickly introduce you to today’s sponsor Gracia AI. Gracia AI is the only app that allows you to experience Gaussian Splatting volumetric videos on a standalone headset, either in VR or MR.

It is a truly impressive experience and I recommend you to try it …

Read more

Capturing Gaussian splatting with a drone is nothing new. What is new is doing it with a flying 360 camera. I recently did that using the Antigravity A1, a drone released by Antigravity, a company incubated by Insta360. It carries a dual-lens 360 camera, records 8K equirectangular footage, and pairs with a set of goggles that let you experience the flights as if you are actually up in the sky.

Get The Antigravity A1

There are plenty of videos with in-depth reviews of the drone, “technical” workflows to reconstruct splats from the footage, but nobody has done a thorough comparison of easy-to-use reconstruction solutions, how they perform, and, most importantly, what it takes to bring these worlds in VR. So there you have it.

This article covers:

• The hardware, the flight experience and the drawbacks of the Antigravity A1 nobody is talking about

• Three easy-to-use tools I tested for processing the footage (with what worked and what didn’t),

• The pipeline I used to turn the result into a VR world, and the open questions that remain.

Antigravity is also hosting a contest that runs until the 24th of June and offers some sweet cash prizes

Project Eternal contest

Before getting into the hardware, it is worth stepping back and asking why this matters.

Gaussian splatting reconstruction depends on coverage. Usually, the more angles you can capture of a scene, the better the final model. With a traditional drone and a standard gimbal camera, you are always pointing in one direction. You can orbit an object, fly a grid, and adjust pitch, but you are fundamentally constrained to whatever the lens is facing.

A 360 camera removes that constraint entirely as you capture all angles simultaneously. In practice, this dramatically compresses flight time and what might take multiple passes with a standard drone can often be covered in a single flight. This is not to say that 360 drones are replacing “normal” drones, because as we will discuss later, the extra coverage and speed come at a cost.

The Hardware

The Antigravity A1 comes as a complete package: the drone itself, a pair of goggles, and a motion controller.

One practical advantage worth noting upfront: the A1 weighs under 250 grams, which gives you significant flexibility around where and when you can fly it without triggering registration requirements in many jurisdictions.

The drone is equipped with two lenses: one pointing upward, one pointing downward. The footage from both is stitched together to produce the equirectangular image. This dual-lens design is what enables the 360 capture, and it is also the source of some of the reconstruction challenges we will get to later.

The controller looks nothing like a standard drone remote. There are no thumbsticks. Instead, you point in the direction you want to fly and use the trigger at the back to control forward speed. A set of additional buttons handles several functions, including recording, menu navigation, return to home and more. It is incredibly easy to use and I would argue even more intuitive for first-time pilots than a “standard” controller with thumbsticks.

The goggles themselves are lighter than they look and more sturdy than you might expect from a consumer product in this category. There is no light leak. The field of view is not comparable to a proper VR headset, but that is not the point. What they offer is a genuinely immersive first-person view of the flight. Even for someone sensitive to motion sickness like me, the latency and refresh rate are stable enough that it is not a problem in practice.

There is also a small screen on the outside of the goggles. Its practical utility is limited, especially in bright light, but it is a detail that adds a significant wow factor.

The overall onboarding and UX is genuinely impressive. From unboxing to first flight is a matter of minutes, and the guided setup inside the goggles removes most of the anxiety that would normally come with flying a drone for the first time.

All the reviews of the Antigravity A1 tend to focus on the flying experience and the 360 footage quality, but 2 aspects get significantly less attention:

• Transmission range: This is the most significant limitation I encountered. During my testing, I was behind a two-story building and at roughly 700 metres, I started losing connection. Shortly after the drone initiated the “return home” procedure. Make sure to keep this in mind. Plan your flight path carefully, stay within line of sight and avoid obstacles between you and the drone. This is an issues that could be improved through firmware updates, and early reports from other users suggest it is a known issue the company is aware of.

• Stringent Obstacle Avoidance: The A1’s obstacle avoidance system is quite conservative and it will not let you get closer than roughly 5 to 7 metres from objects around it. You cannot disable this. Especially for 3D scanning, it limits how close you can get to a subject, which has direct consequences for the reconstruction of details or more intricate structures.

Time to Fly: Capturing and Reconstruction

For the test, I captured a windmill, a confined object, in bright light, with a body of water around it. Bright light is challenging for Gaussian splatting, and the water adds reflective surfaces that tend to cause reconstruction problems. I am also aware this is not the ideal use case for a 360 drone; a larger open landscape would play more to its strengths. But for a like-for-like comparison across three tools, it was a practical choice.

Beyond manual control, the A1 includes a set of automated flight modes: orbit, rocket, spiral, and a few others. For Gaussian splatting capture, the orbit mode and the spiral are the most useful. You draw a selection rectangle around the subject and the drone will circle it automatically. This is a clean way to generate consistent orbital footage around a fixed object. I decided to go for the manual flight option, resulting in a 1:58-second video and a path you can see in the image below.

As mentioned above, I wanted to focus on an accessible workflow without complex setups.

For other options I would suggest you to check out these great examples from Ollie Huttunen and Michael Rubloff that use different approaches for the extraction of the images from the equirectangular footage and for the reconstruction.

When looking at the current offering of accessible tools, I found 3 options: MipMap, Scanniverse, and Splatica.

MipMap

MipMap is a local solution, with no upload required, no cloud costs, and a meaningful free tier that allows you to process up to around 450 images. It gives you a reasonable degree of control over processing parameters, including which cameras to include or exclude and final quality settings.

Unfortunately, my initial results were very poor. Regardless of the path I took or the lighting conditions, I simply couldn’t get a full windmill and had some “ghosting”.

Image positions failed to align, and the scene exhibits obvious layering and misalignment. This is primarily caused by the lack of connectivity between the upward-facing and downward-facing perspectives of the panoramic camera. I contacted the team directly and after applying the MipMap team’s recommendations (deleting upward-facing camera images and adjusting the frame extraction interval), the output improved substantially.

• Number of Splats: 18.3M

• File size: 1 GB

• Pricing: Free up to 450 Images

They, of course, acknowledged that this is not ideal, and they are currently evaluating how to improve the reconstruction results. I still think it is a tool worth watching. The free tier is genuinely useful, and the local processing model has real appeal for users who want to keep their data and avoid recurring costs.

Check MipMap Reconstruction

Scaniverse

Scaniverse is the capturing platform provided by Niantic Spatial, and has recently expanded significantly beyond its origins as a mobile scanning app. It now supports 360° camera footage uploads via a web platform and is targeting large-area reconstruction, such as construction sites, industrial facilities, and city streets. It outputs meshes and Gaussian splats in standard formats (PLY, SPZ, FBX), integrates with a Visual Positioning System for centimetre-accurate localization, and is positioning itself as enterprise infrastructure for spatial AI.

For this test, I exported the footage as MP4 using the Antigravity Studio software (the native .insv format was not accepted at the time of testing) and uploaded it directly.

• Number of Splats: 3.6M

• File size: 866 Mb

• Pricing: 0.016 $/sec

Several wooden elements of the windmill showed incorrect segmentation (sky splats embedded in the geometry), which suggests the reconstruction did not fully separate the structure from the background in those areas. Details look very good in other areas of the reconstruction. Overall, it worked and produced what I was expecting, but it was not flawless.

Check Scaniverse Reconstruction

Splatica

Splatica is a relatively new web-based tool with a specific focus on turning 360 footage into 3DGS. They have a strong partnership with Insta360 and fully support .insv file from both the Antigravity A1 and the Insta X5. The dashboard accepted the .insv file directly and provided the final file after several hours.

• Number of Splats: 7.8M

• File size: 1.7 GB

• Pricing: 0.16 $/sec

Of the three tools, Splatica produced what felt like the cleanest reconstruction (more on the direct comparison right below).

It should also be mentioned that Splatica keeps the scanning path, and you can browse various demo scenes on their platform to give a look at how they were captured and learn from the process and mistakes of others.

Check Splatica Reconstruction

Direct Comparison and my verdict

MipMap is clearly still in development, so I don’t think it would be fair to compare it right now. Scaniverse and Splatica are instead much closer.

These are some of the differences I noticed:

• Scaniverse reconstruction results in sharper details than Splatica

• The hollow wooden structure in Scaniverse is much worse than the one reconstructed in Splatica, which is much more defined and cleaner

• Splatica has many more (grass) floaters than Scaniverse

• The water surface is much more defined and compact in the Scaniverse reconstruction than in Splatica (I wonder if this difference would also translate to glass surfaces often present in buildings)

• File size, splat count (and price) are significantly smaller in Scaniverse than Splatica

Bringing the splat in VR

Having a smooth experience when viewing Gaussian Splatting in VR is still challenging, but there are several things you can do to make your life easier.

• Reduce the number of splats

• Remove the Spherical Harmonics

• Use more compressed file formats

These techniques all revolve around practical ways you can manipulate your scenes and do not address still another major limitation when it comes to optimization on the engine side. Both Unity and Unreal are lagging behind when it comes to the adoption of 3DGS and I strongly recommend you to look into other web-based alternatives such as PlayCanvas and Spark.

Let’s now break down the various options you have available that are also very valuable for splats generated via other methods.

File formats: Splats usually come as .ply files that are recognised as the “uncompressed format”. When looking for a compressed alternative, I suggest you to try .sog or .spz. With these new formats you can achieve a 90% compression without major quality losses

Remove spherical harmonics: Spherical harmonics are the data layer that contribute to the view dependency effects of 3DGS. Removing them reduced significantly the file size but it will also make them “look worse”. This is especially true for reflective surfaces. Keep that in mind and find a good balance by choosing between 0 and 3 (the max usually available). This can be done in SuperSplat before the publishing OR using the tool I am going to mention below.

Reduce the number of splats: This is probably going to be the major contributor to your immersive viewing experience. This can be done in a few ways. First, you can start by removing the scene’s surroundings and keep only the area of interest. You can do that by using SuperSplat since none of the reconstruction tools mentioned above offer editing features.

In the case below, keeping only the windmill brings our splat count from 3.5 M to 1.7 M. The other way you can do this is by merging neighbour splats using the simplify tool on 3DGSViewer.

Try 3DGS Simplify

This web-based tool gives you several options:

• It lets you define the percentage of splats you want to keep

• How stringent you want to be with the removal

• Gives you the option to remove transparent splats and floaters

You should play around with it based on your scenes, but based on my experience, you can keep only 60-70% of the splats without massive quality losses.

Within the same tool you can remove the spherical harmonics and export as an SOG ready to be published on SuperSplat and viewed in a VR Headset.

After all the steps mentioned above I ended up with a .sog file of 1 M Splats, 12 Mb in size that runs on a Quest, and Apple Vision Pro (currently the best runtime environment for this type of reconstruction).

Open questions and what’s next

Getting clean Gaussian splatting from a 360 drone with minimal steps is possible. Cloud tools like Scaniverse and Splatica get you there without having to manage frame extraction, masking pipelines, or COLMAP conversions manually, but unlike local tool you loose a lot of control along the process.

That being said, I think the Antigravity A1 is not a tool for object-focused scanning. I believe a standard drone could capture a better splat due to the higher resolution image captured. I feel that flying “through things”, capturing wider areas or when the surroundings are as important as the main subject itself, that’s when a 360 drone makes more sense. Still keep in mind the transmission issues could hold you back on your flight journey as you explore wider areas.

The other interesting opportunity is the stitching of ground capture with a drone. This is an interesting promise from the Insta360 ecosystem and combining the video taken from an Insta360 X5 with the Antigravity A1 could lead to some impressive detail reconstruction while broader coverage at high angles impossible to capture even with an ultra long selfie stick.

Both these considerations are on top of my list for testing, so make sure to subscribe to the newsletter or follow me on LinkedIn if you want to stay up to date with my latest experiments at the intersection of 3DGS, XR and AI.

Subscribe now

• I have added 3 new tools to create Gaussian Splatting

• An app that turns any image into a splat directly on Vision Pro

• A brand new section with the reconstruction of the most popular game characters in history, ready for downloads

The guide includes an intro to Gaussian Splatting, a list of 21 local and cloud-based solutions for the reconstructions, and 14 apps to experience Gaussian Splatting in VR on Quest or Vision Pro.

Make sure to subscribe to the newsletter, and you will get it directly in your inbox

And now let’s get through everything new you will find in the update.

Everything NEW in the Update

• Splatica: Focuses on turning 360 footage captured from an Insta360 into a 3DGS. It now also supports footage from Antigravity (a drone under 250 gr equipped with a 360 camera). I did a test recently, and I was quite pleased with the result…

Read more

This week, we have a nice mix of new XR apps and updates with a good dose of 3DGenAI tools Gaussian Splatting and a brand new world model. Let’s start with that:

• Reactor is a new world model with remarkable latency and memory. There is an interactive demo on the website you should check out.

Read more

Today, we explore how people behave, spend, and express identity inside social VR platforms. Mila shares findings from a large-scale quantitative study covering avatar choices, gender expression, platform differences, and economic behaviour. The conversation reveals how users navigate identity, why many adopt feminine appearances, and how cultural context shapes these decisions. It also breaks down how money flows through virtual worlds, from avatar customisation and seasonal fashion to events and services.

Before getting into the interview, I wanted to quickly introduce you to today’s sponsor Gracia AI. Gracia AI is the only app that allows you to experience Gaussian Splatting volumetric videos on a standalone headset, either in VR or MR.

It is a truly impressive experience and I recommend you to try it out right now on your Quest or PC-powered headset.

Try Gracia AI

Read more