Minecraft: The Sandbox for AI and Immersive Worldbuilding
Today, I decided to do something slightly different. Given that the Minecraft movie launched last week, I thought this would be a great opportunity to write a recap of how this all-time success turned into a sandbox for AI experimentation and Immersive worldbuilding.
In this article:
I will look at the most recent AI research built on top of Minecraft capabilities
How 2D world-building moved into XR (more and less successfully)
A broader overview of tools and apps (even powered by AI) to scratch your Voxel creative itch.
Before getting into the article, I wanted to quickly introduce you to today’s sponsor Gracia AI. Gracia AI is the only app that allows you to experience Gaussian Splatting volumetric videos on a standalone headset, either in VR or MR.
It is a truly impressive experience and I recommend you to try it out right now on your Quest or PC-powered headset.
Minecraft and AI: A match made in heaven
I wanted to share with you 3 recent examples of how Minecraft has been used as a powerful platform for AI research and development.
The first example is Oasis, a sandbox game that attempts to replicate Minecraft using generative AI, developed by AI company Decart and hardware startup Etched. Released for free on October 31, 2024, the game uses “next-frame prediction” to simulate gameplay based solely on user input and training from millions of hours of Minecraft footage without any underlying game code or memory. You can try it directly on the web right here, and while there are serious problems with consistency and the game gets very trippy after one minute, it gives a glimpse of AI-driven games on top of which other companies like Microsoft and Google Deepmind are building upon.
The other example is Project Sid by Altera. This project explores how hundreds of AI agents in Mincreaft can form complex, human-like societies through large-scale simulations.
The findings were quite mindboggling:
Socially aware agents self-organized into specialized professions, such as farming or art, without being explicitly assigned roles.
Agents could follow societal rules, including tax systems, through democratic voting influenced by group dynamics.
Cultural ideas and religions could also be simulated and spread organically across large-scale societies of 500+ agents.
You can check more on this project here AND Altera also gives the opportunity to play along with other agents in Minecraft with defined personalities. Here is a video on how to do it.
The last and most recent example is from last week and comes directly from Google Deepmind. The findings were published in Nature and describe how Dreamer AI was the first system to mine diamonds in Minecraft entirely from scratch without any human demonstrations, instructions, or pre-training data. Unlike past AI systems trained on human gameplay videos like Oasis mentioned above, Dreamer played the game entirely on its own for nine days, with researchers resetting the game world every 30 minutes to force it to adapt to new environments. In the end, it was able to match human-level performance, mining a diamond in under 30 minutes.
From 2D to XR
Over the years there have been several attempts to bring the Minecraft experience into the headset.
I want to cut it short here and say that Vivecraft is still the most complete Minecraft experience you can get right now, but it only works on PC-based VR. One of the advantages is that you can install mods and texture packs that skyrocket the level of immersion, but you are still required to go through a few setup steps and have a VR-Ready pc.
Another viable option on PC is called Cyube. It is a Minecraft-inspired experience with a much more realistic look and a system that sees the users arranging materials on a 3D grid to craft all you need in the game. With a Very Positive score on Steam and over 1k reviews, it is worth checking out.
On Steam, there has also been a recent release called Voxel Project VR that offers an immersive sandbox set in a fully destructible voxel environment. Reviews are quite positive so far, and there is a demo you can try for free.
On standalone Quest, the lineup of Minecraft clones expands. These include titles like Blockverse, Questcraft, RealmCraft, and Discovery 2.
I have not tried all of them, but based on the reviews, I jumped in Discovery 2 (with a 4.6 rating and 50+ reviews) and it offers a streamlined creative experience in Mixed Reality with hand tracking support. Let’s be clear: you are not playing Minecraft, but if all you wanted was hopping in creative mode and building stuff, then this is your best bet on Quest.
After so many years, it is still strange to believe there are no official Minecraft ports to XR. This is not for lack of trying. Meta ported on Oculus Rift, but the feedback from users was rather clear (with 4K+ reviews and an average rating of 2.8) and it was shut down. Microsoft also launched a port of Minecraft to mobile AR called Minecraft Earth, which was shut down in mid-2021. More recently, Mojang also announced that the VR support for Minecraft Bedrock will be discontinued and that only players on the Java edition with certain mods will be able to continue playing the game in VR. The future for an immersive Minecraft doesn’t look promising. I think Big Tech evaluate the success of a launch by comparing numbers with flat titles (looking at you Ubisoft) and given the user base, that feels like a battle lost from the start.
Creating with Cubes
Cubes and Voxels have a unique charm. Stacking colored cubes to make cute scenes can be fun, rewarding and relaxing… as Lego clearly showed the world. I wanted to share with you some examples of how you can bring that creative experience to the digital world either on a flat screen or in VR and even show you how AI can be part of the experience.
When creating with voxels, it is easy to start from scratch, but you can considerably accelerate the process with the help of AI. Meshy is a 3D gen AI tool that allows you to create 3D voxels starting directly from text and supports export in a variety of formats, including .glb and .vox.
Editing on a PC is possible using a variety of tools. My recommendation is Magica Voxel. It is used by amateurs and professionals alike and the rendering capabilities rival what you get in more advanced 3D editing tools like Blender.
Creating and editing Voxels in VR is possible both on PC-VR and standalone Quest.
On PC-VR my personal favorite is VoxVR because it also supports .vox file import. The UX is not amazing and sadly, the project is abandoned but the tool is free a worth a try.
On Quest, you can now choose between BoxelVR and UnfoldVR. My favourite is BoxelVR because the UX is more streamlined and creating in mixed reality just feels simple and relaxing. Unfortunately, export in .vox format is not supported limiting potential workflows with other 2D tools like magica voxel.
If we take a step back from the pure activity of stacking bloks there are a few other apps worth exploring:
Color-a-cube adapts the familiar color-by-number formula to a 3D white voxel model in VR and MR on Quest
Bunch of Things available on Vision Pro lets the user assemble a small 3D diorama using a vast library of voxel.
Last but not least, I want to mention Rooms (created by the same team behind “Bunch of Things”). This app, available for Web and iOS, allows users to create voxel models from scratch, add simple interactivity via presets and assemble them in beautifully rendered rooms and scenes. Rooms also supports AI and allows creatives to give “personalities” to characters in the scene via an integration with Google Gemini.
I had the chance to interview Jason Toff, the founder of Things Inc. , and we dove deep into the topics of creativity, user-generated content and of course, the role of AI in the process.
The interview will be available on YouTube, Spotify, and Apple Podcast on the 16th of April.
Before closing this up, I wanted to share one last resource that I think is the perfect closure to this article and clearly shows how 3D, Minecraft, and AI are interconnected on so many levels. While researching, I discovered that Minecraft Education, a platform dedicated to teaching subjects like computer science, math and climate science, now has a module on AI. This includes 11 lessons that can be purchased and opened in Minecraft as well as introductory videos that do a very decent job at introducing pre-teens to the very foundations of AI.
If you find this article valuable remember to refer this newsletter to a friend.
It would be greatly appreciated.