Gaussian Splatting on Meadow

by

Joerg Kahlhoefer

November 3, 2023

Unlocking Photorealism in the Metaverse: How Gaussian Splatting Revolutionizes 3D Environments on Meadow

Introduction

Ah, the nearly unreachable quest for photorealism in 3D environments! If you've been following the tech world—or more specifically, the metaverse—you've probably heard the buzz about a recent paper on a technology called "Gaussian Splatting." No, it's not the name of a new avant-garde art movement or a method for cleaning your paint brushes. It's something that's turning the world of 3D rendering on its head.

In simpler terms, Gaussian Splatting is like the missing ingredient in your grandma's secret recipe that suddenly makes everything taste just like you remember from your childhood. It adds that oomph to 3D scenes, making them mind-blowingly photorealistic. For years, we at Bizzlogic have been using photogrammetry to virtualize real-world scenes into our Meadow platform. While the results were impressive, they were, shall we say, a bit like the "diet" version of the real world—less texture, less sparkle, less life.

Photogrammetry scene scan "Greenroom"

But what if I told you that the diet days are over? What if we could bring every nuance, every reflection, and even the glossiness of real-world materials into the digital realm? That’s what Gaussian Splatting promises, and as someone who’s a bit of a tech romantic, it’s like finding the Rosetta Stone of 3D rendering.

So, fasten your seat belts, because you're about to embark on an enthralling journey from the mundane flatlands of photogrammetry to the lush, intricate landscapes enabled by Gaussian Splatting.

The Photogrammetry Odyssey: A Journey of Partial Victories

Photogrammetry, the trusty workhorse in our 3D rendering arsenal! If you're not familiar with the term, photogrammetry is a technique that uses photographs to make measurements and create 3D models. It's been our go-to method for capturing real-world scenes and plugging them into our Meadow platform.

Photogrammetry scan "Car" scene

First, photogrammetry is not a quick process. Think of it as painting by numbers; it's precise, and you have to pay attention to every detail. You're essentially fitting together a complex puzzle and sometimes finding out you're missing some pieces along the way.

As for the cleanup, that's another hurdle. After capturing all the data, you're left with a lot of refining and adjusting. It's similar to cooking a big meal and then realizing you've got a mountain of dishes to wash. At times, the post-processing is so time-consuming it makes you question whether it's all worth the effort.

Photogrammetry scan "Car" scene cleanup

And let's not even get started on the issue of complexity. Real-world scenes are filled with intricate details—thin structures like flowers and foliage, or reflective materials like glass and metal. Photogrammetry looks at these and says, "Uh, maybe next time?" These elements are either poorly captured or skipped altogether, leaving us with a world that's more plastic than fantastic.

While photogrammetry had its drawbacks, it wasn't entirely futile. It served as a valuable precursor and provided a testbed for exploring newer technologies. Think of it as using a flip phone in the smartphone era—it still has its uses, but it's not the most advanced option out there. Essentially, photogrammetry isn't completely out of the game; it still has a role in specific scenarios.

The Gaussian Splatting Revolution: When Science Fiction Becomes Reality

Gaussian Splatting has been a significant advancement for us, similar to finding a useful tool in a video game that makes certain tasks easier. It's not a cure-all, but it certainly opens up new avenues for improving realism in our virtual worlds.

I still remember the "eureka" moment. I tried Gaussian Splatting with some footage of my apartment, and the results left me speechless. We're talking reflections in mirrors, the intricate details of my houseplants, and all the other nuances that make a home a home—all rendered in jaw-dropping 3D that runs in real time. I knew right then that we were onto something monumental.

Gaussian splatting example

So, what's the deal with Gaussian Splatting, you ask? It's a technique that allows us to capture the geometry and appearance of a 3D scene in a more efficient and realistic manner than ever before. Unlike photogrammetry, it doesn't shy away from complexity. Thin structures, reflective materials, you name it—Gaussian Splatting handles them like a pro.

Even better, this technique lets us recycle. Yes, you heard that right! All those old photogrammetry scans we thought were gathering digital dust? Turns out, they were the perfect testing ground for this new technology. It's as if we've been collecting puzzle pieces over the years, and Gaussian Splatting is the missing link that completes the picture.  

Gaussian Splatting technique used on footage of "Hafen-City" example
Photogrammetry technique used on the same footage of "Hafen-City" example

And let's talk about realism. I mean, we've been chasing the holy grail of photorealistic rendering for years. With Gaussian Splatting, we've taken a quantum leap toward that goal. The technology captures view-dependent features like reflections and refractions and makes them look absolutely mind-blowing. If photogrammetry gave us a black-and-white TV, Gaussian Splatting has handed us an IMAX theatre.

Best of all, capturing real environments is now something anyone can do; you just need a smartphone camera that can record a more-or-less decent video. It's democratically magical if you ask me.

Video input material used for both techniques

Making the Impossible, Possible: Practical Applications in Meadow

If you're anything like me, you're probably itching to ask: "That's all well and good, Jörg, but what does this mean for Meadow?" Well, I'm thrilled to say that the implications are as vast as the virtual worlds we're creating.

Firstly, let's talk about accessibility. The painstaking process of capturing real-world environments has just been turned on its head. What used to take weeks of tedious scanning and post-processing can now be done in a fraction of the time. Imagine being able to walk into your company's lobby, whip out your smartphone, take some footage, and voilà! Your very own 3D, photorealistic company lobby is ready to welcome virtual visitors in Meadow. It's not just convenient; it's revolutionary.

Gaussian Splat scan inside Unreal Engine

This isn't just about aesthetics; it's about functionality. Businesses can now use Meadow to create ultra-realistic virtual stores, galleries, or even educational campuses. The applications are as limitless as your imagination. And let's not forget, these spaces are not merely 'visitable'; they're fully interactive worlds where real-life activities can be mimicked, from team meetings to product launches.

And here's where the magic happens: This isn't just a one-off gimmick. The ease with which we can now import real-world environments into Meadow means that we can keep these spaces as up to date as their real-world counterparts. Got a new product launch? Update your virtual store in real-time. Want to add a seasonal touch to your virtual campus? Go ahead, and let it snow.

Remember those old photogrammetry scans I mentioned earlier? We're already using them as the foundation for new Gaussian Splatting experiments. It's like giving a second life to digital assets, and trust me, the results are nothing short of spectacular. It's not just a technological leap; it's an environmental win.

Lastly, let's talk about archiving the real world. With Gaussian Splatting, we can capture and store real-world environments with unprecedented accuracy. Imagine a digital archive of endangered historical sites, natural wonders, or even your grandmother's kitchen. These places can be preserved in the Metaverse for future generations to explore, long after they've changed or disappeared in the real world.

Gaussian Splatting example of "Speicherstadt" scene

The Future is Now: Redefining Realism in the Metaverse

As we venture deeper into the 21st century, the line between the digital and physical world continues to blur. With Gaussian Splatting, that line isn't just blurring; it's vanishing. We're not just mimicking reality; we're capturing it, and in some respects, enhancing it. The term "virtual reality" has never felt more inadequate because what we're creating here is virtually indistinguishable from reality itself.

Meadow has always been at the forefront of innovation in the Metaverse. We've offered unparalleled scalability, accessibility, and a quality of experience that has set industry standards. But with the introduction of Gaussian Splatting into our technological repertoire, we're pushing the boundaries of what's possible. We're giving you not just a space but a world—your world—in all its intricate, photorealistic glory.

And the beauty of it all? This isn't some pie-in-the-sky future tech that you might get to see in a decade. It's here, and it's now. We're already integrating Gaussian Splatting into Meadow, and soon you'll be able to experience it for yourself.

So, what's next? Well, we're in the process of refining this technology, finding new ways to optimize and enhance the user experience further. We're also working on new tools and features that will allow you to interact with these virtual worlds in ways you've never imagined. So, stay tuned, because as incredible as this all sounds, we're just getting started.

________________________________________

And there you have it—our journey from clunky photogrammetry scans to the mind-blowing, real-time photorealism of Gaussian Splatting. It's been quite a ride, and trust me, you'll want to keep your seatbelt fastened. Because in the Metaverse, the only limit is your imagination, and at Meadow, we're in the business of turning imagination into reality.

Thank you for taking the time to read about this groundbreaking technology and its integration into Meadow. We can't wait to show you what we've been working on. So, get ready to step into a world where anything is possible. See you in the Metaverse!

Joerg Kahlhoefer

Jörg Kahlhöfer is a creative director with over 10 years of experience in the metaverse industry. He has worked with brands like BMW, Gillette, and Oral-B on their global metaverse design projects. In addition to his work as a creative director, Jörg is also an accomplished 3D artist and motion designer. He's led multiple other large-scale metaverse projects that have earned him international recognition for his work.

November 1, 2023