What is Photogrammetry? Mirroring Reality in Games and Beyond

Bringing the real world into the virtual, the technique that has and will continue to bring video games to life plays a powerful role in Unreal Engine 5.

Of the many graphical wonders Epic’s Brian Karis and Jerome Platteaux showed off last week with the new Unreal Engine 5 demo, perhaps one of the most exciting was the updated engine’s ability to import highly detailed models directly, without needing to compromise any level of detail or artistic vision.

While this will allow artists to more seamlessly incorporate their original work, complex assets built from the real world using photogrammetry techniques can also be brought into a game with greater efficiency and accuracy than ever before. This will allow game developers to bring in exact replications of real-world locations, physical objects, and even the physical dimensions of actors with ease, speed, and accuracy.

Simply put, photogrammetry is the process of gathering complex information from the data available from photographs. Essentially, the right photographs of a place, object, or person, can provide enough information about different points in 3D space in order to construct a virtual model that can be used in a variety of applications such as scientific studies of archaeological digs sites, special effects in film, 3D models for video games, manufacturing, engineering, and so much more.

Photographs provide an enormous amount of information that can be analyzed and extracted to build 3D representations. This information can be as simple as measuring the distance between two points on an image, or using the available color range and light information to source accurate and reliable 3D location that can then be reconstructed virtually.

Any camera, even smartphone cameras, can do this. Of course, there exist cameras and other technology specifically used to get the most information available, but sometimes all it takes is a handful of pictures of an object to create a usable 3D reference or model.

There isn’t necessarily any one kind of photogrammetry. The term itself represents an umbrella of techniques and disciplines, each with their advantages and disadvantages for certain projects; and the combination of technology such as LiDAR can allow for even greater accuracy. Each method or discipline of photogrammetry can warrant its own technological deep dive. This article will give you just a basic sense of what photogrammetry is and how it can be used.

The term itself, as well as the technique itself, is also nothing new, dating back to the 19th century when Prussian architect Albrecht Meydenbauer first used the term in “Die Photometrographie.” The 1800s saw photogrammetry first used to create reliable and accurate topographic maps. No computers, just photographs and people who knew what to look for in them.

Photogrammetry represents a legacy of human innovation and technology. While bleeding edge technology like Unreal Engine 5 can take us to incredible places using the latest technology and software, it does so by connecting to the past and making the most out of contributions and developments that go back more than a hundred years ago.

Even if you’ve never heard the term before, you’ve probably been seeing photogrammetry for years now. The Matrix is just one of countless films that made use of the technique, not just for its bullet-time scenes, but also for a variety of the trilogy’s set, and actors. It’s also been deployed in games to bring lifelike representations of actors and locations.

The development of Death Stranding made extensive use of this in order to bring Norman Reedus’ likeness into the game for his portrayal of Sam Porter Bridges, as well as other members of the cast. Kojima Productions did this with an approach tailored specifically for their game, demonstrating the flexible ways photogrammetry can be used to suit certain projects. It also presented a challenge for the actors as they had to adapt to performing in a drastically different environment.

“Death Stranding uses Photogrammetry scans made from 360° photographic data, and performance capture technology to record actor/actress performances. These are then integrated and rebuilt in the game engine. These methods are completely different from those used in movies, so performers need to first think ‘This sounds interesting, I’d like to try this new stuff’ or they won’t take part in the project.”

Hideo Kojima on photogrammetry in Death Stranding

2017’s much celebrated Hellblade: Senua’s Sacrifice also relied on photogrammetry to bring its main character, portrayed by Melina Juergens, to life.

Unreal Engine 5 is not Epic’s first rodeo with photogrammetry, however. Not only have artists and designers been using a combination of the technique with the existing Unreal Engine for games, movies, even set design demonstrated through VR and AR, but Epic themselves have been working away at making the technique second nature to its engine.

About a year ago, Epic announced that they had acquired Quixel, the company’s massive library of photogrammetry assets to be used for free in Unreal.

“Building photorealistic 3D content is an expensive endeavor in game development and film production. By coming together with Quixel to make Megascans free for all use in Unreal Engine, this level of artistry is now available to everyone from triple-A studios to indies.”

Tim Sweeney, Epic Games CEO, on the Quixel acquisition

With Unreal Engine 5 and the upcoming hardware found in new consoles and future PC components, we’re sure to see photogrammetry continue to play a dominant role in upcoming titles. Now that developers will have an even greater ability to import models, however, we’re likely to see a pretty substantial jump in the detail and scope in environments and character design that outpaces what we’ve seen before.

Developers will have the tools necessary to bring real-world locations, both urban and natural environments, directly to life in a game, with the freedom to then edit or shape them to better fit an artistic project. The same is true for models of human actors, meaning that we might likely expect to see more Hollywood faces in gaming as the technology to reliably capture their performances is getting more powerful and more accessible. The techniques that allow Unreal Engine 5 to do this with such seamless efficiency are also likely to show up in other engines and practices across the industry.

With creative software implementation, cutting edge hardware, and techniques that have been used and built upon for more than 150 years, the gap between reality and the virtual, is getting smaller every day.

1 comment

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.

%d bloggers like this: