This year, ILMxLAB celebrates its eighth year of producing award winning immersive entertainment. Since its founding, ILMxLAB has created multi-part episodic experiences, innovation experiments and location-based adventures incorporating top-tier narrative content into emerging interactive display mediums.
The release of its updated VR experience, Star Wars: Tales from the Galaxy’s Edge -- Enhanced Edition, scheduled for a February 22 debut on PlayStation VR2’s launch date, is the latest project from the innovative studio.
First introduced as a two-part experience on Meta Quest in 2020 and 2021 as Star Wars: Tales from the Galaxy’s Edge, the Enhanced Edition has been rebuilt to take advantage of the PS VR2 hardware. Eye tracking cameras built into the headset follow a player’s line of sight, foveated rendering keeps the 4K HDR environment richly detailed from a player’s viewpoint, and haptic feedback adds to the sense of immersion.
Players start the experience as a Droid Repair Technician in the Batuu wilds. There, they can infiltrate a First Order facility and face off against the Guavian Death Gang. As they progress through the game, they can travel to other places and eras within the Star Wars galaxy and become other characters, even a Jedi. The state-of-the-art VR experience expands the world of Star Wars: Galaxy’s Edge at Disneyland Resort and Walt Disney World Resort. Vicki Dobbs Beck, vice-president, immersive content, Lucasfilm and ILMxLAB, calls it “storyliving.”
Dobbs Beck led the teams in San Francisco and Singapore and virtually who created this latest project. For her, storyliving is a logical extension of a 30-year journey at ILM into interactive storytelling. In the early 90’s, she was in charge of the now defunct Lucasfilm Learning.
“We were at the intersection of interactive storytelling and high-fidelity media with a lens on project-based learning,” she says. “I became passionate about the intersection, but we were probably decades ahead of the time.”
In 2013/2014, when Dobbs Beck saw the convergence begin to become viable, she supported the idea in a strategic planning role across the Lucasfilm divisions. And then in 2015, she co-founded ILMxLAB.
“I saw an incredible opportunity to invite people to step inside technology in ways that hadn’t been done before,” she says. “We started by inviting people to step into our story. That has now evolved further – from storytelling to storyliving.”
And to award-winning content. Vader Immortal: A Star Wars VR Series created with Oculus Studios received an Emmy nomination. Episode I of Vader Immortal won the VR Experience of the Year from London VR Awards in 2019 and in 2020, the inaugural PGA Innovation Award and GDC’s Best VR/AR game. The collaboration with director Alejandro G. Iñárritu, CARNE y ARENA, a location-based experience that immerses people in an immigrant’s journey, received a Special Achievement Award – an Oscar – in 2018.
Creating a “storyliving” experience incorporates four major elements, Dobbs Beck explains: a holistic approach, discovery, connection, and an unfolding narrative. A rich Star Wars empire that has evolved and expanded over four decades of film, television series, and game content, helps make it possible.
A Holistic Approach
Most of ILMxLAB’s projects, including Marvel Studios’ Avengers: Damage Control, a location-based adventure created in collaboration with The Void, are tied to Disney IP. Many projects involve Star Wars characters and places, and Dobbs Beck provides an example of a holistic approach based on two: The multisensory Star Wars: Secrets of the Empire, created in collaboration with The Void that opened in Disney Springs at Walt Disney World Resort in Orlando and Downtown Disney at Disneyland Resort in Anaheim in 2017, and the Vader Immortal: A Star Wars VR Series experience released on Meta Quest and PlayStation VR.
“Sometimes I talk about a holistic approach as a mosaic of experiences where, as in a mosaic made of tiles, each tile , each experience, can stand on its own, but seen together becomes a rich design,” Dobbs Beck says. “Star Wars: Secrets of the Empire can invoke all the senses in a 100% controlled environment. It is a prequel to Vader Immortal. Each is great on its own, but when you do both, you see a bigger picture. In Empire, for example, you know you’re supposed to recover a crate that becomes relevant for Vader Immortal.”
Discovery
The second element in creating a storyliving experience is what Dobbs Beck calls “discovery,” that is, offering rich, well-detailed worlds that people discover while participating in a story and want to be in.
“It’s super important that when people are exposed to a new device, their first experience with VR headsets is premium,” Dobbs Beck says. “It’s important to us to deliver high-fidelity VR.”
Connection
Third, is connection. The challenge inherent in having a rich world of content to draw from is the necessity to preserve the integrity of that intellectual property. That’s true especially when a player connects and interacts with a character.
“Vader Immortal hinged on whether you could have a meaningful relationship with a compelling character, ”Dobbs Beck says. “The test was in the final product. There’s a scene where you’re in a cell. Now remember we know where your eyes are. The door opens and Darth Vader walks toward you. You hear his classic deep breathing. He comes right up to you and looks you directly in the eyes. He delivers a line that many people don’t even hear because his presence is so overwhelming, so allconsuming. It proved people can have a meaningful relationship with a character in VR, not just observe other people’s relationships.”
An Unfolding Narrative
The fourth storyliving element is creating an unfolding narrative.
“We have focused on curated stories told by master storytellers,” Dobbs Beck says. “Your choices make your journey, your personal experience. But you don’t change the outcome. We’re starting to consider what a branching narrative looks like. You would choose where to go and where you don’t go isn’t part of your experience. We’re also looking at community storytelling where the story unfolds as a result of your and other people’s presence in the world. We’re always moving on to the next bleeding edge project.”
Connecting to VFX
ILM’s advanced technology group supports both ILMxLAB and visual effects with core technology that can be applied to various purposes. The visual effects side takes advantage of VR and AR as production tools, and use Stagecraft, the LED-based volume with real-time graphics. ILMxLAB can use assets from visual effects production, and visual effects can use real-time assets as starting points for films.
“We have a concept of a digital backlot, and it’s a real, meaningful objective,” Dobbs Beck says. “We keep assets in a central repository and do whatever we need to make them useful. The movie assets aren’t a straight port in, but they can get us a long way there. And we can build games on worlds created for the location-based immersive experiences.”
ILMxLAB is also testing ideas that push beyond existing location-based installations and games to, as Dobbs Beck puts it, “transcend the physical, digital and virtual worlds.” For example, they implemented an AR experiment called Project Jigsaw on Lucasfilm’s campus at Letterman Digital Arts Center in San Francisco.
“We had camera phone viewing vignettes in four places on campus and since they were always on, if you went to the right place, you could see them,” she says. “Because our emphasis is on trying to deliver the most cinematic quality experience we can, we pieced together alot of technology that hadn’t been done before. We did off-device rendering for this AR-based storytelling. The phone became a thin client – the computation in the cloud was recomposited on the phone. So, because we knew where the sun was, our AR characters’ shadows made sense in the real world. And, they could pickup reflections. It’s astonishing how real the vignettes looked and felt.”
Future Possibilities
Experiments such as Project Jigsaw offer glimpses into how immersive experiences could evolve. Along those lines, Dobbs Beck offers a set of intriguing possibilities for the future of storyliving.
“I’m excited about connected storytelling,” she says. “If you have filmmakers or storytellers that think multi-dimensionally, you can design an ecosystem of immersive experiences that are compelling beyond anything we’ve seen to date. You can have persistent and evolving worlds that fundamentally change in some way, big or small, based on how people who visit them engage with them. Once anything in your world is available in the virtual world, it can be all around you. Screens are everywhere. Phone, iPad, contact lenses, VR glasses, AR glasses. We’ll see virtual screens with AR glasses. And wherever there’s a screen, there’s an opportunity for a story.”
In 2018, ILMxLAB partnered with designer Steven Tai and the Fashion Innovation Agency in the London College of Fashion on a project called Live CGX. The studio created digital versions of two garments designed by Tai. Then, during London Fashion week, the audience watched on a massive screen without the need for glasses or another device as ILMxLAB realized garments in digital form on alive model in realtime.
“Our relationship to reality is fundamentally changing as digital and real become layered on top of each other, ”Dobbs Beck says, and provides a personal example.
“One of the Project Jigsaw vignettes had stormtroopers,” she says. “I remember the place where I experienced that. I remember that moment with stormtroopers. It’s a very real memory, and it’s a blend of digital and physical.”
Those AR characters were gripping because they looked like they belonged in the environment. The next step would be to have a conversation with an AR character.
“What I think is important for the future is compelling AI characters that can engage in conversation in a way that’s believable,” Dobbs Beck says. “We’ve been talking about it for years, but the latest advancements in AI are extraordinary. We may be closer than we thought. And, these characters may manifest as holograms.”
In fact, holograms are iconic Star Wars elements that date back to the original films, and you see them in the streaming series Andor and The Mandalorian.
“Holographic messages are in most if not all of our Star Wars immersive stories from Vador Immortal toTales from the Galaxy’s Edge,” Dobbs Beck says. “While we’ve played with this element frequently in VR, there’s a real opportunity to bring holograms to life in AR, and we’ve already started. In Project Jigsaw, the story starts with a holographic message from Hondo Ohnaka, a fan-favorite character from Star Wars: Galaxy’s Edge. Seeing his character brought to life in AR as a hologram brings the experience to life in such a delightful way. Beyond Star Wars, the possibilities for applications of AR holograms feel endless, and it’s something ILMxLAB will continue to explore.”
The way we interact with characters might change in interesting ways, as well. “I’m imagining how biometrics, our bodies, will impact virtual experiences,” Dobbs Beck says. “Boundaries are disappearing. Right now, you might unlock something in Vader Immortal with a controller. But what if you unlocked it by reducing your heart rate to a meditative state on your smart watch.”
Moore’s Law has technology advancing by an order of magnitude every five years. Given that, and what the studio has accomplished in the past eight years, the story we’ll all be living in the next eight years seems within the possibilities Dobbs Beck has imagined.