Shiseido Beyond Time

The Challenge

For the launch of Shiseido’s Global Innovation Center in Yokohama, Japan, we were asked to create an experience that would bring to life their new brand purpose: Beauty Innovations for a Better World.

The experience also needed to highlight Shiseido’s long history of scientific research and make visitor’s see Shiseido for the innovative company it is. To achieve all this, we built a time machine.

The Context

Japan has the world’s fastest-aging population. Yet 9 out of 10 Japanese fear growing old.

Shiseido has decades of scientific research on how Japanese age, and a whole department dedicated to skin science. We wanted to harness that data for good–this time, helping people embrace the beauty of ageing.

The Experience

Beyond Time is an immersive, interactive, deep-tech installation that allows two people to seemingly break the boundaries of time. The project leverages Shiseido’s facial data to develop a world-first, scientifically accurate, fully 3D, real-time Age Simulation Engine.

Instead of focusing on how we look at different stages in our lives, the experience was designed to help people reassess their relationship with time–through the lens of their closest relationships.

Participants can experience the spectrum of aging with a loved one, in an instant: a parent becomes younger than their child; lovers glimpse each other before they met; friends grow old together in the blink of an eye. Every time jump prompts conversations around our memories, hopes and dreams, building to a growing realization of what we can look forward to when growing older.

Research & Prototyping

Beyond Time required us to dig deep into facial mapping, three-dimensional data capture and processing, skin and wrinkles science and more. We had to leverage cutting edge scientific research, under intense close collaborations between our design and technology teams and Shiseido’s research team. To accomplish this hefty task, we had to follow a different approach.

We initially kicked off by gathering creative tech and data consultants from the network to assess the feasibility of such complex project. We knew we could achieve some level of accurate age simulation, but we didn’t exactly know how far we could go.

Then we casted a core team of creative technology and experiential design, to research and study different methods to bring a working system to life. The team looked at all existing hardware and software technologies and research for approximately three months. The team created dozens of prototypes, from screen and camera hardware, to facial mapping and facial filter proof of concepts.

We concluded that existing technologies and platforms could not be leveraged in the bespoke way we needed, and setup to put together a team that could built the tech from scratch according to Shiseido’s requirements and the experience goals.

Production Engineering

During the next 6 months, we put together one of the most talented tech teams to the date. We brought together a world-class solution architect from Costa Rica, a creative technologist and a hardware creative prototype expert from Japan, a creative engineer from New York, along with our original creative technologist and researcher from China.

They paired with visual and experience designers to create bespoke solutions for the physical installation, the depth camera solution, the age simulation engine, the textures and shaders renderer and the user interface and animation components, hardware integrations for transparent, electric film walls and synchronized light animations, all running on a distributed system that worked across three different high spec computers.

During these six months we developed breakthrough technologies in three-dimensional data capturing and processing, face feature mapping, real-time shader manipulation, real-time average face model matching, face texture lighting mapping, texture mapping composition and dynamic wrinkling.

The 3D Facial Age Simulation Engine the team developed looks at 1,300 facial data points, and compares them to a vast database of three-dimensional face models we created, enabling a scientifically accurate aging simulation in real-time.

These achievements were done in record time by a small core team, which is in itself an outstanding achievement if you consider that companies like ModiFace or PerfectCorp have invested millions of dollars and years of research with large engineering teams!

Minimum Viable Experience (MVE)

One of the big learnings from this project is the need for focus and getting the end-to-end journey to work as soon as possible. On such complex projects, it’s easy to get distracted on surrounding elements, and focus too much in isolated details.

During the next three months, I shifted gears, and empowered the engineering team to cut off all the noise and focus on a fully working, end-to-end minimal viable experience. Since the experience underlying technology had to work for real, we enabled the team to direct and define the tasks, process and priorities to get the experience working.

Getting to the minimal viable experience allowed us to test the end-to-end journey with real people in a semi-real environment and quickly understand the shortcomings and issues people were reporting. This shift in mindset preventing us from launching an experience that we thought was great but didn’t resonate with actual people. It was precisely this shift that saved the project from failure.

Data Capture

The underlying core engine of this project was the three-dimensional facial model database that enabled scientifically accurate age simulations.

Our core team created a breakthrough data capturing process to capture more than 600 participant’s facial data, create anonymized average face models, deconstruct age textures, and create one of the largest 3D average face databases in the world.

It was because this very core data capturing and processing engine that the team was able to create such an advantage age simulation engine, and all while strictly protecting participant’s private facial data.

We then leveraged Shiseido’s skin and aging science knowledge to re-create the aging textures for different age buckets and facial structures from the three dimensional average face models, enabling the engine to generate real-time, tailored shader models that could adapt to people’s facial structures, current age and follow their movements and light reflections in a seamless way.

Launch

Once we had a working MVE, we could clearly identify the things we needed to prioritize and tackle for launch. Working closely with Shiseido’s creative and research team, along with Nomura Lab architecture firm, we moved our team to the Global Innovation Center and worked for the last few weeks on site, as construction was being completed, to launch the experience.

This was probably one of the most intense faces, having everyone in the team working at full speed to get all details finished, integrated and tested for launch. Elements like physical design, hardware integration, copy and interface final touches were added just in time for the building inauguration.

Results

The first-of-a-kind installation required 14 months of deep technical research and leveraged cutting edge scientific methods to capture, process and recreate scientifically accurate simulations in real time, seamlessly integrating audio, visuals, stories and technology in a unique and intimate experience set out to help people re-evaluate their relationship with aging.

Over the first few months people queued daily to participate in the installation, sometimes waiting up to 4 hours. More than 4,000+ have experienced the installation, sparkling more than 10,000 conversations about aging.

Extending the Story

We also created an extended installation on the second floor of the GIC building, where users could go through the story of how we brought the project to life, the implications of skin science, and a deeper look at challenging the preconceptions of age.

The Team

On a project of this nature, it’s absolutely vital to have the best team talent in place, and foster a close collaboration way of working. Jedy Chen, Senior Creative Technologist and Toru Nagahama, Design Director were the core due that drove this project from start to finish. Additionally, having Marvin Marela, Senior Software Architect, Kumi Tominaga, Hardware Prototyper, and Ayumu Nagamatsu, Senior Creative Technologist complemented our core tech + design team that made this project a reality.

Of course, many more people had an important part on making this project come to life, from visual and experience designers, producers, copywriters, storytellers, researchers, architects and builders.

Also, such a research and experimentation driven project wouldn’t have been possible without a trusted partnership between our client team at Shiseido and the GIC and our internal network teams.