Yokai!

Yokai!

Technical Artist, Experience designer, Production Design

Goals

Explore the potential of interactive narrative by combining Hololens, virtual characters, mobile devices, and physical puzzles.

Summary

Yokai! is an XR installation where users offer physical toys onto a shrine to interact with virtual charactesrs and environemnts. The player is a weary traveler in Tokyo greeted by an unexpected Yokai named Zashiki dwelling in their AirBnB. Zashiki is causing chaos and the only path to peace is to make her happy.

final results research & concept dev worldbuilding experience flow playtesting

Final Results

Research & Concept Development

I began the project by researching Yokai in Japanese culture — what is a Yokai? what's their relationship with people? I found Yokai, an illustrated database of Japanese folklore, especially helpful. Throughout this process, I asked myself the following questions — How do we utilize the physical environment to build a relationship with a digital character? When do these two worlds overlap?

Experience Goals and Limitations

Our goal was to explore potential of interaction between people and digital beings through the combined use of physical and virtual puzzles. Users need to offer Zashiki three items on her shrine in the correct order to pass her test.

initial yokai concept research sketches

Experience goals :


1. Retain authentic cultural relevence


2. Balance the physical and virtual environment


3. Minimize gestural interaction and maximize novel output


The 3rd experience goal accounted for designing within the restrictions of Hololens gestural inaccuracy. Players experience Yokai! with no instructions. Clues come from phone SMS and the digital Yokai, Zashiki.


Zashiki Warashi is a childlike house spirit, often considered to be the god of luck. I based items players offer to Zashiki off of the Yokai's love of candy and toys, and her ability to bring good fortune.


Designing for Hololens :


1. Characters fit comfortably in the focus zone


2. Essential content lies within the safe and focus zone


3. Blur the edge of viewport for smoother transition


There are no hard and fast rules to designing for Hololens. AR headsets are still a developing medium. After multiple initial tests around screen space, color, and interaction, I came up with 3 guidelines.


Hololens Field of View design layout guideline

Worldbuilding

Designing Yokai Characters and Environment

I created all of the assets and animations. To find the most trackable character size on the Hololens, I performed multiple tests with different Zashiki sizes. We went with a smaller character with bold colors and spacial sound effects that also helped players follow her around.

For the set, I created sketches and 3D mockups to give a better sense of space. To emphasize our first experience goal of retaining cultural values, all of the set decor are from traditional Japanese household items — Kotatsu, Tatami mats, Kamidana Alter. For the virtual interactive objects, I hid clues inside of traditonal Japanese painting scrolls that activated based on a player's position.

Zashiki creation process; concept, model, rig, animation, implementation

Physical set creation process; 2D and 3D mockups

Character animations; Zashiki and other Yokai

Experience Flow

Outline and Storyboard

Our approach to designing the experience was quick iteration and repeated playtesting. We first created a text outline to brainstorm the experience flow. I also created storyboard sketches on sticky notes that we could easily reorder.


Throughout this process, I asked the following questions — How do we make Zashiki believable? How can we utilize both physical and virtual spaces? How do we introduce interaction with minimal Player effort?

Experience outline; draft 1 understading experience flow

Design Highlights:


1. Characters react in a range of emotions to Player actions


2. Players solve physical puzzles to interact with virtual characters


3. Players gain insight into goals through multiple access points


To make Zashiki more believable, I created animations that give Players instant feedback to their actions. To add more immersion, I created many virtual household objects that collide with the physical environment.


Each virtual clue links to a physical puzzle piece Players pick up. Connected with RFID tags, the physical objects will trigger changes in the virtual space, giving Players a moment of magic.


Our main audience had no previous Hololens experience. I replaced unfamiliar Hololens interactions with SMS from a mobile phone and gestures from the virtual character. This lowered the barrier of entry.

Storyboard sketches of Yokai! through Hololens POV; cutout characters to easily move around

Experience Map and Story Flow

We initially designed Zashiki to dance in a Player's periperal vision, but Players had difficulty noticing her in the small FOV.

In the final iteration, Zashiki constantly floats within the 'focus zone' of the Hololens FOV. To direct a Player, Zashiki will float towards one side of the screen and gesture until the Player's gaze follows. Animations also helped communicate our intentions.

I created a map alongside a story flow that showed where we should direct Players to trigger the next set of actions.

Experience map; corresponds to player's ideal physcial position in experience flow

Virtual Scroll Clues;
detailed concepts corresponding to items players must offer onto the Shrine

Playtesting

The entire project took about a year to finish from concept to documentation. We conducted about a dozen playtests inside the lab with varrying levels of success and feedback.

Final Results and Feedback

Our metric for success was based on emotional connection to Zashiki and ease of interaction.

Setup of Kotatsu, Tatami mats, and Kamidana Alter

Key Highlights


• All playtesters successfully finished the main experience, but only 1/4 completed the ending.


• Age of playtesters ranged from college undergrads to 50 yr old adults.


• All playtesters showed disbelief and joy when physical actions triggered virtual changes.

Conclusions


• The ending required clearer incentive. Zashiki gestures above the Kotatsu and flys beneath it, but playtesters were not persuaded to look underneath.


• Minimizing hololens specific gestures successfully lowered barrier of entry.


• Successfully maintained novelty by combined use of virtual and physical interactions.

Designing for headsets is challenging, even in a controlled space.
Players get disoriented or miss important information. We must always design with hardware restrictions in mind.

Yokai!

Directed by

Scott Fisher

Producer

Scott Fisher

Production Design

Maryyann Landlord

Scott Fisher

Interaction Design

Scott Fisher

Maryyann Landlord

Visual Development

Maryyann Crichton

Character Modeling

Maryyann Crichton

Animation

Maryyann Crichton

Character Rigging

Maryyann Crichton

Software Engineering

Satrio Dewantono

Hardware Engineering

Satrio Dewantono

Engineering Support

Chinmay Chinara

Awu Chen

Sound Design

Crystal Chan

Sound Advisors

Kojiro Umezaki

Bethany Sparks

Ideation Support

Perry Hoberman

3D Printing

Biayna Bagosian

Japan Cultural Advisors

Kojiro Umezaki

Matt Alt

Hiroko Yoda

Voice Actors

Takashi Uchida

Crew

Yaya Veerasilpa

Enrico Targetti

William Crichton

Keshav Prasad

Wetware
3D Desktop Game
TikTok
Livestreaming AR
Momento
XR interactive installation
Steelcase Virtual Design
VR collaborative workspace