Maryyann Crichton

M.A.A.D

Experience Designer / Software Developer

Project Goals

Explore the potential of interactive XR narrative by combining Hololens, computer vision, and physical puzzles.

Oculus NextGen Symposium
USC Annenberg Graduate Fellowship Research
and Creative Project Symposium 2018

Hololens / Unity / Maya / Vuforia / Photoshop

Summary

M.A.A.D is an XR installation that combines the physical and virtual environment through the use of SFX, lights, and puzzles. Following a classic Hero's Journey archetype, Players meet an Alien named Adam to solve various Rubik's cube puzzles and restore the Power of Transformation back to Earth.

experience flow game development playtesting

Experience Flow

My role was creating a clear line of communication between the technical and creative teams. My teammates did not have prior experience with Hololens or game development. I combined prototyping techniques found in traditional animation storyboarding and game development to ease that collaboration.

Experience Outline

Our goal was to explore the potential of character driven XR narratives, through the combined use of physical and virtual puzzles. We first created a text outline that listed out each task per department. This broke down the main experience but also highlighted major issues.

Experience Outline; maya animation, interaction implementation, and environment installation

Outline Observations :

1. Players lacked context before beginning the experience

2. Experience requires interference from staff to restart

3. Text outline lacked clear visualization of space

The 3rd observation shows the importance of our physical installation. In this outline, the creative team had little context for how to imagine the visuals in the final result.

Solutions :

1. Players learn about lore in waiting line

2. Hand-made holster holds Hololens in same place every start and end of experience

3. Create storyboard paired with SFX notes and scene descriptions

Experience Map and Storyboard

A top-down map helped us visualize the physical layout of the environment and our targets for Player positions at each step of the experience. The installation is a 10ft x 10ft x 10ft physical blue cloth cube, lit only by limited LED lights hanging from the ceiling. I made a storyboard that paired information from the experience maps with visuals and SFX.

experience map; each position can contain multiple actions; rubik's cube puzzle in top-right corner

storyboard; Combines SFX, animation, Player instruction, context

Game Development

We developed the physical and virtual environments together. This was a highly iterative process. To gain better spacial understanding of our project, we role-played the experience in the final physical room. This gave me world positions to transfer into a virtual room and position the virtual character's movements. I created prototypes in Unity to test Vuforia within our physical installation and programmed all of Adam's interactions with the Player.

Rapid Prototyping and Technical Challenges

testing Vuforia in Hololens for stability with moving trackers

Proof of Concept - Vuforia

• Tested stability for detecting tracker design on physical Rubik's cube

• Found good balance between tracker line thickness that allow Players to finish puzzle before triggering final animation

• Decided on 2 x 2 Rubik's cube after testing Player's tolerance

Testing the Experience Flow

• Gained knowledge of limitations betweeen virtual and physical space

• Observed character speed is too quick and needs further iterations

• Discovered Hololens becomes unstable under dim conditions.

Our confined space had limited lighting, specifically designed to give a mysterious mood. However, Hololens did poorly under dim lighting conditions. To combat this issue, but keep our aesthetic we installed strings of LED lights that matched the virtual environment.

testing experience flow in Hololens with designed trackers

Bringing Adam to Life - animation state machine

• Adam captured our hearts

• Character size needed to fit within Hololens FOV safe & focus zone

• Adam required more transition animations than initiallly expected

After down-scaling Adam to fit inside the Hololens FOV, we found it hard to notice Adam in the introduction sequence. I added spacialized sound triggers to Adam and the environment to direct the Player's attention.

Physical installation

Playtesting

MAAD took about 4 months to complete. We conducted about a dozen playtests with varrying levels of success and feedback. This was the first Hololens experience for most Players, so we recorded the playtests for further observation. We also displayed MAAD at the Oculus NextGen Symposium and presented at several events including USC Annenberg Graduate Fellowship Research and Creative Project Symposium 2018 and USC Games Expo 2018.

Feedforward vs Feedback

Our metric for success depended on how well Players understood the story and how easily they navigated the space.

First-time Hololens users testing out MAAD

Feedforward

• Virtual character reacts verbally and visually to proximity of Player

• Rubik's cube is brightly lit with decorative lights to draw attention

• Virtual Character's clue animations mirror Rubik's cube solution

Feedback

• Players reacted positively to Virtual Character, but often confused about goals.

• All Players correctly picked up Rubick's cube puzzle

• All Players successfully solved all 3 puzzles. Some had trouble with Hololens tracking.

Conclusion

Goal — Explore the potential of character driven XR narratives, through the combined use of physical and virtual puzzles.

Result — MAAD was 70% successful.

We achieved our goal of combining the physical and virtual environment into a single believable space through physical puzzles and an adorable virtual character, Adam. We recieved lots of positive reactions towards Adam. The spacialized sound triggers and confined environment helped orient the Player, but Players still have some difficulty determining goals. Sometimes Players are uncertain about what they are allowed to do. Unstable hardware tracking also added to Player's dissatisfaction.