Hanson Robotics
personalizing live audience interaction
UX design, web developer, 3D Animator
Featured in Consumer Electronics Show 2019
Background
Sophia, Hanson Robotics' main humanoid robot, was debuting her new arms and legs at CES 2019. Previously, she had limited body movement and facial gestures. To take advantage of this update, my task was to create an animation library and technical tools that improved robot human interaction at the CES conference.
Impact
Reusable animation library
Stress-tested and showcased new robot arms and legs by creating 50+ digital animations, including dances, greetings, and dialogues.
Robot operation tool
Created a browser-based tool to monitor robot motor health, and control robot interactions in real-time.
Improve live audience experience
Showcased Sophia at CES to thousands of attendees, news anchors, and industry professionals. Improved interaction experience with new robot operation tool.
Goals for Sophia
- Stress test Sophia’s new limbs through advanced animation
- Improve user experience of Sophia during live audience interactions
Building an advanced animation library
Establishing the animation pipeline
Previously the animation system in Sophia was outdated and required a lot of manual oversight. Before creating the animation library, I needed to update the animation pipeline with newest animation tools and connect it to our proprietary engine.
Preserving animation correctness
I worked with engineers to develop a conversion script that would correctly translate all of the animation from Maya onto our physical robot.
Creating robotic digital twins
Working with roboticists, I optimized the existing CAD models and created digital robot rigs based on the robot’s real-world range of motion
Creating and categorizing animations
Futureproofing our library
- I created robot animations guidelines and built an animation library intended for future gestural AI training
- These animations could be remixed and contributed to a larger animation system used across multiple robots.
- I then categorized all animations under various emotions for easy searchability
Benefits of an animation library
- Sophia's dialogue tree automatically paired with animation categories.
- The large number of animation options increased the believability of interactions with Sophia.
- It also allowed us to rapidly block out a physical scene for bodystorming activities with Sophia.
Discovering live interaction challenges
Live interactions with USC students
We brought Sophia to several school classes to interact with students and monitor Sophia’s live audience interactions with her new arms and legs. Through these sessions, we found several interaction challenges that became apparent as we prepared for the upcoming CES conference.
Current live interaction challenges
Challenge 1: Gaps in automated dialogue
Sophia's automated responses don't cover all audience topics and breaks interaction immersion.
Solution: Puppeteering tool
Ability to interject with customized robot responses during special interaction cases.
Challenge 2: Physical motor fatigue over time
Over time, Sophia's physical motors lose accuracy and affect the overall animation performance.
Solution: Monitor motors and adjust animation
Interface to visualize motor health and easily adjust animations on the fly.
Improving live audience interactions
Low-Fi interaction flow
Live audience puppeteering tool
During the CES conference, the show hall will be filled with thousands of people and sounds that could easily interfere with Sophia's 1-1 interactions. I created a tool that would give robot operators the ability to step-in at any time with custom dialogue or animations on the fly.
design 1: Continuously moving timeline
- Form factor familiar to animators
- Focused on upcoming actions
- Live-generated responses can’t take advantage of timeline space.
design 2: log style sequence cards
- Focuses on mainly on the current action.
- Liked the ability to see more information in previous timeline design with less scrolling.
- Wish we can insert animation and speech at the same time.
design 3: combined timeline of sequence cards
- Nice to have option to insert a combination of speech and animation
- timeline + sequence card layout gives more context to operator without constant scrolling
Monitor and adjust motor controls
My goal was to create a tool that gives us insight into when physical motors were not performing as expected and therefore affecting the animation performance. When the motors movements were still within an acceptable range, we could then adjust the intensity of the animation without dismantling the robot, which would take much longer.
Version 1: maximize control
- Good for making precise adjustments
- Most viewers don't notice such tiny detailed changes
- Making changes takes too long and too much effort to map axis to robot animation
Version 2: simplify control
- Simplified controls
- Added animation tweak for entire limb for quick adjustment
- Added visual of robot limb for clarity
Finalized Designs
Showcasing at CES and results
- Successfully debuted Sophia’s new arms and legs with animation library
- Sophia met with thousands of CES attendees and we used our live puppeteering tool to improve the interaction experience during 1-1 conversations.
- Motor motoring system was useful especially during short downtime when there’s enough time to test the result of tweaking the animation intensity.
