everybody move
i Designed a dance game with body-tracking and gesture-recognition technology for a childrens’ museum
Team:
Ash Tiwari
Lead interaction engineer
Kensuke Sato
Interface designer
Michael Cooper
Systems engineer
Timeline:
July 2021 - June 2022
*in conjunction to other projects
Key responsibilities:
game design, development and mechanics
Designed and developed the game in Unity C#, using Azure Kinect for body-tracking and gesture recognition, and incorporated fundamental gameplay mechanics such as scoring, player health, and scene management.
User research and accessibility adjustment
Adjusted the gameplay to accommodate players with varying physical abilities by automatically detecting shoulder and knee positions to switch between seated and standing modes.
pose detection and synchronization
Programmed the game to check pose accuracy at a trigger line, providing visual feedback and synchronizing pose movements to the beat of the music.
overview:
Everybody Move is an engaging body-tracking game designed for kids to encourage exercise and educate them about its benefits. Developed in Unity, the game presents a series of poses for users to mimic, using body-tracking to determine if poses are correctly matched. With three levels of increasing speed and difficulty, the game synchronizes pose appearances to the beat of the music, creating a dynamic and enjoyable experience for children.
setup:
For this exhibit, we used an Azure Kinect camera with depth tracking, allowing us to perform body tracking and gesture recognition. The exhibit is housed in a separate chamber open on the left and right to prevent sunlight or artificial light from affecting the depth tracking
research and development:
Our research started with an analysis of popular dance games like Just Dance. We examined how dance poses are designed for different difficulty levels, the types of bonuses offered, and the factors contributing to the game's replayability.
Using an Azure Kinect camera and Azure Kinect Body Tracking SDK, we designed an initial test in Unity to evaluate the accuracy of input data tracking and processing, testing gestures such as jumps, raising right hand, raising left hand, etc. We mapped body joints to an on-screen mirrored skeleton to assess the model's precision.
Pose creation and synchronization:
Following our research, we developed a system by labeling the joints and creating various poses, then combining them to form a set for users to mimic. We defined a choreography and synchronized each pose to appear on specific beats. A trigger line was included, requiring users to mimic the pose by that point to gain or lose a point.
Our choreography primarily focused on arm movements, with a few leg movements selectively included. This design ensured that leg movements could be easily omitted in the seated mode without significantly impacting the gameplay experience.
user experience and gameplay:
Isolating the user experience in a high-traffic environment:
Since this exhibit is located in a busy children's museum, we needed to isolate the experience to one person at a time and avoid interference from people standing nearby. To achieve this, we designed a mat to be placed in front of the screen that defines the play area. Using depth tracking, the game was programmed to recognize and track only the person within this designated distance, ensuring an uninterrupted experience.
automatic mode detection and accessibility adjustment:
To streamline the experience and avoid inconveniencing users, we chose not to require users to select between standing or seated mode manually. Instead, at the start of the game, the depth tracking system detects the user's joints and measures the distance from shoulder to knee to determine if the user is standing or seated. If seated, the game automatically switches to seated mode, adjusting by removing leg movement poses and changing the bonus action from jumping to raising a hand.
quick and engaging gameplay:
The gameplay was designed to accommodate users with limited time, ensuring the entire experience lasts no more than 5 minutes. Upon detecting a person, the game prompts the user to raise their arms to start. Each user begins with 3 hearts as a health bar and quickly receives brief instructions: fit through walls by mimicking the poses to gain points or lose a heart for each missed pose. The choreography lasts only a couple of minutes and is divided into 3 levels, gradually increasing in difficulty. Completing the choreography or losing all hearts results in a game over, with the final score displayed.
A bonus element was also added where users can earn an extra heart by jumping (for standing players) or raising their hand (for seated players)
final product
✦
final product ✦
final video
A final test during installation at the Health Inside Out museum in Wichita, KS confirmed that the dance game was perfectly functional and enjoyable.