FurElise for Piano Trajectories
AI can play piano with human like motions
Paper: FürElise: Capturing and Physically Synthesizing Hand Motions of Piano Performance (16 Pages)
Researchers from Stanford University are interested in realistically simulating human piano playing.
Hmm..What’s the background?
Physically accurate human motion synthesis has numerous applications in fields like animation, AI, and biomechanics. Current research has focused on functional movements like walking and object interaction. The next challenge is to simulate movements requiring elite-level skill and precision, such as those seen in athletes and musicians. As piano playing is a demanding skill that requires Precision, Coordination and Dexterity, it is important to create a large-scale datasets capturing the complexity of real piano performances.
Ok, So what is proposed in the research paper?
The Fur Elise dataset comprises approximately 10 hours of 3D hand motions from 15 elite pianists playing 153 classical pieces.
Hand motions are reconstructed from the videos and refined using inverse kinematics, guided by high-resolution MIDI data from a Disklavier piano
Provides synchronized audio and motion data, enabling diverse applications in animation, music pedagogy, and even injury prevention
Notably, The model from the paper uses a diffusion model to generate initial hand trajectories and fingering information. Trains a reinforcement learning policy to control physically simulated hands, ensuring interactions with the piano keys are accurate and physically plausible. Experiments demonstrate the ability to handle complex piano techniques like chords, fast wrist motions, and arpeggios, generating natural-sounding music.
What’s next?
For future work:
Sound Amplitude: Incorporate key-pressing velocity into the model to generate music with varying amplitudes, as the current system produces sounds with constant amplitude
Realistic Hand Model: Utilize a more realistic hand musculoskeletal model driven by muscle activation, enabling better biomechanical analysis and injury prevention insights
So essentially,
AI can play piano with human like motions
Learned something new? Consider sharing it!