Teach a robot to dance and it will learn to walk among us

We have spent seventy years teaching machines the things we find hard, on the comfortable assumption that the things we find easy would arrive free with the packaging.

They did not. Hans Moravec noticed this in the eighties and stated it more plainly than anyone has managed since: it is comparatively easy to give a computer adult-level performance on an intelligence test, and hard or impossible to give it the perception and mobility of a one-year-old.¹ Chess fell in 1997. Reaching across a cluttered table for a mug you have never seen before, in a room with a person in it, without breaking either, is still a research programme.

The received answer to that is sensible and I want to give it its due. If you want a machine that can move through a human world, train it on the human world: walking, reaching, opening drawers, everyday activity, hours of teleoperated kitchen work. Teach it the median motion; the median is what it will be doing all day anyway.

I think that trains for the corridor and not for the world.

My claim is this: dance is the hardest, most human motion there is, and it is therefore the motion a machine should learn from if it is ever going to move among us with ease rather than as a hazard or a joke. Everything I am building points at that, and the reasons are not sentimental.

1. Balance is not a state. It is a debt you repay every step

The intuition most people have about walking is that it is a sequence of stable positions, the way a ladder is a sequence of rungs. It is not. Walking is a fall that keeps getting interrupted: you put your weight past the point of support, and the leg arrives in time.

Robotics learned this the elegant way. In 1990 Tad McGeer built a machine with no motors, no sensors and no controller at all, stood it at the top of a shallow slope, and it walked down: a knee, a hip, gravity and good design producing a gait that looked eerily human.² The lesson was not that we can skip the computer. It was that legged locomotion is a conversation with gravity rather than a series of poses held in sequence.

Now watch a dancer. Every interesting thing they do is that same interrupted fall, taken further out and recovered later. Weight thrown past the base of support on purpose, an arm sent out as a counterweight, a foot arriving a fraction after the point where an engineer would have called it a fall. A dancer spends most of a phrase in the region a control system would flag as a failure, and the entire skill lies in getting back: not in avoiding the edge but in living out there and returning, over and over, on time.

Nikolai Bernstein, who was measuring exactly this in Moscow in the 1930s with cyclical photographs of blacksmiths striking, gave us the phrase for the strangest part of it: skilled movement is repetition without repetition.³ A blacksmith who has hit the same chisel ten thousand times does not repeat the same joint trajectory twice. The tip lands in the same place; the body gets it there by a different route every time, because the body is solving a problem, not replaying a tape.

That is the property you actually want in a machine that has to live in a room with furniture and people and a slightly wet floor. Not a perfect trajectory. A thousand different routes to the same landing.

2. Fly the envelope to its edges

Picture every position a human body can take as one enormous space: every angle of every joint at once, the hips, the knees, the twist of the spine, the reach of each shoulder. Anything a body does is a path through it. A walk is a short loop traced over and over; a reach for a mug is a brief excursion off to one side and back.

Now gather the median data, hours of walking and reaching and opening drawers, and mark every place those paths go. They fill a small region near the middle and leave the rest of the space dark. Train a machine on that and its world gets an edge exactly where the data stopped. Inside it, the machine is fluent. Outside it, the machine has never been, and the outside is where everything in the last section lives: the slip on a wet floor, the shoulder that clips you in a crowd, every fall that has to be caught. The limit was never in its motors. It was in its teacher.

Dance goes the other way. Taken across enough dancers and enough nights, it sends every joint toward the end of its range, in combinations no corridor would ever ask for: a leg thrown wide while the spine wrings the other way, the arms overhead with the weight somewhere it should not be. In the limit, dancing maps the whole envelope of the human form, about as near to its edges as people ever get.

Train on that, and the corridor stops being the curriculum. It becomes a small path deep inside known space. Walking down a hall, reaching across a table for a cup: these are gentle cases of moves the machine has already made at full stretch, and recovered from.

Test pilots have a name for this. An aircraft's envelope is the range of speeds, altitudes and angles it can safely fly, and a new one is mapped by flying it out to those edges on purpose, so that ordinary flight afterwards is simply the middle of a space somebody has already explored.

A humanoid deserves the same treatment. Teach it the edges, and the middle comes free.

3. Dance has a deadline

This is the piece I think is genuinely underrated, and it is why I keep coming back to dance rather than to sport or parkour or any of the other places you find athletic motion.

Dance is the only common human movement performed against an external clock that will not wait.

A gymnast takes the vault when she is ready. A climber can hang there and think. A dancer cannot: the drop lands when the drop lands, and either your weight got there or it did not. The music is a deadline you did not negotiate, arriving every half-second, forever, for the length of the track. And because most dancing is improvised, the movement is being composed at the same time it is being executed, against that deadline, with no opportunity to stop and reconsider.

Then add the room. Other bodies, moving unpredictably, at conversational distance, all of them also improvising, all of them also on the clock. A social dance floor is a multi-agent, real-time, contact-adjacent negotiation that ordinary people perform drunk, in the dark, successfully.

This is the exam. Explosive, off-balance, improvised, on time, in company. A machine that can do that has, by construction, solved most of what makes moving among humans hard. A machine trained only on median motion has solved the corridor, and will meet its first real room as a surprise.

I should be fair to the obvious counter-example. You have seen robots dance: the famous one is a company video from December 2020 in which humanoids twist and shuffle to a sixties soul record, and it is a magnificent piece of engineering. But the routine was built by a human choreographer and adapted to the machines' dynamics.⁴ That is a robot performing a dance, which is a very different thing from a robot that has learned from dancing: the same difference as between a player piano and a pianist.

4. What is in the archives, and what is not

If dance is the exam, the obvious question is whether the machines already have the study material. Mostly, they do not.

The canonical benchmark for 3D human pose is a decade old and it is eleven actors performing seventeen everyday scenarios in a lab: walking, eating, sitting, talking on the phone, having a discussion. It is a superb dataset, and by design a catalogue of the median.⁵

The great unification of motion capture, which pulled fifteen separate marker-based datasets into one common body format, amounts to something over forty hours of motion from over three hundred people.⁶ Forty hours. The big video platforms take in more footage of human movement in the time it takes to read this paragraph than exists in the entire scientific record of how a human body moves.

Dance-specific capture exists and I am glad of it: the best-known set is roughly five hours across ten street-dance genres, captured in a studio, and it has driven most of the recent work on generating dance from music.⁷ Five hours is a gift, and also one dancer's decent month.

So the slice is thin, and it is thin in a specific way: what exists was captured in laboratories, in suits, with markers, by people being paid to move for a camera. What does not exist at scale is ordinary, unglamorous, ecstatic movement in ordinary rooms: the stuff people do anyway, on a Tuesday, because they like it. That is not a gap you close by buying more studio time. It is a gap you close by making something people want to use for their own reasons, that produces the data as a by-product of being useful.

5. A video keeps the pixels. A skeleton keeps the movement

This is the house line and it is the whole engineering argument in nine words.

Film a dance and you keep how it looked from one place in the room, under that light, through that lens. Track it and you keep thirty-three joint positions per frame, in metres, in three dimensions: the trajectory of how a body actually moved through space. The first is a picture of the dance. The second is the dance's shape, in numbers: small, durable, replayable from any angle, measurable, and comparable to the same phrase danced a year later.

And crucially, transferable. Putting recorded human motion onto a body with different proportions has been a solved-in-principle problem in graphics since the late nineties,⁸ so a captured phrase can drive a body with longer arms than yours without collapsing into nonsense.

The reason to care, for anything with legs: archives of recorded human motion have already been shown to turn into physical skill. A 2018 graphics result trained simulated characters to reproduce recorded clips of highly dynamic human motion (backflips, cartwheels, spin kicks) and got controllers that survive being shoved, which is not a property the clips themselves contain.⁹ A 2021 follow-up went further and used a whole archive of motion as a style prior, so a character could be told what to achieve and learn to do it while still moving like a person rather than like an optimiser.¹⁰

That last clause is the thesis in miniature. Movement data teaches a machine more than what to do. It teaches it what a body looks like when it is doing something, and everything we would call ease, or grace, or simply not being frightening to stand next to, lives in that second thing.

6. Where the app stands today

What a camera and a pose model give you is kinematics: where the joints went, and how fast. That is real, it is measurable, it is the substrate for everything above, and the app produces it reliably today for every take that goes through it.

What it does not give you yet is kinetics: forces, contacts, how hard the floor pushed back, what the ankle was carrying at the instant it took the weight. Those are measured with instrumented plates and load cells, not read off pixels. Some of it can be estimated from video, but an estimate is not a measurement, and for a machine that will one day put its weight somewhere near a real person, that gap is the whole of the safety case.

So the goal is the complete set: the kinematics we have, and the kinetics to go with them. We have a few ideas for getting there, and they are on our research roadmap. One is to check our skeletons against open datasets that already pair motion with force-plate readings, so we know exactly where our numbers hold. Another is to put pressure insoles on real dancers, so a take can carry its own record of how the floor pushed back.

Kinematics today. Kinetics next.

Sources

1. Hans Moravec, Mind Children: The Future of Robot and Human Intelligence, Harvard University Press, 1988. The observation now known as Moravec's paradox.
2. Tad McGeer, "Passive Dynamic Walking", The International Journal of Robotics Research 9(2), 1990. doi:10.1177/027836499000900206.
3. Nikolai A. Bernstein, The Co-ordination and Regulation of Movements, Pergamon Press, 1967 (English edition of work from the 1930s and 40s). The degrees-of-freedom problem and "repetition without repetition".
4. Boston Dynamics, "Do You Love Me?", published 29 December 2020. On the routine's development with choreographer Monica Thomas, see IEEE Spectrum and Dance Magazine.
5. Catalin Ionescu, Dragos Papava, Vlad Olaru, Cristian Sminchisescu, "Human3.6M: Large Scale Datasets and Predictive Methods for 3D Human Sensing in Natural Environments", IEEE TPAMI 36(7), 2014. 3.6 million 3D poses, 11 professional actors, 17 scenarios. Project page.
6. Naureen Mahmood, Nima Ghorbani, Nikolaus F. Troje, Gerard Pons-Moll, Michael J. Black, "AMASS: Archive of Motion Capture as Surface Shapes", ICCV 2019. 15 datasets unified, over 40 hours, 344 subjects. Project page.
7. Ruilong Li, Shan Yang, David A. Ross, Angjoo Kanazawa, "AI Choreographer: Music Conditioned 3D Dance Generation with AIST++", ICCV 2021. 1,408 sequences, about 5.2 hours, 10 genres, multi-camera studio. Project page.
8. Michael Gleicher, "Retargetting Motion to New Characters", SIGGRAPH 1998. Author's page.
9. Xue Bin Peng, Pieter Abbeel, Sergey Levine, Michiel van de Panne, "DeepMimic: Example-Guided Deep Reinforcement Learning of Physics-Based Character Skills", SIGGRAPH 2018. Project page.
10. Xue Bin Peng, Ze Ma, Pieter Abbeel, Sergey Levine, Angjoo Kanazawa, "AMP: Adversarial Motion Priors for Stylized Physics-Based Character Control", SIGGRAPH 2021. Project page.