Shadow self-modeling · Duke University, 2026
Twenty-one joints, a skin of foamed plastic, and a model of what the wall sees. Not a hand that grasps — a hand that performs.
People communicate constantly through things that are not their bodies. A drawing, a gesture traced in the air, a silhouette behind a paper screen, a shadow puppet thrown on a tent wall by firelight. The authors give this whole family a name: projected visual abstractions — visual forms that come out of a physical body but are not the body itself.
Robots, by contrast, have been stuck expressing themselves through their own hardware. They can paint, dance, walk with attitude, and pull faces, but the audience is always looking directly at the machine. This paper asks a different question: can a robot deliberately shape something that is caused by its body, projected somewhere else, and then read by an observer who never looks at the robot at all?
Shadows are the ideal test case, and not just because hand-shadow puppetry is charming. A shadow is a lossy transformation. Three dimensions collapse into two; texture, color, and depth are thrown away. To aim a shadow at a meaning, the robot cannot simply reason about where its fingers are. It has to reason about what the projection does to its fingers.
The robot stops asking “what is my body doing?” and starts asking “what will someone else see if I do this?”
That shift is the intellectual core of the work. Everything else — the foam skin, the two-stage optimizer, the keyframe clustering — exists to make it physically real.
Before any learning happens, there is a hardware problem that no algorithm can fix. Conventional robot hands are assemblies of rigid links with visible gaps at every joint and between every finger. Point a spotlight at one and the shadow arrives full of holes — light leaks straight through the seams, and the silhouette fragments into disconnected slivers. Human shadow artists never have this problem, because soft tissue seals the gaps: fingers press together and the light simply stops.
So the authors built a hybrid. Inside, a rigid printed skeleton: five identical fingers, each with the three joints of a real finger — the DIP at the fingertip, the PIP in the middle, and the MCP at the knuckle, which gets two degrees of freedom so it can both curl and splay sideways. With a wrist joint at the base, that is 21 degrees of freedom — 21 independently controllable angles, driven by 21 serial-bus servos.
Outside, a compliant skin printed from foaming TPU, a thermoplastic that expands as it prints into a lightweight foam. Printed at low infill with a gyroid lattice and no outer walls, it is floppy enough to add almost no resistance to the skeleton underneath, opaque enough to stop light, and squishy enough that adjacent fingers deform against each other and seal the seam. Rigid where precision is needed, soft where light is concerned.
You can see what that buys them below. The bench is a working toy version of the paper's setup: a spotlight, a hand, and a white backdrop a meter behind it.
The obvious approach is a direct one: train a network that takes a picture of the shadow you want and outputs the 21 joint angles that produce it. This is called an inverse model — from effect back to cause. It fails, and it fails for a reason worth understanding.
The mapping from pose to shadow is many-to-one. Enormous numbers of physically distinct hand configurations cast nearly identical silhouettes — you just watched a version of this when the distance slider grew the shadow without changing the pose. Training a network to invert a many-to-one function is ill-posed: for a single input there are many equally correct outputs, and a network trained on squared error will average them into a pose that is correct nowhere. In the paper's numbers, the inverse baseline barely beats picking a random pose out of the training set.
So they invert the direction of the model. Train a forward map instead — joint angles in, predicted shadow image out — where the answer is unique and well-defined. This is a self-model in the tradition of robots that learn their own morphology from scratch, with one twist: it models not the body but the body's consequences.
The training data costs nothing but simulator time. The hand flails — the authors call it motor babbling, borrowed from infant development — sampling random joint configurations and recording the shadow each one produces. Three quarters of the samples are fully random; the rest deliberately move one or two fingers at a time so the model learns isolated finger motion rather than only tangled full-hand poses. That yields 9.25 million pose–shadow pairs, none of them human demonstrations, none of them scraped video. The robot is its own dataset.
The model itself is deliberately half-analytic. Rather than making a network learn hand geometry from nothing, the first stage is forward kinematics — the exact trigonometry that turns 21 angles into the 3D position and orientation of every joint, written out as transformation matrices. Only then does a neural decoder take over and paint the 256×256 silhouette. Because the kinematics are written in differentiable operations, gradients flow cleanly back through the geometry. Handing the network this structural prior instead of making it rediscover a hand improved performance by 31% on the sign-language targets.
Here is the payoff. Once the forward model is trained, its weights are frozen and it is used backwards — not to invert anything, but as a differentiable stand-in for reality. Feed in a batch of candidate poses, render their predicted shadows, compare each to the target image, then backpropagate the error into the joint angles themselves and nudge them downhill. Gradient descent, but the thing being optimized is a hand pose rather than a set of weights.
The comparison uses three measures at once, because no single one is enough. MAE (mean absolute error) counts pixel-by-pixel disagreement. IoU (intersection over union) measures how much the two silhouettes actually overlap relative to their combined area — a shape-level score that ignores small misalignments. And CLIP similarity compares the two images inside the embedding space of a vision-language model trained on hundreds of millions of captioned images, which gives a rough proxy for whether a human would call them the same thing. Pixels, shape, and semantics, pulling together.
Optimizing through the self-model alone produces poses that look right and cannot be performed. The model learned appearance, not anatomy: nothing in it knows that two fingers cannot occupy the same space. Across the 61 targets, 39% of self-model solutions were self-colliding. The fix is a second stage in a physics simulator — and the order matters.
Gradient descent through the frozen shadow self-model finds a pose that matches the target silhouette. Fast, differentiable, and completely unaware of physics.
That pose becomes the warm start for a hill-climbing search inside the simulator, with collisions switched on: try a small random perturbation, keep it only if the shadow improves and nothing intersects.
A spline-based planner connects the accepted poses into a smooth, self-collision-free trajectory, which the physical hand executes directly.
Hill climbing alone — random local search with no gradient — is both slower and worse, because a 21-dimensional space is far too large to grope through blindly; it recovers coarse blobs but loses eyes, beaks, and thin structures. Gradients alone are fast but physically naive. Used in sequence, the gradient finds the neighbourhood and the simulator makes it executable, cutting total loss by a third over the self-model result.
Total loss over all 61 image targets, lower is better. The nearest neighbour row is the honesty check: it retrieves the best-matching shadow already present in the 9-million-pair training set. Beating it shows the system is composing new poses, not recalling old ones.
A still silhouette is a puzzle. A performance is three more, and each gets its own idea.
Expressiveness concentrates in tiny places: an eye, the gap of a beak, the trailing edge of a wing. A loss function that treats every pixel equally will happily sacrifice a duck's eye to gain a slightly better outline, and the result reads as a blob. The authors define an expressive region mask from two cues — the parts of the silhouette that move between consecutive frames, and the enclosed pockets of light fully surrounded by shadow (found exactly the way the bench above finds them: any background region that cannot reach the image border). Extra IoU and CLIP penalties apply inside that mask. Global accuracy dips slightly; perceived expressiveness climbs sharply — expressive-region error drops by more than half.
The many-to-one ambiguity comes back with a vengeance in video. Optimize each frame independently and consecutive frames may pick different solutions to the same shape — the index finger draws the raven's wingtip in one frame, the ring finger draws it in the next. Both frames look correct in isolation. Between them the hand must perform a violent reshuffle, throwing garbage shadows across the wall.
Two fixes work together. A temporal regularizer penalizes distance from the previous frame's joint angles. But that alone barely helps, because each frame still starts its search from a random place and lands in a different basin. The decisive move is inherited initialization: optimize the first frame with a generous budget, then start every subsequent frame from the previous frame's solution plus a whisper of noise, so the search is confined from the outset to the same region of joint space. Transition success rises from 34.5% to 86.2%. Try the without inherited init toggle on the bench to see the failure mode play out.
A 60-frame clip of a peacock does not contain 60 distinct shapes. The authors compress each frame's expressive-region mask with PCA — principal component analysis, which finds the handful of dimensions along which the frames actually differ — then merge the nearest clusters repeatedly until the remaining groups are meaningfully distinct. One representative frame per cluster becomes an optimization target; every other frame simply inherits its cluster's pose. Optimization cost falls by two thirds to nine tenths, and the motion still reads as the same animal.
Each square is a video frame; clay squares are the keyframes actually optimized.
The evaluation is structured as a widening gap between the target and the robot's body. Twenty-six American Sign Language gestures come first: shapes a hand can obviously make, testing only whether the system can solve the shadow puzzle. Nineteen keyframes of human hand-shadow puppetry follow — duck, deer, peacock, camel — where the silhouettes were made by human fingers that are thinner and more delicate than the robot's. Then sixteen keyframes from raw animal video: a howling wolf, a raven beating its wings. Nothing about a wolf's jaw corresponds to a finger. The targets arrive as bare binary silhouettes, so there is no semantic correspondence to lean on — no instruction that the thumb should play the lower beak.
Across all three tiers the system reproduces global structure and the small details that carry the read, and the trajectories transfer from simulation to the physical hand.
The engineering here is elegant, but the reframing is what lingers. A self-model that predicts the body's appearance to others is a small, concrete, entirely non-mystical version of perspective-taking. The authors are careful about this: nothing in the system is doing cognitive perspective-taking, and they say so plainly. What they have built is a computational mechanism for the same shape of question — a robot optimizing its configuration against a prediction of someone else's view.
There is a second lesson in the hardware. A hand optimized for gripping is not the same hand as one optimized for being seen. Rigid links are excellent for manipulation and terrible for silhouettes. If robots are going to communicate in human spaces, some part of their design will need to answer to perception rather than to force.
The limits are honest and clearly stated. The hand can only cast shadows a hand can cast — targets with disconnected parts or extreme proportions remain out of reach, and a second hand or a prop would help, exactly as it does for human puppeteers. The rig assumes one fixed light and one flat backdrop, where the real world offers moving light and irregular walls. And the optimization runs offline, so the robot cannot yet improvise in front of an audience.
Shadows are only the nearest example of the larger class. Reflections, projections, avatars, holograms — anything physically separate from a robot but causally welded to it — could be steered the same way. The body stops being only a means of acting on the world and becomes a generator of external symbols. Which is, more or less, what a hand was doing on a cave wall by firelight.