Learning Models of People
from Virtual Worlds

Massachusetts Institute of Technology

Where people stop is a stable property of the place; where one person goes next is not, until they start to move. We model people as agents with goals, and the crowd's social structure survives in closed loop.

Abstract

Intelligent agents that share environments with people need models not only of the world, but of the people within it. Human behavior entangles three fundamentally different sources of variation: structure imposed by the environment, dynamics that persist across an individual, and intent that changes from one episode to the next. We learn models of people from shared virtual worlds that separate these sources using repeated observations of people acting across sessions and environments. We find that they generalize in strikingly different ways. Semantic scene representations transfer population-level patterns of where people go across worlds substantially better than geometry alone; individual movement style, including pace and path directness, persists across worlds even when destination preferences do not; and uncertainty within an episode is captured by a belief over latent goals that sharpens as a person acts. Richer observations of the person, including pose, head direction, identity, and social context, add little once this structure is represented. Our results suggest that generalizable models of people should separate where behavior is shaped by the world, how behavior reflects the individual, and what must be inferred about a person’s current intent.

Where people stop is a property of the place

Every evening, people walk into a bar scene and nobody gives them a task. Replayed from telemetry inside the building's own geometry, their paths carve the same corridors. We split the crowd into two halves that share no session and no person, and count where each half's walks end on 1 m cells. Both halves draw the same map (ρ = 0.95 in the bar scene, 0.76 in a hangout scene).

Twenty minutes of real arrivals, scrubbed through time.

Where walks end, rising as columns and morphing between the two halves.

Intent is the bottleneck

The same map is a poor guide to any single walk. Its best guess lands within 3 m of where a walk stops only 5% of the time in the bar scene, and even twenty guesses miss 48% of walks. Watching the walk changes this. A Bayesian goal filter over 512 candidate cells, driven by the walker's heading and the shortest route to each cell, sharpens with every step: after eight seconds the top five candidates include the true goal 40% of the time, against 9% for popularity alone.

One real walk. Columns show the belief over candidate goals; the ring marks where the person actually stopped.

A crowd of agents with goals

We simulate each person as an agent that runs the same loop once a second: decide whether to stay or go, choose a goal that is either a person or a place, route there, and walk. Only who is present comes from the recording; every decision is the model's own.

0

Who is here

Arrivals and departures from the room

→
1

Stay or go?

A departure hazard for everyone standing still

→
2

Who or where?

Head for a person, or for a place

→
3

Route

The shortest walkable path there

→
4

Walk

Step by step, toward the next waypoint

The simulated crowd in the bar scene, rolled out in closed loop.

Zero-shot transfer

Fitted only on the bar scene, the simulator runs unchanged in a pub scene it has never seen. The share of people standing alone is reproduced (error 0.07, against 0.60 for a null model), but the simulated crowd is more restless than the real one and drifts to the upper levels.

Real crowd

Simulated, zero-shot

Results

All splits are person-disjoint. Shaded bands are 95% bootstrap intervals over sessions; hover a chart for exact values.

The individual walk is hard to call at departure

Nearest popular spotMost popular spotsChance

Social structure survives closed-loop rollout, and transfers

Goal simulatorNo social inputsNext-step transformerNull

Watching the walk reveals the goal

Most of the time, people stay put

Explore

Step inside the four recorded walk-ins in 3D: the restored bar as a colour point cloud, everyone in the room as SMPL-X bodies fitted to their recorded skeletons, and the camera wearer's head camera (open full page). Or read the interactive scroll story built from the same data.

BibTeX

@misc{lewis2026learningmodels,
  author    = {Lewis, Sean Hardesty},
  title     = {Learning Models of People from Virtual Worlds},
  year      = {2026},
}