Self-driving companies increasingly test their software in generated worlds rather than only on real roads. For that to work, the simulated cars around the test vehicle have to behave believably, and the simulation has to follow the route you tell it to drive. BirdDreams, a preprint whose authors include researchers who did the work at NVIDIA, tackles both problems by having the model imagine a simple overhead map of the scene alongside the camera view.
The idea is straightforward. Learning how other vehicles move from a dashboard camera alone is hard, because motion is tangled up with appearance and cars hide behind each other. Earlier driving world models got around this by feeding in the future paths of surrounding vehicles, which gives control but means someone has to supply those paths. BirdDreams instead generates two videos at once: the normal perspective view and a schematic bird's-eye view that shows roads and vehicles as plain shapes at a fixed scale. Two video diffusion models, one for each stream, share attention so each informs the other. The overhead view is easier to reason about; the camera view keeps details the map drops, such as traffic lights.
Control comes through the map. To make the simulated car follow a given route, the system shifts and rotates the latest overhead frame along the requested path and uses the result as a soft hint of how the scene should appear to move. At inference it needs only a single camera image, with no high-definition map or vehicle-state inputs. Training used about 670 hours of driving from the RDS-HQ-1M dataset plus the nuScenes training split of 700 scenes, roughly 3.9 hours.
On the PhysicalAI-AV and nuScenes benchmarks, the authors report that BirdDreams follows input routes more faithfully than earlier world models such as Vista, GEM and DrivingDojo. In one comparison its minimum average displacement error was 6.15 against 7.76 for the closest rival, a version of OmniDreams trained on the same data. Human raters judged its surrounding-vehicle behaviour more realistic than a camera-only baseline.
This is research for simulation teams, not a product. The overhead view is rendered from human annotations during training, so the approach depends on well-labelled data. Results come from the authors' own evaluation, the human study details are limited in the abstract, and the work has not been peer reviewed. Whether more plausible generated traffic translates into safer real-world driving software is a separate question the paper does not test.