LEGO: A Lifting-Free Approach for
Exocentric-to-Egocentric Video Generation

1. render

   Learned view synthesizer

   An LVSM-style transformer maps the exocentric frame and the ray maps of both cameras to the egocentric fisheye frame. It is fine-tuned for 10k steps, about 3.5 hours on four H200 GPUs, and then frozen.

2. gate

   Confidence gating

   The synthesizer's attention forms a distribution over exocentric source locations. Its top-16 mass is a per-region confidence, and regions below the threshold are set to neutral gray, so the generator completes them from its own prior.

3. guide

   Adaptive Rendering Consistency

   During the early steps, when the layout is decided, ARC moves the generator's clean estimate toward the render in proportion to the squared confidence. It needs no training and replaces a depth-derived attention bias that prior work trains into the generator.

Results

Comparison with EgoX

Both methods receive the same camera poses, text prompt, and random seed. EgoX is the authors' released checkpoint. EgoHumans and Nymeria are never used for training: LEGO runs with the weights trained on Ego-Exo4D.

Quantitative results

Comparison on Ego-Exo4D

Rows without a mark are quoted from the EgoX paper. ^†The released EgoX checkpoint rerun under our evaluation code. TF, MS, and DD are the VBench temporal flickering, motion smoothness, and dynamic degree scores.

Generalization to other datasets

Both methods use weights trained on Ego-Exo4D only, with no fine-tuning. Each dataset has 200 test clips.

Ablation

What each component contributes

All arms share one training recipe and differ only in the conditioning input and in how denoising is guided.

Confidence gating

Without a gate, the synthesizer render fills every location, including those with no exocentric evidence. The gate sets locations whose confidence falls below the threshold to neutral gray.

Limitations

LEGO inherits the limitations of its view synthesizer. The synthesizer renders each frame from a single exocentric frame, so it cannot recover content that the frame does not show, and it blurs where the correspondence is uncertain. On unfamiliar scenes its colors also fade. The pipeline needs no depth, but it still needs the camera poses of both views. Because the synthesizer is frozen and enters only through its render and confidence, a stronger feed-forward view synthesizer can replace it without changing the diffusion model or ARC.

BibTeX

@misc{cho2026lego,
  title  = {LEGO: A Lifting-Free Approach for Exocentric-to-Egocentric Video Generation},
  author = {Cho, Suhwan and Choi, Yonwoo and Kim, Soongjin and Park, Jicheol and Lim, Taegyu},
  year   = {2026}
}