Elevating Flow-Guided Video Inpainting
with Reference Generation

Suhwan Cho1,* Seoung Wug Oh2 Sangyoun Lee1 Joon-Young Lee2

1Yonsei University2Adobe Research

*Work done during an internship at Adobe Research

AAAI 2025

TL;DRRGVI propagates known pixels across a video with one-shot pixel pulling and generates the rest in a single key frame with a large generative model, then spreads that reference to every frame.

Video object removal on DAVIS 2016. Each clip shows the input with its mask, FuseFormer, ProPainter, and RGVI.

Abstract

Video inpainting (VI) is a challenging task that requires effective propagation of observable content across frames while simultaneously generating new content not present in the original video. In this study, we propose a robust and practical VI framework that leverages a large generative model for reference generation in combination with an advanced pixel propagation algorithm. Powered by a strong generative model, our method not only significantly enhances frame-level quality for object removal but also synthesizes new content in the missing areas based on user-provided text prompts. For pixel propagation, we introduce a one-shot pixel pulling method that effectively avoids error accumulation from repeated sampling while maintaining sub-pixel precision. To evaluate various VI methods in realistic scenarios, we also propose a high-quality VI benchmark, HQVI, comprising carefully generated videos using alpha matte composition. On public benchmarks and the HQVI dataset, our method demonstrates significantly higher visual quality and metric scores compared to existing solutions. Furthermore, it can process high-resolution videos exceeding 2K resolution with ease, underscoring its superiority for real-world applications.

Method

How RGVI works

Coupled approaches train one network to both propagate and generate, which leaves an ambiguity between the two. RGVI decouples them. Content that is visible somewhere in the video is propagated with optical flow, and content that appears nowhere in the video comes from a large generative model.

RGVI pipeline: flow completion, internal propagation, reference generation, reference propagation, and per-frame completion
Overall pipeline of RGVI.
  1. step 1

    Internal pixel propagation

    Known pixels are propagated across all frames with completed optical flows. Colors are pulled from the source frames in both directions, and pixels on which the two directions disagree are marked as unreliable.

  2. step 2

    Reference generation

    A key frame is chosen by how many unknown pixels it connects to in other frames. Stable Diffusion fills its missing area, either as plausible background or as new content from a text prompt.

  3. step 3

    Reference propagation

    The generated pixels in the key frame are propagated to the rest of the frames with the pre-computed flows.

  4. step 4

    Per-frame completion

    A lightweight encoder-decoder network fills the areas that are still missing, including the pixels marked as unreliable.

One-shot pixel pulling

Recurrent pixel warping re-samples colors at every step, so details fade over long propagation. RGVI warps the optical flows instead and pulls each color from its source frame only once, which keeps sub-pixel accuracy without re-sampling artifacts.

Target frame, recurrent pixel warping result, and one-shot pixel pulling result with enlarged crops
Visual comparison between recurrent pixel warping and one-shot pixel pulling, where the source frame image is propagated to the target frame.
Internal propagationReference propagationPSNR ↑SSIM ↑LPIPS ↓VFID ↓
NoneNone27.640.92340.09620.5916
Recurrent warpingNone31.430.95120.05950.2405
One-shot pullingNone31.600.95590.03900.1868
Recurrent warpingRecurrent warping30.170.94780.05580.2153
One-shot pullingOne-shot pulling30.660.95270.03350.1825

Pixel propagation strategies on HQVI at 240p.

Generation mode

In the default removal mode, the text prompt is “Empty background, high resolution”, so the model fills the missing area with content that blends into the scene. In the generation mode, a user prompt decides what appears in the missing area.

Input frames with the bear masked, and outputs generated with the prompts Standing Minions and Sleeping cat
RGVI outputs from the generation mode.

Benchmark

HQVI dataset

HQVI blends foreground objects from VideoMatte240K into real-world videos from Pexels with alpha mattes, so every video comes with a ground truth for object removal. It includes scenarios that need generation for large missing areas, and targets occluded by distracting objects, annotated with a negative mask for the object to remove and a positive mask for the object to keep. Each video is 1200×2160.

Skydivers over a city

Input with masks
RGVI
Ground truth

Workers in front of a graffiti wall

Input with masks
RGVI
Ground truth

Example videos from the HQVI dataset at 480p. Negative masks, the objects to remove, are highlighted in green, while positive masks, the occluding objects to keep, are highlighted in red.

Download HQVI

Results

Video object removal on HQVI

MethodPublicationPSNR ↑SSIM ↑LPIPS ↓VFID ↓Mem.Time
240×432 input
STTN†ECCV'2029.640.93390.05280.25945.2G9s
FGVCECCV'2028.370.93830.04090.24361.3G2m 56s
FuseFormer†ICCV'2129.920.93650.04980.272721.5G17s
E2FGVI†CVPR'2230.630.94270.04010.18857.2G12s
ProPainterICCV'2330.620.94130.03880.21288.1G15s
RGVI w/o Ref.31.600.95590.03900.18688.3G55s
RGVI30.660.95270.03350.18258.3G58s
480×864 input
FGVCECCV'2028.630.94330.03880.04702.4G15m 7s
ProPainterICCV'2330.690.94140.04570.047823.7G1m 12s
RGVI w/o Ref.31.190.95340.04030.04048.3G1m 38s
RGVI30.900.95130.03420.03118.3G1m 41s
1200×2160 input
RGVI w/o Ref.29.810.95010.04030.010117.2G7m 56s
RGVI30.100.94890.03570.005817.2G7m 59s

†Requires a fixed input resolution. Mem. is the maximum GPU memory during inference, and Time is the average inference time per sequence, on a single TITAN RTX GPU.

Video restoration

DAVI and YTVI corrupt the DAVIS 2016 and YouTube-VOS 2018 videos with random free-form masks, and the original videos serve as ground truth. Reference generation is skipped here, since restoration rarely needs it.

DAVIYTVI
MethodPSNR ↑SSIM ↑PSNR ↑SSIM ↑
STTN26.980.847629.960.9102
FuseFormer27.410.858329.180.9001
E2FGVI28.710.888531.180.9280
ProPainter29.550.905831.700.9312
RGVI w/o Ref.29.750.918631.700.9335

240p input resolution.

Analysis

Handling occlusion with an additional mask

Flow-based methods break down when the target is occluded by another object, because its motion corrupts the completed flows. Besides the mask of the object to remove, RGVI can take the mask of the occluding object to keep. The two are removed together, and the kept object is pasted back afterwards, which prevents color bleeding.

Input with negative and positive masks, and results without and with the additional mask
Negative masks are highlighted in green, while positive masks are highlighted in red.
Additional maskPSNR ↑SSIM ↑LPIPS ↓VFID ↓
Without35.130.98130.01370.0779
With37.310.98480.01020.0504

HQVI videos that contain occluding objects, at 240p.

Limitations

Like other flow-based methods, RGVI relies on optical flows, so inaccurate flow completion can cause visible structural misalignment. The generated reference content may also look unnatural at times. Additional positive masks and stronger generative models are possible remedies.

BibTeX

@inproceedings{cho2025elevating,
  title     = {Elevating Flow-Guided Video Inpainting with Reference Generation},
  author    = {Cho, Suhwan and Oh, Seoung Wug and Lee, Sangyoun and Lee, Joon-Young},
  booktitle = {Proceedings of the AAAI Conference on Artificial Intelligence},
  volume    = {39},
  number    = {3},
  pages     = {2527--2535},
  year      = {2025},
  doi       = {10.1609/aaai.v39i3.32255}
}