Link to paper

The full paper is available here.

You can also find the paper on PapersWithCode here.

Abstract

  • Problem of degradation in inpainting quality of neural networks at high resolutions
  • Receptive field remains static when resolution increases
  • Downscaling image prior to inpainting produces coherent structure but lacks detail
  • Optimize intermediate featuremaps of a network to improve inpainting results and establish new state-of-the-art

Paper Content

Introduction

  • Image inpainting is the task of filling missing pixels or regions in an image.
  • It is used in image restoration, image editing, Augmented Reality, and Diminished Reality.
  • Existing approaches often struggle with global consistency when the masked-region is large.
  • To solve this problem, a novel coarse-to-fine iterative refinement approach is proposed.

Multiscale feature refinement

  • Our method follows a coarse-to-fine approach to add detail to an inpainting prediction.
  • We use an image-pyramid of the input RGB image and inpainting mask as network inputs.
  • We split the model into “front” and “rear” sections.
  • We use a single forward pass through the entire inpainting model to get an initial inpainting prediction.

Experiments

  • Iterative multiscale refinement was applied to Big-LaMa.
  • Downscaling factor of 2 was used to build the image pyramid.
  • Output featuremap from the downscaler portion of Big-LaMa was optimized.
  • 15 refinement iterations were performed using Adam optimizer with a learning rate of 0.002.
  • Mask was eroded with a 15 pixel circular kernel prior to applying L1 loss to the inpainted regions.

Results

  • Inpainting networks are typically benchmarked on Places2 dataset
  • Unsplash-Lite Dataset used for high resolution images
  • 1000 images randomly sampled and resized to 1024x1024
  • Masks generated with thin, medium, and thick brush strokes
  • Performance evaluated using FID scores and LPIPS
  • Outperforms state-of-the-art for medium and thick masks
  • Refinement increases inference-time and memory usage
  • Produces infills with stronger global consistency and sharper textures

Conclusion

  • Proposed a multiscale refinement technique to improve inpainting performance of neural networks on high resolution images
  • Technique outperforms other state-of-the-art approaches at high resolution inpainting
  • Performance comparison against recent inpainting approaches on 1k 1024x1024 size images