Link to paper
The full paper is available here.
You can also find the paper on PapersWithCode here.
Abstract
- Problem of degradation in inpainting quality of neural networks at high resolutions
- Receptive field remains static when resolution increases
- Downscaling image prior to inpainting produces coherent structure but lacks detail
- Optimize intermediate featuremaps of a network to improve inpainting results and establish new state-of-the-art
Paper Content
Introduction
- Image inpainting is the task of filling missing pixels or regions in an image.
- It is used in image restoration, image editing, Augmented Reality, and Diminished Reality.
- Existing approaches often struggle with global consistency when the masked-region is large.
- To solve this problem, a novel coarse-to-fine iterative refinement approach is proposed.
Multiscale feature refinement
- Our method follows a coarse-to-fine approach to add detail to an inpainting prediction.
- We use an image-pyramid of the input RGB image and inpainting mask as network inputs.
- We split the model into “front” and “rear” sections.
- We use a single forward pass through the entire inpainting model to get an initial inpainting prediction.
Experiments
- Iterative multiscale refinement was applied to Big-LaMa.
- Downscaling factor of 2 was used to build the image pyramid.
- Output featuremap from the downscaler portion of Big-LaMa was optimized.
- 15 refinement iterations were performed using Adam optimizer with a learning rate of 0.002.
- Mask was eroded with a 15 pixel circular kernel prior to applying L1 loss to the inpainted regions.
Results
- Inpainting networks are typically benchmarked on Places2 dataset
- Unsplash-Lite Dataset used for high resolution images
- 1000 images randomly sampled and resized to 1024x1024
- Masks generated with thin, medium, and thick brush strokes
- Performance evaluated using FID scores and LPIPS
- Outperforms state-of-the-art for medium and thick masks
- Refinement increases inference-time and memory usage
- Produces infills with stronger global consistency and sharper textures
Conclusion
- Proposed a multiscale refinement technique to improve inpainting performance of neural networks on high resolution images
- Technique outperforms other state-of-the-art approaches at high resolution inpainting
- Performance comparison against recent inpainting approaches on 1k 1024x1024 size images