Link to paper

The full paper is available here.

You can also find the paper on PapersWithCode here.

Abstract

  • Videos often contain flickering artifacts.
  • Prior work requires specific guidance to remove flicker.
  • This paper proposes a general flicker removal framework that only requires a single flickering video as input.
  • The approach uses a neural atlas and a neural filtering strategy.
  • Experiments show that the method achieves satisfying deflickering performance.

Paper Content

Introduction

  • Many videos suffer from flickering artifacts due to low-quality hardware, high-speed cameras, or video processing algorithms.
  • Removing flickering from videos is desirable in video processing and computational photography.
  • Blind deflickering is a general approach for deflickering that is agnostic to the patterns or levels of flickering.
  • Blind deflickering is challenging since it is hard to enforce temporal consistency without any extra guidance.
  • Existing techniques usually design specific strategies for each flickering type with specific knowledge.
  • We propose the first approach for blind deflickering that can remove various types of flickering with only an unprocessed video as input.
  • We construct the first dataset containing various types of flickering videos to evaluate the performance of blind deflickering methods.
  • Our method outperforms baselines on our dataset and even outperforms methods that use extra input videos on a public benchmark.
  • Different strategies are designed for specific flickering types
  • Kanj et al. propose a strategy for high-speed cameras
  • Delon et al. present a method for local contrast correction
  • Xu et al. focus on temporal flickering artifacts from GAN-based editing
  • Blind video temporal consistency is designed to remove the flicker for processed videos
  • Bonneel et al. compute the gradient of input frames as guidance
  • Lai et al. input two consecutive input frames as guidance

Method

Overview

  • Approach aims to remove flickering artifacts from video frames
  • Flicker can be global or local, long-term or short-term
  • Framework uses single neural atlas and neural filtering strategy

Flawed atlas generation

  • A good blind deflickering model should have the capacity to track correspondences across all video frames.
  • Neural atlases are introduced to the deflickering task as they fit the task.
  • A mapping network and an atlas network are trained jointly to generate an atlas.
  • The atlas is used to ensure temporal consistency in the video.

Neural filtering and refinement

  • Neural atlas can provide consistent guidance for blind deflickering
  • Neural atlas is not perfect and can’t handle rapid movement or multiple layers
  • Optical flow from flickering video is not accurate
  • Natural changes, such as shadow changes, need to be preserved
  • Neural filtering strategy uses temporal consistency of atlas-based video
  • Local refinement network used to address local flicker
  • Networks trained on MS-COCO and DAVIS datasets

Blind deflickering dataset

  • Collected 5 types of real-world flickering videos
  • Created synthetic dataset with ground truth for quantitative analysis
  • Provided quantitative comparison for processed and synthetic videos
  • Provided user study results for comparison

Experiments

Evaluation setup

  • Constructed Blind Deflickering Dataset for evaluation
  • Warping error used to measure temporal inconsistency
  • Warping error calculated using optical flow and occlusion mask

Comparisons to baselines

  • Baseline inspired by blind video temporal consistency approaches
  • Our results consistently better than baseline
  • User study shows our method outperforms baseline significantly
  • Our approach removes various types of flicker
  • Warping error of our approach lower than all baselines

Ablation study

  • Neural filtering is essential for blind deflickering
  • Removing neural filtering increases temporal inconsistency
  • Local refinement module slightly degrades quantitative performance
  • Local flickering hurts perceptual performance

Comparisons to human experts

  • Our approach is compared to human experts using commercial software for deflickering.
  • Figure 7 shows the comparison results, which show that our approach can obtain competitive results in a fully-automatic manner.

Discussion and future work

  • Our model can be applied to all types of flickering videos.
  • It can also be used for other tasks where flickering artifacts exist.
  • Solving temporal inconsistency of video content is not part of deflickering.

Conclusion

  • Problem named blind deflickering can remove diverse flickering artifacts without extra guidance
  • Core of approach is to adopt a neural atlas with a neural filtering strategy
  • Experiments show approach outperforms baselines on different datasets
  • Ablation study shows atlas and neural filtering strategy reduce temporal inconsistency significantly
  • Results comparable to human experts