Link to paper
The full paper is available here.
You can also find the paper on PapersWithCode here.
Abstract
- Videos often contain flickering artifacts.
- Prior work requires specific guidance to remove flicker.
- This paper proposes a general flicker removal framework that only requires a single flickering video as input.
- The approach uses a neural atlas and a neural filtering strategy.
- Experiments show that the method achieves satisfying deflickering performance.
Paper Content
Introduction
- Many videos suffer from flickering artifacts due to low-quality hardware, high-speed cameras, or video processing algorithms.
- Removing flickering from videos is desirable in video processing and computational photography.
- Blind deflickering is a general approach for deflickering that is agnostic to the patterns or levels of flickering.
- Blind deflickering is challenging since it is hard to enforce temporal consistency without any extra guidance.
- Existing techniques usually design specific strategies for each flickering type with specific knowledge.
- We propose the first approach for blind deflickering that can remove various types of flickering with only an unprocessed video as input.
- We construct the first dataset containing various types of flickering videos to evaluate the performance of blind deflickering methods.
- Our method outperforms baselines on our dataset and even outperforms methods that use extra input videos on a public benchmark.
Related work
- Different strategies are designed for specific flickering types
- Kanj et al. propose a strategy for high-speed cameras
- Delon et al. present a method for local contrast correction
- Xu et al. focus on temporal flickering artifacts from GAN-based editing
- Blind video temporal consistency is designed to remove the flicker for processed videos
- Bonneel et al. compute the gradient of input frames as guidance
- Lai et al. input two consecutive input frames as guidance
Method
Overview
- Approach aims to remove flickering artifacts from video frames
- Flicker can be global or local, long-term or short-term
- Framework uses single neural atlas and neural filtering strategy
Flawed atlas generation
- A good blind deflickering model should have the capacity to track correspondences across all video frames.
- Neural atlases are introduced to the deflickering task as they fit the task.
- A mapping network and an atlas network are trained jointly to generate an atlas.
- The atlas is used to ensure temporal consistency in the video.
Neural filtering and refinement
- Neural atlas can provide consistent guidance for blind deflickering
- Neural atlas is not perfect and can’t handle rapid movement or multiple layers
- Optical flow from flickering video is not accurate
- Natural changes, such as shadow changes, need to be preserved
- Neural filtering strategy uses temporal consistency of atlas-based video
- Local refinement network used to address local flicker
- Networks trained on MS-COCO and DAVIS datasets
Blind deflickering dataset
- Collected 5 types of real-world flickering videos
- Created synthetic dataset with ground truth for quantitative analysis
- Provided quantitative comparison for processed and synthetic videos
- Provided user study results for comparison
Experiments
Evaluation setup
- Constructed Blind Deflickering Dataset for evaluation
- Warping error used to measure temporal inconsistency
- Warping error calculated using optical flow and occlusion mask
Comparisons to baselines
- Baseline inspired by blind video temporal consistency approaches
- Our results consistently better than baseline
- User study shows our method outperforms baseline significantly
- Our approach removes various types of flicker
- Warping error of our approach lower than all baselines
Ablation study
- Neural filtering is essential for blind deflickering
- Removing neural filtering increases temporal inconsistency
- Local refinement module slightly degrades quantitative performance
- Local flickering hurts perceptual performance
Comparisons to human experts
- Our approach is compared to human experts using commercial software for deflickering.
- Figure 7 shows the comparison results, which show that our approach can obtain competitive results in a fully-automatic manner.
Discussion and future work
- Our model can be applied to all types of flickering videos.
- It can also be used for other tasks where flickering artifacts exist.
- Solving temporal inconsistency of video content is not part of deflickering.
Conclusion
- Problem named blind deflickering can remove diverse flickering artifacts without extra guidance
- Core of approach is to adopt a neural atlas with a neural filtering strategy
- Experiments show approach outperforms baselines on different datasets
- Ablation study shows atlas and neural filtering strategy reduce temporal inconsistency significantly
- Results comparable to human experts