Link to paper
The full paper is available here.
You can also find the paper on PapersWithCode here.
Abstract
- Diffusion models (DMs) have potential for generative tasks.
- Watermarking is a solution for copyright protection and content monitoring in DMs.
- A recipe for efficiently watermarking state-of-the-art DMs is provided.
Paper Content
Introduction
- DMs have demonstrated impressive performance on generative tasks
- DMs have advantages over other generative models
- Growing interest in controllable generation has led to the creation of large-scale DMs
- Legal issues arise with the use of DMs, such as copyright protection and detecting generated content
- Watermarks have been used to protect copyright and detect fake content
- This paper develops two watermarking pipelines for DMs
- Ablation studies are conducted to investigate the possibility of watermarking DMs
Related work
- Diffusion models (DMs) are generative learning approaches used in image generation.
- Watermarking technology has been used to protect or identify multimedia contents.
- Watermarking techniques have been proposed for deep neural networks.
- Watermarking generative models is more challenging than discriminative models.
- GANs have been watermarked by embedding binary strings within training images.
Preliminary
- DMs involve a forward process to move data distribution towards a noisy distribution
- The transition probability is a conditional Gaussian distribution
- There is a reverse process with the same marginal distributions as the forward process
- The data score is approximated by a time-dependent DM
- The training objective is to minimize the difference between the data and noise
- During the inference phase, the trained DMs are sampled via stochastic or deterministic solvers
- Samples generated from the DM follow the sampling distribution induced from the DM
Watermarking diffusion models
- DMs have attracted broad interest and have been successful, but legal issues such as copyright protection arise.
- Watermarking has been demonstrated to be an effective solution for similar legal issues, but is underexplored in the DMs literature.
- A recipe is proposed for efficiently watermarking state-of-the-art DMs, taking into account their unique characteristics.
Unconditional/class-conditional generation
- Watermark can be accurately recovered at the cost of degraded generative performance
- Increasing bit length of watermark string leads to larger distribution shift
- Significant increase in bit accuracy occurs at last few steps of sampling process
- Watermarking is robust against perturbations on model weights
Text-to-image generation
- Watermark image can be accurately triggered when λ is small, but generative performance of text-to-image DMs is degraded
- As λ increases, generative performance remains unchanged, but watermark image cannot be accurately triggered
- Need to find a moderate λ for a good trade-off between performance and watermark image generation
- Trigger prompt should be a rare identifier to minimize negative impact on text-to-image DMs
- Rare identifier in a complete sentence does not bring strong connections between trigger prompt and watermark image
Empirical studies
- Conducted large-scale experiments on image generation tasks
- Proposed watermarking pipelines can embed predefined watermark into generated contents and text-to-image DMs
- Discussed design choices and ablation studies of watermarking in greater detail
Detect watermarks from generated contents
- Chosen architectures of watermark encoder and decoder in accordance with prior work
- Bit length of binary string selected as 4, 8, 16, 32, 64
- Complex design of 128 bits degrades quality of generated samples
- Increasing image resolution mitigates performance degradation
- Adam optimizer used with initial learning rate of 0.001
- Models trained on 8 NVIDIA A100 GPUs
- Transferability property assumed and watermark successfully recovered from GAN-generated images
- Performance degradation observed as length and complexity of watermark string increases
- Quality of generated images improves with higher resolution images
Detect watermarks from text-to-image dms
- Use Stable Diffusion as text-to-image DM
- Finetune on 4 NVIDIA A100 GPUs
- Image resolution 512ˆ512
- Trigger prompt “[V]”
- Use Stable Diffusion to detect predefined image-text pair
- Weights-constrained regularization to embed watermark
- Visualize change of weights compared to pretrained weights
- Performance degradation if no regularization used
Extended experiments and analyses
- Conducted extended experiments
- Studied subtleties of watermarked DMs
- Two generation paradigms
Limitations
- Recipe for watermarking different types of DMs is simple and effective
- Injecting a watermark string into all training images results in a distribution shift
- Trade off between generation accuracy of watermark image and generative performance
- Different types of watermark information can be embedded in DMs
Conclusion and discussion
- Conducted empirical study on watermarking of unconditional/class-conditional and text-to-image DMs
- Simple and efficient watermarking pipelines
- One of the first attempts to watermark large-scale DMs
- Findings and experiments pave the way for copyright/ownership information to be added to large-scale DMs
- Positive impact on finetuning of large-scale DMs with few-shot data
Overview of appendix
- Investigated watermarking diffusion models in two major types: unconditional/class-conditional generation and text-to-image generation
- Provided additional implementation details, experiments, and analysis to support proposed methods in main paper
A. additional implementation details
- Watermark encoder and decoder discussed
- Network architecture and objective for optimization during training of watermark encoder and decoder
- Different datasets used for training (CIFAR-10, FFHQ, AFHQv2, ImageNet)
- Inference stage uses same watermark for entire training set
- Watermark information mainly resides at fine-grained levels
- Weights-constrained fine-tuning method proposed
- Rare identifier used as trigger prompt
- Watermark image can be accurately generated given trigger prompt
- Quality of generated images degraded with increased bit length
- Bit accuracy of generated images remains stable with increased bit length
- Robustness of watermarked generated images evaluated by adding Gaussian noise to weights of models
- FID score explodes with increased strength of Gaussian noise added directly to generated images
- Bit accuracy remains stable
- Watermarked text-to-image model can accurately generate predefined watermark image
- Common text as trigger prompt leads to overfitting of watermark image
- Watermark knowledge perturbed after fine-tuning
- Negative impact on resulting watermarked diffusion models
- Potential social and ethical issues if used by malicious users