Link to paper

The full paper is available here.

You can also find the paper on PapersWithCode here.

Abstract

  • Diffusion models (DMs) have potential for generative tasks.
  • Watermarking is a solution for copyright protection and content monitoring in DMs.
  • A recipe for efficiently watermarking state-of-the-art DMs is provided.

Paper Content

Introduction

  • DMs have demonstrated impressive performance on generative tasks
  • DMs have advantages over other generative models
  • Growing interest in controllable generation has led to the creation of large-scale DMs
  • Legal issues arise with the use of DMs, such as copyright protection and detecting generated content
  • Watermarks have been used to protect copyright and detect fake content
  • This paper develops two watermarking pipelines for DMs
  • Ablation studies are conducted to investigate the possibility of watermarking DMs
  • Diffusion models (DMs) are generative learning approaches used in image generation.
  • Watermarking technology has been used to protect or identify multimedia contents.
  • Watermarking techniques have been proposed for deep neural networks.
  • Watermarking generative models is more challenging than discriminative models.
  • GANs have been watermarked by embedding binary strings within training images.

Preliminary

  • DMs involve a forward process to move data distribution towards a noisy distribution
  • The transition probability is a conditional Gaussian distribution
  • There is a reverse process with the same marginal distributions as the forward process
  • The data score is approximated by a time-dependent DM
  • The training objective is to minimize the difference between the data and noise
  • During the inference phase, the trained DMs are sampled via stochastic or deterministic solvers
  • Samples generated from the DM follow the sampling distribution induced from the DM

Watermarking diffusion models

  • DMs have attracted broad interest and have been successful, but legal issues such as copyright protection arise.
  • Watermarking has been demonstrated to be an effective solution for similar legal issues, but is underexplored in the DMs literature.
  • A recipe is proposed for efficiently watermarking state-of-the-art DMs, taking into account their unique characteristics.

Unconditional/class-conditional generation

  • Watermark can be accurately recovered at the cost of degraded generative performance
  • Increasing bit length of watermark string leads to larger distribution shift
  • Significant increase in bit accuracy occurs at last few steps of sampling process
  • Watermarking is robust against perturbations on model weights

Text-to-image generation

  • Watermark image can be accurately triggered when λ is small, but generative performance of text-to-image DMs is degraded
  • As λ increases, generative performance remains unchanged, but watermark image cannot be accurately triggered
  • Need to find a moderate λ for a good trade-off between performance and watermark image generation
  • Trigger prompt should be a rare identifier to minimize negative impact on text-to-image DMs
  • Rare identifier in a complete sentence does not bring strong connections between trigger prompt and watermark image

Empirical studies

  • Conducted large-scale experiments on image generation tasks
  • Proposed watermarking pipelines can embed predefined watermark into generated contents and text-to-image DMs
  • Discussed design choices and ablation studies of watermarking in greater detail

Detect watermarks from generated contents

  • Chosen architectures of watermark encoder and decoder in accordance with prior work
  • Bit length of binary string selected as 4, 8, 16, 32, 64
  • Complex design of 128 bits degrades quality of generated samples
  • Increasing image resolution mitigates performance degradation
  • Adam optimizer used with initial learning rate of 0.001
  • Models trained on 8 NVIDIA A100 GPUs
  • Transferability property assumed and watermark successfully recovered from GAN-generated images
  • Performance degradation observed as length and complexity of watermark string increases
  • Quality of generated images improves with higher resolution images

Detect watermarks from text-to-image dms

  • Use Stable Diffusion as text-to-image DM
  • Finetune on 4 NVIDIA A100 GPUs
  • Image resolution 512ˆ512
  • Trigger prompt “[V]”
  • Use Stable Diffusion to detect predefined image-text pair
  • Weights-constrained regularization to embed watermark
  • Visualize change of weights compared to pretrained weights
  • Performance degradation if no regularization used

Extended experiments and analyses

  • Conducted extended experiments
  • Studied subtleties of watermarked DMs
  • Two generation paradigms

Limitations

  • Recipe for watermarking different types of DMs is simple and effective
  • Injecting a watermark string into all training images results in a distribution shift
  • Trade off between generation accuracy of watermark image and generative performance
  • Different types of watermark information can be embedded in DMs

Conclusion and discussion

  • Conducted empirical study on watermarking of unconditional/class-conditional and text-to-image DMs
  • Simple and efficient watermarking pipelines
  • One of the first attempts to watermark large-scale DMs
  • Findings and experiments pave the way for copyright/ownership information to be added to large-scale DMs
  • Positive impact on finetuning of large-scale DMs with few-shot data

Overview of appendix

  • Investigated watermarking diffusion models in two major types: unconditional/class-conditional generation and text-to-image generation
  • Provided additional implementation details, experiments, and analysis to support proposed methods in main paper

A. additional implementation details

  • Watermark encoder and decoder discussed
  • Network architecture and objective for optimization during training of watermark encoder and decoder
  • Different datasets used for training (CIFAR-10, FFHQ, AFHQv2, ImageNet)
  • Inference stage uses same watermark for entire training set
  • Watermark information mainly resides at fine-grained levels
  • Weights-constrained fine-tuning method proposed
  • Rare identifier used as trigger prompt
  • Watermark image can be accurately generated given trigger prompt
  • Quality of generated images degraded with increased bit length
  • Bit accuracy of generated images remains stable with increased bit length
  • Robustness of watermarked generated images evaluated by adding Gaussian noise to weights of models
  • FID score explodes with increased strength of Gaussian noise added directly to generated images
  • Bit accuracy remains stable
  • Watermarked text-to-image model can accurately generate predefined watermark image
  • Common text as trigger prompt leads to overfitting of watermark image
  • Watermark knowledge perturbed after fine-tuning
  • Negative impact on resulting watermarked diffusion models
  • Potential social and ethical issues if used by malicious users