Link to paper

The full paper is available here.

You can also find the paper on PapersWithCode here.

Abstract

  • Neural fields have been used for 3D vision tasks
  • Several methods have been proposed to estimate distance or density fields using neural fields
  • Neural Radiance Field (NeRF) does not provide density gradient in most empty regions
  • Neural Implicit Surface (NeuS) has limitations in objects’ surface shapes
  • This paper proposes Neural Density-Distance Field (NeDDF) to reciprocally constrain the distance and density fields
  • NeDDF enables explicit conversion from distance field to density field
  • Experiments show NeDDF can achieve high localization performance

Paper Content

Introduction

  • Representing 3D shapes using coordinate-based neural networks
  • Neural Radiance Fields (NeRF) have shown impressive quality for tasks such as novel view-synthesis
  • Proposed Neural Density-Distance Field (NeDDF) achieves robust localization with distance fields while providing object reconstruction quality comparable to NeRF
  • Two main types of 3D shape representation in neural fields: density field and distance field
  • Distance field provides gradient over a wide range even after optimization converges
  • NeDDF has a network that inputs a position and outputs the distance and its gradient, and a converter that explicitly calculates the density
  • Three contributions: extending the distance field, recovering corresponding density, and implementation to alleviate instability of distance gradient

Neural fields

  • Traditional way of representing volumes is to discretize density or distance into voxels
  • Memory-efficient representations such as octree or hash table have been proposed
  • Geometric deep learning methods can handle irregular non-grid structures
  • Neural fields can model output dimensions without increasing model capacity
  • Modeling using gradient information has been proposed

Density field

  • Density field outputs volume density for 3D position
  • Used with color field to enable volume rendering
  • Low density value can describe semi-transparent objects
  • Can model specular reflections
  • NeRF has limitation of known camera pose and static scene
  • Many NeRF-based methods proposed to address this
  • Blank areas with density value of 0 have uncertain gradient directions
  • NeDDF provides consistent distance field while retaining expressiveness of density field
  • Can improve registration performance from rough initial camera poses

Distance field

  • Distance field takes 3D position as input and outputs distance to nearest boundary
  • Widely used in fusion and registration because provides stable surfaces and normal vectors
  • Provides residuals and gradient directions for fast-fitting of two shapes
  • KinectFusion and DynamicFusion use SDF for localization and shape integration
  • DeepSDF, SAL, UDF, IDR, UNISURF, VolSDF, NeuS use neural fields to handle distance fields
  • This study extends distance field to correspond to various density distributions from depth values

Method

  • Distance and density fields are considered
  • Distance field is redefined to interpret arbitrary density fields
  • Conversion formula is introduced to obtain density of independent points from distance and gradient of distance value

Distance field from density field

  • Distance field in boundary surfaces describes the distance to the nearest surface for a given location.
  • Depth value is replaced with the depth value derived from the volume rendering equation.
  • Depth is defined as an integral of the depths at each point.
  • Distance field removes dependence on viewing direction.
  • Gradient of the distance field is used to restore the quantity from the tangent plane.

Density from distance field

  • Extended distance field to shapes with no explicit boundary surface
  • Derived corresponding density field when distance field is known
  • Expression for density can be obtained from equations 7 and 10
  • Density can be calculated using equation 14

Removing cusps

  • Equation 14 assumes distance field is first-order differentiable
  • Distance field is not differentiable at cusps
  • Extend domain of distance field to 4-dimensional hyperspace
  • Use heuristic constraint to constrain shape of auxiliary gradient
  • Introduce weight coefficient to discriminate target point

Reprojection error for volume rendering

  • Previous NeRF-based localization uses photometric error to follow local regions with smooth color changes.
  • NeDDF provides direction and approximate distance from a sampling point.
  • Color information is used to calculate the correspondence points.
  • Penalty is used to record the same color in the gradient direction of the distance field.
  • Reprojection error is used to calculate the distance between the pixel coordinates of the ray and the projected pseudo-correspondence point.
  • NeDDF retains comparable quality of novel view synthesis as NeRF.
  • Optimization by reprojection error can roughly estimate the camera pose and reduce the impact of wrong correspondence points.

Conclusions

  • Proposed NeDDF can represent both distance and density fields
  • NeDDF alleviates discontinuity points by introducing auxiliary gradient
  • Visualization experiments show NeDDF can acquire properties of both distance and density fields
  • Quantitative evaluation shows NeDDF provides competitive quality of novel view synthesis, more stable meshes, and a more comprehensive range of following camera poses than NeRF
  • Limitation of NeDDF is lack of information about distance field inside objects
  • NeDDF has same limitations as NeRF, such as time-consuming optimization and rendering
  • Reprojection error can be calculated from colors in localization
  • Using information with higher uniqueness, such as semantic segmentation, can increase usefulness of reprojection error
  • Network uses MLPs with width of 256
  • Activation function used is tanhExp
  • PE up to L dimensions uses sin and cos of each dimension of position p scaled by powers of 2 from 1 to 2L − 1
  • Intermediate input of conventional γ(p) added in layers after distance output for learning detailed color fields
  • Parameter selection for shape of auxiliary gradient uses hyperparameter α
  • Synthetic dataset for smoke-subject scene produced
  • NeDDF achieves high-quality Novel View Synthesis in smoke-like scenes
  • Camera localization performance improved with reprojection error
  • PSNR reported for quantitative evaluation on synthetic dataset