Link to paper
The full paper is available here.
You can also find the paper on PapersWithCode here.
Abstract
- Neural fields have been used for 3D vision tasks
- Several methods have been proposed to estimate distance or density fields using neural fields
- Neural Radiance Field (NeRF) does not provide density gradient in most empty regions
- Neural Implicit Surface (NeuS) has limitations in objects’ surface shapes
- This paper proposes Neural Density-Distance Field (NeDDF) to reciprocally constrain the distance and density fields
- NeDDF enables explicit conversion from distance field to density field
- Experiments show NeDDF can achieve high localization performance
Paper Content
Introduction
- Representing 3D shapes using coordinate-based neural networks
- Neural Radiance Fields (NeRF) have shown impressive quality for tasks such as novel view-synthesis
- Proposed Neural Density-Distance Field (NeDDF) achieves robust localization with distance fields while providing object reconstruction quality comparable to NeRF
- Two main types of 3D shape representation in neural fields: density field and distance field
- Distance field provides gradient over a wide range even after optimization converges
- NeDDF has a network that inputs a position and outputs the distance and its gradient, and a converter that explicitly calculates the density
- Three contributions: extending the distance field, recovering corresponding density, and implementation to alleviate instability of distance gradient
Neural fields
- Traditional way of representing volumes is to discretize density or distance into voxels
- Memory-efficient representations such as octree or hash table have been proposed
- Geometric deep learning methods can handle irregular non-grid structures
- Neural fields can model output dimensions without increasing model capacity
- Modeling using gradient information has been proposed
Density field
- Density field outputs volume density for 3D position
- Used with color field to enable volume rendering
- Low density value can describe semi-transparent objects
- Can model specular reflections
- NeRF has limitation of known camera pose and static scene
- Many NeRF-based methods proposed to address this
- Blank areas with density value of 0 have uncertain gradient directions
- NeDDF provides consistent distance field while retaining expressiveness of density field
- Can improve registration performance from rough initial camera poses
Distance field
- Distance field takes 3D position as input and outputs distance to nearest boundary
- Widely used in fusion and registration because provides stable surfaces and normal vectors
- Provides residuals and gradient directions for fast-fitting of two shapes
- KinectFusion and DynamicFusion use SDF for localization and shape integration
- DeepSDF, SAL, UDF, IDR, UNISURF, VolSDF, NeuS use neural fields to handle distance fields
- This study extends distance field to correspond to various density distributions from depth values
Method
- Distance and density fields are considered
- Distance field is redefined to interpret arbitrary density fields
- Conversion formula is introduced to obtain density of independent points from distance and gradient of distance value
Distance field from density field
- Distance field in boundary surfaces describes the distance to the nearest surface for a given location.
- Depth value is replaced with the depth value derived from the volume rendering equation.
- Depth is defined as an integral of the depths at each point.
- Distance field removes dependence on viewing direction.
- Gradient of the distance field is used to restore the quantity from the tangent plane.
Density from distance field
- Extended distance field to shapes with no explicit boundary surface
- Derived corresponding density field when distance field is known
- Expression for density can be obtained from equations 7 and 10
- Density can be calculated using equation 14
Removing cusps
- Equation 14 assumes distance field is first-order differentiable
- Distance field is not differentiable at cusps
- Extend domain of distance field to 4-dimensional hyperspace
- Use heuristic constraint to constrain shape of auxiliary gradient
- Introduce weight coefficient to discriminate target point
Reprojection error for volume rendering
- Previous NeRF-based localization uses photometric error to follow local regions with smooth color changes.
- NeDDF provides direction and approximate distance from a sampling point.
- Color information is used to calculate the correspondence points.
- Penalty is used to record the same color in the gradient direction of the distance field.
- Reprojection error is used to calculate the distance between the pixel coordinates of the ray and the projected pseudo-correspondence point.
- NeDDF retains comparable quality of novel view synthesis as NeRF.
- Optimization by reprojection error can roughly estimate the camera pose and reduce the impact of wrong correspondence points.
Conclusions
- Proposed NeDDF can represent both distance and density fields
- NeDDF alleviates discontinuity points by introducing auxiliary gradient
- Visualization experiments show NeDDF can acquire properties of both distance and density fields
- Quantitative evaluation shows NeDDF provides competitive quality of novel view synthesis, more stable meshes, and a more comprehensive range of following camera poses than NeRF
- Limitation of NeDDF is lack of information about distance field inside objects
- NeDDF has same limitations as NeRF, such as time-consuming optimization and rendering
- Reprojection error can be calculated from colors in localization
- Using information with higher uniqueness, such as semantic segmentation, can increase usefulness of reprojection error
- Network uses MLPs with width of 256
- Activation function used is tanhExp
- PE up to L dimensions uses sin and cos of each dimension of position p scaled by powers of 2 from 1 to 2L − 1
- Intermediate input of conventional γ(p) added in layers after distance output for learning detailed color fields
- Parameter selection for shape of auxiliary gradient uses hyperparameter α
- Synthetic dataset for smoke-subject scene produced
- NeDDF achieves high-quality Novel View Synthesis in smoke-like scenes
- Camera localization performance improved with reprojection error
- PSNR reported for quantitative evaluation on synthetic dataset