Better depth#1393
Draft
MrNeRF wants to merge 12 commits into
Draft
Conversation
# Conflicts: # src/python/lfs_plugins/training_panel.py
Contributor
There was a problem hiding this comment.
Pull request overview
This PR expands “better depth” support across the training and visualization stack by replacing legacy depth loss modes with an SSI (scale/shift-invariant) depth prior workflow, adding optional normal-map supervision, and improving dataset sidecar discovery/readback paths (depth + normals).
Changes:
- Introduces SSI depth supervision with optional per-camera depth-anchor fitting + sidecar cache, plus 16-bit depth prior handling and frame-number fallback matching.
- Adds an optional normal render channel to the FastGS rasterizer and implements normal prior loss + depth-normal consistency loss (including “flatten” regularization).
- Adds GT comparison mode (RGB/Normal/Depth) plumbing through render settings, IPC/Python API, and viewport overlay UI; updates training UI and locales for new depth/normal options.
Reviewed changes
Copilot reviewed 70 out of 70 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
| tests/test_normal_prior_decode.cpp | New unit tests covering sRGB normal-prior decode helpers. |
| tests/test_fastgs_kernels.cpp | Adds CUDA tests for normal-channel forward output and rotation gradient correctness. |
| tests/test_colmap_image_layout.cpp | Tests trailing-frame-number fallback and ambiguity handling for sidecar lookup. |
| tests/test_argument_parser.cpp | Updates depth-loss defaults/modes and adds normal-loss CLI parsing tests. |
| tests/python/test_property_system.py | Verifies new normal-loss parameters are exposed/editable via Python. |
| tests/CMakeLists.txt | Registers the new normal-prior decode test file. |
| src/visualizer/rendering/vksplat_viewport_renderer.hpp | Declares output-depth image readback API. |
| src/visualizer/rendering/vksplat_viewport_renderer.cpp | Implements Vulkan output depth image readback via staging buffer. |
| src/visualizer/rendering/rendering_types.hpp | Adds GTComparisonMode + sanitization helper and persists it in RenderSettings. |
| src/visualizer/rendering/rendering_manager.cpp | Sanitizes GT comparison mode alongside other render settings. |
| src/visualizer/ipc/view_context.hpp | Extends IPC view context with gt_comparison_mode. |
| src/visualizer/ipc/render_settings_convert.hpp | Serializes/deserializes gt_comparison_mode and sanitizes it on load. |
| src/visualizer/gui/rmlui/resources/viewport_overlay.rml | Adds GT compare mode selector block to viewport overlay. |
| src/visualizer/gui/rmlui/resources/viewport_overlay.rcss | Styles the new GT compare overlay block. |
| src/visualizer/gui/rmlui/resources/training.rml | Updates depth loss mode choices; adds normal loss controls. |
| src/visualizer/gui/resources/locales/zh.json | Updates depth loss strings and adds normal loss strings (ZH). |
| src/visualizer/gui/resources/locales/en.json | Updates depth loss strings and adds normal loss + GT compare strings (EN). |
| src/training/trainer.hpp | Adds training-state tensors/flags for depth anchors and normal supervision. |
| src/training/rasterization/fastgs/rasterization/src/rasterization_api.cu | Extends FastGS API to forward/backward normal channel and allocates helper buffers. |
| src/training/rasterization/fastgs/rasterization/src/forward.cu | Threads optional normal map output through the blend kernel via template dispatch. |
| src/training/rasterization/fastgs/rasterization/src/backward.cu | Adds optional normal-map gradient path through blend backward via template dispatch. |
| src/training/rasterization/fastgs/rasterization/include/rasterization_api.h | Updates public FastGS forward/backward signatures to include normal channel pointers. |
| src/training/rasterization/fastgs/rasterization/include/kernels_forward.cuh | Computes per-primitive camera-space normals and blends them into an optional normal map. |
| src/training/rasterization/fastgs/rasterization/include/kernels_backward.cuh | Propagates normal-map gradients to per-primitive normals and blend-weight derivatives. |
| src/training/rasterization/fastgs/rasterization/include/kernel_utils.cuh | Adds flatten regularization gradient helper for min-axis scale. |
| src/training/rasterization/fastgs/rasterization/include/fused_adam_types.h | Adds flatten_reg_weight to fused Adam settings. |
| src/training/rasterization/fastgs/rasterization/include/forward.h | Updates internal forward signature to include normal outputs + scratch buffer. |
| src/training/rasterization/fastgs/rasterization/include/backward.h | Updates internal backward signature to include normal gradients + scratch buffers. |
| src/training/rasterization/fast_rasterizer.hpp | Extends render outputs and backward entrypoints with optional normal tensors. |
| src/training/rasterization/fast_rasterizer.cpp | Allocates/returns normal map on demand and wires normal gradient into backward. |
| src/training/optimizer/render_output.hpp | Adds normal tensor to render output struct for training/optimization paths. |
| src/training/kernels/normal_loss.hpp | Declares normal prior supervision kernel API + constants/partials layout. |
| src/training/kernels/normal_loss.cu | Implements alpha-weighted cosine normal-prior loss + gradient emission. |
| src/training/kernels/normal_consistency_loss.hpp | Declares depth-normal consistency loss kernel API + constants/partials layout. |
| src/training/kernels/normal_consistency_loss.cu | Implements depth-derived normal consistency loss with grads into normal/depth/alpha. |
| src/training/kernels/depth_loss.hpp | Reworks depth loss API around SSI + depth-anchor alignment and diagnostics. |
| src/training/depth_anchor/depth_anchor_cache.hpp | Declares depth-anchor fit, fingerprinting, and sidecar read/write API. |
| src/training/depth_anchor/depth_anchor_cache.cpp | Implements anchor computation pipeline (GPU sample collection + CPU robust fit) and JSON sidecar cache. |
| src/training/depth_anchor/CMakeLists.txt | Adds lfs_depth_anchor helper library for trainer + preprocess reuse. |
| src/training/dataset.hpp | Adds optional normal loading to dataset examples and pipelined auxiliary config. |
| src/training/CMakeLists.txt | Builds new normal kernels and adds depth_anchor subdirectory/library dependency. |
| src/python/stubs/lichtfeld/ui/init.pyi | Adds GT comparison mode getter/setter/cycle stubs. |
| src/python/stubs/lichtfeld/init.pyi | Adds OptimizationParams fields for normal loss + updates depth mode docstring. |
| src/python/lfs/py_ui.cpp | Exposes GT comparison mode APIs to Python and wires into rendering manager settings. |
| src/python/lfs/py_rendering.cpp | Adds gt_comparison_mode enum property proxy for UI bindings. |
| src/python/lfs/py_params.cpp | Exposes normal-loss optimization parameters via the Python parameter system. |
| src/python/lfs_plugins/toolbar.py | Mounts/updates the new GT compare overlay controller. |
| src/python/lfs_plugins/gt_compare_controls.py | New controller for GT compare overlay visibility + mode binding. |
| src/preprocessing/preprocess.cpp | Adds 8/16-bit PNG output support, flat-layout image scanning, and depth-anchor precompute after preprocessing. |
| src/preprocessing/CMakeLists.txt | Links preprocess tool with scene IO and lfs_depth_anchor for anchor precompute. |
| src/io/pipelined_image_loader.cpp | Adds normal sidecar prefetch/decode/resize/undistort; supports 16-bit depth decode path. |
| src/io/loaders/blender_loader.cpp | Adds normal sidecar lookup/validation and passes normal_path into Camera. |
| src/io/include/io/pipelined_image_loader.hpp | Extends loader request/ready structs and stats to carry normal sidecars. |
| src/io/include/io/filesystem_utils.hpp | Generalizes sidecar lookup (depth/normals) with trailing-digit fallback + ambiguity rules. |
| src/io/include/io/error.hpp | Adds NORMAL_SIZE_MISMATCH error code. |
| src/io/formats/colmap.cpp | Adds normal sidecar lookup/validation and logs depth/normal match rates. |
| src/core/parameters.cpp | Adds normal-loss parameters serialization/validation and updates depth loss validation/migration. |
| src/core/include/core/parameters.hpp | Adds normal-loss parameters and preprocess bit-depth parameter. |
| src/core/include/core/image_io.hpp | Adds high-bitdepth image loaders, quantization probe, and normal prior decode utilities. |
| src/core/include/core/camera.hpp | Adds normal_path and cached normal loading/decoding API; caches depth quantization step. |
| src/core/image_io.cpp | Implements high-bitdepth loaders, quantization probe, and normal prior decode/sampling helpers. |
| src/core/cuda/lanczos_resize/lanczos_resize.hpp | Extends grayscale resize to accept float32 and adds float CHW resize API. |
| src/core/cuda/lanczos_resize/lanczos_resize.cu | Implements float32 grayscale path and planar 3-channel float Lanczos resize. |
| src/core/camera.cpp | Adds high-bitdepth depth loading path, normal loading/decoding pipeline, and quantization-step caching. |
| src/core/argument_parser.cpp | Adds CLI flags for normal supervision and preprocess output bit depth; updates depth mode help text. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| return "depth_loss_mode must be 'pearson' or 'adaptive-warped-l1'"; | ||
| if (depth_loss_mode != "ssi" && depth_loss_mode != "ssi-disparity" && depth_loss_mode != "ssi-depth" && | ||
| depth_loss_mode != "pearson" && depth_loss_mode != "adaptive-warped-l1") | ||
| return "depth_loss_mode must be 'ssi', 'ssi-disparity', or 'ssi-depth'"; |
Comment on lines
+593
to
+602
| template <typename PixelT> | ||
| void write_png(const fs::path& path, | ||
| int width, | ||
| int height, | ||
| int channels, | ||
| const std::vector<PixelT>& data, | ||
| int png_compression) { | ||
| static_assert(std::is_same_v<PixelT, uint8_t> || std::is_same_v<PixelT, uint16_t>); | ||
| constexpr auto kFormat = std::is_same_v<PixelT, uint16_t> ? OIIO::TypeDesc::UINT16 | ||
| : OIIO::TypeDesc::UINT8; |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.