PRISM: Learning Realistic Depth via Physics-Grounded Noise Disentanglement with Semantic-Geometric Collaboration

The anatomy of depth perception and modeling: physical sensing failures and noise modalities — **Figure 1 · Why does simulated depth fall short?** Ideal simulation omits the structured signal loss and measurement bias of real depth sensing. PRISM models these physically motivated modalities separately, moving beyond a monolithic noise representation.

Abstract

Real-world physical sensing exhibits complex, heterogeneous noise patterns that deviate significantly from idealized simulation, posing a fundamental bottleneck for sim-to-real transfer. Existing sensor modelings typically treat depth noise as a monolithic black-box process, overlooking the distinct physical mechanisms that govern different error modalities. In this work, we introduce a physics-grounded paradigm that disentangles monolithic noise into two complementary modalities: sensing invalidation and measurement inaccuracy, enabling a tailored treatment of noise sources based on their physical origins. Building on this insight, we propose PRISM, a tripartite framework that distills 3D Visual Foundation Model features as rich spatial-semantic priors for physics-based reasoning. To address the inherent sparsity and class imbalance of invalidation regions, we develop Hierarchical Positive-Prioritized Supervision, integrating multi-scale positive-weighted objectives with a positive-preserving dynamic hard mining strategy to enforce precise artifact delineation. Extensive benchmarks demonstrate that PRISM achieves state-of-the-art fidelity in noisy depth synthesis. Furthermore, downstream robotic experiments show that PRISM facilitates a 92.1% average success rate in the real world, marking a significant improvement over monolithic baselines.

Physics gives noise a structure.

Physics-grounded decomposition connects realistic depth synthesis with embodied manipulation learning.

01 / DISENTANGLEMENT

Two complementary modalities

Separate sensing invalidation from measurement inaccuracy, so missing depth and noisy depth receive different treatments.

02 / COLLABORATION

Semantics meets geometry

Bring 3D visual foundation model priors together with simulated geometry to locate and synthesize scene-dependent artifacts.

03 / ROBOT LEARNING

Offline augmentation for policies

Generate realistic depth before policy training, so manipulation policies learn from sensor artifacts without requiring PRISM at deployment.

Reason → Disentangle → Generate

PRISM architecture overview: Semantic-Physics Reasoner, Bimodal Noise Disentangler and Noise Residual Generator — **Figure 2 · Semantic–geometric collaboration** SPR extracts 3D visual priors, BND combines them with simulated geometry to locate sensing invalidation, and NRG generates continuous measurement residuals.

Semantic-Physics Reasoner · SPR

A frozen 3D visual foundation model provides spatial-semantic features. Global context and multi-scale attention turn those features into priors for sensor degradation.

Bimodal Noise Disentangler · BND

Combining RGB semantics and ideal depth geometry, BND predicts the invalidation mask with supervision designed for sparse positives.

Noise Residual Generator · NRG

Conditional diffusion uses simulated geometry and SPR context to synthesize continuous residuals. Mask-aware supervision focuses residual learning on valid sensing regions.

D_syn = (1 − M) · (D_sim + R)

Realistic depth, measured.

On ByteCameraDepth, PRISM improves both overall synthesis fidelity and the localization of invalid sensor measurements.

0.076Overall MAE ↓

0.952Invalidation IoU ↑

0.991Invalidation recall ↑

Original paper table 1: depth synthesis fidelity on ByteCameraDepth — **Table 1 · Depth synthesis on ByteCameraDepth** RealSense D435 in-domain evaluation measures overall fidelity, sensing invalidation and valid-region accuracy. PRISM achieves an overall MAE of 0.076 and invalidation IoU of 0.952.

Original paper Table 2, with all methods and metrics — **Table 2 · Zero-shot cross-camera evaluation** Models trained on ByteCameraDepth are evaluated zero-shot on Kinect NYU-Depth-v2. PRISM achieves overall MAE 0.082 and invalidation IoU 0.923.

Qualitative depth synthesis and baseline comparisons — **Figure 4 · Qualitative comparison with noise synthesis baselines** The comparison shows RGB input, ideal simulation, synthesized depth and the real reference. PRISM reproduces structured missing regions while retaining valid scene geometry.

Appendix · Figures 10–14

Five scenes. Diverse sensing challenges.

Qualitative results across Kitchen, Living Room, Market, Office and Lounge compare ideal depth, predicted and reference noise components, and synthesized depth.

Kitchen

Figure 10

All scenes: Kitchen Living Room Market Office Lounge

From simulation to real-world manipulation.

PRISM enhances the training data offline, enabling zero-shot deployment without a PRISM module at policy inference time.

Sim-to-real data generation, policy learning and deployment pipeline — **Figure 3 · Offline depth augmentation for policy learning** Simulation demonstrations are enhanced before training. Depth-adapted DP3, PPT and RISE policies are then deployed directly on physical robots, without running PRISM during policy inference.

92.1% average real-world success

Across DP3, PPT and RISE, PRISM reaches 92.1% average success on six real-world tasks, compared with 38.9% for clean simulated depth and 89.1% for Stable-Sim2Real.

The gain over clean depth is 53.2 percentage points. The RISE policy reaches 92.5% across the same six tasks. Improvements across all three policy architectures indicate that the enhanced observations support multiple approaches to manipulation learning.

Original paper Table 4, with all methods and metrics — **Table 4 · Success across three policy architectures** Average success (%) over six tasks: 92.1 for PRISM, 89.1 for Stable-Sim2Real and 38.9 for clean simulation. DP3, PPT and RISE are adapted to process depth observations.

Original paper Table 3, with all methods and metrics — **Table 3 · Six manipulation tasks with RISE** Successful trials out of 20, reported as mean ± standard deviation over three random seeds. PRISM achieves 92.5% average success.

What makes the decomposition effective?

Component, backbone and supervision studies connect the modeling choices to depth synthesis and real-world manipulation performance.

Original paper Table 17, with all methods and metrics — **H-PPS ablation · Learning sparse invalidation** Positive-weighted and multi-scale supervision are combined with Dice loss and positive-preserving dynamic hard mining. All invalid pixels remain in training; hard mining selects only negative pixels.

Original paper Table 15, with all methods and metrics — **Component ablation · Reasoning and disentanglement** Removing SPR or BND reduces manipulation success to 83.7% or 81.5%, compared with 92.5% for the complete framework. NRG alone reaches 72.5%.

Original paper Table 16, with all methods and metrics — **Backbone comparison · Geometry-aware visual priors** The comparison spans CNNs, vision transformers and 3D foundation models, with MoGe2-Large performing best.

Appendix · Figure 9

Aligned environments, from real to simulated.

Real and simulated workspaces support demonstration collection, policy learning and evaluation on physical robots.

Figure 9: real-world robot setup on the left and corresponding simulation construction on the right — **Figure 9 · Real and simulated manipulation environments** RealSense D435 captures two robot workspaces paired with aligned simulations. Franka Research 3 handles Tasks 1–3; Realman RM75 handles Tasks 4–6.

Franka Research 3 · Tasks 1–3

Precision manipulation of cans, bananas and stacked cubes.

Realman RM75 · Tasks 4–6

Articulated interactions involving a bucket handle, drawer and microwave.

RealSense D435 RGB-D perception · Geometrically aligned simulation environments.

Appendix · Figures 15–20

Six tasks in the physical world.

Six real-world tasks evaluate grasping, placement and multi-stage articulated interaction on Franka Research 3 and Realman RM75.

T1 · Can-to-Plate

Figure 15

Franka Research 3Specular

Pick up the metallic can and place it upright on the plate.

Sensing challenge. Reflective surfaces create missing depth, requiring robust inference from fragmented boundaries.

Figure 15: complete Can-to-Plate real-world manipulation rollout — **Figure 15 · Can-to-Plate** Pick up the metallic can and place it upright on the plate. The sequence shows a policy trained on PRISM-enhanced simulation data.

All rollouts: T1 · Can-to-Plate T2 · Banana-to-Plate T3 · Stack Cubes T4 · Bucket-to-Plate T5 · Toyball-to-Drawer T6 · Bread-to-Microwave

Current limitations & future work

The current model still needs stronger generalization across sensor families, and per-frame synthesis does not explicitly enforce temporal noise consistency. Interaction-induced noise and extreme optical conditions also remain directions for further study. Extensions for cross-sensor generalization and temporal noise consistency are underway. The project code will be released in the future.

BibTeX

@inproceedings{liang2026prism,
  title     = {{PRISM}: Learning Realistic Depth via Physics-Grounded Noise Disentanglement with Semantic-Geometric Collaboration},
  author    = {Liang, Xiujian and Liu, Jiacheng and Sun, Mingyang and He, Qichen and Cheng, Anda and Lu, Cewu and Sun, Jianhua},
  booktitle = {Proceedings of the 43rd International Conference on Machine Learning},
  volume    = {306},
  pages     = {73301--73335},
  year      = {2026},
  publisher = {PMLR},
  url       = {https://proceedings.mlr.press/v306/liang26n.html}
}

Download BibTeX ↓