DepState-JEPA: Learning Depth-Grounded State Representation for Visual Perception and Manipulation

THE BIG PICTURE

Understand the scene.
Anticipate the change.

DepState-JEPA connects complementary RGB and depth observations, a depth-grounded shared state, and visual perception and manipulation. — **From where things are to what actions change.** Complementary RGB and depth observations ground a common state in metric 3D space. DepState connects spatial queries, action-conditioned feature forecasting, and robot manipulation.

Abstract

Predictive representations for robot manipulation should connect scene geometry with the state changes induced by actions. Our intuition is that understanding what an action will change starts with understanding where things are: complementary RGB and depth observations ground a common scene representation from which to infer those changes. Guided by this intuition, we introduce DepState-JEPA, a dual-stream RGB–Depth joint-embedding predictive architecture that connects spatial understanding with action-conditioned prediction through a shared latent state, DepState.

Built on V-JEPA 2.1, the framework uses depth both as a complementary observation stream and to ground both modalities in metric 3D space, aggregating shared information into DepState while retaining modality-private features for decoding. Ray-conditioned readouts predict local features without target depth; after an action-conditioned transition, the same decoders predict future RGB and depth features using the predicted state and fixed initial private features. Robot-state readouts provide recursive proprioceptive inputs; object-trajectory supervision and calibrated forward kinematics shape state learning through motion targets and joint–TCP consistency.

Comparisons show stronger RGB-encoder transfer after RGB-D adaptation, feature completion, conditioned visual forecasting, and manipulation success. Within-model ablations identify contributions from depth content, metric coordinates, and future-state matching, with additional control gains from FK regularization.

GEOMETRY → SHARED STATE → ACTION PREDICTION

RGB tells what. Depth tells where.
DepState connects both to what happens next.

01 / COMPLEMENTARY OBSERVATIONS

Ground perception in 3D

Independent RGB and depth encoders capture appearance and geometry. Visible metric depth places both streams in a common 3D coordinate frame.

02 / A COMMON STATE INTERFACE

Share what should evolve

DepState aggregates shared information. Private features preserve modality-specific decoding detail and stay fixed throughout temporal rollout.

03 / ACTION-CONDITIONED DYNAMICS

Connect perception to action

Learned transitions evolve DepState under actions and proprioception. Unified readouts support feature prediction, robot-state prediction, and manipulation.

METHOD

One depth-grounded state. Spatial and temporal readouts.

Architecture with independent encoders, metric 3D grounding, shared-private projection, action-conditioned transitions and ray-conditioned decoding. — **A shared interface between geometry and dynamics.** Visible depth and camera calibration ground both streams in 3D. Learned aggregation constructs DepState, while private features retain modality-specific decoding detail. The same readouts decode current or predicted states. Input scenes in this diagram are illustrative.

Depth-grounded state construction

RGB and depth features are lifted using visible depth and camera intrinsics. Shared–private projections separate the aggregation pathway from modality-specific decoding pathways. Learned queries aggregate shared content into DepState.

Ray-conditioned feature prediction

Local features are queried by camera rays without using target depth. The same decoders support masked completion and future visual feature prediction, with the initial private memory fixed during temporal rollout.

Physically supervised dynamics

Action-conditioned transitions predict the next state. Robot-state heads supply recursive proprioception; object-trajectory supervision and calibrated forward kinematics provide motion targets and joint–TCP consistency during training.

STAGE A

RGB-D adaptation

Learn spatial and temporal representations from images and video through masked shared-feature completion and dense modality-feature prediction.

STAGE B

Robot post-training

Learn action-conditioned transitions and physical readouts, with future-state matching, feature targets, physical supervision, and RGB-D replay.

SPATIAL PERCEPTION

Stronger visual representations after RGB-D adaptation.

Frozen probes evaluate monocular depth estimation on NYUv2 and semantic segmentation on VOC12. A separate masked RGB-D completion protocol evaluates the shared state and its decoding paths on held-out LingBot-Depth scenes.

0.088NYUv2 AbsRel ↓

86.0VOC12 mIoU ↑

0.951Completion cosine ↑

Method	NYUv2 AbsRel ↓	NYUv2 RMSE ↓	VOC12 mIoU ↑	Completion L1 ↓	Completion cosine ↑
LeJEPA	0.137	0.490	76.0	0.352	0.891
V-JEPA 2	0.188	0.642	63.9	0.428	0.843
V-JEPA 2.1	0.098	0.350	84.7	0.298	0.931
3D-JEPA †	—	—	—	0.283	0.938
LeMuMo-JEPA †	0.097	0.345	84.2	0.291	0.936
JEPADepth	0.090	0.323	84.5	0.312	0.923
DepState-JEPA Ours	0.088	0.316	86.0	0.247	0.951

Linear probes on NYUv2 and VOC12 use frozen encoders. Completion is measured on held-out LingBot-Depth scenes, with identical RGB/depth masks and masked-region scoring against shared frozen references. 3D-JEPA is not evaluated for RGB probes because it takes point clouds. † Method-specific adaptation.

Monocular depth estimation comparisons on NYUv2 from the paper. — **Monocular depth estimation on NYUv2.** Task-specific probes read frozen encoder features. RGB inputs and ground-truth depth are shown as references, with a common metric color scale for predicted depth.

VISUAL PREDICTION

Predict representations across time.

Starting from matched observations, the model predicts future visual features without future images. The comparison below supplies recorded actions and proprioception at each step and reports cosine similarity in a common RGB reference space.

Method	1 step ↑	8 steps ↑	16 steps ↑	24 steps ↑	32 steps ↑
V-JEPA 2-AC	0.916	0.792	0.701	0.646	0.598
V-JEPA 2.1-AC	0.967	0.874	0.781	0.702	0.636
3D-JEPA-AC †	0.968	0.879	0.789	0.711	0.647
LeMuMo-JEPA-AC †	0.969	0.882	0.778	0.694	0.627
DepState-JEPA-AC Ours	0.976	0.903	0.851	0.811	0.779

Frozen linear readouts map predictions into a shared V-JEPA 2.1 RGB reference space. Readouts remain fixed across horizons; initial private memory remains fixed during DepState-JEPA rollout. This forecasting protocol uses recorded proprioception; control rollouts use predicted proprioception. † Method-specific action-conditioned adaptation.

PCA visualizations of spatial completion and multi-step visual feature forecasting. — **Complete the present. Forecast future features.** The paper visualizes masked spatial completion and temporal predictions using PCA colors. These are feature-space visualizations, rather than generated RGB images. Temporal comparisons are conditioned on recorded actions and proprioception.

EMBODIED LEARNING & CONTROL

From learned predictions to robot manipulation.

The paper evaluates both online policy learning on imagined trajectories and latent model predictive control. Simulation-trained models also transfer to real-world drawer and cup tasks using visual-goal MPC.

90.1%Meta-World · latent MPC

81.1%Real-world drawer success

78.9%Real-world cup success

Method	Online policy learning		Latent MPC		Sim-to-real MPC
Method	DMC return ↑	MW success (%) ↑	DMC return ↑	MW success (%) ↑	Drawer (%) ↑	Cup (%) ↑
V-JEPA 2-AC	682.4 ± 14.7	52.8 ± 2.8	736.5 ± 15.3	66.4 ± 2.4	45.6 ± 5.1	41.1 ± 5.1
V-JEPA 2.1-AC	763.8 ± 15.4	68.6 ± 2.3	807.2 ± 15.0	81.2 ± 2.6	66.7 ± 6.7	64.4 ± 5.1
3D-JEPA-AC †	784.4 ± 12.6	72.3 ± 2.6	826.4 ± 18.2	85.6 ± 3.2	73.3 ± 5.8	71.1 ± 5.1
LeMuMo-JEPA-AC †	790.4 ± 16.1	73.2 ± 2.5	837.5 ± 13.6	87.8 ± 2.2	75.6 ± 6.9	74.4 ± 8.4
DepState-JEPA-AC	813.2 ± 14.3	77.1 ± 2.8	857.8 ± 17.8	90.1 ± 2.8	81.1 ± 5.1	78.9 ± 3.8

DMC: task-averaged episode return over 20 tasks. Meta-World: average success over 50 tasks. Sim-to-real: full-task success over three seeds, 30 trials per seed. † Method-specific action-conditioned adaptation. Complete-system comparisons use the pretraining and capacity stated in the paper.

Simulation and real-world workspaces with two viewpoints of RGB-D manipulation rollouts. — **Matched environments, from simulation to the real robot.** Simulation-trained models remain frozen during real-world visual-goal MPC. The evaluation covers drawer opening and closing, and cup picking and placing. Multiple views record RGB and depth observations.

REAL-ROBOT DEMONSTRATIONS

Watch the robot through RGB and depth.

Explore recorded executions across three action sequences, two camera viewpoints, and two observation modalities. Drawer opening and closing are phases of the full drawer task; the cup sequence shows picking and placing.

RECORDED EXECUTION

Cup pick & place

Cup pick & place · View 1 · RGB observation

Full-task success is reported as mean ± standard deviation over three seeds, with 30 trials per seed. Models are frozen during real-world evaluation. The 60-second robot-execution budget excludes inference and search time.

Multi-view RGB-D rollout sequences

Inspect the paper’s frame sequences alongside the videos. RGB frames appear on the left and depth frames on the right; time advances left to right, then top to bottom.

Multi-view RGB-D rollout of cup picking and placing, viewpoint 1. — **Cup picking and placing · View 1.** Recorded RGB-D observations during real-robot manipulation.

CONTROLLED ABLATIONS

What makes the shared state effective?

Within-model interventions isolate geometry, information sharing, and temporal supervision. L1 and cosine aggregate three prediction tasks; success rate aggregates 50 Meta-World tasks and two real-world tasks, following the paper’s ablation protocol.

Ablation of depth content and metric geometry. — **Geometry-grounded state.** Depth content and metric coordinates both contribute. Combining them raises success from 76.8% to 83.4% and reduces L1 from 0.408 to 0.340.

Ablation of feature sharing and spatial readout. — **Sharing and spatial readout.** Removing private decoding increases L1 from 0.340 to 0.390, with success at 83.2% versus 83.4%. Private paths primarily improve visual readout.

Ablation of temporal prediction and physical grounding. — **Temporal and physical supervision.** Removing future-state matching lowers success from 83.4% to 79.5%. Removing FK regularization gives 82.5%, with direct physical labels retained.

Scope and future directions

Real-world evaluation uses calibrated fixed external cameras, embodiment-specific training, and matched indoor workspaces for drawer and cup manipulation. Moving cameras, zero-shot transfer across embodiments, broader workspace changes, and larger displacements with newly visible surfaces remain directions for further study.

The temporal comparisons measure forecasting conditioned on recorded actions and proprioception. Control rollouts instead use predicted proprioception. Feature accuracy alone does not establish precise physical trajectories or action sensitivity in moving regions.

BibTeX

@misc{liang2026depstatejepa,
  title = {DepState-JEPA: Learning Depth-Grounded
    State Representation for Visual Perception
    and Manipulation},
  author = {Liang, Xiujian and Na, Liuyuan and
    Du, Zehao and Xu, Jiacheng and Song, Weixi and
    Sun, Mingyang and Ma, Xingjun and Lu, Cewu and
    Sun, Jianhua},
  year = {2026},
  eprint        = {2610.29246},
  archivePrefix = {arXiv},
  primaryClass  = {cs.RO},
  url           = {https://arxiv.org/abs/2610.29246}
}

Download BibTeX ↓

Understand the scene.Anticipate the change.