acceptodds
Under review as a conference paper at ICLR 2027

TimeValue: Scaling Robotic Value Foundation Models with Temporal Distance

Abstract

General-purpose reward models are increasingly the bottleneck for scaling robot learning, yet the recipe for learning value-related capabilities from large-scale heterogeneous corpora remains underexplored. Existing approaches tie supervision to task-internal anchors such as preferences or normalized progress, none of which transfer cleanly across embodiments and data sources. We introduce TimeValue, a value foundation model for robotic manipulation that replaces these anchors with temporal distance, the directed cost-to-go from an observation to the language-specified goal. Because temporal-distance labels can be derived directly from timestamps, TimeValue scales to over 7,000 hours and roughly 3M instruction-conditioned clips without preference or progress annotations. To make temporal-value learning reliable at scale, we combine random temporal sampling, temporal-order shuffling, and value-isolation attention, suppressing shortcuts that would leave predictions insensitive to failures and regressions. Trained without preference labels, TimeValue attains an average Kendall's of 0.704 on RBM-EVAL-OOD, surpassing the fully preference-supervised state of the art (0.655) and more than doubling a progress-only counterpart (0.292), while generalizing zero-shot to unseen tasks, embodiments, and viewpoints. As a zero-shot reward model, TimeValue serves a range of downstream applications. Converted into dense rewards via potential-based shaping, it raises real-world policy success from 52.5% to 72.5% online and from 63.8% to 82.5% offline; used for data filtering, it improves multi-task behavior cloning success from 35.0% to 42.5%; and applied as inference-time value guidance, it lifts a frozen policy's success from 67.5% to 80.0%. These results establish temporal distance as a scalable supervision target and practical reward interface for generalist robot policies.

Primary area:
reinforcement learning
Keywords:
Embodied AI;Reinforcement Learning;Value Foundation Models

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_pFgcOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents TimeValue, a language-conditioned robotic value model trained on large-scale heterogeneous data using timestamp-derived temporal-distance supervision. The approach combines absolute and relative temporal prediction with random temporal sampling, temporal-order shuffling, value-isolation attention, and auxiliary language supervision. The resulting model is evaluated through out-of-distribution trajectory ranking, real-world policy post-training, demonstration filtering, and inference-time action selection. Overall, I consider this a substantial empirical contribution with practical value, although its conceptual novelty is relatively limited. My main concern is whether the experiments sufficiently isolate the benefit of absolute time-to-go supervision over normalized progress. My current recommendation is weak rejection, but I am open to raising my score if these concerns are adequately addressed during the rebuttal phase.

Critical strengths

Broad evaluation of practical utility. The evaluation covers both dataset-level trajectory ranking and downstream robotic applications, including online and offline reinforcement learning, data filtering, and inference-time value guidance. This provides meaningful evidence that the learned model can be used in a zero-shot manner for various purposes.

A scalable and useful training recipe. Learning from timestamps and completion boundaries reduces dependence on explicit preference annotation. Applying this approach to a heterogeneous mixture of over 7,000 hours and approximately three million segments is a valuable contribution toward reusable robotic value models.

Critical weaknesses

The central comparison between absolute time-to-go and normalized progress needs stronger isolation. Given that replacing normalized progress estimation with temporal-distance supervision is the first item in the contribution list, I would expect a clean ablation supporting this argument in the main body. I acknowledge that the paper includes Robometer and other progress-based reward models as baselines. However, TimeValue could outperform these methods because of a larger, higher-quality training dataset. I am not sufficiently convinced of the advantage of cost-to-go supervision itself.

The conceptual novelty is limited. Time-to-go value learning has precedents in policy improvement. For example, π0.6∗\pi_{0.6}^{*} learns values corresponding to negative remaining steps for successful episodes, with failure penalties and task-level normalization. TimeValue’s use of absolute seconds and its emphasis on a transferable value model are meaningful distinctions, but the underlying temporal-value interpretation is not new. The proposed shortcut-suppression mechanisms also appear necessary and highly intuitive when predicting time-to-go autoregressively. I do not think they provide much deeper insight.

Questions for authors
  • The time-to-go label decreases as a demonstration proceeds, even if the robot makes a mistake along the way. If the robot drops an object halfway through a trajectory, what teaches the model that this is a regression rather than progress?

  • What does TimeValue filter out in the Drawer task? The drop from 40% to 0% is quite substantial. The paper attributes this to limitations of the camera view, but could the filtering also be removing necessary actions that do not immediately make progress, such as repositioning before grasping?

  • Why is inference-time value guidance evaluated only on Bread and Steak? I would be interested in whether it also helps on Drawer and Bimanual, where judging the outcome of an action may require more precise spatial or coordination information.

  • I found an ablation in Table 9 where, under causal attention, normalized progress achieves 0.667 and absolute time with fixed bins achieves 0.675—a fairly small difference. However, these results are still confounded. Could you provide a more direct comparison in the main body to demonstrate the advantage of predicting time-to-go? Can you provide deeper insight on why TimeValue advance Robotmeter?

4 comments
AuthorsOpenReview ↗

Response to Reviewer PFgc Weaknesses

We thank Reviewer PFgc for the constructive review and for recognizing the breadth of our empirical evaluation. Your feedback helped us sharpen the formal problem definition and Monte Carlo supervision analysis in Section 3 and Appendix G, as well as expand the controlled comparisons in Table 2 and Appendix C.2 (Tables 9–12).

W1. Isolating the contribution of time-to-go supervision from data scale and architecture

We completely agree that cleanly isolating temporal-distance (cost-to-go) supervision from normalized progress is central to our primary contribution. In the revised manuscript, we have placed a direct, unconfounded ablation of the supervision objective into the main body (Section 5, Table 2), holding the training dataset, 8B architecture, and training recipe strictly fixed. Furthermore, because the processed RBM-1M release lacks the original frame-to-physical-time mappings required to construct targets in physical seconds (and original Robometer also relies on trajectory-preference pairs), we pair this main-body ablation with two additional controlled studies in Appendix C.2 (Tables 9–12) to rule out dataset size and quality as confounding factors:

1. Clean main-body ablation of cost-to-go supervision vs. normalized progress. Holding the training data, 8B backbone architecture, and training recipe strictly fixed, replacing normalized progress with quantile-binned temporal-distance (cost-to-go) supervision improves Mean Kendall's τa\tau_a from 0.6670.667 to 0.7040.704:

SupervisionMean Kendall's τa\tau_a ↑\uparrow
Normalized progress0.6670.667
Temporal distance, quantile bins0.7040.704

2. Ruling out dataset size. When subsampling our corpus without replacement to match RBM-1M using shared-source embodiment quotas while preserving instruction coverage, TimeValue-4B (trained with its standard recipe; Appendix C.2, Tables 10–12) still outperforms original Robometer:

ModelMean Kendall's τa\tau_a ↑\uparrow
Robometer (original)0.6550.655
TimeValue-4B (matched datasets)0.66330.6633

3. Ruling out dataset quality alone. Retraining the progress-only variant of Robometer within our framework using normalized-progress and success losses (excluding preference pairs), while retaining its original progress tokens, prediction heads, and image preprocessing, yields substantially lower ranking alignment than TimeValue on the same dataset:

ModelTraining settingMean Kendall's τa\tau_a ↑\uparrow
Robometer-4BProgress/success model retrained on our data pipeline0.55930.5593
TimeValue-4BFull TimeValue training0.6790.679
TimeValue-8BFinal TimeValue model0.7040.704

Together, the clean main-body ablation (0.7040.704 vs. 0.6670.667 under identical data and architecture) and the two dataset-controlled studies (0.66330.6633 vs. 0.6550.655 at matched scale; 0.6790.679/0.7040.704 vs. 0.55930.5593 on the identical data pipeline) provide direct, unconfounded evidence that cost-to-go supervision itself—rather than dataset size or quality alone—is a primary driver of TimeValue's advantage.

W2. Conceptual novelty relative to prior time-to-go value learning

Distinction in formulation. While π0.6∗\pi^*_{0.6} (RECAP) also adopts Monte Carlo time-to-go targets, its successful-episode targets are defined as negative remaining control steps normalized by each task's maximum episode length, paired with a terminal failure penalty (Sections IV-A and V-C). By contrast, TimeValue predicts expected remaining duration in physical seconds over a unified, dataset-level quantile support; our supervision is extracted directly from timestamped demonstrations with identified completion cutoffs, completely eliminating task-specific horizon normalization.

Distinction in role and zero-shot transferability. More fundamentally, our value model serves a distinct paradigm. RECAP fine-tunes a task-specific critic on in-domain deployment rollouts to compute advantages for policy iteration. In contrast, TimeValue is built as a generalist, zero-shot value foundation model: we establish a scalable training recipe over large-scale, heterogeneous multi-embodiment corpora so that a single frozen checkpoint transfers zero-shot to unseen tasks and embodiments without target-domain fine-tuning.

Necessity of shortcut suppression. Finally, while random temporal sampling, order shuffling, and value isolation are clean and modular by design, they are indispensable for scaling multi-frame value learning: removing temporal shuffling alone causes mean Kendall's τa\tau_a to collapse from 0.7040.704 to 0.1590.159 (Table 2), demonstrating that naive autoregressive training fails to learn transferable value representations without explicit shortcut suppression.

Q1. How the model detects regression (e.g., dropping an object) when per-rollout labels always decrease

Intuition. Holding an object near its goal state typically precedes immediate placement, whereas dropping the object requires re-grasping and recovery. Because the pre-drop state is predominantly followed by direct completion while the post-drop state entails a prolonged recovery phase, the dropped-object state exhibits a strictly larger expected remaining time across the demonstration distribution. Because TimeValue estimates the conditional expectation of remaining time given the observed state across the dataset—rather than memorizing individual trajectory indices—it naturally assigns a higher time-to-go (lower value) upon task regression, even though recorded timestamps decrease monotonically within each individual rollout.

Formal analysis (Appendix G.3). Formally, TimeValue estimates dˉ(s)=ED[Y∣S=s]\bar{d}(s) = \mathbb{E}_{\mathcal{D}}[Y \mid S = s], where YY denotes the empirical time-to-go. Appendix G.3 analyzes transitions from a pre-mistake state aa, where demonstrations follow either the direct expert branch a→ba \to b or, with probability ρa\rho_a, the mistake branch a→m→ba \to m \to b, and follow the expert policy elsewhere (both branches ultimately succeed and share the continuation from the recovered state bb). Let hEh_{\mathrm{E}}, hSh_{\mathrm{S}}, and rr denote the expected durations of the direct transition a→ba \to b, the mistake transition a→ma \to m, and the recovery segment m→bm \to b, respectively, where true regression implies r>hEr > h_{\mathrm{E}}. The difference in expected remaining time between the mistake state mm and the pre-mistake state aa is:

dˉ(m)−dˉ(a)=(1−ρa)(r−hE)−ρahS,\bar{d}(m) - \bar{d}(a) = (1 - \rho_a)(r - h_{\mathrm{E}}) - \rho_a h_{\mathrm{S}},

which is strictly positive whenever the mistake probability satisfies:

ρa<11+hSr−hE.\rho_a < \frac{1}{1 + \frac{h_{\mathrm{S}}}{r - h_{\mathrm{E}}}}.

Under this natural distributional condition, the population regression target increases sharply upon entering the mistake state mm and decreases as recovery progresses, proving that conditional expectation over monotonic rollout labels faithfully captures non-monotonic task progress. Figure 11 (Appendix G.3) illustrates this mechanism schematically, and Figure 5 (Appendix C.4) demonstrates it empirically on real trajectories.

Q2. What is filtered out in the Drawer task and whether necessary preparatory/corrective actions are removed

Our quantitative audit in Section 6 and Appendix D.5 identifies two primary factors driving the drop in Drawer success from 4040 to 00 under offline data filtering:

1. Reduced state coverage. Advantage-based filtering prunes a substantial fraction of candidate training transitions, narrowing state-space coverage on precision-critical tasks. Under our filtering configuration (window W=50W = 50, discount λ=0.98\lambda = 0.98, retention criterion At>0A_t > 0), the filter retains 25,68925,689 of 46,03846,038 candidate action-chunk starts on Drawer and removes 20,34920,349 (44.244.2). Across all four tasks, the frame-weighted removal rates over all source episodes (spanning both successful and unsuccessful trajectories) are:

TaskCandidate training starts removed (%)
Bread Basket Placement33.933.9
Steak Serving with a Spatula37.137.1
Box-in-Drawer Placement44.244.2
Bimanual Box Transfer37.137.1

2. Removal of retry and recovery segments. Confirming your intuition, filtering disproportionately prunes retry attempts and corrective adjustments that teach the policy how to recover from initial misalignment. In our case study of a successful Drawer demonstration containing two missed grasp attempts prior to a successful grasp and lift (Appendix D.5, Figure 10), the filter removes 187187 of 286286 candidate starts (65.465.4) across the two unsuccessful attempts and re-approach phases, while retaining 135135 of 138138 starts (97.897.8) during the final successful grasp-and-lift execution. Discarding these corrective segments deprives the imitation policy of essential recovery behaviors, aligning with classic findings on corrective demonstrations.

Q3. Why inference-time value guidance is evaluated only on Bread and Steak

As noted in Section 6, the frozen Cosmos3-Edge-2B world-model proposal policy achieves 7575 unguided success on Bread and 6060 on Steak, but 00 on Drawer and Bimanual despite joint multi-task fine-tuning across all four tasks. We therefore evaluated inference-time value guidance on the two tasks where the proposal distribution generates viable trajectories. With the proposal model held completely fixed, TimeValue guidance boosts mean success from 67.567.5 to 80.080.0—outperforming Robometer guidance (72.572.5) and demonstrating that TimeValue accurately selects superior candidate rollouts at test time without policy updates.

Because inference-time value guidance operates by re-ranking four sampled action/video proposals, its performance is bounded by the support of the proposal distribution: a value model cannot synthesize a successful trajectory if all candidate proposals fail. Since the unguided base model yields 00 success on Drawer and Bimanual, valid candidate rollouts rarely appear among the sampled proposals. Extending value guidance to these high-precision tasks alongside stronger world-model proposal backbones is a promising direction for future work.

Q4. Why Table 9 shows a small gap (0.0080.008) for fixed bins, and deeper insight into why TimeValue advances over Robometer

Why fixed bins yield only a +0.008+0.008 gain over normalized progress. Uniform (fixed) binning is poorly matched to the heavy-tailed distribution of physical remaining times, resulting in severe support under-utilization: the majority of the 256 uniform support centers fall in sparsely populated long-duration tail regions where few training clips reside, whereas the sub-50 s50\text{ s} horizon—where the vast majority of manipulation interactions occur—is compressed into roughly 20 coarse bins. This coarse discretization blunts gradient resolution and limits fine-grained temporal discrimination. By contrast, quantile binning allocates 217 of the 256 support centers (84.884.8) to the critical 0–50 s0\text{–}50\text{ s} regime (Tables 7–8), unlocking the full benefit of physical temporal-distance supervision (0.7040.704 vs. 0.6750.675).

HorizonQuantile support centers (cumulative)Share of all support centers
≤1 s\le 1\text{ s}197.47.4
≤5 s\le 5\text{ s}7529.329.3
≤10 s\le 10\text{ s}11946.546.5
≤20 s\le 20\text{ s}16564.564.5
≤50 s\le 50\text{ s}21784.884.8
≤100 s\le 100\text{ s}24194.194.1
≤200 s\le 200\text{ s}25499.299.2

Why TimeValue advances over Robometer. We present our controlled, unconfounded comparisons directly in Table 2 and Section 5 (with full breakdowns in Appendix C.2), demonstrating that TimeValue's superiority arises from three synergistic pillars:

1. Cross-dynamics formulation and heterogeneity handling. Episode-normalized progress requires well-defined trajectory start points and rescales every trajectory to [0,1][0, 1] by its total duration, discarding physical time-to-go and rendering identical remaining task segments inconsistent across rollouts of differing initial distances or idle pauses. In contrast, our cross-dynamics temporal-distance formulation (Section 3; Appendix G) predicts remaining cost-to-go in physical seconds conditioned on embodiment metadata, preserving suffix invariance and naturally accommodating diverse robot morphologies, control frequencies, and actuation speeds.

2. Scalable training recipe and shortcut-free architecture. Scaling a generalist robotic value foundation model requires a low-cost supervision signal that extracts dense value structure from heterogeneous corpora in the wild. Because physical temporal-distance targets require only frame timestamps and completion cutoffs—avoiding costly human preference annotations or paired failure trajectories—our objective scales seamlessly across real-robot demonstrations, simulation rollouts, and egocentric human videos. Architecturally, TimeValue combines random temporal sampling and order shuffling with training-time value isolation and inference-time causal attention, eliminating sequence-order shortcuts and enforcing visually grounded value estimation.

3. Consistent empirical superiority. Under strictly matched data, architecture, and training recipes, quantile-binned temporal distance elevates Mean Kendall's τa\tau_a from 0.6670.667 to 0.7040.704, translating directly into substantial downstream gains across offline IQL policy learning, online DSRL sample efficiency, and inference-time value guidance.

Dear Reviewer pFgc,

Thank you again for your careful reading. We have revised the manuscript in response to your comments. The following pointers may help you find the changes.

  • W1 — Temporal distance versus normalized progress. The dark green row and discussion in Section 5, Table 2, bring the supervision-target comparison into the main text. Appendix C.2, Tables 9–12, gives the detailed ablations and the composition of the matched training subset.
  • W2 — Contribution relative to prior time-to-go methods. The purple closing sentence of Temporal distance learning in Section 2 clarifies our contribution: learning in physical seconds across heterogeneous datasets and transferring a frozen value model across tasks and embodiments. The purple Monte Carlo supervision paragraph in Section 3 defines the target.
  • Q1 — How decreasing timestamp labels can reveal a setback. The purple analysis in Appendix G.2–G.3, particularly Figure 11, illustrates how a mistake can increase expected remaining time even though the labels along each recorded trajectory decrease. Appendix C.4, Figure 5, shows a corresponding example on a real trajectory.
  • Q2 — What filtering removes on Drawer. The purple discussion in Section 6 and Appendix D.5 examines missed grasps, re-approaches, and placement. Figure 10 shows which parts of two demonstrations were removed, and Table 15 gives the removal rate for each task.
  • Q3 — Why value guidance was evaluated on Bread and Steak. The purple sentence in Section 6, under Inference-time value guidance, explains the task selection and reports the unguided policy’s 0% success on Drawer and Bimanual.
  • Q4 — The small fixed-bin gain and the role of quantile binning. The dark green Value target and tokenization paragraph in Appendix C.2 discusses the three results in Table 9: 0.667 for normalized progress, 0.675 for fixed-bin temporal distance, and 0.704 for quantile-bin temporal distance. Appendix B.2, Tables 7–8, shows how the quantile bins are allocated.

When you have had a chance to read the revision, we would be grateful to hear whether these additions address your concerns.

Reviewer_juTLOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper introduces TimeValue, a value foundation model for robotic manipulation that predicts temporal distance to language-specified task completion. The model is trained on a curated dataset comprising approximately 3M instruction-conditioned trajectory segments. To reduce shortcuts such as correlations between frame position and task progress, the method combines temporal-order shuffling, irregular temporal sampling, and value-isolation attention. On RBM-EVAL-OOD, TimeValue achieves the highest average trajectory-ranking score among the evaluated models, with ablations showing that each component contributes to overall performance. Further experiments demonstrate its utility for downstream applications, including online and offline RL, data filtering for behavior cloning, and inference-time value guidance.

Critical strengths

The paper is well-written with a clear description of the technical approach and architecture. It provides details on data curation, architecture, and training, with thorough discussion on each design choice. The overall approach seems technically sound.

The results are overall thorough and convincing. TimeValue shows overall strong performance, and the ablation study clearly demonstrates the contribution of each component. It also demonstrates that TimeValue is effective for downstream applications such as RL and filtered IL on real robot hardware, with the inclusion of negative results.

Critical weaknesses

Although the ablation study shows the contribution of each component, e.g., temporal-order shuffling, irregular temporal sampling, and value-isolation attention, it doesn’t fully ablate the effect of the dataset. In other words, it is unclear whether TimeValue is outperforming RoboMeter due to a better algorithm & architecture, or it is because it’s trained with more and better data, or both. An additional study with a more comparable data source (processed with temporal-order shuffling, irregular temporal sampling included) to RBM-1M would be ideal.

Questions for authors
  • What is the main consideration behind using absolute time instead of normalized progress? How would TimeValue handle data from the same tasks but with different execution speed?

  • How does TimeValue handle tasks with repetitive nature?

  • For the ablation study, is “w/o Shuffle” referring to using a forward-only sampling strategy (without rewind)?

  • For the value guidance experiment, why does MT-IQL do better than VG in success rate, but does worse in terms of average steps?

5 comments
AuthorsOpenReview ↗

Response to Reviewer JuTL W1

We thank Reviewer JuTL for the thorough review and for recognizing the clarity of our presentation, our real-world evaluation, and our transparent analysis of failure cases. We have revised the manuscript to expand our controlled comparisons with Robometer and clarify the cross-dynamics temporal formulation, the sampling ablations, and the execution-length metric.

W1. Disentangling the effects of algorithm/architecture versus data scale and quality

To directly answer whether TimeValue's gains stem from our algorithm and architecture, our dataset, or both, we conduct two targeted dataset-and-architecture studies that decouple data source and scale from model architecture and algorithm (Section 5, Table 2; Appendix C.2, Tables 10–12). Because the released RBM-1M dataset consists of pre-sampled frame sequences without the original frame-to-physical-time mappings required to construct targets in physical seconds—and original Robometer additionally relies on trajectory-preference supervision—directly training TimeValue on the processed RBM-1M files is not feasible. Instead, following your exact recommendation, we construct an RBM-1M-comparable data split processed with our full pipeline (including irregular temporal sampling and temporal-order shuffling), alongside a controlled architecture evaluation on our full data pipeline:

1. Comparable data source and scale to RBM-1M (processed with irregular temporal sampling and temporal-order shuffling). Directly addressing your suggestion, we subsample our corpus without replacement to match the scale and shared-source embodiment quotas of RBM-1M while preserving instruction coverage (Appendix C.2, Tables 10–12). Trained on this RBM-1M-matched dataset with our full recipe (including irregular temporal sampling and temporal-order shuffling), TimeValue-4B continues to outperform original Robometer:

ModelMean Kendall's τa\tau_a ↑\uparrow
Robometer (original, trained on RBM-1M)0.6550.655
TimeValue-4B (RBM-1M-matched dataset, w/ irregular sampling & shuffling)0.66330.6633

2. Model architecture and algorithm on the identical data pipeline. Conversely, retraining the progress-only variant of Robometer within our framework on the same data pipeline, using normalized-progress and success losses (excluding preference pairs) while retaining its native progress tokens, prediction heads, and image preprocessing, yields substantially lower ranking alignment than TimeValue:

ModelTraining settingMean Kendall's τa\tau_a ↑\uparrow
Robometer-4BProgress/success model retrained on our data pipeline0.55930.5593
TimeValue-4BFull TimeValue training on the same data pipeline0.6790.679
TimeValue-8BFinal TimeValue model on the same data pipeline0.7040.704

Together, these two studies confirm that our algorithmic and architectural designs are the primary drivers of performance, while also scaling effectively with larger data: (i) on a dataset matched to RBM-1M in scale and source composition (processed with irregular sampling and temporal shuffling), TimeValue-4B outperforms original Robometer (0.66330.6633 vs. 0.6550.655); and (ii) training Robometer on our dataset fails to match TimeValue (0.55930.5593 vs. 0.6790.679/0.7040.704), demonstrating that data alone cannot explain the improvement without TimeValue's objective and architecture.

AuthorsOpenReview ↗

Q1&Q2&Q3&Q4

Q1. Rationale for absolute time and handling varying execution speeds

We formulate value learning as cross-dynamics temporal-distance estimation (Section 3; Appendix G.1). Physical seconds establish a unified unit across diverse embodiments and control frequencies while naturally allowing remaining durations to vary under each robot's dynamics PeP_e. This preserves the true physical time-to-completion required to achieve the instructed goal, whereas episode-normalized progress distorts temporal structure by rescaling every rollout by its total duration.

To account for varying execution speeds, we explicitly encode robot embodiment and actuation profiles (such as morphology and nominal execution speed) into the metadata mem_e provided to the shared estimator vθ(ci,ℓ,me)v_\theta(c_i, \ell, m_e), while the multi-frame context cic_i captures real-time kinematic cues from the ongoing rollout. Because the expected remaining-time target is defined per embodiment–instruction pair (e,ℓ)(e, \ell) over demonstrated trajectories (Appendix G.2), execution-speed variations within a task are systematically resolved through both explicit metadata conditioning and the multi-frame visual context.

Q2. Handling tasks with repetitive actions

Multi-frame visual history and persistent scene changes readily distinguish repeated task stages whenever intermediate actions alter the environment—for example, observing whether the first piece of bread has already been placed inside the basket. TimeValue achieves a 100100 offline RL success rate on our two-piece Bread Basket Placement task (Table 3), demonstrating strong capability on multi-stage repetitive tasks with observable state transitions.

For purely cyclic actions that leave no persistent visual trace (e.g., stirring a pot a fixed number of times), a finite observation window cannot recover the exact cycle count once earlier repetitions pass outside the context window. Addressing such unobservable counting tasks requires streaming recurrent memory or explicit state tracking—a fundamental partial-observability constraint that applies equally to downstream VLA policies when visual observations do not disambiguate cycle counts.

Q3. Definition of "w/o Shuffle" in the ablation study

Yes. The "w/o Shuffle" ablation disables temporal-order shuffling and backward rewinds, presenting frames in strictly chronological order while retaining irregular timestamp sampling. Removing temporal shuffling causes mean Kendall's τa\tau_a to drop sharply from 0.7040.704 to 0.1590.159 (Table 2), confirming that temporal-order perturbations are essential for preventing sequence-position shortcuts and ensuring strong ranking accuracy.

Q4. Why MT-IQL achieves higher success rate but higher Avg. Steps than VG

We include MT-IQL as an optimization-based reference point alongside training-free inference-time value guidance (VG). MT-IQL leverages TimeValue reward shaping to directly update policy parameters via gradient optimization, whereas VG steers action selection solely at inference time while keeping both the base policy and value model frozen. Because MT-IQL refines the underlying policy weights, it naturally achieves higher task completion rates than test-time re-ranking over a frozen policy's candidate proposals.

On Bread and Steak, MT-IQL reaches 9090 and 8080 success compared with 8585 and 7575 for VG (TimeValue), while averaging 34.334.3 and 20.820.8 action chunks versus 30.430.4 and 16.416.4 (Figure 3b–c). Crucially, Avg. Steps is evaluated strictly over successful rollouts. Because MT-IQL succeeds on challenging initializations and recovery-heavy trials that require additional corrective steps—trials on which VG fails and is therefore excluded from the step-count average—its average rollout length reflects broader task coverage rather than slower execution on easy trials. Higher success rates and slightly longer average successful rollouts are thus fully consistent.

Reviewer_juTLOpenReview ↗

Thank you for your responses

Thank you for the additional discussions and clarifications, and for the effort to disentangle the effects of the training data and method. It would be helpful to clarify how the extracted subset compares with RBM-1M in terms of data sources and task coverage. I also note that the retrained RoboMeter baseline omits preference supervision, so this comparison only partially isolates the effects of the data and method.

Overall, I believe the paper makes sufficient contributions and provides a thorough empirical study to warrant acceptance. I therefore maintain my original positive rating.

Reviewer_GeEsOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes TimeValue, a language-conditioned value model for robotic manipulation that learns to predict the temporal distance to task completion. Unlike prior approaches that use normalized progress or preferences for supervision, the authors claim that using the temporal distance objective combined with several other tricks can lead to better performance. In their experiments, TimeValue, which was trained on 7000 hours of robot data, outperforms prior reward models on the RBM-EVAL-OOD benchmark. They also show that the trained value model can be used for online/offline RL to actually improve the policy.

Critical strengths

The paper studies an important problem in scaling general-purpose value models for robotics. The idea of using temporal distance as a scalable supervision signal is simple yet interesting. The authors also demonstrate that the learned value model can improve policy performance through real-world RL, which is promising.

Critical weaknesses

W1. While the paper claims the advantage of temporal-distance supervision over normalized progress, it is still unclear how much of the performance improvement comes from the objective itself versus the large-scale training data and architecture. Can the authors provide an apple-to-apple comparison between the two objectives under identical training settings?

W2. The offline RL experiments use a dataset with only 12 failed trajectories. How were these collected? It would be interesting to evaluate the method on datasets with more diverse levels of expertise and a larger proportion of failed trajectories.

Questions for authors

Q1. For the online RL experiments, could the authors provide the learning curve? I am curious to see how TimeValue improves sample efficiency compared to Robometer and sparse rewards.

Q2. Appendix C.2 shows that using causal attention at inference achieves better performance than value-isolation attention. Could the authors explain why this mismatch between training and inference improves performance?

Q3. For the online RL experiment, different reward-shaping coefficients are used for TimeValue and Robometer. How are these values chosen and tuned? I would recommend adding an ablation over these values to see if the performance gain comes from the quality of the learned reward rather than the choice of reward magnitude.

4 comments
AuthorsOpenReview ↗

Response to Reviewer GeEs W1&W2

We thank Reviewer GeEs for recognizing the value of scalable reward supervision and its applicability to robot learning. We have revised the manuscript to thoroughly address your questions regarding supervision targets, data collection, online learning dynamics, attention constraints, and reward scaling.

W1. Disentangling the contributions of temporal supervision, training data, and model architecture

We train on our timestamp-preserving corpus because the processed RBM-1M release lacks the original frame-to-physical-time mappings required to construct targets in physical seconds, and original Robometer additionally relies on trajectory-preference supervision that TimeValue does not use. Directly swapping processed datasets would therefore confound both temporal labels and supervision signals. To rigorously isolate temporal supervision, training data, and model architecture, we conduct three orthogonal controlled comparisons (Section 5, Table 2; Appendix C.2, Tables 9–12):

1. Supervision objective. Holding the training data, 8B backbone architecture, and training recipe strictly fixed (Table 2; Appendix C.2, Table 9), replacing normalized progress with quantile-binned temporal distance elevates Mean Kendall's τa\tau_a from 0.6670.667 to 0.7040.704:

SupervisionMean Kendall's τa\tau_a ↑\uparrow
Normalized progress0.6670.667
Temporal distance, quantile bins0.7040.704

2. Training data scale. When subsampling our corpus without replacement to match RBM-1M using shared-source embodiment quotas while preserving instruction coverage, TimeValue-4B continues to outperform original Robometer:

ModelMean Kendall's τa\tau_a ↑\uparrow
Robometer (original)0.6550.655
TimeValue-4B (matched datasets)0.66330.6633

3. Model architecture. Retraining the progress-only variant of Robometer within our framework using normalized-progress and success losses (excluding preference pairs), while retaining its original progress tokens, prediction heads, and image preprocessing, yields substantially lower ranking alignment than TimeValue under the same data pipeline:

ModelTraining settingMean Kendall's τa\tau_a ↑\uparrow
Robometer-4BProgress/success model retrained on our data pipeline0.55930.5593
TimeValue-4BFull TimeValue training0.6790.679
TimeValue-8BFinal TimeValue model0.7040.704

W2. Collection of the 12 failed trajectories and coverage of expertise levels

The data collection protocol and per-task breakdown are detailed in Appendix D.1 (Table 13). The 12 terminal failures were collected alongside the 398 successful demonstrations during the same human-teleoperation sessions.

Collection setup and outcome labels. A human operator used an SO-100 leader arm to teleoperate the Franka robot on our dual-arm platform without manual physical intervention in the scene. For each task, we targeted roughly 100 successful demonstrations and retained all naturally occurring unsuccessful episodes from the same sessions. Operators assigned binary outcome labels at the end of each rollout, marking an episode as unsuccessful whenever scene disruption prevented task completion. The failure count thus reflects the natural outcome rate of our teleoperation protocol rather than a prescribed success-to-failure ratio.

Suboptimal attempts and recovery. An isolated missed grasp or suboptimal intermediate motion did not automatically render an episode a failure. Whenever the operator corrected the trajectory, retried, and completed the task, we retained the entire rollout—including all unsuccessful attempts and recovery segments—and labeled it successful. For example, one successful Drawer demonstration contains two failed grasp attempts prior to a successful grasp and lift (Appendix D.5). The 12 terminal failures therefore understate the volume of suboptimal and corrective behavior present in the dataset, and both terminal successes and failures are included in the shared offline IQL dataset across all reward variants.

Coverage of expertise. While this dataset captures natural variation in execution quality both within and across episodes, operators were not explicitly stratified by skill level, nor were failure ratios systematically varied. Evaluating value learning across controlled demonstrator expertise levels and higher failure proportions would require collecting a new dataset, which entails a substantial time investment. We therefore leave this evaluation to future work.

Q1. Online RL learning curves and sample efficiency

Figure 8 (Appendix D.4) plots cumulative successful training trajectories against environment interaction steps during online DSRL:

TaskInteraction budgetTimeValueRobometerSparse Reward
Steak Serving with a Spatula12,000 steps403124
Bimanual Box Transfer22,000 steps393629

Each curve tracks a single training run, including warm-up collection, with successes recorded upon episode completion. TimeValue completes substantially more successful episodes under both fixed budgets, achieving 29.029.0 more completions than Robometer and 66.766.7 more than Sparse Reward on Steak Serving with a Spatula. Because faster successful rollouts permit more episodes within a fixed step budget, this metric jointly captures sample efficiency and execution speed.

To examine the learned value signals qualitatively, Figure 9 aligns the predicted QQ-value at the 6,000-step checkpoint with side- and wrist-camera keyframes from a successful Steak rollout. Between 3.00 s3.00\text{ s} and 6.00 s6.00\text{ s}—as the robot repositions its gripper near the spatula handle, lifts the steak clear of the pan, and transports it toward the plate—QQ increases by +5.63+5.63 under TimeValue, whereas it decreases by −5.67-5.67 under Robometer and −8.51-8.51 under Sparse Reward. TimeValue is the only critic that positively rewards this decisive task progress.

Q2. Why training with value isolation and inferring with causal attention improves performance

Thank you for raising this insightful question. The performance gain arises from the complementary roles that attention constraints play during representation learning versus inference-time context aggregation.

Mechanism. During training, value isolation prevents cross-query shortcuts, forcing each value query to ground its prediction directly in the multimodal visual–language observations. Once these grounded representations are established, lifting the isolation constraint at inference allows each query to attend to preceding value-query hidden states, which serve as compact, temporal-distance-supervised summaries of task history. This enables sharp state disambiguation without leaking future observations, aggregating historical progression more effectively than attending solely to raw visual tokens.

Connection to prior work. This training–inference asymmetry aligns closely with recent findings on structured attention constraints. MuToR (Gerontopoulos et al., 2025, §3.2) similarly isolates auxiliary register tokens during training to strengthen backbone representations (though it discards them at inference), and Seeing is Believing (Tan et al., 2025, §4.1) masks speech-token attention during training to enforce visual grounding, demonstrating that removing the mask at inference lowers perplexity from 6.316.31 to 5.655.65 at 3030 masking. Together, these findings validate pairing training-time value isolation with inference-time causal attention for superior ranking accuracy.

References

  • Gerontopoulos, A., Gidaris, S., & Komodakis, N. (2025). Multi-Token Prediction Needs Registers. Advances in Neural Information Processing Systems (NeurIPS 2025).
  • Tan, W., Lian, J., Inaguma, H., Tomasello, P., Koehn, P., & Ma, X. (2025). Seeing is Believing: Emotion-Aware Audio-Visual Language Modeling for Expressive Speech Generation. Findings of the Association for Computational Linguistics: EMNLP 2025.

Q3. Selection of reward-shaping coefficients and attribution of gains

In our main experiments, we fix κ=0.1\kappa = 0.1 for TimeValue and κ=1.0\kappa = 1.0 for Robometer across all four tasks in both offline IQL and online DSRL. This choice directly reflects the models' native output scales: TimeValue predicts remaining duration in physical seconds, whereas Robometer predicts normalized progress in [0,1][0, 1], meaning an identical numerical coefficient would induce mismatched shaping magnitudes relative to the shared terminal sparse reward. To verify that our performance gains stem from reward quality rather than coefficient tuning, we evaluated both models across κ∈0.1,1.0\kappa \in 0.1, 1.0 in offline IQL (Appendix D.3, Table 14):

TimeValue (main setting: κ=0.1\kappa = 0.1)

TaskSuccess (κ=0.1\kappa=0.1)Avg. Steps (κ=0.1\kappa=0.1)Success (κ=1.0\kappa=1.0)Avg. Steps (κ=1.0\kappa=1.0)
Bread Basket Placement100.0100.016.8±3.116.8 \pm 3.160.060.043.1±17.643.1 \pm 17.6
Steak Serving with a Spatula90.090.014.9±4.014.9 \pm 4.0100.0100.022.8±8.822.8 \pm 8.8
Box-in-Drawer Placement90.090.014.9±4.014.9 \pm 4.085.085.032.6±4.332.6 \pm 4.3
Bimanual Box Transfer50.050.033.6±10.533.6 \pm 10.50.00.0—
Mean success82.582.561.2561.25

Robometer (main setting: κ=1.0\kappa = 1.0)

TaskSuccess (κ=0.1\kappa=0.1)Avg. Steps (κ=0.1\kappa=0.1)Success (κ=1.0\kappa=1.0)Avg. Steps (κ=1.0\kappa=1.0)
Bread Basket Placement0.00.0—80.080.018.9±2.718.9 \pm 2.7
Steak Serving with a Spatula50.050.019.6±12.319.6 \pm 12.380.080.019.4±7.219.4 \pm 7.2
Box-in-Drawer Placement0.00.0—50.050.027.3±6.327.3 \pm 6.3
Bimanual Box Transfer0.00.0—45.045.028.7±9.328.7 \pm 9.3
Mean success12.512.563.7563.75

Increasing TimeValue's κ\kappa from 0.10.1 to 1.01.0 lowers mean success from 82.582.5 to 61.2561.25, while reducing Robometer's κ\kappa from 1.01.0 to 0.10.1 drops its mean success from 63.7563.75 to 12.512.5, confirming that each model is evaluated at its best-suited scale. Furthermore, in scale-invariant inference-time value guidance—where arg⁡max⁡\arg\max selection is completely invariant to positive linear scaling—TimeValue still decisively outperforms Robometer (80.080.0 vs. 72.572.5), confirming that our downstream gains derive from superior reward quality rather than reward magnitude.

Dear Reviewer GeEs,

Thank you again for your thoughtful questions. We have updated the manuscript and listed the relevant passages below for ease of reference.

  • W1 — Supervision target, training data, and architecture. The dark green additions in Section 5, Table 2, and Appendix C.2, Tables 9–12, cover the supervision-target ablation, the retrained Robometer baseline, and the comparison on the matched data subset.
  • W2 — Failed demonstrations and variation in operator experience. The blue Data Collection paragraph in Appendix D.1 explains how the 12 terminal failures were collected and how we labeled successful demonstrations that included failed attempts and recovery.
  • Q1 — Online RL learning curves. The blue discussion in Appendix D.4 accompanies Figure 8, which plots cumulative successful episodes against environment steps. Figure 9 provides a complementary example of how critic values evolve during a rollout.
  • Q2 — Training and inference attention masks. The blue Attention paragraph in Appendix C.2 discusses why relaxing value isolation at inference may help.
  • Q3 — Reward-shaping coefficients. The blue discussion in Appendix D.3 explains the choice of coefficients, and Table 14 reports the offline IQL ablation for both reward models.

When you have a chance to look at the revision, we would appreciate hearing whether it addresses your concerns. Thank you again for your time.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Revisions

Dear Program Chairs, Senior Area Chairs, Area Chairs, and Reviewers,

Thank you for your time and constructive feedback. The reviewers’ primary questions were mainly focused on disentangling the contributions of our supervision, architecture, and data scale, as well as requests for downstream RL ablations. We have fully addressed these concerns in our revised manuscript by providing detailed controlled comparisons, new offline RL ablations, and further clarifications.

We are very encouraged by the positive evaluations and recognition from two of our reviewers. We also note that the remaining reviewer explicitly indicated a willingness to raise their score if these primary concerns were resolved. We have provided all the requested experiments and clarifications, though we have not yet received their follow-up response.

We believe our revisions have substantially strengthened the manuscript, and we sincerely appreciate the time and effort the Area Chairs and Reviewers have dedicated to evaluating and guiding our work.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.