acceptodds
Under review as a conference paper at ICLR 2027

RoboValue: A Fine-Grained Sim-and-Real Benchmark for Unified Evaluation of Robotic Value Models

Abstract

General-purpose robotic value models detect task-state changes from visual observations and instructions to score task progress across tasks and embodiments. Yet existing benchmarks test only episode outcomes or isolated process abilities, through model-specific interfaces and mostly on clean successful or failed rollouts, and thus cannot tell whether a model tracks task state or exploits shortcuts such as elapsed time and visual similarity. We introduce RoboValue, a benchmark with a unified protocol for fine-grained evaluation of robotic value models on simulation and real-world tasks. Shared interfaces and adapters enable comparison across heterogeneous models while preserving native value semantics. RoboValue evaluates four complementary dimensions: task-state understanding, temporal progress monitoring, failure and recovery reasoning, and value consistency. Beyond common successful and failed rollouts, it includes diagnostic trajectories with incomplete subtasks, effective and ineffective recovery, visually similar states with different execution histories, and multiple valid action orders. We evaluate 15 model variants from 9 families in zero-shot and one-shot settings, assessing performance under standard conditions and generalization across embodiment and environment shifts. RoboValue provides a public leaderboard and a diagnostic foundation for developing more reliable and general robotic value models. We propose RoboValue-Dataset, which comprises an evaluation set of 2,792 trajectories spanning 15 simulation and 20 real-world manipulation tasks across 4 embodiments, with corresponding training data. The website is available at https://robovalue.github.io/.

Primary area:
datasets and benchmarks
Keywords:
Robotic Value Models, Robot Manipulation, Reward Modeling

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_hFZJOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

RoboValue is a proposed benchmark to evaluate diverse reward/value models for robotic manipulation tasks. It provides teleoperated trajectories from both simulation and real world tasks with different embodiments as the evaluation dataset covering both successful ones and failed ones with recovery. The benchmark also proposes additional evaluation metrics to better asses different reward/value models about their task-state understanding, temporal progress, failure/recovery reasoning, and value consistency, performance under environmental and embodiment shifts.

Critical strengths
  1. The paper is clear and easy to follow with details in the appendix. Incorporating the real failures and recovery into the value/reward model evaluation shows the benefit to evaluate those models more comprehensively.

  2. The analysis of the benchmark results across different models are thorough and shows the pros and cons of different models over different metrics.

Critical weaknesses
  1. The failed trajectories in the benchmark com from the teleoperation and not from actual policy rollout. Unlike human teleoperation which is instructed to fail in certain ways, failures from policy rollouts are more diverse or sometimes more nuanced. Therefore, evaluating only on the teleoperation trajectories may not provide a good proxy to the scenarios where the value function is actually used. Moreover, the paper doesn't evaluate any downstream applications of different reward/value models. For example, for finetuning with RL, it would be interesting to see if any of those metrics can show the correlation between reward quality vs the performance of downstream RL. This may provide more useful signals for the community to know which reward model to choose.
Questions for authors
  1. The episode-level score v is not defined in the paper. This is also critical for the comparison between trajectories. Can the authors clarify whether you are using max, sum, or discounted sum of the per-step value to determine the episode-level score?
  2. Would the authors also be able to evaluate some other simple heuristic baselines in the benchmark to help readers understand better what each metric is evaluating. For example, a random baseline and a baseline where reward is proportional to the timestep can help clarify why some metrics can better evaluate the models that learn a short-cut from the timesteps.
  3. It would be easier for reader to understand if the authors can provide a table to explicitly highlight the difference between the proposed benchmark vs prior benchmark. Also among a list of metrics proposed in the paper, some of them are borrowing from prior work (e.g., VOC). Therefore, it is also better to have a table to clarify which are the novel metrics different from prior work.
3 comments
AuthorsOpenReview ↗

Q1. How the episode score is computed

Thank you for asking how we turn frame-level values into an episode score and how the benchmark can inform reward-model selection. We revised both the main paper and the supplementary material and combined them in a single PDF.

We added evaluations of four general-purpose frontier models (GPT-6.1 Sol, Gemini 4 Argon, Claude Opus 5.5, and DeepSeek-V4.1-Flash), all with eight-frame inputs. Section 4.1 describes the 24 model configurations. Table 1 reports in-domain scores and Table 2 reports results under embodiment and environment shifts. We clarified episode scoring in Section 3.4 and Appendix K.1, added random, constant, time-only, and annotation-based reference curves in Appendix G.3 (Tables 10–11), and compared RoboValue with 23 prior benchmarks and evaluation suites in Appendix B (Table 4).

Q1. How do per-step values determine the episode-level score?

For Success Accuracy (SA), the episode score is the arithmetic mean of the sampled values over the final 2% of the episode duration. This small terminal window smooths fluctuations around the final state. Earlier values have zero weight. For example, a large peak when the robot grasps the lid cannot raise the score if the model assigns low values throughout the terminal window after the lid is dropped.

Within each task and evaluation setting, we compare every successful episode with every partially successful or failed episode. SA is the fraction of these pairs for which the successful episode has a strictly higher terminal mean. A tie earns zero credit.

Task Grounding Accuracy (TGA) uses a different comparison because its question is whether the same video fulfills a given instruction. We average pairwise value gains between observations in the initial and final 2% of that video, once per candidate instruction. The correct instruction must have a larger mean gain than every alternative. Section 3.4 states both rules, and Appendix K.1 gives their equations.

AuthorsOpenReview ↗

Q2. What simple heuristic baselines reveal

Q2. How do random and timestep baselines score, and which metrics expose a temporal shortcut?

We added both baselines you requested, together with a constant baseline and an annotation-based progress oracle (Appendix G.3). Random draws each value independently from a uniform distribution on [0, 1]. Constant always returns 0.5. The linear clock increases with position throughout playback, including when actions are reversed. The oracle uses the annotated subtasks, failures, and recoveries.

PredictorVOCMemory-VOCCycle-VOCVSTRRTGA-CF
Random0.211.54−0.380.5116.8321.45
Constant000000
Linear clock100100010000
Annotation oracle100100100100100100

These are in-domain scores ×100 (Table 10). On a fluent forward demonstration, elapsed time is correlated with completion. A clock can therefore order the states correctly even when the same scene recurs. It also updates at every step and never reverses, earning perfect VS. Constant gets zero on VS because it provides no updates.

Cycle-VOC supplies a controlled test of this shortcut. The video reverses after a shared turning point, with the forward history still available. The clock has rank correlation +1 on the forward half and −1 against progress on the reverse half, giving 0 overall. Its TRR and TGA-CF scores are also zero because it cannot react to failure or distinguish instructions for the same video. Table 11 shows the same 100/100/0 pattern for the clock on VOC, Memory-VOC, and Cycle-VOC under both generalization shifts. This is why we read a high Memory-VOC or VS score together with reversal and grounding results.

We will also manually inspect sample inputs and their corresponding predictions for each model to check for unintended timestamp cues.

AuthorsOpenReview ↗

Q3. Benchmark comparison, metric origins and downstream use

Q3. Which benchmarks already cover these capabilities, and which metric constructions does RoboValue add?

Appendix B now compares 23 prior benchmarks and model-specific evaluation suites with RoboValue. Table 4 maps their coverage across the metrics, simulation and real data, and the two generalization shifts. For example, RBM-EVAL already evaluates outcome ranking, cross-task grounding, and forward progress, while ContextProgress-Bench evaluates history-dependent progress. We summarize the metric origins below.

MetricsPrior work or contribution in RoboValue
SA, TGA-CT, VOCEstablished outcome-ranking, cross-task grounding, and forward-ordering tests. VOC follows Ma et al. (2025).
Memory-VOC, SIAMemory-VOC applies VOC to repeated visual states. SIA scores a model-generated subtask description against the annotated stage using a separate judge.
TGA-CFTests the original instruction against four controlled edits to object, action, placement, or constraint on the same successful video. All four comparisons must pass.
Cycle-VOCUses one continuous forward–reverse sequence and retains the forward history at the turn. RoboBrain 2.5 reported the two playback directions separately.
FPL, TRRProcVLM used the earliest global maximum for failure timing. FPL uses the peak before the largest decline and covers recoverable errors. TRR compares three matched branches and scores the corrective attempt and its observable outcome separately.
VS, CSVCVS combines Kaufman’s efficiency ratio with non-flat time coverage. CSVC aligns the same semantic subtask across valid orders and compares gain dispersion with the magnitude of the shared gain.

The last four rows describe the six constructions we introduce. Sections 3.2–3.4 and Appendix B explain how we apply them across simulation and real trajectories with separate embodiment and environment shifts.

Your concern about teleoperation and downstream RL. The benchmark was collected by teleoperation (Section 3.2 and Appendix N.1). This let us match the initial error scenario across trajectories while varying the outcome. For instance, after dropping the teapot lid the robot can leave it, retrieve and seat it, or retrieve and drop it again (Appendix F.2). A model that increases its reward for both retries but keeps rewarding the second after the lid falls has missed a distinction that matters to a recovery policy. The controlled branches give us examples of each outcome even when a particular policy rarely produces one of them. In the next benchmark release, we plan to add failures from autonomous policy rollouts so we can also examine the errors that arise under learned behavior. Failure and recovery protocols.

The metrics address different decisions. SA and TGA test whether a sparse reward distinguishes success and instruction fulfillment. For data screening, VS helps flag expert trajectories whose predicted values oscillate or stay flat. GPT-6.1 Sol scores 96.23 on in-domain Cycle-VOC, a relevant check for dense rewards when progress is undone. Flora has a smaller FPL error (25.46 versus 29.72 for GPT-6.1 Sol), which matters for an execution monitor that must identify when the failure began, or for a curator retaining the valid episode prefix. Such boundaries are also relevant to intervention data of the kind collected in HG-DAgger. We have not measured whether a higher benchmark score predicts higher downstream RL return. Testing that relationship would require training otherwise matched policies with different reward models and measuring success in autonomous rollouts. Kelly et al., HG-DAgger.

Reviewer_4A93OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces RoboValue, a benchmark of 2792 sim and real trajectories across 35 tasks and 4 embodiments. It evaluates robotic value models on task understanding, progress tracking, failure and recovery reasoning, and consistency across valid solutions. Across 15 model variants, strong outcome discrimination coexists with weak counterfactual instruction grounding, poor execution memory, and near-chance failure localization.

Critical strengths
  1. The dataset includes trajectory types that existing benchmarks lack, such as uncorrected errors, effective and ineffective recovery, repeated similar states, and alternative valid orders. This is very very important in the robotics community.
  2. Cycle-VOC shows this pays off by exposing that one leading model's progress score follows elapsed time.
Critical weaknesses
  1. The metrics are new and unvalidated. There is no oracle ceiling and no link to downstream use, and failure localization sits near the 33% random baseline for every model, which may say as much about the metric as about the models. Tables 1-2 report no variance, and failures are staged by teleoperators, so they may not resemble real policy failures.
  2. Some metrics can actually reward the temporal shortcuts this paper is meaning to target. Memory-VOC measures correlation with time on repetitive tasks, so a timestamp-only predictor could score perfectly without remembering completed actions. TOPReward's reported timestamp dependence and high Memory-VOC illustrate this concern. A monotonically increasing timestamp predictor would also receive perfect Value Stability. Explicit time-only and constant baselines would help interpret these scores. Uncertainty estimates are also needed to assess differences in Tables 1 and 2.
Questions for authors
  1. What scores do your metrics give for an oracle value curve built from your subtask annotations?
  2. Does TOPReward's Cycle-VOC improve when timestamps are removed from its input?
2 comments
AuthorsOpenReview ↗

Q1. Annotation oracle and time-only controls

Thank you for suggesting an oracle built from subtask annotations and asking whether TOPReward tracks reversals better without frame timestamps. We revised both the main paper and the supplementary material and combined them in a single PDF.

We added evaluations of four general-purpose frontier models (GPT-6.1 Sol, Gemini 4 Argon, Claude Opus 5.5, and DeepSeek-V4.1-Flash), all with eight-frame inputs. Section 4.1 describes the 24 model configurations. Table 1 reports in-domain scores and Table 2 reports cross-embodiment and cross-environment scores. The same tables give the new timestamp-free TOPReward results, with the input protocol in Appendix G.2. Appendix G.3 adds the oracle and three controls (Tables 10–11), and Appendix O adds uncertainty estimates for the main results (Tables 76–77).

Q1. What scores does an oracle built from the subtask annotations achieve?

The oracle achieves 100 on TRR, 0.43 on FPL, and 86.26 on CSVC in-domain. On successful executions it assigns equal increments to completed subtasks and interpolates within each subtask. For a mismatched instruction it stays at zero. On a failure branch it drops at the annotated error, rises through a corrective attempt, and drops again if the recovery fails. All four reference curves go through the same scoring code. The scores below are multiplied by 100. Lower is better for FPL.

PredictorVOCCycle-VOCMemory-VOCFPL ↓TRRVSCSVC
Random0.21−0.381.5432.1816.830.51−69.91
Constant00072.4000−100
Time only100010072.40010075.45
Annotation oracle1001001000.4310010086.26

The oracle also scores 100 on SA, TGA-CT, and TGA-CF (Table 10). Table 11 reports oracle FPL of 0.42 under embodiment shift and 0.52 under environment shift. These are measured scores for the annotation-defined progress curve. The theoretical optima on this scale are 0 for FPL and 100 for CSVC.

Your concern about temporal shortcuts is borne out by the controls. On these forward demonstrations, elapsed time orders even the visually repetitive states correctly, and a clock updates on every interval. This gives it perfect Memory-VOC and VS. On the continuous forward–reverse sequence the same clock keeps rising through the reversal, so its two directional correlations cancel. Section 3.4 therefore interprets Memory-VOC alongside Cycle-VOC. Constant scores zero on VS because it never updates.

AuthorsOpenReview ↗

Q2. Removing timestamps, localization uncertainty and policy relevance

Q2. Does TOPReward's Cycle-VOC improve when its input timestamps are removed?

Yes. For the same TOPReward-8B backbone, removing the explicit frame-time text raises in-domain Cycle-VOC from 13.50 to 38.76 (+25.26 points). It also raises Cycle-VOC under both distribution shifts. The paired values below come from Tables 1–2 of the original and revised papers.

Metric and settingWith timestampsWithout timestamps
Cycle-VOC, in-domain13.5038.76
Cycle-VOC, embodiment shift18.8343.37
Cycle-VOC, environment shift8.2324.97
VOC, in-domain94.2981.54
Memory-VOC, in-domain95.5139.06

Removing the cue improves reversal tracking while reducing forward and memory scores. This is consistent with the model using the timestamp text on forward-only tests. Both revised TOPReward backbones omit that text (Appendix G.2). The observed prefix can still convey elapsed time through its length. We will also manually inspect sample inputs and their corresponding predictions for each model to check for unintended timestamp cues.

Failure localization and uncertainty. FPL measures the absolute onset error as a percentage of episode duration. The original 33% reference guessed an onset timestamp uniformly. The new Random curve instead supplies random values to the same onset estimator used for every model, yielding 32.18. The oracle’s 0.43 shows that the estimator can locate the annotated onset when given an event-aligned signal. Among learned models, Flora scores 25.46 (SE 1.35) and GPT-6.1 Sol 29.72 (2.11). DeepSeek-V4.1-Flash, Gemini 4 Argon, and Claude Opus 5.5 score 34.25 (1.44), 34.84 (1.56), and 35.89 (1.27), respectively. The gap between these models and the annotation oracle shows that locating the onset precisely remains difficult for current value models. Figure 12 illustrates one source of error. RoboFAC responds to the actual failure, yet a larger decline during normal motion places its predicted onset too early. Tables 76–77 give variances and standard errors for the reported cells of Tables 1–2, using 50,000 task-bootstrap resamples stratified by simulation and real data. These SEs quantify uncertainty across tasks.

Staged failures and downstream use. Teleoperation lets us compare three responses to the same initial error. After the teapot lid falls, for instance, the robot leaves it, seats it, or retrieves and drops it again (Appendix F.2). This design tests whether a model rewards an attempted correction even after its failure is visible. In the next benchmark release, we plan to add failures from autonomous policy rollouts so that we can compare those failures with the controlled cases. Failure and recovery protocols.

We designed the metrics around specific downstream decisions. For data screening, VS can flag demonstrations whose predicted values oscillate or stay flat. For execution monitoring, FPL measures how accurately a value drop locates the error onset, which also helps a curator decide how much of a failed episode to retain. Human-intervention data, as collected in HG-DAgger, are one setting where that boundary matters. SA and TGA test whether sparse reward signals distinguish outcomes and instructions. Cycle-VOC and TRR test whether dense rewards change direction when progress is undone or recovery fails. Measuring how these scores predict downstream return would require matched RL runs with different reward models. Kelly et al., HG-DAgger.

Reviewer_QbKBOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

RoboValue evaluates robotic value models on 2,792 trajectories from 35 tasks and four embodiments, covering both simulation and real data. It includes successful execution, continued failure, recovery attempts, repeated observations that require history, and different valid subtask orders. The paper compares 15 model variants from nine families using metrics for success discrimination, task grounding, progress, memory, failure localization, recovery, stability, and consistency across solutions. A main finding is that high progress correlation on successful trajectories can coexist with poor failure diagnosis and recovery reasoning.

Critical strengths
  1. The diagnostic cases are well chosen. A model can track elapsed time on a successful demonstration without recognizing a failure or understanding whether a recovery worked. The matched failure and recovery cases, repeated observations, and alternative solution orders test distinctions that a standard progress metric would miss. Including real trajectories makes these tests more relevant to deployment.
  2. Table 1 shows why several evaluation dimensions are needed. Models with strong value-order correlation can behave quite differently on reversed trajectories and memory-sensitive cases. This is useful evidence against treating performance on ordinary successful demonstrations as sufficient validation of a value model.
Critical weaknesses
  1. My main concern is whether the metrics mean the same thing across the models being compared. The paper includes progress estimates, completion signals, and remaining-time predictions. Yet Equation 1 requires value to rise during a recovery attempt even when that recovery ultimately fails. A completion estimate could reasonably stay low during an ineffective attempt. CSVC similarly expects the same subtask to produce comparable value increments in different execution orders, although its effect on completion probability or remaining time can depend on the unfinished steps. These rules may penalize appropriate predictions rather than reveal reasoning failures. The unified comparison needs validation against the intended semantics of each output.
  2. Several details needed to interpret the results are unavailable in the supplied PDF, which ends after the references. The text repeatedly refers to appendices for adapters, prompts, temporal inputs, and metric implementation. Failure localization depends on the ZigZag filter, while memory tests depend on how much history each model receives. Without those details, I cannot tell how much of the observed difference comes from the model and how much comes from its evaluation interface. Diagnostic subset counts and annotation agreement would also help assess reliability. A separate supplement could resolve this concern, but the metric-validity issue would still need to be addressed.
Questions for authors
  1. Why should value rise during an ineffective recovery attempt? Have the TRR and CSVC rules been checked against expert judgments separately for progress, completion, and remaining-time outputs?
  2. Can the referenced appendices be provided, including adapters, frame and history budgets, one-shot training details, diagnostic subset counts, annotation agreement, and ZigZag thresholds?
  3. How do timestamp-only and semantically defined reference predictors score? Are the CSVC and stability rankings sensitive to monotonic output transformations or discretization, and do the memory results hold with matched history inputs?
3 comments
AuthorsOpenReview ↗

Q1. Recovery attempts and subtask credit

Thank you for raising the question of what a score means when one model predicts progress and another predicts completion or remaining time. We revised both the main paper and the supplementary material and combined them in a single PDF.

We added evaluations of four general-purpose frontier models (GPT-6.1 Sol, Gemini 4 Argon, Claude Opus 5.5, and DeepSeek-V4.1-Flash), each with an eight-frame input. Section 4.1 describes the 24 configurations we evaluate. Table 1 gives in-domain results and Table 2 gives results under embodiment and environment shifts. Appendix G.2 specifies the input history and output for each model, Appendix G.3 adds four reference predictors (Tables 10–11), and Appendices K.3–K.4 explain the recovery and subtask-credit criteria.

Q1. Why require a rise during an ineffective recovery, and what validates TRR and CSVC across prediction targets?

We agree that a completion probability can reasonably stay low throughout a failed retry. TRR asks whether a signal used as a dense reward can credit progress that is temporarily regained and then recognize when it is lost. In the tea-set task, for example, the robot drops a lid. In one branch it leaves the lid on the table. In another it retrieves and seats it. In the third it regrasps the lid but drops it again before seating it (Appendix F.2). A dense progress signal can credit the regrasp in both retries, then reverse that credit when the lid falls again.

Equation 1 tests the failed-recovery branch in three stages. It requires a decline during the initial error, an overall rise during the corrective attempt, and a decline once the failed outcome becomes observable. Individual movements can lose ground during the attempt. If a retry never regains any ground, a flat prediction is reasonable and the strict attempt criterion will withhold credit. Figure 11 shows why it helps to inspect the stages separately. TOPReward-32B passes the attempt criterion on 73.3% and the failed-outcome criterion on 28.3%, averaged across tasks and domains.

CSVC compares the gain assigned to the same semantic subtask across valid orders. In Figure 13, for example, the cup can be placed first or last. A change in its gain matters if we use that gain to weight the cup-placement segment in training. A completion forecast can legitimately give the two placements different gains because the remaining steps differ in difficulty. A time-to-go predictor can likewise assign different savings. Appendix K.4 makes this dependence on prediction target and execution context explicit.

Our human annotations mark subtask boundaries, failures, and observable recovery outcomes. We have not collected independent expert judgments of the expected curves separately for progress, completion probability, and remaining time. We therefore interpret TRR and CSVC as diagnostics for using these outputs as local progress feedback. The accuracy of each model on its native prediction target is a separate question.

AuthorsOpenReview ↗

Q2. Model inputs, training, sample counts and annotation

Q2. Where are the adapters, history budgets, one-shot training details, diagnostic counts, annotation protocol, and ZigZag settings?

The appendices now follow the references on pages 17–106 of the same PDF. Section 3.3 defines the shared evaluation interface. Appendix G.2 (pages 52–54) gives the model-specific inputs, prompts, and readouts, including which adapters use scalar differences and which use a native pairwise critic.

Observation history. Each of the four general-purpose models receives the instruction and eight frames from the observed prefix, in temporal order. GPT-6.1 Sol, Gemini 4 Argon, and Claude Opus 5.5 also share the progress prompt and adapter. Robotic value models retain their native frame counts, sampling rules, and camera views. For example, RoboMeter takes its final progress estimate from eight uniformly sampled prefix frames. RynnValue also samples eight prefix frames and returns the negative of its predicted seconds remaining. Flora-Rewarder samples up to 64 frames uniformly from the prefix, including the endpoints. Flora's eight-frame captioner is used for subtask identification.

One-shot protocol. We use one of the 100 training demonstrations for each task, disjoint from the test set (Section 3.3). VLAC uses six reference frames and Robo-Dopamine uses the reference endpoints. ProcVLM fits a task-specific LoRA adapter on remaining-action and progress targets built from that demonstration. The saved simulation adapters use rank 64, LoRA alpha 128, and three epochs. Appendix G.2 describes how the targets and readouts are constructed. Anonymous evaluation code.

Diagnostic sample sizes. The counts below refer to the in-domain evaluation. Table 5 specifies which trajectory types enter each metric.

DiagnosticSimulationReal worldUnit
FPL252306single-error episodes
TRR126147matched three-branch groups
Memory-VOC3090episodes with recurring visual states
CSVC48tasks with multiple valid solutions

Annotation and localization. We manually marked subtask start/end times, the initial failure interval, and, for recovery branches, the first observable outcome (Appendix N.2). We have not measured independent inter-annotator agreement for these boundaries. For FPL we sample at 5 Hz. The ZigZag reversal threshold is 0.2 times the largest drop between an earlier and a later value in that episode. The peak before the largest detected decline is the predicted onset. A trajectory with no detected decline receives its maximum possible timing error (Appendix K.3). Testing coefficients of 0.1 and 0.3 against 0.2 preserved 11/12 pairwise model orderings in simulation and 12/12 in the real-world subset (Appendix H, Table 14).

AuthorsOpenReview ↗

Q3. Reference predictors, score transformations and matched history

Q3. How do time-only and semantic reference predictors score? How sensitive are rankings to score transformations and history inputs?

Reference predictors. Appendix G.3 evaluates Random, Constant, Linear (elapsed time only), and an annotation-based progress oracle. In-domain, the oracle scores 100 on TRR and 86.26 on CSVC. The linear clock scores 0 and 75.45, respectively (Table 10, scores ×100). It also scores 100 on VOC, Memory-VOC, and VS. On the forward expert demonstrations, elapsed time orders the states correctly and the clock updates at every step. During reverse playback it continues to increase as task progress is undone, so its Cycle-VOC is 0. Constant scores 0 on all three temporal metrics and VS. These controls show exactly where a model can score well using time alone. Table 11 gives the corresponding generalization results for the metrics evaluated under each shift.

Removing explicit timestamps from TOPReward-8B raised its in-domain Cycle-VOC from 13.50 to 38.76 (original and revised Table 1). We will also manually inspect sample inputs and their corresponding predictions for each model to check for unintended timestamp cues.

Scale and discretization. Changing scalar units by a positive affine map leaves VS unchanged. It also leaves CSVC unchanged when the adapter computes gains as differences between scalar values. For native pairwise critics, multiplying all reported gains by the same positive factor cancels in CSVC. A nonlinear increasing map can change relative gains and hence the ordering of model scores. For example, the intervals from 0 to 0.25 and from 0.5 to 0.75 both gain 0.25. Squaring the values changes their gains to 0.0625 and 0.3125. For VS, a nonlinear map can change net movement relative to total variation, while rounding can make small updates disappear. The protocol excludes five-level RoboReward from VS and CSVC (Appendix G.4). In the four-model sensitivity study, varying the sampling rate preserved 12/12 VS orderings in each domain. Varying the number of subtask boundary pairs preserved 10/12 CSVC orderings in simulation and 12/12 in the real world (Table 14). Rankings after nonlinear remapping or quantization remain unmeasured.

Matched history. The revision specifies the same eight-frame budget and temporal order for all four general-purpose models (Section 4.1 and Appendix G.2). Their Memory-VOC scores retain the same ordering in all three settings (Tables 1–2, ×100).

ModelIn-domainEmbodiment shiftEnvironment shift
GPT-6.1 Sol89.8278.0188.70
Claude Opus 5.555.6447.9549.41
Gemini 4 Argon52.1540.4346.46
DeepSeek-V4.1-Flash34.1229.7727.41

The first three also use the same adapter and progress prompt, so their differences persist under that tighter comparison. Among robotic models, RoboMeter-4B and RynnValue-4B score 2.36 and −23.89 in-domain. Both sample eight prefix frames, with different rounding and padding rules. The full leaderboard uses native robotic interfaces. A rerun giving every model identical frames and views would be needed to isolate the effect of history selection across the whole benchmark.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.