acceptodds
Under review as a conference paper at ICLR 2027

ProgressCompass: Embodied Progress Reward Models Are Lost Without the Right Context

26 views

Abstract

Embodied agents now take on ever longer tasks. For long tasks, knowing only whether a task finally succeeds or fails says little; the steps along the way matter. Progress Reward Models (PRMs) score how far a task has come at every step, and serve as dense rewards, verifiers and monitors. Yet in long tasks the current frame alone often cannot tell how far the task has come, because progress depends on what happened before. We call this problem context-dependent progress estimation. Existing benchmarks on progress estimation mostly focus on short tasks whose progress can be read from the current observation, and whether PRMs can estimate progress when context is needed remains underexplored. We therefore build ContextProgress-Bench, with 24 manipulation tasks for 120 episodes. The benchmark covers three settings: (i) State Recall, where information needed for progress appeared earlier but is not in the current frame; (ii) Sequence Tracking, where steps follow a fixed order, so progress requires knowing which steps are done and which comes next; and (iii) Recurrence Disambiguation, where look-alike frames sit at very different progress. We then run a paired diagnosis: each PRM keeps the same input format in both runs, and in one run its instruction integrates the right context. Results show that even for PRMs that read the entire history would still get lost in estimating progress. However, with the right context, the same five models cut their progress error by 77–82%. This suggests that Embodied PRMs are thus not incapable in progress estimation, but lost without the right context. We therefore propose ProgressCompass, an autonomous agentic loop that reorients an existing PRM and uses current general-purpose VLMs to supply the context the PRM needs. Wrapped in the loop, the same frozen PRM cuts its progress error by 63% and raises its rank agreement by 76%. With such a compass, PRMs estimate progress far better on longer, more complex tasks.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Progress Reward, Agentic AI, Embodied AI

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_pMqXOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2(+1)
Read review
Paper summary

This paper investigates the problem of progress estimation based on historical context in long-term embodied tasks. The authors constructed ContextProgress-Bench, which includes 24 operational tasks and 120 trajectories, covering three types of scenarios: state recall, sequence tracking, and distinguishing recurring events. Manual context diagnosis revealed that existing PRMs may still fail to accurately assess progress even when provided with historical information; after providing the correct context, the error rates of five models were reduced by 77%–82%. Based on these findings, the paper proposes ProgressCompass, which uses a general-purpose VLM to provide task context to a frozen PRM and advances subtasks through a cycle of completion verification and memory updates. Built on RoboMeter-4B, the system reduces progress error by 63% and improves the Spearman correlation coefficient from 0.53 to 0.93.

Critical strengths

The system significantly improves the progress estimation performance of existing reward models. With the RoboMeter-4B weights frozen, ProgressCompass reduces the MAE on ContextProgress-Bench from 25.4 to 9.3, a decrease of approximately 63%; the Spearman correlation coefficient increased from 0.53 to 0.93. This indicates that the complete system simultaneously improves the accuracy of progress estimates and the temporal consistency of the prediction curves across the context-dependent tasks tested.

Critical weaknesses
  1. The incremental contributions of the method, the roles of its modules, and its applicability across reward models have not yet been fully clarified.

R²VLM has utilized recursive processing to decompose and complete local video maintenance tasks, and the two-stage experiments in the original RoboMeter work also employed a VLM to decompose subtasks, with the reward model determining completion and advancing to the next stage. Therefore, the paper needs to more clearly explain the mechanisms of the independent Verifier, expected state changes, and memory updates following verification—specifically, what new features it adds compared to existing approaches and which errors it specifically addresses that previous methods failed to handle. Existing ablation studies primarily focus on the expected state change τt\tau_t and have not sufficiently validated the individual contributions of the Orienter, PRM, Verifier, and Navigator. Additionally, the Orienter uses Qwen3.5-27B, the Verifier and Navigator use Qwen3.5-9B, and the PRM uses RoboMeter-4B. The lack of control experiments involving module removal or replacement, standardized auxiliary VLM configurations, and inference budget control makes it difficult to distinguish whether performance improvements stem from module design, auxiliary model capabilities, or increased inference calls. In particular, to verify the independent contribution of PRM, it is necessary to replace PRM with a simple scoring rule or direct scoring by the same VLM while retaining the task decomposition, memory, and verification processes.

Furthermore, the complete framework has only been validated on RoboMeter-4B. The manual context diagnosis for the five PRMs relies on manually provided context and subtask boundaries and cannot replace cross-model validation of the complete automated framework. We recommend integrating another representative PRM while keeping the auxiliary VLM and evaluation protocol fixed, and reporting both accuracy and inference overhead to clarify the framework’s scope of applicability.

  1. Differences in sampling settings in the manual context diagnosis require explanation.

Appendix C shows that the sampling interval for Robo-Dopamine was changed from 5 (without context) to 10 (with context), while the maximum number of input frames for TOPReward was changed from 96 to 48. Therefore, in addition to context, the visual input configurations were also altered between the two sets of experiments. Given that the two sets of experiments processed full videos and subtask clips, respectively, these adjustments may be justified; however, the paper has not provided a sufficient explanation. We recommend that the authors explain the reasons for these adjustments, report the actual number of input frames and temporal coverage, and include a control group with consistent sampling settings or provide evidence that these differences do not affect the main conclusions, in order to more fully support the attribution of performance improvements to context.

Questions for authors

Questions:

  1. How is the independent contribution of each module verified? Aside from ablation experiments on the expected state change (\tau_t), have there been any controls where modules were removed or replaced, and where the auxiliary VLM capabilities and inference call budgets were controlled? In particular, when preserving task decomposition, memory, and validation workflows, how much independent benefit does PRM provide compared to simple scoring rules or direct scoring by the same VLM?
  2. Why did the human context diagnosis alter the visual sampling configuration? Please explain the rationale for changing the sampling interval in Robo-Dopamine from 5 to 10 and the maximum number of frames in TOPReward from 96 to 48, and describe how the actual number of input frames and temporal coverage changed. Are there any analyses or comparisons to support the conclusion that these differences do not substantially affect the context-based gains?
  3. Does the R²VLM baseline employ a complete recursive reasoning process? Please provide the specific checkpoint, local video segment length, and frame sampling settings used by R²VLM in Figure 5, and clarify whether the task decomposition and completion status in the historical CoT are propagated and updated round by round. This information is crucial for verifying whether the comparison closest to the baseline is sufficient.

Recommendations for Future Work: Given a fixed auxiliary VLM and evaluation protocol, integrate the complete automated framework into other representative PRMs to verify cross-model applicability; the current empirical conclusions should be explicitly limited to the tested RoboMeter-4B configuration.

4 comments
AuthorsOpenReview ↗

Response to Reviewer pMqX (1/n)

Thank you for recognizing that ProgressCompass improves accuracy and temporal consistency with a frozen RoboMeter-4B. We respond to each concern below.

Concern 1: R²VLM and RoboMeter already use task decomposition. What does ProgressCompass add?

ProgressCompass adds verified context updates to task decomposition. On ContextProgress-Bench, five PRMs have MAEs of 15.59–30.95 without explicit subtask context. Oracle context lowers those errors by 77–82% (Table 1); this condition also supplies annotated boundaries and step positions. Our system estimates progress without those annotations at inference time.

MethodProgress MAE↓Spearman ρ↑
RoboMeter-4B25.40.53
ProgressCompass (based on RoboMeter-4B)9.30.93

We adapted RoboMeter's multi-stage procedure (Sec. IV-Q3 and Appendix E.2): Qwen3.5-27B generates a fixed plan from the instruction and first frame; RoboMeter-4B advances when success probability exceeds 0.6. ProgressCompass instead checks completion and uses verified observations to update context.

MethodState MAE↓Count MAE↓Sequence MAE↓Overall MAE↓Spearman ρ↑Pearson r↑
Qwen3.5-27B decomposition + RoboMeter22.5915.9417.5518.910.6580.711
ProgressCompass10.507.5210.529.820.9410.933

Even against this staged baseline, MAE drops from 18.91 to 9.82 (48%). R²VLM also updates context, through recurrent reasoning. Our design separates local scoring from completion checks and carries verified observations forward.

Concern 2: The expected-transition ablation does not establish the separate roles of the Orienter, PRM, Verifier, and Navigator.

The four components have distinct roles (Section 4 and Appendix B):

ComponentWhat it does
OrienterUses the task instruction, current frame, and verified memory to identify the next subtask and describe what visible change would mark its completion.
PRM (RoboMeter-4B)Scores progress within the current subtask and proposes a candidate completion point.
VerifierChecks the visual evidence at that point. It asks whether the required change actually happened during this subtask.
NavigatorTracks the active step and routes calls. Once the Verifier confirms completion, it carries the confirmed observation into memory and advances the loop.

Model-size ablation. We change only the auxiliary Qwen3.5 model sizes. The code and RoboMeter-4B stay fixed. State, Count, and Sequence report MAE.

OrienterVerifierNavigatorState↓Count↓Sequence↓Overall MAE↓Spearman ρ↑Pearson r↑
27B9B9B10.507.5210.529.820.9410.933
9B9B9B15.9617.6318.3217.350.8670.849
27B4B9B14.6917.2811.5113.950.9200.902
27B9B4B13.8412.2211.5712.500.9230.913
27B4B4B16.5717.1712.8815.150.9240.908
  1. Orienter: Switching to 9B raises overall MAE from 9.82 to 17.35.
  2. Verifier: Switching to 4B raises MAE to 13.95 and more than doubles Count error (7.52 → 17.28).
  3. Navigator: Switching to 4B raises MAE to 12.50. Switching both the Verifier and Navigator to 4B raises it to 15.15.

Expected-transition ablation. Removing the description of what should change from the Verifier raises MAE from 9.3 to 17.5 and stalled episodes from 7 to 40. The Verifier still receives the subtask instruction. This tests the completion description, not whether the Verifier or Navigator can be removed.

SettingProgress MAE↓Stalled Episodes↓
Full ProgressCompass9.37
No expected transition at Verifier17.540
No expected transition at Orienter and Verifier18.345
AuthorsOpenReview ↗

Response to Reviewer pMqX (2/n)

Concern 3: The gain may come from larger auxiliary models or more inference calls.

Directly prompting Qwen3.5-27B gives 19.42 MAE, versus 9.82 for ProgressCompass. The tested frontier models score lower still. The model-size ablation above supports where we use larger models, but these comparisons do not isolate the effect of additional calls.

MethodState MAE↓Count MAE↓Sequence MAE↓Overall MAE↓Spearman ρ↑Pearson r↑
ProgressCompass (based on RoboMeter)10.507.5210.529.820.9410.933
Opus 5.5, direct prediction8.616.787.527.720.9640.951
GPT-6 Astra, direct prediction6.294.717.366.370.9750.961
Qwen3.5-27B, direct prediction26.7821.8012.1119.420.7190.702
Qwen3.5-9B, direct prediction40.9835.7925.0833.060.4330.456
RoboMeter, direct prediction under the comparison settings30.1724.9222.3625.650.5130.504

Concern 4: With decomposition, memory, and verification kept, does the PRM beat a simple rule or direct VLM scoring?

We replaced only the progress estimator. RoboMeter-4B beats the best Qwen replacement by 23% in MAE (9.82 vs. 12.74). “Abs” predicts completion percentage; “delta” predicts the change since the previous sample. All four Qwen variants use temperature 0.7 and top_p 0.8.

Progress EstimatorState↓Count↓Sequence↓Overall MAE↓Spearman ρ↑Pearson r↑
RoboMeter-4B10.507.5210.529.820.9410.933
Qwen3.5-27B, abs12.2314.8411.9812.740.9150.908
Qwen3.5-9B, abs14.9615.6813.3214.440.9070.897
Qwen3.5-9B, delta19.6719.0515.7117.850.8650.853
Qwen3.5-27B, delta23.1918.8715.4818.930.8970.872

On manually annotated subtasks, ProgressCompass also beats the fixed 50% guess (22.43 vs. 25.61); the five standalone PRMs do not. This measures within-subtask changes, separately from the full-loop replacement test.

Concern 5: Why do the visual sampling settings change in the manual-context diagnosis?

We tested whether the sampling differences explain the gains in Table 1. The table reports global progress MAE.

ModelContextSampling settingAvg. countMAE↓Spearman ρ↑Pearson r↑
Robo-DopamineNoInterval 1060.115.590.7450.762
Robo-DopamineNoInterval 5119.715.740.7400.752
Robo-DopamineYesInterval 1013.33.430.9750.962
TOPRewardNoFrame cap 969630.950.5590.542
TOPRewardYesFrame cap 4847.96.700.9050.890
TOPRewardYesFrame cap 9688.16.690.9240.911

Avg. count: scoring points for Robo-Dopamine; input frames for TOPReward.

  • Robo-Dopamine: Halving the interval nearly doubles the number of scoring points without context, but MAE changes by only 0.15. At interval 10, MAE is 15.59 without context vs. 3.43 with context.
  • TOPReward: The 48-frame cap was reached in 539 of 552 subtask clips. Raising it to 96 increases the actual input from 47.9 to 88.1 frames on average, but MAE is nearly unchanged (6.70 vs. 6.69). At the same 96-frame cap, MAE is 30.95 without context vs. 6.69 with context.

The gains remain after matching the sampling parameter. The full-episode and subtask conditions still differ in their boundaries and global step positions, and Robo-Dopamine also receives a different goal image.

Concern 6: Does the R²VLM baseline run the full recurrent process and carry task history between rounds?

We evaluated R²VLM using the original baseline's recurrent inference procedure. The model processes successive video clips and carries its task decomposition and completion history from one round to the next, following the original method.

Concern 7: The complete automated framework is tested only with RoboMeter. Does it work with another PRM?

Why RoboMeter? It can score the current subtask from the instruction and observed frames. Reference-based VLAC, ProgressLM, and our Robo-Dopamine setup need a reference video, a visual or textual demonstration, or a goal image, respectively. We have not tested TOPReward in the full loop.

Can the estimator be swapped? Yes. On the same 120 episodes, Qwen3.5-27B and 9B give MAEs of 12.74 and 14.44, versus 9.82 with RoboMeter-4B. The loop supports other estimators, but a second specialized PRM remains untested.

Reviewer_pMqXOpenReview ↗

Follow-up on the Verifier and PRM generalization

Thank you for your detailed response and the additional experiments. I have two follow-up questions that may help further clarify the contributions and the scope of applicability of this method.

  1. What is the specific contribution of the Verifier?

The ablation study on model scale shows that reducing the Verifier from 9B to 4B increases the MAE from 9.82 to 13.95. While this indicates that the choice of Verifier affects performance, it does not fully quantify the contribution of the Verifier compared to a system that does not use one.

Thank you for providing the baseline using Qwen task decomposition combined with RoboMeter. However, since this baseline employs a fixed schedule, the difference between it and ProgressCompass is not limited solely to the inclusion of a Verifier.

Could you test a version that omits the Verifier—where the system proceeds directly to the next step once the PRM proposes a candidate completion point—while keeping other components as consistent as possible? Additionally, please clarify how memory is updated in this version. Comparing the MAE and the frequency of premature advancements would help quantify the contribution of the verification step to the overall system.

  1. Can the complete framework be generalized to other PRMs?

The experiment involving the replacement of Qwen demonstrates that the framework can accommodate different progress estimators. However, the fully automated framework has currently only been evaluated using one specific PRM: RoboMeter.

Would it be possible to test the complete framework using a different, compatible PRM, while keeping the other components and the evaluation pipeline unchanged? Comparing the performance of that PRM when running independently versus when integrated into ProgressCompass would help determine whether the benefits of this method extend beyond RoboMeter. If such an experiment is difficult to implement, clarifying the scope of applicability for the current results would also be helpful.

AuthorsOpenReview ↗

Wish this could solve your questions

Q1. What does the Verifier contribute?

The Verifier checks whether the expected change is visible before the system advances and updates its memory. We reran the same 120 episodes without the Verifier, keeping the other models, frame sampling, and evaluation protocol fixed.

Advancement and memory. Without the Verifier, the system advances as soon as RoboMeter’s score reaches the threshold. It records the subtask, its frame span, and the Orienter’s predicted after-state. With the Verifier, memory records the state observed at confirmed completion.

SetupOverall MAE ↓Spearman ρ ↑
ProgressCompass with Verifier9.820.941
No Verifier, threshold 0.617.150.870
No Verifier, threshold 0.718.310.860
No Verifier, threshold 0.818.770.883
No Verifier, threshold 0.928.740.898

ProgressCompass with the Verifier achieves lower MAE and higher Spearman ρ than all four variants. The lowest MAE without verification is 17.15, compared with 9.82 for the full system.

Q2. Does ProgressCompass work with other PRMs?

Finding a compatible PRM is difficult because ProgressCompass scores each subtask before its endpoint is known. At that point, the loop has the frames observed so far and the current subtask description. A matching demonstration or goal image may be unavailable, which limits the ProgressLM and Robo-Dopamine fusion configurations we evaluated.

We tested VLAC and TOPReward both standalone and inside the full ProgressCompass loop on the same 120 episodes. We kept the Orienter, Navigator, Verifier, and evaluation protocol fixed across the integrated runs. The loop inferred the subtask boundaries, and VLAC received no reference video in either setup.

PRMStandalone MAE ↓In ProgressCompass MAE ↓MAE reduction ↑
VLAC45.5921.1353.7%
TOPReward30.9514.1354.3%

Both PRMs improved by about 54%, even though their standalone scores were less accurate than RoboMeter’s in our setup. RoboMeter still achieves the lowest MAE in the loop (9.82). These results show that the gains extend to two additional PRMs, while the choice of scorer still affects the final accuracy.

Reviewer_NYd5OpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper identifies a limitation of existing robotic Progress Reward Models (PRMs) on long-horizon tasks: progress often depends on task-relevant context that is unavailable in the current frame and may not be extracted reliably even from the observation history. It introduces a benchmark for such context-dependent tasks and shows that PRMs improve substantially when given ground-truth context. The paper then proposes an agentic pipeline that generates the required context, uses a frozen PRM to estimate local progress, and verifies subtask completion. Experiments show that this pipeline improves over the frozen RoboMeter-4B backbone without retraining it.

Critical strengths
  1. The paper identifies an important limitation in robotics PRMs on long-horizon tasks and introduces a benchmark to demonstrate the limitation across different existing PRMs.

  2. The proposed agentic pipeline is relatively simple, and shows substantial improvement over RoboMeter-4B without any training.

Critical weaknesses
  1. The technical details in the paper are largely described conceptually, lacking essential details in understanding the exact approach and reproducing the results. In particular, the prompts used by the Orienter O, Verifier V and Navigator N are not given. Although a fixed schema is mentioned, it is unclear, without the concrete prompts, that to what extent the prompts are held fixed to generate all results. Example outputs from the Orienter O, Verifier V and Navigator N are also missing.

  2. The scale of the benchmark is limited. It only contains 120 episodes, and 15 held-out negatives (across early stop, extra steps and instruction mismatch). This benchmark is significantly smaller than existing benchmarks such as RoboRewardBench and Robometer RBM-EVAL. Although this is understandable due to complexities of labeling for long-horizon tasks, it is challenging to estimate whether the results from this benchmark are representative. This is especially true for the 15 negatives.

Questions for authors
  • What are the exact prompts used for the Orienter O, Verifier V and Navigator N?

  • Can you provide more details on the benchmark, e.g., how was ground truth progress generated?

  • Does the agentic pipeline provide a similar performance boost for PRMs other than RoboMeter-4B?

  • Can the authors provide an ablation that replaces the PRM with another frozen Qwen model? It seems like since the frozen Qwen model is capable of solving the more complex context inference task, it should be able to estimate progress for subtasks as well.

  • What are the main assumptions for Theorem 1? Please make them explicit rather than hide them in the text and definitions.

  • From the results from Section 5.3, it seems like whether providing the context to the PRM does not significantly change the progress MAE (17 vs 19 overall), yet it is important to provide it to the verifier V. This is not exactly consistent with the main message of the paper, which is that PRM needs context to correctly estimate progress. Can the authors provide a discussion on that?

  • How long does the agentic pipeline take to estimate progress for one frame?

8 comments
AuthorsOpenReview ↗

Response to Reviewer NYd5 (1/n)

Thank you for your helpful comments. We appreciate your recognition of (i) the important problem studied by our benchmark, and (ii) the strong gains of our simple pipeline without additional training. We have carefully considered your questions about implementation details, benchmark size, other models, the theorem, context, and runtime. We respond to each below.

Concern 1: The exact prompts, fixed schemas, and example outputs are missing. How can the method be reproduced?

Thank you for asking for more implementation detail. Appendix D gives the prompt templates and describes the fields returned by the Orienter, Navigator, and Verifier. And we also include a specific case study for easier understanding.

Concern 2: The benchmark is small, especially the 15 negative episodes. Are the results representative?

Thank you for raising this concern. ContextProgress-Bench is a targeted test of progress estimation when the current observation alone is insufficient. We manually selected tasks that meet this criterion and annotated the subtask intervals and the context needed in each interval. Adding more trajectories without this check would increase the sample count without necessarily testing the ability we study.

Our conclusions are supported by five complementary checks:

  1. Coverage across failure modes. The main benchmark covers 24 tasks, 120 episodes, and 552 annotated intervals across three source datasets. It tests whether models remember past states, track ordered steps, and distinguish repeated events.
  2. The context gap appears across models. All five PRMs improve under the oracle subtask setup, with global MAE reductions of 77–82%. This shows a consistent pattern under the tested setup, which provides annotated boundaries and step positions.
  3. The gain survives a matched sampling setting. For TOPReward, raising the frame cap from 48 to 96 increases actual input from 47.9 to 88.1 frames on average, but MAE stays nearly identical (6.70 vs. 6.69). At the same 96-frame cap, the full-episode condition has 30.95 MAE.
  4. The full system improves on a stronger baseline. Even when the baseline uses Qwen3.5-27B to decompose the task and RoboMeter-4B to score and advance through stages, ProgressCompass reduces overall MAE from 18.91 to 9.82 (48%), with lower error in all three categories.
  5. The ablations show which choices matter. Keeping the loop fixed, replacing RoboMeter with the best tested Qwen scorer raises MAE from 9.82 to 12.74; reducing the Orienter from 27B to 9B raises it to 17.35. In the paper's separate verification ablation, removing the expected transition from the Verifier increases stalled episodes from 7 to 40.

The 15 held-out negative episodes serve a separate purpose. They are targeted checks for early stopping, extra steps, and instruction–execution mismatch. They show how the system behaves in those cases; they do not estimate how often such failures occur in general. The main results above come from the 120-episode evaluation, not from those 15 negatives. We will clarify this distinction in the paper.

Concern 3: How were the ground-truth progress labels generated?

Thank you for asking us to clarify the labels. We reviewed every annotated subtask interval by hand, checking its start and end against the video. We then assigned an equal share of total progress to each of the K subtasks and interpolated linearly by frame position within each interval. Subtask k runs from k/K × 100 to (k+1)/K × 100, with k starting at 0.

AuthorsOpenReview ↗

Response to Reviewer NYd5 (2/n)

Concern 4: Since Qwen can infer context, can it also replace the PRM and estimate subtask progress?

Thank you for this suggestion. We tested exactly this replacement. The larger Qwen models can produce progress scores, but those scores are less accurate than the specialized 4B PRM's scores. All other framework components are unchanged.

Progress EstimatorState↓Count↓Sequence↓Overall MAE↓Spearman ρ↑Pearson r↑
RoboMeter-4B10.507.5210.529.820.9410.933
Qwen3.5-27B, abs12.2314.8411.9812.740.9150.908
Qwen3.5-9B, abs14.9615.6813.3214.440.9070.897
Qwen3.5-9B, delta19.6719.0515.7117.850.8650.853
Qwen3.5-27B, delta23.1918.8715.4818.930.8970.872

The best VLM replacement is Qwen3.5-27B with absolute prediction. Its MAE is 12.74, compared with 9.82 for RoboMeter-4B. The specialized 4B PRM therefore has about 23% lower error, despite being much smaller. Its error is lower in every task category. The largest gap is on Count: 7.52 versus 14.84, about 49% lower error. Absolute prediction also works better than delta prediction for both VLM sizes.

Here, “abs” predicts the current subtask completion percentage. “Delta” predicts the change since the previous sample, which is then accumulated. All four VLM replacements use temperature 0.7 and top_p 0.8. Correlations are computed against frame index for each episode and then averaged. Constant curves are excluded: 0 episodes for RoboMeter, 1 for each abs model and 9B delta, and 5 for 27B delta. MAE remains the main comparison because it includes all 120 episodes.

In the same framework, RoboMeter gives lower error than all four Qwen replacements. To check the broader alternative of directly prompting a VLM, we also ran the following comparison.

We also tested direct progress prediction, without the ProgressCompass loop. The new results show a clear gap between the tested open-source VLMs and frontier models.

MethodState MAE↓Count MAE↓Sequence MAE↓Overall MAE↓Spearman ρ↑Pearson r↑
ProgressCompass (based on RoboMeter)10.507.5210.529.820.9410.933
Opus 5.5, direct prediction8.616.787.527.720.9640.951
GPT-6 Astra, direct prediction6.294.717.366.370.9750.961
Qwen3.5-27B, direct prediction26.7821.8012.1119.420.7190.702
Qwen3.5-9B, direct prediction40.9835.7925.0833.060.4330.456
RoboMeter, direct prediction under the comparison settings30.1724.9222.3625.650.5130.504

ProgressCompass has about 49% lower MAE than directly prompting Qwen3.5-27B: 9.82 versus 19.42. Direct Qwen3.5-9B prediction is worse, at 33.06. Thus, directly asking a larger open-source VLM for progress does not match the framework's accuracy in these runs.

Frontier models do better: Opus 5.5 reaches 7.72 and GPT-6 Astra reaches 6.37, both below ProgressCompass's 9.82. We therefore want to close such gap with a much cheaper way by using open-sourced models.

AuthorsOpenReview ↗

Response to Reviewer NYd5 (3/n)

Concern 5: Does the framework give similar gains with PRMs other than RoboMeter? ProgressCompass reduced MAE for all three estimators we tested on the same 120 episodes. Qwen3.5-27B improved from 19.42 to 12.74 (34%), Qwen3.5-9B from 33.06 to 14.44 (56%), and RoboMeter-4B from 25.65 to 9.82 (62%). RoboMeter gave the lowest error in the full loop.

EstimatorSetupState MAE ↓Count MAE ↓Sequence MAE ↓Overall MAE ↓Spearman ρ ↑Pearson r ↑
Qwen3.5-27BStandalone26.7821.8012.1119.420.7190.702
Qwen3.5-27BIn ProgressCompass12.2314.8411.9812.740.9150.908
Qwen3.5-9BStandalone40.9835.7925.0833.060.4330.456
Qwen3.5-9BIn ProgressCompass14.9615.6813.3214.440.9070.897
RoboMeter-4BStandalone30.1724.9222.3625.650.5130.504
RoboMeter-4BIn ProgressCompass10.507.5210.529.820.9410.933

Standalone estimators use the full task instruction and frames sampled uniformly from the first frame through the current scoring point. Under this sampling protocol, RoboMeter's MAE is 25.65 (25.42 under the Table 1 protocol). In ProgressCompass, each estimator scores the active subtask. The Orienter, Navigator, and Verifier are fixed across replacements. Both Qwen variants directly predict subtask completion percentages (the abs setting in Concern 4).

Why we used RoboMeter for the full loop. At each scoring point, ProgressCompass has the active subtask description and the frames observed so far. It must identify subtask completion online, so the end frame is unknown. RoboMeter provides a progress score from those inputs. The other Table 1 configurations need additional inputs or calibration:

  • VLAC. Our reference-based configuration uses a demonstration video. Supplying a reference by cutting out the corresponding subtask from the test video would expose its annotated boundaries. VLAC implementation
  • ProgressLM. The evaluated setup uses demonstration frames. ProgressLM also supports textual demonstrations. The loop has a description of the active subtask, and providing a demonstration for each new subtask would require an additional source of information. ProgressLM implementation
  • Robo-Dopamine. Our Table 1 fusion averages the incremental, forward, and backward predictions and uses the final frame of the scored clip as the goal image. With an unknown subtask end, that goal frame is unavailable.
  • TOPReward. It can score the current prefix online using the log probability of an affirmative completion answer. The normalized progress curve in Table 1 uses the minimum and maximum across the completed clip. Using only prefixes seen so far would put each new maximum at or near 100%, even midway through a subtask. To apply a fixed completion threshold to the raw scores, we would need to calibrate them for this use.
Reviewer_DhJ9OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies progress estimation in embodied tasks where the current observation alone is insufficient to determine how much of a task has been completed. It introduces ContextProgress-Bench, comprising 24 manipulation tasks and 120 episodes, covering three forms of context dependence: state recall, sequence tracking, and recurrence disambiguation. The paper evaluates five progress reward models (PRMs) with and without annotated subtask context, then proposes ProgressCompass, which combines a frozen PRM with VLM-based task decomposition, completion verification, and memory management.

With RoboMeter as its backbone, ProgressCompass reduces progress MAE from 25.4 to 9.3 and increases the Spearman correlation coefficient from 0.53 to 0.93. The evaluation also covers instruction–execution mismatches and includes an ablation of the expected transition descriptions used for verification.

Critical strengths
  1. The paper identifies a relevant problem and provides concrete examples for studying it. Repeated actions, transiently visible states, and ordered action sequences can make progress ambiguous when judged from the current frame alone. All three are relevant to embodied task monitoring, and the annotated benchmark makes the corresponding failure modes explicit. This is a useful contribution that goes beyond evaluating progress estimation on tasks where the current scene already reveals the answer.

  2. The autonomous system delivers substantial gains, and the verification ablation is informative. The improvement over the frozen RoboMeter backbone is considerable. In particular, removing the expected transition descriptions from the Verifier increases MAE from 9.3 to 17.5 and the number of stalled episodes from 7 to 40 (Section 5.3). This supports the practical value of verifying explicit completion conditions when tracking progress across subtasks.

Critical weaknesses
  1. The central diagnostic changes several factors at once and therefore does not isolate the effect of context. In Section 3, the condition with context receives annotated subtask descriptions and manually specified boundaries, and scoring is performed on subtask segments rather than full episodes. The oracle results also use each subtask’s ground-truth position in the overall task. Appendix C further varies sampling rates, frame budgets, and some inference settings across conditions. For Robo-Dopamine, using the final frame of each input segment as the target image also changes the visual goal when moving from episode-level to subtask-level evaluation. These changes make it difficult to attribute the reported 77–82% error reduction specifically to better interpretation of context. The self-chained results provide a useful control, but still rely on annotated subtask boundaries; the improvement drops to 7% for ProgressLM and 30% for VLAC. Knowing which subtask is currently active already provides a substantial amount of information about global progress. A simple baseline that uses the supplied subtask index and assigns a fixed within-subtask progress value (e.g., 0.5) would help establish how much additional information the PRM contributes.

  2. The PRM’s contribution within the proposed VLM-based system remains unclear. ProgressCompass adds a 27B Orienter and a 9B Verifier/Navigator, along with planning, verified memory, and explicit step tracking. The main experiments show that this full system outperforms the original PRM. However, freezing the PRM weights does not establish that the gains specifically come from providing context to that PRM. A comparison is missing that uses the same VLM loop but replaces the PRM with a simple step-based progress estimate or a VLM-based estimate of local progress. The ablation of expected transition descriptions supports the verification mechanism, but does not establish the benefit of the proposed division of labor between the PRM and the VLM. This matters because that division of labor is one of the paper’s central methodological claims.

Questions for authors
  1. Clarification of the evaluation target: How is continuous ground-truth progress within each annotated subtask constructed? Is it based on elapsed time, semantic completion, or another rule? Please also clarify how MAE is aggregated across frames, episodes, and tasks. These choices affect the interpretation of the reported errors and the selection of appropriate simple baselines.

  2. Evidence for the context diagnostic: Could you disentangle the effects of semantic context, subtask segmentation, and global progress anchoring while holding frame sampling and inference settings fixed? In particular, how does an oracle predictor that uses the annotated subtask index and assigns a fixed within-subtask progress value (e.g., 0.5) compare with the PRMs supplied with oracle context?

  3. Evidence for the architectural choice: With the same Orienter, Verifier, and memory mechanism, how does performance change when the PRM is replaced by a simple local progress estimator or a VLM? A focused comparison on the existing benchmark would materially affect my assessment of the proposed division of labor between the PRM and the VLM.

5 comments
AuthorsOpenReview ↗

Response to Reviewer DhJ9 (1/n)

Thank you for your helpful comments. We appreciate your recognition of (i) the value of the three context-dependent settings, and (ii) the strong system results and useful verification ablation. We have carefully considered your concerns about the context comparison, the PRM's role, and evaluation. We respond to each below.

Concern 1: The context comparison also changes subtask boundaries and global position, so the gain cannot be attributed to context alone.

Thank you for raising this point. Giving a model the history does not mean it can use that history correctly. It may still forget where a block came from, mistake the first insertion for a later one, or assign similar scores to repeated button presses (Section 3.2, Figure 3). For example, RoboMeter and TOPReward have access to the episode history through their respective sampling procedures, yet their MAEs without context are 25.42 and 30.95. With oracle subtask context, these fall to 4.91 and 6.70 (Table 1).

Our diagnosis tests whether organizing the task into context-specific subtasks helps. It includes annotated boundaries and, in the oracle setting, the true subtask position; the comparison does not isolate the description alone.

Thank you also for asking about accuracy within each subtask. We additionally evaluated each manually annotated subtask separately on its own 0–100 scale. We reset the prediction to zero at the first sampled frame, multiply its subsequent changes by K, and clip to 0–100. This measures how accurately the predicted changes track progress inside that subtask, without retaining the prediction's overall starting level.

MethodWithin-Subtask Progress-Change Error↓
ProgressCompass (based on RoboMeter)22.43
VLAC27.68
Robo-Dopamine27.93
ProgressLM33.45
RoboMeter35.65
TOPReward38.29

Adding ProgressCompass reduces RoboMeter's within-subtask error from 35.65 to 22.43, a reduction of about 37%. The framework therefore improves progress tracking inside subtasks, not just the overall stage estimate.

Concern 2: A predictor that knows the subtask index and always assigns 50% local progress may already perform well.

Thank you for suggesting this baseline. When the standalone PRMs are lost in context, simply guessing “50% complete” at every point is more accurate than their own predictions. With ProgressCompass supplying and updating context, the model’s predictions become more accurate than this fixed guess. Each constant baseline is evaluated on the same points as its corresponding method.

MethodModel Error↓Always Predict 50%↓Better Than Constant 50%?
ProgressCompass (based on RoboMeter)22.4325.61Yes
VLAC27.6827.20No
Robo-Dopamine27.9325.58No
ProgressLM33.4525.88No
RoboMeter35.6525.62No
TOPReward38.2925.75No

RoboMeter's predicted changes give an error of 35.65, worse than the fixed guess at 25.62. ProgressCompass reverses this pattern: its error is 22.43, about 12% below its constant baseline of 25.61. This is consistent with our context diagnosis: after the framework supplies and updates context, local score changes become more useful than a fixed halfway guess.

Concern 3: Sampling, frame budgets, and some inference settings change between conditions.

Thank you for raising this point. We follow each model's own input format and inference procedure, as detailed in Appendix C. The appendix reports the sampling settings, frame budgets, and post-processing for both conditions, including Robo-Dopamine's intervals of 5 and 10 and TOPReward's frame limits of 96 and 48. Without context, each model scores the full episode; with context, it scores the annotated subtask clips. We will make the reference to Appendix C clearer in the main text.

Concern 4: Robo-Dopamine receives a different goal image when evaluation changes from full episodes to subtask segments.

Thank you for pointing this out. You are correct about Robo-Dopamine: its goal image changes with the input segment. For each subtask, we construct the inputs required by each baseline, including the goal image for Robo-Dopamine, so that the model can be evaluated in its supported input format. The purpose is to test how each model performs when the relevant subtask context is supplied. For Robo-Dopamine, this includes a subtask-specific visual goal, so its improvement should be read as the benefit of that full input condition, not of the text description alone.

AuthorsOpenReview ↗

Response to Reviewer DhJ9 (2/n)

Concern 5: Keeping the PRM frozen does not show that it adds value beyond the VLM loop. Can the same loop use a VLM instead?

Thank you for suggesting this test. Yes, the PRM adds value even when context, verification, and memory are kept. We replaced only the progress estimator with Qwen3.5-27B or Qwen3.5-9B. We tested both absolute and delta prediction on the same 120 episodes. Here, “abs” predicts the current subtask completion percentage. “Delta” predicts the change since the previous sample, which is then accumulated. All four VLM replacements use temperature 0.7 and top_p 0.8.

Progress EstimatorState↓Count↓Sequence↓Overall MAE↓Spearman ρ↑Pearson r↑
RoboMeter-4B10.507.5210.529.820.9410.933
Qwen3.5-27B, abs12.2314.8411.9812.740.9150.908
Qwen3.5-9B, abs14.9615.6813.3214.440.9070.897
Qwen3.5-9B, delta19.6719.0515.7117.850.8650.853
Qwen3.5-27B, delta23.1918.8715.4818.930.8970.872

The best VLM replacement is Qwen3.5-27B with absolute prediction. Its MAE is 12.74, compared with 9.82 for RoboMeter-4B. The specialized 4B PRM therefore has about 23% lower error, despite being much smaller. Its error is lower in every task category. The largest gap is on Count: 7.52 versus 14.84, about 49% lower error. Absolute prediction also works better than delta prediction for both VLM sizes.

These results show that RoboMeter scores progress more accurately than the tested Qwen replacements inside the same loop. We separately test whether VLMs can predict progress directly, without the loop.

We also tested direct progress prediction, without the ProgressCompass loop. The new results show a clear gap between the tested open-source VLMs and frontier models.

MethodState MAE↓Count MAE↓Sequence MAE↓Overall MAE↓Spearman ρ↑Pearson r↑
ProgressCompass (based on RoboMeter)10.507.5210.529.820.9410.933
Opus 5.5, direct prediction (batched)8.616.787.527.720.9640.951
GPT-6 Astra, direct prediction (batched)6.294.717.366.370.9750.961
Qwen3.5-27B, direct prediction26.7821.8012.1119.420.7190.702
Qwen3.5-9B, direct prediction40.9835.7925.0833.060.4330.456
RoboMeter, direct prediction under the comparison settings30.1724.9222.3625.650.5130.504

ProgressCompass has about 49% lower MAE than directly prompting Qwen3.5-27B: 9.82 versus 19.42. Direct Qwen3.5-9B prediction is worse, at 33.06. Thus, directly asking a larger open-source VLM for progress does not match the framework's accuracy in these runs.

Frontier models do better: Opus 5.5 reaches 7.72 and GPT-6 Astra reaches 6.37, both below ProgressCompass's 9.82. We therefore limit the claim to improving the tested open-source models; we do not claim that all VLMs are poor progress estimators or that ProgressCompass beats the frontier models.

The two experiments answer different questions: direct prediction tests the complete alternative, while replacement inside the loop tests which progress estimator works best with the other modules kept.

Concern 6: How is continuous ground-truth progress constructed, and how is MAE averaged?

Thank you for asking us to clarify this. Within each annotated subtask, ground-truth progress increases linearly with frame position. Each of the K subtasks receives an equal share of total progress: subtask k runs from k/K × 100 to (k+1)/K × 100, with k starting at 0. We compute MAE separately within each annotated interval that contains at least three sampled points, then average over intervals. The replacement experiment covers 552 intervals across 120 episodes: State has 50 episodes and 190 intervals; Count has 20 episodes and 129 intervals; Sequence has 50 episodes and 233 intervals.

Thank you for the additional experiments and clarifications. The VLM replacement experiments provide useful evidence for the contribution of RoboMeter.

However, my main concerns remain:

W1: The oracle setting provides ground-truth subtask boundaries and step positions, which are closely related to how the progress labels are constructed. The new within-subtask 50% baseline uses a different evaluation metric and does not resolve this concern. A simple step-index-based baseline under the original global MAE protocol is still needed.

W2: While the VLM replacement results are helpful, a simple step-based estimator within the same agentic loop is still missing, making it difficult to isolate the PRM's contribution.

I appreciate the authors' efforts, but these controls are important for supporting the paper's central claims. I therefore maintain my original rating.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General Response

We sincerely appreciate the time and effort the reviewers have dedicated to evaluating our paper. Thank you for the helpful advice and constructive comments. We are grateful that the reviewers recognized the following aspects of our work:

  • The paper identifies an important problem in long-horizon progress estimation: the current frame may not contain enough information to judge task progress (DhJ9; NYd5).

  • ContextProgress-Bench makes context-dependent failures explicit, with concrete settings for state recall, sequence tracking, and recurrence disambiguation (DhJ9). The benchmark demonstrates limitations across multiple existing PRMs (NYd5).

  • ProgressCompass substantially improves progress estimation without retraining the RoboMeter backbone (DhJ9; pMqX; NYd5). The results show gains in both accuracy and temporal consistency (pMqX).

  • The expected-transition ablation provides useful evidence for checking explicit completion conditions when tracking progress across subtasks (DhJ9).

  • The proposed agentic pipeline is relatively simple and achieves strong gains without training (NYd5).

During the rebuttal period, we spent considerable time and care making the paper easier to read and strengthening the evidence behind its claims. We would be grateful if you could take the time to look through the revision and our responses.

AuthorsOpenReview ↗

Follow-up on the ProgressCompass rebuttal

Dear Reviewers,

Thank you for taking the time to review our paper. We’ve posted our rebuttal with additional experiments and clarifications, and we’d appreciate it if you could take a look.

If our response addresses your concerns, we hope you’ll consider updating your rating. If any concerns remain that affect your rating, we’d be grateful if you could let us know which points still need clarification or further evidence.

Thank you again for your time and feedback.

Best,

The ProgressCompass authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.