- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
SCOPE lets a fixed language model repair robot control programs from execution feedback. A critic labels each run against a shared diagnostic graph of task checks and prerequisites to find a "failure frontier", and the model picks a local repair, stage redesign, or strategy regeneration. After a repair succeeds and changes the accepted program, up to three stored successful runs are replayed against recorded API requests, responses, and state changes. A full match with normal termination certifies success on that recorded instance, under determinism and API-boundary assumptions. Sparse environments running on uncertified configurations give further repair evidence. Frozen programs reach 80.66% on LIBERO-PRO (+6.51 over ASPIRE on the same GLM-5.3-Flash backbone) and 84.48% on Robosuite (+3.48 over ASPIRE with Opus 4.6). On a physical robot, three tasks give 40/60 successes versus 24/60 for ASPIRE. Ablations on two settings and replay time costs are also reported.
Critical strengths
- Same-backbone comparison with three complete learning runs. SCOPE improves all six LIBERO-PRO settings, most on Goal-Task (+13.07) and Spatial-Pos (+10.93), and all seven Robosuite task means.
- Removing any of the three components lowers success in both ablated settings.
- The theorem's assumptions and scope, and the cost and safety limitations, are stated candidly.
Critical weaknesses
- Certification is narrower than its framing. A full match needs every request and pre-call state to equal the tape, so a certified run is one in which the edit altered no API request. Only 38.9-60.6% of checks certify (Table 5); the rest include behavior-changing edits, where regressions arise. Certification also does not gate acceptance (Sec. 3.3, App. H), so "certification of program edits" overstates what the method does. The theorem is a short induction under strong assumptions, and the App. G bound is immediate.
- The source of the gain is not isolated. Ablations cover two of six settings, chosen for the largest gains (App. C), with three runs; drops (3.00-6.80) are comparable to SDs (3.72-9.10). No variant removes all three components, and each single removal keeps 6.27-10.07 of the 13.07-point Goal-Task gain and 4.86-7.66 of the 10.93-point Spatial-Pos gain over ASPIRE (my arithmetic from Table 4). I found no model calls, tokens, or environment episodes per method, though SCOPE adds critic, diagnosis and planning calls, and VLA baselines keep their original protocols, while SCOPE uses 15 development seeds with environment feedback (App. B). The replay cost (4.62-6.59% of environment time) has no re-execution baseline. On hardware, one frozen program per method gives 40/60 vs 24/60; by an unpaired Fisher exact test (my calculation) no single task is significant (p about 0.11, 0.33, 0.11).
Questions for authors
- Which model is the critic, how accurate are its node labels, and are its calls in the budget?
- What are model calls, tokens and environment episodes per method in simulation?
- Does the 4.62-6.59% replay ratio include replays that stop at the first divergence?
- On hardware, what were the certification and divergence rates, and what does a match mean when execution is not deterministic?
- Of the witnesses that diverge, what share fail when diagnosed in the environment? Requests that could change my score (I would raise it if most of the gain remains, and lower it if the gain shrinks under matched calls):
- An all-components-removed ablation on all six settings, ASPIRE with matched model calls, replay versus re-running the same witnesses in the environment, and SCOPE with a second backbone. Suggestion, not part of my decision:
- State in the abstract that certification is feedback, not an acceptance gate.
3 comments
W2 / Q3 / Q6(c). Replay timing and direct re-execution
Yes, Table 5 includes checks that stop at the first divergence. Its 4.62–6.59% divides replay time by the original environment time for the corresponding witnesses. For a new offline timing test, each pair consisted of an edited program from full SCOPE's first run and a historical success it had checked. We sampled up to 20 pairs per setting and status (certified or divergent) and ran each edited program both against its tape and in the simulator restored to the original witness state and randomness. Across 272 pairs, replay took 1,309 s and live execution took 21,798 s (6.00%, excluding restoration).
To test the effect on learning, we compared full SCOPE with SCOPE-Reexec, which replaces historical replay during search. After each accepted source change, each learner samples up to three successes from its own history. SCOPE checks their tapes and runs sparse diagnostics on eligible unresolved cases. Reexec runs every selected configuration in the environment and uses failures for follow-up repair. Both used GLM-5.3-Flash with at most three main repairs and one follow-up per development configuration. We froze each learner's final program after one run per task and tested both on the same held-out seeds. Full SCOPE and Reexec respectively achieved 79.30% and 78.97% on LIBERO-PRO and 85.14% and 84.86% on Robosuite. Using each task's duration and the same parallel schedule gives elapsed learning times of ≈164.9 hours for SCOPE and ≈182.6 hours for Reexec.
W1 / Q5. What fraction of divergent witnesses actually fail?
For online environment diagnostics, the first run recorded 23 failures among 249 fresh resets of unresolved configurations. The task-level counts bound the failure rate for diagnosed divergences at 6.64–9.54%. Of the eight unavailable checks, one was diagnosed successfully and the remaining seven could account for at most seven failures. Allocating the failures within each task gives a lower bound of 16/241 and an upper bound of 23/241 for the divergent cases.
For the original witness instance, the offline pairs used in the timing test above also give a direct task outcome. On each restored instance, the edited program had to terminate normally and satisfy the benchmark evaluator with live API responses. 14/132 divergent pairs failed (10.61%). All 140 sampled certified pairs succeeded.
W1 / W2 / Q4. What did a hardware match establish?
We counted SCOPE's historical checks during one development run on each of the three physical tasks. Of 28 checks, 13 matched (46.4%), 14 diverged (50.0%), and one was unavailable. A match means the edited program reproduced the complete API interaction with the recorded observations, responses and public-state changes. Sensors, object placement and contact can vary on a new execution. The 13 matches resolved the historical checks locally, without manually rebuilding the scenes. All replay took 126.3 s. Of 13 eligible unresolved configurations tested physically, four failed.
We froze one program per method per task and evaluated each on 20 new placements. SCOPE succeeded on 40/60 placements and ASPIRE on 24/60.
W1 / Q7. Certification in the abstract. We will state the order explicitly. “Once a program succeeds on the current configuration, SCOPE accepts it and replays selected historical successes to guide later repairs.” Appendix H gives the recorded-instance guarantee under faithful capture and deterministic execution. Appendix G explains how often an edit must change behavior to affect aggregate success.
Thank you for the authors' response. I will maintain my score.
Thank you for the specific comparisons you suggested. We ran the token-budget comparison, the joint ablation, and the re-execution control with GLM-5.3-Flash, using one learning run per task and the original held-out seeds. We apologize for the time it took to complete these experiments and bring the results together, and hope the delay has not inconvenienced you.
W2 / Q2 / Q6(b). Calls, tokens and environment executions for SCOPE and ASPIRE
We matched each task's input plus output token budget to full SCOPE's first learning run. Model calls are reported as a learning cost. For held-out success, we compared SCOPE with ASPIRE at two endpoints on each task, when ASPIRE first reached or exceeded SCOPE's input plus output token usage, and when ASPIRE completed its full search. The token count included auxiliary requests and retries. At the threshold, ASPIRE could finish validating a candidate it had already generated and freeze its incumbent. If search finished earlier, we used its final program. At both endpoints we evaluated frozen programs on the same 50 seeds per LIBERO-PRO task–setting pair and 100 per Robosuite task.
At the token threshold, SCOPE gained 16.77 points on LIBERO-PRO and 11.00 on Robosuite. Allowing ASPIRE to finish reduced those differences to 8.73 and 5.57 points in the same run.
Across the 67 task–setting pairs, full SCOPE made 3,732 calls and used 1,975 environment executions. Full ASPIRE used about 10,400 calls. The full-run token budgets were ≈3.27B input and ≈0.28B output for SCOPE and ≈4.60B input and ≈0.52B output for ASPIRE. The cutoff experiment used token usage returned by each request. Both learners used the same development configurations. The VLA rows report published scores under their original protocols.
W2 / Q6(a). Joint contribution of the three forms of repair guidance on Goal–Task and Spatial–Pos
In Table 4, each single removal lowered mean success across three runs in both settings. Two-sided Welch tests using the reported run means and SDs give unadjusted p-values from .137 to .594 for the six comparisons.
We compared full SCOPE's first run with a new run that removed all three forms of repair guidance together. We hid certification feedback, the first divergence, and graph and frontier evidence from the planner and code generator, and omitted the choice of repair level. Both learners kept the same repair limits, raw observations, API traces, internal replay, diagnostics and follow-ups. Each learner ran to natural termination, and we froze one program per task. Goal–Task and Spatial–Pos each cover 10 tasks. For each task we evaluated both programs on the same 50 held-out seeds, giving 500 trials per setting.
Q6(d). Second backbone. We evaluated SCOPE only with GLM-5.3-Flash. Opus 4.6 in Table 2 is a published ASPIRE result.
W2 / Q1. Which critic is used, how accurate is it, and is it counted?
The critic uses GLM-5.3-Flash. Its 2,235 calls, including retries, count toward SCOPE's 3,732 calls. We measured label consistency against the execution trace and task prerequisites. We did not score labels against independent annotations. The critic produced a usable assessment on 1,904/1,975 executions (96.4%). Checking evidence windows, prerequisite order and invariant coverage changed 5,554/42,667 node labels (13.0%). Masking graph and frontier evidence lowered held-out success by 6.80 points on Goal–Task and 5.47 on Spatial–Pos over three runs.