- Rating
- 2(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
SpecAgent uses a small model to draft actions and a larger model to review them. The small model executes read-only tool calls during drafting and pauses before an action that changes the environment. The larger model reviews the segment, keeps an accepted prefix, and repairs the first rejected step. Experiments on tau^2-Bench, AutomationBench, and MCP-Atlas report success rate, wall time, estimated computation and memory traffic, and API cost. The strongest result is close to Strong-only success on tau^2-Bench with substantially lower wall time.
Critical strengths
- Executing reads while holding writes for review is a practical design for interactive tasks. On tau^2-Bench, it substantially reduces wall time with little observed loss in success rate.
- The paper reports the additional computation together with latency savings. This helps the reader understand the cost of the method and the serving conditions where it may help.
Critical weaknesses
- DualSpec (2026) already uses a strong critic to check drafted actions and regenerates rejected actions. It is cited but not evaluated.
- The main table gives only one configuration for each collaboration method. Verification frequency, escalation settings, and reasoning budget can all change success rate and cost. These results are not enough to establish a better quality-efficiency frontier.
- The verifier is evaluated mainly through final task success. There are no false-acceptance, false-rejection, or repair-success rates.
- Read-only chunks reaching K steps are committed without review. Replies on tau^2-Bench are also sent before checking. A later rejection cannot remove those earlier steps or withdraw a reply.
- The design ablations are limited to Retail, and K is not varied. Removing fused replacement also causes a clear accuracy drop. Why a separate repair step changes task success so much needs explanation.
- The figures provide little direct evidence about semantic review. Figure 1 illustrates the intended workflow, while Figures 2-4 summarize trade-offs and ablations. Recorded trajectories showing accepted alternatives, missed errors, and successful or failed repairs would make the claimed advantage easier to assess.
- The quality trade-off is larger on the other benchmarks. AutomationBench success falls from 15.2% for Strong-only to 10.5%. On MCP-Atlas, the loss is 8.9-12.9 percentage points, and wall time does not consistently improve. The current analysis does not explain where the method remains useful and where reviewing weak drafts is insufficient.
After rebuttal: I raise my rating from 1 to 2. The DualSpec and step-level comparisons address my main concern, and the logged cases make the verifier’s behavior clearer. The results on tau²-Bench are more convincing now, but the success-rate gaps on AutomationBench and MCP-Atlas remain substantial. It is still unclear when SpecAgent is preferable to using the strong model directly.
Questions for authors
see weaknesses
7 comments
Response to Reviewer XXqQ (2/6)
[W2, W5] Verification frequency, escalation, and fused repair
Following your suggestion, we vary these settings on both τ² domains, on the same test tasks (Retail 104, Airline 40) with 3 runs each. Δ is the paired difference in Pass@1 from SpecAgent with a 95% CI.
Verification frequency: how much the Strong model reviews. Step-level review checks every action; pending-write-only review lets the verifier reject only the held write.
| Pass@1 | Δ [95% CI] | Strong calls | Est. PFLOP | Est. HBM-read (TB) | |
|---|---|---|---|---|---|
| Retail | |||||
| Step-level review | 86.2 | −1.0 [−5.4, +3.5] | 10.4 | 0.94 | 157–165 |
| Pending write only | 81.7 | −5.4 [−9.9, −1.0] | 3.1 | 0.81 | 117–120 |
| SpecAgent | 87.2 | – | 3.0 | 0.78 | 103–106 |
| Airline | |||||
| Step-level review | 79.2 | −5.8 [−12.5, +0.8] | 12.3 | 1.24 | 238–248 |
| Pending write only | 76.7 | −8.3 [−17.5, 0.0] | 3.7 | 1.09 | 176–179 |
| SpecAgent | 85.0 | – | 3.2 | 1.02 | 149–152 |
On both domains SpecAgent is the most accurate and the cheapest. Reviewing every step multiplies Strong calls by 3.5–3.9× without higher success; reviewing only the write is as cheap but loses 5–8 points.
Verification frequency: chunk size K (Retail).
| Pass@1 | Δ [95% CI] | Strong calls | Est. PFLOP | Est. HBM-read (TB) | |
|---|---|---|---|---|---|
| K = 2 | 84.6 | −2.6 [−6.7, +1.6] | 3.2 | 0.81 | 111–114 |
| K = 4 | 83.7 | −3.5 [−8.0, +0.6] | 3.2 | 0.81 | 118–121 |
| K = 6 (default) | 87.2 | – | 3.0 | 0.78 | 103–106 |
| K = 8 | 81.7 | −5.4 [−9.9, −1.0] | 3.1 | 0.81 | 120–123 |
The default K = 6 attains the highest success and the lowest resource use. We report the sweep rather than claim a single optimal K.
Escalation after a rejection: fused repair. With fused repair, the verifier fixes a rejected step in the same pass; without it, every rejection escalates to a separate Strong turn.
| Pass@1 | Strong calls | Separate repair turns | Est. PFLOP | |
|---|---|---|---|---|
| Retail: SpecAgent → fused off | 86.0 → 82.5 | 3.2 → 3.8 | 55 → 95 | 0.83 → 0.86 |
| Airline: SpecAgent → fused off | 85.0 → 83.3 | 3.2 → 3.9 | 95 → 178 | 1.02 → 1.12 |
Retail: the paper's same-batch ablation (one run per task), where fusion is disabled by removing the replacement field from the verifier's output. Airline: new runs that keep the verifier prompt and only discard replacements at runtime. Repair turns are totals over all runs.
On Retail, removing the replacement field changes the design as a whole: the verifier also rejects more often (16.5% → 20.2% of decisions), and every rejection becomes a hand-off to a Strong turn that sees only the retained prefix and a textual reason. More rejections and more hand-offs give more chances to drift. On Airline, where the verifier's behavior is unchanged (288 vs. 295 reviews), separate repairs still nearly double and success drops in the same direction, by less. Fused repair thus helps in two ways: it keeps the verifier's diagnosis and the fix in one pass, and it removes a Strong turn per rejection. For DualSpec, whose escalation is controlled by a calibrated threshold, the comparison is: 86.9 vs. 87.2 (Retail), 86.7 vs. 85.0 (Airline), and 46.3 vs. 45.0 (VitaBench) Pass@1, with 6–10× more Strong calls than SpecAgent.
Revision: We have added the review-scope controls (Table 11) and the K and fused-off ablations (Table 12) to Appendix F.1 of the revised version, and §5 now explains the fused-off result through hand-offs. The original Retail fused-off ablation was already in Figure 4 of our submission.
Response to Reviewer XXqQ (3/6)
[W3] False-acceptance, false-rejection, and repair-success rates
We human-annotated 300 verifier decisions from logged runs on τ²-Bench, VitaBench, and MCP-Atlas. The annotators are three Ph.D. students in computer science who are familiar with tool-using agent benchmarks and received systematic training on our written annotation guidelines before labeling; each annotated a disjoint set of 100 cases. Each case includes its observations, draft, verdict, replacement, executed action, and outcome.
| Rate | Value |
|---|---|
| False acceptance (incorrect accepted prefixes) | 4 / 204 (2.0%) |
| False rejection (incorrect rejections) | 29 / 149 (19.5%) |
| Fused replacement semantically correct | 67 / 85 (78.8%) |
| Executed repair correct (repair success) | 127 / 145 (87.6%) |
Errors are skewed toward caution: false rejections are about 7× more frequent than false acceptances, and they mainly cost an extra repair. In 15 of the 29 false rejections, the action finally executed was still correct. After a correct rejection, the executed repair was correct in 112 of 116 cases (96.6%). Incorrect fused replacements do occur (18 of 85) and are reported as such.
[W6] Recorded trajectories for each category
We give six logged trajectories, covering accepted alternatives, successful repairs, a missed error, and a fused-repair error, in the following comment.
Revision: We have added these rates to §5 (Table 6) of the revised version, with the annotation protocol, annotator background, and recorded cases in Appendix F.3.
Response to Reviewer XXqQ (1/6)
hank you for the review. Following your suggestions, we added the evaluations below, ordered by the weaknesses you raised.
[W1] DualSpec: a different design, and now evaluated
DualSpec and SpecAgent share only the generic draft–verify–fallback pattern. They differ in what is drafted, when and how the strong model is used, and the environments they target:
We nonetheless evaluated DualSpec with the same model pair, keeping its parallel drafts, original Yes/No critic, and fallback. DualSpec accepts a draft by comparing its critic score with a threshold that must be calibrated on development tasks. We therefore split each domain into development and test tasks: Retail 10/104, Airline 10/40, VitaBench (Delivery) 20/80. The threshold was calibrated once on the Retail development tasks and used unchanged in all domains. Every method is compared on the same test tasks, with 3 runs per task on τ² and 2 on VitaBench (Delivery), matching our original VitaBench protocol. We also include a step-level semantic review variant of SpecAgent as a further per-step reference.
Success is statistically indistinguishable between DualSpec and SpecAgent (paired 95% CIs of the difference include zero in all three domains), while DualSpec uses 6–10× more Strong calls, 1.9–2.8× the estimated HBM-read, and 1.3–1.6× the wall time of SpecAgent. The gap follows from the design: DualSpec puts the strong model on every step, while SpecAgent puts it only at state-changing boundaries. The table below, now Table 2 in §2, places both among the related methods:
Multi-step verification: one pass keeps the longest valid prefix of several drafted steps. Environment protection: state-changing actions wait for verification. △: side-effect-free actions only; –: not applicable.
Revision: We have added DualSpec to Table 3 (* = test split) and §4.2 of the revised version, its setting to Appendix E.2, the same-task comparisons to Appendix F.1 (Tables 11 and 13), and the design comparison to §2 (Table 2).