- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper asks whether a component that scores well under teacher forcing is also a good choice when the system runs in a loop. The authors compare transition scorers using the same frozen representations and state-access mechanism, then measure the regret of selecting a scorer by its local accuracy. Gold-prefix and continuation-training experiments help locate the failures. On AppWorld, a route-only metric selects a poor backbone under two agent scaffolds; including call arguments recovers the best aggregate backbone on the evaluated tasks. The paper's main contribution is the evaluation study and its controls.
Critical strengths
The controls are the strongest part of the paper. State access and termination are measured separately, and the gold-prefix experiments help distinguish a bad conditional transition from errors caused by earlier generated actions. The continuation versus STOP-only comparison is also useful: it shows what additional supervision repairs, and at which horizon. The negative results for wider beam search and margin-guided example selection help keep the conclusions grounded (Section 5 and Appendices J-L).
Selection regret is a sensible quantity to report here. A high rank correlation can look reassuring while still giving the wrong deployment choice, and the paper makes that distinction concrete. AppWorld adds a real end-to-end example. I also appreciate that the authors separate this experiment from the frozen-prefix checks and state the limits of the same-task comparison.
Critical weaknesses
My main difficulty is understanding exactly what is being compared in Table 1. For Gemma/Scorer B, teacher-edge accuracy is 51.3%, while exact constrained-route accuracy is 97.5%. If both metrics used the same routes and admissible candidates, a correctly decoded route would also give correct gold-prefix decisions along that route. The caption says the teacher metric includes STOP and comes from an all-valid-edge evaluator, but I could not reconstruct its numerator, denominator, or evaluated edge population from the paper. There may be a straightforward explanation. Until it is clear, though, I am unsure how much of the headline gap comes from a weak local predictor and how much comes from different evaluation definitions.
The AppWorld result also needs a clearer distinction between choosing one backbone for the whole pool and choosing separately for each task. Appendix M reports tied top full-call scores on 80 of 90 tasks, yet the full-call metric selects Qwen3-30B on 88 tasks. With so many ties, the per-task result could depend heavily on the tie-breaking rule. The aggregate ranking is still useful, but the reported selection frequency and regret are harder to interpret. The authors already acknowledge that selection on unseen tasks has not been tested; I would keep the practical conclusion within that scope.
Questions for authors
-
How is teacher-edge accuracy in Table 1 computed? Please specify the evaluated edges, candidate masks, STOP handling, and averaging. What is the gold-prefix skill accuracy on exactly the routes and candidates used for constrained decoding?
-
What happens to AppWorld full-call selection regret under random tie-breaking? How much of the 88/90 Qwen3-30B selection frequency comes from tied scores?
-
Can you break down local scores and regret by the declared core/development/confirmation partitions? Selecting on one partition and evaluating the chosen backbone on another would help establish what the selection result supports.