- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper asks when a real-time LLM agent should reuse an inherited action, update it cheaply, or call a slower thinking model. It first runs paired prompt interventions on Real-Time Reasoning Gym states (Freeway, Snake, Overcooked). Carrying over the previous action raises one-step safety while that action is still safe and sharply lowers it once the action has become unsafe, an effect larger than that of decision age. Oracle validity labels and explicit action outcomes help, but the model's own validity judgments do not. The paper then builds a cascade. A LoRA-tuned Qwen3.5-27B with thinking disabled outputs an action, optionally preceded by a retain/refresh/rerank label. A controller uses Laplace-smoothed counts over exact-match feature buckets to estimate the chance that this action is wrong but AgileThinker's is right, and falls back to AgileThinker when that chance exceeds a validation-selected threshold. Accuracy is offline and one-step: the action must lie in a simulator-derived set of actions that avoid reset or termination. On fresh seeds, selective fallback reaches 96.04% versus 96.65% for always calling AgileThinker, escalating 43.75% of requests and using 57.33% fewer completion tokens.
Critical strengths
S1: The evaluation is careful and candid. Splits are seed-separated, the threshold is frozen before testing, and intervals use state-cluster bootstraps. The authors decline to claim non-inferiority, and they audit both the non-discriminating Overcooked slice and their own label noise (Appendices H, K). The numbers are internally consistent: I recomputed every fresh-seed headline figure from Table 18.
S2: The Section 4 interventions cleanly separate a change in validity from decision age, and an explicit recommendation from carried-over facts. They document a practically relevant anchoring failure: 88.7% of carryover-induced unsafe flips return to the inherited action (Table 2), and the model's own validity judgments are below chance, at 47.9% balanced accuracy (Table 3).
Critical weaknesses
W1: Most headline efficiency comes from a slice where every action counts as correct. Fresh-seed Overcooked accepts every legal action (lines 854–857). I computed from Table 18 that this 39% of the test supplies 64.5% of the token savings. On Freeway and Snake, I recomputed 71.8% escalation and 32.4% token savings, with a 1.0-point accuracy gap (lines 858–859).
W2: The controller's benefit is weakly supported. Its score ignores harmful fallbacks, though Eq. 17 and the Bayes-optimal rule (Jitkrittum et al., 2023) depend on . On Freeway I counted 82 fallbacks (Table 18) that lose 8 correct actions net. The fixed-budget win is 4 versus 1.69 of 7 errors (Table 17); I computed that random routing recovers 4 or more with probability 0.06.
W3: Beyond a standard cascade, the paper's own components do not carry the result. Deferring after seeing the cheap output, with a "cheap wrong, strong right" target, is established (Gupta et al., 2024), contrary to lines 113–114. The structured label matches Action-only accuracy once outputs parse (Table 5), and routing on it loses to random (Table 16). The fresh-seed checkpoints skip diagnostic selection (Table 10).
Questions for authors
- On the fresh-seed test, can you report results for Freeway and Snake only, alongside a random router at the same call rate and a deferral baseline based on the cheap model's action log-probability?
- Why does the score target recoverable errors rather than the net benefit from Eq. 17? Would a net-benefit score avoid the Freeway regression?
- Can the policy be run online in Real-Time Reasoning Gym, where the environment advances during thinking, and scored by episode return as in Wen et al. (2025)? One-step reference membership cannot show long-horizon effects, and Overcooked shows it can fail to discriminate between actions at all.
- How is the "Diagnostic" training set selected? Section 6.2 never defines it. Does it improve the fresh-seed system, whose checkpoints were trained without it (Table 10)?
Minor 1. What's the meaning of contract in "Full contract" in Fig3 Page6? I'm a little confused when reading this.
Dear Reviewers,
I hope this message finds you well.
I would like to remind you that we are currently in the rebuttal phase of the paper review process. During this stage, authors may respond to the comments and feedback you provided. If you receive any responses from the authors, please consider providing your feedback to ensure a smooth review process.
Thank you for your hard work and dedication throughout this review process. Your insights are invaluable in enhancing the quality of the submissions, and we truly appreciate your involvement.
If you have any questions or need further assistance, please do not hesitate to reach out.
Best regards,
AC