acceptodds
Under review as a conference paper at ICLR 2027

Reuse, Refresh, or Replan? Selective Decision Updating for Real-Time LLM Agents

Abstract

Real-time large language models (LLMs) agents must decide whether to reuse a previous decision or update it as the environment evolves. Reuse saves computational cost when the decision remains valid, but an inherited action can mislead the agent once the state changes. Controlled intervention experiments show that prior-action carryover benefits agent performance under safe conditions, yet induces decision anchoring once the action becomes unsafe. Explicit action-outcome information improves the selection of alternative actions. Motivated by these observations, we study selective decision updating supervised via simulator replay. A low-cost model predicts the action for the current state alongside an action-treatment label, while a controller estimates whether further reasoning could correct the action before fallback. The controller triggers AgileThinker if the candidate is unusable or its estimated risk exceeds a threshold selected through validation. On held-out seeds and states, selective fallback trails full AgileThinker by 0.6 percentage points in one-step reference-set accuracy, while reducing mean completion tokens by nearly 60% and invoking the thinking model on about 44% of requests. Collectively, these results show how the expected benefit of further reasoning can guide computational resource allocation for agents updating decisions in dynamic environments, within the evaluated offline, one-step task.

Primary area:
applications to robotics, autonomy, planning
Keywords:
real-time reasoning, adaptive inference

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_wmsMOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper asks when a real-time LLM agent should reuse an inherited action, update it cheaply, or call a slower thinking model. It first runs paired prompt interventions on Real-Time Reasoning Gym states (Freeway, Snake, Overcooked). Carrying over the previous action raises one-step safety while that action is still safe and sharply lowers it once the action has become unsafe, an effect larger than that of decision age. Oracle validity labels and explicit action outcomes help, but the model's own validity judgments do not. The paper then builds a cascade. A LoRA-tuned Qwen3.5-27B with thinking disabled outputs an action, optionally preceded by a retain/refresh/rerank label. A controller uses Laplace-smoothed counts over exact-match feature buckets to estimate the chance that this action is wrong but AgileThinker's is right, and falls back to AgileThinker when that chance exceeds a validation-selected threshold. Accuracy is offline and one-step: the action must lie in a simulator-derived set of actions that avoid reset or termination. On fresh seeds, selective fallback reaches 96.04% versus 96.65% for always calling AgileThinker, escalating 43.75% of requests and using 57.33% fewer completion tokens.

Critical strengths

S1: The evaluation is careful and candid. Splits are seed-separated, the threshold is frozen before testing, and intervals use state-cluster bootstraps. The authors decline to claim non-inferiority, and they audit both the non-discriminating Overcooked slice and their own label noise (Appendices H, K). The numbers are internally consistent: I recomputed every fresh-seed headline figure from Table 18.

S2: The Section 4 interventions cleanly separate a change in validity from decision age, and an explicit recommendation from carried-over facts. They document a practically relevant anchoring failure: 88.7% of carryover-induced unsafe flips return to the inherited action (Table 2), and the model's own validity judgments are below chance, at 47.9% balanced accuracy (Table 3).

Critical weaknesses

W1: Most headline efficiency comes from a slice where every action counts as correct. Fresh-seed Overcooked accepts every legal action (lines 854–857). I computed from Table 18 that this 39% of the test supplies 64.5% of the token savings. On Freeway and Snake, I recomputed 71.8% escalation and 32.4% token savings, with a 1.0-point accuracy gap (lines 858–859).

W2: The controller's benefit is weakly supported. Its score ignores harmful fallbacks, though Eq. 17 and the Bayes-optimal rule (Jitkrittum et al., 2023) depend on erec−ge^{rec} - g. On Freeway I counted 82 fallbacks (Table 18) that lose 8 correct actions net. The fixed-budget win is 4 versus 1.69 of 7 errors (Table 17); I computed that random routing recovers 4 or more with probability 0.06.

W3: Beyond a standard cascade, the paper's own components do not carry the result. Deferring after seeing the cheap output, with a "cheap wrong, strong right" target, is established (Gupta et al., 2024), contrary to lines 113–114. The structured label matches Action-only accuracy once outputs parse (Table 5), and routing on it loses to random (Table 16). The fresh-seed checkpoints skip diagnostic selection (Table 10).

Questions for authors
  1. On the fresh-seed test, can you report results for Freeway and Snake only, alongside a random router at the same call rate and a deferral baseline based on the cheap model's action log-probability?
  2. Why does the score target recoverable errors rather than the net benefit erec−ge^{rec} - g from Eq. 17? Would a net-benefit score avoid the Freeway regression?
  3. Can the policy be run online in Real-Time Reasoning Gym, where the environment advances during thinking, and scored by episode return as in Wen et al. (2025)? One-step reference membership cannot show long-horizon effects, and Overcooked shows it can fail to discriminate between actions at all.
  4. How is the "Diagnostic" training set selected? Section 6.2 never defines it. Does it improve the fresh-seed system, whose checkpoints were trained without it (Table 10)?

Minor 1. What's the meaning of contract in "Full contract" in Fig3 Page6? I'm a little confused when reading this.

Reviewer_QXWPOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies real-time LLM agents that inherit an action decided at an earlier step and must decide whether to reuse it, update it cheaply, or replan with a stronger thinking model. It first runs paired diagnostics on Freeway, Snake and Overcooked states from Realtime-Gym. These show that whether the inherited action is still valid matters more than how old it is, that carrying over an action that has become unsafe anchors the model, and that self-judged validity is near chance while oracle validity and explicit outcome values help (Sec. 4). It then proposes selective refresh. A LoRA-tuned low-cost model (Qwen3.5-27B), trained on simulator-replay labels, predicts an action, optionally with a retain/refresh/rerank label. A bucket-based risk controller estimates whether AgileThinker would correct an error, and invokes it when the risk exceeds a validation-selected threshold (Sec. 5). All evaluation is offline and one-step: an action counts as correct if it is in a simulator-derived reference set of actions that avoid reset and unsafe termination. On fresh seeds, selective fallback reaches 96.04% versus 96.65% for full AgileThinker, while calling the thinking model on 43.75% of requests and cutting mean completion tokens by 57.33% (Table 6). By game, the gains come from Snake, all policies score 100% on Overcooked, and fallback lowers accuracy on Freeway (Table 18).

Critical strengths

S1. A well-controlled analysis of why reusing an old action fails (Sec. 4) The strongest part of the paper is its analysis, not its method. To find out when reusing an earlier action goes wrong, the authors take the same starting state and build two next states: one where the old action is still safe and one where it has become unsafe. They also vary how many steps ago the action was decided, and test every state with and without the carried-over context (the old action and earlier safety judgments) in the prompt. Because only one factor changes at a time, the cause is clear. Whether the old action is still safe matters far more than how old it is (72.50 vs. 20.83 points, Table 1). When the old action is still safe, giving it to the model helps (+23.75 points). When it has become unsafe, the model tends to repeat it anyway, and safety drops by 48.75 points (Sec. 4.1, Table 2). This gives a concrete lesson to anyone building agents that carry decisions across time steps: before passing an old decision to the model, check whether it is still valid, not how recent it is.

S2. The controller targets errors that thinking can actually fix The risk controller is not trained simply to predict whether the low-cost model is wrong. It is trained to predict cases where the low-cost model is wrong and the thinking model would get it right (Eq. 11). So the expensive model is called where it is expected to help, rather than wherever a request looks hard. Allocating compute by the expected benefit of more reasoning is a sound direction.

Critical weaknesses

W1. Task performance and real-time behavior are not evaluated The paper aims to make real-time LLM agents more efficient: check whether the previous action is still valid, and reason further only when needed. The problem is worth studying, but the paper does not show that the method cuts inference cost while keeping task performance.

  • The main metric, synchronization accuracy (Eq. 3), only checks whether the chosen action is in the simulator's reference action set (Eq. 2, Sec. 5.1, App. B). This is not the same as task success rate or episode return. Two actions can both be safe right now while only one moves toward the goal and the other causes an unnecessary detour, and the metric counts both as correct. Picking an action that is acceptable now does not guarantee reaching the goal or a high cumulative reward.
  • The paper is about real-time agents, but the method is never run online, where the environment keeps changing while the agent reasons (Sec. 6.5, Sec. 7). This matters especially for this method. More planning can make an action more accurate, but the time it takes can make the decision outdated. An offline, single-decision evaluation cannot show this. So the current results show that selective fallback keeps one-step accuracy fairly high at a lower cost. They do not show that it keeps the task performance of a real-time agent, which is what the paper aims for. The method should be run in Real-Time Reasoning Gym, measuring task success rate, episode return and actual decision latency (R1).

W2. Much of the reported cost saving comes from Overcooked, where every action counts as correct The abstract reports that selective fallback is 0.61 points less accurate than full AgileThinker while using 57.33% fewer completion tokens (Table 6). However, 256 of the 656 test requests are from Overcooked. There, every legal action is in the reference set, so all three methods score 100% (Table 18). The authors themselves note in App. H that these requests say nothing about cooperation or order completion.

  • Recomputing Table 18 without Overcooked, the thinking-model call rate rises from 43.75% to 71.75%, the token saving drops from 57.33% to 32.40%, and the accuracy gap to full AgileThinker widens from −0.61 to −1.00 points. About 64.5% of all saved tokens come from Overcooked.
  • Skipping the expensive model on easy requests can be a sensible strategy. But because every action is accepted in Overcooked, these requests cannot show whether the risk controller made good decisions. The headline saving therefore depends heavily on how the test set is composed. At a minimum, the main text should report the accuracy–cost trade-off for each game and the results without Overcooked, and analyze the efficiency gain on the games where not every action is correct (R2).

My rating is mainly driven by W1.

Questions for authors

Clarifications

Q1. Should the risk target also count cases where thinking makes things worse? The risk controller predicts recoverable errors: cases where the low-cost model is wrong and the thinking model is right (Eq. 11). The opposite case, where the thinking model replaces a correct low-cost action with a wrong one, is not part of this target, although it does lower the final accuracy (App. A, Eq. 17). This matters because the thinking model is not always better. On Freeway, Refresh-action alone reaches 96.53%, selective fallback drops this to 90.97%, and full AgileThinker gets only 88.19% (Table 18). Instead of estimating only the chance that thinking fixes an error, wouldn't it be better to weigh the expected gain of calling the thinking model against the expected loss?

Q2. How stable are the routing results? In the fixed-budget routing experiment, the bucket scorer beats random routing by about 0.33 points (Table 16), and the test set has only 7 recoverable errors (Table 17). The data-selection experiment also uses a single training seed (Table 9). With these numbers, it is hard to conclude that the proposed risk estimate is clearly better than the alternatives. Could you report more runs and uncertainty estimates?

Q3. What does the structured decision label add? Table 21 (App. K) shows that some records labeled invalid actually have acceptable prior actions. Given this mismatch, how much do the labels help in judging whether a prior action is valid? Does the risk controller perform similarly without the label?

Q4. What happens in buckets not seen in training? For feature buckets not seen in training, the risk falls back to the global smoothed estimate (Eqs. 13–14). With the reported counts (57 recoverable errors in 1,287 usable training pairs, App. D), this estimate is about 0.045, below the selected threshold of about 0.059. So low-cost candidates in unseen buckets would be accepted without fallback. How often does this happen at test time, and how does it affect cost and accuracy?

Q5. Does the risk controller generalize? Would the same risk estimate and threshold still work for new games or a different thinking model?

Requested experiments

R1. Online evaluation in Real-Time Reasoning Gym (most important) Run the full method in Real-Time Reasoning Gym, where the environment keeps moving while the agent reasons. Report task success rate, episode return and actual decision latency, along with the trade-off between task performance and inference cost, compared with Refresh-action alone and full AgileThinker. This is needed to judge the paper's central claim. If it cannot be done within the discussion period, the claims in this version should be narrowed to one-step action acceptability in offline evaluation.

R2. Per-game trade-offs and results without Overcooked Report the accuracy–cost trade-off for each game and the overall results without Overcooked in the main text, and analyze the efficiency gain on the games where not every action is correct.

R3. Comparison with efficiency-oriented methods, and ablations The paper's headline claim is about efficiency: 57.33% fewer completion tokens at a 0.61-point accuracy cost (abstract, Table 6). Judging this claim requires a comparison with existing methods built to cut inference cost, not only with the paper's own routing variants (random, fixed rule, single feature, logistic regression; Sec. 6.4, App. D). Deciding whether to pass a candidate output to a stronger model is already studied in cascade, routing and learning-to-defer work. What is new here is the real-time setting with inherited decisions, so the risk controller itself should be compared directly with these methods. If this cannot be completed within the discussion period, the efficiency claim in this version should be narrowed to gains over the paper's own routing variants.

  1. Comparison with each family of efficiency methods, using the same low-cost model, the same thinking model and the same test requests as Table 6:
    • Confidence-based cascade (FrugalGPT/AutoMix-style): call the thinking model only when the low-cost model's own confidence in its action (e.g., the token probability) is low. None of the controller's 17 features captures this signal (Table 12).
    • Learned routing and cascading: RouteLLM, BEST-Route, cascade routing (Dekoninck et al., 2025) and online cascade learning (Nie et al., 2024). The paper cites these but does not compare against them (Sec. 2), and its threshold-selection objective follows BEST-Route (Eq. 16).
    • Learning to defer (e.g., Mozannar and Sontag, 2020): train a deferral rule on the same paired low-cost and thinking-model outputs used to fit the risk controller.
    • Fast/slow agent switching (SwiftSage-style): a small model acts by default, and the larger planner is called only when a switching rule fires.
  2. Matched-cost curves. Vary each method's threshold or budget and report accuracy against cost (thinking-model call rate and completion tokens), at least on Snake and Freeway, where not every action is correct.
  3. Ablations that show how much the recoverable-error target and the structured decision label each contribute, for example by training the controller to predict plain low-cost-model errors, and by removing the label features. If R1 shows that selective fallback keeps episode-level performance at a clearly lower cost, and the gains hold up in R2 and R3, I would consider raising my score.
Reviewer_aM32OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper aims to reduce repeated long-form reasoning by learning to update prior decisions as the environment changes. The authors argue that current action validity matters more than decision age and show that LLMs can remain anchored to prior action recommendations despite receiving updated observations. They address this through SFT on simulator-derived actions and handling labels assigned by data-construction rules, training a model to produce concise action outputs from current observations and prior actions. Selective refresh runs this model on every request, then either accepts its output or invokes a thinking model based on the likelihood that additional reasoning will correct an error. Experiments in game environments suggest improved action accuracy over the low-cost model alone and reduced inference cost relative to always invoking the thinking model.

Critical strengths

S1. The paper clearly characterizes when reusing prior decisions is beneficial or detrimental.

S2. The paper is well written to understand the content.

Critical weaknesses

W1. The incremental value and cost efficiency of structured refresh remain unclear. In the reported direct comparison, structured refresh matches the accuracy of action-only SFT with fallback and incurs similar generation-token costs, but has a longer recorded execution-path time. Savings relative to full thinking therefore do not establish why learning and generating additional handling labels is preferable to simpler action prediction with fallback.

W2. Transfer of the learned revision capability to related tasks remains insufficiently evaluated. Evaluation is limited to new seeds and states within the same games. Although the SFT model is fixed in the fresh-seed experiments, the controller and threshold are refitted using separate training and validation data. These results support within-game generalization but do not establish whether the learned policy and invocation criterion remain effective under changes to goals or rules.

Questions for authors
  1. Is there existing evidence that jointly learning handling labels improves appropriate retention and revision of prior actions over action-only SFT? Since these labels do not always align with actual action validity, evidence of improved validity-sensitive action selection, beyond label agreement or parsing reliability, would be informative.

  2. Compared with action-only SFT with fallback, what benefits justify the additional cost of structured refresh? Please explain the longer recorded execution-path time for refresh-action and clarify which costs were measured, including whether input processing and controller execution were included.

  3. Are there existing evaluations on related tasks with different goals, rules, or layouts without additional SFT? If so, were the controller and threshold refitted, and how did standalone low-cost model performance, final performance, and the thinking-model invocation rate change?

Comments on the paper

1 comment

Dear Reviewers,

I hope this message finds you well.

I would like to remind you that we are currently in the rebuttal phase of the paper review process. During this stage, authors may respond to the comments and feedback you provided. If you receive any responses from the authors, please consider providing your feedback to ensure a smooth review process.

Thank you for your hard work and dedication throughout this review process. Your insights are invaluable in enhancing the quality of the submissions, and we truly appreciate your involvement.

If you have any questions or need further assistance, please do not hesitate to reach out.

Best regards,

AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.