- Rating
- 3
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper studies how Transformers learn from feedback in closed-loop interactions and how training rewards shape their decisions. The authors propose the Agentic ICL framework, which unifies active learning, bandits, and online reinforcement learning through a state-update-decision decomposition. They prove that Transformers can approximate finite-horizon interactive algorithms under regularity conditions and provide a bound on the resulting return error. Experiments show that supervised distillation enables models to imitate three representative algorithms.
Critical strengths
- The paper clearly defines the problem and distinguishes the feedback available to the model from the rewards used for training.
- The experiments are thorough. In particular, the extensive reward interventions show interpretable changes in decision-making as rewards change. The diagnostics and negative controls are comprehensive: the paper examines state information, update consistency, and the effects of resource changes on decisions, and includes controls that disrupt the correspondence between information and its associated history or target.
- The theoretical proofs are rigorous and well developed.
Critical weaknesses
- The experiments remain removed from real-world tasks. The primary environment contains only four targets and six tools, and the tools' outcome probabilities are explicitly provided to the model. The natural-language experiments use the same task, so they do not establish applicability to real-world search or coding.
- The theory establishes existence but does not show whether standard training converges to such solutions.
- The evidence for long-interaction capabilities is limited. Even when six tool calls are allowed, the model makes only 2.73 calls on average, and only 5% of episodes exceed four calls. Thus, the evaluation provides limited coverage of long-horizon decision-making.
Questions for authors
- Has the model learned multi-step planning, or a shortcut that approximates the results of planning? Beyond comparing candidate policies and decoding posterior beliefs, could the authors design new tasks or interventions on internal states to rule out simple heuristic strategies? The current evidence primarily supports behavior that is close to Bayes DP.
- Can the model solve tasks that require longer interactions? In the current six-call evaluation, even the optimal policy makes only 2.88 tool calls on average. Could the authors construct tasks in which achieving a high return requires interactions longer than those seen during training, to assess genuine long-horizon decision-making?
Response to Reviewer qyec
Thank you for the positive assessment of the proofs, reward controls, and diagnostic controls. We address your questions through the additional unknown-dynamics and long-interaction evaluations.
W1. Distance from real-world tool use
We added two evaluations beyond the original BTS protocol. E1 uses reward-only training in a small unknown-transition MDP, with no sampled transition probabilities, posterior states, or teacher actions supplied to the policy. E2 tests autonomous interaction on tasks where achieving the full oracle value requires more than the training maximum of four calls. Across these evaluations, outcome and outcome-preserving training retain greater alignment with the outcome objective, while training on local process rewards produces different decisions and lower outcome returns. The setups and results are reported in E1–E2 below.
Q1. Multi-step planning versus a shortcut
There is already evidence against particular simple explanations. Receding two-query lookahead is included in Table 8. In Table 10, changing only future horizon or budget substantially changes the outcome model's decisions while leaving current evidence and legal actions unchanged. A policy that depends only on the posterior and current action availability cannot account for that response.
E2 adds new tasks and matched histories that distinguish full-horizon decisions from KG, receding two-step lookahead, entropy-based acquisition, and open-loop acquisition. Paired prompts retain the same evidence and current actions while changing future resources, testing heuristics that can also read the resource fields. On full-DP/two-step disagreement histories, outcome and preserving models agree with full DP on 75.3 and 74.7 of histories, compared with 26.5 for the one-step process model. Their mean correct-direction resource responses are 0.135 and 0.131, compared with 0.015 for the one-step model. These measurements show movement toward the oracle's preferred action, beyond sensitivity to a changed resource field. The E2 heuristic table reports the comparisons and five-seed variation.
W3 and Q2. Tasks that require longer interaction than training
E2 selects tasks from the original BTS generator whose exact full-horizon value exceeds the best value achievable with at most four calls by at least 0.03. The models start from the initial prompt and choose all actions themselves, with no supplied prefix or forced calls. Each task is evaluated under both the four-call cap and the longer horizon, with the original likelihood-generation law.
At horizon eight, outcome and preserving training achieve mean returns of 0.601 and 0.602, exceeding the four-call oracle value of 0.547, while 87.6 and 86.8 of their episodes use more than four calls. The one-step process model returns 0.524, below the capped optimum. Thus, the outcome and preserving models use longer autonomous interactions to attain returns beyond the best four-call policy on this family. The E2 table also reports the horizon-six results and paired return gains.