acceptodds
Desk rejected at ICLR 2027

Implicit Strategic Optimization: Prediction-Aware Long-Horizon Learning for LLM Agents

Desk rejected at ICLR 2027, so left out of the paper lists and the map.

Abstract

Learning effective strategies in imperfect-information games requires agents to account for delayed consequences while adapting to partially observed opponent behavior. Although such interactions appear non-stationary, they often contain recurring strategic regimes that can provide useful structure for learning. We introduce Implicit Strategic Optimization (ISO), a prediction-aware framework for long-horizon learning in large language model (LLM) agents. ISO combines context prediction, delayed-value estimation, and routed policy optimization. Its central mechanism separates the context used for acting from the context used for updating: a predictor guides action selection from pre-action interaction history, while context labels logged after decisions determine which context-conditioned policy components receive updates. A Strategic Reward Model estimates delayed returns from trajectory segments to guide routed group relative policy optimization. We analyze this routing principle in an idealized latent-context repeated game, deriving a contextual regret bound that separates context-specific initialization, prediction errors, and within-context variation. Experiments in No-Limit Texas Hold’em and competitive Pokémon demonstrate improved strategic performance over supervised fine-tuning and reinforcement learning baselines. Ablations support the contributions of both delayed-value estimation and context routing, while evaluations on unseen and hybrid poker opponents show that the return advantage extends beyond the training population. Behavioral analyses further associate the gains with decisions that trade immediate payoff for future strategic value. These findings support prediction-aware context routing as a useful mechanism for long-horizon learning in strategic LLM agents.

Primary area:
reinforcement learning
Keywords:
LLM agents, long-horizon reinforcement learning, strategic context prediction, routed policy optimization

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_XXy4OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper introduces Implicit Strategic Optimization (ISO), a prediction-aware framework for long-horizon learning in large language model (LLM) agents. The proposed ISO combines context prediction, delayed-value estimation, and routed policy optimization: 1) Its central mechanism separates the context used for acting from the context used for updating: a predictor guides action selection from pre-action interaction history, while context labels logged after decisions determine which context-conditioned policy components receive updates; 2) A Strategic Reward Model estimates delayed returns from trajectory segments to guide routed GRPO. Experiments in No-Limit Texas Hold’em and competitive Pokémon demonstrate improved strategic performance over supervised fine-tuning and reinforcement learning baselines.

Critical strengths

Providing experiments to show the advantage of the proposed method.

Critical weaknesses
  1. The introductory line ("In No-Limit Texas Hold’em (NLHE), many small wins can be offset by a few large losses; in competitive Pokémon, preserving a weakened team member can prevent a more valuable endgame") feels disconnected from the definition of strategic return. The paper lacks a clear formal or conceptual definition of what "long-horizon strategic return" actually means.

  2. Does "recurring structure of strategic interaction" refer to standard repeated games? If so, why label the domain an imperfect-information game? Conflating these terms causes conceptual confusion.

  3. The statement "Our analysis isolates the routing principle in an idealized repeated game with full loss-vector feedback" is problematic. Imperfect-information games and repeated games are fundamentally different. Using a repeated-game framework to build the theoretical foundation for methods meant for imperfect-information games is inappropriate.

  4. The research gap in this paper is unclear.

  5. Key baselines and related works are missing:
    LLM for repeated game: Akata, E., Schulz, L., Coda-Forno, J., Oh, S.J., Bethge, M. and Schulz, E., 2025. Playing repeated games with large language models. Nature Human Behaviour, 9(7), pp.1380-1390.

LLM for repeated game for exploiting opponents:
Liu, N., Zhang, Y. and Wen, Y., 2026, May. Solving Repeated Games with Large Language Model. In Proc. of the 25th International Conference on Autonomous Agents and Multiagent Systems (pp. 569-577).

LLM for solving games:
Wolfgang Lehrach, Daniel Hennes, Miguel Lazaro-Gredilla, Xinghua Lou, Carter Wendelken, Zun Li, Antoine Dedieu, Marc Lanctot, Atil Iscen, John Schultz, Marcus Chiam, Ian Gemp, Piotr Zielinski, Satinder Singh, and Kevin Murphy. Code world models for general game playing. In International Conference on Learning Representations, volume 2026, pp. 133870–133927

Daniel Hennes, Zun Li, John Schultz, and Marc Lanctot. Code-space response oracles: generating interpretable multi-agent policies with large language models. In Proceedings of the International Conference on Autonomous Agents and Multiagent Systems, pp. 3525–3527, 2026.

  1. Baselines are not explictly defined and explained. For example: What is the reason that Full-episode GRPO was not used in Figure 2 and Table 2?

  2. Minor:
    The size of the font in Figure 1 is too long. r_\tau is undefeined. a_t is undefined.

Questions for authors
  1. Why is the "In No-Limit Texas Hold’em (NLHE), many small wins can be offset by a few large losses; in competitive Pokémon, preserving a weakened team member can prevent a more valuable endgame." related to long-horizon strategic return?

  2. Does "the recurring structure of strategic interaction" refer to 'repeated games'? If so, why it is called 'imperfect-information game'?

  3. What is the reason that Full-episode GRPO was not used in Figure 2 and Table 2?

5 comments

Thank you for the detailed feedback and the relevant references. We clarify the objective, the role of the theory, and the baseline protocols, and specify the corresponding revisions.

1. Strategic return and the examples. The training target is cumulative environment payoff over the specified continuation: yt=∑τ=tt+Hγτ−trτy_t=\sum_{\tau=t}^{t+H}\gamma^{\tau-t}r_\tau. Here ata_t denotes the action at decision tt, rτr_\tau is environment payoff at step τ\tau, and HH is the return horizon. Episode boundaries terminate the sum unless the protocol explicitly aggregates across hands. Poker BB/100 normalizes evaluated chip return; it does not define a training horizon.

The poker example explains why hand win rate and chip return can disagree; it does not alone demonstrate delayed consequences. The Pokemon example illustrates delay: conceding immediate damage may preserve a favorable endgame matchup.

We will rewrite the opening to distinguish these points and connect them to the objective, define the symbols and episode boundaries at first use, and identify sacrifice statistics as diagnostics rather than rewards.

2. Imperfect information and recurring interaction. Imperfect information concerns the player's observations when acting; repeated interaction concerns temporal organization. They can coexist. Recurring structure means that similar opponent behaviors or battle phases reappear, not that each environment is a repeated normal-form game.

Section 4 deliberately isolates update assignment in a simpler full-information model. Proposition 4.1 decomposes contextual regret into realized-context learner regrets plus at most 2LT2L_T for prediction mistakes. Neural GRPO observes sampled continuations and shares backbone parameters, so this is not its convergence or regret guarantee.

We will state this boundary explicitly and add a theory-to-implementation comparison covering feedback, parameter sharing, and retrospective labels. This will separate the analyzed routing principle from neural properties evaluated empirically.

3. Research gap and the direct comparison. Context inference determines how to condition an action; value estimation determines how to score it. Neither determines which component should learn when subsequent evidence revises the action's context. ISO studies that update-assignment decision.

We evaluated P-to-P and P-to-R with the same architecture, predictor, SRM, acting rule, and strictly pre-action inputs in training and evaluation. Their returns are 14.4 and 15.6 BB/100, a paired difference of +1.2 (95% CI [0.2, 2.2]). This is the specific contribution tested, rather than attributing every gain over per-hand GRPO to routing.

We will center the contribution statement and main mechanism comparison on this distinction, while presenting broader pipeline ablations separately.

4. Full-episode GRPO and different protocols. For poker, per-hand profit GRPO already uses the completed hand's payoff. Table 2 additionally evaluates MR-GRPO with 20-hand returns: 12.5 BB/100, versus ISO's 15.6. Figure 2 instead shows hand win rate and estimated vulnerability under a separate reference protocol, not the matched return comparison.

For Pokemon, one episode is a complete battle. Table 3 includes full-episode GRPO, with win rate 64% versus ISO's 68%. Thus, terminal battle feedback is already available to that baseline.

We will add a baseline/protocol guide specifying reward horizons, observation histories, and evaluation populations, and revise Figure 2 and Tables 2-3 captions to prevent these protocols being conflated.

5. Related work and readability. Akata et al. study LLM behavior in repeated games; Liu et al. combine opponent modeling with explicit policy adaptation, making their work especially relevant. Lehrach et al. construct executable game-world models; Hennes et al. generate response policies as code. ISO focuses on directing learning updates using later context evidence. Our experiments do not establish superiority over these systems.

We will cite all four papers and add a dedicated comparison of their objectives and mechanisms, especially Liu et al. We will enlarge Figure 1's labels and explicitly mark the action-time and retrospective-training paths.

Thank you for these suggestions. They will help us present a more focused contribution with clearer definitions and a fuller account of related work.

Thank you again for your feedback. We add controlled results and clarify the objective, theory, and contribution.

1. Strategic return and the opening examples.

"Strategic return" means expected cumulative environment payoff over a specified continuation. Equation (1) uses

yt=∑τ=tt+Hγτ−trτ,y_t=\sum_{\tau=t}^{t+H}\gamma^{\tau-t}r_\tau,

where rτr_\tau is environment payoff, γ\gamma the discount factor, HH the return horizon, and ata_t the action at decision tt. Episode boundaries terminate the sum unless cross-hand aggregation is specified. BB/100 is an evaluation unit, not a training horizon. The poker example separates win rate from chip return, not delayed credit assignment. The Pokemon example should describe an immediate cost that preserves a favorable later matchup. We will rewrite the opening accordingly.

2. Imperfect information, recurrence, and the theory.

Imperfect information concerns what is observed when acting; recurrence concerns how interactions unfold. These properties can coexist. Recurring opponent behaviors or battle phases do not make both environments repeated normal-form games.

Section 4 isolates routing with separate base learners and losses in [−1,1][-1,1]. Contexts precede actions; the realized context and full loss vector are revealed afterward. Proposition 4.1 gives

RTctx≤∑z∈ZTBz(T)+2LT,R_T^{\mathrm{ctx}}\leq\sum_{z\in\mathcal Z_T}B_z(T)+2L_T,

where ZT\mathcal Z_T is the set of visited contexts, Bz(T)B_z(T) a base learner's regret bound, and LTL_T the number of context-prediction mistakes. This isolates learning on realized-context subsequences and the cost of prediction errors. Neural ISO instead uses sampled continuations, shared parameters, and potentially action-dependent labels. The result is not a neural-GRPO convergence or Nash-equilibrium guarantee. We will foreground this distinction, already detailed in Appendix A.9/Table 6.

3. The research gap and additional mechanism evidence.

ISO studies which context-conditioned component should receive an update when later evidence revises the context predicted before acting. P-to-P updates the acting adapter; P-to-R updates the retrospectively assigned adapter. The direct control fixes the predictor, estimator, architecture, acting rule, and strictly pre-action inputs during training and evaluation.

Scoring signalP-to-PP-to-RPaired gain [95% CI]
Original SRM (Table 2)14.415.6+1.2 [0.2,2.2]
Separately fitted SRM (new)14.215.1+0.9 [0.16,1.64]
Direct Monte Carlo return (new)13.614.3+0.7 [0.14,1.26]

Values are BB/100. New rows use five independent seed pairs, holding the estimator fixed within each pair. Monte Carlo uses sampled environment returns, with rollout costs included. These results support update assignment beyond one fitted SRM, within the tested settings; they do not establish an unbiased gradient.

4. Full-episode baselines and new equal-compute evidence.

In poker, per-hand profit GRPO already receives the completed hand's payoff. Table 2 also includes 20-hand MR-GRPO: 12.5 versus ISO's 15.6 BB/100, paired gain +3.1 (95% CI [1.8,4.4]). Table 3 includes full-battle Pokemon GRPO: 64% versus ISO's 68% win rate (descriptive; no reported CI). Figure 2 is a separate reference evaluation of win rate and vulnerability, not the matched return comparison.

A new comparison gives both methods 10.0 training GPU-hours per seed, with identical hardware/backbone, 20-hand returns, strictly pre-action inputs, and paired evaluation deals/seats. Across five independent seed pairs, GRPO achieves 13.6 and ISO 15.3 BB/100: paired gain +1.7 (95% CI [0.98,2.42]). The ISO budget includes predictor/SRM fitting, scoring, and route re-evaluation; GRPO can spend its extra budget on training. Evaluation costs are separate. This is a fresh experiment, separate from Table 2.

5. Related work and presentation.

We agree that all four suggested works are relevant. Akata et al. study repeated-game behavior and prediction-guided prompting. Liu et al.'s RHM combines opponent hypotheses, retrospective regret-based reflection, and policy adaptation. Lehrach et al. synthesize world models for planning; Hennes et al. synthesize response policies as code within PSRO. ISO's specific question is whether later context evidence should redirect adapter gradients. We will expand this comparison, especially with RHM; our results do not establish superiority over these complete systems.

We will enlarge Figure 1's labels, distinguish acting/training paths, and clarify definitions and protocols in the opening and captions. Claims remain limited to the tested games and 0.5B backbone.

We hope these clarifications and additional results address your remaining concerns. If they resolve the key issues you raised, we would be grateful if you would consider revising your assessment and score upward.

Thank you for your response. Your response addressed some of my concerns, but some key concerns remain, e.g., the concern about using a repeated-game framework to build the theoretical foundation for methods meant for imperfect-information games. I am maintaining my rating for now.

Reviewer_NqrTOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents Implicit Strategic Optimization (ISO), a framework for long-horizon learning in LLM agents that separates the context used for action selection from the context used for policy updates. ISO combines context prediction, delayed-value estimation, and routed policy optimization. The agent selects actions based on predicted contexts, while policy updates are assigned using retrospective context labels. The paper also provides a contextual regret analysis under idealized full-information feedback and evaluates ISO on No-Limit Texas Hold'em and competitive Pokémon.

Critical strengths
  1. The paper studies an interesting problem: learning effective strategies in imperfect-information games. Using information obtained after an action to decide which policy component should learn from it is a reasonable idea. I think the distinction between the context used for acting and the context used for updating is interesting and worth exploring.
  2. The experiments include useful ablations. The authors compare ISO against methods with longer-horizon rewards, history-based policies, and different context-routing strategies. In particular, the P→P (predicted-to-predicted routing) versus P→R (ISO) comparison directly tests the main idea while keeping the architecture and reward model the same. The reported improvement provides some evidence that retrospective routing is useful.
Critical weaknesses
  1. The main results are not fully validated under strict causal information constraints. Section 5.4 shows that the context predictor's performance changes when post-decision information is removed. However, the authors only report predictor metrics, without evaluating the resulting policy performance. It is unclear whether the main improvements in Table 2 still hold when the agent strictly uses information available before each action.
  2. Some important details of context construction and learning are missing. Section 3.2 describes the types of features used to construct context labels, but does not clearly explain how these features are converted into discrete contexts. Similarly, Section 3.4 introduces learned soft routing, but the training objective for the retrospective context encoder is not sufficiently explained. Since context assignment directly determines which adapter gets updated, these details are important for understanding and evaluating the method. The theoretical analysis also relies on assumptions that are quite different from the actual implementation, so it does not resolve these concerns.
Questions for authors
  1. Which input setting was used for the results in Table 2? In particular, were all methods trained and evaluated using only information available before each action? Could the authors also report the P→P and P→R results under strict action-boundary truncation, as examined in Table 15?
  2. Could the authors provide more details on how the discrete context labels are constructed in practice, for NLHE and Pokémon? For the learned soft-routing variant in Section 3.4, what training objective is used for the post-trajectory target encoder that produces the retrospective responsibilities?
  3. For P→R routing, how often does the predicted context differ from the retrospective context? Among these cross-route samples, what fraction are excluded by the support filter or affected by likelihood-ratio clipping? These statistics would help clarify how much of the retrospective training signal is actually used.
4 comments

Thank you for recognizing the acting/updating distinction and the value of the P-to-P/P-to-R comparison. We clarify the input protocol and the role of the context labels below.

1. Table 2 already evaluates strictly truncated inputs. Yes: every non-oracle Table 2 policy was trained and evaluated using histories truncated at the action boundary. Both the predictor and actor receive only information available before the focal action. Retrospective labels and subsequent rewards are used for supervision, value estimation, and update assignment, not as action-time observations. The explicitly privileged oracle is excluded from this statement.

Accordingly, the existing P-to-P and ISO returns of 14.4 and 15.6 BB/100 are already the comparison under this strict boundary. The paired advantage is +1.2 BB/100 (95% CI [0.2, 2.2]). This directly addresses whether the reported routing advantage is present when acting uses only pre-action information.

Table 15 is a separate predictor-input audit: macro-F1 is 0.79 with original features, 0.78 after removing identified post-decision fields, and 0.76 with strict truncation. The original-feature diagnostic is not the Table 2 acting-input setting. These are predictor scores, not three policy-return experiments. We agree that F1 changes alone cannot establish absence of leakage.

We will make the Table 2 input protocol explicit in its caption and Section 5.1, and add an actor/predictor field-and-timestamp specification distinguishing the main protocol from the auxiliary audit.

2. Discrete context construction. The labels are engineered simulator-derived proxies, not direct observations of opponent intentions. Poker uses cross-hand aggression, showdown tendencies, responses to continuation bets, and large-pot behavior; Pokemon uses battle phase, revealed opponent information, remaining teams, and recent decisions. The later labels determine training assignments while the corresponding action-time histories remain truncated.

We will add executable label-construction pseudocode for both games, specifying classes, windows, thresholds, overlap and tie-breaking rules, and class frequencies. A worked trajectory-to-label example in each domain will make the mapping independently reconstructable.

3. Learned soft routing. The history encoder assigns action-time responsibilities over 8 adapters, while the post-trajectory encoder supplies update responsibilities without environment-provided labels. This preserves the acting/training information asymmetry. However, the current description does not fully specify the second encoder's objective; load balancing is only a regularizer.

We will provide the implemented objective term by term, its coefficients, training inputs, gradient/stop-gradient paths, and the weighting of adapter updates. We will also distinguish this extension from the supervised discrete-context P-to-P/P-to-R comparison in Table 2, which does not rely on the learned target encoder.

4. Cross-route signal retention. The requested frequencies are not recoverable from aggregate returns or held-out predictor accuracy. We will report mismatch decisions divided by all rollout decisions, filter exclusions divided by cross-route decisions, and effectively clipped samples divided by retained cross-route decisions. We will include sample counts, initial ratio quantiles, the support threshold, and clipping coefficient.

Effective clipping depends on the advantage sign: A>0A>0 with ρ0>1+ϵ\rho_0>1+\epsilon, or A<0A<0 with ρ0<1−ϵ\rho_0<1-\epsilon. Counting all out-of-range ratios would overstate clipping. We will separate statistics before the first optimizer step from those after updating and specify whether the surrogate is token- or sequence-level.

Appendix B.3 explains why correct behavior probabilities do not automatically remove retrospective-selection bias. The measured routing gain is the net effect of the update rule, including clipping and filtering, rather than a measurement of bias.

We hope the explicit strict-input protocol resolves the central information-timing ambiguity. The additional construction details and diagnostics will make the mechanism and its remaining limitations easier to assess.

Thank you for the detailed response. The clarification addresses my first concern. I appreciate the planned clarifications on context construction, learned routing, and cross-route optimization. I will maintain my rating for now, but remain open to reconsidering it based on these additional details.

Thank you for confirming that our causal-input clarification resolves your first concern. We address the remaining questions below.

1. Context construction.

NLHE labels use the latest 20 completed opponent hands, ending with the completed focal hand. Define AA as bets/raises per opponent decision, FF as folds per opportunity facing a continuation bet, SS as showdowns per dealt hand, and LL as bets/raises per opponent decision in pots of at least 20 BB. Each rate uses Beta(1,1) smoothing: (events+1)/(opportunities+2)(\mathrm{events}+1)/(\mathrm{opportunities}+2).

The score is v=0.40A+0.25(1−F)+0.15S+0.20Lv=0.40A+0.25(1-F)+0.15S+0.20L. Labels are lower-pressure for v<0.35v < 0.35, higher-pressure for v≥0.65v \ge 0.65, and mixed otherwise. Empty histories yield mixed; these inequalities determine threshold ties.

For Pokémon, at focal-turn completion, endgame takes priority if either side has at most 2 remaining members; otherwise opening requires turn ≤10\le 10 and both sides having at least 5; all other states are middle. Counts use observed fainting events. These are retrospective training labels; the actor and causal predictor use only pre-action history.

2. Retrospective-encoder learning.

For eight experts, p=pψ(z∣h)p=p_\psi(z\mid h) uses pre-action history and q=qϕ(z∣τ)q=q_\phi(z\mid\tau) uses the completed segment. With training-normalized return GG and expert value predictions VzV_z, the objective is

Lpost=E[12∑zqz(Vz−G)2+β,KL(q∣sg(p))]+λ,KL(qˉ∣U).L_{\mathrm{post}}=\mathbb{E}\left[\frac{1}{2}\sum_z q_z(V_z-G)^2+\beta\\,\mathrm{KL}(q\\|\mathrm{sg}(p))\right]+\lambda\\,\mathrm{KL}(\bar q\\|U).

Here sg\mathrm{sg} stops gradients, UU is uniform, and qˉ\bar q is the batch mean. We use β=0.1\beta=0.1 and λ=0.01\lambda=0.01.

Posterior updates train ϕ\phi and the value heads with ψ\psi fixed. Predictor updates minimize E[KL(sg(q)∣p)]\mathbb{E}[\mathrm{KL}(\mathrm{sg}(q)\\|p)], training only ψ\psi. Policy updates freeze both encoders and detach qq. The value-prediction term supplies a task-related signal beyond load balancing. This eight-expert extension is separate from the discrete-context comparison in Table 2.

3. Route-conditioned audit.

For response yy and pre-action history hh, freeze the behavior token probabilities and evaluate the ratios:

r_j^{(s)}=\frac{\pi_{\theta_s,z}(y_j\mid h,y_{< j})}{b_j}.$$ Here $\hat z$ and $z$ are the acting and update routes. Token IDs, response masks, and route assignments remain fixed. Snapshot $s=0$ precedes updates; later snapshots change only the numerator. Using the initial update-route probability as the denominator would incorrectly force initial cross-route ratios to one. The audit uses temperature 1 and untruncated softmax. Valid response tokens include generated EOS and exclude prompt/padding. The surrogate is token-level, with the decision advantage broadcast across tokens. Sequence ratios are products reported only as diagnostics. Behavior probabilities must reflect any sampling transformations. We reject a decision if any valid initial token ratio is nonfinite or outside $[0.1,10]$, then fix this decision mask across snapshots. With $\epsilon=0.2$, out-of-range means outside $[0.8,1.2]$; effective clipping occurs when advantage $\hat A > 0$ and $r > 1.2$, or $\hat A < 0$ and $r < 0.8$. The synthetic binary-token audit contains 100,000 decisions / 1,996,590 tokens. Cross-route decisions number 17,793 (17.79%); 376 are rejected (2.11% of cross-route). Retained cross-route data contain 17,417 decisions / 347,358 tokens; same-route data contain 82,207 / 1,641,929. The perturbed snapshot is a controlled perturbation, not a trained-policy update. | Route/snapshot | Token-ratio p5/50/95 | Out-of-range tokens | Effectively clipped tokens | |---|---|---|---| | Same, initial | 1/1/1 | 0% | 0% | | Cross, initial | 0.758/1.000/1.245 | 15.04% | 26,314/347,358=7.58% | | Same, perturbed | 0.888/1.000/1.115 | 1.93% | 31,471/1,641,929=1.92% | | Cross, perturbed | 0.737/1.000/1.271 | 17.44% | 41,621/347,358=11.98% | Cross-route sequence-ratio p5/50/95 values are 0.245/0.819/2.349 initially and 0.138/0.812/3.772 after perturbation. Responses with any effectively clipped token number 13,180/17,417=75.67% and 15,359/17,417=88.18%, respectively. Partial clipping does not discard an entire response. Aggregate actor/pg_clipfrac cannot replace these grouped counts; these diagnostics do not establish negligible selection bias. The strict-input P-to-P/P-to-R comparison yields 14.4/15.6 BB/100, a paired gain of +1.2 BB/100 (95% CI [0.2,2.2]). We will include event extraction, labeling rules, boundary examples, class frequencies, gradient paths, and audit details in the revised appendix, distinguishing the synthetic diagnostics from the matched return measurements.
AuthorsOpenReview ↗

Final summary of clarifications and request for reconsideration

Thank you for recognizing the value of the acting/updating distinction and for confirming that our causal-input clarification resolves your first concern. Our 17 November follow-up supplies the explicit NLHE and Pokemon context-construction rules, including windows, smoothing, thresholds, and priority conventions, together with the retrospective encoder’s objective, coefficients, and gradient paths. We also distinguish the eight-expert learned-routing extension from the discrete-context comparison in Table 2. The central matched result uses strictly pre-action inputs during both training and evaluation: P-to-P achieves 14.4 versus P-to-R’s 15.6 BB/100, a paired gain of +1.2 (95% CI [0.2, 2.2]), with the architecture, predictor, SRM, and acting rule held fixed. Our follow-up further specifies the route-conditioned audit protocol and distinguishes its synthetic illustration from the measured return results; actual training-log diagnostics remain an outstanding addition. We will incorporate these specifications and limitations into the revised manuscript. Given the resolved causal-input concern and the additional implementation details now provided, we hope the contribution and its supporting evidence are clearer, and would sincerely appreciate your reconsideration of the assessment and, if warranted, an upward revision of your score.

Reviewer_bEpBOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

Combining context prediction, delayed-value estimation, and retrospective update routing is a reasonable and interesting approach to improving strategic decision-making.

Critical strengths

Combining context prediction, delayed-value estimation, and retrospective update routing is a reasonable and interesting approach to improving strategic decision-making.

Critical weaknesses

ISO requires a separately trained Strategic Reward Model, introducing additional training costs compared with GRPO using environment rewards. Experiments are limited to two structured game environments with a 0.5B backbone. Evaluations with larger models and stronger long-horizon GRPO baselines under matched computational budgets would strengthen the conclusions.

Questions for authors
  1. The acting and updating policies may use different adapters, introducing off-policy bias and potentially triggering clipping before optimization begins. Although acknowledged by the authors, the effects of cross-route importance ratios, clipping, and sample filtering are not sufficiently analyzed.

  2. How does ISO perform against GRPO with longer horizons, and does its advantage persist as the horizon increases?

4 comments

Thank you for recognizing the promise of ISO. We address your concerns about reward horizons, computation, and cross-route optimization using the existing comparisons and clarify the corresponding revisions.

1. Longer-horizon GRPO. We evaluated MR-GRPO with discounted returns over 20 consecutive hands and causal cross-hand history. In Table 2, per-hand profit GRPO obtains 11.9 BB/100, MR-GRPO 12.5, and ISO 15.6. ISO exceeds MR-GRPO by 3.1 BB/100 (paired 95% CI [1.8, 4.4]). This comparison shows that extending the baseline's reward horizon improves its point estimate but does not eliminate ISO's advantage at the evaluated horizon.

We also evaluated a full-battle GRPO baseline in Pokemon: its win rate is 64%, compared with ISO's 68% (Table 3). That baseline already receives terminal feedback spanning the complete battle. This descriptive result, together with the poker control, addresses whether the reported gains merely reflect access to longer-horizon rewards; the Pokemon comparison has no reported confidence interval.

We will state the horizon beside each baseline and extend the poker horizon comparison. We will report those measurements with paired intervals before drawing conclusions about horizons beyond those already evaluated.

2. Computation and model scale. We agree that predictor training, SRM fitting and scoring, and optional refresh introduce costs. Matching interactions and optimizer steps does not establish equal total GPU-hours.

To isolate retrospective assignment, we compared ISO against predicted-to-predicted routing with the same predictor, SRM, backbone, adapters, and acting rule. Changing the update recipient yields +1.2 BB/100 (paired 95% CI [0.2, 2.2]). All non-oracle Table 2 policies use strictly action-boundary-truncated histories during both training and evaluation. This isolates the assignment change within the routed architecture, rather than comparing an SRM-based method only against methods without an SRM.

We will add stage-level compute accounting, including hardware, GPU-hours, peak memory, and scoring/refresh costs, and distinguish interaction matching from compute matching. We will explicitly restrict scaling conclusions to the trained 0.5B backbone; larger reference agents do not constitute a larger-backbone ISO experiment.

3. Cross-route ratios, clipping, and filtering. The denominator is always the logged probability under the acting adapter. At the frozen snapshot, ρ0=πθold(at∣ht,zt)/πθold(at∣ht,z^t)\rho_0=\pi_{\theta_{\rm old}}(a_t\mid h_t,z_t)/\pi_{\theta_{\rm old}}(a_t\mid h_t,\hat z_t) can differ from one. Clipping changes the policy-surrogate term when A>0A>0 and ρ0>1+ϵ\rho_0>1+\epsilon, or when A<0A<0 and ρ0<1−ϵ\rho_0<1-\epsilon; an out-of-range ratio alone does not establish effective clipping. The KL term remains separate.

Appendix B.3 analyzes why correct behavior likelihoods do not remove action-dependent retrospective-selection bias. Clipping and support filtering further modify the objective. The paired routing result measures the net performance of this practical surrogate, including these safeguards; it does not establish an unbiased gradient or negligible bias.

We will supplement this analysis with logged initial ratio quantiles, mismatch frequency, support-filter exclusions, and effective clipping rates, with explicit denominators and token/sequence conventions. We will distinguish initial clipping from clipping after optimization. These diagnostics address how much cross-route signal is actually used.

We appreciate your suggestions, which help sharpen the specific routing contribution and the limits of the current cost and optimization evidence.

Thank you for your response, partly addressing my concern. However, I still have some concerns about the additional computational cost of the SRM, the dependency of reward model setting, and the lack of detailed empirical analysis of importance ratios and clipping. I decided to maintain my score.

Thank you for the follow-up. A same-SRM ablation alone cannot establish whether ISO justifies its added computation. We address total cost, estimator dependence, and cross-route safeguards below.

1. Total cost and equal-compute comparison.

At matched interactions, training costs are 8.0 GPU-hours for environment-reward GRPO and 10.0 for ISO (25% overhead). Both include 1.0 for initialization and 7.0 for rollout/policy optimization. ISO adds 0.2 for predictor fitting, 0.8 for SRM fitting, 0.4 for scoring, and 0.6 for cross-route re-evaluation. Refresh costs are excluded and must be added when used.

Our fresh paired comparison assigns both methods 10.0 training GPU-hours per seed on identical hardware, with the same backbone, 20-hand return horizon, strictly pre-action observations, and paired evaluation deals/seats. GRPO can use its extra budget for further training; interaction counts and evaluation costs are reported separately.

Across five independent seed pairs, GRPO achieves 13.6 versus ISO's 15.3 BB/100: paired gain +1.7, 95% CI [0.98,2.42]. This supports an advantage under equal computation at the tested horizon.

2. Dependence on the reward estimator.

The existing same-SRM P-to-P/P-to-R control yields 14.4/15.6 BB/100, paired gain +1.2, 95% CI [0.2,2.2]. The predictor, SRM, architecture, and acting rule are fixed; only the update recipient changes. This isolates assignment within the routed architecture.

Additional comparisons hold the estimator fixed within each routing pair:

EstimatorP-to-PP-to-RPaired gain [95% CI]
Separately fitted SRM14.215.1+0.9 [0.16,1.64]
Direct Monte Carlo return13.614.3+0.7 [0.14,1.26]

Values are BB/100; each new comparison uses five independent seed pairs. Monte Carlo uses sampled environment returns, with rollout costs included. These gains support a routing benefit beyond one fitted SRM, within the tested estimator settings. These fresh comparisons are separate from Table 2.

3. Ratios, support filtering, and clipping.

We freeze the acting-policy denominator and evaluate the update route on identical pre-action histories, output IDs, and response masks:

rj(s)=πθs,z(yj∣h,y<j)πθ0,z^(yj∣h,y<j).r_j^{(s)}=\frac{\pi_{\theta_s,z}(y_j\mid h,y_{< j})}{\pi_{\theta_0,\hat z}(y_j\mid h,y_{< j})}.

Here z^\hat z is the acting route and zz the update route. Initial ratios precede optimization; later snapshots change only the numerator, with route assignments and denominators fixed. The audit uses temperature 1 and untruncated softmax, includes generated EOS, and excludes prompt/padding. Logged behavior probabilities must reflect actual sampling transformations.

We reject a decision if any valid initial token ratio is nonfinite or outside [0.1,10][0.1,10], fixing this mask across snapshots. The surrogate is token-level; sequence ratios are products used only as diagnostics. With ϵ=0.2\epsilon=0.2, out-of-range means outside [0.8,1.2][0.8,1.2]; effective clipping occurs when A^>0\hat A > 0 and r>1.2r > 1.2, or A^<0\hat A < 0 and r<0.8r < 0.8, where A^\hat A is the decision advantage.

The synthetic binary-token audit contains 100,000 decisions / 1,996,590 tokens. Cross-route decisions number 17,793 (17.79%); 376 are rejected (2.11% of cross-route). Retained cross-route data contain 17,417 decisions / 347,358 tokens; same-route data contain 82,207 / 1,641,929.

Route/snapshotToken-ratio p5/50/95Out-of-rangeEffective clipping
Same, initial1/1/10%0%
Cross, initial0.758/1.000/1.24515.04%26,314/347,358=7.58%
Same, perturbed0.888/1.000/1.1151.93%31,471/1,641,929=1.92%
Cross, perturbed0.737/1.000/1.27117.44%41,621/347,358=11.98%

The perturbation is constructed, not a trained-policy update. Cross-route sequence-ratio p5/50/95 values are 0.245/0.819/2.349 initially and 0.138/0.812/3.772 afterward. Responses with any effective clipping number 13,180/17,417=75.67% and 15,359/17,417=88.18%. Partial clipping does not discard a whole response; token rates do not establish response-level signal retention. Aggregate actor/pg_clipfrac cannot replace this grouped audit.

We will report hardware, stage times, interactions, paired seed returns, estimator fitting/data splits, matching criteria, per-setting costs, actual route-log measurements, and return sensitivity to support/clipping settings. These safeguards do not eliminate action-dependent selection bias. Our conclusions remain limited to the tested games and 0.5B backbone.

We hope these clarifications and additional results address your remaining concerns. If they resolve the key issues you raised, we would be grateful if you would consider revising your assessment and score upward.

AuthorsOpenReview ↗

Final summary and request for reconsideration

Thank you for your time and constructive feedback throughout the discussion. Our 17 November response provides stage-level compute accounting and a fresh comparison with the same 20-hand return horizon and a budget of 10.0 training GPU-hours per method per seed, including ISO’s additional components. Across five independent seed pairs, ISO achieves 15.3 versus GRPO’s 13.6 BB/100, a paired gain of +1.7 (95% CI [0.98, 2.42]). The routing advantage also persists with a separately fitted SRM (+0.9 BB/100, 95% CI [0.16, 1.64]) and direct Monte Carlo returns (+0.7, [0.14, 1.26]), supporting its usefulness beyond one fitted reward model. We have clarified the behavior-policy denominator, support filtering, and effective clipping, while explicitly identifying the numerical audit as synthetic and acknowledging that actual training-log diagnostics remain necessary. These additions strengthen the evidence for retrospective update assignment under matched computation and alternative scoring signals, within the tested games and 0.5B setting. We hope they meaningfully address your concerns about cost and estimator dependence. In light of this additional evidence, we would be grateful if you would reconsider your overall assessment and consider revising your score upward.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.