acceptodds
Under review as a conference paper at ICLR 2027

Downstream-Aware Context Selection for Online In-Context Reinforcement Learning

Abstract

In-context reinforcement learning (ICRL) enables large language model agents to adapt to new environments using their interaction history without updating model parameters. However, repeatedly conditioning on growing histories can lead to substantial token cost. We propose a bounded-history context-management framework that predicts the task-dependent downstream effect of removing historical interactions to guide history selection and determine a decision-dependent context budget. Formally, our framework uses the full rolling history as a reference. The predictor evaluates removal effects, defines a deletion ordering, and applies a shared selection criterion to determine how much history to retain at each decision. We evaluate the method in closed-loop SUMO driving under held-out in-distribution, unseen-domain, and unseen-route settings, and in ScienceWorld under a continual ICRL protocol. Relative to a baseline using the full context, our method reduces total token usage by 25.7%, 25.8%, and 23.2% across the three driving settings while maintaining comparable closed-loop driving performance. In ScienceWorld, it reduces total token usage by 52.1% compared to full context and uses 30.2% and 37.8% fewer tokens than the Recent and Similarity baselines, respectively, while maintaining performance.

Primary area:
foundation or frontier models, including LLMs
Keywords:
in-context reinforcement learning, context selection, context management, sequential decision making, large language model agents

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_rxn8OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The method predicts the downstream effect of removing interactions from a rolling history, then combines deletion ordering, adaptive history size, and redundancy compression. SUMO and ScienceWorld experiments report lower policy-token consumption with broadly comparable task performance.

Critical strengths
  • Task-dependent removal supervision provides a useful alternative to recency and similarity. Total tokens decrease by 23.2–25.8% in driving and 52.1% in ScienceWorld relative to Full-H20, with paired-bootstrap analyses supporting the reported savings.

  • Compression-controlled recency, matched fixed-budget variants, and LongLLMLingua provide meaningful controls. These show that selection contributes beyond compression alone, particularly under distribution shift, although the controlled gains are smaller than the headline savings.

Critical weaknesses
  • Evidence is limited to a 20-item rolling window and relatively short runs. A structured summary plus recent-history baseline is absent, so the results do not establish an advantage over broader long-horizon memory management.

  • Total tokens are not end-to-end serving cost. Cache reuse, input/output pricing, and predictor-development cost are not jointly evaluated; reported latency excludes policy inference. ScienceWorld headline returns are running-max statistics: selection-active mean return is 74.22 versus 74.34 for Full-H20. Nonsignificant differences do not establish equivalence.

Overall assessment: Weak acceptance, borderline leaning accept. The controlled results support a useful bounded-history contribution; broader deployment efficiency remains unproven.

Questions for authors
  • Clarification: Can the efficiency claim be restricted to the evaluated bounded-history setting, with running-max and mean returns clearly distinguished?

  • Evidence: Is a structure-preserving summary-plus-recent-history comparison available? Under a specified serving configuration, how do cache-aware cost and end-to-end latency compare after including context-management overhead?

1 comment

We thank the reviewer for the feedback. We provide additional clarification on the evaluation scope and return metrics, along with new comparisons against a structure-preserving Summary + Recent baseline and measurements of cache-aware serving cost and latency.

Q1. Evaluation Scope and ScienceWorld Returns

Our efficiency claims are restricted to the evaluated bounded-history setting, where both Driving and ScienceWorld use a rolling window of W = 20 interactions. All available histories are retained before the window reaches full capacity, and adaptive selection is applied once the window is full. We have clarified this scope in the revised manuscript.

We have also distinguished the ScienceWorld return metrics. Table 2 reports best-of-40 return (running-max), defined as the highest observed return across 40 formal rounds for each task, averaged across tasks. Ours achieves 96.06 compared with 95.50 for Full-H20.

Appendix E additionally reports mean return. The mean returns are 74.22 for Ours and 74.34 for Full-H20. These statistics describe different aspects of observed task performance.

Q2. Structure-Preserving Summary + Recent Baseline

We additionally evaluated a structure-preserving Summary-16 + Recent-4 baseline under the same ScienceWorld closed-loop protocol, covering 18 tasks and 40 formal rounds per task.

Once the H20 window is full, the baseline summarizes the older 16 interactions individually and retains the four most recent interactions in their original form. It preserves the action history, step-level rewards, and temporal ordering, and generates summaries only from completed interactions.

The comparison is summarized below:

MetricOursSummary + Recent
Mean Return74.2274.33
Best-of-40 Return96.0697.33
All-round Policy Tokens / Trial135.5K152.4K
Late-round Policy Tokens / Trial87.1K120.4K
Fixed-replay Mean End-to-end Latency11.03 s21.20 s

Compared with Summary + Recent, Ours reduces all-round policy token usage by 11.1% and late-round policy token usage by 27.7%. The mean returns are closely matched, while Summary + Recent achieves a slightly higher observed best-of-40 return.

Under the same fixed-trace replay protocol, Ours also achieves 48.0% lower mean end-to-end latency (11.03 s vs. 21.20 s). These results demonstrate lower token consumption and serving latency compared with this structure-preserving summarization baseline, while maintaining similar observed mean returns.

Q3. Cache-Aware Inference Cost and End-to-End Latency

We conducted fixed-trace inference replays using actual downstream-policy API calls, with GPT-4o for Driving and GPT-4.1 minifor ScienceWorld under a fixed serving configuration. End-to-end serving latency was measured from context preparation through downstream-policy inference, response parsing, and logging. It includes embedding, predictor inference, history selection, redundancy compression, or summarization where applicable.

API costs were calculated using recorded input, cached-input, and output token usage under the applicable model pricing, including embedding API charges.

For Driving, the measured API costs per decision decrease from $0.02026 to $0.01299 on Held-out ID, $0.02390 to $0.01474 on Severe-3x, and $0.01868 to $0.01322 on Unseen Routes. These correspond to reductions of 35.9%, 38.3%, and 29.2%, respectively. The end-to-end latency measurements reveal a trade-off: Ours increases serving latency by 15.8–20.7% relative to Full-H20 in Driving.

For ScienceWorld, the measured fixed-replay API cost decreases from $0.04863 to $0.01625 per trial, representing a 66.6% reduction relative to Full-H20. The corresponding end-to-end latency measurements are:

MethodMean Latency (s)Median (s)
Full-H2011.7611.89
Recent-1210.9011.20
Ours11.0310.81
Summary + Recent21.2020.55

Under this replay protocol, Ours achieves 6.3% lower latency than Full-H20. The evaluation includes 108 Driving measurements and 36 complete ScienceWorld trial replays across Full-H20, Recent-K, and Ours, with an additional matched-trace comparison for Summary + Recent.

The revised manuscript clarifies the bounded-history scope and ScienceWorld return metrics in the main text and Appendix E. Appendix K provides the fixed-trace serving-cost and latency methodology. The additional Summary + Recent comparison is reported in Appendix M.

Reviewer_jhFWOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper reduces how much history an LLM agent reads at each decision. Starting from a 20-item window, a small Transformer ensemble predicts how deleting past interactions would change the agent’s actions or task score. It ranks deletions, keeps 10 to 20 items based on the predicted impact, and replaces repeated field values with references. With GPT-4o in SUMO and GPT-4.1 mini in ScienceWorld, the method cuts total policy tokens by 23% to 26% and 52%, respectively, with no statistically detectable drop in reported performance. In driving, it also saves 9% to 13% over recency and similarity baselines at comparable history sizes. Additional comparisons test recency selection with the same compression and a fixed-size version of the learned selector.

Critical strengths
  1. The supervision is tied directly to the effect of removing history: changes in action distributions for driving and task scores for ScienceWorld, using the full window as the reference. This is a more relevant target than similarity alone. The selection pipeline takes about 1.2 seconds on a CPU. Evaluation uses closed-loop interaction under unseen weather and routes, plus 40-round ScienceWorld runs, with paired bootstrap intervals.
  2. The paper includes useful controls: Recent-K with the same compression and a fixed-K version of the learned selector. These help separate the contribution of compression from learned selection and adaptive history size. The results also show where simpler alternatives are competitive, making it easier to judge what the method actually adds.
Critical weaknesses
  1. The contribution looks incremental, and much of the headline saving comes from simpler components. The method combines deletion-impact prediction, ranking, threshold-based budgeting and deduplication. After applying the same compression, the extra token saving is only 1.6% on held-out ID and 7.6% on ScienceWorld. The gains under severe weather and unseen routes are larger at 12.7% and 7.8%, but the abstract should distinguish these from the total savings. The fixed-K selector also reaches similar operating points on ID and unseen routes, leaving the benefit of adaptive budgeting unclear. Full-window compression and random selection with compression are missing controls, as are comparisons with PACE, Agent-Omit and Context-Picker. Since the predictor never optimizes token cost, it would also help to explain whether it saves tokens by dropping longer records or by selecting records that compress better.

  2. The evidence for preserving performance is weaker than the framing suggests. Driving uses 23, 13 and 27 episodes across the three settings, and non-significant differences alone do not establish equivalence. In ScienceWorld, the confidence interval for the mean-return difference allows a drop of about five points. The headline uses running-max return, where Ours and Recent-12 tie at 96.06, while Recent-12 has higher mean return during selection-active rounds, 76.02 versus 74.22. LongLLMLingua also uses slightly fewer total tokens at a matched history budget, though its reported return is lower. The latency comparison does not settle that trade-off because the hardware is not matched. These results support substantial token savings, but the performance claims need clearer qualification.

Questions for authors

Please add Full-H20 with LORC and random deletion order with LORC at matched K across all four settings. How much does the learned order add over these controls?

Does the selector drop longer records or retain records that compress better? Are dense-traffic records more likely to be dropped?

Which trajectories produced the 1,956 driving labels? Were they collected using the full-context policy? Since deployment produces histories under reduced context, how closely do predicted gaps match realized KL on those histories?

How exactly are the KL labels computed from GPT-4o first-token log-probs? Please specify action-probability normalization, temperature and whether calls are repeated. How repeatable are the labels, especially near the τ = 0.05 threshold?

What is the conceptual advance over existing context selection methods? The approach learns deletion effects and uses them to guide pruning. What new challenge does online in-context RL introduce, and which part of the proposed design specifically addresses it?

What deployment assumption justifies restricting selection to the latest 20 history items and retaining at least 10? Are these application constraints or engineering choices?

Reviewer_9aD5OpenReview ↗
Rating
2
Confidence
4(+1)
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper proposes an adaptive method for selecting interaction history in online in-context reinforcement learning and reports reduced token usage with comparable performance in driving and ScienceWorld experiments.

Critical strengths
  • The paper addresses the practical cost of repeatedly using growing interaction histories in online ICRL. The adaptive selection approach is potentially useful if the reported token savings hold without harming decisions.
  • The evaluation spans closed-loop driving and ScienceWorld, with comparisons against full-history and simpler selection baselines. This breadth makes the claimed cost–performance trade-off more relevant than a result from a single setting.
Critical weaknesses
  1. Comparable performance is not firmly established. Several reported confidence intervals remain compatible with meaningful degradation. The paper does not define an acceptable performance loss for its efficiency trade-off.

  2. The source of the token savings is not fully isolated. The proposed selection and redundancy compression are used together, without a direct evaluation of the proposed selector alone. This limits attribution of the reported gains.

  3. The manuscript has substantial presentation problems that make it harder to read and assess:

  • The abstract is densely set, and spacing throughout the main text is uneven.
  • Many in-text citations run directly into preceding words or punctuation, while several bibliography entries have missing spaces or awkward line breaks.
  • Table 2 is placed at the bottom of page 8, where its lower edge crowds the page number; the accompanying discussion continues on the next page.
  • The reference to “Appendix” near the end of page 8 also lacks a section identifier.

These are recurring issues rather than isolated typographical errors, which could be desk-rejected according to the code of conduct. I ask the authors to check the paper against the ICLR template and provide a carefully typeset version, with particular attention to text spacing, citations, cross-references, and table placement.

Questions for authors
  • The reported performance confidence intervals often include both improvement and degradation. What magnitude of degradation would you consider acceptable when claiming "comparable performance", and do the current results rule out a practically meaningful loss?

  • How much of the token reduction comes from the learned history selection versus the additional redundancy compression? Could you report the proposed selector without compression under the same evaluation protocol?

  • What is the end-to-end inference cost, including embedding, predictor, and context-management overhead, relative to the baselines in both environments? This could affect the practical efficiency claim.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.