acceptodds
Under review as a conference paper at ICLR 2027

When to Prompt, When to Update: Context-Parameter Allocation for Language Agents

Abstract

Language agents must follow structured protocols, adapt to evolving environments, and operate under memory, latency and cost constraints. Adaptation then requires deciding where task-relevant information should live; in the prompt, which is fast and reversible but bounded, or in the parameters, which is persistent but expensive to write. We formalize this as an information-allocation problem. Two limits make the trade-off concrete: parameter writes have diminishing returns, while the usable prompt is bounded not by the context window but by attention dilution, for which we derive and empirically validate a scaling law that provides a measurable prompt-side constraint. We propose an oracle-student-controller architecture that acts on this analysis. A compact student is first distilled to acquire the desired output schema. During deployment a controller preserves the schema by construction, projects the semantic state into a feasible prompt domain with an approximation guarantee, and triggers oracle-supervised parameter updates only on persistent error. The trigger is computable from student and environment observable signals alone, so the oracle is never queried in order to decide whether to use it. Across Multi-Fidelity Bayesian Optimization, all 812 WebArena tasks, and both ALFWorld splits, we studied Llama-3.1-8B and Mistral-7B performance as students. Experimental results show improvement of their baseline performance by integrating the adaptive allocation schema, which is also compared against current state-of-the-art methods.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Language agents, In-context learning, Context management, Parameter-efficient fine-tuning, Continual learning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_CKqqOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper studies adaptation in resource-constrained language agents from an information-allocation perspective, treating prompts and model parameters as two knowledge-storage mechanisms with different capacities, update costs, and persistence. The authors propose an oracle–student–controller architecture. The student first learns the required output interface through offline distillation. During deployment, the controller preserves protocol information and compresses the semantic state into a prompt budget suitable for the student. Observable signals from the student and environment trigger oracle corrections, while a forgetting-risk gate determines whether accumulated corrections should be consolidated into the model parameters. Immediate corrections can affect the current task, whereas parameter updates take place between tasks.

I appreciate the paper's perspective. Its main value lies in bringing context management and parameter adaptation, which are often studied separately, into a unified problem formulation and translating this view into an executable system. The reported empirical improvements also support the usefulness of this perspective.

Critical strengths
  1. The paper frames adaptation as a decision about whether task information should remain in the prompt or be consolidated into model parameters under constraints on capacity, latency, and write costs. This perspective reveals the connection between context management and parameter learning and offers a clear way to reason about transient state and persistent behavioral corrections. I consider this reframing conceptually valuable, with the potential to inform future research on agent memory and continual adaptation.

  2. The authors turn this perspective into a complete workflow involving protocol preservation, semantic-state selection, selective oracle correction, and parameter updates. The system distinguishes immediate assistance within the current task from parameter learning that can benefit subsequent tasks. In particular, the trigger relies only on student and environment signals, allowing it to decide whether to request assistance without first obtaining an oracle response. This makes selective oracle use operationally feasible.

  3. The paper uses prompt length and attention behavior to characterize the student's reliable operating range and uses this characterization to motivate a budget-constrained projection procedure. Although these proxy analyses have explicit assumptions and a limited scope, they provide a testable basis for setting context budgets. The authors also connect theory and experiments through attention measurements, prompt-length sweeps, and calibration across models, improving the framework's interpretability.

  4. For Llama-3.1-8B, success increases from 11.6% with distillation alone to 27.3% on WebArena, and from 56.7% to 79.9% on the unseen ALFWorld split. Under a matched oracle budget, the method also outperforms PivotRL on WebArena, achieving 27.3% versus 23.6%. In addition, the authors separately examine projection, immediate corrections, parameter updates, different oracles, and constrained decoding. These experiments help explain the roles of individual mechanisms beyond demonstrating the full system's improvement over an unadapted model. (Tables 2, 15, 17, 21, and 38–41.)

Critical weaknesses
  1. In Definition 4, If TT denotes the number of applied parameter updates, as described in Algorithm 1, then using the reported values qD/dL=0.5q_D/d_L=0.5, ϵ=0.1\epsilon=0.1, η=2×10−4\eta=2\times10^{-4}, and λ=0.01\lambda=0.01 yields a rejection probability of approximately 3.4×10−63.4\times10^{-6} at T=17T=17. This magnitude appears difficult to reconcile with the update suppression reported in Table 32. Please clarify the actual unit of the counter TT, whether additional scaling or normalization is applied, and provide a corresponding trace of gate values and update decisions. This is a request to clarify the consistency between the mathematical description and implementation, rather than a conclusion that the experimental results are incorrect. Nevertheless, the explanation would affect my assessment of the risk-gating mechanism.

  2. The authors explicitly state that the attention theorem provides a prompt-side proxy constraint and does not explain every form of long-context failure. The calibration across models in Table 9 also provides useful initial evidence. I would like to better understand how much calibration cost the theoretical proxy saves relative to directly sweeping prompt length, or how much it improves budget selection for new models and input types. This would help establish how the analysis guides controller design in practice.

  3. Table 27 indicates that protocol protection is an important source of improvement. I suggest comparing the proposed projection strategy with simple truncation, retrieval, or generic compression methods that preserve the same protocol information and operate under matched token and oracle budgets on WebArena and ALFWorld. The interval-summary comparison already provided for MFBO is helpful. Extending similar comparisons to interactive tasks would better distinguish the benefits of the unified framework from those of the specific information-selection algorithm.

  4. In Table 21, the frozen-adapter control already approaches the performance of the full online system, while Table 37 shows that the full system has higher per-step latency than the distillation-only baseline. These results suggest that the value of continued updates may depend on deployment duration and the extent of environmental change. Learning curves across different task orders and comparisons under matched total compute budgets would help identify when continual adaptation is preferable to adaptation before deployment or less expensive context management. Learning across tasks is consistent with the paper's objective; the concern here is the stability and scope of its benefits.

Questions for authors
  1. How are the update counter, scaling, and normalization in the gate defined in the actual implementation, and can these details explain the differences in update counts in Table 32?

  2. What measurable benefits does the theoretically guided prompt budget provide over direct empirical calibration?

  3. With identical protocol protection and resource budgets, how much additional benefit does the proposed projection strategy offer over simpler methods?

  4. Which deployment durations, environmental changes, or task-stream settings most clearly demonstrate the value of continued parameter updates?

Reviewer_WYevOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies how a language agent should allocate information between its prompt and its parameters. A controller preserves protocol information while selecting semantic context under a token budget, calls an expert oracle when local signals indicate difficulty, and buffers corrections for LoRA updates between tasks. A randomized gate uses a null-space forgetting proxy to decide whether to apply an update. The analysis includes conditional attention-dilution and forgetting bounds and a submodular context-selection guarantee. Experiments cover multi-fidelity Bayesian optimization, WebArena, and ALFWorld with two student models. The main Llama-3.1-8B configuration reports 27.3% WebArena success and 79.9% ALFWorld-unseen success, with oracle-call rates of 6.8% and 5.2%, respectively.

Critical strengths
  1. A practical allocation recipe with unusually extensive controls. The appendix examines matched oracle budgets, frozen adaptation, self-consistency, alternative context selectors, constrained decoding, open-weight teachers, and several continual-learning methods. The reported latency includes uncertainty sampling and amortized training, rather than counting only the student forward pass. These checks make the complete system interesting as a resource-constrained agent design, even though its WebArena success remains substantially below the oracle-only result.

  2. The experiments address when different adaptation mechanisms help. Table 29 distinguishes growing-context tasks from a persistent naming change: projection supplies most of the gain in the former, while parameter updates matter more in the latter. The context ablations and protocol-preservation controls further support a useful empirical distinction between retaining current information and learning recurring corrections. This is more informative than reporting only an aggregate improvement from a combined system.

Critical weaknesses
  1. The theoretical quantities are not adequately connected to the implemented controller's measurements. For the gate, Definition 4 and Table 22 give rejection probabilities of approximately 2.60×10⁻⁶ at T=13 and 1.20×10⁻⁴ at T=600, with an infinite-horizon upper bound of about 0.05. Yet Tables 32–34 attribute substantial, systematic update suppression to it, including 13 updates versus 17 when always updating. A random outcome is not logically impossible, and policies can follow different trajectories, but the stated finite-horizon probabilities do not explain this behavior without a different counter, normalization, or implementation. For prompt sizing, Table 5's Q_eff K_eff≈8.81 and head dimension 128 imply a score spread at most 2QK/√128≈1.56 if those are valid uniform norm bounds. Table 42 instead reports a median spread of 7.9 at length 2048. Appendix A.5.1 measures maxima on a short calibration prompt; those need not bound longer/different prompts. Thus the conditional theorem can remain correct while its empirical sizing interpretation is unsupported. Both discrepancies need the exact calibration and implementation definitions before the claimed analytical validation can be assessed.

  2. The task-level evidence and uncertainty do not yet support a consistent account of the headline gains. In Table 16, the six WebArena site counts sum to 812, and their rounded success rates uniquely imply 223 successes for the proposed method and 174 for PivotRL. These yield 27.5% and 21.4%, rather than the aggregate 27.3% and 21.9%; rounding alone cannot reconcile them. Table 18 similarly implies 108/134 successes for the proposed method on ALFWorld unseen, versus the reported 79.9% (107/134). Separately, the Wilson intervals and task-level McNemar tests do not account for dependence introduced by carrying updated parameters across the ordered evaluation stream. The paper acknowledges this protocol, but the ten-order check reports coefficient/update-count variability rather than success-rate variability. Corrected records and uncertainty across independent streams are needed to establish the size and reliability of the advantage, particularly for the update/gate contribution.

Questions for authors
  1. What exact values of the rejection probability were used for each candidate update in Tables 32–34? Please provide the implemented formula, counter units/reset rules, and any scaling or normalization beyond Definition 4. Can these reproduce the reported update counts under the settings in Table 22? This determines whether the evaluated gate is the proposed one.
  2. Are the norms in Table 5 and the score spreads in Table 42 measured with the same checkpoint, vector definition and prompt family? How are short-prompt maxima validated as bounds at the tested longer lengths? Please reconcile the implied spread of about 1.56 with the measured median of 7.9, or explicitly delimit the rule as a calibrated heuristic.
  3. Can you reconcile the per-site/per-category counts in Tables 16 and 18 with the headline rates and paired discordant counts in Table 19? Using the reconciled records, what is the variability of success differences across independent stream orders or runs, especially against oracle-budget-matched and fixed-count update controls? These requests concern validation of the reported claims.
3 comments

We thank the reviewer for a detailed and quantitative review. We answer each point below.

Q1 — Gate: formula, counter, reset, values

Implemented formula. The gate evaluates Definition 4 directly:

r    = (1 − ηλ)²
E_T  = (q_D/d_L) · η² · (1 − r^T) / (1 − r)
reject with probability min(1, E_T / ε)

It is evaluated once per task, between tasks, and only when |B| ≥ B_min = 32.

Counter units and reset. T counts applied updates. It is incremented only when an update is applied. A rejected candidate leaves T unchanged and keeps its supervision in the buffer, where it is reconsidered at the next candidate. T is reset when accuracy on the fixed 200-prompt schema probe returns to its initial value. This happened twice in the 600-step runs (App. A.7.5).

The value of ε was misreported. The reviewer's arithmetic is correct: with ε = 0.10, as printed in Table 22 and Appendix B, E_T/ε never exceeds 0.05 and the gate would admit essentially every update. The deployed tolerance is ε = 6.53·10⁻⁷, and 0.10 is an error in the paper. The ε column of Table 33 was reported on the same incorrect scale. The correct values are given below.

How ε was set. E_T bounds a normalized forgetting whose normalizer, ½ν²d_Lλ_max(H₀), is not estimated (App. A.3). The tolerance therefore cannot be derived from the bound and has to be set empirically. A value of order 10⁻⁷ is the natural scale: E_T grows by about η²·(q_D/d_L) = 2·10⁻⁸ per applied update. We chose ε on a held-out task stream, separate from the evaluation tasks, as the best balance between schema retention and semantic learning. Table 33 then sweeps ε around this value on the evaluation stream and confirms the choice. The deployed value has probe 0.99, semantic probe 0.56 and the highest success, 27.3%. Smaller tolerances under-learn (semantic probe 0.46–0.52) and larger ones forget (probe 0.92–0.96).

With the correct values, Definition 4 reproduces every update count in Tables 32–34. In the tables below, "expected" is the mean ± s.d. of admitted updates out of the 17 candidates of the always-update run, computed exactly from Definition 4. Probe-triggered resets are not included.

Table 22 (deployed setting)

QuantityPrintedCorrect
ε0.106.53·10⁻⁷

Table 33 (tolerance sweep, B_min = 32)

ε printedε correctReported updatesExpected under Def. 4
0.021.31·10⁻⁷45.3 ± 0.6
0.053.27·10⁻⁷810.1 ± 1.3
0.106.53·10⁻⁷1313.0 ± 1.4
0.201.31·10⁻⁶1614.8 ± 1.2
0.503.27·10⁻⁶1716.1 ± 0.9

The B_min rows of Table 33 (8, 64, 128) use the deployed tolerance, 6.53·10⁻⁷. Their update counts differ because B_min changes how many candidates reach the gate.

Table 34 (misspecified q_D/d_L, ε = 6.53·10⁻⁷)

Assumed q_D/d_LReported updatesExpected under Def. 4
0.1251715.9 ± 1.0
0.251514.8 ± 1.2
0.501313.0 ± 1.4
1.001010.1 ± 1.3
2.0066.4 ± 0.8

With the correct values and no per-row tuning, fourteen of the fifteen reported counts lie within 1.7 standard deviations of the value Definition 4 predicts. The remaining one (ε = 1.31·10⁻⁷: 4 against 5.3 ± 0.6) is at about two standard deviations.

Rejection probability for each candidate. The rejection probability depends on a candidate only through its counter value. The values below are therefore the exact probabilities applied to every candidate in the deployed configuration (Table 32, "Risk-aware gate"). T is the counter value the update would produce if admitted.

T123456789
p_reject0.0310.0610.0920.1220.1530.1840.2140.2450.276
T1011121314151617
p_reject0.3060.3370.3670.3980.4290.4590.4900.520

For any other row of Tables 33 and 34, multiply by (6.53·10⁻⁷/ε) · (q_D/d_L / 0.50). After each probe-triggered reset, the sequence restarts at T = 1.

What Table 32 establishes. App. A.3 states that E_T is a relative exposure score with an unestimated normalizer. Table 32 compares update controls at matched count (13 updates each). At that count, the gate keeps a schema-probe accuracy of 0.99, against 0.92 for the fixed schedule and 0.95 for random selection, while reaching the highest semantic-probe accuracy (0.56). The gate's benefit therefore comes from which updates it admits, not only from admitting fewer.

Q2 — Norms in Table 5 and score spreads in Table 42

Same checkpoint, same vectors, same prompt family. Table 5 and Table 42 use the same distilled Llama-3.1-8B checkpoint and the same prompt family. Both measure the post-projection query and key vectors, before RoPE. They differ in one respect only: prompt length. Table 5 takes maxima over positions of a 64-unit calibration prompt, while Table 42 measures score spreads at length 2048.

The reviewer's comparison therefore isolates exactly the effect we intended to expose. Over the same checkpoint and prompt family, the realized spread grows with length, from the ≈1.56 implied by the short-prompt maxima to a median of 7.9 at 2048. This is why the paper treats the two quantities differently:

  • Sizing (Table 5, Prop. 1). The short-prompt maxima are calibration features, not length-uniform bounds. Proposition 1 combines them with a scalar fitted across models (Table 9). This fitted scalar is what absorbs the growth of the spread with length. The resulting rule predicts held-out models within 10–15%, which is the width of the behavioral transition itself (about 300 tokens around 2160, Table 6).
  • Theorem 1 (Table 42). The theorem is conditional on a spread bound, and it holds with the spread measured at the deployment length. With the median of 7.9 it gives a weaker, but valid, lower bound on the attention mass on R. The measured length exponents (−0.94 overall, −1.02 for diffuse heads, Tables 8 and 42) match the predicted Θ(1/ℓ) dilution.

The paper already states that "measuring finite scores at one length is not enough to establish a length-uniform bound" (App. A.7.6). In the revision, Sec. 5.1 will call Proposition 1 a calibrated sizing rule, so that the short-prompt maxima are not read as uniform bounds.


Q3(a) — Per-site and per-category counts

We thank the reviewer for this reconstruction. It located two transcription errors in the breakdown tables. The aggregates, Table 19 and the per-task records are correct and consistent with each other. The errors were:

  • Table 16, Map row. The entries for all four methods were wrong. Correct values: no distill 7.3% (8/109), distill only 21.1% (23/109), PivotRL 32.1% (35/109), ours 29.4% (32/109).
  • Table 18, Clean & place, ours. The correct value is 77.4% (24/31), not 80.6%.

With these corrections, every column sums exactly to its aggregate:

Table 16 (WebArena, Llama-3.1-8B), corrected

SiteTasksNo distillDistill onlyPivotRLOurs
Shopping1875.3 (10)13.9 (26)24.6 (46)31.0 (58)
Shopping admin1822.7 (5)8.2 (15)18.7 (34)25.3 (46)
Reddit1066.6 (7)17.0 (18)27.4 (29)32.1 (34)
GitLab1801.7 (3)6.1 (11)16.1 (29)23.9 (43)
Map1097.3 (8)21.1 (23)32.1 (35)29.4 (32)
Multi-site480.0 (0)2.1 (1)10.4 (5)18.8 (9)
All8124.1 (33)11.6 (94)21.9 (178)27.3 (222)

Table 18 (ALFWorld unseen, Llama-3.1-8B), corrected

CategoryGamesNo distillDistill onlyPivotRLOurs
Pick & place2450.0 (12)75.0 (18)87.5 (21)91.7 (22)
Examine in light2138.1 (8)61.9 (13)76.2 (16)85.7 (18)
Clean & place3135.5 (11)58.1 (18)71.0 (22)77.4 (24)
Heat & place2330.4 (7)52.2 (12)69.6 (16)78.3 (18)
Cool & place2128.6 (6)52.4 (11)71.4 (15)76.2 (16)
Pick two & place1414.3 (2)28.6 (4)50.0 (7)64.3 (9)
All13434.3 (46)56.7 (76)72.4 (97)79.9 (107)

The totals (33, 94, 178, 222 on WebArena; 46, 76, 97, 107 on ALFWorld-unseen) are exactly those behind Tables 2, 15 and 17. They also satisfy every paired identity in Table 19: on WebArena, b − c = 77 − 33 = 44 = 222 − 178 against PivotRL, and likewise 41, 128, 73, 26, 30 and −10 for the other rows. On pooled ALFWorld, b − c = 28 − 7 = 21 = 225 − 204. Recomputing from these counts reproduces every Wilson interval in Tables 2, 15 and 17 and every McNemar p-value in Table 19. No headline number or statistical test changes.

The corrected Map row changes one qualitative statement. On Map, PivotRL is slightly ahead (32.1 vs. 29.4, 3 tasks), where the original table showed a small margin for us. This sharpens the paper's reading of Table 16. Map has the shortest observations (2.4k tokens), so the projection has the least to do there. Our largest margins are on the sites with the longest observations: GitLab (8.4k, +7.8 points), multi-site (7.1k, +8.4) and Shopping admin (6.8k, +6.6). In Table 18, the largest margin over PivotRL remains the longest-horizon category, Pick two & place (+14.3). The revised PDF includes both corrected tables with k/n counts.

Q3(b) — Dependence from carrying adapters across the stream

We agree that the online protocol makes outcomes order-dependent for configurations that carry parameters across tasks. Several facts in the paper bound how much this can affect the reported comparisons:

  1. Only our online configurations carry state. Distill-only, MemGPT, LLMLingua, FocusAgent, PO-without-oracle and PivotRL evaluate every task with fixed parameters, so their outcomes are independent across tasks. For ours, updates are applied only after a task's outcome has been recorded, so no task is evaluated by parameters trained on that task. Adapters are also reset between the ALFWorld splits.
  2. The frozen-adapter control removes the dependence entirely. With parameters fixed during evaluation (Table 21), the system reaches 25.9 on WebArena. That is still above PivotRL with a matched oracle budget (23.6), PO+periodic updates (24.1) and PivotRL (21.9). The advantage over the baselines therefore does not rely on the online protocol; the online protocol adds a further 1.4 points.
  3. The update path is stable across stream orders. Across ten stream orders, the update count changes by at most one (App. A.7.5). Table 35 measures the effect of individual updates on ALFWorld-unseen: the 13 gated updates have a net effect of +9.7 points, about 0.75 points per update on average. One update more or fewer is therefore an order-of-magnitude smaller effect than the margins over the matched-budget baselines (5.3 points over PivotRL+matched, 6.0 over PO+periodic on ALFWorld-unseen).
  4. The update/gate contribution is measured within the same stream. Table 32 compares the gate with the fixed-schedule and random controls at the same update count and on the same task order. These comparisons therefore do not mix order effects with policy effects.

In the revision we state that the McNemar p-values for the online configuration are conditional on the task order. The frozen-adapter comparison (point 2) is the order-independent version of the main claim.


We thank the reviewer again. The checks on the gate scale, the norm measurements and the breakdown tables will make the presentation more precise. The main results rest on internally consistent per-task records and reproducible statistics, and they hold under the frozen-adapter control, where outcomes are independent across tasks.

Reviewer_iTX6OpenReview ↗
Rating
1
Confidence
3(−1)
Informativeness
2(+1)
Presentation
1
Soundness
1
Read review
Paper summary

The paper treats agent adaptation as deciding whether task information should live in the prompt or in the parameters. A distilled student is deployed with a controller with three parts. It keeps the output schema in the prompt and selects other context under a token budget. It calls an oracle when signals from the student or the environment indicate trouble. It applies buffered corrections as LoRA updates through a randomized forgetting gate. Experiments cover MFBO, WebArena and ALFWorld with Llama-3.1-8B and Mistral-7B students.

Critical strengths

This paper contains serious formatting violations and should be desk reject.

Critical weaknesses

This paper contains serious formatting violations and should be desk reject.

  1. The gate does not select updates by content. By the authors' own response, the rejection probability depends only on the update count and rises by about 0.031 per update. The gate is therefore a random schedule with a tuned slope. It cannot "choose which updates to admit". In Table 32, all success rates (24.8–27.9) are within one confidence interval, each from a single run.
  2. The sizing rule is fitted, not derived. Section 4.1 says the constants are "measured rather than fitted", but Table 9 fits a scalar. Table 9 cannot be reproduced from Eq. (5) with one scalar: prediction / exp(2QK/√d_h) ranges from 193 to 542 across models. The match in Table 1 depends on the chosen m/τ. With the measured spread in Table 42 (7.9), the bound gives about 1.3M tokens.
  3. The main comparison mixes context management with updates. PivotRL and Distill-only run on unmanaged 5.9k-token prompts, above the 2.2k threshold the paper itself identifies. Projection alone already beats PivotRL (22.3 vs. 21.9). The compression baselines do not protect the protocol, which the paper's own ablation credits with 7.9–11.2 points.
  4. Train/test separation on WebArena is unclear. All 812 tasks are evaluated. It is not stated where the "separate training templates" (Table 20) and the "disjoint stream of 200 tasks" for the frozen-adapter control come from.
  5. Key numbers are inconsistent. The experiments use a V100 16GB GPU, but Table 5 assumes 24GB with 16GB of fp16 weights. The update contribution on ALFWorld-unseen is 4.5 points (Sec. 5.5), 9.7 (Table 35) and 1.5 (Table 21). Add to these the ε value misreported by five orders of magnitude and the Table 16/18 errors. It is hard to know which numbers to trust.
  6. No run-to-run variance. Each configuration is run once. Online updates make task outcomes dependent.
Questions for authors

See weaknesses above

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.