acceptodds
Withdrawn from ICLR 2027

Learning What to Refresh: Counterfactual Observation Selection for Tool-Using Agents

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Tool-using language agents often act on cached observations collected earlier in the trajectory, even though the external world may have changed in the meantime. This induces a belief-repair problem: the agent's internal state remains anchored to historical evidence whose relevance depends on the current task objective and the evolving environment. Re-querying every source is prohibitively expensive, while observation age alone is not a reliable indicator of whether a refresh will improve the next decision. We study task-aware observation refresh: selecting a subset of previously observed sources to revisit before the agent continues execution. Rather than treating staleness as a binary signal or a recency heuristic, we learn refresh decisions from counterfactual interventions at the same task checkpoint, comparing downstream task utility under different refresh sets while holding the agent state fixed. A set-conditioned critic estimates the joint value of candidate refresh sets, allowing the policy to account for complementary and redundant observations while balancing expected utility against acquisition cost. The controller operates on agent-visible history, cached observations, timestamps, and query costs, with the language-model backbone kept frozen. We evaluate the approach on a controlled synthetic benchmark with stale observations and task-dependent source interactions, and compare against no-refresh, always-refresh, recency-based refresh, and additive-value baselines. Our goal is to characterize when selective refresh improves the success–cost trade-off and whether the learned policy transfers to unseen task structures and environment dynamics.

Primary area:
applications to robotics, autonomy, planning
Keywords:
LLM Agents, Active Information Acquisition, Counterfactual Supervision, Observation Refresh, Cost-Aware Decision Making

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_STSeOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper formulates selective observation refresh for tool-using agents operating in dynamic environments as a budgeted belief-repair problem. Rather than relying on recency heuristics or assuming independent per-source utility, the authors propose a set-conditioned critic trained via paired counterfactual rollouts from replayable checkpoints while keeping the language-model backbone frozen. However, the empirical evaluation relies entirely on a numerical synthetic simulator (using a hard-coded linear/pairwise formula rather than actual LLM execution), and the proposed set-conditioned selector underperforms a simpler Additive Value baseline across most main and multi-seed benchmark settings.

Critical strengths
  1. Clear problem formulation: Articulates why observation staleness is not merely a function of age, highlighting how complementary and redundant information affect decision value under a budget.

  2. Transparent reporting of negative results and reproducibility: The authors honestly report that the Additive baseline beats their method in aggregate and provide a thorough, multi-seed supplementary appendix (4,000 tasks across 20 seeds) with exact mathematical formulas.

Critical weaknesses
  1. Absence of real LLM agent evaluation: The paper evaluates a purely numerical, 6-variable synthetic script requiring only Python and NumPy, with zero LLM agent trajectories or natural language tool outputs.

  2. Empirical failure of the main hypothesis: Set-conditioned selection consistently underperforms the simpler, cheaper Additive baseline on in-distribution tasks across budgets and staleness scales, winning only in a single sub-family (Delivery Coordination) and OOD/long-horizon bar charts.

Questions for authors
  1. What accounts for the discrepancy between Figure 2 and Figure 5? Why does the Set-conditioned method achieve 0.59 success at B=4B=4 in Figure 2, while "Paired" supervision at B=4B=4 in Figure 5 achieves 0.72 success?

  2. How are unseen templates, unseen sources, and long-horizon drift (Figure 7) implemented? Given the fixed 6-source lookup formula in Appendix A.1, how does the selector generalize to unseen source combinations if bijb_{ij} relies on a dictionary lookup of sorted source identifiers?

Reviewer_GvUcOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies selective observation refresh for tool-using agents operating with potentially stale cached information. It proposes a set-conditioned critic that estimates the downstream value of candidate refresh sets under an acquisition budget, accounting for complementary and redundant sources. The proposed training framework uses paired counterfactual continuations from shared checkpoints while keeping the underlying language-model agent frozen. Evaluation primarily uses a synthetic benchmark to examine selection quality and success–cost trade-offs.

Critical strengths
  1. Task-aware treatment of information value. Considering the current objective and interactions among sources is more principled than refreshing solely according to observation age. Paired counterfactual supervision also provides a reasonable way to connect refresh decisions to downstream consequences.
Critical weaknesses
  1. The practical significance of the setting is insufficiently established. The paper does not clearly demonstrate when stale observations constitute a substantial bottleneck requiring learned refresh selection. For tasks with hard requirements, verifying decision-critical information may be preferable; for cheaper queries, simple refresh policies may suffice. The proposed method’s operating conditions therefore need stronger empirical justification.

  2. The necessity of a separate learned controller remains unclear. An existing agent could potentially select refresh sources directly from the same task context, cached observations, timestamps, and costs. Without an LLM-based selection baseline or a comparison of overall training and deployment costs, the paper does not establish the additional controller’s practical advantage.

  3. A substantial gap remains between the learning framework and the specified implementation. The method describes a trained critic, but its architecture, training data, and optimization procedure are insufficiently documented. The supplementary implementation instead uses a fixed scoring formula with generator-supplied source attributes. This leaves unclear which results validate learning refresh value from realistically available observations.

  4. The evaluation provides limited evidence of broader impact. Success is determined by a synthetic scoring rule rather than completed language-model action sequences. Moreover, set-conditioned selection does not consistently outperform additive selection, including in the main aggregate comparison. These results characterize synthetic selection behavior but do not convincingly establish improved end-to-end agent performance.

Questions for authors
  1. What realistic task settings make selective refresh necessary, and how are refresh decision points, acquisition budgets, and mandatory verification requirements determined?
  2. Which experiments use a trained critic? Please clarify its architecture, training procedure, and the relationship between the supervision comparison in Figure 5 and the fixed-score supplementary implementation.
  3. How are source utility, staleness, relevance, and interaction features obtained from agent-visible history in deployment, without access to current source contents or generator-provided attributes?
  4. How does the method compare with an LLM directly selecting refresh sources under the same information and budget constraints, including end-to-end task success, latency, and overall computational cost?
Reviewer_L5m1OpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper introduces an approach for tool-based agents to selectively refreshed cached observations in their context. The primary motivation for this work is to note that this is a belief-repair problem which surfaces in non-stationary environments where the agents context on prior obesrvations can become stale and thus can lead to suboptimal decision-making. Furthermore, the refresh value is compositional and thus isolated sources might be less valuable than the sum of their parts.

The authors describe their approach, task-aware observation refresh which involves training a critic to identify set-conditioned source refresh.. For learning the critic, the authors use counterfactual supervision wherein rollouts are generated from the same checkpoint both with and without refresh. The resulting reward/cost is used for training. Finally, post learning, a greedy strategy is employed to decide which candidate sources to select for refresh.

The authors then evaluate their approach on a newly created benchmark of task families like route planning etc. They compare against 4 baselines based on how they manage the observation refresh.

Finally they present some ablations and some studies on o.o.d. generalization.

Critical strengths

S1. The paper writing is clear and is broadly accessible to researchers with different backgrounds.

S2. The proposed idea is intuitive and interesting. The use of counterfactual supervision and identifying sources to refresh is novel. This is of particula importance is non-stationarity is a facet of the real-world.

Critical weaknesses

I thank the authors for their effort. Unfortunately, I do not think this paper is ready for publication just yet.

W1. The emprical evaluation is quite weak. There are many things to consider here.

W1.1: Why are the results published with just one seed instead of a few. Std. deviations and error bars in the main paper are essential. I do not understand this decision considering authors say that results with others are available in appendix.

W1.2.: Why are only hand-crafted baselines considered and not ones from related work?

W1.3: Even amongst the baselines considered, their approach does not outperform Additive.

W1.4: The benchmark is newly created. There is no description of what they are, how staleness is introduced, what non-stationarity is. etc. Without using standard benchmarks it is essential to describe this somewhere.

W1.5: The cost comaprison is misleading. I am not able to see how the massive cost of training pairwise counterfactuals is incorporated here. And it seems the training will be per-domain. which makes this cost comparison necessary.

Questions for authors

Interesting work. Please keep polishing. Happy to reconsider my score if some of the weaknesses are addressed.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.