- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper analyzes, from the perspective of approximate beliefs, why LLM agents in multi-turn active reasoning repeatedly issue uninformative queries and stop making progress. The authors formalize this as a Belief Trap Region, or BTR, a forward-invariant region of belief space in which the progress potential no longer decreases, and show that information-gain maximization, or IG-max, can also fall into such traps when the belief approximation error is large enough. To mitigate this, they propose DR-IG, which maximizes the worst-case information gain over an L1 ambiguity set around the current belief, together with a verbalized belief proxy that builds a top-M hypothesis distribution from the LLM's verbalized confidences. On CircuitDecoding and ConceptInduction, IG-max fails in most episodes by repeating the same actions, whereas DR-IG suppresses the repetition and recovers part of the solve rate. The controller is also run on a real browser task, MiniWoB++ guess-number.
Critical strengths
S1. The paper addresses a frequently observed failure. LLM agents repeating the same queries or getting stuck in uninformative actions is a representative failure of multi-turn agents. Defining BTRs through forward invariance and drift provides a clear vocabulary for discussing this phenomenon.
S2. The analysis of the failure mechanism is concrete. The analysis showing that, in trapped episodes, the belief freezes and the shortlist is regenerated identically at every turn illustrates the failure process well. The observation that a single-step ranking inversion is nearly harmless and that the damage arises from accumulation over multiple turns is also interesting.
S3. The ablation that separates escaping repetition from solving the task is useful. The result that a random top-K or an equal mix of the worst case and the mean breaks the repetition but does not solve the task shows well that action diversity alone is insufficient and that an informative direction is needed.
Critical weaknesses
W1. The theory does not support the proposed method, and the difference from prior work is unclear. The abstract and introduction claim to show theoretically that DR-IG restores progress under bounded belief error, but the paper contains no theorem about DR-IG. For Theorem 2.3 on traps, it is unclear how an actual ranking inversion is guaranteed when L is only an upper bound on the change in information gain, and where the conclusion that the same action keeps being selected comes from when the approximate belief changes over time. Moreover, since the BTR concept comes from prior work, the new contribution of this paper lies in DR-IG, yet its difference from existing robust approaches that optimize for the worst case under uncertain beliefs is not discussed.
W2. The observed traps appear to be an artifact of the evaluation pipeline, and the benefit of DR-IG is confined to a narrow setting. That all five LLMs fall into the same repetition pattern at the same step suggests that the loop is produced by the controller rather than by the LLMs, and the authors themselves call it a "framework-induced loop." Moreover, in both tasks an exact belief can be obtained simply by eliminating hypotheses that are inconsistent with the observations, and the Oracle indeed solves every episode, yet the belief is replaced with a handful of hypotheses verbalized by the LLM, which creates the problem artificially. Without a comparison to simple fixes such as excluding already-asked queries, it is impossible to judge whether DR-IG is necessary. The benefit of DR-IG also appears only at a specific ε with α = 0, and on the real browser task DR-IG does not outperform a simple LLM agent, 80.0% versus 85.0%.
W3. The computation of the verbalized proxy and DR-IG is not described in enough detail to be reproduced. For the LLM to name its top-M hypotheses among 500 circuits, the form in which the candidates are presented is crucial, but this is not explained. The predictive updater used to compute agent-side information gain, how particles are sampled within the ambiguity set, how ε was chosen, and the prompt that presents the shortlist to the LLM are also not described. Furthermore, α = 1 in the ablation is defined as the mean over particles, whereas in the original definition of DR-IG it corresponds to the nominal information gain.
Questions for authors
Q1. What are the theorem and assumptions behind the abstract's claim that DR-IG restores negative drift under bounded belief error? In Theorem 2.3, L is an upper bound on the change in information gain; how is an actual ranking inversion guaranteed, and under what assumptions does the same action keep being selected as the approximate belief changes over time? How does DR-IG differ from existing robust approaches such as robust Bayesian experimental design?
Q2. Does the advantage of DR-IG persist when compared with simple fixes such as excluding already-asked queries, smoothing the proxy, or removing hypotheses inconsistent with the observations?
Q3. How are the 500 hypotheses presented to the LLM? What are the predictive updater, the particle sampling scheme, and the procedure for selecting ε?
2 comments
Response to Reviewer y2XD (2/2)
W3 / Q3 — Candidate presentation, scoring, and reproducibility
In the revised CircuitDecoding runs, the elicitation prompt presents each candidate’s outputs on the queried inputs under permuted IDs. Appendix C provides the elicitation and shortlist prompts, parser, fallback rule and logged rate, token limits, and ambiguity-set sampling procedure. Algorithm 1 collects the control step in one place. Appendix D.4 and Algorithm 2 document the verification experiments and their analysis plans.
We also corrected the definition of to use nominal information gain. The original description as a mean over particles was inaccurate. The original was selected after inspecting the evaluation episodes, and we withdraw the single-peak conclusion from the inconsistently scored sweep. Appendix D.3 reports the revised selection on a separate development split.
Your review changed how we interpret the work. The corrected experiments establish where elicited beliefs can still impair action selection, how much simple fixes recover, and what can be guaranteed when a candidate passes an explicit feedback check.
Response to Reviewer y2XD (1/2)
Thank you for asking us to distinguish failures of the elicited belief from failures introduced by the controller. That distinction was decisive. Our audit found that the original CircuitDecoding, ConceptInduction, and LP-500 runs never successfully elicited a belief: the parser expected four-digit answers from another task. On CircuitDecoding, all 25,596 belief steps fell back to the uniform prior, and a replay without an LLM reproduces the loop signature (Appendix E). We withdraw the affected results and the original attribution of the identical cross-model loops to elicited beliefs.
W1 / Q1 — Theory and relationship to robust Bayesian experimental design
The submitted paper contained no theorem supporting the abstract’s negative-drift claim for DR-IG, and Theorem 2.3 was incorrect. As you point out, a Lipschitz upper bound cannot guarantee a ranking inversion. A one-step inversion also does not establish that the same action will be selected as the proxy changes. The revision removes the abstract claim and replaces the original theorem with separate statements. Proposition 2.2 gives δ≥η/(2L) as a necessary condition for a zero-progress IG-max choice. Proposition 2.3 requires the proxy to continue ranking the uninformative action first along the trajectory. Proposition 2.4 gives a guarantee for the exact robust objective when the ambiguity set contains the true belief and an available action has uniformly positive information gain within the set.
We now position DR-IG explicitly within robust Bayesian experimental design and credit prior work on worst-case expected information gain. Proposition 2.5 shows that, with deterministic binary feedback, exact DR-IG gives the same action ranking as nominal IG. The new results concern what can be checked when the belief is elicited. The mass assigned to hypotheses already refuted by feedback gives a lower bound on the belief error, which can be used to rule out coverage by the ambiguity set (Proposition 2.6). In the revised GuessNumber runs, this check rules out coverage in 84–100% of steps. Under the feedback assumptions of Proposition 2.7, directly testing a verified candidate guarantees positive progress without a bound on the elicited belief error.
W2 / Q2 — What survives the pipeline correction, and how simple fixes compare
With verified elicitation, parse failures account for at most 2.4% of steps in the main tables. Some degradation remains: within ten queries, IG-max underperforms a belief-free random shortlist on three of five GuessNumber models (78% vs. 95%, 52% vs. 91%, and 60% vs. 92%). On CircuitDecoding, one model reaches 7% under IG-max versus 33% with a random shortlist. The revised evidence supports a model-dependent degradation, with the original universal looping claim withdrawn.
We added the simple alternatives you asked for: repeat masking, smoothing, and consistency filtering of the named hypotheses. For the CircuitDecoding model that stalls, DR-IG improves identification from 7% to 36%, with the improvement over IG-max replicated on fresh secrets. Its performance is statistically indistinguishable from masking in the pooled comparison. With tuning on a separate development split, DR-IG reaches 51%, versus 56% for both smoothing and masking (Appendix D.3). We also retain the MiniWoB++ result in Appendix F with the direct comparison you highlighted: 80.0% for DR-IG versus 85.0% for the free-form agent.
On GuessNumber, filtering the named hypotheses and then scoring actions by IG had little effect (62.3% versus 59.7% within six queries). Directly testing candidates that passed the same consistency check reached 85.7% on the design secrets. On 300 fresh secrets, the prespecified comparison with free-form reasoning yielded 79.0% versus 78.3% (47 versus 45 discordant pairs, p=0.92). This replication covers the free-form comparison, and the verified-candidate rule uses the known feedback function. Its distinct contribution is the progress guarantee: all 1,241 guesses drawn from the verified pool had positive information gain, as Proposition 2.7 predicts under its assumptions.