acceptodds
Under review as a conference paper at ICLR 2027

Active Hypothesis Analysis for Language Model Reasoning under Ambiguity

Abstract

Reasoning from text about relationships among variables is challenging when multiple structural interpretations are compatible with the evidence. Given correlations and conditional independencies, a reasoning model must determine whether a proposed relation is entailed without knowing which interpretation is correct. We introduce Active Hypothesis Analysis for Language Models under Ambiguity (AHA), an inference-time procedure that explicitly represents competing interpretations and actively tests them. AHA combines a frozen language model with a deterministic controller: the model proposes candidate interpretations and answers targeted local questions, while the controller selects questions by expected information gain and maintains a posterior over hypotheses. We provide a kernel-based result relating representation similarity to differences in hypothesis scores, and a finite-budget guarantee for posterior concentration under candidate coverage, informative queries, and calibrated responses. On Corr2Cause, AHA achieves F1, compared with 51.6 for budget-matched self-consistency and 74.3 for random-query Bayesian aggregation using the same GLM-5.1 backbone. Local-question accuracy ranges from 62.4% to 85.9%, with concentration degrading as response noise approaches random guessing. We further introduce Extended Corr2Cause, an all-negative rejection stress test. Results on both benchmarks show that explicit uncertainty can direct inference-time computation toward questions that distinguish competing interpretations.

Primary area:
foundation or frontier models, including LLMs
Keywords:
language models, reasoning, adaptive belief control, probabilistic search

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_1LcUOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Active Hypothesis Analysis (AHA), an inference-time procedure for LM reasoning when textual premises admit multiple structural interpretations. A frozen language model proposes candidate DAGs and answers local binary questions. A deterministic controller computes candidate predictions, selects questions using expected information gain, and updates a posterior over candidates. The paper also provides a bounded-norm kernel score-margin result and a conditional posterior-concentration guarantee.

Evaluation uses Corr2Cause and an all-negative Extended Corr2Cause stress test. The paper reports substantial improvements over direct prediction, budget-matched self-consistency, and random-query Bayesian aggregation. However, the relationship between the posterior updates and the stated final entailment rule requires clarification before these improvements can be attributed to active querying.

Critical strengths
  1. The research question is meaningful, and the procedure is explicit and inspectable. Allocating additional inference-time computation to questions that distinguish competing interpretations is worth studying. The separation between candidate generation, deterministic predictions, question acquisition, and posterior updating provides a clear framework for investigating this question.

  2. The evaluation includes relevant mechanism-oriented comparisons and acknowledges important limitations. The same-backbone comparisons between direct prediction, self-consistency, random querying, and information-gain querying could provide useful evidence about where improvements originate, provided the scoring pipeline is established. The paper also explicitly distinguishes textual reasoning probes from physical interventions and discusses candidate omission, biased responses, and the conditional scope of its guarantees.

Critical weaknesses
  1. Definition 2 predicts entailment only when every proposed candidate satisfies the hypothesis. Appendix F explicitly states that every reported binary score uses this rule, while Appendix A.4 states that the fixed-budget algorithm retains every candidate. With a fixed candidate pool and fixed candidate-level hypothesis judgments, changing posterior weights cannot change this prediction. Consequently, the reported differences between random queries, information-gain queries, and different response sources in Table 3 cannot be explained by the described scoring procedure.

  2. Extended Corr2Cause contains only negative examples, so always rejecting achieves 100% accuracy, compared with AHA’s reported 83.2%. This slice measures false acceptance but cannot independently establish overall reasoning ability or generalization to larger problems. The nonzero extended F1 values also lack a scoring definition reconciled with this label distribution. Separately, the headline Corr2Cause accuracy is 97.1%, whereas the five-run estimate is 94.1 ± 0.3%; the paper attributes these to different evaluations without sufficiently identifying their samples and correspondence. Table 2 retains historical size-band results that the text says cannot be reconciled with the pooled result.

Questions for authors
  1. How can local responses or posterior updates change the final binary prediction under Definition 2? Please provide the executed scoring procedure and clarify whether any candidate filtering, regeneration, posterior thresholding, or additional hypothesis assessment occurs.

  2. With identical cached candidate pools and hypothesis judgments, what are the results for zero queries, random queries, and information-gain queries? Please also specify how candidate generation, response tokens, and thinking tokens enter the budget comparison.

  3. What exact definition produces the extended F1 values on the all-negative test slice? Please provide confusion matrices and identify the evaluation samples and runs underlying 97.1% versus 94.1 ± 0.3% Corr2Cause accuracy.

  4. What connects concentration on a target candidate in Theorem 2 to correct universal entailment judgments when the premise genuinely admits multiple interpretations? Please clarify which guarantee, if any, applies to the implemented final prediction rule and how the assumed conditional aggregated-response reliability is assessed.

1 comment

We thank the reviewer for their detailed feedback. We present the following responses to the key issues raised.

W1.

"With a fixed candidate pool …, changing posterior weights cannot change this prediction."

We wish to clarify the executed scoring procedure. Every reported label is h evaluated on the MAP candidate after the last round, ℓ̂ = h(G_k̂) with k̂ = argmax_k π_k^(T), which is the candidate Algorithm 1 returns. Definition 2 and Appendix F described a unanimity readout; both now state the MAP readout explicitly (blue). With the MAP readout, posterior updates change the label, which explains the differences between rows in Table 3 (worked example in Sec. 4). New Table 9 confirms this on identical pools: unanimity gives 71.1 F1 with or without questions, while IG + MAP gives 93.2 F1.

W2.

"Extended Corr2Cause contains only negative examples, so always rejecting achieves 100% accuracy."

  1. Always-reject gets 100% on the extended slice; we list it (Table 7) and describe the slice as a false-acceptance test only. Acceptance of entailed hypotheses is measured on Corr2Cause: AHA accepts 1,661 of 1,800 positives (Table 9).
  2. The extended "F1" values are per-template rejection accuracies and their unweighted mean, now relabelled "Macro".
  3. 97.1 is the reported accuracy. 94.1 ± 0.3 appeared only in appendix prose and is removed.
  4. The size-band columns are removed from Table 2, which now reports overall accuracy only (SFT 52.2, DPO 58.0, AHA 83.2), together with the two size-band appendix tables.

Q1.

"How can local responses or posterior updates change the final binary prediction under Definition 2?"

ℓ̂ = h(G_MAP), where G_MAP is the highest-posterior candidate after T rounds. There is no candidate filtering after validation, no regeneration, no posterior threshold and no additional hypothesis assessment. Responses change ℓ̂ by changing the MAP candidate.

Q2.

"With identical cached candidate pools …, what are the results for zero queries, random queries, and information-gain queries?"

New Table 9 gives results on identical cached pools and hypothesis judgments: T = 0 (unanimity / majority), random + MAP, IG + MAP and IG + unanimity. Results (F1 / acc.): T = 0 unanimity 71.1 / 88.8; T = 0 majority 73.6 / 87.6; random + MAP 74.3 / 76.9; IG + unanimity 71.1 / 88.8; IG + MAP (AHA) 93.2 / 97.1. Budget: self-consistency, random and IG each use 60 LLM calls per instance (T × M = 20 × 3) and 30,000 tokens, counting local responses and thinking tokens. Candidate generation is shared by all pool-based rows and is not included in these counts.

Q3.

"What exact definition produces the extended F1 values on the all-negative test slice?"

The extended "F1" was the unweighted mean of per-template rejection accuracies (now "Macro"); 83.2 is the rejection rate over all 18,000 instances. Table 9 gives Corr2Cause confusion matrices. 97.1 is the reported accuracy; the 94.1 mention is removed.

Q4.

"What connects concentration on a target candidate in Theorem 2 to correct universal entailment judgments?"

For δ_c < 1/2, π*⁽ᵀ⁾ > 1 − δ_c makes G* the MAP candidate, so Theorem 2 also bounds Pr(ℓ̂ ≠ h(G*)) (new paragraph after Theorem 2). This equals the benchmark label whenever h is constant on C(P). When consistent interpretations genuinely disagree on h, no single-interpretation readout is guaranteed correct (Proposition 1). Reliability is assessed only marginally: 2,400 isolated local questions give 62.4–85.9% accuracy (App. E.2). These figures do not verify the conditional assumption (Eq. 9), as the paper states.

Reviewer_AjgwOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2(+1)
Read review
Paper summary

The paper proposes AHA, an inference-time procedure for reasoning from textual premises that admit several structural interpretations. A frozen LLM proposes candidate DAGs and answers local binary questions; a deterministic controller selects questions by expected information gain and updates a posterior over the candidates. Experiments on Corr2Cause with GLM-5.1 report 93.2 ± 0.4 positive-class F1, compared with 51.6 for budget-matched self-consistency and 74.3 for random-query Bayesian aggregation. The paper also includes an extended-variable stress test and an all-negative rejection test.

Critical strengths
  1. Keeping explicit alternatives and querying where they disagree is a sensible way to spend inference computation, and the deterministic reachability calculations and posterior updates make the controller easy to inspect.
  2. The paper is careful about what a concentrated posterior can and cannot establish, e.g. Proposition 1's information limit for indistinguishable interpretations.
Critical weaknesses
  1. The stated prediction rule does not appear to use the result of active testing. Under Definition 2 the final label is positive only if every candidate satisfies the hypothesis, and Algorithm 1 keeps the candidate pool fixed while updating only the weights; Appendix F confirms all reported binary scores use this unanimity rule. For a fixed pool, then, the final label cannot change with the questions, responses, posterior updates, or number of rounds. I cannot reconcile this with the reported gains of information-gain queries over random queries, or with the variation across response sources and noise levels. If the implementation actually uses the MAP candidate, pruning, or a posterior threshold, the authors need to say so, and justify it against the universal-entailment target in Eq. (2); selecting one plausible interpretation does not establish entailment across all consistent ones. The shared-pool, candidates-only control needed to assess the contribution is missing, as the authors acknowledge.
  2. The evaluation record is hard to reconcile. If each F1 and accuracy pair in Table 3 comes from a single confusion matrix or pooled counts, the implied positive-class proportions for AHA (93.2 F1, 97.1 accuracy) and random queries (74.3 F1, 76.9 accuracy) cannot both describe the same fixed test set. The appendix also distinguishes the 97.1 headline accuracy from a five-run 94.1 ± 0.3. The authors need to identify the runs, subsets, and aggregation behind Table 3 before the 18.9-point F1 gap can support the acquisition claim. Separately, on the extended test always rejecting achieves 100% versus AHA's 83.2%, so better rejection alone does not demonstrate reasoning generalization; the reported nonzero extended F1 values still lack a clear scoring definition, and the size-band records are inconsistent.
Questions for authors
  1. How exactly is Algorithm 1's output converted into the binary labels in Tables 1–4? If this is the unanimity rule, how can query acquisition or response noise change those labels at all?
  2. Could you provide TP, FP, FN, and TN counts, test-set identifiers, and run and averaging details for each Table 3 row? It would also help to compare no-query, random-query, and information-gain settings with identical candidate pools and the same scoring rule.
  3. Which examples and metric definitions produced the extended F1 columns and the 83.2% rejection result, and do they come from the stated all-negative slice?
  4. Does candidate validation in Algorithm 1 check consistency with all relations in the premise, or only DAG syntax and acyclicity?
2 comments

We thank the reviewer for their detailed feedback. We present the following responses to the key issues raised.

W1.

"The stated prediction rule does not appear to use the result of active testing."

We wish to clarify the prediction rule. Every reported label is h evaluated on the MAP candidate after the last round, ℓ̂ = h(G_k̂) with k̂ = argmax_k π_k^(T), which is the candidate Algorithm 1 returns. Definition 2 and Appendix F described a unanimity readout; both now state the MAP readout explicitly (blue). Questions therefore change the label by changing the MAP candidate (worked example in Sec. 4).

On universal entailment: a single interpretation does not by itself establish entailment over all consistent ones. For a premise-consistent MAP candidate, the MAP readout agrees with Eq. (2) whenever h takes a single value on the premise-consistent set C(P). When consistent interpretations disagree on h, it is correct only if the MAP candidate is a counterexample. We now state this under Definition 2 and after Theorem 2.

The shared-pool control is new Table 9 (App. F). All rows use identical cached pools on the 7,524 Corr2Cause test instances (1,800 positive):

Configuration (same pools)F1Acc.TPFPFNTNDisagree acc.
Candidates only, unanimity (T = 0)71.188.8100039800568570.2
Candidates only, majority (T = 0)73.687.61295430505529466.1
Budget-matched self-consistency51.654.3156532032352521—
Random questions + Bayes, MAP74.376.915901528210419632.0
IG questions + Bayes, unanimity71.188.8100039800568570.2
IG questions + Bayes, MAP (AHA)93.297.1166179139564596.6

Questions cannot change the unanimity readout, so the two unanimity rows coincide. With the MAP readout, IG questions improve on the best no-question readout by +19.6 F1 and +8.3 accuracy points, and by +26.4 accuracy points (70.2 → 96.6) on instances whose pool members disagree on h. This isolates the contribution of acquisition on a fixed pool.

W2.

"The evaluation record is hard to reconcile."

  1. Table 3 F1 vs. accuracy. The F1 column is the unweighted mean of the six per-template F1 scores. This can be checked from Table 5: e.g. GLM-5.1: (95.3 + 94.7 + 92.6 + 91.8 + 88.4 + 96.2)/6 = 93.2, and every row of Tables 5–6 matches. F1 and accuracy are therefore not two summaries of one pooled confusion matrix, which is why the implied positive rates you computed differ. The paper now states this, and Table 9 gives pooled TP/FP/FN/TN for each configuration.
  2. 97.1 vs. 94.1. 97.1 is the reported accuracy. The 94.1 ± 0.3 appeared only in appendix prose and has been removed; Appendix E.2 now reports only the five-run F1 (93.2 ± 0.4).
  3. Extended slice. Always-reject gets 100% (Table 7); we describe the slice as a false-acceptance test only. Acceptance of entailed hypotheses is measured on Corr2Cause: AHA accepts 1,661 of 1,800 positives with 79 false positives (Table 9).
  4. Extended "F1". These columns are per-template rejection accuracies, and the "F1" value was their unweighted mean (81.9 = mean of 84.8, 84.1, 81.2, 80.2, 75.2, 86.0). We relabelled the column "Macro" in Tables 1 and 6.
  5. Size bands. The inconsistent size-band records are removed: the size-band columns of Table 2 and the two size-band appendix tables (old Tables 7–8).

Q1.

"How exactly is Algorithm 1's output converted into the binary labels in Tables 1–4?"

By the MAP readout: ℓ̂ = h(G_MAP) (corrected Definition 2; Algorithm 1, final line). Queries and response noise change which candidate is MAP, and therefore the label.

Q2.

"Could you provide TP, FP, FN, and TN counts … for each Table 3 row?"

New Table 9 gives TP/FP/FN/TN, calls and tokens for each configuration on the 7,524 Corr2Cause test instances. It also compares no-query, random-query and IG settings on identical cached pools under the same scoring rule (both MAP and unanimity); see the table in our response to W1. All rows use the same 7,524 test instances (1,800 positive), and accuracy recomputed from each row's counts matches the reported value. The F1 column is the per-template average (see W2); the headline row is a single run, and the five-run F1 is 93.2 ± 0.4.

Q3.

"Which examples and metric definitions produced the extended F1 columns and the 83.2% rejection result?"

All 18,000 instances of the stated all-negative slice (1,000 per d = 7–24). Template columns are per-template rejection accuracies; the column labelled "F1" was their unweighted mean (now "Macro"); 83.2 is the rejection rate over all instances.

(Continued in the next comment)

Q4.

"Does candidate validation in Algorithm 1 check consistency with all relations in the premise?"

We wish to clarify what validation guarantees. It checks syntax, acyclicity and the premise relations (now stated in Algorithm 1, line 1), but it does not guarantee that every retained candidate is premise-consistent. Table 9 shows this directly: the unanimity readout rejects 800 of the 1,800 entailed hypotheses, which is possible only if pools contain candidates that violate the premise. This is also where the posterior matters. Local answers contradict such candidates and down-weight them, which is why IG + MAP recovers most of these positives (FN falls from 800 to 139). Appendix F now states this explicitly.

Reviewer_txvZOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The goal of this paper is to improve an LLM's performance when there are several different explanations from the given text. The idea is to effectively ask the LLM several yes/no questions on how one variable is causally linked to another (via asking the LLM n times). It keeps track of the explanation that is most likely after each answer. There are some large gains on causal accuracy benchmarks over pure zero-shot answering, and improves reasoning performance across models compared to SFT and DPO.

Critical strengths
  1. There is a clear motivation of this problem of inference time reasoning, and has general applicability to multiple domains. Additionally, the method is training-free, model-agnostic, and is quite simple to implement.
  2. There are strong empirical gains on the Corr2Cause benchmark over six model backbones going up to 100B parameters. At that scale training or any form of weight updates are expensive, so this method of test time scaling would be beneficial. Additionally, the ablations show that these gains arent from extra parameters as repeated calls and random questioning does not work as well as AHA.
Critical weaknesses
  1. The paper says its final answer is entailed only if all the candidate interpretations agree with each other, so its possible that the targeted questions simply change which candidates are favored rather than changing the actual answer. That contradicts the paper’s main evidence that says that the smart evidence beats the random questions by about 19 F1 points, and the paper never actually explains how the questioning actually affects the final answer. So its not clear whether gains come from asking these questions that the method hinges on, or simply by having the model list several interpretations. There is no example showing how the method actually works.

  2. The results in different parts of the paper disagree with each other. The Corr2Cause result is 97.1 in the Table 1a, but there it says 94.1 in the Appendix E.2. Extended Corr2Cause accuracy is 83.2 in Table 1b, but is 86.0 on Table 7a, and 78.2 in Table 7b.

Questions for authors
  1. Why is the Extended Corr2Cause test slice all-negative? The generated corpus contains positive examples.
  2. Can the authors explain what the correct numbers are supposed to be?
  3. What's the performance when the LLM just proposes candidates instead of doing questioning?
1 comment

We thank the reviewer for their detailed feedback. We present the following responses to the key issues raised.

W1.

"So its not clear whether gains come from asking these questions … or simply by having the model list several interpretations."

We wish to clarify how the final label is produced. Every reported label is obtained by evaluating the hypothesis on the MAP candidate after the last round, ℓ̂ = h(G_k̂) with k̂ = argmax_k π_k^(T), which is the candidate Algorithm 1 returns. Definition 2 and Appendix F described a unanimity readout; both now state the MAP readout explicitly (blue).

Under the MAP readout the questions directly determine the label, because they determine which candidate is MAP. Section 4 now has a worked example of this: for h = A ⇝ B, only G₁ satisfies h. Three YES answers make G₁ the MAP candidate (posterior (729,1,9)/739), so ℓ̂ = 1. A NO on the second check instead gives (1,1,9)/11, so ℓ̂ = h(G₃) = 0. This is also why random and IG questions give different scores (74.3 vs 93.2 F1): they lead to different MAP candidates from the same pool.

New Table 9 confirms this on identical cached pools. The unanimity readout gives the same result with and without questions (71.1 F1, 88.8% accuracy), as expected. The MAP readout after IG questions reaches 93.2 F1 and 97.1% accuracy. The gain is largest on instances whose candidates disagree on the hypothesis, where accuracy rises from 70.2% to 96.6%. The candidates-only comparison is given under Q3.

W2.

"The results in different parts of the paper disagree with each other."

We wish to clarify the reported values. The reported values for GLM-5.1 + AHA are 97.1 Corr2Cause accuracy and 83.2 Extended Corr2Cause accuracy. These are the values in the per-model results tables (Tables 5 and 6), and every other occurrence in the paper is taken from them (Tables 1, 2 and 3).

The other figures you list were not separate headline results. They came from supplementary records that were left in the appendix:

  • 94.1 appeared only in appendix prose (Appendix E.2 and Appendix D), never in a results table. To avoid confusion with the reported 97.1, it has been removed, and Appendix E.2 now reports only the five-run F1 (93.2 ± 0.4).
  • 86.0 and 78.2 were the averages of two size-band breakdowns (by number of variables) from earlier runs, shown in the old Tables 7a/7b. Neither pools to the reported 83.2, so we removed both tables. We also removed the size-band columns from Table 2a, which had mixed one of these breakdowns with the 83.2 overall figure. Table 2a now reports overall accuracy only: SFT 52.2, DPO 58.0, AHA 83.2 (gaps of +31.0 and +25.2). No claim in the paper depended on the size-band breakdown.

All changes are marked in blue in the revised PDF. After the revision, each quantity appears with a single value throughout the paper.

Q1.

"Why is the Extended Corr2Cause test slice all-negative?"

We wish to clarify the role of this slice. It is designed as a false-acceptance stress test: it measures whether a method accepts unsupported hypotheses as the number of variables grows. It therefore does not measure acceptance of entailed hypotheses, and an always-reject baseline scores 100% on it; we list that baseline in Table 7 and describe the slice only in these terms. Acceptance of entailed hypotheses is measured on Corr2Cause, where 1,800 of the 7,524 test instances are positive. There, AHA accepts 1,661 of the 1,800 positives (recall 92.3%) with 79 false positives (new Table 9).

Q2.

"Can the authors explain what the correct numbers are supposed to be?"

The correct values are 97.1 (Corr2Cause accuracy) and 83.2 (Extended Corr2Cause accuracy) for GLM-5.1 + AHA, as given in Tables 5 and 6; see W2 for where the other figures came from and how they were removed.

Q3.

"What's the performance when the LLM just proposes candidates instead of doing questioning?"

We added this control (new Table 9, App. F). On identical cached candidate pools with no questions (T = 0):

Configuration (same pools)F1Acc.
Candidates only, unanimity (T = 0)71.188.8
Candidates only, majority (T = 0)73.687.6
Random questions + Bayes, MAP74.376.9
IG questions + Bayes, MAP (AHA)93.297.1

Proposing candidates alone accounts for part of the gain over direct prediction (34.2 → 73.6 F1). Targeted questioning adds a further +19.6 F1 and +8.3 accuracy points over the best no-question readout. Random questions do not provide this gain: their accuracy falls below the T = 0 rows.

Comments on the paper

1 comment
AuthorsOpenReview ↗

A Clarifying Summary

We thank the reviewers for their careful reading and wish to clarify the two questions shared by all three reviews: how the final label is produced, and whether the reported numbers are consistent.

Every reported label is the hypothesis evaluated on the MAP candidate after the final round, which is the candidate Algorithm 1 returns; Definition 2 and Appendix F described a unanimity readout and now state the MAP readout explicitly, so local answers change the label by changing which candidate is MAP, as a new worked example in Section 4 shows. We also add the same-pool control that all three reviewers requested (new Table 9): on identical cached candidate pools over all 7,524 Corr2Cause test instances, reading the label from the candidates alone gives at most 73.6 F1 and 88.8% accuracy, whereas information-gain questions with the MAP readout give 93.2 F1 and 97.1% accuracy (96.6% vs. 70.2% on instances where candidates disagree), while random questions do not produce this gain; the table includes full confusion matrices, and accuracy recomputed from them matches every reported value. On consistency, each quantity now has a single value (97.1 Corr2Cause accuracy, 83.2 Extended accuracy): the 94.1 figure, which appeared only in appendix prose, and the superseded size-band tables have been removed; F1 is the unweighted mean of the six per-template F1 scores, which explains why it is not tied to accuracy through one pooled confusion matrix; and the Extended column labelled F1 is the mean of per-template rejection accuracies, now labelled Macro, with the always-reject baseline listed. The method and its reported results are unchanged; the revision, marked in blue, states the scoring rule precisely and adds the control that isolates the contribution of active questioning.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.