- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes Active Hypothesis Analysis (AHA), an inference-time procedure for LM reasoning when textual premises admit multiple structural interpretations. A frozen language model proposes candidate DAGs and answers local binary questions. A deterministic controller computes candidate predictions, selects questions using expected information gain, and updates a posterior over candidates. The paper also provides a bounded-norm kernel score-margin result and a conditional posterior-concentration guarantee.
Evaluation uses Corr2Cause and an all-negative Extended Corr2Cause stress test. The paper reports substantial improvements over direct prediction, budget-matched self-consistency, and random-query Bayesian aggregation. However, the relationship between the posterior updates and the stated final entailment rule requires clarification before these improvements can be attributed to active querying.
Critical strengths
-
The research question is meaningful, and the procedure is explicit and inspectable. Allocating additional inference-time computation to questions that distinguish competing interpretations is worth studying. The separation between candidate generation, deterministic predictions, question acquisition, and posterior updating provides a clear framework for investigating this question.
-
The evaluation includes relevant mechanism-oriented comparisons and acknowledges important limitations. The same-backbone comparisons between direct prediction, self-consistency, random querying, and information-gain querying could provide useful evidence about where improvements originate, provided the scoring pipeline is established. The paper also explicitly distinguishes textual reasoning probes from physical interventions and discusses candidate omission, biased responses, and the conditional scope of its guarantees.
Critical weaknesses
-
Definition 2 predicts entailment only when every proposed candidate satisfies the hypothesis. Appendix F explicitly states that every reported binary score uses this rule, while Appendix A.4 states that the fixed-budget algorithm retains every candidate. With a fixed candidate pool and fixed candidate-level hypothesis judgments, changing posterior weights cannot change this prediction. Consequently, the reported differences between random queries, information-gain queries, and different response sources in Table 3 cannot be explained by the described scoring procedure.
-
Extended Corr2Cause contains only negative examples, so always rejecting achieves 100% accuracy, compared with AHA’s reported 83.2%. This slice measures false acceptance but cannot independently establish overall reasoning ability or generalization to larger problems. The nonzero extended F1 values also lack a scoring definition reconciled with this label distribution. Separately, the headline Corr2Cause accuracy is 97.1%, whereas the five-run estimate is 94.1 ± 0.3%; the paper attributes these to different evaluations without sufficiently identifying their samples and correspondence. Table 2 retains historical size-band results that the text says cannot be reconciled with the pooled result.
Questions for authors
-
How can local responses or posterior updates change the final binary prediction under Definition 2? Please provide the executed scoring procedure and clarify whether any candidate filtering, regeneration, posterior thresholding, or additional hypothesis assessment occurs.
-
With identical cached candidate pools and hypothesis judgments, what are the results for zero queries, random queries, and information-gain queries? Please also specify how candidate generation, response tokens, and thinking tokens enter the budget comparison.
-
What exact definition produces the extended F1 values on the all-negative test slice? Please provide confusion matrices and identify the evaluation samples and runs underlying 97.1% versus 94.1 ± 0.3% Corr2Cause accuracy.
-
What connects concentration on a target candidate in Theorem 2 to correct universal entailment judgments when the premise genuinely admits multiple interpretations? Please clarify which guarantee, if any, applies to the implemented final prediction rule and how the assumed conditional aggregated-response reliability is assessed.
We thank the reviewer for their detailed feedback. We present the following responses to the key issues raised.
W1.
We wish to clarify the executed scoring procedure. Every reported label is h evaluated on the MAP candidate after the last round, ℓ̂ = h(G_k̂) with k̂ = argmax_k π_k^(T), which is the candidate Algorithm 1 returns. Definition 2 and Appendix F described a unanimity readout; both now state the MAP readout explicitly (blue). With the MAP readout, posterior updates change the label, which explains the differences between rows in Table 3 (worked example in Sec. 4). New Table 9 confirms this on identical pools: unanimity gives 71.1 F1 with or without questions, while IG + MAP gives 93.2 F1.
W2.
Q1.
ℓ̂ = h(G_MAP), where G_MAP is the highest-posterior candidate after T rounds. There is no candidate filtering after validation, no regeneration, no posterior threshold and no additional hypothesis assessment. Responses change ℓ̂ by changing the MAP candidate.
Q2.
New Table 9 gives results on identical cached pools and hypothesis judgments: T = 0 (unanimity / majority), random + MAP, IG + MAP and IG + unanimity. Results (F1 / acc.): T = 0 unanimity 71.1 / 88.8; T = 0 majority 73.6 / 87.6; random + MAP 74.3 / 76.9; IG + unanimity 71.1 / 88.8; IG + MAP (AHA) 93.2 / 97.1. Budget: self-consistency, random and IG each use 60 LLM calls per instance (T × M = 20 × 3) and 30,000 tokens, counting local responses and thinking tokens. Candidate generation is shared by all pool-based rows and is not included in these counts.
Q3.
The extended "F1" was the unweighted mean of per-template rejection accuracies (now "Macro"); 83.2 is the rejection rate over all 18,000 instances. Table 9 gives Corr2Cause confusion matrices. 97.1 is the reported accuracy; the 94.1 mention is removed.
Q4.
For δ_c < 1/2, π*⁽ᵀ⁾ > 1 − δ_c makes G* the MAP candidate, so Theorem 2 also bounds Pr(ℓ̂ ≠ h(G*)) (new paragraph after Theorem 2). This equals the benchmark label whenever h is constant on C(P). When consistent interpretations genuinely disagree on h, no single-interpretation readout is guaranteed correct (Proposition 1). Reliability is assessed only marginally: 2,400 isolated local questions give 62.4–85.9% accuracy (App. E.2). These figures do not verify the conditional assumption (Eq. 9), as the paper states.