acceptodds
Under review as a conference paper at ICLR 2027

Does Probe Conditioning Isolate Composition? An Information-Control Audit of Multi-Hop Evaluation

Abstract

Conditioning multi-hop evaluation on successful component probes is intended to reduce missing-prerequisite confounding. A FULL-FACT condition that supplies both component facts, however, also reveals the answer-bearing second hop, so its performance gain cannot identify composition by itself. We audit this contrast with an information-control suite in MedCompBench, an ontology-controlled biomedical evaluation containing 18,856 real chains and 4,711 counterfactual twins. On Qwen2.5-7B, providing only the bridge improves multiple-choice accuracy by .097 [.059,.137], whereas providing only the answer-bearing second fact improves it by .338 [.286,.392] and providing both facts improves it by .389 [.335,.443]. Across seven Qwen2.5 and Qwen3 checkpoints, the full-fact support gap ranges from .259 to .392 for control-passing models. Independent endpoint checks with Phi-3.5-mini and BioMistral-7B yield gaps of .317 [.264,.374] and .202 [.145,.257]. On matched items, neither endpoint shows a resolved difference from Qwen2.5-7B. Across five prespecified new composed-question wordings, the gap remains positive for all 15 checkpoint–wording pairs. Separately, independently reworded component probes retain only 40.7–58.7% of originally admitted items, showing that the conditional population is strongly prompt-dependent. The original contrast is therefore mainly an answer-support effect rather than an isolated measure of bridge retrieval or composition. Conditional multi-hop evaluations should separate bridge provision, answer-bearing facts, and gate stability before assigning a compositional interpretation.

Primary area:
datasets and benchmarks
Keywords:
language model evaluation, multi-hop reasoning, compositional evaluation, information controls, benchmark methodology, biomedical ontologies, prompt robustness, conditional evaluation

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_C2wZOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper investigates whether conditioning multi-hop evaluation on successful component probes can isolate compositional reasoning. The authors introduce MedCompBench, comprising 18,856 ontology-derived two-hop chains and 4,711 counterfactual twins. They compare LATENT, CoT, and FULL-FACT conditions, alongside information-control interventions, across Qwen, Phi-3.5, and BioMistral models. Results show that most of the FULL-FACT improvement can be attributed to the answer-bearing second-hop fact rather than bridge provision. Independent probes further reveal substantial prompt sensitivity in the conditional evaluation population. The authors conclude that the FULL-FACT–LATENT gap primarily reflects factual support rather than isolated compositional ability.

Critical strengths

1. Systematic experimental controls and robustness analysis. The paper provides a carefully designed information-control suite, including bridge-only, individual-fact, misleading-fact, and length-matched conditions. These interventions help identify the sources of the observed performance gains. Additional analyses across model families, gating criteria, and prompt formulations strengthen the empirical evidence and expose important limitations of conditional multi-hop evaluation.

Critical weaknesses

1. Limited scientific novelty and unclear practical significance. The paper demonstrates that the FULL-FACT versus LATENT gap cannot isolate compositional reasoning because FULL-FACT explicitly provides the answer-bearing second-hop fact. However, this limitation is largely apparent from the experimental design itself. While the additional information controls quantify the effect, the paper does not identify a concrete existing evaluation protocol that makes this specific interpretive error, nor does it demonstrate how its audit changes established scientific conclusions. Furthermore, it does not establish an alternative measure that isolates composition. Consequently, the scientific insight and practical significance beyond identifying an expected information-support effect remain unclear.

2. Insufficient validation of the main findings under independent prerequisite gating. The paper identifies substantial prompt sensitivity in prerequisite gating, which is itself a valuable observation. However, only 970 of 18,856 chains (5.14%) pass the Qwen2.5-7B gate, and independent probes retain just 40.7% to 58.7% of originally admitted examples across models. Although stricter path-specific gates and alternative question wordings are evaluated, the central FULL-FACT and FACT-2 gains are not recomputed on the independently retained subsets. Moreover, 692 of the 970 Qwen2.5-7B examples belong to a single chain type. These limitations leave the stability and representativeness of the central findings insufficiently established.

Questions for authors
  1. Which specific existing evaluation protocols or published conclusions incorrectly interpret the FULL-FACT versus LATENT gap as evidence of compositional reasoning?

  2. What new insight about evaluating compositional reasoning does MedCompBench establish beyond quantifying the expected benefit of providing answer-bearing facts?

  3. What are the LATENT, FACT-2, and FULL-FACT accuracies and corresponding performance gaps when the evaluation population is redefined using the independent held-out prerequisite probes?

  4. How consistent are the main information-control findings across the four ontology chain types, particularly given the strong imbalance in the admitted populations?

Reviewer_Lt94OpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents MedCompBench, which is an ontology-derived biomedical benchmark for auditing multi-hop evaluation. The paper investigates whether the performance difference between latent multi-hop answering and fully supported answering can be interpreted as a measure of compositional reasoning.

Critical strengths
  1. The paper addresses an important issue concerning the validity of multi-hop reasoning evaluation, particularly whether successful component-level probing provides reliable evidence of compositional reasoning.
  2. The paper presents a systematic empirical audit using multiple information controls and robustness analyses.
  3. The proposed MedCompBench provides a structured and reproducible benchmark for evaluating multi-hop reasoning.
Critical weaknesses
  1. Evaluation of compositional reasoning gap in multi-hop QA task is studied in prior works, such as "Measuring and Narrowing the Compositionality Gap in Language Models" (Press et al, EMNLP 2023), where the evaluation and a method for addressing the reasoning gap has been reported on general QA task. The paper should better position the contribution of this paper, against prior works.
  2. In the paper, the key finding is that second-hop factual support accounts for most of the FullFact improvement. This is quite expected, because the second-hop fact contains the target answer.
  3. The model evaluated in the paper is limited to Qwen only. This limits the generalizability of the conclusions being drawn by the paper.
Questions for authors
  1. The paper primarily presents aggregate quantitative results without qualitative case studies examining individual reasoning successes or failures. Detailed examples comparing model responses across different information-control conditions would help illustrate the identified evaluation confounds and provide deeper insights into why models fail to compose individually accessible facts.
  2. In addition to the general prompt instructions in Appendix A and B, can you also include the complete question templates and representative instantiated examples for each evaluation condition to improve clarity and reproducibility.
Reviewer_vV3oOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

An LLM's response to two hop chains can be wrong, but the reason may not be clear. It could be either the LLM doesn't know the first fact, or the second fact or may not know how to combine them despite knowing both facts. In the case the LLM gets the answer right, it could also be due to shortcut association. Hence, the raw score doesn't convey the full story. The paper proposes the benchmark consisting of 18,856 two hop chains from curated medical ontologies. It tests each fact separately and then only tests two hop questions when the model knows both. This benchmark aims to separate composition failure from factual error.

Critical strengths

The paper highlights that the LLM’s ability to do better when facts are given should not be seen as it being able to do compositional reasoning. The LLM might be directly copying the answer. When the facts were given one at a time, the fact containing the answer indeed gave a boost to the performance. Also, when that fact was swapped with incorrect fact, it still copied the incorrect answer. The experiments are well done as only one aspect of question setup is changed in order at each stage with everything left unchanged. This helps to properly locate the cause of performance boost or drop. The authors use large datasets having 18,856 multi-hop questions which is a good artifact for further study.

Critical weaknesses

The paper demonstrates that LLMs simply doing seemingly better in composing after having been given a hint containing answers can't be taken as LLM doing better. However, the paper doesn't yet demonstrate how we can properly measure the composition ability of the LLMs. The paper merely shows that naive measurement can fail. Further, LLM knowing a fact is questionable if it can’t answer the same thing when being rephrased. Only the small models were used for the study, it's not clear if larger LLM struggle similarly or not. Correct answers could also come from a model just memorizing the shortcuts, and its hard to isolate this. Though the dataset has a large number of multi-hop questions, it is only able to use mere ~5% of it. Hence, the main results rest on relatively small numbers of questions.

Questions for authors

As each model is tested on different questions, I wonder if results can be compared fairly. Could authors comment on this?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.