- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper investigates whether conditioning multi-hop evaluation on successful component probes can isolate compositional reasoning. The authors introduce MedCompBench, comprising 18,856 ontology-derived two-hop chains and 4,711 counterfactual twins. They compare LATENT, CoT, and FULL-FACT conditions, alongside information-control interventions, across Qwen, Phi-3.5, and BioMistral models. Results show that most of the FULL-FACT improvement can be attributed to the answer-bearing second-hop fact rather than bridge provision. Independent probes further reveal substantial prompt sensitivity in the conditional evaluation population. The authors conclude that the FULL-FACT–LATENT gap primarily reflects factual support rather than isolated compositional ability.
Critical strengths
1. Systematic experimental controls and robustness analysis. The paper provides a carefully designed information-control suite, including bridge-only, individual-fact, misleading-fact, and length-matched conditions. These interventions help identify the sources of the observed performance gains. Additional analyses across model families, gating criteria, and prompt formulations strengthen the empirical evidence and expose important limitations of conditional multi-hop evaluation.
Critical weaknesses
1. Limited scientific novelty and unclear practical significance. The paper demonstrates that the FULL-FACT versus LATENT gap cannot isolate compositional reasoning because FULL-FACT explicitly provides the answer-bearing second-hop fact. However, this limitation is largely apparent from the experimental design itself. While the additional information controls quantify the effect, the paper does not identify a concrete existing evaluation protocol that makes this specific interpretive error, nor does it demonstrate how its audit changes established scientific conclusions. Furthermore, it does not establish an alternative measure that isolates composition. Consequently, the scientific insight and practical significance beyond identifying an expected information-support effect remain unclear.
2. Insufficient validation of the main findings under independent prerequisite gating. The paper identifies substantial prompt sensitivity in prerequisite gating, which is itself a valuable observation. However, only 970 of 18,856 chains (5.14%) pass the Qwen2.5-7B gate, and independent probes retain just 40.7% to 58.7% of originally admitted examples across models. Although stricter path-specific gates and alternative question wordings are evaluated, the central FULL-FACT and FACT-2 gains are not recomputed on the independently retained subsets. Moreover, 692 of the 970 Qwen2.5-7B examples belong to a single chain type. These limitations leave the stability and representativeness of the central findings insufficiently established.
Questions for authors
-
Which specific existing evaluation protocols or published conclusions incorrectly interpret the FULL-FACT versus LATENT gap as evidence of compositional reasoning?
-
What new insight about evaluating compositional reasoning does MedCompBench establish beyond quantifying the expected benefit of providing answer-bearing facts?
-
What are the LATENT, FACT-2, and FULL-FACT accuracies and corresponding performance gaps when the evaluation population is redefined using the independent held-out prerequisite probes?
-
How consistent are the main information-control findings across the four ontology chain types, particularly given the strong imbalance in the admitted populations?