- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The paper proposes Complex Probabilistic Interventional Reasoning, an intervention-based framework for selecting among real, complex, and heterogeneous dynamics of LLM belief updates, and applies it to LLM-as-a-judge evaluation.
Critical strengths
The manuscript is fluently written and adopts the vocabulary of rigorous methodology throughout (frozen splits, pre-specified gates, matched controls).
Critical weaknesses
- The title centers on phase geometry, but the complex model is not selected on either real LLM (Table 4); the only support comes from synthetic environments where complex dynamics were planted.
- The paraphrase and domain held-out results for both LLMs are described in Sec. 5 but never reported.
- The target is defined as a frequency of binary samples (Eq. 11) but described as a reported probability (Sec. 5); samples per condition and decoding settings are not given.
- The headline 11.1% → 0.7% result is based on 144 candidate pairs (24 scenarios × 6 pairs), i.e., roughly 16 → 1 pair.
- The main text contains one two-row table; all substantive results are in the appendix.
- Several defined metrics (IEE/OEE, NLL, the geometry-consistency residual) and listed baselines (normative Bayes, logistic, scalar real SSM, nonlinear real RNN) never appear in any result, and the gate counts (7/13/6) are never enumerated.
- Multiple cross-references are incorrect: Appendix A does not contain the “complete parameter grid”, Appendix F does not report gate outcomes, Appendix G does not contain the estimators, and the “†” in Table 6 is undefined.
- Several figures have overlapping labels or legends (Figs. 2c, 3a, 3b, 4a, 4b), and Figs. 2a–c and 4a duplicate Tables 2–5.
- On real LLMs, all fitted families have logit RMSE of 1.17–2.43, and no prior-only or normative Bayes baseline is reported, so it is unclear whether any family is better than learning nothing.
Questions for authors
Given the number of incorrect cross-references, unreported metrics, and missing results, could the authors clarify how the manuscript was checked before submission?