- Rating
- 2(+1)
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper focuses on analyzing why LLMs fail when reasoning about physics problems. The paper argues that a single failure to answer a physics question does not provide enough information, since the model might succeed after several more attempts. To address this issue, the paper proposes PACE, a framework for analyzing the causes of model failures. The paper introduces a three-stage failure attribution process that combines repeated attempts with an analysis of different possible causes of failure. Rather than evaluating models simply in terms of success or failure, this approach aims to identify the specific reasons why models fail on physics problems and uncover the deeper causes behind these failures.
Critical strengths
Analyzing model failures is an important research problem. A single failed response does not provide sufficient information about a model's capabilities. Therefore, investigating and identifying the underlying causes of model failures is a worthwhile research direction.
Critical weaknesses
-
The paper is difficult to understand due to its unclear writing. Many terms, such as Difficulty Cards, Benchmark Failure, and Diagnostic Seed, are introduced without sufficient explanation or clear definitions. The use of numerous unfamiliar terms makes the paper difficult to read. In addition, the overall logical flow is not clear, and many important details are missing. For example, in Lines 315–319, the authors do not clearly explain how the different error categories are identified or defined. Similar issues appear throughout the paper, making it very challenging to follow.
-
Although analyzing the causes of model failures is an important research problem, the conclusions of this paper are not clear, and the paper does not provide a sufficient comparison of the strengths and weaknesses of different models. More importantly, the ultimate goal of understanding model failures is to improve model performance. However, the paper does not discuss how the identified underlying causes of failures could be used to improve the models. As a result, the overall contribution of this work to the field is not clearly demonstrated.
-
The experimental setup is also unclear. For example, why is Gemini the only model evaluated in the cross-domain experiments? Why are MiniMax and DeepSeek used for analysis and validation, respectively? Furthermore, the authors do not clearly justify their choice of physics as the target domain. Based on my understanding of the paper, I do not see why the proposed method needs to focus specifically on physics, since it seems applicable to other problems that require reasoning as well.
Questions for authors
I would like the authors to address the issues raised in the Weaknesses section. Specifically, the authors should explain the overall methodology and conclusions in clear and easy-to-understand language, provide the missing details, present clear and concrete findings obtained using the proposed method, and justify their choices of models and domains in the experiments.
5 comments
Response to Reviewer aWSH (2/2)
W2: The main findings and model-specific differences are unclear, and the implications of failure diagnosis for model improvement need further explanation.
Response: We agree that the diagnostic findings and their implications for model improvement deserve greater prominence. PACE makes recurring reasoning weaknesses identifiable and measurable, revealing distinctions beyond accuracy and broad failure categories.
First, failure persistence does not establish mechanism recurrence. Among 451 category-aligned Atomic failed probe–target observations, 107 (23.7%) received non-Same mechanism attributions. Among 117 Gemini self-route probes failing all three trials, 28 (23.9%) received non-Same mechanism attributions. Independent blinded experts further classified 8/46 (17.4%) category-aligned Atomic observations as non-Same. These findings highlight the need to distinguish recurring reasoning mechanisms from repeated incorrect outcomes.
Second, PACE reveals shared and model-specific reasoning weaknesses. Across 1,577 failed Atomic target runs, formula applicability, concept/representation, and hidden assumptions account for 66.6%–71.2% of each model's failures, versus 3.6%–5.6% for calculation/algebra, suggesting that physical formulation, equation selection, and assumption checking deserve particular attention. Gemini has a higher proportion of formula-applicability errors (27.5%), GPT a larger Other category (24.1%), and Kimi more Partial recurrence (20.9%, at the failed-probe level), suggesting distinct diagnostic priorities across models. Notably, GPT and Kimi show almost identical failure occurrence on GPT-source probes (63.8% vs. 63.9%) but markedly different strict mechanism recurrence (67.6% vs. 48.7%, Table 1). Similar failure rates can conceal different reasoning behaviors.
These findings provide concrete priorities for failure-informed training data selection, targeted evaluation, and mechanism-level tracking across model updates. We will make these conclusions and their implications for model improvement more prominent in the revised Results and Discussion.
W3: Why is Gemini used for cross-domain evaluation, why are MiniMax and DeepSeek assigned different roles, and why is physics the chosen testbed?
Response: We appreciate your request for a clearer experimental rationale. Our design separates domain variation, model transfer, and diagnostic responsibilities.
Experimental design. We evaluate Gemini across three physics subfields and conduct a separate 3 × 3 source–target experiment with GPT, Gemini, and Kimi on atomic physics (a shared domain with sufficient diagnostic seeds for controlled cross-model comparison). These complementary settings examine domain variation and model transfer along separate axes (Section 4.1).
Model roles. MiniMax serves as the source-analysis and probe-generation model, while DeepSeek serves as the scientific-validation and mechanism-attribution model. Using distinct model families separates generation from evaluation and avoids generator self-validation, while executable mathematical checks impose additional non-overridable constraints (Section 3; Appendix C).
Why physics? As discussed in Section 1, physics integrates physical modeling, theory and equation selection, and symbolic or numerical execution, combining reasoning demands relevant across scientific disciplines. These interdependent steps make mechanism-level failures particularly informative, while mathematical and scientific constraints enable rigorous probe validation. Physics thus provides a representative and verifiable testbed for PACE, whose underlying diagnostic principle is not limited to physics.
We will clarify these choices in the revised manuscript and upload the revised manuscript and supplementary diagnostic materials by 14 Nov, AoE. Thank you again for your thoughtful feedback; we welcome further discussion and would appreciate your reconsideration.
Revised Manuscript and Supplementary Materials
Dear reviewer,
Thank you again for your constructive feedback and valuable suggestions. We have uploaded the revised manuscript and updated supplementary materials. Approximately 20% of the main text has been revised, with improvements spanning the Introduction through the Discussion, particularly in the Method and Results sections. The core PACE framework, original experimental design, primary results, and central conclusions remain unchanged. Our revisions focus on refining terminology, clarifying methodological definitions and experimental design, strengthening comparisons with related work, and improving the analysis of model-specific findings and practical implications. We have also incorporated additional evidence developed during the discussion, including an independent blinded expert audit and attribution sensitivity analyses. To support practical reuse, the updated supplementary materials include a PACE diagnostic SKILL package with reusable diagnostic resources, expert reference judgments, validation records, and offline replay scripts. Textual revisions and newly incorporated evidence are highlighted in blue, while newly added appendix sections are marked (NEW) to facilitate review.
Regarding your specific concerns, we have made the following improvements to the revised manuscript:
Methodological clarity (W1). We have revised the Introduction and Section 3 to clarify PACE's central diagnostic question, three-phase workflow, and key terminology. In particular, Sections 3.1–3.3 and Appendices B–C now more clearly distinguish source auditing, target-side failure categories, and source-relative mechanism attribution. Section 4.2 and Figure 3 further clarify the source-audit categories you highlighted in your review.
Findings and implications (W2). We have strengthened Sections 4.3–4.4 to present the central findings more explicitly: repeated probe failure and broad category agreement do not necessarily establish recurrence of the source reasoning breakdown. Table 2 and Figure 4(b) now support a clearer discussion of shared reasoning bottlenecks, model-specific differences, and their implications for targeted evaluation and post-training. Section 5 further connects these findings to benchmark auditing, validated probe generation, and tracking reasoning weaknesses across model updates.
Experimental rationale (W3). Section 4.1 now explains the complementary purposes of cross-domain evaluation with Gemini and the 3 × 3 cross-model experiment in Atomic Physics. It also clarifies the separation of MiniMax's generation role from DeepSeek's validation and attribution roles. The Introduction further explains why physics provides a particularly useful setting for mechanism-level diagnosis, combining physical reasoning challenges with scientifically verifiable constraints.
We sincerely appreciate your detailed feedback, which helped us improve the clarity and presentation of the manuscript. We welcome your further assessment of the revised manuscript and hope these improvements address your concerns.
Best regards,
The Authors
Response to Reviewer aWSH (1/2)
Dear Reviewer,
Thank you for your careful review and detailed feedback. We appreciate your recognition of the importance of understanding scientific reasoning failures. Your comments help us identify where the methodology, findings, and experimental rationale can be presented more clearly.
Response: We appreciate this feedback on readability. PACE is designed around a central diagnostic question: whether an identified reasoning breakdown recurs across scientifically validated problem variations. Its three stages follow the evidence required to answer this question: it first audits errors observed in model evaluations to identify attributable reasoning failures; then abstracts their underlying breakdowns into Difficulty Cards and generates scientifically validated probes; finally, it compares source and target reasoning failures and analyzes their error categories to characterize recurring weaknesses and model-specific reasoning patterns.
Four key terms describe how the evidence progresses:
The categories around Lines 315–319 concern source auditing, which determines whether an apparent benchmark failure is attributable to the model. This differs from the failure taxonomy describing individual target errors and the mechanism attribution comparing target failures with the source breakdown. Their criteria are detailed in Appendix B.1 (Table 5), Appendix B.3, and Appendix C.3 (Table 8). We will clarify these distinctions in the main text, particularly around Figure 3(c).
We will improve the clarity and organization of the revised manuscript: sharpen the presentation of PACE's central question and contributions in the Abstract and Introduction; clarify key terminology, the three-stage workflow, and the distinctions among diagnostic categories; and strengthen methodological signposting and the interpretation of experimental findings, particularly around Lines 315–319 and Figure 3(c).
We hope these revisions will make the paper easier to follow, and warmly welcome your further feedback on the revised presentation.