acceptodds
Under review as a conference paper at ICLR 2027

Beyond Accuracy: Re-evaluating Failures in LLM Physics Reasoning

Abstract

Evaluating physics reasoning in large language models (LLMs) requires more than identifying where models fail: it requires determining whether those failures are evidence of a recurring reasoning weakness. An observed error may disappear under repetition, a persistent error may not survive controlled reformulation, and a new failure may arise from a different reasoning breakdown. This distinction is especially consequential in physics, where errors in physical modeling, formula applicability, and symbolic or numerical execution can produce similar incorrect outcomes. We introduce **PACE** (**P**robe-based **A**ssessment and **C**haracterization of **E**rrors), a physics-grounded diagnostic framework that distinguishes *observed*, *stable*, and *attributable* failure evidence. **PACE** combines selective repeated measurement, mechanism-abstracted Difficulty Cards, controlled probe synthesis, scientific validation, and source-relative attribution to test not only whether a model fails again, but whether the same reasoning breakdown recurs. Across frontier reasoning models and multiple physics subfields, we find that repeated failure, failure on a source-guided probe, and broad error-category agreement each provide useful but incomplete diagnostic evidence. In particular, source-guided probe failures do not necessarily reproduce the source reasoning breakdown, and category-level similarity can conceal differences in the underlying reasoning breakdown. These results expose a gap between reproducing an error and diagnosing a recurring weakness, motivating a reliability-aware view of failure evaluation, in which evidence strength is measured rather than assumed.

Primary area:
datasets and benchmarks
Keywords:
Large Language Models, Physics Reasoning, Diagnostic Evaluation, Failure Analysis, Reasoning Reliability

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_aWSHOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper focuses on analyzing why LLMs fail when reasoning about physics problems. The paper argues that a single failure to answer a physics question does not provide enough information, since the model might succeed after several more attempts. To address this issue, the paper proposes PACE, a framework for analyzing the causes of model failures. The paper introduces a three-stage failure attribution process that combines repeated attempts with an analysis of different possible causes of failure. Rather than evaluating models simply in terms of success or failure, this approach aims to identify the specific reasons why models fail on physics problems and uncover the deeper causes behind these failures.

Critical strengths

Analyzing model failures is an important research problem. A single failed response does not provide sufficient information about a model's capabilities. Therefore, investigating and identifying the underlying causes of model failures is a worthwhile research direction.

Critical weaknesses
  1. The paper is difficult to understand due to its unclear writing. Many terms, such as Difficulty Cards, Benchmark Failure, and Diagnostic Seed, are introduced without sufficient explanation or clear definitions. The use of numerous unfamiliar terms makes the paper difficult to read. In addition, the overall logical flow is not clear, and many important details are missing. For example, in Lines 315–319, the authors do not clearly explain how the different error categories are identified or defined. Similar issues appear throughout the paper, making it very challenging to follow.

  2. Although analyzing the causes of model failures is an important research problem, the conclusions of this paper are not clear, and the paper does not provide a sufficient comparison of the strengths and weaknesses of different models. More importantly, the ultimate goal of understanding model failures is to improve model performance. However, the paper does not discuss how the identified underlying causes of failures could be used to improve the models. As a result, the overall contribution of this work to the field is not clearly demonstrated.

  3. The experimental setup is also unclear. For example, why is Gemini the only model evaluated in the cross-domain experiments? Why are MiniMax and DeepSeek used for analysis and validation, respectively? Furthermore, the authors do not clearly justify their choice of physics as the target domain. Based on my understanding of the paper, I do not see why the proposed method needs to focus specifically on physics, since it seems applicable to other problems that require reasoning as well.

Questions for authors

I would like the authors to address the issues raised in the Weaknesses section. Specifically, the authors should explain the overall methodology and conclusions in clear and easy-to-understand language, provide the missing details, present clear and concrete findings obtained using the proposed method, and justify their choices of models and domains in the experiments.

5 comments
AuthorsOpenReview ↗

Response to Reviewer aWSH (1/2)

Dear Reviewer,

Thank you for your careful review and detailed feedback. We appreciate your recognition of the importance of understanding scientific reasoning failures. Your comments help us identify where the methodology, findings, and experimental rationale can be presented more clearly.

W1: The manuscript lacks clear definitions of key terms, a coherent methodological flow, and sufficient explanation of diagnostic categories, particularly around Lines 315–319.

Response: We appreciate this feedback on readability. PACE is designed around a central diagnostic question: whether an identified reasoning breakdown recurs across scientifically validated problem variations. Its three stages follow the evidence required to answer this question: it first audits errors observed in model evaluations to identify attributable reasoning failures; then abstracts their underlying breakdowns into Difficulty Cards and generates scientifically validated probes; finally, it compares source and target reasoning failures and analyzes their error categories to characterize recurring weaknesses and model-specific reasoning patterns.

Four key terms describe how the evidence progresses:

  • Benchmark Failure: An initially observed incorrect evaluation outcome, which may reflect a model error, a reference/scoring issue, or problem ambiguity.
  • Diagnostic Seed: A source failure retained after auditing, with an attributable, scientifically defensible reasoning breakdown.
  • Difficulty Card: A structured description of the source breakdown, including its bottleneck, triggering conditions, characteristic error, and supporting evidence, used to guide probe construction.
  • Mechanism Attribution: A source-relative judgment of whether failed responses to a validated probe reproduce the Card-defined breakdown (Same / Partial / Different, or Unclear when indeterminate).

The categories around Lines 315–319 concern source auditing, which determines whether an apparent benchmark failure is attributable to the model. This differs from the failure taxonomy describing individual target errors and the mechanism attribution comparing target failures with the source breakdown. Their criteria are detailed in Appendix B.1 (Table 5), Appendix B.3, and Appendix C.3 (Table 8). We will clarify these distinctions in the main text, particularly around Figure 3(c).

We will improve the clarity and organization of the revised manuscript: sharpen the presentation of PACE's central question and contributions in the Abstract and Introduction; clarify key terminology, the three-stage workflow, and the distinctions among diagnostic categories; and strengthen methodological signposting and the interpretation of experimental findings, particularly around Lines 315–319 and Figure 3(c).

We hope these revisions will make the paper easier to follow, and warmly welcome your further feedback on the revised presentation.

AuthorsOpenReview ↗

Response to Reviewer aWSH (2/2)

W2: The main findings and model-specific differences are unclear, and the implications of failure diagnosis for model improvement need further explanation.

Response: We agree that the diagnostic findings and their implications for model improvement deserve greater prominence. PACE makes recurring reasoning weaknesses identifiable and measurable, revealing distinctions beyond accuracy and broad failure categories.

First, failure persistence does not establish mechanism recurrence. Among 451 category-aligned Atomic failed probe–target observations, 107 (23.7%) received non-Same mechanism attributions. Among 117 Gemini self-route probes failing all three trials, 28 (23.9%) received non-Same mechanism attributions. Independent blinded experts further classified 8/46 (17.4%) category-aligned Atomic observations as non-Same. These findings highlight the need to distinguish recurring reasoning mechanisms from repeated incorrect outcomes.

Second, PACE reveals shared and model-specific reasoning weaknesses. Across 1,577 failed Atomic target runs, formula applicability, concept/representation, and hidden assumptions account for 66.6%–71.2% of each model's failures, versus 3.6%–5.6% for calculation/algebra, suggesting that physical formulation, equation selection, and assumption checking deserve particular attention. Gemini has a higher proportion of formula-applicability errors (27.5%), GPT a larger Other category (24.1%), and Kimi more Partial recurrence (20.9%, at the failed-probe level), suggesting distinct diagnostic priorities across models. Notably, GPT and Kimi show almost identical failure occurrence on GPT-source probes (63.8% vs. 63.9%) but markedly different strict mechanism recurrence (67.6% vs. 48.7%, Table 1). Similar failure rates can conceal different reasoning behaviors.

These findings provide concrete priorities for failure-informed training data selection, targeted evaluation, and mechanism-level tracking across model updates. We will make these conclusions and their implications for model improvement more prominent in the revised Results and Discussion.

W3: Why is Gemini used for cross-domain evaluation, why are MiniMax and DeepSeek assigned different roles, and why is physics the chosen testbed?

Response: We appreciate your request for a clearer experimental rationale. Our design separates domain variation, model transfer, and diagnostic responsibilities.

Experimental design. We evaluate Gemini across three physics subfields and conduct a separate 3 × 3 source–target experiment with GPT, Gemini, and Kimi on atomic physics (a shared domain with sufficient diagnostic seeds for controlled cross-model comparison). These complementary settings examine domain variation and model transfer along separate axes (Section 4.1).

Model roles. MiniMax serves as the source-analysis and probe-generation model, while DeepSeek serves as the scientific-validation and mechanism-attribution model. Using distinct model families separates generation from evaluation and avoids generator self-validation, while executable mathematical checks impose additional non-overridable constraints (Section 3; Appendix C).

Why physics? As discussed in Section 1, physics integrates physical modeling, theory and equation selection, and symbolic or numerical execution, combining reasoning demands relevant across scientific disciplines. These interdependent steps make mechanism-level failures particularly informative, while mathematical and scientific constraints enable rigorous probe validation. Physics thus provides a representative and verifiable testbed for PACE, whose underlying diagnostic principle is not limited to physics.

We will clarify these choices in the revised manuscript and upload the revised manuscript and supplementary diagnostic materials by 14 Nov, AoE. Thank you again for your thoughtful feedback; we welcome further discussion and would appreciate your reconsideration.

AuthorsOpenReview ↗

Revised Manuscript and Supplementary Materials

Dear reviewer,

Thank you again for your constructive feedback and valuable suggestions. We have uploaded the revised manuscript and updated supplementary materials. Approximately 20% of the main text has been revised, with improvements spanning the Introduction through the Discussion, particularly in the Method and Results sections. The core PACE framework, original experimental design, primary results, and central conclusions remain unchanged. Our revisions focus on refining terminology, clarifying methodological definitions and experimental design, strengthening comparisons with related work, and improving the analysis of model-specific findings and practical implications. We have also incorporated additional evidence developed during the discussion, including an independent blinded expert audit and attribution sensitivity analyses. To support practical reuse, the updated supplementary materials include a PACE diagnostic SKILL package with reusable diagnostic resources, expert reference judgments, validation records, and offline replay scripts. Textual revisions and newly incorporated evidence are highlighted in blue, while newly added appendix sections are marked (NEW) to facilitate review.

Regarding your specific concerns, we have made the following improvements to the revised manuscript:

Methodological clarity (W1). We have revised the Introduction and Section 3 to clarify PACE's central diagnostic question, three-phase workflow, and key terminology. In particular, Sections 3.1–3.3 and Appendices B–C now more clearly distinguish source auditing, target-side failure categories, and source-relative mechanism attribution. Section 4.2 and Figure 3 further clarify the source-audit categories you highlighted in your review.

Findings and implications (W2). We have strengthened Sections 4.3–4.4 to present the central findings more explicitly: repeated probe failure and broad category agreement do not necessarily establish recurrence of the source reasoning breakdown. Table 2 and Figure 4(b) now support a clearer discussion of shared reasoning bottlenecks, model-specific differences, and their implications for targeted evaluation and post-training. Section 5 further connects these findings to benchmark auditing, validated probe generation, and tracking reasoning weaknesses across model updates.

Experimental rationale (W3). Section 4.1 now explains the complementary purposes of cross-domain evaluation with Gemini and the 3 × 3 cross-model experiment in Atomic Physics. It also clarifies the separation of MiniMax's generation role from DeepSeek's validation and attribution roles. The Introduction further explains why physics provides a particularly useful setting for mechanism-level diagnosis, combining physical reasoning challenges with scientifically verifiable constraints.

We sincerely appreciate your detailed feedback, which helped us improve the clarity and presentation of the manuscript. We welcome your further assessment of the revised manuscript and hope these improvements address your concerns.

Best regards,

The Authors

Reviewer_iz4NOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces a framework for diagnosing whether an LLM’s reasoning error reflects a stable and recurring weakness rather than an isolated failure. Starting with errors on physics problems, it summarizes the original failures into structured 'difficulty cards', generates controlled probe questions, and evaluates whether the same failure mechanism is stable across different runs and models.

Its main contribution is that it formulates the failure-mechanism recurrence as an evaluation target and proposes a concrete pipeline for measuring it.

The authors validate their method by applying it to multiple LLMs on physics problems. They compare the mechanism-recurrence labels with some simpler indicators (such as repeated failure and error-category agreement) and conduct human audits of each pipeline stage.

Critical strengths

The paper proposes a new framework to distinguish and analyze different sorts of failure modes, which can be carried out by the following works easily.

Critical weaknesses
  1. PACE is almost entirely LLM-based, but there's not enough evidence supporting its correctness. The paper shows that PACE produces different results from simpler metrics, but it does not show that PACE is more reliable or correct. There's human audit but it's not a fully blinded experiment. (This requirement is not only necessary because of the LLM-based pipeline, but it should be even more important given the practical concerns about LLM reliability in such frameworks.)
  2. The key components of PACE are highly heuristic, and the ablation is totally insufficient to support these design choices. For example they use fully LLM-based methods but there are prior works (e.g. PRISM-Physics) that introduce more rigorous formula-based evaluation and do evaluations in a structured way, so it's the authors' responsibility to explain and support their design choices with more evidence, and also do a more thorough comparison with prior analysis frameworks.
Questions for authors
  1. How sensitive are the results to the heuristic choices in the pipeline? For example, if you slightly change the difficulty card format, judge model, etc., would the conclusions remain stable?
  2. Could the authors compare PACE with simpler or more structured diagnostic baselines to show that the proposed model captures information that existing evaluations cannot provide?
8 comments
AuthorsOpenReview ↗

Response to Reviewer iz4N (1/2)

Dear Reviewer,

Thank you for your careful review and constructive suggestions. We have conducted additional blinded validation and sensitivity analyses to clarify PACE's methodological design and diagnostic contributions.

W1: PACE relies heavily on LLMs, but lacks sufficient evidence of diagnostic reliability and fully blinded human validation.

Response: We appreciate the concern about diagnostic reliability and agree that the connections between PACE's validation procedures and supporting evidence should be clearer.

PACE combines structural checks and executable mathematical verification—including numerical consistency, unit checks, and symbolic equivalence where applicable—with LLM-based scientific review of physical assumptions, applicability conditions, and diagnostic-target preservation (Section 3.2; Appendix C). A failed executable check cannot be overridden by a favorable LLM review. The original four-expert evidence audit found 89.5% of attribution explanations fully supported (95.0% including minor caveats; §4.5, Table 2).

We additionally conducted an independent blinded evaluation of 76 stratified failed probe–target observations (10% of all 758 eligible observations, covering different source–target conditions, physics subfields, and attribution categories). Three physics-background experts independently assessed the source reasoning breakdown and its recurrence using the original source problem, reference solution and answer, supplied source-model responses, generated probe and its reference solution, and three target responses retained in their original order. Experts had no access to PACE's Difficulty Cards, automated labels, or attribution rationales.

AgreementRateκ
Among three experts88.2% (67/76)0.850 (Fleiss)
Expert majority vs. PACE81.6% (62/76)0.665 (Cohen)

The expert-majority comparison shows substantial agreement with PACE's source-relative attributions. We will add the blinded audit to Section 4.5 and Appendix E and clarify the distinct roles of mathematical validation, scientific review, and independent expert assessment.

W2: PACE's heuristic design lacks sufficient justification and empirical support, with inadequate comparison to structured prior frameworks such as PRISM-Physics.

Response: PACE addresses a distinct evaluation question: does an identified source reasoning breakdown recur across scientifically validated problem variations?

Drawing on failure-driven probing, structured process evaluation, and failure-pattern discovery (e.g., StressEval, PRISM-Physics, and ProbeLLM), PACE organizes its components around the evidential requirements of this question: source auditing establishes an attributable failure; Difficulty Cards specify the breakdown to test; controlled generation plus mathematical and scientific validation produce admissible probes; and source-relative attribution assesses recurrence (Section 3; Appendices B–C). These stages jointly establish the evidence needed for source-conditioned recurrence analysis; Q1 below further examines sensitivity to judge choice and Card representation.

The comparison covers scope and construction (Cols. 2–3), analysis and validation (Cols. 4–6), and evaluation objectives (Cols. 7–8):

WorkApplication DomainDynamic GenerationQuality ValidationProcess AnalysisFailure DiagnosisRepeat TestingMechanism Recurrence
Step Verifier [1]Math——✓———
UGPhysics [2]Physics—✓—✓——
PRISM-Physics [3]Physics—✓✓✓——
SymPyBench [4]Physics✓✓—✓——
StressEval [5]Knowledge QA✓✓✓✓——
ProbeLLM [6]Multi-domain✓✓—✓—Partial
PACE (Ours)Physics (Extensible)✓✓✓✓✓✓

PRISM-Physics emphasizes structured process evaluation, while ProbeLLM identifies recurring failure patterns. Building on these complementary directions, PACE introduces a source-conditioned protocol that measures whether an identified reasoning breakdown recurs across controlled, scientifically validated problem variations.

References

[1] Lightman et al. Let's Verify Step by Step. ICLR, 2024.

[2] Xu et al. UGPhysics: A Comprehensive Benchmark for Undergraduate Physics Reasoning with Large Language Models. ICML, 2025.

[3] Zhao et al. PRISM-Physics: Causal DAG-Based Process Evaluation for Physics Reasoning. ICLR, 2026.

[4] Imani et al. SymPyBench: A Dynamic Benchmark for Scientific Reasoning with Executable Python Code. EACL, 2026.

[5] Chen et al. StressEval: Failure-Driven Dynamic Benchmarking for Knowledge-Intensive Reasoning in Large Language Models. IJCAI, 2026.

[6] Huang et al. ProbeLLM: Automating Principled Diagnosis of LLM Failures. ICML, 2026.

AuthorsOpenReview ↗

Response to Reviewer iz4N (2/2)

Q1: How sensitive are PACE's conclusions to heuristic choices, particularly the attribution judge and Difficulty Card representation?

Response: We tested attribution-stage sensitivity on the largest eligible subset underlying our main recurrence findings (649 failed probe–target observations from the Atomic cross-model evaluation). For judge sensitivity, we varied DeepSeek V4 Flash, GLM-5.3-Flash, and Qwen3.8-Flash while holding the source evidence, Difficulty Cards, target responses, and attribution rubric fixed.

MetricDeepSeek V4 Flash (Original)GLM-5.3-FlashQwen3.8-Flash
Strict recurrence56.4%57.6%59.3%
Broad recurrence71.6%74.4%75.2%

For Card sensitivity, we used the same 649 cases with DeepSeek V4 Flash fixed as judge, comparing original Cards against a deterministically condensed version retaining core mechanism information and a prose reformulation preserving the scientific content. A manual spot-check examined information preservation before evaluation.

MetricOriginalCompactProse
Strict recurrence56.4%60.9%58.7%
Broad recurrence71.6%73.7%72.6%

Pairwise label agreement across judges was 83.2%–88.9%; Compact and Prose Cards agreed with the original on 88.0% and 90.0% of cases, respectively. These instance-level results, alongside similar aggregate recurrence rates, support the robustness of PACE's principal mechanism-level findings under the tested judge and Card variations. Detailed configurations and results will be added to the Appendix.

Q2: What diagnostic value does PACE provide beyond simpler metrics or existing structured evaluation methods?

Response: As discussed in W2, structured process evaluation and failure-pattern discovery provide complementary diagnostic capabilities. PACE adds source-relative mechanism recurrence, measuring whether an identified reasoning breakdown reappears across validated problem variations. We demonstrate its additional diagnostic value against simpler behavioral and category-based measures below.

At the mechanism level, even persistent failures need not reproduce the source breakdown: 28/117 (23.9%) Gemini self-route probes failing all three trials received non-Same mechanism attributions (Appendix D.3). Likewise, 107/451 (23.7%) category-aligned Atomic failed probe–target observations received non-Same mechanism attributions (Section 4.4; Appendix D.2). In an independent blinded audit, experts also classified 8/46 (17.4%) category-aligned Atomic observations as non-Same. Together, these findings show that repeated failure and broad error-category agreement do not establish mechanism recurrence.

At the model level, GPT and Kimi show nearly identical failure occurrence on GPT-source probes (63.8% vs. 63.9%) but markedly different strict recurrence (67.6% vs. 48.7%, Table 1), revealing model-specific differences concealed by aggregate failure rates. Figure 4(b) further identifies shared bottlenecks in formula applicability, concept/representation, and hidden assumptions (66.6%–71.2% of failed runs), compared with calculation/algebra (3.6%–5.6%), alongside distinct patterns such as Gemini's higher formula-applicability failures (27.5%) and GPT's more Other failures (24.1%). These findings highlight physical formulation, equation selection, and assumption checking as priorities for targeted evaluation and training.

These results motivate targeted failure diagnosis, failure-informed training data selection, and mechanism-level tracking across model versions. We will make these findings and their implications clearer in Section 4.4 and the Discussion.

Thank you again for your constructive feedback. We will upload the revised manuscript and supplementary diagnostic materials by 14 Nov, AoE. We would appreciate your reconsideration and welcome further discussion.

Sincerely,

The Authors

Thanks to the authors for their detailed feedback. I think these statistics largely addressed my concerns. Overall, I believe the problem is practical, and the proposed method can be useful, but these extra statistics are very necessary to validate that the method is a good method instead of just a randomly designed heuristic method. I think even more can be done in this paper, for example, ablations to validate each segment of the designed Card, etc. However, I understand the discussion period is time-limited, so these can be done further after the rebuttal stage. I am currently raising my score for this as an encouragement, and I look forward to seeing the revised PDF and reconsidering my score.

Reviewer_WS6POpenReview ↗
Rating
4
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper studies what evidence is needed to interpret an observed LLM error as a recurring reasoning weakness. It introduces PACE, a physics-grounded diagnostic framework that distinguishes observed failures, behavioral persistence under repeated evaluation, and source-relative recurrence of a specific reasoning breakdown. The framework combines source auditing, Difficulty Cards, controlled probe generation, scientific validation, and attribution of probe failures relative to the original diagnosis. Its central claim is that repeatedly obtaining incorrect answers—even on targeted probes, or within the same broad error category—does not establish that the same reasoning weakness has recurred. In the atomic-physics cross-model evaluation, only 366 of 649 failed probe–target instances, or 56.4%, are attributed as strictly reproducing the source breakdown. Including partial recurrence raises this to 71.6%. Moreover, among Gemini self-route probes that fail on all three evaluations, 23.9% are still not attributed as strict recurrence. Thus, even persistent failure on a deliberately constructed probe is an imperfect proxy for reproducing the intended weakness. The analysis of error categories strengthens this argument: among 451 instances whose failed runs all match the source error category, 107, or 23.7%, are not attributed as the same concrete breakdown. Broad categories can therefore conceal diagnostically important differences, which is a useful finding for researchers who rely on coarse error taxonomies to characterize model weaknesses.

Critical strengths

The motivation is well founded. Trainers and users of models should not read an evaluation from its scores alone; they also need to know which errors are reproducible and systematic, because only those are worth publicizing as a concrete failure mode of the model and worth fixing case by case. Overall reliability matters as well, but the immediate concern here is individual problems.

The experiments are reasonably solid. I strongly endorse the arrangement in which the AI performs the first pass and humans review afterwards: it is the most economical way to preserve end-to-end quality. Having examined the examples, I find the authors’ analysis reasonable and the AI system well aligned with it.

I think this could be turned into a skill—largely reusing the paper’s own prompts—that provides a workflow for diagnosing model capability deficits during RL. I also see a clear application to failure-driven data synthesis and post-training: before constructing a large training set around a suspected weakness, it is useful to confirm that the diagnosis survives repeated sampling and controlled modifications of the problem.

Critical weaknesses

The use cases of PACE are not analyzed in depth. The authors introduce a good tool, but they do not show clearly how to use it, nor what conclusions the evaluation supports once it has been used. I would like to see more analysis of the reproducible errors: what systematic deficits a given model has, what this tells users to watch out for, which problems’ failures trainers should focus on, and whether the deficits can be strengthened category by category.

Questions for authors
  1. Can PACE give some conclusion on the character of a model? For diagnosis, we can now restrict attention to the stable failure modes. Do different models exhibit clear patterns, and are those patterns worth the attention of model trainers or users? I would like an answer at least for model trainers, since I am one.

  2. Can your system assess the character of problems? Some problems are ambiguous, and models answer them incorrectly at random. I would strongly support using this system to clean existing benchmarks so as to obtain benchmarks with higher signal.

  3. I strongly encourage the release of a reusable SKILL.md containing source problems, supporting responses, Difficulty Cards, validated probes, attribution rubrics, expert reference judgments, and replay scripts. The reproducibility statement already describes supplementary PACE-Bench data and executed prompt templates, while additional diagnostic materials are planned for later release and raw transcripts are excluded. A replayable, expert-anchored subset would substantially improve practical reuse.

4 comments
AuthorsOpenReview ↗

Response to Reviewer WS6P

Dear Reviewer,

Thank you for your encouraging review and thoughtful suggestions, particularly from the perspective of model training. We address your three questions below.

W1 & Q1: Can PACE identify shared and model-specific failure patterns, and provide actionable insights for model trainers?

Response: Yes. PACE identifies both shared reasoning bottlenecks and distinct model-specific failure patterns that directly inform post-training.

Shared bottlenecks. Across 1,577 failed target runs in the Atomic 3 × 3 evaluation, formula applicability, concept/representation, and hidden assumptions jointly account for 66.6%–71.2% of failures across the three models, versus 3.6%–5.6% for calculation/algebra (Figure 4(b)). These findings motivate training data and evaluations emphasizing physical problem formulation, equation selection, and assumption checking, rather than focusing primarily on additional algebraic computation.

Model-specific patterns. Among failed runs, Gemini has a relatively high proportion of formula-applicability errors (27.5%, versus 23.1% GPT and 25.1% Kimi), suggesting targeted tests of equation selection under changing physical conditions. GPT has the largest Other category (24.1%, versus 14.4% Gemini and 20.4% Kimi), motivating finer failure analysis before targeted data synthesis. At the failed-probe level, Kimi exhibits more Partial recurrence (20.9%, versus 11.4% GPT and 12.8% Gemini), suggesting the value of examining which parts of the original reasoning breakdown persist across related problems.

Verifying model improvements. On GPT-source probes, GPT and Kimi have nearly identical failure occurrence (63.8% vs. 63.9%), yet substantially different strict mechanism recurrence (67.6% vs. 48.7%, Table 1). PACE probes can therefore be reused after targeted training or model updates to examine whether the specific weakness being addressed still recurs, rather than relying only on aggregate accuracy.

We will include this trainer-focused analysis in the revised Results and Discussion sections.

Q2: Can PACE identify ambiguous or flawed benchmark problems to improve evaluation quality?

Response: Yes—source auditing is already built into Phase I of PACE (Section 4.2, Appendix B.1). Before generating probes, PACE audits the problem, reference solution, and model traces to filter out flawed benchmark items.

Among 471 initial operational failures, our audit filtered out 236 reference/scoring defects or algebraically equivalent answers (e.g., an image-charge derivation rejected despite equivalence, and a soap-bubble reference answer off by 10×10\times in Appendix B.1), 126 missing-context or convention-ambiguity cases, and 10 unattributable cases—isolating 99 genuine model failures (94 accepted Difficulty Cards). This provides a practical basis for identifying questionable benchmark records and selecting higher-signal diagnostic examples.

Q3: Can PACE release a reusable SKILL.md package with diagnostic evidence, expert judgments, and replay scripts?

Response: We agree that a reusable diagnostic workflow would improve PACE's practical utility. We have prepared three complementary resources: nine task-specific prompt templates, PACE-Bench with 545 records across six model/domain collections, and an agent-ready SKILL.md diagnostic package.

The package includes six complete Source–Response–Card–Probe–Target evidence chains covering Same/Partial/Different, attribution rubrics, probe-validation records, and reference judgments from three physics-trained reviewers. Two additional examples illustrate source-audit and probe-validation rejection.

A lightweight offline replay script checks the saved evidence chains and generates readable reports without API calls; all six bundled cases have been successfully tested.

We will upload the revised manuscript and supplementary diagnostic materials by 14 Nov, AoE. Thank you again for your valuable feedback, and we welcome further discussion.

Sincerely,

The Authors

AuthorsOpenReview ↗

Revised Manuscript and Supplementary Materials

Dear Reviewer,

Thank you again for your constructive feedback and valuable suggestions. We have uploaded the revised manuscript and updated supplementary materials. Approximately 20% of the main text has been revised, with improvements spanning the Introduction through the Discussion, particularly in the Method and Results sections. The core PACE framework, original experimental design, primary results, and central conclusions remain unchanged. Our revisions focus on refining terminology, clarifying methodological definitions and experimental design, strengthening comparisons with related work, and improving the analysis of model-specific findings and practical implications. We have also incorporated additional evidence developed during the discussion, including an independent blinded expert audit and attribution sensitivity analyses. To support practical reuse, the updated supplementary materials include a PACE diagnostic SKILL package with reusable diagnostic resources, expert reference judgments, validation records, and offline replay scripts. Textual revisions and newly incorporated evidence are highlighted in blue, while newly added appendix sections are marked (NEW) to facilitate review.

Regarding your specific suggestions, we have strengthened the trainer-oriented findings in Section 4.4 (Figure 4(b)), identifying shared reasoning bottlenecks and model-specific failure patterns, with practical implications for targeted post-training. Section 4.3 (Table 2) further highlights how similar failure rates can conceal different levels of mechanism recurrence. For benchmark quality, Section 4.2 and Appendix B.1 now more clearly present the source-audit results and representative cases, showing how PACE distinguishes genuine model failures from reference/scoring issues and ambiguous problems. Finally, following your suggestion, we have included a reusable PACE SKILL package in the updated supplementary materials, with complete diagnostic evidence chains, expert reference judgments, validation records, attribution rubrics, and offline replay scripts.

We greatly appreciate your encouraging assessment and insightful suggestions.

Best regards,

The Authors

Apologies for the delayed reply. I am very satisfied with the response, and all of my concerns have been addressed. Thank you for the substantial effort you put into the additional analyses and reusable supplementary materials.

I believe rubric-guided model evaluation combined with human review is a practical approach to studying complex reasoning failures. I am happy to maintain my rating of 4 and remain strongly supportive of acceptance.

Thank you so much for your encouraging feedback and continued strong support! We truly appreciate your thoughtful suggestions, which have helped us strengthen the paper and its practical contributions.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Rebuttal Summary

Dear Area Chair,

We sincerely thank you and all reviewers for the constructive discussion. We have uploaded the revised manuscript and updated supplementary materials, with approximately 20% of the main text revised while keeping the core framework and primary results unchanged. Below, we summarize the rebuttal outcomes and why we believe PACE merits acceptance.

Reviewer Feedback and Rebuttal Outcomes

The discussion resulted in a positive shift in reviewer assessments, with scores changing from 4/1/1 to 4/2/2.

WS6P (4 → 4, Confidence: 4): To address questions about diagnostic insights, benchmark quality, and practical reuse, we strengthened model-specific findings, clarified source-audit evidence, and provided a reusable diagnostic SKILL package. The reviewer confirmed that all concerns were addressed and remained strongly supportive of acceptance.

iz4N (1 → 2, Confidence: 3): To address concerns about methodological validity and the rationale for PACE's design, we added independent blinded expert validation, attribution sensitivity analyses, and clearer comparisons with prior work. The reviewer acknowledged that the additional statistics largely addressed their concerns, recognized PACE's practical value, and raised their score.

aWSH (1 → 2, Confidence: 3): To address concerns about methodological clarity, experimental design, and the interpretation of results, we clarified framework definitions and strengthened model-specific analyses. The reviewer acknowledged that some concerns were addressed, raised their score, and stated that they would not strongly oppose acceptance.

Core Contributions and Acceptance Rationale

  • A new evaluation objective beyond accuracy: PACE distinguishes observed failures, stable failures, and source-relative mechanism recurrence, introducing the latter as a distinct diagnostic objective. Through source auditing, validated probes, and mechanism-level attribution, it evaluates not only whether models fail, but whether the underlying reasoning breakdown recurs across problem variations.
  • Scientifically meaningful diagnostic findings: PACE reveals shared reasoning bottlenecks and model-specific differences that conventional failure metrics cannot distinguish. For example, GPT and Kimi exhibit nearly identical failure occurrence (63.8% vs. 63.9%) but substantially different strict recurrence (67.6% vs. 48.7%).
  • Actionable diagnosis and practical value: PACE supports benchmark quality auditing, comparison of reasoning weaknesses across models and model versions, and identification of recurring mechanisms to guide targeted training and evaluation. Independent blinded expert validation and attribution sensitivity analyses support its diagnostic reliability, while reusable diagnostic resources facilitate practical adoption.

We respectfully ask the Area Chair to consider these contributions, the additional evidence, and the positive changes in reviewer assessments when evaluating PACE for acceptance.

Thank you very much for your time and consideration.

Sincerely,

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.