- Rating
- 2
- Confidence
- 1
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper introduces ProofCover, an evaluation framework for testing whether a model can provide valid supporting evidence after the underlying rule environment changes while the correct answer remains the same. The framework enumerates minimal proofs and uses them to construct controlled updates. In the adaptive setting, it deletes a rule cited in the model’s previously valid certificate while ensuring that an alternative proof survives. The model is then evaluated on both its answer and the validity of its new certificate.
The experiments evaluate eight models on ProofWriter-derived instances and a RuleTaker-derived benchmark. They find that models often return the correct answer together with an invalid certificate. The paper also compares evaluation with and without explicit update history and examines the types of certificate errors.
Critical strengths
I like the basic evaluation question. If a system provides supporting evidence for its answer, it is reasonable to require that this evidence remain valid when the underlying information changes. Keeping the answer fixed while changing which support is valid gives a clean way to formulate this requirement.
The use of an exact symbolic verifier is a strength. The certificate has a precise meaning, and its validity can be checked directly. I also appreciate that the authors distinguish the submitted certificate from the model’s internal reasoning.
The evaluation covers several model families and a second benchmark source. The paper is generally careful about reporting model-specific eligibility and denominators, and about distinguishing successful replacement from refusal. In particular, it does not simply interpret a decrease in invalid certificates as an improvement when the model instead stops answering.
Critical weaknesses
There is an inconsistency in the certificate analysis that I could not reconcile. Adaptive Repair requires that the deleted rule belong to the model’s valid Original certificate. Table 29 defines “unchanged” as a certificate identical to that Original certificate. Therefore, an unchanged certificate must contain the deleted rule. However, Table 29 reports zero deleted-rule citations for every model, while reporting positive unchanged counts for seven models. For example, Qwen3-8B has six unchanged certificates and zero target citations. These statements cannot both hold under the stated definitions. This may be a mislabeled column or an error in how the records were compared, but it matters: the table is used to support the conclusion that failures arise from incomplete reconstruction rather than reuse of the original certificate. I would like this discrepancy resolved before relying on that interpretation. There is also a discrepancy in the judge results: Appendix A10 reports 90.9% stale recall for the Qwen3-32B certificate-aware judge, whereas Table 19 reports 45.5%. The experimental discussion and tables need to be reconciled.
Questions for authors
-
What exactly does “unchanged” measure in Table 29, and how can its positive counts be reconciled with zero deleted-rule citations? Does correcting this discrepancy change the failure decomposition or any headline results?
-
On the same eligible True instances, how does certificate validity change under an edit that preserves the original certificate, compared with the targeted deletion? This comparison would help clarify the interpretation of the main finding.
4 comments
Part II
Q2: Does proof-targeted disruption explain the observed failures?
We conducted an additional matched preserve-versus-target experiment to distinguish the effect of proof disruption from general sensitivity to KB updates.
Both conditions share the same original KB, query, certificate, edit size, and oracle answer (True→True). Their distinction is whether the selected original proof remains valid.
| Condition | Original proof | Alternative proof | Oracle answer |
|---|---|---|---|
| Preserve | Remains valid | Available | True |
| Target | Invalidated | Available | True |
This design controls for answer changes and edit size while directly varying the impact on the original proof.
| Model | Matched n | Preserve S/St/O | Target S/St/O | ΔStale |
|---|---|---|---|---|
| Phi-4-mini | 45 | 37/2/6 | 24/11/10 | +20.0 pp |
| Llama-3-8B | 51 | 37/7/7 | 30/15/6 | +15.7 pp |
| Qwen3-8B | 83 | 47/13/23 | 40/19/24 | +7.2 pp |
| Qwen3-4B | 93 | 61/15/17 | 54/20/19 | +5.4 pp |
| Qwen3-32B | 64 | 46/5/13 | 29/8/27 | +4.7 pp |
| gpt-oss-20b | 177 | 155/22/0 | 145/28/4 | +3.4 pp |
| Total (6 models) | 513 | 383/64/66 | 322/101/90 | +7.2 pp |
S/St/O denotes Success/Stale/Other; ΔStale is the difference in Stale outcomes divided by matched n.
Across the six reported models, targeted proof disruption consistently increases stale-certificate outcomes. Overall, Stale increases from 64/513 to 101/513 (+7.2 pp).
Since the correct answer remains unchanged and the update size is matched, these results provide direct evidence that disrupting the original proof increases the difficulty of producing valid replacement evidence, independently of answer changes.
Importantly, the full Success/Stale/Other distributions distinguish invalid supporting certificates from other output failures.
We have added this matched-control experiment, its construction protocol, and the corresponding analysis to the revised manuscript. The additional evidence strengthens the central conclusion of ProofCover: preserving a correct answer does not guarantee successful adaptation of its supporting proof after a knowledge update.
All revisions described above have been incorporated into the latest submitted PDF of our manuscript, with the corresponding changes highlighted in red for ease of review.
We sincerely thank the reviewer again for the constructive feedback. We would be happy to provide further clarification and discuss any remaining questions.
Dear Reviewer,
Thank you for your careful review. We have corrected the reported inconsistencies and added matched-control experiments to address your questions. We would appreciate any follow-up feedback on our clarifications.
Thank you for your time!
Dear Reviewer,
Thank you for your careful review. We would like to briefly highlight how we have addressed your concerns.
-
Table 29 and judge-statistics inconsistencies (W1/Q1). We have corrected the certificate-reuse definition (37 partial, zero exact reuse) and the Appendix A10 judge statistics. We have independently verified all 821 evaluations, reproducing the original Success/Stale/Other counts.
-
Does targeted proof disruption cause additional certificate failures? (Q2) We have added preserve-versus-target comparisons across 513 matched cases, showing an increase in stale certificates from 64 to 101 (+7.2 pp).
All corrections and additional experiments have been incorporated into the revised manuscript, with changes highlighted in red.
Thank you for your consideration.
Part I
We thank the reviewer for the careful examination of our results and the suggestions for strengthening the experimental evidence. We independently re-examined the reported statistics and conducted additional matched-control experiments to address the concerns.
Table 29 — Certificate reuse. Thank you for highlighting the ambiguity in the certificate-reuse statistics. The 37 cases were computed using nonempty certificate-set containment (Cnew⊆Cold), rather than exact certificate equality.
Our independent re-examination confirms:
All 37 cases are strict subsets of the original certificates. This distinction helps characterize the failure mode: a model may reuse part of the previous evidence without constructing a complete valid replacement proof. Consequently, invalid certificates need not explicitly cite the deleted rule.
We have clarified the column name as Subset of Original Certificate and explicitly specified its criterion in the revised manuscript.
Reliability of the main results. To determine whether the certificate-reuse distinction affects the central findings, we independently rechecked all 821 eligible model–example evaluations against the stored predictions and certificates.
All per-model Success/Stale/Other counts and stale/accepted rates are reproduced exactly. Thus, the clarified certificate-reuse definition does not change the primary empirical finding that models can produce correct answers with invalid supporting certificates.
We have included the verified statistics and clarified the evaluation definitions in the revised manuscript.
Appendix A10 — Judge statistics. We further recomputed the judge metrics directly from the saved confusion matrices. Table 19 is consistent with the evaluation records; two certificate-judge values in the Appendix A10 prose require alignment with those calculations.
We have updated the Appendix A10 descriptions and the corresponding interpretation of judge performance using the verified values.
These judge-level revisions do not affect the primary benchmark outcomes, which are independently determined through certificate validation.