- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper presents SciQuotient, a framework for evaluating and training LLMs to learn transferable scientific state abstractions from failures. It uses finite dynamical systems and exact verification to identify incorrect abstractions and generate corrected training examples. The paper also introduces equivalent representations of the same systems to test whether models learn general rules or simply adapt to specific input formats. Experiments show that training on diverse representations improves transfer to re-encoded tasks and unseen dynamical mechanisms.
Critical strengths
- The use of exact verification provides reliable labels and avoids relying on LLM-based evaluation.
- The experiments cover multiple model sizes and families, with clear improvements under representation changes.
Critical weaknesses
- The methodological novelty is somewhat limited. The framework builds on established concepts such as behavioral equivalence, quotient systems, and data augmentation.
- Exact verification only guarantees correctness within the constructed finite systems. It is unclear how well these systems capture the original scientific failures.
- The experiments focus on simplified, deterministic systems with fully specified dynamics. The results may not generalize to real scientific tasks involving uncertainty and incomplete information.
- The representation-diverse training uses encoding types that also appear during evaluation. Generalization to entirely unseen representation types is not demonstrated.
- The paper shows improved output accuracy but does not establish whether models learn genuinely representation-invariant abstractions. The connection between these results and long-horizon scientific agents remains unclear.
Questions for authors
- What is the main methodological contribution beyond applying existing state abstraction theory and representation-based data augmentation?
- How do the authors ensure that the finite systems constructed from scientific failures preserve the important properties of the original tasks?
- Can the authors evaluate transfer to representation types that are completely absent from training?
- Can the authors provide evidence that these improvements benefit real scientific agents, instead of only the proposed abstraction benchmark?
Summary of Changes in the Revised PDF
Dear Area Chair and Reviewers,
We sincerely thank all reviewers for their careful evaluations and constructive feedback: Reviewer Q2qP for recognizing our mathematical rigor and experimental depth, Reviewer GLUP for positively assessing our exact verification framework and paired consistency diagnostic, and Reviewer w6tB for acknowledging our reliable evaluation across model scales and families.
We have uploaded a revised PDF incorporating the new experiments, audits, and clarifications, summarized below.
1. Direct Evaluation on a Held-Out Third Encoding
2. Structural Diagnostics and Partial-Credit Evaluation
3. Failure Formalization and Training Controls
4. Backbone, Parser, and Human Difficulty Audits
5. Refined Contributions and Scope
All original E1 to H4 results in Tables 1, 3, and 4 and Figures 1 to 3 remain unchanged, with substantive revisions highlighted in blue.
We sincerely thank the reviewers and Area Chair for their valuable feedback and consideration.