acceptodds
Under review as a conference paper at ICLR 2027

SciQuotient: From Exact Failures to Transferable Scientific State Abstractions

Abstract

Scientific agents powered by large language models (LLMs) increasingly solve complex problems through long-horizon cycles of code generation, execution, and revision, accumulating failures that could serve as valuable training experience. Yet repairing a failure does not necessarily yield transferable knowledge, because the model may still misidentify which state distinctions matter. To make this gap measurable, we introduce SciQuotient, an integrated evaluation-and-training framework for learning transferable scientific state abstractions from exact failures. Given a fully specified finite dynamical system and a target observation, an LLM must construct the coarsest exact state abstraction that preserves all future target behavior. Because the correct abstraction can be certified exactly, SciQuotient uses certified quotient-preserving transformations to vary scientifically irrelevant aspects of the state and its representation while leaving the correct abstraction unchanged, exposing hidden abstraction failures that conventional held-out accuracy can miss. Across several model families, training on exact failures in a single representation strongly repairs the targeted behavior and supports partial transfer, but models trained this way remain brittle to mathematically equivalent re-encodings. In contrast, training on equivalent failures across multiple representations substantially broadens transfer to unseen representations and state configurations, with this transfer extending to held-out dynamical mechanisms without additional training. The resulting models can also surpass strong reference LLM baselines on SciQuotient transfer evaluations. Together, these results show that fixing known failures does not guarantee transfer, while representation-diverse exact-failure training helps models carry learned abstractions beyond the conditions in which those failures were observed.

Primary area:
foundation or frontier models, including LLMs
Keywords:
large language models, scientific agents, scientific state abstraction, failure-driven learning, transfer learning, scientific reasoning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_w6tBOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents SciQuotient, a framework for evaluating and training LLMs to learn transferable scientific state abstractions from failures. It uses finite dynamical systems and exact verification to identify incorrect abstractions and generate corrected training examples. The paper also introduces equivalent representations of the same systems to test whether models learn general rules or simply adapt to specific input formats. Experiments show that training on diverse representations improves transfer to re-encoded tasks and unseen dynamical mechanisms.

Critical strengths
  1. The use of exact verification provides reliable labels and avoids relying on LLM-based evaluation.
  2. The experiments cover multiple model sizes and families, with clear improvements under representation changes.
Critical weaknesses
  1. The methodological novelty is somewhat limited. The framework builds on established concepts such as behavioral equivalence, quotient systems, and data augmentation.
  2. Exact verification only guarantees correctness within the constructed finite systems. It is unclear how well these systems capture the original scientific failures.
  3. The experiments focus on simplified, deterministic systems with fully specified dynamics. The results may not generalize to real scientific tasks involving uncertainty and incomplete information.
  4. The representation-diverse training uses encoding types that also appear during evaluation. Generalization to entirely unseen representation types is not demonstrated.
  5. The paper shows improved output accuracy but does not establish whether models learn genuinely representation-invariant abstractions. The connection between these results and long-horizon scientific agents remains unclear.
Questions for authors
  1. What is the main methodological contribution beyond applying existing state abstraction theory and representation-based data augmentation?
  2. How do the authors ensure that the finite systems constructed from scientific failures preserve the important properties of the original tasks?
  3. Can the authors evaluate transfer to representation types that are completely absent from training?
  4. Can the authors provide evidence that these improvements benefit real scientific agents, instead of only the proposed abstraction benchmark?
Reviewer_Q2qPOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors introduce SciQuotient, a method that tests whether AI models can learn the correct scientific state abstractions (meaning whether they can figure out which details matter and which don't) while solving scientific problems. It uses strict mathematical rules to check if the LLMs' or AI's understanding is truly correct or if it's just a lucky guess based on how the problem is written. The core finding is that they have found a flaw in large language models: if you train a large language model to fix mistakes in a specific format, it gets confused when it gets the exactly same problem written in a different format (which should not happen if the model can reason logically); and if you train the AI using the same mistakes written in different mathematically equivalent forms, then it can truly learn the underlying rules and works much better on new and unsolved problems.

Critical strengths

This is a highly rigorous, original and mathematically well-designed work that introduces an interesting way to think about failures in AI models. Experimental depth is very good too.

Critical weaknesses

The critical weakness in my eye is the experiment design and following practical use. It’s dependency on fully specified, finite, and deterministic systems; which is far away from real scientific problems, which are usually continuous, stochastic, and semi-observable. In terms of real world use, the paper claims relevance to long-horizon scientific workflow (hypothesis development to execution to observation to revision); but the actual task here is simply a one-shot construction of a symbolic expression, without tool use, execution, revision or hypothesis development. It contradicts with the framing and practical use of the method.

Also, from evaluation POV, it uses binary metrics, which is not very helpful, specially in tasks which have multiple sub-steps. Also, the paper shows that D outperforms S but it does not say why. The reason behind this phenomenon is like data augmentation, regularization, or invariant risk minimization. Nothing is clear. The paper also cites domain randomization and IRM but does not explore or test these alternative explanations. The findings that these sometimes reduce performance on familiar representations also suggest that there is a coverage optimization trade-off that should be explored too.

Questions for authors
  1. On critical weakness 1, I am interested to know how SciQuotient would need to be extended to handle the different systems, like the stochastic or different continuous systems. What are the main technical problems or hurdles there and would the S versus The distinction still hold in those settings?
  2. Can you please explain or maybe experimentally explore the practical significance of the work, or how the method can be applied to different practical systems, and if it brings any benefit? As I noted in the critical witness, I have concerns that this approach will hold theoretically but at the current stage I see that the practical significance is lower. Can you please explain to me or help me understand it?
  3. Also is it possible to extend the current evaluation into subtasks and assign partial scores to see how the model is progressing? Currently the setup uses a binary task and I think it is hiding some value or some insight that can be revealed with partial scoring based on different subtasks needed to complete the full task.
  4. Also explore why D out of five performs as it does, what the reason behind it is, and any explanation or insight from that perspective. Also can you please check whether this benefit comes from simply seeing more data? Can we do a control that is diverse but matched in terms of total examples to analyze this part?
  5. Can you please develop a human baseline and check how human scientists perform on this task? I think without this it is very hard to interpret if 80% accuracy or 70% accuracy is good, bad, or negligible.
  6. I have a core concern about the overall finding or overall framing of the experimental findings. The paper is basically saying that if we train the model on different types of mathematically equal forms, then it can learn the underlying rules and works much better on nuanced problems. I am concerned about whether it is the result of memorization or not because to me it looks like a memorization problem. If we train it on different things, it memorizes the patterns and this is why it can perform better, but it is not learning anything inherently. If possible can you please verify if it is a memorization thing or if it is actual reasoning?
Reviewer_GLUPOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The presented study asks whether an LLM trained on its own corrected failures learns something reusable or only patches the cases it saw. This is reduced by the authors to a question of state abstraction, i.e. which details of a situation an agent must keep track of and which it can ignore. To investigate this, they build a task on small finite systems where the correct grouping of states can be computed and checked, and ask evaluated models to write a formula that groups states by their future behavior. Because the correct answer is known, task failures and their corrections can be verified by the same procedure. Training data is mined from failures of LLM agents on scientific benchmarks and formalized into such systems. Two training conditions with matched budgets are then compared: One presents each corrected failure in a single notation, the other presents the same failures in two notations. Models are evaluated on seen and held-out systems, on unseen sizes of the irrelevant variable, and in both notations.

The study finds that training in one notation fixes the observed failures, but fails on the identical problems once they are rewritten in the other notation, and it does so by producing well-formed groupings that disagree with its own answers in the original notation. Training in both notations removes this gap, including on held-out systems, while giving no advantage where the notation is familiar. The gap shrinks with model size, and the same pattern roughly repeats across four evaluated model families. Frontier models evaluated without training are all reported to score below fine-tuned models on this task. Based on this, the main conclusion is that fixing failures in one notation does not guarantee transfer to unseen notations, and that presenting the same failures in multiple notations generalizes the learned state abstraction beyond the conditions in which the failures were observed during training.

Critical strengths
  • The task and evaluation are generally well thought-out (for showing transfer across mechanisms, if not representations), since the correct grouping of states can be computed, which makes failures, corrections, and rewrites verifiable. Experiments use sensible controls, fixing dimensions like checkpoints and seeds to isolate effects of interest. Evaluation separates repair from transfer to held-out systems and to unseen sizes of the irrelevant variable, and the comparison is repeated across a representative range of model families and sizes, including unadapted frontier models. The paired consistency check, which asks whether a model's answers in the two notations agree with each other, is a useful diagnostic.

  • The study finds that a model that is well-behaved under one notation, when used with a different notation produces well-formed groupings that are wrong without collapsing into malformed output, in the majority of test cases. This shows that the model has learned a notation-specific rule rather than the abstraction itself, and that held-out accuracy in the training notation cannot detect this. This is an interesting finding that could be expanded upon further to strengthen the overall impact of the paper.

Critical weaknesses
  • A central claim, i.e. that representation-diverse training broadens transfer to unseen representations, is not tested, since the diverse condition is only ever evaluated on the two notations it was trained on. While the paper concedes this in the main text and limitations, the abstract and introduction assert the representation transferability multiple times. What the experiments show instead, is that a model trained on two notations handles new systems in both, while a model trained on one does not. This is a substantial gap and the authors are encouraged to carry out a set of experiments using a third notation held out from both conditions, which would test the claim directly.

  • Findings supported by the conducted experiments have somewhat limited novelty, since 1) sensitivity to equivalent reformulations is already established in the work the paper references, 2) training on several formats as a solution is what the cited domain-randomization work predicts, and 3) on held-out systems in the familiar notation the two compared conditions are indistinguishable (i.e. representation-diverse training is not a prerequisite for the transferability to new systems). The third-notation experiment would decide whether this study extends beyond a replication.

Questions for authors
  • The primary LLM used in the reported experiments is Qwen3-VL-8B-Instruct, a vision-language model, on a task that is entirely textual. Why was this model chosen over text-only Qwen3 checkpoints, and do the authors have any intuition as to whether the multimodal training had any impact on the observed model performance?

  • Four of the nine unadapted frontier models score 0% on every panel, including E1, where a fine-tuned 2B model reaches 100% and Claude Opus 5 reaches 94%. A gap of this size between models may point to a bug in the output parsing logic, rather than a genuine task failure. Did the authors inspect these outputs, and what fraction of them were counted as invalid rather than incorrect (if any)?

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Changes in the Revised PDF

Dear Area Chair and Reviewers,

We sincerely thank all reviewers for their careful evaluations and constructive feedback: Reviewer Q2qP for recognizing our mathematical rigor and experimental depth, Reviewer GLUP for positively assessing our exact verification framework and paired consistency diagnostic, and Reviewer w6tB for acknowledging our reliable evaluation across model scales and families.

We have uploaded a revised PDF incorporating the new experiments, audits, and clarifications, summarized below.

1. Direct Evaluation on a Held-Out Third Encoding

  • CoLex-Fused and shortcut audit: Secs. 3.3, 4.1 (pp. 4 to 5), App. A.5 (p. 16). Introduces nuisance-major CoLex-Fused (strictly excluded from S and D training), proves quotient equivalence, and verifies three shortcuts (including trained fused projection reuse) score 0.00% Exact accuracy on E5 and H5.
  • Unseen-encoding transfer: Sec. 4.6, Table 5 (p. 9). Across seven models on seen (E5) and held-out (H5) mechanisms, D improves Qwen3-VL-8B (+31.25 points on E5, +12.50 on H5), Qwen3-VL-32B (+14.58 on H5), and Gemma-4-12B (+37.50 on E5, +25.00 on H5).
  • Transfer limits: Sec. 4.6, Table 5 (p. 9). OLMo-3-7B and Qwen3-VL-2B score at most 1.04% on E5 and 0.00% on H5 despite strong trained-fused scores, showing direct transfer beyond both training encodings alongside substantial differences across models.

2. Structural Diagnostics and Partial-Credit Evaluation

  • Partial-credit metrics: Sec. 3.2 (p. 4), Sec. 4.3, Table 2 (p. 7), App. B.1, Table 6 (pp. 16 to 17). Extends binary Exact accuracy with Pairwise F1, erroneous merge and unnecessary split rates, and nuisance-orbit invariance.
  • Error analysis: On H2, D lifts Pairwise F1 (0.356 to 0.906) and orbit invariance (17.78% to 97.92%), cuts erroneous merges on H2 (39.58% to 2.78%) and H4 (22.57% to 0.35%), and cuts H2 splits (84.03% to 24.31%).
  • Remaining failures: Sec. 4.3 (p. 7), Sec. 6 (p. 10). Split-only H2 failures under S cause a threefold median state-count inflation, while D mainly retains extra nuisance distinctions rather than merging future-distinct states; all metrics audit observable output partitions rather than internal neural representations.

3. Failure Formalization and Training Controls

  • Failure formalization: Sec. 4.2 (p. 6). Clarifies filtering SciCode, ScienceWorld, ScienceAgentBench, and internal trajectories to isolate state-distinction errors: 100.00% of audited local systems require multi-step refinement beyond immediate observations and 96.96% admit nontrivial quotient compression.
  • Matched budgets: Sec. 4.2 (p. 6), Sec. 5 (p. 10). S and D match record counts, updates, replay schedules, seeds, and parent checkpoints under standard supervised adaptation without an IRM objective, though serialized lengths do not equalize token exposure.
  • Generalization dynamics: Secs. 4.3 to 4.5 (pp. 7 to 8). Explains coordinate-boundary shortcuts under S, separates familiar-encoding competence from unseen-mechanism transfer, and analyzes coverage versus familiar-accuracy trade-offs.

4. Backbone, Parser, and Human Difficulty Audits

  • Backbone and text-only replication: Secs. 4.1, 4.5, Table 4 (pp. 5, 8). Qwen3-VL provides a consistent 2B to 32B ladder on purely textual inputs (no visual tokens, inactive vision encoder), while text-only Llama and OLMo replicate the S versus D gap.
  • Frontier parser audit: Sec. 4.5 (p. 9), App. B.2, Table 7 (pp. 16 to 17). Verifier and independent syntax checks show 76.50% to 99.88% of outputs from four near-zero frontier models form valid but wrong partitions with no valid expressions rejected, ruling out parser errors as the main cause.
  • Human difficulty reference: Sec. 4.5 (p. 9), Ethics Statement (p. 11), App. B.3, Table 8 (p. 17). Three unaided participants average 20.42% Exact accuracy on matched 20-task H1 to H4 subsets versus 82.92% for Qwen3-VL-8B under D.

5. Refined Contributions and Scope

  • Scope and claims: Abstract (p. 1), Sec. 1 (pp. 1 to 3), Sec. 7 (p. 10). Frames SciQuotient as a controlled component-level framework and distinguishes transfer across the two training encodings from generalization to the held-out third encoding.
  • Broader systems: Sec. 6 (p. 10). Discusses stochastic, partially observable, and continuous settings via probabilistic and approximate bisimulation and belief-state abstractions, covering three hurdles (statistical verification, belief-state dimensionality, and long-horizon approximation error) and why S versus D comparisons remain meaningful under equivalence-preserving transformations.

All original E1 to H4 results in Tables 1, 3, and 4 and Figures 1 to 3 remain unchanged, with substantive revisions highlighted in blue.

We sincerely thank the reviewers and Area Chair for their valuable feedback and consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.