acceptodds
Under review as a conference paper at ICLR 2027

Does Learning Protein Folding Generalize to Broader Reasoning?

Abstract

Large language models rely heavily on human text, which often conveys surface answers rather than the spatial and structural logic behind them. Protein folding is a natural testbed, because one solved structure yields thousands of exactly checkable spatial and topological statements. We ask: can learning to fold proteins teach general models reusable reasoning capabilities? To answer this, we build FoldingCorpus, a protein-derived question-answer dataset, and Fold2Reason, a recipe that post-trains on it through two complementary signals: discrete structural answers predicted via the model's native language head, and continuous 3D geometry decoded from the same shared representations. On FoldBench, Fold2Reason achieves structure prediction scores 2.7 to 3.5 times those of Qwen3.5-9B. Beyond protein structure prediction, it improves performance on all 10 benchmarks spanning spatial, graph, scientific, and general reasoning, raising macro-average accuracy from 45.09% to 48.33% (+3.23 pp), with positive gains on all 10 benchmarks, while matched controls built from random, synthetic, and shuffled structure yield substantially smaller or negative gains. Our work shows that non-linguistic, structure-dense scientific data can systematically improve broad reasoning in language models, making a solved scientific problem a practical source of post-training supervision.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large language models, post-training, cross-domain transfer, reasoning, protein-derived supervision, structural representation learning

Reviews

Among:
Shape:
1234this paper, Bottom 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.25 over 4 reviews, above 49% of the others.
Reviewer_rvutOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates whether learning from protein-folding-derived supervision can improve the broader reasoning capabilities of language models. The authors introduce FoldingCorpus, a collection of structural question-answer pairs derived from protein data, and propose Fold2Reason, which combines language-model adaptation with auxiliary geometric supervision. Experiments across spatial, graph, scientific, and general reasoning benchmarks suggest that this training yields transferable performance gains beyond the protein domain.

Critical strengths

1. The paper presents an interesting empirical finding. Post-training on protein-structure-derived supervision improves performance across several reasoning benchmarks beyond the protein domain.

2. The empirical investigation is broad and transparent. The paper evaluates multiple model scales and families, several data scales, component ablations, control tasks, and multiple random seeds. It also analyzes output validity, task-level gain concentration, and structural decodability, while openly discussing several limitations.

Critical weaknesses

1. The current experiments do not establish that the protein-folding origin of the supervision is responsible for the observed transfer. FoldingCorpus provides meaningful, input-dependent structured supervision, but a similarly constructed task such as synthetic graphs or 3D point clouds may also produce comparable gains. The existing controls are not matched alternatives: shuffled labels remove the input-label relationship, format copying is substantially easier, and hidden geometry is input-independent. This issue is consequential because it directly weakens the paper’s central protein-folding-specific claim.

2. The evidence for the proposed geometric mechanism is limited. Removing the geometry objective preserves most of the reported improvement, while the incremental Full-minus-no-Geometry gain is small and its confidence interval spans zero. Moreover, the task-level decomposition shows that a substantial fraction of the gains on FTB-Core and SpatialViz comes from a small number of tasks where the base model exhibits strong answer-selection biases. Although the paper’s validity audit suggests that parsing failures alone do not explain the improvements, it remains difficult to separate improved geometric reasoning from improved answer calibration.

Questions for authors
  1. Could the authors add a matched non-protein control task, such as synthetic graph reasoning or 3D point-cloud reasoning, with comparable data scale, input complexity, and supervision format? Without such evidence, the observed improvements may be attributed to structured QA post-training rather than specifically to learning protein folding.
  2. The authors should rerun the main multiple-choice evaluations using option-order randomization, candidate log-likelihood ranking, or constrained decoding to determine whether the reported gains remain after controlling for answer-selection bias.
  3. Figure 3 shows that the gains diminish as model size increases within the Qwen3.5 family, while Gemma-4-12B shows almost no improvement. Could the authors evaluate a larger model from the same family to determine whether this diminishing trend reflects a scaling limitation rather than architecture dependence? If the transfer effect becomes negligible for larger and more capable models, the practical significance and generalizability of the proposed approach would be substantially limited.
Reviewer_7GTiOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper explores the idea of using information about protein structures to improve the reasoning abilities of large language models (LLMs). Specifically, the authors create a question-answer dataset called FoldingCorpus based on the atom coordinates of protein structures and use it to train a new model, Fold2Reason. This model consists of a pre-trained LLM and a trainable LoRA component and is trained using a combination of a loss based on the FoldingCorpus dataset and a loss based on geometric information from the protein structures. The resulting model shows improved performance on several reasoning tasks, in most of the versions tested. In particular, model versions based on the Qwen3.5 LLM show the largest improvement, while the one based on Gemma-4 is very close to the pretrained baseline. Ablations attribute most of the improvement to the FoldingCorpus loss, while the geometry loss is more relevant for protein structure decodability.

Critical strengths

This is a promising idea, where an existing dataset from a scientific domain is used to automatically generate another dataset more suitable for LLM training to improve its general reasoning abilities. In this paper, spatial knowledge encoded in protein structures (individual atom coordinates) is used, but I can see how the same idea could be used to other types of data, for example imaging data.

Critical weaknesses

I found the description of the method in Section 3 and the appendices quite hard to follow. I hope the authors can spend some time to rewrite these parts, especially Section 3 in the main text, so the paper is more understandable for someone not using an LLM to digest it.

Questions for authors
  • Regarding the performance drop when using Gemma-4: Figure 3a shows what looks like a decrease in improvement as the size of the model goes up. Is this what is happening, or is there something specific about the architecture and model itself that cause pretty much no improvement on Gemma-4? This could be checked if the training and experiments were rerun on another model of the same size as Gemma-4. Is there any relationship between how each of these models has been pretrained and the improvements we see here?
Reviewer_Ty8XOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies whether protein-structure training can improve general reasoning. The authors introduce FoldingCorpus, a set of verifiable questions derived from protein structures, and Fold2Reason, which combines question-answer and geometric supervision. Fold2Reason improves the General-10 average from 45.09% to 48.33%. Ablations suggest that question-answer supervision drives most of the general transfer, while geometric supervision mainly benefits 3D-related tasks and local structural representations.

Critical strengths
  1. The paper presents an interesting way to use scientific structures as verifiable training data for general-purpose models.

  2. The controls and ablations are thoughtful. Removing the protein-specific modules at evaluation also provides stronger evidence that the improvement is stored in the adapted language model.

Critical weaknesses
  1. It is still unclear whether the model learns a reusable reasoning ability or mainly benefits from structured QA training and regularization.

  2. The evidence is based on only three seeds, and the benefit of geometry supervision is relatively small and unstable. The scaling experiment also changes both data size and training steps.

Questions for authors
  1. Why does the hidden-geometry control improve General-10 despite using mismatched structures?

  2. Can the authors provide a compute-matched scaling experiment with a fixed number of training steps?

  3. How does FoldingCorpus compare with an equally sized, non-protein dataset containing verifiable structural questions?

Reviewer_uX6SOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies whether post-training on a protein structure-related corpus can improve the broader reasoning capabilities of general-purpose language models. The authors introduced FoldingCorpus, a dataset of 14,400 structural questions generated from 1,200 known protein structures, and Fold2Reason, a post-training method combining question-answer supervision with an auxiliary geometry objective. The method adapts a language model through LoRA and a shared training-time module, while a frozen decoder provides structural supervision. Downstream evaluation retains only the adapted language model, without protein-specific modules or inputs. On ten benchmarks spanning spatial, graph, scientific, and mixed reasoning, the authors report an approximately 3.23 pp average improvement over the base Qwen3.5-9B. Ablations attribute most of this gain to the question-answer supervision, with geometry supervision contributing smaller, primarily spatial benefits. Additional controls, scaling experiments, and evaluations across model families explore the robustness and scope of the reported transfer.

Critical strengths

This is an interesting idea with concrete implementation. The dataset and empirical findings provide a useful contribution, despite the limitations in attribution and interpretation discussed below. Interesting supervision source and reusable implementation. The paper explores protein structures as supervision for broader LLM reasoning. Automatically computable and verifiable labels reduce annotation costs and make corpus expansion feasible. FoldingCorpus and Fold2Reason provide a concrete implementation potentially applicable to other domains with computable targets, while downstream inference uses only the adapted language model. Together, these make the proposed direction practically useful. Broad empirical evidence for behavioral transfer. The experiments cover multiple reasoning benchmarks, three training seeds, model sizes and families, component ablations, and data scales. Controls matched on key training settings, fixed training endpoints, task-level breakdowns, and output-validity analyses help assess the gains and their limitations. This breadth supports the value of the empirical finding even though the underlying mechanism remains unresolved.

Critical weaknesses

Unclear attribution and limited added benefit of the specialized method. The controls examine copying, mismatched labels, and input-independent geometric targets, but do not compare against matched, learnable non-protein QA supervision. They therefore leave the specific benefit of protein-derived structural content unresolved. Moreover, FoldingCorpus-only Pure LoRA nearly matches full Fold2Reason on General-10 (+3.22 versus +3.23 pp). The +0.30 pp geometry increment is relative to a QA-only configuration retaining the workspace, and its confidence interval spans zero (Appendices G and I). These findings limit the demonstrated contribution beyond simpler QA fine-tuning. The reasoning and folding interpretations require qualification. The canonical Qwen evaluation disables thinking, and Appendix H.1 documents answer-choice biases that complicate attribution of gains to newly acquired reasoning rather than changed response behavior or better use of existing capabilities. The headline folding comparison also changes the output interface: Fold2Reason uses a dedicated decoder, while general-purpose baselines generate coordinates as text. The frozen-base decoder control in Appendix K, Table 17 exceeds Full on TM-score (0.1742 versus 0.1688) and has lower distance MAE (10.753 versus 12.244 Å), although Full improves local metrics. These issues narrow the conclusions supported by the results.

Questions for authors

Specificity of the supervision. Could the gains come from QA fine-tuning generally, rather than protein structural data specifically? If feasible, please compare against one learnable non-protein QA dataset, matching the base model, supervised-token budget, and optimization steps. The current copying and mismatched-label controls do not resolve this distinction. Benefit of the full architecture. Given the nearly identical General-10 gains of Pure LoRA and full Fold2Reason (+3.22 versus +3.23 pp), which benefits justify the additional workspace and geometry objective? Using the existing benchmark-level and seed-level results, could you identify any consistent advantages over Pure LoRA specifically, and clarify their importance to the method's contribution? Relationship to explicit reasoning. How do the authors interpret the answer-only improvements in relation to explicit, multi-step reasoning? Given that the canonical evaluation disables thinking and the training supervises answer tokens rather than reasoning traces, would the authors expect the gains to persist when explicit reasoning is enabled, and why? Please clarify the scope of the current evidence; a conceptual discussion would be sufficient here. Interpretation of the folding results. How should readers reconcile the headline folding improvements with the frozen-base decoder results in Appendix K, Table 17, where the frozen-base control performs better on TM-score and distance MAE? Could you clarify which improvements are attributable to adaptation and which may arise from the different output interfaces, and make this distinction explicit in the main-text comparison?

Comments on the paper

1 comment

Dear Reviewers,

As the discussion period draws to a close, please read and acknowledge the authors’ rebuttal within the next 24 hours. I also encourage you to engage in the discussion and share any remaining questions or concerns.

Thank you for your time and thoughtful reviews.

Best,

Your AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.