acceptodds
Withdrawn from ICLR 2027

Does Reasoning Improve LLMs Beyond Mathematics? A Biological Test on Intrinsically Disordered Protein Regions

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Reasoning improves language models in domains such as mathematics, where premises and answers are well defined. Whether these benefits extend to biology, where evidence is incomplete and heterogeneous, remains unclear. We study this question through prediction of intrinsically disordered protein regions (IDRs). GPT-5.4 generated evidence-grounded teacher traces that were transferred to Qwen3-4B through supervised fine-tuning (SFT) and Group Relative Policy Optimization (GRPO). We combined the resulting models with a frozen ESM-2 residue classifier and calibrated interval decoder. Zero-shot Qwen produced valid structured outputs but no usable IDR intervals, showing that formatting alone does not confer biological prediction ability. Both hybrid models performed strongly on teacher-corpus validation and retained substantial performance on an independently selected, sequence-disjoint test set, numerically exceeding two leading CAID2 Disorder-NOX predictors evaluated with the same labels. GRPO improved interval localization, trace–answer consistency, and reasoning coherence, but did not consistently improve residue classification over SFT. Perturbation analysis identified candidate-region evidence as the dominant predictive signal, while claim-level auditing showed that coherent traces could still contain unsupported statements. Overall, our reasoning-based framework improves selected biological prediction objectives under incomplete evidence, but its benefits are objective-dependent; the strongest results arise from combining language-model reasoning with specialized protein representations and explicit grounding checks.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
Biological reasoning, intrinsically disordered regions, supervised fine-tuning, reinforcement learning, large language models

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_MscyOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies whether reasoning-oriented training helps predict intrinsically disordered protein regions. Qwen3-4B is trained on teacher explanations written with access to reference intervals, then optimized with GRPO using answer, format, and consistency rewards. Its generated intervals are combined with a classifier trained on frozen ESM-2 residue embeddings and processed by a calibrated decoder. Evaluation uses a locked validation cohort from the teacher corpus and an independent test cohort of 277 proteins. The hybrid predictors perform well, and GRPO improves some interval-level metrics without consistently improving residue-level classification. An automated audit finds better trace–answer consistency alongside persistent grounding errors.

Critical strengths

GRPO improves interval overlap on the independent cohort while slightly reducing residue-level AUPRC and F1. The trace audit likewise shows that improved internal consistency can coexist with grounding errors. These results reveal a useful trade-off between interval localization, residue prediction, and explanation quality. Keeping calibration and model selection separate from the independent test strengthens this part of the evaluation.

Critical weaknesses
  1. In 184 of the 277 independent-test SFT cases, the system falls back to scores from the auxiliary ESM-2 classifier. Yet the paper does not report that classifier alone with the same decoder. This is a particularly important baseline: without it, it is hard to say how much the language-model branch contributes to the final prediction. A matched answer-only training control is also missing, so any contribution from that branch still could not be attributed specifically to reasoning supervision. The current results do not answer the central question posed by the paper.
  2. The trace audit concerns the raw generated answer, but fusion, decoding, or fallback can change the intervals that the system ultimately returns. It is unclear whether the explanation remains valid after those changes. Moreover, the automated audit finds substantive or critical grounding issues in 25.6% of the GRPO samples in the validation cohort. Better consistency between a trace and its raw answer therefore does not yet establish a dependable explanation of the final prediction.
Questions for authors
  1. What performance does the auxiliary head achieve on the locked test set with the same decoder and no language-model contribution? How does the system compare with training on the same inputs and answers without reasoning-trace supervision?
  2. How are the 184 SFT fallback cases among the 277 independent-test proteins divided between malformed output, missing intervals, and accession mismatches? What are the corresponding counts for GRPO?
  3. Section 3.4 states that fusion leaves the reasoning text unchanged. How often do the retained explanations agree with the final intervals after fusion or fallback? Can you report this separately for unchanged predictions, changed predictions, and auxiliary-only fallback cases?
  4. How many reference intervals in the independent test set have experimental support, and how many come from computational annotations? Among residues labeled negative, which are known to be ordered and which simply lack a disorder annotation? How sensitive are the results to excluding uncertain labels?
  5. Mean best IoU increases from 0.5224 to 0.5495 after GRPO. Is this improvement spread across many proteins or driven by a few cases? What does it correspond to in terms of boundary errors, and what uncertainty is there around the SFT–GRPO differences on the independent test set?
  6. Table A.1 uses the calibrated decoder for the hybrid models and the returned binary labels for flDPnn2 and DisPredict3.0. How do the residue-F1 differences change when all methods are calibrated on the same training-side calibration set?
Reviewer_6cG8OpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies whether explicit reasoning improves prediction of intrinsically disordered protein regions. GPT-5.4 generates explanations conditioned on supplied reference intervals and biological evidence. These explanations supervise Qwen models, followed by GRPO training. The resulting interval predictions are combined with an ESM-2 auxiliary residue classifier. Evaluation covers two 277-protein cohorts, comparisons with flDPnn2 and DisPredict3.0, component perturbations, and explanation-quality analyses. The authors report strong hybrid prediction performance and improved interval localization after GRPO, but not consistent gains in residue classification or fully supported explanations. The biological prediction experiments are limited to disorder prediction.

Critical strengths

The comparison with flDPnn2 and DisPredict3.0 on a separately selected test cohort provides some evidence that the combined prediction pipeline is useful.

Critical weaknesses

1.The experiments do not establish the benefit of reasoning supervision, which is the paper’s central claim. The framework combines task-specific training, ESM-2 inputs, an auxiliary residue classifier, and interval decoding, but the auxiliary classifier is not evaluated separately and there is no matched answer-only training comparison. This matters particularly because SFT produces usable intervals for only 93 of the 277 independent-test proteins; the remaining predictions retain auxiliary-only scores (Section 4.1). The zero-shot Qwen comparison does not isolate reasoning either, since it lacks the training and auxiliary components used by the full system. The reported gains therefore cannot be confidently attributed to reasoning supervision.

2.The methodological advance appears limited. The paper combines established training components and evaluates only disorder prediction in human proteins containing annotated disordered regions. Without a clearly demonstrated benefit from reasoning, the work remains a narrowly scoped application rather than convincing evidence for its broader framing.

3.The teacher explanations are not sufficiently validated as training targets. GPT-5.4 receives the reference intervals before producing its explanations, so agreement with those intervals says little about whether the explanations are biologically justified. Section 3.2 mainly reports structural completeness checks on the teacher corpus. The more detailed GPT-6 Astra audit concerns student outputs and is itself not validated against blinded expert judgments; it does not resolve the reliability of the explanations used for supervision.

4.Some evaluation results are difficult to reconcile. Table B.4 reports 63.18% valid JSON for SFT, whereas Table B.5 reports 100%. These values may reflect different generation or evaluation settings, but the paper needs to explain the difference before they can be interpreted together.

Questions for authors

1.What are the independent-test results of the ESM-2 auxiliary classifier alone, using exactly the same decoder as the hybrid models?

2.How does the proposed system compare with Qwen trained to predict only the answer, using the same evidence, ESM-2 inputs, data split, and training budget?

3.How well do the precomputed candidate zones perform before Qwen refinement, using the same interval-normalization procedure?

4.What checks of factual support were applied to the teacher explanations before training, beyond structural completeness? Please report the review coverage and any correction or rejection rates, if available.

5.Do Tables B.4 and B.5 use the same SFT-generated outputs and the same definition of valid JSON? If not, please specify the differences in generation settings, parsing rules, and evaluation samples that account for the reported rates of 63.18% and 100%.

Reviewer_ekJjOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper asks whether explicit reasoning helps LLMs in a domain where evidence is incomplete, using intrinsically disordered region (IDR) prediction as the testbed. GPT-5.4 generated evidence-grounded teacher traces, which were distilled into Qwen3-4B by SFT and then refined by GRPO. The generative model proposes IDR intervals, which are fused with per-residue probabilities from a frozen ESM-2 classifier and passed through a calibrated decoder. Evaluation covers a teacher-corpus validation set and an independently selected 277-protein test set, against zero-shot Qwen, flDPnn2 and DisPredict3.0, plus perturbation analysis, ROSCOE coherence scoring, and a GPT-6 claim-level grounding audit.

Critical strengths

The question is worth asking and the testbed is well chosen. Whether reasoning gains transfer from domains with verifiable premises to domains with incomplete, heterogeneous evidence is a real open question, and IDR prediction is a reasonable place to test it.

Critical weaknesses
  1. The method is a standard composition applied to one task, and the paper's own related work shows how small the increment is. Every component is off the shelf: frozen ESM-2 with pooled candidate-region embeddings linearly projected into the Qwen embedding space as soft tokens; LoRA SFT on distilled traces; GRPO; a 328→256→1 MLP probe on frozen embeddings; a hand-set ±0.5 logit fusion; and a threshold/smoothing/min-length/merge-gap decoder. The novelty claim is correspondingly compositional ("the first study to combine human-readable, step-by-step LLM reasoning with residue- and interval-level IDR prediction"), and no new biological knowledge is produced: the teacher explains existing DisProt/MobiDB annotations and the model is trained to reproduce known labels, on one task and one organism.

  2. Sequence-disjoint" means exact-match disjoint only. The independent cohort excluded "teacher-corpus accessions and identical sequences", but no identity-threshold clustering is reported anywhere. Homolog, redundancy, CD-HIT and identity threshold do not appear in the paper.

  3. The effectiveness of the reasoning trace is never established. The one result linking trace quality to prediction quality (mean best IoU 0.7340 / 0.5143 / 0.4029 by grounding label) is explicitly declared non-causal by the paper. And the trace-quality gains are themselves equivocal: the headline DisProt improvement (Jaccard 0.295 → 0.453) rests on n = 20 with the two systems scored on different subsets, reference coverage falls for GRPO on two of three sources (MobiDB-lite 0.516 → 0.493, low-pLDDT 0.738 → 0.701), and unsupported biological rationales rise from 6.41% to 9.01% of claims. Since the reward includes a 0.1-weighted trace–answer consistency term, the coherence gains may simply be the reward shaping the text.

  4. The main results table is in the appendix, and §4.1–4.3 report everything in prose. The only table in the main text is Table 1 (perturbations); the model comparison that the paper's central claims rest on is Table A.1. Table B.1 uses pooled residue F1 while Table B.4 uses macro-averaged residue F1 (noted only in a footnote, and the two are not comparable), and raw-generative and fused numbers are reported in adjacent passages without being labelled as such. Please move Table A.1 into §4, add a single table contrasting raw-generative against fused for SFT and GRPO, and label the metric variant in every table header.

Questions for authors

Please refer to the weaknesses.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.