- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper studies whether reasoning-oriented training helps predict intrinsically disordered protein regions. Qwen3-4B is trained on teacher explanations written with access to reference intervals, then optimized with GRPO using answer, format, and consistency rewards. Its generated intervals are combined with a classifier trained on frozen ESM-2 residue embeddings and processed by a calibrated decoder. Evaluation uses a locked validation cohort from the teacher corpus and an independent test cohort of 277 proteins. The hybrid predictors perform well, and GRPO improves some interval-level metrics without consistently improving residue-level classification. An automated audit finds better trace–answer consistency alongside persistent grounding errors.
Critical strengths
GRPO improves interval overlap on the independent cohort while slightly reducing residue-level AUPRC and F1. The trace audit likewise shows that improved internal consistency can coexist with grounding errors. These results reveal a useful trade-off between interval localization, residue prediction, and explanation quality. Keeping calibration and model selection separate from the independent test strengthens this part of the evaluation.
Critical weaknesses
- In 184 of the 277 independent-test SFT cases, the system falls back to scores from the auxiliary ESM-2 classifier. Yet the paper does not report that classifier alone with the same decoder. This is a particularly important baseline: without it, it is hard to say how much the language-model branch contributes to the final prediction. A matched answer-only training control is also missing, so any contribution from that branch still could not be attributed specifically to reasoning supervision. The current results do not answer the central question posed by the paper.
- The trace audit concerns the raw generated answer, but fusion, decoding, or fallback can change the intervals that the system ultimately returns. It is unclear whether the explanation remains valid after those changes. Moreover, the automated audit finds substantive or critical grounding issues in 25.6% of the GRPO samples in the validation cohort. Better consistency between a trace and its raw answer therefore does not yet establish a dependable explanation of the final prediction.
Questions for authors
- What performance does the auxiliary head achieve on the locked test set with the same decoder and no language-model contribution? How does the system compare with training on the same inputs and answers without reasoning-trace supervision?
- How are the 184 SFT fallback cases among the 277 independent-test proteins divided between malformed output, missing intervals, and accession mismatches? What are the corresponding counts for GRPO?
- Section 3.4 states that fusion leaves the reasoning text unchanged. How often do the retained explanations agree with the final intervals after fusion or fallback? Can you report this separately for unchanged predictions, changed predictions, and auxiliary-only fallback cases?
- How many reference intervals in the independent test set have experimental support, and how many come from computational annotations? Among residues labeled negative, which are known to be ordered and which simply lack a disorder annotation? How sensitive are the results to excluding uncertain labels?
- Mean best IoU increases from 0.5224 to 0.5495 after GRPO. Is this improvement spread across many proteins or driven by a few cases? What does it correspond to in terms of boundary errors, and what uncertainty is there around the SFT–GRPO differences on the independent test set?
- Table A.1 uses the calibrated decoder for the hybrid models and the returned binary labels for flDPnn2 and DisPredict3.0. How do the residue-F1 differences change when all methods are calibrated on the same training-side calibration set?