- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper explores a meaningful direction: training a student LLM on tool-augmented teacher reasoning traces to predict experimental binding outcomes for protein binders. However, the current experiments do not sufficiently establish that the performance gains result from transferring the teacher's reasoning capabilities.
-
the evaluation is limited in scale: the test set covers only four targets, and positive examples are concentrated on PD-L1. Although the authors use a target-disjoint split and examine sequence similarity, they do not strictly separate all homologous clusters or adequately report cross-split similarity among binder sequences. Consequently, the results do not establish robust generalization across protein families. Furthermore, the test set is drawn from examples for which the teacher produced correct predictions, potentially introducing selection bias.
-
the paper lacks non-LLM baselines trained using the same inputs, labels, and data splits. It is therefore unclear whether the complex reasoning-distillation pipeline outperforms conventional supervised learning. If a simple classifier achieves comparable performance, this could raise concerns about possible train–test leakage, and would weaken the justification for the method's complexity.
-
the paper lacks key controls, including label-only supervision, RL directly from the base model, and class-balanced SFT. In particular, the RL stage changes both class sampling and the training objective, making it difficult to disentangle the sources of improvement. The increase in classification performance after RL, accompanied by a decrease in expert-rated explanation quality, further indicates that classification performance alone cannot establish improved scientific reasoning.
-
The positive rate is approximately 10.5% in the SFT training set but increases to 43.4% after resampling for RL, whose reward also incorporates class weights. Consequently, the improvement in recall may partly reflect class rebalancing rather than the training method itself. Class-balanced SFT and comparisons under matched sampling and class-weighting conditions are needed to disentangle these effects.
Critical strengths
- The paper integrates evidence from multiple tools through specialized agents to generate structured teacher reasoning traces. These traces explicitly consider agreement, conflicts, and counterevidence, making their rationale easier to inspect.
Critical weaknesses
-
the paper lacks non-LLM baselines trained using the same inputs, labels, and data splits. It is therefore unclear whether the complex reasoning-distillation pipeline outperforms conventional supervised learning. If a simple classifier achieves comparable performance, this could raise concerns about possible train–test leakage, and would weaken the justification for the method's complexity.
-
the evaluation is limited in scale: the test set covers only four targets, and positive examples are concentrated on PD-L1. Although the authors use a target-disjoint split and examine sequence similarity, they do not strictly separate all homologous clusters or adequately report cross-split similarity among binder sequences.
Questions for authors
The current results do not establish that chain-of-thought supervision improves binder prediction, leaving the added complexity and computational cost of the distillation pipeline insufficiently justified. Moreover, the test set is small and concentrated on a few targets, limiting reliable assessment of the model’s predictive performance and generalization.