acceptodds
Under review as a conference paper at ICLR 2027

Can Post-Training on Agentic Chain-of-Thought Autonomously Screen Protein Binders For Experimental Success?

15 views

Abstract

A well-known bottleneck exists in protein binder design: favorable in silico metrics do not necessarily translate to guaranteed success in the wet-lab. To address this, we introduce BinderReason: we post-train large language models (LLMs) via a two-stage post-training approach on Chain-of-Thought (CoT) reasoning traces generated by our autonomous multi-agent system, BinderLab-Think. We compile a curated collection of 6650 design-target complexes from three open-source datasets, representing 40 targets, of which 18.40% of designs are experimentally validated as good binders. These complexes are co-folded and analysed by BinderLab-Think, our closed-loop system of 11 specialised agents using Plan-and-Solve CoT prompting to predict whether each design binds to its target. We cold-start two open-weight models on the resulting traces to understand the biological, chemical, and physical concepts involved in binder analysis, and to align their internal thinking with a critical, systematic, and motivated approach to verdict generation. We then refine the cold-started Qwen3.5-9B model, BinderReason-SFT, via Group Relative Policy Optimisation (GRPO) with a domain-specific composite reward comprising five components: metric-weighted correctness, reasoning consistency, non-degeneracy, label validity, and format. This reward encourages consistency with structural evidence, suppresses hallucinated metric values and degenerate outputs, and enforces a parsable verdict. The resulting model, BinderReason-RL, achieves 0.8710 balanced accuracy and 0.8517 F1 on a target-disjoint test split, exceeding the strongest of ten zero-shot closed- and open-weight reasoning baselines by 31.2 percentage points in balanced accuracy. Finally, we analyse how post-training affects both classification and generated reasoning. Although performance remains target-dependent, BinderReason represents a step towards interpretable, domain-informed prediction of experimental binder success.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
Agents, Protein Binders, LLMs, Reasoning, CoT, Supervised Fine-Tuning, Reinforcement Learning, GRPO

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_6QzqOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper explores a meaningful direction: training a student LLM on tool-augmented teacher reasoning traces to predict experimental binding outcomes for protein binders. However, the current experiments do not sufficiently establish that the performance gains result from transferring the teacher's reasoning capabilities.

  1. the evaluation is limited in scale: the test set covers only four targets, and positive examples are concentrated on PD-L1. Although the authors use a target-disjoint split and examine sequence similarity, they do not strictly separate all homologous clusters or adequately report cross-split similarity among binder sequences. Consequently, the results do not establish robust generalization across protein families. Furthermore, the test set is drawn from examples for which the teacher produced correct predictions, potentially introducing selection bias.

  2. the paper lacks non-LLM baselines trained using the same inputs, labels, and data splits. It is therefore unclear whether the complex reasoning-distillation pipeline outperforms conventional supervised learning. If a simple classifier achieves comparable performance, this could raise concerns about possible train–test leakage, and would weaken the justification for the method's complexity.

  3. the paper lacks key controls, including label-only supervision, RL directly from the base model, and class-balanced SFT. In particular, the RL stage changes both class sampling and the training objective, making it difficult to disentangle the sources of improvement. The increase in classification performance after RL, accompanied by a decrease in expert-rated explanation quality, further indicates that classification performance alone cannot establish improved scientific reasoning.

  4. The positive rate is approximately 10.5% in the SFT training set but increases to 43.4% after resampling for RL, whose reward also incorporates class weights. Consequently, the improvement in recall may partly reflect class rebalancing rather than the training method itself. Class-balanced SFT and comparisons under matched sampling and class-weighting conditions are needed to disentangle these effects.

Critical strengths
  1. The paper integrates evidence from multiple tools through specialized agents to generate structured teacher reasoning traces. These traces explicitly consider agreement, conflicts, and counterevidence, making their rationale easier to inspect.
Critical weaknesses
  1. the paper lacks non-LLM baselines trained using the same inputs, labels, and data splits. It is therefore unclear whether the complex reasoning-distillation pipeline outperforms conventional supervised learning. If a simple classifier achieves comparable performance, this could raise concerns about possible train–test leakage, and would weaken the justification for the method's complexity.

  2. the evaluation is limited in scale: the test set covers only four targets, and positive examples are concentrated on PD-L1. Although the authors use a target-disjoint split and examine sequence similarity, they do not strictly separate all homologous clusters or adequately report cross-split similarity among binder sequences.

Questions for authors

The current results do not establish that chain-of-thought supervision improves binder prediction, leaving the added complexity and computational cost of the distillation pipeline insufficiently justified. Moreover, the test set is small and concentrated on a few targets, limiting reliable assessment of the model’s predictive performance and generalization.

Reviewer_dgjUOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper targets the gap between favorable in-silico metrics and experimental success in de novo protein binder design. The authors compile over 6k design–target complexes spanning 40 targets from three publicly released, experimentally characterized datasets, and introduce BinderLab-Think, a closed-loop system of eleven specialized agents that co-folds each complex with BoltzGen and analyzes it using a combination of literature retrieval, similarity search, and existing tool use (Rosetta energy calculations) to produce a binary evaluation of design capabilities. In addition to this workflow there is a CoT prompting series that walks the agentic workflow through planning and solving the problem. A staged training procedure is used with SFT followed by GRPO-RL. RL is done using a composite reward framework that captures weighted accuracy, consistency in reasoning, and label format compliance. The evaluation stage of the pipeline uses a target-disjoint test split of four held-out targets. Inference is repeated at three decoding seeds and compared to zero-shot runs of the base models and eight other closed and open-weight reasoning models. The authors also provide a lexical analysis of the generated explanations and a four-axis rubric evaluation provided by two domain experts.

Critical strengths

The dataset assembly and the agentic analysis workflow are a real engineering contribution, and the paper is accurate everywhere I could check it. Combining three experimentally characterized sources, co-folding every pair, and running the same battery of structural, geometric, contact-level, and energetic analyses across all of them is substantial work. The per-agent documentation in Appendix C is detailed enough for another group to rebuild the system. The reported numbers are internally consistent. The dataset counts reconstruct the stated totals and binding rate, the trace counts sum correctly, the per-target accuracies recombine to the aggregate, the quoted margins match the tables, and the rubric composites follow the stated formula. The argument for dropping the standard-deviation normalization in GRPO is also correct, since dividing by the group standard deviation removes the constant factor that the emphasis term is meant to supply. This matters to my recommendation because the problems below are problems of experimental design rather than of execution, and the same group can address them.

The paper reports evidence against its own framing. The expert rubric shows the most accurate model producing the least rigorous explanations, the lexical analysis is described as measuring what a model discusses rather than whether it is correct, and the limitations section names both the target dependence and the retention of correct traces only.

Critical weaknesses
  • One target supplies about half of the held-out rows and is almost entirely positive, with a negative class in the single digits. A rule that looks only at the identity of the target matches or beats the proposed model on every headline metric in the main table. The per-target results are consistent with the model having learned something close to that rule, since it predicts positives almost entirely on the two targets where positives are common and predicts none at all on the other two.

  • Features of the dataset make a shortcut of this kind easy to find. One of the three source datasets contains only binders by construction, so its naming conventions and design campaign are perfectly correlated with a positive label. Binder and target identifiers also appear to stay in the prompt for every condition at both training and test time. The qualitative appendix shows a zero-shot model reaching the right verdict by reading a rank suffix out of an identifier, and the integrity axis in the authors' own rubric treats that behavior as disqualifying.

  • The split is built in a way that makes the aggregate hard to read. Its class balance is inverted relative to training, and the cohort sizes quoted in the appendix suggest the same design–target pairs recur across teacher models. The inference prompt carries only identifiers and sequences, so repeated pairs give identical evaluation items that are being counted as independent observations.

  • Training was a single run at one seed. The reported error bars come from repeating inference at several decoding seeds, so the dominant source of variation is never measured. Either the margin over the zero-shot baselines is real and has not been shown yet, or it sits inside run-to-run variation.

  • There are no ablations, no SFT condition on labels alone, and no equal-compute control. Nothing in the setup separates supervision by agentic reasoning traces from supervision by labels, even though the abstract and the discussion assign separate roles to the traces, the cold start, and the RL stage.

  • The premise in the introduction is never measured either, although it could be. The argument there is that insilico metrics fail to predict experimental success. The authors hold those metrics for every complex, and the aggregated goodness score inside their own reward is a parameter-free classifier that could be thresholded and reported in one line.

  • The traces are described as a systematic expert process worth imitating, but the system that produced them agrees with the experimental label about half the time, which is worse than always answering that a design is a non-binder. Its positive verdicts are correct much less often than that. A trace enters the corpus by agreeing with the label rather than by being sound, so either most retained positive traces are right for reasons the analysis does not support, or the selection step is doing work the paper credits to the reasoning. Neither version supports the framing.

Questions for authors
  1. What are the accuracy, balanced accuracy, and F1 on the held-out split for three predictors that use no language model at all: target identity alone, source dataset alone, and a simple supervised classifier fitted to the nine in-silico metrics on the training targets?
  2. Were binder and target identifiers present in the prompts during training and evaluation? If so, what happens to the main table when they are replaced by neutral tokens, for your model and for at least two zero-shot baselines?
  3. How many distinct design–target pairs are in the 152-row test split, and how many distinct positive pairs? If a pair appears under more than one teacher model its inference prompt is identical, so I would like the main table recomputed with one row per pair.
  4. What is BinderLab-Think's own verdict accuracy on the complexes it analyzed, broken out by class, relative to always predicting a non-binder? Given that number, what is the argument that traces retained by label agreement encode a transferable process?
  5. What fraction of RL completions contain at least one metric value that the consistency extractor parses successfully, at the start and at the end of training? Which loss-masking setting produced the reported SFT checkpoints?
Reviewer_xp4xOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

The paper proposes a new method for protein binder screening. The paper have a teacher system, Binderlab-Think, which utilize a set of 11 agents to generate CoT reasoning traces to predict protein binder success. The paper then trains a language model on a filtered subset of the traces and optimize the predictions with additional rewards via GRPO. The paper show that the BinderReason-RL model achieves good performance on their own held out test set, leading open source models and frontier closed models by a large margin.

Critical strengths

Therapeutic design is an important area of study, the paper is trying to solve an important problem in the space.

Critical weaknesses
  1. There are clear notes that are not cleared out such as "The traces are generated by name the teacher model"/"Report corpus size after filtering and the number of rows dropped." in Appendix E.3
  2. The only baseline in the paper is asking open/closed LLMs if A binds with B. The paper never benchmark Binderlab-Think or BinderReason against any domain-specific computational biology baseline, even though a large fraction of their data source is from another paper that evaluates in silico folding/energy metrics predicts wetlab binding. (Overath 2025) Comparisons against structural scores and supervised classifiers over the computed metrics, or protein sequence based classifiers would help establish if the agentic analysis adds any predictive value. Without these comparisons, it is hard to see the practical advantage of either the Binderlab-Think or BinderReason pipeline.
  3. To generate training and test data for SFT/RL, the authors sampled traces from Binderlab-think and discarded all traces where the prediction does not match the ground truth wetlab outcome. I understand that we want to fine-tune the models on valid traces, but this does not justify building the test set only from examples where the teacher agent is correct. A more appropriate way would be to evaluate on a test set that is not filtered with teacher correctness, and only apply the filtering to the train set. It would also be informative to report performance separately on teacher correct/incorrect cases to see what the student learns from the teacher.
  4. The binder id in the user prompt might be leaking extra information to the model. For example Figure 24, p55, the reasoning traces infers "for a dataset intended to test binder quality identification, a named "rank01" item is statistically overwhelmingly likely to be the positive sample ("good binder")." This looks to me like the model is guessing good binder because of the naming/id instead of sequence chemistry.
Questions for authors

See weaknesses

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.