- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The authors propose TRELLIS, an agent-based framework for automatically reviewing biological sequence-based property predictors. The framework uses LLM coding agents to generate and execute model training code, while MCTS explores different combinations of sequence representations and predictor architectures. The authors evaluate TRELLIS on FLIP protein fitness benchmarks, PeptiVerse peptide property prediction tasks, and MHC class I pathway prediction tasks. The single-agent framework achieves scores above the reported literature references on several tasks, while MCTS improves performance over the single-agent approach on 7/8 selected tasks.
Critical strengths
The authors evaluate the framework on a relatively broad range of protein and peptide property prediction tasks, demonstrating its applicability across different datasets and prediction objectives, with improved predictive performance on several tasks.
Critical weaknesses
-
Limited methodological novelty. I find this paper to be mainly a combination of existing tree search methods, coding agents, and protein foundation models. The resulting predictors mostly rely on existing representations combined with conventional prediction heads. Therefore, I am not fully convinced of the methodological novelty of this work.
-
Unclear motivation and framework design. I am not entirely convinced by the use of the term "agent-orchestrated" in the title, since a substantial part of the framework relies on MCTS rather than being directly orchestrated by LLMs. I am not entirely convinced of the necessity of such a relatively complex framework. Since most resulting predictors rely on pretrained representations combined with conventional prediction heads, it would be useful to examine whether simpler model selection and hyperparameter tuning strategies could provide comparable improvements. Moreover, the reported performance gains are relatively modest, and it remains unclear how much of the predictive performance comes from the pretrained representations rather than the proposed model development framework. More comparisons with simpler approaches would help justify the additional complexity.
-
Unclear baseline comparisons. The specific baseline model names corresponding to the "Paper" scores in Table 1 are not clearly provided, making it difficult to assess how competitive the proposed framework is against existing methods.
-
Writing and presentation. I found some parts of the paper difficult to follow, particularly regarding the nature and positioning of the proposed work. For example, the Related Work section covers several broad research directions but could be more focused on distinguishing this framework from closely related approaches.
Questions for authors
-
Why is MCTS necessary for this framework? Why not allow the LLM orchestrator to directly decide the next predictor configurations based on previous experimental results? Have the authors compared MCTS with simpler search strategies under the same computational budget?
-
What are the specific baseline models corresponding to the "Paper" scores in Table 1? Are these results directly comparable under the same data splits and evaluation protocols?
-
What is the rationale behind the selection of Qwen3.8-27B and GLM-5.3? How would the framework perform with other widely used LLMs, such as GPT, Gemini, or Claude? Would stronger coding models reduce the benefit of the proposed MCTS framework?
2 comments
Comment
Thank you to the authors for their responses and additional experiments. I have carefully reviewed the rebuttal and the comments from other reviewers. However, my main concerns regarding the limited methodological novelty and the necessity of the proposed MCTS-based framework remain unresolved. Therefore, I have decided to maintain my original rating.
We thank the reviewer for these constructive comments. Several points overlap with Reviewer 1 so below contains some similarities in our response.
Motivation and methodological novelty.
We agree that trellis builds on existing components. Our contribution is primarily an application framework for biological sequence property prediction. Biological sequence properties often require different featurizations and more frequently need to be retrained as new assay data comes in or include a new protein representation model. We argue that as LLM agents are increasingly used in protein design, a more flexible framework where the agent is not given a rigid set of features to explore to automatically pick the best feature and instead given just access to a code execution sandbox is beneficial. Our primary motivation behind tree search in our work, is to force such exploration of different representations early on. Once an architecture and representation are chosen, hyperparameter tuning can, in principle be performed even deterministically.
Baseline models corresponding to the "Paper" scores
The “Paper” column reports the best published scores from the source benchmarks used in Table 1. primarily FLIP2 and PeptiVerse and MUNIS for elution. Those papers contain human-designed ml models with different protein language models or classifical sequence features with conventional heads, rather than a single shared architecture. We evaluate trellis on the same train/test splits and report same metrics. In the revision, we can clarify the name of the specific reference model next to each Paper entry
Model Selection Choice
We selected Qwen3.8-27B and GLM-5.3 as moderately sized, reproducible open models suited to a setting where predictors may need repeated retraining as new assay data arrive, under practical compute budgets. The framework is llm model-agnostic: the framework can use other LLMs easily. We are providing comparison with Claude and also BioAutoML approach below.
Traces for all these run are available as html viewer at https://anonymous.4open.science/r/trellis-B12C/reviewer2_responses/index.html