- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper demonstrates a useful agent-based workflow for improving AF3 predictions through adaptive MSA organization. However, the current evaluation establishes the effectiveness of the overall pipeline more clearly than the specific contribution of LLM-driven reasoning. Budget-matched search baselines and component ablations are needed to substantiate the claimed agentic contribution.
Critical strengths
-
Well-motivated problem. The paper addresses the target-dependent effects of MSA depth and cross-chain pairing, an important practical issue in protein-complex structure prediction.
-
Practical, training-free approach. The agent improves a frozen AF3 predictor using prediction-derived feedback, without requiring ground-truth structures during optimization.
-
Broad empirical evaluation. Results cover 1,015 interfaces across three benchmarks, with consistent aggregate improvements and particularly strong gains for inter-species complexes.
Critical weaknesses
Missing budget-matched search baselines. The agent performs multiple AF3 evaluations, whereas the main baselines use fixed MSA recipes. Comparisons with random search or a fixed candidate portfolio under the same prediction budget are needed to establish the value of LLM-driven adaptation.
Limited attribution of the agentic contribution. The paper does not isolate the effects of biological reasoning, reflection, memory, and feedback-driven proposal generation. Consequently, it remains unclear which components are necessary.
Efficiency claims require qualification. Reduced MSA depth does not establish lower end-to-end computational cost, given the repeated AF3 evaluations and additional LLM calls.
Confidence-based selection remains unreliable. On FoldBench-v1, final nomination loses approximately 41% of the improvement discovered during exploration, highlighting a substantial mismatch between predictor confidence and structural accuracy.
Some comparisons are not input-matched. On the FoldBench datasets, the agent and the mMSA/pMSA/uMSA baselines use different MSA resources, so these comparisons do not cleanly isolate construction strategy.
Questions for authors
-
Missing budget-matched search baselines. The agent performs multiple AF3 evaluations, whereas the main baselines use fixed MSA recipes. Comparisons with random search or a fixed candidate portfolio under the same prediction budget are needed to establish the value of LLM-driven adaptation.
-
Limited attribution of the agentic contribution. The paper does not isolate the effects of biological reasoning, reflection, memory, and feedback-driven proposal generation. Consequently, it remains unclear which components are necessary.
-
Efficiency claims require qualification. Reduced MSA depth does not establish lower end-to-end computational cost, given the repeated AF3 evaluations and additional LLM calls.
-
Confidence-based selection remains unreliable. On FoldBench-v1, final nomination loses approximately 41% of the improvement discovered during exploration, highlighting a substantial mismatch between predictor confidence and structural accuracy.
-
Some comparisons are not input-matched. On the FoldBench datasets, the agent and the mMSA/pMSA/uMSA baselines use different MSA resources, so these comparisons do not cleanly isolate construction strategy.
Thank you for the additional points. That's helpful to know. I am not going to change my score, because the issue about overall compute cost remains the primary one, especially seeing the responses to the other reviewers where it's clear that this remains extremely impractical atm, given the number of AF3 evaluations required.