- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper investigates how to increase the effective depth of a protein language model by repeatedly applying a shared Transformer core while maintaining parameter count and residue alignment. The authors adapt an existing depth-recurrent framework to a masked protein encoder, resulting in recAMPLIFY, and propose Gated Mixture Recurrence (GMR), which combines weighted mixing of recurrent states with residual gating at different granularities. Full-scale, fixed-depth recAMPLIFY, using 155M parameters and 2T training tokens, outperforms the AMPLIFY baseline on multiple supervised tasks. GMR is primarily evaluated at a setting of 100K steps and approximately 210B tokens, reporting improvements in the KL-calibration score and a decrease in expected recurrence depth. The paper also analyzes the state stability during depth extrapolation and the relationship between halting depth and protein composition.
Critical strengths
S1. Large-scale pre-training and broad task coverage make the parameter-efficient results substantially valuable. The authors completed pre-training with the same 2T-token budget as AMPLIFY. recAMPLIFY with 155M parameters achieved 69.6% accuracy on fold prediction, a 12.9 percentage point improvement over AMPLIFY-350M's 56.7%; this difference is substantially greater than the fluctuations in supervised probe retraining shown in Table 4. The complete results in Table 6 also show that recAMPLIFY achieved the highest mean on 14 out of 18 supervised tasks. While this does not imply that all improvements are statistically significant, these results demonstrate that weight-tied recurrence can lead to competitive protein representations.
S2. Systematic analysis of depth and gating provides useful design experience. The authors not only compared the final task scores but also examined fixed versus variable depth training, truncated backpropagation through time, input injection, and different gating granularities, and analyzed recurrent-state stability by incorporating state update magnitude, normalized state distance, and attention-map changes. In particular, distinguishing between directional similarity and a true cessation of state change makes the analysis more convincing than a single cosine-similarity diagnostic. The paper also discloses performance degradation during depth extrapolation and the input sensitivity limitations of gating, helping readers determine the applicability of these designs.
Critical weaknesses
W1. Baseline settings and computational budgets are insufficient to clearly define the benefits of the recursive architecture.
- The same token budget does not control computational cost. The effective depth of the full recAMPLIFY in Table 1 is 64 layer applications, while AMPLIFY-350M has 32 layers. Table 5 provides valuable throughput data, but the measurement object is a smaller configuration of 133M and cannot replace the performance–compute comparison corresponding to the main results. Existing results support parameter efficiency but do not fully explain the benefits under the same computational budget.
- There are specific inconsistencies in the baseline learning rate. Appendix B states that the recurrent model uses 10⁻³, which is ten times higher than the 10⁻⁴ of the AMPLIFY baseline. However, the original authors' model card lists the first-stage pre-training learning rate as 10⁻³, while 10⁻⁴ corresponds to subsequent long-sequence training; the public optimizer configuration also uses 10⁻³. This may simply be a descriptive error, but the interpretation of the relevant comparisons remains uncertain until it is clarified whether it affects 100K-step baseline training.
W2. GMR has not yet demonstrated that the input-adaptive computation mechanism brings independent performance or computational benefits.
- Expected depth reduction does not equate to actual computational savings. §5.1 clearly states that evaluation decodes the full state mixture; therefore, the 33–37 expected layer applications in Table 1 cannot be directly compared to the actual depth of a fixed-depth model as the same cost metric. A simple explanation is that even with six cycles of states using input-independent uniform weights, the expected number of cycles is only 3.5, but a full state mixture still requires six cycles of computation. This also demonstrates that expected depth alone is insufficient to validate compute allocation capabilities.
- There is a lack of controls that can isolate input dependence. Appendix G discloses that the variation in expected halting depth for different proteins is only about 1% of the mean. This does not negate the existence of biologically relevant signals, but comparisons with fixed mixture weights or learnable, input-independent gating schedules are needed to determine whether the benefits come from the adaptive computation mechanism. Figure 3b already has shallower fixed-depth models and should also be included in comparisons under the same actual cost.
- State stability cannot replace the above-mentioned evidence of benefits. The cumulative survival weights of Eq. (3) decay the update magnitude, so the vanishing of update magnitudes does not alone prove that the model has completed useful iterative computation. Appendix G further illustrates this with its analysis of the loss of input sensitivity of scalar gates after the removal of input injection, showing that state stability and effective use of inputs need to be verified separately. Existing mechanistic analyses support that gating can suppress drift, but are insufficient to provide evidence for its actual adaptive benefits.
Questions for authors
Q1. Baseline Training Setup Clarification (W1). Is the 10⁻⁴ learning rate for AMPLIFY in Appendix B a textual error, or is it actually used for retraining the 100K-step baseline? If this setting was used, how was the difference from the original authors' 10⁻³ configuration in the first stage determined, and does it affect the comparison results? Please also clarify the relationship between this description and the publicly released 1M-step checkpoint used in Table 6.
Q2. Computational Budget Comparison (W1, Supplementary Evidence). When the training computational budget matches and actual inference FLOPs or latency are reported separately, do the main benefits of recAMPLIFY still hold? What evidence can existing shallower, fixed-depth models provide to suggest that these benefits exceed the effect of simply increasing the number of layer applications?
Q3. Performance After Actual Halting (W2, Clarification and Supplementary Evidence). In the evaluation corresponding to Table 1, what is the distribution of the actual number of recurrence cycles executed? On what inputs did the threshold rule in §4.1 trigger early exit? If the actual average execution is limited to the reported 33–37 layer applications, can downstream performance be maintained? Please compare with the appropriate fixed-depth configuration in Fig. 3b.
Q4. Independent contribution of input dependence (W2, Supplementary Evidence). Under the same training budget and decoding method, how do performance and actual halting behavior change when the input-dependent mixture weights or gating in GMR are replaced with uniform weights or a learnable schedule shared by all inputs? Can this comparison distinguish the benefits of input-adaptive computation from the benefits of state mixing and update attenuation itself?
4 comments
Q4: Input dependence vs. mixing and attenuation. Inference-only controls on the trained models:
| scalar GMR, 100K | KL (T) | KL (NT) | DMS | CASP |
|---|---|---|---|---|
| learned, input-dependent | 49.8 | 36.5 | .346 | .161 |
| uniform mixing weights | 49.8 | 36.4 | .346 | .159 |
| shared, input-independent schedule (drives the updates too) | 49.8 | 36.4 | .346 | .161 |
| no mixing (decode last cycle, keep the learned gates) | 30.1 | 23.6 | .291 | .079 |
- Input dependence contributes nothing measurable. Halt-only GMR behaves the same: on CASP/DMS, on KL.
- The gains come from state mixing combined with gated updates. Removing the mixture collapses the gated model, while removing input dependence changes nothing.
- Caveat on our shared schedule used above. It is the dataset-mean schedule of the trained model since a schedule learned from scratch would require retraining the models.
- Revision. We will reframe GMR as learned cycle mixing with update attenuation rather than input-adaptive compute allocation, and present its small input dependence as an interpretability signal.
**W2: State stability cannot replace the above-mentioned evidence of benefits. ** We agree. Vanishing update norms partly follow from the decaying survival in Eq. . We will describe them as update attenuation rather than convergence, and no longer use them as evidence of useful computation. The input-injection ablation (App. G) still remains the separate test of input use.
Dear Reviewer Q4oE,
As the discussion period ends on November 18, we would be grateful to hear whether our rebuttal addressed your concerns, or if there is anything else we can clarify before the deadline.
If you find that our responses have adequately addressed your concerns, we would sincerely appreciate it if you would consider updating your score accordingly.
Thank you for your time and consideration.
Best regards,
The Authors
Dear Reviewer Q4oE,
We would like to kindly draw your attention to our rebuttal, in which we have provided additional experiments and clarifications addressing the concerns raised in your review. As we are approaching the end of the rebuttal period, we would be grateful if you could take these responses into consideration and, if appropriate, revisit your evaluation.
Thank you for your time and consideration.
We thank the reviewer for the precise and actionable questions. Here we answer each with existing trained checkpoints. We will include these to the revision.
Q1: Learning rate and baseline checkpoints.
Q2: Compute-matched comparison. AMPLIFY-350M and recAMPLIFY share the same block (d = 960, 15 heads, FFN 2560), so layer applications are directly proportional to FLOPs. Our existing 100K fixed-depth sweep (Fig. 3b models) contains recurrent models that cost less than AMPLIFY-350M:
Q3: Actual halting. We measured it on 10K held-out proteins.