acceptodds
Under review as a conference paper at ICLR 2027

recAMPLIFY: Large-Scale Pretraining of Recursive Protein Language Models

Abstract

Protein language models (pLMs) are powerful tools for protein annotation and mutation-effect prediction, but progress has largely depended on scaling model size and data, making them expensive and prone to memorization. Test-time scaling offers a complementary path, but methods developed for autoregressive LLMs do not transfer directly: masked encoders lack a generation loop, and amino acids do not verbalize in a reasoning language. Instead, we adapt recursive pretraining to masked pLMs, where a weight-tied Transformer core refines residue-aligned hidden states over an adjustable number of cycles. We instantiate this recipe as recAMPLIFY, a pLM with M unique parameters trained on the same T tokens as AMPLIFY-M. Despite % fewer parameters, it improves CASP contact recovery and three out of four supervised tasks, notably raising fold accuracy by %. Because optimal depth for recAMPLIFY varies across tasks and assays, we then introduce *Gated Mixture Recurrence* (GMR), a framework that decodes each protein from a probability-weighted mixture of hidden states visited during recursion. GMR employs a learned residual gate at scalar, low-rank, or per-dimension resolution to control how much of each cycle's proposed update is accepted into the state. This unifies halting and gating into a single adaptive computation objective while recovering PonderNet-style halting as a special case without gates. Layered on top of recAMPLIFY, GMR further reduces expected layer depth by up to %, although the accuracy trade-off varies by task. Mechanistic analyses show that residual gating controls recurrence stability beyond the trained depth and that halt depth tracks biological properties.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
Protein language models, Recursive Reasoning, Adaptive computation, Test-time compute scaling, Masked language modeling, Weight-tied recursive Transformers

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_Q4oEOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates how to increase the effective depth of a protein language model by repeatedly applying a shared Transformer core while maintaining parameter count and residue alignment. The authors adapt an existing depth-recurrent framework to a masked protein encoder, resulting in recAMPLIFY, and propose Gated Mixture Recurrence (GMR), which combines weighted mixing of recurrent states with residual gating at different granularities. Full-scale, fixed-depth recAMPLIFY, using 155M parameters and 2T training tokens, outperforms the AMPLIFY baseline on multiple supervised tasks. GMR is primarily evaluated at a setting of 100K steps and approximately 210B tokens, reporting improvements in the KL-calibration score and a decrease in expected recurrence depth. The paper also analyzes the state stability during depth extrapolation and the relationship between halting depth and protein composition.

Critical strengths

S1. Large-scale pre-training and broad task coverage make the parameter-efficient results substantially valuable. The authors completed pre-training with the same 2T-token budget as AMPLIFY. recAMPLIFY with 155M parameters achieved 69.6% accuracy on fold prediction, a 12.9 percentage point improvement over AMPLIFY-350M's 56.7%; this difference is substantially greater than the fluctuations in supervised probe retraining shown in Table 4. The complete results in Table 6 also show that recAMPLIFY achieved the highest mean on 14 out of 18 supervised tasks. While this does not imply that all improvements are statistically significant, these results demonstrate that weight-tied recurrence can lead to competitive protein representations.

S2. Systematic analysis of depth and gating provides useful design experience. The authors not only compared the final task scores but also examined fixed versus variable depth training, truncated backpropagation through time, input injection, and different gating granularities, and analyzed recurrent-state stability by incorporating state update magnitude, normalized state distance, and attention-map changes. In particular, distinguishing between directional similarity and a true cessation of state change makes the analysis more convincing than a single cosine-similarity diagnostic. The paper also discloses performance degradation during depth extrapolation and the input sensitivity limitations of gating, helping readers determine the applicability of these designs.

Critical weaknesses

W1. Baseline settings and computational budgets are insufficient to clearly define the benefits of the recursive architecture.

  • The same token budget does not control computational cost. The effective depth of the full recAMPLIFY in Table 1 is 64 layer applications, while AMPLIFY-350M has 32 layers. Table 5 provides valuable throughput data, but the measurement object is a smaller configuration of 133M and cannot replace the performance–compute comparison corresponding to the main results. Existing results support parameter efficiency but do not fully explain the benefits under the same computational budget.
  • There are specific inconsistencies in the baseline learning rate. Appendix B states that the recurrent model uses 10⁻³, which is ten times higher than the 10⁻⁴ of the AMPLIFY baseline. However, the original authors' model card lists the first-stage pre-training learning rate as 10⁻³, while 10⁻⁴ corresponds to subsequent long-sequence training; the public optimizer configuration also uses 10⁻³. This may simply be a descriptive error, but the interpretation of the relevant comparisons remains uncertain until it is clarified whether it affects 100K-step baseline training.

W2. GMR has not yet demonstrated that the input-adaptive computation mechanism brings independent performance or computational benefits.

  • Expected depth reduction does not equate to actual computational savings. §5.1 clearly states that evaluation decodes the full state mixture; therefore, the 33–37 expected layer applications in Table 1 cannot be directly compared to the actual depth of a fixed-depth model as the same cost metric. A simple explanation is that even with six cycles of states using input-independent uniform weights, the expected number of cycles is only 3.5, but a full state mixture still requires six cycles of computation. This also demonstrates that expected depth alone is insufficient to validate compute allocation capabilities.
  • There is a lack of controls that can isolate input dependence. Appendix G discloses that the variation in expected halting depth for different proteins is only about 1% of the mean. This does not negate the existence of biologically relevant signals, but comparisons with fixed mixture weights or learnable, input-independent gating schedules are needed to determine whether the benefits come from the adaptive computation mechanism. Figure 3b already has shallower fixed-depth models and should also be included in comparisons under the same actual cost.
  • State stability cannot replace the above-mentioned evidence of benefits. The cumulative survival weights of Eq. (3) decay the update magnitude, so the vanishing of update magnitudes does not alone prove that the model has completed useful iterative computation. Appendix G further illustrates this with its analysis of the loss of input sensitivity of scalar gates after the removal of input injection, showing that state stability and effective use of inputs need to be verified separately. Existing mechanistic analyses support that gating can suppress drift, but are insufficient to provide evidence for its actual adaptive benefits.
Questions for authors

Q1. Baseline Training Setup Clarification (W1). Is the 10⁻⁴ learning rate for AMPLIFY in Appendix B a textual error, or is it actually used for retraining the 100K-step baseline? If this setting was used, how was the difference from the original authors' 10⁻³ configuration in the first stage determined, and does it affect the comparison results? Please also clarify the relationship between this description and the publicly released 1M-step checkpoint used in Table 6.

Q2. Computational Budget Comparison (W1, Supplementary Evidence). When the training computational budget matches and actual inference FLOPs or latency are reported separately, do the main benefits of recAMPLIFY still hold? What evidence can existing shallower, fixed-depth models provide to suggest that these benefits exceed the effect of simply increasing the number of layer applications?

Q3. Performance After Actual Halting (W2, Clarification and Supplementary Evidence). In the evaluation corresponding to Table 1, what is the distribution of the actual number of recurrence cycles executed? On what inputs did the threshold rule in §4.1 trigger early exit? If the actual average execution is limited to the reported 33–37 layer applications, can downstream performance be maintained? Please compare with the appropriate fixed-depth configuration in Fig. 3b.

Q4. Independent contribution of input dependence (W2, Supplementary Evidence). Under the same training budget and decoding method, how do performance and actual halting behavior change when the input-dependent mixture weights or gating in GMR are replaced with uniform weights or a learnable schedule shared by all inputs? Can this comparison distinguish the benefits of input-adaptive computation from the benefits of state mixing and update attenuation itself?

4 comments

We thank the reviewer for the precise and actionable questions. Here we answer each with existing trained checkpoints. We will include these to the revision.

Q1: Learning rate and baseline checkpoints.

  • LR 10−410^{-4} is a textual error. All models, including the 100K and 1M AMPLIFY baselines, use a peak LR of 10⁻310^{⁻³}, as in the original AMPLIFY first stage. All use AdamW (0.9, 0.95), 1K warm-up steps and a cosine decay to a 10⁻⁴ floor.
  • Table 6 baselines are our retrains, not the public checkpoints. Both 1M AMPLIFY baselines were retrained with AMPLIFY's public code, data and stage-1 recipe (with max length 512). This gives recAMPLIFY and the baselines the same data, tokens, sequence length and evaluation pipeline. We will correct the caption to avoid any confusion.

Q2: Compute-matched comparison. AMPLIFY-350M and recAMPLIFY share the same block (d = 960, 15 heads, FFN 2560), so layer applications are directly proportional to FLOPs. Our existing 100K fixed-depth sweep (Fig. 3b models) contains recurrent models that cost less than AMPLIFY-350M:

100K stepslayersFLOPs/tok (rel.)inference tok/s (rel.)CASPFoldLocStabKL (T)KL (NT)DMS
AMPLIFY-350M321.001.00.104.570.672.47959.838.8.359
recAMPLIFY, L=2200.62–.129.647.712.71450.136.2.349
recAMPLIFY, L=3280.870.89.130.654.717.64450.336.3.347
  • The gains are not explained by more layer applications. With 62–87% of AMPLIFY-350M's compute, recurrence is better on CASP, Fold, Localization and Stability. Paired bootstrap 95% CIs exclude zero for all four (L=3 vs. AMPLIFY-350M). Training throughput also aligns accordingly: L=3 trains at 0.92× AMPLIFY-350M (Table 5).
  • The trade-off. The wider model keeps its lead on KL calibration and DMS, which we will state explicitly.
  • Inference cost of the actual Table 1 models. We addressed the 133M-only concern by measuring the Table 1 models themselves (A100-40GB, real sequences, bf16):
modelms/seq, batch 1K tok/s, batch 32
AMPLIFY-120M14.3184.7
AMPLIFY-350M18.384.9
recAMPLIFY 1M (155M)29.450.6
  • Full scale. The 1M recAMPLIFY applies 44 layers (2 + 4×10 + 2), i.e. 1.38× AMPLIFY-350M's FLOPs per token, and is 1.7× slower at inference. We will state that the 1M comparison is matched on tokens, not compute. Instead, the 100K sweep above is the compute-matched evidence.

Q3: Actual halting. We measured it on 10K held-out proteins.

  • Executed cycles. We implemented true per-sequence early exit at the paper's threshold ε = 10⁻³. No sequence in any GMR variant exits early (nor at any ε ≤ 0.1), every input runs all 6 cycles. Aligned with the reviewer's comments, the learned halting mass is nearly uniform (≈0.17–0.20 per cycle).
  • Expected vs. executed depth. The 33–37 layers in Table 1 are expected depth, not executed depth. GMR's measured cost equals fixed L=6: 34.7 vs. 32.3 ms/seq at batch 1, and 44.5–44.9K vs. 45.4K tok/s at batch 32. We will report executed depth (52) in Table 1.
  • Performance when actually limited to 36 layers. We stopped GMR after 4 cycles, and compared it with the fixed-depth model trained at L=4 (Fig. 3b, same executed cost, measured 60.6K vs. 61.9K tok/s):
100K, 36 layers executedKL (T)KL (NT)DMSCASPFoldLocStab
fixed L=4 (trained)50.636.4.343.142.658.718.674
scalar GMR, stopped at 4 cycles47.635.5.345.155.620.710.732
(scalar GMR, all 6 cycles)49.836.5.346.161.628.708.737
  • Scalar GMR largely maintains its performance under the real 36-layer budget.
  • It keeps CASP (+9%) and Stability (+8.6%) gains over the equally costly fixed-depth model.
  • It trails on KL, Fold and Localization. Halt-only GMR does not hold up (CASP .135 → .117).
  • We will restrict the claim to these findings and soften the comparison against fixed L=6.

Q4: Input dependence vs. mixing and attenuation. Inference-only controls on the trained models:

scalar GMR, 100KKL (T)KL (NT)DMSCASP
learned, input-dependent49.836.5.346.161
uniform mixing weights49.836.4.346.159
shared, input-independent schedule (drives the updates too)49.836.4.346.161
no mixing (decode last cycle, keep the learned gates)30.123.6.291.079
  • Input dependence contributes nothing measurable. Halt-only GMR behaves the same: ≤0.002≤0.002 on CASP/DMS, ≤0.1≤0.1 on KL.
  • The gains come from state mixing combined with gated updates. Removing the mixture collapses the gated model, while removing input dependence changes nothing.
  • Caveat on our shared schedule used above. It is the dataset-mean schedule of the trained model since a schedule learned from scratch would require retraining the models.
  • Revision. We will reframe GMR as learned cycle mixing with update attenuation rather than input-adaptive compute allocation, and present its small input dependence as an interpretability signal.

**W2: State stability cannot replace the above-mentioned evidence of benefits. ** We agree. Vanishing update norms partly follow from the decaying survival in Eq. (3)(3). We will describe them as update attenuation rather than convergence, and no longer use them as evidence of useful computation. The input-injection ablation (App. G) still remains the separate test of input use.

Dear Reviewer Q4oE,

As the discussion period ends on November 18, we would be grateful to hear whether our rebuttal addressed your concerns, or if there is anything else we can clarify before the deadline.

If you find that our responses have adequately addressed your concerns, we would sincerely appreciate it if you would consider updating your score accordingly.

Thank you for your time and consideration.

Best regards,

The Authors

Dear Reviewer Q4oE,

We would like to kindly draw your attention to our rebuttal, in which we have provided additional experiments and clarifications addressing the concerns raised in your review. As we are approaching the end of the rebuttal period, we would be grateful if you could take these responses into consideration and, if appropriate, revisit your evaluation.

Thank you for your time and consideration.

Reviewer_8gmKOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper introduces a method called recursive pre-training, which helps to scaling test-time compute (in latent space) for protein language models. The method the paper builds on is the one from Geiping et. al.'s work on recurrent depth. The key innovation here is translating it to work for protein language models with bidirectional attention. They find that the number of iterations/recurrence/refinement that the model wants to perform is correlated with factors like amino acid diversity and hydrophobic content.

Critical strengths

This paper adapts the work of Geiping et. al.'s work on recurrent depth to the protein language setting. Some of the empirical findings here are quite interesting:

  1. They are able to use almost 50 percent fewer parameters to give a 20 percent improvement in folding.
  2. This is the first adaptation of latent test-time improvement in the protein language modeling literature that I am aware of.
  3. The Gated Mixture Recurrence is a simple approach that is effective in the experiments shows.
Critical weaknesses

There is some relevant literature that this work should further engage with: Full-bandwidth Transformer by Wang et. el. (uses latent feedback between consecutive autoregressive decoding steps), and Recirculation by Mozer et. al. ( uses recurrent state processing).

The improvements from GMR itself are somewhat inconsistent, in the sense that the best results are not achieved by the same GMR variant. In fact, the halt-only variant improves upon the more elaborate variants on 3out of the 4 main supervised tasks.

Questions for authors
  1. In Table 1, the halt-only vairant outperforms the more complex variants on 3 out of the 4 tasks. However, it seems the per-dimension gating performs best on KL calibration. Why is this? Should one just dispense with fine-grained gating?
3 comments

We thank the reviewer for the positive assessment and the helpful pointers.

Related work. We will add both suggested works to the related work section and discuss how they relate. Like recAMPLIFY, Full-bandwidth Transformer (Wang et al.) and Recirculation (Mozer et al.) feed latent states back into computation. Our setting differs in two main ways:

  • recurrence runs over depth, within a single forward pass of a masked bidirectional encoder with tied blocks
  • there is no decoding loop to carry state across steps.

Inconsistency across GMR variants: should one dispense with fine-grained gating?

To separate real differences from noise, we computed paired bootstrap 95% CIs over test examples (over targets for CASP) for each gated variant against halt-only (100100K steps):

Δ vs. halt-only GMRCASPFoldLocStab
scalar gate+.026 [+.012, +.041]−.009 [−.020, +.001]−.002 [−.019, +.014]+.065 [+.059, +.071]
rank-16 gate+.015 [+.006, +.024]−.020 [−.031, −.009]+.013 [−.003, +.029]+.040 [+.034, +.046]
per-dim gate+.008 [−.002, +.017]−.013 [−.023, −.002]+.016 [+.002, +.031]−.031 [−.039, −.024]

(bold: CI excludes zero)

  • The scalar gate is the only variant with no significant loss. It significantly improves CASP and Stability over halt-only, and its Fold and Localization differences are not significant. It is also the cheapest gate computationally (one value per cycle).

  • Finer gates do not pay off. Rank-16 and per-dim each trade a significant loss on at least one task for their gains.

  • Robust under a real compute budget. Scalar GMR keeps its CASP and Stability advantage when actually stopped after 4 cycles, at the cost of a fixed-depth model trained at L=4. Halt-only GMR does not (CASP .135 → .117).

  • KL calibration is effectively tied across variants (KL (T) 49.7–50.4, KL (NT) 36.3–36.5). The per-dim/rank-1616 lead is smaller than the spread across fixed-depth models of the same family (KL (T) 49.9–50.6 for L = 3–6), so we do not interpret it as an effect of gate resolution.

  • Why halt-only can win on Fold/SSP. Halt-only differs from the gated variants in two further training settings: it mixes raw rather than normalized states, and uses no minimum cycle count. Its comparison with the gated variants is therefore not a clean ablation of gate resolution.

  • Conclusion. Yes, we recommend the scalar gate as the default and will state that finer-grained gating brings no consistent benefit in our experiments. We will also revise the text so it no longer implies a monotone benefit from gate resolution.

Dear Reviewer 8gmK,

As the discussion period ends on November 18, we would be grateful to hear whether our rebuttal addressed your concerns, or if there is anything else we can clarify before the deadline.

If you find that our responses have adequately addressed your concerns, we would sincerely appreciate it if you would consider updating your score accordingly.

Thank you for your time and consideration.

Best regards,

The Authors

Reviewer_8gmKOpenReview ↗

Overview of response

I have looked at the responses to the other authors and to the review I had. You have addressed my concerns.

Overall, I would suggest that you include in the paper the conclusion that "finer-grained gating brings no consistent benefit in our experiments". It would be helpful for future practitioners.

Reviewer_hXHhOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates how recursive computation can improve the parameter efficiency of protein language models. It introduces recAMPLIFY, which repeatedly applies a shared-parameter Transformer core to progressively refine protein representations while preserving alignment between amino acid positions and hidden states. To accommodate potentially different depth requirements across tasks and sequences, the authors further propose GMR, which combines hidden states from different recursive cycles and controls residual updates to enable input-dependent computation. Experiments show that, under the same training token budget, recAMPLIFY uses approximately 56% fewer parameters than AMPLIFY-350M while achieving better results on fold classification, subcellular localization, and contact prediction. Mechanistic analyses examine the effects of gating on recurrence stability and the relationship between halting depth and protein sequence composition.

Critical strengths

1. The full-scale pretraining results demonstrate substantial improvements.

Compared with AMPLIFY-350M, recAMPLIFY reduces the number of unique parameters by approximately 56% while improving fold classification accuracy from 56.7% to 69.6%. Both the magnitude of this improvement and its consistency across repeated downstream prediction-head training runs are convincing.

2. The experiments cover a broad range of tasks and include unfavorable results.

The authors evaluate both zero-shot and supervised tasks. The appendix provides results for the complete 18-task benchmark, alongside ablations of truncated backpropagation, input injection, and architectural choices. The paper also reports that mutation-effect prediction does not outperform the main baseline and that some gating variants exhibit unstable performance, helping clarify the method’s applicability and limitations.

3. The distinction between halting the output computation and stopping state evolution is insightful.

Learning a halting distribution alone does not directly constrain the magnitude of recurrent state updates. The paper investigates this distinction through residual gating and jointly examines state differences, residuals, and changes in attention, providing richer evidence than final task scores alone.

Critical weaknesses

1. The main comparisons do not adequately disentangle parameter efficiency from computational budget.

The full-scale recAMPLIFY has an effective depth of 64 layers, compared with 32 layers for AMPLIFY-350M (Table 1). Matching the number of training tokens therefore does not imply matching training FLOPs or GPU time. Appendix B also states that the recurrent models use a tenfold higher peak learning rate.

These differences do not invalidate the parameter-efficiency results, but they limit attribution of the performance gains. It remains unclear how much of the improvement comes from weight sharing, deeper computation, or different optimization settings. Additional comparisons with feedforward models under matched computational budgets, together with comparably tuned baselines, are needed.

2. The practical benefits of input adaptivity have not been established.

Appendix G states that expected halting depth varies by only approximately 1% around its mean across proteins. This suggests that the model may primarily learn a global recurrence schedule, with relatively weak input-dependent modulation.

Key controls are missing, including a shared, learnable gating schedule, input-independent depth mixing, and a fixed recurrence count chosen under the same average computational budget. Without these comparisons, it remains unclear whether GMR’s improvements require input-dependent allocation of computation.

3. The contributions of GMR’s individual components have not been adequately disentangled.

GMR simultaneously introduces aggregation across multiple depths, residual gating, entropy regularization of the halting distribution, and stop-gradient operations. The existing comparisons do not fully isolate the contributions of these components.

4. Some performance claims and biological interpretations warrant greater caution.

The mean stability prediction score increases from 0.461 to 0.501 (Table 4), but the recurrent model exhibits substantial variance, leaving the robustness of this improvement unclear. The main advantages observed at full scale also cannot be directly extended to GMR, since its experiments are primarily conducted at the 100K-step scale.

The biological analyses support an association between halting depth and sequence composition. However, the explanation that hydrophobic sequences receive more computation because they require more long-range structural evidence remains a mechanistic hypothesis. Similar halting depths among isoforms of the same gene could also be explained by sequence and compositional similarity. Controls such as composition-preserving sequence shuffling and random protein pairs matched for composition would help determine whether the signal extends beyond simple sequence statistics.

Questions for authors
  1. Comparisons under matched computational budgets: Do recAMPLIFY’s advantages over AMPLIFY persist when training FLOPs or GPU hours are matched? Could the authors include a model with unshared parameters and a similar effective depth, while allocating a comparable learning-rate search budget to the baselines?

  2. Actual inference cost: When early stopping is implemented using the threshold specified in the paper, what are GMR’s actual executed cycle counts, single-sequence latency, and batch throughput? How does performance after early stopping compare with full-mixture decoding?

  3. Sources of GMR’s improvements: Could the authors separately ablate state mixing, residual gating, entropy regularization, and stop-gradient operations? How do the authors explain the halt-only variant outperforming finer-grained gating on several downstream tasks?

  4. Robustness and scaling: Which random seeds or training factors account for the substantial variance in stability prediction? Could the authors evaluate at least one GMR variant at the full training scale and report confidence intervals for the key metrics?

  5. Independence of the biological signal: After controlling for amino acid composition, sequence similarity, and protein family, how much association remains between halting depth and structural or functional properties? Does composition-preserving sequence shuffling change the gating schedule and computational allocation?

4 comments

We thank the reviewer for the careful and constructive review. We have addressed their concerns below.

W1 / Q1: Compute vs. parameter efficiency

  • Learning rate: this was a typo. All models, including the AMPLIFY baselines, use a peak LR of 10⁻³ with a cosine decay to a 10⁻⁴ floor. 10⁻⁴ is only the floor value. We verified this against the saved run configurations and logged LR curves. Both families use the identical AMPLIFY recipe (optimizer, schedule, data, tokens).
  • Matched compute. AMPLIFY-350M is exactly the unshared model: it uses the same block as recAMPLIFY (d = 960, 15 heads, FFN 2560), so layer applications translate directly into FLOPs. Our 100K fixed-depth sweep provides recursive models that use less compute than it:
100K stepslayersFLOPs/tok (rel.)inference tok/s (rel.)CASPFoldLocStabKL (T)KL (NT)DMS
AMPLIFY-350M321.001.00.104.570.672.47959.838.8.359
recAMPLIFY, L=2200.62–.129.647.712.71450.136.2.349
recAMPLIFY, L=3280.870.89.130.654.717.64450.336.3.347

We observe the following:

  • At equal or lower compute, recurrence improves CASP, Fold, Localization and Stability.
  • Training throughput agrees: L=3 trains at 0.92×0.92\times AMPLIFY-350M (Table 5).
  • The wider model keeps its lead on KL calibration and DMS, which we will state explicitly.
  • Full scale. The 1M recAMPLIFY applies 44 layers (2 + 4×10 + 2), i.e. 1.38× AMPLIFY-350M's FLOPs per token. Its measured inference is 1.7× slower. We will state that the 1M comparison is matched on tokens, not compute, and that the 100K sweep above is our compute-matched evidence.

W2 / Q2: Input adaptivity and actual inference cost. We agree, and quantified it on 10K held-out proteins.

  • Executed cycles. At the paper's threshold ε = 10⁻³ (and at any ε ≤ 0.1), no sequence exits early in any GMR variant: all inputs run 6 cycles. The learned halting mass is nearly uniform (≈0.17–0.20 per cycle). The 33–37 layers in Table 1 are therefore the expected depth of the mixture weights, not saved compute. We will relabel the columns to avoid any confusion.
  • Latency (A100-40GB). GMR costs the same as fixed L=6: 34.7 vs. 32.3 ms/sequence at batch 1, and 44.5–44.9K vs. 45.4K tokens/s at batch 32.
  • Controls (inference-only, on the trained scalar-gated GMR):
scalar GMR, 100Klayers runKL (T)KL (NT)DMSCASPFoldLocStab
learned, input-dependent5249.836.5.346.161.628.708.737
uniform cycle mixing5249.836.4.346.159
shared input-independent schedule5249.836.4.346.161
early stop at 4 cycles3647.635.5.345.155.620.710.732
fixed L=4 (trained, same cost)3650.636.4.343.142.658.718.674
  • Replacing the input-dependent weights with uniform or shared ones leaves every probe unchanged (halt-only GMR likewise: ≤0.002 on CASP/DMS, ≤0.1 on KL). Input-dependent allocation is thus not what drives GMR's gains.
  • When GMR actually runs only 36 layers, it keeps its CASP and Stability gains over the equally costly fixed-depth model, but lags on Fold and KL.
  • The shared schedule above is the dataset-mean schedule of the trained model, we note that a schedule learned from scratch would require retraining.
  • We will reframe GMR as a learned cycle-mixing and update-damping mechanism, with its small input dependence presented as an interpretability signal (Q5).

W3 / Q3: Sources of GMR's improvements.

  • State mixing (inference-only: decode the last cycle only, keeping the learned gates):
    • Halt-only GMR barely changes: KL (T) 49.7→48.2, CASP 0.135→0.130.
    • Scalar-gated GMR collapses: KL (T) 49.8→30.1, CASP 0.161→0.079, DMS 0.346→0.291.
    • Gated updates alone therefore do not explain the gains. The gated models do rely on the mixture read-out they were trained with.
  • Entropy regularization and stop-gradient: We note that these require retraining and are beyond the rebuttal budget. We thus list this as a limitation. The stop-gradient keeps the ponder cost from influencing the training of the recurrent core. This is standard practice for auxiliary objectives that should not alter a shared representation [1,2], and are likewise central to self-supervised methods [3,4].

References:

[1] van den Oord, Aaron, et al. "Neural Discrete Representation Learning." NeurIPS 2017.

[2] Schuster, Tal, et al. "Confident Adaptive Language Modeling." NeurIPS 2022.

[3] Assran, Mahmoud, et al. "Self-Supervised Learning from Images with a Joint-Embedding Predictive Architecture." CVPR 2023.

[4] Caron, Mathilde, et al. "Emerging Properties in Self-Supervised Vision Transformers." ICCV 2021.

  • Halt-only vs. finer gates. The gaps are small (≤0.016 on Fold and SSP), and the gated variants also differ in two training settings (minimum cycles and mixing normalized states). The comparison is therefore not a clean gate-resolution ablation, and we will not draw conclusions about gate resolution from it.

W4 / Q4: Robustness, confidence intervals, scale.

  • Stability numbers. The quoted values come from the submitted evaluation. After correcting our probe pipeline (standardized regression targets), the stability scores for 1M model are 0.585 ± 0.020 (AMPLIFY-350M) vs. 0.703 ± 0.025 (recAMPLIFY).
  • Source of variance. The spread comes from the probe-head seeds. Each model has a single pretraining run, so pretraining-seed variance is not measured, which we will state.
  • Paired bootstrap 95% Confidence Intervals (CIs) over test examples (CASP: over targets), recAMPLIFY 1M − AMPLIFY-350M: CASP +.032 [+.004, +.066]; Fold +.135 [+.120, +.149]; Loc +.027 [+.008, +.047]; Stab +.171 [+.162, +.180]. We will add CIs for all key comparisons.
  • GMR at 1M steps is beyond our compute budget. We thus restrict all GMR claims explicitly to the 100K scale.

W5 / Q5: Independence of the biological signal.

  • Composition explains only R² = 0.07 (5-fold cross validation) of per-protein halt depth after controlling for length and prediction sharpness.
  • Composition-preserving shuffling (residues permuted within each protein with 10K proteins and all GMR variants):
    • The global schedule is unchanged: same mean depth, still no early exits.
    • Per-protein depth correlates only r ≈ 0.3 with the original, so ~90% of the per-protein variation depends on residue order, not composition.
  • Isoforms vs. composition-matched pairs. Same-gene isoforms remain more similar in halt depth than proteins matched for composition. The within-gene std ratio is 0.66, outside the null's 95% CI. The ratio is 0.56 after additionally regressing out composition.
  • Remaining limits. Isoforms share exons, so sequence similarity cannot be excluded, and we did not run a protein-family control. We will therefore present the isoform result as an association and the "hydrophobic proteins need more long-range evidence" reading as a hypothesis.

Dear Reviewer hXHh,

As the discussion period ends on November 18, we would be grateful to hear whether our rebuttal addressed your concerns, or if there is anything else we can clarify before the deadline.

If you find that our responses have adequately addressed your concerns, we would sincerely appreciate it if you would consider updating your score accordingly.

Thank you for your time and consideration.

Best regards,

The Authors

Thank you for your reply. It has cleared up most of my questions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.