acceptodds
Under review as a conference paper at ICLR 2027

Scaling to Tens of Thousands of Test-Time Iterations with Loop-Native Attention Residuals

38 views

Abstract

In this paper, we argue that looped Transformers need their own residual connections to prevent performance degradation as the number of iterations grows. We observe that increasing loop iterations can reduce reasoning accuracy: noisy state updates overwrite correct intermediate deductions and even undo completed solutions. Under the standard carry-last rule, each new output replaces the recurrent state, leaving subsequent iterations to recover lost information from an already degraded representation. We introduce InfiLoop, a loop-native residual connection that learns which past computations to retain and how much to accept from each new update. InfiLoop combines content-based weighting with learned temporal decay to maintain a running summary of recurrent states. An exact streaming recurrence keeps its persistent aggregation memory constant as the loop count grows. The resulting adaptive update suppresses unreliable proposals and preserves useful intermediate states. Across extensive reasoning tasks, a 7M-parameter InfiLoop model outperforms existing recursive architectures, reaching 97.9% exact accuracy on Sudoku-Extreme, and 13.6% pass@2 on ARC-AGI-2. It is worth noting that on Sudoku-Extreme, InfiLoop scales test-time looping beyond 20,000 effective steps, effectively translating greater computational depth into higher reasoning capacity.

Primary area:
foundation or frontier models, including LLMs
Keywords:
loop transformer, HRM, attention residuals

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_8ojcOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper investigates the performance drop that can occur in looped Transformers as test-time iterations increase, when new states overwrite previously correct intermediate results. It proposes InfiLoop, which aggregates historical hidden states using content-based scoring and temporal decay. Experiments show competitive results on Sudoku, Maze, and ARC; on Sudoku-Extreme, its accuracy continues to improve as the number of executed layers increases.

Critical strengths
  1. The method is simple yet effective. Content-based scoring and temporal decay determine which historical states to retain, and the method can be interpreted as an adaptive residual update.
  2. InfiLoop is memory-efficient. The derived streaming recurrence uses constant aggregation memory with respect to the number of iterations, without storing the full history of states.
  3. The paper includes depth-scaling, ablation, and state-perturbation analyses.
Critical weaknesses
  1. The relationship to related work could be discussed more fully. The paper mentions methods such as Attention Residuals, DenseFormer, and FPRM, but does not sufficiently clarify what InfiLoop adds beyond existing learned residual aggregation and gated recurrent updates.
  2. The experimental scope remains limited to Sudoku, Maze, and ARC, while the main depth-scaling and stability analyses focus on Sudoku-Extreme. It is unclear whether InfiLoop offers similar benefits on more general tasks or standard, non-nested looped Transformers.
Questions for authors

See weaknesses.

2 comments
AuthorsOpenReview ↗

Response to Reviewer 8ojc

W1. "The relationship to related work could be discussed more fully. The paper mentions methods such as Attention Residuals, DenseFormer, and FPRM, but does not sufficiently clarify what InfiLoop adds beyond existing learned residual aggregation and gated recurrent updates."

A1. Due to the page limit, we placed this comparison in Appendix B. We will move it into the main text and expand it as follows.

Full Attention Residuals gives each layer its own query and stores all previous outputs, so it applies to a fixed depth and its memory grows with depth. InfiLoop shares one scorer across all iterations and adds an exponential decay. This gives an exact two-variable recurrence whose memory does not depend on the number of iterations, so one rule applies at any depth. The decay is also what makes this aggregation trainable in a loop: with uniform weights (β=1\beta=1, Attention Residuals with a shared query) the model reaches 38.4% at H3 L6, against 82.1% for InfiLoop (Table 2).

DenseFormer learns one scalar weight per pair of layers. The weights do not depend on the input and the depth is fixed. InfiLoop computes a content-dependent weight per token and is defined for any number of iterations. Removing the content score decreases accuracy by 6.8% (Appendix D).

FPRM stabilizes the iteration with pre-norm, residual scaling and damped fixed-point updates, and halts on convergence. While FPRM introduces scaling on residuals, it does not introduce a residual connection designed for looping, and Pre-LN is known to be insufficient for deep models (Curse of Depth; DepthBench). InfiLoop learns from the task loss which candidate states to keep. InfiLoop reaches 90.9% at 1,872 layers, above FPRM's 90.1% at 3,744 layers, and 92.74% against 91.63% at 24,960 layers (Section 4.4).

A gated update sets the step size from the current input and state. The step size of InfiLoop, γt=et/(βDt−1+et)\gamma_t=e_t/(\beta D_{t-1}+e_t) (Eq. 5), is set relative to the accumulated history weight, which makes the update an exact normalized average over the whole history. Following Reviewer FzAQ, we compared InfiLoop with two learned sigmoid gates on the same backbone and training:

Connection (7,680 puzzles)H3 L6 (training depth)H32 L6
Gate σ(w⋅RMS(z)+b)\sigma(w\cdot\mathrm{RMS}(z)+b)80.782.9
Gate σ(w⋅RMS(z)+u⋅RMS(h)+b)\sigma(w\cdot\mathrm{RMS}(z)+u\cdot\mathrm{RMS}(h)+b)81.684.0
InfiLoop82.189.7

W2. "The experimental scope remains limited to Sudoku, Maze, and ARC, while the main depth-scaling and stability analyses focus on Sudoku-Extreme. It is unclear whether InfiLoop offers similar benefits on more general tasks or standard, non-nested looped Transformers."

A2. We applied InfiLoop to a standard looped Transformer with a single recurrent state: URM, with its official code and recipe and only the connection between loop iterations changed.

Sudoku-Extreme, full test setExact accuracy
URM77.6
URM (Reproduced by us)75.9
URM + InfiLoop84.3

InfiLoop improves the single-state looped Transformer from 77.6% to 84.3%. InfiLoop is defined for a single recurrent state (Eq. 4, Section 3.3), and TRM applies it to each of its two states; Table 2 shows that each state alone also improves with depth.

We run the depth and perturbation analyses on Sudoku-Extreme because every prediction can be checked exactly, which lets us follow individual puzzles and cells across tens of thousands of layers. The depth benefit also holds on ARC-AGI: with 4× the training depth, InfiLoop improves from 49.1% to 52.4% on ARC-AGI-1 and from 13.6% to 14.2% on ARC-AGI-2.

Furthermore, to better address your concern regarding general language tasks and standard non-nested architectures, we conducted continual pre-training (CPT) on Qwen3-1.7B-Base to integrate InfiLoop. Specifically, starting from the 28-layer Qwen3-1.7B-Base without adding any extra layers, all recurrent iterations share the weights of the original layers. We trained on 2B tokens sampled from a Nemotron data mixture, a global batch size of 64 × 1024 tokens, and a cosine learning rate schedule with 2% warmup decaying to 10% of the peak rate (backbone lr = 1e-5, query lr = 1e-3). During training, the number of recurrent iterations KK is uniformly sampled from {1,2,3,4,5}\{1, 2, 3, 4, 5\} at each optimizer step. We evaluated the resulting models across different inference loop counts (K∈{1,2,3,5}K \in \{1, 2, 3, 5\}) on MATH-500:

Model / MethodK=1K=1K=2K=2K=3K=3K=5K=5
Qwen3-1.7B-Base39.2———
Qwen3-1.7B-Base + Standard Loop (Pre-LN)42.243.043.240.4
Qwen3-1.7B-Base + InfiLoop41.843.844.245.8

With more loops at inference, the standard loop degrades from K=3 to K=5 (43.2% → 40.4%), while InfiLoop keeps improving (44.2% → 45.8%).

Dear Reviewer 8ojc,

Thank you again for your questions about InfiLoop’s connection to prior work and its use beyond Sudoku. InfiLoop weights each new state against its history using constant memory across iterations.

On the single-state URM backbone, InfiLoop reached 84.3% versus 75.9% for our reproduction. On MATH-500 with Qwen3-1.7B-Base, increasing inference depth from three to five loops raised accuracy from 44.2% to 45.8% for InfiLoop, while the standard loop fell from 43.2% to 40.4%.

We’d welcome your thoughts on whether these results address your questions.

Best regards,
The Authors

Reviewer_1vpnOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The authors argue that loop transformers should not just substitute an old recurrence state by the new recurrence state (called "carry-last rule" by the authors). The reason is, that already found useful states could be lost. Therefore, they suggest an adapted architecture InfiLoop, which essentially seems to be a weighted average over past states. The weighting is realized by an attention mechanism, which includes learnt queries and a term which incorporates time differences. The authors try to show advantages of their method by experiments, e.g., by demonstrating effects of small perturbations and they argue that InfiLoop might less likely converge away from useful states, while for TRM this might happen. They also have other experiments and ablations, investigating the behavior of their architecture.

Critical strengths
  • relevant topic, they identify: It seems obvious that substituting the whole old state by a new one might lead to instabilities. In the perturbation experiment, they try to demonstrate this.
  • in principle simple modification to avoid "converging away"-problems: An attention-based average seems both to be simple, but powerful.
Critical weaknesses
  • most critical are their comparisons; The most important method, to which authors seem to compare to, seems to be TRM. Therefore, to understand, whether the aggregation is actually useful, requires an ablation of TRM regarding only aggregation. However, the authors seem to use depthwise 3 × 3 convolution. This seems to be more close to URM. According to Table 1 InfiLoop seems to be worse than ARC-1 and ARC-2 compared to URM. However it does not come out very clearly from the paper, that actually this is the much more important comparison than the comparison to TRM due to the usage of convolution. The subsequent experiments then do not compare to URM, but mainly to TRM. The authors therefore have to decide: Do they want to compare their method to URM or do they want to compare it to TRM. As it currently seems, most of the experiments do not make too much sense, since apples (TRM) and oranges (ablated URM-like method = InfiLoop) are compared.

  • quite some unclearness in writing, e.g. no pseudocode for algorithm; therefore it is e.g. not completely clear, how the weighting is trained; it would also be helpful to better understand terminology of the authors, like what they consider as "supervision segments"; computational complexity analysis should also consider the number of tokens

Questions for authors

see critical weakness on comparisons: comparing TRM with an ablated URM-like method is not acceptable interesting would be TRM with the authors' aggregation, but without any other changes interesting would also be URM with the authors' aggregation, but without any other changes best would be to have both: TRM with the author authors' aggregation and URM with the authors' aggregation

8 comments
AuthorsOpenReview ↗

Response to Reviewer 1vpn [1/3]

W1 (part 1). "most critical are their comparisons; The most important method, to which authors seem to compare to, seems to be TRM. Therefore, to understand, whether the aggregation is actually useful, requires an ablation of TRM regarding only aggregation."

A1. As you requested, we provide the ablation of TRM with only the residual connection replaced by InfiLoop. We take TRM-Att with its block (no convolution), recursion schedule, training schedule and evaluation, and replace carry-last with InfiLoop on both of TRM's recurrent states: the latent state z, updated n=6 times per recursion, and the answer state y, updated once per recursion, with T=3 recursions per supervision step. Nothing else changes. TRM-Att at the training depth is the number reported by its authors.

Sudoku-Extreme (full test set, 422,786 puzzles)H3 L6 (training depth)
TRM-Att74.7
TRM-Att + InfiLoop81.9

With only the residual connection changed, InfiLoop improves TRM-Att from 74.7% to 81.9% at the training depth.

The same improvement holds on the convolutional Sudoku block of the paper. We trained a carry-last model that is identical to InfiLoop in every other respect: the same two-layer block with the same convolution, the same nested recurrence, data, optimizer, number of steps and evaluation script. The only difference is whether consecutive iterations are connected by carry-last or by InfiLoop.

Connection (full test set, 422,786 puzzles)H3 L6 (training depth)H8 L6H12 L12H32 L48
Carry-last76.880.683.987.3
InfiLoop83.088.892.695.8

InfiLoop is 6.2% higher at the training depth and 8.2% to 8.7% higher at the larger depths. At H8 L6 (1,344 layers), InfiLoop already exceeds carry-last at H32 L48 (37,632 layers). Both models contain the convolution, so the difference comes from the connection.

We also evaluated this carry-last model under the depth-scaling protocol of Figure 3 (H6 L12, more segments, the same 32,768 test puzzles):

Connection (Figure 3 protocol)936 layers1,872 layers24,960 layers
Carry-last80.082.784.8
InfiLoop86.590.992.74

InfiLoop at 936 layers is above carry-last at 24,960 layers. In summary, our residual connection alone improves TRM-Att by 7.2% at the training depth, and on the convolutional backbone it is ahead of carry-last at every evaluated depth.

AuthorsOpenReview ↗

Response to Reviewer 1vpn [2/3]

W1 (part 2). "However, the authors seem to use depthwise 3 × 3 convolution. This seems to be more close to URM. According to Table 1 InfiLoop seems to be worse than ARC-1 and ARC-2 compared to URM. However it does not come out very clearly from the paper, that actually this is the much more important comparison than the comparison to TRM due to the usage of convolution. The subsequent experiments then do not compare to URM, but mainly to TRM. The authors therefore have to decide: Do they want to compare their method to URM or do they want to compare it to TRM. As it currently seems, most of the experiments do not make too much sense, since apples (TRM) and oranges (ablated URM-like method = InfiLoop) are compared."

Q. "see critical weakness on comparisons: comparing TRM with an ablated URM-like method is not acceptable / interesting would be TRM with the authors' aggregation, but without any other changes / interesting would also be URM with the authors' aggregation, but without any other changes / best would be to have both: TRM with the author authors' aggregation and URM with the authors' aggregation"

A2. Our backbone is TRM, not an ablated URM. It is built on the public TRM codebase and keeps TRM's two recurrent states, the latent z and the answer y, where z is updated n times inside each of the T updates of y, its two-layer shared block, deep supervision and 7M parameters. URM is a single-state universal transformer with a four-layer shared block and 14M parameters. The convolutions also differ. On Sudoku, ours reshapes the 81 cells to the 9×9 board and applies one 3×3 depthwise convolution per block application, so that each cell mixes with its 8 adjacent cells on the board. URM's convolution is placed inside the MLP of every layer, after the SwiGLU gate, and is a 1D convolution of kernel size 2 along the flattened token sequence.

We now provide both ablations the reviewer asks for: TRM with InfiLoop (A1) and URM with InfiLoop. For URM, we take the official URM code and training recipe and replace the carry-last update between loop iterations with InfiLoop. Nothing else changes. URM is the number reported by its authors.

Sudoku-Extreme, full test setExact accuracy
URM77.6
URM (Reproduced by us)75.9
URM + InfiLoop84.3

InfiLoop improves URM from 77.6% to 84.3% without any other change. Together with A1, the connection improves both architectures.

The convolution is used only on Sudoku-Extreme. On Maze-Hard, ARC-AGI-1 and ARC-AGI-2, InfiLoop uses the stock two-layer TRM block without any convolution, the same block as TRM. The architecture paragraph of Section 4.2 describes the Sudoku-Extreme model, whose convolution acts on the 81 board cells (Appendix A). We have stated the block per benchmark in Section 4.2 of the revised PDF (highlighted in blue). On Sudoku-Extreme, A1 compares InfiLoop with carry-last on the same convolutional block, so only the connection differs.

In Table 1, InfiLoop with 7M parameters reaches 97.9% on Sudoku-Extreme against 77.6% for URM with 14M parameters, and 95.8% at a single fixed depth (A1). On ARC-AGI, URM uses twice the parameters, and among the 7M models InfiLoop is best on all four benchmarks. We have discussed URM's architecture and scale in Section 4.3 of the revised PDF (highlighted in blue).

AuthorsOpenReview ↗

Response to Reviewer 1vpn [3/3]

W2. "quite some unclearness in writing, e.g. no pseudocode for algorithm; therefore it is e.g. not completely clear, how the weighting is trained; it would also be helpful to better understand terminology of the authors, like what they consider as "supervision segments"; computational complexity analysis should also consider the number of tokens"

A3. Good points! We have added the following pseudocode and definitions to Appendix E of the revised PDF (highlighted in blue).

The only learned parameters of InfiLoop are a pseudo-query qq and a decay β=e−λ\beta=e^{-\lambda} per recurrent axis, parameterized as β=sigmoid(ℓ)\beta=\mathrm{sigmoid}(\ell) with β0=0.55\beta_0=0.55. They are trained end to end with the rest of the model by the task loss, without any auxiliary loss.

# per token; z: candidate state, h: state passed to the next iteration
# TRM notation: h_L = latent z, h_H = answer y, L = n, H = T, segment = supervision step
score(z):  e = exp(clip(q · RMSNorm(z), -8, 8))
init(z0):  N = score(z0) * z0;  D = score(z0)
step(z):   N = β * N + score(z) * z
           D = β * D + score(z)
           return h = N / D

# one supervision segment
for each outer cycle (H times; the first H-1 without gradient):
    re-initialise the inner accumulators (N_L, D_L) from h_L
    for each inner update (L times):
        h_L = step_L( f(h_L, h_H + x) )
    h_H = step_H( f(h_H, h_L) )
loss = task_loss(head(h_H), y)
carry (h_H, h_L, N_H, D_H) to the next segment, detached

A supervision segment follows the deep supervision of TRM. It is one full forward pass of H outer cycles, each with L inner updates and one outer update. A loss is computed at the end of every segment, and the recurrent state is detached and passed on as the initial state of the next segment. Our Sudoku model is trained with 12 segments of 21 block applications each, and gradients flow through the last 7.

With sequence length nn and width dd, one block application costs O(n2d+nd2)O(n^2d + nd^2). InfiLoop costs O(nd)O(nd) compute and O(nd)O(nd) persistent memory per iteration, independent of the number of iterations TT. Full Attention Residuals stores and reads all previous outputs, which costs O(tnd)O(tnd) at step tt and O(T2nd)O(T^2nd) in total. For Sudoku:

Per puzzle (n=97n=97, d=512d=512)Compute per block applicationPersistent state at 24,960 layers
Transformer block1.36 GFLOPs–
InfiLoop connection0.1 MFLOPs for the content score plus 0.35 M elementwise operations (together 0.03% of the block)388.8 KiB of accumulators on both axes (582.8 KiB with the recurrent states), independent of depth
Full Attention Residualsgrows linearly with tt≈1.15 GiB (12,480 outputs × nn × dd, bf16)

Appendix A gives the per-operator FLOP and memory counts. We have added the nn-dependent formulas to Section 3.4 of the revised PDF (highlighted in blue).

Reviewer_FzAQOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes InfiLoop, a residual connection for looped Transformers that combines content-dependent weighting with exponential temporal decay. An exact streaming recurrence aggregates previous states with constant persistent memory. The method aims to prevent later iterations from overwriting useful computations and reports improvements on Sudoku, Maze, and ARC-AGI benchmarks.

Critical strengths

1.The paper addresses a meaningful limitation of recurrent reasoning: additional computation can undo previously correct solutions.

  1. The proposed mechanism is lightweight, clearly formulated, and supports constant-memory aggregation.
Critical weaknesses
  1. The benefit over simpler learned gates remains unclear. Equation 5 is an adaptive interpolation between the previous state and the new candidate, closely related to gated recurrent updates. The content-score ablation establishes an advantage over content-independent averaging, but does not establish the benefit of the proposed history-dependent gate parameterization. A comparison with a simple learned sigmoid gate under the same backbone and training budget would substantially strengthen the contribution.

2.The relationship between the main results and the scaling experiments is insufficiently specified. Table 1 reports 97.9% Sudoku accuracy, whereas the depth-scaling experiment reaches 92.74% at its largest budget. Different evaluation subsets or inference configurations could explain this gap, but their relationship should be made explicit. Although Table 1 appropriately acknowledges that systems use their respective inference procedures, these comparisons do not fully isolate the contribution of the proposed connection.

Questions for authors

1.How does InfiLoop compare with a simple learned sigmoid gate when the backbone, training procedure, and inference budget are matched?

2.How many previously unsolved puzzles are first solved at the largest inference budgets? Is a similar scaling benefit observed on ARC-AGI?

3 comments
AuthorsOpenReview ↗

Response to Reviewer FzAQ [1/2]

W1. "The benefit over simpler learned gates remains unclear. Equation 5 is an adaptive interpolation between the previous state and the new candidate, closely related to gated recurrent updates. The content-score ablation establishes an advantage over content-independent averaging, but does not establish the benefit of the proposed history-dependent gate parameterization. A comparison with a simple learned sigmoid gate under the same backbone and training budget would substantially strengthen the contribution."

Q1. "How does InfiLoop compare with a simple learned sigmoid gate when the backbone, training procedure, and inference budget are matched?"

A1. We trained two sigmoid gates that replace InfiLoop with the update h←h+g (z−h)h \leftarrow h + g\,(z-h): gate A uses g=σ(w⋅RMSNorm(z)+b)g=\sigma(w\cdot\mathrm{RMSNorm}(z)+b), and gate B also reads the current state, g=σ(w⋅RMSNorm(z)+u⋅RMSNorm(h)+b)g=\sigma(w\cdot\mathrm{RMSNorm}(z)+u\cdot\mathrm{RMSNorm}(h)+b). Both gates are applied to the same two axes as InfiLoop, and gg is initialized at 0.45, InfiLoop's initial steady-state equal-weight step size 1−β01-\beta_0. Backbone, data, optimizer, training steps, checkpoint step and evaluation are those of Table 2.

Connection (7,680 puzzles)H3 L6 (training depth)H32 L6Gain from depth
Gate A80.782.9+2.2
Gate B81.684.0+2.4
InfiLoop82.189.7+7.6

At the training depth the three are within 1.5 points. With more test-time depth the gates gain 2.2 and 2.4 points and InfiLoop gains 7.6. A gate sets its step from the current step alone. InfiLoop compares the new weight with the accumulated history weight, so once high-scoring states have accumulated, later low-scoring candidates cannot overwrite them. Figure 4 shows this behavior: γt\gamma_t stays near 0.40 during search, peaks at 0.91 when the puzzle is solved, settles to the equal-weight value on later valid states, and low-scoring erroneous states receive small weights.


W2. "The relationship between the main results and the scaling experiments is insufficiently specified. Table 1 reports 97.9% Sudoku accuracy, whereas the depth-scaling experiment reaches 92.74% at its largest budget. Different evaluation subsets or inference configurations could explain this gap, but their relationship should be made explicit. Although Table 1 appropriately acknowledges that systems use their respective inference procedures, these comparisons do not fully isolate the contribution of the proposed connection."

A2. The numbers differ in evaluation set, loop schedule and halting. We have stated all three explicitly at the beginning of Section 4 of the revised PDF, and in the caption of Table 1, Sections 4.3 and 4.4, and Appendix A (highlighted in blue).

At a single fixed depth, InfiLoop reaches 95.8% on the full test set of 422,786 puzzles, above FPRM (94.2%) and EqR (93.0%) in Table 1.

92.74% (Figure 3) is the depth-scaling study. It evaluates the three models at matched layer budgets up to 24,960 layers on a fixed sample of 32,768 test puzzles.

97.9% (Table 1) is the complete system on the full test set. Inference runs (H, L) budgets in increasing order of compute and stops at the first output that satisfies the Sudoku constraints (every row, column and box valid and consistent with the givens).

The contribution of the connection is isolated by the same-backbone comparison, where only the residual connection changes:

Connection (full test set)H3 L6 (training depth)H8 L6H12 L12H32 L48
Carry-last76.880.683.987.3
InfiLoop83.088.892.695.8
AuthorsOpenReview ↗

Response to Reviewer FzAQ [2/2]

Q2. "How many previously unsolved puzzles are first solved at the largest inference budgets? Is a similar scaling benefit observed on ARC-AGI?"

A3. On the 32,768 puzzles of the depth study, 619 puzzles are first solved beyond 1,872 layers, 4 of them between 14,977 and 24,960 layers, and no puzzle solved at a smaller depth is lost at the largest depth. Added depth solves new puzzles rather than flipping puzzles between solved and unsolved.

LayersFirst solved in this rangeCumulative accuracy (%)
≤ 1,87229,77090.85
1,873 – 3,74451392.42
3,745 – 7,4888192.66
7,489 – 14,9762192.73
14,977 – 24,960492.74
Solved earlier, lost later0–

On ARC-AGI, accuracy also increases with test-time depth. ARC-AGI follows the standard TRM evaluation, which runs every task over many augmented views on grids of up to 30×30 cells and votes over them. We evaluated the same checkpoints with more outer cycles at test time:

ARC-AGI (pass@2)H3 L6 (training depth, Table 1)H6 L6H12 L6
ARC-AGI-149.149.852.4
ARC-AGI-213.613.814.2

At the training depth, InfiLoop reaches 49.1% on ARC-AGI-1 and 13.6% on ARC-AGI-2, against 44.6% and 7.8% for TRM and 47.5% and 6.2% for FPRM. With 4× the training depth, it reaches 52.4% and 14.2%.

Dear Reviewer FzAQ,

Thank you again for suggesting the learned-gate comparison. With the same backbone and training budget, increasing test-time depth from H3 L6 to H32 L6 improved Sudoku accuracy by 7.6 points for InfiLoop, compared with 2.2 and 2.4 points for the two learned sigmoid gates.

We also clarified the two Sudoku results you asked about. The 97.9% result uses a sequence of inference budgets on the full test set, stopping at the first valid solution. The 92.74% result comes from the matched-depth study on 32,768 puzzles. Of those, 619 were first solved beyond 1,872 layers, including four in the final depth interval. At four times the training depth, ARC-AGI pass@2 also rose from 49.1% to 52.4% on ARC-AGI-1 and from 13.6% to 14.2% on ARC-AGI-2.

We’d welcome your thoughts on whether these comparisons address your questions about gating and test-time depth.

Best regards,

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.