- Rating
- 3
- Confidence
- 1
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper argues that tiny recursive models (TRMs) underuse their recursion depth because their narrow carried states saturate quickly. It adapts Virtual Width Networks to widen these states 5× at nearly constant compute (+5.5% parameters). With 7M-parameter models, this improves ARC-AGI-1 pass@2 by 7.1 points, ARC-AGI-2 by 3.8, and Sudoku-Extreme by 11.2.
Critical strengths
- Simple, cheap, and consistent gains. Widening the carried states adds only 5.5% parameters, no extra depth, and is function-equivalent to TRM at init, yet improves all three benchmarks at every pass@k and nearly every checkpoint.
- Useful diagnosis of the TRM family. Showing that 4× more depth does not help and that halting collapses to 1–2 of 16 steps is a meaningful finding for a line of work that has mostly scaled iteration.
- Transparent evaluation. The non-standard cumulative-ensemble pass@k is clearly specified, top-2 scores are independently re-derived, and the failure to reproduce TRM's ARC-AGI-2 number is reported openly.
Critical weaknesses
-
The central mechanism is asserted, not measured. The fixed-point / saturated-state explanation is never tested directly (e.g., iteration-to-iteration state change or effective rank of z_H, z_L), and the paper's own data cut against it: the wide model halts earlier (1.19 vs. 1.31 steps on ARC-1), and in Fig. 7 its pass@100 rises only 0.75 points from budget 4 to 16 while the pass@1 margin shrinks. The title claim that width makes depth useful is therefore not supported.
-
The gain is not cleanly attributed to width. There is no parameter-matched baseline that simply widens D, no sweep over n, and no static-vs-dynamic GHC ablation, so the improvement may come from learned GHC routing rather than state capacity. VWN is also tested only at H6/L9, and all results are single-seed (the ARC-AGI-2 gap is 16 vs. 9 grids out of 172).
Questions for authors
- Can you report FLOPs, memory, and wall-clock for both models? "Nearly constant compute" is currently unquantified, and the runs used different GPU counts.
- Your ARC-AGI-2 TRM baseline (5.28) is below the published 7.8. What explains the gap, and does VWN still help over a baseline that matches the published number?
- Halting-head accuracy on ARC drops from 74.3% to 63.1%, and q is used to rank candidates. How much does this affect pass@k?
- Can you report standard single-checkpoint pass@2 as the main ARC metric, rather than the cumulative 10-checkpoint ensemble?
- Both ablations in Table 7 are partial runs. Can you complete them?
- What does this paper add beyond VWN (Seed et al., 2025) and the memory–depth trade-off in Sapunov (2026)?
Thank you for the review. Short answers below. Table 5 is in App. D and Table 7 in App. G.
W1 (mechanism). 4x more depth gives TRM no pass@1/2 gain (Sec. 6.1, Fig. 5). Its pool is flat after 4 of 16 steps (Fig. 1), and one wide step beats every narrow budget (Table 5). We will add per-step state change and effective rank of both carried states in the final version.
W1 (halting). Steps used (Table 3) is a training statistic. The head halts once the answer is right. Fewer steps means the wide model gets there sooner, as more capacity per step predicts (Sec. 6.2). Held-out usefulness of later steps is tested with halting disabled in Sec. 6.3.
W1 (Fig. 7). Table 5: from budget 4 to 16 TRM's pass@100 gains nothing, the wide model gains 0.75. From 1 to 16 the gains are 5.50 vs. 3.62. The pass@1 margin dips only from budget 1 to 4 as TRM catches up, then grows again (7.00, 7.24, 7.62).
W2 (attribution). VWN helps at two depths, H6/L9 on ARC (Table 1) and H3/L6 on Sudoku (Table 2). A parameter-matched TRM would only move D from 512 to about 526, since parameters grow as D^2. We will add that baseline, a sweep over n, static vs. dynamic GHC and more seeds in the final version.
Q1 (compute). Backbone FLOPs are unchanged. Attention and FFN run at D = 512, same sequence length. The overhead is one small batched contraction per sublayer (Sec. 4, Cost). We will report FLOPs, memory and wall-clock on identical hardware in the final version.
Q2 (ARC-AGI-2 baseline). Our baseline is the deep H6/L9 config, not the upstream H3/L4 (Sec. 5.1). At H3/L4 we match the published ARC-AGI-1 number, and VWN+TRM (9.03) beats the published 7.8 (Table 1). We will add an H3/L4 pair on ARC-AGI-2 in the final version.
Q3 (halting head). q is only the tie-breaker after vote count (Sec. 3, App. A). If anything a weaker q hurts VWN+TRM, yet its largest ARC-AGI-1 gain is at k = 1 (+9.0, Table 1). We will add a vote-only ranking for both models in the final version.
Q4 (single checkpoint). It is in Table 5 at budget 16, 52.50 vs. 46.75 pass@2 on ARC-AGI-1 (+5.75). Table 1 follows the upstream evaluator (Sec. 3, App. A). We will report single-checkpoint pass@2 next to it for both ARC benchmarks in the final version.
Q5 (partial ablations). Yes, we will train both to the full schedule for the final version. At matched evaluations pseudo-tokens are 1.4 pass@2 below GHC and 1.5 above TRM (Table 7).
Q6 (prior work). VWN was shown in feed-forward LM pretraining. We bring it to weight-tied recursion. Sapunov adds memory tokens on Sudoku. We widen the residual stream itself and test on ARC (Sec. 2). New: depth alone does not help TRM (Sec. 6.1), while state width decides how much is used (Sec. 6.3).