acceptodds
Under review as a conference paper at ICLR 2027

Test-Time Generation

Abstract

Long-context models increasingly rely on fast-weight and test-time-training memories—state-space and linear-attention layers that compress a growing context into a fixed-size state and recall it in linear time. But they read that state through deterministic regression, which under squared loss maps a query key to the conditional mean of its bindings and averages competing values away. Our contribution is to recast the readout, not the memory, as conditional generation: casting fast weights as a conditional associative memory, we show that a key-conditioned denoising readout augments key-space similarity with a value-space posterior term—the term a squared-error endpoint map discards. We realize this as Test-Time Generation ( TTG ), a drop-in readout that leaves the write operator and fixed-size state untouched and so stays orthogonal to the memory architecture: online updates supervise noisy versions of the observed bindings, and a query is answered by sampling a short denoising trajectory rather than regressing to a point. A compressed memory then need not recall by averaging to a single endpoint, but can use the cue to sample the binding it stored. A controlled two-values-per-key probe isolates the mechanism: no deterministic readout—not even one granted a much larger memory—recovers both competing values on any key, whereas the generative readout does at low retrieval load, and freezing its sampling noise removes the effect. The same readout improves both long-context language modeling and novel view synthesis over deterministic baselines, without enlarging the memory or its peak footprint, at a modest readout cost. On language modeling a single readout step already suffices for the gain, so readout depth is matched to training rather than scaled at test time. TTG thus turns a fixed limitation of test-time memory into a readout we can design.

Primary area:
generative models
Keywords:
test-time training, fast weights, conditional generation, rectified flow, in-context retrieval, long-context language modeling

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 2 reviews, above 85% of the others.
Reviewer_Fkk9OpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies generative retrieval from online fast weight memory. It proposes Test-Time Generation (TTG), which replaces deterministic regression readouts with query-conditioned denoising while preserving the fixed-size memory state. Motivated by the inability of deterministic squared-error regression to represent multiple probable values associated with the same key. Experiments evaluate long-context language modeling across 124M–3B scales, novel view synthesis, and controlled two-value associative recall.

Critical strengths

The paper presents an interesting approach to improving memory retrieval independently of the memory update architecture. The integration of conditional generation with online fast-weight memory is a meaningful extension of prior test-time training methods. The experiments cover multiple model scales and modalities, demonstrating improvements in long-context retrieval and novel view synthesis. The controlled two-value recall experiment also provides useful evidence that stochastic readouts can recover multiple stored associations under limited memory load.

Critical weaknesses
  1. Insufficient evidence connecting the proposed generative retrieval to downstream improvements.
  • The central motivation is that iterative generative readouts overcome the conditional-mean limitation of deterministic memory retrieval. However, the headline language-modeling improvements are obtained with a single readout step, where the idealized Bayes-optimal denoiser returns the conditional mean rather than separating modes.

  • While TTG improves long-context retrieval, its language modeling improvements are relatively modest and inconsistent across scales. In particular, deep-context PTL becomes worse than LaCT at 3B. Furthermore, evaluation is largely restricted to Book-3 tail loss, without independent language modeling benchmarks such as WikiText or LAMBADA. This limits the evidence that the proposed readout improves general language modeling capabilities beyond long-context retrieval.

  1. Confounded comparisons.
  • In the 124M family, changes in conditioning architecture and parameter count substantially affect performance, making it unclear how much of the reported improvement is attributable to the proposed mechanism. Moreover, the 760M default configuration outperforms LaCT in the main comparison but not in the separate implementation cohort. More consistently matched, parameter-controlled comparisons and independent-seed evaluations would strengthen the evidence for the proposed readout's effectiveness.
Questions for authors
  1. Please provide more direct evidence that the proposed multimodal generative retrieval mechanism contributes to the downstream improvements? For example, a controlled comparison between single-step and multi-step readouts under matched training configurations, together with an analysis connecting mode recovery to downstream performance, would be helpful.

  2. Please provide additional language modeling evaluations on standard benchmarks such as WikiText/LAMBADA, comparing TTG against LaCT under matched settings. This would help clarify whether the reported retrieval improvements generalize to broader language modeling capabilities.

  3. Please provide a deterministic baseline matched to the 124M TTG configuration (TE + per-block AdaLN, 172.7M parameters), under identical training and evaluation settings. Additionally, multiple-seed results with mean + std would help check robustness of the reported gains.

I'm open to increasing my score if authors address at least two of these three concerns.

6 comments

We thank the reviewer for the insightful suggestion, and we would like to provide a few clarification.

W1.1 & Q1: The downstream improvements lack a clear connection to the claimed multimodal generative retrieval mechanism, as the main gains arise from single-step readouts that theoretically recover only the conditional mean, motivating controlled comparisons with multi-step readouts and analysis of how mode recovery affects downstream performance.

A1: To clarify the connection between iterative generative retrieval and downstream performance, we provide additional evidence from controlled mode-recovery experiments, trained readout-depth comparisons, and downstream analyses of multiple stored values.

  1. Controlled evidence of iterative mode recovery.

Our controlled two-value recall experiments (Section 4.4, Appendix E) directly examine the effect of inference readout depth (S) on recovering multiple values associated with the same key.

At (N=4), increasing (S) from 1 to 4 improves two-value coverage from 22.2% to 65.9%. At (N=16), increasing (S) from 1 to 8 improves coverage from 0% to 34.4%, while deterministic controls retain zero coverage. These experiments demonstrate that iterative generative refinement can improve recovery of distinct stored values under controlled conditions.

  1. Effect of trained readout depth on downstream performance.

To examine whether deeper generative readouts also improve downstream retrieval, we compare 3B TTG models with trained readout depths (d=1) and (d=2), holding write repeats fixed at (r=1). We evaluate both NIAH-MultiValue, which requires recovering multiple values associated with the same key, and overall RULER accuracy across 13 tasks.

Table 1. NIAH-MultiValue (MV) and overall RULER accuracy (%) for 3B TTG models.

MethodMV 4KMV 8KMV 16KMV 32KMV Avg.RULER 4KRULER 8KRULER 16KRULER 32KRULER Avg.
TTG d140.830.220.014.426.459.1438.3425.6916.3334.87
TTG d257.546.132.717.038.359.2541.0627.8818.0136.55

Increasing trained readout depth from (d=1) to (d=2) improves NIAH-MultiValue accuracy from 26.4% to 38.3% (+11.9 points) and overall RULER accuracy from 34.87% to 36.55% (+1.68 points), with improvements at all four context lengths.

The substantially larger gain on NIAH-MultiValue suggests that deeper trained readouts are particularly beneficial for retrieving multiple values associated with the same key.

  1. Evidence of multiple stored values in downstream predictions.

We further investigate whether pretrained language models can access information about multiple stored values beyond the local attention window, using pretrained 760M TTG (r1d1) and LaCT checkpoints.

First, we evaluate how often a stored value outside the local attention window appears in the model's response on RULER-based single-value and four-value retrieval tasks.

Table 2. Fraction of responses containing an out-of-window stored value (%).

ModelSingle 8KSingle 16KSingle 32KMulti 8KMulti 16KMulti 32K
LaCT24.10.00.05.60.30.0
TTG69.02.80.028.26.60.8

TTG shows substantially stronger out-of-window retrieval, particularly at 8K, improving single-value retrieval from 24.1% to 69.0% and four-value retrieval from 5.6% to 28.2%. The advantage persists at 16K for four-value retrieval (6.6% vs. 0.3%), although both models degrade at longer contexts.

To examine whether the decoder retains information about multiple stored values, we further analyze its output probability distribution at the first digit of the answer, using 16K documents in which all four values lie outside the local attention window.

Table 3. First-digit probability analysis on 16K NIAH-MultiValue documents with four out-of-window values.

ModelProbability on stored leading digits (%)Unrelated-document control (%)Stored leading digits with probability ≥0.1
LaCT36.131.90.89
TTG (r1d1)76.937.32.32

TTG assigns 76.9% probability mass to the stored values' leading digits, compared with 36.1% for LaCT. Replacing the target digit set with that from unrelated documents reduces these probabilities to 37.3% and 31.9%, respectively. Furthermore, TTG assigns probability ≥0.1 to an average of 2.32 stored leading-digit candidates, compared with 0.89 for LaCT.

These results provide complementary evidence that TTG's decoder can access information associated with multiple stored values, even when those values are outside the local attention window.

  1. Reconciling single-step improvements with iterative generative retrieval.

We emphasize that improved retrieval and explicit multimodal recovery are related but distinct outcomes. Our single-step language-modeling results demonstrate practical improvements under matched numbers of readout evaluations, without establishing mode separation. The controlled two-value experiments demonstrate that iterative refinement can recover distinct modes, while the additional downstream experiments show that deeper trained readouts improve multi-value retrieval and that TTG provides stronger access to multiple stored values.

Overall, the additional results establish three complementary findings: (i) iterative generative refinement enables stronger mode recovery in controlled memories; (ii) increasing trained readout depth improves downstream multi-value and overall RULER accuracy; and (iii) pretrained TTG exhibits stronger downstream access to multiple stored values.

W1.2 & Q2: TTG shows limited and inconsistent language modeling improvements over LaCT, motivating additional evaluations on standard benchmarks such as WikiText and LAMBADA to assess whether its retrieval gains generalize to broader language modeling capabilities.

A2: To address this concern, we conducted additional evaluations using the 760M model family, comparing TTG against LaCT under matched training and evaluation settings. The new results are summarized in Tables G1–G4.

First, TTG achieves consistently lower perplexity on independent language modeling benchmarks (Table G2). On WikiText and LAMBADA, TTG improves perplexity from 21.62 to 21.54 and from 16.36 to 16.16, respectively. Similar improvements are observed on long-context language modeling datasets, including PG-19 (14.87 → 14.78) and GovReport (6.67 → 6.61), both evaluated on contexts exceeding 30K tokens. Although these improvements are modest, they are consistent across all four datasets, providing evidence that TTG's benefits are not restricted to Book-3 tail loss.

Second, the improvements extend to broader downstream language understanding tasks (Tables G1, G3, and G4). On LongBench v1, TTG outperforms LaCT in five of six task categories, with particularly notable improvements in single-document QA (13.3 → 17.8) and multi-document QA (16.2 → 22.5). On LongBench v2, accuracy increases from 28.0 to 29.6. TTG also improves the average score on six recall-intensive tasks from 45.5 to 47.5, outperforming LaCT on every task. Furthermore, across seven zero-shot commonsense reasoning benchmarks, TTG increases the average accuracy from 43.6 to 44.6.

Taken together, these additional evaluations suggest that TTG's advantages extend beyond synthetic long-context retrieval to standard language modeling, document understanding, and downstream reasoning tasks. The new evidence supports broader generalization of TTG's readout improvements beyond the original retrieval-focused evaluation.

Table G1. LongBench v1 (by category) and LongBench v2

ModelSingle-doc QA ↑Multi-doc QA ↑Summarization ↑Few-shot ↑Synthetic ↑Code ↑LongBench v2 ↑
Transformer12.618.311.323.02.734.327.0
Transformer-SWA16.119.37.920.20.731.328.0
GDN-SWA13.413.99.422.70.933.822.7
DeltaNet-SWA15.819.38.129.40.234.728.0
Mamba-2-SWA15.016.110.528.41.835.320.5
Lact13.316.28.831.32.936.028.0
TTG17.822.511.734.51.238.029.6

Table G2. Short- vs. long-context perplexity

ModelWikiTextLAMBADAPG-19 30K+GovReport 30K+
Transformer23.5022.2315.436.48
Transformer-SWA22.8318.9215.717.10
GDN-SWA22.5020.2015.337.02
DeltaNet-SWA22.5218.1315.326.97
Mamba-2-SWA21.7617.1215.156.84
Lact21.6216.3614.876.67
TTG21.5416.1614.786.61
Reviewer_tRfDOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This manuscript studies memory readout from fixed-size fast-weight states. Existing methods have written context into network weights through online updates at test time. The author note that when the same query corresponds to multiple possible values, the optimal prediction of the squared error target may average distinct memory contents. This manuscript introduce Test-Time Generation (TTG), it uses conditional denoising to write context and generate query-conditioned features from noise.

Critical strengths
  1. This manuscript identify a readout bottleneck in fast-weight memory: even when the state preserves distinct values compatible with a query, deterministic regression may collapse them into a conditional mean.

  2. Replacing deterministic regression with conditional denoising is an interesting approach and it yields positive results in long-context language modeling and novel view synthesis (NVS).

Critical weaknesses
  1. The cost of memory updates during test-time training is not clearly quantified such as the runtime or GPU memory usage of online memory updates under different settings, and it also lacks a direct comparison with LaCT

  2. The TTG Model of 124M model in Table 1 and Figure 3 actually contains 172.7M parameters. And conditional ablation shows that other variables, such as AdaLN, also have a impact on performance. Thus, the gains cannot be clearly attributed to generative readout rather than conditioning or extra parameters.

Questions for authors
  1. What are the runtime and GPU memory costs of online memory updates under different input sizes and configurations?
  2. Why do TTG and LaCT use different view-sampling sequences in the comparison in Section 4.3? Why not using the same sequences to control for differences in the input and target views?
3 comments

We thank the reviewer for the insightful suggestions, and we would like to make a few clarifications.

W1 & Q1: What are the runtime and GPU memory costs of online memory updates under different input sizes and configurations?

A1: We provide a direct comparison with LaCT across model scales, input Lengths, reporting runtime or GPU memory usage of online memory updates.

Table. Per-update runtime or GPU memory usage of online memory updates across context lengths. TTG uses (r=d=1).

Model124M760M
Input Length32K128K32K128K
LaCT time (ms)29.428.759.965.9
TTG time (ms)27.226.853.761.1
LaCT memory (GiB)0.0710.0710.1450.145
TTG memory (GiB)0.0710.0700.1650.164

Setup. Both methods update all fast-weight layers once per 4,096-token chunk. Measurements include gradient computation and the chunk-boundary update. We report median per-update latency and peak transient GPU memory over three runs on a single H200 using BF16 and the same optimized inference engine.

Overall, TTG maintains LaCT-comparable runtime or GPU memory usage of online memory updates.

W2: The TTG Model of 124M model in Table 1 and Figure 3 actually contains 172.7M parameters. And conditional ablation shows that other variables, such as AdaLN, also have a impact on performance. Thus, the gains cannot be clearly attributed to generative readout rather than conditioning or extra parameters.

A2:

To control for parameter count, we introduce a parameter-matched LaCT baseline (LaCT + readout MLP), which adds a deterministic MLP after LaCT's fast-weight output. This increases its total parameters from 137.3M to 172.67M, closely matching TTG (172.70M). Importantly, all added parameters are slow weights, keeping the fast-weight capacity identical (2.95M).

All models use the same codebase, training protocol (10,240 steps, 32K context), and evaluation pipeline. To assess robustness, we evaluate three independent random seeds (42, 43, and 44) and report the mean ± standard deviation.

Table 1. Parameter-matched multi-seed comparison (mean ± standard deviation).

ModelParams (total / fast-weight)SeedsPTL ↓RULER mean ↑
TTG172.70M / 2.95M42, 43, 443.0987 ± 0.001514.78 ± 2.66
LaCT + readout MLP (parameter-matched)172.67M / 2.95M42, 43, 443.1216 ± 0.006512.91 ± 2.10
LaCT137.30M / 2.95M42, 43, 443.1134 ± 0.004211.44 ± 1.18

TTG retains its advantage under parameter matching and across multiple seeds, achieving lower mean PTL (3.0987 vs. 3.1216) and higher mean RULER accuracy (14.78 vs. 12.91) than the parameter-matched LaCT baseline. TTG also outperforms the original LaCT baseline in both metrics. These results indicate that TTG's improvements cannot be explained solely by increased parameter count.

Regarding conditioning, we acknowledge that AdaLN affects performance, as shown in Figure 5 and Table A2. However, our controlled two-value recall experiments (Section 4.4) provide complementary evidence for generative readout. The matched-noise deterministic control preserves TTG's write objective, conditioning, network widths, and denoising loop, replacing stochastic noise with a learned constant. Under these controlled conditions, TTG achieves 65.9% and 34.4% two-value coverage at N=4 and N=16, respectively, whereas deterministic controls achieve zero coverage.

Together, these results show that TTG's performance gains cannot be attributed solely to additional parameters, while the controlled recall experiments demonstrate a distinct capability of stochastic generative readout under matched conditioning.

Q2: Why do TTG and LaCT use different view-sampling sequences in the comparison in Section 4.3? Why not using the same sequences to control for differences in the input and target views?

A3: The original comparison used archived LaCT and Transformer evaluations with seed 9595, while TTG, DeltaNet, and Gated DeltaNet used seed 0. We agree that view-sampling sequences should be matched for a fair comparison. We have now re-evaluated all five models using the same evaluator and identical seed-9595 view sequences, obtaining the following results

ModelGSO PSNRDL3DV PSNR
TTG24.9316.58
LaCT23.9916.14
Transformer25.3617.41
DeltaNet23.9415.82
Gated DeltaNet24.3216.05

All models are evaluated on the same 1,019 GSO objects and 140 DL3DV scenes, with mean PSNR averaged over 23 prediction positions.

Under this controlled evaluation, TTG retains its improvements over LaCT (+0.94 dB on GSO and +0.44 dB on DL3DV) and outperforms both DeltaNet variants on both datasets.

Thanks to the response. Most of my concerns have been addressed. I will retain my current score.

Comments on the paper

6 comments

We thank the reviewer for raising these important concerns, and we would like to clarify a few points.

W1 & Q1: The contributions of noisy writing, conditioning modules, generative readout, and additional parameters to TTG's performance gains remain unclear, motivating controlled ablations under matched parameter counts, training compute, and implementation settings.

A1: To control for TTG's additional parameters, we introduce LaCT + readout MLP, which adds a deterministic MLP after LaCT's fast-weight output. This increases its total parameters from 137.3M to 172.67M, closely matching TTG (172.70M). All added parameters are slow weights, while the fast-weight capacity remains identical (2.95M).

The comparison uses the same codebase, training protocol (10,240 steps, 32K context, seed 42), and evaluation pipeline. The original LaCT and TTG results are included from Table 1.

ModelParams (total / fast-weight)PTL ↓RULER mean ↑4K8K16K32K
LaCT (paper, Table 1)137.3M / 2.95M3.108812.3624.3414.096.974.06
TTG (paper, Table 1)172.70M / 2.95M3.098517.3828.3619.8013.507.84
LaCT + readout MLP (parameter-matched)172.67M / 2.95M3.121414.9930.6415.268.895.17

TTG retains its advantage under parameter matching, achieving lower PTL (3.0985 vs. 3.1214) and higher mean RULER accuracy (17.38 vs. 14.99, +2.39 points), with larger gains at 16K (+4.61) and 32K (+2.67). The matched LaCT baseline performs better at 4K. Adding the deterministic MLP improves LaCT's mean RULER from 12.36 to 14.99 (+2.63 points), indicating that increased slow-weight readout capacity can improve retrieval.

Our existing experiments provide complementary evidence:

Conditioning (Fig. 5, Table A2): Per-block AdaLN consistently improves language-modeling performance over additive conditioning, confirming that conditioning contributes to TTG's gains.

Stochastic readout (Section 4.4): The matched-noise deterministic control preserves TTG's write objective, conditioning, and model widths, but replaces stochastic noise with a learned constant. TTG recovers both stored values at low and moderate memory loads, whereas deterministic controls have zero two-value coverage. This demonstrates the distinct retrieval behavior of stochastic reading.

W2 & Q2: Can the authors clarify under what conditions the actual gated-MLP fast weights can approximate the idealized discrete key-value mixture and provide empirical evidence, approximation bounds, or explicit recovery guarantees to establish the applicability of the theoretical analysis to the actual TTG model?

A2: We thank the reviewer for highlighting the distinction between the idealized mixture and the learned fast-weight state. Section 3 explicitly introduces the effective bindings D_h and association weights π_i(k;h) as a conceptual abstraction, rather than an explicit key-value table. In the actual model, the state consists of gated-MLP parameters h = (W0, W1, W2), updated online through Eq. (1) using the denoising objective in Eq. (4). The theoretical analysis is applicable at the functional level when (i) online updates preserve distinguishable, query-compatible associations with non-negligible weights π_i(k;h), and (ii) the learned denoiser D_theta sufficiently approximates the conditional posterior mean characterized by Eq. (2).

Under these assumptions, Eq. (3) and Proposition C.4 characterize the conditions favoring retrieval: the expected posterior margin decomposes into a key-space prior ratio and a value-space separation term amplified by the denoising signal-to-noise ratio. Observation C.3 further explains that overlapping components initially favor averaging, whereas decreasing noise allows the posterior to distinguish competing values. Crucially, this mechanism requires the corresponding associations to remain represented in the fast-weight state; stronger denoising cannot recover bindings lost during online writing.

We provide direct empirical evidence using the actual online-updated gated-MLP memory in Section 4.4 (Fig. 8 and Appendix E, Table A5). With two values per key, TTG achieves 65.9% two-value coverage at (N=4, S=4) and 34.4% at (N=16, S=8), measured over 32 stochastic reads, while deterministic controls have zero two-value coverage. Freezing the readout noise also eliminates two-value coverage, and quadrupling the deterministic memory capacity does not recover both values. At (N=64), all methods fail, illustrating the effect of write-side interference.

These results provide empirical evidence that its learned fast-weight state can exhibit the multi-valued retrieval behavior characterized by the idealized analysis under moderate memory loads. We will clarify these functional assumptions and explicitly distinguish the conditional theoretical characterization from the empirical behavior of the actual model.

W3 & Q3: Given that direct evidence for multi-value recovery is limited to a synthetic two-value probe, could the authors evaluate whether repeated stochastic reads can recover multiple plausible answers to the same query in realistic language tasks, beyond the single-step readout used in RULER?

A3: To further investigate multi-value retrieval in realistic language tasks, we provide additional downstream experiments and probability-level analyses.

To further investigate multi-value retrieval in realistic language tasks, we provide additional evidence from trained readout-depth comparisons, out-of-window multi-value retrieval, and decoder probability analyses.

  1. Effect of readout depth on downstream multi-value retrieval.

We compare 3B TTG models with trained readout depths (d=1) and (d=2), holding write repeats fixed at (r=1). We evaluate NIAH-MultiValue, which requires retrieving multiple values associated with the same key, at context lengths of 4K, 8K, 16K, and 32K, alongside overall RULER accuracy across 13 tasks.

Table 1. NIAH-MultiValue (MV) and overall RULER accuracy (%) for 3B TTG models.

MethodMV 4KMV 8KMV 16KMV 32KMV Avg.RULER Avg.
TTG d140.830.220.014.426.434.87
TTG d257.546.132.717.038.336.55

Increasing trained readout depth improves NIAH-MultiValue accuracy from 26.4% to 38.3% (+11.9 points), compared with a 1.68-point improvement in overall RULER accuracy. The larger gain on NIAH-MultiValue suggests that deeper generative readouts are particularly beneficial for retrieving multiple values associated with the same key.

  1. Evidence of multiple stored values in pretrained language models.

We further analyze pretrained 760M TTG ((r=1,d=1)) and LaCT checkpoints on RULER-based single-value and four-value retrieval tasks. Specifically, we measure how often a stored value outside the local attention window appears in the model's response at context lengths of 8K, 16K, and 32K.

Table 2. Fraction of responses containing an out-of-window stored value (%).

ModelSingle 8KSingle 16KSingle 32KMulti 8KMulti 16KMulti 32K
LaCT24.10.00.05.60.30.0
TTG69.02.80.028.26.60.8

TTG demonstrates stronger out-of-window retrieval, particularly in the four-value setting, improving from 5.6% to 28.2% at 8K and from 0.3% to 6.6% at 16K compared with LaCT.

To investigate whether multiple stored values influence the decoder's predictions, we further analyze the output probability distribution at the first digit of the answer using 16K NIAH-MultiValue documents in which all four stored values lie outside the local attention window. We measure the total probability assigned to stored leading digits and the number of stored leading-digit candidates receiving probability ≥0.1. We additionally use leading digits from unrelated documents as a control.

Table 3. First-digit probability analysis on 16K four-value retrieval.

ModelStored leading-digit probability (%)Unrelated control (%)Candidates with probability ≥0.1
LaCT36.131.90.89
TTG76.937.32.32

TTG(r1d1) assigns 76.9% probability mass to stored leading digits, compared with 36.1% for LaCT. Furthermore, TTG assigns probability ≥0.1 to an average of 2.32 stored leading-digit candidates, compared with 0.89 for LaCT. These results suggest that TTG's decoder can simultaneously access information associated with multiple stored values, even beyond the local attention window.

  1. Connection to stochastic mode recovery.

Our controlled two-value experiments (Section 4.4, Appendix E) directly demonstrate that increasing inference refinement steps (S) improves distinct-value coverage: from 22.2% to 65.9% when (S) increases from 1 to 4 at (N=4), and from 0% to 34.4% when (S) increases from 1 to 8 at (N=16), while deterministic controls retain zero coverage.

Together, these results provide complementary evidence that iterative refinement supports mode recovery in controlled memories, while deeper trained readouts and pretrained TTG improve access to multiple stored values in downstream language tasks

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.