acceptodds
Under review as a conference paper at ICLR 2027

Test-Time RL for Diffusion Language Models: Geometry-Guided Selective Updates

Abstract

Test-time reinforcement learning (TTRL) enables models to adapt from unlabeled test-time experience, but agreement among sampled answers alone does not establish which targets are reliable enough to reinforce. We introduce Geo.-LiM, a framework for persistent, label-free adaptation of diffusion language models (dLLMs) that exploits complementary evidence from NFE-matched denoising geometries. Geo.-LiM selectively triggers updates by combining strict pooled-majority support across geometries with novelty relative to a deterministic frozen-reference answer, and reinforces only trajectories supporting eligible candidates under fixed-reference regularization. Across three dLLMs and five benchmarks, Geo.-LiM improves post-adaptation performance over the unadapted backbone in 27 of 30 model-task-length settings. Under the same eight-rollout evidence budget, it also outperforms our consensus-based TTRL baseline in 23 of 30 settings while admitting fewer prompts and requiring fewer optimizer steps in every setting, reducing their aggregate counts by 74.2% and 64.4%, respectively. These results show that denoising geometry provides useful evidence for identifying informative test-time updates, enabling selective and efficient persistent adaptation of dLLMs without target labels.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Diffusion Language Models, Test-Time Reinforcement Learning, Test-Time Adaptation, Online Learning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_azjbOpenReview ↗
Rating
3(+1)
Confidence
2(−1)
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The author proposes a new type of TTRL method for DLMs. This new approach considers different views, which is a natural perspective for DLMs. The method is more effective compared to the selected adapted TTRL approach.

Critical strengths
  1. It provides a good perspective on TTRL: "Which answers are worth an update?". The novelty gate makes sense.

  2. Different views are interesting. This is very natural for DLMs.

  3. Makes a very valid point in terms of consensus during the process and brings a lot of efficiency by reducing prompts admitted.

Critical weaknesses
  1. The presentation can use a lot of improvement. The writing is confusing and extremely unclear. For example, until later sections, readers really cannot know what "geometry" stands for in the paper, although it appears a lot in the introduction. "The corresponding grid totals decrease by 74.2% and 64.4%, respectively." in the abstract is really weird, although the reader can recognize this after reading the paper. Figure 7 is quite messy. Having a clear "Setting/Problem" section can help.

  2. Main results are 1-seeded. This is understandable given the time, but more seeds are definitely expected during the rebuttal phase.

  3. Some details/motivations are missing completely.

Questions for authors
  1. Refer to weaknesses3. Why different blocks? For the fixed 2 in the ablation, why {16,64}? How does randomness help in this?

  2. Single-G = “Direct Vote” with one block size, which isn’t specified?

  3. Does the benefit come from using two distinct geometries per prompt, or from mixing many geometries across the stream? An ablation that keeps one pair across prompts but randomizes it per run would separate the two.

Reviewer_7dMpOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper proposes Geometry-Selective Less-is-More TTRL (Geo.-LiM), a framework for label-free, persistent test-time adaptation of diffusion language models (dLLMs). The method generates multiple responses under different denoising geometries and selectively reinforces candidates that satisfy strict pooled-majority support and differ from a frozen reference prediction. Selected trajectories are used to update a persistent LoRA adapter through positive-only replay with KL regularization. Experiments on three dLLMs and five benchmarks show improvements over the unadapted backbone in 27 of 30 settings and over a consensus-based TTRL baseline in 23 settings, while substantially reducing the number of admitted prompts and optimizer steps.

Critical strengths
  1. The paper addresses a relevant problem of enabling diffusion language models to adapt from unlabeled test-time experience without external reward signals.

  2. The proposed selective-update framework is simple and practical, combining denoising-geometry diversity with reference-relative novelty to filter training targets.

  3. The experiments cover multiple dLLM architectures and mathematical reasoning and code-generation benchmarks, demonstrating potential applicability across tasks.

  4. The paper includes several selection ablations, count-matched controls, and post-hoc analyses to investigate the effects of selective adaptation.

Critical weaknesses

1. Limited methodological novelty and insufficient differentiation from prior work.

The main components of Geo.-LiM are closely related to existing approaches. Majority-vote pseudo-rewards and persistent test-time adaptation have been established in TTRL [R1], while aggregating predictions across different dLLM block schedules has been explored in HEX [R2]. Selective pseudo-labeling and reliability-based filtering have also been investigated in recent TTRL methods, including SCRL [R3] and SR-TTRL [R4]. The proposed method primarily combines these existing ideas with a reference-novelty condition and positive-only replay, rather than introducing a substantially new learning algorithm. Although applying this combination to persistent dLLM adaptation is potentially useful, the paper does not sufficiently establish what fundamentally new methodological insight or capability it provides beyond these prior approaches. More direct comparisons with existing selection strategies would help clarify the contribution.

2. The contribution of geometry-guided evidence is not convincingly established.

The central claim is that denoising-geometry diversity provides useful evidence for selecting reliable updates. However, the proposed eligibility rule ultimately relies on pooled majority voting across eight samples. Since each of the two geometries contributes at most four samples, requiring five votes guarantees only that at least one supporting sample comes from each geometry. This is a relatively weak notion of cross-geometry agreement and does not necessarily indicate consistent support across different denoising factorizations. It remains unclear whether the gains arise from meaningful geometric diversity or simply from stochastic sampling and stricter filtering. Although Table 5 includes geometry-source ablations, they are limited to three settings and allow the selected prompts and update counts to change simultaneously. Consequently, the experiments do not convincingly isolate the specific benefit of geometry diversity over conventional sampling-based consensus.

3. Insufficient comparisons to establish the benefit of persistent adaptation and the proposed selection rule.

The main comparison is against CSA-TTRL, an in-paper consensus-based baseline that differs from Geo.-LiM in several components, including geometry construction, eligibility criteria, and replay objectives. Although additional ablations are provided, they cover only a small subset of the evaluation settings. Moreover, simpler alternatives sometimes outperform Geo.-LiM; for example, Confidence-K achieves +3.2 pp on LLaDA MATH500-256, compared with +1.6 pp for Geo.-LiM. A particularly important missing comparison is inference-only geometry-diverse majority voting under the same evidence budget, without parameter updates. Since the method already generates eight trajectories for every test input, it is unclear whether persistent adaptation provides sufficient additional benefit over directly aggregating these predictions, as in geometry-based test-time scaling [R2]. Without this comparison, the necessity and practical value of the proposed learning procedure are difficult to assess.

4. Insufficient statistical reliability of the reported improvements.

All 30 main experimental settings are evaluated using a single adaptation run. This is a significant limitation because the method involves stochastic trajectory generation, geometry assignment, and parameter updates, and the resulting training signal can be extremely sparse. For instance, Geo.-LiM admits only three prompts on LLaDA HumanEval-256, while several reported improvements correspond to only a handful of net corrected examples. Many performance differences between Geo.-LiM and CSA-TTRL are also below one percentage point. Although the authors report paired evaluation statistics, these do not capture variability across adaptation seeds, sampled trajectories, or prompt orders. Therefore, the reported 27/30 and 23/30 improvements do not establish that the method consistently outperforms the baselines. Multi-seed experiments with uncertainty estimates are necessary to substantiate the empirical conclusions.

5. The post-hoc analysis does not convincingly explain why the proposed selection works.

The paper emphasizes that Geo.-LiM eliminates redundant proposals in the post-hoc analysis (Figure 4 and Appendix Table 10). However, this observation appears to follow largely from the reference-novelty condition itself. For tasks with a unique normalized correct answer, a candidate that differs from an already-correct reference cannot also be correct, so selecting only reference-disagreeing candidates structurally excludes correct-to-correct redundant proposals. Thus, the absence of such proposals is not independent evidence that the method identifies more useful updates. Furthermore, Appendix Table 10 shows that Geo.-LiM does not necessarily select more accurate pseudo-labels than CSA-TTRL, despite achieving better final performance. The actual mechanism underlying these improvements remains insufficiently explained, and the current analysis does not clearly distinguish pseudo-label quality, update sparsity, and cross-prompt parameter effects.

6. Limited evidence for generalizable adaptation and practical efficiency.

The evaluation follows a transductive protocol in which the final adapted model is evaluated on the same inputs used during adaptation. While this is a legitimate TTRL setting, the results do not establish whether the learned adapter improves performance on unseen inputs or supports sustained adaptation across changing tasks. The cross-task retention analysis is limited to three transfer settings and does not consistently favor Geo.-LiM. In addition, the claimed efficiency improvements concern the number of admitted prompts and optimizer-bearing steps, whereas every test input still requires eight evidence rollouts and a reference prediction. Since evidence generation can represent a substantial fraction of the computational cost, reductions of 74.2% in admitted prompts and 64.4% in optimizer steps do not directly establish comparable end-to-end efficiency gains. Evaluation on unseen inputs and measurements of total inference and adaptation costs would strengthen the practical contribution.

7. Limited reproducibility due to the absence of source code.

Although the paper describes the high-level algorithm and reports detailed experimental configurations, no source code is provided with the submission. This is particularly concerning because the main contribution depends on implementation-sensitive components, including geometry-specific denoising schedules, supporting-trace selection, replay objectives, answer normalization, and distributed adaptation. The reported prediction ledgers and configuration descriptions may facilitate verification of aggregate metrics, but they do not enable independent reproduction of the training procedure. Combined with the single-seed evaluation, the absence of an executable implementation makes it difficult to verify the reported improvements or assess the robustness of the proposed method.

 

[References]

[R1] "TTRL: Test-Time Reinforcement Learning," NeurIPS, 2025.
[R2] "Test-Time Scaling in Diffusion LLMs via Hidden Semi-Autoregressive Experts," ICLR, 2026.
[R3] "What If Consensus Lies? Selective-Complementary Reinforcement Learning at Test Time," ACL, 2026.
[R4] "Beyond Majority Voting: Self-Reflective Test-Time Reinforcement Learning for LLM Reasoning," ICML, 2026.

 

I also have some minor concerns as follows:

  1. The proposed optimization objective is closer to reward-filtered supervised replay with KL regularization than conventional on-policy reinforcement learning. The distinction should be made clearer when positioning the method as TTRL.

  2. The reference-novelty criterion does not necessarily identify beneficial updates, since a majority-supported candidate may still be incorrect even when the frozen reference is correct.

  3. Code-answer equivalence is determined using Python AST comparison rather than functional equivalence, potentially affecting consensus estimates and update eligibility on HumanEval and MBPP.

  4. The rollout-budget sensitivity results in Appendix Table 8 show non-monotonic performance, suggesting that the method may be sensitive to the evidence budget.

  5. The alternative 4x2 geometry protocol in Appendix Table 12 outperforms the main configuration on several settings, but the comparison simultaneously changes geometry allocation and the novelty condition, limiting its interpretability.

  6. The comparisons with published labeled-RL methods use different supervision and training regimes and therefore provide limited evidence of competitive performance under matched conditions.

Questions for authors

Please refer to the Major Weaknesses section for detailed explanations of the concerns below.

  1. What is the specific methodological contribution beyond combining existing consensus-based TTRL, geometry-diverse decoding, and selective pseudo-labeling techniques [R1-R4]?

  2. What evidence demonstrates that the gains are specifically attributable to denoising-geometry diversity rather than stricter consensus filtering or different selected training samples?

  3. How does Geo.-LiM compare with inference-only geometry-diverse majority voting under the same eight-rollout budget, without parameter adaptation?

  4. Are the reported improvements consistent across multiple adaptation seeds and prompt orders, particularly for settings with small accuracy differences or very few admitted prompts?

  5. Given that the novelty condition structurally excludes certain redundant proposals, what independent evidence supports the claim that Geo.-LiM selects more beneficial updates?

  6. Can the authors provide the implementation details or source code needed to independently reproduce the geometry sampling, trace replay, and adaptation results?

1 comment

Thank you for the thorough rebuttal and the considerable effort devoted to addressing my concerns. I also apologize for providing my follow-up comments so close to the end of the discussion period. Given the extensive responses and additional experiments, I needed some time to carefully examine the results and reconsider my assessment.

I appreciate that the authors have addressed several of my concerns. In particular, the new Maj@8 comparisons provide useful evidence that persistent adaptation can improve performance beyond inference-only majority voting under a matched evaluation protocol. The additional multi-seed experiments, geometry-mixing ablation, and runtime measurements also strengthen the empirical evaluation. I further appreciate the authors' acknowledgment of the limitations of the post-hoc analysis and the distinction between reduced optimizer steps and actual computational savings.

However, my main concerns regarding methodological novelty and the robustness and practical significance of the reported improvements remain only partially addressed.

First, while the rebuttal clarifies the distinction between existing TTRL, geometry-diverse inference, and the proposed persistent adaptation framework, the method still appears to primarily combine established techniques, including consensus-based pseudo-labeling, selective filtering, and regularized replay. More importantly, although the additional geometry-mixing ablation is helpful, the evidence remains insufficient to convincingly establish the independent contribution of geometry diversity. The improvements are relatively small and are evaluated on only a few settings with limited seeds.

Second, I remain concerned about the statistical reliability of the reported improvements. The additional three-seed experiments are appreciated, but they also reveal substantial variability, particularly on MATH500, where the standard deviation exceeds the mean improvement. Most of the original 30 experimental settings still rely on single adaptation runs, making it difficult to determine whether the reported advantages over the baselines are consistently reproducible.

Third, although the new Maj@8 comparisons demonstrate positive gains after adaptation, these improvements are relatively modest (+0.76 to +2.00 pp) and have not been shown to be robust across adaptation seeds. Moreover, the measured runtime reductions of 3.49% and 13.18% are considerably smaller than the reductions in admitted prompts and optimizer steps. While I acknowledge that persistent adaptation is a different objective from inference-only voting, its additional computational cost should be justified by more robust and practically meaningful improvements.

Overall, I acknowledge that the rebuttal has meaningfully strengthened the empirical evidence and clarified several important aspects of the paper. Nevertheless, I remain unconvinced that the proposed method establishes a sufficiently distinct methodological contribution or consistently significant practical advantages over simpler alternatives.

I sincerely appreciate the authors' thoughtful responses and the substantial effort invested in addressing my concerns.

Reviewer_fPTBOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Geometry-Selective Less-is-More TTRL (Geo.-LiM), a framework for persistent, label-free adaptation of diffusion language models (dLLMs). Compared with consensus-based TTRL baseline under the same eight-rollout evidence budget, it achieves higher performance in 23 of the 30 settings while admitting fewer prompts and taking fewer optimizer steps in every setting.

Critical strengths
  1. Compared with consensus-based TTRL baseline under the same eight-rollout evidence budget, it achieves higher performance in 23 of the 30 settings while admitting fewer prompts and taking fewer optimizer steps in every setting.
Critical weaknesses
  1. The presentation and overall written is poor. It requires substantial effort to understand. This paper's core idea is select self rollout data when at least five of the eight samples agree on a valid answer that differs from the valid frozen-reference answer, and then train on the selected trajectories using a block-wise masked diffusion loss with KL regularization. This is kind of like distilling majority vote to the model. However, the paper uses words like 'geometry-diverse evidence', 'reference-novelty gate', 'positive-only replay' to over-package its method.

  2. Lack of direct comparison with maj@8.

Questions for authors

I have no other questions, please see the weakness.

1 comment

I thank the authors for the detailed response and the additional Maj@8 results. After reading the rebuttal, I will keep my score. Frozen Maj@8 still clearly outperforms the adapted policy, so the gains from test-time adaptation remain too minor to justify the added complexity.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.