acceptodds
Under review as a conference paper at ICLR 2027

Uncovering Uncontrolled Repetition through Residual Stream Dynamics

Abstract

Uncontrolled repetition can prolong autoregressive generation in large language models (LLMs) and enable resource consumption attacks. Prior analyses of repetitive generation have identified strongly activated features in intermediate and late layers. However, how uncontrolled repetition activity emerges and develops before becoming prominent in these layers remains insufficiently understood. In this paper, we investigate this question primarily in large vision-language models (LVLMs), which support a richer set of uncontrolled repetitions through both visual and textual inputs. We propose Tokenwise Residual Comparison (TRC), a method that identifies and localizes anomalies associated with repetition from residual dynamics during generation. TRC compares attention and multilayer perceptron writes to the residual stream across generated tokens to identify patterns associated with repetition. It then selectively suppresses coordinates in the residual stream at the identified layer. Experiments show that TRC effectively mitigates uncontrolled repetition, reducing loop rates by 57% on average. Our analysis further shows that repetition semantics emerge in shallow layers and propagate through the residual stream, disrupting normal representations. TRC also generalizes to large language models (LLMs) and large reasoning models (LRMs), where it consistently captures analogous repetition dynamics and achieves effective mitigation. Our work broadens the study of repetitive generation from its prominent internal representations to earlier opportunities for intervention, providing insights for mitigating resource consumption attacks.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Uncontrolled Repetition, Residual Stream, Mechanistic Analysis, Large Vision-Language Models, Resource Consumption Attacks

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_UjwCOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The authors propose Tokenwise Residual Comparison (TRC), a method for identifying and mitigating uncontrolled repetition in LVLMs, LLMs, and LRMs. Unlike previous studies that focus on repetition related features in intermediate or late layers, TRC compares residual stream contributions across repeated token positions to identify early layer signals and selectively suppress relevant hidden dimensions. Experiments show that TRC reduces repetition loop rates by an average of 57% while largely preserving performance on normal tasks.

Critical strengths
  1. TRC analyzes changes in residual stream contributions across token positions, providing a new perspective on the emergence of uncontrolled repetition and enabling fine grained localization and intervention.
  2. The method is tested across LVLMs, LLMs, and LRMs under different attack conditions. The authors also conduct activation restoration experiments to investigate the behavioral relevance of the identified early layer signals.
Critical weaknesses
  1. TRC does not consistently outperform simpler decoding based methods such as no repeat, and its performance varies across models and attack types.
  2. TRC requires attack and benign reference samples to identify the intervention layer and dimensions. Its generalization to unseen attacks and different deployment settings remains unclear.
Questions for authors
  1. I suggest authors evaluate TRC on unseen attack types using intervention masks calibrated on a limited set of known attacks. This would help determine whether TRC can generalize beyond the attack samples used during calibration.
  2. Please conduct additional ablation and activation patching experiments to test the necessity and sufficiency of TRC identified dimensions. These experiments should include different intervention strengths, random dimension controls, and evaluations across multiple models and attack types to better establish the causal role of early layer residual dimensions in uncontrolled repetition.
4 comments
AuthorsOpenReview ↗

Response to Reviewer UjwC (1/4)

Thank you for your careful review of our methodology and practical implications. We believe some aspects of our work may not have been sufficiently clear, and we hope the explanations below help address your concerns.


W1: Comparison with simpler decoding-based defenses

Defense comparisons should also consider their effects on benign task capabilities, and TRC's advantage lies in better preserving complex reasoning capabilities. We added comparative experiments on AIME and GPQA using two textual LLMs, evaluating the task accuracy of No defense, Fixed-length, No-repeat, AUSteer, and TRC. Simple decoding constraints cause substantial accuracy losses on complex tasks, whereas TRC better preserves task capabilities. For example, Qwen's undefended accuracy on AIME is 8.33%, while both Fixed-length and No-repeat reduce it to 1.67%; Llama's accuracy on GPQA decreases from 16.67% to 11.11% under No-repeat. Overall, TRC achieves higher accuracy than Fixed-length and AUSteer in all four settings, demonstrating its advantage in preserving normal reasoning capabilities on complex tasks.

The drawbacks of constraints on token repetition and output length may be less apparent on tasks requiring only short responses. On complex reasoning tasks, however, length truncation can interrupt unfinished reasoning, while repetition blocking can suppress recurring expressions needed for intermediate reasoning steps. These constraints can therefore substantially impair task performance, as illustrated by the accuracy losses above.

Table: Accuracy of different defenses on benign complex tasks (%, higher is better).

ModelDatasetNo defenseFixed-lengthNo-repeatAUSteerTRC
Llama-3.2-3B-InstructAIME1.67%0.00%1.67%1.67%3.33%
Llama-3.2-3B-InstructGPQA16.67%13.64%11.11%10.61%14.14%
Qwen2.5-3B-InstructAIME8.33%1.67%1.67%0.00%8.33%
Qwen2.5-3B-InstructGPQA26.77%14.65%28.28%3.54%27.27%
AuthorsOpenReview ↗

Response to Reviewer UjwC (2/4)

W2&Q1: Generalization to unseen attacks and deployment settings

The existing experiments cover transfer across attack types and input modalities. For LVLMs, masks are calibrated only on RECITE visual attacks; the GCG and LoopLLM textual attacks in Table 1 in the manuscript are not used for calibration, and loop rates decrease in all five evaluated transfer settings. For LLMs and LRMs, masks are calibrated only on GCG, and Figure 2 in the manuscript shows transfer to LoopLLM and Direct: the LoopLLM loop rates of Llama and Qwen decrease from 56% and 36% to 12% and 16%, respectively; the Direct loop rates of the three LRMs decrease from 100%, 100%, and 92% to 8%, 0%, and 0%, respectively. These evaluations cover 3 LVLMs, 2 LLMs, and 3 LRMs, demonstrating that TRC remains effective against attack types not used for calibration across multiple model configurations.

The supplementary experiments further show that attack-calibrated interventions can transfer to spontaneous repetition under normal inputs. On AIME and GPQA, we directly reuse residual masks constructed from GCG examples. Although failures occur in only a small subset of normal inputs, TRC reduces loop rates in three of the four settings, including a reduction from 18.33% to 5.00% for Llama on AIME. These results have been added to Appendix J of the manuscript.

Table: Loop rates after directly transferring attack-calibrated masks to benign tasks (%).

ModelDatasetNo defense loop rateTRC loop rate
Llama-3.2-3B-InstructAIME18.33%5.00%
Llama-3.2-3B-InstructGPQA17.17%14.14%
Qwen2.5-3B-InstructAIME3.33%1.67%
Qwen2.5-3B-InstructGPQA2.02%2.02%
AuthorsOpenReview ↗

Response to Reviewer UjwC (3/4)

Q2: Necessity and sufficiency of TRC-identified dimensions

We thank the reviewer for the attention to method effectiveness and causal evidence. We also recognize the importance of distinguishing correlation-based localization from intervention sites with functional effects. Accordingly, the manuscript already examines bidirectional cross-interventions, random-coordinate controls, and ablations of intervention ratios and locations. Specifically:

(1) Table 5 in the manuscript already includes bidirectional activation patching to examine the role of the selected coordinates in loop maintenance and normal generation. In the first layer, replacing the values of TRC-localized coordinates in attack trajectories with values at the same coordinates in paired normal trajectories restores non-repetitive generation in 80% of cases. This provides direct intervention evidence that the selected coordinates contribute to maintaining uncontrolled repetition. We also performed the reverse intervention by injecting the activation differences between attack and normal trajectories into the corresponding coordinates of normal trajectories. Although this did not reproduce looping, it caused clear generation disruptions and erroneous early termination, indicating that these coordinates have substantive functional effects on generation.

(2) Table 5 in the manuscript also includes controls with an equal number of random coordinates to test whether the intervention effects arise from TRC's coordinate selection. When restoring normal residual values, the recovery rate is 80% for TRC-selected coordinates, compared with 40% for random coordinates. In the reverse direction, injecting the differences at the localized coordinates clearly disrupts normal output, whereas the output remains largely unchanged after the matched random-coordinate intervention. These controls show that the observed behavioral changes do not arise merely from arbitrary modifications to the same number of coordinates; TRC-selected positions have stronger functional relevance.

(3) Appendix F of the manuscript examines different intervention extents by varying the intervention ratio ρ\rho, while Figure 4 in the manuscript further examines the role of intervention location. We set the fraction of selected coordinates to 1%, 5%, 10%, 15%, and 20%, and report loop rates and generation lengths under GCG, RECITE, and LoopLLM, together with changes in benign task accuracy. All tested ratios reduce macro-average loop rates and generation lengths relative to No defense, while benign task accuracy decreases by at most 1.5 percentage points within the tested range. The branch and layer-location ablations in Figure 4 in the manuscript further show that intervening on Attention, MLP, or layers at different depths produces different effects on repetition suppression and benign capability preservation, providing additional evidence for the choice of intervention location.

AuthorsOpenReview ↗

Response to Reviewer UjwC (4/4)

(4) We further added a comparative experiment on the source of examples used to construct suppression directions, examining the intervention effects of the selected coordinates after changing the calibration attack in an LLM and an LVLM. In the main experiments, Llama is calibrated on GCG and Qwen-VL on RECITE; in the supplementary experiment, we instead use LoopLLM and GCG, respectively, following the same TRC construction procedure. Loop rates decrease in all settings, showing that suppression directions constructed from different attack sources remain effective interventions. These results complement the activation patching, random-coordinate controls, and layer-location ablations above, providing further evidence for the functional relevance of early-layer residual coordinates across models and attack conditions.

Table: Loop rates and mean generation lengths after changing the source used to construct suppression directions.

ModelCalibration attackTest attackNo defense loop rateTRC loop rateNo defense mean lengthTRC mean length
Llama-3.2-3B-InstructLoopLLMGCG28.00%16.00%1001.88676.84
Llama-3.2-3B-InstructLoopLLMLoopLLM88.00%12.00%3633.56699.24
Llama-3.2-3B-InstructLoopLLMDirect96.00%52.00%3942.921972.52
Qwen2.5-VL-3B-InstructGCGGCG100.00%60.00%4096.002463.48
Qwen2.5-VL-3B-InstructGCGLoopLLM32.00%16.00%685.12158.60
Qwen2.5-VL-3B-InstructGCGRECITE100.00%0.00%4096.0058.16

We hope these clarifications provide a clearer understanding of our work and look forward to further discussion. Should our responses resolve your concerns, we would greatly appreciate your consideration of a higher rating. Thank you again for your careful evaluation.

Reviewer_1gi3OpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper analyzes the internal mechanisms of uncontrolled repetition in LVLMs and proposes Tokenwise Residual Comparison (TRC) to mitigate it. TRC compares residual stream contributions across the generated token sequences to derive tokenwise repetition signals for each layer. TRC identifies and selectively suppresses layers and coordinates. The results showed that TRC reduced uncontrolled repetition while preserving most of the benign task performance. Unlike prior works that mainly focused on repetition-related features being activated strongly in the intermediate and late layers, this paper confirmed that repetition-related signals can emerge in the shallow layers and propagate through the residual stream. These effects were observed similarly not only in the LVLM but also in the LLM and LRM.

Critical strengths
  • This paper reveals that repetition-related signals appear in shallow layers, unlike prior works that focused on prominent repetition-related activations in intermediate and late layers.
  • The experiments were conducted on diverse model families, such as LVLMs, LLMs, and LRMs, under various visual and textual attacks. The results further demonstrated that TRC can mitigate uncontrolled repetition while preserving benign task accuracy.
  • The comparison between uncontrolled repetition and benign requests containing legitimate repetition is interesting and insightful.
Critical weaknesses
  • Each stage or design choice of the method is not properly explained. The method is complex and difficult to follow, and the reasons and justifications for choosing each component are unclear. For example, N-gram selection in Eq. (1) selects the shortest n-gram covering more than 50% of the output. However, this does not necessarily correspond to the actual repetition period. For instance, ABCABC has a repetition period of 3, but the method selects n = 2. In addition, the 50% threshold is heuristic.
  • In Tables 1 and 2, the no-repeat baselines consistently outperform TRC and provide better performance. The authors do not clearly demonstrate the limitations of these simple baselines, thereby limiting the motivation for TRC. In addition, the experimental setup is limited. The authors suggest that TRC can be applied to LLMs or LRMs; however, using a subsample of only 200 examples is not fully convincing.
  • TRC applies a fixed mask to every query, making it difficult to conclude that there are no side effects. For example, in LRMs, more difficult tasks such as AIME25 or GPQA need to be tested.
Questions for authors
  • Why is low variation preferred in layer selection, while a high attack TRC score is chosen in coordinate selection?
  • Does the TRC-l score itself depend on layer depth or on the characteristics of general residual dynamics?
  • In normal generation, a specific intermediate layer is also selected; how should this result be distinguished from shallow localization in uncontrolled repetition?
  • Since TRC relies on offline calibration, how sensitive is it to the distribution of calibration examples, and does TRC remain effective under different attack distributions?
  • The evaluation focuses on attack-induced repetition. However, uncontrolled repetition can also occur under normal inputs, particularly in small open-source models. Does TRC identify similar signals in this case?
7 comments
AuthorsOpenReview ↗

Response to Reviewer 1gi3 (1/5)

We appreciate your thoughtful assessment of our residual-stream analysis and experimental evaluation. Your feedback is valuable in helping us clarify our contributions, and we hope our responses resolve the remaining questions.


W1: Rationale for repetition-period selection and the coverage threshold

We thank the reviewer for the careful reading and correction. Equation (1) in the manuscript did not accurately express our intended selection order, “first maximize coverage, then select the shortest candidate,” because the argmax over candidate lengths was omitted. The complete selection formula should be

n^\star&=\min\operatorname*{arg\,max}_{n}\left\{\max_{g:\,|\mathcal{O}_n(g)|\ge2}\frac{\left|\bigcup_{t\in\mathcal{O}_n(g)}\{t,\ldots,t+n-1\}\right|}{T}\right\} \end{aligned}$$ Here, the candidate lengths range over $n\in\{1,\ldots,\lfloor T/2\rfloor\}$. If no segment occurs at least twice, or the maximum coverage does not exceed 50%, $n^\star$ is treated as undefined and the original fallback rule is used. For example, in `ABCABC…`, the repeated segment of length 3 achieves maximum coverage. When its integer multiples achieve the same coverage, the shortest-first rule selects 3, rather than a bigram whose coverage is only approximately 2/3. We have corrected the equation and the accompanying explanation in the manuscript. This was an omission in the presentation and does not affect the underlying study. Furthermore, **50% is a coverage hyperparameter for identifying the dominant repetition pattern. It requires the candidate pattern to cover the majority of output tokens, reducing the likelihood that incidental local repetition is used for cycle alignment.** Prior work also uses thresholds on the proportion of repeated content. For example, Gopher [1] uses character-coverage thresholds of 20% and 18% for the most frequent bigram and trigram, respectively. Although the statistical units and application settings differ, this provides a precedent for identifying repetitive content through coverage thresholds. **We choose a higher coverage requirement to obtain clearer evidence of repetition and reduce the risk of using incidental repetition in unsuccessful attack samples as the basis for cycle alignment.** [1] Scaling Language Models: Methods, Analysis & Insights from Training *Gopher*
AuthorsOpenReview ↗

Response to Reviewer 1gi3 (2/5)

W2&W3: Baseline comparisons, sample size, and performance on complex tasks

Regarding sample size, our assessment of TRC's effectiveness is based on combined evidence across models, datasets, and attack configurations, rather than relying solely on a particular evaluation subset of 200 examples. Across settings covering LVLMs, LLMs, and LRMs, TRC consistently exhibits the same trend of suppressing uncontrolled repetition while largely preserving benign task capabilities. This consistency across settings indicates that the observed benefits do not depend on a particular model or data subset.

To strengthen the evaluation of benign task capabilities, we added comparative experiments on capability preservation in complex tasks. The original evaluation already covered 3 LVLMs, 2 LLMs, and 3 LRMs; we have now additionally evaluated AIME and GPQA on two textual LLMs. The results show that TRC achieves higher accuracy than Fixed-length, No-repeat, and AUSteer, with Fixed-length and No-repeat having particularly pronounced effects on benign tasks. For example, Qwen's undefended accuracy on AIME is 8.33%, while both Fixed-length and No-repeat reduce it to 1.67%. These results show that conventional defenses can impair complex task performance while reducing loop rates, whereas TRC better preserves reasoning capabilities on these tasks.

The supplementary experiments also show that TRC can mitigate spontaneous repetitive failures in small models on benign complex tasks. For example, Llama's loop rate on AIME decreases from 18.33% to 5.00%, while accuracy increases from 1.67% to 3.33%. This shows that TRC's benefits extend to benign generation without attacks.

Table: Accuracy and loop rates of different defenses on benign complex tasks (%).

ModelMethodAIME accuracy ↑AIME loop rate ↓GPQA accuracy ↑GPQA loop rate ↓
Llama-3.2-3B-InstructNo defense1.67%18.33%16.67%17.17%
Llama-3.2-3B-InstructFixed-length0.00%1.67%13.64%1.01%
Llama-3.2-3B-InstructNo-repeat1.67%0.00%11.11%0.00%
Llama-3.2-3B-InstructAUSteer1.67%18.33%10.61%23.23%
Llama-3.2-3B-InstructTRC3.33%5.00%14.14%14.14%
Qwen2.5-3B-InstructNo defense8.33%3.33%26.77%2.02%
Qwen2.5-3B-InstructFixed-length1.67%0.00%14.65%1.01%
Qwen2.5-3B-InstructNo-repeat1.67%0.00%28.28%0.00%
Qwen2.5-3B-InstructAUSteer0.00%98.33%3.54%100.00%
Qwen2.5-3B-InstructTRC8.33%1.67%27.27%2.02%
AuthorsOpenReview ↗

Response to Reviewer 1gi3 (3/5)

Q1: Low variation for layer selection versus high TRC scores for coordinate selection

Layer selection and coordinate selection address different levels of granularity, so preferring low layer scores and high coordinate scores is not contradictory. Layer selection concerns “where to intervene.” Using the layer-averaged contribution changes, TRC-l identifies regions with low changes across repetition cycles and limited net cross-layer variation, reflecting relatively stable internal patterns. This reflects persistent repetition-related residual dynamics during uncontrolled repetition. Coordinate selection concerns “which dimensions to intervene on within that region.” Within the selected layer, we rank coordinates by their attack TRC vector scores and select a small number with relatively prominent contribution changes across cycles. These scores measure the mean absolute contribution change of each coordinate, which differs from the normalized composite score used for layer selection.

Figure 3 and Table 5 in the manuscript provide empirical evidence for layer localization and coordinate selection, respectively. Figure 3 in the manuscript shows that uncontrolled repetition has a different layer-localization profile from ordinary requests and legitimate repetition, supporting the use of shallow low-change regions to identify dynamics associated with uncontrolled repetition. Table 5 in the manuscript directly examines the functional effects of the selected coordinates in the first layer: replacing their values in attack trajectories with the values at the same coordinates in paired normal trajectories restores non-repetitive generation in 80% of cases, compared with 40% for an equal number of random coordinates. The examples in the manuscript's appendix further show that intervention on the localized coordinates causes erroneous early termination, while the matched random-coordinate intervention preserves the normal response. These results establish that the selected coordinates are intervention sites with substantive effects on generation, providing functional evidence for coordinate selection after localization.

Q2: Dependence of TRC-l on layer depth versus residual dynamics

The comparison between normal requests and repetition conditions in Figure 3 in the manuscript shows that, in the evaluated settings, TRC-l does not simply exhibit a fixed preference for shallow layers due to depth; its localization distinguishes the residual dynamics of uncontrolled repetition from those of normal generation. Figure 3(a) in the manuscript compares the top-ranked localization positions, while Figure 3(b) in the manuscript presents more detailed changes in rankings across layers. In panel (a), attack-induced repetition is localized to shallow layers, whereas the top-ranked layers for benign references lie deeper in the middle of the network. This shows that the same localization rule does not select shallow layers for every generation state. In panel (b), ordinary normal requests and legitimate repetition exhibit similar ranking profiles, with higher-priority regions concentrated in the middle layers; uncontrolled repetition instead receives higher localization priority in shallow layers, which decreases with depth. Together, these comparisons support that the observed shallow localization is associated with uncontrolled repetition, rather than being explained solely by a uniform preference for shallow layers across all requests.

We have clarified the panel titles and caption of Figure 3 in the manuscript to avoid ambiguity among TRC, TRC-l scores, and localization rankings. The caption now explicitly states that both panels perform layer localization based on TRC-l.

Reviewer_uRosOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes Tokenwise Residual Comparison (TRC), a training-free method to localize and mitigate uncontrolled repetition in autoregressive models by analyzing residual stream dynamics. TRC compares internal representations at positions aligned to a repeated token cycle, selects a layer with unusually small and stable changes relative to benign generation, and applies a sparse, fixed residual-stream mask at that layer to suppress repetition-related coordinates. Across LVLMs, LLMs, and LRMs, TRC reduces loop rates and generation length while minimally affecting benign-task performance, and mechanistic analyses suggest that repetition-related signals emerge early in the network and propagate through residual connections.

Critical strengths
  1. The authors introduce a residual-difference analysis (across repeated n-gram positions) across benign and attacked examples that helps understand where repitition signals emerge in the network.
  2. They propose a training-free defense with negligible inference overhead that can successfully reduce loops in model generation.
Critical weaknesses
  1. The loop rates for GCG (textual) and LoopLLM (textual) for InstructBLIP-7B and Qwen2.5-VL-3B is around 60% (Table 1), which shows that TRC is not effective across all models and benchmarks.
  2. The benign accuracy evaluation using TRC is limited to only two LVLM tasks, ScienceQA and TextVQA (Table 2). An extensive evaluation across standard LVLM benchmarks is necessary to establish no degradation in benign accuracy.
  3. There is an extensive literature on activation steering for compressing the generation length [1,2]. The authors must compare how their approach compares against these steering methods, or decoding strategies that penalize long-range repititions.
  4. The authors do not quantify the time and memory requirement for calibrations. The calibration cost appears to grow as the generation length increases.

[1] Wait, Wait, Wait... Why Do Reasoning Models Loop?, ICML 2026 Spotlight. [2] Activation Steering for Chain-of-Thought Compression, ACL Findings 2026.

Questions for authors
  1. How long is the calibration time and what is the maximum searched repitition length? How does the calibration time and memory scale as the generation length increases?
4 comments
AuthorsOpenReview ↗

Response to Reviewer uRos (1/4)

We sincerely thank you for recognizing the motivation, mechanistic insights, and effectiveness of our work. We also appreciate your constructive suggestions and hope the following clarifications address your concerns.


W1: Defense effectiveness with loop rates

We believe that defense effectiveness should not be assessed solely by absolute metrics. On InstructBLIP-7B, TRC reduces the loop rates under GCG and LoopLLM from 96% and 100% to 56% and 60%, respectively; on Qwen2.5-VL-3B, the GCG loop rate decreases from 100% to 60%. The loop rate decreases by 40 percentage points in all three settings. These gains substantially reduce attack-induced generation overhead. For example, the mean generation length of Qwen2.5-VL-3B under GCG decreases from 4096 to 2469.72 tokens, a reduction of approximately 39.7%; that of InstructBLIP-7B under LoopLLM decreases from 1833.96 to 1229.72 tokens, a reduction of approximately 32.9%. Thus, TRC provides benefits in both loop frequency and attack-induced output length, demonstrating its effectiveness under GCG and LoopLLM on InstructBLIP-7B and Qwen2.5-VL-3B.

AuthorsOpenReview ↗

Response to Reviewer uRos (2/4)

W2: Broader evaluation of benign task accuracy

We thank the reviewer for the suggestion regarding benign task coverage. ScienceQA and TextVQA assess multidisciplinary multimodal knowledge question answering and scene-text understanding, respectively, providing complementary evaluations of benign task capabilities. The experiments in the manuscript show that TRC largely preserves performance on the evaluated multimodal question-answering tasks.

To further examine benign performance on complex reasoning tasks, we supplemented the LLM evaluation with AIME and GPQA, comparing accuracy and loop rates. The supplementary experiments show that TRC achieves task success rates similar to those of the original models on complex tasks, with small fluctuations in either direction. We also find that the evaluated 3B models exhibit some spontaneous repetitive failures on complex tasks without attacks, and TRC can effectively mitigate these naturally occurring failures. On AIME, for example, Llama's loop rate decreases from 18.33% to 5.00%, while the number of correctly answered questions increases from 1 to 2. These results demonstrate TRC's robustness across task types; its offline-calibrated residual mask can also suppress spontaneous repetition on benign complex tasks. We have added this experiment to Appendix J of the manuscript, “Benign Performance and Spontaneous Repetition on Complex Tasks.”

Table: Accuracy and loop rates on benign complex tasks (%).

ModelDatasetNo defense accuracyTRC accuracyNo defense loop rateTRC loop rate
Llama-3.2-3B-InstructAIME1.67%3.33%18.33%5.00%
Llama-3.2-3B-InstructGPQA16.67%14.14%17.17%14.14%
Qwen2.5-3B-InstructAIME8.33%8.33%3.33%1.67%
Qwen2.5-3B-InstructGPQA26.77%27.27%2.02%2.02%
AuthorsOpenReview ↗

Response to Reviewer uRos (3/4)

W3: Distinction from activation steering and decoding methods

TRC's novelty lies in three aspects: its analytical perspective, intervention construction, and scope of applicability. We have presented these distinctions in Appendix B of the manuscript, “Distinction from Interpretability, Activation Steering, and Decoding Methods,” and added comparisons with the two cited works.

a) In terms of analytical perspective, TRC characterizes uncontrolled repetition through changes in internal dynamics across repetition cycles, examining changes at residual connections and thereby providing a framework for internal correction.

b) In terms of intervention construction, TRC uses these signals to select layers and coordinates, directly connecting mechanistic analysis to repetition suppression. Its research objective and technical approach are clearly distinct from [1], which explains the causes of looping through learning errors, and [2], which learns brevity steering vectors from paired long and short chains of thought.

c) In terms of applicability, we demonstrate consistent effectiveness across LVLMs, LLMs, and LRMs under both visual and textual attacks, whereas the two related works primarily address textual reasoning.

In response to the reviewer's suggestion to compare with activation steering methods that shorten generation, we have included an AUSteer-based activation intervention baseline, and the results support TRC's advantages in repetition suppression and benign capability preservation. We would also like to clarify that not all studies on length compression are applicable comparisons for our work, such as [2]. As you noted in the Strengths, our objective is to “reduce loops in model generation,” rather than merely shortening reasoning traces. Therefore, some length-optimization frameworks do not fit our threat setting.

[1] Wait, Wait, Wait... Why Do Reasoning Models Loop?, ICML 2026 Spotlight. [2] Activation Steering for Chain-of-Thought Compression, ACL Findings 2026.

AuthorsOpenReview ↗

Response to Reviewer uRos (4/4)

W4&Q1: Calibration time, memory, and dependence on generation length

TRC's localization statistics require only a finite number of consecutive observations, without expanding the observation window as total generation length increases. As emphasized in the manuscript, TRC aligns generation positions using the repetition period q=n⋆q=n^\star and averages the residual changes between these positions. Thus, when the observation start and spacing are fixed, the number of observations used for localization is controlled by KK, rather than determined by the full output length TT.

We have added calibration experiments showing that a small number of consecutive observations is sufficient for stable localization, with offline processing taking only a few seconds. With only 20 consecutive observations, Llama and Qwen-VL cover 90.85% and 88.24% of the reference dimensions, respectively, with total offline processing times of 621.276 and 404.999 seconds. Regarding memory, caching the residual vectors at the selected positions requires O(KLd)O(KLd) storage for residual statistics across KK observations, LL layers, and hidden dimension dd. Fixing KK therefore bounds this overhead. The complete experiment has been added to Appendix H of the manuscript, “Offline Construction Efficiency.”

This localization is a one-time offline procedure, and the resulting intervention layer and mask can be directly reused for subsequent requests. Online defense only applies the fixed mask at the selected layer, without log loading, observation-chain analysis, or relocalization. Appendix G of the manuscript, “Online Efficiency,” already reports online generation time and throughput.

Table: Localization coverage and offline processing time with different numbers of consecutive observations. Coverage is measured relative to the dimensions selected using 100 observations.

ModelConsecutive observationsCoverageLog loading (s)Localization only (s)Total offline processing time (s)
Llama-3.2-3B-Instruct581.70%0.2597.832621.270
Llama-3.2-3B-Instruct1084.97%0.2657.833621.271
Llama-3.2-3B-Instruct2090.85%0.2717.838621.276
Llama-3.2-3B-Instruct5094.77%0.3327.899621.337
Llama-3.2-3B-Instruct100100.00%0.4378.005621.442
Qwen2.5-VL-3B-Instruct580.39%0.3453.659404.932
Qwen2.5-VL-3B-Instruct1082.35%0.3323.639404.913
Qwen2.5-VL-3B-Instruct2088.24%0.4183.725404.999
Qwen2.5-VL-3B-Instruct5094.12%0.3913.697404.971
Qwen2.5-VL-3B-Instruct100100.00%0.5393.845405.119

We hope our responses have adequately addressed your concerns and would welcome further discussion. If you find our clarifications convincing, we would be grateful if you could consider raising your score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.