acceptodds
Under review as a conference paper at ICLR 2027

A Token Rendering Perspective on Large Language Model Unlearning

Abstract

LLM unlearning seeks to remove targeted knowledge without disrupting retained capabilities. We argue that many existing output-based methods control which predictions to suppress without explicitly parameterizing the contextual relations involved in producing them. We therefore introduce a token rendering perspective, which views autoregressive generation as a hierarchical rendering process in which contextual information is transmitted through attention-mediated token relations and progressively rendered into next-token probabilities. This perspective provides a common first-order characterization of several representative unlearning objectives as forms of target-token renderability control, highlighting that these objectives do not explicitly parameterize the underlying token relations. Based on this view, we propose -Unlearning, which freezes the base LLM and explicitly tunes the renderability of token-token relations. We estimate the sensitivity of semantic relations to forget and retain rendering, attenuate forget-sensitive relations, and constrain changes to retain-sensitive ones. For efficient intervention, relation transmittance is parameterized over semantic token clusters. Experiments on TOFU and WMDP across multiple LLMs and forgetting ratios demonstrate that -Unlearning achieves competitive forgetting–utility trade-offs under the evaluated settings. Code will be released upon acceptance.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Machine Unlearning, LLM Unlearning, Token Rendering.

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_rVEcOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2(+1)
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes T3T^3-Unlearning, which freezes the LLM and learns a bias on the attention logits between groups of similar tokens, weakening the attention paths that produce the forget data while preserving those used by the retained data. Experiments show that it outperforms existing unlearning methods on standard benchmarks.

Critical strengths

Strengths

  1. The paper tackles an important and practical problem. Removing targeted knowledge from LLMs while preserving their general capabilities is an interesting problem closely related to AI safety.
  2. The proposed method consistently outperforms existing unlearning baselines across multiple models and forget ratios.
Critical weaknesses

Weaknesses

  1. The theory renames known quantities without adding anything new to the method. "Token rendering" and "renderability" are just the output distribution and the log-probability, and Prop.1 is a first-order Taylor expansion that holds for any loss on log-probabilities, as used in most prior analyses of fine-tuning dynamics. Likewise, the transmittance τij=exp⁡(−Γij)\tau_{ij} = \exp(-\Gamma_{ij}), said to be borrowed from volume rendering, is just a learnable bias subtracted from the attention logits. The volume rendering and NeRF background is unnecessary and distracting.

  2. The claimed efficiency is unclear. Training Γ\Gamma still requires backpropagation through the full model, as with LoRA, and the table has 10810^8 entries, far more than the rank-8 LoRA used by the baselines.

Questions for authors

Questions

  1. The issue raised by Proposition 1 seems to apply equally to the proposed method, since the relation depth Γ\Gamma is trained with the same gradient-ascent loss, and its effect on the retained data is still not explicitly controlled. Why should updating Γ\Gamma be better than updating the model weights with LoRA?
  2. Since the authors claim that their method is efficient, could they compare training time, peak GPU memory, and inference overhead with the LoRA baselines? This comparison should also account for whether the method can use standard fused attention kernels such as FlashAttention.
Reviewer_GWJzOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper introduces a token rendering perspective for LLM unlearning, viewing autoregressive generation as a hierarchical process where contextual information propagates through token‑token relations in attention layers. It shows that existing output‑level unlearning objectives (e.g., GA, SimNPO, WGA, SatImp, ASU) can be unified as first‑order renderability reweighting that does not explicitly control these underlying relations. Based on this, the authors propose T³‑Unlearning, which freezes the base LLM and tunes a shared relation‑transmittance matrix over semantic token clusters. Forget‑sensitive relations are attenuated while retain‑sensitive ones are preserved. Experiments on TOFU and WMDP demonstrate competitive forgetting‑utility trade‑offs, with additional robustness under relearning and continual unlearning settings.

Critical strengths
  1. Novel and unifying theoretical framing​ – The token‑rendering perspective and the first‑order analysis (Proposition 1) elegantly reveal the common limitation of prior unlearning losses, motivating a principled shift from output suppression to relation‑level intervention.
  2. Parameter‑efficient and modular design​ – By freezing the base model and only optimizing a small, shared relation‑transmittance matrix, T³‑Unlearning avoids catastrophic utility loss and is naturally suited for continual or sequential unlearning requests.
Critical weaknesses
  1. Dependence on semantic clustering quality​ – The method relies on semantic token clusters to parameterize relations, but the paper provides limited details on clustering algorithms, robustness to subword boundaries, entity aliases, or paraphrases. Poor clustering may leave forget‑related pathways intact.
  2. Approximate unlearning with limited attack evaluation​ – T³‑Unlearning only attenuates renderability rather than truly removing knowledge. The relearning experiments use only 1–4 epochs, and stronger white‑box attacks, membership inference, or cross‑lingual rephrasing are not tested, leaving its security guarantees unclear.
Questions for authors
  1. How sensitive is T³‑Unlearning to the choice of clustering method and the number of clusters KK? Could you provide ablation studies on different clustering strategies and their impact on forget‑retain balance?
  2. Beyond 4‑epoch relearning, have you tested stronger recovery attacks (e.g., longer fine‑tuning, prompt‑based extraction, or activation steering)? Would the attenuated relations be sufficient to prevent knowledge leakage under adaptive adversarial queries?
  3. The relation‑transmittance matrix is shared across all layers and heads. Did you observe any cases where this global sharing hurt performance (e.g., early layers needing different relation modulation than deeper layers)? Would layer‑specific transmittance bring further gains?
3 comments

Dear Reviewer GWJz,

Reviewing many assigned papers is a time-consuming task and so we would like to sincerely thank you again for your time and effort to review the paper.

As the rebuttal and discussion period is approaching its end, we would be grateful if you could kindly take a moment to review our responses and additional experimental results. We have carefully addressed your concerns and hope that our clarifications have helped resolve the issues raised in your review.

If any concerns remain, we would be more than happy to provide further clarification or additional evidence. We would greatly appreciate your updated assessment in light of our responses.

Thank you again for your valuable time and consideration.

Best regards,

The Authors

First, thanks for the authors to add the clustering details. The paraphrase-robustness results, and the global-vs-layer ablation answer Q3: global sharing gives the best Avg, while layer-1-only preserves MU but fails to forget. This confirms a genuine forget–retain trade-off rather than dominance on both axes. For Q2, I think the response does not address stronger recovery attacks or the relearning protocol. Again, thanks for the authors to attribute these additional contents. I would hold my score.

Thank you for acknowledging the additional experiments. A relearning evaluation that permits removing or resetting (\Gamma) is not informative for our method, since it restores the original computation without any relearning. Our method implements forgetting through persistent relation modulation, rather than irreversible knowledge deletion from the frozen backbone.

Reviewer_oWCTOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper introduces a token-rendering perspective on LLM unlearning. It views generation as information flowing through token relations. Based on this view, the authors propose an unlearning method. This method keeps the base model frozen and trains a compact relation matrix to control attention. It weakens relations that support targeted knowledge while limiting changes to retained knowledge. Experiments on TOFU and WMDP show competitive forgetting–utility trade-offs across multiple models.

Critical strengths
  • Interesting perspective: Explains unlearning through information flow between tokens
  • Compact design: Trains a small relation matrix while keeping the base model frozen
Critical weaknesses
  • Small and out-dated models in experiments
  • Too many equations in the main content
Questions for authors

Maybe consider to compress Section 3 and add some more experiments? Some references are recommended: "Learning-Time encoding shapes Unlearning LLMs", "Leak-Resistant Unlearning: A New Benchmark for Evaluating Multi-Hop Reasoning Consistency and Recovery Robustness".

1 comment

Dear Reviewer oWCT,

Thank you again for your time and effort in reviewing our paper.

As the discussion period draws to a close, we would greatly appreciate any further feedback on our rebuttal and additional experimental results. We remain happy to clarify any outstanding questions or concerns.

Thank you for your consideration.

Best regards,

The Authors

Comments on the paper

2 comments

Dear reviewers,

Please check your assigned papers for author responses. If the authors have addressed any of your questions or concerns, please leave a brief comment for them indicating which points have been resolved and which, if any, remain open. Even if your assessment is unchanged, a short follow-up is helpful.

Best, AC

Dear Reviewers,

The Author-Reviewer Discussion period is coming to an end soon. Please actively participate in the rebuttal discussion with the authors after carefully reading the authors' responses.

Best, AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.