acceptodds
Under review as a conference paper at ICLR 2027

Diffusion-Inspired Reconfiguration of Transformers for Uncertainty Calibration

Abstract

Uncertainty calibration in pre-trained transformers is critical for their reliable deployment in risk-sensitive applications. Yet, most existing pre-trained transformers do not have a principled mechanism for uncertainty propagation through their feature transformation stack. In this work, we propose a diffusion-inspired reconfiguration of transformers in which each feature transformation block is modeled as a probabilistic mapping. Composing these probabilistic mappings reveals a probability path that mimics the structure of a diffusion process, transporting data mass from the input distribution to the pre-trained feature distribution. This probability path can then be recompiled on a diffusion process with a unified transition model to enable principled propagation of representation uncertainty throughout the pre-trained model’s architecture while maintaining its original predictive performance. Empirical results across a variety of vision and language benchmarks demonstrate that our method achieves superior calibration and predictive accuracy compared to existing uncertainty-aware transformers.

Primary area:
probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
Keywords:
Uncertainty calibration, Transformer

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_dQi1OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper addresses uncertainty calibration in transformers. Building on GP-based attention methods (SGPA, KEP-SVGP), whose attention outputs are Gaussian, the authors regroup the transformer blocks so that each block-to-block transition is Gaussian, making the layer stack resemble the reverse process of a diffusion model. They then distill all block-wise transitions into a single time-conditioned stochastic block that is applied repeatedly, trained by matching the teacher's transitions together with a task loss.

The claimed contributions are this diffusion-style reinterpretation, the distillation algorithm, and an empirical study. Experiments on small vision and language classification benchmarks compare the method with standard calibration baselines and GP-attention models, with additional results on distribution shift, OOD detection, and larger models. The authors report improved accuracy and calibration over the original backbones with fewer parameters.

Critical strengths
  1. Parameter efficiency. The method replaces the full stack of transformer blocks with a single shared block, so the resulting uncertainty-aware model has fewer parameters and a smaller inference memory footprint than the original backbone. This matters to my assessment because existing uncertainty-aware transformers typically add parameters and memory on top of the base model, whereas this approach goes in the opposite direction.

  2. Recovering the accuracy lost by GP-attention models. Transformers with GP-based attention in all blocks (SGPA, KEP-SVGP) are known to trade predictive accuracy for calibration. When the proposed method is applied on top of these models, accuracy is largely recovered while calibration remains competitive, and this holds across the in-distribution and distribution-shift experiments. This is consequential because it addresses the main practical drawback of this line of work and directly supports the paper's motivation.

Critical weaknesses
  1. Unclear source of improvement
  • Eq. 9 only bounds how well the student matches the teacher, so it cannot explain gains beyond the teacher.
  • For vanilla backbones the teacher is deterministic and there is no Gaussian transition to distill, yet the gains are similar.
  • The key notion of "transition correlation" (Fig. 1c) is not defined.
  1. New components are not validated
  • No ablation against a deterministic shared-block distillation baseline.
  • A student trained with plain KD already exceeds the teacher (Table 17).
  • The implemented loss (mean and Cholesky-factor MSE) differs from the derived KL objective.
  1. Limited novelty and missing prior work
  • Distilling transformer layers into a single shared recursive block is known [1,2], but is neither cited nor compared against.
  1. Weak evidence for the calibration claims
  • On ECE the method is best on only one of four datasets in Table 2 and is worse than temperature scaling on CIFAR-100.
  • Large-scale experiments are single runs compared only with the original model.
  • Some results are inconsistent: the MCD and ViT rows in Table 5 are identical, and with 40 runs (Table 16) the MCC gains over KEP backbones on CoLA reported in Table 1 disappear or reverse.
  1. Computational cost
  • Inference is slower than the vanilla backbone despite fewer parameters (Table 12).
  • A GP-attention teacher must be trained first.

[1] Nouriborji et al., "MiniALBERT: Model Distillation via Parameter-Efficient Recursive Transformers," EACL 2023. [2] Wu et al., "Latent Flow Transformer," arXiv:2505.14513, 2025.

Questions for authors
  1. Eq. 9 is minimized when the student equals the teacher. Why, then, does the student outperform the teacher in both accuracy and calibration? What is the source of this gain?

  2. For vanilla backbones (ViT, Transformer, and presumably ViT-B/16 and Qwen-2.5), the teacher is deterministic. What is p(Xt−1∣Xt)p(X_{t-1} \mid X_t) in this case, how does Eq. 9 apply, and where does the student's variance come from?

  3. How is the "correlation" in Fig. 1c defined and computed, and why does sharing parameters across steps preserve it?

  4. Is there any theoretical guarantee of convergence? Specifically, since the matching loss uses teacher-forced inputs and a diagonal covariance, does minimizing it guarantee that the student's rolled-out path distribution converges to the teacher's, or bound the error accumulated over TT steps?

  5. How many samples are used at inference, and are the inference times in Table 12 measured per sample?

  6. How does the method relate to prior work on distilling transformer layers into a single shared recursive block (e.g., MiniALBERT, Latent Flow Transformer)?

  7. Could the authors clarify the identical MCD and ViT rows in Table 5, and why the MCC gains over KEP backbones on CoLA in Table 1 disappear or reverse with 40 runs in Table 16?

  8. Ablations isolating the new components:

    • (a) a deterministic shared-block distillation baseline (same student, no stochastic transition);
    • (b) removing the covariance-matching term for KEP backbones;
    • (c) varying the number of inference samples.
  9. Results on larger datasets and models (e.g., ImageNet-scale) with multiple runs and standard baselines such as temperature scaling and deep ensembles. The current large-scale results are single runs compared only with the original model.

9 comments
AuthorsOpenReview ↗

Response to Reviewer dQi1 (1/4)

We appreciate the reviewer’s constructive feedback and would like to address concerns as follows:

[Q1] and [W1]

We thank the reviewer for this insightful question. We agree that Eq. (9) only provides a theoretical justification for matching the teacher's feature distribution and does not guarantee improvements beyond the teacher. We would like to clarify that DIRECTOR's improvements arise from its joint optimization of distribution matching and predictive performance, rather than from the distillation objective alone.

Specifically, DIRECTOR does not solely minimize Eq. (9). As described in Eqs. (11)–(12), we additionally introduce a performance-aware supervised loss that directly optimizes predictions generated by the unified transition model. Consequently, the student is not constrained to exactly reproduce the teacher's representations, but can learn task-informative representations that potentially improve predictive accuracy.

Furthermore, unlike the original pretrained model, DIRECTOR employs a unified probabilistic transition model that enables uncertainty propagation across transformer blocks. We believe that combining this probabilistic architecture with the distribution-matching and performance-aware objectives contributes to the observed improvements in both accuracy and calibration.

[Q2] and [W1]

We would like to clarify that DIRECTOR does not require a GP-reparameterized teacher and can directly distill feature transformations from a deterministic pretrained backbone.

Specifically, for vanilla backbones (e.g., ViT, Transformer, ViT-B/16, and Qwen-2.5), we treat each deterministic intermediate representation as the mean of a degenerate Gaussian distribution with zero variance. DIRECTOR then learns a unified probabilistic transition model, whose mean is encouraged to match the teacher's representations, while its variance is parameterized and learned by the student.

As detailed in Appendix A.5, the transition model is trained using mean matching, covariance matching, and a performance-aware supervised loss. Thus, the student is not restricted to reproducing the deterministic teacher's predictions, which may explain the observed improvements in accuracy and calibration.

[Q3] and [W1]

We would like to clarify the definition and motivation of transition correlation. In Fig. 1(c), transition correlation refers to the empirical correlation between intermediate feature representations at different transformer depths. Specifically, we measure the correlations between X6X_6 and subsequent representations X4X_4, X2X_2, and X0X_0 on CIFAR-10. We define the correlation between two feature representations using the Pearson correlation coefficient:

=\frac{\operatorname{Cov}(X_i,X_j)} {\sqrt{\operatorname{Var}(X_i)\operatorname{Var}(X_j)}}.$$ **Why parameter sharing helps preserve correlations.** Unlike approaches that independently parameterize probabilistic transitions at each block, DIRECTOR employs a unified spatiotemporal transition model with shared parameters across all timesteps. By jointly learning transitions through the distribution-matching objective in Eq. (10), DIRECTOR encourages consistent feature transformations across the network, which may help preserve cross-layer dependencies. As shown in Fig. 1(c), DIRECTOR maintains a correlation of 0.972 between $X_6$ and $X_0$, compared with 0.621 for KEP-All.
AuthorsOpenReview ↗

Response to Reviewer dQi1 (2/4)

[W2] and [Q8]

We thank the reviewers for their valuable suggestions. To further assess the contributions of DIRECTOR's key components, we conducted the following additional ablation studies:

  • A deterministic shared-block distillation baseline (same student, no stochastic transition). We remove the covariance head to disable stochastic transitions (ST). The results show that DIRECTOR without ST performs worse across all seven metrics, supporting the benefits of stochastic transitions.
MethodACC ↑AURC ↓AUROC ↑FPR95 ↓ECE ↓NLL ↓Brier ↓
ViT83.84±0.094.05±0.1186.42±0.3767.13±1.9812.51±0.2010.91±0.3928.03±0.15
DIRECTOR w/o ST86.00±1.023.01±0.4087.85±0.6062.38±2.389.23±0.616.63±0.4322.93±1.57
DIRECTOR86.27±0.582.90±0.1287.98±0.3262.28±0.929.03±0.446.41±0.1422.54±0.78
  • Removing the covariance-matching term for KEP backbone: As shown in the table below, the covariance-matching term improves uncertainty calibration on KEP-7/7, reducing ECE (5.51 to 4.92), NLL (4.83 to 4.70), and Brier (22.12 to 21.80).
MethodACC ↑AURC ↓AUROC ↑FPR95 ↓ECE ↓NLL ↓Brier ↓
KEP-7/782.68±0.124.46±0.0585.71±0.3466.90±1.786.95±0.365.89±0.1125.90±0.23
DIRECTOR w/o covariance matching85.10±0.473.33±0.2087.53±0.4562.72±0.965.51±0.794.83±0.1922.12±0.66
DIRECTOR85.24±0.283.32±0.1187.42±0.3663.49±1.584.92±0.814.70±0.1321.80±0.34
  • Varying the number of inferences samples. As shown in the table below, DIRECTOR consistently outperforms the pretrained ViT across all metrics on CIFAR-10, with improved uncertainty calibration as the number of inference samples increases, while maintaining stable predictive performance.
MethodACC ↑AURC ↓AUROC ↑FPR95 ↓ECE ↓NLL ↓Brier ↓
ViT83.84±0.094.05±0.1186.42±0.3767.13±1.9812.51±0.2010.91±0.3928.03±0.15
DIRECTOR - 186.27±0.582.90±0.1287.98±0.3262.28±0.929.03±0.446.41±0.1422.54±0.78
DIRECTOR - 286.28±0.612.89±0.1288.00±0.3462.32±1.098.99±0.486.39±0.1522.50±0.78
DIRECTOR - 586.27±0.602.89±0.1288.03±0.3162.17±0.958.98±0.466.38±0.1422.49±0.78
DIRECTOR - 1086.26±0.612.89±0.1288.04±0.3562.26±0.958.99±0.476.38±0.1422.48±0.77

We have updated these results to the manuscript, as highlighted in red color.

[Q4]

We acknowledge that our current framework does not provide a formal theoretical guarantee of convergence or a bound on error accumulation across transition steps. Similar to conventional diffusion models, which typically optimize tractable surrogate objectives without guaranteeing convergence of the learned sampling distribution under finite model capacity and imperfect optimization, DIRECTOR relies on empirical validation of its learned transition process.

Our extensive experiments across different datasets and architectures demonstrate consistent improvements in predictive performance and uncertainty calibration, supporting the practical effectiveness of the proposed approach.

[Q5]

We use a single inference sample for computational efficiency. As shown in our additional ablation study in [W2] and [Q8], using more samples can further improve uncertainty calibration. The inference times reported in Table 12 are measured in seconds over the entire test set, using a batch size of 128 samples during inference.

AuthorsOpenReview ↗

Response to Reviewer dQi1 (3/4)

[Q6] and [W3]

We thank the reviewer for highlighting these relevant works. We would like to clarify the differences between DIRECTOR and these methods.

MiniALBERT [1] compresses pretrained transformers into recursive students using several distillation objectives, including masked language modeling (MLM) loss, attention loss for aligning attention maps, cosine-based hidden-state loss, and KL divergence for matching output distributions. Its primary objective is model compression while preserving predictive performance.

Latent Flow Transformer (LFT) [2] learns a deterministic latent transport operator through flow matching and Flow Walking, primarily targeting model compression and the preservation of latent representations.

Unlike these methods, DIRECTOR focuses on uncertainty calibration through a two-stage framework. First, we structurally reconfigure the pretrained transformer to construct a probabilistic path with Gaussian feature transitions. We then distill these transitions into a unified probabilistic model that jointly parameterizes their means and covariances. Importantly, the distillation targets are the probabilistic transitions induced by our reconfigured architecture, rather than simply the intermediate representations of the original pretrained model. This enables uncertainty-aware feature sampling and propagation across transformer blocks.

Novelty and empirical validation. We clarify that our contribution is not parameter sharing itself, but its integration with Gaussian transition modeling, diffusion-inspired probabilistic reconfiguration, and uncertainty-aware learning. Furthermore, Table 17 provides a comparison with conventional knowledge distillation using the same DiT student architecture, where DIRECTOR achieves better predictive accuracy and calibration.

We appreciate the reviewer's suggestion and will revise the related work section to explicitly acknowledge these approaches and better distinguish DIRECTOR's contributions.

[1] Nouriborji et al., "MiniALBERT: Model Distillation via Parameter-Efficient Recursive Transformers," EACL 2023.

[2] Wu et al., "Latent Flow Transformer," arXiv:2505.14513, 2025.

[Q7] and [W4]

We sincerely thank the reviewer for their careful reading and thoughtful questions.

Regarding the identical MCD and ViT rows in Table 5, we acknowledge an oversight in our experimental configuration. Specifically, we use the TorchUncertainty implementation of MCD, which provides a last_layer option. We accidentally used setting last_layer=True in the OOD experiments, resulting in negligible differences between MCD and the original ViT. We have corrected this by setting last_layer=False, rerun the evaluation, and updated Table 5 accordingly (color red). The results still show that DIRECTOR achieves the best performance. We apologize for this oversight, which does not affect other reported results. For transparency and reproducibility, the code to reproduce all results in the main paper was already provided in the supplementary materials of our original submission.

Regarding the MCC results on CoLA, we would like to clarify the 40-run evaluation in Table 16:

  • DIRECTOR consistently outperforms both KEP backbones across all calibration metrics, supporting the robustness of its calibration improvements.
  • The MCC reversal is observed only for KEP-1/5. Over five runs, the MCC scores are 31.84 ± 1.88 (DIRECTOR) and 30.86 ± 2.03 (KEP). With 40 runs, the scores become 31.41 and 32.40, respectively, both within the previously reported standard deviation ranges. The small difference between these scores indicates that DIRECTOR remains competitive.

[W5]

We thank the reviewer for raising these concerns regarding computational cost. We would like to clarify the following points.

Inference overhead. We acknowledge that DIRECTOR incurs higher inference latency than the vanilla backbone (1.21s vs. 0.93s over the entire test set, using a batch size of 128), although it remains faster than some uncertainty-aware baselines. This overhead primarily arises from iterative stochastic inference despite parameter sharing. Nevertheless, DIRECTOR achieves lower inference memory consumption (0.104 GB vs. 0.151 GB), highlighting a trade-off between inference latency and memory efficiency.

Requirement for a GP-attention teacher. We clarify that DIRECTOR does not require a GP-attention teacher, as it can learn probabilistic transitions directly from deterministic representations, treating them as Gaussian means with zero variance (see our response to [Q2]).

Reviewer_9J3GOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes an uncertainty quantification method (DIRECTOR) tailored for (multi-head) transformer atchitectures. DIRECTOR relies on re-interpretation of the transformer as a series of nested Gaussian Process (GP) and deterministic transformations. The parameters of the GP are sampled randomly from a learned mean and covariance matrix, but for efficiency and expresivity the parameters are learned in a variational formulation in the spectra space predicted from input by a neural network. The parameters are trained on a log-likelihood based score, that is motivated by the similarity of this sequential Gaussian transformation to a (reverse) diffusion process. The paper evaluates their approach against an extensive battery of related works and across two modalities: images and NLP, and generally find it improves both accuracy and calibration statistics.

Critical strengths
  1. The formulation of DIRECTOR rests on a strong probabilistic foundation. DIRECTOR shows consistent and substantial improvements across multiple datasets, supported by thorough experiments and baseline comparisons.
  2. The idea of using interpret a transformer architecture as a stochastic process opens up further directions for improvements
  3. The paper is well written and clear.
  4. DIRECTOR works as a post-hoc method, which is well suited to the current paradigm of using pretrained foundation models for downstream tasks.
Critical weaknesses
  1. More discussion of computational scaling to larger models and datasets would be valuable. The introduction motivates DIRECTOR in the context of large-scale pretraining, but this framing seems somewhat overstated relative to the scale of the computer vision experiments. The main results use CIFAR, as does the vision experiment in the appendix’s “Large-Scale Experiments” section, although CIFAR is generally regarded as a small-scale benchmark in computer vision. Furthermore, the description of DIRECTOR as a lightweight post-hoc method is somewhat undercut by Table 14, where reconfiguration takes roughly half as long as training the ViT backbone. I appreciate that it may be faster than some competing uncertainty quantification methods, but the characterization still seems too strong, especially compared with methods such as temperature scaling. A larger dataset such as ImageNet would be interesting to see and convince me that the current emphasis is justified, as many methods can fail to scale to that kind of scale.
  2. Relatedly, after reading the paper, I would have liked a clearer understanding of the method’s computational complexity, either through a discussion of asymptotic scaling or a breakdown of computational requirements. Since the GP uses a sparse variational representation, it avoids the usual (O(N^3)) scaling of exact GP inference, where (N) is the number of training points. Is the main compute the student transition evaluation or the token length? The appendix provides some details, mainly for CIFAR-10, but corresponding information for the larger Qwen-2.5 experiment would also be useful.
  3. A clearer discussion of where DIRECTOR introduces approximations would help readers understand potential failure cases and identify directions for improvement. Especially, when the theoretical lower bound on the likelihood that is derived, is only approximated by squared errors to means and covariance factors and a classification loss.
Questions for authors
  1. Could the authors expand on the intuition that preserving feature correlations is beneficial for uncertainty quantification? This is presented as a positive property in Figure 1 and the Research Gap section.
  2. Does the setup require a specific transformer architecture in which the reconfigured blocks end with an MHSA block? Do the authors believe the framework can be extended to architectures using Gated DeltaNet blocks, such as Qwen3.5, or the combination of local and global attention in Gemma 3?
  3. Do the authors have an explanation for SGPA’s surprisingly poor performance on certain datasets—for example, its accuracy on CIFAR-10 relative to CIFAR-100 in Table 2, and its OOD detection performance on SVHN and LSUN relative to CIFAR-100 in Table 5? Since DIRECTOR builds on close to the same probabilistic formulation, why does it not exhibit the same fluctuations?
  4. Is there a specific reason why the authors chose to fine-tune an ImageNet-pretrained model on CIFAR-10 rather than evaluate DIRECTOR on ImageNet directly?

Some Minor Improvements:

  1. The layouts of Tables 1–5 could be improved for readability, for example by removing vertical lines, particularly those that do not connect fully.

  2. What is the timing unit in Table 12? Is the reported inference time measured per sample, per batch of 128 samples, or over the full evaluation set?

  3. How are the ensembles in Appendix A.6.2 constructed, and how many models are trained for each ensemble?

4 comments
AuthorsOpenReview ↗

Response to Reviewer 9J3G (1/3)

We sincerely thank the reviewer for their positive evaluation and constructive feedback. We would like to address their concerns as follows:

[W5]

We thank the reviewer for this constructive feedback. We agree that additional experiments on larger-scale datasets and a more careful discussion of computational efficiency would strengthen our work.

Scalability evaluation on ImageNet-1K. To address the reviewer's concern, we conducted additional experiments on ImageNet-1K using a pretrained ViT-L (google/vit-large-patch16-224). The results are summarized below:

MethodAcc (%)ECE (%)NLLBrier
ViT Large81.974.596.9926.12
DIRECTOR81.922.406.9525.99

As shown, DIRECTOR maintains comparable predictive accuracy to the original pretrained model (81.92% vs. 81.97%) while improving all three uncertainty calibration metrics. In particular, DIRECTOR reduces ECE from 4.59% to 2.40%, representing a relative reduction of approximately 47.7%, alongside improvements in NLL (from 6.99 to 6.95) and Brier score (from 26.12 to 25.99).

Computational efficiency. We acknowledge the reviewer's concern regarding the characterization of DIRECTOR as a lightweight post-hoc method. Although DIRECTOR does not require retraining the pretrained backbone, learning the unified transition model incurs non-negligible computational costs, as reported in Table 14. We agree that describing DIRECTOR as lightweight may overstate its computational efficiency, particularly compared with simpler calibration methods such as temperature scaling. Therefore, we will remove this characterization from the revised manuscript.

[W6]

We thank the reviewer for this valuable suggestion. We would like to clarify that DIRECTOR does not require explicitly converting a pretrained transformer into a GP-based model before distillation. Instead, DIRECTOR can directly distill the feature transformations of a standard pretrained transformer into a unified probabilistic transition model, without an additional GP-training stage. Therefore, the computational overhead primarily arises from training the unified transition model, implemented using a single-block DiT.

Computational complexity. For a sequence of (L) tokens with hidden dimension (d), each transition evaluation using the single-block DiT has a computational complexity of approximately (O(L^2d + Ld^2)). Since DIRECTOR reuses the same transition model across (T) reconfigured transformer blocks, a complete rollout requires (T) transition evaluations, resulting in an overall complexity of (O(T(L^2d + Ld^2))). Importantly, DIRECTOR does not require training separate transition models for individual blocks, and the computational overhead can be further controlled by selectively reconfiguring a subset of blocks.

Empirical computational costs. As reported in Appendix A.6.6 (Tables 12–14), DIRECTOR requires 1.75 hours and 3 GB of GPU memory for reconfiguration on CIFAR-10 using a single NVIDIA L40 GPU, excluding backbone training. For larger models such as Qwen-2.5, we reconfigure only the last three transformer blocks to reduce computational overhead, requiring approximately 50 minutes to train for 20 epochs using 8 NVIDIA L40 GPUs.

[W7]

We thank the reviewer for this constructive suggestion. We agree that clarifying the approximations in DIRECTOR would help readers better understand its limitations.

As detailed in Appendix A.5, we approximate the KL divergence in the variational objective using mean matching and Cholesky-factor matching losses, together with a performance-aware classification loss. We adopt a diagonal covariance parameterization to reduce computational complexity.

We have also investigated the impact of different covariance structures in Appendix A.6.5 (Table 11), comparing diagonal and full (dimension-wise) covariance parameterizations. The results show that the diagonal approximation achieves better predictive performance and uncertainty calibration while reducing training time from 2.50 to 1.75 hours. These findings empirically support our choice of covariance parameterization. Nevertheless, we acknowledge that these approximations may not fully capture complex dependencies among feature dimensions.

AuthorsOpenReview ↗

Response to Reviewer 9J3G (2/3)

[Q8]

We would like to clarify on the motivation that preserving the dependencies among intermediate representations established during pre-training is important for reliable uncertainty propagation. Specifically, feature transformations across successive transformer blocks are inherently dependent. Modeling uncertainty at individual blocks without adequately accounting for these dependencies may distort the propagated uncertainty, potentially leading to overconfident or underconfident predictions. In contrast, DIRECTOR distills these transformations into a unified spatiotemporal transition model, encouraging the preservation of cross-block relationships while propagating representation uncertainty throughout the network.

As illustrated in Fig. 1(c), DIRECTOR maintains stronger correlations between intermediate and final representations than KEP-All, suggesting better preservation of these relationships. Together with the improved calibration results in Tables 1 and 2, this provides empirical support for our motivation.

[Q9]

We thank the reviewer for this insightful question. We would like to clarify that the current formulation of DIRECTOR uses MHSA-based blocks to construct Gaussian transitions, as described in Section 2.1. However, the underlying principle of distilling intermediate feature transformations into a unified probabilistic transition model is not inherently restricted to standard MHSA architectures.

Extension to hybrid architectures. For Gemma 3, which combines local and global attention, DIRECTOR could potentially be extended by accounting for the different attention patterns when constructing probabilistic transitions. For Qwen3.5, which incorporates Gated DeltaNet blocks, an extension could selectively reconfigure the standard attention blocks while retaining the remaining blocks, or introduce an alternative probabilistic formulation for Gated DeltaNet.

We acknowledge that these extensions require further theoretical investigation and empirical validation, and consider them promising directions for future research.

[Q10]

We thank the reviewer for this insightful question. We agree that SGPA exhibits noticeable variations in predictive performance and OOD detection across datasets.

SGPA's performance variations. We believe these variations may arise from the trade-off between uncertainty calibration and predictive performance. In particular, SGPA models uncertainty at individual attention blocks without explicitly accounting for cross-block dependencies, which may affect the preservation of discriminative representations. On CIFAR-10 (Table 2), SGPA achieves a low ECE of 1.92% but substantially lower accuracy (75.59%) than the pretrained ViT (83.84%). Furthermore, its OOD detection performance varies across SVHN, CIFAR-100, and LSUN (Table 5), potentially reflecting different distribution shifts and dataset characteristics.

Why DIRECTOR exhibits more consistent improvements. Unlike SGPA, DIRECTOR distills intermediate feature transformations into a unified spatiotemporal transition model, encouraging the preservation of cross-block dependencies while jointly optimizing uncertainty propagation and predictive performance. To further investigate this, we directly applied DIRECTOR to an SGPA backbone, as reported in Appendix A.6.3 (Table 9). On CIFAR-10, DIRECTOR improves SGPA's accuracy from 75.59% to 82.65%, while reducing NLL from 7.11 to 5.30 and Brier score from 33.98 to 25.10, although ECE increases from 1.92% to 4.53%. These results suggest that DIRECTOR can mitigate SGPA's accuracy–calibration trade-off.

[Q11]

We thank the reviewer for this question. Due to limited computational resources, our initial experiments focused on fine-tuning an ImageNet-pretrained ViT-B/16 model on CIFAR-10, allowing us to evaluate DIRECTOR on a larger pretrained architecture while maintaining manageable computational costs.

We agree that evaluating DIRECTOR directly on ImageNet-1K would provide stronger evidence of its scalability. To address this concern, we also conducted additional experiments on ImageNet-1K, as detailed in our response to [W5]. The results demonstrate that DIRECTOR maintains comparable predictive accuracy while improving uncertainty calibration, providing further evidence of its effectiveness on larger-scale datasets.

AuthorsOpenReview ↗

Response to Reviewer 9J3G (3/3) (Minor Improvements)

[Q12]

We thank the reviewer for this helpful suggestion. We agree that the table layouts could be improved for readability and will revise the formatting of Tables 1–5 accordingly in the camera-ready version.

[Q13]

We thank the reviewer for pointing out this ambiguity. The inference times reported in Table 12 are measured in seconds over the entire test set, using a batch size of 128 samples during inference. We will revise the table caption to explicitly clarify the measurement unit and evaluation settings for better reproducibility.

[Q14]

We thank the reviewer for this question. In Appendix A.6.2, we evaluate Deep Ensembles by combining 5 independently trained models with different random seeds. At inference, we aggregate their predicted class probabilities to obtain the ensemble prediction. For DIRECTOR, we apply our diffusion-based reconfiguration to each ensemble member and similarly aggregate their predictions, ensuring a consistent comparison with the corresponding pretrained baselines.

We hope that our responses have adequately addressed the reviewer's concerns. We would sincerely appreciate it if the reviewer could reconsider their rating in light of our clarifications and additional experimental results.

Dear Reviewer 9J3G,

We would like to kindly point out that your follow-up comment appears to have been posted under our response to Reviewer dQi1. We have provided a response to your comment and hope it addresses your remaining concerns.

Best regards,

Authors

Reviewer_KhvrOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper presents a diffusion-based uncertainty quantification (UQ) approach for transformer models. In the proposed approach, each feature transformation within transformer is modelled as a probabilistic mapping, thus making the entire transformer model as a continuous stochastic process. To this end, the authors propose to reconfigure the architecture in a way that each transformation block ends with a MHSA block, as it allows the MHSA block to be Gaussian Process (GP)-parameretized -- outputting a (Gaussian) distribution instead of point estimate. Each transformation block in the transformer (also called a teacher) can be seen to induce a stochastic path with Gaussian transition, or a reverse diffusion process, going from timestep t=Tt=T (or data embedding) to timestep t=0t=0 (the feature representation before the classifier). A unified student DiT is trained to approximate this reverse process, thus making it a "diffusion-like" model. The student model is trained via minimizing the KL divergence between the teacher and student predictions, and a supervised loss using the labels.

The proposed method DIRECTOR, has been evaluated for small scale transformer models on multiple settings, such as in-distribution, distribution shift, and OOD detection, using image recognition datasets: CIFAR10 and 100, and on NLP datasets: IMDB and CoLA. It has been compared with several standard UQ baselines, ranging from Monte Carlo Dropout to Deep Gaussian Process methods. Experimental results demonstrate that the proposed method consistently outperforms the considered baselines in a majority of settings. The authors have also conducted a study using bigger ViT and LLMs, ablation study of the uncertainty-aware backbone and have reported the computational overhead.

Critical strengths
  • [S1] The idea of re-structuring the transformer, so that each transformation block ends with a MHSA operation is elegant, and makes the resulting model amenable to GP.
  • [S2] The idea of "diffusion-like" transition for quantifying uncertainty seems novel to the reviewer.
  • [S3] The proposed method is post-hoc, which makes it appealing from a re-training point of view.
  • [S4] The paper does a good job at introducing the preliminary concepts that are necessary to understand the contributions.
  • [S5] Adding DIRECTOR to existing UQ backbones improves performance across different metrics. Also it outperforms UQ baselines in general.
Critical weaknesses
  • [W1] The state-of-the-art comparison is somewhat limited. While the authors claim that they have established a new state-of-the-art (in L377), the most recent competitor, KEP, is a paper from 2024. It does not consider comparing with recent works on UQ such as Gabetni et al. "Ensembling Pruned Attention Heads For Uncertainty-Aware Efficient Transformers" ICLR 2026. The datasets used for the computer vision tasks is toy-ish. It does not consider ImageNet-1k. Moreover, it does not consider experiments in the multimodal setting, such as classification in CLIP-setting.
  • [W2] The so-called large-scale experiments using bigger models (ViT-B and Qwen-2.5) reported in the appendix is limiting, as they are run using CIFAR-10 and SST-2, which are considered small-scale datasets. Therefore, it is hard to judge is the proposed method is scalable to even bigger models trained on much bigger (both in terms of number of samples and image resolution for image classification) datasets.
  • [W3] From Table 1 one can see that the accuracy of the proposed method is consistently higher than the vanilla transformer/ViT baseline, and often by non-trivial margins. It is not clear why the accuracy improves when both the baselines have been trained with the same supervised objective?
  • [W4] It is not clear why diffusion-inspired UQ leads to a better calibrated model. The authors do mention that the reconfiguration enables an accurate flow of uncertainty propagation across the entire network (L093), but why does diffusion-based network can achieve a better calibrated model than any other UQ technique?
  • [W5] The related works section paragraph that discusses UQ & UC is confusing to a reader as it is not appropriately used to position the paper's contribution.
  • [W6] Deep ensembles is a simple and strong baseline in the UQ literature. Although it is understandable that deep ensembles requires several forward passes and computationally more expensive during inference, why is it not reported as baseline in the tables of the main paper?
Questions for authors
  • [Q1] Can you please explain why diffusion-inspired approach leads to better UQ? A theoretical justification would be convincing. This corresponds to [W4].
  • [Q2] Can you please compare with the most recent baseline mentioned in [W1], including the settings, datasets and backbones used in the aforementioned paper? This can help in demonstrating the true competitiveness of the proposed method as well as demonstrate its scalability to bigger datasets and models.
  • [Q3] Please explain why accuracy improves with the proposed method? This corresponds to [W3].
2 comments
AuthorsOpenReview ↗

Response to Reviewer Khvr (1/2)

We appreciate the reviewer’s constructive feedback and would like to address the concerns as follows:

[W1] and [Q2]

We thank the reviewer for highlighting recent advances in uncertainty quantification (UQ) and suggesting a comparison with Hydra Ensembles [1]. We agree that including more recent baselines and evaluating larger-scale settings would strengthen the empirical validation of DIRECTOR.

Comparison with Hydra Ensembles. To address this concern, we conducted additional experiments comparing DIRECTOR with Hydra Ensembles using a ViT backbone on CIFAR-10 (aaraki/vit-base-patch16-224-in21k-finetuned-cifar10 from HF).

MethodACCECENLLBrier
ViT-B-1697.88018.0212.5646.843
Hydra Ensembles97.62010.6141.8104.902
DIRECTOR97.1701.6111.1224.762

As shown in table above, DIRECTOR consistently outperforms Hydra Ensembles all three uncertainty calibration metrics (ECE, NLL, and Brier score). These additional results provide further evidence of DIRECTOR's competitiveness against recent UQ methods. We have included this comparison in the updated appendix.

Evaluation on larger-scale and multimodal settings. We acknowledge the reviewer's suggestion to extend the comparison to larger datasets (e.g., ImageNet-1k) and multimodal architectures (e.g., CLIP), as explored in [1]. While these evaluations would provide valuable insights into DIRECTOR's scalability and generalizability, reproducing the corresponding experimental settings for both methods would require substantial computational resources beyond those currently available to us. Therefore, we limit our direct comparison with Hydra Ensembles to the ViT/CIFAR-10 setting, while additionally evaluating DIRECTOR against a pretrained ViT model on ImageNet-1K, as detailed in our response below. We acknowledge that more comprehensive comparisons on larger-scale datasets and multimodal architectures remain an important direction for future work.

[W1] and [W2]

To address the scalability concerns, we conducted additional experiments on the ImageNet-1K using a pretrained ViT-L model (google/vit-large-patch16-224) from HF. Unlike CIFAR-10, ImageNet-1K contains approximately 1.28 million training images across 1,000 classes, providing a substantially more challenging benchmark with higher-resolution inputs (224 × 224).

MethodAcc (%)ECE (%)NLLBrier
ViT Large81.974.596.9926.12
DIRECTOR81.922.406.9525.99

As shown in the table, DIRECTOR maintains comparable predictive accuracy to the original pretrained model (81.92% vs. 81.97%) while improving all three uncertainty calibration metrics. In particular, DIRECTOR reduces ECE from 4.59% to 2.40%, representing a relative reduction of approximately 47.7%, alongside improvements in NLL (from 6.99 to 6.95) and Brier score (from 26.12 to 25.99). These additional results complement our existing experiments with larger models in the appendix and provide further empirical evidence of DIRECTOR's effectiveness on larger-scale datasets and higher-resolution images. Nevertheless, we acknowledge that evaluations involving even larger architectures and datasets would provide a more comprehensive assessment of scalability. Due to computational constraints, we leave such extensions for future work.

[W3] and [Q3]

We would like to clarify that the improvements in predictive accuracy arise from DIRECTOR's diffusion-based reconfiguration and its joint optimization of uncertainty propagation and predictive performance, rather than from the supervised objective alone.

Specifically, DIRECTOR interprets the intermediate feature transformations of a pre-trained transformer as samples from a probabilistic path and distills them into a unified spatiotemporal transition model. Unlike the original transformer, which processes features through deterministic attention blocks, DIRECTOR explicitly models representation uncertainty while preserving cross-block correlations established during pre-training. This provides a more coherent mechanism for propagating uncertainty across the network.

Furthermore, as described in Eqs. (10)-(12), DIRECTOR optimizes both a distribution-matching objective and a performance-aware supervised loss. The former encourages the learned transition model to preserve the probabilistic structure of the pre-trained representations, while the latter ensures that the generated representations remain informative for downstream prediction. Although both methods employ the same supervised classification objective, DIRECTOR additionally learns a unified transition model, implemented using a single-block DiT, that refines the mapping from input embeddings to predictive representations.

We believe that the combination of unified uncertainty propagation and performance-aware distillation contributes to the observed accuracy improvements, while simultaneously enhancing uncertainty calibration, as demonstrated in Tab. 1.

AuthorsOpenReview ↗

Response to Reviewer Khvr (2/2)

[W4] and [Q1]

We would like to clarify that the key advantage of DIRECTOR lies in its unified modeling of representation uncertainty across transformer blocks, rather than the diffusion-inspired architecture alone.

Existing UQ methods, such as KEP, model uncertainty at individual attention blocks but do not explicitly account for dependencies across blocks. In contrast, DIRECTOR distills these probabilistic transformations into a unified spatiotemporal transition model, enabling more coherent uncertainty propagation while preserving cross-block feature correlations established during pre-training. As illustrated in Fig. 1(c), DIRECTOR maintains stronger feature correlations across layers than KEP-All, supporting this motivation.

As derived in Eq. (9) and Appendix A.4, the negative log-likelihood of the learned final feature distribution is upper-bounded by the sum of transition-wise KL divergences and a constant entropy term. By minimizing these KL divergences, DIRECTOR encourages the learned probabilistic path to approximate the reference feature distribution. Furthermore, the performance-aware loss in Eqs. (11)–(12) ensures that the resulting representations remain informative for downstream predictions.

While this formulation does not theoretically guarantee better calibration than every alternative UQ method, it provides a principled mechanism for preserving and propagating representation uncertainty across the network. Its effectiveness is further supported by the improved calibration metrics (ECE, NLL, and Brier score) reported in Tables 1 and 2.

[W5]

We thank the reviewer for this constructive feedback. We will revise the Related work section to better distinguish existing approaches, highlight their limitations, and more clearly position DIRECTOR's contributions relative to prior work.

[W6]

We thank the reviewer for this suggestion. Due to space limitations in the main paper, we reported the comparison with Deep Ensembles in Appendix A.6.2 (Table 8), while also highlighting these results in the main text (Lines 429–431). We agree that Deep Ensembles is an important baseline and that presenting these results in the main paper would improve the completeness of our evaluation. Therefore, we will move this comparison to the main text in the camera-ready version.

[1] Gabetni, Firas, Giuseppe Curci, Andrea Pilzer, Subhankar Roy, Elisa Ricci, and Gianni Franchi. "Ensembling pruned attention heads for uncertainty-aware efficient transformers." In International Conference on Learning Representations, vol. 2026, pp. 23296-23334. 2026.

We hope that our responses have adequately addressed the reviewer's concerns. We would sincerely appreciate it if the reviewer could reconsider their rating in light of our clarifications and additional experimental results.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.