- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper addresses uncertainty calibration in transformers. Building on GP-based attention methods (SGPA, KEP-SVGP), whose attention outputs are Gaussian, the authors regroup the transformer blocks so that each block-to-block transition is Gaussian, making the layer stack resemble the reverse process of a diffusion model. They then distill all block-wise transitions into a single time-conditioned stochastic block that is applied repeatedly, trained by matching the teacher's transitions together with a task loss.
The claimed contributions are this diffusion-style reinterpretation, the distillation algorithm, and an empirical study. Experiments on small vision and language classification benchmarks compare the method with standard calibration baselines and GP-attention models, with additional results on distribution shift, OOD detection, and larger models. The authors report improved accuracy and calibration over the original backbones with fewer parameters.
Critical strengths
-
Parameter efficiency. The method replaces the full stack of transformer blocks with a single shared block, so the resulting uncertainty-aware model has fewer parameters and a smaller inference memory footprint than the original backbone. This matters to my assessment because existing uncertainty-aware transformers typically add parameters and memory on top of the base model, whereas this approach goes in the opposite direction.
-
Recovering the accuracy lost by GP-attention models. Transformers with GP-based attention in all blocks (SGPA, KEP-SVGP) are known to trade predictive accuracy for calibration. When the proposed method is applied on top of these models, accuracy is largely recovered while calibration remains competitive, and this holds across the in-distribution and distribution-shift experiments. This is consequential because it addresses the main practical drawback of this line of work and directly supports the paper's motivation.
Critical weaknesses
- Unclear source of improvement
- Eq. 9 only bounds how well the student matches the teacher, so it cannot explain gains beyond the teacher.
- For vanilla backbones the teacher is deterministic and there is no Gaussian transition to distill, yet the gains are similar.
- The key notion of "transition correlation" (Fig. 1c) is not defined.
- New components are not validated
- No ablation against a deterministic shared-block distillation baseline.
- A student trained with plain KD already exceeds the teacher (Table 17).
- The implemented loss (mean and Cholesky-factor MSE) differs from the derived KL objective.
- Limited novelty and missing prior work
- Distilling transformer layers into a single shared recursive block is known [1,2], but is neither cited nor compared against.
- Weak evidence for the calibration claims
- On ECE the method is best on only one of four datasets in Table 2 and is worse than temperature scaling on CIFAR-100.
- Large-scale experiments are single runs compared only with the original model.
- Some results are inconsistent: the MCD and ViT rows in Table 5 are identical, and with 40 runs (Table 16) the MCC gains over KEP backbones on CoLA reported in Table 1 disappear or reverse.
- Computational cost
- Inference is slower than the vanilla backbone despite fewer parameters (Table 12).
- A GP-attention teacher must be trained first.
[1] Nouriborji et al., "MiniALBERT: Model Distillation via Parameter-Efficient Recursive Transformers," EACL 2023. [2] Wu et al., "Latent Flow Transformer," arXiv:2505.14513, 2025.
Questions for authors
-
Eq. 9 is minimized when the student equals the teacher. Why, then, does the student outperform the teacher in both accuracy and calibration? What is the source of this gain?
-
For vanilla backbones (ViT, Transformer, and presumably ViT-B/16 and Qwen-2.5), the teacher is deterministic. What is in this case, how does Eq. 9 apply, and where does the student's variance come from?
-
How is the "correlation" in Fig. 1c defined and computed, and why does sharing parameters across steps preserve it?
-
Is there any theoretical guarantee of convergence? Specifically, since the matching loss uses teacher-forced inputs and a diagonal covariance, does minimizing it guarantee that the student's rolled-out path distribution converges to the teacher's, or bound the error accumulated over steps?
-
How many samples are used at inference, and are the inference times in Table 12 measured per sample?
-
How does the method relate to prior work on distilling transformer layers into a single shared recursive block (e.g., MiniALBERT, Latent Flow Transformer)?
-
Could the authors clarify the identical MCD and ViT rows in Table 5, and why the MCC gains over KEP backbones on CoLA in Table 1 disappear or reverse with 40 runs in Table 16?
-
Ablations isolating the new components:
- (a) a deterministic shared-block distillation baseline (same student, no stochastic transition);
- (b) removing the covariance-matching term for KEP backbones;
- (c) varying the number of inference samples.
-
Results on larger datasets and models (e.g., ImageNet-scale) with multiple runs and standard baselines such as temperature scaling and deep ensembles. The current large-scale results are single runs compared only with the original model.
9 comments
Response to Reviewer dQi1 (2/4)
[W2] and [Q8]
We thank the reviewers for their valuable suggestions. To further assess the contributions of DIRECTOR's key components, we conducted the following additional ablation studies:
- A deterministic shared-block distillation baseline (same student, no stochastic transition). We remove the covariance head to disable stochastic transitions (ST). The results show that DIRECTOR without ST performs worse across all seven metrics, supporting the benefits of stochastic transitions.
| Method | ACC ↑ | AURC ↓ | AUROC ↑ | FPR95 ↓ | ECE ↓ | NLL ↓ | Brier ↓ |
|---|---|---|---|---|---|---|---|
| ViT | 83.84±0.09 | 4.05±0.11 | 86.42±0.37 | 67.13±1.98 | 12.51±0.20 | 10.91±0.39 | 28.03±0.15 |
| DIRECTOR w/o ST | 86.00±1.02 | 3.01±0.40 | 87.85±0.60 | 62.38±2.38 | 9.23±0.61 | 6.63±0.43 | 22.93±1.57 |
| DIRECTOR | 86.27±0.58 | 2.90±0.12 | 87.98±0.32 | 62.28±0.92 | 9.03±0.44 | 6.41±0.14 | 22.54±0.78 |
- Removing the covariance-matching term for KEP backbone: As shown in the table below, the covariance-matching term improves uncertainty calibration on KEP-7/7, reducing ECE (5.51 to 4.92), NLL (4.83 to 4.70), and Brier (22.12 to 21.80).
| Method | ACC ↑ | AURC ↓ | AUROC ↑ | FPR95 ↓ | ECE ↓ | NLL ↓ | Brier ↓ |
|---|---|---|---|---|---|---|---|
| KEP-7/7 | 82.68±0.12 | 4.46±0.05 | 85.71±0.34 | 66.90±1.78 | 6.95±0.36 | 5.89±0.11 | 25.90±0.23 |
| DIRECTOR w/o covariance matching | 85.10±0.47 | 3.33±0.20 | 87.53±0.45 | 62.72±0.96 | 5.51±0.79 | 4.83±0.19 | 22.12±0.66 |
| DIRECTOR | 85.24±0.28 | 3.32±0.11 | 87.42±0.36 | 63.49±1.58 | 4.92±0.81 | 4.70±0.13 | 21.80±0.34 |
- Varying the number of inferences samples. As shown in the table below, DIRECTOR consistently outperforms the pretrained ViT across all metrics on CIFAR-10, with improved uncertainty calibration as the number of inference samples increases, while maintaining stable predictive performance.
| Method | ACC ↑ | AURC ↓ | AUROC ↑ | FPR95 ↓ | ECE ↓ | NLL ↓ | Brier ↓ |
|---|---|---|---|---|---|---|---|
| ViT | 83.84±0.09 | 4.05±0.11 | 86.42±0.37 | 67.13±1.98 | 12.51±0.20 | 10.91±0.39 | 28.03±0.15 |
| DIRECTOR - 1 | 86.27±0.58 | 2.90±0.12 | 87.98±0.32 | 62.28±0.92 | 9.03±0.44 | 6.41±0.14 | 22.54±0.78 |
| DIRECTOR - 2 | 86.28±0.61 | 2.89±0.12 | 88.00±0.34 | 62.32±1.09 | 8.99±0.48 | 6.39±0.15 | 22.50±0.78 |
| DIRECTOR - 5 | 86.27±0.60 | 2.89±0.12 | 88.03±0.31 | 62.17±0.95 | 8.98±0.46 | 6.38±0.14 | 22.49±0.78 |
| DIRECTOR - 10 | 86.26±0.61 | 2.89±0.12 | 88.04±0.35 | 62.26±0.95 | 8.99±0.47 | 6.38±0.14 | 22.48±0.77 |
We have updated these results to the manuscript, as highlighted in red color.
[Q4]
We acknowledge that our current framework does not provide a formal theoretical guarantee of convergence or a bound on error accumulation across transition steps. Similar to conventional diffusion models, which typically optimize tractable surrogate objectives without guaranteeing convergence of the learned sampling distribution under finite model capacity and imperfect optimization, DIRECTOR relies on empirical validation of its learned transition process.
Our extensive experiments across different datasets and architectures demonstrate consistent improvements in predictive performance and uncertainty calibration, supporting the practical effectiveness of the proposed approach.
[Q5]
We use a single inference sample for computational efficiency. As shown in our additional ablation study in [W2] and [Q8], using more samples can further improve uncertainty calibration. The inference times reported in Table 12 are measured in seconds over the entire test set, using a batch size of 128 samples during inference.
Response to Reviewer dQi1 (3/4)
[Q6] and [W3]
We thank the reviewer for highlighting these relevant works. We would like to clarify the differences between DIRECTOR and these methods.
MiniALBERT [1] compresses pretrained transformers into recursive students using several distillation objectives, including masked language modeling (MLM) loss, attention loss for aligning attention maps, cosine-based hidden-state loss, and KL divergence for matching output distributions. Its primary objective is model compression while preserving predictive performance.
Latent Flow Transformer (LFT) [2] learns a deterministic latent transport operator through flow matching and Flow Walking, primarily targeting model compression and the preservation of latent representations.
Unlike these methods, DIRECTOR focuses on uncertainty calibration through a two-stage framework. First, we structurally reconfigure the pretrained transformer to construct a probabilistic path with Gaussian feature transitions. We then distill these transitions into a unified probabilistic model that jointly parameterizes their means and covariances. Importantly, the distillation targets are the probabilistic transitions induced by our reconfigured architecture, rather than simply the intermediate representations of the original pretrained model. This enables uncertainty-aware feature sampling and propagation across transformer blocks.
Novelty and empirical validation. We clarify that our contribution is not parameter sharing itself, but its integration with Gaussian transition modeling, diffusion-inspired probabilistic reconfiguration, and uncertainty-aware learning. Furthermore, Table 17 provides a comparison with conventional knowledge distillation using the same DiT student architecture, where DIRECTOR achieves better predictive accuracy and calibration.
We appreciate the reviewer's suggestion and will revise the related work section to explicitly acknowledge these approaches and better distinguish DIRECTOR's contributions.
[1] Nouriborji et al., "MiniALBERT: Model Distillation via Parameter-Efficient Recursive Transformers," EACL 2023.
[2] Wu et al., "Latent Flow Transformer," arXiv:2505.14513, 2025.
[Q7] and [W4]
We sincerely thank the reviewer for their careful reading and thoughtful questions.
Regarding the identical MCD and ViT rows in Table 5, we acknowledge an oversight in our experimental configuration. Specifically, we use the TorchUncertainty implementation of MCD, which provides a last_layer option. We accidentally used setting last_layer=True in the OOD experiments, resulting in negligible differences between MCD and the original ViT. We have corrected this by setting last_layer=False, rerun the evaluation, and updated Table 5 accordingly (color red). The results still show that DIRECTOR achieves the best performance. We apologize for this oversight, which does not affect other reported results. For transparency and reproducibility, the code to reproduce all results in the main paper was already provided in the supplementary materials of our original submission.
Regarding the MCC results on CoLA, we would like to clarify the 40-run evaluation in Table 16:
- DIRECTOR consistently outperforms both KEP backbones across all calibration metrics, supporting the robustness of its calibration improvements.
- The MCC reversal is observed only for KEP-1/5. Over five runs, the MCC scores are 31.84 ± 1.88 (DIRECTOR) and 30.86 ± 2.03 (KEP). With 40 runs, the scores become 31.41 and 32.40, respectively, both within the previously reported standard deviation ranges. The small difference between these scores indicates that DIRECTOR remains competitive.
[W5]
We thank the reviewer for raising these concerns regarding computational cost. We would like to clarify the following points.
Inference overhead. We acknowledge that DIRECTOR incurs higher inference latency than the vanilla backbone (1.21s vs. 0.93s over the entire test set, using a batch size of 128), although it remains faster than some uncertainty-aware baselines. This overhead primarily arises from iterative stochastic inference despite parameter sharing. Nevertheless, DIRECTOR achieves lower inference memory consumption (0.104 GB vs. 0.151 GB), highlighting a trade-off between inference latency and memory efficiency.
Requirement for a GP-attention teacher. We clarify that DIRECTOR does not require a GP-attention teacher, as it can learn probabilistic transitions directly from deterministic representations, treating them as Gaussian means with zero variance (see our response to [Q2]).
Response to Reviewer dQi1 (1/4)
We appreciate the reviewer’s constructive feedback and would like to address concerns as follows:
We thank the reviewer for this insightful question. We agree that Eq. (9) only provides a theoretical justification for matching the teacher's feature distribution and does not guarantee improvements beyond the teacher. We would like to clarify that DIRECTOR's improvements arise from its joint optimization of distribution matching and predictive performance, rather than from the distillation objective alone.
Specifically, DIRECTOR does not solely minimize Eq. (9). As described in Eqs. (11)–(12), we additionally introduce a performance-aware supervised loss that directly optimizes predictions generated by the unified transition model. Consequently, the student is not constrained to exactly reproduce the teacher's representations, but can learn task-informative representations that potentially improve predictive accuracy.
Furthermore, unlike the original pretrained model, DIRECTOR employs a unified probabilistic transition model that enables uncertainty propagation across transformer blocks. We believe that combining this probabilistic architecture with the distribution-matching and performance-aware objectives contributes to the observed improvements in both accuracy and calibration.
We would like to clarify that DIRECTOR does not require a GP-reparameterized teacher and can directly distill feature transformations from a deterministic pretrained backbone.
Specifically, for vanilla backbones (e.g., ViT, Transformer, ViT-B/16, and Qwen-2.5), we treat each deterministic intermediate representation as the mean of a degenerate Gaussian distribution with zero variance. DIRECTOR then learns a unified probabilistic transition model, whose mean is encouraged to match the teacher's representations, while its variance is parameterized and learned by the student.
As detailed in Appendix A.5, the transition model is trained using mean matching, covariance matching, and a performance-aware supervised loss. Thus, the student is not restricted to reproducing the deterministic teacher's predictions, which may explain the observed improvements in accuracy and calibration.
We would like to clarify the definition and motivation of transition correlation. In Fig. 1(c), transition correlation refers to the empirical correlation between intermediate feature representations at different transformer depths. Specifically, we measure the correlations between X6 and subsequent representations X4, X2, and X0 on CIFAR-10. We define the correlation between two feature representations using the Pearson correlation coefficient:
=\frac{\operatorname{Cov}(X_i,X_j)} {\sqrt{\operatorname{Var}(X_i)\operatorname{Var}(X_j)}}.$$ **Why parameter sharing helps preserve correlations.** Unlike approaches that independently parameterize probabilistic transitions at each block, DIRECTOR employs a unified spatiotemporal transition model with shared parameters across all timesteps. By jointly learning transitions through the distribution-matching objective in Eq. (10), DIRECTOR encourages consistent feature transformations across the network, which may help preserve cross-layer dependencies. As shown in Fig. 1(c), DIRECTOR maintains a correlation of 0.972 between $X_6$ and $X_0$, compared with 0.621 for KEP-All.