acceptodds
Under review as a conference paper at ICLR 2027

Prism Transformer: Progressive Head Schedules for Hierarchical Attention Processing

Abstract

Multi-head attention conventionally partitions the hidden dimension equally across all heads at every layer, enforcing an identical representational subspace dimension () throughout the model’s depth. In this work, we investigate whether this uniform allocation limits the representational capacity available to different stages of a Transformer. To address this, we introduce the Prism Transformer, a novel architectural paradigm that replaces the static, uniform head configuration with a progressive head schedule. By monotonically increasing the head count across layers, the Prism Transformer establishes a local-to-global representational hierarchy: early layers leverage fewer, wider heads to capture complex, local compositional patterns, while deeper layers deploy more, narrower heads to refine these representations into specialized linguistic features. Crucially, this structural shift is parameter-neutral and theoretically compute-neutral, preserving identical weight matrices and theoretical FLOP budgets as the standard Transformer. Across three model scales (124M, 354M, and 757M), the Prism Transformer consistently achieves lower validation loss than uniform baselines while maintaining identical measured training throughput on our 8× H100 GPU setup, alongside improved performance on downstream zero-shot benchmarks including PIQA, HellaSwag, ARC-Easy, and WinoGrande. Our findings suggest that non-uniform subspace allocation can improve the use of model capacity within the standard Transformer parameter and FLOP budget. Our code is available at https://anonymous.4open.science/r/PrismTransformer-593A/.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Transformer Architectures, Multi-Head Attention, Large Language Models, Efficient Deep Learning, Neural Network Architecture Design, Attention Mechanisms

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_fsp1OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

Prism Transformer progressively increases attention head counts across layers while keeping model width fixed. It achieves modest gains across three model scales without increasing parameters or major computational costs. Its head schedules are manually designed.

Critical strengths
  1. The design is clear and easy to understand and follow.

  2. Validation loss improves across three model scales, with results reported over three random seeds.

  3. Reversed schedules and uniformly wide heads provide useful controls supporting the importance of head allocation across depth.

Critical weaknesses
  1. The paper’s contribution appears too limited to support its acceptance. It lacks any technical design, merely proposing the use of varying numbers of attention heads across different layers, with the specific schedules manually chosen based on empirical ablations. Although gains are demonstrated across three model scales, the evidence remains insufficient to establish whether these scheduling rules generalize to broader architectures, context lengths, and training settings.

  2. The quality of Figure 1 is too low. It takes up nearly half a page yet lacks sufficient information.

  3. I notice that changing head dimensions also changes RoPE frequency grids, preventing clear attribution of gains to head allocation alone.

Questions for authors

I think the contributions of this work are insufficient to support its acceptance. Please refer to the Critical Weaknesses.

3 comments
AuthorsOpenReview ↗

Response to Reviewer Concerns

Response to Weakness 1:

We thank the reviewer for raising this concern. We agree that Prism introduces a relatively simple modification to standard MHA. However, its contribution lies in identifying and investigating an under-explored architectural assumption: that every Transformer layer should use the same number of attention heads and per-head dimension.

Although the implementation is simple, determining whether uniform head allocation is optimal across network depth requires systematic investigation. Our motivation is that representations evolve throughout a network, from local compositional patterns in earlier layers towards broader contextual integration in deeper layers. This motivates investigating whether attention capacity should also be allocated differently across depth.

Importantly, our findings go beyond simply introducing non-uniform head counts. Our ablations over ten configurations demonstrate that the direction and structure of allocation matter. Reversing the exact Prism schedule (using the same head dimensions in the opposite order) degrades validation loss from 3.3604 to 3.3857, while uniformly wide heads also underperform the standard baseline (3.3810 vs. 3.3748). This suggests that improvements arise from progressive ordering rather than non-uniformity or wider heads alone.

We further demonstrate transferability of the identified design principles across three model scales, deriving the Medium and Large schedules without independently searching for optimal configurations. Additionally, experiments with GQA demonstrate that the approach is compatible with a different attention mechanism, improving validation loss within both MHA and GQA configurations.

We also conducted additional GQA experiments at the medium scale (24 layers, hidden dimension 1024, 16 query heads, 1 seed), and at different settings, evaluating query-to-KV head ratios of 2 and 4.

GQA ratioBaseline GQAPrism GQA
2:13.00482.9971
4:13.01443.0072

The results show that the progressive head schedule consistently improves validation loss over the corresponding GQA baseline at both grouping ratios.

We acknowledge that evaluations across longer contexts, encoder architectures, and larger scales remain future directions and will clarify these limitations. Nevertheless, we believe the controlled ablations and cross-scale and cross-attention results establish depth-wise attention-capacity allocation as a meaningful architectural design dimension.

Response to Weakness 2:

We agree that Figure 1 underutilizes its space. We will redesign it as a clearer architectural overview illustrating head-count and per-head-dimension transitions across depth, explicitly showing that total model and projection widths remain fixed while individual subspaces progressively narrow.

Response to Weakness 3:

We thank the reviewer for noting this potential confound, as varying per-layer head dimensions in standard RoPE alters the frequency grid.

To isolate positional encoding effects, we conducted an additional controlled experiment at the Medium scale (354M parameters, 24 layers, d_model = 1024, seed 1120), using a fixed-band RoPE configuration.

Instead of constructing a separate RoPE frequency grid for each head dimension, we use a single shared 64-dimensional rotary module across all layers. Wider heads are partitioned into consecutive 64-dimensional chunks, with each chunk receiving the same baseline frequency spectrum. This keeps the frequency grid fixed while retaining progressive attention-capacity allocation.

ConfigurationValidation loss
Uniform baseline2.9954
Prism (standard RoPE)2.9896
Prism (fixed-band RoPE)2.9889

The fixed-band RoPE performs marginally better than the standard Prism. This suggests that the observed improvement does not rely on introducing additional RoPE frequency bands in wider early heads.

We retain dimension-specific RoPE in our main experiments because it provides greater flexibility in selecting intermediate head dimensions and constructing smooth schedules. A single shared frequency band must divide every head dimension, restricting the admissible schedules; for example, our Large configuration also uses a 96-dimensional head stage.

We will include this additional analysis and clarify the motivation for our RoPE implementation in the revised manuscript.

We sincerely thank the reviewer for their detailed feedback and recognition of our design clarity and results. We hope the additional GQA and fixed-band RoPE experiments, together with the proposed revisions, addresses the concerns raised. We hope this work encourages further investigation into alternative head-allocation strategies for improving Transformer representations and efficiency. If the reviewer finds these clarifications satisfactory, we would be grateful if they would consider revisiting their overall assessment and adjusting their rating accordingly.

Thank you for the response. My concerns remain unresolved. The experiments are also not rigorous or comprehensive enough. Validation loss is not a good evaluation metric. Readers cannot easily determine the model’s performance gains on the target tasks from validation loss alone. Overall, I believe the quality of this paper needs improvement. These issues are unlikely to be resolved during the rebuttal period. Therefore, I will keep my score.

AuthorsOpenReview ↗

Additional Analysis and Experiments

Thank you for your follow-up and for sharing your remaining concerns. We understand your assessment and have made further revisions to strengthen the empirical analysis and presentation of the manuscript.

In particular, we have made the following changes:

1. Additional representation analysis

We have added three complementary representation analyses in Section 4.3 (Figure 5): Few-NERD entity-type linear probes, logit-lens analysis, and CKA representational alignment. These analyses examine how entity-type information becomes accessible across layers, how intermediate-layer predictions relate to final-layer predictions, and how representations align across model depths. We believe these provide additional insight into the representational effects of progressive head allocation, beyond validation loss and attention-distance measurements.

2. Evaluation beyond validation loss

We would also like to clarify that our evaluation includes downstream task benchmarks in addition to pre-training validation loss. Figure 2 and Appendix A (Table 3) reports zero-shot results across six benchmarks (HellaSwag, PIQA, WinoGrande, ARC-Easy, BLiMP, and WikiText) for all three model scales, averaged over three independent training seeds. The results show that Prism generally preserves or modestly improves downstream performance.

3. Additional experimental controls

We have included the fixed-band RoPE experiment in Appendix F.2 to investigate whether the observed gains depend on changes to the RoPE frequency grid. The additional Medium-scale experiment (seed 1120) yields the following results:

ConfigurationRoPE settingValidation loss ↓
Uniform baselineStandard2.9954
PrismDimension-specific RoPE2.9896
PrismFixed-band RoPE2.9889

The results show that Prism with fixed-band RoPE performs marginally better than standard Prism. Although this experiment uses a single seed, it provides no evidence that the observed gains depend on the additional RoPE frequency bands.

We have also included additional Medium-scale GQA experiments in Appendix E, evaluating query-to-KV head ratios of 2:1 and 4:1. These experiments extend our evaluation to different attention configurations beyond the standard MHA setup.

4. Figure revision

We have redesigned Figure 1 as a more compact architectural overview, illustrating the progressive head allocation across layers while preserving the overall model and projection dimensions.

We acknowledge that broader evaluations across architectures, context lengths, and training settings remain important directions for future work. We have clarified these limitations in the revised manuscript.

We sincerely appreciate the reviewer's detailed feedback. Although we understand that these revisions may not fully resolve your concerns, we hope they improve the clarity, empirical support, and overall presentation of the work.

Reviewer_vEfJOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes a progressive attention head schedule for decoder-only Transformers. Early layers use fewer, wider heads, and later layers gradually return to the baseline head count, while projection dimensions and parameter counts remain unchanged. Experiments on 124M, 354M, and 757M models show lower validation loss and mostly small improvements on downstream tasks, with matching reported training throughput. Schedule ablations and attention-distance measurements examine the effects of this allocation, and a small-scale experiment tests its compatibility with GQA.

Critical strengths

The modification is simple to implement and improves validation loss at all three model sizes without increasing parameters or the dominant matrix-multiplication FLOPs. The comparisons use matched training budgets and three random seeds. Although the gains are modest, their consistency makes the result useful, particularly given the low implementation cost.

The schedule ablations provide support for the proposed ordering. Both reversing the schedule and using wide heads throughout the network perform worse than the uniform baseline. The medium and large schedules also improve performance without a separate search at each scale, providing some evidence that the small-model findings transfer.

Critical weaknesses

My main concern is the limited evaluation under GQA. The main experiments use MHA, while the GQA comparison covers only the small model with a query-to-KV head ratio of 2. More experiments at the medium and large scales, and with different grouping ratios, are needed to establish whether the gains extend to GQA.

The attention-distance analysis is consistent with the proposed explanation, but does not directly show that wider early heads learn better local representations. A shorter average attention distance indicates a preference for nearby tokens, rather than the quality of the information extracted from them. Similarly, longer attention distances in subsequent layers do not establish that these layers integrate information more effectively. The schedule ablations support the effectiveness of the proposed ordering, but the connection between these attention patterns and the performance gains remains uncertain. Direct evaluation of local representations or controlled local-composition tasks would provide stronger support for this interpretation.

Questions for authors

Have you tested the progressive schedule with GQA at the medium or large scale, or with query-to-KV head ratios other than 2? If so, do the gains and preferred schedules remain consistent?

3 comments
AuthorsOpenReview ↗

Response to Reviewer Concerns

Thank you for raising this concern. To address it, we conducted additional GQA experiments at the medium scale (24 layers, hidden dimension 1024, 16 query heads, 1 seed), evaluating both query-to-KV head ratios of 2 and 4 under the same training setup.

The results are as follows:

GQA ratio Baseline GQA Prism GQA

2:1 3.0048 2.9971

4:1 3.0144 3.0072

The results show that the progressive head schedule consistently improves validation loss over the corresponding GQA baseline at both grouping ratios. Furthermore, after the initial early training phase, the Prism GQA loss curve consistently remains below the corresponding baseline GQA curve throughout training.

We will include these additional experiments in the revised manuscript to strengthen the evaluation of Prism Transformer under GQA.

AuthorsOpenReview ↗

Added Additional Revisions and experiments

Thank you for your feedback. We have made additional revisions and conducted further experiments to address both concerns.

1. Extended GQA Evaluation

Following our previous response, we have included the additional Medium-scale GQA experiments in Appendix E. These cover both query-to-KV head ratios of 2:1 and 4:1.

The results are summarized below:

ScaleQuery:KV ratioUniform GQAPrism GQASeeds
Small (124M)2:13.3868 ± 0.00463.3790 ± 0.00253
Medium (354M)2:13.00482.99711
Medium (354M)4:13.01443.00721

The results show consistent validation-loss improvements with the progressive schedule across both tested grouping ratios at the Medium scale.

We acknowledge that the current evaluation does not include Large-scale GQA experiments or multi-seed replications at the Medium scale. Due to the limited time available during the rebuttal period, we could not complete these additional runs. We plan to extend the evaluation to the Large scale and perform multi-seed experiments, incorporating the results into the final manuscript.

2. Additional Representation Analysis

We agree with the reviewer that attention distance alone does not establish the quality of the information learned by individual heads or demonstrate more effective local information integration.

To investigate the representational differences more directly, we have added a complementary representation analysis in Section 4.3 (Figure 5), covering three aspects:

  • Few-NERD entity-type linear probes: The Few-NERD entity-type probes show that Prism achieves higher probe accuracy across several intermediate layers, with an earlier increase in linearly accessible entity-type information than the baseline. This suggests that Prism makes certain semantic representations accessible earlier in the network, potentially allowing subsequent layers to focus more on developing other aspects of the representations. However, both architectures reach comparable probe accuracy in the final layers, indicating that the primary difference lies in the depth at which entity-type information becomes accessible rather than its final linear accessibility.

  • Logit-lens analysis: Prism exhibits lower KL divergence between intermediate-layer and final-layer predictive distributions across most early and intermediate layers. This indicates that its early-layer predictions more closely approximate the later-layer predictions.

  • CKA representational alignment: The CKA analysis reveals a shift in cross-model layer alignment, suggesting that some shallower Prism representations are more similar to representations at deeper baseline layers, effectively increasing the model’s depth.

The representation analyses cover three seed-matched Prism/baseline model pairs at the Medium scale and two at the Large scale. The third Large-scale run had not completed training within the rebuttal period, so its checkpoint was not available for inclusion in the analysis. We will include the results from this additional run in the final manuscript.

We sincerely thank the reviewer for highlighting these limitations. We hope these revisions adequately address the concerns raised.

Thank you for the additional GQA experiments and representation analyses. These strengthen the empirical evidence and address my concerns about applicability to GQA. The representation results support the proposed explanation, although they do not fully establish that wider early heads improve local representations or that this drives the performance gains. I maintain my weak accept recommendation.

Reviewer_rEhnOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper discusses the uniform distribution of head dimensions in Transformer models versus following a pyramid-like distribution, where shallower layers maintain fewer heads with larger head dimensions while deeper layers use more heads with smaller head dimensions, thereby partitioning the head subspaces across different layers of the model. The paper compares results on three models ranging from 130M to 760M scale. Compared with models using uniform partitioning, their training loss shows a certain degree of reduction. Meanwhile, on downstream tasks, they match or exceed the performance of the original models.

Critical strengths
  1. The paper points out that head dimensions should be kept at different settings, so that the model can better utilize parameter dimensions to distinguish subspace information. In shallow layers, all dimensions should be interacted with more fully, so as to use a larger subspace size. In deep layers, due to the high abstraction of semantics, fewer dimensions can be used while retaining more categories. This idea is relatively feasible.

  2. The paper's current experimental results can basically support its viewpoint.

Critical weaknesses
  1. The paper has relatively little substantive content, and the figures and much of the explanatory text are rather superficial. It is questionable whether the corresponding sections need to be presented at such length; the writing may need to be restructured so that more meaningful parts are included in the main text, such as Fig. 5, which is needed in the main text.

  2. The paper's conclusion is highly similar to that of the variable-width Transformer. Although the latter mainly proposes considerations regarding the partitioning of the entire dimension, it needs to be taken into account.

  3. In Table 5, the paper only tests the uniform partitioning schemes of h=15 and h=3, without clearly showing the results of the others. At the same time, it can be seen from the different settings that, apart from the negative gain brought by the reverse ordering of dimensions, there is no obvious principle of benefit regarding how the forward dimensions are ordered.

Questions for authors

see weakness

4 comments
AuthorsOpenReview ↗

Response to Reviewer Concerns and Clarifications on Prism Transformer

Response to Weakness 1:

We thank the reviewer for this feedback. We agree that the original presentation could better emphasize the empirical analysis of the proposed architecture. We agree that Figure 5 provides important insight into how progressive head allocation changes attention behaviour across network depth and deserves greater prominence.

We will therefore move it and its accompanying analysis from the appendix into the main paper. We will also reduce repetitive architectural motivation and devote more space to the empirical investigation, particularly the relationship between head allocation, attention behaviour, and model performance.

Response to Weakness 2:

We agree that both works share the broader motivation that uniform architectural allocation across Transformer depth may be suboptimal. However, we believe that the two approaches investigate fundamentally different architectural dimensions.

Variable-Width Transformers modifies the overall hidden width of individual blocks, varying both attention and MLP dimensions and requiring non-standard residual-stream construction. Its objective is to redistribute computational capacity across depth.

In contrast, Prism preserves the global model dimension, residual-stream width, and all dense projection and MLP matrix dimensions. It modifies only the partitioning of the attention representation by progressively increasing the number of heads across layers. Thus, Prism investigates how attention representation capacity should be distributed across depth independently of block width.

This distinction also has practical implications. Variable-Width Transformers authors identifies the need for heterogeneous matrix-shape kernel optimization, additional residual-stream operations, and challenges with conventional tensor and pipeline parallelism. Prism avoids the variable-width residual construction and retains standard dense projection shapes. Furthermore, by constraining head dimensions to configurations supported by fused attention kernels, we demonstrate identical measured training throughput and wall-clock time to the uniform baseline across all 3 model scales on eight H100 GPUs.

These approaches are also not mutually exclusive: progressive head allocation could potentially be incorporated into variable-width Transformer blocks. However, Prism isolates attention-head allocation as an independent architectural design dimension, allowing us to study its representational effects without simultaneously changing MLP capacity or the overall block width.

We will revise the related-work and discussion sections to acknowledge Variable-Width Transformers, clarify these architectural distinctions, and discuss their complementary nature.

Response to Weakness 3:

We thank the reviewer for this observation and agree that the role of uniform head configurations can be clarified.

The two uniform configurations in Table 5 were selected as reference points. Uniform (h=12) represents the conventional Transformer baseline, while uniform (h=3) uses the same initial head count as Prism, corresponding to a wider per-head dimension ((d_h=256)), but maintains it across all layers. This control isolates whether gains arise simply from wider attention heads rather than their allocation across depth.

The results show that uniform (h=3) underperforms the conventional (h=12) baseline (3.3810 vs. 3.3748), whereas the progressive Prism schedule achieves 3.3604. Furthermore, our reverse-order control uses exactly the same set of head configurations as Prism but places them in the opposite order, resulting in a validation loss of 3.3857. The difference between these two configurations provides evidence that the direction of head allocation, rather than non-uniformity alone, is important.

We acknowledge that additional uniform configurations would better characterize this design space. We will therefore extend the ablation with uniform (h=6) and (h=8) configurations, keeping the model width, parameter budget, training tokens, and optimization settings unchanged. We will also clarify the interpretation of the existing progressive-schedule experiments.

We sincerely thank the reviewer for their valuable feedback. We hope these clarifications and revisions address the concerns raised and strengthen the paper. If the reviewer finds our responses satisfactory, we would be grateful if they would consider revisiting their overall assessment and adjusting their rating accordingly.

Reviewer_rEhnOpenReview ↗

Response

Thank you for the authors' response. However, my questions have not yet been well addressed. Please note that ICLR allows revised versions of the paper to be uploaded, but regarding my first question about the paper's writing, you have not made corresponding revisions, and I cannot judge whether the improvements match my views. Second, the corresponding results have not yet been supplemented, and at present I am unable to respond to this.

AuthorsOpenReview ↗

Added revised version of paper and experiments

Thank you for following up. We apologize that our previous response did not include the revisions to the manuscript. We have now uploaded a revised version of the manuscript with the changes outlined below.

1. Revisions to the writing and presentation

We have shortened redundant passages and reorganized the analysis to give the empirical findings greater prominence. In particular, the per-layer attention-distance analysis is now presented in the main text in Section 4.1 (Figure 4), followed by a more detailed interpretation in Section 4.2

2. Clarification regarding Variable-Width Transformers

We have added a dedicated paragraph on Variable-Width Transformers in the Related Work section (Section 5). It explicitly clarifies the architectural differences between the two approaches: Variable-Width Transformers modifies the overall block width, including attention and MLP dimensions, whereas Prism preserves the residual-stream width and dense projection dimensions, modifying only the partitioning of attention representations into heads. We also discuss how the two approaches could be complementary.

3. Additional uniform-head ablations

We have supplemented Table 5 in Appendix D with the requested uniform configurations of h=6h=6 and h=8h=8. These additional experiments use the same parameter budget and training compute as the other ablation configurations.

The updated results are summarized below. All validation losses are empirical means over three independent training seeds. The reported difference is calculated as (Delta = L_baseline - L_configuration), so a positive value indicates an improvement over the uniform h=12h=12 baseline.

ConfigurationValidation loss ↓Δ vs. baseline ↑
Reverse-order control3.3857−0.0109
Uniform (h=3h=3)3.3810−0.0062
Uniform (h=12h=12, baseline)3.3748—
Uniform (h=8h=8)3.3720+0.0028
Uniform (h=6h=6)3.3707+0.0041
Prism (progressive)3.3604+0.0144

The results show that uniform head configurations generally underperform compared with progressive schedules. Even when using fewer heads uniformly (h=6h=6 or h=8h=8), the resulting validation losses (3.3707 and 3.3720, respectively) remain higher than those achieved by the progressive schedules. The progressive allocation of heads appears more effective than maintaining a fixed head configuration throughout the network.

We sincerely thank the reviewer for their feedback and for giving us the opportunity to clarify and improve the manuscript. We hope these revisions address the concerns raised and clarify the contributions and empirical findings of the work. We would be grateful if the reviewer could reconsider their assessment in light of the revised manuscript.

Reviewer_rEhnOpenReview ↗

Response

I think this paper has made some improvements compared to the initial version, including the organizational structure and corresponding experimental settings. However, I still feel that the organizational structure of the paper may need to be improved. But considering the simplicity and effectiveness of the method, I have decided to improve my score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.