- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
Prism Transformer progressively increases attention head counts across layers while keeping model width fixed. It achieves modest gains across three model scales without increasing parameters or major computational costs. Its head schedules are manually designed.
Critical strengths
-
The design is clear and easy to understand and follow.
-
Validation loss improves across three model scales, with results reported over three random seeds.
-
Reversed schedules and uniformly wide heads provide useful controls supporting the importance of head allocation across depth.
Critical weaknesses
-
The paper’s contribution appears too limited to support its acceptance. It lacks any technical design, merely proposing the use of varying numbers of attention heads across different layers, with the specific schedules manually chosen based on empirical ablations. Although gains are demonstrated across three model scales, the evidence remains insufficient to establish whether these scheduling rules generalize to broader architectures, context lengths, and training settings.
-
The quality of Figure 1 is too low. It takes up nearly half a page yet lacks sufficient information.
-
I notice that changing head dimensions also changes RoPE frequency grids, preventing clear attribution of gains to head allocation alone.
Questions for authors
I think the contributions of this work are insufficient to support its acceptance. Please refer to the Critical Weaknesses.
3 comments
Thank you for the response. My concerns remain unresolved. The experiments are also not rigorous or comprehensive enough. Validation loss is not a good evaluation metric. Readers cannot easily determine the model’s performance gains on the target tasks from validation loss alone. Overall, I believe the quality of this paper needs improvement. These issues are unlikely to be resolved during the rebuttal period. Therefore, I will keep my score.
Additional Analysis and Experiments
Thank you for your follow-up and for sharing your remaining concerns. We understand your assessment and have made further revisions to strengthen the empirical analysis and presentation of the manuscript.
In particular, we have made the following changes:
1. Additional representation analysis
We have added three complementary representation analyses in Section 4.3 (Figure 5): Few-NERD entity-type linear probes, logit-lens analysis, and CKA representational alignment. These analyses examine how entity-type information becomes accessible across layers, how intermediate-layer predictions relate to final-layer predictions, and how representations align across model depths. We believe these provide additional insight into the representational effects of progressive head allocation, beyond validation loss and attention-distance measurements.
2. Evaluation beyond validation loss
We would also like to clarify that our evaluation includes downstream task benchmarks in addition to pre-training validation loss. Figure 2 and Appendix A (Table 3) reports zero-shot results across six benchmarks (HellaSwag, PIQA, WinoGrande, ARC-Easy, BLiMP, and WikiText) for all three model scales, averaged over three independent training seeds. The results show that Prism generally preserves or modestly improves downstream performance.
3. Additional experimental controls
We have included the fixed-band RoPE experiment in Appendix F.2 to investigate whether the observed gains depend on changes to the RoPE frequency grid. The additional Medium-scale experiment (seed 1120) yields the following results:
| Configuration | RoPE setting | Validation loss ↓ |
|---|---|---|
| Uniform baseline | Standard | 2.9954 |
| Prism | Dimension-specific RoPE | 2.9896 |
| Prism | Fixed-band RoPE | 2.9889 |
The results show that Prism with fixed-band RoPE performs marginally better than standard Prism. Although this experiment uses a single seed, it provides no evidence that the observed gains depend on the additional RoPE frequency bands.
We have also included additional Medium-scale GQA experiments in Appendix E, evaluating query-to-KV head ratios of 2:1 and 4:1. These experiments extend our evaluation to different attention configurations beyond the standard MHA setup.
4. Figure revision
We have redesigned Figure 1 as a more compact architectural overview, illustrating the progressive head allocation across layers while preserving the overall model and projection dimensions.
We acknowledge that broader evaluations across architectures, context lengths, and training settings remain important directions for future work. We have clarified these limitations in the revised manuscript.
We sincerely appreciate the reviewer's detailed feedback. Although we understand that these revisions may not fully resolve your concerns, we hope they improve the clarity, empirical support, and overall presentation of the work.
Response to Reviewer Concerns
Response to Weakness 1:
We thank the reviewer for raising this concern. We agree that Prism introduces a relatively simple modification to standard MHA. However, its contribution lies in identifying and investigating an under-explored architectural assumption: that every Transformer layer should use the same number of attention heads and per-head dimension.
Although the implementation is simple, determining whether uniform head allocation is optimal across network depth requires systematic investigation. Our motivation is that representations evolve throughout a network, from local compositional patterns in earlier layers towards broader contextual integration in deeper layers. This motivates investigating whether attention capacity should also be allocated differently across depth.
Importantly, our findings go beyond simply introducing non-uniform head counts. Our ablations over ten configurations demonstrate that the direction and structure of allocation matter. Reversing the exact Prism schedule (using the same head dimensions in the opposite order) degrades validation loss from 3.3604 to 3.3857, while uniformly wide heads also underperform the standard baseline (3.3810 vs. 3.3748). This suggests that improvements arise from progressive ordering rather than non-uniformity or wider heads alone.
We further demonstrate transferability of the identified design principles across three model scales, deriving the Medium and Large schedules without independently searching for optimal configurations. Additionally, experiments with GQA demonstrate that the approach is compatible with a different attention mechanism, improving validation loss within both MHA and GQA configurations.
We also conducted additional GQA experiments at the medium scale (24 layers, hidden dimension 1024, 16 query heads, 1 seed), and at different settings, evaluating query-to-KV head ratios of 2 and 4.
The results show that the progressive head schedule consistently improves validation loss over the corresponding GQA baseline at both grouping ratios.
We acknowledge that evaluations across longer contexts, encoder architectures, and larger scales remain future directions and will clarify these limitations. Nevertheless, we believe the controlled ablations and cross-scale and cross-attention results establish depth-wise attention-capacity allocation as a meaningful architectural design dimension.
Response to Weakness 2:
We agree that Figure 1 underutilizes its space. We will redesign it as a clearer architectural overview illustrating head-count and per-head-dimension transitions across depth, explicitly showing that total model and projection widths remain fixed while individual subspaces progressively narrow.
Response to Weakness 3:
We thank the reviewer for noting this potential confound, as varying per-layer head dimensions in standard RoPE alters the frequency grid.
To isolate positional encoding effects, we conducted an additional controlled experiment at the Medium scale (354M parameters, 24 layers, d_model = 1024, seed 1120), using a fixed-band RoPE configuration.
Instead of constructing a separate RoPE frequency grid for each head dimension, we use a single shared 64-dimensional rotary module across all layers. Wider heads are partitioned into consecutive 64-dimensional chunks, with each chunk receiving the same baseline frequency spectrum. This keeps the frequency grid fixed while retaining progressive attention-capacity allocation.
The fixed-band RoPE performs marginally better than the standard Prism. This suggests that the observed improvement does not rely on introducing additional RoPE frequency bands in wider early heads.
We retain dimension-specific RoPE in our main experiments because it provides greater flexibility in selecting intermediate head dimensions and constructing smooth schedules. A single shared frequency band must divide every head dimension, restricting the admissible schedules; for example, our Large configuration also uses a 96-dimensional head stage.
We will include this additional analysis and clarify the motivation for our RoPE implementation in the revised manuscript.
We sincerely thank the reviewer for their detailed feedback and recognition of our design clarity and results. We hope the additional GQA and fixed-band RoPE experiments, together with the proposed revisions, addresses the concerns raised. We hope this work encourages further investigation into alternative head-allocation strategies for improving Transformer representations and efficiency. If the reviewer finds these clarifications satisfactory, we would be grateful if they would consider revisiting their overall assessment and adjusting their rating accordingly.