acceptodds
Under review as a conference paper at ICLR 2027

Read, Allocate, Route: Role-Coupled Tokenization for Pixel-Space Generative Transformers

Abstract

Pixel-space generative Transformers typically use one token per image patch, so capturing finer detail requires more tokens and makes global attention increasingly expensive. We observe that finer detail does not necessarily require more global communication: low-frequency structure benefits from global exchange, while high-frequency detail can be modeled locally. Based on this insight, we introduce SpecSlot, a pixel-space rectified-flow Transformer that decouples output detail from the length of the global attention sequence. SpecSlot expands each spatial anchor into multiple slots, each assigned a disjoint band of DCT coefficients; this separates anchor density from spectral subdivision. Only the lowest-frequency slot from each anchor participates in global attention, while all slots interact locally. This allows the model to increase spectral detail without enlarging the global sequence. A spatial head further adds a local correction to the spectral prediction, while the generative state, target, and loss remain in pixel space. On ImageNet-256, SpecSlot achieves a comparable final FID to a FLOPs-matched JiT control, while its FID drops faster early in training. On ImageNet-512, it improves FID by 2.84 over the control with 3.9% fewer FLOPs, and comes within 0.69 FID of JiT-B/16 at 38.2% lower cost.

Primary area:
generative models
Keywords:
Diffusion model

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_vyefOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes SpecSlot, a pixel-space image generation architecture that uses frequency information to guide tokenization and attention. It divides each spatial patch into multiple frequency-based slots, allowing only the lowest-frequency slot to participate in global attention while the others interact locally. This reduces global attention costs while preserving the ability to generate fine details. The model also combines spectral and spatial outputs for image reconstruction.

The authors evaluate SpecSlot on ImageNet-256 and ImageNet-512 against JiT baselines, comparing generation quality, FLOPs, and throughput. They also conduct ablations on token allocation, attention patterns, frequency roles, and output representations. The results show comparable quality at ImageNet-256 and a better quality–compute trade-off at ImageNet-512, where SpecSlot improves FID by 2.84 over a compute-matched JiT baseline with slightly fewer FLOPs.

Critical strengths

1.The paper uses frequency information to guide token communication, allowing only low-frequency slots to participate in global attention. I think this is a simple and interesting way to improve efficiency, and the idea of separating output detail from global communication has potential beyond this specific architecture.

  1. The results are particularly encouraging at higher resolution, where SpecSlot achieves a 2.84 FID improvement over the compute-matched JiT baseline with slightly fewer FLOPs. The larger improvement at ImageNet-512 compared to ImageNet-256 is also a positive signal for its potential to scale to higher resolutions.
Critical weaknesses
  1. While the stronger results at ImageNet-512 are encouraging, the experiments are still limited to relatively small models (~140–160M parameters) and class-conditional image generation. It remains unclear whether the advantage will hold for larger models, higher resolutions, or more complex generation tasks. Since efficiency is a main motivation of the paper, I think broader scaling experiments would strengthen the contribution.
Questions for authors

Can you show that the quality–compute advantage of SpecSlot can hold for larger models (e.g., ~1B parameters)?

3 comments
AuthorsOpenReview ↗

Response to Reviewer vyef

We sincerely thank you for the encouraging review and for recognizing the potential of separating output detail from global communication. We address your concern about broader task coverage and larger models below.

W1: Extend evaluation beyond class-conditional generation.

To address your concern that the experiments are limited to class-conditional ImageNet, we added text-to-image experiments in Section 4.3, p. 8, Table 2. All three models use the same training data, text encoder, and adapter design; the protocol is given in Appendix B, p. 17, Table 5. SpecSlot retains its spectral output roles and carrier/local routing: carriers read the text context during global blocks, while local blocks update the image slots within each anchor. Table a reports the results.

Table a. Text-to-image generation at 256×256256\times256.

ModelForward GFLOPs ↓DPG-Bench (%) ↑GenEval (%) ↑
JiT-B/1650.7159.2224.89
JiT-B/8218.7761.1529.78
SpecSlot W256/K4129.8662.2126.99

SpecSlot achieves the highest DPG-Bench score among these three models, exceeding JiT-B/8 by 1.06 percentage points with 40.6% fewer forward FLOPs. On GenEval, its score lies between the two JiT baselines. These results extend the measured quality–compute trade-off to text-conditioned generation using the same allocation and routing design. Text-to-image examples are shown in Figure 5, p. 8, and Appendix Figure 11, pp. 27–28.

We also strengthened the ImageNet-512 comparison following the JiT-B/16 CFG sweep. With CFG 5.0 for JiT, SpecSlot-K4 retains a 0.88 FID advantage over JiT-CM at slightly lower forward cost (Section 4.2, p. 7, Table 1), updating the original 2.84 margin.

Q1: Does the quality–compute advantage hold at approximately 1B parameters?

Thank you for suggesting this larger-scale comparison. Completing the additional training and evaluation at approximately 1B parameters requires more time than the rebuttal period allows. We will include an approximately 1B-parameter ImageNet-256 comparison in the camera-ready version, comparing SpecSlot with JiT under a shared training and evaluation protocol. We will report FID-50K, forward FLOPs, and measured sampling throughput.

The same carrier rule can be retained when increasing backbone width or depth, keeping the global sequence length independent of the number of output slots. The larger-model comparison will test whether this design continues to provide a favorable quality–compute trade-off at that scale.

Thank you again for the encouraging feedback and for helping us broaden the evaluation. We would be glad to clarify any remaining points during the discussion.

Thank you for your response and clarification. I will maintain my current score.

Dear Reviewer vyef,

We appreciate your encouraging assessment and the opportunity to clarify our work. Your suggestions on broader task coverage and larger models have helped guide our revisions and future experiments. We are grateful for your thoughtful feedback throughout the discussion.

Best regards, The Authors

Reviewer_jeKfOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper points out that in pixel-space patch-grid Transformers such as JiT, increasing output detail inevitably increases global attention cost, as both are tied to the same patch grid. The authors propose SpecSlot, which divides the image into W spatial anchors and expands each anchor into K slots. Each slot predicts a disjoint group of DCT coefficients ordered by radial frequency, and a pixel-space correction head adds a local refinement to this spectral output. Only the slot responsible for the lowest frequencies participates in global attention, while all slots perform local attention within their anchor, so the global sequence length is independent of K. On ImageNet-256, SpecSlot performs comparably to a FLOPs-matched JiT control; on ImageNet-512, it lowers FID by 2.84 relative to this control and approaches JiT-B/16 with 38% less compute.

Critical strengths
  1. Important and timely problem. Global attention cost is a central obstacle to scaling pixel-space generation, and the paper offers a clear framing: decoupling output detail from global communication via anchor density (W) and spectral subdivision (K).

  2. The carrier routing gives a clear efficiency benefit. At matched FLOPs, SpecSlot (GGLL) matches or slightly improves on dense global attention over all slots (6.59 vs. 6.81 FID) while running about 1.5× faster.

Critical weaknesses
  1. The contribution of frequency-based slot assignment is not isolated. The paper attributes its gains to assigning slots disjoint radial DCT groups and routing the lowest-frequency slot globally. However, all slots are expanded from the same anchor vector, so the carrier already holds the full cell's information regardless of its assigned coefficients. Table 3 shows slot 0 is the best carrier given the radial assignment, but no experiment varies the assignment itself. I suggest an ablation that keeps everything else fixed and replaces the radial assignment with a fixed random partition of DCT coefficients across slots.

  2. The ImageNet-512 comparison may be confounded by CFG tuning. SpecSlot’s optimal CFG roughly doubles from 256 to 512 (4.0 → 8.25 for K4; 5.0 → 10.5 for K2), and SF-B/32 uses 9.5, yet all JiT rows remain at 2.9 at both resolutions, and a CFG sweep is shown only for JiT-B/8 at 256. Both headline claims at 512 (the 2.84 FID gain over JiT-CM and the 0.69 gap to JiT-B/16) depend on this. Please report CFG sweeps for the JiT baselines at 512.

  3. The slot representation underperforms plain patches at matched compute. Full-global, where all 1,024 slots attend globally (226 GFLOPs), reaches 5.84 FID, while JiT-B/8 with the same token count and similar compute (219 GFLOPs) reaches 4.34. The quality–compute gains thus appear to come primarily from the compute saved by restricting global attention rather than from a better tokenization. This suggests that decoupling detail from communication comes at a cost in quality, which is offset by the compute savings, rather than being free as the conclusion implies.

Minor:

  1. Section 4 presents the design ablations (Table 1) before the main comparison with baselines (Table 2). Since Table 1 is dense and relies on concepts and configurations introduced only briefly, moving the main results first would make the paper easier to follow.
Questions for authors
  1. If the radial DCT assignment is replaced by a fixed random partition across slots, with everything else unchanged, how does FID change?

  2. Were the JiT baselines at 512 actually swept over CFG? If so, please report the values; if their optimum lies well above 2.9, how do the 512 comparisons change?

7 comments
AuthorsOpenReview ↗

Response to Reviewer jeKf (Part1/2)

We sincerely thank you for the careful review and for recognizing the importance of decoupling output detail from global communication and the efficiency benefit of carrier routing. We address each concern below.

W1 / Q1: Isolate the contribution of frequency-based slot assignment.

Thank you for suggesting this control. The slots share the anchor input, while each slot is assigned a disjoint DCT group to predict (Section 3.2, p. 4). We added the requested fixed random partition into four equal groups, shared across anchors and channels, while keeping the architecture, slot count, carrier slot, and tile mapping fixed (Appendix H.2, pp. 22–23, Table 18). Table a also includes the band-swap controls.

Table a. Spectral assignment and carrier routing on ImageNet-256 with fixed training seed 12345.

DCT assignmentGlobal slot / tileCarrier bandFID-50K ↓
Radial baseline0 / 0Lowest6.4266
Fixed random partition0 / 0Mixed6.8816
Radial, G0/G3G_0/G_3 swapped0 / 0Highest6.6944
Radial, G0/G3G_0/G_3 swapped3 / 3Lowest6.4784

The fixed random partition raises FID from 6.4266 to 6.8816. Radial grouping therefore improves FID by 0.4550 at the same capacity and compute, showing that output assignment affects generation quality even with shared anchor inputs. After the band swap, routing through the new owner of the lowest-frequency group reduces FID from 6.6944 to 6.4784, just 0.0518 FID above the radial baseline. Together, these controls support using the lowest-frequency output role to select the global carrier.

W2 / Q2: Report ImageNet-512 CFG sweeps and update the comparisons.

Thanks for raising this point. We have added coarse and fine CFG sweeps for JiT-B/16 on ImageNet-512 in Appendix D, p. 19, Table 8. CFG 5.0 gives the lowest FID on both grids. Table b reports both sweeps.

Table b. Coarse and fine CFG sweeps for JiT-B/16 on ImageNet-512.

(a) Coarse grid.

CFGFID-50K ↓
1.0037.0192
2.0014.9537
2.908.4469
4.008.0505
5.007.6719
6.007.8720
7.008.2474
8.009.6698

(b) Fine grid.

CFGFID-50K ↓
4.257.8649
4.507.7469
4.757.6882
5.007.6719
5.257.6989
5.507.7411
5.757.8011

We use CFG 5.0 for all three JiT baselines in the revised Section 4.2, p. 7, Table 1. With this setting, SpecSlot-K4 leads JiT-CM by 0.88 FID (previously 2.84), and its gap to JiT-B/16 is 1.47 FID (previously 0.69) at 38.2% fewer forward FLOPs. The FID advantage over JiT-CM therefore remains under the updated CFG setting. Together with the matched-budget routing controls below, these results support SpecSlot's role-based allocation as an effective way to trade generation quality for compute.

AuthorsOpenReview ↗

Response to Reviewer jeKf (Part2/2)

W3: Distinguish slot representation quality from routing efficiency.

Thank you for highlighting this distinction. Full-global slots reach FID 5.84, while JiT-B/8 reaches 4.34 at similar compute (Table 3(c), p. 9; Table 1, p. 7). Decoupling refers to varying the number of output slots independently of the global sequence length. The revised discussion in Section 5.3, p. 9, and conclusion in Section 7, p. 10, present SpecSlot as an intermediate quality–compute operating point.

At the matched forward budget, GGLL achieves a lower mean FID than Dense-global-CM (6.59 vs. 6.81) and 1.56× the measured throughput (Section 5.3, p. 9, Table 3(c)). Compared with Full-global, GGLL saves 42.8% of forward FLOPs with a 0.75 higher FID. These controls support role-based routing as a way to improve the quality–compute trade-off at the intermediate compute budget.

Minor: Present the main results before the design ablations.

Thank you for this suggestion. We have moved the main ImageNet and text-to-image results to Section 4, pp. 6–8, Tables 1–2, followed by a separate Section 5: Ablation Study, pp. 8–9, Table 3. Readers now see the baseline comparisons before the design controls.

Additionally: Text-to-image validation.

To further address your concerns about the quality–compute trade-off on ImageNet, we added text-to-image experiments at 256×256256\times256 (Section 4.3, p. 8, Table 2). SpecSlot outperforms JiT-B/8 on DPG-Bench at lower forward compute. The protocol is given in Appendix B, p. 17, Table 5. Table c reports the results.

Table c. Text-to-image generation at 256×256256\times256.

ModelForward GFLOPs ↓DPG-Bench (%) ↑GenEval (%) ↑
JiT-B/1650.7159.2224.89
JiT-B/8218.7761.1529.78
SpecSlot W256/K4129.8662.2126.99

SpecSlot records the best DPG-Bench score in this comparison, gaining 1.06 percentage points over JiT-B/8 while using 40.6% fewer forward FLOPs. On GenEval, it scores above JiT-B/16 and below JiT-B/8. These results extend the evidence for SpecSlot's quality–compute trade-off from class-conditional to text-to-image generation.

Thank you for the helpful feedback. We would be grateful if you could reconsider your assessment based on these new results and clarifications. We are happy to discuss any remaining questions.

I thank the authors for the additional experiments and for correcting the ImageNet-512 numbers after the CFG sweep. The random-partition control (6.43 → 6.88) addresses my first concern, and I appreciate the revised framing of SpecSlot as an intermediate quality–compute point. A few issues remain:

  1. CFG at 512. The sweep covers only JiT-B/16, and JiT-CM reuses its CFG 5.0. Since JiT-CM is the main comparison, could the authors sweep CFG for JiT-CM as well?

  2. Slot representation vs. patches. At matched compute, Dense-global-CM does not improve over JiT-CM (6.81 vs. 6.68), and Full-global trails JiT-B/8 (5.84 vs. 4.34). The gains therefore seem to come from routing rather than from the tokenization itself. I suggest toning down the claim that detail and communication “can be decoupled through tokenization.”

I will finalize my rating after the response.

Reviewer_hmUaOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

Generative image transformer models require more tokens for finer details. The paper proposes to use locality information for high-frequency structure in the image i.e. details, and global attention for low-frequency structure. The paper uses discrete cosine transform to split the frequencies and introduces spatial anchors that contain multiple slots for the splitted frequencies. Only the slot for the lowest-frequency band participates in global attention. The rest of the slots communicate locally. Evaluations on ImageNet using FID, Inception Score, and computational efficiency metrics. show either comparable or better performance than a JiT baseline that is of the same computational cost, with increasing resolution.

Critical strengths

The idea of linking a token’s output role to its communication pattern is novel. While separating global structure from local details and using carrier tokens are existing ideas*, coupling attention routing to spectral output roles is an interesting architectural contribution.

  • E.g. HiLo Pan et al., FasterViT Hatamizadeh et al., DeCo Ma et al., PixelDiT - Yu et al.
Critical weaknesses

W.1

Section 4.4 (page 8, lines 395 - 397) explains that the improved performance at higher resolutions is because of the quadratic attention costs as the token grid grows. However, in Table 2 the authors show that the JiT baselines maintain the same number of tokens (1,024) by increasing the patch size. Therefore, the better relative performance at ImageNet-512 cannot be attributed to avoiding quadratic growth in attention cost.

W.2

At ImageNet 512, the method achieves FID of 9.14 where the compute matched JiT has an FID 11.98, which is impressive! But, this baseline configuration needs clarification before we can attribute the larger gain to the proposed attention routing.

The paper states that it uses noise with a standard deviation of 1 (Page 3) and does not mention any changes for ImageNet-512. However, the original JiT paper uses a noise standard deviation of 2 at this resolution. It is therefore unclear whether the JiT baselines follow the original settings or use the noise settings described in this paper.

W.3

Please check & cite these two particularly related works:

  • HiLo: Fast Vision Transformers with High-Low Frequency Attention (Pan et al., NeurIPS 2022).
  • FasterViT: Fast Vision Transformers with Hierarchical Attention (Hatamizadeh et al., ICLR 2024).
Questions for authors

See the weaknesses above.

In Section 3.2.2, you divide the same set of DCT coefficients among more slots when increasing the number of slots. However, increasing the number of slots does not introduce additional frequency components. It only subdivides the same set of coefficients among more slots. Could you please clarify whether the improvements result from increased representational capacity? Or is it the proposed spectral subdivision itself?

Besides these, could you clarify whether you considered the fact that changing the carrier slot also changes which spatial tile receives global attention? For instance, did you consider keeping the carrier slot and spatial-tile mapping fixed while changing only its assigned frequency band? This would help isolate whether the improvement comes from the spectral role rather than the spatial assignment.

7 comments
AuthorsOpenReview ↗

Response to Reviewer hmUa (Part1/2)

We sincerely thank the reviewer for the careful review and for recognizing the novelty of linking attention routing to spectral output roles. We address each concern below.

W1: The ImageNet-512 gain cannot be attributed to a growing JiT token grid.

Thank you for this comment. We have revised the explanation in Section 4.2, p. 7, Table 1. JiT-CM and the higher-compute JiT baseline double their patch size at ImageNet-512 and retain 1,024 image tokens. The revised analysis therefore does not attribute the larger margin to growth in JiT's token grid. SpecSlot keeps its global sequence fixed at 288 tokens, while each anchor covers four times as many pixels. Larger spectral windows and the spatial correction may help explain the larger margin.

Our ImageNet-512 sweep identifies CFG 5.0 as the best CFG for JiT-B/16 on both the coarse and fine grids (Appendix D, p. 19, Table 8). We use this value for all three JiT baselines in the revised Table 1. With this stronger baseline, SpecSlot-K4 improves FID over JiT-CM by 0.88, updating the original 2.84 claim, and trails JiT-B/16 by 1.47 FID at 38.2% fewer forward FLOPs. The FID advantage over JiT-CM therefore remains under the updated CFG setting.

W2: Clarify the training and sampling noise at ImageNet-512.

The submitted comparisons used noise standard deviation 1.0 in both training and sampling for JiT and SpecSlot. The comparison thus uses a matched noise protocol across both methods, which we now state explicitly.

To further address your concern, we additionally trained and sampled all four models with noise standard deviation 2.0, matching the noise setting of the original JiT ImageNet-512 experiments. We report these results in Appendix C, p. 18, Table 6, with CFG values retained from the revised main comparison.

Table a. ImageNet-512 comparisons with noise standard deviation 2.0 in training and sampling.

ModelCFGFID-50K ↓Forward GFLOPs ↓
JiT-B/325.08.545451.50
JiT-B/165.07.2278219.56
JiT-CM5.08.2033141.21
SpecSlot W256/K48.258.0583135.64

At noise scale 2.0, SpecSlot-K4 lowers FID by 0.1450 versus JiT-CM with 3.9% fewer forward FLOPs. Its FID gap to JiT-B/16 is 0.83 at 38.2% fewer forward FLOPs. The quality–compute advantage over JiT-CM is therefore retained at noise scale 2.0.

W3: Discuss HiLo and FasterViT.

Thanks for pointing us to these related works. We now cite and discuss both in Section 2, p. 3, and Appendix A.4, p. 16. HiLo [1] splits attention heads between local windows and global attention with pooled keys and values. FasterViT [2] initializes carrier tokens from pooled window features for cross-window exchange. SpecSlot ties communication to explicit output roles: slots predict disjoint DCT groups, and the lowest-frequency output slot serves as the global carrier.

References

[1] Pan, Zizheng, Jianfei Cai, and Bohan Zhuang. "Fast Vision Transformers with HiLo Attention." Advances in Neural Information Processing Systems 35 (2022): 14541–14554.

[2] Hatamizadeh, Ali, et al. "FasterViT: Fast Vision Transformers with Hierarchical Attention." International Conference on Learning Representations, 2024.

AuthorsOpenReview ↗

Response to Reviewer hmUa (Part2/2)

Q1: Does increasing KK add capacity or improve spectral subdivision?

As clarified in Section 3.2, pp. 3–4, at fixed anchor grid and output-window size, increasing KK divides the same coefficients among more slots; it adds no frequency components. More slots provide additional hidden states at higher compute (Section 5.1, p. 8; Table 3(a), p. 9). KK therefore controls both representation resources and the division of output roles. The utilization diagnostic in Appendix H.3, p. 23, Table 19 shows sublinear growth of effective slot rank.

To test spectral assignment separately, we keep K=4K=4, the architecture, and the attention budget fixed. Replacing the radial groups with a fixed random partition raises FID from 6.4266 to 6.8816 (Appendix H.2, p. 23, Table 18). Thus, radial grouping improves FID over the fixed random partition at the same capacity and compute.

Q2: Keep the carrier slot and tile mapping fixed while changing its frequency band.

Thank you for this insightful suggestion. We added the requested control with the final dual-head model in Section 6.2, p. 10, and Appendix H.2, pp. 22–23, Table 18: keep carrier slot 0 and every slot-to-tile mapping fixed, and swap the lowest and highest DCT groups (G0/G3G_0/G_3). Table b summarizes these comparisons.

Table b. Spectral assignment and carrier routing on ImageNet-256.

DCT assignmentGlobal slot / tileCarrier bandFID-50K ↓
Radial baseline0 / 0Lowest6.4266
Fixed random partition0 / 0Mixed6.8816
Radial, G0/G3G_0/G_3 swapped0 / 0Highest6.6944
Radial, G0/G3G_0/G_3 swapped3 / 3Lowest6.4784

With slot 0 and tile 0 fixed, the band swap raises FID from 6.4266 to 6.6944. This directly shows an effect of spectral assignment without changing the carrier's tile. As a further check, routing through the new owner of G0G_0 after the swap gives 6.4784, recovering most of the FID gap to the baseline. Together, these controls support using the lowest-frequency output role to select the global carrier.

Additionally: Text-to-image validation.

To provide further evidence for the method's soundness, we added text-to-image experiments at 256×256256\times256, testing the same allocation and routing design beyond class conditioning (Section 4.3, p. 8, Table 2). SpecSlot achieves a higher DPG-Bench score than JiT-B/8 with lower forward compute. The protocol is given in Appendix B, p. 17, Table 5. Table c reports the results.

Table c. Text-to-image generation at 256×256256\times256.

ModelForward GFLOPs ↓DPG-Bench (%) ↑GenEval (%) ↑
JiT-B/1650.7159.2224.89
JiT-B/8218.7761.1529.78
SpecSlot W256/K4129.8662.2126.99

SpecSlot achieves the highest DPG-Bench score among the three models, exceeding JiT-B/8 by 1.06 percentage points with 40.6% fewer forward FLOPs. Its GenEval score lies between JiT-B/16 and JiT-B/8. These additional results support the allocation and routing design beyond class conditioning and extend the measured quality–compute trade-off to text-to-image generation.

Thank you again for your constructive suggestions. We would appreciate your reconsideration of the paper's soundness in light of these clarifications and new experiments. If you have any further questions, we would welcome the opportunity to address them.

Thank you for the detailed responses. The added discussion of HiLo and FasterViT and the new wording on resolution scaling settle W1 and W3. Table 20 tests what I asked about in Q1 and Q2, and the results support the design.

I have two remaining concerns:

  1. At noise 2.0 (Table 6) the gap to JiT-CM shrinks from 0.88 to 0.145 FID. I would say the two models reach about the same FID at this setting. Noise 2.0 is the setting JiT uses at this resolution, and it gives every model in the table a better FID, including SpecSlot. So, I think the headline claim should be that SpecSlot reaches comparable FID to JiT-CM.

  2. The forward cost of SpecSlot-K4 at ImageNet-512 does not add up for me (Table 1, but also Table 5 & 6). I did not raise this in my original review. Table 5 gives B = 4096. With that value, the layers that grow with resolution add about 15 GFLOPs to the 256 model. That gives about 144 GFLOPs, not 135.64. SpecSlot-K4 would then cost slightly more than JiT-CM, not 3.9% less.

Comments on the paper

2 comments
Area_Chair_ozQBOpenReview ↗

Reviewers, please help to join the discussion.

Dear reviewers,

The authors have uploaded the rebuttal. Please help to join the discussion and the further process.

Thanks!

Your AC

AuthorsOpenReview ↗

General Response

We thank the reviewers for their thorough reviews and constructive feedback. We appreciate the observation that using the frequency band a slot predicts to decide whether it attends globally is a novel design choice (Reviewer hmUa). Another reviewer highlighted the ImageNet-256 comparison, where carrier routing generated more images per second than the compute-matched dense-slot control at comparable FID and forward cost (Reviewer jeKf). We were also encouraged by the interest in using frequency to guide communication and in applying this idea beyond SpecSlot (Reviewer vyef). The discussion helped us sharpen the ImageNet-512 comparison and broaden the evaluation. We have revised the manuscript according to the reviewers’ suggestions (marked in blue).

We here summarize and highlight our responses to the reviewers:

  • We clarified that increasing the slot count adds hidden states and divides the same DCT coefficients more finely (Reviewer hmUa). At a fixed count of four slots, replacing radial groups with a fixed random partition raised FID from 6.43 to 6.88 (Reviewer jeKf). Keeping the carrier and its spatial tile fixed, swapping the lowest and highest bands raised FID to 6.69. Routing through the new owner of the lowest band brought it back to 6.48 (Table 20, Reviewer hmUa). The dense-slot controls also show why we attribute the measured quality–compute trade-off to spectral allocation and carrier routing together. With full global attention, slots trail plain patches at similar compute (Tables 1 and 3, Reviewer jeKf).

  • We independently swept CFG for JiT-B/16 on ImageNet-512 and for JiT-CM at both resolutions. JiT-CM’s best tested CFG was 3.25 at 256 and 5.0 at 512 (Tables 8–10, Reviewer jeKf). We corrected the resolution-scaling explanation because JiT-CM keeps 1,024 image tokens at both resolutions. Profiling the ImageNet-512 checkpoint with B = 4096 corrected SpecSlot-K4’s forward cost from 135.64 to 143.71 GFLOPs, compared with 141.21 for JiT-CM. In the seed-0 comparison at JiT’s training and sampling noise scale of 2.0, their FIDs were 8.06 and 8.20, respectively. Our revised headline is that SpecSlot-K4 reaches comparable FID to JiT-CM at similar forward cost (Tables 1 and 6, Reviewer hmUa).

  • We tested the same allocation and routing design beyond class conditioning on 256 × 256 text-to-image generation. SpecSlot scored 1.06 percentage points higher than JiT-B/8 on DPG-Bench with 40.6% fewer forward FLOPs, and its GenEval score fell between the two JiT baselines (Table 2). To test whether the quality–compute trade-off holds at a larger model size, we will include an approximately 1B-parameter ImageNet-256 comparison with JiT in the camera-ready version (Reviewer vyef).

  • We also discuss HiLo and FasterViT in Section 2 and Appendix A.4 (Reviewer hmUa) and now present the main results before the design ablations (Reviewer jeKf).

We reply to each reviewer’s concerns in detail below their individual reviews. Thank you again for helping us improve the paper. We would be happy to clarify any remaining questions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.