- Rating
- 1
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper proposes Task-Adaptive Inference Block Attention Residuals (TA-IBAR), a variation of Block AttnRes that chooses block boundaries separately for different tasks. It computes pairwise linear CKA between layer residual contributions on task-specific data, then uses dynamic programming to group similar consecutive layers. The partition is fixed while the backbone and task-specific routing parameters are jointly fine-tuned. At inference, a probe selects the corresponding task-adapted configuration. The paper reports validation-loss improvements across Math, Multi-hop, and Code tasks on models from 1.7B to 14B.
Critical strengths
1. Simple structural adaptation idea.
I like the idea of reconsidering fixed Block AttnRes partitions. Using CKA and dynamic programming to select nonuniform boundaries is relatively simple, and could offer a better quality–memory tradeoff.
2. Breadth of post-training experiments.
I appreciate the comparisons across multiple model scales, backbone families, and task domains. The reported validation-loss improvements are promising if reproducible.
Critical weaknesses
1. The experimental results contain substantial reporting inconsistencies and missing evidence.
Figures 2 and 5 are labelled as steps 1–1000, but the reported 350K/500K-token budgets and target of 4,096 global non-padding tokens per optimizer step imply approximately 85/122 optimizer updates. The correspondence between the plotted steps and the stated protocol is not explained. Figure 2 also shows Block AttnRes peak memory around 10.6 GB versus 12.52 GB in Table 2, and its displayed training losses differ substantially from the values reported in that table. Tables 6 and 7 report validation losses of 0.4292 and 0.8836 for what appears to be the default Qwen3-1.7B Math configuration, while Table 11 gives an unexplained 1.76 GB peak memory for Falcon3 Multi-hop Block AttnRes. More importantly, §4.3 claims stable training from scratch at approximately 100M and 220M, but the manuscript provides no numerical comparative results or training curves for these experiments. These reporting gaps make the results difficult to assess.
2. The contribution of task-specific boundary discovery is not adequately isolated, and its practical inference benefits remain unclear.
Tables 2–3 show consistent validation-loss improvements over fixed Block AttnRes under the stated matched post-training conditions. However, the fixed Block baseline's partitions and block counts are not specified, making it unclear whether the improvements arise from the discovered boundary locations or differences in structural granularity. Although Tables 5–6 compare two CKA-derived partitions with the same number of blocks, there are no matched-count uniform, random, or task-swapped partition controls to establish the benefit of the discovery algorithm over simpler alternatives. Furthermore, the method fine-tunes task-specific backbone weights alongside routing parameters, while the complete inference procedure is not evaluated. Without these controls and an assessment of the storage, selection, and inference costs, I do not think the current evidence convincingly establishes the distinct contribution or practical usefulness of task-specific boundary discovery.
Questions for authors
1. Structural discovery and mechanism
(a) Tables 2–3 demonstrate improvements over fixed Block AttnRes, but what are the baseline's exact partitions and block counts? Could you compare against uniform, random, and task-swapped partitions with the same number of blocks and otherwise identical adaptation? Tables 5–6 already compare two CKA-derived 10-block partitions, but this does not establish whether CKA-based boundary placement outperforms simpler alternatives. How much of the improvement is actually attributable to the discovered boundaries rather than structural granularity?
(b) How were the task-specific CKA thresholds selected? Table 6 shows that thresholds 0.45 and 0.50 produce different partitions with the same 10 blocks, while 0.60 achieves the lowest validation loss in the sweep. Why was 0.45 selected for the default Qwen3-1.7B Math configuration?
(c) Why use linear CKA to decide which residuals can be summed? High CKA does not necessarily imply that additive compression preserves information. For example, CKA(X,-X)=1, but X+(-X)=0. The fidelity analysis provides some empirical evidence, but could you compare against a criterion based on actual merging error or downstream loss and demonstrate that CKA is a useful predictor of which boundaries should be retained?
(d) Figure 10 shows negative validation-loss changes for most Multi-hop boundary-removal interventions. How exactly is delta validation loss defined, and how were these interventions performed? If negative values indicate improvements after removing boundaries, why are those boundaries retained? Could you compare the discovered partition against a simplified partition that removes these apparently unnecessary boundaries?
2. Experimental results and reproducibility
(a) What does a plotted “Math task step” mean in Figures 2 and 5, and how does it correspond to optimizer updates and actual token exposure? Could you reconcile the figures with the stated training budgets, the training-loss and memory differences between Figure 2 and Table 2, the apparent default-result discrepancy between Tables 6 and 7, and the Falcon3 memory value in Table 11?
(b) Where are the approximately 100M and 220M training-from-scratch results? Could you provide the training curves, validation losses, training budgets, and matched baselines? Table 8 gives model configurations, but not the outcomes needed to assess the claims in §4.3.
(c) Could you provide an anonymized repository containing the implementation and experimental configurations? Please also identify the starting checkpoint and training stages for each experimental regime. Section 3.2 describes Full-AttnRes training followed by discarding its routing parameters, while Appendix A.3 performs discovery using native residual aggregation. Was a Full-AttnRes training stage actually applied to the pretrained Qwen3 and cross-family models? If so, what data and budget were used, and when was native aggregation restored?
(d) How were the pseudo-queries and normalization parameters initialized? In Table 7, what is the frozen pseudo-query initialized to, and are the key-normalization gains also frozen? With a nonzero frozen query, trainable normalization gains can still change the effective scoring vector; with a zero query, routing remains uniform. These cases give the ablation different interpretations. Please also specify the reference row’s partition and explain its validation-loss discrepancy with Table 6.
3. Task-specific adaptation and inference
(a) My understanding is that each supported task receives a separate discovery procedure and subsequent fine-tuning. What granularity of “task” is intended, and is the deployment setting a fixed collection of supported tasks (sports, medicine, biology, physics) or a general-purpose model handling many overlapping domains? How are unseen or mixed-domain prompts handled?
(b) Equation (7) updates the task-specific backbone theta_t alongside the routing parameters. Does inference therefore require selecting different backbone weights for different tasks, rather than just switching partitions and routing parameters? What are the resulting storage and model-switching costs? Storing multiple copies of a single model would incur significant costs.
(c) Could you evaluate task-specific partitions and routing parameters with a single shared backbone? The existing comparisons already use matched task-specific adaptation protocols, but it remains unclear whether the benefits can be retained without separately fine-tuning and maintaining backbone weights for each task. This would make the proposed task-adaptive inference mechanism considerably more compelling.
(d) Do the reported results use supplied task labels or the probe’s predicted labels? The manuscript describes a probe in §3.5 but does not report its architecture, training procedure, or accuracy. Could you evaluate the complete inference pipeline, including incorrect configuration selection and recomputation from layer 0? If the task identity is supplied externally in the evaluated setting, please distinguish those results from results using automatic task selection.
4. Efficiency and implementation
(a) Could you report actual training throughput, discovery wall-clock time, and prefill/decode latency against an optimized fixed Block AttnRes implementation under matched conditions? The reported peak training-memory savings over fixed Block at 14B are 0.2 GB for Math and 0.5 GB for Multi-hop and Code. These measurements do not establish the inference-time speed or total deployment cost.
5. Evaluation, presentation, and scope
(a) Do the validation-loss improvements translate to answer accuracy on Math and Multi-hop and execution-based correctness on Code? Could you report these task-level outcomes in addition to validation loss?
(b) Could you align the conclusions with the evidence actually reported? In particular, the training-from-scratch claims need their numerical results, while claims about practical task-adaptive inference need an evaluation of that procedure. The current task-specific post-training experiments do not by themselves establish those additional claims.
(c) The paper repeats several descriptions while leaving the complete procedure unclear. For example, the threshold-sensitivity discussion is repeated on pages 8–9, while the relationship between the Full-AttnRes training stage in §3.2, native-residual discovery in Appendix A.3, and the pretrained-model experiments is not explicit. Threshold selection, routing initialization, and probe training also need to be specified. Could you consolidate these descriptions into one end-to-end procedure covering the starting checkpoint, any Full-AttnRes training, structural discovery, task adaptation, configuration selection, and inference? Please distinguish operations performed once offline from those performed for each inference request, including any selection or loading of task-specific backbone weights.
(d) Some introduction citations do not appear to support the specific claims they accompany. Lines 52–53 cite Wang (2026), which studies token KV-cache restrictions, and Fan et al. (2026), whose CORE method partitions smartphone UI elements (completely irrelevant), for the description of existing Block AttnRes. Lines 48–49 cite Wu et al. (2024), which studies shared representations across input modalities for object tracking, for a claim about task-dependent use of representations across network depth. Could you identify the relevant supporting results or correct these citations, and check the remaining claim-to-reference mappings?
6 comments
Comment 2 of 11
W2 and Q1a–b · Baseline blocks, partition transfer and threshold
Fixed Block uses consecutive groups of four Transformer blocks at depths 28/36/36/40, giving 7/9/9/10 groups on 1.7B/4B/8B/14B. Group k spans layers 4k − 3 through 4k (one-based). Each target uses its own training and evaluation data. We change which task supplied the fixed partition and then adapt on the target. Using the other task’s partition raises validation loss by 0.0460 on Math and 0.3512 on Multi-hop. The 10- and 11-block partitions differ in both count and boundaries. Independently, 0.45 and 0.50 both produce 10 Math blocks and give 0.4162 and 0.4313 validation loss. Count-matched uniform and random partitions with identical routing are needed to isolate the benefit over simpler boundary rules.
New Table 10. Partition transfer on Qwen3-1.7B
| Discovery task | Target task | Train loss | Valid. loss | Peak GB |
|---|---|---|---|---|
| Math | Math | 0.4206 | 0.4162 | 10.51 |
| Multi-hop | Math | 0.4871 | 0.4622 | 10.59 |
| Math | Multi-hop | 0.5753 | 0.5957 | 10.68 |
| Multi-hop | Multi-hop | 0.4092 | 0.2445 | 10.51 |
Revised Table 2. Task-specific partitions (submitted Table 1)
| Backbone | Math τ | Math block sizes | Math N | Multi-hop τ | Multi-hop block sizes | Multi-hop N |
|---|---|---|---|---|---|---|
| Qwen3-1.7B | 0.45 | [3,4,4,3,1,4,6,1,1,1] | 10 | 0.40 | [1,2,2,5,4,1,5,4,2,1,1] | 11 |
| Qwen3-4B | 0.50 | [1,2,1,1,2,5,4,4,3,1,7,3,2] | 13 | 0.43 | [1,3,2,3,3,4,1,5,2,1,4,3,4] | 13 |
| Qwen3-8B | 0.55 | [3,2,2,5,4,1,1,1,2,1,2,9,1,2] | 14 | 0.65 | [1,2,1,1,2,5,4,1,1,1,4,10,1,2] | 14 |
| Qwen3-14B | 0.50 | [2,2,4,1,1,1,2,6,1,2,5,1,9,2,1] | 15 | 0.55 | [3,1,2,2,1,7,3,1,1,1,5,1,10,2] | 14 |
Comment 3 of 11
Q1b · Threshold sweep
For Qwen3-1.7B Math, we swept τ over 0.30, 0.40, 0.45, 0.50, and 0.60 using the same cached CKA matrix and matched post-training settings. At 0.45, the default uses 10 blocks at 0.4162 validation loss and 10.51 GB. At 0.60, loss reaches 0.4135 with 16 blocks and 10.53 GB. We favor the more compact partition. The two 10-block partitions at 0.45 and 0.50 give different losses. The sweep measures threshold sensitivity on one matrix. Discovery stability across subsets or seeds remains open.
Revised Tables 8–9. Qwen3-1.7B Math threshold sweep (submitted Tables 5–6)
| τ | Partition | N | Max block | Singletons | Train loss | Valid. loss | Peak GB |
|---|---|---|---|---|---|---|---|
| 0.30 | [3,11,1,11,1,1] | 6 | 11 | 3 | 0.4975 | 0.4437 | 10.50 |
| 0.40 | [3,4,4,3,1,2,9,1,1] | 9 | 9 | 3 | 0.4313 | 0.4328 | 10.50 |
| 0.45 | [3,4,4,3,1,4,6,1,1,1] | 10 | 6 | 4 | 0.4206 | 0.4162 | 10.51 |
| 0.50 | [3,4,5,2,1,4,6,1,1,1] | 10 | 6 | 4 | 0.4184 | 0.4313 | 10.53 |
| 0.60 | [1,2,4,4,1,1,1,1,1,1,3,4,1,1,1,1] | 16 | 4 | 11 | 0.4029 | 0.4135 | 10.53 |
Q1c–d · CKA and boundary interventions
For nondegenerate representations, linear CKA(X, −X) = 1 and the sum X + (−X) is zero. We use CKA as a discovery heuristic. The residual-fidelity analysis is an empirical check. We have yet to compare against a criterion based on merging error or downstream loss. The coherence-based objective is applied before adaptation. Merging the intervals on either side of a boundary can violate the coherence constraint. The loss after merging is a separate empirical question. The sign convention for the loss change in submitted Figure 10 (revised Figure 11) and the effect of jointly removing those boundaries still require verification.
Comment 4 of 11
Q2b and Q4a · Scratch results and throughput
All four methods use 2B FineWeb tokens in Stage 1 and 1.8M GSM8K + SVAMP tokens in Stage 2. The parameter counts are 103.40M and 218.46M (labeled 110M and 220M in the results table). Stage 2 is full-parameter Math adaptation. The table contains endpoint results and measured Stage-2 throughput. From-scratch training curves are not reported.
New Table 1. Stage-2 Math endpoints and throughput
| Method | 110M train | 110M valid. | GB | Tok/s | 220M train | 220M valid. | GB | Tok/s |
|---|---|---|---|---|---|---|---|---|
| Native | 1.7090 | 1.6406 | 4.59 | 9594.21 | 1.6769 | 1.6224 | 6.88 | 6796.64 |
| Full AttnRes | 1.5805 | 1.5072 | 4.83 | 5463.87 | 1.5044 | 1.4496 | 7.34 | 3378.42 |
| Block AttnRes | 1.6371 | 1.6928 | 4.81 | 5187.39 | 1.4552 | 1.4183 | 7.29 | 2925.05 |
| TA-IBAR | 1.5139 | 1.4967 | 4.63 | 6096.58 | 1.4361 | 1.3153 | 6.94 | 4415.83 |
Comment 5 of 11
Q2a and Q2c · Corrected optimization settings
The submitted Table 8 listed scratch sequence lengths of 2,048, revised to 512. Qwen3 maximum lengths at 4B/8B/14B change from 1,024/1,024/2,048 to 2,048/4,096/8,192. The 1.7B target tokens per update change from 4,096 to 1,024. The model-level settings are listed below. Large-model runs use AdamW (β = 0.9/0.95, ε = 1e−8), cosine decay, 3% warmup, gradient clipping at 1.0 and FSDP FULL_SHARD with activation checkpointing. With separate parameter groups, weight decay is 0.1 on the backbone and zero on pseudo-queries and key-normalization gains. The 14B and cross-family runs use no CPU parameter offloading.
Revised Table 12. Model and optimization settings (submitted Table 8)
| Regime | Model | Params. | Depth | Width | Max len. | Tokens/update | Backbone LR | Routing LR | Precision | Hardware |
|---|---|---|---|---|---|---|---|---|---|---|
| Scratch | Scratch-Small | 103.40M | 12 | 448 | 512 | 4096 | 3×10⁻⁴ | 3×10⁻⁴ | BF16 + FP32 master | 4×RTX 3090 |
| Scratch | Scratch-Large | 218.46M | 18 | 672 | 512 | 4096 | 3×10⁻⁴ | 3×10⁻⁴ | BF16 + FP32 master | 4×RTX 3090 |
| Post-training | Qwen3 | 1.7B | 28 | 2048 | 1024 | 1024 | 5×10⁻⁶ | 3×10⁻⁵ | BF16 / FP32 | 2×A100 |
| Post-training | Qwen3 | 4B | 36 | 2560 | 2048 | 4096 | 3×10⁻⁶ | 2×10⁻⁵ | BF16 / FP32 | 2×A100 |
| Post-training | Qwen3 | 8B | 36 | 4096 | 4096 | 4096 | 1×10⁻⁶ | 3×10⁻⁵ | BF16 / FP32 | 4×RTX 3090 |
| Post-training | Qwen3 | 14B | 40 | 5120 | 8192 | 8192 | 1×10⁻⁶ | 3×10⁻⁵ | BF16 / FP32 | 4×RTX PRO 6000 |
| Cross-family | Granite-3.3 | 2B | 40 | 2048 | 1024 | 8192 | 1×10⁻⁶ | 3×10⁻⁵ | BF16 / FP32 | 4×RTX 3090 |
| Cross-family | SmolLM3 | 3B | 36 | 2048 | 1024 | 8192 | 1×10⁻⁶ | 3×10⁻⁵ | BF16 / FP32 | 4×RTX 3090 |
| Cross-family | Phi-4-mini | 3.8B | 32 | 3072 | 1024 | 8192 | 1×10⁻⁶ | 3×10⁻⁵ | BF16 / FP32 | 4×RTX 3090 |
| Cross-family | Llama-3.2 | 3B | 28 | 3072 | 1024 | 8192 | 1×10⁻⁶ | 3×10⁻⁵ | BF16 / FP32 | 4×RTX 3090 |
| Cross-family | Falcon3 | 3B | 22 | 3072 | 1024 | 8192 | 1×10⁻⁶ | 3×10⁻⁵ | BF16 / FP32 | 4×RTX 3090 |
# Response to Reviewer 21is
Comment 1 of 11
W1 and Q2a · Corrected budgets and losses
The submitted configuration listed 350K tokens and a target of 4,096 tokens per update for Qwen3-1.7B Math. The submitted ablation protocol listed 500K tokens. The revised main Math run and Math ablations use 1M tokens. At the revised target of 1,024 non-padding tokens per update, that budget corresponds to about 977 target-sized updates. The run logs are needed to verify how the plotted range through step 1,000 maps to optimizer updates and actual token exposure. The trajectory shows training loss. The revised main Math and Multi-hop tables use the mean of the final 100 optimizer-step training losses with consistent per-token normalization. Validation loss is evaluated at the final step. The submitted default validation entries of 0.4292 (threshold sweep) and 0.8836 (query ablation) are now aligned to 0.4162. The default row is 0.4206 train loss, 0.4162 validation loss and 10.51 GB. The Falcon3 Multi-hop fixed Block memory entry is corrected from 1.76 to 10.76 GB. The Math validation ordering at 1.7B is unchanged.
Revised Table 3. Math (submitted Table 2) (train loss / valid. loss / peak GB)
Revised Table 4. Multi-hop (submitted Table 3) (train loss / valid. loss / peak GB)