acceptodds
Under review as a conference paper at ICLR 2027

TA-IBAR: Task-Adaptive Inference Block Attention Residuals

Abstract

Attention residuals (AttnRes) improve information integration across network depth, but Full AttnRes incurs increasing memory and communication costs as historical representations accumulate. Block AttnRes reduces these costs by compressing consecutive layers into block-level sources, yet uses a fixed depth partition across tasks. We propose Task-Adaptive Inference Block Attention Residuals (TA-IBAR), which derives a task-specific contiguous partition from residual representations using centred Linear CKA and structurally constrained dynamic programming. The partition remains fixed during post-training, while the backbone and lightweight task-specific routing parameters are jointly optimised. At inference time, a lightweight probe selects the corresponding task-adapted configuration. Experiments span training-from-scratch and post-training settings, model sizes from 220M to 14B, and network depths from 28 to 40. Across Qwen3 models from 1.7B to 14B, TA-IBAR achieves the lowest validation loss in all eight Math and Multi-hop settings, including reductions from 0.4810 to 0.4629 on 14B Math and from 9.7513 to 8.9869 on 14B Multi-hop over fixed Block AttnRes. Across five additional model families, TA-IBAR outperforms fixed Block AttnRes in all 10 model–task settings and Full AttnRes in 8 of 10 while retaining the memory efficiency of blockwise residual access.

Primary area:
foundation or frontier models, including LLMs
Keywords:
large language models, efficient inference, task adaptation, depth-wise attention, attention residuals

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.25 over 4 reviews, above 2% of the others.
Reviewer_21isOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes Task-Adaptive Inference Block Attention Residuals (TA-IBAR), a variation of Block AttnRes that chooses block boundaries separately for different tasks. It computes pairwise linear CKA between layer residual contributions on task-specific data, then uses dynamic programming to group similar consecutive layers. The partition is fixed while the backbone and task-specific routing parameters are jointly fine-tuned. At inference, a probe selects the corresponding task-adapted configuration. The paper reports validation-loss improvements across Math, Multi-hop, and Code tasks on models from 1.7B to 14B.

Critical strengths

1. Simple structural adaptation idea.

I like the idea of reconsidering fixed Block AttnRes partitions. Using CKA and dynamic programming to select nonuniform boundaries is relatively simple, and could offer a better quality–memory tradeoff.

2. Breadth of post-training experiments.

I appreciate the comparisons across multiple model scales, backbone families, and task domains. The reported validation-loss improvements are promising if reproducible.

Critical weaknesses

1. The experimental results contain substantial reporting inconsistencies and missing evidence.

Figures 2 and 5 are labelled as steps 1–1000, but the reported 350K/500K-token budgets and target of 4,096 global non-padding tokens per optimizer step imply approximately 85/122 optimizer updates. The correspondence between the plotted steps and the stated protocol is not explained. Figure 2 also shows Block AttnRes peak memory around 10.6 GB versus 12.52 GB in Table 2, and its displayed training losses differ substantially from the values reported in that table. Tables 6 and 7 report validation losses of 0.4292 and 0.8836 for what appears to be the default Qwen3-1.7B Math configuration, while Table 11 gives an unexplained 1.76 GB peak memory for Falcon3 Multi-hop Block AttnRes. More importantly, §4.3 claims stable training from scratch at approximately 100M and 220M, but the manuscript provides no numerical comparative results or training curves for these experiments. These reporting gaps make the results difficult to assess.

2. The contribution of task-specific boundary discovery is not adequately isolated, and its practical inference benefits remain unclear.

Tables 2–3 show consistent validation-loss improvements over fixed Block AttnRes under the stated matched post-training conditions. However, the fixed Block baseline's partitions and block counts are not specified, making it unclear whether the improvements arise from the discovered boundary locations or differences in structural granularity. Although Tables 5–6 compare two CKA-derived partitions with the same number of blocks, there are no matched-count uniform, random, or task-swapped partition controls to establish the benefit of the discovery algorithm over simpler alternatives. Furthermore, the method fine-tunes task-specific backbone weights alongside routing parameters, while the complete inference procedure is not evaluated. Without these controls and an assessment of the storage, selection, and inference costs, I do not think the current evidence convincingly establishes the distinct contribution or practical usefulness of task-specific boundary discovery.

Questions for authors

1. Structural discovery and mechanism

(a) Tables 2–3 demonstrate improvements over fixed Block AttnRes, but what are the baseline's exact partitions and block counts? Could you compare against uniform, random, and task-swapped partitions with the same number of blocks and otherwise identical adaptation? Tables 5–6 already compare two CKA-derived 10-block partitions, but this does not establish whether CKA-based boundary placement outperforms simpler alternatives. How much of the improvement is actually attributable to the discovered boundaries rather than structural granularity?

(b) How were the task-specific CKA thresholds selected? Table 6 shows that thresholds 0.45 and 0.50 produce different partitions with the same 10 blocks, while 0.60 achieves the lowest validation loss in the sweep. Why was 0.45 selected for the default Qwen3-1.7B Math configuration?

(c) Why use linear CKA to decide which residuals can be summed? High CKA does not necessarily imply that additive compression preserves information. For example, CKA(X,-X)=1, but X+(-X)=0. The fidelity analysis provides some empirical evidence, but could you compare against a criterion based on actual merging error or downstream loss and demonstrate that CKA is a useful predictor of which boundaries should be retained?

(d) Figure 10 shows negative validation-loss changes for most Multi-hop boundary-removal interventions. How exactly is delta validation loss defined, and how were these interventions performed? If negative values indicate improvements after removing boundaries, why are those boundaries retained? Could you compare the discovered partition against a simplified partition that removes these apparently unnecessary boundaries?

2. Experimental results and reproducibility

(a) What does a plotted “Math task step” mean in Figures 2 and 5, and how does it correspond to optimizer updates and actual token exposure? Could you reconcile the figures with the stated training budgets, the training-loss and memory differences between Figure 2 and Table 2, the apparent default-result discrepancy between Tables 6 and 7, and the Falcon3 memory value in Table 11?

(b) Where are the approximately 100M and 220M training-from-scratch results? Could you provide the training curves, validation losses, training budgets, and matched baselines? Table 8 gives model configurations, but not the outcomes needed to assess the claims in §4.3.

(c) Could you provide an anonymized repository containing the implementation and experimental configurations? Please also identify the starting checkpoint and training stages for each experimental regime. Section 3.2 describes Full-AttnRes training followed by discarding its routing parameters, while Appendix A.3 performs discovery using native residual aggregation. Was a Full-AttnRes training stage actually applied to the pretrained Qwen3 and cross-family models? If so, what data and budget were used, and when was native aggregation restored?

(d) How were the pseudo-queries and normalization parameters initialized? In Table 7, what is the frozen pseudo-query initialized to, and are the key-normalization gains also frozen? With a nonzero frozen query, trainable normalization gains can still change the effective scoring vector; with a zero query, routing remains uniform. These cases give the ablation different interpretations. Please also specify the reference row’s partition and explain its validation-loss discrepancy with Table 6.

3. Task-specific adaptation and inference

(a) My understanding is that each supported task receives a separate discovery procedure and subsequent fine-tuning. What granularity of “task” is intended, and is the deployment setting a fixed collection of supported tasks (sports, medicine, biology, physics) or a general-purpose model handling many overlapping domains? How are unseen or mixed-domain prompts handled?

(b) Equation (7) updates the task-specific backbone theta_t alongside the routing parameters. Does inference therefore require selecting different backbone weights for different tasks, rather than just switching partitions and routing parameters? What are the resulting storage and model-switching costs? Storing multiple copies of a single model would incur significant costs.

(c) Could you evaluate task-specific partitions and routing parameters with a single shared backbone? The existing comparisons already use matched task-specific adaptation protocols, but it remains unclear whether the benefits can be retained without separately fine-tuning and maintaining backbone weights for each task. This would make the proposed task-adaptive inference mechanism considerably more compelling.

(d) Do the reported results use supplied task labels or the probe’s predicted labels? The manuscript describes a probe in §3.5 but does not report its architecture, training procedure, or accuracy. Could you evaluate the complete inference pipeline, including incorrect configuration selection and recomputation from layer 0? If the task identity is supplied externally in the evaluated setting, please distinguish those results from results using automatic task selection.

4. Efficiency and implementation

(a) Could you report actual training throughput, discovery wall-clock time, and prefill/decode latency against an optimized fixed Block AttnRes implementation under matched conditions? The reported peak training-memory savings over fixed Block at 14B are 0.2 GB for Math and 0.5 GB for Multi-hop and Code. These measurements do not establish the inference-time speed or total deployment cost.

5. Evaluation, presentation, and scope

(a) Do the validation-loss improvements translate to answer accuracy on Math and Multi-hop and execution-based correctness on Code? Could you report these task-level outcomes in addition to validation loss?

(b) Could you align the conclusions with the evidence actually reported? In particular, the training-from-scratch claims need their numerical results, while claims about practical task-adaptive inference need an evaluation of that procedure. The current task-specific post-training experiments do not by themselves establish those additional claims.

(c) The paper repeats several descriptions while leaving the complete procedure unclear. For example, the threshold-sensitivity discussion is repeated on pages 8–9, while the relationship between the Full-AttnRes training stage in §3.2, native-residual discovery in Appendix A.3, and the pretrained-model experiments is not explicit. Threshold selection, routing initialization, and probe training also need to be specified. Could you consolidate these descriptions into one end-to-end procedure covering the starting checkpoint, any Full-AttnRes training, structural discovery, task adaptation, configuration selection, and inference? Please distinguish operations performed once offline from those performed for each inference request, including any selection or loading of task-specific backbone weights.

(d) Some introduction citations do not appear to support the specific claims they accompany. Lines 52–53 cite Wang (2026), which studies token KV-cache restrictions, and Fan et al. (2026), whose CORE method partitions smartphone UI elements (completely irrelevant), for the description of existing Block AttnRes. Lines 48–49 cite Wu et al. (2024), which studies shared representations across input modalities for object tracking, for a claim about task-dependent use of representations across network depth. Could you identify the relevant supporting results or correct these citations, and check the remaining claim-to-reference mappings?

6 comments
AuthorsOpenReview ↗

# Response to Reviewer 21is

Comment 1 of 11

W1 and Q2a · Corrected budgets and losses

The submitted configuration listed 350K tokens and a target of 4,096 tokens per update for Qwen3-1.7B Math. The submitted ablation protocol listed 500K tokens. The revised main Math run and Math ablations use 1M tokens. At the revised target of 1,024 non-padding tokens per update, that budget corresponds to about 977 target-sized updates. The run logs are needed to verify how the plotted range through step 1,000 maps to optimizer updates and actual token exposure. The trajectory shows training loss. The revised main Math and Multi-hop tables use the mean of the final 100 optimizer-step training losses with consistent per-token normalization. Validation loss is evaluated at the final step. The submitted default validation entries of 0.4292 (threshold sweep) and 0.8836 (query ablation) are now aligned to 0.4162. The default row is 0.4206 train loss, 0.4162 validation loss and 10.51 GB. The Falcon3 Multi-hop fixed Block memory entry is corrected from 1.76 to 10.76 GB. The Math validation ordering at 1.7B is unchanged.

Revised Table 3. Math (submitted Table 2) (train loss / valid. loss / peak GB)

ScaleNativeFullBlockTA-IBAR
Qwen3-1.7B0.4971 / 0.4326 / 10.080.4535 / 0.4762 / 12.520.4370 / 0.4182 / 12.520.4206 / 0.4162 / 10.51
Qwen3-4B0.64243 / 0.4347 / 18.880.6091 / 0.4412 / 20.720.7195 / 0.3915 / 20.720.6154 / 0.3697 / 20.83
Qwen3-8B1.1268 / 1.0781 / 22.250.9140 / 0.8743 / 20.010.9319 / 0.9051 / 19.210.8962 / 0.8710 / 19.05
Qwen3-14B0.9704 / 0.5982 / 36.20.7530 / 0.4723 / 37.10.7981 / 0.4810 / 36.60.7409 / 0.4629 / 36.4

Revised Table 4. Multi-hop (submitted Table 3) (train loss / valid. loss / peak GB)

ScaleNativeFullBlockTA-IBAR
Qwen3-1.7B0.7845 / 0.6926 / 10.080.3198 / 0.2853 / 12.400.4513 / 0.4271 / 12.400.4092 / 0.2445 / 10.51
Qwen3-4B3.6251 / 3.1532 / 21.670.2612 / 0.3643 / 24.940.2850 / 0.3245 / 24.940.2264 / 0.3022 / 20.83
Qwen3-8B13.4627 / 12.6283 / 19.509.9362 / 9.8410 / 18.9110.2958 / 10.9455 / 18.859.5727 / 9.3487 / 18.78
Qwen3-14B1.9140 / 10.2445 / 41.41.5365 / 9.7270 / 42.31.5397 / 9.7513 / 42.11.4976 / 8.9869 / 41.6

Comment 2 of 11

W2 and Q1a–b · Baseline blocks, partition transfer and threshold

Fixed Block uses consecutive groups of four Transformer blocks at depths 28/36/36/40, giving 7/9/9/10 groups on 1.7B/4B/8B/14B. Group k spans layers 4k − 3 through 4k (one-based). Each target uses its own training and evaluation data. We change which task supplied the fixed partition and then adapt on the target. Using the other task’s partition raises validation loss by 0.0460 on Math and 0.3512 on Multi-hop. The 10- and 11-block partitions differ in both count and boundaries. Independently, 0.45 and 0.50 both produce 10 Math blocks and give 0.4162 and 0.4313 validation loss. Count-matched uniform and random partitions with identical routing are needed to isolate the benefit over simpler boundary rules.

New Table 10. Partition transfer on Qwen3-1.7B

Discovery taskTarget taskTrain lossValid. lossPeak GB
MathMath0.42060.416210.51
Multi-hopMath0.48710.462210.59
MathMulti-hop0.57530.595710.68
Multi-hopMulti-hop0.40920.244510.51

Revised Table 2. Task-specific partitions (submitted Table 1)

BackboneMath τMath block sizesMath NMulti-hop τMulti-hop block sizesMulti-hop N
Qwen3-1.7B0.45[3,4,4,3,1,4,6,1,1,1]100.40[1,2,2,5,4,1,5,4,2,1,1]11
Qwen3-4B0.50[1,2,1,1,2,5,4,4,3,1,7,3,2]130.43[1,3,2,3,3,4,1,5,2,1,4,3,4]13
Qwen3-8B0.55[3,2,2,5,4,1,1,1,2,1,2,9,1,2]140.65[1,2,1,1,2,5,4,1,1,1,4,10,1,2]14
Qwen3-14B0.50[2,2,4,1,1,1,2,6,1,2,5,1,9,2,1]150.55[3,1,2,2,1,7,3,1,1,1,5,1,10,2]14

Comment 3 of 11

Q1b · Threshold sweep

For Qwen3-1.7B Math, we swept τ over 0.30, 0.40, 0.45, 0.50, and 0.60 using the same cached CKA matrix and matched post-training settings. At 0.45, the default uses 10 blocks at 0.4162 validation loss and 10.51 GB. At 0.60, loss reaches 0.4135 with 16 blocks and 10.53 GB. We favor the more compact partition. The two 10-block partitions at 0.45 and 0.50 give different losses. The sweep measures threshold sensitivity on one matrix. Discovery stability across subsets or seeds remains open.

Revised Tables 8–9. Qwen3-1.7B Math threshold sweep (submitted Tables 5–6)

τPartitionNMax blockSingletonsTrain lossValid. lossPeak GB
0.30[3,11,1,11,1,1]61130.49750.443710.50
0.40[3,4,4,3,1,2,9,1,1]9930.43130.432810.50
0.45[3,4,4,3,1,4,6,1,1,1]10640.42060.416210.51
0.50[3,4,5,2,1,4,6,1,1,1]10640.41840.431310.53
0.60[1,2,4,4,1,1,1,1,1,1,3,4,1,1,1,1]164110.40290.413510.53

Q1c–d · CKA and boundary interventions

For nondegenerate representations, linear CKA(X, −X) = 1 and the sum X + (−X) is zero. We use CKA as a discovery heuristic. The residual-fidelity analysis is an empirical check. We have yet to compare against a criterion based on merging error or downstream loss. The coherence-based objective is applied before adaptation. Merging the intervals on either side of a boundary can violate the coherence constraint. The loss after merging is a separate empirical question. The sign convention for the loss change in submitted Figure 10 (revised Figure 11) and the effect of jointly removing those boundaries still require verification.

Comment 4 of 11

Q2b and Q4a · Scratch results and throughput

All four methods use 2B FineWeb tokens in Stage 1 and 1.8M GSM8K + SVAMP tokens in Stage 2. The parameter counts are 103.40M and 218.46M (labeled 110M and 220M in the results table). Stage 2 is full-parameter Math adaptation. The table contains endpoint results and measured Stage-2 throughput. From-scratch training curves are not reported.

New Table 1. Stage-2 Math endpoints and throughput

Method110M train110M valid.GBTok/s220M train220M valid.GBTok/s
Native1.70901.64064.599594.211.67691.62246.886796.64
Full AttnRes1.58051.50724.835463.871.50441.44967.343378.42
Block AttnRes1.63711.69284.815187.391.45521.41837.292925.05
TA-IBAR1.51391.49674.636096.581.43611.31536.944415.83

Comment 5 of 11

Q2a and Q2c · Corrected optimization settings

The submitted Table 8 listed scratch sequence lengths of 2,048, revised to 512. Qwen3 maximum lengths at 4B/8B/14B change from 1,024/1,024/2,048 to 2,048/4,096/8,192. The 1.7B target tokens per update change from 4,096 to 1,024. The model-level settings are listed below. Large-model runs use AdamW (β = 0.9/0.95, ε = 1e−8), cosine decay, 3% warmup, gradient clipping at 1.0 and FSDP FULL_SHARD with activation checkpointing. With separate parameter groups, weight decay is 0.1 on the backbone and zero on pseudo-queries and key-normalization gains. The 14B and cross-family runs use no CPU parameter offloading.

Revised Table 12. Model and optimization settings (submitted Table 8)

RegimeModelParams.DepthWidthMax len.Tokens/updateBackbone LRRouting LRPrecisionHardware
ScratchScratch-Small103.40M1244851240963×10⁻⁴3×10⁻⁴BF16 + FP32 master4×RTX 3090
ScratchScratch-Large218.46M1867251240963×10⁻⁴3×10⁻⁴BF16 + FP32 master4×RTX 3090
Post-trainingQwen31.7B282048102410245×10⁻⁶3×10⁻⁵BF16 / FP322×A100
Post-trainingQwen34B362560204840963×10⁻⁶2×10⁻⁵BF16 / FP322×A100
Post-trainingQwen38B364096409640961×10⁻⁶3×10⁻⁵BF16 / FP324×RTX 3090
Post-trainingQwen314B405120819281921×10⁻⁶3×10⁻⁵BF16 / FP324×RTX PRO 6000
Cross-familyGranite-3.32B402048102481921×10⁻⁶3×10⁻⁵BF16 / FP324×RTX 3090
Cross-familySmolLM33B362048102481921×10⁻⁶3×10⁻⁵BF16 / FP324×RTX 3090
Cross-familyPhi-4-mini3.8B323072102481921×10⁻⁶3×10⁻⁵BF16 / FP324×RTX 3090
Cross-familyLlama-3.23B283072102481921×10⁻⁶3×10⁻⁵BF16 / FP324×RTX 3090
Cross-familyFalcon33B223072102481921×10⁻⁶3×10⁻⁵BF16 / FP324×RTX 3090
Reviewer_jPuUOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes TA-IBAR, a task-adaptive variant of Block Attention Residuals. It replaces Block AttnRes’s fixed depth partition with a task-specific contiguous partition, identified from residual representations using centred Linear CKA and structurally constrained dynamic programming. The partition is fixed during post-training, when the backbone and task-specific routing parameters are jointly optimized from a common checkpoint. At inference, a lightweight probe selects the task-specific configuration. The authors evaluate TA-IBAR on Qwen3 models from 1.7B to 14B across Math, Multi-hop, and Code, on five additional backbone families, and in training-from-scratch settings. TA-IBAR generally improves validation loss over fixed Block AttnRes and often over Full AttnRes, while keeping blockwise memory costs. The evaluation spans several settings, but does not sufficiently isolate the central causal claim or deployment implications.

Critical strengths

The paper addresses task-dependent depth redundancy and evaluates the method across model scales, task domains, backbone families, memory measurements, and ablations. This scope makes the empirical findings relevant to work on AttnRes and Block AttnRes.

The discovery pipeline operationalises task-adaptive compression by using centred Linear CKA to estimate depth similarity and dynamic programming to derive a contiguous partition. This makes the partitioning procedure interpretable.

Critical weaknesses

The central comparison is confounded because TA-IBAR differs from fixed Block AttnRes in both its partition and its task-specific routing and adaptation. The paper does not report an ablation that combines a fixed Block partition with task-specific routing and backbone adaptation. The observed gains may therefore reflect task-specific post-training or routing rather than adaptive depth partitioning.

The implementation details are underspecified. Section 3.5 states that backbone parameters θ_t and task-specific routing Q_t are jointly optimized, suggesting a separate adapted backbone for each task. If so, TA-IBAR would require multiple full-model copies and a task-identity probe at inference. This raises questions about its claimed memory efficiency and practicality as a single model. The paper reports peak training GPU memory but not inference memory, latency, or probe accuracy and misclassification behavior.

Questions for authors
  1. In Section 3.5, are the backbone parameters Q_t stored separately for each task, or does the method use a single shared backbone with only task-specific routing parameters? Please clarify the inference storage cost and whether TA-IBAR requires multiple full-model copies.

  2. Can you provide an ablation with a fixed Block AttnRes partition and the same task-specific routing or adaptation budget as TA-IBAR? Without this control, how can the gains be attributed to adaptive partitioning rather than task-specific routing or backbone adaptation?

  3. How is the discovery threshold τ selected for each Qwen3 model-task pair? Is it tuned on validation data? How stable are the discovered partitions across discovery subsets or random seeds?

  4. What is the accuracy of the inference-time task probe, and how does TA-IBAR perform when the probe misclassifies a task? Do the benchmark comparisons assume that task identity is known?

  5. Please report inference memory and latency, in addition to peak training memory, when multiple task-specific configurations are retained. How does TA-IBAR compare with fixed Block AttnRes in this deployment setting?

3 comments
Reviewer_jPuUOpenReview ↗

No Response, Score Unchanged

Since there are no responses to my questions, I will not change my score.

AuthorsOpenReview ↗

# Response to Reviewer jPuU

We apologize for the delayed response. We read your comments carefully and took additional time to revise the manuscript and run new experiments in response to your questions. Thank you for your detailed feedback. We address each point below.

Comment 1 of 3

W1 and Q2 · What the partition control isolates

An equal-length control is equivalent to our fixed Block baseline if boundaries, residual aggregation, routing parameterization and initialization, trainable parameters, starting checkpoint, data and budget match. The fixed baseline receives full-parameter adaptation. We have not run a count-matched uniform or random partition with the same retained task-specific routing bank. Each target uses its own training and evaluation data. We change which task supplied the fixed partition and then adapt on the target. Using the other task’s partition raises validation loss by 0.0460 on Math and 0.3512 on Multi-hop. The 10- and 11-block partitions differ in both count and boundaries.

New Table 10. Partition transfer on Qwen3-1.7B

Discovery taskTarget taskTrain lossValid. lossPeak GB
MathMath0.42060.416210.51
Multi-hopMath0.48710.462210.59
MathMulti-hop0.57530.595710.68
Multi-hopMulti-hop0.40920.244510.51

W2 and Q1 · Shared backbone

All methods start with Math and continue sequentially from their own current backbone weights. TA-IBAR switches the active partition, pseudo-queries and key-normalization gains. The model stores a single backbone plus the supported task banks. Peak memory with all task banks loaded and Math performance on the checkpoint after Multi-hop remain unmeasured.

Comment 2 of 3

Q3 · Thresholds and stability

For Qwen3-1.7B Math, we swept τ over 0.30, 0.40, 0.45, 0.50, and 0.60 using the same cached CKA matrix and matched post-training settings. At 0.45, the default uses 10 blocks at 0.4162 validation loss and 10.51 GB. At 0.60, loss reaches 0.4135 with 16 blocks and 10.53 GB. We favor the more compact partition. The two 10-block partitions at 0.45 and 0.50 give different losses. The sweep measures threshold sensitivity on one matrix. Discovery stability across subsets or seeds remains open.

Revised Tables 8–9. Qwen3-1.7B Math threshold sweep (submitted Tables 5–6)

τPartitionNMax blockSingletonsTrain lossValid. lossPeak GB
0.30[3,11,1,11,1,1]61130.49750.443710.50
0.40[3,4,4,3,1,2,9,1,1]9930.43130.432810.50
0.45[3,4,4,3,1,4,6,1,1,1]10640.42060.416210.51
0.50[3,4,5,2,1,4,6,1,1,1]10640.41840.431310.53
0.60[1,2,4,4,1,1,1,1,1,1,3,4,1,1,1,1]164110.40290.413510.53

Revised Table 2. Task-specific partitions (submitted Table 1)

BackboneMath τMath block sizesMath NMulti-hop τMulti-hop block sizesMulti-hop N
Qwen3-1.7B0.45[3,4,4,3,1,4,6,1,1,1]100.40[1,2,2,5,4,1,5,4,2,1,1]11
Qwen3-4B0.50[1,2,1,1,2,5,4,4,3,1,7,3,2]130.43[1,3,2,3,3,4,1,5,2,1,4,3,4]13
Qwen3-8B0.55[3,2,2,5,4,1,1,1,2,1,2,9,1,2]140.65[1,2,1,1,2,5,4,1,1,1,4,10,1,2]14
Qwen3-14B0.50[2,2,4,1,1,1,2,6,1,2,5,1,9,2,1]150.55[3,1,2,2,1,7,3,1,1,1,5,1,10,2]14

Comment 3 of 3

Q4 · Task labels and misclassification

Training uses supplied labels. The downstream inference evaluation selects configurations using the probe. The probe runs the first Transformer block with native residuals and classifies the mean-pooled valid-token hidden states. A recognized task activates its partition and routing bank. Otherwise, the model uses standard AttnRes. In either case, we discard the probe activations and recompute the prompt from layer 0. The probe correctly classified all examples in Table 14. The effect of probe misclassification on answer accuracy remains unmeasured. The task-swap experiment changes the partition before adaptation, which tests a different intervention.

New Table 14. Probe task identification

BackboneTaskEvaluation DataSamplesAccuracy ↑
Qwen3-1.7BMathGSM8K (250) + SVAMP (250)500100%
Qwen3-1.7BMulti-hopCLUTRR500100%
Qwen3-14BCodeMBPP (250) + xCoder-80K (250)500100%

New Table 6. Downstream answer accuracy

ModelTaskDatasetMetricNativeFull AttnResBlock AttnResTA-IBAR
Qwen3-1.7BMathGSM8K + SVAMPAcc. ↑44%56%55%63%
Qwen3-1.7BMulti-hopCLUTRRAcc. ↑1.5%2.0%1.8%2.2%

Q5 · Inference cost

Native uses the official checkpoint, Full uses the checkpoint at 566,588 training tokens, and TA-IBAR uses the final Math checkpoint. The revised paper gives peak memory and decode throughput. Prefill and probe latency, runtime memory bandwidth, and total memory with all routing banks resident were not measured separately.

New Table 7. Inference on Qwen3-1.7B

MethodPeak Mem. (GB) ↓Decode Tok/s ↑
Native3.46431.79
Full AttnRes3.67819.87
Block AttnRes3.61219.63
TA-IBAR3.54620.55
Reviewer_dnfzOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper investigates estimating task-adaptive block configurations for Block Attention Residuals (AttRes) Networks. Task-dependent configurations of maximally coherent blocks are estimated for a task by applying Linear KCA to layer residuals to estimate the coherence of layers that are grouped together in a candidate block (eq. 2,3), and then selecting the overall block configuration based on it's total coherence using dynamic programming (eq. 4). At test time, a lightweight probe selects what task-specific block configuration to employ.

Post-training results suggest that the approach generally performs on-par or better than standard Block AttRes (Table 2,3,4), and qualitative examples illustrate that task-specific block assignments differ appreciably (Table 1, Figure 3). An ablation study (Table 5, 6) suggests that validation loss and memory performance is not very sensitive to the hyperparameter τ\tau, which influences the number of blocks chosen.

Critical strengths
  • The approach is a reasonable and interesting task adaptation scheme for conditional computation.
  • Efficient task-adaptive block-residual attention is an interesting and likely important investigation area.
  • Results are promising.
Critical weaknesses
  • The approach is not baselined against any other task-adaptive block-residual attention approaches that leverage task-level supervision at training time. One could probe and then load (e.g. LoRA) task-adapted residual attention weights, for example.
  • The paper is in generally understandable, but "draft" state, as the following points expand upon.
  • AttRes and BlockAttRes are not properly introduced, and the manuscript is dense with information in some areas, and uninformative in others.
  • Wrt the latter, and example is that training from scratch is reported as "stable", with no other information provided, yet consumes half a page.
  • Figure 2 b has no TA-IBAR memory curve or associated explanation of what curve it matches (block I assume), and Figure 2 a depicts training loss, but is never discussed.
  • Amazingly, the paper provides no information about the task-selection probe which leads to task-adaptive grouping, let alone any results and empirical analysis. How much extra compute is required to compute the probe? How effective is it?
  • Related, are the provided post-training results comparisons computed on a per-task basis, or are all models finetuned and tested on all data, with TA-IBAR's probe estimating the task? The experimental configuration needs to be more clear.
  • Minimizing equation (4) lexicographically as claimed would mean that you always select the minimum feasible NtN_t. If that's what you did, then NtN_t is not determined by the task as later claimed. Please clarify and correct this.
  • Overall, while interesting, I feel that the paper requires significant revision before publication.
Questions for authors

See Limitations section.

2 comments
AuthorsOpenReview ↗

Response to Reviewer dnfz

Comment 1 of 3

W1 · Task-supervised baselines

In Eq. (7), we jointly optimize the backbone and task-specific routing parameters. Backbone updates change the residual representations that the pseudo-queries attend to, and query updates change the residual mixture fed back into the backbone. Training only backbone LoRA adapters would constrain backbone updates to low-rank changes in selected projections and leave the pseudo-queries frozen. Training the routing bank alongside LoRA would be a separate parameter-efficient setting. Table 11 shows the importance of direct query adaptation: freezing the pseudo-queries raises Math validation loss from 0.4162 to 1.4781. The reported comparison uses matched full-parameter adaptation. A matched task-conditioned LoRA comparison remains open.

Revised Table 11. Qwen3-1.7B Math query ablation (submitted Table 7)

VariantTrain lossValid. lossPeak GB
TA-IBAR0.42060.416210.51
w/o Query Adaptation4.32811.478110.63

W2–5 and W9 · Method and from-scratch results

Full AttnRes attends to individual earlier residual contributions. Fixed Block AttnRes aggregates contiguous groups into block-level sources. TA-IBAR discovers the groups and then learns depth-wise routing over the available sources. All four methods use 2B FineWeb tokens in Stage 1 and 1.8M GSM8K + SVAMP tokens in Stage 2. The parameter counts are 103.40M and 218.46M (labeled 110M and 220M in the results table). Stage 2 is full-parameter Math adaptation. The table contains endpoint results and measured Stage-2 throughput. From-scratch training curves are not reported.

New Table 1. Stage-2 Math endpoints and throughput

Method110M train110M valid.GBTok/s220M train220M valid.GBTok/s
Native1.70901.64064.599594.211.67691.62246.886796.64
Full AttnRes1.58051.50724.835463.871.50441.44967.343378.42
Block AttnRes1.63711.69284.815187.391.45521.41837.292925.05
TA-IBAR1.51391.49674.636096.581.43611.31536.944415.83

Comment 2 of 3

W5 · Training loss and peak training memory on pretrained 1.7B

The training-loss trajectory and the final validation measurement answer different questions. The mean of the last 100 training updates and the final validation loss are given below. The peak training-memory results include TA-IBAR at 10.51 GB.

Revised Table 3, Qwen3-1.7B Math

MethodTrain lossValid. lossPeak GB
Native0.49710.432610.08
Full AttnRes0.45350.476212.52
Block AttnRes0.43700.418212.52
TA-IBAR0.42060.416210.51

W6–7 · Probe and evaluation protocol

The probe runs the first Transformer block with native residuals and classifies the mean-pooled valid-token hidden states. A recognized task activates its partition and routing bank. Otherwise, the model uses standard AttnRes. In either case, we discard the probe activations and recompute the prompt from layer 0. This adds one early-block pass and a classification per prompt. Probe latency and accuracy on mixed-domain inputs remain open. All methods receive the supplied training task, train on Math first and continue to the next task from their own Math-trained weights. Task-specific losses use the supplied task. In downstream answer evaluation, TA-IBAR uses the probe’s selected configuration. The probe results are per model and task. The answer accuracy and task-identification accuracy measure distinct outcomes.

New Table 14. Probe task identification

BackboneTaskEvaluation DataSamplesAccuracy ↑
Qwen3-1.7BMathGSM8K (250) + SVAMP (250)500100%
Qwen3-1.7BMulti-hopCLUTRR500100%
Qwen3-14BCodeMBPP (250) + xCoder-80K (250)500100%

New Table 6. Downstream answer accuracy

ModelTaskDatasetMetricNativeFull AttnResBlock AttnResTA-IBAR
Qwen3-1.7BMathGSM8K + SVAMPAcc. ↑44%56%55%63%
Qwen3-1.7BMulti-hopCLUTRRAcc. ↑1.5%2.0%1.8%2.2%

Comment 3 of 3

W8 · Lexicographic minimization and task dependence

A multi-layer interval is admissible when its minimum pairwise CKA reaches the task threshold. Single-layer intervals are always admissible. Eq. (4) first minimizes the number of blocks over feasible contiguous partitions, then maximizes total coherence among partitions with that minimum count. The CKA matrix and threshold set the feasible intervals for each task. For example, 4B gives 13 blocks for both tasks, with the second block ending at layer 3 on Math and layer 4 on Multi-hop.

Revised Table 2. Task-specific partitions (submitted Table 1)

BackboneMath τMath block sizesMath NMulti-hop τMulti-hop block sizesMulti-hop N
Qwen3-1.7B0.45[3,4,4,3,1,4,6,1,1,1]100.40[1,2,2,5,4,1,5,4,2,1,1]11
Qwen3-4B0.50[1,2,1,1,2,5,4,4,3,1,7,3,2]130.43[1,3,2,3,3,4,1,5,2,1,4,3,4]13
Qwen3-8B0.55[3,2,2,5,4,1,1,1,2,1,2,9,1,2]140.65[1,2,1,1,2,5,4,1,1,1,4,10,1,2]14
Qwen3-14B0.50[2,2,4,1,1,1,2,6,1,2,5,1,9,2,1]150.55[3,1,2,2,1,7,3,1,1,1,5,1,10,2]14
Reviewer_P8AjOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

Attention Residuals is a technique that, instead of adding each layer's output directly to the residual stream, converts it into a depth-wise attention form and adds that. It is powerful, but the cost can become large when the outputs of all layers are used. Block AttnRes reduces this cost while preserving the gain by grouping consecutive layers into blocks and using their summed values, but the existing Block AttnRes partitioned the blocks uniformly.

TA-IBAR proposes a method that, for each task, uses discovery examples to compute the CKA of the values that each layer adds to the residual steam, and uses dynamic programming to partition the blocks so that the minimum CKA within a block is at least a threshold and the number of blocks is minimized. Through this, it claims to obtain a lower validation loss for each task while incurring a cost similar to the existing Block Attnres.

Critical strengths
  1. The question of whether blocks must necessarily be partitioned uniformly is valid, and the approach of partitioning differently depending on the task is interesting.
  2. The partitioning method is simple and easy to reproduce, and it appears to be extensible to other tasks as well.
Critical weaknesses
  1. To claim that the partition should differ "per task," the partitioning process must show that the partition is robust to other discovery examples of the same task, but this paper has no such process. This makes it impossible to distinguish whether the partition is due to the characteristics of a specific task or the characteristics of those discovery examples.
  2. The analysis of infrastructure cost, which is the reason Block AttnRes was chosen instead of Full AttnRes, is insufficient, and the Peak GPU Memory presented in the paper does not adequately reflect other costs such as pipeline communication volume and inference memory bandwidth. A theoretical analysis is needed of which values the infrastructure cost depends on and whether it varies with the partitioning scheme.
  3. The comparison is not clear. The number of blocks in the Block AttnRes used in the experiments is not properly specified. Looking at Figure 6, it appears to be partitioned into 7 for Qwen3-1.7B Math, but according to Table 1, the number of blocks in TA-IBAR, which it is compared against, is 10. Since the infrastructure cost varies with the number of blocks and the model gets closer to Full AttnRes as the number grows, this is not an appropriate comparison. In addition, to claim that the partition should differ depending on the task, comparisons are needed, such as a comparison with training on Math using the Multi-hop partition.
Questions for authors
  1. According to Tables 8 and 9, Qwen 1.7B-Math seems to be about 85 steps. What setting do the 1000 steps in the loss graph of Figure 2 correspond to? Also, TA-IBAR is missing from the Peak Memory graph in Figure 2.
  2. At exactly which step is the validation loss measured? If this step is small, then considering the graph in Figure 2, one could claim that TA-IBAR converges faster, but it seems difficult to claim that it is effective once training has progressed sufficiently.
  3. In Block AttnRes, how many blocks are there for each model?
  4. By what criterion was the threshold value determined?
  5. Looking at Table 2, the scale of the Train loss for Qwen3-1.7B Math seems to be severely inconsistent with the train loss graph in Figure 2. Is this a simple error?
7 comments
AuthorsOpenReview ↗

Author Response to Reviewer P8Aj

Response to Reviewer P8Aj

Comment 1 of 3

W1 · Discovery data and stability

We collect residuals with frozen weights, compute centered linear CKA in FP32, and search for a partition offline. The discovery sets are fixed. The discovery table identifies the Qwen3-1.7B Math source as the Math pool. The training-data table separately identifies SVAMP as the adaptation data. We have not repeated discovery on independent subsets of the same task.

Revised Table 15. Qwen3 discovery configurations (submitted Table 10)

BackboneTaskExamples / dataMax lengthτNBlock sizes
Qwen3-1.7BMath200 / Math pool20480.4510[3,4,4,3,1,4,6,1,1,1]
Qwen3-1.7BMulti-hop200 / CLUTRR20480.4011[1,2,2,5,4,1,5,4,2,1,1]
Qwen3-4BMath200 / GSM8K+SVAMP+MATH+OpenMathInstruct-240960.5013[1,2,1,1,2,5,4,4,3,1,7,3,2]
Qwen3-4BMulti-hop200 / CLUTRR40960.4313[1,3,2,3,3,4,1,5,2,1,4,3,4]
Qwen3-8BMath500 / GSM8K 250 + SVAMP 25010240.5514[3,2,2,5,4,1,1,1,2,1,2,9,1,2]
Qwen3-8BMulti-hop500 / CLUTRR10240.6514[1,2,1,1,2,5,4,1,1,1,4,10,1,2]
Qwen3-14BMath300 / GSM8K 150 + SVAMP 15020480.5015[2,2,4,1,1,1,2,6,1,2,5,1,9,2,1]
Qwen3-14BMulti-hop300 / CLUTRR20480.5514[3,1,2,2,1,7,3,1,1,1,5,1,10,2]
Qwen3-14BCode300 / MBPP 224 + xCoder-80K 7640960.6518[1,1,2,1,2,3,1,4,2,1,5,1,4,2,3,1,4,2]

Comment 2 of 3

W2–3 and Q3 · Fixed baseline and measured cost

Our fixed baseline groups four consecutive Transformer blocks. At depths 28/36/36/40 this gives 7/9/9/10 groups. Kimi Team’s report evaluates several block sizes, including S = 4 under its own layer convention. Attention Residuals, §5.3, Table 4 and Figure 6

For 1.7B Math, TA-IBAR uses 10 groups versus 7 for fixed Block. Peak training memory is 10.51 versus 12.52 GB. The inference table shows 3.546 versus 3.612 GB and 20.55 versus 19.63 decode tokens/s. Retaining N depth summaries of width d costs O(Nd) elements per token. Total peak memory also includes model weights, caches and temporary buffers. We have not measured inter-stage traffic separately. Native uses the official checkpoint, Full uses the checkpoint at 566,588 training tokens, and TA-IBAR uses the final Math checkpoint. The revised paper gives peak memory and decode throughput. Prefill and probe latency, runtime memory bandwidth, and total memory with all routing banks resident were not measured separately.

New Table 7. Inference on Qwen3-1.7B

MethodPeak Mem. (GB) ↓Decode Tok/s ↑
Native3.46431.79
Full AttnRes3.67819.87
Block AttnRes3.61219.63
TA-IBAR3.54620.55

W3 · Bidirectional partition transfer

Each target uses its own training and evaluation data. We change which task supplied the fixed partition and then adapt on the target. Using the other task’s partition raises validation loss by 0.0460 on Math and 0.3512 on Multi-hop. The 10- and 11-block partitions differ in both count and boundaries.

New Table 10. Partition transfer on Qwen3-1.7B

Discovery taskTarget taskTrain lossValid. lossPeak GB
MathMath0.42060.416210.51
Multi-hopMath0.48710.462210.59
MathMulti-hop0.57530.595710.68
Multi-hopMulti-hop0.40920.244510.51

Comment 3 of 3

Q1–2 and Q5 · Step budget and corrected losses

The submitted configuration listed 350K tokens and a target of 4,096 tokens per update for Qwen3-1.7B Math. The revision gives 1M global non-padding tokens and a target of 1,024, equivalent to about 977 target-sized updates. The run logs are needed to verify how the plotted range through step 1,000 maps to optimizer updates and actual token exposure. The trajectory shows training loss. The revised main Math and Multi-hop tables use the mean of the final 100 optimizer-step training losses with consistent per-token normalization. Validation loss is evaluated at the final step. The training-loss and peak-memory panels of submitted Figure 2 are now Figures 2 and 3, respectively. The table below gives the corrected peak-memory values, including 10.51 GB for TA-IBAR and 12.52 GB for Block AttnRes. TA-IBAR reaches the lowest validation loss among the compared methods at this budget. Longer-budget convergence remains open.

Revised Table 3, Qwen3-1.7B Math

MethodTrain lossValid. lossPeak GB
Native0.49710.432610.08
Full AttnRes0.45350.476212.52
Block AttnRes0.43700.418212.52
TA-IBAR0.42060.416210.51

Q4 · Threshold selection

For Qwen3-1.7B Math, we swept τ over 0.30, 0.40, 0.45, 0.50, and 0.60 using the same cached CKA matrix and matched post-training settings. At 0.45, the default uses 10 blocks at 0.4162 validation loss and 10.51 GB. At 0.60, loss reaches 0.4135 with 16 blocks and 10.53 GB. We favor the more compact partition. The two 10-block partitions at 0.45 and 0.50 give different losses. The sweep measures threshold sensitivity on one matrix. Discovery stability across subsets or seeds remains open. The model- and task-specific values are in the discovery table in W1.

Revised Tables 8–9. Qwen3-1.7B Math threshold sweep (submitted Tables 5–6)

τPartitionNMax blockSingletonsTrain lossValid. lossPeak GB
0.30[3,11,1,11,1,1]61130.49750.443710.50
0.40[3,4,4,3,1,2,9,1,1]9930.43130.432810.50
0.45[3,4,4,3,1,4,6,1,1,1]10640.42060.416210.51
0.50[3,4,5,2,1,4,6,1,1,1]10640.41840.431310.53
0.60[1,2,4,4,1,1,1,1,1,1,3,4,1,1,1,1]164110.40290.413510.53

Thank you for the response and the revised version. Upon review, the following points are puzzling.

  1. In the revised version, the training budget, training data, order of post-training across tasks, and maximum sequence length changed substantially, yet the validation loss and peak GPU memory in Tables 3, 4, 5, and 16 are identical to those in the submitted version. Were the experiments re-run, or was the description of the settings in the submitted version incorrect?

  2. Figure 2 shows the same training-loss curves as in the submitted version, but the values over the final 100 optimisation steps do not match the Train Loss in Table 3. Are they from the same run?

  3. The CLUTRR accuracy in Table 6 is abnormally low for all methods. In prior work on CLUTRR-SG (Chen et al., 2023, RECKONING), GPT-3.5 achieves much higher accuracy even in the zero-shot setting. It is hard to accept that all methods post-trained on CLUTRR remain at this level.

The key weakness has also not been resolved. In the Qwen3 experiments, fixed Block AttnRes and TA-IBAR use different numbers of blocks, so the effect of the discovered depth partition cannot be judged from this comparison.

As above, there are too many inconsistencies between the experimental settings and the results, which raises strong doubts about the experimental setup itself. Therefore, I maintain my score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.