- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper asks how to combine information from different layers of a frozen language model when a task requires understanding and combining local pieces of text. The authors introduce seven diagnostic tasks and propose Token-Depth Aggregation (ToDA). ToDA first uses attention to combine each token's representations across layers. A graph network then connects the resulting token representations before they are combined into a single text vector.
ToDA improves over final-layer pooling and shared layer mixtures on several tasks and frozen backbones. Some of the largest gains are on target-specific review aggregation with the larger decoder models. The paper also reports improvements on GLUE and IMDB, and competitive or improved results on MTEB.
Critical strengths
1 Diagnostic tasks with a clear purpose I find the task suite useful, especially target-specific review aggregation. Positive and negative reviews are balanced across the input, so overall sentiment cannot give the answer. The model has to identify reviews of the queried business and combine their opinions. This makes the task a clear test of selecting and combining relevant local information. The other tasks cover composition, aspect information, and counting, which helps show where conventional pooling methods struggle.
2 Large gains on several tasks and useful comparisons The natural-text experiments select models on validation data and report test accuracy; the constructed tasks draw from disjoint source banks. Table 23 also compares ToDA with the stronger Non-adaptive ToDA baseline. Across all nine target-specific review settings on the three larger decoders, the paired intervals for the improvements are positive. This supports an advantage for the complete module under the reported procedures. The intervals reflect variation across data seeds, with fixed initialization, and do not by themselves isolate the effect of token-specific layer weights.
Critical weaknesses
1 The methodological contribution is not adequately isolated Section 2 contrasts ToDA with earlier approaches whose layer weights are said to be shared across tokens. This is too broad. SBERT-WK (Wang and Kuo, 2020, Section IV-A, Equations 3–10) already computes word-dependent layer weights from cross-layer alignment and novelty. Its weighting rule is geometric rather than learned final-layer-query attention, so it is not equivalent to ToDA. The paper should define the contribution as its particular learned depth-attention design and integration with the GLOT-style graph, rather than token-dependent layer weighting in general.
The ablation also leaves the source of the gain uncertain. Appendix Table 22 gives 20.21M trainable parameters for ToDA and 1.28M for GLOT on Mistral-7B target-specific reviews with K = 3. Most of ToDA's parameters belong to the dense residual gate in Equation (3). Appendix G describes Non-adaptive ToDA as a shared scalar mixture followed by the same graph network, but does not establish that it retains the full compression, projection, and gating path. Table 23 supports the complete module, but cannot attribute the improvement specifically to adaptive layer weights.
The claim that token-specific weights give the largest gain also needs a narrower scope. In Table 4, on Mistral-7B target-specific reviews with K = 7, adding shared layer mixing to AdaPool raises accuracy from 50.04 to 83.54, a 33.50-point gain. Adding the graph reaches 90.09, and full ToDA reaches 94.13, a further 4.04 points. On Mistral-7B aspect majority, the last step adds only 0.06 points. The larger effect on composed polarity shows that the pattern depends on the task. Moreover, the last step changes more than the weighting rule under the described implementation.
The most decisive control would keep the compression, projections, gate, and graph network while replacing the learned token-dependent weights with shared weights and matching capacity. GLOT applied to a validation-selected intermediate layer would provide a complementary check. The existing shared-mixture graph baseline already addresses shared access to depth; the extra check would test whether explicitly choosing one better layer is sufficient. Figure 3 and Appendix E describe layer-weight patterns, but those patterns alone do not identify the source of the gains.
2 Tuning and evaluation procedures differ across methods Appendix C.1 reports a search over 32 configurations for ToDA and eight for Layer Mix + AdaPool on target-specific reviews with K = 3. Other baselines receive narrower tuning, and selected configurations are reused elsewhere. A separate diagnostic adds another temperature candidate for ToDA. In the 40-epoch diagnostic runs discussed with Figure 2b, TinyLlama accuracy changes from 65.65 to 87.58 when the initial temperature changes. This shows that optimization settings matter, although it does not measure how much of the final advantage comes from the unequal searches. Comparable tuning on the decisive settings is needed.
For GLUE, Appendix F.1 uses the official validation splits both to select configurations and to report the best scores. The published GLOT results also come from hyperparameter selection, using a grid search. The comparison therefore combines separately selected results rather than a common rerun, and ToDA's reported scores are not an independent evaluation after selection. The IMDB split needs clarification. The separate CoLA capacity experiment in Appendix F.3 does hold out development data from training, but that does not resolve the selection issue in the main benchmark table.
For MTEB, Appendix F.2 excludes the six development tasks from the reported evaluation, and all methods undergo six epochs of final retraining. The searches before that stage differ. Baselines receive 32 trials, with batch sizes of 32, 64, or 128 and 12 training epochs per trial. Only ToDA's search includes the option of using hard negatives. Its BERT search uses 30 epochs, and the selected BERT configuration comes from an expanded 40-trial search allowing batch sizes of 128, 256, or 512. That search also allows contrastive temperatures of 0.01 and 0.03, in addition to 0.05 and 0.07, and learning rates up to 0.002, compared with 0.0006 for the baselines. These are search options, not necessarily the final settings. The reported selected ToDA batches are 64–256, and the paper does not say which final configurations use hard negatives. Final MTEB scores come from single training seeds. Table 5 reports gains over mean pooling of 2.38 points for BERT and 1.62 for TinyLlama, so the improvements are not uniformly negligible. On Mistral-7B, however, ToDA scores 48.61 against 48.53, a difference of 0.08 points. Repeated runs are particularly important for judging that small difference. Together with the asymmetric searches, this limits what the results establish about the aggregation architecture itself.
The composed-polarity experiment also reports the selected validation accuracy without a separate test set. This applies to that controlled experiment; the natural tasks do report test results. Using the same selection rule for all methods does not remove the need to evaluate the selected models on separate data.
Questions for authors
- Methodological contribution. How does ToDA's learned weighting differ from the word-dependent weighting in SBERT-WK, beyond the general fact that tokens receive different weights? Can the authors compare learned and shared weights while preserving the compression, projections, gate, graph network, and closely matched capacity on the strongest claimed settings?
- Comparable evaluation. How much of the gain remains with comparable tuning budgets? For GLUE, could model selection use a development split separate from the reported evaluation split, with baselines rerun under the same procedure? Please clarify the IMDB split as well. For MTEB, could the authors match search budgets, training epochs, batch-size ranges, and negative-sampling options, then repeat final training across seeds?
- Layer choice and generality. How does ToDA compare with GLOT using a single intermediate layer selected on validation data? This would complement the existing shared-mixture graph baseline and layerwise linear probes. Can the learned layer behavior also explain why target-specific reviews remain difficult for BERT, RoBERTa, and SmolLM2, as Appendix D acknowledges?
9 comments
Authors Responses (Part 2)
Regarding the matched shared-weight control and the scope of the claim on token-specific weights (W1(d), W1(e), Q1). We are grateful for this suggestion, which separates two things that the last row of Table 4 combines: the learned depth module of ToDA (compression, multi-head projections and residual gate) and the rule that computes the layer weights. We ran the control that you specify, ToDA with shared weights: one learned weight vector per head, shared by all tokens and inputs, replaces the token-specific attention weights, and the normalization, the compression, the values, the output projection, the residual gate, the graph network, the readout and the task head are kept; the query and key projections, whose role is to produce the attention weights, are removed (20.17M trainable parameters, compared with 20.21M for ToDA on Mistral-7B) (Table 35; target-specific reviews, eight data seeds; test accuracy (%)):
| Backbone | K | ToDA with shared weights | ToDA |
|---|---|---|---|
| TinyLlama | 3 | 84.48 | 87.48 |
| TinyLlama | 5 | 87.08 | 87.83 |
| TinyLlama | 7 | 88.35 | 89.54 |
| LLaMA-3B | 3 | 90.74 | 93.04 |
| LLaMA-3B | 5 | 88.98 | 92.04 |
| LLaMA-3B | 7 | 89.71 | 92.09 |
| Mistral-7B | 3 | 92.81 | 94.10 |
| Mistral-7B | 5 | 92.31 | 93.59 |
| Mistral-7B | 7 | 93.34 | 94.13 |
ToDA is higher in all nine settings, by 0.75 to 3.06 points. At accuracies of 84 to 94%, these differences correspond to 6 to 28% of the errors of ToDA with shared weights. Together with ToDA without the graph and the gated layer mix + AdaPool, which has shared weights and no graph, Tables 33 and 35 contain the four combinations of the two stages, each with its own selected configuration, and ToDA is the highest of the four in all nine settings.
Following your comment, Sections 1, 5.2 and 5.3 now state the claim per task. In Table 4, on Mistral-7B at K = 7, shared layer mixing adds 33.50 points to AdaPool (50.04 to 83.54), the graph adds 6.55 (to 90.09), and the last step adds 4.04 (to 94.13). The last step contains the complete depth module: in this setting ToDA with shared weights reaches 93.34 and ToDA 94.13, and ToDA is higher in all nine settings (Table 35). On aspect majority the last step of Table 4 is 0.06 points (80.83 to 80.89), and the multilayer designs perform comparably there (Sections 5.3 and 6). Token-specific weights therefore add a consistent improvement on the task in which tokens play different roles: in target-specific reviews, business names and opinion words place their weight on different layers (Figure 3; Appendix I.1).
You also note that the intervals of Table 23 vary the data seed at a fixed initialization. With two further initializations at K = 3 and K = 7 (eight data seeds), ToDA is above the gated layer mix + AdaPool in all 18 cells and above ToDA with shared weights in all 18. With the initializations and both K pooled (Table 45; test accuracy (%); mean over K = 3 and K = 7, the three initializations and the eight data seeds):
| Backbone | ToDA with shared weights | ToDA |
|---|---|---|
| TinyLlama | 84.42 | 88.01 |
| LLaMA-3B | 90.24 | 92.83 |
| Mistral-7B | 92.78 | 94.39 |
ToDA is higher on the three backbones, and the paired 95% bootstrap interval, which resamples the data seeds, is above zero on each.
Regarding GLOT on a validation-selected layer (W1(f), Q3). Thank you for suggesting this check, which ijXU requests as well. Three layers are shortlisted for each backbone and K by a layer probe; for each seed, validation selects the layer among them, and GLOT is trained on it (Table 33; target-specific reviews, test accuracy (%) at K = 3 / 5 / 7):
| Method | TinyLlama | LLaMA-3B | Mistral-7B |
|---|---|---|---|
| GLOT, selected layer | 84.05 / 87.90 / 89.85 | 91.83 / 90.79 / 91.35 | 93.26 / 92.38 / 93.00 |
| ToDA | 87.48 / 87.83 / 89.54 | 93.04 / 92.04 / 92.09 | 94.10 / 93.59 / 94.13 |
To your question whether explicitly choosing one better layer is sufficient: GLOT on the selected layer reaches 84 to 93%. On LLaMA-3B and Mistral-7B ToDA is 0.7 to 1.3 points higher; on TinyLlama it is 3.4 points higher at K = 3, and the two are at the same level at K = 5 and K = 7. The selection needs a probe at every layer, a shortlist and one training per shortlisted layer and seed, repeated for every backbone and K. On these three decoders, ToDA reaches the same level or a higher one without a probe and without a search over layers: it learns its weights over all layers during training. It is adaptive: every token selects its own depth, and the weights concentrate on the layers that the probe identifies (Figure 2).
Authors Responses (Part 3)
Regarding Figure 3 (W1(g)). We appreciate this comment. Following it, the attribution in Section 5.3 now cites the two interventions of Table 35, ToDA with shared weights and ToDA without the graph; Figure 3 and Appendix E describe the learned layer weights.
Regarding comparable tuning (W2(a) to W2(c), Q2). Thank you for this request, which ijXU shares. The baselines now have their own searches on the decisive setting (target-specific reviews at K = 3; TinyLlama and Mistral-7B; three seeds; selection on validation): GLOT over 96 configurations (learning rate, jumping knowledge, width, depth and threshold); layer mix + AdaPool and the gated layer mix + AdaPool over 32 each, both with the temperature candidates {0.1, 1.0} to which W2(c) refers and with two initializations; and AdaPool over its full grid of 8. The winners were rerun with eight data seeds. Table 30 reports the submitted and the re-tuned configuration of every baseline, because a winner selected on three seeds is not always above the submitted configuration with eight seeds (test accuracy (%) at K = 3 / 5 / 7):
| Method | TinyLlama | LLaMA-3B | Mistral-7B |
|---|---|---|---|
| AdaPool, Table 17 | 50.76 / 50.44 / 49.66 | 50.76 / 49.46 / 49.65 | 51.73 / 51.21 / 50.04 |
| AdaPool, winner of 8 | 49.65 / 50.78 / 50.14 | 49.91 / 49.53 / 49.36 | 50.63 / 51.03 / 50.11 |
| GLOT, Table 17 | 66.71 / 65.19 / 81.85 | 56.90 / 52.76 / 50.60 | 50.51 / 51.11 / 53.59 |
| GLOT, winner of 96 | 64.79 / 74.35 / 78.38 | 49.73 / 49.86 / 49.61 | 51.04 / 64.55 / 68.43 |
| Layer mix + AdaPool, Table 17 | 56.45 / 59.31 / 62.78 | 72.40 / 73.90 / 75.58 | 79.35 / 81.83 / 83.54 |
| Layer mix + AdaPool, winner of 32 | 51.85 / 60.43 / 56.70 | 74.10 / 75.54 / 84.85 | 72.79 / 79.19 / 87.95 |
| Gated layer mix + AdaPool, earlier configuration | 54.16 / 60.58 / 59.91 | 85.98 / 88.04 / 87.81 | 91.21 / 91.69 / 91.35 |
| Gated layer mix + AdaPool, winner of 32 | 58.49 / 68.79 / 62.88 | 87.45 / 87.33 / 87.20 | 90.24 / 89.98 / 91.05 |
| ToDA (32 configurations) | 87.48 / 87.83 / 89.54 | 93.04 / 92.04 / 92.09 | 94.10 / 93.59 / 94.13 |
To your question how much of the improvement remains: ToDA is above every configuration of every baseline of Table 30 in all nine settings, by 1.9 points or more on Mistral-7B, 2.5 or more on LLaMA-3B (relative to the gated layer mix + AdaPool searched on LLaMA-3B itself, below) and 7.7 or more on TinyLlama, with the paired 95% intervals above zero. On the two larger decoders the highest baseline is the gated layer mix + AdaPool, which shares the depth module of ToDA. In the reported results, validation selects among three candidates of ToDA.
Regarding the reuse of configurations (W2(b)): the reuse is the same for every method (Appendix C.1), and in Table 30 LLaMA-3B uses the winners of Mistral-7B. Following your comment, we also ran the searches on LLaMA-3B itself, with the same budgets, and reran the winners with eight seeds (Table 31; test accuracy (%) at K = 3 / 5 / 7):
| Method, searched on LLaMA-3B itself | LLaMA-3B |
|---|---|
| AdaPool | 50.33 / 50.59 / 49.99 |
| GLOT | 50.74 / 50.94 / 49.91 |
| Layer mix + AdaPool | 79.63 / 76.19 / 85.60 |
| Gated layer mix + AdaPool | 90.50 / 86.69 / 88.33 |
| ToDA | 93.04 / 92.04 / 92.09 |
ToDA is above each baseline in all twelve comparisons, with every paired interval above zero (Appendix H.1).
Authors Responses (Part 1)
We sincerely thank you for the precise and constructive review. We are encouraged that you find the task suite "useful, especially target-specific review aggregation", and the task "a clear test of selecting and combining relevant local information". We refer to your points as W1, W2, ... and Q1, Q2, ... in the order of your review, with letters for their parts. We hope that you find our responses satisfactory, and that you will consider revising your score.
In short: your review asks which part of ToDA produces the improvement, and whether it remains under comparable procedures. (1) The control that you describe as the "most decisive" is run, together with GLOT on a validation-selected layer (W1, Q1, Q3). (2) GLOT and the two layer-mix baselines have searches of equal or larger size on the decisive settings, and AdaPool has its full grid (W2, Q2). (3) GLUE and IMDB are rerun with a development split, MTEB has three training seeds and its search is examined in a common space, and composed polarity has a held-out test split (W2, Q2). (4) Section 2 defines the contribution following your comment, and Sections 1, 5.2 and 5.3 state the attribution per task (W1). Each table below gives the rows of the tables of the general response that the answer uses.
Regarding SBERT-WK and the definition of the contribution (W1(a), Q1). Thank you for this precise comment. Following it, Section 2 now states that SBERT-WK [1] computes word-dependent layer weights, and defines the contribution of the paper as its learned depth-attention design and the integration with the token graph. The difference with SBERT-WK is in how the weights are obtained and used. SBERT-WK derives the weights of each word from a fixed geometric rule on its layer trajectory (alignment and novelty across neighboring layers), without training and independently of the task. In ToDA the weights are (i) learned for the task, (ii) computed by attention, with the final-layer state of the token as the query and its layer states as keys, and (iii) used twice: the fused vectors are the node features, and they define the edges of the token graph.
Regarding capacity (W1(b)). Thank you for this observation. Three controls address capacity on target-specific reviews (Table 33; test accuracy (%) at K = 3 / 5 / 7; parameter counts for Mistral-7B):
On the setting that you cite (Mistral-7B, K = 3), the gated layer mix + AdaPool, which contains the gate, reaches 91.21, GLOT widened to 23.1M parameters 50.96, and ToDA with shared weights 92.81, compared with 94.10 for ToDA. The parameters of the gate therefore do not account for the difference with GLOT (43 points) or with the controls that contain the gate.
Regarding Non-adaptive ToDA (W1(c)). Thank you; following your comment, Appendix G now states the definition in one sentence. Non-adaptive ToDA applies GLOT to the layer mixture (Section 5.3, Appendix G): shared scalar weights combine the raw layer states, followed by the same graph network, without the compression, the projections or the gate of ToDA. Table 23 therefore compares the complete module with this baseline, and the control that isolates the weighting rule is the one that you propose in W1(e).