acceptodds
Under review as a conference paper at ICLR 2027

Token-Depth Aggregation for Improved Text Representations

Abstract

Frozen language models are widely used as text encoders, with a trainable aggregation module that combines their token representations into one vector. Existing aggregation methods, from mean pooling to learned attention and token graphs, read the final layer and are mostly evaluated on benchmarks whose labels reflect the overall content of a sentence. Many uses instead require composing local meaning, such as a negated opinion or a chain of family relations, and aggregating it across a text, such as the majority opinion about one business among reviews of several. In this paper, we ask how the representations of a frozen language model should be combined across depth and across tokens when the label is built from such local information. We introduce a suite of composition and aggregation tasks whose labels are built from local judgments while global cues are removed, and propose Token-Depth Aggregation (), in which each token attends over its own representations at all layers before a token graph relates the fused tokens. Our experiments show that existing aggregation methods can remain close to chance level when the answer depends on tying opinions to a specific target, whereas ToDA reaches high accuracy. Across frozen encoder and decoder backbones, ToDA achieves the highest accuracy in nearly every setting of the proposed tasks. On the standard GLUE, IMDB and MTEB benchmarks, ToDA also achieves competitive or improved performance while keeping the backbone frozen and training a small number of parameters.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Sentence Representations, Pretrained Language Models, Multi-Layer Representations, Representation Fusion, Sentence Embeddings

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_NujPOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper asks how to combine information from different layers of a frozen language model when a task requires understanding and combining local pieces of text. The authors introduce seven diagnostic tasks and propose Token-Depth Aggregation (ToDA). ToDA first uses attention to combine each token's representations across layers. A graph network then connects the resulting token representations before they are combined into a single text vector.

ToDA improves over final-layer pooling and shared layer mixtures on several tasks and frozen backbones. Some of the largest gains are on target-specific review aggregation with the larger decoder models. The paper also reports improvements on GLUE and IMDB, and competitive or improved results on MTEB.

Critical strengths

1 Diagnostic tasks with a clear purpose I find the task suite useful, especially target-specific review aggregation. Positive and negative reviews are balanced across the input, so overall sentiment cannot give the answer. The model has to identify reviews of the queried business and combine their opinions. This makes the task a clear test of selecting and combining relevant local information. The other tasks cover composition, aspect information, and counting, which helps show where conventional pooling methods struggle.

2 Large gains on several tasks and useful comparisons The natural-text experiments select models on validation data and report test accuracy; the constructed tasks draw from disjoint source banks. Table 23 also compares ToDA with the stronger Non-adaptive ToDA baseline. Across all nine target-specific review settings on the three larger decoders, the paired intervals for the improvements are positive. This supports an advantage for the complete module under the reported procedures. The intervals reflect variation across data seeds, with fixed initialization, and do not by themselves isolate the effect of token-specific layer weights.

Critical weaknesses

1 The methodological contribution is not adequately isolated Section 2 contrasts ToDA with earlier approaches whose layer weights are said to be shared across tokens. This is too broad. SBERT-WK (Wang and Kuo, 2020, Section IV-A, Equations 3–10) already computes word-dependent layer weights from cross-layer alignment and novelty. Its weighting rule is geometric rather than learned final-layer-query attention, so it is not equivalent to ToDA. The paper should define the contribution as its particular learned depth-attention design and integration with the GLOT-style graph, rather than token-dependent layer weighting in general.

The ablation also leaves the source of the gain uncertain. Appendix Table 22 gives 20.21M trainable parameters for ToDA and 1.28M for GLOT on Mistral-7B target-specific reviews with K = 3. Most of ToDA's parameters belong to the dense residual gate in Equation (3). Appendix G describes Non-adaptive ToDA as a shared scalar mixture followed by the same graph network, but does not establish that it retains the full compression, projection, and gating path. Table 23 supports the complete module, but cannot attribute the improvement specifically to adaptive layer weights.

The claim that token-specific weights give the largest gain also needs a narrower scope. In Table 4, on Mistral-7B target-specific reviews with K = 7, adding shared layer mixing to AdaPool raises accuracy from 50.04 to 83.54, a 33.50-point gain. Adding the graph reaches 90.09, and full ToDA reaches 94.13, a further 4.04 points. On Mistral-7B aspect majority, the last step adds only 0.06 points. The larger effect on composed polarity shows that the pattern depends on the task. Moreover, the last step changes more than the weighting rule under the described implementation.

The most decisive control would keep the compression, projections, gate, and graph network while replacing the learned token-dependent weights with shared weights and matching capacity. GLOT applied to a validation-selected intermediate layer would provide a complementary check. The existing shared-mixture graph baseline already addresses shared access to depth; the extra check would test whether explicitly choosing one better layer is sufficient. Figure 3 and Appendix E describe layer-weight patterns, but those patterns alone do not identify the source of the gains.

2 Tuning and evaluation procedures differ across methods Appendix C.1 reports a search over 32 configurations for ToDA and eight for Layer Mix + AdaPool on target-specific reviews with K = 3. Other baselines receive narrower tuning, and selected configurations are reused elsewhere. A separate diagnostic adds another temperature candidate for ToDA. In the 40-epoch diagnostic runs discussed with Figure 2b, TinyLlama accuracy changes from 65.65 to 87.58 when the initial temperature changes. This shows that optimization settings matter, although it does not measure how much of the final advantage comes from the unequal searches. Comparable tuning on the decisive settings is needed.

For GLUE, Appendix F.1 uses the official validation splits both to select configurations and to report the best scores. The published GLOT results also come from hyperparameter selection, using a grid search. The comparison therefore combines separately selected results rather than a common rerun, and ToDA's reported scores are not an independent evaluation after selection. The IMDB split needs clarification. The separate CoLA capacity experiment in Appendix F.3 does hold out development data from training, but that does not resolve the selection issue in the main benchmark table.

For MTEB, Appendix F.2 excludes the six development tasks from the reported evaluation, and all methods undergo six epochs of final retraining. The searches before that stage differ. Baselines receive 32 trials, with batch sizes of 32, 64, or 128 and 12 training epochs per trial. Only ToDA's search includes the option of using hard negatives. Its BERT search uses 30 epochs, and the selected BERT configuration comes from an expanded 40-trial search allowing batch sizes of 128, 256, or 512. That search also allows contrastive temperatures of 0.01 and 0.03, in addition to 0.05 and 0.07, and learning rates up to 0.002, compared with 0.0006 for the baselines. These are search options, not necessarily the final settings. The reported selected ToDA batches are 64–256, and the paper does not say which final configurations use hard negatives. Final MTEB scores come from single training seeds. Table 5 reports gains over mean pooling of 2.38 points for BERT and 1.62 for TinyLlama, so the improvements are not uniformly negligible. On Mistral-7B, however, ToDA scores 48.61 against 48.53, a difference of 0.08 points. Repeated runs are particularly important for judging that small difference. Together with the asymmetric searches, this limits what the results establish about the aggregation architecture itself.

The composed-polarity experiment also reports the selected validation accuracy without a separate test set. This applies to that controlled experiment; the natural tasks do report test results. Using the same selection rule for all methods does not remove the need to evaluate the selected models on separate data.

Questions for authors
  1. Methodological contribution. How does ToDA's learned weighting differ from the word-dependent weighting in SBERT-WK, beyond the general fact that tokens receive different weights? Can the authors compare learned and shared weights while preserving the compression, projections, gate, graph network, and closely matched capacity on the strongest claimed settings?
  2. Comparable evaluation. How much of the gain remains with comparable tuning budgets? For GLUE, could model selection use a development split separate from the reported evaluation split, with baselines rerun under the same procedure? Please clarify the IMDB split as well. For MTEB, could the authors match search budgets, training epochs, batch-size ranges, and negative-sampling options, then repeat final training across seeds?
  3. Layer choice and generality. How does ToDA compare with GLOT using a single intermediate layer selected on validation data? This would complement the existing shared-mixture graph baseline and layerwise linear probes. Can the learned layer behavior also explain why target-specific reviews remain difficult for BERT, RoBERTa, and SmolLM2, as Appendix D acknowledges?
9 comments
AuthorsOpenReview ↗

Authors Responses (Part 1)

We sincerely thank you for the precise and constructive review. We are encouraged that you find the task suite "useful, especially target-specific review aggregation", and the task "a clear test of selecting and combining relevant local information". We refer to your points as W1, W2, ... and Q1, Q2, ... in the order of your review, with letters for their parts. We hope that you find our responses satisfactory, and that you will consider revising your score.


In short: your review asks which part of ToDA produces the improvement, and whether it remains under comparable procedures. (1) The control that you describe as the "most decisive" is run, together with GLOT on a validation-selected layer (W1, Q1, Q3). (2) GLOT and the two layer-mix baselines have searches of equal or larger size on the decisive settings, and AdaPool has its full grid (W2, Q2). (3) GLUE and IMDB are rerun with a development split, MTEB has three training seeds and its search is examined in a common space, and composed polarity has a held-out test split (W2, Q2). (4) Section 2 defines the contribution following your comment, and Sections 1, 5.2 and 5.3 state the attribution per task (W1). Each table below gives the rows of the tables of the general response that the answer uses.

Regarding SBERT-WK and the definition of the contribution (W1(a), Q1). Thank you for this precise comment. Following it, Section 2 now states that SBERT-WK [1] computes word-dependent layer weights, and defines the contribution of the paper as its learned depth-attention design and the integration with the token graph. The difference with SBERT-WK is in how the weights are obtained and used. SBERT-WK derives the weights of each word from a fixed geometric rule on its layer trajectory (alignment and novelty across neighboring layers), without training and independently of the task. In ToDA the weights are (i) learned for the task, (ii) computed by attention, with the final-layer state of the token as the query and its layer states as keys, and (iii) used twice: the fused vectors are the node features, and they define the edges of the token graph.

Regarding capacity (W1(b)). Thank you for this observation. Three controls address capacity on target-specific reviews (Table 33; test accuracy (%) at K = 3 / 5 / 7; parameter counts for Mistral-7B):

MethodParamsTinyLlamaLLaMA-3BMistral-7B
GLOT, width 2048, final layer23.1M49.89 / 50.28 / 54.3049.93 / 49.46 / 49.6850.96 / 51.45 / 51.10
Gated layer mix + AdaPool19.9M54.16 / 60.58 / 59.9185.98 / 88.04 / 87.8191.21 / 91.69 / 91.35
ToDA with shared weights20.17M84.48 / 87.08 / 88.3590.74 / 88.98 / 89.7192.81 / 92.31 / 93.34
ToDA20.21M87.48 / 87.83 / 89.5493.04 / 92.04 / 92.0994.10 / 93.59 / 94.13

On the setting that you cite (Mistral-7B, K = 3), the gated layer mix + AdaPool, which contains the gate, reaches 91.21, GLOT widened to 23.1M parameters 50.96, and ToDA with shared weights 92.81, compared with 94.10 for ToDA. The parameters of the gate therefore do not account for the difference with GLOT (43 points) or with the controls that contain the gate.

Regarding Non-adaptive ToDA (W1(c)). Thank you; following your comment, Appendix G now states the definition in one sentence. Non-adaptive ToDA applies GLOT to the layer mixture (Section 5.3, Appendix G): shared scalar weights combine the raw layer states, followed by the same graph network, without the compression, the projections or the gate of ToDA. Table 23 therefore compares the complete module with this baseline, and the control that isolates the weighting rule is the one that you propose in W1(e).

AuthorsOpenReview ↗

Authors Responses (Part 2)

Regarding the matched shared-weight control and the scope of the claim on token-specific weights (W1(d), W1(e), Q1). We are grateful for this suggestion, which separates two things that the last row of Table 4 combines: the learned depth module of ToDA (compression, multi-head projections and residual gate) and the rule that computes the layer weights. We ran the control that you specify, ToDA with shared weights: one learned weight vector per head, shared by all tokens and inputs, replaces the token-specific attention weights, and the normalization, the compression, the values, the output projection, the residual gate, the graph network, the readout and the task head are kept; the query and key projections, whose role is to produce the attention weights, are removed (20.17M trainable parameters, compared with 20.21M for ToDA on Mistral-7B) (Table 35; target-specific reviews, eight data seeds; test accuracy (%)):

BackboneKToDA with shared weightsToDA
TinyLlama384.4887.48
TinyLlama587.0887.83
TinyLlama788.3589.54
LLaMA-3B390.7493.04
LLaMA-3B588.9892.04
LLaMA-3B789.7192.09
Mistral-7B392.8194.10
Mistral-7B592.3193.59
Mistral-7B793.3494.13

ToDA is higher in all nine settings, by 0.75 to 3.06 points. At accuracies of 84 to 94%, these differences correspond to 6 to 28% of the errors of ToDA with shared weights. Together with ToDA without the graph and the gated layer mix + AdaPool, which has shared weights and no graph, Tables 33 and 35 contain the four combinations of the two stages, each with its own selected configuration, and ToDA is the highest of the four in all nine settings.

Following your comment, Sections 1, 5.2 and 5.3 now state the claim per task. In Table 4, on Mistral-7B at K = 7, shared layer mixing adds 33.50 points to AdaPool (50.04 to 83.54), the graph adds 6.55 (to 90.09), and the last step adds 4.04 (to 94.13). The last step contains the complete depth module: in this setting ToDA with shared weights reaches 93.34 and ToDA 94.13, and ToDA is higher in all nine settings (Table 35). On aspect majority the last step of Table 4 is 0.06 points (80.83 to 80.89), and the multilayer designs perform comparably there (Sections 5.3 and 6). Token-specific weights therefore add a consistent improvement on the task in which tokens play different roles: in target-specific reviews, business names and opinion words place their weight on different layers (Figure 3; Appendix I.1).

You also note that the intervals of Table 23 vary the data seed at a fixed initialization. With two further initializations at K = 3 and K = 7 (eight data seeds), ToDA is above the gated layer mix + AdaPool in all 18 cells and above ToDA with shared weights in all 18. With the initializations and both K pooled (Table 45; test accuracy (%); mean over K = 3 and K = 7, the three initializations and the eight data seeds):

BackboneToDA with shared weightsToDA
TinyLlama84.4288.01
LLaMA-3B90.2492.83
Mistral-7B92.7894.39

ToDA is higher on the three backbones, and the paired 95% bootstrap interval, which resamples the data seeds, is above zero on each.

Regarding GLOT on a validation-selected layer (W1(f), Q3). Thank you for suggesting this check, which ijXU requests as well. Three layers are shortlisted for each backbone and K by a layer probe; for each seed, validation selects the layer among them, and GLOT is trained on it (Table 33; target-specific reviews, test accuracy (%) at K = 3 / 5 / 7):

MethodTinyLlamaLLaMA-3BMistral-7B
GLOT, selected layer84.05 / 87.90 / 89.8591.83 / 90.79 / 91.3593.26 / 92.38 / 93.00
ToDA87.48 / 87.83 / 89.5493.04 / 92.04 / 92.0994.10 / 93.59 / 94.13

To your question whether explicitly choosing one better layer is sufficient: GLOT on the selected layer reaches 84 to 93%. On LLaMA-3B and Mistral-7B ToDA is 0.7 to 1.3 points higher; on TinyLlama it is 3.4 points higher at K = 3, and the two are at the same level at K = 5 and K = 7. The selection needs a probe at every layer, a shortlist and one training per shortlisted layer and seed, repeated for every backbone and K. On these three decoders, ToDA reaches the same level or a higher one without a probe and without a search over layers: it learns its weights over all layers during training. It is adaptive: every token selects its own depth, and the weights concentrate on the layers that the probe identifies (Figure 2).

AuthorsOpenReview ↗

Authors Responses (Part 3)

Regarding Figure 3 (W1(g)). We appreciate this comment. Following it, the attribution in Section 5.3 now cites the two interventions of Table 35, ToDA with shared weights and ToDA without the graph; Figure 3 and Appendix E describe the learned layer weights.

Regarding comparable tuning (W2(a) to W2(c), Q2). Thank you for this request, which ijXU shares. The baselines now have their own searches on the decisive setting (target-specific reviews at K = 3; TinyLlama and Mistral-7B; three seeds; selection on validation): GLOT over 96 configurations (learning rate, jumping knowledge, width, depth and threshold); layer mix + AdaPool and the gated layer mix + AdaPool over 32 each, both with the temperature candidates {0.1, 1.0} to which W2(c) refers and with two initializations; and AdaPool over its full grid of 8. The winners were rerun with eight data seeds. Table 30 reports the submitted and the re-tuned configuration of every baseline, because a winner selected on three seeds is not always above the submitted configuration with eight seeds (test accuracy (%) at K = 3 / 5 / 7):

MethodTinyLlamaLLaMA-3BMistral-7B
AdaPool, Table 1750.76 / 50.44 / 49.6650.76 / 49.46 / 49.6551.73 / 51.21 / 50.04
AdaPool, winner of 849.65 / 50.78 / 50.1449.91 / 49.53 / 49.3650.63 / 51.03 / 50.11
GLOT, Table 1766.71 / 65.19 / 81.8556.90 / 52.76 / 50.6050.51 / 51.11 / 53.59
GLOT, winner of 9664.79 / 74.35 / 78.3849.73 / 49.86 / 49.6151.04 / 64.55 / 68.43
Layer mix + AdaPool, Table 1756.45 / 59.31 / 62.7872.40 / 73.90 / 75.5879.35 / 81.83 / 83.54
Layer mix + AdaPool, winner of 3251.85 / 60.43 / 56.7074.10 / 75.54 / 84.8572.79 / 79.19 / 87.95
Gated layer mix + AdaPool, earlier configuration54.16 / 60.58 / 59.9185.98 / 88.04 / 87.8191.21 / 91.69 / 91.35
Gated layer mix + AdaPool, winner of 3258.49 / 68.79 / 62.8887.45 / 87.33 / 87.2090.24 / 89.98 / 91.05
ToDA (32 configurations)87.48 / 87.83 / 89.5493.04 / 92.04 / 92.0994.10 / 93.59 / 94.13

To your question how much of the improvement remains: ToDA is above every configuration of every baseline of Table 30 in all nine settings, by 1.9 points or more on Mistral-7B, 2.5 or more on LLaMA-3B (relative to the gated layer mix + AdaPool searched on LLaMA-3B itself, below) and 7.7 or more on TinyLlama, with the paired 95% intervals above zero. On the two larger decoders the highest baseline is the gated layer mix + AdaPool, which shares the depth module of ToDA. In the reported results, validation selects among three candidates of ToDA.

Regarding the reuse of configurations (W2(b)): the reuse is the same for every method (Appendix C.1), and in Table 30 LLaMA-3B uses the winners of Mistral-7B. Following your comment, we also ran the searches on LLaMA-3B itself, with the same budgets, and reran the winners with eight seeds (Table 31; test accuracy (%) at K = 3 / 5 / 7):

Method, searched on LLaMA-3B itselfLLaMA-3B
AdaPool50.33 / 50.59 / 49.99
GLOT50.74 / 50.94 / 49.91
Layer mix + AdaPool79.63 / 76.19 / 85.60
Gated layer mix + AdaPool90.50 / 86.69 / 88.33
ToDA93.04 / 92.04 / 92.09

ToDA is above each baseline in all twelve comparisons, with every paired interval above zero (Appendix H.1).

Reviewer_11ZxOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents a new framework, ToDA, for aggregating representations of a frozen language model into a single text vector. Each token attends over its own representations across layers, and the fused tokens are then related through a graph neural network over a token-similarity graph. It also introduces seven composition and aggregation tasks that depend on selectively combining local information, such as the majority sentiment of reviews for one queried business. ToDA claims to outperform baselines in nearly all of these tasks, and reports improvements on tasks such as GLUE, IMDB and MTEB, among others.

Critical strengths

The analysis is careful to examine tasks that require aggregating local meaning, which existing pooling methods perform poorly on. Standard benchmarks like GLUE and MTEB don’t specifically test whether pooled representations preserve such information. The proposed method shows strong and consistent improvements over baselines on the proposed tasks, such as on target-specific reviews where ToDA reaches around 92-93% while most existing baselines achieve only random chance.

Critical weaknesses

I have a major concern about the limited significance of this paper as we head into 2027. The benchmarks on GLUE, IMDB, and various forms of classification tasks were important several years ago, but today, it costs $0.042 (USD) per million input tokens to run a relatively capable decision model like Jev 1.14 on OpenRouter (output tokens are free). There are open-weight decision models as well which are likewise cheap to download and run locally as desired. Running a full model on text is so inexpensive that generating document embeddings with a frozen LM is no longer necessary for tasks like those contemplated in this paper. The community has, for the most part, advanced beyond classification benchmarks like Yelp reviews, GLUE, IMDB, etc. for understanding the state of the art and the frontier of our field.

This is not to say that sentence and document embeddings don't have applications in modern systems. The major one that comes to mind is dense retrieval, which is foundational to RAG. However, it isn't realistic to use a frozen model to generate the embeddings, since document embedding models are trained specifically for the retrieval task. A model that achieves 38.71 on the MTEB retrieval subset used here is not competitive with a much smaller (but dedicated) embedding model. EmbeddingGemma (308M parameters; Schechter Vera et al., 2025) outperforms ToDA (which uses a frozen Mistral-7B backbone) on every retrieval task the two evaluations share: ArguAna (71.54 vs 41.50), FiQA2018 (47.74 vs 21.27), SCIDOCS (18.43 vs 14.50) and TREC-COVID (80.35 vs 67.80), while being roughly 23x smaller than the backbone ToDA reads from. I don't mean to suggest that the authors need to run a comparison against a Google model that was trained on more data, but I wish to emphasize that a tiny unfrozen model may outperform a much larger frozen backbone in the final analysis.

The comparisons are also somewhat unfair since they are not capacity-matched. ToDA uses 6-20M parameters on target-specific reviews vs 0.76-1.28M for GLOT and 0.53-1.06M for AdaPool in Table 22, for example. ToDA has roughly 8-16x as many trainable parameters as GLOT, so parameter count might explain the difference rather than the proposed design.

Questions for authors

Could the authors examine parameter-matched baselines, like a wider GLOT or the same residual gate added to the baselines?

4 comments
AuthorsOpenReview ↗

Authors Responses (Part 1)

We sincerely thank you for the review and for the constructive feedback. We are encouraged that you find the analysis "careful to examine tasks that require aggregating local meaning" and that you note "strong and consistent improvements over baselines on the proposed tasks". We refer to your points as W1, W2, ... and Q1, Q2, ... in the order of your review, with letters for their parts. We hope that you find our responses satisfactory, and that you will consider revising your score.


In short: your review asks what representations of a frozen LM contribute today, and whether capacity explains the result. (1) Both parameter-matched baselines that you propose are run, and neither reaches ToDA (W3, Q1). (2) The embedding function in our retrieval experiment is trained for retrieval, and on retrieval and STS ToDA improves over mean aggregation on every backbone with three training seeds (W2). (3) We measured the decision model that you name on our tasks (W1).

Regarding the role of representations from frozen LMs (W1). Thank you for this perspective. The object of the paper is the representation that a frozen LM already computes: whether the information that a label needs is accessible, at which depth, and which aggregation reads it. The classification tasks are the instrument of this measurement and not its application. In target-specific reviews the overall sentiment is balanced and the two businesses have opposite majorities, and the label therefore requires binding each opinion to its target. In the query-first format, every method of Table 3 that reads the final layer remains at 49 to 53% on LLaMA-3B and Mistral-7B, and ToDA, in which each token selects its depth, reaches 92 to 94%. This regards every system that reads hidden states of an LM, including embedding models built on large decoders and the retrieval pipelines that you mention. In retrieval, a representation also serves a different use from a prompted model: it is computed once per document and reused for every query and index, whereas a decision model reads every document again for every question.

Inspired by your comment, we also measured the model that you name. We sent the test sets of the five tasks below to Jev 1.14, zero-shot (66,845 calls). Each call contains the text of one test example and one typed question with a criterion for every answer. On target-specific reviews the question is "The text contains reviews of several businesses. Considering only the reviews of X, is the overall opinion about X positive or negative?", with the criteria "More of the reviews about X are positive than negative." and "More of the reviews about X are negative than positive.", where X is the queried business. Appendix L lists the question of every task. We compare the model with ToDA trained on each task over a frozen Mistral-7B (Table 49; accuracy (%)):

TaskJev 1.14, zero-shotToDA, frozen Mistral-7B
Kinship (two / three / four hops)85.6 / 60.5 / 50.399.0 / 96.3 / 96.8
Aspect sentiment56.079.9
Review counting (K = 5 / 9)48.1 / 39.762.9 / 46.4
Aspect majority (K = 1 / 3 / 5)85.4 / 79.6 / 78.587.0 / 80.8 / 80.9
Target-specific reviews (K = 3 / 5 / 7)94.4 / 93.9 / 93.394.1 / 93.6 / 94.1

ToDA is 13 to 46 points above the prompted model on kinship, 24 points above it on aspect sentiment, 7 to 15 points above it on review counting and 1.3 to 2.4 points above it on aspect majority, and the two are at the same level on target-specific reviews. On aspect sentiment, three quarters of the errors of the prompted model are on the neutral class: it assigns a polarity to 441 of the 607 aspects that a sentence mentions without an opinion. The comparison is between a trained aggregation and a zero-shot model and does not rank deployments: it shows that a frozen backbone with a trained aggregation is at the level of the prompted model or above it on these tasks. Section 5.4 now states that the benchmarks compare aggregation methods on a frozen backbone.

AuthorsOpenReview ↗

Authors Responses (Part 2)

Regarding retrieval and dedicated embedding models (W2). Thank you for raising this point, which helped us make the setting explicit. In our retrieval experiment the embedding function is trained for retrieval: the aggregation method and its output projection are trained with a contrastive objective on MS MARCO passage pairs, and the backbone is frozen, with its hidden states computed once and cached (Appendix F.2). The setting is the adaptation of a readout on the states of a fixed LM, without backpropagation through the LM. As you note, a comparison with a model trained end to end on more data is not what the paper sets out to make, and the paper makes no claim relative to dedicated embedding models. Within this setting, ToDA improves over mean aggregation on the two task families that correspond to the use that you describe (Table 40; ×100, mean over the three training seeds; number of categories in parentheses):

Family (categories)Mean aggregation, BERTToDA, BERTMean aggregation, TinyLlamaToDA, TinyLlamaMean aggregation, Mistral-7BToDA, Mistral-7B
Retrieval (8)26.4234.9532.0439.9836.4440.53
STS (9)55.9860.6258.0363.0859.5562.92

The two directions are complementary. A dedicated embedding model, EmbeddingGemma [1] included, still contains an aggregation step, which turns its token states into one vector. Our study characterizes what this final-layer aggregation retains. Mean aggregation is strong for topic-level tasks (classification and clustering, Table 26), whereas on target-specific reviews, whose label requires binding local judgments to a target, no final-layer method of Table 33 exceeds 59% on LLaMA-3B or Mistral-7B in the query-first format, including GLOT with 23.1M parameters. Section 5.4 now states this scope.

Regarding capacity (W3, Q1). Thank you for this important question, which ijXU and NujP share. Both baselines that you propose are run on target-specific reviews (Table 33; test accuracy (%) at K = 3 / 5 / 7; parameter counts for Mistral-7B):

MethodParamsTinyLlamaLLaMA-3BMistral-7B
GLOT, final layer (Table 17)1.28M66.71 / 65.19 / 81.8556.90 / 52.76 / 50.6050.51 / 51.11 / 53.59
GLOT, width 1024, final layer7.9M49.79 / 50.23 / 54.6850.23 / 49.20 / 50.1649.83 / 51.91 / 49.53
GLOT, width 2048, final layer23.1M49.89 / 50.28 / 54.3049.93 / 49.46 / 49.6850.96 / 51.45 / 51.10
Gated layer mix + AdaPool19.9M54.16 / 60.58 / 59.9185.98 / 88.04 / 87.8191.21 / 91.69 / 91.35
ToDA20.21M87.48 / 87.83 / 89.5493.04 / 92.04 / 92.0994.10 / 93.59 / 94.13

(i) The depth module of ToDA with shared weights and an AdaPool readout, which contains the same residual gate: the gated layer mix + AdaPool, with 19.9M trainable parameters on Mistral-7B (ToDA: 20.21M), is below ToDA in all nine settings, with every paired 95% interval above zero. (ii) A wider GLOT: with 7.9M parameters, and with 23.1M parameters, more than ToDA, it remains at 49 to 52% on LLaMA-3B and Mistral-7B and at 50 to 55% on TinyLlama. The D × D projection of the residual gate holds the larger part of the parameters of ToDA (Appendix D); with the same gate, the baseline remains below ToDA in all nine settings (Appendix H.2).


Summary. We thank you again for your feedback, which strengthened our work and the presentation of its contribution. In response to it we added the two parameter-matched baselines (Table 33), MTEB with three training seeds and the task families (Tables 39 and 40) and the measurement of Jev 1.14 on our tasks. We hope that these results answer the points that you raised, and that you will consider revising your score. We would be happy to address any further questions.


References:

[1] EmbeddingGemma: Powerful and Lightweight Text Representations.

Dear Reviewer 11Zx,

Today is the last day of the discussion period, and we have not yet received a reply from you on our responses.

Thank you again for your constructive review. In particular, you asked:

“Could the authors examine parameter-matched baselines, like a wider GLOT or the same residual gate added to the baselines?”\color{red}{\text{“Could the authors examine parameter-matched baselines, like a wider GLOT or the same residual gate added to the baselines?”}}

We therefore followed your guidance and ran these controls, and also added experiments directly addressing your broader concern about the relevance of frozen representations today:

  • Parameter-matched controls: we added a gated layer-mix baseline with 19.9M parameters, compared with 20.21M for ToDA. ToDA remains higher in all 9 target-specific-review settings, with every paired interval above zero. We also widened GLOT to 23.1M parameters, more than ToDA; it remains at approximately 49–52% on LLaMA-3B and Mistral-7B, compared with 92–94% for ToDA.
  • The decision model you specifically discussed: inspired by your comment, we evaluated Jev 1.14 with 66,845 zero-shot calls. ToDA is 13–46 points higher on kinship, 24 points higher on aspect sentiment, 7–15 points higher on review counting, 1.3–2.4 points higher on aspect majority, and at the same level on target-specific reviews.
  • Retrieval and STS: we repeated MTEB with 3 training seeds across 44 categories. On the retrieval subset, ToDA improves over mean aggregation on all three backbones by 4.1–8.5 points, and on STS by 3.4–5.1 points. We also clarified the scope of the paper relative to dedicated end-to-end embedding models.

Following your review, we therefore added the two capacity controls you requested, a direct 66,845-call evaluation of the decision-model alternative you raised, and repeated retrieval evaluation across three backbones and three training seeds.


We believe these additions directly address the central concerns raised in your review and materially strengthen the paper. We hope you find our responses and the new results satisfactory, and that you will consider revising your score.

Thank you again for your time and for the constructive and actionable feedback.

Best regards,
The Authors

AuthorsOpenReview ↗

Final remarks by authors

Dear Reviewer 11Zx,

As the discussion comes to an end, and while we did not receive feedback from you on our responses of 16 November, we would like to conclude by summarizing how we addressed each of your points.

  • Relevance of frozen representations. You wrote:

    "Running a full model on text is so inexpensive that generating document embeddings with a frozen LM is no longer necessary for tasks like those contemplated in this paper."

    Our study measures which local information a frozen LM retains, at which depth, and which aggregation can recover it; classification provides the measurement, and Section 5.4 states this scope. Inspired by your comment, we evaluated Jev 1.14 with 66,845 zero-shot calls and compared it with task-trained ToDA over frozen Mistral-7B: aspect sentiment is 79.9% for ToDA versus 56.0%, and kinship is 99.0/96.3/96.8% versus 85.6/60.5/50.3% at two/three/four hops (Table 49). On target-specific reviews, Jev reaches 94.4/93.9/93.3% versus 94.1/93.6/94.1% for ToDA at K=3/5/7. These results measure the tasks under the stated supervision regimes; the paper compares trained aggregation with zero-shot decisions and does not rank deployment efficiency.

  • Retrieval and dedicated embedding models. You wrote:

    "isn't realistic to use a frozen model to generate the embeddings, since document embedding models are trained specifically for the retrieval task."

    Our embedding function is trained for retrieval: the aggregation module and output projection are trained contrastively on MS MARCO, with the backbone fixed (Appendix F.2). Repeated evaluation with three training seeds gives retrieval scores of 34.95/39.98/40.53 for ToDA versus 26.42/32.04/36.44 for mean aggregation on BERT/TinyLlama/Mistral-7B, and STS scores of 60.62/63.08/62.92 versus 55.98/58.03/59.55 (Tables 39/40). On Mistral-7B the overall 44-category average is 48.20 versus 48.66; the retrieval and STS results identify where depth-aware aggregation helps. Section 5.4 defines the comparison among readouts of a fixed backbone, without a claim of superiority over dedicated embedding models.

  • Capacity-matched baselines. You asked:

    "Could the authors examine parameter-matched baselines, like a wider GLOT or the same residual gate added to the baselines?"

    We ran both controls you proposed (Table 33). On Mistral-7B, gated layer mix + AdaPool has 19.9M parameters versus 20.21M for ToDA and reaches 91.21/91.69/91.35% at K=3/5/7, compared with 94.10/93.59/94.13% for ToDA; ToDA is higher in all nine target-review settings across the three decoders, with every paired 95% interval above zero. Widened final-layer GLOT has 7.9M or 23.1M parameters on Mistral-7B and remains at approximately 49–52% on LLaMA-3B/Mistral-7B, compared with 92–94% for ToDA. The same wide GLOT trains successfully on a validation-selected intermediate layer, reinforcing the distinction between access to depth and parameter count.

We thank you for your review: in our opinion, the work we did following your comments improved our paper, and we hope that you find our responses satisfactory and that you will consider revising your score.

Best regards,
The Authors

Reviewer_ijXUOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies how to pool the hidden states of a frozen LM into a single text vector when the label depends on composing local meaning (e.g., negation) and aggregating local judgments across a text (e.g., the majority opinion about one of two businesses). It makes two contributions.

  1. A suite of seven "composition and aggregation" tasks. Each task is built so that global cues (overall sentiment, length) do not reveal the label. A probe-gap criterion (Table 14) screens the tasks for being non-trivial under mean pooling.
  2. ToDA, a pooling method. For every token, ToDA computes multi-head attention over its own representations at all layers, with the token's final-layer state as the query. The fused token vectors then feed a GLOT-style cosine-threshold token graph, a GAT with jumping knowledge, and an attention readout.

ToDA beats mean/max/CLS pooling, AdaPool, GLOT and shared layer-mix baselines on nearly all proposed tasks across six backbones. The gaps are often very large: on target-specific reviews with 3B/7B decoders, ToDA reaches ~92–94% while final-layer baselines stay at ~50%. The paper also reports gains on GLUE/IMDB (against numbers published by GLOT) and on MTEB.

Critical strengths
  1. Careful task design. The target-specific review task balances the overall sentiment exactly, and the probe-gap filter (which removes plain review majority because mean pooling already solves it) is a good practice that more benchmark papers should adopt. The worked examples in App. B are clear.
  2. Large, consistent effects with seed-level uncertainty. Paired bootstrap intervals are reported for most comparisons, and the 50% → 92–94% jump on target-specific reviews is too large to be noise.
  3. Mechanistic evidence. The learned layer weights (Fig. 2, Fig. 3) concentrate near the layers where a linear probe peaks, and they differ across token categories (opinion words vs. business names). This partly supports the "token-specific depth" motivation.
  4. Clear writing, and honest reporting of limitations of the selection procedure (e.g., the GLUE selection protocol is disclosed in App. F).
Critical weaknesses

W1. Tuning budgets differ across methods, in ToDA's favor. App. C.1 states that ToDA searched 32 configurations plus a separate temperature diagnostic. Layer mix + AdaPool searched 8. "Every other method has a tuned candidate with learning rate 5×10⁻⁵", i.e., two candidates in total. The paper itself shows how sensitive these tasks are to tuning: the tuned configurations raise Layer mix + AdaPool by 26 points on Mistral-7B, and changing ToDA's layer-weighting temperature from 0.1 to 1.0 moves its accuracy from 65.65% to 87.58% (§5.3). With such strong tuning sensitivity, a 2-vs-32 candidate imbalance can produce differences of the size reported. GLOT, the strongest competitor, should receive the same search budget, including its relation threshold, width and depth.

W2. Other confounds favor ToDA.

  • Batch size (App. C.2): on LLaMA-3B/Mistral-7B, ToDA, GLOT and mean pooling use batch 64/32, while the other methods use 16/8, "trained on GPUs with less memory."
  • MTEB (App. F.2): only ToDA's search includes hard negatives. ToDA trains for 30 epochs on BERT versus 12 for the others, and its BERT configuration comes from a separate earlier 40-trial search with larger batch sizes and learning rates up to 2×10⁻³. Each MTEB entry is also a single seed. On Mistral-7B, ToDA's margin over mean pooling is 0.08 points (48.61 vs. 48.53), which is not evidence of an improvement.
  • GLUE/IMDB (App. F.1): ToDA's numbers are the best run of Bayesian searches, both selected and reported on the official validation split. The search space also covers optimizer, weight decay, batch size and GNN architecture. These numbers are compared with GLOT's published numbers from a different procedure. The paper discloses this, but the headline claim of "+2.63 to +8.46 points over GLOT" (Table 5) is not a controlled comparison and should not be presented as one.
  • Capacity: ToDA has ~2× GLOT's parameters on Mistral-7B (19.2M vs. 8.9M), mostly from the D×D residual gate (App. D). A capacity-matched GLOT is needed.

W3. Key ablations and baselines are missing.

  • Per-token layer weights without the graph. Table 4 never trains "all layers + per-token weights + attention readout (no graph)." Without this row, we cannot tell whether the graph stage is still needed once per-token depth is available. This is central to the two-stage design claim.
  • Best single intermediate layer. The probe peaks at layer 2 (TinyLlama, composed polarity) and at layers 14–15 (target-specific reviews). The obvious baseline is AdaPool or GLOT applied to the best layer, chosen on validation. The shared layer-mix baselines should in principle recover this, but they underperform badly (sometimes below final-layer AdaPool), which suggests an optimization or tuning problem rather than a representational limit (see W1).
  • A small transformer over the frozen token states (1–2 layers, from the last or best layer). This is the standard learned aggregator for relational token interactions, and it can express "attach each review to the preceding business name" directly. The cosine-threshold graph of GLOT is a fairly restricted relational module, so beating it says little about relational aggregation in general.
  • Prompt-based last-token pooling for decoders (e.g., placing the query at the end, or PromptEOL-style templates). In target-specific reviews the query comes first. The near-chance last-token results (Table 3) may reflect query position rather than an inherent limitation of final-layer pooling.

W4. The scope of the core claim is narrower than stated. On target-specific reviews, the task with the most striking gains, ToDA stays at chance on BERT, RoBERTa and SmolLM2 (Table 17: ≤57.41%). The paper acknowledges this ("an open challenge for these backbones"). It nonetheless means the main result is "intermediate layers of ≥1B decoders contain the target–review association, and ToDA can read it out", not a general property of token-specific depth aggregation. The per-token weight analysis (Fig. 3c) likewise shows uniform weights on BERT. Please temper the abstract ("ToDA reaches high accuracy" where existing methods are at chance) accordingly.

W5. Novelty relative to prior layer aggregation. Token-dependent (dynamic) aggregation across layers has been studied before. Examples include dynamic layer aggregation for NMT (Dou et al., EMNLP 2018; AAAI 2019) and recent dynamic dense-connection architectures such as MUDDFormer (ICML 2025), where the cross-layer weights depend on the hidden state at each position. DenseFormer and Hyper-Connections are also relevant. The observation that intermediate layers of LLMs give better frozen embeddings has been analyzed systematically (e.g., Skean et al., "Layer by Layer", ICML 2025). The paper's novelty is applying per-token layer attention on top of a frozen LM, before GLOT. That is a reasonable but incremental combination, and this related work should be discussed.

W6. Statistical caveats. (a) The controlled task has no test split. Every method reports its highest validation accuracy over epochs on 200 examples, and the relation threshold is also selected on the same 16-seed validation mean. (b) In the natural-text tasks, the seeds vary the data but the parameter initialization is fixed (App. C.1), so the intervals understate training variability. (c) The bootstrap intervals condition on the selected configurations, so they do not account for selection under the unequal search budgets of W1.

W7. Cost. ToDA reads all layers. A training batch takes 4.6× the time and ~10× the memory of GLOT (61.2 ms / 4.15 GB vs. 13.4 ms / 0.42 GB). Caching all layers of a 7B model is also expensive for retrieval-style use. "The number of trainable parameters does not grow with M" is true but beside the point. The relevant costs are memory and FLOPs, which grow linearly in M.

W8. Minor.

  • In Table 4, all BERT methods, including ToDA (49.18%), are at chance on target-specific reviews. That column carries no information about the components and should be flagged as such.
  • CLUTRR v1 is templated and known to admit surface shortcuts. Training and testing at the same hop count on random 80/10/10 splits does not test systematic generalization, so the kinship numbers (92–99%) should not be read as evidence of multi-hop composition.
  • §1: "ToDA also competitive on …" is missing a verb.

Justification. The task suite is a useful contribution, and some effects (target-specific reviews on large decoders) are large enough that I believe a real phenomenon is behind them. However, the comparisons are not controlled: ToDA had a much larger search budget, extra training options (batch size, hard negatives, epochs) and more parameters. The GLUE comparison uses validation-selected best runs against published numbers. The ablation that would justify the two-stage design (per-token depth without the graph) is missing, and the strongest natural baselines (best single layer, small transformer aggregator, prompt-based readout) are absent. If the authors show that the gains survive equal-budget tuning and these baselines, I would raise my score to 6–8.

Questions for authors
  1. Can you re-run GLOT and Layer mix + AdaPool (and ideally AdaPool) with the same search space and number of candidates as ToDA, including temperature-like knobs where applicable, on target-specific reviews and composed polarity?
  2. What is the accuracy of "per-token layer attention + AdaPool readout, no graph"?
  3. What is the accuracy of AdaPool and GLOT on the best single layer, selected on validation?
  4. How does a 1–2 layer transformer aggregator over frozen token states, with matched parameters, compare?
  5. Do the target-specific review results change if the query is placed after the reviews, or with a prompt-based last-token readout?
  6. For MTEB, can you report results with identical training budgets (epochs, hard negatives) and ≥3 seeds?
  7. Why do shared layer-mix baselines sometimes fall below final-layer AdaPool (e.g., SmolLM2, Table 2), given that they include the final layer? Is this an initialization or temperature issue?
13 comments
AuthorsOpenReview ↗

Authors Responses (Part 1)

We sincerely thank you for the detailed and constructive review, and for stating what would change your assessment: that the improvements "survive equal-budget tuning and these baselines". We are encouraged that you find the task suite "a useful contribution", that you see "a real phenomenon" behind the effects on target-specific reviews with the large decoders, and that you consider the probe-gap filter "a good practice that more benchmark papers should adopt". We refer to your points as W1, W2, ... and Q1, Q2, ... in the order of your review, with letters for their parts. We hope you find our responses satisfactory, and that you will consider revising your score.


In short: your review asks whether the improvements of ToDA remain under equal tuning budgets, matched capacity and the natural baselines. (1) GLOT and the two layer-mix baselines now have searches at least as large as that of ToDA (96, 32 and 32 configurations), and AdaPool has its full grid; ToDA remains above every baseline, submitted or re-tuned, in all nine settings (W1, Q1). (2) The ablation and the baselines that you name are added: ToDA without the graph, AdaPool and GLOT on a validation-selected layer, a transformer on the final and on a selected layer, and the query-after and prompt formats (W3, Q2 to Q5). (3) Capacity, batch size, training seeds, the test split and the initialization are addressed with new runs (W2, W6, Q6). (4) The abstract, Section 2 and the caption of Table 4 are revised following your comments (W4, W5, W8). Each table below gives the rows of the tables of the general response that the answer uses; Table 38 is in this reply.

AuthorsOpenReview ↗

Authors Responses (Part 2)

Regarding equal search budgets (W1, Q1). We appreciate this request, which makes the comparison independent of the size of the search. Following your specification, we searched the baselines on target-specific reviews (K = 3; TinyLlama and Mistral-7B; three seeds; selection on validation; the 20 epochs of Appendix C.1 in every search run): GLOT over 96 configurations (learning rate {5e-5, 1e-4, 2e-4, 5e-4} × jumping knowledge {max, cat} × width {128, 256} × depth {4, 6} × relation threshold {0.3, 0.5, 0.9}); layer mix + AdaPool over 32, with the layer-weighting temperature {0.1, 1.0} and the initialization {zeros, upper bias}; the gated layer mix + AdaPool, which is the depth module of ToDA with shared weights and an AdaPool readout, over 32; and AdaPool over its full grid of 8 (learning rate and scorer width). The winners were rerun with eight data seeds on all nine settings under the protocol of Appendix C.1; LLaMA-3B uses the winner of Mistral-7B, as in Appendix C.1. Table 30 reports the submitted and the re-tuned configuration of every baseline, because a winner selected on three seeds is not always above the submitted configuration with eight seeds (test accuracy (%) at K = 3 / 5 / 7):

MethodTinyLlamaLLaMA-3BMistral-7B
AdaPool, Table 1750.76 / 50.44 / 49.6650.76 / 49.46 / 49.6551.73 / 51.21 / 50.04
AdaPool, winner of 849.65 / 50.78 / 50.1449.91 / 49.53 / 49.3650.63 / 51.03 / 50.11
GLOT, Table 1766.71 / 65.19 / 81.8556.90 / 52.76 / 50.6050.51 / 51.11 / 53.59
GLOT, winner of 9664.79 / 74.35 / 78.3849.73 / 49.86 / 49.6151.04 / 64.55 / 68.43
Layer mix + AdaPool, Table 1756.45 / 59.31 / 62.7872.40 / 73.90 / 75.5879.35 / 81.83 / 83.54
Layer mix + AdaPool, winner of 3251.85 / 60.43 / 56.7074.10 / 75.54 / 84.8572.79 / 79.19 / 87.95
Gated layer mix + AdaPool, earlier configuration54.16 / 60.58 / 59.9185.98 / 88.04 / 87.8191.21 / 91.69 / 91.35
Gated layer mix + AdaPool, winner of 3258.49 / 68.79 / 62.8887.45 / 87.33 / 87.2090.24 / 89.98 / 91.05
ToDA (32 configurations)87.48 / 87.83 / 89.5493.04 / 92.04 / 92.0994.10 / 93.59 / 94.13

ToDA is above both configurations of every baseline of Table 30 in all nine settings, with the paired 95% interval above zero in every comparison. On the two larger decoders the highest baseline is the gated layer mix + AdaPool, which is 1.9 points or more below ToDA on Mistral-7B and 4.0 or more on LLaMA-3B; on TinyLlama it is GLOT, 7.7 points or more below ToDA. With temperature and initialization in its search, the layer mix + AdaPool improves from 83.54 to 87.95 on Mistral-7B and from 75.58 to 84.85 on LLaMA-3B at K = 7, as you anticipated, and it remains 6.2 and 7.2 points below ToDA. The change from 65.65% to 87.58% that you cite is from the 40-epoch runs of Appendix E on TinyLlama; in the reported results, validation selects among three candidates of ToDA.

In Table 30, LLaMA-3B uses the winners of Mistral-7B, as in Appendix C.1. We therefore also ran the same searches on LLaMA-3B itself and reran the winners with eight seeds (Table 31; test accuracy (%) at K = 3 / 5 / 7):

Method, searched on LLaMA-3B itselfLLaMA-3B
AdaPool50.33 / 50.59 / 49.99
GLOT50.74 / 50.94 / 49.91
Layer mix + AdaPool79.63 / 76.19 / 85.60
Gated layer mix + AdaPool90.50 / 86.69 / 88.33
ToDA93.04 / 92.04 / 92.09

ToDA is above each of them, by 2.5 points or more, with the paired interval above zero in all twelve comparisons. On composed polarity, the second task that you name, we searched the baselines with the same sizes on TinyLlama (u = 9, three seeds, selection on validation) and reran the winners with 16 seeds on the held-out test split of Table 42, where ToDA keeps the configuration of Table 42 (Table 32; test accuracy (%) at u = 3 / 5 / 9):

MethodTinyLlama
AdaPool, winner of 876.5 / 71.9 / 68.3
GLOT, winner of 9677.5 / 71.7 / 67.0
Layer mix + AdaPool, winner of 3277.8 / 72.0 / 68.6
ToDA, Table 4298.5 / 98.6 / 95.7

ToDA is above every baseline, searched or as in Table 42, in all nine comparisons, by 14 points or more. These results are reported in Appendix H.1.

AuthorsOpenReview ↗

Authors Responses (Part 3)

Regarding the batch size (W2(a)). Thank you for pointing this out. The difference regards the two studies of Appendix C.2 on LLaMA-3B and Mistral-7B, where ToDA, GLOT and mean aggregation used a larger batch than the remaining methods; on every task of Appendix C.1, all methods on a backbone share the frozen forward pass, the batch size and the batch order. Following your comment, we reran both studies with one batch size per backbone for every method. Composed polarity (Table 42; batch size 16 on LLaMA-3B and 8 on Mistral-7B, the batch sizes of the remaining methods; held-out test split; 16 seeds, eight for ToDA on LLaMA-3B at u = 5; test accuracy (%) at u = 3 / 5 / 9):

MethodLLaMA-3BMistral-7B
Mean aggregation74.4 / 68.9 / 65.978.5 / 71.6 / 68.6
Max aggregation69.9 / 64.8 / 60.773.8 / 67.2 / 63.8
Last token75.8 / 70.1 / 66.777.3 / 71.1 / 66.4
AdaPool76.8 / 71.4 / 68.184.9 / 74.7 / 71.9
GLOT84.0 / 73.9 / 68.191.2 / 77.3 / 71.0
Layer mix + AdaPool79.5 / 74.3 / 70.378.8 / 71.8 / 70.2
ToDA99.1 / 98.3 / 94.399.1 / 97.5 / 94.3

SST-2 paragraphs (Table 44; batch size 32 on LLaMA-3B and 16 on Mistral-7B; 16 data draws; test accuracy (%) at K = 3 / 5 / 9; mean aggregation, GLOT and ToDA are rerun at these batch sizes, and the four remaining methods, which used them in the paper, are repeated from Table 16):

MethodLLaMA-3BMistral-7B
Max aggregation, Table 1669.23 / 66.95 / 63.1567.64 / 66.43 / 63.14
Last token, Table 1670.81 / 67.00 / 64.6468.14 / 64.94 / 62.04
AdaPool, Table 1671.91 / 69.74 / 68.6573.35 / 72.20 / 68.48
Layer mix + AdaPool, Table 1673.89 / 70.81 / 68.8374.19 / 70.89 / 70.24
Mean aggregation, rerun74.93 / 73.44 / 72.8974.14 / 73.53 / 71.85
GLOT, rerun73.70 / 70.91 / 69.0674.29 / 70.64 / 69.38
ToDA, rerun77.10 / 75.78 / 75.4577.76 / 76.63 / 75.41

At the common batch size, ToDA is above every method in both studies, and on SST-2 paragraphs the paired interval is above zero in all twelve comparisons with GLOT and mean aggregation. These results are reported in Appendix J.3.

Comments on the paper

6 comments
AuthorsOpenReview ↗

General Response (Part 1)

We thank the reviewers for their careful reading and for their constructive feedback. We are encouraged that the reviewers found the task suite "a useful contribution" (ijXU) and "useful, especially target-specific review aggregation" (NujP), with a probe-gap filter that is "a good practice that more benchmark papers should adopt" (ijXU); that they noted "strong and consistent improvements over baselines on the proposed tasks" (11Zx) and a difference on target-specific reviews that is "too large to be noise" (ijXU); and that they found the analysis "careful to examine tasks that require aggregating local meaning" (11Zx).

The reviews share one question: whether the improvements come from the design of ToDA or from capacity, search budget and protocol, and which component produces them. Following your comments, we added the experiments summarized below. Unless stated otherwise, they use the protocol of Appendix C.1 (eight data seeds, the epoch selected on validation, test accuracy, paired 95% bootstrap intervals), and every table states the number of seeds where it differs. The main results regard target-specific reviews in the query-first format of the paper, with TinyLlama, LLaMA-3B and Mistral-7B, the setting to which all three reviews point.

  1. Capacity (ijXU, 11Zx, NujP). Both controls proposed by 11Zx are run. The gated layer mix + AdaPool, which is the depth module of ToDA with shared weights and an AdaPool readout (19.9M trainable parameters on Mistral-7B; ToDA 20.21M), reaches 91.21 / 91.69 / 91.35 on Mistral-7B at K = 3/5/7, compared with 94.10 / 93.59 / 94.13 for ToDA, and is below ToDA in all nine settings, with every paired interval above zero. GLOT widened to 7.9M and 23.1M parameters remains at 49 to 52% on LLaMA-3B and Mistral-7B (Table 33).

  2. Search budget (ijXU, NujP). GLOT (96 configurations, including threshold, width and depth), layer mix + AdaPool (32, including temperature and initialization) and the gated layer mix + AdaPool (32) were searched and rerun with eight seeds; AdaPool has its full grid of 8. ToDA is above the submitted and the re-tuned configuration of every baseline in all nine settings, by 1.9 points or more on Mistral-7B, 2.5 or more on LLaMA-3B and 7.7 or more on TinyLlama, and the paired intervals are above zero (Tables 30 and 31).

  3. The two stages of ToDA (ijXU, NujP). In the control that NujP specifies, shared weights replace the token-specific weights inside the module of ToDA: ToDA is higher in all nine settings, by 0.75 to 3.06 points. Without the graph, the row that ijXU requests, accuracy is 2.16 to 4.04 points lower on LLaMA-3B and Mistral-7B, and on TinyLlama the variant rises above random guessing on 1, 2 and 4 of its eight seeds at K = 3, 5 and 7, whereas ToDA does on all eight; every mean is over all eight seeds (Table 35).

  4. Single-layer, transformer and prompt baselines (ijXU, NujP). A transformer on the final layer remains at 49 to 59%. On an intermediate layer selected on validation among three shortlisted layers, AdaPool is 2.1 to 9.0 points below ToDA in all nine settings, and GLOT and a transformer reach 84 to 93%, with ToDA at their level or above. ToDA reaches this accuracy without a probe and without a search over layers: it learns its weights over all layers during training, and every token selects its own depth (Table 33). In the added query formats the last token leaves random guessing, and ToDA remains above it in every cell (Table 38, in our reply to ijXU).

  5. Protocols (ijXU, NujP). MTEB with three training seeds and identical final training: ToDA is above mean aggregation by 2.29 points on BERT and by 1.84 on TinyLlama, in every seed; on Mistral-7B the averages are 48.20 ± 0.35 for ToDA and 48.66 ± 0.15 for mean aggregation, with ToDA higher on retrieval (40.53 and 36.44) and on STS (62.92 and 59.55) (Tables 39 and 40). Composed polarity on a held-out test split: ToDA is above every method of Table 2 in all eighteen settings, by 5 points or more (Table 42). Intervals that repeat the configuration selection inside the bootstrap: 245 of 251 remain above zero and none changes sign. GLUE and IMDB with a development split: ToDA has the highest average over the ten tasks on all six backbones and is above GLOT on 50 of the 60 task and backbone pairs; on the five backbones with all six methods, ToDA has the highest average of the six (Table 41).

The complete tables follow in the next comments, with the list of the added material.

AuthorsOpenReview ↗

General Response (Part 2)

Table 33. Controls on target-specific reviews, query-first format of the paper. Test accuracy (%) at K = 3 / 5 / 7; eight data seeds. Parameter counts are for Mistral-7B. The gated layer mix + AdaPool is the depth module of ToDA with shared weights and an AdaPool readout. The selected layer is chosen per seed on validation among three layers shortlisted by a probe. The transformer rows list the parameters at widths 256 and 768; validation selects the width.

Layers readMethodParamsTinyLlamaLLaMA-3BMistral-7B
Final layerGLOT (Table 17)1.28M66.71 / 65.19 / 81.8556.90 / 52.76 / 50.6050.51 / 51.11 / 53.59
Final layerGLOT, width 10247.9M49.79 / 50.23 / 54.6850.23 / 49.20 / 50.1649.83 / 51.91 / 49.53
Final layerGLOT, width 204823.1M49.89 / 50.28 / 54.3049.93 / 49.46 / 49.6850.96 / 51.45 / 51.10
Final layerTransformer, two layers2.70M / 17.52M50.98 / 49.99 / 50.7451.11 / 49.38 / 49.5350.68 / 52.81 / 58.54
One selected layerAdaPool1.06M81.13 / 85.74 / 85.8884.04 / 85.86 / 86.7186.04 / 86.63 / 87.78
One selected layerGLOT1.28M84.05 / 87.90 / 89.8591.83 / 90.79 / 91.3593.26 / 92.38 / 93.00
One selected layerGLOT, width 204823.1M86.04 / 87.75 / 88.5891.03 / 90.00 / 89.8492.45 / 92.23 / 92.44
One selected layerTransformer, two layers2.70M / 17.52M86.43 / 87.91 / 88.3691.30 / 91.23 / 91.1392.61 / 92.34 / 92.65
All layers, shared weightsLayer mix + AdaPool (Table 17)1.06M56.45 / 59.31 / 62.7872.40 / 73.90 / 75.5879.35 / 81.83 / 83.54
All layers, shared weightsGated layer mix + AdaPool19.9M54.16 / 60.58 / 59.9185.98 / 88.04 / 87.8191.21 / 91.69 / 91.35
All layers, shared weightsToDA with shared weights20.17M84.48 / 87.08 / 88.3590.74 / 88.98 / 89.7192.81 / 92.31 / 93.34
All layers, token-specific weightsToDA without the graph19.98M54.66 / 59.50 / 67.3990.10 / 89.74 / 89.3990.06 / 91.43 / 91.78
All layers, token-specific weightsToDA20.21M87.48 / 87.83 / 89.5493.04 / 92.04 / 92.0994.10 / 93.59 / 94.13

Table 35. The two stages of ToDA: paired comparisons on target-specific reviews. Test accuracy (%) on the same eight data seeds. "ToDA with shared weights": the token-specific layer weights are replaced by weights shared by all tokens and inputs, and the rest of ToDA is kept. "ToDA without the graph": the token-specific weights are kept and the graph is replaced by the AdaPool readout. ToDA is above ToDA with shared weights and above ToDA without the graph in all nine settings. On TinyLlama the variant without the graph rises above random guessing on 1, 2 and 4 of its eight seeds at K = 3, 5 and 7, whereas ToDA does on all eight; every mean is over all eight seeds.

BackboneKToDA with shared weightsToDA without the graphToDA
TinyLlama384.4854.6687.48
TinyLlama587.0859.5087.83
TinyLlama788.3567.3989.54
LLaMA-3B390.7490.1093.04
LLaMA-3B588.9889.7492.04
LLaMA-3B789.7189.3992.09
Mistral-7B392.8190.0694.10
Mistral-7B592.3191.4393.59
Mistral-7B793.3491.7894.13
AuthorsOpenReview ↗

General Response (Part 3)

Table 30. Search budget: the submitted and the re-tuned configuration of every baseline. Target-specific reviews, test accuracy (%) at K = 3 / 5 / 7, eight data seeds. The searches use K = 3 with TinyLlama and Mistral-7B (three seeds, selection on validation); the winners are rerun on all nine settings, and LLaMA-3B uses the winner of Mistral-7B, as in Appendix C.1. A winner selected on three seeds is not always above the submitted configuration with eight seeds, and both are shown. The same searches were also run on LLaMA-3B itself (winners rerun with eight seeds): gated layer mix + AdaPool 90.50 / 86.69 / 88.33, layer mix + AdaPool 79.63 / 76.19 / 85.60, GLOT 50.74 / 50.94 / 49.91, AdaPool 50.33 / 50.59 / 49.99.

MethodTinyLlamaLLaMA-3BMistral-7B
AdaPool, Table 1750.76 / 50.44 / 49.6650.76 / 49.46 / 49.6551.73 / 51.21 / 50.04
AdaPool, winner of 849.65 / 50.78 / 50.1449.91 / 49.53 / 49.3650.63 / 51.03 / 50.11
GLOT, Table 1766.71 / 65.19 / 81.8556.90 / 52.76 / 50.6050.51 / 51.11 / 53.59
GLOT, winner of 9664.79 / 74.35 / 78.3849.73 / 49.86 / 49.6151.04 / 64.55 / 68.43
Layer mix + AdaPool, Table 1756.45 / 59.31 / 62.7872.40 / 73.90 / 75.5879.35 / 81.83 / 83.54
Layer mix + AdaPool, winner of 3251.85 / 60.43 / 56.7074.10 / 75.54 / 84.8572.79 / 79.19 / 87.95
Gated layer mix + AdaPool, earlier configuration54.16 / 60.58 / 59.9185.98 / 88.04 / 87.8191.21 / 91.69 / 91.35
Gated layer mix + AdaPool, winner of 3258.49 / 68.79 / 62.8887.45 / 87.33 / 87.2090.24 / 89.98 / 91.05
ToDA (32 configurations)87.48 / 87.83 / 89.5493.04 / 92.04 / 92.0994.10 / 93.59 / 94.13

Table 39. MTEB with repeated final training. Average over the 44 evaluation categories of Table 5 (×100), mean ± standard deviation over three training seeds. Identical final training for every method (100,000 pairs, six epochs, hard negatives off); the six development tasks are excluded. Table 5 of the paper is unchanged.

BackboneCLS / last tokenMax aggregationMean aggregationGLOTToDA
BERT37.89 ± 0.1339.58 ± 0.0342.18 ± 0.0543.24 ± 0.1644.47 ± 0.30
TinyLlama35.97 ± 0.0643.14 ± 0.0846.74 ± 0.0645.22 ± 0.1648.58 ± 0.29
Mistral-7B39.63 ± 0.1343.15 ± 0.1048.66 ± 0.1546.77 ± 0.5148.20 ± 0.35

Table 40. Mean aggregation and ToDA by task family (×100, mean over the three training seeds; number of categories in parentheses; same runs).

Family (categories)Mean aggregation, BERTToDA, BERTMean aggregation, TinyLlamaToDA, TinyLlamaMean aggregation, Mistral-7BToDA, Mistral-7B
Retrieval (8)26.4234.9532.0439.9836.4440.53
STS (9)55.9860.6258.0363.0859.5562.92
Reranking (3)50.0950.6352.9754.5254.0054.30
Summarization (1)30.5129.0730.0930.7827.1129.50
Classification (11)43.3243.9550.0149.4350.9748.05
Pair classification (2)67.1666.9764.2162.2565.0661.15
Clustering (10)34.9133.3141.0538.7443.3838.72
All 4442.1844.4746.7448.5848.6648.20

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.