acceptodds
Under review as a conference paper at ICLR 2027

Metadata-Aligned Decoding for Accuracy Gains

Abstract

While metadata is commonly used to curate datasets, its potential as a direct decoding signal remains under-explored. We propose LIME (**Li**nguistic **M**etadata **E**mbeddings), a method that enriches token embeddings with linguistic metadata capturing syntax, semantics, and context. LIME substantially improves decoding accuracy across model sizes from 500M to 2B, while introducing only 0.01% additional parameters at negligible compute overhead. Beyond accuracy, LIME improves tokenization and generative task performance. These benefits persist across model scales, and the decoding accuracy gains extend to five typologically diverse languages, where language modeling improvements scale predictably with tokenizer fertility. The gain stems from metadata that anticipates the token to be generated. Supplied directly, as in our guided variant LIME-G, it raises decoding accuracy by over 40% and outperforms constrained decoding in reasoning and arithmetic. Overall, we establish metadata as a powerful interface for controllable generation, demonstrating that inference-time steering can unlock dormant capabilities in reasoning and arithmetic.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Language Models, Metadata Embeddings, Pre-Training, Tokenization, Controllable Decoding, Generation, Inference-Time Steering, Linguistic

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_AtVDOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies linguistic annotations as token-aligned inputs to decoder language models. LIME adds POS/NER embeddings to the current token embedding; LIME-G shifts the annotations so the current input carries metadata about the token to be predicted; LIME-A predicts generated-token annotations with auxiliary heads. Models are pretrained at 500M–2B scale on 302B tokens, with additional 7B, multilingual, and annotation-context experiments. The reported token-accuracy gain is about one percentage point for LIME and about 20 points for reference-guided LIME-G. It transforms the metadata into a powerful controllable generation interface, demonstrating that the shift in reasoning time can unlock dormant reasoning and arithmetic capabilities.

Critical strengths

S1: The paper provides a clear analysis of where the gains come from. Comparisons across annotation contexts show that much of the benefit comes from information about subsequent tokens. This helps clarify why the method works and under what conditions.

S2: The method shows useful gains on some generation tasks. Results on reasoning and addition suggest that class guidance can improve answer correctness and merits further investigation.

Critical weaknesses

W1: Evidence for effectiveness in practical generation remains limited. The main accuracy and data-efficiency results rely on metadata derived from full documents or reference answers. Without this information, gains are smaller and less consistent, limiting the scope of the conclusions.

W2: Key ablations are needed to isolate the source of improvement. Existing comparisons change both training and inference, leaving unclear how much of the gain comes from training with additional class information versus embedding-level guidance at inference.

Questions for authors

Q1. How much of the reported improvement remains when metadata is obtained only from the prompt and previously generated tokens? A comparison separating reference-derived, task-provided, and self-predicted metadata would help clarify the method’s practical benefits.

Q2. Can you isolate the effects of metadata-conditioned training and inference-time guidance? For example, evaluating the same checkpoint with correct, missing, and perturbed guidance would help determine which component drives the improvement.

5 comments

Thank you for the review. We ran the two comparisons you proposed in Q1 and Q2, which both confirm the paper's results. We respond to the weaknesses and questions in order below. Table and section numbers in this response are those of the submitted PDF.


W1. Evidence for practical generation

The paper uses the full document and the reference for one purpose, to measure what the class of the token to be generated is worth, which is the analysis you note in S1. With no annotator seeing the answer, metadata improves generation in both settings the paper proposes: a class determined by the task (LIME-G) and a class predicted by the model (LIME-A). In the paper, LIME-G guided by the digit class alone more than doubles accuracy on ARI-ADD (Sec. 4.5). LIME and LIME-A improve FLenQA with no supplied class, LIME-A even without any annotator after the prompt (Tab. 3). Under Q1 this response adds the same test with one class for all FLenQA items, where LIME-G gains +13 to +26pp across all context lengths, as well as an additional public benchmark with the class taken from the prompt. For clarity, the revised PDF adds an overview of the model variants (Sec. 3) and moves Tab. 20 from the appendix into the main text, where it states the source of the metadata for every result.


Q1. Gains without reference-derived metadata

In decoding accuracy the gain comes with the class of the token to be generated (Tab. 21), which is why the paper supplies exactly this class in generation.

For generation, the table below compares the three sources of the class that your question names. All rows are re-run in one evaluation of the 2B models on the same items, with every model generating token by token, so that the rows are directly comparable. The values are therefore not identical to Tab. 3. The row with one class for all items is new and set in bold.

Source of the classFLenQA 250500100020003000ARI-ADD
none (Base)40.048.834.841.530.021.8
text so far (LIME)50.065.347.345.029.525.0
model's own heads (LIME-A)60.065.848.846.330.523.2
task (LIME-G, one class for all items)62.062.061.355.056.350.2
reference (LIME-G, class of each item)78.074.561.551.036.356.6

The table shows that a class given by the task alone, with no reference, already brings a large gain. With one class for all items, LIME-G improves over Base at every FLenQA context length and stays between 55.0 and 62.0 from 250 to 3000 tokens, while Base falls to 30.0. On ARI-ADD the digit class alone more than doubles accuracy. With no supplied class at all, LIME and LIME-A still gain up to +20pp on FLenQA. LIME-A is level with or above LIME at every context length, which shows that the model's own heads can replace the tagger on generated text.

To test the task-provided class beyond the paper's two tasks, we evaluated LIME-G on BIG-bench QA-Wikidata (Srivastava et al., 2023), a public benchmark whose prompts are templates such as "The country of X is". LIME-G takes its class exclusively from the template. On the items where the template fixes the class, it gains +9.2pp at 2B and +11.1pp at 500M. Over all items the gains are +2.8pp and +7.8pp.

QA-Wikidata2B Base2B LIME-G500M Base500M LIME-G
all items62.565.344.151.9
items where the template fixes the class (61%)60.970.143.554.6

W2. Isolating training from inference-time guidance

We ran the evaluation you proposed and report it under Q2. The paper already contains this test for the switch model, a 500M model with an optional class input (Sec. 4.6, App. A.13). It gains +2.27pp when the class is supplied at inference and stays within 0.3pp of Base without it (Tab. 27).


Q2. One checkpoint with correct, missing and perturbed guidance

To separate what training contributes from what the class at inference contributes, we evaluated the LIME-G 2B checkpoint on the same prompts and changed only the class given at the generated positions. "Missing" is the placeholder class that LIME-G receives in training at the end of a sequence, where no token follows. "Wrong" is a word class on ARI-ADD and a number class on FLenQA.

Class given to LIME-GARI-ADDFLenQA 250 / 500 / 1000 / 2000 / 3000
correct56.678.0 / 74.5 / 61.5 / 51.0 / 36.3
missing4.00.0 / 0.0 / 0.0 / 0.0 / 0.0
wrong0.00.0 / 0.0 / 1.0 / 0.0 / 0.0

The class given at inference drives the improvement, provided the model was pre-trained with shifted metadata. With the correct class the checkpoint reaches 56.6 on ARI-ADD and up to 78.0 on FLenQA, whereas with a missing or wrong class it gives almost no correct answer. This pre-training is required to unlock the gain, because it teaches the model to follow metadata classes (Sec. 5). The class then decides the output, even when it is wrong: with the number class, the model starts its FLenQA answer with a number on 95% to 100% of the items. This is the steering interface the paper establishes: guidance acts inside the model, at the input embedding, while a constraint can only filter the output. The "correct" row is the "reference" row of the table under Q1.

LIME-G follows its class by design, since it always receives one (Eq. 2). For use without a supplied class the paper offers LIME and LIME-A, while the switch model shows that one model can do both (Tab. 27).


Ethics flag

The review flags the paper for ethics review, but we could not find a specific concern in it. The Ethics Statement covers the annotator's weaknesses on non-standard text, dialects and code-mixed input. If you had a concern beyond this in mind, please tell us and we will address it.


Both evaluations you proposed are now in the paper, where they support its central result: a decoder trained with shifted metadata follows the class it is given, which a task can supply at inference with no reference. LIME-A shows in addition that a model can annotate its own output.

On this basis, we kindly ask you to reconsider your score. We are glad to answer any further question.


Changes in the revised PDF

  • Sec. 3: the overview of the variants.
  • Sec. 4: Tab. 20 moves from the appendix into the main text.
  • Sec. 4.2: the text states the annotation regime of its results.
  • App. A.7: the table by source of the class with the new row for one class on all FLenQA items, the same-checkpoint rows (correct, missing, wrong) and the QA-Wikidata result.

Thank you again for your review. We are glad that you see "a clear analysis of where the gains come from" and useful gains on generation tasks. Your summary describes the method as "a powerful controllable generation interface", which the two evaluations you proposed further confirm. Both evaluations are now included in the revised paper, with all changes marked in blue. The table numbers below now refer to the revised PDF.

  • Gains without reference-derived metadata. The comparison you proposed in Q1 is in App. A.7 (Tab. 17). With one class for all items, given by the task alone, LIME-G gains +13 to +26pp over Base at every FLenQA context length and more than doubles accuracy on ARI-ADD.
  • A public benchmark. On QA-Wikidata, LIME-G takes its class from the prompt alone and gains +9.2pp at 2B and +11.1pp at 500M on the items where the prompt fixes the class (Tab. 19).
  • Training and guidance at inference. The evaluation you proposed in Q2 is in Tab. 18. With the correct class, one checkpoint reaches up to 78.0 on FLenQA. With a missing or wrong class, almost no answers are correct, showing that the class given at inference drives the result.
  • Clarity. An overview of the model variants has been added (Tab. 2), and the main text now specifies the source of the metadata for each experiment (Tab. 5).

As the discussion period is coming to an end, we would like to ask whether these additions resolve your concerns. If you have any remaining questions about the soundness of the claims or about the ethics flag, please let us know and we will respond right away.

If your concerns are addressed, we would kindly ask you to update your score.

Best regards,

The Authors

Reviewer_iBcAOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Authors use language metadata to enrich token encoding for LLMs, making it more morphological. Authors test it on variety of model sizes up to 7B, , and up to ~300B token budget on smaller models, and show improvement in decoding and generation accuracy.

Critical strengths

Approach to adding morphological language-dependent information seems sound. For smaller models, up to 2B, authors test on up to 300B tokens, way past Chinchilla optimum, which somewhat addresses my initial concern of whether the gains hold for long training horizon, as models deployed in practice are usually trained on very many tokens. 7B model is only trained on 50B tokens, which is less informative, but more than I'd expect of academic paper given budget constraints.

Critical weaknesses

I see None, though I did not verify reproduceability.

Questions for authors

I could not quite disentangle which tests are entirely causal, and which might have a leak -- i.e. if corpus is used to infer/annotate suffixes, etc.., and then model is tested on the same corpus, the results might be biased.

4 comments

Thank you for the review. We want to address your question in two parts: whether anything is inferred from the corpus the models are tested on, and which evaluations are strictly causal. Table and section numbers in this response are those of the submitted PDF.


Whether anything is inferred from the test corpus

We do not infer anything from the test corpus. The annotator is spaCy's public en_core_web_sm model, which is trained on OntoNotes and not on our data, and every model, Base included, shares the same rule-based word splitter and the Gemma tokenizer (Sec. 4.1). The metadata is never a training target for Base, LIME and LIME-G (Sec. 3).

Additionally, our reported gains are not tied to one corpus, since the 7B models trained on FineWeb-Edu gain +1.24pp on their own held-out set and +1.25pp on the DCLM test set (Tab. 23).


Which evaluations are strictly causal

The table below answers this for every evaluation of LIME, LIME-G and LIME-A and gives the source of its metadata at test time. All evaluations of LIME-A are strictly causal, as are LIME on FLenQA and ARI-ADD and LIME-G wherever the task provides the class.

Model and evaluationMetadata at test timeStrictly causalAgainst Base
LIME: FLenQA, ARI-ADD (Tab. 3)tagger on the text so faryesup to +16pp on FLenQA
LIME-A: all evaluations (Sec. 4.6)its own headsyesup to +16pp on FLenQA
LIME-G: ARI-ADD, digit class alone (App. A.7)class from the taskyes51.4 against 21.8
LIME: decoding accuracy (Sec. 4.2, 4.7)tagger on the full textnoabout +1pp, +2.5pp in five languages
LIME: nine benchmarks (Tab. 2)tagger on context and candidatenoLAMBADA +3.3pp at 500M
LIME-G: decoding accuracy, Tab. 2, Tab. 3class from the referencenoabout +20pp decoding accuracy

To broaden the strictly causal evidence, we add two evaluations in this discussion. On LAMBADA at 500M, LIME-A generates the answer with its own heads supplying the classes and reaches 29.3 against 26.3 for Base. On BIG-bench QA-Wikidata (Srivastava et al., 2023), a public benchmark whose prompts are templates such as "The country of X is", LIME-G takes its class from the template alone and reaches 51.9 against 44.1 for Base at 500M.

The rows marked "no" use annotations of the full text by design, at test time exactly as in training. They measure what the metadata is worth to a decoder, which is how the paper found that the gain lies in the class of the token to be generated (Sec. 4.4, Tab. 21).

This insight is what makes the method usable at inference, because it tells us exactly what a model needs. LIME-G takes that class as an input, which raises decoding accuracy by over 40% (about 20pp) and makes the model steerable: one class given by the task more than doubles accuracy on ARI-ADD, while a hard constraint on the output changes nothing. LIME-A goes one step further and predicts the class itself. Although its two heads have to anticipate the class of a token not yet generated, it matches or exceeds LIME with an external tagger at every FLenQA context length (Sec. 4.6).


Reproducibility

Architectures, hyperparameters and evaluation protocols are in App. A.1, A.5, A.7 and A.8. The LIME Tokenization code is in the supplementary material, with the training and evaluation code to be released.


Your question made the paper clearer and we now state for every experiment whether it is strictly causal. The paper measures what linguistic metadata is worth to a decoder at up to 7B parameters and up to 302B training tokens, locates the signal behind the gain and turns it into a steering interface that works under strictly causal conditions.

Since this question was the open point of your review, we would be grateful if you considered raising your rating. If anything else would help your assessment, please tell us.


Changes in the revised PDF

  • Tab. 20 moves from the appendix into the main text (Sec. 4) and gains a column that marks each experiment as strictly causal or not.
  • Sec. 4.2 states the annotation regime of the decoding-accuracy results in the text.

Thank you again for your review. We are glad that you find the approach sound, see no critical weaknesses and value the long training horizon "way past Chinchilla optimum". We have addressed your question about which tests are entirely causal in our response, and the revised paper now states this directly, with all changes marked in blue. The table numbers below now refer to the revised PDF.

  • Which evaluations are strictly causal. The main text now states, for every experiment, how its metadata is obtained and whether an annotator sees the token to be generated (Tab. 5, Sec. 4.2).
  • New strictly causal results. On LAMBADA at 500M, LIME-A annotates its own output and achieves 29.3, compared to 26.3 for Base (Sec. 4.6). On the public QA-Wikidata benchmark, LIME-G takes its class from the prompt alone and achieves 51.9, compared to 44.1 for Base at 500M (App. A.7).
  • Nothing is inferred from the test corpus. The annotator is a public spaCy model trained on other data, and the metadata is never a training target for Base, LIME or LIME-G.

As the discussion period is coming to an end, we would like to ask whether this resolves your question. If you have any remaining concerns about the soundness or the claims, please let us know and we will respond right away.

As this was the one open point of your review, we would kindly ask you to raise your rating.

Best regards,

The Authors

Thank you for your reply, and the clarification on what is causal. I will maintain my rating of leaning toward accept -- I now feel more confident the paper deserves at least this rating, but not by enough to raise it to 4.

Thank you for your reply and for your support of the paper. We are glad that our clarifications increased your confidence in it.

Reviewer_weogOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces LIME, which incorporates linguistic metadata, such as part of speech and named entity recognition, into the token embeddings. LIME demonstrates consistent improvements in decoding accuracy with negligible parameter overhead. After pre-training with LIME and LIME-G (which augments the token embedding with metadata of the next token), LLMs achieve improved decoding accuracy across various model sizes and can match the final accuracy of the regular tokenizer baseline with fewer training tokens. Performance gains are most pronounced with LIME-G.

Critical strengths

The authors do a good job presenting empirical validation across model sizes from 500M to 7B parameters. The authors are very detailed with their evaluations and included experiments on a variety of benchmarks, multilingual settings, and self-annotations. LIME seems particularly promising for languages with high tokenizer fertility. The authors make a convincing argument that augmenting embeddings with linguistic metadata can prove especially useful for languages with high average tokens per word.

Critical weaknesses
  1. Reliance on future-context metadata. The main decoding accuracy gains of LIME rely on POS and NER annotations derived from full reference documents, which may encode information about future tokens unavailable during autoregressive inference. Appendix A.10 (Table 21) shows that the gains disappear when annotations are restricted to preceding context. This raises concerns about whether the reported improvements translate to realistic inference settings.

  2. Inconsistent performance of self-annotated metadata. Although LIME-A addresses the dependence on externally supplied metadata by predicting its own annotations, its improvements do not consistently generalize across downstream tasks. While LIME-A improves FLenQA and ARI-ADD, Appendix A.12 (Table 26) shows no improvement over the baseline on LAMBADA and TriviaQA, despite both being generative tasks. This raises concerns about the robustness and general applicability of the proposed approach.

Questions for authors
  1. Why does LIME-A improve FLenQA and ARI-ADD but fail to improve LAMBADA and TriviaQA? Since all four are generative tasks, could the authors explain the inconsistent results and investigate whether errors in self-predicted metadata contribute to the performance differences?

  2. Can LIME retain its benefits without metadata at inference time? MeCo [1] demonstrates that metadata-conditioned pretraining followed by a cooldown phase can preserve performance gains without requiring metadata during inference. Could a similar metadata conditioning then cooldown strategy be applied to LIME to address its reliance on future-context annotations?

[1] Gao, T., Wettig, A., He, L., Dong, Y., Malladi, S., & Chen, D. (2025). Metadata Conditioning Accelerates Language Model Pre-training. Proceedings of the 42nd International Conference on Machine Learning (ICML)

5 comments

Thank you for the careful review and the concrete questions. We would like to address your concerns below. Table and section numbers in this response are those of the submitted PDF.


W1. Future-context metadata and realistic inference

Locating where the gain comes from is a central part of the paper's investigation into the role and value of metadata embeddings (Sec. 4.4, with the full results in Tab. 21). This analysis shows that a decoder benefits from one specific signal, the class of the token to be generated. Giving this class directly, as in LIME-G, raises decoding accuracy by about 20pp at every model size (Sec. 5).

In generation the gains carry over. No annotator has to see the answer there, because the task can determine the class (LIME-G) or the model can predict it (LIME-A). On ARI-ADD, guided by nothing but the digit class that the task itself provides, LIME-G reaches 51.4 against 21.8 for Base, with or without a digit constraint (App. A.7). Guidance therefore changes what the model computes, while a constraint only filters what it emits. Without any given class, LIME and LIME-A still gain up to +16pp on FLenQA (Tab. 3).

A class given by the task also helps on a public benchmark that we added for this discussion. The prompts of BIG-bench QA-Wikidata (Srivastava et al., 2023) are templates such as "The country of X is", and LIME-G takes its class from the template alone. On the 61% of the items whose template fixes the class, it gains +9.2pp at 2B (70.1 against 60.9) and +11.1pp at 500M (54.6 against 43.5). Over all items the gains are +2.8pp and +7.8pp.


W2. Performance of self-annotated metadata across tasks

LIME-A is the frozen LIME model with two small heads, trained on about 5% of the pre-training budget, that predict the class of each token before it is generated. They do so with 64.8% accuracy for POS against 13.2% for the majority class (Sec. 4.6), which is enough to replace the external tagger. On FLenQA and ARI-ADD, where the task fixes the class of the answer, LIME-A improves over Base, matches or exceeds LIME at every FLenQA context length and stays within 1.4pp of it on ARI-ADD (Tab. 3).

At 2B, the size at which the paper evaluates LIME-A on the benchmarks (Tab. 26), LIME itself gains little on LAMBADA, which leaves LIME-A on par with Base. On TriviaQA the answer can be any entity, which makes its class hard for the heads to predict (see Q1 in our second comment). LIME-A is below Base there, as on multiple choice (Sec. 4.6): 21.7 against 25.8 (Tab. 26 listed 25.2, the one entry corrected in the revised PDF).

For this response we also evaluated LIME-A at 500M on both tasks, where it improves over Base on LAMBADA and is on par with Base on TriviaQA. On LAMBADA, LIME's gain is larger at this size and LIME-A keeps it, with its own heads supplying the classes of the generated answer: Base reaches 26.3, LIME 29.1 and LIME-A 29.3 (+3.0pp over Base, 95% interval +0.5 to +5.4). On TriviaQA it reaches 7.9 against 8.2 for Base.

Q1. Why the results differ across tasks and whether head errors contribute

The tasks differ in how predictable the class of the answer is. On FLenQA and ARI-ADD the task fixes it. On LAMBADA the heads predict the class of every answer token correctly for about half of the items, but on TriviaQA, where the answer can be any entity, for only 16 to 18%. To test whether these errors contribute, we split the items of LAMBADA and TriviaQA by this criterion. In every setting, LIME-A does better where its heads are right.

LIME-A minus Baseheads correcta head incorrect
LAMBADA, 500M+5.0pp (46% of items)+1.3pp (54%)
TriviaQA, 500M+1.2pp (16%)−0.6pp (84%)
LAMBADA, 2B0.0pp (51%)−0.2pp (49%)
TriviaQA, 2B−2.2pp (18%)−4.3pp (82%)

On LAMBADA at 500M the gain concentrates on the items where the heads are right. The heads are small and trained on about 5% of the pre-training budget, so improving their accuracy is a direct and inexpensive way to improve LIME-A (Sec. 5).


Q2. Keeping the benefit without metadata at inference

We would like to answer in two points: whether the metadata can be phased out as in MeCo's cooldown, and whether guidance can become optional. Our switch model addresses both (App. A.13). Its training leaves the class input empty at 10% of the positions and reduces the supplied class from 50% of the positions to 5%, which serves the same purpose as MeCo's cooldown. The trained switch model gains +2.27pp when the class is supplied and, with the same weights, stays within 0.3pp of Base without it (Tab. 27). Metadata can thus act as a switch that is turned on whenever a class is available.

MeCo's metadata is a document-level prefix whose benefit remains when the prefix is removed after a cooldown. LIME's metadata is a per-token signal that steers each generated token, which a document-level prefix cannot do. A cooldown alone therefore cannot preserve the gain, because the class carries information about the token to be generated that the preceding text does not contain (Sec. 5). We expect the same switch to work for LIME-G when it is trained in this way.


Thank you again for the precise questions, which led to two new analyses that are now in the paper. We also made the setup more explicit: the PDF now has an overview of the model variants (Sec. 3) and states in the main text how the metadata is obtained in each experiment (Tab. 20). Overall, the paper identifies the signal that makes linguistic metadata valuable to a decoder and turns it into a new steering interface for inference. This interface is already powerful at the scales we study: a guided 500M model outperforms an unguided model four times its size (Sec. 4.5).

We kindly ask you to reconsider your score in light of these answers. We are glad to answer any further question.


Changes in the revised PDF

  • Sec. 3: a short overview of the variants.
  • Sec. 4: Tab. 20 moves from the appendix into the main text.
  • Sec. 4.2: the text states the annotation regime of its results.
  • Sec. 4.6 and App. A.12: LIME-A at 500M on LAMBADA, the split by head correctness and the corrected TriviaQA entry of Tab. 26.
  • Sec. 4.6 and App. A.13: the switch model's training regime and the contrast with MeCo.

Thank you again for your thorough review. We are glad that you find the evaluations "very detailed" and see "a convincing argument" for linguistic metadata in languages with many tokens per word. The revised paper directly addresses the concerns about realistic inference settings and LIME-A across tasks, with all changes marked in blue. The table numbers below refer now to the revised PDF.

  • Realistic inference. With nothing but a class given by the task, LIME-G more than doubles accuracy on ARI-ADD (51.4 compared to 21.8 for Base, App. A.7). On the new public benchmark QA-Wikidata, it gains +9.2pp at 2B and +11.1pp at 500M on the items where the prompt fixes the class (Tab. 19).
  • LIME-A across tasks. At 500M, LIME-A improves LAMBADA by +3.0pp over Base, with its own heads annotating the generated answer (Sec. 4.6, App. A.12).
  • Head errors. The split you suggested shows that the difference between LIME-A and Base is higher where the heads are correct, in all four settings, reaching +5.0pp on LAMBADA at 500M (Tab. 31).
  • Cooldown. One model gains +2.27pp when the class is supplied and stays within 0.3pp of Base when it is not. Its training schedule and its relation to MeCo's cooldown are now stated (Sec. 4.6, App. A.13).
  • Clarity. An overview of the model variants has been added (Tab. 2), and the main text now states how the metadata for every experiment is obtained (Tab. 5).

As the discussion period is coming to an end, we would like to ask whether these additions resolve your concerns. If you have any remaining questions about the soundness of the claims, please let us know and we will respond right away.

If your concerns are addressed, we would kindly ask you to update your score.

Best regards,

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.