- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper studies linguistic annotations as token-aligned inputs to decoder language models. LIME adds POS/NER embeddings to the current token embedding; LIME-G shifts the annotations so the current input carries metadata about the token to be predicted; LIME-A predicts generated-token annotations with auxiliary heads. Models are pretrained at 500M–2B scale on 302B tokens, with additional 7B, multilingual, and annotation-context experiments. The reported token-accuracy gain is about one percentage point for LIME and about 20 points for reference-guided LIME-G. It transforms the metadata into a powerful controllable generation interface, demonstrating that the shift in reasoning time can unlock dormant reasoning and arithmetic capabilities.
Critical strengths
S1: The paper provides a clear analysis of where the gains come from. Comparisons across annotation contexts show that much of the benefit comes from information about subsequent tokens. This helps clarify why the method works and under what conditions.
S2: The method shows useful gains on some generation tasks. Results on reasoning and addition suggest that class guidance can improve answer correctness and merits further investigation.
Critical weaknesses
W1: Evidence for effectiveness in practical generation remains limited. The main accuracy and data-efficiency results rely on metadata derived from full documents or reference answers. Without this information, gains are smaller and less consistent, limiting the scope of the conclusions.
W2: Key ablations are needed to isolate the source of improvement. Existing comparisons change both training and inference, leaving unclear how much of the gain comes from training with additional class information versus embedding-level guidance at inference.
Questions for authors
Q1. How much of the reported improvement remains when metadata is obtained only from the prompt and previously generated tokens? A comparison separating reference-derived, task-provided, and self-predicted metadata would help clarify the method’s practical benefits.
Q2. Can you isolate the effects of metadata-conditioned training and inference-time guidance? For example, evaluating the same checkpoint with correct, missing, and perturbed guidance would help determine which component drives the improvement.
5 comments
W2. Isolating training from inference-time guidance
We ran the evaluation you proposed and report it under Q2. The paper already contains this test for the switch model, a 500M model with an optional class input (Sec. 4.6, App. A.13). It gains +2.27pp when the class is supplied at inference and stays within 0.3pp of Base without it (Tab. 27).
Q2. One checkpoint with correct, missing and perturbed guidance
To separate what training contributes from what the class at inference contributes, we evaluated the LIME-G 2B checkpoint on the same prompts and changed only the class given at the generated positions. "Missing" is the placeholder class that LIME-G receives in training at the end of a sequence, where no token follows. "Wrong" is a word class on ARI-ADD and a number class on FLenQA.
| Class given to LIME-G | ARI-ADD | FLenQA 250 / 500 / 1000 / 2000 / 3000 |
|---|---|---|
| correct | 56.6 | 78.0 / 74.5 / 61.5 / 51.0 / 36.3 |
| missing | 4.0 | 0.0 / 0.0 / 0.0 / 0.0 / 0.0 |
| wrong | 0.0 | 0.0 / 0.0 / 1.0 / 0.0 / 0.0 |
The class given at inference drives the improvement, provided the model was pre-trained with shifted metadata. With the correct class the checkpoint reaches 56.6 on ARI-ADD and up to 78.0 on FLenQA, whereas with a missing or wrong class it gives almost no correct answer. This pre-training is required to unlock the gain, because it teaches the model to follow metadata classes (Sec. 5). The class then decides the output, even when it is wrong: with the number class, the model starts its FLenQA answer with a number on 95% to 100% of the items. This is the steering interface the paper establishes: guidance acts inside the model, at the input embedding, while a constraint can only filter the output. The "correct" row is the "reference" row of the table under Q1.
LIME-G follows its class by design, since it always receives one (Eq. 2). For use without a supplied class the paper offers LIME and LIME-A, while the switch model shows that one model can do both (Tab. 27).
Ethics flag
The review flags the paper for ethics review, but we could not find a specific concern in it. The Ethics Statement covers the annotator's weaknesses on non-standard text, dialects and code-mixed input. If you had a concern beyond this in mind, please tell us and we will address it.
Both evaluations you proposed are now in the paper, where they support its central result: a decoder trained with shifted metadata follows the class it is given, which a task can supply at inference with no reference. LIME-A shows in addition that a model can annotate its own output.
On this basis, we kindly ask you to reconsider your score. We are glad to answer any further question.
Changes in the revised PDF
- Sec. 3: the overview of the variants.
- Sec. 4: Tab. 20 moves from the appendix into the main text.
- Sec. 4.2: the text states the annotation regime of its results.
- App. A.7: the table by source of the class with the new row for one class on all FLenQA items, the same-checkpoint rows (correct, missing, wrong) and the QA-Wikidata result.
Thank you again for your review. We are glad that you see "a clear analysis of where the gains come from" and useful gains on generation tasks. Your summary describes the method as "a powerful controllable generation interface", which the two evaluations you proposed further confirm. Both evaluations are now included in the revised paper, with all changes marked in blue. The table numbers below now refer to the revised PDF.
- Gains without reference-derived metadata. The comparison you proposed in Q1 is in App. A.7 (Tab. 17). With one class for all items, given by the task alone, LIME-G gains +13 to +26pp over Base at every FLenQA context length and more than doubles accuracy on ARI-ADD.
- A public benchmark. On QA-Wikidata, LIME-G takes its class from the prompt alone and gains +9.2pp at 2B and +11.1pp at 500M on the items where the prompt fixes the class (Tab. 19).
- Training and guidance at inference. The evaluation you proposed in Q2 is in Tab. 18. With the correct class, one checkpoint reaches up to 78.0 on FLenQA. With a missing or wrong class, almost no answers are correct, showing that the class given at inference drives the result.
- Clarity. An overview of the model variants has been added (Tab. 2), and the main text now specifies the source of the metadata for each experiment (Tab. 5).
As the discussion period is coming to an end, we would like to ask whether these additions resolve your concerns. If you have any remaining questions about the soundness of the claims or about the ethics flag, please let us know and we will respond right away.
If your concerns are addressed, we would kindly ask you to update your score.
Best regards,
The Authors
Thank you for the review. We ran the two comparisons you proposed in Q1 and Q2, which both confirm the paper's results. We respond to the weaknesses and questions in order below. Table and section numbers in this response are those of the submitted PDF.
W1. Evidence for practical generation
The paper uses the full document and the reference for one purpose, to measure what the class of the token to be generated is worth, which is the analysis you note in S1. With no annotator seeing the answer, metadata improves generation in both settings the paper proposes: a class determined by the task (LIME-G) and a class predicted by the model (LIME-A). In the paper, LIME-G guided by the digit class alone more than doubles accuracy on ARI-ADD (Sec. 4.5). LIME and LIME-A improve FLenQA with no supplied class, LIME-A even without any annotator after the prompt (Tab. 3). Under Q1 this response adds the same test with one class for all FLenQA items, where LIME-G gains +13 to +26pp across all context lengths, as well as an additional public benchmark with the class taken from the prompt. For clarity, the revised PDF adds an overview of the model variants (Sec. 3) and moves Tab. 20 from the appendix into the main text, where it states the source of the metadata for every result.
Q1. Gains without reference-derived metadata
In decoding accuracy the gain comes with the class of the token to be generated (Tab. 21), which is why the paper supplies exactly this class in generation.
For generation, the table below compares the three sources of the class that your question names. All rows are re-run in one evaluation of the 2B models on the same items, with every model generating token by token, so that the rows are directly comparable. The values are therefore not identical to Tab. 3. The row with one class for all items is new and set in bold.
The table shows that a class given by the task alone, with no reference, already brings a large gain. With one class for all items, LIME-G improves over Base at every FLenQA context length and stays between 55.0 and 62.0 from 250 to 3000 tokens, while Base falls to 30.0. On ARI-ADD the digit class alone more than doubles accuracy. With no supplied class at all, LIME and LIME-A still gain up to +20pp on FLenQA. LIME-A is level with or above LIME at every context length, which shows that the model's own heads can replace the tagger on generated text.
To test the task-provided class beyond the paper's two tasks, we evaluated LIME-G on BIG-bench QA-Wikidata (Srivastava et al., 2023), a public benchmark whose prompts are templates such as "The country of X is". LIME-G takes its class exclusively from the template. On the items where the template fixes the class, it gains +9.2pp at 2B and +11.1pp at 500M. Over all items the gains are +2.8pp and +7.8pp.