acceptodds
Under review as a conference paper at ICLR 2027

Expression Matters: Task Adaptation through Self-Generated Supervision

Abstract

While the next-token prediction (NTP) paradigm enables large language models (LLMs) to be versatile for various tasks, the NTP interface is not naturally matched to non-generative tasks whose desired outputs are not token sequences. We argue that LLM performance can be limited not only by what models know, but also by how they express what they know. We refer to the latter as the knowledge expression bottleneck. We introduce Self-Knowledge Re-expression (SKR), a task-general method that relies on a model's self-generated signals to align its output mechanism with a task-native format. SKR requires only unlabeled task inputs and an explicit task configuration, without relying on any human-provided annotations or external teacher models' outputs. This makes SKR much more cost-efficient to implement and allows it to be executed locally without sending task data to any external annotation service. Experiments across multiple tasks on an industrial financial-document dataset demonstrate substantial benefits of SKR. It achieves an absolute Recall@1 gain of at least 40% on information retrieval, reduces latency by at least 76% for instruction-conditioned region localization, and achieves an absolute AUPRC gain of at least 33% on imbalanced binary classification. On MMDocRAG, the strongest SKR model exceeds the best published baseline by an absolute 16.9% in Recall@10. These results show that, in addition to knowledge acquisition, knowledge expression is an important bottleneck in LLM task performance and that SKR provides a cost-efficient method for task adaptation.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Knowledge Re-expression, Task Adaptation, Annotation-free Adaptation

Reviews

Among:
Shape:
1234this paper, Bottom 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.25 over 4 reviews, above 49% of the others.
Reviewer_iYVqOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces Self-Knowledge Re-expression (SKR), motivated by the observation that the standard next-token prediction interface of LLMs may be poorly matched to non-generative downstream tasks. The authors formulate this mismatch as a knowledge expression bottleneck. SKR consists of two stages. First, the base model generates task-specific pseudo-targets from unlabeled inputs using its own generative behavior. Second, these pseudo-targets are used to train a task-native output mechanism together with LoRA adapters, while the original model parameters remain frozen. The paper instantiates this framework for three tasks, dense information retrieval, instruction-conditioned region localization, and imbalanced binary classification. Extensive experiments show that SKR substantially improves retrieval quality, reduces localization latency by replacing autoregressive/multi-turn generation with direct regression, and improves AUROC/AUPRC for imbalanced classification.

Critical strengths
  1. The method requires only unlabeled task inputs and task-level specifications, rather than instance-level human annotations or a stronger external teacher. This is practically important for private or domain-specific data, where external annotation can be costly or undesirable. The empirical improvements are also unusually large rather than marginal.
  2. The experimental analysis goes beyond a simple base-versus-method comparison. In particular, the comparison with self-SFT is useful because it shows that merely training on the same self-generated signals while retaining token generation is substantially weaker than moving to a task-native objective. The head-only experiments further show that attaching a new output layer without adapting internal representations is insufficient, while the external-teacher experiments explore when same-model supervision is sufficient versus when stronger supervision can help.
Critical weaknesses
  1. The central “knowledge expression bottleneck” claim is not fully isolated experimentally. Full SKR jointly trains LoRA adapters and task-specific output heads, so the gains may partly come from improved task-specific representations rather than only from changing the output interface. The head-only ablation actually suggests that internal representation adaptation is important. A more controlled baseline would help distinguish “re-expression” from standard self-training or representation learning.
  2. The evidence for broad task generality is still limited. The three tasks require quite different task-specific prompts, output heads, losses, and post-processing procedures. Therefore, SKR currently appears more like a general adaptation recipe than a unified task-general method. In addition, several experiments use relatively older base models or published baselines trained under different settings, making some of the broader comparisons less controlled.
Questions for authors

see weaknesses

3 comments

Thank you for your careful and helpful feedback. We have updated the PDF in response to the reviews. Section 6 expands the discussion of related work. Section 4.2 introduces extra baselines. Section 5.1.2 reports the Qwen3.5’s results on public-datasets and compare SKR with the added baselines. Appendix C.5 adds self-annotation, training, and inference cost information. Appendix D.4 reports classification results using continuous yes/no token probabilities and hard labels.

We respond to your specific concerns below.

Weaknesses

W1: The central “knowledge expression bottleneck” claim is not fully isolated experimentally.

We will clarify this point more precisely. The expression bottleneck does not mean that output heads alone determine task performance. Rather, as you point out, task-specific output heads must be matched with suitable intermediate representations to be effective.

Our paper includes detailed analyses of both the components of SKR and the source of its training signals. We are sorry that these results are distributed across different sections of the paper due to length limitations. Here, we bring the analysis together to clarify the comparisons. We have also added two baselines, LLM2Vec and SELF-GUIDE. These comparisons provide further evidence on whether external supervision is necessary and how the expression mechanism affects performance. We present and discuss the additional results in detail in our response to your next question.

Supervision source compairson (Section 5.2.2 and Figure 3). We replace self-generated targets with GPT-4o-generated targets while keeping the adapted model and procedure fixed. Self-generated targets perform comparably in retrieval, though GPT-4o improves LLaVA’s Recall@5 from 0.665 to 0.685. In localization, GPT-4o targets help LLaVA and Phi-3-VL, while Qwen2-VL benefits more from its own targets. Thus, external supervision can help when self-generated signals are weaker, but is not always necessary.

Adaptation components (Section 5.2.3, Tables 1–3 and 7, and Appendix F). We compare the unadapted base model, self-SFT (base + LoRA), head-only adaptation (base + task-native head), and SKR (base + LoRA + head). Self-SFT and SKR use the same self-generated targets. SKR generally improves performance and efficiency for three tasks, although they differ in both output mechanism and training objective. These comparisons show that, based on the model’s own training signals, re-expressing its knowledge through a task-native output mechanism and adapted representations can substantially improve task performance.

Together, these comparisons highlight the value of self-generated signals and task-native knowledge re-expression, distinguishing SKR from conventional task-specific fine-tuning.

W2: The evidence for broad task generality is still limited.

As stated in Lines 63–69, we present SKR as a general adaptation recipe with task-specific instantiations, rather than as a single task-independent architecture or objective. Its main benefit is to replace the need for instance-level task annotations with a task-level configuration. Across tasks, SKR follows the same procedure: it elicits pseudo-targets from unlabeled inputs using the model being adapted, converts these outputs into training targets, and jointly trains LoRA parameters and a task-native output mechanism. The prompt, output mechanism, objective, and post-processing may vary by task, but the overall adaptation procedure remains the same. We demonstrate this recipe on three substantially different task types.

We appreciate your thoughtful review and your recognition of SKR’s annotation-free setting and the value of our ablation comparisons. In response to your concerns, we brought together experiments from different sections of the paper to clarify how we isolate SKR’s components and knowledge sources step by step. We also clarified that SKR is a general adaptation recipe with task-specific configurations.

In our responses to other reviewers, we also added results for Qwen3.5-9B across the three tasks, together with LLM2Vec and SELF-GUIDE baselines, to strengthen the evaluation. All additional results and explanations have been added to the revised PDF. If you are interested in any of these results, please feel free to ask, and we would be happy to provide further details.

If you have any other remaining concerns, please do not hesitate to let us know. We would be happy to address them. If our rebuttal has addressed your concerns, we would be grateful if you would consider updating your evaluation.

Thank you again for your time and thoughtful review. As the rebuttal deadline is approaching, we wanted to follow up on our response and the additional experiments.

Another reviewer has acknowledged our additional experiments, and we appreciate that they have raised their rating. If our response and new results have addressed your concerns, we would be grateful if you would consider updating your evaluation as well.

Reviewer_P6NCOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Self-Knowledge Re-expression (SKR), which uses a model’s own pseudo-supervision to adapt generative models into retrievers, localization regressors, and classifiers through LoRA and task-specific output mechanisms. Experiments report improvements in effectiveness and efficiency, while novelty, evaluation comparability, and attribution to an “expression bottleneck” require stronger support.

Critical strengths
  1. Reduced reliance on external supervision. Adaptation requires neither human instance-level annotations nor external teacher outputs, offering practical benefits for annotation cost and local deployment.

  2. Breadth across output types. Evaluation spans retrieval, localization, and classification across multiple models and public datasets, demonstrating the recipe’s applicability.

  3. Practical effectiveness and efficiency. Converting expensive generative or multi-turn localization procedures into direct prediction provides meaningful deployment benefits.

Critical weaknesses
  1. Unclear novelty. The paper insufficiently discusses closely related work, including Promptagator [1], LLM2Vec [2], and SELF-GUIDE [3], leaving its contribution beyond self-training and task-specific adaptation unclear.

  2. Uncertain evaluation comparability. The connection between MMDocRAG’s multi-evidence recall and the paper’s single-relevant-item formulation is unclear. Classification baseline metrics also match hard-label scoring, motivating stronger continuous-score comparisons.

  3. Insufficient isolation of the central hypothesis. SKR changes objectives, output mechanisms, and internal parameters simultaneously, preventing a clear distinction between resolving an expression bottleneck and ordinary representation learning or domain adaptation.

[1] Promptagator: Few-shot Dense Retrieval From 8 Examples. ICLR 2023.

[2] LLM2Vec: Large Language Models Are Secretly Powerful Text Encoders. COLM 2024.

[3] SELF-GUIDE: Better Task-Specific Instruction Following via Self-Synthetic Finetuning. COLM 2024.

Questions for authors
  1. Please specify MMDocRAG’s relevance sets, candidate pools, and recall implementation, and verify comparability with the cited baselines using the same evaluation pipeline.

  2. Could you separately report classification results using hard labels, normalized yes/no probabilities, and logit differences?

  3. Could you add matched-budget comparisons with the closest prior methods and further separate the effects of output mechanisms and training objectives under identical pseudo-labels and LoRA settings?

  4. In Figure 3, LLaVA achieves 0.665 with self-annotation versus 0.685 with GPT-4o annotation. Should the claim of matching or outperforming external supervision be revised?

  5. Could you evaluate all three tasks on at least one recent small model from each of the Qwen-3.5 or Gemma-4 families, using optimized single-turn localization and continuous label-scoring baselines, to determine whether SKR’s benefits extend beyond limitations of older models?

4 comments
AuthorsOpenReview ↗

Response to Reviewer P6NC (1/2)

Thank you for your careful and helpful feedback. We have updated the PDF in response to the reviews. Section 6 expands the discussion of related work. Section 4.2 introduces extra baselines. Section 5.1.2 reports the Qwen3.5’s results on public-datasets and compare SKR with the added baselines. Appendix C.5 adds self-annotation, training, and inference cost information. Appendix D.4 reports classification results using continuous yes/no token probabilities and hard labels.

We respond to your specific concerns below.

W1: Novelty compared to existing works.

SKR’s contribution has two aspects.

First, it provides a self-training recipe that we evaluate across distinct task types. The model generates task-specific pseudo-targets from unlabeled inputs, and the same model is adapted to express them through a task-native output mechanism. This differs from Promptagator, which uses the external FLAN-137B model and a small number of task-specific annotated examples to generate synthetic queries, and then trains separate T5-based retrievers and rerankers. SKR instead uses the model being adapted as the source of its task-specific training signals, without an external teacher or query generator. LLM2Vec is a general text-embedding method, evaluated across multiple text-embedding tasks. Its output, however, remains a text representation in embedding space. SKR addresses a broader set of output types, including retrieval embeddings, localization coordinates, and classification scores.

Second, SKR studies whether a model’s task-relevant capabilities are constrained by its output mechanism. Our results show that changing the expression mechanism can improve the use of capabilities that are not fully realized through standard next-token generation. This is distinct from SELF-GUIDE, which also uses self-generated data but fine-tunes the model to improve instruction following while retaining the standard text-generation objective.

We additionally compare LLM2Vec and SELF-GUIDE using Qwen3.5-9B. These comparisons can also emphasize SKR’s difference compared to task-specific fine-tuning. Results are detailed in our response to Q5 below.

We have revised the related-work discussion in Section 6 to include discussions with more related works.

W2, Q1: Evaluation clarification on MMDocRAG.

SKR’s retrieval formulation is not restricted to a single relevant item. It produces embeddings that can rank multiple relevant items for a query. On our financial dataset, each query has one gold item, so we report single-positive retrieval metrics. On MMDocRAG, we use the benchmark’s multiple-evidence annotations and compute recall@k the same as the baselines.

W2, Q2: Could you separately report classification results using hard labels, normalized yes/no probabilities, and logit differences?

We report the two classification scoring approaches of base models here. AUROC1 and AUPRC1 use the normalized yes/no token probabilities as continuous scores. AUROC2 and AUPRC2 use the generated yes/no labels encoded as scores 1 and 0. These are the two methods we mentioned in lines 363-365.

AccAUROC1AUPRC1AUROC2AUPRC2Time
LLaVA-1.6Base0.9300.8940.5310.7170.3933.8
SFT0.8350.8920.3810.6420.1521.4
SKR0.9750.9910.879––1.3
Phi-3-VLBase0.7700.8500.2870.6050.1264.7
SFT0.9450.9120.8500.8810.6577.1
SKR0.9350.9810.933––0.7
Qwen2-VLBase0.9350.8750.5320.8750.5321.1
SFT0.9600.9390.7460.8220.6110.9
SKR0.9600.9840.868––0.7

These two scoring methods and the complete financial-dataset results are now reported in Appendix D.4.

W3, Q3: Isolation of the central hypothesis.

Due to the length limitation of the main text, the relevant controls were not sufficiently visible in the main text. We have included many controlled comparisons in appendix.

First, Tables 1–3 compare SKR with token-level self-SFT using the same self-generated targets and matched per-task training configurations. This tests whether self-generated data alone accounts for the gains, or whether using a task-native output mechanism and objective provides additional benefit.

Second, Table 7 and Appendix F (Tables 16–18) compare full SKR with a head-only variant. Both use the same task-native heads and losses, while head-only freezes the backbone and full SKR additionally trains LoRA adapters. This isolates the contribution of adapting the backbone.

Third, Figure 3 replaces SKR’s self-generated targets with GPT-4o-generated targets while retaining the downstream adaptation setup, testing the effect of the target source. Appendix G further reports CKA analyses of representation changes.

These comparisons address the roles of self-generated supervision, task-native objectives, and LoRA-based backbone adaptation.

AuthorsOpenReview ↗

Response to Reviewer P6NC (2/2)

Q4: In Figure 3, LLaVA achieves 0.665 with self-annotation versus 0.685 with GPT-4o annotation. Should the claim of matching or outperforming external supervision be revised?

We agree that this result should be stated more precisely. The figure is intended to show that the annotation source has a relatively modest effect compared with the much larger differences between the base and adapted model in Table 1. We will make this interpretation explicit.

Q5: Results on recent models.

To strengthen the comparison and assess a more recent model, we evaluate Qwen3.5-9B on public datasets across the three tasks. We also add LLM2Vec and SELF-GUIDE, recommended by reviewers, as baselines. The results are reported below.

We adapted Qwen3.5 into an embedding model using LLM2Vec’s input perturbation approach. The original LLM2Vec is a unimodal method designed for text. To adapt it for cross-modal retrieval, we used the query–image pairs from MMDocRAG to train the LLM2Vec-style baseline. Note that SKR uses only raw images without using queries. Therefore, the LLM2Vec-style baseline uses additional paired information during training. SELF-GUIDE is a cross-task self-synthetic fine-tuning approach. Its few-shot examples guide synthetic label generation, so its configuration uses a small amount of supervision.

IR-MMDocRAG

MethodMRRRecall@10Recall@15Recall@20
Qwen3.5 Base0.2810.5270.6300.715
Qwen3.5 + SKR0.7360.8790.9220.947
SELF-GUIDE0.2700.5080.6120.698
LLM2Vec0.7350.8780.9190.944

IR-DocVQA

MethodMRRRecall@1Recall@3Recall@5Recall@10
Qwen3.5 Base0.8660.7830.9390.9730.996
Qwen3.5 + SKR0.9270.8770.9750.9910.998
SELF-GUIDE0.8620.7760.9370.9730.995
LLM2Vec0.9230.8700.9740.9900.998

IR-SciMMIR

MethodMRRRecall@1Recall@3Recall@5Recall@10
Qwen3.5 Base0.0030.0010.0030.0030.005
Qwen3.5 + SKR0.5420.4400.6040.6630.728
SELF-GUIDE0.0030.0010.0020.0030.004
LLM2Vec0.5320.4330.5970.6510.712

Localization

MethodIoUTime (min/100 cases)
Qwen3.5 Base0.8832.5
Qwen3.5 + SKR,20 epochs0.8890.3
SELF-GUIDE0.8922.2

Imbalanced classification

MethodAccuracyAUROCAUPRCTime (min/100 cases)
Qwen3.5 Base0.9850.8340.4530.9
Qwen3.5 + SKR0.9900.8540.6590.3
SELF-GUIDE0.9900.8210.5780.7

The Qwen3.5 results show that SKR remains effective on a more advanced model. It improves upon the base model and outperforms the baselines. Across the three tasks, its benefits in performance and efficiency are consistent with the trends observed for Qwen2-VL in the paper.

SKR obtains the strongest reported retrieval scores across the three datasets. The LLM2Vec-style baseline comes close, while using extra query information from training data. SELF-GUIDE, which retains a generative objective, does not improve on the base retrieval scores in these experiments. These results indicate that SKR’s self-annotations are effective and task re-expression is necessary.

For localization and classification, the LLM2Vec-style baseline is not applicable as implemented because it is designed for embedding-based retrieval. SELF-GUIDE can be evaluated on both tasks. On localization, with similar comparisons as Table2, SKR achieves comparable IoU to base and SELF-GUIDE, but with reported inference time reduced from 2.54 and 2.16 to 0.31, reducing more than 86%. On classification, SKR improves both AUROC and AUPRC compared to both base and Self-GUIDE, while reducing reported time from 0.85 and 0.66 to 0.32, reducing more than 51%.

Together, these comparisons illustrate two aspects of SKR: its recipe can be instantiated across different tasks, and task-native re-expression can improve effectiveness and efficiency. The Qwen3.5 results show that these patterns also appear on a more recent model.

The Qwen3.5 baseline setup is described in Section 4.2, and results across the three tasks are presented in Section 5.1.2, Tables 4–6.

Thank you for the clarifications. Your response addresses most of my concerns. I've increase my rating to weak acceptance.

Thank you for your thoughtful review and for raising your rating. We appreciate your time and constructive feedback. If you have any further questions, please feel free to ask. We would be happy to address them.

Reviewer_ReuSOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This work argues that LLM's performance on any task is limited not only by what model knows but the also by how it expresses it. Specifically, NTP is a poor representation space for non-generative tasks such as retrieval. To resolve this, the paper presents the Self-Knowledge Re-expression. Experiments use a proprietary financial-document dataset with several LLMs. The paper reports large gains over the raw base models in retrieval, latency improvements for localization, and performance gains for imbalanced classification.

Critical strengths
  • The paper reports the string empirical performance. SKR improves MMDocRAG Recall significantly for Qwen2-VL and delivers substantial performance gains in classification.
  • The approach works across three distinct tasks using small unlabeled datasets, without requiring human annotations or external teacher models.
  • Comparisons against self-SFT, head-only adaptation, and GPT-4o supervision provide evidence for the benefits of task-native output representations.
Critical weaknesses
  • SKR has very limited novelty. Specifically, self-generated pseudo-labeling, task-specific prediction heads, and contrastive learning are established techniques. The paper primarily combines them under a new conceptual framing.
  • The models experiments/evaluated are very old and does not represent the recent SoTA models and limits the practicality and questions the usefulness of SKR against SoTA NTP models.
  • Comparisons against token-level SFT do not fully isolate the proposed mechanism. And stronger baselines using synthetic-data contrastive fine-tuning or conventional self-training are missing.
  • The proprietary classification test set contains only 200 examples, and results lack confidence intervals, multiple seeds, or significance testing.
Questions for authors
  • Can authors compare SKR against the latest SoTA open source models?
  • Can we compare SKR against standard self-supervised or synthetic-data fine-tuning methods using identical pseudo-labels, training budgets, and task-native objectives?
  • Please report multi-seed results and end-to-end computational costs, including self-annotation and adaptation.
4 comments
AuthorsOpenReview ↗

Response to Reviewer ReuS (1/2)

Thank you for your careful and helpful feedback. We have updated the PDF in response to the reviews. Section 6 expands the discussion of related work. Section 4.2 introduces extra baselines. Section 5.1.2 reports the Qwen3.5’s results on public-datasets and compare SKR with the added baselines. Appendix C.5 adds self-annotation, training, and inference cost information. Appendix D.4 reports classification results using continuous yes/no token probabilities and hard labels.

We respond to your specific concerns below.

W1: Limited novelty.

SKR’s novel contributions focus on two aspects. First, it provides a cross-task self-training recipe: the model generates task-specific pseudo-targets from unlabeled inputs, and the same model is then adapted to express them through a task-native output mechanism. Although self-generated pseudo-labeling, task-specific prediction heads, and contrastive learning are established techniques, SKR is not designed for retrieval alone. We demonstrate the recipe across three substantially different tasks: information retrieval, region localization, and imbalanced classification.

Second, SKR decoups a model’s task-relevant knowledge from the mechanism used to express it, and our experiments demonstrate that an NTP interface can limit how effectively a model’s existing task capabilities are used. Adapting the expression mechanism can reveal and improve those capabilities. Thus, our contribution is not a claim that the individual components are new, but a cross-task adaptation recipe and an empirical study of expression as a bottleneck in task performance.

The revised discussion of related work and the distinction between task-native adaptation and generative self-training is in Section 6.

W2, Q1: SOTA NTP models. W3, Q2: Strong synthetic-data baselines.

To strengthen the comparison and assess a more recent model, we evaluate Qwen3.5-9B on public datasets across the three tasks. We also add LLM2Vec and SELF-GUIDE, recommended by reviewers, as baselines. The results are reported below.

We adapted Qwen3.5 into an embedding model using LLM2Vec’s input perturbation approach. The original LLM2Vec is a unimodal method designed for text. To adapt it for cross-modal retrieval, we used the query–image pairs from MMDocRAG to train the LLM2Vec-style baseline. Note that SKR uses only raw images without using queries. Therefore, the LLM2Vec-style baseline uses additional paired information during training. SELF-GUIDE is a cross-task self-synthetic fine-tuning approach. Its few-shot examples guide synthetic label generation, so its configuration uses a small amount of supervision.

IR-MMDocRAG

MRRR@10R@15R@20
Qwen3.5 Base0.2810.5270.6300.715
Qwen3.5 + SKR0.7360.8790.9220.947
SELF-GUIDE0.2700.5080.6120.698
LLM2Vec0.7350.8780.9190.944

IR-DocVQA

MRRR@1R@3R@5R@10
Qwen3.5 Base0.8660.7830.9390.9730.996
Qwen3.5 + SKR0.9270.8770.9750.9910.998
SELF-GUIDE0.8620.7760.9370.9730.995
LLM2Vec0.9230.8700.9740.9900.998

IR-SciMMIR

MRRR@1R@3R@5R@10
Qwen3.5 Base0.0030.0010.0030.0030.005
Qwen3.5 + SKR0.5420.4400.6040.6630.728
SELF-GUIDE0.0030.0010.0020.0030.004
LLM2Vec0.5320.4330.5970.6510.712

Localization

IoUTime
Qwen3.5 Base0.8832.5
Qwen3.5 + SKR,20 epochs0.8890.3
SELF-GUIDE0.8922.2

Imbalanced classification

AccAUROCAUPRCTime
Qwen3.5 Base0.9850.8340.4530.9
Qwen3.5 + SKR0.9900.8540.6590.3
SELF-GUIDE0.9900.8210.5780.7
AuthorsOpenReview ↗

Response to Reviewer ReuS (2/2)

W2, Q1: SOTA NTP models.

The Qwen3.5 results show that SKR remains effective on a more advanced model. It improves upon the base model and outperforms the baselines. Across the three tasks, its benefits in performance and efficiency are consistent with the trends observed for Qwen2-VL in the paper.

W3, Q2: Strong synthetic-data baselines.

SKR obtains the strongest reported retrieval scores across the three datasets. The LLM2Vec-style baseline comes close, while using extra query information from training data. SELF-GUIDE, which retains a generative objective, does not improve on the base retrieval scores in these experiments. These results indicate that SKR’s self-annotations are effective and task re-expression is necessary.

For localization and classification, the LLM2Vec-style baseline is not applicable as implemented because it is designed for embedding-based retrieval. SELF-GUIDE can be evaluated on both tasks. On localization, with similar comparisons as Table2, SKR achieves comparable IoU to base and SELF-GUIDE, but with reported inference time reduced from 2.54 and 2.16 to 0.31, reducing more than 86%. On classification, SKR improves both AUROC and AUPRC compared to both base and Self-GUIDE, while reducing reported time from 0.85 and 0.66 to 0.32, reducing more than 51%.

Together, these comparisons illustrate two aspects of SKR: its recipe can be instantiated across different tasks, and task-native re-expression can improve effectiveness and efficiency. The Qwen3.5 results show that these patterns also appear on a more recent model.

Section 4.2 describes the additional baseline implementations and their supervision; the results across the three tasks appear in Section 5.1.2, Tables 4–6.

W4, Q3: The proprietary classification test set contains only 200 examples, and results lack confidence intervals, multiple seeds, or significance testing. Please report multi-seed results and end-to-end computational costs, including self-annotation and adaptation.

Although the financial test sets are relatively small, they consist of high-quality, practical examples and directly measure performance in realistic application scenarios.

Our reported results are based on three runs. The observed differences between the base and SKR models are statistically significant under a t-test (p<0.05p < 0.05). The run-to-run variation is generally within approximately ±0.5 points. Since the performance differences are substantially larger than this variation, we did not include variance values in the original presentation. We will emphasize this setting in the revision.

Annotation time (min)SKR Training Time (min)SFT Training Time (min)
IR (text-to-image)137.4195.8406.4
Loc166.2232.5306.8
IC36.6122.7135.4

The computational costs are reported in the table above, using the data described in Section A.1 and the devices in Section C.4. As discussed in Section 4.1, SKR can implement effective task adaptation using a relatively small amount of unlabeled task data. Compared with SFT, SKR is faster during both adaptation and inference. SFT still trains a token-generation objective, whereas SKR trains a task-specific output head.

The cost breakdown is now included in Appendix C.5.

We appreciate your thoughtful review and your recognition of SKR’s performance across tasks without human annotations or an external teacher.

In response to your concerns, we evaluated Qwen3.5-9B across the three tasks and added comparisons with LLM2Vec and SELF-GUIDE to include a more recent open-source model and additional baselines. The results show SKR’s effectiveness on a newer model, further emphasizing and clarifying our novelty and contribution. We have also included more experimental information to address your concerns about computational costs and detailed results.

If you have any remaining concerns, please do not hesitate to let us know. We would be happy to address them. If our rebuttal has addressed your concerns, we would be grateful if you would consider updating your evaluation.

Thank you again for your time and thoughtful review. As the rebuttal deadline is approaching, we wanted to follow up on our response and the additional experiments.

Another reviewer has acknowledged our additional experiments, and we appreciate that they have raised their rating. If our response and new results have addressed your concerns, we would be grateful if you would consider updating your evaluation as well.

Reviewer_Q3xROpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The authors introduce Self-Knowledge Re-expression (SKR), which uses self-generated pseudo-labels to adapt LLMs to task-native output formats through LoRA and task-specific heads. Experiments show improvements in performance and efficiency without external supervision.

Critical strengths
  1. The motivation is clear and the research question is meaningful and interesting. The paper highlights the mismatch between next-token prediction and non-generative tasks, providing a unified perspective on task adaptation.
  2. The experiments cover three different output types across multiple LLMs and MLLMs, on both an industrial dataset and public benchmarks, with ablations on supervision sources and adaptation components.
Critical weaknesses
  1. Several closely related works, such as InPars [1], Promptagator [2], E5-mistral [3], LLM2Vec [4], and VLM2Vec [5], are not adequately discussed. Thus, it is unclear what the main methodological novelty of SKR is. Moreover, the experiments do not clearly show whether the improvements come from re-expressing existing knowledge or simply learning task-specific representations through fine-tuning.
  2. Some experimental comparisons could be strengthened. Retrieval gains are mainly measured against unadapted hidden states, and stronger label-free baselines (e.g., LLM2Vec) are missing; the GPT-4o comparison in Sec. 5.2.2 is limited in scope. On MMDocRAG, SKR uses in-domain development data, while baseline results are taken from published evaluations. Latency gains are measured mainly against multi-turn generation, and localization gains are limited or negative for Qwen2-VL. Small test sets, no variance estimates, and unavailable code further limit reliability.

[1] InPars: Unsupervised Dataset Generation for Information Retrieval, SIGIR 2022

[2] Promptagator: Few-shot Dense Retrieval From 8 Examples, ICLR 2023

[3] Improving Text Embeddings with Large Language Models, ACL 2024

[4] LLM2Vec: Large Language Models Are Secretly Powerful Text Encoders, COLM 2024

[5] VLM2Vec: Training Vision-Language Models for Massive Multimodal Embedding Tasks, ICLR 2025

Questions for authors
  1. How does SKR differ from existing methods such as InPars, Promptagator, E5-mistral, LLM2Vec, and VLM2Vec?
  2. How can the authors distinguish knowledge re-expression from conventional task-specific fine-tuning? Can additional experiments directly validate the proposed expression bottleneck?
  3. Can the authors include stronger baselines under matched training conditions? For MMDocRAG, is there any overlap between the development images and the test candidate corpus?
  4. Why does SKR underperform both the multi-turn base model and SFT on Qwen2-VL localization (0.726 vs. 0.739 and 0.741)?
  5. The reported 76% latency reduction drops to approximately 26–29% against one-turn and SFT baselines. Can the authors provide fairer efficiency comparisons, including training and pseudo-label generation costs?
5 comments
AuthorsOpenReview ↗

Response to Reviewer Q3xR (1/2)

Thank you for your careful and helpful feedback. We have updated the PDF in response to the reviews. Section 6 expands the discussion of related work. Section 4.2 introduces extra baselines. Section 5.1.2 reports the Qwen3.5’s results on public-datasets and compare SKR with the added baselines. Appendix C.5 adds self-annotation, training, and inference cost information. Appendix D.4 reports classification results using continuous yes/no token probabilities and hard labels.

We respond to your specific concerns below.

W1, Q1: How does SKR compare to existing methods.

We thank the reviewer for pointing us to these closely related works. Discussing them more explicitly helps clarify SKR’s contribution.

SKR differs from these approaches in two main ways.

First, SKR uses the model being adapted as the sole source of task-specific training signals. It does not use an external teacher or query generator to create its task-specific training signals. InPars uses GPT-3 Curie to generate synthetic queries and trains a monoT5 reranker on the resulting data. Promptagator uses FLAN-137B to generate synthetic queries, then trains a T5-initialized dual encoder and reranker. E5-Mistral uses GPT-4 and GPT-3.5-Turbo to generate synthetic data for fine-tuning Mistral-7B. VLM2Vec trains on labeled data from MMEB. In contrast, SKR obtains its synthetic training signals from the model being adapted. The same model produces the data and is then trained on it, without an external teacher or task-specific human annotations.

Second, SKR is a task-adaptation recipe that we evaluate across distinct task types. The cited works primarily study retrieval and embedding, whereas SKR is evaluated on three different task types in our paper. These results suggest that the self-generated training signal can support adaptation beyond a single retrieval or embedding setup.

We have expanded the related-work discussion in Section 6 to clarify the relationship between SKR and these methods.

W1, Q2: How to distinguish knowledge re-expression from task-specific fine-tuning?

SKR differs from task-specific fine-tuning in two main respects. First, its training signals come from the model itself. It doesn’t rely on task annotations or an external teacher. Second, SKR provides a cross-task adaptation recipe. The same procedure can be instantiated for different output types, as we demonstrate on retrieval, localization, and classification. Other task-specific fine-tuning methods such as LLM2Vec are designed for a particular type of task.

Our paper includes detailed analyses of both the components of SKR and the source of its training signals. These results are distributed across different sections of the paper. Here, we bring the analysis together to clarify the comparisons. We have also added two baselines, LLM2Vec and SELF-GUIDE. These comparisons provide further evidence on whether external supervision is necessary and how the expression mechanism affects performance. We present and discuss the additional results in detail in our response to your next question.

Supervision source comparison (Section 5.2.2 and Figure 3). We replace self-generated targets with GPT-4o-generated targets while keeping the adapted model and procedure fixed. Self-generated targets perform comparably in retrieval, though GPT-4o improves LLaVA’s Recall@5 from 0.665 to 0.685. In localization, GPT-4o targets help LLaVA and Phi-3-VL, while Qwen2-VL benefits more from its own targets. Thus, external supervision can help when self-generated signals are weaker, but is not always necessary.

Adaptation components (Section 5.2.3, Tables 1–3 and 7, and Appendix F). We compare the unadapted base model, self-SFT (base + LoRA), head-only adaptation (base + task-native head), and SKR (base + LoRA + head). Self-SFT and SKR use the same self-generated targets. SKR generally improves performance and efficiency for three tasks, although they differ in both output mechanism and training objective. These comparisons show that, based on the model’s own training signals, re-expressing its knowledge through a task-native output mechanism and adapted representations can substantially improve task performance.

Together, these comparisons highlight the value of self-generated signals and task-native knowledge re-expression, distinguishing SKR from conventional task-specific fine-tuning.

AuthorsOpenReview ↗

Response to Reviewer Q3xR (2/2)

W2, Q3.1: Stronger baselines like LLM2Vec. MMDocRAG data.

We evaluate Qwen3.5-9B across three tasks to include recent model results. We also add LLM2Vec and SELF-GUIDE as baselines. The results are reported below.

Our retrieval results (Table 1) have compared SKR with OpenAI ada-002 and Nomic-text-v2. For context, reported MTEB averages are 56.8 for unsupervised LLM2Vec, 61.0 for ada-002, and 62.39 for Nomic-v1. Therefore, SKR has outperformed baselines better than LLM2Vec.

To further compare with LLM2Vec, we adapted Qwen3.5 into an embedding model using LLM2Vec’s input perturbation approach. LLM2Vec is a unimodal method designed for text. To adapt it for cross-modal retrieval, we used the query–image pairs from MMDocRAG to train. Note that SKR uses only raw images without using queries. Therefore, the LLM2Vec baseline uses additional paired information during training. SELF-GUIDE is a cross-task self-synthetic fine-tuning approach. Its few-shot examples guide synthetic label generation, so its configuration uses a small amount of supervision.

IR-MMDocRAG

MRRR@10R@15R@20
Qwen3.5 Base0.2810.5270.6300.715
Qwen3.5 + SKR0.7360.8790.9220.947
SELF-GUIDE0.2700.5080.6120.698
LLM2Vec0.7350.8780.9190.944

IR-DocVQA

MRRR@1R@3R@5R@10
Qwen3.5 Base0.8660.7830.9390.9730.996
Qwen3.5 + SKR0.9270.8770.9750.9910.998
SELF-GUIDE0.8620.7760.9370.9730.995
LLM2Vec0.9230.8700.9740.9900.998

IR-SciMMIR

MRRR@1R@3R@5R@10
Qwen3.5 Base0.0030.0010.0030.0030.005
Qwen3.5 + SKR0.5420.4400.6040.6630.728
SELF-GUIDE0.0030.0010.0020.0030.004
LLM2Vec0.5320.4330.5970.6510.712

Localization

IoUTime
Qwen3.5 Base0.8832.5
Qwen3.5 + SKR,20 epochs0.8890.3
SELF-GUIDE0.8922.2

Imbalanced classification

AccAUROCAUPRCTime
Qwen3.5 Base0.9850.8340.4530.9
Qwen3.5 + SKR0.9900.8540.6590.3
SELF-GUIDE0.9900.8210.5780.7

SKR achieves the best retrieval results on all sets. LLM2Vec comes close using additional query information. SELF-GUIDE, which retains a generative objective, does not improve on the base retrieval scores in these experiments.

LLM2Vec is not applied to the other two tasks. On localization, similar to comparisons in Table2, SKR has comparable IoU and reduces time by over 85% versus base and SELF-GUIDE. On classification, SKR improves AUROC and AUPRC over both while reducing time by over 51%.

Together, these comparisons illustrate two aspects of SKR: its recipe can be instantiated across different tasks, and task-native re-expression can improve effectiveness and efficiency. The Qwen3.5 results show that these patterns also appear on a more recent model.

The revised PDF describes these baselines in Section 4.2. The new results are in Section 5.1.2.

W2, Q3.2: MMDocRAG data.

We removed development images appearing as evaluation gold answers and did not use evaluation queries for training. To demonstrate SKR’s performance more strictly, we also test the MMDocRAG-trained model on DocVQA-2020 and SciMMIR (Table 11), demonstrating cross-dataset transfer.

W2, Q4: SKR results of Qwen2-VL on localization tasks

As discussed in Section 5.1.1, Qwen2-VL was trained for localization during pre-training and therefore has strong capability. Our results suggest that it does not fully realize this capability under the task-specific instruction. Multi-turn interaction helps elicit it. SKR’s main benefit in this setting is improved efficiency. With an IoU of 0.726, only 0.013 and 0.015 lower than the multi-turn base model and SFT, SKR reduces inference time by at least 76% and 25%.

W2: Small test sets, no variance estimates, and reliability.

Although small, the financial test sets contain high-quality practical example and directly measure performance in realistic application scenarios.

Our reported results are based on three runs. The observed differences between the base and SKR models are statistically significant under a t-test (p<0.05p < 0.05) in cases we claim performance difference. The run-to-run variation is generally within approximately ±0.5 points. Since the performance differences are substantially larger than this variation, we did not include variance values in the presentation.

To support reproducibility, the appendix documents the experiment implementation in detail.

Q5: Latency comparison and training costs comparison.

In the Table 2 localization experiment, self-annotation took 166.2 minutes for both SKR and SFT. SKR and SFT training take 232.5 and 306.8 minutes, respectively, giving totals of 398.7 and 473.0 minutes. SKR trains faster than SFT since SFT optimizes a token-level generation objective, and SKR predicts the task output directly through a task-specific head. Thus, SKR improves efficiency in both training and inference.

The annotation and training costs are now included in Appendix C.5.

I would like to thank the authors for the responses. The additional experiments strengthen the empirical contribution of SKR, but they do not fully address my central concerns regarding methodological novelty and whether the observed improvements specifically result from knowledge re-expression rather than conventional task-specific adaptation. In particular, the new results show that SKR achieves performance comparable to a contrastively trained LLM2Vec variant (MRR 0.736 vs. 0.735 on MMDocRAG). While the two methods use different supervision settings, these results do not clearly establish the unique contribution of the proposed knowledge re-expression mechanism beyond existing representation learning approaches. I maintain my current rating.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.