- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces Self-Knowledge Re-expression (SKR), motivated by the observation that the standard next-token prediction interface of LLMs may be poorly matched to non-generative downstream tasks. The authors formulate this mismatch as a knowledge expression bottleneck. SKR consists of two stages. First, the base model generates task-specific pseudo-targets from unlabeled inputs using its own generative behavior. Second, these pseudo-targets are used to train a task-native output mechanism together with LoRA adapters, while the original model parameters remain frozen. The paper instantiates this framework for three tasks, dense information retrieval, instruction-conditioned region localization, and imbalanced binary classification. Extensive experiments show that SKR substantially improves retrieval quality, reduces localization latency by replacing autoregressive/multi-turn generation with direct regression, and improves AUROC/AUPRC for imbalanced classification.
Critical strengths
- The method requires only unlabeled task inputs and task-level specifications, rather than instance-level human annotations or a stronger external teacher. This is practically important for private or domain-specific data, where external annotation can be costly or undesirable. The empirical improvements are also unusually large rather than marginal.
- The experimental analysis goes beyond a simple base-versus-method comparison. In particular, the comparison with self-SFT is useful because it shows that merely training on the same self-generated signals while retaining token generation is substantially weaker than moving to a task-native objective. The head-only experiments further show that attaching a new output layer without adapting internal representations is insufficient, while the external-teacher experiments explore when same-model supervision is sufficient versus when stronger supervision can help.
Critical weaknesses
- The central “knowledge expression bottleneck” claim is not fully isolated experimentally. Full SKR jointly trains LoRA adapters and task-specific output heads, so the gains may partly come from improved task-specific representations rather than only from changing the output interface. The head-only ablation actually suggests that internal representation adaptation is important. A more controlled baseline would help distinguish “re-expression” from standard self-training or representation learning.
- The evidence for broad task generality is still limited. The three tasks require quite different task-specific prompts, output heads, losses, and post-processing procedures. Therefore, SKR currently appears more like a general adaptation recipe than a unified task-general method. In addition, several experiments use relatively older base models or published baselines trained under different settings, making some of the broader comparisons less controlled.
Questions for authors
see weaknesses
3 comments
We appreciate your thoughtful review and your recognition of SKR’s annotation-free setting and the value of our ablation comparisons. In response to your concerns, we brought together experiments from different sections of the paper to clarify how we isolate SKR’s components and knowledge sources step by step. We also clarified that SKR is a general adaptation recipe with task-specific configurations.
In our responses to other reviewers, we also added results for Qwen3.5-9B across the three tasks, together with LLM2Vec and SELF-GUIDE baselines, to strengthen the evaluation. All additional results and explanations have been added to the revised PDF. If you are interested in any of these results, please feel free to ask, and we would be happy to provide further details.
If you have any other remaining concerns, please do not hesitate to let us know. We would be happy to address them. If our rebuttal has addressed your concerns, we would be grateful if you would consider updating your evaluation.
Thank you again for your time and thoughtful review. As the rebuttal deadline is approaching, we wanted to follow up on our response and the additional experiments.
Another reviewer has acknowledged our additional experiments, and we appreciate that they have raised their rating. If our response and new results have addressed your concerns, we would be grateful if you would consider updating your evaluation as well.
Thank you for your careful and helpful feedback. We have updated the PDF in response to the reviews. Section 6 expands the discussion of related work. Section 4.2 introduces extra baselines. Section 5.1.2 reports the Qwen3.5’s results on public-datasets and compare SKR with the added baselines. Appendix C.5 adds self-annotation, training, and inference cost information. Appendix D.4 reports classification results using continuous yes/no token probabilities and hard labels.
We respond to your specific concerns below.
Weaknesses
W1: The central “knowledge expression bottleneck” claim is not fully isolated experimentally.
We will clarify this point more precisely. The expression bottleneck does not mean that output heads alone determine task performance. Rather, as you point out, task-specific output heads must be matched with suitable intermediate representations to be effective.
Our paper includes detailed analyses of both the components of SKR and the source of its training signals. We are sorry that these results are distributed across different sections of the paper due to length limitations. Here, we bring the analysis together to clarify the comparisons. We have also added two baselines, LLM2Vec and SELF-GUIDE. These comparisons provide further evidence on whether external supervision is necessary and how the expression mechanism affects performance. We present and discuss the additional results in detail in our response to your next question.
Supervision source compairson (Section 5.2.2 and Figure 3). We replace self-generated targets with GPT-4o-generated targets while keeping the adapted model and procedure fixed. Self-generated targets perform comparably in retrieval, though GPT-4o improves LLaVA’s Recall@5 from 0.665 to 0.685. In localization, GPT-4o targets help LLaVA and Phi-3-VL, while Qwen2-VL benefits more from its own targets. Thus, external supervision can help when self-generated signals are weaker, but is not always necessary.
Adaptation components (Section 5.2.3, Tables 1–3 and 7, and Appendix F). We compare the unadapted base model, self-SFT (base + LoRA), head-only adaptation (base + task-native head), and SKR (base + LoRA + head). Self-SFT and SKR use the same self-generated targets. SKR generally improves performance and efficiency for three tasks, although they differ in both output mechanism and training objective. These comparisons show that, based on the model’s own training signals, re-expressing its knowledge through a task-native output mechanism and adapted representations can substantially improve task performance.
Together, these comparisons highlight the value of self-generated signals and task-native knowledge re-expression, distinguishing SKR from conventional task-specific fine-tuning.
W2: The evidence for broad task generality is still limited.
As stated in Lines 63–69, we present SKR as a general adaptation recipe with task-specific instantiations, rather than as a single task-independent architecture or objective. Its main benefit is to replace the need for instance-level task annotations with a task-level configuration. Across tasks, SKR follows the same procedure: it elicits pseudo-targets from unlabeled inputs using the model being adapted, converts these outputs into training targets, and jointly trains LoRA parameters and a task-native output mechanism. The prompt, output mechanism, objective, and post-processing may vary by task, but the overall adaptation procedure remains the same. We demonstrate this recipe on three substantially different task types.