- Rating
- 2(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper analyzes how knowledge stored in the base model’s parameters, provided in the input context, and encoded in LoRA adapters generated by SHINE interacts. On TRCBench, Qwen3-8B preferentially follows the input context over base-model or LoRA knowledge when the sources conflict. On MHRBench, multi-hop reasoning involving knowledge encoded in the generated LoRA adapters is unreliable.
Critical strengths
-
This paper shows that injecting document knowledge through a hypernetwork does not necessarily enable the model to integrate it with its existing parametric knowledge.
-
The experiments are insightful. The dual-chain evaluation distinguishes genuine multi-hop reasoning from answer selection based on surface-level relation cues. Furthermore, the knowledge-swapping experiments reveal source preferences, while the representation analysis and hidden-state interventions show that these preferences can be steered.
Critical weaknesses
-
The experimental setting and scope are narrow. This paper only focuses on a specific hypernetwork, SHINE, and all experiments are conducted with this model. Thus, generalization to other hypernetworks is not discussed.
-
I am concerned about overclaiming because the paper seems to suggest that “all hypernetworks suffer from the reported problems.” Let’s imagine an advanced hypernetwork, SHINE++, trained to handle knowledge conflicts. It might be robust to these problems, which would refute the paper’s general claim. I think the biggest potential contribution from analyzing only SHINE would be to propose a new hypernetwork that addresses the identified problems.
Questions for authors
-
As I mentioned above, have you considered proposing an improved hypernetwork that is robust to the identified problems? Collecting and synthesizing existing datasets to develop such a model could have a significant impact on the research community.
-
Do the main findings still hold in a setting where knowledge is continually injected and modified? Based on the paper’s observations, what potential directions for improvement could be proposed?
2 comments
Thank you for conducting the additional experiments. They have addressed my concerns to some extent, and I have therefore raised my score.
We sincerely appreciate the reviewer’s insightful and positive feedback! We hope the following responses will fully address your concerns.
For Weakness 1, we have provided clarification and new experiment results in our official comment titled “Experiment on Other Models.”
For Weakness 2 and Question 1, we would like to clarify that knowledge conflicts and multi-hop reasoning are two representative settings through which we investigate a broader, more fundamental question: how knowledge interacts within hypernetwork-based continual learning systems. To the best of our knowledge, our work is the first to systematically study this question, and we chose these two settings as concrete starting points.
We agree that performance in these settings could potentially be improved by incorporating more examples of knowledge conflicts and multi-hop reasoning into hypernetwork training or by fine-tuning on such examples. However, these improvements alone would offer limited insight into the underlying mechanisms of knowledge interaction. Our goal is to understand these mechanisms and identify the causes of failure, thereby informing more principled improvements.
In these systems, knowledge interaction involves the interplay among the base model’s weights, the generated Δ weights, and the KV cache. Fully characterizing these interactions and developing a principled solution to the identified limitations remains a challenging research problem. We view our study as an initial step toward this goal.
For Question 2, to the best of our knowledge, no model with publicly available weights currently supports continual knowledge injection through ongoing updates to model weights without incorporating the textual interaction history into either the context or the hypernetwork’s input. If such a model becomes available, we would be interested in investigating knowledge interaction in that setting.