- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes GA-ICL, a demonstration-selection method for hallucination detection using frozen language models. The method extracts hidden representations from labeled training examples and learns a lightweight projection network together with class prototypes. Its objective combines a modified proxy-anchor loss with a local manifold-based loss. Examples nearest to the learned momentum prototypes are selected as a fixed demonstration set and reused across test queries. Experiments on FEVER and three HaluEval tasks show improvements over several ICL selection baselines in many model–task configurations, particularly dialogue and summarization. The paper also provides component ablations, dimensionality analyses, and evaluations on larger models.
Critical strengths
- The problem is relevant and the approach is practically motivated. Improving demonstration selection while keeping the underlying LLM frozen is a useful direction, especially when updating the full model is impractical. The paper explicitly distinguishes training a lightweight selection module from a training-free approach.
Critical weaknesses
- The central equation appears inconsistent with its stated objective. Equation 1 seems to push positive pairs apart and pull negative pairs together. The authors should clarify the implemented loss.
- The necessity of manifold modeling is insufficiently established. Prototype learning and dimensionality reduction also contribute substantially. Comparisons with simpler supervised projections under matched dimensions and training budgets are needed.
- The statistical protocol is unclear. The paper describes deterministic selection and decoding but reports five-seed results. The sources of randomness and the statistical reliability of smaller gains require clarification.
Questions for authors
- Is Equation 1 a typographical error? What is the exact loss used in the implementation?
- How does GA-ICL compare with simpler supervised projection methods under matched dimensions and training budgets?
- What changes across the five seeds for each method, and are the gains statistically significant?
4 comments
Response to Reviewer WTsa(W3/Q3)
W3 / Q3: Statistical protocol; what changes across the five seeds
We thank the reviewer for pressing on this, and we have corrected the manuscript. Given a demonstration order, selection and decoding are deterministic: rerunning a fixed configuration reproduces accuracy exactly.
The five "seeds" in Table 3 shuffle the order of the two selected demonstrations (positive/negative versus negative/positive) for every method. For GA-ICL and AdaICL they also vary the training seed. The reported therefore reflects sensitivity to demonstration order and, where applicable, training seed, and not run-to-run noise.
With two demonstrations there are only two orders, so we now report both explicitly. Order sensitivity is large and method-dependent (Table: Demonstration-order sensitivity): on Qwen3.8-27B, BM25 on QA moves by 6.2 points between orders, while GA-ICL moves by 0.3; on Dialogue the swings are 1.0 (BM25) and 1.0 (GA-ICL).
GA-ICL's worst order exceeds BM25's best order in both tasks (81.3 vs. 76.8 on Dialogue; 89.5 vs. 86.0 on QA).
Table: Demonstration-order sensitivity on Qwen3.8-27B (single run per order, accuracy %). Order 1 and Order 2 are the two possible orders of the two selected demonstrations.
| Task | Method | Order 1 | Order 2 | Swing |
|---|---|---|---|---|
| Dialogue | BM25 | 75.8 | 76.8 | 1.0 |
| GA-ICL | 81.3 | 82.3 | 1.0 | |
| QA | BM25 | 79.8 | 86.0 | 6.2 |
| GA-ICL | 89.8 | 89.5 | 0.3 |
A note for Reviewer WTsa and the AC.
This reply says the five seeds change the demonstration order for every method and also the training seed for GA-ICL and AdaICL. The reply to me says the seeds only change the order, with no mention of training seeds. The order table here also doesn't match Table X in the reply to Reviewer rMup: there, BM25 is shown at its worse order and GA-ICL at its better one in every shared cell, even though the text says all methods use the same order. Since the question was about whether the reported numbers match the described setup, I think these differences are worth a closer look.
Discussion about the experiment results
Seconding Reviewer snq717, this is the same thing I asked in W3. On QA, Order 1 gives GA-ICL a 10.0-point lead (89.8 vs 79.8); Order 2 gives 3.5 (89.5 vs 86.0). So which is in Table 3? If the table in the other thread puts BM25 at its worse order and GA-ICL at its better one, the reported margin isn't a same-order comparison at all. That's a reporting question, not a statistical one.
Two replies of authors also disagree on whether the five seeds include training seeds. With only two orders and deterministic decoding, that's not a detail, it decides what the ±SD means. I would like to know whether authors could provide a detailed discussion.
Response to Reviewer WTsa(except W3/Q4)
We thank the reviewer for the careful reading. Below we address the three weaknesses and the four questions together.
W1 / Q1: Eq. (1) and the exact loss used
Eq. (1) as typeset is a transcription error and not a typo in a single symbol. The signs of the exponents were wrong. The implemented loss matches the textual description: same-class embeddings are pulled toward their proxy and different-class embeddings are pushed away. The error arose when we adapted the proxy-anchor loss from its cosine-similarity form to Euclidean distance, where the signs flip. No experiment is affected, so no result changes and nothing needs to be rerun. The corrected loss is
\mathcal{L}_{PA} = {} & \frac{1}{|\Theta^{+}|} \sum_{\theta_q\in\Theta^{+}} \log\\!\Bigg( 1+\sum_{Z'\in \mathcal{Z}'^{+}_{\theta_q}} \exp\\!\big(\alpha(\lVert h_\theta(Z)-\theta_q\rVert_2-\epsilon)\big) \Bigg) \\\\ &+ \frac{1}{|\Theta|} \sum_{\theta_q\in\Theta} \log\\!\Bigg( 1+\sum_{Z'\in \mathcal{Z}'^{-}_{\theta_q}} \exp\\!\big(-\alpha(\lVert h_\theta(Z)-\theta_q\rVert_2-\epsilon)\big) \Bigg). \end{aligned}$$ This is combined with $\mathcal{L}_{manifold}$ (Eq. 2) as a sum. We have fixed Eq. (1) and re-checked the other equations in Section 2. ### W2 / Q3: Necessity of manifold modeling; comparison with simpler supervised projections at matched dimension and budget We added a compressed proxy-only ablation (Table: Compressed proxy-only vs. GA-ICL). Proxy-only is the simpler supervised projection: the same projection network and prototypes, trained with the proxy-anchor loss alone, at the same $Z'$ values as Table 10, on the same validation split, and with an identical 200-epoch budget as full GA-ICL. Full GA-ICL outperforms it at all 13 matched dimensions, by **1.8–3.3 points on Qwen3-4B QA** and **2.6–5.5 points on Mistral-7B QA**, and the gap widens as $Z'$ shrinks. Proxy-only also improves with compression (Qwen +2.4, Mistral +5.7 across the range), so part of the dimensionality gain comes from supervised compression alone, which we now state explicitly. The remainder, and a growing share at lower dimensions, is attributable to the manifold loss. On Mistral-7B QA, proxy-only is close to the random-projection control (Table 10), so most of the learned gain over random there comes from the manifold objective. We have scoped the claim in Appendix C.2 that proxy-anchor learning alone yields a substantial improvement to Qwen3-4B QA. In the random-projection control, both the projection and the prototypes are randomly initialized and then fixed, with no training. We did not run other supervised projections (e.g., LDA or PCA with a classifier head) and note this as a limitation. **Table: Compressed proxy-only ablation versus full GA-ICL at matched prototype dimensionality $Z'$ (validation accuracy, %).** Both variants use the same validation split and an identical 200-epoch training budget; proxy-only is trained with $\mathcal{L}_{PA}$ only, GA-ICL with $\mathcal{L}_{PA}+\mathcal{L}_{manifold}$. GA-ICL values are those of Table 10. $\Delta$ = GA-ICL $-$ proxy-only. | Model / Task | $Z'$ ($\times 10^3$) | Proxy-only | GA-ICL | $\Delta$ | |---|---:|---:|---:|---:| | **Qwen3-4B QA** | 2.5 | 65.2 | 67.2 | +2.0 | | | 2.3 | 64.8 | 67.5 | +2.7 | | | 2.1 | 66.3 | 68.1 | +1.8 | | | 1.9 | 66.5 | 68.7 | +2.2 | | | 1.7 | 67.1 | 69.5 | +2.4 | | | 1.5 | 67.6 | 70.9 | +3.3 | | **Mistral-7B QA** | 3.8 | 50.7 | 53.4 | +2.7 | | | 3.6 | 52.3 | 56.2 | +3.9 | | | 3.4 | 53.8 | 56.4 | +2.6 | | | 3.2 | 54.6 | 58.7 | +4.1 | | | 3.0 | 55.3 | 60.1 | +4.8 | | | 2.8 | 55.9 | 61.3 | +5.4 | | | 2.6 | 56.4 | 61.9 | +5.5 |