- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper studies the robustness of ordering in training-free LLM embeddings. It reveals an important gap that reordering content inside a query will shift retrieval rankings badly. The proposed method, ASEmbed, extracts attention-sink and projected-value signals from first pass, using them to cut input into blocks, and then reruns the model with bidirectional attention across blocks. On opensource retrieval tasks and internal reordering tests, it beats training-free baselines.
Critical strengths
- The robustness of ordering is a real but mostly ignored problem, and the controlled reordering test are solid.
- The interpretability findings and the following two-pass pipelines are clear stories and well motivated.
Critical weaknesses
- No ablations for the core claim that ASPV signals find good block boundaries. The paper doesn't tests random blocks of the same size, which is a cheap but strong ablation for claiming the core design.
- The paper doesn't say where the calibration set comes from.
- Do corpus documents also need two passes?
Questions for authors
See Weaknesses.
3 comments
**Response to Reviewer do7t(Part 2 / 2)**
W2. Where does the calibration set come from, and how many samples are used in the main experiments?
A2: Thank you for pointing out that the calibration setup was not sufficiently described. In the main experiments, we sample 24 unlabeled inputs from the test split of the corresponding task for calibration. Throughout the process, model parameters remain frozen, and the attention switching layer and readout layers remain unchanged.
Figure 6(c) compares calibration sizes of 8, 12, 24, 48, and 96. Both retrieval metrics improve as the number of calibration samples increases, with diminishing gains.
W3. Do corpus documents also need two passes?
A3: Yes. Both queries and corpus documents require two forward passes. The first pass extracts the input's ASPV signals and determines its partition. The second pass applies AS-Attention and extracts representations from the selected readout layers.
When the corpus and configuration are fixed, document embeddings can be computed and cached during offline indexing. Online retrieval requires only two forward passes for each new query, after which the cached document embeddings can be searched without re-encoding the corpus.
We are happy to see from your comments that you have an overall positive view of our work, and we respectfully ask if you'd be comfortable to adjust your ratings, based on all the new experiments and justification to your questions. Thank you once again for your time and effort. We look forward to incorporating these insightful discussions into the final camera-ready version.
Dear Reviewer,
I hope this message finds you well. As the discussion period nears its end, I would like to check whether our responses have satisfactorily addressed your concerns. Please let me know if there are any additional points or feedback you would like us to consider. Your insights are invaluable, and I am eager to address any remaining issues to improve our work. I hope that the additional evidence and clarifications help clarify our contributions and support your overall assessment.
Thank you for your time and effort in reviewing our paper!
Response to Reviewer do7t(Part 1 / 2)
We thank you for your careful and constructive review and for recognizing our work. To address your questions, we have added partitioning comparison experiments and further clarified the experimental settings. Below, we provide detailed responses to all the comments:
W1. No ablations for the core claim that ASPV signals find good block boundaries?
A1: Thank you for your suggestion. We have added two types of Boundary Construction controls to the ablation study. First, we combine window sizes of 64, 128, and 256 tokens with overlap ratios of 25%, 50%, and 75%, evaluating nine sliding-window configurations. Second, we randomly shuffle the order of the ASPV-derived blocks three times to evaluate retrieval performance under random order perturbations.
The results show that ASPV achieves the highest nDCG@10 and MRR among all partitioning controls. Compared with the control that randomly shuffles the order of the ASPV-derived blocks, ASPV achieves improvements of 1.453 and 1.790 percentage points, respectively, indicating that random perturbations to block order reduce retrieval performance under the current experimental settings.
For the window-based methods, both metrics decrease as the overlap ratio increases for all three window sizes; increasing window overlap provides no performance gains. These results support the retrieval effectiveness of ASPV-guided input-adaptive partitioning on this task and its superiority over random boundary selection.