- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper constructing static chunk indices from question–evidence supervision. QESChunker constructs document-specific chunk indices by optimizing over human-annotated or LLM-generated question–evidence pairs, without access to held-out test queries. The main idea is turning chunking into a supervised combinatorial optimization problem. The method show good emperical results.
Critical strengths
- The paper propose an interesting and novel method.
- It provide clear theoretical analysis.
- The experiments show good improvements over diverse baselines across multiple datasets.
Critical weaknesses
- The hit-gated coverage objective assumes that if a chunk contains at retriever's top- hit atomic block, the chunk has a chance of being retrieved. But retrieval is eventually performed on chunk-level. Then there is a mismatch. Is there any evidence to support this assumption? I think this is important because this is an important assumption supporting the design.
- There are some predefined hyperparameter. It's unclear how sensitive it is to the hyperparameter value change. For example, The non-overlapping DP algorithm requires the number of chunks , rather than determining it adaptively based on document structure and evidence distribution. 3.The overlapping chunk selection formulation measures indexing cost by the total number of atomic blocks But I think there can be some mismatch to actual cost which is highly depens on token counts. I think the setting is a bit unrealistic.
- The experiments primarily evaluate complete-evidence recall. While this metric does not directly show improvements in answer quality. Especially it's unclear to me if the assumption if a chunk contains at retriever's top- hit atomic block, the chunk has a chance of being retrieved is robust or not. Is the retriever doesn't work, complete-evidence recall doesn't mean anything to me.
- All experiments use the same embedding–BM25 hybrid retriever. It remains unclear if we change the retreiver, if this method still works
Questions for authors
I think it will be helpful to report end-to-end computational cost. Generating Question–Evidence pairs can be expensive.
2 comments
Thank you for the response. But I think my main concern(Critical Weaknesses listed above) remains unresolved. I will keep my score.
Response to Reviewer yhrM
We thank the reviewer for comments.
Q End-to-end computational cost
In the experiments, we generate 5N Question–Evidence pairs for each article, where N denote the number of atom blocks of that article. The medium number of atom blocks of qasper, peerqa and privacyqa is about 400, so we generate about 80 Question–Evidence pairs in medium. Using API of qwen-3.7 flash with 16 concurrency, we need several minutes. Actually, we also test the relation between the number of Question–Evidence pairs and the recall in both nonoverlap and overlap assumption in QASPER. The result is as follow.
We find that in 0%(only structural information), the recall is very near paragraph chunker. Between 0% and 40% Question-Evidence pairs, the recall is fluctuating. However, above 40%, recall have a obvious increasing and in 100% it has about 2.25% earning against 0%.
We also find that between 0% and 40% Question-Evidence pairs, the recall is fluctuating. And however, above 40%, recall have a obvious increasing and in 100% it has about 2.13% earning against 0%.