- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes Rilo, which is a method for multi-hop retrieval. It learns a set of relation operators and combines them to predict what evidence should be searched next. These predictions are used to generate search queries. A controller decides which query to use or when to stop searching based on the expected benefit and remaining search budget.
Critical strengths
-
The paper proposes a method that uses learned relation operators to plan the next search. It combines multiple relation steps into a single search query and updates the plan when new documents are retrieved.
-
The paper includes matched controls, ablation studies, experiments with larger document collections, and cross-dataset transfer. It also evaluates the learned operators on relation combinations not seen during training, which helps assess their ability to generalize.
Critical weaknesses
- The proposed method does not show a clear overall improvement over Direct-continuous in the 96-token setting, and its lower performance on MuSiQue is statistically significant. In the 32-token setting, the improvement on MuSiQue is also not statistically significant. These results raise questions about when the learned operators are actually useful.
- The improvements in answer F1 are also small, and RILO performs worse than EfficientRAG on MuSiQue. Given the number of models and the complexity of the proposed method, the additional benefits over simpler approaches are not sufficiently demonstrated.
Questions for authors
- What explains the different results against Direct-continuous across the two query-length settings, particularly the reversal on MuSiQue? Is there evidence that separates errors in operator composition from errors in query generation?
- The short-query protocol specifies a 32-token output limit for all matched methods, but Direct-matched and Direct-continuous report the same scores as in the 96-token comparison (Tables 9, 10, 16). Were these results obtained from separate evaluations? Please clarify why the scores are identical.