- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper introduces four different validity measures to distinguish between the value of information provided to a model for a specific task and the value of an intervention. The validity claims: 1) Opportunity Gap, 2) Instrument Validity 3) Attribution and 4) Decision Validity, provide different sources of signal that are used to understand the utility of information across 8 tasks with the key finding showing oracle information value being the most informative and robust compared to attribution which can change. The paper also conducts a prospective test across two datasets and shows that information value, shortfall attribution and intervention value are different quantities and should be factored in as opposed to relying on one value.
Critical strengths
- The paper introduces and tests the diagnostic framework across a wide range of different tasks and provides ablations across models of different sizes.
- The prospective test is well-grounded but could be improved in its clarity and presentation.
Critical weaknesses
W1: The significance of the framework presented in the paper in contrast to existing approaches remains unclear
Throughout the work, several different metrics that are designed to diagnose the value of information provided or the value of intervention i.e. Opportunity gap, instrument validity, attribution, decision validity, scope, regret, stability etc. It is still unclear what the overall takeaway of these metrics are and how they complement existing work that already report versions of these metrics. It is common practise in RAG benchmarks to report a no-context baseline, an oracle and a retrieval baseline. It remains unclear the significance the metrics this work introduces towards offering new insights that existing approaches already cover. How do the findings of this work translate to how we should evaluate models and how we should select intervention policies -- these questions remain open yet the work sets out to address them. Table E.1. presented in the Appendix provides no citations for any of the prior methods making it hard to do a comparison to existing approaches.
What would help address this concern? In the discussion of the findings, provide a clear explanation of how the results presented differ from existing evaluation approaches and what novel information the distinction between value of information and value of intervention provides. It is important to clarify what the new approach adds that explicitly differs from what existing approaches already report and the practical significance.
W2: Limited error analysis
The results presented on BigCodeBench are interesting but no error analysis is provided to help understand why this behaviour occurs specifically on this dataset or model and whether it generalises to other datasets of similar properties.
What would help address this concern? Beyond just providing the quantitative values for each validity, a futher analysis on BigCodeBench is needed to explain the observed invalid recovery instrument.
W3: A justification on the choice of experimental setup is lacking
The paper does not provide a principled justification as to why certain datasets were specifically used for this task. Most of the tasks used i.e. information retrieval tasks like BrowseComp, HotpotQA are designed to require retrieving from external sources hence some of the findings are not suprising i.e. BrowseComp is designed to be intentionally difficult to solve parametrically.
What would help address this concern? Provide a clear justification on the choice of tasks selected and how it potentially affects the intepretation of the results.
Questions for authors
- The presentation of the four validity claims can be better improved for readability and clarity. Here are a few suggestions I would make:
- Provide a reference or a worked example to show how it relates to each metric e.g. a question from FEVER.
- Use consistent naming of the metrics from when first introduced i.e. the headings in L124, L139 are mismatched from the claim names in Table 1, even though they are written to provide definitions. I would suggest to keep it consistent with the table.
- Reuse notations across the different claim definitions instead of introducing new notations for the same concepts e.g. mean utility without supplied information can be standardised in both Opportunity and Instrument Validity.
-
A clear experimental section that provides details on the choice of datasets used or models evaluated is missing. Were the datasets chosen because they have certain properties that allow you to measure the validity claims, how do you controll for datasets that models may have already been exposed to at pretraining or are saturated e.g. GSM8K, do you do an initial ablation to test for questions that models can already solve prior to providing oracle information? It is interesting to note that for I would suggest to the authors to provide a clear experimental section that provides details of the experiments. L087 - L090 provides the models used but this is misplaced in the introduction.
-
Use consistent naming of the validity terms across the paper. Opportunity shifts to Headroom in Table 2 while uses a different naming when presented in Section 2. I also noted areas wher the term Information Gap is used to refer to Opportunity. I would suggest standardising the use of terms in the paper.
-
Isn't the instrument validity is presented as a numeric value but reported as a binary label between VALID and REJECTED in Table 2? If so, the definition in Line 144 should be clarified to reflect the set options i.e {VALID, REJECTED}. Also in the appendix there is a lack of consistency in the usage of the labels as INVALID is now used, is INVALID and REJECTED the same thing?
-
Are the interventions generalisable and what are is practical utility of the defined action menu beyond the tasks included in the paper?