- Rating
- 1
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 3
Read review
Paper summary
(A quick disclaimer that I found the paper very difficult to follow. I have experience on QA systems over structured data, but I am not an expert in this subarea. I believe that the paper, though almost flawless syntactically, is poorly written, or at the very least poorly self-contained. I will give more details later.) The authors address the problem in a QA system of a rejected candidate result potentially affecting the selection of a candidate to return post-rejection of the candidate. Under specific policies on the answer generation, the authors propose a method to avoid this influence of rejected candidates on accepted candidates. Experiments on existing QA benchmarks show that rejected candidates can influence (thereafter) the final outcome in practice.
Critical strengths
S1: The core idea of avoiding rejected candidates affecting outcomes is a useful axiom / design principle to build better QA systems. The core idea here is intriguing.
S2: Examples are provided that help to partially understand the ideas.
S3: The experiments establish empirically that rejected candidates can affect the final outcome.
Critical weaknesses
W1: Despite some effort, I found the paper exceptionally difficult to follow. I believe this to be due to the paper either being poorly written (though flawless syntactically) or not adequately self-contained. I will provide some examples from the first two paragraphs of the first section:
- "Typed hybrid document-QA" What is this?
- "heterogeneous producers" What makes producers heterogeneous? What are these producers?
- "intermediate selector-visible computation" What is this? What are the selectors? They select the candidates?
- "We study this post-execution, pre-selection integrity boundary." What is the "execution" of "post-execution"? The rejection of a candidate?
- "A table specialist instead copies the visible row label 6" What role does the "table specialist" have here? What makes a row label "visible"? What table is this? (Later I realise that it is a cell in the table of Figure 1(a).)
- "Its trusted-wrapper trace" What is a wrapper here? What does "Its" refer to?
- "Its trusted-wrapper trace records copying rather than the requested difference." I cannot actually parse this sentence.
- "conform to the typed request and fixed policy" What are these? (Figure 1 helps here somewhat.)
More generally:
- The problem is not described in a self-contained way: what is the input, and what output is expected?
- Dense terminology is used but never described.
- Examples are partially helpful, but without the previous two points, do not alone allow for understanding.
- Results like "In the separate schematic selector witness of Figure 1C, admissible reader c1 and calculator c2 have base scores .78 and .74" have no context. Where do these numbers come from?
Similar issues persist throughout the paper.
W2: A more philosophical point, but regarding "This example concerns causal influence over selection, not whether the uninfluenced winner is factually correct". Understandably, rejected candidates should not have a specific influence. But does this outweigh the priority of getting a correct answer?
W3: Theorem 1 and the proof seem informally stated. It is hard to understand what technical claim is made / established.
Questions for authors
Q1: Can you explain in simple terms how your approach works? For example, could you take a single illustrative benchmark example and explain the steps?