acceptodds
Under review as a conference paper at ICLR 2027

TEFS: REJECTED-RECORD NONINTERFERENCE IN TYPED HYBRID DOCUMENT QA

Abstract

Hybrid document-QA runtimes select among records returned by readers, structural parsers, visual models, and symbolic operators. We identify a missing integrity boundary: a completed record that will later be rejected can still influence candidate-dependent computation and change which admissible record is ultimately selected. We formalize rejected-record noninterference (RNI) and characterize it through invariance of the selector-visible view to rejected records. TEFS enforces completed-record admissibility before this view is formed, preventing rejected candidates from affecting downstream features, rankings, or selector state. Natural replay and paired live-return experiments show that rejected records can alter admissible selections even when they are never selected themselves, while controlled return faults distinguish this boundary from call-time authorization and partial post-execution checks. Under its trusted-request and trusted-wrapper assumptions, TEFS guarantees contract-consistent participation rather than semantic correctness, contract inference, or deployment-wide fault prevalence.

Primary area:
interpretability and explainable AI
Keywords:
retrieval, information extraction, QA

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_YozROpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
3
Read review
Paper summary

(A quick disclaimer that I found the paper very difficult to follow. I have experience on QA systems over structured data, but I am not an expert in this subarea. I believe that the paper, though almost flawless syntactically, is poorly written, or at the very least poorly self-contained. I will give more details later.) The authors address the problem in a QA system of a rejected candidate result potentially affecting the selection of a candidate to return post-rejection of the candidate. Under specific policies on the answer generation, the authors propose a method to avoid this influence of rejected candidates on accepted candidates. Experiments on existing QA benchmarks show that rejected candidates can influence (thereafter) the final outcome in practice.

Critical strengths

S1: The core idea of avoiding rejected candidates affecting outcomes is a useful axiom / design principle to build better QA systems. The core idea here is intriguing.

S2: Examples are provided that help to partially understand the ideas.

S3: The experiments establish empirically that rejected candidates can affect the final outcome.

Critical weaknesses

W1: Despite some effort, I found the paper exceptionally difficult to follow. I believe this to be due to the paper either being poorly written (though flawless syntactically) or not adequately self-contained. I will provide some examples from the first two paragraphs of the first section:

  • "Typed hybrid document-QA" What is this?
  • "heterogeneous producers" What makes producers heterogeneous? What are these producers?
  • "intermediate selector-visible computation" What is this? What are the selectors? They select the candidates?
  • "We study this post-execution, pre-selection integrity boundary." What is the "execution" of "post-execution"? The rejection of a candidate?
  • "A table specialist instead copies the visible row label 6" What role does the "table specialist" have here? What makes a row label "visible"? What table is this? (Later I realise that it is a cell in the table of Figure 1(a).)
  • "Its trusted-wrapper trace" What is a wrapper here? What does "Its" refer to?
  • "Its trusted-wrapper trace records copying rather than the requested difference." I cannot actually parse this sentence.
  • "conform to the typed request and fixed policy" What are these? (Figure 1 helps here somewhat.)

More generally:

  • The problem is not described in a self-contained way: what is the input, and what output is expected?
  • Dense terminology is used but never described.
  • Examples are partially helpful, but without the previous two points, do not alone allow for understanding.
  • Results like "In the separate schematic selector witness of Figure 1C, admissible reader c1 and calculator c2 have base scores .78 and .74" have no context. Where do these numbers come from?

Similar issues persist throughout the paper.

W2: A more philosophical point, but regarding "This example concerns causal influence over selection, not whether the uninfluenced winner is factually correct". Understandably, rejected candidates should not have a specific influence. But does this outweigh the priority of getting a correct answer?

W3: Theorem 1 and the proof seem informally stated. It is hard to understand what technical claim is made / established.

Questions for authors

Q1: Can you explain in simple terms how your approach works? For example, could you take a single illustrative benchmark example and explain the steps?

Reviewer_ViHrOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper addresses the domain of hybrid question-answering, where candidate answers can be generated from multiple data sources. When a selector scores candidate answers using a consistency reward strategy (where a consistency reward is added if two candidates provide the same answer), incorrect and invalid candidate answers may cause the overall score of identical incorrect candidate answers to be inflated, leading the selector to ultimately choose the incorrect answer. The authors identified this issue across multiple datasets and resolved it by adding a simple admission check prior to scoring.

Critical strengths
  1. The research question is grounded in practical realities and has potential broader implications for systems that adopt the proposed consensus mechanism.
  2. The experimental design is relatively comprehensive, and the observed conclusions are reliable.
Critical weaknesses
  1. The paper is poorly written; technical terms are introduced without prior definition, and I cannot understand it without AI assistance.
  2. The improvement strategy relies heavily on a specific selector design. If the selector did not have a consistency reward strategy, this problem would not exist; therefore, the applicability of this improvement is limited.
  3. The method itself is more like a general-purpose “trick”; I cannot see any substantial difference between it and existing general-purpose postcondition checking approaches.
Questions for authors
  1. To what extent does this issue have a practical impact on a broader set of candidate selectors?
  2. The paper explicitly treats a correct typed request as a trusted premise. How does TEFS actually perform when there are errors in the typed request or the operation contract itself?
Reviewer_s42ZOpenReview ↗
Rating
2
Confidence
3(−1)
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper looks at hybrid document QA runtimes where several producers each return a candidate record: VLM readers, OCR, table specialists, symbolic operators, caches. A selector then picks one. The claim is that a record which will eventually be rejected for violating the typed request contract, say wrong operation, wrong arguments, stale binding or malformed output, can still sway the selector through set-dependent features like agreement, even though it never wins itself. The authors define rejected-record noninterference, RNI. Theorem 1 says RNI holds for every selector if and only if the selector-visible view does not change when rejected records are appended. TEFS is a kernel that checks an admission predicate, Eq. 1, on each completed record before any feature is computed. In a replay of 6290 TABVERSE requests with a frozen agreement selector, letting rejected records into the features changes the selected admitted record on 82 requests. With pre-feature admission that drops to zero, and EM goes from 50.397 to 50.302. A prompted Qwen2.5-VL-3B joint selector changes five admissible winners in each of three greedy live runs. Injected cache-donor, rotated-operation and stale-binding probes separate this check from call-time authorization. The rest of the paper covers incidence of inadmissible picks by frozen selectors, transfer to DOGR and MMDocBench, a live four-producer pipeline, and controller cost.

Critical strengths
  1. The evidence hygiene is unusually good. Natural incidence, the placement replay and injected faults are kept apart throughout. The paper reports nulls and accuracy effects in both directions: 30 repairs against 36 harms, two worked traces going opposite ways, and separable selectors with zero changes. An independently coded gate reproduces the admitted view on all 6290 requests. Claims are scoped explicitly as participation integrity, not correctness. That makes the paper easy to audit, and for systems work like this that matters.

  2. Splitting selectors into candidate-separable and set-dependent, Table 3, is a clean and practical design rule. With separable scoring, masking at the end is enough. Once a selector looks at candidates jointly, as agreement voting, consensus or an LLM judge given the full set do, rejection has to happen before features are computed. That is a useful checklist item for people building multi-tool pipelines with an LLM selector.

Critical weaknesses
  1. On natural data, the pre-feature boundary is shown to beat late masking, not existing call-time authorization, and it is never shown to prevent harm. In Table 2, pre-execution authorization alone already takes legal-choice changes from 82 to 0, with the same 50.30 EM as TEFS. The paper says as much on p8: the natural replay separates the two only in feature visibility. Everything that distinguishes TEFS from authorization therefore rests on injected probes. Within the replay, the effect comes from one frozen agreement selector, and the Table 7 caption says the first five arms coincide "under this frozen selector and its four-reader tie priority". The agreement scoring is not spelled out. Both p7 traces fit a plurality vote with reader-priority tie breaking: readers answer 15/17/16/17, and the rejected specialist's 16 ties the count. My conjecture is that most of the 82 changes are this one-extra-vote kind, which needs no theorem to predict. Separable selectors give zero changes, and the calibrated consensus row collapses to separable because tuning picked weight zero. The only learned-model case is a prompted 3B judge: five changed winners in each of three greedy runs over the same 629 requests, plus 8, 2 and 10 in three temperature 0.2 runs. That signal is real but small and highly correlated. The live ledger lists only failed or malformed outputs and injected wrapper failures among rejected returns. So I read the live influence as coming from malformed outputs, which an ordinary shape check placed before the selector would also remove. That is my inference, since no reason breakdown is given. Accuracy goes the other way: the late arm is +0.095 EM with an interval crossing zero, and early admission repairs 30 but harms 36. The paper is explicit that it claims influence, not accuracy, and that Figure 1C is a schematic witness, yet its motivating example still ends with TEFS choosing 6.839 over 6.840. The live runs contain no natural post-return binding mismatch, and all 106 warm cache hits match, so cache misbinding and stale versions appear only as injected faults. Even there, Table 8 shows an operation-plus-arguments postcheck stops every rotated-operation and cache-donor fault. Only the nine held-ID stale mutations need the full binding check. The threat model sensibly targets benign integration faults rather than adversarial payloads. Then the paper owes evidence that such faults occur and matter in a natural pipeline.

  2. Accounting and assumption gaps weaken the empirical base. The cache-donor constructor gives 3774 minus 692 minus 48 = 3034 successful, well-formed direct-operation records, and the replay counts imply exactly 3034 admitted specialist records. Yet typed admission in panel a of Table 5 selects 3082 specialists, the conformance audit admits "3,082/3,082 TABVERSE typed records", and the shape/status row of Table 8 admits 4445 = 3082 + 1363. These close exactly if 3082 counts every successful direct-operation return, malformed ones included. Then either panel a and the conformance audit skip the output-shape check that TEFS and the independent gate use to reject the 48, or panel a uses a different ledger from the RQ1 replay. The paper does not say which, and the answer decides whether panel a measures TEFS or a weaker operation-only check. On trust, the supplied labels mark all 20 sampled partition requests as not matching the question. The full-ledger scan finds that all 629 partition reader prompts ask for first and last cell values, and 19 locate cell requests have a blank target, so 648 of 6290 requests carry structural flags. The A/B annotation exports are byte-identical, so there is no independent agreement. To its credit, the paper shows none of the 82 winner changes fall in the flagged strata, so RQ1 survives. The flags still bear on the 1363 out-of-policy records and the panel a utility numbers, since partition is excluded from the direct-extraction policy.

Questions for authors
  1. Which admission check does panel a of Table 5 apply, and is its ledger the same 6290-request pool as the RQ1 replay? The counts close only if its 3082 typed records include the 48 malformed direct-operation returns that the TEFS arm rejects.
  2. Call-time authorization alone removes all 82 natural winner changes in Table 2. Is there any natural, non-injected case where TEFS changes a selection that authorization leaves unchanged?
  3. What are the rejection reasons for the successful rejected returns exposed in the live off arm? If they are all malformed outputs, a plain shape check before the selector would match TEFS on the live runs.
  4. Can you show an RNI violation that lowers accuracy or safety in a natural pool, with a learned joint selector and natural binding faults?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.