acceptodds
Under review as a conference paper at ICLR 2027

Recovering Missing Support: Complementary Generation in Budgeted LLM Auditing

Abstract

Collaborative auditing of large language models (LLMs) fundamentally relies on the availability of diagnostic actions. When no generation source proposes the action needed to expose an error, downstream selection cannot recover it: this is the missing support problem. Collaborative frameworks that focus on aggregating generated proposals can remain vulnerable to candidate anchoring, in which candidate-conditioned sources reproduce or reinforce latent errors. We introduce a two-stage collaborative auditing framework that separates action-support formation from budgeted allocation. First, candidate-conditioned generation is paired with a sealed task-only source that commits diagnostic actions before candidate materialization. Second, both routes feed a unified action pool, where shared triage and reconstruction evidence guide selection by calibrated proxy value under a fixed budget. Theoretically, finite repeated-call records leave irreducible uncertainty about the optimal support value of an allocator that knows task hit probabilities and assigns at most one additional call per task under an expected budget. Under iid tasks and conditionally binomial records, we characterize this uncertainty uniformly across budgets and establish matching minimax estimation rates over a specified call-depth range. In a controlled linear programming (LP) case study with 300 injected errors and 150 paired correct controls, task-only supplementation recovers 66 of 122 targets omitted by the candidate-conditioned route. All are purchased and repaired at , raising repair from 59.00% to 81.00%. Three additional candidate-conditioned calls reach 69.00% at matched nomination slots, with unmatched computation. On identical augmented portfolios, the complete constraint-level verification procedure improves repair by 30.33 percentage points over whole-model verification (95% interval [25.33, 35.67]). All 150 controls remain correct. In this setting, the results support an architectural principle: budgeted auditing benefits from broadening diagnostic coverage before allocating resources.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM auditing, multi-agent LLM systems, candidate anchoring, budgeted allocation, verification, missing support, diagnostic action generation, optimization modeling

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_VCf2OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies auditing failures caused by missing diagnostic checks. It supplements candidate-conditioned proposals with checks generated from the task alone, then selects and executes checks under a budget.

Critical strengths
  1. Distinguishing check coverage, selection, and successful repair provides a useful way to understand auditing failures.

  2. The controlled experiments show a substantial repair improvement.

Critical weaknesses
  1. The experiments do not isolate the benefit of task-only nomination, and the routes differ in prompting, stratification and duplicate handling, as well as candidate access. A candidate-conditioned baseline that systematically considers every requirement and uses the same stratification and deduplication is missing. The improvement could therefore come from broader coverage rather than avoiding candidate anchoring.

  2. The paper’s own ablations leave the value of its scoring and allocation machinery unclear. Stratified random nomination recovers an expected 75.77 missing targets, compared with 66 for the proposed ordering (Table 9). After the same admission filter, simple selectors match the exact allocator on target purchase. These results concern coverage and selection rather than final repair, but they leave the additional benefit of the more elaborate components unestablished.

  3. The missing-support bottleneck is partly imposed by the pipeline. The system already reconstructs all constraints three times and identifies discrepancies, but these discrepancies cannot generate checks outside the nomination pool. A baseline that directly checks those discrepancies under the same execution budget is missing. Comparing against direct regeneration or consensus-based correction would also clarify why the proposed auditing pipeline is preferable, given that full reconstructions have already been computed.

Questions for authors
  1. Do the gains persist when both nomination routes use identical coverage, stratification and duplicate handling, with comparable computation?

  2. How does the method compare with checks generated directly from reconstruction discrepancies and with consensus-based correction?

  3. How many external-baseline failures involve ambiguous task interpretations, and how do the results change when these cases are excluded?

Reviewer_9GJSOpenReview ↗
Rating
3
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper says LLM auditors can only fix errors that someone thinks to check. So the authors add a second helper that picks checks from the task alone, before it sees the answer. They test it on 300 made-up LP tasks, each with one planted error. Repair goes from 59% to 81%. But most of this gain comes from rule groups that match the planted errors. The selection step changes nothing, and the theory is never used in the experiments.

Critical strengths

The paper points to a real problem. If no auditing call asks to check the right rule, the error can’t be fixed later. The authors split the process into three steps: was the check proposed, was it picked, and did the fix work. This makes it easy to see where things fail. The benchmark knows where each error is, and it has 150 correct LPs, so harm can be measured. No harm was found (L452-455). The authors also run many extra tests and are open about the limits (L463-469).

Critical weaknesses
  1. The gain comes from the rule groups, not from the task-only model. The model does not pick the six checks on its own. A fixed procedure sorts the 30 rules into groups using keywords (L1350-1356). It then takes one pick from each of four main groups: weighted normalization, unit conversion, cross-period linkage and aggregation scope. These closely match the four planted error types (L322-323). The authors even say the split is “tailored to the controlled template system” (L1359-1360).

Table 9 (p. 35) shows the effect. With six picks, the model’s scores plus the groups find 66 of the 122 errors the other calls missed. Random picks inside the same groups find 75.77. The model’s scores without groups find only 22, while random picks without groups find 60.17. Four picks show the same pattern (61 for random, 49 for the model). Also, of the 179 errors covered at six picks, only 71 were picked directly. The other 108 were caught because a nearby rule in the same group was picked (L1812-1813). So the model does no better than random. The result mostly shows that it helps to look where the errors were planted. I would like to see a test with groups that are not tied to the error types, or with real errors.

  1. Most of the pipeline doesn’t change the result. Every supported error gets selected (κ = 1, L414). The exact knapsack, two greedy rules, a simple discrepancy rule and random choice all pick the same targets (Table 11, p. 36). The calibrated probability takes only four values (Table 10). An action gets in exactly when the 2-of-3 rebuild flag says so, on all 13,164 pairs (L1921-1922). So the real work is done by the three task-only LP rebuilds. Each is compared with the candidate, and rows where they disagree are flagged. This suggests a simple baseline that is missing: just check every row where two rebuilds disagree with the candidate. Right now, the flag can only score checks that someone else proposed (L1312-1313).

  2. The 30-point verification gain (L424-427, Table 2) is not a fair test. A row is checked only because the rebuilds already disagree with it. The row-level verifier then rebuilds that same row again. So finding the error in 244 of 244 cases (L427) is almost built in. Also, each task has exactly one wrong row (L1517). That is the best case for fixing one row at a time. Now look at the other side in Table 18 (p. 41). Whole-model checking lost 69 cases because the verifier said “pass.” It lost 22 more because the second check blocked the fix. In 31 of those 33 blocked fixes, the fix was already correct (L2189-2196). So the gap points to a weak confirmation rule. It does not show that row-level checking is better in general.

  3. The theory is not linked to the experiments. Theorems 1 and 2 (L198-215) study an ideal helper that knows each task’s hit rate and adds one more call per task. The experiments never estimate this value. This is odd, because seven saved calls per task are available (L756). Appendix A.4 (L1143-1149) also says the theorems can’t rule out that more calls to the same helpers would work just as well. The lower-bound constant is about 4 × 10⁻⁸ (L905-909), so it means little in practice. In the end, about eight pages of proofs are a side result. They don’t shape the method or support the main claim.

  4. The benchmark is small and narrow. All 300 tasks come from one template system. Each has 8 variables, 30 constraints and one planted error in a single row (L1503-1518). Everything uses one model, at low reasoning effort, with one budget of B = 6 (L323-324, L465-469). Most quality checks were done with AI help. The expert check is one author saying they looked at 30% of tasks, with no item-level record (L1540-1543). The external baselines were rebuilt by the authors, and the comparison was chosen after the results were seen (L2201-2203). Their damage on correct LPs also depends on an unclear rule in the benchmark itself. It can be read as aggregate or per pair (L1545-1551, L2361-2363). So the 101/150 damage for Opt-Verifier-LP (L459-460) says as much about the benchmark as about that method.

Questions for authors
  1. I’m not sure the two supplements are matched. The task-only side is one call that scores all 30 rules and gives six distinct picks. The candidate side is three calls with two picks each, which give only 3.43 distinct picks on average (L1811-1812). Only the task-only side uses the groups. The paper itself says this does not show that seeing the candidate causes the misses (L222-224, L1161-1162). Could you run a candidate-aware scorer that also ranks all 30 rules and uses the same groups? Without it, “candidate anchoring” is not really tested, even though it is the main idea of the paper.

  2. The paper is very hard to read. Section 3 brings in support, purchase, rescue, conditional rescue, κ, d+ and d− before any result. Fig. 1 (p. 6) has too many boxes to follow. Many key numbers are only in the appendix (Tables 9-11 and 18). In Table 1 (p. 7), the Support and Purchase columns are the same in every row. I would move the main ablations into the paper and cut the notation. Also, the verification comparison is called preregistered (L424, L1564). Where is the registration record? It would also help to show one full task: the word problem, its 30 rules and the reference LP.

  3. A few smaller points. In Table 9, the model’s picks without groups cover 59 errors directly but only 60 after block checks. Random picks without groups go from 60 to 144.91. Why do block checks add almost nothing for the model’s picks? Do its top picks bunch together? The scale-normalization family goes the other way, with 58 vs 61 repairs (Table 15, p. 39). Why is that? Also, 55 of the 57 unrepaired task-only cases have an empty portfolio (Table 18). What stops the method there? Lastly, the AI-use statement says proof drafting was AI-assisted (L488-490). How were the proofs checked?

Reviewer_d42UOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

When one LLM audits another’s output under a limited budget, it can only fix errors it thinks to check, so if no proposer suggests checking the faulty part, the error is never found; the authors call this “missing support” and attribute it to “candidate anchoring,” where proposers that read the flawed output inherit its blind spots. To fix this, they add a “sealed” proposer that sees only the task description and commits to checks spread across requirement types before the output exists, pool its proposals with the usual candidate-conditioned ones, select checks under a fixed budget using shared evidence, and repair by rebuilding only the selected constraints from the task text. They also prove two theorems showing that a small number of repeated calls cannot reliably reveal how much an ideal allocation of extra calls would gain. On 300 template-generated LP problems, each with one injected wrong constraint, plus 150 correct controls, the usual proposers cover 178 errors, adding the sealed proposer raises coverage to 244 and repairs to 243, three extra regular calls reach only 207, and row-level repair beats whole-model verification by 30 points, from which they conclude that budgeted auditing should broaden where it looks before deciding what to spend on.

Critical strengths
  • On identical portfolios, row-level reconstruction beats whole-model verification, 243/300 vs 152/300, with 91/0/209 discordant pairs and a 95% interval of [25.33, 35.67]. This is the one preregistered result, and it holds up.
  • I see that failures are counted honestly. Errors that no proposer covered stay in the 300 denominator, and every stage (covered, selected, repaired) uses the same base.
Critical weaknesses
  • It seems that the sealed proposer is told where the errors are planted. Its four core strata (App. B.3) are weighted averages, unit conversion, cross-period linkage and aggregation scope. These map one-to-one onto the four injected error families (App. C.2), and the selector must pick one rule from each stratum first. The headline 66/122 recovery is largely built into the design.
  • I think the most obvious baseline is missing. All pools share task-only reconstructions that flag disagreements with the candidate, and admission matches that flag on all 13,164 action pairs (App. E.3.1). The system already knows where the error is, but acts only if a proposer happened to nominate that spot. The paper needs to report how often the flag hits the true error row, and to compare against a baseline that simply checks flagged rows.
  • The statistics appears to overstate certainty. Each configuration was run once despite stochastic generation. Confidence intervals resample tasks only, ignoring LLM run-to-run variance. The headline result is not preregistered, and two of the four preregistered tests go unreported.
Questions for authors

Covered in my weaknesses section.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.