acceptodds
Under review as a conference paper at ICLR 2027

Relational Response Fields: A General Theory of Black-Box LLM Response Consistency and Recovery

Abstract

Black-box language-model reliability is commonly pursued by sampling, prompting, voting, verifying, or iteratively revising individual answers. We ask a prior question: what determines whether a collection of black-box responses is recoverable at all? We represent responses to typed transformations of a query as a relational response field (RRF). Edge transports encode how valid responses must change under paraphrase, scaling, decomposition, refactoring, or other task symmetries; anchors encode independently trusted evidence such as execution or a verifier. For relation operator , anchor operator , and at most corrupted response nodes, we identify as the intrinsic difficulty of black-box response recovery. It is positive exactly when every -node corruption is identifiable; it gives a deterministic stability bound proportional to ; and a matching two-point minimax lower bound shows that no estimator can improve this dependence. Thus consistency is not truth: relation-only methods are blind to null directions, including shared hallucinations. We derive sparse field-repair algorithms while separating information-theoretic identifiability from the stronger null-space conditions required by convex optimization. Controlled theorem tests and black-box mathematics/code experiments evaluate four theory-fixed consequences: consistency–truth separation, anchor phase transitions, redundancy saturation, and cross-model, cross-task prediction of repair difficulty. The results support as a measurable property of a response-recovery instance, rather than a score attached to one repair heuristic.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Black-box LLM Reliability, Relational Response Fields, Sparse Recovery

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_TF5JOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper asks a valid question: When we can spot and fix wrong answers from a black-box language model by looking at its responses to several versions of the same question. These versions are connected by known rules: two answers might need to agree, or one might need to be three times another. Trusted outside checks, called anchors, help tie the answers to something more reliable than the model's own agreement. The authors introduce a number, γk\gamma_k, that captures how well these rules and checks expose errors in up to kk responses. Under the stated assumptions, a positive γk\gamma_k means those errors can be uniquely identified. With noise bounded by ϵ\epsilon, the repair error is at most 2ϵ/γk2\epsilon/\gamma_k, and no method can uniformly beat the ϵ/γk\epsilon/\gamma_k scale. Tests on synthetic data and four open models answering generated arithmetic and coding questions illustrate these findings. The useful insight is that a model can be consistently wrong; what matters is whether the available relations and checks give us enough independent evidence to correct it.

Critical strengths

Maybe summarized in three important overarching bullet points: 1. Generally clear problem statement and an ambitious endeavour. The authors argue that a set of answers can be fully consistent yet still completely wrong. 2. The tasks are generated with exact labels, which makes the experiments easy to audit and be tractable. 3. The paper carefully distinguishes between whether the correct answer can be uniquely determined in principle and whether a particular algorithm can actually recover it.

Critical weaknesses

Paper's main contribution is the idea of treating repair of related LLM answers as a sparse inverse problem, with γk\gamma_k regarded as a measure of how hard that repair is. The authors present the theorems as applications of known compressed sensing results. The paper should therefore be judged on what this framing adds and the implementation achieves rather than on complete novelty. To assess, authors should have compared their work with earlier methods that already fix model answers using relations between related questions, such as ConCoRD (Mitchell et al., 2022), Maieutic prompting (Jung et al., 2022) and BeliefBank (Kassner et al., 2021). Additionally, Fluri et al. who already showed that consistency does not imply correctness—as a side, to my observation, none of these works are properly cited.

A new framing of the problem these methods already address should therefore be providing results that the older methods, in the first place, could not have. What would have been opportune real data experiments collapse to majority vote when no anchor is used and to the answer key when they are, which clouds any claims of improvement.

Furthermore, I noiticed the main experiment cannot fail anyway. The replay test has one wrong answer and no added noise at all. Under such conditions, the theorem guarantees perfect recovery whenever γ\gamma is positive, as Table 6 clearly indicates, so the experiment remains nothing more than a cross-check for the theorem that provides no useful information about LLM errors. Some other predictions also follow from the definitions. Duplicate saturation, the anchor transition and consistency blindness hold by construction. Testing them numerically again adds no evidence whatsoever.

It should also be noted that the main text reports four models, but most appendices cover two (tables). Some numbers disagree between tables as well. Some replay counts are larger than what the dataset allows. For example, more replay fields than code items, several promised results are missing, and "SPARSE-RRF" is never even defined.

While the appendix size is not usually a main concern and could plausibly be kept long for a paper centered on theory, it actually runs for this MS over 50 pages, of which the main text refers to less than half. The only test on unmodified model outputs and the only quantitative test of the 1/γ1/\gamma results appear only in the last appendix, which should be in the main text instead.

Questions for authors
  1. Can there be more than one replay field per item? Where do the correct answers come from for items for which the model produced no correct answer?
  2. Is there any result on real data where a larger positive γ\gamma yields better recovery than a smaller positive γ\gamma?
  3. How does the math anchor differ from the ground-truth label?
  4. Does Theorem 5 still hold when noise affects only the anchor rows?
  5. What regularization was used in the logistic model? Does the γ\gamma coefficient remain finite without it?
  6. Can you show one real task with partial anchors or non-identity relations where combining relations and anchors outperforms using either one alone?
1 comment
Reviewer_TF5JOpenReview ↗

Response to Author's Rebuttal

Thanks for the detailed response and for some of the changes to the current manuscript. The replay accounting, the analysis of anchor-only noise in Proposition 6, and the regularization analysis mostly addressed my earlier technical questions. The expanded discussion of prior work also clarified the contribution of the decoder-independent recoverability certificate. Yet, I still have a few more remaining concerns.

  1. Of the 400 naturally generated fields in the error classes reported in Table 19, 90 contain exactly one incorrect response and remaining 310 contain shared or multiple errors. The 100% recovery observed at positive margins seems to be therefore conditional on an error class identified using the ground-truth answers. So here is a question: Can the corruption budget (k), or the decision to abstain, be determined from the information available at the inference time? Evaluating a rule specified before the evaluation labels are examined would establish how often the certificate can guide repair in practice.

  2. Secondly, each field contains an original claim, two paraphrases, and a negation. Reversing the sign of the response to the negated claim reduces the relations in this experiment to equality constraints. The results demonstrate that partial anchors can improve recovery in this specific domain. A relation that cannot be reduced to agreement by recording individual responses, such as the relation between a total and its components, would provide a more direct test of the framework’s broader applicability in my opinion. For the present experiment though, would you please specify how the retrieval anchors were verified and whether the coverage was varied across fields or within each field, and if so, what would the impact be?

  3. Even after calibration, adding the cross-checkpoint anchor decreases accuracy relative to RRF-D for three of the four target models in Table 26. For Qwen2.5-7B, accuracy decreases further from 82.8% to 81.2%. Reporting paired gains and losses would assist the reader characterize this effect. It would also be informative to examine whether anchor reliability and the extent to which anchor and target errors coincide would guide the decision to use an anchor or maybe not.

The code experiments show that fewer labeled tests can suffice on this benchmark. To assess the practical savings, authors are advised to report the number and cost of unlabeled probe executions and include a baseline with access to the same labeled tests and probe outputs. Although the authors’ revisions have considerably strengthened the paper, the limited significance of the contribution keeps my assessment at Weak Accept (3).

Reviewer_h8DvOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2(+1)
Read review
Paper summary

This paper proposes Relational Response Fields, a framework for characterizing the recoverability of black-box LLM responses connected by known transformations and external verification constraints. By formulating response correction as a group-sparse inverse problem, the authors introduce a relational observability margin γk\gamma_k that characterizes uniform identifiability, worst-case stability, and minimax recovery limits. The framework also includes sparse recovery algorithms and extensions to nonlinear relations. Experiments on synthetic problems and mathematics/code tasks across four LLM checkpoints investigate consistency–truth separation, anchor placement, redundancy, and the relationship between γk\gamma_k and empirical recovery difficulty.

Critical strengths
  1. The paper provides a useful perspective on black-box LLM reliability by separating whether errors are observable from whether a particular correction algorithm can recover them. The distinction between relational consistency and factual correctness, together with the role of external anchors, offers a clear conceptual foundation for analyzing inference-time verification and correction.

  2. The main identifiability, stability, and minimax results are sound under their stated assumptions, with substantial supporting proofs in the supplement. The experimental design also has notable strengths: it reuses authentic model errors across fixed operator interventions, distinguishes controlled replay from natural-response evaluation, and employs grouped statistical analyses to reduce confounding.

Critical weaknesses
  1. The central margin γk\gamma_k is mathematically a group-restricted minimum singular value, and its identifiability, stability, and minimax properties largely follow from classical sparse recovery theory. Moreover, graph-based consistency reasoning and joint answer correction have substantial precedents, including ConCoRD and Maieutic Prompting, while execution-based code selection has been explored in CodeT and MBR-EXEC. Although the paper acknowledges several foundational connections, it does not sufficiently establish what fundamentally new theoretical insight or methodological capability arises from their combination in the LLM setting.

  2. In the real-error replay experiments, three nodes contain a correct response and one contains an authentic model error under a noiseless k=1k=1 corruption model. All reported positive-margin designs achieve 100% exact recovery, as already guaranteed by the identifiability theorem, whereas zero-margin designs exhibit varying success depending on error placement. Consequently, the reported correlations and predictive gains do not establish whether the numerical magnitude of positive γ1\gamma_1 provides information beyond the binary indicator 1[γ1>0]\mathbb{1}[\gamma_1>0]. The supplementary positive-margin noise experiments support the theoretical stability dependence, but they use controlled perturbations rather than naturally occurring LLM recovery errors.

  3. The main paper reports 3,660 replay records across four checkpoints, with a pooled Spearman correlation of 0.451 in Table 3. However, the detailed supplementary breakdown in Appendix L covers only two checkpoints and 2,690 records, while Section 5.3 and Supplement Table 7 report a pooled correlation of 0.449. Several described supplementary robustness analyses also lack corresponding quantitative results.

  4. In the natural-response experiments, the anchor-augmented RRF decoder matches the standalone verifier's accuracy across every reported model–task stratum, with both reaching the candidate-recall ceiling. Since the evaluation uses exact arithmetic verification or reference execution signatures on public tests, the incremental contribution of the relational operator over strong external verification remains unclear. Furthermore, the experiments are restricted to template-generated arithmetic and small integer-function tasks with precisely defined response transformations. While appropriate for validating the theory, these settings provide limited evidence that the framework offers additional practical value when semantic relations, representations, and anchors are less reliable.

Questions for authors
  1. Given that γk\gamma_k is a restricted minimum singular value and that prior methods already combine relational constraints with joint answer or code selection, what specific theoretical insight or experimentally demonstrated capability should be regarded as the paper's primary novel contribution? How does it differ substantively from existing graph-based consistency-repair and execution-based selection methods?

  2. Do the reported cross-model predictive gains persist when controlling for 1[γ1>0]\mathbb{1}[\gamma_1>0], rather than only relation count, anchor count, and other covariates? Since all reported positive-margin replay designs achieve exact recovery, this comparison seems essential for establishing whether the continuous margin contributes information beyond the known identifiability threshold.

  3. Could the authors provide the missing detailed replay results for Qwen2.5-7B and Qwen2.5-14B and clarify why the pooled Spearman correlation differs between the main table (0.451) and the surrounding text/supplement (0.449)? This would help establish which reported results correspond to the complete four-model evaluation.

  4. Given that RRF-D+A and standalone verification achieve identical aggregate accuracy and reach the candidate-recall ceiling, is there evidence that the relational operator changes candidate selection or improves recovery beyond the verifier alone? Clarifying this would help assess the practical contribution of relational structure when strong external anchors are available.

Reviewer_ARXZOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies when errors in black-box LLM responses can be identified and corrected using relationships between answers and trusted reference information. The authors organize parsed responses into a relational response field and define an observability measure, γk(D,A)\gamma_k(D,A), to characterize whether sparse errors can be recovered and how noise affects recovery. The analysis also distinguishes having enough information to recover the answers from whether a particular recovery algorithm can do so.

The paper evaluates the framework through synthetic experiments and small mathematics and code tasks with four language models. The synthetic experiments examine why consistent answers can still be wrong and how anchors and additional relations affect recovery. The main cross-model analysis constructs four-response fields with three correct responses and one real model error, then varies the available relations and anchors. The authors report that the observability measure is associated with recovery performance in these controlled settings, alongside separate results on naturally generated responses.

Critical strengths

S1. The paper provides a unified theoretical characterization of sparse response recovery under a specified linear observation model. Theorems 3–5 connect the same restricted observability margin to uniform identifiability, deterministic stability, and a matching minimax lower bound. This makes the quantity more interpretable than a heuristic consistency score and clarifies which recovery limitations apply regardless of the algorithm used.

S2. The paper clearly distinguishes relational consistency from correctness and information-theoretic recoverability from the guarantees of a particular decoder. The relation–anchor formulation and the discussion in Section 6 explain why mutually consistent responses can still be wrong, and why insufficient observations cannot always be compensated for by a better optimizer.

Critical weaknesses

W1. My main concern is that the practical applicability of the proposed observation model is not sufficiently established. The framework assumes that responses can be parsed into suitable numerical representations, which means their relations can be expressed through known linear transports and informative external anchors are available. However, the experiments mainly consider mathematical transformations and code execution signatures. These settings make the observation model easy to construct, but provide limited evidence about how it would be obtained for less structured LLM tasks, where semantic relations, parsing errors, and trusted evidence are more difficult to specify.

W2. It remains unclear whether the observability margin predicts recovery difficulty for natural LLM errors. Although Appendix Q.5 further tests the margin under controlled perturbations and Appendix Q.4 reports repair results on naturally generated responses, these natural-field results do not show whether larger margins correspond to more successful repair. The experiments support the margin under controlled conditions, but provide limited evidence that it can help predict recovery success in practical use.

Questions for authors

Q1. Could the authors provide a concrete example beyond mathematical transformations and execution-signature equality showing how response representations, relation operators, and informative anchors can be constructed? What trusted information would be required, and what would RRF recovery add beyond directly using that information for verification?

Q2. The paper establishes theoretical recoverability and reports both controlled replay and natural-field repair results. Could the authors clarify how the observability margin can guide recovery decisions on natural response fields, and what additional benefit relational recovery provides over verification using the same trusted evidence?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.