- Rating
- 2(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2(+1)
Read review
Paper summary
The paper studies whether internal error detectors rely on answer-label information rather than correctness. It proposes Answer-Frame Symmetrization (AFS): after generating an answer once, the method permutes the four answer options, teacher-forces the label corresponding to the original semantic answer, and averages internal representations across all 24 permutations. The authors interpret the resulting correctness-related signal as an “Error Field.”
Experiments use a synthetic visual-reasoning benchmark containing 6,400 questions across 16 reasoning types, with additional model-family comparisons and zero-shot transfer to GPQA Diamond. AFS combined with VIB-Probe achieves 71.38% mean Macro-F1 across eight model–domain settings, compared with 63.52% for the strongest baseline selected separately in each setting. The paper also analyzes answer-label decoding, cross-label transfer, linear subspace removal, and representation geometry.
Critical strengths
- The paper identifies a meaningful potential confound in internal error detection. Distinguishing correctness information from answer-label identity is important for evaluating whether probes learn transferable signals. The cross-answer transfer experiments and permutation-based analysis provide useful diagnostics beyond reporting aggregate detection accuracy.
- AFS produces meaningful empirical improvements, particularly in-domain. The approach improves both linear probes and VIB-Probe, and its gains recur across several model families.
Critical weaknesses
-
The formulation and evaluation are restricted to multiple-choice questions. AFS requires a finite set of answer options whose labels can be permuted while preserving the selected semantic answer. This assumption does not directly extend to open-ended generation, summarization, or other common error-detection settings. The paper therefore does not establish that its separation of answer identity and correctness generalizes beyond this narrow setting.
-
The novelty relative to existing work on selection and answer-order bias needs clearer justification. The influence of option order and answer labels on multiple-choice predictions is already extensively studied. Although this paper focuses on internal error detectors, it should more clearly distinguish its contribution from established permutation-based debiasing and aggregation approaches. Naming the resulting signal an “Error Field” does not by itself establish a distinct phenomenon.
-
The significance of the issue at larger model scales remains unclear. The main scaling experiments cover Qwen3.5 models from 0.8B to 9B, whose correctness on the benchmark is only around 30–39%. It is uncertain whether answer-label interference remains comparably consequential in substantially larger, more capable models. Evaluating such models would help establish whether the findings reflect a broad limitation or primarily the behavior of smaller models on difficult tasks.
Questions for authors
- At which exact token position are representations extracted for each detector—before predicting the mapped label or after consuming it?
- Please provide confidence intervals or paired significance analyses for the main comparisons and GPQA transfer.
- Option permutations can change model preferences and confidence. What evidence supports Equation (7)’s assumption that the error component remains unchanged while only an additive answer-token component varies?
8 comments
We have revised the PDF under the title “Answer-Frame Symmetrization Improves Internal Error Detection in Language Models.” The revised paper removes the E-Field definition and additive decomposition, and centers its contributions on fixed-response error detection. Below, we clarify the technical distinctions and provide statistical analyses of the reported results.
W1. Multiple-choice formulation and experimental design
The multiple-choice formulation provides an exact control for studying correctness detection. Relabeling changes the answer frame while preserving the question, option meanings, selected semantic response, and its correctness. This lets us examine whether a detector changes its judgment when the answer being assessed remains the same.
The revised abstract and Section 3 explicitly define this four-option problem. The evaluations cover two complementary settings:
| Evaluation | Experimental purpose | Result |
|---|---|---|
| Controlled visual benchmark: 6,400 questions, 16 reasoning types | Programmatically verified correctness; held-out reasoning types; paired detector comparisons | AFS + VIB achieves 71.38% mean Macro-F1, versus 63.52% for the strongest per-setting baselines. |
| GPQA Diamond: 198 independent text questions | Transfer from visual-source training to a non-synthetic text benchmark, with the source detectors and thresholds frozen | Macro-F1 increases from 31.83% to 57.44% with AFS-4 and to 52.82% with AFS-24. |
The scientific question is how response-preserving transformations affect an error detector, and the finite answer set makes this intervention precisely testable. Extending this construction to open-ended responses requires defining an appropriate family of response-preserving transformations. The results above establish its value for the specified multiple-choice detection problem, including transfer across datasets and modalities.
W2. Novelty relative to answer-selection debiasing
AFS studies the correctness of an already-generated answer. Answer-selection debiasing studies how the model chooses that answer. These objectives lead to different interventions and evaluation targets.
For a concrete comparison, PriDe estimates an option-ID prior and corrects the answer-selection distribution. PA-GRPO, already cited in our Introduction, trains the answering model for more consistent decisions across option permutations. PriDe, ICLR 2024; PA-GRPO, ACL 2026.
| Aspect | Answer-selection debiasing, illustrated by PriDe and PA-GRPO | AFS |
|---|---|---|
| Prediction target | Which semantic answer should the model select? | Is the model's originally selected semantic answer incorrect? |
| Role of permutations | Estimate selection bias or encourage consistent answer choices | Construct equivalent frames for the same fixed response |
| Intervention | Correct answer probabilities or optimize the answering policy | Average internal representations before correctness prediction |
| Assessed response | The method operates on answer selection | The original semantic response is fixed and its mapped label is teacher-forced |
| Output and evaluation | Answer choices, selection bias, and answering performance | Error scores, correctness Macro-F1, and detector transfer |
The contribution also includes detector-specific findings. With the response and its correctness fixed, the original VIB detector's mean agreement across equivalent frames is 77.96% on ID and 86.82% on OOD. AFS improves the linear detector in all eight model–domain settings, and AFS + VIB exceeds the strongest per-setting baselines in seven of eight, by 7.85 pp on average. The new training controls further show that item-specific frame variation can improve transfer even when all evaluation features remain the same complete AFS average.
These are findings about the reliability and transfer of correctness readouts. The review names a related research area but does not identify a prior method that already supplies this fixed-response detection construction or these diagnostic findings. Sharing the permutation operation alone does not establish equivalent contributions. We ask that the novelty assessment consider the prediction target, the fixed-response control, and the measured detector behavior together.
W3. Model scale and task difficulty
The detection gains persist at the largest model in the Qwen scaling study and recur across independently evaluated model families. At Qwen3.5-9B, AFS + VIB improves ID Macro-F1 from 74.69% to 82.34% and OOD Macro-F1 from 60.83% to 65.37% relative to the strongest baselines in those settings.
The additional family comparisons hold each model's generated responses, correctness labels, and test examples fixed between Raw VIB and AFS + VIB:
| Backbone | ID Macro-F1: Raw → AFS (%) | OOD Macro-F1: Raw → AFS (%) | OOD gain, 95% paired CI (pp) |
|---|---|---|---|
| Gemma-4-E4B-IT | 71.04 → 76.51 | 59.32 → 67.51 | +8.19 [6.24, 10.12] |
| Phi-4-Multimodal | 70.29 → 76.90 | 58.42 → 67.14 | +8.72 [6.97, 10.45] |
| GLM-4.6V-Flash | 74.11 → 76.10 | 61.07 → 66.01 | +4.94 [2.93, 6.94] |
The reported 30–39% describes the answering model's correctness. Within each detector comparison, that correctness is identical because the original responses are unchanged. Thus, the measured improvement comes from predicting errors more effectively on the same answers. Macro-F1 gives equal weight to the correct and incorrect classes.
The scale evidence currently covers Qwen3.5-0.8B through 9B; the additional experiments test recurrence across architectures. Establishing the magnitude of the effect in substantially larger models remains an empirical question. These comparisons establish recurrence across the evaluated sizes and architectures.
Q1. Exact token positions used by the detectors
AFS + VIB and AFS + Linear use the mapped answer-label position after that label has been consumed. Let be the position of that label in the supplied sequence. Its representation is computed from a prefix that includes the label; the state used to predict the label itself is at .
| Detector or analysis | Representation or signal used in our implementation |
|---|---|
| AFS + VIB / Raw VIB | Inputs to the full-attention layers' modules at , after consuming the selected or mapped answer label |
| Linear probe; answer-label decoding; INLP diagnostics | Decoder hidden states at , at the specified candidate layers |
| Real-Time Entity Probe | Final decoder-layer hidden state at |
| Semantic Entropy Probes | Hidden states from the final five decoder layers at the final answer token, immediately before EOS in the full sequence |
| HALP | Vision-encoder features, decoder states at the final visual token, and states at the final question token; these positions precede the generated response |
| Lookback Lens | Attention ratios at each supplied response-token position, including , averaged over response positions |
| TOHA | Attention graphs over the prompt and supplied response tokens, through |
| Spilled Energy | Logits immediately before and after each supplied response token, including and for the answer label; minimum pooling over the response |
| INSIDE | Hidden state after consuming each sampled answer label, at the selected middle layer; covariance across sampled answers |
| Semantic Entropy / SelfCheckGPT | Statistics of sampled answers and their agreement |
| P(True) | True/False likelihoods under a separate correctness-assessment prompt |
For AFS, the supplied label denotes the model's own original answer, mapped to the new frame. Section 3.3 now explicitly states the post-consumption extraction position for AFS + VIB. Section 4.3 also treats near-perfect answer-label decoding at this position as an expected reference for the INLP intervention.
Dear reviewer:
Thank you for recognizing the meaningful potential confound and meaningful empirical improvements in our work. Our main contribution is precisely this: identify an overlooked source of answer-frame dependence → address it → improve error detection.
“The formulation of a problem is often more essential than its solution, which may be merely a matter of mathematical or experimental skill.” — Albert Einstein and Leopold Infeld, The Evolution of Physics (1938) The first step toward progress is often the hardest: recognizing a problem that others have learned to overlook. Once a problem is made visible, even a simple solution can represent a meaningful advance. What ultimately matters is not how impressive a method appears, but whether it withstands the test of evidence.
Token prediction ≠ correctness detection. Activations contain information about how an answer is expressed, alongside information useful for assessing correctness. We study whether the detector's judgment changes when the answer's meaning and correctness stay fixed. The original VIB detector's mean agreement with its per-item majority decision across equivalent frames is 77.96% on ID / 86.82% on OOD, demonstrating a concrete monitoring problem.
The table compares the papers' research focus: interference from answer-label tokens (A/B/C/D) in correctness prediction when the semantic answer is held fixed. ✓ = a central research question; ✗ = not a central research question.
AFS: fixed semantic answer → equivalent answer frames → averaged activations → frame-invariant representation. Selection-bias research examines which answer is generated; our work examines whether the fixed answer is correctly judged.
These results show the value of examining the representations used by established detectors. Is identifying a practical weakness and addressing it with a simple, effective method not a substantive scientific contribution?
We are a university research team working with limited GPU resources. Evaluating substantially larger models is constrained by the computing capacity available to us. Within these resources, we have made a sustained effort to evaluate four model scales, multiple families, and transfer beyond the original benchmark. We hope these completed experiments receive appropriate weight in the assessment.
We chose multiple-choice questions because they allow precise control: change the answer frame while preserving meaning and correctness. This makes the problem directly testable. Extending the approach to open-ended natural-language generation is a direction for our future work. A carefully controlled first step can establish a useful contribution and a foundation for broader study.
**Given the meaningful problem and empirical gains already acknowledged in your review, we sincerely hope you could reconsider the rating in light of the clarified research question, the completed experiments, and the demonstrated improvement over established detectors.