acceptodds
Under review as a conference paper at ICLR 2027

Trust Labels Should Calibrate, Not Decide: Diagnosing and Reducing Source-Label Shortcuts in RL-Trained Fact-Checkers

Abstract

Retrieval-augmented fact-checkers often receive a reliability label, such as or trust, for each evidence source. These labels should adjust the model's confidence and its decision to search for more evidence, while the verdict should follow the evidence content. We introduce TrustSwap, a counterfactual test that swaps, lowers, or removes source labels while keeping every evidence text fixed, and measures the verdict, the confidence, and the search decision separately. Across untrained and RL-trained models at two scales, three datasets, and two prompts, confidence and search respond to the labels as intended in 49 of 50 comparisons, yet a label change alone alters 4–23% of confident verdicts for Qwen3 models and up to 50% for an existing RL-trained fact-checker. Standard GRPO fine-tuning amplifies this shortcut at 8B in all six settings. To reduce it, we propose trust-swap augmentation (TSA), which trains GRPO on each claim with both its original and its label-swapped evidence under the same gold verdict. At 4B, TSA lowers the verdict flip rate by 7–35% (relative) in four of six settings, keeps accuracy and the intended confidence and search responses, outperforms reward-based alternatives in the main setting, and carries over to an unseen label-removal perturbation. An added consistency reward helps on the trained-on swap but not on unseen perturbations. At 8B, TSA's effect is not detectable, which makes scale the main open question.

Primary area:
reinforcement learning
Keywords:
fact-checking, fact verification, reinforcement learning, source shortcuts, calibration, variance-aware evaluation, counterfactual augmentation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_oSgpOpenReview ↗
Rating
2
Confidence
3
Informativeness
3(+2)
Presentation
2
Soundness
2
Read review
Paper summary

The work studies how source credibility labels affect fact-checking models before and after RL training. TrustSwap changes these labels while keeping evidence texts and source names fixed, and measures verdicts, confidence, and search flags separately. The results show that labels alone can change verdicts, and GRPO can amplify this effect at 8B. The authors propose trust-swap augmentation (TSA), which trains on original and swapped evidence with the same gold verdict. At 4B, TSA reduces flips in four of six settings, mainly through fewer changes to or from NEI. No significant reduction is detected at 8B.

Critical strengths
  1. TrustSwap provides a controlled test showing that source labels alone can change fact-checking verdicts. Evaluating verdicts, confidence, and search flags separately helps distinguish unwanted verdict changes from intended responses to source credibility.

  2. TSA is simple to incorporate into GRPO and leaves the base reward unchanged. At 4B, it reduces verdict flips in several settings while retaining accuracy and directional confidence and search-flag responses.

Critical weaknesses
  1. TSA's benefit is narrow. In the main 4B setting, almost all of the 1.12 percentage-point reduction comes from fewer abstention flips, with no measurable reduction in SUPPORTS–REFUTES reversals. Since abstention under low trust can be appropriate in some deployments, the practical benefit depends on the stricter contract. The principled prompt's advice to avoid definitive verdicts under low trust further complicates the interpretation of NEI flips.

  2. Evidence for broader effectiveness is unclear. TSA improves on unseen label removal at 4B, but the downshift gain is not significant, and no 8B setting shows a significant reduction. Training uses only 311 claims and 25 steps, with three TSA seeds at 4B; different learning rates also make it difficult to separate scale from optimization effects. An original-only control is still needed to isolate the benefit of swapping labels from extra sampling.

Questions for authors
  1. To distinguish label-driven shortcuts from abstention encouraged by the prompt, could you test a version of the principled prompt that retains verdict invariance but removes the advice to avoid definitive verdicts under low trust?

  2. To isolate the contribution of label swapping from extra sampling, could you add a compute-matched control using two separate groups of G=8 rollouts from the original evidence, with the same reward, within-group advantage normalization, and optimization schedule as TSA?

6 comments

We thank the reviewer for the two concrete experimental suggestions, both of which we have run as specified.

Q1. Principled prompt without the advice to avoid definitive verdicts.

We retained the verdict-invariance rule and changed only the advice. The search rule now reads: "...set needs_more_search: yes, and still give the verdict that the evidence content supports" (originally: "...rather than committing to a definitive verdict").

Table R4. Qwen3-8B, AVeriTeC, paper's GRPO setup: flip rate / abstention flips / reversals (% of confident pairs; GRPO baseline = mean of 5 seeds).

NeutralPrincipledPrincipled, no advice
Untrained10.1 / 6.4 / 3.715.8 / 12.7 / 3.111.2 / 8.8 / 2.4
GRPO baseline15.0 / 11.9 / 3.222.2 / 20.2 / 2.116.5 / 14.4 / 2.0

The results confirm that the advice accounts for part of the abstention flips: removing it lowers the flip rate from 15.8% to 11.2% for the untrained model and from 22.2% to 16.5% after GRPO. However, the shortcut persists without the advice: the untrained model still changes 11.2% of confident verdicts (10.1% with the neutral prompt), and GRPO raises this to 16.5%. Reversals, which the advice cannot explain, remain at 2.0–2.4%. Moreover, our main setting uses the neutral prompt, which contains no such advice, so the main results do not depend on it. On Qwen3.5-4B, removing the advice has little effect on the untrained flip rate (5.9% → 5.4%). In the revision, we will add the no-advice prompt and qualify the §4 paragraph on stating the rule in the prompt.

Q2. Compute-matched original-only control.

We trained the control exactly as specified: two separate groups of G=8 rollouts on the original evidence, with the same reward, within-group advantage normalisation and optimisation schedule as TSA.

Under standard GRPO on Qwen3-8B, the control separates the two effects. Additional rollouts alone increase the main-setting flip rate over the baseline (+1.63 pp [+0.43, +2.93]), whereas TSA decreases it. TSA is 2.99 pp below the control [−4.47, −1.75] and lower in 7 of 9 settings, including both unseen perturbations; a paired bootstrap yields the same conclusion. Under this objective, the gain is therefore attributable to the swapped labels rather than to additional sampling. As in the comparison with the baseline, the reduction is in abstention flips rather than reversals.

Under the paper's GRPO setup, the two cannot be separated. At 8B, neither lowers the flip rate (TSA − control −0.42 pp [−2.14, +0.96]). On Qwen3.5-4B, which we used because Qwen3-4B was withdrawn from the training service and which loses 6–8 pp accuracy with our 4B hyperparameters, both lower it by similar amounts (TSA − control +0.62 pp [−0.97, +2.51]). In the revision, we will add the control to every table and state this dependence on the objective.

W1. The benefit is narrow.

The avoided abstention flips are nevertheless relevant: when the 4B baseline switches to NEI after a label change, the original verdict was correct in 58% of cases, and TSA reduces these cases (−0.51 pp [−0.96, −0.10]). Deployments that prefer to abstain under low trust can do so through the confidence and search outputs, which TSA preserves (see our response to Reviewer onKv, Q1).

W2. Broader effectiveness, seeds and learning rates.

Scale versus optimisation. The 4B learning rate made 8B training unstable, so the 8B learning rate was selected only on the baseline's accuracy and NEI rate, without reference to any TrustSwap metric (§5.1). With this learning rate, switching to standard GRPO makes TSA's effect detectable at 8B: −1.37 pp in the main setting [−2.83, −0.04], −1.62 pp on CONFACT with the neutral prompt, and reductions under both unseen perturbations, including the downshift (−1.48 pp [−2.46, −0.51]), which was not significant at 4B. This is consistent with the explanation in §6 that the 8B result reflects optimisation rather than scale alone, and the compute-matched control does not reproduce this gain (Q2).

Seeds. All new models are trained with 5 seeds. Additional 4B seeds could not be run because the training service withdrew the 4B model (§5).

Training size. We agree that 311 claims and 25 steps constitute a controlled study rather than a full-scale training run; longer training on more claims is discussed as future work (§7).

We hope that these results address the reviewer's concerns, and we would appreciate it if the reviewer would consider updating the score accordingly.

Thank the authors for the response, it partially addresses my concerns, but two main concerns remain.

  1. Reducing transitions from correct verdicts to NEI does not prove that the abstentions are wrong, especially with low-trust evidence. To better assess the overall impact, it would help to include changes in transitions from incorrect verdicts to NEI and to analyze the trade off between accuracy and abstention.

  2. The new results with no-advice prompt show that reversals persist, and there is no evidence that TSA helps mitigate the reversals. Specifically, after GRPO, the reversal rate decreases from 2.4% to 2.0%, while the total flips increase comes from changes in abstention.

We thank the reviewer for the follow-up. We address both points below with additional analyses of the existing model outputs.

1. Are abstentions under low-trust labels useful?

If abstaining on low-trust evidence is a sensible policy, these abstentions should mostly remove wrong verdicts and make the remaining answers more accurate. We tested this on the pairs where the swap lowers the labels from high to low trust and leaves the evidence text unchanged (AVeriTeC, neutral prompt, confident pairs).

ModelMethodCorrect -> NEIWrong -> NEIAccuracy change (pp)Selective accuracy, high -> low trust (%)Confidence-based abstention, same coverage (%)
Qwen3-4BGRPO baseline4.02.7−3.974.5 -> 74.576.7
TSA3.52.3−3.974.5 -> 73.976.6
Qwen3-8B, standard GRPOGRPO baseline5.96.3−5.071.9 -> 73.075.5
Compute-matched control7.57.1−6.771.3 -> 72.375.2
TSA5.25.5−5.370.3 -> 70.273.7

Transitions are per 100 pairs. Selective accuracy is the accuracy on non-NEI answers; the last column shows it when the model instead abstains on the same number of its least-confident answers.

The data do not support this policy. About half or more of the withdrawn verdicts were correct (60% at 4B, 48% at 8B). Lowering the labels costs the baselines 3.9 to 5.0 pp of accuracy, while selective accuracy improves by at most about 1 pp. At the same coverage, confidence-based abstention gives 2.2 to 3.5 pp higher selective accuracy for every model in the table, so abstaining because of the labels is a weaker selective-prediction rule than the model's own confidence.

TSA reduces both transitions, which is the intended effect when the evidence should decide the verdict. At 8B, compared with the compute-matched control, it withdraws 2.3 fewer correct and 1.5 fewer wrong verdicts per 100 pairs and loses 1.4 pp less accuracy (all significant); at 4B, the accuracy cost is unchanged. A system that should be cautious with low-trust evidence can still use the confidence and search outputs, which TSA keeps responsive to the labels. Details are in Appendix M.

2. Reversals

This matches the scope stated in the paper: TSA's gains "come from fewer abstention changes rather than fewer reversals" (Sec. 1), and reversals remain the main open problem (Sec. 7, Sec. 8). We target abstention flips because they are most label-driven flips (78% for the 4B baseline, Sec. 4) and the part that RL fine-tuning amplifies. The no-advice results show the same pattern: after training, abstention flips increase while reversals do not (2.4% -> 2.0%). TSA reduces this amplified part and keeps reversals at the baseline level, with no significant difference in any of the six 4B settings and in eight of the nine 8B settings (Appendix K). Reversals already exist in untrained models and no method in our study reduces them; TrustSwap reports them separately so that future work can target them directly.

We hope this answer can solve the reviewer's remaining concerns.

Reviewer_onKvOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

TRUSTSWAP changes source trust labels while keeping evidence fixed, then measures verdicts, confidence and search requests. Trust-swap augmentation adds the swapped examples to GRPO training. Its clearest reported gains occur at 4B and concern abstention changes.

Critical strengths

1.The distinction between polarity reversals and moves to or from NEI is central to interpreting source-label sensitivity. It separates changes in the answer from changes in willingness to answer.

2.The reward comparisons and unseen perturbations give context for TSA's gains. The 8B results also make the limits of the mitigation clear.

Critical weaknesses

1.Most of TSA's improvement comes from fewer abstention changes. Withdrawing a verdict when source reliability falls can be sensible. The paper reports no clear reduction in polarity reversals under the trained swap, so it needs a stronger reason to treat the avoided abstentions as a practical benefit.

2.TSA receives more rollouts than plain GRPO, while the compute-matched consistency arms also alter the reward or group normalization. Attribution is further complicated by the small, label-correlated training pool and few seeds in several 4B arms. The 8B comparison changes optimization and inference settings as well.

Questions for authors

1.Which intended application makes a switch to NEI after reduced source trust an error?

2.How much of TSA's gain can be separated from extra sampling using the reported comparisons?

5 comments

We thank the reviewer for highlighting the distinction between reversals and abstentions, which we also consider the central design choice of TrustSwap.

Q1 / W1. "Which intended application makes a switch to NEI after reduced source trust an error?"

We see three such applications.

Labels that are inaccurate or manipulated. Trust labels are pipeline metadata, typically derived from third-party ratings, and may be outdated or set by an adversary. If lowering a label suffices to turn a verdict into NEI, a correct fact-check can be suppressed without any change to the evidence. This is frequent in our data: when the 4B baseline switches to NEI after a label change, the original verdict was correct in 58% of cases (46% at 8B). TSA reduces these cases relative to the baseline (2.44% vs. 2.94% of confident pairs; −0.51 pp [−0.96, −0.10]) and, at 8B with standard GRPO, relative to the compute-matched control (−1.24 pp [−1.76, −0.84]).

Automated pipelines without human review, such as automatic claim labelling. In this setting, an NEI leaves a false claim unlabelled.

Deployments that intend to abstain under low trust. Even here, the verdict is not the appropriate output for this decision. The models already lower confidence and request more search when trust decreases, and TSA preserves both responses (§6). A deployer can implement abstention as a threshold on these outputs while retaining the evidence-based verdict; once the verdict itself is NEI, the evidence-based verdict is lost. The stricter contract therefore supports the weaker one, but not the reverse.

We agree that TSA does not reduce reversals (§6), and we regard this as the main open problem.

Q2 / W2. "How much of TSA's gain can be separated from extra sampling?"

We agree that the compute-matched arms in the paper cannot isolate this effect, since they also change the reward or the group structure. We therefore added the control proposed by Reviewer oSgp: two separate groups of G=8 rollouts on the original evidence, with the same reward, advantage normalisation, schedule and rollout count as TSA. The only difference is that the second group in TSA receives swapped labels.

Table R3. Qwen3-8B, main-setting flip rate (%), mean of 5 seeds.

ObjectiveGRPO baselineControlTSATSA − control (pp)
Standard GRPO14.616.213.2−2.99 [−4.47, −1.75]
Paper's setup15.016.716.3−0.42 [−2.14, +0.96]

Under standard GRPO, the gain is attributable to the swapped labels. Additional rollouts alone increase the flip rate (+1.63 pp [+0.43, +2.93]), whereas TSA decreases it. TSA is below the control in 7 of 9 settings, including both unseen perturbations, with no significant difference in the remaining two; a paired bootstrap yields the same conclusion. TSA also has slightly higher accuracy (63.8% vs. 62.3%). The reduction is in abstention flips: TSA's reversal rate remains at the baseline level (+0.09 pp, n.s.) and slightly above the control (+0.58 pp).

Under the paper's setup, the two cannot be separated: at 8B, neither method lowers the flip rate, consistent with the paper. On Qwen3.5-4B (Qwen3-4B has been withdrawn from the training service), both lower it by similar amounts (TSA − control +0.62 pp [−0.97, +2.51]). We place less weight on these runs, as our 4B recipe reduces this model's accuracy by 6–8 pp and no setting in our pre-registered learning-rate search met the paper's accuracy criterion.

In summary, under standard GRPO the swapped labels reduce flips beyond what a second rollout group achieves, whereas additional rollouts alone strengthen the shortcut. In the revision, we will add the control to every table and state this dependence on the objective. The diagnosis, the amplification by GRPO and TSA's advantage over the reward-based methods do not depend on it.

Regarding the remaining points in W2: every new arm has 5 seeds, and within each experiment all arms share the model, objective, learning rate and sampler, so each comparison varies a single factor. The standard-GRPO experiment directly addresses the 8B optimisation setting, and the label–verdict correlation is discussed in our response to Reviewer KzXT.

We hope that these results address the reviewer's concerns, and we would appreciate it if the reviewer would consider updating the score accordingly.

Thanks for running the two-group original-evidence control and for clarifying the target applications.

  1. On Q1/W1, the explanation about stale or manipulated pipeline metadata suppressing right verdicts, and keeping the verdict tied to the evidence so downstream users can threshold confidence or search separately, makes sense as a design argument for the stricter contract. At the same time, since the pre-swap verdict in those NEI switches was right only 58% of the time at 4B and 46% at 8B, preventing switches to NEI also keeps nearly as many wrong confident verdicts as right ones, while direct SUPPORTS/REFUTES reversals remain unchanged.

  2. On Q2/W2, adding the matched two-group (2 x G=8) original-evidence control directly tests my sampling question, and the new Table R3 shows a real separation on Qwen3-8B once you switch to standard GRPO (-2.99 pp [-4.47, -1.75] against the control, with extra rollouts alone raising flips). However, Table R3 and the 4B follow-up also confirm the attribution problem under the submitted paper's own setup: at 8B under the paper's objective, TSA and the control are still indistinguishable (-0.42 pp [-2.14, +0.96]) and neither beats the baseline, while on Qwen3.5-4B both lower flips by similar amounts (+0.62 pp [-0.97, +2.51]) and the original Qwen3-4B tables still lack a valid matched-rollout separation. In addition, even in the favorable 8B standard-GRPO run, the improvement is again purely in abstention flips, while polarity reversals sit slightly above the matched control (+0.58 pp).

Overall, the rebuttal partially addresses my concerns by motivating the contract more clearly and showing where swapped labels help under a modified 8B objective, so I appreciate the extra experiments. Because the submitted 4B gains remain unseparated from extra rollouts, the 8B effect requires changing the training objective and regenerating the main comparisons, and polarity reversals remain unresolved, I will keep my current rating of 2.

One additional presentation comment on Figure 1: the overview diagram looks quite rough, with unclear connecting lines and arrows, almost as if it came directly from an LLM-generated plotting script without manual cleanup; please clarify and redraw it cleanly in the revision.

Reviewer_KzXTOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Authors investigate the way that a fact checking pipeline exploits the certainty labels that the retrieval modules assign to the retrieved documents. They report that these labels are used for making predictions by the pipeline, but argue that in reality they should be only used for either assigning a certainty level to its predictions or asking for further information to make a more reliable prediction.

To fix the issue, they propose to train the predictor using an augmented version of the data, by taking a copy of the data points that have the certainty of high or low (in its set of retrieved documents) and replacing them with low and high respectively--keeping rest of the data identical. They argue that this way the predictor learns to discard the signal extracted from the certainty labels to make the prediction. The experiments show that the proposed method improves the performance of a model with 4B parameters, but does not scale well to an 8B model.


Update after reading the response: I increased my review score from weak reject to weak accept

Critical strengths
  • The paper proposes a clearly defined problem, investigates it in detail, and proposes a simple solution to it. All the aspects of the work are very clear.
  • The paper is very clear about the weaknesses of the method, and the potential next steps
  • The experiments are detailed
Critical weaknesses
  • The improvements are small, this combined with the fact that the method works only when the model is not large enough, then can be a serious problem
  • The experimental setup is limited in my opinion, only one type of llm and RL are used. The data is also spurious as the authors noted themselves--the correlation between trustworthiness and certainty of the sources and their documents.
Questions for authors

None

6 comments

We thank the reviewer for the positive assessment of the problem definition, the clarity and the experimental detail.

W1. "The improvements are small … the method works only when the model is not large enough."

We examined the 8B result further and found that it depends on the RL objective rather than on model size alone. When Qwen3-8B is trained with standard GRPO at the same learning rate as in the paper, TSA significantly lowers the flip rate, and the reduction is larger relative to the compute-matched control (Table R2).

Table R2. Qwen3-8B, standard GRPO. Change in flip rate (pp), 95% CI.

SettingTSA − GRPO baselineTSA − compute-matched control
Main (AVeriTeC, neutral)−1.37 [−2.83, −0.04]−2.99 [−4.47, −1.75]
CONFACT, neutral−1.62 [−2.50, −0.72]−2.08 [−2.90, −1.25]
Downshift (unseen)−1.48 [−2.46, −0.51]−2.20 [−3.04, −1.48]
Label removal (unseen)−1.09 [−2.29, −0.04]−2.19 [−3.34, −1.19]

TSA maintains accuracy (63.8% vs. 63.9%) and keeps the confidence and search responses at the baseline level. Across all 9 settings, TSA is lower than the baseline in 5 and higher in 1 (SciFact, principled prompt, +0.91 pp), and lower than the control in 7. As at 4B, the reduction comes from abstention flips.

Regarding the size of the gain, among the training-time methods in the paper only TSA and TSA + consistency significantly lower the main-setting flip rate; trust-aware reward shaping and the consistency reward alone do not (§6). Moreover, the avoided flips are often costly: when the 4B baseline switches to NEI after a label change, the original verdict was correct in 58% of cases, and TSA reduces these cases (−0.51 pp [−0.96, −0.10]).

Finally, the first two contributions of the paper, the TrustSwap test and the diagnosis it enables, do not depend on TSA, and the new experiments below extend both.

W2 (model families and RL algorithm). "Only one type of LLM and RL are used."

We evaluated two untrained models from other families. Changing only the labels changes 15–27% of confident verdicts for Nemotron and 4–17% for gpt-oss, with reversals in 11 of 12 model–setting pairs. Confidence and search move in the intended direction in 19/20 and 17/20 comparisons; all exceptions occur on small low-trust CONFACT subsets (14–33 pairs), as does the single exception in the paper. Nemotron's accuracy in our output format is low (20–55%), so, as with Veri-R1 in the paper, we use it only as evidence that the shortcut exists; gpt-oss reaches 62–79%.

For the RL algorithm, standard GRPO raises the 8B flip rate over the untrained model in all 9 settings (main: 10.1% → 14.6%), to a similar degree as the paper's setup (15.0%; difference −0.46 pp [−2.48, +1.48]). The amplification is therefore not specific to our setup.

W2 (training data). "The data is also spurious."

We agree that trust labels and verdicts are correlated in the training set, as stated in §5. However, this correlation does not create the shortcut: every untrained model we tested (Qwen3 4B and 8B, Nemotron, gpt-oss) already changes 4–27% of confident verdicts when only the labels change. The correlation is also realistic, since low-reliability sources carry false claims more often, which is why pipelines attach reliability ratings. RL on real data will encounter this correlation, and TrustSwap measures what training does with it: GRPO amplifies the shortcut at 8B under both objectives. Training on decorrelated data is a natural next step, which we discuss in §7.

We hope that these results address the reviewer's concerns, and we would appreciate it if the reviewer would consider updating the score accordingly.

As the rebuttal phase nears its end, we would like to follow up. We hope our response has addressed your concerns. If so, we would appreciate your consideration of an updated score. If any concerns remain, we would be happy to clarify them. Thank you for your time and thoughtful review.

Thank you for your response, in my opinion my questions were not addressed.

Comments on the paper

2 comments

We thank all reviewers for their careful and constructive feedback. The reviewers found the problem clearly defined and the test well controlled (KzXT, oSgp), considered the distinction between reversals and abstentions central (onKv), and noted that the limitations are clearly reported (KzXT, onKv).

The reviews raised three shared concerns: (i) whether the findings generalise beyond one model family and one RL algorithm, (ii) whether TSA's gain comes from the swapped labels or from the additional rollouts, and (iii) whether a switch to NEI under lower trust should be counted as an error. To address them, we conducted four new experiments and one new analysis, summarised in Table R1 and detailed in the individual responses.

Table R1. New experiments.

ConcernExperimentResult
One model familyTwo untrained non-Qwen models: Nemotron-3.5-Lightning-30B-A3B and gpt-oss-20bChanging only the labels changes 15–27% (Nemotron) and 4–17% (gpt-oss) of confident verdicts.
One RL algorithmQwen3-8B trained with standard GRPOGRPO raises the flip rate in all 9 settings (main: 10.1% → 14.6%). TSA lowers it in 5 of 9 (main: −1.37 pp).
Additional rolloutsCompute-matched control: two groups of G=8 rollouts on the original evidence, as proposed by Reviewer oSgpUnder standard GRPO, TSA is 2.99 pp below the control [−4.47, −1.75] and lower in 7 of 9 settings.
Prompt advicePrincipled prompt without the advice to avoid definitive verdictsThe shortcut persists: flip rate 11.2% untrained, 16.5% after GRPO.
Cost of NEI flipsNew analysis of the paper's runsWhen the 4B baseline switches to NEI after a label change, the original verdict was correct in 58% of cases. TSA reduces these cases (−0.51 pp [−0.96, −0.10]).

Setup. We use the terms of the paper. The flip rate is the share of confident pairs (confidence ≥ 0.7 on either version) whose verdict changes when only the source trust labels change; a reversal is a SUPPORTS↔REFUTES change, and an abstention flip is a change to or from NEI. The main setting is AVeriTeC with the neutral prompt. Standard GRPO refers to the original formulation (Shao et al., 2024): advantages normalised by the group standard deviation, a clipped PPO-style loss and two updates per batch. The paper's GRPO setup centres advantages on the group mean without this normalisation (as in Dr. GRPO; Liu et al., 2025) and performs one update per batch. Every new model is trained with 5 random seeds, and all methods within an experiment share the model, learning rate and greedy decoding. We report differences in percentage points (pp) with 95% bootstrap CIs over seeds. The designs and decision rules were pre-registered before any run. The new 8B runs use the current version of the training service, so absolute numbers differ slightly from the paper's 8B table (e.g., untrained flip rate 10.1% vs. 10.5%); all comparisons are made within the new runs.

Planned revisions. We will (1) add the new model families and standard GRPO to the diagnosis (§4); (2) describe the paper's GRPO setup precisely and report the standard-GRPO results (§5–6); (3) add the compute-matched control to every table and state when the effect of the label swap can be separated from additional rollouts (§6); and (4) add the no-advice prompt and the NEI analysis (§4, §7).

AuthorsOpenReview ↗

Summary of the discussion

We thank the reviewers and the AC. Below is a short summary of the discussion; all changes in the revised PDF are shown in blue.

Strengths noted by reviewers. KzXT: the paper "proposes a clearly defined problem, investigates it in detail, and proposes a simple solution", and "the experiments are detailed". onKv: our argument for the stricter contract "makes sense as a design argument", and the new control "directly tests my sampling question". After the discussion, KzXT raised the rating from 2 to 3, and oSgp raised the informativeness score.

Contributions. (1) TrustSwap, a counterfactual test that changes only the source trust labels and measures the verdict, confidence and search separately. (2) A diagnosis: confidence and search follow the labels as intended, but the verdict also changes for 4–23% of confident Qwen3 verdicts, and GRPO training amplifies this at 8B. (3) TSA, a training recipe that reduces the shortcut. Contributions (1) and (2) do not depend on TSA.

What the discussion added (designs fixed before any run; 5 seeds per model):

  • Generality (KzXT). Two other model families, Nemotron and gpt-oss, show the same shortcut (4–27% of confident verdicts), with confidence and search following the labels in 36 of 40 comparisons. Standard GRPO also amplifies it, in all nine 8B settings (Appendices I, K).
  • Swapped labels vs. extra rollouts (onKv, oSgp). We ran the compute-matched control proposed by oSgp. With standard GRPO at 8B, TSA is 2.99 pp below the control [−4.47, −1.75], while extra rollouts alone raise flips by 1.63 pp (Table 3, Appendix K). onKv noted that this shows "a real separation on Qwen3-8B once you switch to standard GRPO".
  • Value of avoided NEI switches (onKv, oSgp). 58% of the verdicts that the 4B baseline withdraws after a label change were correct, and abstaining because of lowered labels is a weaker selective-prediction rule than the model's own confidence (2.2–3.5 pp lower selective accuracy; Appendix M).
  • Prompt advice (oSgp). The shortcut persists without the prompt's advice to avoid definitive verdicts (Appendix J).

Scope of the claims. TSA's gain is in abstention flips, which make up 78% of flips for the 4B baseline and are the part that GRPO amplifies at 8B (abstention flips double while reversals fall from 4.2% to 2.6%, §4); the paper states this (§1, §6, §7) and reports reversals separately. No training-time method in our study reduces reversals: at 4B, the consistency reward raises them (+0.70 pp, significant; Table 2), while TSA leaves them unchanged (−0.01 pp). Table 3 reports the 8B results under both GRPO objectives in the main text, and the abstract states that the effect at 8B appears with standard GRPO. Because the training service no longer supports Qwen3-4B, the 4B results are compared with the GRPO baseline, as §7 states.

We thank the AC for considering the discussion.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.