acceptodds
Withdrawn from ICLR 2027

JudgeBlind: Cross-Encoder Feedback Aliasing in GNN-to-LLM Pipelines

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

GNN-to-LLM pipelines expose a cross-encoder gap: a GNN judge approves graph-dependent feedback, while an LLM student learns from raw tokens through a separate encoder. We show that graph–text edits can preserve the judge's complete non-textual decision transcript—exactly for discrete fields and within fixed componentwise tolerances for continuous fields—while harming student learning. The problem is whether unchanged approval decisions imply unchanged training effects. JudgeBlind studies this boundary through a shared attack–diagnosis–defense mechanism. FC-CEFA constructs transcript-invariant paths with graph-side curvature correction; MC-CMCT diagnoses correctability and surviving student exposure; CT-CAFT suppresses that exposure over continuation tubes under IT/PT coverage constraints. Rank and Taylor analysis characterize exposed directions and second-order closure, yielding a cubic relaxed-state judge residual. A conditional finite-round bound separates probe mismatch from student-update approximation error. On five TAG benchmarks and two victims over five seeds, FC-CEFA attains pre-cap gate-pass rates of 73.8–90.0%. Against the matched first-order control on Cora, it raises feasibility from 51.4% to 90.0% and budget-normalized invariant feedback harm from 0.24 to 0.53. MC-CMCT predicts corrector convergence with AUROC 0.87–0.92. CT-CAFT reduces adaptive harm by 52.2–60.0%; on Cora, it improves over the margin control by 16.7 percentage points while halving clean-accuracy loss from 1.4 to 0.7 points. These results distinguish approval stability from learning stability in heterogeneous feedback pipelines.

Primary area:
learning on graphs and other geometries & topologies
Keywords:
Large Language Model, GNN-to-LLM Pipelines, Graph Learning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_wDBYOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies whether graph-dependent feedback can remain unchanged from a GNN judge’s perspective while becoming harmful for an LLM trained on the underlying text. The attacker edits node text and graph edges while preserving all discrete selection and labeling fields and keeping continuous transcript fields within fixed tolerances. The proposed framework combines a second-order attack construction, a diagnostic for correctability and student exposure, and a defense that reweights feedback under class-wise coverage constraints. Experiments span five text-attributed graph datasets and two feedback pipelines over five seeds. The reported results show improved attack feasibility over matched first-order controls and reduced downstream harm under the proposed defense.

Critical strengths

The threat model isolates a meaningful failure mode. Preserving selected IDs, labels, memberships, and continuous decision fields is substantially more specific than merely preserving prediction accuracy. The paper also explicitly excludes the raw text from the invariant record, which makes the claim interpretable.

The matched controls and mechanism ablations are useful. On Cora, adaptive FC-CEFA increases pre-cap feasibility from 51.4% to 90.0% and unnormalized harm from 0.117 to 0.221 over the first-order control (Table 1). Table 2 separately removes mixed-curvature correction and graph-side Newton refinement. These comparisons provide more direct evidence for the construction than the failures of native attacks that were never optimized for transcript preservation.

Critical weaknesses

The central empirical claim depends on a discrete realization step that is not sufficiently explained. Theorem 5.2 establishes a local result for relaxed states; sparse binary edge edits and actual token substitutions need not retain that cancellation. Appendix A.3.1 describes a top-budget projection and tokenizer-aware beam search, but does not specify the text relaxation, candidate construction, or how the graph correction is recovered after discretization. Tables 1–2 nevertheless report worst-case transcript drift near (10^{-15}) with high acceptance rates. The paper states that the complete gate is checked, but a concrete realized example and an independently recomputed fieldwise transcript would make this unusually strong result much easier to assess. This matters because discrete feasibility is the main bridge from the geometric argument to the claimed vulnerability.

The defense evaluation does not sufficiently specify the adaptive optimization. CT-CAFT suppresses exposure over a tube generated by the same attack family used for evaluation. The paper acknowledges this scope, but it is unclear how attacks are regenerated as defensive weights change, whether gradients account for the defense, and how the inner search is checked for convergence. Theorem 5.3 does not resolve this concern: it assumes an independent student-update approximation bound (\xi_t), which is not numerically established. Consequently, the reported 52.2–60.0% harm reduction is promising, but the current description leaves uncertainty about how strongly the defense was challenged.

Questions for authors

Could you provide one successful discrete attack with the original and edited text, exact edge toggles, and a fresh recomputation of every transcript field? Please clarify whether the judge’s text features are recomputed from the edited raw text and explain how the reported near-machine-precision agreement survives discretization.

What is the precise adaptive attack procedure against CT-CAFT? Are continuation states and directions regenerated after defensive weight updates, does optimization differentiate through or otherwise account for the defense, and does increasing the search budget materially change the reported harm reduction?

In the 20-round co-teaching setting, is each attacked transcript compared against an independently evolving clean trajectory, or against an unedited evaluation at the current attacked model state? How are the token and edge budgets accumulated across rounds when computing IFH?

Reviewer_tDuYOpenReview ↗
Rating
2
Confidence
1
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

NA

Critical strengths

NA

Critical weaknesses

NA

Questions for authors

NA

Reviewer_jaKnOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies GNN-to-LLM pipelines such as GNN-as-Judge, where a GNN judge selects and labels the data used to fine-tune an LLM student. Since the two models use different text encoders, the paper asks whether text and edge edits can keep every judge decision unchanged while harming what the student learns. The attack, FC-CEFA, moves along directions that leave the judge's outputs unchanged to first order and uses a graph-side correction to cancel the second-order drift. Its edits must then pass a strict check on all judge outputs, called the transcript gate. The authors also propose MC-CMCT, a diagnostic that predicts when the correction works, and CT-CAFT, a defense that reweights the fine-tuning examples to reduce the student's exposure. With a Llama-3-8B student on five graph benchmarks and five seeds, the paper reports gate-pass rates of 74 to 90%. On Cora, the attack more than doubles the budget-normalized harm (IFH) of a first-order baseline. Across settings, the defense reduces harm by 52 to 60%.

Critical strengths

The paper studies a useful question for LLM training with automated feedback. In pipelines like GNN-as-Judge, a GNN judge decides which examples an LLM student is fine-tuned on and with which labels. The paper asks whether an attacker can change those examples without changing any of the judge’s decisions. For example, a node’s text could be rewritten so that the judge still accepts it with the same label, while the student is fine-tuned on the rewritten text. If such edits can harm the student, checking a filter’s decisions says little about what the approved data teaches the student.

The strongest part of the paper is that it goes beyond reporting an attack. The attack, the diagnostic and the defense are all built around one precisely defined constraint, the transcript gate, which requires every judge output to stay unchanged. The baselines must pass the same gate, so the comparison is fair. All methods use five seeds and are tuned only on validation data, and Table 2 removes the correction steps one at a time. I also appreciate the paper’s honesty about the limits of its theory. It says that Theorem 5.2’s cubic bound on the judge’s remaining drift covers only relaxed states, not the discrete edits, and that the defense bound in Theorem 5.3 holds only under its stated assumptions.

Critical weaknesses

However, I do not find the evidence sufficient to support the paper's claims about its attack mechanism and its defense.

My main concern is that the role of graph-side correction in the realized edits is not yet shown. The paper defines Max Drift over the states the predictor-corrector keeps. In Table 2, it falls from 10⁻⁸ to 10⁻¹⁵ as correction terms are added, which looks like a Taylor residual on relaxed states. A token edit or edge toggle is a discrete jump, so these values say little about the realized edits unless the drift was recomputed on them. The paper does not say it was. This matters because a realized edit passes the gate only if the judge barely responds to it. If the judge reads bag-of-words features, as in GNN-as-Judge [1], reordered words leave its input unchanged. The harm could then come from such edits rather than from correction. Table 2 does show that correction raises the pass rate from 55% to 90%. But its tangent+gate row, with no correction, already reaches an IFH of 0.41. The first-order control reaches 0.24 and the full method 0.53, so most of the gain appears before any correction. As written, the paper shows that edits passing its gate can raise the student's loss, but does not convincingly demonstrate that graph-side correction is responsible for that harm. Residuals on the realized edits and the promised edit counts would help settle this. Simple controls are also missing, such as random edits at the same budgets that pass the same gate.

My second concern is that the defense results are not yet convincing. CT-CAFT's tube is a set of states produced by FC-CEFA, and the defense is evaluated only against that attack. This is fair for an attack-specific defense, but the paper does not say whether evaluated paths are kept out of the tube or whether the attack is re-optimized against the defended student. Re-optimizing the attack is the standard test for a defense [2], and the paper's white-box attacker has the access to do it. Figure 3(b) also places seven of the nine defenses 0.5 to 0.9 points away from their clean-accuracy changes in Table 9, and the seed-level logs are not included. A re-optimized attack and per-seed results are needed before the reported reduction can be taken as evidence of robustness.

Overall, I think the paper asks an interesting and practical question, but its mechanism and defense need more evidence. The residuals and edit counts should already exist. The random-edit control and a re-optimized attack on Cora should fit in the discussion period. If the realized edits support the claim that correction drives the harm, and FC-CEFA clearly beats the random-edit control, I would raise my score. The defense would also need to hold against the re-optimized attack, or its claim should be narrowed to the evaluated paths. If simple edits that leave the judge's input unchanged explain the harm, that blind spot would itself make a useful paper.

References:

[1] Xu and Ding (2026). GNN-as-Judge: Unleashing the Power of LLMs for Graph Learning with GNN Feedback.

[2] Tramer et al. (2020). On Adaptive Attacks to Adversarial Example Defenses.

Questions for authors
  1. Can the authors clarify whether Max Drift is computed on the realized candidates or on the relaxed states?

  2. What does the judge encoder compute from the node text? One realized example showing the judge’s input features, along with edit counts by type, would help.

  3. Do the passing attacks still pass the gate without their graph-side edits? If not, my first concern would be much more dismissible.

  4. Could the authors explain how Table 2's tangent+gate row differs from the first-order control? As an extra analysis, it'd be great to add random edits at the same budgets, filtered by the same gate.

  5. In what order are the attack and CT-CAFT optimized, and are tube states reused at evaluation? If the attack were re-optimized against the defended student, my second concern wouldn't be as serious.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.