acceptodds
Under review as a conference paper at ICLR 2027

Useful Directions and Useful Partners: Auditing Paired Gradient Subspaces Keywords

Abstract

Pairing related inputs can suggest directions for adapting a classifier. To measure what the original matching contributes, we hold each input's loss gradient and label fixed, shuffle partners within labels, and refit the subspace of gradient differences. Random directions provide a different comparison. We show that identical gains over random references can accompany positive, zero or negative pairing effects on retained target-gradient energy. This ambiguity remains even when random directions lie in the space where gradients can occur. Across visual and multilingual dialogue settings, energy advantages over this restricted random reference are common. Evidence that the original partners improve energy retention is selective, and accuracy gains are rarer. Compared with support-matched random directions, dialogue adaptation also incurs class-balanced prediction costs that ordinary accuracy can hide. The audit separates the quality of the directions, the contribution of the original partners, and the value of updating along those directions.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
gradient subspaces, partner-shuffle audit, distribution shift, test-time adaptation, random controls, identifiability

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_9rDPOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper asks whether performance gains can be attributed to the original pairing when shuffling sample correspondences does not reduce performance. The authors design experiments to compare the original pairing with within-label partner shuffles and random subspaces. Theoretical and empirical results show that gains over random directions do not establish a pairing benefit. Pairing-specific gains are not common, particularly regarding prediction accuracy. Overall, the gradient subspace can be useful even when the original sample correspondence is not.

Critical strengths
  • The paper provides extensive experiments to support its theoretical claims, which are performed across visual and multilingual dialogue tasks.
  • The proposed testing strategy is convincing, which performs shuffle control on pairings to test whether performance gains are still preserved.
Critical weaknesses
  • The paper writing should be improved. The introduction should describe the meaningfulness of your research topic by discussing some related works, but no references are included in the introduction section. It is hard for readers to identify the paper’s contributions.
  • Limited reference papers are included. Only around 20 related papers are shown in the paper.
  • Although the paper presents an interesting diagnostic finding, it offers limited solutions or implications on how to design better pairings or improve the resulting gradient subspaces.
  • The AI Use Statement is not included in the paper, which should be a section to describe your AI usage in producing this paper.
Questions for authors

How well does the proposed audit extend to more complex visual or multilingual tasks, like object detection, image generation, or multilingual text generation?

4 comments

Thank you for finding the testing strategy convincing and the experiments extensive.

Writing. The contributions are stated in the abstract and Section 1, paragraphs 2–4. The introduction states the question first; Section 2 follows immediately with the literature on gradient subspaces, control studies and test-time adaptation.

References. The number of references is not a measure of quality. The paper cites the work it builds on in each area, including SubGD, PGrad, GaLore, LDAdam, saliency sanity checks, probe control tasks, TTT, Tent, CoTTA and EATA. If a specific relevant work is missing, we would be glad to add it.

Implications. The paper gives two: a gain over random directions should not be reported as evidence that pairing helps, and the calibration identifies when pairing does help, namely when partners share a nuisance that subtraction cancels.

Extensions. The audit needs only per-example gradients for each pair, a grouping to shuffle within, and target gradients, all of which are available for detection or generation losses.

AI Use Statement. The revised PDF includes a dedicated AI Use Statement section.

We hope these clarifications lead the reviewer to reconsider the rating.

Thank you again for your review. We hope our response has addressed your concerns. As the discussion period is coming to an end, we would be glad to answer any remaining questions. If our response has resolved your concerns, we would greatly appreciate it if you could reconsider your assessment of our paper in light of it.

Thank you for the rebuttal. It was helpful but ultimately did not change my assessment. I am keeping my score.

Thank you for your response. We respect your decision, but we are still unclear about which specific concerns remain unresolved after our rebuttal and manuscript revisions.

We would appreciate some clarification on the following points:

AI Use Statement: We have added a dedicated section to the revised manuscript. Is this concern now resolved? Implications: We explicitly clarified the implications of our findings, including when pairing provides genuine benefits. What additional implications or solutions do you believe are necessary? Extensions: We explained the general requirements for extending the proposed audit to detection and generation tasks. Do you see any specific methodological limitations that prevent these extensions? Writing and Related Work: Could you identify which specific contributions remain unclear or which relevant prior works are missing? We have made a genuine effort to address each concern raised in the original review. Therefore, it would be helpful if you could identify the particular aspects of our response that you find insufficient, rather than simply reiterating the original assessment.

If there are no remaining substantive concerns beyond those already addressed, we respectfully ask you to reconsider whether maintaining the original rating is fully justified in light of the revised manuscript and our discussion.

Thank you for your time and consideration.

Reviewer_NRG8OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies subspaces built from pairs of inputs, such as an image and an altered copy of it, or a prompt and its reply. For each pair, the authors compute the loss gradient of both inputs, take the difference, and fit a low-rank subspace to these differences. This subspace is then used to adapt the classifier at test time.

The question is whether the original pairing actually matters, or whether the two collections of gradients alone would give the same result. To test this, the authors propose a "partner-shuffle audit." They keep all gradients and labels fixed, shuffle the partners within each label, refit the subspace at the same rank, and compare how much held-out target-gradient energy each subspace retains. They also compare against two random baselines: random directions in the full space, and random directions restricted to the space where gradients can actually lie.

On the theory side, the paper breaks the gain over random directions into three parts:

  • a geometric part;
  • a part that survives shuffling;
  • a part due to the original pairing.

A proposition then shows that the same gain over random baselines can coexist with a positive, zero, or negative pairing effect. So random baselines alone cannot tell whether the pairing helps.

The experiments cover 18 settings: three OOD-CV shifts and three vision models, plus three languages and three models for OpenAssistant reply-quality prediction. The fitted subspaces usually beat support-matched random directions (16 of 18). However, the effect of the original pairing is mostly unclear: 2 positive, 4 negative, and 12 unresolved. Pairing almost never improves accuracy under a fixed pseudo-label adaptation rule. A synthetic experiment with a built-in pairing benefit shows that the audit can detect such a benefit when it exists.

Critical strengths

1. The paper makes a useful distinction and backs it with a sensible control.

The paper separates two claims that are easy to mix up:

  • "the fitted directions are useful";
  • "the original pairing is what made them useful."

Shuffling partners within labels while keeping the gradients fixed is a reasonable way to test the second claim. The paper also explains clearly why shuffling across labels answers a different question.

The support-matched random baseline is especially informative. For ResNet-18, random directions in the full space keep 0.8% of the target energy, random directions in the gradient support keep 44.2%, and the fitted subspace keeps 46.4%. This shows how much of a typical "better than random" result comes from geometry alone. Proposition 1 then shows that even the fairer baseline cannot reveal whether the pairing helps. I checked the construction in the appendix, and it is correct. This is the main lasting value of the paper.

2. The reporting is careful and honest.

The authors report all 18 settings, including negative results and a model that performs poorly (German OPT-125M). They treat intervals containing zero as unresolved, not as evidence of no effect. The corrected preprocessing study shows that small pairing effects remain even when shuffling clearly changes the subspace. The grid of update sizes rules out the simple explanation that the updates were too small. Together with the synthetic check and the stability analyses, this makes the results trustworthy as descriptions of what was measured, even where I disagree with how much they show.

Critical weaknesses

1. The practical importance of the audit is not established.

The audited pipeline is quite specific: an uncentered SVD of paired-input gradient differences, used as a fixed subspace for pseudo-label adaptation. The related work (SubGD, PGrad, GaLore, LDAdam) uses low-dimensional gradient structure in other ways, but none of these methods builds subspaces from paired-input differences. The paper does not point to an existing method that uses this pipeline, or to a published claim about pairing that the audit would change.

An audit does not need to overturn prior work to be valuable. But it does need either a real application or a convincing new setting in which the pairing has a clear reason to matter.

The theory does not fill this gap. The decomposition and Proposition 1 correctly establish the warning that random baselines cannot answer the pairing question. They say little about when the shuffle audit will be informative in practice, or how its result should guide a decision. The burden therefore falls on the experiments, which leads to the second weakness.

2. The experiments give the pairing little chance to matter, so the mostly unresolved results are hard to interpret.

  • Vision. The update is a feature offset, so it only adds a constant shift to the class logits. This can move decision thresholds, but it cannot change how features respond to context, shape, or weather. The paper also gives no reason why a fixed brightness/color change should be related to these three shifts.
  • Dialogue. The prompt gradient is computed using the reply's quality label, from a classifier that only ever sees replies. It is unclear why the difference between prompt and reply gradients should help adapt a reply-quality classifier to a new language.
  • Missing baselines. The construction set in dialogue is labeled target-language data, so the reply gradients alone may explain much of the energy advantage. The paper does not report fits to the first members alone, the partners alone, or the pooled gradients. These are the most direct tests of what the paired-difference construction adds.
  • Limited uncertainty. The intervals come from only three training seeds and never resample target examples. The evaluation sets are small (160 German and 152 French replies). As a result, mostly unresolved intervals are weak evidence that pairing gains are actually rare.

Because of this, the main empirical message ("pairing effects are selective, and accuracy gains are rarer") cannot be separated from a design that gives the pairing little opportunity to help, missing baselines, and limited statistical power. This is the main reason for my recommendation.

Questions for authors

Clarifications and analyses that could change my assessment

  1. Is there an existing method, or a clear task-based reason, that makes the image–view or prompt–reply pairing useful for the chosen adaptation parameters? An application where the audit changes a real conclusion would strongly affect my rating.
  2. At the same rank and in the same coordinates, how do the observed and shuffled fits compare with fits to the first members alone, the partners alone, and the pooled gradients? In dialogue, how much of the energy advantage remains when fitting only to the labeled target replies?
  3. Since the SVD is uncentered, the overall mean difference (and, within labels, each label's mean difference) is the same for every shuffle. What share of the construction energy and target energy comes from these mean components? How do the pairing contrasts change after removing them?
  4. In the synthetic calibration with zero partner correlation, the observed and shuffled difference matrices have the same distribution, so the average contrast over repeated constructions should be exactly zero. Why do two of these intervals exclude zero? Could you check the coverage of the full interval procedure by repeating the whole experiment many times at the same seed count? Given the small evaluation sets, what size of accuracy effect can your setup actually detect?
2 comments

Thank you for verifying Proposition 1 and for describing the reporting as careful and honest. The review lists Q1–Q4 as analyses that could change its assessment; all four are answered below, and all support the paper's claim.

Q1, W1. As the review notes, none of SubGD, PGrad, GaLore or LDAdam "builds subspaces from paired-input differences". Yet paired inputs are common in practice, and the natural evidence for a paired construction is a gain over random directions, which Proposition 1 shows cannot establish a pairing benefit. Our experiments show this is a real risk: the paired subspace beats support-matched random directions in 16 of 18 settings, while the audit and the new unpaired baselines show the advantage comes from the member gradients. The audit changes the conclusion here. To our knowledge, it is the first test that separates the two claims.

W2. We chose links that exist without target labels, an image with its altered view and a prompt with its reply, because an audit should test natural pairings, not ones picked to succeed. Restricted-parameter updates are standard in test-time adaptation; Tent, for example, updates only normalization affine parameters. The energy measure is also computed before any update, so the update form cannot explain the result.

Q2. See the table in our response to Reviewer ySEb: the paired fit is never better than the pooled fit, with 0 better, 10 worse and 8 unresolved. In dialogue, target replies alone retain more energy than the paired fit for MiniLM, 32.5 vs 28.2%, and OPT-125M, 85.9 vs 80.9%, and the same for the lexical MLP, 99.0 vs 99.3%.

Q3. Label-mean differences carry 17.6–63.0% of construction energy in vision and 8.5–51.9% in dialogue. After removing them and refitting, energy counts move from 2/4/12 to 3/6/9 and accuracy counts stay at 1/0/17. The conclusions do not depend on the means.

Q4. The expected contrast at ρ = 0 is zero, as the reviewer says; our sentence referred to the measured contrast of a single finite construction, and the revised PDF now states the expectation explicitly. Over 1,000 full repetitions, the mean contrast is between −8.1×10⁻⁵ and 3.0×10⁻⁵, so the two positive intervals are chance outcomes, not bias. Percentile intervals cover zero in 92.2–93.3% of runs and t intervals in 94.5–95.8%; the revised PDF recommends t intervals and reports all of these analyses in a new appendix. Detectable accuracy effects are mostly 0.2–1.6 pp, as given in our response to Reviewer ySEb.

We respectfully ask the reviewer to reconsider the rating.

Thank you again for your review. We hope our response has addressed your concerns. As the discussion period is coming to an end, we would be glad to answer any remaining questions. If our response has resolved your concerns, we would greatly appreciate it if you could reconsider your assessment of our paper in light of it.

Reviewer_ySEbOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies whether the usefulness of gradient subspaces constructed from paired examples actually depends on the original matching. It introduces a partner-shuffle audit that preserves the gradients and labels while changing their assignments. The theoretical analysis shows that advantages over random directions cannot establish the contribution of pairing. Experiments find useful gradient structure, but limited additional benefits from correct matching, particularly for prediction accuracy.

Critical strengths
  1. Clear scientific distinction. The paper separates the usefulness of extracted directions from the usefulness of the relationships used to construct them.

  2. Well-controlled comparisons. Partner shuffling preserves the individual gradients, labels, and subspace rank, allowing a focused test of the original matching.

Critical weaknesses
  1. Most results leave the benefit of pairing uncertain. In most experiments, the results are compatible with correct pairing being slightly better or slightly worse than shuffled pairing. This does not establish that the difference is too small to matter. With only three training runs and uncertainty estimates that do not account for new test samples, stronger evidence is needed to determine how small the pairing benefit actually is.

  2. The paper does not clearly explain why correct pairing rarely helps. There are two possible explanations: the original relationships contain little useful information, or the chosen adaptation procedure cannot effectively use that information. The experiments do not clearly distinguish these possibilities. Showing a realistic case where correct pairing helps, and identifying what makes that case different, would provide a stronger scientific explanation.

  3. Missing an unpaired-gradient baseline. The paper should compare its directions with those extracted directly from individual gradients, without forming differences. Since correct and shuffled pairs often perform similarly, this would clarify whether the pair-difference construction adds value over a simpler approach.

Questions for authors
  1. Can you add a baseline that extracts the same number of directions directly from all individual gradients, without forming pairwise differences?

  2. Under the same adaptation procedure, does your paired construction outperform this simpler baseline in target-gradient energy retention and prediction accuracy?

2 comments

Thank you for recognizing the distinction and the controlled comparison. We respectfully disagree that the paper has serious soundness problems; each weakness is now answered with new evidence that supports the paper's claims.

W3, Q1, Q2. At the same rank and coordinates, we fit subspaces to first members, partners and pooled gradients. Mean target-energy fractions, in percent:

ModelPairedFirstPartnersPooled
ResNet-1846.447.046.046.4
tiny CNN63.265.265.966.3
descriptor MLP79.180.480.980.9
MiniLM28.232.032.533.2
OPT-125M80.983.985.985.9
lexical MLP99.399.199.099.4

Against pooled gradients the paired fit is 0 better, 10 worse, 8 unresolved. This independently confirms the paper's thesis: the paired subspace beats random directions in 16 of 18 settings, yet the advantage comes from the member gradients, not the pairing. A random-reference comparison alone would have reached the wrong conclusion, as Proposition 1 predicts.

W1. With target examples added to the bootstrap, 90% intervals for the accuracy pairing effect lie within [−0.53, +0.84] pp in vision and [−0.73, +1.11] pp in dialogue. Minimum detectable effects at α = 0.05 and 80% power are mostly 0.2–1.6 pp. Any pairing effect on accuracy is therefore bounded to about 1 pp.

W2. Target energy is measured on the directions before any update, so it does not depend on the adaptation rule. At that level the pairing already adds nothing over the member gradients, so the adaptation rule is not the bottleneck. The calibration shows that the audit does detect real pairing information, at 99–100% for ρ = 0.25 or 0.5 and τ from 0.25 to 2, and when it arises: when partners share a nuisance that subtraction cancels.

We ask the reviewer to reconsider the soundness score and rating.

Thank you again for your review. We hope our response has addressed your concerns. As the discussion period is coming to an end, we would be glad to answer any remaining questions. If our response has resolved your concerns, we would greatly appreciate it if you could reconsider your assessment of our paper in light of it.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.