acceptodds
Under review as a conference paper at ICLR 2027

What Should Consistency Preserve? Readout-Geometric Information Transport for Semi-Supervised Regression

Abstract

Consistency regularization is deceptively underspecified for scalar regression: matching two views does not determine which task information should remain invariant. We show that this missing choice separates useful unlabeled supervision from agreement that distorts the target coordinate or erases label-predictive geometry. We introduce Readout-Geometry Consistency, a unified framework that factorizes consistency into a readout-induced transport space and a source timescale. A task can therefore transport either its scalar coordinate or protected within-fiber geometry, using an online or temporally persistent source, through the same objective. We derive when coordinate transport reduces target risk, characterize the variance–lag trade-off of temporal sources, and establish first-order prediction invariance for protected tangent updates. Across image age estimation, text rating prediction, and speech quality assessment, RGC establishes new best reported means on two benchmarks and matches leading performance on the third, while controlled interventions confirm the predicted roles of view support, source timescale, and transport geometry.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
semi-supervised regression, consistency regularization, representation learning

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_zkVhOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper argues that consistency regularization for scalar regression is underspecified, since matching two views does not say which task information to preserve. Based on the linear readout, it separates what to match (the scalar prediction, or features with the readout and a label-residual direction projected out) from the target source (the online model or an EMA encoder). Three instances result: output consistency, output consistency with an EMA source, and angular consistency on the projected features. One instance is chosen per dataset, by validation where a validation split exists. With 250 labels, RGC reports the best mean results on UTKFace and Yelp and is competitive on BVCC. Crossover experiments suggest that the right consistency object depends on the task.

Critical strengths
  1. The crossover experiments carry a useful message. The best consistency object depends on the task. On BVCC, where both views share the same audio segment, feature consistency beats output consistency in every seed. On Yelp, unprotected feature consistency instead collapses the representation, with effective rank falling from about 45 to below 4.

  2. The method is also simple, adding no parameters or inference cost. Matched continuations from a shared checkpoint make the within-codebase comparisons credible.

Critical weaknesses
  1. The technical contribution is smaller than the framing suggests. By the authors’ own description, C is ordinary weak–strong output consistency and TC is Mean-Teacher-like. On BVCC, T receives identical waveforms in both views, making it essentially dropout-based feature consistency with two directions projected out. Label protection improves MAE by only 0.0038 (Table 8), and without an unprojected control, it is unclear whether the readout geometry matters. The propositions are elementary and leave the choice of instance to validation.

  2. The empirical gains are fragile. On Yelp and BVCC, the differences from the strongest baselines are within seed noise, and with 50 labels on UTKFace, RGC clearly trails SSR-RCUS. On UTKFace-250, C beats the reported Pi-Model and Mean Teacher by about 3.3 and 2.7 MAE despite being closely related to them. This gap likely reflects pipeline choices, but the baselines were copied rather than re-run.

Questions for authors
  1. How do the UTKFace results change if checkpoints are selected on held-out training data, and did the copied baselines use the same selection convention?

  2. The Yelp collapse in Figures 4 and 5 uses the readout-only variant. Why this variant, and does the label-protected T also collapse?

  3. On BVCC, how does T compare with unprojected feature cosine consistency under the same views? This would show whether the readout geometry matters.

  4. If Pi-Model and Mean Teacher are re-run in your pipeline with identical settings, how much of their roughly 3.3 and 2.7 MAE gaps to C remains?

Reviewer_TeLrOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper asks what information consistency learning should preserve in semi-supervised regression. The authors observe that a scalar readout separates feature space into directions that change the prediction and tangent directions that leave it unchanged. Based on this view, they introduce Readout-Geometry Consistency (RGC), which defines consistency through two choices: transport space and source timescale. The framework includes online output consistency (C), output consistency with an EMA encoder source (TC), and tangent consistency that protects both the readout direction and a label-related direction (T). The paper analyzes conditions for reducing risk in output space, the cost of mismatched input support, the variance–lag trade-off of EMA sources, and local gradient orthogonality in feature space. Experiments on UTKFace, Yelp, and BVCC show competitive performance.

Critical strengths
  • The paper addresses an important problem and offers a useful way to understand consistency learning. The distinction between making predictions agree and making predictions more accurate is especially relevant to regression. Separating shared target support, the information being transferred, and source stability provides a useful guide for method design.
  • The method is simple, and the implementation details are fairly complete. RGC keeps the inference architecture unchanged, and C and T introduce no additional learned parameters. The paper describes the protected basis, stop-gradient operations, numerical precision, warm-up, checkpoint selection, and data splits in detail.
Critical weaknesses
  • The experiments do not fully isolate the benefit of the proposed readout geometry. The BVCC C–T comparison changes several elements together: the transported object, the loss form, feature normalization and scaling, and whether the unlabeled loss directly updates the output head. Appendix D states that T does not update the head through the unlabeled loss, whereas C does. The observed improvement could therefore partly reflect differences in how unlabeled training updates the model. Section 5.5 mentions a recipient-matched control, but its precise definition and numerical results seem missing. Reporting this control would address an important concern. Table 8 provides useful evidence for label-related protection, improving MAE from 0.4534 to 0.4496, but it also increases the protected space from one to two dimensions. A dimension-matched control using a random direction orthogonal to the readout would help establish whether choosing the additional direction based on labels is important. Since C and TC are established forms of consistency learning, isolating the value of the proposed protection is central to the paper’s contribution.
  • The cross-task comparisons use different tangent operators, which limits the claim that different tasks require different geometries. The failure and representation collapse reported on Yelp come from a tangent setting that protects only the readout direction, while the final BVCC method also protects a label-related direction. This comparison does not clearly distinguish whether tangent transport is unsuitable for Yelp or whether it needs the additional label protection. The current evidence directly supports the narrower conclusion that the tested readout-only tangent setting fails on Yelp. The view conditions also require clarification: T assumes exact shared support, while Yelp uses an original review and an augmented review. If the Yelp comparison does not satisfy this assumption, its failure cannot be cleanly attributed to the transport geometry. Evaluating the proposed protected tangent operator under suitable view conditions, or narrowing the claims to the settings tested, would address this concern.
  • The link between local feature-space protection and information preservation during training needs stronger evidence. Proposition 4 correctly establishes that the feature gradient is orthogonal to the readout when the protected basis is detached. However, actual encoder updates also depend on the network Jacobian and the optimizer. For a single sample under simple SGD, the induced feature change is approximately Δh≈−ηJJ⊤∇hLT\Delta h \approx -\eta JJ^\top \nabla_h \mathcal L_T, so gradient orthogonality alone does not guarantee that the prediction remains unchanged. The authors acknowledge this distinction. The remaining question is whether the proposed protection also preserves label-related information more effectively during actual parameter updates. The orthogonality logs in Appendix C mainly verify that the projection works as intended, but this does not directly show whether it better preserves label-related information during actual model updates. Figures 4–5 provide useful evidence about the representations at different training stages. However, because the measurement directions are re-estimated at each checkpoint, they do not directly track changes along the same protected directions across updates. A small diagnostic measuring changes in predictions and protected feature components after one unlabeled update from a common checkpoint would strengthen the connection between the local result and actual training behavior.
Questions for authors
  • Which parameters receive gradients in the recipient-matched control mentioned in Section 5.5? Does it include coordinate consistency with the output head detached? Please provide its per-seed results against protected tangent consistency.

  • Did you evaluate tangent consistency with label-related protection on Yelp? Please also clarify whether its views satisfy the assumptions of T. If these results are available, the performance and presence or absence of representation collapse would be useful.

  • Does the gain in Table 8 depend on choosing a label-related direction? If available, please report a comparison against protecting the readout plus a random orthogonal direction, keeping the protected dimension the same.

  • Have you measured the protection effect after actual parameter updates? Beyond gradient orthogonality, I would like to see how predictions and protected feature components change after an unlabeled update from the same checkpoint.

Reviewer_e4CQOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper proposes a discussion on what should be matched in consistency regularization methods for regression problems. Claiming that existing studies do not clearly suggest what information should be preserved invariantly, the authors propose an integrated framework called RGC that decomposes the consistency being matched into two factors: "transport space" induced from the readout and "source timescale". Within this framework, they propose MSE consistency (C) that matches weak-strong predictions, consistency (TC) that matches predictions using EMA, and protected tangent consistency (T) that gives consistency in the orthogonal subspace rather than the label direction, and train models by selecting among them according to the characteristics of each dataset. In this paper, the performance of the proposed methodology is verified on three datasets: UTKFace, Yelp, and BVCC.

Critical strengths

None.

Critical weaknesses
  1. What this paper proposes are two methods called C and TC, and online protected tangent called T. However, the existing C and TC (as a minor writing issue, it is not well understood by what criteria the abbreviations C, TC, and T were made here, and why weak-strong consistency and the Mean Teacher method have to be called separately like this) are traditional methods in semi-supervised learning similar to weak-strong consistency and Mean Teacher that uses EMA as a teacher, and the authors also mention this in Sec. 3.3. However, simply explaining C and TC in this way does not show much novelty by itself. Furthermore, in the case of the T (online protected tangent) method newly proposed in this paper, it seems to be used in only a single dataset (BVCC). Although this paper claims to present a single unified framework, the proposed method actually just finds the best-performing setting among 3 different methods on each different dataset separately, and 2 of them are already widely used methods.

  2. Another serious problem is the writing issue. In the sections explaining the proposed method (Sec. 3, 4), the paper does not provide justification for the claimed method, there is no citation at all, and it merely lists what their method did. Also, content such as Proposition 1 proposed for that explanation only presents an identity that comes out from expanding a quadratic expression in its proof, and does not present any new finding.

Questions for authors
  • In UTKFace and Yelp where C and TC (which the authors said they used existing methods as is) are applied, the paper obtains considerably higher performance compared to other methods. I wonder where this performance improvement came from, and I am concerned whether dataset-specific individual tuning brought most of the performance gain.

  • If there exists a classification rule that uses each method separately for different datasets like this, a justification is needed on why the training objective was defined separately for each dataset, but that part also does not seem appropriate. Already in Table 1, "candidates" are chosen for each dataset, yet the method containing the newly proposed T is still only one (BVCC), and in Table 5, the Tangent method already seems to diverge on UTKFace. Because of this, there remain concerns about the validity even for the method they propose.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.