acceptodds
Under review as a conference paper at ICLR 2027

Fourier Output for Visual Coordinate Regression

Abstract

Coordinate regression is a fundamental component of many computer vision tasks, yet most existing methods rely on direct scalar regression, coordinate discretization and classification, or task-specific output representations. We investigate whether a common representation can improve coordinate prediction across diverse visual tasks. While Fourier encodings are widely used to represent coordinate inputs, their use as representations for coordinate outputs remains underexplored. We introduce Fourier Output, which represents each scalar coordinate using multi-frequency sine–cosine features and directly supervises predictions in the Fourier encoding space, while leaving the underlying backbone unchanged. To efficiently recover coordinates at inference time, we develop a hierarchical decoding algorithm that progressively resolves phase ambiguity from low to high frequencies without candidate search. Through a controlled patch localization study, we systematically analyze the representation, supervision, frequency configuration, and decoding strategy, establishing a practical design for Fourier Output. We then evaluate it on human pose estimation, homography estimation, optical flow, and vision-language model (VLM)-based localization. Across these diverse tasks, Fourier Output consistently improves prediction accuracy over direct coordinate regression with minimal architectural changes, while training curves show that it can reach comparable accuracy with fewer training updates. These results demonstrate that Fourier features, widely used for representing coordinate inputs, can also provide a simple and general representation for visual coordinate outputs.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Computer Vision, Positional Encoding, Fourier Features, Coordinate Regression

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_2EPzOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a frequency-domain output representation for visual coordinate regression. Instead of regressing a coordinate directly with an MSE loss, the method maps the target coordinate to multi-frequency sine–cosine (Fourier) codes, supervises each frequency component separately with MSE, and then decodes the prediction back to the spatial domain via hierarchical phase unwrapping. The overall design is simple.

On patch localization (Tables 1 and 2), the method outperforms direct regression as well as other regression variants. It also improves over direct regression on several downstream vision tasks (Sec. 5.1: 2D pose estimation, homography estimation, and 3D pose estimation).

Critical strengths

S1. Importance of the problem. Coordinate regression is a fundamental problem in computer vision. Developing output representations that do better than plain MSE supervision is a worthwhile direction, as it can broaden the applicability of regression-based models across domains.

S2. A simple yet sound method. Rather than supervising a single scalar value with MSE, the method supervises each frequency component of the Fourier encoding in the frequency domain and recovers the coordinate through phase unwrapping. The approach is simple but sound, and it has clear potential to extend beyond vision to other coordinate regression problems.

Critical weaknesses

W1. Proposed method is not specific to vision. Although the experiments are conducted on visual benchmarks, nothing in the method itself is tied to the visual domain. This could be viewed as a strength, but for the same reason the method should also be compared against regression techniques developed in other domains.

W2. Insufficient baselines (comparison only with plain direct regression). In the experiments (Secs. 4 and 5), the main baseline is plain direct regression. The reported gains may therefore partly come from the increased number of output channels in the Fourier-based head. A comparison with classification-based approaches (i.e., predicting a probability distribution) that use the same number of output channels is needed.

W3. Missing citation. The paper does not cite the following prior work, which also performs regression by applying losses to individual frequency components:

  • 3D Equivariant Pose Regression via Direct Wigner-D Harmonics Prediction (Lee and Cho, NeurIPS 2024)

Please cite this work and discuss the similarities and differences.

W4. Lack of analysis and missing statistics. Tables 4 and 5 and Figure 3 show interesting results, but the experimental settings and statistics are not described in enough detail. Since the improvements could be due to run-to-run noise, please report the number of runs together with the mean and standard deviation.

In addition, there is no analysis of the qualitative results: what the failure cases are, and in which cases the method works well.

W5. Choice of K. Is K in Sec. 3.2 a fixed hyperparameter? Could learning K (or the frequencies) bring further benefits?

Questions for authors

This method demonstrates its effectiveness across a variety of visual coordinate regression tasks with a simple technique. I find this simple yet effective approach quite interesting and worth exploring further.

However, in terms of the overall completeness of the work, I cannot give an acceptance score at this stage.

I would be willing to raise my score if the authors adequately address W1, W2, W3, and W4 above.

3 comments

(W1. Comparisons with regression methods developed in other domains)

We thank the reviewer for clarifying the scope of Fourier Output. We agree that Fourier output parameterization is not inherently vision-specific. To address the connection to regression methods developed beyond vision, we evaluate HL-Gauss, a histogram-based regression objective studied in Improving Regression Performance with Distributional Losses (ICML 2018), in patch localization experiments. We set the number of output channels to 8, the same as in Fourier Output. Across three seeds, the test MAE is 0.0431 ± 0.0006 for HL-Gauss and 0.0371 ± 0.0022 for Fourier. This indicates that Fourier Output still outperforms a general output representation.

(W2. Comparison with classification output representations)

We design two classification-based output representations for patch localization: a 128-channel one-hot distribution (SimCC-style) and an 8-channel two-hot classification distribution that represents the probabilities of the two nearest bins. As shown in the table, Fourier Output outperforms both classification-based output representations.

RepresentationChannels per axisParametersRunsTest MAE ↓
Direct12,971,58630.0629 ± 0.0018
One-hot classification1283,752,12810.08790.0879
Two-hot classification83,014,60830.0450 ± 0.0004
Fourier83,014,60830.0371 ± 0.0022

(W3. Wigner-D)

Thank you for pointing out this relevant work. We will add a citation to it. We agree that both methods share the general idea of predicting harmonic representations of geometric targets instead of directly regressing their original parameters. However, there are two key differences: 1) Wigner-D coefficients are designed for SO(3) rotations, while Fourier Output is designed for 1D translation; 2) Wigner-D requires a specialized equivariant architecture, while Fourier Output can be incorporated into existing direct coordinate regression architectures.

(W4. Protocols, number of runs, means, and standard deviations)

We reran the experiments with 3 seeds. The improvements persist for 2D pose, homography, optical flow, and VLM localization at 6K updates.

Task / backboneMetricRunsDirectFourier
2D pose / ResNet50PCKh (%) ↑281.98 ± 0.2285.29 ± 0.21
Homography / CNNMACE ↓24.188 ± 0.1553.890 ± 0.047
Homography / ViTMACE ↓22.504 ± 0.2841.585 ± 0.047
Flow / Sintel cleanEPE ↓31.275 ± 0.0161.216 ± 0.020
Flow / Sintel finalEPE ↓32.541 ± 0.0312.416 ± 0.068
VLM / 6K updatesmPCK (%) ↑381.206 ± 0.23884.179 ± 0.110

(W4. Failure cases in qualitative results)

We observe left–right limb confusion in some DeepPose predictions. Fourier Output corrects such errors in some examples where Direct fails, but also exhibits the reverse pattern. These cases suggest that ambiguity in joint identity remains a limitation despite the overall accuracy improvement. The analysis has been added to Appendix C.2 (Figure 6) of the revised paper.

(W5. Choice of K)

KK is a fixed hyperparameter, and we study its effect in the ablation in Section 4.2. We tried a progressive learning algorithm that gradually increases KK during training once learning at the lower frequencies has converged, but it does not improve performance. We will add this information to the final version of the paper.

Reviewer_2EPzOpenReview ↗

Post-rebuttal review of Paper ID 29499

I thank the authors for the detailed response and the additional experiments.

W1: partially resolved. W2: resolved. W3: resolved. W4: resolved. W5: resolved.

The authors have addressed most of my concerns, so I raise my score from 2 to 3.

Thank you for your thoughtful feedback. We are glad that our responses and revisions have addressed your main concerns, and we appreciate your time and consideration throughout the discussion.

Reviewer_MgceOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper studies Fourier codes as output representations for visual coordinate regression. Instead of predicting scalar coordinates, a network predicts multi-frequency sine–cosine features supervised in the encoding space. A hierarchical decoder recovers coordinates through coarse-to-fine phase unwrapping with clamped corrections, avoiding explicit candidate search. The paper evaluates 2D pose, homography estimation, 3D pose lifting, optical flow, and VLM-based localization. The proposed configurations improve over the tested direct-regression baselines and sometimes reach comparable accuracy with fewer training updates.

Critical strengths
  1. The evaluation covers different backbones, input modalities, and prediction densities. Improvements in patch localization, 2D pose, and ViT-based homography are substantial, while the flow and VLM results provide additional evidence of practical applicability with limited architectural changes.

  2. The hierarchical decoder offers a computationally attractive alternative to candidate-search inversion. Experiments examine alternative encodings, supervision paths, frequency counts, and phase clamping.

Critical weaknesses
  1. The incremental contribution over existing Fourier output representations is insufficiently established. SACReg already predicts geometrically spaced multi-frequency sine–cosine coordinate codes. The main advances are therefore the decoder and broader empirical study. However, matched comparisons against SACReg-style training and established coordinate-output alternatives are limited, and decoder accuracy is evaluated using a checkpoint selected with the proposed decoder.

  2. Some improvements are confounded by optimization choices and lack uncertainty estimates. Optical flow changes both representation and loss, while the VLM experiment gives Fourier a specialized initialization without an analogous Direct control. All application results use a single training run despite modest gains in several settings. These experiments support the usefulness of the complete configurations, but do not sufficiently isolate the representation’s contribution or establish the robustness of the accuracy and convergence gains.

Questions for authors
  1. Optimization and statistical controls: Do the gains persist with comparably calibrated output initialization for Direct and Fourier, particularly in the VLM experiment? Please provide representative multi-seed results and an optical-flow ablation separating encoding-space MSE from cosine supervision.

  2. Decoder robustness: Algorithm 1 requires ωmin⁡R<π\omega_{\min}R<\pi, whereas the 2D pose configuration uses ωmin⁡=π/256\omega_{\min}=\pi/256 over [0,256][0,256]. Is centering or additional boundary handling applied? Following the stated decoder, a coordinate of 255.99 with a 0.001-radian perturbation only in its lowest-frequency phase can decode to the opposite boundary despite exact higher-frequency phases. How frequently do branch-selection failures occur on actual predictions?

  3. Decoder evaluation: Do the accuracy comparisons hold when checkpoints are selected independently for each decoder? What are the decoding and end-to-end overheads on actual application predictions, particularly the K=8K=8 optical-flow setting?

4 comments

(W1. SACReg-style training)

The training loss used for the output representations in SACReg is similar to that used in Fourier Output. The differences lie in how we set the hyperparameters, select the training loss, and decode the representation. Compared with the SACReg paper, we systematically study the design space of Fourier Output and validate it on various downstream tasks.

(W1. Alternative output representations on downstream tasks)

We extend the DeepPose comparison to polynomial encoding, multiscale sawtooth encoding, and two-hot classification. Two-hot classification improves over Direct, while Fourier achieves the highest mean PCKh, exceeding the strongest alternative by 2.27 percentage points under this training configuration.

Output representationChannels per coordinate[email protected] (%) ↑
Direct181.98 ± 0.22
Polynomial880.76 ± 0.12
Multiscale sawtooth870.79 ± 0.14
Two-hot classification883.02 ± 0.17
Fourier885.29 ± 0.21

(W2, Q1. VLM Initialization)

In the VLM experiment, the baseline applies a sigmoid to the direct output to ensure that the initial coordinate predictions lie near the center of the coordinate range. Motivated by this, we initialized the Fourier head with a similar affine adjustment. This design improves the convergence speed. However, without this design, Fourier Output still outperforms the direct output baseline.

UpdatesDirect default mPCK (%)Fourier default DDP mPCK (%)Fourier affine mPCK (%)
600081.433982.360484.2946
3238586.020486.113086.8311

(W2, Q1. Representative multi-seed results)

We reran the experiments with multiple seeds. The improvements persist for 2D pose, homography, optical flow, and VLM localization at 6K updates.

Task / backboneMetricRunsDirectFourier
2D pose / ResNet50PCKh (%) ↑281.98 ± 0.2285.29 ± 0.21
Homography / CNNMACE ↓24.188 ± 0.1553.890 ± 0.047
Homography / ViTMACE ↓22.504 ± 0.2841.585 ± 0.047
Flow / Sintel cleanEPE ↓31.275 ± 0.0161.216 ± 0.020
Flow / Sintel finalEPE ↓32.541 ± 0.0312.416 ± 0.068
VLM / 6K updatesmPCK (%) ↑381.206 ± 0.23884.179 ± 0.110

(W2, Q1. Optical-flow ablation of encoding-space MSE versus cosine loss)

We reran the optical-flow ablation using the CroCo ViT-B model, comparing Direct MSE, Fourier MSE, and Fourier cosine.

Output and lossChairs val.Sintel cleanSintel final
Direct + MSE2.1731.8113.280
Fourier + MSE1.7491.3992.558
Fourier + cosine1.3141.3722.649

Fourier cosine gives the best source-domain Chairs validation EPE. Although Fourier MSE is worse on Chairs validation, its accuracy is comparable to that of Fourier cosine on Sintel. Both versions outperform the direct version. Thus, the use of Fourier Output, rather than the specific loss, is the main contributor to the accuracy gain.

(Q2. Decoder robustness)

The reviewer is correct: ωmin⁡=π/256\omega_{\min}=\pi/256 leaves no phase margin at 256. Our implementation does not center coordinates. We found that adding a small margin could avoid this issue. For example, we can set ωmin⁡=π/260\omega_{\min}=\pi/260 to provide a safety margin. We use this safety margin in our implementations.

In practice, even if we set ωmin⁡=π/256\omega_{\min}=\pi/256, this issue rarely occurs. In DeepPose experiments with three seeds, we found only 1, 0, and 0 lowest-frequency branch mismatches among 88,076 valid in-crop coordinates per model.

(W1, Q3. Independent checkpoint selection)

We repeated the patch localization experiment and selected checkpoints independently using each decoder's validation MAE. Hierarchical decoding selects step 2400, while Grid and SACReg-style decoding both select step 2280. Hierarchical decoding retains the lowest test MAE at both checkpoints, including when each decoder uses its own selected checkpoint. Thus, the decoder ranking in this experiment is preserved under independent checkpoint selection.

Selected checkpointHierarchicalGrid searchSACReg-style
Step 2400 (hierarchical-selected)0.037270.038520.03842
Step 2280 (search-selected)0.037570.038680.03859

(Q3. Decoding overhead)

We test decoding accuracy and efficiency using the CroCo v2 ViT-L flow estimator on the Sintel dataset. The results are summarized in the following table. Our hierarchical decoding algorithm achieves the lowest error and the fastest decoding speed.

MethodClean EPEFinal EPESingle Batch Decode Time (ms)End-to-end Time (ms)
hierarchical (ours)2.4894.4680.9081206.857
grid search2.4924.4714.8781607.997
SACReg decoder2.5004.47955.8696133.610

Thank you for the clarifications and additional experiments.

W1: partially resolved; the additional comparisons clarify the incremental contribution. W2 / Q1: largely resolved. Q2: largely resolved; the boundary issue is acknowledged, and its observed frequency is low. Q3: checkpoint-selection concern resolved.

Overall, the rebuttal addresses my main concerns, and I maintain my original score.

Thank you for your thoughtful feedback. We are glad that our responses and revisions have addressed your main concerns, and we appreciate your time and consideration throughout the discussion.

Reviewer_WZsTOpenReview ↗
Rating
4(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
3(+1)
Read review
Paper summary

The paper aims to systematically study the potential of Fourier output parameterization for general coordinate regression task. It improves the Fourier coordinate decoding of SACReg with a hierarchical decoding algorithm. It reports evaluation on various sparse/dense coordinate prediction tasks and shows that Fourier output consistently improves accuracy and training efficiency.

Critical strengths
  1. Simple and effective. This simple output parameterization and hierarchical decoder without too many hyperparameters can be easily applied to various tasks and achieve consistent performance and training efficiency improvement. The hierarchical decoding also improves efficiency.
  2. Comprehensive evaluation. Diverse evaluation shows its generalization to different tasks.
Critical weaknesses
  1. Architecture bias for the patch coordinate regression analysis. The main systematic study is on patch coordinate regression with ViT architecture. However, ViT already uses Fourier position embedding to encode each patch, which might favor outputing Fourier directly instead of other alternatives like Multiscale sawtooth.
  2. Limited ablation to alternative decoding baselines. Not enough ablation to the other output decoding alternatives in the downstream task evaluations, especially the main baseline SACReg decoder.
Questions for authors
  1. Comparison to other alternative encodings in downstream tasks.
  2. Using other architectures like CNN for the patch coordinate regression analysis. A potential probe is overfitting single images with homography augmented views to regress the canonical patch coordinate.
3 comments

(W2. Alternative decoding algorithms on downstream tasks)

We test decoding accuracy and efficiency using the CroCo v2 ViT-L flow estimator on the Sintel dataset. The results are summarized in the following table. Our hierarchical decoding algorithm achieves the lowest error and the fastest decoding speed.

MethodClean EPEFinal EPESingle Batch Decode Time (ms)End-to-end Time (ms)
hierarchical (ours)2.4894.4680.9081206.857
grid search2.4924.4714.8781607.997
SACReg decoder2.5004.47955.8696133.610

(Q1. Alternative output representations on downstream tasks)

We extend the DeepPose comparison to polynomial encoding, multiscale sawtooth encoding, and two-hot classification, using the same eight-channel definitions as in the CNN probe (Q2). Two-hot classification improves over Direct, while Fourier achieves the highest mean PCKh, exceeding the strongest alternative by 2.27 percentage points under this training configuration.

Output representationChannels per coordinate[email protected] (%) ↑
Direct181.98 ± 0.22
Polynomial880.76 ± 0.12
Multiscale sawtooth870.79 ± 0.14
Two-hot classification883.02 ± 0.17
Fourier885.29 ± 0.21

(W1, Q2. Fourier Output may be favored by ViT)

In the main paper, DeepPose and HomographyNet have already been evaluated with CNN backbones. Fourier Output outperforms the direct output baseline. Following R1's suggestion, we further evaluate Fourier Output with a CNN backbone on the patch coordinate regression task. Specifically, to avoid feeding two images into the network, we fit each CNN to a single source image. The inputs are homography-augmented views of patches, and the outputs are the corresponding canonical patch coordinates. We test 5 output representations on 100 images from ImageNet, resulting in 500 trained models in total. The results are summarized in the following table. Fourier Output still outperforms other output representations, which indicates that the advantage of Fourier Output also holds for CNN backbones.

RepresentationKMean ± Std
Direct11.2276 ± 0.1794
Polynomial81.1780 ± 0.1877
Multiscale sawtooth81.2771 ± 0.2614
Two-hot Classification80.6504 ± 0.1113
Fourier80.4471 ± 0.0901

Thanks for the accurate response and additional experiments. I found the CNN backbone patch coordinate regression experiments particularly convincing, as they fully addressed my concerns about the architectural bias, and the baseline performance aligns with my own previous experience in similar exploration. Besides accuracy, the efficiency of the decoding algorithm relative to existing decoders constitutes valuable contributions. Overall, the authors have precisely addressed my concerns and I see no remaining critical weakness preventing acceptance, and I will raise my score.

Thank you for your thoughtful feedback. We are glad that our responses and revisions have addressed your concerns, and we appreciate your time and consideration throughout the discussion.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.