acceptodds
Under review as a conference paper at ICLR 2027

SETSUNET: Shared Consensus Geometry for Few-Step Recovery

Abstract

SETSUNET is a shared learned optimizer for few-step geometric and visual recovery. It learns search directions from local gradient and curvature geometry, projects them to complement a measured base solve, and coordinates their contribution across observation views. This construction allows the same graph network to operate on image variables, transformations and graph states. We establish whitening equivariance for the learned directions and numerical solve, and use it to accelerate the complete solver with Cholesky whitening while preserving physical updates and trained weights. Experiments show better recovery than HSLM, Ceres LM, Dogbox and VeLO on multiframe alignment, stereo and biased-graph estimation, with competitive results on pairwise alignment and image restoration. Learned directions also improve recovery over equal-rank random completion. Together, these results demonstrate that learned search geometry can serve as a reusable component for accurate recovery under small update budgets.

Primary area:
optimization
Keywords:
Learned optimization, inverse problems, subspace methods, geometric deep learning, multi-view coordination, whitening equivariance.

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_jJxXOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

SETSUNET proposes a shared learned optimizer for few-step inverse problems. It first computes a numerical base update in a measured subspace, then uses a small graph network to generate complementary search directions from gradient and curvature information. These learned directions are projected to preserve the base solution’s stationarity and are combined using a two-view minimax consensus rule. The same checkpoint is evaluated across geometric, restoration, stereo, and graph optimization tasks. The paper also establishes whitening equivariance, allowing equivalent Cholesky whitening without retraining.

Critical strengths
  1. This design features a clear mechanism.
  2. The learnable modules are well integrated with the analysis solver.
Critical weaknesses

In visual tasks, the contribution of the learned direction has not yet been fully isolated. A controlled ablation study is needed: remove the learned completion information while retaining the remaining solver components.

Having the same number of outer iterations does not guarantee comparable computational costs. Under the same Jacobian budget, some rankings may change, indicating that the advantage depends on how the budget is defined.

Questions for authors

In the whole-block 3D temporal experiment, the theoretical value for learning the completion term is zero; however, on major visual benchmarks, how often does it produce a non-zero correction? How much does performance decline when it is disabled?

Why does the gain of minimax depend on CG refinement when the bias is stronger, while the mean coordination method actually performs better under asymmetric noise? How should coordination rules be selected in practical applications?

2 comments
AuthorsOpenReview ↗

Response to jJxX

Thanks for asking for strictly matched ablations and equal-budget comparisons; we ran both directly to isolate each factor.

W1, Q1: In Sec. 4.4 and Table R1, we added a 120-case matched ablation that disables only learned completion while keeping the base step, configured CG, finite-step rule, initialization, 5-step budget, and shared checkpoint untouched. Counting states where the pre-CG completion-to-base norm ratio exceeds 10−1010^{-10}, completion is active in 120/120 (multiframe), 160/160 (biased graph), 160/160 (noisy graph), and 159/160 (SE(3)) states, cutting mean error by 2.44%2.44\%, 20.70%20.70\%, 48.84%48.84\%, and 6.66%6.66\%. For visual edge cases (Appendix H, Table R4, Fig. 6), full-block stereo without spectral clipping (H=IH=I, d0=−gd_0=-g) and 8-DOF pairwise HPatches (full-rank base) are structural zero-completion boundaries; in small restoration controls, completion activates in 59/60 states with nearly unchanged mean PSNR (Table R4), while general multiblock gains are isolated in Table R1.

W2: Sec. 4.3, Appendix I, and Fig. 8 report head-to-head comparisons under a matched 21-joint-Jacobian (21J) budget. Under 21J, SETSUNET stays best or practically tied (within 2%2\%) on Low light, Stereo, SE(3), and Blur; over Ceres, it gains 1.873 dB in low-light PSNR and cuts stereo and biased-graph error by 5.83%5.83\% and 4.21%4.21\%. Note that in biased graph, the Jacobian is a preconstructed constant AA, yet counting both per-step references caps us at 10 steps / 20J versus HSLM's 21 steps / 21J, highlighting how derivative-request counts can diverge from actual compute work.

Q2: In Sec. 4.5, Fig. 5, and Appendix G, we crossed minimax/mean with CG2/CG0 using the same checkpoint. Under shared curvature, minimax penalizes view disagreement while CG2 cleans up the remaining mean-model residual: at the two stronger bias levels, minimax+CG2 beats mean+CG2 by 14.36%14.36\% and 12.08%12.08\%, whereas without CG the differences are −3.23%-3.23\% and −0.72%-0.72\%. Because asymmetric random noise also creates view disagreement without systematic bias, mean is better there. In practice, one can fix the rule upfront from the measurement model and validation data; across all main benchmarks, we never switch per case and keep minimax fixed.

I thank the authors for the clarifications and additional experiments. The response addresses my specific questions and strengthens the empirical support. However, the additional evidence does not substantially change my assessment of the paper’s overall contribution and informativeness. I therefore maintain my original rating. I have no further questions.

Reviewer_6gUZOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper propose an optimization flow where instead of a learned second order approach they use first (gradient) and second (curvature) to predict future directions of search. They show it is relevant for multiple setups as a general optimization approach.

Critical strengths

Learning to Optimize in an important and relevant field, where we unfold known algorithms and empower them with learning skills on sub-parts. The results here indeed show improvements on most cases from non-learnable setups to fully learnable second order approach.

Critical weaknesses

My main concern here relates to edge cases and lack of research in those directions. First as you flow through the domain the Hessian changes, which means the memory of historical data is not in our favor. That might explain the cases this methods fail to win. The more we try and learn using this approach for functions that are far from quadratic this approach might fail. Next, the Hessian does not have to be PD or PSD. So positive curvatures will only be in subsets of the directions. Again, that will cause edge cases where this approach will fail. The authors do use a PSD replacement but that also comes with a price of failed cases. I believe more could have been done to explore the limitations of this approach.

Questions for authors

Can you define the scenarios this method fails to win ?

1 comment
AuthorsOpenReview ↗

Response to Reviewer 6gUZ

Thanks for pushing us to directly test changing curvature, negative curvature, and history. Your comments motivated a new set of long-trajectory diagnostics.

History and curvature: As clarified in Sec. 3.4, history only supplies the last accepted displacement as a candidate direction, while curvature is recomputed at the current state every step. Without retraining our shared weights, we ran 50-step trajectories across 5 settings ×\times 8 paired seeds (40 paired cases; Appendix J, Table R3, Figs. 9--10), comparing full SETSUNET, completion-off, history-off, and two full-space trust-region references using either the exact objective Hessian or its positive-definite (PD) replacement. Both references have the same 50-step allowance and retain their actual stopping points. All routes use the observed objectives, derivatives, and initialization, with reference states used strictly for post-solve scoring. The main recovery comparisons retain their five-step and 21-Jacobian budgets.

Changing and nonconvex curvature: Across both changing-positive-curvature settings, SETSUNET reaches the same solutions as the trust-region references. At δ=0.1\delta=0.1, all eight runs satisfy ∥g∥∞≤10−8\lVert g\rVert_{\infty}\leq 10^{-8} by step 11. The nonconvex setting β=1\beta=1 also converges successfully: full SETSUNET reaches the reference solution in all eight runs, with every run satisfying the same gradient criterion by step 12 and a mean RMSE of 0.00343. Disabling completion or history leads to poorer local minima in two and one of the eight runs, respectively, with mean RMSEs of 0.0704 and 0.0301 across the corresponding eight-run groups. Thus, both learned completion and the history carrier contribute to successful recovery in this nonconvex control.

A concrete nonconvex edge case: At β=0.5\beta=0.5, three of the eight full-method runs converge to poorer strict local minima than those reached by the trust-region references. These endpoints satisfy ∥g∥∞≤4.12×10−10\lVert g\rVert_{\infty}\leq 4.12\times 10^{-10} and have positive-definite exact objective Hessians, with λmin⁡≥0.9591\lambda_{\min}\geq 0.9591. The method has therefore reached locally stable stationary solutions, but the recovery differs because the trajectories enter different attraction basins. Convergence to a local minimum does not guarantee recovery of the lower-objective solution found by a reference method. The observed limitation concerns basin selection by the local update scheme, rather than a failure to converge within the trajectory allowance.

History and Hessian replacement: Removing history still leaves three of eight runs in poorer basins at β=0.5\beta=0.5, so this limitation persists without the historical displacement carrier. Likewise, the exact-Hessian and PD-Hessian trust-region references reach the same final recovery across all five settings, although PD replacement slows early progress at β=1\beta=1 (step-5 RMSE 0.00528 versus 0.00449). These controls distinguish the early-progress effect of Hessian replacement from the observed basin-selection gap. Together, the diagnostics show successful convergence under changing and nonconvex curvature, while identifying a specific nonconvex setting where the local trajectories do not reach the better reference basin (Appendix J, Table R3, Figs. 9--10).

Reviewer_38enOpenReview ↗
Rating
3
Confidence
1
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a learned optimizer for visual and geometric inverse problems, especially when only a few optimization steps are available. The main idea is to use a GNN to generate additional search directions from gradient and curvature information. These directions are combined with a base numerical solver through projection and two-view consensus, followed by further refinement.

The authors claim that the same learned optimizer can work across different types of problems. They also introduce whitening equivariance to improve computational efficiency without changing the resulting updates. Several experimental results on image alignment, stereo, image restoration, and graph estimation show improved recovery in several settings compared to existing numerical and learned optimizers.

Critical strengths

I should mention that I am not very familiar with numerical optimization, so my assessment of the technical novelty may be limited. Still, I found several positive aspects of this paper.

S1. I find the idea of learning additional search directions while keeping the conventional numerical solver interesting. It seems like a reasonable way to combine learned optimization with classical methods.

S2. I appreciate that the authors evaluate the same learned model across quite different tasks, including image alignment, stereo, and image restoration. This makes the generalization claim more convincing.

S3. The experiments are fairly extensive, and I find it interesting to see comparison between learned and random search directions. The authors also provide ablations on the consensus mechanism and whitening, which help understand the contributions of different components.

Critical weaknesses

W1. I found the introduction somewhat difficult to follow. Several technical concepts are introduced in a short space, but the motivation behind each component is not very clear. I had trouble understanding what specific limitations of existing solvers the authors are trying to address. A simple example illustrating the problem and the proposed solution would help.

W2. The related work covers several research directions, but I found the discussion somewhat brief. For example, HSLM is introduced as another subspace solver, but I had trouble understanding what makes SETSUNET fundamentally different beyond using learned directions. Similarly, the paper mentions that its projection and consensus mechanisms build on existing ideas, but it was not very clear to me which parts are new.

W3. I am also curious about the two-view decomposition. As I understand it, the observation factors are divided into two groups depending on the task. However, it is not clear how sensitive the performance is to this choice. The experiments also show that minimax coordination helps under systematic bias but performs worse under asymmetric noise. I think some discussion or analysis of when this design is beneficial would strengthen the paper.

Questions for authors

Q1. My understanding is that the base solver already uses gradient and Krylov directions, and the GNN adds directions that complement them. I am curious what kind of information the GNN can capture that the base solver misses. Could the authors give a simple example?

Q2. As I understand it, the two observation views are created by splitting the objective factors, and the splitting strategy differs across tasks. How are these partitions chosen? Would a different partition significantly affect the results?

Q3. In Appendix C, the authors mention that the learned minimax correction becomes zero in the full-block stereo setting. I am a bit confused about this. Does the same happen in the stereo experiments in Table 1? If so, where does the performance improvement come from?

Q4. The model is trained mainly on synthetic SE(3) and bundle adjustment problems, but it seems to generalize to quite different tasks. I find this interesting, but I am not sure why it works. What properties of the learned directions make this transfer possible?

2 comments
AuthorsOpenReview ↗

Response to Reviewer 38en

Thanks for the encouraging review and for highlighting our cross-task generalization and ablations.

W1, W2, Q1: We rewrote the intro contributions and Sec. 2 (Mathematical contributions) and added Figs. 1--2 to make the solver flow and differences from prior subspace methods much clearer. Intuitively, the base step only optimizes inside the measured subspace QQ, leaving residual gradients outside QQ untouched. As a simple example, a single global weight can only scale a carrier (1,1)(1,1) into (a,a)(a,a), whereas our GNN's block-dependent weights can turn it into (1,−1)(1,-1), and curvature-orthogonal projection preserves base-space stationarity. Against equal-rank random directions, this cuts mean error by 48.73%48.73\% on biased graphs and 76.30%76.30\% on noisy graphs.

W3, Q2: Each task adapter fixes the two views from measurement structure as factor groups of a single objective (Sec. 3.3). In controlled partition swaps that keep all factors, weights, and the total objective identical (Appendix H.1, Fig. 7, Table R2), restoration mean PSNR shifts by less than 0.05 dB, stereo and pairwise HPatches stay unchanged at reported precision, and four multiframe cases shift by +4.39%+4.39\% / +9.29%+9.29\%. As analyzed in Sec. 4.5, minimax+CG2 helps under systematic bias while mean works better under asymmetric noise; we keep minimax fixed across all main benchmarks.

Q3: Yes, the stereo adapter in Table 1 shares that single full-state block setup without spectral clipping. Because mean whitened curvature is H=IH=I and the gradient-containing base step already reaches d0=−gd_0=-g, any nonzero increment raises the mean quadratic, making exact minimax completion zero; Table 1 therefore reflects the complete solver (Appendix H, Fig. 6, Table R4, with Sec. 4.7 narrowed to whitening timing). In general multiblock problems (Sec. 4.4, Table R1, 120 cases), completion is active in 599/600 states and reduces mean error by 2.44%2.44\%, 20.70%20.70\%, 48.84%48.84\%, and 6.66%6.66\% across the four task groups.

Q4: Transfer works because the GNN reads normalized geometric features and outputs fixed-size mixing weights while the carriers themselves supply the state dimension (Sec. 3.2, Fig. 2). Shared block weights and graph message passing let one checkpoint handle varying block and graph sizes, beating equal-rank random completion by 16.40%16.40\% / 33.09%33.09\% / 24.85%24.85\% at 48 / 192 / 768 variables. We also clarified the training mix in Sec. 3.6 and Appendix B; excluding the three radiometric pretraining sequences keeps our HPatches practical tie (2%2\% margin) and lowest multiframe mean error (Appendix F, Table R5).

I appreciate the authors for their detailed response. I found the revisions to the manuscript and the clarifications in the response helpful. I will remain my positive score. I'll finalize my justification after reading the discussions with the other reviewers.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.