acceptodds
Under review as a conference paper at ICLR 2027

Representation Learning for Exact Preimages

Abstract

Modern neural predictors can model highly nonlinear maps, but many scientific and engineering tasks require reasoning in the opposite direction: given a performance or safety level, the goal is to characterize the preimage, that is, the complete set of inputs which meet the desired target level and optimize over that set. For expressive neural predictors, however, such preimages typically have no explicit representation and are expensive to recover or optimize over. This creates a fundamental three-way challenge between expressive forward prediction, accurate preimage approximation, and tractable optimization over the preimage for downstream tasks. We introduce TRIO (tractable representations for preimage learning and inverse optimization), a framework for learning representations that make these objectives compatible by construction. Our key contribution is a preimage factorization: the forward model remains expressive through nonlinear radial transformations (including neural networks), while, under inversion, each transformation reduces to a single scalar radius, which yields simple geometric level sets. This yields an explicit geometric representation that is reusable for downstream optimization over the preimage, and, for linear objectives, we show that this admits a closed-form global solution. We finally prove a universal approximation theorem which shows that TRIO can approximate any continuous forward map and its entire family of potentially disconnected, nonconvex preimages arbitrarily well. Hence, TRIO combines expressive forward modeling, exact preimage recovery, and tractable global downstream optimization over preimages by design.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
preimage learning, inverse optimization, structured neural predictors, invertible neural networks, geometric deep learning, global optimization, universal approximation, interpretable representations

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_wfPGOpenReview ↗
Rating
3
Confidence
1
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes TRIO, a representation-learning framework whose predictors retain expressive nonlinear forward modeling while admitting exact finite-union-of-ellipsoids preimages.

Critical strengths

This paper is generally clear-written. It is easy to follow the motivation and the technical proposal, which are pretty matching, in the introduction. And the math formulation is natural and clear.

The experiments are also reasonably complete, covering forward accuracy, preimage quality, computational benefit, sensitivity to backbone complexity, and an AC power-flow application, and giving good support to the claims.

Critical weaknesses

The practical scope is still somewhat limited. Most is low-dimensional or structured, while the universal approximation rate scales as O(Q−1/d)O(Q^{-1/d}), leaving scalability to genuinely high-dimensional inverse problems unclear.

The richer anisotropic/neural radial parameterization is not sufficiently isolated from the much simpler isotropic quadratic subclass that already enjoys the paper’s main universality and preimage guarantees.

Questions for authors
  1. How much empirical benefit comes specifically from learned anisotropic metrics and expressive radial functions compared with the isotropic quadratic TRIO subclass at matched QQ or parameter count?

  2. For the bounded-domain optimization experiments, can the authors clarify whether the reported solutions retain exact global-optimality guarantees once the ellipsoidal preimages are intersected with additional domain constraints?

4 comments

Thank you very much for your positive review of our paper! We improved our paper as a result of your comment, and highlighted all changes in blue color in our revised rebuttal PDF.


Responses to Weaknesses:

W1: Scalability

Thank you for raising this important point. We would like to address your concern from three complementary perspectives, together with two new experimental studies.

(i) The approximation rate is a worst-case guarantee (and it does not show the scaling behavior in practice). The constructive rate O(Q−1/d)O(Q^{- 1 / d}) is derived using a highly restricted subclass of isotropic quadratic components. It does not exploit the learned anisotropic geometries or expressive nonlinear radial transformations available in the full TRIO architecture. => Consequently, while the bound correctly reflects the difficulty of approximating arbitrary high-dimensional functions, it does not imply that individual high-dimensional problems necessarily require exponentially many components. The relevant practical complexity depends strongly on the geometry of the target function and its preimages.

(ii) New architectural ablations demonstrate this directly. To quantify this distinction, we conducted a new ablation at fixed Q=64Q=64. Here, we compare isotropic quadratic, anisotropic quadratic, and the full neural radial TRIO.

The results show that learning anisotropic geometries reduces test RMSE by 75.93% and increases mean ground-truth preimage IoU from 0.717 to 0.898. Adding expressive neural radial transformations further reduces RMSE by 31.38% and improves IoU to 0.939. Overall, the full architecture achieves an 83.49% reduction in RMSE relative to the isotropic quadratic subclass on which the theoretical approximation bound relies. These results demonstrate that the architectural flexibility omitted from the constructive bound yields substantial practical benefits at finite QQ.

(iii) New dimensional scaling experiments demonstrate competitive performance through d=64d=64. We additionally introduce a new multimodal high-dimensional benchmark with highly complex, nonlinear sinusoidal interactions, and increase the input dimension from d=2 up to d=64.

With only Q=16Q=16 components fixed across all dimensions, TRIO is highly competitive in forward accuracy compared to an unrestricted MLP, and consistently outperforms the ICNN baseline. Moreover, mean ground-truth preimage IoU remains above 0.89 even for d=32d=32.

However, we also explicitly acknowledge the practical limitation found here: with fixed Q=16, preimage IoU decreases to 0.760.76 at d=64d=64. This shows that increasingly complex high-dimensional preimages require additional representational capacity. Nevertheless, the experiments demonstrate that the worst-case dimensionality dependence is not necessarily representative of practical performance, and that TRIO can remain competitive with unrestricted neural predictors even at much higher dimensions.

Action: We add the new architectural ablation (new Table 3) and the new dimensional scaling experiment (new Figure 7) to our revised Section 5. We further provide full experimental specifications in the Supplement, and explicitly discuss dimensional scaling in our new Limitations section.

W2: Benefits of anisotropic geometry and expressive radial transformations

Thank you for this excellent suggestion! We conducted a new ablation experiment, which provides valuable insights into the different contributions of learned (i) anisotropic geometries and (ii) expressive neural radial transformations.

Specifically, we compare three TRIO variants on the Six-Hump Camel benchmark at Q=64Q=64, with identical training configurations as in our main studies: (i) the restricted isotropic quadratic subclass from Theorem 4.3, (ii) learned anisotropic quadratic components, and (iii) the full anisotropic neural radial TRIO. We evaluate forward test RMSE and mean ground-truth preimage IoU across target levels.

MethodTest RMSE ↓Mean IoU ↑
Isotropic quadratic TRIO0.36954 ± 0.147450.71650 ± 0.05574
Anisotropic quadratic TRIO0.08893 ± 0.001730.89756 ± 0.00916
Neural radial anisotropic TRIO0.06102 ± 0.023510.93850 ± 0.02736
Improvement vs. isotropic quadratic83.49% reduction30.98% increase
Improvement vs. anisotropic quadratic31.38% reduction4.56% increase

The results demonstrate two important findings:

  • (i) Learned anisotropic geometry substantially improves finite-component expressiveness. Compared with the isotropic quadratic subclass, learning anisotropic metrics strongly improves both forward accuracy and preimage reconstruction. This confirms that adapting the shape and orientation of individual components is highly beneficial at finite QQ, even though the simpler subclass already satisfies our universality theorem.
  • (ii) Neural radial transformations provide additional, substantial improvements. Moving from anisotropic quadratic to neural radial TRIO further reduces forward RMSE by 31.38% and improves preimage IoU by 4.56%.

Importantly, the latter corresponds to an approximately 40% reduction in the remaining IoU error (1-IoU). Thus, this improvement is meaningful even when preimage reconstruction is already highly accurate. The results highlight that expressive radial transformations further improve forward modeling and the target-dependent refinement of preimages, but do not sacrifice their exact geometric representation.

These findings demonstrate that the richer parameterization offers substantial practical benefits beyond the theoretical universality guarantees, particularly at finite model capacity.

Action: We add the new ablation study in our new Table 3 in our revised Section 5, to highlight the benefits of anisotropic geometries and expressive radial transformations.


Responses to Questions:

Q1: Benefits of anisotropic geometry and expressive radial transformations

Thank you. We kindly refer to our answer W2.


Q2: Global optimality under bounded-domain constraints

Thank you. We would like to clarify that global optimality is preserved under the additional domain constraints considered in our experiments. Importantly, the exact optimization decomposition in Equation (9) holds for arbitrary input domains X\mathcal X and objectives JJ, since

inf⁡x∈PFθ(g)J(x)=min⁡r∈A(g)inf⁡x∈Er(g)∩XJ(x).\inf_{x\in\mathcal P_{F_\theta}(g)} J(x)=\min_{r\in\mathcal A(g)}\inf_{x\in\mathcal E_r(g)\cap\mathcal X} J(x).

where Er(g)\mathcal E_r(g) denotes the full ellipsoid in Rd\mathbb R^d. Thus, global optimization over the complete learned preimage remains equivalent to independent regionwise optimization, which includes all domain constraints.

In our experiments, the additional domain constraints are convex: the unit disk for the complex-power benchmark and the box for AC power-flow. Since ellipsoids are convex, their intersections with these domains remain convex. Furthermore, the corresponding objectives are convex (Euclidean projection and linear optimization). Hence, each regionwise problem is a convex optimization problem and can be solved globally, which yields the global optimum over the complete, potentially nonconvex learned preimage.

The closed-form expression in Theorem 4.2 assumes X=Rd\mathcal X=\mathbb R^d, but the exact geometric decomposition and global-optimality remain valid for the bounded domains considered here.

Action: We clarify the above in our revised Section 4.2.

Reviewer_wfPGOpenReview ↗

Response to Authors

The authors have addressed my concerns. I remain slightly positive. Thank you.

Dear Reviewer wfPG,

Thank you very much for your detailed review. We appreciate your thoughtful feedback, which has helped us improve our work.

We are glad that we could address your questions and concerns, and we would appreciate it if you would consider increasing your score.

Thank you again for your time!

Reviewer_7sMEOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
3(+1)
Presentation
3
Soundness
2(+1)
Read review
Paper summary

The authors introduce TRIO (tractable representations for preimage learning and inverse optimization): an inverse optimization framework. They tackle the problem of minimizing an objective J:X→RJ : \mathcal{X} \to \mathbb{R}, subject to the restriction f(x)≤αf(x) \leq \alpha. Here, f:X→Rf : \mathcal{X} \to \mathbb{R} is an unknown function. Since approximating α\alpha-sublevel sets of ff can be intractable, the authors propose a representation f~\tilde{f} that is defined as a minimum over anisotropic radial components. The key property of TRIO representations is that sublevel sets of f~\tilde{f} are a union of ellipses with explicit geometries.

The authors also provide two theoretical results. First, they show that their representation can approximate any continuous function f:X→Rf : \mathcal{X} \to \mathbb{R}. Then, they prove that a linear objective JJ minimized over a sublevel set of f~\tilde{f} admits a global closed-form solution.

Critical strengths

The main strength is the closed-form solution for linear objectives. By construction, sublevel sets are the union of ellipses and thus the optimization problem can be broken down into simpler ones.

Critical weaknesses

The experiments are a weak point of the paper. The two 2D experiments do not establish good generalization capabilities: the first one uses too many points on the training set, and the level sets of the second one are approximately unions of ellipses. The 5D (Pandapower) experiment is better, but the claimed real-world application is too strong for the proposed simulation and it does not establish the scaling of the method to higher dimensions. This is particularly concerning because of the proposed ellipsoidal geometry: as nn gets larger, the volume of an nn-sphere when compared to a hypercube gets very low and thus it is hard to cover a toroidal space using ellipses or spheres.

Six-Hump Camel Benchmark (2D)

In this experiment, the authors use the six-hump camel function as the regression target f:[−1,1]2→Rf : [-1, 1]^2 \to \mathbb{R}, which is a six-degree polynomial with two inputs. They used a training set of 100 thousand points to compare TRIO with neural network baselines, which is an unusually large number of points for such a simple function. Under these circumstances, traditional picewise-linear models from computer graphics should also be included in the comparison.

Complex Powers (2D)

In this experiment, the authors used f(x)=Re(e−iα(x1+ix2)p)f(x) = Re(e^{-i\alpha} (x_1 + i x_2)^p), for p=4,⋯ ,12p = 4, \cdots, 12 (defined on the unitary disk) as the target. These are functions on the unit circle with dihedral symmetry, whose level sets look like the union of oval shapes restricted to the unit disk. Since TRIO produces ellipses as the the sublevel sets, these experiments are not representative of general target functions.

Questions for authors

You refer to PREMAP2 as state-of-the-art post-hoc reconstruction method, and yet its IoU on the Six-Hump Camel benchmark only reached .8. This is a simple problem (6 degree polynomial with 2 dimensions) with a large training set (n = 100 000). Can you explain why you chose PREMAP2 as a baseline and why its performance is so poor?

How does TRIO scale to higher dimensional spaces? What are its main limitations?

5 comments

Thank you very much for your helpful review of our paper! We improved our paper as a result of your comment, and highlighted all changes in blue color in our revised rebuttal PDF.


Responses to Weaknesses:

W1: Ellipsoidal geometry and higher-dimensional scalability

Thank you. The decreasing volume ratio between a high-dimensional sphere and its enclosing hypercube does not imply that ellipsoidal preimages become inherently inefficient in higher dimensions. We clarify this distinction and added two new experimental studies to demonstrate the practical expressiveness and scalability of TRIO.

(i) Ambient volume does not determine preimage representation complexity. The sphere-to-hypercube argument is misleading because it conflates covering an ambient domain with representing a particular sublevel set. We would like to explain this with a very simple example:

Consider f(x)=∣x∣22f(x)=|x|_2^2 on X=[−1,1]d\mathcal X=[-1,1]^d. Its preimage at g=1g=1 is represented exactly by a single TRIO component (Q=1), in any dimension, even though the sphere-to-cube volume ratio tends to zero as d→∞d\to \infty.

More generally, TRIO learns the centers, orientations, and anisotropic geometries of its components to represent the relevant preimage. It does not try to cover the input domain with fixed spheres. The relevant complexity depends on the geometry of the target function and its preimages, not simply on ambient dimension or volume ratios.

(ii) New architectural ablations confirm the benefits of expressive geometries. We conducted a ablation for a fixed Q=64Q=64 in our new Table 3. Here, we compare isotropic quadratic, anisotropic quadratic, and full neural radial TRIO. Learning anisotropic geometries reduces test RMSE by 75.93% and improves mean ground-truth preimage IoU from 0.717 to 0.898. Adding expressive neural radial transformations further reduces RMSE by 31.38% and increases IoU to 0.939. Overall, the full architecture achieves an 83.49% RMSE reduction relative to simple isotropic quadratic TRIO.

=> This demonstrates that learned geometry and nonlinear radial transformations substantially improve finite-capacity expressiveness, which is different to what fixed spherical components can capture.

(iii) New dimensional scaling experiments confirm competitive performance in higher dimensions. We introduce a new multimodal benchmark with highly nonlinear sinusoidal interactions, and evaluate TRIO for dimensions d=2,4,8,16,32,64d=2,4,8,16,32,64.

With only Q=16Q=16 components fixed across all dimensions, TRIO has competitive forward RMSE compared to a parameter-matched unrestricted MLP, while consistently outperforming the ICNN baseline. Moreover, mean ground-truth preimage IoU remains above 0.890.89 through d=32d=32. This demonstrates that accurate preimage representations remain possible even with very few components.

Naturally, performance deteriorates as dimensionality increases under a fixed component budget, with preimage IoU decreasing to 0.750.75 at d=64d=64. Nevertheless, these experiments provide direct evidence that higher ambient dimension does not inherently require prohibitively many ellipsoidal components. Practical scalability depends strongly on the structure and geometric complexity of the target problem.

Action: We added the new architectural ablation (new Table 3) and high-dimensional scaling experiments (new Figure 7) on a new nonlinear DGP to our revised Section 5, with complete specifications in the Supplement. Additionally, we include the geometric counterexample and an explanation distinguishing ambient volume from preimage complexity in Supplement A.2, and discuss practical scalability in our new Limitations section.

W2: Motivation for the Six-Hump Camel training size

Thank you. The choice of 100,000 training observations is deliberate for three purposes:

(i) Reliable qualitative evaluation. Our first goal in this benchmark is to study the geometry of learned preimages and the effectiveness of their reconstruction. In this qualitative study, abundant training data reduces the influence of finite-sample estimation errors, which allows us to analyze the qualitative behavior of TRIO under low sampling variability, instead of confounding preimage reconstruction with errors due to sampling variance.

(ii) Parameter efficiency and preimage recovery with lower sample size are studied separately. In Figure 5 and Table 3, we systematically compare TRIO against parameter-matched MLPs and ICNNs under identical training budgets. In Figure 5, we examine how forward accuracy changes with model capacity. Hence, this provides complementary evidence on finite-capacity approximation efficiency for equally well trained methods (TRIO vs ICNN vs MLP). Further, we study how forward and backward accuracy change with task complexity (Table 3). This experiment is performed with moderate training size of 1638416384. => Importantly, TRIO converges quickly to the unrestricted MLP performance (Figure 5) for increasing parameters, and beats all baselines for increasing task complexity (Table 3).

(iii) Isolating exact versus post-hoc preimage recovery. Importantly, Table 2 distinguishes two fundamentally different metrics: certified coverage measures how accurately a method reconstructs its own learned predictor's preimage, whereas Truth IoU measures agreement with the ground-truth preimage. The former is a geometric reconstruction metric and can be evaluated independently of whether the predictor generalizes well. TRIO achieves exact recovery of its learned preimage by construction, whereas PREMAP2 only approximately reconstructs the MLP's preimage, even at substantial computational cost.

Since both TRIO and the MLP are trained on the same abundant data, this comparison isolates the benefits of exact preimage representation over post-hoc recovery, without confounding reconstruction quality with forward estimation error.

In sum, we deliberately chose high nn in these studies to isolate three different effects: (i) qualitative behaviour without confounding through sampling noise, (ii) parameter efficiency, (iii) preimage recovery without confounding through forward errors.

All other experiments (i.e., complex powers (n=16.384n=16.384), pandapower (n=12.000n=12.000) and our new scaling experiments (n=10.000n=10.000)) have moderate training sizes.

Action: We clarify the deliberate training size of the six-hump camel dataset in our revised Section 5.


W3: Benchmark preimages

Thank you. None of the benchmark preimages are ellipsoidal: We respectfully disagree that the complex-power benchmarks have ellipsoidal level sets. With z=x1+ix2=rexp⁡(iθ)z=x_1+ix_2=r \exp (i \theta), the underlying raw functions are

f_p(r,\theta)=r^p\cos(p\theta-\alpha),\ \qquad p\in{4,6,8,10,12},\$$ where $\alpha = 25^°$ is a fixed phase shift. Their level-set boundaries therefore involve **higher-order angular oscillations,** and **not** quadratic ellipsoidal contours. As illustrated in **Supplement Figure 8**, they have strongly nonconvex, multimodal, and disconnected geometries, with substantial changes across target levels (e.g., see the cone-shaped level sets for the 0.5 quantile). Similarly, the Six-Hump Camel benchmark **also** has nonconvex and disconnected sublevel sets with nonlinear boundaries (see **Figure 4**), and so do panda-power and the new scalability dataset. Hence, **neither benchmark has preimages that are ellipsoids by construction.** Instead, approximating these complex geometries through learned unions of ellipsoids is precisely the representation-learning task TRIO is designed to solve. **Action:** In our **revised Section 5**, we clarify the non-ellipsoidal structure of the experimental benchmarks, and highlight the complex powers **Figure 8 in Supplement C.1.**

Responses to Questions:

Q1: PREMAP2 baseline and preimage recovery

Thank you. We chose PREMAP2 because it is a recent state-of-the-art method for certified post-hoc preimage approximation of trained neural networks. We would like clarify two important points:

(i) Training-set size versus preimage recovery: The 100,000 training observations are used only to train the forward predictors (TRIO and the parameter-matched MLP), not by PREMAP2 during preimage reconstruction.

Schematically:

  • TRIO: use data to learn forward predictor → get exact preimage of predictor at basically no cost.

  • MLP + PREMAP2: use data to learn predictor (the MLP) → use PREMAP2 as post-hoc approximation to get some part of the predictor’s preimage.

As explained in W1, we deliberately use abundant training data to reduce forward estimation errors for both methods, and to isolate the computational cost and accuracy of backward preimage recovery both with respect to the trained predictor (“certified coverage”) and the ground truth preimage (“Truth IoU”). This choice is deliberate to isolate two different effects.

(ii) Budget-dependent certified reconstruction: PREMAP2 computes certified approximations of the trained MLP's preimage with linear relaxations and iterative domain partitioning. Its reconstruction is therefore intentionally conservative; regions that cannot yet be certified are excluded, and increasing coverage requires additional refinement at additional cost.

Importantly, PREMAP2 operates on the learned nonlinear MLP, not the original six-degree polynomial, so the simplicity of the analytic data generating function does not eliminate the difficulty of post-hoc neural network inversion.

Table 2 demonstrates precisely this computational trade-off: with a refinement budget of B=20248B=20248, PREMAP2 reconstructs approximately 80% of the learned preimage in 228.29 seconds, while TRIO represents 100% of its learned preimage in approximately 0.01 seconds, i.e., several orders of magnitude faster.

Higher refinement budgets may further improve PREMAP2's coverage, but require additional computation. This is precisely the fundamental advantage of TRIO: exact preimage representation is guaranteed by construction, and not progressively approximated through computationally expensive post-hoc refinement.

Action: We clarify the interpretation of PREMAP2's budget-dependent certified coverage in our revised Section 5.


Q2: Scalability

Thank you. We kindly refer to our answer W1.


Q3: Limitations & scalability

Thank you. We added a new limitations paragraph in our revised Section 5, where we discuss both (i) scalability to high dimensions and (ii) strategies for stable training.

Our new ablation study in Table 3 demonstrates that learned anisotropic geometries and expressive radial transformations substantially improve forward and preimage accuracy at fixed Q, highlighting the practical benefits of TRIO's architectural flexibility.

Furthermore, our new scalability experiments in Figure 7 demonstrate that TRIO remains highly competitive through d=64d=64, even with only Q=16Q=16 components. Ultimately, forward and preimage accuracy can deteriorate in higher dimensions, and parameter-matched unrestricted MLPs may achieve slightly better forward accuracy in some settings. This reflects the trade-off between unrestricted predictive flexibility and TRIO's complementary advantages: exact, explicit learned preimages and tractable downstream optimization.

Finally, we acknowledge that training can be sensitive to component initialization, which we mitigate through targeted farthest-point initialization and soft-minimum warm-up strategies.

Action: We have added: (1) new higher-dimensional scalability experiments in Figure 7, (2) new architectural ablations in Table 3, and (3) an explicit limitations discussion in revised Section 5. We further detail our training strategies in Supplement C.3.

Reviewer_J43qOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

TRIO learns a predictor that simultaneously supports three goals: expressive forward prediction, exact and explicit representation of its preimages, and tractable global optimization over those preimages. The model scores each input as a minimum over learned radial components, where every component applies a monotone scalar transform to a learned Mahalanobis-like distance. Because each transform is monotone, inverting the predictor at any target level reduces to inverting a scalar, so every preimage is an exact finite union of ellipsoids (Thm 4.1). Optimizing over this nonconvex preimage then decomposes into independent per-ellipsoid problems, which for linear objectives admit a closed-form global solution (Thm 4.2). A universal approximation result (Thm 4.3, with a constructive rate) further shows that any continuous function and its full family of preimages can be approximated arbitrarily well, already with shared quadratic radial functions.

Critical strengths
  1. A clean structural idea with exact guarantees that survive contact with practice. The min-of-radial-components construction is simple, the exactness theorem is training-independent, and the empirical results confirm the predicted decoupling: downstream optimization time is flat across a 400× range of radial-backbone sizes. Theory and experiment tell the same story, which is rare.
  2. Convincing end-to-end validation on a real engineering problem. The AC power-flow study is exactly the right test: TRIO is the only method simultaneously globally optimal (w.r.t. the surrogate) and AC-feasible on all runs, at 7 ms/solve. This demonstrates the practical stakes of exact preimages better than any synthetic benchmark could.
Critical weaknesses
  1. Scalar-output restriction and dimension scaling. TRIO handles only scalar maps f: X→R; multi-output constraints (the norm in engineering: several simultaneous safety/performance specs) are not addressed, and the constructive rate O(Q^{-1/d}) signals a curse of dimensionality. Experiments are low-dimensional (2D benchmarks; effectively 5 decision variables in the power-flow study). The paper would benefit from an honest treatment of where the approach breaks.
  2. Exactness is inherited from surrogate quality. The structural guarantee is exactness w.r.t. the learned preimage; nothing protects a user when the surrogate is wrong in low-data regions (safety-relevant regimes are precisely where data is scarce). Some discussion of calibration/conformal-style safeguards, or at least an explicit warning, would temper a reading where "exact preimage" is over-trusted for safety screening.
Questions for authors
  1. Can the construction extend to vector-valued f (e.g., intersection of per-output unions of ellipsoids)? Does anything fundamental break?
  2. How does Q need to scale with d in practice — do you have experiments beyond d≈5–10, or guidance on when ellipsoidal unions become too coarse?
  3. Training uses plain ERM; is optimization of the min-ensemble stable? Any diagnostics on active-component counts (Fig. 4 suggests many components stay inactive)?
5 comments

Thank you very much for your positive review of our paper! We improved our paper as a result of your comment, and highlighted all changes in blue color in our revised rebuttal PDF.

Responses to Weaknesses:


W1a: Extension to multi-output constraints

Thank you. The scalar-output formulation is not a restriction of TRIO. Our framework easily extends to multiple simultaneous output constraints, while preserving all of its key theoretical properties.

To show this, consider a vector-valued map f:X→Rmf:\mathcal X\to\mathbb R^m with constraints fj(x)<=gjf_j(x)<= g_j for j=1,...,mj=1,...,m. We can learn one TRIO predictor FjF_j per output, each of which admits the exact ellipsoidal preimage representation from Theorem 4.1. The joint preimage is then

\bigcap_{j=1}^{m}\mathcal E_{j,r_j}(g_j).\$$ Hence, **the joint preimage remains exact and explicit by construction**, now as a finite union of intersections of ellipsoids. Moreover, **Theorem 4.3 extends componentwise**, i.e., for any continuous vector-valued target, TRIO can approximate all outputs simultaneously, and **preserves the joint preimage inclusions** $$\mathcal P_f(\mathbf g)\subseteq\mathcal P_{F}(\mathbf g)\subseteq\mathcal P_f(\mathbf g+\varepsilon)$$ Importantly, **the global optimization decomposition from Equation (9) also extends**: $$\inf_{\substack{x\in\mathcal X\ F_j(x)\leq g_j,\ \forall j}} J(x)\\$$ $$=\inf_{x\in\mathcal P_F(\mathbf g)}J(x)\\$$ $$=\min_{\mathbf r\in\prod_{j=1}^{m}\mathcal A_j(g_j)} \left( \inf_{x\in\bigcap_{j=1}^{m}\mathcal E_{j,r_j}(g_j)}J(x) \right).$$ Thus, global optimization over the joint preimage reduces exactly to independent regionwise problems. Crucially, the nonlinear radial transformations still factor out, and downstream optimization operates entirely on the resulting geometric representation. **Action:** We discuss the extension to multi-valued targets in our **revised Sections 3 and 4. We also added a formal Theorem with a proof in our new Supplement A.4**.

W1b: Dimension scaling

Thank you for raising this important point. We would like to address your concern from three complementary perspectives, together with two new experimental studies.

(i) The approximation rate is a worst-case guarantee (and not the actual scaling behavior in practice). The constructive rate O(Q−1/d)O(Q^{- 1 / d}) is derived using a highly restricted subclass of isotropic quadratic components. It does not exploit the learned anisotropic geometries or expressive nonlinear radial transformations available in the full TRIO architecture. Consequently, while the bound correctly reflects the difficulty of approximating arbitrary high-dimensional functions, it does not imply that individual high-dimensional problems necessarily require exponentially many components. The relevant practical complexity depends strongly on the geometry of the target function and its preimages.

(ii) Our new architectural ablations demonstrate this directly. To quantify this distinction, we conducted an ablation at fixed Q=64Q=64. Here, we compare isotropic quadratic, anisotropic quadratic, and the full neural radial TRIO.

The results show that learning anisotropic geometries reduces test RMSE by 75.93% and increases mean ground-truth preimage IoU from 0.717 to 0.898. Adding expressive neural radial transformations further reduces RMSE by 31.38% and improves IoU to 0.939. Overall, the full architecture achieves an 83.49% reduction in RMSE relative to the isotropic quadratic subclass on which the theoretical approximation bound relies. These results demonstrate that the architectural flexibility omitted from the constructive bound yields substantial practical benefits at finite QQ.

(iii) New dimensional scaling experiments demonstrate competitive performance through d=64d=64. We additionally introduce a new multimodal high-dimensional benchmark with highly complex, nonlinear sinusoidal interactions, and increase the input dimension from d=2 up to d=64.

With only Q=16Q=16 components fixed across all dimensions, TRIO is highly competitive in forward accuracy compared to an unrestricted MLP, and consistently outperforms the ICNN baseline. Moreover, mean ground-truth preimage IoU remains above 0.89 even for d=32d=32.

However, we also explicitly acknowledge the practical limitation found here: with fixed Q=16, preimage IoU decreases to 0.760.76 at d=64d=64. This shows that increasingly complex high-dimensional preimages require additional representational capacity. Nevertheless, the experiments demonstrate that the worst-case dimensionality dependence is not necessarily representative of practical performance, and that TRIO can remain competitive with unrestricted neural predictors even at much higher dimensions.

Action: We add the new architectural ablation (new Table 3) and the new dimensional scaling experiment (new Figure 7) to our revised Section 5. We provide the full experimental specifications in the Supplement, and explicitly discuss dimensional scaling in our new Limitations section.

W2: Exactness and ground-truth guarantees

Thank you for pointing this out! We would like to clarify two points:

(i) Exactness of the preimage refers to the learned predictor, as is standard in surrogate-based learning. Since the true function ff is unknown and only observed through training data, no method can guarantee exact recovery of the ground-truth preimages without additional assumptions.

Our contribution is different: TRIO guarantees that the complete preimage of the learned predictor is represented exactly, without post-hoc approximation errors. This is a structural guarantee (Theorem 4.1), independent of training accuracy. We now highlight the difference to ground-truth approximation in a new Remark our revised Section 3.

(ii) Forward approximation directly controls ground-truth preimage approximation. Importantly, our theory shows a rigorous connection between forward prediction accuracy and preimage accuracy:

In Supplement A.3, we prove that whenever the learned forward predictor satisfies a uniform forward error bound sup⁡x∈X∣Fθ(x)−f(x)∣≤ε\sup_{x\in\mathcal X}|F_\theta(x)-f(x)|\leq\varepsilon, we obtain, simultaneously for every target level gg, the preimage guarantee

\mathcal P_{F_\theta}(g-\varepsilon)\subseteq \mathcal P_f(g)\subseteq\mathcal P_{F_\theta}(g+\varepsilon).\$$ Thus, **forward approximation errors translate directly into explicit, target-level bounds on the entire family of preimages**. Moreover, the theorem in **Supplement A.3** shows that optimizing over the tightened learned preimage $\mathcal P_{F_\theta}(g-\varepsilon)$ guarantees ground-truth feasibility. Hence, this provides a computable bound on global suboptimality whenever a valid forward error bound is available. Of course, such bounds are not automatically guaranteed by empirical risk minimization (as is the case in **any** predictive ML scenario where the ground-truth $f$ is unknown). For safety-critical applications, they must be supplemented by appropriate safeguards. In our AC power-flow experiment, we already employ validation-based conservative threshold calibration (**Supplement C.6**). **Action:** In our **revised Section 3**, we emphasize the distinction between structural exactness and ground-truth accuracy in a **new Remark**, and highlight the connection between forward approximation and ground-truth preimage recovery in **Supplement A.3**. ____ ### Responses to Questions: > **Q1: Vector-valued $f$** Thank you. We kindly refer to our answer **W1a.** ____ > **Q2: Scaling of $Q$** Thank you. We kindly refer to our answer **W1b.** ____ > **Q3: Optimization and active components** Thank you. We would like to clarify two points: (i) TRIO is trained using **standard** empirical risk minimization, with a **dedicated initialization and training strategy** to ensure stable optimization. (ii) The number of active components in Figure 4 is **threshold-dependent by construction, and should not be confused with inactive components during training**. **(i) Training stability:** We use two complementary strategies to ensure stable training: - a) First, we employ a **mixed low-target initialization**: half of the component centers are initialized using farthest-point sampling across the entire training domain. This does not require any oracle knowledge or test points and ensures good initial coverage of both the overall input space, which is particularly relevant for preimage reconstruction. - b) Second, we initially train using a **differentiable soft minimum**, whose temperature is gradually annealed, before we switch to the exact hard minimum. This helps gradient propagation across components during early training. Importantly, all TRIO models are evaluated using the exact hard minimum, and our experiments across ten independent seeds demonstrate consistent predictive and preimage performance (see **Section 5**). **(ii) Threshold-dependent component activation:** The seemingly inactive components in Figure 4 are **an intended consequence of the exact preimage factorization by TRIO, not because of training failure**. By **Theorem 4.1**, a component contributes to the preimage at level $g$ only if the learned $\beta_r$ satisfy $\beta_r \leq g$. Consequently, low target levels activate fewer components, while increasing $g$ activates additional regions of the same trained predictor. For example, in **Figure 4**, with $Q=256$, the number of active components increases from **61 at $g=-0.1$ to 130 at $g=0.9$**, while preimage IoU stays stable at **0.981 and 0.993**, respectively. Thus, **the number of active components adapts to the target level**, and components excluded at one level may become relevant for another. **Action:** In our **revised Theorem 4.1**, we highlight the behavior of threshold-dependent component activation, and **elaborate on our initialization and annealed training strategy in our new Supplement C.3**.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Response to all reviewers

We thank the reviewers for the comprehensive and helpful feedback on our work.

To improve our work, we made the following additions to our revised paper (highlighted in blue\color{blue}{\text{blue}}):

  1. New high-dimensional scalability experiments: We introduce a new highly nonlinear, multimodal and high-dimensional benchmark in our revised Section 5 (new Figure 7), where we evaluate TRIO across dimensions d=2d=2 to 6464. TRIO achieves forward accuracy highly competitive with parameter-matched unrestricted MLPs, while maintains mean ground-truth preimage IoU above 0.89 even for d=32d=32. The results demonstrate that strong practical performance is possible even in substantially higher dimensions.
  2. New architectural ablation study: In our new Table 3, we isolate the contributions of learned anisotropic geometries and expressive neural radial transformations. At fixed QQ, the full neural TRIO architecture achieves an 83.49% reduction in forward RMSE relative to the isotropic quadratic subclass, and improves mean preimage IoU from 0.7170.717 to 0.9390.939. This demonstrates the practical benefits of both architectural components.
  3. New theoretical extension to multiple output constraints: We extend TRIO to vector-valued predictors and multiple simultaneous constraints in our revised Sections 3 and 4, with a formal statement and proof in new Supplement A.4. The extension preserves exact, explicit joint preimage representations, universal approximation guarantees, and global regionwise optimization. Thereby, we substantially broaden the applicability of our framework.
  4. Clarification of exactness and global optimality: We sharpen the distinction between exact representation of learned preimages and approximation of ground-truth preimages, especially by highlighting the conditional guarantees in Supplement A.3.
  5. Expanded experimental and geometric explanations: We clarify the deliberately large Six-Hump Camel training set, distinguish exactly learned-preimage coverage from ground-truth IoU, and explain PREMAP2's conservative, budget-dependent reconstruction. We also highlight the non-ellipsoidal, nonconvex geometries of our benchmarks, including the visualizations in Supplement Figure 8, and add geometric intuition in Supplement A.2.
  6. Training strategies and limitations: We expand Supplement C.3 with our targeted initialization and soft-minimum training strategies, clarify threshold-dependent component activation, and add an explicit Limitations discussion in revised Section 5.

Taken together, these revisions provide new experimental evidence, new theoretical guarantees, and a clearer understanding of TRIO's capabilities and limitations. We believe they significantly strengthen the paper and its contribution to representation learning and inverse optimization.

We are confident that, with the help of the reviews and our improvements, our work will be an important contribution to the community. Finally, we wanted to say “thank you” – we truly appreciate the help, which allowed us to revise our work and make important improvements.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.