acceptodds
Under review as a conference paper at ICLR 2027

Jacobian Rank Collapse in Decision-Focused Learning

Abstract

Decision-focused learning (DFL) trains predictors through downstream objectives, yet changing the loss does not necessarily create a new direction for updating the model. We study this limitation through the predictor Jacobian. At rank one, all nonzero backpropagated loss gradients are collinear; a spectral bound characterizes near-collinearity when a leading singular direction dominates and the gradients have nonzero components along it. This local restriction does not imply that different losses share an optimum. We test its empirical relevance in sparse index tracking across six equity markets, and examine transfer to weather, air quality, and shortest paths. Across 38 one-parameter equity configurations spanning a 54× range of sparsity, DFL gains over MSE remain below 1.8% (median 0.25%), with no detected monotone association with sparsity (ρ = −0.16, p = 0.35). Structured predictors yield larger gains in several markets and up to 9.4% on air quality. A calibrated effective-dimension proxy organizes these outcomes: the archived blind-evaluation record contains 27 correct predictions among 31 non-abstaining cases, with two additional abstentions. Transfer results also expose the limits of fixed thresholds. For higher-capacity models, covariance regularization mitigates task-training instability. Together, these findings motivate assessing the gradient directions a predictor can express, validating the diagnostic in the target domain, and regularizing task training when needed.

Primary area:
optimization
Keywords:
decision-focused learning, predict-then-optimize, differentiable optimization, no-go theorem, gradient alignment, effective dimension, structured prediction, portfolio optimization

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_kgLmOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The paper studies whether changing from a prediction loss to a decision-focused loss gives useful, additional directions for updating a predictor.

Its main application is sparse index tracking, which setting is not my expertise. From my understanding, a covariance predictor produces Σ^(x;θ)\widehat{\Sigma}(x;\theta), a subset of KK assets is selected and a quadratic program determines the portfolio weights. The paper distinguishes two stages of gradient propagation: covariance-space geometry and parameter-space geometry.

The experiment compares multiple shrinkage settings, factor models, and neural covariance models. Paper reports small DFL gains on scalar shrinkage equity configurations, and larger gains in higher-capacity settings, and the transfer experiments involve spatial data and shortest paths.

My understanding of this paper’s overall contribution is a combination of local gradient analysis, a empirical diagnostic, and a DFL objective. However, given the many obvious numerical inconsistency, theoretical overclaim and abnormal narratives that seem to be generated directly from an LLM, I lean to a rejection.

Critical strengths

1. The question is relevant and worthwhile

Determining when DFL provides enough benefit over its training cost is a useful research problem. This paper studies the predictor as well as the downstream optimizer, specifically, an informative decision gradient can have limited influence after passing through a restricted predictor.

To me, this problem is not far-fetched and it also seems to be highly relevant for DFL researchers.

2. Main theoretical claims are sound, the figures are informative

I have checked the central chain-rule identity, rank-one result and the conditional near-rank-one bound, all appear correct under its stated assumptions. Fig. 2 and 3 also illustrate the two objectives, one predictor and misalignment very well. Moreover, the empirical study part covers multiple markets, universe sizes, sparsity levels, and predictor families.

Critical weaknesses

Despite the strengths above, the paper shows severe obvious numerical inconsistencies, vacuous theoretical guarantees, and abnormal narratives that seem to be generated directly from an LLM. Due to the 1-2 critical weakness limit of this block, I moved few weaknesses to the Question section.

1. Rank-one collinearity overclaim

This is my biggest concern regarding the theoretical and methodological part. Proposition 4 establishes a local geometric fact, but it cannot establish anything like:

MSE and task loss have the same stationary points their training trajectories converge to the same predictor

The paper acknowledges this in Appendix, but it still repeatedly provides stronger claims:

“MSE suffices” and “all paths simply converge.” the family “collapses onto two-stage training.” the theory predicts ‘the same as two-stage’. etc.

These statements do not follow the theorem. An important case is an MSE stationary point: J⊤gMSE=0,J⊤gtask≠0J^\top g_{\mathrm{MSE}}=0, \quad J^\top g_{\mathrm{task}}\ne 0 DFL can move away from that point even when JJ has rank one. The proposition’s restriction to two nonzero gradients does not address this situation.

2. Many numerical inconsistencies across the draft

There are major, obvious numerical contradictions between different sections describing the exact same experiments. For instance:

Structured shrinkage at K=20 is described as gaining 1.6%, p=0.002, 9/9 wins. Table 8 reports p=0.010, 8/9 wins, Figure 9 reports 1.3%, Appendix AB gives 1.26%.

For the neural model at K=20, nine folds, Table 14 gives TE 0.0391 → 0.0363, worst-fold TE 0.0652 → 0.0652. Tables 4, 13 give mean TE 0.0331 → 0.0276, worst-fold TE 0.0564 → 0.0360.

The text claims 14.5% CVaR improvement and 20.5% drawdown improvement, referring to Appendix R. But Table 14 reports 5.9% and 6.9%. Full-capacity negativeIdentity gains are 2.24% in the table and 4.13% in the appendix text.

Questions for authors

1. Abnormal text, mainly in the appendix

There are many abnormal texts, mainly focused in the appendix, which seems to be generated by an LLM without human checking. For instance:

“A reader who suspects our DFL simply fails to train has to explain why it trains here.”

“That is the weakest position a negative result can occupy: it can be dismissed without reading the proof, either as a broken DFL implementation or as a task too easy to show a gap."

“We had previously written 120× from an estimate rather than a measurement. “

“The measured gap is wider than an earlier draft of this paper claimed. “

I am wondering if these weird sentences are the draft revision history or chain-of-though from an LLM.

2. Computational cost are overstated

The approximately 3,300-fold runtime ratio compares a particular shrinkage-DFL implementation with POET. It is not a matched measurement of MSE versus DFL training cost for the same architecture.

3. The shortest-path bottleneck experiments measure the wrong Jacobian

The theory concerns Jθ=∂c^∂θ,J_\theta=\frac{\partial\widehat c}{\partial\theta}, whereas Appendix Y explicitly measures Jx=∂c^∂x.J_x=\frac{\partial\widehat c}{\partial x}.

A bottleneck of width kk bounds the rank of the input Jacobian. It does not generally bound the parameter Jacobian when the decoder is trainable.

  1. Why softmax does not satisfy the simplex constraint? Appendix L says the QP enforces budget and nonnegativity exactly whereas softmax does so approximately. But mathematically, softmax satisfies both constraints exactly. The QP may provide a useful inductive bias, but this explanation is incorrect.

  2. Higher rank alone does not guarantee a distinct task-gradient direction. The summary of Corollary 2 drops the essential noncollinearity assumption. I think the appendix’s formal statement is more careful/proper than the interpretation in main text.

Reviewer_T56yOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies when DFL actually beats just fitting with MSE and then optimizing. Main claim: when the predictor Jacobian has rank 1, MSE and task gradients end up pointing along the same line in parameter space, so switching losses doesn't give the model a new direction to move. They test this on sparse index tracking in six equity markets, plus weather, air quality, and shortest-path toy grids. They find DFL barely helps on one-parameter models, helps more on bigger models, and propose BD-DFL to stop high-capacity training from blowing up.

Critical strengths
  1. The empirical result is valuable for the industry: DFL on simple models costs 3300× for nothing.
  2. The empirical coverage is broad for a DFL study, including six markets and three non-finance domains.
Critical weaknesses
  1. The headline theory is trivial, and the no-go result is built into the design. When the model has only one parameter, there's only one direction to move. Of course two different losses push it along that same direction. And the "DFL doesn't help on simple models" result is obvious too — they tested on a model with literally one knob. Of course two losses trained on one knob land in about the same spot.

  2. This research contribution is not as general as claimed. The math only fits this one specific portfolio QP. The weather and air-quality experiments are the same math with different data, not different problems. The shortest-path tests are tiny toy grids. The thresholds they give practitioners (1.5, 4) are tuned on the same data they report, and the authors admit they don't hold up on a different benchmark. is more like a discovery within the finance or asset management area.

Questions for authors
  1. Can you show the "no-go" result on a model with more than one knob? Right now it looks like you only tested the case where disagreement is impossible.

  2. Do you have any evidence this diagnostic works on a problem that isn't this specific stock-picking QP? If not, can you make the title less broad?

  3. When you add MSE regularization, is DFL actually benefiting from a "unique direction," or is it just that training DFL alone was unstable? Have you tried adding the same regularization to a well-tuned MSE model?

Reviewer_JXdDOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper studies decision-focused learning (DFL) under low-rank predictor Jacobians, primarily in the context of portfolio optimization. The predictor estimates asset covariance matrices, which are subsequently used in a quadratic program to determine portfolio weights. The paper analyzes gradient alignment when the predictor Jacobian has low rank and evaluates different prediction models and training objectives on financial and other datasets.

Critical strengths

None that substantially affect my overall assessment.

Critical weaknesses

Poor presentation and unclear technical exposition

The paper is extremely difficult to follow. The introduction is poorly organized, and the notation and mathematical formulations are not adequately explained, making it difficult to understand the problem setting and the intended contributions.

The figures and experimental tables are also difficult to interpret. The paper presents a large number of empirical comparisons without clearly communicating the main findings or their significance. Furthermore, placing the primary theoretical analysis in Section 7, after the extensive experimental sections, makes the overall presentation difficult to follow.

Questions for authors

Could the authors clarify the central novelty beyond the standard Jacobian rank argument and explain more directly how the theoretical results support the main experimental findings?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.