acceptodds
Under review as a conference paper at ICLR 2027

Learning to Be Wrong: Task-Sufficient Inexact Computation

Abstract

Neural efficiency methods usually optimize how much computation to spend. We study how much local exactness is worth paying for. A cheaper realization may violate a conventional local contract while preserving downstream utility. We evaluate each candidate with measured runtime against a matched exact reference, a mechanism-specific local-fidelity diagnostic, and signed downstream task regret. Across precision-relaxed, incomplete, and contract-relaxed realizations, nominal work, local fidelity, hardware cost, and task utility frequently disagree. Our main case study groups several target positions in ESM-2 masked-marginal scoring so that one forward evaluates multiple mutations under a changed conditioning context. At , the same frozen random-group procedure yields 3.93–3.99 measured scoring speedup with mean signed DMS regret 0.0010–0.0012 on TEM-1, influenza HA, and a DHFR assay held out from policy development; at , speedup reaches 12.34 - 15.59 with assay-dependent regret. Pairwise interference is predictable in development, while held-out DHFR shows that a fidelity-aware grouping rule can improve exact-score agreement and still worsen task regret. These results make task regret the governing quantity for inexact execution: spend exactness where it changes the task.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
approximate computing, efficient neural inference, task-aware computation, machine learning systems, protein language models, variant-effect prediction

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_LdhjOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The paper identifies scoring variants of a given amino acid sequence as a task where exact likelihoods are not needed to approximate the same likelihood. The authors propose simply grouping together multiple single-amino-acid mutations into a single forward pass into the model, and inspecting the likelihoods of masked out tokens at each mask position. They find empirically that the corresponding likelihoods are similar to the ungrouped case, which is then sufficient to produce approximate scores.

Critical strengths

None

Critical weaknesses

As a disclaimer, I am not familiar with the protein language modeling domain. But since compute is entirely spent on prefill, from the presented experiments it does not make sense to provide such speedups. For example, the assay “influenza HA doud” has 565 length and 564 positions, producing total 565⋅564≈319k565 \cdot 564 \approx 319\text k tokens for computing exact per-token marginals. With a 650M model, this is roughly 414 TFLOPs, which can run in tens of seconds on A100 gpus with a reasonable implementation. The speedup from tens of seconds to a few seconds seems negligible in terms of compute cost, if this computation is performed just a single time for each assay. If indeed this inference cost is substantial in practical settings (on the order of, say, multiple hours that is often an experimental bottleneck), then it would be helpful to see comparisons between ESM-2 150M exact g=1 and ESM-2 650M g=4, which correspond to roughly the same FLOPs. It would also be interesting if the described kernel optimizations were more broadly applicable, but they appear to be customized to ESM-2 650M.

The paper optimizes for Spearman correlation, which is reasonable for optimizing against DMS regret. But there is little analysis on the failure modes of the proposed method, e.g. a pairwise analysis shows that pairwise interference is somewhat predictable (at correlation 0.416). Thus, it is difficult to understand precisely in what settings the proposed method disagrees with exact inference.

Questions for authors

None

Reviewer_aS8xOpenReview ↗
Rating
1
Confidence
1
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper studies inexact computation through measured runtime, local fidelity, and downstream task regret. Its main case study groups masked positions in ESM-2 to accelerate mutation scoring, achieving approximately 4× speedup with small observed regret across three protein assays. Additional experiments examine site sensitivity, composition failures, and the mismatch between fidelity-aware grouping and downstream utility.

Critical strengths

The evaluation distinguishes actual runtime gains from nominal computational savings and transparently reports unsuccessful approximations and composition failures.

Grouped masked-context scoring provides a simple, practically useful speed–quality trade-off, with consistent acceleration across three protein assays.

Critical weaknesses

Limited methodological novelty. Task-constrained approximation is already established in work such as ApproxCaliper, while this paper does not establish a transferable task-regret-aware allocation policy.

Limited validation of task sufficiency. The main results cover three assays under one ESM-2 model, without paired confidence intervals for grouped-scoring regret, leaving the robustness of small differences and the held-out reversal unclear.

Questions for authors

How can group size be selected to satisfy a specified regret budget on an unseen protein without DMS labels, and does the frozen learned policy support such selection?

Do the small regret values and the DHFR reversal remain supported under paired, position-clustered confidence intervals and additional held-out assays?

Reviewer_NDWyOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper starts talking about neural efficiency but a more sharper question on how much when intermediate tasks depart from their original path. The paper proposes cheaper proxy tasks. Interestingly the paper questions if the proxy task of local approximations (wrongness) can accommodate errors without degrading the results. This is verified on Protein Language models.

Critical strengths

(i)The papers framing shifting the focus in prior works from spending less compute to paying only for the necessary local exactness. This "Learning to Be Wrong" paradigm offers valuable insights, particularly through its empirical check which reveals that nominal computational savings often vanish due to actual hardware bottlenecks. (ii) The authors' honest reporting of empirical reversal highlights the complexity of the problem and strongly supports their overarching thesis that downstream task utility must be measured directly.

Critical weaknesses

The paper has significant presentation, structural, and notational issues that hinder reading making it hard to grasp the merits of the paper.

1. Severe Notation and Terminology Issues

  • Undefined Symbols: Crucial symbols are used well before they are defined. For example, the utility function UTU_T is introduced in Equation 1 but not defined until Section 4. The parameter ρ\rho (e.g., ρ=0.75\rho = 0.75 in Section 5.2) is never explicitly defined in the text. The group size gg is used in the abstract without definition.
  • Inconsistent Terminology: The authors use terms interchangeably. For example.g the paper swaps between "Task regret," "signed DMS regret," and "regret" without clarification. Similarly, "wrongness," "local distortion," and "local relative error" are used loosely, lacking a unifying definition.

2. Poor Structural Flow and Buried Context

  • Missing Context for Claims: Key claims are presented without immediate context. In Section 6, the authors report an "adjusted 10.4×10.4\times speedup," but the reader must hunt back to Section 4 to discover what "adjusted" actually means (it accounts for test-protein feature forward and CPU planning overhead).
  • Clunky Equations: The notation used in Section 5.3 for the exact reference mutation score is heavy. The distinction between x⟨p⟩x_{\langle p \rangle} and xpWTx_p^{\text{WT}} is not adequately explained in the surrounding text, forcing the reader to infer the mechanics of the masked context.
  • Over-Compressed Case Study: The transition from the general systems framework to the ESM-2 protein scoring case study is abrupt. The authors introduce specific biological assays (TEM-1, influenza HA, DHFR) and reference ProteinGym without providing a brief primer on why masked-marginal scoring is standard in this field. This assumes too much domain knowledge and alienates a broader ML/Systems audience.

3. Missed Discussion of Key Findings

  • The "Reversal" Finding: The most interesting result in the paper is somewhat glossed over. It deserves deeper discussion regarding why the distance heuristic fails to transfer, as this is critical to the paper's core argument about the difficulty of predicting task-sufficient relaxations.
Questions for authors

1. The "Reversal" Phenomenon on the Held-Out Assay (Section 6)

  • Could the authors elaborate on the mechanistic reasons for this failure? Does this suggest that distance heuristics in protein space are fundamentally flawed for predicting functional interference, or does it indicate a broader limitation in using local fidelity metrics to predict downstream task failure? How would one know a priori that a learned policy will suffer this reversal on a new dataset ?

2. Clarification of Notation and Terminology

  • Could you provide a formal definition for the realization parameter ρ\rho (for example, in Section 5.2: ρ=0.75\rho = 0.75)? Is this a fraction of retained FFN dimensions, a dropout rate, or something else?
  • In Section 3, the "Audit rule" is presented as the tuple: (contract(r),  C(r) versus matched exact C(r0),  D(r,r0),  RT(r))(\text{contract}(r), \; C(r) \text{ versus matched exact } C(r_0), \; D(r, r_0), \; R_T(r)) Could you rewrite this in a strictly mathematical form? Specifically, how is the "versus matched exact" relationship between C(r)C(r) and C(r0)C(r_0) formalized as a function or ratio?

3. The Cost of the "Learned Policy" and "Adjusted Speedups"

In Section 6, you report an "adjusted 10.4×10.4\times speedup," which you define in Section 4 as including "measured CPU planning overhead for the one-protein invocation."

  • How does the computational overhead of the prediction model (the ridge predictor) or the policy itself scale?

4. Generalizability Beyond Masked-Marginal Protein Scoring

The "Learning to Be Wrong" framework is presented as a general concept for neural computation, but the empirical evaluation is entirely focused on grouped masked-context scoring in ESM-2.

  • How transferable is this specific case study to standard language models (LLMs) or vision models?
  • For autoregressive LLMs, the exact local contract usually requires sequential decoding. Could "grouped scoring" or a semantic relaxation be applied to, say, speculative decoding or beam search, and would the downstream task regret framework hold up there?

5. Defining the Utility Function UTU_T

In Section 4, the protein task utility is defined as UT(r)=ρS(sr,yDMS)U_T(r) = \rho_S(s_r, y_{\text{DMS}}), which relies on the Spearman correlation against biological ground truth.

  • In domains where a clear "ground truth" does not exist (e.g., open-ended text generation or instruction following), how do you propose practitioners define UTU_T? Do you envision this framework extending to LLM-as-a-judge or human preference models to define the task budget ϵ\epsilon?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.