acceptodds
Withdrawn from ICLR 2027

PROFF: Prompt-Adaptive Reduced-Order Feed-Forward Computation for LLM Agents

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Modern LLM agents repeatedly process long prompts containing system instructions, tool schemas, and interaction history while generating short outputs. This makes prefill a major inference bottleneck, with feed-forward (FF) blocks accounting for a large fraction of its compute. Existing FF acceleration methods largely rely on neuron-level sparsity, which is less effective for modern gated activations such as SwiGLU and GeGLU. We present PROFF (Prompt-adaptive Reduced-Order Feed-Forward computation), which replaces the standard FF block with a reduced-order FF block. PROFF compresses the hidden state into a low-dimensional space, performs mixing in that space, and projects it back to the original hidden dimension. A lightweight predictor selects the reduced dimension for each agent call, adapting FF compute to the input. The surrogate is trained offline through per-layer distillation, while the remaining model parameters are frozen. Across four LLMs and multiple agentic benchmarks, PROFF retains near-baseline task quality while improving prefill throughput by 2.15–2.23 across prompt lengths and model architectures. Across multi-call agent workloads, PROFF further achieves 1.98–2.31 lower end-to-end task latency. These results show that prompt-conditioned reduced-order FF computation provides a practical approach to accelerating prefill-heavy agentic inference.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
AI Agent, Model Inference, Prefill Phase

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_AEPeOpenReview ↗
Rating
1(−1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The authors propose PROFF, which uses distillation to construct lightweight FF replacement modules and dynamically selects their capacity based on structured prompts. The framework aims to reduce computational overhead in agent inference and accelerate prefill, decoding, and end-to-end execution while largely preserving task quality.

Critical strengths

The work provides a clear and convincing motivation, establishing the value and significance of the work. In particular, the introduction is well written.

Critical weaknesses

This paper is far from sufficiently developed. It lacks a clear overview or component diagram that presents the technical details. Insufficient attention has been paid to the figures. The organization and coverage of the main text are also incomplete. I do not believe the main text adequately conveys what readers need to understand the paper. It lacks a critical assessment and discussion of the proposed approach, as well as a related-work discussion that supports and contextualizes the contribution. As a result, I cannot sufficiently understand the work from the main text alone.

The experimental evaluation is insufficient, and the paper does not even provide a well-organized ablation study. The incremental benefits are not adequately explained or substantiated, so I maintain my original assessment and concerns. Table 4 lacks corresponding online-latency measurements for the fixed and adaptive configurations, making it impossible to determine whether adaptation improves the quality–latency tradeoff. The exact architectures of the alternative modules in Table 2 and whether their parameter and training budgets are matched also require clarification.

Questions for authors

At least you need to address the weaknesses I identified in the “Critical Weaknesses” section.

Reviewer_a8zdOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

PROFF accelerates LLM agent inference by replacing pretrained feed-forward blocks with learned reduced-order surrogates. Each surrogate compresses the hidden representation from dimension H to P, applies a precomputed linear transformation in the reduced space, and projects back to H. The surrogates are trained offline through per-layer reconstruction distillation while the remaining model parameters are frozen. A prompt-conditioned predictor selects P from {4,8,16,32} for each call.

The paper evaluates four backbones, including two MoE models, on capability and agentic benchmarks. It reports near-baseline quality, approximately 2.2× higher prefill throughput, and 1.98–2.31× lower end-to-end agent latency.

Critical strengths
  1. A potentially valuable approach to reducing gated FF computation Learning a compact nonlinear replacement avoids relying on zero-valued activations in the original FF block. Preserving the H-dimensional input/output interface also makes the replacement structurally compatible with the existing residual and attention pathways. If the reported quality and speedups are reproducible, this would be a useful compression result.

  2. Evaluation across several architectures and agent workloads The inclusion of dense, hybrid, and MoE backbones, together with fixed-capacity sweeps and multi-call agent evaluation, is a meaningful strength. These experiments address both capability preservation and practical task latency rather than relying solely on reconstruction error or theoretical FLOP reductions.

Critical weaknesses
  1. Key implementation details and architectural benefits are unclear

The paper does not specify the encoder/decoder architectures, essential training settings, or whether MoE compression replaces individual experts or the entire routed mixture. These details are necessary for reproduction. The benefit of the reduced-space mixing is also unclear: depending on the decoder architecture, this fixed linear transformation may be absorbed into the output projection. A comparison with an equal-capacity gated FF student trained using the same distillation procedure would help establish its contribution

  1. The latency results need clarification, and the benefit of adaptivity is unproven

Table 6 reports 15.3 tokens/s for GPT-OSS at 1K tokens, implying about 67 seconds per prefill, while Table 8 reports 55.8 ms for 512 tokens. The units and measurement settings need reconciliation. The paper also lacks a measured quality–latency comparison between adaptive selection and fixed (P=16/32). Finally, the reported 0.016 ms overhead covers only the predictor MLP, leaving the costs of input encoding and online prompt reconstruction unclear.

Questions for authors
  1. What are the exact encoder and decoder equations, distillation objective, training budget, and optimization settings? For each MoE backbone, are individual experts compressed, or is the entire MoE output distilled into a replacement? Please specify router handling and original/reduced FF parameter counts.

  2. How can the prefill measurements in Tables 6 and 8 be reconciled? Please provide the units, batch sizes, precision, parallelism, and timing boundaries. Also, what exactly is the “test reconstruction loss” in Table 2, and why is it nonzero for the unreduced model?

  3. Under the same training budget, how does PROFF compare with an equal-P gated FF student without the matrix-exponential parametrization? Can fixed P=16, fixed P=32, and adaptive P be compared jointly in task quality and measured end-to-end latency?

  4. What are the costs of prompt reconstruction, constructing the predictor input, and dispatching variants? Is prefix caching enabled for both the baseline and PROFF? Since changing FF variants generally changes later-layer KV states, how is cache reuse handled when the selected P changes between calls?

Reviewer_hv8JOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes PROFF to speed up prefill-heavy agent inference. It replaces every feed-forward block with a distilled low-rank block that compresses the hidden state, mixes it in a small latent space and projects it back. A small MLP picks the latent size for each agent call from a few precompiled options, using a version of the prompt that an LLM has rewritten into a structured form. The paper reports that prefill, decode and end-to-end latency are roughly halved across several models, with accuracy close to the original.

Critical strengths

Agent workloads really are dominated by prefill, and it matters that gated activations reduce the sparsity that earlier feed-forward methods relied on.

Critical weaknesses
  1. The main results are hard to believe and don't agree with each other. Each feed-forward block is replaced by a very small latent that is distilled in minutes on general web text with no agent data. Yet the paper says it still matches or even beats the original model on graduate-level reasoning, tool use and agent tasks.

  2. The efficiency numbers also conflict. The tokens-per-second figures in the throughput table don't fit the per-request prefill times in the latency table or the predictor overhead table. The speedup is also nearly the same for every prompt length, phase and model. This contradicts the paper's own argument that the feed-forward share of compute grows with input length, and it doesn't fit the large compute reduction the design implies.

Questions for authors
  1. Can you fix the inconsistencies between the throughput, latency and overhead results, and state which latent size each speedup result uses?

  2. What do the online prompt reconstruction step and keeping several model variants loaded each cost, and does the baseline receive the same reconstructed prompts?

  3. How can such a small latent, briefly distilled on general web text, preserve tool-use and reasoning accuracy, and can you release checkpoints or report per-layer reconstruction errors?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.