acceptodds
Under review as a conference paper at ICLR 2027

Inference-Native Zeroth-Order Optimization for LLMs

Abstract

Zeroth-order (ZO) methods train large language models using only forward passes, yet common ZO execution paths perform substantial work beyond what the algorithm itself requires. To remove this extra execution overhead, we present Infer-ZO, which separates the evaluations required by the algorithm from how they are executed, allowing them to reuse existing inference-engine optimizations. After removing inherited execution work, a complete Infer-ZO step on Qwen3-14B adds only 1.16% wall-clock time over matched inference, bringing ZO execution close to its fundamental inference workload. With Infer-ZO, ZO evaluations run at near-inference cost with frozen base weights and can share GPU batches with ordinary inference requests. In co-serving experiments, foreground and background throughput remain within 1–3% of the corresponding background-request baseline. Across 15 Qwen3, Llama, and OPT models, Infer-ZO achieves 2.10×–9.94× end-to-end step speedups over released LoZO.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
zeroth-order optimization, large language models, inference serving, parameter-efficient fine-tuning, LoRA, machine learning systems

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_3iGYOpenReview ↗
Rating
1
Confidence
3(+1)
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces Infer-ZO, a system that separates zeroth-order optimization algorithms from backend execution. It represents compatible perturbations and accumulated updates using low-rank factors, reducing repeated dense-weight writes and allowing optimization evaluations to run through an inference engine alongside ordinary serving requests.

Critical strengths

Infer-ZO improves the efficiency of ZO optimization by representing perturbations and accumulated updates as low-rank adapters, reducing costly dense weight writes and enabling inference-engine optimizations.

Critical weaknesses

There are system improvement in decomposing the ZO system, but the algorithmic novelty is limited. A system-level optimization for a non-production-useable method ZO has very limited contribution. The experiments are done on SST-2, BoolQ, and CB. These datasets are older datasets and covers very limited capabilities of modern LLMs. The author reported the performance on dev set and validation set, and therefore does not fully establish whether performance are on-par with earlier methods. The model experimented are mainly small models with very limited training steps. It's unclear if the result will transfer to test set in more realistic production setting.

Questions for authors

Could you provide GPU hours vs evaluation results of the proposed Infer-ZO compared to gradient decent methods? What are the performance of the proposed infer-ZO perform after trained to convergence on more recent tasks?

Reviewer_4FLcOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Zeroth-order (ZO) optimization trains models with just forward passes, which saves memory, but it's usually run through training code that does way more work per step than the algorithm actually needs. This paper treats the loss evaluations in a ZO step as plain inference requests that any backend can serve. Their system, Infer-ZO, uses LoZO-style low-rank perturbations so the perturbations and updates can be stored as LoRA adapters on a frozen base model, and runs the loss evaluations as LoRA inference in a modified vLLM. They show it matches LoZO's gradient estimates and learning curves on small OPT models, report big per-step speedups over the released LoZO code across a bunch of models, and claim a training step costs about the same as inference and can share GPU batches with regular inference traffic at little cost.

Critical strengths
  • Really clear breakdown of where the time goes in a ZO step. On Qwen3-14B they trace the gap between the released MeZO/LoZO code and their system to three things (dense weight writes, the LoRA side computation, and padding), and a simple memory-traffic model accounts for the dense-write part. This is useful on its own for anyone building ZO training systems.
  • Good control exps that pin down where the speedup comes from. They show packed PyTorch and vLLM forward passes cost the same, which separates the gains from packing and removing dense writes from any effect of the inference engine itself.
  • Equivalence with LoZO is checked strictly to make sure that the system computes the same algorithm.
  • The co-serving idea, running ZO training requests in the same GPU batches as regular inference traffic, is new and a natural fit for ZO, since each step is just a set of forward passes.
  • Good code supplement
Critical weaknesses
  • Most of the speedup doesn't come from the inference engine. By their own numbers it comes from removing dense weight writes and packing sequences, and packed PyTorch is just as fast as vLLM. The baseline is the released LoZO code on an old software stack, so the big speedups are mostly against unoptimized code.
  • The headline numbers are a bit unclear to me. The largest speedups only appear on the task with the most padding, the overhead number depending on a "matched inference" setup that's defined differently in different places, and co-serving tested only under a heavy background load.
  • Close prior work isn't compared I think? MobiZO and EGGROLL already run low-rank perturbations as batched LoRA inference, and there's no accuracy vs time comparison against MeZO or LoZO.
Questions for authors
  • What does the inference engine let you do that an optimized training framework can't? Your own control shows packed PyTorch matches vLLM, so if the gains come from removing dense writes and packing, it seems the same speedup is available without changing systems. What's the real use case for running ZO training through an inference engine?

-Does this actually change what ZO is practical for? The pitch is that ZO training can cost about the same as inference, but there's no comparison of time to reach a given accuracy or memory against MeZO or LoZO. So it's unclear whether faster steps make ZO more competitive with regular fine-tuning, or just make the existing code less wasteful.

Reviewer_Di6nOpenReview ↗
Rating
4(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper addresses the memory traffic and execution overhead caused by repeated dense weight modifications in zeroth-order optimization. It proposes Infer-ZO, which represents perturbations and accumulated updates through low-rank factors and executes function evaluations as requests through a vLLM-based inference backend, enabling shared execution with ordinary inference requests. Across 15 models, Infer-ZO achieves 2.10×–9.94× end-to-end step speedups over released LoZO while maintaining comparable learning behavior, with only 1.16% overhead over matched inference in a Qwen3-14B experiment. In co-serving experiments, foreground and background throughput remain within approximately 1–3% of the corresponding LoRA-prefill background baseline.

Critical strengths
  • This paper proposes a clear and effective way to alleviate the memory traffic caused by repeated dense weight modifications in zeroth-order optimization. Experimental results show that the low-rank execution approach achieves learning performance comparable to released LoZO, with considerable speedups.
Critical weaknesses
  • Some details of the Infer-ZO system and experimental configurations remain unclear. Although Appendix C.1 reports the hardware platform and some engine settings, the serving configurations for the 15-model evaluation are not fully specified, including GPU counts, parallelism strategies, and other relevant vLLM settings. Similarly, Appendix L specifies priorities for serving and ZO requests, but the detailed scheduling and admission policy remains insufficiently described. These omissions make the systems results harder to reproduce and interpret.
Questions for authors
  1. Could the authors provide the complete execution configuration for each model, including GPU type and count, parallelism strategy, and relevant vLLM engine settings? Why was batch size 16 selected, and how does the speedup change with larger batch sizes?
  2. Beyond the priority assignments reported in Appendix L, what scheduling strategy is used for the serving and ZO queues? How are background requests admitted, and how are prefill, decode, and paired ZO evaluations interleaved?
  3. What are the individual costs of ZO gradient estimation and low-rank parameter updates in the modified vLLM backend? Although these costs are included in the overall speedup comparison, a separate timing breakdown would strengthen the systems evaluation.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.