acceptodds
Under review as a conference paper at ICLR 2027

InterQ-RL: Interaction-Aware Correction of Execution Mismatch in Quantized Agentic RL

Abstract

Multi-turn agent reinforcement learning (RL) relies on long closed-loop rollouts, making generation and KV-cache costs a major bottleneck. Quantized rollout reduces these costs, but its low-precision executor also serves as the behavior policy and can induce execution mismatch with the learner. Existing quantized-RL methods are designed primarily for single-turn generation. They align the learner forward with quantized rollout or correct the current-token rollout–learner ratio, without modeling the policy history that produces each agent state. Even after quantization alignment, residual rollout–learner mismatch accumulates along the state-reaching policy history, so a current action may look locally aligned despite a mismatched history. We introduce InterQ-RL (Interaction-aware Quantized Reinforcement Learning) to correct this post-alignment mismatch. It applies bounded execution-prefix correction over the state-reaching policy history, allocates correction strength across action segments by effective support under a fixed displacement budget, and requires no additional model forward pass. We evaluate dense and mixture-of-experts (MoE) backbones from 4B to 30B on ALFWorld, WebShop, and ScienceWorld, with all quantized arms sharing the same rollout with 8-bit weights, activations, and KV cache (W8A8-KV8). Across four environment–backbone settings, InterQ-RL achieves the strongest quantized endpoint result, matching BF16 to within 0.7 points and exceeding it in two settings. It matches or exceeds the learning speed of alignment alone while avoiding its late performance regression.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Reinforcement Learning, Quantization, Agentic RL

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_VJwwOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This work investigates the impact of strategy distribution mismatch between quantized actors and learners on multi-round agent reinforcement learning training, and proposes InterQ-RL to alleviate this issue through quantization alignment, policy prefix correction, and correction strength allocation based on effective sample size. Experiments are conducted in three environments using different models for validation, and compared with BF16, QAT, and various quantization reinforcement learning baselines. The results show that InterQ-RL achieves the best quantization method endpoint performance in all four environment and model combinations, and performs close to or better than the BF16 baseline on multiple metrics.

Critical strengths
  1. The research question is clear and has practical value. The goal of quantizing the rollout is to reduce the computational and memory costs of generation and KV Cache, but the low-precision executor simultaneously determines the behavior policy of the training data.

  2. The method structure is clear, and the theoretical derivation has a certain completeness. Has a clear algorithmic definition, computational process, and verifiable optimization properties.

Critical weaknesses
  1. The model and environment coverage of the experiments are still limited. Although the experiments include models of different scales from 4B to 30B and both dense and MoE architectures, the backbone models all belong to the Qwen3 series, the quantization settings focus on W8A8-KV8.

  2. The paper takes reducing rollout computation cost as its research motivation and emphasizes that InterQ-RL does not require additional model forward computations. However, current experiments mainly compare task performance, training curves, divergence, and interaction rounds. The paper does not systematically quantify the tokens generated per second and rollout throughput, and the extra overhead from prefix weight statistics and the optimizer is not compared with QAT.

Questions for authors

see weakness

Reviewer_r6P6OpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper studies the mismatch between a quantized rollout executor and the learner in multi-turn agent reinforcement learning. After aligning weight, activation, and KV-cache quantization through QAT, InterQ-RL constructs bounded correction weights from the rollout-learner log-probability differences along the policy-action prefix leading to each state. It allocates correction strength across action segments using an effective-support score under a fixed displacement budget. Across four environment-backbone settings, it reports the best quantized endpoint, improving ALFWorld success over QAT from 90.4% to 96.4%, WebShop score from 74.9 to 77.7, and ScienceWorld macro scores from 65.5/69.3 to 67.8/69.9.

Critical strengths
  1. Clear motivation and a coherent method. The paper explains why a current action can appear locally aligned even when the executor and learner differ along the history that produced its state. The proposed correction follows this motivation closely and reuses existing log probabilities without an additional model forward pass.

  2. Broad empirical evaluation. The experiments cover ALFWorld, WebShop, and ScienceWorld; dense and MoE backbones; and multiple baselines under a common W8A8-KV8 rollout. InterQ-RL reports the strongest quantized endpoint in all four settings. The Prefix/Global and QAT+TIS/AIS ablations provide useful evidence beyond aggregate scores.

Critical weaknesses
  1. Practical cost and budget sensitivity. Although the correction requires no extra model forward pass, the paper does not report training time or peak GPU memory against matched QAT. The Global ablation uses the same correction budget, but the paper does not show how performance or training stability changes when that budget is made smaller or larger.

  2. Some incremental gains over QAT are small and lack multi-seed validation. The ALFWorld improvement is substantial, but the ScienceWorld solved-rate gains are only 0.3 percentage points for each backbone, and the 30B macro-score gain is 0.6 points. Each ScienceWorld result comes from one endpoint checkpoint, and the paper does not report independent-seed replications for those settings. It is therefore unclear whether these smaller differences would persist across training runs.

Questions for authors
  1. What are the wall-clock time per update, total training time, and peak GPU memory of InterQ-RL relative to matched QAT? How much of any difference comes from the prefix statistics and allocation solver?

  2. How sensitive are performance and training stability to the correction budget in Eq. (8)? Could the authors report a small ablation that scales the reference budget below and above its default value?

  3. Do the smaller ScienceWorld gains over QAT persist across independent training seeds? Please report per-seed endpoints or otherwise quantify uncertainty in these comparisons.

Reviewer_6mmCOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper looks into the logprob mismatch between the FP8 rollout engine and the learner. The authors argue that even after the learner is aligned to the quantized engine with QAT, the remaining mismatch builds up over the earlier actions of a multi turn trajectory. InterQ-RL weights each token's GRPO loss by a clipped cumulative rollout learner ratio over all policy tokens up to and including the token. They show on models from 4B to 30B, on the agentic benchmarks, the best quantized results are close to or above BF16 results.

Critical strengths
  1. The problem is practical, and their alignment study is carefully implemented. The paper runs FP8 rollouts for long multi turn agent RL, uses a KV-cache QAT rule matched to the inference kernel, and shows clearly that this alignment cuts the rollout vs learner divergence by about 35 times but still does not remove it.
  2. The comparison is through and well controlled. Three correction methods (Flash-RL, AIS, QuRL) are compared across multiple models.
Critical weaknesses

Most margins are small, and each seems to come from a single training run, so I find it hard to judge how much of the gain would hold across seeds. On WebShop, InterQ-RL leads the best other quantized method by 1.2 points of score, and on ScienceWorld it leads QAT by 0.3-2.3 points, about the same as QAT’s own lead over BF16 (up to 2.3), which I read as a rough guide to how much single runs vary. On ALFWorld the gain over QAT is larger (+6.0), but the two methods learn at the same speed (AULC 82.0 vs 82.3). The gap comes from QAT dropping late in its run, but with one run it is hard to know whether that drop is typical. Appendix D says seed replications of BF16, QAT and InterQ-RL agree with the reported comparisons, but there are no numbers. Still, InterQ-RL is the best quantized method in Table 1, which is encouraging, and reporting the mean and confidence interval over seeds would make the case much clearer.

Questions for authors
  1. Could you report the seed replications from Appendix D (mean and spread for BF16, QAT and InterQ-RL on ALFWorld and WebShop), and seeds for the Prefix and Global variants? Does QAT's late drop on ALFWorld show up in every seed?
  2. How does InterQ-RL compare with existing cumulative or sequence-level ratio methods such as Zhang et al. (2026b), Lei et al. (2026), GSPO and trust-region masking?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.