acceptodds
Under review as a conference paper at ICLR 2027

Which States Should Be Compared? Behavioral Quotients for Interactive Policy Optimization

Abstract

Group-relative policy optimization converts rewards into advantages by normalizing within a reference group. For interactive agents, the appropriate reference group is unclear: exact observation identity splits situations with the same observed behavior, while broad pooling can merge distinctions that matter for control. We study this choice in graph-based path credit. From the action-labeled transition graph of a rollout batch, Behavioral Quotient Policy Optimization (BQPO) uses stable partition refinement to build a terminal-preserving quotient without parsing text, pixels, or task fields. With the unit edge costs used in our experiments, the quotient preserves observed success reachability, shortest-path credit, and action ordering. It therefore changes the group used for normalization, not the raw path target. Because equal path values do not make all samples interchangeable, an exact-first router keeps exact groups when local evidence identifies a clear continuation. It uses quotient groups when exact evidence is insufficient and, for other ambiguous cases, when a nontrivial class supplies complete action comparisons. On WebShop, ALFWorld, and Sokoban, BQPO improves the corresponding GraphGPO result in all seven reported aggregate comparisons, covering 1.5B and 7B language models and a 3B vision-language model.

Primary area:
foundation or frontier models, including LLMs
Keywords:
agentic reinforcement learning, credit assignment

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_aGKeOpenReview ↗
Rating
2(+1)
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This work is a methodology paper that proposes Behavioral Quotient Policy Optimization (BQPO). It addresses how states should be grouped when computing relative advantages for RL in interactive agents. BQPO introduces two components. Firstly, the behavioral quotient, which groups observations with similar observed action-transition structures. Second, the exact-routing, which dynamically chooses whether advantages should be normalised within exact-state groups or larger behavioral-equivalence groups.

Critical strengths
  • The paper identifies an important problem in group-relative RL, which is to determine which experiences should serve as reference for advantage normalisation.
  • The results comprise different environments and model sizes, which I appreciate, and there are several structural diagnostics that demonstrate that meaningful grouping occurs.
  • I also appreciate that the runs were done over 3 seeds which would lead to more stable estimates for the RL results.
Critical weaknesses
  • This paper is pretty hard to read and understand, in terms of the methodology. It is not really clear, intuitively why the method leads to better results.
  • I think that the improvements are rather small (around 2-3%) for both ALFWorld and Webshop, where the variance isn't really reported. Without variance reporting It's hard to see whether the improvements manage yield genuine gains.
  • The ablations are rather limited, and not within the main text. The ablations mainly consider random-partition which is intuitively pretty weak. I would ask the author to include more informative baselines as whether there may be simpler grouping heuristics.
  • Beyond the ablations, although there is a mechanism analysis, this seems to be more descriptive of the training dynamics. The paper seems to lack convincing analyses and experiments to show that the method's mechanisms are working as intended, and are the cause of the gains. I.E. What is it about behavioral quotient/ exact-routing that is responsible for the gains? and can these be demonstrated.
Questions for authors

Questions include the weaknesses above. Besides those, I have a few additional here:

  • Can the authors provide more naive baselines for the ablations (i.e. simpler grouping heuristics) to compare BQPO?
  • Can the authors also improve and simplify the presentation so that it reads as more intuitive?
Reviewer_mUtxOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper argues that in group-relative RL for multi-turn agents, the reference group used to normalize step rewards is a design choice separate from credit assignment. It proposes BQPO, which builds on GraphGPO’s shortest-path graph rewards. First, it builds a “behavioral quotient” of the rollout graph by partition refinement: observations are merged when they share a terminal type and the same action-labeled successor classes. Second, it proves that with unit edge costs this quotient preserves reachability, distances and raw rewards, so only the normalization group changes. Third, a hard per-transition router decides whether each reward is z-scored against its exact-observation group or its larger quotient-class group. On ALFWorld, WebShop and Sokoban, BQPO scores above the GraphGPO numbers reported in that paper, by about 1 to 3 points. WebShop ablations show that pooling by the quotient helps, while a random partition with the same class sizes makes training collapse.

Critical strengths
  • The paper makes the choice of normalization group an explicit design decision.
  • The quotient needs no parser, embeddings or similarity thresholds. It works the same way on text and pixel observations.
  • Theorem 4.2 shows that raw rewards don’t change. Any effect therefore comes from normalization alone, which makes the method easy to interpret.
  • It covers three environments, text and vision-language models, and three model sizes. Gains point the same way on all aggregate metrics.
  • Compression, group-size and routing traces show the mechanism working during training.
Critical weaknesses
  • The main weakness is that the method depends on exact repeats. All benchmarks are deterministic and discrete. The quotient never compares observation content. With noisy or rarely repeated observations it adds little, and this setting is never tested.
  • Truncated dead-end nodes may collapse into one class and inflate the reported compression. No examples of merged observations are shown.
  • Most improvements are within seed variance. There are only 3 seeds and no significance tests.
  • The baselines are taken from the GraphGPO paper, not re-run. BQPO’s exact branch also differs from GraphGPO, so the source of the gains is unclear.
  • Ablations cover only one setting. Always-quotient matches full BQPO within noise, so the router’s value isn’t shown. The random-partition control also changes the rewards, so it doesn’t isolate grouping.
Questions for authors
  • Is the graph built per task instance or across the whole batch?
  • How are horizon-truncated nodes labeled?
  • How does BQPO perform when exact observation repeats are rare?
  • Can you re-run GraphGPO in your codebase and add an exact-only BQPO baseline, with more seeds?
  • Can you ablate the two router routes separately, and add a random-grouping control that keeps the exact rewards?
Reviewer_E1nhOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper discuss a simple heuristic based on state information to learn under agentic environment.

Critical strengths

simple approach

Critical weaknesses

The numerical results are only not sufficient to demonstrate it is a good approach.

Questions for authors

The results are not promising ?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.