acceptodds
Withdrawn from ICLR 2027

OrbitPO: Learning Interface Equivariance for Tool-Using Language Agents

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Tool-using language agents act through software interfaces that are routinely refactored—tools renamed, fields reordered, entity handles relabeled—while the underlying tasks remain unchanged. Because such exact interface changes preserve task semantics yet still break capable agents, robustness to them is a prerequisite for integrating agents into evolving software. First, existing consistency objectives compare candidate-normalized action distributions, which conceal cross-interface shifts in unconditional action probability and yield no signal when only one can- didate is present. Second, view-local reward normalization and prefix matching discard cross-interface supervision precisely when equivalent rollouts fail or di- verge, a loss that grows as episodes lengthen. To address the first challenge, we pro- pose Orbit-Equivariant Policy Optimization (O RBIT PO), whose mass-preserving semantic alignment matches inverse-mapped unconditional candidate probabili- ties together with an explicit residual mass, supported by an exact information decomposition and non-asymptotic bounds that link retained-mass dispersion to alignment. To address the second, O RBIT PO introduces orbit-relative credit and counterfactual complete-history replay, which pool rewards across equivalent in- terfaces and align every visited history without requiring matched prefixes or successful continuations, with a coverage-dependent bound from replay loss to cross-interface return. Experiments on six core benchmarks, four frozen-transfer suites, nine adapted backbones, and controlled OrbitLab programs show that O R - BIT PO improves worst-view aggregate over exposure-matched augmentation on every backbone, by 6.7–9.9 points while keeping one policy call per decision.

Primary area:
reinforcement learning
Keywords:
reinforcement learning, tool-using language agents, interface equivariance, robust tool use, policy optimization, consistency regularization

Reviews

Among:
Shape:
1234this paper, Bottom 25%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.75 over 4 reviews, above 25% of the others.
Reviewer_ETnXOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes OrbitPO, a framework for improving the robustness of tool-using language agents to semantics-preserving interface transformations. It combines mass-preserving semantic alignment, orbit-relative credit assignment, and counterfactual complete-history replay to encourage cross-interface policy equivariance. Theoretical analysis establishes relevant properties under exact interface transformations, while extensive experiments across multiple benchmarks and model backbones demonstrate improvements in worst-view robustness over competitive baselines.

Critical strengths
  1. The mass-preserving alignment objective addresses a limitation of candidate-normalized consistency by retaining residual probability mass.
  2. The experiments results are comprehensive, which cover diverse benchmarks and model backbones. The improvements in worst-view robustness without additional inference calls are particularly encouraging.
  3. The paper is generally well written and easy to follow.
Critical weaknesses
  1. The method relies on exact, invertible, semantics-preserving mappings, whereas real-world API changes often involve semantic or behavioral modifications.
  2. OrbitPO incurs additional training overhead, and its performance falls slightly below LocalCanon-GRPO under approximately matched training FLOPs (53.8 vs. 54.6 WVA, Table 6).
Questions for authors

See Weaknesses

Reviewer_JxKDOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes OrbitPO, a policy optimization method that improves the robustness of tool-using agents to semantics-preserving interface changes. The method combines orbit-relative reward normalization with mass-preserving semantic alignment, allowing the policy to learn from equivalent interface views even when their trajectories diverge. The authors evaluate OrbitPO across multiple tool-use benchmarks and model families, showing consistent improvements over augmentation-based training and several robustness baselines.

Critical strengths

S1. A well-motivated method with clear technical contributions.

I particularly like the distinction between aligning candidate-normalized distributions and preserving unconditional probability mass. The authors show why normalizing over a candidate set can hide meaningful differences between interface views, especially when only one candidate is available. The complete-history replay mechanism also addresses a concrete limitation of prefix-based matching in longer trajectories.

S2. A thorough experimental evaluation.

The paper evaluates the method across several benchmarks and model families, with strong baselines, component ablations, and compute-controlled comparisons. The consistent improvement over Aug-GRPO is encouraging, and the experiments examining mass dispersion and consequential interface decisions provide useful evidence for the proposed mechanisms. I also appreciate that the authors explicitly test settings where their method is not expected to help.

Critical weaknesses

W1. The advantage over the strongest alternative is less convincing under matched compute.

Although OrbitPO substantially improves over Aug-GRPO, its advantage over LocalCanon-GRPO is only 1.3 points on Qwen3-8B, and the reported confidence interval includes zero. More importantly, LocalCanon performs slightly better when training FLOPs are matched (54.6 versus 53.8). This makes it difficult to establish whether OrbitPO offers a clear practical advantage over the simpler canonicalization approach, particularly when training resources are limited.

W2. The exact interface assumption needs further clarification.

The theoretical analysis relies on interface transformations preserving the underlying dynamics and rewards. However, Table 10 reports contract-pass rates between 98.7% and 99.4%, rather than 100%, on the audited histories. I would like to understand what causes these failures and whether they affect the interpretation of the theoretical guarantees or the empirical robustness results.

Questions for authors

Q1. Could the authors further clarify the comparison with LocalCanon-GRPO under matched training FLOPs? Given that LocalCanon slightly outperforms OrbitPO in this setting, it would be useful to know whether OrbitPO offers a consistent advantage at different compute budgets.

Q2. What causes the contract-check failures reported in Table 10, given that the transformations are intended to be exactly semantics-preserving? Were these cases included in the main evaluation, and how sensitive are the results to excluding them?

Reviewer_DiAkOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

OrbitPO aims to make tool-using agents robust to interface changes that leave the underlying task unchanged. It compares rewards across equivalent interfaces, replays histories under those interfaces, and aligns action probabilities without discarding probability outside the selected candidates. The work is well motivated, and the paper reports an extensive set of comparisons and component studies.

Critical strengths

This work is well motivated and the authors reports an extensive set of comparisons and component ablations. The idea itself looks timely and can be very practical.

Critical weaknesses

Sections 6.4–6.5 describe some results as “projected,” and Table 29 says its confidence intervals are “targets,” while the surrounding discussion treats them as computed results. This may be outdated wording, but it needs an explicit explanation, especially whether the statements come from an AI agent.

The comparison with the strongest practical baseline is also less convincing than the gains over augmentation alone. The reported 1.3-point advantage over LocalCanon has a confidence interval spanning zero, and LocalCanon performs better at approximately matched training compute. The authors acknowledge this, but it leaves the practical benefit of the additional training machinery unclear. Table 10 also reports failed checks of interface equivalence, without sufficiently explaining how those cases are handled.

Fig 5 and Table 7 on pg 9 does not look very informative to me.

Questions for authors

Could the authors distinguish completed measurements from projections, provide the records supporting the main results, and explain the failed equivalence checks? The methodological idea is interesting, but I recommend rejection in the current form because the empirical evidence cannot yet be assessed with sufficient confidence.

Reviewer_dSoBOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The authors aim to address the issue of how LLMs interact with evolving software interfaces. Over time the API and returned values from an interface may change and models should be robust to this. They highlight two key challenges:

i) To understand the consistency of a model's actions across two interfaces standard approaches focus on a sub set of the possible actions and discard the probability mass the model places on actions not in this subset. This can be misleading when the model isn't conditioned on this small subset but the entire space of actions.

ii) Improving the training reward signal. The authors highlight that the reward for the same task under different interfaces can be compared despite having different interface specific prompts. They also argue that if interface mappings are known credit assignment can be done effectively in multi-step traces by manually adapting the interface specific calls in previous steps to target a specific setup for a new interface.

Critical strengths

I think the authors have identified an interesting problem and specified two interesting ideas to tackle this problem; namely a more principled consistency analysis and better reward shaping during training. The paper provides theoretical results alongside empirical experiments that appear to show OrbitPO consistently outperforms other GRPO baselines.

Critical weaknesses

In the AI Use Statement the authors mention that "LLMs were employed for polishing the clarity and readability of text". In light of this I strongly advise the authors to re-write much of this paper without an LLM! The ideas I believe the authors are trying to communicate are simple and reasonably motivated, it's such a shame that the writing is what lets down this paper. I struggled to parse many of the ideas and had to ask an AI multiple times to reword and interpret the paper's explanations. It is because of the difficulty I had reading the paper that I have recommended rejecting the paper. Fundamentally I think the paper needs to be re-written and this is beyond the scope of the rebuttal period of any conference.

I would remind the authors that the conference policy states that reviewers are not required to engage with the appendices. Another problem with the communication of this work is that key metrics and baselines are only detailed in the appendix and referred to broadly before this is highlighted to the reader. Specifically:

  • WVA is a key metric in the empirical results section of the paper, used in most figures and tables, the exact details of this metric are buried in appendix E of the paper.
  • GRPO-Aug and other baselines are only fleshed out in Appendix E. It is important that these methods are highlighted in the text to better understand just how important OrbitPO's results are. My advice to the authors is to re-work the results e.g. reduce the size of Figures 4 and 5, to allow more space to properly explain the experiment protocol within the main body of the text.

To justify my statements I include the following extensive but not conclusive list of undefined terms and variables within the opening 4 sections of the paper.

Equation 1 (p. 1)

  • gh, gT, gagh,\ g\mathcal{T},\ ga: what it means to apply a map to a history, an interface or an action is never stated. Eq. 2 then uses the same notation for gτg\tau, got+1go_{t+1} and gP(⋅)gP(\cdot).
  • πθ\pi_\theta, θ\theta: undefined in the text beyond being referred to as the policy, clarified as an LLM policy in the figure.

Section 3 opening and Equation 2 (p. 3)

  • τ\tau in R(τ)R(\tau): the trajectory is never defined, indeed it is more consistently introduced as hh.
  • A(T)\mathcal{A}(\mathcal{T}): the action set is implied, not defined in any detail - are actions freely generated text or specific API calls and functions?
  • PP: the transition kernel is undefined. Is this a simple deterministic concatenation of the next action to the existing sequence as in typical LLM literature?

Equation 3 (p. 3)

  • mm is defined as a view - which is itself never formally defined, I assume it is the same as an interface?
  • xx in μx\mu_x, σx\sigma_x: the subscript is never defined. From context it seems to be the semantic task or another notation for the entire history hh.
  • i,ki, k: never explicitly tied to views and trajectories.
  • ϵ\epsilon: undefined, presumably a numerical-stability constant - but you have to mention this!

Equation 4 (p. 3)

  • ϵc\epsilon_c: the clip range is undefined.
  • tt: the token index is implied by "token likelihood ratio" but never stated. It is unclear how the loss is aggregated over tokens - token-mean, seq-mean-token-mean etc...?

Equation 5 (p. 4)

  • gig_i: presumably the map for view ii, but not stated - again ii index not linked to the view.
  • pθτp_\theta^\tau: Is this the policy - which was previously defined as πθ\pi_{\theta}? The superscript τ\tau is undefined.
  • zi:z_i: Defined as q~(Cx)\tilde{q}(C_x) where q~\tilde{q} is defined on the elements of CxC_x not the set. How does this work?

Equation 7 (p. 4)

  • JSD1/m\mathrm{JSD}_{1/m}: the weighted generalised Jensen–Shannon divergence is not defined.
  • zˉ\bar{z}: undefined, presumably the mean of the ziz_i.
  • wiw_i: undefined.
  • r⋆r_\star: undefined.

Equation 8 (p. 4)

  • λeq\lambda_{\text{eq}}, β\beta: the loss weights are undefined, and no values are given in the main text, presumably both are >0> 0.
  • Rref\mathcal{R}_{\text{ref}}: undefined, presumably a KL-to-reference regulariser.

Proposition 1 / Equation 9 (p. 4)

  • δt(i,j)\delta_t(i,j): the quantity being bounded is undefined.
  • MtM_t: defined in words as "the largest coarsened view-to-barycenter KL," but "coarsened" is undefined.
  • GG in I(G;C~∣xt,Ct)I(G; \tilde{C} \mid x_t, C_t): undefined.
  • C~\tilde{C}: undefined.
  • xtx_t: undefined, and it gains a time index that xx didn't have before.
  • CtC_t: undefined, and inconsistent with CxC_x in Eq. 5.
  • Ji,JjJ_i, J_j: undefined, presumably the expected return in view ii.
  • E[bt]\mathbb{E}[b_t]: the expectation is over an unspecified distribution.

Theorem 1 (p. 4)

  • These are undefined concepts rather than symbols, but they're needed to read the statement: "finite stabilizer," "chart," "legal history," and "applicable groupoid morphism."
Questions for authors

n/a

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.