acceptodds
Under review as a conference paper at ICLR 2027

GEAR: Co-evolving Grounded VLM Agents with Environment-Aligned Rubrics

Abstract

Training multi-turn VLM agents from environment feedback remains challenging because outcome rewards indicate whether a trajectory succeeds, but provide limited supervision for why intermediate visual reasoning steps and actions succeed or fail. Existing dense process rewards are often manually specified or fixed, which makes them difficult to adapt to the evolving trajectory distribution induced by RL. We propose GEAR (Grounded Environment-Aligned Rubrics), a co-evolutionary training framework that jointly optimizes a VLM policy and a rubricator that self-generates grounded rubrics for spatial grounding, action legality, temporal consistency, and transition progress. Each rubric is used as an auxiliary process reward only when its satisfaction is statistically aligned with grounded environment feedback, such as task success, return, legality, or progress, which prevents self-generated supervision from drifting toward textual self-consistency. Experiments on multi-turn VLM agent benchmarks show that GEAR improves over SFT and outcome-centric RL baselines, with ablations confirming that environment-aligned rubric evolution is important for stable and interpretable policy improvement.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
VLM Agents, Reinforcement Learning, Structured Rubrics

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_ip1oOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces GEAR (Grounded Environment-Aligned Rubrics), a novel co-evolutionary training framework designed to improve multi-turn Vision-Language Model (VLM) agents. Traditionally, training these agents via Reinforcement Learning (RL) relies on sparse outcome rewards that only indicate overall success or failure. GEAR solves this by using a single model with two distinct, coupled roles: a solver that interacts with the environment and reasons through tasks, and a rubricator that self-generates evaluation rubrics covering criteria like spatial grounding, action legality, and temporal consistency. Specifically, GEAR validates these self-generated rubrics against actual environment feedback, such as task success or return. This two-stage (SFT and RL) allows the agent's process supervision to evolve dynamically alongside its behavior. Empirically, a 3B parameter GEAR model achieved highly competitive results with an overall score of 0.87, outperforming standard baselines and even surpassing zero-shot proprietary models, particularly in complex 3D navigation tasks.

Critical strengths
  • By introducing a co-evolutionary framework where a single model acts as both the solver and the rubricator, GEAR eliminates the need to continuously query expensive, high-latency proprietary APIs (like GPT-4V) for process supervision during massive RL exploration.
  • Using a relatively small 3B-parameter backbone to demonstrate promising empirical results
  • Because the rubrics are generated in natural language (covering spatial grounding, action legality, temporal consistency, etc.), the reasons for policy improvement or failure are highly interpretable
Critical weaknesses
  • Given the fact that this paper is tackling rubrics-based RL, including an existing rubrics-based RL baseline should demonstrate the advantage of this work, e.g., any method mentioned in the related work.
  • needs space for a Solverthat in the caption of figure 1
  • The notations are somehow confusing, e.g., in Line 184, sparse outcome reward is denoted with subscript t, which I thought sparse reward here should be only final task reward per trajectory.
  • In line 783, Table??
  • For those experiments: (Sokoban and FrozenLake), embodied 3D control (Navigation and PrimitiveSkill), SVG reconstruction, please cite the references in the main context, too.
  • Authors should be carefully define and refer the variables and notations, e.g., in line 204 - line 209, it is mentioned that an environment reward rtenvr_{t}^{env} is received, does it imply the assumption that this work can only deal with dense reward environment? And then later authors describe equation 9, where I did not find the relationship between reward rtenvr_{t}^{env} and z(τ)z(\tau).
Questions for authors
  • From line 57 - line 61, authors argue that a rubric may be more challenging in VLM agents. Could you elaborate it more because I think the issues you mention are all related to interactive settings rather than purely VLM agents themselves.
  • In line 072, does it mean that the initialization of SFT not only requires expert trajectories but also initialized rubrics? Then how do those rubrics are generated in practice?
  • In line 126, why \mathcal{O} and \mathcal{\Omega} are both observation?
  • I am still not fully convinced why outcome rewards provide limited guidance on how an agent should interpret visual states, update its reasoning, or choose actions across intermediate steps if the outcome reward is in dense environment, providing step-wise feedback.
  • In line 369, what is intrinsic reward?
  • Could you elaborate more on section 4.3 and 4.4? It is still not clear to me how the self-evolving rubrics are reliable when evaluating the trajectories (false positive, true negative)? And if we already have step-wise reward, why we still need to have the step-wise rubrics reward? Will the coefficient between different components in the reward function be sensitive?
4 comments
AuthorsOpenReview ↗

Response to Reviewer ip1o

We thank the reviewer for the careful reading and constructive feedback. Below, we clarify the reward formulation, initialization, and reliability of rubric supervision, and address the requested comparisons.

W1. Comparison with existing rubric-based reinforcement learning

We appreciate the suggestion to strengthen the comparison with rubric-based RL. Table 2 already tests two GEAR components: removing validation lowers Sokoban/FrozenLake success from 0.86/0.82 to 0.79/0.76, while removing rubric evolution yields 0.78/0.78. These are component ablations, not reproductions of prior methods.

We additionally evaluate two baselines adapted from prior rubric-RL methods to our VLM environments, using the same backbone, SFT checkpoint, environment-interaction budget, and evaluation protocol. The RaR baseline (Gunjal et al., 2025) pre-generates task-specific rubrics and uses a fixed judge to aggregate their scores into a reward. The DR Tulu baseline (Shao et al., 2025) evolves rubrics but retains them by rollout-score variance rather than environment-outcome alignment.

Table I1. Additional rubric-RL comparisons.

MethodSokoban SRFrozenLake SR
RaR (Gunjal et al., 2025)0.770.76
DR Tulu (Shao et al., 2025)0.800.78
GEAR, matched configuration0.860.82

GEAR exceeds the fixed-judge baseline by 0.09/0.06 and the variance-selected baseline by 0.06/0.04 on Sokoban/FrozenLake. These matched comparisons extend the evidence beyond the component ablations in Table 2.

W2–W6. Reward notation and presentation corrections

Thank you for highlighting this ambiguity. We have clarified the relationship between per-turn environment rewards and trajectory-level feedback in the revised manuscript. A time index does not imply dense rewards: for example, a terminal-success reward can be written as

=\mathbb{I}[t=T-1]\,\mathbb{I}[\tau\text{ succeeds}].$$ Here, $r_t^{\mathrm{env}}$ is the reward used for policy optimization, while $z(\tau)=g_{\mathcal E}(\tau)$ is the trajectory-level feedback used to condition rubric generation (Eq. 9) and validate and select the generated criteria (Eqs. 13–14). In the terminal-success example above, $z(\tau)=r_{T-1}^{\mathrm{env}}$. Appendix A.2 specifies trajectory success as the target for Sokoban/FrozenLake and environment reward for SVG. The same outcome can therefore support both policy learning and rubric validation. We have also made several small edits to further improve notation consistency and presentation, such as: - Changed “Solverthat” to “Solver that” in the Figure 1 caption. - Clarified in Sec. 3.2 that a time-indexed environment reward can be terminal-only. - Replaced “Table ??” in Appendix C.1 with the main-results table reference. - Added the benchmark citations directly to Sec. 5.1. > Q1. What additional challenge arises for VLM agents? We appreciate the opportunity to clarify this point. Sequential credit assignment also arises in text-based agents. Visual interaction adds the need to check reasoning against image evidence: a statement can sound plausible while misidentifying an object's position, overlooking an obstacle, or proposing an action inconsistent with the observed scene. For example, a solver may describe a rightward push as feasible although a wall blocks it. We have revised the Introduction to make this distinction explicit. > Q2. How are rubrics initialized for SFT? Appendix B.1 describes the cold-start corpus: 20,000 mathematical-reasoning examples and 20,000 rubric-generation demonstrations produced by the teacher model specified there. The latter initialize the rubricator’s generation capability, **not a fixed rubric bank**. For these demonstrations, the teacher receives task-specific rubricator instructions and multimodal trajectory context, including the visual observation, solver response, and available history and environment feedback. It produces grounded analysis and rubric entries in the same JSON schema used during RL (`id`, `score`, and criterion `description`). We retain examples with valid visual inputs, parseable rubric outputs, required fields, and aligned multimodal messages. We have clarified this procedure in Appendix B.1; Appendix G provides the task-specific prompt. During RL, **the teacher is not queried and SFT-generated rubrics are not reused**. The current rubricator generates new candidates from the solver’s trajectories, which are then subject to environment-aligned validation. The verifier-distillation data in Appendix B.2 are separate from these SFT demonstrations.

Q3. Observation space versus observation model

Thank you for flagging the similar notation. In Eq. (1), O\mathcal O is the observation space, with ot∈Oo_t\in\mathcal O, while Ω\Omega is the observation model describing how observations arise from states. We have distinguished these explicitly at first use.

Q4. Why add rubric rewards when step-wise feedback is available?

Thank you for raising this distinction. Receiving a new observation after each action does not necessarily provide an informative reward for that decision. Even when step-wise environment rewards are available, they need not assess whether the agent correctly interprets visual evidence or updates its reasoning. For example, a distance-based reward measures progress toward a goal but does not directly check whether the agent identifies an obstacle or updates a stale belief. Validated rubrics provide complementary feedback on these intermediate properties. Our empirical claims concern the reward settings evaluated in the paper, and we have revised the wording to make this scope explicit.

Q5. Meaning of “intrinsic reward”

Thank you for pointing out the unclear terminology. “Intrinsic reward” in Sec. 5.1 refers to the rubric-based criterion reward rtcritr_t^{\mathrm{crit}} defined in Eq. (17), which is combined with the environment reward in Eq. (18). We have replaced the term with “rubric-based criterion reward” and added the equation reference to make this connection explicit.

Q6. Rubric reliability and reward-coefficient sensitivity

GEAR checks rubric reliability at two stages: the frozen verifier judges whether a trajectory satisfies a criterion, and environment-aligned validation checks whether criterion satisfaction is associated with better outcomes. On 467 human-annotated examples, the distilled verifier achieves:

Table I2. Verifier judgment quality.

MetricResult
Accuracy96.36%
Macro-F195.45%
False-positive rate2.95%
True-negative rate97.05%

These metrics assess satisfaction judgments; environment-aligned validation separately assesses whether the judged criteria provide useful training signals. Removing validation reduces success by 7, 6, and 9 percentage points on Sokoban, FrozenLake, and Navigation, respectively (Table 2).

Step-wise environment rewards can indicate progress without checking whether the agent correctly interprets visual evidence or updates its reasoning. Validated rubric rewards provide feedback on these intermediate properties.

The reward-mixture coefficient is λ(u)\lambda^{(u)} in Eq. (18), with maximum value λmax⁡\lambda_{\max}. We evaluated different values of λmax⁡\lambda_{\max} while keeping the initialization, interaction budget, and warmup and gating settings fixed. The 0.20.2 setting corresponds to the main GEAR configuration.

Table I3. Sensitivity to the maximum rubric-reward weight.

λmax⁡\lambda_{\max}Sokoban SRFrozenLake SR
00.760.75
0.10.840.81
0.20.860.82
0.30.850.81
0.50.810.79

Performance is highest at 0.20.2, remains close at 0.10.1 and 0.30.3, and declines at 0.50.5. At λmax⁡=0\lambda_{\max}=0, the solver receives no rubric reward, but the rubricator still updates the shared backbone; this setting is therefore not equivalent to outcome-only PPO.

Thanks for the authors detailed response with more experimental results. The manuscript gets much clear to me and I would like to keep my positive score.

Thank you so much for your positive feedback. We greatly appreciate your time and effort in reviewing our work and considering our clarifications.

Reviewer_V9mGOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces GEAR and uses a shared backbone as a solver and a rubricator simultaneously to train the multi-turn VLM agent. The self-generated rubric is verified by a distilled 3B verifier. Only if the satisfaction has a Pearson correlation higher than \alpha compared to the in-group environment outcome, the rubric will be kept to calculate process rewards. In the VAGEN environment, based on Qwen 2.5-VL-3B, GEAR has an overall performance of 0.87, outperforming VAGEN-Full at 0.82.

Critical strengths
  1. The method is simple and clear. This paper uses the correlation between rubric satisfaction and environmental outcomes to determine whether to keep it or not, and does not need manual process-reward design.

  2. The shared backbone design is efficient: Solver and Rubricator and do not need to maintain two independent models. Table 3 also analyzes memory costs.

  3. The design handles practical failure modes, including degenerative groups, EMA tracking, warm-up, and gating.

Critical weaknesses
  1. There seems to be a label leak. The rubricator input includes z(\tau), and the accepted rubric assigns reward to the same trajectories in filtering. In table 8, an accepted Sokoban rubric has a 1.00 correlation to the success label. So the criterion reward might just be a hindsight relabeling to the outcome rather than an independent process reward. Authors should compare this with simpler turn-level credit assignment.

  2. Verifier ablation is inconsistent with the core claim. A 48% accuracy (near chance) verifier in Table 5 can still achieve a 0.82 success rate in Sokoban and 0.78 success rate in Frozen Lake, which is higher than VAGEN-Full. It is not clear whether the rubric content is helpful or not.

  3. It would be great to add stronger baselines, such as rubric reward with a fixed judge (Gunjal et al., 2025), evolving rubric without environmental verification (Shao et al., 2025;Sheng et al., 2026), and PPO and GRPO from the same SFT checkpoint.

Questions for authors
  1. What if z(\tau) is removed from the rubricator input, and the filtering and reward assignment uses different groups?

  2. Does GEAR include VAGEN world-modeling reward?

  3. How many episodes are in each task evaluation?

4 comments
AuthorsOpenReview ↗

Response to Reviewer V9mG

Thank you for the careful review. We address outcome reuse, verifier reliability, and the experimental comparisons below.

W1 & Q1. Outcome conditioning and same-group selection

Outcome-conditioned discovery and process evaluation serve different roles. We condition candidate generation on environment outcomes to help identify task-relevant criteria (Eq. 9). A selected rubric is then evaluated against local observations, reasoning, actions, and transition evidence to compute the process reward (Eq. 17); the online verifier does not receive the explicit outcome target. For example, a validated no-deadlock criterion can penalize an irreversible push while accepting a safe but unfinished move, even if both episodes ultimately fail. It can therefore distinguish intermediate decisions that share the same terminal outcome.

Same-group selection requires a separate test. The correlation of 1.00 in Table 8 is an estimate for one rubric on one sampled comparison group, not a claim of validity on unseen trajectories. Selecting rubrics and rewarding the same trajectories can favor criteria that happen to match that group's outcomes.

Targeted controls. We compare the following settings using the same SFT checkpoint, total environment-interaction budget, and evaluation protocol:

  1. W/o-z(τ)z(\tau) generation: Remove the explicit outcome input. Outcomes remain available for rubric validation.
  2. Cross-group filtering: Generate and select rubrics using group A, then apply the fixed criteria to a disjoint group B for process rewards. Group B's outcome labels are not used to propose or select its rubrics.
  3. Simpler credit assignment: Compare matched outcome-only PPO with outcome-broadcast PPO, which replaces rubric scores with terminal success broadcast to each turn, using the same auxiliary-reward normalization and mixing schedule.

We also combine the first two controls.

Table R1. Controls for outcome conditioning, sample reuse, and credit assignment.

SettingSokoban SRFrozenLake SR
GEAR0.860.82
W/o-z(τ)z(\tau) generation0.830.79
Cross-group filtering0.810.77
W/o-z(τ)z(\tau) + Cross-group0.780.75
Outcome-only PPO0.700.71
Outcome-broadcast PPO0.720.71

Even under W/o-z(τ)z(\tau) generation and cross-group filtering, GEAR exceeds outcome-only PPO by 0.08/0.04 and outcome-broadcast PPO by 0.06/0.04 on Sokoban/FrozenLake. In this setting, candidate rubrics are still selected using environment feedback on group A and evaluated against local interaction evidence on disjoint group B. The retained advantage supports the value of this environment-validated rubric supervision beyond broadcasting terminal success to each turn.

W2. Verifier quality and rubric content

We thank the reviewer for raising this interesting question. To analyze whether the verifier’s judgments provide useful supervision despite its 48.82% aggregate accuracy, we added a matched random-judgment control. Each verifier judgment is sampled independently at random with equal probability for either outcome; rubric generation, environment rewards, validation, the SFT checkpoint, interaction budget, and reward schedule remain unchanged.

Table R2. Weaker verifier versus random judgments under matched training conditions.

Judgment sourceSokoban SRFrozenLake SR
Qwen3-VL-32B-Instruct0.820.78
Random judgments0.780.75

Table R3. Judgment accuracy by rubric family on the human-annotated evaluation set.

Judgment sourceSpatial GroundingAction LegalityTemporal ConsistencyTransition Progress
Qwen3-VL-32B-Instruct80.34%73.50%23.07%18.10%
Random judgments55.56%45.30%47.86%50.86%

Qwen3 performs substantially better than random judgments on Spatial Grounding and Action Legality. Spatial Grounding checks visible object positions and relations; Action Legality checks immediate constraints such as collisions, boundaries, and invalid pushes. Temporal Consistency and Transition Progress require tracking history and assessing changes toward a goal, where Qwen3 performs much worse. Its 48.82% aggregate accuracy therefore obscures useful judgments in the first two families. Under matched training, Qwen3 exceeds random judgments by 4 percentage points on Sokoban and 3 on FrozenLake, supporting the value of its judgments within GEAR.

Both conditions retain the grounded outcome reward, which helps explain why the random control still reaches 0.78/0.75; environment-aligned validation also limits the influence of misaligned judgments. Table 5 shows that the distilled verifier further raises success from 0.82/0.78 to 0.86/0.82. We have clarified that Qwen3’s advantage over VAGEN-Full reflects the full GEAR pipeline, while the matched random control tests the additional contribution of its verifier judgments.

W3. Stronger rubric-RL baselines and matched initialization

Our existing controls test two relevant mechanisms: fixed rubrics achieve 0.78/0.78, and evolving rubrics without validation achieve 0.79/0.76, compared with GEAR's 0.86/0.82 on Sokoban/FrozenLake (Table 2). These are component ablations, not reproductions of the cited methods.

We further evaluate baselines adapted from Gunjal et al. (2025) and Shao et al. (2025) to our VLM environments, using the same SFT checkpoint, environment-interaction budget, and evaluation protocol. The PPO and GRPO results below are also obtained from this matched SFT checkpoint. The fixed-judge baseline scores rubrics prepared before RL directly using a frozen judge. In the evolving-rubric baseline, we retain rubrics by their score variance across rollouts rather than environment-outcome alignment, while keeping the remaining training setup matched.

Table R4. Additional baselines under matched training conditions.

MethodSokoban SRFrozenLake SR
PPO, outcome-only0.700.71
GRPO, outcome-only0.680.70
Fixed-rubric, fixed-judge RL (Gunjal et al., 2025)0.770.76
Evolving-rubric RL without environment-aligned filtering (Shao et al., 2025)0.800.78
GEAR0.860.82

GEAR exceeds the variance-selected baseline by 0.06/0.04 and the fixed-judge baseline by 0.09/0.06 on Sokoban/FrozenLake. The matched variance-selected comparison supports the contribution of environment-aligned selection beyond rubric evolution alone.

Q2. Does GEAR include VAGEN's world-modeling reward?

No. GEAR combines the environment reward with the validated rubric reward:

rtsol=rtenv+λ(u)rtcrit.r_t^{\mathrm{sol}} = r_t^{\mathrm{env}} + \lambda^{(u)}r_t^{\mathrm{crit}}.

It does not include VAGEN's world-modeling reward.

Q3. Evaluation episodes and samples

We follow VAGEN’s evaluation configuration and use the same episode or sample counts for GEAR and VAGEN-Full. Table R5 lists the counts for each task.

Table R5. Evaluation protocol.

Benchmark / splitEpisodes or samples per evaluation
Sokoban, main table128
FrozenLake, main table128
Navigation — Base60
Navigation — Common60
PrimitiveSkill — Place / Stack / Drawer / Align16 / 16 / 16 / 16
SVG reconstruction32

I thank the authors for their rebuttal. Most of my concerns are resolved and I could raise my score to 3.

Thank you very much for your timely reply. We sincerely appreciate your confirmation that our responses have addressed your concerns. We are also grateful for your valuable comments and constructive suggestions, which have significantly improved the quality and clarity of our manuscript.

Reviewer_QXmSOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces GEAR, a training framework that jointly optimizes weight-sharing solver and rubricator for VLM agents. The rubricator generates process criteria within 4 dimensions: spatial grounding, action legality, temporal consistency and transition progress. A frozen verifier evaluates the criterion with environment outcomes. The final objective is jointly defined by outcome reward, process reward and rubricator reward and optimized through PPO. Experiments across 5 distinct tasks report improvements with open-source models and proprietary models over different baselines.

Critical strengths
  1. Clear motivation from the limitation of fixed process rewards. GEAR generates and validates criterias with environment feedback, providing a mechanism that adapts supervision to task performance.
  2. The paper shows comprehensive experiments to show the effectiveness of the framework and support the utility of the components. Sharing the solver and rubricator backbone makes the pipeline more efficient.
Critical weaknesses
  1. The rubrics are dynamicly generated and verified, while Appendix G provides fixed generation and verification prompts with many details about the tasks. Initial prompts without detailed task prior or the process of how the prompts evolve through the training are better to be included to explain the improvements.
  2. Max number of interaction turns in the tasks is just 4, making the claims about long-horizon reasoning weak.
Questions for authors
  1. Can you provide initial prompts without detailed task prior or the process of how the prompts evolve through the training?
  2. Can you provide the results and observations on tasks requiring more interation turns?
1 comment
AuthorsOpenReview ↗

Response to Reviewer QXmS

We thank the reviewer for the constructive feedback. We clarify the role of the fixed prompts and the scope of the interaction-horizon results below.

W1 & Q1. Fixed prompts and evolving rubrics

Thank you for highlighting the need to distinguish the prompts from their outputs. In GEAR, the generation and verification templates and four rubric dimensions remain fixed throughout each training run. The shared model parameters are updated during RL, allowing the rubricator to produce trajectory-conditioned criteria and weights; environment-aligned validation selects which criteria contribute to training. Thus, the generated rubrics evolve while the prompts remain fixed.

To probe sensitivity to task-specific prompting, we evaluate three generation-and-verification prompt configurations. Reduced-prior prompts retain each task’s basic interface—objects, available actions, and objective—while removing detailed heuristics and fixed thresholds. Generic prompts contain no task-specific objects or action rules and use the supplied task instruction and interaction evidence to generate and verify criteria.

Table Q1. Success rates under different prompt configurations.

Prompt typeSokobanFrozenLake
Generic0.820.79
Reduced-prior0.840.80
Task-specific0.860.82

The maximum difference is 0.04 on Sokoban and 0.03 on FrozenLake. GEAR retains most of its performance with generic and reduced-prior prompts, while task-specific guidance provides a modest additional gain. The exact reduced-prior prompts for Sokoban and generic prompts are provided in Appendix G.3 and G.4, respectively.

W2 & Q2. Interaction horizon and the scope of the claims

Thank you for highlighting the distinction between interaction turns and atomic actions. The original tasks are capped at 2–4 turns. Although a turn may contain several atomic actions, it remains one observation–reasoning–action cycle. We have revised claims such as “long-horizon spatial memory” and “mastering long-horizon dynamics” to reflect the scope of the original experiments.

To examine longer interaction horizons, we retrained GEAR and VAGEN-Full at the benchmark-specific training and evaluation turn limits shown in Table Q2, retaining the same action and observation interfaces and using matched test instances across methods.

Table Q2. Success rate (SR) under extended interaction limits.

BenchmarkTraining/evaluation turn limitVAGEN-Full SRGEAR SR
Navigation40.810.87
Navigation80.840.89
Navigation120.850.90
FrozenLake30.740.82
FrozenLake60.780.84
FrozenLake90.800.85

GEAR also completes some evaluation instances after the fourth interaction turn. For example, on FrozenLake, 3.91% and 4.69% of instances were completed at turn five or later under the 6- and 9-turn limits, respectively.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.