- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces GEAR (Grounded Environment-Aligned Rubrics), a novel co-evolutionary training framework designed to improve multi-turn Vision-Language Model (VLM) agents. Traditionally, training these agents via Reinforcement Learning (RL) relies on sparse outcome rewards that only indicate overall success or failure. GEAR solves this by using a single model with two distinct, coupled roles: a solver that interacts with the environment and reasons through tasks, and a rubricator that self-generates evaluation rubrics covering criteria like spatial grounding, action legality, and temporal consistency. Specifically, GEAR validates these self-generated rubrics against actual environment feedback, such as task success or return. This two-stage (SFT and RL) allows the agent's process supervision to evolve dynamically alongside its behavior. Empirically, a 3B parameter GEAR model achieved highly competitive results with an overall score of 0.87, outperforming standard baselines and even surpassing zero-shot proprietary models, particularly in complex 3D navigation tasks.
Critical strengths
- By introducing a co-evolutionary framework where a single model acts as both the solver and the rubricator, GEAR eliminates the need to continuously query expensive, high-latency proprietary APIs (like GPT-4V) for process supervision during massive RL exploration.
- Using a relatively small 3B-parameter backbone to demonstrate promising empirical results
- Because the rubrics are generated in natural language (covering spatial grounding, action legality, temporal consistency, etc.), the reasons for policy improvement or failure are highly interpretable
Critical weaknesses
- Given the fact that this paper is tackling rubrics-based RL, including an existing rubrics-based RL baseline should demonstrate the advantage of this work, e.g., any method mentioned in the related work.
- needs space for a Solverthat in the caption of figure 1
- The notations are somehow confusing, e.g., in Line 184, sparse outcome reward is denoted with subscript t, which I thought sparse reward here should be only final task reward per trajectory.
- In line 783, Table??
- For those experiments: (Sokoban and FrozenLake), embodied 3D control (Navigation and PrimitiveSkill), SVG reconstruction, please cite the references in the main context, too.
- Authors should be carefully define and refer the variables and notations, e.g., in line 204 - line 209, it is mentioned that an environment reward is received, does it imply the assumption that this work can only deal with dense reward environment? And then later authors describe equation 9, where I did not find the relationship between reward and .
Questions for authors
- From line 57 - line 61, authors argue that a rubric may be more challenging in VLM agents. Could you elaborate it more because I think the issues you mention are all related to interactive settings rather than purely VLM agents themselves.
- In line 072, does it mean that the initialization of SFT not only requires expert trajectories but also initialized rubrics? Then how do those rubrics are generated in practice?
- In line 126, why \mathcal{O} and \mathcal{\Omega} are both observation?
- I am still not fully convinced why outcome rewards provide limited guidance on how an agent should interpret visual states, update its reasoning, or choose actions across intermediate steps if the outcome reward is in dense environment, providing step-wise feedback.
- In line 369, what is intrinsic reward?
- Could you elaborate more on section 4.3 and 4.4? It is still not clear to me how the self-evolving rubrics are reliable when evaluating the trajectories (false positive, true negative)? And if we already have step-wise reward, why we still need to have the step-wise rubrics reward? Will the coefficient between different components in the reward function be sensitive?
4 comments
Q3. Observation space versus observation model
Thank you for flagging the similar notation. In Eq. (1), is the observation space, with , while is the observation model describing how observations arise from states. We have distinguished these explicitly at first use.
Q4. Why add rubric rewards when step-wise feedback is available?
Thank you for raising this distinction. Receiving a new observation after each action does not necessarily provide an informative reward for that decision. Even when step-wise environment rewards are available, they need not assess whether the agent correctly interprets visual evidence or updates its reasoning. For example, a distance-based reward measures progress toward a goal but does not directly check whether the agent identifies an obstacle or updates a stale belief. Validated rubrics provide complementary feedback on these intermediate properties. Our empirical claims concern the reward settings evaluated in the paper, and we have revised the wording to make this scope explicit.
Q5. Meaning of “intrinsic reward”
Thank you for pointing out the unclear terminology. “Intrinsic reward” in Sec. 5.1 refers to the rubric-based criterion reward defined in Eq. (17), which is combined with the environment reward in Eq. (18). We have replaced the term with “rubric-based criterion reward” and added the equation reference to make this connection explicit.
Q6. Rubric reliability and reward-coefficient sensitivity
GEAR checks rubric reliability at two stages: the frozen verifier judges whether a trajectory satisfies a criterion, and environment-aligned validation checks whether criterion satisfaction is associated with better outcomes. On 467 human-annotated examples, the distilled verifier achieves:
Table I2. Verifier judgment quality.
| Metric | Result |
|---|---|
| Accuracy | 96.36% |
| Macro-F1 | 95.45% |
| False-positive rate | 2.95% |
| True-negative rate | 97.05% |
These metrics assess satisfaction judgments; environment-aligned validation separately assesses whether the judged criteria provide useful training signals. Removing validation reduces success by 7, 6, and 9 percentage points on Sokoban, FrozenLake, and Navigation, respectively (Table 2).
Step-wise environment rewards can indicate progress without checking whether the agent correctly interprets visual evidence or updates its reasoning. Validated rubric rewards provide feedback on these intermediate properties.
The reward-mixture coefficient is in Eq. (18), with maximum value . We evaluated different values of while keeping the initialization, interaction budget, and warmup and gating settings fixed. The setting corresponds to the main GEAR configuration.
Table I3. Sensitivity to the maximum rubric-reward weight.
| Sokoban SR | FrozenLake SR | |
|---|---|---|
| 0 | 0.76 | 0.75 |
| 0.1 | 0.84 | 0.81 |
| 0.2 | 0.86 | 0.82 |
| 0.3 | 0.85 | 0.81 |
| 0.5 | 0.81 | 0.79 |
Performance is highest at , remains close at and , and declines at . At , the solver receives no rubric reward, but the rubricator still updates the shared backbone; this setting is therefore not equivalent to outcome-only PPO.
Thanks for the authors detailed response with more experimental results. The manuscript gets much clear to me and I would like to keep my positive score.
Thank you so much for your positive feedback. We greatly appreciate your time and effort in reviewing our work and considering our clarifications.
Response to Reviewer ip1o
We thank the reviewer for the careful reading and constructive feedback. Below, we clarify the reward formulation, initialization, and reliability of rubric supervision, and address the requested comparisons.
We appreciate the suggestion to strengthen the comparison with rubric-based RL. Table 2 already tests two GEAR components: removing validation lowers Sokoban/FrozenLake success from 0.86/0.82 to 0.79/0.76, while removing rubric evolution yields 0.78/0.78. These are component ablations, not reproductions of prior methods.
We additionally evaluate two baselines adapted from prior rubric-RL methods to our VLM environments, using the same backbone, SFT checkpoint, environment-interaction budget, and evaluation protocol. The RaR baseline (Gunjal et al., 2025) pre-generates task-specific rubrics and uses a fixed judge to aggregate their scores into a reward. The DR Tulu baseline (Shao et al., 2025) evolves rubrics but retains them by rollout-score variance rather than environment-outcome alignment.
Table I1. Additional rubric-RL comparisons.
GEAR exceeds the fixed-judge baseline by 0.09/0.06 and the variance-selected baseline by 0.06/0.04 on Sokoban/FrozenLake. These matched comparisons extend the evidence beyond the component ablations in Table 2.
Thank you for highlighting this ambiguity. We have clarified the relationship between per-turn environment rewards and trajectory-level feedback in the revised manuscript. A time index does not imply dense rewards: for example, a terminal-success reward can be written as
=\mathbb{I}[t=T-1]\,\mathbb{I}[\tau\text{ succeeds}].$$ Here, $r_t^{\mathrm{env}}$ is the reward used for policy optimization, while $z(\tau)=g_{\mathcal E}(\tau)$ is the trajectory-level feedback used to condition rubric generation (Eq. 9) and validate and select the generated criteria (Eqs. 13–14). In the terminal-success example above, $z(\tau)=r_{T-1}^{\mathrm{env}}$. Appendix A.2 specifies trajectory success as the target for Sokoban/FrozenLake and environment reward for SVG. The same outcome can therefore support both policy learning and rubric validation. We have also made several small edits to further improve notation consistency and presentation, such as: - Changed “Solverthat” to “Solver that” in the Figure 1 caption. - Clarified in Sec. 3.2 that a time-indexed environment reward can be terminal-only. - Replaced “Table ??” in Appendix C.1 with the main-results table reference. - Added the benchmark citations directly to Sec. 5.1. > Q1. What additional challenge arises for VLM agents? We appreciate the opportunity to clarify this point. Sequential credit assignment also arises in text-based agents. Visual interaction adds the need to check reasoning against image evidence: a statement can sound plausible while misidentifying an object's position, overlooking an obstacle, or proposing an action inconsistent with the observed scene. For example, a solver may describe a rightward push as feasible although a wall blocks it. We have revised the Introduction to make this distinction explicit. > Q2. How are rubrics initialized for SFT? Appendix B.1 describes the cold-start corpus: 20,000 mathematical-reasoning examples and 20,000 rubric-generation demonstrations produced by the teacher model specified there. The latter initialize the rubricator’s generation capability, **not a fixed rubric bank**. For these demonstrations, the teacher receives task-specific rubricator instructions and multimodal trajectory context, including the visual observation, solver response, and available history and environment feedback. It produces grounded analysis and rubric entries in the same JSON schema used during RL (`id`, `score`, and criterion `description`). We retain examples with valid visual inputs, parseable rubric outputs, required fields, and aligned multimodal messages. We have clarified this procedure in Appendix B.1; Appendix G provides the task-specific prompt. During RL, **the teacher is not queried and SFT-generated rubrics are not reused**. The current rubricator generates new candidates from the solver’s trajectories, which are then subject to environment-aligned validation. The verifier-distillation data in Appendix B.2 are separate from these SFT demonstrations.