acceptodds
Under review as a conference paper at ICLR 2027

FORGE: An Execution-Feedback-Guided Framework for Benchmarking Generalist Robot Policies

Abstract

As Vision-Language-Action (VLA) models advance toward open-ended robotic manipulation, reliable evaluation of their performance across diverse capability dimensions becomes increasingly important. Yet comprehensive evaluation remains difficult due to the multimodal nature of VLA inputs and the cost of constructing executable simulation environments and reliable oracles at scale. Existing VLA benchmarks largely rely on manually specified or template-instantiated task-scene-oracle triplets constructed independently of model performance, resulting in limited input diversity, overlooked process failures, and only coarse-grained difficulty levels. In this paper, we introduce **FORGE**, an execution-feedback-guided framework that dynamically generates capability-specific mutations from validated seed tasks and scenes using observed VLA execution behavior, enabling iterative exploration of model failure modes. Using FORGE, we build **FORGE-Bench**, comprising 215 task-scene-oracle variants generated along validated mutation trajectories across five capability dimensions. Evaluation of five representative VLA models shows dimension-specific failures, terminal-process success gaps, and local performance reversals along shared mutation trajectories, providing a discriminative characterization of VLA capabilities.

Primary area:
datasets and benchmarks
Keywords:
VLA benchmark, Embodied AI, Adaptive Benchmark Generation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_jJibOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

The paper uses feedback(human designed rules) from a fixed VLA to keep adjusting scenes and tasks. It combines scene generation, execution checks, and feedback-guided changes into an evaluation framework(In fact, LLM guided env. generation is normal). The experiments evaluate several VLAs. Exp. how that guided changes find harder cases more effectively.

Critical strengths
  1. The pipeline is complete. It reduces the manual work to build tests(from results they find harder cases than random changes).

2.The evaluation covers a useful range of dimensions.Vision, language, deployment settings, long-horizon tasks, and safety, human-designed, offering several ways to test models.

Critical weaknesses
  1. The novelty is limited. Scene generation(LLM-driven,see robotwin series and so on) and feedback-driven testing(last year someone has lifted) have both been explored(everywhere in github). This feels more like an integration of existing ideas.

2.The added value is unclear. The method finds more failures, more test, more data, but what can actually learn from them is less clear. It seems less useful, isn't it? (in today's multiple, much more huge dataset); If you focus on model's limitation and improve it, why not RL? If your work valuable, show me.

3.The search baseline is weak. Human designed, how much the detailed process feedback actually helps unclear. Can you compare with the video method or llm-driven method or something to show your advantage?

4.IF you persist on your work, the exp. is not enough.How well these tests reflect model capabilities?(five root tasks is not enough);the failure labels do not necessarily explain the causes; different tasks or source model maybe have an another conclusion(your work is built around particular models).

Questions for authors

See Critical Weaknesses Please.

Reviewer_7R7vOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces a benchmark that automates evaluation of VLA models by automatically generating a scene for the task using FORGE method, creating a sophisticated oracle to evaluate policy and a process to adjust simulation to fully explore some failure modes of the each particular model. The paper provides good experimental validation for its claims, however, some methodological issues regarding limited number of tasks and Feedback-Guided mutation remain.

Critical strengths

Overall, the paper presents a nice contribution toward robust evaluation of VLA models, allowing one to identify failure points of the policies automatically. Paper covers full lifecycle from automated scene generation, that is improving upon prior methods, to robust oracle/judge that makes sure the task is solved properly (i.e. put banana on the plate is not completed successfully after policy has put all other objects there, as is case in naive completion checkers), and finalizing with model error driven generation of perturbation to properly identify where model fails.

Critical weaknesses

The paper at the current stage has two major critical weaknesses:

  1. Limited number of tasks (N = 5 in Table 12 in Appendix) which are pick-place only, makes this method validated only on the simplest kind of manipulation tasks. It is not certain how pipeline will work with more complex tasks, that require precision (stacking of two objects on top of each other, pouring, pushing into correct spot).
  2. From my understanding, calibration of automatic perturbations requires VLA models. This might introduce significant biases into the whole benchmarking process, since the mutation strengths are primarily a function of calibration VLA competency. This also makes comparing multiple models against each other (say in a RoboLab way) hard to do properly, because of the before mentioned bias.
Questions for authors

Regarding "Critical Weaknesses", I have following questions:

  1. Do you envision any limitations in extension of this framework towards other tasks that are not pick-place like now? What difficulties/problems can occur?
  2. Currently, calibration of mutation strengths requires other VLA models (Section 4.3), which introduces quite a strong model bias when trying to evaluate policies against each other. Is there a possibility to generate those in model-agnostic way?
1 comment

Dear authors,

thank you for providing thorough ablations and explanations. I keep my Weak acceptance decision for the paper.

Reviewer_6MSeOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

FORGE combines automated scene construction, a process-aware oracle, and execution-feedback-guided mutation to benchmark VLA policies. FORGE-Bench contains 215 variants derived from five seed tasks across five capability dimensions. Validation includes scene-construction comparisons on 75 tasks, oracle assessment against 396 human-reviewed rollouts, budget-matched search comparisons, and evaluation of five policies, including two held-out models.

Critical strengths
  1. The process-aware oracle addresses a meaningful evaluation gap: terminal completion can conceal unsafe or unintended interactions. On the annotated sample, it improves agreement with human process judgments from 79.3% to 95.7%, reducing false successes from 75 to 4. This provides concrete evidence for checking execution traces.

  2. Controlled variants expose policy differences that aggregate success rates miss. The held-out evaluations and examples of divergent responses to the same mutation support the benchmark’s diagnostic usefulness beyond its construction policies.

Critical weaknesses
  1. The evidence for the central search advantage is incomplete. Table 4 reports substantial gains under matched rollout budgets, but the task–dimension coverage, random-walk protocol, independent repetitions, and uncertainty are unclear. It is also unclear whether selected improvements were confirmed with fresh rollouts. A comparison against adaptive search using only success rates would help establish the specific value of failure-mode feedback.

  2. The benchmark has limited independent task coverage. All 215 variants originate from five pick-and-place seeds using one fixed-base embodiment, and the listed LONG variants contain two goals. Dense variation is useful, but held-out policy results on these seeds do not establish diagnostic coverage across different task families. The broader 75-task scene-construction experiment does not resolve this limitation.

Questions for authors
  1. Search validation: Which task–dimension pairs and how many independent searches underlie Table 4? Do the gains persist when selected candidates are evaluated with fresh rollouts?

  2. Diagnostic validation: Was the human audit limited to binary terminal/process judgments, or were the individual failure labels used by the selector in Eq. 10 also checked? What evidence supports their accuracy?

  3. Metric clarification: Does reported PSR require both terminal completion and absence of process violations? Section 3.3 describes joint success, whereas Appendix F.1 defines process success separately. Please reconcile these definitions and clarify the interpretation of TSR–PSR.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.