acceptodds
Under review as a conference paper at ICLR 2027

Let the Weak Judge: Annotation-Free Process Rewards for Multimodal E-commerce Relevance

Abstract

Outcome-only reinforcement learning often fails to produce evidence-grounded reasoning in discriminative multimodal tasks: with a small label space, GRPO routinely exploits lexical shortcuts and fabricates forced associations to match the correct label. Process reward models (PRMs) address this in principle, but typically require expensive step-level annotations and can mis-score reasoning outside their training distribution. We propose WISE-GRPO, which turns step-level verification into a verifiable answer-recovery test: we truncate the policy’s reasoning at intermediate steps, mask its upfront answer, and ask a committee of weak models to resume reasoning; a step is rewarded only when its explicit evidence guides the committee back to the correct label. The policy adopts an answer-first Post-CoT paradigm to anchor decisions, while the committee uses a Pre-CoT paradigm for independent continuation-based verification. Increasing step weights emphasize the integrative decisions near the end of the task template. Across four e-commerce benchmarks and two multimodal backbones, WISE-GRPO achieves the best accuracy among the evaluated methods in all eight settings. Its gains over outcome-only GRPO range from to accuracy points, and further analysis shows it reduces forced-association behavior, without requiring step-level human annotation.

Primary area:
reinforcement learning
Keywords:
process reward model, reinforcement learning, GRPO, multimodal large language models, e-commerce relevance, reasoning verification, annotation-free reward, chain-of-thought

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_4FT8OpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes WISE-GRPO, an annotation-free process reward framework for multimodal e-commerce relevance classification. The key idea is to evaluate intermediate reasoning prefixes based on whether a committee of weak models can recover the ground-truth label from them. The method combines answer-masked prefix continuation, weak-model ensemble scoring, and progressively weighted process rewards within GRPO.

Critical strengths

The idea of using independent models to assess the usefulness of intermediate reasoning without requiring human step-level annotations is practically appealing. The paper also includes several controlled analyses and ablations to examine the proposed reward mechanism.

Critical weaknesses

The main idea on whether an intermediate reasoning prefix enables a correct final answer is closely related to prior work on automatic process supervision, including Math-Shepherd, Rewarding Progress, and Free Process Rewards Without Process Labels. These methods have already explored deriving intermediate rewards from downstream outcome correctness without requiring human step-level annotations. Much of the contribution appears to be adapting established ideas to multimodal e-commerce relevance classification.

Although the paper compares against SFT, PC-DPO, outcome-only GRPO, and a static supervised-PRM baseline, it does not directly compare against existing automatic process reward methods such as Math-Shepherd, Rewarding Progress, or implicit process reward approaches.

Despite presenting a general approach to annotation-free process rewards, the experiments are restricted to e-commerce relevance classification. Three of the four benchmarks are internal industrial datasets, and only Amazon ESCI is publicly available. The evaluation uses just two relatively small multimodal backbones, Qwen2.5-VL-3B and Qwen3-VL-4B, with single-seed results.

Questions for authors

How does WISE-GRPO compare with existing annotation-free process supervision methods such as Math-Shepherd, Rewarding Progress, and Free Process Rewards Without Process Labels?

Reviewer_jgfnOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

WISE-GRPO uses a weak-model committee’s continuation accuracy on reasoning prefixes as a process reward for multimodal e-commerce classification. It combines this signal with outcome and format rewards during GRPO training.

Critical strengths

The method achieves consistent accuracy improvements across the evaluated settings without human step-level training annotations. The paper also acknowledges and quantifies its substantial training overhead.

Critical weaknesses

The need for costly online committee evaluation is not established. The reward closely resembles continuation-based process labeling, but the paper does not compare against a PRM trained on the same recovery targets. Given the reported 3.7× training time over outcome-only GRPO, this missing comparison leaves unclear whether online verification provides benefits beyond a potentially cheaper use of the same supervision.

Answer recovery is not sufficiently validated as a measure of evidence-grounded reasoning. An evaluator may recover the correct label from implicit answer cues or persuasive but unsupported reasoning. Masking the explicit answer and comparing against unrelated prefixes do not adequately rule out these possibilities. This weakens the central claim that the reward improves evidence grounding rather than simply making the intended label easier to recover.

Questions for authors
  1. Why not train a PRM using these labels? The method seems closely related to Monte Carlo-based process labeling: continue from a prefix and check whether the answer is correct. Why not use these scores to train a PRM and then use that PRM during RL? This could be much cheaper. Comparing against a PRM trained on the same labels would help explain the value of online committee evaluation.

  2. It is unclear how to choose the evaluators. A strong evaluator may solve the task without the prefix, while a weak one may fail even with useful reasoning. The paper studies this trade-off, but does not give a clear procedure for choosing suitable evaluators for a new task. How sensitive are the results to this choice?

  3. The training cost is substantial. Each rollout requires additional continuations from several evaluators at multiple steps. Training takes 3.7× as long as outcome-only GRPO. The authors acknowledge this, but a comparison under equal compute budgets would help show whether the improvement justifies the cost.

  4. Answer-first generation may encourage post-hoc justification. The paper shows that Post-CoT improves accuracy, but could generating the answer first encourage the model to invent reasoning that supports it? Removing the answer tag also does not guarantee that the reasoning contains no implicit answer cues. How does the method distinguish these cues from genuine evidence?

Reviewer_mhwGOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies multimodal e-commerce relevance classification with a small label space. WISE-GRPO masks the policy's final answer, asks three weak models to continue from an intermediate reasoning prefix, and uses their accuracy in recovering the true label as a process reward. Across ESCI and three private datasets, it improves over OR-GRPO by 1.5–4.8 points. Intact/Random prefix controls and forced-association analyses are used to argue that the reward tracks evidence rather than rhetorical shortcuts.

Critical strengths
  • Defining process reward through whether independent weak models can recover the label from a reasoning prefix avoids step-level annotation. The Intact/Random control also provides useful evidence that the reward responds to relevant content.
  • The phrase-based forced-association analysis is more objective than LLM judging and offers a useful diagnostic for rhetorical shortcuts learned from outcome-only rewards.
Critical weaknesses
  • The results are not yet very reliable. All experiments use a single seed, and the paper does not report test-set sizes. On the 3B model, WISE-GRPO is only 0.86, 1.01, and 0.97 points above SFT on QGR, ESCI, and CGI. The larger headline gains are measured against OR-GRPO, which is itself worse than SFT on some tasks.

  • Several important baselines are unclear. PC-DPO is neither defined nor cited, and the paper does not explain how the supervised reward model in MR-GRPO is trained. It also discusses related methods such as TaoSR-SHE, TaoSR-AGRL, and PAV without comparing against them.

  • In Table 7, the committee clearly helps on ESCI and QGR, but provides little benefit on CGI and NSM. Without an end-to-end ablation, it is unclear whether the committee is necessary.

Questions for authors
  1. Please report the sizes of the four test sets and clarify the implementation of PC-DPO and the training data and labels used for the MR-GRPO reward model. Is the 20K/80K CoT set in Section 4.3 an independent evaluation set?

  2. Why are TaoSR-SHE, TaoSR-AGRL, or PAV-style continuation rewards not compared? Please include at least one related baseline on the public ESCI dataset and report the values of λ\lambda and α1,…,αK\alpha_1,\ldots,\alpha_K in Eq. 2.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.