- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The author propose RLHARNESS, a two-stage training framework that alternates between training the policy and reconstructing dynamically an external"Harness"(skill libraries, protocols to execute skills and few-shot demonstration). This alternates between training: (1) a Harness, extracts distillation traces and then (2) training with SFT and DAPO. It is important to mention that prior to the second stage (DAPO), our method automatically reconstructs the Harness depending on the successful and failed trajectories from the current policy. The effectiveness of our method has successfully been demonstrated on two cases—route planning and decision making via rules—for a total of 4 multimodal tasks.
Critical strengths
1.This article proposes a multi-stage framework that utilizes the current strategy trajectory to synchronously reconstruct skills, skill invocation rules, and their demonstrations. 2.The author deconstructed and compared the impact of three scenarios: the immediate effect of replacing the Harness, the impact of subsequent strategy adaptation, and the result of consuming the same RL (reinforcement learning) training budget under the old Harness. 3.In the tasks of map planning and rule judgment, this article focuses on analyzing the benefits brought by program refactoring, the specific roles of each component, and the limited ability to migrate across tasks.
Critical weaknesses
1.The title and central results of the article attract viewers' attention with"Co-evolving", however, actually the experiment showed a complete evolution cycle only once. In the discussion of limits, the author mentioned this too. I hope the author can apply N$\geq$3 or even more rounds of iterations on experiments in the future to bring more"persuasiveness"to the article.
Questions for authors
1.The paper has no ablation studies to demonstrate if the performance boost comes from the collaboration from RLHarness or whether it is because of the high-quality trajectories and prompts the strong teacher model produces; the authors can also add some controlled experiments, e.g."teacher model changing","weaker or open-source teacher models"or"no-teacher-models"setting. 2.The authors point out that “visual perception bottlenecks” are among the major reasons for model failure, noting that if one landmark disappears the whole path planning process will go awry. It may also help to add an “oracle” experiment—to run RLHarness but give the model perfect, structured visual evidence—to decouple the visual perception errors from the reasoning errors and so measure the contribution of the suggested evolutionary mechanism at the level of reasoning. 3.When validating and selecting candidate Harnesses () using Qwen3.5-plus as a proxy—with the selection performed by a teacher model (such as GPT-5.6-sol)—might this result in the selection of a version that is optimal for Qwen3.5-plus, rather than one that is optimal for the current Qwen3.5-9B(checkpoint-400)?
3 comments
Follow-up on Reviewer Discussion
Dear Reviewer,
I hope you are doing well. With fewer than three days remaining in the discussion period, we wanted to kindly follow up and ask whether our responses have adequately addressed your concerns. If you have any further questions or feedback, we would greatly appreciate hearing from you.
Thank you again for your time and consideration.
Best regards, The Authors
Reviewer Y2Vi, please respond to authors today
Reviewer Y2Vi,
Today is the last day of the discussion period. The authors have prepared a detailed response to your review. Please reply to them as soon as possible to ensure there is a chance for back and forth discussion before the discussion period ends.
AC
Response to Reviewer Y2Vi
W1
Thank you. We extended the experiment to three evolution rounds. R1-Final is the paper's complete-pipeline endpoint, including Harness I/II and both DAPO stages. R2 and R3 each start from the preceding round's final model and Harness and repeat only Stage 2: update the Harness, then perform 200 additional RL global steps under the new Harness. H denotes evaluation after the Harness update and before further RL; RL denotes evaluation after that round's 200 training steps. Metrics are Accuracy for maps and Macro-F1 for strategy-intensive VL tasks (%).
Later map endpoint scores decline. Both map tasks decline across round endpoints. MetroMap-lite recovers from 59.50% to 60.00% after the third Harness update, then reaches 57.75% after RL; TravelMap-lite declines through the later stages. More update rounds and training steps do not yield continued gains.
Later gains in strategy-intensive VL tasks mainly come from Harness updates. Fee-VL and Cancel-VL depend on specific business rules and decision strategies. They gain 0.91/0.92 points from R2-H and 0.82/0.81 from R3-H. Subsequent RL adds only 0.29/0.08 in R2 and loses 0.12/0.07 in R3. In these two strategy-intensive VL tasks, revising rules and decision strategies remains useful, while further RL brings limited gains.
Stopping should follow task and stage gains. From R2-RL to R3-RL, strategy-intensive VL gains 0.70/0.74 points, while maps lose 1.75/3.25. Future scheduling can use validation gains after each stage to decide separately whether to continue Harness updates and RL.
Q1
All three settings use the same teacher and budget to generate solutions for the student to learn. Each student completes two SFT epochs and 600 RL steps. The Harness contains Skills, usage instructions and examples. Metrics are Accuracy for maps and Macro-F1 for strategy-intensive VL tasks (%).
B versus C tests whether updating the Harness helps. Both share demonstrations and training through step 400. For the last 200 steps, B keeps the old Harness; C uses one revised by the teacher from student successes and failures. C gains 3.00/2.50/14.75/5.14 points. The larger gains on Fee-VL and Cancel-VL highlight the value of revising business rules and decision strategies for these strategy-intensive VL tasks.
A versus C tests free-CoT demonstrations plus RL against the full method. With the same teacher and training schedule, C gains 4.50/3.50/11.81/4.51 points. Trace construction and later Harness updating affect results under the same teacher. Varying teacher capability requires a separate comparison.
Q2
Thank you for this suggestion. This work develops a framework for coordinated Harness updates and reinforcement learning on real multimodal tasks. The model combines visual evidence with reasoning, and RL optimizes this process using feedback on the complete task. We therefore focus on overall performance from visual understanding and reasoning working together. Oracle controls offer a valuable way to analyze their individual contributions. We leave this decomposition to future work, alongside extending the framework to other modalities.
Q3
Qwen3.5-9B checkpoint 400 has been trained with the old Skills. After new Skills are introduced, it may still follow its previous solution habits and need further RL to adapt. Evaluating candidates directly with this checkpoint may therefore undervalue new Skills that require additional learning.
An overly capable model, however, may solve tasks without relying on the Skills, making candidate quality harder to distinguish through scores. We need an evaluator that can understand new Skills while still reflecting differences in their quality.
We therefore use the stronger Qwen3.5-plus from the same family to execute candidates. This motivates our choice; proxy–student ranking agreement requires a separate comparison.