- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes FIRE, a post training framework for real-time visual combat. The VLA actor receives raw pixels as input and outputs keyboard/mouse actions. FIRE starts from an SFT policy trained on expert play and runs 10 online rounds with 2 streams (RL stream & Supervised stream). The key component is failure-informed recovery demo acquisition. A VLM diagnoses mistakes from rollouts and groups them into failure modes. Recurring failure modes make requests for record expert’s recovery demonstrations, and retrieved by failure label for policy updates. FIRE reports improvements in Boss HP Reduction and Win Rate across seven in-domain environments under two attribute settings compared with the considered baselines. The paper also provides several ablation studies to analyze the contributions of individual components
Critical strengths
S1. The paper uses a VLM to automates the decision of what human experts should record (failure diagnosis, recording requests, and demo labeling and retrieval). Human experts only record offline demos between rounds, without required continuous monitoring, takeover, or state resets. This matters in real-time game, where such supervision is hard to provide.
Critical weaknesses
W1. The proposed framework largely combines existing components, such as aggregating human demonstrations, VLM-based failure detection, and VLM-derived rewards. While the integration is practically useful, the methodological novelty appears limited.
W2. The main results in Table 1 primarily compare variants of the proposed method and therefore resemble an ablation study rather than a comprehensive comparison against external baselines. The evaluation would be substantially stronger with additional baselines, such as standard RL methods (e.g., PPO) and intervention-based post-training methods. Without such comparisons, it is difficult to assess how much of the reported improvement is attributable specifically to FIRE relative to existing alternatives.
W3. All experiments appear to be conducted with a single random seed, and the paper does not report standard deviations or confidence intervals. Given the potentially high variance in RL training and evaluation, results across multiple seeds are important for assessing the reliability and statistical significance of the reported improvements. Together, W2 and W3 make it difficult to draw strong conclusions from the empirical results.
Questions for authors
Q1. Could you add more external baselines, such as standard RL and intervention-based post-training methods? these methods should use the same SFT policy initialization and a comparable training/data budget to ensure a fair comparison.
Q2. Could you report results across multiple training seed (3+) and confidence intervals on experiments?
5 comments
Response to Reviewer hDyE(Part 2)
W2. Comparison with external baselines
The main results in Table 1 primarily compare variants of the proposed method and therefore resemble an ablation study rather than a comprehensive comparison against external baselines.
We added PPO and HG-DAgger, and FIRE achieves higher final HP reduction and win rate than both in all four conditions. All methods start from the same SFT checkpoint. We evaluate PPO at the ten-round main budget and at thirty rounds because critic fitting was difficult in the multi-boss setting. The latter is an extended-budget comparison. Each checkpoint is evaluated in 20 episodes per condition.
| Method | Rounds | Wind (50/600) | Bear (50/600) | Wind (native) | Bear (native) |
|---|---|---|---|---|---|
| PPO | 10 | 53.43 / 10 | 57.71 / 15 | 40.37 / 0 | 44.20 / 0 |
| PPO | 30 | 56.12 / 10 | 62.90 / 20 | 44.12 / 0 | 50.13 / 5 |
| HG-DAgger | 10 | 62.39 / 20 | 68.56 / 30 | 53.36 / 0 | 52.58 / 5 |
| FIRE | 10 | 70.95 / 35 | 75.07 / 40 | 58.57 / 10 | 59.76 / 20 |
PPO improves with more training, but FIRE at ten rounds achieves higher HP reduction and win rate in all four conditions. The results are in Appendix B.1, Table 10 (p. 31).
HG-DAgger uses approximately 250 minutes of human effort, matching FIRE. The expert continuously monitors the agent and takes over when correction is needed. Monitoring and takeover exhaust this budget by the end of Round 3 and produce about 35 minutes of intervention video. We then stop new interventions and continue training with the collected corrections, mixed by boss with SFT demonstrations. This baseline does not use VLM failure labels or recording requests.
The ten-round comparisons use approximately 770 minutes of regular rollout across seven bosses. FIRE also uses about 90 minutes of diagnostic interaction with the fixed SFT policy before training, which we report separately. Evaluations are performed by the model alone, without human takeover.
We also report the intermediate checkpoint at Round 3, when the HG-DAgger human budget is exhausted, in Appendix B.2, Table 12 (p. 32). HG-DAgger has higher HP reduction than FIRE in three conditions at that point. FIRE continues to improve and achieves better final results in all four conditions. This makes the benefit of targeted recovery collection visible over the course of training.
Response to Reviewer hDyE(Part 3)
W3. Reliability across independent training seeds
All experiments appear to be conducted with a single random seed, and the paper does not report standard deviations or confidence intervals.
Across three independent RL training seeds, FIRE achieves higher mean HP reduction and win rate than Static recovery in all four conditions. We added two runs per method from the same selected SFT checkpoint and evaluated each Round-10 checkpoint in 20 episodes per boss and attribute setting. The table reports mean ± sample SD. HP and WR denote HP reduction and win rate (%).
| Condition | Method | HP | WR |
|---|---|---|---|
| Black Wind King 50/600 | FIRE | 69.74 ± 3.42 | 30.00 ± 5.00 |
| Black Wind King 50/600 | Static | 60.88 ± 2.28 | 21.67 ± 2.89 |
| Black Bear Guai 50/600 | FIRE | 73.33 ± 2.56 | 38.33 ± 2.89 |
| Black Bear Guai 50/600 | Static | 68.20 ± 2.52 | 30.00 ± 5.00 |
| Black Wind King native | FIRE | 58.14 ± 3.38 | 11.67 ± 2.89 |
| Black Wind King native | Static | 52.25 ± 2.06 | 1.67 ± 2.89 |
| Black Bear Guai native | FIRE | 61.68 ± 2.00 | 21.67 ± 2.89 |
| Black Bear Guai native | Static | 53.26 ± 3.03 | 8.33 ± 2.89 |
The following approximate 95% confidence intervals for mean HP reduction use the three run-level means and a Student-t distribution with two degrees of freedom: mean ± 4.303 × SD / √3.
| Condition | Method | 95% interval |
|---|---|---|
| Black Wind King 50/600 | FIRE | [61.25, 78.22] |
| Black Wind King 50/600 | Static | [55.23, 66.53] |
| Black Bear Guai 50/600 | FIRE | [66.97, 79.69] |
| Black Bear Guai 50/600 | Static | [61.94, 74.46] |
| Black Wind King native | FIRE | [49.75, 66.54] |
| Black Wind King native | Static | [47.13, 57.37] |
| Black Bear Guai native | FIRE | [56.71, 66.66] |
| Black Bear Guai native | Static | [45.73, 60.80] |
The original Figure 3 contains one training run per method. The new results measure variation across three training runs at the final checkpoint, with evaluation noise from the 20 episodes also contributing to that variation. PPO at each budget, HG-DAgger, and the fixed-label control are reported with one training run. The comparison tables use the original FIRE and Static runs. These details and the uncertainty estimates are now in Appendix B.4, Tables 14–15 (p. 33).
To show evaluation variability within each checkpoint, Appendix B.6, Figure 19 (p. 35) plots episode outcomes in 10-percentage-point bins, with wins shown separately and the three runs distinguished in each bar. Across 60 episodes per method and condition, FIRE wins 18, 23, 7, and 13 games, compared with 13, 18, 1, and 5 for Static recovery, in the condition order used above.
Response to Reviewer hDyE(Part 1)
Thank you for your careful review. We have added the new experiments and analyses to Appendix B of the revised PDF, with the additions marked in green.
W1. What FIRE contributes beyond its individual components
FIRE closes the loop between policy failures, human demonstration collection, and policy updates. Its contribution is an adaptive correction mechanism that connects curated replay, VLM diagnosis, and expert demonstrations through shared policy experience and an evolving failure vocabulary.
Rule-based feedback and VLM assessments jointly identify informative successes and mistakes for replay, allowing subsequent RL updates to revisit them. The VLM also diagnoses mistake chunks and turns them into specific recovery requests. FIRE prioritizes uncovered failure modes, then allocates recording effort by failure frequency relative to demonstration coverage. Human experts recreate the relevant situations and demonstrate recovery actions. The VLM assigns these demonstrations the same failure labels used for policy mistakes, enabling retrieval by boss and failure mode for supervised updates alongside RL. Replay revisits the actor’s experience, while experts supply corrective action examples. VLM review and human recording occur outside real-time gameplay.
The evolving labels connect diagnosis, collection, and training. During collection rounds, newly observed failures can introduce more specific labels and redirect recording and retrieval as the policy changes. The matched-video controls retain hybrid feedback, replay, RL, and supervised learning while varying demonstration acquisition. Our added fixed-label control further retains VLM diagnosis and adaptive recording priorities but keeps the initial failure vocabulary fixed. We evaluate Round 10 with 20 episodes per condition.
Wind and Bear denote Black Wind King and Black Bear Guai. Cells report HP reduction / win rate (%).
FIRE exceeds the fixed-label control by 2.90–6.76 points in HP reduction and 10 points in win rate across all four conditions. Priority alone improves HP reduction over Static recovery in three conditions, without improving win rate. These results support expanding the failure vocabulary alongside adaptive recording priorities (Appendix B.3, Table 13, p. 32).