acceptodds
Withdrawn from ICLR 2027

VideoGym: Training Memory-Augmented Video Agents with Modality-Aware Curricula

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Long-video QA involves sparse evidence distributed over extended multimodal contexts, making retrieval-based memory a natural interface for revisiting relevant cues instead of relying on fixed keyframes. We instantiate this interface as VideoAgent, a memory-augmented agent that indexes a video into text, graph, and visual stores and retrieves evidence on demand through a multi-turn planner, with the planner trained by reinforcement learning (RL) to coordinate retrieval across modalities. This training, however, is bottlenecked not by capacity but by the modality skew of natural video QA data: text-heavy questions dominate at 58% of samples, while graph-, visual-, and time-range questions appear in only 15%, 17%, and 10% respectively. Fixed-distribution RL therefore concentrates updates on the already-frequent text mode and under-trains the remaining modalities, and naive reweighting of raw data alone is insufficient because rare modalities are scarce in absolute terms. We introduce VideoGym, a curriculum-based training environment that adapts VideoAgent's training distribution online: VideoGym treats per-batch sample selection as an adversarial bandit over (modality, mode) arms and uses EXP3 algorithm to shift sampling toward arms with low answer-level accuracy To supply the augmented arms, a TaskGenerator-Verifier pipeline produces hard-case variants that remain answerable within the same video. Across five long-video benchmarks, VideoGym lifts RL accuracy from to on average, surpassing standalone GPT-5 () with an 8B backbone. Gains concentrate on the under-represented modalities: 10.8 pp on graph-heavy and 7.8 pp on visual-heavy questions, at sample efficiency.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Multimodal agents, Agentic RL

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 4 reviews, above 27% of the others.
Reviewer_RSZ2OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper explores a modality-aware curriculum in the context of the memory-augmented video agent. To solve the issue of long-tail modality distribution in VideoQA caused by text-heavy questions, this paper proposes a curriculum-based training environment named VideoGym that adapts VideoAgent’s training distribution online. The authors conducted experiments on five long-video benchmarks. The results show that the proposed VideoGym improves performance by a large margin.

Critical strengths
  1. The exploration of modality-aware curriculum in video agents is technically sound and shows great sample efficiency in training.

  2. This paper is well-organized and easy to read. The authors provide code, experimental details, and computing costs, which would help researchers follow the work.

  3. The experimental results on multiple long-video qa benchmarks validated the effectiveness of the proposed VideoGym.

Critical weaknesses
  1. The claim that "performance gap tracks the imbalance in the training distribution" needs more evidence to support it. In Table 2, the results of the training-free method also show a performance gap. Therefore, the performance gap may not be caused by imbalance in the training distribution. The authors should provide more experiments on the distribution of data to validate the claim.

  2. The curriculum algorithm is based on EXP3, and the contribution is limited. In addition, the composite reward formulation (Eq. 4) is different from Fig. 2.

  3. There are some typos. For example, "fine-gained frames" in Fig. 1.

Questions for authors

After reading this paper, my bigger concern is that the motivation of this paper is hard to understand. The authors should give more evidence to support the core claim that "performance gap tracks the imbalance in the training distribution" in the Introduction.

Reviewer_dPDQOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper argues that RL training of retrieval-based long-video QA agents suffers from a modality long tail. In the training pool, text-heavy questions make up about 58% and graph-, visual-, and time-range questions only 10 to 17% each, and as Table 2 shows, after SFT the agent is markedly weaker on the rare types. The authors first build VideoAgent, a Qwen3-VL-8B planner that queries offline-built text, graph, and visual memories over up to 10 turns. It is trained with two-stage SFT on about 96K GPT-4o trajectories and then with GRPO under the composite reward of Equation 4. Their main contribution, VideoGym, treats sampling over 8 arms, one for each combination of modality and raw or augmented questions, as an adversarial bandit and updates the arm distribution with EXP3 from per-arm error rates, following Equations 1 to 3. It fills the augmented arms with harder GPT-4o rewrites that a GPT-4o verifier filters. On five long-video benchmarks, VideoGym raises the average accuracy of RL from 60.4 to 64.8 in Table 1, with the largest gains of +10.5 on graph-heavy and +7.5 on visual-heavy questions, and the paper reports 3.3 times better sample efficiency.

Critical strengths

The paper breaks the training problem down cleanly, in a way that could carry over to other settings. The curriculum reads only binary correctness, while GRPO reads the shaped reward, as described in Section 3.4 and Appendix B.4. This makes the scheduling layer independent of the RL algorithm, and the per-type analysis in Table 2 and Figure 3a ties the motivation to measured weaknesses. I think this framing is useful beyond video. The main comparison is evaluated broadly and the numbers are internally consistent. Table 1 covers five benchmarks and twelve baselines and shows the full training progression. The gain of +4.4 over fixed-distribution RL is reported with standard deviations of 0.5 and 0.6 over three seeds and is clearly larger than them. The averages in Tables 1, 3 and 4 and the compute totals in Table 5 are all arithmetically consistent, and Appendix F.4 discusses failure modes openly.

Critical weaknesses

The evidence for the curriculum mechanism in Figure 3 is difficult to reconcile with the stated update rule. Section 3.3 states on page 5 that every arm starts at a probability of 1/8, which is 0.125, but reading Figure 3b, the curves begin at about 0.145 for graph-aug and about 0.115 for text-raw. With the hyperparameters in Appendix B.2, a single update from the uniform start can raise the weight of an arm by at most about 8%. In my own simulation of Equations 1 to 3 with per-type accuracies close to Table 2, graph-aug reaches about 0.14 by period 16 rather than the 0.18 shown. Because Equation 1 divides the loss of each observed arm by its probability, the distribution settles where the probability of each arm is roughly proportional to its error rate. A return to near-uniform would therefore require the error rates of all arms to become nearly equal, whereas Table 2 still shows 57.5 on graph-heavy and 73.9 on text-heavy questions after training, which implies a probability ratio of about 1.6 between the two. Similarly, at a mean reward of 0.906 with 8 samples per group, I would expect the binary-reward advantage std in Figure 3d to be roughly 0.27, whereas the curve in the figure reads about 0.006. Since these plots are the main evidence in the paper for how VideoGym works, logged per-period values would resolve this. The source of the gain is not yet isolated. Uniform sampling over the same arms, which already directs half of the draws to augmented arms, recovers 1.8 of the 4.4 points in Table 4. The 3.3 times efficiency figure in Section 4.5 counts only raw questions and leaves out the GPT-4o-generated ones. The "+7.5 over the strongest baseline" partly reflects the backbone and the distillation data, since the training-free agent already scores 55.1 and exceeds 11 of 12 re-evaluated baselines. In addition, Tables 2 and 4 report no variance, and Table 3 uses one seed and two benchmarks. It would strengthen the paper considerably to add matched-data controls, report seeds throughout, and show the curriculum effect on an existing agent with its original backbone.

Questions for authors
  1. The starting values of Figure 3b and its return to uniform do not seem attainable under Equations 1 to 3 with the hyperparameters in Appendix B.2. Could the authors share the logged per-period arm probabilities and per-arm accuracies, or clarify whether the figure is illustrative?
  2. How exactly is the "GRPO advantage std" in Figure 3d computed? Is it normalized or not, and is it taken per group or per batch? What does the "raw reward of 0.906" refer to, and how does it relate to the per-type accuracies of 44 to 70% in Figure 3a?
  3. Could the authors add two controls, first natural-distribution sampling with the same augmentation budget, and second static inverse-frequency reweighting with augmentation, each with the mean and standard deviation over three seeds? These would separate the effect of scheduling from the effect of the additional generated questions.
  4. Applying VideoGym to an existing agent such as M3-Agent or Conan on its original backbone would show whether the curriculum transfers. It would also help to compare the re-evaluated baseline numbers with the originally reported ones.
  5. Question types are assigned by keyword rules described in Appendix A.2, and in two of the three case studies in Appendix F the label differs from the store that resolves the question. Could the authors report per-type counts on the evaluation sets, agreement with human labels, and the distribution of retrieval actions per type before and after RL?
Reviewer_rvQXOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies memory-augmented agents for long-video QA. The proposed VideoAgent builds text, graph, and visual memories, and uses a multi-turn planner to retrieve evidence from them.

The main contribution, VideoGym, addresses the imbalance of training questions across retrieval modalities. Since text-heavy questions dominate the training pool, standard RL spends more updates on them. VideoGym treats different (modality, raw/augmented) combinations as arms and uses EXP3 to increase sampling of currently weak modalities. A TaskGenerator and Verifier are also used to create harder training questions.

Experiments on five long-video QA benchmarks show that VideoGym improves the RL baseline from 60.4% to 64.8%, with larger gains on graph-heavy and visual-heavy questions.

Critical strengths
  1. The motivation is reasonable and practically relevant. Giving an agent multiple retrieval tools does not mean all of them will be trained equally well. The paper identifies this issue clearly and proposes a simple way to adapt the training distribution.

  2. The empirical gains are strong and consistent with the motivation. The largest improvements appear on graph-heavy and visual-heavy questions, which are exactly the under-represented categories. The evaluation across five benchmarks and the ablations also make the results fairly convincing.

Critical weaknesses
  1. I am not fully convinced that EXP3 itself is necessary. This is my main concern. The paper compares against fixed-distribution RL and uniform sampling, but not against simpler rebalancing strategies. For example, inverse-frequency sampling, balanced modality sampling, or directly sampling according to moving-average per-modality error seem like natural baselines. Since there are only 8 arms, it is unclear whether the improvement comes from the EXP3 formulation itself, or simply from allocating more training budget to rare/hard modalities.

  2. The modality labels and generated questions need stronger validation. The four modality labels are mainly assigned using heuristic rules based on question wording. I am not sure these labels always correspond to the modality actually required to answer the question. I also have some concern about visual-heavy augmentation. The TaskGenerator and Verifier seem to use captions and entity-relation triples, but not the actual visual frames. If so, it is unclear how the Verifier can reliably check whether a generated visual-heavy question is really grounded in the video.

Questions for authors
  1. How much of the improvement actually requires EXP3? Could the authors compare with simple inverse-frequency/balanced sampling, and maybe sampling based directly on moving-average accuracy of each modality? If these baselines perform clearly worse than EXP3 under the same rollout budget, this would address my concern.

  2. Did the authors evaluate the quality of the modality labels manually? For example, on a random subset, how often does a human agree that a graph-heavy question actually requires graph-type evidence, and similarly for visual/time-range? I think this is important because the long-tail modality distribution is the main motivation of the paper.

  3. How exactly are visual-heavy augmented questions verified? My understanding is that the Verifier only receives captions and entity-relation triples, not the actual visual frames. If this is correct, I don't understand how it can reliably verify visual grounding. A small human evaluation of generated questions for each modality (valid/answerable/correct modality) would be very helpful here.

Reviewer_BYgDOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper presents VideoGym, a training framework for video agents that retrieve evidence from text, graph, and visual memories. The agent is trained through two-stage supervised fine-tuning followed by GRPO. During RL, VideoGym adjusts the training mixture across eight groups, defined by four question types and two modes: original and generated questions. It uses an EXP3-based update to give more weight to groups with lower answer accuracy. A question generator and verifier provide additional training examples.

Across five benchmarks, VideoGym improves average accuracy from 60.4% to 64.8% over RL without the curriculum. The largest gains are reported on graph-heavy and visual-heavy questions.

Critical strengths

1. The main comparison provides useful evidence for the training framework.

RL with and without VideoGym starts from the same SFT checkpoint and uses the same composite reward. VideoGym improves results on all five benchmarks, and the paper reports three-seed variation for the overall RL scores. This is stronger evidence than a comparison based only on the final system and unrelated models.

2. The component studies help identify where the gains come from.

Uniform sampling reaches 62.2%, adaptive sampling without augmentation reaches 63.4%, and the full method reaches 64.8%. These results support the value of adjusting the training mixture and adding new questions. Reusing the video memories also keeps the reported extra training time relatively small.

Critical weaknesses

1. The batch update is not clearly justified as EXP3.

Algorithm 1 draws 128 questions per period, computes the mean accuracy of each observed arm, and then divides its capped error by the single-draw probability (p_z). However, the probability that an arm appears at least once in this batch is (1-(1-p_z)^{128}), even before accounting for rejected augmentations. Thus, updating each observed arm once using its mean error divided by (p_z) is not the usual EXP3 estimate. For two frequently observed arms with the same error, the less likely arm receives a larger update simply because its sampling probability is smaller. Reassigning rejected augmented questions to raw arms further changes the probabilities of the recorded observations. This matters because the paper attributes its main gains and training dynamics to this update.

2. The evidence for learning different retrieval skills is not strong enough.

Appendix A.2 assigns types mainly from question wording: on-screen text maps to text, while “why” and “purpose” map to graph. These rules do not necessarily identify the memory needed to answer a question. Indeed, the text-heavy example in Appendix F.3 needs visual retrieval, and the graph-heavy example in F.2 also relies on visual evidence for the final answer. In addition, the generator and verifier receive captions and graph triples, without direct access to the video frames. Their checks therefore do not establish that generated visual questions are correct in the video or require visual retrieval. The results support gains on the assigned question groups, but the stronger claim about improving specific retrieval skills needs further validation.

Questions for authors

1. Could you derive the batch estimator in Eqs.

(1)–(3) and explain how it handles the 128 draws and rejected augmentations? Does the implementation follow these equations exactly? A comparison with a batch update that accounts for sampling counts would help determine whether this issue affects the results.

2. Could you manually check a sample of original and generated questions against the videos?

Please report answer correctness and whether the assigned question type matches the evidence actually needed. A comparison using captions alone would also help show whether the generated visual questions require visual evidence.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.