- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The authors propose UGPO (Use-Grouped Policy Optimization), a framework for credit assignment in long-context memory agents. Instead of broadcasting a trajectory-level reward to every memory operation, the method records the observable lifecycle of each memory item, from writing through retrieval to grounded use in a later response, and routes task, use and cost credit only along these paths. The comparison is done at item-use level, not under exact memory-state equality, because the authors show with a small bound that grouping by exact state lose coverage exponentially when more free-form writes are accumulated. Rewards are normalized for each channel before aggregation, and the delayed uses are replayed to supervise the writes they come from, so there is no need to regenerate the interaction history. Experiments are done on four benchmarks with nine baselines under same budget.
Critical strengths
For me, the theoretical part is a real contribution. The coverage bound for exact-state grouping is simple and clear. It shows where the state-relative methods can be applied and where not in the free-form memory setting. The new grouping key is designed to not include any sampled memory content or reward. This removes the circularity concern, that the groups are built from the outcome which they should predict.
The evaluation protocol is also very complete. The authors fix the three reward channels and only change the assignment operator, so the source of gain can be separated. For statistics, all tables report five seeds with standard deviation, and the main comparison use paired bootstrap intervals with Holm correction. The authors also build failure cases for their own method. In addition, the rollout multiplier keep at one since all supervision comes from the original trace. This is a structural advantage on efficiency compared to the baselines that need branched continuations.
Critical weaknesses
All differences are statistically significant, but compared to the strongest baseline, the improvement on macro average is less than one point, and on each single benchmark it is at most 1.2. The large improvements are mainly on attribution and pollution metrics. So the practical value of this framework depend on how much a real system can benefit from more precise credit and cleaner store, not on the task numbers. For readers who mostly care about task improvement, I am not sure this trade-off is explained clear enough.
Questions for authors
-
For a practitioner who already have a memory agent in service, what is the benefit on task performance if they switch to UGPO? Are the improvements on attribution and pollution worth the extra cost of trace infrastructure and the tuning of multiple channels in real deployment?
-
The grouping key is a combination of item buckets, request strata and delay bins, and there are many design choices in how to combine them. How did you decide this combination? When applying to a new workload, how much of the key need to be redesigned? Besides the sensitivity scan, is there a more principled way to choose the key?
Dear Reviewer bjz2,
We sincerely thank you for your rigorous review and valuable suggestions.
If our rebuttal has addressed your concerns, we kindly hope you might reconsider your evaluation of our paper.
Wish you a wonderful life.
Best regards,