- Rating
- 1
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
FlowRSI evaluates frozen-weight agents that revise persistent prompts, memories, or skills after practice and two mock exams. It independently replays four committed states on the same hidden set of 1,348 flowcharts. The benchmark contains 5,484 images and distinguishes strict relation-set F1, recoverable content, output validity, and visual-domain transfer. Nine systems exhibit improvements, regressions, and early termination; artifact-removal experiments investigate selected final strategies.
Critical strengths
- The evaluation separates mechanisms that endpoint accuracy would obscure. For Qwen3.5-27B, the 55.47-point strict-F1 improvement accompanies only a 6.88-point content-F1 improvement and a substantial increase in output validity (Tables 2–3). This usefully demonstrates why serialization repair should not be equated with improved visual understanding.
- Independent replay, mandatory final-state reporting, and artifact-removal interventions make destructive revisions inspectable. Removing Qwen3.5-9B’s final artifacts recovers 27.48 points in the matched intervention replay (Table 5). The clean, corrupted, and unseen-writer strata add a useful transfer diagnostic.
Critical weaknesses
- System-level conclusions rely on one observed improvement episode per system. Repeated evaluation of fixed states and item-bootstrap intervals do not measure variability in evidence acquisition, diagnosis, or strategy revision. The paper explicitly acknowledges this limitation, but the stability of the improvement and regression patterns remains unresolved for a benchmark intended to compare these capabilities.
- The diagnostic value beyond simpler adaptation controls is insufficiently established. Large strict-score changes are dominated by output-contract behavior, yet the evaluation lacks a fixed format-focused strategy and an equal-budget no-feedback revision control. Content F1 and the careful non-causal interpretation are valuable, but they do not establish how much improvement requires feedback-driven strategy revision rather than a better initial instruction or additional trials.
Questions for authors
- How stable are final gains, destructive revisions, and early-submission patterns across independently restarted full improvement episodes?
- What improvement remains over a fixed format-focused strategy and an equal-budget no-feedback revision control, reporting strict F1, content F1, and validity separately?
2 comments
Dear Authors,
Thank you for the detailed rebuttal and the additional experiments. I appreciate the effort to provide independent trajectories, stage-wise analyses, and stronger feedback controls. These additions improve the transparency of the evaluation. However, my main concerns regarding the benchmark's construct validity and the attribution of recursive self-improvement remain unresolved.
1. The new controls raise a fundamental concern about feedback attribution.
The no-feedback trajectories achieve strict F1 improvements of +0.6400, +0.5348, and +0.2431, compared with +0.3502 and +0.3044 for the reported 27B full-feedback trajectories. More importantly, the no-feedback controls achieve Content F1 improvements of +0.0847, +0.0667, and +0.0516, while the corresponding full-feedback improvements are −0.0046 and +0.0383.
The shuffled-feedback trajectories also reach final strict F1 scores of 0.3109 and 0.4520, comparable to the full-feedback endpoints of 0.3820 and 0.3421.
Although the limited number of trajectories prevents definitive statistical comparisons, these results provide little evidence that meaningful feedback produces a consistent advantage over additional revision opportunities or corrupted feedback.
This directly affects the central construct that FlowRSI aims to evaluate: the ability of agents to use feedback to improve their strategies.
2. Improvements in output compliance remain difficult to separate from substantive self-improvement.
The additional analyses reinforce the observation that much of the strict F1 improvement is associated with output validity and serialization repair. The considerably smaller Content F1 changes, particularly under full feedback, provide limited evidence of improved visual understanding or structural reasoning.
The distinction between strict F1, Content F1, and validity is useful. However, the current evidence leaves the contribution of feedback-specific structural learning insufficiently established. The fixed-format diagnostic also leaves open the need for a quantitative, matched comparison isolating this contribution.
3. The additional trajectories improve reproducibility evidence but leave the broader benchmark conclusions insufficiently supported.
The repeated experiments provide useful examples of improvement, degradation, and early submission. Nevertheless, the substantial variation across trajectories and the limited replication across models make it difficult to draw reliable conclusions about system-level recursive self-improvement capabilities.
I recognize the value of FlowRSI as an auditable evaluation framework for persistent strategy revisions and their failures. However, its central interpretation as a benchmark of feedback-driven recursive self-improvement requires stronger evidence that the measured changes meaningfully distinguish feedback utilization from generic strategy iteration and output-contract repair.
For these reasons, I maintain my original rating of 1 (Clear Rejection). The additional experiments are valuable, but the central methodological concerns remain sufficiently substantial to affect my recommendation.
Stage-wise controls expose revision stability and destructive updates
Thank you for treating revision stability, destructive updates, and early submission as measurable outcomes.
The completed trajectories provide the requested stage-wise evidence.
Q1. Stability of gains, degradation and early submission
The table reports revision paths rather than only endpoint gains. The 27B trajectories finish positive but differ in their final-step behavior. The 9B trajectories finish negative. No-feedback C recovers part of its H2 loss at H3.
Q2. Fixed-format and equal-budget no-feedback controls
All three controls use the same action budget, hidden-test protocol, and evaluator, releasing only mock-exam completion. The results show that extra strategy iterations can improve output compliance while also producing destructive revisions. The fixed-format replay supplies a frozen JSON schema and is reported as a format diagnostic.